Arbenz Peter, Numerical Methods For Computational Science and Engineering
Uploaded by
anthalyaArbenz Peter, Numerical Methods For Computational Science and Engineering
Uploaded by
anthalyaNumerical Methods for Computational Science and Engineering
Numerical Methods for Computational Science
and Engineering
Lecture 1, Sept 18, 2014: Introduction
Peter Arbenz
Computer Science Department, ETH Z
urich
E-mail: [email protected]
NumCSE, Lecture 1, Sept 18, 2014
1/48
Numerical Methods for Computational Science and Engineering
Introduction
Outline of todays lecture
I
What is numerical methods for CSE
Survey of the lecture
Organization of the lecture (exercises/examination)
References
Start of the lecture
NumCSE, Lecture 1, Sept 18, 2014
2/48
Numerical Methods for Computational Science and Engineering
Introduction
Scientific Computing
NumCSE, Lecture 1, Sept 18, 2014
3/48
Numerical Methods for Computational Science and Engineering
A CSE example
A CSE example: particle accelerators
Observed phenomenon
I
Charged particles are accelerated in electromagnetic fields.
I
Cathode ray tubes: beam of electrons deflected by e-field to
create image (classic TV set).
Particle accelerator is a device to propel charged particles to
high speeds by applying electromagnetic fields.
I
Paul Scherrer Institut: Numerous accelerator to investigate
material properties, or for cancer treatment.
CERN: Large hadron collider (LHC) to generate new
elementary particles.
Linear accelerators vs. cyclotrons. Demo: cyclo.m
NumCSE, Lecture 1, Sept 18, 2014
4/48
Numerical Methods for Computational Science and Engineering
A CSE example
A CSE example: particle accelerators (cont.)
NumCSE, Lecture 1, Sept 18, 2014
5/48
Numerical Methods for Computational Science and Engineering
A CSE example
A CSE example: particle accelerators (cont.)
Model
1. Electric fields are modeled by the time-harmonic Maxwell
equations (periodic solution (E(x, t) = e(x)e it ), magnetic
field eliminated, all pysical quantities = 1)
curl curl e(x) = e(x),
n e = 0,
x ,
div e(x) = 0,
x .
is accelerator cavity.
2. Particles move according to Newtons law of motion
dx(t)
= v,
dt
dv(t)
q
=
(E + v B) .
dt
m0
where E = Eext + Eself . Eext is E-field from 1. (q charge, m0
rest mass.)
NumCSE, Lecture 1, Sept 18, 2014
6/48
Numerical Methods for Computational Science and Engineering
A CSE example
A CSE example: particle accelerators (cont.)
Discretization
1. e-field in the Maxwell equation is a continuous function.
Approximate it by a finite element function.
This leads to a large sparse
generalized eigenvalue problem
Ax = Mx.
Size can be in millions.
2. Particles are particles.
Big issue is the computation of the self-field, i.e.,
particle-particle interaction.
NumCSE, Lecture 1, Sept 18, 2014
7/48
Numerical Methods for Computational Science and Engineering
A CSE example
A CSE example: particle accelerators (cont.)
Discretization (cont.)
I
Computing particle-particle interactions costs O(np2 )
operations in the number of particles np .
Remedy: Approximately compute the electric potential (x)
that is due to the charged particles. To that end we have to
solve the Poisson equation
(x) =
1
(x)
0
with some boundary conditions.
The quantity (x) on the right is the charge density which is
in fact approximated on a regular grid.
NumCSE, Lecture 1, Sept 18, 2014
8/48
Numerical Methods for Computational Science and Engineering
A CSE example
A CSE example: particle accelerators (cont.)
Also the solution (x) is obtained on this regular grid.
The gradient of (x) provides the force that acts on the
particle at position x,
Eself = grad (x).
NumCSE, Lecture 1, Sept 18, 2014
9/48
Numerical Methods for Computational Science and Engineering
A CSE example
A CSE example: particle accelerators (cont.)
Solution algorithms
I
We need to solve systems of linear equations
Ax = b,
where A is large and sparse, maybe positive definite.
In a time-dependent problem we may have to solve such a
system in every time step.
For the Poisson problem on regular grids there are fast
Poisson solvers that are related to the fast Fourier transform.
We need to solve eigenvalue problems
Ax = Mx,
where A and M are large and sparse to get the driving e-field.
NumCSE, Lecture 1, Sept 18, 2014
10/48
Numerical Methods for Computational Science and Engineering
A CSE example
A CSE example: particle accelerators (cont.)
Efficiency, accuracy
I
These problems are to be solved efficiently w.r.t. time and
memory.
Linear vs. nonlinear models.
Direct vs. iterative solution of linear systems.
High vs. relaxed accuracy.
Repeated solution of equal/similar problems may amortize
expensive solution procedures.
NumCSE, Lecture 1, Sept 18, 2014
11/48
Numerical Methods for Computational Science and Engineering
Survey of the lecture
Survey of the lecture
1. Introduction
2. Roundoff errors
3. Nonlinear equations in one variable (2 lectures)
4. Linear algebra basics (Read sections 4.14.4 in AscherGreif)
5. Direct methods for linear system, pivoting strategies, sparse
matrices (2)
6. Linear least squares problems (2)
7. Iterative methods for linear system (2)
8. Eigenvalues and singular values (2)
9. Nonlinear systems and optimization (3)
10. (Piecewise) polynomial interpolation (3)
NumCSE, Lecture 1, Sept 18, 2014
12/48
Numerical Methods for Computational Science and Engineering
Survey of the lecture
Survey of the lecture (cont.)
11. Best approximation
12. Filtering algorithms, Fourier transform
13. Numerical differentiation
14. Numerical integration (2)
15. Ordinary differential equations, initial value problems (3)
NumCSE, Lecture 1, Sept 18, 2014
13/48
Numerical Methods for Computational Science and Engineering
About this course
About this course
Focus
I
I
I
on algorithms (principles, scope, and limitations),
on (efficient, stable) implementations in Matlab,
on numerical experiments (design and interpretation).
No emphasis on
I
theory and proofs (unless essential for understanding of
algorithms)
hardware-related issues (e.g. parallelization, vectorization,
memory access)
(These aspects will be covered in the course
High Performance Computing for Science and Engineering
offered by D-INFK)
NumCSE, Lecture 1, Sept 18, 2014
14/48
Numerical Methods for Computational Science and Engineering
About this course
Goals
Knowledge of the fundamental algorithms in numerical
mathematics
Knowledge of the essential terms in numerical mathematics
and the techniques used for the analysis of numerical
algorithms
Ability to choose the appropriate numerical method for
concrete problems
Ability to interpret numerical results
Ability to implement numerical algorithms efficiently in
Matlab
Indispensable: Learning by doing ( exercises)
NumCSE, Lecture 1, Sept 18, 2014
15/48
Numerical Methods for Computational Science and Engineering
About this course
Literature
Uri Ascher & Chen Greif: A First Course in Numerical
Methods. SIAM, 2011.
http://www.siam.org/books/cs07/
NumCSE, Lecture 1, Sept 18, 2014
Excellent reference.
Main reference for large parts of this
course.
Target audience: undergraduate students
in computer science.
I will follow this book quite closely.
16/48
Numerical Methods for Computational Science and Engineering
About this course
Literature (cont.)
W. Dahmen & A. Reusken: Numerik f
ur Ingenieure und
Naturwissenschaftler, Springer, 2006.
A lot of simple examples and good explanations, but also rigorous mathematical treatment. Target
audience: undergraduate students in science and engineering.
H.-R. Schwarz & N. K
ockler: Numerische Mathematik.
Teubner, 2006. 6. Auflage.
Easy to read. Target audience: undergraduate students in science and engineering.
C. Moler: Numerical Computing with Matlab. SIAM 2004.
Good reference for some parts of this course; Target audience: Matlab users and programmers. See
http://www.mathworks.ch/moler/.
W.Gander, M.J. Gander, & F. Kwok: Scientific Computing.
An introduction using Maple and Matlab. Springer 2014.
NumCSE, Lecture 1, Sept 18, 2014
17/48
Numerical Methods for Computational Science and Engineering
About this course
Prerequisites
Essential prerequisite for this course is a solid knowledge in linear
algebra and calculus. Familiarity with the topics covered in the
first semester courses is taken for granted, see
K. Nipp and D. Stoffer, Lineare Algebra, vdf
Hochschulverlag, Z
urich, 5 ed., 2002.
M. Gutknecht, Lineare algebra, lecture notes, SAM, ETH
Z
urich, 2009.
http://www.sam.math.ethz.ch/~mhg/unt/LA/HS07/.
M. Struwe, Analysis f
ur Informatiker. Lecture notes, ETH
Z
urich, 2009.
NumCSE, Lecture 1, Sept 18, 2014
18/48
Numerical Methods for Computational Science and Engineering
Organization
Organization
Lecturer:
Prof. Peter Arbenz
Assistants:
Daniel Hupp
Christian Sch
uller
Alexander Lobbe
Sharan Jagathrakashakan
Alexander Bohn
Manuel Moser
Fabian Th
uring
Timo Welti
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
NumCSE, Lecture 1, Sept 18, 2014
19/48
Numerical Methods for Computational Science and Engineering
Organization
Venue
Classes:
Tutorials:
Mon 10.15-12.00 (CAB G11); Thu 10.15-12.00 (HG G5)
Mon 13.15-15.00
Thu 8.15-10.00
Please register (on course website) for tutorial groups until
September 23th:
http://www.math.ethz.ch/education/bachelor/lectures/
hs2014/math/nummath_cse
Consulting hours: if needed, see the course website.
NumCSE, Lecture 1, Sept 18, 2014
20/48
Numerical Methods for Computational Science and Engineering
Organization
Assignments
The assignment sheets will be uploaded on the course
webpage on Monday every week the latest.
The exercise should be solved until the following tutorial class.
(Hand them in to the assistant.)
NumCSE, Lecture 1, Sept 18, 2014
21/48
Numerical Methods for Computational Science and Engineering
Organization
Examination
Three-hour written examination involving coding problems to
be done at the computer on
TBA
Dry-run for computer based examination:
Does not exist anymore.
Try out a computer in the student labs in HG.
Pre-exam question session:
TBA
NumCSE, Lecture 1, Sept 18, 2014
22/48
Numerical Methods for Computational Science and Engineering
Organization
Examination (cont.)
Topics of examination:
I All topics, that have been addressed in class or in a
homework assignment.
I One exam question will be one of the homework
assignment.
Lecture slides will be available as (a single) PDF file during
the examination.
The Ascher-Greif book will be made available, too.
The exam questions will be asked in English.
NumCSE, Lecture 1, Sept 18, 2014
23/48
Numerical Methods for Computational Science and Engineering
Computing environment: Matlab
Problem solving environment: Matlab
I
I
I
I
We use Matlab for the exercises.
Although most of the algorithm we are dealing with have been
implemented in Matlab, it is useful when you program them
yourselves.
These (little) programs will be building blocks when you will
solve more complex problems in your future.
Matlab help
I
I
I
I
Matlab commands help/doc
Matlab online documentation, e.g.,
http://www.mathworks.nl/help/pdf_doc/allpdf.html
Numerous introductory textbooks / user guides / primers
My own Matlab introduction:
http://people.inf.ethz.ch/arbenz/MatlabKurs/
matlabintro.pdf
NumCSE, Lecture 1, Sept 18, 2014
24/48
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Numerical algorithms and errors
I
The most fundamental feature of numerical computing is the
inevitable presence of errors.
The result of any interesting computation (and of many
uninteresting ones) is typically only approximate, and our goal
is to ensure that the resulting error is tolerably small.
Example: How many loop iterations are there in this little
Matlab program?
x = 0; h = 1/10;
while x<1,
x=x+h;
% do something depending on x
end
NumCSE, Lecture 1, Sept 18, 2014
25/48
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
How to measure errors
I
Can measure errors as absolute or relative, or a combination
of both.
The absolute error in v approximating given scalar quantity u
is |u v |.
|u v |
.
The relative error (assuming u 6= 0) is
|u|
NumCSE, Lecture 1, Sept 18, 2014
1
1
1.5
100
100
0.99
1.01
1.2
99.99
99
Absolute
Error
0.01
0.01
0.3
0.01
1
Relative
Error
0.01
0.01
0.2
0.0001
0.01
26/48
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Approximation example
The Stirling approximation
v = Sn =
2n
n n
e
is used to approximate u = n! = 1 2 n for large n.
The formula involves the constant e = exp(1) = 2.7182818 . . ..
We use the Matlab program Example1 1.m from
http://www.siam.org/books/cs07/programs.zip
to compute the values of u and v for n = 1, . . . , 10.
Notice that the quality of v approximating u increases although
the absolute error increases rapidly!
NumCSE, Lecture 1, Sept 18, 2014
27/48
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Types of errors
I
Errors in the formulation of the problem to be solved.
I
I
Approximation errors
I
I
I
Errors in the mathematical model. Simplifications.
Error in input data. Measurements.
Discretization error.
Convergence error in iterative methods.
Discretization/convergence errors may be assessed by an
analysis of the method used.
Roundoff errors
I
Roundoff errors arise everywhere in numerical computation
because of the finite precision arithmetic.
Roundoff errors behave quite erratic.
NumCSE, Lecture 1, Sept 18, 2014
28/48
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Discretization errors in action
Problem: want to approximate the derivative f 0 (x0 ) of a given
smooth function f (x) at the point x = x0 .
Example: Let f (x) = sin(x), < x < , and set x0 = 1.2.
Thus, f (x0 ) = sin(1.2) 0.932 . . ..
Discretization: Function values f (x) are available only at a discrete
number of points, e.g. at grid points xj = x0 + jh, j Z.
Want to approximate f 0 (x0 ) by values f (xj ).
NumCSE, Lecture 1, Sept 18, 2014
29/48
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Discretization errors in action (cont.)
Taylors series gives us an algorithm to approximate f 0 (x0 ):
NumCSE, Lecture 1, Sept 18, 2014
f 0 (x0 )
f (x0 + h) f (x0 )
h
30/48
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Discretization errors in action (cont.)
Expanding f (x) by a Taylor series around x = x0 gives
f (x0 + h) f (x0 )
h
= f 0 (x0 ) f 00 (),
h
2
x0 < < x0 + h.
So, we expect the error to decrease linearly with h.
h 00 h 00
0
f
(x
+
h)
f
(x
)
0
0
f (x0 )
= f () f (x0 )
h
2
2
Or, using the big-O notation:
0
f (x0 ) Dx ,h (f ) = O(h).
0
NumCSE, Lecture 1, Sept 18, 2014
31/48
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Results
Try for f (x) = sin(x) at x0 = 1.2.
(So, we are approximating cos(1.2) = 0.362357754476674 . . .)
h
0.1
0.01
0.001
104
107
Absolute error
4.71667 102
4.666196 103
4.660799 104
4.660256 105
4.619326 108
These results reflect the discretization error as expected.
Note that f 00 (x0 )/2 = sin(1.2)/2 0.466.
NumCSE, Lecture 1, Sept 18, 2014
32/48
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Results for smaller h
The above results indicate that we can compute the derivative as
accurate as we like, provided that we take h small enough.
If we wanted
cos(1.2) sin(1.2 + h) sin(1.2) < 1010 .
h
We have to set h 1010 /0.466.
The following numbers and plot are generated by
.../Greif/programs/chap01/Example1 3Figure1 3.m
NumCSE, Lecture 1, Sept 18, 2014
33/48
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Results for smaller h
h
108
109
1010
1011
1013
1015
1016
Absolute error
4.36105 1010
5.594726 108
1.669696 107
7.938531 106
6.851746 104
8.173146 102
3.623578 101
These results reflect both discretization and roundoff errors.
NumCSE, Lecture 1, Sept 18, 2014
34/48
Numerical Methods for Computational Science and Engineering
Numerical algorithms and errors
Results for all h
The solid curve interpolates the computed values of
(x0 )
|f 0 (x0 ) f (x0 +h)f
| for f (x) = sin(x) at x0 = 1.2.
h
The dash-dotted straight line depicts the discretization error
without roundoff error.
NumCSE, Lecture 1, Sept 18, 2014
35/48
Numerical Methods for Computational Science and Engineering
Algorithm properties
Algorithm properties
Performance features that may be expected from a good numerical
algorithm.
I
Accuracy
Relates to errors. How accurate is the result going to be when
a numerical algorithm is run with some particular input data.
Efficiency
I
I
I
How fast can we solve a certain problem?
Rate of convergence. Floating point operations (flops).
How much memory space do we need?
These issues may affect each other.
Robustness
(Numerical) software should run under all circumstances.
Should yield correct results to within an acceptable error or
should fail gracefully if not successful.
NumCSE, Lecture 1, Sept 18, 2014
36/48
Numerical Methods for Computational Science and Engineering
Algorithm properties
Complexity I
Complexity/computational cost of an algorithm : number of
elementary operators
Asymptotic complexity =
leading order term of complexity w.r.t.
large problem size parameters
The usual choice of problem size parameters in numerical linear
algebra is the number of independent real variables needed to
describe the input data (vector length, matrix sizes).
operation
inner product
outer product
tensor product
matrix product
description
(x Rn , y Rn ) 7 xH y
(x Rm , y Rn ) 7 xyH
#mul/div
n
nm
#add/sub
n1
0
O(n)
O(mn)
(A Rmn , B Rnk ) 7 AB
mnk
mk(n 1)
O(mnk)
NumCSE, Lecture 1, Sept 18, 2014
37/48
Numerical Methods for Computational Science and Engineering
Algorithm properties
Big-O and notation
For an error e depending on h we denote
e = O(hq )
if there are two positive constants q and C such that
|e| C hq
h > 0 small enough.
Similarly, for w = w (n) the expression
w = O(n log n)
means that there is a constant C > 0 such that
|w | Cn log n
NumCSE, Lecture 1, Sept 18, 2014
as n .
38/48
Numerical Methods for Computational Science and Engineering
Algorithm properties
Big-O and notation
More abstract:
Class O(f ) of functions is defined as
O(f ) = {g | c1 , c2 > 0 : N Z+ : g (N) c1 f (N) + c2 }
The notation signifies a stronger relation than the O notation:
a function (h) for small h (resp., (n) for large n) is ((h))
(resp., ((n))) if is asymptotically bounded both above and
below by .
Example:
O(h2 ) means at least quadratic convergence (see later). (h2 )
is exact quadratic convergence.
NumCSE, Lecture 1, Sept 18, 2014
39/48
Numerical Methods for Computational Science and Engineering
Algorithm properties
Complexity II
To a certain extent, the asymptotic complexity allows to predict
the dependence of the runtime of a particular implementation of
an algorithm on the problem size (for large problems). For
instance, an algorithm with asymptotic complexity O(n2 ) is likely
to take 4 as much time when the problem size is doubled.
One may argue that the memory accesses are more decisive for run
times than floating point operations. Often there is a linear
dependence among the two. So, there is no difference in the O
notation.
NumCSE, Lecture 1, Sept 18, 2014
40/48
Numerical Methods for Computational Science and Engineering
Elementary operations
Scaling
Scaling multiplication with diagonal matrices (with non-zero
diagonal entries) from left and/or right.
It is important to know the different effects of multiplying with a
diagonal matrix from left or right:
DA
vs. AD
with Rnn 3 A, D = diag(d1 , . . . , dn )
Scaling with D = diag(d1 , . . . , dn )
in Matlab:
y = diag(d)*x;
or
y = d.*x;
NumCSE, Lecture 1, Sept 18, 2014
41/48
Numerical Methods for Computational Science and Engineering
Elementary operations
Elementary matrices
Matrices of the form A = I + uvT are called elementary.
Again we can apply A to a vector x in a straightforward and a
more clever way:
Ax = (I + uvT )x
or
Ax = x + u(vT x)
Cf. exercises.
NumCSE, Lecture 1, Sept 18, 2014
42/48
Numerical Methods for Computational Science and Engineering
Elementary operations
Problem conditioning and algorithm stability
Qualitatively speaking:
I
The problem is ill-conditioned if a small perturbation in the
data may produce a large difference in the result.
The problem is well-conditioned otherwise.
The algorithm is stable if its output is the exact result of a
slightly perturbed input.
NumCSE, Lecture 1, Sept 18, 2014
43/48
Numerical Methods for Computational Science and Engineering
Elementary operations
An unstable algorithm
Ill-conditioned problem of computing output values y from input
values x by y = g (x): when x is slightly perturbed to x, the result
y = g (
x ) is far from y .
NumCSE, Lecture 1, Sept 18, 2014
44/48
Numerical Methods for Computational Science and Engineering
Elementary operations
A stable algorithm
An instance of a stable algorithm for computing y = g (x): the
output y is the exact result, y = g (
x ), for a slightly perturbed
input, i.e., x which is close to the input x. Thus, if the algorithm is
stable and the problem is well-conditioned, then the computed
result y is close to the exact y .
NumCSE, Lecture 1, Sept 18, 2014
45/48
Numerical Methods for Computational Science and Engineering
Elementary operations
Unstable algorithm
Problem statement: evaluate the integrals
Z 1
xn
dx, for n = 0, 1, 2, . . . , 30.
yn =
0 x + 10
Algorithm development: observe that analytically, for n > 0,
Z 1 n
Z 1
x + 10x n1
1
yn + 10yn1 =
dx =
x n1 dx = .
x + 10
n
0
0
Also,
Z
y0 =
0
1
dx = log(11) log(10).
x + 10
Algorithm:
I Evaluate y0 = log(1.1).
I For n = 1, 2, . . . , 30, evaluate yn =
NumCSE, Lecture 1, Sept 18, 2014
1
n
10yn1 .
46/48
Numerical Methods for Computational Science and Engineering
Elementary operations
Unstable algorithm (cont.)
Run the Matlab program Example1 6.m by Ascher and Greif to
see the catastrophic amplification of roundoff errors.
This code is available from
http://www.siam.org/books/cs07/programs.zip.
NumCSE, Lecture 1, Sept 18, 2014
47/48
Numerical Methods for Computational Science and Engineering
Elementary operations
Unstable algorithm (cont.)
Roundoff error accumulation
I
In general, if En is error after n elementary operations, cannot
avoid linear roundoff error accumulation
En ' c0 nE0 .
Will not tolerate an exponential error growth such as
En ' c1n E0 ,
for some constant c1 > 1.
This is an unstable algorithm.
NumCSE, Lecture 1, Sept 18, 2014
48/48
Numerical Methods for Computational Science and Engineering
Introduction
Goals for today
I
Unterstand how numbers are stored in a computer
Get a feeling for the almost random nature of rounding error
Understand how standard floating point systems are designed
and implemented
Understand how roundoff errors can cause wrong results
References for today
I
Ascher & Greif: Chapter 2
N. J. Higham: Accuracy and stability of numerical algorithms.
2nd ed. SIAM 2002.
W. Gander, M. Gander, F. Kwok: Scientific Computing.
Springer 2014.
NumCSE, Lecture 2, Sept 22, 2014
2/34
Numerical Methods for Computational Science and Engineering
Introduction
Introduction
One of the important tasks of numerical mathematics is the
determination of the accuracy of results of some computation.
There are three types of errors that limit accuracy:
1. Errors in the mathematical model of the problem to be solved.
Simplified models are easier to solve (shape of objects,
unimportant chemical reactants).
2. Discretization or approximation errors depend on the chosen
algorithm or the type of discretization.
I
May occur even when computing without rounding error:
e =1+
1
1
1
+
+
+
1! 2! 3!
is approximated by a finite number of terms (truncation error).
Function is approximated by, e.g., a piecewise linear function
NumCSE, Lecture 2, Sept 22, 2014
3/34
Numerical Methods for Computational Science and Engineering
Introduction
Introduction (cont.)
I
I
A differential equation by a difference equation.
Iteration is discontinued prematurely.
3. Rounding errors occur if a real number (probably an
intermediate result of some computation) is rounded to the
next nearest machine number.
The propagation of rounding errors from one floating point
operation to the next is the most frequent source of numerical
instabilities.
Since computer memory is finite practically no real number
can be represented exactly in a computer.
We discuss floating point numbers as a representation of real
numbers.
NumCSE, Lecture 2, Sept 22, 2014
4/34
Numerical Methods for Computational Science and Engineering
Introduction
Motivating example: quadrature of a circle
Lets try to compute , the area of a circle with radius r = 1.
We approximate by the area of an inscribed regular polygon:
n := 2
n
Fn = cos 2n sin 2n
An = nFn = n cos
n
n
sin
2
2
as n
[See Gander, Gander, & Kwok: Scientific Computing. Springer 2014.]
NumCSE, Lecture 2, Sept 22, 2014
5/34
Numerical Methods for Computational Science and Engineering
Introduction
Motivating example: quadrature of a circle (cont.)
Of course, we cannot use to compute .
The procedure is to compute A2n from An with an initial n0 for
which An0 does not depend on .
n
n n
n
2
n
2 cos
sin
= sin n = sin
.
An = nFn =
2
2
2
2
2
n
s
p
r
n
1 cos n
1 1 sin2 n
sin 2n = sin
=
=
.
2
2
2
n0 = 6,
6 = 2/n0 = 60 ,
sin 6 =
2
NumCSE, Lecture 2, Sept 22, 2014
6/34
Numerical Methods for Computational Science and Engineering
Introduction
Motivating example: quadrature of a circle (cont.)
s=sqrt(3)/2; A=3*s; n=6;
% initialization
z=[A-pi n A s];
% store the results
while s>1e-10
% terminate if s=sin(alpha) small
s=sqrt((1-sqrt(1-s*s))/2); % new sin(alpha/2) value
n=2*n; A=n/2*s;
% A = new polygonal area
z=[z; A-pi n A s];
end
for i=1:length(z)
fprintf(%10d %20.15f %20.15f %20.15f\n,z(i,2),z(i,3),z(i,1)
end
NumCSE, Lecture 2, Sept 22, 2014
7/34
Numerical Methods for Computational Science and Engineering
Binary representation of numbers
Integers
Also integers suffer from the finiteness of computers.
Matlab represents integers by 32-bit signed ints (in the twos
complement format)
30
P
falls a31 = 0
ai 2 i ,
i=0
a=
30
P
(232
ai 2i ), falls a31 = 1
i=0
Therefore the range is 2147483648 a 2147483647. These
numbers are given by intmin and intmax, respectively.
Note that in Matlab the ints do not cycle when you shoot
across intmax.
Also note that you may be limited in problem size.
NumCSE, Lecture 2, Sept 22, 2014
8/34
Numerical Methods for Computational Science and Engineering
Binary representation of numbers
Real numbers
A number x R (in the binary number system) has the form
x = (1.d1 d2 d3 dt1 dt dt+1 ) 2e
e is an integer exponent, the binary digits di are either 0 or 1.
d1 d2 d3
+ 2 + 3 +
2
2
2
In general, infinitely many digits are needed to represent a real
number.
1.d1 d2 d3 = 1 +
The choice of a binary representation is just one of many
possibilities. It is, indeed, a convenient choice when it comes to
computers.
NumCSE, Lecture 2, Sept 22, 2014
9/34
Numerical Methods for Computational Science and Engineering
Binary representation of numbers
Examples of real numbers
1. 1.1012 2 = (1 + 1/2 + 1/8) 2 = 3.25
2. (10011.01)2 = 24 + 21 + 20 + 22 = 16 + 2 + 1 + 41 = (19.25)
P
P
1 m
3. (0.010101 . . .)2 =
22m = 41
= 14 11 1 = 13
4
m=1
4.
1
5
m=0
= (0.2) = (0.00110011 . . .)2
The last example is of interest insofar as it shows that to a finite
decimal number there may correspond a (nontrivial) infinite binary
representation. (This is not true the other way round. Why?)
So, one cannot assume that a finite decimal number is exactly
representable on a binary computer.
[See Gautschi: Numerical Analysis, 2012]
NumCSE, Lecture 2, Sept 22, 2014
10/34
Numerical Methods for Computational Science and Engineering
Binary representation of numbers
Floating point representation
Given any real number in binary format
d1 d2 d3
dt1 dt
dt+1
x = 1 + 1 + 2 + 3 + + t1 + t + t+1 + 2e
2
2
2
2
2
2
e
= (1.d1 d2 d3 dt1 dt dt+1 ) 2
We somehow represent x in the computer as
fl(x) = sign(x) (1.d1 d2 d3 dt1 dt ) 2e
for some fixed t. Thus we need t bits for the so-called mantissa.
Sign requires 1 bit of storage.
Exponent range: L e U
NumCSE, Lecture 2, Sept 22, 2014
11/34
Numerical Methods for Computational Science and Engineering
Binary representation of numbers
Floating point representation (cont.)
How big can the relative error
|fl(x) x|
|x|
become?
NumCSE, Lecture 2, Sept 22, 2014
12/34
Numerical Methods for Computational Science and Engineering
Binary representation of numbers
Error in floating point number representation
I
Chopping
fl(x) = (1.d1 d2 d3 dt ) 2e
Absolute error is bounded by 2t 2e
Rounding
(
(1.d1 d2 d3 dt ) 2e ,
fl(x) =
(1.d1 d2 d3 dt + 2t ) 2e ,
0.dt+1 dt+2 . . . < 1/2
0.dt+1 dt+2 . . . > 1/2
In case of a tie (0.dt+1 dt+2 . . . = 1/2) round to nearest even
number.
Absolute error bounded by 21 2t 2e .
Relative error bounded by rounding unit, machine precision
NumCSE, Lecture 2, Sept 22, 2014
1
= 2t .
2
13/34
Numerical Methods for Computational Science and Engineering
IEEE floating point numbers
IEEE floating point numbers
ANSI/IEEE Standard 754-1985 for Binary Floating Point
Arithmetic. Acording to the IEEE standard a 32-bit float has the
following structure (from en.wikipedia.org)
The exponent has 8 bits, the mantissa 23 Bit. There is a sign bit.
The value of a normalized 32-bit IEEE floating point number V is
V = (-1)S x 2(E-127) x (1.M)
Normalized means 0 < E < 255 = 28 1. (127 is called a bias.)
NumCSE, Lecture 2, Sept 22, 2014
14/34
Numerical Methods for Computational Science and Engineering
IEEE floating point numbers
IEEE floating point numbers (cont.)
I
double:
1 sign bit
11 bits exponent
52 bits mantissa
The value of a normalized 64-bit IEEE floating point number
V is
V = (-1)S x 2(E-1023) x (1.M)
Normalized means, that 0 < E < 2047 = 211 1.
NumCSE, Lecture 2, Sept 22, 2014
15/34
Numerical Methods for Computational Science and Engineering
IEEE floating point numbers
IEEE floating point numbers (cont.)
If the exponent has only zeros or ones, there is a special cases:
I
I
I
0 (zero): e = 0, m = 0, s arbitrary.
-Infinity, +Infinity: e = all ones, m = 0.
e = all ones, m 6= 0: NaN
There are also non-normalized numbers.
NumCSE, Lecture 2, Sept 22, 2014
16/34
Numerical Methods for Computational Science and Engineering
IEEE floating point numbers
Rounding errors in IEEE
Parameters
Precision
single
double
extended
of IEEE Standard arithmetics with base E = 2.
t
emin
emax
24
23
-125
128
2
6 108
53
52 -1021
1024 2
1 1016
63 -16381 16384 264 5 1020
From the above we can derive the following statements
Lemma
If x 6= 0 is a normalized floating point number and fl(x) obtained
after rounding with t digits, then
|fl(x) x| 2et /2
|fl(x) x|
2t /2
x
NumCSE, Lecture 2, Sept 22, 2014
(max. absolute rounding error)
(max. relative rounding error)
17/34
Numerical Methods for Computational Science and Engineering
Rounding errors
Rounding errors
We assume that all numbers are normalized.
Let t be the length of the mantissa.
Between powers of 2, the floating point numbers are equidistant.
Here, the length of the mantissa is t = 2 and 2 e 2.
Definition
Machine precision = 2(t+1) (half of Matlabs eps)
This is half the distance of the numbers between 1 and 2.
NumCSE, Lecture 2, Sept 22, 2014
18/34
Numerical Methods for Computational Science and Engineering
Rounding errors
Rounding errors are random
Note how the sign
of the floating point
representation error
at nearby arguments
t fluctuates as if
randomly: as a
function of t it is a
non-smooth
error.
Error in sampling exp(t)(sin(2t) + 2) in single precision.
NumCSE, Lecture 2, Sept 22, 2014
19/34
Numerical Methods for Computational Science and Engineering
Rounding errors
Floating point arithmetic
Important to use exact rounding: if x and y are machine numbers,
then
fl(x y ) = (x y )(1 + 1 ),
fl(x y ) = (x y )(1 + 2 ),
|i | .
fl(x/y ) = (x/y )(1 + 3 ),
In other words: The result of a basic operation with two floating
point numbers yields a result that is correct up to a relative error
smaller than .
Thus, the relative errors remain small after each such operation.
This is achieved only using guard digits (intermediate higher
precision).
NumCSE, Lecture 2, Sept 22, 2014
20/34
Numerical Methods for Computational Science and Engineering
Rounding errors
Guard digit
Floating point system with = 10 and t = 4. So, = 21 103 . Let
x = .1103, = 1.103 101 ,
y = 9.963 103 .
Then, x y = .100337. Hence, exact rounding yields 0.1003.
Relative error:
|0.1003370.1003|
0.100337
0.37 103 < .
However, if we were to subtract these two numbers without guard
digits we would obtain .1103 .0099 = 0.1004. Now the obtained
relative error is |0.1003370.1004|
0.63 103 > .
0.100337
Thus, guard digits must be used to produce exact rounding.
NumCSE, Lecture 2, Sept 22, 2014
21/34
Numerical Methods for Computational Science and Engineering
Rounding errors
Rounding error example
For t = 5 we have = 26 = 0.015625.
fl() = fl(21 + 20 + 23 + 25 + ) = 21 + 20 + 23 = 1.10010 21
|fl() |
0.0053
fl()fl() = 10.01110001 22
fl(fl()fl()) = 1.00111 23 = 9.75
So,
NumCSE, Lecture 2, Sept 22, 2014
2 fl(fl()fl()) 0.12
2 fl(fl()fl())
0.012
2
22/34
Numerical Methods for Computational Science and Engineering
Rounding errors
Note on machine epsilon
I
For any number with || we have
fl(1 + ) = 1.
The Matlab command eps returns = 252 , i.e., the
smallest positive number (for the data type double) for which
1 + eps > 1.
eps can have a parameter, see help eps.
In the finite difference example of last week we had for very
small h that fl(f (x + h)) = fl(f (x)). Therefore,
fl(f (x0 + h)) fl(f (x0 ))
0
fl(f (x0 )) = fl(f 0 (x0 )) .
h
NumCSE, Lecture 2, Sept 22, 2014
23/34
Numerical Methods for Computational Science and Engineering
Rounding errors
Rounding errors summary
Lemma
1. With the machine precision we have
fl(x) = x(1 + ) with || .
2. If is an elementary operation then
fl(x y ) = (x y )(1 + ) with || .
Wilkinsons Principle
The result of a numerical computation on the computer
is the exact result with slightly perturbed initial data.
This also holds for good implementations of (library) functions!
NumCSE, Lecture 2, Sept 22, 2014
24/34
Numerical Methods for Computational Science and Engineering
Cancellation
Cancellation
Cancellation (dt. Ausl
oschung) is a special kind of rounding error.
Consider the following two numbers with 5 decimal digits:
1.2345 e0
1.2344 e0
0.0001 e0 = 1.0000 e4
If the two numbers were exact, the result delivered by the
computer would also be exact. But if the first two numbers had
been obtained by previous calculations and were affected by
rounding errors, then the result would at best be 1.xxxx e4,
where the digits denoted by x are unknown.
NumCSE, Lecture 2, Sept 22, 2014
25/34
Numerical Methods for Computational Science and Engineering
Cancellation
Cancellation (cont.)
Suppose z = x y , where x y . Then
|z fl(z)| |x fl(x)| + |y fl(y )|,
from which it follows that the relative error satisfies
|x fl(x)| + |y fl(y )|
|z fl(z)|
,
|z|
|x y |
Numerator: OK.
If the denominator is very close to zero, x y , then the relative
error in z can become large.
NumCSE, Lecture 2, Sept 22, 2014
26/34
Numerical Methods for Computational Science and Engineering
Avoiding cancellation
Subtracting two almost equal numbers
Let a = 1.2 and b = 1.1. We compare two algorithms to compute
a2 b 2 = (a + b)(a b) in decimal arithmetic with two essential
digits.
a2 = 1.44 = fl(a2 ) = 1.4
b 2 = 1.21 = fl(b 2 ) = 1.2
fl(a2 ) fl(b 2 ) = 0.2
(13% error)
a + b = fl(a + b) = 2.3
a b = fl(a b) = 0.10
NumCSE, Lecture 2, Sept 22, 2014
(a + b)(a b) = 0.23
(0% error)
27/34
Numerical Methods for Computational Science and Engineering
Avoiding cancellation
Subtracting two almost equal numbers (cont.)
9970 2 = 9801 9800 =
The 3 formula give the following
# positions 99 70 2
2
1.0
4
0.02000
6
0.00530000
10
0.005050660000
= 0.0050506338833466 .
9801 + 9800
results
9801 9800
0.0
0.01000
0.00510000
0.005050640000
1
9801+ 9800
0.0050
0.005051
0.00505063
0.005050633884
The reason for the numerical instability of the first two algorithms
is the subtraction of almost equal numbers (indicated by appended
zeros).
NumCSE, Lecture 2, Sept 22, 2014
28/34
Numerical Methods for Computational Science and Engineering
Avoiding cancellation
Library function for sinh
The sinus hyperbolicus is defined as
y = sinh(x)
1 x
e e x .
2
If x 0 we have to expect cancellation in the two terms.
However, using the Taylor expansion of sinh:
sinh(x) = x +
5
x3
+
,
6
120
|| < x.
For small enough x we can expect very good approximations.
NumCSE, Lecture 2, Sept 22, 2014
29/34
Numerical Methods for Computational Science and Engineering
Remark on overflow
Remark on overflow
An overflow is obtained when a number is too large to fit into the
floating point system in use.
q
Example: kxk2 = x12 + x22 + + xn2
To make things simple, let n = 2.
If x = (1060 , 1)T , then the exact result is kxk2 = 1060 .
But in the course of the computation we have to form x12 = 10120 .
To avoid overflow, we scale with a positive scalar s, e.g.,
s = max(|xi |)
r
x 2
x1 2 x2 2
n
kxk2 = s
+
+ +
s
s
s
NumCSE, Lecture 2, Sept 22, 2014
30/34
Numerical Methods for Computational Science and Engineering
Quadrature of a circle revisited
Quadrature of a circle revisited
In our initial example to compute by the inscribed regular
polynoms we experienced severe cancelation.
To compute sin(/2) from sin , we used the recurrence:
s
p
n
1 1 sin2 n
sin
=
.
2
2
Since sin n 0, the numerator on the right
p
1 1 2 with small = sin n ,
suffers from severe cancellation.
Therefore the algorithm performed so badly, although theory and
program are both correct.
NumCSE, Lecture 2, Sept 22, 2014
31/34
Numerical Methods for Computational Science and Engineering
Quadrature of a circle revisited
Quadrature of a circle revisited (cont.)
It is possible to rearrange the computation and avoid cancellation:
s
p
n
1 1 sin2 n
sin
=
2
2
v
p
p
u
u 1 1 sin2 n 1 + 1 sin2 n
p
=t
2
1 + 1 sin2 n
s
1 (1 sin2 n )
p
=
2(1 + 1 sin2 n )
sin n
=r
.
p
2 1 + (1 sin n )(1 + sin n )
NumCSE, Lecture 2, Sept 22, 2014
32/34
Numerical Methods for Computational Science and Engineering
Quadrature of a circle revisited
Quadrature of a circle revisited (cont.)
oldA=0; s=sqrt(3)/2; A=3*s; n=6;
% initialization
z=[A-pi n A s];
% store the results
while A>oldA
% terminate if area does
oldA=A;
%
not increase
s=s/sqrt(2*(1+sqrt((1-s)*(1+s)))); % new sin() value
n=2*n; A=n/2*s;
% A = new polygonal area
z=[z; A-pi n A s];
end
for i=1:length(z)
fprintf(%10d %20.15f %20.15f\n,z(i,2),z(i,3),z(i,1))
end
Stopping criterion exploits fact that A6 < < An < A2n <
NumCSE, Lecture 2, Sept 22, 2014
33/34
Numerical Methods for Computational Science and Engineering
Quadrature of a circle revisited
Paper on high precision computation
David H. Bailey, Roberto Barrio, and Jonathan M. Borwein, High
precision computation: Mathematical physics and dynamics,
Applied Mathematics and Computation, vol. 218 (2012),
pp. 1010610121.
http://dx.doi.org/10.1016/j.amc.2012.03.087
Gist of the paper:
In many very large scale problems it is difficult to achieve sufficient
accuracy: for a rapidly growing body of important scientific computing
applications, a higher level of numeric precision is required.
This is facilitated by high-precision software packages.
Software available, but awfully slow.
NumCSE, Lecture 2, Sept 22, 2014
34/34
Numerical Methods for Computational Science and Engineering
Introduction
Goals
I
Want to compute zeros of nonlinear scalar equations.
Want to develop useful methods for a basic, simply stated
problem, including such favourites as fixed point iteration and
the Newtons method.
Want to develop and assess several algorithmic concepts that
are prevalent throughout the field of numerical computing.
Want to study basic algorithms for minimizing a function in
one variable.
NumCSE, Lecture 3, Sept 25, 2014
2/34
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
I
Bisection algorithm
Fixed point iterations
Reference
I
AscherGreif, Chapter 3, Sections 3.13.3.
Next time
I
Newton and other higher order methods
Minimizing a function in one variable
AscherGreif, Chapter 3, Sections 3.43.5.
Matlabs fzero, a combination of some basic methods.
NumCSE, Lecture 3, Sept 25, 2014
3/34
Numerical Methods for Computational Science and Engineering
The problem
The problem
Want to find solutions of the scalar nonlinear equation
f (x) = 0
with continuous f : [a, b] R 7 R
We denote a solution of the equation (called root, or zero) by x .
In contrast to scalar linear equations
ax b = 0
=
a6=0
x =
b
a
nonlinear equations have an undetermined number of zeros.
We denote the set of all continuous functions on the interval [a, b] by C [a, b].
So, above, we require f C [a, b].
NumCSE, Lecture 3, Sept 25, 2014
4/34
Numerical Methods for Computational Science and Engineering
The problem
Examples
1. f (x) = x 1 on [a, b] = [0, 2].
2. f (x) = sin(x)
On [a, b] = [ 2 , 3
2 ] there is one root x = .
On [a, b] = [0, 4] there are five roots, cf. Fig. on next page.
3. f (x) = x 3 30x 2 + 2552 on [0, 20].
4. f (x) = 10 cosh(x/4) on < x <
cosh(t) =
NumCSE, Lecture 3, Sept 25, 2014
1
2
(e t + e t )
5/34
Numerical Methods for Computational Science and Engineering
The problem
Examples (cont.)
NumCSE, Lecture 3, Sept 25, 2014
6/34
Numerical Methods for Computational Science and Engineering
The problem
Why study a nonlinear problem before addressing linear
ones?
I
One linear equation, e.g. ax = b, is too simple, whereas a
system of linear equations can have many complications.
Several important general methods can be described in a
simple context.
Several important algorithm properties can be defined and
used in a simple context.
NumCSE, Lecture 3, Sept 25, 2014
7/34
Numerical Methods for Computational Science and Engineering
The problem
Iterative methods for finding roots
I
Roots that can be expressed analytically are known for very
few special nonlinear functions.
Even for polynomials this holds only for very low orders.
We have to resort to iterative methods:
Starting with an initial guess/iterate x0 we generate a
sequence of iterates x1 , x2 , . . . that (hopefully) converges to a
root of the function.
A rough knowledge of the roots location is required.
(Plot)
Could probe the function and try to find two arguments a, b
s.t. f (a) f (b) < 0.
Intermediate value theorem = x in interval (a, b).
NumCSE, Lecture 3, Sept 25, 2014
8/34
Numerical Methods for Computational Science and Engineering
The problem
Desirable algorithm properties
Iterative method: starting with initial iterate (guess) x0 , generate
sequence of iterates x1 , x2 , . . . , xk , . . ., that hopefully converge to a
root x .
I
Efficient: requires a small number of function evaluations.
Robust: fails rarely, if ever. Announces failure if it does fail.
Requires a minimal amount of additional information such as
the derivative of f .
Requires f to satisfy only minimal smoothness properties.
Generalizes easily and naturally to many equations in many
unknowns.
NumCSE, Lecture 3, Sept 25, 2014
9/34
Numerical Methods for Computational Science and Engineering
Bisection
Bisection
I
Method for finding a root of scalar equation f (x) = 0 in an
interval [a, b].
Assumption: f (a)f (b) < 0 .
Since f is continuous there must be a zero x [a, b].
(This follows from intermediate value theorem.)
Compute midpoint m of the interval and check the value f (m).
Depending on the sign of f (m), we can decide if x [a, m] or
x [m, b].
(Of course, if f (m) = 0 then we are done.)
Robust code on next page from book by Gander et al.
NumCSE, Lecture 3, Sept 25, 2014
10/34
Numerical Methods for Computational Science and Engineering
Bisection
Code bisect
function [x,fx] = bisect(f,a,b,tol)
%BISECT [x,y] = bisect(f,a,b) computes a zero of f(x) = 0
% in interval [a,b] assuming that f(a)f(b) < 0.
fa = f(a);
fb = f(b);
if fa == 0, x = a; fx = fa; return; end
if fb == 0, x = b; fx = fb; return; end
if fa*fb > 0, error(f(a)*f(b) > 0), end
x = (a+b)/2; fx = f(x);
if nargin<4, tol=0; end
NumCSE, Lecture 3, Sept 25, 2014
11/34
Numerical Methods for Computational Science and Engineering
Bisection
Code bisect (cont.)
while (b-a>tol) & ((a < x) &
if fx == 0, break, end
if fa*fx < 0,
b = x; fb = fx;
else
a = x; fa = fx;
end
x = (a+b)/2; fx = f(x);
end
(x < b)),
try with
f=@(x) cos(x)*cosh(x)+1
[x,fx] = bisect(f,0,3,1e-9)
[x,fx] = bisect(f,0,3)
NumCSE, Lecture 3, Sept 25, 2014
12/34
Numerical Methods for Computational Science and Engineering
Bisection
Bisection (cont)
I
If L0 = b a then Lk := L0 /2k is length of search interval in
step k of the procedure.
a
Error estimate: |xk x | b k+1
2
with xk = midpoint of the k-th interval.
If we want the error to satisfy |xk x | , then it suffices to
have (b a)/2k+1 , so that
ba
ba
= log
/ log 2.
k + 1 > log2
Note the machine independent stopping criterion in bisect!
NumCSE, Lecture 3, Sept 25, 2014
13/34
Numerical Methods for Computational Science and Engineering
Bisection
Properties of bisection
I
Simple
Safe, robust (even foolproof)
Requires only that f be continuous
Slow
Hard to generalize to systems
NumCSE, Lecture 3, Sept 25, 2014
14/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Fixed point iteration
An intuitively appealing approach which often leads to simple
algorithms for complicated problems.
The methods discussed now have direct extensions to more
complicated problems, e.g., to systems of nonlinear equations
and to more complex functional equations.
Rewrite the given problem
f (x) = 0
as
x = g (x).
()
so that f (x ) = 0 g (x ) = x .
x is a fixed point of the mapping g .
NumCSE, Lecture 3, Sept 25, 2014
15/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Algorithm: Fixed point iteration
Given a scalar function f (x). Select a function g (x) such that
f (x) = 0
g (x) = x.
Then:
1. Start from an initial guess x0 .
2. For k = 0, 1, 2, . . . set
xk+1 = g (xk ),
k = 0, 1, . . .
until xk+1 satisfies some termination criterion
It is all in the choice of the function g .
NumCSE, Lecture 3, Sept 25, 2014
16/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Examples of fixed point iterations
Note: there are many ways to transform f (x) = 0 into fixed point
form! Not all of them are good in terms of convergence.
We are looking for a zero of
f (x) = xe x 1,
x [0, 1]
Plot function: fplot(@(x) x*exp(x)-1, [0 1], 1e-3)
Different fixed point forms are possible, e.g.:
g1 (x) = e x ,
1+x
g2 (x) =
,
1 + ex
g3 (x) = x + 1 xe x .
NumCSE, Lecture 3, Sept 25, 2014
17/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Examples of fixed point iterations (cont.)
k
0
1
2
3
4
5
6
7
8
9
10
xk+1 := g1 (xk )
0.500000000000000
0.606530659712633
0.545239211892605
0.579703094878068
0.560064627938902
0.571172148977215
0.564862946980323
0.568438047570066
0.566409452746921
0.567559634262242
0.566907212935471
NumCSE, Lecture 3, Sept 25, 2014
xk+1 := g2 (xk )
0.500000000000000
0.566311003197218
0.567143165034862
0.567143290409781
0.567143290409784
0.567143290409784
0.567143290409784
0.567143290409784
0.567143290409784
0.567143290409784
0.567143290409784
xk+1 := g3 (xk )
0.500000000000000
0.675639364649936
0.347812678511202
0.855321409174107
-0.156505955383169
0.977326422747719
-0.619764251895580
0.713713087416146
0.256626649129847
0.924920676910549
-0.407422405542253
18/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Examples of fixed point iterations (cont.)
k
0
1
2
3
4
5
6
7
8
9
10
|xk x |
0.067143290409784
0.039387369302849
0.021904078517179
0.012559804468284
0.007078662470882
0.004028858567431
0.002280343429460
0.001294757160282
0.000733837662863
0.000416343852458
0.000236077474313
NumCSE, Lecture 3, Sept 25, 2014
|xk x |
0.067143290409784
0.000832287212566
0.000000125374922
0.000000000000003
0.000000000000000
0.000000000000000
0.000000000000000
0.000000000000000
0.000000000000000
0.000000000000000
0.000000000000000
|xk x |
0.067143290409784
0.108496074240152
0.219330611898582
0.288178118764323
0.723649245792953
0.410183132337935
1.186907542305364
0.146569797006362
0.310516641279937
0.357777386500765
0.974565695952037
19/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Some questions regarding the fixed point iteration
Suppose that we have somehow determined the continuous
function g C [a, b].
Obvious questions arise about fixed point iteration xk+1 = g (xk ):
1. Is there a fixed point x in [a, b]?
2. If yes, is it unique?
3. Does the sequence of iterates converge to a root x ?
4. If yes, how fast?
5. If not, does this mean that no root exists?
NumCSE, Lecture 3, Sept 25, 2014
20/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Fixed point theorem
If g C [a, b] and a g (x) b for all x [a, b], then there is a
fixed point x in the interval [a, b].
If, in addition, the derivative g 0 exists and there is a constant
< 1 such that the derivative satisfies
|g 0 (x)|
x (a, b),
then the fixed point x is unique in this interval.
See AscherGreif book for a proof.
(This answers questions 1 and 2.)
NumCSE, Lecture 3, Sept 25, 2014
21/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Convergence of the fixed point iteration
3. Does the sequence of iterates converge to a root x ?
|xk+1 x | = |g (xk ) g (x )| = |g 0 ()| |xk x | |xk x |
with [xk , x ].
This is a contraction if the factor < 1. Thus,
|xk+1 x | |xk x | 2 |xk1 x | k+1 |x0 x |.
Since < 1 then k 0 as k .
We have convergence.
NumCSE, Lecture 3, Sept 25, 2014
22/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Convergence of fixed point iterations in 1D
Vastly different behavior of different fixed point iterations:
g1 : linear convergence?
NumCSE, Lecture 3, Sept 25, 2014
g2 : quadratic convergence?
g3 : no convergence?
23/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Geometric interpretation of fixed point iteration
x0
g (x0 )
x1 = g (x0 )
g (x1 )
etc.
start with x0 on the x-axis
go parallel to the y -axis to the graph of g
move parallel to the x-axis to the graph y = x
go parallel to the y -axis to the graph of g
NumCSE, Lecture 3, Sept 25, 2014
24/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
left: 1 < g 0 (x ) 0
at least linear convergence
right: g 0 (x ) < 1
divergence
Note: these are local scenarios.
NumCSE, Lecture 3, Sept 25, 2014
25/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
left: 0 g 0 (x ) < 1
at least linear convergence
right: 1 < g 0 (x )
divergence
Note: these are local scenarios.
NumCSE, Lecture 3, Sept 25, 2014
26/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Rate of convergence
Let x be a fixed point of the iteration xk+1 = g (xk ) and
= |g 0 (x )|
with
0 < < 1.
x0 sufficiently close to x = xk x g 0 (x )(xk1 x ).
|xk x | |xk1 x | k |x0 x |.
is called contraction factor.
Definition: The rate of convergence is defined as rate = log10 .
The rate is higher (faster convergence), the smaller .
NumCSE, Lecture 3, Sept 25, 2014
27/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Rate of convergence (cont.)
About k = d1/ratee iteration steps are needed to reduce the error
by one order of magnitude: k < 0.1.
0.999
0.99
0.90
0.50
0.10
NumCSE, Lecture 3, Sept 25, 2014
rate
0.00044
0.0044
0.046
0.30
1
k
2302
230
22
4
1
28/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Rate of convergence (for earlier numerical example)
The rate of convergence for xk+1 = gi (xk ) is in the previous
scenarios for solving f (x) = xe x 1, with x [0, 1].
Fixed point is x = 0.5671432904.
g10 (x ) = g1 (x ) = x = 0.5671432904.
0 < |g10 (x )| < 1 convergence.
g30 (x ) = 1 x e x e x = 1.76 . . . divergence.
x
f (x )
g20 (x ) = 1 x e 2 =
2 = 0,
(e x + 1)
(e x + 1)
g200 (x ) 6= 0 quadratic convergence (see next time: order of
convergence)
NumCSE, Lecture 3, Sept 25, 2014
29/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Termination criteria
Residual based termination:
|f (xk )|
STOP convergent iteration {xk }, if
=
prescribed tolerance > 0
= no guaranteed accuracy
f
|f (xk )| small 6 |x x | small
NumCSE, Lecture 3, Sept 25, 2014
|f (xk )| small |x x | small
30/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Termination criteria (cont.)
Correction based termination: STOP convergent iteration {xk }, if
abs
|xk+1 xk | or
rel |xk |
abs /rel are prescribed absolute / relative tolerances > 0
Usually (but not always) the relative criterion is more robust than
the absolute one.
A combination of the first two is
NumCSE, Lecture 3, Sept 25, 2014
|xk+1 xk | < (1 + |xk |)
31/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
A posteriori termination criterion
A posteriori termination criterion for linearly convergent iterations
with rate of convergence 0 < < 1:
|xk+1 x | 1 |xk+1 xk |
That is: is we know (or can estimate) then the formula tells us
how close we are to the solution x .
Proof.
Idea of the proof: Bound |xk+m xk+1 |, m > 1, independent of m
and let m .
|xk+2 xk+1 | = |g (xk+1 ) g (xk )| |xk+1 xk |
NumCSE, Lecture 3, Sept 25, 2014
32/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
A posteriori termination criterion (cont.)
|xk+m xk+1 | = |xk+m xk+m1 + xk+m1 xk+m2 +
+ xk+3 xk+2 + xk+2 xk+1 |
|xk+m xk+m1 | + |xk+m1 xk+m2 | +
+ |xk+3 xk+2 | + |xk+2 xk+1 |
m1
m2
m2
+ + )|xk+1 xk |
m3
+ + 1)|xk+1 xk |
1 m1
|xk+1 xk | <
|xk+1 xk |
1
1
m > 2
Let m then xk+m x .
NumCSE, Lecture 3, Sept 25, 2014
33/34
Numerical Methods for Computational Science and Engineering
Fixed point iteration
Useful results from analysis
I
Intermediate value theorem. If f C [a, b] and s is a value
such that f (a) s f (b), then there exists a real number
c [a, b] for which f (c) = s.
Mean value theorem. If f C [a, b] and f is differentiable on
the open interval (a, b), then there exists a real number
(a)
.
c (a, b) for which f 0 (c) = f (b)f
ba
Rolles theorem. If f C [a, b] and f is differentiable on (a, b),
and in addition f (a) = f (b) = 0, then there is a real number
c (a, b) for which f 0 (c) = 0.
NumCSE, Lecture 3, Sept 25, 2014
34/34
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
I
Newtons algorithm and variants
The secant method and inverse interpolation
Zeroin (fzero)
Multiple zeros
Scalar optimization
References
I
AscherGreif, Chapter 3, Sections 3.43.5.
C. Moler: Numerical Computing with Matlab. The
MathWorks. 2004. (Chapter 4: Zeros and roots.)
NumCSE, Lecture 4, Sept 29, 2014
2/40
Numerical Methods for Computational Science and Engineering
The problem
The problem revisited
Want to find solutions of the scalar nonlinear equation
f (x) = 0
with continuous f : [a, b] R 7 R
We denote a solution of the equation (called root, or zero) by x .
We looked at bisection and general fixed point iterations to
determine solutions x of f (x) = 0.
NumCSE, Lecture 4, Sept 29, 2014
3/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Newton iteration
I
Let the function f for which we look for a zero x be
differentiable.
In practice, the derivative f 0 should be cheap to compute
(comparable with the computation of f ).
In Newtons method the function f is linearized at some
approximate value xk x .
Define function t(x) that has at x = xk same function value
and derivative as f
(Taylor polynomial of degree one)
t(x) = f (xk ) + f 0 (xk )(x xk ).
(1)
The single zero of t(x) = 0 yields xk+1 .
NumCSE, Lecture 4, Sept 29, 2014
4/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Newton iteration (cont.)
I
The single zero of t(x) = 0 yields xk+1 :
xk+1 = xk f (xk )/f 0 (xk ),
k = 0, 1, . . .
(2)
Newtons method is a fixed point iteration with iteration
function
g (x) = x f (x)/f 0 (x).
Clearly, g (x ) = x .
Since we have neglected second (and higher) terms of the
Taylor expansion of f at x we can expect that t(x) is a very
good approximation of f (x) if xk x .
NumCSE, Lecture 4, Sept 29, 2014
5/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Newtons method: geometric interpretation
NumCSE, Lecture 4, Sept 29, 2014
6/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Algorithm: Newtons iteration
Given a scalar differentiable function f (x).
1. Start from an initial guess x0 .
2. For k = 0, 1, 2, . . . set
xk+1 = xk
f (xk )
f 0 (xk )
until xk+1 satisfies some termination criterion
NumCSE, Lecture 4, Sept 29, 2014
7/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Newtons method: code
function [x,it]=newton(f,df,x)
%NEWTON Newton iteration for scalar eq f(x)=0.
%
x=newton(f,df,x0)
%
x0: initial approx, df = f
dx = f(x)/df(x); it=0;
while abs(dx) > 1e-10,
x = x - dx;
dx = f(x)/df(x);
it = it+1;
end
NumCSE, Lecture 4, Sept 29, 2014
8/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Example (Convergence of Newton iteration)
Iteration for computing
a, a > 0:
f (x) = x 2 a,
f 0 (x) = 2x,
x2 a
x2 + a
1
a
=
= (x + ),
2x
2x
2
x
2+a
x
1
a
g 0 (x) = 1
= 2,
g 0 ( a) = 0,
2
2x
2 2x
a
1
g 00 (x) = 3 ,
g 00 ( a) = .
x
a
g (x) = x
a
1
xk+1 = (xk + )
2
xk
NumCSE, Lecture 4, Sept 29, 2014
|xk+1
a| =
1
|xk a|2 .
2|xk |
9/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Example (Convergence of Newton iteration (cont.))
Numerical experiment: iterates for a = 2:
k
0
1
2
3
4
5
xk
2.00000000000000000
1.50000000000000000
1.41666666666666652
1.41421568627450966
1.41421356237468987
1.41421356237309492
ek := xk 2
0.58578643762690485
0.08578643762690485
0.00245310429357137
0.00000212390141452
0.00000000000159472
0.00000000000000022
log
|ek |
|ek1 |
: log
|ek1 |
|ek2 |
1.850
1.984
2.000
0.630
The number of significant digits in xk essentially doubles at each
iteration.
When roundoff level is reached, no meaningful improvement can be
obtained any further. The improvement from the 4th to the 5th
iteration (in this example) is minimal.
NumCSE, Lecture 4, Sept 29, 2014
10/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Example (Choice of initial guess)
Consider f (x) = 2 cosh(x/4) x;
g (x) = 2 cosh(x/4).
f has 2 roots: x1 2.35755106, x2 8.50719958.
NumCSE, Lecture 4, Sept 29, 2014
11/40
Numerical Methods for Computational Science and Engineering
Newton iteration
The Newton iteration here is
xk+1 = xk
2 cosh(xk /4) xk
.
0.5 sinh(xk /4) 1
Iterating until |f (xk+1 )| < 108 we observe:
I
Starting from x0 = 2 requires 4 iterations to reach x1 to
within the given tolerance.
Starting from x0 = 4 requires 5 iterations to reach x1 .
Starting from x0 = 8 requires 5 iterations to reach x2 .
Starting from x0 = 10 requires 6 iterations to reach x2 .
NumCSE, Lecture 4, Sept 29, 2014
12/40
Numerical Methods for Computational Science and Engineering
Newton iteration
The values of f (xk ), starting from x0 = 8, are:
k
f (xk )
0
-4.76e-1
1
8.43e-2
2
1.56e-3
3
5.65e-7
4
7.28e-14
5
1.78e-15
Number of significant digits essentially doubles at each iteration.
When roundoff level is reached, no meaningful improvement can be
obtained upon heaping more floating point operations, and the
improvement from the 4th to the 5th iteration in this example is
marginal.
NumCSE, Lecture 4, Sept 29, 2014
13/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Order of convergence
The method is said to be
I
linearly convergent if there is a constant < 1 such that
|xk+1 x | |xk x |,
quadratically convergent if there is a constant M such that
|xk+1 x | M|xk x |2 ,
for k sufficiently large;
for k sufficiently large;
superlinearly convergent if there is a sequence of constants
k 0 such that
|xk+1 x | k |xk x |
for k sufficiently large;
The quadratic case is superlinear with k = M|xk x | 0
NumCSE, Lecture 4, Sept 29, 2014
14/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Convergence of Newtons method
Theorem (Convergence of Newton iteration)
If f C 2 [a, b] and there is a root x in [a, b] such that f (x ) = 0,
f 0 (x ) 6= 0, then there is a number > 0 such that, starting with
x0 from anywhere in the neighborhood [x , x + ], Newtons
method converges quadratically.
Note that f 0 (x ) 6= 0 means that x is a simple root of f .
Sketch of a proof.
(i) It is easy to verify that
g 0 (x) =
f (x)f 00 (x)
.
(f 0 (x))2
So, g 0 (x ) = 0.
NumCSE, Lecture 4, Sept 29, 2014
15/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Convergence of Newtons method (cont.)
(ii) Since g is continuously differentiable there is some
neighborhood [x , x + ] of x in which |g 0 | < 1.
(iii) For convergence order use the Taylor expansion of g at x :
x xk+1 = g (x ) g (xk )
= g (x ) g (x ) + g 0 (x )(xk x )
+ g 00 ()(xk x )2
= g 00 ()(xk x )2
with between x and xk .
NumCSE, Lecture 4, Sept 29, 2014
16/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Another example
The equation f (x) = x 3 3x + 2 = (x + 2)(x 1)2 has two zeros:
2 and 1. The Newton iteration is
2xk
2
+
3
3(xk + 1)
2x(x + 2)
2
2
g 0 (x) =
=
3(x + 1)2
3 3(x + 1)2
xk+1 = g (xk ) =
g 0 (2) = 0
1
g 0 (1) =
2
The double zero makes troubles.
NumCSE, Lecture 4, Sept 29, 2014
17/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Newton iteration for multiple roots
Let x be a multiple root of f : f (x) = (x x )m h(x) with h
differentiable and h(x ) 6= 0. Then,
g (x) = x
f (x)
(x x )m h(x)
=
x
f 0 (x)
m(x x )m1 h(x) + (x x )m h0 (x)
(x x )h(x)
=x
m h(x) + (x x )h0 (x)
h(x) + (x x )h0 (x)
(m h(x) + (x x )h0 (x))
(x x )h(x)(m h0 (x) + h0 (x) + (x x )h00 (x))
,
(m h(x) + (x x )h0 (x))2
1
g 0 (x ) = 1 .
m
g 0 (x) = 1
NumCSE, Lecture 4, Sept 29, 2014
18/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Newton iteration for multiple roots (cont.)
Therefore, Newton iteration converges only linearly to multiple
roots.
For large m the convergence is very slow.
For a double root (m = 2) the contraction factor is = 21 .
Remedy: We have to extend the step length in accordance with
the multiplicity of the zero of f (x),
xk+1 = xk m f (xk )/f 0 (xk )
Note: Often we do not know the multiplicity of a root.
Remark: One may try to apply Newton to the function f (x)/f 0 (x)
that has only simple roots.
NumCSE, Lecture 4, Sept 29, 2014
19/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Simplified Newton iteration
xk+1 = xk f (xk )/f 0 (x0 ),
k = 0, 1, . . .
Linear convergence, convergence factor
:= |g 0 (x )| = |1 f 0 (x )/f 0 (x0 )|.
Simplified Newton iteration can be very effective if, x0 is a good
approximation of x . Then,
1 f (x )/f 0 (x0 ) 1 f (x )/f 0 (x ) = 0
such that the convergence factor is small.
We may update x0 to improve speed of convergence.
NumCSE, Lecture 4, Sept 29, 2014
20/40
Numerical Methods for Computational Science and Engineering
Newton iteration
Damped Newton
To avoid overshooting one can damp (shorten) the Newton step
xk+1 = xk k f (xk )/f 0 (xk ),
k = 0, 1, . . .
k is chosen such that |f (xk+1 )| < |f (xk )|
x = f (x)/f 0 (x);
while (|f (x x)| > |f (x)|)
= /2;
end
Close to convergence we should let k 1 to have the full step
length and quadratic convergence.
Before each iteration step: = min(1, 2);.
NumCSE, Lecture 4, Sept 29, 2014
21/40
Numerical Methods for Computational Science and Engineering
Secant method
Secant method
xk+1 = xk f (xk )
xk xk1
f (xk ) f (xk1 )
secant method.
Notice that the secant method is obtained by approximating the
derivative in Newtons method by a finite difference,
f 0 (xk )
f (xk ) f (xk1 )
.
xk xk1
The secant method is not a fixed point method but a multi-point
method. It can interpreted as follows. The next iterate xk+1 is the
zero of the degree-1 polynomial that interpolates f at xk and xk1 .
Convergence order: 1.618, i.e., superlinear! No derivatives needed!!
NumCSE, Lecture 4, Sept 29, 2014
22/40
Numerical Methods for Computational Science and Engineering
Secant method
Secant method (cont.)
function [x,i] = secant(x0,x1,f,tol,maxit)
% secant method
f0 = f(x0);
for i=1:maxit
f1 = f(x1);
s = f1*(x1-x0)/(f1-f0);
x0=x1;x1=x1-s;
if(abs(s)<tol), x=x1; return; end
f0=f1;
end
x = NaN;
NumCSE, Lecture 4, Sept 29, 2014
23/40
Numerical Methods for Computational Science and Engineering
Secant method
Convergence of secant method
Theorem (Convergence of secant method)
If f C 2 [a, b] and there is a root x in [a, b] such that f (x ) = 0,
f 0 (x ) 6= 0, then there is a number > 0 such that, starting with
x0 , x1 from anywhere in the neighborhood [x , x + ], the
secant method converges superlinearly.
Without proof.
The convergence order is in fact 1.618 . . ., i.e., assymptotically,
|x xk+1 | = c |x xk |1.618...
for some constant c.
NumCSE, Lecture 4, Sept 29, 2014
24/40
Numerical Methods for Computational Science and Engineering
Secant method
Example: Choice of initial guess
Consider again f (x) = 2 cosh(x/4) x = 0.
The secant method requires two initial iterates, x0 and x1 .
I
Starting from x0 = 2, x1 = 4 requires 7 iterations to reach x1 .
Starting from x0 = 10, x1 = 8 requires 7 it.s to reach x2 .
The values of f (xk ), starting from x0 = 10, x1 = 8, are:
k
f (xk )
0
2.26
1
-4.76e-1
2
-1.64e-1
3
2.45e-2
4
-9.93e-4
5
-5.62e-6
6
1.30e-9
Number of significant digits increases more rapidly as we get
nearer to the root (i.e., better than linear order), but not as fast as
with Newtons method. Thus we observe indeed a demonstration
of superlinear convergence.
NumCSE, Lecture 4, Sept 29, 2014
25/40
Numerical Methods for Computational Science and Engineering
Secant method
Higher order methods
Higher order methods could be obtained by (e.g.) determining a
quadratic polynomial q(t) such that
f (t) = q(t),
f 0 (t) = q 0 (t),
f 00 (t) = q 00 (t),
or (for three pairwise different ti )
f (t1 ) = q(t1 ),
f (t2 ) = q(t2 ),
f (t3 ) = q(t3 ),
and then compute its roots.
Questions arise
I
Which of the two roots?
What if the roots are complex?
NumCSE, Lecture 4, Sept 29, 2014
26/40
Numerical Methods for Computational Science and Engineering
Inverse interpolation
Inverse interpolation
Given a data set
(xi , yi = f (xi )),
i = 0, 1, . . . , n.
In inverse interpolation we want to find a position x such that, for
a given y , f (
x ) = y .
If the given function f is monotone in the interval, then for each y
there is only one x for which f (x) = y . In this situation it makes
sense to interpolate the points (yi , xi = f 1 (yi )).
Here: we are looking for x such that x = f 1 (0) f (x ) = 0.
NumCSE, Lecture 4, Sept 29, 2014
27/40
Numerical Methods for Computational Science and Engineering
Inverse interpolation
Inverse linear interpolation
The secant method can be derived as inverse linear interpolation:
The function that linearly interpolates (yk , f 1 (yk )) = (yk , xk ) and
(yk1 , f 1 (yk1 )) = (yk1 , xk1 ) is
f 1 (yk )
y yk1
y yk
f 1 (yk1 )
yk yk1
yk yk1
The value of this function at y = 0 gives the new approximate
xk+1 :
xk yk1 + xk1 yk
xk+1 =
yk yk1
xk xk1
= xk yk
,
yk = f (xk ).
yk yk1
NumCSE, Lecture 4, Sept 29, 2014
28/40
Numerical Methods for Computational Science and Engineering
Inverse interpolation
Inverse quadratic interpolation
fk2 fk1
xk
(fk fk2 )(fk fk1 )
fk2 fk
+
xk1
(fk1 fk2 )(fk1 fk )
fk1 fk
+
xk2
(fk2 fk1 )(fk2 fk )
xk+1 =
Code on next page from Moler, Numerical Computing with
Matlab, SIAM, 2004.
Convergence order: 1.839. No derivatives needed!!
But: IQI needs good starting value!
NumCSE, Lecture 4, Sept 29, 2014
29/40
Numerical Methods for Computational Science and Engineering
Inverse interpolation
Inverse quadratic interpolation (cont.)
function [b,k] = iqi(f,x)
%
x = iqi(f,x) tries to find a zero of f(x) near the initial
%
guesses x=[x0,x1,x2]. If successful, iqi returns the
%
inverse quadratic interpolation approximation to a root of f
%
[x,iter] = iqi(f,x) returns also the number of iterations
%
required to find the root.
a = x(1); b = x(2); c = x(3);
k = 0;
while abs(c-b) > eps*abs(c)
x = polyinterp([f(a),f(b),f(c)],[a,b,c],0);
a = b; b = c; c = x;
k = k + 1;
disp(sprintf(x%d = %16.14f,k,c))
end
NumCSE, Lecture 4, Sept 29, 2014
30/40
Numerical Methods for Computational Science and Engineering
Inverse interpolation
Example for IQI
Find zero of f (x) = xe x 1, x0 = 0, x (1) = 2.5, x (2) = 5.
k
1
2
3
4
5
6
7
8
xk
0.08520390058175
0.16009252622586
0.79879381816390
0.63094636752843
0.56107750991028
0.56706941033107
0.56714331707092
0.56714329040980
NumCSE, Lecture 4, Sept 29, 2014
f (xk )
-0.90721814294134
-0.81211229637354
0.77560534067946
0.18579323999999
-0.01667806436181
-0.00020413476766
0.00000007367067
0.00000000000003
ek := xk x
-0.48193938982803
-0.40705076418392
0.23165052775411
0.06380307711864
-0.00606578049951
-0.00007388007872
0.00000002666114
0.00000000000001
log |ek+1 |log |ek |
log |ek |log |ek1 |
3.33791154378839
2.28740488912208
1.82494667289715
1.87323264214217
1.79832936980454
1.84841261527097
31/40
Numerical Methods for Computational Science and Engineering
Matlabs fzero
Matlabs fzero
Combine the reliability of bisection with the convergence speed of
secant and inverse quadratic interpolation (IQI). Requires only
function evaluation.
Outline:
I
Start with a and b s.t. f (a)f (b) < 0.
Use a secant step to get c between a and b.
Repeat the following steps until |b a| < |b| or f (b) = 0.
Arrange a, b, and c so that
I
I
I
f (a)f (b) < 0
|f (b)| |f (a)|
c is the previous value of b.
If c 6= a, consider an IQI step.
If c = a, consider a secant step.
NumCSE, Lecture 4, Sept 29, 2014
32/40
Numerical Methods for Computational Science and Engineering
Matlabs fzero
Matlabs fzero (cont.)
I
If the IQI or secant step is in the interval [a, b], take it.
If the step is not in the interval, bisect.
This algorithm is foolproof: It never loses track of the zero trapped
in a shrinking interval.
It uses rapidly convergent methods when they are reliable.
It uses a slow, but sure, method when it is necessary.
It only uses function values, no derivatives.
NumCSE, Lecture 4, Sept 29, 2014
33/40
Numerical Methods for Computational Science and Engineering
Comparision of methods
Comparision of methods
Comparison of some methods for computing the smallest zero of
f (x) = cos(x) cosh(x) + 1 = 0. (x 1.8751, tol = 109 )
method
bisection
secant method
secant method
Newton
fzero
fzero
start
[0, 3]
[1.5, 3]
[0, 3]
x0 = 1.5
[0, 3]
[1.5, 3]
# steps
32
8
15
5
8
7
# function evals
34
9
16
11
10
9
Note: (1) These numbers depend on the function f !
Numbers from SchwarzK
ockler, except for fzero.
NumCSE, Lecture 4, Sept 29, 2014
34/40
Numerical Methods for Computational Science and Engineering
Computing multiple zeros
Computing multiple zeros
If we have found a zero z of f (x) = 0 and want to compute
another one, we want to avoid recomputing the already found z.
We can explicitely deflate the zero by defining a new function
f1 (x) :=
f (x)
,
x z
and apply our method of choice to f1 . This procedure can in
particular be done with polynomials which can be error prone if z
is not accurate.
We can proceed similarly for multiple zeros z1 , . . . , zm .
NumCSE, Lecture 4, Sept 29, 2014
35/40
Numerical Methods for Computational Science and Engineering
Computing multiple zeros
Computing multiple zeros (cont.)
For the reciprocal Newton correction for f1 we get
f10 (x)
=
f1 (x)
f 0 (x)
xz
f (x)
(xz)2
f (x)
xz
1
f 0 (x)
.
f (x)
x z
Then a Newton correction becomes
x (k+1) = xk
1
f 0 (xk ) 1
f (xk ) xk z
and similarly for multiple zeros z1 , . . . , zm .
The above procedure is called implicit deflation. f is not modified.
In this way errors in z are not propagated to f1
NumCSE, Lecture 4, Sept 29, 2014
36/40
Numerical Methods for Computational Science and Engineering
Minimizing a function of one variable
Minimizing a function of one variable
I
A major source of applications giving rise to root finding is
optimization.
One-variable version: find an argument x = x that minimizes
a given objective function (x).
Example from earlier: Find the minimum of the function
x
(x) = 10 cosh( ) x
4
over the real line.
Note: maximize function (x) minimize (x) = (x).
NumCSE, Lecture 4, Sept 29, 2014
37/40
Numerical Methods for Computational Science and Engineering
Minimizing a function of one variable
Conditions for a minimum point
Assume that C 2 [a, b]. Denote
f (x) = 0 (x).
An argument x satisfying a < x < b is called a critical point if
f (x ) = 0.
For parameter h small enough so that x + h [a, b] we can
expand in a Taylors series
(x + h) = (x ) + h0 (x ) +
= (x ) +
NumCSE, Lecture 4, Sept 29, 2014
h2 00
(x ) +
2
h2 00
[ (x ) + O(h)].
2
38/40
Numerical Methods for Computational Science and Engineering
Minimizing a function of one variable
Conditions for a minimum point (cont.)
Since |h| can be taken arbitrarily small, it is now clear that at a
critical point:
I
If 00 (x ) > 0, then x = x is a local minimizer of (x). This
means that attains a minimum at x = x in some
neighborhood which includes x .
If 00 (x ) < 0, then x = x is a local maximizer of (x). This
means that attains a minimum at x = x in some
neighborhood which includes x .
If 00 (x ) = 0, then a further investigation at x is required.
NumCSE, Lecture 4, Sept 29, 2014
39/40
Numerical Methods for Computational Science and Engineering
Minimizing a function of one variable
Computation of minimia of functions
If (x) attains a minimum (or maximum) at a point x, then this
point must be critical, i.e., f (
x ) = 0.
We can apply any of the zero finders to find x with f (
x ) = 0.
Example. For the function (x) = 10 cosh(x/4) x we have
0 (x) = f (x) =
x
10
sinh( ) 1,
4
4
00 (x) = f 0 (x) =
10
x
cosh( ).
16
4
Note that for quadratic convergence of Newtons method, (x)
must have three continuous derivatives.
Note: the problem of finding all minima of a given function (x) can be
solved by finding all the critical roots and then checking for each if it is a
minimum by examining the sign of the second derivative of .
NumCSE, Lecture 4, Sept 29, 2014
40/40
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
I
Some examples to review some basic concepts of linear
algebra.
Reference
I
AscherGreif, Section 4.
The basic concepts are found in any book on numerical linear
algebra.
NumCSE, Lecture 5, Oct 2, 2014
2/35
Numerical Methods for Computational Science and Engineering
Data fitting
Example: Data fitting
I
Given measurements, or observations, {(ti , bi )}m
i=1 .
Want to fit to this data a function v (t) that is a linear
combination of n known, linearly independent basis functions
in t, 1 (t), 2 (t), . . . , n (t),
v (t) =
n
X
xi i (t).
i=1
by interpolation at the distinct points t1 < t2 < . . . < tm .
I
So, we seek coefficients x1 , . . . , xn such that
v (ti ) = bi ,
NumCSE, Lecture 5, Oct 2, 2014
i = 1, . . . , m.
(1)
3/35
Numerical Methods for Computational Science and Engineering
Data fitting
Example: Data fitting (cont.)
Setting aij := j (ti ) the interpolation requirements become
a11 x1 + a12 x2 + + a1n xn = b1
a21 x1 + a22 x2 + + a2n xn = b2
..
.
am1 x1 + am2 x2 + + amn xn = bm
Let us assume that m = n (i.e., the same number of data and
unknown coefficients). Then we can write the above equations as
A x = b,
A Rnn
If the j are linearly independent then ... (be careful!) A is
nonsingular.
NumCSE, Lecture 5, Oct 2, 2014
4/35
Numerical Methods for Computational Science and Engineering
Data fitting
Example: Data fitting (cont.)
Special choice of basis functions: j (t) = t j1 (monomials).
Then v (t) is in the set Pn of polynomials of degree < n and
1 t1 t12 t1n1
1 t2 t 2 t n1
2
2
A = . . . .
.. .
..
.. .. ..
.
1 tn tn2
tnn1
This form of a matrix is called Vandermonde matrix. If the ti are
distinct then A is nonsingular, i.e., the polynomial interpolation
problem always has a unique solution.
In fact, det(A) =
1i<jn (tj
ti ).
But under certain conditions there are numerical difficulties in
solving Vandermonde systems!
NumCSE, Lecture 5, Oct 2, 2014
5/35
Numerical Methods for Computational Science and Engineering
Data fitting
Numerical example
t = [0,0.1,0.8,1]; b = [1,-0.9,10,9];
A = fliplr(vander(t)); % notice the definition of
% MATLABs vander function
x = A \ b;
% This solves the system Ax = b
NumCSE, Lecture 5, Oct 2, 2014
6/35
Numerical Methods for Computational Science and Engineering
Data fitting
Overdetermined data fitting problem
I
In real-world applications often not advisable or even feasible
to have as many basis functions as data points.
If noisy laboratory measurements are source for data points,
then interpolation makes little sense.
We may be interested in a trend only.
Thus, want to find x Rn such that
v (ti ) bi ,
i = 1, . . . , m,
where usually m > n, i.e., the system is overdetermined.
Imagine values such as n = 1, 2, or 3 and m = 100.
NumCSE, Lecture 5, Oct 2, 2014
7/35
Numerical Methods for Computational Science and Engineering
Data fitting
Overdetermined data fitting problem (cont.)
Obvious question: in what sense do we want to approximate.
Often natural (or at least convenient) to use the 2-norm:
min
x
m
X
[bi v (ti ; x)]
i=1
= min
x
m
X
i=1
"
bi
n
X
#2
xj j (ti )
i=1
Reusing rectangular matrix A with elements aij = j (ti ) the linear
least squares problem can be written as
minkb Axk2 ,
x
NumCSE, Lecture 5, Oct 2, 2014
A Rmn , b Rm , x Rn , m n.
8/35
Numerical Methods for Computational Science and Engineering
Data fitting
Numerical example continued
t
A
A
x
=
=
=
=
[0,0.1,0.8,1]; b = [1,-0.9,10,9];
fliplr(vander(t));
A(:,1:3); % only quadratic polynomial
A \ b;
% This solves the system Ax = b
NumCSE, Lecture 5, Oct 2, 2014
9/35
Numerical Methods for Computational Science and Engineering
Ordinary differential equation
Example: Ordinary differential equation
I
Consider the recovery of a function v (t) from its given second
derivative g (t) on the interval [0, 1].
Origin: heat equation. Given heat sources/sinks. Recover
steady-state temperature distribution. Here, homogeneous
materials.
Process is not unique: If v1 solves the problem,
v 00 (t) = g (t), then v2 = v1 (t) + + t solves it as well.
Boundary conditions are required to fix , .
Consider two cases:
(1) v (0) = v (1) = 0,
(2) v (0) = v 0 (1) = 0.
NumCSE, Lecture 5, Oct 2, 2014
10/35
Numerical Methods for Computational Science and Engineering
Ordinary differential equation
Example: Ordinary differential equation (cont.)
I
We subdivide interval [0, 1] into N subintervals of size
h = 1/N. Let ti = ih, i = 0, . . . , N and approximate v 00 = g
by (vj v (tj ))
vi1 2vi + vi+1
= g (ti ),
h2
i = 1, . . . , N 1
or
1 2 1
v0
g (t1 )
v1 g (t2 )
1 2 1
..
..
1 2 1
=
.
.
h2
.
.
.
.
.
.
vN1 g (tN2 )
.
.
.
1 2 1
vN
g (tN1 )
NumCSE, Lecture 5, Oct 2, 2014
11/35
Numerical Methods for Computational Science and Engineering
Ordinary differential equation
Example: Ordinary differential equation (cont.)
I
Insert boundary conditions:
(1) v (0) = v (1) = 0,
Set v0 = vN = 0. Then the above system becomes
N 1 N 1 and is spd.
(2) v (0) = v 0 (1) = 0. We set v0 = 0.
At the right end of the interval we add two equations:
vN1 2vN + vN+1
= g (tN )
h2
is the differential equation at t = 1.
vN+1 vN1
=0
2h
is a second-order approximation of the bc v 0 (1) = 0.
NumCSE, Lecture 5, Oct 2, 2014
12/35
Numerical Methods for Computational Science and Engineering
Ordinary differential equation
Example: Ordinary differential equation (cont.)
The system Av = g becomes square:
2
1
h2
1
2 1
1
2 1
..
..
.
.
1
v1
v2
v3
..
.
g (t1 )
g (t2 )
g (t3 )
..
.
.
..
2 1 vN1 g (tN1 )
2
2
vN
g (tN )
The matrix A is not symmetric. It could be symmetrized by
multiplying the last row by 12 .
The last row of the system would then read
1 vN1 + 1 vN =
NumCSE, Lecture 5, Oct 2, 2014
1
g (tN ).
2
13/35
Numerical Methods for Computational Science and Engineering
Global Positioning System
Example: Global Positioning System
A Global Positioning System (GPS)
is a device that uses satellite
readings to determine a receiverss
geocentric coordinates (x, y , z), i.e.,
coordinates relative to center of
earth (0, 0, 0).
These measurements are accurate to
at least 80 meters, usually better.
Picture at
http://www.geologie.ens.fr/~vigny/images/gps-cours1.jpg
NumCSE, Lecture 5, Oct 2, 2014
14/35
Numerical Methods for Computational Science and Engineering
Global Positioning System
Example: Global Positioning System (cont.)
GPS device reads information from a
number of satellites (at least four
satellites are necessary).
The ith satellite will send its
geocentric coordinates (xi , yi , zi ) and
the time of transmission ti .
These values are highly accurate.
Picture from
http://en.wikipedia.org/wiki/
Global_Positioning_System
NumCSE, Lecture 5, Oct 2, 2014
Distances xi , yi , zi are measured in
meters [m] from the center of the
earth, and the time ti is measured in
nanoseconds [nsec]. In GPS, time ti
is the fourth dimension.
15/35
Numerical Methods for Computational Science and Engineering
Global Positioning System
Example: Global Positioning System (cont.)
The receiver gets information of satelite i at time Ti , measured on
the receivers clock. Distance between receiver and satellite can be
computed by i = c(Ti ti ) meters, where c is speed of light:
c = 299792458 m/s = 0.299792458 m/nsec.
An error of one nanosecond yields a 30 cm error in distance. The
i are called pseudorange.
As receiver is a cheap device its clock is out of sync. Since the
satellites clocks are in sync the receiver clock will be out of sync
with each satelite by the same amount.
Let t be the error in the receivers clock. Then the error in the
pseudorange is b := ct.
NumCSE, Lecture 5, Oct 2, 2014
16/35
Numerical Methods for Computational Science and Engineering
Global Positioning System
Example: Global Positioning System (cont.)
x
xi
y
yi
u=
si =
z
zi ,
b
i
| {z }
{z
}
|
data from satellite i
unknown data about receiver
Actual distance of satelite i and receiver:
q
(xi x)2 + (yi y )2 + (zi z)2 + b = i .
Thus,
(xi x)2 + (yi y )2 + (zi z)2 = (i b)2 ,
This is a nonlinear least squares problem.
NumCSE, Lecture 5, Oct 2, 2014
17/35
Numerical Methods for Computational Science and Engineering
Global Positioning System
Example: Global Positioning System (cont.)
Nonlinear least squares problem could be solved (and actually is).
Now we derive an algorithm that is usually much faster since it just
requires a linear least squares solution.
We rearrange terms to get
xi2 + yi2 + zi2 2i 2(xi x + yi y + zi z i b) + (x 2 + y 2 + z 2 b 2 ) = 0.
We define a modified dot product, called Minkovski metric,
hu, vi = u1 v1 + u2 v2 + u3 v3 u4 v4 .
Then the above equation becomes (after division by 2)
NumCSE, Lecture 5, Oct 2, 2014
1
1
hsi , si i hsi , ui + hu, ui = 0,
2
2
i.
(2)
18/35
Numerical Methods for Computational Science and Engineering
Global Positioning System
Example: Global Positioning System (cont.)
If we have n satellites, then we can write the collection of these
equations as
1
x1 y1 z1 1
hs1 , s1 i
x2 y2 z2 2
1
1 hs2 , s2 i
1
B=.
, a = . , e = . , = hu, ui.
.
.
.
.
.
.
.
.
.
2 .
2
.
.
.
.
.
1
xn yn zn n
hsn , sn i
B is an n 4 matrix, a and e are n-vectors, and is a scalar. We
now can simultaneously write the n equations from (2) as
a Bu + e = 0
NumCSE, Lecture 5, Oct 2, 2014
Bu = (a + e)
19/35
Numerical Methods for Computational Science and Engineering
Global Positioning System
Example: Global Positioning System (cont.)
The solution can be written as
u = B + (a + e)
(3)
where B + = (B T B)1 B T is the pseudo-inverse of B.
However, solution u involves which involves unknown u .
We work around this problem with some clever algebraic
manipulations. Substituting u into the definition of gives
1
1
= hu , u i = hB + (a + e), B + (a + e)i
2
2
or
2 hB + e, B + ei + 2(hB + a, B + ei 1) + hB + a, B + ai = 0.
NumCSE, Lecture 5, Oct 2, 2014
(Q)
20/35
Numerical Methods for Computational Science and Engineering
Global Positioning System
Example: Global Positioning System (cont.)
This is a quadratic equation in that has two solutions 1 and 2
that in turn yield
u1 = B + (a + 1 e).
u2 = B + (a + 2 e).
(4)
It turns out that only one of these two solutions is useful, namely
the one that leads to receiver coordinates that are on the surface
of the earth.
To check this, the first three components of u are used.
NumCSE, Lecture 5, Oct 2, 2014
21/35
Numerical Methods for Computational Science and Engineering
Global Positioning System
Example: Global Positioning System (cont.)
To summarize, here is the Bancroft algorithm for finding u :
1. Organize your data into B, a, and e.
2. Solve the quadratic equation (Q) for 1 and 2 .
3. Solve for the least squares solutions u1 and u2 and pick the
one that makes sense.
In point 3, we look for that solution that is on the earths surface.
NumCSE, Lecture 5, Oct 2, 2014
22/35
Numerical Methods for Computational Science and Engineering
Principal Component Analysis
Example: Principal Component Analysis
Let x1 , . . . , xN Rp represent N observations, measurements, or
other data. The sample mean of these observation vectors is
m=
1
(x1 + + xN )
N
Let
xk = xk m,
k = 1, . . . , N.
The columns of the p N matrix
B = [
x1 ,
x2 , . . . ,
xN ]
have a zero sample mean. B is said to be in mean-deviation form.
NumCSE, Lecture 5, Oct 2, 2014
23/35
Numerical Methods for Computational Science and Engineering
Principal Component Analysis
Example: Principal Component Analysis (cont.)
The (sample) covariance matrix is
S=
1
BB T Rpp .
N 1
Clearly, S is symmetric positive semidefinite.
NumCSE, Lecture 5, Oct 2, 2014
24/35
Numerical Methods for Computational Science and Engineering
Principal Component Analysis
Example: Principal Component Analysis (cont.)
Numerical example:
1
4
7
8
5
x1 = 2 , x2 = 2 , x3 = 8 , x4 = 4 , m = 4 .
1
13
1
5
5
4
B = [
x1 ,
x2 ,
x3 ,
x4 ] = 2
4
10
6
8
S = 6
0 8
NumCSE, Lecture 5, Oct 2, 2014
1
2 3
2
4 0 .
8 4 0
0
8
32
25/35
Numerical Methods for Computational Science and Engineering
Principal Component Analysis
Example: Principal Component Analysis (cont.)
Let now x = [x1 , x2 , . . . , xp ]T be a vector that varies over the set of
observation vectors. The first component x1 , e.g., varies over the
first components of the observations xi , 1 i N.
The diagonal entry sjj is called the variance of xj , j = 1, . . . , p.
The variance of xj measures the spread of the values of xj .
In the above example, the variance of x1 is 10, while the variance
of x3 is 32. So, the values in x3 are spread wider than those in x1 .
The total variance of the data is the sum of all variances, i.e. the
trace tr(S) of S.
The entry sij of S for i 6= j is calles the covariance of xi and xj . If
sij = 0 then xi and xj are uncorrelated.
NumCSE, Lecture 5, Oct 2, 2014
26/35
Numerical Methods for Computational Science and Engineering
Principal Component Analysis
Example: Principal Component Analysis (cont.)
Analysis of the multivariate data in x1 , . . . , xN is greatly simplified
if the covariance matrix is diagonal or nearly so.
The goal of the principal component analysis is to find an
orthogonal p p matrix P that determines a change of variables
C = PB
with the property that the new variables y1 , . . . , yp are
uncorrelated and arranged in order of decreasing variance.
NumCSE, Lecture 5, Oct 2, 2014
27/35
Numerical Methods for Computational Science and Engineering
Principal Component Analysis
Example: Principal Component Analysis (cont.)
A=
1
BT
N 1
AT A = S.
Let
UV T = A
be the economic singular value decomposition of A. Then
AT A = V U T UV T = S
V T SV = 2 = diagonal.
The eigenvalues of S are the squares of the singular values
1 2 p 0 of A.
NumCSE, Lecture 5, Oct 2, 2014
total variance = tr(S) =
p
X
j2 .
j=1
28/35
Numerical Methods for Computational Science and Engineering
Principal Component Analysis
Example: Principal Component Analysis (cont.)
Let V = [v1 , v2 , . . . , vp ]. The vectors
Av1 , Av2 , . . . , Avp
are the principal components of B arranged according to
descending variance. Note that Avi = ui i .
Principal component analysis is valuable for applications in which
most of the variation, or dynamic range, in the data is due to
variations in only a few of the transformed variables.
Numerical example above:
12 = 34.5513,
22 = 13.8430,
32 = 1.6057,
tr(S) = 50.
Since 34.5513
+ 13.8430
= 48.3943
= 0.9679 the first two component
50
50
50
contain 97% of the total variation (information).
NumCSE, Lecture 5, Oct 2, 2014
29/35
Numerical Methods for Computational Science and Engineering
Principal Component Analysis
Application: Image processing
From David Lay: Linear Algebra and Its Applications, p. 394.
NumCSE, Lecture 5, Oct 2, 2014
30/35
Numerical Methods for Computational Science and Engineering
Principal Component Analysis
Application: point cloud
NumCSE, Lecture 5, Oct 2, 2014
Graphics objects in R3 are
often represented by point
clouds (scattered data set of
points that together form a
coherent surface).
Laser scanning, medical
imaging, visualization,...
Relevant tasks: clean data
set from noise and outliers,
and display the surface that
gave rise to this cloud in a
pleasing way.
31/35
Numerical Methods for Computational Science and Engineering
Principal Component Analysis
Application: point cloud (cont.)
I
For such applications we need the normal directions at the
points with respect to the implied surface.
Finding normals (without forming the surface first!) can be
done using PCA.
For fixed point p in cloud, we define neighborhood Np of
nearby points, e.g., points whose distance in 2-norm from p is
below some value.
Let us assume there are n = np such neighbors.
We calculate the mean of these neighbors to find the centroid
of Np , call it p
. Our 3 np data matrix A therefore has pip p
for its ith column, where pip is the ith neighbor of p.
NumCSE, Lecture 5, Oct 2, 2014
32/35
Numerical Methods for Computational Science and Engineering
Principal Component Analysis
Application: point cloud (cont.)
I
We find the three singular vectors of A (eigenvectors of the
covariance matrix). The first two principal vectors span the
tangent plane at p. The third one, orthogonal to the first two
and hence to the tangent plane, is our desired unsigned
normal direction.
An additional, global, pass over the points is required to
determine what sign our normal vectors should have.
NumCSE, Lecture 5, Oct 2, 2014
33/35
Numerical Methods for Computational Science and Engineering
Principal Component Analysis
Application: point cloud (cont.)
I
This procedure to consolidate the point cloud can be
complemented by a procedure to represent the surface by
another set of point, called particles. This is a
complementary set of points that enables the more economic
representation of the point cloud.
Its determination (location, distribution) may be a nonlinear
process.
NumCSE, Lecture 5, Oct 2, 2014
34/35
Numerical Methods for Computational Science and Engineering
Further reading
Further reading
I
Uri Ascher, Chen Greif: A First Course in Numerical Methods.
SIAM 2011. Chapter 4.
David Lay: Linear Algebra and Its Applications.
Addison-Wesley, New York. 2000.
Jonathon Shlens: A Tutorial on Principal Component
Analysis. http://arxiv.org/abs/1404.1100.
H. Huang, D. Li, H. Zhang, U. Ascher, D. Cohen-Or:
Consolidation of Unorganized Point Clouds for Surface
Reconstruction. ACM Trans. Graphics (SIGGRAPH Asia),
28(5) (2009).
NumCSE, Lecture 5, Oct 2, 2014
35/35
Numerical Methods for Computational Science and Engineering
Introduction
Goals of the chapter on direct methods for linear systems
I
Learn practical methods to handle the most common problem
in numerical computation, the solution of linear systems of
equations.
Get familiar (again) with the ancient method of Gaussian
elimination in its modern form of LU decomposition.
I
Develop pivoting strategies for its stable computation.
Consider LU decomposition in the very important special
cases of symmetric positive definite and sparse matrices.
Study the expected quality of the computed solution
I
Introduction of the fundamental concept of a condition
number.
NumCSE, Lecture 6, Oct 6, 2014
2/40
Numerical Methods for Computational Science and Engineering
Introduction
General remarks
I
We consider the problem of finding x which solves
Ax = b,
I
I
where A is a given, real, nonsingular, n n matrix, and b is a
given, real vector.
Such problems are ubiquitous in applications! Often most
time critical.
Two solution approaches:
1. Direct methods: yield exact solution in absence of roundoff
errors.
Variations of Gaussian elimination.
Considered today and next time.
2. Iterative methods: iterate in a similar fashion to what we do
for nonlinear problems.
Use only when direct methods are ineffective.
Considered later.
NumCSE, Lecture 6, Oct 6, 2014
3/40
Numerical Methods for Computational Science and Engineering
Introduction
Topics for today
I
Gaussian elimination (GE)
I
I
Pivoting
Diagonally dominant matrices
Cholesky factorization
LAPACK software package and the BLAS
References
I
AscherGreif, Chapter 5, Sections 5.15.5
Golub & van Loan: Matrix Computations, Johns Hopkins
University Press, 3rd ed., 1996.
NumCSE, Lecture 6, Oct 6, 2014
4/40
Numerical Methods for Computational Science and Engineering
Linear systems with matrices of simple structure
Linear systems with matrices of simple structure
I
Special case: A is an upper triangular matrix
a11 a12 a1n
a22 a2n
A=
..
..
.
.
ann
Algorithm: backward substitution
x = zeros(n,1);
for k=n:-1:1
x(k) = (b(k) - a(k,k+1:n)*x(k+1:n))/a(k,k);
end
Complexity: n(n 1)/2 subtractions/multiplications and n
divisions = n(n + 1)/2 = O(n2 ) floating point operations.
NumCSE, Lecture 6, Oct 6, 2014
5/40
Numerical Methods for Computational Science and Engineering
Linear systems with matrices of simple structure
Linear systems with matrices of simple structure (cont.)
I
Example:
x1 4x2 + 3x3 = 2,
5x2 3x3 =
7,
2x3 = 2.
1 4
3
x1
2
0
5 3 x2 = 7
0
0 2
x3
2
Solution: x3 = 1, x2 = 2, x1 = 3.
NumCSE, Lecture 6, Oct 6, 2014
6/40
Numerical Methods for Computational Science and Engineering
Linear systems with matrices of simple structure
Linear systems with matrices of simple structure (cont.)
I
Special case: A is a lower triangular matrix
a11
a12 a22
A= .
.
.
.
.
.
.
.
.
an1 an2
ann
Algorithm: foreward substitution (similar as backward
substitution)
Complexity: O(n2 ) flops.
NumCSE, Lecture 6, Oct 6, 2014
7/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
Gaussian elimination
To solve the linear system of equation A x = b we proceed as
follows:
1. Compute LU factorization
LU = A.
with L a lower unit triangular matrix and U an upper
triangular matrix, respectively.
2. Solve Ly = b by forward substitution. (Can be fused with 1.)
3. Solve Ux = y by backward substitution.
The cost is 32 n3 + O(n2 ) for the LU factorization and 2n2 + O(n)
for each forward and backward substitution, in total
2 3
n + O(n2 ) flops.
3
NumCSE, Lecture 6, Oct 6, 2014
8/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
Achieving the LU factorization
In Gaussian elimination elementary row transformations are used to
reduce the given linear system to an equivalent upper triangular
form.
The resulting system can be solved using backward substitution.
The reason this works is that the solution to our problem Ax = b
is invariant to
I
multiplication of a row by a nonzero constant,
subtraction of a multiple of one row from another, and
row interchanges.
NumCSE, Lecture 6, Oct 6, 2014
9/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
Achieving the LU factorization (cont.)
In the first stage of GE a multiple of the first row is subtracted
from the second, . . ., n-th row in order to introduce zeros in the
first column below the (1, 1) entry. (Note that the (1, 1) entry is
assumed to be nonzero.)
This is achieved by the elementary n n lower triangular matrix
1
`21 1
`31
(1)
1
L =
..
.
.
.
.
`n1
1
After the first step, the modified linear system of equations is
A(1) x = b(1) ,
NumCSE, Lecture 6, Oct 6, 2014
A(1) = L(1) A,
b(1) = L(1) b.
10/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
Achieving the LU factorization (cont.)
In the 2nd stage of GE a multiple of the second row is subtracted
from rows 3, . . ., n to introduce zeros in the second column below
the (2, 2) entry (assumed to be nonzero). This is achieved by
`32 1
(1)
L =
..
.
.
.
.
`n2
1
After the 2nd step, the linear system of equations is
A(2) x = b(2) ,
NumCSE, Lecture 6, Oct 6, 2014
A(2) = L(2) A(1) ,
b(2) = L(2) b(1) .
11/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
Achieving the LU factorization (cont.)
After n 1 stages the original matrix A is tranformed into an
upper triangular matrix U,
U = A(n1) = |L(n1) {z
L(2) L(1)} A.
L1
Likewise, b(n1) = L(n1) L(2) L(1) b.
The matrix L is the product of lower triangular matrices, and thus
itself lower triangular,
1
`21 1
`31 `32 1
(1) 1 (2) 1
(n1) 1
L=L
L
L
=
.
..
..
..
..
.
.
.
.
`n1 `n2 `n3 1
NumCSE, Lecture 6, Oct 6, 2014
12/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
Numerical example
I
Solve Ax = b for
1
A = 1
3
1 3
1 0
2 1
2
b = 5
6
Gaussian elimination stage 1: `21 = 1, `31 = 3, so
1 0 0
1 1 3
2
= 1 1 0 , A(1) = L(1) A = 0 2 3 , b(1) = L(1) b = 7
3 0 1
0 1 8
12
L(1)
Gaussian elimination stage 2: `32 = 1/2, so
1 1
3
2
1 0 0
= 0 1 0 , A(2) = L(2) A(1) = 0 2 3 , b(2) = L(2) b(1) = 7
17
0 0 13
0 12 1
2
2
L(2)
NumCSE, Lecture 6, Oct 6, 2014
13/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
Numerical example (cont.)
I
Note that
[L(1) ]1 [L(2) ]1
1
= 1
3
0
1
0
1
0
0 0
1
0
0
1
1
2
0
1
0 = 1
1
3
0
1
1
2
0
0 = L
1
and
1
LU = 1
3
NumCSE, Lecture 6, Oct 6, 2014
0
1
1
2
0
1
0 0
1
0
1
3
1
2 3 = 1
3
0 13
2
1 3
1 0 = A.
2 1
14/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
Basic Gaussian elimination
for k=1:n-1
for i=k+1:n
l(i,k) = a(i,k)/a(k,k)
for j=k+1:n
a(i,j) = a(i,j) - l(i,k)*a(k,j);
end
b(i) = b(i) - l(i,k)*b(k)
end
end
Then apply backward substitution.
Note: upper part of A is overwritten by U, lower part no longer of
interest. Unit lower triangular L could be stored in lower part of A.
NumCSE, Lecture 6, Oct 6, 2014
15/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
Examples where LU decomposition is useful
I
Multiple right-hand sides: form once LU decomposition
(which costs O(n3 ) flops); then for each right-hand side only
apply forward/backward substitutions (which are
computationally cheap at O(n2 ) flops each).
Can compute A1 by decomposing A = LU once, and then
solving LU x = ek for each column ek of the unit matrix.
n right hand sides, so cost is approximately
8
2 3
n + n 2n2 = n3 flops
3
3
However, typically we try to avoid computing the inverse A1 .
Compute determinant of A by
n
Y
det(A) = det(L) det(U) =
ukk .
NumCSE, Lecture 6, Oct 6, 2014
k=1
16/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
Pivoting
Need for pivoting
Ax = b
0 1
x1
4
=
1 1
x2
7
x1 x2 1
0 1 4
1 1 7
Pivot a11 = 0 we exchange rows 1 and 2.
x1 x2 1
1 1 7
0 1 4
It is rare to hit a precisely zero pivot, but common to hit a very
small one.
NumCSE, Lecture 6, Oct 6, 2014
17/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
Pivoting
Need for pivoting (cont.)
Lets consider the numerical example (5 decimal digits)
x1
x2
1
0.00035 1.2654 3.5267
1.2547 1.3182 6.8541
x1
x2
1
0.00035 1.2654
3.5267
0
4535.0 12636
with
l21 = 1.2547/0.00035 3584.9
u22 = 1.3182 3584.9 1.2654 1.3182 4536.3 4535.0
c2 = 6.8541 3584.9 (3.5267) 6.8541 12643 12636.
NumCSE, Lecture 6, Oct 6, 2014
18/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
Pivoting
Need for pivoting (cont.)
Backsubstitution gives
x2 = 12636/(4535.0) 2.7863
x1 = (3.5267 1.2654 2.7863)/0.00035
(3.5267 3.5258)/0.00035 = 0.0009/0.00035 = 2.5714
The exact solution (up to 5 decimal digits) is
x1 = 2.5354,
x2 = 2.7863
We have cancellation in the computation of x1 !
The very small pivot in the first elimination step induced large
numbers u22 and c2 . Information, e.g. a22 , got lost.
NumCSE, Lecture 6, Oct 6, 2014
19/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
GE with partial pivoting
GE with partial pivoting (GEPP)
The most common strategy for pivoting is to search for the
maximal element in modulus: The index p of the pivot in the k-th
step of GE is determined by
(k1)
|apk
(k1)
| = max |aik
ik
If p > k then rows p and k are exchanged.
This strategy implies that |lik | 1.
NumCSE, Lecture 6, Oct 6, 2014
20/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
GE with partial pivoting
GE with partial pivoting (GEPP) (cont.)
Lets reconsider the 2 2 example with rows 1 and 2 swapped
x1
x2
1
1.2547 1.3182 6.8541
0.00035 1.2654 3.5267
x1
x2
1
1.2547 1.3182 6.8541
0
1.2650 3.5248
Then
l21 = 0.00027895
x2 = 2.7864
x1 = (6.8541 1.3182 2.7864)/1.2547
(6.8541 3.6730)/1.2547 = 3.1811/1.2547 2.5353
There is a deviation from the exact solution in the last digit only.
NumCSE, Lecture 6, Oct 6, 2014
21/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
GE with partial pivoting
GEPP for A x = b
With partial pivoting we have after n 1 stages an upper
triangular matrix U,
U = A(n1) = L(n1) P (n1) L(2) P (2) L(1) P (1) A.
For n = 4 this is
U = L(3) P (3) L(2) P (2) L(1) P (1) A.
This can be rewritten as
U = L(3) (P (3) L(2) P (3)
|
)(P (3) P (2) L(1) P (2)
{z
L1
P (3)
) (P (3) P (2) P (1) ) A.
}|
{z
}
P
This can be generalized for n > 4.
NumCSE, Lecture 6, Oct 6, 2014
22/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
GE with partial pivoting
The GEPP solution procedure
1. Compute LU factorization of A with column pivoting
LU = PA.
Then LU x = PA x = P b.
2. Solve L c = P b by forward substitution.
3. Solve U x = c by backward substitution.
Note: L is a unit lower triangular matrix, i.e., it has ones on the
diagonal and
|lik | 1,
1 i, k n.
Note: P is not stored as a matrix but just as a vector!
NumCSE, Lecture 6, Oct 6, 2014
23/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
GE with partial pivoting
Vectorized GEPP algorithm
p = 1:n;
% initialize permutation vector p
for k=1:n-1
% find row index of relative maximum in column k
[~,q] = max(abs(a(k:n,k)));
q = q + k - 1;
% interchange rows k and q and record this in p
a([k,q],:)=a([q,k],:);
p([k,q])=p([q,k]);
% compute the corresponding column of L
J = k+1:n;
A(J,k) = A(J,k) / A(k,k);
% update submatrix by outer product
A(J,J) = A(J,J) - A(J,k) * A(k,J);
end
NumCSE, Lecture 6, Oct 6, 2014
24/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
GE with partial pivoting
GEPP stability
Question: Does the incorporation of a partial pivoting procedure
guarantee stability of the Gaussian elimination algorithm, in the
sense that roundoff errors do not get amplified by factors that
grow unboundedly as the matrix size n increases?
We have seen that |lik | 1.
What about |uik |?
We would like to be sure that
(k)
gn (A) = max |ai,j |,
i,j,k
i.e., the elements in U do not grow too fast in the course of GE.
Unfortunately, it is not possible in general to guarantee stability of
GEPP.
NumCSE, Lecture 6, Oct 6, 2014
25/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
GE with partial pivoting
Wilkinsons counter example
An =
((aij ))ni,j=1
A10
1
1
1
1
1
1
1
1
1
1
0
1
1
1
1
1
1
1
1
1
1
aij = 1
with
0
0
1
1
1
1
1
1
1
1
0
0
0
1
1
1
1
1
1
1
0
0
0
0
1
1
1
1
1
1
, if i = j j = n
, if i > j
else.
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
0
0
0
1
1
0
0
1 1
1
0
1 1 1
1
1 1 1 1
1
1
1
1
1
1
1
1
1
1
Partial pivoting does not trigger row permutations !
NumCSE, Lecture 6, Oct 6, 2014
26/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
GE with partial pivoting
Complete pivoting
In complete pivoting we look for (one of) the largest element in
modulus
(k1)
(k1)
|aqr | = max |aij
|
ki,jn
at each stage of GE and interchange both row q and column r
with row i and column j, respectively.
This strategy does guarantee stability. However, searching in each
stage (n k)2 elements makes it significantly more expensive than
partial pivoting.
Instances in which (scaled) partial pivoting really fails appear to be
extremely rare in practice = GEPP is the commonly used
approach.
NumCSE, Lecture 6, Oct 6, 2014
27/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
When pivoting is not needed: diagonally dominant matrices
When pivoting is not needed
Definition
A matrix A Rnn is diagonally dominant, if
X
|aii |
|aik |,
i = 1, . . . , n.
()
k6=i
Theorem
If A is nonsingular and diagonally dominant then the LU
factorization can be computed without pivoting.
NumCSE, Lecture 6, Oct 6, 2014
28/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
When pivoting is not needed: diagonally dominant matrices
Proof
We show that after reduction of the first row, the reduced system
is again diagonally dominant. We have
(1)
aik = aik
ai1 a1k
,
a11
i, k = 2, . . . , n.
For the diagonal elements we get the estimate
ai1 a1i
ai1 a1i
(1)
,
|aii |
|aii | = aii
a11
a11
i = 2, . . . , n.
We now want to show that
NumCSE, Lecture 6, Oct 6, 2014
(1)
|aii |
n
X
(1)
|aik |,
i = 2, . . . , n.
k=2
k6=i
29/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
When pivoting is not needed: diagonally dominant matrices
Proof (cont.)
Now for the sum of the moduli of the off-diagonal elements of row
i (i = 2, . . . , n) we get
n
X
k=2
k6=i
(1)
|aik |
n
n
n
X
ai1 X
ai1 a1k X
|aik | +
|a1k |
=
aik a11
a
11
k=2
k=2
k=2
k6=i
n
X
k=1
k6=i
k6=i
k6=i
)
( n
ai1 X
|aik | |ai1 | +
|a1k | |a1i |
a11
k=2
ai1
ai1 a1i (1)
a
|aii | |ai1 | +
{|a11 | |a1i |} = |aii |
ii
a11
a11
NumCSE, Lecture 6, Oct 6, 2014
30/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
When pivoting is not needed: symmetric positive definite systems
Symmetric positive definite systems
Definition
A symmetric matrix A is positive definite, if the corresponding
quadratic form is positive definite, i.e., if
Q(x) := xT A x > 0,
NumCSE, Lecture 6, Oct 6, 2014
for all x 6= 0
31/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
When pivoting is not needed: symmetric positive definite systems
Symmetric, positive definite systems
Theorem
If a symmetric matrix A Kn,n is positive definite, then the
following conditions hold
1. aii > 0 for i = 1, . . . , n.
2 <a a
2. aik
ii kk for i 6= k, i, k = 1, . . . , n.
3. There is a k with maxi,k |aik | = akk
Proof.
1. aii = eT
i Aei > 0
2. (ei + ek )T A(ei + ek ) = aii 2 + 2aik + akk > 0, . This
quadratic equation has no real zero , therefore its
2 4a a
discriminant 4aik
ii kk must be negative.
3. Clear.
NumCSE, Lecture 6, Oct 6, 2014
32/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
When pivoting is not needed: symmetric positive definite systems
Gaussian elimination works w/o pivoting. Since a11 > 0 we have
a11
aT
a11 aT
1
0T
1
1
A=
=
a1 A1
a1 /a11 I
0 A1 a1 aT
1 /a11
T
T
a11 0
1
0
1 aT
/a11
1
=
a1 /a11 I
0
I
0 A(1)
n1 \ {0} we have
with A(1) = A1 a1 aT
1 /a11 . But for any x R
T
a11 0T
0
0
T (1)
x A x=
(1)
x
x
0 A
T
0
0
1
0T
a11 aT
1 aT
/a11
1
1
=
x
x
a1 /a11 I
a1 A1
0
I
|
{z
}
|
{z
}
y
T
y
= yT Ay > 0.
NumCSE, Lecture 6, Oct 6, 2014
33/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
When pivoting is not needed: symmetric positive definite systems
Cholesky decomposition
From the proof we see that
A = LU = LDLT
with
U = DLT
Note that L is a unit lower triangular matrix and D is a diagonal
matrix with positive diagonal entries.
We can easily compute the positive definite diagonal matrix D 1/2
with D 1/2 D 1/2 = D.
Definition
Let L1 = LD 1/2 . Then
A = L1 LT
1
is called the Cholesky decomposition of A.
NumCSE, Lecture 6, Oct 6, 2014
34/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
When pivoting is not needed: symmetric positive definite systems
Matlab Cholesky decomposition
function [A]=mychol1(A)
%MYCHOL1
L=mychol1(A) computes the Cholesky factorization of A
%
using only the lower triangle of A.
%
It overwrites the lower triangle of A with L.
n = size(A,1);
for k=1:n,
if A(k,k) <= 0,
disp(matrix must be positive definite.)
return
end
A(k,k) = sqrt(A(k,k));
A(k+1:n,k) = A(k+1:n,k)/A(k,k);
for j=k+1:n
A(j:n,j) = A(j:n,j) - A(j:n,k)*A(j,k);
end
end
NumCSE, Lecture 6, Oct 6, 2014
35/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
When pivoting is not needed: symmetric positive definite systems
Cholesky decomposition: Solving systems and complexity
Linear system solving with the Cholesky factorization:
A = LLT
Lc = b
LT x = c
(Cholesky decomposition)
(Forward substitution)
(Backward substitution)
The complexity is half that of the LU factorization:
1 3
n + O(n2 ).
3
This can be easily be read from the code.
Matlab function for Cholesky factorization: chol
NumCSE, Lecture 6, Oct 6, 2014
36/40
Numerical Methods for Computational Science and Engineering
Gaussian elimination
Efficient implementation
Efficient implementation
I
In general, the performance of an algorithm depends not only
on the number of arithmetic operations that are carried out
but also on the frequency of memory accesses.
A first step towards efficiently accessing memory is
vectorization.
Matlab supports vector operations in a convenient way.
However, good performance is only achievable by blocking
algorithms.
Only blocks of data increase the number of floating point
operations (flops) versus memory accesses (in Bytes) beyond
O(1).
NumCSE, Lecture 6, Oct 6, 2014
37/40
Numerical Methods for Computational Science and Engineering
LAPACK and the BLAS
LAPACK and the BLAS
I
LAPACK (Linear Algebra PACKage) is a library of
subroutines (functions) for solving the most commonly
occuring problems in numerical linear algebra. It has been
designed to be efficient on a wide range of high-performance
computers with a hierarchy of memory levels.
LAPACK is written in a way that as much as possible of the
computation is performed by calls to the Basic Linear Algebra
Subprograms (BLAS) that are designed to exploit the fast
memories (caches) of todays high-performance computers.
I
I
BLAS-1: vector operations (scaling, norm, saxpy)
BLAS-2: matrix-vector operations (matvec, forward/backward
substitution)
BLAS-3: matrix-matrix operations (blocked operations)
NumCSE, Lecture 6, Oct 6, 2014
38/40
Numerical Methods for Computational Science and Engineering
LAPACK and the BLAS
LAPACK and the BLAS (cont.)
I
The software is freely available at
http://www.netlib.org/lapack
The LINPACK benchmark gives rise to the TOP500 list, the
list of the 500 most powerful computers:
http://www.top500.org.
(LINPACK is a precursor LAPACK.)
NumCSE, Lecture 6, Oct 6, 2014
39/40
Numerical Methods for Computational Science and Engineering
LAPACK and the BLAS
Important remark
I
I
I
I
Do not implement Gaussian elimination yourself!
Use numerical libraries (LAPACK), NAG, MKL, or MATLAB!
MATLAB operator: \
In fact: all implementations are based on LAPACK.
NumCSE, Lecture 6, Oct 6, 2014
40/40
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
I
Gaussian elimination for linear systems of equations Ax = b
continued.
Sparse linear systems
Fill-in
Condition of linear systems
Error estimation
NumCSE, Lecture 7, Oct 9, 2014
2/41
Numerical Methods for Computational Science and Engineering
Sparse matrices
Where do sparse matrices arise?
I
Conceptually, sparsity corresponds to systems which are
loosely coupled.
Networks
I
I
I
I
Low density of significant data or connections.
Internet
Electric circuits
Flow through tubes
Numerical solution of partial differential equations.
I
Finite element, finite difference discretizations.
NumCSE, Lecture 7, Oct 9, 2014
3/41
Numerical Methods for Computational Science and Engineering
Sparse matrices
Example: Electrical network
Node =
junction of wires
(number of nodes 1, . . . , n)
Unknowns: nodal potentials
Uk , k = 1, . . . , n.
(some may be known: grounded
nodes, voltage sources)
Ikj : current from nodes k j,
Ikj = Ijk
Kirchhoff current law (Kirchhoffsche Knotenregel):
Pn
Sum of node currents is zero:
nodes k
j=1 Ikj = 0,
Kirchhoff voltage law (Maschenregel): Directed sum of electrical
potential differences (voltage) around any closed network is zero.
NumCSE, Lecture 7, Oct 9, 2014
4/41
Numerical Methods for Computational Science and Engineering
Sparse matrices
VLSI chips
Modern electric circuits (VLSI chips):
105 107 circuit elements
I Each element is connected to only
a few nodes
I
Each node is connected to only a
few elements
[In the case of a linear circuit]
nodal analysis sparse circuit matrix
VLSI = very-large-scale integration
NumCSE, Lecture 7, Oct 9, 2014
5/41
Numerical Methods for Computational Science and Engineering
Sparse matrices
Introduction to sparse matrices
Introduction to sparse matrices
What is a sparse matrix?
Informal working definition by J.H. Wilkinson (1969):
Any matrix with enough zeros that it pays to take
advantage of them
This is not a real definition.
I
Sparsity not so much a matter of fraction of nonzeros.
Its the economy, ...: memory, operations, computing time.
Remark: More generally: structurally sparse matrices are matrices that
can be represented with few numbers. E.g., the rank-1 matrix H = uvT
may be dense. However, one needs to store only two vectors to have all
the information to do a matrix-vector product.
NumCSE, Lecture 7, Oct 9, 2014
6/41
Numerical Methods for Computational Science and Engineering
Sparse matrices
Introduction to sparse matrices
Introduction to sparse matrices (cont.)
Various practical aspects of sparsity:
I
Storage of the sparse matrix
Access of matrix elements
Operations on the matrix
(here: LU and Cholesky decomposition)
Operations with the matrix
(later: multiplication of a sparse matrix with a vector)
NumCSE, Lecture 7, Oct 9, 2014
7/41
Numerical Methods for Computational Science and Engineering
Sparse matrices
Banded matrices
Banded matrices
a11
..
.
a
p1
A=
I
I
I
I
I
. . . a1q
..
.
..
.
..
.
..
.
..
.
..
..
..
.
..
..
.
..
.
an,np+1 . . .
anq+1,n
..
.
ann
So, aij = 0, if i j + p, or if j i + q.
The bandwidth is p + q 1.
Bandwidth = 1: diagonal matrices (p = q = 1)
Bandwidth = 3: tridiagonal matrices (p = q = 2)
Matrices of this type are supported by LAPACK.
NumCSE, Lecture 7, Oct 9, 2014
8/41
Numerical Methods for Computational Science and Engineering
Sparse matrices
Banded matrices
Band matrices: storage
a11
a21
A=
a31
a12
a22 a23
a32 a33 a34
a42 a43 a44 a45
a53 a54 a55
Option: storage row-wise by diagonals (LAPACK)
a11
a21
a31
a12
a22
a32
a42
a23 a34 a45 }super-diagonal 1
a33 a44 a55
}main diagonal
}sub-diagonal -1
a43 a54
}sub-diagonal -2
a53
Symmetric/Hermitian A: storing only lower/upper half.
NumCSE, Lecture 7, Oct 9, 2014
9/41
Numerical Methods for Computational Science and Engineering
Sparse matrices
Banded matrices
Gaussian Elimination without pivoting
0
0
0
0
0 0 0
0
0
0
0 0
0
0 0
0 0
Essential for structure: no row/column interchanges.
Flop count: if p and q are constant as n grows then cost is O(n),
which is a major improvement over the dense matrix case.
NumCSE, Lecture 7, Oct 9, 2014
10/41
Numerical Methods for Computational Science and Engineering
Sparse matrices
Banded matrices
Gaussian Elimination with pivoting (GEPP)
I
I
Without pivoting, triangular factors inherit band structure
In general, pivoting destroys original band structure:
I
I
The upper bandwidth of U may increase from q to q + p 1.
The matrix L, although it remains unit lower triangular, is no
longer tightly banded (although it has the same number of
nonzeros per column).
In practice: Store L at its original place in A.
Do not apply subsequent permutations to columns of L!
Stick with U = A(n1) = L(n1) P (n1) L(2) P (2) L(1) P (1) A.
NumCSE, Lecture 7, Oct 9, 2014
11/41
Numerical Methods for Computational Science and Engineering
Sparse matrices
Banded matrices
First lessons learned
Seen: first examples of sparse matrices
Issue: Take advantage of zeros (Wilkinson).
Observed:
I
Specialized data structures needed.
Pivoting may conflict with structure of nonzeros.
NumCSE, Lecture 7, Oct 9, 2014
12/41
Numerical Methods for Computational Science and Engineering
Sparse Matrix Storage Formats
Sparse Matrix Storage Formats
Matlab uses coordinate list (COO) format to display sparse
matrices.
You can enter a sparse matrix using the COO format:
S = sparse(i, j, aij ); S = sparse(i, j, aij , n, m);
Simple! But in general not efficient regarding memory access.
Most often used:
Compressed Row Storage (CRS): good for matvec.
Compressed Column Storage (CCS) (Matlab):
suited for column pivoting.
NumCSE, Lecture 7, Oct 9, 2014
13/41
Numerical Methods for Computational Science and Engineering
Sparse Matrix Storage Formats
Example: CRS format
A=
2
3
4
5
6
8
7
10
CRS uses three vectors for storing nonzeros row-wise:
val =
[1, 2, 5, 3, 6, 9, 4, 8, 7, 10]
col ind =
[1, 2, 3, 2, 3, 5, 2, 4, 3, 5]
row ptr =
[1, 2, 4, 7, 9, 11]
Explanation:
1. real/complex vector for the matrix elements,
2. int vector with column index of each matrix element,
3. int vector, encoding the partition of val and row ptr.
NumCSE, Lecture 7, Oct 9, 2014
14/41
Numerical Methods for Computational Science and Engineering
Sparse Matrix Storage Formats
Sparse matrices in Matlab
Initialization:
S = sparse(A);
[i,j,s] = find(S);
[m,n] = size(S);
A = sparse(i,j,s,m,n);
A = sparse(m,n); A = spalloc(m,n,nnz)
A = spdiags(B,d,m,n); A = speye(n); A = spones(S);
spy(A)
NumCSE, Lecture 7, Oct 9, 2014
15/41
Numerical Methods for Computational Science and Engineering
Sparse Factorizations and fill-in
Sparse Factorizations and fill-in
I
We now consider the LU (Cholesky) factorization of a sparse
matrix A.
The factors L and U are in general sparse again.
However, the structure of the nonzeros is in general different.
There may be nonzeros in the triangular factors at positions
where there were zeros in A. This is called fill-in.
Triangular factors generally have more, even many more (!),
nonzero entries than the original matrix.
NumCSE, Lecture 7, Oct 9, 2014
16/41
Numerical Methods for Computational Science and Engineering
Sparse Factorizations and fill-in
Extreme example of fill-in: arrow matrices
Extreme example of fill-in: arrow matrices
Corresponding lower triangular factors (w/o pivoting):
Reordering can make a difference!
NumCSE, Lecture 7, Oct 9, 2014
17/41
Numerical Methods for Computational Science and Engineering
Sparse Factorizations and fill-in
Extreme example of fill-in: arrow matrices
Permutations and ordering strategies
What can we do to obtain little or no fill-in?
I
No perfect solution for this problem.
Occasionally, permuting rows and columns of A significantly
helps in reducing fill-in.
Leads to considerations involving graph theory.
Common goals are:
Reduce the bandwidth of the matrix; e.g., reverse
Cuthill-McKee (RCM)
Matlab: symrcm
Reduce the expected fill-in in the decomposition stage; e.g.,
(approximate) minimum degree (AMD).
Matlab: amd,symamd
NumCSE, Lecture 7, Oct 9, 2014
18/41
Numerical Methods for Computational Science and Engineering
Sparse Factorizations and fill-in
Fill-in reduction by reordering
Example model: Poisson problem
n=10;
A=gallery(poisson,n);
spy(A);
LU decomposition fills the 2 n inner zero diagonals.
NumCSE, Lecture 7, Oct 9, 2014
19/41
Numerical Methods for Computational Science and Engineering
Sparse Factorizations and fill-in
Fill-in reduction by reordering
Model Poisson problem
n=8;
A = gallery(poisson,n);
subplot(1,3,1), spy(A)
p = symrcm(A);
p = symamd(A);
subplot(1,3,2), spy(Arcm,red)
subplot(1,3,3), spy(Aamd,green
Arcm = A(p,p);
Aamd = A(p,p);
pause
L = chol(A,lower);
Lrcm = chol(Arcm,lower);
Lamd = chol(Aamd,lower);
NumCSE, Lecture 7, Oct 9, 2014
subplot(1,3,1), spy(L)
subplot(1,3,2), spy(Lrcm,red)
subplot(1,3,3), spy(Lamd,green)
20/41
Numerical Methods for Computational Science and Engineering
Error analysis
Error estimation
Error estimation
Two questions regarding the accuracy of
x as an approximaton to
the solution x of the linear system of equations A x = b.
1. First we investigate what we can derive from the size of the
residual r = b A
x.
Note that r = b A x = 0.
2. Then, what is the effect of errors in the initial data (b, A) on
the solution x? That is, how sensitive is the solution to
perturbations in the initial data?
NumCSE, Lecture 7, Oct 9, 2014
21/41
Numerical Methods for Computational Science and Engineering
Error analysis
Error estimation
Error estimation (cont.)
Example 5.18. Let
1.2969 0.8648
A=
,
0.2161 0.1441
0.8642
b=
0.1440
Suppose somebody came up with the approximate solution
0.9911
x=
0.4870
108
108
r = b A x =
= krk = 108
2
Since x =
= k
x xk = 1.513 which is 108
2
times larger than the residual!
Then,
NumCSE, Lecture 7, Oct 9, 2014
22/41
Numerical Methods for Computational Science and Engineering
Error analysis
Error estimation
Error estimation (cont.)
How does the residual r := b A
x relate to the error z := x x?
We assume that kAk is any matrix norm and kxk a compatible
vector norm, i.e., we have kAxk kAkkxk for all kxk.
We have
Az = A(
x x) = A
x b = r.
Therefore
kbk = kAxk kAkkxk,
kzk = kA1rk kA1 kkrk
We get an estimate for the relative error of x:
kzk
k
x xk
krk
krk
=
kAkkA1 k
=: (A)
kxk
kxk
kbk
kbk
NumCSE, Lecture 7, Oct 9, 2014
23/41
Numerical Methods for Computational Science and Engineering
Error analysis
Error estimation
Error estimation (cont.)
Definition
The quantity
(A) = kAkkA1 k
is called the condition number of A.
The condition number is at least 1:
1 = kI k = kAA1 k kAkkA1 k = (A).
If A is nonsingular and E is the matrix with smallest norm such
that A + E is singular, then
NumCSE, Lecture 7, Oct 9, 2014
kE k2
1
=
.
kAk2
2 (A)
24/41
Numerical Methods for Computational Science and Engineering
Error analysis
Error estimation
Error estimation (cont.)
Example 5.19. (Previous example continued.)
0.1441 0.8648
1
8
A = 10
,
0.2161
1.2969
This yields
kA1 k = 1.513108 = (A) = 2.1621.513108 3.27108 .
The numbers
1.513
2
<
3.27
0.8642
confirm the estimate
kzk
krk
(A)
kxk
kbk
NumCSE, Lecture 7, Oct 9, 2014
25/41
Numerical Methods for Computational Science and Engineering
Error analysis
Error estimation
Error estimation (cont.)
a21
We now make A singular by replacing a22 by a12
a11 .
Then A + E is singular with
0
0
0
0
E=
,
0 7.7 109
0 0.86480.2161
0.1441
1.2969
So, indeed,
kE k
1
.
kAk
(A)
(This estimate holds in `2 and ` norms.)
NumCSE, Lecture 7, Oct 9, 2014
26/41
Numerical Methods for Computational Science and Engineering
Error analysis
Condition number
Sensitivity on matrix coefficients
I
Input data A and b are often perturbed due, e.g., to rounding.
How big is the change x if the matrix A is altered by A and
b is perturbed by b
The LU decomposition can be considered as the exact
= A + A.
decomposition of a perturbed matrix A
Forward/backward substitition add (rounding) errors.
Altogether: The computed solution
x is the solution of a
perturbed problem
(A + A)
x = b + b,
x = x + x.
How do these perturbations affect the solution?
This is called backward error analysis.
NumCSE, Lecture 7, Oct 9, 2014
27/41
Numerical Methods for Computational Science and Engineering
Error analysis
Condition number
Sensitivity on matrix coefficients (cont.)
(A + A)(x + x) = (b + b)
Multiplying out:
A x + A x + A x + A x = b + b
or, with A x = b,
A x + A x + A x = b
or, equivalently,
NumCSE, Lecture 7, Oct 9, 2014
Ax = b A x A x.
28/41
Numerical Methods for Computational Science and Engineering
Error analysis
Condition number
Sensitivity on matrix coefficients (cont.)
Thus,
x = A1 (b A x A x) .
For compatible norms we have
kxk = kA1 kkb A x A xk
kA1 k (kbk + kAkkxk + kAkkxk)
From this we get
(1 kA1 kkAk) kxk kA1 k (kbk + kAkkxk)
NumCSE, Lecture 7, Oct 9, 2014
29/41
Numerical Methods for Computational Science and Engineering
Error analysis
Condition number
Sensitivity on matrix coefficients (cont.)
Now assume that the perturbation in A is small: kA1 kkAk < 1.
Then
kA1 k
(kbk + kAkkxk)
kxk
(1 kA1 kkAk)
Since we are interested in an estimate of the relative error we use
kbk = kAxk kAkkxk = kxk kbk/kAk =
1
kAk
.
kxk
kbk
Therefore, we have
kxk
kA1 k
kbk
+
kAk
kxk
(1 kA1 kkAk) kxk
kA1 kkAk
kbk kAk
+
(1 kA1 kkAk) kbk
kAk
NumCSE, Lecture 7, Oct 9, 2014
30/41
Numerical Methods for Computational Science and Engineering
Error analysis
Condition number
Sensitivity on matrix coefficients (cont.)
With the condition number (A) = kAkkA1 k we get
kxk
(A)
kbk kAk
+
kxk
kAk
1 (A) kAk kbk
kAk
The condition number (A) is the decisive quantity that describes
the sensitivity of the solution x on both relative changes in A as
well as in b.
If (A) 1, small perturbations in A can lead to large
relative errors in the solution of the linear system of equations.
If (A) 1, a stable algorithm can produce solutions with
large relative error!
A stable algorithm produces (acceptably) small errors if the
problem is well-conditioned (i.e. (A) not too large).
NumCSE, Lecture 7, Oct 9, 2014
31/41
Numerical Methods for Computational Science and Engineering
Error analysis
Condition number
Rule of thumb
Lets assume we compute with d decimal digits such that the
initial errors in the input data are about
kbk
5 10d ,
kbk
kAk
5 10d .
kAk
Lets assume that the condition number of A is 10 .
If 5 10d 1 then we get
kxk
10d+1 .
kxk
NumCSE, Lecture 7, Oct 9, 2014
32/41
Numerical Methods for Computational Science and Engineering
Error analysis
Condition number
Rule of thumb (cont.)
Rule of thumb: If a linear system A x = b is solved with d
digit floating point numbers and (A) 10 then, due to unavoidable errors in the initial data, we have to expect relative
errors in x in the order of 10d+1
Note that this statement is about the largest components of x.
The small components can suffer bigger errors!
NumCSE, Lecture 7, Oct 9, 2014
33/41
Numerical Methods for Computational Science and Engineering
Error analysis
Condition number
Note on A and b
It can be shown that Gaussian elimination with pivoting yield a
perturbation bounded by
kAk (n)gn (A)
where (n) is a low order polynomial in the order of the matrix n
(cubic at most) and is the rounding unit. The bound on forward
and backward substitutions and on b are significantly smaller.
Thus, as long as the pivoting keeps gn (A) growing only moderately
and n is not too large then the overall perturbations A and b are
not larger than a few orders of magnitude times .
NumCSE, Lecture 7, Oct 9, 2014
34/41
Numerical Methods for Computational Science and Engineering
Error analysis
Scaling
Scaling
I
I
It would be nice to have well-conditioned problems.
Can we easily improve the condition of a linear system before
starting the LU factorization?
Scaling is a possibility that often works.
Let D1 and D2 be diagonal matrices. The solution of A x = b can
be found by solving the scaled system
D11 AD2 y = D11 b,
by Gaussian elimination and then setting
x = D2 y.
Scalings of A, b, y, require only O(n2 ) flops.
Often D2 = I , i.e., only row scaling, unless matrix is symmetric.
NumCSE, Lecture 7, Oct 9, 2014
35/41
Numerical Methods for Computational Science and Engineering
Error analysis
Scaling
Scaling (cont.)
Example (Forsythe & Moler)
10 1000 000
1
1
0
x1
100 000
=
x2
2
The equivalent row-scaled problem is
0.0001 1 x1
1
=
1
1 x2
2
The solutions with 3 decimal digits are
x = (0.00, 1.00)T for the
T
unscaled system, and
x = (1.00, 1.00) for the scaled system. The
correct solution is x = (1.0001 . . . , 0.9999 . . .)T
NumCSE, Lecture 7, Oct 9, 2014
36/41
Numerical Methods for Computational Science and Engineering
Error analysis
Scaling
Example 5.11 (modified) from AscherGreif
% Example 5.11 : comparing scaled and unscaled GEPP
% for a special example
% Modified 1.10.2013 (PA)
n = 100; h = 1/(n-1); K = 100;
A = zeros(n,n);
for i = 2:n-1
A(i,i) = -2/h^2 - K;
A(i,i-1) = -1/h^2; A(i,i+1) = -1/h^2;
end
A(1,1) = 1; A(n,n) = 1; % end definition of A
xe = ones(n,1);
b = A*xe;
NumCSE, Lecture 7, Oct 9, 2014
% exact solution of 1s
% corresponding right hand side
37/41
Numerical Methods for Computational Science and Engineering
Error analysis
Scaling
Example 5.11 (modified) from AscherGreif (cont.)
%solve w/o scaling
xu = A\b;
err = norm(xu-xe)
%determine scaling matrix D
D=ones(n,1);
for i=1:n
D(i) = norm(A(i,:),inf);
end
%solve using scaled GEPP
A1 = diag(D)\A;
xs = A1\(D.\b);
err_scaled = norm(xs-xe)
NumCSE, Lecture 7, Oct 9, 2014
38/41
Numerical Methods for Computational Science and Engineering
Error analysis
Scaling
Theorem on scaling
Theorem (`1 row scaling)
Let A Rnn be nonsingular. Let the diagonal matrix Dz be
defined by
1
n
X
dii = |aij | .
j=1
Then
(Dz A) (DA)
for all nonsingular diagonal D
See: Dahmen & Reusken: Numerik f
ur Ingenieure und Naturwissenschaftler.
2nd ed. Springer 2008.
NumCSE, Lecture 7, Oct 9, 2014
39/41
Numerical Methods for Computational Science and Engineering
Error analysis
Remarks
Remark on determinants
Although det(A) = 0 for singular matrices, small determinants do
not indicate bad condition!
Example (Small determinant)
A = diag(0.1, 0.1, . . . , 0.1) Rnn has det(A) = 10n and
(A) = 1.
Example (Unit determinant)
The condition number of A below grows exponentially but
det(A) = 1.
1 1 1
..
..
..
.
.
.
A=
..
. 1
1
NumCSE, Lecture 7, Oct 9, 2014
40/41
Numerical Methods for Computational Science and Engineering
Error analysis
Remarks
Condition estimation
After having solved A x = b via PA = LU we would like to
ascertain the number of correct digits in the computed
x.
Need estimate of (A) = kAk kA1 k
Known: kAk = max
n
P
|aij |
1in j=1
How get at kA1 k ?
Idea: A y = d = kA1 k kyk /kdk
Find d with kdk = 1 such that kyk is as big as possible.
Observation: Estimation of kU 1 k gives a good approximation
of kA1 k . Solve U y = d with di = 1.
NumCSE, Lecture 7, Oct 9, 2014
41/41
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
A basic problem in science is to fit a model to observations subject
to errors.
The more observations that are available, the more accurately will
it be possible to calculate the parameters in the model.
This gives rise to the problem of solving overdetermined linear or
nonlinear systems of equations.
(Ake Bj
orck, Numerical Methods for Least Squares Problems, SIAM,
1996)
NumCSE, Lecture 8, Oct 13, 2014
2/25
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today (cont.)
I
We discuss how to solve overdetermined linear systems of
equations
A x b,
b Rm , x Rn ,
m>n
or
These systems do in general not have a solution.
Today: Normal equations
Next time: QR factorization
m n
Reference
I
AscherGreif, Chapter 6.
NumCSE, Lecture 8, Oct 13, 2014
3/25
Numerical Methods for Computational Science and Engineering
Introduction
Origins of linear least squares problems
Data fitting
http://en.wikipedia.org/wiki/Least_squares
NumCSE, Lecture 8, Oct 13, 2014
4/25
Numerical Methods for Computational Science and Engineering
Introduction
Origins of linear least squares problems (cont.)
Coordinate metrology
Points measured on
two orthogonal lines
g1 and g2 .
Gander, Gander, Kwok: Computational Science, Springer, 2014
NumCSE, Lecture 8, Oct 13, 2014
5/25
Numerical Methods for Computational Science and Engineering
Introduction
Origins of linear least squares problems (cont.)
Procrustes or registration problem
Want to find
orthogonal matrix
Q and translation
vector t such that
Qx + t maps one
tetrahedron into the
other.
Gander, Gander, Kwok: Computational Science, Springer, 2014
NumCSE, Lecture 8, Oct 13, 2014
6/25
Numerical Methods for Computational Science and Engineering
Introduction
Origins of linear least squares problems (cont.)
Image morphing
Schaefer, McPhail, Warren: Image deformation using moving least squares
http://faculty.cs.tamu.edu/schaefer/research/mls.pdf
NumCSE, Lecture 8, Oct 13, 2014
7/25
Numerical Methods for Computational Science and Engineering
Linear least squares
Linear least squares
Consider overdetermined system on m linear equations in n
unknowns:
A x b,
A Rmn ,
b Rm ,
x Rn ,
m > n.
We assume the the matrix A has maximal column rank n.
This means that the columns of A are linearly independent.
In general, this system cannot be solved exactly: b does not lie in
the range space of A.
Reformulate problem:
Find x such that
NumCSE, Lecture 8, Oct 13, 2014
1
kb Axk22
2
is minimized.
8/25
Numerical Methods for Computational Science and Engineering
Linear least squares
Linear least squares (cont.)
We introduce residual
r = b Ax Rm
NumCSE, Lecture 8, Oct 13, 2014
9/25
Numerical Methods for Computational Science and Engineering
Linear least squares
Linear least squares (cont.)
Minimization problem for a smooth scalar function in n variables
min (x),
x
1
with (x) = krk22
2
Necessary conditions for a minimum:
grad (x) = 0
NumCSE, Lecture 8, Oct 13, 2014
(x) = 0,
xk
k = 1, . . . , n.
10/25
Numerical Methods for Computational Science and Engineering
Linear least squares
Linear least squares (cont.)
1
1
(x) = rT r = (Ax b)T (Ax b)
2
2
1 T T
= (x A A x xT AT b bT A x + bT b)
2
1
1
= xT AT A x (AT b)T x + bT b.
2
2
grad (x) = AT A x AT b = AT (A x b) = 0
1
(x + x) = kb A x A xk22
2
1
1
= kb A xk22 (b A x)T A x + kA xk22
2
2
1
2
= (x) + kA xk2 (x)
2
NumCSE, Lecture 8, Oct 13, 2014
11/25
Numerical Methods for Computational Science and Engineering
Linear least squares
Linear least squares (cont.)
B := AT A,
B Rnn .
A has maximal rank = B is SPD with quadratic form
Q(z) = zT B z = zT AT A z = (Az)T (Az) 0,
for all z
Q(z) = 0 Az = 0 z = 0
So, in particular,
1
kA xk22 > 0,
2
for all x 6= 0.
and
(x + x) > (x),
NumCSE, Lecture 8, Oct 13, 2014
for all x 6= 0.
12/25
Numerical Methods for Computational Science and Engineering
Linear least squares
Normal equations
Theorem: Least squares
The least squares problem
min kAx bk2 ,
x
where A has full column rank, has a unique solution that satisfies
the normal equations
(AT A)x = AT b.
1 T
We have x = AT A
A b. The matrix multiplying b is called
the pseudo-inverse of A:
NumCSE, Lecture 8, Oct 13, 2014
A+ = (AT A)1 AT Rnm .
13/25
Numerical Methods for Computational Science and Engineering
Linear least squares
Geometrical interpretation
Geometrical interpretation
From
grad (x) = AT A x AT b = AT (A x b) = 0
we see that AT r = 0.
The blue plane shows R(A). A x
is the orthogonal projection of b
onto R(A).
NumCSE, Lecture 8, Oct 13, 2014
Ax
14/25
Numerical Methods for Computational Science and Engineering
Linear least squares
Algorithm
Algorithm: Least squares via normal equations
1. B = AT A, y = AT b
2. B = LL
3. L z = y,
(normal equations B x = y)
(Cholesky factorization)
LT x
4. r = Ax b
=z
(Forward / backward substitution)
(Compute residual)
Complexity:
I
Step 1: mn2 + O(mn) flops
Step 2:
Steps 3: O(n2 ) flops
Steps 4: O(mn) flops
1 3
3n
+ O(n2 ) flops
Forming B dominates computations cost, in particular if m n.
NumCSE, Lecture 8, Oct 13, 2014
15/25
Numerical Methods for Computational Science and Engineering
Linear least squares
Example
Example 6.1 from AscherGreif
1
2
A=
5
3
1
0
1
3
5
53
3 2
R ,
5
4
6
3
4
2
5
b=
5 R
2
1
For solving the LSP minx kb Axk2 we form B and y:
40 30 10
18
B = AT A = 30 79 47 ,
y = AT b = 5
10 47 55
21
The final solution is x = (0.3472, 0.3990, 0.7859)T , correct to
the number of digits shown. The residual is
r = (4.4387, 0.0381, 0.495, 1.893, 1.311)T .
NumCSE, Lecture 8, Oct 13, 2014
16/25
Numerical Methods for Computational Science and Engineering
Data fitting
Data fitting
Generally, data fitting problems arise as follows:
I
We have observed data b and a model function that for any
candidate model x provides predicted data.
The task is to find x such that the predicted data match the
observed data to the extent possible.
We want to minimize the difference of predicted and observed
data in the least squares sense.
Here, we study the linear case where predicted data are given
by Ax.
(The condition that A has maximal rank means that there is
no redundancy in the representation of the predicted data.)
NumCSE, Lecture 8, Oct 13, 2014
17/25
Numerical Methods for Computational Science and Engineering
Data fitting
Linear regression
Example 6.2 from AscherGreif: Linear regression
Consider fitting a given data set of m pairs (ti , bi ) by a straight
line:
v (t) = x1 + x2 t
=
v (ti ) bi , i = 1, . . . , m.
1 t1
!
Pm
1 t2
m
i=1 ti
A = . .
B = Pm
Pm 2
.. ..
t
t
i=1 i
i=1 i
1 tm
i
ti
bi
NumCSE, Lecture 8, Oct 13, 2014
1
2
3
0.0 1.0 2.0
0.1 0.9 2.0
0.05
x=
0.95
18/25
Numerical Methods for Computational Science and Engineering
Data fitting
Polynomial data fitting
Polynomial data fitting
Extend linear regression formula to a higher degree polynomial fit:
v (t) = x1 +x2 t+ xn t n1
v (ti ) bi ,
i = 1, . . . , m.
The matrix that we now obtain is called a Vandermonde matrix.
1 t0 t0n2 t0n1
1 t1 t n2 t n1
1
1
mn
A = . .
.
..
.. R
.. ..
.
.
n2 t n1
1 tm tm
m
Matlab code lsfit from AscherGreif on next slide.
NumCSE, Lecture 8, Oct 13, 2014
19/25
Numerical Methods for Computational Science and Engineering
Data fitting
Polynomial data fitting
Polynomial data fitting (cont.)
function coefs = lsfit (t, b, n)
%
% function coefs = lsfit (t, b, n)
%
% Construct coefficients of the polynomial of
% degree at most n-1 that best fits data (t,b)
t = t(:); b = b(:); % make sure t and b are column vectors
m = length(t);
% long and skinny A
A = ones(m,n);
for j=1:n-1
A(:,j+1) = A(:,j).*t;
end
NumCSE, Lecture 8, Oct 13, 2014
20/25
Numerical Methods for Computational Science and Engineering
Data fitting
Polynomial data fitting
Polynomial data fitting (cont.)
% normal equations and solution
B = A*A; y = A*b;
coefs = B \ y;
NumCSE, Lecture 8, Oct 13, 2014
21/25
Numerical Methods for Computational Science and Engineering
Data fitting
Polynomial data fitting
Example 6.3 from AscherGreif: Polynomial data fittig
Fit the function f (t) = cos(2t) at 21 equidistant points in the
interval [0, 1] and construct best fits by polynomials of degree at
most n 1 for each n = 1, 2, 3, 4, 5.
NumCSE, Lecture 8, Oct 13, 2014
22/25
Numerical Methods for Computational Science and Engineering
Coordinate metrology
Example of coordinate metrology
Points measured on two orthogonal lines
g1 :
c1 + n1 x + n2 y = 0,
n12 + n22 = 1,
g2 :
c2 n2 x + n1 y = 0,
n12 + n22 = 1,
NumCSE, Lecture 8, Oct 13, 2014
23/25
Numerical Methods for Computational Science and Engineering
Coordinate metrology
Example of coordinate metrology (cont.)
1
1
..
.
..
.
0
0
..
.
x1
x2
..
.
y1
y2
..
.
0
1
1
..
.
xm1
ym1 +1
ym1 +2
..
.
ym1
xm1 +1
xm1 +2
..
.
0 1 ym1 +m2
xm1 +m2
0
0
..
.
c1
c2 0
n1 0
0
n2
..
together with constraint n12 + n22 = 1.
(Without constraint the solution would be c1 = c2 = n1 = n2 = 0.)
NumCSE, Lecture 8, Oct 13, 2014
24/25
Numerical Methods for Computational Science and Engineering
Remarks
Remarks
I
Matlabs backslash operator does a least squares fit if the
matrix is m n, m > n.
Matlab has a function polyfit for polynomial fitting.
You can use polyval to evaluate the polynomial.
We could interpolate right away. But data may be noisy.
The model has fewer parameters anyway.
Other norms give different results. Data fitting with `1 or `
norms is much more involved than linear least squares and
leads to so-called linear programming problems.
NumCSE, Lecture 8, Oct 13, 2014
25/25
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
I
We continue to discuss how to solve overdetermined linear
systems of equations
A x b,
b Rm , x Rn ,
m > n,
where A has maximal column rank n.
I
The normal equations often suffer from ill-conditioned
matrices A.
We investigate more stable algorithms, based on the QR
factorization.
Reference
I
AscherGreif, Chapter 6.
NumCSE, Lecture 9, Oct 16, 2014
2/34
Numerical Methods for Computational Science and Engineering
Linear least squares
Linear least squares
I
The overdetermined system A x b is reformulated:
We actually solve the linear least squares problem:
Find x such that
1
kb Axk22
2
is minimized.
The solution x is found by solving the normal equations:
Bx (AT A)x = AT b.
For m n, most of the algorithm cost is in the formation of
B = AT A.
The way to go for many data fitting problems. But, difficulties
arise when A has (almost) lineraly dependent columns.
NumCSE, Lecture 9, Oct 16, 2014
3/34
Numerical Methods for Computational Science and Engineering
Linear least squares
Sensitivity of normal equations
Condition of normal equations
Let
A=
0
0
1
0
1
0
,
0
1 + 2
1
1
1 + 2
1 .
B = AT A = 1
1
1
1 + 2
For small the columns of A are almost parallel.
The eigenvalues of B are 2 (2) and 3 + 2 .
Thus, (B) 3/2 .
If < we lose all positions in the solution.
NumCSE, Lecture 9, Oct 16, 2014
4/34
Numerical Methods for Computational Science and Engineering
Linear least squares
Sensitivity of normal equations
Illustration of sensitivity of normal equations
The response (output) f (x) of a nonlinear transmission element is
linear in the input x for small x and asymptotically constant for
large x. So, we may write
x
f (x) = 1
+ 2 (1 e x ).
1+x
For a set of inputs xi the responses have been observed:
x=
y=
0.2
0.3
0.5
0.5
1.0
0.8
1.5
1.0
2.0
1.2
3.0
1.3
We determine the coefficient 1 , 2 by solving the overdetermined
linear system
xi
yi = f (xi ) = 1
+ 2 (1 e xi ),
i = 1, . . . , 6.
1 + xi
NumCSE, Lecture 9, Oct 16, 2014
5/34
Numerical Methods for Computational Science and Engineering
Linear least squares
Sensitivity of normal equations
Illustration of sensitivity of normal equations (cont.)
With 6 decimal digits the
0.166667
0.333333
0.500000
0.600000
0.666667
0.750000
overdetermined linear system is
0.3
0.181269
0.393469 0.5
0.632121
1 = 0.8
1.0
0.776870
2
1.2
0.864665
0.950213
1.3
The corresponding normal equation
1.75583 2.23266
1
2.99167
Bx=
=
= AT b
2.23266 2.84134
2
3.80656
1.32508
L=
,
B = LLT
1.68492 0.0487852
NumCSE, Lecture 9, Oct 16, 2014
6/34
Numerical Methods for Computational Science and Engineering
Linear least squares
Sensitivity of normal equations
Illustration of sensitivity of normal equations (cont.)
We get the characteristics
f (x) = 0.384196
x
+ 1.03782(1 e x ).
1+x
with residual
r (0.0478, 0.0364, 0.0481, 0.0368, 0.0465, 0.0257)T .
Since the eigenvalues of B are 1 4.59627 and 2 0.0009006,
the condition number of B is (B) 5104. We lost 4 out of 6
decimal digits.
A computation with 12 decimal digits gives
1 0.382495,
2 1.03915.
The residual does not change showing the sensitivity of the system.
NumCSE, Lecture 9, Oct 16, 2014
7/34
Numerical Methods for Computational Science and Engineering
Linear least squares
Sensitivity of normal equations
Condition number of rectangular matrices
For a rectangular matrix A Rmn , m n, let
1 2 n > 0 be the eigenvalues of B = AT A.
Define singular values 1 , . . . , n of A as
p
i = i ,
i = 1, 2, . . . , n.
Define condition number of the rectangular A
r
1
1
=
.
(A) = 2 (A) =
n
n
Note
1
= 2 (A)2
n
So, problems condition number is squared by going through
normal equations algorithm.
2 (B) =
NumCSE, Lecture 9, Oct 16, 2014
8/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
Orthogonal vectors and matrices
Orthogonal vectors: Two vectors u, v Rn are orthogonal if
uT v = 0.
Vector u is orthonormal if kuk2 = 1.
A square matrix Q is orthogonal if its columns are pairwise
orthonormal, i.e.,
QT Q = I .
Hence also Q 1 = Q T .
Important property: for any orthogonal matrix Q and vector x:
kQxk2 = kxk2 .
Hence
kQk2 = kQ 1 k2 = 1.
NumCSE, Lecture 9, Oct 16, 2014
9/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
Least squares through orthogonal transformations
Normal equations can be bad because the stability of the algorithm
depends on the conditioning of the problem squared.
Instead, transform the problem: for any orthogonal matrix P
kb A xk2 = kP b PA xk2 .
(*)
So, find orthogonal matrix that transforms A to upper triangular
form
R
A=Q
.
O
Now use P = Q T in (*). This is a QR decomposition.
I
How is QR decomposition obtained?
What to do with it? (Answer this one first.)
NumCSE, Lecture 9, Oct 16, 2014
10/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Solving least squares given QR decomposition
R
R
T
kb Axk = kb Q
xk = kQ b
xk
O
O
Denote
c
Q b=
d
T
Then
krk2 = kb Axk2 = kc Rxk2 + kdk2 .
1.
2.
3.
4.
Solve Rx = c by back substitution.
Obtain also krk = kdk.
Stable: the condition number of A is not squared.
Matlab implements this method in the backslash operator
for over-determined systems.
NumCSE, Lecture 9, Oct 16, 2014
11/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Example 6.5 from AscherGreif
1 0
A = 1 1
1 2
In Matlab execute
0.1
b = 0.9
2.0
[Q, T] = qr(A)
[m,n] = size(A);
R = T(1:n, 1:n)
Q*b
In the end, we have to solve
1.7321
1.7321 1.7321
c
R
1.3435
0
1.4142 x
min
x
= min
x
x
d
O
0.1225
0
0
NumCSE, Lecture 9, Oct 16, 2014
12/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Economy size QR
R
R
A=Q
= [ Q1 , Q2 ]
= Q1 R.
|{z} |{z}
O
O
n mn
With the pseudo-inverse of A we get
x = A+ b (AT A)1 AT b = (R T Q1T Q1 R)1 R T Q1T b
= (R T R)1 R T Q1T b = R 1 Q1T b
Notice, that R T R is the Cholesky factorization of B: B = R T R.
NumCSE, Lecture 9, Oct 16, 2014
13/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Algorithm: Least squares via QR decomposition
1. QR factorization:
R
(a) A = Q
(full version) or
O
(b) A = Q1 R (economy version)
2. Compute:
c
(a)
= Q T b (full version) or
d
(b) c = Q1T b (economy version)
3. Backward substitution: R x = c
Matlab: [Q,R] = qr(A);
[Q,R] = qr(A,0);
NumCSE, Lecture 9, Oct 16, 2014
(full version)
(economy version)
14/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Comparing runtimes: normal eqs. vs. QR decomp.
I
The flop count of efficient QR decomposition is
2mn2 2n3 /3, roughly double that of normal equations when
m n. The better stability has its price in efficiency!
Lets see the comparative cost using Matlabs backslash.
for n = 300:100:1000
m = 3*n+1; % m = n+1; % or something else
A = randn(m,n); b = randn(m,1);
% solve and find execution times; first, Matlabs way usin
tic; xqr = A\b; tqr(n/100-2) = toc;
% next use normal equations
tic; B = A*A; y = A*b; xne = B\y; tne(n/100-2) = toc;
end
plot(300:100:1000,tqr./tne) % plot ratio tqr/tne
NumCSE, Lecture 9, Oct 16, 2014
15/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Comparing runtimes: normal eqs. vs. QR decomp. (cont.)
The ratios in the figure below are roughly twice as large as flop counts
predict. Of course, we dont exactly know whats in Matlabs backslash...
Ratio of execution times using QR over normal equations. The number of
rows for each n is 3n + 1 for the upper curve and n + 1 for the lower one.
NumCSE, Lecture 9, Oct 16, 2014
16/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Computing the QR decomposition by Givens rotations
Find unitary row transformation rendering certain matrix elements
zero. 2 2 case tells all.
Rotation
a=
cos
,
sin
> 0,
cos sin
cos
QT a =
sin cos
sin
cos( + )
=
sin( + )
(
Choose s.t.
NumCSE, Lecture 9, Oct 16, 2014
+ = 0, or
+ = .
17/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Givens rotation
The orthogonal matrix G (i, j, ) Rn,n rotates coordinates i and j
of some vector. (The other coordinates are not altered.)
q
can be chosen such that xj = 0. Then, xi = xi2 + xj2 . We do
not need to compute angles, since
xi
c = cos = q
,
2
xi + xj2
xj
s = sin = q
.
xi2 + xj2
In practice, one proceeds as in the function on the next slide.
Compute first the bigger of the two values s or c, then the smaller.
NumCSE, Lecture 9, Oct 16, 2014
18/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Givens rotation (cont.)
function [c,s] = givens(a,b)
%GIVENS [c,s] = givens(a,b)
%
Computes c = cos(phi) and s = sin(phi)
%
of Givens - Rotation G such that
%
G[a] = [c -s] [a] = [r]
%
[b]
[s c] [b]
[0]
if b == 0,
c = 1; s = 0;
else
if abs(b) > abs(a),
tau = -a/b; s = 1/sqrt(1 + tau^2); c = s*tau;
else
tau = -b/a; c = 1/sqrt(1 + tau^2); s = c*tau;
end
end
NumCSE, Lecture 9, Oct 16, 2014
19/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
QR decomposition
Theorem (QR decomposition)
For every matrix A Rmn , m > n, with maximal rank n there is
an orthogonal matrix Q Rmm such that
R
A=Q
, with R Rn,n ,
O
with R upper triangular and nonsingular.
Proof.
By successively zeroing elements in A by the right sequence of
Givens rotations we can prove the statement. The sequence is
(1, 2), (1, 3), . . . , (1, m), (2, 3), (2, 4), . . . , (2, m), (3, 4), . . . (n, m).
Later rotations do not destroy previously generated zeros.
NumCSE, Lecture 9, Oct 16, 2014
20/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Numerical example revisited
We use the QR factorization to determine 1 , 2 in
f (x) = 1
x
+ 2 (1 e x ).
1+x
Now we get
R=
1.68492
,
0.0486849
1.32508
0.0
T
c=Q b=
2.25773
,
0.0505901
0.0478834
0.0363745
0.0481185
r = b Ax =
0.0367843
0.0464831
0.0257141
The seeked parameters become 1 = 0.382528 and 2 = 1.03913
which gives 4 correct positions. We lose just 2 decimal digits which
correspond to the condition number of A.
NumCSE, Lecture 9, Oct 16, 2014
21/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Householder reflectors
Householder reflectors are very special elementary matrices:
H = I 2uuT ,
kuk2 = 1.
The vector u is called Householder vector
It is easy to show that
H2 = I ,
Hu = u,
H T = H 1 ,
Hv = v
if v u.
Similarly as elementary Gauss transformations Householder
reflectors can be used to zero selected components of a vector.
NumCSE, Lecture 9, Oct 16, 2014
22/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Householder reflectors (cont.)
Lets try to reflect x to a multiple of e1 .
Note that because H is orthogonal the multiple must be kxk2 .
Hx = x 2u |{z}
uT x = kxk2 e1 .
Therefore,
v := 2u = x kxk2 e1 ,
i.e., we know the direction of u as soon as we have determined the
sign.
In order to avoid cancellation we choose the sign equal to the sign
of x1 :
v = x + sign(x1 )kxk2 e1 .
NumCSE, Lecture 9, Oct 16, 2014
23/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Householder reflectors (cont.)
kvk22 = kx + sign(x1 )kxk2 e1 k22
= xT x + 2 sign(x1 )kxk2 xT e1 + kxk22 eT
1 e1
= 2kxk22 + 2 sign(x1 ) x1 kxk2
= 2kxk2 (kxk2 + |x1 |).
Example
If x = [3, 1, 5, 1]T and v = [9, 1, 5, 1]T , then
27
vvT
1
9
H =I 2 T =
v v
54 45
9
9
53
5
1
45 9
5 1
29 5
5 53
and Hx = [6, 0, 0, 0]T .
NumCSE, Lecture 9, Oct 16, 2014
24/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Matlab function
function [A,p] = house(A)
% function [A,p] = house(A)
% perform QR decomposition using Householder reflections
% Transformations are of the form P_k = I - 2u_k(u_k^T), so
% store effecting vector u_k in p(k) + A(k+1:m,k). Assume m > n.
[m,n]=size(A); p = zeros(1,n);
for k = 1:n
% define u of length = m-k+1
u = A(k:m,k);
u(1) = u(1) + sign(u(1))*norm(u); u = u/norm(u);
% update nonzero part of A by I-2uu^T
A(k:m,k:n) = A(k:m,k:n) - u*(2*(u*A(k:m,k:n)));
% store u
p(k) = u(1); A(k+1:m,k) = u(2:m-k+1);
end
NumCSE, Lecture 9, Oct 16, 2014
25/34
Numerical Methods for Computational Science and Engineering
Least squares through orthogonal transformations
QR decomposition
Matlab function (cont.)
function [x,nr] = lsol(b,AA,p)
% function [x,nr] = lsol(b,AA,p)
%
% Given the output AA and p of house, which contain Q and R of m
% and given a right-hand-side vector b, solve min || b - Ax ||.
% Return also the norm of the residual, nr = ||r|| = ||b - Ax||.
y = b(:); [m,n] = size(AA);
% transform b
for k=1:n
u = [p(k);AA(k+1:m,k)];
y(k:m) = y(k:m) - 2*u *(u*y(k:m));
end
% form upper triangular R and solve
R = triu(AA(1:n,:));
x = R \ y(1:n); nr = norm(y(n+1:m));
NumCSE, Lecture 9, Oct 16, 2014
26/34
Numerical Methods for Computational Science and Engineering
Householder QR factorization
Householder QR factorization
After the application of three Householder
reflectors to a 20 5 matrix.
I
P (1) zeros the elements of the first column
but the first element.
P (2) zeros the elements of the 2nd column
but the first two elements.
etc.
Complexity of an efficient QR factorization:
2mn2 2n3 /3 flops
Similar as for normal equations method if m n
and twice as expensive if m n.
NumCSE, Lecture 9, Oct 16, 2014
27/34
Numerical Methods for Computational Science and Engineering
Householder QR factorization
Householder QR factorization in Matlab
Matlab functions:
[Q,R] = qr(A);
Q Rmm , R Rmn for A Rmn
[Q,R] = qr(A,0); Q Rmn , R Rnn for A Rmn
(economical QR-factorization)
Computational effort of Householder QR factorization of A Rmn
[Q,R] = qr(A);
Costs O(nm2 )
[Q,R] = qr(A,0);
Costs O(n2 m)
NumCSE, Lecture 9, Oct 16, 2014
28/34
Numerical Methods for Computational Science and Engineering
Householder QR factorization
Timings of QR with m = n2
NumCSE, Lecture 9, Oct 16, 2014
29/34
Numerical Methods for Computational Science and Engineering
Householder QR factorization
Solving the linear systems
Note on solving the linear least squares problem
Standard methods for solving the linear least squares problem
include the following:
1. Normal equations: fast, simple, intuitive, but less robust in
ill-conditioned situations.
2. QR decomposition: this is the standard approach
implemented in general-purpose software. It is more
computationally expensive than the normal equations
approach if m n but is more robust.
3. SVD: used mostly when A is rank deficient or nearly rank
deficient (in which case the QR approach may not be
sufficiently robust). The SVD-based approach is very robust
but is significantly more expensive in general and cannot be
adapted to deal efficiently with sparse matrices. See later.
NumCSE, Lecture 9, Oct 16, 2014
30/34
Numerical Methods for Computational Science and Engineering
Householder QR factorization
Solving the linear systems
Solving linear systems Ax = b by the QR factorization
1. A = QR (QR factorization, Householder reflections)
2. z = Q T b (Transformation of b)
3. R x = z (Backward substitution)
Complexity:
3n2
4
4 3
n + O(n2 ) +
+ O(n) + n2 + O(n) = n3 + O(n2 )
3
2
3
Twice the costs of Gaussian elimination.
NumCSE, Lecture 9, Oct 16, 2014
31/34
Numerical Methods for Computational Science and Engineering
Householder QR factorization
Solving the linear systems
Stable solution of Wilkinsons weird linear system
by means of QR-decomposition
res = [];
for n=10:10:1000
A=tril(-ones(n),-1)+eye(n); A(:,end)=ones(n,1);
x=((-1).^(1:n));
b=A*x;
[Q,R]=qr(A);
errlu=norm(A\b-x)/norm(x);
errqr=norm(R\(Q*b)-x)/norm(x);
res=[res; n,errlu,errqr];
end
semilogy(res(:,1),res(:,2),m-*,...
res(:,1),res(:,3),b-+);
NumCSE, Lecture 9, Oct 16, 2014
32/34
Numerical Methods for Computational Science and Engineering
Householder QR factorization
Solving the linear systems
NumCSE, Lecture 9, Oct 16, 2014
33/34
Numerical Methods for Computational Science and Engineering
Householder QR factorization
Solving the linear systems
Tridiagonal systems
Tridiagonal systems can be solved in O(n) flops by Givens
rotations. (Do not generate Q = G1 Gn1 .)
A = full(gallery(tridiag,10,-2,3,-1));
n = size(A,1);
for k=1:n-1,
[c,s] = givens(A(k,k),A(k+1,k));
A(k:k+1,k:n) = [c -s;s c]*A(k:k+1,k:n);
A(k+1,k) = 0;
end
NumCSE, Lecture 9, Oct 16, 2014
34/34
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
I
Learn simple and effective iterative methods for problems
where direct methods are ineffective.
Analyze methods to see when and where they can be applied
and how effective they are.
Understand modern algorithms, specifically preconditioned
conjugate gradients
References
I
I
U. Ascher & C. Greif: Numerical methods. Chapter 7.
Y. Saad: Iterative Methods for Sparse Linear Systems. SIAM,
2003. 2nd ed.
Barrett et al.: Templates for the Solution of Linear Systems.
SIAM, 1994. Online at URL http://www.netlib.org/
linalg/html_templates/Templates.html.
NumCSE, Lecture 10, Oct 20, 2014
2/40
Numerical Methods for Computational Science and Engineering
Comparison of direct and iterative linear solvers
Why not use a direct method?
A x = b,
nn
nonsingular.
Iterative method: Starting from initial guess x0 , generate iterates
x1 , x2 , . . . , xk , . . ., hopefully converging to solution x.
But why not simply use LU decomposition, or
x = A \ b
in Matlab?
The matrix size n must be large, and the matrix A must be
somehow special to consider iterative methods.
NumCSE, Lecture 10, Oct 20, 2014
3/40
Numerical Methods for Computational Science and Engineering
Comparison of direct and iterative linear solvers
Comparison of direct and iterative linear solvers
Direct solvers
+ Computation is numerically stable in many relevant cases.
+ Can solve economically for several right-hand sides.
+ Accuracy can be improved via iterative refinement.
+ Essentially a black box.
But: fill-in limits usefulness (memory, flops).
Typical scenarios
I Not too big problems (2D)
I High accuracy is required, e.g., in shift-and-invert based
eigensolvers, inverse iteration
I Determinants
I Many linear systems with the same matrix A
I Difficult applications (e.g. circuit simulation)
NumCSE, Lecture 10, Oct 20, 2014
4/40
Numerical Methods for Computational Science and Engineering
Comparison of direct and iterative linear solvers
Comparison of direct and iterative linear solvers (cont.)
Iterative solvers
+ Only a rough approximation to x is required.
+ A good x0 approximating x is known (warm start)
+ Matrix often only implicitly needed via matvec product.
Good preconditioner often necessary for convergence.
Quality often dependent on right choice of parameters, e.g.
start vector, basis size, restart (see that later).
Typical scenarios
I Very large problems
I Inexact Newton-Methods
I Many linear systems with slightly changing matrices
I Matrix-free applications (e.g. matvec product via FFT)
NumCSE, Lecture 10, Oct 20, 2014
5/40
Numerical Methods for Computational Science and Engineering
Examples
The ubiquitous Poisson problem
u(x) = f (x) in domain
and boundary conditions
NumCSE, Lecture 10, Oct 20, 2014
Stationary temperature
distribution.
Electrical potential due to charged
distribution (particles).
Gravitational potential due to mass
distribution.
6/40
Numerical Methods for Computational Science and Engineering
Examples
Somewhat similar problems
Linear elasticity problems (forces displacements stresses /
strains)
Maxwell equation (electric / magnetic fields acceleration of
particles)
NumCSE, Lecture 10, Oct 20, 2014
7/40
Numerical Methods for Computational Science and Engineering
Examples
Test problem: Poisson equation on square
Let us consider the Poisson equation on a rectangular domain
= [0, 1]2
2
2
u(x,
y
)
u(x, y ) = f (x, y ),
x 2
y 2
u(x, y ) = p(x, y ) on ,
u(x, y ) =
We define a rectangular grid with grid points that are a distance h
apart. In each grid point the Laplacian of u can be expanded as
u(x, y ) =
=
NumCSE, Lecture 10, Oct 20, 2014
u(x, y ) 2 u(x, y )
2
x
y
1
(4u(x, y ) u(x + h, y ) u(x h, y )
h2
u(x, y + h) u(x, y h)) + O(h2 )
8/40
Numerical Methods for Computational Science and Engineering
Examples
Test problem: Poisson equation on square (cont.)
Let xi = ih, yj = jh, 0 i, j N + 1. Then
u(xi , yj )
1
(4u(xi , yj ) u(xi+1 , yj ) u(xi1 , yj )
h2
u(xi , yj+1 ) u(xi , yj1 ))
for 0 < i, j < N + 1. The discretized equation now becomes
1
(4u(xi , yj ) u(xi+1 , yj ) u(xi1 , yj ) u(xi , yj+1 ) u(xi , yj1 ))
h2
= f (xi , yj ).
or
4u(xi , yj )u(xi+1 , yj )u(xi1 , yj )u(xi , yj+1 )u(xi , yj1 ) = h2 f (xi , yj ).
NumCSE, Lecture 10, Oct 20, 2014
9/40
Numerical Methods for Computational Science and Engineering
Examples
Test problem: Poisson equation on square (cont.)
NumCSE, Lecture 10, Oct 20, 2014
10/40
Numerical Methods for Computational Science and Engineering
Examples
Test problem: Poisson equation on square (cont.)
At the near-boundary points, the prescribed boundary values are
plugged in. If, e.g., i = 1 and 1 j N we have
1
(4u(x1 , yj ) u(x2 , yj ) p(x0 , yj ) u(x1 , yj+1 ) u(x1 , yj1 ))
h2
= f (x1 , yj ).
We arrange the unknowns u(x1 , y1 ), u(x1 , y2 ), . . . , u(xN , yN )
row-wise in a vector u Rn , n = N 2 :
u(xi , yj ) u(i1)N+j ,
1 i, j n.
Similarly for f : f(i1)N+j h2 f (xi , yj ).
Au = b
NumCSE, Lecture 10, Oct 20, 2014
11/40
Numerical Methods for Computational Science and Engineering
Examples
Test problem: Poisson equation on square (cont.)
u1,1
u2,1
..
.
uN,1
u1,2
u = u2,2 ,
..
.
uN,2
u1,3
..
.
uN,N
NumCSE, Lecture 10, Oct 20, 2014
b1,1
b2,1
..
.
bN,1
b1,2
b = b2,2 ,
..
.
bN,2
b1,3
..
.
bi,j = h2 f (xi , yj )
+ possible boundary
terms
bN,N
12/40
Numerical Methods for Computational Science and Engineering
Examples
Structure of a Poisson matrix
Matlab spy of a matrix discretization of the Poisson equation
u = f in = (0, 1)2 with a N N grid (N = 10).
NumCSE, Lecture 10, Oct 20, 2014
13/40
Numerical Methods for Computational Science and Engineering
Examples
Notes on the linear system
I
The system of linear equations A u = b, consists of n = N 2
unknowns somehow organized in the vector u, and as many
values on the right side in vector b.
Weve seen similar 1D problem: u 00 (x) = g (x).
That problem lead to a tridiagonal matrix of size N.
2-dimensional problem: block-tridiagonal matrix
J
I
A=
NumCSE, Lecture 10, Oct 20, 2014
I
J
..
.
I
..
.
I
..
.
J
I
I
J
4
1
J=
1
4 1
..
..
.
.
1
,
.
4 1
1
4
..
14/40
Numerical Methods for Computational Science and Engineering
Examples
Notes on the linear system (cont.)
I
Band width is N. LU factorization costs O(N 4 ) = O(n2 )
flops.
In 2D matrix orders can easily get large:
N = 100 n = 100 000, N = 1000 n = 106 .
In 3 dimensions: n = N 3 . Bandwidth of A is N 2 .
N = 100 n = 106 , N = 1000 n = 109 .
Gaussian elimination with the original matrix ordering costs
O(N 7 ) = O(n2.33 ) flops.
NumCSE, Lecture 10, Oct 20, 2014
15/40
Numerical Methods for Computational Science and Engineering
Examples
Eigenvalues of the Poisson matrix
For any size N the matrix A is diagonally dominant and
nonsingular.
It can be verified directly that the n = N 2 eigenvalues of A are
given by (recall (N + 1)h = 1)
i,j = 4 2(cos(ih) + cos(jh)),
min 2 2 h2 ,
1 i, j N.
max 8.
Thus i,j > 0 for all 1 i, j N, and we see that the matrix A is
also positive definite.
Knowing the eigenvalues explicitly is helpful in understanding
performance, convergence and accuracy issues related to linear
solvers.
NumCSE, Lecture 10, Oct 20, 2014
16/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods
Stationary iterative methods and fixed point iterations
We can rewrite A x = b as a vector equation
f(x) = b A x = 0.
We construct an equivalent equation in fixed point form
g(x) = x,
and define iteration
xk+1 = g(xk ),
k = 0, 1, . . .
starting from an initial guess x0 .
Convergence order will be linear, differences in rate of convergence.
NumCSE, Lecture 10, Oct 20, 2014
17/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods
Matrix splittings
Stationary iterative methods: Theory
Definition (Matrix splitting)
Let A be nonsingular. Then A = M N, with M nonsingular, is
called a matrix splitting.
Then
Ax = b
M x = Nx + b = M x + b A x.
We consider the fixed point iteration
xk+1 = M 1 Nxk + M 1 b
xk+1 = xk + M 1 rk
Note:
I In practice, M 1 should be a good (but cheaper to invert)
approximation to A1 (preconditioner).
NumCSE, Lecture 10, Oct 20, 2014
18/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods
Matrix splittings
Formulation as fixed-point iteration
1
1
xk+1 = M
| {z N} xk + M
| {z b},
=: c
=: T
xk+1 = T xk + c
N = M A.
fixed point iteration
T = M 1 N = I M 1 A is called the iteration matrix.
Theorem.The error satisfies ek+1 = T ek .
Proof: With fixed point x = A1 b, one has x = T x + c.
NumCSE, Lecture 10, Oct 20, 2014
19/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods
Theorem on convergence
Theorem on convergence
Theorem (Convergence theorem for stationary iterations)
Let A = M N be a matrix splitting with M invertible. Then,
(1) The iteration
Mxk+1 = Nxk + b,
(+)
converges for any x0 if and only if
(T ) < 1,
T = M 1 N = I M 1 A.
Here, (T ) = max{||; is an eigenvalue of T } is the
spectral radius of T .
(2) If, for any vector norm, kT k < 1, then iteration (+)
converges.
NumCSE, Lecture 10, Oct 20, 2014
20/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods
Theorem on convergence
Convergence rate
I
Earlier we derived a convergence rate for fixed point iterations
xk+1 = g (xk ) for solving nonlinear systems.
Here, the situation is similar: xk+1 = g (xk ) = T xk + c.
How many iteration steps are required to reduce the error
norm by an order of magnitude?
0.1ke0 k = (T )k ke0 k.
k
1
log10 (T )
1
log10 (T ).
k
The rate is higher (faster convergence), the smaller .
rate =
NumCSE, Lecture 10, Oct 20, 2014
21/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods: practical schemes
Jacobi iteration
Practical schemes: Jacobi and GaussSeidel iteration
Let A = L + D + U where D is diagonal, L is strictly lower
triangular and U is strictly upper triangular. Let E = D + L.
D=diag(diag(A));
7
A = 3
1
E = tril(A)
3
1
10
2
7 15
Jacobi iteration:
0
0
7
10
0 , E = 3
0 15
1
0
10
7
0
0
15
xk+1 = xk + D 1 rk
GaussSeidel iteration:
NumCSE, Lecture 10, Oct 20, 2014
7
D = 0
0
xk+1 = xk + E 1 rk
22/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods: practical schemes
Jacobi iteration
Jacobi example
7x1 = 3 3x2 x3 ,
10x2 = 4 + 3x1 2x3 ,
15x3 = 2 x1 7x2 .
Evaluate right hand side at current iterate k and left hand side as
unknown at step k + 1:
NumCSE, Lecture 10, Oct 20, 2014
(k+1)
x1
(k+1)
x2
(k+1)
x3
1
(k)
(k)
3 3x2 x3
,
7
1
(k)
(k)
=
4 + 3x1 2x3
,
10
1
(k)
(k)
=
2 x1 7x2
.
15
=
23/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods: practical schemes
Jacobi iteration
GaussSeidel example
7x1 = 3 3x2 x3 ,
10x2 3x1 = 4 2x3 ,
15x3 + x1 + 7x2 = 2.
Evaluate right hand side at current iterate k and left hand side as
unknown at step k + 1:
(k+1)
x1
(k+1)
x2
(k+1)
x3
1
(k)
(k)
3 3x2 x3
,
7
1
(k+1)
(k)
=
4 + 3x1
2x3
,
10
1
(k+1)
(k+1)
=
2 x1
7x2
.
15
=
Always use most recent information.
NumCSE, Lecture 10, Oct 20, 2014
24/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods: practical schemes
Jacobi iteration
Convergence of Jacobi for model Poisson problem
I
Recall matrix is defined by
4ui,j u1+1,j ui1,j ui,j+1 ui,j1 = bi,j ,
1 i, j N.
Eigenvalues
l,m = 4 2(cos
I
l
m
+ cos
),
N +1
N +1
1 l, m N.
Consider Jacobi: M = D. Then
NumCSE, Lecture 10, Oct 20, 2014
1
T = I D 1 A = I A.
4
25/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods: practical schemes
Jacobi iteration
Convergence of Jacobi for model Poisson problem (cont.)
I
So eigenvalues of iteration matrix T are
1
1
l
m
l,m = 1 l,m = (cos
+cos
),
4
2
N +1
N +1
Spectral radius
(T ) = cos
1 l, m N.
1 .
N +1
n
Rate of convergence
rate = log (T ) 1/n.
Thus, O(n) iterations are required for error reduction by a
constant factor.
NumCSE, Lecture 10, Oct 20, 2014
26/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods: practical schemes
Jacobi iteration
Terminating the iteration
How should we terminate the iteration, for any of these methods,
given a convergence error tolerance?
I
One criterion, often used in practice, is to require that the
relative residual be small enough, written as
krk k tol kbk.
However, that this implies
kx xk k
krk k
(A) tol.
kxk
kbk
Thus, the tolerance must be taken suitably small if the
condition number of A is large.
NumCSE, Lecture 10, Oct 20, 2014
27/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods: practical schemes
Jacobi iteration
Terminating the iteration (cont.)
I
If we wish to control the algorithm by measuring the
difference between two consecutive iterations, kxk xk1 k,
then care must be taken when the convergence is slow.
For any vector norm and its induced matrix norm:
kx xk k kT kkx xk1 k kT k(kx xk k + kxk xk1 k).
Thus,
kek k = kx xk k
kT k
kxk xk1 k
1 kT k
For slowly converging iterations (like Jacobi or GS)
kT k 1.
Error may be large even if consecutive iterates are close!
NumCSE, Lecture 10, Oct 20, 2014
28/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods: practical schemes
Properties of Jacobi and Gauss-Seidel relaxation
Properties of Jacobi and Gauss-Seidel relaxation
I
Jacobi more easily parallelized
Jacobi matrix M is symmetric
GS converges whenever Jacobi converges and often (but not
always) twice as fast.
Both methods are simple but slow. Used as building blocks for
faster, more complex methods.
NumCSE, Lecture 10, Oct 20, 2014
29/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods: practical schemes
Successive Over-Relaxation (SOR)
Over-relaxation and under-relaxation
I
There are more sophisticated stationary methods than Jacobi
and GS. The methods introduced below are based on a simple
modification of the ancient ones.
Let xk+1 be obtained from xk by Jacobi or GS.
Modify it further by
xk+1 xk+1 + (1 )xk
where is a parameter.
NumCSE, Lecture 10, Oct 20, 2014
30/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods: practical schemes
Successive Over-Relaxation (SOR)
Over-relaxation and under-relaxation (cont.)
I
Two useful variants:
I
Based on Gauss-Seidel (GS) and 1 < < 2, obtain faster
successive over-relaxation (SOR).
xk+1 = xk + [(1 )D + E ]1 rk xk + M1 rk .
Based on Jacobi and 0.8, obtain slower under-relaxation
which is a good smoother in some applications.
xk+1 = xk + D 1 rk .
This iteration is also caller damped Jacobi.
NumCSE, Lecture 10, Oct 20, 2014
31/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods: practical schemes
Successive Over-Relaxation (SOR)
Optimal parameter for SOR
1. = 1: back to GaussSeidel.
2. For many matrices there is a known optimal parameter for
SOR: opt .
3. For a large class of matrices, the optimal parameter is given by
opt =
2
q
> 1,
1 + 1 2J
with J the spectral radius of the Jacobi iteration matrix.
NumCSE, Lecture 10, Oct 20, 2014
32/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods: practical schemes
Successive Over-Relaxation (SOR)
Optimal parameter for SOR (cont.)
4. For matrices of the form tridiag[-1, 2, -1], the above formula
holds, and opt can be simplified to the following formula:
opt =
2
,
1 + sin n+1
where n is the dimension of the matrix.
5. For this class of matrices (T (opt )) = opt 1.
6. Poisson matrix: gallery(poisson,N) is n n matrix with
n = N2
2
2
opt =
=
.
1 + sin n+1
1 + sin N+1
7. Once the optimal parameter is used, convergence is much
faster than with Jacobi or GaussSeidel.
NumCSE, Lecture 10, Oct 20, 2014
33/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods: practical schemes
Successive Over-Relaxation (SOR)
Symmetric Successive Over-Relaxation (SSOR)
Combination of standard forward and backward SOR:
1
M x(k+ 2 ) = N x(k) + b,
e x(k+1) = N
e x(k+ 12 ) + b,
M
with backward SOR
e = 1 D + U,
M
e = 1 D L.
N
Note:
I
Only used when A symmetric, i.e. U = LT .
Symmetric GaussSeidel: SSOR( = 1).
NumCSE, Lecture 10, Oct 20, 2014
34/40
Numerical Methods for Computational Science and Engineering
Stationary iterative methods: practical schemes
Block iterations
Practical schemes: Block iterations
Let
A11
A21
A= .
..
A12
A22
..
.
Am1 Am2
A1m
A2m
..
.
Amm
Block Jacobi iteration
M = diag(A11 , A22 , . . . , Amm )
Block GaussSeidel iteration
NumCSE, Lecture 10, Oct 20, 2014
A11
A21
M= .
..
A22
..
.
Am1 Am2
..
Amm
35/40
Numerical Methods for Computational Science and Engineering
Numerical experiments
Poisson equation on square 11 11 - grid
N=11; A=gallery(poisson,N);
b = A*[1:N 2 ]; tol = 1e-6, x = zeros(N 2 ,1)
Solver
Jacobi
Block Jacobi
GaussSeidel
Block GaussSeidel
SGS (A s.p.d.)
Block SGS
SOR( = 1.6)
Block SOR( = 1.5)
MATLAB
M = D = diag(diag(A))
M= DB = triu(tril(A,1),-1)
M=tril(A)
M=tril(A,1)
M1 =tril(A)/sqrt(D); M2 = M1T
M1 =tril(A,1)/chol(DB ); M2 = M1T
D/omega + tril(A,-1)
triu(tril(A,1),-1)/omega + tril(A,-m)
nit
341
176
174
90
90
48
32
24
Note: The generation of M for the blocked versions is specific to this
problem!
NumCSE, Lecture 10, Oct 20, 2014
36/40
Numerical Methods for Computational Science and Engineering
Numerical experiments
Poisson equation on slightly larger grids
solver
Jacobi
Block Jacobi
GaussSeidel
Block GaussSeidel
SSOR ( = 1.8)
Block SSOR ( = 1.8)
n = 312
2157
1093
1085
547
85
61
n = 632
7787
3943
3905
1959
238
132
Table 1 : Iteration steps for solving the Poisson equation on a 31-by-31
and on a 63-by-63 grid with an relative residual accuracy of 106 .
NumCSE, Lecture 10, Oct 20, 2014
37/40
Numerical Methods for Computational Science and Engineering
Numerical experiments
Relaxation methods for the model Poisson problem
I
Jacobi requires O(n) iterations hence O(n2 ) flops
Gauss-Seidel is twice as fast, so same O
SOR with optimal requires O(N) iterations hence O(n3/2 )
flops!
Lower bound: the best possible method would require at least
O(n) flops.
So there seems to be room for improvement: are there better
methods?
NumCSE, Lecture 10, Oct 20, 2014
38/40
Numerical Methods for Computational Science and Engineering
Iterative refinement
Iterative refinement/improvement
Let Ax = b be solved via GEPP, PA = LU.
We wish to improve the accuracy of the computed solution
x.
If we execute
r = b A
x
Solve L y = Pr
Solve U z = y
x0 =
x+z
then, in exact arithmetic,
A
x0 = A
x + A z = (b r) + r = b.
Attention: the residual r must be computed with higher precision!
Otherwise there may be no improvement.
Heuristic: In computation with d decimal digits and (A) = 10
d digits can be gained per step. (At most d correct digits.)
NumCSE, Lecture 10, Oct 20, 2014
39/40
Numerical Methods for Computational Science and Engineering
Iterative refinement
Iterative refinement/improvement (cont.)
% Ex. 2.7 from Schwarz-K"ockler
A = [0.29412, 0.41176, 0.52941,
0.42857, 0.57143, 0.71429,
0.36842, 0.52632, 0.42105,
0.38462, 0.53846, 0.46154,
b = [0.17642, 0.21431, 0.15792,
on iterative improvement
0.58824;
0.64286;
0.36842;
0.38462]
0.15380]
x = single(A)\single(b);
r = b - A*x
r = b - A*double(x)
% solve A*x=b in single precision
% residual in single precision
% residual in double precision
X = A\b;
R = b - A*X
% solve A*x=b in double precision
% residual in double precision
x-X, r-R
z = single(A)\r;
x = x + z
% correct single prec. x using double
% prec. residual
NumCSE, Lecture 10, Oct 20, 2014
40/40
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
I
Modern iterative methods for symmetric positive definite
linear systems
I
I
Preconditioning SPD systems
I
Steepest descent method
The conjugate gradient method
(Preconditioned) conjugate gradient (PCG) method
Modern iterative methods for nonsymmetric linear systems
I
I
Krylov space methods
The Generalized Minimal Residual (GMRES) algorithm
References
I
U. Ascher & C. Greif: Numerical methods. Chapter 7.
Y. Saad: Iterative Methods for Sparse Linear Systems. SIAM,
2nd edition, 2003.
NumCSE, Lecture 11, Oct 23, 2014
2/47
Numerical Methods for Computational Science and Engineering
Introduction
Nonstationary methods
I
I
I
I
I
I
None of the stationary methods we have seen is very fast even
for the model Poisson problem when n is really large.
Another, general disadvantage of all these relaxation methods
is that they require an explicitly given matrix A: what if A is
only given implicitly, through matrix-vector products?
Try to take advantage of accumulating knowledge as the
iteration proceeds.
Consider M = Mk , and more generally xk+1 = xk + k pk .
The scalar k > 0 is a step size; pk is a search direction.
Restrict consideration to cases where A is symmetric positive
definite (SPD).
Then solving A x = b is equivalent to
1
min (x) := xT Ax xT b + , any R.
x
2
NumCSE, Lecture 11, Oct 23, 2014
3/47
Numerical Methods for Computational Science and Engineering
Energy norm minimization
Energy norm minimization
Definition: For Ax = b, with A SPD, we define the functional (with
some R)
1
(x) := xT Ax xT b + .
2
I is continuously differentiable, with
(x) = Ax b = r(x).
I
If := 21 bT A1 b (our choice), then
1
1
(x) = (Ax b)T A1 (Ax b) = kr(x)k2A1
2
2
1
= (x A1 b)T A(x A1 b)
2
1
1
= (x x )T A(x x ) = ke(x)k2A .
2
2
NumCSE, Lecture 11, Oct 23, 2014
4/47
Numerical Methods for Computational Science and Engineering
Energy norm minimization
Energy norm minimization (cont.)
I
kxkA =
xT Ax is called the energy norm.
Our goal is to minimize the energy norm of the error, kekA .
I
has a unique minimum satisfying r(x ) = 0 x = A1 b.
(In overdetermined systems there is in general no x for which
r(x) = 0.)
Notice that our choice of makes the minimum of zero.
(If we set = 0 then (x ) = 21 bT A1 b.)
NumCSE, Lecture 11, Oct 23, 2014
5/47
Numerical Methods for Computational Science and Engineering
Steepest descent
Descent directions
Definition: Suppose that for a functional and vectors x, d there
is a 0 such that
(x + d) < (x),
0 < 0 .
Then, d is a descent direction for at x.
For our quadratic functional we have
(x + d) = (x) r(x)T d +
2 T
d Ad.
2
Therefore, d is a descent direction if it has a positive component in
direction of (x) = r(x).
The descent is steepest if d is aligned with r.
NumCSE, Lecture 11, Oct 23, 2014
6/47
Numerical Methods for Computational Science and Engineering
Steepest descent
Local steepest descent minimization of
Let xk be an approximation of x . We want to improve xk in the
steepest descent direction.
xk+1 = xk k (xk ) = xk + k rk
We choose k such that (xk+1 ) is minimized (local line search).
From
2 T
(xk + rk ) = (xk ) rkT rk +
r Ark .
2 k
it follows that
rT rk
d(xk + rk ) !
= 0 k = Tk
.
d
rk Ark
Remark: k changes in each iteration method not stationary
NumCSE, Lecture 11, Oct 23, 2014
7/47
Numerical Methods for Computational Science and Engineering
Steepest descent
The method
Algorithm: Steepest Descent Algorithm
Choose x0 , .
Set r0 = b Ax0 and s0 = Ar0 .
for k := 0, . . . , maxit do
k := rkT rk /rkT sk .
xk+1 := xk + k rk .
rk+1 := rk k sk .
if krk+1 k < kr0 k exit.
Compute sk+1 = Ark+1 .
end for
This is very memory efficient, but how fast is the convergence?
NumCSE, Lecture 11, Oct 23, 2014
8/47
Numerical Methods for Computational Science and Engineering
Steepest descent
Convergence
Convergence of steepest descent
Theorem: For the errors ek = xk x in steepest descent we have
kek+1 kA
max min
kek kA .
max + min
Here, min , max are smallest/largest eigenvalue of A.
From the theorem we have
(A) 1
kek+1 kA
kek kA
(A) + 1
(A) 1
(A) + 1
k+1
ke0 kA .
We always have convergence; but very slow if (A) 1.
Steepest descent is only a local minimization of .
NumCSE, Lecture 11, Oct 23, 2014
9/47
Numerical Methods for Computational Science and Engineering
Steepest descent
Convergence
Slow convergence of steepest descent method
NumCSE, Lecture 11, Oct 23, 2014
10/47
Numerical Methods for Computational Science and Engineering
Steepest descent
Convergence
Iteration steps
How many iteration steps k = k() are needed such that
kep kA ke0 kA ?
We write
1
1 + 1/(A)
1 1/(A)
k
Using log[(1 + s)/(1 s)] = 2(s + s 3 /3 + ) 2s we get
1
k() (A) log(1/).
2
For = 0.1: k(0.1) 21 (A) log(10) 1.15 (A).
2D Poisson matrix: (A) = O(n). No improvement over GS.
NumCSE, Lecture 11, Oct 23, 2014
11/47
Numerical Methods for Computational Science and Engineering
Steepest descent
Convergence
Convergence of steepest descent
SPD matrices R22 :
1.9412 0.2353
A1 =
,
0.2353
1.0588
Eigenvalues: (A1 ) = {1, 2},
A2 =
7.5353 1.8588
1.8588
0.5647
(A2 ) = {0.1, 8}
Notation: spectrum of a matrix Rnn
(A) := { C; is eigenvalue of A}
NumCSE, Lecture 11, Oct 23, 2014
12/47
Numerical Methods for Computational Science and Engineering
Steepest descent
Convergence
Convergence of steepest descent (cont.)
Iterates with steepest descent for
A1 .
NumCSE, Lecture 11, Oct 23, 2014
Iterates with steepest descent for
A2 .
13/47
Numerical Methods for Computational Science and Engineering
Steepest descent
Preconditioning
Preconditioning steepest descent
Let the SPD M be a preconditioner for A with Cholesky
factorization M = LLT
Standard: Instead of solving Ax = b, we apply steepest descent to
the SPD problem
L1 ALT y = L1 b,
LT x = y.
Note:
I
L1 ALT is never formed! Applying L1 ALT means:
(1) Solve with LT ; (2) multiply with A; (3) solve with L.
L1 ALT and M 1 A are similar (have equal eigenvalues):
LT (L1 ALT )LT = M 1 A.
NumCSE, Lecture 11, Oct 23, 2014
14/47
Numerical Methods for Computational Science and Engineering
Steepest descent
Preconditioning
Preconditioning steepest descent (cont.)
Clever: replace all Euclidean inner products by M-inner product
hx, yiM := xT My and solve system
M 1 Ax = M 1 b.
Note: M 1 A is SPD with respect to the M-inner product
hx, yiM := hx, Myi.
hx, M 1 AyiM = xT M(M 1 Ay) = xT Ay
= xT AM 1 My = (M 1 Ax)T My
= hM 1 Ax, yiM
NumCSE, Lecture 11, Oct 23, 2014
15/47
Numerical Methods for Computational Science and Engineering
Steepest descent
Preconditioning
Preconditioning steepest descent (cont.)
So, we can apply the steepest descent algorithm to
M 1 A x = M 1 b
and replace the ordinary Euklidian inner product with the M-inner
product.
We want to retain the notation rk = b A xk for the residual that
we actually care for. We define additionally the preconditioned
residual by
zk = M 1 rk = M 1 (b A xk ).
NumCSE, Lecture 11, Oct 23, 2014
16/47
Numerical Methods for Computational Science and Engineering
Steepest descent
Preconditioning
Preconditioning steepest descent (cont.)
So, in the steepest descent algorithm we simply have to replace
I
A by M 1 A,
rk by zk = M 1 rk , and
Euklidian inner products by M-inner products.
So,
k =
zT
zT
zT
k Mzk
k rk
k Mzk
=
=
TAz
Ts
1 A z )
zT
M(M
z
z
k
k
k
k
k k
where Mzk = rk and sk = Azk .
Note, that rk+1 = rk k A rk translates into
M 1 rk+1 = M 1 rk k M 1 A M 1 rk
| {z }
| {z } | {z }
zk
zk
zk+1
NumCSE, Lecture 11, Oct 23, 2014
rk+1 = rk k sk .
17/47
Numerical Methods for Computational Science and Engineering
Steepest descent
Preconditioning
Algorithm: Preconditioned steepest descent algorithm
Choose x0 , .
Set r0 = b A x0 . Solve M z0 = r0 . Set s0 = A z0 .
for k := 0, . . . , maxit do
T
k := zT
k rk /zk sk .
xk+1 := xk + k zk .
rk+1 := rk k sk .
if krk+1 k < kr0 k return.
Solve M zk+1 = rk+1 .
Compute sk+1 = A zk+1 .
end do
Per iteration step we have to multiply a vector by A and solve a
linear system with M.
NumCSE, Lecture 11, Oct 23, 2014
18/47
Numerical Methods for Computational Science and Engineering
Steepest descent
Preconditioning
Convergence of preconditioned steepest descent
We can adapt the convergence theorem to obtain
(M 1 A) 1
kek+1 kA
kek kA .
(M 1 A) + 1
(Notice that a direct translation of norms gives
1
2
eT
Aek ) = eT
k M(M
k Aek = kek kA .)
The preconditioner M again has the function to reduce the
condition of the original system matrix A.
NumCSE, Lecture 11, Oct 23, 2014
19/47
Numerical Methods for Computational Science and Engineering
Steepest descent
Preconditioning
Convergence of preconditioned steepest descent
SPD matrices R22 :
1.9412 0.2353
A1 =
,
0.2353
1.0588
(A1 ) = {1, 2},
(A1 ) = 2,
7.5353 1.8588
1.8588
0.5647
A2 =
(A2 ) = {0.1, 8},
(A2 ) = 80.
Jacobi: M = D 2 = diag(diag(A2 ))
1/2
(D2
1/2
A2 D2
1/2
) = {0.0989, 1.9011}, (D2
1/2
A2 D2
) = 19.22
1/2
SGS: M = M 2 M2T with M2 = tril(A2 )/D2
(M21 A2 M2T ) = {0.1880, 1.0000}, (M21 A2 M2T ) = 5.3190
NumCSE, Lecture 11, Oct 23, 2014
20/47
Numerical Methods for Computational Science and Engineering
The conjugate gradient (cg) method
The method
The conjugate gradient (cg) method
Method of steepest descent: pk = (xk ) = rk .
Slow convergence if (x) = const form long narrow ellipsoids (i.e.
if (A) 1).
The steepest descent algorithm is a greedy algorithm. It solves the
problem of minimizing locally.
The conjugate gradient (cg) method is a descent method with a
global view : In each iteration step we look for the minimum of
along the line xk + pk where pk is a very cleverly chosen search
direction:
p 0 = r0 ;
pk = rk + k1 pk1 , k > 0.
pk is a descent direction independent of the choice of k1 .
NumCSE, Lecture 11, Oct 23, 2014
21/47
Numerical Methods for Computational Science and Engineering
The conjugate gradient (cg) method
The method
Some formulae for cg:
xk+1 = xk + k pk
(xk+1 )=min
k =
pT
k rk
T
pk Apk
rk+1 = rk k Apk
Search directions must be conjugate:
T
pT
k1 Apk = pk1 A(rk +k1 pk1 ) = 0
k1 =
rkT Apk1
pT
k1 Apk1
Clever manipulations show that
k =
rkT rk
k
= T
,
T
pk Apk
pk Apk
NumCSE, Lecture 11, Oct 23, 2014
k =
k+1
,
k
k = krk k2 .
22/47
Numerical Methods for Computational Science and Engineering
The conjugate gradient (cg) method
Convergence
Finite termination property
rkT rj = pT
k Apj = 0,
Theorem
k 6= j.
(1)
Proof. By induction.
Theorem
In the k-th step of the cg method, xk minimizes (x) in
x0 + Kk (A; r0 ) where the Krylov space Kk (A; r0 ) is defined by
Kk (A; r0 ) = span(r0 , Ar0 , A2 r0 , . . . , Ak1 r0 ).
and r0 = b Ax0 is the initial residual.
In general, dim(Kk (A; r0 )) = k.
NumCSE, Lecture 11, Oct 23, 2014
23/47
Numerical Methods for Computational Science and Engineering
The conjugate gradient (cg) method
Convergence
Finite termination property (cont.)
Theorem
The cg method applied to a SPD n-by-n matrix A finds the
solution after at most n iteration steps.
Proof. Direct consequence of (1).
Note
We do not consider CG as an algorithm that terminates after a
finite number of steps (as Gaussian elimination) but as an iterative
method.
NumCSE, Lecture 11, Oct 23, 2014
24/47
Numerical Methods for Computational Science and Engineering
The conjugate gradient (cg) method
Convergence
Finite termination property (cont.)
Picture: Martin Gutknecht
NumCSE, Lecture 11, Oct 23, 2014
25/47
Numerical Methods for Computational Science and Engineering
The conjugate gradient (cg) method
Convergence
Convergence of CG
Convergence rate:
kek kA 2
!k
p
(A) 1
p
ke0 kA .
(A) + 1
(2)
The number of iterations p() to reduce the error kx xk kA by a
factor is
1p
p()
(A) log(2/).
2
This is in general a huge reduction in comparison with steepest
descent.
p
2D Poisson matrix: (A) = O(N).
NumCSE, Lecture 11, Oct 23, 2014
26/47
Numerical Methods for Computational Science and Engineering
The conjugate gradient (cg) method
Preconditioning CG
Preconditioning CG with SPD M
As with steepest descent we apply the conjugate gradient
algorithm to the SPD problem
M 1 Ax = M 1 b
and replace the ordinary Euklidian inner product with the M-inner
product.
Again we retain the notation rk = b Axk for the residual and
denote the preconditioned residual by zk = M 1 rk .
NumCSE, Lecture 11, Oct 23, 2014
27/47
Numerical Methods for Computational Science and Engineering
The conjugate gradient (cg) method
Preconditioning CG
Algorithm PCG
Choose x0 . Set r0 = b Ax0 . Solve Mz0 = r0 . 0 = zT
0 r0 .
Set p0 = z0 .
for k = 0, 1, . . ., maxit do
sk = Apk .
k = k /pT
k sk .
xk+1 = xk + k pk .
rk+1 = rk k sk .
Solve Mzk+1 = rk+1 .
(Solve with preconditioner)
k+1 = zT
k+1 rk+1 .
if k+1 < 2 0 exit.
(Test for convergence)
k = k+1 /k .
pk+1 = zk+1 + k pk .
endfor
There is one new statement in the algorithm in which the
preconditioned residual zk is computed. Matlab function pcg.
NumCSE, Lecture 11, Oct 23, 2014
28/47
Numerical Methods for Computational Science and Engineering
The conjugate gradient (cg) method
Preconditioning CG
PCG tests
Problem: Poisson equation on square [revisited]
PCG with preconditioner as indicated
preconditioner
Jacobi
block Jacobi
symmetric Gauss-Seidel
symmetric block Gauss-Seidel
SSOR ( = 1.8)
block SSOR ( = 1.8)
n = 312
76
57
33
22
18
15
n = 632
149
110
58
39
26
21
Table 1 : Iteration steps for solving the Poisson equation on a 31-by-31
and on a 63-by-63 grid with a relative residual accuracy of 106 .
NumCSE, Lecture 11, Oct 23, 2014
29/47
Numerical Methods for Computational Science and Engineering
Incomplete factorization preconditioners
Incomplete factorization preconditioners
I
M = A would be the ideal preconditioner.
However, direct solve needs a factorization of A,
LU = A,
LLT = A,
that introduces fill-in.
I
Want to get as close to A as we can without allowing too
much fill-in.
Accept only some fill-in based on some rule.
ILU(0) / IC(0): do not accept fill-in at all.
L and U have same nonzero structure as A.
Matlab: L=ichol(A); [L,U]=ilu(A);
NumCSE, Lecture 11, Oct 23, 2014
30/47
Numerical Methods for Computational Science and Engineering
Incomplete factorization preconditioners
Numerical example
Exec times (it steps)
m = 31
m = 101
NumCSE, Lecture 11, Oct 23, 2014
Jacobi
0.45 (76)
18.0 (234)
Block Jacobi
1.23 (57)
54.1 (166)
Sym. GS
0.34 (33)
10.1 (84)
ICCG(0)
0.28 (28)
8.8 (73)
31/47
Numerical Methods for Computational Science and Engineering
Incomplete factorization preconditioners
Dynamic nonzero patterns
I
ILU(0)/IC(0) are very restricted.
Better to use a dynamic approach.
Execute LU factorization. Drop small elements if they are
smaller than some drop tolerance .
Possibly additional parameter p to limit number of nonzeros
per column of L or row of U.
ICT( ), ILUT( ), ILUTP(p, ).
For details see book by Saad.
Remark: In general it is advisable to apply RCM or AMD
reordering before the factorization.
NumCSE, Lecture 11, Oct 23, 2014
32/47
Numerical Methods for Computational Science and Engineering
Incomplete factorization preconditioners
Example for dynamic fill-in strategies
NumCSE, Lecture 11, Oct 23, 2014
33/47
Numerical Methods for Computational Science and Engineering
Incomplete factorization preconditioners
Nonsymmetric linear systems of equations
We want to solve
Ax = b
with A 6= AT
(3)
We could solve
AT Ax = AT b
by the (preconditioned) conjugate gradient method.
Two potential issues:
1. Each iteration step requires multiplication with A and AT .
2. Condition number (AAT ) = (AT A) = 2 (A).
Consensus:
Only when A is well conditioned and matvec with AT is cheap.
NumCSE, Lecture 11, Oct 23, 2014
34/47
Numerical Methods for Computational Science and Engineering
Krylov spaces
Krylov spaces
In steepest descent and CG we have
rk+1 = rk k Apk ,
pk = rk + k1 pk1 ,
where rk = b Axk is residual and pk is search direction.
One can show that
rk = r0 +
k
X
cj Aj r0
rk = pk (A)r0 ,
pk (0) = 1,
j=1
Moreover,
Pk
j
j=1 cj A rj
= rk r0 = A(xk x0 ). So,
xk = x0
k1
X
cj+1 Aj r0
()
j=0
NumCSE, Lecture 11, Oct 23, 2014
35/47
Numerical Methods for Computational Science and Engineering
Krylov spaces
Krylov spaces (cont.)
Definition
For given A, the m-th Krylov space generated by the vector r is
given by
Km = Km (A, r) := span r, Ar, A2 r, . . . , Am1 r
= {p(A)r | p Pm1 } .
So, from (), we have for all k,
xk x0 + Kk (A, r0 )
NumCSE, Lecture 11, Oct 23, 2014
36/47
Numerical Methods for Computational Science and Engineering
Krylov space solvers
Krylov space solvers
Krylov space solvers search for approximate solutions of Ax = b in
Krylov spaces, by
1. constructing an orthogonal basis for the Krylov subspace;
2. defining an optimality property;
3. using an effective preconditioner.
NumCSE, Lecture 11, Oct 23, 2014
37/47
Numerical Methods for Computational Science and Engineering
Krylov space solvers
Orthogonal basis
Arnoldi & Lanczos algorithms
Task: For j = 1, 2, . . . , m, compute orthonormal bases {v1 , . . . , vj }
for the Krylov spaces
Kj = span r0 , Ar0 , A2 r0 , . . . , Aj1 r0 .
The algorithms that do this are
I
Lanczos algorithm for A symmetric/Hermitian.
Arnoldi algorithm for A nonsymmetric.
Difficulty: Because of ill-conditioning, do not want to explicitly
form r0 , Ar0 , . . . , Aj r0 .
NumCSE, Lecture 11, Oct 23, 2014
38/47
Numerical Methods for Computational Science and Engineering
Krylov space solvers
Orthogonal basis
Arnoldi & Lanczos algorithms (cont.)
Instead of using Aj r0 we proceed with Avj .
(Notice that Avi Ki+1 Kj for all i < j.)
Orthogonalize Avj against v1 , . . . , vj by GramSchmidt:
wj = Avj
j
X
vi hij .
i=1
wj points in the desired new direction (unless it is 0). Therefore,
vj+1 = wj /kwj k2 = wj /hj+1,j ,
hj+1,j := kwj k2 .
Thus,
Avj =
j+1
X
vi hij .
i=1
NumCSE, Lecture 11, Oct 23, 2014
39/47
Numerical Methods for Computational Science and Engineering
Krylov space solvers
Orthogonal basis
Arnoldi: GramSchmidt applied to Km (A, r0 )
v1 = r0 /kr0 k2
for j=1 to k do
wj = Avj ;
for i=1 to j do
Orthogonalize A vj against v1 , . . . , vj :
hi,j = wjT vi ; (Modified Gram-Schmidt)
wj = wj hi,j vi ;
end for
hj+1,j = kwj k2 . (Stop if hj+1,j is zero.)
vj+1 = wj /hj+1,j ;
end for
NumCSE, Lecture 11, Oct 23, 2014
40/47
Numerical Methods for Computational Science and Engineering
Krylov space solvers
Orthogonal basis
Arnoldi relation
Define Vk := [v1 , . . . , vk ]. Then we get the Arnoldi relation
AVk = Vk+1 Hk+1,k .
with
Hk+1,k
NumCSE, Lecture 11, Oct 23, 2014
h11
h21
h12
h22
h3,2
..
.
h1,k
h2,k
h3,k
..
.
R(k+1)k
hk+1,k
41/47
Numerical Methods for Computational Science and Engineering
Krylov space solvers
Orthogonal basis
Arnoldi relation (cont.)
Notice that
Hk,k = VkT AVk .
If A is symmetric Hk,k is tridiagonal!
Avj = vj1 hj1,j + vj hj + vj+1 hj+1,j .
In each step of the Lanczos algorithm, orthogonality has to be
enforced only against three vectors.
NumCSE, Lecture 11, Oct 23, 2014
42/47
Numerical Methods for Computational Science and Engineering
Krylov space solvers
Optimality property: minimal residual
GMRES
The GMRES (Generalized Minimal Residual) algorithm computes
xm Km that leads to the smallest residual exploiting the Arnoldi
relation.
Goal: Cheaply minimize
krm k2 = kb Axm k2 ,
xm x0 + Km (A, r0 ),
using the Arnoldi relation
m
AV m = Vm+1 H
and R(Vm ) = Km (A, r0 ).
Note: Other ways are possible to extract xm , e.g. VmT rm = 0.
NumCSE, Lecture 11, Oct 23, 2014
43/47
Numerical Methods for Computational Science and Engineering
Krylov space solvers
Optimality property: minimal residual
A Hessenberg Least-Squares problem
Arnoldi basis expansion xm = x0 + Vm y gives with := kr0 k2 :
min krm k2 = min kb A(x0 + Vm y)k2
= min kr0 AVm yk2
m yk2
= min kr0 Vm+1 H
m y k2
= min kVm+1 e1 H
m yk2 .
= min ke1 H
I
I
Hessenberg least squares problem of dimension (m + 1) m.
m can be cheaply computed using
QR factorization of H
Givens rotations.
NumCSE, Lecture 11, Oct 23, 2014
44/47
Numerical Methods for Computational Science and Engineering
Krylov space solvers
Preconditioning
Preconditioning
I
For a general matrix A, a preconditioner is often more crucial
than P in PCG for SPD
Incomplete LU (ILU), with drop tolerance (ILUT)
Example: convection-diffusion equation
NumCSE, Lecture 11, Oct 23, 2014
u(x, y ) +
u
u
+
= g (x, y ).
x
y
45/47
Numerical Methods for Computational Science and Engineering
Krylov space solvers
Preconditioning
Preconditioning (cont.)
Restarted GMRES, m = 20, for matrix of size 10000 corresponding
to convection-diffusion equation on 100 100 uniform mesh.
NumCSE, Lecture 11, Oct 23, 2014
46/47
Numerical Methods for Computational Science and Engineering
Krylov space solvers
Preconditioning
Practical issues
I
The GMRES algorithm with m steps needs memory space for
m + O(1) vectors plus matrices A and M.
(CG needs just 4)
Slow convergence may entail expensive memory requirements.
Recipe: GMRES(m). Execute m steps of GMRES. Then
restart with xm as initial approximation of the next GMRES
cycle. Choice of m can be crucial.
GMRES does not care about symmetries. There is no gain in
choosing a symmetric preconditioner M when solving
M 1 A x = M 1 b.
I
I
ILU factorizations are popular preconditioners.
Remark: MINRES = GMRES for symm. indefinite matrices.
Preconditioner must be SPD!
NumCSE, Lecture 11, Oct 23, 2014
47/47
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
1. Eigenvalue problems, definitions
2. Power method and variants
3. Googles pagerank
References
I
U. Ascher & C. Greif: Numerical methods. SIAM 2011.
Chapters 4.1 & 8.1
C. Moler: Numerical Computing with Matlab. The
MathWorks. 2004. (Stuff on pagerank, including code)
W. Dahmen & A. Reusken: Numerik f
ur Ingenieure und
Naturwissenschaftler. 2. Aufl. Springer, 2008.
NumCSE, Lecture 12, Oct 27, 2014
2/34
Numerical Methods for Computational Science and Engineering
Basics
Review of basic concepts
I
Let A Rnn . A scalar and a vector x 6= 0 are an
eigenvalue-eigenvector pair (or eigenpair) if
A x = x.
The set (A) of all eigenvalues form the spectrum of A.
The nullspace N (I A) is called eigenspace of .
dim(N (I A)) = geometric algebraic multiplicity of .
Ax = x
(A + I )x = ( + )x
Ax = x
Ak x = k x
Similarity transformation: Given a nonsingular matrix S, the
matrix S 1 AS has the same eigenvalues as A.
(What about the eigenvectors?)
NumCSE, Lecture 12, Oct 27, 2014
3/34
Numerical Methods for Computational Science and Engineering
Basics
Review of basic concepts (cont.)
I
Spectral decomposition: For a diagonalizable n n real matrix
A there are n (generally complex-valued) eigenpairs (j , xj ),
with X = [x1 , x2 , . . . , xn ] nonsingular.
Then,
AX = X A = X X 1
with = diag(1 , 2 , . . . , n ).
Any vector can be written as y = X =
NumCSE, Lecture 12, Oct 27, 2014
Ay =
n
X
j=1
j Axj =
n
X
xj j and
j j xj .
j=1
4/34
Numerical Methods for Computational Science and Engineering
Basics
Review of basic concepts (cont.)
I
Does every real n n matrix have n eigenvalues?
We can write the eigenvalue problem as a homogeneous linear
system
(I A)x = 0.
Since we want a nontrivial x, this means that I A must be
singular.
Therefore we can find by forming the characteristic
polynomial and finding its roots.
That is, in principle we solve
det(I A) = 0,
which has n in general complex, not necessarily distinct roots.
NumCSE, Lecture 12, Oct 27, 2014
5/34
Numerical Methods for Computational Science and Engineering
Basics
Numerical example
I
Example: The matrix
0.9880
1.8000 0.8793 0.5977 0.7819
1.9417 0.5835 0.1846 0.7250
1.0422
A = 0.6003 0.0287 0.5446 2.0667 0.3961
0.8222
1.4453
1.3369 0.6069
0.8043
0.4187 0.2939
1.4814 0.2119 1.2771
has eigenvalues given approximately by 1 = 2,
2 = 1 + 2.5, 3 = 1 2.5, 4 = 2, and 5 = 2.
It is known that closed form formulas for the roots of a
polynomial do not generally exist if the polynomial is of
degree 5 or higher. Thus we cannot expect to be able to solve
the eigenvalue problem in a finite procedure.
NumCSE, Lecture 12, Oct 27, 2014
6/34
Numerical Methods for Computational Science and Engineering
Basics
Numerical example (cont.)
Eigenvalues in C. For real matrices, the complex eigenvalues
come in pairs. If is an eigenvalue, then so is .
NumCSE, Lecture 12, Oct 27, 2014
7/34
Numerical Methods for Computational Science and Engineering
Example: electric circuit
Example: electric circuit
Nodal analysis of linear electric circuit (composed of resistors,
inductors, capacitors) in frequency domain (at angular frequency
> 0)
C
U
L
R
~
~
Here: three unknown nodal potentials.
NumCSE, Lecture 12, Oct 27, 2014
8/34
Numerical Methods for Computational Science and Engineering
Example: electric circuit
Example: electric circuit (cont.)
The corresponding system of equations is given by ( = 1)
1
1
L
0
C + L
1
2
1
A = L
C + R1 + L
L
1
1
0
L
C + L
0 0 0
L1
0
C 0 0
L
1 1
2
1
= 0 R1 0 + 0 C 0 +
L
L L
1
1
0 0 C
0 0 0
0 L
L
or
A() := W + C
NumCSE, Lecture 12, Oct 27, 2014
S,
W , C , S Rnn symmetric
9/34
Numerical Methods for Computational Science and Engineering
Example: electric circuit
Example: electric circuit (cont.)
Apply rescirc.m:
Scan u = A()1 e1
for 0 < < 2.
Plot of |ui (U())|,
i = 1, 2, 3 for
R=L=C =1
(scaled model)
Blow-up of some
nodal potentials for
certain !
NumCSE, Lecture 12, Oct 27, 2014
10/34
Numerical Methods for Computational Science and Engineering
Example: electric circuit
Example: electric circuit (cont.)
I
If circuit is operated at real resonant frequency: circuit
equations will not have a solution.
A real-world circuit will always behave in a well-defined way,
but the linear model will break down due to extremely large
currents and voltages.
In an experiment this breakdown manifests itself as a rather
explosive meltdown of circuits components. Hence, it is vital
to determine resonant frequencies of circuits in order to avoid
their destruction.
Another somewhat similar experiment:
Collapse of Tacoma narrows bridge on November 7, 1940,
see http://en.wikipedia.org/wiki/Tacoma_Narrows_
Bridge_(1940)
NumCSE, Lecture 12, Oct 27, 2014
11/34
Numerical Methods for Computational Science and Engineering
Example: electric circuit
Example: electric circuit (cont.)
It is important to be able to simulate such problems.
Find C \ {0} such that W + C +
1
S is singular
This is a quadratic eigenvalue problem: find x 6= 0, C \ {0},
A()x = (W + C +
Substitution: y =
1
x
1
S)x = 0
x = y:
W S
x
C 0
x
=
I 0
y
0
I
y
| {z } |{z}
|
{z
}
z
M
B
NumCSE, Lecture 12, Oct 27, 2014
12/34
Numerical Methods for Computational Science and Engineering
Example: electric circuit
Example: electric circuit (cont.)
The quadratic eigenvalue problem is linearized to become a
generalized linear eigenvalue problem:
Find C, z C2n \ {0} such that
Mz = Bz.
In this example one is mainly interested in the eigenvalues ,
whereas the eigenvectors z usually need not be computed.
If B is nonsingular we can write the generalized eigenvalue problem
as
A z = z,
with A = B 1 M.
NumCSE, Lecture 12, Oct 27, 2014
13/34
Numerical Methods for Computational Science and Engineering
Example: electric circuit
Example: electric circuit (cont.)
Resonant frequencies
for circuit (including
decaying modes with
Im() > 0)
NumCSE, Lecture 12, Oct 27, 2014
14/34
Numerical Methods for Computational Science and Engineering
Example: homogeneous linear ODEs
Another example: homogeneous linear ODEs
Analytic solution of autonomous homogeneous linear ordinary
differential equations (ODE):
y = Ay,
If A is diagonalizable, i.e.,
..
A=S
.
A Cnn
if there is a nonsingluar matrix S s.t.
1
S ,
n
{z
}
D
y = Ay, z = S 1 y
S Cnn nonsingular
NumCSE, Lecture 12, Oct 27, 2014
z = Dz
15/34
Numerical Methods for Computational Science and Engineering
Example: homogeneous linear ODEs
Another example: homogeneous linear ODEs (cont.)
Solution of initial value problem:
y = Ay,
y(0) = y0 Cn
z = Dz,
z(0) = z0 = S 1 y0
The initial value problem for the decoupled homogeneous linear
ODE z = Dz has a simple analytic solution:
zi (t) = exp(i t)z0,i = exp(i t) S 1 y0 i
In order to find the transformation matrix S all non-zero solution
vectors (= eigenvectors) x Cn of the linear eigenvalue problem
Ax = x
have to be found.
NumCSE, Lecture 12, Oct 27, 2014
16/34
Numerical Methods for Computational Science and Engineering
Power method
Vector iteration/power method
Now we assume that A is diagonalizable:
A = V V 1 A V = V ,
A vi = i vi ,
and that = diag(1 , . . . , n ) with
|1 | > |2 | |n |.
Starting with an (almost) arbitrary initial vector x0 Rn we form
the vector sequence {xk }
k=0 by defining
xk := Axk1 ,
k = 1, 2, . . .
Clearly,
xk := Ak x0 .
NumCSE, Lecture 12, Oct 27, 2014
17/34
Numerical Methods for Computational Science and Engineering
Power method
Convergence
Let xj = V yj , j 0. Then,
yk = V 1 xk = V 1 Axk1 = V 1 V V 1 xk1 = yk1 .
From this we have
k
yk = y0 =
k
= diag(1,
k1
k
y0 .
2
n
, . . . , )k diag(1, 0, 0, . . . , 0).
k
1
1
Therefore,
1
y
k1 k k
NumCSE, Lecture 12, Oct 27, 2014
(0)
y1 e1 ,
(0)
k 1k yk y1 e1 k = O 2 .
1
1
18/34
Numerical Methods for Computational Science and Engineering
Power method
Convergence (cont.)
From xk = V yk it follows that as k :
xk points in the direction of v1 = V e1 , i.e., into the direction of the
eigenvector coresponding to the largest eigenvalue (in magnitude).
What about the eigenvalue?
Good choice: Rayleigh quotient
(x) =
xT A x
xT x
If x is an eigenvector then (x) is the associated eigenvalue.
Otherwise the Rayleigh quotient of x gives the closest possible
approximation to the eigenvalue in the least squares sense.
NumCSE, Lecture 12, Oct 27, 2014
19/34
Numerical Methods for Computational Science and Engineering
Power method
Algorithm: Power method
Input matrix A and initial guess v0 .
for k := 1, 2, . . . until termination do
v = A vk1
vk =
v/k
vk
(k)
T
1 = vk Avk
end for
The convergence criterion may be, e.g., the angle between vk and
vk1 :
T
T
sin (vk , vk1 ) = kvk vk1 (vk1
vk )k = k(I vk1 vk1
)vk k.
NumCSE, Lecture 12, Oct 27, 2014
20/34
Numerical Methods for Computational Science and Engineering
Power method
Example
A = diag(1, 2, . . . , 30, 31, 32) R3232
B = diag(1, 2, . . . , 30, 30, 32) R3232
Absolute error:
(k)
(k)
|1 1 | |1 32|.
NumCSE, Lecture 12, Oct 27, 2014
31
32
0.968
30
32
0.938
21/34
Numerical Methods for Computational Science and Engineering
Googles page rank
Googles page rank
I
One of the reasons why Google is such an effective search
engine is the PageRank that determines the importance of a
web page.
PageRank is determined entirely by the link structure of the
World Wide Web.
Then, for any particular query, Google finds the pages on the
Web that match that query and lists those pages in the order
of their PageRank.
Lets imagine a surfer going from page to page by randomly
choosing an outgoing link from one page to get to the next.
NumCSE, Lecture 12, Oct 27, 2014
22/34
Numerical Methods for Computational Science and Engineering
Googles page rank
Googles page rank (cont.)
To escape dead ends, a random page of the web is chosen.
To avoid cycles, at a fraction of time, a random page of the
web is chosen.
This theoretical random walk is known as a Markov chain or
Markov process.
NumCSE, Lecture 12, Oct 27, 2014
23/34
Numerical Methods for Computational Science and Engineering
Googles page rank
Googles page rank (cont.)
I
Let W be the set of (reachable) web pages and let n = |W |.
Connectivity matrix G Rnn :
(
1 there is a hyperlink j 7 i,
gij =
0 otherwise.
nnz(G) = number of hyperlinks in W .
Let ri and cj be the row and column sums of G :
X
X
ri =
gij ,
cj =
gij .
j
= rj = in-degree, cj = out-degree of the jth page.
(cj = 0 is a dead end)
NumCSE, Lecture 12, Oct 27, 2014
24/34
Numerical Methods for Computational Science and Engineering
Googles page rank
Googles page rank (cont.)
0
1
0
G =
0
0
1
NumCSE, Lecture 12, Oct 27, 2014
3
0
0
1
1
0
0
4
0
0
0
1
1
1
5
1
0
0
0
0
0
6
0
0
0
0
0
0
1
0
0
0
25/34
Numerical Methods for Computational Science and Engineering
Googles page rank
Googles page rank (cont.)
I
Let A be the matrix with elements
(
gij /cj if cj 6= 0
aij =
1/n
if cj = 0 (dead end).
1
2
0
A=
0
0
1
2
I
0
0
1
2
1
2
0
0
0 1
0 0
0 0
1
3 0
1
3 0
1
3 0
1
6
1
6
1
6
1
6
1
6
1
6
1
0
0
0
Let e = (1, 1, . . . , 1)T . Then AT e = e (or eT A = eT ).
So, 1 (AT ) = (A).
NumCSE, Lecture 12, Oct 27, 2014
26/34
Numerical Methods for Computational Science and Engineering
Googles page rank
Googles page rank (cont.)
I
I
I
To be able to escape cycles or strong components we follows
the links only with a probability .
With probability 1 we choose a random page.
We replace A by the matrix
= A + (1 )peT ,
A
I
I
I
where p is a personalization vectors. (p has nonnegative
elements that sum to 1, kpk1 = 1.
We may choose p = e/n.
= eT
Note that eT A
Most of the elements of A are very small. If n = 4 109 and
= 0.85, then the probability of jumping from one page to
another without following a link is = 3.75 1011 .
NumCSE, Lecture 12, Oct 27, 2014
27/34
Numerical Methods for Computational Science and Engineering
Googles page rank
Googles page rank (cont.)
The PerronFrobenius theorem applies. It states that a nonzero
solution of the equation
x = Ax
exists and is unique to within a scaling factor. If this scaling factor
is chosen so that
n
X
xi = 1
i=1
then x is the state vector of the Markov chain and is Googles
PageRank. The elements of x are all positive and less than one.
This vector x is the eigenvector corresponding to the largest
It can be determined by vector iteration.
eigenvalue of A.
NumCSE, Lecture 12, Oct 27, 2014
28/34
Numerical Methods for Computational Science and Engineering
Googles page rank
Matlab code
function [x,cnt] = pagerankpow(G)
% PAGERANKPOW PageRank by power method with no matrix operations.
% x = pagerankpow(G) is the PageRank of the graph G.
% [x,cnt] = pagerankpow(G) also counts the number of iterations.
% There are no matrix operations. Only the link structure
% of G is used with the power method.
% Link structure
[n,n] = size(G);
for j = 1:n
L{j} = find(G(:,j)); % set of links coming into node j
c(j) = length(L{j}); % in-degree
end
% Power method
p = .85; delta = (1-p)/n;
NumCSE, Lecture 12, Oct 27, 2014
29/34
Numerical Methods for Computational Science and Engineering
Googles page rank
Matlab code (cont.)
x = ones(n,1)/n; z = zeros(n,1);
cnt = 0;
while max(abs(x-z)) > .0001
z = x;
x = zeros(n,1);
for j = 1:n
if c(j) == 0
x = x + z(j)/n;
else
x(L{j}) = x(L{j}) + z(j)/c(j);
end
end
x = p*x + delta;
cnt = cnt+1;
end
NumCSE, Lecture 12, Oct 27, 2014
30/34
Numerical Methods for Computational Science and Engineering
Googles page rank
Shift and invert
I
Convergence of simple vector iteration is potentially very slow.
The quotient of the second largest to the largest eigenvalue
can be very close to 1.
Since a polynomial in A has the same eigenvectors as A we
could try to improve the situation by computing an
eigenvector of p(A) for some polynomial p.
In general not successful if the wanted eigenvalue is very close
to the unwanted.
Instead of a polynomial we transform the matrix by the
rational function f () = 1/( ) where is a so-called shift
that is chosen close to the desired eigenvalue.
NumCSE, Lecture 12, Oct 27, 2014
31/34
Numerical Methods for Computational Science and Engineering
Googles page rank
Shift and invert (cont.)
Simple vector iteration with the matrix (A I )1 is referred to as
inverse vector iteration or shift and invert:
xk := (A I )1 xk1
(A I ) xk := xk1 ,
k = 1, 2, . . .
Convergence rate:
1
2
1
1
1
.
2
Where 1 (2 ) is eigenvalue closest (second closest) to .
Of course, we compute the LU factorization of A I before the
iteration. Then, in every iterations step we just have to do
forward/backward substitutions.
NumCSE, Lecture 12, Oct 27, 2014
32/34
Numerical Methods for Computational Science and Engineering
Googles page rank
Algorithm: Inverse iteration
Input matrix A, initial guess v0 , and shift .
Compute LU / Cholesky factorization of A I .
for k := 1, 2, . . . until termination do
Solve (A I )
v = vk1 using the known factorization
vk =
v/k
vk
(k) = vkT Avk
end for
Rayleigh quotient iteration: Modify shift in every iteration
k = (k) . Requires factorization of a matrix in each iteration
step. In general too expensive.
NumCSE, Lecture 12, Oct 27, 2014
33/34
Numerical Methods for Computational Science and Engineering
Googles page rank
Experiment
A = diag(1, 2, . . . , 30, 31, 32) R3232
Shift = 33, 35
Rayleigh quotient
iteration:
First shift = 33, after
two steps RQ shift.
Cubic convergence if
A is symmetric.
NumCSE, Lecture 12, Oct 27, 2014
34/34
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
1. QR algorithm and Matlabs eig
2. Krylov space methods and Matlabs eigs
3. Singular value decomposition
References
I
U. Ascher & C. Greif: Numerical methods. SIAM 2011.
Chapter 8
C. Moler: Numerical Computing with Matlab. The
MathWorks. 2004.
Golub & van Loan, Matrix Computations, 3rd ed., Johns
Hopkins University Press. 1996.
NumCSE, Lecture 13, Oct 30, 2014
2/34
Numerical Methods for Computational Science and Engineering
Computing multiple eigenpairs
Computing multiple eigenpairs
Lets assume that we have computed an eigenpair (1 , x1 ) of A,
Ax1 = 1 x1 , e.g., by the power method.
We set u1 = x1 . (We assume that kx1 k = 1.) Then,
Au1 = 1 u1 .
(1)
How shall we get another eigenpair (2 , x2 )? Of course, we do not
want to get another copy of (1 , x1 )!
We can compute an eigenpair (2 , u2 ) of
(I u1 uT
1 )Ax = x.
Then
Au2 u1 (uT
Au ) = 2 u2 .
| 1 {z 2}
t12
NumCSE, Lecture 13, Oct 30, 2014
(2)
3/34
Numerical Methods for Computational Science and Engineering
Computing multiple eigenpairs
Computing multiple eigenpairs (cont.)
Combining equations (1) and (2) we get
1 t12
A[u1 , u2 ] = [u1 , u2 ]
.
0 2
This is a partial Schur decomposition. The vectors u1 , u2 span an
invariant subspace under A. (U invariant A U U.)
It is easy to see that
1 t12
T
[u1 , u2 ] A[u1 , u2 ] =
.
0 2
Notice that (2 , u2 ) is not an eigenpair of A unless t12 = 0.
How do we construct an eigenvector x2 corresponding to the
eigenvalue 2 of A?
NumCSE, Lecture 13, Oct 30, 2014
4/34
Numerical Methods for Computational Science and Engineering
Schur decomposition
Theorem (Schur decomposition)
If A Cnn then there is a unitary matrix U Cnn such that
U T AU = T
(o)
is upper triangular. The diagonal elements of T are the
eigenvalues of A.
The Schur decomposition is not unique. The eigenvalues on the
diagonal can be moved around. This changes U.
The first column of U is an eigenvector of A but in general not the
others.
Matlab function schur.
NumCSE, Lecture 13, Oct 30, 2014
5/34
Numerical Methods for Computational Science and Engineering
QR algorithm
The QR algorithm
The QR algorithm (not to be confused with the QR factorization)
is an extremely clever algorithm to compute the Schur
decomposition of an arbitrary matrix A.
At the origin of the algorithm is the observation (mostly due to
Rutishauser) that the sequence {Aj }
j=0 defined by
A0 = A,
Qj Rj = Aj ,
(QR factorization)
Aj+1 = Rj Qj ,
converges to an upper triangular matrix. Since the Aj are similar
the diagonal elements of this upper triangular matrix are the
eigenvalues of A0 .
NumCSE, Lecture 13, Oct 30, 2014
6/34
Numerical Methods for Computational Science and Engineering
QR algorithm
The QR algorithm (cont.)
D = diag([4 3 2 1]);
rand(seed,0);
format short e
S=rand(4); S = (S - .5)*2;
A = S*D/S
% A_0 = A = S*D*S^{-1}
for i=1:20,
[Q,R] = qr(A); A = R*Q
end
NumCSE, Lecture 13, Oct 30, 2014
7/34
Numerical Methods for Computational Science and Engineering
QR algorithm
The QR algorithm (cont.)
A( 0) =
[ -4.4529e-01
[ -6.3941e+00
[ 3.6842e+00
[ 3.1209e+00
4.9063e+00
1.3354e+01
-6.6617e+00
-5.2052e+00
-8.7871e-01
1.6668e+00
-6.0021e-02
-1.4130e+00
6.3036e+00]
1.1945e+01]
-7.0043e+00]
-2.8484e+00]
A(20) =
[ 4.0000e+00
[ -3.9562e-04
[ 5.7679e-07
[ -1.0563e-12
2.0896e-02
2.9998e+00
1.8846e-04
-2.6442e-10
-7.0425e+00
8.7266e-01
2.0002e+00
-1.4682e-06
2.1898e+01]
3.2019e+00]
-3.6424e+00]
1.0000e+00]
A(20)./A(19) =
NumCSE, Lecture 13, Oct 30, 2014
[ 1.0000
[ 0.7495
[ 0.5000
[ -0.2500
0.9752
1.0000
0.6668
-0.3334
1.0000
0.9988
1.0000
-0.4999
-1.0000]
-1.0008]
-1.0001]
1.0000]
8/34
Numerical Methods for Computational Science and Engineering
QR algorithm
The QR algorithm (cont.)
The algorithm has two defects:
1. Each step costs O(n3 ) flops.
Remedy: first transform to Hessenberg form
T AQ
and start QR algorithm with H0
A0 H0 = Q
Each step costs O(n2 ) flops.
2. Slow convergence. Remedy: Spectral shifts.
Qj Rj = Aj I ,
Aj+1 = Rj Qj + I ,
Still Aj+1 Aj .
Right choice of shifts gives quadratic convergence to
individual eigenvalues one by one.
3. Deflation. As soon as one eigenvalue has converged, deflate
and continue computation with smaller submatrix.
NumCSE, Lecture 13, Oct 30, 2014
9/34
Numerical Methods for Computational Science and Engineering
QR algorithm
Matlabs functions schur and eig
N=[10:10:400];
T=[]; T1=[];
for n=10:10:400,
A = rand(n);
tic,[U,L]=schur(A); t=toc; T=[T;t];
tic,[U,L]=eig(A); t=toc; T1=[T1;t];
end
plot(N,[T,T1,(T(end)/N(end)^3)*N.^3,],linewidth,2)
legend(schur,eig,n^3)
NumCSE, Lecture 13, Oct 30, 2014
10/34
Numerical Methods for Computational Science and Engineering
QR algorithm
How to get an eigenvector from the Schur decomposition
The eigenvalues of a (upper) triangular matrix T are located on its
diagonal.
Let i = tii . Then T i I is singular.
The i-th diagonal element of T i I vanishes. Therefore,
t11 i
t12
t1i
t1n
x1
0
t
x
22
2n
i
2i
2 0
.
.
.
.
.
. . ..
..
.. ..
0
tin
1 0
. .
.
.
..
..
.. ..
tnn i
0
0
can be solved easily provided that tkk 6= i for k < i.
NumCSE, Lecture 13, Oct 30, 2014
11/34
Numerical Methods for Computational Science and Engineering
Krylov space methods
Krylov space methods
Popular eigensolvers are based on Krylov spaces
Km = Km (A, x) := span x, Ax, A2 x, . . . , Am1 x .
The Arnoldi/Lanczos algorithms compute orthonormal bases of the
Krylov spaces K1 , K2 , . . ., providing the Arnoldi / Lanczos relations
AV m = Vm Hm + wm eT
m,
(Hm Hessenberg)
or
AV m = Vm Tm + wm eT
m,
NumCSE, Lecture 13, Oct 30, 2014
(A symmetric, Tm tridiagonal)
12/34
Numerical Methods for Computational Science and Engineering
Krylov space methods
Krylov space methods (cont.)
We can consider the eigenvalues of Hm = VmT AV m /
Tm = VmT AV m approximations of the eigenvalues of A.
We compute the eigenvalues of Tm , m = 1, . . . , 10 for
A = diag(1, 2, 3, . . . , 80) R8080
with random initial vector.
A typical observation:
The extremal eigenvalues are approximated well very quickly.
NumCSE, Lecture 13, Oct 30, 2014
13/34
Numerical Methods for Computational Science and Engineering
Krylov space methods
Krylov space methods (cont.)
NumCSE, Lecture 13, Oct 30, 2014
14/34
Numerical Methods for Computational Science and Engineering
Krylov space methods
Computational issues of Arnoldi/Lanczos eigensolvers
There are two main issues:
1. Dimension of the Krylov space is getting too big
2. Speed of convergence
Solutions:
1. Restart with good approxiate eigenvectors (Ritz vectors).
This can be one by an analysis of the matrix Hm /Tm .
2. The eigenvalue approximations (Ritz values) converge fastest
to extremal eigenvalues: A shift-and-invert makes the
eigenvalues close to a target/shift the largest.
(A I )1 x = x,
1
.
3. Solving this equation accurately is another issue.
NumCSE, Lecture 13, Oct 30, 2014
15/34
Numerical Methods for Computational Science and Engineering
Krylov space methods
eigs
I
eigs is the Matlab function for computing some of the
eigenvalues of a large sparse matrix.
eigs is the Matlab interface to the eigensolver package
ARPACK:
R. Lehoucq, D. Sorensen, C. Yang: ARPACK Users Guide,
SIAM, Philadelphia, PA, 1998.
See help eigs.
NumCSE, Lecture 13, Oct 30, 2014
16/34
Numerical Methods for Computational Science and Engineering
Singular value decomposition
Singular value decomposition
Theorem (Singular value decomposition (SVD))
For any matrix A Rmn , m n, there are unitary U Rmm ,
V Rnn and a (generalized) diagonal matrix = (1 , . . . , n ),
1 2 r > 0, r +1 = = m = 0, such that
A = UV T .
NumCSE, Lecture 13, Oct 30, 2014
17/34
Numerical Methods for Computational Science and Engineering
Singular value decomposition
Singular value decomposition (cont.)
The decomposition
A = UV T .
is called singular value decomposition (SVD). There is an economic
variant:
NumCSE, Lecture 13, Oct 30, 2014
18/34
Numerical Methods for Computational Science and Engineering
Singular value decomposition
Some uses of the SVD
Let A Rnn nonsingular and A = UV T its SVD.
I
A nonsingular implies n > 0.
(A Rmn , m > n, rank deficient n = 0.)
Euclidean norm (2-norm) of a matrix
kAk = max kAxk = max kUV T xk = max kV T xk
kxk=1
kxk=1
kxk=1
= max kyk = ke1 k = 1
kyk=1
Frobenius norm of a matrix
kAk2F
n
X
i2 .
i=1
NumCSE, Lecture 13, Oct 30, 2014
19/34
Numerical Methods for Computational Science and Engineering
Singular value decomposition
Some uses of the SVD (cont.)
I
Euclidean norm of the inverse (if it exists)
kA1 k = max kA1 xk = max kV 1 U T xk = = 1/n
kxk=1
kxk=1
Condition number of a matrix:
(A) = 2 (A) = 1 (A)/n (A).
Nullspace of a matrix:
The columns of V corresponding to zero singular values span
N (A).
Range of a matrix:
The columns of U corresponding to positive singular values
span R(A).
NumCSE, Lecture 13, Oct 30, 2014
20/34
Numerical Methods for Computational Science and Engineering
Singular value decomposition
Some uses of the SVD (cont.)
Matlab-functions for computing the SVD:
s = svd(A)
: computes singular values of matrix A
[U,S,V] = svd(A)
: computes singular value decomposition
[U,S,V] = svd(A,0) : economical singular value decomposition
for m > n: U Rmn , Rnn , V Rnn
s = svds(A,k)
:
k largest singular values of sparse A
[U,S,V] = svds(A,k): partial singular value decomposition: U
Rmk , V Rnk , Rk,k diagonal with
k largest singular values of A.
NumCSE, Lecture 13, Oct 30, 2014
21/34
Numerical Methods for Computational Science and Engineering
Singular value decomposition
Matlab code for computing nullspace
function V = kerncomp(A,tol)
% computes an orthonormal basis of nullspace(A) using SVD.
% kernel selection with relative tolerance tol
if (nargin < 2), tol = eps; end
[U,S,V] = svd(A); % Singular Value Decomposition
s = diag(S);
% Extract vector of singular values
% find singular values of relative size < tol*sigma(1)
V = V(:,find(s < tol*s(1))); % rightmost columns of V.
NumCSE, Lecture 13, Oct 30, 2014
22/34
Numerical Methods for Computational Science and Engineering
Singular value decomposition
Computation of the SVD [sketch]
I
We compute orthogonal matrices U1 Rmm and V1 Rnn
such that
.. ..
B is bidiagonal.
U1T AV1 = B =
,
.
.
Then we (implicitely) apply the QR algorithm to the
symmetric tridiagonal matrix B T B.
Tridiagonal matrices are never formed!
For details see, e.g., Golub & van Loan, Matrix Computations.
NumCSE, Lecture 13, Oct 30, 2014
23/34
Numerical Methods for Computational Science and Engineering
Principal component analysis
Principal component analysis
Theorem (Best low rank approximation)
Let A = UV T be the SVD of A Rnm . For
1 k r = rank(A) set Uk = [u1 , . . . , uk ] Rnk ,
Vk = [v1 , . . . , vk ] Rmk , k = diag(1 , . . . , k ) Rk,k , and
Ak Uk k VkT =
k
X
i u i v T .
i=1
Then for kk = kk2 and kk = kkF we have
k+1 = kA Ak k =
NumCSE, Lecture 13, Oct 30, 2014
min
F Rnm
rank(F ) = k
kA F k,
24/34
Numerical Methods for Computational Science and Engineering
Principal component analysis
Compressing image information
I
Image composed of m n pixels (greyscale, BMP format).
Matrix A Rmn , 0 aij 255.
Compression scheme using SVD: Replace A by Ak . k =?
Represent Ak by storing the first k columns of U and V .
Instead of mn numbers only store k(m + n) numbers.
NumCSE, Lecture 13, Oct 30, 2014
25/34
Numerical Methods for Computational Science and Engineering
Principal component analysis
Compressing image information (cont.)
close all
colormap(gray)
load clown.mat;
figure(1)
image(X);
[U,S,V] = svd(X);
figure(2)
r = 20;
colormap(gray)
image(U(:,1:r)*S(1:r,1:r)*V(:,1:r))
NumCSE, Lecture 13, Oct 30, 2014
26/34
Numerical Methods for Computational Science and Engineering
Solving almost singular linear systems
Solving almost singular linear systems
I
Let A Rnn . If (A) is very large, then solving Ax = b can
be an ill-conditioned problem.
With the SVD of A we get
x = V 1 U T b.
If the problem is too ill-conditioned to be of use, then people
often seek to regularize it. This means, replace the given
problem intelligently by a nearby problem which is better
conditioned.
Using SVD this can be done by setting the singular values
below a cutoff tolerance to 0, and minimizing the 2-norm of
the solution to the resulting underdetermined problem.
NumCSE, Lecture 13, Oct 30, 2014
27/34
Numerical Methods for Computational Science and Engineering
Solving almost singular linear systems
Solving almost singular linear systems (cont.)
I
We proceed as follows:
1. Starting from n go backward until r is found such that 1 /r
is tolerable in size. This is the condition number of the
problem that we actually solve.
2. Calculate z = U T b; in fact just the first r components of z are
needed. In other words, if ui = Uei is the ith column vector of
U, then zi = uT
i b, i = 1, . . . , r .
3. Calculate yi = zi /i , i = 1, . . . , r , and set yi = 0, r < i n.
4. Calculate x = V y. This really involves only the first r columns
of V and the first r components of y. P
In other words, if vi is
r
the ith column vector of V , then x = i=1 yi viT
Note: In general, the resulting x does not satisfy Ax = b. But its the
best one can do under certain circumstances, and it produces a
solution x of the smallest norm for a sufficiently
well-conditioned approximate problem.
NumCSE, Lecture 13, Oct 30, 2014
28/34
Numerical Methods for Computational Science and Engineering
Solving almost singular linear systems
Example
We reconsider the problem of solving
1 1
2
A=
,
b=
.
3 3
6
Here, A is singular, but b R(A). Matlab yields the SVD
0.316227766016838 0.948683298050514
U=
0.948683298050514
0.316227766016838
0.707106781186547
0.707106781186548
V =
0.707106781186548 0.707106781186547
4.47213595499958
0
=
0
4.01876204512712e 16
NumCSE, Lecture 13, Oct 30, 2014
29/34
Numerical Methods for Computational Science and Engineering
Solving almost singular linear systems
Example (cont.)
We determine that A has rank r = 1. So,
z1 = 6.32455532033676 y1 = 1.4142135623731 x =
1
.
1
Recall that all solutions for this problem have the form
x = (1 + , 1 )T . Since
k
xk22 = (1 + )2 + (1 )2 = 2(1 + 2 ),
we see that our SVD procedure has stably found the solution of
minimum 2-norm for this singular problem.
NumCSE, Lecture 13, Oct 30, 2014
30/34
Numerical Methods for Computational Science and Engineering
Coordinate metrology
Another use of the SVD
Let A Rnn nonsingular and A = UV T its SVD. Then,
min kAxk = min kUV T xk
kxk=1
kxk=1
= min kV T xk
kxk=1
= min kyk,
kyk=1
y = V T x,
= ken k = n
NumCSE, Lecture 13, Oct 30, 2014
31/34
Numerical Methods for Computational Science and Engineering
Coordinate metrology
Example of coordinate metrology
Points measured on two orthogonal lines
g1 :
c1 + n1 x + n2 y = 0,
n12 + n22 = 1,
g2 :
c2 n2 x + n1 y = 0,
n12 + n22 = 1,
NumCSE, Lecture 13, Oct 30, 2014
32/34
Numerical Methods for Computational Science and Engineering
Coordinate metrology
Example of coordinate metrology (cont.)
1
1
..
.
1
c
A
=
0
n
..
.
0
0
..
.
x1
x2
..
.
y1
y2
..
.
0
1
1
..
.
xm1
ym1 +1
ym1 +2
..
.
ym1
xm1 +1
xm1 +2
..
.
0 1 ym1 +m2
xm1 +m2
0
0
..
.
c1
c2 0
n1 0
0
n2
..
together with constraint knk2 = 1.
(Without constraint the solution would be c1 = c2 = n1 = n2 = 0.)
NumCSE, Lecture 13, Oct 30, 2014
33/34
Numerical Methods for Computational Science and Engineering
Coordinate metrology
Example of coordinate metrology (cont.)
(1) Compute QR factorization of
r11 r12 r13
r22 r23
r33
A: A = QR.
c1
r14
c2
r24
0
r34 n1
n2
r44
(2) Determine n with knk2 = 1, s.t.
r33 r34
n1
= min
n2
r44
n is right singular vector associated with smallest singular value.
(3) Determine c by back-substitution.
NumCSE, Lecture 13, Oct 30, 2014
34/34
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
I
To devise and assess algorithms for (continuous)
unconstrained optimization, a problem setting which arises
frequently in applications.
To solve systems of nonlinear equations, thus extending the
methods introduced for nonlinear scalar equations and
systems of linear equations
I
I
Newtons method and relatives
Descent methods
References
I
U. Ascher & C. Greif: Numerical methods. SIAM 2011.
Chapter 9.19.2.
NumCSE, Lecture 14, Nov 3, 2014
2/27
Numerical Methods for Computational Science and Engineering
Introduction
Introduction: Optimization
I
I
Optimization of high importance to scientists and engineers.
Find the shortest path from A to B; distance, travel time.
Single / multiple objectives.
Find the path from A to B with smallest distance among
those that take less than, e.g., 2 hours.
Contrained optimization.
Find the shape of a car that has the least drag.
Requires a maybe large computation (PDE) for each shape.
Find the cheapest car that runs fastest.
Multiple objectives.
NumCSE, Lecture 14, Nov 3, 2014
3/27
Numerical Methods for Computational Science and Engineering
Introduction
Introduction: Optimization (cont.)
I
Simplest case: optimization problem consists of minimizing
(or maximizing) a real function:
min (x)
x
with x = (x1 , x2 , . . . , xn )T Rn .
(Constrained optimization: x in a true subset of Rn .)
I
Problems where some energy is minimized (bending of beam)
(x) = xT Ax bT x + c.
Approximation in the least squares sense
(x) = kAx bk.
NumCSE, Lecture 14, Nov 3, 2014
4/27
Numerical Methods for Computational Science and Engineering
Introduction
Introduction: Optimization (cont.)
I
Necessary condition for (unconstrained) minimum:
1
x
x
2
grad =
.. = 0.
.
xn
n (nonlinear) equations in n unknowns.
First we look at the more general problem
f(x) = 0
where f1 (x), . . . , fn (x) are n scalar nonlinear equations.
Somewhat similar to scalar (n = 1) nonlinear problem f (x) = 0.
NumCSE, Lecture 14, Nov 3, 2014
5/27
Numerical Methods for Computational Science and Engineering
Introduction
Systems of nonlinear equations
Equipped with knowledge on how to solve scalar nonlinear
equations as well as linear systems, it is time to combine them and
consider systems of nonlinear equations
f1 (x1 , x2 , . . . , xn ) = 0,
f2 (x1 , x2 , . . . , xn ) = 0,
..
.
f(x) = 0
fn (x1 , x2 , . . . , xn ) = 0,
where f : Rn 7 Rn .
We claimed earlier that Newtons method extends directly to
higher-dimensional problems. But we need derivatives.
NumCSE, Lecture 14, Nov 3, 2014
6/27
Numerical Methods for Computational Science and Engineering
Introduction
Example: Parabola meets circle
f1 (x1 , x2 ) = x12 2x1 x2 + 1 = 0
(parabola)
f2 (x1 , x2 ) = x12 + x22 1 = 0
(circle)
Two roots:
1
(1)
x
=
0
0
x (2) =
1
NumCSE, Lecture 14, Nov 3, 2014
7/27
Numerical Methods for Computational Science and Engineering
Multivariate Taylor expansion
Multivariate Taylor expansion
Assume: f : Rn 7 Rm has bounded derivatives up to order 2.
Then, for a direction vector p = (p1 , . . . , pn )T , the Taylor
expansion for each function fi in each coordinate xj yields
f(x + p) = f(x) + J(x) p + O(kpk2 ),
where J(x) is the Jacobian matrix of first derivatives of f at x,
f
f1
f1
1
. . . x
x1
x2
n
f
2 f2 . . . f2
x1 x2
xn
J(x) =
..
..
.. .
.
.
.
fm
fm
fm
. . . xn
x1
x2
Pointwise: fi (x + p) = fi (x) +
n
P
j=1
NumCSE, Lecture 14, Nov 3, 2014
fi
xj pj
+ O(kpk2 ), i = 1, . . . , m.
8/27
Numerical Methods for Computational Science and Engineering
Newtons method
Newtons method
I
Starting from an initial guess x0 we generate a sequence of
iterates x1 , x2 , . . . , xk , . . . where xk+1 is obtained from xk by
linearizing f(x) = 0.
Let x = xk + p, then for small p we have
0 = f(xk + p) f(xk ) + J(xk ) p.
We approximate x by xk+1 = xk + pk where pk is solution of
f(xk ) + J(xk ) pk = 0.
I
I
pk is called Newton correction.
Newtons method is a fixed point iteration:
xk+1 = g(xk ) = xk J(xk )1 f(xk ).
NumCSE, Lecture 14, Nov 3, 2014
9/27
Numerical Methods for Computational Science and Engineering
Newtons method
Algorithm: Newtons method for systems
Choose x0 .
for k = 0, 1, . . ., until convergence do
Solve J(xk ) pk = f(xk ) for the Newton correction pk .
Set xk+1 = xk + pk .
end for
Each iteration step requires the solution of a linear system.
May be expensive!
NumCSE, Lecture 14, Nov 3, 2014
10/27
Numerical Methods for Computational Science and Engineering
Newtons method
Matlab template
Matlab template for Newtons method
Matlab: Newtons method
function [x,k] = newton(x,F,DF,tol)
for i=1:MAXIT
p = DF(x) \ F(x);
x = x - p;
if norm(p) < tol*(1 + norm(x))
return;
end
end
Combined relative and absolute stopping criterion:
I
If kxk small: kpk < tol.
If kxk large: kpk < tolkxk.
NumCSE, Lecture 14, Nov 3, 2014
11/27
Numerical Methods for Computational Science and Engineering
Newtons method
Convergence
Fixed point iterations for nonlinear systems
From Banachs fixed point theorem: g : Rn 7 Rn has fixed point if
I
g : D 7 D for some subset D Rn
kg(x) g(y)k Lkx yk, L < 1, for all x, y D.
If x0 D then all xk D and convergence to unique fixed point
x D is guaranteed.
Let Jg be Jacobian of the iteration function g: Jg (x) := Dg(x).
If kJg (x)k < 1 for compatible matrix norm, then (linear)
convergence to x D is guaranteed.
Alternatively: spectral radius (Jg ) := maxi |i (Jg )| < 1.
NumCSE, Lecture 14, Nov 3, 2014
12/27
Numerical Methods for Computational Science and Engineering
Newtons method
Convergence
Convergence of Newtons method
Newton iteration
g(x) = x D f(x)1 f(x) = x J(x)1 f(x)
Jg (x) = I D(Df(x)1 ) f(x) Df(x)1 Df(x)
f(x ) = 0
Jg (x ) = 0.
Local quadratic convergence of Newtons method, if
Df(x ) nonsingular
NumCSE, Lecture 14, Nov 3, 2014
13/27
Numerical Methods for Computational Science and Engineering
Newtons method
Examples
Example: Parabola meets circle (cont.)
f1 (x1 , x2 ) = x12 2x1 x2 + 1 = 0
f2 (x1 , x2 ) =
x12
x22
(parabola)
1=0
The Jacobian is
J(x) =
2x1 2 1
2x1
2x2
(circle)
For Matlab function newton:
f=@(x) [x(1).^2-2*x(1)-x(2)+1; x(1).^2+x(2).^2-1]
Df=@(x) [2*x(1)-2, -1;2*x(1),2*x(2)]
NumCSE, Lecture 14, Nov 3, 2014
14/27
Numerical Methods for Computational Science and Engineering
Newtons method
Examples
Example: Parabola meets circle (cont.)
We call Newton iteration with initial vector x0 = (1, 1)T .
k
0
1
2
3
4
5
6
(k)
x1
1.000000000000000
1.500000000000000
1.083333333333333
1.015350877192982
1.000124514863663
1.000000034615861
1.000000000000001
NumCSE, Lecture 14, Nov 3, 2014
(k)
x2
1.000000000000000
0
-0.166666666666667
-0.004385964912281
-0.000231826605833
-0.000000015495331
-0.000000000000001
kf(xk )k
1.4142e+00
1.2748e+00
2.6589e-01
3.1300e-02
3.4030e-04
7.0945e-08
1.8841e-15
15/27
Numerical Methods for Computational Science and Engineering
Newtons method
Examples
Example: Parabola meets circle (cont.)
Using Newtons method for finding the two roots, varying the
starting point x0 .
Stopping tolerance tol=1.e-7.
1. Starting at x0 = (0, 0)T is bad because J(x0 ) is singular!
2. Starting at x0 = (1, 1)T obtain root (0, 1)T in 5 iterations.
Observe quadratic convergence.
3. Starting at x0 = (1, 1)T obtain root (1, 0)T in 5 iterations.
Observe quadratic convergence.
NumCSE, Lecture 14, Nov 3, 2014
16/27
Numerical Methods for Computational Science and Engineering
Newtons method
Examples
Example 2: 1D nonlinear differential equation
We earlier looked at problem to recover a function v (t) from its
given second derivative g (t) on the interval [0, 1]:
u 00 (t) = g (t),
0 < t < 1,
with boundary conditions v (0) = v (1) = 0.
We wrote ui u(ti ), ti = ih = i/N.
We approximated u 00 (t) = g (t) by
ui+1 2ui + ui1
= g (ti ),
h2
NumCSE, Lecture 14, Nov 3, 2014
i = 1, . . . , N 1.
17/27
Numerical Methods for Computational Science and Engineering
Newtons method
Examples
Example 2: 1D nonlinear differential equation (cont.)
We modify our linear problem:
u 00 (t) = e u(t) ,
0 < t < 1,
u(0) = u(1) = 0.
The system becomes
2 1
1
2 1
1
1
2 1
2
h
.
.. ... ...
1
2
u1
u2
u3
..
.
e u1
e u2
e u3
..
.
uN1
e uN1
This is a nonlinear equation f(u) = 0 with fi =
NumCSE, Lecture 14, Nov 3, 2014
0
0
0
=
..
.
ui1 2ui +ui+1
h2
0
+ e ui .
18/27
Numerical Methods for Computational Science and Engineering
Newtons method
Examples
Example 2: 1D nonlinear differential equation (cont.)
The Jacobian is thus the matrix with elements Jik =
2 + h2 e u1
1
1
h2
1
2 + h2 e u2
1
fi
uj :
1
2 + h2 e u3
..
.
1
..
.
..
h2 e uN1
2 +
We start a Newton iteration with
v0 = (t1 (1 t1 ), . . . , tN1 (1 tN1 ))T ,
NumCSE, Lecture 14, Nov 3, 2014
19/27
Numerical Methods for Computational Science and Engineering
Newtons method
Examples
Example 2: 1D nonlinear differential equation (cont.)
function [f] = ff(u)
% test function
global T
function J = Dff(u)
% test function Jacobian
global T
f = T*u + exp(u);
J = T + diag(exp(u));
NumCSE, Lecture 14, Nov 3, 2014
20/27
Numerical Methods for Computational Science and Engineering
Newtons method
Examples
Example 2: 1D nonlinear differential equation (cont.)
%%% odedriver.m
global T
n=24; h=1/(n+1);
T = (-2*eye(n) + diag(ones(n-1,1),-1) ...
+ diag(ones(n-1,1), 1))/h^2;
%%alf=1;
t=[1:n]*h;
u0=alf*t.*(1-t);
[u,hist] = newton1(u0,@ff,@Dff,1e-8);
its = size(hist,1) - 1
plot([0;t;1],[0;u;0])
xlabel(t), ylabel(u(t))
NumCSE, Lecture 14, Nov 3, 2014
21/27
Numerical Methods for Computational Science and Engineering
Newtons method
Examples
Example 2: 1D nonlinear differential equation (cont.)
Convergence for = 0, 5, 10, 20, 40. No convergence for = 50.
NumCSE, Lecture 14, Nov 3, 2014
22/27
Numerical Methods for Computational Science and Engineering
Newtons method
Simplified Newton method
Simplified Newton method
I
Problem: Factorization of the Jacobian matrix Df(x) may be
expensive.
Solution (?): Use Df(x0 ) in all steps.
Only one factorization necessary.
Loss of quadratic convergence = linear convergence if kJg k
is small enough.
Other Solution: Compute Df(xk ) for several steps.
Factorization of Jacobian every, say, 5 iteration steps.
NumCSE, Lecture 14, Nov 3, 2014
23/27
Numerical Methods for Computational Science and Engineering
Newtons method
Simplified Newton method
Example 2 continued
Linear convergence
for = 5.
NumCSE, Lecture 14, Nov 3, 2014
24/27
Numerical Methods for Computational Science and Engineering
Newtons method
Simplified Newton method
Example 2 continued (cont.)
function [x] = simpnewton5(x,F,DF,tol)
% MATLAB template for simplified Newton method
% The Jacobian is updated every fifth step
for it=0:50,
if (rem(it,5)==0)
[L,U] = lu(DF(x)); % LU-decomposition
end
h = U\(L\F(x));
x = x-h;
if norm(h) < tol*(1+norm(x)),
break
end
end
NumCSE, Lecture 14, Nov 3, 2014
25/27
Numerical Methods for Computational Science and Engineering
Newtons method
Simplified Newton method
Example 2 continued (cont.)
Piecewise linear
convergence for
= 5.
Jacobian is updated
every 5th iteration
step.
NumCSE, Lecture 14, Nov 3, 2014
26/27
Numerical Methods for Computational Science and Engineering
Newtons method
Simplified Newton method
Summary of Newtons method
I
Converges asymptotically very fast: doubling of number of
significant digits in each step
Often a very small region of convergence, which requires an
initial guess rather close to the solution
In high dimensions n the cost of a Newton iteration is mainly
determined by the number of LU factorizations of the
Jacobian J(xk ).
NumCSE, Lecture 14, Nov 3, 2014
27/27
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
I
Unconstrained optimization
Newtons method and relatives
Descent methods
References
I
U. Ascher & C. Greif: Numerical methods. SIAM 2011.
Chapter 9.19.2.
NumCSE, Lecture 15, Nov 6, 2014
2/40
Numerical Methods for Computational Science and Engineering
Introduction
Optimization
I
Simplest optimization problems consist of minimizing a real
(scalar) function:
min (x),
x
Maximization:
max (x)
x
x Rn .
and
min (x)
x
have same set of solution(s) x.
Problems where some energy is minimized (bending of beam)
(x) = xT Ax bT x + c.
Approximation in the least squares sense
(x) = kAx bk.
NumCSE, Lecture 15, Nov 6, 2014
3/40
Numerical Methods for Computational Science and Engineering
Introduction
Example 9.4:
For n = 2, x = (x1 , x2 )T , we specify the function
(x) = x12 + x24 + 1.
This function obviously has the minimum value of 1 at
x = (0, 0)T = 0.
Matlab:
x = [-2:.1:2]; y = x;
[X,Y] = meshgrid(x,y);
F = X.^2 + Y.^4 + 1;
surf(x,y,F)
shading interp
NumCSE, Lecture 15, Nov 6, 2014
4/40
Numerical Methods for Computational Science and Engineering
Introduction
Example 9.4: (cont.)
NumCSE, Lecture 15, Nov 6, 2014
5/40
Numerical Methods for Computational Science and Engineering
Introduction
Practical issues
I
Typically, we cannot find minimum points by inspection.
In practice often unclear how many local minima a function
(x) has.
How shall we efficiently find the global minimum?
NumCSE, Lecture 15, Nov 6, 2014
6/40
Numerical Methods for Computational Science and Engineering
Taylor series and local minima
Multivariate Taylor expansion
Assume: (x) : Rn 7 R has bounded derivatives up to order 3.
Then, for a direction vector p Rn , the Taylor expansion yields
(x + p) = (x) + grad (x)T p + 12 pT 2 (x)p + O(kpk3 ),
where 2 (x) is the Hessian matrix of second derivatives of at x.
2
2
2
.
.
.
2
x
x
x
x
n
1
2
1
x1
1
2
2
2
x
.
.
.
2
x2
x2 xn
x2
x2 x1
grad (x) =
,
(x)
=
.
..
..
..
..
.
.
.
.
xn
2
2
2
xn x1
xn x2 . . .
x 2
n
O(kpk3 ) depends on derivatives of of order 3 and higher.
NumCSE, Lecture 15, Nov 6, 2014
7/40
Numerical Methods for Computational Science and Engineering
Taylor series and local minima
Local minimum
Let x be a local minimum: (x ) (x) in a neighborhood of x .
Then for any direction p we have
1
(x +p) = (x )+grad (x )T p+ pT 2 (x )p+O(kpk3 ) (x ).
2
For very small kpk we have
(x + p) = (x ) + grad (x )T p + arbitrary small terms.
From this we have grad (x ) = 0, i.e., x is a critical point.
Otherwise: If grad (x ) 6= 0 choose p = grad (x ) to get
(x + p) < (x ).
NumCSE, Lecture 15, Nov 6, 2014
8/40
Numerical Methods for Computational Science and Engineering
Taylor series and local minima
Local minimum (cont.)
At a critical point we thus have (with sufficiently small kpk)
1
(x + p) = (x ) + pT 2 (x ) p + O(kpk3 ) (x ).
2
Therefore, the Hessian matrix 2 (x ) must be positive
semidefinite.
If the Hessian matrix 2 (x ) is positive definite then
(x + p) > (x ) for all sufficiently small p. Then x is a strict
minimum of .
NumCSE, Lecture 15, Nov 6, 2014
9/40
Numerical Methods for Computational Science and Engineering
Taylor series and local minima
Theorem: Unconstrained minimization conditions
Assume that (x) is smooth enough, e.g., suppose it has all derivatives up to third order, bounded. Then:
I
A necessary condition for having a local minimum at a point
x is that x be a critical point, i.e.,
grad (x ) = 0
and that the symmetric Hessian matrix 2 (x ) be positive
semidefinite.
A sufficient condition for having a strict local minimum at a
point x is that x be a critical point and that 2 (x ) be
positive definite.
NumCSE, Lecture 15, Nov 6, 2014
10/40
Numerical Methods for Computational Science and Engineering
Newtons method
Newtons method for unconstrained minimization
I
Starting from an initial guess x0 we generate a sequence of
iterates x1 , x2 , . . . , xk , . . . where xk+1 is obtained from xk by
linearizing
f(x) = grad (x) = 0
The next iterate is defined as xk+1 = xk + pk where pk is the
solution of f(xk ) + J(xk ) pk = 0.
For unconstrained minimization this translates into
grad (xk ) + 2 (xk ) pk = 0.
We call pk a Newton correction.
Of course, we have quadratic convergence order.
NumCSE, Lecture 15, Nov 6, 2014
11/40
Numerical Methods for Computational Science and Engineering
Newtons method
Algorithm: Newtons method for unconstrained
minimization
Choose x0 .
for k = 0, 1, . . ., until convergence do
Solve 2 (xk ) pk = grad (xk ) for pk .
Set xk+1 = xk + pk .
end for
We can use the unmodified Matlab template for Newtons
method. The method does not know that we solve a
minimization problem.
The Matlab code may exploit that the Hessian matrix is
symmetric. Note that it need not be SPD.
NumCSE, Lecture 15, Nov 6, 2014
12/40
Numerical Methods for Computational Science and Engineering
Newtons method
Matlab template
Matlab template for Newtons method
Matlab: Newtons method
function [x,k] = newton(x,F,DF,tol)
for i=1:MAXIT
p = DF(x) \ -F(x);
x = x + p;
if norm(p) < tol*(1 + norm(x))
return;
end
end
Combined relative and absolute stopping criterion:
I
If kxk small: kpk < tol.
If kxk large: kpk < tolkxk.
NumCSE, Lecture 15, Nov 6, 2014
13/40
Numerical Methods for Computational Science and Engineering
Newtons method
Example
Example 9.5.
Consider minimizing
1
(x) =
2
2
2
2 !
3
9
21
2
3
x1 (1 x2 ) +
x1 (1 x2 ) +
x1 (1 x2 )
.
2
4
8
A critical point is given by grad (x) = f(x) = 0, where
3
9
f1 (x1 , x2 ) = ( x1 (1 x2 ))(1 x2 ) + ( x1 (1 x22 ))(1 x22 )
2
4
21
3
3
+ ( x1 (1 x2 ))(1 x2 ),
8
3
9
f2 (x1 , x2 ) = ( x1 (1 x2 ))x1 + 2( x1 (1 x22 ))x1 x2
2
4
21
+ 3( x1 (1 x23 ))x1 x22 .
8
NumCSE, Lecture 15, Nov 6, 2014
14/40
Numerical Methods for Computational Science and Engineering
Newtons method
Example
Example 9.5. (cont.)
Global mimimum at
x = (3, .5)T with
(x ) = 0.
Found starting at
(8, .2)T .
Saddle point at
x = (0, 1)T .
Found starting at
(8, .8)T .
Look at the Hessian
matrix at
x.
NumCSE, Lecture 15, Nov 6, 2014
15/40
Numerical Methods for Computational Science and Engineering
Newtons method
Example
Example 9.5. (cont.)
k
kxk x k
0
1
2
3
4
5
6
7
8
5.01e+00
8.66e-01
6.49e-02
1.39e-01
2.10e-02
1.38e-03
3.03e-06
2.84e-11
|(xk )
(x )|
4.09e+01
1.21e+00
1.20e-02
1.72e-03
6.91e-05
1.43e-07
1.09e-12
6.16e-23
fkT pk
kxk
xk
7.3e+01
2.3e+00
2.3e-02
3.2e-03
1.4e-04
2.9e-07
2.2e-12
8.00e+00
6.95e+00
7.21e+00
1.36e+01
3.16e+00
4.44e-01
6.90e-02
2.43e-04
1.50e-09
|(xk )
(
x)|
6.08e+00
6.97e+00
6.99e+00
2.80e+02
1.90e+01
8.78e-01
2.12e-03
1.42e-08
0.00e+00
fkT pk
1.7e+00
2.9e-02
3.2e-01
4.7e+02
3.2e+01
1.9e+00
4.2e-03
2.8e-08
5.4e-19
fk = grad (xk ).
NumCSE, Lecture 15, Nov 6, 2014
16/40
Numerical Methods for Computational Science and Engineering
Descent methods
Descent direction
I
There is a difference of unconstrained minimization and plain
nonlinear system solving: .
Would like to have (xk+1 ) = (xk + pk ) < (xk ).
The directional derivative of in direction of p must be
negative:
grad (x)T p < 0.
A vector p for which this inequality holds is called a descent
direction.
Remember solving SPD systems: there
(x) = 21 xT Ax bT x + and grad (x) = Ax b = r.
p is descent direction if rT p > 0
NumCSE, Lecture 15, Nov 6, 2014
17/40
Numerical Methods for Computational Science and Engineering
Descent methods
Descent methods
I
Lets consider methods of the form
xk+1 = xk + k pk ,
pk = Bk1 grad (xk )
If Bk is SPD then pk is a descent direction.
Newtons method: Bk = 2 (xk ).
Advantages/disadvantages of Newtons method:
quadratic convergence, sparsity (?) of Hessian
Existence/evaluation of Hessian
Bk may not be SPD away from minimum.
No control on convergence.
How to choose Bk ?
NumCSE, Lecture 15, Nov 6, 2014
18/40
Numerical Methods for Computational Science and Engineering
Descent methods
Gradient descent
Gradient descent: Bk = I
pk = grad (xk )
+ Garanteed reduction of .
+ No linear system to be solved.
We know from system solving: slow convergence.
NumCSE, Lecture 15, Nov 6, 2014
19/40
Numerical Methods for Computational Science and Engineering
Descent methods
Trust region method
Trust region method: Bk = 2 (xk ) + k I
pk = Bk1 grad (xk )
As expensive as Newton
k 0 is chosen such that Bk is SPD. How?
Then pk is a descent method.
NumCSE, Lecture 15, Nov 6, 2014
20/40
Numerical Methods for Computational Science and Engineering
Descent methods
Line search
Search for an k such that (xk+1 ) = (xk + k pk ) < (xk ).
Simple backtracking algorithm:
Set k = 1.
while (xk + k pk ) > (xk ) do
Set k = k /2;
end while
More sophisticated approach:
Starting with
= 1, determine quadratic polynomial () such
that (0) = (xk ), 0 (0) = pT
k grad (xk ), and
(
) = ((xk +
pk ) where
is current unsatisfactory step size.
Minimize () to get k .
NumCSE, Lecture 15, Nov 6, 2014
21/40
Numerical Methods for Computational Science and Engineering
Descent methods
Example 9.6.
With (x) = x14 + x1 x2 + (1 + x2 )2 we get
4x13 + x2
12x12 1
2
grad (x) =
,
(x) =
.
x1 + 2(1 + x2 )
1
2
has unique minimum at x (0.6959, 1.3479)T , where
(x ) 0.5824.
I
Starting from (0.75, 1.25)T , plain Newton converges in
4 steps.
Starting from (0, 0.3)T , plain Newton diverges.
The Hessian matrix is SPD only for x12 > 1/24. The xk are
trapped in a bad region.
NumCSE, Lecture 15, Nov 6, 2014
22/40
Numerical Methods for Computational Science and Engineering
Descent methods
Example 9.6. (cont.)
I
Starting from (0.75, 1.25)T and backtracking with Newton
direction does not change the iterates.
Starting from (0, 0.3)T and backtracking converges after 6
steps. Backtracking is needed in 3 out of 6 steps.
k
0
1
2
3
4
5
6
1
0.147
0.386
0.479
1
1
1
NumCSE, Lecture 15, Nov 6, 2014
kxk x k
1.79e+00
1.88e+00
8.89e-01
3.37e-01
2.85e-03
1.14e-05
1.83e-10
|(xk ) (x )|
2.27e+00
2.09e+00
6.43e-01
1.31e-01
1.74e-05
2.76e-10
2.22e-16
fkT pk
1.38e+00
4.15e+00
1.63e+00
2.62e-01
3.46e-05
5.51e-10
1.42e-19
23/40
Numerical Methods for Computational Science and Engineering
Descent methods
Inexact Newton
I
Large problems with sparse Hessian matrices: iterative
methods to determine Newton correction pk is appropriate.
Incorporating this yields a method with an inner loop inside an
outer loop.
Outer loop: Newton-like iteration with (e.g.)
Bk = 2 (xk ) + k I with k 0 such that Bk definite and
sparse.
Inner loop: Solves linear system of equation. PCG is favorite
method.
Issue: accuracy of inner solve vs. order of convergence of
outer loop.
Intuitive: fewer steps when far, more when close to x .
NumCSE, Lecture 15, Nov 6, 2014
24/40
Numerical Methods for Computational Science and Engineering
Quasi-Newton methods
Quasi-Newton methods
What to do when Hessian 2 (x) is not available and numerical
differentiation is too expensive?
I
Do not want to form or evaluate Hessian matrix explicitly;
also,
want a positive definite Bk that is easy to invert.
Note by Taylors expansion for the gradient
grad (xk ) grad (xk+1 ) 2 (xk+1 )(xk+1 xk )
Essential action of Hessian is in direction of wk = xk+1 xk .
Thus, require
Bk+1 wk = yk ,
NumCSE, Lecture 15, Nov 6, 2014
yk = grad (xk+1 ) grad (xk ).
25/40
Numerical Methods for Computational Science and Engineering
Quasi-Newton methods
Quasi-Newton methods (cont.)
I
Construct Bk+1 as a positive definite rank-2 update of Bk ,
thus satisfying the above requirements.
Note that this can be considered as a generalization of the
secant method to dimensions > 1. In 1D:
f 00 (xk )
f 0 (xk+1 ) f 0 (xk )
xk+1 xk
(difference quotient)
Bk+1 approximates 2 in the direction of xk+1 xk . In this
direction Bk+1 coincides approximately with that of
2 (xk+1 ).
NumCSE, Lecture 15, Nov 6, 2014
26/40
Numerical Methods for Computational Science and Engineering
Quasi-Newton methods
Quasi-Newton methods (cont.)
Bk+1 wk = yk ,
wk = xk+1 xk ,
yk = grad (xk+1 )grad (xk ).
Many matrices Bk+1 satisfy this equation! Choose Bk+1 as a low
rank modification of Bk .
There are a number of variants available. Most popular is BFGS
(BroydenFletcherGoldfarbShanno).
In this algorithm Gk = Bk1 is used. So, solving the linear system
Bk pk = grad (xk ) is replaced by a matvec:
pk = Gk grad (xk ).
NumCSE, Lecture 15, Nov 6, 2014
27/40
Numerical Methods for Computational Science and Engineering
Quasi-Newton methods
Algorithm: The BFGS iteration
Choose x0 and G0 (e.g., G0 = I , or (I)C factorization of 2 ).
for k = 0, 1, . . ., until convergence do
pk = Gk grad (xk ).
Find a suitable stepsize k .
xk+1 = xk + k pk .
wk = k pk .
yk = grad
(xk+1 )Tgrad(xk ). T
y w
w wT
w y
Gk+1 = I Tk k Gk I kT k + Tk k .
yk w k
wk y k
yk wk
end for
Note that Gk is defined recursively!
BFGS is implemented in Matlabs function fminunc.
NumCSE, Lecture 15, Nov 6, 2014
28/40
Numerical Methods for Computational Science and Engineering
Quasi-Newton methods
Example for Broydens quasi-Newton method
0
10
10
Small 2 2 example.
10
The numerical example
shows that the method is:
Normen
10
10
slower than Newton, but
better than simplified
Newton.
10
10
Broyden: ||F(x(k))||
Broyden: Fehlernorm
Newton: ||F(x(k))||
Newton: Fehlernorm
Newton (vereinfacht)
12
10
14
10
5
6
Iterationsschritt
10
11
Broyden method is worthwile for dimensions n 1 and low
accuracy requirements.
NumCSE, Lecture 15, Nov 6, 2014
29/40
Numerical Methods for Computational Science and Engineering
Quasi-Newton methods
Quasi-Newton pros and cons
I
Cheap update of Gk = Bk1 using rank-2 updates.
Local superlinear convergence.
Only descent directions because Gk = Bk 1 symmetric
positive definite.
The method of choice for most problems.
May lose Hessian sparsity, though.
Limited memory versions L-BFGS exist for large, sparse
matrices. They have their pros and cons...
NumCSE, Lecture 15, Nov 6, 2014
30/40
Numerical Methods for Computational Science and Engineering
Nonlinear least squares
Nonlinear least squares
Given observed data b and a model function g(x) which predicts
the data for each parameter x:
minkg(x) bk
x
g : Rn 7 Rm , m > n, g C 2 .
Similarly as in the case of linear least squares problems (g(x) = Ax)
we have an overdetermined non-linear system of equations.
We want to determine x such that the residual r = r(x) gets
minimal length.
m
1X
1
(x) := kg(x) bk2 =
[gi (x) bi ]2 .
2
2
i=1
NumCSE, Lecture 15, Nov 6, 2014
31/40
Numerical Methods for Computational Science and Engineering
Nonlinear least squares
Nonlinear least squares (cont.)
At the minimum we have to have
m
(x) X
gi (x)
=
[gi (x) bi ]
= 0,
xj
xj
j = 1, . . . , n.
i=1
or with the Jacobian of g (i.e. A(x) = D g(x) =
g
x )
grad (x ) = A(x )T (g(x ) b) = 0.
These are n nonlinear equations in n unknowns.
(These are normal equations in linear least squares (g(x) = Ax))
NumCSE, Lecture 15, Nov 6, 2014
32/40
Numerical Methods for Computational Science and Engineering
Nonlinear least squares
Nonlinear least squares (cont.)
The Hessian of this problem is
2 (x) = A(x)T A(x) + L(x)
with
Li,j (x) =
m
X
2 gk
(gk bk )
xi xj
k=1
We could solve this nonlinear system of equations, but very often
this is too expensive.
It is mostly more economic to linearize the error equations
g(x) b = r.
NumCSE, Lecture 15, Nov 6, 2014
33/40
Numerical Methods for Computational Science and Engineering
Nonlinear least squares
Nonlinear least squares (cont.)
Given the approximation xk of the solution x and
x = xk + p.
Then
g(x ) g(xk ) + D g(xk ) p g(xk ) + A(xk ) p.
We plug the right side into the error equation:
g(xk ) + A(xk ) p b =: k .
The correction p is obtained by solving
A(xk ) p dk = k 0,
dk = b g(xk ),
in the least squares sense.
NumCSE, Lecture 15, Nov 6, 2014
34/40
Numerical Methods for Computational Science and Engineering
Nonlinear least squares
Nonlinear least squares (cont.)
We get the GaussNewton method:
Choose x(0) , d(0) = b g(x(0) ), k = 0,
repeat
Solve A(xk ) pk dk for pk ,
xk+1 = xk + pk ,
dk+1 = b g(xk+1 ),
k = k + 1,
until kpk k < tol.
NumCSE, Lecture 15, Nov 6, 2014
35/40
Numerical Methods for Computational Science and Engineering
Nonlinear least squares
Example 9.8:
We generate data using the function u(t) = e 2t cos(20t), see the
solid blue curve in the next figure. At the 51 points ti = .02(i 1),
i = 1, . . . , 51, we add 20% random noise to u(ti ) to generate data
bi (green circles in figure).
We now pretend that we have never seen the blue curve and
attempt to fit to this b a function of the form
v (t) = x1 e x2 t cos(x3 t).
In the above notation we have m = 51, n = 3, and the ith rows of
g and A are
gi = x1 e x2 ti cos(x3 ti ),
ai,2 = ti gi ,
NumCSE, Lecture 15, Nov 6, 2014
ai,1 = e x2 ti cos(x3 ti ),
ai,3 = ti x1 e x2 ti sin(x3 ti ),
1 i m.
36/40
Numerical Methods for Computational Science and Engineering
Nonlinear least squares
Example 9.8: (cont.)
NumCSE, Lecture 15, Nov 6, 2014
37/40
Numerical Methods for Computational Science and Engineering
Nonlinear least squares
Example 9.8: (cont.)
I
Start GaussNewton method with x0 = (1.2, 1.9, 18)T , we
obtain after 7 iterations pk < 107 ,
x (1.0471, 2.1068, 19.9588)T , and g b .42. The
predicted data are displayed as red diamonds in the figure.
This is a very satisfactory result.
With the noise level at 100% we obtain after 15 iterations
g b 1.92 for x (1.0814, 3.0291, 19.8583)T .
When the model approximates the data less well, more
iterations are required.
Using rougher initial guess x0 = (1.2, 1.9, 10)T quickly leads
to another solution.
NumCSE, Lecture 15, Nov 6, 2014
38/40
Numerical Methods for Computational Science and Engineering
Nonlinear least squares
GaussNewton vs. Newton
GaussNewton differs from Newton. Newtons method reads
2 (xk )pk = grad (xk ) = A(xk )T (b g(xk )),
The right-hand side is the same as for the normal equations of the
GaussNewton iteration, but the matrix 2 (xk ) contains the
extra term L, which is dropped in the GaussNewton iteration.
I
Advantage of the GaussNewton method: second derivative of
not needed, cheaper.
Drawback of the GaussNewton method: no local quadratic
convergence.
GaussNewton direction is a descent direction w.r.t. . (AT A
is SPD while AT A + L may not be so.)
NumCSE, Lecture 15, Nov 6, 2014
39/40
Numerical Methods for Computational Science and Engineering
Summary
Summary of nonlinear system solving
I
Newtons method is method of choice if applicable.
I
I
I
Problems with computation of derivatives
I
I
Very fast asymptotic convergence
However, often very small region of convergence
Stabilization (damped Newton, line search)
Numerical differentiation
Quasi-Newton
Problems with linear system solving
I
I
I
Simplified Newton
Quasi-Newton
Inexact Newton: iterative solution of Newton step
(preconditioner?)
NumCSE, Lecture 15, Nov 6, 2014
40/40
Numerical Methods for Computational Science and Engineering
Introduction
Goals of chapter on interpolation
I
To motivate the need for interpolation of data and of
functions;
to derive three different methods for computing a polynomial
interpolant, each particularly suitable for certain
circumstances;
to derive error expressions for the polynomial interpolation
process;
to construct Chebyshev interpolants, which can provide very
accurate approximations for complex functions using stable
high degree polynomial interpolation at special points; and
to consider cases where not only function values but also
values of derivatives are to be interpolated.
NumCSE, Lecture 16, Nov 10, 2014
2/41
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
I
What is interpolation
Monomial basis
Lagrange basis
Newton basis and divided differences
References
I
U. Ascher & C. Greif: Numerical methods. SIAM 2011.
Chapter 10.110.3.
NumCSE, Lecture 16, Nov 10, 2014
3/41
Numerical Methods for Computational Science and Engineering
Introduction
Interpolating data
We are given a collection of data samples {(xi , yi )}ni=0
I
The {xi }ni=0 are called the abscissae, the {yi }ni=0 are called
the data values.
Want to find a function v (x) which can be used to estimate
sampled function for x 6= xi .
Interpolation:
v (xi ) = yi ,
i = 0, 1, . . . , n.
Why? We often get discrete data from sensors or
computation, but we want information as if the function were
not discretely sampled.
Want a reasonable looking interpolant. If possible, v (x)
should be inexpensive to evaluate for a given x.
NumCSE, Lecture 16, Nov 10, 2014
4/41
Numerical Methods for Computational Science and Engineering
Introduction
Interpolating data (cont.)
Want a reasonable looking interpolant. Example:
If possible, v should be inexpensive to construct.
If possible, v(x) should be inexpensive to evaluate for given x.
NumCSE, Lecture 16, Nov 10, 2014
5/41
Numerical Methods for Computational Science and Engineering
Introduction
Interpolating functions
A function f (x) may be given explicitly or implicitly. Want
interpolant v (x) such that
v (xi ) = f (xi ),
i = 0, 1, . . . , n.
Same algorithms as for interpolating data, but here
I
May be able to choose abscissae xi .
May be able to estimate interpolation error.
There are lots of ways to define a function v (x) to interpolate the
data: polynomials, trigonometric functions, exponential, rational
functions (fractions), wavelets, etc.
NumCSE, Lecture 16, Nov 10, 2014
6/41
Numerical Methods for Computational Science and Engineering
Introduction
Interpolation and approximation
I
Interpolation means that the approximating function g
coincides with the function at prescribed points:
v (xi ) = f (xi ) = yi , i = 0, . . . , n.
Approximation means that some norm kv yk of the
difference of the vectors v = [v (x0 ), . . . , v (xn )] and
y = [y0 , . . . , yn ] is minimized, or the difference of the given
function f (x) and v (x).
Quality of approximation/interpolation depends on the
method and the choice of v .
Interpolation: g (z) is computed at some new value z 6= xi
inside the range of the interpolation points x0 , . . . , xn .
Extrapolation: the new value z is outside this range.
NumCSE, Lecture 16, Nov 10, 2014
7/41
Numerical Methods for Computational Science and Engineering
Introduction
The need for interpolation/extrapolation
I
For prediction: we can use v (x) to find approximate values of
the underlying function at locations x other than the data
abscissae, x0 , . . . , xn .
I
I
For manipulation: an instance is finding approximations for
derivatives and integrals of the underlying function.
I
Tabulated numbers
Stock performance
Remember the computation of in the step size in a line search
in optimization.
The interpolating function should be easy to evaluate and
manipulate, and of course to generate.
NumCSE, Lecture 16, Nov 10, 2014
8/41
Numerical Methods for Computational Science and Engineering
Interpolation formulation
Interpolation formulation
Assume a linear form of interpolation
v (x) =
n
X
cj j (x) = c0 0 (x) + + cn n (x)
j=0
where {ci }ni=0 are unknown coefficients or parameters and
{i (x)}ni=0 are predetermined basis functions.
The basis functions are assumed to be linearly independent.
There are n + 1 coefficients to be determined by n + 1 equations.
NumCSE, Lecture 16, Nov 10, 2014
9/41
Numerical Methods for Computational Science and Engineering
Interpolation formulation
Interpolation formulation (cont.)
The n + 1 equations v (xi ) = yi , i = 0, 1, . . . , n yield
0 (x0 ) 1 (x0 ) 2 (x0 ) n (x0 )
y0
c0
0 (x1 ) 1 (x1 ) 2 (x1 ) n (x1 ) c1 y1
..
..
..
.. .. = ..
.
.
.
. . .
0 (xn ) 1 (xn ) 2 (xn )
n (xn )
cn
yn
Choices for the j (x):
I
Polynomial interpolation with monomial basis j (x) = x j .
Piecewise polynomial interpolation: soon to come.
Trigonometric interpolation: j (x) = cos(jx).
NumCSE, Lecture 16, Nov 10, 2014
10/41
Numerical Methods for Computational Science and Engineering
Examples
Example 1: linear data fitting
{(xi , yi )} = {(2, 14), (6, 24), (4, 25), (7, 15)}
I
Requires four basis functions: {j (x)} = {1, x, x 2 , x 3 }.
The interpolant will be {p(x) = c0 1 + c1 x + c2 x 2 + c3 x 3 }.
Construct linear system
1 2 4
8
1 6 36 216
A=
1 4 16 64
1 7 49 343
14
24
y=
25
15
Solve c = A\y. We find c (3.800, 2.767, 1.700, 0.267)T .
NumCSE, Lecture 16, Nov 10, 2014
11/41
Numerical Methods for Computational Science and Engineering
Examples
Example 1: linear data fitting (cont.)
x=[2 6 4 7];
y=[14 24 25 15];
c=polyfit(x,y,3)
xx=[1:.02:8];
yy=polyval(c,xx);
plot(x,y,ro,xx,yy)
NumCSE, Lecture 16, Nov 10, 2014
12/41
Numerical Methods for Computational Science and Engineering
Examples
Example 2: fitting a rational function
Consider the function
Zx
f (x) =
e sin t dt
some values of which are tabulated:
x
y
0.4
0.5
0.6
0.7
0.8
0.9
0.4904 0.6449 0.8136 0.9967 1.1944 1.4063
What is the value f (0.66)?
Lets choose
g (x) =
NumCSE, Lecture 16, Nov 10, 2014
a
x b
13/41
Numerical Methods for Computational Science and Engineering
Examples
Example 2: fitting a rational function (cont.)
We determine the parameters a and b by requiring that g
interpolates at both neighboring points of z:
a
= 0.8136,
0.6 b
a
g (0.7) =
= 0.9967.
0.7 b
g (0.6) =
The result is a = 0.4429 and b = 1.1443 thus
g (0.66) =
0.4429
= 0.9145 f (0.66) = 0.9216
0.66 1.1443
Interpolation error is rather large: |g (0.66) f (0.66)| = 0.0071.
Choice of the model function g was not clever: linear function
g (x) = ax + b leads to smaller error 0.0019.
NumCSE, Lecture 16, Nov 10, 2014
14/41
Numerical Methods for Computational Science and Engineering
Polynomial interpolation
Polynomial interpolation
I
A common choice of model functions for interpolation is
polynomials, which are easy to evaluate and smooth, i.e.,
infinitely differentiable ( C ).
Given n + 1 points xi , we are looking for a polynomial p(x)
such that
p(xi ) = f (xi ) = yi ,
i = 0, . . . , n.
()
As we have n + 1 constraints to satisfy, we need n + 1 degrees
of freedom. Consider the n-th degree polynomial
pn (x) = c0 + c1 x + + cn1 x n1 + cn x n .
The n + 1 coefficients ci have to be determined in such a way
that () is satisfied.
NumCSE, Lecture 16, Nov 10, 2014
15/41
Numerical Methods for Computational Science and Engineering
Polynomial interpolation
Polynomial interpolation (cont.)
This leads to the linear system of equations
c0 + c1 x0 + + cn1 x0n1 + cn x0n = y0
c0 + c1 x1 + + cn1 x1n1 + cn x1n = y1
..
.
c0 + c1 xn + + cn1 xnn1 + cn xnn = yn .
In matrix
1
1
..
.
form, the system reads
x0 . . . x0n1 x0n
x1 . . . x1n1 x1n
..
..
..
.
.
.
1 xn . . . xnn1 xnn
{z
|
V
NumCSE, Lecture 16, Nov 10, 2014
c0
..
.
cn1
cn
} | {z
c
y0
y1
..
.
(1)
yn
}
| {z }
=
y
16/41
Numerical Methods for Computational Science and Engineering
Polynomial interpolation
Polynomial interpolation (cont.)
I
V is called a Vandermonde matrix. Its elements are the
powers of the nodes.
The determinant of V is
det(V ) =
(xi xj )
i6=j
I
If all the nodes xi are distinct, then V is nonsingular and the
linear system of equations has a unique solution.
We will prove this statement without using determinants.
NumCSE, Lecture 16, Nov 10, 2014
17/41
Numerical Methods for Computational Science and Engineering
Polynomial interpolation
Uniqueness of interpolation
The polynomial pn (x) in () is the only polynomial of degree n
that interpolates f at the n + 1 distinct points x0 , . . . , xn .
Proof.
If qn (x) Pn was another such polynomial:
pn (xk ) = qn (xk ) = yk ,
k = 0, 1, . . . , n.
Then the difference polynomial d(x) := pn (x) qn (x) Pn has
n + 1 zeros x0 , . . . , xn . By the fundamental theorem of algebra a
nonzero polynomial of degree n has at most n zeros. But d has
n + 1 distinct zeros; hence, it must be identically zero, meaning
that qn (x) pn (x).
Note: this statement proves nonsingularity of Vandermonde matrix.
NumCSE, Lecture 16, Nov 10, 2014
18/41
Numerical Methods for Computational Science and Engineering
Polynomial interpolation
Polynomials in Matlab
Matlab displays polynomials as row vectors containing the
coefficients ordered by descending powers.
p(x) = c0 + c1 x + . . . + cn x n
c = [cn , . . . , c1 , c0 ]
>> x=[0:4]; y=[0 1 0 1 0];
>> V=vander(x)
V =
0
0
0
0
1
1
1
1
1
1
16
8
4
2
1
81
27
9
3
1
256
64
16
4
1
>> a=V\y;
>> X=[0:.1:5]; plot(X, polyval(a,X))
NumCSE, Lecture 16, Nov 10, 2014
19/41
Numerical Methods for Computational Science and Engineering
Polynomial interpolation
Polynomials in Matlab (cont.)
NumCSE, Lecture 16, Nov 10, 2014
20/41
Numerical Methods for Computational Science and Engineering
Polynomial interpolation
Monomial basis assessment
+ Simple!
+ Matrix A is a Vandermonde matrix: nonsingular.
Hence uniqueness: there is precisely one interpolating
polynomial.
Construction cost O(n3 ) flops (high if n is large)
+ Evaluation cost using Horner, O(n) flops (low)
Coefficients cj not indicative of f (x) and all change if data are
modified.
Potential stability difficulties if degree is large or abscissae
spread apart.
NumCSE, Lecture 16, Nov 10, 2014
21/41
Numerical Methods for Computational Science and Engineering
Lagrange interpolation
Lagrange interpolation
Lagrange polynomials are polynomials of degree n:
Li (x) =
n
Y
x xj
,
x
i xj
j=0
i = 0, 1, . . . , n,
j6=i
(x x0 ) (x xi1 )(x xi+1 ) (x xn )
(xi x0 ) (xi xi1 )(xi xi+1 ) (xi xn )
They satisfy
(
1, if i = k,
Li (xk ) = ik =
0, if i 6= k.
Therefore,
NumCSE, Lecture 16, Nov 10, 2014
pn (x) =
n
X
f (xj ) Lj (x),
(2)
j=0
22/41
Numerical Methods for Computational Science and Engineering
Lagrange interpolation
Lagrange interpolation (cont.)
Interpolation is guaranteed:
n
X
pn (xi ) =
f (xj ) Lj (xi ) = f (xi ) Li (xi ) = f (xi ).
j=0
Lagrange polynomials of
degree 4, with abszissae at
x = 2, 4, 6, 7, and value 1 at
one of these x.
NumCSE, Lecture 16, Nov 10, 2014
23/41
Numerical Methods for Computational Science and Engineering
Lagrange interpolation
Construction and evaluation
Construction: Construct auxiliary quantities independent of x
j =
(xj xi ),
wj =
i6=j
1
,
j
j = 0, . . . , n.
The quantities wj are called barycentric weights.
Evaluation:
(x) =
n
Y
(x xi ).
i=0
Then,
p(x) = (x)
NumCSE, Lecture 16, Nov 10, 2014
n
X
j=0
wj yj
,
(x xj )
yj = f (xj ).
24/41
Numerical Methods for Computational Science and Engineering
Lagrange interpolation
Construction and evaluation (cont.)
If we interpolate the constant function f (x) 1, then we get
n
X
wj 1
.
1 = (x)
(x xj )
j=0
1
n
X
w
1
j
and
Therefore, (x) =
(x xj )
j=0
n
X
p(x) =
j=0
n
X
j=0
NumCSE, Lecture 16, Nov 10, 2014
wj yj
(x xj )
wj
(x xj )
25/41
Numerical Methods for Computational Science and Engineering
Lagrange interpolation
Algorithm: Lagrange polynomial interpolation
1. Construction:
weights
Given data {xi , yi }ni=0 .
wj = Q
1
,
(x
i6=j j xi )
Compute barycentric
j = 0, 1, . . . , n,
and also quantities wj yj .
2. Evaluation: Given an evaluation point x not equal to one of the
data points {xi }ni=0 . Compute
wj yj
j=0 (xxj )
Pn
wj
j=0 (xxj )
Pn
p(x) =
NumCSE, Lecture 16, Nov 10, 2014
26/41
Numerical Methods for Computational Science and Engineering
Lagrange interpolation
Lagrange basis assessment
Not so simple.
+ Matrix A is the identity: the coefficients are immediately
obtained.
+ Construction cost O(n2 ) flops (OK even if n is large)
+ Evaluation cost O(n) flops (low but not lowest)
+ Coefficients cj indicative of data and useful for function
manipulation such as integration and differentiation.
+ Stable even if degree is large or abscissae spread apart.
NumCSE, Lecture 16, Nov 10, 2014
27/41
Numerical Methods for Computational Science and Engineering
Newton interpolation
Newton interpolation
For Newton interpolation we represent the interpolation polynomial
in a different way:
pn (x) := c0 + c1 (x x0 ) + c2 (x x0 )(x x1 ) +
+ cn (x x0 )(x x1 ) (x xn1 ).
All terms but the first vanish at x0 :
Pn (x0 ) = c0
c0 = y0 .
All terms but the first two vanish at x1 :
pn (x1 ) = c0 + c1 (x1 x0 )
NumCSE, Lecture 16, Nov 10, 2014
c1 =
y1 y0
.
x1 x0
28/41
Numerical Methods for Computational Science and Engineering
Newton interpolation
Newton interpolation (cont.)
All terms but the first three vanish at x2 :
pn (x2 ) = c0 + c1 (x2 x0 ) + c2 (x2 x0 )(x2 x1 )
=
c2 =
y2 y0
y1 y0
x1 x0 (x2
x0 )
(x2 x0 )(x2 x1 )
y2 y0 y1 y0
x2 x0 x1 x0
c2 =
x2 x1
The ci are defined recursively: they are ratios of differences of
previously computed ratios. These so-called divided differences can
be defined recursively as follows.
NumCSE, Lecture 16, Nov 10, 2014
29/41
Numerical Methods for Computational Science and Engineering
Newton interpolation
Example: quadratic polynomial
{(xi , yi )} = {(1, 1), (2, 3), (4, 3)}
0 (x) = 1, 1 (x) = x x0 = x 1,
2 (x) = (x x0 )(x x1 ) = (x 1)(x 2).
1 = p(1) = c0 0 (1) + c1 1 (1) + c2 2 (1) = c0 1 + c1 0 + c2 0 = c0 .
3 = p(2) = 1 + c1 (2 1) + c2 0
3 = p(4) = 1 + 2(4 1) + c2 (4 1)(4 2)
c1 = 2.
=
2
c2 = .
3
2
p2 (x) = 1 + 2(x 1) (x 1)(x 2).
3
NumCSE, Lecture 16, Nov 10, 2014
30/41
Numerical Methods for Computational Science and Engineering
Newton interpolation
Example: cubic polynomial
{(xi , yi )} = {(2, 14), (6, 24), (4, 25), (7, 15)}
Use Newton basis.
I Four basis functions:
0 (x) = 1,
1 (x) = (x 2),
2 (x) = (x 2)(x 6),
I
Construct linear system
1
1
A=
1
1
I
3 (x) = (x 2)(x 6)(x 4).
0
0 0
4
0 0
2 4 0
5
5 15
and solve Ac = y to find c [14, 2.5, 1.5, 0.2667]T
Same polynomial as before, but different representation!
NumCSE, Lecture 16, Nov 10, 2014
31/41
Numerical Methods for Computational Science and Engineering
Newton interpolation
Newton interpolation
Zeroth divided difference:
f [xi ] := f (xi ).
First divided difference:
f [x0 , x1 ] :=
f (x1 ) f (x0 )
f [x1 ] f [x0 ]
=
x1 x0
x1 x0
k-th divided difference:
f [xi , xi+1 , . . . , xi+k ] :=
f [xi+1 , xi+2 , . . . , xi+k ] f [xi , xi+1 , . . . , xi+k1 ]
.
xi+k xi
where k = 2, 3, . . . , n and i = 0, 1, . . . , n k.
NumCSE, Lecture 16, Nov 10, 2014
32/41
Numerical Methods for Computational Science and Engineering
Newton interpolation
Newton interpolation (cont.)
For example, the first divided difference would evaluate to
f [x0 , x1 ] =
f [x1 ] f [x0 ]
f (x1 ) f (x0 )
=
.
x1 x0
x1 x0
The second divided difference would amount to
f [x1 , x2 ] f [x0 , x1 ]
f [x0 , x1 , x2 ] =
=
x2 x0
f (x2 )f (x1 )
x2 x1
f (x1 )f (x0 )
x1 x0
x2 x0
The third divided difference is
f [x0 , x1 , x2 , x3 ] =
NumCSE, Lecture 16, Nov 10, 2014
f [x1 , x2 , x3 ] f [x0 , x1 , x2 ]
.
x3 x0
33/41
Numerical Methods for Computational Science and Engineering
Newton interpolation
Newton interpolation (cont.)
The divided difference are best arranged in a triangular array:
x0 f [x0 ] = c0
f [x0 , x1 ] = c1
x1 f [x1 ]
f [x0 , x1 , x2 ] = c2
f [x1 , x2 ]
x2 f [x2 ]
f [x0 , x1 , x2 , x3 ] = c3
f [x1 , x2 , x3 ]
f [x2 , x3 ]
x3 f [x3 ]
Here is the case n = 3.
NumCSE, Lecture 16, Nov 10, 2014
34/41
Numerical Methods for Computational Science and Engineering
Newton interpolation
Matlab code
function [c,D] = coeffnewton0(x,y)
% COEFFNEWTON computes the divided differences needed for
%
constructing the interpolating polynomial
%
through (x_i,y_i)
n = length(x)-1;
% degree of interpolating polynomial
for i=1:n+1
% divided differences
D(i,1) = y(i);
for j=1:i-1
D(i,j+1) = (D(i,j)-D(i-1,j))/(x(i)-x(i-j));
end
end
c=diag(D);
NumCSE, Lecture 16, Nov 10, 2014
35/41
Numerical Methods for Computational Science and Engineering
Newton interpolation
Matlab code: construction of vector c without table of
divided difference
function [c] = coeffnewton(x,y)
% COEFFNEWTON computes the divided differences needed for
%
constructing the interpolating polynomial
%
through (x_i,y_i) in place
n = length(x)-1;
% degree of interpolating polynomial
for k=1:n+1
% divided differences
c(k) = y(k);
end
for k=1:n
for i=n+1:-1:k+1
c(i)=(c(i)-c(i-1))/(x(i)-x(i-k));
end
end
c=c(:);
NumCSE, Lecture 16, Nov 10, 2014
36/41
Numerical Methods for Computational Science and Engineering
Newton interpolation
Evaluation of Newton polynomial by Horner scheme
We evaluate the interpolation polynomial for a new argument z by
a procedure similar to Horners rule
pn (z) = c0 + (z x0 ) (c1 + (z x1 ) (c2 +
+(z xn1 ) (cn ) ))
by the recurrence
p := cn
p := (z xi )p + ci ,
i = n 1, n 2, . . . , 0.
Complexity. The computation of the Newton coefficients costs
3n2 /2 flops. The evaluation of the Newton polynomial costs 3n
flops per data point.
NumCSE, Lecture 16, Nov 10, 2014
37/41
Numerical Methods for Computational Science and Engineering
Newton interpolation
Matlab code
function y = intnewton(x,c,z)
% INTNEWTON evaluates the Newton interpolating polynomial
%
at the new points z: y = P_n(z) using the
%
Horner form and the diagonal c of the
%
divided difference scheme.
n = length(x)-1;
y = c(n+1);
for i= n:-1:1
y = y.*(z - x(i)) + c(i);
end
NumCSE, Lecture 16, Nov 10, 2014
38/41
Numerical Methods for Computational Science and Engineering
Newton interpolation
Matlab example
We want to compute log(1.57) = 0.451075619360217. We have a
table with 4 values of the natural logarithm:
x
1.4
1.5
1.6
1.7
y
0.336472236621213
0.405465108108164
0.470003629245736
0.530628251062170
With the above Matlab functions we can easily verify that
indices
1,2,3
2,3,4
1,2,3,4
NumCSE, Lecture 16, Nov 10, 2014
(1.57)
p...
0.451109779691149
0.451053032333184
0.451077622854969
error
3.416 105
2.259 105
2.003 106
39/41
Numerical Methods for Computational Science and Engineering
Newton interpolation
Algorithm: Newton polynomial interpolation
1. Construction: Construct the table of divided differences.
Extract the diagonal to obtain the coefficients cj , j = 0, 1, . . . , n.
2. Evaluation: Given an evaluation point x. Compute the value
p(x) =
n
X
j=0
cj
j1
Y
(x xi )
i=0
by Horners rule.
NumCSE, Lecture 16, Nov 10, 2014
40/41
Numerical Methods for Computational Science and Engineering
Newton interpolation
Algorithms comparison
Basis
name
j (x)
construction
cost
evaluation
cost
selling
feature
Monomial
xj
2 3
3n
2n
Lagrange
Lj (x)
n2
5n
simple
cj = yj
most stable
3 2
2n
2n
Newton
j1
Q
(x xi )
i=0
NumCSE, Lecture 16, Nov 10, 2014
adaptive
41/41
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
I
Error of polynomial interpolation
Chebyshev points
Interpolating derivatives
References
I
U. Ascher & C. Greif: Numerical methods. SIAM 2011.
Chapter 10.110.3.
NumCSE, Lecture 17, Nov 13, 2014
2/25
Numerical Methods for Computational Science and Engineering
Polynomial interpolation
Polynomial interpolation
I
Given n + 1 mutually distinct points xi , we are looking for a
polynomial p(x) such that
p(xi ) = f (xi ) = yi ,
i = 0, . . . , n.
(+)
Interpolation: p(z) is evaluated at some new value z 6= xi
inside the range of the interpolation points x0 , . . . , xn .
As we have n + 1 constraints to satisfy, we need n + 1 degrees
of freedom. Consider the n-th degree polynomial
pn (x) = a0 + a1 x + . . . + an1 x n1 + an x n .
The n + 1 coefficients ai are determined uniquely by (+).
NumCSE, Lecture 17, Nov 13, 2014
3/25
Numerical Methods for Computational Science and Engineering
Newton interpolation
Newton interpolation
For Newton interpolation we represent the interpolation polynomial
in a different way:
pn (x) := c0 + c1 (x x0 ) + c2 (x x0 )(x x1 ) +
+ cn (x x0 )(x x1 ) (x xn1 ).
The coefficient cj is determined by the j-th divided differences
cj = f [x0 , x1 , . . . , xj ],
NumCSE, Lecture 17, Nov 13, 2014
j = 0, . . . , n.
4/25
Numerical Methods for Computational Science and Engineering
Newton interpolation
Newton interpolation (cont.)
The divided differences are defined recursively,
f [xi ] := f (xi ),
f [xi , xi+1 , . . . , xi+k ] :=
i = 0, . . . , n.
f [xi+1 , xi+2 , . . . , xi+k ] f [xi , xi+1 , . . . , xi+k1 ]
.
xi+k xi
where k = 1, 2, . . . , n and i = 0, 1, . . . , n k.
NumCSE, Lecture 17, Nov 13, 2014
5/25
Numerical Methods for Computational Science and Engineering
Newton interpolation
Newton interpolation (cont.)
The divided difference are best arranged in a triangular array:
x0 f [x0 ] = c0
f [x0 , x1 ] = c1
x1 f [x1 ]
f [x0 , x1 , x2 ] = c2
f [x1 , x2 ]
x2 f [x2 ]
f [x0 , x1 , x2 , x3 ] = c3
f [x1 , x2 , x3 ]
f [x2 , x3 ]
x3 f [x3 ]
Here is the case n = 3.
NumCSE, Lecture 17, Nov 13, 2014
6/25
Numerical Methods for Computational Science and Engineering
Error of polynomial interpolation
Error expression
I
Assume yi = f (xi ), i = 0, 1, . . . , n. Interpolant pn (x).
Want to estimate error
en (x) = f (x) pn (x)
and see how it depends on the choice of n and the properties
of f
For some x 6 {xi }ni=0 pretend we are adding as new data
point (x, f (x)).
Using the properties of the Newton basis and divided
differences
f (x) = pn+1 (x) = pn (x) + f [x0 , . . . , xn , x]
NumCSE, Lecture 17, Nov 13, 2014
n
Y
(x xj )
j=0
7/25
Numerical Methods for Computational Science and Engineering
Error of polynomial interpolation
Error expression (cont.)
I
Rearranging
en (x) = f (x) pn (x) = f [x0 , . . . , xn , x]
n
Y
(x xj )
j=0
I
Let a = mini xi , b = maxi xi and assume x [a, b] (otherwise
pn (x) is extrapolating)
Relationship between divided differences and derivatives:
[a, b]
such that
f [x0 , x1 , . . . , xn , x] =
f (n+1) ()
(n + 1)!
Take upper bounds
NumCSE, Lecture 17, Nov 13, 2014
8/25
Numerical Methods for Computational Science and Engineering
Error of polynomial interpolation
Theorem: polynomial interpolation error
If pn interpolates f at the n + 1 points x0 , . . . , xn and f has
n+1 bounded derivatives on an interval [a, b] containing these
points, then for each x [a, b] there is a point = (x)
[a, b] such that
n
f (x) pn (x) =
f (n+1) () Y
(x xj )
(n + 1)!
j=0
By consequence
|en (x)| max
NumCSE, Lecture 17, Nov 13, 2014
atb
ken k
n
Y
max (s xj )
(n + 1)! asb
j=0
|f (n+1) (t)|
kf (n+1) k
(b a)n+1
(n + 1)!
9/25
Numerical Methods for Computational Science and Engineering
Error of polynomial interpolation
Example
Consider {(x0 , y0 ), (x1 , y1 )}.
So n = 1, hence n + 1 = 2.
1
ke1 k kf 00 k max |(s x0 )(s x1 )|.
asb
2
Maximum at
x0 +x1
2 ,
Thus,
NumCSE, Lecture 17, Nov 13, 2014
so maxs |(s x0 )(s x1 )| = 41 (x1 x0 )2 .
1
ke1 k (x1 x0 )2 kf 00 k .
8
10/25
Numerical Methods for Computational Science and Engineering
Error of polynomial interpolation
Example 10.5: Determine temperature
Maximum daily temperatures recorded every 3rd day during August
Day
Temp (C)
3
31.2
6
32.0
9
35.3
12
34.1
15
35.0
18
35.5
21
34.1
24
35.1
27
36.0
Data values are not monotonically decreasing/increasing.
Estimate maximum temperature at day x = 13 of that month.
First, construct cubic polynomial using 4 specific points near 13.
We choose x0 = 9, x1 = 12, x2 = 15, x3 = 18. So, we get
p3 (13) = 34.29.
For the linear interpolation of the two closest neighbors, at
x = 12, 15, we get p1 (13) = 34.4.
This provides some confidence that our calculations are probably
on the mark.
In summary, it was hot on that day, too.
NumCSE, Lecture 17, Nov 13, 2014
11/25
Numerical Methods for Computational Science and Engineering
Error of polynomial interpolation
About theorem on interpolation error
I
We do in general
not know the
n + 1-st derivative
of f .
Maybe we can
choose the
interpolation
points xi ?
Compare
equidistant with
Chebyshev points.
Here n = 8.
NumCSE, Lecture 17, Nov 13, 2014
12/25
Numerical Methods for Computational Science and Engineering
Error of polynomial interpolation
Minimizing error bound?
|en (x)| max
atb
n
Y
max (s xj )
(n + 1)! asb
j=0
|f (n+1) (t)|
Assume we can evaluate f (x) at any n + 1 points xi . What
should those be?
Knowing nothing more about the interpolated function f (x),
choose the n abscissae x i attempting to minimize
Q
maxs[a,b] nj=0 (s xj ) .
This leads to Chebyshev points over [1, 1]
2i + 1
xi = cos
, i = 0, . . . , n.
2(n + 1)
NumCSE, Lecture 17, Nov 13, 2014
13/25
Numerical Methods for Computational Science and Engineering
Error of polynomial interpolation
Chebyshev points
I
These points solve the min-max problem
=
min
max |(x x0 )(x x1 ) (x xn )|
x0 ,x1 ,...,xn 1x1
yielding the value = 2n .
I
This leads to the Chebyshev interpolation error bound
max |f (x) pn (x)|
1x1
I
2n (n
1
max |f (n+1) (t)|
+ 1)! 1t1
For a general interval [a, b], scale and translate [1, 1] onto
[a, b]
ba
x =a+
(t + 1), t [1, 1].
2
NumCSE, Lecture 17, Nov 13, 2014
14/25
Numerical Methods for Computational Science and Engineering
Error of polynomial interpolation
Example
Example 10.6: Runge example
Problems with polynomial interpolation are possible strong
oscillations of interpolating polynomials of high degree on
uniformly spaced nodes!
Runges example:
1
,
1 < x < 1.
1 + 25x 2
In this example, interpolation with Chebyshev points that cluster at
the ends of the interval are gives much better results than
interpolation with equidistant points.
f (x) =
NumCSE, Lecture 17, Nov 13, 2014
15/25
Numerical Methods for Computational Science and Engineering
Error of polynomial interpolation
Example
Example 10.6: Runge example (cont.)
Left: Interpolation with n = 4 (red) and n = 9 (green) equidistant
points. Right: n = 19.
NumCSE, Lecture 17, Nov 13, 2014
16/25
Numerical Methods for Computational Science and Engineering
Error of polynomial interpolation
Example
Example 10.6: Runge example (cont.)
Left: Interpolation with n = 4 (red) and n = 9 (green) Chebyshev
points. Right: n = 19.
NumCSE, Lecture 17, Nov 13, 2014
17/25
Numerical Methods for Computational Science and Engineering
Error of polynomial interpolation
Example
Example 10.7: A more difficult example
f (x) = e 3x sin(200x 2 )/(1 + 20x 2 ),
NumCSE, Lecture 17, Nov 13, 2014
0 < x < 1.
18/25
Numerical Methods for Computational Science and Engineering
Interpolating derivatives
Interpolating derivatives
We now admit also derivatives at the interpolation points.
f (0) = 1.5,
Hence
f 0 (0) = 1,
f (20) = 0.
{(xi , yi )} = {(0, 1.5), (0, 1), (20, 0)}.
Using the simplest basis: monomial, p(x) = c0 + c1 x + c2 x 2 .
Then
1.5 = p(0) = c0 ,
1 = p 0 (0) = c1 ,
0 = p(20) = c0 + 20c1 + 400c2 = c2 =
and p(x) = 1.5 + x
NumCSE, Lecture 17, Nov 13, 2014
(1.5 20)
21.5
=
400
400
21.5 2
400 x .
19/25
Numerical Methods for Computational Science and Engineering
Interpolating derivatives
Hermite interpolation
I
Derivatives at the interpolation points means that there may
be points z where we prescibe the function value f (z)
together with f 0 (z), f 00 (z), etc.
We can surprisingly still use divided differences.
Let xk be a double interpolation point. Then,
f (xk + h) f (xk )
f (xk + h) f (xk )
= lim
= f 0 (xk ).
h0
h0
(xk + h) xk
h
f [xk , xk ] = lim
I
More generally, we can write
1
f [xk , . . . , xk ] =
f (m1) (xk ).
| {z } (m 1)!
m-fold
Note relation to Taylor polynomial!
NumCSE, Lecture 17, Nov 13, 2014
20/25
Numerical Methods for Computational Science and Engineering
Interpolating derivatives
Examples
Hermite interpolation: example
Lets consider the following interpolation problem. Let
x0 , x1 , . . . , xn be disjoint points. We are looking for the polynomial
of degree 2n + 1 such that the 2n conditions are satisfied:
P2n+1 (xi ) = yi ,
0
P2n+1
(xi ) = yi0 ,
i = 0, . . . , n.
To indicate how we proceed we consider the case n = 1:
x0
x0
x1
x1
f [x0 ] = y0 = c0
f [x0 , x0 ] := y00 = c1
f [x0 ]
f [x0 , x0 , x1 ] = c2
f [x0 , x1 ]
f [x0 , x0 , x1 , x1 ] = c3
f [x1 ] = y1
f [x0 , x1 , x1 ]
f [x1 , x1 ] := y10
f [x1 ]
NumCSE, Lecture 17, Nov 13, 2014
21/25
Numerical Methods for Computational Science and Engineering
Interpolating derivatives
Examples
Hermite interpolation: example (cont.)
The interpolating polynomial has now quadratic terms:
p3 (x) = c0 + c1 (x x0 ) + c2 (x x0 )2 + c3 (x x0 )2 (x x1 ).
NumCSE, Lecture 17, Nov 13, 2014
22/25
Numerical Methods for Computational Science and Engineering
Interpolating derivatives
Examples
Hermite interpolation: numerial example
We want to determine the interpolant of the the function
f (x) = e x sin(5x),
with derivative
f 0 (x) = e x (sin(5x)+5 cos(5x))
in the interval [0, 1] and compare with Newton interpolation. We
choose n + 1 equidistant interpolation points. For n = 1 we get
(with 4 decimal digits)
0
0 = c0
5 = c1
7.6066 = c2
2.6066
1
2.6066
2.6066
11.4620 = c3
3.8554
1.2487
Thus, PH (x) = 5x 7.6066x 2 + 11.4620x 2 (x 1)
NumCSE, Lecture 17, Nov 13, 2014
23/25
Numerical Methods for Computational Science and Engineering
Interpolating derivatives
Examples
Hermite interpolation: numerial example (cont.)
Except for the coarsest
case the errors are about
equal for equal amount of
work.
n+1
2
3
4
5
Hermite error
1.8
0.077
0.0024
0.000043
NumCSE, Lecture 17, Nov 13, 2014
2n + 2
4
6
8
10
Newton error
0.40
0.038
0.0018
0.000044
24/25
Numerical Methods for Computational Science and Engineering
Interpolating derivatives
Examples
Example 10.10: Another interpolation example with
derivatives
ti
8.3
8.6
f (ti )
17.564921
18.505155
f 0 (ti )
3.116256
3.151762
f 00 (ti )
0.120482
(t0 , t1 , t2 , t3 , t4 ) = (8.3, 8.3, 8.3, 8.6, 8.6)
ti
8.3
8.3
8.3
8.6
8.6
f []
17.564921
17.564921
17.564921
18.505155
18.505155
NumCSE, Lecture 17, Nov 13, 2014
f [, ]
f [, , ]
f [, , , ]
f [, , , ]
3.116256
3.116256
3.130780
3.151762
0.060241
0.048413
0.069400
-0.039426
0.071756
0.370604
25/25
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
I
Piecewise polynomial interpolation
Spline interpolation
Parametric interpolation
References
I
U. Ascher & C. Greif: Numerical methods. SIAM 2011.
Chapter 11, Sections 11.1 11.4.
H.-R. Schwarz & N. K
ockler: Numerische Mathematik.
Teubner 2006. Section 3.2.
NumCSE, Lecture 18, Nov 17, 2014
2/40
Numerical Methods for Computational Science and Engineering
Introduction
Whats wrong with polynomial interpolation?
We are (still) given a collection of data samples {(xi , yi )}ni=0
looking for a function v (x) that satisfies
v (xi ) = yi ,
I
I
i = 0, 1, . . . , n.
kf (n+1) k
is not.
(n + 1)!
High order polynomials tend to oscillate unreasonably.
The error bound may not be small if
Data often suggest only piecewise smooth, whereas
polynomials are infinitely differentiable.
No locality: changing any one data value may drastically alter
the entire interpolant.
NumCSE, Lecture 18, Nov 17, 2014
3/40
Numerical Methods for Computational Science and Engineering
Piecewise polynomial interpolation
Piecewise polynomial interpolation
I
Divide [a, b] into subintervals
a = t0 < t1 < t2 < < tr = b
where ti are the breakpoints, or knots.
Construct interpolant
v (x) = si (x),
ti x ti+1 ,
i = 0, . . . , r 1
where each si (x) is a polynomial of low degree (e.g., piecewise
cubic: m = 3).
I
As before v (x) must satisfy the interpolation conditions. In
addition, we require a global smoothness property.
NumCSE, Lecture 18, Nov 17, 2014
4/40
Numerical Methods for Computational Science and Engineering
Piecewise polynomial interpolation
A piecewise polynomial function with break points ti = i, i = 0, 1, . . . , 6.
NumCSE, Lecture 18, Nov 17, 2014
5/40
Numerical Methods for Computational Science and Engineering
Piecewise polynomial interpolation
ti and xi
I
Distinguish between abscissae xi where data are specified
a x0 x1 xn1 xn b
and knots (break points) of the piecewise polynomial
a = t0 < t1 < t2 < < tr = b
For future purposes, set
h = max (ti ti1 ).
1ir
Rely on h being small (while b a is not small) for quality
approximation.
NumCSE, Lecture 18, Nov 17, 2014
6/40
Numerical Methods for Computational Science and Engineering
Local interpolants
Local interpolants: C 0 -linear and C 1 -cubic
I
Local piecewise polynomial interpolants are simply
constructed from local polynomial interpolation on each
subinterval [ti1 , ti ].
Some global smoothness conditions are automatically satisfied.
Interpolation error bounds are easily derived from polynomial
interpolation.
The interpolant depends on the data only locally.
Limitation: dont have control over how much global
smoothness is easily achieved.
NumCSE, Lecture 18, Nov 17, 2014
7/40
Numerical Methods for Computational Science and Engineering
Piecewise linear interpolation
Piecewise linear interpolation
The simplest continuous interpolant.
I
Assuming {xi }ni=0 are distinct and ordered, identify ti = xi ,
yi = f (xi ).
Determine si (x) as the straight line interpolant at xi and xi+1 .
Thus, m = 1 and
v (x) = si (x) = f (xi ) + f [xi , xi+1 ](x xi ),
with xi x xi+1 , 0 i n 1.
Inherently, v (x) is continuous but not smoother than that.
Shape preservation. No new minima or maxima are
invented by v (x) for f (x).
Matlabs command plot uses this interplant by default.
NumCSE, Lecture 18, Nov 17, 2014
8/40
Numerical Methods for Computational Science and Engineering
Piecewise linear interpolation
Data and broken line interpolation
NumCSE, Lecture 18, Nov 17, 2014
9/40
Numerical Methods for Computational Science and Engineering
Piecewise linear interpolation
Hat functions: a compact basis for piecewise linears
x xi1
, xi1 < x < xi
xi xi1
i = x xi+1 , xi < x < xi+1
xi xi+1
0,
otherwise
v (x) =
n
X
f (xi )i (x)
i=0
(cf. Lagrange interpolation)
NumCSE, Lecture 18, Nov 17, 2014
10/40
Numerical Methods for Computational Science and Engineering
Piecewise linear interpolation
Polynomial error estimate and bound
Recall:
I
Assume x [ti , ti+1 ]. Consider polynomial interpolation of
degree m with abscissae x0 , x1 , . . . , xm [ti , ti+1 ].
Then
|em (x)| max
atb
NumCSE, Lecture 18, Nov 17, 2014
Y
m
max (s xj )
(m + 1)! asb
j=0
|f (m+1) (t)|
11/40
Numerical Methods for Computational Science and Engineering
Piecewise linear interpolation
Error analysis for piecewise linear interpolation
I
Linear polynomials: m = 1, ti = xi , so for xi x xi+1
|v (x) f (x)|
=
max
xi txi+1
max
xi txi+1
|f 00 (t)|
max |(s xi )(s xi+1 )|
2 xi sxi+1
|f 00 (t)| |xi+1 xi |2
2
4
Recall notation h = maxi (ti+1 ti ). Then
kf v k
h2 00
kf k .
8
The error is O(h2 ). If we have n + 1 equidistant data points,
then h = (b a)/n and the error is O( n12 ).
NumCSE, Lecture 18, Nov 17, 2014
12/40
Numerical Methods for Computational Science and Engineering
Piecewise cubic Hermite Interpolation
Piecewise cubic Hermite interpolation
I
Most popular in applications other than plot.
v (x) = si (x) = ai + bi (x ti ) + ci (x ti )2 + di (x ti )3 ,
ti x ti+1 ,
i = 0, . . . , r 1.
We have 4r coefficients. Of these, 2r are used to satisfy
interpolation of f values as in piecewise linear case. What
about the remaining 2r coefficients?
Use either to interpolate more data or to require higher degree
of smoothness
NumCSE, Lecture 18, Nov 17, 2014
13/40
Numerical Methods for Computational Science and Engineering
Hermite Interpolation
Piecewise cubic Hermite interpolation
Cleanest piecewise cubic.
I Values of f 0 (ti ) are also provided. So
(x0 , x1 , x2 , . . . , xn1 , xn ) = (t0 , t0 , t1 , t1 , . . . , tr , tr ).
I
I
I
Each cubic si (x), 0 i r 1 satisfies
si (ti ) = f (ti ),
si (ti+1 ) = f (ti+1 )
si0 (ti )
si0 (ti+1 ) = f 0 (ti+1 ).
= f (ti ),
Obtain v (x) C 1 [a, b]. Construction entirely local!
Error bound obtained directly from polynomial:
|f (x) v (x)|
NumCSE, Lecture 18, Nov 17, 2014
h4
max |f 0000 (s)|
384 asb
14/40
Numerical Methods for Computational Science and Engineering
Hermite Interpolation
Piecewise cubic Hermite basis functions corresponding to an
interior node.
Knots t = [0, 0.5, 1.5, 2, 2.5]
NumCSE, Lecture 18, Nov 17, 2014
15/40
Numerical Methods for Computational Science and Engineering
Hermite Interpolation
Computational issues
Piecewise cubic polynomials: computational issues
Lets look at an interval
i = 0, 1, . . . , n 1
[xi , xi+1 ],
In [xi , xi+1 ], s(x) is a cubic polynomial that we call si .
(Note that i does not refer here to the degree of the polynomial but to
the number of the interval!)
The polynomial si must satisfy the following conditions:
si (xi ) = yi ,
si (xi+1 ) = yi+1
si0 (xi )
si0 (xi+1 )
yi0 ,
NumCSE, Lecture 18, Nov 17, 2014
0
yi+1
interpolate
continuous derivative
16/40
Numerical Methods for Computational Science and Engineering
Hermite Interpolation
Computational issues
Piecewise cubic polynomials: computational issues (cont.)
To simplify computations we make a change of variables
t :=
x xi
hi
with
hi := xi+1 xi .
t is a local variable in the i-th interval. Set
qi (t) := si (xi + thi ).
The derivative is
qi0 (t) = hi si0 (xi + thi ),
and thus
qi (0) = yi ,
qi (1) = yi+1
qi0 (0)
0
qi0 (1) = hi yi+1
NumCSE, Lecture 18, Nov 17, 2014
hi yi0 ,
17/40
Numerical Methods for Computational Science and Engineering
Hermite Interpolation
Computational issues
Piecewise cubic polynomials: computational issues (cont.)
We can write
0
qi (t) = yi H1 (t) + yi+1 H2 (t) + hi yi0 H3 (t) + hi yi+1
H4 (t)
(1)
with
H1 (t) = 1 3t 2 + 2t 3
H3 (t) = t 2t 2 + t 3
H2 (t) = 3t 2 2t 3
H4 (t) = t 2 + t 3
qi could be evaluated by the expression (1); however, a scheme
based on Newton interpolation is more efficient.
NumCSE, Lecture 18, Nov 17, 2014
18/40
Numerical Methods for Computational Science and Engineering
Hermite Interpolation
Computational issues
Piecewise cubic polynomials: computational issues (cont.)
Local shape functions of piecewise cubic Hermite interpolation.
H1 (t) := ( ti+1hit ),
i
H2 (t) := ( tt
hi )
H3 (t) := hi ( ti+1hit ),
i
H4 (t) := hi ( tt
hi )
( ) := 3 2 2 3 ,
( ) := 3 2 ,
hi := ti+1 ti
NumCSE, Lecture 18, Nov 17, 2014
19/40
Numerical Methods for Computational Science and Engineering
Hermite Interpolation
Computational issues
Piecewise cubic polynomials: computational issues (cont.)
Using a scheme based on Newton interpolation we can write
0
yi = c 0
[yi , yi ] = hi yi0 = c1
yi
[yi , yi , yi+1 ] = c2
[yi , yi+1 ]
yi+1
[yi , yi , yi+1 , yi+1 ] = c3
[yi , yi+1 , yi+1 ]
0
[yi+1 , yi+1 ] = hi yi+1
yi+1
In this way we obtain the coefficients c0 , c1 , c2 and c3 and we can
compute
qi (t) = c0 + (c1 + (c2 + c3 (t 1))t)t.
NumCSE, Lecture 18, Nov 17, 2014
20/40
Numerical Methods for Computational Science and Engineering
Hermite Interpolation
Computational issues
Piecewise cubic polynomials: computational issues (cont.)
To interpolate with v (x) = s(x) for a value z, x0 z xn , we
proceed in three steps:
1. Determine the interval which contains z, i.e. compute the
index i for which xi z < xi+1 .
2. Compute the local variable t = (z xi )/(xi+1 xi ).
3. Evaluate s(z) = qi (t) by the Newton scheme.
To find the interval which contains z, we can use a binary search.
This is done in the following (homegrown) Matlab function
c1hermite.
NumCSE, Lecture 18, Nov 17, 2014
21/40
Numerical Methods for Computational Science and Engineering
Hermite Interpolation
Computational issues
Piecewise cubic polynomials: computational issues (cont.)
function g = c1hermite(x,y,ys,z);
% C1HERMITE interpolates at z the data (x,y,ys)
%
with a piecewise cubic polynomial.
n = length(x);
a = 1; b = n; i = a;
while a+1 ~= b,
i = floor((a+b)/2);
if x(i)<z, a = i; else b = i; end
end
i = a; h = (x(i+1)-x(i));
t = (z-x(i))/h;
c0 = y(i);
c1 = h*ys(i); tmp = y(i+1)-c0; % = f[x(i),x(i+1)]
c2 = tmp-c1; tmp = h*ys(i+1)-tmp; % = f[x(i),x(i+1),x(i+1)]
c3 = tmp-c2;
g = c0+(c1+(c2+c3*(t-1))*t)*t;
NumCSE, Lecture 18, Nov 17, 2014
22/40
Numerical Methods for Computational Science and Engineering
Piecewise cubic spline interpolation
Piecewise cubic spline interpolation
Most popular general-purpose interpolant.
I
Given only function values, {xi , f (xi )}, x0 < x1 < < xn .
Set ti = xi , r = n and use 2r conditions for C 0 interpolation
as before.
Use 2(r 1) additional coefficients to impose a global C 2
smoothness. (Sufficient to create smooth-looking curves.)
si (xi )
0
si (xi+1 )
= f (xi ),
=
0
(xi+1 ),
si+1
si (xi+1 ) = f (xi+1 ),
si00 (xi+1 )
00
si+1
(xi+1 ),
i = 0, . . . , n1,
i = 0, . . . , n2.
We do not prescribe yi0 anymore! In fact, we want to
determine the yi0 such that the above equations hold.
I
What to do with the remaining two coefficients?
NumCSE, Lecture 18, Nov 17, 2014
23/40
Numerical Methods for Computational Science and Engineering
Piecewise cubic spline interpolation
Matching spline si and si1 at xi .
NumCSE, Lecture 18, Nov 17, 2014
24/40
Numerical Methods for Computational Science and Engineering
Piecewise cubic spline interpolation
Construction of C 2 spline
We have v (x) = si (x) for xi x xi+1 , i = 0, . . . , n 1
si (x) = ai + bi (x xi ) + ci (x xi )2 + di (x xi )3
si0 (x) = bi + 2ci (x xi ) + 3di (x xi )2
si00 (x) = 2ci + 6di (x xi )
Clearly,
ai = f (xi ),
i = 0, . . . , n 1
(2)
Continuity of v , i.e., si (xi+1 ) = f (xi+1 ):
ai + hi bi + hi2 ci + hi3 di = f (xi+1 ),
NumCSE, Lecture 18, Nov 17, 2014
hi = xi+1 xi .
bi + hi ci + hi2 di = f [xi , xi+1 ].
(3)
25/40
Numerical Methods for Computational Science and Engineering
Piecewise cubic spline interpolation
Construction of C 2 spline (cont.)
0 (x
Continuity of v 0 , i.e. si0 (xi+1 ) = si+1
i+1 ):
bi + 2hi ci + 3hi2 di = bi+1 ,
i = 0, . . . , n 2.
(4)
00 (x
Continuity of v 00 , i.e. si00 (xi+1 ) = si+1
i+1 ):
ci + 3hi di = ci+1 ,
i = 0, . . . , n 2.
(5)
At this point we have 4n 2 equations for 4n unknowns.
Now we express the bi and di in terms of ci . (3) and (5) give
di =
ci+1 ci
,
3hi
bi = f [xi , xi+1 ]
NumCSE, Lecture 18, Nov 17, 2014
i = 0, . . . , n 2,
hi
(2ci + ci+1 ),
3
i = 0, . . . , n 2.
26/40
Numerical Methods for Computational Science and Engineering
Piecewise cubic spline interpolation
Construction of C 2 spline (cont.)
Writing (4) for interval i 1 gives
2
bi1 + 2hi1 ci1 + 3hi1
di1 = bi ,
i = 1, . . . , n 1.
Substituting the expressions for bi and di we get
hi1 ci1 + 2(hi1 + hi )ci + hi ci+1 = 3(f [xi , xi+1 ] f [xi1 , xi ]), (6)
for i = 1, . . . , n 1.
NumCSE, Lecture 18, Nov 17, 2014
27/40
Numerical Methods for Computational Science and Engineering
Piecewise cubic spline interpolation
Construction of C 2 spline (cont.)
In matrix notation we get a linear system of n 1 equations with
n + 1 unknowns
A c = ,
()
with c = (c0 , c1 , . . . , cn )T the vector of unknowns and
A=
h0
2(h0 + h1 )
h1
h1
2(h1 + h2 )
..
.
h2
..
.
hn2
..
.
2(hn2 + hn1 ) hn1
a tridiagonal (rectangular) matrix and
i = 3(f [xi , xi+1 ] f [xi1 , xi ]),
i = 1, . . . , n 1.
Need two more equations to determine the ci uniquely
= boundary conditions.
NumCSE, Lecture 18, Nov 17, 2014
28/40
Numerical Methods for Computational Science and Engineering
Piecewise cubic spline interpolation
Natural boundary conditions
Boundary conditions
1. Natural boundary conditions:
00
s000 (x0 ) = sn1
(xn ) = 0
c0 = cn = 0.
First/last column of A can be deleted. n n linear system of
equations with a tridiagonal matrix. The matrix is symmetric
positive definite.
Leads to lower accuracy of O(h2 ).
NumCSE, Lecture 18, Nov 17, 2014
29/40
Numerical Methods for Computational Science and Engineering
Piecewise cubic spline interpolation
Clamped plate
Clamped plate
2. Clamped plate
Derivatives of f are prescribed:
b0 = f 0 (x0 ),
bn = f 0 (xn ).
Two additional equations in ():
2h0 c0 + h0 c1 = 3(f [x0 , x1 ] f 0 (x0 )),
hn1 cn1 + 2hn1 cn = 3(f 0 (xn ) f [xn1 , xn ]).
Accuracy O(h4 ), good for theory, but requires extra data.
NumCSE, Lecture 18, Nov 17, 2014
30/40
Numerical Methods for Computational Science and Engineering
Piecewise cubic spline interpolation
Not-a-knot condition
Not-a-knot condition
3. Not-a-knot condition of de Boor: The two polynomials in the
first/last two intervals shall be the same:
s0 (x) s1 (x)
and sn2 (x) sn1 (x).
Since spline v (x) already in C 2 it suffices to enforce that
s0000 (x1 ) = s1000 (x1 )
000
000
and sn2
(x1 ) = sn1
(x1 )
or
d0 = d1 ,
dn2 = dn1 .
The tridiagonal linear system is not SPD but diag. dominant.
Accuracy O(h4 ). This option used in practice.
NumCSE, Lecture 18, Nov 17, 2014
31/40
Numerical Methods for Computational Science and Engineering
Piecewise cubic spline interpolation
Not-a-knot condition
The Runge example using cubic spline interpolation
Not-a-knot cubic
spline interpolation
for the Runge
Example at 20
equidistant points.
pp = csape(x,y,not-a-knot);
NumCSE, Lecture 18, Nov 17, 2014
fnplt(pp)
32/40
Numerical Methods for Computational Science and Engineering
Piecewise cubic spline interpolation
Summary
Obtaining the cubic spline
I
Form and solve tridiagonal system of size n(1).
Construction cost is linear in n, but approximation is not
entirely local.
Still, much better than polynomial interpolation.
Interpolant
Piecewise
constant
Broken
line
Piecewise
cubic Hermite
Spline
(not-a-knot)
NumCSE, Lecture 18, Nov 17, 2014
Local?
Order
Smooth?
yes
bounded
yes
C0
yes
C1
not quite
C2
Selling features
Accommodates
general f
Simple, max and min
at data values
Elegant and
accurate
Accurate, smooth,
requires only f data
33/40
Numerical Methods for Computational Science and Engineering
Shape preservation
Shape preservation
Interpolation that preserves monotonicity and shape of the data.
Matlab function:
v = pchip(t,y,x);
t:
y:
x:
v:
Sampling points
Sampling values
Evaluation points xi
Vector s(xi )
Local interpolation operator
! Nonlinear interpolation operator
This is cubic Hermite interpolation with carefully chosen slopes at
the break points.
NumCSE, Lecture 18, Nov 17, 2014
34/40
Numerical Methods for Computational Science and Engineering
Parametric curves
Parametric curves
We would like to generate a curve through the n given points in
the plane (xi , yi ), i = 0, 1, . . . , n.
Note: The numbering of the points is crucial; reordering them will
give us another curve.
Plane curves are represented by parametric functions
(x(s), y (s))
with
s0 s sn .
We can interpret the given points as function values for some (yet
to be determined) parametrization
x(si ) = xi ,
NumCSE, Lecture 18, Nov 17, 2014
y (si ) = yi
i = 0, 1, . . . , n.
35/40
Numerical Methods for Computational Science and Engineering
Parametric curves
Parametric curves (cont.)
Sequence {si } can be chosen arbitrarily, but it must be strictly
monotonically increasing: si < si+1 .
Often parameter s is chosen to be arc length of the polygonal line
s0 = 0
si+1 = si +
q
(xi+1 xi )2 + (yi+1 yi )2 ,
i = 0, . . . , n 1
After computing the {si } we have two independent sets of data
s
x
s1 s2 . . . sn
x1 x2 . . . xn
s
y
s1 s2 . . . sn
y1 y2 . . . yn
and
NumCSE, Lecture 18, Nov 17, 2014
36/40
Numerical Methods for Computational Science and Engineering
Parametric curves
Parametric curves (cont.)
Both functions x(s) and y (s) can be interpolated with any of the
variants for spline interpolation.
For closed curves it is important to use the periodic boundary
conditions to avoid the cusp at the endpoints.
Example
For the points
x
y
1.31 2.89 5.05 6.67 3.12 2.05 0.23 3.04 1.31
7.94 5.50 3.47 6.40 3.77 1.07 3.77 7.41 7.94
we obtain the curve of the following Figure by using the Matlab
function csape.
NumCSE, Lecture 18, Nov 17, 2014
37/40
Numerical Methods for Computational Science and Engineering
Parametric curves
Parametric curves (cont.)
x =[1.31, 2.89, 5.05, 6.67, 3.12, 2.05, 0.23, 3.04, 1.31];
y =[7.94, 5.50, 3.47, 6.40, 3.77, 1.07, 3.77, 7.41, 7.94];
xy=[x; y]
n=length(x);
t(1)=0;
for i=2:n, t(i)=t(i-1) + norm(xy(:,i)-xy(:,i-1)); end
infty = csape(t, xy, periodic);
fnplt(infty, 2)
hold on
plot(xy(1,:),xy(2,:),ro,LineWidth,2)
hold off
NumCSE, Lecture 18, Nov 17, 2014
38/40
Numerical Methods for Computational Science and Engineering
Parametric curves
Parametric curves (cont.)
NumCSE, Lecture 18, Nov 17, 2014
39/40
Numerical Methods for Computational Science and Engineering
Introduction
Introduction
Computer Aided Geometric Design (CAGD) applications make
extensive use of parametric curve fitting.
These applications require
I
rapid generation of curves, and
easy modification of curves. Modifications should be local.
From second criterion: piecewise (mostly cubic) polynomials
Bezier curves are parametric curves frequently used in computer
graphics and related fields.
Bezier curves were publicized 1962 by the French engineer Bezier
designed automobile bodies in the 1960s. De Casteljau suggested
a numerically stable recursive algorithm to evaluate Bezier curves.
NumCSE, Lecture 19, Nov 20, 2014
2/39
Numerical Methods for Computational Science and Engineering
Introduction
Topics today
I
Bezier polynomials
Bezier curves
2-dimensional tensor splines
References
I
U. Ascher & C. Greif: Numerical methods. SIAM 2011.
Chapter 11, Sections 11.1 11.4.
H.-R. Schwarz & N. K
ockler: Numerische Mathematik.
6th edition. Teubner 2006. Section 3.5.
NumCSE, Lecture 19, Nov 20, 2014
3/39
Numerical Methods for Computational Science and Engineering
Bernstein polynomials
Bernstein polynomials
From the binomial theorem we have for any n
n
X
n
n
1 = ((1 t) + t) =
(1 t)ni t i .
i
i=0
The Bernstein polynomials of degree n on the interval [0, 1] are
defined as
n
Bi,n (t) :=
(1 t)ni t i , i = 0, 1, . . . , n.
(1)
i
They are positive in [0, 1] and form a partition of unity:
NumCSE, Lecture 19, Nov 20, 2014
n
X
Bi,n (t) = 1.
i=1
4/39
Numerical Methods for Computational Science and Engineering
Bernstein polynomials
Bernstein polynomials (cont.)
The Bernstein polynomials of degree 1 are
B0,1 (t) = (1 t),
B1,1 (t) = t.
The Bernstein polynomials of degree 2 are
B0,2 (t) = (1 t)2 ,
B1,2 (t) = 2(1 t)t,
B2,2 (t) = t 2 .
The Bernstein polynomials of degree 3 are
B0,3 (t) = (1 t)3 ,
B1,3 (t) = 3(1 t)2 t,
B2,3 (t) = 3(1 t)t 2 ,
B3,3 (t) = t 3 .
NumCSE, Lecture 19, Nov 20, 2014
5/39
Numerical Methods for Computational Science and Engineering
Bernstein polynomials
Bernstein polynomials (cont.)
Bernstein polynomials of degree 6.
NumCSE, Lecture 19, Nov 20, 2014
6/39
Numerical Methods for Computational Science and Engineering
Bernstein polynomials
Bernstein polynomials (cont.)
To obtain the Bernstein polynomials of degree n on an arbitrary
interval [a, b] we use the transformation
u [a, b] 7 t [0, 1] :
t=
ua
ba
to get
Bi,n (u; a, b) := Bi,n
NumCSE, Lecture 19, Nov 20, 2014
ua
ba
1
n
=
(b u)ni (u a)i .
n
(b a) i
7/39
Numerical Methods for Computational Science and Engineering
Bernstein polynomials
Bernstein polynomials (cont.)
Almost evident properties of Bernstein polynomials:
1. t = 0 is an i-fold zero of Bi,n
2. t = 1 is an n i-fold zero of Bi,n
3. Bi,n (t) = Bni,n (1 t) (symmetry)
4. 0 Bi,n (t) 1 for all t [0, 1],
Bi,n (t) > 0 for all t (0, 1).
5. In the interval [0, 1], Bi,n (t) has precisely one maximum at
t = i/n.
NumCSE, Lecture 19, Nov 20, 2014
8/39
Numerical Methods for Computational Science and Engineering
Bernstein polynomials
Bernstein polynomials (cont.)
Theorem (recurrence relations)
Bernstein polynomials satisfy the following recurrences:
B0,n (t) = (1 t)B0,n1 (t),
Bi,n (t) = t Bi1,n1 (t) + (1 t) Bi,n1 (t),
i = 1, . . . , n 1,
()
Bn,n (t) = tBn1,n1 (t).
NumCSE, Lecture 19, Nov 20, 2014
9/39
Numerical Methods for Computational Science and Engineering
Bernstein polynomials
Bernstein polynomials (cont.)
Theorem (linear independence)
The polynomials {Bi,n (t)}ni=0 are linearly independent and form a
basis of Pn .
In fact, every element of the monomial basis can be written as
k
nk
t = t ((1 t) + t)
nk
X
j=0
n
X
r =k
nk
r k
n
r
nk
(1 t)nkj t j+k
j
n
(1 t)nr t r
r
|
{z
}
Brn (t)
In particular, t =
n
P
r =k
NumCSE, Lecture 19, Nov 20, 2014
r
n Brn (t).
10/39
Numerical Methods for Computational Science and Engineering
Bernstein polynomials
Bernstein polynomials (cont.)
By consequence: any polynomial p Pn can be written as
p(t) =
n
X
i Bi,n (t).
(+)
i=0
Definition
I
Eq. (+) is called Bezier representation of p.
The i are the Bezier coefficients.
The points (i/n, i ) are called Bezier points.
The connection of the Bezier points is called Bezier polygon.
NumCSE, Lecture 19, Nov 20, 2014
11/39
Numerical Methods for Computational Science and Engineering
Bernstein polynomials
Example
Bezier points: (0, 0), (1/3, 1), (2/3, 1), (1, 1)
Bezier polynomial p(t) = B13 B23 + B33 = t 3 + 3t 2 3t.
NumCSE, Lecture 19, Nov 20, 2014
12/39
Numerical Methods for Computational Science and Engineering
Bernstein polynomials
Example (cont.)
Theorem
I
The graph of the Bezier polynomials is in the convex hull of
the Bezier points.
The graph of a polynomial and its Bezier polygon meet at the
end points t = 0 and t = 1 and have equal slopes.
In fact:
0
p (t) =
n
X
i=0
n d
[(1 t)ni t i ]
i
i dt
= 0 [n(1 t)n1 ] + 1 n[(n 1)(1 t)n2 t + (1 t)n1 ] + . . .
p 0 (0) = n(1 0 )
NumCSE, Lecture 19, Nov 20, 2014
13/39
Numerical Methods for Computational Science and Engineering
Casteljau algorithm
Casteljau algorithm
The Casteljau algorithm is an algorithm to compute value of the
Bezier polynomial. It exploits the recurrences ().
Given the Bezier polynomial p(t) =
n
P
i Bi,n (t). Then
i=0
p(t) =
n
X
i Bi,n (t)
i=0
= 0 (1 t)B0,n1 (t) +
n1
X
i [t Bi1,n1 (t) + (1 t) Bi,n1 (t)]
i=1
+ n tBn1,n1 (t)
=
n1
X
[i (1 t) + i+1 t]Bi,n1 (t).
i=0
NumCSE, Lecture 19, Nov 20, 2014
14/39
Numerical Methods for Computational Science and Engineering
Casteljau algorithm
Casteljau algorithm (cont.)
p(t) =
n
X
i=0
n1
X
i Bi,n (t)
[i (1 t) + i+1 t]Bi,n1 (t)
i=0
n2
X
n1
X
(1)
i Bi,n1 (t)
i=0
(2)
i Bi,n2 (t)
i=0
= =
(n)
= 0
(k)
with i
(k1)
= i
NumCSE, Lecture 19, Nov 20, 2014
(k1)
(1 t) + i+1 t.
15/39
Numerical Methods for Computational Science and Engineering
Casteljau algorithm
Casteljau algorithm (cont.)
Geometric interpretation:
(0)
(0)
Set (xi , yi ) = (i/n, i ), i = 0, . . . , n
The point (xi , yi ) with
(1)
(1)
xi
(1)
(0)
(0)
= (1t)xi +t xi+1 =
i +t
,
n
(1)
yi
(1)
= i
(0)
(0)
= (1t)yi +t yi+1
lies on the straight line between ( ni , i ) and ( i+1
n , i+1 )
(on the Bezier polygon).
I
(2)
(2)
xi
I
(2)
The point (xi , yi ) with
(1)
= (1 t)xi
(1)
+ txi+1 ,
(2)
yi
(1)
= (1 t)yi
(1)
+ yi+1 t
etc.
NumCSE, Lecture 19, Nov 20, 2014
16/39
Numerical Methods for Computational Science and Engineering
Casteljau algorithm
Casteljau algorithm (cont.)
NumCSE, Lecture 19, Nov 20, 2014
17/39
Numerical Methods for Computational Science and Engineering
B
ezier curves
Bezier curves
Bezier curves are polynomial curves represented in the basis of
Bernstein polynomials:
Pn (t) =
n
X
Bi,n (t)Pi ,
t [0, 1]
i=0
where Bi,n (t) is the Bernstein polynomial of degree n. Each
component of Pn (t) is a Bezier polynomial!
The points Pi are called control points of the Bezier curve.
Given 2 points P0 and P1 the Bezier curve of degree 1 is the
straight line
P1 (t) = (1 t)P0 + tP1
joining the two points.
NumCSE, Lecture 19, Nov 20, 2014
18/39
Numerical Methods for Computational Science and Engineering
B
ezier curves
Bezier curves (cont.)
Given three points P0 , P1 and P2 the Bezier curve of degree 2 is
P2 (t) = (1 t)2 P0 + 2(1 t)tP1 + t 2 P2 .
P2 (0) = P0 and P2 (1) = P2 so that the curve interpolates the end
points. However, P1 is not part of the curve.
Furthermore:
P02 (0) = 2P0 + 2P1 ,
P02 (1) = 2P1 + 2P2 ,
such that the derivative of the Bezier curve is proportional to that
of the Bezier polygon through the control points.
NumCSE, Lecture 19, Nov 20, 2014
19/39
Numerical Methods for Computational Science and Engineering
B
ezier curves
Bezier curves (cont.)
Theorem
The set of points of a Bezier curve
(
n
X
P(t) =
Bi,n (t)Pi ,
)
t [0, 1]
i=0
is contained in the convex hull of the Bezier points P0 , P1 , , Pn .
Theorem
For the end points of a Bezier curve we have
P(0) = P0 ,
0
P (0) = n(P1 P0 ),
NumCSE, Lecture 19, Nov 20, 2014
P(1) = Pn
P0 (1) = n(Pn Pn1 ).
20/39
Numerical Methods for Computational Science and Engineering
B
ezier curves
Example
Example
The Bezier curve of
degree three defined by
the control points
(2, 1), (3, 8), (8, 7), (10, 3)
NumCSE, Lecture 19, Nov 20, 2014
21/39
Numerical Methods for Computational Science and Engineering
B
ezier curves
Algorithm of de Casteljau
Algorithm of de Casteljau
The following Matlab function bezier computes a point on the
Bezier curve with the de Casteljau algorithm.
function [y] = bezier(t,P)
% BEZIER evaluates the Bezier polynomial defined
%
by the control points P for parameter t
%
using the de Casteljau scheme
[m,n] = size(P); n=n-1;
T = zeros(m,n+1,n+1);
for i = 0:n
T(:, i+1,1) = P(:,i+1);
for j = 1:i
T(:,i+1,j+1) = (1-t)*T(:,i,j) + t*T(:,i+1,j);
end;
NumCSE, Lecture 19, Nov 20, 2014
22/39
Numerical Methods for Computational Science and Engineering
B
ezier curves
Algorithm of de Casteljau
Algorithm of de Casteljau (cont.)
end;
y = T(:,n+1, n+1);
Here, a 3-dimensional Matlab array T is used to represent the
Casteljau scheme. For the control points of the previous example
P =
2
1
3
8
8
7
10
3
Calling U = bezier(0.5,P) gives the following:
NumCSE, Lecture 19, Nov 20, 2014
23/39
Numerical Methods for Computational Science and Engineering
B
ezier curves
Algorithm of de Casteljau
Algorithm of de Casteljau (cont.)
T(:,:,1) =
2
3
1
8
8
7
10
3
T(:,:,2) =
0
0
2.5000
4.5000
5.5000
7.5000
9.0000
5.0000
T(:,:,3) =
0
0
0
0
4.0000
6.0000
7.2500
6.2500
T(:,:,4) =
0
0
0
0
0
0
5.6250
6.1250
NumCSE, Lecture 19, Nov 20, 2014
24/39
Numerical Methods for Computational Science and Engineering
B
ezier curves
Algorithm of de Casteljau
Algorithm of de Casteljau (cont.)
U =
5.6250
6.1250
The point U = (5.6250, 6.1250) computed for t = 0.5 is marked on
the Bezier curve of degree 3 in the previous example/figure by .
NumCSE, Lecture 19, Nov 20, 2014
25/39
Numerical Methods for Computational Science and Engineering
B
ezier curves
Algorithm of de Casteljau
Efficient evaluation of P(t)
We have
B0,n (t)
n
B1,n (t)
X
P(t) =
Bi,n (t)Pi = [P0 , P1 , . . . , Pn ] .
..
i=0
Bn,n (t)
Let Bi,n (t) =
Pn
j=0 bij t
nj .
Then,
tn
t n1
P(t) = [P0 , P1 , . . . , Pn ]Bn . ,
..
1
NumCSE, Lecture 19, Nov 20, 2014
Bn = ((bij ))
26/39
Numerical Methods for Computational Science and Engineering
B
ezier curves
Algorithm of de Casteljau
Efficient evaluation of P(t) (cont.)
Numerical example continued:
P3 (t) =
x(t)
y (t)
NumCSE, Lecture 19, Nov 20, 2014
3
t
3 3 1
1
t 2
3
6
3
0
2 3 8 10
=
3
0 0 t
1 8 7 3 3
1
1
0
0 0
3
t2
7
12 3 2
t
=
5 24 21 1
t
1
7
12
3
2
=
t+
t+
t+
5
24
21
1
27/39
Numerical Methods for Computational Science and Engineering
Segmented B
ezier curves
Segmented Bezier curves
I
In real applications Bezier curves can usually not be generated
by a single curve. They are composed of segments.
Continuity is easy to achieve: The last control point of one
segment, say j, has to be equal to the first control point of
the next segment j + 1.
The continuity of the derivative requires that the second to
last control point of segment j, the second control point of
segment j + 1 and the control point common to segments j
and j + 1 lie on a line.
The distances of these points is determined by the length of
the segments.
NumCSE, Lecture 19, Nov 20, 2014
28/39
Numerical Methods for Computational Science and Engineering
Segmented B
ezier curves
Segmented Bezier curves (cont.)
I
Let the number of segments be m.
On each segment there is a Bezier curve of equal degree n.
The whole curve is parametrized by parameter u:
u0 < u1 < u2 < < um1 < um .
On the jth segment [uj1 , uj ] the curve can be written as
Pj (u) =
n
X
Pij Bin (u; uj1 , uj ),
u [uj1 , uj ].
i=0
where P0j , P1j , . . . , Pnj are the Bezier points.
NumCSE, Lecture 19, Nov 20, 2014
29/39
Numerical Methods for Computational Science and Engineering
Segmented B
ezier curves
Segmented Bezier curves (cont.)
I
Continuity from segment j to segment j + 1 is guaranteed if
Pn,j = P0,j+1
Since
d
d
Bin (u; uj1 , uj ) =
Bin
du
du
u uj1
uj uj1
=
d
1
Bin (t) ,
dt
hj
we get with the theorem on page 20:
n(Pn,j Pn1,j )/hj = n(P1,j+1 P0,j+1 )/hj+1 .
Pn,j = P0,j+1 = Pn,j =
I
hj+1
hj
Pn1,j +
P1,j+1
hj + hj+1
hj + hj+1
Continuity of higher derivatives can be enforced similarly.
NumCSE, Lecture 19, Nov 20, 2014
30/39
Numerical Methods for Computational Science and Engineering
Segmented B
ezier curves
Examples
Example
I
Want to approximate quarter of unit circle by (one segment)
of Bezier curve.
NumCSE, Lecture 19, Nov 20, 2014
31/39
Numerical Methods for Computational Science and Engineering
Segmented B
ezier curves
Examples
Example (cont.)
P(t) =
3
X
Pi Bi3 (t)
i=0
1
1
0 3
3
2
2
=
(1 t) +
3(1 t) t +
3(1 t)t +
t
0
1
1
I
How to choose ? Lets try to have the Bezier curve meet the
circle at t = 1/2:
1
p
1 1 0 1
3 = p1/2
0 1 1 8 3
1/2
1
p
1 p
= 3+4 = 8 1/2 = = (8 1/24) 0.552285.
3
NumCSE, Lecture 19, Nov 20, 2014
32/39
Numerical Methods for Computational Science and Engineering
Segmented B
ezier curves
Examples
Examples of segmented Bezier curves
Profile of an airfoil.
Most pictures today from Schwarz/K
ockler: Numerische Mathematik.
Springer.
NumCSE, Lecture 19, Nov 20, 2014
33/39
Numerical Methods for Computational Science and Engineering
Segmented B
ezier curves
Examples
Examples of segmented Bezier curves (cont.)
NumCSE, Lecture 19, Nov 20, 2014
34/39
Numerical Methods for Computational Science and Engineering
B
ezier surfaces
Bezier surfaces
Similarly as curves it is possible to define surfaces in 3D:
Let Pij , i = 0, 1, . . . , n; j = 0, 1, . . . , m be 3-dimensional vectors.
We define a surface by
x(s, t) :=
n X
m
X
Pij Bin (s)Bjm (t),
s, t [0, 1].
i=0 j=0
Bin and Bjm are Bernstein polynomials of the proper degree.
NumCSE, Lecture 19, Nov 20, 2014
35/39
Numerical Methods for Computational Science and Engineering
Tensor splines
Tensor splines
Let (xi , yj ), i = 0, 1, . . . , n; j = 0, 1, . . . , m be a regular grid.
Let Bi (x), Bj (y ) be the piecewise linear functions that satisfy
Bi (xk ) = ik ,
Bj (y` ) = j` .
We can interpolate a function f defined on [x0 , xn ] [y0 , ym ] by
n X
m
X
f(xi , yj )Bi (x)Bj (y ),
(x, y ) [x0 , xn ] [y0 , ym ]
i=0 j=0
Inside a rectangle [xi , xi+1 ] [yj , yj+1 ] this function is bilinear, i.e.
of the form
+ x + y + xy .
The 4 values , , , are determined by the 4 values of f at the
corners of the rectangle.
NumCSE, Lecture 19, Nov 20, 2014
36/39
Numerical Methods for Computational Science and Engineering
Tensor splines
Tensor splines (cont.)
Piecewise
bilinear basis
function.
This is the
way
Matlab
works.
NumCSE, Lecture 19, Nov 20, 2014
37/39
Numerical Methods for Computational Science and Engineering
Tensor splines
Tensor splines (cont.)
In the same way we can interpolate with piecewise cubic Hermite
polynomials which gives a smooth interpolant. We have to take
derivatives
f
(xi , yj ),
x
f
(xi , yj ),
y
2f
(xi , yj ),
xy
into account.
NumCSE, Lecture 19, Nov 20, 2014
38/39
Numerical Methods for Computational Science and Engineering
Tensor splines
Tensor splines (cont.)
Piecewise
bicubic basis
function.
NumCSE, Lecture 19, Nov 20, 2014
39/39
Numerical Methods for Computational Science and Engineering
Introduction
Goal of the lecture
I
To develop some elegant, classical theory of best
approximation using least squares and weighted least squares
norms;
to develop important families of orthogonal polynomials;
to examine in particular Legendre and Chebyshev polynomials.
NumCSE, Lecture 20, Nov 24, 2014
2/43
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
1. Best least squares approximation
2. Orthogonal basis functions
3. Trigonometric basis function
4. Weighted least squares
5. Chebyshev polynomials
References
I
U. Ascher & C. Greif: Numerical methods. SIAM 2011.
Chapter 12.
Examples from H.-R. Schwarz & N. K
ockler: Numerische
Mathematik.
NumCSE, Lecture 20, Nov 24, 2014
3/43
Numerical Methods for Computational Science and Engineering
Introduction
Approximation, not just interpolation
I
Previously, we were given a complicated (or implicit) function
f (x) which may be evaluated anywhere on the interval [a, b].
We looked for a simpler approximation
v (x) =
n
X
cj j (x),
j=0
where j (x) are known basis functions with some useful
properties.
I
We determined the unknown parameters cj by imposing n + 1
interpolation conditions:
v (xi ) f (xi ) = 0
NumCSE, Lecture 20, Nov 24, 2014
4/43
Numerical Methods for Computational Science and Engineering
Introduction
Approximation, not just interpolation (cont.)
I
If there is significant noise in the measured values of f then
measuring the approximation error at m + 1 > n + 1 points
may yield more plausible or meaningful results. This leads to
overdetermined linear systems which are most easily solved
using discrete least squares methods; see earlier.
Here, however, we stick to the continuous, rather than
discrete level, and use integrals rather than vectors.
Typically we may seek a function v (x), i.e., coefficients cj , so
as to minimize
Rb
kv f k22 = (v (x) f (x))2 dx
a
NumCSE, Lecture 20, Nov 24, 2014
5/43
Numerical Methods for Computational Science and Engineering
Introduction
Function norms and orthogonality
I
For all integrable functions g (x) and f (x) on an interval
[a, b], a norm kk is a scalar function satisfying
I
I
I
kg k 0; kg k = 0 iff g (x) = 0,
kg k = ||kg k for all scalars ,
kf + g k kf k + kg k.
The set of all functions whose norm is finite forms a function
space associated with that particular norm.
Some popular function norms and corresponding function
spaces are:
L2 : kg k2 =
R
b
a
g (x)2 dx
1/2
(least squares)
Rb
L1 : kg k1 = |g (x)|dx,
a
L : kg k = max |g (x)|,
NumCSE, Lecture 20, Nov 24, 2014
axb
(maximum norm)
6/43
Numerical Methods for Computational Science and Engineering
Introduction
Function norms and orthogonality (cont.)
I
Two square-integrable functions f , g L2 are orthogonal if
their inner product vanishes,
Z
f (x)g (x) dx = 0.
a
I
We have seen the L norm in action when estimating errors
in polynomial interpolation. Here we concentrate on the L2
norm, occasionally in weighted form.
NumCSE, Lecture 20, Nov 24, 2014
7/43
Numerical Methods for Computational Science and Engineering
Introduction
Why continuous approximation?
I
We have dealt with discrete least squares earlier,
1
min kb Axk2 ,
x 2
for, e.g., polynomial data fitting.
Now, we stick to the continuous case that allows developing a
mathematically elegant theory.
Moreover, classical families of orthogonal basis functions are
developed which will prove useful later.
NumCSE, Lecture 20, Nov 24, 2014
8/43
Numerical Methods for Computational Science and Engineering
Best least squares approximation
Best least squares approximation
I
We are still approximating a given f (x) on interval [a, b] using
v (x) =
n
X
cj j (x)
j=0
but we are no longer interpolating.
I
Determine the coefficients cj as those that solve the
minimization problem
kf
NumCSE, Lecture 20, Nov 24, 2014
v k22
Z
min
c
f (x)
n
X
2
cj j (x) dx
j=0
9/43
Numerical Methods for Computational Science and Engineering
Best least squares approximation
Best least squares approximation (cont.)
I
Taking first derivative with respect to each ck in turn and
equating to 0 gives
Zb
f (x)
n
X
cj j (x) k (x) dx = 0,
k = 0, . . . , n.
j=0
or
n
X
j=0
Zb
cj
Zb
j (x)k (x) dx =
|a
NumCSE, Lecture 20, Nov 24, 2014
{z
jk
B
f (x)k (x) dx ,
|a
{z
bk
k = 0, . . . , n.
10/43
Numerical Methods for Computational Science and Engineering
Best least squares approximation
Algorithm: Continuous least squares
Given data function f (x) on [a, b], construct approximation v (x) =
Pn
j=0 cj j (x) as follows:
1. Calculate
jk =
B
j (x)k (x) dx,
j, k = 0, 1, . . . , n,
bj =
f (x)j (x) dx,
j = 0, 1, . . . , n.
2. Solve linear (n + 1) (n + 1) system
=b
Bc
for the coefficients c = (c0 , c1 , . . . , cn )T .
NumCSE, Lecture 20, Nov 24, 2014
11/43
Numerical Methods for Computational Science and Engineering
Best least squares approximation
Example 12.1: f (x) = cos(2x) for n = 0, . . . , 4
I
Example: using a monomial basis j (x) = x j for a polynomial
approximation on [0, 1], we obtain
j,k =
B
x j+k dx =
1
,
j +k +1
j, k = 0, 1, . . . , n.
This is the extremely ill-conditioned Hilbert matrix.
5 105 .
For n = 4 we have (B)
I
The integrals
bj =
f (x) x j dx
can be approximated by numerical integration (still to come).
NumCSE, Lecture 20, Nov 24, 2014
12/43
Numerical Methods for Computational Science and Engineering
Best least squares approximation
Example 12.1: f (x) = cos(2x) for n = 0, . . . , 4 (cont.)
NumCSE, Lecture 20, Nov 24, 2014
13/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Orthogonal basis functions
I
The simple example using monomials leads to stability
problems (bad conditioning in B).
Using monomial and other simple-minded bases, must
evaluate many integrals when n is large.
is diagonal!
Idea: Construct basis functions such that B
This means
Z b
j (x)k (x) dx = 0,
j 6= k.
a
I
Take advantage of the fact that here, unlike interpolation,
there are no input abscissae: can construct families of
orthogonal basis functions in advance!
NumCSE, Lecture 20, Nov 24, 2014
14/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Best least squares using orthogonal basis functions
I
The construction algorithm becomes simply:
1. Set
jj =
dj = B
2j (x) dx,
j = 0, 1, . . . , n,
in advance, and calculate
Z 1
bj =
f (x) j (x) dx,
j = 0, 1, . . . , n.
2. The sought coefficients are
cj = bj /dj ,
I
j = 0, 1, . . . , n.
Next, restrict to polynomial best approximation and ask, how
to construct a family of orthogonal polynomials?
NumCSE, Lecture 20, Nov 24, 2014
15/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Legendre polynomials
Legendre polynomials
Defined on the interval [1, 1] by
0 (x) = 1,
1 (x) = x,
j
2j + 1
x j (x)
j1 (x),
j+1 (x) =
j +1
j +1
j 1.
So,
1
2 (x) = (3x 2 1),
2
5
2
1
3 (x) = x(3x 2 1) x = (5x 3 3x).
6
3
2
NumCSE, Lecture 20, Nov 24, 2014
16/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Legendre polynomials
Legendre polynomials: properties
I
Orthogonality:
Z1
j (x)k (x) dx =
1
0,
2
,
2j + 1
j 6= k,
j = k.
Calibration: |j (x)| 1, 1 x 1, and j (1) = 1.
Oscillation: j (x) has degree j (not less).
All its j zeros are simple and lie inside the interval (1, 1).
Hence the polynomial oscillates j times in this interval.
NumCSE, Lecture 20, Nov 24, 2014
17/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Legendre polynomials
Legendre polynomials: the picture
You can determine which curve corresponds to which j by the number
of oscillations.
NumCSE, Lecture 20, Nov 24, 2014
18/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Legendre polynomials
Best approximation on a general interval
Let the given interval be [a, b].
We apply an affine transformation [1, 1] 3 x t [a, b] defined
by
1
t = [(b a)x + (a + b)].
2
We see that
Z b
Z 1
(b a)
ba
2
dx =
.
dj =
[j (t)] dt =
[j (x)]2
2
2j + 1
a
1
NumCSE, Lecture 20, Nov 24, 2014
19/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Legendre polynomials
Legendre polynomials: example
I
I
Find polynomial best approximations for f (x) = cos(2x)
over [0, 1] using n orthogonal polynomials.
Using Legendre polynomials (on [0, 1]),
R1
bj = 0 cos(2t)j (2t 1) dt, and cj = (2j + 1)bj .
Here f is in green.
Obtain the same
approximation for n = 2l
and n = 2l + 1 due to
symmetry of f . Note
improvement in error as l
increases.
NumCSE, Lecture 20, Nov 24, 2014
20/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Trigonometric polynomials
Trigonometric polynomials
I
Not all interesting orthogonal basis functions are polynomials.
Of particular interest are the periodic basis functions
1
2`1 (x) = sin(`x),
1
0 (x) = ,
2
1
2` (x) = cos(`x),
for ` = 1, 2, . . .
I
These are orthonormal on [, ]:
(
0, j 6= k,
j (x)k (x) dx =
1, j = k.
Leads to the famous Fast Fourier transform (next time).
NumCSE, Lecture 20, Nov 24, 2014
21/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Fourier Analysis
Fourier Analysis
For 2-time-periodic real-valued functions f one often considers a
frequency or Fourier decomposition:
f (t)
X
a0 X
+
ak cos(kt) +
bk sin(kt),
2
k=1
(1)
k=1
with the real Fourier coefficients
Z
1
f (t) cos(kt) dt,
Z
1
bk =
f (t) sin(kt) dt,
ak =
k 0,
k 1.
?
in (1) means formal expansion. Convergence f (t)
NumCSE, Lecture 20, Nov 24, 2014
22/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Fourier Analysis
Fourier Analysis (cont.)
Theorem (Convergence)
Let f (x) be a 2 periodic, piecewise continuous function with
piecewise continuous first derivative. Then the Fourier series p(x)
converges to
a) f (x0 ),
b)
1
2 (f0
if f is continuous at x0 ,
f0+ ),
if f is discontinuous at x0 .
Here,
lim f (x0 h) = f0 ,
h+0
NumCSE, Lecture 20, Nov 24, 2014
lim f (x0 + h) = f0+ .
h+0
23/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Fourier Analysis
Example 1: f (x) = |x|
The 2-periodic function f (x) is defined in (, ) as
f (x) = |x|,
< x < .
This function is related to hat function, but it is periodic. f is
continuous and even. Therefore, the sine terms in the Fourier
series vanish, bk = 0 for all k.
1
|x| dx =
x dx =
Z
2
2 1
1
ak =
x cos(kx) dx =
x sin(kx)
sin(kx) dx
0
k
k 0
0
2
2
=
cos(kx) =
[(1)k 1], k > 0.
2
k
k 2
a0 =
NumCSE, Lecture 20, Nov 24, 2014
24/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Fourier Analysis
Example 1: f (x) = |x| (cont.)
The corresponding Fourier series is
4 cos(x) cos(3x) cos(5x)
+
+
+
g (x) =
2
12
32
52
Periodic hat function with approximating Fourier polynomal g3 (x). Picture
from Schwarz & K
ockler.
NumCSE, Lecture 20, Nov 24, 2014
25/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Fourier Analysis
Example 2: f (x) = x 2
The 2-periodic function f (x) is defined in (0, 2) as
f (x) = x 2 ,
0 < x < 2.
This function jumps at xk = 2k, k Z, but satisfies the
assumptions in the theorem. The Fourier coefficients are
a0 =
8 2
,
3
ak =
4
,
k2
bk =
4
,
k
such that
4 2 X
+
g (x) =
3
k=1
NumCSE, Lecture 20, Nov 24, 2014
4
4
cos(kx)
sin(kx)
k2
k
26/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Fourier Analysis
Example 2: f (x) = x 2 (cont.)
Periodic funktion with jumps. From Schwarz & K
ockler.
NumCSE, Lecture 20, Nov 24, 2014
27/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Fourier Analysis
Low pass filter
As frequency k
increases, sin(kx) /
cos(kx) oscillate more.
Observed data = true
signal plus noise
oscillating at higher
frequencies, then best
least squares
approximation with the
first few trigonometric
basis functions naturally
filters out noise.
Low pass filter.
NumCSE, Lecture 20, Nov 24, 2014
28/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Weighted least squares
Weighted least squares
I
An important generalization of best least squares
approximation: with a weight function w (x) that is never
negative, has bounded L2 norm, and may be equal 0 only at
n
P
cj j (x) so
isolated points, seek coefficients cj for v (x) =
j=0
as to minimize
Z
w (x)(v (x) f (x))2 dx.
a
I
The orthogonality condition correspondingly generalizes to
Z
w (x)j (x)k (x) dx = 0,
j 6= k.
NumCSE, Lecture 20, Nov 24, 2014
29/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Weighted least squares
Weighted least squares (cont.)
I
If this holds then the best approximation solution is given by
cj = bj /dj ,
where
Z
dj =
w (x)2j (x) dx > 0
and
bj =
w (x)f (x)j (x) dx, j = 0, 1, . . . , n.
a
Important families of orthogonal polynomials may be obtained
for good choices of the weight function.
NumCSE, Lecture 20, Nov 24, 2014
30/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Weighted least squares
GramSchmidt process for constructing orthogonal
polynomials
I
Given an interval [a, b] and a weight function w (x), this
algorithm produces an orthogonal polynomial family via a
short three-term recurrence!
1 (x) = x 1 ,
Set 0 (x) = 1,
j (x) = (x j )j1 (x) j j2 (x),
j 2.
where
Rb
j = Ra b
a
Rb
j =
NumCSE, Lecture 20, Nov 24, 2014
xw (x)[j1 (x)]2 dx
w (x)[j1 (x)]2 dx
xw (x)j1 (x)j2 (x)dx
,
Rb
2
a w (x)[j2 (x)] dx
j 1,
j 2.
31/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Weighted least squares
GramSchmidt process for constructing orthogonal
polynomials (cont.)
I
For w (x) = 1 we again find the Legendre polynomials, scaled
in a different way.
Orthogonal basis vs. orthonormal basis.
Let the roots of the jth Legendre polynomials be
1 < x1 < < xj < 1. Then we can write
j (x) = (x x1 )(x x2 ) (x xj ),
6= 0.
For all polynomials q(x) of degree less than j we have
Z
q(x)(x x1 )(x x2 ) (x xj ) dx = 0.
1
NumCSE, Lecture 20, Nov 24, 2014
32/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Weighted least squares
GramSchmidt process for constructing orthogonal
polynomials (cont.)
Will use this to derive highly accurate numerical integration
formulae. Zeros of Legendre polynomial are called Gauss
points.
I
Intimately connected to derivation of Krylov subspace iterative
methods for symmetric linear systems (CG, MINRES).
In fact this is the Lanczos procedure with the matrix A as a
map of Rn Rn replaced by multiplication by x as an
operator in the space of polynomials,
p(x)
x p(x).
Here, we choose the constant polynomial 0 (x) 1 as the
initial vector.
NumCSE, Lecture 20, Nov 24, 2014
33/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Weighted least squares
Example 12.4: Laguerre polynomials
Transformation from [0, 1) to infinite interval [0, ):
[0, 1) 3 t = 1 e x x [0, ).
Now, dt = e x dx, so for any function g L2 (0, 1) we have
Z
Z 1
x
2
e g (x) dx =
g (t)2 dt <
0
e x
For w (x) =
and [a, b] = [0, ) the Laguerre polynomials form
an orthogonal family:
0 (x) = 1,
1 (x) = 1 x,
2j + 1 x
j
j+1 (x) =
j (x)
j1 (x),
j +1
j +1
NumCSE, Lecture 20, Nov 24, 2014
j > 0.
34/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Weighted least squares
Example 12.5: Hermite polynomials
2
w (x) = e x ,
[a, b] = (, )
Hermite polynomials
0 (x) = 1,
1 (x) = 2x,
j+1 (x) = 2xj (x) 2jj1 (x),
NumCSE, Lecture 20, Nov 24, 2014
j > 0.
35/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Chebyshev polynomials
Chebyshev polynomials
I
This is a family of orthogonal polynomials on [1, 1] with
respect to the weight function
w (x) =
1
1 x2
Define
j (x) = Tj (x) = cos(j arccos(x)) = cos(j),
where x = cos().
So, these polynomials are naturally defined in terms of an
angle, . Clearly, the larger j the more oscillatory Tj .
NumCSE, Lecture 20, Nov 24, 2014
36/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Chebyshev polynomials
Chebyshev polynomials (cont.)
I
Easy to verify orthogonality:
Z 1
Z
w (x)Tj (x)Tk (x) dx =
1
Z
=
0
I
Tj (x)Tk (x)
dx
1 x2
(
0, j 6= k,
cos(j) cos(k)d =
2 , j = k.
These polynomials obey the simple 3-term recurrence
T0 (x) = 1,
T1 (x) = x,
Tj+1 (x) = 2x Tj (x) Tj1 (x),
They satisfy |Tj (x)| 1,
NumCSE, Lecture 20, Nov 24, 2014
j 1.
1 x 1.
37/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Chebyshev polynomials
Chebyshev polynomials (cont.)
I
The roots (zeros) of Tn are the Chebyshev points
2k 1
,
k = 1, . . . , n.
xk = cos
2n
The interleaving extremal points (where Tn0 (x) = 0) are
k
,
k = 0, 1, . . . , n.
k = cos
n
Tn (k ) = (1)k .
(The k include the interval end points.)
NumCSE, Lecture 20, Nov 24, 2014
38/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Chebyshev polynomials
Chebyshev polynomials: the picture
You can determine which curve corresponds to which j = Tj by the
number of oscillations. Note the perfect calibration.
NumCSE, Lecture 20, Nov 24, 2014
39/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Chebyshev polynomials
Chebyshev polynomials: Min-max property
I
Chebyshev points tend to concentrate towards the interval
ends (recall w (x)).
Maximum norm of Tn satisfies
kTn k = max |Tn (x)| = 1,
1x1
Note that outside of [1, 1] |Tn (x)| goes to .
I
Leading coefficient of Tn (x) (what multiplies x n ) equals 2n1
NumCSE, Lecture 20, Nov 24, 2014
40/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Chebyshev polynomials
Chebyshev polynomials: Min-max property (cont.)
Theorem (Chebyshev Min-Max Property)
Let x1 , . . . , xn be the Chebyshev points and define
n (x) =
T
n
Y
(x xk ).
k=1
n (x) uniquely has the
Then, of all monic polynomials of degree n, T
smallest maximum magnitude over the interval [1, 1], and
n (x)| =
max |T
1x1
NumCSE, Lecture 20, Nov 24, 2014
1
2n1
41/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Chebyshev polynomials
Chebyshev polynomials: Min-max property (cont.)
The monic Chebyshev polynomials are
0 (x) = 1,
T
1 (x) = x,
T
2 (x) = x 2 1 ,
T
2
n
Y
T
(x)
n (x) = n
T
=
(x xk ),
2n1
n 1.
k=1
The min-max property follows from the oscillation of Tn (x) at its
extremum points k , where its first derivative vanishes and
n (k ) =
T
NumCSE, Lecture 20, Nov 24, 2014
1
(1)k .
2n1
42/43
Numerical Methods for Computational Science and Engineering
Orthogonal basis functions
Chebyshev polynomials
Chebyshev polynomials: Min-max property (cont.)
Proof.
For any monic polynomial pn (x) of degree n, we now show that if
1
n .
max1x1 |pn (x)| 2n1
, then necessarily pn T
n (x) pn (x). Then, q(x) has degree at most n 1.
Let q(x) = T
At the points k we know that for k = 0, 1, . . . , n
(
0, k even,
1
k
q(k ) = n1 (1) pn (k )
2
0, k odd.
So, q(x) changes sign at least n times on the interval [1, 1], and
therefore it has at least n zeros there. This implies that q 0, and
hence pn = Tn .
NumCSE, Lecture 20, Nov 24, 2014
43/43
Numerical Methods for Computational Science and Engineering
Introduction
Goals
I
Understand the approximation properties of the famous
Fourier transform;
See the power and elegance of this transform in important
applications;
Understand the fast Fourier transform (FFT) algorithm, one
of the most popular algorithms of all times;
Consider the discrete cosine transform (DCT), used in basic
image compression software such as jpeg.
NumCSE, Lecture 21, Nov 27, 2014
2/38
Numerical Methods for Computational Science and Engineering
Introduction
Topics today
I
Fourier transform in real and complex arithmetic
Discrete Fourier transform (DFT)
Fast Fourier transform (FFT)
Filtering
References
I
U. Ascher & C. Greif: Numerical methods. SIAM 2011.
Chapter 13.
NumCSE, Lecture 21, Nov 27, 2014
3/38
Numerical Methods for Computational Science and Engineering
Fourier Analysis
Approximation using orthogonal trigonometric polynomials
I
We have considered polynomial approximation in various
forms,
n
X
v (x) =
cj j (x),
j=0
where j (x) are known basis functions, usually polynomials of
degree j.
I
We now look more closely at the case where
0 (x) = 1,
2k1 (x) = sin(kx),
2k (x) = cos(kx),
k > 0.
This describes a signal f as a sum or integral of components
of higher and higher frequencies k.
NumCSE, Lecture 21, Nov 27, 2014
4/38
Numerical Methods for Computational Science and Engineering
Fourier Analysis
Why Fourier transform?
I
Not so great as a general-purpose approximation!
But great for smooth, periodic functions.
Note: a function f defined on the real line is periodic if there
is a positive scalar T , called the period, such that
f (x + T ) = f (x) for all x.
Through change of variable x = 2t/T , we can assume
w.l.o.g. that T = 2, i.e., we can consider 2-periodic
functions only.
NumCSE, Lecture 21, Nov 27, 2014
5/38
Numerical Methods for Computational Science and Engineering
Fourier Analysis
Why Fourier transform? (cont.)
I
But the Fourier transform is more than just another pretty
approximation!
1. Decomposes a given function into components that are more
and more oscillatory.
2. Useful for describing natural phenomena signals such as light
and sound, which are smooth and periodic.
3. Useful for many image processing applications (e.g.,
deblurring), time series analysis, image and sound compression
(mp3, jpeg,. . . ), solving PDEs, and more.
NumCSE, Lecture 21, Nov 27, 2014
6/38
Numerical Methods for Computational Science and Engineering
Fourier Analysis
The Fourier transform
I
I
P
Consider (yet again) approximation v (x) = nj=0 cj j (x).
Family of orthogonal basis fcts: trigonometric polynomials
0 (x) = 1,
2k1 (x) = sin(kx),
2k (x) = cos(kx),
for k = 1, 2, . . . They are orthogonal on [, ]:
Z
j (x)k (x) dx = 0,
j 6= k.
This leads to Fourier transform. The real form Fourier series is
f (t) =
k=1
k=1
X
a0 X
+
ak cos(kt) +
bk sin(kt).
2
Finite range of the sum provides a computable approximation
v (x) for f (x).
NumCSE, Lecture 21, Nov 27, 2014
7/38
Numerical Methods for Computational Science and Engineering
Fourier Analysis
The Fourier transform (cont.)
I
The coefficients of the Fourier series are given by
Z
1
f (t) cos(kt) dt, k 0,
ak =
Z
1
bk =
f (t) sin(kt) dt, k 1.
Best least squares approximation (n = 2`):
v (x) =
n
X
j=0
cj j (x)
`1
a0 X
+ (ak cos(kt)+bk sin(kt))+a` cos(`t).
2
k=1
This leaves out the higher frequencies, i.e., it is a low pass
filter.
NumCSE, Lecture 21, Nov 27, 2014
8/38
Numerical Methods for Computational Science and Engineering
Fourier Analysis
The Fourier transform (cont.)
NumCSE, Lecture 21, Nov 27, 2014
9/38
Numerical Methods for Computational Science and Engineering
Fourier Analysis
Neater in complex arithmetic form
I
Recall identity e = cos() + sin(). So define complex basis
functions
k (x) = e kx = cos(kx) + sin(kx),
k = 0, 1, 2, . . .
Obtain Fourier series
f (x) =
ck k (x)
j=
with Fourier coefficient
1
ck =
2
NumCSE, Lecture 21, Nov 27, 2014
f (t)e kt dt,
10/38
Numerical Methods for Computational Science and Engineering
Fourier Analysis
Complex number refresher
I
A complex number z can be written as
z = x + y ,
where x and y are real and = 1. x = Re(z), y = Im(z).
p
p
I Magnitude: |z| =
x 2 + y 2 = (x + y )(x y ) = z
z , where
z = x y is the conjugate of z.
I
Euler identity
e = cos() + sin(),
where the angle is in radian.
In polar coordinates
z = re = r cos() + r sin() = x + y ,
where r = |z| and tan() = y /x.
NumCSE, Lecture 21, Nov 27, 2014
11/38
Numerical Methods for Computational Science and Engineering
Applications
Application: convolution
I
Problem: evaluate
Z
g (x s)f (s)ds,
(x) =
for two real, 2-periodic functions.
P
Solution: writing (x) =
j= ck k (x) we have
Z
Z Z
1
1
kx
(x)e
dx =
g (x s)f (s)ds e kx dx
ck =
2
2
Z Z
1
=
g ()e k d e ks f (s) ds,
= x s,
2
Z
= ckg
e ks f (s) ds = 2ckg ckf .
So, convolution becomes simple multiplication in frequency
domain!
NumCSE, Lecture 21, Nov 27, 2014
12/38
Numerical Methods for Computational Science and Engineering
Applications
Application: differentiation
I
Problem: evaluate g (x) = f 0 (x) with Fourier transform of f
given.
Solution: simply write
Z
Z
1
1
kx
g (x)e
dx = (k)
f (x)e kx dx = (k)ckf .
2
2
Thus,
ckg = k ckf .
So, differentiation becomes replacing ck by kck in the
frequency domain!
NumCSE, Lecture 21, Nov 27, 2014
13/38
Numerical Methods for Computational Science and Engineering
Discrete Fourier Transform
Discrete Fourier Transform
I
For convenience, we use the interval [0, 2].
Define the uniformly spaced abscissae
xi =
i
2i
= ,
2n
n
i = 0, 1, . . . , 2n 1.
We want to determine the trigonometric polynomial
n1
X
1
p2n (x) = (a0 + an cos(nx)) +
[ak cos(kx) + bk sin(kx)].
2
k=1
by interpolation
yi = p2n (xi ),
NumCSE, Lecture 21, Nov 27, 2014
i = 0, 1, . . . , 2n 1.
14/38
Numerical Methods for Computational Science and Engineering
Discrete Fourier Transform
Discrete Fourier Transform (cont.)
I
Then we have the important discrete orthogonality:
2n1
0 k 6= j
1 X
cos(kxi ) cos(jxi ) = 1 0 < k = j < n
i=1
2 k = j = 0, or k = j = n,
(
2n1
0 k 6= j
1 X
sin(kxi ) sin(jxi ) =
n
1 0<k =j <n
i=1
2n1
1 X
sin(kxi ) cos(jxi ) = 0,
n
j, k.
i=1
NumCSE, Lecture 21, Nov 27, 2014
15/38
Numerical Methods for Computational Science and Engineering
Discrete Fourier Transform
Discrete Fourier Transform (cont.)
I
DFT:
ak =
bk =
I
2n1
1 X
yi cos(kxi ),
n
k = 0, 1, . . . , n,
1
n
k = 1, 2, . . . , n 1.
i=0
2n1
X
yi sin(kxi ),
i=0
Interpolating trigonometric polynomial:
n1
X
1
p2n (x) = (a0 + an cos(nx)) +
[ak cos(kx) + bk sin(kx)].
2
k=1
Inverse DFT: For i = 0, 1, . . . , 2n 1:
n1
X
1
yi = p2n (xi ) = (a0 +an cos(nxi ))+ [ak cos(kxi )+bk sin(kxi )].
2
k=1
NumCSE, Lecture 21, Nov 27, 2014
16/38
Numerical Methods for Computational Science and Engineering
Discrete Fourier Transform
Application: DFT interpolation
Fourier transform is for much more than just approximation, but
this property is important and can yield surprises, so lets see it in
action.
1.
2
g (t) = t 2 (t+1)r (t2)r e t sinr (t+1) sinr (t2),
2.
NumCSE, Lecture 21, Nov 27, 2014
1 t 2.
(
1, 1 x + 1,
f (x) =
0, otherwise.
17/38
Numerical Methods for Computational Science and Engineering
Discrete Fourier Transform
Application: DFT interpolation (cont.)
Trigonometric polynomial interpolation for the hat function with p3 (x).
Interpolate at 4 points: the 5th at x = 2 is a freebe due to periodicity.
Bad approximation because of low continuity.
NumCSE, Lecture 21, Nov 27, 2014
18/38
Numerical Methods for Computational Science and Engineering
Discrete Fourier Transform
Application: DFT interpolation (cont.)
Trigonometric polynomial interpolation for smooth function g(t) with
r = 2 using p3 (x) (top left), p7 (x) (top right), p15 (x) (bottom left) and
p31 (x) (bottom right). Approximated function in dashed green.
NumCSE, Lecture 21, Nov 27, 2014
19/38
Numerical Methods for Computational Science and Engineering
Discrete Fourier Transform
Application: DFT interpolation (cont.)
Lousy approximation because of low continuity.
Worst near the jump discontinuities: Gibbs phenomenon.
Dispersion effect not local (dies out slowly).
NumCSE, Lecture 21, Nov 27, 2014
20/38
Numerical Methods for Computational Science and Engineering
Trigonometric Interpolation
Trigonometric interpolation
2n1
Let {yj }j=0
be an 2n-periodic sequence. We define a
trigonometric polynomial of degree 2n by
p2n (x) = a0 +
n
X
ak cos(kx) +
k=1
n1
X
bk sin(kx).
k=1
Does there exist a trigonometric polynomial of degree 2n such that
p2n (xj ) = yj
for
xj = j/n,
j = 0, 1, . . . , 2n 1.
Is the polynomial unique?
NumCSE, Lecture 21, Nov 27, 2014
21/38
Numerical Methods for Computational Science and Engineering
Trigonometric Interpolation
Trigonometric interpolation
To construct a trigonometric polynomial of degree 2n, we first
rewrite it in complex form: for n k n, we let
2 (ak bk ), 1 k n 1,
1 (a + b ), n + 1 k 1,
k
k
ck = 2
a0 ,
k = 0,
a ,
k = n.
n
We also use that
e z = cos(z) + sin(z),
1
cos(z) = (e z + e z ),
2
NumCSE, Lecture 21, Nov 27, 2014
sin(z) =
1 z
(e e z ).
2
22/38
Numerical Methods for Computational Science and Engineering
Trigonometric Interpolation
Trigonometric interpolation (cont.)
Using this definition, we calculate
n
n1
1X
1 X
p2n (x) = a0 +
ak (e kx + e kx ) +
bk (e kx e kx )
2
2
k=1
k=1
1
= a0 + an (e nx + e nx ) +
2
1
= cn e nx +
2
NumCSE, Lecture 21, Nov 27, 2014
n1
X
k=n+1
n1
X
k=1
n1
ak bk kx X ak + bk kx
e +
e
2
2
k=1
1
ck e kx + cn e nx .
2
23/38
Numerical Methods for Computational Science and Engineering
Trigonometric Interpolation
Trigonometric interpolation (cont.)
To determine if there is a unique set of cn , . . . , cn1 such that
1
p2n (xj ) = yj = cn e nxj +
2
n1
X
k=n+1
1
ck e kxj + cn e nxj .
2
we observe the following:
1. e jk/n = e (j+2n)k/n is a 2n-periodic sequence,
2. cn = cn
We extend the finite sequence cn , . . . , cn1 to a 2n-periodic
sequence and write
yj =
2n1
X
k=0
NumCSE, Lecture 21, Nov 27, 2014
ck e
jk/n
2n1
X
jk
ck 2n
,
m = e 2/m .
k=0
24/38
Numerical Methods for Computational Science and Engineering
Trigonometric Interpolation
Trigonometric interpolation (cont.)
In matrix form, we
1
1
1 2n
2
1 2n
.
..
.
.
.
have
2n1
1 2n
|
{z
V
2n1
2n
2(2n1)
2n
..
(2n1)2
2n
c0
c1
c2
..
.
y0
y1
y2
..
.
=
,
c2n1
y2n1
} | {z } | {z }
c
y
or V c = y where 2n = e /n is 2n-th root of unity.
A unique trigonometric interpolating polynomial of degree
2n exists V is non-singular.
NumCSE, Lecture 21, Nov 27, 2014
25/38
Numerical Methods for Computational Science and Engineering
Trigonometric Interpolation
Trigonometric interpolation (cont.)
Proof that V is nonsingular.
We show the existence and uniqueness of an interpolating
trigonometric polynomial by explicitly producing the inverse of V .
T.
We show that V V = N IN , N = 2n, where V = V H = V
Since
(j1)(k1)
Vj,k = N
1 j, k N,
we have for 1 k, l N
[V V ]kl =
N
X
j=1
(k1)(j1) (j1)(l1)
N
N1
X
j=0
j(lk)
N1
X
mj
N
,
j=0
with m = l k.
NumCSE, Lecture 21, Nov 27, 2014
26/38
Numerical Methods for Computational Science and Engineering
Trigonometric Interpolation
Trigonometric interpolation (cont.)
I
m = 1, thus [V V ]
Case k = l: m = 0 and N
kk = N.
m 6= 1. Thus, we can write
Case k 6= l: m 6= 0, N
N1
X
jm
N
=
j=0
Nm
1 N
m = 0.
1 N
Therefore, [V V ]kl = 0.
I
So, V V = N IN , or, V 1 =
c=
Note:
1 V
N
NumCSE, Lecture 21, Nov 27, 2014
1
NV
and
1
V y.
N
is unitary (complex orthogonal).
27/38
Numerical Methods for Computational Science and Engineering
Discrete Fourier Transform
Discrete Fourier Transform
Theorem (Discrete Fourier Transform)
The unique trigonometric polynomial of even degree n that
satisfies pN (xj ) = yj for xj = 2j/N is given by
pN (x) =
N1
X
ck e kx
k=0
where {ck }kZ is the n-periodic sequence given by the discrete
Fourier transform (DFT)
N1
N1
1 X
1 X
jk
ck = (FN y)j =
yj N =
yj e 2jk/N ,
N
N
j=0
NumCSE, Lecture 21, Nov 27, 2014
(+)
j=0
28/38
Numerical Methods for Computational Science and Engineering
Discrete Fourier Transform
Discrete Fourier Transform (cont.)
Vice versa, the yj can be obtained from the ck via the inverse
discrete Fourier transform (IDFT)
yj = (FN1 c)j =
N1
X
k=0
NumCSE, Lecture 21, Nov 27, 2014
jk
ck N
=
N1
X
ck e 2jk/N .
()
k=0
29/38
Numerical Methods for Computational Science and Engineering
Fast Fourier Transform
Fast Fourier Transform
I
A straightforward implementation of the discrete Fourier
transform c = FN y as a matrix-vector product costs O(N 2 )
floating point operations.
If N = 2m is a power of 2, then there are algorithms that can
calculate FN y in O(N log2 N) operations only.
Such algorithms are collectively known as the fast Fourier
transform. They were popularized by J.W. Cooley and
J.W. Tukey: An algorithm for the machine calculation of
complex Fourier series. Math. Comput. 19 (1965) 297301.
NumCSE, Lecture 21, Nov 27, 2014
30/38
Numerical Methods for Computational Science and Engineering
Fast Fourier Transform
Fast Fourier Transform (cont.)
We only show the algorithm for the inverse Fourier transform, the
one for the forward transform being similar. For k = 0, . . . , N 1:
(FN1 c)k =
N1
X
jk
c j N
,
N = 2n
j=0
n1
X
j=0
n1
X
j=0
2jk
c2j N
+
n1
X
(2j+1)k
c2j+1 N
j=0
k
c2j njk + N
n1
X
c2j+1 njk ,
2jk
N
= njk
j=0
(Split the sum in even and odd parts.)
NumCSE, Lecture 21, Nov 27, 2014
31/38
Numerical Methods for Computational Science and Engineering
Fast Fourier Transform
Fast Fourier Transform (cont.)
The two sums have exactly the same form as the inverse Fourier
transform of length n = N/2.
k
(FN1 c)k = (Fn1 ce )k + N
(Fn1 co )k ,
0 k < N.
It is useful to rewrite this equation for k = n + k:
k
(FN1 c)k = (Fn1 ce )k N
(Fn1 co )k ,
0 k < n.
Therefore, we can call our IFFT routine on two vectors of length n.
Then we add/subtract the second from the first result vector to
get the full IFFT.
This procedure can be applied recursively to the smaller IFFTs.
NumCSE, Lecture 21, Nov 27, 2014
32/38
Numerical Methods for Computational Science and Engineering
Fast Fourier Transform
Matrix representation of one step of IFFT (n = N/2)
y = FN c
0
In n
y
=
In n y00
In
In
In
In
=
=
N 2
N
n =
..
n
Fn
even ck
n
Fn
odd ck
n
Fn
PN c
n
Fn n
m1
N
(1)
PnN is the N-by-N permutation matrix that picks first the even and
then the odd components of the N-vector c.
NumCSE, Lecture 21, Nov 27, 2014
33/38
Numerical Methods for Computational Science and Engineering
Fast Fourier Transform
The recursive application for N = 2n = 4m:
y = FN x
In n
Fn
=
PN x
In n
Fn n
F
I
m
m
m
n
Pm
Fm
I
n
Im m
= n
n P
Pm
Fm
Im m
In n
Fm
Im m
After the permutations, the directly neighbored vector elements
with indices 2k and 2k + 1 are combined. In the next step,
elements that are two apart are combined. Finally, elements that
are four (= n/2) apart are combined.
NumCSE, Lecture 21, Nov 27, 2014
34/38
Numerical Methods for Computational Science and Engineering
Fast Fourier Transform
function y = myIFFT(x)
%MYIFFT y = myIFFT(x) computes the inverse Fourier transform of
%
vector x having length equal a power of 2
%
using the Cooley-Tukey recursive algorithm.
n = length(x);
if (n == 1)
y = x;
else
w = exp(2i*pi/n*[0:n/2-1]);
ze = myIFFT(x(1:2:n-1));
zo = myIFFT(x(2:2:n));
y = [ze+w.*zo; ze-w.*zo];
end
To compute the forward Fourier transform, one merely needs to replace
e 2k/n by e 2k/n and divide the final result by n.
NumCSE, Lecture 21, Nov 27, 2014
35/38
Numerical Methods for Computational Science and Engineering
Fast Fourier Transform
Example: IFFT
;
ifft(1) = 1
1
ifft
3
1
2
ifft
3
4
&
1+13= 4
1 1 3 = 2
4 + 1 6 = 10
2 + i (2) = 2 2i
4 1 6 = 2
2 i (2) = 2 + 2i
> ifft(2) = 2
2
ifft
4
'
ifft(3) = 3
2+14= 6
2 1 4 = 2
ifft(4) = 4
NumCSE, Lecture 21, Nov 27, 2014
36/38
Numerical Methods for Computational Science and Engineering
Fast Fourier Transform
Matlab
The Matlab built-in functions fft and ifft implement the
discrete Fourier transform and its inverse in compiled code.
Unlike in the definitions (+) and (), the FFT in Matlab puts
the division by n in the inverse transform, i.e., Matlab uses
(Fn y)k =
n1
X
yj e 2jk/n ,
j=0
n1
1X
(Fn1 c)j =
ck e 2jk/n .
n
k=0
NumCSE, Lecture 21, Nov 27, 2014
37/38
Numerical Methods for Computational Science and Engineering
Fast Fourier Transform
Complexity
Complexity
Let C (n) be the complexity to execute the Fourier transform of
length n = 2q . Then,
n
C (n) = 3n
2 + 2C ( 2 )
3n
n
= 3n
2 + 2 4 + 4C ( 4 )
..
.
= 3n
2 log2 n + n = O(n log2 n)
Notice that these are complex flops. As a complex
addition/subtraction amounts to two real additions/subtractions
and a complex multiplication amounts to four real multiplications
and two additions/subtractions, the real complexity of the FFT of
length n = 2q is
NumCSE, Lecture 21, Nov 27, 2014
C (n) = 5n log2 n + n real flops.
38/38
Numerical Methods for Computational Science and Engineering
Topics today
We continue with the discussion of the Discrete Fourier Transform,
keeping in mind its fast variant, the FFT.
I
Discrete cosine transform (DCT)
Application of the Fast Fourier transform (FFT) in signal
processing
Convolution
Filtering
Circulant matrices
NumCSE, Lecture 22, Dec 1, 2014
2/38
Numerical Methods for Computational Science and Engineering
Discrete cosine transform (DCT)
I
Discrete cosine transform (DCT) similar to DFT, operating on
real data with even symmetry (since the Fourier transform of
a real and even function is real and even).
A related transform is the discrete sine transform (DST),
which is equivalent to a DFT of real and odd functions.
Most common variant of discrete cosine transform is the
type-II DCT, which is often called simply the DCT, its
inverse, the type-III DCT, is correspondingly often called
simply the inverse DCT or the IDCT.
In image compression (e.g., jpeg) a two-dimensional DCT
variant is used.
NumCSE, Lecture 22, Dec 1, 2014
3/38
Numerical Methods for Computational Science and Engineering
Discrete cosine transform (DCT) (cont.)
Given m = n + 1 data values (xi , yi ), with xi =
(i+1/2)
,
m
i = 0, 1, . . . , n.
DCT defines coefficients
ak =
n
2 X
yi cos(kxi ),
m
k = 0, 1, . . . , n.
i=1
Interpolating trigonometric polynomial:
n
pn (x) =
X
1
a0 +
ak cos(kx)
2
k=1
Inverse DCT gives back the data values
n
yi = pn (xi ) =
X
1
a0 +
ak cos(kxi ),
2
i = 0, 1, . . . , n.
k=1
NumCSE, Lecture 22, Dec 1, 2014
4/38
Numerical Methods for Computational Science and Engineering
Discrete cosine transform (DCT) (cont.)
In image compression
(e.g., jpeg) a
two-dimensional
variant of DCT is used.
Here you see basis
functions of a 8 8
fragment of a
(grayscale) image.
Lossy compression:
omit terms with small
weight.
http://en.wikipedia.org/wiki/Discrete_cosine_transform
NumCSE, Lecture 22, Dec 1, 2014
5/38
Numerical Methods for Computational Science and Engineering
DFT vs. Chebyshev interpolation
DFT vs. Chebyshev interpolation
2
g (t) = t 2 (t +1)r (t 2)r e t sinr (t +1) sinr (t 2),
1 t 2.
Periodic function continous up to r -th derivative.
Error of DFT with n equidistant interpolation points.
n
31
63
127
error (r = 2)
9.8e-3
2.4e-3
3.9e-4
error (r = 10)
2.0e-5
2.2e-8
2.5e-11
If the function f (x) to be interpolated is smooth and periodic, and
certainly if it is only given sampled on a uniform mesh, then a
trigonometric polynomial interpolation (Fourier) is preferable.
NumCSE, Lecture 22, Dec 1, 2014
6/38
Numerical Methods for Computational Science and Engineering
DFT vs. Chebyshev interpolation
DFT vs. Chebyshev interpolation (cont.)
Chebyshev interpolation
n
31
63
127
Error (r = 2)
1.3e-13
2.1e-15
3.7e-15
Error (r = 10)
6.5e-6
2.5e-14
6.4e-14
If f (x) is smooth and given anywhere on the interval of interest,
but it is not periodic, then Chebyshev interpolation wins.
But no fast transform available.
Note: noise filtering possible with Chebyshev interpolation, too.
NumCSE, Lecture 22, Dec 1, 2014
7/38
Numerical Methods for Computational Science and Engineering
Signal processing
Signal processing
I
In signal processing, it is often useful to study the different
representations of a signal f (t).
For sound signals, it is often the case that f (t) is periodic (or
nearly periodic) with period T , i.e., that
f (t + T ) = f (t),
for all t R.
Sampling: Signal is
sampled at discrete
equidistant times ti .
vectors t, y,
yi = f (ti ).
NumCSE, Lecture 22, Dec 1, 2014
8/38
Numerical Methods for Computational Science and Engineering
Signal processing
Signal processing (cont.)
I
Discrete finite linear time-invariant causal channel (filter)
Output signal y is obtained by input signal x through some
manipulation that involves a filter characterized by a finite
vector h: y = h x (convolution).
We are mostly interested in periodic signals.
NumCSE, Lecture 22, Dec 1, 2014
9/38
Numerical Methods for Computational Science and Engineering
Discrete convolutions
Filter
Discrete finite linear time-invariant causal channel (filter)
xk
yk
input signal
output signal
(widely used model for digital communication channels, e.g. in
wireless communication theory)
Mathematically speaking, a (discrete) channel/filter is a linear
mapping from the space of input sequences {xk }kZ to output
sequences {yk }kZ .
NumCSE, Lecture 22, Dec 1, 2014
10/38
Numerical Methods for Computational Science and Engineering
Discrete convolutions
Impulse response
In order to link digital filters to linear algebra, we have to assume
certain properties that are indicated by the attributes finite ,
linear, time-invariant and causal:
I
finite: impulse response of finite duration = it can be
described by a vector h of finite length n.
time-invariant: when input is shifted in time, output is shifted
by the same amount of time.
linear: input 7 output-map is linear
output( signal 1 + signal 2)
= output(signal 1) + output(signal 2)
causal (or physical, or nonanticipative): output depends only
on past and present inputs, not on the future.
NumCSE, Lecture 22, Dec 1, 2014
11/38
Numerical Methods for Computational Science and Engineering
Discrete convolutions
Impulse response (cont.)
impulse
response
h1
hm
h2
t0 t1 t2
tn
h3
hn
t0 t1 t2
tm tn
Output = linear superposition of impulse responses:
NumCSE, Lecture 22, Dec 1, 2014
12/38
Numerical Methods for Computational Science and Engineering
Discrete convolutions
Impulse response (cont.)
NumCSE, Lecture 22, Dec 1, 2014
13/38
Numerical Methods for Computational Science and Engineering
Discrete convolutions
Impulse response (cont.)
NumCSE, Lecture 22, Dec 1, 2014
14/38
Numerical Methods for Computational Science and Engineering
Discrete convolutions
0
h0
0
0
..
..
h0
0
.
..
..
h
hn1
.
0
..
.
yn
0
hn1
.
.
..
+ + xn1
.. = x0 0 + x1 0 + x2
hn1
0
..
..
.
.
.
0
h0
..
.
.
.
..
y2n3
..
..
..
0
y2n2
0
0
hn1
y0
y1
..
.
yk =
n1
P
hkj xj ,
k = 0, . . . , 2n2,
(hk = 0 for k < 0 and k n).
j=0
NumCSE, Lecture 22, Dec 1, 2014
15/38
Numerical Methods for Computational Science and Engineering
Discrete convolutions
x = (x0 , . . . , xn1 )T Rn input s. 7 y = (y0 , . . . , y2n2 )T R2n1 output signal
h
0
h
1
y ..
.
0
... ..
.
.. h
. = n1
.. 0
.
..
y2n2
.
..
.
0
Matrix notation
NumCSE, Lecture 22, Dec 1, 2014
0 0
..
.. ..
. .
.
x
..
.. .. ..
0
. . . .
...
.. ..
. . 0
h1 h0
...
..
..
..
.
.
.
.. x
.. ..
n1
. .
.
.
.. ..
. . ..
0 hn1
(Toeplitz matrix)
16/38
Numerical Methods for Computational Science and Engineering
Discrete convolutions
Discrete convolution
Definition (discrete convolution)
Given x = (x0 , . . . , xn1 )T Rn , h = (h0 , . . . , hn1 )T Rn their
discrete convolution is the vector y R2n1 with components
yk =
n1
X
hkj xj
k = 0, . . . , 2n 2
(hj := 0 for j < 0)
j=0
Notation: y = h x.
x0 , . . . , xn1
FIR h0 , . . . , hn1
NumCSE, Lecture 22, Dec 1, 2014
h0 , . . . , hn1
FIR x0 , . . . , xn1
17/38
Numerical Methods for Computational Science and Engineering
Discrete convolutions
Multiplication of polynomials
Multiplication of polynomials
The bilinear operation that takes two input vectors and produces
an output vector with double the number of entries also governs
the multiplication of polynomials:
p(z) =
n1
P
q(z) =
k=0
n1
P
k=0
ak z k ,
bk z k
= (pq)(z) =
2n2
k
X X
aj bkj
zk
j=0
k=0
{z
=: ck
Here the roles of hk , xk are played by the ak and bk .
= coefficients of product polynomial can be obtained by discrete
convolution of coefficients of polynomial factors!
NumCSE, Lecture 22, Dec 1, 2014
18/38
Numerical Methods for Computational Science and Engineering
Discrete convolutions
Convolution of sequences
Convolution of sequences
The notion of a discrete convolution extends naturally to
sequences N0 7 R:
The (discrete) convolution of two sequences (xj )jN0 , (yj )jN0 is
the sequence (zj )jN0 defined by
zk :=
k
X
j=0
xkj yj =
k
X
xj ykj ,
k N0 .
j=0
(Recall product formula for power series, Cauchy product.)
NumCSE, Lecture 22, Dec 1, 2014
19/38
Numerical Methods for Computational Science and Engineering
Periodic convolutions
Linear filtering of periodic signals
n-periodic signal (n N) = sequence (xj )jZ with xj+n = xj j .
The n-periodic signal (xj )jZ is determined by any consecutive n
elements of the sequence, in particuler by
x0 , . . . , xn1 vector x = (x0 , . . . , xn1 )T Rn .
Whenever the input signal of a time-invariant filter is n-periodic, so
will be the output signal. Thus, in the n-periodic setting, a causal
linear time-invariant filter will give rise to a linear mapping
H : Rn 7 Rn according to
yk =
n1
X
hkj xj
for some h0 , . . . , hn1 R hk := hkn for all k Z
j=0
NumCSE, Lecture 22, Dec 1, 2014
20/38
Numerical Methods for Computational Science and Engineering
Periodic convolutions
Linear filtering of periodic signals (cont.)
Since H is a linear operator on the space of n-periodic sequences,
we can write it in matrix form:
h0 hn1 h1
y0
h1
h0 h2
y1
Hy = .
. . .
..
..
. .. ..
hn1 hn2
h0
yn1
H is a circulant matrix.
Every circulant matrix can be written as a convolution, and
vice versa.
The fundamental relationship between convolutions and
discrete Fourier transforms is given by the following lemma.
NumCSE, Lecture 22, Dec 1, 2014
21/38
Numerical Methods for Computational Science and Engineering
Periodic convolutions
Linear filtering of periodic signals (cont.)
Lemma
Let y and z be two periodic sequences. Then
Fn (y z) = n(Fn (y) Fn (z)),
where denotes pointwise multiplication.
Note that the lemma implies that the convolution operator is
symmetric, i.e., y z = z y, despite the apparent asymmetry in
the definition.
NumCSE, Lecture 22, Dec 1, 2014
22/38
Numerical Methods for Computational Science and Engineering
Periodic convolutions
Linear filtering of periodic signals (cont.)
Proof.
n1 n1
1 XX
(Fn (y z))k =
yl zjl e 2jk/n
n
j=0 l=0
1
n
n1 X
n1
X
yl zjl e 2(jl)k/n e 2lk/n
j=0 l=0
n1
n1
1 X 2lk/n X
zjl e 2(jl)k/n
=
yl e
n
l=0
=n
j=0
X
X
n1
n1
1
1
yl e 2lk/n
zj e 2jk/n
n
n
l=0
j=0
= n (Fn y)k (Fn z)k .
NumCSE, Lecture 22, Dec 1, 2014
23/38
Numerical Methods for Computational Science and Engineering
Periodic convolutions
Discrete periodic convolution
Definition (discrete periodic convolution)
The discrete periodic convolution of two n-periodic sequences (xk ),
k Z, (yk ), k Z yields the n-periodic sequence
(zk ) := (xk ) n (yk )
zk :=
n1
X
xkj yj =
j=0
n1
X
ykj xj
k Z
j=0
Definition (circulant matrix)
A matrix C = (cij )ni,j=1 Rn,n is circulant
some n-periodic sequence (uk )kZ .
NumCSE, Lecture 22, Dec 1, 2014
cij = uji for
24/38
Numerical Methods for Computational Science and Engineering
Periodic convolutions
How to make a discrete a periodic convolution
Recall discrete convolution of a = (a0 , . . . , an1 )T Rn ,
b = (b0 , . . . , bn1 )T Rn :
(a b)k =
n1
X
aj bkj ,
k = 0, . . . , 2n 2
j=0
Expand a0 , . . . , an1 and b0 , . . . , bn1 to 2n 1-periodic sequences
by zero padding:
(
(
ak , if 0 k < n,
bk , if 0 k < n,
xk :=
yk :=
0, if n k < 2n 1,
0,
if n k < 2n 1,
and periodic extension: xk = x2n1+k , yk = y2n1+k for all k Z.
NumCSE, Lecture 22, Dec 1, 2014
25/38
Numerical Methods for Computational Science and Engineering
Periodic convolutions
How to make a discrete a periodic convolution (cont.)
a0
an1
n
=
(a b)k = (x 2n1 y)k ,
NumCSE, Lecture 22, Dec 1, 2014
0
2n
3n
4n
k = 0, . . . , 2n 2.
26/38
Numerical Methods for Computational Science and Engineering
Periodic convolutions
Matrix notation
h0
y0
...
h
.1
.
..
..
..
..
.
.
= hn1
y
n1
yn
0
..
..
.
.
..
.
..
.
y2n2
0
|
NumCSE, Lecture 22, Dec 1, 2014
0
.. ..
. .
.. ..
. .
..
.
..
.
.. ..
. .
..
.
..
..
.
h1
0
..
.
..
.
0
h0
hn1
..
.
0
.. . .
.
.
..
.
0
h1 h0
..
.
h1
..
..
..
. .
.
0 hn1 hn2
{z
0
..
.
..
.
..
..
..
.
..
.
circulant matrix of order 2n 1
h1
..
.
..
.
x0
..
.
..
.
..
.
hn1
0
xn1
..
..
.
.
..
.
0
0
h0
}
27/38
Numerical Methods for Computational Science and Engineering
Circulant matrices
Circulant matrices
Let C be a circulant matrix:
c0
c1
C = .
..
cn1
c0
..
.
cn1 cn2
c1
c2
.. .
.
c0
Then,
C=
n1
X
ck Jnk
k=0
where
NumCSE, Lecture 22, Dec 1, 2014
0 1
. .
Jn = . . . . Rn,n .
0 1
1
0
28/38
Numerical Methods for Computational Science and Engineering
Circulant matrices
Circulant matrices (cont.)
If is an eigenvalue of Jn then I Jn is singular. Because
1
0
.. ..
0
. .
.. = .. .
.
.
1
n 1
1
n1
is an eigenvalue of Jn if n = 1.
The numbers nk e 2k/n , k = 0, . . . , n 1, are n different
solutions of n = 1. Therefore, with
1 1
1
NumCSE, Lecture 22, Dec 1, 2014
n1
1 n n
2(n1)
2
1
n
n
Fn =
.
.
.
..
.. ..
(n1)2
1 nn1 n
29/38
Numerical Methods for Computational Science and Engineering
Circulant matrices
Circulant matrices (cont.)
We have
1
Jn = Fn n Fn1 = Fn n Fn
n
n2
n =
.
.
..
where
nn1
The eigenvalue decomposition of C is now easily found:
C=
n1
X
ck Jnk =
k=0
n1
X
k=0
n1
X
1
ck Fn k
F = Fn
n
n n
ck k
n
k=0
|
NumCSE, Lecture 22, Dec 1, 2014
{z
1
F .
n n
30/38
Numerical Methods for Computational Science and Engineering
Circulant matrices
Circulant matrices (cont.)
:=
C
n1
X
ck k
n
k=0
is a diagonal matrix, the j diagonal element being
cj =
n1
X
ck nkj .
k=0
Therefore,
NumCSE, Lecture 22, Dec 1, 2014
c0
c1
..
.
c0
c1
..
.
c
=
F
c
=
F
.
n
n
cn1
cn1
31/38
Numerical Methods for Computational Science and Engineering
Circulant matrices
Computing convolutions via FFT
Convolution of two n-periodic sequences h and x
We exploit the fact that
h x Hx,
and
y = Fn x =
H = circ(h0 , . . . , hn1 ).
1
F x,
n n
x = Fn1 y = Fn y.
= Fn h and
1. Use FFTs to compute h
x = Fn x.
2. Compute w
= n(h
x) (elementwise product).
3. Use the IFFT to compute y = Fn1 w.
Since steps 1 and 3 each cost O(n log n) and step 2 costs O(n),
the overall cost is O(n log n).
NumCSE, Lecture 22, Dec 1, 2014
32/38
Numerical Methods for Computational Science and Engineering
Circulant matrices
Computing convolutions via FFT
Matlab functions
I
Discrete periodic convolution: straightforward implementation
function z=pconv(u,x)
n = length(x); z = zeros(n,1);
for i=1:n,
z(i)=dot(conj(u), x([i:-1:1,n:-1:i+1]));
end
Discrete periodic convolution: DFT implementation
function z=pconvfft(u,x)
% Implementation of convolution using FFTs
z = ifft(fft(u).*fft(x));
NumCSE, Lecture 22, Dec 1, 2014
33/38
Numerical Methods for Computational Science and Engineering
Circulant matrices
Computing convolutions via FFT
Matlab functions (cont.)
I
Discrete convolution: DFT implementation
function y = myconv(h,x)
n = length(h);
% Zero padding
h = [h;zeros(n-1,1)];
x = [x;zeros(n-1,1)];
% Periodic discrete convolution of length 2n-1
y = pconvfft(h,x);
Same is done by Matlabs conv.
NumCSE, Lecture 22, Dec 1, 2014
34/38
Numerical Methods for Computational Science and Engineering
Frequency filtering via DFT
Example: Frequency identification with DFT
Example: Frequency identification with DFT
Extraction of characteristic frequencies from a distorted discrete
periodical signal:
t = [0:63]; x = sin(2*pi*t/64)+ sin(7*2*pi*t/64);
y = x + randn(size(t));
%distortion
3
20
18
2
16
14
|ck|2
Signal
12
10
8
1
6
4
2
2
10
20
30
40
50
Sampling points (time)
NumCSE, Lecture 22, Dec 1, 2014
60
70
10
15
20
25
30
Coefficient index k
35/38
Numerical Methods for Computational Science and Engineering
Frequency filtering via DFT
Example: Frequency identification with DFT
Example: Frequency identification with DFT (cont.)
Picture generated by
c = fft(y); p = (c.*conj(c))/64;
figure(Name,power spectrum);
bar(0:31,p(1:32),r);
set(gca,fontsize,14);
axis([-1 32 0 max(p)+1]);
xlabel(index k of Fourier coefficient,...
Fontsize,14,FontWeight,bold);
ylabel(|c_k|^2,Fontsize,14,FontWeight,bold);
Pronounced peaks in the power spectrum point to periodic structure of
the data. Location of peaks tells lengths of dominant periods.
NumCSE, Lecture 22, Dec 1, 2014
36/38
Numerical Methods for Computational Science and Engineering
Frequency filtering via DFT
Frequency filtering by DFT
Example: Frequency filtering by DFT
Frequency filtering of real discrete periodic signals by suppressing
certain Fourier coefficients.
Im
function [low,high] = freqfilter(y,k)
m = length(y)/2; c = fft(y);
clow = c; clow(m+1-k:m+1+k) = 0;
chigh = c-clow;
low = real(ifft(clow));
high = real(ifft(chigh));
Map y 7 low in code
=
low pass filter.
High frequencies
Re
Low frequencies
Map y 7 high in code
=
high pass filter.
NumCSE, Lecture 22, Dec 1, 2014
37/38
Numerical Methods for Computational Science and Engineering
Frequency filtering via DFT
Frequency filtering by DFT
Example: Frequency filtering by DFT (cont.)
Noisy signal:
n = 256; y = exp(sin(2*pi*((0:n-1))/n)) + 0.5*sin(exp(1:n));
Frequency filtering with k = 120.
3.5
350
signal
noisy signal
low pass filter
high pass filter
300
2.5
250
2
200
|ck|
1.5
150
0.5
100
0
50
0.5
0.1
0.2
0.3
0.4
0.5
time
NumCSE, Lecture 22, Dec 1, 2014
0.6
0.7
0.8
0.9
20
40
60
80
100
120
140
No. of Fourier coefficient
38/38
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
1. NewtonCotes Rules
I
I
Trapezoidal rule
Simpsons rule
2. Romberg integration
3. Gauss quadrature
4. Adaptive quadrature
References
I
U. Ascher & C. Greif: Numerical methods. 15.115.5.
C. Moler: Numerical Computing with Matlab.
W. Gander, M. Gander, & F. Kwok: Scientific Computing.
Springer 2014.
NumCSE, Lecture 23, Dec 4, 2014
2/40
Numerical Methods for Computational Science and Engineering
Introduction
Introduction
Let f (x) be a continuous real-valued function of a real variable,
defined on a finite interval a x b.
We seek to compute the value of the integral,
Rb
f (x) dx.
Quadrature means the approximate evaluation of such a definite
integral.
Principle idea: Interpolate f by (piecewise) polynomial, then
integrate the polynomial.
NumCSE, Lecture 23, Dec 4, 2014
3/40
Numerical Methods for Computational Science and Engineering
Introduction
Introduction (cont.)
The fundamental additive property of definite integrals is the basis
for many quadrature formulae. For a < c < b we have
Zc
Zb
f (x) dx =
a
Zb
f (x) dx +
f (x) dx
c
In particular, we can subdivide an interval in many subintervals,
integrate f in the subintervals, and finally add up these partial
integrals to get the overall result.
We can subdivide subintervals that indicate big errors recursively in
an adaptive procedure.
NumCSE, Lecture 23, Dec 4, 2014
4/40
Numerical Methods for Computational Science and Engineering
Introduction
Basic quadrature rules
Let h = b a be the length of the interval. The two simplest
quadrature rules are
I
midpoint rule
M = hf
a+b
2
trapezoidal rule
f (a) + f (b)
2
The accuracy of a quadrature rule can be predicted in part by
examining its behavior on polynomials.
T =h
Order of a quadrature rule is the degree of the lowest degree
polynomial that the rule does not integrate exactly.
The orders of both midpoint and trapezoidal rule are 2.
NumCSE, Lecture 23, Dec 4, 2014
5/40
Numerical Methods for Computational Science and Engineering
NewtonCotes rules
NewtonCotes rule
One obvious way to approximate f is by polynomial interpolation
and then integrate the polynomial.
Let a = x0 < x1 < x2 < < xn = b be equidistant points in
[a, b]. The Lagrange interpolating polynomial is given by
Pn (x) =
n
X
Li (x)f (xi ),
Li (x) =
i=0
n
Y
x xj
,
xi xj
j=0
i = 0, . . . , n.
j6=i
Then
Zb
Zb
f (x) dx
Pn (x) dx =
a
n
X
i=0
Zb
f (xi )
Li (x) dx =
a
n
X
f (xi )wi .
i=0
Clearly, such a method has at least order n + 1.
NumCSE, Lecture 23, Dec 4, 2014
6/40
Numerical Methods for Computational Science and Engineering
NewtonCotes rules
Trapezoidal rule
Trapezoidal rule again
The trapezoidal rule is the NewtonCotes rule for n = 1:
P1 = f (a)
bx
x a
+ f (b)
ba
ba
and (with x = a + h)
Zb
bx
dx =
ba
Zb
x a
dx = h
ba
Z1
d =
T =
h
(f (a) + f (b)).
2
h
.
2
So,
NumCSE, Lecture 23, Dec 4, 2014
7/40
Numerical Methods for Computational Science and Engineering
NewtonCotes rules
Simpsons rule
Simpsons rule
Simpsons rule is the NewtonCotes rule for n = 2: (x = a + h)
P2 () = f (a)(1 2)(1 ) + f (c)4(1 ) + f (b)(2 1).
S=
h
(f (a) + 4f (c) + f (b)),
6
c=
a+b
.
2
Note that
2
1
S = M + T.
3
3
It turns out that Simpsons rule also integrates cubics exactly, but
not quartics, so its order is four.
NewtonCotes methods with order up to 8 exist.
Even higher order methods can be devised, but some of their
weights wi get negative which is undesirable for potential round-off.
NumCSE, Lecture 23, Dec 4, 2014
8/40
Numerical Methods for Computational Science and Engineering
NewtonCotes rules
Summary
Summary of basic quadrature rules
Quadrature
Midpoint
Trapezoidal
Simpson
Rule
a+b
)
(b a) f (
2
(b a)
[f (a) + f (b)]
2
a+b
(b a)
f (a) + 4f (
) + f (b)
6
2
NumCSE, Lecture 23, Dec 4, 2014
Error
f 00 (1 )
(b a)3
24
f 00 (2 )
(b a)3
12
5
f (4) (3 ) b a
90
2
9/40
Numerical Methods for Computational Science and Engineering
Composite Rules
Composite Rules
I
Rb
Applying a quadrature rule to a f (x) dx may not yield an
approximation with the desired accuracy.
To increase the accuracy, one can partition the interval [a, b]
into subintervals and apply the quadrature rule to each
subinterval.
The resulting formula is known as a composite rule.
NumCSE, Lecture 23, Dec 4, 2014
10/40
Numerical Methods for Computational Science and Engineering
Composite Rules
Composite Rules (cont.)
Picture from Moler: Numerical Computing with Matlab.
NumCSE, Lecture 23, Dec 4, 2014
11/40
Numerical Methods for Computational Science and Engineering
Composite Rules
Trapezoidal rule
Composite trapezoidal rule
Let [a, b] be partitioned into n equidistant subintervals (xi , xi+1 ) of
length h = xi+1 xi = (b a)/n.
Then we apply the trapezoidal rule to each subinterval to obtain
the composite trapezoidal rule
T (h) = h
1
2 y0
+ y1 + + yn1 + 12 yn ,
The error of the composite trapezoidal rule is
b
Z
2
f (x)dx T (h) = (b a)h |f 00 ()|,
12
yi = f (xi ).
[a, b].
NumCSE, Lecture 23, Dec 4, 2014
12/40
Numerical Methods for Computational Science and Engineering
Composite Rules
Trapezoidal rule
Composite trapezoidal rule (cont.)
Now that we have computed T (h), how do we compute T (h/2)?
h h
h 1
1
h
y0 + y1 + + y2n1 + y2n
T ( ) = S( ) =
2
2 2
2 2
2
h 1
1
=
y0 + y2 + + y2n2 + y2n
2 2
2
h
+ (y1 + y3 + + y2n1 )
2
h
h
= S(h) + (y1 + y3 + + y2n1 )
2
2
NumCSE, Lecture 23, Dec 4, 2014
13/40
Numerical Methods for Computational Science and Engineering
Composite Rules
Trapezoidal rule
function T=trapez(f,a,b,tol);
% TRAPEZ(f,a,b,tol) tries to integrate int_a^b f(x) dx
% to a relative tolerance tol using the composite
% trapezoidal rule.
%
h = b-a; s = (f(a)+f(b))/2;
tnew = h*s; zh = 1; told = 2*tnew;
fprintf(h = 1/%2.0f, T(h) = %12.10f\n,1/h, tnew)
while abs(told-tnew)>tol*abs(tnew),
told = tnew; zh = 2*zh; h = h/2;
s = s+sum(f(a+[1:2:zh]*h));
tnew = h*s;
fprintf(h = 1/%2.0f, T(h) = %12.10f\n,1/h, tnew)
end;
T = tnew;
NumCSE, Lecture 23, Dec 4, 2014
14/40
Numerical Methods for Computational Science and Engineering
Composite Rules
Trapezoidal rule
Example
Z1
I =
xe x
e 2
= 0.3591409142 . . .
dx =
2
(x + 1)
2
Intermediate approximations calculated by trapez(f,0,1,1e-4):
hi
T (hi )
0.339785228
1
2
1
4
1
8
1
16
1
32
1
64
0.353083866
0.31293
0.357515195
0.26840
0.358726477
0.25492
0.359036783
0.25125
0.359114848
0.25031
0.359134395
0.25007
2
3
4
5
6
If
f 00
T (hi ) I
T (hi1 ) I
does not vary too much, the error decreases by factor 4/step.
NumCSE, Lecture 23, Dec 4, 2014
15/40
Numerical Methods for Computational Science and Engineering
Composite Rules
Simpsons rule
Composite Simpsons rule
For the composite Simpson Rule, we need to partition the interval
[a, b] into 2n subintervals.
Let h = ba
2n , xi = a + ih and yi = f (xi ) for i = 0, 1, 2, . . . , 2n.
Applying Simpsons Rule to two consecutive subintervals
xi+1
Z
f (x) dx
h
(yi1 + 4yi + yi+1 )
3
xi1
we obtain the composite Simpson rule
S(h) =
h
(y0 + 4y1 + 2y2 + 4y3 + + 2y2n2 + 4y2n1 + y2n ) .
3
NumCSE, Lecture 23, Dec 4, 2014
16/40
Numerical Methods for Computational Science and Engineering
Composite Rules
Simpsons rule
Composite Simpsons rule (cont.)
The integration error is
b
Z
4
f (x)dx S(h) = (b a)h
180
(4)
f
()
,
with a b.
NumCSE, Lecture 23, Dec 4, 2014
17/40
Numerical Methods for Computational Science and Engineering
Composite Rules
Simpsons rule
Composite Simpsons rule (cont.)
Again Matlab function that computes approximations to the
integral by halving the integration step until a given tolerance is
met.
We introduce variables for sums of function values with the same
weight s1 , s2 and s4 then we have the following update
formula to compute the new Simpson approximation:
h
(s1 + 4s4 + 2s2 )
3
= s1
S(h) =
s1new
s2new = s2 + s4
s4new = sum of new function values
h/2 new
S(h/2) =
(s
+ 4s4new + 2s2new ).
3 1
NumCSE, Lecture 23, Dec 4, 2014
18/40
Numerical Methods for Computational Science and Engineering
Composite Rules
Simpsons rule
function S=simpson(f,a,b,tol);
% SIMPSON (f,a,b,tol) tries to integrate int_a^b f(x) dx
% to a relative tolerance tol using the composite
% Simpson rule.
%
h = (b-a)/2; s1 = f(a)+f(b); s2 = 0;
s4 = f(a+h); snew = h*(s1+4*s4)/3;
zh = 2; sold = 2*snew;
fprintf(h = 1/%2.0f, S(h) = %12.10f\n,1/h, snew)
while abs(sold-snew)>tol*abs(snew),
sold = snew; zh = 2*zh; h = h/2; s2 = s2+s4;
s4 = sum(f(a +[1:2:zh]*h));
snew = h*(s1+2*s2+4*s4)/3;
fprintf(h = 1/%2.0f, S(h) = %12.10f\n,1/h, snew)
end
S = snew;
NumCSE, Lecture 23, Dec 4, 2014
19/40
Numerical Methods for Computational Science and Engineering
Composite Rules
Simpsons rule
Same exampleZ1
I =
xe x
e 2
= 0.3591409142 . . .
dx =
2
(x + 1)
2
Intermediate approximations calculated by simpson(f,0,1,1e-8)
hi
S(hi )
0.357516745
1
2
1
4
1
8
1
16
1
32
1
64
0.358992305
0.09149
0.359130237
0.07184
0.359140219
0.06511
0.359140870
0.06317
0.359140911
0.06267
0.359140914
0.06254
2
3
4
5
6
NumCSE, Lecture 23, Dec 4, 2014
S(hi ) I
S(hi1 ) I
20/40
Numerical Methods for Computational Science and Engineering
Romberg method
Romberg method
The asymptotic expansion for the approximation of an integral by
the trapezoidal rule is given by
Zb
T (h) =
g (t) dt + K1 h2 + K2 h4 + + Km h2m + Rm .
We can extrapolate to the limit T (0) by interpolating the values
T (hi ) for some hi > 0 by a polynomial.
Error term Rm is small values T (hi ) are close to the values of
polynomial
2
P2m (h) = K0 + K1 h + K2 h + + Km h
NumCSE, Lecture 23, Dec 4, 2014
2m
Z
, with K0 =
g (t) dt.
a
21/40
Numerical Methods for Computational Science and Engineering
Romberg method
Romberg method (cont.)
Only even powers of h consider polynomial in variable x = h2 .
We extrapolate to x = 0 the data:
x
y
h02
h12
T (h0 ) T (h1 )
hi2
T (hi )
We assume that hi = hi1 /2 = h0 /2i .
We use the Newton polynomial interpolation to determine the
coefficients Ki , i = 0, 1, . . . , m.
NumCSE, Lecture 23, Dec 4, 2014
22/40
Numerical Methods for Computational Science and Engineering
Romberg method
Romberg method (cont.)
This is the triangular scheme:
T (h0 ) = T00
T (h1 ) = T10 T11
..
..
..
..
.
.
.
.
T (hk ) = Tk0 Tk1
Tkk
with
Ti+1,j =
NumCSE, Lecture 23, Dec 4, 2014
4j Ti+1,j1 Ti,j1
,
4j 1
0 j i k.
23/40
Numerical Methods for Computational Science and Engineering
Romberg method
Same example again
Z1
xe x
e 2
dx =
= 0.3591409142 . . .
2
(x + 1)
2
Intermediate approximations calculated by
[a z]=romberg(f,0,1,1e-8)
NumCSE, Lecture 23, Dec 4, 2014
24/40
Numerical Methods for Computational Science and Engineering
Romberg method
Same example again (cont.)
Romberg scheme columns 0 to 3
0.33978522855738
0.35308386657870
0.35751519587192
0.35872647716421
0.35903678355577
0.35911484861929
0.35913439576246
0.35751674591915
0.35899230563633
0.35913023759497
0.35914021901962
0.35914087030714
0.35914091147685
0.35909067628414
0.35913943305888
0.35914088444793
0.35914091372630
0.35914091422149
0.35914091422123
0.35914091422951
0.35914091422952
0.35914020697594
0.35914090748585
0.35914091419104
0.35914091422935
columns 4 to 6
0.35914091023295
0.35914091421734
0.35914091422950
NumCSE, Lecture 23, Dec 4, 2014
25/40
Numerical Methods for Computational Science and Engineering
Romberg method
Another example
Lets integrate
R1
0
x dx = 2/3 by Romberg method.
[I,T]=romberg(@sqrt,0,1,1e-8)
generates a warning message that 15 extrapolation steps were not
sufficient to meet the tolerance of 108 .
romberg is hardly more accurate than simpson!
Reason: the function f (x) = x is not differentiable at x = 0
= the asymptotic expansion T (0) = K0 + K1 h2 + does not
hold.
NumCSE, Lecture 23, Dec 4, 2014
26/40
Numerical Methods for Computational Science and Engineering
Gauss quadrature
Gauss quadrature
I
I
I
So far: quadrature rules for fixed location of nodes
(i = a + ih) at which the function was evaluated.
Gauss quadrature rules allow the nodes to vary as well
The additional freedom can be used to increase the accuracy
of the quadrature rule.
We consider Gauss quadrature rules in the interval [1, 1].
A transformation
[a, b] 3 x =
a+b
2x b a
ba
t+
t =
[1, 1]
2
2
ba
may be needed.
Z b
Z
ba 1
ba
a+b
f (x) dx =
f
t+
dt.
2
2
2
a
1
NumCSE, Lecture 23, Dec 4, 2014
27/40
Numerical Methods for Computational Science and Engineering
Gauss quadrature
Gauss quadrature (cont.)
For a given number of nodes n, we want to find the location of
nodes k and the weights wk in the quadrature rule
Z
f (x) dx
1
n
X
wk f (k )
(Gauss)
k=1
that allows us to integrate exactly a polynomial of degree as high
as possible.
Theorem
The order m of an n-point quadrature rule (Gauss) is bounded by
2n 1.
NumCSE, Lecture 23, Dec 4, 2014
28/40
Numerical Methods for Computational Science and Engineering
Gauss quadrature
Gauss quadrature (cont.)
We consider the case n = 2 for deriving conditions on nodes and
weights.
Four degrees of freedom (2 nodes 1 , 2 / 2 weights w1 , w2 ).
We can satisfy 4 nonlinear equations: We derive a quadrature rule
that is exact for polynomials up to degree 3:
Z1
Z1
1 dx = 2 = 1w1 + 1w2
Z1
x dx = 0 = 1 w1 + 2 w2
1
2
x dx = = 12 w1 + 22 w2
3
2
NumCSE, Lecture 23, Dec 4, 2014
Z1
x 3 dx = 0 = 13 w1 + 23 w2 .
29/40
Numerical Methods for Computational Science and Engineering
Gauss quadrature
Gauss quadrature (cont.)
The solution is
n
o
w1 = 1, w2 = 1, 1 = 1/ 3, 2 = 1/ 3
and the corresponding GaussLegendre rule for n = 2 is
Z 1
1
1
f (x) dx f
+f
3
3
1
This approach (solving a nonlinear system of 2n equations) doesnt
work for big n.
NumCSE, Lecture 23, Dec 4, 2014
30/40
Numerical Methods for Computational Science and Engineering
Gauss quadrature
GaussLegendre quadrature, general case
Let f (x) be a polynomial of degree at most 2n 1, i.e. f P2n1 .
Then, by polynomial division we can write
f (x) = q(x) n (x) + r (x),
where r (x), q(x) Pn1 are quotient and remainder polynomial
and n (x) Pn is the n-th Legendre polynomial.
Then
Z 1
Z 1
Z 1
f (x) dx =
[q(x) n (x) + r (x)] dx =
r (x) dx.
1
since n (x) is orthogonal to all polynomials of degree < n.
NumCSE, Lecture 23, Dec 4, 2014
31/40
Numerical Methods for Computational Science and Engineering
Gauss quadrature
GaussLegendre quadrature, general case (cont.)
Let 1 < 1 < < n < 1 be the zeros of n (x) (Gauss points).
Also, let the weights be given by
Z 1
Z 1Y
n
x j
dx
wi =
Li (x) dx =
1
1 j=1 i j
j6=i
where the Li (x) are Lagrange polynomials. Then,
n
X
wi f (i ) =
i=1
n
X
i=1
Z 1
wi [q(i ) n (i ) + r (i )] =
wi r (i )
i=1
n
X
f (x) dx.
r (i )Li (x) dx =
1 i=1
NumCSE, Lecture 23, Dec 4, 2014
n
X
{z
r (x)
32/40
Numerical Methods for Computational Science and Engineering
Gauss quadrature
GaussLegendre quadrature, general case (cont.)
I
Advantage of Gaussian quadrature:
Exact integration of polynomials of high order.
I
I
Disadvantage of Gaussian quadrature: integration points xi
depend on n.
I
I
n function evaluations for order 2n.
Used for exact integration of polynomials.
Function values f (xi ) cannot be reused if n is increased.
Function values f (xi ) cannot be reused if composite rule is
refined.
Variants GaussLobato/GaussRadau that include interval
endpoints.
I
Prescribed abscissae reduce order of precision.
NumCSE, Lecture 23, Dec 4, 2014
33/40
Numerical Methods for Computational Science and Engineering
Adaptive Quadrature
Adaptive Quadrature
I
So far: we used composite rules in which all integration steps
had same size h.
Controlled integration error by halving the step size h h/2.
We generated equidistant subintervals and applied the same
quadrature rule to each subinterval.
In each subinterval there will be a different error depending on
the local behavior of the function.
The global error will be localized
To minimize the number of function evaluations, need to
adapted length of subintervals to local behavior of integrand.
The partition should be such that the error becomes roughly
the same for each subinterval.
NumCSE, Lecture 23, Dec 4, 2014
34/40
Numerical Methods for Computational Science and Engineering
Adaptive Quadrature
Adaptive Quadrature: Example
Lets again consider
R1
0
x dx = 2/3.
T=trapez(@sqrt,0,1,1e-5) needs 2048 function evaluations.
T=trapez(@sqrt,0,1,1e-10) needs 4194304 function
evaluations to complete.
Most of the error is in the leftmost interval. For the second
subinterval [h, 2h] we have (since maxhx2h |f 00 (x)| = h3/2 /4)
|I T (h)| =
|f 00 ()| 3 h3/2
h
= 6.357 107 < 105 .
12
48
For the last subinterval [1 h, 1] the analogous error estimate
gives the bound 1.94 1011 .
NumCSE, Lecture 23, Dec 4, 2014
35/40
Numerical Methods for Computational Science and Engineering
Adaptive Quadrature
Adaptive Quadrature: Example (cont.)
In an adaptive approach one proceeds as follows (according to the
principle of divide and conquer):
I
First, integrate f using two different numerical integration
schemes (or the same method with two different step sizes) to
get the approximations I1 and I2 .
If I1 / I2 one accepts I1 as value of integral (|I2 I1 | < tol).
Otherwise, the interval [a, b] is divided, e.g., into two equal
parts [a, m] and [m, b], where m = (a + b)/2, and the two
respective integrals are computed (conquered) independently.
NumCSE, Lecture 23, Dec 4, 2014
36/40
Numerical Methods for Computational Science and Engineering
Adaptive Quadrature
Adaptive Quadrature: Example (cont.)
function [Q,fcount] = quadtx(F,a,b,tol,varargin)
...
% Initialization
c = (a + b)/2;
fa = F(a,varargin{:});
fc = F(c,varargin{:});
fb = F(b,varargin{:});
% Recursive call
[Q,k] = quadtxstep(F, a, b, tol, fa, fc, fb, varargin{:});
fcount = k + 3;
From Moler: Numerical Computing with Matlab.
NumCSE, Lecture 23, Dec 4, 2014
37/40
Numerical Methods for Computational Science and Engineering
Adaptive Quadrature
Adaptive Quadrature: Example (cont.)
function [Q,fcount] = quadtxstep(F,a,b,tol,fa,fc,fb,varargin)
% Recursive subfunction used by quadtx.
h = b - a;
c = (a + b)/2;
fd = F((a+c)/2,varargin{:});
fe = F((c+b)/2,varargin{:});
Q1 = h/6 * (fa + 4*fc + fb);
Q2 = h/12 * (fa + 4*fd + 2*fc + 4*fe + fb);
if abs(Q2 - Q1) <= tol
Q = Q2 + (Q2 - Q1)/15;
fcount = 2;
else
[Qa,ka] = quadtxstep(F, a, c, tol, fa, fd, fc, varargin{:});
[Qb,kb] = quadtxstep(F, c, b, tol, fc, fe, fb, varargin{:});
Q = Qa + Qb;
fcount = ka + kb + 2;
end
NumCSE, Lecture 23, Dec 4, 2014
38/40
Numerical Methods for Computational Science and Engineering
Adaptive Quadrature
Adaptive Quadrature: Example (cont.)
How should tol be chosen in the two subroutines calls in
quadtxstep?
tol/2 in each subinterval? This is too conservative.
Errors in general overestimated. (Extrapolated Simpsons rule)
We can allow one of the two recursive calls to have an error close
to the tolerance because the other subinterval will probably have a
much smaller error. For these reasons, the same value of tol is
used in each recursive call.
See choice of tol in quadtxstep.
NumCSE, Lecture 23, Dec 4, 2014
39/40
Numerical Methods for Computational Science and Engineering
Adaptive Quadrature
Comparison
With f (x) =
xe x
we get
(x + 1)2
trapez(f,0,1,1e-8): 8192 function calls
trapez(@sqrt,0,1,1e-8): 262144 function calls
simpson(f,0,1,1e-8): 128 function calls
simpson(@sqrt,0,1,1e-8): 131072 function calls
romberg(f,0,1,1e-8): 64 function calls
romberg(@sqrt,0,1,1e-8): 131072 function calls
quadtx(f,0,1,1e-8): 61 function calls
quadtx(@sqrt,0,1,1e-8): 153 function calls
NumCSE, Lecture 23, Dec 4, 2014
40/40
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
1. Single-step methods
I
I
I
Forward / backward Euler
Midpoint
Trapezoidal
2. Theory for single-step methods
References
I
U. Ascher & C. Greif: Numerical Methods, 16.116.2.
C. Moler: Numerical Computing with Matlab. The
MathWorks. 2004. (Chapter 7: Ordinary Differential
Equations.)
NumCSE, Lecture 24, Dec 8, 2014
2/37
Numerical Methods for Computational Science and Engineering
Introduction
Differential Equations
I
Differential equations arise in all branches of science and
engineering, economics, finance, computer science.
Relate physical state to rate of change, e.g., rate of change of
particle is velocity,
dx(t)
= v(t) = g(t, x),
dt
a < t < b.
Ordinary differential equation (ODE): one independent
variable (time).
Partial differential equation (PDE): several independent
variables.
NumCSE, Lecture 24, Dec 8, 2014
3/37
Numerical Methods for Computational Science and Engineering
Introduction
Differential Equations (cont.)
Example: f (t, y ) = y + t defined for t 0 and any y R gives
y 0 (t) = y + t,
t 0.
Solution is
y (t) = t 1 + e t ,
any R.
Additional condition needed to make solution unique: initial
condition:
y (0) = c
NumCSE, Lecture 24, Dec 8, 2014
y (t) = t 1 + (c + 1)e t .
4/37
Numerical Methods for Computational Science and Engineering
Introduction
Differential Equations (cont.)
I
Initial value problem for ordinary differential equation (ODE):
find function y (t) that satisfies
y 0 (t) =
dy (t)
= f (t, y (t))
dt
(1)
together with initial condition
y (t0 ) = y0 .
(2)
that makes the solution unique.
I
Note that y (t) is only defined implicitly, through the
differential equation. This makes the task of approximating it
more difficult than in previous chapters.
NumCSE, Lecture 24, Dec 8, 2014
5/37
Numerical Methods for Computational Science and Engineering
Introduction
Differential Equations (cont.)
Fundamental theorem of calculus gives important connection
between differential equations and integrals:
Z t
f (s, y (s)) ds,
a t b.
y (t) = c +
a
We cannot use numerical quadrature directly because we do not
know the function y (s) (except if f (s, y ) = f (s)).
NumCSE, Lecture 24, Dec 8, 2014
6/37
Numerical Methods for Computational Science and Engineering
Examples
Harmonic oscillator
Example: Pendulum (Harmonic oscillator)
Newtons law of motion (masses
accelerations = forces) gives
d 2 (t)
00 (t) = g sin((t))
dt 2
()
where g is the scaled constant of
gravity, e.g., g = 9.81, and t is time.
Second order differential equation can be written as system of two
first order equations.
(This can be extended easily to higher order differential equations.)
NumCSE, Lecture 24, Dec 8, 2014
7/37
Numerical Methods for Computational Science and Engineering
Examples
Harmonic oscillator
Example: Pendulum (Harmonic oscillator) (cont.)
I
Write () as first order ODE system:
y1 (t) = (t), y2 (t) = 0 (t). Then,
y10 = y2 ,
y20 = g sin(y1 ).
ODE in standard form for the pendulum
0 0
y2
y1
0
= f(t, y(t))
y (t) = 0 = 00 =
g sin(y1 )
y2
Initial values: (0) and 0 (0) are given.
An alternative not considered further is a boundary value
problem, where e.g. (0) and () are given.
NumCSE, Lecture 24, Dec 8, 2014
8/37
Numerical Methods for Computational Science and Engineering
Examples
Two-body problem
Example: Two-body problem
Two-body problem describes the orbit of one body under the
gravitational attraction of a much heavier body. Using Cartesian
coordinates, u(t) and v (t), centered in the heavy body, the
equations are (dimensionless)
u 00 (t) = u(t)/r (t)3 ,
v 00 (t) = v (t)/r (t)3 ,
p
with r (t) = u(t)2 + v (t)2 . The vector y(t) has four
components:
u(t)
u 0 (t)
v (t)
v 0 (t)
y(t) =
y0 (t) =
u 0 (t) ,
u(t)/r (t)3 .
v 0 (t)
v (t)/r (t)3
NumCSE, Lecture 24, Dec 8, 2014
9/37
Numerical Methods for Computational Science and Engineering
Examples
Predator-prey model
Example: Predator-prey model
A classical model in mathematical ecology is the LotkaVolterra
predator-prey model.
Predators and prey coexist in an ecosystem. Population of
prey/predators increase/decrease without the other. Population of
prey/predators decrease/increase with the other.
u 0 = ( v )u,
0
v = (u )v .
0
y = f(y)
u
with y =
,
v
(prey)
(predator)
( v )u
f(y) =
.
(u )v
Here we assume that all parameters , , , are positive.
NumCSE, Lecture 24, Dec 8, 2014
10/37
Numerical Methods for Computational Science and Engineering
Examples
Predator-prey model
Example: Predator-prey model (cont.)
NumCSE, Lecture 24, Dec 8, 2014
11/37
Numerical Methods for Computational Science and Engineering
Examples
Predator-prey model
NumCSE, Lecture 24, Dec 8, 2014
12/37
Numerical Methods for Computational Science and Engineering
Eulers method
Forward Euler
I
Simplest method(s) to solve the initial value ODE
y 0 (t) = f (t, y (t)),
y (a) = c.
Use method to demonstrate general concepts:
I
I
I
I
I
I
Method derivation
Explicit vs. implicit methods
Local truncation error and global error
Order of accuracy
Convergence
Absolute stability and stiffness.
NumCSE, Lecture 24, Dec 8, 2014
13/37
Numerical Methods for Computational Science and Engineering
Eulers method
Forward Euler: derivation
I
Mesh points t0 < t1 < < tN with
fixed step size h = tj+1 tj .
Proceed to march from one mesh point to next (step by step).
For simplest forward difference:
y 0 (tj ) =
y (tj+1 ) y (tj ) h 00
+ y (j ).
h
2
Therefore,
y (tj+1 ) = y (tj ) + h f (tj , y (tj )) +
NumCSE, Lecture 24, Dec 8, 2014
h2 00
y (j ).
2
14/37
Numerical Methods for Computational Science and Engineering
Eulers method
Forward Euler: derivation (cont.)
I
Approximate solution yj y(tj ).
Then, set
y0 = y (t0 ) = c,
yj+1 = yj + h f (tj , yj ),
j = 0, . . . , N 1.
Possible Matlab code for Eulers method that stores all
intermediate values.
t = [a:h:b];
y(1) = y0;
for j=1:N
y(j+1) = y(j) + h*f(t(j),y(j));
end
NumCSE, Lecture 24, Dec 8, 2014
15/37
Numerical Methods for Computational Science and Engineering
Eulers method
Forward Euler: simple example
I
For the problem y 0 = y , y (0) = 1, the exact solution is y (t) = e t .
Forward Euler: y0 = 1, yi+1 = (1 + h)yi , i = 0, 1, . . .
In typical figure, red is exact y (t), blue is approximate yi , h = 0.1.
Measuring error max1n1/h |yn y (tn )|, it behaves like O(h).
NumCSE, Lecture 24, Dec 8, 2014
16/37
Numerical Methods for Computational Science and Engineering
Eulers method
Forward Euler: nonlinear example
A system of 8 ODEs arising in plant physiology. In Matlabdefine
function f = hires(t,u)
%
% f = hires(t,u)
% High irradiance response function arising in plant physiology
f = u;
f(1) =
f(2) =
f(3) =
f(4) =
f(5) =
f(6) =
f(7) =
f(8) =
-1.71*u(1) + .43*u(2) + 8.32*u(3) + .0007;
1.71*u(1) - 8.75*u(2);
-10.03*u(3) + .43*u(4) + .035*u(5);
8.32*u(2) + 1.71*u(3) - 1.12*u(4);
-1.745*u(5) + .43*u(6) + .43*u(7);
-280*u(6)*u(8) + .69*u(4) + 1.71*u(5) - .43*u(6) + .69*u(7);
280*u(6)*u(8) - 1.81*u(7);
-280*u(6)*u(8) + 1.81*u(7);
NumCSE, Lecture 24, Dec 8, 2014
17/37
Numerical Methods for Computational Science and Engineering
Eulers method
Forward Euler: nonlinear example (cont.)
Integrate from a = 0 to b = 322 starting from
y(0) = y0 = (1, 0, 0, 0, 0, 0, 0, .0057)T .
Use the script (note: wasteful in terms of storage, but simple)
t = 0:.001:322; h = .001;
y = u * ones(1,length(t));
for i = 2:length(t)
y(:,i) = y(:,i-1) + h*hires(t(i-1),y(:,i-1));
end
plot(t,y(6,:))
Note the simplicity of the code, as well as the rather small h, which yields
322,000 time steps.
Execute Matlab program Example16_4Figure16_3.m.
NumCSE, Lecture 24, Dec 8, 2014
18/37
Numerical Methods for Computational Science and Engineering
Eulers method
Backward Euler: implicit vs. explicit
Instead of forward, we use backward difference
y 0 (tj+1 ) =
y (tj+1 ) y (tj ) h 00
+ y (j ).
h
2
So,
y (tj+1 ) = y (tj ) + h f (tj+1 , y (tj+1 )) + O(h2 ).
The initial value problem is then solved by
y0 = c,
yj+1 = yj + h f (tj , yj+1 ),
j = 0, . . . , N 1.
But now yi+1 appears implicitly! A (usually nonlinear) system of
equations to be solved for yj+1 :
NumCSE, Lecture 24, Dec 8, 2014
yj+1 yj h f (tj+1 , yj+1 ) = 0.
19/37
Numerical Methods for Computational Science and Engineering
Higher order one step methods
Trapezoidal rule
I
Euler Methods are only first order accurate: want higher
accuracy.
Write ODE y 0 = f (t, y ) as
Z
y (ti+1 ) = y (ti ) +
ti+1
f (t, y (t)) dt,
ti
and use basic quadrature rule.
I
Implicit trapezoidal method:
yj+1 = yj +
NumCSE, Lecture 24, Dec 8, 2014
h
(f (tj , yj ) + f (tj+1 , yj+1 )).
2
20/37
Numerical Methods for Computational Science and Engineering
Higher order one step methods
Trapezoidal rule (cont.)
In the explicit trapezoidal rule (Heuns method) we take a tentative
Euler step across the whole interval, then we average slopes:
s1 = f (tj , yj ),
s2 = f (tj + h, yj + h s1 ),
h
yj+1 = yj + (s1 + s2 ),
2
tj+1 = tj + h.
While Eulers methods have accuracy order 1 Heuns method has
accuracy order 2.
See below for the definition of order of accuracy.
NumCSE, Lecture 24, Dec 8, 2014
21/37
Numerical Methods for Computational Science and Engineering
Higher order one step methods
Midpoint rule
Let us try the analog to the midpoint rule for integrating functions.
This requires a second point of evaluation of f .
s1 = f (tj , yj ),
h
h
s2 = f (tj + , yj + s1 ),
2
2
yj+1 = yj + h s2 ,
tj+1 = tj + h.
The midpoint rule has accuracy order 2.
NumCSE, Lecture 24, Dec 8, 2014
22/37
Numerical Methods for Computational Science and Engineering
Higher order one step methods
Comparison
We compare the four methods (all except implicit trapezoidal rule)
for the initial value problem
y 0 (t) = 2ty 2 ,
y (0) = 1.
The exact solution is
1
.
1 + t2
We see that midpoint rule and trapezoidal rule are much more
accurate than both Euler methods.
y (t) =
NumCSE, Lecture 24, Dec 8, 2014
23/37
Numerical Methods for Computational Science and Engineering
Higher order one step methods
Comparison (cont.)
NumCSE, Lecture 24, Dec 8, 2014
24/37
Numerical Methods for Computational Science and Engineering
Theory on single-step methods
Theory on single-step methods
Let a = t0 < t1 < t2 < < tN = b, h = tj+1 tj for all j.
The initial value problem (1)(2) is solved by the single-step
method
yj+1 = yj + h (tj , yj , h),
y0 = y (t0 ).
(Ah )
In the case of (explicit) Euler we have (tj , yj , h) = f (tj , yj ).
In the case of the explicit trapezoidal rule we have
1
(t, y , h) = (f (t, y ) + f (t + h, y + h f (t, y )))
2
h f
f
= f (t, y ) +
(t, y ) +
(t, y ) f (t, y ) + O(h2 ).
2 t
y
NumCSE, Lecture 24, Dec 8, 2014
25/37
Numerical Methods for Computational Science and Engineering
Theory on single-step methods
Theory on single-step methods (cont.)
Definition (Local truncation error)
The local truncation error is defined by
h (t) :=
1
(y (t + h) y (t)) (t, y (t), h).
h
(3)
For any reasonable method we have that |h (t)| 0 as h 0.
This means that
lim
h0
1
(y (t + h) y (t)) = f (t, y ) = (t, y , 0)
h
for all (t, y ) S = [a, b] R.
NumCSE, Lecture 24, Dec 8, 2014
26/37
Numerical Methods for Computational Science and Engineering
Theory on single-step methods
Theory on single-step methods (cont.)
Definition (Consistency)
A single-step method (Ah ) is called consistent with (1)(2) if
lim (t, y , h) = f (t, y )
h0
for all (t, y ) S = [a, b] R and all f with bounded
f
y (t, y ).
Definition (Order of accuracy)
The single-step method (Ah ) has order of accuracy p if
h (t, y ) = O(hp )
for all (t, y ) S and f with bounded derivatives up to order p.
NumCSE, Lecture 24, Dec 8, 2014
27/37
Numerical Methods for Computational Science and Engineering
Theory on single-step methods
Theory on single-step methods (cont.)
Definition (Global error)
The global error is defined by
eh (tj ) = y (tj ) yj .
A Lipschitz condition is needed to show global convergence of a
method (Ah ): There is some 0 < L < such that
|(t, y , h) (t, y , h)| L|y y |
for all (t, y ) S = [a, b] R and all sufficiently small h.
Note: If we have to deal with ODE systems then |y | becomes kyk.
NumCSE, Lecture 24, Dec 8, 2014
28/37
Numerical Methods for Computational Science and Engineering
Theory on single-step methods
Theory on single-step methods (cont.)
We subtract the two equations
y (tj+1 ) = y (tj ) + h (tj , y (tj ), h) + h h (tj ),
(3)
and
yj+1 = yj + h (tj , yj , h).
(Ah )
Then
eh (tj+1 ) = eh (tj ) + h ((tj , y (tj ), h) (tj , yj , h)) + hh (tj )
Using the Lipschitz condition on we get
|eh (tj+1 )| |eh (tj )| + hL |y (tj ) yj | + h|h (tj )|
= (1 + hL)|eh (tj )| + h|h (tj )|
NumCSE, Lecture 24, Dec 8, 2014
29/37
Numerical Methods for Computational Science and Engineering
Theory on single-step methods
Theory on single-step methods (cont.)
Let
max|h (tj )| D.
j
Then,
|eh (tj+1 )| (1 + hL)|eh (tj )| + hD.
Applying this inequality recursively to j = n, n 1, . . . , 1 yields
|eh (tn )|
(1 + hL)n 1
D + (1 + hL)n |eh (t0 )|.
L
Assuming that eh (t0 ) = y (t0 ) y0 = 0 and since 1 + hL e hL we
get
D
|eh (tn )| (e nhL 1).
L
NumCSE, Lecture 24, Dec 8, 2014
30/37
Numerical Methods for Computational Science and Engineering
Theory on single-step methods
Theory on single-step methods (cont.)
If we can bound the local truncation error by
|h (tj )| Mhp
then we get
Theorem
The error of the method (Ah ) at tn = nh + t0 is
|eh (tn )|
M hp L(tn t0 )
(e
1).
L
The local order of convergence determines the global order of
convergence.
NumCSE, Lecture 24, Dec 8, 2014
31/37
Numerical Methods for Computational Science and Engineering
Theory on single-step methods
Absolute stability
I
Convergence is for h 0, but in computation the step size is
finite and fixed. How is the method expected to perform?
Consider simplest test equation model problem
y 0 (t) = y ,
y (0) = y0 ,
Solution: y (t) = y0 e t .
I
Forward Euler: yi+1 = (1 + h)yi = = (1 + h)i+1 y0 .
If > 0 then y (t) grows. So does numerical approximation.
Equal qualitative behavior.
NumCSE, Lecture 24, Dec 8, 2014
32/37
Numerical Methods for Computational Science and Engineering
Theory on single-step methods
Absolute stability (cont.)
I
If < 0 then y (t) decreases.
The numerical approximation decreases if |1 + h| < 1.
Or,
1 < 1 + h < 1
The right inequality is always satisfied (h > 0, < 0).
The left inequality is critical:
For < 0 must require
h
NumCSE, Lecture 24, Dec 8, 2014
2
||
33/37
Numerical Methods for Computational Science and Engineering
Theory on single-step methods
Absolute stability and stiffness
I
I
I
The restriction on the step size is an absolute stability
requirement.
Note: its a stability, not accuracy, requirement.
If absolute stability requirement is much more restrictive than
accuracy requirement, the problem is called stiff.
Example
y 0 = 1000(y cos(t)) sin(t),
y (0) = 1,
Exact solution y (t) = cos(t): varies slowly and smoothly.
And yet, applying forward Euler, must require
2
= 0.002,
1000
or roundoff error rapidly builds up, leading to useless results.
h
NumCSE, Lecture 24, Dec 8, 2014
34/37
Numerical Methods for Computational Science and Engineering
Theory on single-step methods
Stiffness
Definition
The initial-value ODE problem is stiff if the step size needed to
maintain absolute stability of the forward Euler method is much
smaller than the step size needed to represent the solution
accurately.
Remark: The example of the previous slide would require h 0.1
to accuratly represent cos(t). It is the stability of forward Euler
that requires h 0.002.
Remark: The example can be made even stiffer by increasing the
constant 1000.
NumCSE, Lecture 24, Dec 8, 2014
35/37
Numerical Methods for Computational Science and Engineering
Theory on single-step methods
Backward Euler and implicit methods
I
Apply backward Euler to the model problem
yi+1 = yi + hyi+1
yi+1 =
1
yi .
1 h
Here |yi+1 | |yi | for any h > 0 and < 0: no annoying
absolute stability restriction.
For the example of previous slide, integrating from 0 to /2
using forward Euler with h .001 leads to blowup.
Integrating using backward Euler with h = .001 gives error
3.2 109 . Using backward Euler with h = .1 still results in
respectable error 1.7 105 .
NumCSE, Lecture 24, Dec 8, 2014
36/37
Numerical Methods for Computational Science and Engineering
Theory on single-step methods
Stiff problems more generally
I
I
I
Stiff systems do arise a lot in practice.
If problem is very stiff, explicit methods are not effective.
Simplest case of a system: y0 = Ay, with A a constant m m
diagonalizable matrix.
There is a similarity transformation T so that
T 1 AT = diag(1 , . . . , n ).
Then for x = T 1 y obtain m test equations
x0j = xj ,
I
I
j = 1, . . . , m.
For forward Euler must require |1 + hj | 1, j = 1, . . . , m.
The big complication: the eigenvalues j may be complex!
NumCSE, Lecture 24, Dec 8, 2014
37/37
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
1. RungeKutta methods
2. Stability of single-step methods
3. Multistep methods
References
I
U. Ascher & C. Greif: Numerical Methods, 16.316.5.
SIAM, 2012.
C. Moler: Numerical Computing with Matlab. The
MathWorks. 2004. (Chapter 7: Ordinary Differential
Equations.)
NumCSE, Lecture 25, Dec 11, 2014
2/44
Numerical Methods for Computational Science and Engineering
Introduction
Introduction
Initial value problem for ordinary differential equation (ODE): find
function y (t) that satisfies
y0 (t) =
dy(t)
= f (t, y(t))
dt
(1)
together with initial condition
y(t0 ) = y0 .
NumCSE, Lecture 25, Dec 11, 2014
(2)
3/44
Numerical Methods for Computational Science and Engineering
Single-step methods
Single-step methods: Explicit/implicit Euler
The simplest numerical method for the solution of initial value
problems is the explicit Euler method.
yj+1 = yj + h f(tj , yj ),
tj+1 = tj + h.
with initial conditions y0 = y(t0 ).
The implicit Euler method is
yj+1 = yj + h f(tj+1 , yj+1 ),
tj+1 = tj + h.
Both methods have local trucation error O(h).
NumCSE, Lecture 25, Dec 11, 2014
4/44
Numerical Methods for Computational Science and Engineering
Single-step methods
Single-step methods: 2nd order methods
Midpoint rule
s1 = f(tj , yj ),
h
h
s2 = f(tj + , yj + s1 ),
2
2
yj+1 = yj + h s2 ,
tj+1 = tj + h.
Explicit trapezoidal rule
s1 = f(tj , yj ),
s2 = f(tj + h, yj + h s1 ),
h
yj+1 = yj + (s1 + s2 ),
2
tj+1 = tj + h.
Both methods have local trucation error O(h2 ).
NumCSE, Lecture 25, Dec 11, 2014
5/44
Numerical Methods for Computational Science and Engineering
Single-step methods
2nd order methods
The two previous methods can be written as
(t, y, h) := a1 k1 + a2 k2 = a1 f(t, y) + a2 f(t + p1 h, y + p2 hf(t, y)),
Using a Taylor series expansion, we get
(t, y, h) = (a1 +a2 )f(t, y)+a2 h[p1 ft (t, y)+p2 fy (t, y)f(t, y)]+O(h2 ).
Since the exact solution y(t) of the ODE satisfies
y(t + h) = y(t) + hy0 (t) +
h2 00
hp
y (t) + + y(p) (t + h)
2
p!
y(t + h) y(t)
h
= f(t, y) + [ft (t, y) + fy (t, y)f(t, y)] +
h
2
NumCSE, Lecture 25, Dec 11, 2014
6/44
Numerical Methods for Computational Science and Engineering
Single-step methods
2nd order methods (cont.)
We need to have
a1 + a2 = 1,
1
a2 p1 = ,
2
a2 p2 =
1
2
to get 2nd order methods.
Midpoint rule: a1 = 0, a2 = 1, p1 = p2 = 12 .
Trapezoidal rule: a1 = 21 , a2 = 21 , p1 = p2 = 1.
NumCSE, Lecture 25, Dec 11, 2014
7/44
Numerical Methods for Computational Science and Engineering
Single-step methods
RungeKutta methods
Classical RungeKutta method
Systematic derivation of single-step methods of higher order go
back to Runge and Kutta (1905). The classical RungeKutta
method (1895) is obtained in a more involved approach:
1
(t, y, h) := [k1 + 2k2 + 2k3 + k4 ]
6
with
k1 := f(t, y),
1
1
k2 := f(t + h, y + hk1 ),
2
2
1
1
k3 := f(t + h, y + hk2 ),
2
2
k4 := f(t + h, y + hk3 ).
A cumbersome computation gives h (t, y) = O(h4 ).
NumCSE, Lecture 25, Dec 11, 2014
8/44
Numerical Methods for Computational Science and Engineering
Single-step methods
RungeKutta methods
Classical RungeKutta method (cont.)
function [t,y] = rk4(f,tspan,y0,h)
%
% [t,y] = rk4(f,tspan,y0,h)
%
% A simple integration routine to solve the
% initial value ODE
y = f(t,y), y(a) = y0,
% using the clascical 4-stage Runge-Kutta method
% with a fixed step size h.
% tspan = [a b] is the integration interval.
% Note that y and f can be vector functions
y0 = y0(:);
% make sure y0 is a column vector
m = length(y0); % problem size
t = tspan(1):h:tspan(2); % output abscissae
N = length(t)-1; % number of steps
NumCSE, Lecture 25, Dec 11, 2014
9/44
Numerical Methods for Computational Science and Engineering
Single-step methods
RungeKutta methods
Classical RungeKutta method (cont.)
y = zeros(m,N+1);
y(:,1) = y0; % initialize
% Integrate
for i=1:N
% Calculate the four stages
K1 = feval(f, t(i),y(:,i)
);
K2 = feval(f, t(i)+.5*h, y(:,i)+.5*h*K1);
K3 = feval(f, t(i)+.5*h, y(:,i)+.5*h*K2);
K4 = feval(f, t(i)+h,
y(:,i)+h*K3
);
% Evaluate approximate solution at next step
y(:,i+1) = y(:,i) + h/6 *(K1+2*K2+2*K3+K4);
end
NumCSE, Lecture 25, Dec 11, 2014
10/44
Numerical Methods for Computational Science and Engineering
Single-step methods
RungeKutta methods
Example with known solution
y 0 = y 2 , y (0) = 1 = y (t) = 1/t.
Table shows absolute errors at t = 1 as well as observed
convergence rates. To obtain small errors, higher order methods
are more effective.
h
0.2
0.1
0.05
0.02
0.01
0.005
0.002
Euler
4.7e-3
2.3e-3
1.2e-3
4.6e-3
2.3e-4
1.2e-4
4.6e-5
rate
RK2
3.3e-4
7.4e-5
1.8e-5
2.8e-6
6.8e-7
1.7e-7
2.7e-8
1.01
1.01
1.00
1.00
1.00
1.00
where rate(h) = log2 e(2h)
e(h)
NumCSE, Lecture 25, Dec 11, 2014
rate
2.15
2.07
2.03
2.01
2.01
2.00
RK4
2.0e-7
1.4e-8
8.6e-10
2.2-11
1.4e-12
8.7e-14
1.9e-15
rate
3.90
3.98
4.00
4.00
4.00
4.19
11/44
Numerical Methods for Computational Science and Engineering
Single-step methods
RungeKutta methods
Example: predator-prey model
We integrate the following system of ODEs from a = 0 to b = 100
with step size h = 0.01.
y10 = .25y1 .01y1 y2 ,
y1 (0) = 80
y20
y2 (0) = 30
= y2 + .01y1 y2 ,
Prey: y1 , predator: y2
NumCSE, Lecture 25, Dec 11, 2014
Phase plane: y1 vs. y2
12/44
Numerical Methods for Computational Science and Engineering
Single-step methods
RungeKutta methods
RungeKutta methods
Definition: A s-stage (explicit) RungeKutta method is a
single-step method where (t, y, h) is defined by the
2s 1 + s(s 1)/2 real numbers ij , 2 i s, 1 j < i, and
c1 , . . . , cs , 2 , . . . , s :
(t, y, h) := c1 k1 + + cs ks
where
k1 := f(t, y),
k2 := f(t + 2 h, y + h21 k1 ),
k3 := f(t + 3 h, y + h(31 k1 + 32 k2 )),
..
.
ks := f(t + s h, y + h(s1 k1 + + s,s1 ks1 )).
NumCSE, Lecture 25, Dec 11, 2014
13/44
Numerical Methods for Computational Science and Engineering
Single-step methods
RungeKutta methods
RungeKutta methods (cont.)
The RungeKutta methods are characterized by the (Butcher)
table
0
2 21
3 31 32
..
..
..
.
.
.
s
s1 s2
c1 c2
s,s1
cs1 cs
(t, y, h) is consistent if c1 + c2 + + cs = 1.
For high order approximation we have to have i =
i1
P
i` .
`=1
NumCSE, Lecture 25, Dec 11, 2014
14/44
Numerical Methods for Computational Science and Engineering
Single-step methods
RungeKutta methods
RungeKutta methods (cont.)
Butcher scheme for some explicit RK s-stage methods
I
Explicit Euler method (order 1)
Explicit trapezoidal rule (order 2)
0 0
1
0
I
Explicit midpoint rule (order 2)
NumCSE, Lecture 25, Dec 11, 2014
1
2
0 0
1 1
0
0
1
2
1
2
0 0
1
2 0
0 1
15/44
Numerical Methods for Computational Science and Engineering
Single-step methods
RungeKutta methods
RungeKutta methods (cont.)
0
Classical 4th-order RK s-stage methods: 21
1
(order 4,
2
related to Simpson quadrature rule)
1
Kuttas 3/8-rule: (order 4)
1
2
0
0
0
0
0
0
1
0
0
0
0
0
0
1
6
2
6
2
6
1
6
0
0
1
1 1
0
0
0
1
0
0
0
0
1
8
3
8
1
8
1
2
1
3
2
3
1
3
31
NumCSE, Lecture 25, Dec 11, 2014
3
8
16/44
Numerical Methods for Computational Science and Engineering
Single-step methods
RungeKutta methods
Semi-implicit RungeKutta methods
In explicit methods the ij matrix in the Butcher table is strictly
lower triangular.
In fully-implicit RK methods the ij matrix is full.
It is difficult to find such methods and expensive to execute.
Therefore, fully-implicit RK methods are applied rarely, e.g., with
particularly stiff systems.
In semi-implicit RK methods the ij matrix is lower triangular with
nonzero diagonal.
NumCSE, Lecture 25, Dec 11, 2014
17/44
Numerical Methods for Computational Science and Engineering
Single-step methods
Stability
Stability of single-step methods
We have earlier noticed problems with the (explicit) Euler method
when solving the so-called model problem
y 0 (t) = y ,
y (0) = 1,
C.
for negative Re(). The exact solution of the model equation is
y (t) = e t .
For forward Euler we have (yj y (tj ), tj = jh)
yj+1 = yj + h f (tj , yj ) = yj + hyj = F (h) yj .
with F (h) = 1 + h.
NumCSE, Lecture 25, Dec 11, 2014
18/44
Numerical Methods for Computational Science and Engineering
Single-step methods
Stability
Stability of single-step methods (cont.)
For the classical RungeKutta method we get
k1 = f (tj , yj ) = yj ,
1
k2 = f (tj + h, yj +
2
1
k3 = f (tj + h, yj +
2
1
1
hk1 ) = (yj + hk1 ) = ( +
2
2
1
1
hk2 ) = (yj + hk2 ) = ( +
2
2
1 2
h )yj ,
2
1 2 1 2 3
h + h )yj ,
2
2
1
1
k4 = f (tj + h, yj + hk3 ) = (yj + hk3 ) = ( + h2 + h2 3 + h3 4 )yj
2
4
For
yj+1 = yj +
h
{k1 + 2k2 + 2k3 + k4 }
6
we get
NumCSE, Lecture 25, Dec 11, 2014
19/44
Numerical Methods for Computational Science and Engineering
Single-step methods
Stability
Stability of single-step methods (cont.)
h
yj+1 = yj + {k1 + 2k2 + 2k3 + k4 }
6
1 2 2 1 3 3
1 4 4
= 1 + h + h + h + h yj
2
6
24
with
= F (h) yj
1
1
1
F (h) = 1 + h + h2 2 + h3 3 + h4 4 .
2
6
24
Note that these are the first five terms of the Taylor series of e h .
NumCSE, Lecture 25, Dec 11, 2014
20/44
Numerical Methods for Computational Science and Engineering
Single-step methods
Stability
Stability of single-step methods (cont.)
I
Multiplier F (h) is a good approximation of e h for small h
For important case of decreasing solutions where < 0 the yj
should decrease in accordance with y (tj ).
This is the case only if |F (h)| < 1.
Since F (h) is a polynomial we must have
|F (h)| if h
Therefore:
|F (h)| < 1 cannot hold for all negative values of h.
This holds similarly for the case C. If Re() < 1 then
e h is decaying.
NumCSE, Lecture 25, Dec 11, 2014
21/44
Numerical Methods for Computational Science and Engineering
Single-step methods
Stability
Stability of single-step methods (cont.)
Definition
1. For single-step methods for which the model problem leads to
an iteration yj+1 = F (h) yj , the domain
B := { C||F ()| < 1} ,
= h,
is called domain of absolute stability.
2. A method is called absolutely stable if the left complex half
plane is contained in B.
Remark:
The implicit trapezoidal rule is absolutely stable.
NumCSE, Lecture 25, Dec 11, 2014
22/44
Numerical Methods for Computational Science and Engineering
Single-step methods
Stability
Domains of absolute stability for RK methods
Stability regions for
q-stage explicit RK
methods of order
q = 1, 2, 3, 4.
Inner circle corresponds
to fwd Euler (q = 1).
The larger q, the larger
the stability region.
NumCSE, Lecture 25, Dec 11, 2014
23/44
Numerical Methods for Computational Science and Engineering
Single-step methods
Time-marching method
Time-marching method
Lets try to determine the temperature u(x) in a 1D room of
width 1, when the temperatures at the walls are 0 and 1. The heat
(diffusion) equation is
u
2u
=
,
t
x 2
u(0, t) = 0,
u(1, t) = 1.
The stationary equation is
2u
= 0,
x 2
u(0) = 0,
u(1) = 1.
Lets discretize this equation by finite differences with a grid of
mesh width x = 0.1.
NumCSE, Lecture 25, Dec 11, 2014
24/44
Numerical Methods for Computational Science and Engineering
Single-step methods
Time-marching method
Time-marching method (cont.)
% form matrix and right side of linear system
n=9;
dx=.1;
A = 2*eye(n)-diag(ones(n-1,1),-1)-diag(ones(n-1,1),1);
A = -A/(dx^2);
b=zeros(n,1); b(n) = -1/(dx^2);
% direct solve
u=A\b
plot(u)
What do we do if we do not want or cannot factor the matrix?
NumCSE, Lecture 25, Dec 11, 2014
25/44
Numerical Methods for Computational Science and Engineering
Single-step methods
Time-marching method
Time-marching method (cont.)
Lets solve the time dependent problem. With uj u(x, jt) we
get:
uj+1 uj
= Auj ,
u(0) = 0, u(1) = 1.
t
In matrix form this is (boundary conditions are in b)
uj+1 = uj t (b Auj ),
Lets try with Matlab. Note that the choice of t is crucial for
convergence. (Cf. convergence analysis for iterative methods.)
u0=rand(n,1);
u=u0; dt=1;
for i=1:100, u=u-dt*(b-A*u); plot(u); pause(.05), end
NumCSE, Lecture 25, Dec 11, 2014
26/44
Numerical Methods for Computational Science and Engineering
Single-step methods
Time-marching method
Time-marching method (cont.)
For forward Euler the stability requires a time step t with
t < 2/i ,
for all i.
Note, that max(i ) (x)2 !!!
The implicit trapezoidal rule (aka. Crank-Nicolson) is A-stable.
(I
t
t
A)uj+1 = (I +
A)uj + t b
2
2
Lets try it.
u=u0; dt=dx;
for i=1:50,
u = (eye(9)-(dt/2)*A)\((eye(9)+(dt/2)*A)*u - dt*b);
end
NumCSE, Lecture 25, Dec 11, 2014
27/44
Numerical Methods for Computational Science and Engineering
Single-step methods
Assessment of RungeKutta methods
Assessment of RungeKutta methods
Some of the advantages and disadvantages of RK methods:
+ Simplicity in concept and in starting the integration process.
+ Flexibility in varying the step size.
+ Flexibility in handling discontinuities in f and other events.
The number of function evaluations required for higher order
RK methods is high, compared to multistep methods.
Deriving higher order RK methods and proving their order of
accuracy can be challenging.
More involved and possibly more costly procedures are
required for stiff problems than for multistep methods.
NumCSE, Lecture 25, Dec 11, 2014
28/44
Numerical Methods for Computational Science and Engineering
Multistep methods
Multistep methods
I
The RungeKutta (RK) family of methods are one-step
methods, increasing the order of the method by repeated
evaluations of f in [ti , ti+1 ].
Another approach for increasing order is to use past values
yi+1j and corresponding fi+1j = f (ti+1j , yi+1j ),
j = 0, 1, . . . , s.
NumCSE, Lecture 25, Dec 11, 2014
29/44
Numerical Methods for Computational Science and Engineering
Multistep methods
Multistep methods (cont.)
I
The general linear multistep formula is
s
X
j=0
j yi+1j = h
s
X
j fi+1j .
j=0
Here, j , j are coefficients; set 0 = 1. The method is
explicit if and only if 0 = 0.
I
One-step s = 1 examples: forward Euler (1 = 1, 1 = 1),
backward Euler (1 = 1, 0 = 1), implicit trapezoid
(1 = 1, 0 = 1 = .5).
However, the explicit RK2 and RK4 are not in the linear
multistep form.
NumCSE, Lecture 25, Dec 11, 2014
30/44
Numerical Methods for Computational Science and Engineering
Multistep methods
Multistep methods (cont.)
I
Starting with the integral form
Z
y(ti+1 ) = y(ti ) +
ti+1
f(t, y(t)) dt.
ti
integrate polynomial interpolant of f (t, y ) using values
fi , fi1 , . . . , fi+1s , which gives an explicit AdamsBashforth
method of order s; or
fi+1 , fi , fi1 , . . . , fi+1s , which gives an implicit
AdamsMoulton method of order s + 1.
NumCSE, Lecture 25, Dec 11, 2014
31/44
Numerical Methods for Computational Science and Engineering
Multistep methods
AdamsBashforth methods
Example: explicit 2-step AdamsBashforth (s = 2)
The polynomial interpolates (ti1 , fi1 ) and (ti , fi ):
p(t) = p(ti + h) = fi + (fi fi1 ).
Integration gives
Z
ti+1
p(t) dt = h
ti
p(ti + h) d
0
[fi + (fi fi1 )] d = h
=h
0
3
1
fi fi1
2
2
h
[3fi fi1 ]
2
h (ti+1 ) = Ch2 y 000 (ti ) + O(h3 ). The method has order s = 2.
So, the method is yi+1 = yi +
NumCSE, Lecture 25, Dec 11, 2014
32/44
Numerical Methods for Computational Science and Engineering
Multistep methods
AdamsBashforth methods
Example: explicit 2-step AdamsBashforth (s = 2) (cont.)
y 0 = y 2 , y (0) = 1 = y (t) = 1/t.
Errors and observed rates for AdamsBashforth methods;
(s, q) denotes the s-step method of order q.
h
0.2
0.1
0.05
0.02
0.01
0.005
0.002
(1,1)error
4.7e-03
2.3e-03
1.2e-03
4.6e-04
2.3e-04
1.2e-04
4.6e-05
NumCSE, Lecture 25, Dec 11, 2014
rate
1.01
1.01
1.00
1.00
1.00
1.00
(2,2)error
9.3e-04
2.3e-04
5.7e-05
9.0e-06
2.3e-06
5.6e-07
9.0e-08
rate
2.02
2.01
2.01
2.00
2.00
2.00
(4,4)error
1.6e-04
1.2e-05
7.9e-07
2.1e-08
1.4e-09
8.6e-11
2.2e-12
rate
3.76
3.87
3.94
3.97
3.99
3.99
33/44
Numerical Methods for Computational Science and Engineering
Multistep methods
AdamsMoulton methods
Example: implicit 2-step AdamsMoulton (s = 2)
The polynomial interpolates (ti1 , fi1 ) and (ti , fi ) and (ti+1 , fi+1 ):
p(t) = p(ti + h) = fi + (fi fi1 ) + ( + 1)(fi+1 2fi + fi1 )
2
Integrating gives
Z ti+1
p(t) dt =
ti
h
[fi1 + 8fi + 5fi+1 ] .
12
Therefore the method is
yi+1 = yi +
h
[fi1 + 8fi + 5fi+1 ]
12
h (ti+1 ) = Ch3 y (4) (ti ) + O(h4 ). The method has order s + 1 = 3.
NumCSE, Lecture 25, Dec 11, 2014
34/44
Numerical Methods for Computational Science and Engineering
Multistep methods
AdamsMoulton methods
Example: implicit 2-step AdamsMoulton (s = 2) (cont.)
y 0 = y 2 , y (0) = 1 = y (t) = 1/t.
Errors and observed rates for AdamsMoulton methods;
(s, q) denotes the s-step method of order q.
h
0.2
0.1
0.05
0.02
0.01
0.005
0.002
(1,1)error
4.6e-03
2.3e-03
1.1e-03
4.6e-04
2.3e-04
1.2e-04
4.6e-05
NumCSE, Lecture 25, Dec 11, 2014
rate
0.99
1.00
1.00
1.00
1.00
1.00
(1,2)error
1.8e-04
4.5e-05
1.1e-05
1.8e-06
4.5e-07
1.1e-07
1.8e-08
rate
2.01
2.00
2.00
2.00
2.00
2.00
(3,4)error
1.1e-05
8.4e-07
5.9e-08
1.6e-09
1.0e-10
6.5e-12
1.7e-13
rate
3.73
3.85
3.92
3.97
3.98
3.99
35/44
Numerical Methods for Computational Science and Engineering
Multistep methods
Theory
Theorem on multistep method order
Let
C0 =
s
X
j ,
j=0
s
s
X
X
1
1
Ci = (1)i
j i j +
j i1 j ,
i!
(i 1)!
j=1
i = 1, 2, . . .
j=0
Then the linear multistep method has order p if and only if
C0 = C1 = = Cp = 0,
Cp+1 6= 0.
Furthermore, the local truncation error is given by
h (ti ) = Cp+1 hp y (p+1) (ti ) + O(hp+1 ).
NumCSE, Lecture 25, Dec 11, 2014
36/44
Numerical Methods for Computational Science and Engineering
Multistep methods
Theory
Deriving the AdamsBashforth 2-step method
Based on the theorem, lets derive the AB2 method.
We know: 0 = 1, 1 = 1, 0 = 0. Determine 1 and 2 by
C1 = (1) [1 + 1 + 2 ] = 0,
1
C2 =
1 + 1 + 2 2 = 0,
2
1
1
5
C3 = (1) 1 + (1 + 4 2 ) =
6= 0,
6
2
12
or
1 1
1 2
1
1
=
2
0.5
h (ti ) =
NumCSE, Lecture 25, Dec 11, 2014
3
1 = ,
2
1
2 = .
2
5 2 000
h y (ti ) + O(h3 ).
12
37/44
Numerical Methods for Computational Science and Engineering
Multistep methods
BDF methods
BDF methods
I
These methods are called backward differentiation formulas
(BDF), and use f value only at the unknown level i + 1.
Starting with the form
s
X
j yi+1j = h0 fi+1 .
j=0
determine the coefficients using polynomial interpolation of
yi+1 , yi , yi1 , . . . , yi+1s , which gives an implicit method of
order s.
I
BDF are implicit and absolutely stable = good for stiff
problems.
NumCSE, Lecture 25, Dec 11, 2014
38/44
Numerical Methods for Computational Science and Engineering
Multistep methods
BDF methods
BDF methods (cont.)
I
I
The 1-step BDF is backward Euler.
The theorem on multistep method orders provides BDF as
well.
For s = 2 we get (with 0 = 1)
C0 = 0 + 1 + 2 = 0
C1 = (1)(1 + 22 + 0 ) = 0
C2 = 1 + 42 = 0
So, the 2-step BDF becomes
1
2h
4
yi+1 = yi yi1 + f (ti+1 , yi+1 ).
3
3
3
(BDF2)
It has accuracy order 2.
NumCSE, Lecture 25, Dec 11, 2014
39/44
Numerical Methods for Computational Science and Engineering
Multistep methods
BDF methods
BDF methods (cont.)
y 0 = y 2 , y (0) = 1 = y (t) = 1/t.
Errors and calculated rates for BDF methods; (s, q) denotes the
s-step method of order q.
h
0.2
0.1
0.05
0.02
0.01
0.005
0.002
(1,1)error
4.6e-03
2.3e-03
1.1e-03
4.6e-04
2.3e-04
1.2e-04
4.6e-05
NumCSE, Lecture 25, Dec 11, 2014
rate
0.00
0.99
1.00
1.00
1.00
1.00
1.00
(2,2)error
7.3e-04
1.8e-04
4.5e-05
7.2e-06
1.8e-06
4.5e-07
7.2e-08
rate
0.00
2.02
2.00
2.00
2.00
2.00
2.00
(4,4)error
7.6e-05
6.1e-06
4.3e-07
1.2e-08
7.8e-10
4.9e-11
1.3e-12
rate
0.00
3.65
3.81
3.91
3.96
3.98
3.99
40/44
Numerical Methods for Computational Science and Engineering
Multistep methods
PECE methods
Predictor-corrector methods
I
Big advantage of multistep methods: cost per step w.r.t.
function evaluations.
AdamsBashforth s-step method has order s but severe
stability limitations. Only used with s 2.
Implicit AdamsMoulton s-step method has order s + 1 and is
more stable.
AdamsMoulton methods need solution of a nonlinear system.
Predictor-corrector methods compute a predictor by the
AdamsBashforth method and then correct just once by one
step of the AdamsMoulton method.
NumCSE, Lecture 25, Dec 11, 2014
41/44
Numerical Methods for Computational Science and Engineering
Multistep methods
PECE methods
Algorithm: predictor-corrector step (PECE)
At step i, given yi , fi , fi1 , . . . , fi+1s :
0 ,
1. Use an s-step AdamsBashforth method to calculate yi+1
calling the result the Predicted value.
0 = f (t
0
2. Evaluate fi+1
i+1 , yi+1 ).
0 for the
3. Apply an s-step AdamsMoulton method using fi+1
unknown, calling the result the Corrected value yi+1 .
4. Evaluate fi+1 = f (ti+1 , yi+1 ).
Note: The result of the last step is needed only in the subsequent
PECE iteration.
NumCSE, Lecture 25, Dec 11, 2014
42/44
Numerical Methods for Computational Science and Engineering
Multistep methods
PECE methods
Example: AdamsBashforthMoulton with s = 3
Prediction:
0
yj+1
= yj +
h
[5fj2 16fj1 + 23fj ]
12
Correction:
yj+1 = yj +
h
0
9f (tj+1 , yj+1
) + fj2 5fj1 + 19fj
24
s-step AdamsBashforthMoulton (ABM) methods have order
s + 1.
NumCSE, Lecture 25, Dec 11, 2014
43/44
Numerical Methods for Computational Science and Engineering
Multistep methods
Assessment of multistep methods
Assessment of multistep methods
I
Multistep methods are good for non-stiff problems, in
particular for high accuracy.
Important advantages of the multistep family are the cheap
high order PECE pairs with the local error estimation that
comes for free.
The important disadvantages are the need for additional
startup procedure and the relatively cumbersome adjustment
to local changes such as lower continuity, event location, and
drastically adapting the step size.
RK methods are more popular for their much higher flexibility.
However, for stiff problems, the BDF methods are still popular.
NumCSE, Lecture 25, Dec 11, 2014
44/44
Numerical Methods for Computational Science and Engineering
Introduction
Topics of today
1. Stiff problems
2. Step size control
3. Wrap-up of the whole lecture
References
I
U. Ascher & C. Greif: Numerical Methods, 16.516.6.
SIAM, 2012.
C. Moler: Numerical Computing with Matlab. The
MathWorks. 2004. (Chapter 7: Ordinary Differential
Equations.)
Gander, Gander & Kwok: Scientific Computing. Springer
2014. (Chapter 10)
NumCSE, Lecture 26, Dec 15, 2014
2/28
Numerical Methods for Computational Science and Engineering
Introduction
Introduction
Initial value problem for ordinary differential equation (ODE): find
function y (t) that satisfies
dy (t)
= f (t, y (t))
dt
(1)
together with initial condition
y (t0 ) = y0 .
(2)
Numerical solution to this problem generates a sequence of values
t0 , t1 , . . ., and a corresponding sequence of values y0 , y1 , . . ., so
that yn approximates solution y (tn ) at tn :
yn y (tn ), n = 0, 1, . . .
NumCSE, Lecture 26, Dec 15, 2014
3/28
Numerical Methods for Computational Science and Engineering
Stiff problems
Stiff problems
I
A problem is stiff if the solution being sought varies slowly, but
there are nearby solutions that vary rapidly, so the numerical
method must take small steps to obtain satisfactory results.
So, an unreasonably small step size h must be used for e.g.
forward Euler. Also explicit RK or multistep are inadequate
resort to appropriate implicit methods.
Stiffness is an efficiency issue. If we werent concerned with
how much time a computation takes, we wouldnt be
concerned about stiffness. (Nonstiff methods can solve stiff
problems; they just take a long time to do it.)
Method is A-stable if absolute stability region contains entire
left half z-plane, as backward Euler and implicit trapezoidal
rule.
Method is L-stable if |F (z)| 0 as |z| .
Backward Euler is L-stable, implicit trapezoidal rule is not.
NumCSE, Lecture 26, Dec 15, 2014
4/28
Numerical Methods for Computational Science and Engineering
Stiff problems
Stiff system
I
For a nonlinear ODE system, must look at the eigenvalues of
the Jacobian
f
f1
1
. . . y
y1
n
f
..
.
.
..
..
= .
J=
.
y
fm
fm
. . . yn
y1
In fact, if our solution y(t) is a perturbation of nearby solution
y(t) and w(t) = y(t)
y(t) then
f(y) = f(
y) + J(
y)w + O(kwk2 ) J(
y)w.
The propagation of the error is determined by the spectral
properties of the Jacobian.
NumCSE, Lecture 26, Dec 15, 2014
5/28
Numerical Methods for Computational Science and Engineering
Stiff problems
Stiff system (cont.)
I
It makes sense to investigate the model equation. Since
eigenvalues of J may be complex, we must consider C.
Moreover, the stability restriction for an explicit method may
dictate different step sizes at different t.
An implicit method requires the solution of a nonlinear
algebraic system at each step. Use some variant of Newtons
method (and not a cheap fixed point iteration).
NumCSE, Lecture 26, Dec 15, 2014
6/28
Numerical Methods for Computational Science and Engineering
Stiff problems
Example: the hires system
Return to the hires system of plant physiology, for which
J(y ) =
1.71 .43
8.32
0
0
0
0
0
1.71 8.75
0
0
0
0
0
0
0
0 10.03 .43
.035
0
0
0
0
8.32 1.71 1.12
0
0
0
0
0
0
0
1.745
.43
.43
0
0
0
0
.69
1.71 280y8 .43 .69 280y6
0
0
0
0
0
280y8
1.81 280y6
0
0
0
0
0
280y8
1.81 280y6
Initial condition: y(0) = y0 = (1, 0, 0, 0, 0, 0, 0, .0057)T .
NumCSE, Lecture 26, Dec 15, 2014
7/28
Numerical Methods for Computational Science and Engineering
Stiff problems
Example: the hires system (cont.)
I
At t = 0 where y = y0 , the eigenvalues are approximately
0, 10.4841, 8.2780, 0.2595, 0.5058,
2.6745 0.1499, 2.3147.
So, h = 0.1 is a safe choice for forward Euler.
At t 10.7, the eigenvalues are approximately
211.7697, 10.4841, 8.2780, 2.3923,
2.1400, 0.4907, 3 105 , 3 1012 .
Here, h = 0.005 is required for forward Euler.
This innocent-looking problem is mildly stiff.
NumCSE, Lecture 26, Dec 15, 2014
8/28
Numerical Methods for Computational Science and Engineering
Stiff problems
Example of a stiff problem
A model of flame propagation (Shampine), taken from Molers
book.
If one lights a match, the ball of flame grows rapidly until it reaches
a critical size. Then it remains at that size because the amount of
oxygen being consumed by the combustion in the interior of the
ball balances the amount available through the surface.
y = y 2 y 3 ,
y (0) = ,
0 t 2/.
Scalar variable y (t) represents the radius of the ball.
Critical parameter is the initial radius, , which is small. We
seek the solution over a length of time that is inversely
proportional to .
NumCSE, Lecture 26, Dec 15, 2014
9/28
Numerical Methods for Computational Science and Engineering
Stiff problems
Example of a stiff problem (cont.)
Solutions with non-stiff solver ode45.
delta = 0.01;
F = @(t,y) y^2 - y^3;
opts = odeset(RelTol,1.e-4);
ode45(F,[0 2/delta],delta,opts);
delta = 0.00001;
ode45(F,[0 2/delta],delta,opts);
Solution with stiff solver ode23s.
delta = 0.00001;
ode23s(F,[0 2/delta],delta,opts);
NumCSE, Lecture 26, Dec 15, 2014
10/28
Numerical Methods for Computational Science and Engineering
Stiff problems
Another example of a stiff problem
An artificial problem taken from the SchwarzK
ockler book.
0.5 32.6 25.7
4
0 48
9 y,
y0 = A y =
y0 = 13 .
0
9 72
1
The eigenvalues of A are 1 = 0.5, 2 = 45, and 3 = 75.
The exact solution can be written as
15
12
1
0.5t
45t
y(t) = 0 e
+ 12 e
+ 1 e 75t .
0
4
3
The system has very different decay rates.
NumCSE, Lecture 26, Dec 15, 2014
11/28
Numerical Methods for Computational Science and Engineering
Stiff problems
Another example of a stiff problem (cont.)
I
The component corresponding to 3 decays the fastest.
To have 4 digits accuracy we set h such that e 75h coincides with F (75h) of RK4 to 5 decimal digits: h 0.0025.
At t = 0.15: e 750.15 0.000013, much smaller than
e 450.15 0.001171.
From this point on, we do not need to integrate the e 75t
term accurately. It just must not blow up!
From Slide 25/23 we have h h = 2.78/75 0.0371.
We can proceed with h such that e 45h coincides with
F (45h) to five decimal digits: h 0.005.
At t = 0.3 also the e 45t term has become small that we just
have to make sure that it does not blow up. (h h is fine.)
Can solve the system with ode45 or ode23s.
NumCSE, Lecture 26, Dec 15, 2014
12/28
Numerical Methods for Computational Science and Engineering
Matlabs ODE solvers
Matlabs ODE solvers
Excerpt from Matlab Reference Manual.
ode45: Explicit RungeKutta (4,5) formula. Most of the time, try
ode45 first.
ode23: Explicit RungeKutta (2,3) pair. Often more efficient than
ode45 at crude tolerances and in presence of moderate stiffness.
ode113 variable-order AdamsBashforthMoulton PECE
algorithm. For problems with stringent error tolerances or for
solving computationally intensive problems.
ode23s. Stiff problems, low accuracy. Use for large error
tolerances with stiff systems or with a constant mass matrix.
NumCSE, Lecture 26, Dec 15, 2014
13/28
Numerical Methods for Computational Science and Engineering
Matlabs ODE solvers
Error control and estimation
I
Want to build a function which returns, given f (t, y ), initial
value information a, y (a), tolerance tol and a set of output
points t1 , . . . , tN , corresponding set of solution values
accurate at these points to within tol.
However, in contrast to quadrature, the error here propagates
and accumulates from one time step (or mesh subinterval) to
the next, hence the task of estimating the global error is much
more awkward.
Often in practice, a tolerance on the global error is not known,
and we only want to control the step size hi = ti+1 ti from
the current known (ti , yi ) to the next (ti+1 , yi+1 ).
NumCSE, Lecture 26, Dec 15, 2014
14/28
Numerical Methods for Computational Science and Engineering
Matlabs ODE solvers
Error control and estimation (cont.)
I
Thus, select hi so that the local error in the ith step
li+1 = y (ti+1 ) yi+1 ,
is below a (local) tolerance hi tol. Here, y (ti ) is the local
exact (unknown) ODE solution satisfying y (ti ) = yi .
There are examples where this sort of error control leads to
very large global errors, but in many applications good results
are obtained very efficiently!
NumCSE, Lecture 26, Dec 15, 2014
15/28
Numerical Methods for Computational Science and Engineering
Matlabs ODE solvers
Using a pair of RK methods
I
How can we estimate y (ti+1 ) effectively?
Suppose we use an RK method of order q for yi+1 , given yi .
Then use another RK method of order q + 1 to calculate yi+1 ,
given yi , and estimate
|li+1 | |
yi+1 yi+1 |.
So, at step i, with h = hi , if
|
yi+1 yi+1 | h tol
then accept step: set yi+1 yi+1 , i i + 1.
NumCSE, Lecture 26, Dec 15, 2014
16/28
Numerical Methods for Computational Science and Engineering
Matlabs ODE solvers
Using a pair of RK methods (cont.)
I
If step is not accepted, then decrease h to h
h = h
h tol
|
yi+1 yi+1 |
1
q
and repeat. This selection is based on the asymptotic
behavior li+1 hq+1 .
I
In the correction formula for h, there is a safety factor , e.g.
= 0.9.
NumCSE, Lecture 26, Dec 15, 2014
17/28
Numerical Methods for Computational Science and Engineering
Matlabs ODE solvers
Embedded methods and step size control
I
The local error |h (tj )| can be estimated in various ways. We
have seen applying the same method with different step sizes
h and h/2 in quadrature.
In RK methods we use methods with different stages/orders.
In order to reduce the work the two methods should use
mostly the same ki .
In the following we have a 5-stage 4th order method
embedded in a 6-stage 5th order method.
There are many variants of this kind. In particular Matlabs
ode45 see http:
//en.wikipedia.org/wiki/Dormand-Prince_method
NumCSE, Lecture 26, Dec 15, 2014
18/28
Numerical Methods for Computational Science and Engineering
Matlabs ODE solvers
Embedded methods and step size control (cont.)
k1 = f (tj , yj )
2
2
k2 = f (tj + h, yj + hk1 )
9
9
1
1
1
k3 = f (tj + h, yj + hk1 + hk2 )
3
12
4
3
69
243
135
k4 = f (tj + h, yj +
hk1 +
hk2 +
hk3 )
4
128
128
64
17
27
27
16
k5 = f (tj + h, yj + hk1 + hk2 + hk3 + hk4 )
12
4
5
15
1
9
16
1
yj+1 = yj + h
k1 + k3 + k4 + k5
9
20
45
12
NumCSE, Lecture 26, Dec 15, 2014
19/28
Numerical Methods for Computational Science and Engineering
Matlabs ODE solvers
Embedded methods and step size control (cont.)
5
65
5
13
4
5
k6 = f (tj + h, yj +
hk1 + hk2 + hk3 + hk4 +
hk5 )
6
432
16
16
27
144
47
12
32
1
6
yj+1 = yj + h
k1 + k3 +
k4 + k5 + k6
450
25
225
30
25
The error is estimated by k
yj+1 yj+1 k
h (tj+1 )
NumCSE, Lecture 26, Dec 15, 2014
h
k2k1 + 9k3 64k4 15k5 + 72k6 k
300
20/28
Numerical Methods for Computational Science and Engineering
Matlabs ODE solvers
Example with Matlab
Oscillation with increasing amplitude and frequency
y 0 = (sin(t 3 ) + 3t 3 cos(t 3 ))y ,
y (0) = 1.
The exact solution is
y = exp(t sin(t 3 ))
We solve the equation
with Matlabs ode45
which is an embedded
4th/5th order RK method.
NumCSE, Lecture 26, Dec 15, 2014
21/28
Numerical Methods for Computational Science and Engineering
Matlabs ODE solvers
Example with Matlab (cont.)
f=@(t,y) -(sin(t^3)+3*t^3*cos(t^3))*y
t=[0:.01:3];
y=exp(-t.*sin(t.^3));
ode45(f,[0 3],1)
[t,y]=ode45(f,[0 3],1);
plot(t,y)
plot(t(1:end-1),t(2:end)-t(1:end-1))
% options = odeset(reltol,0.001,abstol,1e-5,stats,on);
% options = odeset(reltol,0.01,abstol,1e-3,stats,on);
% [t,y]=ode45(f,[0 3],1,options)
NumCSE, Lecture 26, Dec 15, 2014
22/28
Numerical Methods for Computational Science and Engineering
Matlabs ODE solvers
Example 16.20: Restricted 3-body problem
Two bodies of masses = 0.012277471 and
= 1 (moon and
earth) in a planar motion, and a third body of negligible mass
(satellite) moving in the same plane.
u1
u1 +
,
D1
D2
u2
u2
u200 = u2 2u10
,
D1
D2
u100 = u1 + 2u20
D1 = ((u1 + )2 + u22 )3/2 ,
D2 = ((u1
)2 + u22 )3/2 .
Initial conditions:
u1 (0) = 0.994, u2 (0) = 0, u10 (0) = 0,
u20 (0) = 2.00158510637908.
Solution is periodic with period 17.0652 . . .
NumCSE, Lecture 26, Dec 15, 2014
23/28
Numerical Methods for Computational Science and Engineering
Matlabs ODE solvers
Example 16.20: Restricted 3-body problem (cont.)
I
RK4 with a uniform step size: 10,000 steps needed for a
qualitatively correct approximation.
ode45 with default tolerances: 309 steps required, with
max hi = 0.1892, min hi = 1.5921 107 .
NumCSE, Lecture 26, Dec 15, 2014
24/28
Numerical Methods for Computational Science and Engineering
Matlabs ODE solvers
Another Example with Matlab
We consider again the two body problem:
u 00 (t) = u(t)/r (t)3 ,
v 00 (t) = v (t)/r (t)3 ,
r (t) =
q
u(t)2 + v (t)2 .
function ydot = twobody(t,y,y0)
r = sqrt(y(1)^2 + y(2)^2);
ydot = [y(3); y(4); -y(1)/r^3; -y(2)/r^3];
options = odeset(reltol,0.001,abstol,1e-5,stats,on);
ode45(@twobody,[0 4],[-1;0;0.1;-0.1],options);
[t,y]=ode45(@twobody,[0 4],[-1;0;0.1;-0.1],options);
plot(t,y); plot(t(1:end-1),t(2:end)-t(1:end-1))
NumCSE, Lecture 26, Dec 15, 2014
25/28
Numerical Methods for Computational Science and Engineering
Epilog
Epilog
I
Review of the lecture
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
Numerical algorithms (AG 1)
Roundoff errors (AG 2)
Nonlinear equations in one variable (AG 3)
Direct methods for linear system (AG 4.14.3/5)
Linear least squares problems (AG 6)
Iterative methods for linear system (AG 7)
Eigenvalues and singular values (AG 8)
Nonlinear systems and optimization (AG 9.19.2)
(Piecewise) polynomial interpolation (AG 10.15/11.14),
Bezier curves
Best approximation (AG 12)
Fourier transform (AG 13), Filtering
Numerical integration (AG 15.15)
Ordinary differential equations (AG 16.16)
NumCSE, Lecture 26, Dec 15, 2014
26/28
Numerical Methods for Computational Science and Engineering
Epilog
Epilog (cont.)
I
Examination
I
I
I
I
I
I
Saturday 24 January 9:0012:00
Mainly on computer
AscherGreif book and lecture notes are available as PDF.
No aids
Questions similar to assignments.
Room accessible 10 min before start of exam.
Questions and answers session on Tuesday 13 January
14:0016:00 CAB G51.
NumCSE, Lecture 26, Dec 15, 2014
27/28
Numerical Methods for Computational Science and Engineering
Epilog
Epilog (cont.)
Merry Christmas and a happy new year!
Picture from www.springer.com.
NumCSE, Lecture 26, Dec 15, 2014
28/28
You might also like
- Computer Awareness Capsule For IBPS RRB PO (By Studyniti)100% (3)Computer Awareness Capsule For IBPS RRB PO (By Studyniti)59 pages
- Numerical Techniques in Electromagnetics - Sadiku PDF75% (4)Numerical Techniques in Electromagnetics - Sadiku PDF750 pages
- Numerical Methods: Intro To Numerical Methods & Intro To Matlab Programming100% (1)Numerical Methods: Intro To Numerical Methods & Intro To Matlab Programming45 pages
- An Introduction To Numerical Methods For The Physical SciencesNo ratings yetAn Introduction To Numerical Methods For The Physical Sciences168 pages
- Paul DeVries A First Course in Computational Physics100% (1)Paul DeVries A First Course in Computational Physics435 pages
- SP Curr Advanced End User Computing 5 20No ratings yetSP Curr Advanced End User Computing 5 2034 pages
- (Graduate Texts in Mathematics 103) Serge Lang-Complex Analysis-Springer (2003)100% (5)(Graduate Texts in Mathematics 103) Serge Lang-Complex Analysis-Springer (2003)504 pages
- Zhi Dong Bai, Random Matrix Theory and Its Applications100% (1)Zhi Dong Bai, Random Matrix Theory and Its Applications174 pages
- Lab - Introduction To Finite Element Methods and Matlab'S Pdetoolbox100% (1)Lab - Introduction To Finite Element Methods and Matlab'S Pdetoolbox15 pages
- John P. D'Angelo, Several Complex Variables and The Geometry of Real HypersurfacesNo ratings yetJohn P. D'Angelo, Several Complex Variables and The Geometry of Real Hypersurfaces288 pages
- CH E 374 Computational Methods in EngineeringNo ratings yetCH E 374 Computational Methods in Engineering4 pages
- Numerical Methods For Computational Science and Engineering: Lecture 1, Sept 19, 2013: IntroductionNo ratings yetNumerical Methods For Computational Science and Engineering: Lecture 1, Sept 19, 2013: Introduction40 pages
- Lecture 6,7 Ethical Hacking Framework and FootprintingNo ratings yetLecture 6,7 Ethical Hacking Framework and Footprinting16 pages
- Coding With Lua Cheatsheet: Create New Scripts Run CodeNo ratings yetCoding With Lua Cheatsheet: Create New Scripts Run Code2 pages
- Multivariable and Vector Calculus 696f PDF100% (11)Multivariable and Vector Calculus 696f PDF319 pages
- Foundations of Geometry - Venema, G., Second Edition93% (15)Foundations of Geometry - Venema, G., Second Edition407 pages
- Statistical Regression and Classification - From Linear Models To Machine Learning100% (10)Statistical Regression and Classification - From Linear Models To Machine Learning532 pages
- A Concise Introduction To Statistical Inference100% (5)A Concise Introduction To Statistical Inference231 pages
- Purbanchal University: Biratangar, NepalNo ratings yetPurbanchal University: Biratangar, Nepal15 pages
- Computational Methods in Physics and EngineeringNo ratings yetComputational Methods in Physics and Engineering11 pages
- (Embedded) - Realtime Operating Systems - Concepts & Implementation of Microkernels For Embedded Systems (2spp)No ratings yet(Embedded) - Realtime Operating Systems - Concepts & Implementation of Microkernels For Embedded Systems (2spp)106 pages
- Full Download Numerical Methods in Engineering With Python First Edition Jaan Kiusalaas PDFNo ratings yetFull Download Numerical Methods in Engineering With Python First Edition Jaan Kiusalaas PDF51 pages
- Numerical Methods For Physicists: Volker Hohmann Institute of Physics University of Oldenburg, GermanyNo ratings yetNumerical Methods For Physicists: Volker Hohmann Institute of Physics University of Oldenburg, Germany90 pages
- Introduction To Mathematical Proofs - A Transition To Advanced Mathematics (PDFDrive)100% (12)Introduction To Mathematical Proofs - A Transition To Advanced Mathematics (PDFDrive)406 pages
- An Introduction To Linear Algebra For Science and Engineering - 3rd Ed - Norman100% (16)An Introduction To Linear Algebra For Science and Engineering - 3rd Ed - Norman592 pages
- CH00-Introduction To Numerical Methods For PhysicsNo ratings yetCH00-Introduction To Numerical Methods For Physics21 pages
- Strategic Thinking in Complex Problem Solving100% (16)Strategic Thinking in Complex Problem Solving300 pages
- A Fast Algorithm For Particle Simulations Greengard RokhlinNo ratings yetA Fast Algorithm For Particle Simulations Greengard Rokhlin13 pages
- Differential Equations in Technology and Medicine. Computational Concepts, Adaptive Algorithms, and Virtual LabsNo ratings yetDifferential Equations in Technology and Medicine. Computational Concepts, Adaptive Algorithms, and Virtual Labs63 pages
- Bookreviews: I. Foundations & Basic MethodsNo ratings yetBookreviews: I. Foundations & Basic Methods8 pages
- Essential Calculus Skills Practice Workbook With Full Solutions95% (84)Essential Calculus Skills Practice Workbook With Full Solutions528 pages
- Field Call Dynamic DNS Infrastructure v11.2 (Slides)No ratings yetField Call Dynamic DNS Infrastructure v11.2 (Slides)37 pages
- CS 294-73 Software Engineering For Scientific Computing Lecture 12: Particle Methods Homework 3No ratings yetCS 294-73 Software Engineering For Scientific Computing Lecture 12: Particle Methods Homework 332 pages
- Ian Stewart, David Tall - Complex Analysis-Cambridge University Press (2018) PDF100% (11)Ian Stewart, David Tall - Complex Analysis-Cambridge University Press (2018) PDF404 pages
- ECE 595, Section 10 Numerical Simulations Lecture 1: Introduction To Numerical SimulationsNo ratings yetECE 595, Section 10 Numerical Simulations Lecture 1: Introduction To Numerical Simulations24 pages
- Toaz - Info Numerical Solutions To Ce Problems PRNo ratings yetToaz - Info Numerical Solutions To Ce Problems PR2 pages
- Automated Student Attendance System Using Fingerprint RecognitionNo ratings yetAutomated Student Attendance System Using Fingerprint Recognition16 pages
- Assignment: Course Code: MAT-211 Course Title: Engineering MathematicsNo ratings yetAssignment: Course Code: MAT-211 Course Title: Engineering Mathematics8 pages
- Today!: Allan P. Engsig-Karup Scientific Computing Section DTU Informatics Technical University of DenmarkNo ratings yetToday!: Allan P. Engsig-Karup Scientific Computing Section DTU Informatics Technical University of Denmark12 pages
- Computational Methods in Engineering (ME 5201) : Course ContentNo ratings yetComputational Methods in Engineering (ME 5201) : Course Content2 pages
- 50 Challenging Calculus Problems (Fully Solved) - Chris McMullen100% (16)50 Challenging Calculus Problems (Fully Solved) - Chris McMullen236 pages
- Problem Set 1: A Set-Theory Diagnostic: Your Name: Due: Tuesday, February 9No ratings yetProblem Set 1: A Set-Theory Diagnostic: Your Name: Due: Tuesday, February 93 pages
- Problem Set 1: A Set-Theory Diagnostic: A 2A A 2A A 2A A 2ANo ratings yetProblem Set 1: A Set-Theory Diagnostic: A 2A A 2A A 2A A 2A3 pages
- Automatic Irrigation System Using Arduino UNO: Kriti Taneja Sanmeet BhatiaNo ratings yetAutomatic Irrigation System Using Arduino UNO: Kriti Taneja Sanmeet Bhatia4 pages
- Recommended Bibliography For Environmental EnginierNo ratings yetRecommended Bibliography For Environmental Enginier9 pages
- Problem Set 2: Topological Spaces: Your Name: Due: Thursday, February 11No ratings yetProblem Set 2: Topological Spaces: Your Name: Due: Thursday, February 112 pages
- Problem Set 3: Limits and Closures: Your Name: Due: Thursday, February 18No ratings yetProblem Set 3: Limits and Closures: Your Name: Due: Thursday, February 182 pages
- Introduction To Linear Algebra For Science and Engineering 1st Ed90% (58)Introduction To Linear Algebra For Science and Engineering 1st Ed550 pages
- Two Questions of 10 Marks Will Be Set From Each Unit, One Needs To Be Answered From Each UnitNo ratings yetTwo Questions of 10 Marks Will Be Set From Each Unit, One Needs To Be Answered From Each Unit7 pages
- ME 704 Computational Methods in Thermal and Fluids Engineering (Introduction)No ratings yetME 704 Computational Methods in Thermal and Fluids Engineering (Introduction)3 pages
- All The Math You Missed - But Need To Know For Graduate School100% (35)All The Math You Missed - But Need To Know For Graduate School417 pages
- Problem Set 2: Topological Spaces: Your Name: Due: Thursday, February 11No ratings yetProblem Set 2: Topological Spaces: Your Name: Due: Thursday, February 115 pages
- (Industrial and Applied Mathematics) Martin Brokate, Pammy Manchanda, Abul Hasan Siddiqi - Calculus For Scientists and Engineers (2019, Springer)100% (14)(Industrial and Applied Mathematics) Martin Brokate, Pammy Manchanda, Abul Hasan Siddiqi - Calculus For Scientists and Engineers (2019, Springer)655 pages
- Solutions To Problem Set 3: Limits and ClosuresNo ratings yetSolutions To Problem Set 3: Limits and Closures4 pages
- Laplace Transforms and Their Applications to Differential EquationsFrom EverandLaplace Transforms and Their Applications to Differential Equations4.5/5 (2)
- Elementary Theory and Application of Numerical Analysis: Revised EditionFrom EverandElementary Theory and Application of Numerical Analysis: Revised EditionNo ratings yet
