Approach Based on Taylor Series

Interpolation-Based Approach
Complex Step Derivative

MATH 3Q03: Differentiation with Finite

Differences

Bartosz Protas

Department of Mathematics & Statistics

Email: [email protected]
Office HH 326, Ext. 24116
Course Webpage: http://www.math.mcmaster.ca/˜bprotas/MATH3Q03

March 29, 2017

B. Protas MATH3Q03, Winter 2017

 Approach Based on Taylor Series
Interpolation-Based Approach
Complex Step Derivative

Agenda

Approach Based on Taylor Series

Interpolation-Based Approach

Complex Step Derivative

B. Protas MATH3Q03, Winter 2017

 Approach Based on Taylor Series
Interpolation-Based Approach
Complex Step Derivative

I Assumptions :
I f : Ω → R is a smooth function, i.e. is continuously
differentiable sufficiently many times,
I the domain Ω = [a, b] is discretized with a uniform grid
{x1 = a, . . . , xN = b}, such that xj+1 − xj = hj = h (extensions
to nonuniform grids are straightforward)

I Problem — given the nodal values of the function f, i.e.,

fj = f (xj ), j = 1, . . . , N approximate the nodal values of the
function derivative
df
(xj ) = f 0 (xj ) =: fj0 , j = 1, . . . , N
dx

δf


I The symbol δx j
will denote the approximation of the
0
derivative f (x) at x = xj
B. Protas MATH3Q03, Winter 2017
 Approach Based on Taylor Series
Interpolation-Based Approach
Complex Step Derivative

I The simplest approach — Derivation of finite difference

formulae via Taylor–series expansions
(xj+1 − xj )2 00 (xj+1 − xj )3 000
fj+1 = fj + (xj+1 − xj )fj0 + fj + fj + . . .
2! 3!
h2 00 h3 000
= fj + hfj0 + f + fj + . . .
2 j 6

I Rearrange the expansion

fj+1 − fj h fj+1 − fj
fj 0 = − fj00 + · · · = + O(h),
h 2 h
where O(hα ) denotes the contribution from all terms with powers of
h greater or equal α (here α = 1).
I Neglecting O(h), we obtain a first order
forward–difference formula :
 
δf fj+1 − fj
=
δx j h

B. Protas MATH3Q03, Winter 2017

 Approach Based on Taylor Series
Interpolation-Based Approach
Complex Step Derivative

I Backward difference formula is obtained by expanding fj−1 about xj

and proceeding as before:
 
fj − fj−1 h δf fj − fj−1
fj 0 = − fj 00 + . . . =⇒ =
h 2 δx j h

I Neglected term with the lowest power of

 h is the leading–order


approximation error , i.e., Err = f 0 (xj ) − δx
δf
 ≈ Chα

j

I The exponent α of h in the leading–order error represents the

order of accuracy of the method — it tells how quickly
the approximation error vanishes when the resolution is refined

I The actual value of the approximation error depends on the

constant C characterizing the function f

I In the examples above Err = − h2 fj00 , hence the methods are

first–order accurate
B. Protas MATH3Q03, Winter 2017
 Approach Based on Taylor Series
Interpolation-Based Approach
Complex Step Derivative

Higher–Order Formulas (I)

I Consider two expansions:
h2 00 h3 000
fj+1 = fj + hfj0 + f + f + ...
2 j 6 j
h2 h3 000
fj−1 = fj − hfj0 + fj00 − f + ...
2 6 j

I Subtracting the second from the first:

h3 000
fj+1 − fj−1 = 2hfj0 + f + ...
3 j

I Central Difference Formula

fj+1 − fj−1 h2 000
 
δf fj+1 − fj−1
fj 0 = − fj + . . . =⇒ =
2h 6 δx j 2h

B. Protas MATH3Q03, Winter 2017

 Approach Based on Taylor Series
Interpolation-Based Approach
Complex Step Derivative

Higher–Order Formulas (II)

2
I The leading–order error is h6 fj000 , thus the method is
second–order accurate

I Manipulating four different Taylor series expansions one can

obtain a fourth–order central difference formula :
h4 (v )
 
δf −fj+2 + 8fj+1 − 8fj−1 + fj−2
= , Err = f
δx j 12h 30

B. Protas MATH3Q03, Winter 2017

 Approach Based on Taylor Series
Interpolation-Based Approach
Complex Step Derivative

Approximation of the Second Derivative

I Consider two expansions:
h2 00 h3 000
fj+1 = fj + hfj 0 + fj + fj + . . .
2 6
2 3
h h 000
fj−1 = fj − hfj 0 + fj 00 − fj + . . .
2 6
I Adding the two expansions
h4 iv
fj+1 + fj−1 = 2fj + h2 fj 00 + fj + . . .
12

I Central difference formula for the second derivative:

h2 (iv ) δ2 f
 
fj+1 − 2fj + fj−1 fj+1 − 2fj + fj−1
fj 00 = − f + . . . =⇒ =
h2 12 j δx 2 j h2

h 2 (iv )
I The leading–order error is 12 fj , thus the method is
second–order accurate
B. Protas MATH3Q03, Winter 2017
 Approach Based on Taylor Series
Interpolation-Based Approach
Complex Step Derivative

Taylor Table
I A general method for choosing the coefficients of a finite
difference formula to ensure the highest possible order of
accuracy

I Example:
P2 consider a one–sided finite difference formula
p=0 ap fj+p , where the coefficients ap , p = 0, 1, 2 are to be
determined.

I Form an expression for the approximation error

2
X
fj0 − ap fj+p = 
p=0

and expand it about xj in the powers of h

B. Protas MATH3Q03, Winter 2017
 Approach Based on Taylor Series
Interpolation-Based Approach
Complex Step Derivative

I Expansions can be collected in a Taylor table

fj fj 0 fj 00 fj 000
0
fj 0 1 0 0
−a0 fj −a0 0 0 0
2 3
−a1 fj+1 −a1 −a1 h −a1 h2 −a1 h6
(2h)2 (2h)3
−a2 fj+2 −a2 −a2 (2h) −a2 2 −a2 6
I the leftmost column contains the terms present in the
expression for the approximation error
I the corresponding rows (multiplied by the top row) represent
the terms obtained from expansions about xj
I columns represent terms with the same order in h — sums of
columns are the contributions to the approximation error with
the given order in h

I The coefficients ap , p = 0, 1, 2 can now be chosen to cancel

the contributions to the approximation error with the lowest
powers of h
B. Protas MATH3Q03, Winter 2017
 Approach Based on Taylor Series
Interpolation-Based Approach
Complex Step Derivative

I Setting the coefficients of the first three terms to zero:



 −a0 − a1 − a2 = 0

−a1 h − a2 (2h) = −1 3 2 1

=⇒ a0 = − , a1 = , a2 = −
2 2 2h h 2h
 −a1 h − a2 (2h) = 0



2 2

I The resulting formula:

 
δf −fj+2 + 4fj+1 − 3fj
=
δx j 2h

I The approximation error — determined the evaluating the

first column with non–zero coefficient:
h3 (2h)3 h2 000
 
−a1 − a2 fj 000 = fj
6 6 3

The formula is thus second–order accurate

B. Protas MATH3Q03, Winter 2017
 Approach Based on Taylor Series
Interpolation-Based Approach
Complex Step Derivative

I An alternative derivation of a finite–difference scheme:

I Find an N–th order accurate interpolating function p(x) which
interpolates the function f (x) at the nodes xj , j = 1, . . . , N, i.e., such
that p(xj ) = f (xj ), j = 1, . . . , N
I Differentiate the interpolating function p(x) and evaluate at the nodes
to obtain an approximation of the derivative p 0 (xj ) ≈ f 0 (xj ),
j = 1, . . . , N

I Example:
I for j = 2, . . . , N − 1, let the interpolant have the form of a quadratic
polynomial pj (x) on [xj−1 , xj+1 ] (Lagrange interpolating polynomial)
(x − xj )(x − xj+1 ) −(x − xj−1 )(x − xj+1 ) (x − xj−1 )(x − xj )
pj (x) = fj−1 + fj + fj+1
2h2 h2 2h2
(2x − xj − xj+1 ) −(2x − xj−1 − xj+1 ) (2x − xj−1 − xj )
pj0 (x) = fj−1 + fj + fj+1
2h2 h2 2h2
f −f
I Evaluating at x = xj we obtain f 0 (xj ) ≈ pj0 (xj ) = j+12h j−1
(i.e., second–order accurate center–difference formula)
B. Protas MATH3Q03, Winter 2017
 Approach Based on Taylor Series
Interpolation-Based Approach
Complex Step Derivative

I Generalization to higher–orders straightforward

I Example:
I for j = 3, . . . , N − 2, one can use a fourth–order polynomial as
interpolant pj (x) on [xj−2 , xj+2 ]
I Differentiating with respect to x and evaluating at x = xj we
arrive at the fourth–order accurate finite–difference formula
h4 (v )
 
δf −fj+2 + 8fj+1 − 8fj−1 + fj−2
= , Err = f
δx j 12h 30

I Order of accuracy of the finite–difference formula is one less

than the order of the interpolating polynomial

I The set of grid points needed to evaluate a finite–difference

formula is called stencil

I In general, higher–order formulas have larger stencils

B. Protas MATH3Q03, Winter 2017
 Approach Based on Taylor Series
Interpolation-Based Approach
Complex Step Derivative

Subtractive Cancellation Errors

I Subtractive cancellation errors — when comparing
two numbers which are almost the same using finite–precision
arithmetic , the relative round–off error is proportional to the
inverse of the difference between the two numbers
I Thus, if the difference between the two numbers is decreased
by an order of magnitude, the relative accuracy with which
this difference may be calculated using finite–precision
arithmetic is also decreased by an order of magnitude.
I Problems with finite difference formulae when h → 0 — loss
of precision due to finite–precision arithmetic ( subtractive
cancellation ), e.g., for double precision:
1.0000000000012345 − 1.0≈1.2e − 12 (2.8% error)
1.0000000000001234 − 1.0≈1.0e − 13 (19.0% error)
...
B. Protas MATH3Q03, Winter 2017
 Approach Based on Taylor Series
Interpolation-Based Approach
Complex Step Derivative

I Consider the complex extension f (z), where z = x + iy , of

f (x) and compute the complex Taylor series expansion
h2 00 h3
f (xj + ih) = fj + ihfj 0 − fj − i fj 000 + O(h4 )
2 6
df (z)
Need to assume that f (z) is analytic ! Then f 0 = dz

I Take imaginary part and divide by h

h2
 
0 =(f (xj + ih)) δf =(f (xj + ih))
fj = + fj 000 + O(h3 ) =⇒ =
h 6 δx j h

I Note that the scheme is second order accurate — where is

conservation of complexity?
I The method doesn’t suffer from cancellation errors, is easy to
implement and quite useful
I Reference:
I J. N. Lyness and C. B.Moler, “Numerical differentiation of
analytical functions”, SIAM J. Numer Anal 4, 202-210, (1967)
B. Protas MATH3Q03, Winter 2017