Various lecture notes for 18311.

R. R. Rosales
April 28, 2011 version 01.
Abstract

Notes, both complete and/or incomplete, for MIT’s 18.311 (Principles of Applied Mathematics).
These notes will be updated from time to time. Check the date and version.

Contents
1 Convergence of numerical Schemes. 2
1.1 The initial value problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 The numerical scheme. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Consistency and stability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Lax convergence theorem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.5 Example: von Neumann stability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Problem: complete the details for the von Neumann stability example. . . . . . 7

2 DFT, FFT, and Fourier series. 7

2.1 Introduction and motivation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
The Discrete Fourier Transform (DFT). . . . . . . . . . . . . . . . . . . . . . . . . . 8
Problem: verify the DFT and iDFT formulas. . . . . . . . . . . . . . . . . . . 9
2.2 Alternative formulations for the DFT. . . . . . . . . . . . . . . . . . . . . . . . . . 9
®
MATLAB implementation of the DFT and iDFT formulas. . . . . . . . . . . . . . 10
2.3 Relationship between Fourier series and the DFT/iDFT. . . . . . . . . . . . . . . . 10
2.3.1 Fourier series, DFT, and derivatives. . . . . . . . . . . . . . . . . . . . . . . 13
This subsection is yet to be written. . . . . . . . . . . . . . . . . . . . 13
2.4 Simple convergence results for Fourier series. . . . . . . . . . . . . . . . . . . . . . 13
Problem: compute the asymptotic value of an integral. . . . . . . . . . . . . . 16
2.4.1 Convergence in the weak sense. . . . . . . . . . . . . . . . . . . . . . . . . . 17
This subsubsection is yet to be written. . . . . . . . . . . . . . . . . . 17
2.5 The main idea behind the FFT algorithm. . . . . . . . . . . . . . . . . . . . . . . . 17
This subsection is yet to be written. . . . . . . . . . . . . . . . . . . . 17
2.6 Trapezoidal rule and the Euler-Maclaurin formula. . . . . . . . . . . . . . . . . . . 17
Trapezoidal rule for periodic functions. . . . . . . . . . . . . . . . . . . . . . . 19
2.6.1 Bernoulli polynomials. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Generating function for the Bernoulli polynomials. . . . . . . . . . . . . . . . . 20
Problem: compute the Bernoulli polynomials’ generating function. . . . . . . . 20

1
Various lecture notes for 18311. Rosales, MIT 2

1 Convergence of numerical schemes.

In this section we introduce some theory dealing with the question of the convergence of numer­
ical schemes for partial differential equations. For simplicity we will consider only the case of an
homogeneous, one step in time, linear scheme for an initial value problem (IVP) for an
homogeneous linear system of first order in time partial differential equations (pde) in 1-D.

1.1 The initial value problem.

The IVP to be solved has the form

u t = L u for xL < x < xR and t > 0, (1.1.1)

with u (x, 0) = u 0 (x), where u (x, t) is vector1 valued, L is a linear differential operator, and appro­
priate homogeneous2 boundary conditions (BC) apply.

Remark 1.1.1 We will assume that (1.1.1) is a well posed problem, with a solution that is as
smooth as needed (this is specified later).

Example 1.1.1 Linear scalar equation. ut + a ux = b u, where u = u(x, t) is scalar valued, (a, b)
are given functions of (x, t), and periodic BC apply: u(xL , t) = u(xR , t).

Example 1.1.2 Heat equation. ut = (ν ux )x , where u = u(x, t) is scalar valued, ν > 0 is a given
function of (x, t), and either of the following BC apply (this list of BC is not exhaustive)
(i) u(xL , t) = u(xR , t) and ux (xL , t) = ux (xR , t) (periodic).
(ii) u(xL , t) = u(xR , t) = 0.
(iii) ux (xL , t) = ux (xR , t) = 0.
(iv) u(xL , t) = ux (xR , t) = 0.

Example 1.1.3 Wave equation: (u1 )t = u2 and (u2 )t = c2 (u1 )x x , where u = (u1 , u2 ), c is a given
function of (x, t), and the same BC as in example 1.1.2 apply.

Example 1.1.4 Klein-Gordon equation. (u1 )t = u2 and (u2 )t = c2 (u1 )x x − m2 u1 , where u =

(u1 , u2 ), (c, m) are given functions of (x, t), and the same BC as in example 1.1.2 apply.

Example 1.1.5 Korteweg-de Vries equation. ut + a ux + b uxxx = 0, where u = u(x, t) is scalar

valued, (a, b) are given functions of (x, t), and periodic BC apply: u(xL , t) = u(xR , t), ux (xL , t) =
ux (xR , t), and uxx (xL , t) = uxx (xR , t).
1
i ∈ Rd , is a d-column vector, for some d = 1, 2, 3, . . ..
u
2
Homogeneous BC means BC that yield: If Ouj solves (1.1.1) for the initial values Ou0 = U O j (j = 1 or j = 2), then
O 1 + α2 U
Ou = α1 Ou1 + α2 Ou2 solves (1.1.1) for the initial value Ou0 = α1 U O 2 , where α1 and α2 are arbitrary constants.
Various lecture notes for 18311. Rosales, MIT 3

1.2 The numerical scheme.

Assume an appropriate3 grid in space-time, with constant grid separations Δx > 0 and Δt > 0.
For example
1
Δx = (xR − xL ) , xn = xL + n Δx for 1 ≤ n ≤ N , and tj = j Δt for j ≥ 0, (1.2.1)
N +1
where N > 1 is any sufficiently “large” integer.4 We assume that the numerical scheme to solve the
(1.1.1) IVP has the form
uj+1 = Sj uj for j ≥ 0, (1.2.2)
where uj ∈ R d ×N is a d × N matrix for each j > 0, and {Sj }j≥0 is a sequence of linear operators
in R d ×N — which could be represented as d N × d N matrices if needed. The operators Sj generally
depend on: Δt and Δx, possibly some numerical parameters (e.g.: artificial viscosity), and the details
of the pde to be solved. Furthermore, each of the columns of uj is a d-vector, which we denote by
u nj — for 1 ≤ n ≤ N . These vectors are interpreted as the approximations to the values of the solution
at the grid points. Namely u nj ≈ u (xn , tj ). (1.2.3)
0
In particular u n = u 0 (xn ) (1.2.4)
should be used to initialize the numerical scheme.
Note that:
1.2a The interpretation in (1.2.3–1.2.3) is not unique. For example, in many schemes u nj is taken
as the average value of the solution over the nth cell: |x − xn | ≤ 12 Δx.
1.2b Schemes with meshes where tj+1 − tj = Δt is not a constant are frequently used. Similarly,
xn+1 − xn = Δx need not be a constant.
1.2c In (1.2.1) the end points xL and xR are not included in the numerical grid. This would be
appropriate if the solution is prescribed there — e.g.: u (xL , t) = u (xR , t) = 0. For periodic
BC, an appropriate choice is to include one of the end points but not the other, as in: xn =
xL + (n − 1) Δx for 1 ≤ n ≤ N , with Δx = (xR − xL )/N . Many other choices are possible.

Example 1.2.1 Consider the heat equation, as in example 1.1.2 – with the BC in (ii). Then, with
the choice of grid in (1.2.1), a scheme of the form in (1.2.2) is given by
Δt j
uj+1 = ujn + ujn+1 − ujn − νn−
j
unj − ujn−1 ,



n νn+ 1 1 (1.2.5)
(Δx)2 2 2

j 1 j j


where νn± 1 = ν xn ± 2 Δx, tj , and u0 = uN +1 = 0 is used when evaluating the right hand side of
2
equation (1.2.5) for n = 1 and n = N . This scheme makes sense for any N ≥ 1.
3
See item 1.2c below.
4
The numerical scheme should be defined for any N large enough — where large enough is usually not very large
— see example 1.2.1. We are, however, interested in the limit N → ∞ here.
Various lecture notes for 18311. Rosales, MIT 4

1.3 Consistency and stability.

Let · Nbe a norm in R d ×N , where the uj belong — see equation (1.2.2). We assume the following:
⎫
Let u = u (x) be a suficiently nice d-vector valued function. Define u ∈ R d ×N by ⎪
⎬
th
u n = u (xn ), where u n is the n column of u, for 1 ≤ n ≤ N . Then u N → u ∗ (1.3.1)
⎪
as N → ∞, where · ∗ is a norm defined for d-vector valued functions.
⎭

N xR
Example 1.3.1 For d = 1, let u N = 1 |un |2 Δx. Then u ∗ = xL
|u(x)|2 dx, and in
(1.3.1) “sufficiently nice” means continuous.

N N −1
Example 1.3.2 For d = 1, let u N = 1 |un |2 Δx + 2 |un+1 − un |2 Δx. Then u ∗ =
xR xR
xL
|u(x)|2 dx + xL
|u' (x)|2 dx, and in (1.3.1) “sufficiently nice” means C 1 — i.e.: u has a
5
continuous derivative.

Remark 1.3.1 Recall that a norm is a real valued function defined on a vector space V such that,
for any v ∈ V and w ∈ V, and scalars a and b, the following applies: (i) v ≥ 0, (ii) v = 0 if
and only if v = 0, and (iii) v + w ≤ v + w .

Definition 1.3.1 The numerical scheme in § 1.2 is consistent if and only if the following applies:
Let u = U u (x, t) be the solution to the IVP (1.3.1) for some arbitrary initial condition u 0 . Assume
u is is sufficiently smooth and define Uj ∈ R d ×N by U
that U uj=U u (xn , tj ), where 1 ≤ n ≤ N , j ≥ 0,
n
and Uu j is the nth column of Uj . Then
n

Uj+1 − Sj Uj N ≤ fc (tj+1 ) Δt ((Δt)p + (Δx)q ) , (1.3.2)

where p > 0 is the order of the method in time, q > 0 is the order of the method in space, and
0 < fc (t) < ∞ is some grid independent bounded function — determined by the solution U u and its
partial derivatives up to some order,a as well as the coefficients b of the equation in the IVP (1.1.1).
(a) This is why Uu needs to be sufficiently smooth.
(b) These coefficients must also be sufficiently smooth.

Example 1.3.3 Consider the numerical scheme in example 1.2.1. In this case (1.3.2) applies with
p = 1, q = 2, and
fc (t) = max ( 1 N ) max(M1 , M2 ), (1.3.3)
N

where
(i) 1 N indicates the norm of the vector all whose entries are one — since 1 N → 1 ∗ as
N → ∞, { 1 N }N is a bounded sequence with a maximum.
5
Actually, less is needed — e.g.: an integrable bounded derivative will do (dominated convergence theorem).
Various lecture notes for 18311. Rosales, MIT 5

1
(ii) M1 = M1 (t) is the maximum of 2
|Utt (x, s)|, for xL ≤ x ≤ xR , and 0 ≤ s ≤ t.
1
(iii) M2 = M2 (t) is the maximum of 24
|Ghhhh (x, h, s)|, for xL + h ≤ x ≤ xR − h, and 0 ≤ s ≤ t
— where G is defined in (1.3.5).

Proof. Using in (1.2.5) Taylor expansions with remainder it follows that

Δt  j j j j


Unj+1 − Unj − j j


ν 1 U n+1 − U n − ν 1 Un − Un−1
(Δx)2 n+ 2 n− 2

1 1
= Utt (xn , τnj ) (Δt)2 − Ghhhh (xn , hjn , tj ) (Δx)2 Δt, (1.3.4)
2 24
for some tj ≤ τnj ≤ tj+1 and 0 ≤ hjn ≤ Δx, where
1 1
G(x, h, t) = ν(x + h) (U (x + h, t) − U (x, t)) − ν(x − h) (U (x, t) − U (x − h, t)) . (1.3.5)
2 2
Hence (1.3.3) follows. ♣

Remark 1.3.2 Methods exist for which (1.3.2) does not strictly apply. For example, one may have
 
j+1 j p+1 ql
kU − Sj U kN ≤ fc (tj+1 ) (Δt) + (Δx) . (1.3.6)

However, in a numerical method one is interested in the situations where both Δt and Δx are small,6
and generally Δt and Δx are related to each other — e.g. Δt = constant Δx, in which case (1.3.2)
and (1.3.6) are equivalent.

Definition 1.3.2 The numerical scheme in § 1.2 is stable if and only if

kuj kN ≤ fs (tj ) kuk kN , for any 0 ≤ k ≤ j, (1.3.7)

where 0 < fs (t) < ∞ is some grid (and solution) independent a bounded function. Note that, for
equation (1.3.7) to apply, restrictions might be needed on Δt and Δx — such as: Δt ≤ constant Δx
or Δt ≤ constant (Δx)2 . These restrictions must allow Δt and Δx to vanish simultaneously.
(a) Of course, fs will depend on the coefficients of the equation in the IVP (1.1.1).

1.4 Lax convergence theorem.

Theorem 1.4.1 If the scheme in § 1.2 is consistent and stable, then it converges — in any fixed
time interval 0 ≤ t ≤ T — as Δt → 0 and Δx → 0 (provided that any restrictions on Δt and Δx
required by (1.3.7) apply). By converges we mean that

kuj − Uj kN → 0, for 0 ≤ tj ≤ T , as Δt + Δx → 0, (1.4.1)

where Uj is as in definition 1.3.1, uj is the numerical solution (1.2.2) — initialized as in (1.2.4),

and the convergence is uniform in 0 ≤ tj ≤ T .
6
Formally, Δt → 0 and Δx → 0.
Various lecture notes for 18311. Rosales, MIT 6

Proof: we have uj+1 − Uj+1 = (uj+1 − Sj uj ) + (Sj uj − Sj Uj ) + (Sj Uj − Uj+1 ). Hence, since
uj+1 − Sj uj = 0, we can write uj+1 − Uj+1 = Sj (uj − Uj ) + (Sj Uj − Uj+1 ). Recursive application
of this then yields

uj − Uj = Sj−1 Sj−2 . . . S1 S0 (u0 − U0 ) +(Sj−1 Uj−1 − Uj ) + Sj−1 (Sj−2 Uj−2 − Uj−1 ) +

" "
A
Sj−1 Sj−2 (Sj−3 Uj−3 − Uj−2 ) + . . . + Sj−1 Sj−2 . . . S1 (S0 U0 − U1 ), (1.4.2)

where A = 0 — since u0 = U0 . Let 0 < K < ∞ be a bound on fc and fs for 0 ≤ t ≤ T . Then, from
stability and consistency

Sj−1 Sj−2 . . . Sc+1 (Sc Uc − Uc+1 )kN ≤ K Sc Uc − Uc+1 kN ≤ K 2 ((Δt)p + (Δx)q ) Δt, (1.4.3)

for any 0 ≤ f < j. Using this in (1.4.2) then yields

kuj − Uj kN ≤ K 2 ((Δt)p + (Δx)q ) tj ≤ K 2 ((Δt)p + (Δx)q ) T , (1.4.4)

from which (1.4.1) follows. ♣

1.5 Example: von Neumann stability.

Consider now the situation when the equation in (1.1.1) has constant coefficients, and periodic BC
apply. This is the case where stability can be ascertained using a von Neumann analysis, as we show
next. For simplicity we will assume a scalar equation (i.e. d = 1), and a normalized period = 2 π,
with xL = 0 and xR = 2 π. We then use the space grid xn = n Δx, 1 ≤ n ≤ N , with Δx = 2 π/N .
The initial data for the scheme u0 = {u0n }n=1
N
can be written in the form (see § 2)
N
N N
N N
N N
N
u0n = a c e i c xn = ac ei ke n = ac ei 2 π c n/N = ac w c n for 1 ≤ n ≤ N , (1.5.1)
c=1 c=1 c=1 c=1

where w = ei 2 π/N is the N th fundamental root of unity, kc = f Δx = 2 π f/N , and the {ac }N
c are
j j N
some (complex) constants. Then a von Neumann analysis shows that the solution u = {un }n=1 to
the numerical scheme has the form
N
N
ujn = ac (Gc )j ei c xn for 1 ≤ n ≤ N , and j ≥ 0, (1.5.2)
c=1

where the {Gc }N

c are constants that depend on Δt, Δx, the coefficients of the equation, and any
numerical parameters.
Remark 1.5.1 Finite differences approximations to constant coefficients IVP with periodic bound­
ary conditions generally yield numerical schemes for which a von Neumann analysis works. Namely:
in (1.2.2) Sj = S is independent of j, and S applied to a uj whose components are proportional to
exponentials of the form ei k n , for some k, yields an answer of the same type.
Various lecture notes for 18311. Rosales, MIT 7

We point out that it is possible to produce numerical schemes (not necessarily using finite differences)
for constant coefficients IVP with periodic boundary conditions, for which a von Neumann analysis
does not work. Here we assume that this is not the case.

Apply now the norm introduced in example 1.3.1 to the expression in equation (1.5.2). Using the
fact that N n (.1 −.2 )
P
n=1 w = N δ.1 .2 for 1 ≤ f1 , f2 ≤ N , we obtain

N
N
j
ku k N = 2π |ac |2 (|Gc |2 )j for j ≥ 0. (1.5.3)
c=1

Define now G = max |G. |. Then

1≤.≤N

kuj kN ≤ Gj−k kuk kN for 0 ≤ k ≤ j. (1.5.4)

Comparing this with (1.3.7), we see that stability can be ascertained by studying the behavior of Gj
as j → ∞ with tj = j Δt bounded. In particular:

If G ≤ 1, the scheme is stable. (1.5.5)

Problem 1.5.1 Complete the details for the von Neumann stability example.
PN
Show that 7 n=1 wn (c1 −c2 ) = N δc1 c2 for 1 ≤ f1 , f2 ≤ N , where N > 1 is an integer, and w = ei 2 π/N
is the N th fundamental root of unit. Then use this to derive (1.5.3).
Hint: wM = 1 if and only if M is a multiple of N .

2 DFT, FFT, and Fourier series.

This section deals with the Discrete Fourier Transform (DFT), it fast implementation using the Fast
Fourier Transform (FFT), and the relationship of the DFT with Fourier series for periodic functions.

2.1 Introduction and motivation.

In von Neumann stability analysis — see § 1.5, we conclude that a numerical scheme for a situation
with periodic boundary conditions8 is stable if and only if the solutions to the scheme of the form

ujN = Gj ei k n (2.1.1)
7
6 j, and δj j = 1.
The notation δ£ j is used for the Kronecker delta: δ£ j = 0 if g =
8
Example: finite differences for a 1-D linear, constant coefficients, equation for wave propagation.
Various lecture notes for 18311. Rosales, MIT 8

remain bounded as M → ∞, for 0 ≤ tj = j Δt ≤ T , Δt = T /M , and T fixed — perhaps with a

constraint9 relating Δx to Δt. In particular, this happens if |G| ≤ 1 for all solutions of this form.
However, for the result in § 1.5 to be true, it must be that all the solutions are of the form in (2.1.1),
or linear combinations of them. This motivates the question: Given any sequence {un }∞ −∞ , with
N
un+N = un for some integer N > 0, can we write un = a c ei k e n (2.1.2)
for some finite set of coefficients ac and wave numbers kc ? c

The answer to this question is yes, and is given in detail by theorem 2.1.1 below.
Before going into details, notice that periodicity un+N = un constraints the possible wave num­
bers that can occur in (2.1.2), since it requires that kc N be a multiple of 2π. Thus
⎫
the wave numbers are restricted to the set k. = 2Nπ £, where £ is an integer. ⎪ ⎪
i ke n i ke+N n
⎪
Furthermore, e =e for any integer n, hence
⎬
(2.1.3)
the wave numbers can be selected in any range £∗ ≤ £ ≤ £∗ + N − 1, where ⎪ ⎪
⎪
⎭
£∗ is some arbitrary integer.
Theorem 2.1.1 Given any periodic sequence {un }n=+∞
n=−∞ of complex numbers, with period N —
un+N = un , one can write
∗ +N −1
c=cN
2π
un = ac ei ke n , where f∗ is any integer, kc = f, (2.1.4)
c=c∗
N

and {ac }c=+∞

c=−∞ is the periodic, with period N , sequence of complex numbers given by
∗ +N −1
n=nN
1
ac = un e−i kn c , where n∗ is any integer. (2.1.5)
N n=n∗

The transformation between periodic sequences {un } → {ac } in (2.1.5), giving the coefficients ac in
(2.1.4), is the Discrete Fourier Transform (DFT). It’s inverse in (2.1.4) is the Inverse Discrete
Fourier Transform (iDFT). The names follow from the connection with Fourier series — see § 2.3.

Remark 2.1.1 Consider the case when f∗ = n∗ = 1. Then (2.1.4) and (2.1.5) become
c=N n=N
N 1 N
un = ac ei ke n
and ac = un e−i kn c . (2.1.6)
c=1
N n=1
Because of the periodicity, we need only consider {un } and {ac } for 1 ≤ n, f ≤ N . Hence, in terms of
(a) the N -vectors ui and i a whose components are {un } and {ac }, respectively,
(b) the N × N matrix D whose entries are D. n = ei k£ n = w. n , where w = ei 2 π/N ,
we can write
1 † 1 †
u = D ua and ua = D u , ⇐⇒ D−1 = D, (2.1.7)
N N
where † denotes the adjoint of a matrix.
9
Examples: Δt ≤ constantΔx or Δt ≤ constant(Δx)2 . The constraint must allow Δx to vanish as Δt → 0, so
that convergence can occur.
Various lecture notes for 18311. Rosales, MIT 9

Problem 2.1.1 Verify the DFT and iDFT formulas.

Show that
(i) If {ac }c=+∞
c=−∞ is a periodic sequence of period N , the value of the right hand side in (2.1.4)
is independent of the choice of f∗ . Similarly, if {un }n=+∞
n=−∞ is a periodic sequence of period
N , the value of the right hand side in (2.1.5) is independent of the choice of n∗ .
(ii) For any set of constants10 ac , f∗ ≤ f < f∗ + N , {un } — as given by (2.1.4), is periodic of
period N . Similarly, for any set of constants un , n∗ ≤ n < n∗ + N , {ac } — as given by
(2.1.5), is periodic of period N .
(iii) Substituting (2.1.5) into the right hand side of (2.1.4) yields the left hand side.
(iv) Substituting (2.1.4) into the right hand side of (2.1.5) yields the left hand side.

Hints:
1) For part (i), denote by Sa = Sa (f∗ ) the value of the right hand side of (2.1.4) as a function of f∗
— for some given, fixed, periodic sequence {ac }+∞ −∞ . Then show that S(f∗ + 1) = S(f∗ ) for any
f∗ , from which S = constant follows. The same idea works for (2.1.5).
PN n (.1 −.2 )
2) You will need the following result, obtained in problem 1.5.1: n=1 w = N δ.1 .2 for
i 2 π/N th
1 ≤ f1 , f2 ≤ N , where N > 1 is an integer, w = e is the N fundamental root of unit,
6 j, and δj j = 1.
and δi j denotes the Kronecker delta: δi j = 0 if i =
3) Note that ei k£ n = w. n , and e−i kn . = w−. n .

2.2 Alternative formulations for the DFT.

If we define vn = un−1 and b. = a.−1 , then the equations in (2.1.4–2.1.5) take the form11
+N −1
c∗ N +N −1
n∗N
1
vn = bc w (c−1) (n−1)
and bc = vn w−(n−1) (c−1) . (2.2.1)
c=c∗
N n=n∗
" {z " " {z "
iDFT DFT

where f∗ , n∗ are arbitrary integers, and w = ei 2 π/N . These two formulas constitute an alternative
formulation of the DFT:{vn } → {b. }, and the iDFT:{b. } → {vn }, relating N -periodic sequences.
In particular, if we select f∗ = n∗ = 1, we obtain
N N
N 1 N
vn = bc w (c−1) (n−1)
and bc = vn w−(n−1) (c−1) . (2.2.2)
c=1
N n=1
10
For this part of the problem there is no sequence {a£ }, just N constants.
11
To get the equations here, replace g∗ → g∗ − 1 in (2.1.4) and n∗ → n∗ − 1 in (2.1.4).
Various lecture notes for 18311. Rosales, MIT 10

If v is the N -vector array whose entries are the {vn }, and b is the N -vector array whose entries are
the {bc }, then the transformations in (2.2.2) are executed by the MATLAB commands
1
b= fft(v) and v = N ifft(b). (2.2.3)
N
Remark 2.2.1 A numerical implementation of the DFT and iDFT, as written in (2.2.2), has an
O(N 2 ) operation, and it is thus rather costly. Fortunately, there is a way (algorithm) to organize
the calculations that leads to computation whose operation count is O(N ln(N )). This algorithm is
known by the name of the Fast Fourier Transform (FFT), with inverse given by the inverse Fast
Fourier Transform (iFFT). This algorithm, of course, is the one MATLAB implements.12
The FFT is important because it allows the fast/efficient implementation of the DFT and iDFT,
which allow the approximate, efficient, and accurate, calculation of Fourier series and Fourier
coefficients.13 Since Fourier series appear in very many applications, the FFT and iFFT are widely
used. A brief description of the main idea behind the FFT algorithm is included in § 2.5.

2.3 Relationship between Fourier series and the DFT/iDFT.

Let f = f (x) be a sufficiently nice14 periodic function — assume (for simplicity) that the period is
2 π, so that f (x + 2 π) = f (x). Then f has a Fourier series
∞
N
f (x) = Cn (f ) ei n x , (2.3.1)
−∞
x0 +2 π
1
where Cn = f (x) e−i n x dx for n = 0, ±1, ±2, ±3, . . . (2.3.2)
2 π x0
Here x0 is any real number — the Fourier coefficients Cn do not depend on the choice of x0 .
There are many notions (types of convergence) in which the right hand side in (2.3.1) can represent
f . We will explore this question (lightly) in § 2.4. The important point here is that, the smoother the
function f is, the better and faster (less terms needed to get a good approximation) the convergence
of the series in (2.3.1), in particular: for smooth functions the convergence is very fast:
N
N
f (x) = Cn (f ) ei n x + EN , (2.3.3)
−N

where EN → 0 vanishes faster than any power N −p as N → ∞.

12
Hence the command names fft and ifft in (2.2.3).
13
We study the relationship between the DFT/iDFT and Fourier series in § 2.3.
14
Example: f has two derivatives, with f " " integrable — see § 2.4. Note that far less is needed for a Fourier series
to exist and converge — e.g.: see § 2.4.1.
Various lecture notes for 18311. Rosales, MIT 11

Here we will assume that f is “nice” enough to justify the calculations below (two continuous
derivatives is enough, but not necessary). To be precise, here we assume that
⎫
The series in (2.3.1) converges absolutely and uniformly. In fact, we further ⎪
⎪
⎪
assume that ⎪
⎬
C (2.3.4)
|Cn (f )| ≤ 6 0,
for n =
|n|p ⎪
⎪
⎪
⎪
⎭
where C > 0 and p > 1 are constants — e.g. see remark 2.4.1.
Introduce a numerical grid on the line by
2π
xn = (n − 1) Δx, where Δx = (2.3.5)
N
n runs over the integers, and N is some “large” natural number. Then use the trapezoidal rule
to approximate the integrals in (2.3.2), thus obtaining discrete approximations for the Fourier
coefficients:
Z 2π N N
1 1 N 1 N
Cc = f (x) e−i c x dx ≈ fn e−i c xn Δx = fn w−(n−1) c (2.3.6)
2π 0 2 π n=1 N n=1

where fn = f (xn ) and (as usual) w = ei 2 π/N . Thus we write

N
1 N
Cc ≈ cc where cc = fn w−(n−1) c . (2.3.7)
N n=1

Now, question:
How good an approximation to C. is c. ? (2.3.8)

Remark 2.3.1 From the results in § 2.6, we should expect the approximation to be quite good,
but some caution is needed: the results in § 2.6 indicate that, for a fixed “nice enough” function,
the trapezoidal rule provides a very good approximation to the integral of the function as N → ∞.
However, here we have a whole sequence of functions that we are integrating (one for each f), so
the results in § 2.6 have to be taken with a grain of salt. At best they can be used to state that, for a
fixed set of Cc , the approximation in (2.3.7) gets better very fast as N → ∞ (at least for functions
with many derivatives).

Remark 2.3.2 A second note of caution comes from the observation that, when f is comparable
with N in size, the integrand in (2.3.6) has O(1) oscillations that occur on the same scale as Δx.
Clearly, when this happens, (2.3.7) can be a good approximation by accident only — definitely not
for generic functions f , no matter how nice they might be!

Remark 2.3.3 A final note of caution comes from the observation that (2.3.7) defines cc for all
integer values of f, in such a way that c.+N = c. . This makes Cc ≈ cc compatible with (2.3.4) only
for the trivial case Cc ≡ 0. Clearly, in general C. ≈ c. must fail for |£| = O(N ) or larger.
Various lecture notes for 18311. Rosales, MIT 12

When (2.3.4) applies, an explicit formula relating the cc to the Cc is possible, which can be used to
answer (2.3.8). The idea is to plug in the Fourier series for f into the definition of cc . This yields15
N
� ∞ � ∞ N ∞
1 N N i j xn −(n−1) c 1 N N
(n−1)(j−c)
N
cc = Cj (f ) e w = Cj (f ) w = Cc+k N , (2.3.9)
N n=1 j=−∞ N j=−∞ n=1 k=−∞

where we used that J = N (n−1) q

P
n=1 w = 0, unless q = multiple of N , when J = N . Note that this
formula also shows that {cc } is periodic of period N . This last equation can also be rewritten as

cc = Cc + (Cc+N + Cc−N ) + (Cc+2N + Cc−2N ) + . . . (2.3.10)

This makes it clear that the approximation in (2.3.7–2.3.8) is not very good outside the range
|f| « N . On the other hand, it also shows that we can write
N
c£ = C£ + O(N −p ) for |£| ≤ , (2.3.11)
2
where we have used (2.3.4). Hence, when p is large (the function has many derivatives), the
approximation in (2.3.7) is very good, at least for |£| ≤ N2 .

Remark 2.3.4 Note that, for f ≈ ± 12 N the relative error in (2.3.11) is quite large, since then all
the terms have (roughly) the same size. However, in this case both cc and Cc are small, and then it
does not matter.

Guided by the results above, we now complete the discretization of the Fourier series formulas by
adding to the approximation in (2.3.7) of the Fourier coefficients, the following approximations to
the Fourier series
NM
f (x) ≈ cc ei c x for N = 2 M + 1 odd.
c=−M
M −1
1 N 1
f (x) ≈ c−M e−i M x + cc ei c x + cM ei M x for N = 2 M even.
2 c=1−M
2
When used on the grid points xn these yield
M
N N
N −1
c (n−1)
fn = cc w = cc wc (n−1) ,
c=−M 0
M −1 N −1
1 N 1 N
fn = c−M w−M (n−1) + cc wc (n−1) + cM wM (n−1) = cc wc (n−1) ,
2 c=1−M
2 0

where in each case we use the periodicity properties to shift the summations to the range 0 ≤ f < N .
15
Absolute convergence justifies all the calculations here.
Various lecture notes for 18311. Rosales, MIT 13

Putting this all together yields the following discrete approximation to the Fourier series:
N −1 N
N 1 N
fn = c. w . (n−1)
and c. = fn w−(n−1) . , (2.3.12)
0
N n=1

where the {fn } correspond to the function values at the grid points — fn ≈ f (xn ), and the {ce ll}
are approximations to the Fourier coefficients — see (2.3.11).

Equation (2.3.12) should be compared with the DFT and iDFT ⎬

⎫

formulas in (2.2.2) and (2.2.3). It should be clear that, with (2.3.13)

⎭
the identification vn = fn and b. = c.−1 , they are the same.

2.3.1 Fourier series, DFT, and derivatives.

This subsection is yet to be written.

2.4 Simple convergence results for Fourier series.

In this subsection we present some simple convergence results for Fourier series

Theorem 2.4.1 Let f = f (x) be a periodic function of period 2 π — f (x + 2 π) = f (x). Assume

that f has at least two derivatives, with f ' ' integrable. Then the Fourier series in (2.3.1) converges
to f absolutely and uniformly. In fact
N N 1 � � Z 2π
N
'' 1 ''1
f (x) − Cn (f ) e inx
≤ 2 kf k1 2
=O 2
, where kf k1 = |f ' ' (x)| dx
−N n>N
n N 2π 0
(2.4.1)
and N ≥ 0 is any natural number.
Proof. Integrate by parts twice, and use the periodicity, to obtain
Z 2π Z 2π
1 ' −i n x 1
Cn (f ) = f (x) e dx = − f ' ' (x) e−i n x dx, (2.4.2)
2πin 0 2 π n2 0
from which Z 2π
1 1 ''
|Cn (f )| ≤ |f ' ' (x)| dx = kf k1 . (2.4.3)
2 π n2 0 n2
It follows that the series on the right hand side of (2.3.1) converges absolutely and uniformly. Hence it
defines a continuous function ∞
N
g(x) = Cn (f ) ei n x . (2.4.4)
−∞

Clearly, it is also the case that Cn (g) = Cn (f ) for all n. Hence, from theorem 2.4.2, g = f . The result
in (2.4.1) then follows from (2.4.3). ♣
Various lecture notes for 18311. Rosales, MIT 14

Remark 2.4.1 It should be clear that, if f is smoother than in the statement of the theorem above,
then the convergence properties of the Fourier series are even better. For example, if f has p ≥ 2
derivatives, with f (p) integrable, then (2.4.1) generalizes to
N N 1 
� �  Z 2π
N
inx (p) 1 (p) 1
f (x) − Cn (f ) e ≤ 2 kf k1 p
=O p
, where kf k1 = f (p) (x) dx
−N n>N
n N 2 π 0
(2.4.5)
16
and N ≥ 0 is any natural number. In particular: for smooth functions, the Fourier series
converges faster than any negative power of N = number of terms in the partial sums.
On the other hand, Fourier series also converge for periodic functions that are less smooth than in
the statement of the theorem above — although the convergence is then, generally, in weaker senses
than absolutely and uniformly. For example:
(1) If f is square integrable, then the Fourier series for f converges to f in the square norm. This
means that
N Z 2π
N
inx 1
f (x) − Cn (f ) e → 0 as N → ∞, where kg k2 = |g 2 (x)| dx (2.4.6)
−N
2 π 0
2

for any function square integrable function g.

(2) Let f be such that: (i) f is discontinuous at a finite set of points per period. (ii) Let a and b
be two successive points of discontinuity for f . Then f is twice continuously differentiable for
a ≤ x ≤ b. In this case the Fourier series for f converges as follows:
— To f (x) at every point x at which f is continuous.
— To 12 (f (x + 0) + f (x − 0)) at every point x at which f is discontinuous.
It is important to note that, in this case the convergence is not uniform. In fact, near each
discontinuity the partial Fourier sums over-shoot and under-shoot, by amounts that do not vanish
as N → ∞, and exhibit a oscillations in a small17 interval near each discontinuity. This is the
Gibbs’ phenomenon, of which we will see more later.

There are many, many, more known results characterizing how Fourier series converge under various
conditions. Another example is given in § 2.4.1.

Theorem 2.4.2 Let f (x) and g(x) be two continuous, 2π-periodic functions with the same Fourier
coefficients. Then f = g.
Proof. Clearly, Cn (h) = 0 for all n, where h = g − f . Hence
Z a+π N 
−i n x
h(x) cn e dx = 0, (2.4.7)
a−π " {z "
p=p(x)

16
The proof of this follows by performing p integrations by parts, instead of only two, in (2.4.2).
17
Vanishes as N → ∞.
Various lecture notes for 18311. Rosales, MIT 15

where a is arbitrary and the summation is over a finite number of exponentials, with coefficients cn —
we call p a trigonometric polynomial. We show below that this implies that h ≡ 0. (2.4.8)
Define the following trigonometric polynomials:
m
e + 2 + e−i x
� ix
 �
2 x
  m
−1 −1
pm (x) = γm = γm cos , (2.4.9)
4 2
where m = 1, 2, 3, . . . and Z +π  x 2 m
γm = cos dx > 0. (2.4.10)
−π 2
It should be clear that the pm have the properties below — see figure 2.4.1.
R +π
(i) pm (−π) = pm (π) = 0 and pm (x) > 0 for −π < x < π, with −π pm (x) dx = 1.
(ii) For any f, δ > 0 there exists 0 < M < ∞ such that pm (x) ≤ f for δ ≤ |x| ≤ π and m ≥ M .

These properties imply that pm (x) behaves like the Dirac “delta function” as m → ∞. In particular, let
y = y(x) be a continuous function in −π ≤ x ≤ π, then
Z +π
lim y(x) pm (x) dx = y(0). (2.4.11)
m→∞ −π

This is proved in lemma 2.4.1. Thus we can write, using (2.4.7)

Z a+π Z +π
0 = lim h(x)pm (x − a) dx = lim h(x + a)pm (x) dx = h(a). (2.4.12)
m→∞ a−π m→∞ −π

Since a is arbitrary, this proves (2.4.8). ♣

Lemma 2.4.1 Equation (2.4.11) applies.

Proof. Let Y = max |y(x) − y(0)|. Also, because y is continuous, for any f > 0, there is a δ > 0
−π≤x≤π

such that |y(0) − y(x)| ≤ f for |x| ≤ δ. Select now M as in item (ii) above (2.4.11). Then, for m ≥ M
Z +π Z −δ Z π
y(x) pm (x) dx − y(0) = (y(x) − y(0)) pm (x) dx + (y (x) − y(0)) pm (x) dx
−π −π δ
" {z " " {z " " {z "
I I1 I2
Z +δ
+ (y(x) − y(0)) pm (x) dx, (2.4.13)
−δ
" {z "
I3

where we have used item (i) above (2.4.11). Clearly |I1 | ≤ f Y (π − δ) and |I2 | ≤ f Y (π − δ),
while Z +δ
|I2 | ≤ f pm (x) dx ≤ f.
−δ
Various lecture notes for 18311. Rosales, MIT 16
Trigonometric polynomial pm for various values of m.

1.5

p
1

0.5

0
−3 −2 −1 0 1 2 3
Figure 2.4.1: Trigonometric polynomials pm for: m = 1 (blue), m = 4 (red), m = 16 (black), and
m = 64 (magenta). As m → ∞, pm ∼ δ(x) for |x| ≤ π — where δ(x) is the Dirac “delta function”.

Hence Z +π
y(x) pm (x) dx − y(0) ≤ f + 2 f Y (π − δ),
−π

which shows that I can be made as small as desired by taking m large enough. ♣

Problem 2.4.1 Compute the asymptotic value of an integral.

Find the leading order contribution to the integral defining γm in equation (2.4.10), when m » 1.
Hint. As items (i-ii) below equation (2.4.10) show — see also figure 2.4.1, as m → ∞, the main
contribution to the integral defining γm arises from a small neighborhood of the origin. To be precise
Z δ Z +π 
x 2 m x 2 m 3
γm = 2 cos dx + 2 cos dx, where δ = 2 m− 8 . (2.4.14)
0 2 δ 2
" {z "
I

1 2
x
≤ e− 8 x , it follows that


However, since 0 ≤ cos 2
Z +π 1
m m
x2 δ2
0<I≤2 e− 4 dx ≤ 2 (π − δ) e− 4 ≤ 2 π e−m 4 = exponentially small. (2.4.15)
δ

Hence, for m » 1, Z δ
x 2 m  1
−m 4
γm = 2 cos dx + O e . (2.4.16)
0 2
This can be exploited to find a simple approximation to the value of γm for m » 1, as follows:
Various lecture notes for 18311. Rosales, MIT 17

x


2 m x



(a) Write cos 2
= exp 2 m ln cos 2
.
x

x


2 m
(b) Expand ln cos 2
in powers of x, to obtain an approximation to cos 2
, valid for 0 ≤ x ≤ δ.
(c) Substitute the result of (b) into (2.4.16), and do the integral. This should give a very simple
formula for the approximate value of γm for m » 1.
R∞ 2 √
(d) You will need the fact that 0 e−x dx = 12 π.

2.4.1 Convergence in the weak sense.

This subsubsection is yet to be written.

2.5 The main idea behind the FFT algorithm.

This subsection is yet to be written.

2.6 Trapezoidal rule and the Euler-Maclaurin formula.

The purpose of this subsection is to produce explicit formulas for the error in the trapezoidal rule
of integration. In particular, we will show that: when integrating a periodic function over a period,
the order of the error is directly proportional to the degree of smoothness of the integrand.
We begin by considering a function f = f (x) with N continuous derivatives, defined for 0 ≤ x ≤ 1.
Then we use integration by parts, using the properties of the Bernoulli polynomials — see § 2.6.1
R1
— to obtain a formula for 0 f (x) dx in terms of the values of the function and its derivatives at
the end points. Note that
Z 1 Z 1
f (x) dx = B0 (x) f (x) dx,
0 0
Z 1 Z 1 Z 1
' 1
B0 (x) f (x) dx = B1 (x) f (x) dx = (f (1) + f (0)) − B1 (x) f ' (x) dx,
2
Z 01 Z0 1 ' 0
Z 1
' B2 (x) ' β2 ' ' B2 (x) ' '
B1 (x) f (x) dx = f (x) dx = (f (1) − f (0)) − f (x) dx,
0 0 2 2 0 2
Z 1 Z 1 ' Z 1
B2 (x) ' ' B3 (x) ' ' β3 ' ' '' B3 (x) (3)
f (x) dx = f (x) dx = (f (1) − f (0)) − f (x) dx,
0 2 0 6 6 0 6
and so on, with the general formula for n ≥ 1 being
Z 1 Z 1 '
Bn (x) (n) Bn+1 (x) (n)
f (x) dx = f (x) dx
0 n! 0 (n + 1)!
Z 1
βn+1 (n) (n)

Bn+1 (x) (n+1)
= f (1) − f (0) − f (x) dx.
(n + 1)! 0 (n + 1)!
Various lecture notes for 18311. Rosales, MIT 18

Putting this all together, we obtain

Z 1
1 β2 ' β3 ' '
f (x) dx = (f (1) + f (0)) − (f (1) − f ' (0)) + (f (1) − f ' ' (0)) + . . . +
0 2 2 6
Z 1
N +1 βN (N −1) (N −1) N BN (x) (N )


(−1) f (1) − f (0) + (−1) f (x) dx
N! 0 N!
" {z "
EN =EN (f , 0, 1)
N −1
1 N βn+1
(−1)n f (n) (1) − f (n) (0) + EN .


= (f (1) + f (0)) + (2.6.1)
2 n=1
(n + 1)!

We are now ready to write the Euler-Maclaurin formula, which provides an expression for the error
in approximating the integral of a function using the trapezoidal rule. Let g = g(x) be a function
with N continuous derivatives, defined in some interval a ≤ x ≤ b.
Introduce a numerical grid in a ≤ x ≤ b by defining x. = a + £ h for 0 ≤ £ ≤ M , where M > 0
is a natural number and h = Δx = b−a M
. Then we can write
Z b M
N −1 Z xe +h M
N −1 Z 1
g(x) dx = g(x) dx = h g(xc + h x) dx. (2.6.2)
a c=0 xe c=0 0

We now apply the equality in (2.6.1) to each one of the terms in the sum on the right hand side in
(2.6.2), with f (x) = fc (x) = g(xc + h x) in each case. It is then easy to see that this yields
Z b � M −1
�
! N −1
1 N 1 N βn+1
(−h)n+1 g (n) (b) − g (n) (a) +


g(x) dx = g(a) + g(xc ) + g(b) h −
a 2 2 (n + 1)!
" c=1
{z " n=1
Trapezoidal rule.
M
N −1
h EN (fc , 0, 1), (2.6.3)
c=0
" {z "
EN (g)=EN (g, a, b)

which is the Euler-Maclaurin summation formula. In particular, note that

Z 1
N BN (x) (N )
EN (fc , 0, 1) = (−h) g (xc + h x) dx. (2.6.4)
0 N!

Hence, using (2.6.12) we see that

Z 1 Z xe +h
N (N ) N −1
|EN (fc , 0, 1)| ≤ h g (xc + h x) dx = h g (N ) (x) dx, (2.6.5)
0 xe

from which it follows that Z b

N
|EN (g, a, b)| ≤ h g (N ) (x) dx. (2.6.6)
a
Various lecture notes for 18311. Rosales, MIT 19

Remark 2.6.1 Trapezoidal rule for periodic functions. Suppose that g above in (2.6.3) is peri­
odic of period b − a. Then
Z b M
N
g(x) dx = h g(xc ) + EN (g, a, b), where EN (g, a, b) = O(hN ). (2.6.7)
a c=1

In particular: for smooth periodic functions, the error the trapezoidal rule approximation to
their integral over one period vanishes, as h → 0, faster than any power of h.
By contrast, notice that Equations (2.6.3) and (2.6.6) show that, for generic functions (with, at
least, a second derivative that is integrable) the trapezoidal rule is second order only
Z b � M −1
�
!
1 N 1 1
g(x) dx = g(a) + g(xc ) + g(b) h − h2 β2 (g ' (b) − g ' (a)) + E2 (g, a, b), (2.6.8)
a 2 c=1
2 2

where E2 (g, a, b) = O(h2 ).

2.6.1 Bernoulli polynomials.

The Bernoulli polynomials Bn = Bn (x) and the Bernoulli numbers βn are defined as follows
⎫
(a) B0 = 1, ⎪
⎪
Bn'
⎪
(b) = nB for n > 0, ⎬
R 1 n−1 (2.6.9)
(c) 0 = 0
Bn (x) dx for n > 0, ⎪
⎪
⎪
⎭
(d) βn = Bn (0),
d
where the prime denotes dx and n = 0, 1, 2, 3, . . . These equations determine the polynomials
recursively — the arbitrary constant of integration in (b) is determined by the condition in (c).
The first few Bernoulli polynomials are B0 = 1, B1 = x − 21 , B2 = 12 x2 − 12 x + 12 1
, ...
Notice that
Bn is a degree n polynomial. (2.6.10)
Bn (0) = Bn (1) = βn for n ≥ 2. (2.6.11)

The first statement here follows by induction from (a) and (b) in (2.6.9). The second follows from
(b) and (c) in (2.6.9).

Lemma 2.6.1 Define the constants Mn by Mn = max IBn (x)I. Then Mn ≤ n!. (2.6.12)
0≤x≤1

Proof: Clearly true for n = 0. For n > 0, (c) in (2.6.9) shows that Bn has a zero at some 0 < xn < 1.
Rx
Hence from (b) in (2.6.9) Bn (x) = n xn Bn−1 (s) ds, thus |Bn (x)| ≤ n Mn−1 |x − xn | ≤ n Mn−1 . ♣
Various lecture notes for 18311. Rosales, MIT 20

Definition 2.6.1 The generating function for the Bernoulli polynomials is defined by
∞
N 1 n
G(x, t) = t Bn (x). (2.6.13)
n=0
n!

Note that, from (2.6.12), the series defining G converges for all 0 ≤ x ≤ 1 and |t| < 1 — in fact,
it can be show that it converges for all x and all |t| < 2 π.

Problem 2.6.1 Compute the Bernoulli polynomials’ generating function.

Find an explicit expression for G in (2.6.13).
Hint: Use (b) in (2.6.9) to compute Gx and find a simple ode in x that G solves, for each t. Write the
general solution to this ode — this solution will have a free “constant” — which is actually a function
of t, say f = f (t). Determine f = f (t) using (a) and (c) in (2.6.9), which can be used to obtain the
R1
value of 0 G(x, t) dx.

The End.
MIT OpenCourseWare
http://ocw.mit.edu

18.311 Principles of Applied Mathematics

Spring 2014

For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.