Numerical Methods for Partial

Differential Equations

Seongjai Kim

Department of Mathematics and Statistics

Mississippi State University
Mississippi State, MS 39762 USA
Email: [email protected]

April 13, 2023

Seongjai Kim, Department of Mathematics and Statistics, Mississippi State
University, Mississippi State, MS 39762-5921 USA Email: [email protected].
The work of the author is supported in part by NSF grant DMS-1228337.
Prologue

In the area of “Numerical Methods for Differential Equations", it seems very

hard to find a textbook incorporating mathematical, physical, and engineer-
ing issues of numerical methods in a synergistic fashion. So the first goal of
this lecture note is to provide students a convenient textbook that addresses
both physical and mathematical aspects of numerical methods for partial dif-
ferential equations (PDEs).
In solving PDEs numerically, the following are essential to consider:

• physical laws governing the differential equations (physical understand-

ing),
• stability/accuracy analysis of numerical methods (mathematical under-
standing),
• issues/difficulties in realistic applications, and
• implementation techniques (efficiency of human efforts).

In organizing the lecture note, I am indebted by Ferziger and Peric [23], John-
son [32], Strikwerda [64], and Varga [68], among others. Currently the lecture
note is not fully grown up; other useful techniques would be soon incorporated.
Any questions, suggestions, comments will be deeply appreciated.

3
4
Contents

Title 2

Prologue 3

Table of Contents 9

1 Mathematical Preliminaries 1
1.1. Taylor’s Theorem & Polynomial Fitting . . . . . . . . . . . . . . . . . . . . . . . 2
1.2. Finite Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2.1. Uniformly spaced grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2.2. General grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.3. Overview of PDEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.4. Difference Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
1.5. Homework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2 Numerical Methods for ODEs 31

2.1. Taylor-Series Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.1.1. The Euler method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.1.2. Higher-order Taylor methods . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.2. Runge-Kutta Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.2.1. Second-order Runge-Kutta method . . . . . . . . . . . . . . . . . . . . . . 41
2.2.2. Fourth-order Runge-Kutta method . . . . . . . . . . . . . . . . . . . . . . 44
2.2.3. Adaptive methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.3. Accuracy Comparison for One-Step Methods . . . . . . . . . . . . . . . . . . . . 47
2.4. Multi-step Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.5. High-Order Equations & Systems of Differential Equations . . . . . . . . . . . 52
2.6. Homework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

3 Properties of Numerical Methods 55

3.1. A Model Problem: Heat Conduction in 1D . . . . . . . . . . . . . . . . . . . . . . 56
3.2. Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

5
6 Contents

3.3. Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.4. Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.4.1. Approaches for proving stability . . . . . . . . . . . . . . . . . . . . . . . 70
3.4.2. The von Neumann analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.4.3. Influence of lower-order terms . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.5. Boundedness – Maximum Principle . . . . . . . . . . . . . . . . . . . . . . . . . 77
3.5.1. Convection-dominated fluid flows . . . . . . . . . . . . . . . . . . . . . . . 78
3.5.2. Stability vs. boundedness . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.6. Conservation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.7. A Central-Time Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.8. The θ-Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
3.8.1. Stability analysis for the θ-Method . . . . . . . . . . . . . . . . . . . . . . 84
3.8.2. Accuracy order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.8.3. Maximum principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
3.8.4. Error analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
3.9. Homework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

4 Finite Difference Methods for Elliptic Equations 91

4.1. Finite Difference (FD) Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.1.1. Constant-coefficient problems . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.1.2. General diffusion coefficients . . . . . . . . . . . . . . . . . . . . . . . . . 96
4.1.3. FD schemes for mixed derivatives . . . . . . . . . . . . . . . . . . . . . . 98
4.1.4. L∞ -norm error estimates for FD schemes . . . . . . . . . . . . . . . . . . 98
4.1.5. The Algebraic System for FDM . . . . . . . . . . . . . . . . . . . . . . . . 105
4.2. Solution of Linear Algebraic Systems . . . . . . . . . . . . . . . . . . . . . . . . . 109
4.2.1. Direct method: the LU factorization . . . . . . . . . . . . . . . . . . . . . 110
4.2.2. Linear iterative methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
4.2.3. Convergence theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
4.2.4. Relaxation methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
4.2.5. Line relaxation methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
4.3. Krylov Subspace Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
4.3.1. Steepest descent method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
4.3.2. Conjugate gradient (CG) method . . . . . . . . . . . . . . . . . . . . . . . 135
4.3.3. Preconditioned CG method . . . . . . . . . . . . . . . . . . . . . . . . . . 138
4.4. Other Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
4.4.1. Incomplete LU-factorization . . . . . . . . . . . . . . . . . . . . . . . . . . 140
4.5. Numerical Examples with Python . . . . . . . . . . . . . . . . . . . . . . . . . . 144
4.6. Homework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

5 Finite Element Methods for Elliptic Equations 153

Contents 7

5.1. Finite Element (FE) Methods in 1D Space . . . . . . . . . . . . . . . . . . . . . . 154

5.1.1. Variational formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
5.1.2. Formulation of FEMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
5.2. The Hilbert spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
5.3. An error estimate for FEM in 1D . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
5.4. Other Variational Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
5.5. FEM for the Poisson equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
5.5.1. Integration by parts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
5.5.2. Defining FEMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
5.5.3. Assembly: Element stiffness matrices . . . . . . . . . . . . . . . . . . . . 189
5.5.4. Extension to Neumann boundary conditions . . . . . . . . . . . . . . . . 191
5.6. Finite Volume (FV) Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
5.7. Average of The Diffusion Coefficient . . . . . . . . . . . . . . . . . . . . . . . . . 198
5.8. Abstract Variational Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
5.9. Numerical Examples with Python . . . . . . . . . . . . . . . . . . . . . . . . . . 203
5.10.Homework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206

6 FD Methods for Hyperbolic Equations 209

6.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
6.2. Basic Difference Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
6.2.1. Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
6.2.2. Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
6.2.3. Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
6.2.4. Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
6.3. Conservation Laws . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
6.3.1. Euler equations of gas dynamics . . . . . . . . . . . . . . . . . . . . . . . 227
6.4. Shocks and Rarefaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
6.4.1. Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
6.4.2. Weak solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
6.5. Numerical Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
6.5.1. Modified equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
6.5.2. Conservative methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
6.5.3. Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
6.5.4. Godunov’s method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
6.6. Nonlinear Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
6.6.1. Total variation stability (TV-stability) . . . . . . . . . . . . . . . . . . . . 252
6.6.2. Total variation diminishing (TVD) methods . . . . . . . . . . . . . . . . . 254
6.6.3. Other nonoscillatory methods . . . . . . . . . . . . . . . . . . . . . . . . . 255
6.7. Numerical Examples with Python . . . . . . . . . . . . . . . . . . . . . . . . . . 260
8 Contents

6.8. Homework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262

7 Domain Decomposition Methods 265

7.1. Introduction to DDMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
7.2. Overlapping Schwarz Alternating Methods (SAMs) . . . . . . . . . . . . . . . . 269
7.2.1. Variational formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
7.2.2. SAM with two subdomains . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
7.2.3. Convergence analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
7.2.4. Coarse subspace correction . . . . . . . . . . . . . . . . . . . . . . . . . . 274
7.3. Nonoverlapping DDMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
7.3.1. Multi-domain formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
7.3.2. The Steklov-Poincaré operator . . . . . . . . . . . . . . . . . . . . . . . . 279
7.3.3. The Schur complement matrix . . . . . . . . . . . . . . . . . . . . . . . . 281
7.4. Iterative DDMs Based on Transmission Conditions . . . . . . . . . . . . . . . . 284
7.4.1. The Dirichlet-Neumann method . . . . . . . . . . . . . . . . . . . . . . . 284
7.4.2. The Neumann-Neumann method . . . . . . . . . . . . . . . . . . . . . . . 286
7.4.3. The Robin method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
7.4.4. Remarks on DDMs of transmission conditions . . . . . . . . . . . . . . . 288
7.5. Homework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294

8 Multigrid Methods∗ 297

8.1. Introduction to Multigrid Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 298
8.2. Homework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299

9 Locally One-Dimensional Methods 301

9.1. Heat Conduction in 1D Space: Revisited . . . . . . . . . . . . . . . . . . . . . . . 302
9.2. Heat Equation in Two and Three Variables . . . . . . . . . . . . . . . . . . . . . 308
9.2.1. The θ-method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
9.2.2. Convergence analysis for θ-method . . . . . . . . . . . . . . . . . . . . . . 311
9.3. LOD Methods for the Heat Equation . . . . . . . . . . . . . . . . . . . . . . . . . 314
9.3.1. The ADI method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
9.3.2. Accuracy of the ADI: Two examples . . . . . . . . . . . . . . . . . . . . . 321
9.3.3. The general fractional step (FS) procedure . . . . . . . . . . . . . . . . . 324
9.3.4. Improved accuracy for LOD procedures . . . . . . . . . . . . . . . . . . . 326
9.3.5. A convergence proof for the ADI-II . . . . . . . . . . . . . . . . . . . . . . 333
9.3.6. Accuracy and efficiency of ADI-II . . . . . . . . . . . . . . . . . . . . . . . 335
9.4. Homework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337

10 Special Schemes 339

10.1.Wave Propagation and Absorbing Boundary Conditions . . . . . . . . . . . . . . 340
Contents 9

10.1.1. Introduction to wave equations . . . . . . . . . . . . . . . . . . . . . . . . 340

10.1.2. Absorbing boundary conditions (ABCs) . . . . . . . . . . . . . . . . . . . 341
10.1.3. Waveform ABC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342

11 Projects∗ 347
11.1.High-order FEMs for PDEs of One Spacial Variable . . . . . . . . . . . . . . . . 347

A Basic Concepts in Fluid Dynamics 349

A.1. Conservation Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
A.2. Conservation of Mass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350
A.3. Conservation of Momentum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
A.4. Non-dimensionalization of the Navier-Stokes Equations . . . . . . . . . . . . . 354
A.5. Generic Transport Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
A.6. Homework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356

B Elliptic Partial Differential Equations 359

B.1. Regularity Estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
B.2. Maximum and Minimum Principles . . . . . . . . . . . . . . . . . . . . . . . . . 361
B.3. Discrete Maximum and Minimum Principles . . . . . . . . . . . . . . . . . . . . 363
B.4. Coordinate Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
B.5. Cylindrical and Spherical Coordinates . . . . . . . . . . . . . . . . . . . . . . . . 366

C Helmholtz Wave Equation∗ 369

D Richards’s Equation for Unsaturated Water Flow∗ 371

E Orthogonal Polynomials and Quadratures 373

E.1. Orthogonal Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373
E.2. Gauss-Type Quadratures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375

F Some Mathematical Formulas 379

F.1. Trigonometric Formulas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379
F.2. Vector Identities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379

G Finite Difference Formulas 381

Bibliography 383

Index 391
10 Contents
Chapter 1

Mathematical Preliminaries

In the approximation of derivatives, we consider the Taylor series expansion

and the curve-fitting as two of most popular tools. This chapter begins with
a brief review for these introductory techniques, followed by finite difference
schemes, and an overview of partial differential equations (PDEs).
In the study of numerical methods for PDEs, experiments such as the im-
plementation and running of computational codes are necessary to under-
stand the detailed properties/behaviors of the numerical algorithm under con-
sideration. However, these tasks often take a long time so that the work can
hardly be finished in a desired period of time. Particularly, it is the case for
the graduate students in classes of numerical PDEs. Basic software will be
provided to help you experience numerical methods satisfactorily.

1
2 CHAPTER 1. MATHEMATICAL PRELIMINARIES

1.1. Taylor’s Theorem & Polynomial Fitting

While the differential equations are defined on continuous variables, their nu-
merical solutions must be computed on a finite number of discrete points. The
derivatives should be approximated appropriately to simulate the physical
phenomena accurately and efficiently. Such approximations require various
mathematical and computational tools. In this section we present a brief re-
view for the Taylor’s series and the curve fitting.
Theorem 1.1. (Taylor’s Theorem). Assume that u ∈ C n+1 [a, b] and let
c ∈ [a, b]. Then, for every x ∈ (a, b), there is a point ξ that lies between x and c
such that
u(x) = pn (x) + En+1 (x), (1.1)
where pn is a polynomial of degree ≤ n and En+1 denotes the remainder defined
as n
X u(k) (c) k u(n+1) (ξ)
pn (x) = (x − c) , En+1 (x) = (x − c)n+1 .
k! (n + 1)!
k=0

The formula (1.1) can be rewritten for u(x + h) (about x) as follows: for
x, x + h ∈ (a, b),
n
X u(k) (x) k u(n+1) (ξ) n+1
u(x + h) = h + h (1.2)
k! (n + 1)!
k=0
1.1. Taylor’s Theorem & Polynomial Fitting 3

Curve fitting
Another useful tool in numerical analysis is the curve fitting. It is often the
case that the solution must be represented as a continuous function rather
than a collection of discrete values. For example, when the function is to be
evaluated at a point which is not a grid point, the function must be interpo-
lated near the point before the evaluation.
First, we introduce the existence theorem for interpolating polynomials.
Theorem 1.2. Let x0 , x1 , · · · , xN be a set of distinct points. Then, for arbi-
trary real values y0 , y1 , · · · , yN , there is a unique polynomial pN of degree ≤ N
such that
pN (xi ) = yi , i = 0, 1, · · · , N.
4 CHAPTER 1. MATHEMATICAL PRELIMINARIES

Lagrange interpolating polynomial

Let {a = x0 < x1 < · · · < xN = b} be a partition of the interval [a, b].
Then, the Lagrange form of interpolating polynomial is formulated as a
linear combination of the so-called cardinal functions:
N
X
pN (x) = LN,i (x)u(xi ). (1.3)
i=0

Here the cardinal functions are defined as

N  
Y x − xj
LN,i (x) = ∈ PN , (1.4)
j=0
xi − xj
j 6= i

where PN is the set of polynomials of degree ≤ N , which satisfy

LN,i (xj ) = δij , i, j = 0, 1, · · · , N.

1.1. Taylor’s Theorem & Polynomial Fitting 5

Newton polynomial
The Newton form of the interpolating polynomial that interpolates u at
{x0 , x1 , · · · , xN } is given as
N h
X k−1
Y i
pN (x) = ak (x − xj ) , (1.5)
k=0 j=0

where the coefficients ak , k = 0, 1, · · · , N , can be computed as divided differ-

ences
ak = u[x0 , x1 , · · · , xk ]. (1.6)
Definition 1.3. (Divided Differences). The divided differences for the
function u(x) are defined as

u[xj ] = u(xj ),
u[xj+1 ] − u[xj ]
u[xj , xj+1 ] = ,
xj+1 − xj (1.7)
u[xj+1 , xj+2 ] − u[xj , xj+1 ]
u[xj , xj+1 , xj+2 ] = ,
xj+2 − xj
and the recursive rule for higher-order divided differences is
u[xj , xj+1 , · · · , xm ]
u[xj+1 , xj+2 , · · · , xm ] − u[xj , xj+1 , · · · , xm−1 ] (1.8)
= ,
xm − xj
for j < m.
6 CHAPTER 1. MATHEMATICAL PRELIMINARIES

Table 1.1: Divided-difference table for u(x).

xj u[xj ] u[ , ] u[ , , ] u[ , , , ] u[ , , , , ]
x0 u[x0 ]
x1 u[x1 ] u[x0 , x1 ]
x2 u[x2 ] u[x1 , x2 ] u[x0 , x1 , x2 ]
x3 u[x3 ] u[x2 , x3 ] u[x1 , x2 , x3 ] u[x0 , x1 , x2 , x3 ]
x4 u[x4 ] u[x3 , x4 ] u[x2 , x3 , x4 ] u[x1 , x2 , x3 , x4 ] u[x0 , x1 , x2 , x3 , x4 ]

Example
1.1. Taylor’s Theorem & Polynomial Fitting 7

Figure 1.1: A Maple program

Interpolation Error Theorem

Theorem 1.4. (Interpolation Error Theorem). Let the interval be par-
titioned into {a = x0 < x1 < · · · < xN = b} and pN interpolate u at the nodal
points of the partitioning. Assume that u(N +1) (x) exists for each x ∈ [a, b].
Then, there is a point ξ ∈ [a, b] such that
N
u(N +1) (ξ) Y
u(x) = pN (x) + (x − xj ), ∀ x ∈ [a, b]. (1.9)
(N + 1)! j=0

Further, assume that the points are uniformly spaced and max |u(N +1) (x)| ≤
x∈[a,b]
M , for some M > 0. Then,
M  b − a N +1
max |u(x) − pN (x)| ≤ . (1.10)
x∈[a,b] 4(N + 1) N
8 CHAPTER 1. MATHEMATICAL PRELIMINARIES

1.2. Finite Differences

In this section, we present bases of finite difference (FD) approximations. Tay-
lor series approaches are more popular than curve-fitting approaches; how-
ever, higher-order FD schemes can be easily obtained by curve-fitting ap-
proaches, although grid points are not uniformly spaced.

1.2.1. Uniformly spaced grids

• Let h = (b − a)/N , for some positive integer N , and

xi = a + ih, i = 0, 1, · · · , N.

• Define ui = u(xi ), i = 0, 1, · · · , N .

Then, it follows from (1.2) that

uxx (xi ) 2 uxxx (xi ) 3
(a) ui+1 = ui + ux (xi )h + h + h
2! 3!
uxxxx (xi ) 4 uxxxxx (xi ) 5
+ h + h + ··· ,
4! 5! (1.11)
uxx (xi ) 2 uxxx (xi ) 3
(b) ui−1 = ui − ux (xi )h + h − h
2! 3!
uxxxx (xi ) 4 uxxxxx (xi ) 5
+ h − h + ··· .
4! 5!
1.2. Finite Differences 9

One-sided FD operators
Solve the above equations for ux (xi ) to have
ui+1 − ui uxx (xi ) uxxx (xi ) 2
ux (xi ) = − h− h
h 2! 3!
uxxxx (xi ) 3
− h + ··· ,
4! (1.12)
ui − ui−1 uxx (xi ) uxxx (xi ) 2
ux (xi ) = + h− h
h 2! 3!
uxxxx (xi ) 3
+ h − ··· .
4!
By truncating the terms including hk , k = 1, 2, · · · , we define the first-order
FD schemes
ui+1 − ui
ux (xi ) ≈ Dx+ ui := , (forward)
h (1.13)
− ui − ui−1
ux (xi ) ≈ Dx ui := , (backward)
h
where Dx+ and Dx− are called the forward and backward difference operators,
respectively.
10 CHAPTER 1. MATHEMATICAL PRELIMINARIES

Central FD operators
The central second-order FD scheme for ux : Subtract (1.11.b) from (1.11.a)
and divide the resulting equation by 2h.
ui+1 − ui−1 uxxx (xi ) 2
ux (xi ) = − h
2h 3! (1.14)
uxxxxx (xi ) 4
− h − ··· .
5!
Thus the central second-order FD scheme reads
ui+1 − ui−1
ux (xi ) ≈ Dx1 ui := . (central) (1.15)
2h
Note that the central difference operator Dx1 is the average of the forward and
backward operators, i.e.,
1 Dx+ + Dx−
Dx = .
2

A FD scheme for uxx (xi ): Add the two equations in (1.11) and divide the
resulting equation by h2 .
ui−1 − 2ui + ui+1 uxxxx (xi ) 2
uxx (xi ) = − 2 h
h2 4! (1.16)
uxxxxxx (xi ) 4
−2 h − ··· .
6!
Thus the central second-order FD scheme for uxx at xi reads
ui−1 − 2ui + ui+1
uxx (xi ) ≈ Dx2 ui := . (1.17)
h2
Note that
Dx2 = Dx− Dx+ = Dx+ Dx− . (1.18)

1.2.2. General grids

Taylor series approaches

For {a = x0 < x1 < · · · < xN = b}, a partition of the interval [a, b], let

hi = xi − xi−1 , i = 1, 2, · · · , N.
1.2. Finite Differences 11

The Taylor series expansions for ui+1 and ui−1 (about xi ) become
uxx (xi ) 2
(a) ui+1 = ui + ux (xi )hi+1 + hi+1
2!
uxxx (xi ) 3
+ hi+1 + · · · ,
3! (1.19)
uxx (xi ) 2
(b) ui−1 = ui − ux (xi )hi + hi
2!
uxxx (xi ) 3
− hi + · · · .
3!
which correspond to (1.11).
12 CHAPTER 1. MATHEMATICAL PRELIMINARIES

The second-order FD scheme for ux

Multiply (1.19.b) by ri2 (:= (hi+1 /hi )2 ) and subtract the resulting equation
from (1.19.a) to have
ui+1 − (1 − ri2 )ui − ri2 ui−1
ux (xi ) =
hi+1 + ri2 hi
h3i+1 + ri2 h3i
− uxxx (xi ) − · · ·
6(hi+1 + ri2 hi )
h2i ui+1 + (h2i+1 − h2i )ui − h2i+1 ui−1
=
hi hi+1 (hi + hi+1 )
hi hi+1
− uxxx (xi ) − · · · .
6
Thus the second-order approximation for ux (xi ) becomes
h2i ui+1 + (h2i+1 − h2i )ui − h2i+1 ui−1
ux (xi ) ≈ . (1.20)
hi hi+1 (hi + hi+1 )

Note: It is relatively easy to find the second-order FD scheme for ux in nonuni-

form grids, as just shown, using the Taylor series approach. However, for
higher-order schemes, it requires a tedious work for the derivation. The curve
fitting approached can be applied for the approximation of both ux and uxx
more conveniently.
1.2. Finite Differences 13

Figure 1.2: The curve fitting by the interpolating quadratic polynomial.

Curve fitting approaches

An alternative way of obtaining FD approximations is to

• fit the function to an interpolating polynomial &

• differentiate the resulting polynomial.

For example, the quadratic polynomial that interpolates u at {xi−1 , xi , xi+1 }

can be constructed as (see Figure 1.2)

p2 (x) = a0 + a1 (x − xi−1 ) + a2 (x − xi−1 )(x − xi ), (1.21)

where the coefficients ak , k = 0, 1, 2, are determined by e.g. the divided differ-

ences:
ui − ui−1
a0 = ui−1 , a1 = ,
hi
hi (ui+1 − ui ) − hi+1 (ui − ui−1 )
a2 = .
hi hi+1 (hi + hi+1 )
Thus
ux (xi ) ≈ p02 (xi ) = a1 + a2 hi
h2i ui+1 + (h2i+1 − h2i )ui − h2i+1 ui−1 (1.22)
= ,
hi hi+1 (hi + hi+1 )
which is second-order and identical to (1.20).
14 CHAPTER 1. MATHEMATICAL PRELIMINARIES

Higher-order FDs for ux(xi)

For higher-order approximations for ux (xi ), the function must be fit to higher-
degree polynomials that interpolate u at a larger set of grid points including xi .
For a fourth-order approximation, for example, we should construct a fourth-
degree polynomial.
Let pi−2,4 (x) be the fourth-order Newton polynomial that interpolates u at
{xi−2 , xi−1 , xi , xi+1 , xi+2 }, i.e.,
4 h
X k−1
Y i
pi−2,4 (x) = ai−2,k (x − xi−2+j ) , (1.23)
k=0 j=0

where
ai−2,k = u[xi−2 , xi−1 , · · · , xi−2+k ], k = 0, · · · , 4.
Then it follows from the Interpolation Error Theorem (1.9) that
ux (xi ) = p0i−2,4 (xi )
u(5) (ξ)
+ (xi − xi−2 )(xi − xi−1 )(xi − xi+1 )(xi − xi+2 ).
5!
Therefore, under the assumption that u(5) (x) exists, p0i−2,4 (xi ) approximates
ux (xi ) with a fourth-order truncation error.
1.2. Finite Differences 15

FDs for uxx(xi)

The second-derivative uxx can be approximated by differentiating the inter-
polating polynomial twice. For example, from p2 in (1.21), we have
hi (ui+1 − ui ) − hi+1 (ui − ui−1 )
uxx (xi ) ≈ p002 (xi ) = 2
hi hi+1 (hi + hi+1 )
(1.24)
hi+1 ui−1 − (hi + hi+1 )ui + hi ui+1
= 1 .
2 hi hi+1 (hi + h i+1 )
The above approximation has a first-order accuracy for general grids. How-
ever, it turns out to be second-order accurate when hi = hi+1 ; compare it with
the one in (1.17).

A higher-order FD scheme for uxx can be obtained from the twice differen-
tiation of pi−2,4 in (1.23):
uxx (xi ) ≈ p00i−2,4 (xi ), (1.25)
which is a third-order approximation and becomes fourth-order for uniform
grids.

The thumb of rule is to utilize higher-order interpolating polynomials for

higher-order FD approximations.
16 CHAPTER 1. MATHEMATICAL PRELIMINARIES

1.3. Overview of PDEs

Parabolic Equations
The one-dimensional (1D) differential equation

ut − α2 uxx = f (x, t), x ∈ (0, L), (1.26)

is a standard 1D parabolic equation, which is often called the heat/diffusion

equation.
The equation models many physical phenomena such as heat distribution
on a rod: u(x, t) represents the temperature at the position x and time t, α2 is
the thermal diffusivity of the material, and f (x, t) denotes a source/sink along
the rod.
When the material property is not uniform along the rod, the coefficient α
is a function of x. In this case, the thermal conductivity K depends on the
position x and the heat equation becomes

ut − ∇ · (K(x) ux )x = f (x, t). (1.27)

Note: To make the heat equation well-posed (existence, uniqueness, and sta-
bility), we have to supply an initial condition and appropriate boundary con-
ditions on the both ends of the rod.
1.3. Overview of PDEs 17

Heat equation in 2D/3D

In 2D or 3D, the heat equations can be formulated as

ut − ∇ · (K∇u) = f, (x, t) ∈ Ω × [0, J]

u(x, t = 0) = u0 (x), x ∈ Ω (IC) (1.28)
u(x, t) = g(x, t), (x, t) ∈ Γ × [0, J] (BC)

where Γ = ∂Ω, the boundary of Ω.

18 CHAPTER 1. MATHEMATICAL PRELIMINARIES

Hyperbolic Equations
The second-order hyperbolic differential equation
1
utt − uxx = f (x, t), x ∈ (0, L) (1.29)
v2
is often called the wave equation. The coefficient v is the wave velocity, while
f represents a source. The equation can be used to describe the vibration of a
flexible string, for which u denotes the displacement of the string.
In higher dimensions, the wave equation can be formulated similarly.

Elliptic Equations
The second-order elliptic equations are obtained as the steady-state solu-
tions (as t → ∞) of the parabolic and hyperbolic equations. For example,
−∇ · (K∇u) = f, x ∈ Ω
(1.30)
u(x) = g(x), x ∈ Γ

represents a steady-state heat distribution for the given heat source f and the
boundary condition g.
1.3. Overview of PDEs 19

Fluid Mechanics
The 2D Navier-Stokes (NS) equations for viscous incompressible fluid flows:

Momentum equations
ut + px − R1 ∆u + (u2 )x + (uv)y = g1

vt + py − R1 ∆v + (uv)x + (v 2 )y = g2 (1.31)

Continuity equation
ux + vy = 0

Here (u, v) denote the velocity fields in (x, y)-directions, respectively, p is the
pressure, R is the (dimensionless) Reynolds number, and (g1 , g2 ) are body
forces. See e.g. [23] for computational methods for fluid dynamics.
20 CHAPTER 1. MATHEMATICAL PRELIMINARIES

Finance Modeling
In option pricing, the most popular model is the Black-Scholes (BS) differ-
ential equation
1 2 2 ∂ 2u ∂S −
ut + σ S + rS S ru = 0 (1.32)
2 ∂S 2 ∂u
Here

• S(t) is the stock price at time t

• u = u(S(t), t) denotes the price of an option on the stock
• σ is the volatility of the stock
• r is the (risk-free) interest rate

Note that the BS model is a backward parabolic equation, which needs

a final condition at time T . For European calls, for example, we have the
condition
u(S, T ) = max(S − X, 0),
while for a put option, the condition reads

u(S, T ) = max(X − S, 0),

where X is the exercise price at the expiration date T .

• Call option: the right to buy the stock

• Put option: the right to sell the stock
1.3. Overview of PDEs 21

Image Processing

• As higher reliability and efficiency are required, PDE-based mathemati-

cal techniques have become important components of many research and
processing areas, including image processing.
• PDE-based methods have been applied for various image processing tasks
such as image denoising, interpolation, inpainting, segmentation, and ob-
ject detection.

Example: Image denoising

• Noise model:
f =u+η (1.33)
where f is the observed (noisy) image, u denotes the desired image, and η
is the noise.
• Optimization problem
Minimize the total variation (TV) with the constraint
ˆ
min |∇u|dx subj. tokf − uk2 = σ 2 . (1.34)
u Ω

Using a Lagrange multiplier, the above minimization problem can be

rewritten as ˆ λ
ˆ 
2
min |∇u|dx + (f − u) dx , (1.35)
u Ω 2 Ω
from which we can derive the corresponding Euler-Lagrange equation
 ∇u 
−∇ · = λ(f − u), (1.36)
|∇u|
which is called the TV model in image denoising [58].

Remarks:

• Many other image processing tasks (such as interpolation and inpaint-

ing) can be considered as “generalized denoising." For example, the main
22 CHAPTER 1. MATHEMATICAL PRELIMINARIES

issue in interpolation is to remove or significantly reduce artifacts of easy

and traditional interpolation methods, and the artifacts can be viewed as
noise [8, 34].
• Variants of the TV model can be applied for various image processing
tasks.
1.3. Overview of PDEs 23

Numerical methods for PDEs

• Finite difference method: Simple, easiest technique. It becomes quite

complex for irregular domains
• Finite element method: Most popular, due to most flexible over com-
plex domains
• Finite volume method: Very popular in computational fluid dynamics
(CFD).

– Surface integral over control volumes

– Locally conservative
• Spectral method: Powerful if the domain is simple and the solution is
smooth.
• Boundary element method: Useful for PDEs which can be formulated
as integral equations; it solves the problem on the boundary to find the
solution over the whole domain.

– The algebraic system is often full

– Not many problems can be written as integral equations. for example,
nonlinear equations

• Meshless/mesh-free method: Developed to overcome drawbacks of mesh-

ing and re-meshing, for example, in crack propagation problems and large
deformation simulations
24 CHAPTER 1. MATHEMATICAL PRELIMINARIES

1.4. Difference Equations

In this section, we will consider solution methods and stability analysis for
difference equations, as a warm-up problem.
Problem: Find a general form for yn by solving the recurrence relation
2yn+2 − 5yn+1 + 2yn = 0
(1.37)
y0 = 2, y1 = 1

Solution: Let
yn = α n . (1.38)
and plug it into the first equation of (1.37) to have

2αn+2 − 5αn+1 + 2αn = 0,

which implies
2α2 − 5α + 2 = 0. (1.39)
The last equation is called the characteristic equation of the difference
equation (1.37), of which the two roots are
1
α = 2, .
2
1.4. Difference Equations 25

Thus, the general solution of the difference equation reads

 1 n
n
yn = c1 2 + c2 , (1.40)
2
where c1 and c2 are constants. One can determine the constants using the
initial conditions in (1.37).
c2
y0 = c1 + c2 = 2, y1 = 2 c1 + =1
2
which implies
c1 = 0, c2 = 2. (1.41)
What we have found is that
 1 n
yn = 2 = 21−n . (1.42)
2
26 CHAPTER 1. MATHEMATICAL PRELIMINARIES

A small change in the initial conditions

Now, consider another difference equation with a little bit different initial
conditions from those in (1.37):
2wn+2 − 5wn+1 + 2wn = 0
(1.43)
w0 = 2, w1 = 1.01

Then, the difference equation has the general solution of the form as in (1.40):
 1 n
n
wn = c1 2 + c2 . (1.44)
2
Using the new initial conditions, we have
c2
w0 = c1 + c2 = 2, w1 = 2 c1 + = 1.01,
2
Thus, the solution becomes
1 n 299  1 n
wn = 2 + . (1.45)
150 150 2

Comparison

y0 = 2 w0 = 2
y1 = 1 w1 = 1.01
.. ..
. .
y10 = 9.7656 × 10−4 w10 = 6.8286
y20 = 9.5367 × 10−7 w20 = 6.9905 × 103

Thus, the difference equation in (1.37) or (1.43) is unstable.

1.4. Difference Equations 27

Stability Theory
Physical Definition: A (FD) scheme is stable if a small change in the initial
conditions produces a small change in the state of the system.

• Most aspects in the nature are stable.

• Some phenomena in the nature can be represented by differential equa-
tions (ODEs and PDEs), while they may be solved through difference
equations.
• Although ODEs and PDEs are stable, their approximations (finite differ-
ence equations) may not be stable. In this case, the approximation is a
failure.

Definition: A differential equation is

• stable if for every set of initial data, the solution remains bounded as
t → ∞.
• strongly stable if the solution approaches zero as t → ∞.
28 CHAPTER 1. MATHEMATICAL PRELIMINARIES

Stability of difference equations

Theorem 1.5. A finite difference equation is stable if and only if

(a) |α| ≤ 1 for all roots of the characteristic equation, and

(b) if |α| = 1 for some root, then the root is simple.

Theorem 1.6. A finite difference equation is strongly stable if and only if

|α| < 1 for all roots of the characteristic equation.
1.5. Homework 29

1.5. Homework
1. For an interval [a, b], let the grid be uniform:
b−a
xi = ih + a; i = 0, 1, · · · , N, h= . (1.46)
N
Second-order schemes for ux and uxx , on the uniform grid given as in
(1.46), respectively read
ui+1 − ui−1
ux (xi ) ≈ Dx1 ui = ,
2h
(1.47)
ui−1 − 2ui + ui+1
uxx (xi ) ≈ Dx2 ui = Dx+ Dx− ui = .
h2
(a) Use Divided Differences to construct the second-order Newton poly-
nomial p2 (x) which passes (xi−1 , ui−1 ), (xi , ui ), and (xi+1 , ui+1 ).
(b) Evaluate p02 (xi ) and p002 (xi ) to compare with the FD schemes in (1.47).
2. Find the general solution of each of the following difference equations:
(a) yn+1 = 3yn
(b) yn+1 = 3yn + 2
(c) yn+2 − 8yn+1 + 12yn = 0
(d) yn+2 − 6yn+1 + 9yn = 1
3. Determine, for each of the following difference equations, whether it is
stable or unstable.
(a) yn+2 − 5yn+1 + 6yn = 0
(b) 8yn+2 + 2yn+1 − 3yn = 0
(c) 3yn+2 + yn = 0
(d) 4yn+4 + 5yn+2 + yn = 0
30 CHAPTER 1. MATHEMATICAL PRELIMINARIES
Chapter 2

Numerical Methods for ODEs

The first-order initial value problem (IVP) is formulated as follows: find {yi (x) :
i = 1, 2, · · · , M } satisfying
dyi
= fi (x, y1 , y2 , · · · , yM ),
dx i = 1, 2, · · · , M, (2.1)
yi (x0 ) = yi0 ,

for a prescribed initial values {yi0 : i = 1, 2, · · · , M }.

We assume that (2.1) admits a unique solution in a neighborhood of x0 .
For simplicity, we consider the case M = 1:
dy
= f (x, y),
dx (2.2)
y(x0 ) = y0 .

It is known that if f and ∂f /∂y are continuous in a strip (a, b) × R containing

(x0 , y0 ), then (2.2) has a unique solution in an interval I, where x0 ∈ I ⊂ (a, b).

31
32 Chapter 2. Numerical Methods for ODEs

In the following, we describe step-by-step methods for (2.2); that is, we start
from y0 = y(x0 ) and proceed stepwise.

• In the first step, we compute y1 which approximate the solution y of (2.2)

at x = x1 = x0 + h, where h is the step size.
• The second step computes an approximate value y2 of the solution at x =
x2 = x0 + 2h, etc..

We first introduce the Taylor-series methods for (2.2), followed by Runge-

Kutta methods and multi-step methods. All of these methods are applicable
straightforwardly to (2.1).
2.1. Taylor-Series Methods 33

2.1. Taylor-Series Methods

Here we rewrite the initial value problem (IVP):
(
y0 = f (x, y),
(IVP) (2.3)
y(x0 ) = y0 .

For the problem, a continuous approximation to the solution y(x) will not be
obtained; instead, approximations to y will be generated at various points,
called mesh points, in the interval [x0 , T ] for some T > x0 .
Let

• h = (T − x0 )/nt , for an integer nt ≥ 1

• xn = x0 + nh, n = 0, 1, 2, · · · , nt
• yn be the approximate solution of y at xn
34 Chapter 2. Numerical Methods for ODEs

2.1.1. The Euler method

Let us try to find an approximation of y(x1 ), marching through the first subin-
terval [x0 , x1 ] and using a Taylor-series involving only up to the first-derivative
of y.
Consider the Taylor series

0h2 00
y(x + h) = y(x) + hy (x) + y (x) + · · · . (2.4)
2
Letting x = x0 and utilizing y(x0 ) = y0 and y 0 (x0 ) = f (x0 , y0 ), the value y(x1 )
can be approximated by
y1 = y0 + hf (x0 , y0 ), (2.5)
where the second- and higher-order terms of h are ignored.
Such an idea can be applied recursively for the computation of solution on
later subintervals. Indeed, since
h2 00
y(x2 ) = y(x1 ) + hy 0 (x1 ) + y (x1 ) + · · · ,
2
by replacing y(x1 ) and y 0 (x1 ) with y1 and f (x1 , y1 ), respectively, we obtain

y2 = y1 + hf (x1 , y1 ), (2.6)

which approximates the solution at x2 = x0 + 2h.

2.1. Taylor-Series Methods 35

Figure 2.1: The Euler method.

In general, for n ≥ 0,

yn+1 = yn + hf (xn , yn ) (2.7)

which is called the Euler method.

Geometrically it is an approximation of the curve {x, y(x)} by a polygon of
which the first side is tangent to the curve at x0 , as shown in Figure 2.1. For
example, y1 is determined by moving the point (x0 , y0 ) by the length of h with
the slope f (x0 , y0 ).
36 Chapter 2. Numerical Methods for ODEs

Convergence of the Euler method

Theorem 2.1. Let f satisfy the Lipschitz condition in its second variable,
i.e., there is λ > 0 such that

kf (x, y1 ) − f (x, y2 )k ≤ λky1 − y2 k, ∀ y1 , y2 . (2.8)

Then, the Euler method is convergent; more precisely,

C
kyn − y(xn )k ≤ h[(1 + λh)n − 1], n = 0, 1, 2, · · · . (2.9)
λ

Proof. The true solution y satisfies

y(xn+1 ) = y(xn ) + hf (xn , y(xn )) + O(h2 ). (2.10)

Thus it follows from (2.7) and (2.10) that

en+1 = en + h[f (xn , yn ) − f (xn , y(xn ))] + O(h2 )
= en + h[f (xn , y(xn ) + en ) − f (xn , y(xn ))] + O(h2 ),

where en = yn − y(xn ). Utilizing (2.8), we have

ken+1 k ≤ (1 + λh)ken k + Ch2 . (2.11)

Here we will prove (2.9) by using (2.11) and induction. It holds trivially when
n = 0. Suppose it holds for n. Then,
ken+1 k ≤ (1 + λh)ken k + Ch2
C
≤ (1 + λh) · h[(1 + λh)n − 1] + Ch2
λ
C
= h[(1 + λh)n+1 − (1 + λh)] + Ch2
λ
C
= h[(1 + λh)n+1 − 1],
λ
which completes the proof.
2.1. Taylor-Series Methods 37

2.1.2. Higher-order Taylor methods

These methods are based on Taylor series expansion.
If we expand the solution y(x), in terms of its mth-order Taylor polynomial
about xn and evaluated at xn+1 , we obtain

0 h2 00
y(xn+1 ) = y(xn ) + hy (xn ) + y (xn ) + · · ·
2!
m (2.12)
h (m) hm+1 (m+1)
+ y (xn ) + y (ξn ).
m! (m + 1)!
Successive differentiation of the solution, y(x), gives

y 0 (x) = f (x, y(x)), y 00 (x) = f 0 (x, y(x)), · · · ,

and generally,
y (k) (x) = f (k−1) (x, y(x)). (2.13)
Thus, we have
h2 0
y(xn+1 ) = y(xn ) + hf (xn , y(xn )) + f (xn , y(xn )) + · · ·
2!
(2.14)
hm (m−1) hm+1 (m)
+ f (xn , y(xn )) + f (ξn , y(ξn ))
m! (m + 1)!
38 Chapter 2. Numerical Methods for ODEs

The Taylor method of order m corresponding to (2.14) is obtained by

deleting the remainder term involving ξn :

yn+1 = yn + h Tm (xn , yn ), (2.15)

where
h 0
Tm (xn , yn ) = f (xn , yn ) + f (xn , yn ) + · · ·
2! (2.16)
hm−1 (m−1)
+ f (xn , yn ).
m!
Remarks

• m = 1 ⇒ yn+1 = yn + hf (xn , yn )
which is the Euler method.
h h 0 i
• m = 2 ⇒ yn+1 = yn + h f (xn , yn ) + f (xn , yn )
2
• As m increases, the method achieves higher-order accuracy; however, it
requires to compute derivatives of f (x, y(x)).
2.1. Taylor-Series Methods 39

Example: For the initial-value problem

y 0 = y − x3 + x + 1, y(0) = 0.5, (2.17)

find T3 (x, y).

• Solution: Since y 0 = f (x, y) = y − x3 + x + 1,

f 0 (x, y) = y 0 − 3x2 + 1
= (y − x3 + x + 1) − 3x2 + 1
= y − x3 − 3x2 + x + 2

and
f 00 (x, y) = y 0 − 3x2 − 6x + 1
= (y − x3 + x + 1) − 3x2 − 6x + 1
= y − x3 − 3x2 − 5x + 2
Thus
h 0 h2 00
T3 (x, y) = f (x, y) + f (x, y) + f (x, y)
2 6
h
= y − x3 + x + 1 + (y − x3 − 3x2 + x + 2)
2
2
h
+ (y − x3 − 3x2 − 5x + 2)
6
40 Chapter 2. Numerical Methods for ODEs

2.2. Runge-Kutta Methods

The Taylor-series method of the preceding section has the drawback of re-
quiring the computation of derivatives of f (x, y). This is a tedious and time-
consuming procedure for most cases, which makes the Taylor methods seldom
used in practice.
Runge-Kutta methods have high-order local truncation error of the Taylor
methods but eliminate the need to compute and evaluate the derivatives of
f (x, y). That is, the Runge-Kutta Methods are formulated, incorporating a
weighted average of slopes, as follows:

yn+1 = yn + h (w1 K1 + w2 K2 + · · · + wm Km ) , (2.18)

where

• wj ≥ 0 and w1 + w2 + · · · + wm = 1
• Kj are recursive evaluations of the slope f (x, y)
• Need to determine wj and other parameters to satisfy

w1 K1 + w2 K2 + · · · + wm Km ≈ Tm (xn , yn ) + O(hm ) (2.19)

That is, Runge-Kutta methods evaluate an average slope of f (x, y) on the

interval [xn , xn+1 ] in the same order of accuracy as the mth-order Taylor
method.
2.2. Runge-Kutta Methods 41

2.2.1. Second-order Runge-Kutta method

Formulation:
yn+1 = yn + h (w1 K1 + w2 K2 ) (2.20)
where
K1 = f (xn , yn )
K2 = f (xn + αh, yn + βhK1 )
Requirement: Determine w1 , w2 , α, β such that
w1 K1 + w2 K2 = T2 (xn , yn ) + O(h2 )
h
= f (xn , yn ) + f 0 (xn , yn ) + O(h2 )
2
Derivation: For the left-hand side of (2.20), the Taylor series reads

0 h2 00
y(x + h) = y(x) + hy (x) + y (x) + O(h3 ).
2
Since y 0 = f and y 00 = fx + fy y 0 = fx + fy f ,
h2
y(x + h) = y(x) + hf + (fx + fy f ) + O(h3 ). (2.21)
2
42 Chapter 2. Numerical Methods for ODEs

On the other hand, the right-side of (2.20) can be reformulated as

y + h(w1 K1 + w2 K2 )
= y + w1 hf (x, y) + w2 hf (x + αh, y + βhK1 )
= y + w1 hf + w2 h(f + αhfx + βhfy f ) + O(h3 )

which reads
y + h(w1 K1 + w2 K2 )
(2.22)
= y + (w1 + w2 )hf + h2 (w2 αfx + w2 βfy f ) + O(h3 )

The comparison of (2.21) and (2.22) drives the following result, for the
second-order Runge-Kutta methods.
Results:
1 1
w1 + w2 = 1, w2 α = , w2 β = (2.23)
2 2
2.2. Runge-Kutta Methods 43

Common Choices:
1
I. w1 = w2 = , α=β=1
2
Then, the algorithm becomes
h
yn+1 = yn + (K1 + K2 ) (2.24)
2

where
K1 = f (xn , yn )
K2 = f (xn + h, yn + hK1 )
This algorithm is the second-order Runge-Kutta (RK2) method, which
is also known as the Heun’s method.
1
II. w1 = 0, w2 = 1, α = β =
2
For the choices, the algorithm reads
 h h 
yn+1 = yn + hf xn + , yn + f (xn , yn ) (2.25)
2 2

which is also known as the modified Euler method.

44 Chapter 2. Numerical Methods for ODEs

2.2.2. Fourth-order Runge-Kutta method

Formulation:

yn+1 = yn + h (w1 K1 + w2 K2 + w3 K3 + w4 K4 ) (2.26)

where
K1 = f (xn , yn )
K2 = f (xn + α1 h, yn + β1 hK1 )
K3 = f (xn + α2 h, yn + β2 hK1 + β3 hK2 )
K4 = f (xn + α3 h, yn + β4 hK1 + β5 hK2 + β6 hK3 )
Requirement: Determine wj , αj , βj such that

w1 K1 + w2 K2 + w3 K3 + w4 K4 = T4 (xn , yn ) + O(h4 )
2.2. Runge-Kutta Methods 45

The most common choice: The most commonly used set of parameter val-
ues yields
h
yn+1 = yn + (K1 + 2K2 + 2K3 + K4 ) (2.27)
6

where
K1 = f (xn , yn )
1 1
K2 = f (xn + h, yn + hK1 )
2 2
1 1
K3 = f (xn + h, yn + hK2 )
2 2
K4 = f (xn + h, yn + hK3 )

The local truncation error for the above RK4 can be derived as
h5 (5)
y (ξn ) (2.28)
5!
for some ξn ∈ [xn , xn+1 ]. Thus the global error becomes
(T − x0 )h4 (5)
y (ξ) (2.29)
5!
for some ξ ∈ [x0 , T ]
46 Chapter 2. Numerical Methods for ODEs

2.2.3. Adaptive methods

• Accuracy of numerical methods can be improved by decreasing the step
size.
• Decreasing the step size ≈ Increasing the computational cost
• There may be subintervals where a relatively large step size suffices and
other subintervals where a small step is necessary to keep the truncation
error within a desired limit.
• An adaptive method is a numerical method which uses a variable step
size.
• Example: Runge-Kutta-Fehlberg method (RKF45), which uses RK5 to
estimate local truncation error of RK4.
2.3. Accuracy Comparison for One-Step Methods 47

2.3. Accuracy Comparison for One-Step Methods

For an accuracy comparison among the one-step methods presented in the
previous sections, consider the motion of the spring-mass system:
κ F0
y 00 (t) + y= cos(µt),
m m (2.30)
y(0) = c0 , y 0 (0) = 0,

where m is the mass attached at the end of a spring of the spring constant
κ, the term F0 cos(µt) is a periodic driving force of frequency µ, and c0 is the
initial displacement from the equilibrium position.

• It is not difficult to find the analytic solution of (2.30):

F0
y(t) = A cos(ωt) + cos(µt),
m(ω 2 − µ2 )
p
where ω = κ/m is the angular frequency and the coefficient A is deter-
mined corresponding to c0 .
• Let y1 = y and y2 = −y10 /ω. Then, we can reformulate (2.30) as

y10 = −ωy2 , y0 (0) = c0 ,

F0 (2.31)
y20 = ωy1 − cos(µt), y2 (0) = 0.
mω
See § 2.5 on page 52 for high-order equations.
• The motion is periodic only if µ/ω is a rational number. We choose

m = 1, F0 = 40, A = 1 (c0 ≈ 1.33774), ω = 4π, µ = 2π. (2.32)

Thus the fundamental period of the motion

2πq 2πp
T = = = 1.
ω µ
See Figure 2.2 for the trajectory of the mass satisfying (2.31)-(2.32).
48 Chapter 2. Numerical Methods for ODEs

Figure 2.2: The trajectory of the mass satisfying (2.31)-(2.32).

Accuracy comparison

Table 2.1: The `2 -error at t = 1 for various time step sizes.

1/h Euler Heun RK4

100 1.19 3.31E-2 2.61E-5
200 4.83E-1 (1.3) 8.27E-3 (2.0) 1.63E-6 (4.0)
400 2.18E-1 (1.1) 2.07E-3 (2.0) 1.02E-7 (4.0)
800 1.04E-1 (1.1) 5.17E-4 (2.0) 6.38E-9 (4.0)

Table 2.1 presents the `2 -error at t = 1 for various time step sizes h, defined
as
 2  h 2 1/2
|ynht h
− y(1)| = y1,nt − y1 (1) + y2,nt − y2 (1) ,

where ynht denotes the computed solution at the nt -th time step with h = 1/nt .

• The numbers in parenthesis indicate the order of convergence α, defined

2.3. Accuracy Comparison for One-Step Methods 49

as
log(E(2h)/E(h))
α := ,
log 2
where E(h) and E(2h) denote the errors obtained with the grid spacing to
be h and 2h, respectively.
• As one can see from the table, the one-step methods exhibit the expected
accuracy.
• RK4 shows a much better accuracy than the lower-order methods, which
explains its popularity.
50 Chapter 2. Numerical Methods for ODEs

2.4. Multi-step Methods

The problem: The first-order initial value problem (IVP)
(
y0 = f (x, y),
(IVP) (2.33)
y(x0 ) = y0 .

Numerical Methods:

• Single-step/Starting methods: Euler’s method, Modified Euler’s, Runge-

Kutta methods
• Multi-step/Continuing methods: Adams-Bashforth-Moulton

Definition: An m-step method, m ≥ 2, for solving the IVP, is a difference

equation for finding the approximation yn+1 at x = xn+1 , given by
yn+1 = a1 yn + a2 yn−1 + · · · + am yn+1−m
+h[b0 f (xn+1 , yn+1 ) + b1 f (xn , yn ) + · · · (2.34)
+bm f (xn+1−m , yn+1−m )]

The m-step method is said to be

(
explicit or open, if b0 = 0
implicit or closed, if b0 6= 0
2.4. Multi-step Methods 51

Fourth-order multi-step methods

Let yi0 = f (xi , yi ).

• Adams-Bashforth method (explicit)

h
yn+1 = yn + (55yn0 − 59yn−1
0 0
+ 37yn−2 0
− 9yn−3 )
24
• Adams-Moulton method (implicit)
h 0
yn+1 = yn + (9yn+1 + 19yn0 − 5yn−1
0 0
+ yn−2 )
24
• Adams-Bashforth-Moulton method (predictor-corrector)
h
∗
yn+1 = yn + (55yn0 − 59yn−1
0 0
+ 37yn−2 0
− 9yn−3 )
24
h ∗
yn+1 = yn + (9y 0 n+1 + 19yn0 − 5yn−1
0 0
+ yn−2 )
24
where y 0 ∗n+1 = f (xn+1 , yn+1
∗
)

Remarks

• y1 , y2 , y3 can be computed by RK4.

• Multi-step methods may save evaluations of f (x, y) such that in each step,
they require only one new evaluation of f (x, y) to fulfill the step.
• RK methods are accurate enough and easy to implement, so that multi-
step methods are rarely applied in practice.
• ABM shows a strong stability for special cases, occasionally but not
often [11].
52 Chapter 2. Numerical Methods for ODEs

2.5. High-Order Equations & Systems of Differ-

ential Equations
The problem: 2nd-order initial value problem (IVP)
(
y 00 = f (x, y, y 0 ), x ∈ [x0 , T ]
(2.35)
y(x0 ) = y0 , y 0 (x0 ) = u0 ,

Let u = y 0 . Then,
u0 = y 00 = f (x, y, y 0 ) = f (x, y, u)

An equivalent problem: Thus, the above 2nd-order IVP can be equivalently

written as the following system of first-order DEs:
(
y 0 = u, y(x0 ) = y0 ,
x ∈ [x0 , T ] (2.36)
u0 = f (x, y, u), u(x0 ) = u0 ,

Notes:

• The right-side of the DEs involves no derivatives.

• The system (2.36) can be solved by one of the numerical methods (we have
studied), after modifying it for vector functions.
2.6. Homework 53

2.6. Homework
1. For the IVP in (2.17),
(a) Find T4 (x, y).
(b) Perform two steps of the 3rd and 4th-order Taylor methods, with h =
1/2, to find an approximate solutions of y at x = 1.
(c) Compare the errors, given that the exact solution
7
y(x) = 4 + 5x + 3x2 + x3 − ex
2
2. Derive the global error of RK4 in (2.29), given the local truncation error
(2.28).
3. Write the following DE as a system of first-order differential equations.
x00 + x0 y − 2y 00 = t,
−2y + y 00 + x = e−t ,

where the derivative denotes d/dt.

54 Chapter 2. Numerical Methods for ODEs
Chapter 3

Properties of Numerical Methods

Numerical methods compute approximate solutions for differential equations

(DEs). In order for the numerical solution to be a reliable approximation of
the given problem, the numerical method should satisfy certain properties. In
this chapter, we consider properties of numerical methods that are most com-
mon in numerical analysis such as consistency, convergence, stability, accuracy
order, boundedness/maximum principle, and conservation.

55
56 Chapter 3. Properties of Numerical Methods

3.1. A Model Problem: Heat Conduction in 1D

Let Ω = (0, 1) and J = (0, T ], for some T > 0. Consider the following simplest
model problem for parabolic equations in one-dimensional (1D) space:
ut − uxx = f, (x, t) ∈ Ω × J,
u = 0, (x, t) ∈ Γ × J, (3.1)
u = u0 , x ∈ Ω, t = 0,

where f is a heat source, Γ denotes the boundary of Ω, i.e., Γ = {0, 1}, and u0
is the prescribed initial value of the solution at t = 0.
3.1. A Model Problem: Heat Conduction in 1D 57

Finite difference methods

We begin with our discussion of finite difference (FD) methods for (3.1) by
partitioning the domain. Let
∆t = T /nt , tn = n∆t, n = 0, 1, · · · , nt ;
∆x = 1/nx , xj = j∆x, j = 0, 1, · · · , nx ;

for some positive integers nt and nx . Define unj = u(xj , tn ).

Let
S n := Ω × (tn−1 , tn ] (3.2)
be the nth space-time slice. Suppose that the computation has been performed
for uk = {ukj }, 0 ≤ k ≤ n − 1. Then, the task is to compute un by integrating the
equation on the space-time slice S n , utilizing FD schemes.

The basic idea of FD schemes is to replace derivatives by FD approxima-

tions. It can be done in various ways; here we consider most common ways
that are based on the Taylor’s formula.
58 Chapter 3. Properties of Numerical Methods

Recall the central second-order FD formula for uxx presented in (1.16):

ui−1 − 2ui + ui+1 uxxxx (xi ) 2
uxx (xi ) = − 2 h
h2 4! (3.3)
uxxxxxx (xi ) 4
−2 h − ··· .
6!
Apply the above to have

n
unj−1 − 2unj + unj+1
uxx (xj , t ) =
∆x2 (3.4)
uxxxx (xj , tn ) 2
−2 ∆x + O(∆x4 ).
4!

For the temporal direction, one can also apply a difference formula for the
approximation of the time-derivative ut . Depending on the way of combining
the spatial and temporal differences, the resulting scheme can behave quite
differently.
3.1. A Model Problem: Heat Conduction in 1D 59

Explicit Scheme
The following presents the simplest scheme:
n−1
vjn − vjn−1 vj−1 − 2vjn−1 + vj+1
n−1
− 2 = fjn−1 (3.5)
∆t ∆x

which is an explicit scheme for (3.1), called the forward Euler method.
Here vjn is an approximation of unj .

The above scheme can be rewritten as

vjn = µ vj−1
n−1
+ (1 − 2µ) vjn−1 + µ vj+1
n−1
+ ∆tfjn−1 (3.6)

where
∆t
µ=
∆x2
60 Chapter 3. Properties of Numerical Methods

3.2. Consistency
The bottom line for an accurate numerical method is that the discretization
becomes exact as the grid spacing tends to zero, which is the basis of consis-
tency.

Definition 3.1. Given a PDE P u = f and a FD scheme P∆x,∆t v = f ,

the FD scheme is said to be consistent with the PDE if for every smooth
function φ(x, t)

P φ − P∆x,∆t φ → 0 as (∆x, ∆t) → 0,

with the convergence being pointwise at each grid point.

Not all numerical methods based on Taylor series expansions are consis-
tent; sometimes, we may have to restrict the manner in which ∆x and ∆t
approach zero in order for them to be consistent.
3.2. Consistency 61

Example 3.2. The forward Euler scheme (3.5) is consistent.

Proof. For the heat equation in 1D,
∂ ∂2 
Pφ ≡ − 2 φ = φt − φxx .
∂t ∂x
The forward Euler scheme (3.5) reads
φnj − φn−1
j φn−1 n−1
j−1 − 2φj + φn−1
j+1
P∆x,∆t φ = − 2
∆t ∆x
The truncation error for the temporal discretization can be obtained applying
the one-sided FD formula:

n−1
φij − φn−1
j
φt (xj , t ) =
∆t (3.7)
φtt (xj , tn−1 )
− ∆t + O(∆t2 ).
2!
It follows from (3.4) and (3.7) that the truncation error of the forward Euler
scheme evaluated at (xj , tn−1 ) becomes

(P φ − P∆x,∆t φ) (xj , tn−1 )

φtt (xj , tn−1 ) φxxxx (xj , tn−1 ) 2
=− ∆t + 2 ∆x (3.8)
2! 4!
+O(∆t2 + ∆x4 ),

which clearly approaches zero as (∆x, ∆t) → 0.

62 Chapter 3. Properties of Numerical Methods

Truncation Error
Definition 3.3. Let u be smooth and

P u(xj , tn ) = P∆x,∆t unj + Tunj , (3.9)

Then, Tunj is called the truncation error of the FD scheme P∆x,∆t v = f eval-
uated at (xj , tn ).

It follows from (3.8) that the truncation error of the forward Euler scheme
(3.5) is
O(∆t + ∆x2 )
for all grid points (xj , tn ).
3.3. Convergence 63

3.3. Convergence
A numerical method is said to be convergent if the solution of the FD scheme
tends to the exact solution of the PDE as the grid spacing tends to zero. We
define convergence in a formal way as follows:

Definition 3.4. A FD scheme approximating a PDE is said to be conver-

gent if
u(x, t) − vjn → 0, as (xj , tn ) → (x, t) and (∆x, ∆t) → 0,
where u(x, t) is the exact solution of PDE and vjn denotes the the solution of
the FD scheme.

Consistency implies that the truncation error

(P u − P∆x,∆t u) → 0, as (∆x, ∆t) → 0.

So consistency is certainly necessary for convergence, but may not be suffi-

cient.
64 Chapter 3. Properties of Numerical Methods

Example 3.5. The forward Euler scheme (3.5) is convergent, when

∆t 1
µ= ≤ . (3.10)
∆x2 2

Proof. (The scheme) Recall the explicit scheme (3.5):

vjn − vjn−1 vj−1

n−1
− 2vjn−1 + vj+1
n−1
− 2 = fjn−1 (3.11)
∆t ∆x
which can be expressed as

P∆x,∆t vjn−1 = fjn−1 (3.12)

On the other hand, for the exact solution u,

P∆x,∆t un−1
j + Tun−1
j = fjn−1 (3.13)

(Error equation) Let

enj = unj − vjn ,
where u is the exact solution of (3.1). Then, from (3.12) and (3.13), the error
equation becomes
P∆x,∆t en−1
j = −T un−1
j ,
which in detail reads
enj − ejn−1 en−1 n−1
j−1 − 2ej + en−1
j+1
− 2 = −Tun−1
j . (3.14)
∆t ∆x
In order to control the error more conveniently, we reformulate the error equa-
tion
enj = µ ej−1
n−1
+ (1 − 2µ) en−1
j + µ en−1 n−1
j+1 − ∆t T uj . (3.15)

(Error analysis with `∞ -norm) Now, define

E n = max |enj |, T n = max |T unj |, Tb = max T n .

j j n

Note that vj0 = u0j for all j and therefore E 0 = 0.

3.3. Convergence 65

It follows from (3.15) and the assumption (3.10) that

|enj | ≤ µ |en−1 n−1
j−1 | + (1 − 2µ) |ej | + µ |en−1
j+1 |

+∆t |T un−1
j |
≤ µ E n−1 + (1 − 2µ) E n−1 + µ E n−1 (3.16)
+∆t T n−1
= E n−1 + ∆t T n−1 .

Since the above inequality holds for all j, we have

E n ≤ E n−1 + ∆t T n−1 , (3.17)

and therefore
E n ≤ E n−1 + ∆t T n−1
≤ E n−2 + ∆t T n−1 + ∆t T n−2
≤ ··· (3.18)
n−1
X
0
≤ E + ∆t T k .
k=1

Since E 0 = 0,
E n ≤ (n − 1)∆t Tb ≤ T Tb , (3.19)
where T is the upper bound of the time available. Since Tb = O(∆t + ∆x2 ), the
maximum norm of the error approaches zero as (∆x, ∆t) → 0.
66 Chapter 3. Properties of Numerical Methods

Remarks
• The assumption µ ≤ 1/2 makes coefficients in the forward Euler scheme
n−1 n−1 n−1
(3.6) nonnegative, which in turn makes vjn a weighted average of {vj−1 , vj , vj+1 }.
• The analysis can often conclude

E n = O(Tb ), ∀ n

• Convergence is what a numerical scheme must satisfy.

• However, showing convergence is not easy in general, if attempted in a
direct manner as in the previous example.
• There is a related concept, stability, that is easier to check.
3.3. Convergence 67

An Example: µ ≤ 1/2

Figure 3.1: The explicit scheme (forward Euler) in Maple.

The problem:
ut − α2 uxx = 0, (x, t) ∈ [0, 1] × [0, 1],
u = 0, (x, t) ∈ {0, 1} × [0, 1], (3.20)
u = sin(πx), x ∈ [0, 1], t = 0,
The exact solution:
2
u(x, t) = e−π t sin(πx)
68 Chapter 3. Properties of Numerical Methods

Parameter setting:
a := 0; b := 1; T := 1; α := 1; f := 0;
nx := 10;

Numerical results:
nt := 200 (µ = 1/2) kunt − v nt k∞ = 7.94 × 10−6
nt := 170 (µ ≈ 0.588) kunt − v nt k∞ = 1.31 × 109

• For the case µ ≈ 0.588, the numerical solution becomes oscillatory and
blows up.
3.4. Stability 69

3.4. Stability
The example with Figure 3.1 shows that consistency of a numerical method is
not enough to guarantee convergence of its solution to the exact solution. In
order for a consistent numerical scheme to be convergent, a required property
is stability. Note that if a scheme is convergent, it produces a bounded solution
whenever the exact solution is bounded. This is the basis of stability. We first
define the L2 -norm of grid function v:
 X 1/2
2
kvk∆x = ∆x |vj | .
j

Definition 3.6. A FD scheme P∆x,∆t v = 0 for a homogeneous PDE P u = 0

is stable if for any positive T , there is a constant CT such that
M
X
n
kv k∆x ≤ CT kum k∆x , (3.21)
m=0

for 0 ≤ tn ≤ T and for ∆x and ∆t sufficiently small. Here M is chosen to

incorporate the data initialized on the first M + 1 levels.
70 Chapter 3. Properties of Numerical Methods

3.4.1. Approaches for proving stability

There are two fundamental approaches for proving stability:

• The Fourier analysis (von Neumann analysis)

It applies only to linear constant coefficient problems.
• The energy method
It can be used for more general problems with variable coefficients and
nonlinear terms. But it is quite complicated and the proof is problem
dependent.

Theorem 3.7. (Lax-Richtmyer Equivalence Theorem). Given a well-

posed linear initial value problem and its FD approximation that satisfies
the consistency condition, stability is a necessary and sufficient condition
for convergence.

The above theorem is very useful and important. Proving convergence is

difficult for most problems. However, the determination of consistency of a
scheme is quite easy as shown in §3.2, and determining stability is also easier
than showing convergence. Here we introduce the von Neumann analysis of
stability of FD schemes, which allows one to analyze stability much simpler
than a direct verification of (3.21).
3.4. Stability 71

Theorem 3.8. A FD scheme P∆x,∆t v = 0 for a homogeneous PDE P u = 0

is stable if
kv n k∆x ≤ (1 + C∆t)kv n−1 k∆x , (3.22)
for some C ≥ 0 independent on ∆t

Proof. Recall ∆t = T /nt , for some positive integer nt . A recursive application

of (3.22) reads
kv n k∆x ≤ (1 + C∆t)kv n−1 k∆x ≤ (1 + C∆t)2 kv n−2 k∆x
(3.23)
≤ · · · ≤ (1 + C∆t)n kv 0 (= u0 )k∆x .

Here the task is to show (1 + C∆t)n is bounded by some positive number CT

for n = 1, · · · , nt , independently on ∆t. Since ∆t = T /nt , we have

(1 + C∆t)n = (1 + CT /nt )n
≤ (1 + CT /nt )nt
h iCT
nt /CT
= (1 + CT /nt )
≤ eCT ,

which proves (3.21) with by CT := eCT .

72 Chapter 3. Properties of Numerical Methods

3.4.2. The von Neumann analysis

• Let φ be a grid function defined on grid points of spacing ∆x and φj =
φ(j∆x). Then, its Fourier transform is given by, for ξ ∈ [−π/∆x, π/∆x],
∞
1 X −ij∆xξ
φ(ξ) = √
b e φj , (3.24)
2π j=−∞

and the inverse formula is

ˆ π/∆x
1
φj = √ eij∆xξ φ(ξ)dξ.
b (3.25)
2π −π/∆x

• Parseval’s identity
kφn k∆x = kφbn k∆x , (3.26)
where ∞
 X 1/2
n 2
kφ k∆x = |φj | ∆x ,
j=−∞
ˆ π/∆x 1/2
bn
kφ k∆x = |φ(ξ)|
b 2
dξ
−π/∆x

• The stability inequality (3.21) can be replaced by

M
X
n
kb
v k∆x ≤ CT v m k∆x .
kb (3.27)
m=0

• Thus stability can be determined by providing (3.27) in the frequency

domain.
3.4. Stability 73

Example
To show how one can use the above analysis, we exemplify the forward Euler
scheme (3.6), with f = 0:
vjn = µ vj−1
n−1
+ (1 − 2µ) vjn−1 + µ vj+1
n−1
(3.28)

• The inversion formula implies

ˆ π/∆x
1
vjn =√ eij∆xξ vbn (ξ) dξ. (3.29)
2π −π/∆x

Thus it follows from (3.28) and (3.29) that

ˆ π/∆x
n 1
vj = √ F∆x,j (ξ) dξ, (3.30)
2π −π/∆x
where
F∆x,j (ξ) = µei(j−1)∆xξ vbn−1 (ξ)
+(1 − 2µ)eij∆xξ vbn−1 (ξ)
+µei(j+1)∆xξ vbn−1 (ξ)
= eij∆xξ [µ e−i∆xξ + (1 − 2µ) + µ ei∆xξ ] vbn−1 (ξ)
• Comparing (3.29) with (3.30), we obtain
vbn (ξ) = [µ e−i∆xξ + (1 − 2µ) + µ ei∆xξ ] vbn−1 (ξ) (3.31)

• Letting ϑ = ∆xξ, we define the amplification factor for the scheme (3.6)
by
g(ϑ) = µ e−i∆xξ + (1 − 2µ) + µ ei∆xξ
= µ e−iϑ + (1 − 2µ) + µ eiϑ
(3.32)
= (1 − 2µ) + 2µ cos(ϑ)
= 1 − 2µ(1 − cos(ϑ)) = 1 − 4µ sin2 (ϑ/2)
• Equation (3.31) can be rewritten as
vbn (ξ) = g(ϑ) vbn−1 (ξ) = g(ϑ)2 vbn−2 (ξ) = · · · = g(ϑ)n vb0 (ξ). (3.33)
Therefore, when g(ϑ)n is suitably bounded, the scheme is stable. In fact,
g(ϑ)n would be uniformly bounded only if |g(ϑ)| ≤ 1 + C∆t.
74 Chapter 3. Properties of Numerical Methods

• It is not difficult to see

|g(ϑ)| = |1 − 2µ(1 − cos(ϑ))| ≤ 1

only if
0 ≤ µ ≤ 1/2 (3.34)
which is the stability condition of the scheme (3.6).
3.4. Stability 75

The von Neumann analysis: Is it complicated?

A simpler and equivalent procedure of the von Neumann analysis can be sum-
marized as follows:

• Replace vjn by g n eijϑ for each value of j and n.

• Find conditions on coefficients and grid spacings which would satisfy |g| ≤
1 + C∆t, for some C ≥ 0.

The forward Euler scheme (3.6):

vjn = µ vj−1
n−1
+ (1 − 2µ) vjn−1 + µ vj+1
n−1

Replacing vjn with g n eijϑ gives

g n eijϑ = µ g n−1 ei(j−1)ϑ + (1 − 2µ) g n−1 eijϑ + µ g n−1 ei(j+1)ϑ

Dividing both sides of the above by g n−1 eijϑ , we obtain

g = µ e−iϑ + (1 − 2µ) + µ eiϑ

which is exactly the same as in (3.32)

76 Chapter 3. Properties of Numerical Methods

3.4.3. Influence of lower-order terms

Let us consider the model problem (3.1) augmented by lower-order terms

ut = uxx + aux + bu (3.35)

where a and b are constants.

We can construct an explicit scheme
vjn − vjn−1 n−1
vj−1 − 2vjn−1 + vj+1
n−1 n−1
vj+1 n−1
− vj−1
= 2 +a + b vjn−1 (3.36)
∆t ∆x 2∆x
From the von Neumann analysis, we can obtain the amplification factor
a∆t
g(ϑ) = 1 − 4µ sin2 (ϑ/2) + i sin(ϑ) + b∆t, (3.37)
∆x
which gives

2  a∆t
2
2 2
|g(ϑ)| = 1 − 4µ sin (ϑ/2) + b∆t + sin(ϑ)
∆x

2
= 1 − 4µ sin2 (ϑ/2) + 2 1 − 4µ sin2 (ϑ/2) b∆t

 a∆t 2
2
+(b∆t) + sin(ϑ)
∆x
Hence, under the condition 0 < µ = ∆t/∆x2 ≤ 1/2,

2 |a|2
2
|g(ϑ)| ≤ 1 + 2|b|∆t + (b∆t) + ∆t

2 2 (3.38)
≤ 1 + (|b| + |a|2 /4) ∆t .

Thus, lower-order terms do not change the stability condition. (Homework for
details.)
3.5. Boundedness – Maximum Principle 77

3.5. Boundedness – Maximum Principle

Numerical solutions should lie between proper bounds. For example, physical
quantities such as density and kinetic energy of turbulence must be positive,
while concentration should be between 0 and 1.

In the absence of sources and sinks, some variables are required to have
maximum and minimum values on the boundary of the domain. The above
property is call the maximum principle, which should be inherited by the
numerical approximation.
78 Chapter 3. Properties of Numerical Methods

3.5.1. Convection-dominated fluid flows

To illustrate boundedness of the numerical solution, we consider the convection-
diffusion problem:
ut − εuxx + aux = 0. (3.39)
where ε > 0.
When the spatial derivatives are approximated by central differences, the
algebraic equation for unj reads
h −un−1 + 2un−1 − un−1 un−1 n−1 i
j−1 j j+1 j+1 − uj−1
unj = ujn−1 − ε +a ∆t,
∆x2 2∆x
or
σ  n−1
  σ  n−1
= d+unj u n−1
+ (1 − 2d)uj + d − u , (3.40)
2 j−1 2 j+1
where the dimensionless parameters are defined as
ε∆t a∆t
d= and σ = .
∆x2 ∆x
• σ: the Courant number
• ∆x/a: the characteristic convection time
• ∆x2 /ε: the characteristic diffusion time
These are the time required for a disturbance to be transmitted by con-
vection and diffusion over a distance ∆x.
3.5. Boundedness – Maximum Principle 79

3.5.2. Stability vs. boundedness

The requirement that the coefficients of the old nodal values be nonnegative
leads to
|σ|
(1 − 2d) ≥ 0, ≤ d. (3.41)
2

• The first condition leads to the limit on ∆t as

∆x2
∆t ≤ ,
2ε
which guarantees stability of (3.40). Recall that lower-order terms do
not change the stability condition (§3.4.3).

• The second condition imposes no limit on the time step. But it gives a
relation between convection and diffusion coefficients.

• The cell Peclet number is defined and bounded as

|σ| |a|∆x
Pecell := = ≤ 2. (3.42)
d ε
which is a sufficient (but not necessary) condition for boundedness of
the solution of (3.40).
80 Chapter 3. Properties of Numerical Methods

3.6. Conservation
When the equations to be solved are from conservation laws, the numerical
scheme should respect these laws both locally and globally. This means that
the amount of a conserved quantity leaving a control volume is equal to the
amount entering to adjacent control volumes.
If divergence form of equations and a finite volume method is used, this
is readily guaranteed for each individual control volume and for the solution
domain as a whole.
For other discretization methods, conservation can be achieved if care is
taken in the choice of approximations. Sources and sinks should be carefully
treated so that the net flux for each individual control volume is conservative.
Conservation is a very important property of numerical schemes. Once
conservation of mass, momentum, and energy is guaranteed, the error of con-
servative schemes is only due to an improper distribution of these quantities
over the solution domain.
Non-conservative schemes can produce artificial sources or sinks, changing
the balance locally or globally. However, non-conservative schemes can be
consistent and stable and therefore lead to correct solutions in the limit of
mesh refinement; error due to non-conservation is appreciable in most cases
only when the mesh is not fine enough.
The problem is that it is difficult to know on which mesh the non-conservation
error is small enough. Conservative schemes are thus preferred.
3.7. A Central-Time Scheme 81

3.7. A Central-Time Scheme

Before we begin considering general implicit methods, we would like to men-
tion an interesting scheme for solving (3.1):
vjn+1 − vjn−1 vj−1
n
− 2vjn + vj+1
n
− 2 = fjn , (3.43)
2∆t ∆x
of which the truncation error

Trunc.Err = O(∆t2 + ∆x2 ). (3.44)

To study its stability, we set f ≡ 0 and substitute vjn = g n eijϑ into (3.43) to
obtain
g − 1/g e−iϑ − 2 + eiϑ
− = 0,
2∆t ∆x2
or
g 2 + (8µ sin2 (ϑ/2))g − 1 = 0. (3.45)
We see that (3.45) has two distinct real roots g1 and g2 which should satisfy

g1 · g2 = −1. (3.46)

Hence the magnitude of one root must be greater than one, for some modes
and for all µ > 0, for which we say that the scheme is unconditionally un-
stable.
This example warns us that we need be careful when developing a FD
scheme. We cannot simply put combinations of difference approximations to-
gether.
82 Chapter 3. Properties of Numerical Methods

3.8. The θ-Method

Let A1 be the central second-order approximation of −∂xx , defined as
n
vj−1 − 2vjn + vj+1
n
A1 vjn := − .
∆x2
Then the θ-method for (3.1) is
v n − v n−1
+ A1 θv n + (1 − θ)v n−1 = f n−1+θ ,
 
(3.47)
∆t
for θ ∈ [0, 1], or equivalently

(I + θ∆tA1 )v n
(3.48)
= [I − (1 − θ)∆tA1 ]v n−1 + ∆tf n−1+θ .

The following three choices of θ are popular.

• Forward Euler method (θ = 0): The algorithm (3.48) is reduced to

v n = (I − ∆tA1 )v n−1 + ∆tf n−1 , (3.49)

which is the explicit scheme in (3.6), requiring the stability condition

∆t 1
µ= ≤ .
∆x2 2
3.8. The θ-Method 83

• Backward Euler method (θ = 1): This is an implicit method written

as
(I + ∆tA1 )v n = v n−1 + ∆tf n . (3.50)

– The method must invert a tridiagonal matrix to get the solution in

each time level.
– But it is unconditionally stable, stable independently on the choice
of ∆t.

• Crank-Nicolson method (θ = 1/2):

 ∆t  n  ∆t  n−1
I+ A1 v = I − A1 v + ∆tf n−1/2 . (3.51)
2 2
– It requires to solve a tridiagonal system in each time level, as in the
backward Euler method.
– However, the Crank-Nicolson method is most popular, because it is
second-order in both space and time and unconditionally stable.
– The Crank-Nicolson method can be viewed as an explicit method in
the first half of the space-time slice S n (:= Ω×(tn−1 , tn ]) and an implicit
method in the second half of S n . Hence it is often called a semi-
implicit method.
84 Chapter 3. Properties of Numerical Methods

3.8.1. Stability analysis for the θ-Method

Setting f ≡ 0, the algebraic system (3.48) reads pointwisely
n
−θµ vj−1 + (1 + 2θµ)vjn − θµ vj+1
n

n−1
(3.52)
= (1 − θ)µ vj−1 + [1 − 2(1 − θ)µ]vjn−1 + (1 − θ)µ vj+1
n−1
,

where µ = ∆t/∆x2 .
For an stability analysis for this one-parameter family of systems by uti-
lizing the von Neumann analysis in §3.4.2, substitute g n eijϑ for vjn in (3.52) to
have
g −θµ e−iϑ + (1 + 2θµ) − θµ eiϑ
 

= (1 − θ)µ e−iϑ + [1 − 2(1 − θ)µ] + (1 − θ)µ eiϑ .

That is,
1 − 2(1 − θ)µ (1 − cos ϑ)
g =
1 + 2θµ (1 − cos ϑ)
(3.53)
1 − 4(1 − θ)µ sin2 (ϑ/2)
= .
1 + 4θµ sin2 (ϑ/2)
Because µ > 0 and θ ∈ [0, 1], the amplification factor g cannot be larger than
one. The condition g ≥ −1 is equivalent to

1 − 4(1 − θ)µ sin2 (ϑ/2) ≥ − 1 + 4θµ sin2 (ϑ/2) ,

 

or
ϑ 1
(1 − 2θ)µ sin2 ≤ .
2 2
Thus the θ-method (3.48) is stable if
1
(1 − 2θ)µ ≤ . (3.54)
2
In conclusion:

• The θ-method is unconditionally stable for θ ≥ 1/2

• When θ < 1/2, the method is stable only if
∆t 1
µ= ≤ , θ ∈ [0, 1/2). (3.55)
∆x2 2(1 − 2θ)
3.8. The θ-Method 85

3.8.2. Accuracy order

We shall choose (xj , tn−1/2 ) for the expansion point in the following derivation
for the truncation error of the θ-method.
The arguments in §1.2 give
unj − ujn−1 h uttt  ∆t 2 in−1/2
= ut + + ··· . (3.56)
∆t 6 2 j

Also from the section, we have

`
h uxxxx 2 uxxxxxx 4 i`
A1 uj = − uxx + ∆x + 2 ∆x + · · · , ` = n − 1, n.
12 6! j

We now expand each term in the right side of the above equation in powers
of ∆t, about (xj , tn−1/2 ), to have

(n− 12 )± 12
h uxxxx 2 uxxxxxx 4 in−1/2
A1 uj = − uxx + ∆x + 2 ∆x + · · ·
12 6! j
∆t h uxxxxt 2 uxxxxxxt 4 in−1/2
∓ uxxt + ∆x + 2 ∆x + · · · (3.57)
2 12 6! j
1  ∆t 2 h uxxxxtt 2 in−1/2
− uxxtt + ∆x + · · · − ··· .
2 2 12 j

It follows from (3.56) and (3.57) that

unj − ujn−1 uttt  ∆t 2
+ O(∆t4 )
 n n−1

+ A1 θuj + (1 − θ)uj = ut +
∆t 6 2
 uxxxx 2 uxxxxxx 4
− uxx + ∆x + 2 ∆x + · · ·
12 6! (3.58)
∆t uxxxxt 2 uxxxxxxt 4 
− (2θ − 1) uxxt + ∆x + 2 ∆x + · · ·
2   12 6!
1 ∆t 2 uxxxxtt 2 
− uxxtt + ∆x + · · · − · · · ,
2 2 12
of which the right side is evaluated at (xj , tn−1/2 ).
86 Chapter 3. Properties of Numerical Methods

So the truncation error T u(:= P u − P∆x,∆t u) turns out to be

n−1/2
 1 uxxxx 2 uttt 2 uxxtt 2
T uj = θ − uxxt ∆t + ∆x − ∆t + ∆t
2 12 24 8
 1  uxxxxt uxxxxxx 4
+ θ− ∆t∆x2 + 2 ∆x + · · ·
2 12 6!
h 1 ∆x2 i ∆t2
= θ − ∆t + uxxt + uttt
2 12 12
h 1 ∆x2 i ∆x2  1 2
+ θ − ∆t + uxxxxt − 2
− uxxxxxx ∆x4 + · · · ,
2 12 12 12 6!
(3.59)
where we have utilized ut = uxx + f .
Thus the accuracy order reads
1

2 2

 O(∆t + ∆x ) when θ = ,

 2
2 4 1 ∆x2 (3.60)
O(∆t + ∆x ) when θ = − ,
2 12∆t



O(∆t + ∆x2 ) otherwise.


Note that the second choice of θ in (3.60) is less than 1/2, which is equivalent
to
∆t 1
2 = .
∆x 6(1 − 2θ)
Hence it satisfies (3.55); the method is stable and we can take large time steps
while maintaining accuracy and stability. For example, when ∆x = ∆t = 0.01,
we have θ = 21 − 1200
1
for the (2, 4)-accuracy scheme in time-space.
3.8. The θ-Method 87

3.8.3. Maximum principle

For heat conduction without interior sources/sinks, it is known mathemati-
cally and physically that the extreme values of the solution appear either in
the initial data or on the boundary. This property is called the maximum
principle.

• It is quite natural and sometimes very important to examine if the nu-

merical solution satisfies the maximum principle.
• Once the scheme satisfies the maximum principle, the solution will never
involve interior local extrema.
88 Chapter 3. Properties of Numerical Methods

Theorem 3.9. (Maximum principle for θ-method) Let f = 0 and the

θ-method be set satisfying θ ∈ [0, 1] and
1
(1 − θ)µ ≤ . (3.61)
2
If the computed solution v has an interior maximum or minimum, then v
is constant.

Proof. We rewrite the component-wise expression of the θ-method, (3.52), in

the form
(1 + 2θµ)vjn = θµ(vj−1
n n
+ vj+1 n−1
) + (1 − θ)µ(vj−1 n−1
+ vj+1 )
(3.62)
+[1 − 2(1 − θ)µ]vjn−1 .

Under the hypotheses of the theorem all coefficients in the right side of the
above equation are nonnegative and sum to (1 + 2θµ). Hence this leads to
the conclusion that the interior point (xj , tn ) can have a local maximum or
minimum only if all five neighboring points, related to the right side of (3.62),
have the same maximum or minimum value. The argument then implies that
v has the same value at all grid points including those on the boundary. This
completes the proof.
3.8. The θ-Method 89

3.8.4. Error analysis

Let
enj = unj − vjn ,
where unj = u(xj , tn ) with u being the exact solution of (3.1). Define
n−1/2
E n = max |enj |, T n−1/2 = max |T uj |,
j j

n−1/2
where T uj is the truncation error at (xj , tn−1/2 ) defined in (3.59).

1
Theorem 3.10. Let θ ∈ [0, 1] and (1 − θ)µ ≤ 2 for the θ-method. Then,
n
X
n
E ≤ ∆t T k−1/2 . (3.63)
k=1

It follows from (3.63) that

E n ≤ n∆t max T k−1/2 ≤ T max T k−1/2 , (3.64)

k k

where T is the upper limit of the time variable.

90 Chapter 3. Properties of Numerical Methods

3.9. Homework
1. The energy method can be utilized to prove stability of the forward Euler
scheme for ut − uxx = 0:
vjn = µ vj−1
n−1
+ (1 − 2µ) vjn−1 + µ vj+1
n−1
(3.65)
The analysis requires you to prove
kv n k2∆x ≤ (1 + C∆t)2 kv n−1 k2∆x , (3.66)
for some C ≥ 0. Prove it, assuming 1 − 2µ ≥ 0 and using the following
hint
• Start with squaring (3.65).
a2 + b 2
• Apply the inequality |ab| ≤ .
2
• Use the observation
X X X
n−1 2 n−1 2 n−1 2
|vj−1 | = |vj | = |vj+1 |
j j j

2. Verify (3.37) and (3.38).

3. Use the arguments in the proof of Example 3.5 on page 64 to prove Theo-
rem 3.10.
4. This problem shows a different way of maximum principle for FD meth-
ods. Prove that the solution of the forward Euler method (3.5) satisfies

min vjn−1 ≤ vjn ≤ max vjn−1 (3.67)

j j
when f ≡ 0 and µ ≤ 1/2.
5. Consider the problem in (3.20):
ut − uxx = 0, (x, t) ∈ [0, 1] × [0, 1],
u = 0, (x, t) ∈ {0, 1} × [0, 1], (3.68)
u = sin(πx), x ∈ [0, 1], t = 0
(a) Implement a code for the θ-method.
(b) Compare its performances for θ = 0, 1, 1/2.
Choose ∆x = 1/10, 1/20; set either ∆t = ∆x or ∆t to satisfy the stabil-
ity limit.
Chapter 4

Finite Difference Methods for Elliptic

Equations

This chapter introduces finite difference methods for elliptic PDEs defined on
1-dimensional (1D), 2-dimensional (2D), or 3-dimensional (3D) regions.

91
92 Chapter 4. Finite Difference Methods for Elliptic Equations

4.1. Finite Difference (FD) Methods

Let Ω = (ax , bx ) × (ay , by ) in 2D space. Consider the model problem

(a) −∇ · (a∇u) + cu = f, x ∈ Ω
(4.1)
(b) auν + βu = g, x ∈ Γ,

where the diffusivity a(x) > 0 and the coefficient c(x) ≥ 0.

• When c ≡ 0 and β ≡ 0, the problem (4.1) has infinitely many solutions.

– If u(x) is a solution, so is u(x) + C, for ∀ C ∈ R.

– Also we can see that the corresponding algebraic system is singular.
– The singularity is not a big issue in numerical simulation; one may
impose a Dirichlet condition at a grid point on the boundary.
• We may assume that (4.1) admits a unique solution.

To explain the main feature of the central FD method, we may start with
the problem (4.1) with the constant diffusivity, i.e., a = 1.
4.1. Finite Difference (FD) Methods 93

4.1.1. Constant-coefficient problems

Consider the following simplified problem (a ≡ 1):
−uxx − uyy + cu = f (x, y), (x, y) ∈ Ω,
(4.2)
uν + βu = g(x, y), (x, y) ∈ Γ,

Furthermore, we may start with the 1D problem:

(a) −uxx + cu = f, x ∈ (ax , bx ),
(b) −ux + βu = g, x = ax , (4.3)
(c) ux + βu = g, x = bx .

Select nx equally spaced grid points on the interval [ax , bx ]:

b x − ax
xi = ax + ihx , i = 0, 1, · · · , nx , hx = .
nx
Let ui = u(xi ) and recall (1.16) on page 10:
−ui−1 + 2ui − ui+1 uxxxx (xi ) 2
−uxx (xi ) ≈ + hx + · · · . (4.4)
h2x 12
94 Chapter 4. Finite Difference Methods for Elliptic Equations

Apply the FD scheme for (4.3.a) to have

−ui−1 + (2 + h2x c)ui − ui+1 = h2x fi . (4.5)

However, we will meet ghost grid values at the end points. For example, at
the point ax = x0 , the formula becomes

−u−1 + (2 + h2x c)u0 − u1 = h2x f0 . (4.6)

Here the value u−1 is not defined and we call it a ghost grid value.
Now, let’s replace the value by using the boundary condition (4.3.b). Recall
the central FD scheme (1.15) for ux at x0 :
u1 − u−1 uxxx (x0 ) 2
ux (x0 ) ≈ , Trunc.Err = − hx + · · · . (4.7)
2hx 6
Thus he equation (4.3.b) can be approximated (at x0 )

u−1 + 2hx βu0 − u1 = 2hx g0 . (4.8)

Hence it follows from (4.6) and (4.8) that

(2 + h2x c + 2hx β)u0 − 2u1 = h2x f0 + 2hx g0 . (4.9)

The same can be considered for the algebraic equation at the point xn .
4.1. Finite Difference (FD) Methods 95

The problem (4.3) is reduced to finding the solution u1 satisfying

A1 u1 = b1 , (4.10)

where
2 + h2x c + 2hx β

−2

 −1 2 + h2x c −1 
... ... ...
 
A1 =  ,
 
 
 −1 2 + h2x c −1 
2
−2 2 + hx c + 2hx β

and    
h2x f0 2hx g0
 h2x f1   0 
.. ..
   
b1 =  . + . .
   
 2   
 hx fnx −1   0 
h2x fnx 2hx gnx
Such a technique of removing ghost grid values is called outer bordering.
We can use it for the 2D problem (4.2) along the boundary grid points.

Symmetrization: The matrix A1 is not symmetric! You can symmetrize it

by dividing the first and the last rows of [A1 |b1 ] by 2. For the 2D problem, you
have to apply “division by 2" along each side of boundaries. (So, the algebraic
equations corresponding to the corner points would be divided by a total factor
of 4, for a symmetric algebraic system.)
96 Chapter 4. Finite Difference Methods for Elliptic Equations

4.1.2. General diffusion coefficients

Let the 1D problem read
(a) −(aux )x + cu = f, x ∈ (ax , bx ),
(b) −aux + βu = g, x = ax , (4.11)
(c) aux + βu = g, x = bx .

The central FD scheme for (aux )x can be obtained as follows.

• The term (aux ) can be viewed as a function and approximated as

(aux )i+1/2 − (aux )i−1/2
(aux )x (xi ) ≈ + O(h2x ), (4.12)
hx
where (aux )i+1/2 denotes the value of (aux ) evaluated at xi+1/2 := (xi +
xi+1 )/2.
• The terms (aux )i+1/2 and (aux )i−1/2 can be again approximated as

ui+1 − ui uxxx (xi+1/2 )  hx 2

(aux )i+1/2 ≈ ai+1/2 − ai+1/2 + ··· ,
hx 3! 2
(4.13)
ui − ui−1 uxxx (xi−1/2 )  hx 2
(aux )i−1/2 ≈ ai−1/2 − ai−1/2 + ··· .
hx 3! 2

• Combine the above two equations to have

−ai−1/2 ui−1 + (ai−1/2 + ai+1/2 )ui − ai+1/2 ui+1
−(aux )x (xi ) ≈ , (4.14)
h2x
of which the overall truncation error becomes O(h2x ). See Homework 4.1
on page 150.
4.1. Finite Difference (FD) Methods 97

Notes
• The y-directional approximation can be done in the same fashion.
• The reader should also notice that the quantities ai+1/2 evaluated at mid-
points are not available in general.
• We may replace it by the arithmetic/harmonic average of ai and ai+1 :
  −1
ai + ai+1 1 1 1
ai+1/2 ≈ or + . (4.15)
2 2 ai ai+1

• The harmonic average is preferred; the resulting system holds the con-
servation property. See §5.7.
98 Chapter 4. Finite Difference Methods for Elliptic Equations

4.1.3. FD schemes for mixed derivatives

The linear elliptic equation in its general form is given as

−∇ · (A(x)∇u) + b · ∇u + cu = f, x ∈ Ω ⊂ Rd , (4.16)

where 1 ≤ d ≤ 3 and
X ∂  ∂u

−∇ · (A(x)∇u) = − aij (x) .
i,j
∂x i ∂x j

Thus we must approximate the mixed derives whenever they appear.

As an example, we consider a second-order FD scheme for (aux )y on a mesh
of grid size hx × hy :
aux (xp,q+1 ) − aux (xp,q−1 )
(aux )y (xpq ) ≈ + O(h2y )
2hy
ap,q+1 (up+1,q+1 − up−1,q+1 ) − ap,q−1 (up+1,q−1 − up−1,q−1 ) (4.17)
≈
4hx hy
2 2
+O(hx ) + O(hy ).

• There may involve difficulties in FD approximations when the diffusion

coefficient A is a full tensor.
• Scalar coefficients can also become a full tensor when coordinates are
changed.

4.1.4. L∞-norm error estimates for FD schemes

Let Ω be a rectangular domain in 2D and Γ = ∂Ω. Consider

−∆u = f, x ∈ Ω,
(4.18)
u = g, x ∈ Γ,

where x = (x, y) = (x1 , x2 ) and

∂2 ∂2 ∂2 ∂2
∆=∇·∇= + = + .
∂x2 ∂y 2 ∂x21 ∂x22
4.1. Finite Difference (FD) Methods 99

Let ∆h be the discrete five-point Laplacian:

∆h upq = (δx2 + δy2 )upq
up−1,q − 2upq + up+1,q up,q−1 − 2upq + up,q+1 (4.19)
:= + .
h2x h2y
100 Chapter 4. Finite Difference Methods for Elliptic Equations

Consistency: Let uh be the FD solution of (4.18), i.e.,

−∆h uh = f, x ∈ Ωh ,
(4.20)
uh = g, x ∈ Γh ,

where Ωh and Γh are the sets of grid points on Ω◦ and Γ, respectively. Note
that the exact solution u of (4.18) satisfies

−∆h u = f + O(h2 ∂ 4 u), x ∈ Ωh . (4.21)

Thus it follows from (4.20) and (4.21) that for some C > 0 independent of h,

k∆h (u − uh )k∞,Ωh ≤ Ch2 k∂ 4 uk∞,Ωh , (4.22)

where k · k∞,Ωh denotes the maximum norm measured on the grid points Ωh .
4.1. Finite Difference (FD) Methods 101

Convergence: We are more interested in an error estimate for (u − uh)

rather than for ∆h (u − uh ). We begin with the following lemma.
Lemma 4.1. Let Ω is a rectangular domain and vh be a discrete function
defined on a grid Ωh of Ω with vh = 0 on the boundary Γh . Then

kvh k∞,Ωh ≤ Ck∆h vh k∞,Ωh , (4.23)

for some C > 0 independent on h.

Proof. Let the function fh be defined as

fh := −∆h vh , x ∈ Ωh .

Then obviously
(a) kfh k∞,Ωh = k∆h vh k∞,Ωh ,
(4.24)
(b) −kfh k∞,Ωh ≤ −∆h vh ≤ kfh k∞,Ωh .
Let x
b = (x̂, ŷ) be the centroid of Ω and consider
1 1
b|2 = (x − x̂)2 + (y − ŷ)2 ,


wh (x) = |x − x x ∈ Ωh .
4 4
Then wh has its maximum on the boundary, bounded by a constant C > 0
independent on h, and
−∆h wh = −1, x ∈ Ωh .
So from (4.24.b) we have

−∆h (vh + kfh k∞,Ωh wh ) = −∆h vh − kfh k∞,Ωh ≤ 0

and therefore from the discrete maximum principle for subharmonic func-
tions, Theorem B.7 on page 363,

vh + kfh k∞,Ωh wh ≤ kfh k∞,Ωh kwh k∞,Γh ≤ C kfh k∞,Ωh .

Since wh ≥ 0,
vh ≤ C kfh k∞,Ωh . (4.25)
The argument in the proof can be applied for the same conclusion, when vh is
replaced by −vh . Thus, (4.23) follows from (4.24.a) and (4.25).
102 Chapter 4. Finite Difference Methods for Elliptic Equations

Clearly, (u − uh ) in (4.22) can be considered as a discrete function on the

unit square with u − uh = 0 on Γh . Therefore, with a aid of Lemma 4.1, one
can conclude
Theorem 4.2. Let u and uh be the solutions of (4.18) and (4.20), respectively.
Then
ku − uh k∞,Ωh ≤ Ch2 k∂ 4 uk∞,Ωh , (4.26)
for some C > 0 independent on the grid size h.
4.1. Finite Difference (FD) Methods 103

Generalization: The above theorem can be expanded for more general

elliptic problems of the form
Lu := −∇ · (A(x)∇u) + b(x) · ∇u = f, x ∈ Ω,
(4.27)
u = g, x ∈ Γ,

where A(x) = diag(a11 (x), a22 (x)).

Let Lh be the five-point central discretization of L and uh be the solution of

Lh uh = f, x ∈ Ωh ,
(4.28)
uh = g, x ∈ Γh .

Theorem 4.3. Let u and uh be the solutions of (4.27) and (4.28), respectively.
Assume h is sufficiently small for the case b 6= 0. Then

ku − uh k∞,Ωh ≤ Ch2 , (4.29)

for some C = C(Ω, ∂ 3 u, ∂ 4 u) > 0 independent on the grid size h.

104 Chapter 4. Finite Difference Methods for Elliptic Equations

Proof. Note that

Lh u = f + O(h2 ),
x ∈ Ωh .
Lh uh = f,
Thus, we have
kLh (u − uh )k∞,Ωh ≤ Ch2 , (4.30)
for some C > 0 independent on h. Now, follow the same arguments utilized in
Lemma 4.1, with Theorem B.7 replaced by Theorem B.8, to get

kvh k∞,Ωh ≤ CkLh vh k∞,Ωh , (4.31)

for discrete functions vh such that vh = 0 on Γh . The inequality (4.29) follows

from (4.30) and (4.31) with vh = u − uh .
4.1. Finite Difference (FD) Methods 105

4.1.5. The Algebraic System for FDM

Let Ω = [ax , bx ] × [ay , by ] and Γ = ∂Ω. Consider (4.18):

−∆u = f, x ∈ Ω,
(4.32)
u = g, x ∈ Γ.

Define, for some positive integers nx , ny ,

b x − ax by − ay
hx = , hy =
nx ny
and
xp = ax + p hx , p = 0, 1, · · · , nx
yq = ay + q hy , q = 0, 1, · · · , ny
Let ∆h be the discrete five-point Laplacian (4.19):
∆h upq = (δx2 + δy2 )upq
up−1,q − 2upq + up+1,q up,q−1 − 2upq + up,q+1 (4.33)
:= + .
h2x h2y
106 Chapter 4. Finite Difference Methods for Elliptic Equations

Then, when the grid points are ordered row-wise, the algebraic system for
the FDM reads
Au = b, (4.34)
where  
B −I/h2y 0
 −I/h2 B −I/hy2 
 y 
A=
 . .. ... ... 
 (4.35)
−I/h2y −I/h2y
 
 B 
0 −I/h2y B
with I being the identity matrix of dimension nx − 1 and B being a matrix of
order nx − 1 given by
 
d −1/h2x 0
 −1/h2 d −1/h2x 
 x 
B=
 . . . . . . . . .

 (4.36)
2 2
−1/hx −1/hx 
 
 d
0 −1/h2x d
2 2
where d = + .
h2x h2y
On the other hand,
gp−1,q gp+1,q
bpq = fpq + δp−1,0 + δp+1,nx
h2x h2x
gp,q−1 gp,q+1 (4.37)
+ 2 δq−1,0 + δq+1,ny
hy h2y

Here, the global point index for the row-wise ordering of the interior points,
i = 0, 1, 2, · · · , becomes

i = (q − 1) ∗ (nx − 1) + p − 1 (4.38)
4.1. Finite Difference (FD) Methods 107

Saving and managing the algebraic system

• For the FDM we just considered, the total number of interior nodal points
is
(nx − 1) ∗ (ny − 1)
Thus, you may try to open the matrix and other arrays based on this
number.
• Saving nonzero entries only, the matrix A can be stored in an array of the
form
A[M ][5] or A[ny − 1][nx − 1][5], (4.39)
where M = (nx − 1) ∗ (ny − 1).
• However, it is often more convenient when the memory objects are opened
incorporating all the nodal points (including those on boundaries). You
may open the matrix as

A[ny + 1][nx + 1][5]. (4.40)

108 Chapter 4. Finite Difference Methods for Elliptic Equations

• The matrix A in (4.35) can be saved, in Python, as

rx, ry = 1/hx**2, 1/hy**2

d = 2*(rx+ry)
for q in range(1,ny):
for p in range(1,nx):
A[q][p][0] = -ry
A[q][p][1] = -rx
A[q][p][2] = d
A[q][p][3] = -rx
A[q][p][4] = -ry

• Let the solution vector u be opened in u[ny+1][nx+1] and initialized

along the boundaries. Then, the Gauss-Seidel iteration can be carried
out as

import numpy as np; import copy

from numpy import abs,sqrt,pi,sin,cos

# the Jacobi matrix

T = copy.deepcopy(A) # np.ndarray((ny+1,nx+1,5),float)
for q in range(1,ny):
for p in range(1,nx):
for c in [0,1,3,4]:
T[q][p][c] = -T[q][p][c]/T[q][p][2]

# A function for the Gauss-Seidel iteration

def Gauss_Seidel(T,u,itmax=1):
ny,nx = leng(u)-1, len(u[0])-1
for it in range(0,itmax):
for q in range(1,ny):
for p in range(1,nx):
u[q][p] = T[q][p][0]*u[q-1][p] \
+T[q][p][1]*u[q][p-1] \
+T[q][p][3]*u[q][p+1] \
+T[q][p][4]*u[q+1][p]
4.2. Solution of Linear Algebraic Systems 109

4.2. Solution of Linear Algebraic Systems

In this section, we consider solution methods for the following linear system

Ax = b, (4.41)

where A ∈ Cn×n and b ∈ Cn . In most applications of PDEs, the matrix A

is real-valued and sparse. By being sparse we mean that a large portion of
entries in A is zero. For example, the maximum number of nonzero entries in
a row is five for the central FD application to the Poisson equation in 2D.
110 Chapter 4. Finite Difference Methods for Elliptic Equations

4.2.1. Direct method: the LU factorization

Let the matrix
A = [aij ]
be factorized into LU , where

L = [lij ], U = [uij ]

are respectively lower and upper triangular matrices with lii = 1.

Then (4.41) reads

Ax = LU x = b, (4.42)
which can be solved by
Ly = b,
(4.43)
U x = y,
by the forward elimination and backward substitution.
4.2. Solution of Linear Algebraic Systems 111

The LU factorization can be carried out by the Gauss elimination proce-

(1)
dure. Define A(1) = [aij ] = [aij ] and
 
(k) (k) (k)
a a · · · · · · · · · a1n
 11 12 (k) (k) 
 a 22 · · · · · · · · · a 2n 


... · · · · · · .. 

 . 
(k) (k)
A(k) =  akk · · · akn  . (4.44)
 
 (k) (k) 
 0 ak+1,k · · · ak+1,n 
.. . .. 
 

 . . . . 
(k) (k)
ank ··· ann

Using the Gauss elimination procedure, A(k+1) and the entries of L can be
determined as
( (k)  (k)  (k)  (k)
(k+1) aij − aik akk akj , for i = k + 1, · · · , n, j = k, · · · , n,
aij = (k)
aij , else, (4.45)
lkk = 1,
(k)  (k)
lik = aik akk , i = k + 1, · · · , n.

Then, finally
(n)
U = A(n) = [aij ]. (4.46)
112 Chapter 4. Finite Difference Methods for Elliptic Equations

The above procedure can be summarized into the following pseudocode:

For k = 1 to n − 1
For i = k + 1 to n



 mi ← aik /akk ;
  if mi = 0, continue ; (4.47)
 
  aik ← mi ;
 
  For j = k + 1 to n

aij ← aij − mi akj ;

In the output of the algorithm, the upper part including the main diagonal
becomes U , while its strictly lower part is the corresponding part of L.
Algorithm (4.47) should be modified to incorporate the so-called partial piv-
oting when a pivot akk is expected to be zero or small in modulus.
4.2. Solution of Linear Algebraic Systems 113

The LU factorization with partial pivoting must look like the following:
Fork = 1 to n − 1
amax ← 0 ; imax ← 0 ; /*find pivot*/
 For i = k to n
 
 ik | > amax )
if (|a



 amax ← |aik | ; imax ← i ;
 if (i
 max = 0) stop ; /*A is singular*/
 if (imax 6= k)

for j = 1 to n


 /*row interchange*/



 tmp ← akj ;
   akj ← aimax ,j ;
(4.48)
 
aimax ,j ← tmp ;
 
 
 itmp ← intch[k] ;


 /*save interchange*/
 intch[k] ← intch[imax ] ;
 


 intch[imax ] ← itmp ;
 For i = k + 1 to n /*row operations*/


mi ← aik /akk ;



  if mi = 0, continue ;
 
  aik ← mi ;
 
  For j = k + 1 to n

aij ← aij − mi akj ;

In the above algorithm, the array “intch" must be initialized in advance

intch[i]=i. You can use the array resulting from (4.48) to reorder the en-
tries of the right-hand side b. That is,

b[i] ← b[intch[i]], i = 1, · · · , n
114 Chapter 4. Finite Difference Methods for Elliptic Equations

Banded matrices: For a square matrix A = [aij ], if

aij = 0 for |i − j| > d, ∀ i, j,

the matrix is called to be banded with the bandwidth d.

• In most applications with the numerical solution of PDEs, the algebraic

system is banded.
• For banded matrices, the LU factorization algorithms presented in (4.47)
and (4.48) can be easily modified. For example, for the algorithm (4.47),
simply replace the integers n appeared as the last indices of the i- and
j-loops by min(n, k + d).
4.2. Solution of Linear Algebraic Systems 115

4.2.2. Linear iterative methods

Basic concepts: For solving linear algebraic systems, linear iterative

methods begin with splitting the matrix A by

A = M − N, (4.49)

for some invertible matrix M .

Then, the linear system equivalently reads

M x = N x + b. (4.50)

Associated with the splitting is an iterative method

M xk = N xk−1 + b, (4.51)

or, equivalently,

xk = M −1 (N xk−1 + b) = xk−1 + M −1 (b − Axk−1 ), (4.52)

for an initial value x0 .

Notes:

• Methods differ for different choices of M .

• M must be easy to invert (efficiency) and
M −1 ≈ A−1 (convergence).
116 Chapter 4. Finite Difference Methods for Elliptic Equations

4.2.3. Convergence theory

Let
ek = x − x k ;
from (4.50) and (4.51), we obtain the error equation

M ek = N ek−1

or, equivalently,
ek = M −1 N ek−1 . (4.53)
Since
kek k ≤ kM −1 N k · kek−1 k
≤ kM −1 N k2 · kek−2 k
.. (4.54)
.
≤ kM −1 N kk · ke0 k,
a sufficient condition for the convergence is

kM −1 N k < 1. (4.55)

Let σ(B) be the spectrum, the set of eigenvalues of the matrix B, and ρ(B)
denote the spectral radius defined by

ρ(B) = max |λi |.

λi ∈σ(B)

Theorem 4.4. The iteration converges if and only if

ρ(M −1 N ) < 1. (4.56)

4.2. Solution of Linear Algebraic Systems 117

Graph theory for the estimation of the spectral

radius
Definition 4.5. A permutation matrix is a square matrix in which each
row and each column has one entry of unity, all others zero.

Definition 4.6. For n ≥ 2, an n × n complex-valued matrix A is reducible if

there is a permutation matrix P such that
 
A 11 A 12
P AP T = ,
0 A22

where A11 and A22 are respectively r × r and (n − r) × (n − r) submatrices,

0 < r < n. If no such permutation matrix exists, then A is irreducible.

The geometrical interpretation of the concept of the irreducibility by means

of graph theory is useful.
118 Chapter 4. Finite Difference Methods for Elliptic Equations

Geometrical interpretation of irreducibility

Figure 4.1: The directed paths for nonzero aii and aij .

Figure 4.2: The directed graph G(A) for A in (4.57).

• Given A = (aij ) ∈ Cn×n , consider n distinct points

P1 , P2 , · · · , Pn

in the plane, which we will call nodes or nodal points.

−→
• For any nonzero entry aij of A, we connect Pi to Pj by a path Pi Pj , di-
rected from the node Pi to the node Pj ; a nonzero aii is joined to itself by a
directed loop, as shown in Figure 4.1.
• In this way, every n × n matrix A can be associated a directed graph G(A).
For example, the matrix
 
2 −1 0
A =  −1 2 −1  (4.57)
0 −1 2

has a directed graph shown in Figure 4.2.

4.2. Solution of Linear Algebraic Systems 119

Definition 4.7. A directed graph is strongly connected if, for any ordered
pair of nodes (Pi , Pj ), there is a directed path of a finite length
−→ −→ −→
Pi Pk1 , Pk1 Pk2 , · · · , Pkr−1 Pkr =j ,

connecting from Pi to Pj .
The theorems to be presented in this subsection can be found in [68] along
with their proofs.

Theorem 4.8. An n × n complex-valued matrix A is irreducible if and only if

its directed graph G(A) is strongly connected.

It is obvious that the matrices obtained from FD/FE methods of the Poisson
equation are strongly connected. Therefore the matrices are irreducible.
120 Chapter 4. Finite Difference Methods for Elliptic Equations

Eigenvalue locus theorem

For A = [aij ] ∈ Cn×n , let
n
X
Λi := |aij |
j=1
j 6= i

Theorem 4.9. (Eigenvalue locus theorem) Let A = [aij ] be an irreducible

n × n complex matrix. Then,

1. (Gerschgorin [25]) All eigenvalues of A lie in the union of the disks in

the complex plane
|z − aii | ≤ Λi , 1 ≤ i ≤ n. (4.58)

2. (Taussky [65]) In addition, assume that λ, an eigenvalue of A, is a

boundary point of the union of the disks |z − aii | ≤ Λi . Then, all the n
circles |z − aii | = Λi must pass through the point λ, i.e., |λ − aii | = Λi for
all 1 ≤ i ≤ n.

For example, for  

2 −1 0
A =  −1 2 −1 
0 −1 2
Λ1 = 1, Λ2 = 2, and Λ3 = 1. Since aii = 2, for i = 1, 2, 3,

|λ − 2| < 2

for all eigenvalues λ of A.

4.2. Solution of Linear Algebraic Systems 121

Positiveness
Definition 4.10. An n × n complex-valued matrix A = [aij ] is diagonally
dominant if n
X
|aii | ≥ Λi := |aij |, (4.59)
j=1
j 6= i
for all 1 ≤ i ≤ n. An n × n matrix A is irreducibly diagonally dominant if A is
irreducible and diagonally dominant, with strict inequality holding in (4.59)
for at least one i.

Theorem 4.11. Let A be an n × n strictly or irreducibly diagonally dominant

complex-valued matrix. Then, A is nonsingular. If all the diagonal entries of
A are in addition positive real, then the real parts of all eigenvalues of A are
positive.

Corollary 4.12. A Hermitian matrix satisfying the conditions in Theorem 4.11

is positive definite.
Corollary 4.13. The FD/FE matrices from diffusion equations (including the
Poisson equation) are positive definite, when it is symmetric.
122 Chapter 4. Finite Difference Methods for Elliptic Equations

Regular splitting and M-matrices

Definition 4.14. For n × n real matrices, A, M , and N , A = M − N is a
regular splitting of A if M is nonsingular with M −1 ≥ 0, and N ≥ 0.
Theorem 4.15. If A = M − N is a regular splitting of A and A−1 ≥ 0, then

−1 ρ(A−1 N )
ρ(M N) = < 1. (4.60)
1 + ρ(A−1 N )

Thus, the matrix M −1 N is convergent and the iterative method of (4.51) con-
verges for any initial value x0 .
Definition 4.16. An n × n real matrix A = [aij ] with aij ≤ 0 for all i 6= j is an
M-matrix if A is nonsingular and A−1 ≥ 0.
Theorem 4.17. Let A = (aij ) be an n × n M -matrix. If M is any n × n matrix
obtained by setting certain off-diagonal entries of A to zero, then A = M − N
is a regular splitting of A and ρ(M −1 N ) < 1.
Theorem 4.18. Let A be an n×n real matrix with A−1 > 0, and A = M1 −N1 =
M2 − N2 be two regular splittings of A. If N2 ≥ N1 ≥ 0, where neither N2 − N1
nor N1 is null, then

1 > ρ(M2−1 N2 ) > ρ(M1−1 N1 ) > 0. (4.61)

4.2.4. Relaxation methods

We first express A = (aij ) as the matrix sum

A = D − E − F, (4.62)

where
· · · , ann ),
D = diag(a11 , a22 ,
−aij , if i > j,
E = (eij ), eij =
 0, else,
−aij , if i < j,
F = (fij ), fij =
0, else.
4.2. Solution of Linear Algebraic Systems 123

Then, a relaxation method can be formulated by selecting M and N for a

regular splitting:
A=M −N (4.63)

Popular examples are

Table 4.1: Relaxation methods

Methods M N
Jacobi method D E+F
Gauss-Seidel method D−E F
1 1−ω
SOR method D−E D+F
ω ω
Richardson method I I −A

SOR stands for Successive Over Relaxation.

124 Chapter 4. Finite Difference Methods for Elliptic Equations

Jacobi method
It is formulated as
Dxk = (E + F )xk−1 + b, (4.64)
which is the same as choosing

M = D, N =E+F

The i-th component of (4.64) reads

i−1
X n
X
aii xki =− aij xk−1
j − aij xk−1
j + bi
j=1 j=i+1

or, equivalently,
 i−1
X n
X .
xki = bi − aij xjk−1 − aij xk−1
j aii , (4.65)
j=1 j=i+1

for i = 1, · · · , n.
4.2. Solution of Linear Algebraic Systems 125

Gauss-Seidel method
For the choice
M = D − E, N = F,
we obtain the Gauss-Seidel method:

(D − E)xk = F xk−1 + b. (4.66)

Its i-th component reads

i
X n
X
aij xkj = −aij xk−1
j + bi ,
j=1 j=i+1

which is equivalent to
 i−1
X n
X .
xki = bi − k
aij xj − k−1
aij xj aii , i = 1, · · · , n. (4.67)
j=1 j=i+1

Note:

• The difference of the Gauss-Seidel method (4.67) out of the Jacobi method
(4.65) is to utilize the updated values xkj , j = 1, · · · , i − 1.s
• It makes the method converge or diverge twice faster asymptotically.
126 Chapter 4. Finite Difference Methods for Elliptic Equations

Successive over-relaxation (SOR) method

Now, we consider the third basic linear iterative method for solving Ax = b.
Choose
1 1−ω
M = D − E, N = D + F, ω ∈ (0, 2),
ω ω
where ω is called the relaxation parameter which is often set larger than one.
With the splitting, the SOR method can be formulated as

(D − ωE)xk = (1 − ω)D + ωF xk−1 + ωb.

 
(4.68)

Since the above equation equivalently reads

Dxk = (1 − ω)Dxk−1 + ω b + Exk + F xk−1 ,

the i-th component of SOR becomes

 i−1
X n
X .
xkGS,i = bi − aij xkj − aij xk−1
j aii ,
j=1 j=i+1
(4.69)
xki = (1 − ω) xk−1
i + ω xkGS,i .

for i = 1, · · · , n. Note that SOR turns out to be the Gauss-Seidel method when
ω = 1.
4.2. Solution of Linear Algebraic Systems 127

Convergence of relaxation methods

Let B, L1 , and Lω be respectively the iteration matrices of the Jacobi, Gauss-
Seidel, and SOR methods. That is,
B = D−1 (E + F ), L1 = (D − E)−1 F,
Lω = (D − ωE)−1 (1 − ω)D + ωF .
 

Theorem 4.19. (Stein and Rosenberg [62]) On and only one of the follow-
ing mutually exclusive relations is valid:
1. ρ(B) = ρ(L1 ) = 0,
2. 0 < ρ(L1 ) < ρ(B) < 1,
(4.70)
3. ρ(B) = ρ(L1 ) = 1,
4. 1 < ρ(B) < ρ(L1 ).

Thus the Jacobi and Gauss-Seidel methods are either both convergent or both
divergent.
Theorem 4.20. (Ostrowski [55]) Let A = D − E − E ∗ be an n × n Hermitian
matrix, where D is Hermitian and positive definite and D − ωE is nonsingular
for 0 ≤ ω ≤ 2. Then,

ρ(Lω ) < 1 ⇐⇒ A is positive definite & 0 < ω < 2. (4.71)

Note that the matrices D and E in Ostrowski’s theorem need not to be diago-
nal and strictly lower triangular matrices.
128 Chapter 4. Finite Difference Methods for Elliptic Equations

Optimal parameter for SOR: For algebraic systems of good proper-

ties, it is theoretically known that the convergence of SOR can be optimized
when
2
ω= p , (4.72)
1 + 1 − ρ(B)
where B is the Jacobi iteration matrix.

However, in most cases you can find a better parameter for a given algebraic
system.
4.2. Solution of Linear Algebraic Systems 129

4.2.5. Line relaxation methods

• The standard Jacobi, Gauss-Seidel, and SOR schemes are called point
relaxation methods.
• We can compute a whole line of new values using a direct method, e.g.,
Gauss elimination.
• this leads to line relaxation methods.

Algebraic interpretation: As in §4.1.5, consider

−∆u = f, x ∈ Ω,
(4.73)
u = g, x ∈ Γ,

where Ω is a rectangular domain in R2 , and its discrete five-point Laplacian

∆h upq = (δx2 + δy2 )upq
up−1,q − 2upq + up+1,q up,q−1 − 2upq + up,q+1 (4.74)
:= + .
h2x h2y
130 Chapter 4. Finite Difference Methods for Elliptic Equations

Then, for the column-wise point ordering, the algebraic system for the FDM
reads
Au = b, (4.75)
where  
C −I/h2x 0
 −I/h2 C −I/h2x 
 x 
A=
 ... ... ... 
 (4.76)
−I/h2x −I/h2x
 
 C 
0 −I/h2x C
with I being the identity matrix of dimension ny − 1 and C being a matrix of
order nx − 1 given by
 
d −1/h2y 0
 −1/h2 d −1/h2y 
 y 
C=
 . . . . . . . . .

 (4.77)
2 2
−1/hy −1/hy 
 
 d
0 −1/h2y d
2 2
where d = + .
h2x h2y

• A line relaxation method can be viewed as a (standard) relaxation method

which deals with the matrix C like a single entry of a tridiagonal matrix.
• Once a point relaxation method converges, its line method converges
twice faster asymptotically.
• Line methods can employ the line solver in alternating directions of (x, y).
4.2. Solution of Linear Algebraic Systems 131

Convergence comparison: For (4.73) on p.129, we choose

Ω = (0, 1)2 , n = nx = ny .

The following table includes the spectral radii of iteration matrices ρ(T ) and
the required iteration counts k for the convergence to satisfy the tolerance
kek k ke0 k < 10−6 .


Table 4.2: Convergence comparison

Point Jacobi Line Jacobi Point GS Line GS
n ρ(T ) k ρ(T ) k ρ(T ) k ρ(T ) k
5 0.8090 66 0.6793 36 0.6545 33 0.4614 18
10 0.9511 276 0.9067 142 0.9045 138 0.8221 71
20 0.9877 1116 0.9757 562 0.9755 558 0.9519 281
40 0.9969 4475 0.9939 2241 0.9938 2238 0.9877 1121

Final remarks for relaxation methods

• GS methods converge asymptotically twice faster than Jacobi methods, in

either point or line iterations. SOR is yet faster and the line SOR is again
twice faster.
• Relaxation methods sweep over either points or groups of points. For a
faster convergence, you may let them visit the points in an order followed
by the opposite order.
• For line methods, the tridiagonal matrix can be stored in a 3-column ar-
ray, instead of a square big-fat array.
132 Chapter 4. Finite Difference Methods for Elliptic Equations

4.3. Krylov Subspace Methods

We consider Krylov subspace methods for solving

Ax = b, (4.78)

when A is symmetric positive definite.

Given an initial guess x0 ∈ Rn , find successive approximations xk ∈ Rn of
the form
xk+1 = xk + αk pk , k = 0, 1, · · · , (4.79)
where pk is the search direction and αk > 0 is the step length. Different meth-
ods differ in the choice of the search direction and the step length.

In this section, we consider the gradient method (also known as the

steepest descent method, or the Richardson’s method), the conjugate gra-
dient (CG) method, and preconditioned CG method. For other Krylov
subspace methods, see e.g. [3, 33].
Note that (4.78) admits a unique solution x ∈ Rn , which is equivalently
characterized by
1
minn f (η), f (η) = η · Aη − b · η, (4.80)
η ∈R 2
where a · b = aT b.
4.3. Krylov Subspace Methods 133

4.3.1. Steepest descent method

We denote the gradient and Hessian of f by f 0 and f 00 , respectively:

f 0 (η) = Aη − b, f 00 (η) = A.

Given xk+1 as in (4.79), we have by Taylor’s formula

f (xk+1 ) = f (xk + αk pk )
αk2
= f (xk ) + αk f 0 (xk ) · pk + pk · f 00 (ξ)pk ,
2
for some ξ. Since the element of f 00 is bounded (As a matter of fact, we assumed
it!),
f (xk+1 ) = f (xk ) + αk f 0 (xk ) · pk + O(αk2 ), as αk → 0.

The goal: to find pk and αk such that

f (xk+1 ) < f (xk ),

which can be achieved if

f 0 (xk ) · pk < 0 (4.81)
and αk is sufficiently small.
Choice: (4.81) holds if we choose, when f 0 (xk ) 6= 0,

pk = −f 0 (xk ) = b − Axk =: rk (4.82)

134 Chapter 4. Finite Difference Methods for Elliptic Equations

Optimal step length: We may determine αk such that

f (xk + αk pk ) = min f (xk + αpk ),

α

in which case αk is said to be optimal. If αk is optimal, then

d
0 = f (xk + αpk ) = f 0 (xk + αk pk ) · pk
dα α=αk
= (A(xk + αk pk ) − b) · pk
= (Axk − b) · pk + αk pk · Apk .

So,
rk · pk
αk = . (4.83)
pk · Apk
Convergence of the steepest descent method: For the method, the fol-
lowing is known
 k
1
k x − xk k2 ≤ 1 − k x − x0 k2 . (4.84)
κ(A)
Thus, the number of iterations required to reduce the error by a factor of ε is
in the order of the condition number of A:
1
k ≥ κ(A) log . (4.85)
ε

Definition 4.21. The condition number of a matrix A is

κ(A) = kAk · kA−1 k, (4.86)

for a matrix norm.

4.3. Krylov Subspace Methods 135

4.3.2. Conjugate gradient (CG) method

In this method the search directions pk are conjugate, i.e.,
pi · Apj = 0, i 6= j,
and the step length αk is chosen to be optimal.
The following is the original version of the CG method.
CG Algorithm, V.1
Select x0 , ε;
r0 = b − Ax0 , p0 = r0 ;
Do k = 0, 1, · · ·
αk = rk · pk /pk · Apk ; (CG1)
xk+1 = xk + αk pk ; (CG2)
(4.87)
rk+1 = rk − αk Apk ; (CG3)
if k rk+1 k2 < ε k r0 k2 , stop;
βk = −rk+1 · Apk /pk · Apk ; (CG4)
pk+1 = rk+1 + βk pk ; (CG5)
End Do
Remarks:
• αk in (CG1) is designed such that rk+1 · pk = 0. You may easily verify it
using rk+1 in (CG3).
• rk = b − Axk , by definition. So,
rk+1 = b − Axk+1 = b − A(xk + αk pk )
= b − Axk − αk Apk = rk − αk Apk ,
which is (CG3).
• βk in (CG4) is determined such that pk+1 · Apk = 0. Verify it using pk+1 in
(CG5).
• The CG method finds the iterate
xk ∈ x0 + span{r0 , Ar0 , · · · , Ak−1 r0 }
so that (x − xk ) · A(x − xk ) is minimized.
136 Chapter 4. Finite Difference Methods for Elliptic Equations

Theorem 4.22. For m = 0, 1, · · · ,

span{p0 , · · · , pm } = span{r0 , · · · , rm }
(4.88)
= span{r0 , Ar0 , · · · , Am r0 }.

Theorem 4.23. The search directions and the residuals satisfy the orthogo-
nality,
pi · Apj = 0; ri · rj = 0, i 6= j. (4.89)

Theorem 4.24. For some m ≤ n, we have Axm = b and

κ(A) − 1 k
p 
k x − xk kA ≤ 2 p k x − x0 kA . (4.90)
κ(A) + 1
So the required iteration number to reduce the error by a factor of ε is
1p 2
k≥ κ(A) log . (4.91)
2 ε
Proofs of the above theorems can be found in e.g. [32].
4.3. Krylov Subspace Methods 137

Simplification of the CG method: Using the properties and iden-

tities involved in the method, one can derive a more popular form of the CG
method.
CG Algorithm, V.2
Select x0 , ε;
r0 = b − Ax0 , p0 = r0 ;
Compute ρ0 = r0 · r0 ;
Do k = 0, 1, · · ·
αk = ρk /pk · Apk ;
xk+1 = xk + αk pk ;
(4.92)
rk+1 = rk − αk Apk ;
if k rk+1 k2 < ε k r0 k2 , stop;
ρk+1 = rk+1 · rk+1 ;
βk = ρk+1 /ρk ;
pk+1 = rk+1 + βk pk ;
End Do
Note:
rk · pk = rk · (rk + βk−1 pk−1 ) = rk · rk ,
αk
βk = −rk+1 · Apk /pk · Apk = −rk+1 · Apk
ρk
= rk+1 · (rk+1 − rk )/ρk = ρk+1 /ρk .
138 Chapter 4. Finite Difference Methods for Elliptic Equations

4.3.3. Preconditioned CG method

The condition number of A is the critical point for the convergence of the CG
method. If we can find a matrix M such that

M ≈A

and it is easy to invert, we may try to apply the CG algorithm to the following
system
M −1 Ax = M −1 b. (4.93)
Since
κ(M −1 A)  κ(A) (4.94)
(hopefully, κ(M −1 A) ≈ 1), the CG algorithm will converge much faster.
4.3. Krylov Subspace Methods 139

In practice, we do not have to multiply M −1 to the original algebraic system

and the algorithm can be implemented as
Preconditioned CG
Select x0 , ε;
r0 = b − Ax0 , M z0 = r0 ;
p0 = z0 , compute ρ0 = z∗0 r0 ;
Do k = 0, 1, · · ·
αk = ρk /p∗k Apk ;
xk+1 = xk + αk pk ;
rk+1 = rk − αk Apk ; (4.95)
if k rk+1 k2 < ε k r0 k2 , stop;
M zk+1 = rk+1 ;
ρk+1 = z∗k+1 rk+1 ;
βk = ρk+1 /ρk ;
pk+1 = zk+1 + βk pk ;
End Do

Here the superscript * indicates the transpose complex-conjugate; it is the

transpose for real-valued systems.
140 Chapter 4. Finite Difference Methods for Elliptic Equations

4.4. Other Iterative Methods

4.4.1. Incomplete LU-factorization
Here, we introduce Stone’s strongly implicit procedure (SIP) [63] to solve
the following linear system
Ax = b. (4.96)
As for other iterative methods, SIP is based on a regular splitting, A =
M − N, with M being an incomplete LU (ILU) factorization;

M = LI UI = A + N, (4.97)

where LI and UI are respectively the lower and upper triangular components
of the ILU factorization of A, where the entries of the main diagonal of UI are
all one.
The iteration corresponding to the splitting (4.97) is formulated as

LI UI xk = N xk−1 + b, (4.98)

or, since N = LI UI − A,
(a) rk−1 = b − Axk−1 ,
(b) LI UI δ k = rk−1 , (4.99)
(c) xk = xk−1 + δ k .

The iteration (4.98) converges fast, when we choose elements of LI and UI in

a way that N is as small as possible.
4.4. Other Iterative Methods 141

Figure 4.3: Systematic presentation of LI UI = M . The subscripts S, W , E,

N , and C denote respectively south, west, east, north, and center. Note that
diagonals of M marked by subscripts SE and N W are not found in A.

Derivation of SIP: For a 2D problem in a rectangular mesh where the

grid points are ordered in the row-wise manner, the ILU factorization is in
the form as in Figure 4.3 and the row of M corresponding to the (`, m)-th grid
point is given as

MS`,m = L`,m
S ,
`,m
MSE = L`,m
S UE
`,m−1
,
`,m
MW = L`,m
W ,
MC`,m = L`,m
S UN
`,m−1
+ L`,m
W UE
`−1,m
+ L`,m
C , (4.100)
ME`,m = L`,m `,m
C UE ,
MN`,m `,m `−1,m
W = LW UN ,
MN`,m = L`,m `,m
C UN .
142 Chapter 4. Finite Difference Methods for Elliptic Equations

The (`, m)-th component of N x is

`,m
(N x)`,m = NC`,m x`,m + NS`,m x`,m−1 + NW x`−1,m + NE`,m x`+1,m
(4.101)
+NN`,m x`,m+1 + MSE
`,m
x`+1,m−1 + MN`,m
W x`−1,m+1 .

By utilizing the approximations

x`+1,m−1 ≈ α(x`,m−1 + x`+1,m − x`,m ),
0 < α ≤ 1, (4.102)
x`−1,m+1 ≈ α(x`,m+1 + x`−1,m − x`,m ),

we can rewrite (4.101) as

`,m
(N x)`,m ≈ (NC`,m − αMSE − αMN`,m
W )x`,m
+(NS`,m + αMSE
`,m `,m
)x`,m−1 + (NW + αMN`,m
W )x`−1,m (4.103)
+(NE`,m + `,m
αMSE )x`+1,m + (NN`,m + αMN`,m
W )x`,m+1 .

Set each of coefficients in the right-side of (4.103) to be zero. Then, it follows

from (4.100) that entries of N are presented by those of LI and UI :

NS`,m = −αMSE
`,m
= −αL`,m
S UE
`,m−1
,
`,m
NW = −αMN`,m `,m `−1,m
W = −αLW UN ,
NC`,m = α(MSE
`,m
+ MN`,m `,m `,m−1
W ) = α(LS UE + L`,m
W UN
`−1,m
), (4.104)
NE`,m = −αMSE
`,m
= −αL`,m
S UE
`,m−1
,
NN`,m = −αMN`,m `,m `−1,m
W = −αLW UN .
4.4. Other Iterative Methods 143

Now, utilizing M = A + N , (4.100), and (4.104), one can obtain Stone’s SIP
[63]:
L`,m
S = A`,m
S /(1 + αUE
`,m−1
),
L`,m
W = A`,m `−1,m
W /(1 + αUN ),
L`,m
C = A`,m `,m `,m−1
C + α(LS UE + L`,m
W UN
`−1,m
)
(4.105)
−L`,m
S UN
`,m−1
− L`,m
W UE
`−1,m
,
UE`,m = (A`,m `,m `,m−1
E − αLS UE )/L`,m
C ,
UN`,m = (A`,m `,m `−1,m
N − αLW UN )/L`,m
C .

Remark: The approximations in (4.102) are second-order accurate when α =

1. But the algorithm (4.105) can be unstable for the case; the parameter α is
often chosen between 0.92 and 0.96 [23]. Entries of LI and UI used in (4.105)
whose indices are outside the index boundaries should be set zero.
144 Chapter 4. Finite Difference Methods for Elliptic Equations

4.5. Numerical Examples with Python

Here we demonstrate a Python code for solving
−∆u = f, x ∈ Ω = (0, 1)2
(4.106)
u = g, x ∈ ∂Ω

The exact solution is chosen as

u(x, y) = sin(πx) sin(πy) (4.107)

so that the right-hand side becomes

f (x, y) = 2π 2 sin(πx) sin(πy)

With the number of grid points n = nx = ny , the maximum errors are as

follows

Table 4.3: The maximum error ku − uh k∞ .

n 10 20 40 80
ku − uh k∞ 0.00827 0.00206 0.00050 6.42e-05
4.5. Numerical Examples with Python 145

Figure 4.4: Contour plots of computed solution with n = 40 (left) and the
10000-times magnified error (right)

The whole code is attached below.

146 Chapter 4. Finite Difference Methods for Elliptic Equations

#=======================================================
# Elliptic_2D.py
# This module solves, by the 2nd-order FD method & SOR
# -(u_xx+u_yy)=f, (x,y) in (ax,bx)x(ay,by)
# u=g, (x,y) on its boundary
# Supporting functions are built in "util_ellip2D.py"
#=======================================================
from util_ellip2D import *

##----------------------
## User Input
##----------------------
ax,bx = 0., 1.
ay,by = 0., 1.
nx= 40; ny=nx

itmax = 1000
tol = 1.e-6
omega = 1.8

level = 2
##----------------------
## End of "User Input"
##----------------------

print 'Elliptic_2D: (ax,bx)x(ay,by)=(%g,%g)x(%g,%g),\

(nx,ny)=(%d,%d)' % (ax,bx,ay,by, nx,ny)

## build up coefficient matrix & others

A = coeff_matrix(ax,bx,ay,by,nx,ny,level)
b = get_rhs(ax,bx,ay,by,nx,ny,level)
U = get_exact_sol(ax,bx,ay,by,nx,ny,level)
X = init_X(U)

## solve with SOR

sol_SOR(A,X,b,omega,tol,itmax,level)
4.5. Numerical Examples with Python 147

## Checking error
if level:
print " Max-error=%g" % (error8(U,X,level))

## Want to see the figure?

if level>=3:
contourplot(U,ax,bx,ay,by,'Exact Solution',2)
contourplot(X,ax,bx,ay,by,'Computed Solution',2)

##===================================================
## util_ellip2D.py
##===================================================
import numpy as np
from numpy import abs,sqrt,pi,sin,cos
import matplotlib.pyplot as plt
from matplotlib.mlab import griddata
from copy import deepcopy

def coeff_matrix(ax,bx,ay,by,nx,ny,level=0):
matA = np.ndarray((ny+1,nx+1,5),float)
hx,hy= (bx-ax)/nx, (by-ay)/ny
for p in range(0,nx+1):
matA[0][p]=[0,0,1,0,0]; matA[ny][p]=[0,0,1,0,0]
for q in range(0,ny+1):
matA[q][0]=[0,0,1,0,0]; matA[q][nx]=[0,0,1,0,0]
rx,ry = 1./hx**2, 1./hy**2
d = 2*(rx+ry)
for q in range(1,ny):
for p in range(1,nx):
matA[q][p][0] = -ry
matA[q][p][1] = -rx
matA[q][p][2] = d
matA[q][p][3] = -rx
matA[q][p][4] = -ry
return matA
148 Chapter 4. Finite Difference Methods for Elliptic Equations

def get_rhs(ax,bx,ay,by,nx,ny,level=0):
vec_b = np.ndarray((ny+1,nx+1),float)
hx,hy = (bx-ax)/nx, (by-ay)/ny
for q in range(0,ny+1):
y = ay+q*hy
for p in range(0,nx+1):
x = ax+p*hx
vec_b[q][p] = funct_f(x,y)
return vec_b

def get_exact_sol(ax,bx,ay,by,nx,ny,level=0):
vec_u = np.ndarray((ny+1,nx+1),float)
hx,hy = (bx-ax)/nx, (by-ay)/ny
for q in range(0,ny+1):
y = ay+q*hy
for p in range(0,nx+1):
x = ax+p*hx
vec_u[q][p] = funct_u(x,y)
return vec_u

def funct_f(x,y):
return 2*pi**2*sin(pi*x)*sin(pi*y)

def funct_u(x,y):
return sin(pi*x)*sin(pi*y)

def contourplot(XX,ax,bx,ay,by,title,level=0):
ny,nx = len(XX),len(XX[0])
xi = np.linspace(ax,bx,nx)
yi = np.linspace(ay,by,ny)
X,Y= np.meshgrid(xi, yi)
Z = griddata(X.ravel(),Y.ravel(),XX.ravel(),xi,yi)
CS = plt.contour(X, Y, Z, linewidths=2,colors='k')
plt.clabel(CS, inline=2, fmt='%1.1f', fontsize=12)
plt.title(title)
4.5. Numerical Examples with Python 149

plt.show()

def init_X(U,level=0):
X = deepcopy(U)
ny,nx = len(U),len(U[0])
for q in range(1,ny-1):
for p in range(1,nx-1):
X[q][p] = 0.
return X

def sol_SOR(A,X,b,omega,tol,itmax,level=0):
ny,nx = len(X),len(X[0])
for it in range(0,itmax):
err=0.
for j in range(1,ny-1):
for i in range(1,nx-1):
gs =( b[j][i]-(A[j][i][0]*X[j-1][i]\
+A[j][i][1]*X[j][i-1]\
+A[j][i][3]*X[j][i+1]\
+A[j][i][4]*X[j+1][i]) )\
/ A[j][i][2]
xnew = (1.-omega)*X[j][i]+omega*gs
err = max(err, abs(X[j][i]-xnew))
X[j][i] = xnew
if err<tol:
if level>=1:
print "sol_SOR: converged it= %d" %(it+1)
break

def error8(X,Y,level=0):
ny,nx = len(X),len(X[0])
err8=0.
for q in range(0,ny):
for p in range(0,nx):
err8=max(err8,abs(X[q][p]-Y[q][p]))
return err8
150 Chapter 4. Finite Difference Methods for Elliptic Equations

4.6. Homework
1. Verify that the overall truncation error for the FD scheme (4.14) is second-
order in hx . Hint: Define
uxxx (x)  hx 2
K(x) = a(x) + ··· ,
3! 2
for the truncation errors appeared in (4.13). Then the truncation error for
the approximation of (aux )i+1/2 − (aux )i−1/2 becomes K(xi+1/2 ) − K(xi−1/2 ) =
hx K 0 (xi ) + · · · .
2. Implement a code to solve
(
−(uux )x = 0, x ∈ (0, 2),
(4.108)
u(0) = gL , u(2) = gR ,

utilizing the second-order FD scheme (4.14) on a uniform grid. At the grid

point xi , your approximation will read
−u2i−1 + 2u2i − u2i+1
= 0. (4.109)
h2x
For the solver, you may use the simplest method (the Jacobi!) and its
variant. For the number of grid points, you may choose a convenient
number, e.g., nx = 20.

(a) Derive (4.109).

(b) Solve to plot the FD solution
√ for gL = 0 and gR = 2.
(The exact solution u = 2x and you may assume that the numerical
solution is nonnegative.)
 √ for gL = −1 and
(c) Solve to plot the FD solution gR = 1.
 x − 1, x ≥ 1,
√

The exact solution u = The FD equation (4.109)
− 1 − x, x < 1.
q
reads ui = ± (u2i−1 + u2i+1 )/2. You have to modify the iterative algo-
rithm to choose the right one. This step will be so hard, but I believe
it is fun to conquer.
(d) (Optional) Do you have any idea overcoming the difficulty involved in
(4.2c)?
4.6. Homework 151

3. For the 3D Poisson equation

−(uxx + uyy + uzz ) = f, x = (x, y, z) ∈ Ω = (0, 1)3 ,
(4.110)
u = 0, x = (x, y, z) ∈ ∂Ω
(a) Apply the central second-order FD method, with a uniform grid size
h = hx = hy = hz , to get difference equations.
(b) Show that the maximum principle still applies.
(c) Prove that
h2
ku − uh k∞ ≤ max(|uxxxx | + |uyyyy | + |uzzzz |), (4.111)
24 x∈Ω
where uh is the finite difference solution.
4. Consider the eigenvalue problem
−∆u = λu, (x, y) ∈ Ω = (0, 1)2 ,
(4.112)
u = 0, (x, y) ∈ ∂Ω,
where the eigenfunction u(x, y) 6= 0. Prove that the eigenvalues and the
corresponding eigenfunctions are
λmn = (m2 + n2 )π 2 ,
(4.113)
umn (x, y) = sin(mπx) sin(nπy),
for m, n = 1, 2, · · · . (Hint: Set u(x, y) = X(x)Y (y) to plug it in (4.112).)
5. Modify the Python code in §4.5 to add a line SOR method, for the line
either in the x-direction or in the y-direction. Provide a convergence anal-
ysis comparing convergence speeds between the point SOR and the line
SOR.
6. Edit once more the Python code you just modified for Homework 4.5 to
solve more general elliptic problem of the form
−[d1 (x, y)ux ]x − [d2 (x, y)uy ]y + r(x, y)u = f, x ∈ Ω = (0, 1)2
(4.114)
u = g, x ∈ ∂Ω.
(a) Choose f and g accordingly such that the exact solution
u(x, y) = (1 − x2 )(y 3 − y) (4.115)
and the coefficients
d1 (x, y) = 2 + x2 − y 2 , d2 (x, y) = exy , r(x, y) = x + 2y.
152 Chapter 4. Finite Difference Methods for Elliptic Equations

(b) Estimate the convergence rate by running different mesh sizes, for
example, n = 10, 20, 40, 80.
(c) Visualize computed solutions with 3D mesh/surface plots in Python.
7. (Optional) Let A = (aij ) be a nonsingular square matrix, obtained from
a FD/FE approximation of an elliptic problem of the form
−∇ · (a(x)∇u) + b(x) · ∇u + c(x)u = f (x), x ∈ Ω,
(4.116)
α(x)uν + β(x)u = g(x), x ∈ Γ,

where a > 0, c ≥ 0, α ≥ 0, and Ω is a bounded domain in Rd , 1 ≤ d ≤ 3,

with its boundary Γ = ∂Ω. Assume that
(i) The elements in the main diagonal of A are positive and the other
elements are nonpositive, i.e., for each i,

aii > 0; aij ≤ 0, i 6= j.

(ii) A is diagonally dominant, i.e., for each i,

X
aii ≥ |aij |,
j6=i

and at least one of the inequalities is strict.

(iii) The directed graph of A is strongly connected. (The standard FD/FE
methods always satisfy this condition.)

(a) Prove the following generalized maximum principle:

Theorem 4.25. (Maximum Principle) Suppose that A satisfies all
the above assumptions and that

Au ≤ 0 (Au ≥ 0).

Then, the solution u has its maximum (minimum) on the boundary.

(b) Let Ω = (0, 1)3 and consider the 7-point FD method for the problem
in (4.116). Find conditions on the coefficients and the mesh size h
with which the numerical solution of (4.116) satisfies the maximum
principle.
Chapter 5

Finite Element Methods for Elliptic

Equations

This chapter consideres finite element and finite volume methods for elliptic
PDEs defined on 1D and 2D regions.

153
154 Chapter 5. Finite Element Methods for Elliptic Equations

5.1. Finite Element (FE) Methods in 1D Space

Consider the model problem formulated in 1D space:

−u00 = f, x ∈ I = (0, 1),


(D) (5.1)
u = 0, x = 0, 1,

which we call the differential problem (D).

FEM begins with a variational formulation for the given differential
problem. The variational formulation is sometimes called the weak formu-
lation.

5.1.1. Variational formulation

Define the product ˆ
(v, w) = v(x)w(x)dx (5.2)
I
and the linear space
V = {v : v ∈ C 0 [0, 1]; v 0 is piecewise continuous
(5.3)
and bounded on [0, 1]; v(0) = v(1) = 0}.
5.1. Finite Element (FE) Methods in 1D Space 155

Variational problem: Use the integration by parts to have

ˆ 1
ˆ ˆ
00 0 0 0
−u v = −u v + uv = u0 v 0 .
I 0 I I

Then, (5.1) can be written as

(u0 , v 0 ) = (f, v), ∀ v ∈ V. (5.4)

Now, we define the variational problem (V) corresponding to the differ-

ential problem (5.1):

(V) Find u ∈ V such that

(5.5)
(u0 , v 0 ) = (f, v), ∀ v ∈ V.

Claim 5.1. The problem (D) is equivalent to the problem (V), when solutions
are sufficiently smooth.
Proof. ((D) ⇒ (V)): Clear.
((D) ⇐ (V)): Let u be a solution of (V). Then,

(u0 , v 0 ) = (f, v), ∀ v ∈ V. (5.6)

Now, assume that u00 exists. Then, because

ˆ 1
ˆ
(u0 , v 0 ) = u0 v 0 = u0 v − u00 v = (−u00 , v),
I 0 I

Equation (5.6) reads

(u00 + f, v) = 0, ∀ v ∈ V.
So u should satisfy (5.1).
156 Chapter 5. Finite Element Methods for Elliptic Equations

Minimization problem:
Define a functional F : V → R as
1
F (v) = (v 0 , v 0 ) − (f, v), v ∈ V. (5.7)
2

Then, the minimization problem (M) is formulated as

(M) Find u ∈ V such that

(5.8)
F (u) ≤ F (v), ∀ v ∈ V.
5.1. Finite Element (FE) Methods in 1D Space 157

Claim 5.2. The minimization problem (M) is equivalent to the variational

problem (V).
Proof. (⇒): Let u be a solution of (M). Then,

F (u) ≤ F (u + εv), ∀ v ∈ V, ∀ε ∈ R. (5.9)

Define g(ε) = F (u + εv). Then, g 0 (0) = 0. Since

1 ε2
g(ε) = (u0 , u0 ) + ε(u0 , v 0 ) + (v 0 , v 0 ) − (f, u) − ε(f, v), (5.10)
2 2
we have
g 0 (ε) = [(u0 , v 0 ) + ε(v 0 , v 0 ) − (f, v)] = 0, ∀ v ∈ V.
ε=0 ε=0
0 0
So, we conclude (u , v ) = (f, v), ∀ v ∈ V .
(⇐): Now, let u be a solution of (V). Then, the objective is to show F (u) ≤
F (v), ∀ v ∈ V. For given v ∈ V , let w = v − u. Then, w ∈ V and
1
F (v) = F (u + w) = (u0 + w0 , u0 + w0 ) − (f, u + w)
2
1 0 0 1
= (u , u ) − (f, u) + (w0 , w0 ) + (u0 , w0 ) − (f, w).
2 2
The last two terms in the right side of the above equation become zero, because
u be a solution of (V). So
1
F (v) = F (u) + (w0 , w0 ) ≥ F (u), ∀ v ∈ V,
2
which completes the proof.
158 Chapter 5. Finite Element Methods for Elliptic Equations

Claim 5.3. The problem (V) admits a unique solution.

Proof. Existence and uniqueness can be proved in an abstract mathematical
theory for variational problems, using the Lax-Milgram lemma, as in Theo-
rem 5.12 on p.202. Here we will consider uniqueness only.
(Uniqueness): Let u1 and u2 be two solutions of (V). Then,
(u01 , v 0 ) = (f, v), ∀ v ∈ V,
(u02 , v 0 ) = (f, v), ∀ v ∈ V,

which reads
(u01 − u02 , v 0 ) = 0, ∀ v ∈ V.
Thus, by choosing v = (u1 − u2 ), we reach at
ˆ
(u01 − u02 )2 dx = 0,
I

which implies u01 − u02 = 0 and therefore u1 − u2 = c, a constant. Since u1 (0) =

u2 (0) = 0, the constant c must be zero. Thus u1 ≡ u2 , which completes the
proof.

In summary:

• (D) ⇔ (V) ⇔ (M). (when u00 exists)

• They admit a unique solution.
5.1. Finite Element (FE) Methods in 1D Space 159

5.1.2. Formulation of FEMs

In designing a FEM, the following steps are to be performed:

• Partitioning: The domain should be partitioned into a collection of ele-

ments of the mesh size h.
• Subspace Vh ⊂ V and basis functions {ϕj (x)}: A subspace is set to
represent the numerical solution that is a linear combination of basis
functions. That is,
M
X
uh (x) = ξj ϕj (x). (5.11)
j=1

For example, ϕj (x) are piecewise polynomials (splines).

• Application of variational principles: Different variational princi-
ples produce various FEMs.

– the minimization principle (Rayleigh-Ritz)

– weighted residual approaches with the weights being either the basis
functions (Galerkin) or different functions (Petrov-Galerkin)
– least-square approaches
– collocation method

• Assembly for a linear system: The linear system can be assembled

for (ξ1 , ξ2 , · · · , ξM )T with the integrals approximated by numerical quadra-
ture.
160 Chapter 5. Finite Element Methods for Elliptic Equations

Step 1. Partitioning: Let

0 = x0 < x1 < · · · < xM < xM +1 = 1

be a partition of the unit interval. Define

hj = xj − xj−1 , Ij = [xj−1 , xj ], j = 1, 2, · · · , M + 1

and
h= max hj .
1≤j≤M +1

Step 2. Subspace and basis functions: Define a finite-dimensional

subspace of V as
Vh = {v ∈ V : v is a polynomial of
(5.12)
degree ≤ k on each Ij }.

Notes:

• Corresponding basis functions are determined depending on the choice of

polynomial degree k ≥ 1 and therefore on the nodal points.
• Each of basis functions is related to a nodal point.
• Basis functions ϕj ∈ Vh are defined to satisfy

1, if i = j,
ϕj (xi ) = δij :=
0, else.
5.1. Finite Element (FE) Methods in 1D Space 161

Figure 5.1: The basis function ϕj .

Example: k = 1 (the linear FEM): The basis function ϕj is depicted in

Figure 5.1:
1


 (x − xj−1 ), x ∈ [xj−1 , xj ],



 hj

ϕj (x) = −1 (5.13)
(x − xj+1 ), x ∈ [xj , xj+1 ],



 hj+1

0, elsewhere.


Notes:

• The functions v ∈ Vh can be expressed as a linear combination of the basis

functions as
M
X
v(x) = ηj ϕj (x), x ∈ [0, 1].
j=1

• The above expression is unique for given v ∈ Vh ; in fact,

ηj = v(xj ), j = 1, 2, · · · , M.
162 Chapter 5. Finite Element Methods for Elliptic Equations

Example: k > 1 (higher-order FEMs):

• For each interval Ij = [xj−1 , xj ], the degree of freedom of k-th order poly-
nomials is k + 1.
It requires to choose k + 1 nodal points in each interval.
• As for the linear FEM, the two endpoints can naturally become nodal
points.
We should select k − 1 extra nodal points inside the interval Ij .
• In the literature, a common practice is to select those nodal points in
such a way that the numerical quadrature of the integrals is as accurate
as possible when the nodal points are used as quadrature points.
• Such selection is related to the family of orthogonal polynomials such as
Legendre polynomials and Chebyshev polynomials; see Appendix E for
details.
5.1. Finite Element (FE) Methods in 1D Space 163

Step 3. Application of variational principles: The most popular

FEM is the Galerkin method, which is a weighted residual approach with
the weights being basis functions.

Weighted residual approaches: Let P (u) = −u00 . For the differential

problem (5.1), define the residual R as

R(v) = P (v) − f (5.14)

Then, we have
R(u) = P (u) − f = 0.
M
X
However, for uh (x) = ξj ϕj (x),
j=1

R(uh ) = P (uh ) − f 6= 0, in general. (5.15)

Weighed residual approaches are seeking an approximate solution

M
X
uh (x) = ξj ϕj (x)
j=1

which satisfies ˆ
R(uh ) w(x) dx = 0, (5.16)
I
for a sequence of weight functions w(x) ∈ {wi (x)}, which is also called
trial functions.
When the integration by parts is utilized, (5.16) reads

(u0h , w0 ) = (f, w) (5.17)

164 Chapter 5. Finite Element Methods for Elliptic Equations

The linear Galerkin method: For the subspace Vh of linear basis functions
{ϕj (x)}, let
wi (x) = ϕi (x) (5.18)
Then, the linear Galerkin FEM for the differential problem (5.1) is formulated
as
Find uh ∈ Vh s.t. (u0h , ϕ0i ) = (f, ϕi ), ∀ ϕi ∈ Vh (5.19)

As in §5.1.1, one can show that (5.19) admits a unique solution.

5.1. Finite Element (FE) Methods in 1D Space 165

Step 4. Assembly for a linear system:

• Given basis functions {ϕj (x)} ⊂ Vh , the numerical solution uh is uniquely
expressed as
XM
uh (x) = ξj ϕj (x). (5.20)
j=1

• The numerical solution must be the solution of a variational formulation.

For example, the solution of the linear Galerkin FEM satisfies

(u0h , ϕ0i ) = (f, ϕi ), ∀ ϕi ∈ Vh (5.21)

The next objective is to assemble the linear system for the unknown vector
ξ := (ξ1 , ξ2 , · · · , ξM )T . From (5.20) and (5.21),
M
X
(u0h , ϕ0i ) = ξj (ϕ0j , ϕ0i ) = (f, ϕi ), ∀ ϕi ∈ Vh .
j=1

We rewrite the above equation

M
X
(ϕ0j , ϕ0i )ξj = (f, ϕi ), i = 1, · · · , M. (5.22)
j=1

Define
aij = (ϕ0j , ϕ0i ), bi = (f, ϕi ). (5.23)
166 Chapter 5. Finite Element Methods for Elliptic Equations

Then, (5.22) equivalently reads the algebraic system of the form

Aξ = b, (5.24)

where A = (aij ) is an M × M matrix and b = (b1 , b2 , · · · , bM )T is the source

vector.

• The matrix A has good properties such as being symmetric and positive
definite.
• We will show them later; we first consider details for the computation of
aij and bi .
• Note that
ˆ
aij = (ϕ0j , ϕ0i ) = ϕ0j (x)ϕ0i (x)dx = 0, if |i − j| ≥ 2,
I

because the support of ϕj is [xj−1 , xj+1 ]. Thus, there are only three cases
for nonzero entries of A:

j = i − 1, i, i + 1.
5.1. Finite Element (FE) Methods in 1D Space 167

Computation of aij and bi : Recall

1


 (x − xj−1 ), x ∈ [xj−1 , xj ],



 hj

ϕj (x) = −1 (5.25)
(x − xj+1 ), x ∈ [xj , xj+1 ],



 hj+1

0, elsewhere.


Case j = i − 1: It follows from (5.25) that

ˆ xi
ai,i−1 = (ϕ0i−1 , ϕ0i ) = ϕ0i−1 (x)ϕ0i (x)dx
ˆ xi xi−1
−1 1 −1
= · dx = .
xi−1 h i h i h i

Case j = i: Again utilizing (5.25), we have

ˆ xi+1
ai,i = (ϕ0i , ϕ0i ) = ϕ0i (x)ϕ0i (x)dx
ˆ xi ˆ xi+1xi−1
1 1
= + ϕ0i (x)ϕ0i (x)dx = + .
xi−1 xi hi hi+1

Case j = i + 1:
ˆ xi+1
ai,i+1 = (ϕ0i+1 , ϕ0i ) = ϕ0i+1 (x)ϕ0i (x)dx
ˆ xi+1
xi
1 −1 −1
= · dx = .
xi hi+1 hi+1 hi+1

Computation of bi : Finally, it can be done as

ˆ xi+1
hi + hi+1
bi = (f, ϕi ) = f (x)ϕi (x)dx ≈ fi ,
xi−1 2

where f has been approximated by fi = f (xi ) on [xi−1 , xi+1 ].

168 Chapter 5. Finite Element Methods for Elliptic Equations

Properties of the algebraic system:

Definition 5.4. A matrix S = (sij ) ∈ RM ×M is said to be positive definite if
M
X
η · Sη = ηi sij ηj > 0, ∀ η ∈ RM , η 6= 0.
i,j=1

It has been known that a matrix S is symmetric positive definite if and only
if all eigenvalues of S are strictly positive.
Lemma 5.5. The matrix A in (5.24) is symmetric positive definite.
Proof. Symmetry is easy to see, because

aij := (ϕ0j , ϕ0i ) = (ϕ0i , ϕ0j ) =: aji .

M
X
M
Given η ∈ R , we define v(x) = ηj ϕj (x). Then
j=1

M
X M
X
η · Aη = ηi aij ηj = ηi (ϕ0i , ϕ0j )ηj
i,j=1 i,j=1
M
X M  (5.26)
X
= ηi ϕ0i , ηj ϕ0j ≥ 0,
i j

with equality satisfied only if v 0 = 0, and therefore only if v = 0 because

v(0) = 0; which implies that equality holds only if η = 0. This completes the
proof.
5.1. Finite Element (FE) Methods in 1D Space 169

Figure 5.2: The element Ii = [xi−1 , xi ] and the basis functions for the cubic FE
method.

Higher-order FEMs:
• Higher-order FE methods introduce higher-order basis functions.
• Figure 5.2 presents the element Ii = [xi−1 , xi ] and the basis functions each
of which is cubic in Ii .
• Since the degree of freedom for cubic polynomials is four, we need to pro-
vide four independent information to determine the polynomial uniquely.
• For the purpose, one can choose four distinct points (including two edge
points), as shown in Figure 5.2. The points are called the nodal points.
170 Chapter 5. Finite Element Methods for Elliptic Equations

Construction of cubic basis functions:

• Let the nodal points be given and denoted by `p , p = 0, · · · , 3.
• Then the local basis functions ϕj on the element Ii must read

ϕj (`p ) = δjp , j, p = 0, · · · , 3.

• The above property can be satisfied the cardinal functions:

3 x−` 
m
Y
ϕj (x) = , j = 0, · · · , 3, (5.27)
`j − `m
m=0
m 6= j

and they can serve as basis functions.

• It is often to choose Gauss-Lobatto points for the nodal points; see Ap-
pendix E for details.
5.1. Finite Element (FE) Methods in 1D Space 171

Construction of general-order basis functions: We generalize

the above argument for FE methods utilizing piecewise kth-order polynomials
k ≥ 1, as follows:

• Select extra (k −1) nodal points such that each element Ii has (k +1) nodal
points including the two edge points.
• Denote them by `m , m = 0, · · · , k.
• Define the local basis functions as
k x−` 
m
Y
ϕj (x) = , j = 0, · · · , k.
`j − `m
m=0
m 6= j

• The basis functions associated with the edge points must be extended
both side for the final form of the basis functions.
172 Chapter 5. Finite Element Methods for Elliptic Equations

5.2. The Hilbert spaces

We first define the space of square integrable functions on I:
ˆ
L2 (I) = {v : v is defined on I and v 2 dx < ∞}.
I

The space L2 (I) is a Hilbert space with the scalar product

ˆ
(v, w) = v(x)w(x)dx
I

and the corresponding norm (the L2 -norm)

ˆ 1/2
1/2 2
kvk = (v, v) = [v(x)] dx .
I

In general, for an integer r ≥ 0, we define a Hilbert space

H r (I) = {v ∈ L2 (I) : v (k) ∈ L2 (I), k = 1, · · · , r}

with the corresponding norm (the H r (I)-norm)

ˆ Xr h i2 1/2
(k)
kvkr = v (x) dx ,
I k=0

where v (k) denotes the k-th derivative of v. It is often convenient to define

ˆ h i2 1/2
(r)
|v|r = v (x) dx , v ∈ H r (I).
I

Note that L2 (I) = H 0 (I) and k · k = k · k0 = | · |0 .

The following shall be useful for the error estimate to be presented in §5.3.
5.2. The Hilbert spaces 173

The Cauchy-Schwarz inequality reads

|(v, w)| ≤ kvk · kwk. (5.28)

Consider the problem (D) in (5.1). Then, it is well known that

kuks+2 ≤ Ckf ks , s = 0, 1, · · · , (5.29)

for some C > 0, independent of u and f . The above regularity estimate holds
for higher-dimensional problems (the Poisson equation in 2D and 3D) when
the boundary is smooth enough. See Appendix B.1 for the details.
174 Chapter 5. Finite Element Methods for Elliptic Equations

5.3. An error estimate for FEM in 1D

Let u and uh be the solutions of Problem (V) in (5.5) and Problem (Vh ) in (5.19),
respectively. Then,
(u0 , v 0 ) = (f, v), ∀ v ∈ V,
(u0h , v 0 ) = (f, v), ∀ v ∈ Vh .
Note that Vh ⊂ V . Thus it follows from the above equations that

(u0 − u0h , v 0 ) = 0, ∀ v ∈ Vh . (5.30)

Theorem 5.6. For any v ∈ Vh , we have

k(u − uh )0 k ≤ k(u − v)0 k. (5.31)

Proof. Given v, an arbitrary function in Vh , let w = uh −v ∈ Vh . Then, utilizing

(5.30) and the Cauchy-Schwarz inequality, we have
k(u − uh )0 k2 = ((u − uh )0 , (u − uh )0 )
= ((u − uh )0 , (u − uh )0 ) + ((u − uh )0 , w0 )
= ((u − uh )0 , (u − uh + w)0 )
= ((u − uh )0 , (u − v)0 )
≤ k(u − uh )0 k · k(u − v)0 k,

from which (5.31) follows.

5.3. An error estimate for FEM in 1D 175

Figure 5.3: The solution u and its interpolant πh u.

Notes
• The inequality (5.31) allows us to analyze the error k(u − uh )0 k quantita-
tively.
• That is, we can choose v ∈ Vh suitably to estimate the right side of (5.31).
• We shall choose v to be the interpolant of u, πh u, which interpolates u at
all the nodal points xj . See Figure 5.3.

Now, one can prove that for x ∈ [0, 1],

h2
|u(x) − πh u(x)| ≤ max |u00 (ξ)|, (5.32)
8 ξ∈(0,1)

|u0 (x) − πh u0 (x)| ≤ h max |u00 (ξ)|. (5.33)

ξ∈(0,1)

(See Homework 5.2.) The above inequalities hold for any (sufficiently smooth)
function u and its interpolant πh u. The estimates are called the interpolation
estimates.
176 Chapter 5. Finite Element Methods for Elliptic Equations

It follows from (5.33) and Theorem 5.6 that

k(u − uh )0 k0 ≤ Ch|u|2 , (5.34)

for some constant C > 0, independent of h.

Since ˆ x
|(u − uh )(x)| = (u − uh )0 (t)dt
ˆ x
0
1/2
0 2
≤ k(u − uh ) k0 · 1 dt
0
0
≤ k(u − uh ) k0 ,
we have
|(u − uh )(x)| ≤ Ch|u|2 . (5.35)
Therefore, from (5.34) and (5.35),

ku − uh k1 ≤ Ch|u|2 , (5.36)
5.3. An error estimate for FEM in 1D 177

Estimation of ku − uhk0

Theorem 5.7. Let u and uh be the solutions of Problem (V) and Problem
(Vh ), respectively. Then

ku − uh k0 ≤ Ch2 |u|2 , (5.37)

where C > 0 is independent on h.

Proof. Let e = u − uh . Then, we know from (5.30) that

(e0 , v 0 ) = 0, ∀v ∈ Vh . (5.38)

We shall estimate (e, e) = kek20 using the so-called duality argument which
is popular in FEM error analysis. Let φ be the solution of the following dual
problem
−φ00 = e, x ∈ I,
(5.39)
φ = 0, x = 0 or 1.
Then, from (5.29) with s = 0,

kφk2 ≤ Ckek0 , (5.40)

where C > 0 is independent on e. Using the integration by parts and the fact
that e(0) = e(1) = 0,

(e, e) = (e, −φ00 ) = (e0 , φ0 ) = (e0 , φ0 − πh φ0 ),

where πh φ ∈ Vh denotes the interpolant of φ. Now, apply the interpolation

estimate (5.33) to φ and use the regularity estimate (5.40) to get

kek20 ≤ kek1 · kφ − πh φk1 ≤ kek1 · Ch|φ|2 ≤ Chkek1 · kek0 .

Thus dividing by kek0 and utilizing (5.36), we finally reach at

kek0 ≤ Chkek1 ≤ Ch2 |u|2

and the proof is complete.

Summary: Error estimate for the linear FEM: The error estimates in
(5.36) and (5.37) can be rewritten as
178 Chapter 5. Finite Element Methods for Elliptic Equations

ku − uh ks ≤ Ch2−s |u|2 , s = 0, 1. (5.41)

Error estimate for general-order FEMs: When piecewise k-th order poly-
nomials (k ≥ 1) are employed for the basis functions, one can use the same
arguments presented in this section to show

ku − uh ks ≤ Chk+1−s |u|k+1 , s = 0, 1, · · · , k. (5.42)

5.4. Other Variational Principles 179

5.4. Other Variational Principles

The FEM we have consider so far is the Galerkin method, one of weighted
residual approaches.
There have been other variational principles such as

• the minimization principle (Rayleigh-Ritz methods),

• least-square approaches,
• collocation methods, and
• weighted residual approaches with the weights being different from the
basis functions (Petrov-Galerkin methods).
180 Chapter 5. Finite Element Methods for Elliptic Equations

5.5. FEM for the Poisson equation

Let Ω ⊂ R2 be bounded domain with its boundary Γ = ∂Ω being smooth.
Consider 
−∆u = f, x ∈ Ω,
(D) (5.43)
u = 0, x ∈ Γ,
where x = (x, y) = (x1 , x2 ).

5.5.1. Integration by parts

To derive a variational form for (5.43), we first introduce the divergence theo-
rem. Let A = (A1 , A2 ) be a vector-valued function on R2 . Then divergence of A
is defined as
∂A1 ∂A2
∇·A= + .
∂x1 ∂x2
Let n = (n1 , n2 ) be the outward unit normal to Γ and
∂v ∂v ∂v
vn = = ∇v · n = n1 + n2 .
∂n ∂x1 ∂x2

Theorem 5.8. (Divergence theorem) Let A = (A1 , A2 ) be a vector-valued

differentiable function on a bounded region Ω in R2 . Then
ˆ ˆ
∇ · Adx = A · nds, (5.44)
Ω Γ

where s is the element of arc length.

5.5. FEM for the Poisson equation 181

Apply the divergence theorem to A = (vw, 0) and A = (0, vw) to read

ˆ ˆ
∂
(vw)dx = vwn1 ds,
ˆΩ ∂x 1 ˆΓ
∂
(vw)dx = vwn2 ds,
Ω ∂x 2 Γ

which implies
ˆ ˆ ˆ
∂v ∂w
wdx = vwni ds − v dx, i = 1, 2. (5.45)
Ω ∂x i Γ Ω ∂xi
Thus we have the Green’s formula
ˆ ˆ  
∂v ∂w ∂v ∂w
∇v · ∇wdx ≡ +
Ω ∂x1 ∂x1 ∂x ∂x
Ω
ˆ ˆ 2 22
∂w ∂ w
= v n1 ds − v 2
dx
∂x ∂x
Γ
ˆ 1 Ω
ˆ 1
∂w ∂ 2w
+ v n2 ds − v 2
dx
ˆ Γ ∂x 2 ˆ Ω ∂x 2
∂w
= v ds − v∆wdx.
Γ ∂n Ω

That is,

(∇v, ∇w) =< v, wn > −(v, ∆w), (5.46)

´
where < v, w >= Γ vwds.
182 Chapter 5. Finite Element Methods for Elliptic Equations

The linear space: Now, define the linear space

V = {v : v ∈ C 0 (Ω); ∇v is piecewise continuous
(5.47)
and bounded on Ω; v(x) = 0, x ∈ Γ}.

Let ˆ
a(u, v) = ∇u · ∇vdx.
Ω
Define the variational problem (V)
(
Find u ∈ V such that
(V) (5.48)
a(u, v) = (f, v), ∀ v ∈ V,

and the minimization problem (M)

(
Find u ∈ V such that
(M) (5.49)
F (u) ≤ F (v), ∀ v ∈ V,

where
1
F (v) = a(v, v) − (f, v).
2
Then, as for the 1D model problem in §5.1.1, one can prove that

• problems (D), (V), and (M) are equivalent when the solution u is suffi-
ciently smooth, and
• they admit a unique solution.
5.5. FEM for the Poisson equation 183

5.5.2. Defining FEMs

To define an FEM for the Poisson equation (5.48), we need to follow steps as
for the FE method for the 1D problem presented in §5.1.2:

• Triangulation
• Subspace Vh ⊂ V and basis functions
• Application of variational principles
• Assembly for the linear system
184 Chapter 5. Finite Element Methods for Elliptic Equations

Figure 5.4: Triangulation Th of Ω.

Step 1. Triangulation: Let Kj , j = 1, · · · , m, be nonoverlapping trian-

gles such that
Ω = ∪m
j=1 Kj ;

we assume that no vertex of a triangle lies on the edge of another triangle as

shown in Figure 5.4.
Let h be the longest side of edges of the triangles, i.e.,

h = max diam(Kj ).
j

Then the collection of such triangles composes the finite elements

Th = {K1 , K2 , · · · , Km }.

An FE mesh consists of
nPT the number of vertices (points)
nEL the number of elements/triangles
(x, y)i the vertices
(n1 , n2 , n3 )j the connectivity
5.5. FEM for the Poisson equation 185

Figure 5.5: Two meshes Dr. Kim made, using the Python package MeshPy.

Step 2. Subspace Vh ⊂ V and basis functions: For the linear

FE method, we define a subspace of V as

Vh = {v ∈ V : v is linear on each Kj }. (5.50)

The corresponding basis functions {ϕj } are as

ϕj (Ni ) = δij ,

where Ni are the vertices, the nodal points.

Each basis function ϕi restricted on an element Kj , one vertex of which is
Ni , is linear of the form

ϕi (x) = ax1 + bx2 + c, x ∈ Kj .

186 Chapter 5. Finite Element Methods for Elliptic Equations

Step 3. Application of variational principles: The linear Galerkin

FEM for (5.48) can be formulated as
(
Find uh ∈ Vh such that
(Vh ) (5.51)
a(uh , v) = (f, v), ∀ v ∈ Vh .

The error analysis for the linear Galerkin method can be carried out fol-
lowing the arguments in §5.3.

Theorem 5.9. Let u and uh be the solutions of (5.48) and (5.51), respec-
tively. Then
ku − uh ks ≤ Ch2−s |u|2 , s = 0, 1, (5.52)
where C > 0 is a constant independent on h.

It is fun to prove the theorem; challenge it for an extra credit, or more impor-
tantly, for your pride!
5.5. FEM for the Poisson equation 187

Step 4. Assembly for the linear system: Let

M
X
uh (x) = ξj ϕj (x), for some M > 0.
j=1

Then, the algebraic system for (5.51) can be formulated as

Aξ = b, (5.53)

where ξ = (ξ1 , · · · , ξM )T is the solution vector and

A = (aij ), aij := a(ϕj , ϕi ),

b = (b1 , · · · , bM )T , bi := (f, ϕi ).

Notes:

• As for the 1D problem in §5.1.2, the matrix A is symmetric and positive

definite.
• Thus the system (5.53) admits a unique solution.
188 Chapter 5. Finite Element Methods for Elliptic Equations

Stiffness matrix A:
Let the stiffness matrix be A = (aij ). Then,
X
aij = a(ϕj , ϕi ) = aK
ij , (5.54)
K∈Th

where ˆ
aK
ij = aK (ϕj , ϕi ) = ∇ϕj · ∇ϕi dx. (5.55)
K
Definition 5.10. The element stiffness matrix AK of the element K is
 K K K 
a11 a12 a13
 K K K 
AK =  a21 a22 a23  ,
aK K K
31 a32 a33

where each component can be computed from (5.55).

• The stiffness matrix A can be constructed through the contributions from

the element stiffness matrices AK , K ∈ Th .
• Looks complicated? We will deal with an efficient method for the compu-
tation of aK
ij in a separate section; see §5.5.3.
5.5. FEM for the Poisson equation 189

b → K.
Figure 5.6: The affine mapping F : K

5.5.3. Assembly: Element stiffness matrices

• The computation of the element stiffness matrix

AK := aK 3×3


ij ∈ R

is not a simple task for the element K ∈ Th in a general geometry.

• To overcome the complexity, we introduce the reference element K
b and
b → K. See Figure 5.6.
an affine mapping F : K

c It has the following three vertices

The reference element K:

a1 = [0, 0]T , b
b a2 = [1, 0]T , b
a3 = [0, 1]T , (5.56)

and the corresponding reference basis functions are

ϕ x) = 1 − x
b1 (b b1 − x
b2 , ϕ
b2 (b
x) = x
b1 , ϕ
b3 (b
x) = x
b2 . (5.57)
190 Chapter 5. Finite Element Methods for Elliptic Equations

b → K (b
Affine mapping F : The mapping F : K x 7→ x) must be defined as

ai = F (b
ai ), ϕi (x) = ϕ
bi (b
x), i = 1, 2, 3. (5.58)

That is, the corners and the basis functions of K are defined as the affine
images of those of K.
b
Let J be the Jacobian of the affine mapping F :
    " ∂x1 ∂x1 #
∂Fi ∂xi ∂b
x ∂b
x
J := = = ∂x12 ∂x22 . (5.59)
∂b
xj ∂bxj ∂b
x ∂b
x 1 2

Then, it follows from the chain rule that

∇ϕj = J −T ∇ϕ
bj , j = 1, 2, 3, (5.60)

where J −T is the transpose of J −1 , which implies

ˆ
aK
ij := ∇ϕj · ∇ϕi dx
ˆK
(5.61)
= (J −T ∇ϕ bj ) · (J −T ∇ϕ
bi ) |detJ| db
x.
K
b

Notes:

• Every affine mapping in Rn has the form Bb

x + s, where B ∈ Rn×n and
s ∈ Rn .
• From some algebra, it can be shown that

x) = [a2 − a1 , a3 − a1 ] x
F (b b + a1 (5.62)

Thus
J = [a2 − a1 , a3 − a1 ] ∈ R2×2 . (5.63)
5.5. FEM for the Poisson equation 191

5.5.4. Extension to Neumann boundary conditions

Consider the following problem of Neumann boundary condition
−∆u + u = f, x ∈ Ω,
(5.64)
un = g, x ∈ Γ.

For the problem, it is natural to choose V = H 1 (Ω) for the linear space.
Integration by parts: It follows from the Green’s formula (5.46) that (5.64)
reads
(∇u, ∇v) + (u, v) = (f, v)+ < g, v >, v ∈ V. (5.65)
Define
a(u, v) = (∇u, ∇v) + (u, v),
1
F (v) = a(v, v) − (f, v)− < g, v > .
2
Then, one can formulate the variational problem
(
Find u ∈ V such that
(V) (5.66)
a(u, v) = (f, v)+ < g, v >, ∀ v ∈ V,

and the minimization problem

(
Find u ∈ V such that
(M) (5.67)
F (u) ≤ F (v), ∀ v ∈ V.
192 Chapter 5. Finite Element Methods for Elliptic Equations

Notes:

• In (5.66) the boundary condition is implicitly imposed. Such a boundary

condition is called a natural boundary condition.
• On the other hand, the Dirichlet boundary condition as in (5.43) is called
a essential boundary condition.

• For the problem (5.66), an FEM can be formulated as for (5.48); a similar
error analysis can be obtained.
5.6. Finite Volume (FV) Method 193

5.6. Finite Volume (FV) Method

Here we will discuss one of easiest FV methods formulated on a rectangu-
lar domain. For problems on more general domains or convection-dominated
problems, the FV method can be more complicated. However, the major ideas
would be near around the same corner.

Consider the following problem of general diffusion coefficients

−∇ · (a∇u) = f, x ∈ Ω,
(5.68)
u = 0, x ∈ Γ.
194 Chapter 5. Finite Element Methods for Elliptic Equations

Figure 5.7: Cell-centered FV method on a uniform mesh of grid size hx × hy .

For this case, each cell is a control volume.

Formulation of FV methods
1. Triangulation: Let Ω be a rectangular domain partitioned into elements,
called cells. For simplicity, we assume all cells are rectangular of size hx × hy .
See Figure 5.7.
2. Localization: Let φpq be the characteristic function of the cell Kpq , i.e.,

1, if x ∈ Kpq ,
φpq (x) =
0, else.
3. Variational principle: Multiplying the first equation of (5.68) by φpq and
integrating the result over the domain Ω, we have
ˆ ˆ ˆ
−∇ · (a∇u)φpq dx = −∇ · (a∇u)dx = f dx.
Ω Kpq Kpq

Therefore, from the divergence theorem,

ˆ ˆ
− aunpq ds = f dx, (5.69)
∂Kpq Kpq

where s is the edge element and npq denotes the unit out normal to ∂Kpq .
4. Approximation and evaluation: Now we have to evaluate or approxi-
mate the quantity aunpq along the boundary of the cell Kpq .
5.6. Finite Volume (FV) Method 195

On ∂Kpq ∩ ∂Kp+1,q (“East", the right vertical edge), for example, it can be
approximated as
up+1,q − up,q
aunpq (x) ≈ ap+1/2,q , x ∈ ∂Kpq ∩ ∂Kp+1,q , (5.70)
hx
where the approximation is second-order accurate.

Thus ˆ
hy
(E) aunpq (x)ds ≈ ap+1/2,q (up+1,q − up,q ). (5.71)
Kpq ∩∂Kp+1,q hx
The same can be applied for other edges. That is,
ˆ
hy
(W) aunpq (x)ds ≈ ap−1/2,q (up−1,q − up,q )
ˆKpq ∩∂Kp−1,q hx
hx
(N) aunpq (x)ds ≈ ap,q+1/2 (up,q+1 − up,q ) (5.72)
ˆ pq
K ∩∂Kp,q+1
hy
hx
(S) aunpq (x)ds ≈ ap,q−1/2 (up,q−1 − up,q )
Kpq ∩∂Kp,q−1 hy
196 Chapter 5. Finite Element Methods for Elliptic Equations

The right-hand side term: The right-hand side term of (5.69) can be inte-
grated by the mass-lumping technique to become hx hy fpq . That is,
ˆ
f dx ≈ hx hy fpq . (5.73)
Kpq

For (5.69), combine (5.71), (5.72), and (5.73) and divide the resulting equa-
tion by hx hy to have
h 1 1
− 2 ap+1/2,q (up+1,q − up,q ) + 2 ap−1/2,q (up−1,q − up,q )
hx hx
1 1 i
+ 2 ap,q+1/2 (up,q+1 − up,q ) + 2 ap,q−1/2 (up,q−1 − up,q )
hy hy
−ap−1/2,q up−1,q + (ap−1/2,q + ap+1/2,q )up,q − ap+1/2,q up+1,q (5.74)
= 2
hx
−ap,q−1/2 up,q−1 + (ap,q−1/2 + ap,q+1/2 )up,q − ap,q+1/2 up,q+1
h2y
= fpq

which is the same as the finite difference equation for interior nodal points.
Convection term: When a convection term b · ∇u appears in the differential
equation, the same idea can be applied. For example, since b · ∇u = b1 ux + b2 uy
in 2D,
ˆ ˆ
b · ∇uφpq dx = (b1 ux + b2 uy )dx
Ω Kpq (5.75)
 up+1,q − up−1,q up,q+1 − up,q−1 
≈ hx hy b1,pq + b2,pq ,
2hx 2hy
which is again the same as the FD method.
5.6. Finite Volume (FV) Method 197

Remarks:
• The idea used in the above is the basis for the finite volume method de-
fined on control volumes (CVs).
• Here we have put the nodal points at the center of the rectangular cells
and used the cells for the CVs. Thus the method is sometimes called the
cell-centered finite difference method.
• At interior points, the algebraic equations obtained from the FV method
are equivalent to those of the second-order FD method (on rectangular
meshes) or the linear FE method (on triangular meshes).
• Boundary conditions must be treated accurately. See Homework 5.3.
• When the nodal points are set on the corners of the cells, the CV should be
determined such that it contains the nodal point in an appropriate way;
the CVs are nonoverlapping and their union becomes the whole domain.
198 Chapter 5. Finite Element Methods for Elliptic Equations

5.7. Average of The Diffusion Coefficient

Remarks
• The conormal flux aun on a interface denotes the mass or energy move-
ment through the interface.
• Thus it must be continuous (mass/energy conservation), on the interfaces
of finite elements or control volumes. That is,

aunpq (x) = −aunp+1,q (x), x ∈ ∂Kpq ∩ ∂Kp+1,q (5.76)

• Such a physical consideration gives a way of approximating the diffusion

coefficient a to get a more physical (and therefor more accurate) numeri-
cal solution.
5.7. Average of The Diffusion Coefficient 199

Approximation of the diffusion coefficient

• Let a be locally constant, i.e., constant on each cell.

• Then conormal flux in (5.69) on ∂Kpq ∩ ∂Kp+1,q can be approximated as
ue − upq
aunpq (x) ≈ apq , x ∈ ∂Kpq ∩ ∂Kp+1,q , (5.77)
hx /2
where ue is introduced to represent the solution on the interface ∂Kpq ∩
∂Kp+1,q .
• From the other side of the interface, we have
ue − up+1,q
aunp+1,q (x) ≈ ap+1,q , x ∈ ∂Kpq ∩ ∂Kp+1,q . (5.78)
hx /2

• Here the goal is to find e

a such that
ue − upq up+1,q − ue up+1,q − upq
apq = ap+1,q =e
a . (5.79)
hx /2 hx /2 hx

• It can be solved as   −1

1 1 1
a=
e + , (5.80)
2 apq ap+1,q
which is the harmonic average of apq and ap+1,q .
200 Chapter 5. Finite Element Methods for Elliptic Equations

5.8. Abstract Variational Problem

Let V be a normed space and consider the following abstract variational prob-
lem:
Find u ∈ V such that

a(u, v) = f (v), ∀v ∈ V, (5.81)

where a(·, ·) : V × V → R is a continuous bilinear form and f : V → R is a

continuous linear form.

Theorem 5.11. (Lax-Milgram Lemma) Suppose that V is a Hilbert

space with norm k · k. Let a(·, ·) : V × V → R is a continuous V -elliptic
bilinear form in the sense that

∃α s.t. αkvk2 ≤ a(v, v), ∀v ∈ V, (5.82)

and f : V → R, a continuous linear form. Then, the abstract variational

problem (5.81) has one and only one solution.
5.8. Abstract Variational Problem 201

Existence and uniqueness of the solution: Consider the Laplace

equation
−∆u = f x ∈ Ω,
(5.83)
u=0 x ∈ Γ = ∂Ω.
Then, using the Green’s formula, its variational problem is formulated as fol-
lows:

Find u ∈ V = H01 (Ω) such that

a(u, v) ≡ (∇u, ∇v) = (f, v) ≡ f (v), ∀v ∈ V. (5.84)

Here the Hilbert space

H01 (Ω) = {v : v ∇v are square-integrable and v|Γ = 0}

equipped with the norm k · k1 defined as

kvk21 = kvk20 + k∇vk20

202 Chapter 5. Finite Element Methods for Elliptic Equations

Theorem 5.12. The variational problem (5.84) has a unique solution.

Proof. Application of the Cauchy-Schwarz inequality shows that

|(∇u, ∇v)| ≤ k∇uk0 · k∇vk0 ≤ k∇uk1 · k∇vk1 ,

which implies that a(·, ·) is continuous on H01 (Ω) × H01 (Ω).

Using the Poincaré inequality,
ˆ ˆ
u2 dx ≤ C |∇u|2 dx, ∀v ∈ H01 (Ω), (5.85)
Ω Ω
or
kvk20 ≤ Ck∇vk20 = Ca(v, v),
we obtain
kvk20 + k∇vk20 ≤ (1 + C)k∇vk20 = (1 + C)a(v, v).
That is,
1
kvk21 ≤ a(v, v) (5.86)
1+C
which shows that a(·, ·) is V -elliptic. Hence, by the Lax-Milgram lemma, the
variational problem has a unique solution.
The V -ellipticity is sometimes said to be coercive.
5.9. Numerical Examples with Python 203

5.9. Numerical Examples with Python

A Python code is implemented for solving
−uxx = f, x ∈ (0, 1)
(5.87)
u = g, x = 0, 1,

using high-order Galerkin FE methods.

The exact solution is chosen as

u(x) = sin(πx) (5.88)

so that the right-hand side becomes

f (x, y) = π 2 sin(πx)

For various number of grid points nx and the order of basis functions k, the
maximum errors are found as in the table.

Table 5.1: The maximum error ku − uh k∞ .

k
nx 1 2 3 4
2 0.234 0.00739 0.000428 1.67e-05
4 0.053(2.14) 0.000562(3.72) 1.45e-05(4.88) 3.37e-07(5.63)
8 0.013(2.03) 3.67e-05(3.94) 4.61e-07(4.98) 5.58e-09(5.92)
16 0.00322(2.01) 2.31e-06(3.99) 1.45e-08(4.99) 8.84e-11(5.98)

The numbers in parentheses denote convergence rates. Note that super-

convergence is observed for k ≥ 2.
204 Chapter 5. Finite Element Methods for Elliptic Equations

The following shows the main routine FEM_1D_High_Order.py, the user pa-
rameter file USER_PARS.py, and the core functions for the construction of the
stiffness matrix.

## FEM_1D_High_Order.py
##-- read USER_PARS and util ---------
from USER_PARS import *
from util_FEM_1D import *

level = 2
print_USER_PARS(level)
from fem_1d import *

#------------------------------------
A = stiffness_mtx(level)
b = get_rhs(level)
dirichlet_BC(A)

ALU = mtx_banded_lu(A,level)
mtx_banded_lusol(ALU,b)

U = exact_sol(level)
print "L8-error = %.3g" %(max_difference(U,b))

## USER_PARS.py
##-----------------------
ax,bx = 0.,1.0;
nx = 20
poly_order = 3

## fem_1d.py
##-----------------------
def stiffness_mtx(level=0):
A = np.ndarray((row,col),float)
init_array(A)
for e in range (nx):
5.9. Numerical Examples with Python 205

g0,g1 = e*kpoly,(e+1)*kpoly
xl,xr = XG[e],XG[e+1]
E = element_stiffness(xl,xr,kpoly)
for i in range(kpoly+1):
for j in range(kpoly+1):
A[g0+i][kpoly+j-i] += E[i][j]
return A

def element_stiffness(xl,xr,kpoly):
m = kpoly+1
E = np.ndarray((m,m),float)
init_array(E)
XL,WT = local_points_weights(xl,xr,kpoly)
XT = get_XT(XL)
for i in range(m):
for j in range(m):
for l in range(m):
dphi_i_xl=eval_dphi(i,kpoly,XL[i],XL[l],XT)
dphi_j_xl=eval_dphi(j,kpoly,XL[j],XL[l],XT)
E[i][j]+=(dphi_i_xl*dphi_j_xl*WT[l])
return E
206 Chapter 5. Finite Element Methods for Elliptic Equations

5.10. Homework
1. Consider the model problem (5.1). Verify that the algebraic system from
the linear Galerkin method is equivalent to that of finite difference method
when the mesh is uniform, i.e.,

h = hi , i = 1, · · · , M + 1,

2. Prove (5.32) and (5.33). Hint: In each subinterval Ij = [xj−1 , xj ], the differ-
ence between u and its linear interpolant can be expressed as follows: for
x ∈ Ij ,
u00 (ξj )
u(x) − πh u(x) = (x − xj−1 )(x − xj ), for some ξj ∈ Ij .
2!
(See (1.9)on p.7.)
3. Let Ω = (0, 1)2 and Γ = ∂Ω and consider

−∇ · (a(x)∇u) = f, x ∈ Ω,
u = gD , x ∈ ΓD , (5.89)
aun = gN , x ∈ ΓN ,
where Γ = ΓD ∪ ΓN and ΓD and ΓN are distinct nonempty boundary por-
tions corresponding to the Dirichlet and Neumann boundary conditions,
respectively. Consider a FV method on a rectangular cells with cell-
centered nodal points, as considered in Section 5.6. Design to suggest
numerical methods for an effective treatment for each of the boundary
conditions. (You may assume gD = gN ≡ 0, if you want.)
4. Consider the following 1D elliptic problem of general form
−((1 + x2 )ux )x + 5ux = f, x ∈ (0, 1)
(5.90)
ux (0) = gN , u(1) = gD
Choose the exact solution as in (5.88):

u(x) = sin(πx)

and correspondingly the right side f and the boundary data, gN and gD .
(a) Formulate the Galerkin method for (5.90).
5.10. Homework 207

(b) Modify the Python code in §5.9 to solve the above problem.
(c) Carry out an error analysis as in Table 5.1.
5. Assume that v(x) ∈ C 1 [a, b] and v(a) = 0. Prove that the one-dimensional
Poincaré inequality
b−a
kvk0 ≤ √ kv 0 k0 . (5.91)
2
Hint: You may begin with
ˆ x ˆ x
0
v(x) = v(a) + v (t)dt = v 0 (t)dt.
a a

Thus, by the Cauchy-Schwarz inequality

ˆ x  ˆ x 1/2  ˆ x 1/2
0 0 2
|v(x)| ≤ |v |dt ≤ dt (v ) dt
√a a a (5.92)
≤ x − a kv 0 k0

Now, square the inequality and then integrate over the interval.
6. (Optional) Use the arguments in the proof of Homework 5.5 to prove the
Poincaré inequality (5.85) when Ω = (0, 1)2 :
ˆ ˆ
2
u dx ≤ C |∇u|2 dx, ∀v ∈ H01 (Ω), (5.93)
Ω Ω

for some C ˆ> 0. Try to ˆdetermine the constant ˆ 1as small as possible.
ˆ 1C
 1ˆ 1 
Note that f (x) dx = f (x, y) dxdy = f (x, y) dydx.
Ω 0 0 0 0
208 Chapter 5. Finite Element Methods for Elliptic Equations
Chapter 6

FD Methods for Hyperbolic Equations

This chapter considers finite difference methods for hyperbolic PDEs. We be-
gin with numerical methods for the linear scalar wave equation. Then, numer-
ical methods for conservation laws are treated along with nonlinear stability.
A Python code is included for the Lax-Wendroff scheme to solve the one-way
wave equation.

209
210 Chapter 6. FD Methods for Hyperbolic Equations

6.1. Introduction
Consider the initial value problem
ut + A ux = 0
(6.1)
u|t=0 = u0 (x),

where A = [aij ] ∈ Rm×m and u is a vector function of m components, m ≥ 1.

• The problem (6.1) is well-posed if and only if all eigenvalues of A are real
and there is a complete set of eigenvectors [27].
• Such a system is called (strongly) hyperbolic.
• We will restrict our discussions to such hyperbolic problems.
6.1. Introduction 211

Let {φ1 , · · · , φm } be the complete set of eigenvectors corresponding to the

eigenvalues {λ1 , · · · , λm }. Define a matrix

S = [φ1 , · · · , φm ], Γ = diag(λ1 , · · · , λm ).

Then, from linear algebra theory, we obtain

A = SΓS −1 . (6.2)

Apply S −1 to (6.1) to have

S −1 ut + Γ S −1 ux = 0
(6.3)
S −1 u|t=0 = S −1 u0 (x).

e = S −1 u. Then, (6.3) is reduced to the following m scalar equations

Let u
u
ei,t + λi u
ei,x = 0, i = 1, · · · , m,
(6.4)
ei |t=0 = u
u ei,0 (x).
212 Chapter 6. FD Methods for Hyperbolic Equations

Hence the chapter begins with discussions focusing on the scalar equation:
ut + aux = 0, (x, t) ∈ Ω × J,
(6.5)
u(x, 0) = u0 (x), x ∈ Ω, t = 0,

where Ω = (ax , bx ) ⊂ R and J = (0, T ], T > 0, the time interval. Here the
boundary condition is ignored for simplicity. (Or, we may assume Ω = R.)
When a is a constant, (6.5) has the exact solution

u(x, t) = u0 (x − at). (6.6)

6.2. Basic Difference Schemes 213

6.2. Basic Difference Schemes

We begin with our discussion of finite difference (FD) schemes for (6.5) by
defining grid points in the (x, t) plane.
Let ∆x and ∆t be the spatial and temporal grid sizes, respectively; then the
grid will be the points
(xm , tn ) = (m∆x, n∆t)
for integers m and n ≥ 0. For a function v defined either on the grid or for
n
continuously varying (x, t), we write vm for the value of v at (xm , tn ), i.e.,
n
vm = v(xm , tn ).

Let
S n := Ω × (tn−1 , tn ]
be the nth space-time slice. Suppose that the computation has been performed
for uj = {ujm }, 0 ≤ j ≤ n − 1. Then, the task is to compute un by integrating
the equation on the space-time slice S n , utilizing FD schemes.
214 Chapter 6. FD Methods for Hyperbolic Equations

The following presents examples of the forward-time (explicit) schemes for

(6.5):
n n−1 n−1 n−1
vm − vm vm − vm−1
(a) +a = 0,
∆t ∆x
n n−1 n−1 n−1
vm − vm vm+1 − vm
(b) +a = 0,
∆t ∆x
n n−1 n−1 n−1
vm − vm vm+1 − vm−1
(c) +a = 0, (6.7)
∆t 2∆x
n n−2 n−1 n−1
vm − vm vm+1 − vm−1
(d) +a = 0, (leapfrog)
2∆t 2∆x
n−1 n−1
n vm+1 +vm−1 n−1 n−1
vm − 2 vm+1 − vm−1
(e) +a = 0. (Lax-Friedrichs)
∆t 2∆x
These explicit schemes shall be exemplified in describing properties of nu-
merical methods.
6.2. Basic Difference Schemes 215

6.2.1. Consistency
The bottom line for accurate numerical methods is that the discretization be-
comes exact as the grid spacing tends to zero, which is the basis of consistency.
Recall the definition of consistency.
Definition 6.1. Given a PDE P u = f and a FD scheme P∆x,∆t u = f , the
FD scheme is said to be consistent with the PDE if for every smooth function
φ(x, t)
P φ − P∆x,∆t φ → 0 as (∆x, ∆t) → 0,
with the convergence being pointwise at each grid point.
Not all numerical methods based on Taylor series expansions are consis-
tent.
216 Chapter 6. FD Methods for Hyperbolic Equations

Example 6.2. The forward-time forward-space scheme is consistent.

Proof. For the one-way wave equation (6.5),
 
∂ ∂
Pφ ≡ +a φ = φt + aφx .
∂t ∂x
For the forward-time forward-space scheme (6.7b),
φnm − φn−1
m φn−1 n−1
m+1 − φm
P∆x,∆t φ = +a .
∆t ∆x
To find the truncation error of the numerical scheme, we begin with the Taylor
series in x and t about (xm , tn ):

∆t2
φnm= n−1
φm n−1
+ ∆t φt (xm , t ) + φtt (xm , tn−1 ) + O(∆t3 ),
2
∆x2
n−1 n−1 n−1
φm+1 = φm + ∆x φx (xm , t ) + φxx (xm , tn−1 ) + O(∆x3 ).
2
With some algebra, one can obtain
∆t ∆x
P∆x,∆t φ = φt + aφx + φtt + a φxx + O(∆x2 + ∆t2 ).
2 2
Thus, as (∆x, ∆t) → 0,
∆t ∆x
P φ − P∆x,∆t φ = − φtt − a φxx + O(∆x2 + ∆t2 ) → 0.
2 2
Therefore, the scheme is consistent.
6.2. Basic Difference Schemes 217

6.2.2. Convergence
A numerical method is said to be convergent if the solution of the FD scheme
tends to the exact solution of the PDE as the grid spacing tends to zero. We
redefine convergence in a formal way as follows:
Definition 6.3. A FD scheme approximating a PDE is said to be convergent
if
u(x, t) − unm → 0 as (xm , tn ) → (x, t) as (∆x, ∆t) → 0,
where u(x, t) is the exact solution of PDE and unm denotes the the solution of
the FD scheme.
Consistency implies that the truncation error

(P u − P∆x,∆t u) → 0

as ∆x and ∆t approach zero. So consistency is certainly necessary for con-

vergence. But as the following example shows, a numerical scheme may be
consistent but not convergent.
218 Chapter 6. FD Methods for Hyperbolic Equations

Figure 6.1: The characteristic curve passing the origin of the xt-plane.

Example 6.4. The forward-time forward-space scheme for (6.5) is not con-
vergent, when a > 0.
Proof. The scheme (6.7b) is consistent from Example 6.2. The problem (6.5)
has the exact solution
u(x, t) = u0 (x − at),
a shift of u0 by at. The lines having the slope 1/a in the xt-plane become
characteristics of the problem; when a > 0, the characteristic curve passing
the origin is shown in Figure 6.1.
On the other hand, the scheme (6.7b) can be rewritten as
n n−1 n−1 n−1 n−1 n−1
vm = vm − aλ(vm+1 − vm ) = (1 + aλ)vm − aλvm+1 , (6.8)

where λ = ∆t/∆x. Let the initial data be given


1, if x ≤ 0,
u0 (x) =
0, else.

Since it is natural for the scheme to take the initial data


0 1, if xm ≤ 0,
vm =
0, else,

it follows from (6.8) that

n
vm ≡ 0 ∀ m > 0, n ≥ 0.
6.2. Basic Difference Schemes 219

Figure 6.2: The forward-time forward-space scheme for ut + aux = 0, a > 0.

n
See Figure 6.2. The above holds for any choices of ∆x and ∆t. Therefore, vm
cannot converge to the exact solution u(x, t) in (6.6).
Showing that a given consistent scheme is convergent is not easy in gen-
eral, if attempted in a direct manner as in Homework 6.1. However, there is
a related concept, stability, that is easier to check.
220 Chapter 6. FD Methods for Hyperbolic Equations

6.2.3. Stability
Example 6.4 shows that consistency is not enough for a numerical method to
guarantee convergence of its solution to the exact solution. In order for a con-
sistent numerical scheme to be convergent, the required property is stability.
Recall the L2 -norm of grid function v:
 X∞ 1/2
2
kvk∆x = ∆x |vm | .
m=−∞

Definition 6.5. A FD scheme P∆x,∆t v = 0 for a homogeneous PDE P u = 0 is

stable if for any positive T , there is a constant CT such that
J
X
n
kv k∆x ≤ CT kv j k∆x , (6.9)
j=0

for 0 ≤ tn ≤ T and for ∆x and ∆t sufficiently small. Here J is chosen to

incorporate the data initialized on the first J + 1 levels.
6.2. Basic Difference Schemes 221

Example 6.6. The schemes (6.7a) and (6.7b) can be written of the form
n n−1 n−1
vm = αvm + βvm∓1 .

Then they are stable if |α| + |β| ≤ 1.

Proof. Indeed, for the scheme (6.7a),
∞
X ∞
X
n 2 n−1 n−1 2
|vm | = |αvm + βvm−1 |
m=−∞ m=−∞
∞
X
n−1 2 n−1 n−1 n−1 2
≤ |αvm | + 2|αβvm vm−1 | + |βvm−1 |
m=−∞
∞
X
≤ |α|2 |vm
n−1 2 n−1 2
| + |α||β|(|vm n−1 2
| + |vm−1 | ) + |β|2 |vm−1
n−1 2
|
m=−∞
∞
X
= (|α| + |β|)2 |vm
n−1 2
|.
m=−∞

Thus the scheme is stable if |α| + |β| = |1 − aλ| + |aλ| ≤ 1, where λ = ∆t/∆x.
Therefore, a sufficient condition for stability of (6.7a) is 0 ≤ aλ ≤ 1. The
analysis is similar for (6.7b); it is stable if −1 ≤ aλ ≤ 0.
The stability inequality (6.9) can be easily satisfied when

kv n k∆x ≤ (1 + C∆t)kv n−1 k∆x , (6.10)

for some C ≥ 0 independent on ∆t.

222 Chapter 6. FD Methods for Hyperbolic Equations

Theorem 6.7. (Lax-Richtmyer Equivalence Theorem). Given a well-

posed linear initial value problem and its FD approximation that satisfies the
consistency condition, stability is the necessary and sufficient condition for
convergence.
The above theorem is very useful and important. Providing convergence
is difficult for most problems. However, the determination of consistency of a
scheme is quite easy as shown in §6.2.1, and determining stability is also eas-
ier than showing convergence. Here we introduce the von Neumann analysis
of stability of FD schemes, which allows one to analyze stability much simpler
than a direct verification of (6.9).

The von Neumann analysis

A simple procedure of the von Neumann analysis reads
n
• Replace vm by g n eimϑ for each value of m and n.
• Find conditions on coefficients and grid spacings which would satisfy |g| ≤
1 + C∆t, for some C ≥ 0.
6.2. Basic Difference Schemes 223

The Courant-Friedrichs-Lewy (CFL) condition

The von Neumann analysis is not easy to utilize for rather general problems,
in particular, for nonlinear problems. In computational fluid dynamics
(CFD), a more popular concept is the so-called CFL condition.

Theorem 6.8. Given an explicit scheme for ut + aux = 0 of the form

n n−1 n−1 n−1
vm = αvm−1 + βvm + γvm+1

with λ = ∆t/∆x held constant, a necessary condition for stability is the

Courant-Friedrichs-Lewy (CFL) condition

|aλ| ≤ 1.

Proof. Let ∆t = 1/n, for some n ≥ 1. Then the physical domain of dependence
for the exact solution at the point (x, t) = (0, 1) must be (±a, 0), i.e.,
u(0, 1) = u0 (±a).

On the other hand, it follows from the FD scheme that the numerical solu-
0
tion v0n depends on vm , |m| ≤ n. Since
m∆x = m∆t/λ ≤ n∆t/λ = 1/λ,
we can see that the numerical solution at (0, 1), v0n , depends on x for |x| ≤ 1/λ.
Suppose |aλ| > 1. Then we have |a| > 1/λ. So v0n depends on x for
|x| ≤ 1/λ < |a|.
Thus v0n cannot converge to the exact value u(0, 1) = u0 (±a) as ∆x → 0 with
λ = ∆t/∆x keeping constant. This proves the theorem.

One can see from the above theorem and proof that
stability requires the numerical domain of dependence contain the physical
domain of dependence.
This physical observation is very useful for stability analysis for certain
nonlinear problems [40].
224 Chapter 6. FD Methods for Hyperbolic Equations

6.2.4. Accuracy
We define the order of accuracy for numerical schemes for PDEs.
Definition 6.9. (Order of accuracy). Let P∆x,∆t u = R∆x,∆t f be a numerical
scheme for P u = f . Assume that for every smooth function φ,

P∆x,∆t φ = R∆x,∆t (P φ) + O(∆xp ) + O(∆tq ).

Then, the scheme is said to have the p-th order accuracy in space and the q-th
order accuracy in time, and denoted by the “accuracy order (p, q) in space-
time".
For example, the forward-time forward-space, forward-time central-space,
and leapfrog schemes for (6.5) have the accuracy orders (1, 1), (2, 1), and (2, 2)
in space-time, respectively.
6.2. Basic Difference Schemes 225

Crank-Nicolson (CN) scheme: Consider the one-way wave equation with a

source term
ut + aux = f. (6.11)
The scheme is based on central differences about (x, tn−1/2 ), where tn−1/2 =
(tn−1 + tn )/2. Since
unm − um n−1
n−1/2
ut (xm , t ) = + O(∆t2 ),
∆t
ux (xm , tn ) + ux (xm , tn−1 )
ux (xm , tn−1/2 ) = + O(∆t2 )
2
1 unm+1 − unm−1 un−1 − n−1 

u
= + m+1 m−1
+ O(∆x2 ) + O(∆t2 ),
2 2∆x 2∆x
n n−1
fm + fm
f (xm , tn−1/2 ) = + O(∆t2 ),
2
we obtain the CN scheme
n n−1
 n n n−1 n−1 
vm − vm a vm+1 − vm−1 vm+1 − vm−1 n−1
f n + fm
+ + = m , (6.12)
∆t 2 2∆x 2∆x 2
where the truncation error is

O(∆x2 ) + O(∆t2 ).

Thus the CN scheme has the accuracy order (2, 2).

226 Chapter 6. FD Methods for Hyperbolic Equations

It follows from the von Neumann analysis presented in §6.2.3 that the am-
plification factor for the CN scheme is
1 − i aλ
2 sin ϑ ∆t
g(ϑ) = , λ= .
1 + i aλ
2 sin ϑ
∆x

Thus its magnitude is identically one and therefore the CN scheme is stable
for every choice of ∆x and ∆t (unconditional stability).
Note: The numerical solution of the CN method (6.12) may involve oscilla-
tions when the initial data is nonsmooth.

For a wide range of PDEs, the CN scheme is unconditionally stable and

of a second-order accuracy in both space and time. These two advantageous
properties have made the scheme quite popular.
6.3. Conservation Laws 227

6.3. Conservation Laws

The conservation laws in one-dimensional (1D) space have the form

∂ ∂
u(x, t) + f (u(x, t)) = 0. (6.13)
∂t ∂x
Here
u : R × R → Rm
and f : Rm → Rm is called the flux function. For simplicity, we may consider
the pure initial value problem, or Cauchy problem, in which (6.13) holds for
−∞ < x < ∞ and t ≥ 0. In this case we must specify initial conditions only

u(x, 0) = u0 (x), −∞ < x < ∞. (6.14)

We assume that the system (6.13) is hyperbolic. That is, the Jacobian ma-
trix f 0 (u) of the flux function is

• of real eigenvalues, and

• diagonalizable, i.e., there is a complete set of m linearly independent
eigenvectors.

In 2D, a system of conservation laws can be written as

ut + f (u)x + g(u)y = 0, (6.15)

where
u : R2 × R → Rm , f, g : Rm → Rm .

6.3.1. Euler equations of gas dynamics

Consider “a tube" where properties of the gas such as density and velocity are
assumed to be constant across each cross section of the tube. Let ρ(x, t) and
v(x, t) be respectively the density and the velocity at point x and time t. Then
ˆ x2
mass in [x1 , x2 ] at time t = ρ(x, t)dx.
x1
228 Chapter 6. FD Methods for Hyperbolic Equations

Assume that the walls of the tube are impermeable and that mass is neither
created nor destroyed. Then the mass in a section [x1 , x2 ] can change only
because of gas flowing across the end points x1 and x2 . The rate of flow, or flux
of gas at (x, t) is given by

mass flux at (x, t) = ρ(x, t) v(x, t).

Thus, the change rate of mass in [x1 , x2 ] is

ˆ
d x2
ρ(x, t)dx = ρ(x1 , t) v(x1 , t) − ρ(x2 , t) v(x2 , t), (6.16)
dt x1

which is one integral form of conservation law.

6.3. Conservation Laws 229

Integrate (6.16) in time from t1 to t2 to have

ˆ x2 ˆ x2
ρ(x, t2 )dx = ρ(x, t1 )dx
x1 xˆ ˆ
1
t2 t2 (6.17)
+ ρ(x1 , t) v(x1 , t)dt − ρ(x2 , t) v(x2 , t)dt.
t1 t1

This is another integral form of conservation law.

Geometric interpretation for (6.17):
230 Chapter 6. FD Methods for Hyperbolic Equations

Derivation of differential form: Now, assume ρ and v are differentiable.

Since ˆ t2
∂
ρ(x, t2 ) − ρ(x, t1 ) = ρ(x, t) dt,
t1 ˆ ∂t
x2
∂
ρ(x2 , t) v(x2 , t) − ρ(x1 , t) v(x1 , t) = (ρ(x, t) v(x, t)) dx,
x1 ∂x
the equation (6.17) reads
ˆ t 2 ˆ x2 h
∂ ∂ i
ρ(x, t) + (ρ(x, t) v(x, t)) dx dt = 0. (6.18)
t1 x1 ∂t ∂x
Since this must hold for any section [x1 , x2 ] and for any time interval [t1 , t2 ],
the integrand in (6.18) must be identically zero, i.e.,

ρt + (ρv)x = 0. (conservation of mass) (6.19)

6.3. Conservation Laws 231

Euler equations of gas dynamics:

ρt + (ρv)x = 0, (conservation of mass)
(ρv)t + (ρv 2 + p)x = 0, (conservation of momentum) (6.20)
Et + (v(E + p))x = 0. (conservation of energy)

The rule of thumb (in the derivation of conservation laws) is that

• For any quantity z which is advected with the flow will have a contribu-
tion to the flux of the form zv.
• Besides advection, there are forces on the fluid that cause acceleration
due to Newton’s laws. Since we assume there is no outside forces, the
only force is due to variations in the fluid itself; it is proportional to the
pressure gradient for momentum and proportional to the gradient of vp
for energy.
232 Chapter 6. FD Methods for Hyperbolic Equations

The pressure variable can be replaced by additional equations of physics,

called the state equations. For gases,
1
E = ρv 2 + ρe, (total energy)
2
p = R ρ T, (pressure: ideal gas law)
e = cv T, (specific internal energy: polytropic gas)
h = e + p/ρ = cp T, (enthalpy: polytropic gas)
γ = cp /cv , (ratio of specific heat)
R = cp − cv . (polytropic gas)

The polytropic gas is such that the internal energy is proportional to the tem-
perature, so the coefficients cv and cp are constants, called respectively the
specific heat at constant volume and the specific heat at constant pressure. (In
general, “specific" means “per unit mass".)
6.3. Conservation Laws 233

The equation of state for a polytropic gas: Note that T = p/(Rρ) so that
cv p cv p 1 p
e = cv T = = = .
R ρ cp − cv ρ γ − 1 ρ
Thus the equation of state for a polytropic gas is
p 1
E= + ρv 2 . (6.21)
γ−1 2

Isothermal flow: Assume the temperature is constant through the tube.

Then, from the ideal gas law,

p = R ρ T = a2 ρ,
√
where a = RT is the sound speed. Thus the isothermal equations read
   
ρ ρv
+ = 0. (6.22)
ρv t ρv 2 + a2 ρ x
234 Chapter 6. FD Methods for Hyperbolic Equations

6.4. Shocks and Rarefaction

6.4.1. Characteristics
Consider the linear advection equation
ut + aux = 0,
(6.23)
u(x, 0) = u0 (x).

The exact solution is simply

u(x, t) = u0 (x − at), t ≥ 0.

The solution is constant along each ray x − at = x0 . Such rays are known as
the characteristics of the equation.
Note that the characteristics are curves in the x-t plane satisfying the ODE
x0 (t) = a, x(0) = x0 . Let us differentiate u(x, t) along one of these curves to find
the change rate of the solution along the characteristics:
d ∂ ∂
u(x, t) = u(x, t) + u(x, t)x0 = ut + aux = 0,
dt ∂t ∂x
which confirms that u is constant along the characteristics.
There is a fundamental property of linear hyperbolic equations: singulari-
ties propagate only along characteristics.
6.4. Shocks and Rarefaction 235

Nonsmooth data: We consider the so-called vanishing-viscosity approach.

Let uε be the solution of
ut + aux = εuxx . (6.24)
Then uε is smooth for t > 0 even if u0 is not smooth, because it is the solution
of a parabolic equation.
Note that (6.24) simplifies if we make a change of variables to follow the
characteristics:
v ε (x, t) = uε (x + at, t).
Then v ε satisfies the heat equation

vtε (x, t) = εvxx

ε
(x, t).

Thus, after solving the heat equation, we can compute uε (x, t) = v ε (x − at, t)
explicitly. It is easy to verify that the vanishing-viscosity solution is equal to
u0 (x − at):
lim uε (x, t) = u(x, t) = u0 (x − at).
ε→0
236 Chapter 6. FD Methods for Hyperbolic Equations

6.4.2. Weak solutions

A natural way to define a generalized solution of the inviscid equation that
does not require differentiability is to go back to the integral form of the con-
servation law. We say u(x, t) is a generalized solution if (6.17) is satisfied for
all x1 , x2 , t1 , and t2 .
There is another approach that results in a different integral formulation
that is often more convenient to work with.
Let φ ∈ C01 (R × R+ ). Multiply ut + f (u)x = 0 by φ and integrate over space
and time to have ˆ ∞ˆ ∞
[φut + φf (u)x ] dx dt = 0.
0 −∞
Using integration by parts gives
ˆ ∞ˆ ∞ ˆ ∞
[φt u + φx f (u)] dx dt = − φ(x, 0)u(x, 0) dx. (6.25)
0 −∞ −∞

Definition 6.10. The function u(x, t) is called a weak solution of ut +f (u)x = 0

if (6.25) holds for all φ ∈ C01 (R × R+ ).
6.4. Shocks and Rarefaction 237

Known facts:

• Any weak solution satisfies the original integral conservation law.

• The vanishing-viscosity generalized solution is a weak solution.
• For nonlinear problems, weak solutions are often not unique, and there-
fore an additional problem is often considered to identify which weak so-
lution is the physically correct vanishing-viscosity solution.
• There are other conditions to avoid working with the viscous equation di-
rectly. They are usually called the entropy conditions. Thus the vanishing-
viscosity solution is also called the entropy solution.
238 Chapter 6. FD Methods for Hyperbolic Equations

6.5. Numerical Methods

6.5.1. Modified equations
In this subsection, we briefly review accuracy and stability for the Riemann
problem of the linear advection equation:
ut + aux = 0, x ∈ R, t ≥ 0,
1, x < 0, (6.26)
u0 (x) =
0, x > 0.

The exact solution is given

u(x, t) = u0 (x − at). (6.27)

Consider the following numerical schemes:

Ujn+1 − Ujn n
Ujn − Uj−1
+a = 0, (explicit one-sided)
k h
n n
Uj+1 +Uj−1
Ujn+1 − 2
n
Uj+1 n
− Uj−1
+a = 0, (Lax-Friedrichs)
k 2h (6.28)
Ujn+1 − Ujn n
Uj+1 n
− Uj−1
+a
k 2h
k 2 Uj+1 − 2Ujn + Uj−1
n n
− a = 0. (Lax-Wendroff)
2 h2
6.5. Numerical Methods 239

Lax-Wendroff scheme: Note that

n
Ujn+1 − Ujn k k2
ut (xj , t ) = − utt − uttt − · · · .
k 2 6
Since
ut = −aux ,
we have
utt = (ut )t = (−aux )t = −auxt = −autx
= −a(ut )x = −a(−aux )x = a2 uxx
Therefore, the Lax-Wendroff scheme can be obtained by taking care of utt =
a2 uxx by the central scheme; its truncation error is
k2 h2 k2 3 h2
− uttt − a uxxx + · · · = a uxxx − a uxxx + · · ·
6 6 6 6
2  2
h k 2 
= a 2 a − 1 uxxx + · · ·
6 h

Thus, when h and k are sufficiently small, solving (6.26) by the Lax-Wendroff
scheme is equivalent to solving the following equation exactly:
h2  k 2 2 
ut + aux = a 2 a − 1 uxxx . (6.29)
6 h

Equation (6.29) is called the modified equation of (6.26) for the Lax-Wendroff
scheme. By analyzing (6.29) in PDE sense, one can understand the Lax-
Wendroff scheme.
240 Chapter 6. FD Methods for Hyperbolic Equations

Finite difference equation was introduced in the first place because it is eas-
ier to solve than a PDE; on the other hand, it is often easier to predict qualita-
tive behavior of a PDE than difference equations.
Dispersion analysis: Equation (6.29) is a dispersive equation of the form

ut + aux = µuxxx . (6.30)

To look at a Fourier series solution to this equation, take u(x, t) as

ˆ ∞
u(x, t) = b(ξ, t)eiξx dξ,
u
−∞

where ξ is the wave number. Here the purpose is to see that the Fourier
components with different wave number ξ propagate at different speeds (dis-
persion).
Due to linearity, it suffices to consider each wave number in isolation, so
suppose that we look for solution of (6.30) of the form

u(x, t) = ei(ξx−ct) , (6.31)

where c = c(ξ) is called the frequency. Plugging this into (6.30) gives

c(ξ) = aξ + µ ξ 3 . (6.32)

This expression is called the dispersion relation for (6.30).

6.5. Numerical Methods 241

Define
cp (ξ) = c(ξ)/ξ, (phase velocity)
cg (ξ) = c0 (ξ). (group velocity)
The phase velocity is the speed of wave peaks or in single frequency, while the
group velocity is the speed of energy in wavetrain.
Then, for the modified equation of Lax-Friedrichs scheme in (6.29), we have

cp = a + µ ξ 2 , cg = a + 3µ ξ 2 . (6.33)
Recall that the CFL condition reads

|aλ| = |ak/h| ≤ 1.

Thus, when the Lax-Friedrichs scheme is stable, the coefficient µ for (6.29)
must be nonpositive, i.e.,
h2  k 2 2 
µ = a 2 a − 1 ≤ 0, (6.34)
6 h
which implies from (6.33) that both the phase velocity and the group velocity
are smaller than the actual velocity a.
242 Chapter 6. FD Methods for Hyperbolic Equations

Remarks:

• For the step function in (6.26), the Fourier spectrum decays only as

ub0 (ξ) = O(1/ξ), as |ξ| → ∞.

(For smooth solutions, its Fourier spectrum decays exponentially.)

• Thus for the Lax-Wendroff scheme, dispersion becomes visible near

x = cg t.

(although the scheme satisfies the stability condition.)

• The numerical solution is oscillatory in the upstream (behind).
6.5. Numerical Methods 243

Beam-Warming scheme: This method is one-sided second-order veni-

son of the Lax-Wendroff scheme:
Ujn+1 − Ujn 3Ujn − 4Uj−1
n n
+ Uj−2
+a
k 2h (6.35)
k 2 Ujn − 2Uj−1
n n
+ Uj−2
− a = 0. (Beam-Warming)
2 h2
Then the associated modified equation reads
h2  3k k2 
ut + aux = µuxxx , µ = a 2 − a + 2 a2 . (6.36)
6 h h
Remarks:

• Since µ > 0 for sufficiently small k, the group velocity will be larger than
the actual speed a; there must be oscillation propagating faster than the
shock speed.
• Here the point is that a upwind modification is not sufficient enough to
cure oscillation.
244 Chapter 6. FD Methods for Hyperbolic Equations

Upwind (one-sided) scheme: For the explicit one-sided scheme in

(6.28), one can find its modified equation as
1  k 
ut + aux = ε uxx , ε = ha 1 − a . (6.37)
2 h
Note that the stability requires ε ≥ 0. This is a heat equation; the solution
must be diffusive.
When the dispersion analysis is applied for (6.37), the dispersion relation
is complex-valued as
c(ξ) = aξ − iεξ 2 .
It is not appropriate to analyze dispersive behavior of the solution. What we
can claim is that the solution is diffusive.
6.5. Numerical Methods 245

6.5.2. Conservative methods

Consider the Burgers’s equation in conservation form:
 u2 
ut + = 0. (6.38)
2 x
It can be rewritten in advection form

ut + uux = 0. (6.39)

When we consider the advection form, a natural (explicit) numerical scheme

reads
Ujn+1 − Ujn U n − Uj−1
n j
n
+ Uj = 0. (6.40)
k h
When e.g. the initial value is given as

0 1, j < 0,
Uj =
0, j ≥ 0,

one can easily verify that

Uj1 = Uj0 , ∀ j.
For other initial values, the scheme easily involves a large error in the shock
speed. Why? Answer: It is not conservative.
246 Chapter 6. FD Methods for Hyperbolic Equations

Conservative methods: Consider the following conservative form of

conservation law
ut + f (u)x = 0. (6.41)
Its simple and natural numerical method can be formulated as
Ujn+1 − Ujn n
F (Uj−p n
, Uj−p+1 n
, · · · , Uj+q n
) − F (Uj−p−1 n
, Uj−p+1 n
, · · · , Uj+q−1 )
+ = 0,
k h
(6.42)
for some F of p + q + 1 arguments, called the numerical flux function.
In the simplest case, p = 0 and q = 1. Then, (6.42) becomes

k
Ujn+1 = Ujn − [F (Ujn , Uj+1
n n
) − F (Uj−1 , Ujn )]. (6.43)
h
6.5. Numerical Methods 247

The above numerical scheme is very natural if we view Ujn as an approxi-

mation of the cell average unj ,
ˆ
n 1 xj+1/2
uj = u(x, tn )dx.
h xj−1/2

Consider the integral form of the conservation law (6.17),

ˆ xj+1/2 ˆ xj+1/2
n+1
u(x, t )dx = u(x, tn )dx
xj−1/2 x
ˆ tn+1 j−1/2 ˆ tn+1 (6.44)
+ f (u(xj−1/2 , t))dt − f (u(xj+1/2 , t))dt.
tn tn

Then, dividing by h, we have

ˆ tn+1 ˆ tn+1 
1
un+1
j = unj − f (u(xj+1/2 , t))dt − f (u(xj−1/2 , t))dt . (6.45)
h tn tn

Comparing this with (6.43), we can see that the numerical flux F (Ujn , Uj+1 n
)
n n+1
plays the role of an average flux at x = xj+1/2 over the time interval [t , t ]:

ˆ tn+1
1
F (Ujn , Uj+1
n
) ≈ f (u(xj+1/2 , t))dt. (6.46)
k tn

The Godunov’s method is based on this approximation, assuming that the

solution is piecewise constant on each cell (xj−1/2 , xj+1/2 ).
248 Chapter 6. FD Methods for Hyperbolic Equations

Upwind scheme: For the Burgers’s equation (6.38), the upwind scheme in
conservative form reads
n+1 n k h 1 n 2 1 n 2i
Uj = Uj − (U ) − (Uj−1 ) , (6.47)
h 2 j 2
where
1
F (Ujn , Uj+1
n
) = (Ujn )2 .
2
Lax-Friedrichs scheme: The generalization of the Lax-Friedrichs scheme
to the conservation law takes the form
n+1 1 n n kh n n
i
Uj = (Uj−1 + Uj+1 ) − f (Uj+1 ) − f (Uj−1 ) , (6.48)
2 2h
which can be rewritten in the conservation form by taking
h n 1
F (Ujn , Uj+1
n
)= n
(Uj − Uj+1 ) + (f (Ujn ) + f (Uj+1
n
)). (6.49)
2k 2
6.5. Numerical Methods 249

6.5.3. Consistency
The numerical method (6.43) is said to be consistent with the original conser-
vation law if the numerical flux F reduces to the true flux f for the constant
flow. That is, if u(x, t) ≡ u
b, say, then we expect

F (b
u, u
b) = f (b
u), ∀u
b ∈ R. (6.50)

b if there is a constant K ≥ 0 (which

We say F is Lipschitz continuous at u
may depend on u
b) such that

|F (v, w) − f (b
u)| ≤ K max(|v − u
b|, |w − u
b|).

Note that the Lipschitz continuity is sufficient for consistency.

250 Chapter 6. FD Methods for Hyperbolic Equations

6.5.4. Godunov’s method

k
Ujn+1 = Ujn − [F (Ujn , Uj+1
n n
) − F (Uj−1 , Ujn )], (6.51)
h
where
ˆ tn+1
1
F (Ujn , Uj+1
n
) ≈ u(xj+1/2 , t))dt = f (u∗ (Ujn , Uj+1
f (e n
)). (6.52)
k tn

Here

• u
e(x, t) is the piecewise constant representation of the solution, over the
grid cell (xj−1/2 , xj+1/2 ).
• u∗ (Ujn , Uj+1
n
) is the Riemann solution on {xj+1/2 } × [tn , tn+1 ].
• The method is consistent.
• Stability of the method requires to choose k small enough to satisfy
k
σ= max |f 0 (Ujn )| ≤ 1,
h j
where σ is called the Courant number.
6.6. Nonlinear Stability 251

6.6. Nonlinear Stability

To guarantee convergence, we need some form of stability, just as for linear
problems. Unfortunately, the Lax-Richtmyer Equivalence Theorem no longer
holds and we cannot use the same approach to prove convergence. In this sec-
tion, we will consider one form of nonlinear stability that allows us to prove
convergence results for a wide class of practical problems. So far, this ap-
proach has been completely successful only for scalar problems. For general
systems of equations with arbitrary initial data, no numerical method has
been prove to be stable or convergent, although convergence results have been
obtained in some special cases.
252 Chapter 6. FD Methods for Hyperbolic Equations

6.6.1. Total variation stability (TV-stability)

We first define the total variation (TV) over [0, T ] by
ˆ ˆ
1 T ∞
T VT (u) = lim sup |u(x + ε, t) − u(x, t)| dx dt
ε→0 ε 0ˆ −∞ ˆ (6.53)
1 T ∞
+ lim sup |u(x, t + ε) − u(x, t)| dx dt.
ε→0 ε 0 −∞
Define ˆ ˆ ˆ
T T ∞
kvk1,T = kvk1 dt = |v(x, t)| dx dt
0 0 −∞
and

K = {u ∈ L1,T : T VT (u) ≤ R and Supp(u(·, t)) ⊂ [−M, M ], ∀ t ∈ [0, T ]}. (6.54)

When we consider numerical solution U = {Ujn }, piecewise constant, then

T /k ∞ h i
X X
n n n+1 n
T VT (U ) = k|Uj+1 − Uj | + h|Uj − Uj |
n=0 j=−∞
T /k h
(6.55)
X i
= k T V (U n ) + kUjn+1 − Ujn k1 .
n=0
6.6. Nonlinear Stability 253

Definition 6.11. We will say that a numerical method is total variation

stable (TV-stable), if all approximations Uk for k < k0 lie in some fixed set of
the form (6.54) (where R and M may depend on the initial data u0 and the flux
function f (u), but not on k).
Theorem 6.12. Consider a conservative method with a Lipschitz continuous
numerical flux F (U ; j). Suppose that for each initial data u0 , there exists some
k0 , R > 0 such that

T V (U n ) ≤ R, ∀ n, k with k < k0 , nk ≤ T. (6.56)

Then, the method is TV-stable.

Theorem 6.13. Suppose Uk is generated by a numerical method in conser-
vation form with Lipschitz continuous numerical flux, consistent with some
scalar conservation law. If the method is TV-stable, then it is convergent in
the following sense
dist(Uk , W) → 0, as k → 0, (6.57)
where W = {w : w(x, t) is a weak solution}.
254 Chapter 6. FD Methods for Hyperbolic Equations

6.6.2. Total variation diminishing (TVD) methods

We have just seen that TV-stability of a consistent and conservative numer-
ical method is enough to guarantee convergence, in the sense in (6.57). One
easy way to ensure TV-stability is to require that the TV be nonincreasing as
time evolves, so that the TV at any time is uniformly bounded by the TV of
the initial data. This requirement gives rise to the very important class of
methods.
Definition 6.14. The numerical method Ujn+1 = H(U n ; j) is called total vari-
ation diminishing (TVD) if

T V (U n+1 ) ≤ T V (U n ) (6.58)

for all grid functions U n .

It can be shown that the true solution to the scalar conservation law has
this TVD property, i.e., any weak solution u(x, t) satisfies

T V (u(·, t2 )) ≤ T V (u(·, t1 )) for t2 ≥ t1 . (6.59)

Thus it is reasonable to impose TVD on the numerical solution as well, yield-

ing a TV-stability and hence convergence method.
6.6. Nonlinear Stability 255

6.6.3. Other nonoscillatory methods

Monotonicity preserving methods : A method is monotonicity preserving
if U n , n ≥ 1, are monotone for a monotone initial data u0 .
Theorem 6.15. Any TVD method is monotonicity preserving.
Another attractive feature of the TVD requirement is that it is possible to
derive methods with a high order of accuracy that are TVD. By contrast, if we
define “stability" by mimicking certain other properties of the true solution,
we find that accuracy is limited to first order. Nevertheless, we introduce
some of these other concepts, because they are useful and frequently seen in
the literature.
256 Chapter 6. FD Methods for Hyperbolic Equations

l1 -contracting methods : Any weak solution of a scalar conservation law

satisfies
ku(·, t2 )k1 ≤ ku(·, t1 )k1 , for t2 ≥ t1 . (6.60)
More generally: If u and v are both entropy solutions of the same conservation
law (but possibly with different data), and if u0 − v0 has compact support, then

ku(·, t2 ) − v(·, t2 )k1 ≤ ku(·, t1 ) − v(·, t1 )k1 , for t2 ≥ t1 . (6.61)

This property is called L1 -contraction. In discrete space l1 , for grid functions
U = {Uj } we define the l1 -norm by
∞
X
kU k1 = h |Uj |.
j=−∞

In analogy to the L1 -contraction property (6.61) of the true solution operator,

we say that a numerical method

Ujn+1 = H(U n ; j) (6.62)

is l1 -contracting if any two grid functions U n and V n for which U n − V n has

compact support satisfy

kU n+1 − V n+1 k1 ≤ kU n − V n k1 . (6.63)

6.6. Nonlinear Stability 257

Theorem 6.16. Any l1 -contracting numerical method is TVD.

Proof. The proof depends on the following important relation between the
1-norm and TV: Given any grid function U , define V by shifting U as

Vj = Uj−1 , ∀ j.

Then
1
kU − V k1 .
T V (U ) =
h
Now, suppose the method (6.62) is l1 -contracting. Define Vjn = Uj−1
n
. Note that
the methods under consideration are translation invariant, i.e.,

Vjn+1 = H(V n ; j).

Thus l1 -contraction implies

1 n+1
T V (U n+1 ) = kU − V n+1 k1
h
1
≤ kU n − V n k1
h
= T V (U n )

and hence the method is TVD.

Example 6.17. The upwind method is l1 -contracting and therefore TVD,
provided the CFL condition is satisfied.
258 Chapter 6. FD Methods for Hyperbolic Equations

Monotone methods : Another useful property of the entropy-satisfying

weak solution is as following: If we take two sets of initial data u0 and v0 , with

v0 (x) ≥ u0 (x), ∀ x,

then the respective entropy solutions u and v satisfy

v(x, t) ≥ u(x, t), ∀ x, t. (6.64)

The numerical method Ujn+1 = H(U n ; j) is called a monotone method if

Vjn ≥ Ujn ⇒ Vjn+1 ≥ Ujn+1 , ∀ j. (6.65)

To prove that a method is monotone, it suffices to check that

∂
H(U n ; j) ≥ 0, ∀ i, j, U n . (6.66)
∂Uin

This means that if we increase the value of any Uin then the value of Ujn+1
cannot decrease as a result.
Example 6.18. The Lax-Friedrichs scheme (6.48) (See page 248) is monotone
provided that the CFL condition is satisfied, because

n 1 n n kh n n
i
H(U ; j) = (Uj−1 + Uj+1 ) − f (Uj+1 ) − f (Uj−1 )
2 2h
satisfies
1 k 0 n 
 

 1 + f (Uj−1 ) , i = j − 1,
∂

 2 h
n 1 k 0 n 
H(U ; j) =

∂Uin 1 − f (Uj+1 ) , i = j + 1,
 2 h



0, otherwise.
6.6. Nonlinear Stability 259

Theorem 6.19. Any monotone method is l1 -contracting.

To summarize the relation between the different types of methods consid-
ered above, we have

monotone ⇒ l1 -contracting ⇒ TVD

⇒ monotonicity preserving

Theorem 6.20. A monotone method is at most first-order accurate.

Theorem 6.21. The numerical solution computed with a consistent mono-
tone method with k/h fixed converges to the entropy solution as k → 0.
Note that the numerical solution by a TVD method converges to a weak
solution that may not be the entropy solution. However, the notion of TV-
stability is much more useful, because it is possible to derive TVD methods
that have better than first-order accuracy.
We close the chapter with the following well-known theorem:
Theorem 6.22. (Godunov). A linear, monotonicity preserving method is at
most first-order accurate.
260 Chapter 6. FD Methods for Hyperbolic Equations

6.7. Numerical Examples with Python

A Python code is implemented for the Lax-Wendroff scheme in (6.28), for solv-
ing
ut + aux =0, (x, t) ∈ (−1, 6) × (0, 2]
1, x ∈ [0, 2] (6.67)
u(x, 0) =
0, elsewhere,
where a = 1.

Figure 6.3: The Lax-Wendroff scheme: (left) The initial solution and (right)
the solution at t = 2.
6.7. Numerical Examples with Python 261

The following shows the main routine lax_wendroff.py:

def lax_wendroff(U0,ax,bx,nx,T,nt,a,level=0):
hx,ht = (bx-ax)/nx, T/nt
if level>=1:
print("Lax-Wendroff: a=%g, nx=%d, nt=%d, hx=%g, ht=%g")\
%(a,nx,nt,hx,ht)

U =np.ndarray((2,nx+1),float)
for i in range(nx+1):
U[0][i]=U0[i]; U[1][i]=0.

alam = a*ht/hx
alam2= alam**2
for n in range(0,nt):
id0,id1 = n%2,(n+1)%2
for j in range (1,nx):
U[id1][j]=U[id0][j]-(alam/2.)*(U[id0][j+1]-U[id0][j-1
+(alam2/2.)*(U[id0][j+1]-2.*U[id0][j]+U[id0][j-1
return U[id1]
262 Chapter 6. FD Methods for Hyperbolic Equations

6.8. Homework
1. Find conditions on a and λ with which the FD schemes in (6.7.a)-(6.7.c)
are stable or unstable.
2. Consider the leapfrog scheme (6.7.d).
(a) Derive the relation
∞
X
n+1 2 n 2 n+1 n n+1 n
|vm | + |vm | + aλ(vm vm+1 − vm+1 vm )
m=−∞
∞
X
n 2 n−1 2 n n−1 n n−1
= |vm | + |vm | + aλ(vm vm+1 − vm+1 vm )
m=−∞
∞
X
1 2 0 2 1 0 1 0
= |vm | + |vm | + aλ(vm vm+1 − vm+1 vm )
m=−∞

n+1 n−1
(Hint: Multiply the leapfrog scheme by vm + vm and sum over all
m.)
(b) Show that
∞
X ∞
X
n+1 2 n 2 1 2 0 2
(1 − |aλ|) |vm | + |vm | ≤ (1 + |aλ|) |vm | + |vm |.
m=−∞ m=−∞

(Hint: Use the inequality − 21 (x2 + y 2 ) ≤ xy ≤ 12 (x2 + y 2 ).)

(c) Conclude the scheme is stable if |aλ| < 1.
3. Consider finite difference schemes of the form
n+1 n n
vm = αvm+1 + βvm−1 .

(a) Show that they are stable if |α| + |β| ≤ 1.

(Use the arguments as in Example 6.6 rather than the Von Neumann
analysis.)
(b) Conclude that the Lax-Friedrichs scheme (6.7.e) is stable if |aλ| ≤ 1,
where λ = k/h.
4. Verify the modified equation of the Beam-Warming scheme presented in
(6.36).
6.8. Homework 263

5. Derive the conservation form for the Lax-Friedrichs scheme applied to

the conservation law and presented in (6.48). (Use (6.49).)
6. Modify the Python code in § 6.7 to solve the one-way wave equation (6.67)
by the Beam-Warming scheme (6.35).
264 Chapter 6. FD Methods for Hyperbolic Equations
Chapter 7

Domain Decomposition Methods

The development of high-performance parallel computers has promoted the

effort to search for new efficient parallel algorithms for scientific computa-
tion rather than parallelize existing sequential algorithms. In the last two
decades, domain decomposition (DD) methods have been studied extensively
for the numerical solution of PDEs.

265
266 Chapter 7. Domain Decomposition Methods

7.1. Introduction to DDMs

The earliest DD method for elliptic problems is the alternating method discov-
ered by Hermann A. Schwarz in 1869 [60], so it is called Schwarz alternating
method (SAM).
Schwarz used the method to establish the existence of harmonic functions
on the nonsmooth domains that were constructed as a union of regions where
the existence could be established by some other methods; see Figure 7.1.

Figure 7.1: The domain used by Schwarz to show the existence of harmonic
solutions on irregular domains.
7.1. Introduction to DDMs 267

• Indeed, for a given initial value, SAM provided a convergent sequence

with a limit that is the harmonic function satisfying the given boundary
condition.
• Each iteration of the method consists of two fractional steps.

– In the first step, the previous approximation on Ω1 is replaced by the

e12 (:= ∂Ω1 ∩ Ω2 ) is
harmonic function for which the Dirichlet data on Γ
given by the previous approximation on Ω2 .
– The second step, in which new approximation is obtained on Ω2 , is
carried out similarly.
• Therefore, an arbitrarily accurate approximation of the harmonic func-
tion in the domain Ω1 ∪ Ω2 can be computed by using only solvers for
circles and rectangles. The method of separation of variables can be used
for the solution of these subdomains.
268 Chapter 7. Domain Decomposition Methods

SAM: Historical Backgrounds

• SAM offers a process that can be carried out by a series of fast solvers on
relatively smooth subdomains.
• Over last two decades, Schwarz’s idea has been extensively applied to
various problems defined on general domains.
• It has offered a possibility of efficient numerical algorithms for poorly-
conditioned large-scale problems and of parallelism for the very large sys-
tems of linear or nonlinear algebraic equations that arise from discretiza-
tions of elliptic problems in fluid dynamics, elasticity, wave propagation,
and other important areas.
• The main question for the classical SAM and its modern extensions has
been to show that the convergence rate of the iteration is satisfactory and
that it is independent or grows slowly when the mesh is to be refined
and/or when the number of subdomains increases.
• It is not surprising that reducing the amount of overlap without a dete-
rioration of the convergence rate has become an important issue in theo-
retical analyses and numerical simulations using SAM.
7.2. Overlapping Schwarz Alternating Methods (SAMs) 269

Ω1 Ω2

0.3 (2,4)
Ω
e1
Ω
e2

Figure 7.2: Nonoverlapping and overlapping partitions of Ω.

7.2. Overlapping Schwarz Alternating Methods

(SAMs)
7.2.1. Variational formulation
Let Ω be a bounded domain in Rd , d ≤ 3, with Lipschitz boundary Γ = ∂Ω.
Consider the following elliptic problem with a homogeneous Dirichlet bound-
ary condition: Find u ∈ V = H01 (Ω) such that

Lu := −∇ · (a(x)∇u) = f (x), x ∈ Ω,
(7.1)
u = 0, x ∈ Γ,

where we assumed that 0 < a∗ ≤ a(x) ≤ a∗ < ∞.

The problem (7.1) in its variational form reads

a(u, v) = (f, v), v ∈ V, (7.2)

where ˆ ˆ
a(u, v) = a∇u · ∇vdx, (f, v) = f v dx.
Ω Ω
270 Chapter 7. Domain Decomposition Methods

7.2.2. SAM with two subdomains

In the simplest form, SAM decomposes the original domain into two overlap-
ping subdomains Ω e 1 and Ω
e 2 ; see Figure 7.2. Let

Vej = {v ∈ V : v = 0 on Ω \ Ω
e j }, j = 1, 2.

Then, Vej are subspaces of V and V = Ve1 + Ve2 . Let an initial guess u0 = {u01 , u02 } ∈
V be given. Then, the iterate un ∈ V is determined from un−1 by sequentially
solving
n−1/2
(a) Lu1 = f, in Ω
e 1,
n−1/2
(b) u1 = 0, on Γ
e1,
n−1/2
(c) u1 = un−1
2 , on Γ12 ,
e
(7.3)
(d) Lun2 = f, in Ω
e 2,
(e) un2 = 0, on Γ
e2,
n−1/2
(f) un2 = u 1 , on Γ
e21 ,

where Γ e j ∩ ∂Ω and Γ
ej = ∂ Ω e j ∩ Ωk .
ejk = ∂ Ω

• This multiplicative Schwarz method solves at each iteration a series of

smaller problems restricted on subdomains.
• These subproblems require an additional boundary condition on the inte-
rior (artificial) boundaries Γ
ejk .
• The Schwarz method is easy to implement and can be applied to more
general elliptic differential operators and domains.
7.2. Overlapping Schwarz Alternating Methods (SAMs) 271

7.2.3. Convergence analysis

Let us consider the error propagation operator of (7.3); see [47, 70] for details.
n−1/2 n−1/2
In (7.3), one may extend u1 by un−1
2 on Ω2 and un2 by u1 on Ω1 . In the
variational form, (7.3) reads
n−1/2 n−1/2
a(u1 , v) = (f, v), v ∈ Ve1 , u1 − un−1 ∈ Ve1 ,
(7.4)
a(un2 , v) = (f, v), v ∈ Ve2 , un2 − un−1/2 ∈ Ve2 .

Since
(f, v) = a(u, v), v ∈ Vej , j = 1, 2,
one can rewrite (7.4) as
n−1/2 n−1/2
a(u1 − un−1 , v) = a(u − un−1 , v), v ∈ Ve1 , u1 − un−1 ∈ Ve1 ,
(7.5)
a(un2 − un−1/2 , v) = a(u − un−1/2 , v), v ∈ Ve2 , un2 − un−1/2 ∈ Ve2 .

It is easy and convenient to describe the method in terms of two projections

Pj , j = 1, 2, onto Vej , defined by

a(Pj v, w) = a(v, w), ∀ w ∈ Vej .

Then, (7.5) obviously means

un−1/2 − un−1 = P1 (u − un−1 ),
un − un−1/2 = P2 (u − un−1/2 ),

or equivalently
u − un−1/2 = (I − P1 ) (u − un−1 ),
u − un = (I − P2 ) (u − un−1/2 ),
where I is the identity operator. Therefore, the error propagates as

u − un = (I − P2 ) (I − P1 ) (u − un−1 ). (7.6)

Domain Decomposition for FEMs: Now, let V h be the piecewise linear

FE subspace of V corresponding to a regular triangulation Th . Then the FE
method for the variational problem (7.2) can be formulated as follows: Find
uh ∈ V h such that
a(uh , v h ) = (f, v h ), v h ∈ V h . (7.7)
272 Chapter 7. Domain Decomposition Methods

The FE procedure corresponding to the DDM (7.3) is formulated by finding

iterates {un−1/2 , un } from V h . One can consider analogous projections Pj , j =
1, 2, onto Vejh (:= Vej ∩ V h ) for FE methods. Then, the error for the FE methods
propagates as
uh − uh,n = (I − P2 ) (I − P1 ) (uh − uh,n−1 ). (7.8)

So, the FE formulation of (7.3) can be viewed as an iterative method for

solving
(P1 + P2 − P2 P1 ) uh = g h , (7.9)
with an appropriate right hand side g h . Here the upshot/hope is that the
condition number of (P1 + P2 − P2 P1 ) is much smaller than that of the original
algebraic system.
7.2. Overlapping Schwarz Alternating Methods (SAMs) 273

Notes
• The multiplicative Schwarz method has an important variant, i.e., the
additive Schwarz method which decouples the subproblems (7.3.a)-(7.3.c)
and (7.3.d)-(7.3.f). In additive Schwarz method, (7.3.f) is replaced by

un2 = un−1
1 , on Γ
e21 ;

the additive algorithm is a simple iterative method for solving

(P1 + P2 )uh = g0h , (7.10)

for some g0h ; see Exercise 7.1.

• Such Schwarz methods can be generalized immediately to any number of
overlapping subdomains Ω e j expanded from the original nonoverlapping
subdomains Ωj , j = 1, 2, · · · , M .
274 Chapter 7. Domain Decomposition Methods

7.2.4. Coarse subspace correction

Let Hj measure the size of Ωj and

H = max Hj .
j=1,··· ,M

It is known that a DD preconditioner for which the new iterate is updated by

the former solutions on local subregions of diameter on the order of H has a
condition number which grows at least as fast as 1/H 2 ; see [19] and references
therein.
To overcome this difficulty, one can introduce the coarse subspace correction
technique as a preconditioner. Then, our FE space is represented as the sum
of M + 1 subspaces
V h = V0h + Ve1h + · · · + VeMh , (7.11)
where V0h = V H , the piecewise linear FE space on the coarse mesh defined
by the nonoverlapping partition {Ωj }. (We have implicitly assumed that each
subdomain is triangle.)
The corresponding additive algorithm can be viewed as an iterative method
for solving
P uh = (P0 + P1 + · · · + PM )uh = Gh , (7.12)
for an appropriate Gh , where P0 is the projection from V h to V H .
7.2. Overlapping Schwarz Alternating Methods (SAMs) 275

Known: Let λ∗ > 0 and λ∗ > 0 be the minimum and the maximum eigen-
values for a symmetric positive definite (SPD) matrix A, respectively. The
condition number of A, κ(A), is defined by

κ(A) = λ∗ /λ∗ .

The
prequired
 iteration number for the CG method to solve SPD systems is
O κ(A) for a given accuracy. (For more general systems, GMRES [59] and
QMR [24] can be used.) The following result was established by Dryja and
Widlund [19].
Theorem 7.1. Let δ = min dist(∂Ωj \ ∂Ω, ∂ Ω e j \ ∂Ω) > 0. Assume the
j=1,··· ,M
problem coefficient a is continuous on Ω̄. Then, the condition number of the
additive Schwarz method for solving (7.12) satisfies

κ(P ) ≤ C(1 + H/δ), (7.13)

where C is independent of H, h, and δ.

If there is no coarse subspace correction, (7.13) must be replaced by (see
[45])  
1 H
κ(P ) ≤ C 1 + 2 ,
Hmin δ
where Hmin is the minimum diameter of the subdomains.
276 Chapter 7. Domain Decomposition Methods

Final Notes

• Introducing a global solver at a modest cost is the key to efficiency of

iterative algorithms.
• On the other hand, if the overlap is a fraction of H, the condition number
in (7.13) is bounded uniformly by a constant.
• In numerical simulations, however, the requirement on the amount of
overlap may degrade the algorithm due to a heavy cost of local solvers.
Consider the algorithm with a small overlap. The number of CG itera-
tions is higher in such a case, but this can be compensated for by cheaper
local problem solvers.
• The condition number for DD methods incorporating a small overlap to-
gether with a coarse subspace solver is often bounded by

κ(P ) ≤ C(1 + log(H/h))r , r = 2, 3, or 4, (7.14)

where r depends on the amount of overlap and the regularity of the diffu-
sion coefficient a.
• The convergence analysis of Schwarz method is more complicated when
the subdomains overlap less. See [47] and the survey papers [19, 45] for
details.
7.3. Nonoverlapping DDMs 277

7.3. Nonoverlapping DDMs

7.3.1. Multi-domain formulation
Recall the model problem: Find u ∈ V = H01 (Ω) such that

Lu := −∇ · (a(x)∇u) = f (x), x ∈ Ω,
(7.15)
u = 0, x ∈ Γ,

where we assumed that 0 < a∗ ≤ a(x) ≤ a∗ < ∞.

Consider a nonoverlapping partition {Ωj : j = 1, 2, · · · , M } of Ω:

Ω = ∪Mj=1 Ωj ; Ωj ∩ Ωk = ∅, j 6= k;
Γj = Γ ∩ ∂Ωj ; Γjk = Γkj = ∂Ωj ∩ ∂Ωk .

Let uj denote the restriction of u to Ωj .

278 Chapter 7. Domain Decomposition Methods

Then, the problem (7.15) can be formulated as follows: Find {uj } such that

(a) Luj = f, x ∈ Ωj ,
(b) uj = 0, x ∈ Γj ,
(c) uj = uk , x ∈ Γjk , (7.16)
∂uj ∂uk
(d) =− , x ∈ Γjk ,
∂νL,j ∂νL,k
where the conormal derivative is defined as
∂uj
= a∇uj · nj ,
∂νL,j
where nj indicates the unit outer normal from ∂Ωj .

• Equations (7.16.c)-(7.16.d) are the transmission conditions which impose

the continuity of the solution and its conormal fluxes on the subdomain
interfaces.
• Nonoverlapping DDMs can be characterized depending on how the trans-
mission conditions are incorporated in the iteration procedure.

We first introduce the Steklov-Poincaré operator which is useful for the con-
vergence analysis for the variational formulation of the DDMs.
7.3. Nonoverlapping DDMs 279

7.3.2. The Steklov-Poincaré operator

Let λjk be the unknown value of u on Γjk . Consider the following Dirichlet
problems:
Lwj = f, x ∈ Ωj ,
wj = 0, x ∈ Γj , (7.17)
wj = λjk , x ∈ Γjk ,
for j = 1, · · · , M . Then, we can state that

wj = u0j + u∗j , (7.18)

where {u0j } and {u∗j } are defined as the solutions of

Lu0j = 0, x ∈ Ωj ,
u0j = 0, x ∈ Γj , (7.19)
u0j = λjk , x ∈ Γjk ,

and
Lu∗j = f, x ∈ Ωj ,
u∗j = 0, x ∈ Γj , (7.20)
u∗j = 0, x ∈ Γjk ,
Note that when a(x) = 1, u0j is the harmonic extension of {λjk } (for k’s such that
Γjk 6= ∅) into Ωj ; for general coefficients, we still call it the harmonic extension
and denote by Hj λjk . We will also write Gj f instead of u∗j , j = 1, · · · , M .
280 Chapter 7. Domain Decomposition Methods

It follows from comparing (7.16) with (7.17) that


uj = wj , ∀ j = 1, · · · , M
n ∂w
j ∂wk o (7.21)
⇐⇒ =− , ∀ j, k such that Γjk 6= ∅ .
∂νL,j ∂νL,k
The latter condition equivalently amounts to the requirement that each of
{λjk } satisfies the Steklov-Poincaré interface equation

Sjk λjk = χjk , (7.22)

where S = {Sjk } is the Steklov-Poincaré operator defined as

∂ ∂
Sjk η = Hj η + Hk η, (7.23)
∂νL,j ∂νL,k
for η defined on Γjk (6= ∅), and
 
∂ ∂
χjk =− Gj f + Gk f . (7.24)
∂νL,j ∂νL,k
The operator S is symmetric, positive definite (coercive), and continuous.
Here the goal is to find {λjk } such that λjk = u Γjk , which must satisfy (7.22).
Some DDMs update the iterates {λnjk } by iteratively solving (7.22), of which
each step solves the subproblems in (7.19) and (7.20). The process can be
understood easily by considering the algebraic system of the discrete Steklov-
Poincaré operator, which is known as the Schur complement matrix.
7.3. Nonoverlapping DDMs 281

7.3.3. The Schur complement matrix

Consider the FE method for the variational form (7.7). Let Nj denote the
number of interior nodes in Ωj , j = 1, 2, · · · , M , and NB be the number of
nodal points on ∪Γjk . Thus the total number of nodes are N1 + · · · + NM + NB .
We order the interior nodes of {Ωj }’first and those on ∪Γjk next. Then, the
algebraic system of (7.7) can be written as
    
AII AIB uI fI
Au := = , (7.25)
ABI ABB uB fB

where AII is a block diagonal matrix and ABI = ATIB :

AII = diag(A11 , A22 , · · · , AM M ),

ABI = (AB1 , AB2 , · · · , ABM ).

Here the sr-th entry of Ajj , the `r-th entry of ABj , and the `m-th entry of ABB
are given by

(Ajj )sr = aj (ϕ(j) (j)

r , ϕs ), s, r = 1, · · · , Nj ,
(B)
(ABj )`r = aj (ϕ(j) , ϕ` ), ` = 1, · · · , NB , r = 1, · · · , Nj ,
Xr (B)
(B)
(ABB )`m = aj (ϕm , ϕ` ), `, m = 1, · · · , NB ,
j

(j) (B)
where aj (·, ·) is the restriction of a(·, ·) to Ωj , and ϕs and ϕ` are the basis
functions associated with nodes lying in Ωj and ∪Γjk , respectively.
282 Chapter 7. Domain Decomposition Methods

By eliminating all degrees of freedom that are associated with interior

nodes of subdomains, (7.25) reduces to the following interface problem:

Σ uB = fB − ATIB A−1
II fI , (7.26)

where Σ is the Schur complement matrix defined as

Σ = ABB − ATIB A−1

II AIB .

The matrix Σ is exactly the algebraic counterpart of the discrete Steklov-

Poincaré operator; it can be proved symmetric positive definite, as for the
Steklov-Poincaré operator.
In early substructuring techniques of the 1960’s, the interface problem
(7.26) was solved by a direct solver (for which a frontal method was often
employed mainly due to insufficient computer memory). Most of the recent it-
erative nonoverlapping DDMs can be explained as preconditioning techniques
for solving the interface problem by the CG method.
Each matrix-vector multiplication with Σ involves M subdomain solves,
i.e.,
A−1 −1 −1
II = diag(A11 , · · · , AM M ),

which can be carried out in parallel.

7.3. Nonoverlapping DDMs 283

Convergence

• As reported in Le Tallec [45], the condition number of Σ is bounded as

H
κ(Σ) ≤ C 2 ,
hHmin
where H and Hmin are respectively the maximum and minimum diame-
ters of the subdomains.
• Thus a mathematical challenge is to construct a preconditioner for Σ such
that the convergence rate of the preconditioned iterative method becomes
independent on both h and H.
• However, in practice the incorporation of such an optimal preconditioner
may not imply that the resulting algorithm is fastest in computation time.
We refer interested readers to Quarteroni and Valli [57].
284 Chapter 7. Domain Decomposition Methods

7.4. Iterative DDMs Based on Transmission Con-

ditions
7.4.1. The Dirichlet-Neumann method
As it is called, some subproblems are solved using Dirichlet data on the inter-
faces and the others use Neumann data. We may separate the subdomains
into two groups by a red-black coloring.
Let IR and IB be respectively the indices of the red and black subdomains.
Then, the method is formulated as follows: For given {λ0jk }, find {unj }, n ≥ 1,
by recursively solving
 n
 Luj = f, x ∈ Ωj ,

(a) unj = 0, x ∈ Γj , j ∈ IB ,
 un = λn−1 , x ∈ Γ ,

j jk jk

Lunj = f,

x ∈ Ωj ,
(7.27)


 un = 0,

x ∈ Γ ,
j j
(b) n
j ∈ IR ,
∂u n
 j ∂u
= − k , x ∈ Γjk ,



∂νL,j ∂νL,k
(c) λnjk = θjk unj,R + (1 − θjk )λn−1
jk ,

where {θjk } > 0 is an acceleration parameter and unj,R denotes the solution
from the subdomains colored red.
7.4. Iterative DDMs Based on Transmission Conditions 285

The acceleration parameter is often set less than one; the method without
relaxation (i.e., θjk ≡ 1) is not necessarily convergent, unless special assump-
tions are made on the size of the subdomains. We refer readers interested in
the Dirichlet-Neumann method to [4, 6, 52] and [57] for details.
286 Chapter 7. Domain Decomposition Methods

7.4.2. The Neumann-Neumann method

This method requires solving the subproblems twice, one with Dirichlet-Dirichlet
data and the other with Neumann-Neumann data: For given {λ0jk }, find {unj },
n ≥ 1, satisfying
 n
 Luj = f, x ∈ Ωj ,

(a) unj = 0, x ∈ Γj ,
 un = λn−1 , x ∈ Γ ,

j jk jk

Lvjn = 0,

x ∈ Ωj ,
(7.28)


 v n = 0,

x ∈ Γj ,
j
(b)
 ∂vjn ∂unj ∂unk
= + , x ∈ Γjk ,



∂νL,j ∂νL,j ∂νL,k
(c) λnjk = λjkn−1
− θjk σjk vjn + (1 − σjk )vkn Γjk , j > k,

where {θjk } > 0 is again an acceleration parameter and {σjk } is an averaging

coefficient.
The Neumann-Neumann method was studied in [1, 5, 12, 50]. It is known
that the method is efficient when the subdomains are similar [45]. The re-
sulting condition number (without a coarse grid solver) has been shown to be
[12]  2
C H
κ(M −1 A) ≤ 2 1 + log ,
H h
where M is the Neumann-Neumann preconditioning matrix for A.
7.4. Iterative DDMs Based on Transmission Conditions 287

7.4.3. The Robin method

The method was first suggested by Lions [48] and has been applied to various
physical problems with a great efficiency; see e.g. [13, 17, 36, 38, 41, 42, 53].
For given {u0j }, find {unj }, n ≥ 1, satisfying

(a) Lunj = f, x ∈ Ωj ,
(b) unj = 0, x ∈ Γj ,
(7.29)
∂unj ∂un−1
(c) + θjk uj = − k + θjk un−1
n
k , x ∈ Γjk ,
∂νL,j ∂νL,k
where {θjk } ≥ 0 is an acceleration parameter with

θjk + θkj > 0.

Lions [48] proved the convergence of the method through an energy estimate
on the interfaces.
Note that (7.29.c) is defined twice on each of Γjk from both sides of the
interface:
∂unj ∂un−1
+ θjk uj = − k + θjk un−1
n
k ,
∂νL,j ∂νL,k
∂unk n
∂un−1
j
+ θkj uk = − + θkj un−1
j .
∂νL,k ∂νL,j
When the iterates converge, the limit {uj } would satisfy the above equations
in the same way (without the superscripts n and n − 1). By subtracting
and adding the equations, one can get the transmission conditions (7.16.c)-
(7.16.d).
288 Chapter 7. Domain Decomposition Methods

7.4.4. Remarks on DDMs of transmission conditions

• The DDMs based on transmission conditions ((7.27), (7.28), and (7.29)) re-
quire to choose appropriate acceleration parameters to either guarantee
or accelerate convergence. However, there is no guide line to be applied
to various problems; finding the acceleration parameter is problematic.
• For the Robin method applied, Kim [37, 44] suggested an automatic way
of choosing the acceleration parameter to solve the Helmholtz wave prob-
lem.

• A very important accuracy issue is related to the discrete transmission

conditions. Recall that the standard discretization methods such as the
FD and FE methods allow the conormal flux to be discontinuous at the
element interfaces.
• Since the transmission conditions impose the continuity of both the so-
lution and its conormal flux on the subdomain interfaces, there will be
a flux conservation error, i.e., the discrete solution uh would not satisfy
(7.16.c)-(7.16.d) unless it is linear across the subdomain interfaces.
7.4. Iterative DDMs Based on Transmission Conditions 289

Flux conservation error

• In practice, the flux conservation error can severely deteriorate accuracy
of the computed solution.
• Thus the conormal flux must be treated with a special care, in particular,
when the DDM is to be utilized as the main solver.
• When the DDM is used as a preconditioner, i.e., another algorithm such
as a Krylov subspace method is applied as an outer iteration, the flux
conservation error may affect the convergence speed of the resulting al-
gorithm; however, the required accuracy of the solution can be achieved
by the main solver (the outer iteration).
290 Chapter 7. Domain Decomposition Methods

W O E

S
Ωj Γjk Ωk

Figure 7.3: The five point stencil at a grid point on the interface Γjk .

Discretization of the Robin boundary condition: To illustrate

a way of dealing with the conormal flux, consider the Robin method applied to
the Poisson equation, L = −∆:
(a) −∆unj = f, x ∈ Ωj ,
(b) unj = 0, x ∈ Γj ,
(7.30)
∂unj ∂un−1
(c) + βuj = − k + βun−1
n
k , x ∈ Γjk ,
∂νj ∂νk
where β > 0 is a constant acceleration parameter.
Let the domain be discretized into uniform cells of edge size h and the sub-
domain interfaces {Γjk } coincide with parts of grid lines. Let ∂b,jk uj and ∂f,jk uj
be the backward and forward differences for ∂uj /∂νj on Γjk , respectively. For
example, at the nodal point O ∈ Γjk in Figure 7.3, they are defined as

∂b,jk uj (O) = (uj (O) − uj (W) )/h, ∂f,jk uj (O) = (uj (E) − uj (O) )/h,
∂b,kj uk (O) = (uk (O) − uk (E) )/h, ∂f,kj uk (O) = (uk (W) − uk (O) )/h.

(Here we have employed an exterior bordering of the subdomains.)

7.4. Iterative DDMs Based on Transmission Conditions 291

Let ∆h uj be the central five-point difference approximation of ∆uj . Then

the DD iterative algorithm in the FD formulation can be defined as follows:
For given {u0j }, find {unj }, n ≥ 1, by recursively solving

(a) −∆h unj = f, x ∈ Ωj ,

(b) unj = 0, x ∈ Γj , (7.31)
(c) ∂f,jk unj + β unj = −∂b,kj un−1
k + β un−1
k , x ∈ Γjk .

Note that (7.31.c) imposes the continuity of the discrete solution only, when
the algorithm converges. Such a treatment of the Robin condition, a forward-
backward difference matching, was introduced by Kim [36, 38] to enforce
equivalence of the DD method to the original discrete problem of the multi-
linear FE methods.
292 Chapter 7. Domain Decomposition Methods

Equivalence: In the following, we will check the equivalence of algorithm

(7.31) to the original discrete problem. It suffices to consider the algebraic
equations of (7.31) at interface grid points. At the point O (in Figure 7.3), the
equation (7.31.a) reads

4 unj,O − unj,E − unj,W − unj,S − unj,N = h2 fO , (7.32)

where unj,O = unj (O), the value of unj at the point O, and the others are similarly
defined.
The term unj,E in (7.32) evaluated at a point out of the subdomain Ωj can be
substituted by using (7.31.c). Equation (7.31.c) is written as

unj,E − unj,O n
un−1 n−1
k,E − uk,O
+ β uj,O = + β un−1
k,O ,
h h
or equivalently

unj,E − (1 − βh) unj,O = un−1 n−1

k,E − (1 − βh) uk,O . (7.33)

Adding (7.32) and (7.33) reads

[4 − (1 − βh)] unj,O − unj,W − unj,S − unj,N = h2 fO + un−1 n−1

k,E − (1 − βh) uk,O . (7.34)
7.4. Iterative DDMs Based on Transmission Conditions 293

In the same manner, one can treat cross points arising in a box-type decom-
position of the domain. When the algorithm converges, the limit would clearly
satisfy the original algebraic equation

4uO − uE − uW − uS − uN = h2 fO ,

which proves the equivalence of (7.31) to the original discrete problem.

• It should be noticed that the standard FE formulation of (7.30) fails to get

the original discrete solution, unless the original solution is linear across
the subdomain interfaces. The forward-backward difference matching
can be incorporated into the FE formulation to overcome the difficulty.
See Exercises 7.2 and 7.3.
• For FD schemes, the normal derivatives in (7.30) can be approximated
by the central differences, without a failure for the original FD solution.
However, the convergence speed of the iteration may matter.
294 Chapter 7. Domain Decomposition Methods

7.5. Homework
1. Derive (7.10) for the additive Schwarz method for two overlapping subdo-
mains.
2. Consider the bilinear FE method of grid size h on the unit square applied
to the DD method (7.30): Given {uh,0
j }, uj
h,0
∈ Vjh := V h Ωj , j = 1, · · · , M ,
find {uh,n
j }, n ≥ 1, satisfying
X
(∇uh,n
j , ∇v)Ωj + hβuh,n
j , viΓjk = (f, v)Ωj
k
X h,n−1
∂uk X (7.35)
+ h− , viΓjk + hβuh,n−1
k , viΓjk , v∈ Vjh .
∂νk
k k

(a) Show that the algebraic equation of (7.35) at the boundary nodal point
O as given in Figure 7.3 reads

1 n 1 n h2
(2 + βh) unj,O
− unj,W
− uj,S − uj,N = fO + un−1 n−1
k,E − (1 − βh) uk,O , (7.36)
2 2 2
provided that the mass-lumping quadrature rule is used.
(b) Show that (7.36) is equivalent to (7.34), in their limits, if the discrete
solution is linear across the subdomain boundary Γjk .
3. A modification of (7.35) can be obtained incorporating the forward-backward
difference matching (7.31.c) as follows: Given {uh,0 h,0 h
j }, uj ∈ Vj , j = 1, · · · , M ,
find {uh,n
j }, n ≥ 1, satisfying
X
(∇uh,nj , ∇v)Ωj + h−∂c,jk uh,n
j , viΓjk = (f, v)Ωj , v ∈ Vjh ,
k (7.37)
n n n−1 n−1
∂f,jk uj + β uj = −∂b,kj uk + β uk , x ∈ Γjk ,
∂uh,n
where ∂c,jk uh,nj is the central approximation of ∂νj j , i.e., ∂c,jk = (∂b,jk +
∂f,jk )/2. (We have assumed the outer bordering.) Equations (7.37) can be
rewritten as
h,n
X 1
(∇uj , ∇v)Ωj + h (−∂b,jk uh,n n
j + β uj ), viΓjk
2
k
X 1 (7.38)
h,n−1 n−1 h
= (f, v)Ωj + h (−∂b,kj uk + β uk ), viΓjk , v ∈ Vj .
2
k
7.5. Homework 295

Prove that the algorithm (7.38) solves the original discrete solution if it
converges.
296 Chapter 7. Domain Decomposition Methods
Chapter 8

Multigrid Methods∗

See sepatate hand-out.

297
298 Chapter 8. Multigrid Methods∗

8.1. Introduction to Multigrid Methods

8.2. Homework 299

8.2. Homework
1.
300 Chapter 8. Multigrid Methods∗
Chapter 9

Locally One-Dimensional Methods

Explicit schemes for parabolic equations are easy to implement, but they are
stable only if the time step size is chosen sufficiently small: ∆t = O(∆x2 ).
Implicit methods are often unconditionally stable; however, a large algebraic
system must be solved (directly or iteratively) for the time integration on each
of the space-time slices. In this chapter, we will introduce the locally one-
dimensional (LOD) methods such as the alternating direction implicit (ADI)
method and the fractional step (FS) method, in order to solve the algebraic
system of equations efficiently. The LOD methods can be viewed as a pertur-
bation of standard implicit methods.

301
302 Chapter 9. Locally One-Dimensional Methods

9.1. Heat Conduction in 1D Space: Revisited

Let Ω = (0, 1) and J = (0, T ], for some T > 0. Consider the following simplest
model problem for parabolic equations in 1D:
ut − uxx = 0, (x, t) ∈ Ω × J,
u = 0, (x, t) ∈ Γ × J, (9.1)
u = u0 , x ∈ Ω, t = 0,

where Γ is the boundary of Ω, i.e., Γ = {0, 1}, and u0 is the prescribed initial
value of the solution at t = 0.
Let
∆t = T /nt , tn = n∆t, n = 0, 1, · · · , nt ;
∆x = 1/nx , xj = j∆x, j = 0, 1, · · · , nx ;
for some positive integers nt and nx . Define unj = u(xj , tn ). Let A1 be the central
second-order approximation of −∂xx , defined as
−unj−1 + 2unj − unj+1
A1 unj := .
∆x2
Then the θ-method for (9.1) is
v n − v n−1
+ A1 θv n + (1 − θ)v n−1 = 0,
 
θ ∈ [0, 1], (9.2)
∆t
or equivalently

(I + θ∆tA1 )v n = [I − (1 − θ)∆tA1 ]v n−1 , θ ∈ [0, 1]. (9.3)

9.1. Heat Conduction in 1D Space: Revisited 303

Forward Euler method (θ = 0): The algorithm (9.3) is reduced to

v n = (I − ∆tA1 )v n−1 ,

which is explicit and cheap to compute the solution in each time level. How-
ever, we shall see later that its stability requires to choose ∆t small enough to
satisfy
∆t 1
µ= ≤ .
∆x2 2
Backward Euler method (θ = 1): This is an implicit method written as

(I + ∆tA1 )v n = v n−1 .

The method must invert a tridiagonal matrix to get the solution in each time
level. But it is stable independently on the choice of ∆t.
Crank-Nicolson method (θ = 1/2):
 ∆t  n  ∆t  n−1
I+ A1 v = I − A1 v .
2 2
It requires to solve a tridiagonal system in each time level, as in the backward
Euler method. However, the Crank-Nicolson method is most popular, because

• it is unconditionally stable
• its error = O(∆x2 + ∆t2 )

It is often called a semi-implicit method.

304 Chapter 9. Locally One-Dimensional Methods

Stability analysis
Components of the algebraic system (9.3) are
n
−θµ vj−1 + (1 + 2θµ)vjn − θµ vj+1
n

n−1 (9.4)
= (1 − θ)µ vj−1 + [1 − 2(1 − θ)µ]vjn−1 + (1 − θ)µ vj+1
n−1
,

where µ = ∆t/∆x2 .
For an stability analysis for this one-parameter family of systems, substi-
tute g n eijϑ for vjn in (9.4) to have

g −θµ e−ijϑ + (1 + 2θµ) − θµ eijϑ

 

= (1 − θ)µ e−ijϑ + [1 − 2(1 − θ)µ] + (1 − θ)µ eijϑ ,

i.e.,
1 − 2(1 − θ)µ (1 − cos ϑ) 1 − 4(1 − θ)µ sin2 ϑ2
g= = .
1 + 2θµ (1 − cos ϑ) 1 + 4θµ sin2 ϑ2
Because µ > 0 and θ ∈ [0, 1], the amplification factor g cannot be larger than
one. The condition g ≥ −1 is equivalent to

2 ϑ 2 ϑ
h i
1 − 4(1 − θ)µ sin ≥ − 1 + 4θµ sin ,
2 2
or
ϑ 1
(1 − 2θ)µ sin2 ≤ .
2 2
Thus (9.3) is stable if
1
(1 − 2θ)µ ≤ . (9.5)
2
9.1. Heat Conduction in 1D Space: Revisited 305

In conclusion:

• The θ-method is unconditionally stable for θ ≥ 1/2, because every choice

of µ satisfies the above inequality.
• When θ < 1/2, the method is stable only if
∆t 1
µ= 2 ≤ , θ ∈ [0, 1/2). (9.6)
∆x 2(1 − 2θ)

• For example, the forward Euler method (θ = 0) is stable only if

∆t ≤ ∆x2 /2;

∆t must be chosen sufficiently small for stability.

306 Chapter 9. Locally One-Dimensional Methods

Maximum principle
For heat conduction without interior sources/sinks, it is known mathemat-
ically and physically that the extreme values of the solution appear either
in the initial data or on the boundary. This property is called the maximum
principle. It is quite natural and sometimes very important to examine if the
numerical solution satisfies the maximum principle, too.
Theorem 9.1. (Maximum principle for the θ-method). Let the θ-method
be set satisfying θ ∈ [0, 1] and
1
(1 − θ)µ ≤ .
2
If the computed solution v has an interior maximum or minimum, then v is
constant.
9.1. Heat Conduction in 1D Space: Revisited 307

Error analysis
Let
enj = unj − vjn ,
where unj = u(xj , tn ) with u being the exact solution of (9.1). Define
n−1/2
E n = max |enj |, T n−1/2 = max |T uj |,
j j

n−1/2
where T uj is the truncation error expanded at (xj , tn−1/2 ). Note that vj0 =
u0j , j = 0, · · · , nx , and therefore E 0 = 0.
Theorem 9.2. Let the θ-method be set satisfying θ ∈ [0, 1] and (1 − θ)µ ≤ 21 .
Then,
Xn
n
E ≤ ∆t T k−1/2 . (9.7)
k=1

It follows from (9.7) that

E n ≤ n∆t max T k−1/2 ≤ T max T k−1/2 ,

k k

where T is the upper limit of the time variable.

308 Chapter 9. Locally One-Dimensional Methods

9.2. Heat Equation in Two and Three Variables

Let Ω be a bounded domain in Rm , m = 2 or 3, with boundary Γ = ∂Ω. Consider
the parabolic problem
ut − ∇ · (a∇u) + cu = f, (x, t) ∈ Ω × J,
α1 uν + α2 u = g, (x, t) ∈ Γ × J, (9.8)
u = u0 , x ∈ Ω, t = 0,

where

• a > 0, c ≥ 0, α1 ≥ 0, and α2 ≥ 0 are given functions, α1 + α2 > 0,

• the subscript ν denotes the outer unit normal on Γ,
• u0 is the prescribed initial value of the solution at t = 0, and
• f and g represent external sources and sinks.
9.2. Heat Equation in Two and Three Variables 309

9.2.1. The θ-method

Let Th be the mesh of Ω consisting of elements of which the maximum edge
size is h. Let A be the approximation of −∇ · a∇ + c on the mesh Th , having
the p-th order accuracy, i.e.,

Au ≈ −∇ · (a∇u) + cu + O(hp ).

Then, the θ-method for (9.8) reads1

v n − v n−1
+ A θv n + (1 − θ)v n−1 = f n−1/2 ,
 
θ ∈ [0, 1], (9.9)
∆t
and the truncation error for the n-th time level is

δ n−1/2 = O (1 − 2θ)∆t + ∆t2 + hp .

Note that A is symmetric and nonnegative; it is positive definite when c > 0

or α2 > 0.
Let vn be the solution vector in the n-th time level. Then the method (9.9)
in its matrix representation reads

[I + θ∆tA] vn = ∆tf n−1/2 + [I − (1 − θ)∆tA] vn−1 . (9.10)

1
Here we used f n−1/2 , instead of f n−1+θ , for a simplier presentation.
310 Chapter 9. Locally One-Dimensional Methods

Notes:

• When θ > 0, it is necessary to invert a matrix, either exactly or approxi-

mately, to get the solution in the new time level.
• When the domain is rectangular or cubic, the algebraic system (9.10) can
be perturbed to become a series of traditional systems; the resulting prob-
lem can be solved very efficiently. This is the basic idea of the locally
one-dimensional (LOD) methods to be treated in this chapter later.
9.2. Heat Equation in Two and Three Variables 311

9.2.2. Convergence analysis for θ-method

For a simpler presentation, we define

n v n − v n−1
∂ tv = .
∆t
Let
en = un − v n ,
where un is the exact solution of (9.8) at the time level tn . Then, the error
equation associated with the θ-method (9.9) is

∂ t en + A[θen + (1 − θ)en−1 ] = δ n−1/2 . (9.11)

Choose ∂ t en as a test function. Then, for n ≥ 1,

(∂ t en , ∂ t en ) + A[θen + (1 − θ)en−1 ], ∂ t en = (δ n−1/2 , ∂ t en ).

(9.12)

Note that
1
θen + (1 − θ)en−1 = (en + en−1 ) + (2θ − 1)(en − en−1 )


2
and therefore
A[θen + (1 − θ)en−1 ], ∂ t en ∆t

1h
= (Aen , en ) − (Aen−1 , en−1 ) (9.13)
2 i
n n 2
+(2θ − 1)(A∂ t e , ∂ t e )∆t , n ≥ 1.
312 Chapter 9. Locally One-Dimensional Methods

Multiply (9.12) by ∆t and utilize (9.13) to have

2θ − 1
k∂ t en k2 ∆t + (A∂ t en , ∂ t en )∆t2
2
1 (9.14)
+ (Ae , e ) − (Aen−1 , en−1 )
n n

2
= (δ n−1/2 , ∂ t en )∆t, n ≥ 1.

Summing (9.14) beginning at n = 1 reads

n n
X 2θ − 1 X 1
j 2
k∂ t e k ∆t + (A∂ t ej , ∂ t ej )∆t2 + (Aen , en )
j=1
2 j=1 2
n (9.15)
1 0 0
X
= (Ae , e ) + (δ j−1/2 , ∂ t ej )∆t.
2 j=1

Now, we apply the inequality (|ab| ≤ (a2 + b2 )/2) to the last term in (9.15) to
obtain the following inequality:
n
X n
X
j 2
k∂ t e k ∆t + (2θ − 1) (A∂ t ej , ∂ t ej )∆t2 + (Aen , en )
j=1 j=1
n
X (9.16)
≤ (Ae0 , e0 ) + kδ j−1/2 k2 ∆t.
j=1

Thus, the estimation of the error generated by the θ-method is reduced to

bounding the errors in v 0 and the truncation error.
9.2. Heat Equation in Two and Three Variables 313

Note: The estimate (9.16) also indicates that

• The θ-method is unconditionally stable for θ ∈ [1/2, 1].

• When θ ∈ [0, 1/2), it is stable if

1 + (2θ − 1)ρ(A)∆t ≥ 0,

where ρ(A) is the spectral radius of A (the largest eigenvalue of A in

modulus). Since
ρ(A) ≈ 4mkak∞ /h2 ,
where m is the dimensionality and kak∞ = max |a(x)|, the θ-method is
x∈Ω
stable if
∆t 1
≤ , θ ∈ [0, 1/2). (9.17)
h2 4(1 − 2θ)mkak∞
The inequality in (9.17) is compared to the analysis in (9.6).

• The θ-method is particularly interesting when θ = 1/2, because the trun-

cation error becomes second-order in time. This case is called the Crank-
Nicolson or semi-implicit method. The spatial derivatives can be approx-
imated to have a p-th order accuracy, p ≥ 2, independently on θ or ∆t.
314 Chapter 9. Locally One-Dimensional Methods

9.3. LOD Methods for the Heat Equation

Over the last five decades or so, many time-stepping procedures have been
introduced to allow multidimensional parabolic problems to be approximated
accurately and efficiently. These procedures treat the spatial variables in-
dividually in a cyclic fashion; we shall call any such a procedure a locally
one-dimensional (LOD) method. Here we will be mainly concerned with two
families of these methods, namely the alternating direction implicit (ADI)
methods [14, 18, 56] and the fractional-step (FS) procedures [20, 51, 71, 72].
These methods can be interpreted as perturbations of some underlying im-
plicit multidimensional numerical method, such as the Crank-Nicolson or the
backward Euler method. Recently, a unified approach of these LOD methods,
along with strategies for virtual elimination of the splitting error, has been
studied by Douglas and Kim [16].
9.3. LOD Methods for the Heat Equation 315

9.3.1. The ADI method

Consider the parabolic problem (9.8) defined on a rectangular domain Ω ⊂ R2 .
Let Th be a uniform mesh of rectangular elements of which the edge lengths
are hx and hy , h = max(hx , hy ). Define
1 1
A1 u ≈ −(aux )x + cu, A2 u ≈ −(auy )y + cu,
2 2
which are finite difference or finite element approximations on the mesh Th
having a truncation error of O(hp ), p ≥ 2. Let

A = A1 + A2 .

Then the Crank-Nicolson difference equation for the heat equation (9.8) reads

v n − v n−1 1
+ A(v n + v n−1 ) = f n−1/2 + O(hp + ∆t2 ), (9.18)
∆t 2
where
1
f n−1/2 = (f n + f n−1 ).
2
The truncation error for the CN procedure (9.18) is

O(∆x2 + ∆t2 ).
316 Chapter 9. Locally One-Dimensional Methods

The original ADI method:

The ADI method of Douglas-Peaceman-Rachford [14, 18, 56] is a pertur-
bation of the Crank-Nicolson difference equation that has a splitting error of
O(∆t2 ), so that it is second-order correct in time.
Let us formulate it in an equivalent way that will coincide with the general
formulation in Douglas-Gunn [15] of ADI methods. Given an approximation
w0 to u0 , find wn , n ≥ 1, by solving
w∗ − wn−1 1
+ A1 (w∗ + wn−1 ) + A2 wn−1 = f n−1/2 ,
∆t 2 (9.19)
wn − wn−1 1 1
+ A1 (w∗ + wn−1 ) + A2 (wn + wn−1 ) = f n−1/2 ,
∆t 2 2
or, equivalently,
 ∆t  ∗  ∆t 
1+ A1 w = 1 − A1 − ∆tA2 wn−1 + ∆tf n−1/2 ,
2 2 (9.20)
 ∆t  n ∆t
1+ A2 w = w∗ + A2 wn−1 .
2 2
Here w∗ is an intermediate value.
9.3. LOD Methods for the Heat Equation 317

Splitting error of ADI: The intermediate solution w∗ can be found (implic-

itly) as
∆t
w∗ = wn + A2 (wn − wn−1 ).
2
Thus, by plugging it into the first equation of (9.20), we have
 ∆t  ∆t  n  ∆t  n−1
1+ A1 1 + A2 w = 1 − A w
2 2 2
∆t2
+ A1 A2 wn−1 + ∆tf n−1/2 .
4
Multiply out the left hand side and rewrite the result as
wn − wn−1 1 ∆t
+ A(wn + wn−1 ) + A1 A2 (wn − wn−1 ) = f n−1/2 . (9.21)
∆t 2 4
Thus, compared with (9.18), the splitting error is given by
∆t
A1 A2 (wn − wn−1 ), (9.22)
4
which is O(∆t2 ) for a smooth solution.
318 Chapter 9. Locally One-Dimensional Methods

Notes:

• Some theoretical aspects of the method were treated in detail in Douglas

[14], while practical aspects of the method were considered in the com-
panion paper by Peaceman-Rachford [56].
• In each half of the calculation, the matrix to be inverted is tridiagonal, so
that the algorithm requires O(N := nt nx ny ) flops.
• The ADI (9.19) can be equivalently formulated in many different ways.
The modelcode ADI_HEAT.CF.tar in GRADE [35] is implemented based
on the following formulation:
 ∆t  ∗  ∆t  n−1 ∆t n−1/2
1+ A1 w = 1 − A2 w + f
2 2 2 (9.23)
 ∆t  n  ∆t  ∗ ∆t n−1/2
1+ A2 w = 1 − A1 w + f .
2 2 2
9.3. LOD Methods for the Heat Equation 319

General ADI procedure

Consider a parabolic problem of the form
m
X
ut + Ai u = f, (x, t) ∈ Ω × J, (9.24)
i=1

with an appropriate initial data and boundary condition. If A = A1 + · · · + Am ,

then the basic Crank-Nicolson approximation to (9.24) is given by
wn − wn−1 1
+ A(wn + wn−1 ) = f n−1/2 , n ≥ 1. (9.25)
∆t 2
(Here, we are interested in the time discretization of (9.24); consequently, we
shall ignore spatial discretization for the moment.)
The Douglas-Gunn algorithm [15] for ADI time discretization of (9.24) is as
follows: For κ = 1, . . . , m, find wn,κ such that
κ m
wn,κ − wn−1 1 X n,i n−1
X
+ Ai (w + w ) + Ai wn−1 = f n−1/2 , (9.26)
∆t 2 i=1 i=κ+1

and then to set

wn = wn,m . (9.27)
In the above,
m
X
Ai wn−1 := 0.
m+1
320 Chapter 9. Locally One-Dimensional Methods

The Douglas-Gunn algorithm equivalently reads

m
 ∆t  n,1  ∆t X 
1+ A1 w = 1− A1 − ∆t Ai wn−1 + ∆tf n−1/2 ,
2 2 i=2
 ∆t  ∆t (9.28)
1+ Aκ wn,κ = wn,κ−1 + Aκ wn−1 , κ = 2, . . . , m,
2 2
n n,m
w = w .

Splitting error: The intermediate values wn,1 , · · · , wn,m−1 can be eliminated

by recursively operating on the second equation of (9.28) by (1 + ∆t 2 Aκ ) for
κ = m − 1, · · · , 1:
wn − wn−1 1
+ A(wn + wn−1 ) + B∆t (wn − wn−1 ) = f n−1/2 , (9.29)
∆t 2
where
∆t X ∆t2 X
B∆t = Ai1 Ai2 + Ai1 Ai2 Ai3
4 1≤i <i ≤m 8 1≤i <i <i
1 2 1 2 3 ≤m (9.30)
m−1
∆t
+ ··· + A1 A2 · · · Am .
2m
The splitting perturbation is given by B∆t (wn − wn−1 ), and for sufficiently
smooth solutions u,
B∆t (un − un−1 ) = O(∆t2 ), (9.31)
which is of the same order in ∆t as the Crank-Nicolson truncation error. But
the splitting error can be much larger than the truncation error as shown in
the following.
9.3. LOD Methods for the Heat Equation 321

9.3.2. Accuracy of the ADI: Two examples

Let Ω × J = (0, 1)2 × (0, 1), a = α1 ≡ 1, and c = α2 ≡ 0 in (9.8). Consider two
different solutions:
u+ = sin(2πνt t) + sin(2πνx x) + sin(2πνy y),
(9.32)
u× = sin(2πνt t) · sin(2πνx x) · sin(2πνy y).

For the moment, take νt = νx = νy = 1.

The sources f and g are evaluated so that (9.8) is satisfied. Also, let n :=
nt = nx = ny . To compare computation cost and accuracy, we implemented
three algorithms:

• an LU-based algorithm,
• a PCG-ILU0 procedure for the Crank-Nicolson equation derivable from
(9.9), and
• the ADI procedure of (9.19).

Here, PCG-ILU0 denotes the conjugate gradient method preconditioned by

the zero-level (not allowing fill-in) incomplete LU-factorization. The PCG-
ILU0 procedure was initialized at each time level by the extrapolation

un,0 = 2un−1 − un−2 , n ≥ 2,

and the iteration stopped when the residual was reduced by a factor of 10−5 .
322 Chapter 9. Locally One-Dimensional Methods

n = 40 n = 80 n = 160
CPU L2 -error CPU L2 -error CPU L2 -error
LU-based 0.74 4.10e-3 9.07 1.00e-3 126 2.47e-4
PCG-ILU0 0.46 4.11e-3 5.67 1.00e-3 53.4 2.47e-4
ADI 0.26 4.10e-3 2.16 1.00e-3 17.9 2.47e-4

Table 9.1: The performances of the LU-based, PCG-ILU0, and ADI methods
for u = u+ . The elapsed time (CPU) is measured in seconds and the L2 -norm
of the error is evaluated at t = 1.
n = 40 n = 80 n = 160
CPU L2 -error CPU L2 -error CPU L2 -error
LU-based 0.91 2.46e-4 10.5 5.98e-5 136 1.47e-5
PCG-ILU0 0.83 2.46e-4 12.5 5.97e-5 121 1.42e-5
ADI 0.45 8.44e-3 3.62 2.02e-3 29.0 4.90e-4

Table 9.2: The performances of the LU-based, PCG-ILU0, and ADI methods
for u = u× .

Table 9.1 presents the elapsed times and numerical errors for u = u+ for
various grid sizes. As one can see from the table, the three different algo-
rithms show the same errors and their second-order convergence.
Table 9.2 shows the results for u = u× . The computation cost for the ADI
method increases linearly as the number of grid points grows, while the PCG-
ILU0 calculation shows a slight superlinearity in its computation cost. How-
ever, the ADI method produces an error approximately 34 times larger than
that for the LU-based or PCG-ILU0 methods for the same grid size.
9.3. LOD Methods for the Heat Equation 323

Truncation error vs. splitting error: The truncation error for the Crank-
Nicolson difference equation is of the form
4 4 3
     
∂ u ∂ u ∂ u
O h2x 4 + O h2y 4 + O ∆t2 3 ,
∂x ∂y ∂t
while the splitting error of the ADI method is
2 2
 
∂ ∂ ∂
O ∆t2 2 2 u .
∂x ∂y ∂t
This is, roughly speaking, why the ADI method introduces no splitting error
for u+ and a large splitting error for u× .
Now, since the operators Ai usually represent second-order differential op-
erators in an xi direction, it should not be surprising that the higher-order
derivatives in B∆t contribute bigger errors than the truncation error. We shall
see in §9.3.4 that it is not only possible but also quite feasible to modify the
algorithm (9.26) in a rather simple fashion to reduce the splitting error to
O(∆t3 ).
324 Chapter 9. Locally One-Dimensional Methods

9.3.3. The general fractional step (FS) procedure

We shall consider the same parabolic problem (9.24) for a FS time discretiza-
tion. For reasons that will appear below, it is not the usual case to look for
an FS procedure based on the Crank-Nicolson equation (9.25); however, it is
useful for us to do so.
The appropriate FS algorithm is given by
wn,1 − wn−1 1
+ A1 (wn,1 + wn−1 ) = f n−1/2 ,
∆t 2
n,κ n,κ−1
w −w 1
+ Aκ (wn,κ + wn−1 ) = 0, κ = 2, . . . , m − 1, (9.33)
∆t 2
n n,m−1
w −w 1
+ Am (wn + wn−1 ) = 0.
∆t 2
Equivalently,
 ∆t  n,1  ∆t  n−1
1+ A1 w = 1− A1 w + ∆tf n−1/2 ,
2 2
 ∆t  n,κ ∆t
1+ Aκ w = wn,κ−1 − Aκ wn−1 , κ = 2, . . . , m − 1, (9.34)
2 2
 ∆t  n ∆t
1+ Am w = wn,m−1 − Am wn−1 .
2 2
9.3. LOD Methods for the Heat Equation 325

Splitting error of FS procedure: Again, the intermediate values can be

eliminated:
wn − wn−1 1
+ A(wn + wn−1 ) + B∆t (wn + wn−1 ) = f n−1/2 , (9.35)
∆t 2
with B∆t being the same as for the ADI; see (9.30).
Thus, for the Crank-Nicolson version of the FS method, the splitting per-
turbation term becomes B∆t (wn + wn−1 ). We know that

B∆t (un + un−1 ) = O(∆t); (9.36)

i.e., the splitting error term is worse than the inherent local error in the
Crank-Nicolson equation.
This is the reason that (9.33) is not common; the FS methods have been
employed for the backward Euler method rather than the Crank-Nicholson
method. However, we shall be able to modify the procedure (9.33) in an
equally simple fashion to reduce the splitting error to O(∆t3 ) below.
326 Chapter 9. Locally One-Dimensional Methods

9.3.4. Improved accuracy for LOD procedures

We present a strategy to reduce the perturbation error of ADI and FS proce-
dures and essentially to recover the accuracy of the Crank-Nicolson difference
equation for an additional computational cost that is a small fraction of the
standard ADI or FS cost.
Correction term for the ADI method: Observation from (9.26),
(9.29), and (9.30) is that

if the right hand side term of (9.26) is f n−1/2 , then the right hand side
of (9.29) is also f n−1/2 and the splitting error is given by B∆t (wn −
wn−1 ).

If we could add B∆t (wn − wn−1 ) to the right hand side of (9.29), then we
could cancel the perturbation term completely; but since we do not know wn ,
we cannot make this modification in the algorithm.
Our best estimate for (wn − wn−1 ) is (wn−1 − wn−2 ).
9.3. LOD Methods for the Heat Equation 327

Modification of the ADI: Let us modify the ADI algorithm to the fol-
lowing: For n ≥ 2,
n
FAD = f n−1/2 + B∆t (z n−1 − z n−2 ),
   m 
∆t ∆t X
1+ A1 z n,1 = 1− A1 − ∆t Ai z n−1 + ∆t FAD n
,
2 2 i=2 (9.37)
 
∆t ∆t
1+ Aκ z n,κ = z n,κ−1 + Aκ z n−1 , κ = 2, . . . , m,
2 2
zn n,m
= z .

The evaluation of z 1 will be discussed below by interpreting the modified

method as an iterative procedure; for practical purposes, assume that z 1 is
obtained by solving the Crank-Nicolson equation for this single time step.
328 Chapter 9. Locally One-Dimensional Methods

Splitting error: By eliminating the intermediate values (or referring to

(9.29)), we see that z n satisfies
z n − z n−1 1
+ A(z n + z n−1 ) + B∆t (z n − 2z n−1 + z n−2 )
∆t 2 (9.38)
= f n−1/2 , n ≥ 2.

Now, for a smooth solution u of (9.8),

B∆t (un − 2un−1 + un−2 ) = O(∆t3 ), (9.39)

and the splitting error is now higher order in ∆t than the truncation error of
the Crank-Nicolson equation.
We shall both prove the convergence of the solution of (9.37) to that of (9.8)
under certain circumstances and demonstrate that the error in the solution of
(9.37) is reduced essentially to that of the Crank-Nicolson procedure for the
example u× considered above, for which the splitting error was many times as
large as the Crank-Nicolson error.
9.3. LOD Methods for the Heat Equation 329

Algebraic interpretation: We will interpret (9.38) as the iterative proce-

dure related to the matrix splitting [67]
∆t  ∆t 
1+ A= 1+ A + B∆t − B∆t .
2 2
Consider the algorithm: Find ζ , ≥ 1, by recursively solving
 ∆t 
−1
 ∆t 
1+ A + B∆t ζ = B∆t ζ + 1 − A γ + f n−1/2 . (9.40)
2 2
The solution wn of the original ADI method (9.26) is the first iterate ζ 1 of (9.40)
for γ = wn−1 starting with the initial value

ζ 0 = wn−1 . (9.41)

On the other hand, the solution z n of (9.37) is the first iterate of (9.40) with
γ = z n−1 and the initial value

ζ 0 = 2z n−1 − z n−2 . (9.42)

Consequently, the algorithm (9.37) is called the alternating direction implicit

method with improved initialization (ADI-II) [16].
If the general time step code for (9.37) is written to perform the iteration
(9.40), then, for n ≥ 2, (9.42) would be used to initialize the “iteration" and one
step of iteration calculated, while for n = 1, (9.41) would be used to initialize
the iteration and two or more iterations would give z 1 to the desired accuracy.
330 Chapter 9. Locally One-Dimensional Methods

Reformulation of ADI-II: As for ADI, ADI-II (9.37) can be formu-

lated in a various way. For the 2D problem (m = 2), the ADI-II routine in
ADI_HEAT.CF.tar is implemented based on
 ∆t  n,1  ∆t  n−1
I+ A1 z = I− A z + ∆tf n−1/2
2 2
∆t2 (9.43)
+ A1 A2 (2z n−1 − z n−2 ),
4
 ∆t  n
I+ A2 z = z n,1 .
2
• It might seem reasonable to use a higher-order extrapolation than (9.42),
but experiments have shown that instability can result unless the time
step is small enough.
• It has also been observed that (9.42) can over-correct for large time steps,
and it is possible that the use of

ζ 0 = z n−1 + η(z n−1 − z n−2 ), 0 ≤ η ≤ 1, (9.44)

could lead to better computational results for large time steps.

• However, experiments have shown that, when the time step is reason-
ably chosen (e.g., ∆t . ah), ADI-II methods have worked better than
ADI methods for various heterogeneous media; see Tables 9.3 and 9.4
in §9.3.6. So, (9.44) does not seem necessary for solving heat equations in
practice.
9.3. LOD Methods for the Heat Equation 331

Correction term for the FS method

The FS difference equation (9.35) preserves the right hand side of the FS
algorithm (9.34) and exhibits the splitting perturbation B∆t (wn +wn−1 ). Modify
(9.34) as follows. For n ≥ 2, let

FFnS = f n−1/2 + B∆t (3z n−1 − z n−2 ),

   
∆t ∆t
1+ A1 z n,1 = 1− A1 z n−1 + ∆t FFnS ,
2  2
(9.45)

∆t ∆t
1+ Aκ z n,κ = z n,κ−1 − Aκ z n−1 , κ = 2, · · · , m − 1,
 2  2
∆t ∆t
1+ Am z n = z n,m−1 − Am z n−1 .
2 2

After the intermediate values are eliminated, we see that z n satisfies

z n − z n−1 1
+ A(z n + z n−1 ) + B∆t (z n − 2z n−1 + z n−2 ) = f n−1/2 , (9.46)
∆t 2
which is identical to the equation (9.38) satisfied by the solution of the ADI-II
algorithm (9.37).
332 Chapter 9. Locally One-Dimensional Methods

Remarks [16]:

• We have not only shown how to reduce the splitting errors for the ADI
and FS methods but also discovered that their improved procedures lead
to identical results “(after several decades of being considered to be
different techniques)."
• Again, it is advisable to obtain z 1 as discussed earlier.
• If the values of Ai z n−1 are saved, then there is essentially no difference in
the implementation of algorithms (9.37) and (9.45). That being the case,
we shall address both algorithms as pertaining to the ADI-II method.
9.3. LOD Methods for the Heat Equation 333

9.3.5. A convergence proof for the ADI-II

Let k · k denote the L2 (Ω) or 2 (Ω) norm and k · k1 the norm on either H 1 (Ω) or
h1 (Ω), as appropriate. (That is, depending on spatial discretization by finite
elements or finite differences.) Assume that the operators {Ai } commute:
Ai Aj = Aj Ai , i, j = 1, . . . , m, (9.47)
and that
(Ai z, z) ≥ αkzk21 , α > 0. (9.48)
By (9.47) and (9.48), it follows that
(B∆t z, z) ≥ 0.

Let ∂ t v n = (v n − v n−1 )/∆t and en = un − z n . Then, the error equation associ-

ated with ADI-II (9.38) is
1
∂ t en + A(en + en−1 ) + B∆t (en − 2en−1 + en−2 ) = δ n , (9.49)
2
where δ n is the truncation error on the n-th level, i.e.,
δ n = O(∆t2 + hp ), p ≥ 2, (9.50)
for any reasonable spatial discretization. Choose ∂ t en as a test function. Then,
for n ≥ 2,
n n 1 2
 2 n

A(e + e ), ∂ t e + ∆t B∆t ∂ t e , ∂ t e = (δ n , ∂ t en ). (9.51)
n n−1 n n


(∂ t e , ∂ t e ) +
2
Multiply (9.51) by ∆t and sum beginning at n = 2 to have
n n
X 1 X 2
k∂ t e k ∆t + (Aen , en ) + ∆t2
j 2
(B∆t ∂ t ej , ∂ t ej )∆t
j=2
2 j=2
n (9.52)
1 X
= (Ae1 , e1 ) + (δ j , ∂ t ej )∆t.
2 j=2

Now, since b2 − ab ≥ (b2 − a2 )/2, we have

n n
X 2 X
(B∆t ∂ t ej , ∂ t ej )∆t = (B∆t [∂ t ej − ∂ t ej−1 ], ∂ t ej )
j=2 j=2 (9.53)
1 1
≥ (B∆t ∂ t en , ∂ t en ) − (B∆t ∂ t e1 , ∂ t e1 ).
2 2
334 Chapter 9. Locally One-Dimensional Methods

Apply the inequality (|ab| ≤ (a2 +b2 )/2) to the last term in (9.52). Then utilizing
(9.53), one can obtain the following inequality:
n
X
k∂ t ej k2 ∆t + (Aen , en ) + ∆t2 (B∆t ∂ t en , ∂ t en )
j=2
n
X (9.54)
≤ kδ j k2 ∆t + (Ae1 , e1 ) + ∆t2 (B∆t ∂ t e1 , ∂ t e1 ), n ≥ 2.
j=2

Thus, the estimation of the error generated by the ADI-II method is, in
the commutative case, reduced to bounding the errors in z 0 and z 1 , thereby
emphasizing the remarks above on the evaluation of z 1 . Try to compare the
above analysis with (9.16) when θ = 1/2.
9.3. LOD Methods for the Heat Equation 335

9.3.6. Accuracy and efficiency of ADI-II

To check the accuracy and efficiency of the ADI-II algorithm, let us choose
the domain Ω = (0, 1)2 and the time interval J = (0, 1], along with the four
diffusion coefficients
a1 (x, y) = 1,
a2 (x, y) = 1/(2 + cos(3πx) · cos(2πy)),
1 + 0.5 · sin(5πx) + y 3 ,

if x ≤ 0.5,
a3 (x, y) = 2 3 (9.55)
1.5/(1 + (x − 0.5) ) + y , else,
 
a2 (x, y) 0
a4 (x, y) = .
0 a3 (x, y)

• The first time step to obtain z 1 for the ADI-II was made by following the
w1 -ADI calculation by SOR iterations to get the Crank-Nicolson value.
• Here, we compare the results of four different algorithms, namely the
LU-based, PCG-ILU0, ADI, and ADI-II methods.
336 Chapter 9. Locally One-Dimensional Methods

a = a1 a = a2 a = a3
CPU L2 -error CPU L2 -error CPU L2 -error
LU-based 23.6 1.10e-3 27.2 3.52e-3 24.2 5.35e-3
PCG-ILU0 21.6 1.09e-3 24.0 3.52e-3 24.7 5.36e-3
ADI 7.14 1.70e-2 10.9 1.02e-2 7.91 2.67e-2
ADI-II 7.77 1.10e-3 11.3 3.54e-3 8.46 5.35e-3

Table 9.3: The performances of the LU-based, PCG-ILU0, ADI, and ADI-II
methods with c = α2 ≡ 0, νt = 1, νx = 4, νy = 3, nx = ny = nt = 100 for u = u× .
∆t = 2h ∆t = h ∆t = h/2 ∆t = h/4
CPU L2 -error CPU L2 -error CPU L2 -error CPU L2 -error
LU-based 28.4 2.12e-3 49.6 2.13e-3 92.1 2.13e-3 176 2.13e-3
PCG-ILU0 24.9 2.14e-3 36.5 2.15e-3 57.6 2.14e-3 96.8 2.13e-3
ADI 8.19 2.01e-1 16.3 6.76e-2 32.4 1.75e-2 64.5 4.86e-3
ADI-II 8.80 1.10e-2 16.9 2.17e-3 33.2 2.13e-3 66.1 2.13e-3

Table 9.4: The performances of the LU-based, PCG-ILU0, ADI, and ADI-II
methods with a = a4 , c = α2 ≡ 0, νt = 2.0, νx = 6.25, νy = 7, h = hx = hy = 1/120,
and u = u× .

Table 9.3 presents the performances of the four algorithms for the first
three diffusion coefficients in (9.55) for u = u× with νt = 1, νx = 4, and νy = 3.
The error for the ADI method is 16, 3, and 5 times larger than the Crank-
Nicolson error for a = a1 , a2 , and a3 , respectively. The ADI-II method requires
only about 5-7% extra cost over the ADI method and its accuracy hardly differs
from that of the direct, LU-based solver, when ∆t ≤ h.
Table 9.4 shows numerical results for various time steps, when a = a4 (an
anisotropic diffusivity), c = α2 ≡ 0, νt = 2, νx = 6.25, and νy = 7, and h = hx =
hy = 1/120. The ADI calculations show large splitting errors, even for small
time steps. Here again the improved initialization (9.42) greatly improves
the accuracy of the alternating direction procedure, for a few percent of extra
cost. However, as one can see from the table, the ADI-II algorithm generates a
splitting error that is a few times the Crank-Nicolson error for ∆t = 2h. Thus
one has to choose ∆t sufficiently small, although the splitting error is O(∆t3 ).
9.4. Homework 337

9.4. Homework
1. Show that all of (9.19), (9.20), and (9.23) are equivalent to each other.
Count and compare the required operations for (9.20) and (9.23) in each
time level.
2. Show that (9.28) is equivalent to (9.29)-(9.30), for m = 3.
3. Check if (9.37) is equivalent to (9.43), when m = 2. Count to compare the
required operations for them.
4. The given code in Matlab is an implementation for the ADI (9.20) solving
the heat equation in 2D. Adjust the code for ADI-II (9.37) with m = 2.
(a) The major step you should fulfill is to adjust F in xy_sweeps.m.
(b) Perform error analysis comparing errors from ADI and ADI-II.
(c) Report your additions to the code.
338 Chapter 9. Locally One-Dimensional Methods
Chapter 10

Special Schemes

In this chapter, we will deal with

• Absorbing boundart conditions (ABCs) for wave propagation

• Numerical techniques for PDE-based image processing
• ...

339
340 Chapter 10. Special Schemes

10.1. Wave Propagation and Absorbing Bound-

ary Conditions
10.1.1. Introduction to wave equations
Wave equations are often imposed by a suitable radiation condition at infinity.
Such problems can be solved numerically by
• first truncating the given unbounded domain,
• imposing a suitable ABC on the boundary of the truncated bounded do-
main,
• approximating the resulting problem by discretization methods such as
finite differences and finite element methods, and then
• applying computational algorithms to the resulting algebraic system.
Let Ω ⊂ Rm , 1 ≤ m ≤ 3, be a bounded domain with its boundary Γ = ∂Ω and
J = (0, T ], T > 0. Consider
1
(a) utt − ∆u = S(x, t), (x, t) ∈ Ω × J,
v2
1 (10.1)
(b) ut + uν = 0, (x, t) ∈ Γ × J,
v
(c) u(x, 0) = g0 (x), ut (x, 0) = g1 (x), x ∈ Ω,
where v = v(x) > 0 denotes the normal velocity of the wavefront, S is the
wave source/sink, ν denote the unit outer normal from Γ, and g0 and g1 are
initial data. Equation (10.1.b) is popular as a first-order absorbing boundary
condition (ABC), since introduced by Clayton and Engquist [9]. We will call
(10.1.b) the Clayton-Engquist ABC (CE-ABC).
Equation (10.1) has been studied extensively as a model problem for second-
order hyperbolic problems; see e.g. [2, 7, 10, 46, 61]. It is often the case that
the source is given in the following form
S(x, t) = δ(x − xs )f (t),
where xs ∈ Ω is the source point. For the function f , the Ricker wavelet of
frequency λ can be chosen, i.e.,
2 2 2
f (t) = π 2 λ2 (1 − 2π 2 λ2 t2 ) e−π λ t
. (10.2)
10.1. Wave Propagation and Absorbing Boundary Conditions 341

10.1.2. Absorbing boundary conditions (ABCs)

The CE-ABC (10.1.b) has been studied and applied widely, representing a
first-order ABC which allows normally incident waves to pass out of Ω trans-
parently. Various other ABCs have been introduced to absorb the energy pass-
ing the boundary more effectively.
Consider the Fourier transform (time to frequency) of the CE-ABC (10.1.b):

iω
u
b+ubν = 0, (10.3)
v
where i is the imaginary unit, ω (:= 2πλ) denotes the angular frequency, and
ˆ ∞
1
b(x, ω) = √
u u(x, t)e−iωt dt.
2π −∞
In order to suppress the boundary reflection, Kim et al. [43] introduced the
following ABC
iω τν u
b+u
bν = 0, (10.4)
where τ is an appropriate solution of the eikonal equation
1
|∇τ | = , τ (xs ) = 0, (10.5)
v
which can be solved effectively by employing optimal solvers such as the group
marching method (GMM) [39] and a high-order ENO-type iterative method
[40].
For the time domain simulation of the acoustic waves, we apply the inverse
Fourier transform to (10.4) to obtain
τν ut + uν = 0, (10.6)
which will be called the traveltime ABC (TT-ABC). Note that τν ≥ 0 for out-
going waves and
cos θ
τν = ∇τ · ν = |∇τ | cos θ = ,
v
where θ is the angle of the wave measured with respect to the normal of the
boundary. Thus the TT-ABC is a canonical form of the first-order ABC [29].
For normally incident wavefronts, τν = |∇τ | and therefore the TT-ABC (10.6)
acts like the CE-ABC (10.1.b).
342 Chapter 10. Special Schemes

• See Engquist-Majda [22] and Higdon [29, 30] for a hierarchy of ABCs
which approximate the nonlocal, pseudodifferential ABC [21].
• See [28, 31, 49, 66] for recent strategies for effective ABCs.

10.1.3. Waveform ABC

In this subsection, we introduce a new ABC which incorporates local wave-
form information in order to accurately estimate the incident angles of wave-
fronts, without using the first-arrival traveltime.
We begin with an observation that ∇τ is parallel to ∇u (in acoustic media).
Thus, since |∇τ | = 1/v, we have
1 ∇u
∇τ = ± . (10.7)
v |∇u|
Recall that τν ≥ 0 for out-going wavefronts. Hence it follows from (10.7) that
1 |uν |
τν = ∇τ · ν = . (10.8)
v |∇u|
Note that the above equation must be satisfied for every wavefront that ap-
proaches to the boundary, including multiple arrivals. Thus an effective ABC
can be formulated as follows:
1 |uν |
ut + uν = 0, (10.9)
v |∇u|
which we will call the waveform ABC (WF-ABC).
Remarks:
• The TT-ABC (10.6) must be identical to the WF-ABC (10.9) for the first
arrival. However, for later arrivals having different incident angles, the
TT-ABC may introduce a large boundary reflection. The WF-ABC is de-
signed in such a way that all wavefronts can pass out of the domain with
no noticeable reflection.
• Since it is in the form of first-order ABCs, it can be easily implemented as
a stable boundary condition.
• For normally incident wavefronts, we have |uν | = |∇u| and therefore the
WF-ABC acts like the CE-ABC (10.1.b).
10.1. Wave Propagation and Absorbing Boundary Conditions 343

Approximation of WF-ABC: Here we present numerical strategies for

the approximation of the WF-ABC.
For example, let Ω = (0, 1)2 and ∆x = 1/nx , ∆y = 1/ny , for some positive
integers nx and ny ; let the grid points be given as

xij = (xi , yj ) := (i∆x, j∆y), i = 0, 1, · · · , nx , j = 0, 1, · · · , ny .

Let ∆t be the timestep and tn = n∆t.

Assume that we have computed uk (≈ u(·, tk )), k ≤ n, and un+1 is to be ob-
tained. Then, we may approximate (10.9) as
n+1
1 n u − un−1 |unν |
Q(u ) + (∇h un ) · ν = 0, n
Q(u ) ≈ , (10.10)
v 2∆t |∇un |
where ∇h is an spatial approximation of ∇. Here the quantity Q(un ) must
evaluate accurately the cosine of the incident angle of the wavefront.
344 Chapter 10. Special Schemes

Figure 10.1: A boundary point B and a corner point C.

Let Ω = (ax , bx ) × (ay , by ) and

∆x = (bx − ax )/nx , ∆y = (by − ay )/ny ,
for some positive integers nx and ny ; let the grid points be given as
xij = (xi , yj ) := (i∆x, j∆y), i = 0, · · · , nx , j = 0, · · · , ny .
For the boundary points B and C as in Figure 10.1, we may apply difference
schemes to determine un .
• For both B and C, the second-order FDM approximates the main equa-
tion (10.1.a) as
1 un+1 n n−1
O − 2uO + uO −unW + 2unO − unE
+
v2 ∆t2 ∆x2
(10.11)
−unS + 2unO − unN
+ 2 = SOn .
∆y
• For the point B, unS is a ghost value to be eliminated. The WF-ABC
(10.10) reads
n+1 n−1
1 n uO − uO unS − unN
QS (u ) + = 0, (10.12)
v 2∆t 2∆y
where QS (un ) = | − uny |/|∇un |.
2
Perform (10.11)+ (10.12) and then solve the resulting equation for un+1 O at the poi
∆y
h 1 QS (un ) i n+1 2unO − un−1 O QS (un ) n−1
+ u = + u
v 2 ∆t2 v∆t∆y O v 2 ∆t2 v∆t∆y O
−unW + 2unO − unE 2unO − 2unN
+SOn − − .
∆x2 ∆y 2
10.1. Wave Propagation and Absorbing Boundary Conditions 345

Multiplying both sides of the above equation by v 2 ∆t2 , we reach at

(At the boundary point B):
h QS (un ) i n+1
1 + v∆t uO = (2unO − un−1
O )
∆y
QS (un ) n−1 (10.13)
+v∆t uO
∆y
2 2
h
n −unW + 2unO − unE 2unO − 2unN i
+v ∆t SO − − .
∆x2 ∆y 2

• For the point C, unS and unW are ghost values to be eliminated. The WF-
ABC (10.10) reads

1 un+1 − un−1 un − unE

(a) QW (un ) O O
+ W = 0,
v 2∆t 2∆x
n+1 n−1
(10.14)
1 n uO − uO unS − unN
(b) QS (u ) + = 0,
v 2∆t 2∆y

where QW (un ) = | − unx |/|∇un |.

2 2
Perform (10.11)+ (10.14.a)+ (10.14.b) and then solve the resulting
∆x ∆y
n+1
equation for uO at the point C:

h 1 QW (un ) QS (un ) i n+1 2unO − un−1

O
2 2 + + uO = 2 2
v ∆t v∆t∆x v∆t∆y v ∆t
 Q (un ) Q (un ) 
W S
+ + un−1
v∆t∆x v∆t∆y O
n 2unO − 2unE 2unO − 2unN
+SO − − .
∆x2 ∆y 2

Multiplying both sides of the above equation by v 2 ∆t2 , we reach at

346 Chapter 10. Special Schemes

(At the corner point C):

h  Q (un ) Q (un ) i
W S
1 + v∆t + un+1
O = (2unO − un−1
O )
∆x ∆y
 Q (un ) Q (un ) 
W S
(10.15)
+v∆t + un−1
O
∆x ∆y
2 2

n 2unO − 2unE 2unO − 2unN 
+v ∆t SO − − .
∆x2 ∆y 2
Chapter 11

Projects∗

11.1. High-order FEMs for PDEs of One Spacial

Variable
The provided Python code is implemented for solving
−uxx = f, x ∈ (a, b)
(11.1)
u = g, x = a, b,

using high-order Galerkin FE methods.

Through the project, you will modify the code for the numerical solution of
more general problems of the form
−(Kux )x + ru = f, x ∈ (a, b)
(11.2)
Kuν = g, x = a, b,

where K = K(x) and r are prescribed continuous positive functions.

347
348 Chapter 11. Projects∗

Here are your objectives:

• Derive Galerkin FEMs for (11.2) of Neumann boundary conditions.

• Modify the code for the problem. You may have to spend a certain amount
of time to understand the code. Please save new functions in a new file;
do not add any extra functions to util_FEM_1D.py.
• Test your code for its convergence, for example, for

– (a, b) = (0, π)
– K(x) = 1 + x
– r(x) ≡ 1
– The exact solution u(x) = sin(x).

You have to set f and g correspondingly; for example, g(0) = 1 and g(π) =
−(1 + π).
• Report your results by Tue Nov 24, 2015, in hard copies, including new
functions (you implemented) and convergence analysis. The project is
worth 100 points.
Appendix A

Basic Concepts in Fluid Dynamics

Physical properties of fluid flow under consideration must be known if one is

to either study fluid motion or design numerical methods to simulate it. This
appendix is devoted to introducing basic concepts of fluid flows.

A.1. Conservation Principles

Conservation laws can be derived by considering a given quantity of matter or
control mass (CM) and its extensive properties such as mass, momentum, and
energy. This approach is used to study the dynamics of solid bodies, where the
CM is easily identified. However, it is difficult to follow matter in fluid flows.
It is more convenient to deal with the flow in a certain spatial region, called
the control volume (CV).
We first consider the conservation laws for extensive properties: mass and
momentum. For mass, which is neither created nor destroyed, the conserva-
tion equation reads
dm
= 0, (A.1)
dt
where t is time and m represents mass. On the other hand, the momentum can
be changed by the action of forces and its conservation equation is Newton’s
second law of motion
d(mv) X
= f, (A.2)
dt
where v is the fluid velocity and f is forces acting on the control mass.

349
350 Appendix A. Basic Concepts in Fluid Dynamics

We will reformulate these laws with incorporation of the control volume.

The fundamental variables will be intensive, rather than extensive, properties
that are independent of the amount of matter. Examples are density ρ (mass
per unit volume) and velocity v (momentum per unit mass).
For any intensive property φ, the corresponding extensive property Φ is by
definition given as ˆ
Φ= ρφ dΩ, (A.3)
ΩCM
where ΩCM is the volume occupied by the CM. For example, φ = 1 for mass
conservation, φ = v for momentum conservation, and for a scalar property, φ
represents the conserved property per unit mass. Using (A.3), the left hand
side of each of conservation equations, (A.1) and (A.2), can be written as
ˆ ˆ ˆ
d d
ρφ dΩ = ρφ dΩ + ρφ(v − vb ) · n dS, (A.4)
dt ΩCM dt ΩCV ∂ΩCV

where ΩCV is the CV, n denotes the unit outward normal to ∂ΩCV , dS repre-
sents the surface element, v is the fluid velocity, and vb denotes the velocity
of the CV surface ∂ΩCV . The equation (A.4) is called the control volume equa-
tion or the Reynolds’s transport equation. For a fixed CV, vb = 0 and the first
derivative on the right hand side of (A.4) becomes a local (partial) derivative:
ˆ ˆ ˆ
d ∂
ρφ dΩ = ρφ dΩ + ρφ v · n dS. (A.5)
dt ΩCM ∂t ΩCV ∂ΩCV

Note that the material derivative applied to the control volume is

d ∂
= + vb · ∇.
dt ∂t
For a detailed derivation of this equation, see e.g. [54, 69].

A.2. Conservation of Mass

The integral form of the mass conservation equation follows from the control
volume equation (A.5), by setting φ = 1:
ˆ ˆ
∂
ρdΩ + ρv · n dS = 0, (A.6)
∂t Ω ∂Ω
A.3. Conservation of Momentum 351

where we have omitted the subscript CV from Ω. The above equation is also
called the continuity equation. Recall the Gauss’s divergence theorem
ˆ ˆ
∇ · A dΩ = A · n dS, (A.7)
Ω ∂Ω
for any vector field A defined in the control volume Ω. Applying (A.7) to (A.6)
and allowing the CV to become infinitesimally small, we have the following
differential coordinate-free form of the continuity equation
∂ρ
+ ∇ · (ρv) = 0, (A.8)
∂t
and its Cartesian form
∂ρ ∂(ρvi ) ∂ρ ∂(ρu) ∂(ρv) ∂(ρw)
+ = + + + = 0, (A.9)
∂t ∂xi ∂t ∂x ∂y ∂z
where xi (i = 1, 2, 3) or (x, y, z) are the Cartesian coordinates and vi or (u, v, w)
are the Cartesian components of the velocity v. Here we have utilized the
Einstein convention that whenever the same index appears twice in any term,
summation over the range of that index is applied.

A.3. Conservation of Momentum

Using (A.2) and (A.5) with φ = v, one can obtain the integral form of the
momentum conservation equation
ˆ ˆ
∂ X
ρv dΩ + ρv v · n dS = f. (A.10)
∂t Ω ∂Ω
The right hand side consists of the forces:
– surface forces: pressure, normal and shear stresses, surface tension, etc.;
– body forces: gravity, electromagnetic forces, etc..
The surface forces due to pressure and stresses are the microscopic momen-
tum flux across the surface. For Newtonian fluids, the stress tensor T , which
is the molecular transport rate of momentum, reads
  
2
T = 2µD + κ − µ ∇ · v − p I, (A.11)
3
352 Appendix A. Basic Concepts in Fluid Dynamics

where p is the static pressure, µ and κ are respectively the shear coefficient
of viscosity and the bulk coefficient of viscosity, I is the unit (identity) tensor,
and D is the rate of strain (deformation) tensor defined by
1 T

D= ∇v + (∇v) . (A.12)
2
The following notation is often used in the literature to denote the viscous
part of the stress tensor
  
2
τ = 2µD + κ − µ ∇ · v I. (A.13)
3
Thus the stress tensor can be written as

T = τ − pI (A.14)

and its components read

Tij = τij − pδij , (A.15)
where    
2 1 ∂vi ∂vj
τij = 2µDij + κ − µ δij ∇ · v, Dij = + .
3 2 ∂xj ∂xi
Assume that gravity g is the only body force. Then, the integral form of the
momentum conservation equation becomes
ˆ ˆ ˆ ˆ
∂
ρv dΩ + ρv v · n dS = T · n dS + ρg dΩ. (A.16)
∂t Ω ∂Ω ∂Ω Ω

A coordinate-free vector form of the momentum conservation equation is read-

ily obtained by applying the Gauss’s divergence theorem (A.7) to the convec-
tive and diffusive flux terms of (A.16):
∂(ρv)
+ ∇ · (ρvv) = ∇ · T + ρg. (A.17)
∂t
The continuity equation (A.8) and the momentum equations (A.17) are called
the Navier-Stokes equations.
The corresponding equation for the ith component of (A.17) is
∂(ρvi )
+ ∇ · (ρvi v) = ∇ · Ti + ρgi , (A.18)
∂t
A.3. Conservation of Momentum 353

where Ti in the Cartesian coordinates can be expressed as

    
∂vi ∂vj 2
Ti = µ + Ij + κ − µ ∇ · v − p Ii , (A.19)
∂xj ∂xi 3
where Ii is the Cartesian unit vector in the direction of the coordinate xi .
The integral form of (A.18) reads
ˆ ˆ ˆ ˆ
∂
ρvi dΩ + ρvi v · n dS = Ti · n dS + ρgi dΩ. (A.20)
∂t Ω ∂Ω ∂Ω Ω

In index notation, (A.18) can be rewritten as

∂(ρvi ) ∂(ρvj vi ) ∂p ∂τij
+ =− + + ρgi . (A.21)
∂t ∂xj ∂xi ∂xj
In approximating the momentum equations by finite difference schemes, it is
often more convenient to deal with the following non-conservative form
 
∂vi
ρ + v · ∇vi = ∇ · Ti + ρgi . (A.22)
∂t
Here we describe the momentum equations for the incompressible Newto-
nian fluid of constant density and viscosity. In this case, since ∇ · v = 0, (A.21)
becomes
∂ 2 vi
 
∂vi ∂vi ∂p
ρ + vj =− + ρgi + µ . (A.23)
∂t ∂xj ∂xi ∂xj ∂xj
In 2D Cartesian coordinates, (A.23) reads
 2
∂ v1 ∂ 2 v1
  
∂v1 ∂v1 ∂v1 ∂p
(a) ρ + v1 + v2 =− + ρg1 + µ 2
+ ,
 ∂t ∂x ∂y  ∂x  ∂x ∂y 2 
(A.24)
∂v2 ∂v2 ∂v2 ∂p ∂ 2 v2 ∂ 2 v2
(b) ρ + v1 + v2 = − + ρg2 + µ + .
∂t ∂x ∂y ∂y ∂x2 ∂y 2
Thus the complete set of the Navier-Stokes equations for incompressible ho-
mogeneous flows becomes (in Gibbs notation)
(a) ∇ · v = 0,
∂v (A.25)
(b) + (v · ∇)v = −∇p0 + g + ν∆v.
∂t
where p0 = p/ρ and ν = µ/ρ is the kinematic viscosity coefficient.
354 Appendix A. Basic Concepts in Fluid Dynamics

In the case of frictionless (inviscid) flow, i.e., µ = 0, the equation of motion

(A.25.b) reduces to the Euler’s equation,
∂v
+ (v · ∇)v = −∇p0 + g. (A.26)
∂t

A.4. Non-dimensionalization of the Navier-Stokes

Equations
Now we will discuss some scaling properties of the Navier-Stokes equations
with the aim of introducing a parameter (the Reynolds number) that mea-
sures the effect of viscosity.
Let L be a reference length L and U a reference velocity. These number are
chosen in an arbitrary way. For example, if we consider a free-stream flow
past a sphere, L can be either the radius or the diameter of the sphere and U
can be the magnitude of the fluid velocity at infinity. The choice determines a
time scale T = L/U . We measure x, v, and t as fractions of these scales, i.e.,
we introduce the following dimensionless quantities
x v t
x0 = , v0 = , t0 = .
L U T
Consider the change of variables e.g. for the x-component of the Navier-Stokes
equations in 2D Cartesian coordinates (A.24.a):
∂(U v10 ) ∂t0 0 0
∂(U v10 ) ∂y 0
 
0 ∂(U v1 ) ∂x
ρ + U v1 + U v2
∂t0 ∂t ∂x0 ∂x ∂y 0 ∂y
∂p ∂x0 ∂ 2 (U v10 ) ∂ 2 (U v10 )
=− 0 + ρg1 + µ + ,
∂x ∂x ∂(Lx0 )2 ∂(Lx0 )2
or
U 2 ∂v10 0 0
U ∂ 2 v10 ∂ 2 v10
   
0 ∂v1 0 ∂v1 1 ∂p
ρ + v1 0 + v2 0 = − + ρg1 + µ 2 + 02 .
L ∂t0 ∂x ∂y L ∂x0 L ∂x0 2 ∂y
Thus we have
∂v10 0 0
∂ 2 v10 ∂ 2 v10
 
0 ∂v1 0 ∂v1 1 ∂p L ν
+ v1 + v2 = − + g1 + + 02 .
∂t0 ∂x0 ∂y 0 ρU 2 ∂x0 U 2 LU ∂x0 2 ∂y
A.5. Generic Transport Equations 355

It is straightforward to apply the change of variables to the x-component (and

also the other ones) of the Navier-Stokes equations in 3D. It follows from the
change of variables that (A.25) becomes
(a) ∇0 · v0 = 0,
∂v0 0 0 0 0 0 0 1 0 0 (A.27)
(b) + v · ∇ v = −∇ p + g + ∆v,
∂t0 R
where
p Lg LU
p0 = , g 0
= , R = .
ρU 2 U2 ν
Here the dimensionless quantity R is the Reynolds number. The equations
(A.27) are the the Navier-Stokes equations in dimensionless variables. (The
gravity term g0 is often ignored.)
When R is very small, the flow transport is dominated by the diffusion/dissipation
and the convection term (sometimes, called inertia) v · ∇v becomes much
smaller than the diffusion term R1 ∆v, i.e.,
1
|v · ∇v| 
∆v .
R
Ignoring the convection term, we have the Stokes’s equations
(a) ∇ · v = 0,
∂v 1 (A.28)
(b) = −∇p + g + ∆v.
∂t R

A.5. Generic Transport Equations

The integral form of the equation describing conservation of a scalar quantity
φ is analogous to the previous equations and reads
ˆ ˆ
∂ X
ρφ dΩ + ρφ v · n dS = fφ , (A.29)
∂t Ω ∂Ω
where fφ represents any sources and sinks and transport of φ by mechanisms
other than convection. Diffusive transport fφd is always present and usually
expressed by a gradient approximation
ˆ
d
fφ = D∇φ · n dS, (A.30)
∂Ω
356 Appendix A. Basic Concepts in Fluid Dynamics

where D is the diffusivity for φ. The equation (A.30) is called Fick’s law for
mass diffusion or Fourier’s law for heat diffusion. Since the sources/sinks can
be expressed as ˆ
fφs = qφ dΩ,
Ω
setting fφ = fφd + fφs and applying the Gauss’s divergence theorem, one can
obtain the generic transport equation, the coordinate-free form of the equation
(A.29):
∂(ρφ)
+ ∇ · (ρφv) = ∇ · (D∇φ) + qφ . (A.31)
∂t
The lecture note will first focus on the numerical methods for (A.31). More pre-
cisely, we will consider numerical methods for the convection-diffusion equa-
tion of the form
∂c
(a) + ∇ · (vc) − ∇ · (D∇c) = f, (x, t) ∈ Ω × J,
∂t (A.32)
(b) (D∇c) · ν = 0, (x, t) ∈ Γ × J,
(c) c = c0 , x ∈ Ω, t = 0,
where c is the unknown (e.g. concentration), Ω ⊂ Rd , 1 ≤ d ≤ 3, is a bounded
domain with its boundary Γ = ∂Ω and J = (0, T ] the time interval, T > 0.
Here v = v(c) is the fluid velocity, ν is the outward normal to Γ, and f = f (c)
denotes chemical reactions and source/sink. The diffusion tensor D = D(v, c)
is symmetric and positive definite:
DT = D; D∗ |y|2 ≤ yT D(x)y ≤ D∗ |y|2 , ∀ x ∈ Ω, ∀ y ∈ Rd ,
for some positive constants D∗ and D∗ . The velocity either can be obtained
by solving another equation such as the pressure equation or is given from
experiments.
Special features of the continuity and momentum equations (Navier-Stokes
equations) will be considered afterwards as applications of the numerical meth-
ods for the generic equation.

A.6. Homework
1. Use ∇ · (ρvi v) = vi ∇ · (ρv) + ρv · ∇vi to derive (A.22) from (A.9) and (A.18).
A.6. Homework 357

2. Derive (A.23).
358 Appendix A. Basic Concepts in Fluid Dynamics
Appendix B

Elliptic Partial Differential Equations

B.1. Regularity Estimates

The quasilinear second-order elliptic equation in 2D is defined as

−∇ · (A(x)∇u) + b(x, u, ∇u) = f (x), (B.1)

where b is a general function and A is symmetric positive definite, i.e.,

 
a11 a12
A= , a11 > 0, a22 > 0, a11 a22 > a212 .
a12 a22

For simplicity, we begin with the constant coefficient linear equation

−∇ · (A∇u) + b · ∇u + cu = f, (B.2)

where b = (b1 , b2 ).
The Fourier transform in 2D reads
ˆ
1
u
b(ξ) = u(x)e−ix·ξ dx;
2π R2
its inverse formula is ˆ
1
u(x) = u(ξ)eix·ξ dξ.
2π R2
The Fourier transform satisfies the Parseval’s identity
ˆ ˆ
2
|u(x)| dx = u(ξ)|2 dξ.
|b (B.3)
R2 R2

359
360 Appendix B. Elliptic Partial Differential Equations

Let ∂x = (∂x1 , ∂x2 ), where ∂xi = ∂/∂xi , i = 1, 2. For α = (α1 , α2 ), a pair of

nonnegative integers, define

|α| = α1 + α2 , ξ α = ξ1α1 ξ2α2 , ∂xα = (∂xα11 , ∂xα22 ).

Since
|α| α
∂d
x u=i ξ u
α
b, (B.4)
equation (B.2) in its Fourier transform becomes

P (ξ) u
b(ξ) = fb(ξ), (B.5)

where
P (ξ) = ξ · Aξ + ib · ξ + c.
From the ellipticity requirements: a11 > 0, a22 > 0, and a11 a22 > a212 , we see

ξ · Aξ ≥ C0 |ξ|2 ,

for some C0 > 0. Thus there are C1 > 0 and R0 ≥ 0 such that

|P (ξ)| ≥ C1 |ξ|2 , if |ξ| ≥ R0 , (B.6)

and therefore we have

|fb(ξ)|
|b
u(ξ)| ≤ C2 , if |ξ| ≥ R0 , (B.7)
|ξ|2
for some C2 > 0. Thus, from (B.3), (B.4), and (B.7),
ˆ ˆ
2
α
|∂x u| dx = b|2 dξ
|ξ α u
R2 R2
ˆ ˆ b2
2|α| 2 2|α| |f | (B.8)
≤ |ξ| |b u| dξ + C2 |ξ| 2
dξ
|ξ|≤Rˆ0 ˆ |ξ |≥R 0
|ξ|
2|α|
≤ R0 u|2 dξ + C2
|b |ξ|2|α|−2 |fb|2 dξ.
R2 R2

For nonnegative integer s, the H s (R2 )-norm is defined as

Xˆ
kuk2s = |∂xα u|2 dx.
|α|≤s R2
B.2. Maximum and Minimum Principles 361

Then, it follows from (B.8) and the Parseval’s identity that

kuk2s+2 ≤ C(kf k2s + kuk20 ), s ≥ 0, (B.9)

for some C = C(s, A, b, c) > 0.

The inequality (B.9) is called a regularity estimate. Note that when b = 0
and c ≥ 0, (B.6) holds with R0 = 0. Thus the regularity estimate reads

kuks+2 ≤ Ckf ks , s ≥ 0, if b = 0 and c ≥ 0. (B.10)

When (B.2) is defined on bounded domain Ω ⊂ R2 whose boundary is suf-

ficiently smooth, one can obtain an interior regularity estimate of the form

kuk2s+2,Ω1 ≤ C(kf k2s,Ω + kuk20,Ω ), s ≥ 0, (B.11)

where Ω1 ⊂ Ω is such that its boundary is contained in the interior of Ω, and
the constant C = C(s, A, b, c, Ω, Ω1 ) > 0.

B.2. Maximum and Minimum Principles

This section presents the maximum and minimum principles for subharmonic
and superharmonic functions, respectively, following Gilberg and Trudinger
[26, Ch.2].
The function u is called harmonic (subharmonic, superharmonic) in Ω ⊂ Rn
if it satisfies
−∆u = 0 (≤ 0, ≥ 0), x ∈ Ω.
The following is known as the mean value theorems , which characterize har-
monic functions.
Theorem B.1. Let u ∈ C 2 (Ω) satisfy −∆u = 0 (≤ 0, ≥ 0) in Ω. Then, for any
ball B = BR (y) ⊂⊂ Ω, we have
ˆ
1
u(y) = (≤, ≥) u ds,
|∂B|ˆ ∂B
1 (B.12)
u(y) = (≤, ≥) u dx.
|B| B
362 Appendix B. Elliptic Partial Differential Equations

With the aid of Theorem B.1, the strong maximum principle for subhar-
monic functions and the strong minimum principle for superharmonic func-
tions can be derived as follows.
Theorem B.2. Let −∆u ≤ 0 (≥ 0) in Ω and suppose there is a point y ∈ Ω
such that
u(y) = sup u (inf u).
Ω Ω

Then u is constant. Therefore a harmonic function cannot assume an interior

maximum or minimum value unless it is constant.
Proof. Let −∆u ≤ 0 in Ω, M = supΩ u and ΩM = {x ∈ Ω : u(x) = M }.
By assumption, ΩM 6= ∅. Furthermore since u is continuous, ΩM is closed
relative to Ω. We are going to show ΩM is also open relative to Ω to conclude
ΩM = Ω. Let z is a point in ΩM . Apply the mean value inequality (B.12) to the
subharmonic function u − M in a ball B = BR (z) ⊂⊂ Ω to get
ˆ
1
0 = u(z) − M ≤ (u − M ) dx ≤ 0.
|B| B
Since u − M ≤ 0 in in BR (z), we must have u = M in BR (z), which implies ΩM
is open. The result for superharmonic functions follows by replacing u by −u.

Theorem B.2 implies the following weak maximum and minimum princi-
ples.
Theorem B.3. Let u ∈ C 2 (Ω) ∩ C 0 (Ω) with −∆u ≤ 0 (≥ 0) in Ω. Then,
provided that Ω is bounded,
sup u = sup u (inf u = inf u).
Ω ∂Ω Ω ∂Ω

Therefore, for a harmonic function u,

inf u ≤ u(x) ≤ sup u, x ∈ Ω.
∂Ω ∂Ω

The uniqueness theorem for the classical Dirichlet problem for the Poisson
equation in bounded domains follows from Theorem B.3.
Theorem B.4. Let u, v ∈ C 2 (Ω) ∩ C 0 (Ω) satisfy −∆u = −∆v in Ω and u = v
on ∂Ω. Then u = v in Ω.
B.3. Discrete Maximum and Minimum Principles 363

Proof. Let w = u − v. Then −∆w = 0 in Ω and w = 0 on ∂Ω. It follows from

Theorem B.3 that w ≡ 0 in Ω.
Now, consider the linear elliptic operator of the form

Lu = −∇ · (A(x)∇u) + b(x) · ∇u + c(x)u. (B.13)

A function u satisfying Lu = 0 (≤ 0, ≥ 0) in Ω is called a solution (subsolution,

supersolution) of Lu = 0 in Ω. Analogues to Theorems B.3 and B.4 can be
proved for L. See [26, §3.1] for proofs.
Theorem B.5. Let L be elliptic in a bounded domain Ω with c = 0. Suppose
u ∈ C 2 (Ω) ∩ C 0 (Ω) with Lu ≤ 0 (≥ 0) in Ω. Then

sup u = sup u (inf u = inf u).

Ω ∂Ω Ω ∂Ω

Theorem B.6. Let L be elliptic with c ≥ 0. Suppose u, v ∈ C 2 (Ω) ∩ C 0 (Ω)

satisfy Lu = Lv in Ω and u = v on ∂Ω. Then u = v in Ω. If Lu ≤ Lv in Ω and
u ≤ v on ∂Ω, then u ≤ v in Ω.

B.3. Discrete Maximum and Minimum Principles

Let ∆h be the discrete five-point Laplacian defined on grid points Ωh = {xpq ∈
Ω}, where h is the grid size and Ω is a bounded region in 2D.
Theorem B.7. Let Ω be a rectangular region and −∆h u ≤ 0 (≥ 0) on Ωh . If u
has an interior maximum (minimum), then u is constant on Ωh . Therefore

max u = max u (min u = min u).

Ωh ∂Ωh Ωh ∂Ωh

Proof. First, consider the case −∆h u ≤ 0; let u have a maximum value at an
interior point xpq . The condition −∆h u ≤ 0 is equivalent to
1
upq ≤ 2
(up−1,q + up+1,q + r2 up,q−1 + r2 up,q+1 ), (B.14)
2 + 2r
where r = hx /hy . Hence this easily leads to the conclusion that the interior
point xpq can have a (local) maximum only if all neighboring points have the
364 Appendix B. Elliptic Partial Differential Equations

same maximum value and that the inequality is actually an equality. The
argument then implies that u has the same value at all grid points includ-
ing those on the boundary. This proves the discrete maximum principle for
−∆h u ≤ 0. Now, the discrete minimum principle for the superharmonic func-
tions can be proved by replacing u by −u and following the same argument.

The following generalizes Theorem B.7.

Theorem B.8. Let L = −∇ · A(x)∇ + b(x) · ∇ be an elliptic operator defined
in a rectangular region Ω, where A(x) = diag(a11 (x), a22 (x)), and Lh be the a
five-point FD discretization of L. Assume that h is sufficiently small when
b 6= 0. Suppose a function u satisfies Lh u ≤ 0 (≥ 0) on Ωh and has an interior
maximum (minimum), then u is constant on Ωh . Thus

max u = max u (min u = min u)

Ωh ∂Ωh Ωh ∂Ωh

and therefore, for a solution u of Lh u = 0,

inf u ≤ u(x) ≤ sup u, x ∈ Ωh .

∂Ωh ∂Ωh

Proof. Let u have a maximum at an interior point xpq . The condition Lh u ≤ 0

is equivalent to
1 pq pq pq pq

upq ≤ pq −ap−1,q up−1,q − ap+1,q up+1,q − ap,q−1 up,q−1 − ap,q+1 up,q+1 , (B.15)
apq
where apq rs is the matrix entry corresponding to the relationship of Lh from upq
to urs . Note that for five-point FD schemes,
pq pq pq pq
apq
pq = −(ap−1,q + ap+1,q + ap,q−1 + ap,q+1 ) > 0. (B.16)

When b = 0, it is easy to see that the coefficients apq rs , (pq) 6= (rs), are all
strictly negative; for the case b 6= 0, one needs to choose the grid size h suffi-
ciently small in order for the four off-diagonal entries of the algebraic system
to remain negative. Now, let upq be an interior (local) maximum. Then it fol-
lows from (B.15), (B.16), and apq rs < 0, (pq) 6= (rs), that all the neighboring
values must be the same as the maximum, which implies u is constant on
B.4. Coordinate Changes 365

Ωh . This proves the discrete maximum principle for subsolutions. As in the

proof of Theorem B.7, the discrete minimum principle for supersolutions can
be proved by replacing u by −u and following the same argument.
See Exercise 4.7, on page 152, for the maximum principle applied for more
general elliptic problems.

B.4. Coordinate Changes

Often we have to solve the PDEs on a domain that is not a rectangle or other
easy shape. In the case it is desirable to change coordinates so that the solu-
tion can be computed in a convenient coordinate system. We begin with the
elliptic equation
−∇ · (A(x)∇u) = f (x), (B.17)
where A = [aij ] is symmetric positive definite. Let ξ be another coordinate
system:
ξ = ξ(x). (B.18)
Then we see 
∂ξi
∇x = J T ∇ξ , J= , (B.19)
∂xj
and therefore
∇x · A∇x = ∇ξ · JAJ T ∇ξ . (B.20)
Note that B(:= JAJ T ) is symmetric; its positiveness can be shown for certain
cases.
As an example consider the Poisson equation defined on a trapezoidal do-
main:
Ω = {(x1 , x2 ) : 0 < x1 < 1, 0 < x2 < (1 + x1 )/2}.
Define a new coordinate system ξ ∈ (0, 1)2 ,
2x2
ξ1 = x1 , ξ2 = .
1 + x1
Then the Jacobian reads
 
1 0
J=
−ξ2 /(1 + ξ1 ) 2/(1 + ξ1 )
366 Appendix B. Elliptic Partial Differential Equations

and  
ξ2
1 −
T
 T 1 + ξ1 
B = JAJ = JJ =  .
 ξ2 ξ22 + 4 
−
1 + ξ1 (1 + ξ1 )2
The matrix B(ξ) is clearly symmetric and positive definite on the unit square.
The problem
−∇ · B(ξ)∇ u = f (ξ), ξ ∈ (0, 1)2 ,
can be approximated by the standard second-order FD method.

B.5. Cylindrical and Spherical Coordinates

The cylindrical coordinates (ρ, φ, z) determine a point P whose Cartesian co-
ordinates are
x = ρ cos φ, y = ρ sin φ, z = z. (B.21)
Thus ρ and φ are the polar coordinates in the xy-plane of the point Q, where
Q is the projection of P onto that plane. Relations (B.21) can be written as
p
ρ = x2 + y 2 , φ = tan−1 (y/x), z = z. (B.22)

It follows from (B.21) and (B.22) that

∂u ∂u ∂ρ ∂u ∂φ x ∂u y ∂u ∂u sin φ ∂u
= + = − 2 = cos φ − .
∂x ∂ρ ∂x ∂φ ∂x ρ ∂ρ ρ ∂φ ∂ρ ρ ∂φ
Replacing the function u in the above equation by ∂u ∂x , we see

∂ 2u
   
∂ ∂u sin φ ∂ ∂u
= cos φ −
∂x2 ∂ρ  ∂x ρ ∂φ ∂x  
∂ ∂u sin φ ∂u sin φ ∂ ∂u sin φ ∂u
= cos φ cos φ − − cos φ −
∂ρ ∂ρ ρ ∂φ ρ ∂φ ∂ρ ρ ∂φ
2 2 2 2
∂ u 2 sin φ cos φ ∂ u sin φ ∂ u
= cos2 φ 2 − +
∂ρ ρ ∂φ∂ρ ρ2 ∂φ2
2
sin φ ∂u 2 sin φ cos φ ∂u
+ + .
ρ ∂ρ ρ2 ∂φ
(B.23)
B.5. Cylindrical and Spherical Coordinates 367

In the same way, one can show that

∂u ∂u cos φ ∂u
= sin φ +
∂y ∂ρ ρ ∂φ
and
∂ 2u 2 ∂ u
2
2 sin φ cos φ ∂ 2 u cos2 φ ∂ 2 u
= sin φ 2 + +
∂y 2 ∂ρ ρ ∂ φ∂ρ ρ2 ∂φ2 (B.24)
cos2 φ ∂u 2 sin φ cos φ ∂u
+ − .
ρ ∂ρ ρ2 ∂φ
From (B.23) and (B.24), the Laplacian of u in cylindrical coordinates is
∂ 2 u 1 ∂u 1 ∂ 2u ∂ 2u
∆u = + + +
∂ρ2 ρ ∂ρ ρ2 ∂φ2 ∂z 2 (B.25)
1 1
= (ρuρ )ρ + 2 uφφ + uzz .
ρ ρ
The spherical coordinates (r, φ, θ) of a point are related to x, y, and z as
follows:
x = r sin θ cos φ, y = r sin θ sin φ, z = r cos θ. (B.26)
Using the arguments for the cylindrical coordinates, one can see that the
Laplacian of u in spherical coordinates is
∂ 2 u 2 ∂u 1 ∂ 2u 1 ∂ 2 u cot θ ∂u
∆u = + + + + 2
∂r2 r ∂r r2 sin2 θ ∂φ2 r2 ∂θ2 r ∂θ (B.27)
1 2 1 1
= 2 (r ur )r + 2 2 uφφ + 2 (uθ sin θ)θ .
r r sin θ r sin θ
368 Appendix B. Elliptic Partial Differential Equations
Appendix C

Helmholtz Wave Equation∗

To be included.

369
370 Appendix C. Helmholtz Wave Equation∗
Appendix D

Richards’s Equation for Unsaturated

Water Flow∗

To be included.

371
372 Appendix D. Richards’s Equation for Unsaturated Water Flow∗
Appendix E

Orthogonal Polynomials and

Quadratures

E.1. Orthogonal Polynomials

Let w be a given function defined on (−1, 1) and positive there. (The function
w is often called a weight function.) Let f and g be defined on the interval
(−1, 1). Define the scalar product of the functions f and g on (−1, 1) as

ˆ 1
(f, g)w = f (x)g(x)w(x)dx. (E.1)
−1

Then, the orthogonal polynomials on (−1, 1) with respect to the weight func-
tion w are a series of polynomials {Pk }k=0,1,2,··· satisfying

Pk ∈ Pk ; (Pk , Pm )w = 0, k 6= m, (E.2)

where Pk denotes the space of polynomials of degree ≤ k.

Those orthogonal polynomials satisfy a three-term recurrence relation of the
form

Pk+1 (x) = Ak (x − Bk )Pk (x) − Ck Pk−1 (x), k = 0, 1, 2, · · · , (E.3)

373
374 Appendix E. Orthogonal Polynomials and Quadratures

where
P−1 ≡ 0,
αk+1
Ak = ,
αk
(xPk , Pk )w
Bk = ,
 Sk
 arbitrary, k = 0,
Ck = Ak Sk
, k > 0.
Ak−1 Sk−1


Here αk is the leading coefficient of Pk and Sk is defined as

Sk = (Pk , Pk )w .

Example E.1. Legendre Polynomials {Lk }: the weight function

w(x) ≡ 1.

With this choice of the weight function, starting with L0 (x) = 1, one can get

2k + 1 k
Ak = , Bk = 0, Ck = ,
k+1 k+1
where a normalization is applied for Lk (1) = 1. Thus the Legendre polynomi-
als satisfy the following three-term recurrence relation

(2k + 1)xLk (x) − kLk−1 (x)

Lk+1 (x) = . (E.4)
k+1
A few first Legendre polynomials are

L0 (x) = 1,
L1 (x) = x,
 
3 2 1
L2 (x) = x − ,
2 3  (E.5)
5 3 3
L3 (x) = x − x ,
2  5 
35 4 6 2 3
L4 (x) = x − x + .
8 7 35
E.2. Gauss-Type Quadratures 375

Relevant properties are

|Lk (x)| ≤ 1, ∀ x ∈ [−1, 1],
Lk (±1) = (±1)k ,
|L0k (x)| ≤ k(k + 1)/2, ∀ x ∈ [−1, 1], (E.6)
L0k (±1) = (±1)k k(k + 1)/2,
(Lk , Lk )w=1 = (k + 1/2)−1 .

Example E.2. Chebyshev Polynomials {Tk }: the weight function

w(x) := (1 − x2 )−1/2 .
With this choice of the weight function, one can get the three-term recurrence
relation for the Chebyshev polynomials
Tk+1 (x) = 2xTk (x) − Tk−1 (x). (E.7)
A few first Chebyshev polynomials are
T0 (x) = 1,
T1 (x) = x,
T2 (x) = 2x2 − 1, (E.8)
T3 (x) = 4x3 − 3x,
T4 (x) = 8x4 − 8x2 + 1.
Relevant properties are
|Tk (x)| ≤ 1, ∀ x ∈ [−1, 1],
Tk (±1) = (±1)k ,
|Tk0 (x)| ≤ k 2 , ∀ x ∈ [−1, 1],
(E.9)
Tk0 (±1) = (±1)k k 2 ,

π, if k = 0,
(Tk , Tk )w =
π/2, if k ≥ 1.

E.2. Gauss-Type Quadratures

There are close relations between orthogonal polynomials and Gauss-type in-
tegration quadrature formulas on the interval [−1, 1]. We first review the
376 Appendix E. Orthogonal Polynomials and Quadratures

Gauss-type integration formulas.

Theorem E.3. Gauss Integration. Let {x0 , x1 , · · · , xn } be the zeros of the
(n + 1)-th orthogonal polynomial Pn+1 . Let {w0 , w1 , · · · , wn } be the solution of
the linear system
Xn ˆ 1
i
(xj ) wj = xi w(x)dx, i = 0, 1, · · · , n.
j=0 −1

Then, (1). wj > 0, j = 0, 1, · · · , n, and

ˆ 1 n
X
f (x)w(x) = f (xj )wj , ∀ f ∈ P2n+1 . (E.10)
−1 j=0

(2). There is no xj and wj , j = 0, 1, · · · , n, such that (E.10) holds for all f ∈

P2n+2 .
The Gauss integration formula is well known. However, the zeros of Pn+1
are all in the interior of [−1, 1]. Thus, it shows a drawback when a bound-
ary condition is to be imposed. In particular, most finite element methods
require the continuity of the solution on element boundaries and introduce
nodal points on the boundary. The following Gauss-Lobatto formula is more
useful than the Gauss formula in numerical PDEs.
Theorem E.4. Gauss-Lobatto Integration. Let x0 = −1, xn = 1, and xj ,
j = 1, 2, · · · , n − 1, be the zeros of the first-derivative of the n-th orthogonal
polynomial, Pn0 . Let {w0 , w1 , · · · , wn } be the solution of the linear system
Xn ˆ 1
i
(xj ) wj = xi w(x)dx, i = 0, 1, · · · , n.
j=0 −1

Then,
ˆ 1 n
X
f (x)w(x) = f (xj )wj , ∀ f ∈ P2n−1 . (E.11)
−1 j=0

For the Legendre polynomials, the explicit formulas for the quadrature
nodes are not known. Thus the nodal points and the corresponding weights
must be computed numerically as zeros of appropriate polynomials and the
E.2. Gauss-Type Quadratures 377

solution of a linear system, respectively. On the other hand, for Chebyshev

series, the points and weights are known explicitly. Here we collect those
formulas and explicit expressions:
Legendre-Gauss:
xj (= zeros of Ln+1 ), j = 0, 1, · · · , n,
2 (E.12)
wj = , j = 0, 1, · · · , n.
(1 − xj )[L0n+1 (xj )]2
2

Legendre-Gauss-Lobatto:
x0 = −1, xn = 1; xj (= zeros of L0n ), j = 1, 2, · · · , n − 1,
2 (E.13)
wj = , j = 0, 1, · · · , n.
n(n + 1)[Ln (xj )]2

Chebyshev-Gauss:
 
(2j + 1)π π
xj = − cos , wj = , j = 0, 1, · · · , n. (E.14)
2n + 2 n+1

Chebyshev-Gauss-Lobatto:
  
jπ π/(2n), j = 0, n,
xj = − cos , wj = (E.15)
n π/n, j = 1, · · · , n − 1.

The following shows a few examples for the Legendre-Gauss-Lobatto points

and the corresponding weights on the interval [−1, 1]:

Legendre-Gauss-Lobatto points weights

n = 1 −1 1 1 1
1 4 1
n = 2 −1 0 1
 1 1/2  1 1/2 3 3 3 (E.16)
1 5 5 1
n = 3 −1 − 1
 53 1/2 5 
3 1/2
6 6 6 6
1 49 64 49 1
n = 4 −1 − 0 1
7 7 10 90 90 90 10
378 Appendix E. Orthogonal Polynomials and Quadratures
Appendix F

Some Mathematical Formulas

F.1. Trigonometric Formulas

The following trigonometric formulas are useful

(a) sin(x + y) = sin x cos y + cos x sin y,

(b) cos(x + y) = cos x cos y − sin x sin y,
   
x+y x−y
(c) sin x + sin y = 2 sin cos ,
 2   2 
x+y x−y
(d) sin x − sin y = 2 cos sin , (F.1)
 2   2 
x+y x−y
(e) cos x + cos y = 2 cos cos ,
 2   2 
x+y x−y
(f) cos x − cos y = −2 sin sin .
2 2

By setting x = 2θ and y = 0 in (F.1.e), one also can have

2 sin2 θ = 1 − cos(2θ), 2 cos2 θ = 1 + cos(2θ). (F.2)

F.2. Vector Identities

Let A, B, C, and D be vectors in R3 and f is scalar. Let

A · B = A1 B1 + A2 B2 + A3 B3

379
380 Appendix F. Some Mathematical Formulas

and
A × B = (A2 B3 − A3 B2 , A3 B1 − A1 B3 , A1 B2 − A2 B1 )
 
jb1 jb2 jb3
= det  A1 A2 A3  ,
B1 B2 B3
where jbi is the unit vector in the xi -direction. Then
A · B = |A| |B| cos θ, A × B = |A| |B| sin θ n
b,
where θ is the angle between A and B and n b is the unit normal vector from the
plane containing A and B whose orientation is determined by the right-hand
rule. (When four fingers grab directing from A to B, then the direction of the
thumb determines n b .) Let ∇× denote the curl operator defined as
 
∂A3 ∂A2 ∂A1 ∂A3 ∂A2 ∂A1
∇×A= − , − , − .
∂y ∂z ∂z ∂x ∂x ∂y
Then,
A · (B × C) = B · (C × A) = C · (A × B),
A × (B × C) = (A · C)B − (A · B)C,
(A × B) · (C × D) = (A · C)(B · D) − (A · D)(B · C),
∇(A · B) = A × (∇ × B) + B × (∇ × A) + (A · ∇)B + (B · ∇)A,
∇ · (A × B) = B · (∇ × A) − A · (∇ × B),
(F.3)
∇ × (f A) = f (∇ × A) − A × (∇f ),
∇ × (A × B) = (B · ∇)A − (A · ∇)B + A(∇ · B) − B(∇ · A),
∇ · (∇ × A) = 0,
∇ × (∇f ) = 0,
∇ × (∇ × A) = ∇(∇ · A) − ∇2 A.
Associated with vectors are the following integrals.
Gauss’s divergence theorem:
ˆ ˛
∇ · B dx = B · n ds
V A
Stokes’s theorem: ˆ ˛
(∇ × B) · n ds = B · dl
A C
Appendix G

Finite Difference Formulas

Here we summarize second- and fourth-order finite difference formulas. In

the following, h(> 0) is the spatial variable and ui = u(x0 + ih).

Central 2nd-order FD schemes:

u1 − u−1
ux (x0 ) ≈
2h
u1 − 2u0 + u−1
uxx (x0 ) ≈
h2 (G.1)
u2 − 2u1 + 2u−1 − u−2
uxxx (x0 ) ≈
2h3
u2 − 4u1 + 6u0 − 4u−1 + u−2
u(4) (x0 ) ≈
h4

Central 4th-order FD schemes:

−u2 + 8u1 − 8u−1 + u−2

ux (x0 ) ≈
12h
−u2 + 16u1 − 30u0 + 16u−1 − u−2
uxx (x0 ) ≈
12h2 (G.2)
−u3 + 8u2 − 13u1 + 13u−1 − 8u−2 + u−3
uxxx (x0 ) ≈
8h3
−u3 + 12u2 − 39u1 + 56u0 − 39u−1 + 12u−2 − u−3
u(4) (x0 ) ≈
6h4
381
382 Appendix G. Finite Difference Formulas

One-sided 2nd-order FD schemes:

−3u0 + 4u±1 − u±2
ux (x0 ) ≈ ±
2h
2u0 − 5u±1 + 4u±2 − f±3
uxx (x0 ) ≈
h2 (G.3)
−5u0 + 18u±1 − 24u±2 + 14f±3 − 3u±4
uxxx (x0 ) ≈ ±
2h3
3u0 − 14u±1 + 26u±2 − 24f±3 + 11u±4 − 2u±5
u(4) (x0 ) ≈
h4
Bibliography

[1] V. A GHOSKOV, Poincaré–Steklov’s operators and domain decomposition

methods in finite dimensional spaces, in First International Symposium
on Domain Decomposition Method for Partial Differential Equations,
R. Glowinski, G. Golub, G. Meurant, and J. Periaux, eds., SIAM, Philadel-
phia, 1988, pp. 73–112.

[2] W. A MES AND D. L EE, Current development in the numerical treatment

of ocean acoustic propagation, Appl. Numer. Math., 3 (1987), pp. 25–47.

[3] R. B ARRETT, M. B ERRY, T. C HAN, J. D EMMEL , J. D ONATO, J. D ON -

GARRA , V. E IJKHOUT, R. P OZO, C. R OMINE , AND H. VAN DER V ORST ,
Templates for the solution of linear systems: Building blocks for iterative
methods, SIAM, Philadelphia, 1994. The postscript file is free to down-
load from http://www.netlib.org/templates/ along with source
codes.

[4] P. B JORSTAD AND O. W IDLUND, Iterative methods for the solution of el-
liptic problems on regions partitioned into substructures, SIAM J. Numer.
Anal., 23 (1986), pp. 1097–1120.

[5] J.-F. B OURGAT, R. G LOWINSKI , P. L E T ALLEC, AND M. V IDRASCU,

Variational formulation and algorithm for trace operator in domain de-
composition calculations, in Domain Decomposition Methods, T. Chan,
R. Glowinski, J. Periaux, and O. Widlund, eds., SIAM, Philadelphia,
1989, pp. 3–16.

[6] J. B RAMBLE , J. PASCIAK , AND A. S CHATZ, An iterative method for ellip-

tic problems on regions partitioned into substructures, Math. Comput., 46
(1986), pp. 361–369.

383
384 BIBLIOGRAPHY

[7] S. C ANDEL, A review of numerical methods in acoustic wave propagation,

in Recent Advances in Aeroacoustics, A. Krothapalli and C. A. Smith,
eds., Springer-Verlag, New York, 1986, pp. 339–410.
[8] Y. C HA AND S. K IM, Edge-forming methods for color image zooming,
IEEE Trans. Image Process., 15 (2006), pp. 2315–2323.
[9] R. C LAYTON AND B. E NGQUIST, Absorbing boundary conditions for
acoustic and elastic wave calculations, Bull. Seismol. Soc. Amer., 67
(1977), pp. 1529–1540.
[10] G. C OHEN, P. J OLY, AND N. T ORDJMAN, Construction and analysis of
higher order finite elements with mass lumping for the wave equation,
in Second International Conference on Mathematical and Numerical As-
pects of Wave Propagation, R. Kleinman, T. Angell, D. Colton, F. Santosa,
and I. Stakgold, eds., SIAM, Philadelphia, 1993, pp. 152–160.
[11] G. D AHLQUIST, A special stability problem for linear multistep methods,
BIT, 3 (1963), pp. 27–43.
[12] Y. D E R OECK AND P. L E T ALLEC, Analysis and test of a local do-
main decomposition preconditioner, in Fourth International Symposium
on Domain Decomposition Method for Partial Differential Equations,
R. Glowinski, G. Meurant, J. Periaux, and O. B. Widlund, eds., SIAM,
Philadelphia, 1991, pp. 112–128.
[13] B. D ESPRÉS, Domain decomposition method and the Helmholtz problem,
in Mathematical and Numerical Aspects of Wave Propagation Phenom-
ena, G. Cohen, L. Halpern, and P. Joly, eds., Philadelphia, 1991, SIAM,
pp. 44–52.
2 2
[14] J. D OUGLAS, J R ., On the numerical integration of ∂∂xu2 + ∂∂yu2 = ∂u
∂t by implicit
methods, J. Soc. Indust. Appl. Math., 3 (1955), pp. 42–65.
[15] J. D OUGLAS, J R . AND J. G UNN, A general formulation of alternating di-
rection methods Part I. Parabolic and hyperbolic problems, Numer. Math.,
6 (1964), pp. 428–453.
[16] J. D OUGLAS, J R . AND S. K IM, Improved accuracy for locally one-
dimensional methods for parabolic equations, Mathematical Models and
Methods in Applied Sciences, 11 (2001), pp. 1563–1579.
BIBLIOGRAPHY 385

[17] J. D OUGLAS, J R ., P. PAES L EME , J. R OBERTS, AND J. WANG, A parallel

iterative procedure applicable to the approximate solution of second order
partial differential equations by mixed finite element methods, Numer.
Math., 65 (1993), pp. 95–108.

[18] J. D OUGLAS, J R . AND D. P EACEMAN, Numerical solution of two-

dimensional heat flow problems, American Institute of Chemical Engi-
neering Journal, 1 (1955), pp. 505–512.

[19] M. D RYJA AND O. W IDLUND, Some recent results on Schwarz type do-
main decomposition algorithms, in Domain Decomposition Methods in
Science and Engineering, A. Quarteroni, J. Periaux, Y. Kuznetsov, and
O. Widlund, eds., vol. 157 of Contemporary Mathematics, Philadelphia,
1994, SIAM, pp. 53–61.

[20] E. D’ YAKONOV, Difference schemes with split operators for multidimen-

sional unsteady problems (English translation), USSR Comp. Math., 3
(1963), pp. 581–607.

[21] B. E NGQUIST AND A. M AJDA, Absorbing boundary conditions for the nu-
merical simulation of waves, Math. Comp., 31 (1977), pp. 629–651.

[22] B. E NGQUIST AND A. M AJDA, Radiation boundary conditions for acous-

tic and elastic wave calculations, Comm. Pure Appl. Math., 32 (1979),
pp. 314–358.

[23] J. F ERZIGER AND M. P ERIC, Computational methods for fluid dynamics,

2nd Edition, Springer-Verlag, Berlin, Heidelberg, New York, 1999.

[24] R. W. F REUND, Conjugate gradient–type methods for linear systems with

complex symmetric coefficient matrices, SIAM J. Sci. Stat. Comput., 13
(1992), pp. 425–448.

[25] S. G ERSCHGORIN, Über die abgrenzung der eigenwerte einer matrix, Izv.
Akad. Nauk SSSR Ser. Mat., 7 (1931), pp. 746–754.

[26] D. G ILBERG AND N. T RUDINGER, Elliptic Partial Differential Equations

of Second Order, Springer-Verlag, Berlin, Heidelberg, New York, Tokyo,
1983.
386 BIBLIOGRAPHY

[27] B. G USTAFSSON, H.-O. K REISS, AND J. O LIGER, Time Dependent Prob-

lems and Difference Methods, Wiley-Interscience, New York, 1996.
[28] I. H ARARI AND R. D JELLOULI, Analytical study of the effect of wave num-
ber on the performance of local absorbing boundary conditions for acous-
tic scattering, Appl. Numer. Math., 50 (2004), pp. 15–47.
[29] R. L. H IGDON, Absorbing boundary conditions for difference approxima-
tions to the multi-dimensional wave equation, Math. Comp., 47 (1986),
pp. 437–459.
[30] , Numerical absorbing boundary conditions for the wave equation,
Math. Comp., 49 (1987), pp. 65–90.
[31] F. Q. H U, Absorbing boundary conditions, Int. J. Comput. Fluid Dyn., 18
(2004), pp. 513–522.
[32] C. J OHNSON, Numerical Solutions of Partial Differential Equations by
the Finite Element Method, Cambridge University Press, New York, New
Rochelle, Melbourne, Sydney, 1987.
[33] C. K ELLY, Iterative methods for linear and nonlinear equations, SIAM,
Philadelphia, 1995.
[34] H. K IM , Y. C HA , AND S. K IM, Curvature interpolation method for image
zooming, IEEE Trans. Image Process., 20 (2011), pp. 1895–1903.
[35] S. K IM, GRADE: Graduate Research and Applications for Differential
Equations. The modelcode library is under construction for educa-
tion and research in Industrial and Computational Mathematics, initi-
ated in Spring 1999; the codes are available through internet access to
www.msstate.edu/∼skim/GRADE.
[36] , Numerical treatments for the Helmholtz problem by domain decom-
position technique, Contemporary Mathematics, 180 (1994), pp. 245–250.
[37] , Parallel multidomain iterative algorithms for the Helmholtz wave
equation, Appl. Numer. Math., 17 (1995), pp. 411–429.
[38] , Domain decomposition iterative procedures for solving scalar waves
in the frequency domain, Numer. Math., 79 (1998), pp. 231–259.
BIBLIOGRAPHY 387

[39] , An O(N ) level set method for eikonal equations, SIAM J. Sci. Com-
put., 22 (2001), pp. 2178–2193.
[40] S. K IM AND R. C OOK, 3D traveltime computation using second-order
ENO scheme, Geophysics, 64 (1999), pp. 1867–1876.
[41] S. K IM AND S OOHYUN K IM, Multigrid simulation for high-frequency so-
lutions of the Helmholtz problem in heterogeneous media, SIAM J. Sci.
Comput., 24 (2002), pp. 684–701.
[42] S. K IM AND M. L EE, Artificial damping techniques for scalar waves in the
frequency domain, Computers Math. Applic., 31, No. 8 (1996), pp. 1–12.
[43] S. K IM , C. S HIN, AND J. K ELLER, High-frequency asymptotics for the
numerical solution of the Helmholtz equation, Appl. Math. Letters, 18
(2005), pp. 797–804.
[44] S. K IM AND W. S YMES, Multigrid domain decomposition methods for
the Helmholtz problem, in Mathematical and Numerical Aspects of Wave
Propagation, J. A. DeSanto, ed., SIAM, Philadelphia, 1998, pp. 617–619.
[45] P. L E T ALLEC, Domain decomposition methods in computational me-
chanics, Comput. Mech. Advances, 1 (1994), pp. 121–220.
[46] H. L IM , S. K IM , AND J. D OUGLAS, J R ., Numerical methods for viscous
and nonviscous wave equations, Appl. Numer. Math., 57 (2007), pp. 194–
212.
[47] P. L IONS, On the Schwarz alternating method I, in First International
Symposium on Domain Decomposition Method for Partial Differential
Equations, R. Glowinski, G. Golub, G. Meurant, and J. Periaux, eds.,
Philadelphia, PA, 1988, SIAM, pp. 1–42.
[48] , On the Schwarz alternating method III: a variant for nonoverlap-
ping subdomains, in Domain Decomposition Methods for Partial Differ-
ential Equations, T. Chan, R. Glowinski, J. Periaux, and O. Widlund, eds.,
Philadelphia, PA, 1990, SIAM, pp. 202–223.
[49] F. M AGOULÈS , F.-X. R OUX , AND L. S ERIES, Algebraic way to derive
absorbing boundary conditions for the Helmholtz equation, J. Comput.
Acoust., 13 (2005), pp. 433–454.
388 BIBLIOGRAPHY

[50] J. M ANDEL, Two-level domain decomposition preconditioning for the p-

version finite element method in three dimensions, Int. J. Numer. Methods
Engrg., 29 (1990), pp. 1095–1108.

[51] G. M ARCHUK, Methods of numerical mathematics, Springer-Verlag, New

York, Heidelberg, and Berlin, 1982.

[52] L. M ARINI AND A. Q UARTERONI, A relaxation procedure for domain

decomposition methods using finite elements, Numer. Math., 55 (1989),
pp. 575–598.

[53] L. M C I NNES, R. S USAN -R ESIGA , D. K EYES, AND H. A TASSI, Additive

Schwarz methods with nonreflecting boundary conditions for the paral-
lel computation of Helmholtz problems, in Domain Decomposition Meth-
ods 10, J. Mandel, C. Farhat, and X.-C. Cai, eds., vol. 218 of Contempo-
rary Mathematics, Providence, RI, 1998, American Mathematical Soci-
ety, pp. 325–333. Proceedings of the Tenth International Conference on
Domain Decomposition Methods, August 10-14, 1997, Boulder, CO.

[54] R. M EYER, Introduction to mathematical fluid dynamics, Dover Publica-

tions, Inc., New York, 1982.

[55] A. O STROWSKI, On the linear iteration procedures for symmetric matri-

ces, Rend. Mat. e Appl., 14 (1954), pp. 140–163.

[56] D. P EACEMAN AND H. R ACHFORD, The numerical solution of parabolic

and elliptic differential equations, J. Soc. Indust. Appl. Math., 3 (1955),
pp. 28–41.

[57] A. Q UARTERONI AND A. VALLI, Domain Decomposition Methods for Par-

tial Differential Equations, Oxford University Press, Oxford, New York,
1999.

[58] L. R UDIN, S. O SHER , AND E. FATEMI, Nonlinear total variation based

noise removal algorithms, Physica D, 60 (1992), pp. 259–268.

[59] Y. S AAD AND M. S CHULTZ, GMRES: A generalized minimal residual al-

gorithm for solving nonsymmetric linear systems, SIAM J. Sci. Stat. Com-
put., 7 (1986), pp. 856–869.
BIBLIOGRAPHY 389

[60] H. S CHWARZ, Ueber einige abbildungsaufgaben, J. Reine Angew. Math.,

70 (1869), pp. 105–120.

[61] A. S EI AND W. S YMES, Dispersion analysis of numerical wave propa-

gation and its computational consequences, J. Sci. Comput., 10 (1995),
pp. 1–27.

[62] P. S TEIN AND R. R OSENBERG, On the solution of linear simultaneous

equations by iteration, J. London Math. Soc., 23 (1948), pp. 111–118.

[63] H. S TONE, Iterative solution of implicit approximations of multidimen-

sional partial differential equations, SIAM J. Numer. Anal., 5 (1968),
pp. 530–558.

[64] J. C. S TRIKWERDA, Finite Difference Schemes and Partial Differential

Equations, Wadsworth & Brooks/Cole, Pacific Grove, California, 1989.

[65] O. T AUSSKY, Bounds for characteristic roots of matrices, Duke Math. J.,
15 (1948), pp. 1043–1044.

[66] O. VACUS, Mathematical analysis of absorbing boundary conditions for

the wave equation: the corner problem, Math. Comp., 74 (2005), pp. 177–
200.

[67] R. VARGA, Matrix Iterative Analysis, Prentice-Hall, Englewood Cliffs,

NJ, 1962.

[68] , Matrix Iterative Analysis, 2nd Ed., Springer-Verlag, Berlin, Heidel-

berg, 2000.

[69] S. W HITAKER, Introduction to fluid mechanics, R.E. Krieger Publishing

Company, Malabar, Florida, 1968.

[70] O. W IDLUND, Optimal iterative refinement methods, in Domain Decom-

position Methods, T. Chan, R. Glowinski, J. Periaux, and O. Widlund,
eds., SIAM, Philadelphia, 1989, pp. 114–125.

[71] N. YANENKO, Convergence of the method of splitting for the heat con-
duction equations with variable coefficients (English translation), USSR
Comp. Math., 3 (1963), pp. 1094–1100.
390 BIBLIOGRAPHY

[72] , The method of fractional steps, Springer-Verlag, Berlin, Heidelberg,

and New York, 1971. (English translation; originally published in Rus-
sian, 1967).
Index

L1 -contraction, 135 Black-Scholes differential equation,

θ-method, 44, 162 10
l1 -contracting method, 135 boundedness, 42
Burgers’s equation, 130
absorbing boundary condition, 178
abstract variational problem, 109 cardinal functions, 2, 94
accuracy, 122 Cauchy problem, 123
accuracy order, 46 Cauchy-Schwarz inequality, 95, 110
acoustic wave equation, 177 cell Peclet number, 43
Adams-Bashforth method, 27 cell-centered FDM, 108
Adams-Bashforth-Moulton method, central difference operator, 6
27 CFL condition, 121
Adams-Moulton method, 27 CG method, 72
adaptive methods, 24 characteristic equation, 13
additive Schwarz method, 145 characteristic function, 106
ADI method, 165 characteristics, 126
ADI-II, 172 Chebyshev polynomials, 206
advection form, 131 Chebyshev-Gauss formula, 208
affine mapping, 104 Chebyshev-Gauss-Lobatto formula,
alternating direction implicit 208
method, 165 Clayton-Engquist ABC, 178
amplification factor, 40 coarse subspace correction, 145
average slope, 22 coercivity, 110
collocation method, 88
backward difference operator, 5 column-wise point ordering, 69
backward Euler method, 45, 160 condition number, 72, 146
banded matrix, 61 conjugate gradient method, 72
bandwidth, 61 conormal flux, 108
Beam-Warming scheme, 130 conservation, 43, 54

391
392 INDEX

conservation form, 130 dispersion relation, 129

conservation laws, 123 dispersive equation, 128
conservation of mass, 186 divergence theorem, 99, 186, 210
conservation of momentum, 187 divided differences, 3
conservation principles, 185 dual problem, 97
conservative method, 130 duality argument, 97
consistency, 33, 55, 117, 132
continuity equation, 186 eigenvalue locus theorem, 64
control mass, 185 eigenvalue problem, 82
control volume, 106, 108, 185 eikonal equation, 178
control volume equation, 186 Einstein convention, 186
convection-diffusion equation, 190 element stiffness matrix, 103
convergence, 34, 118 elliptic equation, 9
coordinate change, 198 energy method, 38, 49
Courant number, 42, 133 error analysis, 48
Courant-Friedrichs-Lewy condition, error equation, 35
121 error estimate for FEM, 95
Crank-Nicolson method, 45, 160, essential boundary condition, 105
165 Euler equations, 124
Crank-Nicolson scheme, 122 Euler method, 18
curl, 210 Euler’s equation, 189
curve fitting, 1, 2 explicit scheme, 32
curve fitting approach, 7 explicit schemes, 117
cylindrical coordinates, 199 extensive property, 185

diagonal dominance, 65, 83 FD schemes, central 2nd-order, 211

difference equation, 12 FD schemes, central 4th-order, 211
differential form, 124 FD schemes, one-sided 2nd-order,
differential problem, 85 211
directed graph, 64 Fick’s law, 190
Dirichlet-Neumann method, 150 finite difference formulas, 211
discrete five-point Laplacian, 55, 57, finite difference method, 31, 51, 116
69 finite element method, 85
discrete maximum principle, 56, 196 finite volume method, 105
discrete minimum principle, 196 first-order ABC, 178
dispersion, 129 fluid mechanics, 10
dispersion analysis, 128 flux conservation error, 152
INDEX 393

flux function, 123 harmonic extension, 148

forward difference operator, 5 harmonic function, 195
forward Euler method, 32, 45, 160 heat equation, 9
forward-backward difference match- Hessian, 71
ing, 153 Heun’s method, 23
Fourier transform, 178, 193 high-order Galerkin methods, 110,
Fourier’s law, 190 183
fourth-order Runge-Kutta method, higher-order FEMs, 89
23 Higher-order Taylor methods, 20
fractional-step method, 165 Hilbert space, 94
frequency, 129 hyperbolic, 123
fundamental period of the motion, hyperbolic equation, 9
25
ILU, 75
Galerkin method, 88, 90 image denoising, 11
Gauss elimination, 60 image processing, 11
Gauss integration, 207 incomplete LU-factorization, 75
Gauss-Lobatto integration, 207 initial value problem, 17
Gauss-Lobatto points, 94 integral form, 124
Gauss-Seidel method, 66, 67 integration by parts, 85
generalized solution, 127 intensive property, 185
generic transport equation, 190 interior regularity estimate, 195
ghost grid value, 52 interpolation error theorem, 3
ghost value, 181 interpolation estimate, 96
Gibbs notation, 189 irreducible matrix, 63
global error, 24 isothermal equations, 125
global point index, 58 isothermal flow, 125
Godunov theorem, 137
Jacobi method, 66
Godunov’s method, 132
Jacobian, 104
gradient, 71
Green’s formula, 99 kinematic viscosity coefficient, 189
group marching method, 178 Krylov subspace method, 71
group velocity, 129
L2 -norm, 94
Hr (Ω)-norm, 95 Lagrange interpolating polynomial,
Hs (R2 )-norm, 194 2
harmonic average, 109 Lax-Friedrichs scheme, 117, 132
394 INDEX

Lax-Milgram Lemma, 109 modified equation, 128

Lax-Milgram lemma, 87 modified Euler method, 23
Lax-Richtmyer Equivalence Theo- momentum conservation, 187
rem, 38, 120 momentum conservation equation,
Lax-Wendroff scheme, 128 187
leapfrog scheme, 117 monotone method, 136
least-square approach, 88 monotonicity preserving method,
Legendre polynomials, 206 134
Legendre-Gauss formula, 208 multi-step methods, 27
Legendre-Gauss-Lobatto formula, multiplicative Schwarz method, 143
208
natural boundary condition, 105
line relaxation methods, 69
Navier-Stokes (NS) equations, 10
line SOR method, 83
Navier-Stokes equations, 188
linear FEM, 89
Neumann-Neumann method, 151
linear Galerkin method, 90
Newton polynomial, 2
linear iterative method, 62
Newtonian fluid, 187
linear space, 85
nodal point, 63, 93
Lipschitz condition, 19
non-dimensionalization, 189
Lipschitz continuity, 132
nonlinear stability, 133
local truncation error, 24
nonoverlapping DD method, 147
locally one-dimensional method,
numerical flux function, 131
165
LOD method, 165 one-sided 2nd-order FD schemes,
LU factorization, 59 211
optimal step length, 72
M-matrix, 65
order of accuracy, 122
m-step method, 27
orthogonal polynomials, 205
mass conservation, 186
outer bordering, 53, 155
material derivative, 186
overlapping Schwarz method, 142
matrix splitting, 171
maximum principle, 42, 47, 56, 83, parabolic equation, 9
161, 195, 196 Parseval’s identity, 39, 193
mean value theorems, 195 partial pivoting, 61
mesh points, 18 PCG, 75
minimization problem, 86 PCG-ILU0, 168
minimum principle, 195, 196 Peclet number, 43
mixed derivatives, 54 permutation matrix, 63
INDEX 395

Petrov-Galerkin method, 88 SIP, 75

phase velocity, 129 SOR method, 66, 67, 83
pivot, 61 space-time slice, 32, 116
Poincaré inequality, 110, 113 SPD, 146
point relaxation method, 69 specific heat, 125
polar coordinates, 199 spectral radius, 63
polytropic gas, 125 spectrum, 63
positive definite, 92 spherical coordinates, 199
preconditioned CG method, 74, 75 spline, 88
Python code, 77, 110, 137 spring-mass system, 25
quadrature, 207 stability, 14, 36, 119
quasilinear elliptic equation, 193 stability condition, 40
stability theory, 14
Rayleigh-Ritz method, 88 state equations, 125
red-black coloring, 150 steepest descent method, 71
reducible matrix, 63 Steklov-Poincaré interface equa-
reference element, 103 tion, 148
regular splitting, 65, 75 Steklov-Poincaré operator, 148
regularity estimate, 95, 194 step length, 71
relaxation methods, 66, 69 step-by-step methods, 17
relaxation parameter, 67 stiffness matrix, 103
Reynolds number, 190 Stokes’s equations, 190
Reynolds’s transport equation, 186 Stokes’s theorem, 210
Ricker wavelet, 178 strain tensor, 187
right-hand rule, 210 stress tensor, 187
Robin method, 151 strong maximum principle, 195
row-wise point ordering, 57, 58 strong minimum principle, 195
Runge-Kutta methods, 21
strong stability, 14
Runge-Kutta-Fehlberg method, 24
strongly connected, 83
SAM, 141 strongly connected directed graph,
Schur complement matrix, 148, 149 64
Schwarz alternating method, 141 strongly hyperbolic, 115
search direction, 71 strongly implicit procedure, 75
second-order Runge-Kutta method, subharmonic function, 195
22, 23 successive over-relaxation method,
semi-implicit method, 45, 160 67
396 INDEX

super-convergence, 111 TVD method, 134

superharmonic function, 195
symmetric positive definite, 71 unconditional stability, 123
symmetric positive definite matrix, unconditionally stable, 45
146 unconditionally unstable, 44
symmetrization, 53 upwind scheme, 130, 132

Taylor method of order m, 20 vanishing-viscosity approach, 126

Taylor series approach, 6 variational formulation, 85
Taylor’s theorem, 1 variational problem, 86
Taylor-series methods, 17 vector identities, 209
three-term recurrence relation, 205 von Neumann analysis, 38, 39, 121
total variation, 133
total variation diminishing method, wave equation, 9
134 wave number, 129
total variation stability, 133 waveform ABC, 179
transmission conditions, 147 weak formulation, 85
traveltime ABC, 178 weak maximum principle, 196
trial functions, 90 weak minimum principle, 196
trigonometric formulas, 209 weak solution, 127
truncation error, 33 weight function, 90
TV model, 11 weighted residual approach, 88
TV-stability, 133 well-posed equation, 9