Copyright

by
Jack Lesly Poulson
2012
 The Dissertation Committee for Jack Lesly Poulson
certifies that this is the approved version of the following dissertation:

Fast Parallel Solution of Heterogeneous

3D Time-harmonic Wave Equations

Committee:

Lexing Ying, Supervisor

Björn Engquist

Sergey Fomel

Omar Ghattas

Robert van de Geijn

 Fast Parallel Solution of Heterogeneous
3D Time-harmonic Wave Equations

by

Jack Lesly Poulson, B.S.As.E.; M.S.E.; M.S.C.A.M.

DISSERTATION
Presented to the Faculty of the Graduate School of
The University of Texas at Austin
in Partial Fulfillment
of the Requirements
for the Degree of

DOCTOR OF PHILOSOPHY

THE UNIVERSITY OF TEXAS AT AUSTIN

December 2012
 Acknowledgments

I would like to begin by thanking Lexing Ying for his immense amount
of help and advice. He has always made me feel comfortable discussing my
conceptual stumbling blocks, and I am now convinced that there are few things
he cannot teach with five minutes and a blackboard. Looking back, it is hard
to express how much he has helped me learn over the past three years. Modern
numerical analysis is far more intellectually satisfying that I could have ever
possibly known, and I look forward to continually expanding upon the many
insights he has been kind enough to share with me.

I also owe a large amount of thanks to Robert van de Geijn. My

master’s thesis was on parallel dense linear algebra, and most of the insights it
contained were due to our many conversations. We both have an appreciation
for continually refining and expanding core concepts and notation, and so we
have continually collaborated towards this goal. I would have never expected
so many insights from dense linear algebra to play a role in this work.

I would also like to thank Sergey Fomel for introducing me to seismic

applications and for helping to teach me some of the language of the field. I am
also grateful for his help in interfacing the Parallel Sweeping Preconditioner
with Madagascar, and for his and Siwei Li’s in working with the Overthrust
model. Computational geophysics is an exciting field and I look forward to
working in the area for years to come.

I would also like to thank Paul Tsuji for sharing his hard-earned expe-
rience in extending sweeping preconditioners to such a large class of problems

iv
and discretizations. He has been a good friend and a better colleague, and I
look forward to many fruitful future collaborations.

v
 Fast Parallel Solution of Heterogeneous
3D Time-harmonic Wave Equations

Publication No.

Jack Lesly Poulson, Ph.D.

The University of Texas at Austin, 2012

Supervisor: Lexing Ying

Several advancements related to the solution of 3D time-harmonic wave

equations are presented, especially in the context of a parallel moving-PML
sweeping preconditioner for problems without large-scale resonances. The
main contribution of this dissertation is the introduction of an efficient parallel
sweeping preconditioner and its subsequent application to several challenging
velocity models. For instance, 3D seismic problems approaching a billion de-
grees of freedom have been solved in just a few minutes using several thousand
processors. The setup and application costs of the sequential algorithm were
also respectively refined to O(γ 2 N 4/3 ) and O(γN log N ), where N denotes the
total number of degrees of freedom in the 3D volume and γ(ω) denotes the
modestly frequency-dependent number of grid points per Perfectly Matched
Layer discretization.

Furthermore, high-performance parallel algorithms are proposed for

performing multifrontal triangular solves with many right-hand sides, and a
custom compression scheme is introduced which builds upon the translation

vi
invariance of free-space Green’s functions in order to justify the replacement
of each dense matrix within a certain modified multifrontal method with the
sum of a small number of Kronecker products. For the sake of reproducibility,
every algorithm exercised within this dissertation is made available as part of
the open source packages Clique and Parallel Sweeping Preconditioner (PSP).

vii
 Table of Contents

Acknowledgments iv

Abstract vi

List of Tables xi

List of Figures xii

Chapter 1. Introduction 1
1.1 Fundamental problem . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Time-harmonic wave equations . . . . . . . . . . . . . . . . . . 3
1.3 Solving Helmholtz equations . . . . . . . . . . . . . . . . . . . 6
1.4 Background material . . . . . . . . . . . . . . . . . . . . . . . 8
1.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Chapter 2. Multifrontal methods 11

2.1 Dense Cholesky factorization . . . . . . . . . . . . . . . . . . . 12
2.2 Arrowhead matrices . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.1 Factorization . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2.2 Triangular solves . . . . . . . . . . . . . . . . . . . . . . 19
2.2.3 Separators . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3 Introduction to multifrontal methods . . . . . . . . . . . . . . 24
2.3.1 Elimination forests . . . . . . . . . . . . . . . . . . . . . 25
2.3.2 Factorization . . . . . . . . . . . . . . . . . . . . . . . . 29
2.3.3 Triangular solves . . . . . . . . . . . . . . . . . . . . . . 32
2.3.4 Computational complexity . . . . . . . . . . . . . . . . 36
2.3.5 Graph partitioning . . . . . . . . . . . . . . . . . . . . . 42
2.4 Parallel multifrontal methods . . . . . . . . . . . . . . . . . . . 43

viii
 2.4.1 Symbolic factorization . . . . . . . . . . . . . . . . . . . 45
2.4.2 Numeric factorization . . . . . . . . . . . . . . . . . . . 45
2.4.3 Standard triangular solves . . . . . . . . . . . . . . . . . 49
2.4.4 Selective inversion . . . . . . . . . . . . . . . . . . . . . 52
2.4.5 Solving with many right-hand sides . . . . . . . . . . . . 54
2.4.6 Graph partitioning . . . . . . . . . . . . . . . . . . . . . 55
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Chapter 3. Sweeping preconditioners 58

3.1 Radiation boundary conditions . . . . . . . . . . . . . . . . . . 58
3.2 Moving PML sweeping preconditioner . . . . . . . . . . . . . . 60
3.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.4 Parallel sweeping preconditioner . . . . . . . . . . . . . . . . . 70
3.4.1 The need for scalable triangular solves . . . . . . . . . . 71
3.4.2 Global vector distributions . . . . . . . . . . . . . . . . 72
3.4.3 Parallel preconditioned GMRES(k) . . . . . . . . . . . . 77
3.5 Experimental results . . . . . . . . . . . . . . . . . . . . . . . 78
3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

Chapter 4. A compressed sweeping preconditioner 86

4.1 Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.2 Interpretation, application, and parallelization . . . . . . . . . 100
4.2.1 Mapping a single vector . . . . . . . . . . . . . . . . . . 103
4.2.2 Mapping several vectors . . . . . . . . . . . . . . . . . . 103
4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
4.4 Summary and future work . . . . . . . . . . . . . . . . . . . . 106

Chapter 5. Contributions and future work 108

5.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.1.1 High-performance multifrontal triangular solves . . . . . 108
5.1.2 Parallel moving-PML sweeping preconditioner . . . . . . 109
5.1.3 Compressed moving-PML sweeping preconditioner . . . 110
5.1.4 High-performance compressed front applications . . . . 110

ix
 5.1.5 Reproducibility . . . . . . . . . . . . . . . . . . . . . . . 111
5.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

Appendices 113

Appendix A. Finite-dimensional spectral theory 114

A.1 Vector spaces and matrices . . . . . . . . . . . . . . . . . . . . 115
A.2 Existence of the spectrum . . . . . . . . . . . . . . . . . . . . 120
A.3 Schur decompositions . . . . . . . . . . . . . . . . . . . . . . . 126
A.4 The spectral theorem . . . . . . . . . . . . . . . . . . . . . . . 137
A.5 Generalized eigenspaces and matrix polynomials . . . . . . . . 139
A.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

Appendix B. Krylov subspace methods 147

B.1 Rayleigh quotients . . . . . . . . . . . . . . . . . . . . . . . . . 148
B.2 Krylov decompositions . . . . . . . . . . . . . . . . . . . . . . 150
B.3 Lanczos and Arnoldi decompositions . . . . . . . . . . . . . . . 152
B.4 Introduction to FOM and GMRES . . . . . . . . . . . . . . . 154
B.5 Implementing GMRES . . . . . . . . . . . . . . . . . . . . . . 165
B.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

Appendix C. Dense linear algebra algorithms 169

Bibliography 185

x
 List of Tables

2.1 Storage and work upper bounds for the multifrontal method
applied to d-dimensional grid graphs with n degrees of freedom
in each direction (i.e., N = nd total degrees of freedom) . . . . 41

3.1 The number of iterations required for convergence for four model
problems (with four forcing functions per model). The grid sizes
were 5003 and roughly 50 wavelengths were spanned in each di-
rection. Five grid points were used for all PML discretizations,
four planes were processed per panel, and the damping factors
were all set to 7. . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.2 Convergence rates and timings (in seconds) on TACC’s Lon-
estar for the SEG/EAGE Overthrust model, where timings in
parentheses do not make use of selective inversion. All cases
used a complex double-precision second-order finite-difference
stencil with five grid spacings for all PML (with a magnitude
of 7.5), and a damping parameter of 2.25π. The preconditioner
was configured with four planes per panel and eight processes
per node. The ‘apply’ timings refer to a single application of
the preconditioner to four right-hand sides. . . . . . . . . . . . 85

4.1 Compression of the top-panel, which consists of 14 planes, of a

2503 (and 5003 ) waveguide with ten points per wavelength using
relative tolerances of 1e-2 and 5e-2 for the diagonal blocks and
connectivity, respectively . . . . . . . . . . . . . . . . . . . . . 106

xi
 List of Figures

2.1 An 8×8 structurally-symmetric sparse matrix and its associated

graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2 (a) The arrowhead sparsity pattern of an 8×8 matrix reordered
via graph partitioning and (b) the associated graph with the
connections to the separator clipped . . . . . . . . . . . . . . . 23
2.3 A two-level arrowhead matrix and its Cholesky factor, L . . . 25
2.4 (a) A symmetric two-level arrowhead matrix and (b) its elimi-
nation tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.5 A 15 × 15 grid graph. Each node is connected to its nearest
horizontal and vertical neighbors. . . . . . . . . . . . . . . . . 37
2.6 A separator-based elimination tree (right) for a 15 × 15 grid
graph (left) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.7 An elimination tree based on cross separators (right) of a 15×15
grid graph (left) . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.8 An example of a (slightly imbalanced) elimination tree dis-
tributed over four processes: the entire set of processes shares
the root node, teams of two processes share each of the two
children, and each process is assigned an entire subtree rooted
at the dashed line. . . . . . . . . . . . . . . . . . . . . . . . . 44
2.9 Overlay of the process ranks (in binary) of the owning subteams
of each supernode from an elimination tree assigned to eight
processes; a ‘*’ is used to denote both 0 and 1, so that ‘00∗’
represents processes 0 and 1, ‘01∗’ represents processes 2 and 3,
and ‘∗ ∗ ∗’ represents all eight processes. . . . . . . . . . . . . 44
2.10 An example of the portion of a symbolic factorization to be
computed by a single process (enclosed in the dashed blue line)
for an elimination tree distributed over eight processes. Each
step towards the top of the tree requires an interaction with the
team which computed the factored structure of the other child
(each of these interactions is surrounded with dashed red lines). 46

3.1 An x1 x3 cross section of a cube with PML on its x3 = 0 face.

The domain is shaded in a manner which loosely corresponds
to its extension into the complex plane. . . . . . . . . . . . . . 62

xii
3.2 (Left) A depiction of the portion of the domain involved in the
computation of the Schur complement of an x1 x2 plane (marked
with the dashed line) with respect to all of the planes to its left
during execution of Alg. 3.1. (Middle) An equivalent auxiliary
problem which generates the same Schur complement; the orig-
inal domain is truncated just to the right of the marked plane
and a homogeneous Dirichlet boundary condition is placed on
the cut. (Right) A local auxiliary problem for generating an
approximation to the relevant Schur complement; the radiation
boundary condition of the exact auxiliary problem is moved
next to the marked plane. . . . . . . . . . . . . . . . . . . . . 64
3.3 A separator-based elimination tree (right) over a quasi-2D sub-
domain (left) . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.4 Overlay of the owning process ranks of an 7 × 7 matrix dis-
tributed over a 2 × 3 process grid in the [MC , MR ] distribution,
where MC assigns row i to process row i mod 2, and MR assigns
column j to process column i mod 3 (left). The process grid is
shown on the right. . . . . . . . . . . . . . . . . . . . . . . . . 73
3.5 Overlay of the owning process ranks of a vector of height 7
distributed over a 2 × 3 process grid in the [VC , ?] vector dis-
tribution (left) and the [VR , ?] vector distribution (right). . . . 75
3.6 Overlay of the owning process ranks of a vector of height 7
distributed over a 2 × 3 process grid in the [MC , ?] distribution
(left) and the [MR , ?] distribution (right). . . . . . . . . . . . . 75
3.7 A single x2 x3 plane from each of the four analytical velocity
models over a 5003 grid with the smallest wavelength resolved
with ten grid points. (Top-left) the three-shot solution for the
barrier model near x1 = 0.7, (bottom-left) the three-shot so-
lution for the two-layer model near x1 = 0.7, (top-right) the
single-shot solution for the wedge model near x1 = 0.7, and
(bottom-right) the single-shot solution for the waveguide model
near x1 = 0.55. . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.8 Three cross-sections of the SEG/EAGE Overthrust velocity model,
which represents an artificial 20 km × 20 km × 4.65 km domain,
containing an overthrust fault, using samples every 25 m. The
result is an 801 × 801 × 187 grid of wave speeds varying discon-
tinuously between 2.179 km/sec (blue) and 6.000 km/sec (red). 83
3.9 Three planes from an 8 Hz solution with the Overthrust model
at its native resolution, 801 × 801 × 187, with a single localized
shot at the center of the x1 x2 plane at a depth of 456 m: (top)
a x2 x3 plane near x1 = 14 km, (middle) an x1 x3 plane near
x2 = 14 km, and (bottom) an x1 x2 plane near x3 = 0.9 km. . . 84

xiii
4.1 (Top) The real part of a 2D slice of the potential generated by a
source in the obvious location and (bottom) the same potential
superimposed with that of a mirror-image charge. The potential
is identically zero along the vertical line directly between the two
charges. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.2 A set of six unevenly spaced points (in blue) along the line seg-
ment {0} × [0, 1] × {0} and their four evenly spaced translations
(in red) in the direction ê3 . . . . . . . . . . . . . . . . . . . . . 93
4.3 Convergence of compressed moving PML sweeping precondi-
tioner in GMRES(20) for a 2503 waveguide problem, with ten
points per wavelength and relative tolerances of 0.01 and 0.05
for the Krocker product approximations of the top-left and
bottom-left quadrants of each frontal matrix, respectively . . . 106

xiv
 Chapter 1

Introduction

The holy grail of exploration geophysics is to be able to take a boat out

to sea, fire off air guns near the surface of the water, and to use measurements
of the resulting echoes as a means of quickly constructing an accurate model
for the wave propagation characteristics throughout the portion of the earth
below the boat. This is an important example of an inverse problem, and
a large percentage of the seismic community is dedicated to at least finding
qualitative solutions to questions such as, “Do the material properties imply
a rock formation which is likely to contain hydrocarbons?”.

1.1 Fundamental problem

The name inverse problem is a reference to the usual method of attack:
if we assume the necessary material properties throughout the relevant portion
of the domain then we may use a forward model, such as the 3D elastic wave
equation,
∂2
−∇ · (C(x)∇U(x, t)) + ρ(x) 2 U(x, t) = F(x, t), (1.1)
∂t
to model the evolution of the displacement field, U(x, t), given the forcing
function F(x, t), which is perhaps the result of a blast from an air gun or stick
of dynamite. The difference between this simulation and a physical experi-
ment can then be used to drive an iterative method which hopefully converges
towards the “true” mass density, ρ(x), and fourth-order stiffness tensor, C(x).

1
 While the details of effective inversion techniques are well beyond the
scope of this document, it is important to recognize a few common character-
istics:

• the forward model must be used many times,

• the same forward model is typically used for many experiments,

• many researchers prefer to work in the temporal-frequency domain [115],

and, perhaps most importantly,

• the domain often spans large numbers of wavelengths, which requires a

fine-scale discretization (and therefore a large-scale linear system).

In order to drive the last point home, let us recognize that the speed of sound
in water is roughly 1.5 km/sec, which implies that it would take roughly 10
seconds for sound to propagate across a distance of 15 km. If a sound wave
oscillates at a frequency of 30 Hz, then, throughout the course of the 10 seconds
it takes to travel this distance, it will oscillate 300 times.

In order to capture these oscillations over a 15 km × 15 km region

of the surface, significantly more than 300 grid points would be required in
each of the two horizontal directions. We might say that such a domain spans
300 wavelengths and describe the quality of the discretization in terms of the
number of points per wavelength (eight to ten points per wavelength is quite
common in industry).1

1
It is important to combat so-called pollution effects [15], for instance using high-order
spectral elements [123, 124], but this issue is somewhat orthogonal to the focus of this
dissertation.

2
1.2 Time-harmonic wave equations
Consider the scalar analogue of the elastic wave equation, the 3D acous-
tic wave equation [121],

1 ∂2
 
−∆ + U (x, t) = F (x, t), (1.2)
c2 (x) ∂t2
where U (x, t) is the time and spatially-varying pressure field, F (x, t) is the
forcing function, and c(x) is the sound speed. If we were to perform a for-
mal Fourier transform [84] in the time variable of each side of this equation,
we would decouple the solution U (x, t) into a sum of solutions of the form
u(x)e−iωt , each driven by a forcing function of the form f (x)e−iωt , where ω is
the temporal frequency, which usually has units of radians per second.2 Since
∂2 
u(x)e−iωt = −ω 2 u(x)e−iωt ,
  
∂t 2

for each frequency ω, we can instead pose a Helmholtz equation,

ω2
 
−∆ − 2 u(x) = f (x). (1.3)
c (x)

Because of the implicit e−iωt time-dependence of the solution and forcing func-
tion, the Helmholtz equation is (the prototypical example of) a time-harmonic
wave equation.

It is important to recognize that the negative Laplacian operator, −∆,

when discretized over a finite domain with homogeneous boundary conditions,
results in a real, symmetric, positive-definite matrix, with eigenvalues on the
positive real-line which become arbitrarily close to the origin and approach

2
Different communities have different conventions for Fourier transforms. It is also com-
mon to use a time-dependence of eiωt , but this essentially only requires conjugating the
appropriate terms in the following discussion.

3
infinity as the discretization is refined. The Helmholtz operator, −∆ − ω 2 ,
therefore has the same eigenvalues, but shifted left by a distance of ω 2 in the
complex plane. Thus, for any nonzero frequency ω and sufficiently fine dis-
cretization, its eigenvalues span both sides of the origin on the real line (and
are often quite close to the origin). This admittedly crude interpretation of the
spectrum of the Helmholtz operator explains the essential difficulty with solv-
ing Helmholtz equations via iterative methods. For instance, because of the
shape of the spectrum, it is not possible to construct an ellipse in the complex
plane which contains the spectrum but not the origin (see Definition B.4.4).

An analogous, though significantly more complex, technique can be

used to reduce Maxwell’s equations to time-harmonic form [123]. Consider
the constant-coefficient Maxwell’s equations:

∇ × E(x, t) = −Ḣ(x, t), (1.4)

∇ × H(x, t) = Ė(x, t) + J(x, t), (1.5)
i
∇ · E(x, t) = ∇ · J(x, t), and (1.6)
ω
∇ · H(x, t) = 0, (1.7)

where E(x, t) and H(x, t) respectively model the three-dimensional electric and
magnetic fields, and J(x, t) is the current distribution, which is also a three-
dimensional vector quantity. If we again perform a formal Fourier transform in
the time variable of each set of equations, then we may represent the solutions
E(x, t) and H(x, t) as sums of terms of the form E(x)e−iωt and H(x)e−iωt ,
which are each driven by a harmonic component J (x)e−iωt of the current
distribution, J(x, t). Substituting these harmonic quantities into Maxwell’s

4
equations yields

∇ × E(x) = iωH(x), (1.8)

∇ × H(x) = −iωE(x) + J (x), (1.9)
i
∇ · E(x) = ∇ · J (x), and (1.10)
ω
∇ · H(x) = 0. (1.11)

The last step in deriving the time-harmonic form of Maxwell’s equations

involves the vector identity

∇ × ∇ × a = ∇(∇ · a) − ∆a, (1.12)

which may be readily proved with so-called index notation [104] via the prod-
uct rule and knowledge of the Levi-Civita symbol, which is also known as the
alternating symbol. In particular, if we take the curl of both sides of Equa-
tion (1.8), then our vector identity allows us to write

∇(∇ · E) − ∆E = iω∇ × H.

We may then substitute Equations (1.10) and (1.11) in order to arrive at the
result
i
−∆ − ω 2 E(x) = fE (J ) ≡ iωJ − ∇(∇ · J ),
 
(1.13)
ω
which was purposely written in a form similar to that of Equation (1.3). We
may similarly take the curl of Equation (1.9) in order to find

−∆ − ω 2 H(x) = fH (J ) ≡ ∇ × J ,
 
(1.14)

which is clearly also similar in structure to the Helmholtz equation. Note that
these time-harmonic equations for the electric and magnetic fields, which are

5
ostensibly decoupled, will typically interact through an appropriate choice of
boundary conditions.

Before we move on, it is important to emphasize that the equations

for 3D time-harmonic linear elasticity are essentially an anisotropic form of
the Helmholtz equation, which roughly means that waves do not propagate
radially outwards equally in each direction. In particular, a formal Fourier
transform of the elastic wave equation implies the time-harmonic elastic wave
equation,
−(Cijkl Uk,l ),j − ω 2 ρUi = Fi , (1.15)

which we have written in the usual index notation.3 The basic idea is that xi
refers to the i’th coordinate of the vector x, xi,j refers to the derivative of the
i’th coordinate in the j’th direction, and a repeated index implies that it is a
dummy summation variable, e.g., Aik Bkj is an expression of the (i, j) entry of
the matrix AB, and Aki Bkj represents the (i, j) entry of AT B.

1.3 Solving Helmholtz equations

Until recently, doubling the frequency of Equation (1.3) not only in-
creased the size of the linear system by at least a factor of two in each dimen-
sion, it also doubled the number of iterations required for convergence with
preconditioned Krylov methods [23, 45, 47]. Thus, denoting the number of de-
grees of freedom in a three-dimensional finite-element or finite-difference dis-
cretization as N = Ω(ω 3 ), every linear solve required Ω(ω 4 ) work with iterative
techniques. Engquist and Ying recently introduced two classes of sweeping pre-

3
For an introduction to index notation and linear elasticity, please consult a book on
continuum mechanics, such as [104].

6
conditioners for Helmholtz equations without internal resonance [43, 44]. Both
approaches approximate a block LDLT factorization of the Helmholtz opera-
tor in block tridiagonal form in a manner which exploits a radiation boundary
condition [78]. The first approach performs a block tridiagonal factorization
algorithm in H-matrix arithmetic [59, 66], while the second approach approx-
imates the Schur complements of the factorization using auxiliary problems
with artificial radiation boundary conditions. Though the H-matrix sweeping
preconditioner has theoretical support for two-dimensional problems [43, 92],
there is not yet justification for three-dimensional problems.

This dissertation therefore focuses on the second approach, which relies

on multifrontal factorizations [41, 54, 91, 112] of approximate auxiliary prob-
lems in order to achieve an O(γ 2 N 4/3 ) setup cost and an O(γN log N ) ap-
plication cost, where γ(ω) denotes the number of grid points used to dis-
cretize each radiation boundary condition (in our case, a Perfectly Matched
Layer [19, 29, 78]), and we note that γ typically grows logarithmically with
frequency. While the sweeping preconditioner is competitive with existing
techniques even for a single right-hand side, its main advantage is for prob-
lems with large numbers of right-hand sides, as the preconditioner appears
to converge in O(1) iterations for problems without internal resonance [44].
Thus, after setting up the preconditioner, typically only O(γN log N ) work
is required for each solution. As mentioned at the beginning of this chapter,
frequency-domain seismic inversion requires the solution of large numbers of
time-harmonic wave equations posed over domains spanning large numbers of
wavelengths, and so we will focus on the efficient implementation of a parallel
sweeping preconditioner in order to achieve solution costs closer to O(ω 3 ) than
O(ω 4 ).

7
1.4 Background material
A more detailed discussion of the sweeping preconditioner requires a
thorough knowledge of sparse-direct methods and at least a passing familiar-
ity with iterative methods and wave equations. Chapter 2 gives a detailed
introduction to multifrontal Cholesky factorization and triangular solves un-
der the assumption that the reader is well-versed in numerical linear algebra.
If this is not the case, it is recommended that the reader work through the
proofs in Appendix A leading up to the Hermitian spectral decomposition
(Corollary A.2). In addition, Appendix B is provided for those who would
like to familiarize themselves with the theory behind the Generalized Min-
imum Residual method (GMRES) [108] used in Chapters 3 and 4. Lastly,
Appendix C is provided for readers who are interested in the (parallel) dense
linear algebra algorithms used at the core of the multifrontal algorithms dis-
cussed in Chapter 2. Their implementations are all part of the Elemental
library [99], which is written in a style derived from FLAME notation [60]. It
is also strongly recommended that the reader familiarize themselves with the
fundamentals behind BLAS [37], LAPACK [5], and MPI [39]. In particular,
readers interested in gaining a better understanding of collective communica-
tion algorithms should consult [28].

1.5 Contributions
There are essentially four significant contributions in this dissertation:

1. two high-performance algorithms for multifrontal triangular solves are

introduced,

8
 2. a parallelization of the sweeping preconditioner is introduced and applied
to large-scale challenging heterogeneous 3D Helmholtz equations,

3. a compressed variant of the sweeping preconditioner is introduced which

attempts to exploit the (approximate) translation invariance of free-space
Green’s functions, and

4. a high-performance algorithm for applying the compressed frontal ma-

trices in their Kronecker-product form is proposed.

Together, these advancements allow for the solution of large-scale 3D Helmholtz

equations at unprecedented rates. In particular, it will be shown in Chapter 3
that, after spending a few minutes setting up the sweeping preconditioner, 3D
systems of equations approaching a billion degrees of freedom can be solved
in a matter of seconds.

1.6 Outline
The remainder of this dissertation is organized as follows:

• Chapter 2 provided a detailed introduction to sequential and parallel

algorithms for multifrontal Cholesky factorization from the point of view
of reordering the operations in a dense Cholesky factorization. Readers
who are not concerned with the inner-workings of sparse-direct solvers
may simply want to familiarize themselves with Table 2.1.

• Chapter 3 provides an introduction to the moving PML sweeping pre-

conditioner, describes a careful parallelization, and then discusses results
for several challenging velocity models.

9
 • Chapter 4 describes how the translation invariance of free-space Green’s
functions can be used to compress certain half-space Green’s functions
associated with the diagonal blocks of a modified multifrontal factoriza-
tion, as well as how these compressed matrices may be efficiently applied
as part of a multifrontal triangular solve.

• And, lastly, Chapter 5 gives a brief review of the contributions made

throughout this dissertation and then concludes with possible future
work and the websites of the software produced as part of this disserta-
tion.

As previously mentioned, Appendix A is meant to provide a brief introduction

to finite-dimensional spectral theory, Appendix B gives a short introduction
to GMRES, and Appendix C is meant to provide a concise description of the
dense linear algebra algorithms used throughout this dissertation.

10
 Chapter 2

Multifrontal methods

This chapter attempts to give a self-contained, elementary introduction

to efficient distributed-memory algorithms for multifrontal [41, 91] Cholesky
factorization, as later chapters will rely upon an essentially identical approach
for indefinite sparse LDLT factorization as part of a sweeping preconditioner [44].
Many numerical analysts will undoubtedly feel uneasy about unpivoted LDLT
factorizations, but this ostensibly naı̈ve approach is at least partially justified
for the following reasons:

• sparse-direct solvers will only be used to solve approximate auxiliary

problems as part of a preconditioner,

• the spectra are typically well-separated from the origin, and

• no instabilities have been encountered, even for large-scale models.

Nevertheless, scalable pivoted variants of the algorithms discussed in this chap-

ter will be investigated as part of future work. Readers who are not interested
in the inner-workings of multifrontal algorithms may safely proceed to Chap-
ter 3 after familiarizing themselves with the asymptotic complexities listed in
Table 2.1.

Although a major portion of this chapter will be a review of well-

known results, we will deviate from standard approaches to scalable multi-

11
frontal Cholesky factorization and solution [56, 63, 64, 113] in the following
two ways:

1. element-wise matrix distributions [71, 93, 99] will be used for parallelizing
dense linear algebra operations, and

2. distributed sparse-direct triangular solve algorithms built upon dense

distributed TRSM and GEMM [36, 99] kernels will be introduced, where
the latter exploits selective inversion [102, 103, 114].

Before we start, it is also important to briefly mention the context of

multifrontal methods within the world of sparse solvers. By the end of this
chapter, we will have shown that a multifrontal factorization is essentially a re-
ordering of the operations within a right-looking dense Cholesky factorization,
where the ability to reorder said operations is precisely due to the sparsity
of the matrix being factored. It is, of course, possible to modify other dense
Cholesky factorization algorithms in order to produce alternative sparse fac-
torization algorithms, such as the column fan-in [10] approach. Also, a short
note on the name multifrontal itself: it is a reference to the earlier frontal
scheme of Irons [75], which was an improvement to banded Cholesky factoriza-
tion. Rather than discussing this progression in detail, we will instead tran-
sition from dense to sparse matrices through the class of (block) arrowhead
matrices.

2.1 Dense Cholesky factorization

Definition 2.1.1 (Hermitian positive (semi-)definite). A matrix is called Her-
mitian positive semi-definite (HPSD) if it is Hermitian and every eigenvalue is

12
non-negative, and Hermitian positive-definite (HPD) when every eigenvalue is
positive.

Theorem 2.1.1 (semi-definite Cholesky decomposition). Every Hermitian

positive semi-definite matrix has a Cholesky decomposition,

A = RH R,

where R is upper-triangular.

Proof. As a consequence of Corollary A.2, every HPSD matrix A ∈ Mn , where

Mn is the space of n × n complex matrices, has a spectral decomposition

A = V ΛV H ,

where V ∈ Mn is unitary and, by assumption, every eigenvalue along the

diagonal of Λ is non-negative. We may thus define the square-root of the
diagonal matrix Λ entry-wise along its diagonal, and we denote this result as
√
Λ. Then the operator
√
A1/2 = V ΛV H

is also HPSD, and A1/2 A1/2 = A, which justifies the choice of notation.

Now, let us apply Theorem A.9 to A1/2 to show there exists some
unitary matrix Q ∈ Mn and upper-triangular matrix R ∈ Mn such that A1/2 =
QR. But, A1/2 is Hermitian, and so A1/2 = RH QH , and we find that

A = RH QH QR = RH R.

13
Remark 2.1.1. We have now justified an interpretation of a Cholesky factor
of a Hermitian positive semi-definite matrix A as the square-root of A: it may
always be chosen to be the upper-triangular factor, R, of a QR decomposition
of A1/2 . We will now strengthen our assumptions and give a constructive proof
of uniqueness which will serve as the starting point for the remainder of this
chapter.

Lemma 2.1.2. If A ∈ Mn is Hermitian positive-definite, and X ∈ Mn,k has

linearly independent columns, then X H AX is also Hermitian positive-definite.

Proof.
(y, X H AXy) = ((Xy), A(Xy)) ≥ 0,

with equality if and only if Xy = 0. But, because the columns of X are linearly
independent, Xy must be nonzero.

Theorem 2.1.3 (Cholesky factorization). Every Hermitian positive-definite

matrix has the unique Cholesky factorization

A = LLH ,

where L is lower-triangular with positive diagonal entries.

Proof. If A ∈ Mn is HPD, then Theorem 2.1.1 guarantees the existence of

some upper-triangular matrix R ∈ Mn such that A = RH R. Let us partition
this matrix expression as
2 H
      
α1,1 aH2,1 ρ 1,1 0 ρ 1,1 r H
1,2 |ρ 1,1 | ρ 1,1 r1,2
= H = H H .
a2,1 A2,2 r1,2 R2,2 0 R2,2 ρ1,1 r1,2 r1,2 r1,2 + R2,2 R1,1

Since A is nonsingular, R must also be nonsingular, and thus Theorem A.1

guarantees that the diagonal entries of R are nonzero. We can thus recognize

14
that the top-left entry of A, α0,0 = |ρ0,0 |2 , is positive, and so we can legally take
its square root and divide by it. Consider the partitioned matrix expression

λ1,1 `H
  2
λ1,1 `H
    
α1,1 aH2,1 λ1,1 0 1,0 λ1,1 2,1
= = ,
a2,1 A2,2 `2,1 L2,2 0 LH 2,2 λ1,1 `H H
2,1 `2,1 `2,1 + L2,2
 
λ1,1 0
where L = is the lower-triangular Cholesky factor we wish to
`2,1 L2,2
show can be chosen to have a positive diagonal. Clearly we may set λ1,1 =
√
α1,1 and `2,1 := a2,1 /λ1,1 , and then find that LH H
2,2 L2,2 = A2,2 − `2,1 `2,1 , but,

in order to recurse on the updated bottom-right quadrant, we must first show

that the Schur complement A2,2 − `2,1 `H
2,1 inherits the positive-definiteness of

A. But this Schur complement is the result of the product X H AX, where
 
−a2,1 /α1,1
X=
I

has linearly independent columns. The preceding lemma therefore yields the
result.

Remark 2.1.2. The recursive procedure for computing the lower-triangular

Cholesky factor of a Hermitian positive-definite matrix given in the previous
theorem is called a right-looking Cholesky factorization, and its pseudocode
is given in Algorithm C.16. Furthermore, it can easily be generalized into
a so-called blocked algorithm [5], which expresses most of the factorization
in terms of matrix-matrix multiplication in order to minimize data movement
between different levels of the memory hierarchy. The pseudocode for a blocked
right-looking Cholesky factorization is given in Algorithm C.17, though the
upcoming subsection will work with the decomposition
 H
A0,0 AH H
L0,0 LH H
   
1,0 A2,0 L0,0 0 0 1,0 L2,0
A1,0 A1,1 AH 2,1
 = L1,0 L1,1 0   0 LH H 
1,1 L2,1 ,
A2,0 A2,1 A2,2 L2,0 L2,1 L2,2 0 0 LH 2,2

15
which can be directly computed with Algorithm 2.1 using the notation
   
L1,0 A1,1 A1,2
L1:2,0 ≡ , A1:2,1:2 ≡ .
L2,0 A2,1 A2,2

Algorithm 2.1: unrolled 3 × 3 block Cholesky factorization

1 L0,0 := Cholesky(A0,0 )
−H
2 L1:2,0 := A1:2,0 L0,0
H
3 A1:2,1:2 := A1:2,1:2 − L1:2,0 L1:2,0
4 L1,1 := Cholesky(A1,1 )
−H
5 L2,1 := A2,1 L1,1
H
6 A2,2 := A2,2 − L2,1 L2,1
7 L2,2 := Cholesky(A2,2 )

2.2 Arrowhead matrices

Now suppose that A1,0 = AH
0,1 = 0 so that we may write

A0,0 0 AH
 
2,0
A =  0 A1,1 AH 2,1
. (2.1)
A2,0 A2,1 A2,2

We say that such a matrix has arrowhead [133] form.1 This class of matrices
is of interest because it allows us to easily motivate many of the basic ideas
behind the multifrontal method. In particular, the fact that A1,0 = 0 allows
us to decouple and reorganize many of the operations in Algorithm 2.1.

1
We should perhaps call A a block-arrowhead matrix for the same reason that a block-
diagonal matrix should not be called diagonal. Despite their usefulness for Hermitian eigen-
value problems, we will not make use of unblocked arrowhead matrices in this dissertation,
and so we will drop the block qualifier.

16
2.2.1 Factorization

Because A1,0 = 0, it is easy to see that Step 2 of Algorithm 2.1 will

compute L1,0 := 0 (as well as L2,0 := A2,0 L−H
0,0 ), so that the Cholesky factor,

L, has the block structure

 
L0,0 0 0
L =  0 L1,1 0  . (2.2)
L2,0 L2,1 L2,2
Once we recognize that A1,0 = L1,0 = 0, we may easily modify Algorithm 2.1
into Algorithm 2.2.

Algorithm 2.2: arrowhead Cholesky

1 L0,0 := Cholesky(A0,0 )
−H
2 L2,0 := A2,0 L0,0
H
3 A2,2 := A2,2 − L2,0 L2,0
4 L1,1 := Cholesky(A1,1 )
−H
5 L2,1 := A2,1 L1,1
H
6 A2,2 := A2,2 − L2,1 L2,1
7 L2,2 := Cholesky(A2,2 )

It is important to notice that Steps 4-5 of Algorithm 2.2 are independent

of Steps 1-3, and thus we may execute Steps 1-2 in parallel with Steps 4-5.
In addition, the vast majority of the work involved in Steps 3 and 6 lies in
the matrix-matrix multiplication required for the formation of the Hermitian
updates, say Uj ≡ −L2,j LH
2,j , j = 0, 1, as their subsequent addition to A2,2 has

a lower-order cost. For the sake of parallelism, it is then natural to form and
store these updates for later use. The result is Algorithm 2.3.

We can now recognize that Steps 2-4 of Algorithm 2.3 are equivalent
to the partial Cholesky factorization of the original frontal matrix
 
Aj,j ?
F̂j = ,
A2,j 0

17
 Algorithm 2.3: reorganized arrowhead Cholesky
1 foreach j ∈ {0, 1} do
2 Lj,j := Cholesky(Aj,j )
3 L2,j := A2,j L−H
j,j
4 Uj := −L2,j LH 2,j
5 end
6 A2,2 := A2,2 + U0 + U1
7 L2,2 := Cholesky(A2,2 )

where we have used a ? to denote a quadrant that will not be accessed due to
implicit symmetry (i.e., A2,j = AH
j,2 ). If a right-looking Cholesky factorization

algorithm applied to F̂j is stopped just before beginning the factorization of

the bottom-right quadrant, then the resulting factored frontal matrix will be
 
Lj,j ?
Fj = .
L2,j Uj

We will refer this partial-factorization process as processing the front. We

can also define the initial state of the root front to be F̂2 = A2,2 , which must
be updated with U0 and U1 and then completely factored. It is useful to
interpret the root front in the same manner as the previous fronts, but with a
0 × 0 bottom-right quadrant. Processing the root front is then equivalent to
computing its full Cholesky factorization. Algorithm 2.3 can now be concisely
summarized in the language of fronts (see Algorithm 2.4).

The scheme we have just described is essentially a non-recursive version

of the multifrontal method [41, 91], which derives its name from the fact that
several fronts are simultaneously maintained. The next subsection will give an
overview of how to perform triangular solves using the processed fronts, and
then we will dive into the details behind applying the multifrontal method
to more general classes of matrices. We also note that all of the above (and

18
 Algorithm 2.4: “multifrontal” arrowhead Cholesky
1 foreach j ∈ {0, 1} do
2 Fj := F̂j
3 Process Fj
4 end

5 F2 := F̂2
6 Add bottom-right quadrants of F0 and F1 onto F2
7 Process F2

proceeding) algorithms may be trivially modified to perform unpivoted LDLT

and LDLH factorizations, which allows us to handle a significant subset of
complex-symmetric and Hermitian arrowhead matrices.

2.2.2 Triangular solves

The first thing to notice is that the lower-triangular matrix L, expressed

in the form of Equation (2.2), is completely represented by the top-left and
bottom-left quadrants of the processed fronts, {F0 , F1 , F2 }. It is then reason-
able to expect to modify dense triangular solve algorithms for applying L−1
and L−H so that they exploit arrowhead structure and access L through the
processed fronts.

Before we consider the arrowhead triangular solve

   −1  
X0 L0,0 0 0 X0
X1  :=  0 L1,1 0  X1  , (2.3)
X2 L2,0 L2,1 L2,2 X2

let us recognize that the dense triangular solve

   −1  
X0 L0,0 0 0 X0
X1  := L1,0 L1,1 0  X1  (2.4)
X2 L2,0 L2,1 L2,2 X2

19
can be performed with Algorithm 2.5, which simply executes a matrix version
of forward substitution (see Algorithm C.6). Clearly L1,0 = 0 implies that
Step 2 of Algorithm 2.5 can be replaced with X2 := X2 − L2,0 X0 , and the re-
sulting optimized algorithm is given in Algorithm 2.6. Just as we introduced
temporary storage in order to decouple the updates within an arrowhead fac-
torization, temporarily storing the update matrices Zj ≡ −L2,j Xj , j = 0, 1,
allows us to effectively decouple Steps 1-2 and 3-4 within Algorithm 2.6, re-
sulting in Algorithm 2.7.

Algorithm 2.5: unrolled 3 × 3 block triangular solve

−1
1 X0 := L0,0 X0
2 X1:2 := X1:2 − L1:2,0 X0
−1
3 X1 := L1,1 X1
4 X2 := X2 − L2,1 X1
−1
5 X2 := L2,2 X2

Algorithm 2.6: arrowhead solve

−1
1 X0 := L0,0 X0
2 X2 := X2 − L2,0 X0
−1
3 X1 := L1,1 X1
4 X2 := X2 − L2,1 X1
−1
5 X2 := L2,2 X2

Algorithm 2.7: reorganized arrowhead solve

1 foreach j ∈ {0, 1} do
2 Xj := L−1j,j Xj
3 Zj := −L2,j Xj
4 end
5 X2 := X2 + Z0 + Z1
−1
6 X2 := L2,2 X2

20
 Applying the inverse of LH , say,
 H −1  
L0,0 0 LH
 
X0 2,0 X0
X1  :=  0 LH1,1 LH 
2,1
 X1  , (2.5)
H
X2 0 0 L2,2 X2

is an equally straight-forward modification of dense backward substitution (Al-

gorithm C.11). The resulting steps are given by Algorithm 2.8 and show that,
if we interpret the processed fronts, {F0 , F1 , F2 }, as a tree, where F2 is the
root and F0 and F1 are the leaves, then we see that applying L−H requires a
downward traversal of the frontal tree, while applying L−1 requires an upward
traversal. In the following subsection we will see that, for the full multifrontal
method, factorizations and forward substitution will both correspond to an up-
ward traversal of a frontal tree, while backward substitution will correspond
to a downward traversal.

Algorithm 2.8: reorganized arrowhead adjoint solve

Input: L,X
Output: X := L−H X
−H
1 X2 := L2,2 X2
2 foreach j ∈ {0, 1} do
3 Xj := Xj − LH 2,j X2
−H
4 Xj := Lj,j Xj
5 end

2.2.3 Separators

There is an important interpretation of arrowhead matrices in the lan-

guage of graphs. Let us define the graph, G = (V, E), of a matrix A ∈ Mn
to be the pairing of a set of n vertices, V = {0, 1, . . . , n − 1}, and a set of
edges, E ⊂ V × V , such that the graph G contains an edge (i, j) ∈ E if and

21
only if the (i, j) entry of A is nonzero (it is therefore common to associate
vertex j with the j’th diagonal entry of A). Because A1,0 = AH
0,1 = 0 within a

Hermitian 3 × 3 arrowhead matrix, we see that no edges connect the vertices

associated with the diagonal entries of A0,0 and A1,1 , and so it is natural to
say that the remaining vertices (those of A2,2 ) separate the graph associated
with the matrix A.

Example 2.2.1. An example of a graph associated with a 8 × 8 symmetric

sparse matrix is shown in Figure 2.1. In its current state, this matrix is

1
 
0 X X X X 5

 1 X X 


 X 2 X X

 2
 
X X 3 X X
3
 
X 4 
 
X X 5 X
 
0

 
X X 6 
X X X 7 6 4 7

(a) symmetric sparsity pattern (b) associated graph

Figure 2.1: An 8 × 8 structurally-symmetric sparse matrix and its associated

graph

clearly not in arrowhead form, but it turns out that investigating its graph
leads to a way to permute the matrix into the desired form: If we clipped all
external connections to vertices {3,7}, we would end up with the graph shown
in Figure 2.2b. It should be clear from the clipped graph that the remaining set
of vertices can be grouped into two subsets, V0 = {1, 2, 5} and V1 = {0, 4, 6},
which do not directly interact; we therefore refer to vertices V2 = {3, 7} as a
separator. If we now reorder the sparse matrix so that vertex set V0 appears

22
  
1 X X
X 2 X X 
 
X X 5 X
 
 
 0 X X X X
  1 0 3

 X 4 

X 6 X
 



 5 4 6 7
 X X X 3 X
X X X 7 2

(a) reordered sparsity pattern (b) clipped graph

Figure 2.2: (a) The arrowhead sparsity pattern of an 8 × 8 matrix reordered

via graph partitioning and (b) the associated graph with the connections to
the separator clipped

first, followed by V1 , and finally the separator V2 , then the result, shown in
Figure 2.2a, is in arrowhead form.

Remark 2.2.1. The fact that the top-left and bottom-right blocks of the
matrix in Figure 2.2a are dense corresponds to their associated sets of vertices,
V0 and V2 , being fully connected subsets of the induced subgraph.

Definition 2.2.1 (clique). A subset of vertices of a graph such that every

vertex in the subset is directly connected to every other vertex in the subset
is referred to as a clique.

Definition 2.2.2 (induced subgraph). Given a graph G = (V, E) and a subset

of vertices, say U ⊂ V , the induced subgraph G|U = (U, E|U ), the restriction
of the graph G onto the vertex set U , is defined by setting

E|U = {{vi , vj } ∈ E : vi , vj ∈ U }.

Definition 2.2.3 (complete graph). A graph G = (V, E) is called complete,

or fully connected, if every vertex is directly connected to every other vertex.

23
Remark 2.2.2. We can now exercise our new terminology: the vertex subsets
V0 and V2 from the previous example are both cliques, and thus their induced
subgraphs, G|V0 and G|V2 , are complete/fully-connected.

Definition 2.2.4 (separator). We say that a vertex subset V2 ⊂ V of a graph

G = (V, E) separates the graph G into vertex subsets V0 and V1 if the induced
graph G|V\V2 = (V \V2 , E|V\V2 ) is the union of two disconnected sets of vertices,
i.e.:

1. V \V2 = V0 ∪ V1 , and

2. {v0 , v1 } ∈ E|V\V2 : v0 ∈ V0 , v1 ∈ V1 = ∅.

We will delay discussion of the important problem of how to compute

graph partitions until much later in this chapter. For now, it will suffice to
mention that it is an NP-complete problem [53] that is usually attacked with
multi-level heuristics [69, 70, 81].

2.3 Introduction to multifrontal methods

It should not be surprising that our next step is to recurse on the graph
partitioning approach described in the previous section. That is, given a par-
titioning V = V0 ∪ V1 ∪ V2 , where V2 separates V0 and V1 , nested dissection [54]
recursively partitions the restricted graphs G|V0 and G|V1 in order to expose a
recursive arrowhead structure (see Figure 2.3 for a two-level example).

If we now revisit Algorithm 2.3, then, for a two-level arrowhead ma-

trix, the dense Cholesky factorization in step 2 should be a recursive call to
another arrowhead Cholesky, and step 3, which requires an application of the
inverse of the factored matrix, should use the arrowhead solve algorithms, i.e.,

24
    
A0,0 A0,2 A0,6 L0,0

 A1,1 A1,2 A1,6 


 L1,1 

A2,0 A2,1 A2,2 A2,6  L2,0 L2,1 L2,2 
   

 A3,3 A3,5 A3,6 


 L3,3 


 A4,4 A4,5 A4,6 


 L4,4 

 A5,3 A5,4 A5,5 A5,6   L5,3 L5,4 L5,5 
A6,0 A6,1 A6,2 A6,3 A6,4 A6,5 A6,6 L6,0 L6,1 L6,2 L6,3 L6,4 L6,5 L6,6

Figure 2.3: A two-level arrowhead matrix and its Cholesky factor, L

Algorithms 2.7 and 2.8. It turns out that it is also possible to exploit sparsity
within the off-diagonal blocks, but it will help to introduce more notation be-
fore tackling this issue. Note that our following discussions will assume that
an n × n Hermitian positive-definite matrix A has already been reordered in
some beneficial manner, typically via nested dissection.

2.3.1 Elimination forests

Definition 2.3.1 (supernode [11, 13, 91]). For any n × n Hermitian positive-
definite matrix A, if the n vertices are partitioned into m contiguous subsets,
say D = (D0 , D1 , . . . , Dm−1 ), such that j < k implies that every member of Dj
is less than every member of Dk , then each set Di is referred to as a (relaxed)
supernode.2

Remark 2.3.1. If possible, these supernodes should be chosen such that each
corresponding lower-triangular diagonal block of the Cholesky factor of A, say
L(Ds , Ds ), is sufficiently dense. We will frequently refer to supernodes by their
indices, e.g., by speaking of “supernode s ” rather than of “supernode Ds ”.
In all of the calculations performed within this dissertation, these supernodes

2
The usual graph-theoretic definition of a supernode is much more restrictive, as the
subset must be a clique whose edges satisfy a certain property. We must therefore make it
clear that our relaxed [11] definition is different.

25
correspond to the degrees of freedom of the separators produced during nested
dissection. Please glance ahead to Figures 2.5 and 2.6 for a visual interpreta-
tion of supernodes (and their associated elimination tree, which will soon be
introduced).

Definition 2.3.2 (original structure). The (lower) original structure of su-

pernode s, denoted by L̂s , is the set of all indices of rows of A which have
nonzero entries directly below the diagonal block A(Ds , Ds ). We will also de-
fine L̂ to be the tuple of the original structures, that is, L̂ = (L̂0 , L̂1 , . . . , L̂m−1 ).

Definition 2.3.3 (factored structure). If we use the notation D0:s = ∪0≤j≤s Dj ,

then the (lower) factored structure of supernode s can be inductively defined
as
[
Ls = L̂s ∪ Lj \ D0:s . (2.6)
0≤j<s
Lj ∩Ds 6=∅

Each contribution in the right term, say Lj \D0:s , is meant to model the effects
of an outer-product update

A(Lj , Lj ) := A(Lj , Lj ) − L(Lj , Dj )L(Lj , Dj )H

on the portion of A below supernode s, and clearly the columns with indices
Ds can only be effected when Lj intersects Ds .

We will also define L to be the tuple of the factored structures, i.e.,

L = (L0 , L1 , . . . , Lm−1 ).

Remark 2.3.2. When the supernodes each consist of a single vertex, this
definition yields the precise nonzero structure of the Cholesky factor of A,
assuming that no exact numerical cancellation took place. When the supern-
odes contain multiple vertices, this formula is meant to return a relatively tight
superset of the true nonzero structure.

26
Definition 2.3.4 (ancestors, parent, c.f. [112]). We may define the set of
ancestors of supernode s as all supernodes which have a nonzero intersection
with the factored structure of supernode s, that is, supernodes

A(s) = {j : Ls ∩ Dj 6= ∅}. (2.7)

When this set is non-empty, we may define the parent of supernode s as

parent(s) = min A(s). (2.8)

Definition 2.3.5 (descendants, children). The descendants of a supernode are

simply those which have a particular supernode as an ancestor, i.e.,

A−1 (s) = {j : s ∈ A(j)} = {j : Lj ∩ Ds 6= ∅}. (2.9)

We also say that the children of supernode s are the supernodes with Ds as
their parent, i.e.,

C(s) = parent−1 (s) = {c : parent(c) = s}. (2.10)

Definition 2.3.6 (elimination forest, c.f. [97, 112]). The structure we have
been describing is extremely important to multifrontal methods and was first
formalized in [112] in the context of supernodes composed of single vertices,
or, more simply, nodes. Simply connecting each supernode with its parent
supernode, when it exists, implies a graph, and parent(s) > s further implies
that this graph is actually a collection of trees. This collection of trees is
known as the elimination forest, which we will denote by E(A, D), and it can
be used to guide a multifrontal factorization.

Remark 2.3.3. The elimination forest, which becomes an elimination tree

when the underlying matrix is irreducible, will be at the heart of our further

27
discussions of the multifrontal method. We will begin by showing that a much
simpler formula for the factored structure can be constructed if we think in
terms of the elimination forest.

Example 2.3.1. A non-trivial elimination tree for a two-level arrowhead ma-

trix is shown in Figure 2.4.
 
0 ? ?
1 ? ?
 
 

A2,0 A2,1
 6
 2 ? 


 3 ? ? 


 4 ? ? 
 2 5
 
 A5,3 A5,4 5 ? 
A6,0 A6,1 A6,2 A6,3 A6,4 A6,5 6 0 1 3 4
(a) two-level arrowhead matrix (b) elimination tree

Figure 2.4: (a) A symmetric two-level arrowhead matrix and (b) its elimination
tree

Proposition 2.3.1 (optimal symbolic factorization [55]). The factored struc-

ture of a supernode s, Ls , can be directly formed from the original structure of
supernode s, L̂s , and the factored structure of its children. In particular,
[
Ls = L̂s ∪ Lc \ D s . (2.11)
c∈C(s)

This result immediately implies Algorithm 2.9.

Proof. By definition of the descendants of supernode s, we may rewrite Equa-

tion (2.3.1) as
[
Ls = L̂s ∪ Lj \ D0:s .
j∈A−1 (s)

28
Now consider recursively defining the related sets
[
Gs = L̂s ∪ Gj ,
j∈A−1 (s)

which satisfy the property that j ∈ A−1 (c) =⇒ Gj ⊂ Gc , and thus

[ [
Gj = Gc .
j∈A−1 (s) c∈C(s)

On the other hand, Ls = Gs \ D0:s , so we may write

[
Ls = L̂s ∪ Lc \ D0:s .
c∈C(s)

We now have only to show that, for each c ∈ C(s), Lc \ D0:s = Lc \ Ds . But
this is equivalent to the requirement that, for each j such that c < j < s,
Lc ∩ Dj = ∅, which follows from the fact that s = parent(c).

Algorithm 2.9: Optimal symbolic factorization

Input: supernodes, D, initial structure, L̂, and root node, s
Output: the factored structure, L
1 foreach c ∈ C(s) do Recurse(D,L̂,c)
S
2 Ls := L̂s ∪ c∈C(s) Lc \ Ds

2.3.2 Factorization

When we first introduced the notion of structure, we recognized that

an update A(Lj , Lj ) := A(Lj , Lj ) − L(Lj , Dj )L(Lj , Dj )H will only effect the
columns assigned to supernode s if Lj ∩ Ds 6= ∅, and we can now recognize
this as equivalent to supernode s being an ancestor of supernode j. Thus, a
supernode’s diagonal block will be ready for factorization as soon as each of

29
its descendants’ updates has been applied, and so a factorization algorithm
must work “up” the elimination forest. Since each elimination tree is indepen-
dent, from now on we will assume that all matrices are irreducible, with the
understanding that the following algorithms should be run on each tree in the
elimination forest.

Algorithm 2.10 is simply a blocked right-looking Cholesky factoriza-

tion (where the blocksize of the j’th step is the size of the j’th supernode)
which makes use of the elimination tree in order to expose different possible
execution orders (but unfortunately no trivial parallelism). Just as we reorga-
nized Algorithm 2.2 into Algorithm 2.3, we can also expose latent parallelism
in Algorithm 2.10 by storing temporary copies of update matrices. The key
difference is that, because a particular supernode’s portion of the matrix will
be effected by all of its descendants update matrices, it is beneficial to take
updates which came from grandchildren, great-grandchildren, etc., and to add
them into the update matrices of the direct descendants. This process is jus-
tified by Proposition 2.3.1 and demonstrated by Algorithm 2.11, which makes
use of the so-called extend-add operator [91], , to represent the process of
adding the entries of the update matrix Uc into the appropriate rows and
columns of a larger matrix. In particular, Uc is a square matrix meant to
update the (Lc , Lc ) submatrix of A, and Lc ⊂ Ds ∪ Ls implies that Uc can
be extended by zero into a matrix meant to update the (Ds ∪ Ls , Ds ∪ Ls )
submatrix of A. Thus, in this context, can be interpreted as extending its
right argument by zero and then performing standard matrix addition.

Algorithm 2.11 is actually the classical multifrontal Cholesky factoriza-

tion algorithm in disguise. If, instead of working directly with the matrix A,
we instead initialize a frontal tree, say F(A, D), which associates each node s

30
Algorithm 2.10: Cholesky factorization via an elimination tree
Input: HPD matrix, A, supernodes, D, and root node, s
Output: a lower triangular matrix, L, such that A = LLH
1 foreach c ∈ C(s) do Recurse(A,D,c)
2 L(Ds , Ds ) := Cholesky(A(Ds , Ds ))
−H
3 L(Ls , Ds ) := A(Ls , Ds )L(Ds , Ds )
H
4 A(Ls , Ls ) := A(Ls , Ls ) − L(Ls , Ds )L(Ls , Ds )

Algorithm 2.11: Verbose multifrontal Cholesky factorization

Input: HPD matrix, A, elimination tree, E, and root node, s
Output: a lower triangular matrix, L, such that A = LLH
1 foreach c ∈ C(s) do Recurse(A,E,c)
2 Us := zeros(|Ls |, |Ls |)
3 foreach c ∈ C(s) do
   
A(Ds , Ds ) A(Ds , Ls ) A(Ds , Ds ) A(Ds , Ls )
:= Uc
A(Ls , Ds ) Us A(Ls , Ds ) Us
4 L(Ds , Ds ) := Cholesky(A(Ds , Ds ))
−H
5 L(Ls , Ds ) := A(Ls , Ds )L(Ds , Ds )
H
6 Us := Us − L(Ls , Ds )L(Ls , Ds )

31
in the elimination tree with the original frontal matrix
 
A(Ds , Ds ) A(Ds , Ls )
F̂s = , (2.12)
A(Ls , Ds ) 0

then we can use the same language as Algorithm 2.4 to condense Algorithm 2.11
into Algorithm 2.12, which produces the factored frontal tree, F, with each
front Fs equal to  
L(Ds , Ds ) ?
Fs = , (2.13)
L(Ls , Ds ) ?
where the irrelevant quadrants have been marked with a ?.3

Algorithm 2.12: Multifrontal Cholesky factorization

Input: original frontal tree, F̂, and root node, s
Output: factored frontal tree, F
1 foreach c ∈ C(s) do Recurse(F̂,c)
2 Fs := F̂s
3 foreach c ∈ C(s) do Fs := Fs Uc
4 Process Fs

2.3.3 Triangular solves

We will now suppose that a multifrontal factorization of our HPD ma-

trix A has already been performed and that the factored frontal tree F is
available. Since each factored front Fs is simply a compact representation of
the nonzero entries within supernode s’s set of columns of the Cholesky factor
L, we will now introduce analogues of Algorithms 2.7 and 2.8 which apply to
arbitrary frontal trees. Just as with multifrontal factorization, the key idea is
to accumulate updates through several generations of an elimination tree so

3
In fact, a practical implementation will free the memory required for these quadrants
the last time they are used within the multifrontal factorization.

32
that each supernode must only interact with its children and its parent (when
it exists). We will see that multifrontal lower triangular solves (X := L−1 X)
work up the elimination tree making use of extend-add operations, but the
adjoint operation (X := L−H X) works down the elimination tree and requires
more care. We will begin by explaining the simple case and will then move on
to adjoint solves.

Consider performing the triangular solve X := L−1 X, where the lower

triangular matrix is the Cholesky factorization of a two-level arrowhead matrix
(see the right side of Figure 2.3). It is then natural to conformally partition
X to express the solve as the operation
   −1  
X0 L0,0 X0
X1   L1,1  X1 
 
X2 

L2,0 L2,1 L2,2
  
 X2 
X3  :=  L3,3 X3  .
     

X   L
 X 
 4  4,4   4
X5   L5,3 L5,4 L5,5  X5 
X6 L6,0 L6,1 L6,2 L6,3 L6,4 L6,5 L6,6 X6

It should be apparent that, for any block row s of L which is only nonzero at
its diagonal block, Ls,s , the solution for that block row may be found by simply
setting Xs := L−1
s,s Xs . Since this condition on block row s holds precisely when

supernode s has no descendants in the elimination tree (such a supernode is

called a leaf of the tree), we see that a multifrontal lower-triangular solve can
start working from all of the leaves of the elimination tree at once rather than
just the first set of degrees of freedom as in dense forward substitution.

Once the solution X` has been computed for each leaf ` in the elimina-
tion tree, each set of degrees of freedom can be eliminated via the update

X(L` , :) := X(L` , :) − L(L` , D` )X(D` , D` ),

33
which we see, by definition, only effects the portions of X assigned to the
ancestors of leaf `. As was mentioned earlier, our goal is to find a way to ac-
cumulate these updates up the elimination tree so that, for instance, when we
reach the root node, its updates from the entire rest of the tree have all been
merged into the update matrices of its children. Such an approach is demon-
strated by Algorithm 2.13, which is quite similar to the verbose description of
multifrontal factorization in Algorithm 2.11, which can likewise be condensed
using the language of fronts. In particular, if we introduce a right-hand side
tree, say Y, which assigns to supernode s the matrix
 
X(Ds , :)
Ys = ,
zeros(|Ls |, k)

where k is the number of columns of X, then we can express the solve using
the compact language of Algorithm 2.14. The term trapezoidal elimination [61]
simply refers to viewing the last two steps of Algorithm 2.13 as a partial
elimination process with the matrices
   
L(Ds , Ds ) ? X(Ds , :)
Fs = , and Ys = .
L(Ls , Ds ) ? Zs

Notice that the nonzero structure of the left half of Fs is trapezoidal: L(Ds , Ds )
is lower-triangular and L(Ls , Ds ) is, in general, dense.

Now consider the operation, X := L−H X, which, for a two-level arrow-

head matrix, can written as
−1 
LH LH LH
   
X0 0,0 2,0 6,0 X0
X1   LH H
1,1 L2,1 LH
6,1  X1 
LH LH
 
X2 
   
 2,2 6,2  X2 
X3  :=  LH LH LH X3  .
     
3,3 5,3 6,3 
X   LH LH LH
 X 
 4  4,4 5,4 6,4   4
X5   LH
5,5 LH
6,5
 X5 
X6 LH
6,6 X6

34
 Algorithm 2.13: Verbose multifrontal lower solve
Input: lower triangular matrix, L, right-hand side matrix of
width k, X, elimination tree, E, and root node, s
Output: X := L−1 X
1 foreach c ∈ C(s) do Recurse(L,X,E,c)
2 Zs := zeros(|Ls |, k)
   
X(Ds , :) X(Ds , :)
foreach c ∈ C(s) do := Zc
3 Zs Zs
−1
4 X(Ds , :) := L(Ds , Ds ) X(Ds , :)
5 Zs := Zs − L(Ls , Ds )X(Ds , :)

Algorithm 2.14: Multifrontal lower solve

Input: frontal tree, F, right-hand side tree, Y, and root node, s
1 foreach c ∈ C(s) do Recurse(F,Y,c)
2 foreach c ∈ C(s) do Ys := Ys Zc
3 Trapezoidal forward elimination of Ys with Fs

This time, the only part of the solution which we can immediately compute
corresponds to the root node: Xs := L−H
s,s Xs (in this case, s = 6). These

degrees of freedom can then be eliminated from each of the descendants d ∈

A−1 (s) via the update Xd := Xd − LH
s,d Xs , and we can recognize that the

sparsity in L implies that the update may be simplified to

X(Dd , :) := X(Dd , :) − L(Ld ∩ Ds , Dd )H X(Ld ∩ Ds , :).

One possible approach would be to update all of the descendants in this manner
and then to recurse on each of the children of the root node.

On the other hand, suppose that we were interest interested in simulta-

neously applying all of the updates to a particular supernode from the entire
set of its ancestors, i.e.,
X
X(Ds , :) := X(Ds , :) − L(Ls ∩ Da , Ds )H X(Ls ∩ Da , :).
a∈A(s)

35
Since the set of ancestors of node s is precisely equal to the set of supernodes
which intersect its lower structure, this update simplifies to

X(Ds , :) := X(Ds , :) − L(Ls , Ds )H X(Ls , :),

which implies the top-down approach of Algorithm 2.15. The downside of

this approach is that there is no clear notion of data locality: each supernode
continually indexes out of the global right-hand side matrix, X.

Algorithm 2.15: Verbose multifrontal lower adjoint solve

Input: lower triangular matrix, L, right-hand sides, X,
elimination tree, E, and root node, s
Output: X := L−H X
H
1 if A(s) 6= ∅ then X(Ds , :) := X(Ds , :) − L(Ls , Ds ) X(Ls , :)
−H
2 X(Ds , :) := L(Ds , Ds ) X(Ds , :)
3 foreach c ∈ C(s) do Recurse(L,X,E,c)

On the other hand, if the parent of supernode s is supernode p, then

Ls ⊂ Dp ∪ Lp , and so if each supernode s pulls X(Ls , :) down from its parent
and then performs all necessary computation on X(Ds , :), it will have locally
computed all of the pieces of X needed for its children to repeat the process.
A recursive procedure based on this idea is given in Algorithm 2.16, where
initially  
X(Ds , :)
Ys = ,
zeros(|Ls |, k)
and Extract(Ys , Yp ) refers to the process of extracting X(Ls , :) out of Yp =
X(Dp ∪ Lp , :) and placing it into the bottom section of Ys .

2.3.4 Computational complexity

We have now derived the main algorithms required for solving sparse
Hermitian positive-definite linear systems with the multifrontal algorithm (with

36
 Algorithm 2.16: Multifrontal lower adjoint solve
Input: frontal tree, F, right-hand side tree, Y, and root node, s
1 if A(s) 6= ∅ then Extract(Ys , Yparent(s) )
H
2 Trapezoidal backward elimination of Ys with Fs
3 foreach c ∈ C(s) do Recurse(F,Y,c)

Figure 2.5: A 15 × 15 grid graph. Each node is connected to its nearest

horizontal and vertical neighbors.

the notable exception of graph partitioning approaches, which we will discuss

at the end of the chapter) and are ready to begin discussing storage and work
requirements. We will begin by mirroring George’s argument [54] that, if the
graph of a Hermitian positive-definite matrix is an n × n grid graph [90] (see
Figure 2.5), then nested dissection may be combined with the multifrontal al-
gorithm in order to factor the sparse matrix with only O(n3 ) work (as opposed
to the O(n6 ) work required for a dense factorization). This sort of sparsity
pattern frequently arises for two-dimensional finite-difference equations, and
an example of an elimination tree for a 15 × 15 grid is shown in Figure 2.6.

While the separators chosen in Figure 2.6 are what we would choose
in practice, in order to keep our analysis simple we will choose so-called cross
separators [61], which are approximately twice as large as necessary but do
not change the asymptotics of the method. Figure 2.7 shows an elimination

37
Figure 2.6: A separator-based elimination tree (right) for a 15 × 15 grid graph
(left)

tree for a 15 × 15 grid graph using cross separators. In particular, we will be

considering n × n grids where n is one less than a power of two, which result
in the cross separators on level ` of the tree (counting down from zero from
the root node) having heights and widths of

n+1
w` = − 1.
2`

We thus know that, if s ∈ level(`) (that is, supernode s is on the `’th level of
the elimination tree), then |Ds | = 2wl − 1. We can also easily bound |Ls | from
above by 4w` (the maximum number of nodes that can border the w` × w` box
spanned by the cross separator and its descendants).

We can easily see that the work required for processing front Fs is
approximately equal to

1
Work(Fs ) = |Ds |3 + |Ds |2 |Ls | + |Ds ||Ls |2 , (2.14)
3

where the first term corresponds to the Cholesky factorization of the |Ds | ×
|Ds | diagonal block, the second term corresponds to a triangular solve of the
diagonal block against the |Ls | × |Ds | submatrix A(Ls , Ds ), and the third term

38
Figure 2.7: An elimination tree based on cross separators (right) of a 15 × 15
grid graph (left)

corresponds to a rank-|Ds | Hermitian update of an |Ls | × |Ls | matrix. We can

now plug the previously discussed level-dependent values and bounds for |Ds |
and |Ls | into a summation over all of the fronts to find the total work for the
multifrontal factorization (ignoring the lower-order extend-add costs):
blog2 (n)c  
X
2` 1
Work(F) ≤ 2 (2w` )3 + (2w` )2 (4w` ) + (2w` )(4w` )2
`=0
3
blog2 (n)c
X
≤C 22` w`3
`=0
blog2 (n)c
X
≤ C(n + 1) 3
2−` ∈ O(n3 ).
`=0

We have just shown that nested dissection of regular 2d grids leads only re-
quires O(n3 ) = O(N 3/2 ) work, where N = n2 is the total number of degrees of
freedom. Soon after George showed this result [54], Lipton and Tarjan gener-
alized the approach to the entire class of planar graphs [90], which is simply
the class of graphs which may be drawn on paper without any overlapping
edges (that is, embedded in the plane).

Before we discuss more abstract classes of graphs, we will first extend

39
our analysis to d-dimensional grid graphs with n vertices in each direction, for
a total of N = nd vertices, which only interact with their nearest neighbors in
each of the d cardinal directions. This time, the cross separators will consist of
d overlapping hyperplanes of dimension d − 1 (i.e., in 3D, the cross separators
consist of three planes), and we can bound |Ds | from above by dw`d−1 , which
is exact other than ignoring the overlap of the d hyperplanes. Also, a d-
dimensional cube has 2d faces, and so our best bound for |Ls | is 2dw`d−1 . The
resulting bound for d-dimensional nested dissection is
blog2 (n)c  
X
d` 1 d−1 3 d−1 3 d−1 3
Work(F) ≤ 2 (dw` ) + 2(dw` ) + 4(dw` )
`=0
3
blog2 (n)c
X 3(d−1)
=C 2d` w`
`=0
blog2 (n)c
X
3(d−1)
≤ C(n + 1) 2`(3−2d) .
`=0

When we set d = 1, we find that the cost is O(n), as it should be for a

tridiagonal matrix, when we set d = 2, we recover our previous result, and
when we set d = 3 we find that the cost is O(n6 ) = O(N 2 ). We can also see
that, for d ≥ 2, the cost is O(n3(d−1) ), which is the same asymptotic complexity
as a dense factorization over a root separator of size nd−1 × nd−1 .

Now let us recall that, after factorization, all that must be kept from
each frontal matrix Fs is its top-left and bottom-left quadrants, i.e., L(Ds , Ds )
and L(Ls , Ds ), which respectively require 21 |Ds |2 and |Ds ||Ls | units of storage.
We therefore write
1
Mem(Fs ) = |Ds |2 + |Ds ||Ls |. (2.15)
2
If we make use of the same bounds on |Ds | and |Ls | as before, then we find that,
for a d-dimensional grid graph, the memory required for storing the Cholesky

40
 d work storage
1 O(N ) O(N )
3/2
2 O(N ) O(N log N )
2
3 O(N ) O(N 4/3 )
3(1−1/d)
d ≥ 3 O(N ) O(N 2(1−1/d) )

Table 2.1: Storage and work upper bounds for the multifrontal method applied
to d-dimensional grid graphs with n degrees of freedom in each direction (i.e.,
N = nd total degrees of freedom)

factorization is
blog2 (n)c  
X 1
Mem(F) ≤ d`
2 (dw`d−1 )2 + 2(dw`d−1 )2
`=0
2
blog2 (n)c
X
2(d−1)
= C(n + 1) 2`(2−d) .
`=0

When we set d = 1, Mem(F) is O(n), which agrees with the fact that the
Cholesky factor should be bidiagonal, when d = 2, we see that the memory
requirement is O(n2 log n) = O(N log N ), for d = 3 it is O(n4 ) = O(N 4/3 ),
and, for arbitrary d ≥ 3, it is O(n2(d−1) ), which is equal to the amount of
space required for the storage for a nd−1 × nd−1 dense matrix. Because our
multifrontal triangular solve algorithms perform O(1) flops per unit of storage,
the work required for the triangular solves has the same asymptotic complexity
as the memory requirements of the factorization.

It should now be overwhelmingly clear that, for nested-dissection based

multifrontal methods, the deciding factor for the cost of the multifrontal method
is the minimum separator size required for bisecting the underlying graph. For
one-dimensional grid graphs, these separators consist of a single vertex, for
two-dimensional grid graphs, a separator can be chosen as a line of n vertices,
for three dimensions the separator can be chosen as a plane of n2 vertices,

41
and, in d dimensions, the separator can be chosen as a hyperplane of nd−1
vertices. Interestingly, it was shown in [57] that, if a graph can be embedded
in a surface of genus g (for example, the surface of a sphere with g handles
√ √
attached), then a separator of size O( gN + N ) may be found. This implies
that d-dimensional grid graphs have separators comparable to those of graphs
d−2
of genus N d .

2.3.5 Graph partitioning

As was previously mentioned, graph partitioning is an NP-complete

problem [53], and so practical solution techniques invariably make use of
heuristics. The most successful approach to date is multilevel graph parti-
tioning [69, 70], which resembles multigrid methods [25, 26] in that the basic
workflow is to successively coarsen the original problem, directly solve the
problem once it is sufficiently coarse, and then push the solution back up the
hierarchy.

In the case of graphs, this coarsening process merges sets of vertices

into a single vertex whose edge list is the union of the edge lists of its con-
stituents; one might say that each step of the coarsening process constructs a
new graph out of a set of supernodes for the previous graph. After the graph
is sufficiently coarsened, a brute-force graph partitioning algorithm can be ap-
plied, and then, by construction, this partition implies a valid (albeit likely
suboptimal) partition for the parent graph. It is then natural to attempt to
refine the candidate partition at each step via an algorithm like Kernighan-Lin
refinement [83], though practical implementations should make use of variants
of Fiduccia and Mattheyses’s improvement of KL-refinement [48], e.g., the one
used by METIS [81]. How to effectively parallelize this refinement process is

42
still an open question.

2.4 Parallel multifrontal methods

It is now time to discuss the parallelization of each of the major steps
of the multifrontal solution of a sparse Hermitian positive-definite system of
equations:

1. reordering the unknowns (especially via nested dissection),

2. symbolic factorization,

3. numeric factorization, and

4. triangular solves.

We will begin with the easiest step, symbolic factorization, move on to nu-
meric factorization, spend a while discussing several options for the triangular
solves, and then briefly touch on remaining challenges with respect to parallel
reorderings.

Our approach to the parallelization of each of these stages (with the pos-
sible exception of the reordering phase) exploits the independence of siblings
within the elimination tree in order to expose as much trivial parallelism as
possible, and fine-grain parallelism is therefore only employed towards the top
of the tree, usually via distributed-memory dense linear algebra algorithms.
The fundamentals of this approach are due to George, Liu, and Ng, who
proposed a so-called subtree-to-subcube mapping [56] which splits the set of
processes into two disjoint teams at each junction of a binary elimination
tree. The term subcube was used because this decomposition maps naturally

43
Figure 2.8: An example of a (slightly imbalanced) elimination tree distributed
over four processes: the entire set of processes shares the root node, teams of
two processes share each of the two children, and each process is assigned an
entire subtree rooted at the dashed line.

∗∗∗

0∗∗ 1∗∗

00∗ 01∗ 10∗ 11∗

000 001 010 011 100 101 110 111

000 000 001 001 010 010 011 011 100 100 101 101 110 110 111 111

Figure 2.9: Overlay of the process ranks (in binary) of the owning subteams
of each supernode from an elimination tree assigned to eight processes; a ‘*’ is
used to denote both 0 and 1, so that ‘00∗’ represents processes 0 and 1, ‘01∗’
represents processes 2 and 3, and ‘∗ ∗ ∗’ represents all eight processes.

to hypercube network topologies, but the idea trivially generalizes to general

elimination forests and non-power-of-two numbers of processes, and so we will
speak of subtree-to-subteam mappings. Figures 2.8 and 2.9 both demonstrate
subtree-to-subteam mappings of an elimination tree. For the sake of simplic-
ity, we will assume that the elimination tree is binary above the roots of the
local subtrees.

44
2.4.1 Symbolic factorization

There is a natural way to combine subtree-to-subteam mappings with

the symbolic factorization approach in Algorithm 2.9: every process begins
the algorithm at the root of the tree, and, each time a set of recursions was
to be launched over the children of that node, the set of processes is split
into (roughly) even teams which each handle the subtree rooted at one of the
children. For the sake of simplicity, we require that the distributed portion
of the tree is binary. Thus, if p processes are available, b p2 c processes can
recurse on the first child while the remaining processes handle the second
child, and, after both recursions finish, the two teams can share the factored
structure of the two children. Any time the recursive routine is launched with
a team consisting of a single process, we can simply revert to the sequential
factorization algorithm. This approach is summarized in Algorithm 2.17, and
an example of the part of the elimination tree touched by a single process is
shown in Figure 2.10. From now on, when a subscript of loc is added to a
tree data structure, e.g., Lloc , the implication is that it is the restriction of the
data structure to a particular process’s portion of the tree.

2.4.2 Numeric factorization

The numeric factorization can be parallelized in a manner very similar

to the symbolic factorization; the main difference is that, rather than each
process performing a few local set operations, teams must work together to
perform part of a distributed dense Cholesky factorization (this corresponds
to processing the supernode, recall Algorithm 2.12). As was noted in [113]
and, for the dense case, [38], two-dimensional (or higher) decompositions of
the fronts are required in order to perform their partial factorizations in a scal-

45
 Algorithm 2.17: Distributed symbolic factorization
Input: local original structure, L̂loc , local elimination tree, Eloc ,
root node, s, number of processes, p, and process rank, r
Output: local factored structure, Lloc
1 if p > 1 then
2 if r < b p2 c then
3 Recurse(L̂loc , Eloc , c0 (s),b p2 c,r)
4 Exchange Lc0 (s) for Lc1 (s) with other team
5 else
6 Recurse(L̂loc , Eloc , c1 (s),p−b p2 c, r−b p2 c)
7 Exchange Lc1 (s) for Lc0 (s) with other team
8 else
9 foreach c ∈ C(s) do Recurse(L̂loc , Eloc ,c,1,0)
S
10 Ls := L̂s ∪ c∈C(s) Lc \ Ds

Figure 2.10: An example of the portion of a symbolic factorization to be com-

puted by a single process (enclosed in the dashed blue line) for an elimination
tree distributed over eight processes. Each step towards the top of the tree
requires an interaction with the team which computed the factored structure
of the other child (each of these interactions is surrounded with dashed red
lines).

46
able manner. And, in the words of Schreiber, a scalable dense factorization of
the root front is a sine qua non for the overall scalability of the multifrontal
method. As was discussed in the previous chapter, when memory is not con-
strained, it is worthwhile to consider so-called 2.5D [116] and 3D [9, 76] dense
factorizations, which replicate the distributed matrix across several teams of
processes in order to lower the communication volume.

Algorithm 2.18: Distributed multifrontal factorization

2D
Input: local part of 2D-distributed original frontal tree, F̂loc ,
root node, s, number of processes, p, and process rank, r
2D
Output: local part of 2D-distributed factored frontal tree, Floc
1 if p > 1 then
2 if r < b p2 c then Recurse(F̂loc2D
, c0 (s),b p2 c,r)
3 else Recurse(F̂loc2D
, c1 (s),p−b p2 c, r−b p2 c)
4 Fs := F̂s
5 foreach j ∈ {0, 1} do AllToAll to add Ucj into Fs
6 Partial factorization of Fs with 2D (or higher) distribution
7 else
2D
8 Apply Algorithm 2.12 to F̂loc beginning at node s

Although this chapter is focused on multifrontal approaches to solving

sparse linear systems, it is important to briefly survey the wide variety of exist-
ing distributed-memory sparse-direct solvers [33]. The solvers PSPASES [80]
and WSMP [62] are both descendants of the approach in [63], which is also
the foundation for the approach followed in this dissertation. Another related
solver is DSCPACK [103], which was the testing ground for selective inver-
sion [102], an approach which we will devote one of the following subsections
to. DSCPACK follows the same basic framework as PSPASES and WSMP,
though it unfortunately does not make use of level 3 BLAS for processing its
distributed fronts [103].

47
 The packages SPOOLES [12] and MUMPS [3] both implement distributed-
memory multifrontal methods, but the former makes use of one-dimensional
distributions for all of its fronts in order to simplify partial pivoting, while
the latter only makes use of two-dimensional distributions for the root front.
Thus, neither approach can be expected to exhibit comparable scalability to
the approach of [63]. In addition, MUMPS was originally designed to make
use of a master-slave paradigm, and a number of artifacts still remain that ob-
struct its usage for truly large-scale problems (for example, MUMPS requires
that right-hand sides be stored entirely on the root process at the beginning
of the solution process [1]).

The remaining two commonly used packages are PaStiX [72] and
SuperLU_Dist [87], both of which use alternatives to the multifrontal method.
PaStiX couples a column fan-in, or left-looking supernodal approach [10] with
an intelligent selection criteria for choosing between one-dimensional and two-
dimensional dense distributions, though it appears to only be competitive
with the approach of [63] for modest numbers of processors [72]. On the
other hand, SuperLU_Dist focuses on structurally non-symmetric matrices
and uses a row-pipelined approach [88] which resembles a sparse version of
the High Performance Linpack algorithm [95, 126]. One potential pitfall to
this approach is that the size of supernodes must be artificially restricted in
order to preserve the efficiency of the parallelization: for instance, if the root
separator arising from nested dissection of a 3D grid graph was treated as a
single supernode, then the runtime and memory usage of the parallel scheme
√ √
would at best be O(N 2 / p) and O(N 4/3 / p), respectively.

As was mentioned at the beginning of the chapter, the difference be-

tween the factorization approach used later in this dissertation versus that of

48
PSPASES and WSMP is which two-dimensional distribution is chosen for the
fronts. While PSPASES and WSMP make use of block-cyclic distributions,
Clique, the implementation which will be used within our parallel sweeping
preconditioner, makes use of element-wise distributions [71, 93, 99] and does
not alter the details of the data distribution so that parallel extend-add op-
erations only require communication with a single process [63]. The major
advantage Clique is its flexibility with respect to triangular solve algorithms
(though a custom Kronecker-product compression scheme is also employed in
a later chapter).

2.4.3 Standard triangular solves

We will now assume that a distributed factorization has already been

performed and that O(1) right-hand sides are ready to be solved in the form
of the matrix X, which we will arrange into a right-hand side tree, Y. Fur-
thermore, we will also apply the subtree-to-subteam mapping to Y in order to
determine which teams of processes share each matrix in the tree. However,
we must still decide how each matrix should be shared between its team of
processes. Let us recall that each member of Y has the form
 
X(Ds , :)
Ys = ,
zeros(|Ls |, k)

when X has k columns.

Consider the multifrontal triangular solve described by Algorithm 2.13.

It should be clear that essentially all of the work of the multifrontal triangular
solve lies within Steps 6 and 7, which respectively perform dense triangular
solves and a dense matrix-matrix multiplication. For now, we will consider
the regime where only a small number of right-hand sides need to be solved,

49
and so Steps 6 and 7 should temporarily be interpreted as a dense triangular
solve with a single right-hand side and a dense matrix-vector multiply. Thus,
the distribution of each member of the right-hand side tree Y can be chosen
to maximize the efficiency of the dense triangular solves and matrix-vector
multiplication.

Unfortunately, the critical path of forward/backward substitution is

O(n) for an n × n dense triangular matrix, and so the O(n2 ) work can only
be effectively distributed amongst p processes, i.e., have a runtime of O(n2 /p),
when p ≤ n. Thus, the per-process memory requirements for an efficient
triangular solve increase proportionally with the size of the dense triangular
matrix, n. This observation is consistent with the isoefficiency analysis of
[65] and [79], which respectively analyzed pipelined distributed multifrontal
triangular solves with 1D and 2D distributions of the frontal matrices (cf. [68]).
For both distribution schemes, their conclusion was that, in order to maintain
constant efficiency in the multifrontal triangular solves arising from 2D and
3D grid graphs, the number of processes, p, can respectively only grow as
fast as Ω(N 1/2 ) and Ω(N 2/3 ) (where the former term holds within logarithmic
factors of N ). Thus, the O(N log N/p) and O(N 4/3 /p) per-process memory
requirements must respectively grow at rates of Ω(N 1/2 ) and Ω(N 2/3 ) in order
to ensure that the triangular solves remain efficient (the former term again
only holds within logarithmic factors of N ).

Under most circumstances, this scalability issue can be ignored, as a

multifrontal triangular solve requires asymptotically less work than a multi-
frontal factorization (e.g., O(N log N ) vs. O(N 3/2 ) in 2D and O(N 4/3 ) vs. O(N 2 )
in 3D). Thus, as long as efficiency of the parallel triangular solve does not drop
too quickly (e.g., at a rate faster than O(N 1/2 / log N ) in 2D and O(N 2/3 ) in

50
3D), the factorization can be expected to remain the dominant term. Exam-
ples of parallel multifrontal forward and backward eliminations based upon
element-wise 2D frontal distributions and 1D right-hand side distributions are
therefore given in Algorithms 2.19 and 2.20, though it is also reasonable to
redistribute the fronts into one-dimensional distributions and to make use of
multifrontal triangular solves based upon Algorithms C.7 and C.15.4 One-
dimensional distributions should be used for each right-hand side submatrix
Ys because, by assumption, there are only O(1) right-hand sides, and so one-
dimensional distributions must be used if the right-hand sides are to be dis-
tributed amongst O(p) processes.

Algorithm 2.19: Distributed multifrontal forward substitution

2D
Input: local part of 2D-distributed factored frontal tree, Floc ,
1D
local part of 1D-distributed right-hand side tree, Yloc ,
root node, s, number of processes, p, and process rank, r
1 if p > 1 then
2 if r < b p2 c then Recurse(Floc 2D 1D
, Yloc ,c0 (s), b p2 c,r)
3 else Recurse(Floc 2D 1D
, Yloc p
,c1 (s), p−b 2 c, r−b p2 c)
4 foreach j ∈ {0, 1} do AllToAll to add Zcj into Ys
5 X(Ds , :) := X(Ds , :)L(Ds , Ds )−1 via Algorithm C.8
6 Zs := Zs − L(Ls , Ds )X(Ds , :) via Algorithm C.1
7 else
2D 1D
8 Apply Algorithm 2.14 to Floc and Yloc beginning at node s

Note that, when more than O(1) triangular solves must be performed
with each factorization, efficient parallelizations are important. There are
essentially two important situations:

4
We will not discuss pipelined dense triangular solves since they are less practical with
element-wise distributions. See [79] for a pipelined two-dimensional dense triangular solve.

51
 Algorithm 2.20: Distributed multifrontal backward substitution
2D 1D
Input: Floc , Yloc , s, p, r
1 if p > 1 then
2 if A(s) 6= ∅ then AllToAll to update Ys with Yparent(s)
3 X(Ds , :) := X(Ds , :) − L(Ls , Ds )H Zs via Algorithm C.2
4 X(Ds , :) := X(Ds , :)L(Ds , Ds )−H via Algorithm C.12
5 if r < b p2 c then Recurse(Floc 2D 1D
, Yloc ,c0 (s), b p2 c,r)
6 else Recurse(Floc 2D 1D
, Yloc ,c1 (s), p−b p2 c, r−b p2 c)
7 else
8 if A(s) 6= ∅ then AllToAll to update Ys with Yparent(s)
9 X(Ds , :) := X(Ds , :) − L(Ls , Ds )H Zs
10 X(Ds , :) := X(Ds , :)L(Ds , Ds )−H
2D 1D
11 foreach c ∈ C(s) do Recurse(Floc , Yloc , c,1,0)

1. many right-hand sides must be solved one at a time, and

2. many right-hand sides can be solved at once.

The following subsection addresses the first case, which is important for iter-
ative methods which make use of sparse-direct triangular solves (usually on
subproblems) as part of a preconditioner.

2.4.4 Selective inversion

An examination of the triangular solve algorithms from the previous

subsection, Algorithms 2.19 and 2.20, reveals that all of the scalability prob-
lems result from the dense triangular solves against the diagonal blocks of the
frontal matrices in Steps 8 and 5, respectively. The idea of selective inver-
sion [102, 103] is to perform extra work between the multifrontal factorization
and solution phases in order to invert the lower-triangular diagonal blocks
in place, so that the problematic distributed dense triangular solves become
distributed dense matrix-vector multiplication (see Algorithms 2.21 and 2.22).

52
Significantly less work is required for this inversion process than the origi-
nal multifrontal factorization and it has been observed that, for moderate to
large numbers of processes, the extra time spent within selective inversion is
recouped after a small number of triangular solves are performed [98, 102].
Large performance improvements from making use of selective inversion are
shown in Chapter 3. It is also important to note that a similar idea appeared
in earlier work on out-of-core dense solvers [114], where the diagonal blocks of
dense triangular matrices were inverted in place in order to speed up subse-
quent triangular solves.

Algorithm 2.21: Selective inversion

2D
Input: Floc , s, p, r
Output: local part of 2D-distributed selectively-inverted frontal
2D
tree, Gloc
1 Gs := Fs
2 if p > 1 then
3 Invert L(Ds , Ds ) in top-left corner of Gs via Algorithm C.21
4 if r < b p2 c then Recurse(Floc 2D
, c0 (s),b p2 c,r)
5 else Recurse(Floc 2D
, c1 (s),p−b 2 c, r−b p2 c)
p

6 else
7 foreach d ∈ A−1 (s) do Gd := Fd

The numerical stability of our replacement of triangular solves with

an explicit calculation of the inverse followed by matrix-vector multiplication
warrants at least a short discussion. Although avoiding direct inversion is
perhaps the first lesson taught to budding numerical analysts, results dating
back to Wilkinson [40, 132] show that, under reasonable assumptions, and with
a careful calculation of the inverse, x := inv(A) x can be as accurate as with a
backwards-stable solver. Although these results hold for essentially arbitrary
square matrices, we need only apply them to lower triangular matrices in

53
 Algorithm 2.22: Distributed multifrontal forward substitution af-
ter selective-inversion
2D 1D
Input: Gloc , Yloc , s, p, r
1 if p > 1 then
2 if r < b p2 c then Recurse(Gloc 2D 1D
, Yloc ,c0 (s), b p2 c,r)
3 else Recurse(Gloc 2D 1D
, Yloc ,c1 (s), p−b p2 c, r−b p2 c)
4 foreach j ∈ {0, 1} do AllToAll to add Zcj into Ys
5 X(Ds , :) := X(Ds , :) inv(L(Ds , Ds )) via Algorithm C.1
6 Zs := Zs − L(Ls , Ds )X(Ds , :) via Algorithm C.1
7 else
2D 1D
8 Apply Algorithm 2.14 to Gloc and Yloc beginning at node s

order to exploit selective inversion with confidence. A detailed analysis of the

requirements for accurately forming both inv(L) b and inv(L)H b will be the
subject of future work.

2.4.5 Solving with many right-hand sides

When many right-hand sides are available at once, which is the case in
many seismic inversion [121] algorithms, there are many more paths to scal-
able multifrontal triangular solves. In particular, let us recall that the dense
operation TRSM [37], which performs triangular solves with multiple right-hand
sides, can be made scalable when there are sufficiently many right-hand sides
due to the independence of each solve. This is in stark contrast to triangular
solves with O(1) right-hand sides, which are difficult (if not impossible) to
efficiently parallelize, and so it is natural to consider building a multifrontal
triangular solve on top of a distributed dense TRSM algorithm, such as Algo-
rithm C.9.

As in the single right-hand side case, the distribution of each submatrix

of the right-hand side tree, Y, should be chosen in order to maximize the

54
efficiency of the most expensive operations which will be performed with it.
In the case of many right-hand sides, which we will loosely define to be at
√
least O( p) right-hand sides when p processes are to be used, Steps 8-9 and
5-6 from Algorithms 2.19 and 2.20 should each be thought of as a pair of
distributed TRSM and GEMM calls. If each matrix Ys in the right-hand side tree
is stored in a two-dimensional data distribution, then we may directly exploit
Algorithms C.9 and C.3 in order to build a high-performance multifrontal
triangular solve with many right-hand sides (and the adjoint dense kernels
may be used to construct an efficient adjoint solve). The resulting forward
elimination approach is listed in Algorithm 2.23.

Algorithm 2.23: Distributed multifrontal forward substitution

with many right-hand sides
2D 2D
Input: Floc , Yloc , s, p, r
1 if p > 1 then
2 if r < b p2 c then Recurse(Floc 2D 2D
, Yloc ,c0 (s), b p2 c,r)
3 else Recurse(Floc 2D 2D
, Yloc ,c1 (s), p−b p2 c, r−b p2 c)
4 foreach j ∈ {0, 1} do AllToAll to add Zcj into Ys
5 X(Ds , :) := X(Ds , :)L(Ds , Ds )−1 via Algorithm C.9
6 Zs := Zs − L(Ls , Ds )X(Ds , :) via Algorithm C.3
7 else
2D 2D
8 Apply Algorithm 2.14 to Floc and Yloc beginning at node s

2.4.6 Graph partitioning

The elephant in the room for parallel sparse direct solvers is that no
scalable implementations of effective heuristics for graph partitioning yet exist.
Although ParMETIS [82] is extremely widely used and useful for partitioning
graphs which cannot be stored in memory on a single node, there are still
difficulties in the parallelization of partition refinements, and, in practice [89],

55
 Algorithm 2.24: Distributed multifrontal forward substitution
with many right-hand sides after selective-inversion
2D 2D
Input: Gloc , Yloc , s, p, r
1 if p > 1 then
2 if r < b p2 c then Recurse(Gloc 2D 2D
, Yloc ,c0 (s), b p2 c,r)
3 else Recurse(Gloc 2D 2D
, Yloc ,c1 (s), p−b p2 c, r−b p2 c)
4 foreach j ∈ {0, 1} do AllToAll to add Zcj into Ys
5 X(Ds , :) := X(Ds , :) inv(L(Ds , Ds )) via Algorithm C.3
6 Zs := Zs − L(Ls , Ds )X(Ds , :) via Algorithm C.3
7 else
2D 2D
8 Apply Algorithm 2.14 to Gloc and Yloc beginning at node s

distributed sparse-direct solvers often default to the sequential algorithms of

METIS. It is therefore beneficial for large-scale applications to analytically
perform nested dissection if their discretization is structured enough to allow it.
This is the approach taken for the parallel sweeping preconditioner discussed
in later chapters.

2.5 Summary
An introduction to nested-dissection based multifrontal Cholesky fac-
torization and triangular solves was provided in the context of a reordered
dense right-looking Cholesky factorization. Parallelizations of these opera-
tions were then discussed, Raghavan’s selective inversion [102] was reviewed,
and then two new parallel multifrontal triangular solve algorithms targeted
towards simultaneously solving many right-hand sides were introduced. The
author expects that these algorithms will be a significant contribution to-
wards frequency-domain seismic inversion procedures, which typically require
the solution of roughly O(N 2/3 ) linear systems. In the following chapter, an
algorithm will be described which replaces a full 3D multifrontal factorization

56
with a preconditioner which only requires O(N 4/3 ) work to set up, yet only
requires O(N log N ) work to solve usual seismic problems. It is easy to see
that, if O(N 2/3 ) linear systems are to be solved, asymptotically more work is
performed solving the linear systems and setting up the preconditioner.

57
 Chapter 3

Sweeping preconditioners

The fundamental idea of both this and the following chapter is to or-
ganize our problem in such a manner that we may assign physical meanings
to the temporary matrices generated during a factorization process, and then
to exploit our knowledge of the physics in order to efficiently construct (or,
in the case of the next chapter, compress) an approximation to these tempo-
rary products. In particular, we will order our degrees of freedom so that the
Schur complements generated during a block tridiagonal LDLT factorization
of a discretized boundary value problem may be approximately generated by
the factorization of a boundary value problem posed over a much smaller sub-
domain. More specifically, if the original problem is three-dimensional, multi-
frontal factorizations may be efficiently employed on quasi-2D subdomains in
order to construct an effective preconditioner.

3.1 Radiation boundary conditions

It can easily be seen that, if we had instead transformed the acous-
tic wave equation, Equation (1.2), into time-harmonic form using a time-
dependence of eiωt instead of e−iωt , then we would arrive at the exact same
Helmholtz equation. But this implies that the transformation t 7→ −t, which is
simply time reversal, does not have an effect on the resulting Helmholtz equa-

58
tion. It is therefore the responsibility of the boundary conditions1 to break
this symmetry so that we may describe cause and effect in a sensical manner.
For instance, if we create a disturbance at some point in the domain, the de-
sired response is for a wave front to radiate outwards from this point as time
progresses, rather than coming in “from infinity” at the beginning of time in
anticipation of our actions.

Sommerfeld introduced a radiation condition [111] for precisely this

reason; in three dimensions, it is the requirement that the solution u(x) satisfies
 
∂u
lim |x| − iωu = 0, (3.1)
|x|→∞ ∂|x|

which can be seen to hold for the radiating 3D Helmholtz Green’s function,

eiω|x−y|
Gr (x, y) = , (3.2)
4π|x − y|

but not the absorbing Green’s function,

e−iω|x−y|
Ga (x, y) = Gr (x, y) = . (3.3)
4π|x − y|

It is instructive to consider the evolution of these functions in their time-

harmonic forms when the point source is at the origin, for example, an anima-
tion of
eiω(|x|−t)
Gr (x, 0)e−iωt =
4π|x|
shows that waves are propagating outwards from the origin, while Ga (x, 0)e−iωt
can be seen to be absorbing waves into the origin.2

1
some readers will perhaps insist that we speak of remote conditions instead of applying
boundary conditions “at infinity”
2
And thus, if a time-dependence of eiωt is desired, the radiation condition should be
conjugated and the definitions of Ga and Gr should be reversed (i.e., conjugated).

59
 Essentially the same argument holds for 3D time-harmonic Maxwell’s
equations, which are typically equipped with the Silver-Müller radiation con-
ditions [124],

lim (H × x − |x|E) = 0, and (3.4)

|x|→∞

lim (E × x + |x|H) = 0. (3.5)

|x|→∞

Likewise, the 3D time-harmonic linear elastic equations make use of the Kupradze-
Sommerfeld radiation conditions [24, 123],
 
∂us
lim |x| − iκs us = 0, and (3.6)
|x|→∞ ∂|x|
 
∂up
lim |x| − iκp up = 0, (3.7)
|x|→∞ ∂|x|
where us and up respectively refer to the solenoidal (divergence-free) and ir-
rotational (curl-free) components of the solution, u, and κs and κp refer to the
wave numbers associated with shear and pressure waves, respectively. Note
that the decomposition of the solution into its solenoidal and irrotational parts
is referred to as the Helmholtz decomposition [32].

3.2 Moving PML sweeping preconditioner

The focus of this dissertation is on parallelization of a class of sweeping
preconditioners which makes use of auxiliary problems with artificial radiation
boundary conditions in order to approximate the Schur complements that arise
during a block LDLT factorization. The approach is referred to as a moving
PML preconditioner since the introductory paper represented the artificial
radiation boundary conditions using Perfectly Matched Layers [19, 29, 44].

One interpretation of radiation conditions is that they allow for the

analysis of a finite portion of an infinite domain, as their purpose is to enforce

60
the condition that waves propagate outwards and not reflect at the bound-
ary of the truncated domain. This concept is crucial to understanding the
Schur complement approximations that take place within the moving PML
sweeping preconditioner which is reintroduced in this document for the sake
of completeness.

For the sake of simplicity, we will describe the preconditioner in the

context of a second-order finite-difference discretization over the unit cube,
with PML used to approximate a radiation boundary condition on the x3 =
0 face and homogeneous Dirichlet boundary conditions applied on all other
boundaries (an x1 x3 cross-section is shown in Fig. 3.1). If the domain is
discretized into an n×n×n grid, then ordering the vertices in the grid such that
vertex (i1 , i2 , i3 ) is assigned index i1 + i2 n + i3 n2 results in a block tridiagonal
system of equations, say

A0,0 AT1,0
    
u0 f0
.
A1,0 A1,1 . . u1 f1
    
   
 ... ... ...  .. ..
= , (3.8)
  



 . .
.. ..  u   
 fn−2 
 . . ATn−1,n−2  n−2 

An−1,n−2 An−1,n−1 un−1 fn−1

where Ai,j propagates sources from the i’th x1 x2 plane into the j’th x1 x2 plane,
and the overall linear system is complex symmetric (not Hermitian) due to the
imaginary terms introduced by the PML [44].

If we were to ignore the sparsity within each block, then the naı̈ve
factorization and solve algorithms shown in Algorithms 3.1 and 3.2 would be
appropriate for a direct solver, where the application of Si−1 makes use of
the factorization of Si . While the computational complexities of these ap-
proaches are significantly higher than the multifrontal algorithms discussed in

61
 region
PML of inter-
est
x1

x3

Figure 3.1: An x1 x3 cross section of a cube with PML on its x3 = 0 face. The
domain is shaded in a manner which loosely corresponds to its extension into
the complex plane.

the previous chapter, which have an O(N 2 ) factorization cost and an O(N 4/3 )
solve complexity for regular three-dimensional meshes, they are the conceptual
starting points of the sweeping preconditioner.3

Algorithm 3.1: Naı̈ve block tridiagonal LDLT factorization.

O(n7 ) = O(N 7/3 ) work is required.
1 S0 := A0,0
2 Factor S0
3 for i = 0, ..., n − 2 do
4 Si+1 := Ai+1,i+1 − Ai+1,i Si−1 ATi+1,i
5 Factor Si+1
6 end

Suppose that we paused Algorithm 3.1 after computing the i’th Schur
complement, Si , where the i’th x1 x2 plane is in the interior of the domain
(i.e., it is not in a PML region). Due to the ordering imposed on the degrees
of freedom of the discretization, the first i Schur complements are equivalent

3
In fact, they are the starting points of both classes of sweeping preconditioners. The
H-matrix approach essentially executes these algorithms with H-matrix arithmetic.

62
 Algorithm 3.2: Naı̈ve block LDLT solve. O(n5 ) = O(N 5/3 ) work
is required.
// Apply L−1
1 for i = 0, ..., n − 2 do
2 ui+1 := ui+1 − Ai+1,i (Si−1 ui )
3 end
// Apply D−1
4 for i = 0, ..., n − 1 do
5 ui := Si−1 ui
6 end
// Apply L−T
7 for i = n − 2, ..., 0 do
8 ui := ui − Si−1 (ATi+1,i ui+1 )
9 end

to those that would have been produced from applying Algorithm 3.1 to an
auxiliary problem formed by placing a homogeneous Dirichlet boundary con-
dition on the (i + 1)’th x1 x2 plane and ignoring all of the subsequent planes
(see the left half of Fig. 3.2). While this observation does not immediately
yield any computational savings, it does allow for a qualitative description of
the inverse of the i’th Schur complement, Si−1 : it is the restriction of the half-
space Green’s function of the exact auxiliary problem onto the i’th x1 x2 plane
(recall that PML can be interpreted as approximating the effect of a domain
extending to infinity).

The main approximation made in the sweeping preconditioner can now

be succinctly described: since Si−1 is a restricted half-space Green’s function
which incorporates the velocity field of the first i planes, it is natural to ap-
proximate it with another restricted half-space Green’s function which only
takes into account the local velocity field, i.e., by moving the PML next to the
i’th plane (see the right half of Fig. 3.2).

63
 = ≈
x1 x1 x1

x3 x3 x3

Figure 3.2: (Left) A depiction of the portion of the domain involved in the
computation of the Schur complement of an x1 x2 plane (marked with the
dashed line) with respect to all of the planes to its left during execution of
Alg. 3.1. (Middle) An equivalent auxiliary problem which generates the same
Schur complement; the original domain is truncated just to the right of the
marked plane and a homogeneous Dirichlet boundary condition is placed on
the cut. (Right) A local auxiliary problem for generating an approximation to
the relevant Schur complement; the radiation boundary condition of the exact
auxiliary problem is moved next to the marked plane.

64
 If we use γ(ω) to denote the number of grid points of PML as a function
of the frequency, ω, then it is important to recognize that the depth of the
approximate auxiliary problems in the x3 direction is Ω(γ).4 If the depth of the
approximate auxiliary problems was O(1), then combining nested dissection
with the multifrontal method over the regular n × n × O(1) mesh would only
require O(n3 ) work and O(n2 log n) storage [54]. However, if the PML size is
a slowly growing function of frequency, then applying 2D nested dissection to
the quasi-2D domain requires O(γ 3 n3 ) work and O(γ 2 n2 log n) storage, as the
number of degrees of freedom in each front increases by a factor of γ and dense
factorizations have cubic complexity.

Let us denote the quasi-2D discretization of the local auxiliary problem

for the i’th plane as Hi , and its corresponding approximation to the Schur
complement Si as S̃i . Since S̃i is essentially a dense matrix, it is advantageous
to come up with an abstract scheme for applying S̃i−1 : Assuming that Hi was
ordered in a manner consistent with the (i1 , i2 , i3 ) 7→ i1 + i2 n + i3 n2 global
ordering, the degrees of freedom corresponding to the i’th plane come last and
we find that  
? ?
Hi−1 = , (3.9)
? S̃i−1
where the irrelevant portions of the inverse have been marked with a ?. Then,
trivially,       
0 ? ? 0 ?
Hi−1 = = , (3.10)
ui ? S̃i−1 ui S̃i−1 ui
which implies a method for quickly computing S̃i−1 ui given a factorization of
Hi :

4
In all of the experiments in this chapter, γ(ω) was either 5 or 6, and the subdomain
depth never exceeded 10.

65
 Algorithm 3.3: Application of S̃i−1 to ui given a multifrontal fac-
torization of Hi . O(γ 2 n2 log n) work is required.
1 Form ûi as the extension of ui by zero over the artificial PML
−1
2 Form v̂i := Hi ûi
−1
3 Extract S̃i ui from the relevant entries of v̂i

From now on, we write Ti to refer to the application of the (approximate)

inverse of the Schur complement for the i’th plane.

There are two more points to discuss before defining the full serial al-
gorithm. The first is that, rather than performing an approximate LDLT
factorization using a discretization of A, the preconditioner is instead built
from a discretization of an artificially damped version of the Helmholtz oper-
ator, say
(ω + iα)2
 
J ≡ −∆ − , (3.11)
c2 (x)
where α ≈ 2π is responsible for the artificial damping. This is in contrast
to shifted Laplacian preconditioners [18, 46], where α is typically O(ω) [47],
and our motivation is to avoid introducing large long-range dispersion error
by damping the long range interactions in the preconditioner. Just as A refers
to the discretization of the original Helmholtz operator, A, we will use J to
refer to the discretization of the artificially damped operator, J .

The second point is much easier to motivate: since the artificial PML
in each approximate auxiliary problem is of depth γ(ω), processing a single
plane at a time would require processing O(n) subdomains with O(γ 3 n3 ) work
each. Clearly, processing O(γ) planes at once has the same asymptotic com-
plexity as processing a single plane, and so combining O(γ) planes reduces
the setup cost from O(γ 3 N 4/3 ) to O(γ 2 N 4/3 ). Similarly, the memory usage

66
becomes O(γN log N ) instead of O(γ 2 N log N ).5 If we refer to these sets of
O(γ) contiguous planes as panels, which we label from 0 to m − 1, where
m = O(n/γ), and correspondingly redefine {ui }, {fi }, {Si }, {Ti }, and {Hi },
we have the following algorithm for setting up an approximate block LDLT
factorization of the discrete artificially damped Helmholtz operator:

Algorithm 3.4: Setup phase of the sweeping preconditioner.

O(γ 2 N 4/3 ) work is required.
1 S0 := J0,0
2 Factor S0
3 parallel for i = 1, ..., m − 1 do
4 Form Hi by prefixing PML to Ji,i
5 Factor Hi
6 end

Once the preconditioner is set up, it can be applied using a straightfor-

ward modification of Algorithm 3.2 which avoids redundant solves by combin-
ing the application of L−1 and D−1 (see Algorithm 3.5). Given that the pre-
conditioner can be set up with O(γ 2 N 4/3 ) work, and applied with O(γN log N )
work, it provides a near-linear scheme for solving Helmholtz equations if only
O(1) iterations are required for convergence. The experiments in this chapter
seem to indicate that, as long as the velocity model is not completely sur-
rounded by reflecting boundary conditions and/or high-velocity barriers, the
sweeping preconditioner indeed only requires O(1) iterations.

Although this chapter is focused on the parallel solution of Helmholtz

equations, Tsuji et al. have shown that the moving PML sweeping precon-
ditioner is equally effective for time-harmonic Maxwell’s equations [124, 125],

5
Note that increasing the number of planes per panel provides a mechanism for interpo-
lating between the sweeping preconditioner and a full multifrontal factorization.

67
 Algorithm 3.5: Application of the sweeping preconditioner.
O(γN log N ) work is required.
// Apply L−1 and D−1
1 for i = 0, ..., m − 2 do
2 ui := Ti ui
3 ui+1 := ui+1 − Ji+1,i ui
4 end
5 um−1 := Tm−1 um−1
// Apply L−T
6 for i = m − 2, ..., 0 do
T
7 ui := ui − Ti (Ji+1,i ui+1 )
8 end

and current work supports its effectiveness for time-harmonic linear elasticity.
The rest of this document will be presented in the context of the Helmholtz
equation, but we emphasize that all parallelizations should carry over to more
general wave equations in a conceptually trivial way.

3.3 Related work

A domain decomposition variant of the sweeping preconditioner was
recently introduced [120] which results in fast convergence rates, albeit at the
expense of requiring PML padding on both sides of each subdomain. Recalling
our previous analysis with respect to the PML size, γ, the memory usage of the
preconditioner scales linearly with the PML size, while the setup cost scales
quadratically. Thus, the domain decomposition approach should be expected
to use twice as much memory and require four times as much work for the
setup phase. On the other hand, doubling the subdomain sizes allows for
more parallelism in both the setup and solve phases, and less sweeps seem to
be required.

68
 Another closely related method is the Analytic ILU factorization [52].
Like the sweeping preconditioner, it uses local approximations of the Schur
complements of the block LDLT factorization of the Helmholtz matrix rep-
resented in block tridiagonal form. There are two crucial differences between
the two methods:

• Roughly speaking, AILU can be viewed as using Absorbing Boundary

Conditions (ABC’s) [42] instead of PML when forming approximate sub-
domain auxiliary problems. While ABC’s result in strictly 2D local sub-
problems, versus the quasi-2D subdomain problems which result from
using PML, they are well-known to be less effective approximations of
the Sommerfeld radiation condition (and thus the local Schur comple-
ment approximations are less effective). The sweeping preconditioner
handles its non-trivial subdomain factorizations via a multifrontal algo-
rithm.

• Rather than preconditioning with an approximate LDLT factorization

of the original Helmholtz operator, the sweeping preconditioner sets up
an approximate factorization of a slightly damped Helmholtz operator in
order to mitigate the dispersion error which would result from the Schur
complement approximations.

These two improvements are responsible for transitioning from the O(ω) iter-
ations required with AILU to the O(1) iterations needed with the sweeping
preconditioner (for problems without internal resonance).

Two other iterative methods warrant mentioning: the two-grid shifted-

Laplacian approach of [27] and the multilevel-ILU approach of [23]. Although
both require O(ω) iterations for convergence, they have very modest memory

69
requirements. In particular, [27] demonstrates that, with a properly tuned
two-grid approach, large-scale heterogeneous 3D problems can be solved with
impressive timings.

There has also been a recent effort to extend the fast-direct meth-
ods presented in [134] from definite elliptic problems into the realm of low-
to-moderate frequency time-harmonic wave equations [130, 131]. While their
work has resulted in a significant constant speedup versus applying a clas-
sical multifrontal algorithm to the full 3D domain [131], their results have
so far still demonstrated the same O(N 2 ) asymptotic complexity as standard
sparse-direct methods.

3.4 Parallel sweeping preconditioner

The setup and application stages of the sweeping preconditioner (Algs.
3.4 and 3.5) essentially consist of m = O(n/γ) multifrontal factorizations and
solves, respectively. The most important detail is that the subdomain factor-
izations can be performed in parallel, while the subdomain solves must happen
sequentially.6 When we also consider that each subdomain factorization re-
quires O(γ 3 n3 ) work, while subdomain solves only require O(γn2 log n) work,
we see that, relative to the subdomain factorizations, subdomain solves must
extract m = O(n/γ) more parallelism from a factor of O(γ 2 n/ log n) less op-
erations. We thus have a strong hint that, unless the subdomain solves are
carefully handled, they will be the limiting factor in the scalability of the
sweeping preconditioner.

6
While it is tempting to try to expose more parallelism with techniques like cyclic reduc-
tion (which is a special case of a multifrontal algorithm), their straightforward application
destroys the Schur complement properties that we exploit for our fast algorithm.

70
3.4.1 The need for scalable triangular solves

Roughly speaking, the analysis in [79] shows that, if pF processes are

used in the multifrontal factorization of our quasi-2D subdomain problems,
1/2
then we must have γn = Ω(pF ) in order to maintain constant efficiency as
pF is increased; similarly, if pS processes are used in the multifrontal triangu-
lar solves for a subdomain, then we must have γn ≈ Ω(pS ) (where we use ≈
to denote that the equality holds within logarithmic factors). Since we can
simultaneously factor the m = O(n/γ) subdomain matrices, we denote the
total number of processes as p and set pS = p and pF = O(p/m); then the
subdomain factorizations only require that n3 = Ω(p/γ), while the subdomain
solves have the much stronger constraint that n ≈ Ω(p/γ). This last constraint
should be considered unacceptable, as we have the conflicting requirement that
n3 ≈ O(p/γ) in order to store the factorizations in memory. It is therefore
advantageous to consider more scalable alternatives to standard multifrontal
triangular solves, even if they require additional computation. In particular,
for modest numbers of right-hand sides, we will make use of selective inver-
sion [102], and for large numbers of right-hand sides, we may make use of
the GEMM and TRSM-based algorithms discussed at the end of Chapter 2.
Since each application of the sweeping preconditioner requires two multifrontal
solves for each of the m = O(n/γ) subdomains, which are relatively small
and likely distributed over a large number of processes, we will later see that
high-performance triangular solves are crucial for the competitiveness of the
sweeping preconditioner on parallel architectures.

71
Figure 3.3: A separator-based elimination tree (right) over a quasi-2D subdo-
main (left)

3.4.2 Global vector distributions

The goal of this subsection is to describe an appropriate scheme for

distributing vectors which are supported over the entire domain (as opposed to
only over a panel auxiliary problem). And while the factorizations themselves
may have occurred on subteams of O(p/m) processes each, in order to make use
of all available processes for every subdomain solve, at this point we assume
that each auxiliary problem’s frontal tree has been selectively inverted and
is distributed using a subtree-to-subteam mapping (recall Fig. 2.9) over the
entire set of p processes.7

Since a subtree-to-subteam mapping will assign each supernode of an

auxiliary problem to a team of processes, and each panel of the original 3D
domain is by construction a subset of the domain of an auxiliary problem,
it is straightforward to extend the supernodal subteam assignments to the
full domain. We should then be able to distribute global vectors so that no

7
In cases where the available solve parallelism has been exhausted but the problem cannot
be solved on less processes due to memory constraints, it would be preferable to solve with
subdomains stored on subsets of processes.

72
  
0 2 4 0 2 4 0

 1 3 5 1 3 5 1 


 0 2 4 0 2 4 0 
 0 − 2 − 4

 1 3 5 1 3 5 1 
 | | |

 0 2 4 0 2 4 0 
 1 − 3 − 5
 1 3 5 1 3 5 1 
0 2 4 0 2 4 0

Figure 3.4: Overlay of the owning process ranks of an 7 × 7 matrix distributed

over a 2 × 3 process grid in the [MC , MR ] distribution, where MC assigns row
i to process row i mod 2, and MR assigns column j to process column i mod 3
(left). The process grid is shown on the right.

communication is required for readying panel subvectors for subdomain solves

(via extension by zero for interior panels, and trivially for the first panel). Since
our nested dissection process does not partition in the shallow dimension of
quasi-2D subdomains (see Fig. 3.3), extending a vector defined over a panel of
the original domain onto the PML-padded auxiliary domain simply requires
individually extending each supernodal subvector by zero in the x3 direction.

Consider an element-wise two-dimensional cyclic distribution [99] of a

frontal matrix F over q processes using an r × c process grid, where r and c
√
are O( q). Then the (i, j) entry will be stored by the process in the (i mod
r, j mod c) position in the process grid (see Figure 3.4). Using the notation
from [99], this distributed front would be denoted as F [MC , MR ], while its
top-left quadrant would be referred to as FT L [MC , MR ].

Loosely speaking, F [X, Y ] means that each column of F is distributed

using the scheme denoted by X, and each row is distributed using the scheme
denoted by Y . For the element-wise two-dimensional distribution used for F ,
[MC , MR ], we have that the columns of F are distributed like Matrix Columns
(MC ), and the rows of F are distributed like Matrix Rows (MR ). While

73
this notation may seem vapid when only working with a single distributed
matrix, it is useful when working with products of distributed matrices. For
instance, if a ‘?’ is used to represent rows/columns being redundantly stored
(i.e., not distributed), then the result of every process multiplying its local
submatrix of A[X, ?] with its local submatrix of B[?, Y ] forms a distributed
matrix C[X, Y ] = (AB)[X, Y ] = A[X, ?] B[?, Y ], where the last expression
refers to a per-process local multiplication.

We can now decide on a distribution for each supernodal subvector,

say xs , based on the criteria that it should be fast to form FT L xs and FTTL xs
in the same distribution as xs , given that FT L is distributed as FT L [MC , MR ].
Suppose that we define a Column-major Vector distribution (VC ) of xs , say
xs [VC , ?], to mean that entry i is owned by process i mod q, which corresponds
to position (i mod r, bi/rc mod c) in the process grid (if the grid is constructed
with a column-major ordering of the process ranks; see the left side of Fig. 3.5).
Then a call to MPI_Allgather [39] within each row of the process grid would
allow for each process to collect all of the data necessary to form xs [MC , ?], as
for any process row index s ∈ {0, 1, ..., r − 1},
c−1
[
{i ∈ N0 : i mod r = s} = {i ∈ N0 : i mod q = s + tr}. (3.12)
t=0

See the left side of Fig. 3.6 for an example of an [MC , ?] distribution of a 7 × 3
matrix.

Similarly, if xs was distributed with a Row-major Vector distribution

(VR ), as shown on the right side of Fig. 3.5, say xs [VR , ?], so that entry i is
owned by the process in position (bi/cc mod r, i mod c) of the process grid,
then a call to MPI_Allgather within each column of the process grid would

74
    
0 0

 1 


 2 


 2 


 4 


 3 ,


 1 


 4 


 3 

 5   5 
0 0

Figure 3.5: Overlay of the owning process ranks of a vector of height 7 dis-
tributed over a 2 × 3 process grid in the [VC , ?] vector distribution (left) and
the [VR , ?] vector distribution (right).

   
{0, 2, 4} {0, 1}

 {1, 3, 5} 


 {2, 3} 


 {0, 2, 4} 


 {4, 5} 


 {1, 3, 5} ,


 {0, 1} 


 {0, 2, 4} 


 {2, 3} 

 {1, 3, 5}   {4, 5} 
{0, 2, 4} {0, 1}

Figure 3.6: Overlay of the owning process ranks of a vector of height 7 dis-
tributed over a 2 × 3 process grid in the [MC , ?] distribution (left) and the
[MR , ?] distribution (right).

75
provide each process with the data necessary to form xs [MR , ?]. Under rea-
sonable assumptions, both of these redistributions can be shown to have per-
√
process communication volume lower bounds of Ω(n/ p) (if FT L is n × n)
√
and latency lower bounds of Ω(log2 ( p)) [28]. We also note that translating
between xs [VC , ?] and xs [VR , ?] simply requires permuting which process owns
each local subvector, so the communication volume would be O(n/p), while
the latency cost is O(1).

We have thus described efficient techniques for redistributing xs [VC , ?]

to both the xs [MR , ?] and xs [MC , ?] distributions, which are the first steps for
our parallel algorithms for forming FT L xs and FTTL xs , respectively: Denoting
the distributed result of each process in process column t ∈ {0, 1, ..., c−1} mul-
tiplying its local submatrix of FT L [MC , MR ] by its local subvector of xs [MR , ?]
Pc−1 (t)
as z (t) [MC , ?], it holds that (FT L xs [MC , ?] = t=0 z [MC , ?]. Since Equa-

tion (3.12) also implies that each process’s local data from a [VC , ?] distri-
bution is a subset of its local data from a [MC , ?] distribution, a simultane-
ous summation and scattering of {z (t) [MC , ?]}c−1
t=0 within process rows, per-

haps via MPI_Reduce_scatter or MPI_Reduce_scatter_block, yields the de-

sired result, (FT L xs )[VC , ?]. An analogous process with (FT L [MC , MR ])T =
FTTL [MR , MC ] and xs [MC , ?] yields (FTTL xs )[VR , ?], which can then be cheaply
permuted to form (FTTL xs )[VC , ?]. Both calls to MPI_Reduce_scatter_block
can be shown to have the same communication lower bounds as the previously
discussed MPI_Allgather calls [28].

As discussed at the beginning of this section, defining the distribution

of each supernodal subvector specifies a distribution for a global vector, say
[G, ?]. While the [VC , ?] distribution shown in the left half of Fig. 3.5 simply
assigns entry i of a supernodal subvector xs , distributed over q processes, to

76
process i mod q, we can instead choose an alignment parameter, σ, where
0 ≤ σ < q, and assign entry i to process (i + σ) mod q. If we simply set σ = 0
for every supernode, as the discussion at the beginning of this subsection
implied, then at most O(γn) processes will store data for the root separator
supernodes of a global vector, as each root separator only has O(γn) degrees
of freedom by construction. However, there are m = O(n/γ) root separators,
so we can easily allow for up to O(n2 ) processes to share the storage of a
global vector if the alignments are carefully chosen. It is important to notice
that the top-left quadrants of the frontal matrices for the root separators can
each be distributed over O(γ 2 n2 ) processes, so O(γ 2 n2 ) processes can actively
participate in the corresponding triangular matrix-vector multiplications.

3.4.3 Parallel preconditioned GMRES(k)

Since, by hypothesis, only O(1) iterations of GMRES(k) will be re-

quired for convergence with the sweeping preconditioner, a cursory inspec-
tion of Algorithm 3.5 reveals that the vast majority of the work required
for GMRES(k) will be in the multifrontal solves during each preconditioner
application, but that a modest amount of time will also be spent in sparse
matrix-vector multiplication with the discrete Helmholtz operator, A, and the
off-diagonal blocks of the discrete artificially damped Helmholtz operator, J.
It is thus important to parallelize the sparse matrix-vector multiplies, but it
is not crucial that the scheme be optimal, and so we simply distribute A and
J in the same manner as vectors, i.e., with the [G, ?] distribution derived from
the auxiliary problems’ frontal distributions.

We also note that, if we are to apply the preconditioner to several

vectors at once, then we must make use of a version of GMRES(k) which can

77
handle several right-hand sides at once. The approach taken in the following
experiments is a straight-forward modification of Algorithm B.1 which simply
maintains a separate Krylov subspace and Rayleigh quotient for each right-
hand side and iterates until all relative residuals are sufficiently small.

3.5 Experimental results

Most of the techniques discussed thus far were instantiated as part
of a distributed-memory multifrontal solver, Clique [100], and a distributed-
memory moving-PML sweeping preconditioner, Parallel Sweeping Precondi-
tioner (PSP) [101]. Our experiments were performed on the Texas Advanced
Computing Center (TACC) machine, Lonestar, which is comprised of 1,888
compute nodes, each equipped with two hex-core 3.33 GHz processors and
24 GB of memory, which are connected with QDR InfiniBand using a fat-tree
topology. Our tests launched eight MPI processes per node in order to provide
each MPI process with 3 GB of memory.

Our experiments took place over five different 3D velocity models:

1. A uniform background with a high-contrast barrier. The domain is the

unit cube and the wave speed is 1 except in [0, 1] × [0.25, 0.3] × [0, 0.75],
where it is 1010 .

2. A wedge problem over the unit cube, where the wave speed is set to 2
if Z ≤ 0.4 + 0.1x2 , 1.5 if otherwise Z ≤ 0.8 − 0.2x2 , and 3 in all other
cases.

3. A two-layer model defined over the unit cube, where c = 4 if x2 < 0.5,
and c = 1 otherwise.

78
 2 +|x 2)
4. A waveguide over the unit cube: c(x) = 1.25(1−0.4e−32(|x1 −0.5| 2 −0.5|
).

5. The SEG/EAGE Overthrust model [4], see Fig. 3.8.

In all of the following experiments, the shortest wavelength of each

model is resolved with roughly ten grid points and the performance of the
preconditioner is tested using the following four forcing functions:

2
1. a single shot centered at x0 , f0 (x) = ne−10nkx−x0 k ,
P2 2
2. three shots, f1 (x) = i=0 ne−10nkx−xi k ,
2
3. a Gaussian beam centered at x2 in direction d, f2 (x) = eiωx·d e−4ωkx−x2 k ,
and

4. a plane wave in direction d, f3 (x) = eiωx·d ,

where x0 = (0.5, 0.5, 0.1), x1 = (0.25, 0.25, 0.1), x2 = (0.75, 0.75, 0.5), and
√
d = (1, 1, −1)/ 3. Note that, in the case of the Overthrust model, these
source locations should be interpreted proportionally (e.g., an x3 value of 0.1
means a depth which is 10% of the total depth of the model). Due to the
oscillatory nature of the solution, we simply choose the zero vector as our
initial guess in all experiments.

The first experiment was meant to test the convergence rate of the
sweeping preconditioner over domains spanning 50 wavelengths in each direc-
tion (resolved to ten points per wavelength) and each test made use of 256
nodes of Lonestar. During the course of the tests, it was noticed that a signifi-
cant amount of care must be taken when setting the amplitude of the derivative
of the PML takeoff function (i.e., the “C” variable in Equation (2.1) from [44]).

79
 velocity model
barrier wedge two-layers waveguide
Hz 50 75 50 37.5
PML amplitude 3.0 4.0 4.0 2.0
# of its. 28 49 48 52

Table 3.1: The number of iterations required for convergence for four model
problems (with four forcing functions per model). The grid sizes were 5003
and roughly 50 wavelengths were spanned in each direction. Five grid points
were used for all PML discretizations, four planes were processed per panel,
and the damping factors were all set to 7.

For the sake of brevity, hereafter we refer to this variable as the PML ampli-
tude. A modest search was performed in order to find a near-optimal value for
the PML amplitude for each problem. These values are reported in Table 3.1
along with the number of iterations required for the relative residuals for all
four forcing functions to reduce to less than 10−5 .

It was also observed that, at least for the waveguide problem, the con-
vergence rate would be significantly improved if 6 grid points of PML were
used instead of 5. In particular, the 52 iterations reported in Table 3.1 reduce
to 27 if the PML size is increased by one. Interestingly, the same number
of iterations are required for convergence of the waveguide problem at half
the frequency (and half the resolution) with five grid points of PML. Thus,
it appears that the optimal PML size is a slowly growing function of the fre-
quency.8 We also note that, though it was not intentional, each of the these
first four velocity models is invariant in one or more direction, and so it would
be straightforward to sweep in a direction such that only O(1) panel factoriza-
tions would need to be performed, effectively reducing the complexity of the

8
A similar observation is also made in [120].

80
 8e-05
0.0002

6e-05
0.00015

4e-05
0.0001

2e-05
5e-05

0
0

-2e-05
-5e-05

-4e-05
-0.0001

-6e-05
-0.00015

-8e-05
-0.0002

6e-05
0.0015

4e-05
0.001

0.0005 2e-05

0 0

-0.0005 -2e-05

-0.001 -4e-05

-0.0015 -6e-05

Figure 3.7: A single x2 x3 plane from each of the four analytical velocity models
over a 5003 grid with the smallest wavelength resolved with ten grid points.
(Top-left) the three-shot solution for the barrier model near x1 = 0.7, (bottom-
left) the three-shot solution for the two-layer model near x1 = 0.7, (top-right)
the single-shot solution for the wedge model near x1 = 0.7, and (bottom-right)
the single-shot solution for the waveguide model near x1 = 0.55.

81
setup phase to O(γ 2 N ).

The last experiment was meant to simultaneously test the convergence

rates and scalability of the new sweeping preconditioner using the Overthrust
velocity model (see Fig. 3.8) at various frequencies, and the results are reported
in Table 3.2. It is important to notice that this is not a typical weak scaling
test, as the number of grid points per process was fixed, not the local memory
usage or computational load, which both grow superlinearly with respect to
the total number of degrees of freedom. Nevertheless, including the setup
phase, it took less than three minutes to solve the 3.175 Hz problem with four
right-hand sides with 128 cores, and just under seven and a half minutes to
solve the corresponding 8 Hz problem using 2048 cores. Also, while it is by
no means the main message of this section, the timings without making use
of selective inversion are also reported in parentheses, as the technique is not
widely implemented.9

3.6 Summary
A parallelization of the moving-PML sweeping preconditioner was pre-
sented and successfully applied to five challenging heterogeneous velocity mod-
els, including the SEG/EAGE Overthrust model. The primary challenges
were:

• ensuring that the subdomain multifrontal solves were scalable (e.g., through
selective inversion),

• ensuring that the PML profile was appropriately matched to the velocity

9
Other than Clique, the only other implementation appears to be in DSCPACK [103].

82
Figure 3.8: Three cross-sections of the SEG/EAGE Overthrust velocity model,
which represents an artificial 20 km × 20 km × 4.65 km domain, containing an
overthrust fault, using samples every 25 m. The result is an 801 × 801 × 187
grid of wave speeds varying discontinuously between 2.179 km/sec (blue) and
6.000 km/sec (red).

83
Figure 3.9: Three planes from an 8 Hz solution with the Overthrust model at
its native resolution, 801 × 801 × 187, with a single localized shot at the center
of the x1 x2 plane at a depth of 456 m: (top) a x2 x3 plane near x1 = 14 km,
(middle) an x1 x3 plane near x2 = 14 km, and (bottom) an x1 x2 plane near
x3 = 0.9 km.
84
 number of processes
128 256 512 1024 2048
Hz 3.175 4 5.04 6.35 8
grid 319×319×75 401×401×94 505×505×118 635×635×145 801×801×187
setup 48.40 (46.23) 66.33 (63.41) 92.95 (86.90) 130.4 (120.6) 193.0 (175.2)
apply 1.87 (4.26) 2.20 (5.11) 2.58 (9.61) 2.80 (13.3) 3.52 (24.5)
3 digits 42 44 42 39 40
4 digits 54 57 57 58 58
5 digits 63 69 70 68 72

Table 3.2: Convergence rates and timings (in seconds) on TACC’s Lonestar
for the SEG/EAGE Overthrust model, where timings in parentheses do not
make use of selective inversion. All cases used a complex double-precision
second-order finite-difference stencil with five grid spacings for all PML (with
a magnitude of 7.5), and a damping parameter of 2.25π. The preconditioner
was configured with four planes per panel and eight processes per node. The
‘apply’ timings refer to a single application of the preconditioner to four right-
hand sides.

field, and

• avoiding unnecessary communication by distributing right-hand sides in

a manner which conforms to the distributions of the multifrontal factor-
izations of the subdomain auxiliary problems.

The results presented in this chapter are admittedly quite conservative in

their usage of ten grid points per shortest wavelength, and, in fact, several
experiments were withheld which aggressively solved the Overthrust model
at a frequency of 30 Hz over a 1201 × 1201 × 287 grid using a second-order
finite-difference stencil. Although the pollution error [15] was almost certainly
quite high, we note that the number of iterations required for convergence was
essentially the same as for the 8 Hz example which we described in detail, and
that the pollution effects can be greatly mitigated by switching to spectral
elements [123, 124] without a significant increase in computational cost.

85
 Chapter 4

A compressed sweeping preconditioner

We have shown that the sweeping preconditioner requires O(γ 2 N 4/3 )

work in its setup phase and O(γN log N ) storage space, where γ is the number
of grid points used for artificial PML. Since γ appears to need to grow roughly
logarithmically with the number of wavelengths spanned by the domain [98,
120], it would be worthwhile to find a means of lessening its impact on the
memory requirements of the sweeping preconditioner. We will shortly see
that the translation invariance of the Green’s functions for homogeneous free-
space time-harmonic wave equations provides theoretical support for such a
compression scheme.

Rather than listing the free-space Green’s functions for the most im-
portant classes of time-harmonic wave equations and showing that their trans-
lation invariance is self-evident, we will instead consider the translation invari-
ance as a direct result of the lack of a frame of reference for a problem posed
over an infinite homogeneous domain. We will use the 3D Helmholtz Green’s
function as an example, but it is important to keep in mind that the techniques
generalize to other time-harmonic wave equations.

86
4.1 Theory
Consider evaluating the Green’s function for the free-space 3D Helmholtz
equation, (
exp(iω|x−y|)
4π|x−y|
, x 6= y,
G(x, y) =
+∞, x = y,
with x and y restricted to a set of points evenly spaced over a line segment,
say x, y ∈ {jhê}q−1 3 3
j=0 ⊂ R , where h > 0 and ê ∈ R is an arbitrary unit vector.

Despite the fact that there are q 2 different choices for the pair (x, y), |x − y|
can only take on q different values, namely, {0, h, . . . , (q − 1)h} (and thus G
can also only take on at most q unique values). This redundancy is primarily
due to the translation invariance of G, i.e.,

G(x, y) = G(x + t, y + t) ∀ t ∈ R3 .

Our next step will be to discuss situations where G is only translation invariant
in a particular direction.

Lemma 4.1.1. Let G : Rd × Rd → C ∪ {∞} be invariant under translation in

some direction ê ∈ Rd . That is to say,

G(x, y) = G(x + t, y + t) ∀ x, y ∈ R3 and t ∝ ê.

If we extrude two points u, v ∈ Rd with spacing h ∈ R to form the sets U =

{u + khê}q−1 q−1
k=0 and V = {v + `hê}`=0 , then G restricted to U × V can take on

at most 2q − 1 unique values.

If we further assume that G satisfies the mirror symmetry condition

G(u + khê, v) = G(u − khê, v), k = 0, . . . , q − 1,

then the image of U × V through G consists of at most q unique values.

87
Proof. Due to the translation invariance of G in the direction ê,

G(u + khê, v + `hê) = G(u + (k − `)hê, v),

and so −q < k − ` < q implies the first result.

Now suppose that G(u + hkê, v) = G(u − hkê, v) for k = 0, . . . , q − 1.

Then
G(u + (k − `)hê, v) = G(u + |k − `|hê, v),

where 0 ≤ |k − `| < q.

Remark 4.1.1. The mirror symmetry condition,

G(u + khê, v) = G(u − khê, v),

is useful in many cases. Consider the function G(x, y) = eiω|x−y| , where x, y ∈

R3 , and suppose that u − v ⊥ ê.

Lemma 4.1.2. Let G : Rd × Rd → C ∪ {∞} be invariant under translation

in some direction ê ∈ Rd . Then, for any {(ui , vi )}m−1 d d
i=0 ⊂ R × R , the m × q
2

matrix

K(i, j1 + qj2 ) = G(ui + j1 hê, vi + j2 hê), 0 ≤ i < m, 0 ≤ j1 , j2 < q,

has at most 2q − 1 unique columns.

If we also assume that G satisfies the mirror symmetry condition

G(ui + hkê, vi ) = G(ui − hkê, vi ), ∀ 0 ≤ i < m, 0 ≤ k < q,

then K has at most q unique columns.

88
Proof. Using translation invariance,

K(i, j1 + qj2 ) = G(ui + (j1 − j2 )hê, vi ),

and thus, since −q < j1 − j2 < q, there are at most 2q − 1 unique columns.

If we then assume that G(ui + hkê, vi ) = G(ui − hkê, vi ) for each

i = 0, . . . , m − 1 and k = 0, . . . , q − 1, then

K(i, j1 + qj2 ) = G(ui + |j1 − j2 |hê, vi ),

where 0 ≤ |j1 − j2 | < q.

Remark 4.1.2. Although Lemma 4.1.2 may seem overly specific, transla-
tion invariance in a particular direction arises naturally for Green’s func-
tions of constant-coefficient problems posed over unbounded domains which
are invariant in a particular direction, such as the infinitely tall rectangle
[0, 1] × (−∞, +∞). We will now show how the method of mirror images [77]
can be used to extend these ideas to semi-infinite domains, e.g., [0, 1]×(−∞, 0]
with a zero Dirichlet boundary condition imposed over [0, 1] × {0}. Once we
have done so, we will have a strong argument for the compressibility of the
frontal matrices arising from the semi-infinite subdomain auxiliary problems
posed by the sweeping preconditioner.

Example 4.1.1. Suppose that G : R3 × R3 → C ∪ {∞} is a Green’s function

representing the potential generated at some point x ∈ R3 due to a point source
at y ∈ R3 , and that G is both translation invariant and mirror symmetric
(even) with respect to its last coordinate, i.e., if u, v ∈ R2 × {0}, then

G(u + γê3 , v) = G(u − γê3 , v).

89
Then we can use the original Green’s function, G, to construct a half-space
Green’s function, say G̃, with domain (R2 ×(−∞, 0])2 and boundary condition
G̃(x, y) = 0 when x ∈ R2 × {0}. In particular, we may set

G̃(u, v) = G(u, v) − G(u, R(v)),

where R(v) is the reflection of v over the plane R2 × {0} and represents the
location of an artificial charge which counteracts the charge located at v over
the plane R2 × {0}.

The contribution of the conceptual mirror image charge, −G(u, R(v)),

is often referred to as a reflection of the charge located at v off of the boundary
located at R2 × {0} (see Figure 4.1). In general, the introduction of the
Dirichlet boundary condition destroys translation invariance in the direction
normal to the boundary condition, but we will now show that an analogue of
Lemma 4.1.2 still holds.

Lemma 4.1.3. Let a function G : Rd × Rd → C ∪ {∞} be invariant under

translation in the direction ê ∈ Rd and let {(ui , vi )}m−1 d d
i=0 ⊂ R × R , where each

vi is orthogonal to ê. If we then define

G̃(x, y) = G(x, y) − G(x, R(y)),

where R(y) = y − 2ê(ê, y), then the m × q 2 matrix

K(i, j1 + qj2 ) = G̃(ui + j1 hê, vi + j2 hê), 0 ≤ i < m, 0 ≤ j1 , j2 < q,

has a rank of at most 3q − 2.

If G also satisfies the mirror symmetry condition

G(ui + hjê, vi ) = G(ui − hjê, vi ), 0 ≤ i < m, 0 ≤ j < q,

then K has a rank of at most 2q − 1.

90
Figure 4.1: (Top) The real part of a 2D slice of the potential generated by a
source in the obvious location and (bottom) the same potential superimposed
with that of a mirror-image charge. The potential is identically zero along the
vertical line directly between the two charges.

Proof. By definition,

K(i, j1 + qj2 ) = G(ui + j1 hê, vi + j2 hê) − G(ui + j1 hê, R(vi + j2 hê)),

and, since vi ⊥ ê, R(vi ) = vi , and we find that

K(i, j1 + qj2 ) = G(ui + (j1 − j2 )hê, vi ) − G(ui + (j1 + j2 )hê, vi ).

The columns of K are therefore linear combinations of the vectors

pj (i) = G(ui + jhê, vi ),

where −q < j < 2q − 1, and so we have established the first result.

If we additionally assume that

G(ui + hjê, vi ) = G(ui − hjê, vi ), ∀ 0 ≤ i < m, 0 ≤ j < q,

then

K(i, j1 + qj2 ) = G(ui + |j1 − j2 |hê, vi ) − G(ui + (j1 + j2 )hê, vi ).

91
It follows that the columns of K are again linear combinations of the vectors

pj (i) = G(ui + jhê, vi ),

but now 0 ≤ j < 2q − 1, which yields the second result.

Example 4.1.2. Consider the Green’s function G : R3 × R3 → C ∪ {∞} for

the free-space 3D Helmholtz equation,
(
exp(iω|x−y|)
4π|x−y|
, x 6= y,
G(x, y) =
+∞, x = y,

and set G̃(x, y) = G(x, y) − G(x, R(y)), where R(y) = y − 2ê3 (ê3 , y). Then G̃
restricted to (R2 × (−∞, 0])2 is a Green’s function for the constant-coefficient
half-space Helmholtz problem,

−∆ − ω 2 u(x) = f (x),
 

where x ∈ R2 × (−∞, 0] and u(x) = 0 when x lies on the half-space boundary,

R2 × {0}.

Now suppose that we are interested in representing G̃ over a portion

of a plane lying against the half-space boundary, say {0} × [0, 1] × [0, λ] (see
Figure 4.2). If we let {ui }s−1
i=0 be an arbitrary collection of points from {0} ×

[0, 1] × {0}, and set ê = ê3 , then for any integer q > 1, Lemma 4.1.3 implies
that the rank of the s2 × q 2 matrix
j1 λ j2 λ
K(i1 + si2 , j1 + qj2 ) = G̃(ui1 + ê3 , ui2 + ê3 ),
q−1 q−1
where 0 ≤ i1 , i2 < s and 0 ≤ j1 , j2 < q, is at most 3q − 2.

Furthermore, since every vector ui is orthogonal to ê3 , so is every linear

combination of the ui ’s. Thus,
p
k(ui1 + γê3 ) − ui2 k2 = |ui1 − ui2 |2 + |γ|2 ,

92
 1

ê2

ê1
ê3 λ

Figure 4.2: A set of six unevenly spaced points (in blue) along the line segment
{0} × [0, 1] × {0} and their four evenly spaced translations (in red) in the
direction ê3 .

and so the sign of γ is irrelevant when evaluating G̃(ui1 +γê3 , ui2 ). We therefore
satisfy the mirror symmetry conditions for Lemma 4.1.3 and find that the rank
of K is at most 2q − 1.

Remark 4.1.3. Although our previous example used identical source and
target sets, notice that this is not a requirement of Lemma 4.1.3.

Remark 4.1.4. The s2 × q 2 matrix K formed in Example 4.1.2 is somewhat

of an unconventional way to represent a discrete Green’s function: the (j1 +
qj2 )’th column of K stores the potentials generated over s points on the line
{0} × [0, 1] × {j1 h} for each of s different point sources lying on the line
{0}×[0, 1]×{j2 h}, resulting in s2 entries per column. It is much more common
to instead work with the square qs × qs matrix

G̃(i, j) = G̃(xi , xj ),

where {xi }qs−1

i=0 is an enumeration of the entire set of samples of the plane

93
segment {0} × [0, 1] × [0, λ]. This representation is satisfying because G̃ can
be interpreted as mapping a charge distribution over the plane segment to its
resulting potential field over the same region.

It is natural to now ask how a low-rank representation of K can be

translated into a data-sparse representation of G̃. The following theorem
makes use of the fact that, with a particular ordering of G̃, G̃ is a q × q matrix
of blocks of size s × s, where each block is a column of K. The result is that G̃
may be represented as the sum of O(q) Kronecker-product matrices [128, 129].

Definition 4.1.1 (Kronecker-Zehfuss product). Given an m1 × m2 matrix Y

and an n1 × n2 matrix Z, the Kronecker product Y ⊗ Z, sometimes also called
the Zehfuss product, is defined as the m1 n1 × m2 n2 matrix
 
ψ0,0 Z ··· ψ0,m2 −1 Z
Y⊗Z= .. ... ..
,
 
. .
ψm1 −1,0 Z · · · ψm1 −1,m2 −1 Z

where ψi,j is the (i, j) entry of Y.

Definition 4.1.2. The “vec” operator takes an m × n array X and produces a

one-dimensional array of length mn, where x is formed by successively stacking
the columns of X. That is, the (i, j) entry of X becomes the i + jm entry of
x, and we write x = vec(X).

Remark 4.1.5. The previous definition was purposefully written to avoid

specifying that the result of the vec operator must be a vector, as it is often
useful to reshape two-dimensional arrays whose entries are more general than
scalar values. In particular, the following theorem uses the vec operator on a
two-dimensional array of points from Rd .

94
Theorem 4.1.4. Let G : Rd × Rd → C ∪ {∞} be invariant under translation
in the direction ê ∈ Rd and define

G̃(x, y) = G(x, y) − G(x, R(y)),

a −1 b −1
where R(y) = y −2ê(ê, y). Furthermore, let {ai }si=0 , {bi }si=0 ⊂ Rd where each
bi is orthogonal to ê, and define an array of qsa target points, x = vec(X),
and an array of qsb source points, y = vec(Y ), where X(i, j) = ai + jhê and
Y (i, j) = bi + jhê. Then the qsa × qsb matrix

G̃(i, j) = G̃(xi , yj )

can be represented as a sum of Kronecker products,

3q−3
X
G̃ = Yt ⊗ Zt ,
t=0

where each Yt is q × q and each Zt is sa × sb .

If we further assume the mirror symmetry conditions

G(ai + hkê, bj ) = G(ai − hkê, bj ), ∀ 0 ≤ i < sa , 0 ≤ j < sb , 0 ≤ k < q,

then G̃ can be represented as the sum of only 2q−1 Kronecker product matrices.

Proof. Partition G̃ as
 
G̃0,0 · · · G̃0,q−1
G̃ =  ... .. ..
,
 
. .
G̃q−1,0 · · · G̃q−1,q−1

where each submatrix G̃i,j is sa ×sb . Due to the ordering imposed upon G̃, the
matrix K formed such that its i + qj’th column is equal to vec(G̃i,j ) satisfies

95
the conditions for Lemma 4.1.3, which shows that the rank of K is at most
3q − 2. That is, we may decompose K as
3q−3
X
K= zt ytT .
t=0

Since each column of K corresponds to some sa × sb block of G̃, each rank-one

contribution to K corresponds to a Kronecker product contribution to G̃, say
Yt ⊗ Zt , where Yt is q × q, Zt is sa × sb , vec(Yt ) = yt , and vec(Zt ) = zt . This
shows the first result.

Now suppose that

G(ai + hkê, bj ) = G(ai − hkê, bj ), ∀ 0 ≤ i < sa , 0 ≤ j < sb , 0 ≤ k < q.

Then K satisfies the second condition of Lemma 4.1.3 and has a rank of at
most 2q − 1.

Lemma 4.1.5. For any matrices Y and Z and conforming vector x,

(Y ⊗ Z)x = vec(ZXYT ),

where X is defined such that vec(X) = x.

Proof. Let Y be m × n. Then, by definition of the Kronecker product,

 
Zxk ψ0,k
n−1 
 Zxk ψ1,k 
X 
(Y ⊗ Z)x =  .. ,
k=0
 . 
Zxk ψm−1,k
where ψi,j is the (i, j) entry of Y. If we then denote the i0 th row of Y by ŷi
and define X = [x0 , x1 , · · · , xn−1 ], so that vec(X) = x, we see that
 
ZXŷ0T
 ZXŷT 
1 
(Y ⊗ Z)x =   = vec(ZXYT ).

..
 . 
T
ZXŷm−1

96
Corollary 4.1.6. Given a q × q matrix Y, an sa × sb matrix Z, and a vector x
of length qsb , the product (Y ⊗ Z)x can be formed with 2q 2 min{sa , sb } + 2qsa sb
total additions and multiplications.

Proof. The product of an m × k matrix and a k × n matrix can be computed

with 2mnk operations and we are free to form ZXYT as either Z(XYT ) or
(ZX)YT .

Theorem 4.1.7. Suppose a qsa × qsb matrix G̃ satisfies the first set of condi-
tions for Theorem 4.1.4. Then the Kronecker product representation of G̃ only
requires the storage of (3q − 2)(q 2 + sa sb ) scalars, and this representation can
be be used to linearly transform a vector with (3q − 2)(2q 2 min{sa , sb } + 2qsa sb )
floating-point operations.

If G̃ satisfies the second set of conditions for Theorem 4.1.4, then only
(2q − 1)(q 2 + sa sb ) scalars of storage are required, and the operator may be
applied in this form with (2q − 1)(2q 2 min{sa , sb } + 2qsa sb ) total additions and
multiplications.

Proof. This is essentially an immediate consequence of Theorem 4.1.4 and

Corollary 4.1.6; we need only recognize that the each Kronecker product Yt ⊗
Zt only requires storage of the q × q matrix Yt and the sa × sb matrix Zt .

Remark 4.1.6. It is now important to compare the results from Theorem

4.1.7 to the case where G̃ is simply stored as a dense matrix, which would
require q 2 sa sb unit of storage and 2q 2 sa sb floating-point operations for naı̈vely
mapping a vector. In particular, let us consider the case where q = sa = sb .
Then the memory requirement for the Kronecker product scheme, assuming

97
that the second set of conditions for 4.1.4 are satisfied, is essentially 4q 3 entries
of storage versus the q 4 entries required for the standard dense storage scheme,
which is clearly a factor of q/4 compression. On the other hand, if q  sa , sb ,
then we could expect a factor of q/2 compression.

This lossless compression scheme comes at somewhat of a price. A

direct comparison of the work required for performing the matrix-vector mul-
tiplications shows that, as long as q ≤ min{sa , sb }, the Kronecker product
scheme may require as much as four times the work as the dense storage
scheme. On the other hand, the Kronecker product approach makes use of
matrix-matrix multiplication instead of matrix-vector multiplications and can
therefore take better advantage of cache (i.e., it can use level 3 BLAS opera-
tions instead of level 2). A proper sequential implementation of the Kronecker
product scheme should therefore not be significantly slower than the standard
scheme.

Example 4.1.3. Suppose that we are interested in representing the Green’s

function for the constant-coefficient 3D Helmholtz equation,

−∆ − ω 2 x GD (x, y) = δ(x − y), in [0, 1] × [0, 1] × (−∞, ∞),

 

with zero Dirichlet boundary conditions on the finite boundaries and radi-
ation conditions posed in the remaining two directions, where the subscript
x implies that the operator −∆ − ω 2 acts on the x variable of the Green’s
function GD (x, y). Due to Dirichlet boundary conditions being posed on both
sides of the domain in the first two dimensions, mirror imaging techniques for
constructing the Green’s function for this problem from the free-space Green’s
function would require an infinite summation, as the reflections from the left
wall would reflect off the right wall, which would then reflect off of the left
wall, ad infinitum.

98
 We will avoid discussion of the convergence of such summations [21] and
simply recognize that the resulting Green’s function will remain translation
invariant in the third coordinate. Then we may use the method of mirror
imaging in the third dimension to construct another Green’s function, say
G̃D (x, y) = GD (x, y) − GD (x, R(y)), for the problem

−∆ − ω 2 x G̃D (x, y) = δ(x − y), in [0, 1] × [0, 1] × (−∞, 0],

 

where R(y) = y − 2ê3 (ê3 , y) is responsible for reflecting y over the newly
imposed zero Dirichlet boundary condition on [0, 1] × [0, 1] × {0}. If x and
y are both perpendicular to ê3 , then GD retains the free-space Helmholtz
property that GD (x + γê3 , y) = GD (x − γê3 , y) for any γ ∈ R, and we see that
Theorem 4.1.7 can be invoked to show that samples of G̃D over a grid which
is uniform in the ê3 direction are compressible.

Theorem 4.1.8. The Green’s function, say G̃D , for the Helmholtz problem

−∆ − ω 2
 
x
G̃D (x, y) = δ(x − y), in [0, 1] × [0, 1] × (−∞, 0],

with zero Dirichlet boundary conditions on the finite boundaries and the Som-
merfeld radiation condition in the remaining direction, is compressible in the
following sense. For any points {ai }s−1 ⊥
i=0 ⊂ ê3 , if we define the array of points

x = vec(X), where X is the s × q matrix defined entrywise as X(i, j) =

ai + jhê3 , the matrix
G̃D (i, j) = G̃D (xi , xj )

has the Kronecker product decomposition

2q−2
X
G̃ = Yt ⊗ Zt ,
t=0

99
where Yt is q × q and Zt is s × s. Furthermore, this representation requires
only (2q − 1)(q 2 + s2 ) units of storage and can be used to linearly transform a
vector with (2q − 1)(2q 2 s + 2qs2 ) units of work.

4.2 Interpretation, application, and parallelization

We have just provided a significant amount of theoretical support for
a compression scheme for the frontal matrices formed during the multifrontal
factorization of each auxiliary problem posed by the sweeping preconditioner.
In particular, the inverse of the Schur complement of each frontal matrix, say

Ss−1 = [L(Ds , Ds )]−T [D(Ds , Ds )]−1 [L(Ds , Ds )]−1 ,

is closely related to the Green’s function resulting from the subproblem posed
over the subdomain covered by supernode s and its descendants, but with zero
Dirichlet boundary conditions posed over the artificial boundaries introduced
by the ancestor separators (consider Figure 3.3); Theorem 4.1.8 was specifically
designed to handle these artificial Dirichlet boundary conditions.

There are several points worth discussing:

• The theory from the previous section demonstrates precise cases where
the discrete Green’s function can be losslessly compressed as a sum of
Kronecker products, but we will make use of a thresholded singular value
decomposition in order to find an approximate compression in more gen-
eral scenarios (e.g., heterogeneous media and PML boundary conditions).

• We must modify our multifrontal scheme to directly compute each front’s

inverse Schur complement, and then compress this inverse. Unfortu-

100
 nately, much less can be said about the compression of the bottom-left
quadrant of the fronts.

Our Cholesky-like multifrontal LDLT factorization will thus need to be mod-

ified in order to perform a block LDLT factorization (see Algorithm 4.1), and
the top-left and bottom-left quadrants of each front will then need to be shuf-
fled into a different form and then compressed via a singular value decompo-
sition (see Algorithm 4.2), where each rank-one contribution to the permuted
matrix is a Kronecker-product contribution to the original matrix. We can
then alter triangular solves with an uncompressed block LDLT factorization
(shown in Algorithm 4.3) so that each operation which originally involved a
large dense matrix multiplication is replaced with the application of a sum of
Kronecker product matrices based upon Lemma 4.1.5.

Algorithm 4.1: Block LDLT multifrontal factorization

Input: Symmetric matrix, A, elimination tree, E, and root node, s
Output: a block LDLT factorization of A
1 parallel foreach c ∈ C(s) do Recurse(A,E,c)
2 Us := zeros(|Ls |, |Ls |)
3 foreach c ∈ C(s) do
   
A(Ds , Ds ) A(Ds , Ls ) A(Ds , Ds ) A(Ds , Ls )
:= Uc
A(Ls , Ds ) Us A(Ls , Ds ) Us
−1 −1
4 Ss := A(Ds , Ds )
5 L(Ls , Ds ) := A(Ls , Ds )
−1 H
6 Us := Us − L(Ls , Ds )Ss L(Ls , Ds )

For example, Step 4 of Algorithm 4.3 and Step 2 of Algorithm 4.4 must
each be replaced with an operation of the form
r−1
X
X(Ds , :) := (Yt ⊗ Zt )X(Ds , :),
t=0

where kSs−1 − Yt ⊗ Zt k2 ≤ kSs−1 k2 .

P
t

101
Algorithm 4.2: Structured Kronecker product compression
Input: rsa × rsb matrix G and tolerance, 
r−1 r−1
Output: r × r matrices
P {Yt }t=0 and sa × sb matrices {Zt }t=0 such
that kG − t Yt ⊗ Zt k2 ≤ kGk2
2
1 Form the sa sb × r matrix K such that
K(:, i+jr) = vec(G(isa : (i+1)sa −1, jsb : (j +1)sb −1))
H
2 Compute SVD of K, K = U ΣV
3 Remove every triplet (σ, u, v) such that σ < kAk2 to form the

truncated SVD, K ≈ Û Σ̂V̂ H = r−1 H

P
t=0 (σt ut )vt
4 parallel foreach t = 0 : r − 1 do
5 Form Yt = vec−1 (vt )
6 Form Zt = vec−1 (σut )
7 end

Algorithm 4.3: Block LDLT multifrontal forward solve

Input: block LDLT factorization, J, right-hand side matrix of
width k, X, elimination tree, E, and root node, s
Output: X := D−1 L−1 X
1 parallel foreach c ∈ C(s) do Recurse(J,X,E,c)
2 Zs := zeros(|Ls |, k)
   
X(Ds , :) X(Ds , :)
foreach c ∈ C(s) do := Zc
3 Zs Zs
−1
4 X(Ds , :) := Ss X(Ds , :)
5 Zs := Zs − L(Ls , Ds )X(Ds , :)

Algorithm 4.4: Block LDLT multifrontal backward solve

Input: block LDLT factorization, J, right-hand side matrix of
width k, X, elimination tree, E, and root node, s
Output: X := L−T X
H
1 if A(s) 6= ∅ then X(Ds , :) := X(Ds , :) − L(Ls , Ds ) X(Ls , :)
−1
2 X(Ds , :) := Ss X(Ds , :)
3 parallel foreach c ∈ C(s) do Recurse(J,X,E,c)

102
4.2.1 Mapping a single vector

For the moment, let us suppose that the submatrix X(Ds , :) has only
a single column, which we will simply refer to as the vector b. Then we must
perform the update
r−1
X r−1
X
b := (Yt ⊗ Zt )b = vec(Zt W YtT ),
t=0 t=0

where vec(W ) = b. But this is clearly equivalent to

  
T
r−1 W Y 0
(Yt ⊗ Zt )b = vec  Z0 , · · · Zr−1  ...  ≡ vec(ZE),
X

(4.1)
 
t=0 T
W Yr−1
which we might form in the five steps shown in Algorithm 4.5. Notice that
both the second and fourth steps can be performed with standard algorithms
for parallel matrix-matrix multiplication, such as Scalable Universal Matrix
Multiplication (SUMMA) [109, 127].

Algorithm 4.5: Mapping a vector with a sum of Kronecker prod-

Pr−1
ucts, b := t=0 (Yt ⊗ Zt )b
1 W := vec−1 (b)
2
T
Ẽ := W [Y0T , . . . , Yr−1 ]
3 Shuffle row-panel Ẽ into column-panel E
4 M := [Z0 , . . . , Zr−1 ]E
5 b := vec(M )

4.2.2 Mapping several vectors

Let us now investigate an alternative to Equation (4.1) designed for

simultaneously mapping several vectors via matrix-matrix multiplication. We
will begin by generalizing the vec operator in order to keep our notation com-
pact.

103
Definition 4.2.1. Given an m × qk matrix W , which we may partition as


W = W0 , W 1 , · · · Wk−1 ,

where each Wj is m × q, vec(W, k) denotes the mq × k matrix whose column

with index j is given by vec(Wj ). Furthermore, given an mq × k matrix B,
when the dimensions of the result are clear, we may denote the unique m × qk
matrix W such that vec(W, k) = B as vec−1 (B, k).

We will now let W = vec−1 (B, k) be the unique m × qk matrix such

that vec(W, k) = B. Then, if we define Ej based upon column j of B in the
same manner as in Equation (4.1) and recall that Z = [Z0 , Z1 , . . . , Zr−1 ], we
may write
r−1
X
(Yt ⊗ Zt )B = [vec(ZE0 ), . . . , vec(ZEk−1 )] = vec(ZE, k), (4.2)
t=0

where  
W0 Y0T ··· Wk−1 Y0T
 W0 Y T ··· Wk−1 Y1T 
1
E =  .. , (4.3)
 
.. ..
 . . . 
T T
W0 Yr−1 ··· Wk−1 Yr−1
which is a permuted version of the matrix
 
W0
 W1 
≡ W̃ Ỹ T .


Ẽ =  ..  Y0T Y1T · · · T
Yr−1 (4.4)
 
 . 
Wk−1
Pr−1
We may thus form the update B := t=0 (Yt ⊗ Zt )B in an analogous manner
as in the previous subsection (see Algorithm 4.6). As before, the entire process
involves two dense matrix-matrix multiplications (Steps 2 and 4) and several
permutations.

104
 AlgorithmP4.6: Mapping a matrix with a sum of Kronecker prod-
ucts, B := r−1
t=0 (Yt ⊗ Zt )B
1 W̃ := vec−1 (B, k)
2
T
Ẽ := W̃ [Y0T , . . . , Yr−1 ]
3 Shuffle Ẽ into E
4 M := ZE
5 B := vec(M, k)

4.3 Results
We now reconsider the waveguide model discussed in the previous chap-
ter using 250 × 250 × 250 and 500 × 500 × 500 second-order finite-difference
stencils with frequencies of 18.75 Hz and 37.5 Hz, respectively. Table 4.1
demonstrates that our compression scheme is quite successful for the discrete
Green’s functions (the diagonal blocks) produced within our modified multi-
frontal factorization. In particular, for the 2503 grid, a factor of 16.41 memory
compression was measured for the top-most panel of the waveguide (which
consists of 14 planes), whereas the compression ratio was 12.51 for the 5003
grid. It is perhaps unsurprising that the compression algorithm is less suc-
cessful when applied in an ad-hoc manner to the connectivity between the
supernodes, as these matrices do not correspond to discrete Green’s functions
and therefore do not enjoy any obvious benefits from the (approximate) trans-
lation invariance of the free-space Green’s function. It was also observed (see
Figure 4.3) that the compressed preconditioner behaved essentially the same
as the original preconditioner for the 2503 waveguide example.

105
 original compressed ratio
diagonal blocks 2544 MB (11295 MB) 155.0 MB (903.2 MB) 16.41 (12.51)
connectivity 6462 MB (31044 MB) 1336 MB (7611 MB) 4.836 (4.079)
total 9007 MB (42339 MB) 1491 MB (8513 MB) 6.039 (4.973)

Table 4.1: Compression of the top-panel, which consists of 14 planes, of a 2503

(and 5003 ) waveguide with ten points per wavelength using relative tolerances
of 1e-2 and 5e-2 for the diagonal blocks and connectivity, respectively

100
max kb − Axk2 /kbk2

10−1
10−2
10−3
10−4
b

10−5
0 10 20 30
Iteration

Figure 4.3: Convergence of compressed moving PML sweeping preconditioner

in GMRES(20) for a 2503 waveguide problem, with ten points per wavelength
and relative tolerances of 0.01 and 0.05 for the Krocker product approximations
of the top-left and bottom-left quadrants of each frontal matrix, respectively

4.4 Summary and future work

We have introduced a compression scheme for the sweeping precon-
ditioner motivated by the lossless compression of certain constant-coefficient
discrete Green’s functions in terms of sums of Kronecker products and ex-
plained how to express each of the significant operations in a multifrontal
triangular solve with the compressed frontal trees in terms of dense matrix-
matrix multiplication. A detailed study of how to properly adapt SUMMA to
applying sums of Kronecker products for the typical matrix sizes of our ap-

106
plication will be left as future work. The author would also like to emphasize
that, while several other researchers have investigated hierarchical low-rank
compression schemes for the dense frontal matrices [110, 134], the proposed
scheme is based upon Kronecker-product approximations designed to exploit
the translation invariance of the free-space Green’s function. Another differ-
ence is that the primary goal is to lower the memory requirements rather than
to reduce the computational cost.

While the observed factor of five memory compression is useful, it is

clear from Table 4.1 that most of the benefits come from the compression of
the diagonal blocks of the subdomain multifrontal factorizations (where the
developed theory directly applies). If a similar compression scheme was found
for the off-diagonal blocks, then it would perhaps allow us to solve problems at
twice the frequency as with the standard sweeping preconditioner. It is there-
fore worthwhile to develop theory targeted directly towards the interactions
between supernodes.

107
 Chapter 5

Contributions and future work

5.1 Contributions
The major contributions detailed throughout the previous chapters es-
sentially boil down to:

• high-performance multifrontal triangular solves,

• an efficient parallel moving-PML sweeping preconditioner,

• a compressed moving-PML sweeping preconditioner, and

• a high-performance algorithm for applying compressed fronts.

We will now briefly summarize each of these items and then provide pointers
to the source code and documentation for the supporting software.

5.1.1 High-performance multifrontal triangular solves

Two new high-performance multifrontal triangular solve schemes were

proposed in Chapter 2, one of which is an extension of selective inversion to
large amounts of right-hand sides, and the other avoids selective inversion and
exploits the fact that, while dense triangular solves with a few right-hand sides
are not scalable, dense triangular solves with many right-hand sides are. The

108
fundamental idea behind each of these approaches is to eschew traditional one-
dimensional distributions for each supernode’s portion of the right-hand side
vectors in favor of two-dimensional data distributions which can be used within
scalable dense matrix-matrix multiplication and triangular solve algorithms.
Both of these approaches will be of significant interest for applications which
require the solution of large numbers of linear systems with the same sparse
matrix (especially frequency-domain seismic inversion).

5.1.2 Parallel moving-PML sweeping preconditioner

A parallelization of the moving-PML sweeping preconditioner was pre-

sented in Chapter 3 and efficiently applied to several large-scale analytical
models, as well as a realistic seismic model (Overthrust [4]) using several thou-
sand cores. The main ideas were:

• ensuring that the subdomain multifrontal solves were scalable, for in-
stance, through the usage of selective inversion,

• distributing right-hand sides in a manner which conforms to the dis-

tributed subdomain multifrontal factorizations in order to avoid unnec-
essary communication before and after each subdomain triangular solve,
and

• making use of a scalable scheme for indefinite multifrontal factorization

which does not needlessly sacrifice performance in the name of numerical
stability.

109
5.1.3 Compressed moving-PML sweeping preconditioner

A new Kronecker-product compression scheme for the fronts of the sub-

domain auxiliary problems was theoretically motivated through the translation
invariance of the underlying free-space Green’s function and a significant com-
pression ratio was demonstrated for a large waveguide model. In particular,
it was shown that translation invariance is only required in the direction nor-
mal to the half-space boundary in order to apply mirror-imaging techniques to
yield a compression scheme for the half-space Green’s function. This compres-
sion scheme was shown to allow for the replacement of the diagonal blocks of
the sweeping preconditioners subdomain multifrontal factorizations with the
sum of a small number of Kronecker product matrices.

5.1.4 High-performance compressed front applications

It was shown that, through the usage of the identity

(Y ⊗ Z)x = vec(ZXY T ),

where the vec operator concatenates the columns of its input matrix and X is
defined such that vec(X) = x, that the frontal matrices compressed as a sum
of a small number of Kronecker products may be applied to (sets of) vectors
using scalable algorithms for dense matrix-matrix multiplication. The result
is that, after the Kronecker product compression, standard multifrontal trian-
gular solves need only be slightly modified in order to perform the necessary
linear transformations directly with the compressed data.

110
5.1.5 Reproducibility

Each of the major software efforts involved in the research behind this
dissertation is available under an open-source license along with a significant
amount of documentation:

• Parallel Sweeping Preconditioner (PSP) is available at

http://bitbucket.org/poulson/psp,

• Clique is available at http://bitbucket.org/poulson/clique,

• Elemental is available at http://code.google.com/p/elemental, and

• Madagascar is available at http://ahay.org.

Each of these packages is actively used by a number of researchers scattered

throughout the world, and significant efforts have been expended in order to
make all published claims easily repeatable.

5.2 Future work

While many significant contributions were made in this thesis, there
are invariably a number of lines of future work. Several of the areas which the
author believes to be the most important are listed here:

• Performing large-scale experiments to measure the benefits of the new

high-performance triangular solve algorithms.

• Performing a careful stability analysis of selective inversion.

111
 • Extending the current implementation of the parallel sweeping precon-
ditioner to more sophisticated discretizations for both time-harmonic
Maxwell’s and linear elastic equations.

• Making use of an appropriate scheme for choosing a priori profiles for

the Perfectly Matched Layers based upon the material coefficients.

• Performing a detailed study of distributed Kronecker-product based matrix-

matrix multiplication.

• And, perhaps most importantly, fundamental research should be con-

ducted towards effectively solving time-harmonic wave equations which
are close to resonance (e.g., cavity problems).

Finally, as was mentioned in the introductory chapter, one of the main ap-
plications of this work will be seismic inversion. As these techniques mature,
the author will begin investigating their incorporation into effective inversion
procedures.

112
Appendices

113
 Appendix A

Finite-dimensional spectral theory

The purpose of this appendix is to provide a brief introduction to finite-

dimensional spectral theory, especially in preparation for discussions of Krylov
subspace methods in Appendix B. Arguably the most useful tool in under-
standing in and analyzing matrices (and, more generally, linear operators) is
the notion of a spectrum, which, informally, is the set of scalars ξ associated
with a linear operator A such that A − ξI does not have a well-defined inverse.
Of course, it is important to specify precisely what we mean by scalars, and so,
for the remainder of this document, all of our analysis will take place within
the complex field, C. The reason for this generality is two-fold:

• the fundamental theorem of algebra yields the existence of n roots to

polynomials of degree n in C, but there is no analogue in the set of real
numbers, R, and

• the sweeping preconditioner applies to matrices which are complex by

construction.

A proof of the fundamental theorem of algebra is beyond the scope of this dis-
sertation, and so we will simply state it without proof and point the interested
reader to [117] for a concise proof based upon Liouville’s theorem, or to [30]
for a more elementary argument based upon the minimum-modulus principle
which dates back to the amateur mathematician, Jean-Robert Argand [7].

114
 The motivating ideas of this appendix are essentially a blend of [14],
[74], [58], and [67]. Like [14], we prove the existence of eigenpairs without
resorting to determinants,1 whereas our proof of the existence of a shared
eigenvector among two commuting matrices is due to [74], which leads to a
beautifully simple (and seemingly nonstandard) proof of the spectral theorem
for normal matrices. Our proof of the singular value decomposition is essen-
tially identical to that of [58], and our debt to [67] and primarily stylistic.

The basic outline of this appendix is as follows:

1. an introduction of vector spaces and matrices,

2. a proof of the existence of eigenpairs via the introduction of Krylov

subspaces,

3. a proof of the existence of the Schur decomposition (as well as for pairs
of commuting matrices) and of the singular value decomposition,

4. the specialization of Schur decompositions to normal matrices (the spec-

tral decomposition) and its relation to the SVD, and, finally,

5. a discussion of generalized eigenspaces, the Cayley-Hamilton theorem,

and the class of diagonalizable matrices.

A.1 Vector spaces and matrices

In order to avoid getting sidetracked during the introduction of a spec-
trum, we will first define the basic terminology which is used for describing

1
via an argument based upon Krylov subspaces, despite the fact that the term Krylov
never appears in [14]

115
vector spaces. Ideally the reader is already intimately familiar with the con-
cept of linear independence and the notion of a basis of a vector space, but their
definitions will be provided for the sake of completeness. However, we will not
list all of the properties required for a proper definition of a vector space or
provide a proof of the finite-dimensional Riesz representation theorem. Please
see [67] for a detailed discussion of this background material.

Definition A.1.1 (complex vector space). The symbol Cn will be used to

denote the set of all complex vectors of length n. For each n ≥ 1, Cn is called
a vector space, as, in addition to various other properties inherited from C, for
any scalars α, β ∈ C and vectors x, y ∈ Cn , the linear combination αx + βy is
also a member of Cn .

Definition A.1.2 (linear independence). A set of k vectors, say {xj }k−1 n

j=0 ⊂ C ,

is called linearly independent if no nontrivial combination of its members is

equal to the zero vector. More specifically, {xj }k−1
j=0 is linearly independent if

k−1
X
αj xj 6= 0
j=0

for every set of coefficients {αj }k−1

j=0 ⊂ C which has at least one nonzero entry.

Notice that this is equivalent to the statement that no member of the set of
vectors can be written as a linear combination of the others.

Definition A.1.3 (subspace). A (linear) subspace of the vector space Cn is

any set W ⊂ Cn such that, for any w1 , w2 ∈ W and α, β ∈ C, αw1 + βw2 ∈ W.
Notice that {0} is a subspace of Cn .

Definition A.1.4 (basis). Any set of vectors {xj }k−1 n

j=0 ⊂ C which can be

combined to form every member of some subspace W ⊂ Cn is called a basis

116
for that subspace. More specifically, {xj }k−1
j=0 is a basis for W if, for every

w ∈ W, there exist coefficients {αj }k−1

j=0 ⊂ C such that

k−1
X
w= αj xj .
j=0

Definition A.1.5 (span). We say that a basis for a subspace spans that
subspace, and, for any arbitrary set of vectors {xj }k−1 n
j=0 ⊂ C , we may define

the set
k−1
X
span {xj }k−1
j=0
n
= {x ∈ C : x = αj xj for some {αj }k−1
j=0 ⊂ C}. (A.1.1)
j=0

It is easy to see that this set is in fact a linear subspace of Cn . We will also
occasionally use the shorthand span A for denoting the span of the columns of
a matrix A.

Definition A.1.6 (standard basis vectors). The j’th standard basis vector,
ej ∈ Cn , is defined component-wise as being zero everywhere except for entry
j, which is equal to one. Clearly {ej }n−1 n
j=0 spans C .

Definition A.1.7 (dimension). The dimension of a subspace W ⊂ Cn is the

minimum number of members of W which may span W, and it is denoted by
dim W. We also call such a set of vectors a minimal basis.

Proposition A.1. For every subspace W ⊂ Cn , at most dim W linearly in-

dependent vectors may be chosen from W.

Proof. Please see [67] for an inductive proof.

Corollary A.2. If dim W members of W are linearly independent, then they

span W.

117
Proof. Suppose not. Then there exists a member of W which is independent
of the original d vectors, which implies a set of d + 1 linearly independent
vectors, which contradicts the previous theorem.

Definition A.1.8 (linear transformation,linear functional). Any function A :

Cn → Cm is called linear if, for any scalars α, β ∈ C and vectors x, y ∈ Cn ,

A(αx + βy) = αAx + βAy.

When m = 1 (i.e., the result lies in C), we will call such a transformation a
linear functional.

Definition A.1.9 (inner product). For any x, y ∈ Cn , we define their (Eu-

clidean) inner product as
n−1
X
(y, x) = ηj ξj , (A.1.2)
j=0

where ξj and ηj are respectively the j’th entries of x and y. The inner product
(y, x) will also often be denoted by y H x. It is easy to see that the Euclidean
inner product satisfies the following properties for any α, β ∈ C and x, y ∈ Cn :

1. (conjugate symmetry), (y, x) = (x, y)

2. (sesquilinearity), (βy, αx) = β(y, αx) = αβ(y, x), and

3. (positive-definiteness), (x, x) ≥ 0, with equality if and only if x = 0.

Theorem A.3 (Riesz representation theorem [67]). Every linear functional

f : Cn → C can be represented as the inner-product of its argument with a
particular vector, say vf :

f (x) = (yf , x) = yfH x. (A.1.3)

We say that the Riesz representation theorem identifies each linear functional
on Cn with a vector in Cn .

118
Proof. Please see [67] for details.

Remark A.1.1. Every linear transformation A : Cn → Cm can be decom-

posed into m linear functionals, say {fi }m−1 n
i=0 , where, for any x ∈ C ,
 
f0 (x)
 f1 (x) 
Ax =  .
 
..
 . 
fm−1 (x)
Definition A.1.10 (matrices). Every linear transformation A : Cn → Cm
can be identified with an m × n array of numbers called a matrix, where each
row of the matrix is the conjugate of the Riesz representation of the linear
functional fi . Then, y = Ax implies that
X
ηi = αi,j ξj ,
j

where αi,j refers to the (i, j) entry of the matrix associated with A, ξj refers to
the j’th entry of ξ, and ηi refers to the i’th entry of y. From now on, we will
identify all finite-dimensional linear transformations with their corresponding
matrix.

Definition A.1.11. We will follow the convention of [74] and denote the set
of m × n matrices with complex coefficients as Mm,n , and the set of n × n
(square) matrices as Mn .

Definition A.1.12 (transpose and Hermitian-transpose). We define the trans-

pose of A ∈ Mm,n entry-wise as the n × m matrix

AT (i, j) = A(j, i). (A.1.4)

Likewise, we define the Hermitian-transpose of A, also known as the conjugate-

transpose and adjoint of A, as the n × m matrix

AH (i, j) = A(j, i).

119
Proposition A.4. For any square matrix A ∈ Mn ,

(y, Ax) = (AH y, x)

for all x, y ∈ Cn . This is the usual setting in mind when referring to AH as

the adjoint of A.

Proof. Once we recognize that the (i, j) entry of a matrix A is given by

(ei , Aej ), the entry-wise definition of the adjoint implies that (y, Ax) = (AH y, x)
at least holds when x and y are both standard basis vectors. We can then ex-
tend this result to arbitrary x and y in Cn by representing each as a sum
of standard basis vectors (and making use of the sesqui-linearity of the inner
product).

Proposition A.5. For any square matrices A, B ∈ Mn ,

(AB)H = B H AH , and (AB)T = B T AT . (A.1.5)

Proof. We may apply the previous proposition with standard basis vectors in
order to show that both equations hold entrywise. For example,

(ei , (AB)H ej ) = ((AB)ei , ej ) = (Bei , AH ej ) = (ei , B H AH ej ),

which shows the first result. A similar argument holds for (AB)T .

A.2 Existence of the spectrum

Definition A.2.1 (singular matrix). A matrix A ∈ Mn is called singular if
there exists some nonzero vector x ∈ Cn such that Ax = 0.

120
Definition A.2.2 (kernel). Given a matrix A ∈ Mn , the set of all vectors
x ∈ Cn such that Ax = 0 is called the kernel or null space of A. That is,

ker A = {x ∈ Cn : Ax = 0}.

Proposition A.1. The kernel of any matrix A ∈ Mn is in fact a subspace of

Cn , and, if it is nontrivial, then A is not injective, and therefore not invertible.

Proof. Suppose that α, β ∈ C and x, y ∈ ker A. Then A(αx + βy) = αAx +

βAy = 0. For the second claim, we recognize that, if Ax = 0, then A(αx+z) =
Az for any α ∈ C and z ∈ Cn .

Definition A.2.3 (spectrum). The spectrum of a matrix A ∈ Mn , denoted

by σ(A), is the set of all scalars λ ∈ C such that A − λI is singular.2

Definition A.2.4 (eigenvalue,eigenvector,eigenpair). Suppose that λ is a mem-

ber of the spectrum of some A ∈ Mn . Then, since (A − λI)x = 0 for some
x ∈ Cn , we may trivially rearrange this equation into the form

Ax = λx,

which shows that, in some sense, A behaves like the scalar λ when acting on
the vector x. For this reason, each λ ∈ σ(A) is called an eigenvalue of A, and
its corresponding vector, x, is called an eigenvector. It is also common to refer
to the pair (λ, x) as an eigenpair of a matrix.

2
This definition must become significantly more technical when considering infinite-
dimensional linear operators. Roughly speaking, the spectrum of a linear operator A is
the set of all ξ ∈ C such that the resolvent operator, (ξI − A)−1 , is not well-defined and
bounded. See [84] for details.

121
Remark A.2.1. We will now follow the approach of [14] in showing the crucial
fact that every complex finite-dimensional square matrix, A ∈ Mn , where
n ≥ 1, has at least one eigenpair. Along the way, we will introduce the
extremely-important notion of a Krylov subspace, which plays a central role in
the body of this document.

Lemma A.2. For any two scalars ξ1 , ξ2 ∈ C, and any matrix A ∈ Mn ,

(A − ξ1 I)(A − ξ2 I) = (A − ξ2 I)(A − ξ1 I).

More generally, for any finite set of shifts {ξj }k−1

j=0 ⊂ C, the ordering of the

terms in the product

k−1
Y
(A − ξj I)
j=0

is irrelevant.

Definition A.2.5 (commuting matrices [74]). Any two matrices A, B ∈ Mn

such that AB = BA are said to commute.

Definition A.2.6. The polynomials of degree n, Pn , are the functions p : C →

C which can be expressed as
n
X
p(z) = αj z j ,
j=0

where each αj ∈ C is called a coefficient of the polynomial p.

Definition A.2.7 (Krylov subspace). The Krylov subspace of matrix A ∈ Mn

and vector r ∈ Cn of order k is defined as

Kk (A, r) = span {r, Ar, . . . , Ak−1 r}, (A.2.6)

which is precisely the space of all polynomials of degree less than or equal to
k − 1 of A acting on r, which we denote by Pk−1 (A)r.

122
Lemma A.3 (Krylov lemma). For any matrix A ∈ Mn and vector x ∈ Cn ,
there exists a polynomial p ∈ Pn such that p(A)x = 0.

Proof. Consider the Krylov subspace

Kn+1 (A, x) = span {x, Ax, . . . , An x} = Pn (A)x

for some arbitrary vector x ∈ Cn . Since we know that at most n linearly

independent vectors may be chosen from Cn (Lemma A.1), there exists some
linear combination of the vectors {x, Ax, . . . , An x} which is equal to zero, i.e.,
there exists some nonzero p ∈ Pn such that p(A)x = 0.

Theorem A.4 (fundamental theorem of algebra [7]). Every polynomial p of

degree n ≥ 1 can be factored as

p(z) = γn (z − ξ0 )(z − ξ1 ) · · · (z − ξn ),

where, for each ξj ∈ C, p(ξj ) = 0. Each ξj is called a root of the polynomial

p.

Proof. See [117] for details.

Theorem A.5 (existence of an eigenpair [14]). Every matrix A ∈ Mn , where

n ≥ 1, has at least one eigenpair.

Proof. Let r ∈ Cn and let p ∈ Pn be the polynomial guaranteed by Lemma A.3

such that p(A)r = 0. We can now employ the fundamental theorem of algebra
(Theorem A.4) in order to assert the existence of the factorization
n
Y
p(z) = γn (z − ξj ),
j=0

123
where γn 6= 0, and, when combined with Lemma A.2, we see that
n
!
Y
p(A)r = γn (A − ξj I) r = 0,
j=0

where the ordering of the factors A − ξj I is irrelevant. Let us therefore simply

write
p(A)r = γn (A − ξn I) · · · (A − ξ1 I)(A − ξ0 I)r = 0,

and consider evaluating the middle expression from right to left, e.g., by con-
sidering the sequence {rj }n+1
j=0 , where r0 ≡ r and

rj ≡ (A − ξj−1 I) · · · (A − ξ1 I)(A − ξ0 I)r = (A − ξj−1 I)rj−1 ,

when j ≥ 1. Since we know that r0 is nonzero, and rn+1 = p(A)r = 0, let rq be

the first term which is zero (as all subsequent terms must also be zero). Then
(A − ξq−1 I)rq−1 = 0 even though rq−1 6= 0, and so, by definition, (ξq−1 , rq−1 )
is an eigenpair of A.

Remark A.2.2. With just a few more lemmas at our disposal, we can actually
prove that, if two matrices commute, they share a common eigenvector. In
fact, this is true for any family of commuting matrices [74], but we will simply
stick to pairs of commuting matrices.

Definition A.2.8 (invariant subspace). Any subspace W ⊂ Cn such that

A(W) ⊂ W is called an invariant subspace of A ∈ Mn .

Remark A.2.3. Note that the one-dimensional space spanned by an eigen-

vector of A is an invariant subspace, as is the trivial subspace, {0}. In fact,
we will now show that every non-trivial invariant subspace of A contains an
eigenvector of A.

124
Lemma A.6. If a subspace W ⊂ Cn is invariant under A ∈ Mn , then there
exists an eigenvector of A within W.

Proof. Let W be a matrix whose columns form a minimal basis for the in-
variant subspace W. Then, for each w ∈ W, we may express w as a linear
combination of the columns of W , i.e., w = W y. Since W is invariant, each
column of AW therefore lies within W, and so we may find a matrix Z such
that
AW = W Z.

Now, if we apply Theorem A.5 to the matrix Z, we know that Zy = λy

for some y of appropriate dimension. Then, we may multiply both sides of
AW = W Z by y to find that

W Zy = W (λy) = AW y,

which shows that W y is an eigenvector of A with eigenvalue λ.

Lemma A.7. If a matrices A, B ∈ Mn commute, then, for any finite-degree

polynomial p, A and p(B) also commute.

Proof. Since, for any k ≥ 1,

AB k = (AB)B k−1 = (BA)B k−1 = B(AB k−1 ),

we may move a single power of B from the right to the left side of A as many
times as we like, which shows that A commutes with every monomial of B,
say B k . Since polynomials are merely linear combinations of monomials, the
result follows.

125
Lemma A.8. If, given some matrix A ∈ Mn and vector x ∈ Cn , the vectors
{x, Ax, . . . , Ak−1 x} are linearly dependent, then Kk (A, x) is invariant under
A.

Proof. Since the result is trivial for k = 1, 2, suppose that we have shown the
result for k < j. Then, if {x, Ax, . . . , Aj−1 x} is linearly dependent, then there
are two possibilities:

• {x, Ax, . . . , Aj−2 x} is linearly dependent, which implies that Kj−1 (A, x)
is invariant, which shows that Kk (A, x) is invariant for all k ≥ j, or

• Aj−1 x can be expressed as a linear combination of {x, Ax, . . . , Aj−2 x},

which implies that A(Aj−1 x) ∈ Kj (A, x) so that A(Kj (A, x)) ⊂ Kj (A, x).

In both cases, if {x, Ax, . . . , Aj−1 x} is linearly dependent, Kj (A, x) is invariant

under A, and so the proof holds by induction.

Theorem A.9 (existence of a shared eigenpair [74]). If the matrices A, B ∈

Mn commute, then they share a common eigenvector.

Proof. We first apply Theorem A.5 to yield an eigenvector x of A. Since

{x, Bx, . . . , B n+1 x} is linearly dependent, Lemma A.8 shows that Kn+2 (B, x) is
invariant under B, and thus Lemma A.6 provides an eigenvector y ∈ Kn+2 (B, x)
of B. Since every member of Kn+2 (B, x) may be written as p(B)x for some
p ∈ Pn+1 , Lemma A.7 shows that p(B)x is also an eigenvector of A.

A.3 Schur decompositions

We will now introduce the classes of triangular and unitary matrices,
and then show that every matrix A ∈ Mn can be decomposed into a product of

126
such matrices. We will then show that this decomposition provides a significant
amount of insight into the spectrum of A.

Definition A.3.1 (triangular matrices). A matrix L ∈ Mn is called lower-

triangular if it is zero below its diagonal, i.e., if i > j implies that L(i, j) = 0.
Likewise, a matrix U ∈ Mn is called upper-triangular if it is zero above its
diagonal, i.e., i < j implies that U (i, j) = 0.

Lemma A.1. For any triangular matrix T ∈ Mn , λ ∈ σ(T ) if and only if

there is a diagonal entry of T equal to λ.

Proof. Since T − λI has a zero diagonal entry if and only if T has a diagonal
value equal to λ, we need only show that a triangular matrix is singular if and
only if it has a zero diagonal entry.

Suppose that a triangular matrix T has no nonzero diagonal entries, and

that T x = 0 for some vector x ∈ Cn . Then we may reverse this multiplication
process, through a process known as backward elimination, in order to find
that x is identically zero. Notice that backward elimination is only well-defined
when the diagonal of T does not contain a zero. In particular, let us partition
the equation T x = 0 into
    
T1,1 t1,2 x1 0
=
0 τ2,2 χ2 0

in order to see that χ2 = 0. But we then recursive application of this procedure

to T1,1 x1 = 0 shows that x = 0.

Now suppose that the j’th diagonal entry of T is zero. Then the first j
columns of T lie within the j − 1-dimensional subspace span {e0 , e1 , . . . , ej−2 },
and thus, by Lemma A.1, they cannot be linearly independent. We may

127
therefore define a vector x, which is only nonzero in its first j components
such that T x = 0.

Definition A.3.2 (two-norm). The two-norm of the vector x ∈ Cn , also

known as the Euclidean norm, is defined as
p
kxk2 = (x, x), (A.3.7)

where the square-root can be interpreted in the usual sense because (x, x) ≥ 0,
with equality if and only if x = 0. The two-norm is perhaps the most natural
way to define the length of a vector.

Definition A.3.3 (unitary matrix). A matrix Q ∈ Mn is called unitary if,

for every x, y ∈ Cn , (x, y) = (Qx, Qy); as a special case, we also have that
kQxk2 = kxk2 . The interpretation of the matrix Q as having unit-magnitude
(hence the name unitary) can be made rigorous using the following definition.

Proposition A.2. The product of two unitary matrices is also unitary.

Proof. Let U, V ∈ Mn both be unitary. Then, for any x, y ∈ Cn ,

(x, y) = (V x, V y) = (U (V x), U (V y)).

Definition A.3.4 (operator two-norm). The operator, or induced two-norm

of the matrix A ∈ Mn is defined as

kAk2 = max
n
kAxk2 , (A.3.8)
x∈C ,
kxk2 =1

where the maximum can be shown to exist due to the Heine-Borel theo-
rem [106], but we will avoid diving into real analysis in order to explain this

128
technical detail. This operator norm of A can be geometrically interpreted
as the maximum length of any transformed vector which originated on the
surface of the unit-ball in Cn .

Proposition A.3. For any unitary matrix Q, kQk2 = 1.

Proof. By definition of the operator two-norm,

kQk2 = maxn kQxk2 ,

x∈C
kxk2 =1

and, by the definition of unitary matrices, kQxk2 = kxk2 . Since the maximum
is only taken over the unit ball, where kxk2 = 1, the result follows.

Proposition A.4. A matrix is unitary if and only if its adjoint is its inverse.

Proof. We will first show that, if Q is unitary, QH Q = I. The (i, j) entry of

QH Q is given by (ei , QH Qej ), and, by Proposition A.4, it is also equivalent to
(Qei , Qej ). By the definition of a unitary transformation, (Qei , Qej ) = (ei , ej ),
which satisfies the desired properties. A similar argument shows that QQH =
I.

Now suppose that AH A = I for some matrix A ∈ Mn . Since (ei , ej ) =

(ei , AH Aej ) = (Aei , Aej ) for every pair of standard basis vectors, ei and ej ,
by the sesquilinearity of the inner product we find that (x, y) = (Ax, Ay) for
every x, y ∈ Cn , which shows that A is unitary by definition.

Definition A.3.5. A set of vectors {xj }k−1 n

j=0 ⊂ C is called orthonormal if

(xi , xj ) = 0 when i 6= j, and (xi , xj ) = 1 when i = j. Notice that we have just

shown that the rows and columns of a unitary matrix are both orthonormal.

129
Proposition A.5. Every subspace W ⊂ Cn has an orthonormal basis of size
dim W.

Proof. Let {wj }d−1

j=0 ⊂ W be a linearly independent basis of W, where d =

dim W. Then, we will construct an orthonormal basis through a process known

as Gram-Schmidt orthogonalization. We begin by putting q0 = w0 /kw0 k2 ,
which must have unit norm. Then we may define q̃1 = w1 − q0 (q0 , w1 ), which is
easily verified to be a nonzero vector (the original basis is linearly independent)
which is orthogonal to w0 . Since q̃1 is nonzero, we may normalize it to construct
q1 = q̃1 /kq̃1 k2 so that {q0 , q1 }1j=0 is orthonormal.

Now, suppose that we have constructed an orthonormal set {qj }k−1

j=0 ,

where k < d. Then we may define

k−1
X
q̃k = wk − qj (qj , wk ),
j=0

which effectively removes all of the components of q̃k in the directions of the
previous orthonormal set. Since the linear independence of the {wj }d−1
j=0 basis

proves that q̃k is nonzero, we may normalize it in order to construct a new

orthonormal set, {qj }kj=0 , which completes the proof by induction.

Definition A.3.6 (orthogonal complement). The orthogonal complement for

a subspace W ⊂ Cn is defined as

W ⊥ = {x ∈ Cn : ∀ w ∈ W, (w, x) = 0}.

Proposition A.6. For any subspace W ⊂ Cn , W ⊥ is also a subspace of Cn .

Proof. Let x, y ∈ W ⊥ and α, β ∈ C. Then (w, αx + βy) = α(w, x) + β(w, y) by

the sesquilinearity of the inner product, and, because x and y lie in W ⊥ , each
of these terms is identically zero, and thus αx + βy also lies within W ⊥ .

130
Proposition A.7. For any subspace W ⊂ Cn ,

dim W + dim W ⊥ = n,

and each x ∈ Cn may be uniquely decomposed as x = w + z, where w ∈ W

and z ∈ W ⊥ . We may say that Cn is a direct sum of W and its orthogonal
complement, which we denote by

W ⊕ W ⊥ = Cn .

Proof. Let W = {wj }d−1 k−1

j=0 ⊂ W and Z = {zj }j=0 ⊂ W
⊥
respectively be
orthonormal bases for W and W ⊥ , where d = dim W and k is an as-of-yet
undetermined number.

Since each zi ∈ W ⊥ , (zi , wj ) = (wj , zi ) = 0 for every pair (i, j). Because
each of these sets is orthonormal, the combined set is orthonormal as well (and,
of course, also linearly independent). This implies that dim W ⊥ ≤ n − d.

Now suppose that dim W ⊥ < n − d. Then, by Proposition A.1, there

exists some vector x ∈ Cn which is orthogonal to both W and W ⊥ . But this
is a contradiction, as, by definition, any vector which is orthogonal to W is a
member of W ⊥ . We have thus shown that dim W ⊥ = n − d.

In order to show the second property, for x ∈ Cn , set

d−1
X
y= wj (wj , x),
j=0

and
n−d−1
X
z= zj (zj , x),
j=0

so that y ∈ W, z ∈ W ⊥ . But then we can see that x = y +z, as y +z = QQH x,

where Q is the unitary matrix with columns equal to (w0 , . . . , wd−1 , z0 , . . . , zn−d−1 ).

131
 In order to show uniqueness, let x = ŷ + ẑ be another such decomposi-
tion, so that y − ŷ = ẑ − z. Since the left side of the equation lies in W, and
the right side lies in W ⊥ , it can only be satisfied when both y − ŷ and z − ẑ
are zero.

Corollary A.8. Every vector x ∈ Cn of unit length can be extended into an

orthonormal basis for Cn .

Proof. If we define W = span x, then Lemma A.7 shows that there exists
an orthonormal basis of dimension n − 1 for x⊥ , which implies that, when
combined with x, is an orthonormal basis for all of Cn .

Theorem A.9 (QR decomposition). Every matrix A ∈ Mm,n has a QR de-

composition,
A = QR,

where Q ∈ Mm is unitary and R ∈ Mm,n is zero in every entry (i, j) such that
i > j (and is therefore referred to as quasi upper-triangular).

Proof. We will construct Q from a Gram-Schmidt orthogonalization process

similar to the one used in Proposition A.5, but when we reach the j’th column
of A, if it is linearly dependent on the previous columns, we will choose its
corresponding column of Q from the orthogonal complement of the span of
the previous columns. If we continue this process until the first min{m, n}
columns have been processed, forming some matrix Q̂ ∈ Mm,min{m,n} with
orthonormal columns with same span as the columns of A, then we may apply
Proposition A.7 in order to extend it into a unitary matrix Q ∈ Mm . But
then, for any j < min{m, n}, column i of Q is orthogonal to column j of A if
i > j. This is precisely equivalent to the statement that QH A = R for some
quasi upper-triangular matrix R, which shows the result.

132
Remark A.3.1. The following theorem is being presented in somewhat of a
different context than normal. In particular, please keep in mind that we have
only proven that the spectrum of A ∈ Mn is non-empty. However, once we es-
tablish the following theorem (via an admittedly tedious inductive argument),
we will immediately be able to say much more about the spectrum of A.

Theorem A.10 (Schur decomposition [74]). Every square matrix A ∈ Mn

can be decomposed as
A = QT QH , (A.3.9)

where Q is unitary and T is upper triangular.

Proof. Due to Theorem A.5, there exists an eigenpair (λ0 , x0 ) of A, so that

Ax0 = λ0 x0 , and, without loss of generality, we may choose x to have unit
norm. Then Corollary A.8 implies that we may extend x0 into an orthonormal
basis for Cn , and so we may define the unitary matrix Q0 = (x0 , X0 ) via this
extension. Then
 
λ r0
QH
0 (AQ0 ) = QH
0 (λx0 , AX0 ) = ,
0 B0

where the zero in the bottom-left of the last term is due to the orthonormality
of the columns of Q0 . We can of course rewrite this equation as
 
λ r0
A = Q0 QH0 ,
0 B0

and it is natural to attempt to recursively apply our approach to B0 , as Propo-

sition A.2 shows that the product of unitary matrices is also unitary. The
reader need only recognize that
   H  
I 0 RT L RT R U I 0 RT L RT R
= ,
0 U 0 RBR 0 U 0 U RBR U H

133
  
I 0
and that U unitary implies that is also unitary in order to complete
0 U
the proof.

Corollary A.11. Given a Schur decomposition QT QH of a matrix A ∈ Mn ,

the diagonal values of the upper-triangular matrix T are the eigenvalues of
A. More precisely, λ ∈ σ(A) if an only if there is some index j such that
T (j, j) = λ.

Proof. Since A − λI = QT QH − λI = Q(T − λI)QH , λ ∈ σ(A) if and only if

λ ∈ σ(T ). But Lemma A.1 shows that λ ∈ σ(T ) if and only if λ lies on the
diagonal of T .

Remark A.3.2. We conclude this section with a slight generalization of The-

orem A.10 with important consequences which will be discussed in the next
section.

Theorem A.12 (simultaneous Schur decomposition [74]). Every pair of com-

muting matrices A, B ∈ Mn has a simultaneous Schur decomposition

A = QSQH , B = QT QH ,

where Q is unitary and both S and T are upper-triangular.

Proof. We may essentially repeat the proof of Theorem A.10, but, at each step,
we may apply Theorem A.9 instead of Theorem A.5 in order to simultaneously
reduce both matrices to triangular form.

Theorem A.13 (singular value decomposition [58]). Any matrix A ∈ Mm,n

can be decomposed as
A = U ΣV H ,

134
where U ∈ Mm and V ∈ Mn are unitary, and Σ ∈ Mm,n is at most nonzero in
the first min{m, n} entries of its main diagonal, and these non-negative entries
are called the singular values. The columns of U and V are respectively referred
to as the left and right singular vectors. We will denote the j’th diagonal value
of Σ by σj , where 0 ≤ j < min{m, n}.

Proof. We may prove this important decomposition in a manner very similar

to the Schur decomposition, but rather than choosing an eigenpair at each step
of the recursive process, we instead choose a unit-vector which, when linearly
transformed by A, produces a vector of maximal norm. In particular, we begin
by choosing some unit-vector x0 ∈ Cn such that kAx0 k2 = kAk2 (and such a
vector exists due to the Heine-Borel theorem). Then we may choose another
unit-vector y0 such that Ax0 = σ0 y0 , where σ0 = kAx0 k2 = kAk2 , and then
respectively extend x0 and y0 into unitary matrices U0 = (x0 , X0 ) ∈ Mm and
V0 = (y0 , Y0 ) ∈ Mn via Corollary A.8.

We then find that

 
σ0 r0H
U0H AV0 = U0H (σ0 y0 , AX0 ) = ,
0 B0
for some vector r0 ∈ Cn−1 and matrix B0 ∈ Mm−1,n−1 . The key observation [58]
is that    2 
σ0 σ0 + r0H r0
(U0H AV0 ) = ,
r0 B0 r0
which implies that
p
(σ02 + r0H r0 )2 + kB0 r0 k22
q
kAk2 = kU0H AV0 k2 ≥ p ≥ σ02 + r0H r0 ,
σ02 + r0H r0
which contradicts kAk2 = σ0 whenever r0 6= 0. We have therefore shown that
r0 = 0 so that we may write
 
σ0 0
A = U0 V0H ,
0 B0

135
and an inductive technique essentially identical to that of Theorem A.10 com-
pletes the proof.

Definition A.3.7 (range,rank,nullity). The range of a matrix A ∈ Mm,n is

the set of all vectors y ∈ C m such that Ax = y for some x ∈ Cn . We denote it
as
ran A = {y ∈ Cn : Ax = y for some x ∈ Cn }.

The dimension of the range is referred to as the rank of the matrix, while the
dimension of the kernel is called its nullity.

Corollary A.14 (rank-nullity). The dimensions of the range and null space
of a matrix A ∈ Mn add up to n, and, in particular, given a singular value
decomposition A = U ΣV H ,

ran A = ran U Σ, (A.3.10)

and
ker A = ker ΣV H . (A.3.11)

That is to say, the rank is equal to the number of nonzero singular values, and
the nullity is equal to n minus this number.

Proof. The result is essentially immediate now that we have the existence of
the SVD: the number of linearly independent columns in U Σ is the number of
nonzero singular values, and a similar argument holds for ΣV H .

Definition A.3.8 (consistent norm). A matrix norm is said to be consistent

if, for any A, B ∈ Mn ,
kABk ≤ kAkkBk.

136
Proposition A.15 (consistency of two-norm). The operator two-norm is con-
sistent.

Proof. Let A, B ∈ Mn have singular value decompositions A = UA ΣA VAH and

B = UB ΣB VBH , and recognize that kAk2 is the largest singular value of A.
Then

kABk2 = kUA ΣA VAH UB ΣB VBH k2 = kΣA VAH UB ΣB k2

≤ kAk2 kBk2 kVAH UB k2 = kAk2 kBk2 .

A.4 The spectral theorem

Definition A.4.1 (normal matrix). A matrix A is said to be normal if it
commutes with its adjoint, which is to say,

AAH = AH A.

Theorem A.1 (spectral theorem). A matrix A ∈ Mn has a spectral decom-

position,
A = QΛQH , (A.4.12)

where Q is unitary and Λ is diagonal, if and only if A is normal. Since we

may rewrite this equation as
AQ = QΛ,

we see that

1. the j’th column of q and diagonal value of Λ are an eigenpair of A, and

2. the eigenvectors of A span Cn .

137
These observations justify the name spectral theorem.

Proof. Suppose that A is normal. Then A commutes with its adjoint, and
Theorem A.12 shows that we may find a simultaneous Schur decomposition of
A and AH , say A = QSQH and AH = QT QH , where Q is unitary and S and
T are both upper-triangular. Then, we may equate the adjoint of the Schur
decomposition of A with the Schur decomposition of AH in order to find that
S H = T , where S H is lower-triangular and T is upper-triangular. But, this is
only possible when both S and T are diagonal, so that S = T and A = QSQH
is in fact a spectral decomposition of A.

Now suppose that A has a spectral decomposition A = QΛQH . Then

AAH = (QΛQH )(QΛH Q) = Q|Λ|2 QH = (QΛH QH )(QΛQH ) = AH A.

Definition A.4.2 (Hermitian matrix). A matrix A ∈ Mn is said to be Her-

mitian when it equals its adjoint, i.e.,

A = AH .

Since every matrix commutes with itself, every Hermitian matrix is normal.

Corollary A.2 (Hermitian spectral decomposition). A matrix A ∈ Mn can

be decomposed as
A = QΛQH ,

where Q is unitary and Λ is a real, diagonal matrix consisting of the eigenvalues

of A.

Proof. Since A is normal, we know that A = QΛQH for some unitary Q and
diagonal Λ (consisting of the eigenvalues of A), but we do not yet know that
Λ is real. But this is easily shown since A = AH implies that Λ = ΛH , and
since Λ is diagonal, its values must also be real.

138
Remark A.4.1. Given any matrix A ∈ Mm,n , its singular value decompo-
sition, A = U ΣV H , can be interpreted in terms of the Hermitian eigenvalue
decompositions of AAH and AH A. In particular,

AH A = (U ΣV H )H (U ΣV H ) = V (ΣH Σ)V H ,

which shows that the right singular vectors, V , are the eigenvectors of AH A,
and the associated eigenvalues lie along the diagonal of ΣH Σ. Due to the
structure of Σ, the first min{m, n} eigenvalues are equal to σj2 , while the
remaining n − min{m, n} eigenvalues are equal to zero. Likewise,

AAH = U (ΣΣH )U H ,

which also has σj2 as its j’th eigenvalue, for 0 ≤ j < min{m, n}, and the
remaining m − min{m, n} eigenvalues are zero.

A.5 Generalized eigenspaces and matrix polynomials

We still have much left to prove about the spectrum of general matrices.
For instance, though we have shown that a scalar λ is an eigenvalue of a general
matrix A ∈ Mn if and only if it lies on the diagonal of the triangular matrix
produced by its Schur decomposition, we have said nothing about how many
times it appears. We will now provide definitions of the multiplicity of an
eigenvalue in the spirit of [14].

Definition A.5.1 (eigenspace). The eigenspace of an eigenvalue λ of a matrix

A ∈ Mn is the set of all eigenvectors corresponding to the eigenvalue λ,

Wλ = {x ∈ Cn : (A − λI)x = 0}. (A.5.13)

Notice that Wλ = ker (A − λI).

139
Example A.5.1. Unfortunately not all square matrices have eigenspaces which
span their vector space. Consider the matrix
 
0 1
T = ,
0 0

which, because it is upper triangular, has the spectrum σ(T ) = {0}. It is easy
to see that its only eigenspace is
 
1
W0 = span ,
0

which clearly does not span C2 . Thankfully we may loosen our definition of
an eigenspace in order to span the entire vector space.

Definition A.5.2 (generalized eigenvector [14]). A nonzero vector x ∈ Cn

is called a generalized eigenvector for a matrix A ∈ Mn if there exists some
integer k ≥ 1 and some scalar λ ∈ C such that

(A − λI)k x = 0.

Proposition A.1. Every λ ∈ C which admits a generalized eigenvector for a

matrix A ∈ Mn must be a member of the spectrum of A.

Proof. Since the product of nonsingular matrices is again nonsingular, A − λI

must be singular in order for (A − λI)k to map a nonzero vector x to the zero
vector.

Definition A.5.3 (generalized eigenspace). The generalized eigenspace for an

eigenvalue λ ∈ σ(A) is defined as

Gλ = {x ∈ Cn : (A − λI)k x = 0 for some k ≥ 1}.

Notice that, Wλ ⊂ Gλ by definition.

140
Proposition A.2 ([14]). For any A ∈ Mn and λ ∈ σ(A),

Gλ = ker (A − λI)n .

Proof. Since (A − λI)k x = 0 implies that (A − λI)k+1 x = 0, we need only show

that, if (A−λI)k x = 0, where k > n, then we also have that (A−λI)n x = 0. In
order to do so, we will show that, if k is the first index such that (A−λI)k x = 0,
then the vectors {x, (A − λI)x, . . . , (A − λI)k−1 x} are linearly independent,
and the result will follow from Lemma A.1.

Suppose that
k−1
X
αj (A − λI)j x = 0.
j=0

Since each of the terms (A − λI)j , j < k, is nonzero by assumption, showing

that each αj is zero will show that {x, (A − λI)x, . . . , (A − λI)k−1 } are linearly
independent. If we multiply each term in this equation by (A − λI)k−1 , then
(A − λI)j+k−1 x = 0 for each j ≥ 1, which shows that α0 = 0. Similarly, if
we multiply each term by (A − λI)k−2 , we discover that α1 = 0, and we may
repeat the process to find that each αj is 0. Since the vectors {(A−λI)j x}k−1
j=0 }

are linearly independent, we must have that k ≤ n.

Lemma A.3. Suppose that the subspace W ⊂ Cn is invariant under A ∈ Mn ,

and Q is a matrix whose columns form an orthonormal basis for W. Then, if
y is a generalized eigenvector of QH AQ, Qy is a generalized eigenvector of A.

Proof. If y is a generalized eigenvector of QH AQ, then (QH AQ−λI)k y = 0 for

some λ ∈ C and k ≤ n. Then, we may express the last equation term-by-term
(using the binomial formula) as
k  
X k
(QH AQ)k−j (−λ)j y = 0,
j=0
j

141
and, due to the invariance of W under A, QQH AQ = AQ, and QH AQz = 0 if
and only if AQz = 0, so we may write
k  
!
X k
(−λ)j Ak−j Qy = 0.
j=0
j

But this is precisely equal to the statement

(A − λI)k (Qy) = 0,

which shows that Qy is a generalized eigenvector of A.

Definition A.5.4 (Rayleigh quotient). For any matrix A ∈ Mn and matrix

Q ∈ Mn,m with orthonormal columns, the matrix

RA (Q) = QH AQ

is called the Rayleigh quotient of A with respect to Q. In some sense, it is

the projection of A onto the space spanned by the columns of Q. It will be
fundamental to our discussion of Krylov subspace methods in Appendix B.

Lemma A.4 ([14]). For any matrix A ∈ Mn and eigenvalue λ ∈ σ(A), we

may strengthen the rank-nullity theorem for (A − λI)n to say that

ran (A − λI)n ⊕ ker (A − λI)n = Cn .

Proof. We need only show that ran (A − λI)n ∩ ker(A − λI)n = {0} in order to
have the result follow from the rank-nullity theorem. So, suppose x ∈ Cn is in
both the range and kernel of (A − λI)n . Because x is in the range of (A − λI)n ,
there is some y such that (A−λI)n y = x. But then (A−λI)2n y = (A−λI)n x =
0, where the last equality follows since x is in the kernel of (A − λI)n . But
we have then shown that y ∈ Gλ (A), which implies that (A − λI)n y = 0 by
Proposition A.2. But then x = 0, which shows the result.

142
Theorem A.5 ([14]). The generalized eigenvectors of a matrix A ∈ Mn span
Cn .

Proof. The result follows from an inductive argument which decomposes Cn

into the range and kernel of (A−λI)n , via Lemma A.4, and then uses Lemma A.3
in order to show that the generalized eigenvectors of the projection of (A−λI)n
onto its range are also generalized eigenvectors of (A − λI)n . Please see [14]
for a detailed proof.

Theorem A.6. Given a matrix A ∈ Mn , every vector x ∈ Cn can be written

as a unique linear combination of members of the generalized eigenspaces of
A, i.e.,
M
Gλ = Cn .
λ∈σ(A)

Furthermore, if dim Gλ = k and y ∈ Gλ , then (A − λI)k y = 0.

Proof. Please see [14] for details.

Corollary A.7. The eigenvectors of a matrix A ∈ Mn span Cn if and only if

Wλ = Gλ for each λ ∈ σ(A).

Definition A.5.5 (characteristic polynomial). For any A ∈ Mn , define its

characteristic polynomial, pA ∈ Pn , as
Y
pA (z) = (z − λ)dim Gλ .
λ∈σ(A)

Theorem A.8 (Cayley-Hamilton). Every matrix satisfies its characteristic

polynomial, i.e., pA (A) = 0.

143
Proof. Let x be an arbitrary member of Cn , which, by Theorem A.6, we may
decompose as
X
x= gλ ,
λ∈σ(A)

where gλ ∈ Gλ . Then
 
Y
pA (A)x =  (A − λI)dim Gλ  x,
λ∈σ(A)

and each of the matrices in the product commutes. Thus, for each component
gλ of x,  
Y
pA (A)gλ =  (A − ξI)dim Gξ  gλ ,
ξ∈σ(A)

and, if we evaluate (A − λI)dim Gλ gλ first, Theorem A.6 implies that it is zero,

so we see that
X
pA (A)x = p(A)gλ = 0.
λ∈σ(A)

Definition A.5.6 (eigenvalue multiplicity [14]). We say that the algebraic

multiplicity of an eigenvalue λ ∈ σ(A) is the dimension of its corresponding
generalized eigenspace, Gλ , while its geometric multiplicity is the dimension
of its standard eigenspace, Wλ . This is consistent with the usual definitions
based upon the determinant.

Proposition A.9 (diagonalizability). Every matrix A ∈ Mn which can be

decomposed as
A = XΛX −1 ,

where Λ is diagonal, is called diagonalizable. Since AX = XΛ is an expression

of the eigenpairs of A, and X can be inverted if and only if the eigenvectors

144
span Cn , Corollary A.7 shows that A is diagonalizable if and only if the geo-
metric multiplicity of every eigenvalue is equal to its algebraic multiplicity.

Remark A.5.1. We have already shown that all normal matrices are diago-
nalizable (in fact, with a unitary eigenvector matrix).

Proposition A.10. For any polynomial p ∈ Pk and diagonalizable matrix

A ∈ Mn , where A = XΛX −1 ,

p(A) = Xp(Λ)X −1 .

Proof. Since each polynomial is a sum of monomials, we need only consider

(XΛX −1 )k , where the result immediately follows.

Remark A.5.2. It can be shown that the eigenvalues of a matrix A ∈ Mn

depend continuously on the entries of A [74], which implies that an arbitrarily
small perturbation of any non-diagonalizable matrix may be used to separate
every cluster of eigenvalues with geometric multiplicity less than its algebraic
multiplicity. But then every matrix is arbitrarily close to a diagonalizable
matrix (that is to say, the class of diagonalizable matrices is dense in Mn ), so
it is tempting to think that one could always find a suitably accurate eigenvalue
decomposition for a nondiagonalizable matrix. The problem with this idea is
that the resulting eigenvector matrix would almost certainly be so close to
singular as to be numerically useless. For detailed discussions of the subject of
perturbation theory, please see one of the many excellent texts on the subject,
such as [133], [73], and [34].

A.6 Summary
A compact introduction to the classical linear algebra theorems needed
for discussing Krylov subspace methods has been given, with an emphasis

145
placed upon using basic dimensional arguments (in the spirit of [14]) as a re-
placement for cumbersome arguments based upon determinants. The following
appendix makes use of these theorems in order to provide a rigorous intro-
duction to Krylov subspace methods and the Generalized Minimum Residual
method (GMRES), which are both fundamental to prerequisites for Chapters 3
and 4.

146
 Appendix B

Krylov subspace methods

Krylov subspace methods, which are intimately related to the power

method [119], provide an economical means of finding approximate eigenpairs
and, as a sideproduct, also yield algorithms for the approximate solution of
linear systems. Let us recall that power iteration proceeds by computing the
sequence of vectors r, Ar, A2 r, . . . , Ak−1 r in the hope that, for a randomly
chosen starting vector r, Ak−1 r quickly becomes an accurate approximation to
the dominant eigenvector of A.

If we also recall the definition of a Krylov subspace (Definition A.2.7),

then we see that power iteration produces precisely the vectors which generate
the Krylov subspace,

Kk (A, r) = span {r, Ar, . . . , Ak−1 r} = Pk−1 (A)r, (B.1)

where Pk−1 is the space of polynomials of degree k − 1. The difference between

a Krylov subspace eigensolver and the power method is qualitatively simple:
rather than choosing the eigenvector estimate to be the most recent iterate,
Ak−1 r, Krylov eigensolvers find the “best” estimate available from linear com-
binations of the entire set of iterates.

Arguably the most important feature of Krylov subspaces is that, with

a typically modest amount of computation, one arrives at a subspace which is
approximately invariant under A. More specifically, given a matrix V whose

147
columns span a Krylov subspace, it is often the case that the columns of AV
are approximately contained in the same Krylov subspace. It will be helpful
to introduce a few more concepts before attempting to clarify this statement
and its important implications.

It is worth noting that this appendix draws heavily from the content
and style of [119], [107], and [122]. In particular, the focus on Rayleigh quo-
tients and Krylov decompositions is due to [119], and much of the Chebyshev
polynomial approximation discussion is drawn from [107].

B.1 Rayleigh quotients

For an eigenvector estimate v, the corresponding eigenvalue estimate
can be cheaply computed through the Rayleigh quotient,

v H Av
RA (v) = ,
vH v

which simplifies to RA (v) = v H Av when v is a unit vector. In particular,

because RA (v) is continuous, and the Rayleigh quotient of an exact eigenvector
is its exact eigenvalue, we can expect approximate eigenvectors to be mapped
to approximate eigenvalues.

We now generalize the notion of a Rayleigh quotient from translating an

approximate eigenvector into an approximate eigenvalue into a mapping which
translates an approximate invariant subspace into a matrix whose eigenval-
ues approximate those of A. As in Definition A.5.4, for any matrix V with
orthonormal columns, the Rayleigh quotient is equal to

RA (V ) = V H AV.

148
Definition B.1.1 (projection matrix). The projection of a vector x ∈ Cn onto
the space spanned by the (orthonormal) columns of a matrix V is given by

PV x = V V H x,

while the projection of x onto the orthogonal complement (Definition A.3.6)

of the space spanned by the columns of V is equal to

PV ⊥ x = (I − V V H )x.

Notice that, if x ∈ span V , then PV x = x, and PV ⊥ x = 0.

Proposition B.1. If V is a matrix with orthonormal columns such that span V

is an invariant subspace of a matrix A, then

A(V y) = λ(V y) ⇔ RA (V )y = λy.

Proof. We first assume that A(V y) = λ(V y). Then V H AV y = λV H V y = λy

since V has orthonormal columns.

Now assume that V H AV y = λy. Then V V H A(V y) = λ(V y), which

is identically equal to PV (AV y) = λ(V y). Since V y is a member of span V ,
which is invariant under A, A(V y) = PV (AV y).

Due to the continuity of the Rayleigh quotient, we can expect any V

whose range is nearly an invariant subspace of A to map to a matrix whose
eigenpairs yields approximate eigenpairs of A: For any matrix V with or-
thonormal columns, we can set Z = RA (V ) = V H AV and decompose

AV = PV AV + PV ⊥ AV = V Z + PV ⊥ AV. (B.1.2)

149
We can thus characterize how close V is to being an invariant subspace through
the size of the residual matrix,

E ≡ PV ⊥ AV = (I − V V H )AV = AV − V RA (V ). (B.1.3)

Given any eigenpair of the Rayleigh quotient, say Zy = λy, we see that

AV y = λV y + Ey, (B.1.4)

which shows that (λ, V y) can be interpreted as an approximate eigenpair of

A, known as a Ritz pair. The norm of the vector Ey then provides a means of
characterizing the error in the approximation.

Another useful interpretation is that a Ritz pair (λ, V y) is an exact

eigenpair of a perturbation of A, i.e., if we set  = A − EV H , then

ÂV y = λV y, (B.1.5)

where we have simply made use of the fact that V H V = I in order to rearrange
Equation (B.1.4). Since kA − Âk2 = kEV H k2 = kEk2 , we see that, when V
is close to an invariant subspace of A, Ritz pairs are eigenpairs of a matrix
which is close to A.

B.2 Krylov decompositions

It is easy to see that the residual matrix resulting from a basis for a
Krylov subspace is at most rank-one.

Lemma B.1. Using the shorthand notation that Kk ≡ Kk (A, r),

Kk+1 ∩ Kk⊥ = PKk⊥ (Kk+1 ) = span PKk⊥ (Ak r).

150
Proof. By definition,
Kk+1 = span {Kk , Ak r},

and thus
PKk⊥ (Kk+1 ) = span PKk⊥ (Ak r).

Lemma B.2. If a matrix V has orthonormal columns whose span is equal to

the Krylov subspace Kk (A, r), then we may find vectors v and w such that

PV ⊥ AV = vwH ,

where v ∈ span PV ⊥ (Ak r) and kvwH k2 = kwk2 .

Proof. Due to Lemma B.1, each column of PV ⊥ AV is a scalar multiple of

u = PV ⊥ (Ak r). If u = 0, we may set both v and w equal to zero. Otherwise,
we may set v = u/kuk2 and the j’th entry of w equal to PV ⊥ (Avj )H v.

We can now state the main result for this section.

Theorem B.3 (Orthonormal Krylov decomposition). If a matrix V has or-

thonormal columns whose span is equal to the Krylov subspace Kk (A, r), then
we may decompose its image AV as

AV = V Z + vwH ,

where Z = RA (V ), v ∈ span PV ⊥ (Ak r), and kvwH k2 = kwk2 .

Proof. Equation (B.1.2) shows that, for any matrix V with orthonormal columns,
we may decompose AV as V RA (V ) + PV ⊥ AV . Lemma B.2 exploits the fact
that span V is a Krylov subspace in order to yield the necessary rank-one
decomposition of the second term.

151
 The “orthonormal” qualifier hints at the fact that, with a proper gen-
eralization of the Rayleigh quotient, we may lift the requirement that V must
have orthonormal columns [119]. However, for the purposes of this disserta-
tion, orthonormal Krylov decompositions are more than sufficient.

B.3 Lanczos and Arnoldi decompositions

Despite their generality, Krylov decompositions are a relatively modern
means of working with Krylov subspaces [118]. Traditional approaches place
further restrictions on the basis vectors in order to expose additional structure.
For the remainder of this section, we will assume that the Krylov subspace
Kk (A, r) is full-rank and make use of the conventions that K0 (A, r) = {0} and
Vj = [v0 , . . . , vj−1 ].

Lemma B.1. If a matrix V has orthonormal columns chosen such that

vj ∈ span PVj⊥ (Aj r) = Kj+1 ∩ Kj⊥ , (B.3.6)

then the Rayleigh quotient RA (V ) is upper Hessenberg.

Proof. The (i, j) entry of RA (V ) = V H AV is given by viH Avj , and vi ∈ Ki⊥ is

orthogonal to Avj ∈ Kj+2 as long as i ≥ j + 2.

Lemma B.2. If a matrix V consists of k orthonormal columns satisfying

Equation (B.3.6), then the residual matrix E = PV ⊥ AV is can only contain
nonzero entries in its last column.

Proof. Avj ∈ Kj+2 ⊂ Kk = span V for all j ≤ k − 2, and so PV ⊥ (Avj ) = 0 for

the first k − 1 columns of V .

152
Theorem B.3 (Arnoldi decomposition [8]). If a matrix V consists of k or-
thonormal columns satisfying Equation (B.3.6), then its image AV can be
decomposed as
AV = V H + v(βek )H ,

where H = RA (V ) is upper Hessenberg, v ∈ span PV ⊥ (Ak r), and kv(βek )H k2 =

β.

Proof. We can again apply Equation (B.1.2) to show that, for any matrix V
with orthonormal columns, we may decompose AV as V RA (V ) + PV ⊥ AV .
Lemma B.1 then shows that RA (V ) is upper Hessenberg, and Lemma B.2
shows that only the last column of PV ⊥ AV can be nonzero. If we label this
last column as u, then, when u = 0, we may set v = 0 and β = 0, otherwise,
we may set v = u/kuk2 and β = kuk2 .

Corollary B.4 (Lanczos decomposition [85]). If a matrix V consists of k

orthonormal columns satisfying Equation (B.3.6), and the underlying operator
A is Hermitian, then the image AV can be decomposed as

AV = V T + v(βek )H ,

where T = RA (V ) is Hermitian and tridiagonal, v ∈ span PV ⊥ (Ak r), and

kv(βek )H k2 = β.

Proof. We may simply combine Theorem B.3 with the fact that, if A is Hermi-
tian, RA (V ) = V H AV must also be Hermitian. Since Hermitian Hessenberg
matrices are tridiagonal, we have finished the proof.

153
B.4 Introduction to FOM and GMRES
Suppose that we are interested in the solution of the nonsingular system
of equations
Ax = b,

and that we have constructed a matrix V whose columns are orthonormal

and whose span both contains b and is invariant under A. It is natural to
expect that we can project the system of equations onto span V in order to
quickly solve the linear system. The following theorem shows that this is in
fact possible.

Proposition B.1. If a matrix V has orthonormal columns whose span is

invariant under the nonsingular matrix A, then:

(i) The Rayleigh quotient Z = RA (V ) is invertible,

(ii) A−1 b = V Z −1 V H b for all b ∈ span V , and

(iii) A : span V → span V is a bijection.

Proof. Since span V is invariant under A, we may invoke Proposition B.1 to

conclude that every eigenvalue of Z is also an eigenvalue of A. Because A is
nonsingular, it cannot have a zero eigenvalue, and therefore Z must also be
nonsingular.

Due to the invariance of span V , A(span V ) ⊂ span V , and because A

is invertible, A : span V → span V is injective.

Given that both A and Z are now known to be invertible, we may

rewrite the invariance relation AV = V Z as A−1 (V V H ) = V Z −1 V H , which

154
shows that, for any b ∈ span V , A−1 b = V Z −1 V H b ∈ span V . A : span V →
span V is therefore surjective.

Corollary B.2. If a matrix V has orthonormal columns whose span is the

Krylov subspace Kk (A, b), then Kk (A, b) = Kk+1 (A, b) implies that Kk (A, b) is
invariant under A, and that, if A is nonsingular, the unique solution to the
linear system Ax = b is given by x = V Z −1 V H b.

Proof. We have only to show that Kk (A, b) = Kk+1 (A, b) implies that Kk (A, b)
is invariant under A. This fact can easily be seen since AKk (A, b) ⊂ Kk+1 (A, b) =
Kk (A, b).

Definition B.4.1 (Full Orthogonalization Method (FOM)). Given a nonsin-

gular linear system, Ax = b, and a matrix V composed of orthonormal columns
whose span is the Krylov subspace Kk (A, b), any method which attempts to
approximate the solution x with V Z −1 V H b, even when span V is not necessar-
ily invariant under A, is referred to as a Full Orthogonalization Method (FOM).
The approximate solution, if it exists, will be denoted by xk .

Remark B.4.1. Given any vector x0 , we can instead apply FOM to the linear
system
A(x − x0 ) = b − Ax0 ,

where x0 is referred to as the initial guess, and r0 ≡ b − Ax0 is called the

initial residual. FOM could then be applied by attempting to construct the
approximate solution xk = x0 + V Z −1 V H r0 , where the columns of V span
Kk (A, r0 ) instead of Kk (A, b).

Just as in the previous sections, we have extended an insight which is

precise for invariant subspaces to subspaces which are ideally close to being in-

155
variant. We will unfortunately now see that the FOM can fail catastrophically
when span V is not invariant.
   
0 1 1
Example B.4.1. Let A = and b = , and set V = b so that V
1 0 0
has a single orthonormal column which spans the Krylov subspace K1 (A, b).
Even though A is both nonsingular and Hermitian, the Rayleigh quotient
RA (V ) = bH Ab is identically zero.

Despite the fact that FOM can completely fail on trivial problems, it
turns out that a relatively minor modification results in an extremely robust
algorithm. We will now show that, even though the inverse of the Rayleigh
quotient can fail to exist, a unique solution can always be found in the least-
squares sense.

Lemma B.3. If dim Kk+1 (A, b) = k + 1 and span V = Kk (A, b), then the
matrix AV has full column-rank.

Proof. Because Kk+1 (A, b) = span{b, AV }, the dimension of the column space
of AV is at least dim Kk+1 (A, b) − 1 = k, and thus rank(AV ) = k.

Theorem B.4. Given an Arnoldi decomposition AV = V H + v(βek )H , where

span V = Kk (A, b), there exists a unique least-squares solution to the equivalent
problems
arg min kAV y − bk2 = arg min kH̃y − βe1 k2 ,
y y
 
H
where H̃ = .
kbk2 eH
k

Proof. If Kk (A, b) is invariant, then Corollary B.2 implies that y = Z −1 V H b is

the unique (exact) solution. Otherwise, we must have that dim Kk+1 (A, b) =

156
k + 1, and so Lemma B.3 shows that the matrix AV has full column rank,
which implies that a unique least-squares solution can be found, e.g., through
a QR decomposition of AV .

If we set β = kbk2 , then b = V (βe1 ), and we can make use of the

Arnoldi decomposition to find

AV y − b = (V H + v(βek )H )y − V (βe1 ).

Since the matrix [V v] has orthonormal columns, if we define H̃ as above, then

arg min kAV z − bk2 = arg min kH̃z − βe1 k2 ,

z z

where the vector e1 is now of length k + 1.

Definition B.4.2 (Generalized minimum residual method (GMRES) [108]).

Given a nonsingular linear system, Ax = b, the label GMRES applies to any
method which exploits an Arnoldi decomposition, say AV = V H + v(βek )H ,
to find the minimizer

y = arg min kAV z − bk2 = arg min kH̃z − βe1 k2 ,

z z

which implies that the minimum-residual solution is xk = V y.

Remark B.4.2. Just like FOM, GMRES can easily incorporate an initial
guess x0 by first finding

y = arg min kAV z − r0 k2 ,

z

where r0 = b − Ax0 and the columns of V span Kk (A, r0 ). The approximate

solution to Ax = b is then given by xk = x0 + V y.

157
Definition B.4.3 (Minimum residual method (MINRES) [94]). Given a Her-
mitian nonsingular linear system, Ax = b, the label MINRES applies to any
method which exploits a Lanczos decomposition, say AV = V T + v(βek )H , to
find the minimizer

y = arg min kAV z − bk2 = arg min kT̃ z − βe1 k2 ,

z z

which implies that the minimum-residual solution is xk = V y. That is to say,

MINRES is the specialization of GMRES to Hermitian systems of equations.

Proposition B.5. If GMRES is applied to the nonsingular linear system Ax =

b, yielding an approximate solution xk , then

kb − Axk k2 = min kb − Ap(A)bk2 = min km(A)bk,

p∈Pk−1 m∈Mk

where Pk−1 is the space of k − 1 degree polynomials and

Mk = {p ∈ Pk : p(0) = 1}

is the space of monic polynomials of degree k.

Proof. The left term immediately follows from the definition of GMRES and
the fact that the search space is Kk (A, b) = Pk−1 (A)b. The right term follows
from the fact that p ∈ Pk−1 if and only if 1 + xp(x) ∈ Mk .

Corollary B.6. If GMRES is applied to the nonsingular system Ax = b,

where A is diagonalizable, say, A = W ΛW −1 , then

kb − Axk k2 = min kW m(Λ)W −1 bk2 ≤ κ2 (W ) · min kmkσ(A) · kbk2 .

m∈Mk m∈Mk

158
Proof. The first equality only requires that, for any polynomial p, p(A) =
p(W ΛW −1 ) = W p(Λ)W −1 , which was shown in Proposition A.10. The in-
equality relies on the consistency of the matrix two-norm (Proposition A.15),
i.e.,

kW m(Λ)W −1 k2 ≤ kW k2 kW −1 k2 km(Λ)k2 = κ2 (W ) · max |m(λ)|.

λ∈σ(A)

Corollary B.7. If GMRES is applied to the nonsingular linear system Ax = b,

where A is normal, and the spectral decomposition of A is QΛQH , then

kb − Axk k2 = min km(Λ)QH bk2 ≤ min kmkσ(A) · kbk2 .

m∈Mk m∈Mk

Proof. We have made use of two elementary facts about unitary matrices in
order to improve upon Corollary B.6: κ2 (Q) = 1, and kQxk2 = kxk2 for any
vector x.

Remark B.4.3. The previous bounds show that the residual norms produced
by GMRES are at least weakly controlled by the conditioning of the eigenvec-
tors and the distribution of the eigenvalues. Before we begin to discuss the
important subject of preconditioning, which, in the context of GMRES, loosely
corresponds to mitigating the effects of these two terms, it is important that
we touch on how to replace the kmkσ(A) term with something more concrete.
The main idea is that, if the spectrum is known to be contained within some
well-defined region, then we may propose a monic polynomial, evaluate its
maximum magnitude over the region of interest, and then use this value as an
upper-bound for the minimum value over all polynomials.

159
Example B.4.2. Suppose that A = I − γU , where |γ| < 1 and U is an ar-
bitrary unitary matrix with spectral decomposition QΛQH . Then each eigen-
value of U lies on the unit circle, and, since A = Q(I − γΛ)QH , the eigenvalues
of A lie on the circle of radius |γ| centered at 1. It is then natural to investigate
the candidate polynomial

m(z) = (1 − z)k ∈ Mk ,

which evaluates to 1 at the origin and γ k over the spectrum of A. Then, by

Corollary B.7, we have that

kb − Axk k2 ≤ |γ|k kbk2 .

Remark B.4.4. It turns out that, as the spectrum becomes dense on the
boundary of the circle, the candidate polynomial from the last example be-
comes optimal. This can easily be shown using the following theorem due to
S. Bernstein, which we state without proof.

Theorem B.8 (S. Bernstein [6, 20]). If p ∈ Pk and kpkC1 (0) = 1, then kpkCR (0) ≤
Rk for R ≥ 1, with equality only if p(z) = γz k , with |γ| = 1.

Corollary B.9. For any radius r < |z0 |,

rk
min kmkCr (z0 ) = ,
m∈Mk |z0 |k

and the minimizing monic polynomial is given by (z0 − z)k /z0k .

Proof. Showing that this value is attainable simply requires a rescaling of the
previous example. In particular, we can define the monic polynomial
1
m(z) = (z − z)k ∈ Mk ,
k 0
z0

160
which, for any z ∈ Cr (z0 ), evaluates to rk /|z0 |k .

Now suppose that there exists m̃(z) ∈ Mk such that

ηk rk
km̃kCr (z0 ) = < .
|z0 |k |z0 |k

We may then define the polynomial

|z0 |k
p(z) = m̃(zr + z0 ),
ηk

then we see that kpkC1 (0) = 1, and, setting R = |z0 |/r,

|z0 |k |z0 |k
p(−z0 /r) = k m̃(0) = k > Rk ,
η η

which contradicts Theorem B.8.

Corollary B.10. For any radius r < |z0 |,

rk
min kmkD̄r (z0 ) = ,
m∈Mk |z0 |k

and the minimizing monic polynomial is given by (z0 − z)k /z0k .

Proof. Since Cr (z0 ) ⊂ D̄r (z0 ) and (z0 − z)k /z0k reaches its extrema on the
boundary of Dr (z0 ), (z0 − z)k /z0k is also optimal over the entire closed disc
D̄r (z0 ).

Remark B.4.5. The optimal polynomial only depends upon the center, and
not the radius, of the disc.

While the optimal monic polynomial is easily found for the case where
the spectrum is only known to be contained within a particular circle in the
complex plane which does not contain the origin, there are several other cases

161
of significant interest. In particular, the following well-known theorem [51],
due to Flanders and Shortley, but apparently inspired by Tukey and Grosch,
can be immediately used to show that monic Chebyshev polynomials are the
optimally small monic polynomials over intervals of the real line which are
separated from the origin.

Theorem B.11. Among all polynomials p ∈ Pk satisfying p(γ) = 1, where

|γ| > 1, the unique minimizer of kpk[−1,1] is given by Tk (z)/Tk (γ), where Tk is
the k’th-order Chebyshev polynomial of the first kind. In addition, the maxi-
mum absolute value of this polynomial over [−1, 1] is 1/|Tk (γ)|.

Corollary B.12. Let [α, β] be an interval of the real line which does not
contain the origin. Then
1
min kmk[α,β] =   ,
m∈Mk α+β
Tk β−α
   
α+β+2z α+β
and the unique minimizing monic polynomial is Tk β−α
/Tk β−α
.

Proof. We may perform a change of variables on each m ∈ Mk , say

 
µ−z
m(z) = p ,
w
where µ = (α + β)/2, and w = (β − α)/2, where z 7→ (µ − z)/w provides a
bijection between [α, β] and [−1, 1]. Since m(0) = 1 implies that p(µ/w) = 1,
p is a candidate for Theorem B.11 if we choose γ = µ/w.

Corollary B.13. Let i[α, β] be an interval of the imaginary axis which does
not contain the origin. Then
1
min kmk[α,β] =   ,
m∈Mk α+β
Tk β−α
   
α+β+2iz α+β
and the unique minimizing polynomial is Tk β−α
/Tk β−α
.

162
Proof. The proof is essentially identical to that of Corollary B.12, but we
instead set µ = i(α + β)/2 and w = i(β − α)/2.

The last classical case concerns the minimization of a monic polynomial

over an ellipse which is separated from the origin. Interestingly enough, though
the monic Chebyshev polynomials cannot provide an optimal solution [50],
they are asymptotically optimal in the sense that, as the polynomial order
increases to infinity, the supremum norm of the best candidate converges to
the minimal value. In order to avoid diving into the connections between
Chebyshev polynomials, the Joukowski map, and Bernstein ellipses [96], we
simply define the near-best Chebyshev polynomial. Please see [107] or [49] for
details.

Definition B.4.4 (Near-best Chebyshev polynomial [107]). Given an ellipse

ε, with center z0 and focal distance d, we say that the monic Chebyshev
polynomial
z0 −z


Tk d

m̃(z) = z0
Tk d
is the near-best polynomial for the minimization problem

arg min kmkε ,

m∈Mk

and it satisfies
a


Tk d

km̃kε = z0
,
Tk d

where a is the (possibly complex) semi-major axis of the ellipse ε.

On the other hand, it is important to recognize that upper bounds

based upon the supremum norm of monic polynomials over intervals containing
the spectrum can be extremely far from optimal, even for normal matrices.

163
The reason is that the right-hand side, b, may only have components in the
directions of a handful of eigenvectors of A, and a monic polynomial can be
constructed with its roots at the appropriate eigenvalues such that m(A)b is
exactly zero.

Example B.4.3. Suppose that the right-hand side, b, is an eigenvector of the

nonsingular matrix A with eigenvalue λ. Then GMRES applied to the system
Ax = b will converge in a single iteration, as b − A(1/λ)b = 0, and 1/λ ∈ P0 .

Proposition B.14. If a vector b lies in a subspace spanned by a set of eigen-

vectors of a nonsingular matrix A with r distinct eigenvalues, then GMRES
applied to the system Ax = b will converge in at most r iterations.

Proof. Let {λj }rj=1 be the eigenvalues corresponding to the eigenvectors which
comprise b and set
r
Y λj − z
m(z) = ∈ Mr .
j=1
λj
Then !
r
Y −1
m(A)b = (A − λ1 I) · · · (A − λr I)b.
j=1
λj
If we express b as a linear combination of the relevant eigenvectors, say
r
X
b= γj yj ,
j=1

then the fact that

r
X r
X
(A − λi I) γj yj = (λj − λi )γj yj ,
j=1 j=1

shows that !
r r Y
r
X −1 X
m(A)b = (λj − λi )γj yj = 0,
j=1
λj j=1 i=1
Qr
since each term i=1 (λj − λi )γj yj is identically zero.

164
B.5 Implementing GMRES
A completely naı̈ve implementation of GMRES would essentially exe-
cute the following steps:

1. sequentially construct each vector b, Ab, A(Ab), A(A(Ab)), etc., to form

a matrix B whose columns span the Krylov subspace Kk (A, b),

2. apply Gram-Schmidt orthogonalization to the columns of the matrix B

in order to find an orthonormal basis for Kk (A, b), say V ,

3. form the Rayleigh quotient H = V H (AV ) in the obvious manner,

4. solve the least-squares problem y = arg minz kH̄ − βe1 k2 ,

5. set the GMRES estimate to be xk = V y,

6. compute the residual rk = kb − Axk k2 , and, finally,

7. stop if the residual is small enough, or, otherwise, repeat the entire
process with a larger value for k.

There are unfortunately numerous practical problems with this ap-

proach, such as:

• B is likely to be extremely ill-conditioned, perhaps to the point of arti-

ficially losing linear independence of its columns due to round-off error,

• the classical Gram-Schmidt algorithm is well-known to be numerically

unstable, and should be replaced with the modified Gram-Schmidt algo-
rithm [58],

165
 • if we choose to increase k in order to use a large Krylov subspace then
the previous work need not be thrown away,

• for large N × N sparse systems, the O(N k) memory requirements for

storing the basis for Kk (A, b) can quickly get out of hand, and so it is
important to come up with a so-called restart technique, and, finally,

• for problems which do not converge quickly, it is crucial to apply GM-

RES to a preconditioned operator, say M −1 A, where the preconditioner
M must be carefully designed in order to balance its construction and
application costs with its effectiveness at accelerating GMRES conver-
gence.

In practice, a restarted version of GMRES called GMRES(k) is typi-

cally used. We may describe its fundamental details in one sentence: after k
iterations of GMRES, if the residual is not sufficiently small, the current esti-
mate xk is used to restart GMRES. We will conclude the appendix by noting
that, because k is typically quite small, e.g., 20, the only steps of GMRES(k)
which need to be parallelized are the application of the sparse matrix and
the preconditioner, and the computation of inner products and norms [16, 17].
Please see Algorithm B.1 for a straight-forward, but reasonably practical, im-
plementation of GMRES(k).

B.6 Summary
A brief introduction to the theory and implementation of the Gener-
alized Minimum Residual (GMRES) method was provided in order to serve
as a reference to those without extensive experience implementing iterative

166
Algorithm B.1: Pseudocode for a simple implementation of pre-
conditioned GMRES(k)
Input: Nonsingular matrix, A, preconditioner, M , right-hand
side, b, and initial guess, x0
Output: Estimate, x, for the solution, A−1 b
1 x := x0
2 r := b − Ax, ρ := krk2
3 while ρ/kbk2 < tolerance do
// Run GMRES for k steps with x as initial guess
4 x0 := x, H := zeros(k, k)
5 w := M −1 r
6 β := kwk2 , v0 := w/β
7 for j = 0 : k − 1 do
8 d := Avj , δ := kdk2
9 d := M −1 d
// Run j’th step of Arnoldi
10 for i = 0 : j do
11 H(i, j) := viH d
12 d := d − H(i, j)vi
13 end
14 H(j + 1, j) = δ := kdk2
15 if j + 1 6= k then vj+1 := vj+1 /δ
// Solve for residual minimizer
16 y := arg minz kH(0 : j +1, 0 : j)z − βe0 k2
// Form next iterate and residual
17 x := x0 + Vj y
18 r := b − Ax, ρ := krk2
19 if ρ/kbk2 < tolerance then break
20 end
21 end

167
methods. The author hopes that this appendix will serve to dispel some of
the mystique surrounding GMRES(k), as it is surprisingly straight-forward to
implement, both sequentially and in parallel.

168
 Appendix C

Dense linear algebra algorithms

Several algorithms are listed in this appendix for the sole purpose of
providing an anchor for the (compressed) multifrontal techniques discussed in
Chapters 2 and 4. The following fundamental dense linear algebra operations
are covered:

• matrix-matrix multiplication,

• triangular solves,

• Cholesky factorization, and

• triangular inversion.

Matrix-matrix multiplication is perhaps the most important operation

to understand, as it is at the heart of most high-performance dense linear
algebra operations (by design), as well as the high-performance Kronecker-
product application scheme discussed in Chapter 4. Please see [127] and [109]
for detailed discussions of the ideas behind the matrix-matrix multiplication
algorithms which follow. The triangular solve algorithms form the foundation
for the multifrontal triangular solve algorithms of Chapter 2, dense Cholesky
factorization is, of course, at the heart of our multifrontal Cholesky factoriza-
tion scheme, and, finally, dense triangular inversion is used as part of selective
inversion.

169
 The notation used within the following algorithms is in the style of
FLAME [60] and Elemental [99]. In addition, the operations described here
are essentially all implemented as part of the Elemental library, and thorough
benchmarks are provided for several of these operations within [99] and [109],
as well as thorough discussions of the underlying notation (which is closely
followed by the actual implementations). In most cases, the progression follows
that of LAPACK [5], ScaLAPACK [31], and PLAPACK [2], which is to build
an “unblocked” implementation for small sequential problems, use this at the
core of a “blocked” algorithm for larger sequential problems, and then to embed
this blocked algorithm within a distributed-memory algorithm.

Algorithm C.1: Distributed dense matrix-matrix multiplication

with few right-hand sides
Input: Scalar, α, m × k matrix, A, k × n matrix, B, scalar, β,
and m × n matrix, C. A is in a 2D distribution, but B
and C are in 1D distributions.
Output: C := αAB + βC
1 B[MR , ?] ← B[VC , ?]

2 Ẑ[MC , ?] := αA[MC , MR ]B[MR , ?]

3 C[VC , ?] := βC[VC , ?] + SumScatter(Ẑ[MC , ?])

Algorithm C.2: Distributed dense adjoint matrix-matrix multi-

plication with few right-hand sides
Input: Scalar, α, k × m matrix, A, k × n matrix, B, scalar, β,
and m × n matrix, C. A is in a 2D distribution, but B
and C are in 1D distributions.
Output: C := αAH B + βC
1 B[MC , ?] ← B[VC , ?]
H
2 Ẑ[MR , ?] := αA [MR , MC ]B[MC , ?]

3 C[VC , ?] := βC[VC , ?] + SumScatter(Ẑ[MR , ?])

170
Algorithm C.3: Distributed dense matrix-matrix multiplication
with many right-hand sides
Input: Scalar, α, m × k matrix, A, k × n matrix, B, scalar, β,
m × n matrix, C, and a blocksize, nb . A, B, and C are
each in 2D distributions.
Output: C := αAB + βC
1 C := βC
2 b := min{k, nb }
 

B1
A → A1 A2 and B → , where A1 is m × b and B1 is b × n
3 B2
// C := C + αA1 B1
4 A1 [MC , ?] ← A1 [MC , MR ]
5 B1 [?, MR ] ← B1 [MC , MR ]
6 C[MC , MR ] := C + αA1 [MC , ?]B1 [?, MR ]
7 if k > nb then Recurse(α,A2 ,B2 ,1,C,nb )

Algorithm C.4: Distributed dense adjoint matrix-matrix multi-

plication with many right-hand sides
Input: Scalar, α, k × m matrix, A, k × n matrix, B, scalar, β,
m × n matrix, C, and a blocksize, nb . A, B, and C are
each in 2D distributions.
Output: C := αAH B + βC
1 C := βC
2 b := min{k, nb }
   
A1 B1
A→ and B → , where A1 is b × m and B1 is b × n
3 A2 B2
// C := C + αAH 1 B1
4 A1 [?, MC ] ← A1 [MC , MR ]
5 B1 [?, MR ] ← B1 [MC , MR ]
H
6 C[MC , MR ] := C + αA1 [MC , ?]B1 [?, MR ]
7 if k > nb then Recurse(α,A2 ,B2 ,1,C,nb )

171
Algorithm C.5: Unblocked dense triangular solve
Input: n × n lower-triangular matrix L and an n × k matrix X
−1
Output:
 X := L  X  
λ1,1 0 x1
L→ and X → , where λ1,1 is 1 × 1 and x1 is
1 `2,1 L2,2 X2
1×k
2 x1 := x1 /λ1,1
3 if n > 1 then
4 X2 := X2 − `2,1 x1
5 Recurse(L2,2 ,X2 )

Algorithm C.6: Dense triangular solve

Input: n × n lower-triangular matrix, L, an n × k matrix, X, and
a blocksize, nb
Output: X := L−1 X
1 b := min{n, nb }
   
L1,1 0 X1
L→ and X → , where L1,1 is b × b and X1 is
2 L2,1 L2,2 X2
b×k
−1
3 X1 := L1,1 X1 via Algorithm C.5
4 if n > nb then
5 X2 := X2 − L2,1 X1
6 Recurse(L2,2 ,X2 )

172
Algorithm C.7: Dense triangular solve with few right-hand sides
(1D distribution)
Input: n × n lower-triangular matrix, L, an n × k matrix, X, and
a blocksize, nb . L and X are both in 1D distributions.
Output: X := L−1 X
1 b := min{n, nb }
   
L1,1 0 X1
L→ and X → , where L1,1 is b × b and X1 is
2 L2,1 L2,2 X2
b×k
// X1 := L−1 1,1 X1
3 X1 [∗, ∗] ← X1 [VC , ∗]
4 L1,1 [∗, ∗] ← L1,1 [VC , ∗]
−1
5 X1 [∗, ∗] := L1,1 [∗, ∗]X1 [∗, ∗] via Algorithm C.6
6 X1 [VC , ∗] ← X1 [∗, ∗]
7 if n > nb then
// X2 := X2 − L2,1 X1
8 X2 [VC , ∗] := X2 [VC , ∗] − L2,1 [VC , ∗]X1 [∗, ∗]
9 Recurse(L2,2 ,X2 )

173
Algorithm C.8: Dense triangular solve with few right-hand sides
(2D distribution)
Input: n × n lower-triangular matrix, L, an n × k matrix, X, and
a blocksize, nb . L is in a 2D distribution, but X is in a 1D
distribution. An n × k matrix, Ẑ, initially filled with zeros
and in an [MC , ∗] distribution, should also be passed in.
Output: X := L−1 X
1 b := min{n, nb }
     
L1,1 0 X1 Ẑ1
L→ ,X→ , and Ẑ → , where L1,1 is
2 L2,1 L2,2 X2 Ẑ2
b × b and X1 and Ẑ1 are b × k
// Combine and apply all previous partial updates to X1
3 X1 [VC , ∗] := X1 [VC , ∗] + SumScatter(Ẑ1 [MC , ∗])
// X1 := L−1 1,1 X1
4 X1 [∗, ∗] ← X1 [VC , ∗]
5 L1,1 [∗, ∗] ← L1,1 [MC , MR ]
−1
6 X1 [∗, ∗] := L1,1 [∗, ∗]X1 [∗, ∗] via Algorithm C.6
7 X1 [VC , ∗] ← X1 [∗, ∗]
8 if n > nb then
// Add portion of −L2,1 X1 onto past partial updates
9 X1 [MR , ∗] ← X1 [∗, ∗]
10 Ẑ2 [MC , ∗] := Ẑ2 [MC , ∗] − L2,1 [MC , MR ]X1 [MR , ∗]
11 Recurse(L2,2 ,X2 ,Ẑ2 )

174
Algorithm C.9: Distributed dense triangular solve with many
right-hand sides
Input: n × n lower-triangular matrix, L, an n × k matrix, X, and
a blocksize, nb . L and X are both in 2D distributions.
Output: X := L−1 X
1 b := min{n, nb }
   
L1,1 0 X1
L→ and X → , where L1,1 is b × b and X1 is
2 L2,1 L2,2 X2
b×k
// X1 := L−1 1,1 X1
3 X1 [∗, VR ] ← X1 [MC , MR ]
4 L1,1 [∗, ∗] ← L1,1 [MC , MR ]
−1
5 X1 [∗, VR ] := L1,1 [∗, ∗]X1 [∗, VR ] via Algorithm C.6
6 if n > nb then
// X2 := X2 − L2,1 X1 and delayed write of X1
7 L2,1 [MC , ∗] ← L2,1 [MC , MR ]
8 X1 [∗, MR ] ← X1 [∗, VR ]
9 X2 [MC , MR ] := X2 [MC , MR ] − L2,1 [MC , ∗]X1 [∗, MR ]
10 X1 [MC , MR ] ← X1 [∗, MR ]
11 Recurse(L2,2 ,X2 )

Algorithm C.10: Unblocked dense adjoint triangular solve

Input: n × n lower-triangular matrix L and an n × k matrix X
−H
Output:
 X := L  X  
L0,0 0 X0
L→ and X → , where λ1,1 is 1 × 1 and x1 is
1 `1,0 λ1,1 x1
1×k
2 x1 := x1 /λ1,1
3 if n > 1 then
4 X0 := X0 − `H 1,0 x1
5 Recurse(L0,0 ,X0 )

175
Algorithm C.11: Dense adjoint triangular solve
Input: n × n lower-triangular matrix L, an n × k matrix X, and a
blocksize, nb
Output: X := L−H X
1 b := min{n, nb }
   
L0,0 0 X0
L→ and X → , where L1,1 is b × b and X1 is
2 L1,0 L1,1 X1
b×k
−H
3 X1 := L1,1 X1 via Algorithm C.10
4 if n > nb then
5 X0 := X0 − LH 1,0 X1
6 Recurse(L0,0 ,X0 )

Algorithm C.12: Dense adjoint triangular solve with few right-

hand sides (2D distribution)
Input: n × n lower-triangular matrix L, an n × k matrix X, and a
blocksize, nb . L is in a 2D distribution and X is in a 1D
distribution. An n × k matrix, Ẑ, initially filled with zeros
and in an [MC , ∗] distribution, should also be passed in.
Output: X := L−H X
1 b := min{n, nb }
   
L0,0 0 X0
L→ and X → , where L1,1 is b × b and X1 is
2 L1,0 L1,1 X1
b×k
// Combine and apply all previous partial updates to X1
3 X1 [VC , ∗] := X1 [VC , ∗] + SumScatter(Ẑ1 [MR , ∗])
// X1 := L−H 1,1 X1
4 X1 [∗, ∗] ← X1 [VC , ∗]
5 L1,1 [∗, ∗] ← L1,1 [MC , MR ]
−H
6 X1 [∗, ∗] := L1,1 [∗, ∗]X1 [∗, ∗] via Algorithm C.10
7 X1 [VC , ∗] ← X1 [∗, ∗]
8 if n > nb then
// Add portion of −LH 1,0 X1 onto past partial updates
9 Ẑ0 [MR , ∗] := Ẑ0 [MR , ∗] − LH1,0 [MR , MC ]X1 [MC , ∗]
10 Recurse(L0,0 ,X0 ,Ẑ0 )

176
Algorithm C.13: Distributed dense adjoint triangular solve with
many right-hand sides
Input: n × n lower-triangular matrix L, an n × k matrix X, and a
blocksize, nb . L and X are both in a 2D distribution.
Output: X := L−H X
1 b := min{n, nb }
   
L0,0 0 X0
L→ and X → , where L1,1 is b × b and X1 is
2 L1,0 L1,1 X1
b×k
// X1 := L−H 1,1 X1
3 X1 [∗, VR ] ← X1 [MC , MR ]
4 L1,1 [∗, ∗] ← L1,1 [MC , MR ]
−H
5 X1 [∗, VR ] := L1,1 [∗, ∗]X1 [∗, VR ] via Algorithm C.10
6 if n > nb then
// X0 := X0 − LH 1,0 X1 and delayed write of X1
7 L1,0 [∗, MC ] ← L1,0 [MC , MR ]
8 X1 [∗, MR ] ← X1 [∗, VR ]
9 X0 [MC , MR ] := X0 [MC , MR ] − LH 1,0 [MC , ∗]X1 [∗, MR ]
10 X1 [MC , MR ] ← X1 [∗, MR ]
11 Recurse(L0,0 ,X0 )

177
Algorithm C.14: Dense adjoint triangular solve (lazy variant)
Input: n × n lower-triangular matrix L, an n × k matrix X, and a
blocksize, nb
−H
Output:
 X := L X  
LT L 0 XT
L→ and X → , where LBR is 0 × 0 and XB
1 LBL LBR XB
is 0 × k
2 while size(LBR ) 6= size(L) do
3 b := min{height(LT L ), nb }
 
  L0,0 0 0
LT L 0
→  L1,0 L1,1 0  and
LBL LBR
4 L L2,1 L2,2
  2,0
  X0
XT
→  X1 , where L1,1 is b × b and X1 is b × k
XB
X2
H
5 X1 := X1 − L2,1 X2
6 X1 := L−H1,1 X1 via Algorithm C.10
 
  L0,0 0 0
LT L 0
←  L1,0 L1,1 0  , and
LBL LBR
7 L L2,1 L2,2
  2,0
  X0
XT
←  X1 
XB
X2
8 end

178
Algorithm C.15: Dense adjoint triangular solve with few right-
hand sides (lazy variant, 1D distribution)
Input: n × n lower-triangular matrix L, an n × k matrix X, and a
blocksize, nb
−H
Output:
 X := L X  
LT L 0 XT
L→ and X → , where LBR is 0 × 0 and XB
1 LBL LBR XB
is 0 × k
2 while size(LBR ) 6= size(L) do
3 b := min{height(LT L ), nb }
 
  L0,0 0 0
LT L 0
→  L1,0 L1,1 0  and
LBL LBR
4 L L2,1 L2,2
  2,0
  X0
XT
→  X1 , where L1,1 is b × b and X1 is b × k
XB
X2
// X1 := X1 − LH 2,1 X2
H
5 Ẑ1 [∗, ∗] := −L2,1 [∗, VC ]X2 [VC , ∗]
6 X1 [VC , ∗] := X1 [VC , ∗] + SumScatter(Ẑ1 [∗, ∗])
// X1 := L−H 1,1 X1
7 X1 [∗, ∗] ← X1 [VC , ∗]
8 L1,1 [∗, ∗] ← L1,1 [VC , ∗]
9 X1 [∗, ∗] := L−H1,1 [∗, ∗]X1 [∗, ∗] via Algorithm C.10
10 X1 [VC , ∗] ← X1 [∗, ∗]
 
  L0,0 0 0
LT L 0
←  L1,0 L1,1 0  , and
LBL LBR
11 L L2,1 L2,2
  2,0
  X0
XT
←  X1 
XB
X2
12 end

179
Algorithm C.16: Unblocked dense right-looking Cholesky
Input: n × n HPD matrix, A
Output:
 A is overwritten
 with its lower Cholesky factor, L
α1,1 ?
A→ , where α1,1 is a scalar
1 a2,1 A2,2
√
2 λ1,1 := α1,1
3 if n > 1 then
4 `2,1 := a2,1 /λ1,1
5 A2,2 := A2,2 − `2,1 `H2,1
6 Recurse(A2,2 )

Algorithm C.17: Dense right-looking Cholesky

Input: n × n HPD matrix, A, and blocksize, nb
Output: A is overwritten with its lower Cholesky factor, L
1 b := min{n, nb }
 
A1,1 ?
A→ , where A1,1 is b × b
2 A2,1 A2,2
3 L1,1 := Cholesky(A1,1 )
4 via Algorithm C.16
5 if n > nb then
6 L2,1 := A2,1 L−H
1,1
7 A2,2 := A2,2 − L2,1 LH2,1
8 Recurse(A2,2 ,nb )

180
Algorithm C.18: Distributed dense right-looking Cholesky
Input: n × n HPD matrix, A, in a 2D distribution, and a
blocksize nb
Output: A is overwritten with its lower Cholesky factor, L
1 b := min{n, nb }
 
A1,1 ?
A→ , where A1,1 is b × b
2 A2,1 A2,2
// L1,1 := Cholesky(A1,1 )
3 A1,1 [∗, ∗] ← A1,1 [MC , MR ]
4 L1,1 [∗, ∗] := Cholesky(A1,1 [∗, ∗]) via Algorithm C.17
5 L1,1 [MC , MR ] ← L1,1 [∗, ∗]
6 if n > nb then
// L2,1 := A2,1 L−H 1,1
7 A2,1 [VC , ∗] ← A2,1 [MC , MR ]
8 L2,1 [VC , ∗] := A2,1 [VC , ∗]L−H
1,1 [∗, ∗]
// A2,2 := A2,2 − L2,1 LH 2,1 and delayed write of L2,1
9 L2,1 [MC , ∗] ← L2,1 [VC , ∗]
10 L2,1 [MR , ∗] ← L2,1 [VC , ∗]
11 A2,2 [MC , MR ] := A2,2 [MC , MR ] − L2,1 [MC , ∗]LH
2,1 [∗, MR ]
12 L2,1 [MC , MR ] ← L2,1 [MC , ∗]
13 Recurse(A2,2 ,nb )

181
Algorithm C.19: Unblocked dense triangular inversion
Input: n × n lower-triangular matrix, L
Output:
 L is overwritten
 with its inverse
LT L 0
L→ , where LT L is 0 × 0
1 LBL LBR
2 while size(LT L ) 6= size(L) do
 
  L0,0 0 0
LT L 0
→  `1,0 λ1,1 0 , where λ1,1 is a scalar
LBL LBR
3 L2,0 `2,1 L2,2
4 `1,0 := −`1,0 /λ1,1
5 L2,0 := L2,0 + `2,1 `1,0
6 `2,1 := `2,1 /λ1,1
7 λ1,1 := 1/λ1,1
 
  L0,0 0 0
LT L 0
←  `1,0 λ1,1 0 
LBL LBR
8 L2,0 `2,1 L2,2
9 end

182
Algorithm C.20: Dense triangular inversion
Input: n × n lower-triangular matrix, L, and blocksize, nb
Output:
 L is overwritten
 with its inverse
LT L 0
L→ , where LT L is 0 × 0
1 LBL LBR
2 while size(LT L ) 6= size(L) do
3 b := min{height(LBR ), nb }
 
  L0,0 0 0
LT L 0
→  L1,0 L1,1 0 , where L1,1 is b × b
LBL LBR
4 L2,0 L2,1 L2,2
−1
5 L1,0 := −L1,1 L1,0
6 L2,0 := L2,0 + L2,1 L1,0
7 L2,1 := L2,1 L−1
1,1
8 L1,1 := L−1
1,1 via Algorithm C.19
 
  L0,0 0 0
LT L 0
←  L1,0 L1,1 0 
LBL LBR
9 L2,0 L2,1 L2,2
10 end

183
Algorithm C.21: Distributed dense triangular inversion
Input: n × n lower-triangular matrix, L, in a 2D distribution, and
a blocksize, nb
Output:
 L is overwritten
 with its inverse
LT L 0
L→ , where LT L is 0 × 0
1 LBL LBR
2 while size(LT L ) 6= size(L) do
3 b := min{height(LBR ), nb }
 
  L0,0 0 0
LT L 0
→  L1,0 L1,1 0 , where L1,1 is b × b
LBL LBR
4 L2,0 L2,1 L2,2
−1
// L1,0 := −L1,1 L1,0
5 L1,0 [∗, VR ] ← L1,0 [MC , MR ]
6 L1,1 [∗, ∗] ← L1,1 [MC , MR ]
7 L1,0 [∗, VR ] := −L−1 1,1 [∗, ∗]L1,0 [∗, VR ]
// L2,0 := L2,0 + L2,1 L1,0 and delayed write of L1,0
8 L2,1 [MC , ∗] ← L2,1 [MC , MR ]
9 L1,0 [∗, MR ] ← L1,0 [∗, VR ]
10 L2,0 [MC , MR ] := L2,0 [MC , MR ] + L2,1 [MC , ∗]L1,0 [∗, MR ]
11 L1,0 [MC , MR ] ← L1,0 [∗, MR ]
// L2,1 := L2,1 L−1 1,1
12 L2,1 [VC , ∗] ← L2,1 [MC , ∗]
13 L2,1 [VC , ∗] := L2,1 [VC , ∗]L−1 1,1 [∗, ∗]
14 L2,1 [MC , MR ] ← L2,1 [VC , ∗]
// L1,1 := L−1 1,1
15 L1,1 [∗, ∗] := L−11,1 [∗, ∗] via Algorithm C.20
16 L1,1 [MC , MR ] ← L1,1 [∗, ∗]
 
  L0,0 0 0
LT L 0
←  L1,0 L1,1 0 
LBL LBR
17 L2,0 L2,1 L2,2
18 end

184
 Bibliography

[1] MUltifrontal Massively Parallel Solver Users’ guide: version 4.10.0. http:
//graal.ens-lyon.fr/MUMPS/doc/userguide_4.10.0.pdf, May 10, 2011.

[2] Philip Alpatov, Greg Baker, Carter Edwards, John A. Gunnels, Greg
Morrow, James Overfelt, Robert A. van de Geijn, and Yuan-Jye J. Wu.
PLAPACK: Parallel Linear Algebra Package – design overview. In
Proceedings of Supercomputing, 1997.

[3] Patrick R. Amestoy, Iain S. Duff, Jacko Koster, and Jean-Yves L’Excellent.
A fully asynchronous multifrontal solver using distributed dynamic schedul-
ing. SIAM Journal on Matrix Analysis and Applications, 23(1):15–41,
2001.

[4] Fred Aminzadeh, Jean Brac, and Tim Kunz. 3-D Salt and Overthrust
models. In SEG/EAGE 3-D Modeling Series 1, Tulsa, OK, 1997. Soci-
ety for Exploration Geophysicists.

[5] Edward J. Anderson, Zhaojun Bai, Christian H. Bischof, James W. Dem-

mel, Jack J. Dongarra, J. Du Croz, Anne Greenbaum, Sven Hammar-
ling, A. McKenney, and Danny C. Sorensen. LAPACK: A portable
linear algebra library for high-performance computers. In Proceedings
Supercomputing ’90, pages 2–11. IEEE Computer Society Press, Los
Alamitos, California, 1990.

[6] Nesmith C. Ankeny and Theodore J. Rivlin. On a theorem of S. Bern-

stein. Pacific Journal of Mathematics, 5(2):849–852, 1955.

185
 [7] Jean-Robert Argand. Essai sur une manière de représenter des quan-
tités imaginaires dans les constructions géométriques. les Annales de
mathématiques pures et appliquées, 4:133–147, 1813.

[8] Walter E. Arnoldi. The principle of minimized iteration in the solution

of the matrix eigenvalue problem. Quarterly of Applied Mathematics,
9:17–29, 1951.

[9] Cleve Ashcraft. A taxonomy of distributed dense LU factorization meth-

ods. Boeing Computer Services Technical Report ECA-TR-161, March
1991.

[10] Cleve Ashcraft, Stanley C. Eisenstat, and Joseph W.-H. Liu. A fan-in
algorithm for distributed sparse numerical factorization. SIAM Journal
on Scientific and Statistical Computing, 11(3):593–599, 1990.

[11] Cleve Ashcraft and Roger G. Grimes. The influence of relaxed su-
pernode partitions on the multifrontal method. ACM Transactions on
Mathematical Software, 15(4):291–309, 1989.

[12] Cleve Ashcraft and Roger G. Grimes. SPOOLES: An object-oriented

sparse matrix library. In Proceedings of the SIAM Conference on Par-
allel Processing for Scientific Computing, page 10, March 22-27, 1999.

[13] Cleve Ashcraft, Roger G. Grimes, John G. Lewis, Barry W. Peyton, and
Horst D. Simon. Progress in sparse matrix methods for large sparse
linear systems on vector supercomputers. International Journal of Su-
percomputer Applications, 1:10–30, 1987.

[14] Sheldon Axler. Down with determinants! American Mathematical

Monthly, 102(2):139–154.

186
[15] Ivo M. Babuška and S. A. Sauter. Is the pollution effect of the FEM
avoidable for the Helmholtz equation considering high wave numbers?
SIAM Review, 42(3):451–484, 2000.

[16] Satish Balay, Jed Brown, , Kris Buschelman, Victor Eijkhout, William D.
Gropp, Dinesh Kaushik, Matthew G. Knepley, Lois Curfman McInnes,
Barry F. Smith, and Hong Zhang. PETSc Uusers manual. Technical
Report ANL-95/11 - Revision 3.3, Argonne National Laboratory, 2012.

[17] Satish Balay, Jed Brown, Kris Buschelman, William D. Gropp, Dinesh
Kaushik, Matthew G. Knepley, Lois Curfman McInnes, Barry F. Smith,
and Hong Zhang. PETSc Web page. http://www.mcs.anl.gov/
petsc, 2012.

[18] Alvin Bayliss, Charles I. Goldstein, and E. Turkel. An iterative method

for the Helmholtz equation. Journal of Computational Physics, 49:443–
457, 1983.

[19] Jean-Pierre Bérenger. A perfectly matched layer for the absorption of

electromagnetic waves. Journal of Computational Physics, 114:185–200,
1994.

[20] S. Bernstein. Sur l’ordre de la meilleure approximation des fonctions

continues par des polynomes de degré donné. Mém. Acad. Roy. Belg.,
1912.

[21] Gregory Beylkin, Christopher Kurcz, and Lucas Monzón. Fast algo-
rithms for Helmholtz Green’s functions. Proceedings of the Royal Society
A: Mathematical, Physical, and Engineering Sciences, 464(2100):3301–
3326, 2008.

187
[22] Matthias Bollhöfer and Yousef Saad. Multilevel preconditioners con-
structed from inverse-based ILUs. SIAM Journal on Scientific Comput-
ing, 27:1627–1650, 2006.

[23] Matthis Bollhöfer, Marcus Grote, and Olaf Schenk. Algebraic multi-
level preconditioner for the Helmholtz equation in heterogeneous media.
SIAM Journal on Scientific Computing, 31:3781–3805, 2009.

[24] James H. Bramble and Joseph E. Pasciak. A note on the existence and
uniqueness of solutions of frequency domain elastic wave problems: a pri-
ori estimates in H 1 . Mathematical Analysis and Applications, 345:396–
404, 2008.

[25] Achi Brandt. Multi-level adaptive solutions to boundary-value prob-

lems. Mathematics of Computation, 31(138):333–390, 1977.

[26] William L. Briggs, Van Emden Henson, and Steve F. McCormick. A

multigrid tutorial. SIAM, 2000.

[27] Henri Calandra, Serge Gratton, Xavier Pinel, and Xavier Vasseur. An
improved two-grid preconditioner for the solution of three-dimensional
Helmholtz problems in heterogeneous media. Technical Report TR/PA/12/2,
CERFACS, Toulouse, France, 2012.

[28] Ernie Chan, Marcel Heimlich, Avi Purkayastha, and Robert van de
Geijn. Collective communication: theory, practice, and experience.
Concurrency and Computation: Practice and Experience, 19(13):1749–
1783, 2007.

188
[29] Weng Cho Chew and William H. Weedon. A 3D perfectly matched
medium from modified Maxwell’s equations with stretched coordinates.
Microwave and Optical Technology Letters, 7(13):599–604, 1994.

[30] Lindsay N. Childs. A concrete introduction to higher algebra. Under-

graduate Texts in Mathematics. Springer-Verlag, New York, NY, 2008.

[31] J. Choi, J. J. Dongarra, R. Pozo, and D. W. Walker. ScaLAPACK: A

scalable linear algebra library for distributed memory concurrent com-
puters. In Proceedings of the Fourth Symposium on the Frontiers of
Massively Parallel Computation, pages 120–127. IEEE Comput. Soc.
Press, 1992.

[32] Robert Dautray and Jacques-Louis Lions. Spectral theory and appli-
cations, volume 3 of Mathematical analysis and numerical methods for
science and technology. Springer-Verlag, New York, NY, 1985.

[33] Tim Davis. Summary of available software for sparse direct methods.
http://www.cise.ufl.edu/research/sparse/codes/, April 2012.

[34] James W. Demmel. Applied numerical linear algebra. SIAM, Philadel-

phia, PA, 1997.

[35] James W. Demmel, Stanley C. Eisenstat, John R. Gilbert, Xiaoye S. Li,

and Joseph W. H. Liu. A supernodal approach to sparse partial pivot-
ing. SIAM Journal on Matrix Analysis and Applications, 20(3):720–755,
1999.

[36] Jack J. Dongarra, Jeremy Du Croz, Sven Hammarling, and Iain S. Duff.
A set of Level 3 Basic Linear Algebra Subprograms. ACM Transactions
on Mathematical Software, 16(1):1–17, 1990.

189
[37] Jack J. Dongarra, Jeremy Du Croz, Sven Hammarling, and Richard J.
Hanson. An extended set of Fortran Basic Linear Algebra Subprograms.
ACM Transactions on Mathematical Software, 14(1):1–17, 1988.

[38] Jack J. Dongarra, Robert A. van de Geijn, and David W. Walker. A

look at scalable dense linear algebra libraries. In Proceedings of Scalable
High Performance Computer Conference, pages 372–379, Williamsburg,
VA, 1992.

[39] Jack J. Dongarra and David W. Walker. MPI: a standard message

passing interface. Supercomputer, 12(1):56–68, 1996.

[40] Alex Druinsky and Sivan Toledo. How accurate is inv(a) ∗ b? ArXiv
e-prints, January 2012.

[41] Iain S. Duff and John K. Reid. The multifrontal solution of indefinite
sparse symmetric linear equations. ACM Transactions on Mathematical
Software, 9:302–325, 1983.

[42] Björn Engquist and Andrew Majda. Absorbing Boundary Conditions

for the numerical simulation of waves. Mathematics of Computation,
31:629–651, 1977.

[43] Björn Engquist and Lexing Ying. Sweeping preconditioner for the
Helmholtz equation: hierarchical matrix representation. Communica-
tions on Pure and Applied Mathematics, 64:697–735, 2011.

[44] Björn Engquist and Lexing Ying. Sweeping preconditioner for the
Helmholtz equation: moving perfectly matched layers. SIAM Journal
on Multiscale Modeling and Simulation, 9:686–710, 2011.

190
[45] Yogi A. Erlangga. Advances in iterative methods and preconditioners
for the Helmholtz equation. Archives of Computational Methods in
Engineering, 15:37–66, 2008.

[46] Yogi A. Erlangga, Cornelis Vuik, and Cornelis W. Oosterlee. On a

class of preconditioners for solving the Helmholtz equation. Applied
Numerical Mathematics, 50:409–425, 2004.

[47] Oliver G. Ernst and Martin J. Gander. Why it is difficult to solve

Helmholtz problems with classical iterative methods. In I. Graham,
T. Hou, O. Lakkis, and R. Scheichl, editors, Numerical Analysis of Mul-
tiscale Problems, pages 325–363, New York, NY, 2011. Springer-Verlag.

[48] C. M. Fiduccia and R. M. Mattheyses. A linear time heuristic for

improving network partitions. In Proceedings of 19th IEEE Design Au-
tomation Conference, pages 175–181. IEEE, 1982.

[49] Bernd Fischer and Roland Freund. On the constrained Chebyshev ap-
proximation problem on ellipses. Journal of Approximation Theory,
62(3):297–315, 1990.

[50] Bernd Fischer and Roland Fruend. Chebyshev polynomials are not
always optimal. Journal of Approximation Theory, 65:261–272, 1991.

[51] Donald A. Flanders and George Shortley. Numerical determination of

fundamental modes. Journal of Applied Physics, 21(1326):1326–1332,
1950.

[52] Martin J. Gander and Frédéric Nataf. AILU for Helmholtz problems: a
new preconditioner based on the analytic parabolic factorization. Jour-
nal of Computational Acoustics, 9:1499–1506, 2001.

191
[53] M. R. Garey, David S. Johnson, and Larry J. Stockmeyer. Some sim-
plified NP-complete problems. In Proceedings of the sixth annual ACM
symposium on Theory of computing, pages 47–63, New York, NY, 1974.
ACM.

[54] Alan George. Nested dissection of a regular finite element mesh. SIAM
Journal on Numerical Analysis, 10:345–363, 1973.

[55] Alan George and Joseph W. H. Liu. An optimal algorithm for sym-
bolic factorization of symmetric matrices. SIAM Journal on Computing,
9:583–593, 1980.

[56] Alan George, Joseph W. H. Liu, and Esmond G. Ng. Communication

reduction in parallel sparse Cholesky factorization on a hypercube. In
Michael T. Heath, editor, Hypercube Multiprocessors, pages 576–586,
Philadelphia, PA, 1987. SIAM.

[57] John R. Gilbert, Joan P. Hutchinson, and Robert Endre Tarjan. A

separator theorem for graphs of bounded genus. Journal of Algorithms,
5(3):391–407, 1984.

[58] Gene H. Golub and Charles F. van Loan. Matrix computations. Johns
Hopkins University Press, Baltimore, MD, 1996.

[59] Lars Grasedyck and Wolfgang Hackbusch. Construction and arithmetics

of H-matrices. Computing, 70(4):295–334, 2003.

[60] John A. Gunnels, Fred G. Gustavson, Greg M. Henry, and Robert A.

van de Geijn. FLAME: Formal Linear Algebra Methods Environment.
ACM Transactions on Mathematical Software, 27(4):422–455, 2001.

192
[61] Anshul Gupta. Analysis and design of scalable parallel algorithms for
scientific computing. Ph.D. Dissertation, University of Minnesota, Min-
neapolis, Minnesota, 1995.

[62] Anshul Gupta and Mahesh V. Joshi. WSMP: A high-performance

shared- and distributed-memory parallel sparse linear equations solver.
IBM Research Report RC 22038 (98932), 2001.

[63] Anshul Gupta, George Karypis, and Vipin Kumar. A highly scalable
parallel algorithm for sparse matrix factorization. IEEE Transactions
on Parallel and Distributed Systems, 8(5):502–520, 1997.

[64] Anshul Gupta, Seid Koric, and Thomas George. Sparse matrix factor-
ization on massively parallel computers. In Proceedings of Conference
on High Performance Computing Networking, Storage, and Analysis,
number 1, New York, NY, 2009. ACM.

[65] Anshul Gupta and Vipin Kumar. Parallel algorithms for forward elimi-
nation and backward substitution in direct solution of sparse linear sys-
tems. In Proceedings of Supercomputing, San Diego, CA, 1995. ACM.

[66] Wolfgang Hackbusch. A sparse matrix arithmetic based on H-matrices.

I. Introduction to H-matrices. Computing, 62(2):89–108, 1999.

[67] Paul R. Halmos. Finite-dimensional vector spaces. Undergraduate

Texts in Mathematics. Springer-Verlag, New York, NY, 1987.

[68] Michael T. Heath and Padma Raghavan. LAPACK Working Note 62:
distributed solution of sparse linear systems. Technical Report UT-CS-
93-201, University of Tennessee, Knoxville, TN, 1993.

193
[69] Bruce A. Hendrickson and Robert W. Leland. The Chaco Users Guide:
Version 2.0. Sandia Technical Report SAND94–2692, 1994.

[70] Bruce A. Hendrickson and Robert W. Leland. A multi-level algorithm

for partitioning graphs. In Proceedings of Supercomputing, San Diego,
CA, 1995. ACM.

[71] Bruce A. Hendrickson and David E. Womble. The torus-wrap mapping

for dense matrix calculations on massively parallel computers. SIAM
Journal on Scientific and Statistical Computing, 15:1201–1226, 1994.

[72] Pascal Henon, Pierre Ramet, and Jean Roman. PaStiX: A parallel
sparse direct solver based on static scheduling for mixed 1D/2D block
distributions. In Proceedings of Irregular’2000 workshop of IPDPS, vol-
ume 1800 of Lecture Notes in Computer Science, pages 519–525, Cancun,
Mexico, 2000. Springer Verlag.

[73] Nicholas J. Higham. Accuracy and stability of numerical algorithms.

SIAM, Philadelphia, PA, 1996.

[74] Roger A. Horn and Charles R. Johnson. Matrix analysis. Cambridge

University Press, Cambridge, NY, 1985.

[75] Bruce M. Irons. A frontal solution program for finite element analysis.
International Journal for Numerical Methods in Engineering, 2:5–32,
1970.

[76] Dror Irony and Sivan Toledo. Trading replication for communication in
parallel distributed-memory dense solvers. Parallel Processing Letters,
pages 79–94, 2002.

194
[77] James Jeans. The mathematical theory of electricity and magnetism.
Cambridge University Press, 1908.

[78] Steven G. Johnson. Notes on Perfectly Matched Layers (PMLs). Tech-

nical report, Massachusetts Institute of Technology, Cambridge, MA,
2010.

[79] Mahesh V. Joshi, Anshul Gupta, George Karypis, and Vipin Kumar. A
high performance two dimensional scalable parallel algorithm for solv-
ing sparse triangular systems. In Proceedings of the Fourth Interna-
tional Conference on High-Performance Computing, HIPC ’97, pages
137–, Washington, DC, 1997. IEEE Computer Society.

[80] Mahesh V. Joshi, George Karypis, Vipin Kumar, Anshul Gupta, and
Fred Gustavson. PSPASES: Building a high performance scalable par-
allel direct solver for sparse linear systems. In Tianruo Yang, editor,
Parallel Numerical Computations with Applications, pages 3–18, Nor-
well, MA, 1999. IEEE.

[81] George Karypis and Vipin Kumar. A fast and high quality multilevel
scheme for partitioning irregular graphs. SIAM Journal on Scientific
Computing, 20(1):359–392, 1998.

[82] George Karypis and Vipin Kumar. A parallel algorithm for multilevel
graph partitioning and sparse matrix ordering. Parallel and Distributed
Computing, 48:71–85, 1998.

[83] Brian W. Kernighan and Shen Lin. An efficient heuristic procedure for
partitioning graphs. Bell Systems Technical Journal, 49:291–307, 1970.

195
[84] Erwin Kreyszig. Introductory functional analysis with applications. Wi-
ley, 1978.

[85] Cornelius Lanczos. An iteration method for the solution of the eigen-
value problem of linear differential and integral operators. Journal of
Research of the National Bureau of Standards, 45:255–282, 1950.

[86] C. L. Lawson, R. J. Hanson, D. R. Kincaid, and F. T. Krogh. Basic

Linear Algebra Subprograms for Fortran usage. ACM Transactions on
Mathematical Software, 5(3):308–323, 1979.

[87] Xiaoye S. Li. An overview of SuperLU: Algorithms, implementation,

and user interface. ACM Transactions on Mathematical Software, 31(3):302–
325, 2005.

[88] Xiaoye S. Li and James W. Demmel. A scalable sparse direct solver

using static pivoting. In Proceedings of the SIAM Conference on Parallel
Processing for Scientific Computing, page 10, March 22-24, 1999.

[89] Xiaoye S. Li, James W. Demmel, John R. Gilbert, Laura Grigori, Meiyue
Shao, and Ichitaro Yamazaki. SuperLU Users’ Guide. Technical Report
LBNL-44289, October 2011.

[90] Robert J. Lipton and Robert Endre Tarjan. A separator theorem for
planar graphs. SIAM Journal on Applied Mathematics, 36(2), 1979.

[91] Joseph W. H. Liu. The multifrontal method for sparse matrix solution:
theory and practice. SIAM Review, 34(1):82–109, 1992.

196
[92] Per-Gunnar Martinsson and Vladimir Rokhlin. A fast direct solver for
scattering problems involving elongated structures. Journal of Compu-
tational Physics, 221(1):288–302, 2007.

[93] Dianne P. O’Leary and G. W. Stewart. Data-flow algorithms for parallel

matrix computations. Communications of the ACM, 28:840–853, 1985.

[94] Chris C. Paige and Michael A. Saunders. Solution of sparse indefi-

nite systems of linear equations. SIAM Journal of Numerical Analysis,
12(4):617–629, 1975.

[95] Antoine Petitet, R. Clint Whaley, Jack J. Dongarra, and Andy Cleary.
HPL - a portable implementation of the High-Performance Linpack bench-
mark for distributed-memory computers. http://www.netlib.org/
benchmark/hpl, 2012.

[96] Rodrigo B. Platte, Lloyd N. Trefethen, and Arno B. J. Kuijlaars. Impos-

sibility of fast stable approximation of analytic functions from equispaced
samples. SIAM Review, 53(2):308–318, 2011.

[97] Alex Pothen and Sivan Toledo. Elimination structures in scientific com-
puting. In Dinesh Mehta and Sartaj Sahni, editors, Handbook on Data
Structures and Applications, pages 1–59.29. CRC Press, 2004.

[98] Jack Poulson, Björn Engquist, Siwei Li, and Lexing Ying. A paral-
lel sweeping preconditioner for heterogeneous 3D Helmholtz equations.
ArXiv e-prints, March 2012.

[99] Jack Poulson, Bryan Marker, Robert A. van de Geijn, Jeff R. Hammond,
and Nichols A. Romero. Elemental: A new framework for distributed

197
 memory dense matrix computations. ACM Transactions on Mathemat-
ical Software, 39(2).

[100] Jack Poulson and Lexing Ying. Clique. http://bitbucket.org/

poulson/clique/, November 2012.

[101] Jack Poulson and Lexing Ying. Parallel Sweeping Preconditioner. http:
//bitbucket.org/poulson/psp/, November 2012.

[102] Padma Raghavan. Efficient parallel sparse triangular solution using

selective inversion. Parallel Processing Letters, 8(1):29–40, 1998.

[103] Padma Raghavan. Domain-Separator Codes for the parallel solution of

sparse linear systems. Technical Report CSE-02-004, The Pennsylvania
State University, University Park, PA, 2002.

[104] Junuthula N. Reddy. An introduction to continuum mechanics with

applications. Cambridge University Press, New York, NY, 2008.

[105] Theodore J. Rivlin. Chebyshev polynomials: From approximation theory

to algebra and number theory. Wiley-Interscience, second edition, 1990.

[106] Walter Rudin. Principles of mathematical analysis. McGraw-Hill, New

York, NY, third edition, 1976.

[107] Yousef Saad. Iterative methods for sparse linear systems. SIAM,
Philadelphia, PA, 2003.

[108] Yousef Saad and Martin H. Schultz. A generalized minimum residual

algorithm for solving nonsymmetric linear systems. SIAM Journal on
Scientific and Statistical Computing, 7(3):856–869, 1986.

198
[109] Martin D. Schatz, Jack Poulson, and Robert A. van de Geijn. Scal-
able Universal Matrix Multiplication: 2D and 3D variations on a theme.
Technical Report, University of Texas at Austin, Austin, TX, 2012.

[110] Phillip G. Schmitz and Lexing Ying. A fast direct solver for elliptic
problems on general meshes in 2D. Journal of Computational Physics,
231:1314–1338, 2012.

[111] Steven H. Schot. Eighty years of Sommerfeld’s radiation condition.

Historia Mathematica, 19:385–401, 1992.

[112] Robert Schreiber. A new implementation of sparse Gaussian elimina-

tion. ACM Transactions on Mathematical Software, 8(3):256–276, 1982.

[113] Robert Schreiber. Scalability of sparse direct solvers. In A. George,

J. R. Gilbert, and J. W. H. Liu, editors, Graph Theory and Sparse Matrix
Computation, pages 191–209, New York, NY, 1993. Springer-Verlag.

[114] David S. Scott. Out of core dense solvers on Intel parallel supercom-
puters. In Proceedings of Frontiers of Massively Parallel Computation,
pages 484–487, 1992.

[115] Laurent Sirgue and R. Gerhard Pratt. Efficient waveform inversion and
imaging: A strategy for selecting temporal frequencies. Geophysics,
69(1):231–248, 2004.

[116] Edgar Solomonik and James W. Demmel. Communication-optimal par-

allel 2.5D matrix multiplication and LU factorization. In Emmanuel
Jeannot, Raymond Namyst, and Jean Roman, editors, Euro-Par 2011
Parallel Processing, volume 6853 of Lecture Notes in Computer Science,
pages 90–109. Springer Berlin Heidelberg, 2011.

199
[117] Elias M. Stein and Rami Shakarchi. Complex analysis. Princeton
Lectures in Analysis. Princeton University Press, Princeton, NJ, 2003.

[118] G. W. Stewart. A Krylov–Schur algorithm for large eigenproblems.

SIAM Journal on Matrix Analysis and Applications, 23(3):601–614, March
2001.

[119] G. W. Stewart. Matrix algorithms Volume II: Eigensystems. SIAM,

Philadelphia, PA, 2001.

[120] Christiaan C. Stolk. A rapidly converging domain decomposition method

for the Helmholtz equation. ArXiv e-prints, August 2012.

[121] Albert Tarantola. Inversion of seismic reflection data in the acoustic

approximation. Geophysics, 49:1259–1266, 1984.

[122] Lloyd N. Trefethen and David Bau. Numerical linear algebra. SIAM,
Philadelphia, PA, 1997.

[123] Paul Tsuji. Fast algorithms for frequency-domain wave propagation.

Ph.D. Dissertation, University of Texas at Austin, Austin, TX, 2012.

[124] Paul Tsuji, Björn Engquist, and Lexing Ying. A sweeping precondi-
tioner for time-harmonic Maxwell’s equations with finite elements. Jour-
nal of Computational Physics, 231(9):3770–3783, 2012.

[125] Paul Tsuji, Björn Engquist, and Lexing Ying. A sweeping precondi-
tioner for Yee’s finite difference approximation of time-harmonic Maxwell’s
equations. Frontiers of Mathematics in China, 7(2):347–363, 2012.

200
[126] Robert A. van de Geijn. Massively parallel LINPACK benchmark on
the Intel Touchstone DELTA and iPSC/860 systems. In Proceedings of
Intel Supercomputer Users Group, Dallas, TX, 1991.

[127] Robert A. van de Geijn and Jerrell Watts. SUMMA: Scalable Universal
Matrix Multiplication Algorithm. Concurrency: Practice and Experi-
ence, 9:255–274, 1997.

[128] Charles F. van Loan. The ubiquitous Kronecker product. Journal of

Computational and Applied Mathematics, 123(1-2):85–100, 2000.

[129] Charles F. van Loan and Nikos Pitsianis. Approximation with Kro-
necker products. In Linear Algebra for Large Scale and Real Time
Applications, pages 293–314. Kluwer Publications, 1993.

[130] Shen Wang, Maarten V. de Hoop, and Jianlin Xia. On 3D modeling

of seismic wave propagation via a structured parallel multifrontal direct
Helmholtz solver. Geophysical Prospecting, 59:857–873, 2011.

[131] Shen Wang, Xiaoye S. Li, Jianlin Xia, Yingchong Situ, and Maarten V.
de Hoop. Efficient scalable algorithms for hierarchically semiseparable
matrices. Technical report, Purdue University, West Lafayette, IN,
2011.

[132] James H. Wilkinson. Rounding errors in algebraic processes. Prentice-

Hall, Englewood Cliffs, NJ, 1963.

[133] James H. Wilkinson. The algebraic eigenvalue problem. Numerical

Mathematics and Scientific Computation. Oxford University Press, Ox-
ford, England, 1965.

201
[134] Jianlin Xia, Shiv Chandrasekaran, Ming Gu, and Xiaoye S. Li. Superfast
multifrontal method for large structured linear systems of equations.
SIAM Journal on Matrix Analysis and Applications, 31(3):1382–1411,
2009.

202