Maeda et al.

Earth, Planets and Space (2017) 69:102

DOI 10.1186/s40623-017-0687-2

TECHNICAL REPORT Open Access

OpenSWPC: an open‑source integrated

parallel simulation code for modeling seismic
wave propagation in 3D heterogeneous
viscoelastic media
Takuto Maeda1* , Shunsuke Takemura2 and Takashi Furumura1

Abstract
We have developed an open-source software package, Open-source Seismic Wave Propagation Code (OpenSWPC),
for parallel numerical simulations of seismic wave propagation in 3D and 2D (P-SV and SH) viscoelastic media based
on the finite difference method in local-to-regional scales. This code is equipped with a frequency-independent
attenuation model based on the generalized Zener body and an efficient perfectly matched layer for absorbing
boundary condition. A hybrid-style programming using OpenMP and the Message Passing Interface (MPI) is adopted
for efficient parallel computation. OpenSWPC has wide applicability for seismological studies and great portability to
allowing excellent performance from PC clusters to supercomputers. Without modifying the code, users can conduct
seismic wave propagation simulations using their own velocity structure models and the necessary source represen-
tations by specifying them in an input parameter file. The code has various modes for different types of velocity struc-
ture model input and different source representations such as single force, moment tensor and plane-wave incidence,
which can easily be selected via the input parameters. Widely used binary data formats, the Network Common Data
Form (NetCDF) and the Seismic Analysis Code (SAC) are adopted for the input of the heterogeneous structure model
and the outputs of the simulation results, so users can easily handle the input/output datasets. All codes are written in
Fortran 2003 and are available with detailed documents in a public repository.
Keywords: Seismic waves, Numerical simulation, Finite difference method, Parallel computing, Open-source software

Background models and high-performance computer systems have

The numerical simulation of seismic wave propagation enabled the use of 3D numerical simulations as a practi-
is a fundamental tool for seismological studies such as cal tool for regular data processing studies with currently
estimation of the heterogeneous velocity structure (e.g., available parallel computers.
Tape et al. 2009; Chen and Lee 2015), the study of seis- Of the wide variety of numerical methods used to
mic source processes (e.g., Lee et al. 2006; Imperatori and simulate seismic wave propagation, the staggered-grid
Gallovič 2017), wave propagation in the heterogeneous finite difference method (FDM) has been widely used
Earth (e.g., Frankel and Clayton 1986; Emoto et al. 2010) in the seismological community. The staggered-grid
and hazard assessment (e.g., Graves et al. 2010; Maeda FDM with second-order accuracy in space and time to
et al. 2016). Significant improvements in numerical sim- simulate seismic waves was first proposed by Madariaga
ulation techniques, high-resolution velocity structure (1976); then, it was expanded to fourth-order accuracy
in space by Levander (1988). The 3D FDM simulation
of seismic wave propagation at regional scales has been
*Correspondence: [email protected]‑tokyo.ac.jp
1
conducted after Olsen et al. (1995) and Graves (1996).
Earthquake Research Institute, The University of Tokyo, 1‑1‑1 Yayoi,
Bunkyo 113‑0032, Tokyo, Japan
Since then, the FDM simulations have been widely used
Full list of author information is available at the end of the article for the numerical modeling of seismic wave propagation

© The Author(s) 2017. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License
(http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium,
provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license,
and indicate if changes were made.
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 2 of 20

(see Moczo et al. 2014 and references therein) due to The development of OpenSWPC was in particular
the relative simplicity of its numerical algorithm as well intended to improve its usability for non-experts who are
as its computational efficiency in parallel computing. not so familiar with numerical simulation techniques.
Multiple available FDM codes, such as GMS (Aoi and For this purpose, OpenSWPC combines the necessary
Fujiwara 1999; Aoi et al. 2004), FDMPI (Bohlen 2002), pre- and post-processing functions in its main simu-
AWP-ODC (Cui et al. 2013), FDSim (Moczo et al. 2014) lation code (Fig. 1), and all the necessity information
and SW4 (Petersson and Sjogreen 2014), have been for the simulations are defined in input text parameter
used to numerically model seismic wave propagation. files. Therefore, users do not need to modify the code
Many other available community codes including other to apply it to individual simulation targets. We adopt
numerical methods such as the spectral-element method de facto standard formats for the binary input and out-
are described in Igel (2016). put datasets; the velocity structure input model is given
In this study, we developed a new staggered-grid FDM in the Network Common Data Form (NetCDF; Rew and
code to model seismic waves in 3D and 2D viscoelastic Davis 1990), and the simulation results are output in the
media in local-to-regional scale. By improving the usa- NetCDF and the Seismic Analysis Code (SAC; Goldstein
bility of the code, called the Open-source Seismic Wave et al. 2003; Helffrich et al. 2013) formats. Following the
Propagation Code (OpenSWPC), the numerical simula- input parameters, OpenSWPC dynamically allocates
tion with parallel computing of a wide variety of targets the computer memory, automatically generates grids for
is made accessible to users. The original version of this a heterogeneous velocity model and performs parallel
code, designed to be implemented on supercomputers numerical simulations of seismic wave propagation.
(Furumura and Chen 2005; Inoue et al. 2013), was fully In the following, we briefly review the strategy of the
restructured to have excellent performance on a variety FDM simulation of seismic wave propagation in hetero-
of computer architectures to be fully open to the seismo- geneous viscoelastic media with frequency-independent
logical community. attenuation and the numerical techniques adopted in the

Input Parameter/Data OpenSWPC Output Data

Controll files (text) Pre-Processing Waveform at stations (SAC)
map projection 400
34˚N

33˚N

300
epicenter distance [km]

32˚N

31˚N

200
30˚N

128˚E 129˚E 130˚E 131˚E 132˚E 133˚E 134˚E 135˚E 136˚E

parallel model generation 100

0
0 40 80 120 160 200
Layered velocity model (NetCDF) elapsed time [s]

Snapshots of wavefield (NetCDF)

Staggered-Grid FDM Simulation

Post-Processing
data gathering
binary data formatting

Fig. 1 A schematic illustration of the flow of the wave propagation simulation using OpenSWPC
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 3 of 20

present code. Then, examples of numerical simulations As a consequence of introducing the viscoelasticity, the
are demonstrated to show how the present code works. phase speeds of the body waves become a function of the
frequency due to the physical dispersion effect (Aki and
Methods Richards 2002). Therefore, they follow the given velocity
In this section, we briefly introduce the staggered-grid structure at a given reference frequency fR.
FDM method for simulating seismic waves in viscoelastic In the actual Earth medium, QP and QS do not signifi-
media and the algorithms adopted in OpenSWPC. The cantly depend on frequency over a wide frequency range
details of theories for viscoelasticity and numerical algo- below ∼1 Hz. To yield an approximately constant Q over
rithms can be found in Moczo et al. (2014) and Igel (2016). a frequency range, we introduced memory variables fol-
lowing Robertsson et al. (1994) with the τ-method pro-
Seismic wave modeling with a viscoelastic body posed by Blanch et al. (1995). The τ-method assumes that
We solve the equation of motion of continuum mechan- the ratios between the relaxation and creep times are
ics described by the velocity and stress components in constant among all Zener bodies (ℓ = 1, . . . , L):
the Cartesian coordinate system such that
τℓǫP τℓǫS
∂vi D N
∂σij τP ≡ − 1, τS ≡ − 1. (4)
τℓσ τℓσ

ρ
∂t
=
∂xj
+ fi , (1)
j=1
From a given set of relaxation times, the τ-method gives
where ND is the model dimension (ND = 2 or 3), vi is the optimized values for the parameters τ P and τ S using the
particle velocity of the elastic motion in the ith compo- least squares method, so that the attenuation QP and QS
nent, ρ is the density, σij is the i, jth component of the become approximately constant over a given frequency
stress tensor and fi is the ith component of the body range. The constitutive equation based on this τ-method
force. In the following, we primarily describe the three- is written as follows:
dimensional case, ND = 3; however, it is straightforward
to reproduce the two-dimensional P-SV or SH formulae ∂σii     
= (R + 2µR ) 1 + τ P − 2µR 1 + τ S
from this case. ∂t
3 L
The stress is related to the particle velocity via the con-  ∂vk   ∂v
i

stitutive equation. In this study, we adopted the General- × + 2µR 1 + τ S + riiℓ
∂xk ∂xi
k=1 ℓ=1
ized Zener Body (GZB) (e.g., JafarGandomi and Takenaka  ∂v 
2007; Maeda et al. 2013) of the viscoelasticity, which is ∂σij 
i ∂vj
= µR 1 + τ S +
expressed as a parallel connection of Zener bodies (e.g., ∂t ∂xj ∂xi
Aki and Richards 2002, Chapter 5) having different physi- L
(5)

cal constants. Note that the GZB is known to be equiva- + rijℓ , (for i � = j),
lent to the generalized Maxwell body (Moczo and Kristek ℓ=1
2005; Cao and Yin 2014), which is also widely used for
seismic wave modeling (e.g., Emmerich and Korn 1987). where rijℓ is the memory variable of the (i, j)th compo-
Following Robertsson et al. (1994), the constitutive nent of the stress tensor and the ℓth Zener body and sat-
equation of the viscoelastic body is written as isfy the following auxiliary equation:
 
σij = ψ̇π (t) − 2ψ̇µ (t) ∗ δij ǫkk + 2ψ̇µ (t) ∗ ǫij , (2)

∂riiℓ 1 1  
= − σ riiℓ − σ (R + 2µR )τ P − 2µR τ S
where ǫij is the i, jth component of the strain tensor, ψ̇π (t) ∂t τℓ Lτℓ
and ψ̇µ (t) are the time derivatives of the relaxation func- 3

tions for two independent relaxed moduli πR ≡ R + µR ×
 ∂vk
+ 2µR τ S
∂vi
and µR, respectively. For the GZB, they are represented k=1
∂xk ∂xi
using the L different relaxation times τℓσ (ℓ = 1, . . . , L)  
∂rijℓ 1 1 ∂vi ∂vj
and the creep times of the P- and S-waves (τℓǫP , τℓǫS , = − σ rijℓ − σ µR τ S +
respectively) such that ∂t τℓ Lτℓ ∂xj ∂xi
(6)
(for i � = j).
L
   
1 τℓǫP −t/τ σ
ψ̇π (t) = πR 1 − 1− σ e ℓ H (t)
L τℓ The selection of the relaxation times, τℓσ, is not a triv-
ℓ=1
 L
   (3) ial issue in the τ-method. We adopted logarithmically
1 τℓǫS −t/τ σ spaced relaxation times in the given frequency range.
ψ̇µ (t) = µR 1 − 1− σ e ℓ H(t).
L
ℓ=1
τℓ As in an example in Fig. 2, the adopted model can yield
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 4 of 20

0.02

vz
0.01

σxz σyz
0.005
Q−1

0.002 σii
vx vy
0.001 σxy

0.0005
x
y
z
0.0002
10−3 10−2 10−1 100 101 10 2 Fig. 3 Layout of the three-dimensional Cartesian coordinate
staggered-grid adopted in this study
frequency [Hz]
Fig. 2 An example of a nearly constant Q model using three Zener
bodies (bold black curve), considering a constant Q (=100) in the modulus is necessary when evaluating the shear stress
frequency range between 0.02 and 2 Hz. Dotted lines show the contri-
components and the accompanying memory variables
butions from three Zener bodies
(Eqs. 5, 6). For the averaging, we adopted an arithmetic
and a harmonic averaging for the density and the relaxed
an approximately constant Q over two orders of the rigidity modulus, respectively. Notice that no averaging is
frequency range below 2 Hz using three Zener bodies necessary for R because it is used only for updating the
(L = 3). Frequency range of the approximately constant normal stress components.
Q is adjustable by input parameter of the code.
Boundary conditions
Discretization The free surface and ocean bottom boundary condi-
The equation of motion, Eq. (1), and the constitutive tions are implemented in the FDM simulation based on
equations, Eqs. (5) and (6), are solved numerically based the efficient Heterogeneity, Oceanic layer and Topogra-
on a staggered-grid FDM with fourth-order accuracy in phy (HOT) FDM method (Nakamura et al. 2012). This
space and second-order accuracy in time (e.g., Levander method was originally developed for a second-order
1988). We adopt the Cartesian coordinate system with FDM method to include the topographic variations of
x1 = x and x2 = y as the horizontal direction and vertical volcanoes (Ohminato and Chouet 1997). Later, it was
axis x3 = z was taken to be positive downward with aver- shown that this condition could be applied effectively for
age sea level height at z = 0. Figure 3 shows the layout of fluid–solid boundaries (Okamoto and Takenaka 2005).
the staggered-grid adopted in this study. Currently, HOT-FDM is widely applied in regional-scale
We adopted the first-order Crank–Nicolson method to modeling of seismic wave propagation in coastal areas
calculate the time integration of the constitutive equa- (e.g., Maeda and Furumura 2013; Maeda et al. 2013, 2014;
tion, Eq. (5), and the auxiliary equation for the memory Nakamura et al. 2012, 2015; Noguchi et al. 2016; Todoriki
variables, Eq. (6). This enables us to solve the original et al. 2017), in high-frequency seismic wave simulations of
implicit scheme equations explicitly. The discretized for- topographic scattering (Takemura et al. 2015), and in the
mula can be found in Maeda et al. (2013). synthesis of elastic wave propagation in cylinder-shaped
The evaluation of the medium properties defined in materials on the scale of 10 cm (Yoshimitsu et al. 2016),
the staggered-grid system requires appropriate averag- demonstrating the ability of the HOT-FDM to include
ing of the medium parameters defined on the neighbor- irregular topography and curved surfaces in FDM.
ing grids. In this code, all medium properties (relaxed In this HOT-FDM method, the air column is treated as
medium parameters R and µR, density ρ, and attenuation a medium with a very small density and zero wavespeeds
parameters τ P and τ S) are defined on the same grid as for the P- and S-waves as αR = βR = 0 [km/s]. This col-
the normal stress components (Fig. 3). Averaging of the umn is treated as a vacuum where no seismic wave can
density on neighboring grids is necessary when evaluat- propagate due to zero wavespeeds. The ocean column
ing the velocity, Eq. (1), and averaging the relaxed rigidity is treated as an elastic medium having ρ = 1.0 [g/cm3],
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 5 of 20

αR = 1.5 [km/s] and βR = 0.0 [km/s]. On the free surface using multiple point sources (Graves and Wald 2001;
and seafloor, a reduced, second-order FDM scheme is Takenaka and Fujii 2008). In this code, there is no
applied instead of the fourth-order FDM. To apply such restriction on the number of point sources; the code
a reduced-order FDM across the boundaries, the location automatically detects the number of sources and allo-
of the boundaries are detected automatically in Open- cates the required memory. We also implemented a sin-
SWPC by searching for the grid positions where µR and gle-force source that is widely used for modeling seismic
R change from zero to a finite value. signals excited in volcanic environments (e.g., Ohmi-
Except in whole-Earth seismic wave modeling, an nato et al. 1998). Both moment tensor and single-force
appropriate absorbing boundary condition surrounding sources are placed on the nearest grid point of the nor-
the bounded model is necessary to avoid artificial reflec- mal stress component. This source grid point is averaged
tions from the boundaries of the computational model. with the neighboring source grid points if necessary
We adopt the Perfectly Matched Layer (PML) boundary when updating the shear stress and velocity components
condition (after Chew and Liu 2011) to minimize artifi- (Coutant et al. 1995).
cial reflections. Up to now, various PML methods have The seismic source time function is represented by
been developed (for example, Kristek et al. 2009 and bell-like functions (Fig. 4) of various forms such as box-
references therein); we adopted an implementation pro- car, triangle and cosine functions, Herrmann’s quadratic
posed by Zhang and Shen (2010) to consider its effective- function (Herrmann 1979) and Küpper’s wavelet of a sin-
ness in and applicability to OpenSWPC. gle cycle (Mavroeidis and Papageorgiou 2003). They have
This PML method solves the auxiliary differential equa- a common cutoff frequency, which is a reciprocal of the
tions in addition to the equations of motion and the con- source duration time, but have different roll-offs above
stitutive equations in the absorbing zone with complex the cutoff frequency.
and frequency-shifted absorbing functions. Typically The incidence of plane P- or S-waves with near-vertical
10–20 grids are used for the thickness of the PML zone angles was also implemented in OpenSWPC. The plane-
surrounding the model; this can be defined in the input wave incidence uses a solution of the up-going wave of
parameters. To avoid large computational loads when the 1D wave equation in a homogeneous medium as an
solving the auxiliary equations of the PML, we assumed initial condition of the velocity and stress components
that the material in the PML zone was perfectly elastic at the bottom of the model (e.g., Emoto et al. 2010). A
and did not calculate the memory variables (Eq. 6), for bell-like-shaped spatial distribution corresponding to the
the anelasticity. source time function (Fig. 4) is assumed as initial condi-
Even though the PML efficiently absorbs outgoing tions. Even though the OpenSWPC allows a non-vertical
waves from the model, it occasionally causes severe incidence angle to be set for the plane wave, note that
instabilities during the calculation, in particular when artificial reflections may arise at both sides of the plane
the seismic waves propagate in highly heterogeneous wave traveling in the absorbing boundaries with increas-
structures with very large velocity contrasts (Maeda et al. ing plane-wave incidence angles.
2013). Therefore, we also implemented the stable sponge Simulations exchanging source and station positions
boundary condition of Cerjan et al. (1985), which simply are sometimes conducted because the reciprocity theo-
attenuates the waves in the absorbing layer by multiply- rem (e.g., Aki and Richards 2002) implies the equivalence
ing small values to the stress and velocity components of both simulation results under certain conditions. The
in the absorbing zone at each time step so that users can use of the reciprocity theorem is very effective to obtain
select the appropriate boundary condition. Note that a large number of synthetic seismograms for a small
the sponge condition is perfectly stable but less efficient number of stations when there is a very large number of
when absorbing outgoing waves. source grids, e.g., when calculating Green’s function with
reduced computational cost (e.g., Graves and Wald 2001;
Seismic source representation Eisner and Clayton 2001; Zhao et al. 2006; Hsieh et al.
The seismic moment tensor source can be implemented 2016; Petukhin et al. 2016). OpenSWPC is equipped with
in the FDM either by couples of body forces (Graves this feature. In this mode, a single-force source is placed
1996) or based on a stress discontinuity representa- at a specified station grid to export displacement wave-
tion (Coutant et al. 1995; Pitarka 1999). In OpenSWPC, forms and spatial derivatives of displacement at the mul-
the stress discontinuity representation was adopted to tiple source grid locations. Using a source having much
implement the moment tensor sources. The seismic shorter source time functions compared to the dominant
source can be implemented in OpenSWPC with a sin- period of the simulation target, the simulation results can
gle point source. A finite-fault source can be represented be used as Green’s functions.
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 6 of 20

a Boxcar b Triangle

100 100
2 2
10−1 10−1

10−2 10−2

1 10−3 1 10−3

10−4 10−4

10−5 10−5
0 0
10−6 10−6
0.0 0.5 1.0 0.1 1 10 0.0 0.5 1.0 0.1 1 10

c Herrmann d Cosine

100 100
2 2
10−1 10−1

10−2 10−2

1 10−3 1 10−3

10−4 10−4

10−5 10−5
0 0
10−6 10−6
0.0 0.5 1.0 0.1 1 10 0.0 0.5 1.0 0.1 1 10

e Küpper f t−exp

100 100
2 2
10−1 10−1

10−2 10−2

1 10−3 1 10−3

10−4 10−4

10−5 10−5
0 0
10−6 10−6
0.0 0.5 1.0 0.1 1 10 0.0 0.5 1.0 0.1 1 10
t/TR f TR t/TR f TR
Fig. 4 Implemented source time functions and their amplitude spectra: a boxcar function, b triangle function, c Herrmann function, d cosine func-
tion, e Küpper’s wavelet and f t exp[−t] function. In each panel, the left figure shows the temporal variation of the source time function and the right
figure shows the amplitude spectrum. The horizontal axes are normalized by the rise time TR

Software implementations libraries of Message Passing Interface (MPI) and NetCDF

Parallel computing for parallel computing and data I/O, respectively.
All codes in OpenSWPC were written following to the Note that these external libraries also work in many
standards of Fortran 2003 (e.g., Adams et al. 2008) with environments.
very few compiler-unique functions; therefore, this code In OpenSWPC, all definitions such as parallelization
should work in a variety of computational environ- strategies, model size, discretization and output file-
ments without modification. This code uses the external names are defined in the input parameter files. Because
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 7 of 20

the computational memory is allocated dynamically, users to reduce the data communication overhead (Furumura
do not need to re-compile the code to perform simula- and Chen 2005).
tions with different model sizes. OpenSWPC adopts a The simulations are performed with a mix of single-
domain-partitioning procedure for the parallel comput- and double-floating-point-precision arithmetic. Static
ing using MPI. It performs 2D and 1D model partition- parameters such as the medium velocities or density are
ing in the horizontal directions for the 3D and 2D codes, defined in single precision; however, the FDM calculation
respectively (see Fig. 5 for the partitioning for the 3D and the stress discontinuity at the sources are evaluated
code). Then, OpenSWPC sets up the velocity structure more accurately in double precision. We found that sim-
model for the partitioned domain on each CPU or CPU ulations in single precision often lead to numerical insta-
core, and calculates the seismic wave propagation in the bility after long-time step calculations, as demonstrated
domain using MPI data communication at each time step. in Fig. 6. The panels in Fig. 6a show snapshots of the par-
To maintain wavefield continuities across the par- ticle velocity amplitude in the seismic wavefield obtained
titioned domain, the velocity and stress components by a 2D P-SV numerical simulation using single-precision
defined at the two-point-thick outermost layer are arithmetic. After a long elapsed time, it was confirmed
exchanged with those of the neighboring partitioned that a randomly oscillating noise was continuously being
domains at every time step using MPI (Fig. 5b). We radiated from the source location. Such unstable noise is
adopted hybrid-style programming using OpenMP and suspected to occur due to the wide dynamic range of the
MPI (e.g., Furumura and Chen 2005) to make full use of wavefield amplitude near the seismic source, which can-
many CPUs with multiple CPU cores; directive-based not be handled accurately using single-precision arith-
OpenMP parallel programming is applied to parallel- metic. The accumulation of this small error over long
ize the computation among the CPU cores, and the MPI time steps may cause problems or instabilities in the
data communication between the CPUs is performed simulation. The use of double-precision arithmetic can
explicitly. Compared to the flat-MPI application which effectively avoid this problem (Fig. 6b); however, it leads
explicitly exchanges data for all cores, this hierarchical to a considerable increase in the memory usage and com-
OpenMP/MPI parallel programming method is expected putation time. OpenSWPC has an option to evaluate all

nodes
a nproc_y
np b
roc
_x
no
de
s
node (nproc_x, 1)

node (4, 1)

node (ip+1, jp)

node (1, npoc_y)

MPI node (ip+1,jp+1)

node (3, 1)

node (1,4)
node (2, 1)

node (1,3)
node (1,2)
node (1,1)
node (1, 1)

MPI MPI

node (ip, jp) MPI node (ip, jp+1)

MP
Id
ata
co x
x mm
y un ation s
ica
tio ta co mmunic y
ns MPI da
z
Fig. 5 a A schematic illustration of the domain partition model for MPI data communication in a 3D simulation. b A schematic illustration of the
data communication strategy using MPI. (Reproduced from figure B1 of Maeda et al. (2013) with modification. Copyright by the Seismological
Society of America)
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 8 of 20

a Single Precision
0
t=050 s t=500 s

50

100
z [km]

150

200

b Mixed Precision
0
t=050 s t=500 s

50

100
z [km]

150

200

−100 −50 0 50 100 −100 −50 0 50 100

x [km] x [km]

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
velocity amplitude at t=50 s [m/s] velocity amplitude at t=500 s [10−4 m/s]
Fig. 6 A comparison of simulations based on a single-precision arithmetic calculations and b mixed-precision calculations. In each frame, the
absolute amplitude of the horizontal components of the velocity at the elapsed times of 50 and 500 s are shown in the color scales. Circles show the
location of the point source, which is an isotropic radiation of seismic waves

calculations using single-precision arithmetic to reduce The topography is treated as a staircase boundary with
the computational cost. a sufficiently small FDM grid. We assume that z = 0 on
the depth axis corresponds to the average sea level; the
Model input topography deeper than zero is treated as the seafloor,
For the 3D simulation of seismic wave propagation, we and the seawater column is filled between the seafloor
consider a layered structure model formed by a set of and z = 0.
velocity layers with varying depths. Each layer describes We adopted NetCDF as the input data format for
the medium discontinuities with the definitions of each layer. The NetCDF format is commonly used in the
medium parameters of density, P- and S-wave velocities, seismological community, such as in Generic Mapping
and attenuation (QP , QS) beneath the layer. The depth of Tools (GMT; Wessel et al. 2013). In OpenSWPC, a set of
the shallow-most layer corresponds to the topography. NetCDF files and a list of these files associated with the
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 9 of 20

medium parameters are used as input to OpenSWPC. The snapshots of the wavefield are stored in NetCDF
Each NetCDF file is defined in geographical coordinates format files. These cross sections can be taken vertically
(longitude and latitude) and contains the depth informa- or horizontally, or along the topography or bathymetry.
tion at each coordinate location. The snapshots contain the three-component velocity or
Even though the input NetCDF files describe the depths the displacement motions, or divergence and rotation of
of the boundary at the locations defined by the geograph- the velocity, which are related to the P- and S-waves.
ical coordinates, the FDM simulation model needs the
depths of the boundary at grids in the Cartesian coordi- Checkpoint and restart
nate system. Therefore, a coordinate transform is neces- Most computer systems restrict the maximum com-
sary to model seismic waves in the Cartesian coordinate puter time for a single simulation job via their job queu-
system. For this purpose, a coordinate transform based ing systems. To cope with this restriction, OpenSWPC
on a power-series expression of the Gauss–Krüger trans- is equipped with a checkpoint–restart function that
form (Kawase 2011) is embedded in OpenSWPC. Users halts the simulation at a specified checkpoint prior to
are requested to input the center of the simulation model the allowed CPU time and exports all data on memory
in longitude and latitude for the coordinate transform, to files. Then, a new job restarts the simulation from the
and the OpenSWPC automatically generates the veloc- checkpoint by reading the stored data files of the previ-
ity structure model in the Cartesian coordinate system. ous job.
Because the grid locations in a Cartesian coordinate sys-
tem are usually not identical to the grid locations of the Two‑dimensional codes
input NetCDF files, a bicubic interpolation is applied to OpenSWPC also provides 2D codes for the P-SV and SH
obtain the depth of the boundary. models. The 2D FDM simulations use the same input
The layered velocity structure model can be superim- parameter file, except they perform the wave propaga-
posed by a random velocity fluctuation described by sta- tion simulations in the x–z plane. A 2D simulation along
tistical characteristics such as the Gaussian or von Kármán a cross section of the 3D medium has a much lighter
power spectrum density functions (Sato et al. 2012), to sim- computational load. The horizontal direction of the cross
ulate the scattering of high-frequency seismic waves in het- section can be chosen arbitrarily by adjusting the coor-
erogeneous structures (e.g., Furumura and Kennett 2005; dinate rotation parameter. Note that the 2D codes adopt
Takemura et al. 2015, 2016). The random velocity pertur- one-dimensional MPI domain partitioning along the x
bation is calculated in wavenumber space based on the direction.
statistical characteristics of the random medium (Klimeš
2002; Sato et al. 2012). Utility programs to calculate these Computation performance
random velocity fluctuations in 2D and 3D are provided Strong‑scale performance measurement
in OpenSWPC. The utility programs provide the random The computational efficiency of the parallel simula-
velocity fluctuation data in the NetCDF format, which can tion was examined using a strong scaling test, where
be used in OpenSWPC. By customizing the model gen- the computation time of a fixed-sized numerical model
eration subroutine in OpenSWPC, it is easy to implement is measured for parallel computing using different CPU
another type of velocity model in OpenSWPC if necessary. numbers. We performed this test on different computer
systems and two different model sizes, a cluster-type
Simulation data output computer of the Earthquake Information Center (EIC)
The simulation results are exported as two types of system of the Earthquake Research Institute at the Uni-
datasets: waveforms at specified station points and 2D versity of Tokyo with up to 36 Intel Xeon E5-2680 v3
snapshots of the seismic wavefield on 2D horizontal (403.2 GFlops) CPUs, and the Earth Simulator (ES)
and vertical cross sections. For the waveforms, veloc- supercomputer of the Japan Agency for Marine–Earth
ity and displacement traces from specified station loca- Science and Technology (JAMSTEC) with up to 2048 the
tions (in either geographical or Cartesian coordinates) NEC SX-ACE (256 GFlops) CPUs.
are exported in the SAC data format. Because the FDM The parallel performance was measured for two model
simulations are performed with much smaller time steps sizes: 512 × 512 × 512 and 1024 × 1024 × 1024 on the
compared to the dominant period of the seismic waves, EIC and 1024 × 1024 × 1024 and 2048 × 2048 × 2048
a temporal decimation is often applied to reduce the on the ES. In these experiments, the average compu-
data size. The decimation factor is specified in the input tation time per time step was normalized by the total
parameter file. Note that the integration of velocity number of grids. The results (Fig. 7) demonstrate a linear
waveforms with respect to time to obtain displacement decrease in the computation time (a linear increase in the
records is performed before the decimation. computational performance) with increasing numbers
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 10 of 20

10−8 (x 10−9)
2

flat−MPI
normalized computation time [s/grid]

hybrid−MPI
10−9

normalized computation time [s/grid]

1

10−10 pe
rfe
ct
st
ro
ng
sc
al 0.5
in
EIC (size: 5123) g
10−11
EIC (size: 10243)

ES (size: 10243)

ES (size: 20483)

10−12
100 101 102 103 104
#CPU 0.2
6 12 18 24 36
Fig. 7 Speedup of the computational time of the 3D simulation code
#CPU
with increasing numbers of CPUs for different model sizes and com-
puter systems. The computation time of each model is normalized by Fig. 8 Comparison of the normalized computation times between
the FDM grid numbers (see text for details) flat-MPI and OpenMP/MPI hybrid parallel computing. The experi-
ments were conducted on the EIC computer (12 Cores/CPU) for the
case of 10243 grids model with different number of CPUs

of CPUs for all models. This parallel performance fol-

lows a near perfect theoretical scaling, except for cases
of over parallelization using very large numbers of CPUs Memory requirements and maximum frequency of the 3D
for the small model simulations (e.g., the result of the FDM
1024 × 1024 × 1024 model on the ES). This result sup- The maximum size of the 3D FDM simulation and the
ports the efficiency of the parallel simulation of Open- highest frequency of the seismic waves in the model are
SWPC using a variety of numbers of CPUs from a small bounded by the size of the computer memory. To con-
(2) to large (2048) number of CPUs. sider the maximum frequency of the simulation, suppose
Figure 8 compares the computational speeds by adopting a simulation model has a square horizontal surface with an
OpenMP/MPI hybrid computation (hybrid-MPI) relative area of S and a depth of D. OpenSWPC requires a memory
to pure MPI (flat-MPI) computation. The measurement of mg = 188 bytes per grid when using mixed-precision
was taken for the model size of 1024 × 1024 × 1024 on arithmetic calculations. Discretizing the 3D computational
the EIC. A CPU of the EIC has 12 CPU cores. For the flat- volume with a uniform grid spacing of h indicates that
MPI parallel computing, the velocity structure model was OpenSWPC requires a total memory size, MR, of
partitioned and assigned into all cores, and explicit data mg SD
communications between CPU cores have been made via MR = . (7)
h3
MPI. On the other hand, the hybrid-MPI computing uti-
lizes the shared memory among CPU cores in the CPU, Further, the spatial grid size h controls the highest fre-
and the MPI data communications were performed only quency allowed for the FDM simulation considering the
for inter-CPU data exchange. The comparison shows con- minimum wavespeed, vmin, such that
siderable improvement in computational performance for vmin
the hybrid-MPI, up to 25% speedup compared to the flat- fmax = , (8)
7h
MPI. The speedup ratio slightly increases with increasing
numbers of CPUs. These performance improvements in where we assume that at least seven grid points per mini-
the hybrid programming are in particular because of the mum wavelength are necessary to restrict the numerical
efficient reuse of the data on the cache of CPU which are dispersion of the FDM calculation within the required
shared among CPU cores (Inoue et al. 2013). We note that level. Combining the above equations (Eqs. 7, 8), the maxi-
the efficiency of the hybrid-MPI may depend on model mum frequency of the 3D FDM, fmax, can be estimated by
configurations such as model size or number of CPUs and the size of the simulation, the minimum wavespeed and
computer architecture. the memory size of the computer Mmax such that
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 11 of 20

vmin

MR
1/3
vmin

Mmax
1/3 synthetic seismograms in laterally homogeneous layered
fmax = ≤ . (9) structures. In this experiment, we assumed the layered
7 mg SD 7 mg SD
structure model of Kubo et al. (2002) and a double-
We evaluated the maximum frequency for the 3D FDM couple point source (strike: 70◦, dip: 30◦ and rake: −50◦)
simulation on the EIC and the ES as a function of hori- placed at a depth of 25 km. The FDM simulation was
zontal distance in the model and minimum wavespeed conducted with a model of 1024 × 1024 × 1024 grid
(Fig. 9). In this evaluation, we assumed a square simulation
√ points, a grid size of 0.5 km and a time step of 0.025 s.
model in the horizontal directions with a size of S and a The reference frequency for Q was set to 1 Hz to match
fixed model depth of D = 200 km as a typical simulation the parameter of the wavenumber integration code. Fig-
model on a regional scale. For example, assuming a mini- ure 10 compares the vertical-component seismograms
mum S-wavespeed of 1.5 km/s, which corresponds to the along the x direction obtained by the wavenumber inte-
P-wavespeed in the ocean column, and a horizontal scale gration method and OpenSWPC. The same band-pass
of 200 km, OpenSWPC can simulate a high-frequency filter of 0.01–0.5 Hz was applied to both traces, consid-
seismic wavefield up to 2 and 10 Hz using the EIC and the ering the expected maximum frequency (∼0.89 Hz) of
ES, respectively. this simulation. There is excellent agreement between
the synthetic seismograms of OpenSWPC and those
Simulation examples of the wavenumber integration method from short to
To demonstrate the effectiveness of OpenSWPC, here we large distances and in the wide frequency range below
show examples of the seismic wave propagation simula- ~1 Hz, demonstrating the effectiveness of OpenSWPC
tions performed by OpenSWPC. for simulations of seismic wave propagation in an elastic
medium.
Comparison with the wavenumber integration method
We first demonstrate the accuracy of the simulation Simulation and visualization with a 3D velocity structure
for a laterally homogeneous medium compared to the The next example demonstrates the result of a simula-
result of the wavenumber integration method using the tion of seismic wave propagation in a 3D heterogeneous
FKRPOG code (Saikia 1994), which is distributed along model of the Japan Integrated Velocity Structure Model
with the TDMT_INVC moment tensor estimation code (JIVSM; Koketsu et al. 2012). The JIVSM consists of a set
(Dreger 2003). The wavenumber integration method is of sedimentary layers, Conrad and Moho discontinuities,
often employed as a reference to validate the accuracy of and Pacific and Philippine Sea Plate interfaces associated

Fig. 9 Maximum frequency of the 3D FDM simulation bounded by the maximum memory size for a the EIC computer (2.2 TB) and b the Earth
Simulator (128 TB) as a function of the horizontal size of the 3D model and the minimum value of the wavespeed in the medium. The color scale is
shown in the right
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 12 of 20

200

180

160

140
distance [km]

120

100

80

60

40

20

0 10 20 30 40 50 60 70
time [s]
Fig. 10 Comparison of vertical-component velocity seismograms obtained by the frequency–wavenumber integral method of Saikia (1994) (black
curve) and OpenSWPC (red curve) in the frequency range of 0.01–0.5 Hz under the laterally homogeneous structure of Kubo et al. (2002). The ampli-
tude at each distance is normalized by the maximum amplitude of the trace obtained by the frequency–wavenumber integral method. Note that
the two traces at the same distance are slightly shifted vertically so that they do not overlap

with oceanic Mohos and topography. These are rep- wave packet is clearly seen in the snapshots of the seis-
resented by the depth variations of each layer and the mic wave propagation shown in Fig. 12. It is demonstrated
medium parameters below each depth. that the direct waves were first radiated from the epicenter
We performed a 3D FDM simulation of the seismic and then were trapped in the very low-wavespeed layer of
wave propagation for the 2005 West Off Fukuoka Pre- the accretionary prism, which was formed by the subduc-
fecture ( MW = 6.6) earthquake. The source location and tion of the Philippine Sea Plate (mark A). Then, part of the
the fault mechanism were taken from the F-net moment trapped seismic wave energy was released from the accre-
tensor catalog (Fukuyama et al. 1998). The simulation tionary prism and propagated back to the Japanese Archi-
model has a spatial grid size of 2000 × 2560 × 500 with a pelago (marks B and C), as demonstrated by Furumura
grid spacing of 0.5 km and a time step of 0.025 s with the et al. (2008) and Guo et al. (2016). As a result, apparently
minimum wavespeed of 1.5 km/s for the shallow-most incoherent arrivals appear in the record section (Fig. 11)
sedimentary layer. We evaluated the seismic wave propa- due to significant off-great-circle propagation.
gation in the long period band using a source time func- Note that the visualized seismic wavefield (Fig. 12) was
tion with a rise time of 20 s. obtained using an associated tool included in the Open-
The record section of the vertical-component veloc- SWPC package. This program reads the wavefield and
ity traces (Fig. 11) at the station locations of High Sensi- topography from a NetCDF-formatted output file of the
tivity Seismograph Network Japan (Hi-net) operated by simulation NetCDF file, and the spatiotemporal wave-
National Research Institute for Earth Science and Disas- fields over the topography map are plotted in sequentially
ter Resilience (Okada et al. 2004) shows very complicated numbered portable bitmap files.
waves as a result of propagation in the heterogeneous
structure. In particular, there are significant wave pack- Finite‑fault rupture and coseismic deformation
ets after the arrival of dispersive surface waves at epicen- The finite source rupture over the fault is represented in
tral distances over 500 km that seem to be incoherent in OpenSWPC using multiple sources. To demonstrate this
neighboring stations. The development of this peculiar feature, we set up a finite-fault model in a homogeneous
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 13 of 20

1000

800
epicenter distance [km]

600
b

600 km
38˚ N

400 km
400

200 km
36˚ N

34˚ N
200

32˚ N

800 km 1000 km
0
0 100 200 300 400 500 132˚ E 136˚ E 140˚ E
elapsed time [s]
Fig. 11 Seismic wave traces simulated by OpenSWPC. a Vertical-component velocity traces in the frequency range of 0.025–0.1 Hz. The amplitude
of each trace is normalized by its maximum amplitude. b An index map showing the station (circles) and source locations. Dashed curves show the
iso-distance contour at every 200 km from the epicenter

half-space
√ as depicted in Fig. 13. A Poisson medium (1985) (Fig. 14). The result of the FDM simulation for the
(α/β = 3) is assumed with an S-wavespeed of coseismic deformation of the finite-fault source over-
β = 3500 m/s and a density of ρ = 2700 kg/m3. No all agrees with the analytic solution as demonstrated in
attenuation was included in the simulation. We previous studies (e.g., Wald and Graves 2001; Maeda
assumed slip with a dip angle of 45◦ and a fault size of and Furumura 2013). Note that the conventional sponge
100 km × 50 km. The slip amount is 7.5 m, which cor- absorbing boundary condition of Cerjan et al. (1985)
responds to a moment magnitude of MW = 8.0. In this leads to an inaccurate estimation of final deformation
simulation, the finite-sized fault was represented by pattern close to the edge of the fault. In this case, this
100 × 50 (=5000) point sources along the strike and dip is most significant in the horizontal dip (y) direction of
directions, respectively. The moment release was equally the fault, and the error grows gradually with increasing
distributed over the point sources assuming homogene- time even after the termination of the fault rupture. Con-
ous slip on the fault. The rupture propagation starts from versely, the estimation using the PML boundary agrees
one corner of the fault and spreads over the fault with an quite well with the analytic solution even if the boundary
assumed constant rupture speed of 2.5 km/s. The source is very close to the fault.
rupture is expressed in OpenSWPC as the delay of the
initiation time of each point source. The FDM simula- Plane‑wave incidence
tion was performed with a 3D model of 600 × 600 × 400 Analysis of regional wave propagation from far-field body
grid points with a spatial grid size of 0.25 km and time waves (Maeda et al. 2014), core reflected phases (Toya
step of 0.01 s. et al. 2017) and the amplification and reflection of seis-
The simulation result 200 s after the rupture starting mic waves in the shallow structure can be effectively eval-
time was compared with the analytic solution of Okada uated by considering the incidence of plane waves. An
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 14 of 20

a b

(A)

c d
4700

(C)

height [m]
(B) 0

−10000
Fig. 12 Example of a visualization of a seismic wave propagation simulation by OpenSWPC at the elapsed times of a 100 s, b 200 s, c 300 s and d
400 s. Superimposed on the topography map (color scale is shown in the right), scaled absolute amplitude of vertical- and horizontal-component
ground velocity motions on the ground surface are shown in red and green, respectively. An additional file contains a time-sequential movie of the
seismic wave propagation (Additional file 1)

example of an FDM simulation for vertical incidences of

plane P- and S-waves is shown in Fig. 15. This simulation
was performed using the 2D P-SV code with a spatial grid
size of 0.2 km and a time step of 0.01 s. The layered veloc-
−80 −60 −40 −20 0

ity structure model of Kubo et al. (2002) was assumed,

and up-going P- and SV-wave packets with characteris-
z [km]

tic wavelengths of 20 km were set near the bottom of the

simulation model as initial conditions.
150
150
It was found that the use of the PML absorbing bound-
100
100 ary condition is essential for FDM simulations with inci-
]
m

50 ]
dent plane waves as well as with coseismic deformation.
[k

50 x [km
y

0 0
With the use of the conventional sponge absorbing bound-
Fig. 13 Geometry of the finite-fault simulation used to infer the
coseismic deformation. The blue rectangle shows the fault plane, and
ary, artificial reflections from the edge of the model at both
the red rectangle shows the free surface sides of the plane wave make it difficult to recognize later
arrivals of reflected and converted phases (Fig. 15a). Such
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 15 of 20

a Analytic Solution
150
ux uy uz

100

50

0
0 50 100 150 0 50 100 150 0 50 100 150

b PML
150
ux uy uz

100

50

0
0 50 100 150 0 50 100 150 0 50 100 150

c Sponge
150
ux uy uz

100

50

0
0 50 100 150 0 50 100 150 0 50 100 150

−1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 −1.5 −1.0 −0.5 0.0 0.5 1.0 1.5 −4 −2 0 2 4
ux [m] uy [m] uz [m]
Fig. 14 Comparison of the coseismic deformation of a an analytic solution of Okada (1985), b the result of a simulation by OpenSWPC with
the PML boundary condition and c the result of a simulation by OpenSWPC with the sponge boundary condition of Cerjan et al. (1985). In each
panel, three-component displacements (ux , uy, and uz) are plotted with the color scales shown below. The gray shaded areas in b and c indicate the
absorber regions. Time-sequential movies corresponding to b and c are provided in Additional file 2, Additional file 3, respectively

artificial reflections are inevitable for plane-wave inci- propagating out of the computational domain, and a part
dence because the attenuation of up-going plane waves of this packet reflects back inside the model. The situa-
in the absorbing boundary leads to artificial diffraction tion is much worse for S-wave incidence (Fig. 15b) due
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 16 of 20

a P−wave Sponge b S−wave Sponge

0

10
time [s]

20

30

40

0 25 50 75 100 0 25 50 75 100
x [km] x [km]
c P−wave PML d S−wave PML
0

10
time [s]

20

30

40

0 25 50 75 100 0 25 50 75 100
x [km] x [km]
Fig. 15 The seismic traces for vertical plane-wave incidences obtained by the 2D P-SV simulations. a Vertical-component traces for P-wave inci-
dence with the sponge boundary condition. b Horizontal-component traces for S-wave incidence with the sponge boundary condition. c, d same
as a, b but with the PML boundary condition. Waveform traces within the absorbing layer are shown as green curves, and traces at the center of the
model are shown as thick red curves

to strong S-to-P conversion at the boundary. Because the Simulations using the reciprocity theorem
PML boundary condition does not modify seismic waves An example of a reciprocity calculation comparing an
propagating parallel to the boundary, it minimizes the ordinary forward simulation from the source to receiver
occurrence of such artificial reflections (Fig. 15c, d). and a simulation using the reciprocity theorem with
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 17 of 20

a x−comp y−comp z−comp

Mxx 10−12 m

Myy 10−12 m

Mzz 10−12 m

Myz 10−12 m b
41˚N

40˚N
Mxz 10−12 m

39˚N N.KSNF

Mxy 10−12 m 38˚N

100 km
37˚N
138˚E 140˚E 142˚E 144˚E
0 20 40 60 80 100 0 20 40 60 80 100 0 20 40 60 80 100
time [s] time [s] time [s]
Fig. 16 An example of a computation using the reciprocity theorem. a Three-component Green’s functions of the six moment tensor components
(listed on the left). Black and blue curves represent the traces calculated using the forward simulation and the reciprocity theorem, respectively. Red
curves show the residual between the forward and reciprocity calculations. b An index map showing the locations of the epicenter (star) and the
observed station (triangle)

exchanged source and receiver is shown in Fig. 16. In Conclusions and future perspectives
this simulation, we adopted the JIVSM model (Koketsu This newly developed FDM simulation code for mod-
et al. 2012) in northeastern Japan including the topog- eling of seismic wave propagation in heterogeneous vis-
raphy variation and seawater column and calculated the coelastic media in 2D and 3D, OpenSWPC, has wide
seismic waveforms at a F-net station location (N.KSNF) applicability for seismological studies and great port-
from a source on the Pacific Plate boundary. Both sim- ability, allowing excellent performance from PC clusters
ulations were performed with the same model size of to supercomputers. OpenSWPC was designed to apply
3000 × 2400 × 600 grid points with a discretization of various models from small to regional scales with the
0.25 km in space and 0.01 s in time. The seismic source is use of various seismic source representations and veloc-
assumed to have a finite source duration time of 4 s. ity models. Note that all examples presented in this study
Figure 16 compares the two simulations without were obtained by setting necessary parameters and veloc-
applying a filter. They show excellent agreement with ity model files without modifying the simulation code
very small residual amplitudes between the two results. itself. Using OpenSWPC, we can simulate seismic waves
Note that the reciprocity theorem (Aki and Richards at regional scales of up to approximately 1 Hz using PC
2002) assures the exchange of the source and receiver clusters and up to approximately 10 Hz using high-per-
positions only if the absorbing and free surface bound- formance supercomputers. All the codes are available
ary conditions are satisfied (Eisner and Clayton 2001). with detailed documents at a public repository with an
Therefore, the good agreement between the two simula- open-source (MIT) license.
tions demonstrates the effectiveness of the free surface In OpenSWPC, we implemented a frequency-inde-
and the absorbing boundary conditions in the FDM pendent attenuation model by adopting the GZB. The
simulation. assumption of frequency-independent attenuation is
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 18 of 20

valid in the low-frequency regime below ∼ 1 Hz, at least Institute for Earth Science and Disaster Resilience. We used computer
resources of the EIC system of the Earthquake Research Institute at the Uni-
to a first-order approximation (e.g., Aki and Richards versity of Tokyo and the Earth Simulator of the Japan Agency for Marine–Earth
2002). However, at frequencies above about 1 Hz, attenu- Science and Technology.
ation appears to follow a power-law frequency depend-
Competing interests
ence (e.g., Carcolé and Sato 2010; Phillips et al. 2014). The authors declare that they have no competing interests.
Incorporation of such different frequency-dependent
attenuation properties for high frequencies can also be Availability of data and materials
Project name: OpenSWPC
modeled by applying the memory variable approach Project home page: http://github.com/takuto-maeda/OpenSWPC
(Withers et al. 2015). Such implementation of more real- Archived version: Version 3.7
istic attenuation characteristics to OpenSWPC is a pros- https://github.com/takuto-maeda/OpenSWPC/releases
Operating system: platform independent
pect for future development. Programing language: Fortran2003 with message passing interface
The OpenSWPC adopts the Cartesian coordinate sys- Other requirements: NetCDF library (http://www.unidata.ucar.edu/netcdf )
tem to mainly apply to small-to-regional-scale seismic License: MIT license
Any restriction to use by non-academics: None.
wave modeling. Recently, Takenaka et al. (2017) proposed
a method to incorporate the effect of spherical geometry
of the Earth into FDM simulation model in the Cartesian Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in pub-
coordinate system. Incorporation of such technique may lished maps and institutional affiliations.
lead wider applications of OpenSWPC in global seismology
Received: 27 February 2017 Accepted: 14 July 2017
without significant modification of present set of the code.

Additional files

References
Additional file 1. A time-sequential movie of the seismic wave propaga-
Adams JC, Brainerd WS, Hendrickson RA, Maine RE, Martin JT, Smith BT (2008)
tion associated with Fig. 12. (6570 KB).
The Fortran 2003 Handbook, the complete syntax, features and proce-
Additional file 2. A time-sequential movie of the displacement field dures. Springer, London. doi:10.1007/978-1-84628-746-6
from a finite-fault rupture with the PML absorbing boundary condition Aki K, Richards PG (2002) Quantitative seismology, 2nd edn. University Science
associated with Fig. 14. (81.5 KB). Books, Sausalito
Additional file 3. A time-sequential movie of the displacement field Aoi S, Fujiwara H (1999) 3D finite-difference method using discontinuous
from a finite-fault rupture with the sponge absorbing boundary condition grids. Bull Seismol Soc Am 89:918–930
associated with Fig. 14. (94.7 KB). Aoi S, Hayakawa T, Fujiwara H (2004) Ground motion simulator: GMS. Butsuri
Tansa 57:651–666 (in Japanese with English abstract)
Blanch JO, Robertsson JOA, Symes WW (1995) Modeling of a constant Q:
methodology and algorithm for an efficient and optimally inexpensive
Abbreviations viscoelastic technique. Geophysics 60:176–184. doi:10.1190/1.1443744
FDM: Finite Difference Method; EIC: Earthquake Information Center; ES: Earth Bohlen T (2002) Parallel 3-D viscoelastic finite difference seismic modelling.
Simulator; F-net: Full-range seismograph network; GZB: Generalized Zener Comput Geosci 28:887–899. doi:10.1016/S0098-3004(02)00006-7
Body; HOT: Heterogeneity, Oceanic layer and Topography; JIVSM: Japan Cao D, Yin X (2014) Equivalence relations of generalized rheological models for
Integrated Velocity Structure Model; MPI: Message Passing Interface; NetCDF: viscoelastic seismic-wave modeling. Bull Seismol Soc Am 104:260–268.
Network Common Data Form; PML: Perfectly Matched Layer. doi:10.1785/0120130158
Carcolé E, Sato H (2010) Spatial distribution of scattering loss and intrinsic
Authors’ contributions absorption of short-period S waves in the lithosphere of Japan on the
TM developed the code based on the original version of the software devel- basis of the Multiple Lapse Time Window Analysis of Hi-net data. Geo-
oped by TF performed simulations with ST, and drafted the manuscript. ST phys J Int 180:268–290. doi:10.1111/j.1365-246X.2009.04394.x
contributed to the design and application of the numerical tests of the code. Cerjan C, Kosloff D, Kosloff R, Reshef M (1985) A nonreflecting boundary
TF conceived the study and participated in the conception and design of condition for discrete acoustic and elastic wave equations. Geophysics
the study. All authors participated in discussions and equally contributed to 50:705–708. doi:10.1190/1.1441945
revising the draft of the manuscript. All authors read and approved the final Chen P, Lee EJ (2015) Full-3D seismic waveform inversion. Springer, Cham.
manuscript. doi:10.1007/978-3-319-16604-9
Chew WC, Liu QH (2011) Perfectly matched layers for elastodynamics: a
Author details new absorbing boundary condition. J Comput Acoust 04:341–359.
1
Earthquake Research Institute, The University of Tokyo, 1‑1‑1 Yayoi, Bun- doi:10.1142/S0218396X96000118
kyo 113‑0032, Tokyo, Japan. 2 National Research Institute for Earth Science Coutant O, Virieux J, Zollo A (1995) Numerical source implementation in a
and Disaster Resilience, 3‑1 Tennodai, Tsukuba 305‑0006, Ibaraki, Japan. 2D finite difference scheme for wave propagation. Bull Seismol Soc Am
85:1507–1512
Acknowledgements Cui Y, Poyraz E, Olsen KB, Zhou J, Withers K, Callaghan S, Larkin J,
We appreciate Phil Cummins, the editor, and two anonymous reviewers for Guest C, Choi D, Chourasia A, Shi Z, Day SM, Maechling JP, Jordan
their constructive comments that greatly improved the manuscript. This TH (2013) Physics-based seismic hazard analysis on petascale
study was supported by a Collaborative Research Program of the Earthquake heterogeneous supercomputers. In: Proceedings of SC13 70.
Research Institute at the University of Tokyo (2015-B-01). TM was partly sup- doi:10.1145/2503210.2503300
ported by the Core-to-Core Collaborative Research Program of the Earthquake Dreger DS (2003) TDMT_INV: time domain seismic moment tensor inversion.
Research Institute at the University of Tokyo and the Disaster Prevention In: Lee WHK, Kanamori H, Jennings PC, Kisslinger C (eds) The IASPEi
Research Institute at Kyoto University (2016-K-06). We used the F-net moment international handbook of earthquake and engineering seismology.
tensor catalog and station location information of the National Research Academic Press, London
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 19 of 20

Eisner L, Clayton RW (2001) A reciprocity method for multiple-source simula- Koketsu K, Miyake H, Suzuki H (2012) Japan integrated velocity structure
tions. Bull Seismol Soc Am 91:553–560. doi:10.1785/0120000222 model version 1. In: Proceedings of 15th WCEE:1773
Emmerich H, Korn M (1987) Incorporation of attenuation into time-domain Kristek J, Moczo P, Galis M (2009) A brief summary of some PML formulations
computations of seismic wave fields. Geophysics 52:1252–1264. and discretizations for the velocity-stress equation of seismic motion.
doi:10.1190/1.1442386 Stud Geophys Geodaet 53:459–474. doi:10.1007/s11200-009-0034-6
Emoto K, Sato G, Nishimura T (2010) Synthesis of vector wave envelopes on Kubo A, Fukuyama E, Kawai H, Nonomura K (2002) NIED seismic moment
the free surface of a random medium for the vertical incidence of a plane tensor catalogue for regional earthquakes around Japan: qual-
wavelet based on the Markov approximation. J Geophys Res 115:B08306. ity test and application. Tectonophysics 356:23–48. doi:10.1016/
doi:10.1029/2009JB006955 S0040-1951(02)00375-X
Frankel A, Clayton RW (1986) Finite difference simulations of seismic scatter- Lee S-J, Ma K-F, Chen H-W (2006) Three-dimensional dense strong motion
ing: implications for the propagation of short-period seismic waves in the waveform inversion for the rupture process of the 1999 Chi-Chi, Taiwan,
crust and models of crustal heterogeneity. J Geophys Res 91:6465–6489. earthquake. J Geophys Res 111:B11308. doi:10.1029/2005JB004097
doi:10.1029/JB091iB06p06465 Levander AR (1988) Fourth-order finite-difference P-SV seismograms. Geo-
Fukuyama E, Ishida M, Dreger DS, Kawai H (1998) Automated seismic moment physics 53:1425–1436. doi:10.1190/1.1442422
tensor determination by using on-line broadband seismic waveforms. Madariaga R (1976) Dynamics of an expanding circular fault. Bull Seismol Soc
Zisin2 51:149–156 (in Japanese with English Abstract) Am 66:639–666
Furumura T, Chen L (2005) Parallel simulation of strong ground motions during Maeda T, Furumura T (2013) FDM Simulation of seismic waves, ocean acoustic
recent and historical damaging earthquakes in Tokyo, Japan. Parallel waves, and tsunamis based on tsunami-coupled equations of motion.
Comput 31:149–165. doi:10.1016/j.parco.2005.02.003 Pure Appl Geophys 170:109–127. doi:10.1007/s00024-011-0430-z
Furumura T, Kennett BLN (2005) Subduction zone guided waves and the Maeda T, Furumura T, Noguchi S, Takemura S, Sakai S, Shinohara M, Iwai K,
heterogeneity structure of the subducted plate: Intensity anoma- Lee S-J (2013) Seismic- and tsunami-wave propagation of the 2011 Off
lies in northern Japan. J Geophys Res 110:B10302. doi:10.1029/200 the Pacific Coast of Tohoku Earthquake as inferred from the tsunami-
4JB003486 coupled finite-difference simulation. Bull Seismol Soc Am 103:1456–1472.
Furumura T, Hayakawa T, Nakamura M, Koketsu K, Baba T (2008) Development doi:10.1785/0120120118
of long-period ground motions from the Nankai Trough, Japan, earth- Maeda T, Furumura T, Obara K (2014) Scattering of teleseismic P-waves by the
quakes: observations and computer simulation of the 1944 Tonankai (Mw Japan Trench: a significant effect of reverberation in the seawater col-
8.1) and the 2004 SE Off-Kii Peninsula (Mw 7.4) earthquakes. Pure Appl umn. Earth Planet Sci Lett 397:101–110. doi:10.1016/j.epsl.2014.04.037
Geophys 165:585–607. doi:10.1007/s00024-008-0318-8 Maeda T, Iwaki A, Morikawa N, Aoi S, Fujiwara H (2016) Seismic-hazard analysis
Goldstein P, Dodge D, Firpo M, Minner L (2003) SAC2000: signal processing of long-period ground motion of megathrust earthquakes in the Nankai
and analysis tools for seismologists and engineers. In: Lee WHK, Kanamori Trough based on 3d finite-difference simulation. Seismol Res Lett
H, Jennings PC, Kisslinger C (eds) The IASPEI international handbook of 87:1265–1273. doi:10.1785/0220160093
earthquake and engineering seismology. Academic Press, London Mavroeidis GP, Papageorgiou AS (2003) A mathematical representation
Graves RW (1996) Simulating seismic wave propagation in 3D elastic of near-fault ground motions. Bull Seismol Soc Am 93:1099–1131.
media using staggered-grid finite differences. Bull Seismol Soc Am. doi:10.1785/0120020100
86:1091–1106 Moczo P (2002) 3D Heterogeneous staggered-grid finite-difference modeling
Graves RW, Wald DJ (2001) Resolution analysis of finite fault source inversion of seismic motion with volume harmonic and arithmetic averaging
using one- and three-dimensional Green’s functions: 1. Strong motions. J of elastic moduli and densities. Bull Seismol Soc Am 92:3042–3066.
Geophys Res 106:8745–8766. doi:10.1029/2000JB900436 doi:10.1785/0120010167
Graves R, Jordan TH, Callaghan S, Deelman E, Field E, Juve G, Kesselman C, Moczo P, Kristek J (2005) On the rheological models used for time-domain
Maechling P, Mehta G, Milner K, Okaya D, Small P, Vahi K (2010) Cyber- methods of seismic wave propagation. Geophys Res Lett 32:L01306. doi:1
shake: a physics-based seismic hazard model for southern California. Pure 0.1029/2004GL021598
Appl Geophys 168:367–381. doi:10.1007/s00024-010-0161-6 Moczo P, Kristek J, Galis M (2014) The finite-difference modelling of earth-
Guo Y, Koketsu K, Miyake H (2016) Propagation mechanism of long-period quake motions. Cambridge University Press, Cambridge. doi:10.1017/
ground motions for offshore earthquakes along the Nankai Trough: CBO9781139236911
effects of the accretionary wedge. Bull Seismol Soc Am 106:1176–1197. Nakamura T, Takenaka H, Okamoto T, Kaneda Y (2012) FDM simulation of
doi:10.1785/0120150315 seismic-wave propagation for an aftershock of the 2009 Suruga bay
Helffrich G, Wookey J, Bastow I (2013) The Seismic Analysis Code—a primer earthquake: effects of ocean-bottom topography and seawater layer. Bull
and user’s guide. Cambridge University Press, Cambridge. doi:10.1017/ Seismol Soc Am 102:2420–2435. doi:10.1785/0120110356
CBO9781139547260 Nakamura T, Takenaka H, Okamoto T, Ohori M, Tsuboi S (2015) Long-period
Herrmann RB (1979) SH-wave generation by dislocation sources—a numerical ocean-bottom motions in the source areas of large subduction earth-
study. Bull Seismol Soc Am 69:1–15 quakes. Sci Rep 5:16648. doi:10.1038/srep16648
Hsieh MC, Zhao L, Ji C, Ma K-F (2016) Efficient inversions for earthquake Noguchi S, Maeda T, Furumura T (2016) Ocean-influenced Rayleigh waves
slip distributions in 3D structures. Seismol Res Lett 87:1342–1354. from outer-rise earthquakes and their effects on durations of long-period
doi:10.1785/0220160050 ground motion. Geophys J Int 205:1099–1107. doi:10.1093/gji/ggw074
Igel H (2016) Computational seismology: a practical introduction. Oxford Ohminato T, Chouet BA (1997) A free-surface boundary condition for includ-
University Press, Oxford ing 3D topography in the finite-difference method. Bull Seismol Soc Am
Imperatori W, Gallovič F (2017) Validation of 3D velocity models using earth- 87:494–515
quakes with shallow slip: case study of the 2014 Mw 6.0 South Napa, Ohminato T, Chouet BA, Dawson P, Kedar S (1998) Waveform inversion of
California, event. Bull Seismol Soc Am 107. doi:10.1785/0120160041 very long period impulsive signals associated with magmatic injec-
Inoue S, Tsutsumi S, Maeda T, Minami K (2013) Performance optimization of tion beneath Kilauea volcano. Hawaii. J Geophys Res 103:23839–23862.
seismic wave simulation code on the K computer. IPSJ Trans Adv Comput doi:10.1029/98JB01122
Syst 6:22–30 (in Japanese with English abstract) Okada Y (1985) Surface deformation due to shear and tensile faults in a half-
JafarGandomi A, Takenaka H (2007) Efficient FDTD algorithm for plane-wave space. Bull Seismol Soc Am 75:1135–1154
simulation for vertically heterogeneous attenuative media. Geophysics Okada Y, Kasahara K, Hori S, Obara K, Sekiguchi S, Fujiwara H, Yamamoto A
72:H43–H53. doi:10.1190/1.2732555 (2004) Recent progress of seismic observation networks in Japan—
Kawase K (2011) A general formula for calculating meridian arc length and its Hi-net, F-net, K-NET and KiK-net—. Earth Planets Space 56:xv–xxviii.
application to coordinate conversion in the Gauss–Krüger projection. Bull doi:10.1186/BF03353076
Geosp Inf Auth Jpn 59:1–13 Okamoto T, Takenaka H (2005) Fluid–solid boundary implementation in the
Klimeš L (2002) Correlation functions of random media. Pure Appl Geophys velocity-stress finite-difference method. Zisin2 57:355–364 (in Japanese
159:1811–1831. doi:10.1007/s00024-002-8710-2 with English abstract)
Maeda et al. Earth, Planets and Space (2017) 69:102 Page 20 of 20

Olsen KB, Pechmann JC, Schuster GT (1995) Simulation of 3D elastic wave Takenaka H, Fujii Y (2008) A compact representation of spatio-temporal slip
propagation in the Salt Lake Basin. Bull Seismol Soc Am 85:1688–1710 distribution on a rupturing fault. J Seismol 12:281–293. doi:10.1007/
Petersson NA, Sjogreen B (2014) SW4 user’s guide. Lawrence Livermore s10950-007-9087-6
National Laboratory technical report LLNL-SM-662014. https://geody- Takenaka H, Komatsu M, Toyokuni G, Nakamura T, Okamoto T (2017) Quasi-
namics.org/cig/software/github/sw4/v1.1/SW4-v1.1-UsersGuide.pdf. Last Cartesian finite-difference computation of seismic wave propagation
accessed June 2017 for a three-dimensional sub-global model. Earth Planets Space 69:67.
Petukhin A, Miyakoshi K, Tsurugi M, Kawase H, Kamae K (2016) Visualiza- doi:10.1186/s40623-017-0651-1
tion of Green’s function anomalies for megathrust source in Nankai Tape C, Liu Q, Maggi A, Tromp J (2009) Adjoint tomography of the southern
Trough by reciprocity method. Earth Planets Space 68:4. doi:10.1186/ California crust. Science 325:988–992. doi:10.1126/science.1175298
s40623-016-0385-5 Todoriki M, Furumura T, Maeda T (2017) Effects of sea water on elongated
Phillips WS, Mayeda KM, Malagnini L (2014) How to invert multi-band, duration of ground motion as well as variation in its amplitude for off-
regional phase amplitudes for 2-D attenuation and source parameters: shore earthquakes. Geophys J Int 208:226–233. doi:10.1093/gji/ggw388
tests using the USArray. Pure Appl Geophys 171:469–484. doi:10.1007/ Toya M, Kato A, Maeda T, Obara K, Takeda T, Yamaoka K (2017) Down-dip
s00024-013-0646-1 variations in a subducting low-velocity zone linked to episodic tremor
Pitarka A (1999) 3D Elastic finite-difference modeling of seismic motion using and slip: a new constraint from ScSp waves. Sci Rep 7:2868. doi:10.1038/
staggered grids with nonuniform spacing. Bull Seismol Soc Am 89:54–68 s41598-017-03048-6
Rew R, Davis G (1990) NetCDF: an interface for scientific data access. IEEE Wald DJ, Graves RW (2001) Resolution analysis of finite fault source inver-
Comput Graph Appl 10:76–82. doi:10.1109/38.56302 sion using one- and three-dimensional Green’s functions: 2. Combining
Robertsson JOA, Blanch JO, Symes WW (1994) Viscoelastic finite-difference seismic and geodetic data. J Geophys Res 106:8767–8788. doi:10.1029/2
modeling. Geophysics 59:1444–1456. doi:10.1190/1.1443701 000JB900435
Saikia CK (1994) Modified frequency-wavenumber algorithm for regional Wessel P, Smith WHF, Scharroo R, Luis J, Wobbe F (2013) Generic Mapping
seismograms using Filon’s quadrature: modelling of Lg waves in eastern Tools: improved version released. EOS Trans AGU 94:409–410. doi:10.100
North America. Geophys J Int 118:142–158. doi:10.1111/j.1365-246X.1994. 2/2013EO450001
tb04680.x Withers KB, Olsen KB, Day SM (2015) Memory-efficient simulation of
Sato H, Fehler MC, Maeda T (2012) Seismic wave propagation and scat- frequency-dependent Q. Bull Seismol Soc Am 105:3129–3142.
tering in the heterogeneous Earth, 2nd edn. Springer, Berlin. doi:10.1785/0120150020
doi:10.1007/978-3-642-23029-5 Yoshimitsu N, Furumura T, Maeda T (2016) Geometric effect on a laboratory-
Takemura S, Furumura T, Maeda T (2015) Scattering of high-frequency seismic scale wavefield inferred from a three-dimensional numerical simulation. J
waves caused by irregular surface topography and small-scale velocity Appl Geophys 132:184–192. doi:10.1016/j.jappgeo.2016.07.002
inhomogeneity. Geophys J Int 201:459–474. doi:10.1093/gji/ggv038 Zhang W, Shen Y (2010) Unsplit complex frequency-shifted PML implementa-
Takemura S, Maeda T, Furumura T, Obara K (2016) Constraining the source tion using auxiliary differential equations for seismic wave modeling.
location of the 30 May 2015 (Mw 7.9) Bonin deep-focus earthquake using Geophysics 75:T141–T154. doi:10.1190/1.3463431
seismogram envelopes of high-frequency P waveforms: occurrence of Zhao L, Chen P, Jordan TH (2006) Strain Green’s tensors, reciprocity, and their
deep-focus earthquake at the bottom of a subducting slab. Geophys Res applications to seismic source and structure studies. Bull Seismol Soc Am
Lett 43:4297–4302. doi:10.1002/2016GL068437 96:1753–1763. doi:10.1785/0120050253