Uni Ed Astronomy Thesaurus Concepts

The Astrophysical Journal Supplement Series, 263:32 (25pp), 2022 December https://doi.org/10.
3847/1538-4365/ac9279
© 2022. The Author(s). Published by the American Astronomical Society.
SADHANA: A Doubly Linked List-based Multidimensional Adaptive Mesh Refinement

Framework for Solving Hyperbolic Conservation Laws with Application to Astrophysical
Hydrodynamics and Magnetohydrodynamics
Somdeb Bandopadhyay1,2 and Hsien Shang (尚賢)2
1
Institute of Applied Mechanics, National Taiwan University, Taipei 106, Taiwan; [email protected], [email protected]
2
Institute of Astronomy and Astrophysics, Academia Sinica, Taipei 10617, Taiwan; [email protected]
Received 2022 May 21; revised 2022 September 10; accepted 2022 September 14; published 2022 December 5
Abstract
We report the development of a nested block-structured adaptive mesh framework to solve multidimensional, time-
dependent hyperbolic equations encountered in astrophysics. An approach based on a tabular list is used to
construct variants of Hilbert space-filling curves in an iterative fashion to maintain the connectivity of locally
refined mesh configurations using a doubly linked list. Modifications are made to conventional boundaries of
computational blocks to aid the adaptive mesh. We also describe a well-defined, computationally efficient data
structure to hold self-similar mesh units for this purpose. The flexibility of this code is demonstrated by the
performance of various Riemann solvers implemented in this computational framework.
Unified Astronomy Thesaurus concepts: Magnetohydrodynamics (1964); Hydrodynamics (1963); Computational
methods (1965); Algorithms (1883); Astrophysical fluid dynamics (101)
1. Introduction AMR algorithm for Cartesian meshes. The technique can be

extended to multiple space dimensions, and the data structure
The search for a balance between ever-increasing demand
required for this approach is explained in detail in Berger
for higher resolutions of numerical solutions and available
(1986). The computation begins with a coarse Cartesian grid
computational power has resulted in the rise of the study of
that is considered to be the root-level mesh. Afterward,
adaptive mesh refinement (AMR) algorithms on multiple CPU-
computational cells are marked for refinement as the calcul-
based computational units. A successful implementation of the
ation progresses. These newly refined individual cells are
AMR algorithm can distribute the computational cost effi-
organized and maintained in different patches, and these cells
ciently at discrete locations in the domain where higher
are further refined, depending on requirements, creating
resolutions are required. Since these discrete locations are
additional nested patches. In a cell-based AMR, each cell
usually governed by time-dependent local solutions, the
may be refined individually and is stored using a tree-type data
required adaptive character of the mesh has to be dynamic
structure (quadtree for 2D and octree for 3D). This version of
and capable of changing with the evolution of the global
an AMR was again pioneered by Berger & Leveque (1989) and
solution itself. To achieve this, light-weight signatures of the
has been adopted, modified, and improved by various studies,
complete or partial mesh structure are usually stored in a
dynamic fashion to control the global mesh. These light-weight including Powell et al. (1993), DeZeeuw & Powell (1993),
signatures are often termed proxy-data or meta-data, and the Coirier & Powell (1995), Aftosmis et al. (1998), and Aftosmis
mesh adaption is performed using a replication and/or et al. (2000). The flexibility of a tree-type data structure readily
distribution of these proxy-data among different processes. In accounts for the local refinement of the mesh by keeping track
some cases, particularly when the requirement of higher of the computational cell connectivity as new grid points are
resolution is scattered sparsely all over the domain, the task generated by the refinement process (four and eight children
of mesh distribution among various process may lead to a cells in 2D and 3D, respectively). Most of these cell-based
considerable increase in computational cost, and therefore, approaches have so far been restricted to a numerical solution
these mesh-adaption techniques need to be designed with care of hyperbolic systems on Cartesian meshes, mainly because
and caution. In practice, some form of indicator function is elliptic operators are more challenging to implement in this
used to identify the regions in which mesh adaption is needed, type of mesh structure. In a block-based AMR strategy,
and space-filling curves (SFCs), constructed from the proxy- commonly known as block-structured adaptive mesh refine-
data, are used to decompose the computational load among ment (BSAMR), mesh adaptation is accomplished by the
multiple processes. Although numerous variations exist, AMR dividing and coarsening of entire predefined mesh blocks or
for structured mesh or structured adaptive mesh refinement groupings of cells. Although not required, each of the groups or
(SAMR) methods can be divided into three main categories, blocks generally has an equal number of cells. As in the case of
namely patch-based, cell-based, and block-based AMRs. The a cell-based AMR, tree-type data structures are commonly used
patch-based AMR was introduced by Berger & Oliger (1984) for tracking the block connectivity and mesh refinement.
and Berger & Colella (1989) in their pioneering work on the However, the tree structure needed in this case is lighter (i.e., it
needs less information and is hence computationally cheaper)
compared to cell-based AMR methods. As such, even
Original content from this work may be used under the terms
of the Creative Commons Attribution 4.0 licence. Any further considering the typical larger numbers of mesh cells created
distribution of this work must maintain attribution to the author(s) and the title during the refinement process (i.e., typically more than the
of the work, journal citation and DOI. corresponding number of cells introduced in cell-based AMR
1
The Astrophysical Journal Supplement Series, 263:32 (25pp), 2022 December Bandopadhyay & Shang
approaches), block-based AMR methods are arguably easier to fashion to provide a computationally efficient dynamic load-
develop and maintain in terms of computational efficiency and balancing performance according to the mesh adaption for
scalability in parallel implementations via domain decomposi- numerical solution. The current paper is organized in the
tion, as discussed in Groth et al. (1999). following fashion: we first briefly review the governing
There exist various software frameworks, each of which equations to be used for the targeted physics suitable for our
provides an AMR facility for a structured mesh, with multiple solver in Section 2; afterward, we give a detailed description of
options for the detector function and load balancing. To name a the SADHANA AMR solver in Section 3, followed by
few, BoxLib3 (Zhang et al. 2016), Chombo4 (Adams et al. Section 4, where we consider various conventional test-cases
2016), PARAMESH (MacNeice et al. 2000), and p4est library to show the practicality and versatility of the SADHANA AMR
(Burstedde et al. 2011) are widely used general-purpose SAMR framework and test its computational performance. Finally, in
software packages. The key differences among these codes are Section 5, we summarize the work done and provide
based on the choice of SFC, on the design of inherent data concluding remarks.
structures and algorithms to control proxy-data-related tasks
among different processes during AMR operations. BoxLib, 2. Governing Equations
now succeeded by AMReX5, is one of the oldest computational
SADHANA aims to solve a generalized system of hyper-
libraries and is devoted to building high-performance AMR
bolic equations with source terms in multidimensional form.
applications. Chombo started as a fork of BoxLib in 1998 and
Strictly speaking, in 3D Cartesian coordinates, the PDE system
eventually became a separate multipurpose package itself.
in consideration is
Various codes used in the astrophysics community have started
using these software libraries to add an AMR facility in the ¶U ND
¶Fx i
simulation. For example, the FLASH code (Dubey et al. 2009) = -å + S, (1 )
¶t i = 1 ¶xi
has mesh-refinement support provided by the octree-based
PARAMESH package, while Chombo is used in the finite- where x1: = x, x2: = y, and x3: = z are the directions, ND is
volume code Pluto (Mignone et al. 2012). Another numerical dimension of the problem, U is a vector holding conservative
framework that has gained considerable interest in the states of dependent variables, Fxi is the directional flux vector,
astrophysical community is Cactus (Goodale et al. 2003), and finally, S is a vector holding all the source terms to be
which uses Carpet6 (Schnetter et al. 2004) for its AMR
considered. The value of U is computed from the corresp-
operations.
Although the use of external frameworks makes it possible onding Q vector holding the primitive states of the independent
to implement an AMR facility in a particular code without any variables following a physics-based equation of state (EOS).
need of extreme care for the AMR strategy itself, the solver that Detailed structures and values of Q, U, Fxi , and S are directed
is considered ends up having a strong dependence on the data by the physics of the problem in consideration. We now discuss
structure that is used in these packages. As a result, a how different types of governing equations can be implemented
significant amount of time and effort is needed to go through in the system expressed by Equation (1) to account for typical
and understand these aforementioned data structures before a hydrodynamics and magnetohydrodynamics (MHD) problems.
specific AMR package can be used efficiently. Also, the code
needs to be updated regularly to keep it compatible with newer 2.1. Hydrodynamics
versions of the AMR framework that is used. To avoid this
hindrance, some codes, including Enzo (O’Shea et al. 2005; In an inviscid hydrodynamics system, the primitive and
Bryan et al. 2014), RAMSES (Teyssier 2002), and NIRVANA conservative state vectors, directional flux vectors, and the
(Ziegler 1999, 2008), have integrated their own AMR source vector are defined as
algorithms in these solvers. The development of the Enzo-P/ r r
Cello (Bordner & Norman 2012) code is a good example to ⎛v ⎞ ⎛ rv ⎞
x x
⎜ ⎟ ⎜ ⎟
show complications associated with a stand-alone AMR code Q = vy , U = rvy ,
development even with an extremely capable group of ⎜ vz ⎟ ⎜ rv ⎟
developers because it is a modification and redesign of Enzo, ⎜p ⎟ ⎜ z⎟
where special care is given to overcome scaling bottlenecks. A ⎝ ⎠t ⎝E⎠
full review of all AMR codes is beyond the scope of this paper. rvx i
⎛ ⎞
We refert to Dubey et al. (2014) for a study of various SAMR ⎜ rvx i vx + pt d x i x ⎟
packages, their design, performance, and limitations. Fx i = ⎜ rvx i vy + pt d x i y ⎟,
The key contribution of the current work is in the direction ⎜ rvx i vz + pt d x i z ⎟
of AMR-based code development without any major depend- ⎜ (E + p ) v ⎟
ence on external libraries. Also, our work differs from most of ⎝ t xi ⎠
other related studies in the way we define the neighbors, and
how, instead of using an explicit and well-optimized tree-type ⎛0⎞
⎜0⎟
data structure, we use doubly linked lists to organize and S= 0 . (2 )
maintain our AMR mesh. The doubly linked list data structure ⎜ ⎟
0
⎜ ⎟
allows us to implement Hilbert-type SFCs in an implicit
⎝0⎠
3
https://github.com/BoxLib-Codes It should be noted that while we have ignored gravity and any
4
https://commons.lbl.gov/display/chombo other possible forcing function, if needed, we can easily modify
5
https://amrex-codes.github.io/amrex the source vector for a specific problem (e.g., where gravity is
6
https://carpetcode.org needed, or when we need to consider viscous effect). Following
2
conventional symbols in fluid mechanics, the density, velocity, 3. SADHANA Adaptive Mesh Refinement Framework
and thermal pressure are represented by ρ, v = [vx , vy , vz ]T , and
SADHANA uses a simple but versatile data structure to
pt, respectively. The total energy E can be calculated as
control the nested-mesh system, which is dynamically adapted
E = ρein + Ekin, where the specific internal energy ein is obtained
p and modified as per requirement during computation. The main
from the adiabatic EOS for ideal gas, i.e., ein = r (g -t 1) , with γ
structure of the code is maintained in a modular and optimized
being the adiabatic index, and the kinetic energy Ekin is fashion such that adjustments can easily be implemented to
1 1
calculated as Ekin = 2 r ∣ v ∣2 = 2 r (vx2 + vy2 + vz2 ). account for specific physics problems, as described in
Section 2. We now proceed to describe the AMR implementa-
2.2. Magnetohydrodynamics tion in SADHANA. The general configuration of the nested
block-structured mesh is explained in Section 3.1, followed by
Because SADHANA at the time of writing is developed to a short discussion of new definitions of the block boundaries
deal with cell-centered quantities alone, the classical numerical used for our work in Section 3.2. The related data structure,
approach for an MHD system is difficult to apply in the solver. SFC generation, and load-balancing procedures are discussed
This is due to the conventional approach of storing the in Sections 3.3, 3.4, and 3.5, respectively. The procedure of
magnetic field B = [Bx , By , Bz ]T in a staggered arrangement so checking the refinement criteria for AMR and associated
that it is numerically easier to keep a divergence-free condition, numerical adjustments are given in Sections 3.6 and 3.7.
i.e., ∇ • B = 0 is satisfied throughout the calculation. Fortu- Finally, Section 3.8 gives a detailed description of the physics-
nately, much effort is made to prepare reliable algorithms for a independent basic numerical approach, including flux calcul-
numerical treatment of MHD in nonstaggered grids, including ation and time integration to solve the system of equations.
Chacón (2004), Toth (2000), Rossmanith (2006), Helzel et al.
(2011), and Dedner et al. (2002). We have chosen to follow the
3.1. Nest of Blocks
extended generalized Lagrange multiplier (EGLM) formulation
of MHD equations as pioneered by Dedner et al. (2002), due to At any particular time, the domain is covered by a nest of
its simplicity, where a scalar field ψ is evolved to maintain the multiple blocks that can have a user-defined number of
divergence-free condition. As such, we have primitive and computational cells (by default, we set it to 10) at each direction.
conservative state vectors, directional flux vectors, and the As such, each block ends up with the same {nx, ny, nz} numbers
source vector of the EGLM-MHD system, defined as of mesh in the {x, y, z} directions represented by {i,j,k} integers
for loop operations. If needed, these blocks can be divided into
r
⎛v ⎞
r
⎛ rv ⎞ fixed numbers (2, 4, and 8 for 1D, 2D, and 3D AMR,
x
⎜v ⎟
x
⎜ rv ⎟ respectively) of child blocks. The division is done in such a
y y
⎜v ⎟
z
⎜ rv ⎟
z
way that we have exactly the same physical dimension for any
⎜p⎟ ⎜ ⎟
Q = ⎜ t ⎟, U = ⎜ E ⎟,
two child blocks that have the same mother block (i.e., sibling
B
⎜ x⎟
B
⎜ x⎟
blocks). This operation is repeated in an iterative fashion based on
By B some criteria, as explained in Section 3.6, to provide the required
⎜ ⎟ ⎜ y⎟
⎜ Bz ⎟ ⎜ Bz ⎟ resolution at various locations of interest. Owing to the fact that
⎜ ⎟ ⎜ ⎟
⎝y⎠ ⎝y⎠ the individual cell size of any block with a certain level of
rv xi ⎛ 0 ⎞ refinement (LOR) is constant throughout the simulation and
⎛
1
⎞ ⎜
å
3
¶B xi ⎟
B because the number of mesh at each block is kept at fixed values
⎜ rv x vx + p d x x + ∣ B ∣2 d x x - B x Bx ⎟ ⎜ i = 1 ¶x i
x
⎟
⎜
i t i
2 i i
⎟
regardless of its LOR, we can define the refinement ratios between
⎜ rv v + p d + 1 ∣ B ∣2 d - B B ⎟
⎜ 3 ¶B ⎟ mesh intervals of any two LOR values of (l − 1) and l as
⎜å
xi
xi y t xi x xi x xi x By ⎟ [Dx ]
⎜ 2 ⎟ ⎜ i=1
¶ x i
⎟ [rx ]l = [ry ]l = [rz ]l = 2, where [rxi ]l = [Dix l]- 1 , {i = 1, 2, 3}.
⎜ rv v + p d + 1 ∣ B ∣2 d - B B ⎟ ⎜
3
¶B xi ⎟ i l
⎜
xi z t xi x
2
xi x xi x
⎟ å ¶x z ⎟ B Note that the LOR of each child block is one level higher than that
Fxi = ⎜ ⎜
⎟, S = -⎜ i = 1 . (3 ) of its mother block. Finally, the nest of blocks with different
i
3
1 ¶y ⎟
⎜⎛E + pt + ∣ B ∣ ⎞ v xi - B xi ⎜ å B xn v xn ⎟⎟
2 ⎛ ⎞
2
3
⎜å i = 1 Bx ⎟ refinement levels is organized in such a fashion that the maximum
⎜⎝ ⎠ ⎝n = 1 ⎠⎟ ¶x i i
v B - B v + yd
⎜ ⎟ value for the difference in LOR between any two neighboring
0
⎜ xi x xi x xi x
v xi By - B xi vy + yd xi y
⎟ ⎜
0
⎟ blocks is 1, and thus, we always have at least one LOR = (l − 1)
⎜ ⎟
⎜ v xi Bz - B xi vz + yd xi z ⎟
⎜
0
⎟ block between a LOR = l and a LOR = (l − 2) block.
⎜ 2 ⎟
⎜ ⎟ c
ch
2
B x
⎜
⎜
h
y ⎟
⎟
⎝ i ⎠ cp2
3.2. Modified Definition of Block Boundaries
⎝ ⎠
In a conventional numerical approach, for any computational
Just as before, we can modify the source terms to consider any volume, only two boundaries are considered in each direction, i.e.,
nonideal term (e.g., magnetic resistance) and other forcing LEFT(ΩLft) and RIGHT(ΩRht) for x, TOP (ΩTop) and
terms (e.g., gravity or viscous effects). The magnetic field is BOTTOM(ΩBot) for y, and finally, FRONT(ΩFrn) and BACK
scaled by m0 , with μ0 being the magnetic permeability of a (ΩBck) for z. Only the neighboring blocks with the same LOR
medium. The total energy of the MHD system is value are connected because the block interface covers the block
E = ρein + Ekin + Emag, with the magnetic energy Emag = boundaries completely in these cases. Because we only perform
1 1 the computation on leaf blocks (i.e., blocks without a child),
2
∣ B ∣2 = 2 (Bx2 + By2 + Bz2 ). The values of cp and ch are whenever there is any need of communication between two
adjusted by the user to account for the characteristic neighboring leaf blocks, we first check the neighbor at the same
propagation speed and the damping rate. For an in-depth LOR, and if it is not a leaf block, we then iterate through its
description of the EGLM-MHD system with a detailed mother/child to access the actual leaf block. While the use of the
explanation of every term, we refer to Dedner et al. (2002). optimized tree-type data structure makes this leaf-search efficient,
3
3D case
in this paper, we follow a different approach altogether. We now 6. WBck ⟹ WBck Bck Bck
BckTopLft, WBckTopRht, WBckBotLft, and
discuss some key arguments and explain why some modifications 4 corners
in the definition of block neighbors can be a welcome step to WBck
BckBotRht.
simplify the neighbor leaf block search. We consider two The subscripts in these new boundary definitions represent the
neighboring blocks A and B that share the boundaries WbA and corner of the original block that is shared by both the old and
WbB , respectively. Because the common interface Winterface = new boundaries. While no changes are made for a 1D AMR,
WbA Ç WbB will always completely cover both WbA and WbB in case we would like to represent ΩLft and ΩRht by WLft Rht
Lft and WRht,
of a 1D AMR, we now make some observations for exclusively respectively, for closure. Figure 1 gives a schematic presenta-
2D and 3D AMR implementations. The following three situations tion of a 2D AMR block with these modified definitions. In
can occur when we wish to check the relation among Ωinterface, Figure 1 we also show the conventional definition of block
WbA, and WbB . boundaries for comparison. It should be noted that depending
on the physics and/or numerical methods, we may need to
1. Case 1: Both A and B are at the same LOR, i.e., LORA = consider the complete neighborhood of a block for 2D and 3D
LORB. In this case, we have Winterface = WbA = WbB . problems. As a result, we need to consider four boundaries in
2. Case 2: A is at a higher LOR, i.e., LORA = LORB + 1. In the corners for the 2D case, and eight boundaries in corners and
this case, we have Winterface = WbA and Winterface = 24 boundaries along the edges (each edge would lead to two
bWbB , where β = 0.5 for a 2D AMR and β = 0.25 for a boundaries, as discussed before) for the 3D case.
3D AMR. As explained in the next sections, child blocks are
3. Case 3: A is at the lower LOR, i.e., LORA = LORB − generated from each mother block that undergoes the
1. In this case, we have Winterface = WbB and refinement procedure. The neighbors along boundaries for
Winterface = b WbA, where β = 0.5 for a 2D AMR and these newly created blocks are needed to be updated as soon
β = 0.25 for a 3D AMR. as they are created. Following the child id for each new block,
Note that we always need to iterate through mother/child for these boundaries can be identified in an iterative fashion.
Winterface ¹ WbA or Winterface ¹ WbB , which is always true When the neighborhood of the mother block is already
for Case 2 and Case 3 following the conventional definitions defined, this process can be directly derived from the location
for boundaries. As such, considering the 2:1 refinement, we and position of each child block in the inner domain of the
modify these definitions such that Winterface = WbA = WbB mother block. For simplicity, we restrict ourselves to a 2D
remains true regardless of the LOR difference between A scenario and proceed to provide a detailed list of boundaries
and B. for all children.
For a 2D AMR, we divide each of the ΩLft, ΩRht, ΩTop, and Lft
1. For Child 1, we have the edge boundaries defined as 1WTopLft
Bot
Ω into two equal parts. We therefore have the following set
= 0WLft Lft Lft
BotLft, 1WBotLft = 0 WBotLft, 1WTopRht =
Rht
of eight new boundaries: Rht Top
2D case
Child 2, 1WBotRht = Child 2, 1WTopLft = Child 3,
Lft
1. WLft ⟹ WTopLft and WLft
BotLft
Top Bot Bot Bot
1WTopRht = Child 3, 1WBotLft = 0 WBotLft, and 1WBotRht =
2 corners Bot
2D case 0 WBotLft. Similarly, we have the corner boundaries defined as
2. WRht Rht
⟹ WTopRht and WRht TopLft Lft TopRht BotLft
1WTopLft = 0 WTopLft, 1WTopRht = Child 4, 1WBotLft =
BotRht
2 corners
BotLft BotRht Bot
0 WBotLft, and 1WBotRht = 0 WBotRht.
2D case
Top
3. WTop ⟹ WTop
TopLft and WTopRht
2 corners 2. For Child 2, we have the edge boundaries defined as
2D case Lft Lft Rht
4. WBot ⟹ WBot Bot 2 WTopLft = Child 1, 2 WBotLft = Child 1, 2 WTopRht =
BotLft and WBotRht.
Rht Rht Rht Top
0 WBotRht, 2 WBotRht = 0 WBotRht, 2 WTopLft = Child 4,
2 corners
For a 3D AMR, we obtain a total set of 24 new boundaries Top Bot Bot Bot
2 WTopRht = Child 4, 2 WBotLft = 0 WBotRht, and 2 WBotRht
by dividing each of the ΩLft, ΩRht, ΩTop, ΩBot, ΩFrn, and
= 0WBotBotRht. The corner boundaries are defined as
ΩBck into four equal parts. In a similar way as before, the TopLft TopRht Rht BotLft
division is done in such a way that each new boundary covers a 2 WTopLft = Child 3, 2 WTopRht = 0 WTopRht, 2 WBotLft =
Bot BotRht BotRht
single and different corner of the original boundary itself. 0 WBotLft, and 2 WBotRht = 0 WBotRht.
3D case
3. For Child 3, we have the edge boundaries defined as
Lft Lft Lft Lft Rht
3WTopLft = 0 WTopLft, 3WBotLft = 0 WTopLft, 3WTopRht =
Lft Lft
1. WLft ⟹ WLft
BckTopLft, WBckBotLft, WFrnTopLft, and
Lft
4 corners
Child 4, 3WBotRht = Child 4, 3WTopLft = 0WTop
Rht Top
TopLft,
WFrnBotLft Top Top Bot
3D case
Rht Rht
W
3 TopRht = W
0 TopLft , W
3 BotLft = Child 1, and
2. WRht ⟹ WRht
BckTopRht, WBckBotRht, WFrnTopRht, and Bot
4 corners 3WBotRht = Child 1. The corner boundaries are defined
Top
Rht
WFrnBotRht as 3WTopLft TopLft TopRht
TopLft = 0 WTopLft, 3WTopRht = 0 WTopRht, 3WBotLft
BotLft
3D case Lft BotRht
3. WTop ⟹ WTop Top Top = 0WBotLft, and 3WBotRht = Child 2.
BckTopLft, WBckTopRht, WFrnTopLft, and
4 corners 4. For Child 4, we have the edge boundaries defined as
Top
WFrnTopRht Lft Lft
4 WTopLft = Child 3, 4 WBotLft = Child 3, 4 WTopRht =
Rht
3D case Rht Rht Rht Top Top
0 WTopRht, 4 WBotRht = 0 WTopRht, 4 WTopLft = 0 WTopRht,
Bot Bot
4. WBot ⟹ WBot
BckBotLft, WBckBotRht, WFrnBotLft, and
Top Top Bot
4 WTopRht = 0 WTopRht, 4 WBotLft = Child 2, and
4 corners
Bot
WFrnBotRht Bot
3D case 4 WBotRht = Child 2. The corner boundaries are defined
Frn
5. WFrn ⟹ WFrnTopLft, WFrn Frn
FrnTopRht, WFrnBotLft, and as 4WTopLft Top TopRht TopRht
4 corners TopLft = 0 WTopLft, 4 WTopRht = 0 WTopRht,
BotLft BotRht Rht
WFrn
FrnBotRht 4 WBotLft = Child 1, and 4 WBotRht = 0 WBotRht.
4
Figure 1. Figure showing the 2D AMR setup. (a) A schematic diagram of a 2D AMR block. (b) Mother-children AMR boundaries following the modified definition.
(c) Mother-children AMR boundaries following the conventional definition. In the case of panel (b), we show the mother block (0) and its children (1, 2, 3, and 4). 0Ω
represents the mother-block boundaries, while iΩ with i = 1, 2, 3, 4 are the boundaries of its children.
Similar relationships can be found for the third case by for this structure is such that every individual block should be
following the child id and the child position in the mother block responsible for the same amount of computational load,
as well. regardless of its LOR. Figure 2 presents the Block_Unit
structure for a 2D AMR application. Because most of the time,
independent parameters such as the numbers and size of the
3.3. Block_Unit: Data Structure for Single Block Information cells in different directions and the size of the physical domain
We now focus our attention on the data structure to be used associated with the block are either constant or depend only on
for storing information of individual blocks. The requirement the LOR, we do not need to save them explicitly in this data
5
Figure 2. Data structure holding the information of a single block for a 2D AMR. Note that the corner boundaries ΩTopLft, ΩTopRht, ΩBotLft, and ΩTopLft have been
considered in neighbor blocks.
structure. Block boundaries being [xi min, xi max] for ith levels. One of the unique features in our work is the complete
directions are saved explicitly. Following the structure of independence of the requirement of having a well-optimized
doubly linked lists, each Block_Unit has two pointers. They are explicit tree data structure, unlike most other AMR codes. We
directed toward the previous and the next member in a doubly take advantage of the fact that after refinement, the quadrants/
linked list consisting of leaf blocks at any time during the octants of a computational block become the children (similar
simulation. Every time we perform a refinement/derefinement to the logic behind octree/quadtree data structures). First, we
process, this doubly linked list must be updated following the assign unique integers to each child of a mother. The locations
appropriate ordering, as explained before. Similar types of of each child block along with the associated numbers used in
pointers (directed toward another Block_Unit structure) are this study (1D, 2D, and 3D) are given in Table 1. Now, we
used to maintain information regarding the boundaries of these define different sets of specific sequences using these child
blocks. A set of array pointers are used to hold the conservative
numbers and call them “base patterns”. When the centers of
and primitive variables as well as the interface fluxes for the
children are connected according to these sequences, a graph is
computational cells inside these blocks. Integer-type variables
are used to hold the values for the refinement level, the current formed representing a variant of a first-order Hilbert curve.
processor to which the block belongs, the SFC value (explained Basically, these base patterns are indicators of the linked-list-
in detail in the next section), the refinement/derefinement flag, based connectivity among the children, i.e., how the previous
and the index as child for the current block from its mother and next Block_Unit pointers would be prepared for the child
block. blocks.
For example, the 2D type-I base pattern is defined as
(1 → 2 → 4 → 3), which means that the first-order Hilbert
3.4. Doubly Linked List and Generation of Nested Hilbert SFC curve for this pattern can be retrieved by connecting the centers
We now describe our approach to preparing a face- of Child 1 to Child 3 to Child 4 and to Child 2. Similarly, we
continuous Hilbert SFC from the active blocks of different end up with 2, 4, and 12 different variants of the basic patterns
6
Table 1 Table 2
Location of Child Blocks Compared to the Mother Block from Which They Ordering of Child Blocks for Different Types of Base Pattern
Are Created
1D AMR
1D AMR
Base Pattern Ordering of the Children
Child ID xc
Type I 1→2
1 0.25Lx
Type II 2→1
2 0.75Lx
2D AMR
2D AMR
Child ID xc yc
Type I 1 → 2→4 → 3
1 0.25Lx 0.25Ly
Type II 1 → 3→4 → 2
2 0.75Lx 0.25Ly
Type III 4 → 2→1→3
3 0.25Lx 0.75Ly
Type IV 4 → 3→1 → 2
4 0.75Lx 0.75Ly
3D AMR
3D AMR
Child ID xc yc zc
Type I 1 → 2→4→3→7→8→6 → 5
1 0.25Lx 0.25Ly 0.25Lz
Type II 1 → 3→7→5→6→8→4 → 2
2 0.75Lx 0.25Ly 0.25Lz
Type III 1 → 5→6→2→4→8→7 → 3
3 0.25Lx 0.75Ly 0.25Lz
Type IV 4 → 2→6→8→7→5→1 → 3
4 0.75Lx 0.75Ly 0.25Lz
Type V 4 → 3→1→2→6→5→7 → 8
5 0.25Lx 0.25Ly 0.75Lz
Type VI 4 → 8→7→3→1→5→6 → 2
6 0.75Lx 0.25Ly 0.75Lz
Type VII 6 → 2→1→5→7→3→4 → 8
7 0.25Lx 0.75Ly 0.75Lz
Type VIII 6 → 5→7→8→4→3→1 → 2
8 0.75Lx 0.75Ly 0.75Lz
Type IX 6 → 8→4→2→1→3→7 → 5
Note. With [x1 = x, x2 = y, x3 = z], the children are centered at [xic ], with the
Type X 7 → 3→4→8→6→2→1 → 5
mother block dimension in xi direction being L xi , and the mother block is
centered at [xi = 0.5L xi ]. i = 1 for 1D, i = 1,2 for 2D, and i = 1, 2, 3 in 3D. Type XI 7 → 5→1→3→4→2→6 → 8
in 1D, 2D, and 3D, respectively, which are described in Type XII 7 → 8→6→5→1→2→4 → 3
Table 2.
We now demonstrate how the refinement is performed with our
are given by Table 3 for 1D, 2D, and 3D cases. Returning to
algorithm. Consider the pre- and post-refinement situation shown
our example, we now select the Type-II pattern for the 2D
in Figure 3. We assume that we have done the first level (l = 1) of
AMR as given by Table 3 for the current scenario. Finally, we
refinement and a Type-II pattern has been assigned to current 2x2
merge the patterns together by following the new sequence,
blocks. As such, the Hilbert curve is presented by the sequence
1l1  l31  l41  2l1 . 1l1  l31  l412  l432
 l442  l422  2l1 ,
The dimension corresponding to our global domain of
interest is [Lx, Ly]. When the child block with ID = 4 is selected where the subscript and superscript represent the child number
for refinement by the predefined criterion (see Section 3.6 for and refinement level. Note that this process would change
details), we need to prepare a modified Hilbert curve for this according to the pattern type for first level (l = 1) of refinement.
configuration. To do this, we first assign a new pattern to this
This process can be recursively repeated to create a set of
⎣
L L 3L 3L
(
localized zone of dimension ⎡ 2x , 2y ⎤ centered at 4 x , 4 y and
⎦ ) underlying self-similar patterns that can form nested Hilbert
then merge it with the base pattern of the previous level to curves of an arbitrary order. Whenever we refine a block in the
create the new Hilbert curve. The type of this new base pattern implicit nested tree, we simply push the ordering of child
should be selected in such a way that the local direction of the blocks in the original tree. Afterward, using these basic patterns
curve does not change. Before we continue with our example,
at each active level, a sorted index list is prepared for all the
we explore our observation that the choice for the type of
active (or leaf) blocks. This sorted index list is then used to
pattern, to be assigned at any particular child after refinement,
depends on the type of pattern in the previous level and on the distribute the blocks among various processes. The algorithm
number of this specific child where the refinement operation is and corresponding data structure described here is based on
done. We can thus set up a list of all possible choices three building blocks: (i) Select an SFC with self-similar
corresponding to a specific SFC-type and child for this new property and set up the basic pattern for the same, (ii) select the
pattern. When the Hilbert SFC is considered, the possibilities number of children to support the chosen SFC, and (iii) set up a
7
Figure 3. Example showing how the linked list is modified with the refinement according to the underlying SFC skeleton with a Type-II base pattern for the mother
block in pre-refinement configuration.
list similar to Table 3. This approach can easily be used for multiprocessing (OpenMP). Before the simulation starts, we
different SFCs and with slight modifications to the data construct an initial SFC map on a single process until a user-
structures (e.g., adding another parameter to our lists, namely, defined LOR is achieved (note that this initial maximum LOR
the current level). is different than the maximum refinement to be allowed during
the simulation). Afterward, domain decomposition is done
following this SFC, which is then used to distribute the
3.5. Parallelization and Load Balancing computational workload to different processes for parallel
At present, parallelization in SADHANA is only done by computation on distributed memory architectures. When this is
means of a message-passing interface (MPI), and we have no done, we prepare and assign the field variables based on a user-
immediate plan to implement any facility for open defined initial condition. As such, each process evolves the
8
Table 3
Pattern Types of Mother and Child Blocks
1D AMR
Pattern for Mother Pattern for Children

Child 1 Child 2
I I II
II II I
2D AMR

Child 1 Child 2 Child 3 Child 4
I II I III I
II I IV II II
III III III I IV
IV IV II IV III
3D AMR

Child 1 Child 2 Child 3 Child 4 Child 5 Child 6 Child 7 Child 8
I II III III V V X X IX
II III I I XI XI VIII VIII VI
III I II II VII VII IV IV XII
IV VI V V IX IX XII XII III
V IV VI VI I I VII VII XI
VI V IV IV X X II II VIII
VII VIII IX IX III III XI XI V
VIII IX VII VII XII XII VI VI II
IX VII VIII VIII IV IV I I X
X XII XI XI VI VI IX IX I
XI X XII XII II II V V VII
XII XI X X VIII VIII III III IV
simulation in a subset of the full domain, which is defined by a 3.6. Refinement and Derefinement Operations
corresponding subset of the 1D block list already prepared and The refinement/derefinement operation is performed using a
sorted as discussed before. Each of the processes only saves the consecutive series of substeps. Each block has two flag
time-dependent variables associated with the subdomain variables for mesh-update and mesh-refinement operations.
assigned to the same process alone. At any particular time, The value of the update flag can be 0 (no update) or 1 (update
data for a given block belong to only one process, and thus the needed), while the refinement flag can be −1 (derefinement), 0
decomposition remains fully parallel among multiple proces- (no refinement), or 1 (refinement). The default values for all
sors. At any point of time, suppose we have a total number of these flags are set to 0. Refinement or derefinement is
Nb blocks to be distributed over Np processors. As such, we performed only if the mesh-update flag is set to 1. First an
have n and (n + 1) number of blocks distributed over p1 and p2 identification step is performed in an iterative manner over all
processors, respectively, such that Nb = n*p1 + (n + 1) * p2, leaf blocks to mark the required zones for refinement. In this
where Np = p1 + p2. The maximum possible load difference work, we have used a second-order gradient-based error
between two processors at any given time is ΔN = nx * ny * nz, indicator function following Jameson et al. (1981) for the
which is basically the computational load of a single block, and identification operation. For a scalar quantity f, the definitions
for the ideal case, we obtain p2 = ΔNp = (Np − p1) = 0 of this function in 1D (hfi ), 2D (hfi, j ), and 3D (hfi, j, k ) are given by
9
∣ fi + 1 - 2fi + fi - 1∣
h fi =
 + fi + 1 + 2fi + fi - 1
(fi + 1, j - 2fi, j + fi - 1, j )2 + (fi, j + 1 - 2fi, j + fi, j - 1)2
h if, j =
+ (fi + 1, j + 2fi, j + fi - 1, j )2 + (fi, j + 1 + 2fi, j + fi, j - 1)2
(fi + 1, j, k - 2fi, j, k + fi - 1, j, k )2 + (fi, j + 1, k - 2fi, j, k + fi, j - 1, k )2 + (fi, j, k+ 1 - 2fi, j, k + fi, j, k- 1)2

h fi, j, k = , (4 )
+ (fi + 1, j, k + 2fi, j, k + fi - 1, j, k )2 + (fi, j + 1, k + 2fi, j, k + fi, j - 1, k )2 + (fi, j, k+ 1 + 2fi, j, k + fi, j, k- 1)2
where ò is a nonzero number of value 10−20 just to prevent a covered by the coarse cell is defined as
division-by-zero situation. We evaluate the value of η against f
2 ND
å p = 1DVpf f
ˆ ND
two user-defined tolerance factors, namely tolref and tolderef. If Cartesian ⎛2 ˆ f⎞
(0.5 ND ) ⎜ å f
c p
the value of η in any cell of a block becomes greater than tolref, f
ˆ = ⟹ p ⎟, (5 )
DV c Coordinate =
then the value of the mesh refinement flag is associated with the ⎝ p 1 ⎠
current block and its neighbors are all set to 1, tagging them for
where ND is the dimension of the AMR application.
refinement. If η is less than tolderef for all cells in the block and A similar approach is used to correct the values of the face-
its sibling blocks (which are created from the same mother averaged interface fluxes in the boundaries of the coarser
block), then the value of the mesh refinement flag is set to −1 block for ND > 1. Although we can directly compute these
such that we get back the mother block after the derefinement fluxes across coarser cell boundaries connected with finer
operation. When the marking is completed, we iterate over all cells, the conservative property of the numerical scheme
active blocks first to perform derefinement and replace the would be distorted because these values are unlikely to
entries corresponding to all sibling child blocks in the linked exactly match the corresponding interface flux that is
list by their mother during the operation. Next, we perform the computed on the cells belonging to the finer block (one level
refinement process in a similar iterative fashion and push the higher LOR) due to the difference in truncation errors during
entries of newly created children in the linked list to replace discretization. A flux-correction procedure must therefore be
their mother block. After new blocks at finer level are enforced by correcting the interface fluxes on these coarse
produced, linear spatial interpolation is used to project the cells using the values calculated on the finer cells. Given the
solutions from the mother (coarse) block to its children, face area and interface flux of the finer cells in the shared
holding a finer mesh. The details of these numerical operations coarse/fine interface DApf and Fpf , then the correct value of
are given next. Finally, we again reset the values of the the interface flux F c for the coarser mesh with a face area
refinement flags to 0 for all active blocks in the newly updated ΔA c is calculated as
Hilbert SFC. (ND - 1)
å p = 1 DApf Fpf
2
Fc =
DAc
3.7. Restriction, Flux Correction, and Prolongation (ND - 1)
Cartesian 1 ⎛2 ⎞
⟹ ⎛ (N - 1) ⎞ ⎜ å Fpf ⎟. (6 )
Every derefinement and refinement operation is associated Coordinate ⎝ 2 D
⎠ ⎝ p=1 ⎠
with a restriction and prolongation procedure. Whenever
sibling blocks are derefined to create a mother block with Similar to the scenario for restriction, we need to perform a
reduced (by one) LOR, volume-averaged conserved variables prolongation process for the volume-averaged, cell-centered
stored at the cell centers of the computational cells belonging quantities in the coarser cells to calculate the corresponding
to the child blocks are restricted to compute the volume- values in cells belonging to the newly created refined blocks due
averaged value of the variable for cells in the newly created to the refinement process or the ghost cells of a finer block sharing
mother block at lower LOR. The same procedure must also a boundary with a coarser block. Unlike the restriction process,
the prolongation itself is not conservative, and as such, it may be
be followed to obtain the values in ghost cells of a coarse
useful to apply it on the primitive variables. We describe the
block that shares a boundary with a finer (one level higher
f prolongation process for a 3D setup because the equations for
LOR) block. Let DVpf and f̂p be the cell-volume and volume- f
lower dimensions are analogous. f̂p and xpf = [xpf , ypf , zpf ] are
averaged cell-centered value of the variable f (the definition the volume-averaged value of the variable f and the location of
for the volume averaging is given in the next section) for fine finer cells that collectively engulf the domain covered by a coarser
c
cells, while ΔV c and f̂ are the same for a coarse cell. The cell located at xic, j, k = [xic , yjc , zkc], respectively. As such,
c
restriction procedure to obtain the value of f̂ from all the considering a 3D problem, we can follow Table 1 to calculate
finer cells collectively engulfing the computational domain the locations of the finer cells in terms of the directional indices of
10
the coarse block as values of variables such as density and/or pressure are negative
or zero during the prolongation, the simulation must be
Dx ⎞ ⎛ c
c Dy c ⎛zkc - Dz ⎞ ⎤
c
restarted with an increased minimum resolution, which can be
[x1f , y1f , z1f ] = ⎡⎛xic - , y - ⎞,
⎢
⎣⎝ 4 ⎠ ⎝ j 4 ⎠ ⎝ 4 ⎠⎥ ⎦ achieved by increasing the number of computational cells per
each block.
Dx c ⎞ ⎛ y c + Dy ⎞ , D
c c
z
[x 2f , y2f , z 2f ] = ⎡⎛xic - , j
⎛zkc - ⎞⎤ Figure 4 shows a situation in which a block with a higher
⎢
⎣⎝ 4 ⎠ ⎝ 4 ⎠ ⎝ 4 ⎠⎥ ⎦ LOR having coarse cells is placed on top of two finer blocks.
Dx c ⎞ D c
⎛ y c - y ⎞, ⎛zkc - Dz ⎞ ⎤
c
Following the nodes marked for prolongation and restriction,
[x3f , y3f , z3f ] = ⎡⎛xic + , j
⎢
⎣⎝ 4 ⎠ ⎝ 4 ⎠ ⎝ 4 ⎠⎥ ⎦ we can clearly see that the restriction process should be
performed before prolongation because the required nodes for
Dx c ⎞ ⎛ y c + Dy ⎞, ⎛zkc - Dz ⎞ ⎤
c c
[x4f , y4f , z 4f ] = ⎡⎛xic + , j
restriction remain inside the block’s inner zone, while the
⎢⎝
⎣ 4 ⎠ ⎝ 4 ⎠ ⎝ 4 ⎠⎦ ⎥
nodes required for prolongation often require the coarse cells in
Dx c ⎞ ⎛ y c - Dy ⎞, ⎛zkc + Dz ⎞ ⎤
c c the ghost region of the coarse block.
[x5f , y5f , z 5f ] = ⎡⎛xic + , j
⎢
⎣⎝ 4 ⎠ ⎝ 4 ⎠ ⎝ 4 ⎠⎥ ⎦
3.8. Base Numerical Schemes and Time Integration
Dx c ⎞ D y c
D z c
[x 6f , y6f , z 6f ] = ⎡⎛xic + , ⎛y +
j
c ⎞ , ⎛z k +
c ⎞⎤ The block-based semidiscrete form of Equation (1) for the
⎢
⎣⎝ 4 ⎠ ⎝ 4 ⎠ ⎝ 4 ⎠⎥ ⎦ 3D problem is given by
Dx c ⎞ ⎛ c Dy c ⎞ ⎛zkc + Dz ⎞ ⎤
c
[x 7f , y7f , z7f ] = ⎡⎛xic - , yj - , d (uˆ i, j, k )b 1 ˜
⎢
⎣⎝ 4 ⎠ ⎝ 4 ⎠ ⎝ 4 ⎠⎥ ⎦ =- ( fx, i + 1 , j, k - f˜x, i - 1 , j, k )b
dt Db x 2 2
Dx c ⎞ ⎛ c Dy c ⎞ D z c
[x8f , y8f , z8f ] = ⎡⎛xic - , yj + , ⎛zkc + ⎞ ⎤. 1 ˜
⎢
⎣⎝ 4 ⎠ ⎝ 4 ⎠ ⎝ 4 ⎠⎥ ⎦ - ( fy, i, j + 1 , k - f˜y, i, j - 1 , k )b
Db y 2 2
(7 )
1 ˜
Then the prolongation procedure for the finer cells is defined - ( fz , i, j, k+ 1 - f˜z , i, j, k- 1 )b + (ˆsi, j, k )b , (10)
Db z 2 2
as
~ where ()b represents the calculation being done for the bth
f c
fˆ p (xpf ) = fˆ i, j, k + (¶x fˆ )i, j, k (x¯ic - x¯pf ) block, and û, ŝ are the volume-averaged values of U, S stored
~ ~ in the cell centers of the block. They are defined as
+ (¶y fˆ )i, j, k ( y¯jc - y¯pf ) + (¶z fˆ )i, j, k (z¯kc - z¯pf ) , (8)
1 x i + 1 , b yj + 1 , b
(uˆ i, j, k )b := ò 2 ò 2
where x¯pf = [x¯pf , y¯pf , z¯pf ] and x¯ic, j, k = [x¯ic , y¯jc , z¯kc] are the Db Vi, j, k x i - 12 yj - 12 , b
volume-averaged cell centers of the finer cell with child id p ´ò
zk+ 1 ,b
2 U (t , x , y , z) dVb,
and the coarse cell, respectively. For a Cartesian coordinate zk- 1 ,b
2
system, the volume-averaged cell centers coincide with the 1 x i + 1 , b yj + 1 , b
geometric cell centers, i.e., x¯pf = xpf and x¯ic, j, k = xic, j, k . (ˆsi, j, k )b := ò 2
ò 2
Db Vi, j, k i - 12 , b yj - 12 , b
x
Approximated values of the directional gradients are calculated zk+ 1 ,b
using the simple minmod limiter as ´ò 2 S (t , x , y , z) dVb.
zk- 1 ,b
2
~
(¶x fˆ )i, j, k = minmod Similarly, f˜x , f˜y , f˜z are the average directional flux across cell
⎛ fˆ i + 1, j, k - fˆ i, j, k fˆ i, j, k - fˆ i - 1, j, k ⎞ interfaces at xi + 12 , yj + 1 , zk + 12 , respectively. For example, f̃x is
´⎜ , defined as
2
x¯ic+ 1 - x¯ic x¯ic - x¯ic- 1 ⎟

⎝ ⎠
~ ( f˜x, i + 1 , j, k )b :=
1 yj + 1 , b
(¶y fˆ )i, j, k = minmod 2
ò 2
Db Ax, i + 12 , j, k yj - 12 , b
⎛ fˆ i, j + 1, k - fˆ i, j, k fˆ i, j, k - fˆ i, j - 1, k ⎞ ´ò
zk+ 1 ,b
fx (t , xi + 12 , y , z) dAx, b .
´⎜
2
, zk- 1 ,b
y¯jc+ 1 - y¯jc y¯jc - y¯jc- 1 ⎟ 2
⎝ ⎠
~ In these definitions, we use ΔbVi,j,k as the volume of the
(¶z fˆ )i, j, k = minmod computational cell in the bth block with the cell center located at
⎛ fˆ i, j, k+ 1 - fˆ i, j, k fˆ i, j, k - fˆ i, j, k- 1 ⎞ (xi, b , yj, b , zk, b ) and Db Ax, i + 12 , j, k as the surface normal to ( f˜x )b at
´⎜ , . (9 ) xi + 12 , b . For a Cartesian coordinate system, as considered here,
z¯kc+ 1 - z¯kc z¯kc - z¯kc- 1 ⎟
⎝ ⎠ we have Db Vi, j, k = (Db x )(Db y)(Db z ) and Db Ax, i + 12 , j, k =
It should be noted that the choice for the slope limiter in the (Db y)(Db z ). For the xi - 12 , b interface and all the other fluxes,
prolongation process is restricted to minmod alone because in the definition is analogous. The full simulation algorithm in our
the end, we try to construct a multidimensional profile using a AMR solver uses a second-order Runge–Kutta time integration to
linear combination of 1D gradients. Our numerical tests suggest update the solution (uˆ i(,nj,)k )b , b = 1, 2,¼,(nb )n available at the nth
that any other limiter that is sharper than minmod leads to
significant oscillations near the interfaces. This is consistent time step to (uˆ i(,nj,+k 1) )b , b = 1, 2 ,.., (nb )n + 1 at the (n + 1)th time
with the recent work done by Stone et al. (2020), and as such, step. Note that at each processor, if (nb )n is the maximum number
we refrain from finding alternative options at present. When the of leaf blocks at the nth time step, then we can have
11
Figure 4. Schematic diagram of a coarse 2D block on top of two fine blocks. The regions near the shared coarse/fine boundaries between the top block and bottom
block engage in the restriction and prolongation process.
(nb )n = (nb )n + 1 or (nb )n > (nb )n + 1 or (nb )n < (nb )n + 1 depend- center using these formulations:
ing on the refinement criteria.
1
qîL+ 1 , j, k = qî, j, k + a (rqˆx )(qî + 1, j, k - qî, j, k )
2 2
3.8.1. Flux Calculation 1
qîR- 1 , j, k = qî, j, k - a (rqˆx )(qî + 1, j, k - qî, j, k ) , (11)
The interface fluxes, namely f˜x, i  1 , j, k , f˜y, i, j  1 , k , and f˜z, i, j, k  1 , 2 2
2 2 2
ignoring the ()b indices for simplification, are reconstructed from qî, j, k - qî - 1, j, k
ûi, j, k following a two-step procedure. The calculation procedure where rqˆx = qî + 1, j, k - qî, j, k
is the ratio of the differences
of f˜x, i  1 , j, k is explained here: along the x-direction, and α(r) is a limiter function to
be calculated using rqˆx . We have implemented a
2
1. The cell-centered values of the primitive state vector q̂i, j, k variety of second-order limiters including van Leer
are obtained from the conservative counterpart following
the physics-driven EOS. We follow the second-order
( r+ ∣ r ∣
)
avl = 1+ ∣ r ∣ as in van Leer (1974), monotonized-
central (amc = max [0, min (2r , 0.5 (1 + r ), 2)]) as in
variation of the uonotone Upwind-central schemes for
Van Leer (1977), minmod (amm = max [0, min (1, r )])
conservation laws (MUSCL) scheme used by Nessyahu and superbee (asb = max [0, min (2r , 1) min (r , 2)]) as
& Tadmor (1990) and Kurganov & Tadmor (2000). In
this approach, the left state qîL+ 1 , j, k and the right state
r + r2
in Roe (1986), and van Albada ava = 1 + r 2 as in van ( )
Albada et al. (1997). Unless specified otherwise, we
2
1
( 1
)
qîR- 1 , j, k at the interface i + 2 , j, k and i - 2 , j, k ,
2
( ) have used monotonized-central type limiters for all the
respectively, is computed from q̂i, j, k available at the cell numerical tests reported in this paper.
12
Figure 5. 1D HD Sod result at t = 0.2.
2. Afterward, a physics-specific Riemann solver is applied

to calculate the interface flux using the interpolated face-
2. Calculate Δt as Dt = CFL * min ( Dx Dy Dz
lx
, l , l
y z ) , with
l
averaged values of the left and right states. Basically, the l = 1, 2,¼,l max being the active refinement levels, and
Riemann solver defines a flux function  Riemann such that λx, λy, λz being the maximum characteristic speeds of the
system at x, y, z directions.
f˜x, i  1 , j, k =  Riemann(qîL 1 , j, k , qîR 1 , j, k ). 3. For each bth leaf block, calculate the interface fluxes
(n ) (n ) (n )
2 2 2 ( f˜x, i  1 , j, k )b , ( f˜y, i, j  1 , k )b , and ( f˜z, i, j, k  1 )b using the solu-
tion available at the nth time step, i.e., U( n) following the
2 2 2
In this work, we have used HLLC (as discussed in

previously described procedures. Perform flux synchro-
Appendix A) and HLLD (as discussed in Appendix B)
nization over the boundaries shared by different refine-
Riemann solvers for hydrodynamic and MHD systems,
ments following Equation (6).
respectively. 4. When the interface fluxes are prepared, loop over all leaf
blocks to calculate the hyperbolic increment along with
the source term and obtain the intermediate solution
3.8.2. Time Integration (uˆ i(,1j,)k )b for every bth block as
SADHANA currently uses a second-order accurate Runge– (uˆ i(,1j,)k )b = (uˆ i(,nj,)k )b
Kutta method without any subcycling for time integration. The
procedure followed is presented here. Dt ˜ (n) (n )
-⎡ ( fx, i + 1 , j, k - f˜x, i - 1 , j, k )b
1. A projection approach is used at the beginning of ⎣ Db x
⎢ 2 2
each time integration to project the solution available Dt ˜ (n) (n )

+ ( fy, i, j + 1 , k - f˜y, i, j - 1 , k )b
at the nth time step to the AMR framework for Db y 2 2
the (n + 1) th time step such that we obtain Dt ˜ (n)

[(uˆ i(,nj,)k )bn , bn = 1, 2,¼,(nb )n ]  [(uˆ i(,nj,)k )bn + 1, bn + 1 = 1, 2 ,.., (nb )n + 1], + (f ˜ (n)
1 - fz , i, j, k - 1 )b
⎤
with nb being the maximum number of leaf blocks per Db z z , i, j, k + 2 2 ⎥
⎦
processing unit. This is attained through the following set + (ˆsi(,nj,)k )b Dt. (12)
of operations:
(a) check the refinement criterion of all active blocks at We should note that for the EGLM-MHD case, we do not
the current process, include the source term for the ψ equation here, as that is
(b) derefine and restrict selected sibling blocks following taken care of later.
Equation (5), 5. Loop over all leaf blocks at different LOR and perform
(c) refine and perform prolongation blocks following prolongation/restriction on (uˆ i(,1j,)k )b as required in the
Equation (8), regions covered by ghost cells and apply the boundary
(d) update the doubly linked list of the leaf block, condition on (uˆ i(,1j,)k )b .
(e) redistribute leaf blocks equally among all processors (1) (1)
6. Calculate the interface fluxes ( f˜x, i  1 , j, k )b , ( f˜y, i, j  1 , k )b , and
following the modified Hilbert SFC as explained before, 2 2
(1)
(f) finally, we have (uˆ i(,nj,)k )bn + 1, bn + 1 = 1, 2 ,.., (nb )n + 1. ( f˜z, i, j, k  1 )b using the intermediate solution and proceed to
2
13
Figure 6. 1D HD Shu–Osher result at t = 1.8.
perform flux synchronization over the boundaries shared 9. We need to make a final update of the solution, where we
by different refinements for these fluxes like before. loop over all leaf blocks at different LOR and perform
7. Now loop over all leaf blocks to calculate the hyperbolic prolongation/restriction on (uˆ i(,nj,+k 1) )b , as required in the
increment along with the source term using the regions covered by ghost cells and apply the boundary
intermediate solution and compute (uˆ i(,nj,+k 1) )b at (n + 1) condition on (uˆ i(,nj,+k 1) )b for each bth leaf block at the
th time as domain boundaries.
1 (n) 1
(uˆ i(,nj,+k 1) )b = (uˆ i, j, k ) + (uˆ i(,1j,)k )
2 2 4. Numerical Examples and Computational Performance
⎡
-⎢
( ) 1
2
Dt
(1) (1)
( f˜x, i + 1 , j, k - f˜x, i - 1 , j, k )b
While the main novelty of this work includes the aforemen-
tioned modified definition of block boundaries and the listed
⎢ Db x 2 2
approach to prepare face-continuous Hilbert-type SFC, we also
⎣
show the flexibility and application of our adaptive mesh solver
+
( Dt ) ( f˜
1
2 (1) (1)
- f˜y, i, j - 1 , k )b
by means of numerical examples. Both hydrodynamics and
y, i, j + 12 , k MHD type of physics have been considered in these examples.
Db y 2
+
( ) ( f˜
1
2
Dt (1) (1) ⎤
- fˆz , i, j, k- 1 )b⎥
4.1. Hydrodynamic Tests
z , i, j, k + 12
Db z 2
⎥ 4.1.1. 1D Sod Shock Tube
⎦
1 Following the conventional approach of code validation, we
+ (˜si(,1j,)k )b ⎛ Dt ⎞. (13) start with the classic shock tube test as mentioned in Sod
⎝2 ⎠
(1978). The initial condition for the test is given by
Once again, we exclude the source term for the ψ
equation, as before. Q = (r , vx , vy , vz , pt )T
8. Now that the integration is completed, we need to address
(1,0, 0,0,1)T if x  xs ,
some physics-specific operations. For the EGLM-MHD =⎧ T if x > x .
(14)
c2 ⎨ ( 0.125,0, 0,0,1 )
system, this is where we accommodate the ch2 y term ⎩ s
p
in S for the ψ equation, and we add the source for
the ψ equation analytically. With ψ being the value
old We set the initial discontinuity location at xs = 0.5 and the
of ψ at the end of the previous step, the new updated adiabatic index γ = 1.4. A variety of cells/block is considered,
and we use CFL = 0.4 for all cases. We run the simulation for
value of ψ becomes y new = y old exp ⎡-ap Dhh ⎤, where
c
⎢ tmax = 0.2 in the computation domain [0 x 1]. As can be
⎣ ( Dt ) ⎥⎦ seen from the results given in Figure 5, all three regions of the
ch
ap = Dh 2 . The value of Δh is taken as system, namely left rarefaction, contact discontinuity, and right
c p
min (Dx, Dy, Dz ) over all the leaf blocks. shock discontinuity, are captured in our solution.
14
Figure 7. 2D HD, isentropic vortex problem: L1 and L2 error of density and thermal pressure at t = 10.0.
Figure 8. 2D HD, Kelvin–Helmholtz instability problem: Results shown in terms of density contour at t = 1.50. While both simulations are done with maximum
refinement as l max = 7, in panels (a) and (b), we have 4 × 4 and 16 × 16 nodes in each block, respectively, leading to peak resolutions of 256 × 256 for panel (a) and
1024 × 1024 for panel (b).
4.1.2. 1D Shu–Osher Shock Tube 4.1.3. 2D Isentropic Vortex

Next we consider the shock tube test described in Shu & The problem is described in Yee et al. (2000) and is a very
Osher (1988). The initial profile is given by good test for the accuracy of hydrodynamic solvers because an
exact solution is available. The initial condition can be defined
Q = (r , vx , vy , vz , pt )T in primitive state as
(3.857143, 2.629369,0,0, 10.3333)T if x  xs,
=⎧ (15) ⎧
1
(g - 1) b 1 - r 2⎤g- 1 ⎫
⎨(1 + Ar sin ( fr x ) , 0,0, 0,1)T if x > xs. ⎪r = ⎡1 - e ⎪ ⎧ vx = 1⎫
⎩ Q= ⎢
⎣ 8gp ⎥
⎦ , v0 = vy = 1 ,
⎨ v = v0 + dv ⎬ ⎨ ⎬
We set the initial discontinuity location at xs = −4.0, and the ⎪ g ⎪ ⎩ vz = 0 ⎭
⎩ pt = r ⎭
density perturbation is defined by the amplitude Aρ = 0.2 and
⎧dvx = b e 1 -2r ( y - y ) ⎫
2
the frequency fρ = 5.0. The computational domain is set as ⎪ 2 p 0
⎪
⎪ ⎪
[−5 x 5], and the zero-gradient boundary condition is used dv = b 1-r 2 . (16)
⎨dvy = e 2 (x - x 0 )⎬
at both boundaries. With γ = 1.4 and CFL = 0.4, we run the ⎪ 2p ⎪
⎪dvz = 0 ⎪
simulation until tmax = 1.8. Figure 6 shows the final result. ⎩ ⎭
15
Figure 9. 2D HD double Mach reflection problem: Schematic diagram of the simulation setup is given in panel (a), while the density contour at time t = 0.2 is shown
in panels (b) and (c). For the computation shown in panel (b), we define each block to have 4 × 4 nodes, while for panel (c), we use 4 × 16 nodes for each block.
Because the maximum LOR is defined as l max = 7, the peak resolution for both simulations remains the same, i.e., 256 × 1024.
Figure 10. 1D MHD Brio–Wu result at t = 0.12.
16
Figure 11. 1D MHD Shu–Osher–Susanto result at t = 0.7.
Figure 12. 2D MHD isodensity vortex problem: L1 and L2 error of the thermal pressure and x-component of the magnetic field at t = 10.0.
We have used x0 = 0, y0 = 0 as the initial location of the 4.1.4. 2D Kelvin–Helmholtz Instability

vortex and β = 5. The spherical symmetry of the system is This problem is designed and described in McNally et al.
disturbed by v0 because otherwise, our system would lead to (2012). The setup includes a periodic domain of size
a time-invariant state because the centrifugal force is exactly 1
[0 x L, 0 y L] and two shear layers at y = 4L and
balanced by the pressure gradient toward the vortex center. 3
The simulation is run using CFL = 0.4 with periodic y = 4L . The initial condition is represented in primitive
boundaries in all direction. The computational domain is variables as
defined by [−5 x 5, −5 y 5] and an adiabatic index Q = (r , vx , vy, vz , p )T
with the value γ = 7/5 is used. We run the simulation until
= ⎧ g ([r ]1 , [r ]2 ), g ([vx ]1 , [vx ]2 ),
tmax = 10 . Second-order convergence is achieved in L1 and ⎨
⎩
L2 norms of error for both density and pressure, as shown in k (2px ) ⎞
T
Figure 7. lsin ⎛ , 0, 2.5⎫ (17)

⎝ L ⎠ ⎬
⎭
17
where we use a smoothing function g() for the density and the waves. The initial state is given by
x-component of the velocity, parameterized by a smoothing
Q = (r , vx , vy , vz , pt , Bx , By , Bz )T
parameter s,
(1,0, 0,0,1, 0.75, 1.0,0)T if x  xs ,
g ([a]1 , [a]2 ) =⎧ (20)
y/L -1/4
⎨
⎩ ( 0.125,0, 0,0,1, 0.75, - 1.0,0 ) T if x > x .
s
⎧[a] 1
- ([a]1 - [a]2 ) e s / L if 1/4 > y/L  0
⎪ 1 2 We set xs = 0.5 and the adiabatic index γ = 2. We simulate the
⎪ -y / L + 1 / 4
⎪[a]2 1
+ ([a]1 - [a]2 ) e s / L if 1/2 > y/L  1/4
problem until tmax = 0.12 with CFL = 0.4 in a computational
=
⎪ 2 . domain of [0 x 1]. At both boundaries, we keep the
y/L -3/4
⎨ 1 boundary values the same as the initial profile. The final results
⎪[a]2 + ([a]1 - [a]2 ) e s / L if 3/4 > y/L  1/2
⎪ 2 are shown in Figure 10.
-y / L + 3 / 4
⎪ 1
⎪[a]1 - ([a]1 - [a]2 ) e s / L if 1 > y/L  3/4 4.2.2. 1D Shu–Osher–Susanto Shock Tube
⎩ 2
(18) We consider a modified version of the Shu–Osher shock tube
case, as developed by Susanto for an MHD problem
We take L = 1, s = 0.025, [r ]1 = 1.0 , [r ]2 = 2.0 , [vx ]1 = 0.5, (Susanto 2014). The initial conditions for this test are
[vx ]2 = -0.5, k = 2, and λ = 0.01. Other relevant computa-
5 Q = (r , vx , vy, vz , pt , Bx , B y , Bz )T
tional parameters are CFL = 0.4 and the adiabatic index g = 3 .
Figure 8 shows the results at t = 1.5 and can directly be ⎧ (3.5, 5.8846, 1.1198,0, 42.0267,1, 3.6359,0) if x  xs ,
T
=
compared with Figure 2 of McNally et al. (2012). ⎨ (1 + ar sin ( fr x ), 0,0, 0,1, 1,1,0) T
if x > xs .
⎩
(21)
4.1.5. 2D Double Mach Reflection
This problem is described in Woodward & Colella (1984) Similar to the hydrodynamic case, we solve the problem in the
and is a classical two-dimensional test case for hydrodynamics [−5 x 5] domain with a zero-gradient boundary condition
solvers. The computational domain for this problem is and set xs = − 4. The adiabatic index is taken as γ = 5/3, and
[0 x 4, 0 y 1], and a plate lies at the bottom of the we run the simulation until tmax = 0.7 with CFL = 0.4. This
computational domain starting form x = 1/6. Initially, a right- problem has no analytic solution, so we use a highly refined
moving Mach = 10 shock is positioned across x = 1/6, y = 0 grid (with N = 131072) to compute the reference solution. The
and makes a 60° angle with the x-axis. final result is given in Figure 11.
Q = (r , vx , vy , vz, pt )T
T
4.2.3. 2D Isodensity MHD Vortex
⎛p ⎞ ⎛p ⎞
⎧⎛ y
⎪ 8, 8.25 cos ⎝ 6 ⎠, -8.25 cos ⎝ 6 ⎠, 0, 116.5 if x < xs = x 0 +
⎞
= ⎝ ⎠ 3, This problem was described in Balsara (2004), and, just like
⎨
(1.4,0, 0,0,1 ) T if x  xs = x 0 +
y Section 4.1.3, is an excellent candidate for a convergence study
⎪
⎩ 3 of an MHD solver because an exact solution is available. The
(19) initial condition of the primitive state vector Q is defined by
adding a perturbation δQ to the unperturbed background Q̄
with x0 = 1/6. We set the Dirichlet boundary values corresp- such that
onding to the postshock state at the left boundary (x = 0) and
part of the bottom boundary (y = 0), precisely in ⎡0  x < 6 ⎤
1 Q = Q¯ + dQ,
⎣ ⎦
and for the rest of the bottom boundary [x 0 , 4], we have set a ⎧r = 1 ⎫
⎧ vx = 1⎫
⎪ ⎪
reflecting boundary condition. An outflow boundary condition ⎪ ⎪
dQ
with dx = 0 is used at the right boundary (x = 4). At the upper ⎪ v0 = ⎨ vy = 1⎬ ⎪
⎪ ⎩ vz = 0 ⎭ ⎪
boundary, y = 1, the flow has to be imposed such that there is Q¯ =
⎨ pt = 1 ⎬
no interaction with the moving shock. The exact location of the ⎪
B = 1
⎪
shockwave at y = 1 at any time t is given by s (t ) = ⎪ ⎧ x ⎫⎪
⎪ B0 = By = 1 ⎪
1 + 20t ⎨ ⎬⎪
x0 + . As such, we use postshock values for 0 x < s(t) ⎪
⎩ Bz = 0 ⎭⎭
3 ⎩
and preshock values at s(t) x 4. Figure 9 shows the result at
⎧dr = 0 ⎫ (22)
t = 0.2. This test is a very good example to show the capability ⎪dv = - y e0.5(1 - r 2) ⎪
of SADHANA of using both square and rectangular blocks for ⎪ x 2p ⎪
⎪ x 0.5(1 - r 2) ⎪
hydrodynamic problems. ⎪dvy = e ⎪
⎪ 2p ⎪
⎪dvz = 0 ⎪
4.2. Magnetohydrodynamic Tests ⎪ ⎪
dQ = x 2 r 2 (1 - r 2 ) .
⎨dpt = -⎛ ⎞ e ⎬
4.2.1. 1D Brio–Wu Shock Tube ⎪ ⎝ 2p ⎠ 2 ⎪
⎪ y 0.5(1 - r 2) ⎪
dB = - e
An extension of the Sods shock-tube problem with magnetic ⎪ x
2p ⎪
fields was given in Brio & Wu (1988). This test can be used to ⎪ x 0.5(1 - r 2) ⎪
⎪dBy = e ⎪
verify the ability of an MHD solver to capture the characteristic ⎪ 2 p ⎪ (2)
⎪dBz = 0 ⎪
waves, i.e., the shocks, rarefactions, and contact discontinuity ⎩ ⎭
18
Figure 13. 2D MHD Orszag–Tang vortex problem: Results at t = 0.25, 0.35, and 0.50 in panels (a), (b), and (c), respectively, are shown in terms of density (I),
hydrodynamic Mach number (II), and magnitude of the magnetic field (III). During this simulation, each block has 16 × 16 nodes, with the maximum LOR defined as
l max = 7, leading to an effective resolution of 1024 × 1024 over our computational domain. The dynamic mesh refinement at different times of computation is shown
in II.
We have r = (x - x 0 )2 + ( y - y0 )2 , with x0 = 0, y0 = 0 We have plotted the L1 and L2 norms of error for the pressure
being the vortex location during initialization. Since the vortex and x-component of the magnetic field as obtained from the
is basically being advected under equilibrium conditions, solution at t = 10 in Figure 12. As we can see, the results
similar to the isentropic vortex problem discussed before, follow a second-order convergence rate, which is expected
we can obtain the exact solution at any time t as given our numerical method.
U (x , t ) = U (x - v0 t , 0). The problem is solved in the domain
defined as [−5 x 5, −5 y 5] with periodic boundary 4.2.4. 2D Orszag–Tang Vortex
boundaries in all directions. Other noteworthy conditions for This problem was first studied in Orszag & Tang (1979) for
the problem include a CFL value of 0.4 and an adiabatic index incompressible MHD equations. Many authors have used the
of γ = 5/3. The simulation is run until tmax = 10 when the Orszag–Tang vortex for compressible MHD equations as a test
vortex returns exactly to the location where it was at t = 0. problem in order to know how robust the employed numerical
19
Figure 14. 2D MHD cloud–shock interaction problem: Results at t = 0.0, 0.05, and 0.10 in terms of density and thermal pressure contour. For I.(a)–(c), each AMR
block has 4 × 4 nodes, while for II.(a)–(c), each AMR block has 4 × 8 nodes. Since we use the maximum refinement level = 7 for both simulations, the peak
resolution remains the same, i.e., 256 × 512.
scheme is at handling the formation and the interactions of covering the computational domain of [0 x 2, 0 y 1]
MHD shocks. We set the problem up in a computational and a high-density circular cloud with radius rc = 0.15 located
domain defined as [0 x 1, 0 y 1]. With an adiabatic at xc = 1.6, yc = 0.5. The cloud properties are defined by Qc
index γ = 5/3, the initial condition for the problem is defined such that the cloud is in hydrostatic equilibrium with the
as ambient medium,
Q = (r , vx , vy, vz, pt , Bx , By, Bz )T
Qa = (r, vx , vy, vz , pt , Bx , By, Bz ) T
= (g 2 , - sin (2py), sin (2px ), 0, g , sin (2py) , sin (4px ) , 0)T .
(3.86859,0, 0,0, 167.345,0, 2.1826182, -2.1826182) T if x  xs ,
(23) =⎧
⎩ (1, -11.2536,0, 0,1,0, 0.56418958, 0.56418958) if x > xs
⎨ T
We consider periodic boundaries in all directions and run the Qc = (r, vx , vy, vz , pt , Bx , By, Bz ) T
simulation until tmax = 1.0 with CFL = 0.4. Figure 13 shows = (10, -11.2536,0, 0,1,0, 0.56418958, 0.56418958) T .
the results at different times. (24)
4.2.5. 2D Cloud–Shock Interaction We evolve the system with CFL = 0.4 and adiabatic index
This problem describes a a strong shockwave interacting γ = 5/3 until tmax = 0.1. The transmissive-type boundary
with a dense cloud. We use the well-discussed setup described condition is used at the domain boundaries. Figure 14 shows
in Dai & Woodward (1994) and used in Toth (2000). We have how the solution evolves in terms of density and pressure
the ambient condition Qa with a strong shock at xs = 1.2 contour. Just like Section 4.1.5, this test case presents the
20
Figure 15. Strong (left) and weak (right) scaling results for isentropic vortex (HD) and isodensity vortex (MHD) problems using SADHANA with AMR.
capability of SADHANA of using both square and rectangular AMR. A second-order accurate reconstruction scheme has been
blocks for MHD problems. used throughout the paper, and approximate Riemann solvers
for HD and MHD systems are implemented. While in this
4.3. Computational Performance and Scaling work, we have limited ourselves on implementation of ideal
HD-MHD physics to test and demonstrate capabilities of
To assess the computational efficiency of SADHANA, we SADHANA, more advanced physics can be added with little
have run several strong and weak scaling tests. The isentropic effort. Throughout our numerical tests, our solver has
(Section 4.1.3) and isodensity (Section 4.2.3) vortex problems demonstrated to be a robust and versatile numerical framework,
are extended to 3D with [−5 z 5] and are chosen as model and it can be used to investigate realistic astrophysical
problems for an HD and MHD scaling test, respectively. Each problems in the future. SADHANA is capable of using
block has 163 computational cells for strong scaling tests, and we arbitrary even numbers of grid points in each direction, which
run the simulation for 1000 time steps. We vary the number of makes it a very good candidate for the implementation of high-
mpi processes as [Np = 4, 8, 16, 32, 64, 128, 256] and record the order schemes in the future. It should be explicitly noted that at
time taken for each case. Afterward, for each case, the present, we have only used a Cartesian coordinate system in
computational time is normalized by the time taken for Np = 4 our code. Extensions to other coordinate systems are left for
to reach the final speedup result. The weak scaling test for our future development. Also, we restrained ourselves from using
AMR framework appeared to be tricky because we needed to any subcycling approach in our implementation of time
maintain the same amount of computational load for each MPI
integration. The solver currently uses cell-centered data for
process. As such, we ended up having a higher resolution for our
each computational cell. While we intend to modify SAD-
simulation as we kept increasing the number of MPI processes
HANA to handle staggered data in the future, it is not on our
(Np) and the refinement criteria needed to be modified to make
priority list at present. For postprocessing purposes, SAD-
sure that the mesh was refined in the same regions for all cases.
HANA saves all block definitions and the associated variables
Each MPI process in our weak scaling tests has eight blocks, and
within the blocks in a file, and Python-based scripts are used to
each block has 163 computational cells, just like the case for
plot/analyze these data. We are interested in preparing a better
strong scaling tests. The HPC cluster uses Intel Xeon Platinum
option to view/process the output data in the future.
8280L CPUs and is equipped with the Simple Linux Utility for
Resource Management (SLURM) as its workload manager.
The authors acknowledge grant support for the Computational
Figure 15 shows the final result of our scaling tests, which is
Astrophysical Sciences (CompAS) group under Theory from the
more than adequate for our use at present.
Institute of Astronomy and Astrophysics, Academia Sinica
(ASIAA), and the National Science and Technology Council
5. Summary and Conclusion (NSTC) in Taiwan through grants 105-2119-M-001-044-MY3,
We have described the relevant algorithms used in a newly 108-2112-M-001-009-, 109-2112-M-001-028-, and 110-2112-
developed AMR framework called SADHANA. A new M-001-019-. The authors also acknowledge the access to high-
definition of finite-volume boundaries is given, which can performance facilities (TIARA cluster and storage) in ASIAA,
assist the development of block-structured AMR codes. A and thank the National Center for High-performance Computing
novel list-based method is also presented here to generate (NCHC) of National Applied Research Laboratories (NARLabs)
multidimensional Hilbert SFCs, which can be applied beyond in Taiwan for providing computational and storage resources.
21
Appendix A
HLLC Riemann Solver
The Harten–Lax–vanLeer–Contact (HLLC) solver, as presented by Toro et al. (1994), is used for adiabatic hydrodynamic
problems in SADHANA. It is a modification of the very frequently used Harten–Lax–vanLeer (HLL) solver pioneered by Harten
et al. (1983). Further developments in this direction were made by Toro & Chakraborty (1994) and Batten et al. (1997). For
simplification purposes, we use qL and qR instead of qîL+ 1 , j, k and qîR+ 1 , j, k for left and right states. The conservative states uL/R are
2 2
calculated from their primitive counterparts qL/R following* the specific EOS (adiabatic hydrodynamics in this case).
The Riemann fan is divided into one intermediate state u obtained between three waves with slopes S L, S R, and S M. The values of
these slopes are estimated from qL and qR as
SL = min [min (vxL - c L , vxR - c R) , 0]

SR = max [max (vxL + c L , vxR + c R) , 0]
r RvxR (SR - vxR) - r LvxL (SL - vxL ) - ( ptR - ptL )
S* = .
r R (SR - vxR) - r Lx (SL - vxL )
* *
Afterward, we estimate the intermediate states as determined for uL and uR as
K* ⎧ ⎛ 1 ⎞⎫
⎧ r ⎫ ⎪ S *
⎪ r K *v ⎪
K * ⎜ ⎟⎪
x ⎪ v K ⎪
⎪ ⎪ ⎪ S K - vK ⎜ y ⎟⎪
*
uK * = r K *vy K
= rK K x
⎜ vzK ⎟ , K = L, R. (A1)
⎨ ⎬ ⎨ S - S* ⎬
⎪ r K *vzK *⎪ ⎪ ⎜
ptK ⎟⎪
⎪ K* K*⎪ ⎪ ⎜ e K + (S * - vxK ) ⎡S * + ⎤⎟ ⎪
⎩r e ⎭ ⎪ ⎜ ⎢
⎣
K ⎥⎟
r (S - u x ) ⎦ ⎪
K K
⎩ ⎝ ⎠⎭
And finally, we have the HLLC flux function,
L
⎧ f (u ) if SL > 0
⎪ f (uL ) + SL (uL * - uL ) if SL  0  S *
 HLLC(uL , uR ) = . (A2)
⎨ f (uR ) + SR (uR * - uR ) if S *  0  SR
⎪ R
⎩ f (u ) if SR < 0
Appendix B
HLLD Riemann Solver
SADHANA uses the Harten–Lax–van Leer Discontinuities (HLLD) Riemann solver, as proposed in Miyoshi & Kusano (2005),
for adiabatic MHD problems. Some modifications are made to the classic HLLD algoriothm to accommodate the EGLM-MHD
system (or rather the GLM-MHD system because the structure of the HLLD solver does not change, regardless of whether we
consider GLM or EGLM) without changing the basic HLLD scheme. As in the case of the HLLC solver, we use qL and qR for the left
and right states instead of qîL+ 1 , j, k and qîR+ 1 , j, k . At first, we calculate the conservative states uL/R from their primitive counterparts qL/R.
2 2
Afterward, we calculate the variables that are responsible for the construction of our GLM system,
⎧ 1 (BxR + BxL ) - 1
(y R - y L ) ⎫
B ⎪ ⎪
QGLM =⎧ GLM ⎫= 2 2ch
.
⎨ y
⎩ GLM ⎭ ⎨ 1 (y R + y L ) -
⎬ ch R ⎬
⎪2 (Bx - BxL ) ⎪
⎩ 2 ⎭
* * * *
In the HLLD approach, the Riemann fan is divided into four intermediary states uL , uL *, uR *, and uR obtained between the five
waves, namely, two fast magnetosonic waves [l-f = vx - cf and l+f = vx + cf ], one entropy wave [λe = vx], and two Alfvén waves
gpt
[l-A = vx - cA and l+A = vx + cA]. With the sound speed being c = r
, the fast magnetosonic speed cf and Alfvén sound speed cA
22
are given by
Bx
cA =
r
BGLM
=
r
2
1 ⎡⎛ gp ⎧⎛ gp gp B 2 ⎫ ⎤
cf = ⎢⎜ + Bx2 + By2 + Bz2⎞⎟ + ⎜ + Bx2 + By2 + Bz2 ⎞⎟ - 4 ⎛⎜ ⎞⎟ ⎛⎜ x ⎞⎟ ⎥
2 ⎢⎝ r ⎨⎝ r ⎝ r ⎠⎝ r ⎠⎬
⎣
⎠ ⎩ ⎠ ⎭⎥
⎦
2
1 ⎡⎛ gp ⎧⎛ gp gp B 2 ⎫⎤
= ⎢⎜ + BGLM2
+ By2 + Bz2⎞⎟ + ⎜ + BGLM
2
+ By2 + Bz2 ⎞⎟ - 4 ⎛⎜ ⎞⎟ ⎛⎜ GLM ⎞⎟ ⎥
2 ⎢⎝ r ⎨⎝ r ⎝ r ⎠⎝ r ⎠⎬
⎣
⎠ ⎩ ⎠ ⎭⎥
⎦
1 2
= [(c + ∣ B ∣2 ) + (c2 + ∣ B ∣2 )2 - 4c2cA2 ] .
2
* *
The corresponding slopes S L, S R, S M, S L , and S R in the Riemann fan are estimated from the components of both qL and qR,
S L = min [min (vxL , vxR) - max (cfL , cfR) , 0]

S R = max [max (vxL , vxR) + max (cfL , cfR) , 0]
(S R - vxR ) r RvxR - (S L - vxL ) r LvxL - ( ptR - ptL )
SM =
(S R - vxR ) r R - (S L - vxL ) r L
∣ BGLM ∣
SL* = S M -
SL - v L
r L S L - SxM
∣ BGLM ∣
SR* = S M + .
SR - v R
r R S R - SxM
The total pressure inside the Riemann fan is considered to have a constant value pT*,
(s R - vxR ) r RpTL - (s L - vxL ) r LpTR - r Lr R (S R - vxR)(S L - vxL )(vxR - vxL )
pT* =
(s R - vxR ) r R - (s L - vxL ) r L
1
with pTL R = ptL R + 2 [BGLM
2
+ (ByL R )2 + (BzL R )2 ]. We can use the Rankine–Hugoniot condition for slopes S L and S R to obtain
* *
the first set of intermediate states in terms of conservative variables, namely uL and uR ,
⎧ S K - vxK ⎫
⎪ rK ⎪
SK - SM
⎪ ⎪
⎪ S K - vxK M ⎪
rK K S
⎪ S - SM ⎪
⎧ r * ⎫
K
⎪ ⎪
⎪ r K *v K * ⎪ ⎪ S K - vxK ⎛ K BGLM (S M - vxK ) ⎞ ⎪
x ⎪ ⎪ rK K v - ByK K K
M⎜ y ⎟
⎪
⎪ S -S ⎝ r (S - vxK )(S K - S M ) - BGLM 2
⎪ r K *vyK * ⎪ ⎪ ⎠ ⎪
⎪ K K ⎪ ⎪ S K
- v K
x ⎛ K B ( S M
- v K
) ⎞ ⎪
⎪ r *vz * ⎪
GLM
⎪ rK K ⎜vz - Bz
K x
⎟ ⎪
K* =
⎪ ⎪ ⎪ S - SM ⎝ r K (S K - vxK )(S K - S M ) - BGLM
2
⎠
⎪
u r K *e K * = , K = L , R. (B1)
⎨ ⎬ ⎨ (S K - v K ) r K e K - p K v K + p * S M + B ⎬
GLM (S BGLM + vy By + vz Bz - v * · B *)
M K K K K K K
K* x T x T
⎪ x B ⎪ ⎪ ⎪
⎪ K* ⎪ ⎪ SK - SM ⎪
⎪ y B ⎪ ⎪ ⎪
BGLM
⎪ B K* ⎪ ⎪ ⎪
⎪ z ⎪ ⎪ K r (S - vx ) - BGLM
K K K 2 2
⎪
⎪ y K* ⎪ ⎪ B y ⎪
⎩ ⎭ ⎪ r K (S K - vxK )(S K - S M ) - BGLM 2
⎪
⎪ K r K (S K - vxK ) 2 - BGLM2
⎪
⎪ Bz K K ⎪
⎪ r ( S - v K
x )( S K
- S M
) - B 2
GLM ⎪
⎪
⎩ yGLM ⎪
⎭
23
K*
We should note that according to this formulation, all other components of u need to be estimated before we calculate the energy
term. Also we need to avoid division by a zero situation for r K (S K - vxK )(S K - S M ) - BGLM 2
= 0 and having complex
(BGLM )2 *
magnetosonic speeds for S M = uxK , S K = uxK  cfK , ByK = BzK = 0, and gptK
 1. In these cases, we assume uK = uK and
* *
ByK * = BzK * = 0 . To estimate the conservative variables uL * and uR * related to the other two intermediate states, we again apply the
* * * *
Rankine–Hugoniot condition for slopes S L and S R with the assumption that ρ K * = ρ K , which again comes from application of the
continuity equation over the jumps with the same x-direction velocity, leading to the following formulation:
⎧ S K - vxK ⎫
rK K
⎪ S - SM ⎪
⎪ ⎪
⎪ S - vx M
K K
⎪
rK K S
⎪ S - SM ⎪
K **
⎧ r ⎫ ⎪
K - vK ⎡ v L *
r L * + v R *
r R * + ( B R *
- B L *
) sign ( B )
⎪
⎪r v ⎪** ** ⎪ S y y y y GLM ⎤⎪
⎪r SK - S M ⎢
K K K x
⎪ x
⎪ * * ⎥⎪
r + r
L R
⎪ r K **vyK **⎪ ⎪ ⎣ ⎦⎪
⎪ K ** K **⎪ ⎪ L* R*
⎪
r vz ⎪ ⎪ K S K - vx ⎡ vz r + vz
K L* r R * + (BzR * - BzL * ) sign (BGLM ) ⎤⎪
⎪
⎪ ⎪ ⎪r K ⎥⎪
uK ** = r K **e K ** = S - SM ⎢ r L* + r R*
⎨ ⎬ ⎨ ⎣ ⎦⎬, K = L, R. (B2)
K **
⎪ Bx ⎪ ⎪ E K *  r K * (v K * · B K * - v K ** · B K **) sign (BGLM ) ⎪
⎪ K ** ⎪ ⎪ ⎪
⎪ By ⎪ ⎪ BGLM ⎪
⎪ B K ** ⎪ ⎪ B R * r L * + B L * r R * + r L *r R * (v R * - v L * ) sign (BGLM ) ⎪
⎪ z ⎪ ⎪ y y y y
⎪
⎪ y K* ⎪ ⎪ r + r
L * R * ⎪
⎩ ⎭
⎪ ⎪
⎪ BzR * r L * + BzL * r R * + r L *r R * (vzR * - vzL * ) sign (BGLM ) ⎪
⎪ ⎪
⎪ r L* + r R* ⎪
⎪ y ⎪
⎩ GLM ⎭
In the energy equation, the () operator refers to the (−) operation for K = L and to the (+) operation for K = R. It should be noted
*
that the formulation of uK * leads to undefined values for BGLM = 0 because the value of sign (0) itself is undefined. For this case, we
* * * *
consider S L = S R = S M, and the intermediate states become uK * = uK , i.e., the solver in this case becomes a two-state type. When
we complete the calculation of slopes and intermediate states, we can finally calculate the numerical flux  HLLD(uL , uR ) as
⎧ f (u L ) if SL > 0
⎪ f (u ) + S L (u L * - u L )
L if SL  0  SL*
⎪
⎪ f (uL ) + S L *uL ** - (S L * - S L ) uL * - S LuL if SL*  0  S M
 HLLD(uL , uR ) = . (B3)
⎨ f (uR ) + S R *uR ** - (S R * - S R) uR * - S RuR if S M  0  SR*
⎪
⎪ f (u R ) + S R (u R * - u R ) if SR*  0  SR
⎪ f (u R ) if SR < 0
⎩
ORCID iDs Berger, M. J., & Leveque, R. J. 1989, in 9th Computational Fluid Dynamics
Conf. (Reston, VA: AIAA), AIAA 1989-1930
Somdeb Bandopadhyay https://orcid.org/0000-0002-4086-7632 Bordner, J., & Norman, M. L. 2012, arXiv:1810.01319
Hsien Shang (尚賢) https://orcid.org/0000-0001-8385-9838 Brio, M., & Wu, C. C. 1988, JCoPh, 75, 400
Bryan, G. L., Norman, M. L., O’Shea, B. W., et al. 2014, ApJS, 211, 19
Burstedde, C., Wilcox, L. C., & Ghattas, O. 2011, SIAM J. Sci. Comput.,
33, 1103
References Chacón, L. 2004, CoPhC, 163, 143
Coirier, W. J., & Powell, K. G. 1995, JCoPh, 117, 121
Adams, M., Colella, P., Graves, D. T., et al. 2016, Chombo Software Package Dai, W., & Woodward, P. R. 1994, ApJ, 436, 776
for AMR Applications–Design Document, Technical Report, LBNL-6616E, Dedner, A., Kemm, F., Króner, D., et al. 2002, JCoPh, 175, 645
(Lawrence Berkeley National Laboratory) DeZeeuw, D., & Powell, K. G. 1993, JCoPh, 104, 56
Aftosmis, M. J., Berger, M. J., & Adomavicius, G. 2000, in 38th Aerospace Dubey, A., Almgren, A., Bell, J., et al. 2014, JPDC, 74, 3217
Sciences Meeting and Exhibit (Reston, VA: AIAA), AIAA 2000-808 Dubey, A., Antypas, K., Ganapathy, M. K., et al. 2009, ParC, 35, 512
Aftosmis, M. J., Berger, M. J., & Melton, J. E. 1998, AIAAJ, 36, 952 Goodale, T., Allen, G., Lanfermann, G., et al. 2003, in High Performance
Balsara, D. S. 2004, ApJS, 151, 149 Computing for Computational Science—VECPAR 2002 (Berlin: Springer)
Batten, P., Clarke, N., Lambert, C., & Causon, D. M. 1997, SIAM J. Sci. 197
Comput., 18, 1553 Groth, C. P. T., De Zeeuw, D. L., Powell, K. G., et al. 1999, in 14th
Berger, M. J. 1986, SIAM J. Sci. Stat. Comput., 7, 904 Computational Fluid Dynamics Conf. (Reston, VA: AIAA), AIAA
Berger, M. J., & Colella, P. 1989, JCoPh, 82, 64 1999-3273
Berger, M. J., & Oliger, J. 1984, JCoPh, 53, 484 Harten, A., Lax, P. D., & van Leer, B. 1983, SIAMR, 25, 35
24
Helzel, C., Rossmanith, J. A., & Taetz, B. 2011, JCoPh, 230, 3803 Shu, C.-W., & Osher, S. 1988, JCoPh, 77, 439
Jameson, A., Schmidt, W., & Turkel, E. 1981, in 14th Fluid and Plasma Sod, G. A. 1978, JCoPh, 27, 1
Dynamics Conf. (Reston, VA: AIAA), AIAA 1981-1259 Stone, J. M., Tomida, K., White, C. J, & Felker, K. G 2020, ApJS, 249, 4
Kurganov, A., & Tadmor, E. 2000, JCoPh, 160, 241 Susanto, A. 2014, PhD thesis, Univ. Waterloo
MacNeice, P., Olson, K., Mobarry, C., de Fainchtein, R., & Packer, C. 2000, Teyssier, R. 2002, A&A, 385, 337
CoPhC, 126, 330 Toro, E. F., & Chakraborty, A. 1994, AeJ, 98, 325
McNally, C. P., Lyra, W., & Passy, J.-C. 2012, ApJS, 201, 18 Toro, E. F., Spruce, M., & Speares, W. 1994, ShWav, 4, 25
Mignone, A., Zanni, C., Tzeferacos, P., et al. 2012, ApJS, 198, 7 Toth, G. 2000, JCoPh, 161, 605
Miyoshi, T., & Kusano, K. 2005, JCoPh, 208, 315 van Albada, G. D., van Leer, B., & Roberts, W. W. 1997, A Comparative
Nessyahu, H., & Tadmor, E. 1990, JCoPh, 87, 408 Study of Computational Methods in Cosmic Gas Dynamics (Berlin:
Orszag, S. A., & Tang, C.-M. 1979, JFM, 90, 129 Springer), 95
O’Shea, B. W., Bryan, G., Bordner, J., et al. 2005, in Adaptive Mesh van Leer, B. 1974, JCoPh, 14, 361
Refinement – Theory and Applications (Berlin: Springer), 341 Van Leer, B. 1977, JCoPh, 23, 263
Powell, K. G., Roe, P. L., & Quirk, J. 1993, in Algorithmic Trends in Woodward, P. R., & Colella, P. 1984, JCoPh, 54, 115
Computational Fluid Dynamics (New York: Springer), 303 Yee, H. C., Vinokur, M., & Djomehri, M. J. 2000, JCoPh, 162, 33
Roe, P. L. 1986, AnRFM, 18, 337 Zhang, W., Almgren, A., Day, M., et al. 2016, SIAM J. Sci. Comput., 38, S156
Rossmanith, J. A. 2006, SIAM J. Sci. Comput., 28, 1766 Ziegler, U. 1999, CoPhC, 116, 65
Schnetter, E., Hawley, S. H., & Hawke, I. 2004, CQGra, 21, 1465 Ziegler, U. 2008, CoPhC, 179, 227
25

Uni Ed Astronomy Thesaurus Concepts

Uploaded by

Copyright:

Available Formats

Uni Ed Astronomy Thesaurus Concepts

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Uni Ed Astronomy Thesaurus Concepts

Uploaded by

Copyright:

Available Formats

The Astrophysical Journal Supplement Series, 263:32 (25pp), 2022 December https://doi.org/10.

SADHANA: A Doubly Linked List-based Multidimensional Adaptive Mesh Reﬁnement

1. Introduction AMR algorithm for Cartesian meshes. The technique can be

Pattern for Mother Pattern for Children

Pattern for Mother Pattern for Children

III III III I IV

Pattern for Mother Pattern for Children

II III I I XI XI VIII VIII VI

III I II II VII VII IV IV XII

IV VI V V IX IX XII XII III

VII VIII IX IX III III XI XI V

VIII IX VII VII XII XII VI VI II

IX VII VIII VIII IV IV I I X

XI X XII XII II II V V VII

XII XI X X VIII VIII III III IV

(fi + 1, j, k - 2fi, j, k + fi - 1, j, k )2 + (fi, j + 1, k - 2fi, j, k + fi, j - 1, k )2 + (fi, j, k+ 1 - 2fi, j, k + fi, j, k- 1)2

x¯ic+ 1 - x¯ic x¯ic - x¯ic- 1 ⎟

Figure 5. 1D HD Sod result at t = 0.2.

2. Afterward, a physics-speciﬁc Riemann solver is applied

In this work, we have used HLLC (as discussed in

each time integration to project the solution available Dt ˜ (n) (n )

the (n + 1) th time step such that we obtain Dt ˜ (n)

Figure 6. 1D HD Shu–Osher result at t = 1.8.

4.1.2. 1D Shu–Osher Shock Tube 4.1.3. 2D Isentropic Vortex

Figure 10. 1D MHD Brio–Wu result at t = 0.12.

Figure 11. 1D MHD Shu–Osher–Susanto result at t = 0.7.

We have used x0 = 0, y0 = 0 as the initial location of the 4.1.4. 2D Kelvin–Helmholtz Instability

Figure 7. lsin ⎛ , 0, 2.5⎫ (17)

SL = min [min (vxL - c L , vxR - c R) , 0]

S L = min [min (vxL , vxR) - max (cfL , cfR) , 0]

You might also like