The Implementation of An Implicit Coupled Density based

Solver based on OpenFOAM

Shuai Ye WenJing Yang* XinHai Xu
College of Computer, National College of Computer, National College of Computer, National
University of Defense Technology. University of Defense Technology. University of Defense Technology.
State Key Laboratory of High State Key Laboratory of High State Key Laboratory of High
Performance Computing. Performance Computing. Performance Computing.
Changsha, Hunan, China Changsha, Hunan, China Changsha, Hunan, China
[email protected] [email protected] [email protected]
Hao Li Chao Huang
College of Computer, National State Key Laboratory of High
University of Defense Technology. Performance Computing.
State Key Laboratory of High Changsha, Hunan, China
Performance Computing. [email protected]
Changsha, Hunan, China
[email protected]

ABSTRACT flexible engineering simulation in CFD. Users can develop their

In this paper, we develop an implicit coupled density based solver solvers based on OpenFOAM and extend it [6]–[8]. OpenFOAM
called coupledFoam in the framework of OpenFOAM. The solver is very friendly to users and easy to use since it supports DSL
uses the Steger-Warming flux vector splitting to achieve an (Domain Specific Language). More and more functions are
implicit discretization. BoomerAMG precondition and GMRES supported by OpenFOAM after decays of development.
iterative method are applied to solve the coupled equation. Data Explicit and implicit scheme are two types of schemes that are
structure has been designed to store the coupled matrix. To usually used in time discretization in CFD. Due to the simple
support the linear solver in Hypre, data interface is designed. The operation in explicit scheme, the computational cost of each time
details of the implementation are presented in this paper. The step is small. The implicit scheme has a larger computational
steady-state oblique shock test case and unsteady-state forward overhead since it need to solve a linear system each time step.
step test case are tested. The flow field characteristics of However, since the explicit scheme uses only the prior flow field
coupledFoam are smoother than that of rhoCentralFoam. information, and the implicit scheme uses the current flow field
information, the solution obtained in implicit scheme is generally
CCS Concepts more stable. Seokkwan yoon and Chen,R.F use an implicit LU-
•Computing methodologies →Modeling and simulation SGS scheme early in reference [9], [10]. The results in their
papers shows a smaller residual compared with the explicit
Keywords scheme.
OpenFOAM; coupled solver; flux vector splitting; Hypre.
It is called a coupled equations if the equations are coupling with
1. INTRODUCTION each other. The variables in one of the equations are coupling with
Computational Fluid Dynamics (CFD) is widely used to model other variables. Under this circumstances, the dependent
fluid flow and optimize industrial processes [1]. It is a branch of equations can be combined to form one coupled linear system.
fluid mechanics using numerical simulation to solve and analyze The variables in the coupled equation are solved simultaneously.
problems that involve fluid flows [2], [3]. It is an important
application of information technology using numerical simulation The official version of OpenFOAM has developed a density-based
with computer. There are a variety of CFD software, including explicit compressible solver called rhoCentralFoam, which uses a
some famous commercial software such as Fluent, CFX, Star-cd, central-upwind schemes of Kruganov and Tadmor to capture the
etc [4], and some open source software [5]. OpenFOAM is a shocks in the simulations. Reference [11], [12] have developed an
widely used framework of the open source C++ CFD toolbox for implicit LU-SGS density-based solver in the official version of
OpenFOAM, which does not yet support solving equations in
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are coupled form. The DensityBasedTurbo solver provided by the
not made or distributed for profit or commercial advantage and that extend version of OpenFOAM supports more flux schemes such
copies bear this notice and the full citation on the first page. Copyrights as Roe, AUSM+, etc [13]. Although the coupled equations are
for components of this work owned by others than ACM must be supported in the extend version, there are no coupled density-
honored. Abstracting with credit is permitted. To copy otherwise, or based implicit solvers yet.
republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from In this paper, we develop a coupled implicit density-based solver
[email protected]. in the framework of the official OpenFOAM. The equations are
ICCIS 2017, November 7-9, 2017, Wuhan, China discretized into a coupled linear system in accordance with the
© 2017 Association for Computing Machinery. coupled algorithm, and variables of the equations are solved
ACM ISBN 978-1-4503-5348-9/17/11…$15.00 simultaneously. BoomerAMG is used as the precondition, and
DOI: https://doi.org/10.1145/ 3158233.3159341

121
GMRES is used as the iterative method to solve this coupled Here, Tadmor is the flux scheme. In rhoCentralFoam, the three
linear system. The library of Hypre is called in the solving variables, i.e. rho, rhoU, rhoE, are discretized separately to obtain
procedure. The main contributions of this paper are: three linear systems. Each linear system are independent of others,
these linear systems can be solved separately.
 The CSR data structure is designed in OpenFOAM to
store the coupled matrix, the interface between 2.2 Implicit coupled discretization
OpenFOAM and Hypre is designed and implemented to Considering the following two-dimensional Euler equations:
support the calling of BoomerAMG and GMRES in
Hypre.  ρ  ρu ρv 
 ρu   ρu 2  p ρuv 

 t    div 
A coupled implicit density-based solver, coupledFoam, is
0. (1)
implemented in the framework of OpenFOAM. The  ρv   ρuv ρv 2  p 
Steger-Warming flux vector splitting method is used to    
capture shocks in discontinuous problems. E  u( E  p ) v( E  p ) 
Where ρ is the density, u, v are the velocity components in x and y
 Experiments are designed to verify and validate directions, p is the pressure, and E is the total energy. The first
coupledFoam. The steady-state oblique shock and term in the equations is called transient term, and the second is
unsteady-state forward step are tested. Compared with called convective term. After changing Euler equations into a
rhoCentralFoam, the flow field characteristics of conservation equation of the form, we will have:
coupledFoam is smoother. Specially, the residual of
coupledFoam in oblique shock case is smaller than t u( x, t )  divF ( u( x, t ))  0 . (2)
rhoCentralFoam.
In the process of the numerical discretization, the computational
The rest of the paper is organized as follows: Section 2 introduces domain Ω is partitioned into multiple computational domains  j .
some related work and background, including the procedure of a
standard solver in OpenFOAM and implicit coupled discretization After discretization in each domain, we obtain the following
method. Section 3 introduces the design and implementation of formulation:
coupledFoa, followed by the experiments in section 4, which
 Ωi Rin 
shows the comparison of the results. Finally, Section 5 concludes
 I   Ui   Ri .
n n
(3)
the paper.  t U i 

2. OPENFOAM INTRODUCTION AND n

Where Ri is the residual of the variables in cell i in the n-th time
NUMERICAL METHOD Rin
is the flux Jacobian, I is the identity matrix, Ui is
n
step,
2.1 Introduction to OpenFOAM U i
The solver in OpenFOAM generally includes three procedures,
the difference of the variables in cell i, it can be represented by
that is, mathematical modeling, equations discretization and linear
another form:
system solving. The user firstly need to establish the mathematical
model of the problem, e.g Euler equations, Navier-Stokes Uin  Uin 1  Uin . (4)
equations etc. Relative data structure such as volScalarField need
to be created by user. Then the user need to write the code After applying the Gauss’ law, equation (3) is written as follows:
combined with the model using DSL, which must meet the syntax
Uin 1  Uin
of OpenFOAM. The procedures of discretization and solving after Ωi   Ac ,ij Uijn Sij   Rin . (5)
this procedure is carried out by the kernel of OpenFOAM. ti jN ( i )
OpenFOAM uses finite volume method to discrete the terms in Where j is the label of the cells shared face with cell i. To support
the equations. Fvm and fvc in the code respectively represent the capturing of shock in nonlinear system, Steger-Warming split
implicit discretization and explicit discretization. The matrix flux is used. The split flux is represented as follows:
coefficient A and the RHS of the linear system b are returned
when fvm is used, and field is returned when fvc is used. After all Ac ,ij Uijn Sij  ( Ac,ij Uin  Ac,ij U nj ) Sij . (6)
the terms are discretized, the overload operator will return a linear
system. The linear system is solved in OpenFOAM with its library. 
In the equation, Ac ,ij represents the flux Jacobian on the face
The main procedure of rhoCentralFoam is shown in fig.1
shared by cell i and cell j. This formula shows that the flux on the
shared face can be split into the sum of flux flowing from cell i

and cell j. In the Euler equation, Ac ,ij is calculated by the
following formula:

1 1
A  Ac  (| V | c ) I . (7)
2 2
Where V is the velocity flux flowing through Sij , |V|+c is the
spectral radius of the convective flux Jacobian. For the details, we
refer the readers to the reference [13]. To obtain Ac , which is the
Figure 1: The procedure of rhoCentralFoam
convective flux Jacobian, we refer the reader to reference [14].

122
Put all the cells together, we can get the following equation at n-th When the two-dimensional Euler equation is solved, the number
time step: of unknown variables is 4. According to equation (6), the variable
Uin of every cell is a vector of length 4，and Jacobian matrix A
AU   Rn . (8)
is a 4×4 matrix. Thus, the discrete matrix A in coupledFoam is a
It is important to note that the A here is a large matrix of all the
sparse block matrix, which need to be stored in other data
computing cells, and U is the vector of all unknown variables.
structures.
That is, all the equations are discretized into a coupled matrix, and
all variables are solved in a coupled linear system. Fig.4 is a structured mesh with a simple cell number of 4. Fig.5
shows the data structure of the matrix A under the grid and the
3. THE DESIGN AND IMPLEMENTATION data structure of the block matrix . Block coefficients
OF COUPLEDFOAM corresponding to the cell 1 will be located from the fourth row to
3.1 The procedure of coupledFoam seventh row in the matrix A. The corresponding block coordinates
for are (1, 0).
The procedure of coupledFoam is different from that of the
standard solver in OpenFOAM. As is shown in fig.2. In
coupledFoam, the discretization of the DSL language and the 2 3
solving of the linear system are replaced by the procedure of
0 1
coupled discretization. In addition, there is an assembly of the
coupled equations and a matrix format conversion process before
Figure 4: A simple mesh with four cells
solving the linear system in coupledFoam. CoupledFoam uses
Hypre’s library to solve the linear system instead of using the 0 1 2 3
OpenFOAM’s library.
0 A00 A01 A02

Mathematical modeling x00 x01 x02 x03

Mathematical modeling 1 A10 A11 A13
A10 = x1 0 x1 1 x1 2 x1 3
x2 0 x2 1 x2 2 x2 3
2 A20 A22 A23 x30 x31 x32 x33
Coupled discretizations
3 A31 A32 A33
Equations discretizations

Coupled matrix assemble

A10 rows=4 cols=4

Linear system solving Hypre lib row 4 5 6 7

col 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3
rhoCentralFoam coupledFoam

val x00 x0 1 x02 x03 x10 x1 1 x12 x13 x20 x21 x22 x23 x30 x31 x3 2 x33
Figure 2: The difference of procedures between
rhoCentralFoam and coupledFoam Figure 5: The data structure of the coupled matrix
When the matrix of Hypre is set, the matrix insert operation is
3.2 Data structure in coupledFoam performed using the data structure mentioned above and the
In OpenFOAM, the cell is numbered sequentially, the following interface.
discretization of each cell is only related to its adjacent cells, so
the obtained matrix is usually sparse. Compressed Sparse Row HY PRE IJM atrixSetValues(A,rownum,
format (CSR) is usually used to store such a sparse matrix. CSR
can save storage space since it stores only nonzero elements in the &ncol, &rows, &cols, &vals);
matrix.
Where rownum represents the number of rows to insert, ncol
Specifically, the CSR structure has three arrays of row, col, and represents the number of elements to be inserted, and rows
val, where row represents the row number of the matrix, col represents the corresponding row number. Cols represents the
represents the column number, and val represents the column number of the corresponding element, and vals indicates
corresponding element value in the matrix. Fig.3 is an example of the value of the element to insert. For specific use of related
a simple CSR structure. Hypre functions, see [15].
0 1 2 3
3.3 Coupled algorithm
0 1 2 0 0 According to formula (6), the flux on each surface of can be
row 0 1 2 3 divided into two parts, and . Such that the equation (5)
1 0 3 0 4
col 0 1 1 3 2 3 3
can be written as:
2 0 0 5 6
i 1
3 0 0 0 7
val 1 2 3 4 5 6 7 [  (| V | c)]I U i 
ti 2

j  neighbor
Ac, j Sij U j
(9)
Figure 3: The data structure CSR of an example sparse  
j  owner
Ac, j Sij U j  Ri
matrix

123
In OpenFOAM, each surface on the grid unit is shared by two cell,
which are labeled as the owner and neighbor of the face. The
owner of a face is always the smaller label in the two shared cell,
and the neighbor is the other larger label. The symbols of Sij are
different in owner and neighbor, thus it is important to pay
attention to the symbols in the formula. In formula (9), the Figure 7: The grid of forward step
coefficient of U i will be located in the matrix A with block id of We use the Euler equation introduced in section 2.2 as the
 mathematical solution Model.
(i, i). The coefficient block Ac , j will be located in A with block
id of (i, j). The coupled algorithm is shown in algorithm 1. 4.2 Methodology
To illustrate the correctness of coupledFoam, we firstly simulate
the oblique shock test case and forward step with rhoCentralFoam,
which is the official solver in OpenFOAM. The results of these
Algorithm 1: Procedure of coupled matrix discretization and cases using rhoCentralFoam are compared with the results using
assemble coupledFoam. The shock positions in these results should be
consistent. In our experiments, we focus on the convergence rate
1: for all patchi mesh.boundary() do in these two cases. We define the relative L2 normal of residual
2: pFaceCells= mesh.boundary()[patchi].faceCells(); Res in time step n as follows:
3: for all facei  mesh.boundary()[patchi] do

4: calculate Ac , o ; ρin  ρin 1 2

N
( ) /N
5: ownerLabel=pFaceCells[facei];
Res n 
i 0
t

6: rhs[4*ownerLabel] -= Ac U ; Res 0
7: end for All the courant number is set 0.5 in these cases.
8: end for 4.3 Experimental Results
9: for all facei  internal faces do
10: label ownerLabel = faceOwner()[facei]; 4.3.1 Oblique shock
11: label neighbour = faceNeighbor()[facei]; The contour lines of the ρ field is shown in Fig.8. Notice the
 positions of the shocks are consistent. Also notice that the result
12: calculate Ac , o ; of coupledFoam is more stable than rhoCentralFoam. The
13: set block id (o,n); consistent in these results show the correctness of coupledFoam.
The convergence histories of residual in terms of iteration number
14: set A[offset(o,n)] += Ac, o .
is shown in Fig.9. The number of iterations here is the number of

15: calculate Ac , n ; steps in time marching. It can be seen that coupledFoam can
converge to smaller residuals than rhoCentralFoam.
16: set block id (n,o);
17: set A[offset(n,o)] -= Ac, n .
18: end for

4. EXPERIMENT
(a) The result of rhoCentralFoam
4.1 Experiment set up
The operating system of the computer we use in this experiment is
Red Hat Enterprise Linux Server 6.5, with OpenFOAM version
4.0 installed on it. In this experiment, we have tested two cases.

(b) The result of coupledFoam

Figure 8: The comparison of contour lines of ρ in oblique
shock case

Figure 6: The grid used in the oblique shock test case

The first case called oblique shock is a tutorial case of
rhoCentralFoam solver provided in OpenFOAM. This case
simulates two flows flowing from the top and left of a field in an
angle. Shocks will appear and stay steady in the field in this case.
We use a structured grid in this test, as is shown in fig. 6.
The second test case is the forward step shock test, a transient
problem that is usually used to test the ability of numerical
methods in capturing shock waves. The forward step test is also a
test case for rhoCentralFoam in OpenFOAM. Figure 7 is the grid
used for the test.
Figure 9: The residuals of oblique shock with different solvers

124
4.3.2 Forward step 6. ACKNOWLEDGMENTS
The contour lines of the flow’s density at specific time are shown The authors would like to thank the Science Challenge Project
in fig.10. Both these two solvers are able to capture shock wave in (No.TZ2016002), the National Key Research and Development
the flow field. These results also verify the correctness of the Program of China (No. 2016YFB0201301).
coupledFoam. By looking closely at the contours of density, it is
found that the contour lines obtained by rhoCentralFoam 7. REFERENCES
[1] M. Safinowski, M. Szudarek, R. Szewczyk, and W.
t=0.5s t=0.5s Winiarski, Capabili- ties of an Open-Source Software, Elmer
FEM, in Finite Element Analysis of Fluid Flow. Cham:
Springer International Publishing, 2017, pp. 118– 126.
t=1s t=1s [2] Blazek J. Computational fluid dynamics: principles and
applications[M]. Butterworth-Heinemann, 2015.
[3] Biswas G. Computational fluid dynamics[M]. Alpha Science
t=2s t=2s Intl Ltd, 2013.
[4] Jeong W, Seong J. Comparison of effects on technical
variances of computational fluid dynamics (CFD) software
t=4s based on finite element and finite volume methods[J].
t=4s
International Journal of Mechanical Sciences, 2014, 78: 19-
26..
[5] Moukalled F, Mangani L, Darwish M. The finite volume
rhoCentralFoam coupledFoam method in computational fluid dynamics[J]. 2016.
Figure 10. The comparison of contour lines of density in [6] H. Jasak, A. Jemcov, Z. Tukovic et al., “Openfoam: A c++
forward step in different time. library for complex physics simulations,” in International
workshop on coupled methods in numerical dynamics, vol.
1000. IUC Dubrovnik, Croatia, 2007, pp. 1–20.
[7] Q. Wang, X.-G. Ren, X.-H. Xu, C. Li, H.-Y. Ji, and X.-J.
Yang, “Coupling strategies investigation of hybrid atomistic-
continuum method based on state variable coupling,”
Advances in Materials Science and Engineering, vol. 2017,
2017.
[8] T. T. Zhang, W. J. Yang, Y. F. Lin, Y. Cao, M. Wang, Q.
Wang, and Y. X. Wei, “Numerical study on flow rate
limitation of open capillary channel flow through a wedge,”
Advances in Mechanical Engineering, vol. 8, no. 4, 2016.
[9] S. Yoon and A. Jameson, “Lower-upper symmetric-gauss-
seidel method for the euler and navier-stokes equations,”
Figure 11. The residuals of forward step with
Aiaa Journal, vol. 26, no. 9, p. 1025, 1988.
rhoCentralFoam and coupledFoam.
[10] R. F. Chen and Z. J. Wang, “Fast, Block Lower-Upper
contain more oscillations, while the results obtained by
Symmetric Gauss-Seidel Scheme Introduction,” 2000.
coupledFoam are smoother. The convergence histories of residual
in terms of iteration number is shown in Fig.11. The residual [11] C. Shen, X. L. Xia, Y. Z. Wang, F. Yu, and Z. W. Jiao,
convergence of coupledFoam is almost consistent with that of “Implementation of density-based implicit lu-sgs solver in
rhoCentralFoam. the framework of openfoam,” Advances in Engineering
Software, vol. 91, pp. 80–88, 2016.
5. CONCLUSION [12] J. Kim and K. Kim, “A development and verification of
We have developed a coupled implicit density-based solver, density based solver using lu-sgs algorithm in openfoam,”
coupledFoam, in the framework of OpenFOAM. The correctness 2014.
of coupledFoam is verified by the both the steady-state oblique
shock case and the unsteady-state forward step case. The [13] C. Shen, F. Sun, and X. Xia, “Analysis on capabilities of
comparisons of the results obtained by coupledFoam and density-based solvers within openfoam to distinguish
rhoCentralFoam respectively show that, coupledFoam can obtain aerothermal variables in difusion boundary layer,” Chinese
a smoother flow field. Especially in the oblique shock case, Journal of Aeronautics, vol. 26, no. 6, pp. 1370–1379, 2013.
coupledFoam can converge to lower residuals. This paper has [14] A. Rohde, “Eigenvalues and eigenvectors of the euler
some limitations since coupledFoam has a high time and space equations in general geometries,” Aiaa Journal, 2001.
overhead, we will focus on the memory and performance
optimization in the future. [15] R. Falgout and U. Yang, “Hypre user’s manual,” Revision
2.8. Technical Report, Lawrence Livermore National
Laboratory. Available at: http://www. llnl. gov/CASC/hypre,
Tech. Rep., 2011.

125