Computers and Fluids 241 (2022) 105481

Contents lists available at ScienceDirect

Computers and Fluids

journal homepage: www.elsevier.com/locate/compfluid

Model order reduction method based on (r)POD-ANNs for parameterized

time-dependent partial differential equations
Fangxiong Cheng a , Hui Xu b ,∗, Xinlong Feng a
a College of Mathematics and System Sciences, Xinjiang University, Urumqi 830046, PR China
b
School of Aeronautics and Astronautics, Shanghai Jiao Tong University, Shanghai 200240, PR China

ARTICLE INFO ABSTRACT

Keywords: In this paper, we propose a non-intrusive data-driven model order reduction method, also known as (r)POD-
Non-intrusive ANNs model order reduction method. In this method, (r)POD priori dimension reduction is performed on
Data-driven model order reduction method high-fidelity data set, and then the mapping relationship between parameter space and generalized coordinates
High-fidelity data
of (r)POD is implicitly constructed by using the universal approximation property of neural network. Through
Universal approximation property
pre-training on small batch data sets, updating neural network parameters every several time steps, and
Neural network
combination of L-BFGS optimization algorithm and LM optimization algorithm, time cost of the reduced order
model in the off-line calculation stage is reduced. This makes the (r)POD-ANNs model order reduction method
suitable for high-fidelity models with larger spatial degrees of freedom and higher complexity. Finally, we
verify effectiveness of the proposed method by comparing with the data-driven model reduction method, and
then apply it to the Allen-Cahn equations with strong nonlinearity and cylinder flow problem with large spatial
degrees of freedom.

1. Introduction problem described by HF model. For traditional MOR methods, such as

reduced basis (RB) method, because it is based on linear superposition
In recent years, numerical simulation of parameterized of reduced basis, it has certain limitations in dealing with nonlinear
time-dependent partial differential equations (PDEs) has become more parameterized time dependent PDEs. The method of constructing re-
and more important in engineering and applied science, due to achieve- duced basis is usually the proper orthogonal decomposition (POD) and
ments in numerical analysis and scientific computation. In the field of the greedy algorithm [2]. Therefore, for intrusive ROM methods such as
aerospace engineering, mechanical engineering and life science, numer- POD-Galerkin projection method, there are problems in instability and
ical simulations of solving PDEs provide a virtual experimental testing nonlinear low efficiency [3–6]. For some non-intrusive ROM methods,
platform. For some complex systems, it can be solved by some high- POD is mainly combined with interpolation methods such as RBF,
fidelity numerical techniques, such as finite element method (FEM), DEIM, etc. [7–9], and effect of order reduction is mainly determined
finite difference method (FDM) and finite volume method (FVM). by interpolation function and sample data.
However, the high-fidelity (HF) numerical simulations of a complex In recent years, due to strong function fitting ability of neural net-
system require a lot of computing time and storage space, especially
work, it can fit almost any function theoretically. There exist many ex-
when the problem solved in real time and multi-queries are required.
amples of using neural networks to solve forward and inverse problems
The difficulties confronted in traditional numerical methods greatly
of PDE systems, such as physics-informed neural networks (PINN) [10],
promote development of model order reduction (MOR).
and its generalizations conservative physics-informed neural networks
The MOR method provides possibility of efficiently implementing
(CPINN) and extended physics-informed neural networks (XPINN) [11,
numerical simulation of complex systems [1], especially when multiple
12]. In [13], deep neural network is used to solve PDEs of complex
computations are required for different parameters, which are related
geometric region. Deep learning is used to determine the basis function
to fluid properties, source terms, geometry, initial and boundary con-
ditions, etc. The MOR method reduces the computational complexity and the corresponding weight of meshless configuration method to
through a simplified low-dimensional system, and controls the accuracy solve PDEs [14]. PINN is successfully applied to turbulence prediction
loss while reducing the computational burden, thus saving the cost of and MOR for nonlinear steady state problems [15,16]. There are other
numerical simulation and expressing the physical characteristics of the examples of learning governing equations through data [17–21]. With

∗ Corresponding author.
E-mail addresses: [email protected] (F. Cheng), [email protected] (H. Xu), [email protected] (X. Feng).

https://doi.org/10.1016/j.compfluid.2022.105481
Received 28 December 2021; Received in revised form 8 March 2022; Accepted 27 April 2022
Available online 4 May 2022
0045-7930/© 2022 Elsevier Ltd. All rights reserved.
F. Cheng et al. Computers and Fluids 241 (2022) 105481

excellent performance of machine learning (ML) in data mining, it system dynamics. The input parameters 𝐮(𝑡) ∈  represent the relevant
is possible to construct non-intrusive MOR methods using machine geometric or physical parameters of the dynamic system, where the
learning methods. parameter space  ∈ R𝑁𝑢 is a bounded closed set. And 𝐲(𝑡) ∈ R𝑁𝑦
At present, there are some successful cases combining data-driven represents the relevant output information of the dynamical system that
modeling with POD, mainly through POD to extract a set of roughly we are interested in or want to know about.
simplified basis functions from high-fidelity solutions, and use neural The dimension of full-order state variable 𝐒(𝑡) is closely related to
network to recover the coefficients of the basis functions [22–26]. finite element space discretization. When the equation of state variables
In [27], POD is used to generate basis function of flow field, coef- is required to describe more complex physical dynamics system, or HF
ficient of the basis function is taken as low-dimensional feature, a model solution is required to have high accuracy, then 𝑁ℎ may be very
time-convolution neural network is used to build the low-dimensional large. The solution of (1) will require a lot of time and storage space.
feature model, and the feed-forward neural network is used to build Especially when the HF solution for different parameters needs to be
the low-dimensional feature model [28]. In [29], the PDEs space is solved many times, the time and storage space spent will be unbearable
completely discretized to obtain the first-order equations of time, and to the computer. This forces us to find a method that can quickly solve
ANN is used to construct the mapping relationship between input the high-precision approximate solution of (1). We expect it to have the
and output, and this method is applied to the efficient simulation of following form
cardiac electromechanics [30]. In [31], the DL-ROM framework can {
̇ = 𝐟(𝐬(𝑡), 𝐮(𝑡)), 𝑡 ∈ (0, 𝑇 ],
𝐬(𝑡)
solve parameterized time-dependent PDEs and is applied to the reduced (2)
𝐬(0) = 𝐬0
order model of cardiac electrophysiology [32]. By further extension, the
POD-DL-ROM method can solve large-scale high-precision systems and ̃
𝐲(𝑡) = 𝐠(𝐬(𝑡)), 𝑡 ∈ (0, 𝑇 ],
be extended to vector equations [33,34].
where 𝐬(𝑡) ∈ R𝑛 (𝑛 ≪ 𝑁ℎ ) represents the state variable of the MOR
Data-driven modeling has been popular in the field of computational
system, which refers to reduced state variable. And 𝐟 ∶ R𝑛 × R𝑁𝑢 → R𝑛
fluid dynamics (CFD) to obtain reduced order models of complex
and 𝐠 ∶ R𝑛 → R𝑁𝑦 are easier to calculate than 𝐅 and 𝐆. And ̃ 𝐲(𝑡) is a
fluid dynamics, such as unsteady aerodynamics modeling. In [35], the
high precision approximation of 𝐲(𝑡), that is 𝐲(𝑡) ≈ ̃
𝐲(𝑡).
author introduces three typical data-driven methods (including system
When the dynamic system model is known, we can obtain its
identification, feature extraction and data fusion) and methods to im-
simplified form (2) from the HF model, which we call the model-
prove the accuracy, stability and generalization ability of data-driven
based MOR method. In [29], a model-based MOR method, namely
models. In [36], low order linear model representation of aerodynamics
(Petrov)-Galerkin projection method, is considered. That is finding an
system is provided based on system identification method. In [37],
approximate solution to (1) in the subspace Col(𝐕) spanned by the
a reduced-order modeling method for aerodynamics optimization is
orthogonal basis for the 𝑁 columns of the orthogonal matrix 𝐕𝑁 ∈
proposed, and principal component analysis (PCA) is used to reduce the
R𝑁ℎ ×𝑁 , i.e.
number of modes so as to improve optimization efficiency. A layered {
reduced-order modeling approach for nonlinear unsteady aerodynam- ̇ = 𝐖T 𝐅(𝐕𝐬(𝑡), 𝐮(𝑡)), 𝑡 ∈ (0, 𝑇 ]
𝐬(𝑡)
(3)
ics comprising both linear and nonlinear characteristics is proposed 𝐬(0) = 𝐖T 𝐒0 ,
in [38].
̃
𝐲𝑅𝐵 (𝑡) = 𝐆(𝐕𝐬(𝑡)), 𝑡 ∈ (0, 𝑇 ].
In this paper, by spatial discretization of parameterized time-
dependent PDEs, a set of first-order differential equations about time, Where the choice of 𝐖 and 𝐕 corresponds to different projection meth-
also known as the equation of state variables, is obtained. By reducing ods. In this work, we consider the POD-Galerkin projection method,
dimension of state variables, a low order system about state variables which satisfies 𝐖 = 𝐕.
is obtained. The mapping relationship between input and output is When the model is unknown, we assume that it can be expressed
directly constructed [29,39], and a low-order system is obtained to as a full-order state variable of the form (1), and its HF solution
replace the original high-order model. In [33], the author considers can be obtained by some measurement means. If we want to get a
the prior dimension reduction based on POD on basis of the DL-ROM simplified expression of the form (2), we can only use data-driven
method. Therefore, before considering the data-driven MOR method for MOR method. In contrast, data-driven MOR method are more general.
state variables, POD prior dimension reduction is carried out for system For the data-driven MOR method, we mainly consider the nonlinear
output data, which may accelerate the process of data-driven MOR in system identification method mentioned in [35,40], which uses the
the offline stage. flexibility of neural network to realize a dynamic nonlinear reduced
The outline of the paper is given as follows. In Section 2, we detail state space model and reconstruct its internal state through HF input
the construction of the proposed (r)POD-ANN model order reduction and output data. We introduce neural network functions into (2), that
method. We consider feasibility of the (r)POD-ANNs model order reduc- is, continuous functions 𝐟 and 𝐠 are represented by ANN respectively,
tion method from the aspect of numerical calculation and optimization so it has the following form
methods in Section 3. Its reliability is evaluated numerically in Sec- {
̇ = 𝐟𝐴𝑁𝑁 (𝐬(𝑡), 𝐮(𝑡); 𝝁), 𝑡 ∈ (0, 𝑇 ]
𝐬(𝑡)
tion 4 by considering four different test cases. Finally, the work of this (4)
𝐬(0) = 𝐬0 ,
paper is summarized in Section 5.
̃
𝐲(𝑡) = 𝐠𝐴𝑁𝑁 (𝐬(𝑡); 𝝂), 𝑡 ∈ (0, 𝑇 ],
2. Model order reduction strategy
where 𝝁, 𝝂 represents the weight and bias information of the networks
respectively. This data-driven MOR method implicitly constructs a
For generality, we consider a class of parameterized time-dependent
nonlinear mapping of 𝐮(𝑡) → 𝐲(𝑡), where the state variable 𝐬(𝑡) acts as
PDEs. No matter what kind of finite element space discretization is
an auxiliary variable that drives the evolution of the model over time.
used, it can be expressed as a class of first-order differential equations,
For the parameterized time-dependent HF model with the form
namely, the following full-order state variable equations
(1), if it has a low order approximation model of the form (2), then
{
̇ = 𝐅(𝐒(𝑡), 𝐮(𝑡)), 𝑡 ∈ (0, 𝑇 ],
𝐒(𝑡) its reduced order model has obvious advantages in computing. How-
(1)
𝐒(0) = 𝐒0 ever, when HF model needs high-precision solution, the huge HF
data will bring computational pressure to optimize neural network
𝐲(𝑡) = 𝐆(𝐒(𝑡)), 𝑡 ∈ (0, 𝑇 ], parameters. Therefore, here we consider a (r)POD-ANNs model order
where 𝐒(𝑡) ∈ R𝑁ℎ is the state variable of HF model, with 𝐅 ∶ R𝑁ℎ × reduction method, which can also be regarded as a two-step dimension-
R𝑁𝑢 → R𝑁ℎ and 𝐆 ∶ R𝑁ℎ → R𝑁𝑦 are (nonlinear) function, encoding the ality reduction method for the HF model spatio-temporal separation.

2
F. Cheng et al. Computers and Fluids 241 (2022) 105481

Firstly, (r)POD prior dimension reduction is performed on HF data set. parameter sets and generalized coordinates of (r)POD. Therefore, the
Then, the ANNs is used to construct the mapping relationship between reduced state variable equation has the following form
parameter space and (r)POD generalized coordinates. {
̇ = 𝐟𝑛 (𝐬(𝑡), 𝐮(𝑡); 𝜽), 𝑡 ∈ (0, 𝑇 ],
𝐬(𝑡)
(6)
𝐬(0) = 𝐬0 ,
2.1. (r)POD-ANNs model order reduction method
̃
𝐲(𝑡) = 𝐕𝑁 𝐠𝑛 (𝐬(𝑡); 𝝑), 𝑡 ∈ (0, 𝑇 ]
Firstly, for parameter space  ⊆ R𝑁𝑢 , take parameter set 𝛩 = where 𝜽 and 𝝑 are the weight and bias of continuous neural networks
{𝝁 , 𝝁2 , … , 𝝁𝑁𝑡𝑟𝑎𝑖𝑛 } and time interval 0 = 𝑡1 < 𝑡2 < ⋯ < 𝑡𝑁𝑡 = 𝑇 .
1
functions 𝐟𝑛 and 𝐠𝑛 respectively. Where the reduced state variable
𝐲𝑗𝑘 = 𝐲(𝑢𝑗 , 𝑡𝑘 ) is denoted as HF solution of the model at 𝑢𝑗 (𝑡𝑘 ), 𝑗 = 𝐬(𝑡) ∈ R𝑛 , the neural network output 𝐠𝑛 (𝐬(𝑡); 𝝑) ∈ R𝑁 represents
1, 2, … , 𝑁𝑡𝑟𝑎𝑖𝑛 , 𝑘 = 0, 1, … , 𝑁𝑡 and 𝐲𝑢𝑗 = {𝐲(𝑢𝑗 , 𝑡1 ), 𝐲(𝑢𝑗 , 𝑡2 ), … , 𝐲(𝑢𝑗 , 𝑡𝑁𝑡 )} the approximation of the (r)POD generalized coordinates of the HF
is used to represent the solution set of parameter 𝑢𝑗 (𝑡), then the solution solution (where 𝑛 < 𝑁 ≪ 𝑁ℎ ). Therefore, ̃ 𝐲(𝑡) = 𝐕𝑁 𝐠𝑛 (𝐬(𝑡); 𝝑) gives
set 𝐲𝛩 = {𝐲𝑢1 , 𝐲𝑢2 , … , 𝐲𝑢𝑁 } is the set of corresponding solutions of all an approximation of the HF solution, which we have 𝐲(𝑡) ≃ ̃ 𝐲(𝑡) =
𝑡𝑟𝑎𝑖𝑛
parameter sets, so the snapshot matrix 𝐘 is formed by 𝐕𝑁 𝐠𝑛 (𝐬(𝑡); 𝝑).
Because of the universal approximation property of ANNs, where
𝐘 = [𝐲𝑢1 |𝐲𝑢2 | ⋯ |𝐲𝑢𝑁 ] ∈ 𝐑𝑁ℎ ×𝑁𝑡𝑟𝑎𝑖𝑛 𝑁𝑡 .
𝑠 the hypothesis space to which the continuous function 𝐟𝑛 belongs is
By singular value decomposition (SVD) of matrix 𝐘, we get 𝐘 = ̂ ∶=  0 (R𝑛 × R𝑁𝑢 ; R𝑛 ), that is 𝐟𝑛 ∈ 
 ̂ . And again, we have 𝐠𝑛 ∈ ̂ ∶=
𝐕Σ𝐙, where 𝐕 ∈ R𝑁ℎ ×𝑁ℎ and 𝐙 ∈ R𝑁𝑡𝑟𝑎𝑖𝑛 𝑁𝑡 ×𝑁𝑡𝑟𝑎𝑖𝑛 𝑁𝑡 are orthogonal  0 (R𝑛 ; R𝑁 ) and 𝐬(0) ∈ 𝐒𝑛 ≡ R𝑛 . According to (6), we can know that for
matrices, their columns are left singular vectors and right singular neural network function 𝐟𝑛 , parameter 𝐮(𝑡) and reduced state variable
vectors respectively. And where Σ = diag(𝜎1 , 𝜎2 , … , 𝜎𝑘 ) ∈ R𝑁ℎ ×𝑁𝑡𝑟𝑎𝑖𝑛 𝑁𝑡 𝐬(𝑡) serve as input, and output is the approximation of the first-order
satisfies 𝜎1 ≥ 𝜎2 ≥ ⋯ ≥ 𝜎𝑘 ≥ 0, 𝑘 ≤ min(𝑁ℎ , 𝑁𝑡𝑟𝑎𝑖𝑛 × 𝑁𝑡 ). derivative function of the reduced state variable. For neural network
POD takes advantage of the SVD of 𝐘 and obtain an 𝑁-dimensional function 𝐠𝑛 , the reduced state variable is the input, and the output is the
subspace spanned by the columns of the orthogonal matrix 𝐕𝑁 , where approximation of generalized coordinates of (r)POD. Here the reduced
matrix 𝐕𝑁 is formed by the first 𝑁 columns of matrix 𝐕. According to state variable has no physical meaning, just an auxiliary variable to
relevant theoretical analysis, when the matrix 𝐕𝑁 is spanned by the left construct the mapping 𝐮(𝑡) → ̃ 𝐲(𝑡).
singular vectors corresponding to the first 𝑁 largest singular values, the
error can be evaluated by 2.2. Continuous solution strategy
𝑁𝑡𝑟𝑎𝑖𝑛 𝑁𝑡 𝑁𝑡𝑟𝑎𝑖𝑛 ×𝑁𝑡
∑ ∑ ‖ 𝑘 ‖2 ∑
𝐸𝑒𝑟𝑟 = ‖𝐲𝑗 − 𝐕𝑁 𝐕T𝑁 𝐲𝑗𝑘 ‖ 𝑁ℎ = 𝜎𝑖2 . (5) For the continuous problem (6), suppose we have obtained its 𝑁𝑡𝑟𝑎𝑖𝑛
‖ ‖R
𝑗=1 𝑘=1 𝑖=𝑁+1 input–output data pairs {̂ 𝐲𝑗 (𝑡)}, 𝑗 = 1, ⋯, 𝑁𝑡𝑟𝑎𝑖𝑛 . The transforma-
𝐮𝑗 (𝑡), ̂
And according to the relative information loss value tion basis matrix 𝐕𝑁 is obtained by prior dimension reduction of the
HF output set  ∶=  0 ([0, 𝑇 ]; R𝑁𝑦 ), that is, for any ̂𝐲𝑗 (𝑡) ∈ , 𝑗 = 1, ⋯,
∑𝑁𝑡𝑟𝑎𝑖𝑛 ∑𝑁𝑡 ‖ 𝑘 ‖2 ∑𝑁𝑡𝑟𝑎𝑖𝑛 ×𝑁𝑡 2
‖𝐲 − 𝐕𝑁 𝐕T𝑁 𝐲𝑗𝑘 ‖ 𝑁ℎ
𝑘=1 ‖ 𝑗 𝜎𝑖 𝑁𝑡𝑟𝑎𝑖𝑛 , we have
𝑗=1 ‖R 𝑖=𝑁+1
𝜀𝑛 = = ∑𝑁 ×𝑁 ,
∑𝑁𝑡𝑟𝑎𝑖𝑛 ∑𝑁𝑡 ‖ 𝑘 ‖2 𝑡𝑟𝑎𝑖𝑛 𝑡 2
𝜎𝑖 ̂
𝐲𝑗 (𝑡) ≃ 𝐕𝑁 𝐂𝑗 (𝑡)
𝑗=1
‖𝐲 ‖
𝑘=1 ‖ 𝑗 ‖R𝑁ℎ 𝑖=1

which can help us determine the value of 𝑁. In practical applications, where 𝐂𝑗 (𝑡) ∈  ∶=  0 ([0, 𝑇 ]; R𝑁 ) is (r)POD generalized coordinate
the value of 𝑁 is usually not preset by users, but the optimal value of set and satisfies 𝐂𝑗 (𝑡) ≃ 𝐕T𝑁 ̂ 𝐲𝑗 (𝑡), and the approximate error loss can
𝑁 is found according to the value of 𝜀𝑛 . be guaranteed by (5). Then, ANNs is used to implicitly construct the
However, for large-scale problems, the SVD of snapshot matrix 𝐘 is mapping set 𝛷 = {𝜑 ∶  → } from the parameter set  ∶=
not a simple problem to calculate. When the spatial dimension 𝑁ℎ is  0 ([0, 𝑇 ]; R𝑁𝑢 ) to the set . For any 𝜑, there is 𝐂𝑗 (𝑡) ≃ 𝜑(̂ 𝐮𝑗 (𝑡)) is
expected to be obtained. Therefore, the loss function is obtained by
large, the parameter space is large and the time step is small, the cost
∑𝑁𝑡𝑟𝑎𝑖𝑛 𝑇 2
of memory space and calculation time required by SVD is inestimable. 𝐽 (𝜑) = 21 𝑗=1 ∫0 |𝐂𝑗 (𝑡) − 𝜑(̂ 𝐮𝑗 (𝑡))| 𝑑𝑡
Therefore, we use a low rank matrix approximation method based on 1 ∑𝑁𝑡𝑟𝑎𝑖𝑛 𝑇 2 (7)
= 2 𝑗=1 ∫0 |𝐕T𝑁 ̂ 𝐲𝑗 (𝑡) − 𝜑(̂ 𝐮𝑗 (𝑡))| 𝑑𝑡,
randomization, namely the rPOD method [41]. It uses matrix factor-
ization to accomplish the basic tasks of linear algebra, which makes it so our goal is to find 𝜑∗ ∈ 𝛷, so that
possible to deal with practical calculations of large-scale problems.
𝜑∗ = argmin𝐽 (𝜑) (8)
First, for a given Gaussian random matrix 𝐗 ∈ R𝑁ℎ ×𝑁 , where 𝑁 < 𝜑∈𝛷
𝑚 < 𝑁ℎ is satisfied, where 𝑚 − 𝑁 is called the over-sampling parameter. Since we use ANNs to implicitly build map 𝜑 ∶  → , so there is
The matrix ( )
𝐮𝑗 (𝑡)) = 𝐠𝑛 𝐬𝑗 (𝑡); 𝝑 exists, and the equivalent expression of (7) can
𝜑(̂
𝛷 = (𝐘𝐘T )𝑞 𝐘𝐗, be obtained as
𝑁𝑡𝑟𝑎𝑖𝑛
1 ∑ ( )|2
𝑇
when the singular value of the matrix 𝐘 decays slowly, we introduce | T̂
𝐽= |𝐕 𝐲 (𝑡) − 𝐠𝑛 𝐬𝑗 (𝑡); 𝝑 | 𝑑𝑡, (9)
the power iteration 𝑞 = 1 or 𝑞 = 2. Then QR factorization of 𝛷 gives 2 𝑗=1 0 | 𝑁 𝑗
∫ |
𝛷 = 𝐐𝐑, where 𝐐 is an orthogonal matrix, we have 𝐘 ≈ 𝐐𝐐T 𝐘. Here, therefore, taking (9) as the loss function and the reduced state variable
we limit 𝐐 = 𝐐(∶, 1 ∶ 𝑁), and let 𝐁 = 𝐐T 𝐘, and then perform SVD Eq. (6) as the constraint condition, the optimization problem (8) can be
decomposition on 𝐁 to obtain transformed into a class of optimization problem with constraints [42],
𝐁 = 𝐐T 𝐘 = 𝐕
̃Σ ̃
̃ 𝐙. namely

̃Σ ̃ Such that 𝐕𝑁 = 𝐐𝐕,

̃ 𝐙. ̃ then 𝐕𝑁 is an ⎧ 𝑁𝑡𝑟𝑎𝑖𝑛
So there is 𝐘 ≈ 𝐐𝐁 = 𝐐𝐕 1 ∑ ( )|2
𝑇
⎪ | T̂
orthogonal matrix and all the columns of 𝐕𝑁 span an 𝑁-dimensional min |𝐕 𝐲 (𝑡) − 𝐠𝑛 𝐬𝑗 (𝑡); 𝝑 | 𝑑𝑡
⎪ ̂ ,𝐠𝑛 ∈,𝐬
𝐟𝑛 ∈ ̂ 0 ∈𝑛 2 𝑗=1 ∫0 | 𝑁 𝑗 |
subspace. ⎨ ( )
⎪ s.t. 𝐬̇ 𝑗 (𝑡) = 𝐟𝑛 𝐬𝑗 (𝑡), ̂
𝐮𝑗 (𝑡); 𝜽 , 𝑗 = 1, … , 𝑁𝑡𝑟𝑎𝑖𝑛 , 𝑡 ∈ (0, 𝑇 ].
For the (r)POD-ANNs model order reduction method, firstly, a priori ⎪
⎩ 𝐬𝑗 (0) = 𝐬0 , 𝑗 = 1, … , 𝑁𝑡𝑟𝑎𝑖𝑛 .
(r)POD dimension reduction is performed for the HF data set, and ANNs
are used to implicitly construct the mapping relationship between (10)

3
F. Cheng et al. Computers and Fluids 241 (2022) 105481

From the above equation, we know that the mapping 𝜑 depends on 𝐟𝑛 , the following simplified version can be obtained
𝐠𝑛 and 𝐬0 , which refers to 𝜑𝐟 𝐠 𝐬 . That is, to find the best representation
𝑛 𝑛 0
⎧ 𝑁𝑡𝑟𝑎𝑖𝑛
̂ , ̂ and 𝑛 . 1 ∑ ( )|2
of 𝐟𝑛 , 𝐠𝑛 and 𝐬0 in the corresponding function space  𝑇
⎪ | T̂
min |𝐕 𝐲 (𝑡) − 𝐠𝑛 𝐬𝑗 (𝑡); 𝝑 | 𝑑𝑡
For the optimization problem (10) with equality constraints, we use ⎪ ̂ ,𝐠𝑛 ∈̂
𝐟𝑛 ∈ 2 𝑗=1 ∫0 | 𝑁 𝑗 |
the Lagrange multiplier method to solve it. So the Lagrange function is ⎨ ( )
⎪ s.t. 𝐬̇ 𝑗 (𝑡) = 𝐟𝑛 𝐬𝑗 (𝑡), ̂
𝐮𝑗 (𝑡); 𝜽 , 𝑗 = 1, … , 𝑁𝑠 , 𝑡 ∈ (0, 𝑇 ].
obtained as follows ⎪ 𝐬𝑗 (0) = 𝟎, 𝑗 = 1, … , 𝑁𝑠 .
⎩
}) 1 ∑
𝑁𝑡𝑟𝑎𝑖𝑛
( { } { 𝑇
| T̂ ( )|2 (15)
 𝜽, 𝝑, 𝐬𝑗 (𝑡) , 𝐰𝑗 (𝑡) = |𝐕 𝐲 (𝑡) − 𝐠𝑛 𝐬𝑗 (𝑡); 𝝑 | 𝑑𝑡
2 𝑗=1 ∫0 | 𝑁 𝑗 |
Compared with (10), the optimization variables are only 𝐟𝑛 and 𝐠𝑛
∑
𝑁𝑡𝑟𝑎𝑖𝑛 𝑇 ( ( )) related to neural network functions, we only need to find the optimal
− 𝐰𝑗 (𝑡) ⋅ 𝐬̇ 𝑗 (𝑡) − 𝐟𝑛 𝐬𝑗 (𝑡), ̂
𝐮𝑗 (𝑡); 𝜽 𝑑𝑡
𝑗=1
∫0 representation of 𝐟𝑛 and 𝐠𝑛 in the corresponding continuous function
space through optimization methods. The optimization problem is fur-
∑
𝑁𝑡𝑟𝑎𝑖𝑛
( )
− 𝐰𝑗 (0) ⋅ 𝐬𝑗 (0) − 𝐬0 , ther simplified by reducing the optimization variables. Since (15) is
𝑗=1 a simplified version of (10), we can still solve it using the Lagrange
(11) multiplier method.

where 𝐰𝑗 (𝑡) and 𝐰𝑗 (0) are Lagrange multipliers. By setting the first
derivative of the dual variable to 0, we get the Lagrange multiplier 3. Optimizing strategy
information as follows
⎧−𝐰̇ 𝑗 (𝑡) = ∇𝐬 𝐠𝑛 (𝐬𝑗 (𝑡); 𝝑)T (∇𝐬 𝐠𝑛 (𝐬𝑗 (𝑡); 𝝑) − 𝐕T𝑁 ̂
𝐲𝑗 (𝑡))
⎪ We have obtained the solution strategy of (r)POD-ANNs model
⎨ + ∇𝐬 𝐟𝑛 (𝐬𝑗 (𝑡), ̂𝐮𝑗 (𝑡); 𝜽)T 𝐰𝑗 (𝑡) 𝑡 ∈ [0, 𝑇 ) (12) order reduction method in the case of time continuity, that is, the
⎪
⎩ 𝐰𝑗 (𝑇 ) = 𝟎. MOR problem is transformed into a class of constrained optimiza-
tion problem, and obtained the optimization problem about neural
Once the Lagrange multiplier information is obtained, for (10), we
network functions. Next let us start with a fully discrete low-order
need to find the optimal neural network functions 𝐟𝑛 and 𝐠𝑛 , where
approximation model, consider the discrete optimization problems,
function 𝐟𝑛 depends on its weight and bias information 𝜃 and function 𝐠𝑛
optimization of neural network functions and optimization methods are
depends on 𝜗. For the optimization of neural network functions, the fa-
used.
mous back propagation algorithm in machine learning is adopted [42].
Here, we take Lagrange function (11) as the loss function of optimizing
neural network functions 𝐟𝑛 and 𝐠𝑛 , and obtain the first-order sensitivity
of neural network parameters 𝜃 and 𝜗 in the form of 3.1. The discrete solution
𝑁𝑡𝑟𝑎𝑖𝑛
⎧ ∑ 𝑇
⎪ ∇𝜽  = 𝐮𝑗 (𝑡); 𝜽)T 𝐰𝑗 (𝑡)𝑑𝑡
∇𝜽 𝐟𝑛 (𝐬𝑗 (𝑡), ̂ In order to solve the problem numerically and verify the reliability
∫
⎪ 𝑗=1 0
of the (r)POD-ANNs model order reduction method. We discrete the
⎨ (13)
𝑁𝑡𝑟𝑎𝑖𝑛
⎪ ∑ 𝑇 time layer, consider the constant time step 𝛥𝑡 (which can also generalize
⎪∇𝝑  = ∇𝝑 𝐠𝑛 (𝐬𝑗 (𝑡); 𝝑)T (𝐠𝑛 (𝐬𝑗 (𝑡); 𝝑) − 𝐕T𝑁 ̂
𝐲𝑗 (𝑡))𝑑𝑡
⎩ ∫ to variable time steps), such that 𝑡𝑘 = 𝑘𝛥𝑡, 𝑘 = 0, 1, ⋯, 𝑁𝑡 , then we
𝑗=1 0
will collect time instants 0 = 𝑡0 < 𝑡1 < ⋯ < 𝑡𝑁𝑡 = 𝑇 . And for the
Due to the large number of HF data sets and neural network pa- 𝐲𝑗𝑘 to represent
sake of expression, we use 𝐮𝑘𝑗 = 𝐮̂ 𝑗 (𝑡𝑘 ), 𝐬𝑘𝑗 = 𝐬𝑗 (𝑡𝑘 ) and ̂
rameters, the computation of second-order sensitivity of neural network the corresponding input values, reduced state variables and HF solu-
parameters is too large, which consumes storage space. Therefore, opti- tions at the discrete time. Therefore, the corresponding discrete loss
mization methods with first-order sensitivity information are generally function is
used. 𝑁𝑡𝑟𝑎𝑖𝑛 𝑁𝑡 −1
1 ∑ ∑ 2
𝐽= 𝛥𝑡 |𝐕T𝑁 ̂
𝐲𝑗𝑘 − 𝐠𝑛 (𝑠𝑘𝑗 ; 𝝑)| . (16)
2.3. Specific constraints on 𝐬0 2 𝑗=1 𝑘=0

Similarly, we obtain the discrete corresponding version of optimiza-

Because different ANNs can represent the same function, we know
tion problem (10) as follows
that the representation of 𝜑𝐟 𝐠 𝐬 is not unique. For any reversible and
𝑛 𝑛 0
regular mapping 𝐡 ∶ R𝑛 ↦ R𝑛 , when ̂𝐬 = 𝐡(𝐬), and then ⎧ 𝑁𝑡𝑟𝑎𝑖𝑛 𝑁𝑡 −1 ( )|2
⎪ 1 ∑ ∑ || T 𝑘
min | 𝐲𝑗 − 𝐠𝑛 𝐬𝑘𝑗 ; 𝝑 ||
𝐕𝑁 ̂
̂ ⎪ 𝑁 +𝑁
(𝜽,𝝑)∈R 𝐟𝑛 𝐠𝑛 2 𝑗=1 𝑘=0 | |
𝐟𝑛 (̂𝐬, 𝐮) = (∇𝐡◦𝐡−1 )𝐟𝑛 (𝐡−1 (̂𝐬), 𝐮), 𝐠𝑛 (̂𝐬) = 𝐠𝑛 (𝐡−1 (̂𝐬)),
̂ 𝐬̂0 = 𝐡(𝐬0 ) ⎪ ( )
⎨ s.t. 𝑘+1 𝑘 𝑘 𝑘
𝐬𝑗 = 𝐬𝑗 + 𝛥𝑡𝐟𝑛 𝐬𝑗 , 𝐮𝑗 ; 𝜽 , (17)
(14) ⎪
⎪ 𝑗 = 1, … , 𝑁𝑡𝑟𝑎𝑖𝑛 , 𝑘 = 1, … , 𝑁𝑡 − 1
then we have 𝜑𝐟 = 𝜑̂ , so mapping 𝜑𝐟 is equivalent to ⎪ 𝐬0𝑗 = 𝐬0 , 𝑗 = 1, … , 𝑁𝑡𝑟𝑎𝑖𝑛 .
𝑛 𝐠𝑛 𝐬0 𝐟𝑛 ̂
𝐠𝑛 𝐬̂
0 𝑛 𝐠𝑛 𝐬0 ⎩
mapping 𝜑̂ . For example, for any non-zero real number 𝑘 ∈ R∖{0},
𝐟𝑛 ̂
𝐠𝑛 𝐬̂
0 What needs to be noted here is that for the continuous optimization
there exists a linear transformation 𝐡(𝐱) = 𝑘𝐱, we have ̂ 𝐟𝑛 (̂𝐬, 𝐮) = problem (10), we need to find the optimal neural network functions 𝐟𝑛
𝑘𝐟𝑛 (̂𝐬∕𝑘, 𝐮), ̂𝐠𝑛 (̂𝐬) = 𝐠𝑛 (̂𝐬∕𝑘), and ̂𝐬0 = 𝑘𝐬0 , such that triplet (𝐟𝑛 , 𝐠𝑛 , 𝐬0 ) and and 𝐠𝑛 in the corresponding continuous function space. But for discrete
triplet (̂ 𝐠𝑛 ,̂𝐬0 ) represent the same model. Since optimization problem
𝐟𝑛 , ̂ problems, we need to find the optimal weight and bias information 𝜽
(10) is an ill-posed problem, we consider adding constraints to the and 𝝑 in the corresponding parameter space 𝑁𝐟𝑛 and 𝑁𝐠𝑛 . Here, we
optimization variables here. consider it as an equivalent transformation, because when the structure
Consider that for model 𝜑𝐟 𝐠 𝐬 ∈ 𝛷, suppose there is an invertible and activation function of neural network are determined, the weight
𝑛 𝑛 0
transformation 𝐡(𝐱) = 𝐱 − 𝐬0 , by applying (14), we can obtain ̂ 𝐟𝑛 (̂𝐬, 𝐮) = and bias information will directly affect the output result of neural
𝐠𝑛 (̂𝐬) = 𝐠𝑛 (̂𝐬 + 𝐬0 ), and ̂𝐬0 = 𝟎. Without loss of generality, we
𝐟𝑛 (̂𝐬 + 𝐬0 , 𝐮), ̂ network function. When the optimal 𝜽 and 𝝑 are found, the optimal
make 𝐬0 = 𝟎 by converting 𝐡(𝐱). Thus, for optimization problem (10), network functions 𝐟𝑛 and 𝐠𝑛 can be obtained accordingly.

4
F. Cheng et al. Computers and Fluids 241 (2022) 105481

Fig. 1. Left: the structure of ANNs; Right: the inner structure of the 𝑖th neuron.

According to the time continuous Lagrangian function (11), the mathematical expression
following fully discrete Lagrangian function is obtained as follows
𝛹 (𝐗, 𝐖) = 𝐖𝐿−1 𝑓𝑎𝑐𝑡 (⋯ 𝐖2 𝑓𝑎𝑐𝑡 (𝐖1 𝐗 − 𝐛1 ) − 𝐛2 ⋯) − 𝐛𝐿−1 , (21)
( { } { }) 𝑁𝑡𝑟𝑎𝑖𝑛 𝑁𝑡 −1
∑ ∑ | ( )|2 where 𝐖𝑖 and 𝐛𝑖 (𝑖 = 1, 2, … , 𝐿 − 1), respectively represent the weight
1 |𝐕T ̂
 𝜽, 𝝑, 𝐬𝑘𝑗 , 𝐰𝑘𝑗 = 𝛥𝑡 | 𝑁 𝐲𝑗𝑘 − 𝐠𝑛 𝐬𝑘𝑗 ; 𝝑 ||
and bias of the 𝑖 hidden layer, 𝐗 is the input vector, and 𝐖 is the set
𝑗=1 𝑘=0 | |
2
of weight and bias of the neural network function.
𝑁𝑡𝑟𝑎𝑖𝑛 𝑁𝑡
∑ ∑ ( ( ))
− 𝐰𝑘𝑗 ⋅ 𝐬𝑘𝑗 − 𝐬𝑘−1 − 𝛥𝑡𝐟𝑛 𝐬𝑘−1 𝑘−1 The output of the network depends on the connection mode of the
𝑗 𝑗 , 𝐮𝑗 ; 𝜽
𝑗=1 𝑘=1 network, the initialization of weights and the selection of activation
𝑁𝑡𝑟𝑎𝑖𝑛 ( ) functions. In this work, the fully connected neural network is mainly
∑
− 𝐰0𝑗 ⋅ 𝐬0𝑗 − 𝐬0 , adopted, and the initial weights and biases are randomly initialized in
𝑥 −𝑥
𝑗=1 interval [0, 1], and activation function is selected with 𝑓𝑎𝑐𝑡 (𝑥) = 𝑒𝑒𝑥 −𝑒
+𝑒−𝑥
(18) to increase the nonlinearity of neural network function.
According to the discrete optimization problem (17), we can obtain
taking (18) as the loss function, HF data set is used to train the optimal the neural network structure of the (r)POD-ANNs model order reduc-
representation of neural network functions. In the process of back tion method as shown in Fig. 2. The discrete reduced state variable
propagation, it is necessary to use the first-order sensitivity information equation is the bridge connecting the two neural network functions.
of the loss function about optimization variables 𝜽 and 𝝑, that is, the
discretized version corresponding to (13) as follows 3.2.1. Data preprocessing
𝑁𝑡𝑟𝑎𝑖𝑛 𝑁𝑡
First of all, in order to eliminate the effect of different units and
∑ ∑ orders of magnitude between parameters, we standardized training
T 𝑘
∇𝜽  = 𝛥𝑡 ∇𝜽 𝐟𝑛 (𝐬𝑘−1 𝑘−1
𝑗 , 𝐮𝑗 ; 𝜽) 𝐰𝑗
𝑗=1 𝑘=1 data and test data. In order to accelerate the off-line training stage of
(19) the reduced order model, we mainly consider adopting some strategies.
𝑁𝑡𝑟𝑎𝑖𝑛 𝑁𝑡 −1
∑ ∑
∇𝝑  = 𝛥𝑡 ∇𝐬 𝐠𝑛 (𝐬𝑘𝑗 ; 𝝑)T (𝐠𝑛 (𝐬𝑘𝑗 ; 𝝑) − 𝐕T𝑁 ̂
𝐲𝑗𝑘 ), For example, (i) The optimization algorithm with fast convergence
𝑗=1 𝑘=0 speed is used to find a good initial value for the weight and bias of
the initial network on a small batch data set. (ii) In terms of time
where
layer, we update the parameter value of neural network every few time
⎧ 𝑘 𝑘+1 𝑘 𝑘 T 𝑘+1 𝑘 T 𝑘 T 𝑘 steps, but the time step should not be too large. (iii) if the loss function
⎪ 𝐰𝑗 =𝐰𝑗 + 𝛥𝑡∇𝐬 𝐟𝑛 (𝐬𝑗 , 𝐮𝑗 ; 𝜽) 𝐰𝑗 + 𝛥𝑡∇𝐬 𝐠𝑛 (𝐬𝑗 ; 𝝑) (𝐠𝑛 (𝐬𝑗 ; 𝝑) − 𝐕𝑁 ̂
𝐲𝑗 )
value of the neural network does not decrease every 500 epochs, the
⎨ 𝑀𝑗
⎪𝐰𝑗 =0, regularization strategy of early stop is adopted.
⎩
(20) 3.3. Optimizing algorithm

is the discretized counterpart of (12). For the least squares problem

1
min 𝑓 (Θ) = |𝐫(Θ)|2 (22)
3.2. Artificial neural network
𝑁 +𝑁
Θ∈R ̃𝐟 ̃𝐠 2
where Θ represents the weight and bias information of the ANN
Artificial neural network (ANN) is a mathematical model of infor- function. For (22), if used Newton update iteration, e.g
mation processing similar to the structure of synaptic connections in the
Θ𝑘+1 = Θ𝑘 − (∇𝐫(Θ𝑘 )T ∇𝐫(Θ𝑘 ) + ∇2 𝐫(Θ𝑘 )T 𝐫(Θ𝑘 ))−1 ∇𝐫(Θ𝑘 )T 𝐫(Θ𝑘 ).
brain, which is mainly composed of a large number of interconnected
neurons. As shown in Fig. 1(Left), ANN mainly has input layer, hidden (23)
layer and output layer, among which the hidden layer can be multiple
As Θ is a large parameter set, the training of neural network re-
layers [43]. There are weights between neurons of different layers that
quires a large amount of HF data, so we ignore the second-order
represent their connection information. According to Fig. 1 (Right), it
information. Considering that in the process of neural network back
can be found that the information of the previous layer is transferred
propagation [44], the gradient value of the loss function approaches 0,
to the neuron of the next layer by weighting, which is the output of the
we consider adding 𝜆𝑘 𝐈 to the gradient term to avoid the phenomenon
neuron after bias and activation function.
of gradient disappearance in the training process. Since the Newton
ANN is optimized according to the learning method of mathematical
direction is not necessarily the descending direction, one dimensional
statistics type, so ANN is also a practical application of mathemat-
search is added to the Newton direction, namely LM algorithm
ical statistics method. It is composed of nonlinear functions 𝜎𝑗 =
∑
𝑓𝑎𝑐𝑡 ( 𝑖 𝑤𝑖 𝑥𝑖 − 𝑏𝑗 ) nested with each other, so it has the following Θ𝑘+1 = Θ𝑘 − 𝛼𝑘 (∇𝐫(Θ𝑘 )T ∇𝐫(Θ𝑘 ) + 𝜆𝑘 𝐈)−1 ∇𝐫(Θ𝑘 )T 𝐫(Θ𝑘 ). (24)

5
F. Cheng et al. Computers and Fluids 241 (2022) 105481

Fig. 2. The ANN structure of the POD-ANN model reduction method.

( )
Where 𝜆𝑘 = 𝑚𝑖𝑛{|𝑟(Θ0 )|2 , |∇𝐫(Θ𝑘 )T ∇𝐫(Θ𝑘 )|} can ensure that the algo- can be guaranteed by mathematical theory (𝜖𝑟𝑒𝑙 ̂ 𝐲𝑗𝑘 , 𝐕𝑁 𝐕T𝑁 𝐂𝑘𝑗 ≈ 10−4 ).
rithm is superlinear convergence. The other part is the loss error when using neural network to construct
LM algorithm needs a good initial iteration value Θ𝟎 , so for large- implicitly the mapping relation between parameters set and generalized
scale optimization problems, we consider using L-BFGS algorithm (see coordinates of (r)POD, this can be done by minimizing the loss function
Algorithms 𝟏) to find a relatively good initial value Θ𝟎 . Since L-BFGS (18).
only stores the information of the latest 𝑚 iterations, the storage space
of data is greatly reduced. 4. Numerical experiments

Algorithm 1: L-BFGS algorithm

In this section, we apply the proposed (r)POD-ANNs model order
Step 1: Select the initial point Θ0 , and tolerance 𝜀 > 0, and
reduction method to some parameterized time-dependent PDEs. In the
store the data of the latest 𝑚 iteration.
first example we mainly compare the model-based MOR method, the
Step 2: 𝑘 = 0, 𝐇0 = 𝐈, 𝐝 = ∇𝑓 (Θ0 ).
data-driven MOR method and the (r)POD-ANNs model order reduction
Step 3: If there is ‖∇𝑓 (Θ𝑘+1 )‖ ≤ 𝜀, return the optimal solution
method. Then, we consider the application of (r)POD-ANNs model
Θ𝑘+1 , otherwise return Step 4.
order reduction method to strongly nonlinear parameterized Allen-
Step 4: Calculate the feasible direction 𝐩𝑘 = −𝐝𝑘 of this
Cahn equation. And for the two-dimensional cylinder flow problem
iteration.
with high spatial complexity, we consider using rPOD to perform prior
Step 5: Compute step 𝛼𝑘 > 0 and perform a one-dimensional
dimensionality reduction for HF data, and comparing the effect of
search on the following formula
adding pre-training and combining LM algorithm with L-BFGS algo-
𝑓 (Θ𝑘 + 𝛼𝑘 𝐩𝑘 ) = min 𝑓 (Θ𝑘 + 𝛼𝐩𝑘 ).
rithm. Finally, the possibility of (r)POD-ANNs model order reduction
Step 6: Update weight and bias information Θ𝑘+1 = Θ𝑘 + 𝛼𝑘 𝐩𝑘 .
method for the 3D cylinder flow problem with higher spatial complexity
Step 7: If 𝑘 > 𝑚 is true, retain the data of the latest 𝑚 iterations
is considered.
and delete (𝐰𝑘−𝑚 , 𝐯𝑘−𝑚 ).
Step 8: Compute and hold 𝐰𝑘 = Θ𝑘+1 − Θ𝑘 ,
4.1. 2D Burger’s equation
𝐯𝑘 = ∇𝑓 (Θ𝑘+1 ) − ∇𝑓 (Θ𝑘 ).
Step 9: Let us use two-loop recursion algorithm to solve for
We first consider the basic equation of fluid dynamics, which is
𝐝𝑘 = 𝐁𝑘 ∇𝑓 (Θ𝑘 ), here 𝐁𝑘 depends on the value of 𝐰𝑘
the Burger’s equation, refer to [45]. This is a non-true-solution prob-
and 𝐯𝑘 𝑚 times before and 𝐇0 .
lem where the initial conditions and boundary conditions have the
Step 10: 𝑘 = 𝑘 + 1, and return to Step 3.
following form
In order to evaluate the performance of (r)POD-ANN model order ⎧ 𝜕𝑢 𝜕𝑢 𝜕𝑢 𝜕2 𝑢 𝜕2 𝑢
+𝑢 +𝑢 =𝝂 +𝝂 (𝑥, 𝑦, 𝑡) ∈ 𝛺 × (0, 𝑇 ]
reduction method, we mainly use( the)following two error indicators: ⎪ 𝜕𝑡 𝜕𝑥 𝜕𝑦 2
𝜕 𝑥 𝜕2 𝑦
⎨ (27)
𝐲𝑗𝑘 , ̃
(1) The error indicator 𝜖𝑟𝑒𝑙 ̂ 𝐲𝑗𝑘 is as follows 𝑢(𝑥, 𝑦, 𝑡) = 0 (𝑥, 𝑦, 𝑡) ∈ 𝜕𝛺 × (0, 𝑇 ]
⎪
√ ⎩ 𝑢(𝑥, 𝑦, 0) = sin(2𝜋𝑥) cos(2𝜋𝑦) (𝑥, 𝑦) ∈ 𝛺,
⎛ ∑𝑁𝑡 −1 ⎞
( ) 𝑁𝑡𝑒𝑠𝑡
∑⎜ ‖̂ 𝐲𝑗𝑘 ‖2 ⎟
𝐲𝑗𝑘 − ̃ with𝑇 = 1 and 𝛺 = [0, 1]2 . And the viscosity coefficient 𝝂 ∈ [0.01, 0.1].
1 𝑘=0
𝐲𝑗𝑘 , ̃
𝜖𝑟𝑒𝑙 ̂ 𝐲𝑗𝑘 = ⎜ √ (25)
∑𝑁𝑡 −1 𝑘 2 ⎟⎟
.
𝑁𝑡𝑒𝑠𝑡 𝑗=0 ⎜
Here, we use the HF solution technology FEM with spatial freedom of
⎝ 𝑘=0
‖̂
𝐲𝑗 ‖ ⎠ 𝑁ℎ = 441 for numerical solution, and obtain the HF solution snapshot
( ) when 𝑡 = 0.3, 0.6 and 1 as shown in Fig. 3.
𝐲𝑗𝑘 , ̃
(2) The error indicator 𝜖𝑎𝑏𝑠 ̂ 𝐲𝑗𝑘 is as follows
We firstly compare the data-driven MOR method with the POD-
( ) |̂𝑘 ̃𝑘 | ANNs model order reduction method, for the training the ANNs, we
|𝐲𝑗 − 𝐲𝑗 |
𝐲𝑗𝑘 , ̃
𝜖𝑎𝑏𝑠 ̂ 𝐲𝑗𝑘 = √ | | . consider 𝑁𝑡 = 10 time instances, 𝑁𝑡𝑟𝑎𝑖𝑛 = 140 training-parameter
√ (26)
√ 𝑁𝑡 −1
1 ∑ 𝑘 2
instances and 𝑁𝑡𝑒𝑠𝑡 = 60 test-parameter instances uniformly distributed
√ ‖̂
𝐲 ‖ in parametric domain. That is, directly build the mapping relationship
𝑁𝑡 𝑘=0 𝑗
between parameters set and output set through (4), and perform POD
Because the (r)POD-ANNs model order reduction method can be prior dimensionality reduction on output data firstly, and build the
regarded as a two-step dimensionality reduction method. Therefore, mapping relationship between parameter set and generalized coordi-
the error also includes two parts. One is the loss error of the prior nates of POD through (6). In Table 1, with different reduced state
dimension reduction of spatial data based on (r)POD method, which variables dimension 𝑛 = 1, 2, 3, 4, where the data-driven MOR method

6
F. Cheng et al. Computers and Fluids 241 (2022) 105481

Fig. 3. HF snapshots at different time.

Table 1
Error comparison between data-driven MOR method and the POD-ANN model order reduction method.
Case Data-driven MOR method POD-ANNs model reduction method
𝑛 1 2 3 4 1 2 3 4
Train errors 5.48e−02 5.48e−02 5.48e−02 5.48e−02 3.31e−03 8.37e−04 8.31e−04 8.31e−04
Test errors 5.80e−02 5.79e−02 5.79e−02 5.79e−02 4.50e−03 1.11e−03 1.12e−03 1.11e−03
Offline time 18.56 h 16.12 h 17.69 h 18.52 h 3.52 h 5.32 h 4.20 h 6.34 h

Fig. 4. MOR results of the RB method (top), the data-driven MOR method (middle) and the POD-ANN model reduction method (bottom) at 𝑡 = 0.3.

with 𝐹𝑙𝑎𝑦𝑒𝑟 = 𝑛 + 2 (number of hidden layer units of neural network Finally, we compare the relative error loss and relative online
𝐟𝐴𝑁𝑁 ) and 𝐺𝑙𝑎𝑦𝑒𝑟 = 2 (number of hidden layer units of neural network calculation time of the proposed POD-ANNs model order reduction
function 𝐠𝐴𝑁𝑁 )). For the POD-ANNs model order reduction method, method with the data-driven MOR method and the RB method in Fig. 5.
we set the subspace dimension of POD prior dimension reduction as It can be found that the relative information loss and relative error of
𝑁 = 40, neural network function 𝐟𝑛 has 12 hidden layer units, and RB method decrease with the increase of the dimension of the reduced
function 𝐠𝑛 has 4 hidden layer units. By comparing the effect of the state variables, but its online calculation time increases gradually. For
two model reduction methods, it can be found that the loss error of the data-driven MOR method, under the same dimension of the reduced
the POD-ANNs model order reduction method is at least one order of state variable, compared with RB method, the low-order model with
magnitude lower than the data-driven MOR method. smaller relative error can be obtained, and its relative online calcula-
In Fig. 4, we compared the MOR effects of the traditional RB tion time is also less. However, compared with the POD-ANNs model
method, the data-driven MOR method and the POD-ANNs model order order reduction method, the latter can obtain a low-order approximate
reduction method for different reduced state variable dimension 𝑛 = model with higher reliability, and its online computing time is slightly
1, 2, 3, 4, which can be compared with the left of the HF solution different according to the complexity of neural network. For the model
snapshot in Fig. 3. reduction method based on neural network, once the structure of neural

7
F. Cheng et al. Computers and Fluids 241 (2022) 105481

Fig. 5. Comparison of error and online computing time based on RB method, the data-driven MOR method and the POD-ANN model reduction method.

Fig. 6. HF snapshots at different times with parameter 𝜀 = 0.0384.

( )
network is determined, when the difference of parameter space is not 𝐲𝑗𝑘 , 𝐕𝑁 𝐕T𝑁 ̂
(𝜖𝑟𝑒𝑙 ̂ 𝐲𝑗𝑘 ≈ 1.88𝑒 − 4) respectively, then L-BFGS algorithm
big, its loss error eventually converges to its minimum possible error. is used for neural network optimization training. We get the solu-
The POD-ANNs model order reduction method has certain advantages tion snapshot corresponding to the POD-ANNs model order reduction
both in terms of relative error indicator and online calculation time. method when 𝑁 = 40, as shown in Fig. 7 (above). Compared with the
HF numerical solution, the calculation accuracy is lower. Therefore, we
4.2. Allen-Cahn equations adopt more low-dimensional features of POD. For example, by making
𝑁 = 80, we obtain a numerical solution snapshot of the POD-ANNs
model order reduction method, as shown in Fig. 7 (below).
Then we consider the strongly nonlinear Allen-Cahn equation with
Finally, in Table 2, for the subspace dimensions of prior dimension
the following form, which can be referred to [46]
reduction are 𝑁 = 40 and 𝑁 = 80 respectively, and different dimensions
⎧ 𝜕𝑢 − 𝜀2 𝛥𝑢 − 𝑢 + 𝑢3 = 0 (𝑥, 𝑦, 𝑡) ∈ 𝛺 × (0, 𝑇 ] of reduced state variables, we compare the time required by a single
⎪ 𝜕𝑡 iteration of L-BFGS algorithm for different numbers of neural network
⎨ 𝜕𝑢 ∣𝜕𝛺 = 0 𝑡 ∈ [0, 𝑇 ] (28) ( )
⎪ 𝜕𝐧 optimization parameters. And the mean 𝐿2 error indicator 𝜖𝑟𝑒𝑙 ̂ 𝐲𝑗𝑘 , ̃
𝐲𝑗𝑘
⎩ 𝑢(𝑥, 𝑦, 0) = 𝑢0 (𝑥, 𝑦) ∈ 𝛺,
of numerical solutions of MOR model under different dimensions of
where 𝑢 represents a distribution function in the mixed component, reduced state variables. It can be found that as 𝑁 increases, we can
and 𝐧 represents the extra-unit normal vector of the boundary. And we get better low-order approximation model. This is also in line with our
consider the following initial condition expectation that the more low-dimensional feature information POD
retains, the more we can obtain a low-order approximation model with
⎧ 1 (𝑥 + 12 )2 + 𝑦2 < 1
⎪ 16 higher numerical accuracy.
⎪ 1 1 2
(𝑥 − 2 ) + 𝑦2 < 16 1
𝑢0 = ⎨ (29)
1 1
⎪ 1 ∣ 𝑥 ∣< 2 , ∣ 𝑦 ∣< 8 4.3. 2D Cylinder flow
⎪ −1 otherwise.
⎩
And parameter 𝜀 ∈ [0.025, 0.045] represents the interface width. We use We now consider a Navier–Stokes equations for unsteady incom-
a space discretization made by 𝑃2 finite elements technology to obtain pressible flows [47] through a long cylinder with initial boundary value
the HF solution with 𝑁ℎ = 61×61 grid points and the time step 𝛥𝑡 = 0.01 conditions of the following form
over the interval (0, 𝑇 ) with 𝑇 = 50. We obtained the HF snapshots of 𝜕𝐮
⎧ 𝜌 + 𝜌𝐮 ⋅ ∇𝐮 − ∇ ⋅ 𝝈(𝐮, 𝑝) = 𝟎, (𝐱, 𝑡) ∈ 𝛺 × (0, 𝑇 )
parameter 𝜀 = 0.0384 at different moments 𝑡 = 0, 𝑡 = 10, 𝑡 = 30 and ⎪ 𝜕𝑡
𝑡 = 50 as shown in Fig. 6. ⎪ ∇ ⋅ 𝐮 = 0, (𝐱, 𝑡) ∈ 𝛺 × (0, 𝑇 )
For the training phase, we consider 𝑁𝑡 = 100 time instances, ⎪ 𝐮 = 𝟎, (𝐱, 𝑡) ∈ 𝛤1 × (0, 𝑇 )
⎨ (30)
and 𝑁𝑡𝑟𝑎𝑖𝑛 = 50 training-parameter instances and 𝑁𝑡𝑒𝑠𝑡 = 25 testing- ⎪ 𝐮 = 𝐡, (𝐱, 𝑡) ∈ 𝛤2 × (0, 𝑇 )
parameter instances. The dimension of reduced state variable 𝑛 = 4, ⎪ 𝝈(𝐮, 𝑝)𝐧 = 𝟎, (𝐱, 𝑡) ∈ 𝛤𝑁 × (0, 𝑇 )
⎪
6, 8 and 10 is considered respectively, where the hidden layer unit of ⎩ 𝐮(0) = 𝟎, 𝐱 ∈ 𝛺, 𝑡 = 0.
neural network function 𝐟𝑛 is 𝐹𝑙𝑎𝑦𝑒𝑟 = 2 × 𝑛, and the hidden layer unit of
Where 𝜌 is the fluid density, 𝐧 is the normal vector outside the unit and
neural network function 𝐠𝑛 is 𝐺𝑙𝑎𝑦𝑒𝑟 = 𝑛. We first consider the prior
𝝈(𝐮, 𝑝) is the stress tensor, satisfying
dimension reduction of HF(data, and consider ) POD prior dimension
reduction of 𝑁 = 40 (𝜖𝑟𝑒𝑙 ̂ 𝐲𝑗𝑘 , 𝐕𝑁 𝐕T𝑁 ̂
𝐲𝑗𝑘 ≈ 9.70𝑒 − 3) and 𝑁 = 80 𝝈(𝐮, 𝑝) = −𝑝𝐈 + 2𝜈𝜺(𝐮),

8
F. Cheng et al. Computers and Fluids 241 (2022) 105481

Fig. 7. Solution snapshot of (r)POD-ANNs method when 𝑁 = 40 and 𝑁 = 80 with parameter 𝜀 = 0.0384.

Fig. 8. Geometry of flow around a two-dimensional cylinder with boundary conditions.

Fig. 9. HF solution for the parameter instances 𝜇 = 1.0462 (𝑅𝑒 = 70) and 𝜇 = 1.6551 (𝑅𝑒 = 110) at 𝑡 = 5.60 s.

Table 2
Error and time comparison of the POD-ANNs model order reduction methods under 𝑁 = 40 and 𝑁 = 80.
Case 𝑁 = 40 N=80
𝑛 4 6 8 10 4 6 8 10
Parameters 304 496 728 1000 504 776 1088 1440
L-BFGS time 3.31 s 3.99 s 4.49 s 5.39 s 4.67 s 6.11 s 7.17 s 10.90 s
Train errors 4.34e−02 4.34e−02 2.32e−02 1.10e−02 4.07e−03 4.34e−03 4.34e−03 4.41e−03
Test errors 3.54e−02 3.54e−02 1.79e−02 9.50e−03 3.38e−03 3.54e−03 3.54e−03 3.62e−03

here 𝜈 denotes the dynamic viscosity of the fluid, while 𝜺(𝐮) is the strain 𝐶0.05 (0.2, 0.2) represents a circle with (0.2, 0.2) as the center and 𝑟 =
tensor, 0.05 as the radius. Boundary conditions are divided into three parts,
including Dirichlet boundary conditions 𝛤1 = {𝑥 ∈ [0, 2.2], 𝑦 = 0} ∪ {𝑥 ∈
1
𝜺(𝐮) = (∇𝐮 + ∇𝐮T ) [0, 2.2], 𝑦 = 0.41} ∪ 𝜕𝐶0.05 (0.2, 0.2), the inlet 𝛤2 = {𝑥 = 0, 𝑦 ∈ [0.0.41]},
2
and the outlet 𝛤𝑁 = {𝑥 = 2.2, 𝑦 ∈ [0.0.41]}.
The region 𝛺 is a two-dimensional pipe with a circular section, as The density of the fluid is 𝜌 = 1, and the dynamic viscosity 𝜈 =
shown in Fig. 8. That is 𝛺 = [0, 2.2] × [0, 0.41] ⧵ 𝐶0.05 (0.2, 0.2), where 1.0 × 10−3 . no-slip boundary conditions are applied on 𝛤1 , a parabolic

9
F. Cheng et al. Computers and Fluids 241 (2022) 105481

Fig. 10. Solution snapshot of (r)POD-ANNs method for the parameter instances 𝜇 = 1.0462 (𝑅𝑒 = 70) and 𝜇 = 1.6551 (𝑅𝑒 = 110) at 𝑡 = 5.60 s.

Table 3
Comparison of the performance of standard POD and rPOD.
Case: 𝑁 = 40 Accuracy loss Computation time (s)
POD rPOD POD rPOD
𝐘 ∈ R14513×4010 1.01e−04 1.19e−04 1.40e+01 6.83e+01
𝐘 ∈ R14513×8020 1.03e−04 1.17e−04 4.57e+02 1.33e+02
𝐘 ∈ R14513×10025 1.01e−04 1.15e−04 9.41e+02 1.59e+02
𝐘 ∈ R14513×12832 9.46e−05 1.07e−04 1.51e+03 2.10e+02

Table 4
Comparison of computation time of algorithm.
L-BFGS time 2.01e+01
LM time 1.30e+01

the neural network, that is, the size of the parameter space of the neural
Fig. 11. Relative error for the testing-parameter instances 𝜇 = 1.1869 (top) and network, according to the performance of the trained neural network
𝜇 = 1.6551 (bottom) at 𝑡 = 5.60. on the verification set. Finally, the effect of the rPOD-ANNs model
order reduction method is evaluated by the performance of neural
network function on the test set. In Fig. 9, we give the HF snapshots
of parameters 𝜇 = 1.0462 (𝑅𝑒 = 70) and 𝜇 = 1.6551 (𝑅𝑒 = 110) at time
𝑡 = 5.60 s.
Considering the large degree of spatial freedom, we should conduct
a priori dimension reduction for the HF snapshot matrix by rPOD
method. In Table 3, we compare the calculation time and accuracy loss
of standard POD and rPOD in different scale of snapshot matrices 𝐘 ∈
R𝑁ℎ ×𝑁𝑡𝑟𝑎𝑖𝑛 𝑁𝑡 . It can be found that when the matrix size is small, standard
Fig. 12. Geometry of 3D cylinder flow with boundary conditions. POD has advantages in both calculation time and accuracy. With the
increase of matrix size, standard POD is time-consuming, while rPOD
has certain advantages in calculation time despite the loss of accuracy.
We can choose more rPOD bases to avoid a certain loss of accuracy,
inflow profile
and selecting 𝑁 = 120 as dimension of the rPOD ( basis for)the velocity
( )
4𝑈 (𝑡, 𝜇)𝑦(0.41 − 𝑦) components. In this way, error indicator 𝜖𝑟𝑒𝑙 ̂ 𝐲𝑗𝑘 , 𝐕𝑁 𝐕T𝑁 ̂
𝐲𝑗𝑘 ≈ 10−4 can
𝐡(𝐱, 𝑡; 𝜇) = ,𝟎 , where 𝑈 (𝑡; 𝜇) = 𝜇 sin(𝜋𝑡∕8)
0.412 be satisfied.
We consider that neural network function 𝐟𝑛 has 𝐹𝑙𝑎𝑦𝑒𝑟 = 24 hidden
(31)
layer neurons, neural network function 𝐠𝑛 has 𝐺𝑙𝑎𝑦𝑒𝑟 = 4 hidden
is prescribed at the inlet 𝛤2 , while zero-stress condition is imposed layer neurons, and the dimension of the reduced state variable is 𝑛 =
at the outlet 𝛤𝑁 . We consider as parameter 𝜇 ∈  = [1, 2], which 8. In order to reduce the off-line training time, some optimization
reflects on the Reynolds number varying in the range [66, 133]. We are strategies are adopted here. (1) The initial weight and bias (𝜽, 𝝑) of the
interested in reconstructing the velocity field, and the HF solution is neural network are obtained by L-BFGS algorithm on a small training
obtained by FEM high fidelity technique, for which the spatial degree set ((with 𝑁 ) 𝑡𝑟𝑎𝑖𝑛 =20 and 𝑁𝑡𝑒𝑠𝑡 =10). When the relative error indicator
of freedom is equal to 𝑁ℎ = 14513 and the time step is 𝛥𝑡 = 2.0 × 10−3 . 𝜖𝑟𝑒𝑙 ̂ 𝐲𝑗𝑘 < 10−2 , the regularization strategy of early stop is adopted.
𝐲𝑗𝑘 , ̃
We obtain available data set every 10 time steps, i.e., 𝑁𝑡 = 400 time (2) Neural network parameters are updated every 5 time steps. And
instances for 𝛥𝑡 = 2.0 × 10−2 . It includes 𝑁𝑡𝑟𝑎𝑖𝑛 = 40 training-parameter, under the same neural network parameter space, the time required by
𝑁𝑡𝑒𝑠𝑡 = 20 testing-parameter and 𝑁𝑣𝑎𝑙 = 10 validating-parameter single iteration of L-BFGS algorithm and LM algorithm is compared in
instances uniformly distributed over . We choose the complexity of Table 4.

10
F. Cheng et al. Computers and Fluids 241 (2022) 105481

Fig. 13. HF snapshots for the parameter instances 𝜇 = 2.4832 (𝑅𝑒 = 110) at 𝑧 = 0.2, 𝑡 = 5 (above) and 𝑧 = 0.05, 𝑡 = 8 (below).

Where 𝜌 = 1 is the fluid density, 𝐧 denotes the normal unit vector to

𝜕𝛺 and 𝝈(𝐮, 𝑝) is the stress tensor, i.e.

𝝈(𝐮, 𝑝) = −𝑝𝐈 + 2𝜈𝜺(𝐮),

where 𝜈 = 1.0 × 10−3 denotes the kinematic viscosity of the fluid, while
𝜺(𝐮) is the strain tensor,
1
𝜺(𝐮) = (∇𝐮 + ∇𝐮T ).
2
The domain consists a cuboid with a circular cross sections, its
geometric sketch and boundary conditions are shown in Fig. 12. That
is 𝛺 = [0, 0.41]×[0, 2.2]×[0, 0.41]⧵𝐵0.05 (0.2, 0.5, 𝑧), where 𝐵0.05 (0.2, 0.5, 𝑧)
represents a cylinder with (0.2, 0.2, 𝑧) as the center, 𝑟 = 0.05 as the
radius, and length 𝑍 = 0.41. The boundary conditions include Dirichlet
boundary 𝛤1 = {𝑥 ∈ [0, 0.41], 𝑦 ∈ [0, 2.5], 𝑧 = 0} ⧵ 𝐵0.05 (0.2, 0.5, 0) ∪ {𝑦 ∈
[0, 2.5], 𝑧 ∈ [0, 0.41], 𝑥 = 0} ∪ {𝑥 ∈ [0, 0.41], 𝑦 ∈ [0, 2.5], 𝑧 = 0.41} ⧵
𝐵0.05 (0.2, 0.5, 0.41)∪{𝑦 ∈ [0, 2.5], 𝑧 ∈ [0, 0.41], 𝑥 = 0.41}∪𝜕𝐵0.05 (0.2, 0.5, 𝑧),
Fig. 14. Error variation in training neural network with L-BFGS algorithm. the inflow plane 𝛤2 = {𝑥 ∈ [0, 0.41], 𝑧 ∈ [0.0.41], 𝑦 = 0}, and the outflow
plane 𝛤𝑁 = {𝑥 ∈ [0, 0.41], 𝑧 ∈ [0.0.41], 𝑦 = 2.5}.
The no-slip boundary conditions are applied on 𝛤1 , a parabolic
The snapshot of the rPOD-ANNs model order reduction method for inflow profile
( )
the parameter instances 𝜇 = 1.0462 (𝑅𝑒 = 70) and 𝜇 = 1.6551 (𝑅𝑒 = 110) 𝐡(𝐱, 𝑡; 𝜇) = 𝟎, 16𝑈 (𝑡,𝜇)𝑥𝑧(0.41−𝑥)(0.41−𝑧) ,𝟎 ,
at 𝑡 = 5.6 s is shown in Fig. 10, where the value of the velocity field in 0.414 (33)
where 𝑈 (𝑡; 𝜇) = 𝜇 sin(𝜋𝑡∕8)
circle 𝐶0.05 (0.2, 0.2) defaults to 𝟎.
Finally, we evaluate the performance( of rPOD-ANNs model order is prescribed at the inlet 𝛤2 , while zero-stress Neumann conditions are
)
reduction method by error indicator 𝜖𝑎𝑏𝑠 ̂𝐲𝑗𝑘 , ̃
𝐲𝑗𝑘 . As shown in Fig. 11, imposed at the outlet 𝛤𝑁 . We consider parameter 𝜇 ∈  = [1.5, 3],
the above is the error indicator corresponding to parameter 𝜇 = 1.1869 which reflects on the Reynolds number varying in the range [67, 133].
at 𝑡 = 5.60, and the below is the error indicator corresponding to The HF solution was obtained by using the HF technology based on
parameter 𝜇 = 1.6551 at 𝑡 = 5.60, where the value of the error indicator FEM, where the spatial degree of freedom was 𝑁ℎ = 37165 and the time
in circle 𝐶0.05 (0.2, 0.2) defaults to 𝟎. step was 𝛥𝑡 = 2.0 × 10−3 . And HF snapshots for the parameter instances
𝜇 = 2.4832 (𝑅𝑒 = 110) at 𝑧 = 0.2, 𝑡 = 5 and 𝑧 = 0.05, 𝑡 = 8 as show in
Fig. 13.
We are interested in reconstructing the velocity field, considering
4.4. 3D Cylinder flow 𝑁𝑡 = 200 time instances, i.e. 𝛥𝑡 = 0.04, 𝑁𝑡𝑟𝑎𝑖𝑛 = 20 training-parameter
instances, 𝑁𝑡𝑒𝑠𝑡 = 10 testing-parameter instances and 𝑁𝑣𝑎𝑙 = 10
validating-parameter instances uniformly distributed in the parameter
Finally, we extend the rPOD-ANNs model order reduction method to space . Considering the HF data has a large spatial degree of freedom
the 3D cylinder flow problem mentioned in [47,48], and for which the 𝑁ℎ , we first use rPOD method to perform prior ( dimension ) reduction
conservation equations of mass and momentum are written as follows for the HF data so that 𝑁 = 80 can satisfy 𝜖𝑟𝑒𝑙 ̂ 𝐲𝑗𝑘 , 𝐕𝑁 𝐕T𝑁 ̂
𝐲𝑗𝑘 ≈ 10−3 .
Here, L-BFGS algorithm is also used for pre-training, when the error loss
⎧ 𝜕𝐮
𝜌 + 𝜌𝐮 ⋅ ∇𝐮 − ∇ ⋅ 𝝈(𝐮, 𝑝) = 𝟎, (𝐱, 𝑡) ∈ 𝛺 × (0, 𝑇 ) did not decrease within 500 epochs, the early stop strategy is adopted,
⎪ 𝜕𝑡
⎪ ∇ ⋅ 𝐮 = 0, (𝐱, 𝑡) ∈ 𝛺 × (0, 𝑇 ) and the loss error changes in the training process are shown in Fig. 14.
⎪ 𝐮 = 𝟎, (𝐱, 𝑡) ∈ 𝛤1 × (0, 𝑇 ) The weights and bias values of the neural network obtained by L-BFGS
⎨ (32)
⎪ 𝐮 = 𝐡, (𝐱, 𝑡) ∈ 𝛤2 × (0, 𝑇 ) algorithm were taken as the initial values, then LM algorithm was used
⎪ 𝝈(𝐮, 𝑝)𝐧 = 𝟎, (𝐱, 𝑡) ∈ 𝛤𝑁 × (0, 𝑇 ) for training. In the training process, network parameters were updated
⎪ every two time steps in order to speed up the off-line training stage.
⎩ 𝐮(0) = 𝟎, 𝐱 ∈ 𝛺, 𝑡 = 0.

11
F. Cheng et al. Computers and Fluids 241 (2022) 105481

Fig. 15. Solution snapshot of (r)POD-ANNs method for the parameter instances 𝜇 = 2.4832(𝑅𝑒 = 110) at 𝑧 = 0.2, 𝑡 = 5 (left) and 𝑧 = 0.05, 𝑡 = 8 (right).

Fig. 16. Relative error for the testing-parameter instances 𝜇 = 2.4832 at 𝑧 = 0.2, 𝑡 = 5 (left) and 𝑧 = 0.05, 𝑡 = 8 (right).

By observing the performance of the neural network loss function References

on the verification set, considering the dimension of the reduced state
variable is 𝑛 = 8, and the neural network with 𝐹𝑙𝑎𝑦𝑒𝑟 = 26 and 𝐺𝑙𝑎𝑦𝑒𝑟 = [1] Schilders W. Introduction to model order reduction. Springer; 2008.
[2] Quarteroni A, Manzoni A, Negri F. Reduced basis methods for partial differential
12, the numerical results of the rPOD-ANNs model reduction method
equations: An introduction. Springer; 2015.
are shown in Fig. 15.
( Compared) with its HF snapshot in Fig. 13, its [3] Kunisch K, Volkwein S. Galerkin proper orthogonal decomposition methods for
𝐲𝑗𝑘 , ̃
error indicator 𝜖𝑎𝑏𝑠 ̂ 𝐲𝑗𝑘 is shown in Fig. 16. a general equation in fluid dynamics. Soc Ind Appl Math 2002;40(2):492–515.
[4] Abbasi F, Mohammadpour J. Nonlinear model order reduction of Burgers’
equation using proper orthogonal decomposition. In: Proceedings of the American
5. Conclusions control conference. http://dx.doi.org/10.1109/ACC.2015.7170798.
[5] Benner P, Gugercin S, Willcox K. A survey of projection-based model re-
duction methods for parametric dynamical systems. Soc Ind Appl Math
In this paper, we proposed a non-intrusive data-driven (r)POD-ANNs
2015;57(4):483–531.
model order reduction method. We first perform a prior dimension [6] Xiao D. Error estimation of the parametric non-intrusive reduced order model
reduction for the high-precision numerical solution, and then implicitly using machine learning. Comput Methods Appl Mech Engrg 2019;355:513–34.
construct the nonlinear mapping function between the parameter space [7] Xiao D, Fang F, Buchan AG, Pain CC, Navon IM, Muggeridge A. Non-intrusive
reduced order modelling of the Navier–Stokes equations. Comput Methods Appl
and the generalized coordinates of (r)POD using neural network func-
Mech Engrg 2015;293:522–42.
tions. By transforming a class of model reduction problems into a class [8] Xiao D, Fang F, Buchan AG, Pain CC, Navon IM, Du J, et al. Non-linear model
of constrained optimization problems, the low-order approximation of reduction for the Navier–Stokes equations using residual DEIM method. J Comput
HF models with certain accuracy can be obtained through optimiza- Phys 2014;263:1–18.
[9] Xiao D, Fang F, Pain CC, Navon IM. A parameterized non-intrusive reduced
tion solution. Some optimization strategies are adopted to accelerate
order model and error analysis for general time-dependent nonlinear partial
the off-line reconstruction of low order systems. Then, through the differential equations and its applications. Comput Methods Appl Mech Engrg
computational performance and numerical accuracy of (r)POD-ANNs 2017;317:868–89.
model order reduction method for several time-dependent problems [10] Raissi M, Perdikaris P, Karniadakis GE. Physics-informed neural networks: A deep
are evaluated, we find that the (r)POD-ANNs model order reduction learning framework for solving forward and inverse problems involving nonlinear
partial differential equations. J Comput Phys 2019;378:686–707.
method has some advantages both in off-line computing time and [11] Jagtap AD, Kharazmi E, Karniadakis GE. Conservative physics-informed neural
in maintaining certain accuracy, especially for large-scale numerical networks on discrete domains for conservation laws: Applications to forward and
computation. inverse problems. Comput Methods Appl Mech Engrg 2020;365:113028.
[12] Jagtap AD, Karniadakis GE. Extended physics-informed neural networks
(XPINNs): A generalized space–time domain decomposition based deep learning
Declaration of competing interest framework for nonlinear partial differential equations. Commun Comput Phys
2020;28:2002–41.
[13] Berg J, Nyström K. A unified deep artificial neural network approach to partial
The authors declare that they have no known competing finan-
differential equations in complex geometries. Neurocomputing 2018;317:28–41.
cial interests or personal relationships that could have appeared to [14] Brink AR, Najera-Flores DA, Martinez C. The neural network collocation method
influence the work reported in this paper. for solving partial differential equations. Neural Comput Appl 2021;33:5591–608.
[15] Chen W, Wang Q, Hesthaven JS, Zhang C. Physics-informed machine learning for
reduced-order modeling of nonlinear problems. J Comput Phys 2021;446:110666.
Acknowledgments [16] Wang R, Kashinath K, Mustafa M, Albert A, Yu R. Towards physics-informed deep
learning for turbulent flow prediction. Association for Computing Machinery;
This work is supported by the Natural Science Foundation of China 2020, http://dx.doi.org/10.1145/3394486.3403198.
[17] Rudy SH, Brunton SL, Proctor JL, Kutz JN. Data-driven discovery of partial
(No. U19A2079, 11671345, 91630205, 91852106 and 92152301) and
differential equations. Sci Adv 2017;3(4):1602614.
the Research Fund from Key Laboratory of Xinjiang Province (No. [18] Raissi M, Perdikaris P, Karniadakis GE. Machine learning of linear differential
2020D04002). equations using Gaussian processes. J Comput Phys 2017;348:683–93.

12
F. Cheng et al. Computers and Fluids 241 (2022) 105481

[19] Audouze C, Vuyst FD, Nair PB. Nonintrusive reduced-order modeling of [33] Frescaa S, Manzonia A. POD-DL-ROM: Enhancing deep learning-based re-
parametrized time-dependent partial differential equations. Numer Methods duced order models for nonlinear parametrized PDEs by proper orthogonal
Partial Differential Equations 2013;29(5):1587–628. decomposition. 2021, arXiv:2101.11845v1.
[20] Wu K, Xiu D. Numerical aspects for approximating governing equations using [34] Pagani S, Manzoni A, Quarteroni A. Numerical approximation of parametrized
data. J Comput Phys 2019;384:200–21. problems in cardiac electrophysiology by a local reduced basis method. Comput
[21] Arridge S, Maass P, Öktem O, Schönlieb CB. Solving inverse problems using Methods Appl Mech Engrg 2018;340:530–58.
data-driven models. Acta Numer 2019;28:1–174. http://dx.doi.org/10.1017/ [35] Kou J, Zhang W. Data-driven modeling for unsteady aerodynamics and
S0962492919000059. aeroelasticity. Prog Aerosp Sci 2021;125:100725.
[22] Hesthaven JS, Ubbiali S. Non-intrusive reduced order modeling of nonlinear [36] Hickner MK, Fasel U, Nair AG, Brunton BW, Brunton SL. Data-driven unsteady
problems using neural networks. Art J Comput Phys 2018;363:55–78. aeroelastic modeling for control. 2022, https://arxiv.org/abs/2111.11299.
[23] Guo M, Hesthaven JS. Data-driven reduced order modeling for time-dependent [37] Wang Z, Zhang W, Wu X, Chen K. A novel unsteady aerodynamic reduced-
problems. Comput Methods Appl Mech Engrg 2019;345:75–99. order modeling method for transonic aeroelastic optimization. J Fluids Struct
[24] Mücke NT, Bohté SM, Oosterlee CW. Reduced order modeling for parameterized 2018;82:308–28.
time-dependent PDEs using spatially and memory aware deep learning. J Comput [38] Kou J, Zhang W. Layered reduced-order models for nonlinear aerodynamics and
Sci 2021;53:101408. aeroelasticity. J Fluids Struct 2017;68:174–93.
[25] Salvador M, Dede L, Manzoni A. Non intrusive reduced order modeling of [39] Peherstorfer B, Gugercin S, Willcox K. Data-driven reduced model construction
parametrized PDEs by kernel POD and neural networks. Comput Math Appl with time-domain Loewner models. SIAM J Sci Comput 2017;39(5):A2152–78.
2021;104:0898–1221. [40] Nelles O. Nonlinear system identification: From classical approaches to neural
[26] Fresca S, Manzoni A. Real-time simulation of parameter-dependent fluid flows networks, fuzzy models, and Gaussian processes. Springer; 2021.
through deep learning-based reduced order models. Fluids 2021. http://dx.doi. [41] Halko N, Martinsson PG, Tropp JA. Finding structure with randomness: Prob-
org/10.3390/fluids6070259. abilistic algorithms for constructing approximate matrix decompositions. SIAM
[27] Wu P, Sun J, Chang X, Zhang W, Arcucci R, Guo Y, et al. Data-driven reduced Rev 2011;53(2):217–88.
order model with temporal convolutional neural network. Comput Methods Appl [42] Negri F. A model order reduction framework for parametrized nonlinear PDE
Mech Engrg 2020;360:112766. constrained optimization. Mathematics Institute of Computational Science and
[28] Wang Q, Hesthaven JS, Ray D. Non-intrusive reduced order modeling of unsteady Engineering; 2015, Preprint.
flows using artificial neural networks with application to acombustion problem. [43] Goodfellow I, Bengio Y, Courville A. Deep learning. The MIT Press; 2016,
J Comput Phys 2019;384:289–307. http://www.deeplearningbook.org.
[29] Regazzoni F, Dedè L, Quarteroni A. Machine learning for fast and re- [44] Bottou L, Curtis FE, Nocedal J. Optimization methods for large-scale machine
liable solution of time-dependent differential equations. J Comput Phys learning. Soc Ind Appl Math 2018;60(2):223–311.
2019;397:108852. [45] Zhang J, Yan G. Lattice Boltzmann method for one and two-dimensional Burgers’
[30] Regazzoni F, Dedè L, Quarteroni A. Machine learning of multiscale active equation. Physica A 2008;387:4771–86.
force generation models for the efficient simulation of cardiac electromechanics. [46] Jia J, Zhang H, Xu H, Jiang X. An efficient second order stabilized scheme
Comput Methods Appl Mech Engrg 2020;370:113268. for the two dimensional time fractional Allen-Cahn equation. Appl Numer Math
[31] Frescaa S, Dedè L, Manzonia A. A comprehensive deep learning-based approach 2021;165:216–31.
to reduced order modeling of nonlinear time-dependent parametrized PDEs. J [47] Schäfer M, Turek S, Durst F, Krause E, Rannacher R. Benchmark computations
Sci Comput 2021;87:61. of laminar flow around a cylinder. In: Flow simulation with high-performance
[32] Fresca S, Manzoni A, Dedè L, Quarteroni A. Deep learning-based reduced order computers II. 1996.
models in cardiac electrophysiology. PLoS One 2020;15(10):0239416. [48] Bayraktar E, Mierka O, Turek S. Benchmark computations of 3D laminar flow
around a cylinder with CFX, openFOAm and featflow. Int J Comput Sci Eng
2012;7:253–66.

13