Data-driven Nonlinear Parametric Model Order

Reduction Framework using Deep Hierarchical

Variational Autoencoder
SiHun Lee1 , Sangmin Lee1 , Kijoo Jang1 , Haeseong Cho2 ,
SangJoon Shin1,3*
1 Department of Aerospace Engineering, Seoul National University,
Seoul, 08226, Republic of Korea.
arXiv:2307.06816v1 [cs.LG] 10 Jul 2023

2 Department of Aerospace Engineering, Jeonbuk National University,

Jeonju, 54896, Republic of Korea.

3* Institute of Advanced Aerospace Technology, Seoul National

University, Seoul, 08226, Republic of Korea.

*Corresponding author(s). E-mail(s): [email protected];

Contributing authors: [email protected]; [email protected];
[email protected]; [email protected];

Abstract
A data-driven parametric model order reduction (MOR) method using a deep
artificial neural network is proposed. The present network, which is the least-
squares hierarchical variational autoencoder (LSH-VAE), is capable of performing
nonlinear MOR for the parametric interpolation of a nonlinear dynamic system
with a significant number of degrees of freedom. LSH-VAE exploits two major
changes to the existing networks: a hierarchical deep structure and a hybrid
weighted, probabilistic loss function. The enhancements result in a significantly
improved accuracy and stability compared against the conventional nonlinear
MOR methods, autoencoder, and variational autoencoder. Upon LSH-VAE, a
parametric MOR framework is presented based on the spherically linear interpo-
lation of the latent manifold. The present framework is validated and evaluated
on three nonlinear and multiphysics dynamic systems. First, the present frame-
work is evaluated on the fluid-structure interaction benchmark problem to assess
its efficiency and accuracy. Then, a highly nonlinear aeroelastic phenomenon,
limit cycle oscillation, is analyzed. Finally, the present framework is applied to a
three-dimensional fluid flow to demonstrate its capability of efficiently analyzing
a significantly large number of degrees of freedom. The performance of LSH-VAE

1
 is emphasized by comparing its results against that of the widely used nonlinear
MOR methods, convolutional autoencoder, and β-VAE. The present framework
exhibits a significantly enhanced accuracy to the conventional methods while still
exhibiting a large speed-up factor.

Keywords: Machine Learning, Nonlinear Model Order Reduction, Multiphysics

Analysis, Variational Autoencoder, Parametric Interpolation

1 Introduction
Modern high-fidelity, nonlinear computational analysis is mostly computationally
intensive in terms of time and memory. In particular, many multiphysics analysis
adopt a partitioned method in which the solvers regarding each type of physics are
executed separately. Such an approach also requires computation for the data inter-
polation among different types of discretization and executes iterative computation
within a single time step, demanding even more intensive computation. Consequently,
model order reduction (MOR) has been suggested to alleviate the computational time
and memory consumption. Two types of MOR frameworks exist: intrusive and non-
intrusive. Intrusive MOR depends on the governing equation to construct the reduced
bases. Galerkin projection is one of the most widely used approaches which projects
an ensemble of the full-order model (FOM) results into the governing equation [1, 2].
However, a parametric analysis may become extremely challenging when the algo-
rithm is not explicitly established as it manipulates the governing equation directly
[3]. Instead, a completely data-driven approach, non-intrusive MOR (NIMOR) may
be considered. NIMOR aims to discover the embedded pattern in the FOM dataset
and rescale those to a much smaller dimensionality. Unlike intrusive MOR, NIMOR
is independent of the governing equation, making it to be extremely versatile.
Among MOR methods, linear subspace MOR (LS-MOR) has been widely consid-
ered as they are mathematically rigorous and efficient. LS-MOR has been successfully
employed in fluid dynamics, flow control, structural dynamics, aeroelasticity, and fluid-
structure interaction (FSI) [4–11]. However, LS-MOR may require an excessive number
of the subspaces to accurately represent a nonlinear, complex FOM. For example, in
complex turbulent fluid flows, proper orthogonal decomposition (POD) extracts its
modes with respect to the energy ratio and details are filtered out [12]. Those details
are usually excluded because they contain very small energy and the correspond-
ing coefficients are quite random. LS-MOR methods are generally known to be less
effective on advection-dominated, sharp-gradient, multiphysics systems, and especially
systems with slowly decaying Kolmogorov n-width [12–15].
Recent exponential development in the field of machine learning has enabled neural
networks to be used for MOR. Specifically, autoencoder has become a viable nonlin-
ear MOR method where a shallow, well-trained autoencoder with a linear activation
function is known to behave similarly to POD [16–18]. Instead of the linear activa-
tion functions, many autoencoders adopt nonlinear activation functions, using them
to generate nonlinear subspace [17, 18]. Such an autoencoder-based method has been

2
implemented widely to reduce the dimensionality of various engineering problems
including fluid dynamics, convection problems, and structural dynamics [14, 19–24].
However, the performance of an autoencoder as a generative ANN is known to be
quite limited [25]. The deterministic aspect of its loss function, which was designed to
only reconstruct the input, limits autoencoders to generate diverse outputs. Attempts
to enhance the generative capability have led to the development of the variational
autoencoder (VAE) and generative adversarial network (GAN) [26, 27]. These meth-
ods implement probabilistic loss functions that construct a dense and smooth latent
space. Between the two alternatives, VAE is selected for use in this study owing to
its stable training property [10]. VAE has been widely studied for use in the field of
computer vision but it has also been used to interpolate dynamic systems [10, 11].
VAE in its simplest form, vanilla VAE, is capable of generating data of signif-
icantly superior quality compared with the autoencoder. However, VAE commonly
suffers from a phenomenon known as posterior collapse, where the generative model
learns to ignore a subset of the latent variables [28]. The posterior collapse was easily
alleviated by applying a technique known as Kullback-Leibler divergence (KL diver-
gence) annealing, or β-VAE [29–34]. Another problem with vanilla VAE is that it is
restricted to a shallow network, limiting its expressiveness. Vanilla VAE tends to per-
form worse as the network becomes deeper due to the loss of long-range correlation
and its performance was found to be insufficient when complex data were processed
[10, 32]. Deep hierarchical VAEs, such as the LVAE, IAF-VAE, and NVAE, have been
developed to enhance the performance of vanilla VAE [30, 32, 35]. These VAEs mainly
adopt a type of residual cells that connect the encoder and decoder directly without
passing through the latent space. Similar to U-nets, the skip connections allow bidi-
rectional information sharing between the encoder and decoder, thereby preventing
the loss of long-range correlation.
Recently, various types of VAEs are being adopted as a nonlinear MOR method
owing to their superior generative capability compared to conventional autoencoders.
VAEs have been adopted on flow problems [36–39], transonic flow [40, 41], numerics
[42], biology [43], brain MRI images [44], and anomaly detection [45, 46]. While earlier
studies adopt the simplest convolutional VAE, many recent studies consider β-VAE
due to its near-orthogonal latent space [33, 34]. Previous studies show that β-VAE
may successfully construct nonlinear subspace, but the majority of networks used in
those studies were quite shallow. The use of shallow networks may result in insufficient
expressiveness if the input data consists of a large number of DOF and exhibits a
complex response.
Instead, a deep hierarchical VAE is proposed, least-squares hierarchical VAE (LSH-
VAE) for nonlinear MOR of a dynamic system. LSH-VAE is a very deep hierarchical
network that incorporates a modified loss function similar to that of β-VAE. The deep
hierarchical structure enables a very deep, stable network (>100 layers) with highly
expressive and accurate interpolation results. The modified loss function consists of a
hybrid weighted least-squares and Kullback-Leibler divergence function that alleviates
posterior collapse and enhances orthogonality of the latent space [33, 34, 47]. The
least-squares error in the loss function is also known to enhance the accuracy when
used on the continuous dataset [11].

3
 There has been no report on a very deep VAE (>100 layers) implemented for
nonlinear MOR. The present framework is validated by solving the following three
problems. First, a standard two-dimensional FSI benchmark problem developed by
Turek and Hron will be exemplified [48]. Then, the highly nonlinear aeroelastic phe-
nomenon of limit cycle oscillation (LCO) will be considered to examine the accuracy
of the proposed framework under nonlinearity. Finally, the flow surrounding a three-
dimensional cylinder is to be analyzed to establish the capability of the current
framework to accommodate a system with a significantly large number of degrees
of freedom. The computational efficiency and accuracy will be assessed as well as
comparison to the existing nonlinear MOR methods will be presented.

2 Machine-Learning Methods
This section provides the theoretical background of the machine learning methods.
Based on the existing convolutional autoencoder and β-VAE, the formulation of the
proposed network, LSH-VAE is presented.

2.1 Convolutional autoencoder (CAE)

A convolutional autoencoder (CAE) is an ANN that is trained to output data that
are similar to its input. The typical architecture of the CAE, shown in Fig. 1, enables
the encoder to compress the input data into a smaller latent dimensionality. The
decoder then expands the latent code back to its original dimensionality. By training
both the encoder and decoder, CAE learns to extract important features of the input
dataset. The latent codes contain the embedded features recognized by the CAE that
can be used as the reduced bases in the ROM. The interpolation of data using CAE
is conducted by interpolating the latent codes. The interpolated latent code contains
the interpolated features, which leads to the interpolation of the input data.

Latent
code

Input Output
𝒙𝒙 �
𝒙𝒙

Encoder Decoder

Fig. 1: Architecture of a typical convolutional autoencoder

The loss function of CAE is quite intuitive. CAE takes the input, x, and passes it
through the encoder, Φ, to obtain the latent vector, z. Then, the decoder, Ψ, receives
the latent vector and generates the output, y. The output, y, is compared against the
input, x, using the mean squared error (M SE) loss function. In this way, the CAE is
trained such that the difference between y and x is reduced, aiming for a more accurate

4
reconstruction of the input. The equations for the encoder and decoder network are
presented in Eq. (1), where the loss function is shown in Eq. (2).

z = Φ(x), y = Ψ(z) (1)

L = M SE(Ψ(Φ(x)) − x) (2)
The simplest form of CAE, known as the vanilla CAE, has been shown to produce
unsatisfactory interpolation outcomes [25]. Hence, derivatives thereof such as VAE,
and GAN may be utilized to enhance the performance.

2.2 Variational autoencoder (VAE)

VAE and autoencoder share a similar architecture. The largest difference lies in that
the encoder of VAE utilizes probabilistic latent values instead of discrete latent codes.
The probabilistic encoder models the latent feature probability distribution. The
resultant latent space is continuous and smooth, enabling higher quality generated
outcomes. The encoder of VAE extracts the mean, µ, and the variance, σ, which are
used to generate the latent code, z. A typical VAE structure can be observed in Figure
2.

𝜺𝜺
Latent
code
𝝈𝝈
Input Output
𝒙𝒙 �
𝒙𝒙
𝝁𝝁

Encoder Decoder

Fig. 2: Architecture of a typical VAE

VAE aims to efficiently infer the intractable posterior distribution, p(z|x). It is

performed by adopting an approximate posterior, q(z|x), because determining the true
posterior is quite challenging. Here, the encoder or inference network is represented
by q(z|x), whereas the decoder network is denoted as p(x|z).
Kullback-Leibler (KL) divergence is the expectation of the difference between two
distributions, which is always a positive value. KL divergence between the approximate
and the real posterior is written as Eq. (3).
Z
p(z|x)
DKL (q(z|x)||p(z|x)) = − q(z|x) log( )dz ≥ 0 (3)
q(z|x)

5
 Applying Bayes’ theorem to Eq. (3) yields Eq. (4).
Z
p(x|z)p(z)
DKL (q(z|x)||p(z|x)) = − q(z|x) log( )dz
q(z|x)p(x)
Z (4)
p(x|z)p(z)
= − q(z|x) log( )dz + log p(x) ≥ 0
q(z|x)
Equation (4) can be rewritten as Eq. (5). Applying the rules of logarithm to Eq.
(5) will yield Eq. (6).
Z
p(x|z)p(z)
log p(x) ≥ q(z|x) log dz (5)
q(z|x)
Z Z
p(z)
log p(x) ≥ q(z|x) log( )dz + q(z|x) log p(x|z)dz
q(z|x) (6)
≥ Eq(z|x) [log p(x|z)] − DKL (q(z|x)||p(z))
The right hand side of Eq. (6) is the evidence lower bound (ELBO). VAE aims to
maximize ELBO which maximizes the logarithmic probability of the data by proxy.
Following the convention of minimizing the loss function, the right hand side of Eq.
(6) is converted as Eq. (7), which is the goal of VAE.

min[−Eq(z|x) [log p(x|z)] + DKL (q(z|x)||p(z))] (7)

The goal of VAE is to minimize both the reconstruction and KL divergence loss. In
Eq. (7), the first term corresponds to the reconstruction loss and the second term cor-
responds to KL divergence loss. KL divergence loss enforces the decoder (approximate
posterior) to become similar to the inverse of the encoder.
The loss function in Eq. (7) has to be differentiable to minimize it during the
training. Usually, KLD term can be integrated analytically [26]; however, the recon-
struction loss is not directly differentiable. To enforce the reconstruction loss to be
differentiable, the reparameterization technique is adopted [26].
First, Gaussian sampled random noise, ε will be introduced. The latent code z, is
formulated as shown in Eq. (8), introducing the mean and standard deviation to the
equation.

z = µ + (σ × ε), ε ∼ N (0, 1) (8)

Since the latent code is formulated as Eq. (8), KL divergence in Eq. (7) is rewritten
as Eq. (9), assuming the posterior and prior follow the Gaussian distribution.
1X 2
DKL (q(z|x)||p(z)) = (σ + µ2 − (log(σ 2 ) + 1)) (9)
2
The latent code with the reparameterization technique enforces the latent space
to be stochastically determined. The reparameterization enables the reconstruction
loss to be differentiable by Monte Carlo method. For further details and step-by-step
derivation of the VAE loss function, reference can be found in works by Kingma and
Odaibo [26, 49].

6
2.3 Least-squares hierarchical variational autoencoder
(LSH-VAE)
Conventional vanilla VAE is limited to shallow networks due to vanishing gradients and
the loss of long-range correlation. However, shallow networks may lack expressiveness
on complex systems with a significant number of DOFs. In this study, a deep VAE
with a hierarchical structure is proposed to enhance the performance. Specifically,
to alleviate the loss of long-range correlation and stabilize the training process of
a very deep network. The hierarchical structure creates direct passages between the
earlier layers of the encoder and the latter layers of the decoder, circumventing the
middle layers. Those direct passages enable bidirectional information sharing between
the encoder and decoder network. The bidirectional information enables the earlier
layers of the VAE to greatly affect the outcome, thus, alleviating the loss of long-range
correlation. The diagram in Fig. 3 shows the hierarchical structure of LSH-VAE.

sampling
𝒛𝒛𝑳𝑳 𝑳𝑳 − 𝟏𝟏𝒕𝒕𝒕𝒕 group
Bi-directional
information

𝒛𝒛𝑳𝑳−𝟏𝟏 top-down

sampling
bottom-up

⋮ ⋮ ⋮
𝟐𝟐𝒏𝒏𝒏𝒏 group
Bi-directional
information

𝒛𝒛𝟐𝟐
𝟏𝟏𝒔𝒔𝒔𝒔 group
sampling Bi-directional
information

𝒛𝒛𝟏𝟏

sampling

𝒙𝒙 �
𝒙𝒙

Fig. 3: Hierarchical structure of LSH-VAE

7
 In the hierarchical VAE, the latent variables are divided into L groups. By the
divided latent dimension, the prior and posterior distributions are rewritten as in Eq.
(10) and Eq. (11).
L−1
Y
p(z) = p (zL ) p (zi |zi+1 )
i=1
(10)
L
Y
q(z|x) = q (z1 |x) q (zi |zi−1 )
i=2

p(zi |zi+1 ) = N (zi |µ(zi+1 ), σ 2 (zi+1 ))

p(zL ) = N (zL |0, I)
(11)
q(zi |zi−1 ) = N (zi |µ(zi−1 ), σ 2 (zi−1 ))
q(z1 |x) = N (z1 |µ(x), σ 2 (x))
The loss function for hierarchical VAE is shown in Eq. (12), which is obtained
by computing the KL divergence separately for each group. By breaking down the
KL divergence into groups, bidirectional information flows are created between the
inference and generative network. Detailed descriptions about the deep hierarchical
structure of VAE can be found in [30].

min[−Eq(z|x) [log p(x|z)] + DKL (q(z|x) |p(z))

L−1
X (12)
+ Eq(z<i |x) [DKL (q(zi |z<i , x) |p(zi |z>i ))]]
i=1
The present LSH-VAE adopts hierarchical structures motivated by LVAE, IAF-
VAE, and NVAE [30, 32, 35]. The latent codes in the hierarchical VAE are formed
by both bottom-up and top-down information. The latent codes of each of the groups
output shared information (from the encoder and decoder) to the next decoder block.
Because the information of the encoder and decoder network is shared via latent code,
the network delivers higher performance.
Upon the hierarchical structure, LSH-VAE implements a hybrid weighted loss func-
tion. The loss function consists of the mean squared error (MSE) and KL divergence
instead of conventional binary cross entropy. The use of MSE as a reconstruction error
has been known to be successful for continuous datasets [11]. The loss function of
LSH-VAE is shown in Eq. (13), where the coefficients α and β denote the weights of
the MSE and KL divergence, respectively.

min[αM SE(x, x̃) + βDKL (q(z|x) |p(z))

ϕ,θ
L−1
X (13)
+ Eq(z<i |x) [βDKL (q(zi |z<i , x) |p(zi |z>i ))]]
i=1
Usually, the weights α and β are set to be α/βtarget ≈ 106 . During the training, α is
a fixed value whereas β is a variable that varies with respect to the epochs. The variable
β is implemented to prevent posterior collapse in which some latent variables become

8
inactive. This method is known as KL-annealing or β-VAE, where β is formulated as
Eq. (14) [29].
(
1 × 10−4 βtarget if epoch < 0.3nepochs
β= (14)
βtarget nepoch
epochs
if epoch > 0.3nepochs
During the training, β is assigned a low value at the start such that LSH-VAE
behaves as an autoencoder. During the first few epochs, input data will be mapped
on the latent space. Beyond a few prescribed epochs, β will be gradually ramped up
such that LSH-VAE may behave as a VAE, generating smooth latent space.

3 Present Framework
3.1 Architecture of the least-squares hierarchical VAE
(LSH-VAE)
LSH-VAE adopts a one-dimensional (1D) convolutional layer to accommodate the
transient response of the unstructured grids. The use of a 1D convolutional layer
enables the temporal continuity of the physical variables to be considered. The encoder
and decoder of the LSH-VAE consist of the blocks discussed in the previous section,
where a detailed schematic of these blocks is shown in Fig. 4.

(𝒊𝒊 - 1)-th decoder block

Bottom-up info.
Latent code
z𝒊𝒊 Top-down info.
Concat
Reshape Sigmoid Shared info.
SN-TC1D-ELU
Dense Dense

BN
ELU 𝒊𝒊-th decoder block
SN-TC1D-ELU
Conv1D(7) SN
BN
SN TransConv1D(7)
Swish
Swish ELU
𝒊𝒊-th encoder block SN-TC1D-ELU
BN

Dense Dense
Bottom-up info. Latent code
z𝒊𝒊−𝟏𝟏 Top-down
Reshape Sigmoid info.
Top-down info.
(𝒊𝒊 - 1)-th encoder block
Concat
Shared info.
SN-TC1D-ELU

Fig. 4: Detailed architecture of the encoder and decoder blocks of LSH-VAE

9
 Being a deep neural network (DNN), LSH-VAE encoder and decoder blocks are
composed of stacks of multiple layers. These layers consist of the following layers:
spectral normalization (SN), 1D convolution, dense, exponential linear unit (ELU),
Swish, and batch normalization (BN). Swish, and ELU nonlinear activation functions
are chosen as their continuous derivatives enhance the stability of a DNN [50]. The
LSH-VAE implements a normalization-activation sequence instead of the conventional
activation-normalization sequence. Such sequence is known to deliver benign perfor-
mance empirically when used before the convolutional computation [32]. The output
of the encoder block is branched in three ways. The first branch connects to the input
of the next block and the remaining two branches form µ, and σ. The encoder latent
code is formulated by reparameterizing µ, and σ. The reparameterized latent code and
ELU layer infer bottom-up information transfer, shown in green in Fig. 4.
In the current configuration, the decoder network is significantly deeper and more
complex than the encoder network. The deep decoder network enables an expres-
sive output when accompanied by a system with many DOFs. The decoder network
receives two inputs: top-down information from the predecessor decoder block and
encoder-decoder shared information from the latent code. Through a series of layers,
the decoder outputs top-down information, shown in blue. The decoder block gener-
ates the decoder latent code and input for the next block. The encoder latent code
and the decoder latent code are added to generate shared latent code, z i . The shared
latent code contains both top-down and bottom-up information, enabling bidirectional
information sharing.

3.2 Preprocessing dataset

Acquiring many FOM samples may be quite cumbersome. In particular, many-queried
FOM computations are extremely time-consuming if FOM is highly nonlinear, includes
multiphysics, and involves a significant number of DOFs. Acquiring those FOM data
through experiments and simulations is considered prohibitive for computational,
financial reasons. Instead, data augmentation is considered to sample sparsely and
expand the amount of training data. A larger amount of training data improves the
generalization of ANN and thus enhances the accuracy. Similar to the data augmenta-
tion typically performed on images, the pre-acquired FOM results are processed using
the following three methods. First, temporal data are resampled by shortening the
timestep, i.e. frequency elongation. Then, the training data are augmented by chang-
ing the amplitude and adding a random number within the bound of ±30% for every
epoch. Training the ANN using the augmented data ensures that the ANN is effectively
trained against a very large dataset, resulting in a high-performance network.

3.3 LSH-VAE training and interpolation

The current framework performs MOR directly on FOM results. The LSH-VAE
employs 1D convolutional layers which requires a three-dimensional input of the for-
mat (batch, sequence, channel). In the current configuration, the temporal continuity
of the FOM results is considered in the convolutional dimension. The resultant input
composition of LSH-VAE becomes (batch, Nt , NDOF ), where Nt denotes the number

10
of time steps and NDOF denotes the number of DOFs in the dynamic system. LSH-
VAE receives such input and compresses it into latent vectors via the encoder. The
dimensionality change throughout LSH-VAE is expressed in Eq. 15, where Ni repre- P
sents the latent dimension in the i-th latent group. The total latent dimension, Ni
is much smaller than the FOM dimension, achieving MOR.
 X 
encoder decoder
(, Nt , NDOF ) −−−−−→ , Ni −−−−−→ (, Nt , NDOF ) (15)

Algorithm 1 Training of LSH-VAE

Require: Hyperparameters, αLR , β1 , β2 , NA , qϕenc , pdec
θ
x = N (v i ) = v̄ i → [−0.7, 0.7] (i = 1, · · · , NDOF )
xtrain = [x, R(x)1 , R(x)2 , ... , R(x)NA ] (resampling aug.)
for i = 1, Nepoch do
xtrain = 0.3α × (xtrain + 0.3β), α, β ∼ N (0, 1) (amplitude, offset aug.)
for j = 1, L do
if j = 1 then
enc
qϕ,1 (xtrain ) = {u1 , µq,1 , σq,1 }
z1 = µq,1 + σq,1 ε, ε ∼ N (0, 1)
else if 1 < j < L then
enc
qϕ,j (uj−1 ) = {uj , µq,j , σq,j }
zj = µq,j + σq,j ε, ε ∼ N (0, 1)
else if j = L then
enc
qϕ,L (uL−1 ) = {µq,L , σq,L }
end if
end for
zL = µq,L + σq,L × ε, ε ∼ N (0, 1)
for k = L, −1, 1 do
if k = L then
pdec
θ,k (zL ) = {dk , µp,k , σp,k }
dec
zL−1 = µp,L + σp,L ε, ε ∼ N (0, 1)
dec
zL−1 = zL−1 + zL−1
else if 1 ≤ k < L then
pdec
θ,k (dk+1 , zk ) = {dk , µp,k , σp,k }
dec
zk−1 = µp,k + σp,k ε, ε ∼ N (0, 1)
dec
zk−1 = zk−1 + zk−1
end if
end for
LLSH−V AE →Eq. (7), where x = xtrain , x̃ = d1
Adamax(∇ϕ,θ LLSH−V AE , αLR , β1 , β2 ) → ϕ, θ
end for

The training algorithm for LSH-VAE is shown in Algorithm 1. The algorithm starts
by normalizing the physical variables of interest, v. v is normalized to the range of [-0.7,

11
0.7] for each DOF by the normalizing function, N (). The normalized variable is then
augmented by resampling for NA instances. Then, the training dataset, xtrain is con-
structed by concatenating the original normalized variable with the augmented ones.
The training dataset of the network becomes, xtrain = [x, R(x)1 , R(x)2 , ... , R(x)NA ],
where R(x)n denotes the resampled normalized variable of interest.
The training dataset is further augmented for amplitude and offset. The amplitude
and offset augmentation is performed by using random values for every epoch. The
network receives a different input in every epoch, enabling the network to be trained
against a very large dataset. After the data augmentation is completed, the encoder
and the decoder networks are trained. After the decoder is trained, the loss function
can be obtained by Eq. 13. The training of LSH-VAE is optimized by the Adamax
optimizer, which has shown good performance compared with the conventional Adam
and SGD optimizers.
Generative ANNs usually require latent vectors to be sought. This is required
owing to the probabilistic formulation that is used to parameterize the latent vector.
However, we empirically found that sufficient epochs and a small number of parameters
obviate the need for latent searching. In this study, rather than attempting latent
searching, the latent vectors are calculated by the mean value from the encoder network
directly.
Upon acquiring the latent vectors, slerp interpolation is performed to collect the
targeted latent vector. The latent space created by VAEs is in the form of a well-
structured, multi-dimensional hypersphere, which enables complex operation by vector
arithmetic [51]. It is possible since the reparameterization trick introduces Gaussian
random number, which attributes to the vector length and angle in the latent hyper-
sphere. The slerp interpolation shown in Algorithm 2 not only interpolates the rotation
angle of vectors, but it also interpolates the arc length. Such slerp interpolation enables
the latent vectors to be interpolated following the path of the complex latent man-
ifold. The use of slerp interpolation has been widely accepted for performing latent
interpolation [52, 53].

Algorithm 2 Interpolation of latent codes

Require: N () from Algorithm 1, z i
Set the target parametric value, ptgt , and two adjacent values, p1 and p2
Find the ratio, k, such that ptgt = k × p1 + (1 − k) × p2
zptgt = slerp(zp1 , zp2 , k)
= cos−1 (p1 · p2 ) = p1 sin(1−k)θ
sin θ + p2 sin kθ
sin θ
−1 dec
vptgt = N (fϕ zptgt )

4 Numerical results
This section presents the numerical results obtained by the proposed framework. First,
the framework is applied to solve a FSI benchmark problem previously developed by
Turek and Hron [48]. The accuracy of the current method is evaluated and compared
against that obtained by the conventional nonlinear MOR, CAE. Then, the proposed

12
framework is examined on a wing section that undergoes limit cycle oscillation (LCO).
LCO analysis is performed to evaluate the accuracy of the proposed framework on the
nonlinear multiphysics phenomenon. Last, the applicability of LSH-VAE to a system
with many DOFs is demonstrated by analyzing a three-dimensional fluid flow.
The numerical results presented in this paper are obtained by intentionally sam-
pling a small number of initial FOM results. Sparse sampling is performed because
ANN replicating its training data often leads to enough accuracy when the sampling
is performed densely. In addition, sparse sampling is attempted as dense and iterative
computations on a nonlinear system with many DOFs are rather unrealistic.
For all of the results, the same LSH-VAE network is used for each variable of
interest. The hyperparameters used for the training are shown in Table. 1. In Table
1, the first value for the latent dimension criterion denotes the latent dimension in
which the interpolation is performed. The latter value denotes the latent dimension
used for information sharing between the encoder and decoder network. LSH-VAE
used for the following numerical results consists of 7 encoder and decoder blocks, with
a total of 107 layers. While detailed optimization of the hyperparameters would yield
better accuracy, such procedure is not performed to emphasize the generability of the
framework. However, different batch sizes are used considering the number of DOF,
limited by the VRAM of GPU.

Table 1: Hyperparameters of LSH-VAE

Criterion Value Criterion Value
Epochs 5,000 Latent dim. [32, 8]
α 1 × 106 Number of filters [64,32,16,8,4,2,1]
βtarget 1 NA 5
Learning rate 1 × 10−3

For all of the results presented in this paper, computations are carried out on AMD
3950X CPU to obtain the FOM results. ANN are trained using NVIDIA GeForce
GTX 3090 GPU.

4.1 Turek-Hron FSI benchmark

4.1.1 Description of the analysis
The widely accepted FSI benchmark developed by Turek and Hron is described in this
section [48]. The benchmark problem consists of a rigid cylinder with a diameter of 0.1
m and a highly flexible tail. The fluid flowed from the inlet to the outlet with laminar
separation occurring behind the cylinder. Von Kàrmàn vortex street created by the
flow separation excites the tail, which exhibits a large deflection. A hyperbolic inlet
profile is used to consider the no-slip initial wall boundary condition at the upper and
lower computational domain. A detailed schematic regarding the analysis is shown in
Fig. 5.
The current framework requires a few parametric initial FOM samples to extract
the embedded patterns. For Turek-Hron FSI benchmark problem, seven initial FOM

13
 25D
𝑼𝟎 (𝒚) Rigid body No-slip wall

h=0.2D Flexible body

4.1D A
D L=3.5D
Outlet
Inlet No-slip wall

Fig. 5: Analysis setup of the Turek-Hron FSI problem

results are collected. The inflow speed was selected as a parameter and speeds ranging
from 0.7 m/s to 1.3 m/s, in 0.1 m/s intervals were sampled. The FOM samples are
analyzed using Navier-Stokes computational fluid dynamics (CFD) and finite element
method (FEM) two-way FSI analysis provided in the commercial software, ANSYS.
The flow field is discretized by 29,788 CFD nodes and the flexible body is discretized
by 954 FEM nodes.
The ensemble of FOM results is constructed by collecting 2 s of the fully converged
response in intervals of 0.01 s. The pre-acquired FOM ensemble is then subjected
to interpolation by LSH-VAE shown in Table 1. After the training of LSH-VAE is
completed, the latent code is interpolated. In the present case, the target parameter
is selected as the unseen inflow speed of 0.95m/s. The latent code corresponding to
0.95m/s is acquired by the slerp interpolation shown in Algorithm 2. The interpolated
latent code is then decoded by the decoder network where the resultant interpolated
variables are generated.

4.1.2 Accuracy and efficiency

The accuracy of the current framework is assessed by comparing the results of the
ROM against those obtained with the FOM. Five physical variables, dX, dY, u, v, and
p are considered for interpolation in this case. Among them, the first two variables
denote the grid deformation in x- and y-direction. Using the interpolated variables,
the interpolated FSI field will be constructed. The interpolated FSI field and FOM
are shown in Fig. 6.
Evaluation of the results shown in Fig. 6 verifies that the proposed framework is rea-
sonably accurate. Subsequently, the accuracy of LSH-VAE is compared against that of
CAE and β-VAE. For comparison, the CAE and β-VAE networks are constructed using
the same hyperparameters that were used for LSH-VAE. The comparison between
CAE, β-VAE, and LSH-VAE is performed by comparing the extent to which their
results differed from those of FOM. The discrepancy contours of various networks are
shown in Fig. 7. The minimum and maximum of each variable are matched for the
respective variable.
Overall, LSH-VAE exhibits the smallest discrepancy while β-VAE performs the
worst. Interestingly, the regions that exhibit a relatively larger discrepancy are found
to be quite similar for all of the networks. This is caused by the finite number of
latent dimensions considered in the generative networks. Small details of FOM would

14
 (a) LSH-VAE, u (b) FOM, u

(c) LSH-VAE, v (d) FOM, v

(e) LSH-VAE, p (f) FOM, p

Fig. 6: Original and interpolated FSI field for Turek-Hron FSI problem at t = 2 s

have been neglected in the finite latent representation, which lead to the discrepancy
in the similar areas. Another one to note is that the pressure contour of CAE and
β-VAE shows a considerably larger discrepancy compared against that by LSH-VAE.
This is caused by the large variation between the maximum and minimum values of
the pressure. The inability of CAE and β-VAE to generate an expressive output is
considered to be the reason for small details being neglected by large variations.
Then, the efficiency of the proposed framework is assessed. The computational
procedures for the proposed framework comprise four stages and the computational
time required for each stage is listed in Table 2. For Turek-Hron FSI problem, each
FOM query requires 109.0 h whereas the online stage consumes 0.11 h. The proposed
framework therefore exhibits a speed-up factor of 990 for each unseen parametric
estimation. The expected computational time in terms of the number of computations
is shown in Fig. 8.

4.2 Limit cycle oscillations

4.2.1 Description of the analysis
Limit cycle oscillation (LCO) is a nonlinear periodic oscillation with limited amplitude
on an aerodynamic surface. LCO of an aircraft is a highly nonlinear FSI phenomenon
that is caused by nonlinearities in both the fluid and structure. Typical causes of
LCO include flow separation, transonic shock, geometric nonlinearity, and nonlinear

15
 (a) LSH-VAE, dX (b) CAE, dX (c) β−VAE, dX

(d) LSH-VAE, dY (e) CAE, dY (f) β−VAE, dY

(g) LSH-VAE, u (h) CAE, u (i) β−VAE, u

(j) LSH-VAE, v (k) CAE, v (l) β−VAE, v

(m) LSH-VAE, p (n) CAE, p (o) β−VAE, p

Fig. 7: Discrepancy contour of the interpolated FSI field using LSH-VAE, CAE, and
β−VAE at t = 2 s

stiffness of the control surface. For an aircraft, LCO may result in structural fatigue
in the wings, thus requiring high-fidelity analysis for safety. During the design stage
of an aircraft, iterative LCO analysis is performed to satisfy the vibration criterion.
Such parametric LCO analysis is considered to be quite cumbersome and tedious as it
is highly nonlinear and involves many DOFs. In this section, the proposed framework
is used to conduct a simplified nonlinear parametric LCO analysis of a wing section.
The wing section considered in this analysis is derived from that reported by O’Neil
et al. [54]. In it, a two-dimensional wing section was constrained by the pitch and
heave springs as shown in Fig. 9. The pitch and heave stiffnesses are nonlinear in their

16
 Table 2: Computational time requirements for Turek-Hron
FSI problem
Procedure Computational time[h]
FOM 763.1
Offline Algorithm 1 3.51
Total offline stage 766.6
Algorithm 2 <0.01
Online Variable reconstruction/write 0.11
Total online stage 0.11
Total sum 766.7

1200
FOM
LSH-VAE
Computational time[hours]

1000

800

600

400

200

0
0 2 4 6 8 10
Queries
Fig. 8: Computational time in terms of the parametric queries for Turek-Hron FSI
problem

cubic terms, which are expressed in Eq. 16. LCO is caused by the cubic stiffness in
the structure and LCO is observed at the inflow stream speed of 15.5 m/s to 50 m/s.

Kα = 2.57(α + 500α3 )
(16)
Kh = 0.09(h + 2860h3 ))

𝑲𝒉

𝑼∞ = 𝟐𝟎~𝟒𝟓 𝒎/𝒔
𝑲𝜶

𝟎. 𝟐𝟏𝟐𝟖𝒎

Fig. 9: Analysis setup of the LCO

17
 The inflow speed is chosen as the parameter in this analysis. The initial FOM sam-
ples are collected by adjusting the inflow speed from 20 m/s to 45 m/s in increments
of 5 m/s. The relevant flow field is discretized by 19,381 nodes and solved using the
commercial Navier-Stokes solver, ANSYS. The initial FOM samples are obtained by
collecting 2 s of the fully converged response in intervals of 0.01 s. The FOM ensemble
is subjected to MOR and interpolation by LSH-VAE.
After LSH-VAE is trained, the latent code for the desired parameter is acquired
via slerp interpolation. The target parameter is an unseen inflow speed of 32.5 m/s,
and the corresponding latent code is interpolated using Algorithm 2. The interpolated
latent code is then decoded by the decoder and the interpolated FSI field is generated.

4.2.2 Evaluation of accuracy and efficiency

The accuracy of LSH-VAE is assessed by comparing the ROM results against those
produced by FOM. In this case, the five physical variables discussed in the previous
section were considered. The interpolated variables were used to generate the FSI field,
where the interpolated FSI field and FOM are shown in Fig. 10.

(a) LSH-VAE, u (b) FOM, u

(c) LSH-VAE, v (d) FOM, v

(e) LSH-VAE, p (f) FOM, p

Fig. 10: Original and interpolated FSI field for LCO at t = 2 s

In Fig. 10, the interpolated FSI field constructed by LSH-VAE is found to be

accurate. Then, the accuracy of LSH-VAE is compared against that of CAE and β-
VAE. The discrepancy contours between LSH-VAE, CAE, and β-VAE are shown in

18
Fig. 11. The minimum and maximum of the variable are each matched for the same
variable.

(a) LSH-VAE, dX (b) CAE, dX (c) β−VAE, dX

(d) LSH-VAE, dY (e) CAE, dY (f) β−VAE, dY

(g) LSH-VAE, u (h) CAE, u (i) β−VAE, u

(j) LSH-VAE, v (k) CAE, v (l) β−VAE, v

(m) LSH-VAE, p (n) CAE, p (o) β−VAE, p

Fig. 11: Discrepancy contour of the interpolated FSI field using LSH-VAE, CAE, and
β−VAE at t = 2 s

19
 Similar to Turek-Hron problem, LSH-VAE exhibits the smallest discrepancy. How-
ever in this case, β-VAE performed better than CAE. For dX, all networks exhibit a
similar discrepancy, as the wing section is constrained in x-direction. Only the pitching
motion affects the deformation of surrounding grids in x-direction, resulting in a small
variation. dY , however, shows different behavior. The discrepancy is spread evenly
as the wing heaves and LSH-VAE shows a significantly reduced discrepancy. Another
important point to note is that the discrepancy regarding the pressure is quite small.
This is due to the stagnation point which creates a concentrated high-pressure region.
The efficiency of the proposed framework is also assessed. The computational
time required for each stage is summarized in Table 3. The offline FOM computation
required 280.1 h including six initial FOM sample computations. LSH-VAE training
required 3.52 h for the five variables of interest, resulting in a total offline stage of
283.6 h. For the online stage, FSI field reconstruction and saving to disk requires the
most time as it requires 0.06 h. The present framework exhibits a speed-up factor of
660 for each unseen parametric estimation. The expected computational time in terms
of the unseen parametric queries is shown in Fig. 16.

Table 3: Computational time requirements for LCO

Procedure Computational time[h]
FOM 280.1
Offline Algorithm 1 3.52
Total offline stage 283.6
Algorithm 2 <0.01
Online Variable reconstruction/write 0.06
Total online stage 0.06
Total sum 283.7

500
FOM
LSH-VAE
400
Computational time[hours]

300

200

100

0
0 2 4 6 8 10
Number of computations
Fig. 12: Computational time in terms of the parametric queries for LCO

20
4.3 Three-dimensional fluid flow
4.3.1 Description of the analysis
Finally, fluid flow surrounding a simple stationary three-dimensional (3D) cylinder is
analyzed. The analysis of the 3D fluid serves to demonstrate the use of the proposed
framework to analyze a system with a significant number of DOFs. A 3D cylinder with
a diameter of 1 m was subjected to a uniform inflow, as shown in Fig. 13. Similar to
Turek-Hron FSI benchmark, a von Kàrmàn vortex is formed behind the cylinder. For
CFD analysis, a cuboid computational domain of 20m × 10m × 10m was discretized
into 1,121,000 tetrahedral elements. The Reynolds number of the inflow varied from
100 to 160 in intervals of 10.

10D

Re = 100~160

10D

20D

Fig. 13: Analysis setup of the 3D fluid flow

The initial FOM samples are obtained by using the ANSYS Navier-Stokes solver
and 2s of FOM data are collected in intervals of 0.01 s. Then, the LSH-VAE is
trained against the FOM ensemble and interpolation is performed with respect to the
parameter.
After LSH-VAE is trained, the latent code representing the targeted parameter is
acquired. The target parameter is selected as an unseen inflow Reynolds number of
Re = 125. The latent code corresponding to Re = 125 is acquired by the interpolation
shown in Algorithm 2. The interpolated latent code is then decoded and the resultant
interpolated flow field is generated.

4.3.2 Evaluation of the accuracy and efficiency

The accuracy of LSH-VAE is assessed by comparing the results of ROM with those
obtained using FOM. In this case, four physical variables, u, v, w, and p are considered
for the interpolation. Using the interpolated variables, the interpolated flow field is
generated. The interpolated and original flow fields are displayed in Fig. 14.
The interpolated flow field constructed by LSH-VAE is found to be quite accurate.
Particularly, the velocity in z-direction, w, is accurately interpolated even though w
exhibits quite a complex response. As the initial physical variables are interpolated

21
 (a) LSH-VAE, u (b) FOM, u

(c) LSH-VAE, v (d) FOM, v

(e) LSH-VAE, w (f) FOM, w

(g) LSH-VAE, p (h) FOM, p

Fig. 14: Original and interpolated FSI field for 3D fluid flow at t = 2 s

well, the relationship between the variables is inspected. Comparison against CAE and
β-VAE is not conducted in this case as the large number of DOF caused instability of
the networks. Instead, the normalized Q-criterion is considered to assess whether the
interpolated flow field preserves its vorticity. In Fig.15, the normalized Q-criterion is
obtained using the interpolated variables shown in Fig. 14. Figure 15 shows the iso-
surface generated based on the normalized Q-criterion. The iso-surface is colored by
u-velocity and pressure for visualization.
The good agreement in terms of the Q-criterion indicates that LSH-VAE interpo-
lates the direct variables sufficiently well such that the relationship between variables
may be well preserved.
Lastly, the efficiency of the present framework is assessed. The computational time
required for each stage is listed in Table 4. The offline FOM computation requires 193.7
h including the seven initial FOM samples. LSH-VAE training requires 11.3 h resulting
in a total offline stage of 205.0 h. For the online stage, variable reconstruction and
writing to disk requires the most time as it required 2.02 h. The proposed framework

22
 (a) LSH-VAE, Q = 0.02,u

(b) FOM, Q = 0.02,u

(c) LSH-VAE, Q = 0,p

(d) FOM, Q = 0,p

Fig. 15: Original and interpolated flow fields: iso-surface of 3D fluid flow at t = 1.5s

exhibits a speed-up factor of 14 for each unseen parametric estimation. The expected
computational time in terms of queries is as shown in Fig. 16.

5 Conclusions
This paper proposes a nonlinear data-driven parametric MOR framework based on
a neural network. The present framework adopts a novel neural network, LSH-VAE,
to perform parametric MOR and interpolation. The present validations demonstrates
that the LSH-VAE is capable of the parametric interpolation of dynamic system while
significantly reducing the computational time. The following results are obtained in
this study.

23
 Table 4: Computational time requirements for 3D fluid
flow
Procedure Computational time[h]
FOM 193.7
Offline Algorithm 1 11.3
Total offline stage 205.0
Algorithm 2 <0.01
Online Variable reconstruction/write 2.02
Total online stage 2.02
Total sum 207.0

400
FOM
350 LSH-VAE
Computational time[hours]

300

250

200

150

100

50

0
0 2 4 6 8 10
Number of computations
Fig. 16: Computational time in terms of the parametric queries for the 3D fluid flow

• A novel machine-learning method, LSH-VAE, is developed for nonlinear MOR and

the parametric interpolation of nonlinear, dynamic systems.
• LSH-VAE is assessed on three nonlinear and multiphysics dynamic systems with
many DOFs. The proposed framework is proven to be accurate and to significantly
reduce the computational time.
• Compared against the existing nonlinear MOR methods, convolutional autoencoder
and β-VAE, LSH-VAE demonstrates significantly higher accuracy.
The performance of LSH-VAE is assessed on three nonlinear dynamic systems:
FSI benchmark, LCO, and three-dimensional flow. For all of the systems, LSH-VAE is
capable of constructing an accurate parametric ROM. Especially, LSH-VAE exhibited
a significantly enhanced accuracy compared to CAE and β-VAE. Also, LSH-VAE is
found to be effective as not only did it interpolate the variables well, but it also
interpolated the vorticity with high accuracy, which is embedded in the patterns of
variables. Upon the accurate parametric MOR, LSH-VAE exhibites a speed-up factor
of 990, 660, and 14 respectively.

24
 Such results are possible owing to the improvements in the LSH-VAE. First, it
adopts a hierarchical structure that enables a much deeper and more stable network.
Second, it adopts a hybrid weighted loss function consisting of mean-squared error
and KL divergence. The use of mean-squared error improved the performance against
continuous datasets while the hybrid weights reduced posterior collapse. Lastly, the
use of slerp interpolation instead of linear interpolation in the latent space significantly
enhanced the interpolation quality following the complex latent manifolds.
However, there still exist a few challenges to be dealt with. First, LSH-VAE may
require a significant amount of video random access memory (VRAM) if it is incor-
porated with an extensive number of DOF. The excessive VRAM requirement stems
from its deep structure. By adopting a deep structure, LSH-VAE is capable of gen-
erating an expressive result at the cost of training an extensive number of learnable
nodes. The excessive VRAM requirements necessitate limiting the batch size for the
3D fluid flow example. Yet, VRAM limitations may be alleviated by adopting paral-
lel computing and utilizing many GPUs. Splitting the DOFs into several groups and
merging them after interpolation may also be considered as a solution. Second, extrap-
olation is limited in the proposed framework. Accurate extrapolation would require
dense sampling in the parametric space. However, the construction of ROM with suf-
ficiently dense sampling accompanied by an effective latent manifold tracking method
would make reasonable extrapolation viable. Finally, the effectiveness of the proposed
framework decreases as the FOM becomes simpler and increasing DOFs are involved.
An example of this tendency is demonstrated in the 3D fluid flow example where the
speed-up factor diminished to 14 compared to 990 and 660 in the previous cases.
In the future, the plan is to extend the evaluation of the proposed framework to
various multiphysics problems such as the analysis of the heat-structure systems. Con-
sidering that the present framework is purely data-driven, LSH-VAE is expected to
be used in its current form. In addition, multi-parametric analysis coupled with sam-
pling algorithms such as Latin hypercube will be attempted by adopting conditional
tokens in the latent space.
Acknowledgments. This research was supported by the Basic Science Research
Program through the National Research Foundation of Korea (NRF), funded by the
Ministry of Science, ICT and Future Planning (2023R1A2C1007352).

Declarations
The authors declare that they have no conflict of interest.

References
[1] Rowley, C.W., Colonius, T., Murray, R.M.: Model reduction for compressible
flows using pod and galerkin projection. Physica D: Nonlinear Phenomena 189(1-
2), 115–129 (2004)

25
 [2] Carlberg, K., Bou-Mosleh, C., Farhat, C.: Efficient non-linear model reduction
via a least-squares petrov–galerkin projection and compressive tensor approxima-
tions. International Journal for Numerical Methods in Engineering 86(2), 155–181
(2011)

[3] Chen, H., et al.: Blackbox stencil interpolation method for model reduction. PhD
thesis, Massachusetts Institute of Technology (2012)

[4] Berkooz, G., Holmes, P., Lumley, J.L.: The proper orthogonal decomposition in
the analysis of turbulent flows. Annual Review of Fluid Mechanics 25(1), 539–575
(1993)

[5] Ravindran, S.S.: A reduced-order approach for optimal control of fluids using
proper orthogonal decomposition. International Journal for Numerical Methods
in Fluids 34(5), 425–448 (2000)

[6] Rajagopal, K., Balakrishnan, S.N., Nguyen, N.T., Kumar, M.: Proper orthogonal
decomposition technique for near-optimal control of flexible aircraft wings. In:
AIAA Guidance, Navigation, and Control (GNC) Conference, p. 4935 (2013)

[7] Shane, C., Jha, R.: Structural health monitoring of a composite wing model
using proper orthogonal decomposition. In: 48th AIAA/ASME/ASCE/AHS/ASC
Structures, Structural Dynamics, and Materials Conference, p. 1726 (2007)

[8] Kim, Y., Cho, H., Park, S., Kim, H., Shin, S.: Advanced structural analysis based
on reduced-order modeling for gas turbine blade. AIAA Journal 56(8), 3369–3373
(2018)

[9] Lee, S., Cho, H., Kim, H., Shin, S.-J.: Time-domain non-linear aeroelastic analysis
via a projection-based reduced-order model. The Aeronautical Journal 124(1281),
1798–1818 (2020)

[10] Lee, S., Jang, K., Cho, H., Kim, H., Shin, S.: Parametric non-intrusive model
order reduction for flow-fields using unsupervised machine learning. Computer
Methods in Applied Mechanics and Engineering 384, 113999 (2021)

[11] Lee, S., Jang, K., Lee, S., Cho, H., Shin, S.: Parametric model order reduction
by machine learning for fluid–structure interaction analysis. Engineering with
Computers, 1–16 (2023)

[12] Carlberg, K., Barone, M., Antil, H.: Galerkin v. least-squares petrov–galerkin
projection in nonlinear model reduction. Journal of Computational Physics 330,
693–734 (2017)

[13] Kim, Y., Choi, Y., Widemann, D., Zohdi, T.: A fast and accurate physics-informed
neural network reduced order model with shallow masked autoencoder. Journal
of Computational Physics 451, 110841 (2022)

26
[14] Xu, J., Duraisamy, K.: Multi-level convolutional autoencoder networks for para-
metric prediction of spatio-temporal dynamics. Computer Methods in Applied
Mechanics and Engineering 372, 113379 (2020)

[15] Kim, Y., Choi, Y., Widemann, D., Zohdi, T.: Efficient nonlinear manifold reduced
order model. arXiv preprint arXiv:2011.07727 (2020)

[16] Brunton, S.L., Noack, B.R., Koumoutsakos, P.: Machine learning for fluid
mechanics. Annual review of fluid mechanics 52, 477–508 (2020)

[17] Kramer, M.A.: Nonlinear principal component analysis using autoassociative

neural networks. AIChE journal 37(2), 233–243 (1991)

[18] DeMers, D., Cottrell, G.: Non-linear dimensionality reduction. Advances in neural
information processing systems 5 (1992)

[19] Gonzalez, F.J., Balajewicz, M.: Deep convolutional recurrent autoencoders for
learning low-dimensional feature dynamics of fluid systems. arXiv preprint
arXiv:1808.01346 (2018)

[20] Kadeethum, T., Ballarin, F., Choi, Y., O’Malley, D., Yoon, H., Bouklas, N.:
Non-intrusive reduced order modeling of natural convection in porous media
using convolutional autoencoders: comparison with linear subspace techniques.
Advances in Water Resources 160, 104098 (2022)

[21] Kadeethum, T., Ballarin, F., O’Malley, D., Choi, Y., Bouklas, N., Yoon, H.:
Reduced order modeling with barlow twins self-supervised learning: Navigat-
ing the space between linear and nonlinear solution manifolds. arXiv preprint
arXiv:2202.05460 (2022)

[22] Kim, H., Cheon, S., Jeong, I., Cho, H., Kim, H.: Enhanced model reduction
method via combined supervised and unsupervised learning for real-time solution
of nonlinear structural dynamics. Nonlinear Dynamics, 1–31 (2022)

[23] Omata, N., Shirayama, S.: A novel method of low-dimensional representation

for temporal behavior of flow fields using deep autoencoder. Aip Advances 9(1),
015006 (2019)

[24] Lee, K., Carlberg, K.T.: Model reduction of dynamical systems on nonlinear man-
ifolds using deep convolutional autoencoders. Journal of Computational Physics
404, 108973 (2020)

[25] Berthelot, D., Raffel, C., Roy, A., Goodfellow, I.: Understanding and improv-
ing interpolation in autoencoders via an adversarial regularizer. arXiv Preprint
arXiv:1807.07543 (2018)

[26] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint

27
 arXiv:1312.6114 (2013)

[27] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair,
S., Courville, A., Bengio, Y.: Generative adversarial nets. Advances in neural
information processing systems 27 (2014)

[28] Lucas, J., Tucker, G., Grosse, R., Norouzi, M.: Understanding posterior collapse
in generative latent variable models (2019)

[29] Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A.M., Jozefowicz, R., Bengio, S.:
Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349
(2015)

[30] Sønderby, C.K., Raiko, T., Maaløe, L., Sønderby, S.K., Winther, O.: Ladder
variational autoencoders. Advances in neural information processing systems 29
(2016)

[31] Fu, H., Li, C., Liu, X., Gao, J., Celikyilmaz, A., Carin, L.: Cyclical anneal-
ing schedule: A simple approach to mitigating kl vanishing. arXiv preprint
arXiv:1903.10145 (2019)

[32] Vahdat, A., Kautz, J.: Nvae: A deep hierarchical variational autoencoder.
Advances in Neural Information Processing Systems 33, 19667–19679 (2020)

[33] Burgess, C.P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., Lerch-
ner, A.: Understanding disentangling in beta-vae. arXiv preprint arXiv:1804.03599
(2018)

[34] Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed,
S., Lerchner, A.: beta-vae: Learning basic visual concepts with a constrained
variational framework. In: International Conference on Learning Representations
(2017)

[35] Kingma, D.P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., Welling,
M.: Improved variational inference with inverse autoregressive flow. Advances in
neural information processing systems 29 (2016)

[36] Eivazi, H., Veisi, H., Naderi, M.H., Esfahanian, V.: Deep neural networks for
nonlinear model order reduction of unsteady flows. Physics of Fluids 32(10),
105104 (2020)

[37] Eivazi, H., Le Clainche, S., Hoyas, S., Vinuesa, R.: Towards extraction of orthog-
onal and parsimonious non-linear modes from turbulent flows. Expert Systems
with Applications 202, 117038 (2022)

[38] Solera-Rico, A., Vila, C.S., Gómez, M., Wang, Y., Almashjary, A., Dawson,
S., Vinuesa, R.: β-variational autoencoders and transformers for reduced-order

28
 modelling of fluid flows. arXiv preprint arXiv:2304.03571 (2023)

[39] Mrosek, M., Othmer, C., Radespiel, R.: Variational autoencoders for model order
reduction in vehicle aerodynamics. In: AIAA Aviation 2021 Forum, p. 3049 (2021)

[40] Kang, Y.-E., Yang, S., Yee, K.: Physics-aware reduced-order modeling of transonic
flow via β-variational autoencoder. Physics of Fluids 34(7), 076103 (2022)

[41] Wang, J., He, C., Li, R., Chen, H., Zhai, C., Zhang, M.: Flow field prediction of
supercritical airfoils via variational autoencoder based deep learning framework.
Physics of Fluids 33(8), 086108 (2021)

[42] Zhu, J., Shi, H., Song, B., Tao, Y., Tan, S.: Information concentrated variational
auto-encoder for quality-related nonlinear process monitoring. Journal of Process
Control 94, 12–25 (2020)

[43] Kneifl, J., Rosin, D., Röhrle, O., Fehr, J.: Low-dimensional data-based surrogate
model of a continuum-mechanical musculoskeletal system based on non-intrusive
model order reduction. arXiv preprint arXiv:2302.06528 (2023)

[44] Miolane, N., Holmes, S.: Learning weighted submanifolds with variational
autoencoders and riemannian variational autoencoders. In: Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
14503–14511 (2020)

[45] Wang, K., Forbes, M.G., Gopaluni, B., Chen, J., Song, Z.: Systematic devel-
opment of a new variational autoencoder model based on uncertain data for
monitoring nonlinear processes. IEEE Access 7, 22554–22565 (2019)

[46] Lee, S., Kwak, M., Tsui, K.-L., Kim, S.B.: Process monitoring using variational
autoencoder for high-dimensional nonlinear processes. Engineering Applications
of Artificial Intelligence 83, 13–27 (2019)

[47] Kullback, S., Leibler, R.A.: On information and sufficiency. The annals of
mathematical statistics 22(1), 79–86 (1951)

[48] Turek, S., Hron, J.: Proposal for Numerical Benchmarking of Fluid-structure
Interaction Between an Elastic Object and Laminar Incompressible Flow.
Springer, ??? (2006)

[49] Odaibo, S.: Tutorial: Deriving the standard variational autoencoder (vae) loss
function. arXiv preprint arXiv:1907.08956 (2019)

[50] Clevert, D.-A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep net-
work learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289
(2015)

[51] Larsen, A.B.L., Sønderby, S.K., Larochelle, H., Winther, O.: Autoencoding

29
 beyond pixels using a learned similarity metric. In: International Conference on
Machine Learning, pp. 1558–1566 (2016). PMLR

[52] White, T.: Sampling generative networks. arXiv preprint arXiv:1609.04468

(2016)

[53] Agustsson, E., Sage, A., Timofte, R., Van Gool, L.: Optimal transport maps for
distribution preserving operations on latent spaces of generative models. arXiv
preprint arXiv:1711.01970 (2017)

[54] O’Neil, T., Strganac, T.W.: Aeroelastic response of a rigid wing supported by
nonlinear springs. Journal of Aircraft 35(4), 616–622 (1998)

30