0% found this document useful (0 votes)
64 views15 pages

Physics-Informed Neural Network Method For Solving One-Dimensional Advection Equation Using Pytorch

1) The document discusses using physics-informed neural networks (PINNs) to solve the one-dimensional advection equation. PINNs are neural networks trained to solve supervised learning tasks while respecting any given physics laws described by nonlinear partial differential equations. 2) The authors implement PINNs using PyTorch to find numerical solutions to the advection equation under conditions that allow an analytical solution. They compare the PINN solution to the analytical solution to examine accuracy. 3) Of all the numerical schemes tested, including finite difference approximations, only the PINN approximation accurately predicted the outcome defined by the analytical solution to the advection equation. The authors conclude PINNs have potential for real-time physics simulations without costly

Uploaded by

Waridho Iskandar
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
64 views15 pages

Physics-Informed Neural Network Method For Solving One-Dimensional Advection Equation Using Pytorch

1) The document discusses using physics-informed neural networks (PINNs) to solve the one-dimensional advection equation. PINNs are neural networks trained to solve supervised learning tasks while respecting any given physics laws described by nonlinear partial differential equations. 2) The authors implement PINNs using PyTorch to find numerical solutions to the advection equation under conditions that allow an analytical solution. They compare the PINN solution to the analytical solution to examine accuracy. 3) Of all the numerical schemes tested, including finite difference approximations, only the PINN approximation accurately predicted the outcome defined by the analytical solution to the advection equation. The authors conclude PINNs have potential for real-time physics simulations without costly

Uploaded by

Waridho Iskandar
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 15

Physics-Informed Neural Network Method for Solving One-Dimensional

Advection Equation Using PyTorch.

Shashank Reddy Vadyala; Sai Nethra Betgeri.


Department of Computational Analysis and Modeling, Louisiana Tech University, Ruston, LA
United States.

Abstract:

Numerical solutions to the equation for advection are determined using different finite-difference
approximations and physics-informed neural networks (PINNs) under conditions that allow an
analytical solution. Their accuracy is examined by comparing them to the analytical solution. We
used a machine learning framework like PyTorch to implement PINNs. PINNs approach allows
training neural networks while respecting the PDEs as a strong constraint in the optimization as
apposed to making them part of the loss function. In standard small-scale circulation simulations,
it is shown that the conventional approach incorporates a pseudo diffusive effect that is almost as
large as the effect of the turbulent diffusion model; hence the numerical solution is rendered
inconsistent with the PDEs. This oscillation causes inaccuracy and computational uncertainty. Of
all the schemes tested, only the PINNs approximation accurately predicted the outcome. We
assume that the PINNs approach can transform the physics simulation area by allowing real-time
physics simulation and geometry optimization without costly and time-consuming simulations on
large supercomputers.

Keywords: Data-driven scientific computing, Partial differential equations, Physics, Machine


learning, Finite element method

1. Introductions

Partially differential equations (PDEs), whose states exist in infinite-dimensional spaces, are often
used to model physical science and engineering processes [1]. Due to the lack of computational
methods in most situations, finite-dimensional approximations are used instead, which are based
on conventional numerical techniques that have been developed and improved over the years.
Traditional numerical solvers, on the other hand, frequently necessitate considerable
computational effort, particularly for complex systems with multiscale/multiphysics features, and
may not be feasible in real-time or many-query applications, such as optimization, inverse
problem, and uncertainty quantification (UQ), which necessitate many repeated simulations[2].
Solving PDE structures for the best possible combination between precision and reliability is
always a problem [3]. Some data-driven approaches to solve PDEs with deep neural networks
(DNNs) exist nowadays. The benefits of using DNNs to approximate PDE solutions[4]. DNNs can
Identify nonlinear interactions, which are confirmed mathematically by universal approximation
theorems.
Forward assessments of trained DNNs are quick, which is ideal for real-time or multiple-query
applications. DNN models are analytically differentiable; derivative information can be easily
derived for optimization and control problems using automatic differentiation[5]. Recently,
researchers have attempted to use DNNs in numerous sectors, including education[6], media[7],
power[8], and healthcare[9]. There are several problems in various fields for which there is no
analytical approach. Since certain constants are believed to be fixed, even problems with analytical
solutions have them. The analytical solution to a simplistic dilemma, on the other hand, teaches us
a lot about the system's actions. On the other hand, if no analytical solution approach is available,
numerical methods can be used to investigate problems rapidly. However, extreme caution must
be exercised to ensure that a converged solution is achieved. This means we need to figure out if
the step sizes are minimal enough to discover the solutions to the equations we are trying to
understand. The numerical method FEM is an excellent tool to solve complicated geometrical
shapes with a boundary and load condition that is difficult to describe with analytical expressions
available. There are generally three approaches by which scientific problems/equations are solved:
Analytical, Numerical, and Experimental. However, we cannot perform the experimental method
every time because of cost and time constraints. The traditional approaches for solving problems
are analytical methods. However, we cannot solve equations analytically due to limitations
imposed by complex geometry, boundary conditions, and other factors.
Consequently, we have been pushing towards numerical methods for several years because they
can produce almost reliable results in addition to analytical methods, and they can do so in a much
shorter and easier period. The famous Navier-Stoke equation has never been solved analytically,
but it can be solved quickly using Numerical Schemes. The disadvantages of traditional Numerical
Schemes are they are challenging in representing irregular boundaries, not optimized for
unstructured meshes, and momentum, energy, and mass are not conserved[10]. Lastly, they are
not well suited for turbulent flow slow for significant problems, and they tend to be biased towards
edges and one-dimensional physics.
To solve the drawbacks of traditional methods, we investigate the use of physics-informed neural
networks (PINNs) as a solution approximation for PDEs in this paper. PINNs – neural networks
trained to solve supervised learning tasks while respecting any given physics law described by
general nonlinear PDEs Fig. 1 and Eq. (1) summarizes the PINNs[11]. We will talk about PyTorch
as a Python implementation for PINNs. We assume these PINNs would affect real-world
applications where reduced-order physics models are possible and widely used. The numerical
efficiency of reduced-order models usually comes at the expense of physical consistency. We
illustrate how PINNs can be used in reduced-order models to reduce epistemic (model-form)
ambiguity for the advection equation. PINNs architectures can help close the distance between
forecasts and observable results. Our method begins with an abstract formulation and then moves
on to numerical integration methods before arriving at a neural network implementation.

PINNs = Data + Neural Networks + Physical Laws Eq. (1)


2
Fig.1. Schematic of a PINN for solving the diffusion equation (𝜕𝑢⁄𝜕𝑡 =⋏ 𝜕 𝑢⁄𝜕𝑡 2 )
The rest of the paper is structured in the following way. Section 2 background of advection
equation, section 3.1 specifies the implementation choices in terms of language and libraries, and
public repositories (needed for replicating results). Section 3.2 presents the formulation and
implementation for integrating the first-order ordinary differential equation with the simple Euler's
forward method. Section 4 presents results, and section 5 closes conclusions and future work.
Finally, Appendix summarizes concepts about neural networks used in this paper.

2. Background

Whether it is waves on a lake, sound waves, or an earthquake, everybody has seen traveling waves.
Writing down a reasonable-looking finite difference approximation to a wave equation is not
complicated. Most of the time is spent attempting to decide if the process succeeds. Since
numerical wave propagation has its own set of complexities, we will start with one of the most
basic mathematical equations for generating moving waves[12]. The advection equation is written
as

𝜕𝑢⁄ + 𝑎 𝜕𝑢⁄ = 0, 𝑓𝑜𝑟 {−∞ < 𝑥 < ∞


𝜕𝑡 𝜕𝑥 0<𝑡 Eq. (2)

The convenient feature of model Eq. (2) is that it has an analytical solution in Eq. (3):
𝑢 = 𝑢0 𝑓(𝑥 − 𝑎𝑡) Eq. (3)

Which represents a wave propagating with a constant velocity a with unchanged shape. When a>0,
the wave propagating in the positive x-direction, whereas for a<0, the wave propagates in the
negative x-direction. Equation (2) is significant in various applications should not be overlooked
due to its mathematical simplicity. The momentum equation for gas motion, for example, is an
inhomogeneous version of equation (2), that is dependent on u. Therefore, in the physical model
of interstellar gas and its motion due to the solar wind, a nonlinear version of the advection
equation occurs[13]. Another application is in analyzing traffic flow along a highway[14]. The
density of cars along the road is represented by u (x, t). Their speed is represented by a. Equation
(2) may serve as a model-equation for a compressible fluid, e.g., if u denote pressure, it represents
a pressure wave propagating. In a compliant pipe, such as a blood vessel, the advection equation
is often used to model pressure or flow transmission. The applications of fractional advection-
dispersion equations for anomalous solute transport in surface and subsurface water. The
generalization of equation 𝜕𝑢⁄𝜕𝑡 + 𝜕𝐹⁄𝜕𝑥 = 0, where for the linear advection equation F(u) =
au.

PDEs are equations that contain unknown multivariate functions and their partial derivatives in
Eq. (4).

∅𝑡 + ∇ ∙ (𝑢 ∙ ∅) = ∇ ∙ (Γ∇∅) Eq. (4)

Where ∅ is the dependent variable, ∅𝑡 = 𝜕∅⁄𝜕𝑡 is the derivative of ∅𝑡 concerning time, t, 𝛻 =


(𝜕∅⁄𝜕𝑡 , 𝜕∅⁄𝜕𝑡 , ) is the nabla operator, u = (𝑢, 𝑣, 𝑤) is the velocity, and Γ is the diffusivity.
𝜕∅
𝜕𝑡
The independent variables are space, x = (𝑥, 𝑦, 𝑧), and time, t. Equation (4) is known as the
convection-diffusion equation. Diffusion and convection are two mechanisms that explain how
particles, electricity, and other physical quantities are transported within a physical structure.
Concerning diffusion, 𝛻 ∙ (𝛤𝛻∅), The concentration of a chemical can be assumed by ∅. As
concentration is minimal in one region relative to the surrounding areas (e.g., a local minimum of
concentration), the substance diffuses from the surrounding areas, increasing concentration. The
net diffusion is proportional to the Laplacian (or second derivative) of concentration. if the
diffusivity 𝛤 is a constant, 𝛤𝛻 2 ∅. On the other hand, concerning convection, 𝛻 ∙ (𝑢 ∙ ∅), Assume
the chemical is being carried down a canal, and we are calculating the concentration of the water
per second. Someone spills a barrel of the chemical into the river upstream. The concentration will
suddenly rise and fall as the field with the elevated chemicals passes. The problems presented in
this study as examples will consist of finding the function ∅(𝑥, 𝑡) that, for a given geometry,
original, and boundary conditions, satisfies the PDE. In a typical approach such as finite volume,
we divide the statistical domain into small regions and consider the average volume size ∅ at a
given moment Eq. (5).

1
∅𝑛𝑖 = ∫ ∅(𝑥, 𝑡 𝑛 ) Eq. (5)
𝑉𝑖 𝑉𝑖

Where 𝑉𝑖 denotes the volume of the i𝑡ℎ discretized element in the computational domain. The
global solution can then be obtained by combining the different solutions for all volumes Eq. (6).

∅𝑛 = ∑ ∅𝑛𝑖 Eq. (6)


𝑉𝑖

This function is piecewise constant and cannot be deduced, demonstrating one of the first
drawbacks of conventional numerical methods. Many applications need derivable PDE solutions.
Specific exciting effects, such as heat fluxes and mass transfer, are computed with derivatives; a
derivable solution would be more reliable than a derivatives approximation using piecewise
functions. We discretize the various operators in equation one and use a time integration scheme
to change the time. In its most basic form, the first-order Euler algorithm Eq. (7),

∆𝑡 Eq. (7)
∅𝑛+1
𝑖 = ∅𝑛𝑖 − ∑ 𝐹𝑓 𝐴𝑓
𝑉𝑖 𝑓𝑖

where ∆𝑡 is the time step, 𝐹𝑓 is the flux ∅ across the face f of the volume 𝑉𝑖 and 𝐴𝑓 is the area of
the face. Additional computational schemes to calculate these unknown values are used to
calculate the fluxes at the faces. Some popular choices are central difference or upwind methods.
Two more aspects are required to complete the problem: the original and boundary conditions. To
begin, keep in mind that an initial state is simply a time-dimensional boundary condition. This is
important because initial and boundary conditions are typically considered independently, but they
will be treated similarly in our approach. The initial condition sets the value for ∅(𝑥, 𝑡 = 0) and
functions as the first values to start the computation in the previously described time-marching
algorithm. Boundary conditions set the value for ∅(𝑥 ∈ 𝐷, 𝑡 > 0) where D is the total number of
points on the domain's boundaries. We must specify a specific set of rules to change these points,
not to see values outside the scope. There are many boundary conditions, but the ones used in our
examples are as follows:

• Periodic: The domain folds itself to bind boundaries when periodic conditions are assumed.
• Dirichlet: For this type of boundary condition, we will fix ∅ it at boundaries.
• Newmann: For this type of boundary condition, we will fix ∇∅ at boundaries.

For the treatment of walls, inflows, and outflows, specific boundaries may be needed. In the
temporal dimension, an initial state is a Dirichlet boundary condition. The approach used for other
conventional approaches such as finite difference or finite elements differs slightly. The basic
principle remains the same: discretize the computational domain into small regions where the
solution is assumed and bring them back together to retrieve the global solution. Therefore, there
are no derivable piecewise alternatives. Furthermore, since we use time-marching algorithms, new
computations are needed once the free-parameters, current, or boundary conditions are changed.

3 Methods:

3.1 Implementation

This part will teach you how to solve equation (2) using a neural network. Python is being used to
solve the PDEs. Python, a programming language, has grown in popularity in recent years. For the
following reasons: Since there is no need for an explicit declaration of variables or a separate
compilation period, it is quick and simple to code and use for small "prototyping" tasks. It is
available for free on most computing systems and has a vast repository of packages covering a
wide range of applications. Python also has features that make developing and documenting
massive, well-structured program structures easier. Python is not appropriate for running extended
computational computations since it is an interpreted language. However, calls to precompiled
library routines are often used in the code for such computations. Many of these routines are
available directly from Python using the PyTorch[15], NumPy[16], and SciPy[17] packages. Most
operating systems (OS), including Linux, OSX, and MSWindows, have these kits available for
free.

The aim is to achieve a trained multilayer perceptron (MLP) that can give the output φ(x, t), where
x and t are set as inputs that satisfy equation 1. The basic principle is that a forward transfer on the
network using the independent variables as PINN inputs gives one the value of the dependent
variables evaluated. Since PINNs are derivable, we can compute the dependent variables (outputs).
The dependent variables (inputs) to calculate the different derivatives that appear in the original
PDEs. We create a loss function that fits the PDEs with this derivative, and we use it throughout
the training phase. We will assume that our PINNs are a solution to the PDEs if the loss function
approaches a near-zero value. The teaching is done in an unsupervised setting. There are
continuous and derivable solutions that can be applied throughout the whole domain. PINNs also
have the advantage of allowing one to use the PDEs-free parameters in the solution. Therefore, a
solution trained for various values of these parameters will generalize to different conditions rather
than a single scenario, eliminating the need for new calculations any time a parameter is modified.
This property is of particular interest in optimization studies. In more depth, we describe a
collection of points within our domain in the same way as standard approaches would. These points
are divided into two categories: preparation and validation after training. We also differentiate
between internal and external points. This will be dealt with in compliance with the boundary
conditions that have been established. Then, we define the MLP architecture: several layers and a
number of hidden units in each layer. The number of inputs would be the same as the number of
independent variables in the PDEs with any free parameters we want to add. The number of outputs
would be the same as the number of unknowns that need to be solved. Table 1 summarizes the
PINNs algorithm. Once we have training data and the PINNs defined, follows the steps:

• We compute the network's outputs at all points, ∅(𝑥, 𝑡) and the derivatives concerning the
inputs: ∅𝑡 , ∅𝑥 , ∅𝑥𝑥 .
• We use a loss function that fits our PDE for internal points. This is the role we'd like to
improve: ∅𝑡 + 𝛻 ∙ (𝑢 ∙ ∅) − 𝛻 ∙ (𝛤𝛻∅) = 0
• We can create an MSE loss function to fulfill the given condition for boundary points since
we fix values.
• Update the parameters of the PINNs for each loss function.

Table 1: PINNs Algorithm.


PINNs Algorithm
Input: dynamics parameters ∅, start time 𝑡0 , stop time 𝑡1 , final state z (𝑡1 ).
• Specify the two training sets 𝑇𝑓 and 𝑇𝑏 for the equation and boundary/intial conditions.
3.2•PINN
Define
fordynamics on augmented
Solving Partial state. Equations
Differential
• Specify a loss function for PDE equation and boundary condition.
• Train the PINNs to find the best parameters ∅ by minimizing the loss function.

3.2 Solving one-dimensional advection equation using PINNs.


We aim to create a qualified multilayer perceptron (MLP) that can output (x, t) when given x and
t as inputs and satisfy equation (2). Consider the one-dimensional advection Eq. (8), which is a
simplification of equation (1) for a 1D,

∅𝑡 + 𝑢∅𝑥 = 0 Eq. (8)

Where ∅(𝑥, 𝑡) is the unknown function, x and t are the independent variables, u is a constant
parameter and ∅𝑡 and ∅𝑥 are the derivatives of ∅ concerning t and x, respectively. This PDE has
an analytical solution, which is ∅(𝑥, 𝑡) = ∅(𝑥, 𝑥 − 𝑢𝑡). We may assume that the initial condition
was physical ∅(𝑥, 𝑡 = 0) moves in x at speed u. In the case that ∅(𝑥, 𝑡 = 0) =
1 𝑥−𝑎𝑡 2 1 𝑥−𝑎𝑡 2
𝑒𝑥𝑝 (− ( ) ) /𝐿 the solution is ∅(𝑥, 𝑡) = 𝑒𝑥𝑝 (− ( ) ) /𝐿 as illustrated in Fig. 3. To
2 0.4 2 0.4
solve the equation, we define a set of points for training.
∆𝑡
Discretized equation: 𝑢𝑖𝑛+1 = 𝑢𝑖𝑛 − 𝑎 ∆𝑥 (𝑢𝑖𝑛 − 𝑢𝑖−1
𝑛
) Eq.
(9)

Fig. 3. Example of the solution to the 1D advection equation with the initial condition
1 𝑥−𝑎𝑡 2
𝑒𝑥𝑝 (− 2 ( ) )
0.4

Defining a ∆𝑥 and ∆𝑡 allows us to build a uniform grid of points in the entire domain used in the
discretized Eq. (9). We describe internal and boundary points, each of which would have a separate
loss function associated with it. When t = 0, the initial condition will use a Mean Square Error
(MSE) loss function, which will equate the known initial condition with the PINNs output. For the
spatial boundary condition, we use a periodic condition that compares the solutions using an MSE
loss function at x = 0 and x = L for any t and forces them to be equal. As seen in Fig. 4, we
characterize our solution as an MLP with two inputs (number of independent variables), hidden
layers, and one output. Fig. 4 shows a general scheme's flowchart for converting a differential
equation into the PINNs structure.
Fig. 4. Flowchart for designing the PINNs for a general PDE

The training steps as follows:


• Compute the network outputs for the internal points: ∅(0 < 𝑥 < 𝐿; 𝑡 > 0).
• Compute the gradients of the outputs concerning the inputs: ∅𝑥 , ∅𝑡 .
• Create the internal point loss function: L1 =MSE (∅𝑥 + ∅𝑡 )
• Calculate boundary state outputs: ∅(0 < 𝑥 < 𝐿; 𝑡 = 0), ∅(𝑥 = 0; 𝑡) and ∅(𝑥 = 𝐿, 𝑡).
• Establish a failure function to account for boundary conditions: L2 =MSE (∅(0 < x <
1 𝑥−𝑎𝑡 2
L; t = 0) − 𝑒𝑥𝑝 (− 2 ( ) ) /𝐿 , L3 = MSE (∅(𝑥 = 0; 𝑡) − (∅(𝑥 = 𝐿; 𝑡))
0.4
• Update the PINNs parameters for the other losses.

MLP with two inputs, x and t, one output ∅(𝑥, 𝑡), sigmoid activation function, and five hidden
layers with 32 neurons shown in Fig. 5. Moreover, we here assume that t is a function of ∅(𝑥, 𝑡),
the first derivative of ∅(𝑥, 𝑡) concerning the set of inputs and physical parameters, θ. The training
points are randomly uniformly drawn from their corresponding domains. {𝑥 𝑛,𝑖 ,𝑢𝑛,𝑖 } corresponding
to data at 𝑡 𝑛 𝑎𝑛𝑑 Euler scheme equation (7) now allows us to infer the latent solution u(t, x) in a
sequential method. Starting from initial data {𝑥 𝑛,𝑖 ,𝑢𝑛,𝑖 } at time 𝑡 𝑛 and data at the domain
boundaries x = 0 and x = 2 shown in Fig. 5, we can use the loss function to train the networks and
predict the solution at the time 𝑡 𝑛+1 . A Euler time-stepping scheme would then use this prediction
as initial data for the next step and proceed to train again and predict. We use an optimization
algorithm to perform well in applying the PINNs method for parameter estimation. The Adam
optimizer with a learning rate of 0.001 is used until the optimization problem's solution converges
with the prescribed tolerance. We set the maximum number of Adam's epoch to be 10,000 and use
the mini-batch training; the PINNs weights are randomly initialized using the Xavier scheme[18].
Fig. 5. Two distributions of training points of PINNs for one-dimensional advection equation
(Left). PINNs Multilayer perceptrons (Right)

These steps are usually small in the classical numerical analysis due to stability constraints for
explicit schemes or computational complexity constraints for implicit formulations. If the number
of Euler scheme stages increases, these restrictions become more serious. Thousands to millions
of such measures are needed for most realistic interest problems before the solution is resolved up
to a desired final period. In comparison to conventional approaches, we can use an implicit Euler
scheme with an infinitely large number of stages at no added expense. This helps one to overcome
the entire Spatio-temporal solution in a single step while ensuring consistency and high predictive
precision. Table 2 shows the parameters and constants used implementation of PINNs and different
hyperbolic schemes.

Table 2: Constants and parameters

Constants and parameters Values


a (Wave Speed) 1.0
𝑡𝑚𝑖𝑛 , 𝑡𝑚𝑎𝑥 (start and stop time of simulation) 0.0,2.0
𝑁𝑥 (Number of spatial points) 80
C (Courant number, need C<=1 for stability) 0.9

4. Results

A one-dimensional advection equation is solved to understand the main process of solving a PDE
with a PINNs. Compared PINNs with a traditional FTBS [19], Lax Wendroff[20], and Lax
Friedrich[21] all time-integrated with an Euler scheme. Our trained PINNs, unlike the FVM
solution, are continuous and derivable over the entire domain. Since physical effects do not limit
PINNs, our experiments demonstrate that a much larger mesh will provide better results in this
specific case. In comparison to conventional methods, we find that PINNs are capable of
minimizing error. Our findings are summarized in table 3 below.

Table 3: MSE reported. MSE between the predicted ∅(𝑥, 𝑡) and the exact solution u(x, y) for the
different methods and PINNs.
Methods MSE

FTBS 3.87e-03

Lax Wendroff 4.32e-03

Lax Fredrich 4.78e-03

PINNs 1.65e-03

The computing time needed to integrate the problem in a small grid of points is a few seconds.
Still, if the domain's discretization contains about 10000 points for each variable, the computing
time of the 10000 Euler method is not similar to that of the PINNs training. However, the
computing time requirement of the PINNs method is not comparable with traditional numerical
methods. Any numerical method gives solution points over a selected grid. The computing time,
in this case, increases as the number of grid points increases. So, computing the solution for an
infinite number of points will need infinite time. On the other hand, the PINNs find a continuous
solution over the real domain in a finite time.

The PINNs were tested with points that did not participate in the training. Fig. 5 shows the MSE
of the trained network as a function of the number of testing points. It can be observed that the
averaged error is stable for an increasing number of testing points, which indicates that the solution
found is a good approximation of the real solution not only for the training points but also over the
whole domain of the problem.

Fig. 5. MSE of the trained PINNs vs the number of testing points into the integration domain.
Fig. 6 shows the errors in the resulting PINNs solution ∅(𝑥, 𝑡) for the reference one-dimensional
advection equation u(x, t) field with the maximum point errors provided in Table 4 for the
prediction in 0 < t < 2. The FTBS, Lax Wendroff, and Lax Friedrich approximation produced a
less accurate solution and have more intense oscillation than PINNs. As can be seen, training the
PINNs requires optimization concerning many loss functions (as many as PDEs and different
boundary conditions), challenging complex problems. We see that PINNs more sharply resolve
discontinuities at the cost of adding oscillations, which leads to a net reduction in error. We can
use this result to hypothesize that PINNs will further improve the solution when the discontinuity
is more significant. We verify our hypothesis by plotting the error ratio against discontinuity width
in 0 < t < 2 and demonstrating that more significant discontinuities lead to lower error ratios. In
the solutions obtained with Lax Friedrich, FTBS, and Lax Wendroff, an anomalous oscillation is
present when the grid spacing is too large to follow the quantity's variations advected closely. Lax
Friedrich, FTBS, and Lax Wendroff results converge faster and produce more significant
predictive errors than PINNs at almost all training data points. The accuracy of the predictions Lax
Friedrich, FTBS, and Lax Wendroff is very good at the initial conditions t=0, and the boundary
conditions are controlled or well known, but they struggle at t=2. MSE indicates that the PINNs
loss can produce an excellent approximation to the solution (0 < t < 2) without explicit
regularization.
Fig. 6: A range of hyperbolic schemes are implemented 1D advection equation with the initial
1 𝑥−𝑎𝑡 2
condition 𝑒𝑥𝑝 (− 2 ( ) ). The colored lines with a marker "*" denote different hyperbolic
0.4
schemes. The solid black line denotes the exact solution.

Table 4: 1D advection equation: Relative final prediction error measure in the L2 norm for
different hyperbolic schemes. Here, the time-step size to t = 0.5.

Model t = 0 min t = 0.5 t=1 t = 1.5 t=2


FTBS 2.33e-04 5.29e-04 5.46e-03 6.01e-03 7.12e-03
Lax Wendroff 2.03e-04 4.90e-04 5.26e-03 6.71e-03 8.92e-03
Lax Friedrich 2.53e-04 5.80e-04 6.56e-03 7.21e-03 9.32e-03
PINNs 1.03e-04 1.10e-7 2.26e-03 2.45e-03 3.42e-03

5. Conclusions:

By comparing numerical solutions of advection equation with known analytic solutions and PINNs
have been demonstrated. The FTBS, Lax Wendroff, and Lax Friedrich introduced a pseudo
diffusion effect that led to a large error in t=2 and would make accurate turbulent diffusion
modeling impossible. In those cases where the value of the property being advected varied rapidly
in the space of a few grid intervals, variations which seem to be typical in many models, the Lax
Friedrich, FTBS, and Lax Wendroff introduced an anomalous oscillation into the distribution of
the quality being advected at t=2. The oscillations grew steadily with time and could easily
introduce instability into a numerical model. Of the methods investigated, Only the PINNs
approximation moved the advected field correctly at 0 < t < 2. Traditional numerical methods to
find a consistent solution with more straightforward and faster methods led to inaccurate results.
Only the PINNs method produces an accurate and consistent approximation with the PDEs.
Traditional numerical methods also require a large amount of computer core storage. PINNs
approach can transform the physics simulation area by allowing real-time physics simulation and
geometry optimization without costly and time-consuming simulations on large supercomputers.

6. References:
[1] J. Sirignano and K. Spiliopoulos, "DGM: A deep learning algorithm for solving partial
differential equations," J. Comput. Phys., vol. 375, pp. 1339–1364, Dec. 2018, doi:
10.1016/j.jcp.2018.08.029.
[2] U. Rüde, K. Willcox, L. C. McInnes, and H. D. Sterck, "Research and Education in
Computational Science and Engineering," SIAM Rev., vol. 60, no. 3, pp. 707–754, Jan. 2018, doi:
10.1137/16M1096840.
[3] M. Gunzburger, "Finite Element Solution of Boundary Value Problems—Theory and
Computation (O. Axelsson and V. A. Barker)," SIAM Rev., vol. 30, no. 1, pp. 143–144, Mar. 1988,
doi: 10.1137/1030025.
[4] M. Mahmoudabadbozchelou, M. Caggioni, S. Shahsavari, W. H. Hartt, G. Em Karniadakis,
and S. Jamali, "Data-driven physics-informed constitutive metamodeling of complex fluids: A
multifidelity neural network (MFNN) framework," J. Rheol., vol. 65, no. 2, pp. 179–198, Feb.
2021, doi: 10.1122/8.0000138.
[5] J. Xu, Q. Tao, Z. Li, X. Xi, J. A. K. Suykens, and S. Wang, "Efficient hinging hyperplanes
neural network and its application in nonlinear system identification," ArXiv190506518 Cs, Nov.
2019, Accessed: Mar. 14, 2021. [Online]. Available: http://arxiv.org/abs/1905.06518.
[6] A. Albalowi and A. Alhamed, Big data and learning analytics in higher education:
Demystifying variety, acquisition, storage, NLP and analytics. 2017.
[7] K. Denecke, “Extracting Medical Concepts from Medical Social Media with Clinical NLP
Tools : A Qualitative Study,” 2014. /paper/Extracting-Medical-Concepts-from-Medical-Social-
NLP-Denecke/8ed7609f41aa772a93c4a46f5ae7a9266559376e (accessed Mar. 14, 2021).
[8] E. Strubell, A. Ganesh, and A. McCallum, "Energy and Policy Considerations for Deep
Learning in NLP," ArXiv190602243 Cs, Jun. 2019, Accessed: Mar. 14, 2021. [Online]. Available:
http://arxiv.org/abs/1906.02243.
[9] S. R. Vadyala, S. N. Betgeri, E. A. Sherer, and A. Amritphale, "Prediction of the Number of
COVID-19 Confirmed Cases Based on K-Means-LSTM," ArXiv200614752 Phys. Q-Bio, Jun.
2020, Accessed: Mar. 14, 2021. [Online]. Available: http://arxiv.org/abs/2006.14752.
[10] C. Loredo, D. Banks, and N. Roqueñí, "Evaluation of analytical models for heat transfer in
mine tunnels," Geothermics, vol. 69, pp. 153–164, Sep. 2017, doi:
10.1016/j.geothermics.2017.06.001.
[11] M. Raissi, P. Perdikaris, and G. E. Karniadakis, "Physics-informed neural networks: A deep
learning framework for solving forward and inverse problems involving nonlinear partial
differential equations," J. Comput. Phys., vol. 378, pp. 686–707, Feb. 2019, doi:
10.1016/j.jcp.2018.10.045.
[12] D. Dutykh and F. Dias, "Water waves generated by a moving bottom," in Tsunami and
Nonlinear Waves, A. Kundu, Ed. Berlin, Heidelberg: Springer, 2007, pp. 65–95.
[13] R. Bruno and V. Carbone, "The Solar Wind as a Turbulence Laboratory," Living Rev. Sol.
Phys., vol. 10, no. 1, p. 2, May 2013, doi: 10.12942/lrsp-2013-2.
[14] L. Romero and F. G. Benitez, "OUTLINE OF DIFFUSION-ADVECTION IN TRAFFIC
FLOW MODELLING," p. 17.
[15] E. Stevens, L. Antiga, T. Viehmann, and S. Chintala, Deep learning with PyTorch. 2020.
[16] T. Oliphant, Guide to NumPy. 2006.
[17] "Learning SciPy for Numerical and Scientific Computing," Packt.
https://www.packtpub.com/product/learning-scipy-for-numerical-and-scientific-
computing/9781782161622 (accessed Mar. 14, 2021).
[18] S. K. Kumar, "On weight initialization in deep neural networks," ArXiv170408863 Cs, May
2017, Accessed: Mar. 14, 2021. [Online]. Available: http://arxiv.org/abs/1704.08863.
[19] W. Long, J. Kirby, and Z. Shao, "A numerical scheme for morphological bed level
calculations," Coast. Eng., vol. 55, pp. 167–180, Feb. 2008, doi:
10.1016/j.coastaleng.2007.09.009.
[20] P. L. Roe, Generalized Formulation of TVD Lax-Wendroff Schemes. National Aeronautics
and Space Administration, Langley Research Center, 1984.
[21] G. Chen, "CONVERGENCE OF THE LAX–FRIEDRICHS SCHEME FOR
ISENTROPIC GAS DYNAMICS (III)," Acta Math. Sci., vol. 6, no. 1, pp. 75–120, Jan. 1986, doi:
10.1016/S0252-9602(18)30535-6.

Appendix. PINNs method

PINNs is a simple multilayer feedforward neural network with depth D that contains an input layer,
D − 1 hidden layers, and output layer. Without loss of generalization, we assume that there are 𝑁𝑑
neurons in the 𝑑𝑡ℎ hidden layer. Then, the 𝑑𝑡ℎ hidden layer receives the post-activation output
𝑥 𝑑−1 ∈ 𝑅𝑁𝑑−1 of the previous layers as its input, and the specific affine transformation is of the
form shown in Eq. (A.1):

𝜌𝑑 (𝑋 𝑑−1 )≡𝑊 𝑑 𝑥 𝑑−1 + 𝑏 𝑑 Eq. (A.1)

where the network weight 𝑊 𝑑 ∈ 𝑅𝑁𝑑−1 and the bias term 𝑏 𝑑 ∈ 𝑅𝑁𝑑 be learned are initialized
using some special strategies, such as Xavier initialization or initialization. The nonlinear
activation function σ(·) is applied component-wise to the affine output 𝜌𝑑 of the present layer. For
specific regression issues, this nonlinear activation is not used in the output layer, or we may
assume that the identity activation is used in the output layer. As a result, the neural network can
be denoted as 𝑞(𝑥, 𝛩) = (𝜌𝑑 ◦ σ ◦ 𝜌𝑑−1◦ · · · ◦ σ ◦ 𝜌1 )(x). Where the operator " ◦ " is the composition
operator, Θ = {𝑊 𝑑 , 𝑏 𝑑 }𝐷 ∈ P represents the learnable parameters to be optimized later in the
network, and P is the parameter space, q, and 𝑥 0 = x is the output and input of the network,
respectively. The residual networks 𝑓𝑢 (x, t) := 𝑢𝑡 + 𝑁𝑢 (u, 𝑢𝑥 , 𝑢𝑥𝑥 , 𝑢𝑥𝑥𝑥 , · · ·), and 𝑓𝑣 (x, t):= 𝑣𝑡
+ 𝑁𝑣 (v, 𝑣𝑥 , 𝑣𝑥𝑥 , 𝑣𝑥𝑥𝑥 , · · ·). Then the solution q (x, t) will be trained to satisfy these two physical
constraint conditions. Which play a vital role in regularization and have been embedded into the
mean-squared objective function, that is, the loss function 𝑙𝑜𝑠𝑠𝛩 = 𝑙𝑜𝑠𝑠𝑢 + 𝑙𝑜𝑠𝑠𝑣 + 𝑙𝑜𝑠𝑠𝑓𝑢 + 𝑙𝑜𝑠𝑠𝑓𝑣

Appendix. Multilayer perceptrons (MLP) shown in Fig. A.1

Fig. A.1 illustrates the popular MLP.


Each layer can have one or more perceptrons (nodes in the graph). A perceptron applies a linear
combination to the input variables followed by an activation function.
𝑣 = 𝑓(𝑧) and 𝑧 = 𝑊 𝑇 𝑢 + 𝑏
where 𝑣 is the perceptron output, u is the inputs; w and b are the perceptron hyperparameters, and
𝑓(.) is the activation function. Throughout this paper, we used the hyperbolic tangent (tanh),
sigmoid, and the exponential linear unit (elu) activation functions (although others could also be
used, such as the rectified exponential linear unit):
𝑒 𝑧 −𝑒 −𝑧 1 𝑧 𝑤ℎ𝑒𝑛 𝑧 > 0
tanh(𝑧) = 𝑒 𝑧 +𝑒 −𝑧 , sigmoid(𝑧) = 1+𝑒 −𝑧 and elu(𝑧) ={ 𝑧
𝑒 −1

You might also like