Copyright © 2019 Tech Science Press CMC, vol.59, no.1, pp.

345-359, 2019

Artificial Neural Network Methods for the Solution of Second

Order Boundary Value Problems

Cosmin Anitescu 1, Elena Atroshchenko2, Naif Alajlan3 and Timon Rabczuk3, *

Abstract: We present a method for solving partial differential equations using artificial
neural networks and an adaptive collocation strategy. In this procedure, a coarse grid of
training points is used at the initial training stages, while more points are added at later
stages based on the value of the residual at a larger set of evaluation points. This method
increases the robustness of the neural network approximation and can result in significant
computational savings, particularly when the solution is non-smooth. Numerical results
are presented for benchmark problems for scalar-valued PDEs, namely Poisson and
Helmholtz equations, as well as for an inverse acoustics problem.

Keywords: Deep learning, adaptive collocation, inverse problems, artificial neural

networks.

1 Introduction
Artificial neural networks (ANNs) have been a topic of great interest in the machine
learning community due to their ability to solve very difficult problems, particularly in
the fields of image processing and object recognition, speech recognition, medical
diagnosis, etc. More recently, applications have been found in engineering, especially
where large data sets are involved. From a mathematical point of view, neural networks
are also interesting due to their ability to efficiently approximate arbitrary functions
[Cybenko (1989)].
A natural question is to determine whether ANNs can be used to approximate the solution
of partial differential equations which commonly appear in physics, engineering and
mathematical problems. Several articles and even a book [Yadav (2015)] have been very
recently devoted to this topic. In most of the approaches considered, a collocation-type
method is employed which attempts to fit the governing equations and the boundary
conditions at randomly selected points in the domain and on the boundary. Among these
methods we mention the Deep Galerkin Method [Sirignano and Spiliopoulos (2018)],
Physics Informed Neural Networks [Raissi, Perdikaris, and Karniadakis (2019)], as well
as the earlier works in Lagaris et al. [Lagaris, Likas and Fotiadis (1998); Lagaris, Likas

1 Institute of Structural Mechanics, Bauhaus-Universität Weimar, 99423, Weimar, Germany.

2 School of Civil & Environmental Engineering, University of New South Wales, Sydney, Australia.
3 Department of Computer Engineering, College of Computer and Information Sciences, King Saud

University, Riyadh, Saudi Arabia.

*
Corresponding Author: Timon Rabczuk. Email: [email protected].

CMC. doi:10.32604/cmc.2019.06641 www.techscience.com/cmc

346 Copyright © 2019 Tech Science Press CMC, vol.59, no.1, pp.345-359, 2019

and Papageorgiou (2000); van Milligen, Tribaldos and Jiménez (1995); Kumar and Yadav
(2011); McFall and Mahan (2009)]. This method appears to produce reasonably accurate
results, particularly for high-dimensional domains [Han, Jentzen and Weinan (2018)] and
domains with complex geometries [Berg and Nyström (2018)], where the meshfree
character of these methods makes them competitive with established discretization methods.
Another related approach is to use an energy minimization formulation of the governing
equation as in Weinan et al. [Weinan and Yu (2018); Wang and Zhang (2019)]. This
formulation has the advantage that only the first derivatives need to be computed for a 2nd
order problem, however it requires a more precise integration procedure and not all
governing equations can be cast in an energy-minimization framework.
In this work, we employ a collocation formulation for solving 2nd order boundary value
problems such as Poisson’s equation and Helmholtz equation. Different from existing
methods which typically use a randomly scattered set of collocation points, we present an
adaptive approach for selecting the collocation points based on the value of the residual at
previous training steps. This method can improve the robustness of collocation method,
particularly in cases when the solution has a non-smooth region where increasing the
number of training points is beneficial.
The paper is structured as follows: in Section 2 we give an overview of artificial neural
networks and briefly discuss their approximation properties. The application of ANNs to
forward and inverse boundary-value problems is discussed in Section 3. Detailed
numerical results are presented in Section 4, followed by concluding remarks.

2 Structure of neural network

In this section, we briefly describe the anatomy of a neural network and its properties for
approximating arbitrary functions. We focus in particular on simple feed-forward neural
networks which are used in the subsequent sections.

2.1 Feed-forward networks

A feed-forward network can be seen as a computational graph consisting of an input
layer, an output layer and an arbitrary number of intermediary hidden layers, where all
the neurons (units) in adjacent layers are connected with each other. It can be used to
represent a function 𝑢𝑢: ℝ𝑛𝑛 → ℝ𝑚𝑚 by using n neurons in input layer and m neurons in the
output layer, see Fig. 1. We index the layers, starting with the input layer at 0, and the
output layer as 𝐿𝐿, and we denote the number of neurons in each layer by 𝑘𝑘0 =
𝑛𝑛, 𝑘𝑘1 , … , 𝑘𝑘𝐿𝐿 = 𝑚𝑚. To each connection between the i-th neuron in layer 𝑙𝑙 − 1 and the j-th
neuron in layer 𝑙𝑙, with 0 < 𝑙𝑙 ≤ 𝐿𝐿, we associate a weight 𝑤𝑤𝑗𝑗𝑗𝑗𝑙𝑙 and to each neuron in the
layers 0 < 𝑙𝑙 ≤ 𝐿𝐿 we associate a bias 𝑏𝑏𝑗𝑗 , 𝑖𝑖 = 1, … , 𝑘𝑘𝑙𝑙 . Moreover, we define an activation
function 𝜎𝜎 𝑙𝑙 : ℝ → ℝ between the layers 𝑙𝑙 and 𝑙𝑙 − 1. Then the values at each neuron can be
written in terms of the activation function applied to a linear combination of the neurons
in the previous layer given by the corresponding weights and biases, i.e.,
Artificial Neural Network Methods for the Solution 347

Figure 1: Schematic of a feed-forward neural network with 𝑳𝑳 − 𝟏𝟏 hidden layers

𝑘𝑘 𝑙𝑙 𝑙𝑙−1
𝑢𝑢𝑘𝑘𝑙𝑙 = 𝜎𝜎 𝑙𝑙 �∑𝑗𝑗=1
𝑙𝑙−1
𝑤𝑤𝑘𝑘𝑗𝑗 𝑢𝑢𝑗𝑗 + 𝑏𝑏𝑘𝑘𝑙𝑙 � (1)
This can be written more compactly in matrix form as:
𝒖𝒖𝑙𝑙 = 𝜎𝜎 𝑙𝑙 (𝑾𝑾𝑙𝑙 𝒖𝒖𝑙𝑙−1 + 𝒃𝒃𝑙𝑙 ) for 𝑙𝑙 = 1,2, … , 𝐿𝐿, (2)
𝑙𝑙
where 𝑾𝑾 is a matrix of weights corresponding to the connections between layers 𝑙𝑙 − 1
and 𝑙𝑙, 𝒖𝒖𝑙𝑙 = [𝑢𝑢𝑗𝑗𝑙𝑙 ] and 𝒃𝒃𝑙𝑙 = [𝑏𝑏𝑗𝑗𝑙𝑙 ] are column vectors and the activation function is applied
element-wise.
Existing computational frameworks such as Tensorflow or Pytorch perform very efficient
parallel execution, also on GPUs if available, of computational graphs like the ones
defined by (2). Moreover, the input values can be defined as multi-dimensional arrays
(tensors) and the computation of the corresponding outputs is efficiently vectorized and
distributed across the available computational resources.
A key observation is that, for a given a neural network, the partial derivatives of the outputs
with respect to the weights and biases can also be efficiently computed by the back-
propagation algorithm. The idea is to perform the chain rule starting with the last layer and
store the intermediary values in a computational graph where the order of the layers is
reversed. The back-propagation algorithm enables the application of gradient-based
minimization algorithms were a loss function based on the output of the neural network is
to be minimized. Moreover, the partial derivatives of the outputs with respect to the inputs
or with respect to some other prescribed parameters can be calculated in a similar way.
In typical applications of deep learning, a neural network is trained on a set of matching
inputs and outputs by seeking to minimize the difference between the predicted values
and some known correct outputs. This often requires large data sets that often must be
manually processed and are themselves subject to different types of errors. We avoid this
by defining a loss function which minimizes the residuals of the governing equations at a
chosen set of training points. The training points can be simply generated as randomly
scattered points in the domain as in Raissi et al. [Raissi, Perdikaris and Karniadakis
(2019)]. In this work, we adopt instead an adaptive procedure which iteratively adds
more points to the training set where the residual values are higher than some prescribed
threshold, as detailed in Section 3.
348 Copyright © 2019 Tech Science Press CMC, vol.59, no.1, pp.345-359, 2019

Aside from the choice of the size of the neural networks (the number of hidden layers and
the number of neurons in each layers) and that of the training points, other important
parameters are related to the selection of the activation function and the choice of the
minimization algorithm. Typical activation functions used are ramp functions like ReLU,
sigmoid (logistic) function, and the hyperbolic tangent function (tanh). In this work, we
use the tanh activation function which is preferable due to its smoothness. For
optimization, we use the Adam (adaptive momentum) optimizer which based on
stochastic gradient descent followed by a quasi-Newton method (L-BFGS) which builds
an approximated Hessian at each gradient-descent step.

2.2 Theoretical approximation properties of neural networks

It is well-known that artificial neural networks have very good approximation properties
when the function to be approximated is continuous. This has been established since the
1980s, where it was shown (Hornik, Stinchcombe and White 1989) that under mild
assumptions on the activation function, a given function 𝒇𝒇(𝒙𝒙) can be approximated within
any chosen tolerance by a network with single hidden layer and a finite number of neurons.
It was later shown, e.g., [Lu, Pu, Wang et. al. (2017)] that by choosing an non-linear
activation function and forming deeper networks, the number of neurons can be
significantly reduced. Various estimates of the approximation properties of neural
networks for approximating PDEs have been more recently derived [Sirignano and
Spiliopoulos (2018)].
We note however, that while the neural networks are in theory capable to represent
complex functions in a very compact way, finding the actual parameters (weights and
biases) which solve a given differential equation within a given approximation tolerance
can be quite difficult. In the following, we present a method for selecting the training data
which gives some control over the performance of the solver and optimization algorithms.

3 Collocation solver for PDEs

In the collocation method, the main idea is to define a loss function which is based on the
strong form of the governing equations and the boundary conditions. The loss function is
evaluated at chosen sets of points in the interior on the domain as well as on the boundary.
More specifically, let us assume that a general boundary value problem can be written as:
ℒ�𝐮𝐮(𝐱𝐱)� = 𝑓𝑓(𝐱𝐱) for 𝐱𝐱 ∈ Ω, (3)
𝒢𝒢�𝐮𝐮(𝐱𝐱)� = 𝑔𝑔(𝐱𝐱) for 𝐱𝐱 ∈ 𝜕𝜕Ω, (4)
where Ω ∈ ℝ𝑑𝑑 is the problem domain with boundary 𝜕𝜕Ω, ℒ, 𝒢𝒢 are interior and boundary
differential operators, and 𝑓𝑓, 𝑔𝑔 are prescribed functions (e.g., loading data).

3.1 Poisson equation

We consider equations of the form:
−Δ𝑢𝑢(𝐱𝐱) + 𝑘𝑘𝑢𝑢(𝐱𝐱) = 𝑓𝑓(𝐱𝐱), for 𝐱𝐱 ∈ Ω (5)
𝑢𝑢(𝐱𝐱) = 𝑢𝑢�(𝐱𝐱)for 𝒙𝒙 ∈ 𝜕𝜕Ω𝐷𝐷 , (6)
𝜕𝜕𝜕𝜕(𝐱𝐱)
𝜕𝜕𝐧𝐧
= 𝑔𝑔(𝐱𝐱), for 𝐱𝐱 ∈ ∂ΩN , (7)
Artificial Neural Network Methods for the Solution 349

where 𝑢𝑢(𝐱𝐱) is the unknown solution, Ω ⊂ ℝ𝑑𝑑 is the computational domain, 𝑘𝑘 ≥ 0 is a

given constant, 𝑓𝑓 is the given source term, 𝑢𝑢� is the prescribed Dirichlet data on the
Dirichlet boundary 𝜕𝜕Ω𝐷𝐷 , 𝐧𝐧 is the outer normal vector, and 𝑔𝑔 is the Neumann data. When
𝑘𝑘 = 0, the problem is a standard Poisson equation; we also consider a more general
setting when e.g., 𝑘𝑘 = 1.
We define a loss function, ℂ(𝑢𝑢; 𝑘𝑘, 𝑓𝑓, 𝑢𝑢�, 𝑔𝑔) using a mean squared error (MSE) for the
interior and boundary governing equations as:
𝑁𝑁𝑖𝑖𝑖𝑖𝑖𝑖
1
ℂ(𝑢𝑢; 𝑘𝑘, 𝑓𝑓, 𝑢𝑢�, 𝑔𝑔) ≔ �[Δ𝑢𝑢(𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ ) − 𝑘𝑘𝑢𝑢(𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ ) + 𝑓𝑓(𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ )]2
𝑁𝑁𝑗𝑗𝑛𝑛𝑖𝑖
𝑗𝑗=1
𝑁𝑁𝑑𝑑𝑖𝑖𝑑𝑑
𝛾𝛾𝑑𝑑𝑗𝑗𝑑𝑑 2
+ � �𝑢𝑢��𝑥𝑥𝑗𝑗𝑑𝑑𝑗𝑗𝑑𝑑 , 𝑦𝑦𝑗𝑗𝑑𝑑𝑗𝑗𝑑𝑑 � − 𝑢𝑢�𝑥𝑥𝑗𝑗𝑑𝑑𝑗𝑗𝑑𝑑 , 𝑦𝑦𝑗𝑗𝑑𝑑𝑗𝑗𝑑𝑑 ��
𝑁𝑁𝑑𝑑𝑗𝑗𝑑𝑑
𝑗𝑗=1
𝑁𝑁𝑖𝑖𝑛𝑛𝑛𝑛
𝛾𝛾𝑛𝑛𝑛𝑛𝜕𝜕 𝑛𝑛𝑛𝑛𝜕𝜕 𝑛𝑛𝑛𝑛𝜕𝜕 )
𝜕𝜕𝑢𝑢 𝑛𝑛𝑛𝑛𝜕𝜕 𝑛𝑛𝑛𝑛𝜕𝜕 2
+ � �𝑔𝑔(𝑥𝑥𝑘𝑘 , 𝑦𝑦𝑘𝑘 − (𝑥𝑥 , 𝑦𝑦𝑘𝑘 )� ,
𝑁𝑁𝑛𝑛𝑛𝑛𝜕𝜕 𝜕𝜕𝐧𝐧 𝑘𝑘
𝑘𝑘=1
where 𝑁𝑁𝑗𝑗𝑛𝑛𝑖𝑖 , 𝑁𝑁𝑑𝑑𝑗𝑗𝑑𝑑 and 𝑁𝑁𝑛𝑛𝑛𝑛𝜕𝜕 represent the number of points (𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ ), �𝑥𝑥𝑗𝑗𝑑𝑑𝑗𝑗𝑑𝑑 , 𝑦𝑦𝑗𝑗𝑑𝑑𝑗𝑗𝑑𝑑 �, and
(𝑥𝑥𝑘𝑘𝑛𝑛𝑛𝑛𝜕𝜕 , 𝑦𝑦𝑘𝑘𝑛𝑛𝑛𝑛𝜕𝜕 ) in the interior domain, on the Dirichlet boundary and the Neumann
boundary respectively. Moreover, 𝛾𝛾𝑑𝑑𝑗𝑗𝑑𝑑 and 𝛾𝛾𝑛𝑛𝑛𝑛𝜕𝜕 represent penalty terms for the Dirichlet
and Neumann boundaries. While in some cases it is enough to set 𝛾𝛾𝑑𝑑𝑗𝑗𝑑𝑑 = 𝛾𝛾𝑛𝑛𝑛𝑛𝜕𝜕 = 1, the
convergence of the loss function can be improved by increasing one or both of these
values to ensure that the boundary conditions are satisfied. During the minimization
process, the terms in the cost function can be monitored and usually more accurate results
can be obtained when the loss for the interior points is of the same order as the losses
corresponding to the Dirichlet and Neumann boundaries.

3.2 Helmholtz equation

The governing equation for the homogeneous Helmholtz equation is of the form:
Δ𝑢𝑢(𝑥𝑥, 𝑦𝑦) + 𝑘𝑘 2 𝑢𝑢(𝑥𝑥, 𝑦𝑦) = 0 for (𝑥𝑥, 𝑦𝑦) ∈ Ω. (8)
Here 𝑢𝑢(𝑥𝑥, 𝑦𝑦) can be complex-valued function. This equation is a time-independent form
of the wave equation and it has applications in the study of various physical phenomena,
such as acoustics, seismology and electromagnetic radiation. For many problems, the
domain Ω is not bounded and the solution 𝑢𝑢(𝑥𝑥, 𝑦𝑦) can be highly oscillatory, which creates
difficulties in standard finite element analysis.
Different types of boundary conditions, usually of the Neumann type can be imposed
depending on the problem. As for Poisson’s equation, we construct a loss function which
seeks to minimize the residual of the governing equation at collocation points. While in
theory neural networks are defined in ℝ𝑛𝑛 , where n is the number of neurons in the input
layer and can capture unbounded domains, we limit ourselves here to finite domains.

3.3 Inverse problems

In this class of problems, one is given a particular solution 𝐮𝐮∗ (𝐱𝐱), which satisfies the
350 Copyright © 2019 Tech Science Press CMC, vol.59, no.1, pp.345-359, 2019

governing equations of the form:

ℒ(𝐮𝐮∗ (𝐱𝐱); 𝝀𝝀∗ ) = 𝑓𝑓(𝐱𝐱) for 𝐱𝐱 ∈ Ω, (9)
∗ (𝐱𝐱); ∗ )
𝒢𝒢(𝐮𝐮 𝝀𝝀 = 𝑔𝑔(𝐱𝐱) for 𝐱𝐱 ∈ 𝜕𝜕Ω, (10)
∗
where 𝝀𝝀 is unknown. The problem is then to determine the unknown parameter or vector
of parameters 𝝀𝝀∗ . This can be reformulated as an optimization problem, where we start
with an initial guess 𝝀𝝀, and we approximate the solution 𝐮𝐮 which satisfies the governing
equations. In the framework of neural networks, we can use gradient descent to minimize
‖𝐮𝐮∗ − 𝐮𝐮‖ under some suitable norm. Formally, we define a cost function of the form:
𝑁𝑁𝑖𝑖𝑖𝑖𝑖𝑖 𝑁𝑁𝑏𝑏𝑖𝑖𝑑𝑑
1 2 𝛾𝛾𝑏𝑏𝑛𝑛𝑑𝑑 2
ℂ(𝐮𝐮, 𝛌𝛌) ≔ ��ℒ(𝐮𝐮(𝐱𝐱𝑗𝑗∗ ); 𝝀𝝀) − 𝑓𝑓(𝐱𝐱𝑗𝑗∗ )� + � �𝒢𝒢�𝐮𝐮�𝐱𝐱𝑗𝑗𝑏𝑏𝑛𝑛𝑑𝑑 �; 𝝀𝝀� − 𝑔𝑔�𝐱𝐱𝑗𝑗𝑏𝑏𝑛𝑛𝑑𝑑 ��
𝑁𝑁𝑗𝑗𝑛𝑛𝑖𝑖 𝑁𝑁𝑏𝑏𝑛𝑛𝑑𝑑
𝑗𝑗=1 𝑗𝑗=1
𝑁𝑁𝑖𝑖𝑖𝑖𝑖𝑖
1 2
+ ��𝐮𝐮∗ (𝐱𝐱𝑗𝑗∗ ) − 𝐮𝐮(𝐱𝐱𝑗𝑗∗ )� ,
𝑁𝑁𝑗𝑗𝑛𝑛𝑖𝑖
𝑗𝑗=1
where are interior collocation points, 𝐱𝐱𝑗𝑗𝑏𝑏𝑛𝑛𝑑𝑑 are boundary collocation points, 𝛾𝛾𝑏𝑏𝑛𝑛𝑑𝑑 is a
𝐱𝐱𝑗𝑗∗
penalty term for enforcing boundary conditions. This method seeks to find both 𝐮𝐮 and λ
simultaneously which approach 𝐮𝐮∗ and 𝝀𝝀∗ respectively. In this work, we investigate
applying this method to the Helmholtz equation where the wave number k is unknown.

3.4 Adaptive collocation

Several methods can be proposed to select the collocation points in the interior and the
boundary of the domain. Here we propose a method where we start with a coarse grid and
then select additional points based on the evaluation of the residual. We apply this
method for the selection of the interior points, for which evaluating the governing
equations requires the most computational effort, although in principle it can also be
applied to the boundary points.
The main idea of the method is described in Fig. 2. The blue dots represent the training
(collocation points) in the interior, the green dots represent the model evaluation points
after the first training is complete, and the purple dots represent the additional training
points. We note that evaluating the model at a larger number of points is quite
inexpensive computationally, while the number of training points impacts the
performance much more significantly as the governing equations need to be evaluated at
the training points at each gradient descent step. Therefore, this method provides a
criterion for selecting the collocation points in an efficient manner. Once the training is
completed for one step, the network weights and biases can be carried over to the
subsequent step, resulting in faster convergence.
Artificial Neural Network Methods for the Solution 351

a) 1st training set b) Evaluation set c) 2nd training set

Figure 2: The steps of the adaptive collocation method, assuming the residual values are
higher in the center of the domain

4 Numerical results
4.1 Poisson equation on the unit square
We first consider a Poisson equation with Dirichlet and Neumann boundary conditions:
−Δ𝑢𝑢(𝑥𝑥, 𝑦𝑦) = 8sin(2𝜋𝜋𝑥𝑥)cos(2𝜋𝜋𝑦𝑦) for (𝑥𝑥, 𝑦𝑦) ∈ (0,1)2 ,
𝑢𝑢(𝑥𝑥, 𝑦𝑦) = 0 for 𝑥𝑥 = 0,
𝑑𝑑𝑢𝑢
= 0 for 𝑦𝑦 = 0 and 𝑦𝑦 = 1, and
𝑑𝑑𝑦𝑦
𝑑𝑑𝑢𝑢
= 2𝜋𝜋 cos(2𝜋𝜋𝑥𝑥) cos(2𝜋𝜋𝑦𝑦) for 𝑥𝑥 = 1.
𝑑𝑑𝑥𝑥
The exact solution of this equation is 𝑢𝑢(𝑥𝑥, 𝑦𝑦) = sin(2𝜋𝜋𝑥𝑥) cos(2𝜋𝜋𝑦𝑦). We consider a loss
function of the form:
𝑁𝑁𝑖𝑖𝑖𝑖𝑖𝑖 𝑁𝑁𝑙𝑙𝑛𝑛𝑙𝑙𝑖𝑖
1 1 2
𝑙𝑙𝑛𝑛𝑙𝑙𝑖𝑖 𝑙𝑙𝑛𝑛𝑙𝑙𝑖𝑖
ℂ(𝑢𝑢) ≔ �[Δ𝑢𝑢(𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ ) + 2𝜋𝜋 2 sin(𝜋𝜋𝑥𝑥𝑗𝑗 )cos(𝜋𝜋𝑦𝑦𝑗𝑗 )]2 + � 𝑢𝑢�𝑥𝑥𝑗𝑗 , 𝑦𝑦𝑗𝑗 � +
𝑁𝑁𝑗𝑗𝑛𝑛𝑖𝑖 𝑁𝑁𝑙𝑙𝑛𝑛𝑙𝑙𝑖𝑖
𝑗𝑗=1 𝑗𝑗=1
𝑁𝑁𝑏𝑏𝑡𝑡𝑖𝑖𝑖𝑖𝑡𝑡𝑏𝑏 𝑁𝑁𝑖𝑖𝑡𝑡𝑡𝑡
1 𝜕𝜕𝑢𝑢 𝑏𝑏𝑏𝑏𝑖𝑖𝑖𝑖𝑏𝑏𝑚𝑚 𝑏𝑏𝑏𝑏𝑖𝑖𝑖𝑖𝑏𝑏𝑚𝑚 2 1 𝜕𝜕𝑢𝑢 𝑖𝑖𝑏𝑏𝑡𝑡 𝑖𝑖𝑏𝑏𝑡𝑡 2
+ � � �𝑥𝑥𝑗𝑗 , 𝑦𝑦𝑗𝑗 �� + � � �𝑥𝑥𝑗𝑗 , 𝑦𝑦𝑗𝑗 �� +
𝑁𝑁𝑏𝑏𝑏𝑏𝑖𝑖𝑖𝑖𝑏𝑏𝑚𝑚 𝜕𝜕𝑦𝑦 𝑁𝑁𝑖𝑖𝑏𝑏𝑡𝑡 𝜕𝜕𝑦𝑦
𝑗𝑗=1 𝑗𝑗=1
𝑁𝑁𝑑𝑑𝑖𝑖𝑟𝑟ℎ𝑖𝑖
2
1 𝜕𝜕𝑢𝑢 𝑑𝑑𝑗𝑗𝑟𝑟ℎ𝑖𝑖 𝑑𝑑𝑗𝑗𝑟𝑟ℎ𝑖𝑖 𝑑𝑑𝑗𝑗𝑟𝑟ℎ𝑖𝑖 𝑑𝑑𝑗𝑗𝑟𝑟ℎ𝑖𝑖
+ � � �𝑥𝑥𝑗𝑗 , 𝑦𝑦𝑗𝑗 � − 𝜋𝜋 cos�𝜋𝜋𝑥𝑥𝑗𝑗 � cos�𝜋𝜋𝑦𝑦𝑗𝑗 �� ,
𝑁𝑁𝑑𝑑𝑗𝑗𝑟𝑟ℎ𝑖𝑖 𝜕𝜕𝑥𝑥
𝑗𝑗=1
𝑙𝑙𝑛𝑛𝑙𝑙𝑖𝑖 𝑙𝑙𝑛𝑛𝑙𝑙𝑖𝑖
where (𝑥𝑥𝑗𝑗 . 𝑦𝑦𝑗𝑗 ) are interior collocation points, and (𝑥𝑥𝑗𝑗 , 𝑦𝑦𝑗𝑗 ), (𝑥𝑥𝑗𝑗𝑏𝑏𝑏𝑏𝑖𝑖𝑖𝑖𝑏𝑏𝑚𝑚 , 𝑦𝑦𝑗𝑗𝑏𝑏𝑏𝑏𝑖𝑖𝑖𝑖𝑏𝑏𝑚𝑚 ),
∗ ∗

𝑖𝑖𝑏𝑏𝑡𝑡 𝑖𝑖𝑏𝑏𝑡𝑡 𝑑𝑑𝑗𝑗𝑟𝑟ℎ𝑖𝑖 𝑑𝑑𝑗𝑗𝑟𝑟ℎ𝑖𝑖

(𝑥𝑥𝑗𝑗 , 𝑦𝑦𝑗𝑗 ), and (𝑥𝑥𝑗𝑗 , 𝑦𝑦𝑗𝑗 ) are boundary collocation points. Moreover, 𝑁𝑁𝑗𝑗𝑛𝑛𝑖𝑖 , 𝑁𝑁𝑙𝑙𝑛𝑛𝑙𝑙𝑖𝑖 ,
𝑁𝑁𝑏𝑏𝑏𝑏𝑖𝑖𝑖𝑖𝑏𝑏𝑚𝑚 , 𝑁𝑁𝑖𝑖𝑏𝑏𝑡𝑡 and 𝑁𝑁𝑑𝑑𝑗𝑗𝑟𝑟ℎ𝑖𝑖 are the number of collocation points in the interior and the four
boundaries. For this example, in the initial training stage we have used 𝑁𝑁𝑗𝑗𝑛𝑛𝑖𝑖 = 192
equally spaced points in the interior of the domain and 𝑁𝑁𝑙𝑙𝑛𝑛𝑙𝑙𝑖𝑖 = 𝑁𝑁𝑖𝑖𝑏𝑏𝑡𝑡 = 𝑁𝑁𝑏𝑏𝑏𝑏𝑖𝑖𝑖𝑖𝑏𝑏𝑚𝑚 =
𝑁𝑁𝑑𝑑𝑗𝑗𝑟𝑟ℎ𝑖𝑖 = 41 uniformly distributed points on each of the four edges. The test (evaluation
352 Copyright © 2019 Tech Science Press CMC, vol.59, no.1, pp.345-359, 2019

set) consists of a grid of 292 equally-spaced points. The training (collocation) points at
subsequent iterations were chosen by selecting the top 30% of the points with the highest.
The results obtained for a shallow network with one hidden layer of 10 neurons are
shown in Figure 3. The blue dots represent the interior collocation points, while the red
and green dots represent the points corresponding to the Dirichlet and Neumann
boundary conditions respectively. We note that even for this simple network with 41
parameters (30 weights and 11 biases), an accurate solution can be obtained. Because the
solution is smooth throughout the domain, the training points are generally evenly
distributed. However, more points are selected near the corners and boundaries since the
residuals and the actual errors are higher there.
The relative 𝐿𝐿2 errors obtained by increasing the number of layers while keeping the
number of neurons per layer fixed and using the same refinement strategy are shown in
Tab. 1. It can be observed that except for the single-layer network, the error decreases
significantly as more training points are used. Moreover, the error for deeper networks is
greatly reduced compared to the single-layer network, although the number of parameters
and computational cost increases as well.
Table 1: Relative 𝑳𝑳𝟐𝟐 errors for different levels of refinement and different numbers of
layers for the Poisson equation on the unit square
Number of Relative 𝐿𝐿2 Relative 𝐿𝐿2 Relative 𝐿𝐿2 error
training points error for 1 error for 2 for 3 layers
layer layers
Refinement 1 361 0.07475152 0.00486584 0.00116222
Refinement 2 816 0.03761188 0.00119753 0.00043185
Refinement 3 1271 0.08465629 0.00026268 0.00026697

4.2 Source problem on a quarter-annulus domain

We now consider a 2nd order problem with pure Dirichlet boundary conditions on a non-
rectangular domain. The governing equation is given by:
−Δ𝑢𝑢(𝑥𝑥, 𝑦𝑦) + 𝑢𝑢(𝑥𝑥, 𝑦𝑦) = 𝑓𝑓(𝑥𝑥, 𝑦𝑦), for (𝑥𝑥, 𝑦𝑦) ∈ Ω
𝑢𝑢(𝑥𝑥, 𝑦𝑦) = 0, for (𝑥𝑥, 𝑦𝑦) ∈ 𝜕𝜕Ω,
where Ω is quarter of an annulus located in the first quadrant and centered at the origin,
with inner radius 𝑟𝑟𝑗𝑗𝑛𝑛𝑖𝑖 = 1 and outer radius 𝑟𝑟𝑛𝑛𝑒𝑒𝑖𝑖 = 4. Here 𝑓𝑓(𝑥𝑥, 𝑦𝑦) is chosen such that the
exact solution is 𝑢𝑢(𝑥𝑥, 𝑦𝑦) = (𝑥𝑥 2 + 𝑦𝑦 2 − 1)(𝑥𝑥 2 + 𝑦𝑦 2 − 16) sin(𝑥𝑥) sin(𝑦𝑦), i.e.,
𝑓𝑓(𝑥𝑥, 𝑦𝑦) = (3𝑥𝑥 4 − 67𝑥𝑥 2 − 67𝑦𝑦 2 + 3𝑦𝑦 4 + 6𝑥𝑥 2 𝑦𝑦 2 + 116) sin(𝑥𝑥) sin(𝑦𝑦) +
(68𝑥𝑥 − 8𝑥𝑥 3 − 8𝑥𝑥𝑦𝑦 2 ) cos(𝑥𝑥) sin(𝑦𝑦) + (68𝑦𝑦 − 8𝑦𝑦 3 − 8𝑦𝑦𝑥𝑥 2 ) cos(𝑥𝑥) sin(𝑦𝑦).
As before, we define a cost function consisting of term corresponding to the interior
governing equation and another term corresponding to the boundary conditions. To
ensure that the boundary conditions are satisfied during the minimization process, we
apply a penalty factor 𝛾𝛾 = 100, so that the cost function becomes:
Artificial Neural Network Methods for the Solution 353

𝑁𝑁𝑖𝑖𝑖𝑖𝑖𝑖 𝑁𝑁𝑏𝑏𝑖𝑖𝑑𝑑
1 𝛾𝛾 2
ℂ(𝑢𝑢) ≔ �[Δ𝑢𝑢(𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ ) − 𝑢𝑢(𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ ) + 𝑓𝑓(𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ )]2 + � 𝑢𝑢�𝑥𝑥𝑗𝑗𝑏𝑏𝑛𝑛𝑑𝑑 , 𝑦𝑦𝑗𝑗𝑏𝑏𝑛𝑛𝑑𝑑 �
𝑁𝑁𝑗𝑗𝑛𝑛𝑖𝑖 𝑁𝑁𝑏𝑏𝑛𝑛𝑑𝑑
𝑗𝑗=1 𝑗𝑗=1
We first choose a neural network with 2 hidden layers with 10 neurons each and the tanh
activation function. The initial set of collocation points consists of 𝑁𝑁𝑗𝑗𝑛𝑛𝑖𝑖 = 192 points in
the interior and 𝑁𝑁𝑏𝑏𝑛𝑛𝑑𝑑 = 168 points on the boundary, spaced uniformly as shown in Fig. 4.
Subsequent refinements are done according to the same procedure as in the first example.
The relative 𝐿𝐿2 error is calculated as 0.00053738 for the initial training and decreases to
0.00036766 and 0.00043207 as the number of collocation points is increased.

a) Computed solution 𝑢𝑢𝑐𝑐𝑏𝑏𝑚𝑚𝑡𝑡 b) Error 𝑢𝑢𝑛𝑛𝑒𝑒 − 𝑢𝑢𝑐𝑐𝑏𝑏𝑚𝑚𝑡𝑡

c) Initial training set d) Final training set

Figure 3: Computed solution with error and the training sets of collocation points using a
network with one hidden layer and 10 neurons for the Poisson equation on a unit square

4.3 Poisson equation with a corner singularity

To investigate the ability of the proposed refinement scheme to approximate solutions
with sharp gradients, we next consider the Poisson equation on a domain with an internal
boundary which results in a corner-type singularity appearing in the solution. The domain
considered is given by Ω ≔ (−1,1)2 − [0,1) in Cartesian coordinates and we seek an
unknown solution which satisfies:
−Δ𝑢𝑢(𝑥𝑥, 𝑦𝑦) = 0, for (𝑥𝑥, 𝑦𝑦) ∈ Ω
𝑢𝑢(𝑟𝑟, 𝜃𝜃) = 𝑟𝑟1/2 sin(𝜃𝜃/2) for (𝑟𝑟 , 𝜃𝜃) = (�𝑥𝑥 2 + 𝑦𝑦 2 , arctan(𝑦𝑦/𝑥𝑥)) ∈ 𝜕𝜕Ω.
354 Copyright © 2019 Tech Science Press CMC, vol.59, no.1, pp.345-359, 2019

a) Computed solution 𝑢𝑢𝑐𝑐𝑏𝑏𝑚𝑚𝑡𝑡 b) Error 𝑢𝑢𝑛𝑛𝑒𝑒 − 𝑢𝑢𝑐𝑐𝑏𝑏𝑚𝑚𝑡𝑡

c) Initial training set d) Final training set

Figure 4: Computed solution with error and the training sets of collocation points using a
network with two hidden layer and 10 neurons each for the source equation on a quarter-
annulus domain

The exact solution in polar coordinates is 𝑢𝑢𝑛𝑛𝑒𝑒 (𝑟𝑟, 𝜃𝜃) = 𝑟𝑟1/2 sin(𝜃𝜃/2) , which has the
singular term 𝑟𝑟1/2 creating approximation difficulties near the origin. In finite elements
methods, a more refined mesh is typically required to obtain a good approximation. This
problem was also investigated in Weinan et al. [Weinan and Yu (2018)] using an energy
minimization method.
The geometry is modelled by considering 3 rectangular subdomains (−1,0) × (−1,1),
(0,1) × (−1,0), and (0,1) × (0,1). We define a loss function of the form:
𝑁𝑁𝑖𝑖𝑖𝑖𝑖𝑖 𝑁𝑁𝑏𝑏𝑖𝑖𝑑𝑑
1 𝛾𝛾 2
ℂ(𝑢𝑢) ≔ �[Δ𝑢𝑢(𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ )]2 + � �𝑢𝑢�𝑟𝑟𝑗𝑗𝑏𝑏𝑛𝑛𝑑𝑑 , 𝜃𝜃𝑗𝑗𝑏𝑏𝑛𝑛𝑑𝑑 � − 𝑢𝑢𝑛𝑛𝑒𝑒 �𝑟𝑟𝑗𝑗𝑏𝑏𝑛𝑛𝑑𝑑 , 𝜃𝜃𝑗𝑗𝑏𝑏𝑛𝑛𝑑𝑑 �� .
𝑁𝑁𝑗𝑗𝑛𝑛𝑖𝑖 𝑁𝑁𝑏𝑏𝑛𝑛𝑑𝑑
𝑗𝑗=1 𝑗𝑗=1
In the initial grid we choose equally spaced points with a distance of 0.05 in the x and y
directions. For the points on the boundary, we choose more densely spaced points, with a
distance of 0.025 in Cartesian coordinates and we set a penalty factor of 𝛾𝛾 = 500 to
ensure that the boundary conditions are respected. As before, we evaluate the model on
grids with more points and append the points where the residual value is large to the
training set in the next step.
The results obtained by the adaptive collocation scheme using a network with 3 hidden
Artificial Neural Network Methods for the Solution 355

layers and 30 neurons each are shown in Fig. 5. In general, the residual values in a
narrow region around the singularity are much larger than in the rest of the domain and
they are selected in the subsequent training step. Also, larger residuals are observed along
the line 𝑦𝑦 = 0, with 𝑥𝑥 < 0 as the neural network with a coarser training grid has
difficulties in capturing correctly the end of the internal boundary. However, as can be
seen from the plots, the error diminishes as the number of training points increases. The
accuracy can be further improved by choosing larger networks although the number of
training points needs to be increased as well.

Figure 5: Error between the exact solution and computed solution for the Poisson
equation with a singularity at origin and the training sets at each refinement step for a
network with 4 hidden layers and 30 neurons per layer

4.4 Acoustic duct problem

To investigate the applicability of neural networks to approximate the oscillatory
solutions such as those obtained by solving the Helmholtz equation in acoustics. The
benchmark problem under consideration has complex-valued governing equations of the
form:
Δ𝑢𝑢(𝑥𝑥, 𝑦𝑦) + 𝑘𝑘 2 𝑢𝑢(𝑥𝑥, 𝑦𝑦) = 0 for (𝑥𝑥, 𝑦𝑦) ∈ Ω ≔ (0,2) × (0,1)
𝜕𝜕𝑢𝑢 𝜕𝜕𝑢𝑢
= cos(𝑚𝑚𝜋𝜋𝑥𝑥) , for 𝑥𝑥 = 0; = −𝑖𝑖𝑘𝑘𝑢𝑢, for 𝑥𝑥 = 2
𝜕𝜕𝑛𝑛 𝜕𝜕𝑛𝑛
𝜕𝜕𝑢𝑢
= 0, for 𝑦𝑦 = 0 and 𝑦𝑦 = 1
𝜕𝜕𝑛𝑛
Here we select 𝑘𝑘 = 12 as the wave number and 𝑚𝑚 = 2 as the mode number. This
problem admits an analytical solution which can be written as:
𝑢𝑢𝑛𝑛𝑒𝑒 (𝑥𝑥, 𝑦𝑦) = cos(𝑚𝑚𝜋𝜋𝑦𝑦)(𝐴𝐴1 exp(−𝑖𝑖𝑘𝑘𝑒𝑒 𝑥𝑥) + 𝐴𝐴2 exp(𝑖𝑖𝑘𝑘𝑒𝑒 𝑥𝑥),
where 𝑘𝑘𝑒𝑒 = �𝑘𝑘 2 − (𝜋𝜋𝑚𝑚)2 , and 𝐴𝐴1 and 𝐴𝐴2 are the solution of the 2 × 2 linear system:
356 Copyright © 2019 Tech Science Press CMC, vol.59, no.1, pp.345-359, 2019

𝑖𝑖𝑘𝑘𝑒𝑒 −𝑖𝑖𝑘𝑘𝑒𝑒 𝐴𝐴 1
� � � 1 � = � �.
(𝑘𝑘 − 𝑘𝑘𝑒𝑒 )exp(−2𝑖𝑖𝑘𝑘𝑒𝑒 ) (𝑘𝑘 + 𝑘𝑘𝑒𝑒 )exp(2𝑖𝑖𝑘𝑘𝑒𝑒 ) 𝐴𝐴2 0
In the following, we compute only the real part of the solution 𝑢𝑢(𝑥𝑥, 𝑦𝑦) as the imaginary
part can be computed by a similar procedure.
As before, we define a loss functions which minimizes the residual of the governing
equation at interior and boundary points:
𝑁𝑁𝑖𝑖𝑖𝑖𝑖𝑖
1
ℂ(𝑢𝑢) ≔ �[Δ𝑢𝑢(𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ ) + 𝑘𝑘 2 𝑢𝑢(𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ )]2
𝑁𝑁𝑗𝑗𝑛𝑛𝑖𝑖
𝑗𝑗=1
𝑁𝑁𝑏𝑏𝑖𝑖𝑑𝑑 2
𝛾𝛾 𝜕𝜕𝑢𝑢𝑛𝑛𝑒𝑒 𝑏𝑏𝑛𝑛𝑑𝑑 𝑏𝑏𝑛𝑛𝑑𝑑 𝜕𝜕𝑢𝑢 𝑏𝑏𝑛𝑛𝑑𝑑 𝑏𝑏𝑛𝑛𝑑𝑑
+ � � �𝑥𝑥𝑗𝑗 , 𝑦𝑦𝑗𝑗 � − �𝑥𝑥 , 𝑦𝑦𝑗𝑗 �� .
𝑁𝑁𝑏𝑏𝑛𝑛𝑑𝑑 𝜕𝜕𝑛𝑛 𝜕𝜕𝑛𝑛 𝑗𝑗
𝑗𝑗=1
The results of the adaptive collocation method are shown in Fig. 6. We have used a
neural network with 3 hidden layers of 30 neurons each and a grid of 99 × 49 uniformly
spaced points in the interior of the domain in the initial step. For the boundary, we have
used 𝑁𝑁𝑏𝑏𝑛𝑛𝑑𝑑 = 400 uniformly spaced points and a penalty parameter of 𝛾𝛾 = 100. As
before, the size of the training set is increased based on the residual value on a finer grid
(with double the points in each direction) in subsequent steps. Due to the oscillatory
nature of the solution, the additional training points are also generally evenly distributed
in the domain with higher concentration in the areas where the residual value was initially
larger than average.

4.5 Inverse acoustic problem

Here we consider the same governing equation and boundary conditions as in the
previous example, but we seek instead to solve for the wave number 𝑘𝑘 while the value of
the solution (for the correct 𝑘𝑘 ) is given. In this case we build a loss function that
minimizes the difference between the given solution and the residual of the governing
equations simultaneously:
𝑁𝑁𝑖𝑖𝑖𝑖𝑖𝑖 𝑁𝑁𝑖𝑖𝑖𝑖𝑖𝑖
1 1
ℂ(𝑢𝑢, 𝑘𝑘) ≔ �[Δ𝑢𝑢(𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ ) + 𝑘𝑘 2 𝑢𝑢(𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ )]2 + �[𝑢𝑢𝑛𝑛𝑒𝑒 (𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ ) − 𝑢𝑢(𝑥𝑥𝑗𝑗∗ , 𝑦𝑦𝑗𝑗∗ )]2
𝑁𝑁𝑗𝑗𝑛𝑛𝑖𝑖 𝑁𝑁𝑗𝑗𝑛𝑛𝑖𝑖
𝑗𝑗=1 𝑗𝑗=1
𝑁𝑁𝑏𝑏𝑖𝑖𝑑𝑑 2
𝑛𝑛𝑒𝑒
𝛾𝛾 𝜕𝜕𝑢𝑢 𝜕𝜕𝑢𝑢 𝑏𝑏𝑛𝑛𝑑𝑑 𝑏𝑏𝑛𝑛𝑑𝑑
+ � � �𝑥𝑥𝑗𝑗𝑏𝑏𝑛𝑛𝑑𝑑 , 𝑦𝑦𝑗𝑗𝑏𝑏𝑛𝑛𝑑𝑑 � − �𝑥𝑥 , 𝑦𝑦𝑗𝑗 �� .
𝑁𝑁𝑏𝑏𝑛𝑛𝑑𝑑 𝜕𝜕𝑛𝑛 𝜕𝜕𝑛𝑛 𝑗𝑗
𝑗𝑗=1
Artificial Neural Network Methods for the Solution 357

a) Computed solution 𝑢𝑢𝑐𝑐𝑏𝑏𝑚𝑚𝑡𝑡 b) Error 𝑢𝑢𝑛𝑛𝑒𝑒 − 𝑢𝑢𝑐𝑐𝑏𝑏𝑚𝑚𝑡𝑡

c) Initial training set d) Final training set

Figure 6: Computed solution, the error and the sets of training points for the acoustic
duct benchmark problem with 𝒌𝒌 = 𝟏𝟏𝟐𝟐 and 𝒎𝒎 = 𝟐𝟐
Here 𝑢𝑢𝑛𝑛𝑒𝑒 has the same form as in the previous section but with 𝑘𝑘 = 4 and 𝑚𝑚 = 1. We
start with 𝑘𝑘 = 1 as an initial guess and seek to minimize the loss function with 𝑘𝑘 as a free
parameter. For this problem, we choose a grid of 149 × 29 equally spaced points in the
interior of the domain and 𝑁𝑁𝑏𝑏𝑛𝑛𝑑𝑑 = 800 boundary collocation points and 𝛾𝛾 = 100.
The results for this example are presented in Fig. 7. We can observe that the solution has
been represented with reasonable accuracy both in terms of 𝑢𝑢(𝑥𝑥, 𝑦𝑦) as well 𝑘𝑘. The
relative 𝐿𝐿2 error for 𝑢𝑢(𝑥𝑥, 𝑦𝑦) in this example is 0.084, while the computed 𝑘𝑘 is 3.882 as
compared to 4 in the reference solution. As in the other examples, we have used the
Adam optimizer followed by a quasi-Newton method (L-BFGS). It can be noted that the
latter converges significantly faster, however in many cases performing a stochastic
gradient-descent like Adam helps the solver to avoid being trapped in early local minima.
358 Copyright © 2019 Tech Science Press CMC, vol.59, no.1, pp.345-359, 2019

b) Error 𝑢𝑢𝑛𝑛𝑒𝑒 − 𝑢𝑢𝑐𝑐𝑏𝑏𝑚𝑚𝑡𝑡

a) Computed solution 𝑢𝑢𝑐𝑐𝑏𝑏𝑚𝑚𝑡𝑡

d) Final training set

c) Initial training set
Figure 7: Computed solution, the error and the convergence of the loss function and
wave number 𝒌𝒌 where the reference solution has 𝒌𝒌 = 𝟒𝟒 for the inverse acoustic problem

5 Conclusions
We have presented a collocation method for solving boundary values problem using
artificial neural networks. The method is completely mesh-free as only scattered sets of
points are used in the training and evaluation sets. Although uniform grids of training
points have been used in the initial training step, the method could be easily adapted to
scattered data obtained e.g. by Latin hypercube sampling methods. The method was
shown to produce results with good accuracy for the parameters chosen, although as
common in deep learning methods, parameter selection may require some manual tuning.
A more detailed study of the convergence and approximation properties of neural networks,
as well as selecting robust minimization procedures remain open as possible research topics.
Moreover, the applicability of these methods to energy minimization formulations, for the
differential equations which allow it, can be investigated in future work.

Acknowledgements: N. Alajlan and T. Rabczuk acknowledge the Distinguished

Scientist Fellowship Program (DSFP) at King Saud University for supporting this work.
Artificial Neural Network Methods for the Solution 359

References
Berg, J.; Nyström K. (2018): A unified deep artificial neural network approach to partial
differential equations in complex geometries. Neurocomputing, vol. 317, pp. 28-41.
Cybenko, G. (1989): Approximation by superpositions of a sigmoidal function.
Mathematics of Control, Signals and Systems, vol. 2, no. 4, pp. 303-314.
Weinan, E.; Yu, B. (2018): The deep Ritz method: a deep learning-based numerical
algorithm for solving variational problems. Communications in Mathematics and
Statistics, vol. 6, no. 1, pp. 1-12.
Han, J.; Jentzen, A.; Weinan, E. (2018): Solving high-dimensional partial differential
equations using deep learning. Proceedings of the National Academy of Sciences, vol.
115, no. 34, pp. 8505-8510.
Hornik, K.; Stinchcombe, M.; White, H. (1989): Multilayer feedforward networks are
universal approximators. Neural Networks, vol. 2, no. 5, pp. 359-366.
Kumar, M.; Yadav, N. (2011) Multilayer perceptrons and radial basis function neural
network methods for the solution of differential equations: a survey. Computers &
Mathematics with Applications, vol. 62, no. 10, pp. 3796-3811.
Lagaris, I. E.; Likas, A. C.; Papageorgiou, D. G. (2000): Neural-network methods for
boundary value problems with irregular boundaries. IEEE Transactions on Neural
Networks, vol. 11, no. 5, pp. 1041-1049.
Lagaris, I. E.; Likas, A.; Fotiadis, D. I. (1998): Artificial neural networks for solving
ordinary and partial differential equations. IEEE Transactions on Neural Networks, vol. 9,
no. 5, pp. 987-1000.
Lu, Z.; Pu, H.; Wang, F.; Hu, Z.; Wang, L. (2017): The expressive power of neural
networks: a view from the width. Advances in Neural Information Processing Systems,
vol. 30, pp. 6231-6239.
McFall, K. S.; Mahan, J. R. (2009): Artificial neural network method for solution of
boundary value problems with exact satisfaction of arbitrary boundary conditions. IEEE
Transactions on Neural Networks, vol. 20, no. 8, pp. 1221-1233.
Raissi, M.; Perdikaris, P.; Karniadakis, G. E. (2019): Physics-informed neural
networks: a deep learning framework for solving forward and inverse problems involving
nonlinear partial differential equations. Journal of Computational Physics, vol. 378, pp.
686-707.
Sirignano, J.; Spiliopoulos, K. (2018): DGM: a deep learning algorithm for solving
partial differential equations. Journal of Computational Physics, vol. 375, pp. 1339-1364.
van Milligen, B. Ph.; Tribaldos, V.; Jiménez, J. A. (1995): Neural network differential
equation and plasma equilibrium solver. Physical Review Letters, vol. 75, no. 20, pp.
3594-3597.
Wang, Z.; Zhang, Z. (2019): A mesh-free method for interface problems using the deep
learning approach. arXiv:1901.00618.
Yadav, N. (2015). An Introduction to Neural Network Methods for Differential
Equations. Springer, Netherlands.