Basic Numerical Solution Methods for Differential Equations

Sebastian Merkel
February 14, 2019

1 Ordinary Differential Equations (ODEs)

1.1 Preliminaries
• A first-order ordinary differential equation is an equation of the form

f (x, y, y 0 ) = 0 (1)

with a function f : Rn × Rn × R → Rn . A solution is a differentiable function y : R → Rn with the

property
∀x ∈ R : f (y(x), y 0 (x), x) = 0.
Definitions are adjusted in an obvious way, if f is just defined on a subset of Rn × Rn × R and/or
one is only interested in a solution y defined on a subset of R.
• A k-th order ODE is similarly an equation of the form

f (y, y 0 , ..., y (k) , x) = 0

with a fucntion f : Rn × · · · × Rn × R → Rn . Such an equation can always be reduced to a k · n-

dimensional first-order ODE by associating it with an equivalent equation for the k · n-dimensional
vector function  
y
 y0 
z :=  ..  ,
 
 . 
y (k−1)
whose first deriative is
y0
 
y (1) 
0
z =  . .
 
 .. 
y (k)
The equivalent equation is
h(z, z 0 , x) = 0

1
 with
   0  
z1 z1 (
 ..   ..   fi (z1 , z10 , . . . , zk0 , x) , i = 1, . . . , n
hi  .  ,  .  , x = 0
zj−1,i − zj,i , j ∈ {2, ..., k}, i = (j − 1)n + 1, ..., jn
zk zk0

for i = 1, ..., k · n. For this reason we focus in the following on first-order ODEs.
• An equation of the form (1) is called (fully) implicit. Often, the equation can be written in explicit
form,
y 0 = g(x, y). (2)

• For explicit ODEs there are simple sufficient conditions for existence and uniqueness of a solution
satisfying y(x0 ) = y0 for a given initial value (x0 , y0 ) ∈ R × Rn :1
– Existence (Peano theorem): if g is continuous, then a solution y to (2) such that y(x0 ) = y0
exists locally around x0 (meaning a solution function exists on some interval [x0 − ε, x0 + ε])
– Existence and Uniqueness (Picard–Lindelöf theorem): if g is continuous and satisfies a local
Lipschitz condition with respect to y, that is there is a constant L < ∞, such that |g(x, y) −
g(x, z)| ≤ L|y − z| for all y, z in some neighborhood of y0 and all x in some neighborhood of
x0 , then there is a unique solution to (2) satisfying y(x0 ) = y0 – at least locally around x0
(meaning there is an interval [x0 − ε, x0 + ε] and a unique solution on that interval)
∂g
Local Lipschitz conditions are e.g. satisfied, if ∂y (x, y) exists and is bounded (locally around
(x0 , y0 )), so regularly existence and uniqueness is not a problem in applications.
Note that these local solutions can usually be extended to a “global” solution, if they do not blow
up/reach inadmissible values in finite time.

1.2 Numerical Solutions of ODEs

1.2.1 Explicit Euler Method
Let the following objects be given: some explicit ODE of the form (2), an initial condition (x0 , y0 ) and
a desired solution domain [x0 , x̄]. A simple solution approach is to discretize the solution domain [x0 , x̄]
into N + 1 points,
x0 < x1 < · · · < xN = x̄
−yi−1
dy
and approximate the derivatives y 0 (xi−1 ) = dx (xi−1 ) by finite differences xyii −xi−1
for i = 1, ..., N , where
yi is an approximation for the unknown y(xi ). With this approximation, equation (2) evaluated at the
points xi−1 becomes
yi − yi−1
= g (xi−1 , yi−1 ) ,
xi − xi−1
which can be solved for yi :
yi = yi−1 + g (xi−1 , yi−1 ) (xi − xi−1 ) . (3)
1 If the first-order system is a reduction from a k-th order system as above, this means that at x the values of all first
0
k − 1 derivatives must be pinned down by the initial value.

2
Together with the initial values y0 , equation (3) defines a simple recursion for the approximate function
values {yi }N
i=0 .
Alternatively, one may view the Euler method as a sequence of first-order Taylor approximations to
the function y: a Taylor expansion around xi−1 evaluated in xi yields

y(xi ) ≈ y(xi−1 ) + y 0 (xi−1 ) (xi − xi−1 ) = y(xi−1 ) + g(xi−1 , y(xi−1 )) (xi − xi−1 ) ,

where the second equation holds, because y satisfied ODE (3). Replacing the true function values y(xi−1 )
and y(xi ) by their approximations yi−1 and yi yields again recursion (3).

1.2.2 Implicit Euler Method

Again, let an initial condition (x0 , y0 ), a solution domain [x0 , x̄] and a discretization {xi }N i=0 of that
−yi−1
domain be given. The explicit Euler method approximates derivatives y 0 (xi−1 ) by xyii −x i−1
and uses the
ODE in the points {x0 , ..., xN −1 } to derive an explicit recursion for {yi }N i=0 . The implicit Euler method
instead approximates y 0 (xi ) by xyii −yi−1
−xi−1 and uses the ODE in the points {x 1 , ..., x N }, which leads to the
equations
yi − yi−1
= g(xi , yi ), i = 1, ..., N
xi − xi−1
Now, there is in general no explicit solution for yi , instead one has to solve in each step a – potentially
nonlinear – equation system to find them. While this looks much more complicated than the explicit
method, the implicit method has often superior stability properties (compare Problem 4 of Problem Set
1).
This method does not require the ODE to be given in explicit form (2). It works equally for a fully
implicit ODE (1). Then the equation system to solve in each step is
 
yi − yi−1
f xi , yi , = 0.
xi − xi−1

1.2.3 Other Methods

There are a lot of other methods, that we do not cover explicitly. Two common classes of methods are
(both exist in implicit and explicit variants):
• Runge-Kutta methods: like the Euler method they are single-step methods in that when computing
the solution yi over one discretization interval [xi−1 , xi ], they only use the function value information
from the last step (xi−1 , yi−1 ) as an initial condition. However, each step is computed using multiple
stages with the goal of increasing the order of consistency2 . Each stage involves an evaluation of
g or – in the case of implicit methods – an equation to be solved. Matlab’s standard solvers for
nonstiff problems are mainly based on these methods.
• Multi-step methods: use information (xj , yj , yj0 ) from multiple previous steps j ≤ i − 1 when
computing (yi , yi0 ) with the aim of gaining efficiency. Implicit versions of (linear) multi-step methods
have worse stability properties than certain single-step methods for consistency orders larger than
2.
2 In each step, a numerical scheme produces an error, the so-called truncation error. The method is said to be consistent

of order p, if the truncation error in step i converges to 0 faster than (xi −xi−1 )p as the the step-width xi −xi−1 approaches
0. Euler methods are consistent of order 1.

3
Details on solvers pre-implemented in Matlab can be found in the ODE section of the Matlab help. You
can also execute the command odeexamples for example code using the different Matlab solvers.

2 Partial Differential Equations (PDEs)

2.1 Some Basic Definitions
A general partial differential equation in n variables of order k is an equation of the form

F x, u, Du, D2 u, . . . , Dk u = 0,

   
∂u ∂2u
where x ∈ Rn , u : Rn → R is a function and D1 u = ∂x i
, D 2
u = ∂xi1 ∂xi2 ,
  i=1,...,n i1 ,i2 =1,...,n
3
∂ u
D2 u = ∂xi ∂x i ∂xi
etc. A (classical) solution u is a k times continuously differentiable
1 2 3 i1 ,i2 ,i3 =1,...,n
function that satisfies the equation for all x. The solution theory of PDEs is much more complex than the
one of ODEs and often, solutions only exist according to some weaker solution concept.3 The focus here
will be on second order PDEs4 and in particular on PDEs that have the following quasilinear structure
n X
n
X ∂2u
aij (x, u, Du) (x) + G(x, u, Du) = 0. (4)
i=1 j=1
∂xi ∂xj

Despite the name “quasilinear”, such PDEs can be highly nonlinear. These PDEs are typically classified
into parabolic, hyperbolic and elliptic equations, although this classification does not cover all possible
cases:5
• The equation is called parabolic, if the matrix (aij (x, y, z)) is (positive or negative) semidefinite
with exactly one eigenvalue assuming the value 0 for all x ∈ Rn , y ∈ R, z ∈ Rn
• The equation is called hyperbolic, if the matrix (aij (x, y, z)) has one eigenvalue that is positive
or negative and the remaining n − 1 eigenvalues have opposite sign (but are 6= 0) for all x ∈ Rn ,
y ∈ R, z ∈ Rn
• The equation is called elliptic, if the matrix (aij (x, y, z)) is (positive or negative) definite for all
x ∈ Rn , y ∈ R, z ∈ Rn
Equations typically encountered in economics and finance are elliptic or parabolic. If the equation is
parabolic, then the eigenvalue 0 of the (aij (x, y, z)) matrix usually has an Euklidian basis vector as an
eigenvector. The associated coordinate can usually be interpreted as “time” and is separated in notation
3 This may be also relevant for economists. For many PDEs in economic problems, particularly HJB-type equations, the

concept of a viscosity solutions is useful.

4 Continuous-time models with Brownian noise naturally lead to second-order PDEs. Higher order PDEs are less relevant

for economic applications. You may sometimes encounter first-order PDEs. They can be viewed as degenerate second order
PDEs and solved by the same methods (see however the discussion of upwind and monotone schemes below). Alternatively,
it may sometimes be useful to apply the “method of characteristics” to gain insights into the solution structure of these
equations.
5 The terminology as it is defined here is also not the most general, but even more generally the classification does not

exhaust all possible cases.

4
(so write (x, t) instead of just x). A quasilinear parabolic equation of the form (4) with a separate time
dimension looks like
n X n
∂2u
 
X ∂u
aij (x, t, u, Du) (x, t) + G x, t, u, Dx u, = 0, (5)
i=1 j=1
∂xi ∂xj ∂t

where Dx u is the vector of all first-order spatial derivatives. Because for a parabolic equation the
eigenvalue 0 may appear only once, the matrix (aij (x, t, y, z)) must be (positive or negative) definite for
all x, t, y, z. For this reason, parabolic and elliptic equations are closely connected.
PDE problems that contain a time dimension are often initial value problems. Unlike for ODEs the
initial value does not just consist of a vector of function values at the initial time t0 , but of an initial
function u0 : Rn → R that assigns an initial value to each point x in the space domain of the problem.
If the space domain has boundaries (say is an interval instead of R) there may be additional (potentially
time-varying) boundary conditions. The goal is to find a solution u to the PDE that satisfies the initial
condition u(x, t0 ) = u0 (x) for all x and potentially some boundary conditions for all times t ≥ t0 . A
simple example of an initial value problem without boundary conditions is the heat equation in one space
dimension,
∂u ∂2u
=a 2
∂t ∂x
with an initial temperature distribution u(x, 0) = u0 (x) over the whole real line (for all x ∈ R). A simple
example of an initial value problem with additional boundary conditions is the same heat equation over a
finite interval [x, x] where the temperature at the interval boundaries is held fixed over time, u(t, x) = u,
u(t, x) = u for all t ≥ 0.
PDE problems without a time dimension are typically pure boundary value problems. The goal is to
find a solution u to the PDE that satisfies some additional restrictions on function values or derivatives
on the boundary of the solution domain. A simple example is the stationary heat equation

∂2u
0=a
∂x2
over some interval [x, x] with boundary conditions as in the example above, u(x) = u, u(x) = u.

2.2 Spatial Discretization and Method of Lines

2.2.1 Method of Lines for Parabolic PDEs
Consider a quasilinear parabolic PDE and assume for ease of exposition that the first time derivative
enters the equation also linearly such that it can be written in the form
n n
∂u X X ∂2u
= aij (x, t, u, Du) (x, t) + G (x, t, u, Dx u) . (6)
∂t i=1 j=1
∂xi ∂xj

By interpreting the function (x, t) 7→ u(x, t) as a (infinite-dimensional) vector function t 7→ (u(x, ·))x∈Rn ,
one can conceptually view this equation as an inifinite-dimensional system of ODEs in time and try to
use ODE methods to solve the equation. The idea is to use a space discretization to approximate the
infinite-dimensional vector (u(x, ·))x∈Rn by some finite-dimensional object. Let X ⊂ Rn , X finite, be a
grid of discretization points. For any function y : Rn → R define the vector ŷ = (y(x))x∈X of function

5
 ∂y
values on the grid and denote by di (ŷ, x) some approximation scheme for the first derivative ∂x i
(x) with
respect to xi that only uses information in the vector ŷ (information of function values on the grid).
Here, x ∈ X is a grid point. Similarly, let d2ij (ŷ, x) be some approximation scheme for second derivatives
∂2y
For n = 1 and a uniform grid x0 < x1 < · · · < xN , xi = xi−1 + ∆x, common approximation
∂xi ∂xj (x).
schemes are (let for this example ŷi = y(xi ))
 ŷ −ŷ
i+1 i−1
 2∆x
 (central differences)
ŷi −ŷi−1 ŷi+1 + ŷi−1 − 2ŷi
d(ŷ, xi ) = ∆x (left differences) d2 (ŷ, xi ) = .
 ŷi+1 −ŷi
 ∆x2
∆x (right differences)

Section 3 discusses in more detail how to design appropriate approximation schemes d, d2 . Given a space
grid X and derivative approximation schemes, a discretized version of (6) is
n n
dû XX
(t) = aij (x, t, û(t), d1 (û(t), x), ..., dn (û(t), x)) d2ij (û(t), x)+G (x, t, û(t), d1 (û(t), x), ..., dn (û(t), x)) , x ∈ X,
dt i=1 j=1

where û : R → R|X | . This is a |X |-dimensional first-order ODE and can be solved using any of the ODE
methods discussed in section 1.
A very common case is that both the space discretization and the time steps in the ODE solver are
based on a finite-difference discretization (this is for example true for the example schemes d and d2
given above and an Euler method for the time step). Such a solution method is called a finite difference
method.

2.2.2 Method of Lines for Elliptic PDEs

Elliptic PDEs are often steady-state/stationary versions of some dynamics that are represented by a
parabolic PDE.6 If one has a good reason to believe that these dynamics eventually converge to the
stationary solution,7 then it may be possible to solve the elliptic equation by solving the underlying
(dynamic) parabolic equation until convergence.
If there is no time-dependent representation of the system readily available, one can also convert an
elliptic PDE into a parabolic PDE by adding a false time transient: take a generic PDE of the type (5)
that does not contain a time variable and replace the 0 on the right-hand side by ∂u ∂t . If the stationary
∂u
solution has been found, then ∂t = 0 and u solves the original equation, otherwise the modified equation
implies a time drift of u. Whether this procedure converges or not depends on the problem at hand.
Sometimes, it may be easier to solve elliptic PDEs directly without introducting an additional time
dimension. This is regularly true for linear problems, but may not be the case in nonlinear problems.8
6 Examples are a (stationary) value function of an infinite-horizon decision maker as the (backward) limit of (nonsta-

tionary) finite-horizon value functions as the time horizon becomes large or the invariant density of an equilibrium model
as the (forward) limit of the equilibrium state distribution as the system runs for a long time.
7 E.g. when iterating backward on a Bellman equation with discounting, intuition (and in many cases theory) suggests

that the process should converge to the infinite-horizon solution. The same is true for continuous-time value function
iteration based on the HJB equation.
8 Even a more “direct” method must solve a nonlinear equation system which usually requires some iterative method.

Designing an iterative algorithm based on a related parabolic PDE problem may be superior to a generic Newton-type
equation solver, if one can exploit additional (economic/physical/other) insights about the problem at hand to understand
convergence behavior (economic insights often imply that a value function iteration of a single decision maker should
converge to an infinite-horizon value function, physical insights may imply that something that looks like a heat diffusion
process should eventually reach an equilibrium).

6
2.3 A Simple Linear Example
2.3.1 Implicit and Explicit Euler Methods for the Heat Equation
To make the abstract considerations of the last subsection more concrete and add some details on issues
that come up in practice, consider the following simple linear parabolic PDE (one-dimensional heat
equation)
∂u ∂2u
=a 2 (7)
∂t ∂x
with some parameter a > 0. Suppose this is given together with a initial condition u(x, 0) = u0 (x)
defined on the real line. To solve this problem using a finite difference method, one typically proceeds
as follows:
2
1. Choose a space discretization, x0 < x1 < · · · < xN and replace ∂∂xu2 (xi , t) on the right-hand side of
the PDE for all i by an expression that only depends on the vector (u(x0 , t), . . . , u(xN , t)).
Example: choose the number N of gridpoints, arbitrary boundaries x0 < xN , and use a uniform
2
grid, xi = x0 + i∆x with ∆x = xNN−x0 . Replace ∂∂xu2 (xi , t) by a three-point finite difference
approximation
∂2u u(xi+1 , t) + u(xi−1 , t) − 2u(xi , t)
2
(xi , t) ≈ a .
∂x ∆x2
2. At the end points x0 and xN impose some artificial boundary conditions,9 ideally using insights
about the problem at hand.
Example: choose constants consistent with the initial value u(x0 , t) = u0 (x0 ), u(xN , t) = u0 (xN ),
consistent with long-run decay to 0, u(x0 , t) = u(xN , t) = 0, or something in between (e.g. letting
the function on the boundary decay from the initial condition to 0). In the following let uL (t),
uR (t) be some generic boundary functions.
3. Solve the resulting N − 1-dimensional ODE initial value problem for the vector function û(t) =
(u(x1 , t), . . . , u(xN −1 , t)) using a suitable ODE method, for example implicit or explicit Euler.
The last step is discussed in more detail. Given the choices in the example, the ODE to be solved is
dûi (t) ûi+1 (t) + ûi−1 (t) − 2ûi (t)
=a i = 1, ..., N − 1 (8)
dt ∆x2
and û0 (t) = uL (t), ûN (t) = uR (t) under the initial condition ûi (t) = u0 (xi ) for all i = 1, ..., N − 1.
Solving this ODE problem using an explicit Euler discretization in time with time step width ∆t
means solving in each step the equation
ûi (tj+1 ) − ûi (tj ) ûi+1 (tj ) + ûi−1 (tj ) − 2ûi (tj )
=a
∆t ∆x2
for ûi (tj+1 ). The solution has an explicit recursive form (just as expected from the discussion in section
1),
a∆t
ûi (tj+1 ) = ûi (tj ) + (ûi+1 (tj ) + ûi−1 (tj ) − 2ûi (tj )) . (9)
∆x2
9 If the problem is stated on a compact space domain and has actual boundary conditions, one should use these. But if

the domain is unbounded like R, additional boundary conditions have to imposed to transform the problem to a compact
space domain.

7
 For an implicit Euler discretization in time one has to solve for each j the equation system

ûi (tj+1 ) − ûi (tj ) ûi+1 (tj+1 ) + ûi−1 (tj+1 ) − 2ûi (tj+1 )
=a , i = 1, ..., N − 1
∆t ∆x2
Rearranging, writing the system in matrix form and imposing the boundary conditions yields the linear
equation
a∆t
   
 a∆t a∆t
 û1 (tj+1 ) û1 (tj ) + ∆x 2 uL (tj+1 )
1 + 2 ∆x2 − ∆x2
 − a∆t2 a∆t a∆t   û2 (tj+1 )   û2 (tj )
   
∆x 1 + 2 ∆x 2 − ∆x 2
.. .


 .. .. ..

  .  =
 
 .
. .

 . . .    
a∆t a∆t
ûN −2 (tj+1 )  ûN −2 (tj ) 
− ∆x2 1 + 2 ∆x2 a∆t
ûN −1 (tj+1 ) ûN −1 (tj+1 ) + ∆x2 uR (tj+1 )
(10)
While the equation system (10) is certainly harder to solve than the explicit recursion (9), it is usually not
too costly to solve this equation system even for large N , because one can exploit the special tridiagonal
structure of the coefficient matrix. This structure also means that the matrix is very sparse for large
N (most of its entries are 0) and there is no need to store a dense matrix with (N − 1)2 entries.10 To
work with sparse matrices in Matlab, check the function documentation of the functions sparse (generic
creation function), sparsealloc (allocating space for sparse matrix to be populated later), spdiags
(defining sparse matrices with a particular diagonal structure) and speye (sparse identity).

2.3.2 Stability Considerations

Through the ODE representation (8) the stability insights from Problem 4 of Problem Set 1 can be used
to assess the stability of the two methods. As you have seen in the numerical examples there, a failure
of stability can lead to bad error propagation properties of the algorithm and substantially distort the
numerical solution.
The system (8) can be written in matrix form,

dû(t)
= Aû(t) + b(t)
dt
with    
−2 1 uL (t)
1 −2 1   0 
a  1 −2 1
 1  .. 

A= , b(t) = .
 
∆x2  ∆x2  . 
 
.. .. .. 
 . . .  0 
1 −2 uR (t)

Ignoring the constant b(t), the associated homogeneous equation dû(t)

dt = Aû(t) looks structurally identical
to the one-dimensional test problem y 0 = λy analyzed in Problem 3. Indeed, the numerical stability of a
scheme for the vector ODE is related to numerical stability of the (same) scheme for the test equations
y 0 = λy, if λ are the eigenvalues of the matrix A. The matrix A has only negative eigenvalues, so by the
analysis of Problem 4 one would expect the explicit Euler scheme to be stable, if 1 + λ∆t lies inside the
10 The tridiagonal structure is a special case for n = 1 (one space dimension). In multiple space dimensions, the equation

matrix will still be sparse, but not tridiagonal.

8
unit circle for all eigenvalues
λ of A.

Here, it is only relevant to check the smallest eigenvalue, which is
2a N −1 4a 11
given by λ = ∆x 2 cos π N − 1 ≈ − ∆x2 for N large. Thus, the explicit Euler scheme is stable, if

a∆t 1
2
≤ .
∆x 2
This condition severely limits the size of the time step: whenever one refines the space grid by increasing
the number of grid points by a factor m, the number of grid points in the time grid has to increase by
a factor m2 for the method to remain stable (at least asymptotically for large m). This is particularly
bad, if one solves a parabolic, time-dependent problem only to get to the stationary solution of a related
elliptic problem, because then one would like to take large time steps (for fast convergence), but have a
fine space discretization (for accuracy of the stationary solution).
Because the implicit Euler scheme for ODEs is A-stable, a PDE algorithm based on implicit time
steps as in equation (10) is stable regardless of the size of the eigenvalues of A, as long as they remain all
negative. In this sense an implicit time discretization is often preferrable to an explicit one for problems
with a large diffusion term (large a).12

2.3.3 Advection Terms/First-order Derivatives

Additional care must be taken in the presence of first-order derivatives in the PDE, even for the linear
case. To illustrate this point, consider the simple one-dimensional advection equation
∂u ∂u
=b . (11)
∂t ∂x
The heat equation (7) models undirected movements of particles through diffusion or movement of prob-
ability mass through Brownian-type fluctuations, whereas equation (11) models a directed flow/stream
of particles (say in a moving fluid) or the movement of probability mass through (deterministic) time
drift:
11 Details of the eigenvalue computation are omitted here.
12 While the discussion was limited to the simple heat equation (7), these insights apply more generally.

9
 Diffusion Advection

In the physical example, suppose u(x, t) describes some characteristic, say the mass, of the particle
located at point x at time t. For b > 0, the equation describes a movement to the left at speed b, implying
that the very same particle at location x at time t must be located at x − b∆t at time t + ∆t. If there is
no interaction between different particles that changes the mass of a particle, then the mass observable
at location x − b∆t at time t + ∆t should exactly equal the mass observable at location x at time t, i.e.
u(x, t) = u(x − b∆t, t + ∆t). This reasoning implies u(x, t) = u(x + bt, 0) = f (x + bt) with some function
f and it is easy to check that for any differentiable f , u(x, t) = f (x + bt) solves indeed equation (11).
Now consider an explicit time discretization of (11),
u(x, t + ∆t) = u(x, t) + ∆t · bux (x, t). (12)
Using u(x, t + ∆t) = f (x + bt + b∆t) = u(x + b∆t, t), this implies
u(x + b∆t, t) − u(x, t)
ux (x, t) = .
b∆t
So if one was to choose ∆x = b∆t and approximate ux (x, t) in (11) by right first differences,
u(x + ∆x, t) − u(x, t)
ux (x, t) ≈ ,
∆x
then the Euler scheme (11) would give exactly the correct solution. If one had used instead left differences
u(x, t) − u(x − ∆x, t)
ux (x, t) ≈ ,
∆x

10
then the Euler scheme (12) would imply (still for ∆x = b∆t) u(x, t+ ∆t) = 2u(x, t) − u(x − ∆x, t). To the
extent that u(x, t) − u(x − ∆x, t) ≈ u(x + ∆x, t) − u(x, t), this does not matter too much, but if there is a
substantial difference between the two (because there is substantial curvature in u), the left differencing
scheme may introduce a significant error (a similar point applies to central differences). Of course, also
the right differences do not lead to the exact solution, if ∆x 6= b∆t, but one would nevertheless expect
them to yield a better approximation, because one still looks into the right direction when searching for
the time t position of the particle that is located at x at time t + ∆t.
The following figure illustrates what could happen in a numerical solution based on an Euler scheme
with left differences to approximate first derivatives:

left differences (solid) vs true solution (dashed)

t=0 t
t=1 t
t=2 t
t=3 t

The discussion so far focused on the case b > 0 and concluded that right differences are the superior
choice. Of course, everything is symmetric in the orientation of the real axis and consequently for b < 0
the above discussion would have concluded that left differences are the superior choice. These insights
suggest the following simple heuristic may be reasonable in the design of numerical schemes even beyond
this simple example:
Upwind scheme: If (in some PDE) the coefficient multiplying ∂u ∂x is positive in the grid point (ti , xi ),
∂u
use right differences to approximate ∂x (ti , xi ), if the coefficient multyplying ∂u
∂x is negative in (ti , xi ), use
left differences to approximate ∂u
∂x (ti , xi ).13

13 Moregenerally, if the PDE is of the form (6) with G nonlinear in ∂u∂x

instead of a positive/negative coefficient the
requirement would be a positive/negative derivative of G with respect to ∂u
∂x
.

11
2.4 Monotone Schemes
Upwind schemes are a special case of schemes that satisfy a certain monotonicity property that turns out
to be key for results on the convergence of numerical schemes (to viscosity solutions). In particular, by
the Barles-Souganidis theorem under certain conditions on the (parabolic/elliptic) equation, a numerical
scheme converges, if it is consistent, stable and monotone.14 Here, the discussion will be restricted to the
monotonicity requirement. The Barles-Souganidis result applies to generic parabolic PDEs of the form
∂u
= F (x, t, u, Du, D2 u)
∂t
under the requirement that F is elliptic, which means in this case that for two symmetric matrices A,
B the inequality A ≤ B (in the sense that B − A is positive semidefinite) implies F (x, t, y, z, A) ≤
F (x, t, y, z, B) for all x, t, y, z.15 A numerical scheme for this PDE is said to be monotone, if it satisfies
a similar monotonicity condition as F .
Let S be a function that describes the numerical scheme in the following sense: for each grid point
(x, t), the value û(x, t) is defined through the scheme by the equation
S(x, t, û(x, t), (û(ξ, τ ))(ξ,τ )6=(x,t) ) = 0,
where (û(ξ, τ ))(ξ,τ )6=(x,t) is the vector of approximate solution values on all grid points (ξ, τ ) other than
(x, t). Examples of such functions are (here, ûi,j = û(xi , tj ))
ûi−1,j + ûi+1,j − 2ûi,j ûi,j+1 − ûi,j
S ee (xi , tj+1 , ûi,j , (ûi0 ,j 0 )(i0 ,j 0 )6=(i,j) ) = a 2
− (13)
∆x ∆t
for an explicit Euler scheme for equation (7) and
ûi−1,j+1 + ûi+1,j+1 − 2ûi,j+1 ûi,j+1 − ûi,j
S ie (xi , tj+1 , ûi,j , (ûi0 ,j 0 )(i0 ,j 0 )6=(i,j) ) = a
2
− (14)
∆x ∆t
for an implicit Euler scheme for the same equation. A scheme S is called monotone, if it is (weakly)
increasing in its last argument (the vector of function values other than at the current grid point).16
This section is concluded by checking the monotonicity property for some simple numerical schemes:
1. The scheme S ee defined in (13) for the heat equation:
Rearranging yields
 
a a 1 2a 1
S ee (· · · ) = 2
ûi−1,j + ûi+1,j + − ûi,j − ûi,j+1 .
∆x ∆x2 ∆t ∆x2 ∆t
The coefficients in front of ûi−1,j and ûi+1,j are always positive, the coefficient in front of ûi,j+1 can
be ignored (this is the value this equation of the scheme is solving for), so the scheme is monotone,
if and only if the coefficient in front of ûi,j is nonnegative, which is equivalent to
1 2a a∆t 1
− ≥0⇔ ≤ .
∆t ∆x2 ∆x2 2
This condition happens to coincide with the stability condition for the explicit Euler scheme.
14 Convergence here does not mean that some (false) time transient method based on a parabolic equation converges to

a (stationary) solution of an elliptic equation (like in “convergence of value function iteration”), but that the scheme for
the PDE converges to the true solution of the full PDE (including transitory dynamics) as grids are refined.
15 For an equation of the form (6) the definition of ellipticity (for the right-hand side) given above is sufficient for the

condition stated here to be satisfied.

16 “weakly decreasing” works, too: one can just replace S by −S.

12
 2. The scheme S ie defined in (14) for the heat equation:
Rearranging yields
 
ie a a 1 1 2a
S (· · · ) = ûi−1,j+1 + ûi+1,j+1 + ûi,j + − − ûi,j+1 .
∆x2 ∆x2 ∆t ∆t ∆x2
Again, the coefficient in front of ûi,j+1 is irrelevant for monotonicity and all other coefficients are
positive. Consequently, the implicit Euler scheme for the heat equation is always monotone.
3. Consider three explicit schemes for the advection equation (11)
ûi,j − ûi−1,j ûi,j+1 − ûi,j
S l (· · · ) = b −
∆x ∆t
c ûi+1,j − ûi−1,j ûi,j+1 − ûi,j
S (· · · ) = b −
2∆x ∆t
r ûi+1,j − ûi,j ûi,j+1 − ûi,j
S (· · · ) = b −
∆x ∆t
It is easy to see that S l can never be monotone for b > 0. If b ≤ 0, S l is monotone, if and only if
b 1 −b∆t r
∆x + ∆t ≥ 0 ⇔ ∆x ≤ 1. Symmetrically, S can never be monotone for b < 0 and if b ≥ 0, it
b∆t
is monotone, if and only if ∆x ≤ 1. Clearly, the scheme S c based on central differencing is never
monotone (unless b = 0).
4. You should check yourself that for a linear advection-diffusion equation

∂u ∂2u ∂u
= a(x, t) 2 + b(x, t) + c(x, t)
∂t ∂x ∂x
an implicit Euler method that uses an upwind scheme for first derivatives is always monotone.

2.5 A Hybrid Implicit-Explicit Method for Nonlinear Equations

The discussion so far has concluded that implicit schemes are preferrable, but has restricted attention
to linear equations. For nonlinear problems, an implicit scheme requires to solve a (high-dimensional)
nonlinear equation in each step. While sometimes feasible, this is often hard.17 A (very heuristic) idea to
overcome this issue is to choose a time discretization that is explicit in the nonlinear parts and implicit
in the linear parts, such that it is still sufficient to solve in each time step a sparse linear equation
system of the form (10). While I am not aware of any rigorous analysis of the numerical properties of
such a scheme, for the economic models studied in this course this procedure works sufficiently well. To
illustrate the idea, consider a quasilinear equation of the form (5), but write the function G in a “more
linear form”,18
n n n
∂u X X ∂2u X ∂u
= aij (x, t, u, Du) (x, t)+ bi (x, t, u, Du) (x, t)+c (x, t, u, Du) u(x, t)+d(x, t, u, Du)
∂t i=1 j=1
∂xi ∂xj i=1
∂xi

17 And it may introduce an additional source of instability, if the equation system has multiple solutions and the solver

finds the “wrong” one for some grid values of x.

18 This can be done in an arbitrary way, but one could also use a Taylor expansion of G. In a particular model, there is

often a relatively natural way how to do this.

13
The idea is now to use an explicit time discretization for the coefficients a, b, c, d,

âij (x, tk ) := aij (x, tk , u(x, tk ), Du(x, tk ))

b̂i (x, tk ) := bi (x, tk , u(x, tk ), Du(x, tk ))
ĉ(x, tk ) := c (x, tk , u(x, tk ), Du(x, tk ))
ˆ tk ) := d (x, tk , u(x, tk ), Du(x, tk ))
d(x,

but an implicit time discretization for the remaining linear parts,

n n n
u(x, tk+1 ) − u(x, tk ) X X ∂2u X ∂u ˆ tk )
= âij (x, tk ) (x, tk+1 )+ b̂i (x, tk ) (x, tk+1 )+ĉ(x, tk )u(x, tk+1 )+d(x,
tk+1 − tk i=1 j=1
∂xi ∂xj i=1
∂xi

After a suitable space discretization, this leads to a (sparse) linear equation system for the time step.
The hope is that for sufficiently small time steps the error introduced by the explicit evaluation of the
coefficient functions a, b, c, d is not too large.

3 Differentiation Formulas
Given a grid X = {x0 , ..., xN } ⊂ R, x0 < · · · < xN and a vector of function values û = (ûi )i=1,...,N =
(u(x))x∈X on that grid, what is a good way to compute derivatives u0 and u00 of the unknown function
using just information available from the vector û?19 There are mainly two ways to approach this
question:

1. Choose a smooth global approximation function v and approximate the unknowns u0 (xi ) and u00 (xi )
by v 0 (xi ) and v 00 (xi ). This approach encompasses many specific methods, for example:
(a) Choose for v a sum of smooth basis functions with global support and interpolate such that
v solves v(xi ) = u(xi ) for i = 0, ..., N . Typical choices for v are polynomials or trigonometric
polynomials. This is called a spectral method with collocation.
(b) Choose for v a sum of basis functions that have only small local support and interpolate such
that v solves v(xi ) = u(xi ) for i = 0, ..., N . An example would be spline interpolation. This
is called a finite element method with collocation.
In these approaches the derivative at every grid point xi usually depends on the full vector û. They
may thus lead to dense equation systems in implicit time steps.

2. For each reference grid point xi choose only a small set of adjacent grid points Y(xi ) = {y1 , ..., yk } ⊂
X \ {xi } and use only local information u(xi ), u(y1 ), ..., u(yk ) around xi , ignoring all other compo-
nents of û. The advantage is that this makes equations to be solved in implicit time steps sparser.
There are two ways to work with the restricted information on Y(xi ) ∪ {xi }:

(a) Use any global approximation method on the smaller grid Y(xi ) ∪ {xi }, e.g. a polynomial
interpolation.
19 The focus on the one-dimensional case is here for ease of exposition only. Everything applies analogously to multiple

dimensions.

14
 (b) Use a Taylor expansion of the unknown u to design a finite difference scheme that is accurate
to the highest possible order.
Here, the method 2(b) is discussed in more detail. It leads to idenical results as method 2(a) with
polynomial interpolation. The goal is to find a linear differentiation formula
k
X
µ
d u(xi ) = α0µ u(xi ) + αjµ u(yj )
j=1

to approximate u(µ) (xi ) (µ = 1, 2). A m-th order Taylor expansion of u on a point y ∈ Y(xi ) around xi
yields
m ν
X (y − xi ) (ν) m
u(y) = u(xi ) + u (xi ) + o ((y − xi ) ) .
ν=1
ν!
and substituting this into the differentiation formula implies
   
k m k ν  
X X X (yj − x i )  u(ν) (xi ) + o max (yj − xi )m .
dµ u(xi ) = α0µ + αjµ  u(xi ) +  αjµ (15)
j=1 ν=1 j=1
ν! j

The true derivative satisfies

µ−1
X ∞
X
u(µ) (xi ) = 0 · u(ν) (xi ) + 1 · u(µ) (xi ) + 0 · u(ν) (xi ) (16)
ν=0 ν=µ+1

In order to maximize the order of convergence to the true derivative as maxj (yj − xi ) → 0 (the order of
consistency of the approximation), it makes sense to choose the unknown k + 1 coefficients α0µ , α1µ , . . . ,
αkµ in a way to equalize the first k + 1 lowest order terms in equations (15) and (16). This results in the
linear equation system
k
X
α0µ + αjµ = 0
j=1
k
(
ν
X (yj − xi ) 1, ν=µ
αjµ =
j=1
ν! 0, ν ≤ k, ν 6= µ

that can be solved for α0µ , α1µ , . . . , αkµ . Obviously, this only makes sense for µ ≤ k.
Examples:
• if Y(xi ) = {xi−1 , xi+1 } and the grid is uniform, xi − xi−1 = xi+1 − xi = ∆x, solving the coefficient
equations leads to
u(xi+1 ) − u(xi−1 )
d1 u(xi ) = ,
∆x
u(xi+1 ) + u(xi−1 ) − 2u(xi )
d2 u(xi ) = ,
∆x2
the standard central differencing formulas for the first and second derivatives.

15
• if Y(xi ) = {xi−1 }, then (also for non-uniform grids)

u(xi ) − u(xi−1 )
d1 u(xi ) = ,
xi − xi−1

that is standard left differences. Similarly, Y(xi ) = {xi+1 } leads to right differences.

16