Advanced Numerical Analysis

Lecture Notes
Prof. Daniel Kressner
EPFL / SB / MATHICSE / ANCHP
Spring 2015

February 19, 2015

Remarks
• Although not necessary, you may find it helpful to complement these lecture
notes with additional literature. These lecture notes are based on material
from

[DB] P. Deuflhard and F. Bornemann. Scientific computing with ordinary

differential equations. Texts in Applied Mathematics, 42. Springer, 2002.
[E] F. Eisenbrand. Lecture notes on Discrete Optimization, 2013. Available
from http://disopt.epfl.ch/Opt2013.
[HLW] E. Hairer, Ch. Lubich, and G. Wanner. Geometric numerical integra-
tion. Structure-preserving algorithms for ordinary differential equations.
Springer Series in Computational Mathematics, 31. Springer, 2010.
[HNW] E. Hairer, S. P. Nørsett, and G. Wanner. Solving ordinary differen-
tial equations. I. Nonstiff problems. Second edition. Springer Series in
Computational Mathematics, 8. Springer, 1993.
[HW] E. Hairer and G. Wanner. Solving ordinary differential equations. II.
Stiff and differential-algebraic problems. Second revised edition. Springer
Series in Computational Mathematics, 14. Springer-Verlag, Berlin, 2010.
[K] C. T. Kelley. Iterative Methods for Optimization. SIAM, 1999.
[Kr] J. Krieger. Equations Différentielles Ordinaires, 2014. Lecture notes
available from http://pde.epfl.ch/page-44573-en.html.
[N] Yu. Nesterov. Introductory Lectures on Convex Optimization. A Basic
Course, 2004.
[NW] J. Nocedal and S. J. Wright. Numerical optimization. Second edi-
tion. Springer Series in Operations Research and Financial Engineering.
Springer, 2006.
[UU] M. Ulbrich and S. Ulbrich. Nichtlineare Optimierung, Birkhäuser Ver-
lag, 2012.
Please note: None of these books is needed to prepare for the exam. These
lecture notes are entirely sufficient.

• Sections or paragraphs marked by a ? contain complementary material that

is not part of the exam. You may skip reading these sections, but it may be
worthwhile not to do so.
• If you have any suggestions or spot mistakes, please send a message to

[email protected]
or [email protected]
Chapter 1

Introduction and Review

The first part of this course starts where the course Analyse Numérique ended: the
numerical solution of ordinary differential equations (ODEs). In this first chapter,
we will therefore very briefly review (and slightly extend) some of the material
covered in Analyse Numérique.
In general, an initial value problem [problème de Cauchy] takes the form
ß 0

y (t) = f t, y(t) t ∈ I,
(1.1)
y(t0 ) = y0 ,

where:

• t ∈ R denotes time.

• I is a real interval containing the initial time t0 .

• y(t) ∈ D ⊂ Rd is the desired solution vector at time t. It contains d time-

dependent functions y1 (t), . . . , yd (t). Often, y(t) is called the state [état] of
the ODE. The open set D containing all admissible states is called the phase
space [espace des phases] and Ω := I × D ⊂ Rd+1 is called the augmented
phase space.

• y0 (t) is the entry-wise derivative of y with respect to time:

Ö 0 è Ö ∂ è
y1 (t) ∂t y1 (t)
y0 (t) = .. = .. .
. .
yd0 (t) ∂
∂t yd (t)

• f : Ω → Rd is a continuous function depending on time and state. If there is

no direct dependence on time (for example, f (t, y(t)) = y(t)2 ) then the ODE
is called autonomous [équation différentielle autonome] and we can write
f (y(t)) instead of f (t, y(t)).

3
4 Version February 19, 2015 Chapter 1. Introduction and Review

1.1 Existence and uniqueness of solutions

In the following, we briefly recall classical results on the existence of a solution
y ∈ C 1 (I, Rd ) to the initial value problem (1.1). See Lectures 2 and 3 of [Kr].

Theorem 1.1 (Peano existence theorem) Consider an initial value prob-

lem (1.1) with f ∈ C 0 (Ω, Rd ) and initial data (t0 , y0 ) ∈ Ω. Then there is at
least one solution y ∈ C 1 ([t0 , t0 + α]) for some α > 0.

Uniqueness requires a condition on f that is stronger than continuity.

Definition 1.2 Consider a mapping f ∈ C(Ω, R) with Ω = I × D ⊂ R × Rd .

Then f is called locally Lipschitz continuous [localement lipschitzienne] on Ω
with respect to the state variable if for any compact subset K ⊂ Ω there exists a
Lipschitz constant LK such that

kf (t, y) − f (t, y
e )k ≤ LK ky − y
ek ∀(t, y), (t, y
e ) ∈ K.

In the definition above, k · k denotes an arbitrary vector norm.

Theorem 1.3 (Picard–Lindelöf theorem) Consider an initial value prob-

lem (1.1) with f ∈ C 0 (Ω, Rd ) locally Lipschitz continuous on Ω with respect
to the state variable, and (t0 , x0 ) ∈ Ω. Then there exists α > 0 such that the
following holds: There exists one and only one solution y ∈ C 1 ([t0 , t0 + α]) of
the initial value problem on the interval [t0 , t0 + α].

Often, solutions can be continued beyond the interval [t0 , t0 + α] guaranteed by

Theorem 1.3. For harmless ODEs, this continuation may extend until T = ∞. If
the extension of a solution y(t) breaks down at some point T then because of one
of the following two reasons:

1. Blow up of the solution: lim ky(t)k = ∞.

t→T
t<T

2. Collapse of the solution: lim dist (t, y(t)), ∂Ω = 0.
t→T
t<T

First, an example for blow up.

Example 1.4 Let Ω = R × R and consider

y0 = y2 , y(0) = 1.
1.1. Existence and uniqueness of solutions Version February 19, 2015 5

1
Then, by separation of variables, we obtain the solution y(t) = 1−t on the interval
] − ∞ < t < 1. Hence, the solution blows up at time T = 1. Let us apply the
Matlab function ode23, which uses an adaptive time stepping strategy, to this
IVP:
Matlab
ode23(@blowup,[0,2],1)

function f = blowup(t,y)
f = y.^2;

Figure 1.1 shows that the time of blow up is detected quite accurately, and the
method actually refuses to continue beyond T = 1, delivering the following error
message:

Warning: Failure at t=1.001616e+00. Unable to meet integration

tolerances without reducing the step size below the smallest
value allowed (3.552714e-15) at time t.

In contrast, the Euler method fails to accurately detect the time of blow up. 
13
x 10 200
7 10

6
150
10
5

4 100
10
3

2 50
10
1
0
0 10
0 0.5 1 0 0.5 1 1.5 2

Figure 1.1. Left plot: ode23 applied to Example 1.4. Right plot: Euler
method with step size h = 0.02 applied to Example 1.4.

Collapse happens when the solution approaches the boundary of the augmented
phase space Ω in finite time. It is important to note that this notion makes most
sense when Ω has been maximally chosen, according to the properties of f .

Example 1.5 Let Ω = R×] − ∞, 0[ and consider

y 0 = −y −1/2 , y(0) = 1.

Up to t < 2/3 the unique solution is given by y(t) = (1 − 3t/2)2/3 . At t = 2/3,

however, the trajectory runs into the boundary of the phase space and collapses
due to the singularity of f at 0. 
6 Version February 19, 2015 Chapter 1. Introduction and Review

In the previous example, the collapse could be easily detected as the solution ap-
proaches −∞ as t → 2/3. A more subtle situation occurs when the solution remains
bounded even when approaching the boundary of the phase space.

Example 1.6 Let Ω = R×] − ∞, 0[ and consider

y 0 = sin(1/y) − 2, y(0) = 1.

It can be shown that the solution remains bounded. The function sin(1/y) is con-
tinuously differentiable on the phase space, but cannot be extended continuously
to a larger domain. It turns out that the solution collapses at time T ≈ 0.76741.
Figure 1.2 shows that both ode23 and the Euler method fail to notice the collapse.

1 1

0.8 0.8

0.6 0.6

0.4 0.4

0.2 0.2

0 0

−0.2 −0.2

−0.4 −0.4
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1

Figure 1.2. Left plot: ode23 applied to Example 1.6. Right plot: Euler
method with step size h = 0.01 applied to Example 1.6.

(At least, the very small steps taken by ode23 when approaching T indicate a prob-
lematic behavior.) In fact, the highly oscillatory behavior causing the collapse of
the solution is “averaged out” by the numerical methods. This is a good example
where mathematical analysis cannot be replaced by numerics. 

1.2 Simple one-step methods

The (forward) Euler method [méthode d’Euler (explicite/progressive)] is the
simplest numerical method for solving an IVP. Given a small step size h, the idea is
to replace ẏ(t) in the differential equation ẏ(t) = f (t, y(t)) by the forward difference.
At t = t0 , this yields
1

y1 − y0 = f (t0 , y0 ),
h
where t1 = t0 + h and y1 ≈ y(t1 ). Rearranging this equation gives

y1 = y0 + hf (t0 , y0 ). (1.2)

Repeating this procedure at t1 , t2 , . . . gives the following algorithm:

1.3. Consistency of one-step methods Version February 19, 2015 7

1. Choose step size h = (T − t0 )/N .

2. for j = 0, 1, . . . , N − 1 do

tj+1 = tj + h, yj+1 = yj + hf (tj , yj ). (1.3)

For one-step methods, it is sufficient to specify the first step (1.2); the iteration (1.3)
immediately follows.
The backward Euler method [méthode d’Euler (implicite/retrograde)] is ob-
tained in a similar way as the forward Euler method, but using the backward dif-
ference instead of the forward difference. At t = t1 , this yields

1

y1 − y0 = f (t1 , y1 ),
h
where t1 = t0 + h and y1 ≈ y(t1 ). Rearranging this equation gives

y1 = y0 + hf (t1 , y1 ).

In contrast to forward Euler, we need to solve a system of nonlinear equations to

determine y1 ! The solution of such nonlinear systems will be discussed in some
detail for more general implicit methods in Chapter 3.
As a final for a simple one-step method, we mention the explicit trapezoidal
method
h

y1 = y0 + f (t0 , y0 ) + f (t1 , y0 + hf (t0 , y0 ) . (1.4)
2

1.3 Consistency of one-step methods

To discuss the local error committed by a single step of the methods introduced
above, it suffices to consider the first step:

e(h) = y(t0 + h) − y1 . (1.5)

(Note that e(h) corresponds to the quantity ε1 in Analyse Numérique.) As a mini-

mal requirement, e(h) should converge faster to zero than h, that is

e(h) = o(h).

A method satisfying this property is called consistent. More generally, we have

the following definition.

Definition 1.7 A method has (consistency) order p if ke(h)k = O(hp+1 ) holds

for all sufficiently smooth f .
8 Version February 19, 2015 Chapter 1. Introduction and Review

It is crucial to have the exponent p + 1 (and not p) in the definition of consistency.

The underlying rationale is that we lose one power in the eventual convergence
result because we have to perform O(1/h) steps to reach the endpoint of an interval
of constant length.
In principle, it is a simple matter to determine the order of a method by com-
paring the Taylor expansions of y(t0 + h) and y1 as functions of h. Assuming that
f is sufficiently smooth, we obtain for the forward Euler method that

y(t0 + h) = y(t0 ) + hy0 (t0 ) + O(h2 ) = y0 + hf (t0 , y(t0 )) + O(h2 ),

y1 = y0 + hf (t0 , y(t0 )).

Hence, ke(h)k = O(h2 ) and therefore the forward Euler method has order 1.
For the backward Euler method, y1 (h) = y0 + hf (t0 + h, y1 (h)). By applying
the implicit function theorem, we obtain y10 (0) = f (t0 , y1 (0)) = f (t0 , y0 ). Hence,
we obtain the Taylor expansion

y1 = y0 + hf (t0 , y(t0 )) + O(h2 ),

which implies that the backward Euler method has order 1 as well.
For (reasonable) one-step methods, consistency of order p implies convergence
of order p. This has already been discussed in Analyse Numérique; we will have a
somewhat more elaborate discussion in Section 2.2.2 on this topic.
Chapter 2

Explicit Runge-Kutta
methods

The goal of this chapter is to construct explicit one-step methods of higher order.
By far, the most popular methods of this type are Runge-Kutta methods, which –
as we will see – are behind the Matlab commands ode23 and ode45.

2.1 A first Runge-Kutta method

The starting point for the development of simple Runge-Kutta methods is the re-
formulation of the IVP (1.1) on the interval [t0 , t0 + h] as the integral equation
Z t0 +h
y(t0 + h) = y0 + f (t, y(t)) dt. (2.1)
t0

When approximating the integral by a rectangle with area h×f (t0 , y(t0 )), we obtain

y(t0 + h) ≈ y1 = y0 + hf (t0 , y(t0 )),

which is the forward Euler method, so nothing new is gained. When using the
midpoint rule, we approximate the integral in (2.1) by a rectangle with area h ×
f (t0 + h/2, y(t0 + h/2)), leading to
Å ã
h h
y(t0 + h) ≈ y0 + hf t0 + , y(t0 + ) . (2.2)
2 2

This does not look very promising either, as we do not know the value for y(t0 + h2 ).
In the absence of a better idea, we approximate it by one step of the forward Euler
method with step size h/2. Inserted into (2.2), this leads to Runge’s method:

k1 = f (t0 , y0 )
Å ã
h h
k2 = f t0 + , y0 + k1 (2.3)
2 2
y1 = y0 + hk2 .

9
10 Version February 19, 2015 Chapter 2. Explicit Runge-Kutta methods

On first sight, it appears contradictory to invoke the forward Euler method for de-
signing a higher-order method. However, the crucial point is that k2 gets multiplied
by h, which weakens the impact of the error made when approximating y(t0 + h2 )
within k2 .
As a warm-up to the error analysis of general Runge-Kutta methods, let us have
a look at the Taylor expansion of y1 in (2.3) as a function of h. When attempting
to do so, we have to face the unpleasant situation that we have to expand both
arguments of the function f appearing in k2 . We will therefore need to make use
of the multivariate Taylor expansion.
Remark 2.1 For a function f ∈ C k (Ω, Rd ) with an open set Ω ⊂ Rm , the kth
derivative f (k) (x) at x ∈ Ω applied to h1 , . . . , hk ∈ Rm is defined as
m
X ∂ k f (x)
f (k) (x) · (h1 , . . . , hk ) = h1,i1 . . . hk,ik . (2.4)
i1 ,...,ik =1
∂xi1 · · · ∂xik

Thus,
f (k) (x) : Rm × · · · × Rm → Rd ,
| {z }
k times

is a (symmetric) k-linear map.

For h1 = · · · = hk ≡ h, the sum in (2.4) can be represented in more compact
form. Let α = (α1 , . . . , αm ), where each 0 ≤ αj ≤ k counts the differentiations with
respect to the jth variable xj . By the usual multi-index notation, we define

∂ |α| f ∂ |α| f
|α| := α1 + · · · + αm , := αm , hα := hα αm
1 . . . hm .
1

∂xα ∂xα
1 · · · ∂xm
1

k! k!
Each α corresponds to α! = α1 !···αm ! terms in the sum of (2.4), and hence (2.4) is
equivalent to
X k! ∂ |α| f (x)
f (k) (x) · (h, . . . , h) = hα . (2.5)
α! ∂xα
|α|=k


Theorem 2.2 Let f ∈ C p+1 (Ω, Rd ) with an open set Ω and x ∈ Ω. Then
p
X 1 (k)
f (x + h) = f (x) · (h, . . . , h) + O(khkp+1 )
k!
k=0

for all sufficiently small h ∈ Rm .

Theorem 2.2 is proven by applying the usual univariate Taylor expansion in each
coordinate direction separately and collecting all terms.
2.1. A first Runge-Kutta method Version February 19, 2015 11

Example 2.3 For d = 1 and p = 2, Theorem 2.2 results in

1
f (x + h) = f (x) + f 0 (x) · h + f 00 (x) · (h, h) + O(khk3 )
2
1
= f (x) + ∇f (x) · h + hT H(x)h + O(khk3 ),
2
Ä 2 äm
where ∇f is the gradient and H = ∂x∂i ∂x f
j
is the Hessian [matrice hessienne]
i,j=1
of f . 
For simplicity, we consider the initial value problem (1.1) for an autonomous
differential equation: ß 0

y (t) = f y(t) t ≥ t0 ,
(2.6)
y(t0 ) = y0 .
Let us now recursively construct a third-order Taylor expansion for the exact
solution y(t0 + h). The first-order Taylor expansion is given by
y(t0 + h) = y0 + hy0 (t0 ) + O(h2 ) = y0 + hf (y0 ) + O(h2 ).
Inserting this into the differential equation (2.6) gives
∂
y(t0 + h) = f y(t0 + h) = f y0 + hf (y0 ) + O(h2 )



∂h
= f + hf 0 f + O(h2 ),
where we have used the multivariate Taylor expansion from Theorem 2.2 and omit-
ted the argument y0 to save space. Integration of this relation with respect to h
gives the second-order Taylor expansion
h2 0
y(t0 + h) = y0 + hf + f f + O(h3 ).
2
Let us repeat this process:
h2
Å ã
∂
y(t0 + h) = f y0 + hf + f 0 f + O(h3 )
∂h 2
Å 2 ã
h 1
= f + f 0 hf + f 0 f + f 00 (hf, hf ) + O(h3 )
2 2
2
h
= f + hf 0 f + f 0 f 0 f + f 00 (f, f ) + O(h3 ).


2
By integration, the third-order Taylor expansion follows:
h2 0 h3 0 0
f f f + f 00 (f, f ) + O(h4 ).


y(t0 + h) = y0 + hf + ff+ (2.7)
2 6
This will be compared with the Taylor expansion of y1 , produced by apply-
ing (2.3) to (2.6), as a function of h:
Å ã
h
y1 = y0 + hf y0 + f (y0 )
2
h2 0 h3
= y0 + hf + f f + f 00 (f, f ) + O(h4 ). (2.8)
2 8
12 Version February 19, 2015 Chapter 2. Explicit Runge-Kutta methods

Subtracting this from the expansion (2.7) of the exact solution gives

h3 0 0
h3
e(h) = y(t0 + h) − y1 = f f f + f 00 (f, f ) − f 00 (f, f ) + O(h4 )
6 8
Hence, the local error satisfies ke(h)k = O(h3 ), provided that all second partial
derivatives of f remain bounded. This shows that the order of Runge’s method is
2.

2.2 General form of explicit Runge-Kutta methods

Definition 2.4 A method of the form

k1 = f (t0 + c1 h, y0 )
k2 = f (t0 + c2 h, y0 + ha21 k1 )
k3 = f (t0 + c3 h, y0 + ha31 k1 + ha32 k2 )
..
.
 s−1
X 
ks = f t0 + cs h, y0 + h as` k` ,
`=1
s
X
y1 = y0 + h bi k i ,
i=1

with all coefficients ai` , bi , ci being real numbers, is called an s-stage explicit
Runge-Kutta method.

The coefficients defining a Runge-Kutta method can be organized compactly in a

so-called Butcher tableau:
c1 0 ··· ··· 0
.. ..
c2 a21 . .
c A .. .. ..
:= .. ..
bT . . . . .
cs as1 ··· as,s−1 0
b1 ··· bs−1 bs
Here, A is strictly lower triangular s × s matrix and b, c are vectors of length
s. All explicit one-step methods we have discussed so far are special cases of the
Runge-Kutta method:
0 0 0
0 0
Forward Euler: Explicit Trapezoidal: 1 1 0
1 1 1
0 2 2
2.2. General form of explicit Runge-Kutta methods Version February 19, 2015 13

0 0 0
1 1
Runge’s method: 2 2 0
0 0 1
Another example for an explicit Runge-Kutta method is the classical Runge-
Kutta method (RK4) defined by the tableau

0 0 0 0 0
1 1
2 2 0 0 0
1 1
2 0 2 0 0
1 0 0 1 0
1 2 2 1
6 6 6 6

According to Definition 2.4, this leads to the following algorithm.

Classical Runge-Kutta method:

1. Choose step size h = (T − t0 )/N .

2. for j = 0, 1, . . . , N − 1 do
k1 = f (tj , yj )
k2 = f (tj + h/2, yj + h/2 k1 )
k3 = f (tj + h/2, yj + h/2 k2 )
k4 = f (tj + h, yj + h k3 )
yj+1 = yj + h6 k1 + 2h 2h h
6 k2 + 6 k3 + 6 k4
tj+1 = tj + h
end for

What is so special about the classical Runge-Kutta method? When designing an s-

stage Runge-Kutta method, one goal could be to attain the maximal possible order.
As we will discuss later, there is a certain limit which order can be achieved, but
certainly it cannot become larger than s (see Lemma 2.5 below). One could now
proceed as in Section 2.1 and compare the Taylor expansion of the exact solution
with the Taylor expansion of y1 produced by (2.4). This appears to be a tedious
task for larger values of s, but we will discuss a clever way of organizing it in the
the following section. Matching both expansions up to a certain order leads to a
system of nonlinear equations in the coefficients defining the Runge-Kutta method.
Finding all solutions to this system by hand is a nearly impossible task, except for
tiny s. For example, the case s = 4 is covered in Section II.1 of [HNW], giving
a fairly general parametrization of 4-stage explicit Runge-Kutta methods having
order 4.

2.2.1 Order conditions for Runge-Kutta methods

The ultimate goal of this section is to develop an elegant way that avoids the te-
diousness of Taylor expansions when analyzing the order of a Runge-Kutta method.
Before, we will make some basic observations.
14 Version February 19, 2015 Chapter 2. Explicit Runge-Kutta methods

Lemma 2.5 If an s-stage explicit Runge-Kutta method has order p for all f ∈
C ∞ (Ω, Rd ), then p ≤ s.
Proof. We apply the Runge-Kutta method to the scalar IVP y 0 (t) = y(t), y(0) =
y0 = 1. Then each stage ki is a polynomial of degree at most i − 1 in h. Hence, y1
hs+1
is a polynomial of degree at most s in h. Consequently, the (s + 1)th term (s+1)!
in the Taylor expansion of the exact solution y(h) = eh cannot be matched by y1 .
Hence, the order of the Runge-Kutta method is at most s.

Lemma 2.6 An explicit Runge-Kutta method is consistent for all f ∈ C(Ω, Rd ) if

and only if
Xs
bi = 1. (2.9)
i=1
Proof. The first term in the expansion of y1 takes the form
s
X
y1 = y0 + h bi f (t0 , y0 ) + · · ·
i=1

This matches the first two terms in the Taylor expansion (2.7) if and only if (2.9)
is satisfied.

The analysis greatly simplifies if we only need to consider autonomous IVPs, as

we did in Section 2.1. There is a simple trick to transform (1.1) to autonomous
form by appending t to the variables:
Å ã0 Å ã Å ã Å ã
t 1 t t
= , = 0 . (2.10)
y f (t, y) y t=t y0
0

This trick makes perfect sense if we have a correspondence between the Runge-
Kutta method applied to (2.10) and applied to (1.1). Ideally, the approximation
produced by one step of the Runge-Kutta method applied to (2.10) takes the form
Å ã
t0 + h
.
y1

A Runge-Kutta method having this property is called invariant under autono-

mization.

Lemma 2.7 An explicit Runge-Kutta method is invariant under autonomization if

and only if it is consistent and satisfies

i−1
X
ci = aij , i = 1, . . . , s.
j=1
2.2. General form of explicit Runge-Kutta methods Version February 19, 2015 15

Proof. The stages of the Runge-Kutta method applied to (2.10) take the form

!
Å ã
θi 1
=
 P P 
k̂i f t0 + h j aij θj , y0 + h j aij k̂j
!
1
=
 P P 
f t0 + h j aij , y0 + h j aij k̂j .

P
Clearly, we have k̂i = k̂i for general f if and only if j aij = ci . The approximate
solution is then given by

Å P ã Å P ã
t 0 + h i bi θ i t 0 + h i bi
P = .
y0 + h i bi ki y1

P
Hence, invariance holds if we additionally require that i bi = 1, which – by
Lemma 2.6 – is equivalent to the consistency of the Runge-Kutta method.

Letting e = (1, . . . , 1)T ∈ Rs denote the vector of all ones, the conditions of
Lemma 2.6 and Lemma 2.7 can be compactly written as

bT e = 1, c = Ae.

For the rest of this section, we will assume that these conditions are satisfied and
only consider autonomous IVPs.

Elementary differentials and rooted trees. The biggest problem when attempting
to verify (high) orders of Runge-Kutta methods is to keep track of the terms appear-
ing in the Taylor expansions of the exact and the approximate solutions. As can be
seen from (2.7) and (2.8), these terms are elementary differentials of f . Counting
the number of the various elementary differentials appearing in the expansion is a
combinatorial problem, for which we will use the concept of rooted trees.
Roughly speaking, an elementary differential is a multilinear form that is fully
described by the recursive structure of its arguments, a more precise definition
will be given below. To give a specific example, let us consider the differential
f 000 (f 0 f, f 0 f, f ). We can deduce the presence of the third derivative from the fact
that three arguments need to be provided. To detail the specific structure of the
arguments, we can use rooted trees, such as:
16 Version February 19, 2015 Chapter 2. Explicit Runge-Kutta methods

for f 000 (f 0 f, f 0 f, f )

for f 0 f 00 (f, f ) for f 00 (f 0 f, f )

Each node of the rooted tree corresponds to a derivative of f at y0 , with the order
k of the derivative determined by the number of children. The substitution of the
arguments proceeds recursively starting at the root . Note that the order of the
children is not important, due the symmetry of the multilinear form.
When the root is deleted from a tree β then β decomposes into a forest of k
trees β1 , . . . , βk , whose roots correspond to the k children of . For example:

β1 β2 β3

In the other direction, the tree β can be composed from an unordered tuple of
subtrees:
β = [β1 , . . . , βk ], #β = 1 + #β1 + · · · + #βk ,
where #β denotes the number of the nodes in a tree β. For creating a tree consisting
solely of the root, we use the convention = []. Two examples for the composition
of trees:

= = ,

This decomposition can be continued recursively until the subtrees consist of roots
only. Two examples:

f 0 f 00 (f, f ) corresponds to [[ , ]],

f 00 (f 0 f, f ) corresponds to [[ ], ].

With the notation defined above, we are ready to define elementary differentials.
2.2. General form of explicit Runge-Kutta methods Version February 19, 2015 17

Definition 2.8 The elementary differential f (β) (y0 ) corresponding to a

rooted tree β = [β1 , . . . , βk ] is defined recursively as
 
f (β) (y0 ) = f (k) · f (β1 ) (y0 ), . . . , f (βk ) (y0 ) .

In the definition above, the argument y0 is given for clarity; we will (again) often
omit it in our subsequent considerations.

Remark 2.9 The Hopf algebra of rooted trees does not only play a role in the
theory of Runge–Kutta methods, but also in noncommutative geometry and renor-
malization methods in quantum field theory. Quite remarkably, it was developed
first in the context of numerical analysis, by John Butcher. We refer to [Brouder,
Ch. Trees, renormalization and differential equations. BIT 44 (2004), no. 3, 425–
438] for a very accessible exposition of these connections. 

Taylor expansion of the exact solution. Let us recall the Taylor expansion (2.7):

h2 0 h3 0 0
f f f + f 00 (f, f ) + O(h4 ).


y(t0 + h) = y0 + hf + ff+
2 6

A closer inspection suggests that the terms for hk involve all elementary differen-
tials corresponding to rooted trees with k nodes. Lemma 2.10 below confirms this
impression in the general case. For this purpose, we need to introduce two combi-
natorial quantities. The factorial of a tree β = [β1 , . . . , βk ] is recursively defined
by
β! = (#β)β1 !β2 ! · · · βk !.
Moreover, we define recursively

δβ
αβ = αβ αβ · · · αβk ,
k! 1 2
where δβ denotes the number of different possibilities for associating an an ordered
k-tuple with the unordered k-tuple β = [β1 , . . . , βk ].

Lemma 2.10 Let f ∈ C p (Ω, Rd ) with an open set Ω and y0 ∈ Ω. Then

X h#β
y(t0 + h) = y0 + αβ f (β) (y0 ) + O(hp+1 ).
β!
#β≤p

Proof. By induction over p. For p = 0, the statement trivially holds. We now

show the expansion for p + 1, assuming that it holds for p. The proof proceeds,
as in the analysis of Section 2.1, by inserting the pth order expansion into the
18 Version February 19, 2015 Chapter 2. Explicit Runge-Kutta methods

differential equation and using the multivariate Taylor expansion. Setting h =

X h#β
αβ f (β) , we obtain
β!
#β≤p

p
∂ X 1 (k)
y(t0 +h) = f (y0 +th) = f y0 + h + O(hp+1 ) = f (h, . . . , h)+O(hp+1 ).


∂h k!
k=0

Using the multilinearity of f (k) and omitting terms smaller than hp , we obtain

1 (k)
f (h, . . . , h)
k!
1 X X h#β1 +···+#βk
αβ1 · · · αβk f (k) f (β1 ) , . . . , f (βk )


= ···
k! β1 ! · · · βk !
#β1 ≤p #βk ≤p

1 X h#β1 +···+#βk
αβ1 · · · αβk f (k) f (β1 ) , . . . , f (βk ) + O(hp+1 )


=
k! β1 ! · · · βk !
#β1 +···+#βk ≤p
X #β · h#β−1 δβ
= αβ1 · · · αβk f (β) + O(hp+1 ),
#β≤p+1
β! k!
| {z }
β=[β1 ,...,βk ]
=αβ

where we have used in the last step that the symmetry of f (k) allows to use un-
ordered tuples [β1 , . . . , βk ] instead of ordered tuples. This introduces the extra
factor δβ , which counts the number of ordered tuples associated with [β1 , . . . , βk ].
To sum up, we obtain

∂ X #β · h#β−1
y(t0 + h) = αβ f (β) + O(hp+1 ).
∂h β!
#β≤p+1

Integrating this relation gives

X h#β
y(t0 + h) = y0 + αβ f (β) + O(hp+2 ).
β!
#β≤p+1

This corresponds exactly to the claimed expansion for p + 1.

Examples for the coefficients β! and αβ can be found in Table 2.1.

Taylor expansion of the numerical solution. The expansion of the numerical so-
lution y1 produced by the Runge-Kutta method can also be represented with the
help of rooted trees. For a rooted tree β = [β1 , . . . , βk ], we recursively define the
vector A(β) ∈ Rs via its components:
   
(β)
Ai = A · A(β1 ) · · · A · A(βk ) , i = 1, . . . , s.
i i
2.2. General form of explicit Runge-Kutta methods Version February 19, 2015 19

Table 2.1. Rooted trees up to order 5. Table taken from Page 150 of [DB].

Again, this definition is independent of the order of the children β1 , . . . , βk . Note

that Ö è
1
A ( )
= .. ∈ Rs .
.
1
20 Version February 19, 2015 Chapter 2. Explicit Runge-Kutta methods

Further examples for A(β) can be found in Table 2.1, where the assumed relation
c = Ae has been used extensively to simplify the formulas.
Lemma 2.11 Let f ∈ C p (Ω, Rd ) with an open set Ω and y0 ∈ Ω. Then the ap-
proximation y1 obtained by applying the Runge-Kutta method (Definition 2.4) to
the autononomous IVP (2.6) satisfies
X
y1 = y0 + h#β αβ · bT A(β) f (β) (y0 ) + O(hp+1 ).
#β≤p

Proof. Since
s
X
y1 = y0 + h bi k i ,
i=1
it suffices to show the Taylor expansions
(β)
X
ki = h#β−1 αβ Ai f (β) + O(hp ), i = 1, . . . , s (2.11)
#β≤p

for the stages ki . The proof of (2.11) proceeds by induction over p. It clearly holds
for p = 0 and we now aim to derive (2.11) for p + 1, assuming that it holds for p.
We then have
i−1
!
X
ki = f y0 + h aij kj
j=1
Ñ é
i−1
(β)
X X
=f y0 + aij h#β αβ Ai f (β) + O(hp+1 )
j=1 #β≤p
Ñ é
X  
=f y0 + h#β αβ A · A(β) f (β) + O(hp+1 ) .
i
#β≤p
X  
Setting h = h#β αβ A · A(β) f (β) , the multivariate Taylor expansion com-
i
#β≤p
(k)
bined with the multilinearity of f yield
p
X 1 (k)
ki = f · (h, . . . , h) + O(hp+1 )
k!
k=0
p
X 1 X    
= h#β1 +···+#βk · αβ1 · · · αβk A · A(β1 ) · · · A · A(βk )
k! i i
k=0 #β1 +···+#βk ≤p
 
· · · f (k) · f (β1 ) , . . . f (βk ) + O(hp+1 )
p
X X δβ (β)
= h#β−1 · αβ · · · αβk · Ai f (β) + O(hp+1 )
k! 1
k=0 #β≤p+1
β=[β1 ,...,βk ]

(β)
X
= h#β−1 αβ · Ai f (β) + O(hp+1 ).
#β≤p+1
2.2. General form of explicit Runge-Kutta methods Version February 19, 2015 21

This completes the proof, as the last line corresponds exactly to (2.11) for p + 1.

Order conditions. After all these preparations, we can now combine Lemma 2.10
and Lemma 2.11 to obtain a pleasingly elegant necessary and sufficient condition
for a Runge-Kutta method having order p.

Theorem 2.12 A Runge-Kutta method is of order p if and only if

1
bT A(β) = (2.12)
β!
holds for all rooted trees β of order #β ≤ p.

Proof. The sufficiency of condition (2.12) follows immediately from comparing

the expansions in Lemma 2.10 and Lemma 2.11. For showing that (2.12) is also
necessary, one needs to show that the elementary differentials for different trees are
linearly independent. This is proven, e.g., in Lemma 4.25 of [DB].

Example 2.13 For p = 4, Theorem 2.12 leads to the following conditions:

Ps 1 P 3
1 = i=1 bi 4 = i bi c i
1 P 1 P
2 = i bi ci 8 = i,j bi ci aij cj
1 P 2 , 1 P 2
3 = i bi ci 12 = i,j bi aij cj
1 P 1 P
6 = i,j bi aij cj 24 = i,j,k bi aij ajk ck

It is a simple exercise to verify that RK4 satisfies this conditions and therefore has
order 4. 

From Example 2.13 one may obtain the misleading impression that the number of
conditions grows only slowly with p. In fact, for p = 10 one has 1205 conditions
and for p = 20 one has 20247374 conditions!

2.2.2 Convergence results

By definition, the local error e(h) = y(t0 + h) − y1 of a Runge-Kutta method of
order p satisfies ke(hk ≤ Chp+1 for some constant C.
We are now concerned with analyzing the global error, which is the error of
the computed solution after several steps: y0 , y1 , y2 , y3 , . . .. For this purpose, we
will write
yi+1 = yi + hi Φ(ti , yi , hi ), hi = ti+1 − ti ,
22 Version February 19, 2015 Chapter 2. Explicit Runge-Kutta methods

y(tN )

E1
exact solutions
y0
e1
E2
y1 e2
E3
y2
e3 ..
.
y3
EN = e N
yN

t0 t1 t2 t3 ··· tN = T

Figure 2.1. Lady Windemere’s fan [l’éventail de Lady Windermere].

where Φ is the increment function of the method. Every one-step method can
be written in this form. Our task is to estimate the global error

E = y(T ) − yN , with T = tN . (2.13)

This estimate is found by transporting all local errors ei = y(tj ) − yi , which satisfy

kei k ≤ Chp+1
i−1 , (2.14)

to the end point tN and adding them up. The principle is illustrated in Figure 2.1.

The following lemma can be used to estimate the magnification of the error
during the transport.

Lemma 2.14 Suppose that y(t) and v(t) are solutions to y0 = f (t, y) with initial
values y(t0 ) = y0 and v(t0 ) = v0 , respectively. If

kf (t, v) − f (t, y)k ≤ Lkv − yk (2.15)

then we have the error estimate

kv(t) − y(t)k ≤ eL(t−t0 ) · kv0 − y0 k. (2.16)

0 0
Proof. Integration of v − y = f (t, v) − f (t, y) with respect to t gives
Z t

v − y = v0 − y0 + f (s, v(s)) − f (s, y(s)) ds.
t0
2.2. General form of explicit Runge-Kutta methods Version February 19, 2015 23

Hence, Z t
kv − yk ≤ kv0 − y0 k + L kv(s) − y(s)k ds. (2.17)
t0

We can apply the (continuous) lemma of Gronwall to this situation (see Lemma 2.15
below), with u(t) = kv(t) − y(t)k, ρ = kv0 − y0 k, and w(t) ≡ L. This immediately
gives (2.16).

Lemma 2.15 (Lemma of Gronwall) Let u, w ∈ C([t0 , T ]) be nonnegative func-

tions and ρ ≥ 0. Then
Z t
u(t) ≤ ρ + u(s)w(s) ds for all t ∈ [t0 , T ]
t0

implies the estimate

t
ÇZ å
u(t) ≤ ρ exp w(s) ds for all t ∈ [t0 , T ].
t0

Proof. EFY, cf. Ex.1,3(b).

Using Lemma 2.14, it remains to add the transported local errors in order to
obtain an estimate for the global error.



Theorem 2.16 Let U be a neighborhood of t, y(t) : t0 ≤ t ≤ T such that
the local error estimate (2.14) and the estimate

∂f
≤L (2.18)
∂y

hold in U . Then the global error (2.13) satisfies

C Ä L(T −t0 ) ä
kEk ≤ hp e −1 , (2.19)
L
where h = maxi hi is small enough for the numerical solution to remain in U .

Proof. Let Ei denote the ith local error ei transported to the end point, see
Figure 2.1. Then Lemma 2.14 applied at time ti with initial values yi vs. y(ti )
yields
kEi k ≤ eL(T −ti ) · kei k ≤ CeL(T −ti ) hp+1
i−1 .

Using hp+1 p
i−1 ≤ h hi−1 yields

N
X N
X Z T
L(T −ti )
kEk ≤ kEi k ≤ Chp
e hi−1 ≤ Ch p
eL(T −s) ds.
i=1 i=1 t0
24 Version February 19, 2015 Chapter 2. Explicit Runge-Kutta methods

eL(tN −t)

t
t0 t1 t2 · · · tN −1 tN

Figure 2.2. Lower Darboux sum used in the proof of Theorem 2.16.

In the last inequality, we have used that the sum is actually a lower Darboux sum
[somme de Darboux inférieure] for the integral on the right-hand side, see Figure 2.2.
RT
The elementary fact t0 eL(T −s) ds = L1 eL(T −t0 ) − 1 concludes the proof.

Clearly, Theorem 2.16 indicates that we may run into severe problems (that is,
exponentially growing error) as T − t0 grows. Still, ODEs are routinely integrated
over relatively long time, especially in molecular dynamics and astronomy. However,
this may require the use of special methods, see [HLW].
Remark 2.17 ? As the Lipschitz constant L is positive, Theorem 2.16 always
implies an exponential growth of the error. This is clearly not very satisfactory for
differential equations such as y 0 = −y, where both the solution and the transported
error are damped as t increases. The guilty party is Lemma 2.14, which is too
pessimistic about the effect of transport on the error. We can derive a different
estimate as follows. First, note that

kv(t + h) − y(t + h)k = v(t) − y(t) + h f (t, v) − f (t, y) + O(h2 )

Å ã
∂f
≤ max I + h (t, x) · kv(t) − y(t)k + O(h2 ),
x∈V ∂y
where we applied the mean value theorem and V is a connected neighborhood
containing both v and y. Setting f (t) := kv(t) − y(t)k, we therefore obtain
Ñ é
∂f
m(t + h) − m(t) I + h ∂y (t, x) − 1
≤ max m(t) + O(h). (2.20)
h x∈V h

As h → 0, the factor on the right-hand side features a quantity known as the

logarithmic “norm” of a matrix.
Definition 2.18 Let A ∈ Cn×n . Then
kI + hAk − 1
µ(A) := lim
h→0
h>0
h
2.2. General form of explicit Runge-Kutta methods Version February 19, 2015 25

is called the logarithmic norm [norme logarithmique] of A.

Of course, Definition 2.18 depends on the choice of matrix norm k·k. For the matrix
2-norm, it is not hard to show that µ(A) = 12 λmax (A + A∗ ).
From (2.20) it now follows that the upper Dini derivative m0+ satisfies
Å ã
∂f
m0+ (t) ≤ max µ (t, x) m(t).
x∈V ∂y
Ä ä
∂f
Integrating this inequality gives (2.17) but with L = max µ ∂y (t, x) replacing the
x∈V
Lipschitz constant. As a consequence, we get an improved variant of Theorem 2.16.
The assumption (2.18) is replaced by
Å ã
∂f
µ ≤L
∂y

and the global error estimate (2.19) reads as

C 0 Ä L(T −t0 ) ä
kEk ≤ hp e −1 .
L
Here, C 0 = C for L > 0. For L < 0, one has to adjust the Darboux sum in the
proof of Theorem 2.16 and obtains C 0 = Ce−Lh .