Numerical Analysis II

Xiaojing Ye, Math & Stat, Georgia State University

Spring 2020

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 1
 Section 1

Initial Value Problems for ODEs

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 2
IVP of ODE

We study numerical solution for initial value problem (IVP) of

ordinary differential equations (ODE).
I A basic IVP:
dy
= f (t, y ), for a ≤ t ≤ b
dt
with initial value y (a) = α.

Remark
I f is given and called the defining function of IVP.
I α is given and called the initial value.
I y (t) is called the solution of the IVP if
I y (a) = α;
I y 0 (t) = f (t, y (t)) for all t ∈ [a, b].

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 3
IVP of ODE

Example
The following is a basic IVP:

y 0 = y − t 2 + 1, t ∈ [0, 2], and y (0) = 0.5

I The defining function is f (t, y ) = y − t 2 + 1.

I Initial value is y (0) = 0.5.
I The solution is y (t) = (t + 1)2 − et
2 because:
I y (0) = (0 + 1)2 − e2 = 1 − 12 = 21 ;
0

I We can check that y 0 (t) = f (t, y (t)):

0 et
y (t) = 2(t + 1) −
2
2 et 2
2 et
f (t, y (t)) = y (t) − t + 1 = (t + 1) − − t + 1 = 2(t + 1) −
2 2

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 4
IVP of ODE (cont.)
More general or complex cases:
I IVP of ODE system:


 dy1

 = f1 (t, y1 , y2 , . . . , yn )

 dt



 dy
 2 = f2 (t, y1 , y2 , . . . , yn )
dt for a ≤ t ≤ b

 ..

 .





 dyn
 = fn (t, y1 , y2 , . . . , yn )
dt

with initial value y1 (a) = α1 , . . . , yn (a) = αn .

I High-order ODE:

y (n) = f (t, y , y 0 , . . . , y (n−1) ) for a ≤ t ≤ b

with initial value y (a) = α1 , y 0 (a) = α2 , . . . , y (n−1) (a) = αn .

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 5
Why numerical solutions for IVP?

I ODEs have extensive applications in real-world: science,

engineering, economics, finance, public health, etc.
I Analytic solution? Not with almost all ODEs.
I Fast improvement of computers.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 6
Some basics about IVP

Definition (Lipschitz functions)

A function f (t, y ) defined on D = {(t, y ) : t ∈ R+ , y ∈ R} is
called Lipschitz with respect to y if there exists a constant L > 0

|f (t, y1 ) − f (t, y2 )| ≤ L|y1 − y2 |

for all t ∈ R+ , and y1 , y2 ∈ R.

Remark
We also call f is Lipschitz with respect to y with constant L, or
simply f is L-Lipschitz with respect to y .

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 7
Some basics about IVP

Example
Function f (t, y ) = t|y | is Lipschitz with respect to y on the set
D := {(t, y )|t ∈ [1, 2], y ∈ [−3, 4]}.

Solution: For any t ∈ [1, 2] and y1 , y2 ∈ [−3, 4], we have

 
|f (t, y1 ) − f (t, y2 )| = t|y1 | − t|y2 | ≤ t|y1 − y2 | ≤ 2|y1 − y2 |.

So f (t, y ) = t|y | is Lipschitz with respect to y with constant

L = 2.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 8
 5.2 A set D ⊂ R2 is said to be convex if whenever (t1 , y1 ) and (t2 , y2 ) belong to D, then
Some
((1 − λ)tbasics
1 + λt2 , (1about
− λ)y1 +IVP
λy2 ) also belongs to D for every λ in [0, 1].

In geometric terms, Definition 5.2 states that a set is convex provided that whenever
two points belong to the set, the entire straight-line segment between the points also belongs
to the set. (See Figure
Definition 5.1.) The
(Convex sets we consider in this chapter are generally of the form
sets)
D = {(t, y) | a ≤ t ≤ b and − ∞ < y < ∞} for some constants a and b. It is easy to verify
(seeAExercise R these
2
set D 7)∈ that is convex
sets are if whenever (t1 , y1 ), (t2 , y2 ) ∈ D there is
convex.
(1 − λ)(t1 , y1 ) + λ(t2 , y2 ) ∈ D for all λ ∈ [0, 1].
e 5.1

(t 2, y 2)
(t 2, y 2) (t1, y1)

(t1, y1)

Convex Not convex

5.3 Suppose f (t, y) is defined on a convex set D ⊂ R2 . If a constant L > 0 exists with
Numerical Analysis II – Xiaojing Ye, ! Georgia State University
! Math & Stat, 9
Some basics about IVP

Theorem
∂f
If D ⊂ R2 is convex, and | ∂y (t, y )| ≤ L for all (t, y ) ∈ D, then f is
Lipschitz with respect to y with constant L.

Remark
This is a sufficient (but not necessary) condition for f to be
Lipschitz with respect to y .

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 10
Some basics about IVP

Proof.
For any (t, y1 ), (t, y2 ) ∈ D, define function g by

g (λ) = f (t, (1 − λ)y1 + λy2 )

for λ ∈ [0, 1] (need convexity of D!). Then we have

g 0 (λ) = ∂y f (t, (1 − λ)y1 + λy2 ) · (y2 − y1 )

So |g 0 (λ)| ≤ L|y2 − y1 |. Then we have

Z 1  Z 1 
   
|g (1) − g (0)| =  g 0 (λ) dλ ≤ L|y2 − y1 | dλ = L|y2 − y1 |
0 0

Note that g (0) = f (t, y1 ) and g (1) = f (t, y2 ). This completes the
proof.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 11
Some basics about IVP

Theorem
Suppose D = [a, b] × R, a function f is continuous on D and
Lipschitz with respect to y , then the initial value problem
y 0 = f (t, y ) for t ∈ [a, b] with initial value y (a) = α has a unique
solution y (t) for t ∈ [a, b].

Remark
This theorem says that there must be one and only one solution of
the IVP, provided that the defining f of the IVP is continuous and
Lipschitz with respect to y on D.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 12
Some basics about IVP

Example
Show that y 0 = 1 + t sin(ty ) for t ∈ [0, 2] with y (0) = 0 has a
unique solution.

Solution: First, we know f (t, y ) = 1 + t sin(ty ) is continuous on

[0, 2] × R. Second, we can see
   
 ∂f   2
  = t cos(ty ) ≤ |t 2 | ≤ 4
 ∂y 

So f (t, y ) is Lipschitz with respect to y (with constant 4). From

theorem above, we know the IVP has a unique solution y (t) on
[0, 2].

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 13
Some basics about IVP
Definition (Well-posedness)
An IVP y 0 = f (t, y ) for t ∈ [a, b] with y (a) = α is called
well-posed if
I It has a unique solution y (t);
I There exist 0 > 0 and k > 0, such that ∀ ∈ (0, 0 ) and
function δ(t), which is continuous and satisfies |δ(t)| <  for
all t ∈ [a, b], the perturbed problem z 0 = f (t, z) + δ(t) with
initial value z(a) = α + δ0 (where |δ0 | ≤ ) satisfies

|z(t) − y (t)| < k, ∀t ∈ [a, b].

Remark
This theorem says that a small perturbation on defining function f
by δ(t) and initial value y (a) by δ0 will only cause small change to
original solution y (t).
Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 14
Some basics about IVP

Theorem
Let D = [a, b] × R. If f is continuous on D and Lipschitz with
respect to y , then the IVP is well-posed.

Remark
Again, a sufficient but not necessary condition for well-posedness
of IVP.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 15
 Euler’s method

Given an IVP y 0 = f (t, y ) for t ∈ [a, b] and y (a) = α, we want to

compute y (t) on mesh points {t0 , t1 , . . . , tN } on [a, b].
ER 5 Initial-Value Problems for Ordinary Differential Equations

To this end, we partition [a, b] into N equal segments: set

h =The
b−agraph
, of the
and function
define t i
highlighting
= a + ih y(ti ) is
for i shown
= 0, in Figure
1, . . . , 5.2.Here
N. One step
h in
is Euler’s
methodNappears in Figure 5.3, and a series of steps appears in Figure 5.4.
called the step size.
Figure 5.2
y
y(t N) ! y(b) y" ! f (t, y),
y(a) ! α
...

y(t 2)
y(t 1)
y(t 0) ! α

t0 ! a t1 t2 . . . tN ! b t

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 16
Euler’s method

From Taylor’s theorem, we have

0 1 00
y (ti+1 ) = y (ti ) + y (ti )(ti+1 − ti ) + y (ξi )(ti+1 − ti )2
2
for some ξi ∈ (ti , ti+1 ). Note that ti+1 − ti = h and
y 0 (ti ) = f (ti , y (ti )), we get

y (ti+1 ) ≈ y (ti ) + hf (t, y (ti ))

Denote wi = y (ti ) for all i = 0, 1, . . . , N, we get the Euler’s

method:
(
w0 = α
wi+1 = wi + hf (ti , wi ), i = 0, 1, . . . , N − 1

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 17
 y(t 1) y(t 1)
y(t 0) ! α y(t 0) ! α
Euler’s method
t0 ! a t1 t2 . t.0. !t aN !tb1 t 2t . . . tN ! b t

y Figure 5.3 y Figure 5.4 y Figure 5.4 y

y" ! f (t, y), y" ! f (t, y),
y" ! f (t, y), y" ! f (t, y), y(b) y(b)
y(a) !α y(a) ! α
y(a) ! α y(a) ! α wN wN

α)
Slope y"(a) ! f (a, Slope y"(a) ! f (a, α) w2 w2
w1 w1 w1 w1
α α α α
t0 ! a t1 t2 .t .0 .! taN !t 1b t 2 t. . . t N ! b t t0 ! a t1 t2 . .t .0 !t Na! bt 1 t2 t . . . tN ! b

ple 1 Euler’s method

Example 1 was usedmethod
Euler’s in the first
wasillustration
used in thewith = 0.5 to approximate
firsthillustration thetosolution
with h = 0.5 approximate the so
to the initial-value
toproblem
the initial-value problem

y′ = yYe,−Math
Numerical Analysis II – Xiaojing t 2 +&1,
Stat, ′0 ≤t≤
Georgia 2, y(0) =
2 University
y = y − t + 1,
State
0 ≤0.5.
t ≤ 2, y(0) = 0.5. 18
Euler’s method

Example
Use Euler’s method with h = 0.5 for IVP y 0 = y − t 2 + 1 for
t ∈ [0, 2] with initial value y (0) = 0.5.

Solution: We follow Euler’s method step-by-step:

t0 = 0 : w0 = y (0) = 0.5
t1 = 0.5 : w1 = w0 + hf (t0 , w0 ) = 0.5 + 0.5 × (0.5 − 02 + 1) = 1.25
t2 = 1.0 : w2 = w1 + hf (t1 , w1 ) = 1.25 + 0.5 × (1.25 − 0.52 + 1) = 2.25
t3 = 1.5 : w3 = w2 + hf (t2 , w2 ) = 2.25 + 0.5 × (2.25 − 12 + 1) = 3.375
t4 = 2.0 : w4 = w3 + hf (t3 , w3 ) = 3.375 + 0.5 × (3.375 − 1.52 + 1) = 4.4375

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 19
Error bound of Euler’s method

Theorem
Suppose f (t, y ) in an IVP is continuous on D = [a, b] × R and
Lipschitz with respect to y with constant L. If ∃M > 0 such that
|y 00 (t)| ≤ M (y (t) is the unique solution of the IVP), then for all
i = 0, 1, . . . , N there is
  hM  L(t −a) 
y (ti ) − wi  ≤ e i −1
2L

Remark
I Numerical error depends on h (also called O(h) error).
I Also depends on M, L of f .
I Error increases for larger ti .

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 20
Error bound of Euler’s method

Proof. Taking the difference of

1 00
y (ti+1 ) = y (ti ) + hf (ti , yi ) + y (ξi )(ti+1 − ti )2
2
wi+1 = wi + hf (ti , wi )

we get

Mh2
|y (ti+1 ) − wi+1 | ≤ |y (ti ) − wi | + h|f (ti , yi ) − f (ti , wi )| +
2
Mh2
≤ |y (ti ) − wi | + hL|yi − wi | +
2
Mh2
= (1 + hL)|yi − wi | +
2

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 21
Error bound of Euler’s method
Proof (cont).
Denote di = |y (ti ) − wi |, then we have

Mh2  hM  hM
di+1 ≤ (1 + hL)di + = (1 + hL) di + −
2 2L 2L
for all i = 0, 1, . . . , N − 1. So we obtain
hM  hM 
di+1 + ≤ (1 + hL) di +
2L 2L
 hM 
≤ (1 + hL)2 di−1 +
2L
≤ ···
 hM 
≤ (1 + hL)i+1 d0 +
2L
hM hM
and hence di ≤ (1 + hL)i · 2L − 2L (since d0 = 0).
Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 22
Error bound of Euler’s method

Proof (cont).
Note that 1 + x ≤ e x for all x > −1, and hence (1 + x)a ≤ e ax if
a > 0.
Based on this, we know (1 + hL)i ≤ e ihL = e L(ti −a) since
ih = ti − a. Therefore we get

L(ti −a) hM hM hM L(ti −a)

di ≤ e · − = (e − 1)
2L 2L 2L
This completes the proof.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 23
Error bound of Euler’s method

Example
Estimate the error of Euler’s method with h = 0.2 for IVP
y 0 = y − t 2 + 1 for t ∈ [0, 2] with initial value y (0) = 0.5.

∂f
Solution: We first note that ∂y = 1, so f is Lipschitz with respect
to y with constant L = 1. The IVP has solution
2 et 00 et e2
y (t) = (t − 1) − 2 so |y (t)| = | 2 − 2| ≤ 2 − 2 =: M. By
theorem above, the error of Euler’s method is
  hM  L(t −a)  0.2(0.5e 2 − 2)  
y (ti ) − wi  ≤ e i −1 = e ti − 1
2L 2

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 24
Error bound of Euler’s method

Example
Estimate the error of Euler’s method with h = 0.2 for IVP
y 0 = y − t 2 + 1 for t ∈ [0, 2] with initial value y (0) = 0.5.
Solution: (cont) 5.2 Euler’s Metho

Table 5.1 ti wi yi = y(ti ) |yi − wi |

0.0 0.5000000 0.5000000 0.0000000
0.2 0.8000000 0.8292986 0.0292986
0.4 1.1520000 1.2140877 0.0620877
0.6 1.5504000 1.6489406 0.0985406
0.8 1.9884800 2.1272295 0.1387495
1.0 2.4581760 2.6408591 0.1826831
1.2 2.9498112 3.1799415 0.2301303
1.4 3.4517734 3.7324000 0.2806266
1.6 3.9501281 4.2834838 0.3333557
1.8 4.4281538 4.8151763 0.3870225
2.0 4.8657845 5.3054720 0.4396874

Note
Numerical Analysis II – Xiaojing that&the
Ye, Math Stat,error
Georgiagrows slightly
State University as the value of t increases.
25 This contr
Round-off error of Euler’s method

Due to round-off errors in computer, we instead obtain

(
u0 = α + δ0
ui+1 = ui + hf (ti , ui ) + δi , i = 0, 1, . . . , N − 1

Suppose ∃δ > 0 such that |δi | ≤ δ for all i, then we can show
  1  hM δ  
y (ti ) − ui  ≤ + e L(ti −a) − 1 + δe L(ti −a) .
L 2 h
hM δ hM δ
Note that 2 +hqdoes not approach 0 as h → 0. 2 + h reaches
minimum at h = 2δM (often much smaller than what we choose in
practice).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 26
Higher-order Taylor’s method
Definition (Local truncation error)
We call the difference method
(
w0 = α
wi+1 = wi + hφ(ti , wi ), i = 0, 1, . . . , N − 1

to have local truncation error

yi+1 − (yi + hφ(ti , yi ))
τi+1 (h) =
h
where yi := y (ti ).

Example
Euler’s method has local truncation error
yi+1 − (yi + hf (ti , yi )) yi+1 − yi
τi+1 (h) = = − f (ti , yi )
h h
Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 27
Higher-order Taylor’s method

Note that Euler’s method has local 00

truncation error
τi+1 (h) = yi+1h−yi − f (ti , yi ) = hy 2(ξi ) for some ξi ∈ (ti , ti+1 ). If
|y 00 | ≤ M we know |τi+1 (h)| ≤ hM 2 = O(h).

Question: What if we use higher-order Taylor’s approximation?

h2 00
0 hn (n)
y (ti+1 ) = y (ti ) + hy (ti ) + y (ti ) + · · · + y (ti ) + R
2 n!
hn+1 (n+1) (ξ )
where R = (n+1)! y i for some ξi ∈ (ti , ti+1 ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 28
Higher-order Taylor’s method

First note that we can always write y (n) using f (t, y (t)):

y 0 (t) = f
y 00 (t) = f 0 = ∂t f + (∂y f )f
y 000 (t) = f 00 = ∂t2 f + (∂t ∂y f + (∂y2 f )f )f + ∂y f (∂t f + (∂y f )f )
···
y (n) (t) = f (n−1) = · · ·

albeit it’s quickly getting very complicated.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 29
Higher-order Taylor’s method
Now substitute them back to high-order Taylor’s approximation
(ignore residual R)

h2 00
0 hn (n)
y (ti+1 ) = y (ti ) + hy (ti ) + y (ti ) + · · · + y (ti )
2 n!
h2 0 hn (n−1)
= y (ti ) + hf + f + · · · + f
2 n!
We can get the n-th order Taylor’s method:
(
w0 = α
wi+1 = wi + hT (n) (ti , wi ), i = 0, 1, . . . , N − 1

where

(n) h 0 hn−1 (n−1)

T (ti , wi ) = f (ti , wi ) + f (ti , wi ) + · · · + f (ti , wi )
2 n!

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 30
Higher-order Taylor’s method

I Euler’s method is the first order Taylor’s method.

I High-order Taylor’s method is more accurate than Euler’s
method, but at much higher computational cost.
I Together with Hermite interpolating polynomials, it can be
used to interpolate values not on mesh points more accurately.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 31
Higher-order Taylor’s method

Theorem
If y (t) ∈ C n+1 [a, b], then the n-th order Taylor method has local
truncation error O(hn ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 32
Runge-Kutta (RK) method

Runge-Kutta (RK) method attains high-order local truncation error

without expensive evaluations of derivatives of f .

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 33
Runge-Kutta (RK) method

To derive RK method, first recall Taylor’s formula for two variables

(t, y ):
f (t, y ) = Pn (t, y ) + Rn (t, y )
∂ n f (t0 ,y0 )
where ∂tn−k ∂yk f = ∂t n−k ∂y k
and

Pn (t, y ) = f (t0 , y0 ) + (∂t f · (t − t0 ) + ∂y f · (y − y0 ))

1 2 
+ ∂t f · (t − t0 )2 + 2∂y ∂t f · (t − t0 )(y − y0 ) + ∂y2 f · (y − y0 )2
2
Xn  
1 n n−k k
+ ··· + ∂t ∂y f · (t − t0 )n−k (y − y0 )k
n! k
k=0
n+1 
X 
1 n + 1 n+1−k k
Rn (t, y ) = ∂t ∂y f (ξ, µ) · (t − t0 )n+1−k (y − y0 )k
(n + 1)! k
k=0

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 34
Runge-Kutta (RK) method

The second order Taylor’s method uses

(2) h 0 h
T (t, y ) = f (t, y ) + f (t, y ) = f (t, y ) + (∂t f + ∂y f · f )
2 2
to get O(h2 ) error. Suppose we use af (t + α, y + β) (with some

a, α, β to be determined) to reach the same order of error. To that

end, we first have
 
af (t + α, y + β) = a f + ∂t f · α + ∂y f · β + R

where R = 21 (∂t2 f (ξ, µ) · α2 + 2∂y ∂t f (ξ, µ) · αβ + ∂y2 f (ξ, µ) · β 2 ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 35
Runge-Kutta (RK) method

Suppose we try to match the terms of these two formulas (ignore

R):

(2) h hf
T (t, y ) = f + ∂t f + ∂y f
2 2
af (t + α, y + β) = af + aα∂t f + aβ∂y f

then we have
h h
a = 1, α= , β = f (t, y )
2 2

So instead of T (2) (t, y ), we use

h h  
af (t + α, y + β) = f t + , y + f (t, y )
2 2

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 36
Runge-Kutta (RK) method

Note that R we ignored is

  h 2  h 2  h 2 
1
R= ∂t2 f (ξ, µ) · + 2∂y ∂t f (ξ, µ) · f + ∂y2 f (ξ, µ) · f2
2 2 2 2

which means R = O(h2 ).

Also note that

h h 
R = T (2) (t, y ) − f t + , y + f (t, y ) = O(h2 )
2 2

and the error of T (2) (t, y ) is of O(h2 ), we know

 
f t + h2 , y + h2 f (t, y ) has error of O(h2 ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 37
Runge-Kutta (RK) method

This is the RK2 method (Midpoint method):



 w0 = α
 h h 

wi+1 = wi + h f ti + , wi + f (ti , wi ) , i = 0, 1, . . . , N − 1.
2 2

Remark
If we have (ti , wi ), we only need to evaluate f twice (i.e., compute
k1 = f (ti , wi ) and k2 = f (ti + h2 , wi + h2 k1 )) to get wi+1 .

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 38
Runge-Kutta (RK) method
We can also consider higher-order RK method by fitting

(3) h 0 h 00
T (t, y ) = f (t, y ) + f (t, y ) + f (t, y )
2 6
with af (t, y ) + bf (t + α, y + β) (has 4 parameters a, b, α, β).
hf 00
Unfortunately we can’t make match to the term of T (3) , which 6
h2
contains 6 f · (∂y f )2 , by this way. But it leaves us open choices if
we’re OK with O(h2 ) error: let a = b = 1, α = h, β = hf (t, y ),
then we get the modified Euler’s method:


 w0 = α
 h 
wi+1 = wi + f (ti , wi ) + f (ti+1 , wi + hf (ti , wi )) , i = 0, 1, . . . , N − 1.
2

Also need evaluation of f twice in each step.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 39
Runge-Kutta (RK) method

Example
Use Midpoint method (RK2) and Modified Euler’s method with
h = 0.2 to solve IVP y 0 = y − t 2 + 1 for t ∈ [0, 2] and y (0) = 0.5.

Solution: Apply the main steps in the two methods:


h h 
Midpoint : wi+1 =wi + h f ti + , wi + f (ti , wi )
2 2
h  
Modified Euler’s : wi+1 =wi + f (ti , wi ) + f (ti+1 , wi + hf (ti , wi ))
2

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 40
 Midpoint method: w2 = 1.22(0.828) − 0.0088(0.2)2 − 0.008(0.2) + 0.218
Runge-Kutta (RK) method
= 1.21136;
Modified Euler method: w2 = 1.22(0.826) − 0.0088(0.2)2 − 0.008(0.2) + 0.216
Example = 1.20692,
Use Midpoint method (RK2) and Modified Euler’s method with
Table 5.6 lists all the results
0 of the calculations.
2 For this problem, the Midpoint method
h =is0.2 to solve
superior IVP y Euler
to the Modified − t + 1 for t ∈ [0, 2] and y (0) = 0.5.
= ymethod.
Solution: (cont)
Table 5.6 Midpoint Modified Euler
ti y(ti ) Method Error Method Error
0.0 0.5000000 0.5000000 0 0.5000000 0
0.2 0.8292986 0.8280000 0.0012986 0.8260000 0.0032986
0.4 1.2140877 1.2113600 0.0027277 1.2069200 0.0071677
0.6 1.6489406 1.6446592 0.0042814 1.6372424 0.0116982
0.8 2.1272295 2.1212842 0.0059453 2.1102357 0.0169938
1.0 2.6408591 2.6331668 0.0076923 2.6176876 0.0231715
1.2 3.1799415 3.1704634 0.0094781 3.1495789 0.0303627
1.4 3.7324000 3.7211654 0.0112346 3.6936862 0.0387138
1.6 4.2834838 4.2706218 0.0128620 4.2350972 0.0483866
1.8 4.8151763 4.8009586 0.0142177 4.7556185 0.0595577
2.0 5.3054720 5.2903695 0.0151025 5.2330546 0.0724173

Midpoint (RK2) method is better than modified Euler’s method.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 41
Runge-Kutta (RK) method

We can also consider higher-order RK method by fitting

h h 2
T (3) (t, y ) = f (t, y ) + f 0 (t, y ) + f 00 (t, y )
2 6
with af (t, y ) + bf (t + α1 , y + δ1 (f (t + α2 , y + δ2 f (t, y )) ) (has 6
parameters a, b, α1 , α2 , δ1 , δ2 ) to reach O(h3 ) error.

For example, Heun’s choice is a = 14 , b = 43 , α1 = 2h

3 , α2 = h3 ,
δ1 = 2h
3 f , δ2 = h
3f .

Nevertheless, methods of order O(h3 ) are rarely used in practice.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 42
4-th Order Runge-Kutta (RK4) method

Most commonly used is the 4-th order Runge-Kutta method

(RK4): start with w0 = α, and iteratively do


 k1 = f (ti , wi )



 h h

 k2 = f (ti + , wi + k1 )

 2 2


h h
k3 = f (ti + , wi + k2 )

 2 2



 k4 = f (ti+1 , wi + hk3 )



 h

wi+1 = wi + (k1 + 2k2 + 2k3 + k4 )
6

Need to evaluate f for 4 times in each step. Reach error O(h4 ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 43
4-th Order Runge-Kutta (RK4) method
Example
Use RK4 (with h = 0.2) to solve IVP y 0 = y − t 2 + 1 for t ∈ [0, 2]
and y (0) = 0.5.
Solution: With h = 0.2, we have N = 10 and ti = 0.2i for
i = 0, 1, . . . , 10. First set w0 = 0.5, then the first iteration is

k1 = f (t0 , w0 ) = f (0, 0.5) = 0.5 − 02 + 1 = 1.5

h h
k2 = f (t0 + , w0 + k1 ) = f (0.1, 0.5 + 0.1 × 1.5) = 1.64
2 2
h h
k3 = f (t0 + , w0 + k2 ) = f (0.1, 0.5 + 0.1 × 1.64) = 1.654
2 2
k4 = f (t1 , w0 + hk3 ) = f (0.2, 0.5 + 0.2 × 1.654) = 1.7908
h
w1 = w0 + (k1 + 2k2 + 2k3 + k4 ) = 0.8292933
6
So w1 is our RK4 approximation of y (t1 ) = y (0.2).
Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 44
 k3 = 0.2f (0.1, 0.664) = 0.3308

4-th Order Runge-Kutta (RK4)

k4 = 0.2f method
(0.2, 0.8308) = 0.35816
1
Example w1 = 0.5 + (0.3 + 2(0.328) + 2(0.3308) + 0.35816) = 0.829293
6
Use RK4 (with h = 0.2) to solve IVP y 0 = y − t 2 + 1 for t ∈ [0, 2]
The remaining results and their errors are listed in Table 5.8.
and y (0) = 0.5.
Solution: (cont) Continue with i = 1, 2, · · · , 9:
Table 5.8 Runge-Kutta
Exact Order Four Error
ti yi = y(ti ) wi |yi − wi |
0.0 0.5000000 0.5000000 0
0.2 0.8292986 0.8292933 0.0000053
0.4 1.2140877 1.2140762 0.0000114
0.6 1.6489406 1.6489220 0.0000186
0.8 2.1272295 2.1272027 0.0000269
1.0 2.6408591 2.6408227 0.0000364
1.2 3.1799415 3.1798942 0.0000474
1.4 3.7324000 3.7323401 0.0000599
1.6 4.2834838 4.2834095 0.0000743
1.8 4.8151763 4.8150857 0.0000906
2.0 5.3054720 5.3053630 0.0001089
Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 45
High-order Runge-Kutta method

Can we use even higher-order method to improve accuracy?

#f eval 2 3 4 5≤n≤7 8≤n≤9 n ≥ 10
Best error O(h2 ) O(h3 ) O(h4 ) O(hn−1 ) O(hn−2 ) O(hn−3 )
So

RK4 is the sweet spot.

Remark
Note that RK4 requires 4 evaluations of f each step. So it would
make sense only if it’s accuracy with step size 4h is higher than
Midpoint with 2h or Euler’s with h!

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 46
 High-order Runge-Kutta method

Example
Use RK4 (with h = 0.1), Midpoint (with h = 0.05), and Euler’s
method (with h = 0.025) to solve IVP y 0 = y − t 2 + 1 for
t ∈ [0, 0.5] and y (0) = 0.5.
5.4 Runge-Kutta Methods
Solution:
Table 5.10 Modified Runge-Kutta
Euler Euler Order Four
ti Exact h = 0.025 h = 0.05 h = 0.1
0.0 0.5000000 0.5000000 0.5000000 0.5000000
0.1 0.6574145 0.6554982 0.6573085 0.6574144
0.2 0.8292986 0.8253385 0.8290778 0.8292983
0.3 1.0150706 1.0089334 1.0147254 1.0150701
0.4 1.2140877 1.2056345 1.2136079 1.2140869
0.5 1.4256394 1.4147264 1.4250141 1.4256384

RK4 is better with same computation cost!

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 47
Error control

Can we control the error of Runge-Kutta method by using variable

step sizes?

Let’s compare two difference methods with errors O(hn ) and

O(hn+1 ) (say, RK4 and RK5) for fixed step size h, which have
schemes below:

wi+1 = wi + hφ(ti , wi , h) O(hn )

w̃i+1 = w̃i + hφ̃(ti , w̃i , h) O(hn+1 )

Suppose wi ≈ w̃i ≈ y (ti ) =: yi . Then for any given  > 0, we want

to see how small h should be for the O(hn ) method so that its
error |τi+1 (h)| ≤ ?

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 48
Error control

We recall that the local truncation errors of these two methods are:
yi+1 − yi
τi+1 (h) = − φ(ti , yi , h) ≈ O(hn )
h
yi+1 − yi
τ̃i+1 (h) = − φ̃(ti , yi , h) ≈ O(hn+1 )
h
Given that wi ≈ w̃i ≈ yi and O(hn+1 )  O(hn ) for small h, we see

τi+1 (h) ≈ τi+1 (h) − τ̃i+1 (h) = φ̃(ti , yi , h) − φ(ti , yi , h)

w̃i+1 − w̃i wi+1 − wi
≈ φ̃(ti , w̃i , h) − φ(ti , wi , h) = −
h h
w̃i+1 − wi+1
≈ ≈ Khn
h
for some K > 0 independent of h, since τi+1 (h) ≈ O(hn ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 49
Error control

Suppose that we can scale h by q > 0, such that

n n n n |w̃i+1 − wi+1 |
|τi+1 (qh)| ≈ K (qh) = q Kh ≈ q ≤
h
So we need q to satisfy
 h 1/n
q≤
|w̃i+1 − wi+1 |

I q < 1: reject the initial h and recalculate using qh.

I q ≥ 1: accept computed value and use qh for next step.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 50
Runge-Kutta-Fehlberg method

The Runge-Kutta-Fehlberg (RKF) method uses specific

4th-order and 5th-order RK schemes, which share some computed
values and together only need 6 evaluation of f , to estimate
 h 1/4  h 1/4
q= = 0.84
2|w̃i+1 − wi+1 | |w̃i+1 − wi+1 |

This q is used to tune step size so that error is always bounded by

the prescribed .

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 51
Multistep method

Definition
Let m > 1 be an integer, then an m-step multistep method is
given by the form of

wi+1 = am−1 wi + am−2 wi−1 + · · · + a0 wi−m+1

 
+ h bm f (ti+1 , wi+1 ) + bm−1 f (ti , wi ) + · · · + b0 f (ti−m+1 , wi−m+1 )

for i = m − 1, m, . . . , N − 1.

Here a0 , . . . , am−1 , b0 , . . . , bm are constants. Also

w0 = α, w1 = α1 , . . . , wm−1 = αm−1 need to be given.
I bm = 0: Explicit m-step method.
I bm 6= 0: Implicit m-step method.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 52
Multistep method

Definition
The local truncation error of the m-step multistep method above
is defined by
yi+1 − (am−1 yi + · · · + a0 yi−m+1 )
τi+1 (h) =
 h 
− bm f (ti+1 , yi+1 ) + bm−1 f (ti , yi ) + · · · + b0 f (ti−m+1 , yi−m+1 )

where yi := y (ti ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 53
Adams-Bashforth Explicit method

Adams-Bashforth Two-Step Explicit method:



 w0 = α, w1 = α1 ,
h h i

wi+1 = wi + 3f (ti , wi ) − f (ti−1 , wi−1 )
2

for i = 1, . . . , N − 1.

The local truncation error is

5 000
τi+1 (h) = y (µi )h2
12
for some µi ∈ (ti−1 , ti+1 ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 54
Adams-Bashforth Explicit method: local truncation error

(k)
We denote yi := y (k) (ti ) for short. If wj = yj for j ≤ i, then

h2 00 h3 000
yi+1 = yi + + hyi0y + y (ξi ), ξi ∈ (ti , ti+1 )
2 i 6
h
wi+1 = yi + hyi0 + (yi0 − yi−1
0
),
2
0 0 00 h2 000
yi−1 = yi − hyi + y (ηi ), ηi ∈ (ti−1 , ti )
2

Plugging the equations above into the formula of local truncation error:

yi+1 − wi+1 1 1 000  5 000

000
τi+1 (h) = = y (ξi ) + y (ηi ) h2 = y (µi )h2
h 6 4 12

for some µi ∈ (ti−1 , ti+1 ), where in the last equality we used the intermediate value
1 y 000 (ξ )+ 1 y 000 (η )
i i
theorem and y ∈ C3 (so y 000 is continuous) to obtain y 000 (µi ) = 6
1+1
4
6 4
which is between y 000 (ξi ) and y 000 (ηi ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 55
Adams-Bashforth Explicit method

Adams-Bashforth Three-Step Explicit method:



 w0 = α, w1 = α1 , w2 = α2 ,
h h i

wi+1 = wi + 23f (ti , wi ) − 16f (ti−1 , wi−1 ) + 5f (ti−2 , wi−2 )
12

for i = 2, . . . , N − 1.

The local truncation error is

3 (4)
τi+1 (h) = y (µi )h3
8
for some µi ∈ (ti−2 , ti+1 ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 56
Adams-Bashforth Explicit method

Adams-Bashforth Four-Step Explicit method:



 w0 = α, w1 = α 1 , w2 = α 2 , w3 = α 3
 h h i
wi+1 = wi + 55f (ti , wi ) − 59f (ti−1 , wi−1 ) + 37f (ti−2 , wi−2 ) − 9f (ti−3 , wi−3 )
24

for i = 3, . . . , N − 1.

The local truncation error is

251 (5)
τi+1 (h) = y (µi )h4
720
for some µi ∈ (ti−3 , ti+1 ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 57
Adams-Bashforth Explicit method

Adams-Bashforth Five-Step Explicit method:



 w0 = α, w1 = α1 , w2 = α2 , w3 = α3 , w4 = α4



h
wi+1 = wi + [1901f (ti , wi ) − 2774f (ti−1 , wi−1 ) + 2616f (ti−2 , wi−2 )

 720


 − 1274f (ti−3 , wi−3 ) + 251f (ti−4 , wi−4 )]

for i = 4, . . . , N − 1.

The local truncation error is

95 (6)
τi+1 (h) = y (µi )h5
288
for some µi ∈ (ti−4 , ti+1 ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 58
Adams-Moulton Implicit method

Adams-Moulton Two-Step Implicit method:



 w0 = α, w1 = α1 ,
 h
wi+1 = wi + [5f (ti+1 , wi+1 ) + 8f (ti , wi ) − f (ti−1 , wi−1 )]
12

for i = 1, . . . , N − 1.

The local truncation error is

1 (4)
τi+1 (h) = − y (µi )h3
24
for some µi ∈ (ti−1 , ti+1 ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 59
Adams-Moulton Implicit method

Adams-Moulton Three-Step Implicit method:



 w0 = α, w1 = α 1 , w2 = α 2
 h
wi+1 = wi + [9f (ti+1 , wi+1 ) + 19f (ti , wi ) − 5f (ti−1 , wi−1 ) + f (ti−2 , wi−2 )]
24

for i = 2, . . . , N − 1.

The local truncation error is

19 (5)
τi+1 (h) = − y (µi )h4
720
for some µi ∈ (ti−2 , ti+1 ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 60
Adams-Moulton Implicit method

Adams-Moulton Four-Step Implicit method:



 w0 = α, w1 = α1 , w2 = α2 , w3 = α3



h
wi+1 = wi + [251f (ti+1 , wi+1 ) + 646f (ti , wi ) − 264f (ti−1 , wi−1 )

 720


 + 106f (ti−2 , wi−2 ) − 19f (ti−3 , wi−3 )]

for i = 3, . . . , N − 1.

The local truncation error is

3 (6)
τi+1 (h) = − y (µi )h5
160
for some µi ∈ (ti−3 , ti+1 ).

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 61
Steps to develop multistep methods

I Construct interpolating polynomial P(t) (e.g., Newton’s

backward difference method) using previously computed
(ti−m+1 , wi−m+1 ), . . . , (ti , wi ).
I Approximate y (ti+1 ) based on
Z ti+1 Z ti+1
y (ti+1 ) = y (ti ) + y 0 (t) dt = y (ti ) + f (t, y (t)) dt
ti ti
Z ti+1
≈ y (ti ) + f (t, P(t)) dt
ti

and construct difference method:

wi+1 = wi + hφ(ti , . . . , ti−m+1 , wi , . . . , wi−m+1 )

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 62
Explicit vs. Implicit

I Implicit methods are generally more accurate than the explicit

ones (e.g., Adams-Moulton three-step implicit method is even
more accurate than Adams-Bashforth four-step explicit
method).
I Implicit methods require solving for wi+1 from

h
wi+1 = ··· + f (ti+1 , wi+1 ) + · · ·
xxx
which can be difficult or even impossible.
I There could be multiple solutions of wi+1 when solving the
equation above in implicit methods.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 63
Predictor-Corrector method

Due to the aforementioned issues, implicit methods are often cast

in “predictor-corrector” form in practice.

In each step i:
I Prediction: Compute wi+1 using an explicit method φ to get
wi+1,p using

wi+1,p = wi + hφ(ti , wi , . . . , ti−m+1 , wi−m+1 )

I Correction: Substitute wi+1 by wi+1,p in the implicit method

φ̃ and compute wi+1 using

wi+1 = wi + hφ̃(ti+1 , wi+1,p , ti , wi , . . . , ti−m+1 , wi−m+1 )

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 64
Predictor-Corrector method
Example
Use the Adams-Bashforth 4-step explicit method and
Adams-Moulton 3-step implicit method to form the Adams
4th-order Predictor-Corrector method.
With initial value w0 = α, suppose we first generate w1 , w2 , w3
using RK4 method. Then for i = 3, 4, . . . , N − 1:
I Use Adams-Bashforth 4-step explicit method to get a
predictor wi+1,p :
h h i
wi+1,p = wi + 55f (ti , wi ) − 59f (ti−1 , wi−1 ) + 37f (ti−2 , wi−2 ) − 9f (ti−3 , wi−3 )
24

I Use Adams-Moulton 3-step implicit method to get a corrector

wi+1 :
h
wi+1 = wi + [9f (ti+1 , wi+1,p ) + 19f (ti , wi ) − 5f (ti−1 , wi−1 ) + f (ti−2 , wi−2 )]
24

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 65
 respectively,method
Predictor-Corrector compared to those of Runge-Kutta, which were accurate, re

|2.1272027 − 2.1272892| = 2.69 × 10− 5

and |2.6408227 − 2.6408
Example
Use AdamsThe remaining predictor-corrector
Predictor-Corrector Method withapproximations were generated
h = 0.2 to solve IVP using
y 0 = y − t 2are
+ shown
1 for tin∈Table
[0, 2]5.14.
and y (0) = 0.5.

Table 5.14 Error

ti yi = y(ti ) wi |yi − wi |
0.0 0.5000000 0.5000000 0
0.2 0.8292986 0.8292933 0.0000053
0.4 1.2140877 1.2140762 0.0000114
0.6 1.6489406 1.6489220 0.0000186
0.8 2.1272295 2.1272056 0.0000239
1.0 2.6408591 2.6408286 0.0000305
1.2 3.1799415 3.1799026 0.0000389
1.4 3.7324000 3.7323505 0.0000495
1.6 4.2834838 4.2834208 0.0000630
1.8 4.8151763 4.8150964 0.0000799
2.0 5.3054720 5.3053707 0.0001013

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 66
Other Predictor-Corrector method

We can also use Milne’s 3-step explicit method and Simpson’s

2-step implicit method below:
4h h i
wi+1,p = wi−3 + 2f (ti , wi ) − f (ti−1 , wi−1 ) + 2f (ti−2 , wi−2 )
3
h
wi+1 = wi−1 + [f (ti+1 , wi+1,p ) + 4f (ti , wi ) + f (ti−1 , wi−1 )]
3

This method is O(h4 ) and generally has better accuracy than

Adams PC method. However it is more likely to be vulnerable to
round-off error.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 67
Predictor-Corrector method

I PC methods have comparable accuracy as RK4, but often

require only 2 evaluations of f in each step.
I Need to store values of f for several previous steps.
I Sometimes are more restrictive on step size h, e.g., in the stiff
differential equation case later.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 68
Variable step-size multistep method

Now let’s take a closer look at the errors of the multistep methods.
Denote yi := y (ti ).

The Adams-Bashforth 4-step explicit method has error

251 (5)
τi+1 (h) = y (µi )h4
720
The Adams-Moulton 3-step implicit method has error
19 (5)
τ̃i+1 (h) = − y (µ̃i )h4
720
where µi ∈ (ti−3 , ti+1 ) and µ̃i ∈ (ti−2 , ti+1 ).

Question: Can we find a way to scale step size h so the error is

under control?

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 69
Variable step-size multistep method
Consider the their local truncation errors:
251 (5)
yi+1 − wi+1,p = y (µi )h5
720
19 (5)
yi+1 − wi+1 =− y (µ̃i )h5
720

Assume y (5) (µi ) ≈ y (5) (µ̃i ), we take their difference to get

1 3 (5)
wi+1 − wi+1,p = (19 + 251)y (µi )h ≈ y (µi )h5
(5) 5
720 8
So the error of Adams-Moulton (corrector step) is

|yi+1 − wi+1 | 19|wi+1 − wi+1,p |

τ̃i+1 (h) = ≈ = Kh4
h 270h
where K is independent of h since τ̃i+1 (h) = O(h4 ).
Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 70
Variable step-size multistep method

If we want to keep error under a prescribed , then we need to find

q > 0 such that with step size qh, there is

|y (ti + qh) − wi+1 | 19q 4 |wi+1 − wi+1,p |

τ̃i+1 (qh) = ≈ <
qh 270h
This implies that

270h 1/4  h 1/4
q< ≈2
19|wi+1 − wi+1,p | |wi+1 − wi+1,p |

To be conservative, we may replace 2 by 1.5 above.

In practice, we tune q (as less as possible) such that the estimated

error is between (/10, )

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 71
System of differential equations

The IVP for a system of ODE has form



 du1

 = f1 (t, u1 , u2 , . . . , um )

 dt



 du
 2 = f2 (t, u1 , u2 , . . . , um )
dt for a ≤ t ≤ b

 ..

 .





 dum
 = fm (t, u1 , u2 , . . . , um )
dt

with initial value u1 (a) = α1 , . . . , um (a) = αm .

Definition
A set of functions u1 (t), . . . , um (t) is a solution of the IVP above
if they satisfy both the system of ODEs and the initial values.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 72
 N subintervals with the mesh points

System of differential equations

t = a + jh, j for each j = 0, 1, . . . , N.

Use the notation wij , for each j = 0, 1, . . . , N and i = 1, 2, . . . , m, to denote an approx-

imation to ui (tj ). That is, wij approximates the ith solution ui (t) of (5.45) at the jth mesh
point tj . For the initial conditions, set (see Figure 5.6)

w1,0 = α1 , w2,0 = α2 , . . . , wm,0 = αm . (5.48)

In this case, we will solve for u1 (t), . . . , um (t) which are
Figure 5.6
interdependent according to the ODE system.
y y y

w11 w23 u2(t) wm3 um(a) ! αm

w12 w22 wm2 um(t)
w13
u1(t) w21 wm1
u1(a) ! α1
u2(a) ! α2
a ! t0 t1 t2 t3 t a ! t0 t1 t2 t3 t a ! t0 t1 t2 t3 t

Suppose that the values w1, j , w2, j , . . . , wm, j have been computed. We obtain w1, j+1 ,
w2, j+1 , . . . , wm, j+1 by first calculating

k1,i = hfi (tj , w1, j , w2, j , . . . , wm, j ), for each i = 1, 2, . . . , m; (5.49)

! "
h 1 1 1
k2,i = hfi tj + , w1, j + k1,1 , w2, j + k1,2 , . . . , wm, j + k1,m , (5.50)
2
Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University
2 2 2 73
System of differential equations

Definition
A function f is called Lipschitz with respect to u1 , . . . , um on
D := [a, b] × Rm if there exists L > 0 s.t.
m
X
|f (t, u1 , . . . , um ) − f (t, z1 , . . . , zm )| ≤ L |uj − zj |
j=1

for all (t, u1 , . . . , um ), (t, z1 , . . . , zm ) ∈ D.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 74
System of differential equations

Theorem
∂f
If f ∈ C 1 (D) and | ∂u j
| ≤ L for all j, then f is Lipschitz with
respect to u = (u1 , . . . , um ) on D.

Proof.
Note that D is convex. For any (t, u1 , . . . , um ), (t, z1 , . . . , zm ) ∈ D,
define

g (λ) = f (t, (1 − λ)u1 + λz1 , . . . , (1 − λ)um + λzm )

R1
for all λ ∈ [0, 1]. Then from |g (1) − g (0)| ≤ 0 |g 0 (λ)| dλ and the
definition of g , the conclusion follows.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 75
System of differential equations

Theorem
If f ∈ C 1 (D) and is Lipschitz with respect to u = (u1 , . . . , um ),
then the IVP with f as defining function has a unique solution.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 76
System of differential equations

Now let’s use vector notations below

a = (α1 , . . . , αm )
y = (y1 , . . . , ym )
w = (w1 , . . . , wm )
f(t, w) = (f1 (t, w), . . . , fm (t, w))

Then the IVP of ODE system can be written as

y0 = f(t, y), t ∈ [a, b]

with initial value y(a) = a. So the difference methods developed

above, such as RK4, still apply.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 77
System of differential equations

Example
Use RK4 (with h = 0.1) to solve IVP for ODE system
(
I10 (t) = f1 (t, I1 , I2 ) = −4I1 + 3I2 + 6
I20 (t) = f2 (t, I1 , I2 ) = −2.4I1 + 1.6I2 + 3.6

with initial value I1 (0) = I2 (0) = 0.

Solution: The exact solution is
(
I1 (t) = −3.375e −2t + 1.875e −0.4t + 1.5
I2 (t) = −2.25e −2t + 2.25e −0.4t

for all t ≥ 0.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 78
 As a consequence,
System of differential equations
1
I1 (0.1) ≈ w1,1 = w1,0 + (k1,1 + 2k2,1 + 2k3,1 + k4,1 )
6
Example
1
= 0
Use RK4 (with h = 0.1) to + + 2(0.534)
solve
(0.6 IVP for+ODE
2(0.54072)
system + 0.4800912) = 0.5382552
6
(
and I10 (t) = f1 (t, I1 , I2 ) = −4I1 + 3I2 + 6
0 1I ) = −2.4I + 1.6I + 3.6
I2 (0.1) I≈2 (t)
w2,1=
= fw
2 (t,
2,0
I
+1 , 2 1,2 + 2k2,2 +1 2k3,2 + k24,2 ) = 0.3196263.
(k
6
withThe remaining
initial value entries
I1 (0) in
= Table
I2 (0)5.19
= 0.are generated in a similar manner. !
Solution: (cont) The result by RK4 is
Table 5.19 tj w1,j w2,j |I1 (tj ) − w1,j | |I2 (tj ) − w2,j |
0.0 0 0 0 0
0.1 0.5382550 0.3196263 0.8285 × 10− 5
0.5803 × 10− 5

0.2 0.9684983 0.5687817 0.1514 × 10− 4

0.9596 × 10− 5

0.3 1.310717 0.7607328 0.1907 × 10− 4

0.1216 × 10− 4

0.4 1.581263 0.9063208 0.2098 × 10− 4

0.1311 × 10− 4

0.5 1.793505 1.014402 0.2193 × 10− 4

0.1240 × 10− 4

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 79
High-order ordinary differential equations

A general IVP for mth-order ODE is

y (m) = f (t, y , y 0 , . . . , y (m−1) ), t ∈ [a, b]

with initial value y (a) = α1 , y 0 (a) = α2 , . . . , y (m−1) (a) = αm .

Definition
A function y (t) is a solution of IVP for the mth-order ODE
above if y (t) satisfies the differential equation for t ∈ [a, b] and all
initial value conditions at t = a.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 80
High-order ordinary differential equations

We can define a set of functions u1 , . . . , um s.t.

u1 (t) = y (t), u2 (t) = y 0 (t), ..., um (t) = y (m−1) (t)

Then we can convert the mth-order ODE to a system of first-order

ODEs (total of m coupled ODEs):


 u10 = u2



 u20 = u3
.. for a ≤ t ≤ b

 .



u 0 = f (t, u , u , . . . , u )
m 1 2 m

with initial values u1 (a) = α1 , . . . , um (a) = αm .

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 81
High-order ordinary differential equations

Example
Use RK4 (with h = 0.1) to solve IVP for ODE system

y 00 − 2y 0 + 2y = e 2t sin t, t ∈ [0, 1]

with initial value y (0) = −0.4, y 0 (0) = −0.6.

Solution: The exact solution is
y (t) = u1 (t) = 0.2e 2t (sin t − 2 cos t). Also u2 (t) = y 0 (t) = u10 (t)
but we don’t need it.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 82
High-order ordinary differential equations

Example
Use RK4 (with h = 0.1) to solve IVP for ODE system

y 00 − 2y 0 + 2y = e 2t sin t, t ∈ [0, 1]

with initial value y (0) = −0.4, y 0 (0) = −0.6.

336 CHAPTER 5 Initial-Value Problems for Ordinary Differential Equations
Solution: (cont) The result by RK4 is
Table 5.20
tj y(tj ) = u1 (tj ) w1,j y′ (tj ) = u2 (tj ) w2,j |y(tj ) − w1,j | |y′ (tj ) − w2,j |
0.0 − 0.40000000 − 0.40000000 − 0.6000000 − 0.60000000 0 0
0.1 − 0.46173297 − 0.46173334 − 0.6316304 − 0.63163124 3.7 × 10− 7 7.75 × 10− 7

0.2 − 0.52555905 − 0.52555988 − 0.6401478 − 0.64014895 8.3 × 10− 7 1.01 × 10− 6

0.3 − 0.58860005 − 0.58860144 − 0.6136630 − 0.61366381 1.39 × 10− 6 8.34 × 10− 7

0.4 − 0.64661028 − 0.64661231 − 0.5365821 − 0.53658203 2.03 × 10− 6 1.79 × 10− 7

0.5 − 0.69356395 − 0.69356666 − 0.3887395 − 0.38873810 2.71 × 10− 6 5.96 × 10− 7

0.6 − 0.72114849 − 0.72115190 − 0.1443834 − 0.14438087 3.41 × 10− 6 7.75 × 10− 7

0.7 − 0.71814890 − 0.71815295 0.2289917 0.22899702 4.05 × 10− 6 2.03 × 10− 6

0.8 − 0.66970677 − 0.66971133 0.7719815 0.77199180 4.56 × 10− 6 5.30 × 10− 6

0.9 − 0.55643814 − 0.55644290 1.534764 1.5347815 4.76 × 10− 6 9.54 × 10− 6

1.0 − 0.35339436 − 0.35339886 2.578741 2.5787663 4.50 × 10− 6 1.34 × 10− 5

In Maple the nth derivative y(n) (t) We can also use dsolve from Maple on higher-order equations. To define the differential
is specified by (D@@n)(y)(t). equation in Example 1, use
Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 83
A brief summary

The difference methods we developed above, e.g., Euler’s,

midpoints, RK4, multistep explicit/implicit, predictor-corrector
methods, are
I based on step-by-step derivation and easy to understand;
I widely used in many practical problems;
I fundamental to more advanced and complex techniques.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 84
Stability of difference methods

Definition (Consistency)
A difference method is called consistent if
 
lim max τi (h) = 0
h→0 1≤i≤N

where τi (h) is the local truncation error of the method.

Remark
Since local truncation error τi (h) is defined assuming previous
wi = yi , it does not take error accumulation into account. So the
consistency definition above only considers how good φ(t, wi , h) in
the difference method is.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 85
Stability of difference methods

For any step size h > 0, the difference method

wi+1 = wi + hφ(ti , wi , h) can generate a sequence of wi which
depend on h. We call them {wi (h)}i . Note that wi gradually
accumulate errors as i = 1, 2, . . . , N.
Definition (Convergent)
A difference method is called convergent if
 
lim max |yi − wi (h)| = 0
h→0 1≤i≤N

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 86
Stability of difference methods

Example
Show that Euler’s method is convergent.
Solution: We have showed before that for fixed h > 0 there is
  hM  L(t −a)  hM  
y (ti ) − wi  ≤ e i −1 ≤ e L(b−a) − 1
2L 2L
for all i = 0, . . . , N. Therefore we have
  hM  L(b−a) 
max y (ti ) − wi  ≤ e −1 →0
1≤i≤N 2L
 
as h → 0. Therefore limh→0 (max1≤i≤N y (ti ) − wi ) = 0.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 87
Stability of difference method

Definition
A numerical method is called stable if its results depend on the
initial data continuously.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 88
Stability of difference methods

Theorem
For a given IVP y 0 = f (t, y ), t ∈ [a, b] with y (a) = α, consider a
difference method wi+1 = wi + hφ(ti , wi , h) with w0 = α. If there
exists h0 > 0 such that φ is continuous on [a, b] × R × [0, h0 ], and
φ is L-Lipschitz with respect to w , then
I The difference method is stable.
I The difference method is convergent if and only if it is
consistent (i.e., φ(t, y , 0) = f (t, y )).
I If there exists bound τ (h) such that |τi (h)| ≤ τ (h) for all
i = 1, . . . , N, then |y (ti ) − wi | ≤ τ (h)e L(ti −a) /L.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 89
Stability of difference methods

Proof.
Let h be fixed, then wi (α) generated by the difference method are
functions of α. For any two values α, α̂, there is
|wi+1 (α) − wi+1 (α̂)| = |(wi (α) − hφ(ti , wi (α), h)) − (wi (α̂) − hφ(ti , wi (α̂), h))|
≤ |wi (α) − wi (α̂)| + h|φ(ti , wi (α), h) − φ(ti , wi (α̂), h)|
≤ |wi (α) − wi (α̂)| + hL|wi (α) − wi (α̂)|
= (1 + hL)|wi (α) − wi (α̂)|
≤ ···
≤ (1 + hL)i+1 |w0 (α) − w0 (α̂)|
= (1 + hL)i+1 |α − α̂|
≤ (1 + hL)N |α − α̂|

Therefore wi (α) is Lipschitz with respect to α (with constant at

most (1 + hL)N ), and hence is continuous with respect to α. We
omit the proofs for the other two assertions here.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 90
Stability of difference method

Example
Use the result of Theorem above to show that the Modified Euler’s
method is stable.
Solution: Recall the Modified Euler’s method is given by
h 
wi+1 = wi + f (ti , wi ) + f (ti+1 , wi + hf (ti , wi ))
2
So we have φ(t, w , h) = 12 (f (t, w ) + f (t + h, w + hf (t, w ))).
Now we want to show φ is continuous in (t, w , h), and Lipschitz
with respect to w .

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 91
Stability of difference method

Solution: (cont) It is obvious that φ is continuous in (t, w , h)

since f (t, w ) is continuous. Fix t and h. For any w , w̄ ∈ R, there is
1
|φ(t, w , h) − φ(t, w̄ , h)| ≤ |f (t, w ) − f (t, w̄ )|
2
1
+ |f (t + h, w + hf (t, w )) − f (t + h, w̄ + hf (t, w̄ ))|
2
L L
≤ |w − w̄ | + |(w + hf (t, w )) − (w̄ + hf (t, w̄ ))|
2 2
Lh
≤ L|w − w̄ | + |f (t, w ) − f (t, w̄ )|
2
L2 h
≤ L|w − w̄ | + |w − w̄ |
2
L2 h
= (L + )|w − w̄ |
2

So φ is Lipschitz with respect to w . By first part of Theorem

above, the Modified Euler’s method is stable.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 92
Stability of multistep difference method

Definition
Suppose a multistep difference method given by

wi+1 = am−1 wi + am−2 wi−1 + · · · + a0 wi−m+1 + hF (ti , h, wi+1 , . . . , wi−m+1 )

Then we call the following the characteristic polynomial of the

method:
λm − (am−1 λm−1 + · · · + a1 λ + a0 )

Definition
A difference method is said to satisfy the root condition if all the
m roots λ1 , . . . , λm of its characteristic polynomial have
magnitudes ≤ 1, and all of those which have magnitude =1 are
single roots.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 93
Stability of multistep difference method

Definition
I A difference method that satisfies root condition is called
strongly stable if the only root with magnitude 1 is λ = 1.
I A difference method that satisfies root condition is called
weakly stable if there are multiple roots with magnitude 1.
I A difference method that does not satisfy root condition is
called unstable.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 94
Stability of multistep difference method

Theorem
I A difference method is stable if and only if it satisfies the root
condition.
I If a difference method is consistent, then it is stable if and
only if it is convergent.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 95
Stability of multistep difference method

Example
Show that the Adams-Bashforth 4-step explicit method is strongly
stable.
Solution: Recall that the method is given by
h h i
wi+1 = wi + 55f (ti , wi ) − 59f (ti−1 , wi−1 ) + 37f (ti−2 , wi−2 ) − 9f (ti−3 , wi−3 )
24

So the characteristic polynomial is simply λ4 − λ3 = λ3 (λ − 1),

which only has one root λ = 1 with magnitude 1. So the method
is strongly stable.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 96
Stability of multistep difference method

Example
Show that the Milne’s 4-step explicit method is weakly stable but
not strongly stable.
Solution: Recall that the method is given by
4h h i
wi+1 = wi−3 + 2f (ti , wi ) − f (ti−1 , wi−1 ) + 2f (ti−2 , wi−2 )
3

So the characteristic polynomial is simply λ4 − 1, which have

roots λ = ±1, ±i. So the method is weakly stable but not strongly
stable.
Remark
This is the reason we chose Adams-Bashforth-Moulton PC rather
than Milne-Simpsons PC since the former is strongly stable and
likely to be more robust.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 97
Stiff differential equations

Stiff differential equations have e −ct terms (c > 0 large) in their

solutions. These terms → 0 quickly, but their derivatives (of form
c n e −ct ) do not, especially at small t.

Recall that difference methods have errors proportional to the

derivatives, and hence they may be inaccurate for stiff ODEs.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 98
Stiff differential equations

Example
Use RK4 to solve the IVP for a system of two ODEs:


 0 1
u1 = 9u1 + 24u2 + 5 cos t − sin t
3

 0 1
u2 = −24u1 − 51u2 − 9 cos t + sin t
3

with initial values u1 (0) = 4/3 and u2 (0) = 2/3.

Solution: The exact solution is


 −3t −39t 1
u1 (t) = 2e −e + cos t
3

 −3t −39t 1
u2 (t) = −e + 2e − cos t
3
for all t ≥ 0.
Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 99
 1 1
u1 (t) = 2e− 3t
− e− 39t
+ cos t, u2 (t) = − e− 3t
+ 2e− 39t
− cos t.
3 3
Stiff differential equations
The transient term e− 39t in the solution causes this system to be stiff. Applying Algorithm
5.7, the Runge-Kutta Fourth-Order Method for Systems, gives results listed in Table 5.22.
When h = 0.05, stability results and the approximations are accurate. Increasing the step
size to h = 0.1, however, leads to the disastrous results shown in the table. !
Solution: (cont) When we apply RK4 to this stiff ODE, we obtain
Table 5.22 w1 (t) w1 (t) w2 (t) w2 (t)
t u1 (t) h = 0.05 h = 0.1 u2 (t) h = 0.05 h = 0.1
0.1 1.793061 1.712219 − 2.645169 − 1.032001 − 0.8703152 7.844527
0.2 1.423901 1.414070 − 18.45158 − 0.8746809 − 0.8550148 38.87631
0.3 1.131575 1.130523 − 87.47221 − 0.7249984 − 0.7228910 176.4828
0.4 0.9094086 0.9092763 − 934.0722 − 0.6082141 − 0.6079475 789.3540
0.5 0.7387877 9.7387506 − 1760.016 − 0.5156575 − 0.5155810 3520.00
0.6 0.6057094 0.6056833 − 7848.550 − 0.4404108 − 0.4403558 15697.84
0.7 0.4998603 0.4998361 − 34989.63 − 0.3774038 − 0.3773540 69979.87
0.8 0.4136714 0.4136490 − 155979.4 − 0.3229535 − 0.3229078 311959.5
0.9 0.3416143 0.3415939 − 695332.0 − 0.2744088 − 0.2743673 1390664.
1.0 0.2796748 0.2796568 − 3099671. − 0.2298877 − 0.2298511 6199352.

which can blow up for larger step size h.

Although stiffness is usually associated with systems of differential equations, the
approximation characteristics of a particular numerical method applied to a stiff system can
be predicted by examining the error produced when the method is applied to a simple test
equation,

y′ = λy, y(0) = α, where λ < 0. (5.64)

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 100
Stiff differential equations

Now let’s use a simple example to see why this happens: consider
an IVP y 0 = λy , t ≥ 0, and y (0) = α. Here λ < 0. We know the
problem has solution y (t) = αe λt .

Suppose we apply Euler’s method, which is

wi+1 = wi + hf (ti , wi ) = wi + hλwi = (1 + λh)wi

= · · · = (1 + λh)i+1 w0 = (1 + λh)i+1 α

Therefore we simply have wi = (1 + λh)i α. So the error is

|y (ti ) − wi | = |αe λih − (1 + λh)i α| = |e λih − (1 + λh)i ||α|

In order for the error not to blow up, we need at least |1 + λh| < 1,
2
which yields h < |λ| . So h needs to be sufficiently small for large λ.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 101
Stiff differential equations

Similar issue occurs for other one-step methods, which for this IVP
can be written as wi+1 = Q(λh)wi = · · · = (Q(λh))i+1 α. For the
solution not to blow up, we need |Q(λh)| < 1.

For example, in nth-order Taylor’s method, we need

 λ2 h 2 λn hn 

|Q(λh)| = 1 + λh + + ··· + <1
2 n!
which requires h to be very small.

The same issue occurs for multistep methods too.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 102
Stiff differential equations

A remedy of stiff ODE is using implicit method, e.g., the implicit

Trapezoid method:
h
wi+1 = wi + (f (ti+1 , wi+1 ) + f (ti , wi ))
2
In each step, we need to solve for wi+1 from the equation above.
Namely, we need to solve for the root of F (w ):

h
:
F (w ) = w − wi − (f (ti+1 , w ) + f (ti , wi )) = 0
2
We can use fixed point iteration or Newton’s method to solve
F (x) = 0.

Numerical Analysis II – Xiaojing Ye, Math & Stat, Georgia State University 103