Advanced Computational Methods

Lecture 1
Review of Numerical Methods

Advanced Computational Methods, AKU

Lecture 1
Solutions of Nonlinear Equations and System of Linear Equations

February 26, 2024

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
1 / 153
 Outlines
1 Basic Numerical methods:

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
2 / 153
 Outlines
1 Basic Numerical methods:
• Numerical Solution of Nonlinear Equations

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
2 / 153
 Outlines
1 Basic Numerical methods:
• Numerical Solution of Nonlinear Equations
• Numerical Solution of Linear System of Equations

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
2 / 153
 Outlines
1 Basic Numerical methods:
• Numerical Solution of Nonlinear Equations
• Numerical Solution of Linear System of Equations
2 Numerical Solution for Ordinary Differential Equations

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
2 / 153
 Outlines
1 Basic Numerical methods:
• Numerical Solution of Nonlinear Equations
• Numerical Solution of Linear System of Equations
2 Numerical Solution for Ordinary Differential Equations
3 Numerical Solution for Partial Differential Equations

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
2 / 153
 Outlines
1 Basic Numerical methods:
• Numerical Solution of Nonlinear Equations
• Numerical Solution of Linear System of Equations
2 Numerical Solution for Ordinary Differential Equations
3 Numerical Solution for Partial Differential Equations
4 Finite difference method:

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
2 / 153
 Outlines
1 Basic Numerical methods:
• Numerical Solution of Nonlinear Equations
• Numerical Solution of Linear System of Equations
2 Numerical Solution for Ordinary Differential Equations
3 Numerical Solution for Partial Differential Equations
4 Finite difference method:
• Solving linear and transient conduction problems

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
2 / 153
 Outlines
1 Basic Numerical methods:
• Numerical Solution of Nonlinear Equations
• Numerical Solution of Linear System of Equations
2 Numerical Solution for Ordinary Differential Equations
3 Numerical Solution for Partial Differential Equations
4 Finite difference method:
• Solving linear and transient conduction problems
5 Finite element and Finite volume methods:

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
2 / 153
 Outlines
1 Basic Numerical methods:
• Numerical Solution of Nonlinear Equations
• Numerical Solution of Linear System of Equations
2 Numerical Solution for Ordinary Differential Equations
3 Numerical Solution for Partial Differential Equations
4 Finite difference method:
• Solving linear and transient conduction problems
5 Finite element and Finite volume methods:
• Solution of diffusion and convective form of partial differential
equations

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
2 / 153
 Outlines
1 Basic Numerical methods:
• Numerical Solution of Nonlinear Equations
• Numerical Solution of Linear System of Equations
2 Numerical Solution for Ordinary Differential Equations
3 Numerical Solution for Partial Differential Equations
4 Finite difference method:
• Solving linear and transient conduction problems
5 Finite element and Finite volume methods:
• Solution of diffusion and convective form of partial differential
equations
6 Pressure-velocity coupling algorithms: linear and non-linear
programing; simulation and optimization techniques

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
2 / 153
 Outlines
1 Basic Numerical methods:
• Numerical Solution of Nonlinear Equations
• Numerical Solution of Linear System of Equations
2 Numerical Solution for Ordinary Differential Equations
3 Numerical Solution for Partial Differential Equations
4 Finite difference method:
• Solving linear and transient conduction problems
5 Finite element and Finite volume methods:
• Solution of diffusion and convective form of partial differential
equations
6 Pressure-velocity coupling algorithms: linear and non-linear
programing; simulation and optimization techniques
7 Applications of numerical and programming software in solving
fluid flow, heat transfer and mass transfer problems in chemical,
environmental and other related engineering disciplines
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
2 / 153
 Introduction to Numerical Methods

• Numerical methods are techniques used to approximate

solutions to mathematical problems that are otherwise difficult or
impossible to solve analytically.
• They are commonly used in various fields such as engineering,
computer science, physics, and economics.

Types of Numerical Methods

1 Root Finding Methods
2 Interpolation Methods
3 Numerical Integration
4 Differential Equations

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
3 / 153
 Errors in Numerical Methods

• Errors can arise from various sources such as round-off errors,

truncation errors, and algorithmic errors.
• It is important to understand and minimize errors in numerical
computations to obtain accurate results.

Convergence and Stability

• Convergence refers to the property of a numerical method to
approach the exact solution as the number of iterations
increases.
• Stability refers to the property of a numerical method to produce
consistent results in the presence of small perturbations.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
4 / 153
 Applications of Numerical Methods
• Engineering: Solving complex engineering problems in structural
analysis, fluid dynamics, and electromagnetics.
• Finance: Financial modeling, risk assessment, option pricing, and
portfolio optimization.
• Computer Science: Algorithms for optimization, machine
learning, data analysis, and computer graphics.
• Physics: Simulations of physical systems, quantum mechanics,
and astrophysics.
• Geology: Modeling geological processes such as seismic wave
propagation, groundwater flow, and landform evolution.
• Weather Forecasting: Numerical weather prediction models for
forecasting weather patterns and climate change.
• Image and Signal Processing: Image reconstruction, denoising,
compression, and feature extraction in medical imaging, remote
sensing, and computer vision applications.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
5 / 153
 Solving Nonlinear Equations Using Numerical
Methods

• Nonlinear equations are equations in which the variables appear

in nonlinear terms.
• Examples: f (x) = x 2 − 3x + 2 = 0, g(x) = sin(x) − x = 0
• Nonlinear equations do not have simple algebraic solutions and
require numerical methods for finding roots.
• Common numerical methods for solving nonlinear equations
include:
• Bisection method
• Newton’s method
• Secant method
• Fixed-point iteration
• Steffensen’s method
• It is important to choose an appropriate method based on the
characteristics of the equation and desired precision.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
6 / 153
 Bisection Method
• The bisection method is a root-finding algorithm that repeatedly
bisects an interval and then selects a subinterval in which a root
must lie for further processing.
• It is a simple and robust method for finding roots of nonlinear
equations.
• The bisection method is based on the intermediate value
theorem.
• It requires an initial interval [a, b ] such that f (a) · f (b ) < 0, which
guarantees the existence of a root in that interval.

Algorithm
a+b
1 Start with the interval [a, b ] and compute the midpoint c = 2 .
2 Evaluate f (c) and determine the new interval [a, c] or [c, b ] based
on the sign of f (a) · f (c).
3 Repeat until the desired accuracy is achieved.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
7 / 153
 Example
Use the Bisection method to find a root of the equation
x 3 − 4x − 8.95 = 0 accurate to three decimal places.

Solution
Here, f (x) = x 3 − 4x − 8.95 = 0

f (2) = 23 − 4(2) − 8.95 = −8.95 < 0

f (3) = 33 − 4(3) − 8.95 = 6.05 > 0

Hence, a root lies between 2 and 3.
f ′ (x) = 3x 2 − 4 > 0 for x in the interval (2, 3). Hence, we have a = 2 and
b = 3. The results of the algorithm for Bisection method are shown in
Table 1.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
8 / 153
 n a b x0 b − x0 f (x0 )
1 2 3 2.5 0.5 -3.25
2 2.5 3 2.75 0.25 0.846
3 2.75 2.75 2.625 0.125 -1.362
4 2.75 2.625 2.6875 -0.0625 -0.289
5 2.6875 2.6875 2.718 -0.0312 0.2709
6 2.71875 2.7187 2.703 -0.015 -0.011
7 2.71875 2.703 2.7109 -0.007 0.129
8 2.7109 2.7109 2.7148 -0.0039 0.20005
Table 1: Bisection method results x 3 − 4x − 8.95 = 0

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
9 / 153
 Example
Consider a mining company trying to estimate the depth of a mineral
deposit within a mine. The company has drilled test boreholes at two
locations, A and B, and has collected data on the mineral content at
those locations.
Let f (x) be the function that represents the mineral content as a
function of depth at location A and B. The company wants to find the
depth at which the mineral content is equal at both locations.
• Initial depth interval: [a, b ] = [0, 100] meters
• f (a) = mineral content at location A at depth a
• f (b ) = mineral content at location B at depth b
• The company uses the bisection method to iteratively narrow
down the interval and estimate the depth where the mineral
content is equal at both locations.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
10 / 153
 Example
The floating ball has a specific gravity of 0.6 and has a radius of
5.5 cm. We are asked to find the depth to which the ball is submerged
when floating in water.
The equation that gives the depth x to which the ball is submerged
under water is given by

x 3 − 0.165x 2 + 3.993 × 10−4 = 0

Use the bisection method of finding roots of equations to find the

depth x to which the ball is submerged under water. Conduct three
iterations to estimate the root of the above equation. Find the
absolute relative approximate error at the end of each iteration, and
the number of significant digits at least correct at the end of each
iteration.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
11 / 153
 Newton’s Method

• Newton’s method is an iterative numerical technique used to find

the roots of a nonlinear equation.
• It is based on linear approximation of the function at each
iteration.
• The formula for Newton’s method is given by:

f (xn )
xn+1 = xn − (1)
f ′ (xn )

where xn is the current estimate and f ′ (xn ) is the derivative of the

function.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
12 / 153
 Steps for Newton’s Method

1 Choose an initial guess x0 .

2 Calculate f (x0 ) and f ′ (x0 ).
3 Update the estimate using the formula:

f (xn )
xn+1 = xn − (2)
f ′ (xn )

4 Repeat steps 2 and 3 until convergence criteria are met.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
13 / 153
 Example

Let’s solve the equation f (x) = x 2 − 4 using Newton’s method.

• Initial guess: x0 = 2
• Derivative: f ′ (x) = 2x

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
14 / 153
 Iteration

• Iteration 1:
• x1 = 2 − 24 = 1.5
• Iteration 2:
• x2 = 1.5 − −0.25
3 = 1.41
• Continue iterating until convergence is achieved.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
15 / 153
 Introduction

When faced with a system of linear equations of the form:

a11 x1 + a12 x2 + · · · + a1n xn = b1

a21 x1 + a22 x2 + · · · + a2n xn = b2
..
.
an1 x1 + an2 x2 + · · · + ann xn = bn

where aij are coefficients, xi are unknowns, and bi are constants, we

can use numerical methods to find solutions.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
16 / 153
 Direct Methods

Direct methods involve manipulating the system of equations to solve

for the unknowns. Some examples of direct methods include:
• Gaussian Elimination
• LU Decomposition
• Cholesky Decomposition

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
17 / 153
 Iterative Methods

Iterative methods involve finding approximate solutions to the system

of equations. Some examples of iterative methods include:
• Jacobi Method
• Gauss-Seidel Method
• Successive Over-Relaxation (SOR) Method

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
18 / 153
 Jacobi and Gauss-Seidel Methods

Jacobi Method
- Decomposes matrix A into D , L , and U .
- Iteration formula updates each component of x based on previous
values.
- New estimates for unknown variables are calculated using previous
estimates.

Gauss-Seidel Method
- Decomposes matrix A into L , D , and U .
- Updates each component of x in a sequential manner.
- Uses most recent component values for calculating subsequent
values.
Both methods converge to the exact solution under certain
conditions, such as diagonal dominance of A or positive definiteness.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
19 / 153
 In order to apply these iterative techniques, the equations must
satisfy diagonal dominance criteria.
Consider the systems of algebraic equations

a11 x1 + a12 x2 + a13 x3 + · · · + a1n xn = b1

a21 x1 + a22 x2 + a23 x3 + · · · + a2n xn = b2
..
.
an1 x1 + an2 x2 + an3 x3 + · · · + ann xn = bn

Dividing each equation by the leading diagonal term, we get

n
b1 ¼ a1j xj
x1 = −
a11 a11
j =2,j ,1
n
b2 ¼ a2j xj
x2 = −
a22 a22
j =1,j ,2
..
.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of n Equations
of Numerical
Nonlinear
¼ Methods and System of Linear Equations
20 / 153
 Cont..
 (k −1) 
 x 
 1 
 (k −1) 
x
Let x (k −1) =  2 .  be the (k − 1)th approximation of the given

 

 .
.


 (k −1) 
xn
system of equation. Then the k th approximation is
n (k −1)
b1 ¼ a1j xj
x1k = −
a11 a11
j =2,j ,1
n (k −1)
b2 ¼ a2j xj
x2k = −
a22 a22
j =1,j ,2
..
.
n (k −1)
bi ¼ aij xj
xik = −
a
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics
ii of of Numerical aii and System of Linear Equations
Nonlinear Equations
Methods 21 / 153
 The process is continued till convergence is secured. That is, if the
error
| x k − x (k −1) |< ê
• In the absence of any better estimates, the initial approximations
(0) (0) (0)
are taken as : x1 = 0, x1 = 0, · · · , xn = 0
• If some of the diagonal elements are zeros, or the system is not
diagonally dominant, we may have to rearrange the equations.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
22 / 153
 Example
Solve the system of equations using the Jacobi method correct to two
decimalplace.
5x1 + x2 + 2x3 = 10
−3x1 + 9x2 + 4x3 = −14 .
x1 + 2x2 + 7x3 = −33.
 
 0 
Taking the initial approximation x (0) =  0 
 
0
 

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
23 / 153
 Solution
We have  
 5 1 2 
 
A =  −3 9 4 
1 2 7
 

|a11 | = 5 > |a12 | + |a13 | =1 + 2 = 3

|a22 | = 9 > |a21 | + |a23 | =3 + 4 = 7
|a33 | = 7 > |a31 | + |a32 | =1 + 2 = 3
Thus, the system is diagonally dominant. Rearranging the equations
we get
1 (k −1)
 
(k ) (k −1)
x1 =2 − x2 + 2x3
5
14 1
 
(k ) (k −1) (k −1)
x2 = − − −x1 + 4x3
9 9
33 1 (k −1)
 
(k ) (k −1)
x3 = − − x1 + 2x2
7 7
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
24 / 153
 First approximation: Substituting the initial guesses of the unknowns
in the above governing equation, we get the first approximations as:
1 (0)
 
(1) (0)
x1 =2 − x2 + 2x3 = 2
5
14 1
 
(1) (0) (0)
x2 = − − −x1 + 4x3 = −1.5556
9 9
33 1 (0)
 
(1) (0)
x3 = − − x1 + 2x2 = −4.7143
7 7
Second approximation: Substituting the first approximation of the
unknowns in the abovegoverning equation, we get the second
approximations as:
1 (1)
 
(2) (1)
x1 =2 − x2 + 2x3 = 4.1968
5
14 1
 
(2) (1) (1)
x2 = − − −x1 + 4x3 = 1.2063
9 9
33 1 (1)
 
(2) (1)
x3 = − − x1 + 2x2 = −4.5556
7 7
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
25 / 153
 Jacobi Method

Consider a system of linear equations:

3x1 + x2 = 9
x1 + 2x2 = 8

We can rewrite this system in the form Ax = b as:

" #" # " #
3 1 x1 9
=
1 2 x2 8

Using the Jacobi method, we can iterate to find the solution to this
system.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
26 / 153
 Jacobi Method (Contd.)

h i
tarting with an initial guess x (0) = 0 0 , the Jacobi iteration formula is
given by:

x (k +1) = D −1 (b − (L + U )x (k ) )
where D is the diagonal of A , L is the lower triangular part of A , and U
is the upper triangular part of A .
We iterate until convergence to get the solution.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
27 / 153
 Gauss-Seidel Method

• This method is a modification of the Jacobi’s method. Because

the new values can be immediately stored in the location that
held the old values, the storage requirement for x with the
Gauss-Seidel method is half what it would be the Jacobi method
and the rate of convergence is more rapid.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
28 / 153
 Gauss-Seidel Method

• This method is a modification of the Jacobi’s method. Because

the new values can be immediately stored in the location that
held the old values, the storage requirement for x with the
Gauss-Seidel method is half what it would be the Jacobi method
and the rate of convergence is more rapid.
• Consider the systems of algebraic equations

a11 x1 + a12 x2 + a13 x3 + · · · + a1n xn = b1

a21 x1 + a22 x2 + a23 x3 + · · · + a2n xn = b2
..
.
an1 x1 + an2 x2 + an3 x3 + · · · + ann xn = bn

Now, the system can be re-written as follows:

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
28 / 153
 b1 1
x1 = − (a x + a13 x3 · · · + a1n xn )
a11 a11 12 2
b 1
x2 = 2 − (a x + a23 x3 · · · + a2n xn )
a22 a22 21 1
b 1
x3 = 3 − (a x + a32 x2 · · · + a3n xn )
a33 a33 31 1
..
.
b 1
xn = n − (a x + an2 x2 · · · + a1n xn )
ann ann n1 1
(k )
 

 x1 
(k ) 
x2 
 
(k )  th

Let x =  ..  be the (k ) approximation of the given system of


 .  
 (k ) 
xn

equation.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
29 / 153
 Then the (k + 1)th approximation is

b 1
 
(k ) (k ) (k )
x1 (k +1) = 1 − a12 x2 + a13 x3 + · · · a1n xn
a11 a11
b2 1
 
(k +1) (k ) (k ) (k )
x2 = − a x + a23 x3 + · · · a2n xn
a22 a22 21 1
b3 1
 
(k +1) (k ) (k ) (k )
x3 = − a x + a32 x2 + · · · a3n xn
a33 a33 31 1
..
.
bn 1
 
(k +1) (k ) (k ) (k )
xn = − a x + an2 x3 + · · · an(n−1) xn−1
ann ann n1 1
The process is continued till convergence is secured.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
30 / 153
 Example
Solve the system of equations

5x1 + x2 + 2x3 = 10
−3x1 + 9x2 + 4x3 = −14
x1 + 2x2 + 7x3 = −33

using the Gauss-Seidel method correct to two decimal places, with the
h iT
initial approximation x (0) = 0 0 0 .

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
31 / 153
 Solution
• The system is diagonally dominant. Rearranging the equations
we get
1 (k −1)
 
(k ) (k −1)
x1 =2 − x2 + 2x3
5
14 1
 
(k ) (k ) (k −1)
x2 = − − −x1 + 4x3
9 9
33 1 (k )
 
(k ) (k )
x3 = − − x1 + 2x2
7 7
• First approximation:

1 (0)
 
(1) (0)
x1 =2 − x2 + 2x3 = 2
5
14 1
 
(1) (1) (0)
x2 = − − −x1 + 4x3 = −0.8889
9 9
33 1 (1)
 
(1) (1)
x3 = − − x + 2x2 = −4.746
7 7 1
• Second approximation:
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
32 / 153
 Gauss-Seidel Method

Consider the same system of linear equations:

3x1 + x2 = 9 x1 + 2x2 =8

We can use Gauss-Seidel method to find the solution to this system.

The iteration formula for Gauss-Seidel method is given by:
 
i −1 n
(k +1) 1  ¼ (k +1)
¼ 
(k ) 
xi = bi − aij xj − aij xj 

aii  
j =1 j =i +1

We can iterate until convergence to find the solution.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
33 / 153
 Applications of Jacobi and Gauss-Seidel Methods

• Used in solving systems of linear equations

• Computational fluid dynamics
• Finite element analysis
• Image processing
• Machine learning
• Signal processing

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
34 / 153
 Numerical Solution for Ordinary Differential Equations

Introduction
• A differential equation is an equation that contains derivatives of
an unknown function.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
35 / 153
 Numerical Solution for Ordinary Differential Equations

Introduction
• A differential equation is an equation that contains derivatives of
an unknown function.
• ODE: An equation involving the derivatives of an unknown
function y of a single variable x over an interval x ∈ (I ).

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
35 / 153
 Numerical Solution for Ordinary Differential Equations

Introduction
• A differential equation is an equation that contains derivatives of
an unknown function.
• ODE: An equation involving the derivatives of an unknown
function y of a single variable x over an interval x ∈ (I ).
• Ordinary Differential Equations (ODEs) are equations involving an
unknown function of one variable and its derivatives.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
35 / 153
 Numerical Solution for Ordinary Differential Equations

Introduction
• A differential equation is an equation that contains derivatives of
an unknown function.
• ODE: An equation involving the derivatives of an unknown
function y of a single variable x over an interval x ∈ (I ).
• Ordinary Differential Equations (ODEs) are equations involving an
unknown function of one variable and its derivatives.
• Numerical methods are often used to approximate the solutions
of ODEs when analytical solutions are not possible.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
35 / 153
 Ordinary Differential Equation (ODE)

• Any function y = f (x) which satisfies this equation over the

interval (I ) is called a solution of the ODE.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
36 / 153
 Ordinary Differential Equation (ODE)

• Any function y = f (x) which satisfies this equation over the

interval (I ) is called a solution of the ODE.
• A first-order ODE involves the first derivative of the dependent
variable with respect to the independent variable.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
36 / 153
 Ordinary Differential Equation (ODE)

• Any function y = f (x) which satisfies this equation over the

interval (I ) is called a solution of the ODE.
• A first-order ODE involves the first derivative of the dependent
variable with respect to the independent variable.
• For example, if x is the independent variable and y is the
dependent variable, the equation has combinations of the
variables x, y, and dydx

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
36 / 153
 Ordinary Differential Equation (ODE)

• Any function y = f (x) which satisfies this equation over the

interval (I ) is called a solution of the ODE.
• A first-order ODE involves the first derivative of the dependent
variable with respect to the independent variable.
• For example, if x is the independent variable and y is the
dependent variable, the equation has combinations of the
variables x, y, and dydx
• An ODE is said to be order n, if y (n) is the highest order derivative
occurring in the equation.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
36 / 153
 Ordinary Differential Equation (ODE)

• Any function y = f (x) which satisfies this equation over the

interval (I ) is called a solution of the ODE.
• A first-order ODE involves the first derivative of the dependent
variable with respect to the independent variable.
• For example, if x is the independent variable and y is the
dependent variable, the equation has combinations of the
variables x, y, and dydx
• An ODE is said to be order n, if y (n) is the highest order derivative
occurring in the equation.
• Differential equations are possibly the most widely used form of
mathematics in science and engineering.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
36 / 153
 Ordinary Differential Equation (ODE)

• Any function y = f (x) which satisfies this equation over the

interval (I ) is called a solution of the ODE.
• A first-order ODE involves the first derivative of the dependent
variable with respect to the independent variable.
• For example, if x is the independent variable and y is the
dependent variable, the equation has combinations of the
variables x, y, and dydx
• An ODE is said to be order n, if y (n) is the highest order derivative
occurring in the equation.
• Differential equations are possibly the most widely used form of
mathematics in science and engineering.
• An equation involving the partial derivatives of a function of more
than one variable is called PDE.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
36 / 153
 Ordinary Differential Equation (ODE)

• Any function y = f (x) which satisfies this equation over the

interval (I ) is called a solution of the ODE.
• A first-order ODE involves the first derivative of the dependent
variable with respect to the independent variable.
• For example, if x is the independent variable and y is the
dependent variable, the equation has combinations of the
variables x, y, and dydx
• An ODE is said to be order n, if y (n) is the highest order derivative
occurring in the equation.
• Differential equations are possibly the most widely used form of
mathematics in science and engineering.
• An equation involving the partial derivatives of a function of more
than one variable is called PDE.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
36 / 153
 Initial Value Problems (IVPs)

• An initial value problem is an ordinary differential equation of the

form

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
37 / 153
 Initial Value Problems (IVPs)

• An initial value problem is an ordinary differential equation of the

form
dy
= f(x, y) where y(xo ) = yo is given and y ∈ ‘n . (3)
dx
• To solve any differential equation we need to put it into the
standard form given by (3).

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
37 / 153
 Initial Value Problems (IVPs)

• An initial value problem is an ordinary differential equation of the

form
dy
= f(x, y) where y(xo ) = yo is given and y ∈ ‘n . (3)
dx
• To solve any differential equation we need to put it into the
standard form given by (3).
• One of the simplest numerical methods for solving ODEs is Euler’s
Method.
• It approximates the solution at each time step by using the
derivative of the function at that time step.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
37 / 153
 Euler’s Method

• Let’s consider the first-order ODE:

dy
= f (t, y)
dt
• The Euler method updates the solution at each time step ti as:

yi +1 = yi + hf (ti , yi )

where h is the step size.

• Euler’s method is the simplest technique for solving a first-order
ODE of the form:
dy
= f (t, y)
dt
with the initial condition: y(t0 ) = y0 .

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
38 / 153
 Explicit Euler Method example

Example
dy
Consider the differential equation dt = −2y with the initial condition
y(0) = 1.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
39 / 153
 Explicit Euler Method example

Example
Consider the differential equation dy
dt = −2y with the initial condition
y(0) = 1.Approximate the solution using the explicit Euler method with
a time step size of h = 0.1.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
39 / 153
 Explicit Euler Method example

Example
Consider the differential equation dy
dt = −2y with the initial condition
y(0) = 1.Approximate the solution using the explicit Euler method with
a time step size of h = 0.1.

Solution
At time t = 0, y(0) = 1. Using the explicit Euler method:

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
39 / 153
Explicit Euler Method example

Example
Consider the differential equation dy
dt = −2y with the initial condition
y(0) = 1.Approximate the solution using the explicit Euler method with
a time step size of h = 0.1.

Solution
At time t = 0, y(0) = 1. Using the explicit Euler method:

y1 = 1 + 0.1 · (−2 · 1) = 1 − 0.2 = 0.8

y2 = 0.8 + 0.1 · (−2 · 0.8) = 0.8 − 0.16 = 0.64
y3 = 0.64 + 0.1 · (−2 · 0.64) = 0.64 − 0.128 = 0.512
Explicit Euler Method example

Example
Consider the differential equation dy
dt = −2y with the initial condition
y(0) = 1.Approximate the solution using the explicit Euler method with
a time step size of h = 0.1.

Solution
At time t = 0, y(0) = 1. Using the explicit Euler method:

y1 = 1 + 0.1 · (−2 · 1) = 1 − 0.2 = 0.8

y3 = 0.64 + 0.1 · (−2 · 0.64) = 0.64 − 0.128 = 0.512

 Explicit Euler Method example

Example
Consider the differential equation dy
dt = −2y with the initial condition
y(0) = 1.Approximate the solution using the explicit Euler method with
a time step size of h = 0.1.

Solution
At time t = 0, y(0) = 1. Using the explicit Euler method:

y1 = 1 + 0.1 · (−2 · 1) = 1 − 0.2 = 0.8

y2 = 0.8 + 0.1 · (−2 · 0.8) = 0.8 − 0.16 = 0.64
y3 = 0.64 + 0.1 · (−2 · 0.64) = 0.64 − 0.128 = 0.512

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
39 / 153
 Example

• Consider the ODE dy

dt = t − y with initial condition y(0) = 1.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
40 / 153
 Example

• Consider the ODE dydt = t − y with initial condition y(0) = 1.

• Let’s use Euler’s method with step size h = 0.1 to approximate the
solution.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
40 / 153
Example

• Consider the ODE dydt = t − y with initial condition y(0) = 1.

• Let’s use Euler’s method with step size h = 0.1 to approximate the
solution.

yi +1 = yi + h (ti − yi )
y1 = 1 + 0.1(0 − 1) = 1 − 0.1 = 0.9
y2 = 0.9 + 0.1(0.1 − 0.9) = 0.9 + 0.1(−0.8) = 0.9 − 0.08 = 0.82
..
.

• The numerical approximation is y(1) ≈ 0.83.

Example

• Consider the ODE dydt = t − y with initial condition y(0) = 1.

• Let’s use Euler’s method with step size h = 0.1 to approximate the
solution.

yi +1 = yi + h (ti − yi )

y2 = 0.9 + 0.1(0.1 − 0.9) = 0.9 + 0.1(−0.8) = 0.9 − 0.08 = 0.82

..
.

• The numerical approximation is y(1) ≈ 0.83.

Example

• Consider the ODE dydt = t − y with initial condition y(0) = 1.

• Let’s use Euler’s method with step size h = 0.1 to approximate the
solution.

yi +1 = yi + h (ti − yi )

..
.

• The numerical approximation is y(1) ≈ 0.83.

 Example

• Consider the ODE dydt = t − y with initial condition y(0) = 1.

• Let’s use Euler’s method with step size h = 0.1 to approximate the
solution.

yi +1 = yi + h (ti − yi )
y1 = 1 + 0.1(0 − 1) = 1 − 0.1 = 0.9
y2 = 0.9 + 0.1(0.1 − 0.9) = 0.9 + 0.1(−0.8) = 0.9 − 0.08 = 0.82
..
.

• The numerical approximation is y(1) ≈ 0.83.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
40 / 153
 Example (Population growth Examples)
Given the population growth rate of 2% per year, we can model the
population size using the differential equation:

dP
= 0.02P
dt
Using the explicit Euler method with a step size h = 1 year:
Let’s assume an initial population size of P0 = 1000.
Iteration 1:
P1 = 1000 + 1 · (0.02 · 1000) = 1020
Iteration 2:
P2 = 1020 + 1 · (0.02 · 1020) = 1040.4
Iteration 3:

P3 = 1040.4 + 1 · (0.02 · 1040.4) = 1061.208

Therefore, the approximated population size at t = 0, 1, 2, 3 years are

1000, 1020, 1040.4, and 1061.208, respectively.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
41 / 153
 Example (Electrical Circuits:)
Let’s consider a simple circuit with a resistor R = 5 ohms and a voltage
source V = 10 volts. The voltage across the resistor can be modeled
by the differential equation:
dVR V − VR
=
dt R
Using the explicit Euler method with a step size of h = 1:
Assuming an initial voltage across the resistor of VR = 0:
Iteration 1:
VR 1 = 0 + 1 · ((10 − 0)/5) = 2
Iteration 2:
VR 2 = 2 + 1 · ((10 − 2)/5) = 2.6
Iteration 3:
VR 3 = 2.6 + 1 · ((10 − 2.6)/5) = 3.08
Therefore, the approximated voltage across the resistor at t = 0, 1, 2, 3
seconds are 0, 2, 2.6, and 3.08 volts, respectively.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
42 / 153
 Example
Consider a mining company extracting a mineral from a deposit over
time. The rate of extraction is modeled by the ODE:
dM
= −kM
dt
where M is the amount of mineral in the deposit at time t, and k is the
extraction rate constant.
Let’s use Euler’s method with h = 0.5 and M (0) = 1000 to approximate
the amount of mineral remaining after 5 units of time.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
43 / 153
 Solution
The update equation using Euler’s method is:

Mi +1 = Mi − hkMi

Applying this formula iteratively, we get:

M1 = 1000 − 0.5 × k × 1000

M2 = M1 − 0.5 × k × M1
M3 = M2 − 0.5 × k × M2
M4 = M3 − 0.5 × k × M3
M5 = M4 − 0.5 × k × M4

Finally, M5 gives us the approximate amount of mineral remaining

after 5 units of time using Euler’s method.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
44 / 153
 Example (Leaching Process)
In the leaching process, a solvent is used to extract valuable minerals
from an ore. The rate of extraction can be modeled by the ODE:
dM
= −k1 M + k2
dt
where M is the amount of mineral in the ore at time t, and k1 , k2 are
rate constants.
Apply Euler’s method to estimate the amount of mineral remaining
after 10 units of time with initial condition M (0) = 500.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
45 / 153
 Example (Slope Stability Analysis)
In slope stability analysis, the rate of erosion of a sloping terrain can
be modeled by the ODE:
dH
= −k3 H + k4
dt
where H is the height of the slope at time t, and k3 , k4 are constants.
Use Euler’s method with h = 0.1 to predict the height of the slope after
15 units of time with initial condition H (0) = 200.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
46 / 153
 Euler Algorithm

Algorithm 1 Forward Euler method

Inputs: initial condition y0 , end time tend .

Initialize: t0 = 0, y0 .
for n = 0 to tend /h do
tn = nh
yn+1 ← yn + hf (tn , yn )
end for
return y vector.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
47 / 153
 Matlab code for Forward Euler method
1 function y = f o r w a r d E u l e r ( y0 , tEnd , h )
2 % I n p u t s : i n i t i a l c o n d i t i o n y0 , end time tEnd , time step
size h
3 % Initialize
4 t0 = 0 ;
5 y = y0 ;
6
7 % Loop through time steps
8 f o r n = 0 : tEnd /h
9 tn = n ∗ h ;
0 yn = y ( end ) ;
1 ynew = yn + h ∗ f ( tn , yn ) ; % Forward E u l e r update
2 y = [ y ynew ] ;
3 end
4 end
5
6 % D e f i n e the f u n c t i o n f ( t , y )
7 function dydt = f ( t , y )
8 % D e f i n e the ODE f u n c t i o n here
9 dydt = % ODE f u n c t i o n
0 Advanced Computational
end Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
48 / 153
 Example
Use Euler’s explicit method to solve the ODE dy dx = −1.2y + 7e
−0.3x from

x = 0 to x = 2.5 with the initial condition y = 3 at x = 0.

(a) Solve by hand using h = 0.5.
(b) Write a MATLAB program in a script file that solves the equation
using h = 0.1.
(c) Use the program from part (b) to solve the equation using h = 0.01.
In each part, compare the results with the exact (analytical) solution:
y = 7e −0.3x − 2e −1.2x .

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
49 / 153
 Euler’s Method - Iterations

• At x = 0.5: y(0.5) ≈ 4.7

• At x = 1.0: y(1.0) ≈ 4.89245
• At x = 1.5: y(1.5) ≈ 4.54985
• At x = 2.0: y(2.0) ≈ 4.03649
• At x = 2.5: y(2.5) ≈ 3.53539

x y f (x, y) yapprox
0.0 3.0 -1.2(3) + 7 3.0
0.5 4.7 -1.2(4.7) + 7e −0.3(0) 4.7
1.0 4.89245 -1.2(4.89245) + 7e −0.3(0.5) 4.89245
1.5 4.54985 -1.2(4.54985) + 7e −0.3(1.0) 4.54985
2.0 4.03649 -1.2(4.03649) + 7e −0.3(1.5) 4.03649
2.5 3.53539 -1.2(3.53539) + 7e −0.3(2.0) 3.53539

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
50 / 153
 i xi Yi (numerical) Yi (exact)
1 0.0 3.0 3.0
2 0.5 4.70 4.072
3 1.0 4.893 4.323
4 1.5 4.55 4.170
5 2.0 4.052 3.835
6 2.5 3.542 3.436

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
51 / 153
 Comparison of Exact and Euler Solution for ODE Comparison of Exact and Euler Solution for ODE Comparison of Exact and Euler Solution for ODE
5 4.5 4.4
Exact Solution Exact Solution Exact Solution
4.8 Euler Solution Euler Solution Euler Solution
4.2
4.6

4.4 4
4
4.2
3.8
4
y

y
3.6
3.8
3.5
3.6 3.4

3.4
3.2
3.2

3 3 3
0 0.5 1 1.5 2 2.5 0 0.5 1 1.5 2 2.5 0 0.5 1 1.5 2 2.5
x x x

(a) h = 0.5 (b) h = 0.1 (c) h = 0.01

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
52 / 153
 Runge-Kutta Methods
• Runge-Kutta methods are a class of numerical methods used to
solve ordinary differential equations (ODEs) by approximating the
solution at discrete points.
• The most commonly used Runge-Kutta method is the 4th order
method, which is given by the following formula:

k1 = hf (tn , yn )
h k
k2 = hf (tn + , yn + 1 )
2 2
h k2
k3 = hf (tn + , yn + )
2 2
k4 = hf (tn + h , yn + k3 )
1
yn+1 = yn + (k1 + 2k2 + 2k3 + k4 )
6
where h is the step size, tn is the current time, yn is the current
approximate solution, and f (t, y) is the ODE.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
53 / 153
 Example
Consider the initial value problem:
dy
= y −t
dt
y(0) = 1

Using the 4th order Runge-Kutta method, let’s approximate y(0.1) with
a step size of h = 0.05.

k1 = 0.05 · (yn − tn )
k2 = 0.05 · (yn + 0.025k1 − (tn + 0.025))
k3 = 0.05 · (yn + 0.025k2 − (tn + 0.025))
k4 = 0.05 · (yn + k3 − (tn + 0.05))
1
yn+1 = yn + (k1 + 2k2 + 2k3 + k4 )
6
Solving these equations step by step, we get y(0.1) ≈ 1.0545.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
54 / 153
 Runge-Kutta Method Examples

Example (Flow Rate in Underground Mine)

In an underground mine, the flow rate of water into the mine can be
modeled by the ODE:
dQ
= −k5 Q + k6
dt
where Q is the flow rate of water at time t, and k5 , k6 are given
parameters.
Apply the 2nd order Runge-Kutta method to estimate the flow rate
after 8 units of time with initial condition Q (0) = 100.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
55 / 153
 Solution
Given ODE: dQ dt = −k5 Q + k6
Initial condition: Q (0) = 100
Parameters: k5 = 0.5, k6 = 10
We will use the 2nd order Runge-Kutta method to estimate the flow
rate after 8 units of time.
Calculate k 1 and k 2:

k 1 = hf (tn , Qn ) = 1(−0.5 × 100 + 10) = 5

k 2 = hf (tn + h , Qn + k 1) = 1(−0.5 × (100 + 5) + 10) = 2.5

Update the solution:

k1 + k2
Qn+1 = Qn +
2
5 + 2.5
= 100 +
2
= 101.25

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
56 / 153
 Example (Rock Stability Analysis)
For rock stability analysis, the rate of deformation in a rock layer can
be described by the ODE:
dD
= −k7 D 2 + k8
dt
where D is the deformation of the rock layer at time t, and k7 , k8 are
constants.
Utilize the 4th order Runge-Kutta method with step size h = 0.2 to
predict the deformation after 12 units of time with initial condition
D (0) = 10.

Solution
Given ODE: dP dt = 0.1P
Initial condition: P (0) = 100
Using 4th order Runge-Kutta method to estimate the population
growth after 5 time units:
Calculate k 1, k 2, k 3, and k 4 at each step using the formula:
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
57 / 153
 Solution

k 1 = hf (tn , Pn ) = 0.1 × 100 = 10

h k1 10
k 2 = hf (tn + , Pn + ) = 0.1 × (100 + ) = 11
2 2 2
h k2 11
k 3 = hf (tn + , Pn + ) = 0.1 × (100 + ) = 11.1
2 2 2
k 4 = hf (tn + h , Pn + k 3) = 0.1 × (100 + 11.1) = 11.11

Update the solution:

k 1 + 2k 2 + 2k 3 + k 4
Pn+1 = Pn +
6
10 + 2(11) + 2(11.1) + 11.11
= 100 +
6
≈ 123.92

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
58 / 153
 More Runge-Kutta Method Examples

Example (Ventilation System Dynamics)

In a mining ventilation system, the rate of air flow can be modeled by
the ODE:
dV
= −k9 V + k10
dt
where V is the air flow rate at time t, and k9 , k10 are parameters.
Apply the 4th order Runge-Kutta method to estimate the air flow rate
after 6 units of time with an initial condition of V (0) = 500.

Solution
Given ODEs:
dv
= −9.81 − 0.1v
dt
dx
=v
dt
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
59 / 153
 Solution
Initial conditions: x(0) = 0 and v(0) = 10
Using 4th order Runge-Kutta method to estimate the position and
velocity after 2 seconds:
Calculate k 1, k 2, k 3, and k 4 at each step using the given ODEs:

k1v = h (−9.81 − 0.1 × 10) = −10.81

k1x = h × 10 = 20
k1v
k2v = h (−9.81 − 0.1 × (10 + )) = −10.667
2
k1x
k2x = h × (10 + ) = 20.5
2
k2v
k3v = h (−9.81 − 0.1 × (10 + )) = −10.523
2

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
60 / 153
 k2x
k3x = h × (10 + ) = 20.25
2
k4v = h (−9.81 − 0.1 × (10 + k3v )) = −10.37
k4x = h × (10 + k3x ) = 20.5

Update the solution:

k1v + 2k2v + 2k3v + k4v
vn+1 = 10 +
6
−10.81 + 2(−10.667) + 2(−10.523) − 10.37
≈ 10 + ≈ 8.956
6
k + 2k2x + 2k3x + k4x
xn+1 = 0 + 1x
6
20 + 2(20.5) + 2(20.25) + 20.5
≈ 0+ ≈ 41.5
6
After 2 seconds, the estimated velocity is approximately 8.956 m/s
and the estimated position is approximately 41.5 meters.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
61 / 153
 Example (Mine Shaft Pumping)
For mine shaft pumping, the water level in a shaft can be described by
the ODE:
dh
= −k11 h 2 + k12
dt
where h is the water level at time t, and k11 , k12 are given constants.
Use the 5th order Runge-Kutta method with a step size of h = 0.1 to
predict the water level after 10 units of time with an initial condition of
h (0) = 20.

Justification
Given ODE: dh 2
dt = −k11 h + k12
Given parameters: k11 , k12 , h (0) = 20, h = 0.1, t = 10.
Define function f (h , t) = −k11 h 2 + k12 .

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
62 / 153
 Detailed Solution
• Apply 5th Order Runge-Kutta:

k 1 = h · f (hn , tn )
k1 h
k 2 = h · f (hn + , tn + )
4 4
3k 1 9k 2 3h
k 3 = h · f (hn + + , tn + )
32 32 8
1932k 1 7200k 2 7296k 3 12h
k 4 = h · f (hn + − + , tn + )
2197 2197 2197 13
439k 1 3680k 3 845k 4
k 5 = h · f (hn + − 8k 2 + − ,t + h)
216 513 4104 n
8k 1 3544k 3 1859k 4 11k 5 h
k 6 = h · f (hn − + 2k 2 − + − , tn + )
27 2565 4104 40 2
25(k 1 + 6k 3 + 8k 4 − 9k 5 + 2k 6)
hn+1 = hn +
216
• Perform calculations iteratively until t = 10.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
63 / 153
 Rock Fragmentation during Blasting

Problem Statement
• A mining company is conducting a blasting operation in a rock
quarry to fragment the rock.
• The rate of change of the size of rock fragments during blasting
is given by the differential equation:
dD
= −k1 D + k2
dt

• Where:
• D is the average diameter of rock fragments
• k1 = 0.02 (fracture constant)
• k2 = 0.5 (constant rate of fragmentation)
• Initial condition: D (0) = 50 cm
• Time step: h = 0.1 seconds

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
64 / 153
 Solution using Fourth Order Runge-Kutta Method
• Let’s apply the Fourth Order Runge-Kutta method to solve the
differential equation for rock fragmentation during blasting.
• Given differential equation:

dD
= −k1 D + k2
dt
• Initial condition: D (0) = 50 cm
• Constants: k1 = 0.02, k2 = 0.5
• Time step: h = 0.1 seconds
• Using the Fourth Order Runge-Kutta method, the update formula
is:
1
Di +1 = Di + (k1 + 2k2 + 2k3 + k4 )
6
Where:

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
65 / 153
 k1 = h (−k1 Di + k2 )
h
   
k2 = h −k1 Di + k1 + k2
2
h
   
k3 = h −k1 Di + k2 + k2
2
k4 = h (−k1 (Di + hk3 ) + k2 )

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
66 / 153
 Mine Shaft Ventilation Analysis

Problem Statement
• An underground mine requires adequate ventilation to maintain a
safe working environment for miners.
• The differential equation governing air flow in the shaft is given
by:
dQ
= −k3 Q + k4
dz
Where:
• Q is the air flow rate
• z is the depth of the mine shaft
• k3 = 0.01 (decrease rate of air flow)
• k4 = 2 (constant rate of air supply)
• Initial condition: Q (0) = 10 m3 /min
• Step size: h = 0.5 meters
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
67 / 153
 Solution using Fourth Order Runge-Kutta Method
• Let’s apply the Fourth Order Runge-Kutta method to solve the
differential equation for mine shaft ventilation analysis.
• The update formula for the Fourth Order Runge-Kutta method is:

1
Qi +1 = Qi + (k1 + 2k2 + 2k3 + k4 )
6
Where:

k1 = h (−k3 Qi + k4 )
h
   
k2 = h −k3 Qi + k1 + k4
2
h
   
k3 = h −k3 Qi + k2 + k4
2
k4 = h (−k3 (Qi + hk3 ) + k4 )

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
68 / 153
 Introduction to Partial Differential Equations

• Partial differential equations (PDEs) involve functions of several

variables and their partial derivatives.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
69 / 153
 Introduction to Partial Differential Equations

• Partial differential equations (PDEs) involve functions of several

variables and their partial derivatives.
• They arise in various areas of physics, engineering, and
mathematics to describe the behavior of systems that depend on
more than one independent variable.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
69 / 153
 Introduction to Partial Differential Equations

• Partial differential equations (PDEs) involve functions of several

variables and their partial derivatives.
• They arise in various areas of physics, engineering, and
mathematics to describe the behavior of systems that depend on
more than one independent variable.
• The general form of a partial differential equation is:

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
69 / 153
 Introduction to Partial Differential Equations

• Partial differential equations (PDEs) involve functions of several

variables and their partial derivatives.
• They arise in various areas of physics, engineering, and
mathematics to describe the behavior of systems that depend on
more than one independent variable.
• The general form of a partial differential equation is:

u u u 2 u
F (x1 , x2 , . . . , xn , u, , ,..., , , . . .) = 0 (4)
x1 x2 xn x1 x2

where u(x1 , x2 , . . . , xn ) is the unknown function and its partial

derivatives are involved in the equation.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
69 / 153
 Classification of Partial Differential Equations

• PDEs can be classified into different types based on their order

and linearity.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
70 / 153
 Classification of Partial Differential Equations

• PDEs can be classified into different types based on their order

and linearity.
• The order of a PDE is determined by the highest order of the
partial derivatives involved in the equation.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
70 / 153
 Classification of Partial Differential Equations

• PDEs can be classified into different types based on their order

and linearity.
• The order of a PDE is determined by the highest order of the
partial derivatives involved in the equation.
• The order can be 1st, 2nd, or higher.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
70 / 153
 Classification of Partial Differential Equations

• PDEs can be classified into different types based on their order

and linearity.
• The order of a PDE is determined by the highest order of the
partial derivatives involved in the equation.
• The order can be 1st, 2nd, or higher.
• PDEs can also be classified as linear or nonlinear based on how
the unknown function u and its derivatives appear in the
equation.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
70 / 153
 Classification of Partial Differential Equations

• PDEs can be classified into different types based on their order

and linearity.
• The order of a PDE is determined by the highest order of the
partial derivatives involved in the equation.
• The order can be 1st, 2nd, or higher.
• PDEs can also be classified as linear or nonlinear based on how
the unknown function u and its derivatives appear in the
equation.
• A linear PDE is of the form:

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
70 / 153
 Classification of Partial Differential Equations

• PDEs can be classified into different types based on their order

and linearity.
• The order of a PDE is determined by the highest order of the
partial derivatives involved in the equation.
• The order can be 1st, 2nd, or higher.
• PDEs can also be classified as linear or nonlinear based on how
the unknown function u and its derivatives appear in the
equation.
• A linear PDE is of the form:

2 u 2 u 2 u
a1 + a2 + . . . + an = f (x1 , x2 , . . . , xn ) (5)
x12 x22 xn2

where ai (x1 , x2 , . . . , xn ) and f (x1 , x2 , . . . , xn ) are functions of the

independent variables.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
70 / 153
 Types of Partial Differential Equations

Some common types of PDEs include:

• Heat Equation: Describes the distribution of heat in a material
over time.
• Wave Equation: Describes the propagation of waves in a medium.
• Laplace’s Equation: Describes the steady-state distribution of a
physical quantity.
• Schrodinger Equation: Describes the quantum behavior of
particles.
These types of PDEs have specific forms and properties that govern
their behavior and solutions.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
71 / 153
 Mathematical Equations

Heat Equation
The heat equation is given by:

2 u 2 u 2 u
!
u
=k + + (6)
t x 2 y 2 z 2

where u(x, y, z, t) is the temperature distribution function and k is the

thermal conductivity.

Laplace’s Equation
Laplace’s equation is given by:

2 u 2 u 2 u
+ + =0 (7)
x 2 y 2 z 2

where u(x, y, z) is the function that satisfies Laplace’s equation.

Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
72 / 153
 Wave Equation
The wave equation is given by:

2 u 2 2 u 2 u
!
2  u
= c + + (8)
t 2 x 2 y 2 z 2

where u(x, y, z, t) is the wave function and c is the speed of

propagation of the wave.

Schrödinger equation
The Schrödinger equation in one dimension is given by:

Ñ (x, t) ℏ2 2 Ñ (x, t)
iℏ =− + V (x)Ñ (x, t) (9)
t 2m x 2
where: Ñ (x, t) is the wave function of the particle at position x and
h
time t, i is the imaginary unit, ℏ is the reduced Planck constant ( 2á ), m
is the mass of the particle, - V (x) is the potential energy of the particle
at position x.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
73 / 153
 Introduction to Finite Differences

Definition
Finite differences is a numerical method for approximating the
solutions of differential equations by replacing derivatives with finite
differences.

• Different types of finite difference schemes: forward, backward,

central differences.
• Applications in solving ordinary and partial differential equations.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
74 / 153
 Forward Difference Approximation

Forward Difference Formula

The forward difference approximation of the first derivative f ′ (x) at a
point x is given by:
f (x + h ) − f (x)
f ′ (x) ≈
h

Example
Consider the function f (x) = e x and its derivative f ′ (x) = e x . Using
forward difference approximation with h = 0.1 at x = 0, we have:

f (0.1) − f (0) 1.1052 − 1

f ′ (0) ≈ ≈ ≈ 1.0528
0.1 0.1

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
75 / 153
 Central Difference Approximation

Central Difference Formula

The central difference approximation of the first derivative f ′ (x) at a
point x is given by:
f (x + h ) − f (x − h )
f ′ (x) ≈
2h

Example
Consider the function f (x) = sin(x) and its derivative f ′ (x) = cos(x).
Using central difference approximation with h = 0.01 at x = 0, we have:

f (0.01) − f (−0.01) 0.9999998333 − 0.9999998333

f ′ (0) ≈ ≈ ≈0
0.02 0.02

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
76 / 153
 Boundary Value Problems
Examples of ODE/PDE
1 Initial value problems (IVP). A canonical first-order system is

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
77 / 153
 Boundary Value Problems
Examples of ODE/PDE
1 Initial value problems (IVP). A canonical first-order system is
dy
= f (t, y); y(t0 ) = y0 ; (10)
dt
and a single higher-order differential equation may be rewritten
as a first-order system. For example, a second-order ODE

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
77 / 153
 Boundary Value Problems
Examples of ODE/PDE
1 Initial value problems (IVP). A canonical first-order system is
dy
= f (t, y); y(t0 ) = y0 ; (10)
dt
and a single higher-order differential equation may be rewritten
as a first-order system. For example, a second-order ODE

u ′′ (t) + a(t)u ′ (t) + b (t)u(t) = f (t); (11)

with initial conditions

u(0) = u0 ,
u ′ (0) = v0 .

can be converted into a first-order system by setting y1 (t) = u and

y2 (t) = u ′ (t).
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
77 / 153
 An ODE IVP can often be solved using Runge-Kutta methods, with
adaptive time steps. In Matlab, there is the ODE-Suite which includes
ode45, ode23, ode23s, ode15s, etc. For a stiff ODE system, either
ode23s or ode15s is recommended
2 Boundary value problems (BVP). An example of an ODE BVP is

u ′′ (x) + a(x)u ′ (x) + b (x)u(x) = f (x); 0 < x < 1; (12)

u(0) = u0 ; u(1) = u1 ; (13)

and a PDE BVP example is

uxx + uyy = f (x, y); (x, y) ∈ Ò; (14)

u(x, y) = u0 (x, y); (x, y) ∈ Ò; (15)

2 2
where uxx = xu2 and uyy = yu2 , in a domain Ò with boundary Ò.
The above PDE is linear and classified as elliptic, and there are
two other classifications for linear PDE, namely, parabolic and
hyperbolic.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
78 / 153
 3 Boundary value problem (BVP) and initial value problem (IVP), e.g.,

ut = auxx + f (x, t); (16)

u(0, t) = g1 (t); u(1, t) = g2 (t); BC
u(x, 0) = u0 (x); IC;

where BC and IC stand for boundary condition(s) and initial

condition, respectively, where ut = u
t
.
4 Eigenvalue problems, e.g.,

u ′′ (x) = Ýu(x); (17)

u(0) = 0; u(1) = 0; (18)

In this example, both the function u(x) (the eigenfunction) and the
scalar Ý (the eigenvalue) are unknowns.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
79 / 153
 5 Diffusion and reaction equations, e.g.,

u
= ∇ · (Ô∇u) + a · ∇u + f (u) (19)
t
where a is a vector, ∇ · (Ô∇u) is a diffusion term, a · ∇u is called an
advection term, and f (u) a reaction term.
6 Systems of PDE. The incompressible Navier–Stokes model is an
important nonlinear example:
!
u
â + (u · ∇)u = ∇p + ÞÉu + F ; (20)
t

∇ · u = 0; (21)

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
80 / 153
 A linear second-order PDE has the following general form:

a(x, y)uxx +2b (x, y)uxy +c(x, y)uyy +d (x, y)ux +e(x, y)uy +g(x, y)u(x, y) = f (x, y
(22)
where the coefficients are independent of u(x, y) so the equation is
linear in u and its partial derivatives. The solution of the 2D linear PDE
is sought in some bounded domain Ò, and the classification of the
PDE form (22) is:
• Elliptic if b 2 − ac < 0 for all (x, y) ∈ Ò,
• Parabolic if b 2 − ac = 0 for all (x, y) ∈ Ò, and
• Hyperbolic if b 2 − ac > 0 for all (x, y) ∈ Ò.
The appropriate solution method typically depends on the equation
class. For the first-order system:

u u
= A (x) (23)
t x
the classification is determined from the eigenvalues of the coefficient
matrix A (x).
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
81 / 153
 Numerical Solutions of Boundary Value Problems

Definition
A boundary value problem is a differential equation together with a
set of additional constraints, called boundary conditions, that the
solution must satisfy at the boundaries of the domain.

Example
Consider the boundary value problem:

u ′′ (x) = −u(x), 0<x <1

u(0) = 1, u(1) = 2

• To solve this boundary value problem numerically, we can use

finite difference methods.
• Let’s discretize the domain [0, 1] into N points, and approximate
the second derivative with a central difference scheme:
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
82 / 153
 Numerical Solutions of Boundary Value Problems
(cont’d)

ui +1 − 2ui + ui −1
u ′′ (x) ≈
h2
where h = N1 is the step size. Applying the central difference scheme
to the boundary value problem, we obtain the following system of
equations:
ui +1 − 2ui + ui −1
= −ui , 1 ≤ i ≤ N −1
h2
u0 = 1
uN = 2

This system of equations can be represented as a tridiagonal matrix

equation:

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
83 / 153
  2 −1 0 · · ·
    
0  u1   u1 
−1 2 −1 · · · 0  u2   u2 
    
 0 −1 2 · · ·
     
0  u3  = −h 2  u3 
 .. .. .. ..   ..   . 

..  . 
 .
 . . . .   .   . 
0 −1
   
0 0 2 uN −1 uN −1

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
84 / 153
 Finite Difference Approximations

• Finite difference and finite element methods are suitable

techniques to solve differential equations (ODEs and PDEs)
numerically.
• There are other methods as well, for example, finite volume
methods, collocation methods,spectral methods, etc.
• Finite difference methods are numerical techniques used to solve
differential equations by approximating derivatives with finite
differences.
• They are widely used in various fields such as physics,
engineering, and mathematics.
• The basic idea behind finite difference methods is to approximate
derivatives with finite differences. Consider a function f (x) and its
derivative f ′ (x).

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
85 / 153
 • The derivative at a point can be approximated using the central
difference formula:
f (x + h ) − f (x − h )
f ′ (x) ≈ (24)
2h
where h is the step size.
• Similarly, the second derivative can be approximated using the
central difference formula:
f (x + h ) − 2f (x) + f (x − h )
f ′′ (x) ≈ (25)
h2
• These approximations can be used to discretize a differential
equation and solve it numerically.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
86 / 153
 Finite Difference Methods

• Often relatively simple to use, and quite easy to understand.

• Easy to implement for regular domains, e.g., rectangular domains
in Cartesian coordinates, and circular or annular domains in
polar coordinates.
• Their discretization and approximate solutions are pointwise, and
the fundamental mathematical tool is the Taylor expansion.
• There are many fast solvers and packages for regular domains,
e.g., the Poisson solvers Fishpack (Adams et al.) and Clawpack
(LeVeque, 1998).
• Difficult to implement for complicated geometries.
• Have strong regularity requirements (the existence of high-order
derivatives).

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
87 / 153
 Finite Element Methods

• Very successful for structural (elliptic type) problems.

• Suitable approach for problems with complicated boundaries.
• Sound theoretical foundation, at least for elliptic PDE, using
Sobolev space theory.
• Weaker regularity requirements.
• Many commercial packages, e.g., Ansys, Matlab PDE Tool-Box,
Triangle, and PLTMG.
• Usually coupled with multigrid solvers.
• Mesh generation can be difficult, but there are now many
packages that do this, e.g., Matlab, Triangle, Pltmg, Fidap, Gmsh,
and Ansys.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
88 / 153
 A Simple Example of a Finite Difference Method

Let us consider a model problem

u ′′ (x) = f (x); 0 < x < 1; u(0) = ua ; u(1) = ub ;

to illustrate the general procedure using a finite difference method as

follows.
1 Generate a grid. A grid is a finite set of points on which we seek
the function values that represent an approximate solution to the
differential equation. For example, given an integer parameter
n > 0, we can use a uniform Cartesian grid
1
xi = ih ; i = 0, 1, . . . , n; h= .
n
The parameter n can be chosen according to accuracy
requirement. If we wish that the approximate solution has four
significant digits, then we can take n = 100 or larger, for example.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
89 / 153
 • Represent the derivative by some finite difference formula at
every grid point where the solution is unknown, to get an
algebraic system of equations.
• Note that for a twice differentiable function æ(x), we have

æ(x − Éx) − 2æ(x) + æ(x + Éx)

æ′′ (x) = lim .
Éx→0 (Éx)2

Thus at a grid point xi , we can approximate u ′′ (xi ) using nearby

function values to get a finite difference formula for the
second-order derivative:
u(xi − h ) − 2u(xi ) + u(xi + h )
u ′′ (xi ) ≈ ,
h2

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
90 / 153
 with some error in the approximation. In the finite difference method,
we replace the differential equation at each grid point xi by

u(xi − h ) − 2u(xi ) + u(xi + h )

= f (xi ) + error,
h2
where the error is called the local truncation error and will be
reconsidered later. Thus we define the finite difference (FD) solution
(an approximation) for u(x) at all xi as the solution Ui (if it exists) of
the following linear system of algebraic equations.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
91 / 153
 ua − 2U1 + U2
= f (x1 )
h2
U1 − 2U2 + U3
= f (x2 )
h2
U2 − 2U3 + U4
= f (x3 )
h2
.. .
. = ..
Ui −1 − 2Ui + Ui +1
= f (xi )
h2
.. .
. = ..
Un−3 − 2Un−2 + Un−1
= f (xn−2 )
h2
Un−2 − 2Un−1 + ub
= f (xn−1 )
h2
Note that the finite difference equation at each grid point involves
solution values at three grid points, i.e., at xi −1 , xi , and xi +1 . The set of
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
92 / 153
 • Solve the system of algebraic equations to get an approximate
solution at each grid point. The system of algebraic equations
can be written in the matrix and vector form as:

  U1   f (x1 ) − ua /h 2 
    
−2/h 1/h 0 ... 0 0
 1/h −2/h 1/h ... 0 0   U2   f (x2 )
     

     
 0 1/h −2/h ... 0 0   U3   f (x3 ) 
 . .. .. .. .. ..   .  =  ..

 . .   .   
 . . . . .   .   . 

    
 0
 0 0 ... 1/h −2/h  Un−2   f (xn−2 ) 

0 0 0 ... 1/h −2/h Un−1 f (xn−1 ) − ub /h 2
   

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
93 / 153
 Solution of Tridiagonal System of Linear Equations

The tridiagonal system of linear equations above can be solved

efficiently in O (Cn) operations by the Crout or Cholesky algorithm, as
described in Burden and Faires (2010), where C is a constant (typically
C = 5 in this case).
1 Implement and debug the computer code.
2 Run the program to get the output.
3 Analyze and visualize the results (tables, plots, etc.).
Error Analysis: Algorithmic consistency and stability imply the
convergence of the finite difference method, which will be discussed
later. The convergence is point-wise, i.e., limh →0 ||u(xi ) − Ui ||∞ = 0. The
finite difference method requires the solution u(x) to have up to
second-order continuous derivatives.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
94 / 153
 Matlab Code for the Model Problem
A Matlab function called two-point.m is presented below for the
model problem. This function uses the centered finite difference
scheme to solve a two-point boundary value problem:
function [x,U] = two_point(a,b,ua,ub,f,n)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% This matlab function two_point solves the following %
% two-point boundary value problem: u’’(x) = f(x) %
% using the centered finite difference scheme. %
% Input: %
% a, b: Two end points. %
% ua, ub: Dirichlet boundary conditions at a and b %
% f: external function f(x). %
% n: number of grid points. %
% Output: %
% x: x(1),x(2),...x(n-1) are grid points %
% U: U(1),U(2),...U(n-1) are approximate solution at %
% grid points %
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
95 / 153
 Using Matlab for Solving the Model Problem

• We can call the Matlab function two-point directly in a Matlab

command window, but a better way is to put all Matlab commands
in a Matlab file (called an M-file), referred to here as main.m.
• This approach allows us to keep a record of our work and revisit
or modify the file whenever needed.
• To illustrate, let’s consider a differential equation defined in the
interval (0, 1) with f (x) = −á2 cos(áx), u(0) = 0, and u(1) = −1.
Here is a sample Matlab M-file:

%%%%%%%% Clear all unwanted variables and graphs.

clear; close all
%%%%%%% Input
a=0; b=1; n=40;
ua=1; ub=-1;

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
96 / 153
 %%%%%% Call the solver: U is the FD solution at the grid poin
[x,U] = two_point(a,b,ua,ub,’f’,n);
%%%%%%%%%%%%%%%%% Plot and show the error %%%%%%%%%%%%%%%%%%
plot(x,U,’o’); hold % Plot the computed solution
u=zeros(n-1,1);
for i=1:n-1,
u(i) = cos(pi*x(i));
end
plot(x,u) %%% Plot the true solution at the grid points on th
%%%%%%% Plot the error
figure(2); plot(x,U-u)
norm(U-u,inf) %%% Print out the maximum error.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
97 / 153
 Fundamentals of Finite Difference Methods

• The Taylor expansion is the most important tool in the analysis of

finite difference methods.
• It may be written as an infinite series

h 2 ′′ h k (k )
u(x + h ) = u(x) + hu ′ (x) + u (x) + . . . + u (x) + . . . (2.2)
2 k!
if u(x) is "analytic" (differentiable to any order), or as a finite sum

h 2 ′′ h k (k )
u(x + h ) = u(x) + hu ′ (x) + u (x) + . . . + u (à); (2.3)
2 k!
where x < à < x + h (or x + h < à < x if h < 0), if u(x) is differentiable
up to k -th order.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
98 / 153
 • The second form of the Taylor expansion is sometimes called the
extended mean value theorem.
• As indicated earlier, we may represent derivatives of a differential
equation by finite difference formulas at grid points to get a linear
or nonlinear algebraic system.
• There are several kinds of finite difference formulas to consider,
but in general their accuracy is directly related to the magnitude
of h (typically small).

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
99 / 153
 Forward, Backward, and Central Finite Difference
Formulas for u ′ (x)

• The forward difference formula is given by:

u(x + h ) − u(x)
u ′ (x) ≈
h
• The backward difference formula is given by:

u(x) − u(x − h )
u ′ (x) ≈
h
• The central difference formula is given by:

u(x + h ) − u(x − h )
u ′ (x) ≈
2h

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
100 / 153
 Solving a Boundary Value Problem using Forward
Difference

Consider the second-order boundary value problem:

d 2y
= −2y, y(0) = 1, y(1) = 0
dx 2
Discretize the problem using the forward difference method:

d 2 y y(x + h ) − 2y(x) + y(x − h )

≈
dx 2 h2
Rearranging the terms:

y(x + h ) = 2y(x) − y(x − h ) − 2h 2 y(x)

Given the boundary conditions y(0) = 1 and y(1) = 0, solve the system
of equations iteratively to find the solution.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
101 / 153
 2
Given equation: ddx y2 = −2y, with boundary conditions y(0) = 1 and
y(1) = 0.
To solve this differential equation using the forward finite difference
method with a step size of h=0.5, we need to discretize the second
derivative using the following formula:

d 2 y yi +1 − 2yi + yi −1
≈
dx 2 h2
Substitute this approximation into the original differential equation,
we get
Step 1:

y0 = 1
y1 = 2(1 − 0.52 ) · 1 − 0 = 1.75

Step 2:

y1 = 1.75
y = 2(1 − 0.52 ) · 1.75 − 1 = 1
Advanced Computational Methods, AKU Lecture
2 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
102 / 153
 Example
Consider the ODE:
d 2y dy
2
−3 + 2y = 0
dx dx
with boundary conditions:

y(0) = 1, y ′ (1) = 0

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
103 / 153
 Finite Difference Method

• To solve this ODE using the finite difference method, we discretize

the domain and approximate the derivatives.
• Let xi = i Éx where Éx is the step size.
• The second derivative can be approximated as:

d 2 y yi −1 − 2yi + yi +1
≈
dx 2 (Éx)2

The first derivative can be approximated as:

dy yi +1 − yi −1
≈
dx 2Éx

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
104 / 153
 Finite Difference Method Cont.

• Substitute the finite difference approximations into the ODE:

yi −1 − 2yi + yi +1 y −y
2
− 3 i +1 i −1 + 2yi = 0
(Éx) 2Éx
• Rearranging terms gives:

yi −1 − (2 + 3Éx)yi + yi +1 = −2Éx 2 yi

• This leads to a system of equations that can be solved using

matrix methods.
• After solving the system of equations, we obtain the values of yi
at each grid point.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
105 / 153
 Solution of PDE using Finite Difference Method

• Consider the following one-dimensional heat conduction

equation:
u 2 u
=Ó 2 (26)
t x
subject to the initial condition u(x, 0) = f (x) and boundary
conditions u(0, t) = u(L , t) = 0, where Ó is the thermal diffusivity, L
is the length of the domain, and f (x) is the initial temperature
distribution.
• Let’s solve this equation using the finite difference method.
• We discretize the spatial domain into N points with spacing Éx
and the time domain into M points with spacing Ét.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
106 / 153
 • The finite difference approximation for the second derivative is
given by:
2 u ui +1 − 2ui + ui −1
≈ (27)
x 2 (Éx)2
where ui = u(xi , tn ).
• By substituting the finite difference approximation into the heat
conduction equation, we obtain the following update formula:
ÓÉt n
uin+1 = uin + (u − 2uin + uin−1 ) (28)
(Éx)2 i +1
• Now, let’s consider an example where f (x) = sin(áx), Ó = 1, L = 1,
N = 10, M = 100, Éx = 0.1, and Ét = 0.01.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
107 / 153
 Solving Poisson’s Equation using Finite Element
Method

Consider the 1D Poisson’s equation:

d 2u
− = f (x) in (0, 1)
dx 2
with Dirichlet boundary conditions:

u(0) = 0, u(1) = 1

We will use the finite element method to solve this equation.

• Discretize the domain: Divide the interval (0, 1) into N
sub-intervals.
• Approximate the solution: Use piecewise linear basis functions.
• Assemble the system of equations: Formulate the finite element
matrix equation.
• Computational
Advanced Solve forMethods,
the AKU
unknowns: Use
Lecture 1 Solutions
Basics of numerical
of Numerical
Nonlinear Equations methods
Methods and to Equations
System of Linear solve the
108 / 153
 Finite Element Formulation

The finite element formulation for the Poisson’s equation can be

written as:
Z 1 Z 1
du dv
dx = fv dx
0 dx dx 0
where u is the approximate solution and v is a test function.
Using the Galerkin method, we can rewrite the equation as:
Z 1 Z 1
du dv
dx = fv dx
0 dx dx 0

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
109 / 153
 Examples

Example
Consider the 1D heat equation:

u 2 u
=k 2 (29)
t x
Discretizing this equation using finite difference methods, we get:

uin+1 − uin u n − 2uin + uin−1

= k i +1 (30)
Ét (Éx)2

where uin is the value of u at grid point i and time step n. This
equation can be solved iteratively to approximate the solution of the
heat equation.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
110 / 153
 Example
Consider the 2D Poisson equation:

∇2 u = f (x, y) (31)

Discretizing this equation using finite difference methods, we get:

1
ui ,j = (u + ui −1,j + ui ,j +1 + ui ,j −1 − Éx 2 fi ,j ) (32)
4 i +1,j
where ui ,j is the value of u at grid point (i , j ) and fi ,j is the value of the
forcing function at that point. This equation can be solved iteratively
to approximate the solution of the Poisson equation.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
111 / 153
 linear and transient conduction

Steady-State Conduction
• We discuss how to solve linear and transient conduction
problems using mathematical techniques.
• We focus on one-dimensional heat conduction in a solid material,
where the temperature distribution varies with both time and
position.
• Let’s start by considering the steady-state conduction equation in
one dimension:
d 2T
=0 (33)
dx 2
where T (x) is the temperature distribution along the material and
x is the position coordinate.
• The solution to this equation can be found by integrating twice
and applying boundary conditions.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
112 / 153
 Example: Heat Conduction in a Rod

Example
Consider a rod of length L with a constant heat source at one end and
insulated at the other end. The boundary conditions are:

T (0) = T0 (34)
dT
| =q −k (35)
dx x=L
where T0 is the constant temperature at x = 0, k is the thermal
conductivity, and q is the heat flux at x = L .
By solving the steady-state conduction equation and applying the
boundary conditions, we can find the temperature distribution in the
rod.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
113 / 153
 Transient Conduction

Next, let’s consider transient conduction, where the temperature

distribution varies with time as well as position.
The transient conduction equation in one dimension is given by:

T 2 T
=k 2 âc (36)
t x
where â is the density, c is the specific heat, and k is the thermal
conductivity.
This equation describes how the temperature changes over time due
to conduction in the material.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
114 / 153
 Example: Cooling of a Hot Plate

Example
Consider a hot plate initially at a temperature T0 and suddenly cooled
to a lower temperature at time t = 0. The initial condition is:

T (x, 0) = T0 (37)
By solving the transient conduction equation with the initial condition,
we can find the temperature distribution in the hot plate as it cools
over time.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
115 / 153
 Finite Element and Finite Volume Methods

Finite Element Method (FEM)

• The finite element method is a numerical technique for finding
approximate solutions to boundary value problems for partial
differential equations.
• It breaks down the domain of the problem into smaller, simpler
subdomains called elements.
• The solutions in each element are approximated by piecewise
polynomial functions.
• The method requires the formulation of a system of algebraic
equations, which can be solved using methods such as matrix
inversion or iterative techniques.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
116 / 153
 Finite Volume Method (FVM)
• The finite volume method is also a numerical technique for
solving partial differential equations.
• It divides the domain into control volumes and conservatively
calculates the fluxes of the variable of interest across the
boundaries of each control volume.
• The method ensures that the integral form of the conservation
law is satisfied in each control volume.
• FVM is particularly suitable for solving problems involving
complex geometries and irregular mesh structures.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
117 / 153
 Example: Finite Element Method

Example
Consider the one-dimensional heat conduction equation:

T 2 T
=Ó 2 (38)
t x
with initial condition T (x, 0) = T0 and boundary conditions T (0, t) = T1
and T (L , t) = T2 .
Apply the finite element method using linear shape functions to
discretize the domain 0 ≤ x ≤ L into N elements.
N
¼
Th (x, t) = Ni (x)Ti (t) (39)
i =1

where Ni (x) are the linear shape functions.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
118 / 153
 Example: Finite Volume Method

Example
Consider the one-dimensional advection-diffusion equation:

â (âu) 2 â
+ =È 2 (40)
t x x
where â is the scalar quantity, u is the velocity field, and È is the
diffusion coefficient.
Apply the finite volume method to discretize the domain 0 ≤ x ≤ L into
N control volumes.

d âi F −F
= − i +1/2 i −1/2 (41)
dt Éx
where Fi +1/2 is the flux at the interface between control volumes i and
i + 1.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
119 / 153
 Example: Finite Element Method

Example
Consider the steady-state heat conduction equation in one dimension:

d 2T
−k = Q (x) (42)
dx 2
with boundary conditions T (0) = T1 and T (L ) = T2 .
Using the finite element method with linear shape functions, derive
the weak form of the equation and solve for the temperature
distribution.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
120 / 153
 Example: Finite Volume Method

Example
Consider the one-dimensional Burger’s equation:

u u 2 u
+u =ß 2 (43)
t x x
where u is the velocity field and ß is the kinematic viscosity.
Apply the finite volume method to discretize the domain 0 ≤ x ≤ L into
N control volumes and solve the Burger’s equation.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
121 / 153
 Example: Finite Element Method

Example
Consider the wave equation in one dimension:

2 u 2
2 u
= c (44)
t 2 x 2
with initial conditions u(x, 0) = f (x) and u
t
(x, 0) = g(x).
Using the finite element method with linear shape functions, derive
the weak form of the equation and solve for the displacement function
u(x, t).

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
122 / 153
 Solution of Diffusion and Convective Form of PDEs

Diffusion Equation
: The general form of the diffusion equation in one dimension is given
by:
æ 2 æ
=D 2 (45)
t x
with initial condition æ(x, 0) = æ0 and boundary conditions
æ æ
x
x = 0 = 0 and x x = L = 0.
The solution to this equation can be found using techniques like finite
difference method or finite element method.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
123 / 153
 Convective Form of PDE
:
The convective form of a PDE is given by:

â
+ ∇ · (âu) = 0 (46)
t
where â is the density and u is the velocity field.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
124 / 153
 Example: Diffusion Equation

Example
Consider the one-dimensional diffusion equation:

æ 2 æ
=D 2 (47)
t x
with initial condition æ(x, 0) = 0 and boundary conditions æ(0, t) = A
and æ(L , t) = 0.
Apply the finite difference method to discretize the domain 0 ≤ x ≤ L
into N grid points and solve for the concentration profile æ(x, t).

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
125 / 153
 Example: Convective Form of PDE

Consider the 2D convective diffusion equation:

2 ⠁2 â
!
â (âu) (âv)
+ + =D + (48)
t x y x 2 y 2

with initial condition â(x, y, 0) = â0 and boundary conditions

â(0, y, t) = 0, â(L , y, t) = 0, â(x, 0, t) = 0, and â(x, W , t) = 0.
Apply the finite volume method to discretize the domain 0 ≤ x ≤ L ,
0 ≤ y ≤ W into control volumes and solve for the density field â(x, y, t).

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
126 / 153
 Example: Finite Element Method for Diffusion Equation

Example
Consider the one-dimensional diffusion equation:

æ 2 æ
=D 2 (49)
t x
with initial condition æ(x, 0) = æ0 and boundary conditions æ(0, t) = A
and æ(L , t) = 0.
Apply the finite element method using linear shape functions to solve
for the concentration profile æ(x, t).

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
127 / 153
 Example: Finite Volume Method for Convective Form of
PDE

Example
Consider the 2D convective diffusion equation:

2 ⠁2 â
!
â (âu) (âv)
+ + =D + (50)
t x y x 2 y 2
with initial condition â(x, y, 0) = â0 and boundary conditions
â(0, y, t) = 0, â(L , y, t) = 0, â(x, 0, t) = 0, and â(x, W , t) = 0.
Apply the finite volume method to discretize the domain 0 ≤ x ≤ L ,
0 ≤ y ≤ W into control volumes and solve for the density field â(x, y, t).

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
128 / 153
 Example: Finite Difference Method for Diffusion
Equation

Example
Consider the one-dimensional diffusion equation:

æ 2 æ
=D 2 (51)
t x
with initial condition æ(x, 0) = 0 and boundary conditions æ(0, t) = A
and æ(L , t) = 0.
Apply the finite difference method to discretize the domain 0 ≤ x ≤ L
into N grid points and solve for the concentration profile æ(x, t).

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
129 / 153
 Introduction to Optimization

• Optimization is the process of finding the best solution for a given

problem, often under certain constraints.
• It is an essential tool in various fields such as engineering,
economics, and computer science.
• Optimization methods aim to minimize or maximize a function,
known as the objective function, by adjusting the input variables.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
130 / 153
 Linear Programming

• Linear programming is a method used to achieve the best

outcome in a mathematical model with linear relationships.
• It involves optimizing a linear objective function subject to linear
equality and inequality constraints.
• Example:

Maximize Z = 3x + 4y
Subject to
2x + y ≤ 10
x + 2y ≤ 8
x ≥ 0, y ≥ 0

• Here, x and y are decision variables, and Z is the objective

function.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
131 / 153
 Non-linear Programming

• Non-linear programming involves optimizing a non-linear

objective function subject to non-linear constraints.
• This is more complex compared to linear programming and often
requires specialized techniques for optimization.
• Example:

Maximize Z = x 2 + 2y 2
Subject to
x 2 + y 2 ≤ 12
x ≥ 0, y ≥ 0

• Here, x and y are decision variables, and Z is the non-linear

objective function.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
132 / 153
 Single Variable Optimization Problem

• Single-variable optimization is a fundamental concept in

mathematics and plays a crucial role in various fields such as
economics, engineering, and physics.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
133 / 153
 Single Variable Optimization Problem

• Single-variable optimization is a fundamental concept in

mathematics and plays a crucial role in various fields such as
economics, engineering, and physics.
• Consider a function f (x) that depends on a single variable x.
• The optimization problem is to find the value of x that maximizes
or minimizes the function f (x).
• The general form of an optimization problem can be written as:

Maximize f (x)
Subject to g(x) = 0
h (x) ≤ 0

where g(x) and h (x) are constraints on the variable x.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
133 / 153
 Derivatives and Optimization

• One of the key concepts in single-variable optimization is the use

of derivatives.
• The derivative of a function f (x) at a point x = a gives us
information about the slope of the function at that point.
• To find the optimal solution for an optimization problem, we can
use the first and second derivatives of the function f (x).

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
134 / 153
 Methods for Optimization

• There are different methods for solving optimization problems,

some of which include:
• Analytical methods: using calculus and derivatives to find critical
points and determine the optimal solution.
• Numerical methods: using algorithms such as gradient descent or
Newton’s method to iteratively find the optimal solution.
• The choice of method depends on the complexity of the function
and constraints in the optimization problem.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
135 / 153
 Example
Consider the function f (x) = x 2 − 2x + 1. We want to find the minimum
value of f (x).

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
136 / 153
 Example
Consider the function f (x) = x 2 − 2x + 1. We want to find the minimum
value of f (x).
To find the critical points, we first compute the derivative of f :

d 2
f ′ (x) = (x − 2x + 1)
dx
= 2x − 2

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
136 / 153
 Example
Consider the function f (x) = x 2 − 2x + 1. We want to find the minimum
value of f (x).
To find the critical points, we first compute the derivative of f :

d 2
f ′ (x) = (x − 2x + 1)
dx
= 2x − 2

Next, we set f ′ (x) = 0 and solve for x:

2x − 2 = 0
2x = 2
x =1

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
136 / 153
 Example
Consider the function f (x) = x 2 − 2x + 1. We want to find the minimum
value of f (x).
To find the critical points, we first compute the derivative of f :

d 2
f ′ (x) = (x − 2x + 1)
dx
= 2x − 2

Next, we set f ′ (x) = 0 and solve for x:

2x − 2 = 0
2x = 2
x =1

Therefore, the critical point is x = 1.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
136 / 153
 Multivariate optimization
• Multivariate optimization refers to the process of optimizing a
function with multiple variables.
• In simple terms, it involves finding the maximum or minimum
value of a function that depends on two or more variables.
• In multivariate optimization, the goal is to find the values of the
variables that result in the optimal (maximum or minimum) value
of the function.
• This process often involves techniques such as finding the
gradient of the function, setting it to zero to find critical points,
and using the Hessian matrix to determine whether these critical
points are maxima, minima, or saddle points.
• Multivariate optimization has applications in various fields such
as machine learning, economics, engineering, and statistics,
where the goal is to optimize complex functions with multiple
variables to achieve the best possible outcome.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
137 / 153
 General Form of Multivariate Optimization Problem
• The general form of a multivariate optimization problem can be
expressed as follows:
• Given a function f (x1 , x2 , . . . , xn ) with n variables, the goal is to find
the values of the variables x1 , x2 , . . . , xn that maximize or minimize
the function.
• Mathematically, this can be formulated as:

minimize f (x1 , x2 , . . . , xn )
subject to gi (x1 , x2 , . . . , xn ) ≤ 0, for i = 1, 2, . . . , m
and hj (x1 , x2 , . . . , xn ) = 0, for j = 1, 2, . . . , p

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
138 / 153
 • where:
• f (x1 , x2 , . . . , xn ) is the objective function that needs to be optimized.
• gi (x1 , x2 , . . . , xn ) are inequality constraints that the variables must
satisfy.
• hj (x1 , x2 , . . . , xn ) are equality constraints that the variables must
satisfy.
• The variables x1 , x2 , . . . , xn are often subject to certain bounds or
constraints, and the optimization problem seeks to find the
values of these variables that satisfy all constraints while
optimizing the objective function.
• Solving multivariate optimization problems involves techniques
such as gradient descent, Lagrange multipliers, Newton’s
method, and other optimization algorithms to find the optimal
solution that minimizes or maximizes the objective function while
satisfying all constraints.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
139 / 153
 Problem Statement
Find the minimum of the function f (x1 , x2 ) = 3x12 + 4x22 − 2x1 x2 + x1 − 2x2 .

Solution
The first step in solving this problem is to find the gradient of the
function: !
6x1 − 2x2 + 1
∇f =
8x2 − 2x1 − 2
Setting the gradient to zero, we get the system of equations:

6x1 − 2x2 + 1 = 0
8x2 − 2x1 − 2 = 0

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
140 / 153
 3 1
Solving this system, we find the critical point to be (x1 , x2 ) = (− 10 , 5 ).
We can then use the Hessian matrix to determine if this point is a
minimum, maximum, or saddle point:
!
6 −2
H=
−2 8

The eigenvalues of the Hessian matrix are 4 and 10, both positive,
indicating that the critical point is a minimum.
3 1 9
Therefore, the minimum of the function is f (− 10 , 5 ) = − 10 .

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
141 / 153
 Example 1
Suppose a mining company wants to optimize the production of a
mineral using two factors: the amount of labor and the amount of
equipment used.
The production output P can be modeled as a function of labor L and
equipment E :
P = 2L + 3E
The company wants to maximize production while keeping the total
cost of labor and equipment below $500.
This can be formulated as the following optimization problem:

Maximize P = 2L + 3E
Subject to 2L + 3E ≤ 500

Solving this optimization problem will help the company maximize

production within their budget constraints.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
142 / 153
 Example 2
Consider a mining operation that wants to optimize the extraction
process for two minerals: gold and silver. The extraction rates for gold
(G ) and silver (S ) are given by the following equations:

G = 0.5L + 0.3E
S = 0.2L + 0.5E

The company wants to maximize the total value of extracted gold and
silver, given constraints on the labor and equipment costs. This leads
to the following optimization problem:

Maximize V = 10G + 5S
Subject to 0.5L + 0.3E ≤ 100
0.2L + 0.5E ≤ 80

Solving this optimization problem will help the mining operation

maximize the total value of extracted minerals within their cost
constraints.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
143 / 153
 Example 3
In open-pit mining, the dimensions of the pit can affect the cost of
extraction and the amount of material that can be removed.
Let the length (L ), width (W ), and depth (D ) of the pit be the factors
that influence these parameters. The cost of extraction (C ) and the
volume of material removed (V ) can be modeled as:

C = 100L + 150W + 200D

V = 3LW + D

The mining company wants to minimize the cost of extraction while

maximizing the volume of material removed. This leads to the
following optimization problem:

Minimize C = 100L + 150W + 200D

Subject to 3LW + D ≤ 5000

Solving this optimization problem will help the company optimize the
dimensions of the pit to achieve the desired cost and volume targets.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
144 / 153
 Gradient-based Methods

Gradient-based methods are iterative algorithms that use the

gradient of the objective function to find the optimal solution.
Examples include:
• Gradient descent
• Newton’s method
• Conjugate gradient method

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
145 / 153
 Evolutionary Algorithms

Evolutionary algorithms are stochastic optimization methods inspired

by the process of natural selection. Examples include:
• Genetic algorithms
• Particle swarm optimization
• Ant colony optimization

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
146 / 153
 Example: Mining Optimization

Example
Consider a mining operation that is trying to maximize the extraction
of a mineral resource from a mine while minimizing operating costs.
Let x be the amount of the mineral extracted per day and y be the cost
of operating the mine per day. The objective is to maximize the profit,
given by the equation:

Maximize Z = 1000x − 500y

Subject to 2x + 3y ≤ 10
5x + 2y ≤ 8
x ≥ 0, y ≥ 0

Here, x represents the mineral extraction rate, y represents the

operating cost, and the constraints represent the physical and
financial limitations of the mining operation.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
147 / 153
 Example: Mine Production Schedule Optimization

Example
In mining operations, it is essential to optimize the production
schedule to maximize output while minimizing costs. Let x represent
the production rate of a mine and y represent the total operating cost
of the mine. The objective is to maximize the revenue generated from
the mine’s production, given by the equation:

Maximize Z = 2000x − 1000y

Subject to 3x + 2y ≤ 15
2x + 4y ≤ 10
x ≥ 0, y ≥ 0

Here, the constraints represent the production capacity and financial

constraints of the mining operation.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
148 / 153
 Example: Mine Planning and Design Optimization

Example
In mine planning and design, engineers need to optimize the layout of
the mine to maximize resource extraction and minimize excavation
costs. Let x represent the layout design parameters and y represent
the total excavation costs. The objective is to minimize the total
excavation costs, given by the equation:

Minimize Z = 5000x + 2000y

Subject to
2x + 3y ≤ 12
4x + 2y ≤ 8
x ≥ 0, y ≥ 0

Here, the constraints represent the geometric and cost constraints of

the mine planning and design process.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
149 / 153
 Simulation Techniques

Simulation techniques involve using computer programs to model the

behavior of complex systems over time. This allows for the evaluation
of different scenarios, decision-making under uncertainty, and
optimization of system performance. Some common simulation
techniques include:
• Monte Carlo simulation
• Discrete-event simulation
• Agent-based simulation
• System dynamics
These techniques are used to model and analyze systems in a wide
range of fields, including manufacturing, healthcare, finance, and
transportation.

Advanced Computational Methods, AKU Lecture 1 Solutions

Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
150 / 153
 Summary
• Consider a manufacturing company trying to optimize its
production process.
• The goal is to minimize production costs while meeting demand.
Using optimization techniques, the company can model different
production scenarios, evaluate the cost and output of each
scenario, and identify the optimal production plan.
• By formulating the problem as an optimization model and
applying optimization techniques such as linear programming or
evolutionary algorithms, the company can improve efficiency,
reduce costs, and better meet customer demand.
• Simulation and optimization techniques are powerful tools for
modeling, analyzing, and optimizing complex systems.
• By using these techniques, decision-makers and engineers can
make informed decisions, improve performance, and achieve
efficient solutions in a wide range of applications.
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
151 / 153
 % Objective function coefficients
f = [-1000, 500];

% Inequality constraint matrix

A = [2, 3; 5, 2];
b = [10; 8];

% Lower bounds
lb = [0; 0];

% Solve the linear programming problem

[x, fval] = linprog(f, A, b, [], [], lb);

disp(’Optimal mineral extraction rate (x):’);

disp(x(1));

disp(’Optimal operating cost (y):’);

disp(x(2));
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
152 / 153
 % Objective function
fun = @(x) 100*(x(2) - x(1)^2)^2 + (1 - x(1))^2;

% Initial guess
x0 = [0,0];

% Nonlinear constraints
nonlcon = @(x)deal([], x(1)^2 + x(2)^2 - 1);

% Bounds
lb = [-1, -1];
ub = [1, 1];

% Solve the nonlinear programming problem

[x,fval] = fmincon(fun, x0, [], [], [], [], lb, ub, nonlcon);

disp(’Optimal x1:’);
disp(x(1));
Advanced Computational Methods, AKU Lecture 1 Solutions
Basics of
of Numerical
Nonlinear Equations
Methods and System of Linear Equations
153 / 153