LECTURE NOTE ON EIE 419

(NUMERICAL METHODS II)

BY

Engr. Grace Olaleru

 ITERATIVE METHODS FOR SOLVING
LINEAR SYSTEMS AND CONVERGENCE

2
Learning Objectives
By the end of this lecture, students will be able to:
1. Apply iterative methods for solving linear
systems.
2. Understand the convergence criteria and apply
it in engineering problems.
 Iterative methods
• Iterative methods are crucial numerical techniques for
solving large systems of linear equations when direct
methods become computationally expensive. These
methods start with an initial guess and progressively
refine it to approach the actual solution.
• In the context of the learning objectives, the Jacobi and
Gauss-seidel methods will be considered.
JACOBI’s method

5
Assumptions for the Jacobi’s method
Given the system of equations:
𝑎11 𝑥1 + 𝑎12 𝑥2 + 𝑎13 𝑥3 = 𝑏1
𝑎21 𝑥1 + 𝑎22 𝑥2 + 𝑎23 𝑥3 = 𝑏2
𝑎31 𝑥1 + 𝑎32 𝑥2 + 𝑎33 𝑥3 = 𝑏3
𝑎11 𝑎12 𝑎13
𝐴 = 𝑎21 𝑎22 𝑎23
𝑎31 𝑎32 𝑎33
1. The system must have unique solution
2. The coefficient matrix A, should have non-zero diagonal elements,
i.e., 𝑎11 , 𝑎22 , …, 𝑎𝑛𝑛 should be non-zero.
 Convergence requirement for the
Jacobi’s Method
• For convergence of the approximation to the exact solution, A
must be a diagonally dominant matrix. That is,
𝑎11 ≥ 𝑎1𝑧 + 𝑎13
𝑎22 ≥ 𝑎21 + 𝑎23
𝑎33 ≥ 𝑎31 + 𝑎32

• However, if A is not diagonally dominant, the solution converge

slowly or not converge at all.
 Jacobi’s Algorithm
For a system of n equations:
1. Verify if A is a diagonally dominant matrix.
(0) 0 0 0 𝑇
2. Start with an initial guess 𝑥 = [𝑥1 𝑥2 𝑥3 ]
3. For each component 𝑖 = 1,2, … , 𝑛:
𝑘+1 1
𝑥𝑖 = 𝑏𝑖 − ෍ 𝑎𝑖𝑗 ∗ 𝑥𝑗𝑘 𝑗≠i
𝑎𝑖𝑖
4. Repeat until convergence or stopping criterion.
 Jacobi’s Algorithm
• Step 2 for a 3x3 matrix:

For k ≥ 0
1
𝑥1𝑘+1 = 𝑏1 − 𝑎12 𝑥2𝑘 − 𝑎13 𝑥3𝑘
𝑎11
1
𝑥2𝑘+1 = 𝑏2 − 𝑎21 𝑥1𝑘 − 𝑎23 𝑥3𝑘
𝑎22
1
𝑥3𝑘+1 = 𝑏3 − 𝑎31 𝑥1𝑘 − 𝑎32 𝑥2𝑘
𝑎33
 Stopping Criteria
1. When we reach the exact solution
2. When we have exhausted the number of iterations
specified.
3. Absolute error = 𝑥 (𝑘+1) − 𝑥 (𝑘) < 𝜀 or

𝑥 (𝑘+1) −𝑥 (𝑘)
Relative error = < 𝜀,
𝑥 (𝑘)

where 𝜀 = error tolerance given in the question

 Example 1

Solve the system of equations using Jacobi Iteration

Method. Perform only two iterations.
5𝑥1 − 𝑥2 + 2𝑥3 = 12
3𝑥1 + 8𝑥2 − 2𝑥3 = −25
𝑥1 + 𝑥2 + 4𝑥3 = 6
 Solution
1. Verify if A is diagonally dominant

5 −1 2
𝐴= 3 8 −2
1 1 4

5 ≥ −1 + 2 𝑇𝑟𝑢𝑒
8 ≥ 3 + −2 𝑇𝑟𝑢𝑒
4 ≥ 1 + 1 𝑇𝑟𝑢𝑒
 Solution
(0) 0 0 0 𝑇
2. Start with an initial guess 𝑥 = [𝑥1 𝑥2 𝑥3 ] = [0 0 0]𝑇 .
3. For k ≥ 0
1 𝑘 𝑘
𝑥1𝑘+1
= 12 + 𝑥2 − 2𝑥3
5
1
𝑥2 = −25 − 3𝑥1𝑘 + 2𝑥3𝑘
𝑘+1
8
1
𝑥3 = 6 − 𝑥1𝑘 − 𝑥2𝑘
𝑘+1
4
 Solution
• For k = 0
1
𝑥11 = 12 + 0 − 2(0) = 2.4
5
1
𝑥21 = −25 − 3(0) + 2(0) = −3.125
8
1
1
𝑥3 = 6 − 0 − 0 = 1.5
4
 Solution
• 𝑥 (1) = [2.4 − 3.125 1.5]𝑇 .
• For k = 1
1
𝑥12
= 12 − 3.125 − 2(1.5) = 1.175
5
2
1
𝑥2 = −25 − 3(2.4) + 2(1.5) = −3.65
8
2
1
𝑥3 = 6 − 2.4 + 3.125 = 1.68125
4
Stopping criterion is met.
 Example 2

Solve the system of equations using Jacobi Iteration

Method. Stop at convergence.
8𝑥1 + 5𝑥2 + 2𝑥3 = 25
2𝑥1 + 10𝑥2 − 2𝑥3 = 20
𝑥1 + 3𝑥2 + 6𝑥3 = 30
 Solution
1. Verify if A is diagonally dominant

8 5 2
𝐴 = 2 10 −2
1 3 6

8 ≥ 5 + 2 𝑇𝑟𝑢𝑒
10 ≥ 2 + −2 𝑇𝑟𝑢𝑒
6 ≥ 1 + 3 𝑇𝑟𝑢𝑒
 Solution
(0) 0 0 0 𝑇
2. Start with an initial guess 𝑥 = [𝑥1 𝑥2 𝑥3 ] = [0 0 0]𝑇 .
3. For k ≥ 0
1 𝑘 𝑘
𝑥1𝑘+1 = 25 − 5𝑥2 − 2𝑥3
8
𝑘+1
1
𝑥2 = 20 − 2𝑥1𝑘 + 2𝑥3𝑘
10
1
𝑥3 = 30 − 𝑥1𝑘 − 3𝑥2𝑘
𝑘+1
6
 Solution
For k = 0
1
𝑥11= 25 − 5(0) − 2(0) = 3.125
8
1
1
𝑥2 = 20 − 2(0) + 2(0) = 2
10
1
1
𝑥3 = 30 − 0 − 3(0) = 5
6
 Solution
• 𝑥 (1) = [3.125 2 5]𝑇 .
• For k = 1
1
𝑥11
= 25 − 5(2) − 2(5) = 0.625
8
1
1
𝑥2 = 20 − 2(3.125) + 2(5) = 2.375
10
1
1
𝑥3 = 30 − 3.125 − 3(2) = 3.479
6
 Solution
• 𝑥 (2) = [0.625 2.375 3.479]𝑇 .
• For k = 2
Keep solving until convergence
 No of Iterations

1 2 3 4 5 6 7 8

𝑥1 3.125 0.625 0.771 0.591 0.612 0.599 0.601 0.6

𝑥2 2 2.375 2.571 2.587 2.599 2.599 2.6 2.6

𝑥2 5 3.479 3.708 3.586 3.608 3.599 3.601 3.6

 Example 3

Verify the convergence of the system of equations below

using the Jacobi’s method.
𝑥1 + 𝑥2 − 𝑥3 = −2
2𝑥1 − 𝑥2 + 𝑥3 = 5
−𝑥1 + 2𝑥2 + 2𝑥3 = 1
 Solution
1. Verify if A is diagonally dominant
1 1 −1
𝐴 = 2 −1 1
−1 2 2
1 ≥ 1 + −1 𝐹𝑎𝑙𝑠𝑒
−1 ≥ 2 + 1 𝐹𝑎𝑙𝑠𝑒
2 ≥ −1 + 2 𝐹𝑎𝑙𝑠𝑒

The solution may not converge or may converge slowly.

Advantages of Jacobi’s method

• Simple to understand and implement.

• Easily parallelizable as calculations are independent.

• Good for diagonally dominant matrices.

Disadvantages of Jacobi’s method

• Slower convergence compared to Gauss-Seidel.

• Requires more storage for new values.
GAUSS-SEIDEL method

27
 Gauss-Seidel method for solving system
of equations
Improves upon Jacobi by immediately using updated
values as they become available.
 Example 4

Solve the system of equations using Gauss-Seidel Iteration Method.

5𝑥1 − 𝑥2 + 2𝑥3 = 12
3𝑥1 + 8𝑥2 − 2𝑥3 = −25
𝑥1 + 𝑥2 + 4𝑥3 = 6
 Solution
1. Verify if A is diagonally dominant

5 −1 2
𝐴= 3 8 −2
1 1 4

5 ≥ −1 + 2 𝑇𝑟𝑢𝑒
8 ≥ 3 + −2 𝑇𝑟𝑢𝑒
4 ≥ 1 + 1 𝑇𝑟𝑢𝑒
 Solution
(0) 0 0 0 𝑇
2. Start with an initial guess 𝑥 = [𝑥1 𝑥2 𝑥3 ] = [0 0 0]𝑇 .
3. For k ≥ 0
1 𝑘 𝑘
𝑥1𝑘+1
= 12 + 𝑥2 − 2𝑥3
5
1
𝑥2 = −25 − 3𝑥1𝑘+1 + 2𝑥3𝑘
𝑘+1
8
1
𝑥3 = 6 − 𝑥1𝑘+1 − 𝑥2𝑘+1
𝑘+1
4
 Solution
• For k = 0
1
𝑥11 = 12 + 0 − 2(0) = 2.4
5
1
𝑥21 = −25 − 3(2.4) + 2(0) = −4.025
8
1
1
𝑥3 = 6 − 2.4 + 4.025 = 1.906
4
 Solution
• 𝑥 (1) = [2.4 − 4.025 1.906]𝑇 .
• For k = 1
1
𝑥12
= 12 − 4.025 − 2(1.906) = 0.8326
5
2
1
𝑥2 = −25 − 3(0.8326) + 2(1.906) = −2.961
8
2
1
𝑥3 = 6 − 0.8326 + 2.961 = 2.0321
4
 Solution
• 𝑥 (2) = [0.8326 − 2.961 2.0321]𝑇 .
• For k = 2
1
𝑥13 = 12 − 2.961 − 2(2.0321) = 0.995
5
1
𝑥23 = −25 − 3(0.995) + 2(2.0321) = −2.9901
8
3
1
𝑥3 = 6 − 0.995 + 2.9901 = 1.999
4
 Solution
• 𝑥 (3) = [0.995 − 2.9901 1.999]𝑇 .
• For k = 3
1
𝑥14
= 12 − 2.9901 − 2(1.999) = 1.002
5
4
1
𝑥2 = −25 − 3(1.002) + 2(1.999) = −3.001
8
4
1
𝑥3 = 6 − 1.002 + 3.001 = 2.000
4
 Solution
• 𝑥 (4) = [1.002 − 3.001 2]𝑇 .
• For k = 4
1
𝑥15
= 12 − 3.001 − 2(2) = 1.00
5
5
1
𝑥2 = −25 − 3(1.00) + 2(2) = −3.00
8
5
1
𝑥3 = 6 − 1.00 + 3.00 = 2.000
4
Advantages of Gauss-Seidel method

• Faster convergence than Jacobi.

• Less storage required.

• Better numerical stability.

Disadvantages of Gauss-Seidel method
• Less suitable for parallel computing.
• More complex implementation.
Comparative Analysis (Jacobi vs. Gauss-Seidel)
Aspects Jacobi Gauss-Seidel

Convergence Rate Slower Faster

Parallelization Better Limited

Memory Usage More Less

Implementation Simpler More Complex

 ASSIGNMENT

1. Write the MATLAB code for each for the methods

discussed in this lecture note.

2. What is parallel computing?