Iterative methods for linear system of equations

1. Gauss-Seidel Method
2. Jacobi Method
Gauss-Seidel Method

An iterative method.

Basic Procedure:
-Algebraically solve each linear equation for xi
-Assume an initial guess solution array
-Solve for each xi and repeat
-Use absolute relative approximate error after each iteration
to check if error is within a pre-specified tolerance.
Gauss-Seidel Method

Why?
The Gauss-Seidel Method allows the user to control round-off error.

Elimination methods such as Gaussian Elimination and LU

Decomposition are prone to round-off error.

Also: If the physics of the problem are understood, a close initial

guess can be made, decreasing the number of iterations needed.
 Gauss-Seidel Method

Algorithm
A set of n equations and n unknowns:
If: the diagonal elements are
a11x1 + a12 x2 + a13 x3 + ... + a1n xn = b1 non-zero
a21x1 + a22 x2 + a23 x3 + ... + a2n xn = b2 Rewrite each equation solving
. .
. . for the corresponding unknown
. .
ex:
an1x1 + an 2 x2 + an3 x3 + ... + ann xn = bn
First equation, solve for x1
Second equation, solve for x2
 Gauss-Seidel Method

Algorithm
Rewriting each equation
c1 − a12 x2 − a13 x3  − a1n xn
x1 = From Equation 1
a11

c2 − a21 x1 − a23 x3  − a2 n xn
x2 = From equation 2
a22
  
c − a x − a x  − an −1,n −2 xn −2 − an −1,n xn From equation n-1
xn −1 = n −1 n −1,1 1 n −1, 2 2
an−1,n −1
cn − an1 x1 − an 2 x2 −  − an,n −1 xn −1
xn = From equation n
ann
Gauss-Seidel Method

Algorithm
General Form of each equation
n

a
n
c1 −  a1 j x j cn −1 − n −1, j xj
j =1 j =1
j  n −1
x1 =
j 1
xn −1 =
a11 an −1,n −1
n
n
c2 −  a2 j x j c n −  a nj x j
j =1 j =1
j n
x2 =
j 2
xn =
a 22 a nn
Gauss-Seidel Method

Algorithm
General Form for any row ‘i’
n
ci −  aij x j
j =1
j i
xi = , i = 1,2,, n.
aii

How or where can this equation be used?

Stopping Criteria for Iterative Methods

Calculate the Absolute Relative Approximate Error

x new
−x old
a i = i
new
i
100
x i

So when has the answer been found?

The iterations are stopped when the absolute relative

approximate error is less than a prespecified tolerance for all
unknowns.
Convergence of Iterative Methods

One class of system of equations always converges: One with a diagonally

dominant coefficient matrix.

Diagonally dominant: [A] in [A] [X] = [C] is diagonally dominant if:

n
aii   aij
n
aii   aij for all ‘i’ and for at least one ‘i’
j =1 j =1
j i j i
 Convergence of Iterative Methods

Diagonally dominant: The coefficient on the diagonal must be at least

equal to the sum of the other coefficients in that row and at least one row
with a diagonal coefficient greater than the sum of the other coefficients
in that row.
Which coefficient matrix is diagonally dominant?

 2 5.81 34 124 34 56 

A =  45 43 1  [B] =  23 53 5 
123 16 1   96 34 129

Most physical systems do result in simultaneous linear equations that

have diagonally dominant coefficient matrices.
Gauss-Seidel Method: Example

Given the system of equations The coefficient matrix is:

12x1 + 3x2- 5x3 = 1 12 3 − 5

x1 + 5x2 + 3x3 = 28 A =  1 5 3 
3x1 + 7 x2 + 13x3 = 76  3 7 13 

With an initial guess of Will the solution converge using the

Gauss-Siedel method?
 x1  1
 x  = 0 
 2  
 x3  1
 Gauss-Seidel Method: Example

Checking if the coefficient matrix is diagonally dominant

a11 = 12 = 12  a12 + a13 = 3 + − 5 = 8
12 3 − 5
A =  1 5 3  a 22 = 5 = 5  a 21 + a 23 = 1 + 3 = 4
 3 7 13 
a33 = 13 = 13  a31 + a32 = 3 + 7 = 10

The inequalities are all true and at least one row is strictly greater than:
Therefore: The solution should converge using the Gauss-Siedel Method
Gauss-Seidel Method: Example

Rewriting each equation With an initial guess of

12 3 − 5  a1   1   x1  1
 1 5 3  a  = 28  x  = 0 
   2    2  
 3 7 13  a3  76  x3  1

1 − 3 x 2 + 5 x3 1 − 3(0) + 5(1)
x1 = x1 = = 0.50000
12 12

28 − x1 − 3x3 28 − (0.5) − 3(1)

x2 = x2 = = 4.9000
5 5

76 − 3x1 − 7 x 2 76 − 3(0.50000) − 7(4.9000)

x3 = x3 = = 3.0923
13 13
 Gauss-Seidel Method: Example

The absolute relative approximate error

0.50000 − 1.0000
a 1 = 100 = 100.00%
0.50000

4.9000 − 0
a 2
= 100 = 100.00%
4.9000

3.0923 − 1.0000
a 3
= 100 = 67.662%
3.0923

The maximum absolute relative error after the first iteration is 100%
Gauss-Seidel Method: Example

After Iteration #1
 x1  0.5000
 x  = 4.9000
 2  
 x3  3.0923

Substituting the x values into the After Iteration #2

equations
1 − 3(4.9000) + 5(3.0923)
x1 =
12
= 0.14679 𝑥1
28 − (0.14679) − 3(3.0923)
𝑥2
x2 = = 3.7153
5 𝑥3
76 − 3 0.14679 − 7 3.0923
𝑥3 = = 4.1419
13 0.14679
= 3.7153
Gauss-Seidel Method: Example

Iteration #2 absolute relative approximate error

0.14679 − 0.50000
a 1 = 100 = 240.61%
0.14679
3.7153 − 4.9000
a 2
= 100 = 31.889%
3.7153
3.8118 − 3.0923
a 3
= 100 = 18.874%
3.8118
The maximum absolute relative error after the first iteration is 240.61%

This is much larger than the maximum absolute relative error obtained in
iteration #1. Is this a problem?
 Gauss-Seidel Method: Example

Repeating more iterations, the following values are obtained

Iteration a1 a 1 % a2 a 2 % a3 a 3 %

1 0.50000 100.00 4.9000 100.00 3.0923 67.662

2 0.14679 240.61 3.7153 31.889 3.8118 18.876
3 0.74275 80.236 3.1644 17.408 3.9708 4.0042
4 0.94675 21.546 3.0281 4.4996 3.9971 0.65772
5 0.99177 4.5391 3.0034 0.82499 4.0001 0.074383
6 0.99919 0.74307 3.0001 0.10856 4.0001 0.00101

 x1  0.99919  x1  1 
The solution obtained  x  =  3.0001  is close to the exact solution of  x  =  3 .
 2    2  
 x3   4.0001   x3  4
Gauss-Seidel Method

The Gauss-Seidel Method can still be used

 3 7 13 
The coefficient matrix is not
diagonally dominant
A =  1 5 3 
12 3 − 5
But this is the same set of 12 3 − 5
equations used in example which
did converge.
A =  1 5 3 
 3 7 13 

If a system of linear equations is not diagonally dominant, check to see if

rearranging the equations can form a diagonally dominant matrix.
Gauss-Seidel Method

Not every system of equations can be rearranged to have a

diagonally dominant coefficient matrix.
Observe the set of equations

x1 + x2 + x3 = 3
2x1 + 3x2 + 4x3 = 9
x1 + 7 x2 + x3 = 9

Which equation(s) prevents this set of equation from having a

diagonally dominant coefficient matrix?
 Iterative Formula for Gauss-Seidel Method

(𝑘−1) (𝑘−1) (𝑘−1)

(𝑘) 𝑐1 − 𝑎12 𝑥2 − 𝑎13 𝑥3 … … − 𝑎1𝑛 𝑥𝑛
𝑥1 = From Equation 1
𝑎11

𝑘 𝑘−1 (𝑘−1)
(𝑘) c2 − 𝑎21 𝑥1 − 𝑎23 𝑥3 … … − 𝑎2𝑛 𝑥𝑛 From equation 2
𝑥2 =
𝑎22
⋮ ⋮ ⋮
(𝑘) (𝑘) (𝑘) (𝑘−1)
(𝑘) 𝑐𝑛−1 − 𝑎𝑛−1,1 𝑥1 − 𝑎𝑛−1,2 𝑥2 … … − 𝑎𝑛−1,𝑛−2 𝑥𝑛−2 − 𝑎𝑛−1,𝑛 𝑥𝑛 From equation n-1
𝑥𝑛−1 =
𝑎𝑛−1𝑛−1

(𝑘) (𝑘) (𝑘)

(𝑘) 𝑐𝑛 − 𝑎𝑛1 𝑥1 − 𝑎𝑛2 𝑥2 − ⋯ … − 𝑎𝑛,𝑛−1 𝑥𝑛−1
𝑥𝑛 = From equation n
𝑎𝑛𝑛
Gauss-Seidel Method

Solve for the unknowns

Assume an initial guess for [X] Use rewritten equations to solve for
each value of xi.
Important: Remember to use the
 x1  most recent value of xi. Which
x  means to apply values calculated to
 2 the calculations remaining in the
current iteration.
  
 
 xn -1 
 xn 
 Gauss-Seidel Iterative Technique

Consider the following set of equations.

10x1 − x2 + 2 x3 =6
− x1 + 11x2 − x3 + 3x4 = 25
2 x1 − x2 + 10x3 − x4 = −11
3x2 − x3 + 8 x4 = 15
 1 ( k −1 ) 1 ( k −1 ) 3
= − x3 +
(k)
x1 x2
10 5 5
1 ( k −1 ) 1 ( k −1 ) 3 ( k −1 ) 25
= + x3 − x4 +
(k )
x2 x1
11 11 11 11
1 ( k −1 ) 1 ( k −1 ) 1 ( k −1 ) 11
= − x1 + x2 + x4 −
(k )
x3
5 10 10 10
3 ( k −1 ) 1 ( k −1 ) 15
= − x2 + x3 +
(k )
x4
8 8 8
 1 ( k −1) 1 ( k −1) 3
= − x3 +
(k)
x1 x2
10 5 5
1 (k) 1 ( k −1) 3 ( k −1) 25
= + x3 − x4 +
(k)
x2 x1
11 11 11 11
1 (k) 1 (k) 1 ( k −1) 11
= − x1 + + −
(k)
x3 x2 x4
5 10 10 10
3 (k) 1 (k) 15
= − x2 + x3 +
(k)
x4
8 8 8
Results of Gauss-Seidel Iteration:
(Blue numbers are for Jacobi iterations.)
k 0 1 2 3
(k ) 0.0000 0.6000 1.0300 1.0065
x1
0.6000 1.0473 0.9326
(k ) 0.0000 2.3272 2.0370 2.0036
x2 2.2727 1.7159 2.0530
(k ) 0.0000 -0.9873 -1.0140 -1.0025
x3
-1.1000 -0.8052 -1.0493
(k ) 0.0000 0.8789 0.9844 0.9983
x4 1.8750 0.8852 1.1309
The solution is: x1= 1, x2 = 2, x3 = -1,
x4 = 1
It required 15 iterations for Jacobi
method and 7 iterations for Gauss-
Seidel method to arrive at the solution
with a tolerance of 0.00001.
 Iterative formula for Jacobi Method

(𝑘−1) (𝑘−1) (𝑘−1)

(𝑘) 𝑐1 − 𝑎12 𝑥2 − 𝑎13 𝑥3 … … − 𝑎1𝑛 𝑥𝑛
𝑥1 = From Equation 1
𝑎11

𝑘−1 𝑘−1 (𝑘−1)

(𝑘) c2 − 𝑎21 𝑥1− 𝑎23 𝑥3 … … − 𝑎2𝑛 𝑥𝑛 From equation 2
𝑥2 =
𝑎22
⋮ ⋮ ⋮
(𝑘−1) (𝑘−1) (𝑘−1) (𝑘−1)
(𝑘) 𝑐𝑛−1 − 𝑎𝑛−1,1 𝑥1 − 𝑎𝑛−1,2 𝑥2 … … − 𝑎𝑛−1,𝑛−2 𝑥𝑛−2 − 𝑎𝑛−1,𝑛 𝑥𝑛 From equation n-1
𝑥𝑛−1 =
𝑎𝑛−1𝑛−1

(𝑘−1) (𝑘−1) (𝑘−1)

(𝑘) 𝑐𝑛 − 𝑎𝑛1 𝑥1 − 𝑎𝑛2 𝑥2 − ⋯ … − 𝑎𝑛,𝑛−1 𝑥𝑛−1
𝑥𝑛 = From equation n
𝑎𝑛𝑛
 Jacobi Iterative Technique

Consider the following set of equations.

10x1 − x2 + 2 x3 =6
− x1 + 11x2 − x3 + 3x4 = 25
2 x1 − x2 + 10x3 − x4 = −11
3x2 − x3 + 8 x4 = 15
Convert the set Ax = b in the form of x = Tx + c.

1 1 3
x1 = x2 − x3 +
10 5 5
1 1 3 25
x2 = x1 + x3 − x4 +
11 11 11 11
1 1 1 11
x3 = − x1 + x2 + x4 −
5 10 10 10
3 1 15
x4 = − x2 + x3 +
8 8 8
Start with an initial approximation of:
= 0, x 2 = 0, x3 = 0 and x4 = 0.
(0) (0) (0) (0)
x1

1 (0) 1 (0) 3
= − x3 +
(1)
x1 x2
10 5 5
1 (0) 1 (0) 3 (0) 25
= + x3 − x4 +
(1)
x2 x1
11 11 11 11
1 (0) 1 (0) 1 (0) 11
= − x1 + + −
(1)
x3 x2 x4
5 10 10 10
3 (0) 1 (0) 15
= − x2 + x3 +
(1)
x4
8 8 8
 1 1 3
= − (0) +
(1)
x1 (0)
10 5 5
1 1 3 25
= + (0) − (0) +
( 1)
x2 (0)
11 11 11 11
1 1 1 11
= − (0) + (0) + (0) −
( 1)
x3
5 10 10 10
3 1 15
= − (0) + (0) +
(1)
x4
8 8 8

= 0.6000, x2 = 2.2727,
(1) (1)
x1

= −1.1000 and x4 = 1.8750

(1) (1)
x3
 (2)
1 (1) 1 (1) 3
𝑥1 = 𝑥 − 𝑥3 +
10 2 5 5
(2)
1 (1) 1 (1) 3 (1) 25
𝑥2 = 𝑥 + 𝑥3 − 𝑥4 +
11 1 11 11 11
1 1 (1) 1 11
𝑥3 (2) = − 𝑥1 (1) + 𝑥 + 𝑥4 (1) −
5 10 2 10 10
3 1 15
𝑥4 (2) = − 𝑥2 (1) + 𝑥3 (1) +
8 8 8
 1 ( k −1) 1 ( k −1) 3
x1 = − x3 +
(k)
x2
10 5 5
1 ( k −1) 1 ( k −1) 3 ( k −1) 25
= + x3 − x4 +
(k)
x2 x1
11 11 11 11
1 ( k −1) 1 ( k −1) 1 ( k −1) 11
= − x1 + + −
(k)
x3 x2 x4
5 10 10 10
3 ( k −1) 1 ( k −1) 15
= − x2 + x3 +
(k)
x4
8 8 8
Results of Jacobi Iteration:

k 0 1 2 3
(k ) 0.0000 0.6000 1.0473 0.9326
x1
(k ) 0.0000 2.2727 1.7159 2.0530
x2
(k ) 0.0000 -1.1000 -0.8052 -1.0493
x3
(k ) 0.0000 1.8750 0.8852 1.1309
x4
Use gauss-Seidel method to solve the following system. Assume the error
tolerance as 0.0001.
15𝑥1 − 2𝑥2 − 6𝑥3 = 300

−2𝑥1 + 12𝑥2 − 4𝑥3 − 𝑥4 = 0

−6𝑥1 − 4𝑥2 + 19𝑥3 − 9𝑥4 = 0

−𝑥2 − 9𝑥3 − 21𝑥4 = 0

1
𝑥1 = [300 + 2𝑥2 + 6𝑥3 ]
15
1
𝑥2 = [2𝑥1 + 4𝑥3 + 𝑥4 ]
12
1
𝑥3 = [6𝑥1 + 4𝑥2 + 9𝑥4 ]
19
−1
𝑥4 = [𝑥2 + 9𝑥3 ]
21
Iterative formula ( for k=1,2,3…..)

(𝑘) 1 (𝑘−1) (𝑘−1)

𝑥1 = [300 + 2𝑥2 + 6𝑥3 ]
15
(𝑘) 1 (𝑘) (𝑘−1) (𝑘−1)
𝑥2 = [2𝑥1 + 4𝑥3 + 𝑥4 ]
12
(𝑘) 1 (𝑘) (𝑘) (𝑘−1)
𝑥3 = [6𝑥1 + 4𝑥2 + 9𝑥4 ]
19
(𝑘) 1 (𝑘) (𝑘)
𝑥4 = [𝑥2 + 9𝑥3 ]
21

Assume initial approximation as

𝑋 (0) = 0, 0, 0, 0 T

(0) (0) (0) (0)

𝑥1 = 0, 𝑥2 = 0, 𝑥3 = 0, 𝑥4 = 0
 (1)
𝑥1 = 20
(1) 1(1) (0) (0)
𝑥2 = 2𝑥1 + 4𝑥3 + 𝑥4
12
(1) 1
𝑥2 = 2 20 + 0 + 0 = 3.3333
12
(1) 1 (1) (1) (0)
𝑥3 = 6𝑥1 + 4𝑥2 + 9𝑥4
19
(1) 1
𝑥3 = 6(20) + 4(3.3333) + 9(0) = 7.0175
19
(1) 1 (1) (1)
𝑥4 = 𝑥2 + 9𝑥3
21
(1) 1
𝑥4 = 3.3333 + 9(7.0175) = 3.1662
21
𝑋 (1) = 20, 3.3333, 7.0175, 3.1662