Numerical Methods

Iterative Methods for Systems of Linear

Equations

Topic: Jacobi Method

Introduction
• If systems of linear equations are very large, the
computational effort of direct methods is
prohibitively expensive
• Three common classical iterative techniques for
linear systems
• The Jacobi method
• Gauss-Seidel method
• Successive Over Relaxation (SOR) method
• Matlab’s built-in functions
Introduction
• For systems that have coefficient matrices with the
appropriate structure – especially large, sparse systems
(many coefficients whose value is zero) – iterative
techniques may be preferable
 4 1 0 0 1 0 0 0 0 0 0 0 0 0 0   u1  0.08 
0
 1 4  1 0 0 1 0 0 0 0 0 0 0 0 0   u  0.16 
0
   2  
 0 1 4 1 0 0 1 0 0 0 0 0 0 0 0   u3  0.36 
0
     
0 0 1 4 0 0 0 1 0 0 0 0 0 0 0   u 4  1.64 
0
 1 0 0 0 4 1 0 0 1 0 0 0 0 0 0 0   u5  0.16 
     
 0 1 0 0 1 4 1 0 0 1 0 0 0 0 0 0   u 6   0. 0 
0 0 1 0 0 1 4 1 0 0 1 0 0 0 0 0   u 7   0. 0 
     
0 0 0 1 0 0 1 4 0 0 0 1 0 0 0 0   u8   1.0 
     
0 0 0 0 1 0 0 0 4 1 0 0 1 0 0 0   u9  0.36 
0 0 0 0 0 1 0 0 1 4 1 0 0 1 0 0  u10   0 
     
0 0 0 0 0 0 1 0 0 1 4 1 0 0  1 0  u11   0 
0 0 0 0 0 0 0 1 0 0 1 4 0 0 0  1 u12   1.0 
     
0 0 0 0 0 0 0 0 1 0 0 0 4 1 0 0  u13  1.64 
0 0 0 0 0 0 0 0 0 1 0 0  1 4  1 0  u14   1.0 
   
0 0 0 0 0 0 0 0 0 0 1 0 0  1 4  1 u15   1.0 
     
0 0 0 0 0 0 0 0 0 0 0 1 0 0  1 4  u16   2.0 
Iterative solution procedure
• Write the system Ax=b in an equivalent form
x=Ex+f (like x=g(x) for fixed-point iteration)
• Starting with x0, generate a sequence of
approximations {xk} iteratively by
xk+1=Exk+f
• Representation of E and f depends on the type
of the method used
• But for every method E and f are obtained from
A and b, but in a different way
Convergence
• As k, the sequence {xk} converges to the
solution vector under some conditions on E
matrix
• This imposes different conditions on A matrix for
different methods
• For the same A matrix, one method may
converge while the other may diverge
• Therefore for each method the relation between
A and E should be found to decide on the
convergence
Basic Idea on Jacobi method
Convert the system: Ax  B
into the equivalent system: x  Cx  d
a12 a13 b1
x1   x2  x3 
a11 a11 a11
a11x1  a12 x2  a13 x3  b1
a21 a23 b2
a21x1  a22 x1  a23 x3  b2 x2   x1  x3 
a31x1  a32 x1  a33 x3  b3 a22 a22 a22
a31 a b3
x3   x1  32 x2 
a33 a33 a33
• Generate a sequence of approximation
x (1) , x ( 2 ) ,... x ( k )  Cx ( k 1)  d
Jacobi iteration method
a11 x1  a12 x2    a1n xn  b1  x10 
a21 x1  a22 x2    a2 n xn  b2  0
 x2 
x 
0
 
 0
an1 x1  an 2 x2    ann xn  bn  xn 

1  i 1 n 
x1 
1 1
(b1  a12 x2    a1n xn )
0 0 k 1
xi  bi 
aii 
 aij x   aij x 
k
j
k
j
a11 j 1 j  i 1 
1
x2 
1
(b2  a21 x10  a23 x30    a2 n xn0 )
a22
1
x1n  (bn  an1 x10  an 2 x20    ann1 xn01 )
ann
xk+1=Exk+f iteration for Jacobi method

A can be written as A=L+D+U (not decomposition)

 a11 a12 a13   0 0 0 a11 0 0  0 a12 a13 
a a a   a 0 0  0 a 0   0 0 a 
 21 22 23   21   22   23 

a31 a32 a33  a31 a32 0  0 0 a33  0 0 0 

Ax=b  (L+D+U)x=b Dxk+1 =-(L+U)xk+b

1  i 1 n  xk+1=-D-1(L+U)xk+D-1b
bi   aij x j   aij x j 
k 1
xi  k k

aii  j 1 j i 1  E=-D-1(L+U)
 
Dxk+1 Lx k Uxk f=D-1b
Jacobi Method with examples
• Consider the two-by-two system x
1
y3
2x  y  6 2
x  2y  6 1
• Start with x (1)  y (1)  1/ 2 y   x3
2

• Simultaneous updating
• New values of the variables are
not used until a new iteration
step is begun

y ( 2)
1 (1) 1 11
x ( 2)   y 3   3 
2 4 4
y (1)
1 (1) 1 11
y ( 2)   x 3   3 
2 4 4 x (1) x ( 2)
Jacobi Method
• Con’t

1 ( 2) 11 13
x (3)
  y 3  3
2 8 8
1 11 13
y ( 3)   x ( 2)  3    3 
2 8 8
Jacobi Method
• Consider the three-by-three system
2 x1  x2  x3  1
x1  2 x2  x3  6
x1  x2  2 x3  3
x1  0.5 x2  0.5 x3  0.5
x2  0.5 x1  0.5 x3  3.0
x3  0.5 x1  0.5 x2  1.5

 x1( k )   0.0 0.5  0.5  x1( k 1)   0.5

 (k )     x ( k 1)    3.0 
 2  
x   0.5 0.0 0.5   2   
 x ( k )   0.5 0.5 0.0   x ( k 1)    1.5 
 3     3   

• Start with x (0)  (0,0,0)

Jacobi Method
y
• Discussion
x  2y  6 x  2 y  6
2x  y  6 y  2 x  6

y ( 2)
• A necessary and sufficient
condition for the
convergence of the Jacobi y (1)
method x (1) x ( 2) x
• The magnitude of the largest
eigenvalue of the iteration
matrix C be less than 1
Jacobi method : Example 1
Consider a circuit shown in figure here; currents i1, i2, and i3 are given by

9i1  0i2 - 5i3  10 4Ω 8Ω

0i1  20i2 - 12i3  - 2 2V
i2
- 5i1  12i2  20i3  0 i1
10 V 12Ω
The matrix form is:
5Ω
9 0  5  i1   10  i3
0 20  12  i    2
  2    3Ω
 5  12 20  i3   0 
Notice that magnitude of any diagonal element is greater than the sum of
the magnitudes of other elements in that row

A matrix with this property is said to be Diagonally dominant.

Jacobi method : Example 1
The set of equations:
9i1  0i2 - 5i3  10
0i1  20i2 - 12i3  - 2
- 5i1  12i2  20i3  0
Let us write for i1, i2 and i3 as

i1  10  5i3 /9  1.1111  0.5556i2 (1)

i2  - 2  12i3 /20  - 0.1000  0.6000i2 ( 2)
i3  5i1  12i3 /20  0.2500i1  0.6000i2 (3)

Let us make an initial guess as i1 = 0.0; i2 =0.0 and i3 = 0.0

First iteration results: i1 = 1.1111; i2 =-0.1000 and i3 = 0.0

Jacobi method : Example 1
i1  10  5i3 /9  1.1111  0.5556i2 (1)
i2  - 2  12i3 /20  - 0.1000  0.6000i2 ( 2)
i3  5i1  12i3 /20  0.2500i1  0.6000i2 (3)

First iteration results: i1 = 1.1111; i2 =-0.1000 and i3 = 0.0

2nd iteration results: i1 = 1.1111; i2 =-0.1000 and i3 = 0.22

3rd iteration results: i1 = 1.23; i2 = 0.03 and i3 = 0.22

4th iteration results: i1 = 1.23 ; i2 = 0.03 and i3 = 0.33

5th iteration results: i1 = 1.29; i2 = 0.1 and i3 = 0.33

6th iteration results: i1 = 1.29; i2 = 0.1 and i3 = 0.38
Jacobi method : Example 2
Consider the following set of equations.
10 x1  x2  2 x3 6
 x1  11x2  x3  3x4  25
2 x1  x2  10 x3  x4  11
3 x2  x3  8 x4  15
Convert the set Ax = b in the form of x = Tx + c.
1 1 3
x1  x2  x3 
10 5 5
1 1 3 25
x2  x1  x3  x4 
11 11 11 11
1 1 1 11
x3   x1  x2  x4 
5 10 10 10
3 1 15
x4   x2  x3 
8 8 8
Jacobi method : Example 2
1 (0) 1 ( 0) 3
  
(1)
x1 x2 x3
10 5 5
1 (0) 1 (0) 3 ( 0) 25
   
(1)
x2 x1 x3 x4
11 11 11 11
1 (0) 1 ( 0) 1 (0) 11
  x1   
(1)
x3 x2 x4
5 10 10 10
3 (0) 1 ( 0) 15
  x2  
(1)
x4 x3
8 8 8

 0, x2  0, x3  0 and x4  0.
( 0) ( 0) ( 0) ( 0)
x1
1 1 3
  
(1)
x1 (0) (0)
 0.6000,
10 5 5 (1)
1 1 3 25 x1
   
(1)
x2 (0) (0) (0)
 2.2727,
11 11 11 11 (1)
1 1 1 11 x2
  (0)   
(1)
x3 (0) (0)
5 10 10 10
 1.1000
(1)
3 1 15 x3
  (0)  
(1)
x4 (0)
8 8 8
 1.8750
(1)
x4
Jacobi method : Example 2
1 (1) 1 (1) 3
  
(2)
x1 x2 x3
10 5 5
1 (1) 1 (1) 3 (1) 25
  x3  
( 2)
x2 x1 x4
11 11 11 11
1 (1) 1 (1) 1 (1) 11
  x1   
( 2)
x3 x2 x4
5 10 10 10
3 (1) 1 (1) 15
  x2  
( 2)
x4 x3
8 8 8

1 ( k 1) 1 ( k 1) 3
x1   x3 
(k)
x2
10 5 5
1 ( k 1) 1 ( k 1) 3 ( k 1) 25
  x3  
(k )
x2 x1 x4
11 11 11 11
1 ( k 1) 1 ( k 1) 1 ( k 1) 11
  x1   x4 
(k )
x3 x2
5 10 10 10
3 ( k 1) 1 ( k 1) 15
  x2  x3 
(k )
x4
8 8 8
Jacobi method : Example 2
Results:
iteration 0 1 2 3

(k ) 0.0000 0.6000 1.0473 0.9326

x1
(k ) 0.0000 2.2727 1.7159 2.0530
x2
(k ) 0.0000 -1.1000 -0.8052 -1.0493
x3
(k ) 0.0000 1.8750 0.8852 1.1309
x4
Jacobi method : Example 3
A diverging case study:
 2 1 5  x1   15  0 
 4  8 1  x    21 x 0  0
  2    b  Ax0  26.7395
2
 4  1 1  x3   7  0

The matrix is not diagonally dominant

 15  x20  5 x30  15
x 
1
1
  7.5
2 2 b  Ax1  54.8546
21  4 x10  x30 21 2
x2 
1
  2.625
8 8
x31  7  4 x10  x20  7.0
Jacobi method : Example 3
 15  2.625  5  7
x 
1
1  11.3125
2
21  4  7.5  7
x12   0.25
8 b  Ax2  208.3761
2
x31  7  4  7.5  2.625  39.625

The residual term is increasing at each iteration,

so the iterations are diverging.
Note that the matrix is not diagonally dominant
Pseudo-Code For Jacobi Method
1) build A, b
2) build modified A with diagonal zero  Q
3) set initial guess x=0

4) do { 1  jN

a) compute: xi   bi   Qij x j 
A ii  j 1 
max  xi  xi 
i 1,.., N
b) compute error: err 
max  bi 
i 1,.., N

xi  xi
c) update x:
}while err>tol
The rules
We set up the Jacobi iteration but did not ask the question “when
will the Jacobi iteration converge and how fast?”

Definition: N

A matrix A is diagonally dominant if a

j 1, j  i
ij  aii

Theorem:
If the matrix A is diagonally dominant then Ax=b has a unique
solution x and the Jacobi iteration produces a sequence which
converges to x for any initial guess

Informally:
The “more diagonally dominant ” a matrix is the faster it will
converge… this holds some of the time.
 Numerical Methods

The Jacobi and Gauss-

Siedel Iterative
Techniques
The Jacobi and Gauss-Siedel Iterative Techniques
In this section we describe the
Jacobi and the Gauss-Seidel
iterative methods

We want to solve the following  10 -1 2 0   x1   6 

linear system  x   25 
 -1 11 -1 3  2    
 2 -1 10 -1   x3    11
  
𝐴𝑥 = 𝑏  0 3 -1 8   4 
x

 15


𝑨 𝒙 𝒃
Example
solve the linear system

10 x1  x2  2 x3 6
 x1  11x2  x3  3x4  25
2 x1  x2  10 x3  x4  11
3 x2  x3  8 x4  15
The Jacobi and Gauss-Siedel Iterative Techniques

We want to solve the following linear system

Example
Keep the diagonal on the left hand side
solve the linear system

10 x1  x2  2 x3 6 10 x1  x2  2 x3 6
 x1  11x2  x3  3x4  25 11x2  x1  x3  3x4  25
2 x1  x2  10 x3  x4  11 10 x3  2 x1  x2  x4  11
3 x2  x3  8 x4  15  8 x4   3x2  x3  15
Jacobi Iterative Method
From the initial approx. 𝒙(𝟎) = (𝟎, 𝟎, 𝟎, 𝟎)𝑻 we 𝒙(𝟏) Change the coeff of LHS to be one by division

x1(1)  ( x2( 0 )  2 x3( 0 )  6) / 10 x1  ( x2  2 x3  6) / 10

x2  ( x1  x3  3x4  25) / 11
x (1)
( x ( 0)
x ( 0)
 3x ( 0)
 25) / 11
2 1 3 4
x3  (2 x1  x2  x4  11) / 10
x (1)
 ( 2 x
( 0)
 x ( 0)
 x ( 0)
 11) / 10
3 1 2 4
x4  (  3x2  x3  15) /( 8)
x (1)
4 (  3x ( 0)
2 x( 0)
3  15) /( 8)
The Jacobi and Gauss-Siedel Iterative Techniques

From the initial approx. 𝒙(𝟎) = (𝟎, 𝟎, 𝟎, 𝟎)𝑻 we 𝒙(𝟏)

x1(1)  ( x2( 0 )  2 x3( 0 )  6) / 10  0  6 / 10   0.600

 0  25 / 11   2.272
x2(1)  ( x1( 0 )  x3( 0 )  3x4( 0 )  25) / 11 x ( 0)   x (1)    
 0  11 / 10   1.100 
x3(1)  ( 2 x1( 0 )  x2( 0 )  x4( 0 )  11) / 10       1.875 
 0   15 / 8   
x4(1)  (  3x2( 0 )  x3( 0 )  15) /( 8)

Jacobi Iterative Method

From the k-th approx. 𝒙(𝒌) = (𝟎, 𝟎, 𝟎, 𝟎)𝑻 we 𝒙(𝒌+𝟏)

x1( k 1)  ( x2( k )  2 x3( k )  6) / 10  0.600  1.047 

 2.272  1.715
( k 1) x (1)   x(2)  
x 2 ( x (k )
1 x (k )
3  3x (k )
4  25) / 11   1.100   0.805
   
x3( k 1)  ( 2 x1( k )  x2( k )  x4( k )  11) / 10   1.875   0.885
x4( k 1)  (  3x2( k )  x3( k )  15) /( 8)
The Jacobi and Gauss-Siedel Iterative Techniques
Jacobi Iterative Method

 0
x1( k 1)  ( x2( k )  2 x3( k )  6) / 10  0
x ( 0)  
x2( k 1)  ( x1( k )  x3( k )  3x4( k )  25) / 11  0
 
x3( k 1)  ( 2 x1( k )  x2( k )  x4( k )  11) / 10  0
x4( k 1)  (  3x2( k )  x3( k )  15) /( 8)

𝒌=𝟏 𝒌=𝟐 𝒌=𝟑 𝒌=𝟒 𝒌=𝟓

(𝒌)
𝒙𝟏
0.6000 1.0473 0.9326 1.0152 0.9890
(𝒌)
𝒙𝟐 2.2727 1.7159 2.0533 1.9537 2.0114
(𝒌)
𝒙𝟑 -1.1000 -0.8052 -1.0493 -0.9681 -1.0103
(𝒌)
𝒙𝟒 1.8750 0.8852 1.1309 0.9738 1.0214

𝒌=6 𝒌=7 𝒌=8 𝒌=9 𝒌 = 10

(𝒌)  1
𝒙𝟏 1.0032 0.9981 1.0006 0.9997 1.0001  2
x  
(𝒌)
𝒙𝟐 1.9922 2.0023 1.9987 2.0004 1.9998 *
(𝒌)
𝒙𝟑 -0.9945 -1.0020 -0.9990 -1.0004 -0.9998  1
 
(𝒌)
𝒙𝟒 0.9944 1.0036 0.9989 1.0006 0.9998  1
The Jacobi and Gauss-Siedel Iterative Techniques
Gauss Seidel Method
From the k-th approx. 𝒙(𝒌) = (𝟎, 𝟎, 𝟎, 𝟎)𝑻 we 𝒙(𝒌+𝟏)
x1( k 1)  ( x2( k )  2 x3( k )  6) / 10
Note that in the Jacobi iteration one
does not use the most recently x2( k 1)  ( x1( k )  x3( k )  3x4( k )  25) / 11
available information.
x3( k 1)  ( 2 x1( k )  x2( k )  x4( k )  11) / 10
x4( k 1)  (  3x2( k )  x3( k )  15) /( 8)

From the k-th approx. 𝒙(𝒌) = (𝟎, 𝟎, 𝟎, 𝟎)𝑻 we 𝒙(𝒌+𝟏)

x1( k 1)  ( x2( k )  2 x3( k )  6) / 10

x2( k 1)  ( x1( k 1)  x3( k )  3x4( k )  25) / 11
x3( k 1)  ( 2 x1( k 1)  x2( k 1)  x4( k )  11) / 10
x4( k 1)  (  3x2( k 1)  x3( k 1)  15) /( 8)
𝒌=𝟏 𝒌=𝟐 𝒌=𝟑 𝒌=𝟒 𝒌=𝟓
(𝒌)
𝒙𝟏
Note that Jacobi’s method in this 0.6000 1.0302 1.0066 1.0009 1.0001
(𝒌)
𝒙𝟐
example required twice as many 2.3273 2.0369 2.0036 2.0003 2.0000
(𝒌)
iterations for the same accuracy. 𝒙𝟑 -0.9873 -1.0145 -1.0025 -1.0003 -1.0000
(𝒌)
𝒙𝟒 0.8789 0.9843 0.9984 0.9998 1.0000
The Jacobi and Gauss-Siedel Iterative Techniques

 a11  a1n   x1   b1 
Jacobi iteration for general n:           
    
for i  1 : n an1  ann   xn  bn 
 i 1 n 
( k 1)
x   bi   aij x j   aij x j 
 (k ) (k )
i a
 j 1 j i 1  ii
end

Gauss-Seidel iteration for general n:

for i  1 : n
 i 1 n 
( k 1)
x   bi   aij x j   aij x j 
 ( k 1) (k )
i a
 j 1 j i 1  ii
end
The Jacobi and Gauss-Siedel Iterative Techniques

We want to solve the following linear system Dx  ( L  U ) x  b

Example
Keep the diagonal on the left hand side
solve the linear system

10 x1  x2  2 x3 6 10 x1  x2  2 x3 6
 x1  11x2  x3  3x4  25 11x2  x1  x3  3x4  25
2 x1  x2  10 x3  x4  11 10 x3  2 x1  x2  x4  11
3 x2  x3  8 x4  15  8 x4   3x2  x3  15

( D  ( L  U )) x  b Change the coeff of LHS to be one by division

Jacobi Iterative Method

x1  ( x2  2 x3  6) / 10
x2  ( x1  x3  3x4  25) / 11
1 1 x3  (2 x1  x2  x4  11) / 10
x  D (L  U )x  D b
x4  (  3x2  x3  15) /( 8)
The Jacobi and Gauss-Siedel Iterative Techniques
Convergence Criterion for the Gauss-Seidel Method
Definition

A is strictly diagonally dominant That is, the diagonal coefficient in

𝑛 each of the equations must be
larger than the sum of the
𝑎𝑖𝑖 > ෍ 𝑎𝑖𝑗 , 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑖
absolute values of the other
𝑗=1
𝑗≠𝑖
coefficients in the equation

Example1 Example2
A A
10 -1 2 0 10 -1 2 0
-1 11 -1 3 -1 4 -1 3
2 -1 10 -1 2 -1 4 -1
0 3 -1 8 0 3 -1 2

THM:
A is strictly diagonally dominant GS Convg
The Jacobi and Gauss-Siedel Iterative Techniques

Definition Example1

The L2 norms for the vector 𝑥 are defined by 3

𝑥= 2
𝑥1 −1
𝑥 = 𝑥12 + 𝑥22 + ⋯ + 𝑥𝑛2 𝑥= ⋮
𝑥𝑛 𝑥 = 14

norm of the vector x = (x1, x2, x3)t gives the length of the
straight line joining the points (0, 0, 0) and (x1, x2, x3).

the distance between two vectors is defined as

the norm of the difference of the vectors
The Jacobi and Gauss-Siedel Iterative Techniques