Chapter 11

Iterative Methods for Linear and

Nonlinear System of Equations

“These notes are only to be used in class presentations”

1
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
 Linear Algebraic Equations
• A linear equation in n variables is
a1x1+a2x2+ … +anxn = b

•A linear algebraic system is a system of equations where

all the functions are linear

2
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
In matrix form;
[ A]{ X }  {B}
where A (coefficient matrix) is a n x n matrix, and X and B
are n x 1 vectors.

3
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
 Matrix definitions

4
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
• Transpose

aijT  a ji

•Determinant
n
D  det A   (1)i  j aij M ij for any i=1, 2 ...n
j 1

Minor Mij is the determinant of the (n-1)x(n-1) submatrix of A

obtained by deleting the ith row and jth column of the matrix A.

•Inverse
If matrix A is nonsingular (det≠0) and square, A-1 is the inverse such
that
1
A A I
5
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
 Matrix operations

For addition and subtraction A

and B should have the same
dimensions.

Cnxl = AnxmBmxl

AB  BA

6
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
 If you need to work more than one linear equations, a system
of linear equations must be solved simultaneously.

We need to solve for X

A1 AX  A1B
X  A1B

How do we get A-1?

– It is non-trivial
– Not very efficient

• Usually use other methods to solve for X

– Elimination Methods
– Iterative Methods 7
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
 Iterative Methods

• Iterative methods give approximate results for linear

algebraic systems
AX  B
and provide an alternative to the elimination methods
• They are efficient for large systems with a high percentage of
0 entries (A is sparse) in terms of both computer storage and
computation.
•As any other iterative methods, they need initial guesses and
they may not converge or converge slowly.

8
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
 Jacobi Iterative Method

To solve n linear equations;

1. Choose an initial approximation x (0)
A simple way to obtain initial guesses is to assume that
they are zero.
2. Convert the system AX=B into an equivalent system of the
form
X  TX  C
It consists of solving ith equation AX=B for xi to obtain (aii≠0)
n 
1  
xi   
aii j 1
 a x
ij j  bi i  1,2  n
 j i 
9
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
3. Generate sequence of approximate solution vectors by
computing
( k 1) (k )
X  TX C k: iteration number

n 
1  
xi ( k 1)   
aii j 1
( aij x j )  bi 
(k )
i  1,2  n
 j i 
10
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
 4. Terminate the procedure when

a   s
or max. number of iterations is reached.

How to define εa
We have vectors so we’ll use norms to to define εa

X ( k 1)  X ( k )
 a ( k 1) 
X ( k 1)

A norm is a real-valued function that provides a measure of the size or

length of multi-component mathematical entities such as vectors or
matrices. Norm of the vector gives a measure for the distance between an
arbitrary vector and zero vector.
11
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
 x  x1 x2  xn T
Maximum magnitude norm

x   max xi
1i n
p norm
1/ p

 n
p

x p   xi 

i 1 

n
p=1 x 1   xi
i 1
1/ 2
 2
n
p=2 x 2   xi  Euclidean norm
i 1  12
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
The distance between two vectors is defined as the norm of the
difference of the vectors.

y   y1 y2  yn  T

x  y   max xi  yi
1i n

1/ 2
 n
2
x  y 2   ( xi  yi ) 
i 1 

13
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Example 11.1
Solve the linear system with Jacobi iterative method
4 x1  x2  x3  7
4 x1  8 x2  x3  21 x ( 0)
 1 2 2 T

 2 x1  x2  5 x3  15

Carry out four iterations.

iter. no x1 x2 x3
0 1 2 2
1 1.75 3.375 3
2 1.8438 3.875 3.025
3 1.9625 3.925 2.963
4 1.9906 3.9766 3

14
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
 Gauss-Seidel Method
The Gauss-Seidel method is a commonly used iterative
method. It is same as Jacobi technique except that it uses the
latest values of x’s.

1  i 1 n 
  ij j )  
( k 1) ( k 1)
xi   ( a x (aij x (jk ) )  bi  i  1,2n
aii  j 1 j i 1 
15
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
Example 11.2
Solve the linear system with Gauss-Seidel iterative method
5 x1  2 x2  3x3  1
 3x1  9 x2  x3  2
x ( 0)  0 0 0
T

2 x1  x2  7 x3  3
Iterate until εa ≤ 0.02 . Use maximum magnitude norm to
calculate εa .

iter.no x1 x2 x3 εa
0 0 0 0
1 -0.2000 0.1556 -0.5079
2 0.1670 0.3343 -0.4286 0.8562
3 0.1909 0.3335 -0.4217 0.0567
4 0.1864 0.3312 -0.4226 0.0107

16
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
 Convergence criterion

If the coefficient matrix A is Diagonally Dominant Jacobi

iteration and Gauss-Seidel is guaranteed to converge.
For each equation i :
Diagonally Dominant 
n
aii   aij
j 1
j i

For each row, the absolute value of the diagonal element is

greater than the sum of the absolute values of the rest of the
elements.
Note that this is not a necessary condition, i.e. the system may
still have a chance to converge even if A is not diagonally
dominant. 17
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
 Example 11.3

Solve the linear system with Gauss-Seidel iterative method

4 x1  8 x2  x3  21
4 x1  x2  x3  7 x(0)  1 2 2T
 2 x1  x2  5 x3  15
Iterate until εa ≤ 0.001 . Use maximum magnitude norm to
calculate εa . Ensure convergence before starting to iterate.

iter.no x1 x2 x3 εa
0 1 2 2

1 1.75 3.75 2.95

2 1.95 3.9688 2.9863 0.0551
3 1.9956 3.9961 2.9990 0.0114
4 1.9993 3.9995 2.9998 0.0009
18
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
 Improvement of Convergence Using Relaxation

Relaxation represents a slight modification of the Gauss-Seidel

method and is designed to enhance convergence.
After each new value of x is computed , that value is modified
by a weighted average of the results of the previous and the
present iterations

new old new

xi  (1  w) xi  wxi
0 w2
19
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
•For choices of w with 0 < w < 1, the procedures are called
under-relaxation methods
•For choices of w with 1 < w, the procedures are called over-
relaxation methods
These methods are abbreviated SOR, for Successive Over-
Relaxation.
Substitute w in Gauss-Seidel iterative equation

xi ( k 1)  (1  w) xi( k )  wxi(k 1) i  1,2 n

 i 1 n 
bi   aij x (jk 1)   aij x (jk ) 
( k 1) (k ) w
xi  (1  w) xi 
aii     
 j 1 j i 1 
i  1,2  n 20
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
 Example 11.4
Use SOR method with w=1.25 to solve the linear system
4 x1  3x2  24
3x1  4 x2  x3  30
x ( 0)
 1 1 1 T
 x2  4 x3  24
Compute three iterations.
The result is given for 7 iterations and compared with Gauss-Seidel Method.

21
Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display.