Numerical Methods

King Saud University

Aims

In this lecture, we will . . .

I Consider the System of Linear Equations
I Introduce the Gaussian elimination method
I Introduce the Jacobi Iterative Method
I Introduce the Gauss-Seidel Iterative Method
Introduction

When engineering systems are modeled, the mathematical description is

frequently developed in terms of set of algebraic simultaneous equations.
Sometimes these equations are non-linear and sometimes linear. Here we discuss
systems of simultaneous linear equations and describe the numerical methods
for the approximate solutions of such systems.
Important Points
I. We will look for the solutions of systems of linear equations.
II. Linear systems may be simultaneous (number of linear equations and
unknowns variables are equal) or underdetermined (number of linear equations
less than unknowns variables) or overdetermined (number of linear equations more
than unknowns variables). Here, we shall discuss only simultaneous systems.
III. Matrix form of linear system is Ax = b, where A called coefficient matrix,
column matrix b is right hand constant and column matrix x be the unknowns.
IV. Linear systems may be nonhomogeneous (right hand vector b 6= 0) or
homogeneous (b = 0).
V. Linear systems may have unique solution or no solution or infinitely many
solutions.
VI. Linear systems may be nonsingular (determinant of coefficients matrix A not
equal to zero) or singular (determinant of coefficients matrix A equal to zero).
Nonsingular systems have unique solution while singular systems have either no
solution or infinitely many solutions.
VII. Solutions of linear systems can be obtained by both direct and indirect
(iterative) methods.
VII. Linear systems may be well-conditioned (small condition number) or
ill-conditioned (large condition number) .
Definition 1
(Linear equation)
It is an equation in which the highest exponent in a variable term is no more than
one. The graph of such equation is a straight line. •
Alinear equation in two variables x1 and x2 is an equation that can be written
in the form

a1 x1 + a2 x2 = b,

where a1 , a2 and b are real numbers. Note that this is the equation of a straight
line in the plane. For example, the equations

4
5x1 + 2x2 = 2, x1 + 2x2 = 1, 2x1 − 4x2 = π,
5

are all linear equations in two variables.

A linear equation in n variables x1 , x2 , . . . , xn is an equation that can be written as

a1 x1 + a2 x2 + · · · + an xn = b,

where a1 , a2 , . . . , an are real numbers and called the coefficients of unknown

variables x1 , x2 , . . . , xn and the real number b, the right-hand side of equation, is
called the constant term of the equation.
Definition 2
(System of Linear Equations)
A system of linear equations (or linear system) is simply a finite set of linear
equations.
For example,
4x1 − 2x2 = 5
3x1 + 2x2 = 4
is the system of two equations in two variables x1 and x2 , while

2x1 + x2 − 5x3 + 2x4 = 9

4x1 + 3x2 + 2x3 + 4x4 = 3
x1 + 2x2 + 3x3 + 2x4 = 11

is the system of three equations in the four variables x1 , x2 , x3 and x4 .

In order to write a general system of m linear equations in the n variables
x1 , . . . , xn , we have

a11 x1 + a12 x2 + ··· + a1n xn = b1

a21 x1 + a22 x2 + ··· + a2n xn = b2
.. .. .. .. (1)
. . ··· . .
am1 x1 + am2 x2 + ··· + amn xn = bm

or, in compact form the system (1) can be written

n
X
aij xj = bi , i = 1, 2, . . . , m. (2)
j=1

For such system we seek all possible ordered sets of numbers c1 , . . . , cn which
satisfies all m equations when they are substituted for the variables x1 , x2 , . . . , xn .
Any such set {c1 , c2 , . . . , cn }, is called a solution of the system of linear equations
(1) or (2).
Theorem 3
(Solution of a Linear System)
Every system of linear equations has either no solution, exactly one solution, or
infinitely many solutions. •
Linear System in Matrix Notation

To write the general simultaneous system of n linear equations in the n unknown

variables x1 , x2 , . . . , xn , is

a11 x1 + a12 x2 + ··· + a1n xn = b1

a21 x1 + a22 x2 + ··· + a2n xn = b2
.. .. .. .. .. (3)
. . . . .
an1 x1 + an2 x2 + ··· + ann xn = bn

The system of linear equations (3) can be written as the single matrix equation

a11 a12 ··· a1n x1 b1

    
 a21 a22 ··· a2n  x2   b2 
.. .. .. .. .. = .. , (4)
    
 
 . . . .  .   . 
an1 an2 ··· ann xn bn
where
a11 a12 ··· a1n x1 b1
     
 a21 a22 ··· a2n   x2   b2 
A= . . . . , x= .. , b= .. ,
     
 .. .. .. ..   .   . 
an1 an2 ··· ann xn bn

the coefficient matrix, the column matrix of unknowns, and the column
matrix of constants, respectively, then the system (3) can be written very
compactly as
Ax = b, (5)
which is called the matrix form of the system of linear equations (3). The column
matrices x and b are called vectors.
Solutions of Linear Systems of Equations

Now we shall discuss numerical methods for solving system of linear equations.
We shall discuss both direct and indirect (iterative) methods for the solution of
given linear systems. In direct method we shall discuss the familiar technique
called the method of elimination to find the solution of linear systems. This
method starts with the augmented matrix of the given linear system and obtain a
matrix of a certain form. This new matrix represents a linear system that has
exactly the same solutions as the given origin system. In indirect methods we shall
discuss Jacobi and Gauss-Seidel methods.
Gaussian Elimination Method
Simple Gaussian Elimination Method

The Gaussian elimination procedure start with forward elimination, in which the
first equation in the linear system is used to eliminate the first variable from the
rest of (n − 1) equations. Then the new second equation is used to elimination
second variable from the rest of (n − 2) equations, and so on. If (n − 1) such
elimination is performed then the resulting system will be the triangular form.
Once this forward elimination is completed, we can determine whether the system
is overdetermined or underdetermined or has a unique solution. If it has a unique
solution, then the backward substitution is used to solve the triangular system
easily and one can find the unknown variables involve in the system.
Now we shall describe the method in detail for a system of n linear equations.
Consider the following a system of n linear equations:

a11 x1 + a12 x2 + a13 x3 + ··· + a1n xn = b1

a21 x1 + a22 x2 + a23 x3 + ··· + a2n xn = b2
a31 x1 + a32 x2 + a33 x3 + ··· + a3n xn = b3
(6)
.. .. .. .. .. ..
. . . . . .
an1 x1 + an2 x2 an3 x3 + ··· + ann xn = bn

Forward Elimination
Consider first equation of the given system (6)

a11 x1 + a12 x2 + a13 x3 + · · · + a1n xn = b1 , (7)

as first pivotal equation with first pivot element a11 . Then the first equation times
multiples mi1 = (ai1 /a11 ), i = 2, 3, . . . , n, is subtracted from the ith equation to
eliminate first variable x1 , producing an equivalent system

a11 x1 + a12 x2 + a13 x3 + ··· + a1n xn = b1

(1) (1) (1) (1)
a22 x2 + a23 x3 + ··· + a2n xn = b2
(1) (1) (1) (1)
a32 x2 + a33 x3 + ··· + a3n xn = b3 (8)
.. .. .. .. ..
. . . . .
(1) (1) (1) (1)
an2 x2 + an3 x3 + ··· + ann xn = bn
Now consider a second equation of the system (8), which is
(1) (1) (1) (1)
a22 x2 + a23 x3 + · · · + a2n xn = b2 , (9)

(1)
as second pivotal equation with second pivot element a22 . Then the second
(1) (1)
equation times multiples mi2 = (ai2 /a22 ), i = 3, . . . , n, is subtracted from the
ith equation to eliminate second variable x2 , producing an equivalent system

a11 x1 + a12 x2 + a13 x3 + ··· + a1n xn = b1

(1) (1) (1) (1)
a22 x2 + a23 x3 + ··· + a2n xn = b2
(2) (2) (2)
a33 x3 + ··· + a3n xn = b3 (10)
.. .. .. ..
. . . .
(2) (2) (2)
an3 x3 + ··· + ann xn = bn
Now consider a third equation of the system (10), which is
(2) (2) (2)
a33 x3 + · · · + a3n xn = b3 , (11)

(2)
as the third pivotal equation with third pivot element a33 . Then the third
(2) (2)
equation times multiples mi3 = (ai3 /a33 ), i = 4, . . . , n, is subtracted from the
ith equation to eliminatethird variable x3 . Similarly, after (n-1)th steps, we have
the nth pivotal equation which have only one unknown variable xn , that is

a11 x1 + a12 x2 + a13 x3 + ··· + a1n xn = b1

(1) (1) (1) (1)
+ a22 x2 + a23 x3 + ··· + a2n xn = b2
(2) (2) (2)
+ a33 x3 + ··· + a3n xn = b3 (12)
.. ..
. .
(n−1) (n−1)
ann xn = bn

(n−1)
with nth pivotal element ann . After getting the upper-triangular system which
is equivalent to the original system, the forward elimination is completed. After
the triangular set of equations has been obtained, the last equation of the system
(12) yields the value of xn directly. The value is then substituted into the
equation next to the last one of the system (12) to obtain a value of xn−1 , which
is, in turn, used along with the value of xn in the second to the last equation to
obtain a value of xn−2 , and so on.
Example 0.1
Solve the following linear system using the simple Gaussian elimination method

x1 + 2x2 + x3 = 2
2x1 + 5x2 + 3x3 = 1
x1 + 3x2 + 4x3 = 5

solution. The process begins with the augmented matrix form

 
..
 1 2 1 . 2 
 
.
 2 5 3 .. 1  .
 
 
 .. 
1 3 4 . 5

Since a11 = 1 6= 0, so we wish to eliminate the elements a21 and a31 by

subtracting from the second and third rows the appropriate multiples of the first
2 1
row. In this case the multiples are m21 = = 2 and m31 = = 1. Hence
1 1
 
.
 1 2 1 .. 2 
 
..
 0 1 1 . −3  .
 
 
 .. 
0 1 3 . 3
 (1) (1)
As a22 = 1 6= 0, therefore, we eliminate entry in a32 position by subtracting the
1
multiple m32 = = 1 of the second row from the third row, to get
1
 
.
 1 2 1 .. 2 
 
.
 0 1 1 .. −3  .
 
 
 . 
0 0 2 . . 6

Obviously, the original set of equations has been transformed to an

upper-triangular form. Since all the diagonal elements of the obtaining
upper-triangular matrix are nonzero, which means that the coefficient matrix of
the given system is nonsingular and therefore, the given system has a unique
solution. Now expressing the set in algebraic form yields

x1 + 2x2 + x3 = 2
x2 + x3 = −3
2x3 = 6

Now using backward substitution, we get

2x3 = 6, gives x3 = 3,
x2 = −x3 − 3 = −(3) − 3 = −6, gives x2 = −6,
x1 = 2 − 2x2 − x3 = 2 − 2(−6) − 3, gives x1 = 11,

which is the required solution of the given system. •

Example 0.2
Solve the following linear system using the simple Gaussian elimination method

x2 + x3 = 1
x1 + 2x2 + 2x3 = 1
2x1 + x2 + 2x3 = 3

Solution. Writing the given system in the augmented matrix form

 
..
 0 1 1 . 1 
 
.
 1 2 2 .. 1  .
 
 
 .. 
2 1 2 . 3

To solve this system, the simple Gaussian elimination method will fail
immediately because the element in the first row on the leading diagonal, the
pivot, is zero. Thus it is impossible to divide that row by the pivot value. Clearly,
this difficulty can be overcome by rearranging the order of the rows; for example
by making the first row the second, gives
 
.
 1 2 2 .. 1 
 
.
 0 1 1 .. 1  .
 
 
 .. 
2 1 2 . 3
Now we use the usual elimination process. The first elimination step is to
eliminate the element a31 = 2 from the third row by subtracting a multiple
2
m31 = = 2 of row 1 from row 3, gives
1
 
..
 1 2 2 . 1 
 
..
.
 
 0 1 1 . 1
 
 .. 
0 −3 −2 . 1
We finished with the first elimination step since the element a21 is already
eliminated from second row. The second elimination step is to eliminate the
(1) −3
element a32 = −3 from the third row by subtracting a multiple m32 = of row
1
2 from row 3, gives  
.
 1 2 2 .. 1 
 
.
 0 1 1 .. 1  .
 
 
 . 
0 0 1 . 4 .
Obviously, the original set of equations has been transformed to an
upper-triangular form. Now expressing the set in algebraic form yields
x1 + 2x2 + 2x3 = 1
x2 + x3 = 1
x3 = 4
Using backward substitution, we get, x1 = −1, x2 = −3, x3 = 4, the solution of
the system. •
Example 0.3
For what values of α the following linear system has (i) Unique solution, (ii) No
solution, (iii) Infinitely many solutions, by using the simple Gaussian elimination
method. Use smallest positive integer value of α to get the unique solution of the
system.

x1 + 3x2 + αx3 = 4
2x1 − x2 + 2αx3 = 1
αx1 + 5x2 + x3 = 6

Solution. Writing the given system in the augmented matrix form

 
1 3 α 4
 2 −1 2α 1  ,
α 5 1 6

and by using the following multiples m21 = 2 and m31 = α, we get

 
1 3 α 4
 0 −7 0 −7  .
0 5 − 3α 1 − α2 6 − 4α
5 − 3α
Now using the multiple m32 = , gives
−7
 
1 3 α 4
 0 −7 0 −7  .
0 0 1 − α2 1−α
So if 1 − α2 6= 0, then we have the unique solution of the given system while for
α = ±1, we have no unique solution. If α = 1, then we have infinitely many
solution because third row of above matrix gives

0x1 + 0x2 + 0x3 = 0,

and when α = −1, we have

0x1 + 0x2 + 0x3 = 2,

which is not possible, so no solution.

Since we can not take α = 1 for the unique solution, so can take next positive
integer α = 2, which gives us upper-triangular system of the form

x1 + 3x2 + 2x3 = 4
− 7x2 = −7
− 3x3 = −1

Solving this system using backward substitution, we get,

x1 = 1/3, x2 = 1, x3 = 1/3, the required unique solution of the given system
using smallest positive integer value of α. •
Theorem 4
An upper-triangular matrix A is nonsingular if and only if all its diagonal
elements are different from zero. •
Example 0.4
Use the simple Gaussian elimination method to find all the values of α which
make the following matrix singular.
 
1 −1 α
A=  2 2 1 .
0 α −1.5

Solution. Applying the forward elimination step of the simple Gaussian

elimination on the given matrix A and eliminate the element a21 by subtracting
from the second row the appropriate multiple of the first row. In this case the
multiple is given as  
1 −1 α
 0 4 1 − 2α  .
0 α −1.5
We finished with the first elimination step. The second elimination step is to
(1) α
eliminate element a32 = α by subtracting a multiple m32 = of row 2 from row
4
3, gives
1 −1 α
 
 0 4 1 − 2α 
.
 α(1 − 2α) 
0 0 −1.5 −
4
To show that the given matrix is singular, we have to set the third diagonal
element equal to zero (by Theorem 4), that is

α(1 − 2α)
−1.5 − = 0, or 2α2 − α − 6 = 0.
4
3
Solving the above quadratic equation, we get, α = − and α = 2, the possible
2
values of α which make the given matrix singular. •
Procedure
[Gaussian Elimination Method]

1. Form the augmented matrix, B = [A|b].

2. Check first pivot element a11 6= 0, then move to the next step; otherwise,
interchange rows so that a11 6= 0.
ai1
3. Multiply row one by multiplier mi1 = and subtract to the ith row for
a11
i = 2, 3, . . . , n.
4. Repeat the steps 2 and 3 for the remaining pivots elements unless coefficient
matrix A becomes upper-triangular matrix U .
bn−1
n
5. Use backward substitution to solve xn from the nth equation xn = and
ann
solve the other (n-1) unknowns variables by using

(n−1)
bn 
xn =


(n−1)

ann






1   

(n−2) (n−2)

xn−1 = bn−1 − an−1n xn 

(n−2)

an−1n−1 (13)
..



.


  


n 
1  X 

x1 = b1 − a1j xj 



a11 j=2


Partial Pivoting

In using the Gaussian elimination by partial pivoting(or row pivoting), the basic
approach is to use the largest (in absolute value) element on or below the diagonal
in the column of current interest as the pivotal element for elimination in the rest
of that column.
One immediate effect of this will be to force all the multiples used to be not
greater than 1 in absolute value. This will inhibit the growth of error in the rest of
elimination phase and in subsequent backward substitution.
At stage k of forward elimination, it is necessary, therefore, to be able to identify
the largest element from |akk |, |ak+1,k |, . . . , |ank |, where these aik ’s are the
elements in the current partially triangularized coefficient matrix. If this
maximum occurs in row p, then pth and kth rows of the augmented matrix are
interchange and the elimination proceed as usual. In solving n linear equations, a
n(n + 1)
total of N = coefficients must be examined.
2
Example 0.5
Solve the following linear system using the Gaussian elimination with partial
pivoting
x1 + x2 + x3 = 1
2x1 + 3x2 + 4x3 = 3
4x1 + 9x2 + 16x3 = 11

Solution. For the first elimination step, since 4 is the largest absolute coefficient
of first variable x1 , therefore, the first row and the third row are interchange,
giving us
4x1 + 9x2 + 16x3 = 11
2x1 + 3x2 + 4x3 = 3
x1 + x2 + x3 = 1
Eliminate first variable x1 from the second and third rows by subtracting the
2 1
multiples m21 = and m31 = of row 1 from row 2 and row 3 respectively, gives
4 4
4x1 + 9x2 + 16x3 = 11
− 3/2x2 − 4x3 = −5/2
− 5/4x2 − x3 = −7/5
For the second elimination step, −3/2 is the largest absolute coefficient of second
variable x2 , so eliminate second variable x2 from the third row by subtracting the
5
multiple m32 = of row 2 from row 3, gives
6
4x1 + 9x2 + 16x3 = 11
− 3/2x2 − 4x3 = −5/2
1/3x3 = 1/3

Obviously, the original set of equations has been transformed to an equivalent

upper-triangular form. Now using backward substitution, gives,
x1 = 1, x2 = −1, x3 = 1, the required solution. •
Strictly Diagonally Dominant Matrix

Definition 5
A square matrix is said to be strictly diagonally dominant (SDD) if the absolute
value of each element on the main diagonal is greater than the sum of the absolute
values of all the other elements in that row. Thus, strictly diagonally
dominant matrix is defined as
n
X
|aii | > |aij |, for i = 1, 2, . . . , n. (14)
j=1
j6=i
Example 0.6
The matrix  
7 3 1
A= 1 6 3 ,
−2 4 8
is strictly diagonally dominant since

|7| > |3| + |1|, that is, 7 > 4,

|6| > |1| + |3|, that is, 6 > 4,
|8| > | − 2| + |4|, that is, 8 > 6,

but the following matrix  

6 −3 4
B= 3 7 3 ,
5 −4 10
is not strictly diagonally dominant since

|6| > | − 3| + |4|, that is, 6 > 7,

Theorem 6
If a matrix A is strictly diagonally dominant, then:
1. Matrix A is nonsingular.
2. Gaussian elimination without row interchange can be performed on the linear
system Ax = b. •

Example 0.7
Solve the following linear system using the simple Gaussian elimination method.

5x1 + x2 + x3 = 7
2x1 + 6x2 + x3 = 9
x1 + 2x2 + 9x3 = 12

solution. Start with the augmented matrix form

 
.
 5 1 1 .. 7 
 
..
,
 
 2 6 1 . 9
 
 .. 
1 2 9 . 12
and since a11 = 5 6= 0, so we can eliminate the elements a21 and a31 by
subtracting from the second and third rows the appropriate multiples of the first
row. In this case the multiples are given
2 1
m21 = and m31 = .
5 5
Hence  
..
 5 1 1 . 7 
 
 .. 
.
 0 28/5 3/5 . 31/5 
 
 .. 
0 9/5 44/5 . 53/5
(1) (1)
As = 28/5 6= 0, therefore, we eliminate entry in a32 position by subtracting
a22
1.8
the multiple m32 = = 9/28 of the second row from the third row, to get
5.6
 
.
 5 1 1 .. 7 
 
. .
.
 
 0 28/5 3/5 . 31/5 
 
 .. 
0 0 43/5 . 43/5
Obviously, the original set of equations has been transformed to an
upper-triangular form. Since all the diagonal elements of the obtaining
upper-triangular matrix are nonzero, which means that the coefficient matrix of
the given system is nonsingular and therefore, the given system has a unique
solution. Now expressing the set in algebraic form yields

5x1 + x2 + x3 = 7
(28/5)x2 + (3/5)x3 = 31/5
(43/5)x3 = 43/5

Now using backward substitution to get the solution of the system as

(43/5)x3 = 43/5, gives x3 = 1,

(28/5)x2 = −(3/5)x3 + 31/5, gives x2 = 1,
5x1 = 7 − x2 − x3 , gives x1 = 1.
. •
Norms of Vectors and Matrices

For solving linear systems, we discuss a method for quantitatively measuring the
distance between vectors in Rn , the set of all column vectors with real
components, to determine whether the sequence of vectors that results from using
an direct method converges to a solution of the system. To define a distance in
Rn , we use the notation of the norm of a vector.
Vector Norms
It is sometimes useful to have a scalar measure of the magnitude of a vector. Such
a measure is called a vector norm and for a vector x is written as kxk.
A vector norm on Rn is a function, from Rn to R satisfying:
1. kxk > 0 for all x ∈ Rn .
2. kxk = 0 if and only if x = 0.
3. kαxk = |α|kxk, for all α ∈ R, x ∈ Rn .
4. kx + yk ≤ kxk + kyk, for all x, y ∈ Rn .
There are three norms in Rn that are most commonly used in applications, called
l1 -norm, l2 -norm, and l∞ -norm, and are defined for the given vectors

x = [x1 , x2 , . . . , xn ]T

as !1/2
n
X n
X
kxk1 = |xi |, kxk2 = x2i , kxk∞ = max |xi |.
1≤i≤n
i=1 i=1

The l1 -norm is called the absolute norm, the l2 -norm is frequently called the
Euclidean norm as it is just the formula for distance in ordinary three-dimensional
Euclidean space extended to dimension n. Finally, the l∞ -norm is called the
maximum norm or occasionally the uniform norm. All these three norms are also
called the natural norms.
Here we will consider the vector l∞ -norm only.
Example 0.8
Compute lp -norms (p = 1, 2, ∞) of the vector x = [−5, 3, −2]T in R3 .
Solution. These lp -norms (p = 1, 2, ∞) of the given vector are:

kxk1 = |x1 | + |x2 | + |x3 | = | − 5| + |3| + | − 2| = 10.

h i1/2
kxk2 = (x21 + x22 + x23 )1/2 = (−5)2 + (3)2 + (−2)2 ≈ 6.1644.

kxk∞ = max{|x1 |, |x2 |, |x3 |} = max{| − 5|, |3|, | − 2|} = 5.

Matrix Norms

A matrix norm is a measure of how well one matrix approximates another, or,
more accurately, of how well their difference approximates the zero matrix. An
iterative procedure for inverting a matrix produces a sequence of approximate
inverses. Since in practices such a process must be terminated, it is desirable to
have some measure of the error of approximate inverse.
So a matrix norm on the set of all n × n matrices is a real-valued function, k.k,
defined on this set, satisfying for all n × n matrices A and B and all real number α
as follows:
1. kAk > 0, A 6= 0.
2. kAk = 0, A = 0.
3. kIk = 1, I is the identity matrix.
4. kαAk = |α|kAk, for some scalar α ∈ R.
5. kA + Bk ≤ kAk + kBk.
6. kABk ≤ kAkkBk.
7. kA − Bk ≥ kAk − kBk .
Several norms for matrices have been defined, we shall use the following three
natural norms l1 , l2 , and l∞ for a square matrix of order n:
n
!
X
kAk1 = max |aij | = maximum column-sum.
j
i=1

kAk2 = max kAxk2 = spectral norm.

kxk2 =1
 
n
X
kAk∞ = max  |aij | = row-sum norm.
i
j=1

For m × n matrix, we can paraphrase the Frobenius norm (or Euclidean norm),
which is not a natural norm and is define as
 1/2
m X
X n
2
kAkF =  |aij |  .
i=1 j=1

It can be shown that q

kAkF = tr(AT A),

where tr(AT A) is the trace of a matrix AT A, that is, the sum of the diagonal
entries of AT A. The Frobenius norm of a matrix is a good measure of the
magnitude of a matrix. It is to be noted that kAkF 6= kAk2 . For a diagonal
matrix, all norms have the same values.
Also, here we will consider the matrix l∞ -norm only.
Example 0.9
Compute lp -norms (p = ∞, F ) of the following matrix
 
4 2 −1
A= 3 5 −2  .
1 −2 7

l∞ -norm is defined as
3
X
|a1j | = |4| + |2| + | − 1| = 7,
j=1
X3
|a2j | = |3| + |5| + | − 2| = 10,
j=1
X3
|a3j | = |1| + | − 2| + |7| = 10,
j=1

so
kAk∞ = max{7, 10, 10} = 10.
In addition, we have the lF -norm of the matrix as

kAkF = (16 + 4 + 1 + 9 + 25 + 4 + 1 + 4 + 49)1/2 ≈ 10.6301,

the Frobenius norm of the given matrix. •

Iterative Methods for Solving Linear Systems

The methods discussed in the previous section for the solution of the system of
linear equations have been direct, which required a finite number of arithmetic
operations. The elimination methods of solving such systems usually yield
sufficiently accurate solutions for approximately 20 to 25 simultaneous equations,
where most of the unknowns are present in all of the equations. When the
coefficients matrix is sparse (has many zeros), a considerably large number of
equations can be handled by the elimination methods. But these methods are
generally impractical when many hundreds or thousands of equations must be
solved simultaneously.
There are, however, several methods which can be used to solve large numbers of
simultaneous equations. These methods are, called iterative methods by which an
approximation to the solution of a system of linear equations may be obtained.
Here, we consider just two of these iterative methods. These two forms the basis
of a family of methods which are designed either to accelerate the convergence or
to suit some particular computer architecture.
Jacobi Iterative Method
This is one of the easiest iterative method to find the approximate solution of the
system of linear equations
Ax = b, (15)
To explain its procedure, consider a system of three linear equations as follows:
a11 x1 + a12 x2 + a13 x3 = b1
a21 x1 + a22 x2 + a23 x3 = b2
a31 x1 + a32 x2 + a33 x3 = b3

The solution process starts by solving for the first variable x1 from first equation,
second variable x2 from second equation and third variable x3 from third
equation, gives
a11 x1 = b1 − a12 x2 − a13 x3
a22 x2 = b2 − a21 x1 − a23 x3
a33 x3 = b3 − a31 x1 − a32 x2
Divide both sides of the above three equations by their diagonal elements, a11 , a22
and a33 respectively, to have
1 h i
x1 = b1 − a12 x2 − a13 x3
a11
1 h i
x2 = b2 − a21 x1 − a23 x3
a22
1 h i
x3 = b3 − a31 x1 − a32 x2
a33
 (k) T
h i
(k) (k)
Let x(k) = x1 , x2 , x3 be an initial solution of the exact solution x of the
linear system (22), then we define an iterative sequence

(k+1) 1 h (k) (k)

i
x1 = b1 − a12 x2 − a13 x3
a11

(k+1) 1 h (k) (k)

i
x2 = b2 − a21 x1 − a23 x3 (16)
a22

(k+1) 1 h (k) (k)

i
x3 = b3 − a31 x1 − a32 x2
a33
where k is the number of iterative steps. Then the form (16) is called the Jacobi
formula for system of three equations. For a general system of n linear equations,
the Jacobi method is defined by
 
i−1 n
(k+1) 1  X (k)
X (k) 
xi = bi − aij xj − aij xj (17)
aii j=1 j=i+1

i = 1, 2, . . . , n, k = 0, 1, 2, . . . ,
provided that the diagonal elements aii 6= 0 for each i = 1, 2, . . . , n. If the diagonal
elements equal to zero, then reordering of the equations can be performed so that
no element in the diagonal position equal to zero. As usual with iterative
(0)
methods, an initial approximation xi must be supplied. If we don’t have
(0)
knowledge about the exact solution, it is conventional to start with xi = 0 for all
i. The iterations defined by (17) are stopped when

kx(k+1) − x(k) k < , (18)

or by using other possible stopping criteria

kx(k+1) − x(k) k
< , (19)
kx(k+1) k

where  is a preassigned small positive number. For this purpose, any convenient
norm can be used, the most usual being the l∞ -norm.
Example 0.10
Solve the following system of equations using the Jacobi iterative method, using
 = 10−6 in the l∞ -norm.

5x1 − x2 + x3 = 10
2x1 + 8x2 − x3 = 11
−x1 + x2 + 4x3 = 3

Start with the initial solution x(0) = [0, 0, 0]T .

Solution. The Jacobi iterative method for the given system has the form

(k+1) 1h (k) (k)

i
x1 = 10 + x2 − x3
5
(k+1) 1h (k) (k)
i
x2 = 11 − 2x1 + x3
8
(k+1) 1h (k) (k)
i
x3 = 3 + x1 − x2
4
 (0) (0) (0)
and starting with initial approximation x1 = 0, x2 = 0, x3 = 0, then for k = 0,
we obtain
(1) 1h (0) (0)
i 1h i
x1 = 10 + x2 − x3 = 10 + 0 − 0 = 2,
5 5
(1) 1h (0) (0)
i 1h i
x2 = 11 − 2x1 + x3 = 11 − 0 + 0 = 1.375,
8 8
(1) 1 h
(0) (0)
i 1h i
x3 = 3 + x1 − x2 = 3 + 0 − 0 = 0.75.
4 4
The first and subsequent iterations are listed in Table 1.

Table: Solution of the Example 0.10

(k) (k) (k)

k x1 x2 x3
0 0.000000 0.000000 0.000000
1 2.000000 1.375000 0.750000
2 2.125000 0.968750 0.906250
.. .. .. ..
. . . .
15 2.000000 0.999999 1.000000
16 2.000000 1.000000 1.000000
Note that the Jacobi method converges and after 16 iterations we obtained what is
obviously the exact solution. Ideally the iteration should stop automatically when
we obtained the required accuracy using one of the stopping criteria mentioned by
(18) or (19).
To get the above results using MATLAB command, we do the following:
>> Ab = [A|b] = [5 − 1 1 10; 2 8 − 1 11; −1 1 4 3];
>> x = [0 0 0]; acc = 0.5e − 6; JacobiM (Ab, x, acc);
Example 0.11
Solve the following system of equations using the Jacobi iterative method.

2x1 + 8x2 − x3 = 11
5x1 − x2 + x3 = 10
−x1 + x2 + 4x3 = 3

Start with the initial solution x(0) = [0, 0, 0]T .

Solution. Results for this linear system are listed in Table 2. •

Table: Solution of the Example 0.11

(k) (k) (k)

k x1 x2 x3
0 0.000000 0.000000 0.000000
1 5.500000 -10.0000 0.750000
2 45.87500 18.25000 4.625000
3 -65.1875 224.0000 7.656250

Notice that Jacobi method diverges rapidly. Although the given linear system is
same as the linear system of the previous Example 0.10 except the first and second
equations are interchanged. From this example we concluded that Jacobi iterative
method is not always convergent.
 Program 3.10
MATLAB m-file for the Jacobi Iterative Method for Linear System
function x=JacobiM(Ab,x,acc)
[n,t]=size(Ab); b=Ab(1:n,t); R=1; k=1; d(1,1:n+1)=[0 x]; while R > acc
for i=1:n; sum=0; for j=1:n; if j ˜ =i
sum = sum + Ab(i, j) ∗ d(k, j + 1); end; x(1, i) = (1/Ab(i, i)) ∗ (b(i, 1) − sum);end;end
k=k+1; d(k,1:n+1)=[k-1 x]; R=max(abs((d(k,2:n+1)-d(k-1,2:n+1))));
if k > 10 & R > 100 (’Jacobi Method is diverges’) break; end; end; x=d;

Procedure [Jacobi Method]

1. Check the coefficient matrix A is strictly diagonally dominant (for guaranteed
convergence).
2. Initialize the first approximation x(0) and pre-assigned accuracy .
bi
3. Compute the constant c = D−1 b = , for i = 1, 2, . . . , n.
aii
4. Compute the Jacobi iteration matrix TJ = −D−1 (L + U ).
(k+1) (k)
5. Solve for the approximate solutions xi = T J xi + c, i = 1, 2, . . . , n
and k = 0, 1, . . .
(k+1) (k)
6. Repeat step 5 until kxi − xi k < .
Gauss-Seidel Iterative Method

This is one of the most popular and widely used iterative method to find the
approximate solution of the system of linear equations. This iterative method is a
modification of the Jacobi iterative method and give us good accuracy by using
the most recently calculated values.
From the Jacobi iterative formula (17), it is seen that the new estimates for
solution x are computed from the old estimates and only when all the new
estimates have been determined are then used in the right-hand side of the
equation to perform the next iteration. But the Gauss-Seidel method is to make
use of the new estimates in the right-hand side of the equation as soon as they
become available. For example, the Gauss-Seidel formula for the system of three
equations can be define an iterative sequence
 (k+1) 1 h (k) (k)
i
x1 = b1 − a12 x2 − a13 x3
a11

(k+1) 1 h (k+1) (k)

i
x2 = b2 − a21 x1 − a23 x3 (20)
a22

(k+1) 1 h (k+1) (k+1)

i
x3 = b3 − a31 x1 − a32 x2
a33
For a general system of n linear equations, the Gauss-Seidel iterative method
defined as
 
i−1 n
(k+1) 1  X (k+1)
X (k) 
xi = bi − aij xj − aij xj (21)
aii j=1 j=i+1

i = 1, 2, . . . , n, k = 0, 1, 2, . . .

The Gauss-Seidel iterative method is sometimes called the method of successive

iteration, because the most recent values of all xi are used in the calculation.
Example 0.12
Solve the following system of equations using the Gauss-Seidel iterative method,
with  = 10−6 in l∞ -norm.

5x1 − x2 + x3 = 10
2x1 + 8x2 − x3 = 11
−x1 + x2 + 4x3 = 3

Start with the initial solution x(0) = [0, 0, 0]T .

Solution. The Gauss-Seidel iteration for the given system is

(k+1) 1h (k) (k)

i
x1 = 10 + x2 − x3
5
(k+1) 1h (k+1) (k)
i
x2 = 11 − 2x1 + x3
8
(k+1) 1h (k+1) (k+1)
i
x3 = 3 + x1 − x2
4
 (0) (0) (0)
and starting with initial approximation x1 = 0, x2 = 0, x3 = 0, then for k = 0,
we obtain
(1) 1h (0) (0) 1h
i i
x1 = 10 + x2 − x3 10 + 0 − 0 = 2,
=
5 5
(1) 1h (1) (0)
i 1h i
x2 = 11 − 2x1 + x3 = 11 − 4 + 0 = 0.875,
8 8
(1) 1h (1) (1)
i 1h i
x3 = 3 + x1 − x2 = 3 + 2 − 0.875 = 1.03125.
4 4
The first and subsequent iterations are listed in Table 3.

Table: Solution of the Example 0.12

(k) (k) (k)

k x1 x2 x3
0 0.000000 0.000000 0.000000
1 2.000000 0.875000 1.031250
2 1.968750 1.011719 0.989258
3 2.004492 0.997534 1.001740
.. .. .. ..
. . . .
9 2.000000 0.999999 1.000000
10 2.000000 1.000000 1.000000
Note that the Gauss-Seidel method converged and required 10 iterations to obtain
the correct solution for the given system, which is 6 iterations less than required
by the Jacobi method for the same Example 0.10.

The above results can be obtained using MATLAB command as follows:

>> Ab = [A|b] = [5 − 1 1 10; 2 8 − 1 11; −1 1 4 3];
>> x = [0 0 0]; acc = 0.5e − 6; GaussSM (Ab, x, acc);
Example 0.13
Solve the following system of equations using the Gauss-Seidel iterative method.

2x1 + 8x2 − x3 = 11
5x1 − x2 + x3 = 10
−x1 + x2 + 4x3 = 3

Start with the initial solution x(0) = [0, 0, 0]T .

Solution. Results for this linear system are listed in Table 4. Note that in this
case the Gauss-Seidel method diverges rapidly. Although the given linear system
is same as the linear system of the previous Example 0.12 except the first and
second equations are interchanged. From this example we concluded that the
Gauss-Seidel iterative method is not always convergent. •

Table: Solution of the Example 0.13

(k) (k) (k)

k x1 x2 x3
0 0.000000 0.000000 0.000000
1 5.500000 17.5000 -2.25000
2 -65.6250 -340.375 69.43750
3 1401.719 7068.031 -1415.83
Procedure
[Gauss-Seidel Method]

1. Check the coefficient matrix A is strictly diagonally dominant (for guaranteed

convergence).
2. Initialize the first approximation x(0) ∈ R and pre-assigned accuracy .
3. Compute the constant c = (D + L)−1 b.
4. Compute the Gauss-Seidel iteration matrix TG = −(D + L)−1 U .
(k+1) (k)
5. Solve for the approximate solutions xi = T G xi + c, i = 1, 2, . . . , n
and k = 0, 1, . . .
(k+1) (k)
6. Repeat step 5 until kxi − xi k < .
Note that from the Examples (0.10) and (0.12), we noted that the solution by the
Gauss-Seidel method converges more quickly than the Jacobi method. In general,
we may state that if both the Jacobi method and the Gauss-Seidel
method are converge, then the Gauss-Seidel method converges more
quickly. This is generally the case but not always true. In fact, there are some
linear systems for which the Jacobi method converges but the Gauss-Seidel
method does not, and others for which the Gauss-Seidel method converges but the
Jacobi method does not.
Matrix Forms of Iterative Methods for Linear System

The iterative methods to solve the system of linear equations

Ax = b, (22)

start with an initial approximation x(0) ∈ R to the solution x of the linear system
(22), and generates a sequence of vectors {x(k) }∞
k=0 that converges to x. Most of
these iterative methods involve a process that converts the system (22) into an
equivalent system of the form
x = T x + c, (23)
for some square matrix T and vector c. After the initial vector x(0) is selected, the
sequence of approximate solutions vector is generated by computing

x(k+1) = T x(k) + c, for k = 0, 1, 2, . . . (24)

The sequence is terminated when the error is sufficiently small, that is

kx(k+1) − x(k) k < , for small positive . (25)

Note that a matrix T is called iteration matrix and a vector c is a column
matrix. We can find the forms of these matrices easily for both iterative methods
as follows. Let a matrix A can be written as

A = L + D + U, (26)

where L is strictly lower-triangular, U is strictly upper-triangular, and D is the

diagonal parts of the coefficients matrix A, that is

0 0 0 ··· 0 0 a12 a13 ··· a1n

   
 a21 0 0 ··· 0   0 0 a23 ··· a2n 
 a a32 0 ··· 0 0 0 0 ··· a3n
   
L =  31 , U = ,
  
 .. .. .. .. ..   .. .. .. .. .. 
 . . . . .   . . . . . 
an1 an2 an3 ··· 0 0 0 0 ··· 0

and
a11 0 0 ··· 0
 
 0 a22 0 ··· 0 
0 0 a33 ··· 0
 
D= .
 
 .. .. .. .. .. 
 . . . . . 
0 0 0 ··· ann
Then the linear system (22) can be written as

(L + D + U )x = b. (27)

Now we find forms of both matrices T and c which help us to solve the linear
system.
 TJ = −D−1 (L + U ) and cj = D−1 b, (28)
are called Jacobi iteration matrix and Jacobi constant column matrix,
respectively, and their elements are defined by
aij
tij = , i, j = 1, 2, . . . , n, i 6= j,
aii

tij = 0, i = j,

bi
ci = , i = 1, 2, . . . , n.
aii
Note that the diagonal elements of Jacobi iteration matrix TJ are always zero.

TG = −(L + D)−1 U and cG = (L + D)−1 b, (29)

are called Gauss-Seidel iteration matrix and Gauss-seidel constant
column matrix, respectively.
Example 0.14
Consider the following system

6x1 + 2x2 = 1
x1 + 7x2 − 2x3 = 2
3x1 − 2x2 + 9x3 = −1

Find the matrix form of iterative (Jacobi and Gauss-Seidel) methods.

Solution. Since the given matrix A is
 
6 2 0
A= 1 7 −2  ,
3 −2 9

and so
     
0 0 0 0 2 0 6 0 0
A=L+U +D = 1 0 0 + 0 0 −2  +  0 7 0 .
3 −2 0 0 0 0 0 0 9
Jacobi Iterative Method
Since the matrix form of the Jacobi iterative method can be written as

x(k+1) = TJ x(k) + cJ , k = 0, 1, 2, . . . ,

where
TJ = −D−1 (L + U ) and cJ = D−1 b.
One can easily compute the Jacobi iteration matrix TJ and the vector cJ as
follows:
 1   2   1 
0 0 0 − 0
 6   6   6 
      
  0 2 0    
 1   1 2   2 
TJ = −  0
 0   1 0 −2  = 
 −7 0  and c= 
 7 
 3 −2 0  7 


 7 

     
 1   3 2   1 
0 0 − 0 −
9 9 9 9
Thus the matrix form of Jacobi iterative method is
 2   1 
0 − 0

 6 


 6 

   
1 2  (k)  2
x(k+1) = 
   
 −7 x +
0 , k = 0, 1, 2.
 7   7 

   
 3 2   1 
− 0 −
9 9 9
Gauss-Seidel Iterative Method
Now by using Gauss-Seidel method, first we compute the Gauss-Seidel iteration
matrix TG and the vector cG as follows:
 1   1 
0 − 0

 3 


 6 
   
 1 2   11 
TG =  0  and cG =  .

 21 7 

 42 
   
 23 4   41 
0 −
189 63 378
Thus the matrix form of Gauss-Seidel iterative method is
 1   1 
0 − 0

 3 


 6 
   
1 2 11
x(k+1) =   x(k) + 
   
, k = 0, 1, 2.
 0 21 7   42 
  
   
 23 4   41 
0 −
189 63 378
Theorem 7
(Second Sufficient Condition for Convergence)
For any initial approximation x(0) ∈ R, the sequence {x(k) }∞
k=0 of
approximations defined by

x(k+1) = T x(k) + c, for each k ≥ 0, and c 6= 0, (30)

converges to the unique solution of x = T x + c if kT k < 1 for any natural

matrix norm, and the following error bounds hold:

kx − x(k) k ≤ kT kk kx(0) − xk,

kT kk
kx − x(k) k ≤ kx(1) − x(0) k.
1 − kT k

Note that smaller the value of the kT k, faster the convergence of the iterative
methods.
Example 0.15
Consider the following nonhomogeneous linear system Ax = b, where
   
5 0 −1 1
A =  −1 3 0  and b =  2  .
0 −1 4 4

Find the matrix form of iterative (Jacobi and Gauss-Seidel) methods and show
that Gauss-Seidel iterative method converges faster than Jacobi iterative method
for the given system.
Solution. Here we will show that the l∞ -norm of the Gauss-Seidel iteration
matrix TG is less than the l∞ -norm of the Jacobi iteration matrix TJ , that is

kTG k∞ < kTJ k∞ .

The Jacobi iteration matrix TJ can be obtained from the given matrix A as follows
 1 
0 0
 −1   
 5 
5 0 0 0 0 −1  
−1
 1 
TJ = −D (L+U ) = −  0 3 0   −1 0 0 =  3 0 0 .

0 0 4 0 −1 0 



 1 
0 0
4
Thus the matrix form of Jacobi iterative method is
 1  1 
0 0


 5    5 
   
 1
x(k+1) = 
 (k)  
0 0 x
 + 2 , k ≥ 0.
 3  





 3 

 1 
0 0 1
4
Similarly, Gauss-Seidel iteration matrix TG is defined as
 −1  
5 0 0 0 0 −1
TG = −(D + L)−1 U = −  −1 3 0   0 0 0 ,
0 −1 4 0 0 0

and it gives
 1   1 
0 0 0 0

 5 


  5 


 1
 0 0 −1  
1   1 
TG = −
 15 0   0 0 0 =
 0 0 .
 3 
 0 0 0  15 

   
 1 1 1   1 
0 0
60 15 4 60
So the matrix form of Gauss-Seidel iterative method is
 1   1 
0 0

 5 
 5



   
(k+1)
 1  (k) 

 11

x = 0 0
 x + , k ≥ 0.
 15  
 15



   
 1   71 
0 0
60 60
Since the l∞ -norm of the matrix TJ is
 
1 1 1 1
kTJ k∞ = max , , = = 0.3333 < 1,
5 3 4 3

and the l∞ -norm of the matrix TG is

 
1 1 1 1
kTG k∞ = max , , = = 0.2000 < 1.
5 15 60 5

Since kTG k∞ < kTJ k∞ , which shows that Gauss-Seidel method will converge
faster than Jacobi method for the given linear system. •
Example 0.16
Consider the following linear system of equations

4x1 − x2 + x3 = 12
−x1 + 3x2 + x3 = 1
x1 + x2 + 5x3 = −14

(a) Show that both iterative methods (Jacobi and Gauss-Seidel) will converge by
using kT k∞ < 1.
(b) Find second approximation x(2) when the initial solution is x(0) = [4, 3, −3]T .
(c) Compute the error bounds for your approximations.
(d) How many iterations needed to get an accuracy within 10−4 .
Solution. From (26), we have
       
4 −1 1 0 0 0 0 −1 1 4 0 0
A =  −1 3 1  =  −1 0 0 + 0 0 1 + 0 3 0 
1 1 5 1 1 0 0 0 0 0 0 5

= L + U + D.
Jacobi Method:
(a) Since the Jacobi iteration matrix is defined as

TJ = −D−1 (L + U ),

and by using the given information, we have

 1   1 1 
0 0 0 −
 4  
0 −1 1
  4 4 
   
   
 1    1 1 
TJ = −  0 3 0  −1 0 1 = 0 −  .

   3 3 
   
   
 1  1 1 0  1 1 
0 0 − − 0
5 5 5
Then the l∞ norm of the matrix TJ is
 
2 2 2 2
kTJ k∞ = max , , = < 1.
4 3 5 3

Thus the Jacobi method will converge for the given linear system.
(b) The Jacobi method for the given system is

(k+1) 1h (k) (k)

i
x1 = 12 + x2 − x3
4
(k+1) 1h (k) (k)
i
x2 = 1 + x1 − x3
3
(k+1) 1h (k) (k)
i
x3 = − 14 − x1 − x2
5
(0) (0) (0)
Starting with initial approximation x1 = 4, x2 = 3, x3 = −3, and for k = 0, 1,
we obtain the first and the second approximations as

x(1) = [4.5, 2.6667, −4.2]T and x(2) = [4.7167, 3.2333, −4.2333]T .

(c) Using the error bound formula (31), we obtain

   
4.5 4
(2) (2/3)2  4
kx − x k ≤ 2.6667  −  3  ≤ (1.2) = 1.6.
1 − 2/3 −4.2 −3 3
(d) To find the number of iterations, we use the formula (31) as

kTJ kk
kx − x(k) k ≤ kx(1) − x(0) k ≤ 10−4 .
1 − kTJ k

It gives
(2/3)k 10−4
(1.2) ≤ 10−4 , or (2/3)k ≤ .
1/3 3.6
Taking ln on both sides, we obtain
 −4 
10
k ln(2/3) ≤ ln , gives k ≥ 25.8789, or k = 26,
3.6

which is the required number of iterations.

Gauss-Seidel Method:
(a) Since the Gauss-Seidel iteration matrix is defined as

TG = −(D + L)−1 U,

and by using the given information, we have

 1   1 1 
0 0 0 −

 4   0 −1
 1


 4 4 

    
 1 1    1 5 
TG = −  0  0 0 1 = 0 − .

 12 3 
  
 12 12 

   
 4 1 1  0 0 0  4 8 
− − 0 −
60 15 5 60 60
Then the l∞ norm of the matrix TG is
 
2 6 12 1
kTG k∞ = max , , = < 1.
4 12 60 2

Thus the Gauss-Seidel method will converge for the given linear system.
(b) The Gauss-Seidel method for the given system is

(k+1) 1h (k) (k)

i
x1 = 12 + x2 − x3
4
(k+1) 1h (k+1) (k)
i
x2 = 1 + x1 − x3
3
(k+1) 1h (k+1) (k+1)
i
x3 = − 12 − x1 − x2
5
(0) (0) (0)
Starting with initial approximation x1 = 4, x2 = 3, x3 = −3, and for k = 0, 1,
we obtain the first and the second approximations as

x(1) = [4.5, 2.8333, −4.2667]T and x(2) = [4.775, 3.3472, −4.4244]T .

(c) Using the error bound formula (31), we obtain

   
4.5 4
(2) (1/2)2  1
kx − x k ≤ 2.8333  −  3  ≤ (1.2667) = 0.6334.
1 − 1/2 −4.2667 −3 2
(d) To find the number of iterations, we use the formula (31) as

kTJ kk
kx − x(k) k ≤ kx(1) − x(0) k ≤ 10−4 .
1 − kTJ k

It gives
(1/2)k 10−4
(1.2667) ≤ 10−4 , or (1/2)k ≤ .
1/2 2.5334
Taking ln on both sides, we obtain

10−4
 
k ln(1/2) ≤ ln , gives k ≥ 14.6084 or k = 15,
2.5334

which is the required number of iterations. •

Errors in Solving Linear Systems

Any computed solution of a linear system must, because of round-off and other
errors, be considered an approximate solution. Here we shall consider the most
natural method for determining the accuracy of a solution of the linear system.
One obvious way of estimating the accuracy of the computed solution x∗ is to
compute Ax∗ and to see how close Ax∗ comes to b. Thus if x∗ is an approximate
solution of the given system Ax = b, we compute a vector

r = b − Ax∗ , (31)

which is called the residual vector and can be easily calculated. The quantity

krk kb − Ax∗ k
= ,
kbk kbk

is called the relative residual.

The smallness of the residual then provides a measure of the goodness of the
approximate solution x∗ . If every component of vector r vanishes, then x∗ is the
exact solution. If x∗ is a good approximation then we would expect each
component of r to be small, at least in a relative sense. For example, the following
linear system
x1 + 2x2 = 3
1.0001x1 + 2x2 = 3.0001
has the exact solution x = [1, 1]T but has a poor approximate solution
x∗ = [3, 0]T . To see how good this solution is, we compute the residual,
r = [0, −0.0002]T , and so krk∞ = 0.0002. Although the norm of the residual
vector is small, the approximate solution x∗ = [3, 0]T is obviously quite poor; in
fact kx − x∗ k∞ = 2.
Program 3.12
MATLAB m-file for finding Residual Vector
function r=RES(A,b,x0)
[n,n]=size(A);
for i=1:n; R(i) = b(i); for j=1:n
R(i)=R(i)-A(i,j)*x0(j);end; RES(i)=R(i); end; r=RES’
To get above results using MATLAB command, we do the following:
>> A = [1 2; 1.0001 2]; b = [3 3.0001]; x0 = [3 0];
>> RESID(A, b, x0); x = [1 1]; Error = norm((x − x0), inf );
We can conclude from the residual that the approximate solution is correct to at
most three decimal places. Also, the following linear system

1.0000x1 + 0.9600x2 + 0.8400x3 + 0.6400x4 = 3.4400

0.9600x1 + 0.9214x2 + 0.4406x3 + 0.2222x4 = 2.5442
0.8400x1 + 0.4406x2 + 1.0000x3 + 0.3444x4 = 2.6250
0.6400x1 + 0.2222x2 + 0.3444x3 + 1.0000x4 = 2.2066

has exact solution x = [1, 1, 1, 1]T and having the approximate solution due to the
Gaussian elimination without pivoting is

x∗ = [1.0000322, 0.99996948, 0.99998748, 1.0000113]T ,

and the residual is

r = [0.6 × 10−7 , 0.6 × 10−7 , −0.53 × 10−5 , −0.21 × 10−4 ]T

The approximate solution due to the Gaussian elimination with partial pivoting is

x∗ = [0.9999997, 0.99999997, 0.99999996, 1.0000000]T ,

and the residual is

r = [0.3 × 10−7 , 0.3 × 10−7 , 0.6 × 10−7 , 0.1 × 10−8 ]T .

We found that all the elements of the residual for second case (with pivoting) are
less than 0.6 × 10−7 , whereas for first case (without pivoting) they are as large as
0.2 × 10−4 . Even without knowing the exact solution, it is clear that the solution
obtained in second case is much better than that of first case. The residual
provides a reasonable measure of the accuracy of a solution in those cases where
the error is primarily due to the accumulation of round-off errors.
Intuitively it would seem reasonable to assume that when krk is small for a given
vector norm, then the error kx − x∗ k would be small as well. In fact this is true
for some systems. However, there are systems of equations which do not satisfy
this property. Such systems are said to be ill-conditioned.
Conditioning of Linear Systems

In solving the linear system numerically we have to see the problem conditioning,
algorithm stability, and cost. Above we discussed efficient elimination schemes to
solve a linear system and these schemes are stable when pivoting is employed. But
there are some ill-conditioned systems which are tough to solve by any method.
Definition 8
(Condition Number of a Matrix)
The number kAkkA−1 k is called the condition number of a nonsingular matrix A
and is denoted by K(A), that is

cond(A) = K(A) = kAkkA−1 k. (32)

Note that the condition number K(A) for A depends on the matrix norm used
and can, for some matrices, vary considerably as the matrix norm is changed.
Since
1 = kIk = kAA−1 k ≤ kAkkA−1 k = K(A),
therefore, the condition number is always in the range 1 ≤ K(A) ≤ ∞ regardless
of any natural norm. The lower limit is attained for identity matrices and
K(A) = ∞ if A is singular. So the matrix A is well-behaved (well-conditioned) if
K(A) is close to 1 and is increasingly ill-conditioned when K(A) is significantly
greater than 1, that is, K(A) → ∞.
Example 0.17
Compute the condition number of the following matrix using the l∞ -norm
 
2 −1 0
A=  2 −4 −1  .
−1 0 2

Solution. Since the condition number of a matrix is defined as

K(A) = kAk∞ kA−1 k∞ .

First we calculate the inverse of the given matrix which is

 8 2 1 
− −
 13 13 13 
 
 
3 4 2
A−1 = 
 
 13 − 13 − 13  .

 
 
 4 1 6 
−
13 13 13
Now we calculate the l∞ -norm of both the matrices A and A−1 . Since the
l∞ -norm of a matrix is the maximum of the absolute row sums, we have

kAk∞ = max{|2| + | − 1| + |0|, |2| + | − 4| + | − 1|, | − 1| + |0| + |2|} = 7,

and
n 8 −2 −1 3 −4 −2 4 −1 6 o
kA−1 k∞ = max + + , + + , + + ,
13 13 13 13 13 13 13 13 13
which gives
11
kA−1 k∞ = .
13
Therefore,  
11
K(A) = kAk∞ kA−1 k∞ = (7) ≈ 5.9231.
13
Depending on the application, we might consider this number to be reasonably
small and conclude that the given matrix A is reasonably well-conditioned. •
Example 0.18
If the condition number of following matrix A is 8.8671, then find the l∞ -norm of
its inverse matrix, that is, kA−1 k∞
 
10.2 2.4 4.5
A =  −2.3 7.7 11.1  .
−5.5 −3.2 0.9

Solution. Since the condition number of a matrix is defined as

K(A) = kAk∞ kA−1 k∞ .

First we calculate the l∞ -norm of the given matrix A which is the maximum of
the absolute row sums, we have

kAk∞ = max{17.1000, 21.1000, 9.6} = 21.1000,

and as it is given K(A) = 8.8671, so we have

8.8671 = (21.1000)kA−1 k∞ .

Simplifying this, we get kA−1 k∞ = 0.4202. •

Theorem 9
(Error in Linear Systems)
Suppose that x∗ is an approximation to the solution x of the linear system
Ax = b and A is a nonsingular matrix and r is the residual vector for x∗ . Then
for any natural norm, the error is

kx − x∗ k ≤ krkkA−1 k, (33)

and the relative error is

kx − x∗ k krk
≤ K(A) , provided that x 6= 0, b 6= 0. (34)
kxk kbk
Example 0.19
Find the condition number of the following matrix (for n = 2, 3, . . .)
 
1 1
An = .
1 1 − 1/n

If n = 2 and x∗ = [−1.99, 2.99]T be the approximate solution of the linear system

Ax = [1, −0.5]T , then find the relative error.
Solution. We can easily find the inverse of the given matrix as
     
1 1 − 1/n −1 1 − 1/n −1 1−n n
A−1
n = = −n = .
(1 − 1/n) − 1 −1 1 −1 1 n −n

Then the l∞ -norm of both matrices An and A−1

n are

kAn k∞ = 2 and kA−1

n k∞ = 2n,

and so the condition number of the matrix can be computed as follows:

K(A) = kAn k∞ k|A−1

n k∞ = (2)(2n) = 4n and lim K(A) = ∞,
n→∞

which shows that the matrix An is obviously ill-conditioned.

Here we expect that the relative error in the calculated solution to a linear system
of the form An x = b could be as much as 4n times the relative residual.
The residual vector (by taking n = 2) can be calculated as
      
1 1 1 −1.99 0.000
r = b − A2 x∗ = − = ,
−0.5 1 0.5 2.99 −0.005

and it gives krk∞ = 0.005. Now using (35), we obtain

kx − x∗ k krk 0.005
≤ K(A) = (8) = 0.0400,
kxk kbk 1

which is the required relative error. •

Example 0.20
Consider a following linear system

x1 + x2 − x3 = 1
x1 + 2x2 − 2x3 = 0
−2x1 + x2 + x3 = −1

(a) Discuss the ill-conditioning of the given linear system.

(b) If x∗ = [2.01, 1.01, 1.98]T be an approximate solution of the given system, then
find the residual vector r and its norm krk∞ .
(c) Estimate the relative error using

kx − x∗ k krk
≤ K(A) , provided that x 6= 0, b 6= 0. (35)
kxk kbk
.
(d) Use the simple Gaussian elimination method to find approximate error using

A(x − x∗ ) = r, or x − x∗ = A−1 r. (36)

.
Solution. (a) Given the matrix
 
1 1 −1
A= 1 2 −2  ,
−2 1 1
and whose inverse can be computed as
 
2 −1 0
−1
A =  1.5 −0.5 0.5  .
(b) The residual vector can be calculated as
      
1 1 1 −1 2.01 −0.04
∗
r = b − Ax =  0  −  1 2 −2   1.01  =  −0.07  ,
−1 −2 1 1 1.98 0.03

and it gives
krk∞ = 0.07.

>> A = [1 1 − 1; 1 2 − 2; −2 1 1]; b = [1 0 − 1]0 ;

>> x0 = [2.01 1.01 1.98]0 ; r = RES(A, b, x0); rnorm = norm(r, inf );
(c) From (35), we have
kx − x∗ k krk
≤ K(A) .
kxk kbk
By using above parts (a) and (b) and the value kbk∞ = 1, we obtain

kx − x∗ k (0.07)
≤ (22.5) = 1.575.
kxk 1

>> RelErr = (K(A) ∗ rnorm)/norm(b, inf );

(d) To solve the linear system Ae = r, where
   
1 1 −1 −0.04
A =  1 2 −2  and r =  −0.07  ,
−2 1 1 0.03

and e = x − x∗ . Writing the above system in the augmented matrix form

 
.
 1 1 −1 .. −0.04 
 
.
 1 2 −2 .. −0.07  .
 
 
 .. 
−2 1 1 . 0.03

After applying forward elimination step of the simple Gauss elimination method,
we obtain  
..
 1 1 −1 . −0.04 
 
.
 0 1 −1 .. −0.03  .
 
 
 .. 
0 0 2 . 0.04
Now by using the backward substitution, we obtain the solution

e∗ = [−0.01, −0.01, 0.02]T ,

which is the required approximation of the exact error. •

Summary

In this lecture, we ...

I Considered the System of Linear Equations
I Introduced the Gaussian elimination method
I Introduced the Jacobi Iterative Method
I Introduced the Gauss-Seidel Iterative Method