NATIONAL INSTITUTE OF TECHNOLOGY : TIRUCHIRAPPALLI

Dr. K. Murugesan
Professor(HAG)
Department of Mathematics
National Insitute of Technology, Tiruchirappalli 620015,
Tamil Nadu, India
Email : [email protected]

Lecture notes on Solution of Linear Systems:

Content
Direct methods:

• Gauss elimination method

• Gauss-Jordan method

Iterative methods:

• Gauss-Jacobi method

• Gauss-Seidel method

LU decomposition method

1
 Solving system of linear equations
1 Direct methods:

1.1 Gauss Elimination method

This method is based on the idea of reducing the given system of equations Ax=b, to an upper triangular
system of equations Ux=z, using elementary row operations.
Consider a 3 × 3 system

a11 x1 + a12 x2 + a13 x3 =b1

a21 x1 + a22 x2 + a23 x3 =b2 (1.1)
a31 x1 + a32 x2 + a33 x3 =b3

.
Augmented matrix [A..b] is
 
a11 a12 a13 b1
 
 a21
 a22 a23 b2 
 (1.2)
a31 a32 a33 b3

Step 1:
Assume a11 6= 0.
It is called the first pivot.
Reduce all the elements below this pivot in the first column as zeros. For this multiply the first row in (1.1)
by a12 /a11 and a31 /a11 respectively and subtract from the second and third rows.That is,perform elementary
row operations R2 → R2 − (a21 /a11 )R1 and R3 → R3 − (a31 /a11 )R1 respectively.
The new augmented matrix is
 
a11 a12 a13 b1
 0 0 0 
 0
 a22 a23 b2 
 (1.3)
0 0 0
0 a32 a33 b3

Step 2:
0
Assume a22 6= 0.
This element is called the second pivot.
Reduce all the elements below this pivot in the second column as zeros. For this multiply the second
0 0
row in (1.3) by a32 /a22 and subtract from the third row.That is,perform elementary row operation R3 →
0 0
R3 − (a32 /a22 )R2 .

2
Now the augmented matrix is
 
a11 a12 a13 b1
 0 0 0 
 0
 a22 a23 b2  (1.4)
00 00
0 0 a33 b3

00
a33 is the third pivot.
This system is now in the required upper triangular form [U z]. By back substitution the solution vector
is obtained.
From third row of (1.4),
00 00
x3 = b3 /a33 .
From the second row of (1.4),
0 0 0
x2 = (b2 − a23 x3 )/a22 .
From the first row of (1.4),
x1 = (b1 − a12 x2 − a13 x3 )/a11 .

1.1.1 Remark

Gauss elimination method fails when anyone of the pivot is zero or it is a very small number,as the elimination
progresses.

1.1.2 Problems

1. Solve the system of equations using Gauss elimination method.

x1 + 2x2 + x3 =3
2x1 + 3x2 + 3x3 =10
3x1 − x2 + 2x3 =13

Sol.
.
Augmented matrix [A..b] is
 
1 2 1 3
 
 2 3 3 10 
 
3 −1 2 13

     
1 2 1 3 1 2 1 3 1 2 1 3
     
2 3 3 10 R2 →R2 −2R1 0 −1 1 4 R3 →R3 −7R2 0 −1 1 4 
  R3 →R3 −3R1    
3 −1 2 13 0 −7 −1 4 0 0 −8 −24

3
 By back substitution,

x1 + 2x2 + x3 =3
−x2 + x3 =4
−8x3 = − 24.

=⇒ x1 = 2, x2 = −1 and x3 = 3.

2. Solve the system of equations using Gauss elimination method.

3x1 + 3x2 + 4x3 =20

2x1 + x2 + 3x3 =13
x1 + x2 + 3x3 =6

Sol.
.
Augmented matrix [A..b] is
 
3 3 4 20
 
 2 1 3 13 
 
1 1 3 6

     
3 3 4 20 1 1 4/3 20/3 1 1 4/3 20/3
     
2 1 3 13 R1 →R1 /3 2 1 3 13  R2 →R2 −2R1 0 −1 1/3 −1/3
    R3 →R3 −R1  
1 1 3 6 1 1 3 6 0 0 5/3 −2/3

By back substitution,

x1 + x2 + 4/3x3 =20/3
−x2 + 1/3x3 = − 1/3
5/3x3 = − 2/3.

=⇒ x1 = 7, x2 = 1/5 and x3 = −2/5.

1.1.3 Problems to try:

3. Solve the system of equations using Gauss elimination method.

2x + y + z − 2w = − 10
4x + 2z + w =8
3x + 2y + 2z =7
x + 3y + 2z − w = − 5

4
 4. Solve the system of equations using Gauss elimination method.

x1 + 10x2 − x3 =3
2x1 + 3x2 + 20x3 =7
10x1 − x2 + 2x3 =4.

1.2 Gauss-Jordan method

In this method, the given system of equations Ax=b is reduced to a diagonal system of equations Ix=d,where
I is the identity matrix, using elementary row operations.
After eliminations are completed the augmented matrix for a 3 × 3 system is of the form
 
1 0 0 d1
 
 0
 1 0 d2 
 (1.5)
0 0 1 d3

and the solution is xi = di , i = 1, 2, 3.

1.2.1 Elimination procedure

Step 1: Make the elements below the first pivot as zeros,using elementary row operations.
Step 2: Make elements below and above the pivots as zeros,using elementary row operations.
Step 3: Divide each row by its pivot so that the final augmented matrix is of the form (1.5).

1.2.2 Problems

1. Solve the following system of equations using Gauss-Jordan method.

x1 + x2 + x3 =1
4x1 + 3x2 − x3 =6
3x1 + 5x2 + 3x3 =4.

Sol.
Augmented matrix is
 
1 1 1 1
 
 4
 3 −1 6 

3 5 3 4

 
1 1 1 1
 
= 0 −1 −5
 2 R2 →R2 −4R1
 R3 →R3 −3R1
0 2 0 1

5
  
1 1 1 1
 
= 0
 1 5 −2
 R2 →R2 /−1
0 2 0 1

 
1 0 −4 3
 
=
0 1 5 −2 R1 →R1 −R2

0 2 0 1

 
1 0 −4 3
 
= 0
 1 5 −2 R3 →R3 −2R2

0 0 −10 5

 
1 0 −4 3
 
= −2  R3 →R3 /−10
0 1 5 
0 0 1 −1/2

 
1 0 0 1
 
= 0
 1 5 −2  R1 →R1 +4R3

0 0 1 −1/2

 
1 0 0 1
 
= 0
 1 0 1/2  R2 →R2 −5R3

0 0 1 −1/2

Hence the solution is x1 = 1, x2 = 1/2, x3 = −1/2.

2. Solve the following system of equations using Gauss-Jordan method.

x + 2y + z =3
2x + 3y + 3z =10
3x − y + 2z =13.

Sol.
Augmented matrix is
 
1 2 1 3
 
 2 3 3 10 
 
3 −1 2 13

6
      
1 2 1 3 1 2 1 3 1 2 1 3
     
R2 →R2 −2R1   R2 →−R2
 R3 →R3 −3R1 0 −1 1 4
2 3 3 10 0 1 −1 −4
  
3 −1 2 13 0 −7 −1 4 0 −7 −1 4
     
1 0 3 11 1 0 3 11 1 0 0 2
     
R1 →R1 −2R2 0 1 −1 −4  R →(−1/8)R R →R −3R
0 1 −1 −4 0 1 0 −1
 3 3   1 1 3 
R3 →R3 +7R2  R2 →R2 +R3 
0 0 −8 −24 0 0 1 3 0 0 1 3

Hence the solution is x = 2, y = −1, z = 3.

1.2.3 Problems to try:

3. Solve the system of equations using Gauss-Jordan method

5x + y + z + w =4
x + 7y + z + w =12
x + y + 6z + w = − 5
x + y + z + 4w = − 6

4. Solve the system of equations using Gauss-Jordan method

x1 + 10x2 − x3 =3
2x1 + 3x2 + 20x3 =7
10x1 − x2 + 2x3 =4.

2 Iterative methods:
Consider a 3x3 system of algebraic equations.

a11 x1 + a12 x2 + a13 x3 =b1

a21 x1 + a22 x2 + a23 x3 =b2 (2.1)
a31 x1 + a32 x2 + a33 x3 =b3

7
2.1 Gauss-Jacobi iteration method
(Method of simultaneous displacement)
Assume that the pivots aii 6= 0,for all i.
Write the equation (2.1) as

a11 x1 =b1 − a12 x2 − a13 x3

a22 x2 =b2 − a21 x1 − a23 x3
a33 x3 =b3 − a31 x1 − a32 x2

The Jacobi iteration method is defined as

(k+1) 1 (k) (k)

x1 = [b1 − a12 x2 − a13 x3 ] (2.2)
a11
(k+1) 1 (k) (k)
x2 = [b2 − a21 x1 − a23 x3 ]
a22
(k+1) 1 (k) (k)
x3 = [b3 − a31 x1 − a32 x2 ], k = 0, 1, 2, . . .
a33

Note: Take the initials approximation as x1 = x2 = x3 = 0

2.1.1 Remark:

• A sufficient condition for the convergence of Jacobi method is that the system of equations is diago-
nally dominant,i.e.,the coefficient matrix A is diagonally dominant.If the system is not diagonally dominant,
exchange the equations,if possible,such that the new system is diagonally dominant and convergence is guar-
anteed.
• The necessary and sufficient condition for convergence is that the spectral radius of the iteration matrix
H is less than one,i.e.,ρ(H) < 1,where ρ(H) is the largest eigen value in magnitude of H.
•If no suitable initial approximation is available,choose x = 0,i.e.,xi = 0for all i.Then the initial approxi-
bi
mation becomes xi = aii
, for all i.

2.1.2 Problems

1. Solve the system of equations using Gauss Jacobi iteration method.

4x1 + x2 + x3 =2
x1 + 5x2 + 2x3 = − 6
x1 + 2x2 + 3x3 = − 4.

8
 Sol.
The given system is diagonally dominant.

(k+1) 1 (k) (k)

x1 = [2 − x2 − x3 ]
4
(k+1) 1 (k) (k)
x2 = [−6 − x1 − 2x3 ]
5
(k+1) 1 (k) (k)
x3 = [−4 − x1 − 2x2 ], k = 0, 1, 2, . . .
3

Take the initial approximation as xi = 0, i = 1, 2, 3.

Then

(1) 1 (0) (0) 1

x1 = [2 − x2 − x3 ] = [2 − 0 − 0] = 0.5
4 4
(1) 1 (0) (0) 1
x2 = [−6 − x1 − 2x3 ] = [−6 − 0 − 0] = −1.2
5 5
(1) 1 (0) (0) 1
x3 = [−4 − x1 − 2x2 ] = [−4 − 0 − 0] = −1.33333
3 3

iteration x1 x2 x3
0 0 0 0
1 0.5 -1.2 -1.33333
2 1.13333 -0.76668 -0.7
3 0.86667 -1.14667 -1.19998
4 1.08666 -0.89334 -0.85777
5 0.93778 -1.07422 -1.09998
6 1.0435 -0.9476 -0.9298
7 0.9694 -1.0368 -1.0494
8 1.0216 -0.9741 -0.9653
9 0.985 -1.0182 -1.0123
10 1.0076 -0.9921 -0.9829
11 0.9938 -1.0084 -1.0078

Hence the solution is x1 = 0.9938, x2 = −1.0084, x3 = −1.0078.

The exact solution is x1 = 1, x2 = −1, x3 = −1.

2. Solve the system of equations using Gauss Jacobi iteration method

10x − 5y − 2z =3
4x − 10y + 3z = − 3
x + 6y + 10z = − 3.

9
 Sol.
The given system is diagonally dominant.

1
x(k+1) = [3 + 5y (k) + 2z (k) ]
10
1
y (k+1) = [3 + 4x(k) + 3z (k) ]
10
1
z (k+1) = [−3 − x(k) − 6y (k) ], k = 0, 1, 2, . . .
10

Take the initial approximation as xi = 0, i = 1, 2, 3.

iteration x1 x2 x3
0 0 0 0
1 0.3 0.3 -0.3
2 0.39 0.33 -0.51
3 0.363 0.303 -0.537
4 0.3441 0.2841 -0.50487
5 0.33843 0.28101 -0.502449
6 0.339531 0.283911 -0.502449
7 0.3414657 0.2850797 -0.5042997
8 0.34167891 0.28529637 -0.50519319
9 0.341609547 0.285113607 -0.505345803

Hence the solution is x = 0.341, y = 0.285, z = −0.505.

2.1.3 Problems to try:

3. Solve the system of equations using Gauss-Jacobi method

8x − 3y + 2z = 20
4x + 11y − z = 33
6x + 3y + 12z = 35

4. Solve the system of equations using Gauss-Jacobi method

x1 + 10x2 − x3 =3
2x1 + 3x2 + 20x3 =7
10x1 − x2 + 2x3 =4.

10
2.2 Gauss-Seidel iteration method
(Method of successive displacement)
Assume that the pivots aii 6= 0, for all i.
Write the equation (2.1) as

a11 x1 =b1 − a12 x2 − a13 x3

a22 x2 =b2 − a21 x1 − a23 x3
a33 x3 =b3 − a31 x1 − a32 x2

The Gauss-Seidel iteration method is defined as

(k+1) 1 (k) (k)

x1 = [b1 − a12 x2 − a13 x3 ] (2.3)
a11
(k+1) 1 (k+1) (k)
x2 = [b2 − a21 x1 − a23 x3 ]
a22
(k+1) 1 (k+1) (k+1)
x3 = [b3 − a31 x1 − a32 x2 ], k = 0, 1, 2, . . .
a33

Note: Take the initials approximation as x2 = x3 = 0

2.2.1 Remark:

• A sufficient condition for the convergence of Gauss-Seidel method is that the system of equations is diago-
nally dominant,i.e.,the coefficient matrix A is diagonally dominant.If the system is not diagonally dominant,
exchange the equations,if possible,such that the new system is diagonally dominant and convergence is guar-
anteed.
• The necessary and sufficient condition for convergence is that the spectral radius of the iteration matrix
H is less than one,i.e.,ρ(H) < 1,where ρ(H) is the largest eigen value in magnitude of H.
• If both the Gauss-Jacobi and Gauss-Seidel method converge, then Gauss-Seidel method converges at least
two times faster than the Gauss-Jacobi method.

2.2.2 Problems

1. Find the solution of the system of equations correct to three decimal places,using the Gauss-Seidel
iteration method

8x − 3y + 2z = 20
4x + 11y − z = 33
6x + 3y + 12z = 35

11
 Sol.
The given system of equations is diagonally dominant.

1
x(i+1) = [20 + 3y (i) − 2z (i) ]
8
1
y (i+1) = [33 − 4x(i+1) + z (i) ]
11
1
z (i+1) = [35 − 6x(i+1) − 3y (i+1) ]
12

Take y (0) = 0, z (0) = 0.

Then

1 1
x(1) = [20 + 3y (0) − 2z (0) ] = [20 + 3(0) − 2(0)] = 2.5
8 8
1 1
y (1) = [33 − 4x(1) + z (0) ] = [33 − 4(2.5) + 0] = 2.091
11 11
1 1
z (1) = [35 − 6x(1) − 3y (1) ] = [35 − 6(2.5) − 3(2.091)] = 1.144
12 12

iteration x y z
0 0 0
1 2.5 2.091 1.144
2 2.998 2.014 0.914
3 3.027 1.982 0.908
4 3.016 1.986 0.912
5 3.107 1.986 0.912

Hence, the solution is x = 3.017, y = 1.986, z = 0.912

2. Find the solution of the system of equations correct to three decimal places,using the Gauss-Seidel
iteration method

10x − 5y − 2z =3
4x − 10y + 3z = − 3
x + 6y + 10z = − 3.

Sol.
The given system is diagonally dominant.

1
x(k+1) = [3 + 5y (k) + 2z (k) ]
10
1
y (k+1) = [3 + 4x(k+1) + 3z (k) ]
10
1
z (k+1) = [−3 − x(k+1) − 6y (k+1) ], k = 0, 1, 2, . . .
10

12
 iteration x y z
0 0 0
1 0.3 0.42 -0.582
2 0.3936 0.28284 -0.509064
3 0.3396072 0.28312368 -0.503834928
4 0.340795 0.2851675 -0.50517926
5 0.34155 0.28507 -0.505193

Hence, the solution is x = 0.341, y = 0.285, z = −0.505

2.2.3 Problems to try:

3. Solve the system of equations using Gauss-Seidal method

4x1 + x2 + x3 =2
x1 + 5x2 + 2x3 = − 6
x1 + 2x2 + 3x3 = − 4.

4. Solve the system of equations using Gauss-Seidal method

x1 + 10x2 − x3 =3
2x1 + 3x2 + 20x3 =7
10x1 − x2 + 2x3 =4.

3 LU Decomposition/Triangularization method
This method is based on the fact that a square matrix A can be factorized into the form LU,where L is
lower triangular and U is upper triangular, if all the principal minors of A are non-singular, i.e. if
a11 a12 a13
a11 a12
a11 6= 0, 6= 0, a21 a22 a23 6= 0 , etc.
a21 a22
a31 a32 a33
Consider a linear system

a11 x1 + a12 x2 + a13 x3 + · · · + a1n xn =b1

a21 x1 + a22 x2 + a23 x3 + · · · + a2n xn =b2
a31 x1 + a32 x2 + a33 x3 + · · · + a3n xn =b3
....................................
an1 x1 + an2 x2 + an3 x3 + · · · + ann xn =bn
(3.1)

13
which can be written in the form AX=B.
Let A=LU
   
l11 0 0 ... 0 u11 u12 u13 ... u1n
 l21 l22 0 ... 0 0 u22 u23 ... u2n 
   
 
   
where L=  l31 l32 l33 ... 0 , U= 0 0 u33 ... u3n 
  

 . .. .. ..   .. .. .. .. 
 .
 . . . ... . . . . ... . 
  
 
ln1 ln2 ln3 ... lnn 0 0 0 ... unn

li1 u1j + li2 u2j + · · · + lin unj = aij , j = 1(1)n, where lij = 0, j > i and uij = 0, i < j.
(3.2)

Thus there are n parameter family of solutions. To produce a unique solution, it is convenient to choose
either uii = 1(Crout’s method) or lii = 1(Doolittle’s method), i = 1(1)n.
Take uii = 1, i = 1(1)n.
Solution of (3.2) may be written as

j−1
X
lij =aij − lik ukj , i ≥ j
k=1
i−1
X
uij =(aij − lik ukj )/lij , i < j.
k=1

First column of L is identical with first column of A

i.e., li1 = ai1 , i = 1(1)n.
aij
Also, uij = l11
, j = 2(1)n.
Thus first column of L and first row of U have been determined. Now proceed to determine second column
of L and second row of U.

li2 = ai2 − li1 u12 , i = 2(1)n

u2j = (a2j − l21 u1j )/l22 , j = 3(1)n

Next third column of L followed by third row of U.Thus, for the relevant indices i and j, elements are
computed in the order
li1 , u1j ; li2 , u2j ; li3 , u3j ; . . . ; li,n−1 , un−1,j , lnn .
Thus Ax=b becomes LUx=b
=⇒ Ux=z (back substitution) Lz=b(forward substitution)
z = L−1 b, x = U −1 z
A−1 = U −1 L−1

14
3.0.1 Remark:

• The method fails if any of the diagonal elements uij or lii is zero.
• LU decomposition is guaranteed when A is positive definite (sufficient condition).

3.0.2 Problems

1. Solve the given system of equations

x1 + x2 + x3 =1
4x1 + 3x2 − x3 =6
3x1 + 5x2 + 3x3 =4.

Sol.
 
  
1 1 1 l 0 0 u u12 u13
   11   11 
4 3
 −1= l21
  l22 0 0
  u22 u23 

3 5 3 l31 l32 l33 0 0 u33

Take uii = 1
 
   
1 1 1 l 0 0 1 u12 u13 l l11 u12 l11 u13
   11    11 
4 3
 −1= l21
  l22 0  0
  1 u23 = l21
  l21 u12 + l22 l21 u13 + l22 u23 

3 5 3 l31 l32 l33 0 0 1 l31 l31 u12 + l32 l31 u13 + l32 u23 + l33

Comparing first column: l11 = 1, l21 = 4, l31 = 3

Comparing first row: u12 = 1, u13 = 1
Comparing second column: l22 = 3 − 4 = −1, l32 = 5 − 3 = 2
Comparing second row: u23 = (−1 − 4)/ − 1 = 5
Comparing third column:l33 = 3 − 3 − 10 = −10
   
1 0 0 1 1 1
   
L=4 −1
 0  , U=0
  1 5
3 2 −10 0 0 1
    
1 0 0 z 1
   1  
Lz = b =⇒ 4 −1
 0  z2 =
 
6
 
3 2 −10 z3 4

z1 =1
4z1 − z2 =6
3z1 + 2z2 − 10z3 =4

15
  
1
 
z=
 −2 

−1/2
    
1 1 1 x1 1
    
U x = z =⇒ 
0 1 5 x2  =  −2 
   
0 0 1 x3 −1/2

x1 + x2 + x3 =1
x2 + 5x3 = − 2
x3 = − 1/2
 
1
 
Hence,x = 
 1/2 

−1/2
 
3 2 1
 
2. Find the inverse of A=2
 3 , take uii = 1
2
1 2 2
Sol.
    
3 2 1 l 0 0 1 u12 u13 l l11 u12 l11 u13
   11    11 
2
 3 2 = l21
  l22 0  0
  1 u23 = l21
  l21 u12 + l22 l21 u13 + l22 u23 

1 2 2 l31 l32 l33 0 0 1 l31 l31 u12 + l32 l31 u13 + l32 u23 + l33

Comparing first column: l11 = 3, l21 = 2, l31 = 1

Comparing first row: u12 = 2/3, u13 = 1/3
Comparing second column: l22 = 3 − 4/3 = 5/3, l32 = 2 − 2/3 = 4/3
Comparing second row: u[ 23] = (2 − 2/3)/(5/3) = 4/5
Comparing third column:l33 = 2 − 1/3 − (4/5)(4/3) = 3/5
   
3 0 0 1 0 0
1
  −1
 
L=
2 5/3 0 , L = 3
−6/5
 9/5 0

1 4/3 3/5 1 −4 5

   
1 2/3 1/3 1 −2/3 1/5
  −1
 
U=0
 1 , U
4/5 = 0
 1 −4/5

0 0 1 0 0 1

     
1 −2/3 1/5 1 0 0 2 −2 1
1 1
     
A−1 = U −1 L−1 =
0 1 −4/5
 3
−6/5
 9/5 =
0 3
−2
 5 −4

0 0 1 1 −4 5 1 −4 5

16
 3.0.3 Problems to try

    
1 1 −1 x1 2
    
3. Show that LU decomposition fails to solve 2
 2 5  x2  = −3
   
 whose exact solution is
2 2 −3 x3 6
x1 = 1, x2 = 0, x3 = −1.
    
2 1 1 −2 x1 −10
    
4 0 2 1  x2 
  =  8 
 
4. Solve 
3 2 2 0  x   7 
   3  
1 3 2 1 x4 −5

17