BGE 2113/Chapter 3: Systems of Linear Equations

Prepared by N am 2011
1
CHAPTER 3: SYSTEMS OF LINEAR EQUATIONS
3.0 Introduction
Concept Mapping
System of Linear Equations
Key Term (English Bahasa Melayu)
Backward substitution - pengganntian ke belakang
Cholesky method - kaedah cholesky
Crout method - kaedah Crout
Diagonally dominant matrix - matrik dominan pepenjuru tegas
Doolittle method - kaedah Doolittle
Forward substitution - penggantian ke hadapan
Gauss Elimination Method - kaedah penghapusan Gauss
Gauss Elimination Method with Pivoting - kaedah penghapusan Gauss dengan
pemangsian separa
Gauss Seidel iteration method - kaedah lelaran Gauss Seidel
Lower triangular matrix - matrix segitiga bawah
Symmetric positive-definite matrix - matrix tentu positif simetri
Tridiagonal matrix - matrix tiga penjuru
Upper triangular matrix - matrix segitiga atas
Elimination LU Factorization Method Iteration Model
Gauss
Eliminatio
n Method
Gauss
Elimination
with Pivoting
Method
Gauss Seidel
Iteration
Method
Doolittle
Method
Crout
Method
Thomas
Method
BGE 2113/Chapter 3: Systems of Linear Equations
Prepared by N am 2011
2
A system of linear equations is a set of linear equations such as:
3 5 2
2 2 3
1 2
3 2 1
3 2 1
3 2 1
= +
= + +
= +
x x x
x x x
x x x
Solving the system is finding the values of unknows
2 1
, x x and
3
x which satisfy all three
equations simultaneously.
In general, a system with m linear equations and n unknowns can be written as
n n mn m m
n n
n n
b x a x a x a
b x a x a x a
b x a x a x a
= + + +
= + + +
= + + +
......... ..........
:
:
......... ..........
......... ..........
2 2 1 1
2 2 2 22 1 21
1 1 2 12 1 11
Or in matrix form
|
|
|
|
|
.
|

\
|
=
|
|
|
|
|
.
|

\
|
|
|
|
|
|
.
|

\
|
m n mn m mi
n
n
b
b
b
x
x
x
a a a
a a a
a a a
: :
.....
: : :
.....
.....
2
1
2
1
2
2 22 21
1 12 11
Or b Ax =
Where ,
:
,
.....
: : :
.....
.....
2
1
2
2 22 21
1 12 11
|
|
|
|
|
.
|

\
|
=
|
|
|
|
|
.
|

\
|
=
n mn m mi
n
n
x
x
x
x
a a a
a a a
a a a
A and
|
|
|
|
|
.
|

\
|
=
m
b
b
b
b
:
2
1
Or in augmented matrix form ,
:
.....
: : :
.....
.....
2
1
2
2 22 21
1 12 11
|
|
|
|
|
.
|

\
|
=
m mn m mi
n
n
b
b
b
a a a
a a a
a a a
A
For any given system of linear equations, the system has only the possible solutions:
a) Unique solution
b) Infinite many solutions
c) No solution
BGE 2113/Chapter 3: Systems of Linear Equations
Prepared by N am 2011
3
For this chapter, only the case of unique solution where matrix A must be square matrix
will be discussed.
3.1 Gauss Elimination Method
Consider a n linear equations with n unknowns
n
x x x ,....., ,
2 1
. The concept of
elimination method is eliminate all the variables
1
x below equation 1, eliminate all the
variables
2
x below equation 2, , eliminate the variable
1 n
x below equation( ) 1 n . After
the elimination process, an upper triangular system will be obtained. It can be solved by
forward substitution.
Definition
Upper Triangular System
An n n upper triangular system f Ux = is in the form of
|
|
|
|
|
.
|

\
|
m nn
n
n
f
f
f
u
u u
u u u
:
0 ...... 0
: : :
..... 0
.....
2
1
2 22
1 12 11
3.1.1 Gauss Elimination Method
Let discuss algorithm for the case of 3 linear equations with unknowns, but the concept
can be extended to n linear equations with n unknowns. For 3 linear equations with 3
unknowns as below:
) / 3 ( .
) / 2 ( .
) / 1 (
3 3 3 33 2 32 1 31
2 2 3 23 2 22 1 21
1 1 3 13 2 12 1 11
R Eq b x a x a x a
R Eq b x a x a x a
R Eq b x a x a x a
= + +
= + +
= + +
Can be written as
|
|
|
|
.
|

\
|
3
2
1
32 31
23 22 21
13 12 11
33
b
b
b
a a a
a a a
a a a
The following steps are the algorithm of Gaussian elimination method.
BGE 2113/Chapter 3: Systems of Linear Equations
Prepared by N am 2011
4
Step 1: Do the row operation below on
2
R to eliminate
1
x from
2
R .
2 2 1
11
21
R R R
a
a
+
|
|
.
|

\
|

( ) B A a a +
11 21
(Calculator Formula)
Step 2: Do the row operation below on
3
R to eliminate
1
x from
3
R .
3 3 1
11
31
R R R
a
a
+
|
|
.
|

\
|

( ) D C a a +
11 31
(Calculator Formula)
After step 1 and 2, matrix A and b will be reduced to matrix C and d as
below:
) / 6 ( .
) / 5 ( .
) / 4 (
6 3 3 33 2 32
5 2 3 23 2 22
4 1 3 13 2 12 1 11
R Eq d x c x c
R Eq d x c x c
R Eq b x a x a x a
= +
= +
= + +
Or
( )
( )
( )
6
5
4
3
2
1
32
23 22
13 12 11
/ 6
/ 5
/ 4
0
0
33
R Eq
R Eq
R Eq
d
d
b
c c
c c
a a a
|
|
|
|
.
|

\
|
Where matrix
|
|
|
.
|

\
|
=
33
32
23 22
13 12 11
0
0
c c
c c
a a a
C and
|
|
|
.
|

\
|
=
3
2
1
d
d
b
d
Note: Step 1and step 2 make the changes on (Eq 2) and (Eq 3) only.
Step 3: Do the row operation below on
6
R to eliminate
2
x from
6
R .
6 6 5
22
32
R R R
c
c
+
|
|
.
|

\
|

( ) F E c c +
22 32
(Calculator Formula)
After step 3, matrix C and d will be reduced to matrix E and f as below:
BGE 2113/Chapter 3: Systems of Linear Equations
Prepared by N am 2011
5
) / 9 ( .
) / 8 ( .
) / 7 (
9 3 3 33
8 2 3 23 2 22
7 1 3 13 2 12 1 11
R Eq f x e
R Eq d x c x c
R Eq b x a x a x a
=
= +
= + +
Or
( )
( )
( )
9
8
7
3
2
1
23 22
13 12 11
/ 9
/ 8
/ 7
0 0
0
33
R Eq
R Eq
R Eq
f
d
b
e
c c
a a a
|
|
|
|
.
|

\
|
Where matrix
|
|
|
.
|

\
|
=
33
0 0
0
23 22
13 12 11
e
c c
a a a
E and
|
|
|
.
|

\
|
=
3
2
1
f
d
b
f
Notice that step 3 make the changes on (Eq 6) only. Now matrix A is reduced to matrix E
which upper triangular matrix.
Step 4: Use back substitution to solve for x .
|
|
|
.
|

\
|
=
|
|
|
.
|

\
|
|
|
|
.
|

\
|
3
2
1
3
2
1
23 22
13 12 11
33
0 0
0
f
d
b
x
x
x
e
c c
a a a
or
|
|
|
|
.
|

\
|
3
2
1
23 22
13 12 11
33
0 0
0
f
d
b
e
c c
a a a
33
3
3 3 3 33
e
f
x f x e = =
22
3 23 2
2 2 3 23 2 22
c
x c d
x d x c x c

= = +
11
3 13 2 12 1
1 1 3 13 2 12 1 11
a
x a x a b
x b x a x a x a

= = + +
Example 3.1
Solve the system of linear equations below by using Gauss elimination method.
2 . 7 4
6 3 5
9 10 3 2
3 2 1
3 2 1
3 2 1
= + +
= +
= + +
x x x
x x x
x x x
Solution:
BGE 2113/Chapter 3: Systems of Linear Equations
Prepared by N am 2011
6
Example 3.2
Solve the system of linear equations below by using Gauss elimination method.
3 11 10
2 8 7 6
0 5 4 3
1 2
4 3
4 3 2
3 2 1
2 1
= +
= + +
= + +
= +
x x
x x x
x x x
x x
Solution:
3.1.2 Gauss Elimination Method with Pivoting
Gauss elimination with pivoting is to solve the problem of division zero in case of
11
a or
0
22
= a . So the algorithm of Gauss elimination with pivoting is almost the same Gauss
elimination except before doing elimination process to eliminate
1
x from
2
R and
3
R ,
we have to swap rows if necessary to get the maximum of
11
a if compare with the
entries below first column, and before doing elimination process to eliminate
2
x from
6
R , we have to swap rows if necessary to get the maximum of
22
a if compare with the
entries below second column.
Example 3.3
Solve the linear system below by using Gauss elimination method with pivoting,
2 . 7 4
6 3 5
9 10 3 2
3 2 1
3 2 1
3 2 1
= + +
= +
= + +
x x x
x x x
x x x
Solution:
BGE 2113/Chapter 3: Systems of Linear Equations
Prepared by N am 2011
7
Example 3.4
Solve the linear system below by using Gauss elimination method with pivoting,
3 11 10
2 8 7 6
0 5 4 3
1 2
4 3
4 3 2
3 2 1
2 1
= +
= + +
= + +
= +
x x
x x x
x x x
x x
Solution:
3.2 LU Factorization Method
For linear system
b Ax =
Use substitution of LU A = , where L is a Lower triangular matrix, and U is upper
triangular matrix. Thus,
b LUx =
Let
Y Ux =
Yields
b LY = .
Definition
Lower Triangular Matrix
A n n lower triangular matrix, L is in the form of
(
(
(
(

=
mn n n
l l l
l l
l
L
......
: :
: ....
0 .... 0
2 1
22 21
11
BGE 2113/Chapter 3: Systems of Linear Equations
Prepared by N am 2011
8
Lower Triangular Matrix
A n n upper triangular matrix, U is in the form of
(
(
(
(

=
mn
n
n
u
u u
u u u
U
0 ....... 0
: :
.... 0
....
2 22
1 12 11
Procedure
Steps of solving a linear system b Ax = by LU factorization method
Step 1: From LU A = , solve for L and U.
Step 2: From b LY = , solve for Y by forward substitution.
Step 3: From Y Ux = , solve for x by backward substitution.
There are 4 types of LU factorization methods.
1. Doolittle method where diagonal of matrix L are 1.
2. Crout method where diagonal of matrix U are 1
3. Thomas method where diagonal of matrix U are 1 and matrix A is a tridiagonal
matrix ( 0 =
ij
a if 1 > j i ). From Thomas method, we can derive algorithm which
is easier to program it to solve for a larger system.
4. Cholesky method where
T
L U = and matrix A must be symmetric positive-definite.
3.2.1 Doolittle Method
Example 3.5
Solve the system of linear equations below by Doolittle method.
|
|
|
.
|

\
|
=
|
|
|
.
|

\
|
|
|
|
.
|

\
|
2
0
3
1 4 2
5 3 0
1 2 1
3
2
1
x
x
x
Solution:
BGE 2113/Chapter 3: Systems of Linear Equations
Prepared by N am 2011
9
Example 3.6
Solve the system of linear equations below by Doolittle method.
|
|
|
|
|
.
|

\
|
=
|
|
|
|
|
.
|

\
|
|
|
|
|
|
.
|

\
|
8
7
0
3
6 5 4 3
2 3 0 9
8 7 2 5
4 3 2 1
4
3
2
1
x
x
x
x
Solution:
3.2.2 Crout Method
Crout method: diagonal of matrix U are 1.
Example 3.7
Solve the system of linear equations below by Crout method.
|
|
|
.
|

\
|
=
|
|
|
.
|

\
|
|
|
|
.
|

\
|
2
0
3
1 4 2
5 3 0
1 2 1
3
2
1
x
x
x
Solution:
BGE 2113/Chapter 3: Systems of Linear Equations
Prepared by N am 2011
10
3.3 Iteration Method:
The iteration method must satisfy convergence criteria, that A must be diagonally
dominant.
Definition
Diagonal Dominant
Matrix A is diagonally dominant if and only if . ...., , 2 , 1 ,
1
11
n i a a
n
i j
j
ij
= >

=
=
Example 3.8
Show that matrix
|
|
|
.
|

\
|

=
7 1 3
2 5 2
2 6 9
A is diagonally dominant.
Solution:
Example 3.9
Show that matrix
|
|
|
.
|

\
|

=
7 6 3
3 5 2
5 1 9
A is not diagonally dominant.
Solution:
BGE 2113/Chapter 3: Systems of Linear Equations
Prepared by N am 2011
11
Gauss Seidel Iteration Method.
Consider n equations with n unknows below:
n n nn n n n
n n
n n
n n
b x a x a x a x a
b x a x a x a x a
b x a x a x a x a
b x a x a x a x a
= + + + +
= + + + +
= + + + +
= + + + +
..........
:
:
..........
..........
..........
3 3 2 2 1 1
3 3 3 33 2 32 1 31
2 2 3 23 2 22 1 21
1 1 3 13 2 12 1 11
Algorithm for Gauss-Seidel iteration method is as below
Step 1: If necessary, rearrange the equation to achieve convergence criteria
Step 2: Write down Gauss-Seidel iteration formula
( )
( ) ( ) ( )
( )
( ) ( ) ( )
( )
( ) ( ) ( )
( )
( ) ( ) ( )
( )
( )
( )
( ) ( )
n i
a
x a x a b
x
a
x a x a x a x a b
x
a
x a x a x a b
x
a
x a x a x a b
x
a
x a x a x a b
x
ii
n
j
k
j ij
i
j
k
j ij i
k
i
nn
k
n n n
k
n
k
n
k
n n k
n
k
n n
k k
k
k
n n
k k
k
k
n n
k k
k
,....., 3 , 2 , 1 ,
......
:
:
......
......
......
1 1
1
1
1
1
1
1 1
1
3 3
1
2 2
1
1 1 1
33
3
1
2 32
1
1 31 3 1
3
22
2 3 23
1
1 21 2 1
2
11
1 3 13 2 12 1 1
1
=

=

=

=

=

=

+ =
=
=
+
+
+

+ + +
+
+ +
+
+
+
+
BGE 2113/Chapter 3: Systems of Linear Equations
Prepared by N am 2011
12
For the case of 3 equations with 3 unknowns:
( )
( ) ( )
( )
( ) ( )
( )
( ) ( )
33
1
2 32
1
1 31 3 1
3
22
3 23
1
1 21 2 1
2
11
3 13 2 12 1 1
1
a
x a x a b
x
a
x a x a b
x
a
x a x a b
x
k k
k
k k
k
k k
k
+ +
+
+
+
+

=

=

=
Step 3: Take initial guesses as
( )
( )
T
x 0 0 0
0
=
Step 4: Iterate using Gauss-Seidel iteration formula until
( ) ( )
{ }
( ) ( )
{ } < =
+
s s
+ k
i
k
i
n i
k k
x x x x
1
1
1
max max
Example 3.10
Solve the system of linear equations below by Gauss-Seidel iteration method.
2 7 4
6 3 5
9 10 3 2
3 2 1
3 2 1
3 2 1
= + +
= +
= + +
x x x
x x x
x x x
Where
( ) ( )
{ }
( ) ( )
{ } 005 . 0 max max
1
1
1
= < =
+
s s
+

k
i
k
i
n i
k k
x x x x
Solution:
Example 3.11
Solve the system of linear equations below by Gauss-Seidel iteration method.
|
|
|
.
|

\
|
=
|
|
|
.
|

\
|
|
|
|
.
|

\
|
9
6
9
3 2 8
9 2 0
0 5 1
3
2
1
x
x
x
Where
( ) ( )
{ }
( ) ( )
{ } 005 . 0 max max
1
1
1
= < =
+
s s
+

k
i
k
i
n i
k k
x x x x
Solution: