Module 3: Numerical Solution of

Simultaneous Algebraic Equations

(Part 1)

Prepared by:
Engr. Ma. Grace C. Sumaria

ESCi 118 – Numerical Methods in Eng’g Problems 1st Semester AY 2020-2021

OUTLINE
Review of Matrix Concepts
System of Linear Equations
Types of Solution
Introduction to Methods
Direct Methods
-- Gauss Elimination
-- Gauss-Jordan Elimination
-- LU Decomposition
-- Inverse Method
-- Cramer’s Rule

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
MATRIX
A rectangular array of numbers or
functions arranged in rows and
columns
Designated by a capital letter
Enclosed by brackets, parentheses or
double bars

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
SIZE OF MATRIX
m x n
where m = number of rows
n = number of columns

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
ELEMENTS OF MATRIX
𝒂𝒊𝒋
where i = particular row 1≤𝑖≤𝑚
j = particular column 1 ≤ 𝑗 ≤n

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
SPECIAL TYPES OF MATRICES
Row Matrix (Row Vector)

Column Matrix (Column Vector)

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
SPECIAL TYPES OF MATRICES
Square Matrix (m=n)

• Order of Matrix (3)

• Principal (or Main) Diagonal

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
SPECIAL TYPES OF MATRICES
Upper Triangular Matrix
-- Square matrix with all elements below
the principal diagonal equal to 0
-- (𝑎𝑖𝑗 = 0 for i > j)

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
SPECIAL TYPES OF MATRICES
Lower Triangular Matrix
-- Square matrix with all elements above
the principal diagonal equal to 0
-- (𝑎𝑖𝑗 = 0 for i < j)

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
SPECIAL TYPES OF MATRICES
Diagonal Matrix
-- Square matrix that is both an upper
triangular and lower triangular matrix
-- (aij = 0 for i ≠ j)

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
SPECIAL TYPES OF MATRICES
Identity Matrix
-- Diagonal matrix where all elements
along the main diagonal are equal to 1
-- Represented by 𝐼𝑛

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
SPECIAL TYPES OF MATRICES
Null Matrix
-- Matrix where all elements are zero
-- Represented by 𝑂

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
SPECIAL TYPES OF MATRICES
Banded (or Band) Matrix
-- Matrix whose non-zero elements
are confined to particular diagonals

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
SPECIAL TYPES OF MATRICES
Tridiagonal Matrix
-- All elements not on the principal diagonal
or on the two diagonals surrounding the
principal diagonal are zero

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
SPECIAL TYPES OF MATRICES
Sparse Matrix
-- Most of the elements are zero
-- Usually seen in engineering and
scientific applications

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
MATRIX OPERATIONS
Given: 2  1  1 3
A  5 3  B   5 2
6 1   3 7

Addition of two matrices (A+B):

2  1  1 3 3 2
 5 3     5 2   0 5 
     
6 1   3 7  9 8 

Subtraction of two matrices (A-B):

2  1  1 3  1  4
5 3    5 2  10 1 
     
6 1   3 7  3  6
Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
MATRIX OPERATIONS
• Matrix Multiplication
A = m x n matrix = (3x4)
B = n x p matrix = (4x2)
A B=F= m x p matrix = (3x2)

for i = 1 to m
for j = 1 to p

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
MATRIX OPERATIONS
• Matrix Multiplication
b b12 
 c11 c12   a11 a14   11
b22 
a12 a13
c c 22   a 21 a 22 a 23 a 24   21
b
 21 b b23 
c 31 c 32   a 31 a 32 a 33 a 34   31 
b41 b42 

c11  a11b11  a12 b21  a13 b31  a14 b41

Engr. MGCSumaria
 REVIEW OF MATRIX CONCEPTS
MATRIX OPERATIONS
• Matrix Multiplication
2  1 2  4    1 5 2  9   1 2 2 1   1 4 2  3   1 6
8 3      8  4   3  5 8  9  3  2 8 1  3  4 8  3  (3  6) 
4 9 1 3
 
 5 2 4 6   
6 7     6  4   7  5 6  9  7  2 6 1  7  4 6  3  7  6 

 13 16  2  12
  17 78 20  6 
- 11 68 34 24 

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
MATRIX OPERATIONS
• Matrix Transposition
--𝑎𝑖𝑗 = 𝑎𝑗𝑖

*Transpose of a Symmetric Matrix is equal

to itself.

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
MATRIX PROPERTIES
Let O = null matrix; I = identity matrix

• Matrix Addition
-- A + O = A [Additive Identity]
-- A + (-A) = O [Additive Inverse]
-- A + B = B + A [Commutative Property]
-- (A + B) + C = [Associative Property]
A + (B + C)

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
MATRIX PROPERTIES
• Matrix Multiplication

-- A(BC) = (AB)C [Associative Property]

-- A(B + C) = AB + AC [Left Distributive]
≠ BA + CA
-- (A + B)C = AC + BC [Right Distributive]
-- AI = IA = A [Multiplicative Identity]

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
MATRIX PROPERTIES
• Scalar Multiplication
-- Ф x A = O
-- 1 x A = A
-- kI(A) = k(IA) = I(kA)
-- (k + I)A = kA + IA
-- k(A + B) = kA + kB

*where Ф = zero (scalar)

Engr. MGCSumaria
REVIEW OF MATRIX CONCEPTS
MATRIX PROPERTIES
• Matrix Transposition

-- 𝐴𝑇 𝑇
=A
-- 𝐴 + 𝐵 𝑇 = 𝐴 𝑇 + 𝐵 𝑇

-- 𝑘𝐴 𝑇 = 𝑘𝐴𝑇
-- 𝐴𝐵𝐶𝐷𝐸 𝑇 = 𝐸 𝑇 𝐷𝑇 𝐶 𝑇 𝐵𝑇 𝐴𝑇

Engr. MGCSumaria
SYSTEMS OF LINEAR EQUATIONS

In matrix notation:

Engr. MGCSumaria
SYSTEMS OF LINEAR EQUATIONS

Engr. MGCSumaria
SYSTEMS OF LINEAR EQUATIONS

ILLUSTRATION
Two equations in two variables

Engr. MGCSumaria
SYSTEMS OF LINEAR EQUATIONS

ILLUSTRATION
Three equations in three variables

Engr. MGCSumaria
TYPES OF SOLUTION

DETERMINANT
• Denoted by det(A) or |A|
• If det(A) = 0, system will have no
solution.
• Diagonal Method
-- For square matrices with order less than
or equal to three

Engr. MGCSumaria
TYPES OF SOLUTION

DETERMINANT
Method of Cofactors
-- Expansion about a row (or a column)

• Complementary Minor
-- det(Mij) or |Mij|
-- Determinant of the submatrix Mij obtained
after eliminating the ith row and jth column
of given matrix A

Engr. MGCSumaria
TYPES OF SOLUTION

DETERMINANT
Method of Cofactors
-- Expansion about a row (or a column)

• Algebraic Complement
-- 𝑨𝒊𝒋
-- Signed minor obtained from:
𝑨𝒊𝒋 = −𝟏 𝒊+𝒋 𝑴𝒊𝒋

Engr. MGCSumaria
TYPES OF SOLUTION

DETERMINANT
Method of Cofactors

*We may choose any row or any column (as

basis) in obtaining the determinant of the
matrix.

Engr. MGCSumaria
TYPES OF SOLUTION

RANK OF A MATRIX
The order of the largest square submatrix
with a nonzero determinant.
-- The minimum rank is 1 if not all elements
in the matrix are zero.

 Rank is not altered by any sequence of

elementary row (or column) operations.

Engr. MGCSumaria
TYPES OF SOLUTION

ILLUSTRATION
Two equations in two variables

Engr. MGCSumaria
TYPES OF SOLUTION

ILLUSTRATION

Engr. MGCSumaria
INTRO TO METHODS
DIRECT METHODS
Based on algebraic elimination that
obtains the solution in fixed number of
operations
1. Gauss Elimination
2. Gauss-Jordan Elimination
3. LU Decomposition
Crout’s Method
Doolittle’s Method
Cholesky’s Method
4. Inverse Method
5. Cramer’s Rule

Engr. MGCSumaria
INTRO TO METHODS
DIRECT METHODS
• Generally used when one or more of the
following condition holds:
1. Number of equations is small (100 or less)

2. Most of the coefficients 𝑎𝑖𝑗 are nonzero

3. The system of equations is not diagonally

dominant

4. The system of equations is ill-conditioned

Engr. MGCSumaria
INTRO TO METHODS
ITERATIVE METHODS
• Consist of guessing a value/solution and
then using a systematic method to obtain
improved/refined estimates

• Generally used when the number of

equations is large ( > 100)
1. Jacobi Method
2. Gauss-Seidel Method
3. Relaxation Methods (SOR)

Engr. MGCSumaria
INTRO TO METHODS
DIRECT VS. ITERATIVE
Direct Method
Obtains the solution in a fixed number of
operations based on algebraic elimination

Iterative Method
Obtains the solution asymptotically by an
iterative* procedure

*Repetition of a process usually with the aim of

approaching a desired goal or target or result.

Engr. MGCSumaria
DIRECT METHODS
GAUSS ELIMINATION METHOD

• Reduces a set of n equations in n

unknowns to an equivalent upper
triangular form

Engr. MGCSumaria
DIRECT METHODS
GAUSS ELIMINATION METHOD

Involves forward elimination and back

substitution

Engr. MGCSumaria
DIRECT METHODS
GAUSS ELIMINATION METHOD
For forward elimination, the following
elementary operations apply:
-- Interchange any two rows
-- Multiply a row by a non-zero constant k
-- Add/subtract a factor of another row to a
row

 Backward substitution is done to obtain

the solution starting from the last equation
(one unknown) going up.

Engr. MGCSumaria
DIRECT METHODS

GAUSS ELIMINATION METHOD

PROCEDURE:
Consider the following augmented matrix C:

Engr. MGCSumaria
DIRECT METHODS

GAUSS ELIMINATION METHOD

PROCEDURE:
By using (n-1) elimination steps (via
elementary row operations), matrix C is
reduced as follows:

Engr. MGCSumaria
DIRECT METHODS

GAUSS ELIMINATION METHOD

PROCEDURE:
The initial elements of 𝐶 𝐶 0 are given by

The first elimination step is applied to obtain:

Engr. MGCSumaria
 DIRECT METHODS

GAUSS ELIMINATION METHOD

PROCEDURE:
The elimination procedure is continued by using
rows 1, 2, n-1 in sequence as pivot rows with 𝑎𝑖𝑖 (𝑘)
as the pivot element in the kth elimination step

NAIVE GAUSS ELIMINATION

NAÏVE GAUSS ELIMINATION

𝑎𝑘𝑘 (𝑘−1) cannot be zero

Engr. MGCSumaria
 DIRECT METHODS

GAUSS ELIMINATION METHOD

PROCEDURE:
If the pivot element is zero, rearrange the equations
in each step to achieve diagonal dominance of the
coefficient matrix

Partial Pivoting
 Avoids the possibility of division by zero
 Improves the accuracy of the resulting solution

Engr. MGCSumaria
 DIRECT METHODS

GAUSS ELIMINATION METHOD

PROCEDURE:
At the end of (n-1) elimination steps, perform back
substitution:

Engr. MGCSumaria
 DIRECT METHODS

GAUSS ELIMINATION METHOD

HOW TO IMPROVE SOLUTION:

More significant figures

Partial or complete pivoting

Scaling of values

Engr. MGCSumaria
 DIRECT METHODS

GAUSS ELIMINATION METHOD

EXAMPLE:
Find the solution of the following equations
using Gauss Elimination Method.

Engr. MGCSumaria
 DIRECT METHODS

GAUSS ELIMINATION METHOD

EXAMPLE:
• Using the scaling process and dividing each row of
the original system of equation by the magnitude of
the largest coefficient appearing in that equation:

Engr. MGCSumaria
 DIRECT METHODS

GAUSS JORDAN METHOD

Reduces all the off-diagonal elements both below

and above the diagonal to zero

No need for back substitution

Uses the same elementary operations

Engr. MGCSumaria
 DIRECT METHODS

GAUSS JORDAN ELIMINATION METHOD

PROCEDURE:
Search the leftmost non-zero column.

Interchange the row with zero in that particular

column to that of a row that has a non-zero
element in the same column.

Normalize the pivot element 𝑎𝑖𝑖 .

Eliminate all elements below and above the pivot

element within the column.

Engr. MGCSumaria
 DIRECT METHODS

GAUSS JORDAN ELIMINATION METHOD

PROCEDURE: (Index Notation)
Initialization

Normalization

Engr. MGCSumaria
 DIRECT METHODS

GAUSS JORDAN ELIMINATION METHOD

PROCEDURE: (Index Notation)
Reduction

NOTE:
Normalization and reduction steps are to be
used with k = 1,2,…,n.

Engr. MGCSumaria
 DIRECT METHODS

LU DECOMPOSITION METHOD
Any square matrix A may be factored into a product
of lower (L) and upper (U) triangular matrices.

A = LU

Engr. MGCSumaria
 DIRECT METHODS

LU DECOMPOSITION METHOD
Illustrating the decomposition of a 3 x 3 matrix:

A = LU

Engr. MGCSumaria
 DIRECT METHODS

LU DECOMPOSITION METHOD

There are a total of (n²+n) entries (unknowns) in L

and U matrices, while there are only n² entries
(known) in A.

n additional conditions are to be imposed in order

to determine matrices L and U
-- Crout’s Method
-- Doolittle’s Method
-- Cholesky’s Method

Engr. MGCSumaria
 DIRECT METHODS

Crout’s Method
-- All diagonal elements of U are equal to 1
-- 𝑢𝑖𝑖 = 1 for i = 1, 2, …, n

Engr. MGCSumaria
 DIRECT METHODS

Doolittle’s Method
-- All diagonal elements of L are equal to 1
-- 𝑙𝑖𝑖 = 1 for i = 1, 2, …, n

Engr. MGCSumaria
 DIRECT METHODS

Cholesky’s Method
-- Useful for symmetric matrices
-- 𝑢𝑖𝑖 =𝑙𝑖𝑖 for i = 1, 2, …, n

Engr. MGCSumaria
 DIRECT METHODS

Cholesky’s Method
-- Useful for symmetric matrices
-- 𝑢𝑖𝑖 =𝑙𝑖𝑖 for i = 1, 2, …, n

Engr. MGCSumaria
 DIRECT METHODS

Given: Ax = B
(LU)x = B

By associative property,
L(Ux) = B
LY = B

Two-stage solution:
LY = B by Forward Substitution
Ux = Y by Backward Substitution

Engr. MGCSumaria
 DIRECT METHODS

LU DECOMPOSITION METHOD
If Crout’s Method is to be applied:

DECOMPOSITION

Engr. MGCSumaria
 DIRECT METHODS

LU DECOMPOSITION METHOD
If Crout’s Method is to be applied:

DECOMPOSITION
-- Compute the elements in the order

Engr. MGCSumaria
 DIRECT METHODS

LU DECOMPOSITION METHOD
If Crout’s Method is to be applied:

SOLUTION
First Stage:

Second Stage:

Engr. MGCSumaria
 DIRECT METHODS

LU DECOMPOSITION METHOD

If Cholesky’s Method [𝑈 𝑇 𝑈] is to be applied

(for symmetric matrices)

DECOMPOSITION

Engr. MGCSumaria
 DIRECT METHODS

LU DECOMPOSITION METHOD

If Cholesky’s Method [𝑈 𝑇 𝑈] is to be applied

(for symmetric matrices)

DECOMPOSITION

Engr. MGCSumaria
 DIRECT METHODS

LU DECOMPOSITION METHOD
NOTES:
LU Decomposition is more efficient than Gauss
Elimination when the solution of linear systems is to
be found with the same coefficient matrix, but with
different right-hand side vectors.

The U in Doolittle’s Method is the same as the U

found in Gauss Elimination Method.

Engr. MGCSumaria
 DIRECT METHODS

LU DECOMPOSITION METHOD
EXAMPLE:
Find the solution of the following equations using LU
Decomposition Method.

Engr. MGCSumaria
 DIRECT METHODS

INVERSE METHOD

Adjoint of a Matrix
• Transpose of the matrix of cofactors of the elements of a given
matrix A

Engr. MGCSumaria
 DIRECT METHODS

INVERSE METHOD
For a 2x2 matrix:

Engr. MGCSumaria
 DIRECT METHODS

CRAMER’S RULE
Ax = B

Let A = coefficient matrix

xi = 𝑖𝑡ℎ variable/unknown
B = right-hand side constants
𝐴𝑖 = matrix resulting from replacing the
𝑖𝑡ℎ column of A by column vector
of constants B

Engr. MGCSumaria
 DIRECT METHODS

CRAMER’S RULE
• The solution of the system of equations can be
determined by using the formula:
𝐴𝑖
𝑋𝑖 =
𝐴

• It is suggested that the determinant of the

coefficient matrix be the first to be calculated.

Engr. MGCSumaria
 DIRECT METHODS

COMPARATIVE EFFICIENCES
Gauss Elimination Method

Forward Elimination

𝟐𝒏𝟑 +𝟑𝒏𝟐 −𝟓𝒏 𝒏𝟑 −𝒏

𝑵𝒎𝟏 = 𝑵𝒂𝟏 =
𝟔 𝟑

Backward Substitution
𝒏𝟐 − 𝒏
𝑵𝒎𝟐 =
𝟐

Engr. MGCSumaria
 DIRECT METHODS

COMPARATIVE EFFICIENCES
Gauss Elimination Method

Number of multiplications & divisions (𝑵𝒎 )

𝟏 𝟑
𝑵𝒎 = 𝑵𝒎𝟏 + 𝑵𝒎𝟐 = 𝒏 + 𝟑𝒏𝟐 − 𝒏
𝟑

Number of additions & subtractions(𝑵𝒂 )

𝟏
𝑵𝒂 = 𝑵𝒂𝟏 + 𝑵𝒂𝟐 = 𝟐𝒏𝟑 + 𝟑𝒏𝟐 − 𝟓𝒏
𝟔

Engr. MGCSumaria
 DIRECT METHODS

COMPARATIVE EFFICIENCES
Gauss-Jordan Elimination Method

Number of multiplications & divisions (𝑁𝑚 )

1 3 2
1
𝑁𝑚 = 𝑛 +𝑛 − 𝑛
2 2

Number of additions & subtractions(𝑁𝑎 )

1 3 1
𝑁𝑎 = 𝑛 − 𝑛
2 2

Engr. MGCSumaria
 DIRECT METHODS

COMPARATIVE EFFICIENCES
LU Decomposition Method
Cholesky’s Method

𝑵𝟏 = 𝒏 𝑠𝑞𝑢𝑎𝑟𝑒 𝑟𝑜𝑜𝑡𝑠
𝟏 𝟑 𝟏
𝑵𝟐 = 𝒏𝟑 + 𝒏𝟐 + 𝒏 multiplications & divisions
𝟔 𝟐 𝟑
𝟏 𝟕
𝑵𝟑 = 𝒏𝟑 + 𝒏𝟐 − 𝒏 additions & subtractions
𝟔 𝟔

Engr. MGCSumaria
 DIRECT METHODS

COMPARATIVE EFFICIENCES

Engr. MGCSumaria
 DIRECT METHODS

COMPARATIVE EFFICIENCES
Inverse Method
• Computation of the adjoint matrix of A:
-- 𝑛2 determinants of order (n-1)

8
• Computation of solution:≈ 𝑛3
3
2
-- Compared to Gauss Elimination: ≈ 𝑛3
3

Engr. MGCSumaria
 DIRECT METHODS

COMPARATIVE EFFICIENCES
Cramer’s Rule
• The determination of all 𝑥𝑖 requires evaluation of
(n+1) determinants of matrices with order i

• If in terms of cofactors:
𝑛 − 1 𝑛! arithmetic operations

• Thus, the solution set will require

𝑛 − 1 (𝑛 + 1)𝑛! arithmetic operations

Engr. MGCSumaria
 DIRECT METHODS

COMPARATIVE EFFICIENCES
Notes:
• The round-off error, as well as the time required to
solve the equations, depends on the total number
of arithmetic operations performed.
-- Time required to perform
multiplication/division operation is approximately
the same
-- Time required to perform addition/subtraction
is considerably small compared to multiplication
or division

Engr. MGCSumaria
 DIRECT METHODS

COMPARATIVE EFFICIENCES
Notes:
• If the number of equations is very large, the round-
off errors might accumulate to such an extent that
the resulting solution might not be accurate.

Engr. MGCSumaria
 DIRECT METHODS

COMPARATIVE EFFICIENCES
Notes:
• If the number of equations (n) is less than 50, an
accurate solution can be found by using even
single-precision arithmetic.

• If the value of n lies between 50 and a few hundred,

it is better to use double-precision arithmetic.
However, if n is on the order of thousands,
numerical errors will result even with double
precision arithmetic.

Engr. MGCSumaria
 DIRECT METHODS

COMPARATIVE EFFICIENCES
Notes:
• Gauss Elimination Method can be applied,
theoretically, to any number of equations, but the
accuracy suffers from round-off errors when n > 50.

Engr. MGCSumaria
END

Engr. MGC Sumaria