CE 30125 - Lecture 17

LECTURE 17

DIRECT SOLUTIONS TO LINEAR SYSTEMS OF ALGEBRAIC EQUATIONS

• Solve the system of equations

AX = B



a 1 1 a 1 2 a 1 3 a 1 4 x 1 b1
a 2 1 a 2 2 a 2 3 a 2 4 x 2 b2
=
a 3 1 a 3 2 a 3 3 a 3 4 x 3 b3
a 4 1 a 4 2 a 4 3 a 4 4 x 4 b4

• The solution is formally expressed as:

X = A –1 B

p. 17.1
 CE 30125 - Lecture 2 - Fall 2004

• Typically it is more efficient to solve for X directly without solving for A –1 since
finding the inverse is an expensive (and less accurate) procedure

• Types of solution procedures

• Direct Procedures
• Exact procedures which have infinite precision (excluding roundoff error)
• Suitable when A is relatively fully populated/dense or well banded
• A predictable number of operations is required
• Indirect Procedures
• Iterative procedures
• Are appropriate when A is
• Large and sparse but not tightly banded
• Very large (since roundoff accumulates more slowly)
• Accuracy of the solution improves as the number of iterations increases

p. 2.2
 CE 30125 - Lecture 17

Cramer’s Rule - A Direct Procedure

• The components of the solution X are computed as:

Ak
x k = ---------
A

where
A k is the matrix A with its kth column replaced by vector B

A is the determinant of matrix A

• For each B vector, we must evaluate N + 1 determinants of size N where N defines the
size of the matrix A

• Evaluate a determinant as follows using the method of expansion by cofactors

N N

A =  aI j  cof  aI j   =  ai J  cof  ai J  

j=1 i=1

p. 17.3
 CE 30125 - Lecture 2 - Fall 2004

where
I = specified value of i

J = specified value of j

cof  a i j  =  – 1  i + j  minor  a i j  

minor  a i j  = determinant of the sub-matrix obtained by deleting the ith row and the
jth column

• Procedure is repeated until 2  2 matrices are established (which has a determinant by

definition):

a 1 1 a 1 2
A = = a 1 1 a 2 2 – a 2 1 a 1 2
a 2 1 a 2 2

p. 2.4
 CE 30125 - Lecture 17

Example

• Evaluate the determinant of A

a 1 1 a 1 2 a 1 3
det  A  = A = a 2 1 a 2 2 a 2 3 

a 3 1 a 3 2 a 3 3

a 2 2 a 2 3 a 2 1 a 2 3
det  A  = a 1 1  – 1   1 + 1  + a 1 2  – 1   1 + 2 
a 3 2 a 3 3 a 3 1 a 3 3

a 2 1 a 2 2
+ a 1 3  – 1   1 + 3  
a 3 1 a 3 2

det  A  = a 1 1  +1   a 2 2 a 3 3 – a 3 2 a 2 3  + a 1 2  – 1   a 2 1 a 3 3 – a 3 1 a 2 3 

+ a 1 3  +1   a 2 1 a 3 2 – a 3 1 a 2 2 

p. 17.5
 CE 30125 - Lecture 2 - Fall 2004

• Note that more efficient methods are available to compute the determinant of a matrix.
These methods are associated with alternative direct procedures.

• This evaluation of the determinant involves O  N  3 operations

• Number of operations for Cramers’ Rule O  N  4

2  2 system  O  2 4  = O  16 

4  4 system  O  4 4  = O  256 

8  8 system  O  8 4  = O  4096 

• Cramer’s rule is not a good method for very large systems!

• If A = 0 and A k  0  no solution! The matrix A is singular

• If A = 0 and A k = 0  infinite number of solutions!

p. 2.6
 CE 30125 - Lecture 17

Gauss Elimination - A Direct Procedure

• Basic concept is to produce an upper or lower triangular matrix and to then use back-
ward or forward substitution to solve for the unknowns.

Example application
• Solve the system of equations

a 1 1 a 1 2 a 1 3 x 1 b1
a 2 1 a 2 2 a 2 3 x 2 = b 2
a 3 1 a 3 2 a 3 3 x 3 b3

• Divide the first row of A and B by a 1 1 (pivot element) to get

1 a' 1 2 a' 1 3 x 1 b' 1

a 2 1 a 2 2 a 2 3 x 2 = b2
a 3 1 a 3 2 a 3 3 x 3 b3

p. 17.7
 CE 30125 - Lecture 2 - Fall 2004

• Now multiply row 1 by a 2 1 and subtract from row 2

and then multiply row 1 by a 3 1 and subtract from row 3

1 a' 1 2 a' 1 3 x 1 b' 1

0 a' 2 2 a' 2 3 x 2 = b' 2
0 a' 3 2 a' 3 3 x 3 b' 3

• Now divide row 2 by a' 2 2 (pivot element)

1 a' 1 2 a' 1 3 x 1 b' 1

0 1 a'' 2 3 x 2 = b'' 2
0 a' 3 2 a' 3 3 x 3 b' 3

p. 2.8
 CE 30125 - Lecture 17

• Now multiply row 2 by a' 3 2 and subtract from row 3 to get

1 a' 1 2 a' 1 3 x 1 b' 1

0 1 a'' 2 3 x 2 = b'' 2
0 0 a'' 3 3 x 3 b'' 3

• Finally divide row 3 by a'' 3 3 (pivot element) to complete the triangulation procedure
and results in the upper triangular matrix

1 a' 1 2 a' 1 3 x 1 b' 1

0 1 a'' 2 3 x 2 = b'' 2
0 0 1 x3 b''' 3

• We have triangularized the coefficient matrix simply by taking linear combinations of

the equations

p. 17.9
 CE 30125 - Lecture 2 - Fall 2004

• We can very conveniently solve the upper triangularized system of equations

1 a' 1 2 a' 1 3 x 1 b' 1

0 1 a'' 2 3 x 2 = b'' 2
0 0 1 x3 b''' 3

• We apply a backward substitution procedure to solve for the components of X

x 3 = b''' 3

x 2 + a'' 2 3 x 3 = b'' 2  x 2 = b'' 2 – a'' 2 3 x 3

x 1 + a' 1 2 x 2 + a' 1 3 x 3 = b' 1  x 1 = b' 1 – a' 1 2 x 2 – a' 1 3 x 3

• We can also produce a lower triangular matrix and use a forward substitution procedure

p. 2.10
 CE 30125 - Lecture 17

• Number of operations required for Gauss elimination

1
• Triangularization --- N 3
3
1
• Backward substitution --- N 2
2

• Total number of operations for Gauss elimination equals O  N  3 versus O  N  4 for

Cramer’s rule
• Therefore we save O  N  operations as compared to Cramer’s rule

Gauss-Jordan Elimination - A Direct Procedure

• Gauss Jordan elimination is an adaptation of Gauss elimination in which both elements
above and below the pivot element are cleared to zero  the entire column except the
pivot element become zeroes

1 0 0 0 x1 b 1 
0 1 0 0 x2 b 2 
=
0 0 1 0 x3 b 3 
0 0 0 1 x4 b 4 

• No backward/forward substitution is necessary

p. 17.11
 CE 30125 - Lecture 2 - Fall 2004

Matrix Inversion by Gauss-Jordan Elimination

• Given A , find A –1 such that

AA –1  I

1 0 0 0 0 0
0 1 0 0 0 0
where I = identity matrix = 0 0 1 0 0 0
0 0 0 1 0 0
0 0 0 0 1 0
0 0 0 0 0 1

• Procedure is similar to finding the solution of AX = B except that the matrix A –1

assumes the role of vector X and matrix I serves as vector B

• Therefore we perform the same operations on A and I

p. 2.12
 CE 30125 - Lecture 17

• Convert A  I through Gauss-Jordan elimination

AA –1 = I



AA –1 = I

• However through the manipulations A  A = I and therefore

IA –1 = I



A –1 = I

• The right hand side matrix, I , has been transformed into the inverted matrix

p. 17.13
 CE 30125 - Lecture 2 - Fall 2004

• Notes:
• Inverting a diagonal matrix simply involves computing reciprocals

a 11 0 0
A = 0 a 22 0
0 0 a 33

1/a 11 0 0
A –1 = 0 1/a 22 0 

0 0 1/a 33

AA –1 = I

• Inverse of the product relationship

 A 1 A 2 A 3  –1 = A 3–1 A 2–1 A 1–1

p. 2.14
 CE 30125 - Lecture 17

Gauss Elimination Type Solutions to Banded Matrices

Banded matrices
• Have non-zero entries contained within a defined number of positions to the left and
right of the diagonal (bandwidth)
INSERT FIGURE NO. 122

NxN System Compact Diagonal

x x o x o o o o o o o o o o o x x o x
x x x x o o o o o o o o o o x x x x o
o x x o x x o o o o o o o o x x o x x
x o x x o o x o o o o o x o x x o o x
o x o x o x
x o o o o o x x o x o x o
o o o x x x x o x o o o stored o x x x x o x
o o o x o x x x o x o o as x o x x x o x
o o o o o o x x x x o o o o x x x x o
o o o o o x o x x x o x x o x x x o x
o o o o o o x o x x x o x o x x x o o
o o o o o o o o x x x x o x x x x o o
o o o o o o o o x o x x x o x x o o o

halfbandwidth bandwidth
M + 1 =4 M
2
bandwidth M = 7
storage required = N2 storage required = NM

p. 17.15
 CE 30125 - Lecture 2 - Fall 2004

• Notes on banded matrices

• The advantage of banded storage mode is that we avoid storing and manipulating
zero entries outside of the defined bandwidth
• Banded matrices typically result from finite difference and finite element methods
(conversion from p.d.e.  algebraic equations)
• Compact banded storage mode can still be sparse (this is particularly true for large
finite difference and finite element problems)

Savings on storage for banded matrices

• N 2 for full storage versus NM for banded storage
where N = the size of the matrix and M = the bandwidth

• Examples:

N M full banded ratio

400 20 160,000 8,000 20
106 103 1012 109 1000

p. 2.16
 CE 30125 - Lecture 17

Savings on computations for banded matrices

• Assuming a Gauss elimination procedure
O  N 3  versus O  NM 2 
(full) (banded)

• Therefore save O  N 2 /M 2  operations since we are not manipulating all the zeros outside
of the bands!

• Examples:

N M full banded ratio

400 20 O(6.4x107) O(1.6x105) O(400)
106 103 O(1018) O(1012) O(106)

p. 17.17
 CE 30125 - Lecture 2 - Fall 2004

Symmetrical banded matrices

• Substantial savings on both storage and computations if we use a banded storage mode
• Even greater savings (both storage and computations) are possible if the matrix A is
symmetrical
• Therefore if a ij = a ji we need only store and operate on half the bandwidth in a
banded matrix (half the matrix in a full storage mode matrix)
INSERT FIGURE NO. 123a

(M + 1)/2 (M + 1)/2

store
only half

p. 2.18
 CE 30125 - Lecture 17

Alternative Compact Storage Modes for Direct Methods

• Skyline method defines an alternative compact storage procedure for symmetrical

matrices

• The skyline goes below the last non-zero element in a column

INSERT FIGURE NO. 123b

a11 a12 o a14 o o

Skyline goes above the last
a22 a23 o o o non-zero element in a column

a33 a34 o a36

a44 a45 a46

symmetrical a55 a56

a66

p. 17.19
 CE 30125 - Lecture 2 - Fall 2004

• Store all entries between skyline and diagonal into a vector as follows:
INSERT FIGURE NO. 124

A(1) A(3) o A(9) o o

A(2) A(5) A(8) o o

A(4) A(7) o A(15)

A(6) A(11) A(14)

symmetrical
A(10) A(13)

A(12)

• Accounting procedure must be able to identify the location within the matrix of
elements stored in vector mode in A  i  Store locations of diagonal terms in A  i 

1
2
MaxA = 4
6
10
12

p. 2.20
 CE 30125 - Lecture 17

• Savings in storage and computation time due to the elimination of the additional zeroes
e.g. storage savings:

full symmetrical banded skyline

N 2 = 36 M +1 7+1
-------------- N =  ------------ 6 = 24 15
 2   2 

• Program COLSOL (Bathe and Wilson) available for skyline storage solution

p. 17.21
 CE 30125 - Lecture 2 - Fall 2004

Problems with Gauss Elimination Procedures

Inaccuracies originating from the pivot elements

• The pivot element is the diagonal element which divides the associated row

• As more pivot rows are processed, the number of times a pivot element has been modi-
fied increases.

• Sometimes a pivot element can become very small compared to the rest of the elements
in the pivot row
• Pivot element will be inaccurate due to roundoff
• When the pivot element divides the rest of the pivot row, large inaccurate numbers
result across the pivot row
• Pivot row now subtracts (after being multiplied) from all rows below the pivot row,
resulting in propagation of large errors throughout the matrix!

Partial pivoting
• Always look below the pivot element and pick the row with the largest value and switch
rows

p. 2.22
 CE 30125 - Lecture 17

Complete pivoting
• Look at all columns and all rows to the right/below the pivot element and switch so that
the largest element possible is in the pivot position.

• For complete pivoting, you must change the order of the variable array

• Pivoting procedures give large diagonal elements

• minimize roundoff error

• increase accuracy

• Pivoting is not required when the matrix is diagonally dominant

• A matrix is diagonally dominant when the absolute values of the diagonal terms is
greater than the sum of the absolute values of the off diagonal terms for each row

p. 17.23