CHAPTER III

Solving Systems of Linear Equations By

Gaussian Elimination
Nabil R. Nassif and Dolly K. Fayad
March 2010

Mathematical Preliminaries

In this chapter we consider the issue of computing the solution of a system of n linear
equations in n unknowns. The scalar form of that system is as follows:

a11 x1 +a12 x2 +... +... +a1n xn = b1

a21 x1 +a22 x2 +... +... +a2n xn = b2

(S)
...
...
...
...
...

an1 x1 +an2 x2 +... +... +ann xn = bn

Written in matrix form, (S) is equivalent to:
(1)

Ax = b,

where the matrix A Rn,n , x, b Rn,1

= Rn . Specifically,

a11 a12 ... ... a1n

a21 a22 ... ... a2n

A=
... ... ... ... ...
an1 an2 ... ... ann

x1
b1
x2
b2

x=
... , b = ... .
xn
bn
We assume the basic property of linear algebra for systems of linear equations like (1).
Proposition 1 The following statements are equivalent:
1. System (1) has a unique solution.
2. det(A) 6= 0.
3. A is invertible.
In this chapter, our objective is to present the basic ideas of a linear system solver. It
consists of two main procedures allowing to solve efficiently (1).
1. The first, referred to as Gauss elimination (or reduction) reduces (1) into an
equivalent system of linear equations, which matrix is upper triangular, specifically
one shows in section 4 that
Ax = b U x = c,
where c Rn and U Rn,n is given by:

u11 u12 ...

...
u1n
0 u22 ...
...
u2n

...
...
...
...
...
U =

0
0 ... un1,n1 un1,n
0
0 ...
0
unn
1

Thus, uij = 0, i > j. Consequently, one observes that A is invertible if and only if
ni=1 uii = u11 u22 ...unn 6= 0, i.e. uii 6= 0, i.
2. The second consists in solving by back substitution the upper triangular system
(2)

U x = c.

A picture that describes the two steps of the linear solver is the following.

Input A, b Gauss Reduction Output U, c Back Substitution Output x

Our plan in this chapter is as follows. We start in section 2 by discussing issues related to
computer storage. It is followed in section 3 by the presentation of the back substitution
procedure that solves upper triangular systems, such as (2). Finally in section 4 we present
various versions of Gauss reduction, the simplest of which is Naive Gaussian elimination.

Computer Storage for matrices. Data Structures

The data storage for A and b is through one data

AG Rn,n+1 , given by:

a11 a12 ... ...

a21 a22 ... ...
AG =
... ... ... ...
an1 an2 ... ...

structure: the augmented matrix

a1n
a2n
...
ann

b1
b2

...
bn

We generally assume that the matrix A is a full matrix, that is most of its elements are
non-zero. Storing the augmented matrix AG for a full matrix in its standard form, would
then require N = n (n + 1) words of computer memory. If one uses single precision, 4N
bytes would be necessary, while using double precision would necessitate 8N bytes for that
storage.
Example 1 When the matrix size is n = 2k , the computer memory, for double precision
computation, should exceed N = 8 2k (2k + 1) O(22k+3) bytes.
The following table illustrates the magnitude of memory requirements.
k

n = 2k

N = n (n + 1)

3
6
8
10

8
64
256
1024

72
4160
65792
1049600

in Megabytes
IEEE single precision IEEE double precision
2.7 104
5.5 104
1.6 102
3.2 102
0.25
0.5
4
8
2

Practically,computer storage is usually one-dimensional. As a result, matrix elements are

either read column-wise or row-wise. In the case where the elements of the augmented matrix
AG are contiguously stored by columns, this storage would obey the following sequential
pattern:
| a11 a21 ... an1 | a12 ... an2 | ...| a1n ... ann | b1 b2 ..., bn |,
{z
} | {z }
| {z } | {z }
|
column 1 column 2
column n column n+1
while if stored by line, the storage pattern for the N augmented matrix elements becomes:
| a11 a12 ... a1n b1 | a21 ... a2n b2 | ...| an1 ... ann bn |.
|
{z
} |
{z
}
|
{z
}
line 1 1
line 2
line n
Once Gauss reduction has been applied to the original system Ax = b, the resulting upper
triangular system U x = c would necessitate the storage of the upper triangular matrix U
and the right hand side vector c. Obviously, the augmented matrix for this system is given
as follows:

u11 u12 ... ... u1n c1

0 u22 ... ... u2n c2

UG =
... ... ... ... ... ...
0 ... ... 0 unn cn
Since by default, the lower part of the matrix U consists of zeros, this part of the storage
shall not be waisted but used for other purposes, particularly that of storing the multiplying
factors, which are essential parameters to carry out Gauss elimination procedure. Hence,
at this stage we may consider the data structure U G whether stored by rows or by columns
as consisting of the elements of U and c and unused storage space:

u11
u12 ...
...
u1n c1
unused u22 ...
...
u2n c2

UG =

...
... ...
...
... ...
unused

...

...

unused

unn cn

We turn now to the Back substitution procedure.

Back Substitution for Upper Triangular Systems

Although this procedure comes after the completion of the Gauss Reduction step, we shall
deal with it from the start. It indeed provides the importance of this global approach.
Considering (2) in its scalar form, with all diagonal elements uii 6= 0, gives:

u11 u12
0 u22
... ...
0
0
0
0

...
...
u1n
...
...
u2n
...
...
...
... un1,n1 un1,n
...
0
unn

x1
x2
...
xn1
xn

c1
c2
...

=

cn1
cn

Solving (2) by the back substitution procedure reduces such procedure to solving n equations, each one in one unknown only. Starting with the last equation yields xn , then moving
backwards, substituting xn by its computed value - in appropriate rows or columns - yields
xn1 . We reiterate this procedure until the first equation is reached, and where x1 is computed.
We give two versions of the back-substitution process: the first one is column oriented,
while the second one is row oriented. We then evaluate the computational complexity of
each version. Both forms require the same number of flops.

1. Column-version: The steps are as follows:

(a) Solve the last equation for xn : xn = cn /unn
(b) In all rows above: from row (n 1) to row 1, replace xn by its computed value,
and compute the new right hand side vector that results by shifting last column
of the xn -terms. The new system to solve at this step is as follows:

u11 u12
0 u22

... ...
0
0

...
...
...
...

... u1,n1
x1
x2
... u2,n1

...
...
...
0 un1,n1
xn1

c1 u1n xn
c2 u2n xn

...
cn1 un1,n xn

This process is repeated till the 1st row is reached, where:

u11 x1 = c1 u1,n xn u1,n1 xn1 ... u12 x2
leading thus to the value of x1 = (c1 u1,n xn u1,n1 xn1 ... u12 x2 )/u11 .
The corresponding pseudocode is as follows:
% U is an upper-triangular invertible matrix
% Storage is column oriented
function x = ColBackSubstitution(U,c)
4

n=length(c) ;
for j=n:-1:1
x(j)=c(j)/U(j,j);
% In all rows above (1(j-1)) replace x(j) by its computed value, shift
the x(j)- terms to the right hand side. Calculate the new right hand side.
for i=1:j-1
c(i)=c(i)- U(i,j) * x(j) ;
end
end
The number of flops needed to execute this algorithm is calculated as follows:
For every j = 1 to n: 1 Division is needed to compute x(j); therefore, a total of n
flops.
For every j = 1 to n, and for every i = 1 to (j 1), to compute the new right hand
side : 1 Addition + 1 Multiplication are used, i.e. a total of:
j1
n X
X

2=

j=1 i=1

n
X

2[(j 1) 1 + 1] = 2(1 + 2 + ... + (n 1)) = n(n 1) = n2 n

j=1

Therefore, the total number of flops needed is: n + n(n 1) = n2 .

As for the 2nd version, the rows are successively and completely solved, starting with
the nth one.
2. Row-version:
% U is an upper-triangular invertible matrix
% Storage is row oriented
function x = RowBackSubstitution(U,c)
n=length(c) :
x(n)=c(n)/U(n,n);
for i=n-1:-1:1
for j=i+1:n
c(i)=c(i)-x(j)*U(i,j);
end
x(i)=c(i)/U(i,i);
end
The total number of flops needed to execute this algorithm, as in the column version, is
2

n.
5

Gauss Reduction

Our starting point is to assume ideal mathematical conditions allowing to carry the reduction without any safeguard. Before setting formally these assumptions, we work out the
following example:
Example 2 Consider the reduction into an upper triangular form of the system:

6x1
2x2 +2x3 +4x4
= 16

12x1 8x2 +6x3 +10x4 = 26

(3)
3x1 13x2 +9x3 +3x4 = 19

6x1 +4x2 +x3 18x4 = 34

The augmented matrix for (3) can

6
12

3
6

be easily derived to yield:

2 2 4
16
8 6 10
26

13 9 3 19
4 1 18 34

We proceed by applying successively, 3 Gauss reductions. In each of these reductions,

the following linear algebra elementary operation is being used: for a fixed k, k = 1, ..., n 1,
and for i = k + 1, ..., n
(4)

(New) Equ i (Previous) Equ i (multiplier) Pivot Equ k, i 6= k,

1. Reduction 1. The pivot equation is the 1st equation (k = 1), the pivot element is
a11 = 6. The aim in this step is to eliminate x1 from equations i, where i = 2, 3,
4, by creating 0 as its coefficient. The respective multipliers for each equation i are
1i
= 2, 12 , 1}. Thus, performing (4) repeatedly:
{ aa11
Equation 2 Equation 2 2 Pivot Equation 1,
1
Pivot Equation 1,
2
Equation 4 Equation 4 (1) Pivot Equation 1,
Equation 3 Equation 3

These operations yield the new augmented matrix:

6 2 2 4
16
0 4 2 2
6

0 12 8 1 27
0 2 3 14 18

In order not to waste the implicitly zero storage locations, we use these locations to
place the multipliers. Hence, at the accomplishment of this reduction, the augmented
matrix takes the form:

6
2 2 4
16
2
4 2 2
6

1/2 12 8 1 27
-1
2 3 14 18
with the understanding that boxed elements are the multipliers used in this 1st
reduction.
2. Reduction 2. Perform repeatedly operation(4) with the second pivot equation
(k = 2), the pivot element being a22 = 4, and i successively 3,4. The multipliers are
2i
= 3, 12 }. Thus, to create 0 as coefficient for x2 :
respectively { aa22
Equation 3 Equation 3 3 Equation 2,

The 2nd

1
Equation 4 Equation 4 ( ) Equation 2,
2
reduction yields the following augmented matrix:

6 2 2 4
16
0 4 2 2
6

0 0 2 5 9 .
0 0 4 13 21

After storing the multipliers of the second reduction, the contents of the augmented
matrix updated data structure are as follows:

6
2 2 4
16
2
4 2 2
6

1/2
.
3
2
5
9

-1

-1/2

4 13 21

Finally, we come to the last reduction step.

3. Reduction 3. Perform operation(4) with the third pivot equation (k = 3), the
3i
pivot element being a33 = 2, and the sole multiplier being { aa33
= 2}, for i = 4.
Specifically:
Equation 4 Equation 4 (2) Equation 3,

yields the augmented matrix:

6 2 2 4 16
0 4 2 2 6

0 0 2 5 9 .
0 0 0 3 3
With the storage of the multipliers, the updated augmented matrix is then:

6
2
2
4 16
2
4
2
2 6

.
1/2
3
2
5
9

-1

-1/2

3 3

We may now discuss the assumptions leading to the Naive Gauss elimination.

4.1

Naive Gauss Elimination

It is the simplest form of Gaussian elimination. The adjective naive applies because this form
is not usually suitable for automatic computation unless essential modifications are made.
We give first the conditions that allow the procedure to work out.
Definition 1 A square matrix An has the principal minor property, if all its principal
sub-matrices Ai , i = 1, ..., n are invertible.

a11 a12 ... ... a1i

a21 a22 ... ... a2i

Ai =
... ... ... ... ...
ai1 ai2 ... ... aii
If a matrix A verifies Definition 1 then b Rn,1 , the following algorithms can be applied on
the augmented matrix [A|b]. The first one assumes that the matrix A is stored column-wise.
% The algorithm is column oriented
% The matrix A is assumed to have the principal minor property
% At reduction k, the kth equation is the pivot equation and A(k,k) is
% the pivot element, and equations 1,..,k remained unchanged
function[c,U]=naiveGauss(A,b)
n=length(b) ;
for k=1:n-1
% Get the pivot element and the multipliers proceeding by columns
piv=A(k,k);
for i=k+1:n
8

A(i,k)=A(i,k)/piv;
end
% Modify the body of matrix A proceeding by columns
for j=k+1:n
for i=k+1:n
A(i,j)=A(i,j)-A(i,k)*A(k,j);
end
end
% Modify the right hand side b
for i=k+1:n
b(i)=b(i)-A(i,k)*b(k);
end
end
% Extract U and c proceeding by columns
c=b;
for j=1:n
for i=1:j
U(i,j)=A(i,j);
end
end
It can be shown in case the matrix A verifies Definition 1, that the pivot element at each
reduction is well located on the main diagonal.
The operations count for this algorithm can be easily verified:
1. To find the multipliers:
n1 X
n
X

1=

k=1 i=k+1

n1
X

n k = 1 + 2 + ... + (n 1) =

k=1

n(n 1)
divisions
2

2. To modify the body of the matrix:

n1 X
n
n
X
X

2=

k=1 j=k+1 i=k+1

=2

n1 X
n
X

2(n k) = 2

k=1 j=k+1

n(n1)(2n1)
6

n1
X

(n k)2 = 2[12 + 22 + ... + (n 1)2 ]

k=1

3. To to change the right hand side vector:

n1 X
n
X

n1
X

n(n 1)
2=2
n k = 2[1 + 2 + ... + (n 1)] = 2
2
k=1 i=k+1
k=1
9

In term of flops, this would total:

n(n 1) n(n 1)(2n 1)
n(n 1)
2n3
+
+ n(n 1) =
(7 + 4n) = O(
).
2
3
6
3
The next version requires the same number of flops and is row oriented
% The algorithm is row oriented
% The matrix A is assumed to have the principal minor property
% At reduction k, the kth equation is the pivot equation and A(k,k)
% is the pivot element, and equations 1,..,k remained unchanged
function[c,U]=naiveGauss(A,b)
n=length(b) ;
for k=1:n-1
% Get the pivot element
piv=A(k,k);
% Proceed by row: get the multiplier for equation i
for i=k+1:n
A(i,k)=A(i,k)/piv;
% and modify its remaining coefficients, then its right hand side
for j=k+1:n
A(i,j)=A(i,j)-A(i,k)*A(k,j);
end
b(i)=b(i)-A(i,k)*b(k);
end
end
% Extract U and c
c=b;
for i=1:n
c(i)=b(i);
for j=i:n
U(i,j)=A(i,j);
end
end
The above 2 versions are the simplest expressions of Gaussian elimination. They do not take
into account the eventual sensitivity of the system to propagate round-off errors.

4.2

Partial Pivoting strategies

There are several situations where the application of the above algorithm fails although
the matrix A may verify the principal minor property; for example, if the pivot element is
relatively small, this would lead to large multipliers that worsen the round-off errors.To that
end, consider the following example on a linear system of 2 equations in 2 unknowns:
10

Example 3 Naive Gaussian elimination fails.


x1 + x2 = 1
(5)
x1 + x2 = 2
where  is a small number different from 0.
The exact solution to this problem is x1 1 and x2 1. Naive Gauss elimination where the
pivot is  leads to:

x1 + x2 = 1
(1 1 )x2 = 2 1
Finally, the back substitution procedure would give:
(
21/
x2 = 11/
2
x1 = 1x

Now 1/ will be large, and so if this calculation is performed by a computer that has a
fixed word length, then for a small values of , both (2 1/) and (1 1/) would be 1/.
Therefore, the computed values would be: x2 1 and x1 0.
On the other hand, if we perform a permutation of the equations before the reduction
then the system becomes:

x1 + x2 = 2
x1 + x2 = 1
Carried out, Gauss reduction would lead to:

x1
+x2
=2
(1 )x2 = 1 2
Back substitution in this case would clearly lead to x2 1 and x1 1.
From the previous example, it is clear that some type of strategy is needed for selecting
new pivots at each stage in Gaussian elimination. Perhaps the best approach is complete
pivoting that involves at each reduction, searches over all the entries in the sub-matrices for
the largest entry in absolute value and then interchanging rows and columns to move that
entry into the pivot position. This would be quite expensive since it involves a great amount
of searching and data movement. However, searching just the 1st column in the sub-matrix
at each reduction accomplishes most of what is needed , thus avoiding small or zero pivots.
This is partial pivoting. It is the most common approach, however it does not involve
an examination of the elements in the rows, since it only looks at column entries. A more
elaborated version would be the scaled partial pivoting, where we advocate a strategy
that stimulates a scaling of the row vectors and then selects as a pivot element the relatively
largest absolute value entry in a column. This process would load balance the entries of the
matrix.
11

Also, rather than interchanging rows to move the desired element into the pivot position, we
use an indexing array to avoid the data movement. Thus, the order in which the equations
are employed is denoted by the row vector IV called the Index Vector. At first, IV is set to
[1, 2, ..., n], then at each reduction, if there would be a permutation in the rows, it is done
only on IV which acts as a vector of pointers to the memory location of the rows. At each
reduction, IV=[i1 , i2 , ..., in ], where the i0j s are integers from 1 to n, in a possibly different order. This eliminates the time consuming and unnecessary process of moving the coefficients
of equations around in the computer memory.
1. Gaussian Elimination with Partial Pivoting
Partial Pivoting consists in first finding at each reduction k, the maximum absolute
value element in column k starting from row k to row n.
- At reduction k = 1 , seek i1 in the set {1, 2, ..., n} such that:
|ai1 ,1 | = max |ai1 | = max {|a11 |, |a21 |, ..., |an1 |}
1in

then perform a permutation of row 1 and row i1 in IV only. Row i1 is the first
Pivot Equation, and ai1,1 is the Pivot Element. We write IV ([1, i1 ]) = IV ([i1 , 1]),
meaning that at this stage,
IV = [i1 , 2, ..., 1..n]
- At reduction k, seek ik in {k, ..., n}, such that:
|aik ,k | = max |aik | = max {|ak,k |, |ak+1,k |, ..., |an,k |}
kin

then perform a permutation of row k and row ik in the last IV. Row ik is
the the Pivot Equation and aik,k is the Pivot element. Therefore, one writes:
IV ([k, ik ]) = IV ([ik , k]).
- At reduction (n-1), seek in1 in {n 1, n}, such that:
|ain1 ,n1 | = max |ai,n1 | = max {|an1,n |, |an,n |}
n1in

then perform a permutation of rows (n 1) and in1 in IV.

As such the Naive Gauss Elimination is modified as follows:

12

% The algorithm is column oriented

% At reduction k, a search is made in the kth column (in rows k to n)
% to find the maximum absolute value column element (p=max)
for k=1:n-1
[p,ik]=max(abs(A(k:n,k)));
% Permutation of rows k and ik is then performed
A([k ik])=A([ik k]);
piv=A(k,k);
....................
If an index vector is referred to, then the algorithm proceeds as follows.
% An index vector is used to keep track of the location of the rows
IV=1:n
%At reduction k, a search is made to find
% the absolute value maximum column element
for k=1:n-1
[p ik]=max(abs(A(IV(k:n),k)));
% find the position of ik in last IV
ik=ik + k - 1 ;
% Permutation of rows k and ik is then performed through the index vector
IV([k ik])=IV([ik k]);
% Identify the pivots
Piv=A(IV(k),k);
% Find the multipliers
for i=k+1:n
A(IV(i),k)=A(IV(i),k)/piv;
end
% Modify the body of matrix A and right hand side b
for j=k+1:n
for i=k+1:n
A(IV(i),j)=A(IV(i),j)-A(IV(i),k)*A(IV(k),j);
end
end
for i=k+1:n
b(IV(i))=b(IV(i))-A(IV(i),k)*b(IV(k));
end
Extract U,b
We turn now to the final version of Gauss Elimination that introduces a scale for each
equation.

13

2. Gaussian Elimination with scaled Partial Pivoting

In this strategy, besides the index vector IV that is created to keep track of the equationpermutations of the system, a scale factor must be computed for each equation. We
define the maximum element of each row si , such that:
si = max {|aij |} ; 1 i n
1jn

These n numbers are recorded in the scale vector s = [s1 , s2 , ..., sn ]T .

In starting the forward elimination process, we do not arbitrarily use the first equation
as the pivot equation as in the Naive-Gauss elimination, nor do we select the row with
maximum absolute value in the entries of the first column, as in the Partial Pivoting
strategy. Instead we scan first in column 1 the ratios


|ai,1 |
, i = 1, ..., n
si
and select the equation (or row) for which this ratio is greatest. Let i1 be the 1st index
for which the ratio is greatest, then:


|ai1 ,1 |
|ai,1 |
= max
1in
si 1
si
Interchange i1 and 1 in the index vector only, which is now stored as IV=[i1 , i2 , ...in ].
So, Row i1 is the 1st pivot equation, and the pivot is then ai1 ,1 . In a similar way,
proceed next to the 2nd reduction.
Notice that through this procedure, the scale factors are computed once. They are
not changed after each pivot step.
We give now a new version of the newly devised algorithm.
% Initialize IV
IV=[1,2,...,n]
% Seek the scales
for i=1:n
s(i)=max(abs(A(i,1:n))
end
% At reduction k, a search is made to find the absolute value
maximum scaled column element
for k=1:n-1
[p ik]=max(abs(A(IV(k:n),k)/s(IV(k:n)) ;
14

ik=ik+k-1;
IV([k ik])= IV([ik k]) ;
.........Same as Gauss Partial Pivoting..............
3. Example 4
We first set the index vector to IV = [1 2 3 4], and evaluate the scale of the system
IV
1
2
3
4

Augmented matrix
Scales
3 13 9 3 19
13
6 4 1 18 34
18
6 2 2 4
16
6
12 8 6 10
26
12

(a) Reduction 1 Seek the pivot equation:

max {3/13, 6/18, 1, 1}.
First occurrence of the maximum is the 3rd one, i.e. at IV(3)=3 (meaning that
the 3rd component of IV is equation 3). So perform the permutation of rows 1
and 3 through the index vector, which becomes IV = [3 , 2 , 1 , 4]. The pivot
equation is equation 3 and the pivot element is 6. Computing the multipliers
ai,1 /6 , i = 2, 3, 4 of the permutated system would give:
IV
3
2
1
4

Augmented
1/2 13 9
-1
4 1
6
2 2
2
8 6

matrix
Scales
3 19
13
18 34
18
4
16
6
10
26
12

Modifying the body of matrix and right hand side leads to:
IV
3
2
1
4

Augmented
1/2 12 8
-1
2 3
6
2 2
2
4 2

15

matrix
Scales
1 27
13
14 18
18
4
16
6
2
6
12

(b) Reduction 2 Similarly to reduction 1, one starts with a search for the pivot
equation:
{

max
IV (2),IV (3),IV (4)

|aIV (2),2 | |aIV (3),2 | |aIV (4),2 |

,
,
} = max {2/18, 12/13, 4/12}.
sIV (2)
sIV (3)
sIV (4)

The maximum 12/13 occurs at IV (3) = 1. Hence we perform the permutation

of Equations IV (2) = 2 and IV (3) = 1. Thus the index vector becomes IV =
[3 , 1 , 2 , 4], the pivot equation is equation 1 and the pivot element is 12.
Computing the multipliers and proceeding into the modifications of the remaining
part of the augmented matrix leads to the following profile of the global data
structure:
IV
Augmented matrix
Scales
2
1/2 12
8
1
27
13
3
1
4

-1
6
2

-1/6
2
1/3

13/3 83/6 45/2

2
4
16
2/3 5/3
3

18
6
12

 13
2
|; | 312
| =
(c) Reduction 3 This last step keeps the index vector unchanged since max | 318
13
. It is easily verified at the end of the process the contents of the data struc318
ture are as follows.
IV
2

1/2

3
1
4

-1
6
2

Augmented matrix
12
8
1
-1/6
2
1/3

13/3
2
-2/13

27

83/6 45/2
4
16
6/13 6/13

Scales
13
18
6
12

Obviously, back substitution yields:

x4 = 1, x3 = 2, x2 = 1, x1 = 3.

4.3

LU Decomposition

A major by-Product of Gauss Elimination is the decomposition of a matrix into the product
of a unit lower triangular matrix by an upper triangular one. We will base our arguments
on examples 2 and 4.
1. First case : Naive Gauss
Example 2
Going back to example 2, and on the basis of the multipliers of Gauss elimination, let
L and U be respectively the unit lower and upper triangular matrices:
16

1
0
2
1
L=
1
3
2
1 12
Note that the product LU

1
0 0 0
2
1 0 0
1
(6)

3 1 0
2
1 12 2 1

0
0

0
1

0
0
1
2

verifies:

6
0

0
0

2
4
0
0

6 2 2 4
0 4 2 2

U =
0 0 2 5
0 0 0 3

2 4
6 2 2 4
12 8 6 10
2 2
=
2 5 3 13 9 3
0 3
6 4 1 18

which is precisely:
LU = A.
This identity obeys to the following theorem ([2], [3]):
Theorem 1 If a square matrix A Rn,n verifies the principal minor property, then
it can be decomposed into the product of a unit lower triangular matrix L by an upper
triangular matrix U , such that A = LU , where L and U are derived from the Gaussian
elimination process; L is the lower triangular matrix obtained from the multipliers at
each reduction,and U is the final upper triangular matrix of the process.
2. Second case : Gauss Partial Pivoting
Example 4
Consider now the final stage of Gauss reduction in example 4. According to the index
vector IV , we extract successively the upper triangular matrix

6 2
2
4
0 12

8
1

U =
0 0 13/3 83/6
0 0
0
6/13
and the lower unit triangular one,

1
0
0
1/2
1
0
L=
1 1/6
1
2
1/3 2/13
Computation of the product LU

1
0
0
0
1/2

1
0
0

.
1 1/6

1
0
2
1/3 2/13 1

0
0

0
1

gives:

6 2
2
4
6 2 2 4
3 13 9 3
0 12
8
1
=
0 0 13/3 83/6 6 4 1 18
0 0
0
6/13
12 8 6 10
17

The product matrix is the matrix A up to a permutation matrix, P = P (IV ),

associated with the final status of the index vector . We write then LU = P (IV )A,
where the Permutation matrix P is defined as follows.
Definition 2 Let I Rn,n , be the identity matrix defined by its rows, the unit vectors
of Rn {e1 = (1, 0, ...0), 2 = (0, 1, 0, ..., 0), en = (0, 0, ..., 0, 1)}, with

e1
e2

I=
...
en
Let IV = [i1 , i2 , ..., in ], where {i1 , i2 , ..., in } a permutation of the integers {1, 2, ..., n}.
The permutation matrix P associated with IV is a permutation of the identity matrix
I, and is given by the row matrix:

ei1
ei2

P (IV ) =
...
ein
In example 4, the index vector final status is IV = [3, 1, 2, 4]. Thus,

e3
0 0 1 0
e1 1 0 0 0

P = P (IV ) =
e2 = 0 1 0 0
e4
0 0 0 1
Note then that the product:

0
1
PA =
0
0

0
0
1
0

1
0
0
0

0
3 13 9 3

0
6 4 1 18
0 6 2 2 4
1
12 8 6 10

is precisely the product LU found above. Hence the general LU decomposition theorem
which generalizes theorem 1 stands as follows.
Theorem 2 Let a square matrix An be processed through partial pivoting Gauss reduction.
If the unit lower triangular matrix L , the upper triangular matrix U and the index vector
IV are extracted from the final status of the process then:
P (IV )A = LU,
18

where P (IV ) is the permutation matrix associated with the last status of the index vector
IV .
Within the scope of this course, the LU decomposition or factorization of A is particularly helpful in computing the determinant of A, in solving different systems of equations
Ax = b, where the coefficient matrix A is held constant, or also in computing the inverse of A,

4.4

Computing the Determinant of a Matrix

Clearly, from theorems 1 and 2, we conclude respectively that in the first case:
det(A) = det(L) det(U )
while in the second case
det(P ) det(A) = det(L) det(U )
where P = P (IV ). Obviously, det(L) = 1, and det(P ) = (1)s , where s is the number of
permutations needed to transform the last IV back to its initial value [1, 2, ..., n].
These results are formulated as follows:
Theorem 3

(a) Under the hypothesis of Theorem 1,

det(A) =

n
Y

uii ,

i=1

where uii , , i = 1, ..., n are the diagonal elements of the upper triangular matrix U.
(b) Under the hypothesis of Theorem 2,
det(A) = (1)s

n
Y

uii ,

i=1

where s is the number of permutations needed to transform the last index vector IV
obtained, back to its initial status [1, 2, ..., n].
One easily finds in example 2 that:
det(A) = 6 (4) 2 (3) = 144,
and in example 4,
det(P ) det(A) = 6 (12) 13/3 (6/13) = 144
Since s = 2 and det(P ) = 1.

19

4.5

Solving Linear Systems using LU Factorization

The LU decomposition of a matrix is particularly useful for solving systems of equations

that involve the same coefficient matrix A, and different right hand side vectors b.
We consider again successively 2 different cases.
1. First case : NaIve Gauss
Under the hypothesis of Theorem 1, once that the LU factorization is available, we
can solve the system Ax = b , by writing instead:
LU x = b
Letting U x = y, we solve then successively 2 triangular systems:
(i) The Lower triangular system Ly = b, where the vector y is obtained by forward
substitution.
(ii) The Upper triangular system U x = y, where the required vector x is obtained by
backward substitution.
Example 5

2 1 2
Let A = 2 3 3 be a matrix satisfying the Principal minor property.
6 1 8
The Nave Gauss elimination process leads to the LU

1 0 0
L = 1 1 0 and U =
3 1 1

decomposition of A, where:

2 1 2
0 2 1
0 0 3

Using this LU decomposition, to solve the systems of equations Ax = bi , with b1 =

[2, 5, 0]T and b2 = [1, 1, 3]T , we solve successively the 2 triangular systems:

Ly = b
Ux = y

1 0 0
y1
2
(a) 1 1 0 y2 = 5
3 1 1
y3
0
By Forward substitution, one obtains: y1 = 2, y2 = 3, y3 = 3

20

2 1 2
x1
2
(b) 0 2 1 x2 = 3
0 0 3
x3
3
By Backward substitution, one obtains: x1 = 1, x2 = 2, x3 = 1.
Changing the right-hand-side, we solve successively:

1 0 0
y1
1
(a) 1 1 0 y2 = 1
3 1 1
y3
3
By Forward substitution, one obtains: y1 = 1, y2 = 0, y3 = 0.

2 1 2
x1
1

0 2 1
x2
0
=
(b)
0 0 3
x3
0
By Backward substitution, one obtains: x1 = 1/2, x2 = 0, x3 = 0.
2. Second case : Gauss Partial Pivoting
Under the hypothesis of Theorem 2, solving Ax = b is equivalent to solving P Ax = P b
or equivalently:
LU x = P b,
Letting P b = c and U x = y, leads to solving successively 2 triangular systems, as in
the preceding situation, i.e. :
Ly = c followed by U x = y.
Example 6

1 1 2
Let A = 2 1 1 .
4 1 2
decomposition of A, where:

1
0

1/2 1
L=
2 2

The Scaled Partial pivoting process leads to the following LU

0
2
1
1
0 1 0
0 ; U = 0 1/2 3/2 ; P = 1 0 0
1
0
0
3
0 0 1

Using this LU decomposition, to solve the systems of equations Ax = bi , with b1 =

[2, 5, 0]T and b2 = [1, 1, 3]T , we solve successively the 2 triangular systems:
21

Ly = P b
Ux = y

1
0 0
y1
5
1. 1/2 1 0 y2 = 2
2 2 1
y3
0
By Forward substitution: y1 = 5, y2 = 9/2, y3 = 19

2
1
1
x1
5
2. 0 1/2 3/2 x2 = 9/2
0
0
3
x3
19
By Backward substitution: x1 = 131/6, x2 = 10, x3 = 19/3.
Changing the right-hand-side, we solve successively:

1
0 0
y1
1
1. 1/2 1 0 y2 = 1
2 2 1
y3
3
By Forward substitution: y1 = 1, y2 = 3/2, y3 = 8.

1
2
1
1
x1
2. 0 1/2 3/2 x2 = 3/2
8
0
0
3
x3

By Backward substitution: x1 = 2/3, x2 = 5, x3 = 8/3.

4.6

Computing the Inverse of A

Let A1 be the inverse matrix of A. Then, if I is the identity matrix,

AA1 = I
The above identity is equivalently written as:
A[c1 c2 ... cn ] = [e1 e2 ... en ]
where cj and ej , for j = 1, ..., n are respectively the
22

column vectors

of A1 and I.

1. Under the hypothesis of Theorem 1, since

LU = A, then (LU )A1 = I
To obtain A1 , it is therefore enough to solve the following n systems of equations
LU cj = ej , j = 1 : n
as detailed in the preceding section.
2. Under the hypothesis of Theorem 2, since
LU = P A
one has:
LU (A1 ) = P A(A1 )
that is:
LU (A1 ) = P
To find A1 one needs then to solve successively the following n-systems of n equations:
LU cj = pj , j = 1 : n,
where pj is the jth column of the permutation matrix P .
Remark 1 Note that in Definition 2, the Permutation matrix P is defined in terms of its
rows, while in the process of computing A1 , one has first to identify the columns of P.
Example 7

2 4 2
Let A = 1 3 4 be a given invertible matrix . The Scaled Partial Pivoting Gauss
5 2 0
elimination process leads to the LU decomposition of A, where:

1
0
0
5
2
0
1
0 and U = 0 16/5 2
L = 2/5
1/5 13/16 1
0
0
45/8
The last status of the index vector being IV = [3, 1, 2], the rows of the permutation matrix
P are the row vectors e3 , e1 , e2 and P is then:

0 0 1
P = 1 0 0
0 1 0
with column vectors: e2 , e3 , e1 .
Thus to find A1 one needs to solve successively the following 3-systems of equations:
LU c1 = e2 , LU c2 = e3 , LU c3 = e1
23

1. LU c1 = e2
Letting U c1 = y, we solve successively the 2 triangular systems:

Ly = [0, 1, 0]T
U c1 = y
Forward then Backward substitutions lead to c1 = [4/45, 2/9, 13/90]T
2. LU c2 = e3
Letting U c2 = y, we solve successively the 2 triangular systems:

Ly = [0, 0, 1]T
U c2 = y
Forward then Backward substitutions lead to c2 = [2/45, 1/9, 8/45]T
3. LU c3 = e1
Letting U c3 = y, we solve successively the 2 triangular systems:

Ly = [1, 0, 0]T
U c3 = y
Forward then Backward substitutions lead to c3 = [11/45, 1/9, 1/45]T
The inverse matrix is therefore as follows:

4/45 2/45 11/45

1/9
1/9
A1 = 2/9
13/90 8/45 1/45
One verifies that
AA1 = A1 A = I

24

PROBLEMS
1. Solve each of the following systems using Naive Gaussian elimination- Carry four significant figures.

3x1 + 4x2 + 3x3 = 10

x1 + 5x2 x3 = 7
(a)

6x1 + 3x3 + 7x3 = 15

3x1 + 2x2 5x3 = 0

2x1 3x2 + x3 = 0
(b)

x1 + 4x2 x3 = 4

1 1 2 1
x1
1
3 2 1 4 x2 1

(c)
5 8 6 3 x3 = 1
4 2 5 3
x4
1

3x1 + 2x2 x3 = 7
5x1 + 3x2 + 2x3 = 4
(d)

x1 + x2 3x3 = 1

x1 + 3x2 + 2x3 + x4 = 2

4x1 + 2x2 + x3 + 2x4 = 2

(e)
2x1 + x2 + 2x3 + 3x4 = 1

x1 + 2x2 + 4x3 + x4 = 1
2. Solve each of the following systems using Gaussian elimination with partial pivoting
and scaled partial pivoting. Carry four significant figures. What are the contents of
the index array at each step?

3x1 + 4x2 + 3x3 = 10

x1 + 5x2 x3 = 7
(a)

6x1 + 3x3 + 7x3 = 15

3x1 + 2x2 5x3 = 0

2x1 3x2 + x3 = 0
(b)

x1 + 4x2 x3 = 4

1 1 2 1
x1
1
3 2 1 4 x2 1

(c)
5 8 6 3 x3 = 1
4 2 5 3
x4
1

3x1 + 2x2 x3 = 7
5x1 + 3x2 + 2x3 = 4
(d)

x1 + x2 3x3 = 1

25


x1 + 3x2 + 2x3 + x4 = 2

4x1 + 2x2 + x3 + 2x4 = 2

(e)
2x1 + x2 + 2x3 + 3x4 = 1

x1 + 2x2 + 4x3 + x4 = 1
3. Using scaled partial pivoting, show how the computer would solve this system of equations. Show the scale array, tell how the pivot rows are selected, and carry out the
computations. Include the index array for each step. You should follow exactly the
scaled-partial-pivoting code, except that you can include the right-hand side of the
system in your calculations , when necessary, as you go along.

2x1 x2 +3x3 +7x4 = 15

4x1 +4x2
+7x4 = 11
(a)
2x1 +x2 +x3 +3x4 = 7

6x1 +5x2 +4x3 +17x4 = 31

2 3 4 1
1 1 0 2

(b)
3 3
4
3
4 1
0
4

2
1 1 2
x1
(c) 2 1 1 x2 = 2
1
4 1 2
x3

1 0 3 0
0 1 3 1

(d)
3 3 0 6
0 2 4 6
.

4 7
3
2
(e) 1 3
2 4 1

8 1 4 9 2
1
0 3 9 7

(f ) 5 0 1 3 5

4
3 2 2 7
3
0 0 0 9

2x1 + 4x2 2x3 = 6

x1 + 3x2 + 4x3 = 1
(g)

5x1 + 2x2
=2
26


x1 + x2
3x4 = 4

x1
+ 3x3 + x4 = 0
(h)
x

2 x3 x4 = 3

3x1
+ x3 + 2x4 = 1
4. For each coefficient matrix A of problems 1, 2 and 3, use the LU decomposition of A
to find the determinant oA, and the inverse of A.

27

References
[1] CHENEY, W., KINCAID D., Numerical Mathematics and Computing. 4th edition.
Brooks, Cole 1999.
[2] CIARLET, P., Analyse Numrique Matricielle et Optimisation, Masson, 1985
[3] GOLUB, G.H., VAN LOAN C.F., Matrix Computations. 2nd Edition John Hopkins
University, 1993.
[4] HIGHAM, N., Accuracy and Stability of numerical algorithms. 1996

28