MATHx6101 LU Factorization 2024

1 Gaussian Elimination
We begin by giving a traditional description of Gaussian elimination (GE) for solving a linear
system Ax = b, where A ∈ Rn×n is nonsingular.
The aim of GE is to reduce a problem we can’t solve (a full linear system) to one that we
can (a triangular system), using elementary row operations. There are n − 1 stages, beginning
with A(1) := A, b(1) := b, and finishing with the upper triangular system A(n) x = b(n) .
At the start of the kth stage we have converted the original system to A(k) x = b(k) , where
 (k) (k) 
(k) A11 A12
A = (k) ,
0 A22
(k)
with A11 ∈ R(k−1)×(k−1) upper triangular.
The purpose of the kth stage of the elimination is to zero the elements below the diagonal
in the kth column of A(k) . This is accomplished by the operations
(k+1) (k) (k)
aij = aij − mik akj , i = k + 1 : n, j = k + 1 : n,
(k+1) (k) (k)
bi = bi − mik bk , i = k + 1 : n,

where
(k) (k)
mik = aik /akk , i = k + 1: n
are called the multipliers.
At the end of the (n − 1)st stage we have the upper triangular system A(n) x = b(n) , which
is solved by back substitution (see later). For an n × n matrix, GE requires 2n3 /3 flops, which
is about half the work required by a Householder QR factorization.

2 LU Factorization
Much insight into GE is obtained by expressing it in matrix notation. We can write
 
Ik−1

 1 

 −mk+1,k 1 
A(k+1) = Mk A(k) := 
 (k)
.. A .

 −mk+2,k . 

 .
.. . ..


−mn,k 1

The matrix Mk is called a Gauss transformation and can be expressed compactly as

Mk = I − mk eTk ,

where mk = [0, . . . , 0, mk+1,k , . . . , mn,k ]T and ek is the kth unit vector.

To invert Mk , just flip the signs of the multipliers: Mk−1 = I + mk eTk (see Exercise ??).
Overall,
Mn−1 Mn−2 . . . M1 A = A(n) =: U,
MATHx6101: LU Factorization Page 2

and so

A = M1−1 M2−1 . . . Mn−1

−1
U
= (I + m1 e1 )(I + m2 eT2 ) . . . (I + mn−1 eTn−1 )U
T

 Xn−1 
T
= I+ mi ei U [why?]
i=1
1
 
 m21 1 
 . ..
.

=
 .. m32 .  U =: LU.
 . .. ...

. .

mn1 mn2 ... mn,n−1 1

Conclusion. GE computes an LU factorization of A

A = LU ,

where L is unit lower triangular (“unit” means 1s on the diagonal) and U is upper triangular.

Theorem 1 There exists a unique LU factorization of A ∈ Rn×n if and only if Ak is nonsin-

gular for k = 1 : n − 1, where Ak is the leading principal submatrix of order k.
Proof. Suppose Ak is nonsingular for k = 1 : n − 1. Suppose Ak−1 has the unique LU
factorization Ak−1 = Lk−1 Uk−1 (this supposition clearly holds for k − 1 = 1). We look for a
factorization     
Ak−1 b Lk−1 0 Uk−1 u
Ak = = =: Lk Uk .
cT akk lT 1 0 ukk
The equations to be satisfied are

Lk−1 u = b,
T
Uk−1 l = c,
akk = lT u + ukk .

The matrices Lk−1 and Uk−1 are nonsingular, since 0 6= det(Ak−1 ) = det(Lk−1 ) det(Uk−1 ), so
the equations have a unique solution, completing the induction.
We prove the converse, under the assumption that A is nonsingular (for singular A, see [1,
Prob. 9.1]). Suppose an LU factorization exists. Then Ak = Lk Uk for k = 1 : n, which gives

det(Ak ) = det(Uk ) = u11 . . . ukk . (1)

Setting k = n we find that 0 6= det(A) = u11 . . . unn , and hence det(Ak ) = u11 . . . ukk 6= 0,
k = 1 : n − 1.
Here is an example of an LU factorization (function g ep is not a built-in MATLAB function;
it is described on the following “LU Factorization Example” handout):
MATHx6101: LU Factorization Page 3

>> A = gallery(’binomial’,4), [L,U] = gep(A)

A =
1 3 3 1
1 1 -1 -1
1 -1 -1 1
1 -3 3 -1
L =
1 0 0 0
1 1 0 0
1 2 1 0
1 3 3 1
U =
1 3 3 1
0 -2 -4 -2
0 0 4 4
0 0 0 -8

The following alternative proof of uniqueness illustrates a useful technique. Suppose A is

nonsingular and that A has two LU factorizations: A = L1 U1 = L2 U2 . The Ui are necessarily
nonsingular (why?), so L−1 −1
2 L1 = U2 U1 . The left side of this equation is unit lower triangular
and the right side upper triangular; therefore both sides must equal the identity matrix, which
means that L1 = L2 and U1 = U2 , as required.

Historical Note
In a 1948 paper [2], Alan Turing formulated the LU factorization, showed
that GE computes it, and proved the “if” part of Theorem ??. He also
introduced the term “condition number” (he used the Frobenius norm) and
the term “preconditioning” that we will use in the iterative methods section
of the course.

3 Relation of Cholesky with GE

If A is symmetric positive definite then A = RT R (Cholesky) and A = LU are the same
factorizations up to diagonal scaling of L and U :
1/2
A = LU = LD · D−1 U ≡ RTR, D = diag(uii ).

GE for general A ∈ Rn×n costs 2n3 /3 flops—twice the cost of Cholesky since there is no
symmetry.

4 Gaussian Elimination with Partial Pivoting

There are two problems with GE as described above.
(k)
1. There is a breakdown with division by zero if akk = 0.
MATHx6101: LU Factorization Page 4

2. If we are working in finite precision and some multiplier mik is large, then there is a
(k) (k) (k)
possible loss of significance: in the subtraction aij − mik akj , low-order digits of aij
could be lost. Losing these digits could correspond to making a relatively large change to
the original matrix A.
 
 1
Example: The simplest example of this second phenomenon is for the matrix A = .
1 1
GE without pivoting produces
    
 1 1 0  1
= = LU.
1 1 1/ 1 0 −1/ + 1

Let  < u (the unit roundoff). Then in floating point arithmetic the factors are approximated
by    
1 0  1
f l(L) = =: L, f l(U ) =
b =: Ub,
1/ 1 0 −1/
which would be the exact answer if we changed a22 from 1 to 0. Hence L bU
b = A + ∆A
with k∆Ak∞ /kAk∞ = 1/2  u. This shows that for this matrix GE does not compute the LU
factorization of a matrix close to A, which means that GE is behaving as an unstable algorithm.

These observations motivate the strategy of partial pivoting. At the start of the kth stage,
the kth and rth rows are interchanged, where
(k) (k)
|ark | := max |aik |.
k≤i≤n

Partial pivoting ensures that the multipliers are nicely bounded:

|mik | ≤ 1, i = k + 1 : n.

The effect of partial pivoting is easily seen by considering the case n = 4. We have

U = M3 P3 M2 P2 M1 P1 A, where Pk swaps rows k, r (r ≥ k),

= M3 · P3 M2 P3 · P3 P2 M1 P2 P3 · P3 P2 P1 A
| {z } | {z } | {z }
0 0 0
=: M3 M2 M1 P A,

where, for example, M10 = P3 P2 (I − m1 eT1 )P2 P3 = I − (P3 P2 m1 )eT1 =: I − m01 eT1 . For k = 1, 2, 3,
Mk0 =: I − m0k eTk is the same as Mk except the multipliers are interchanged, since eTi m0k = 0 for
i = 1 : k. Hence, for n = 4, GE with partial pivoting (GEPP) applied to A is equivalent to GE
without pivoting applied to the row-permuted matrix P A. This conclusion is true for any n:

GEPP computes a factorization P A = LU , where P is a permutation matrix.

Here is what GEPP produces for the same matrix as in the example above:
MATHx6101: LU Factorization Page 5

>> A = gallery(’binomial’,4), [L,U,P] = gep(A,’p’)

A =
1 3 3 1
1 1 -1 -1
1 -1 -1 1
1 -3 3 -1
L =
1.0000 0 0 0
1.0000 1.0000 0 0
1.0000 0.6667 1.0000 0
1.0000 0.3333 1.0000 1.0000
U =
1.0000 3.0000 3.0000 1.0000
0 -6.0000 0 -2.0000
0 0 -4.0000 1.3333
0 0 0 -2.6667
P =
1 0 0 0
0 0 0 1
0 0 1 0
0 1 0 0
>> norm(P*A-L*U,1) % Check the residual is at roundoff level.
ans =
4.4409e-16

5 Linear Systems
We wish to exploit LU factorization to solve the linear system

Ax = b,

where A ∈ Rn×n , b ∈ Rn are given and x ∈ Rn is to be calculated. We first consider how to

solve a triangular system.

5.1 Triangular Systems

Consider the system
Lx = b,
where L ∈ Rn×n is lower triangular and nonsingular (so that lii 6= 0, i = 1 : n). The solution
can be calculated by the forward substitution algorithm:

x1 = b1 /l11
 k−1
X 
x k = bk − lkj xj /lkk , k = 2 : n.
j=1
MATHx6101: LU Factorization Page 6

In a analogous way, the nonsingular upper triangular system

Ux = b

can be solved using the backward substitution algorithm. We express this in pseudocode:

1 xn = bn /unn
2 for i = n − 1: −1: 1
3 s = bi
4 for j = i + 1: n
5 s = s − uij xj
6 end
7 xi = s/uii
8 end

Both algorithms require n2 flops. Forward and backward substitution are extremely stable
algorithms, as the next result shows.

Theorem 2 Let the triangular system T x = b, where T ∈ Rn×n is nonsingular be solved by

substitution. Then the computed solution x̂ satisfies

(T + ∆T )x̂ = b,

where |∆tij | ≤ nu|tij | for all i and j, where u is the unit roundoff.

5.2 LU Factorization to Solve Ax = b

There are three steps in solving Ax = b by LU factorization.

(1) Use Gaussian elimination with partial pivoting to factorize A ∈ Rn×n as

P A = LU,

where L is unit lower triangular and U is upper triangular.

(2) Solve the lower triangular system

Ly = P b
by forward substitution.

(3) Solve the upper triangular system

Ux = y
by back substitution.

MATLAB’s backslash (A\b) carries out these steps when presented with a square linear
system.
Step (1) costs 2n3 /3 flops, and steps (2) and (3) both cost n2 flops. Therefore the cost of
the solution process is dominated (for large n) by the cost of the factorization.
MATHx6101: LU Factorization Page 7

Maxim 1 If we have to solve several systems Axi = bi , i = 1 : r, all with the same coefficient
matrix A, we should factorize A once and re-use the factors when doing the triangular
solves for each right-hand side.

Maxim 2 Never solve Ax = b by forming x = A−1 × b. Reasons:

1. Forming A−1 costs 2n3 flops, so this approach is more expensive than (1)–(3).
2. The use of A−1 is less stable and accurate than (1)–(3).
3. For sparse matrices, A−1 is usually full.

TOP500
The TOP500 list (http://www.top500.org) ranks the world’s fastest com-
puters by their speed (measured in flops per second) in solving a very large
random linear system Ax = b by GE.

6 Error Analysis
Traditionally, error analysis for GE is expressed in terms of the growth factor
(k)
maxi,j,k |aij |
ρn = ≥ 1,
maxi,j |aij |
(k)
which involves all the elements aij (k = 1 : n) that occur during the elimination. Using the
(i)
bound |uij | = |aij | ≤ ρn maxi,j |aij | we obtain the following classic theorem.

Theorem 3 (Wilkinson, 1961) Let A ∈ Rn×n and suppose GE with partial pivoting produces
a computed solution x
b to Ax = b. Then

(A + ∆A)b
x = b, k∆Ak∞ ≤ p(n)ρn ukAk∞ ,

where u is the unit roundoff and p(n) is a cubic polynomial.

Comments

• Ideally, we would like k∆Ak∞ ≤ ukAk∞ which reflects the uncertainty caused by storing
A on the computer.

• Unfortunately, the bound is bigger by a factor p(n)ρn . The danger term is the growth
factor ρn ≥ 1.

• The theorem shows that a pivoting strategy should  aim  to keep ρn small. If no pivoting
is done ρn can be arbitrarily large. E.g., for A = 1 11 (0 <   1), ρn = 1/.

• For partial pivoting it can be shown that ρn ≤ 2n−1 , but in practice ρn is almost always
of modest size (ρn ≤ 50, say).
MATHx6101: LU Factorization Page 8

It is important to identify classes of matrices for which pivoting is not necessary, that is,
for which the LU factorization A = LU exists and ρn is nicely bounded. One such class is the
symmetric positive definite matrices, for which it can be shown that ρn = 1 (see Exercise ??).
Another class is diagonally dominant matrices. A ∈ Rn×n is diagonally dominant by rows if
X
|aij | ≤ |aii |, i = 1 : n,
j6=i

and diagonally dominant by columns if AT is diagonally dominant by rows.

Theorem 4 (Wilkinson) Let A ∈ Rn×n be nonsingular. If A is diagonally dominant by rows

or columns then A has an LU factorization without pivoting and the growth factor ρn ≤ 2.

7 Other Pivoting Strategies

While partial pivoting is the pivoting strategy almost always used in practice for general ma-
trices, two other strategies are also important. It suffices to describe how they choose the kth
pivot.

7.1 Complete Pivoting

At the start of the kth stage rows k and r and columns k and s are interchanged where
(k)
|a(k)
rs | := max |aij |;
k≤i,j≤n

in other words, a pivot of maximal magnitude is chosen over the remaining submatrix.

• The cost of the pivoting is O(n3 ) comparisons in total, compared with O(n2 ) for partial
pivoting. This is of the same order of magnitude as the number of flops, which is why
complete pivoting is usually avoided.

• For complete pivoting we have a much smaller bound on the growth factor ρn (Wilkinson):
1
ρn ≤ n1/2 (2 · 31/2 · · · n1/(n−1) )1/2 ∼ c n1/2 n 4 log n .

7.2 Rook Pivoting

At the start of the kth stage, rows k and r and columns k and s are interchanged, where
(k) (k)
|a(k)
rs | = max |ais | = max |arj |;
k≤i≤n k≤j≤n

in other words, a pivot is chosen that is the largest in magnitude in both its column (as for
partial pivoting) and its row. The pivot search is done by repeatedly looking down a column
and across a row for the largest element in modulus; see Figure ??.
Rook pivoting is a relatively recent strategy. Its cost, and the size of the growth factor,
are typically intermediate between partial and complete pivoting (but O(n3 ) comparisons are
required in the worst case).
MATHx6101: LU Factorization Page 9

•
1 10 1 2 4 5
• •
0 5 2 7 8 2
•
2 0 3 1 9 4
• •
3 2 4 2 1 0
• •
1 4 5 6 3 1
1 0 3 4 0 12

Figure 1: Illustration of how rook pivoting searches for the first pivot for a particular 6 × 6
matrix (with the positive integer entries shown). Each dot denotes a putative pivot that is
tested to see if it is the largest in magnitude in both its row and its column.

7.3 Assessment

Both complete pivoting and rook pivoting yield a factorization P AQ = LU , where P and Q
are permutation matrices. They tend to be used in two circumstances:

• when a stable factorization is required at any cost (especially for very small n, where cost
is hardly an issue),

• when a rank-revealing factorization is required (roughly speaking, one in which when A

is close to a matrix of rank k the last n − k rows of U are small).
 MATHx6101: LU Factorization Page 10

Exercises
1. Let    
2 1 1 1
A = 4 3 3 and b = 1  .

8 7 9 1
Use Gaussian elimination to reduce the full linear system Ax = b to a triangular system U x = c
and solve the system.

2. Let Mk = I − mk eTk and eTi mk = 0 for i ≤ k. Show that Mk−1 = I + mk eTk .

3. Show that for an n × n matrix, Gaussian elimination requires 2n3 /3 flops.

4. Show that the product of two upper (lower) triangular matrices is upper (lower) triangular.
Show also that the inverse of a triangular matrix is triangular.

5. Suppose the factorization P A = LU has been computed by Gaussian elimination with partial
pivoting. Explain in detail how to

• Compute det(A).
• Compute A−1 . State the total cost.

6. Let A ∈ Rn×n be nonsingular. Explain how to compute xT A−1 y without computing A−1 .

7. Show that if GE with partial pivoting is applied to A ∈ Rn×n then the growth factor ρn ≤ 2n−1 .

8. Let A ∈ Rn×n be symmetric positive definite. Show that the reduced submatrix B of order
n − 1 at the endof the first stage of Gaussian elimination is also symmetric positive definite.
T T
[Hint: A = αa aC ⇒ A(2) = α0 aB . Find B and show that xT Bx > 0 by taking a particular y
in the expression y T Ay > 0.]
(k) (k−1) (1)
(Harder) Deduce that 0 < akk ≤ akk ≤ · · · ≤ akk = akk and hence that the growth factor
ρn ≤ 1.

References
[1] Nicholas J. Higham. Accuracy and Stability of Numerical Algorithms. Second edition,
Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2002. xxx+680
pp. ISBN 0-89871-521-0.

[2] A. M. Turing. Rounding-off errors in matrix processes. Quart. J. Mech. Appl. Math., 1:
287–308, 1948.