Bindel, Fall 2009 Matrix Computations (CS 6210)

Week 5: Wednesday, Sep 23

Sherman-Morrison-Woodbury
The Sherman-Morrison formula describes the solution of A + uv T when there
is already a factorization for A. An easy way to derive the formula is through
block Gaussian elimination. In order to compute the product (A + uv T )x, we
would usually first compute = v T x and then compute (A+uv T )x = Ax+u.
So we can write (A + uv T )x = b in terms of an extended linear system
    
A u x b
= .
v T 1 0
We can factor the matrix in this extended system as
    
A u I 0 A u
= T 1 ,
v T 1 v A 1 0 1 v T A1 u
apply forward substitution with the block lower triangular factor,

y = b,
= v T A1 y,

and apply backward substitution with the block upper triangular factor,

= (1 v T A1 u)1
x = A1 (y u).

If we put all the algebra together, we find

A1 uv T A1
 
1
x= A b,
1 + v T A1 u
or
A1 uv T A1
(A + uv T )1 = A1 .
1 + v T A1 u
This last formula is usually called the Sherman-Morrison formula. The
Sherman-Morrison-Woodbury formula is the generalization to a rank k mod-
ification to A:

(A + U V T )1 = A1 A1 U (I + V T A1 U )1 V T A1 .
Bindel, Fall 2009 Matrix Computations (CS 6210)

Backward error in Gaussian elimination

In lecture, I cut straight to the point: solving Ax = b in finite precision using
Gaussian elimination followed by forward and backward substitution yields
a computed solution x exactly satisfies

(1) (A + A)x = b,

where |A| . 3nmach |L||U |, assuming L and U are the computed L and U
factors.
Though I didnt do this in class, here I will briefly sketch a part of the
error analysis following Demmels treatment (2.4.2). Mostly, this is because
I find the treatment in 3.3.1 of our book less clear than I would like but
also, the bound in Demmels book is marginally tighter. Here is the idea.
Suppose L and U are the computed L and U factors. We obtain ujk by
repeatedly subtracting lji uik from the original ajk , i.e.
j1
!
X
ujk = fl ajk lji uik .
i=1

Regardless of the order of the sum, we get an error that looks like
j1
X
ujk = ajk (1 + 0 ) lji uik (1 + i ) + O(2 )
mach
i=1

where |i | (j 1)mach . Turning this around gives

j1
!
1 ljj ujk +
X
lji uik (1 + i ) + O(2 )
ajk = mach
1 + 0 i=1
j1
X
= ljj ujk (1 0 ) + lji uik (1 + i 0 ) + O(2 )
mach
i=1
 
= LU + Ejk ,
jk

where
j1
X
Ejk = ljj ujk 0 + lji uik (i 0 ) + O(2 )
mach
i=1
Bindel, Fall 2009 Matrix Computations (CS 6210)

is bounded in magnitude by (j 1)mach (|L||U |)jk + O(2mach )1 . A similar

argument for the components of L yields

A = LU + E, where |E| nmach |L||U | + O(2mach ).

In addition to the backward error due to the computation of the LU

factors, there is also backward error in the forward and backward substitution
phases, which gives the overall bound (1).

Pivoting
The backward error analysis in the previous section is not completely satis-
factory, since |L||U | may be much larger than |A|, yielding a large backward
error overall. For example, consider the matrix
    
1 1 0 1
A= = 1 .
1 1 1 0 1 1

If 0 <  1 then kLk kU k 2 , even though kAk 2. The problem

is that we ended up subtracting a huge multiple of the first row from the
second row because is close to zero that is, the leading principle minor
is nearly singular. If were exactly zero, then the factorization would fall
apart even in exact arithmetic. The solution to the woes of singular and near
singular minors is pivoting; instead of solving a system with A, we re-order
the equations to get
    
1 1 1 0 1 1
A = = .
1 1 0 1

Now the triangular factors for the re-ordered system matrix A have very
modest norms, and so we are happy. If we think of the re-ordering as the
effect of a permutation matrix P , we can write
     
1 0 1 1 0 1 1
A= = = P T LU.
1 1 1 0 1 0 1
1
Its obvious that Ej k is bounded in magnitude by 2(j 1)mach (|L||U |)jk + O(2mach ).
We cut a factor of two if we go down to the level of looking at the individual rounding
errors during the dot product, because some of those errors cancel.
Bindel, Fall 2009 Matrix Computations (CS 6210)

Note that this is equivalent to writing P A = LU where P is another permu-

tation (which undoes the action of P T ).
If we wish to control the multipliers, its natural to choose the permutation
P so that each of the multipliers is at most one. This standard choice leads
to the following algorithm:

for j = 1:n-1

% Find ipiv >= j to maximize |A(i,j)|

[absAij, ipiv] = max(abs(A(j:n,j)));
ipiv = ipiv + j-1;

% Swap row ipiv and row j

Aj = A(j,j:n);
A(j,j:n) = A(ipiv,j:n);
A(ipiv,j:n) = Aj;

% Record the pivot row

piv(j) = ipiv;

% Update trailing submatrix

A(j+1:n,j+1:n) = A(j+1:n,j+1:n) - A(j+1:n,j)*A(j,j+1:n);

end

By design, this algorithm produces an L factor in which all the elements

are bounded by one. But what about the U factor? There exist pathological
matrices for which the elements of U grow exponentially with n. But these
examples are extremely uncommon in practice, and so, in general, Gaussian
elimination with partial pivoting does indeed have a small backward error.
Of course, the pivot growth is something that we can monitor, so in the
unlikely event that it does look like things are blowing up, we can tell there
is a problem and try something different.
When problems do occur, it is more frequently the result of ill-conditioning
in the problem than of pivot growth during the factorization.
Bindel, Fall 2009 Matrix Computations (CS 6210)

Iterative refinement revisited

Recall from last lecture that if we have a solver for A = A + E with E small,
then we can use iterative refinement to clean up the solution. The matrix
A could come from finite precision Gaussian elimination of A, for example,
or from some factorization of a nearby easier matrix. To get the refinement
iteration, we take the equation

(2) Ax = Ax Ex = b,

and think of x as the fixed point for an iteration

(3) Axk+1 Exk = b.

Note that this is the same as

Axk+1 (A A)xk = b,

or
xk+1 = xk + A1 (b Axk ).
Note that this latter form is the same as inexact Newton iteration on the
equation Axk b = 0 with the approximate Jacobian A.
If we subtract (2) from (3), we see

A(xk+1 x) E(xk x) = 0,

or
xk+1 x = A1 E(xk x).
Taking norms, we have

kxk+1 xk kA1 Ekkxk xk.

Thus, if kA1 Ek < 1, we are guaranteed that xk x as k . At least,

this is what happens in exact arithmetic. In practice, the residual is usually
computed with only finite precision, and so we might expect to stop making
progress at some point. This is the topic of the first problem in HW 2.