CS 450 – Numerical Analysis

†
Chapter 2: Systems of Linear Equations

Prof. Michael T. Heath

Department of Computer Science

University of Illinois at Urbana-Champaign
[email protected]

January 28, 2019

† Lecture slides based on the textbook Scientific Computing: An Introductory

Survey by Michael T. Heath, copyright c 2018 by the Society for Industrial and
Applied Mathematics. http://www.siam.org/books/cl80
 2

Systems of Linear Equations

 3

Review: Matrix-Vector Product

  
a1,1 a1,2 ··· a1,n x1
 a2,1 a2,2 ··· a2,n  x2 
Ax = 
  
.. .. .. ..  .. 
 . . . .  . 
am,1 am,2 ··· am,n xn
 
a1,1 x1 + a1,2 x2 + · · · + a1,n xn
 a2,1 x1 + a2,2 x2 + · · · + a2,n xn 
= 
 
.. 
 . 
am,1 x1 + am,2 x2 + · · · + am,n xn
     
a1,1 a1,2 a1,n
 a2,1   a2,2   a2,n 
= x1  .  + x2  .  + · · · + xn  .
     
 ..   ..   ..


am,1 am,2 am,n

Definition: For A ∈ Rm×n , span(A) = {Ax : x ∈ Rn }

 4

System of Linear Equations

Ax =b

I Given m × n matrix A and m-vector b, find unknown n-vector x

satisfying Ax = b
I System of equations asks whether b can be expressed as linear
combination of columns of A, or equivalently, is b ∈ span(A)?
I If so, coefficients of linear combination are components of solution
vector x
I Solution may or may not exist, and may or may not be unique
I For now, we consider only square case, m = n
 5

Singularity and Nonsingularity

n × n matrix A is nonsingular if it has any of following equivalent

properties

1. Inverse of A, denoted by A−1 , exists such that AA−1 = A−1 A = I

2. det(A) 6= 0

3. rank(A) = n

4. For any vector z 6= 0, Az 6= 0

 6

Existence and Uniqueness

I Existence and uniqueness of solution to Ax = b depend on whether

A is singular or nonsingular
I Can also depend on b, but only in singular case
I If b ∈ span(A), system is consistent

A b # solutions
nonsingular arbitrary 1

singular b ∈ span(A) ∞

singular b∈
/ span(A) 0
 7

Geometric Interpretation

I In two dimensions, each equation determines straight line in plane

I Solution is intersection point of two straight lines, if any
I If two straight lines are not parallel (nonsingular), then their
intersection point is unique solution
I If two straight lines are parallel (singular), then they either do not
intersect (no solution) or else they coincide (any point along line is
solution)
I In higher dimensions, each equation determines hyperplane; if matrix
is nonsingular, intersection of hyperplanes is unique solution
 8

Example: Nonsingularity

I 2 × 2 system

2x1 + 3x2 = b1
5x1 + 4x2 = b2

or in matrix-vector notation
    
2 3 x1 b
Ax = = 1 =b
5 4 x2 b2

is nonsingular and thus has unique solution regardless of value of b

 T  T
I For example, if b = 8 13 , then x = 1 2 is unique solution
 9

Example: Singularity

I 2 × 2 system     
2 3 x1 b
Ax = = 1 =b
4 6 x2 b2
is singular regardless of value of b
 T
I With b = 4 7 , there is no solution
 T  T
I With b = 4 8 , x = γ (4 − 2γ)/3 is solution for any real
number γ, so there are infinitely many solutions
 10

Norms and Condition Number

 11

Vector Norms

I Magnitude (absolute value, modulus) for scalars generalizes to norm

for vectors
I We will use only p-norms, defined by

n
!1/p
X p
kxkp = |xi |
i=1

for integer p > 0 and n-vector x

I Important special cases
Pn
I 1-norm: kxk1 = i=1 |xi |
Pn
1/2
I 2-norm: kxk2 = i=1 |xi |2
I ∞-norm: kxk∞ = maxi |xi |
 12

Example: Vector Norms

I Drawing shows unit “circle” in two dimensions for each norm

I Norms have following values for vector shown

kxk1 = 2.8, kxk2 = 2.0, kxk∞ = 1.6

h interactive example i
 13

Equivalence of Norms

I In general, for any vector x in Rn , kxk1 ≥ kxk2 ≥ kxk∞

I However, we also have
√
I kxk1 ≤ n · kxk2
√
I kxk2 ≤ n · kxk∞
I kxk1 ≤ n · kxk∞

I For given n, norms differ by at most a constant, and hence are

equivalent: if one is small, all must be proportionally small
I Consequently, we can use whichever norm is most convenient in
given context
 14

Properties of Vector Norms

I For any vector norm

I kxk > 0 if x 6= 0
I kγxk = |γ| · kxk for any scalar γ
I kx + y k ≤ kxk + ky k (triangle inequality)

I In more general treatment, these properties taken as definition of

vector norm
I Useful variation on triangle inequality
I | kxk − ky k | ≤ kx − y k
 15

Matrix Norms

I Matrix norm induced by a given vector norm is defined by

kAxk
kAk = maxx6=0
kxk

I Norm of matrix measures maximum relative stretching matrix does

to any vector in given vector norm
 16

Example Matrix Norms

I Matrix norm induced by vector 1-norm is maximum absolute column

sum
Xn
kAk1 = max |aij |
j
i=1

I Matrix norm induced by vector ∞-norm is maximum absolute row

sum
Xn
kAk∞ = max |aij |
i
j=1

I Handy way to remember these is that matrix norms agree with

corresponding vector norms for n × 1 matrix
I No simple formula for matrix 2-norm
 17

Properties of Matrix Norms

I Any matrix norm satisfies

I kAk > 0 if A 6= 0
I kγAk = |γ| · kAk for any scalar γ
I kA + Bk ≤ kAk + kBk

I Matrix norms we have defined also satisfy

I kABk ≤ kAk · kBk
I kAxk ≤ kAk · kxk for any vector x
 18

Condition Number

I Condition number of square nonsingular matrix A is defined by

cond(A) = kAk · kA−1 k

I By convention, cond(A) = ∞ if A is singular

I Since    −1
−1 kAxk kAxk
kAk · kA k= max · min
x6=0 kxk x6=0 kxk

condition number measures ratio of maximum stretching to

maximum shrinking matrix does to any nonzero vectors
I Large cond(A) means A is nearly singular
 19

Properties of Condition Number

I For any matrix A, cond(A) ≥ 1

I For identity matrix I , cond(I ) = 1

I For any matrix A and scalar γ, cond(γA) = cond(A)

max |di |
I For any diagonal matrix D = diag(di ), cond(D) =
min |di |

h interactive example i
 20

Computing Condition Number

I Definition of condition number involves matrix inverse, so it is

nontrivial to compute
I Computing condition number from definition would require much
more work than computing solution whose accuracy is to be assessed

I In practice, condition number is estimated inexpensively as

byproduct of solution process
I Matrix norm kAk is easily computed as maximum absolute column
sum (or row sum, depending on norm used)
I Estimating kA−1 k at low cost is more challenging
 21

Computing Condition Number, continued

I From properties of norms, if Az = y , then

kzk
≤ kA−1 k
ky k

and this bound is achieved for optimally chosen y

I Efficient condition estimators heuristically pick y with large ratio
kzk/ky k, yielding good estimate for kA−1 k
I Good software packages for linear systems provide efficient and
reliable condition estimator
I Condition number useful in assessing accuracy of approximate
solution
 22

Assessing Accuracy
 23

Error Bounds
I Condition number yields error bound for approximate solution to
linear system
I Let x be solution to Ax = b, and let x̂ be solution to Ax̂ = b + ∆b
I If ∆x = x̂ − x, then

b + ∆b = A(x̂) = A(x + ∆x) = Ax + A∆x

which leads to bound

k∆xk k∆bk
≤ cond(A)
kxk kbk

for possible relative change in solution x due to relative change in

right-hand side b

h interactive example i
 24

Error Bounds, continued

I Similar result holds for relative change in matrix: if (A + E )x̂ = b,

then
k∆xk kE k
≤ cond(A)
kx̂k kAk

I If input data are accurate to machine precision, then bound for

relative error in solution x becomes

kx̂ − xk
≤ cond(A) mach
kxk

I Computed solution loses about log10 (cond(A)) decimal digits of

accuracy relative to accuracy of input
 25

Error Bounds – Illustration

I In two dimensions, uncertainty in intersection point of two lines

depends on whether lines are nearly parallel

h interactive example i
 26

Error Bounds – Caveats

I Normwise analysis bounds relative error in largest components of

solution; relative error in smaller components can be much larger
I Componentwise error bounds can be obtained, but are somewhat
more complicated

I Conditioning of system is affected by relative scaling of rows or

columns
I Ill-conditioning can result from poor scaling as well as near
singularity
I Rescaling can help the former, but not the latter
 27

Residual

I Residual vector of approximate solution x̂ to linear system Ax = b

is defined by
r = b − Ax̂

I In theory, if A is nonsingular, then kx̂ − xk = 0 if, and only if,

kr k = 0, but they are not necessarily small simultaneously

I Since
k∆xk kr k
≤ cond(A)
kx̂k kAk · kx̂k
small relative residual implies small relative error in approximate
solution only if A is well-conditioned
 28

Residual, continued

I If computed solution x̂ exactly satisfies

(A + E )x̂ = b

then
kr k kE k
≤
kAk kx̂k kAk
so large relative residual implies large backward error in matrix, and
algorithm used to compute solution is unstable

I Stable algorithm yields small relative residual regardless of

conditioning of nonsingular system

I Small residual is easy to obtain, but does not necessarily imply

computed solution is accurate
 29

Example: Small Residual

I For linear system
    
0.913 0.659 x1 0.254
Ax = = =b
0.457 0.330 x2 0.127

consider two approximate solutions

   
0.6391 0.999
x̂1 = , x̂2 =
−0.5 −1.001

I Norms of respective residuals are

kr1 k1 = 7.0 × 10−5 , kr2 k1 = 2.4 × 10−2

I Exact solution is x = [1, −1]T , so x̂2 is much more accurate than x̂1 ,
despite having much larger residual
I A is ill-conditioned (cond(A) > 104 ), so small residual does not
imply small error
 30

Solving Linear Systems

 31

Solving Linear Systems

I General strategy: To solve linear system, transform it into one whose

solution is same but easier to compute
I What type of transformation of linear system leaves solution
unchanged?
I We can premultiply (from left) both sides of linear system Ax = b
by any nonsingular matrix M without affecting solution
I Solution to MAx = Mb is given by
x = (MA)−1 Mb = A−1 M −1 Mb = A−1 b
 32

Example: Permutations

I Permutation matrix P has one 1 in each row and column and zeros
elsewhere, i.e., identity matrix with rows or columns permuted
I P T reverses permutation, so P −1 = P T
I Premultiplying both sides of system by permutation matrix,
PAx = Pb, reorders rows, but solution x is unchanged
I Postmultiplying A by permutation matrix, APx = b, reorders
columns, which permutes components of original solution
x = (AP)−1 b = P −1 A−1 b = P T (A−1 b)
 33

Example: Diagonal Scaling

I Row scaling: premultiplying both sides of system by nonsingular

diagonal matrix D, DAx = Db, multiplies each row of matrix and
right-hand side by corresponding diagonal entry of D, but solution x
is unchanged
I Column scaling: postmultiplying A by D, ADx = b, multiplies each
column of matrix by corresponding diagonal entry of D, which
rescales original solution
x = (AD)−1 b = D −1 A−1 b
 34

Triangular Linear Systems

I What type of linear system is easy to solve?

I If one equation in system involves only one component of solution
(i.e., only one entry in that row of matrix is nonzero), then that
component can be computed by division
I If another equation in system involves only one additional solution
component, then by substituting one known component into it, we
can solve for other component
I If this pattern continues, with only one new solution component per
equation, then all components of solution can be computed in
succession.
I System with this property is called triangular
 35

Triangular Matrices

I Two specific triangular forms are of particular interest

I lower triangular : all entries above main diagonal are zero, aij = 0 for
i <j
I upper triangular : all entries below main diagonal are zero, aij = 0
for i > j

I Successive substitution process described earlier is especially easy to

formulate for lower or upper triangular systems
I Any triangular matrix can be permuted into upper or lower
triangular form by suitable row permutation
 36

Forward-Substitution

I Forward-substitution for lower triangular system Lx = b

 
i−1
X
x1 = b1 /`11 , xi = bi − `ij xj  / `ii , i = 2, . . . , n
j=1

for j = 1 to n { loop over columns }

if `jj = 0 then stop { stop if matrix is singular }
xj = bj /`jj { compute solution component }
for i = j + 1 to n
bi = bi − `ij xj { update right-hand side }
end
end
 37

Back-Substitution

I Back-substitution for upper triangular system Ux = b

 
Xn
xn = bn /unn , xi = bi − uij xj  / uii , i = n − 1, . . . , 1
j=i+1

for j = n to 1 { loop backwards over columns }

if ujj = 0 then stop { stop if matrix is singular }
xj = bj /ujj { compute solution component }
for i = 1 to j − 1
bi = bi − uij xj { update right-hand side }
end
end
 38

Example: Triangular Linear System

    
2 4 −2 x1 2
0 1 1 x2  = 4
0 0 4 x3 8

I Using back-substitution for this upper triangular system, last

equation, 4x3 = 8, is solved directly to obtain x3 = 2
I Next, x3 is substituted into second equation to obtain x2 = 2
I Finally, both x3 and x2 are substituted into first equation to obtain
x1 = −1
 39

Elementary Elimination Matrices

 40

Elimination

I To transform general linear system into triangular form, need to

replace selected nonzero entries of matrix by zeros
I This can be accomplished by taking linear combinations of rows
 
a
I Consider 2-vector a = 1
a2
I If a1 6= 0, then
    
1 0 a1 a
= 1
−a2 /a1 1 a2 0
 41

Elementary Elimination Matrices

I More generally, we can annihilate all entries below kth position in
n-vector a by transformation
    
1 ··· 0 0 ··· 0 a1 a1
 .. . . .. .. . . ..   ..   .. 
.
 . . . . .  .   . 
   
0 · · · 1 0 ··· 0  ak  ak 
Mk a = 0 · · · −mk+1
 = 
 1 ··· 0  ak+1   0 
   
 .. . . .. .. . . ..   ..   .. 
. . . . . .  .   . 
0 ··· −mn 0 ··· 1 an 0

where mi = ai /ak , i = k + 1, . . . , n
I Divisor ak , called pivot, must be nonzero
I Matrix Mk , called elementary elimination matrix, adds multiple of
row k to each subsequent row, with multipliers mi chosen so that
result is zero
 42

Elementary Elimination Matrices, continued

I Mk is unit lower triangular and nonsingular

I Mk = I − mk ekT , where mk = [0, . . . , 0, mk+1 , . . . , mn ]T and ek is

kth column of identity matrix

I Mk−1 = I + mk ekT , which means Mk−1 = Lk is same as Mk except

signs of multipliers are reversed

I If Mj , j > k, is another elementary elimination matrix, with vector

of multipliers mj , then

Mk Mj = I − mk ekT − mj ejT + mk ekT mj ejT

= I − mk ekT − mj ejT

which means their product is essentially their “union” and similarly

for product of inverses, Lk Lj
 43

Example: Elementary Elimination Matrices

 
2
I For a =  4,
−2
    
1 0 0 2 2
M1 a = −2 1 0  4 = 0
1 0 1 −2 0

and     
1 0 0 2 2
M2 a = 0 1 0  4 = 4
0 1/2 1 −2 0
 44

Example, continued

I Note that
   
1 0 0 1 0 0
L1 = M1−1 = 2 1 0 , L2 = M2−1 = 0 1 0
−1 0 1 0 −1/2 1

and
   
1 0 0 1 0 0
M1 M2 = −2 1 0 , L1 L2 =  2 1 0
1 1/2 1 −1 −1/2 1
 45

LU Factorization by Gaussian Elimination

 46

Gaussian Elimination
I To reduce general linear system Ax = b to upper triangular form,
first choose M1 , with a11 as pivot, to annihilate first column of A
below first row
I System becomes M1 Ax = M1 b, but solution is unchanged
I Next choose M2 , using a22 as pivot, to annihilate second column of
M1 A below second row
I System becomes M2 M1 Ax = M2 M1 b, but solution is still unchanged

I Process continues for each successive column until all subdiagonal

entries have been zeroed
I Resulting upper triangular linear system
Mn−1 · · · M1 Ax = Mn−1 · · · M1 b
MAx = Mb
can be solved by back-substitution to obtain solution to original
linear system Ax = b
I Process just described is called Gaussian elimination
 47

LU Factorization

I Product Lk Lj is unit lower triangular if k < j, so

L = M −1 = M1−1 · · · Mn−1
−1
= L1 · · · Ln−1

is unit lower triangular

I By design, MA = U is upper triangular

I So we have
A = LU
with L unit lower triangular and U upper triangular

I Thus, Gaussian elimination produces LU factorization of matrix into

triangular factors
 48

LU Factorization, continued

I Having obtained LU factorization A = LU, equation Ax = b

becomes
LUx = b
which can be solved by
I solving lower triangular system Ly = b for y by forward-substitution
I then solving upper triangular system Ux = y for x by
back-substitution

I Note that y = Mb is same as transformed right-hand side in

Gaussian elimination

I Gaussian elimination and LU factorization are two ways of expressing

same solution process
 49

LU Factorization by Gaussian Elimination

for k = 1 to n − 1 { loop over columns }

if akk = 0 then stop { stop if pivot is zero }
for i = k + 1 to n { compute multipliers
mik = aik /akk for current column }
end
for j = k + 1 to n
for i = k + 1 to n { apply transformation to
aij = aij − mik akj remaining submatrix }
end
end
end
 50

Example: Gaussian Elimination

I Use Gaussian elimination to solve linear system

    
2 4 −2 x1 2
Ax =  4 9 −3 x2  =  8 = b
−2 −3 7 x3 10

I To annihilate subdiagonal entries of first column of A,

    
1 0 0 2 4 −2 2 4 −2
M1 A = −2 1 0  4 9 −3 = 0 1 1 ,
1 0 1 −2 −3 7 0 1 5
    
1 0 0 2 2
M1 b = −2 1 0  8 =  4
1 0 1 10 12
 51

Example, continued
I To annihilate subdiagonal entry of second column of M1 A,
    
1 0 0 2 4 −2 2 4 −2
M2 M1 A = 0 1 0 0 1 1 = 0 1 1 = U,
0 −1 1 0 1 5 0 0 4
    
1 0 0 2 2
M2 M1 b = 0 1 0   4 = 4 = Mb

0 −1 1 12 8

I We have reduced original system to equivalent upper triangular

system     
2 4 −2 x1 2
Ux = 0 1 1 x2  = 4 = Mb
0 0 4 x3 8
 
−1
which can now be solved by back-substitution to obtain x =  2
2
 52

Example, continued

I To write out LU factorization explicitly,

    
1 0 0 1 0 0 1 0 0
L1 L2 =  2 1 0 0 1 0 =  2 1 0 = L
−1 0 1 0 1 1 −1 1 1

so that
    
2 4 −2 1 0 0 2 4 −2
A= 4 9 −3 =  2 1 0 0 1 1 = LU
−2 −3 7 −1 1 1 0 0 4
 53

Pivoting
 54

Row Interchanges

I Gaussian elimination breaks down if leading diagonal entry of

remaining unreduced matrix is zero at any stage

I Easy fix: if diagonal entry in column k is zero, then interchange row

k with some subsequent row having nonzero entry in column k and
then proceed as usual

I If there is no nonzero on or below diagonal in column k, then there

is nothing to do at this stage, so skip to next column

I Zero on diagonal causes resulting upper triangular matrix U to be

singular, but LU factorization can still be completed

I Subsequent back-substitution will fail, however, as it should for

singular matrix
 55

Partial Pivoting

I In principle, any nonzero value will do as pivot, but in practice pivot

should be chosen to minimize error propagation

I To avoid amplifying previous rounding errors when multiplying

remaining portion of matrix by elementary elimination matrix,
multipliers should not exceed 1 in magnitude

I This can be accomplished by choosing entry of largest magnitude on

or below diagonal as pivot at each stage

I Such partial pivoting is essential in practice for numerically stable

implementation of Gaussian elimination for general linear systems

h interactive example i
 56

LU Factorization with Partial Pivoting

I With partial pivoting, each Mk is preceded by permutation Pk to

interchange rows to bring entry of largest magnitude into diagonal
pivot position
I Still obtain MA = U, with U upper triangular, but now

M = Mn−1 Pn−1 · · · M1 P1

I L = M −1 is still triangular in general sense, but not necessarily lower

triangular
I Alternatively, we can write

PA = L U

where P = Pn−1 · · · P1 permutes rows of A into order determined by

partial pivoting, and now L is lower triangular
 57

Complete Pivoting
I Complete pivoting is more exhaustive strategy in which largest entry
in entire remaining unreduced submatrix is permuted into diagonal
pivot position

I Requires interchanging columns as well as rows, leading to

factorization
PAQ = L U
with L unit lower triangular, U upper triangular, and P and Q
permutations

I Numerical stability of complete pivoting is theoretically superior, but

pivot search is more expensive than for partial pivoting

I Numerical stability of partial pivoting is more than adequate in

practice, so it is almost always used in solving linear systems by
Gaussian elimination
 58

Example: Pivoting

I Need for pivoting has nothing to do with whether matrix is singular

or nearly singular

I For example,  
0 1
A=
1 0
is nonsingular yet has no LU factorization unless rows are
interchanged, whereas  
1 1
A=
1 1
is singular yet has LU factorization
 59

Example: Small Pivots

I To illustrate effect of small pivots, consider
 
 1
A=
1 1

where  is positive number smaller than mach

I If rows are not interchanged, then pivot is  and multiplier is −1/,

so
   
1 0 1 0
M= , L= ,
−1/ 1 1/ 1
   
 1  1
U= =
0 1 − 1/ 0 −1/
in floating-point arithmetic, but then
    
1 0  1  1
LU = = 6= A
1/ 1 0 −1/ 1 0
 60

Example, continued

I Using small pivot, and correspondingly large multiplier, has caused

loss of information in transformed matrix
I If rows interchanged, then pivot is 1 and multiplier is −, so
   
1 0 1 0
M= , L= ,
− 1  1
   
1 1 1 1
U= =
0 1− 0 1
in floating-point arithmetic
I Thus,     
1 0 1 1 1 1
LU = =
 1 0 1  1
which is correct after permutation
 61

Pivoting, continued

I Although pivoting is generally required for stability of Gaussian

elimination, pivoting is not required for some important classes of
matrices

I Diagonally dominant
n
X
|aij | < |ajj |, j = 1, . . . , n
i=1, i6=j

I Symmetric positive definite

A = AT and x T Ax > 0 for all x 6= 0

 62

Residual
 63

Residual

I Residual r = b − Ax̂ for solution x̂ computed using Gaussian

elimination satisfies
kr k kE k
≤ ≤ ρ n2 mach
kAk kx̂k kAk

where E is backward error in matrix A and growth factor ρ is ratio

of largest entry of U to largest entry of A

I Without pivoting, ρ can be arbitrarily large, so Gaussian elimination

without pivoting is unstable

I With partial pivoting, ρ can still be as large as 2n−1 , but such

behavior is extremely rare
 64

Residual, continued

I There is little or no growth in practice, so

kr k kE k
≤ / n mach
kAk kx̂k kAk

which means Gaussian elimination with partial pivoting yields small

relative residual regardless of conditioning of system

I Thus, small relative residual does not necessarily imply computed

solution is close to “true” solution unless system is well-conditioned

I Complete pivoting yields even smaller growth factor, but additional

margin of stability is not usually worth extra cost
 65

Example: Small Residual

I Use 4-digit decimal arithmetic to solve

    
0.913 0.659 x1 0.254
=
0.457 0.330 x2 0.127

I Gaussian elimination with partial pivoting yields triangular system

    
0.9130 0.6590 x1 0.2540
=
0 0.0002 x2 −0.0001

I Back-substitution then gives solution

 T
x̂ = 0.6391 −0.5

I Exact residual norm for this solution is 7.04 × 10−5 , as small as we

can expect using 4-digit arithmetic
 66

Example, continued

I But exact solution is

 T
x = 1.00 1.00

so error is almost as large as solution

I Cause of this phenomenon is that matrix is nearly singular

(cond(A) > 104 )

I Division that determines x2 is between two quantities that are both

on order of rounding error, and hence result is essentially arbitrary

I When arbitrary value for x2 is substituted into first equation, value

for x1 is computed so that first equation is satisfied, yielding small
residual, but poor solution
 67

Implementing Gaussian Elimination

 68

Implementing Gaussian Elimination

I Gaussian elimination has general form of triple-nested loop

for
for
for
aij = aij − (aik /akk )akj
end
end
end

I Indices i, j, and k of for loops can be taken in any order, for total of
3! = 6 different arrangements

I These variations have different memory access patterns, which may

cause their performance to vary widely on different computers
 69

Uniqueness of LU Factorization

I Despite variations in computing it, LU factorization is unique up to

diagonal scaling of factors

I Provided row pivot sequence is same, if we have two LU

factorizations PA = LU = L̂Û, then L̂−1 L = ÛU −1 = D is both
lower and upper triangular, hence diagonal

I If both L and L̂ are unit lower triangular, then D must be identity

matrix, so L = L̂ and U = Û

I Uniqueness is made explicit in LDU factorization PA = LDU, with L

unit lower triangular, U unit upper triangular, and D diagonal
 70

Storage Management

I Elementary elimination matrices Mk , their inverses Lk , and

permutation matrices Pk used in formal description of LU
factorization process are not formed explicitly in actual
implementation

I U overwrites upper triangle of A, multipliers in L overwrite strict

lower triangle of A, and unit diagonal of L need not be stored

I Row interchanges usually are not done explicitly; auxiliary integer

vector keeps track of row order in original locations
 71

Complexity of Solving Linear Systems

I LU factorization requires about n3 /3 floating-point multiplications

and similar number of additions

I Forward- and back-substitution for single right-hand-side vector

together require about n2 multiplications and similar number of
additions

I Can also solve linear system by matrix inversion: x = A−1 b

I Computing A−1 is tantamount to solving n linear systems, requiring

LU factorization of A followed by n forward- and back-substitutions,
one for each column of identity matrix

I Operation count for inversion is about n3 , three times as expensive

as LU factorization
 72

Inversion vs. Factorization

I Even with many right-hand sides b, inversion never overcomes higher

initial cost, since each matrix-vector multiplication A−1 b requires n2
operations, similar to cost of forward- and back-substitution

I Inversion gives less accurate answer; for example, solving 3x = 18 by

division gives x = 18/3 = 6, but inversion gives
x = 3−1 × 18 = 0.333 × 18 = 5.99 using 3-digit arithmetic

I Matrix inverses often occur as convenient notation in formulas, but

explicit inverse is rarely required to implement such formulas

I For example, product A−1 B should be computed by LU

factorization of A, followed by forward- and back-substitutions using
each column of B
 73

Gauss-Jordan Elimination
I In Gauss-Jordan elimination, matrix is reduced to diagonal rather
than triangular form
I Row combinations are used to annihilate entries above as well as
below diagonal
I Elimination matrix used for given column vector a is of form
1 ··· 0 −m1 0 ··· 0 a1 0
    
 .. .. .. .. .. .. ..   ..   .. 
. . . . . . .  .   . 
   

0
 ··· 1 −mk−1 0 ··· 0 
ak−1   0 
  
0
 ··· 0 1 0 ··· 0   ak  = ak 
   
0
 ··· 0 −mk+1 1 ··· 0  ak+1 
 
 0
 
. .. .. .. .. .. ..   .  .
 .. . . . . . .   ..   .. 
0 ··· 0 −mn 0 ··· 1 an 0

where mi = ai /ak , i = 1, . . . , n
 74

Gauss-Jordan Elimination, continued

I Gauss-Jordan elimination requires about n3 /2 multiplications and

similar number of additions, 50% more expensive than LU
factorization
I During elimination phase, same row operations are also applied to
right-hand-side vector (or vectors) of system of linear equations
I Once matrix is in diagonal form, components of solution are
computed by dividing each entry of transformed right-hand side by
corresponding diagonal entry of matrix
I Latter requires only n divisions, but this is not enough cheaper to
offset more costly elimination phase

h interactive example i
 75

Updating Solutions
 76

Solving Modified Problems

I If right-hand side of linear system changes but matrix does not, then
LU factorization need not be repeated to solve new system

I Only forward- and back-substitution need be repeated for new

right-hand side

I This is substantial savings in work, since additional triangular

solutions cost only O(n2 ) work, in contrast to O(n3 ) cost of
factorization
 77

Sherman-Morrison Formula

I Sometimes refactorization can be avoided even when matrix does

change

I Sherman-Morrison formula gives inverse of matrix resulting from

rank-one change to matrix whose inverse is already known

(A − uv T )−1 = A−1 + A−1 u(1 − v T A−1 u)−1 v T A−1

where u and v are n-vectors

I Evaluation of formula requires O(n2 ) work (for matrix-vector

multiplications) rather than O(n3 ) work required for inversion
 78

Rank-One Updating of Solution

I To solve linear system (A − uv T )x = b with new matrix, use

Sherman-Morrison formula to obtain

x = (A − uv T )−1 b
= A−1 b + A−1 u(1 − v T A−1 u)−1 v T A−1 b

which can be implemented by following steps

I Solve Az = u for z, so z = A−1 u
I Solve Ay = b for y , so y = A−1 b
I Compute x = y + ((v T y )/(1 − v T z))z

I If A is already factored, procedure requires only triangular solutions

and inner products, so only O(n2 ) work and no explicit inverses
 79

Example: Rank-One Updating of Solution

I Consider rank-one modification

    
2 4 −2 x1 2
 4 9 −3 x2  =  8
−2 −1 7 x3 10

(with 3, 2 entry changed) of system whose LU factorization was

computed in earlier example

I One way to choose update vectors is

   
0 0
u =  0 and v = 1
−2 0

so matrix of modified system is A − uv T

 80

Example, continued

I Using LU factorization of A to solve Az = u and Ay = b,

   
−3/2 −1
z =  1/2 and y =  2
−1/2 2

I Final step computes updated solution

     
−1 −3/2 −7
vTy 2
x =y+ z =  2 +  1/2 =  4
1 − vTz 1 − 1/2
2 −1/2 0

I We have thus computed solution to modified system without

factoring modified matrix
 81

Improving Accuracy
 82

Scaling Linear Systems

I In principle, solution to linear system is unaffected by diagonal

scaling of matrix and right-hand-side vector

I In practice, scaling affects both conditioning of matrix and selection

of pivots in Gaussian elimination, which in turn affect numerical
accuracy in finite-precision arithmetic

I It is usually best if all entries (or uncertainties in entries) of matrix

have about same size

I Sometimes it may be obvious how to accomplish this by choice of

measurement units for variables, but there is no foolproof method
for doing so in general

I Scaling can introduce rounding errors if not done carefully

 83

Example: Scaling

I Linear system     
1 0 x1 1
=
0  x2 
has condition number 1/, so is ill-conditioned if  is small

I If second row is multiplied by 1/, then system becomes perfectly

well-conditioned

I Apparent ill-conditioning was due purely to poor scaling

I In general, it is usually much less obvious how to correct poor scaling

 84

Iterative Refinement

I Given approximate solution x0 to linear system Ax = b, compute

residual
r0 = b − Ax0
I Now solve linear system Az0 = r0 and take

x1 = x0 + z0

as new and “better” approximate solution, since

Ax1 = A(x0 + z0 ) = Ax0 + Az0

= (b − r0 ) + r0 = b

I Process can be repeated to refine solution successively until

convergence, potentially producing solution accurate to full machine
precision
 85

Iterative Refinement, continued

I Iterative refinement requires double storage, since both original

matrix and its LU factorization are required

I Due to cancellation, residual usually must be computed with higher

precision for iterative refinement to produce meaningful
improvement

I For these reasons, iterative improvement is often impractical to use

routinely, but it can still be useful in some circumstances

I For example, iterative refinement can sometimes stabilize otherwise

unstable algorithm
 86

Special Types of Linear Systems

 87

Special Types of Linear Systems

I Work and storage can often be saved in solving linear system if

matrix has special properties
I Examples include
I Symmetric : A = AT , aij = aji for all i, j
I Positive definite : x T Ax > 0 for all x 6= 0
I Band : aij = 0 for all |i − j| > β, where β is bandwidth of A
I Sparse : most entries of A are zero
 88

Symmetric Positive Definite Matrices

I If A is symmetric and positive definite, then LU factorization can be
arranged so that U = LT , which gives Cholesky factorization

A = L LT

where L is lower triangular with positive diagonal entries

I Algorithm for computing it can be derived by equating
corresponding entries of A and LLT
I In 2 × 2 case, for example,
    
a11 a21 l 0 l11 l21
= 11
a21 a22 l21 l22 0 l22

implies
√
q
l11 = a11 , l21 = a21 /l11 , l22 = 2
a22 − l21
 89

Cholesky Factorization

I One way to write resulting algorithm, in which Cholesky factor L

overwrites lower triangle of original matrix A, is

for k = 1 to n { loop over columns }

√
akk = akk
for i = k + 1 to n
aik = aik /akk { scale current column }
end
for j = k + 1 to n { from each remaining column,
for i = j to n subtract multiple
aij = aij − aik · ajk of current column }
end
end
end
 90

Cholesky Factorization, continued

I Features of Cholesky algorithm for symmetric positive definite
matrices
I All n square roots are of positive numbers, so algorithm is well
defined
I No pivoting is required to maintain numerical stability
I Only lower triangle of A is accessed, and hence upper triangular
portion need not be stored
I Only n3 /6 multiplications and similar number of additions are
required

I Thus, Cholesky factorization requires only about half work and half
storage compared with LU factorization of general matrix by
Gaussian elimination, and also avoids need for pivoting

h interactive example i
 91

Symmetric Indefinite Systems

I For symmetric indefinite A, Cholesky factorization is not applicable,

and some form of pivoting is generally required for numerical
stability

I Factorization of form
PAP T = LDLT
with L unit lower triangular and D either tridiagonal or block
diagonal with 1 × 1 and 2 × 2 diagonal blocks, can be computed
stably using symmetric pivoting strategy

I In either case, cost is comparable to that of Cholesky factorization

 92

Band Matrices

I Gaussian elimination for band matrices differs little from general

case — only ranges of loops change
I Typically matrix is stored in array by diagonals to avoid storing zero
entries
I If pivoting is required for numerical stability, bandwidth can grow
(but no more than double)
I General purpose solver for arbitrary bandwidth is similar to code for
Gaussian elimination for general matrices
I For fixed small bandwidth, band solver can be extremely simple,
especially if pivoting is not required for stability
 93

Tridiagonal Matrices
I Consider tridiagonal matrix
 
b1 c1 0 ··· 0
 .. .. 
a
 2 b2 c2 . . 


A= .. .. .. 
0 . . . 0 

 ..
 
.. 
. . an−1 bn−1 cn−1 
0 ··· 0 an bn

I Gaussian elimination without pivoting reduces to

d1 = b1
for i = 2 to n
mi = ai /di−1
di = bi − mi ci−1
end
 94

Tridiagonal Matrices, continued

I LU factorization of A is then given by

   
1 0 ··· ··· 0 d1 c1 0 ··· 0
 .. ..   .. .. 
m
 2 1 . . 
0
 d2 c2 . . 

 .. .. .. ..  . .. .. .. 
L=
0 . . . . ,  ..
U = . . . 0 
 
 ..  ..
   
..  .. 
 . . mn−1 1 0 . . dn−1 cn−1 
0 ··· 0 mn 1 0 ··· ··· 0 dn
 95

General Band Matrices

I In general, band system of bandwidth β requires O(βn) storage, and

its factorization requires O(β 2 n) work

I Compared with full system, savings is substantial if β  n

 96

Iterative Methods for Linear Systems

I Gaussian elimination is direct method for solving linear system,

producing exact solution in finite number of steps (in exact
arithmetic)
I Iterative methods begin with initial guess for solution and
successively improve it until desired accuracy attained
I In theory, it might take infinite number of iterations to converge to
exact solution, but in practice iterations are terminated when
residual is as small as desired
I For some types of problems, iterative methods have significant
advantages over direct methods
I We will study specific iterative methods later when we consider
solution of partial differential equations
 97

Software for Linear Systems

 98

LINPACK and LAPACK

I LINPACK is software package for solving wide variety of systems of
linear equations, both general dense systems and special systems,
such as symmetric or banded

I Solving linear systems is of such fundamental importance in

scientific computing that LINPACK has become standard benchmark
for comparing performance of computers

I LAPACK is more recent replacement for LINPACK featuring higher

performance on modern computer architectures, including many
parallel computers

I Both LINPACK and LAPACK are available from Netlib.org

I Linear system solvers underlying MATLAB and Python’s NumPy and

SciPy libraries are based on LAPACK
 99

BLAS – Basic Linear Algebra Subprograms

I High-level routines in LINPACK and LAPACK are based on lower-level

Basic Linear Algebra Subprograms (BLAS)

I BLAS encapsulate basic operations on vectors and matrices so they

can be optimized for given computer architecture while high-level
routines that call them remain portable

I Higher-level BLAS encapsulate matrix-vector and matrix-matrix

operations for better utilization of memory hierarchies such as cache
and virtual memory with paging

I Generic versions of BLAS are available from Netlib.org, and many

computer vendors provide custom versions optimized for their
particular systems
 100

Examples of BLAS

Level Data Work Examples Function

1 O(n) O(n) saxpy Scalar × vector + vector
sdot Inner product
snrm2 Euclidean vector norm
2 O(n2 ) O(n2 ) sgemv Matrix-vector product
strsv Triangular solution
sger Rank-one update
3 O(n2 ) O(n3 ) sgemm Matrix-matrix product
strsm Multiple triang. solutions
ssyrk Rank-k update

Level-3 BLAS have more opportunity for data reuse, and hence higher
performance, because they perform more operations per data item than
lower-level BLAS
 101

Summary - Solving Linear Systems

I Solving linear systems is fundamental in scientific computing

I Sensitivity of solution to linear system is measured by cond(A)

I Triangular linear system is easily solved by successive substitution

I General linear system can be solved by transforming it to triangular

form by Gaussian elimination (LU factorization)

I Pivoting is essential for stable implementation of Gaussian

elimination

I Specialized algorithms and software are available for solving

particular types of linear systems