Errors in Solutions to Systems of Linear Equations

System of Equations
Errors in Solutions to Systems of Linear Equations
Objective: Solve [A]{x} = {b}
Problem:
Round-off errors may accumulate and even be
exaggerated by the solution procedure. Errors are
often exaggerated if the system is ill-conditioned
Possible remedies to minimize this effect:
1. Partial or complete pivoting
2. Work in double precision
3. Transform the problem into an equivalent system of
linear equations by scaling or equilibrating

E. T. S. I. Caminos, Canales y Puertos 1

Errors in Solutions to Systems of Linear Equations

Ill-conditioning
• A system of equations is singular if det|A| = 0
• If a system of equations is nearly singular it is ill-conditioned.

Systems which are ill-conditioned are extremely sensitive to

small changes in coefficients of [A] and {b}. These systems
are inherently sensitive to round-off errors.

Question:
Can we develop a means for detecting these situations?

E. T. S. I. Caminos, Canales y Puertos 2

Errors in Solutions to Systems of Linear Equations

Ill-conditioning of [A]{x} = {b}:
Consider the graphical interpretation for a 2-equation system:

 a11 a12   x1   b1  (1)

a   =  
 21 a 22   x 2  b 2  (2)
We can plot the two linear equations on a graph of x 1 vs. x2.

x2 a11x1+ a12x2 = b1

b2/a22 b2/a21
b1/a12
x1
b1/a11 a21x1+ a22x2 = b2
E. T. S. I. Caminos, Canales y Puertos 3
Errors in Solutions to Systems of Linear Equations
Ill-conditioning of [A]{x} = {b}:
Consider the graphical interpretation for a 2-equation system:
 a11 a12   x1   b1  (1)
a   =  
 21 a 22   x 2  b2  (2)
We can plot the two linear equations on a graph of x 1 vs. x2.

x1 x1

x2 x2
Uncertainty
Uncertainty
in x2
in x2
Well-conditioned Ill-conditioned
E. T. S. I. Caminos, Canales y Puertos 4

Errors in Solutions to Systems of Linear Equations

Ways to detect ill-conditioning:

1. Calculate {x}, make small change in [A] or {b} and
determine the change in the solution {x}.
2. After forward elimination, examine diagonal of upper
triangular matrix. If aii << ajj, i.e. there is a relatively
small value on diagonal, then this may indicate ill-
conditioning.
3. Compare {x}single with {x}double
4. Estimate the "condition number" for A.
Substituting the calculated {x} into [A]{x} and checking this
against {b} will not always work!!!

E. T. S. I. Caminos, Canales y Puertos 5

Errors in Solutions to Systems of Linear Equations

Ways to detect ill-conditioning:

• If det|A| = 0 the matrix is singular

==> the determinant may be an indicator of conditioning
• If det|A| is near zero is the matrix ill-conditioned?
• Consider:  0.01 0 0 
det  0 0.01 0   106
 0 0 0.01

After scaling: 1 0 0
det  0 1 0   1
 0 0 1 
==> det|A| will provide an estimate of conditioning if it is
normalized by the "magnitude" of the matrix.
E. T. S. I. Caminos, Canales y Puertos 6
 Norms

Norms and the Condition Number

We need a quantitative measure of ill-conditioning.
This measure will then directly reflect the possible magnitude
of round off effects.
To do this we need to understand norms:

Norm: Scalar measure of the magnitude of a matrix or vector

("how big" a vector is).

Not to be confused with the dimension of a matrix.

E. T. S. I. Caminos, Canales y Puertos 7

Vector Norms
Vector Norms: Scalar measure of the magnitude of a vector
Here are some vector norms for n x 1 1/ p
 n 
vectors {x} with typical elements xi.
Each is in the general form of a p norm
x p



x
i 1
i
p



defined by the general relationship:
 n 
1. Sum of the magnitudes: 
1 

 i 1
xi 
x



2. Magnitude of largest element: x 
 max xi
i
(infinity norm)
1/ 2
 n 
3. Length or Euclidean norm: x 2



x
i 1
i
2



E. T. S. I. Caminos, Canales y Puertos 8

Norms

Vector Norms
Required Properties of vector norm:
1. ||x||  0 and ||x|| = 0 if and only if [x]=0
2 ||kx|| = k ||x|| where k is any positive scalar
3. ||x+y||  ||x|| + ||y|| Triangle Inequality
For the Euclidean vector norm we also have
4. ||x•y||  ||x|| ||y||
because the dot product or inner product property
satisfies:
||xy|| = ||x||•||y|| |cos()|  ||x|| • ||y||.
E. T. S. I. Caminos, Canales y Puertos 9
 Matrix Norms

Matrix Norms: Scalar measure of the magnitude of a matrix.

Matrix norms corresponding to vector norms above are defined
by the general relationship:

A p
 max  A   x p
x 1

n
1. Largest column sum: A 1 max
j a ij
(column sum norm) i 1

n
2. Largest row sum:
(row sum norm)
A 
 max
i a
j1
ij

(infinity norm)
E. T. S. I. Caminos, Canales y Puertos 10

Matrix norms

3. Spectral norm: ||A|| 2 = (µmax)1/2

where µmax is the largest eigenvalue of [A]T[A]
If [A] is symmetric, (µmax)1/2 = max , is the largest
eigenvalue of [A].

(Note: this is not the same as the Euclidean or Frobenius

norm, seldom used:

n n
Ae  A
i 1 j 1
2
ij

E. T. S. I. Caminos, Canales y Puertos 11

Matrix norms

Matrix Norms
For matrix norms to be useful we require that
0. || Ax ||  || A || ||x ||
General properties of any matrix norm:
1. || A ||  0 and || A || = 0 iff [A] = 0
2. || k A || = k || A || where k is any positive scalar
3. || A + B ||  || A || + || B || "Triangle Inequality"
4. || A B ||  || A || || B ||
Why are norms important?
 Norms permit us to express the accuracy of the solution {x} in
terms of || x ||
 Norms allow us to bound the magnitude of the product [ A ] {x}
and the associated errors.

E. T. S. I. Caminos, Canales y Puertos 12

 Error Analysis

Forward and backward error analysis can estimate the effect of truncation
and roundoff errors on the precision of a result. The two approaches are
alternative views:
1. Forward (a priori) error analysis tries to trace the accumulation of
error through each process of the algorithm, comparing the
calculated and exact values at every stage.
2. Backward (a posteriori) error analysis views the final solution as the
exact solution to a perturbed problem. One can consider how
different the perturbed problem is from the original problem.
Here we use the condition number of a matrix [A] to specify the amount
by which relative errors in [A] and/or {b} due to input, truncation, and
rounding can be amplified by the linear system in the computation of {x}.

E. T. S. I. Caminos, Canales y Puertos 13

Error Analysis

Backward Error Analysis of [A]{x} = {b} for errors in {b}

Suppose the coefficients {b} are not precisely represented. What might
be the effect on the calculated value for {x + dx}?

Lemma: [A]{x} = {b} yields ||A|| ||x||  ||b|| or

b 1 A
x   
A x b
Now an error in {b} yields a corresponding error in {x}:
[A ]{x + dx} = {b + db}

[A]{x} + [A]{ dx} = {b} + {db}

Subtracting [A]{x} = {b} yields:

[A]{dx} = {db} ––> {dx} = [A]-1{db}
E. T. S. I. Caminos, Canales y Puertos 14

Error Analysis
Backward Error Analysis of [A]{x} = {b} for errors in {b}
1
Taking norms we have: x  A b

And using the lemma: 1 A


x b

we then have : x b b
 A A1 
x b b
Define the condition number as
k = cond [A]  ||A-1|| ||A||  1
If k  1 or k is small, the system is well-conditioned
If k >> 1, system is ill conditioned.
1 = || I || = || A-1A ||  || A-1 || || A || = k = Cond(A)
E. T. S. I. Caminos, Canales y Puertos 15
 Error Analysis
Backward Error Analysis of [A]{x} = {b} for errors in [A]
If the coefficients in [A] are not precisely represented, what
might be effect on the calculated value of {x+ dx}?
[A + dA ]{x + dx} = {b}
[A]{x} + [A]{ dx} + [dA]{x+dx} = {b}
Subtracting [A]{x} = {b} yields:
[A]{ dx} = – [dA]{x+dx}
or
{dx} = – [A]-1 [dA] {x+dx}
Taking norms and multiplying by || A || / || A || yields :
x A A
 A A 1 
x  x A A
E. T. S. I. Caminos, Canales y Puertos 16

Error Analysis
Estimate of Loss of Significance:
Consider the possible impact of errors [dA] on the precision of {x}.
A x A
If ~ 10 p , then 
A x  x A
x
implies that if ~ 10s , then 10s   10  p
x  x
Or, taking log of both sides: s > p - log10()
• log10() is the loss in decimal precision; i.e., we start with p
decimal figures and end-up with s decimal figures.
• It is not always necessary to find [A] -1 to estimate k = cond[A].
Instead, use an estimate based upon iteration of inverse matrix
using LU decomposition.
E. T. S. I. Caminos, Canales y Puertos 17