Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

5. Direct Methods for Solving Systems of Linear Equations

They are all over the place . . .

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 1 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

5.1. Preliminary Remarks

Systems of Linear Equations

• Another important field of application for numerical methods is numerical linear

algebra that deals with solving problems of linear algebra numerically
(matrix-vector product, finding eigenvalues, solving systems of linear equations).
• Here, the solution of systems of linear equations, i.e.

for A = (ai,j )1≤i,j≤n ∈ Rn,n , b = (bi )1≤i≤n ∈ Rn ,

find x ∈ Rn mit A · x = b ,
is of major significance. Linear systems of equations are omnipresent in numerics:

– interpolation: construction of the cubic spline interpolant (see section 3.3)

– boundary value problems (BVP) of ordinary differential equations (ODE) (see
chapter 8)
– partial differential equations (PDE)
– ...

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 2 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

• In terms of the problem given, one distinguishes between:

– full matrices: the number of non-zero values in A is of the same order of
magnitude as the number of all entries of the matrix, i.e. O(n2 ).
– sparse matrices: here, zeros clearly dominate over the non-zeros (typically
O(n) or O(n log(n)) non-zeros); those sparse matrices often have a certain
sparsity pattern (diagonal matrix, tridiagonal matrix (ai,j = 0 for |i − j| > 1),
general band structure (ai,j = 0 for |i − j| > c) etc.), which simplifies solving
the system.
0 ∗ 0 ∗ ∗
1 1
B ∗ C B ∗ ∗ ∗ C
∗ ∗ ∗ ∗
B C B C
B C B C
∗ ∗ ∗ ∗
B C B C
B C B C
∗ ∗ ∗ ∗
B C B C
B C B C
∗ ∗ ∗ ∗
B C B C
B C B C
∗ ∗ ∗ ∗
B C B C
B C B C
@ ∗ A @ ∗ ∗ ∗ A
∗ ∗ ∗

diagonal tridiagonal
0 ∗ ∗ ∗ 1 0 ∗ ∗ ∗ 1
B ∗ ∗ C B ∗ ∗ ∗ C
B ∗ ∗ B ∗ ∗ ∗
C B C
B C C
B ∗ ∗ C B
∗ ∗ ∗
C
←
→ ...
B C B C
∗ 2c ∗ ∗ ∗ ∗
B C B C
B C B C
∗ ∗ ∗ ∗ ∗
B C B C
B C B C
∗ ∗ ∗ ∗ ∗
B C B C
B C B C
@
∗ ∗
A @ ∗ ∗ ∗ A
∗ ∗ ∗ ∗ ∗ ∗

band (bandwidth 2c) block-diagonal

5. Direct Methods for Solving Systems of Linear Equations
Numerical Programming I (for CSE), Hans-Joachim Bungartz page 3 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

Systems of Linear Equations (2)

• One distinguishes between different solution approaches:

– direct solvers: provide the exact solution x (modulo rounding errors)
(covered in this chapter);
– indirect solvers: start with a first approximation x(0) and compute iteratively
a sequence of (hopefully increasingly better) approximations x(i) , without
ever reaching x (covered in chapter 7).
• Reasonably, we will assume in the following an invertible or non-singular matrix A,
i.e. det(A) 6= 0 or rank(A) = n or Ax = 0 ⇔ x = 0, respectively.
• Two approaches that seem obvious at first sight are considered as numerical
mortal sins for reasons of complexity:
– x := A−1 b, i.e. the explicit computation of the inverse of A;
– The use of Cramer’s rule (via the determinant of A or rather the n matrices
which result from A by substituting a column with the right hand side b).

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 4 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

• Of course, the following general rule also applies to numerical linear algebra:
Have a close look at the way a problem is posed before starting, because even the
simple term

y := A · B · C · D · x , A, B, C, D ∈ Rn , x ∈ Rn ,

can be calculated stupidly via

y := (((A · B) · C) · D) · x

with O(n3 ) operations (matrix-matrix products!) or efficiently via

y := A · (B · (C · (D · x)))

with O(n2 ) operations (only matrix-vector products!)!

• Keep in mind for later: Being able to apply a linear mapping in form of a matrix (i.e.
to be in control of its effect on an arbitrary vector) is generally a lot cheaper than
via the explicit design of the matrix!

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 5 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

Vector Norms

• In order to analyze the condition of the problem of solving systems of linear

equations as well as to analyze the behavior of convergence of iterative methods
in chapter 7, we need a concept of norms for vectors and matrices.
• A vector norm is a function k.k : Rn → R with the three properties
– positivity: kxk > 0 ∀x 6= 0;
– homogeneity: kαxk = |α| · kxk for arbitrary α ∈ R;
– triangle inequality: kx + yk ≤ kxk + kyk.
• The set {x ∈ Rn : kxk = 1} is called norm sphere regarding the norm k.k.

• Examples for vector norms relevant in our context (verify the above norm
attributes for every one):
Pn
– Manhattan norm: kxk1 := i=1 |xi |
n
pP
– Euclidean norm: kxk2 := 2
i=1 |xi | (the common vector length)
– maximum norm: kxk∞ := max1≤i≤n |xi |

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 6 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

Matrix Norms
• By extending the concept of vector norm, a matrix norm can be defined or rather
induced according to
kAk := max kAxk .
kxk=1

In addition to the three properties of a vector norm (rephrased correspondingly),

which also apply here, a matrix norm is
– sub-multiplicative: kABk ≤ kAk · kBk;
– consistent: kAxk ≤ kAk · kxk.
• The condition number κ(A) is defined as

maxkxk=1 kAxk
κ(A) := .
minkxk=1 kAxk
– κ(A) indicates how strongly the norm sphere is deformed by the matrix A or
by the respective linear mapping.
– In case of the identity matrix I (ones in the diagonal, zeros everywhere else)
and for certain classes of matrices there are no deformations at all – in these
cases we have κ(A) = 1 .
– For non-singular A, we have
κ(A) = kAk · kA−1 k .
5. Direct Methods for Solving Systems of Linear Equations
Numerical Programming I (for CSE), Hans-Joachim Bungartz page 7 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

The Condition of Solving Linear Systems of Equations

• Now, we can begin to determine the condition of the problem of solving a system
of linear equations:
– In (A + δA)(x + δx) = b + δb,
we now have to deduce the error δx of the result x from the perturbations
δA, δb of the input A, b . Of course, δA has to be so small that the perturbed
matrix remains invertible (this holds for example for changes with
kδAk < kA−1 k−1 ).
– Solve the relation above for δx and estimate with the help of the
sub-multiplicativity and the consistency of an induced matrix norm (this is
what we needed it for!):
kA−1 k
kδxk ≤ · (kδbk + kδAk · kxk) .
1 − kA−1 k · kδAk
– Now, divide both sides by kxk and bear in mind that the relative input
perturbations kδAk/kAk as well as kδbk/kbk should be bounded by ε
(because we assume small input perturbations when analyzing the
condition).
– With this, it follows „ «
kδxk εκ(A) kbk 2εκ(A)
≤ · +1 ≤
kxk 1 − εκ(A) kAk · kxk 1 − εκ(A)
because kbk = kAxk ≤ kAk · kxk.
5. Direct Methods for Solving Systems of Linear Equations
Numerical Programming I (for CSE), Hans-Joachim Bungartz page 8 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

The Condition of Solving Linear Systems of Equations (2)

• Because it’s so nice and so important, once more the result we achieved for the
condition:
kδxk 2εκ(A)
≤ .
kxk 1 − εκ(A)

– The bigger the condition number κ(A) is, the bigger our upper bound on the
right for the effects on the result becomes, the worse the condition of the
problem “solve Ax = b” gets.
– The term “condition number” therefore is chosen reasonably – it represents a
measure for condition.
– Only if εκ(A)  1, which is restricting the order of magnitude of the
acceptable input perturbations, a numerical solution of the problem makes
sense. In this case, however, we are in control of the condition.
– At the risk of seeming obtrusive: This only has to do with the problem (i.e. the
matrix) and nothing to do with the rounding errors or approximate
calculations!
• Note: The condition of a problem can often be improved by adequate rearranging.
If the system matrix A in Ax = b is ill-conditioned, the condition of the new system
matrix M A in M Ax = M b might be improved by choosing a suitable prefactor M .

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 9 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

The Residual

• An important quantity is the residual r. For an approximation x̃ of x, r is defined

as
r := b − Ax̃ = A(x − x̃) =: −Ae
with the error e := x̃ − x .
• Caution: Error and residual can be of very different order of magnitude. In
particular, from r = O(ε̄) does not at all follow e = O(ε̄) – the correlation even
contains the condition number κ(A), too.
• Nevertheless the residual is helpful: Namely,

r = b − Ax̃ ⇔ Ax̃ = b − r

shows that x̃ can be interpreted as exact result of slightly perturbed input data
(A original, instead of b now b − r) for small residual; therefore, x̃ is an
acceptable result in terms of chapter 2!
• We will mainly use the residual for the construction of iterative solving methods in
chapter 7: In contrast to the unknown error, the residual can easily be determined
in every iteration step.

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 10 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

5.2. Gaussian Elimination

The Principle of Gaussian Elimination

• The classical solution method for systems of linear equations, familiar from linear
algebra, is Gaussian elimination, the natural generalization of solving two
equations with two unknowns:
– Solve one of the n equations (e.g. the first one) for one of the unknowns (e.g.
x1 ).
– Replace x1 by the resulting term (depending on x2 , . . . , xn ) in the other
n − 1 equations – therefore, x1 is eliminated from those.
– Solve the resulting system of n − 1 equations with n − 1 unknowns
analogously and continue until an equation only contains xn , which can
therefore be explicitly calculated.
– Now, xn is inserted into the elimination equation of xn−1 , so xn−1 can be
given explicitly.
– Continue until at last the elimination equation of x1 provides the value for x1
by inserting the values for x2 , . . . , xn (known by now).
• Simply spoken, the elimination means that A and b are modified such that there
are only zeros below a1,1 in the first column. Note that the new system (consisting
of the first equation and the remaining x1 -free equations), of course, is solved by
the same vector x as the old one!
5. Direct Methods for Solving Systems of Linear Equations
Numerical Programming I (for CSE), Hans-Joachim Bungartz page 11 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

The Algorithm

• Those thoughts result in the following algorithm:

Gaussian elimination:

for j from 1 to n do
for k from j to n do r[j,k]:=a[j,k] od;
y[j]:=b[j];
for i from j+1 to n do
l[i,j]:=a[i,j]/r[j,j];
for k from j+1 to n do a[i,k]:=a[i,k]-l[i,j]*r[j,k] od;
b[i]:=b[i]-l[i,j]*y[j]
od
od;
for i from n downto 1 do
x[i]:=y[i];
for j from i+1 to n do x[i]:=x[i]-r[i,j]*x[j] od;
x[i]:=x[i]/r[i,i]
od;

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 12 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

initial system

1 2 −2 −1 1 −8 1 0 0 0 0
0 1 0 1 0 1
B 2 3 −3 2 3 C B −34 C B 0 1 0 0 0 C
1 2 5 3 −2 C , 43 C , 0 0 1 0 0
B C B C B C
B B B C
@ 3 −3 2 1 −2 A @ 19 A @ 0 0 0 1 0 A
1 2 3 −1 4 57 0 0 0 0 1

first column eliminated

1 2 −2 −1 1 −8 1 0 0 0 0
0 1 0 1 0 1
B 0 −1 1 4 1 C B −18 C B 2 1 0 0 0 C
B 0 0 7 4 −3 C , B 51 C , B 1 0 1 0 0
B C B C B C
C
@ 0 −9 8 4 −5 A @ 43 A @ 3 0 0 1 0 A
0 0 5 0 3 65 1 0 0 0 1

second column eliminated

1 2 −2 −1 1 −8 1 0 0 0 0
0 1 0 1 0 1
B 0 −1 1 4 1 C B −18 C B 2 1 0 0 0 C
B 0 0 7 4 −3 C , B 51 C , B 1 0 1 0 0
B C B C B C
C
@ 0 0 −1 −32 −14 A @ 205 A @ 3 9 0 1 0 A
0 0 5 0 3 65 1 0 0 0 1

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 13 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

third column eliminated

0 1 0 1 0 1
1 2 −2 −1 1 −8 1 0 0 0 0
B
B 0 −1 1 4 1 C
C
B −18 C
B C
B 2
B 1 0 0 0 C
C
B 0 0 7 4 −3 C , B 51 C , B 1 0 1 0 0 C
B −220 −101 C B 1486 C B −1 C
@ 0 0 0 7 7
A @ A
7
@ 3 9 7
1 0 A
−20 36 200 5
0 0 0 7 7 7
1 0 7
0 1
last column eliminated
0 1 0 1 0 1
1 2 −2 −1 1 −8 1 0 0 0 0
B
B 0 −1 1 4 1 C
C
B −18 C
B C
B 2
B 1 0 0 0 C
C
R := B 0 0 7 4 −3 C , B 51 C , B 1 0 1 0 0 C =: L
B −220 −101 C B 1486 C B −1 C
@ 0 0 0 7 7
A @ A
7
@ 3 9 7
1 0 A
497 714 5 1
0 0 0 0 77 77
1 0 7 11
1
factors L and R
0 10 1
1 0 0 0 0 1 2 −2 −1 1 0
1 2 −2 −1 1
1
B
B 2 1 0 0 0 CB
CB 0 −1 1 4 1 C
C B 2 3 −3 2 3 C
B 1 0 1 0 0 C·B 0 0 7 4 −3 C = B 1
B
2 5 3 −2
C
C
B −1 CB −220 −101 C
@ 3 9 7
1 0 A@ 0 0 0 7 7
A @ 3 −3 2 1 −2 A
5 1 497 1 2 3 −1 4
1 0 7 11
1 0 0 0 0 77

we have L · R = A

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 14 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

Discussion of Gaussian Elimination

• outer j-loop: eliminate variables (i.e. cancel subdiagonal columns) one after the
other
• first inner k-loop: store the part of the row j located in the upper triangular part
in the auxiliary matrix R (we only eliminate below the diagonal);
store the (modified) right hand side bj in yj ;
the values rj,k stored this way as well as the values yj won’t be changed anymore
in the following process!
• inner i-loop:
– determine the required factors in column j for the cancellation of the entries
below the diagonal and store them in the auxiliary matrix L (here, we silently
assume rj,j 6= 0 for the first instance)
– subtract the corresponding multiple of the row j from row i
– also modify the right side accordingly (such that the solution x won’t be
changed)
• finally: substitute the calculated components of x backwards (i.e. starting with xn )
in the equations stored in R and y (ri,i 6= 0 is silently assumed again)
• Counting the arithmetic operations accurately gives a total effort of
2 3
n + O(n2 )
3
basic arithmetic operations.
5. Direct Methods for Solving Systems of Linear Equations
Numerical Programming I (for CSE), Hans-Joachim Bungartz page 15 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

Discussion of Gaussian Elimination (2)

• Together with the matrix A, the matrices L and R appear in the algorithm of
Gaussian elimination. We have (cf. algorithm):
– In R, only the upper triangular part (inclusive the diagonal) is populated.
– In L, only the strict lower trianguler part is populated (without the diagonal).
– If filling the diagonal in L with ones, we get the fundamental relation

A = L·R.

Such a decomposition or factorization of a given matrix A in factors with

certain properties (here: triangular form) is a very basic technique in
numerical linear algebra.
• The insight above means for us: Instead of using the classical Gauss elimination,
we can solve Ax = b with the triangular decomposition A = LR, namely with the
algorithm resulting from

Ax = LRx = L(Rx) = Ly = b.

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 16 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

• algorithm of the LR decomposition:

– 1. step: triangular decomposition: decompose A into factors L (lower
triangular matrix with ones in the diagonal) and R (upper triangular matrix);
the decomposition is – with this specification of the respective diagonal
values – unique!
– 2. step: forward substitution: solve Ly = b (by inserting, from y1 to yn )
– 3. step: backward substitution: solve Rx = y (by inserting, from xn to x1 )
• In English, this is usually denoted as LU factorization with a lower matrix L and
an upper matrix U .

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 17 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

LU Factorization

• The previous considerations lead to the following algorithm:

LU factorization:
for i from 1 to n do
for k from 1 to i-1 do
l[i,k]:=a[i,k];
for j from 1 to k-1 do l[i,k]:=l[i,k]-l[i,j]*u[j,k] od;
l[i,k]:=l[i,k]/u[k,k]
od;
for k from i to n do
u[i,k]:=a[i,k];
for j from 1 to i-1 do u[i,k]:=u[i,k]-l[i,j]*u[j,k] od
od
od;
for i from 1 to n do
y[i]:=b[i];
for j from 1 to i-1 do y[i]:=y[i]-l[i,j]*y[j] od
od;
for i from n downto 1 do
x[i]:=y[i];
for j from i+1 to n do x[i]:=x[i]-u[i,j]*x[j] od;
x[i]:=x[i]/u[i,i]
od;

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 18 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

Discussion of LU factorization
• To comprehend the first part (the decomposition into the factors L and R), start
with the general formula for the matrix product:
n
X
ai,k = li,j · uj,k .
j=1

However, here quite unusually A is known, and L and R have to be determined

(the fact that this works is due to the triangular shape of the factors)!
– In case of i > k, one only has to sum up to j = k, and li,k can be determined
(solve the equation for li,k : everything on the right hand side is known
already):
0 1
k−1
X k−1
X
ai,k = li,j ·uj,k +li,k ·uk,k and, therefore, li,k := ai,k −
@ li,j · uj,k A /uk,k .
j=1 j=1

– If i ≤ k, one only has to sum up to j = i, and ui,k can be determined (solve

the equation for ui,k : again, all quantities on the right are already given; note
li,i = 1):
i−1
X i−1
X
ai,k = li,j · uj,k + li,i · ui,k and, therefore, ui,k := ai,k − li,j · uj,k .
j=1 j=1

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 19 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

Discussion of LU Factorization (2)

• It is clear: If you fight your way through all variables with increasing row and
column indices (the way it’s happening in the algorithm, starting at i = k = 1),
every li,k and ui,k can be calculated one after the other – on the right hand side,
there is always something that has already been calculated!
• The forward substitution (second i-loop) follows and – as before at Gaussian
elimination – the backward substitution (third and last i loop).
• It can be shown that the two methods “Gauss elimination” and “LU factorization”
are identical in terms of carrying out the same operations (i.e. they particularly
have the same cost); only the order of the operations is different!
• In the special case of positive definite matrices A (A = AT and xT Ax > 0 for all
x 6= 0), this can be accomplished even cheaper than with the algorithm just
shown:
– decompose the factor U of A = LU into a diagonal matrix D and an upper
triangular matrix Ũ with ones at the diagonal (this is always possible):

A = L · U = L · D · Ũ with D = diag(u1,1 , . . . , un,n ) ;

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 20 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

– then, it follows from the symmetry of A

AT = (L · D · Ũ )T = Ũ T · D · LT = L · D · Ũ = A ,

and the uniqueness of the decomposition forces L = Ũ T and thus

A = L · D · LT =: L̃ · L̃T ,

if splitting the diagonal factor D in equal shares into the triangular factors
√
( ui,i in both diagonals; the values ui,i are all positive because A is positive
definite).

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 21 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

The Cholesky Factorization

• The method described above, with which the calculation of the ui,k , i 6= k, in the
LU factorization can be avoided and with that about half of the total computing
time and required memory, is called Cholesky factorization oder Cholesky
decomposition. We write A = LLT .

Cholesky factorization:

for k from 1 to n do
l[k,k]:=a[k,k];
for j from 1 to k-1 do l[k,k]:=l[k,k]-l[k,j]ˆ2 od;
l[k,k]:=(l[k,k])ˆ0.5;
for i from k+1 to n do
l[i,k]:=a[i,k];
for j from 1 to k-1 do l[i,k]:=l[i,k]-l[i,j]*l[k,j] od;
l[i,k]:=l[i,k]/l[k,k]
od
od;
• Of course, the algorithm above only delivers the triangular decomposition. As
before, forward and backward substitution still have to be carried out to solve the
system of linear equations Ax = b.

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 22 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

Discussion of the Cholesky Factorization

• At the construction of the Cholesky algorithm, we once more start with the formula
for the matrix product:
Xn k
X k
X
T T
ai,k = li,j · lj,k = li,j · lj,k = li,j · lk,j , i ≥ k.
j=1 j=1 j=1

• From this, calculate L (i.e. the lower triangular part) column by column, starting in
every column with the diagonal element
v
k
X
u
u k−1
X
2 2 ,
ak,k = lk,j , lk,k := tak,k − lk,j
j=1 j=1

and then process the remaining rows i > k:

0 1
X k k−1
X
ai,k = li,j · lk,j , li,k := @ai,k − li,j · lk,j A /lk,k .
j=1 j=1

• As mentioned earlier, the cost decreases to about half, i.e.

1 3
n + O(n2 ) .
3
• To close this chapter, we now turn to the division by the diagonal element, so far
always silently assumed to be possible.
5. Direct Methods for Solving Systems of Linear Equations
Numerical Programming I (for CSE), Hans-Joachim Bungartz page 23 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

5.3. Choice of Pivot

Pivots
• In the algorithm just introduced, we assumed that divisions of the kind ai,j /uj,j or
xi /ui,i do not cause any problems, i.e. particularly that there occur no zeros in
the diagonal of U .
• As everything centers on those values in the diagonal, they are called Pivots
(French word).
• In the positive definite case, all eigenvalues are positive, which is why zeros are
impossible in the diagonal – i.e. in the Cholesky methods everything is alright.
• However, in the general case, the requirement uj,j 6= 0 for all j is not granted. If a
zero emerges, the algorithm has to be modified, and a feasible situation, i.e. a
non-zero in the diagonal, has to be forced by permutations of rows or columns
(which is possible, of course, when A is not singular!). Consider for example the
matrix 0 1
1 1 0
A= @ 1 1 2 A.
0 1 1
• A possible partner for exchange of a zero ui,i in the diagonal can be found either
in the column i below the diagonal (column pivot search) or in the entire
remaining matrix (everything from the row and the column i + 1 onward, total
pivot search).
5. Direct Methods for Solving Systems of Linear Equations
Numerical Programming I (for CSE), Hans-Joachim Bungartz page 24 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

Pivot Search

• Partial pivoting or column pivot search:

If ui,i = 0, then one has to search in the column i below the row i for an entry
ak,i 6= 0, k = i + 1, . . . , n:
– If there is no such ak,i 6= 0: The remaining matrix has a first column that
completely vanishes, which means det(A) = 0 (case of error).
– If there are non-zeros: Then usually the element with the biggest absolute
value (let it be ak,i ) is chosen as pivot and the rows k and i in the matrix and
on the right hand side are switched.
– Big pivots are convenient because they lead to small elimination factors lk,i
and they do not increase the entries in L and U too much.
– Of course, such a switching of rows does not change the system of equations
and its solution at all!
• Total pivoting or total pivot search:
Here, one searches not only in the column i of the remaining matrix but in the total
remaining matrix, instead of an exchange of rows also exchanges of columns
(rearranging of the unknowns xk ) can be realized here; total pivoting therefore is
more expensive.
• Even if no zeros occur – for numerical reasons, pivoting is always advisable!

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 25 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

5.4. Applications for Direct Solving Methods

Systems of Linear Equations in CSE
• From the multitude of problems that lead to solving a system of linear equations,
we pick one: the so called radiosity method in computer graphics with which
photo-realistic images can be produced. Radiosity is particularly suited for
ambient light (fume, fog, dust in sunlight . . . ).
– The image generation is carried out in four steps: Division of the entire
surface of the scene into several patches, computation of the form factors
between the patches (they describe the light flow), setting up and solving of
the radiosity system of equations (aha!) with the radiosity values (energy
densities) as variables, rendering of the patches with the calculated radiosity
values.
– For the radiosity Bi of the patch i, the relation
n
X
Bi = Ei + %i · Bj Fij ,
j=1

holds, where Ei denotes the emissivity of the patches (how much light is
newly produced and emitted – especially important for light sources), %i the
absorption and Fij the form factor. In short: On patch i there is the light
which is produced (emitted) there or has arrived from other patches.
5. Direct Methods for Solving Systems of Linear Equations
Numerical Programming I (for CSE), Hans-Joachim Bungartz page 26 of 27
Preliminary Remarks Gaussian Elimination Choice of Pivot Applications

– For Fij the relations

n
cos θi cos θj X
Fij = Aj Vij , Fii = 0 , Fij = 1
πr2 j=1

hold. Here, θi denotes the angle between the normal of patch i and the
vector of length r which connects the centers of the patches i and j. The
term Aj denotes the area of patch j, and Vij indicates the visibility (1 for free
sight from patch i to patch j, 0 else).
• The relation above obviously forms a system of linear equations – n equations
with n variables Bi .

5. Direct Methods for Solving Systems of Linear Equations

Numerical Programming I (for CSE), Hans-Joachim Bungartz page 27 of 27