Contents

1 Resolution of linear systems . . . . . . . . . . . . . . . . . . . . . . . . 5

1.1 Matrix Theory: Definitions and Key Properties . . . . . . . . . . . . . . . . 6
1.2 Direct methods for solving linear systems . . . . . . . . . . . . . . . . . . . 12
1.2.1 Gaussian Elimination method . . . . . . . . . . . . . . . . . . . . . . 13
1.2.2 Pivoting Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.2.3 Gauss-Jordan Elimination . . . . . . . . . . . . . . . . . . . . . . . 19
1.2.4 LU-Factorisation . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3
Numerical Analysis

4
1 Resolution of linear systems

Kirchhoff’s laws are fundamental tools in electrical circuit analysis. They allow us to derive
linear systems of equations that describe the currents and voltages in a circuit. In this exam-
ple, we will analyze a simple circuit with two loops and three resistors to demonstrate how
Kirchhoff’s laws lead to a system of linear equations.
Consider the following circuit:

R1

V1 R2

R3

Kirchhoff ’s Current Law (KCL)

Kirchhoff’s Current Law states that the sum of currents entering a node is equal to the sum of
currents leaving the node. For the node between R1 , R2 , and R3 , we have:

I1 = I2 + I3

where:
• I1 is the current through R1 ,

• I2 is the current through R2 ,

• I3 is the current through R3 .

Kirchhoff ’s Voltage Law (KVL)
Kirchhoff’s Voltage Law states that the sum of voltages around any closed loop in a circuit is
zero. We apply this law to the two loops in the circuit.

• Loop 1: V1 → R1 → R3
Starting from V1 and moving clockwise:

V1 − I1 R1 − I3 R3 = 0

• Loop 2: R2 → R3
Starting from R2 and moving clockwise:

−I2 R2 + I3 R3 = 0

5
Numerical Analysis

Combining the equations from Kirchhoff’s laws, we obtain the following system of linear
equations: 
I1 − I2 − I3 = 0,

R1 I1 + R3 I3 = V1 ,

−R2 I2 + R3 I3 = 0.


Let us assign the following values to the components:

• V1 = 10 V, R1 = 2 Ω, R2 = 3 Ω and R3 = 5 Ω.

The system becomes: 

I1 − I2 − I3 = 0,

2I1 + 5I3 = 10,

−3I2 + 5I3 = 0.


This latter can be written as

     
1 −1 −1 I1 0
2 0 5   I2  =  10
0 −3 5 I3 0
| {z } | {z } | {z }
A x b

This example demonstrates how Kirchhoff’s laws can be used to derive a system of linear
equations from an electrical circuit. Solving this system allows us to determine the unknown
currents I1 , I2 , and I3 in the circuit.
Linear systems are extensively used across various domains, including engineering, physics, eco-
nomics, computer science, operations research, biology, and finance, among others. They play
a crucial role in modeling, analyzing, and solving real-world problems, ranging from simulating
physical systems to predicting financial trends. Their versatility makes them essential tools in
addressing complex challenges across multiple fields. In this chapter, we will focus on solving
linear systems using both direct and iterative methods. Direct methods yield exact solutions
in a finite number of steps, while iterative methods are particularly useful for large systems
where direct methods may be computationally expensive or infeasible. By exploring these ap-
proaches, we aim to enhance our understanding of how to effectively tackle linear systems in
diverse contexts.

1.1 Matrix Theory: Definitions and Key Properties

Before diving into solving linear systems, it’s essential to recall some fundamental concepts
from linear algebra, particularly matrix definitions, and to expand upon their properties.

• The space of real (resp. complex) m × n matrices is indicated by Rm×n (resp. Cm×n ).

• The inner product of two vectors x = (x1 , x2 , . . . , xn ) and y = (y1 , y2 , . . . , yn ) in Rn is

defined as:
Xn
hx, yi = xi y i
i=1

6
Numerical Analysis

• The inner product of two vectors x = (x1 , x2 , . . . , xn ) and y = (y1 , y2 , . . . , yn ) in Cn is

defined as: n
X
hx, yi = xi y i
i=1

• A matrix is a rectangular array of numbers arranged in rows and columns. A general

matrix A of size m × n is written as:
 
a11 a12 · · · a1n
 a21 a22 · · · a2n 
A =  .. ..  ,
 
.. . .
 . . . . 
am1 am2 · · · amn

where aij represents the element in the ith row and jth column.

• The transpose of a matrix A ∈ Rm×n is denoted by AT ∈ Rn×m , where the rows of A

become columns of AT .  
a11 a21 · · · am1
 a12 a22 · · · am2 
AT =  .. ..  .
 
.. ..
 . . . . 
a1n a2n · · · amn

• The adjoint (or conjugate transpose) of a matrix A ∈ Cm×n is denoted A∗ ∈ Cn×m . It is

defined as
A∗ = ĀT .

• The conjugate of a matrix A, denoted A, is obtained by replacing each element of A

with its complex conjugate.

aij = aij

• Let A ∈ Cm×n and B ∈ Cn×p . Their product AB ∈ Cm×p is defined component-wise by:
n
X
(AB)ij = aik bkj , 1 ≤ i ≤ m, 1 ≤ j ≤ p
k=1

Property 1. (i) Associative: A(BC) = (AB)C

(ii) Distributive: A(B + C) = AB + AC
(iii) Non-commutative: AB 6= BA in general

Let A ∈ Cm×s and B ∈ Cs×n then

– (AB)> = B > A> .

– (AB)∗ = B ∗ A∗ .

• A square matrix is a matrix where the number of rows equals the number of columns
(m = n).

7
Numerical Analysis

• A square matrix is diagonal if all its non-diagonal elements are zero:

 
d11 0 · · · 0
 0 d22 · · · 0 
D =  .. ..  .
 
.. . .
 . . . . 
0 0 · · · dnn

• A square diagonal matrix with all diagonal elements equal to 1. For size n:
 
1 0 ··· 0
0 1 · · · 0
In =  .. .. . . ..  .
 
. . . .
0 0 ··· 1

• The rank of a matrix A, denoted rank(A), is the dimension of the vector space spanned
by its rows or columns. It is equal to the number of linearly independent rows or columns.

• The null space of a matrix A ∈ Cm×n is the set of all vectors x ∈ Cn such that:

Ax = 0

• An n-by-n square matrix A is invertible if there exists an n-by-n square matrix B such
that
AB = BA = In .

• The inverse of a square matrix A, denoted A−1 , is a matrix such that:

AA−1 = A−1 A = In .

In this case, we say that A is non singular.

Let A, B ∈ Cn×n , two invertible matrices then

– The product AB is invertible and (AB)−1 = B −1 A−1 .

– A∗ is invertible and (A∗ )−1 = (A−1 )∗ .

– For any scalar α 6= 0, αA is invertible and (αA)−1 = α1 A−1 .

– A> is invertible and (A> )−1 = (A−1 )> .

– (A−1 )−1 = A.

• A square matrix is:

– Upper triangular if all entries below the main diagonal are zero (aij = 0 for i > j).
– Lower triangular if all entries above the main diagonal are zero (aij = 0 for i < j).

8
Numerical Analysis

– The product of two upper triangular matrices is an upper triangular matrix

and the product of two lower triangular matrices is a lower triangular matrix.

– The transpose of a lower triangular matrix is an upper triangular matrix, and

vice versa.

– The inverse of a lower triangular matrix is a lower triangular matrix, and the
inverse of an upper triangular matrix is an upper triangular matrix.

• An n-by-n square matrix A is called Hermitian if A∗ = A.

• An n-by-n square matrix A is skew-Hermition if A = −A∗ .

• An n-by-n square matrix A is called symmetric if: A = A> .

• An n-by-n square matrix A is skew-symmetric (antisymmetric) if: A = −A> .

• An n-by-n square matrix A is orthogonal if: Q> Q = QQ> = In .

• An n-by-n square matrix A is normal if: AA∗ = A∗ A.

• An n-by-n square matrix A is unitary if: U ∗ U = U U ∗ = In .

• The determinant of a square matrix A ∈ Cn×n , denoted det(A) (or |A|), is a scalar
value that can be computed recursively or using properties like row reduction.

Let A, B ∈ Cn×n , then

– det(AB) = det(BA) = det(A) · det(B).

– the matrix A is invertible if and only if det(A) 6= 0.

– If A is invertible then: det(A−1 ) = 1

det(A)
.

– For any scalar α ∈ C,det(αA) = αn det(A).

– det(In ) = 1.

– The determinant of aQtriangular matrix is equal to the product of its diagonal

elements: det(A) = ni=1 aii .

– det(A> ) = det(A).

– det(A∗ ) = det(A).

• The trace of a square matrix A, denoted tr(A), is the sum of its diagonal elements:
n
X
tr(A) = aii .
i=1

9
Numerical Analysis

Let A, B, C ∈ Cn×n , then

– tr(ABC) = tr(BCA) = tr(CAB).

– For any scalar α ∈ C, tr(αA) = α tr(A).

– tr(In ) = n.

– tr(A> ) = tr(A).

– tr(A∗ ) = tr(A).

• For a square matrix A ∈ Cn×n , a scalar λ is called an eigenvalue if there exists a non-zero
vector v (called an eigenvector) such that:

Av = λv

The set Λ(A) of all eigenvalues is the spectrum.

• The characteristic polynomial of a square matrix A is defined as:

pA (λ) = det(A − λIn )

The roots of pA (λ) are the eigenvalues of A.

Property 2 (Cayley-Hamilton Theorem). Every matrix satisfies its own characteristic
equation: pA (A) = 0.

• A square matrix A is diagonalizable if it can be expressed as:

A = P DP −1

where D is a diagonal matrix and P is an invertible matrix.

Schur decomposition

• For any square matrix A ∈ Cn×n , there exists a unitary matrix U (i.e., U ∗ U = I)
and an upper triangular matrix T such that: A = U T U ∗ .

• For any square normal matrix A ∈ Cn×n , there exists a unitary matrix U (i.e.,
U ∗ U = I) and a diagonal matrix D such that: A = U DU ∗ .

• For any square orthogonal matrix A ∈ Cn×n , there exists an orthogonal matrix O
(i.e., U ∗ U = I) and a diagonal matrix D such that: A = ODO> .

• Let V be a vector space over a field (typically R or C). A norm is a function k·k : V → R
that satisfies the following properties for all vectors u, v ∈ V and all scalars α ∈ K:

1. Positive Definiteness:

kvk ≥ 0, and kvk = 0 ⇐⇒ v = 0.

10
Numerical Analysis

2. Absolute Homogeneity (or Scalar Multiplication):

kαvk = |α| · kvk.

3. Triangle Inequality:
ku + vk ≤ kuk + kvk.
Common norms:
– kvk1 = |v1 | + |v2 | + · · · + |vn | = ni=1 |vi |.
P
1
– kvk2 = |v1 |2 + |v2 |2 + · · · + |vn |2 = ( ni=1 vi2 ) 2 .
p P

– kvk∞ = max(|v1 |, |v2 |, . . . , |vn |) = max1≤i≤n |vi |.

– In general, for p ≥ 1, the p-norm of a vector v = (v1 , v2 , . . . , vn ) is:

n
! p1
X
kvkp = (|v1 |p + |v2 |p + · · · + |vn |p )1/p . = |vi |p
i=1

The Euclidean norm is a special case of the p-norm with p = 2.

• A function k · k : Cm×n → R is a matrix norm if:

(i) kAk ≥ 0 with equality iff A = 0

(ii) kcAk = |c|kAk
(iii) kA + Bk ≤ kAk + kBk
(iv) Submultiplicativity: kABk ≤ kAkkBk

Common norms:
kAxkp
– Operator norm: kAkp = supx6=0 kxkp
.
kAvk1
= max1≤j≤n ni=1 |aij |.
P
– kAk1 = supv∈Cn ,v6=0 kvk1

– kAk2 = supv∈Cn ,v6=0 kAvk

p p
kvk2
2
= ρ(A∗ A) = ρ(AA∗ ) = kA∗ k2 .
– kAk∞ = supv∈Cn ,v6=0 kAvk = max1≤i≤n nj=1 |aij |.
∞
P
kvk∞
qP
– Frobenius norm: kAkF = 2
i,j |aij | .

• For A ∈ Cn×n , the spectral radius is:

ρ(A) = max{|λ| : λ ∈ σ(A)}

For any induced matrix norm: ρ(A) ≤ kAk.

• The condition number of A ∈ Cn×n (invertible) is:

κ(A) = kAkkA−1 k

– For the identity matrix: κ(I) = 1

11
Numerical Analysis

– For singular matrices: κ(A) = ∞

κp (A) = kAkp kA−1 kp .

Perturbation in the Right-Hand Side

Let A be an invertible matrix. Let x and x + ∆x be the respective solutions of the
systems:
Ax = b and A(x + ∆x) = b + ∆b.
If b 6= 0, then the following inequality holds:

k∆xk k∆bk
≤ κ(A) ,
kxk kbk

where κ(A) = kAk · kA−1 k is the condition number of A.

Perturbation in the Matrix

Let A and A + ∆A be two invertible matrices. Let x and x + ∆x be the respective
solutions of the systems:

Ax = b and (A + ∆A)(x + ∆x) = b.

If b 6= 0, then the following inequality holds:

k∆xk k∆Ak
≤ κ(A) .
kx + ∆xk kAk

1.2 Direct methods for solving linear systems

Consider a system of n linear equations with n unknowns:


 a11 x1 + a12 x2 + · · · + a1n xn = b1

a21 x1 + a22 x2 + · · · + a2n xn = b2

.. .. . . . .


 . . . .. ..

a x + a x + · · · + a x = b
n1 1 n2 2 nn n n

This system can be expressed in matrix form as:

Ax = b (1.1)

where:

• A = (aij ) ∈ Mn (K) is the coefficient matrix

• b ∈ Kn is the constant term vector

• x ∈ Kn is the unknown vector

12
Numerical Analysis

Existence and Uniqueness of Solutions

While the theoretical solution x = A−1 b exists for invertible matrices (i.e det(A) 6= 0) , practical
computation faces two major hurdles:

• Size Matters: For large systems (n > 1000), direct inversion becomes computationally
expensive and memory-intensive
• Error Amplification: Small input errors can cause large solution errors in sensitive
systems (ill-conditioned matrices)

1.2.1 Gaussian Elimination method

The Gaussian elimination method systematically transforms a linear system Ax = b into an
equivalent upper triangular system U x = b̃, where:
• A ∈ Rn×n is the coefficient matrix
• U is an upper triangular matrix
• b̃ is the modified right-hand side vector
The algorithm consists of n − 1 stages, each eliminating subdiagonal elements in one column.
Stage 1: First Column Elimination
For each row i > 1:
ai1
1. Compute multiplier: mi1 = a11

2. Update row: Ri ← Ri − mi1 R1

Resulting augmented matrix:
 
a11 a12 · · · a1n | b1
 0 a(1) · · · a(1) (1)
| b2 
 22 2n 
 .. .. . . .. .. 
 . . . . | . 
(1) (1) (1)
0 an2 · · · ann | bn
Stage 2: Second Column Elimination
For each row i > 2:
(1)
ai2
1. Compute multiplier: mi2 = (1)
a22

2. Update row: Ri ← Ri − mi2 R2

Resulting augmented matrix:
a11 a12 a13 · · · a1n | b1
 
 0 a(1)
22
(1) (1)
a23 · · · a2n
(1)
| b2 
 
(2) (2) (2)
0
 0 a33 · · · a3n | b3  
 . .. .. .. .. .. 
 .. . . . . | . 
(2) (2) (2)
0 0 an3 · · · ann | bn
General Stage k
For each row i > k:

13
Numerical Analysis

(k−1)
aik
1. Compute multiplier: mik = (k−1)
akk

2. Update row: Ri ← Ri − mik Rk

Final Triangular System
After n − 1 stages, we obtain:
    
a11 a12 · · · a1n x1 b1
 0 a(1) · · · a(1)   x2   b(1) 
22 2n     2 
..   ..  =  .. 

 .. . . . .
 . . . .  .   . 
(n−1) (n−1)
0 · · · 0 ann xn bn
Important Considerations
(k−1)
• Pivot Elements: The diagonal elements akk must be non-zero
• Row Swaps: Required when encountering zero pivots
Numerical Example
Solve: 
2x + 4y + 6z = 4

x + 3y + 2z = 3

3x + y + 5z = 2


Stage 1 Transformations

1
R2 ← R2 − R1
2
3
R3 ← R3 − R1
2
Resulting system:  
2 4 6 | 4
0 1 −1 | 1 
0 −5 −4 | −4
Stage 2 Transformations

R3 ← R3 + 5R2

Final triangular system:  

2 4 6 | 4
0 1 −1 | 1
0 0 −9 | 1
Back Substitution
1
z=−
9
8
y =1+z =
9
4 − 4y − 6z 5
x= =
2 9

14
Numerical Analysis

Ill-Conditioned System Demonstration

Consider the system of equations (S):
(
10−4 x + y = 1
x+y =2

Exact Solution
The exact solution of (S) is:
10000 9998
x= ≈ 1, y= ≈ 1.
9999 9999

Case 1: Using 10−4 as the Pivot

Applying Gaussian elimination with 10−4 as the pivot, we derive the equivalent system:
(
1 10−4 x + y = 1
R2 ← R2 − −4 R1
10 (x + y) − 101−4 (10−4 x + y) = 2 − 101−4

Solving with limited precision:

(
10−4 x + y = 1
9998
9999y = 9998 ⇒ y= 9999
≈1

Substituting y = 1 into the first equation: 10−4 x + 1 = 1 ⇒ x = 0. This yields the inaccurate
result:
x = 0, y = 1.

Case 2: Swapping Equations for Stability

To improve accuracy, we reorder the equations before elimination:
(
x+y =2
10−4 x + y = 1

Elimination Process: Multiply the first equation by 10−4 and subtract from the second:
(
x+y =2
10−4 x + y − 10−4 (x + y) = 1 − 2 × 10−4

Simplifying the second equation:

(
x+y =2
0.9999y = 0.9998 ⇒ y = 1.00
9998 104
Substituting y = 9999 into x + 1 = 2 gives: x = 9999
.
This produces the accurate result:
9998 104
x= , y= .
9999 9999

15
Numerical Analysis

1.2.2 Pivoting Strategies

To improve the numerical stability of Gaussian elimination, two common strategies are adopted.

Partial Pivoting
In partial pivoting, one examines the entries in the current column (from the diagonal entry
downwards) and selects as pivot the entry with the largest absolute value. That is, at the kth
step one chooses the pivot from the set
(k)
{ |aik | : i = k, . . . , n }.
(k)
If akk is zero or nearly zero, a row with a larger entry is swapped into the kth position.
Example. Consider the system

x1 + 2x2 + x3 = 5

3x1 + x2 + 4x3 = 6

2x1 + 3x2 + 2x3 = 7


Initial augmented Matrix:  

1 2 1 | 5
3 1 4 | 6
2 3 2 | 7
Stage 1 (k=1):

• Pivot: max(|1|, |3|, |2|) = 3 at row 2

• Swap R1 ↔ R2:
 
3 1 4 | 6
1 2 1 | 5
2 3 2 | 7
Elimination:
1
R2 ← R2 − R1
3
2
R3 ← R3 − R1
3
 
3 1 4 | 6
0 5 − 1 | 3
3 3
0 37 − 23 | 3
Stage 2 (k=2):

• Pivot: max(| 35 |, | 73 |) = 7
3
at row 3

• Swap R2 ↔ R3:
 
3 1 4 | 6
0 7 − 2 | 3
3 3
0 35 − 13 | 3

16
Numerical Analysis

Final Elimination:
5
R3 ← R3 − R2
7
 
3 1 4 | 6
0 7 − 2 | 3 
3 3
0 0 17 | 67
Back Substitution
x3 = 6
x2 = 3
x1 = −7

Complete (Total) Pivoting

Complete pivoting extends the idea of partial pivoting by selecting the pivot as the entry of
largest absolute value in the entire submatrix
(k)
{ aij : i, j = k, . . . , n }.

Because column interchanges may be required, it is necessary to keep track of the permutation
of the variables.
Example. Consider the system

2x1 + 4x2 + x3 = 8

x1 + 3x2 + 5x3 = 10

3x1 + x2 + 2x3 = 5


Initial Matrix:  
2 4 1 | 8
1 3 5 | 10
3 1 2 | 5
Stage 1 (k = 1):
• Global pivot: 5 at position (2, 3)

• Swap R1 ↔ R2, C1 ↔ C3

 
5 3 1 | 10
1 4 2 | 8 
2 1 3 | 5
Elimination:
1
R2 ← R2 − R1
5
2
R3 ← R3 − R1
5
 
5 3 1 | 10
0 3.4 1.8 | 6 
0 −0.2 2.6 | 1
Stage 2 (k = 2):

17
Numerical Analysis

• Global pivot: 3.4 at position (2, 2)

• No swaps needed

Final elimination:
−0.2
R3 ← R3 − R2
3.4
 
5 3 1 | 10
0 3.4 1.8 | 6 
46
0 0 17 | 23
17

Now, the system (after column permutation) is:


5x3 + 3x2 + x1 = 10

3.4x2 + 1.8x1 = 6

 46
x = 23
17 1 17

Solution :
x1 = 0.5, x2 = 1.5, x3 = 1

In Gaussian elimination, besides performing row interchanges, one may also swap columns
especially when using complete (or total) pivoting. Each column exchange introduces a
sign change in the determinant, much like a row swap.
Suppose we perform:

• p row swaps, and

• q column swaps.

After reducing the matrix A to an upper triangular matrix U , the determinant of A is

given by
n
Y
p+q
det(A) = (−1) uii ,
i=1

where uii are the diagonal elements of U .

Gaussian Elimination as Matrix Multiplication

• Every row operation in Gaussian elimination corresponds to left-multiplication by

an elementary matrix. For a matrix A, the elimination process can be written as:

Ek · · · E2 E1 A = U,

where U is upper triangular and E1 , E2 , . . . , Ek are elementary matrices.

• Every row swap in Gaussian elimination is equivalent to left-multiplying the current

matrix by a permutation matrix.

18
Numerical Analysis

Example
 
1 2
Let A = .
3 4

1. Step 1: Eliminate the entry 3 in row 2, column 1. Use Eadd with c = −3:
   
1 0 1 2
E1 = , E1 A = .
−3 1 0 −2

2. Result: U = E1 A, and L = E1−1 :

 
1 0
L= .
3 1

1.2.3 Gauss-Jordan Elimination

Following the discussion on Gaussian elimination, we now describe the Gauss-Jordan elimina-
tion method. This procedure refines the upper triangular matrix further to obtain the reduced
row-echelon form (RREF), from which the solution can be directly read without the need for
back substitution.
In Gauss-Jordan elimination, for a given augmented matrix [A | b], the method consists of:

1. Pivot Selection and Normalization: Choose a nonzero pivot in the current row and
scale the row so that the pivot becomes 1.

2. Elimination in All Directions: Use the normalized pivot row to eliminate all other
nonzero entries in the pivot’s column (both below and above the pivot).

The final matrix is in RREF if every pivot is 1 and is the sole nonzero entry in its column.
Consider an augmented matrix for a system:
 (1) (1) (1) (1) 
a11 a12 · · · a1n | b1
a(1) a(1) · · · a(1) (1)
| b2 
[A | b] =  21 22 2n
.
 
.
 .. .
.. . .. .. .. 
. | . 
(1) (1) (1) (1)
an1 an2 · · · ann | bn

Step 1: First Pivot

(1)
If a11 6= 0, begin by normalizing the first row:

(2) 1 (1)
L1 = (1)
L1 .
a11

Then, for each row i = 2, . . . , n, eliminate the first column entries:

(2) (1) (1) (2)
Li = Li − ai1 L1 .

19
Numerical Analysis

After this step, the augmented matrix takes the form:

 (2) (2) (2) 
1 a12 · · · a1n | b1
0 a(2) · · · a(2) | b(2) 
22 2n 2 
. .

. . . .
.. .
. . . .
. | .. 
(2) (2) (2)
0 an2 · · · ann | bn
Step 2: Second Pivot
(2)
Next, if a22 6= 0, normalize the second row:
(3) 1 (2)
L2 = (2)
L2 .
a22
Then, for every row i 6= 2, eliminate the entry in column 2:
(3) (2) (2) (3)
Li = Li − ai2 L2 .
This results in:  (3) (3) (3) 
1 0 a13 · · · a1n | b1
(3) (3) (3)
0 1 a23 · · · a2n | b2 
.
 
. .. .. .. .. .. 
 .. . . . . | . 
(3) (3) (3)
0 0 an3 · · · ann | bn
Step n: Final Pivot
(n)
Assuming ann 6= 0, normalize the nth row:
1
L(n+1)
n = (n)
L(n)
n .
ann
Then, for each row i = 1, . . . , n − 1, eliminate the entry in column n:
(n+1) (n) (n)
Li = Li − ain Ln(n+1) .
At the end, the augmented matrix is transformed into:
 (n) 
1 0 ··· 0 | b1
0 1 · · · 0 (n)
| b2 
.
 
. . .
 .. .. . . ... .. 
| . 
(n)
0 0 ··· 1 | bn
Thus, the solution of the system is directly given by:
(n)
x i = bi , i = 1, . . . , n.
Example

Consider the system: 

x1 + x2 + x3 = 0,


2x1 − x2 + x3 = 2,


x − x + 2x = 3.
1 2 3

20
Numerical Analysis

The corresponding augmented matrix is:

 
1 1 1 | 0
2 −1 1 | 2 .
1 −1 2 | 3
(1)
Step 1: Since a11 = 1 (nonzero), leave the first row as is, and eliminate the first entry in
rows 2 and 3:
(2) (1) (1) (2) (1) (1)
L2 = L2 − 2 L1 , L3 = L3 − L1 .
The matrix becomes:  
1 1 1 | 0
0 −3 −1 | 2 .
0 −2 1 | 3
(2)
Step 2: Normalize the second row by dividing by −3 (since a22 = −3):

(3) 1 (2)
L2 = L .
−3 2
Then, eliminate the second column entries in the other rows:
(3) (2) (3) (3) (2) (3)
L1 = L1 − 1 · L2 , L3 = L3 − (−2) L2 .

After these operations, the matrix is transformed into:

 
1 0 ∗ | ∗
0 1 ∗ | ∗ .
0 0 ∗ | ∗

Step 3: Finally, normalize the third row and eliminate the third column entries in the rows
above. At this point, the augmented matrix becomes:
 
1 0 0 | x1
0 1 0 | x2  .
0 0 1 | x3

The solution is read off directly:

x1 = 0, x2 = −1, x3 = 1.

1.2.4 LU-Factorisation
Building on Gaussian elimination, we now formalize its matrix factorization interpretation.
Definition 1.2.1. For a square matrix A ∈ Cn×n , an LU decomposition is a factorization:

A = LU

where:
• L is lower triangular with unit diagonal entries

• U is upper triangular

21
Numerical Analysis

Since the system Ax = b can be rewritten as LU x = b, solving it is equivalent to solving

the two triangular systems:
Ly = b and U x = y.

Definition 1.2.2 (Leading Principal Submatrix). For a square matrix A ∈ Cn×n , the k-th
leading principal submatrix Ak is the k × k submatrix formed from the first k rows and columns
of A:  
a11 · · · a1k
Ak :=  ... . . . ...  , 1 ≤ k ≤ n
 
ak1 · · · akk

Definition 1.2.3 (Leading Principal Minor). The k-th leading principal minor is the determi-
nant of the k-th leading principal submatrix:

∆k := det(Ak )

Existence and Uniqueness

Let A ∈ Cn×n . Then A admits an LU decomposition if and only if all leading principal
minors are nonsingular. When it exists, the decomposition is unique.

Proof. We prove both directions:

(⇒) LU Decomposition =⇒ Non-zero Leading Minors:
Assume A = LU exists. For any 1 ≤ k ≤ n, partition:

Ak = Lk Uk

where Lk is the k × k leading submatrix of L, and Uk similarly for U .

Since:

• Lk is unit lower triangular =⇒ det(Lk ) = 1

• Uk is upper triangular =⇒ det(Uk ) = ki=1 uii

Q

Thus:
k
Y
det(Ak ) = det(Lk ) det(Uk ) = uii 6= 0
i=1

because diagonal entries uii are non-zero (they are Gaussian elimination pivots).
(⇐) Non-zero Leading Minors =⇒ LU Decomposition:
Gaussian Elimination without Permutations:
Step 1: The first minor
det(A1 ) = a11 6= 0.
The first pivot u11 = a11 .
Step k: Assume the first k − 1 pivots are non-zero. The minor

det(Ak ) 6= 0

22
Numerical Analysis

guarantees that the pivot ukk (after elimination) is non-zero.

Inductive Construction: The Gaussian multipliers are stored in L, and the pivots in U .
Uniqueness:
Assume A = L1 U1 = L2 U2 . Since L1 and L2 have 1s on the diagonal:
L−1 −1
1 L2 = U1 U2 .

This equality implies that L1 = L2 and U1 = U2 , as a lower triangular matrix with 1s on the
diagonal can only equal an upper triangular matrix if both are the identity matrix.

Let A be an invertible square matrix of order n. Then, there exists a permutation matrix
P such that all the pivots of P A are nonzero. Consequently, we can factorize P A as

P A = LU,

where L is a lower triangular matrix with ones on its diagonal and U is an upper triangular
matrix.
The linear system Ax = b can therefore be reformulated as

P Ax = P b.

The solution is then obtained by following these steps:

1. Construct the matrices U , L, and P .

2. Compute P b.

3. Solve the lower triangular system Ly = P b for y.

4. Solve the upper triangular system U x = y for x.

Consider the system of equations:


2x1 − x2 + 2x3 = 1,


−6x1 − 2x3 = 2,


8x − x + 5x = 1.
1 2 3

We wish to solve this system using LU factorization, where the coefficient matrix A is decom-
posed as:
A = LU,
with    
1 0 0 u11 u12 u13
L = l21 1 0 and U =  0 u22 u23  .
   
l31 l32 1 0 0 u33
LU Decomposition
Multiplying L and U gives:
 
u11 u12 u13
LU = l21 u11 l21 u12 + u22 l21 u13 + u23 .
 
l31 u11 l31 u12 + l32 u22 l31 u13 + l32 u23 + u33

23
Numerical Analysis

By matching the entries with those of the original matrix

 
2 −1 2
A = −6 0 −2 ,
 
8 −1 5
one obtains the values:
   
1 0 0 2 −1 2
L = −3 1 0 , U = 0 −3 4 .
   
4 −1 1 0 0 1
Solving
  the System
1
Let b = 2. The system Ax = b can be solved in two stages:
1
Step 1: Solve Ly = b
We first solve:     
1 0 0 y1 1
−3 1 0  y2 = 2 .
    
4 −1 1 y3 1
This yields:
y1 = 1,
−3y1 + y2 = 2 ⇒ y2 = 2 + 3 = 5,
4y1 − y2 + y3 = 1 ⇒ y3 = 1 − 4 + 5 = 2.
 
1
Thus, y = 5.
2
Step 2: Solve U x = y
Next, we solve:     
2 −1 2 x1 1
0 −3 4 x2 = 5 .
    
0 0 1 x3 2
Starting from the bottom:
x3 = 2,
−3x2 + 4x3 = 5 ⇒ −3x2 + 8 = 5 ⇒ −3x2 = −3 ⇒ x2 = 1,
2x1 − x2 + 2x3 = 1 ⇒ 2x1 − 1 + 4 = 1 ⇒ 2x1 + 3 = 1 ⇒ 2x1 = −2 ⇒ x1 = −1.
Hence, the solution is:  
−1
x=  1 .
2
Conclusion
By decomposing the matrix A into L and U , we solved the system Ax = b in two stages: first,
finding y from Ly = b, and then x from U x = y. The final solution to the system is:
x1 = −1, x2 = 1, x3 = 2.

24