MA206L-Numerical Methods

Lecture 20: Numerical Linear Algebra - LU Decomposition

Panchatcharam Mariappan1

1 Associate Professor

Department of Mathematics and Statistics

IIT Tirupati, Tirupati

October 7, 2024
Linear System
• One of the important problem in many science and engineering.
• To solve an algebraic linear system

Ax = b

for the unknown vector x.

• The coefficient matrix A and the right-hand side vector b are known.
• What is the size of A, x and b?
• Assume: A is n × n, x and b are n−dimensional vector.
• The system may or may not have a solution.
• It may have an infinite solutions or unique solution.

1
Linear System
• Gaussian elimination method: one of the standard method for solving
the linear system
• Computer or calculator
• School days.
• However, in pure mathematics, the solution is given by

x = A−1 b

where A−1 denotes the inverse of the matrix A.

• But, in most of the real applications, it is not advised to find A−1
explicitly, indeed, it is recommended to solve for x.

2
Linear System
• But, in most of the real applications, it is not advised to find A−1
explicitly, indeed, it is recommended to solve for x.
• In applied mathematics, the largest and fastest computers can also face
difficulty to solve the system accurately when the number of unknowns
are in millions.

3
Linear System: Important 10 Questions
1. How do we store such a large linear system in the computer?
2. How do we know that the computed answers are correct?
3. How does the computer precision affects the results?
4. Can the algorithm fail?
5. How long will it take to compute the answers?
6. What is the asymptotic operation count of the algorithm?
7. Will the algorithm be stable for perturbation?
8. Can stability be controlled by pivoting?
9. Which strategy of pivoting should be used?
10. How do we know whether the matrix is ill-conditioned?

4
Diagonal Linear System
Definition 1 (Diagonal linear system)
A matrix A of order n is diagonal if all its nonzero entries are on its diagonal. A
diagonal linear system of order n is one whose coefficient matrix is diagonal.
That is (
dii ̸= 0 if i = j
Aij =
0 if i ̸= j

A diagonal linear system can be represented as

    
d11 0 0 ··· 0 x1 b1
 0 d22 0 · · · 0   x2   b2 
    
 0
 0 d33 · · · 0   x3   b3 
  =  
 .. .. .. . . .
.  .   . 
 . . . . .   ..   .. 
0 0 0 · · · dnn xn bn
5
Diagonal Linear System
Finding the solution of the linear system is easy because each equation
determines the value of one unknown, provided that each diagonal entry is
nonzero. The solution is given by

bi
xi = , i = 1, 2, · · · , n
dii

Example 1
Consider the following linear system and find the solution.

9x1 = 3, 25x2 = 5, , 16x3 = 4, 36x4 = 6, 49x5 = 7

6
Diagonal Linear System
It is a diagonal linear system and it can be written as
    
9 0 0 0 0 x1 3
0 25 0 0 0  x2  5
    
0 0 16 0 0  x3  = 4
    
0 0 0 36 0  x4  6
0 0 0 0 49 x5 7

Without constructing this matrix also we can immediately conclude that

  1
x1 3
x2   1 
   51 
x3  =  
   41 
x4   
6
1
x5 7
7
Upper Triangular Linear System
Definition 2 (Upper triangular linear system)
A matrix A of order n is upper triangular if all its nonzero entries are on its
diagonal or strictly on the upper triangular entries. An upper triangular linear
system of order n is one whose coefficient matrix is upper triangular. That is
(
uij if i ≥ j
Aij =
0 if i < j

An upper triangular linear system can be represented as

    
u11 u12 u13 · · · u1n x1 y1
 0 u22 u23 · · · u2n   x2   y2 
    
 0
 0 u33 · · · u3n   x3  =  y3 
   
 .. .. .. .. .
..   .  .
 ..   .. 

 . . . .
0 0 0 · · · unn xn yn 8
Backward Substitution
In order to solve an upper triangular linear system, we need a backward
substitution which is given by the following algorithm

9
Backward Substitution
In order to solve an upper triangular linear system, we need a backward
substitution which is given by the following algorithm
n
X
yi − uij xj
j=i+1
xi = , i = n, n − 1, · · · , 1 (1)
uii

9
Backward Substitution
Example 2
Find the solution of the following linear system.
    
2 2 3 4 x1 20
0 5 6 7  x2  34
0 0 8 9  x3  = 25
    

0 0 0 10 x4 10
Backward Substitution
Example 2
Find the solution of the following linear system.
    
2 2 3 4 x1 20
0 5 6 7  x2  34
0 0 8 9  x3  = 25
    

0 0 0 10 x4 10

It is an upper triangular system. Using backward substitution, we obtain that

10 25 − 9x4 16
x4 = = 1 =⇒ x3 = =⇒ x3 = =2
10 8 8
34 − 6x3 − 7x4 15
=⇒ x2 = =⇒ x2 = =3
5 5
20 − 2x2 − 3x3 − 4x4 4
=⇒ x1 = =⇒ x1 = = 2
2 2
10
Forward Substitution
Definition 3 (Lower triangular linear system)
A matrix A of order n is upper triangular if all its nonzero entries are on its
diagonal or strictly on the lower triangular entries. A lower triangular linear
system of order n is one whose coefficient matrix is lower triangular. That is
(
lij if i ≤ j
Aij =
0 if i > j

A lower triangular linear system can be represented as

    
l11 0 0 ··· 0 y1 b1
 l21 l22 0 · · · 0   y2   b2 
    
 l31 l32 u33 · · · 0   y3   b3 
   =  
 .. .. .. . . ..   ..   .. 
 . . . . .  .   . 
ln1 ln2 ln3 · · · lnn yn bn 11
Forward Substitution
In order to solve a lower triangular linear system, one can apply forward
substitution. In the backward substitution, we obtain the value of xn first and
then using xn , we obtain the value of xn−1 Using xn , xn−1 , xn−2 is calculated.
Finally the value of x1 is calculated. On the other hand, in forward
substitution, we obtain the value of y1 first. Using y1 , we obtain the value of y2
and so on. The value of

12
Forward Substitution
In order to solve a lower triangular linear system, one can apply forward
substitution. In the backward substitution, we obtain the value of xn first and
then using xn , we obtain the value of xn−1 Using xn , xn−1 , xn−2 is calculated.
Finally the value of x1 is calculated. On the other hand, in forward
substitution, we obtain the value of y1 first. Using y1 , we obtain the value of y2
and so on. The value of
i−1
X
bi − lij xj
j=1
yi = , i = 1, 2, · · · , n (2)
lii

12
LU Decomposition

13
LU Decomposition
If we can decompose a square matrix A by in the form of

A = LU

then one can easily obtain the solution of the linear system

Ax = b

Ax = b =⇒ LU x = b =⇒ Ly = b where U x = y
Since Ly = b is a lower triangular system, by the forward substitution, we can
solve for y. Using y in U x = y, which is an upper triangular system, and the
backward substitution, we can obtain x.

14
LU Decomposition
The following steps are used to solve the linear system Ax = b.
1. Find L and U such that A = LU
2. Using forward substitution, Solve for y from Ly = b
3. Use y in U x = y
4. Using backward substitution, Solve for x from U x = y
As per the LU decomposition, we can write A as
    
a11 a12 a13 · · · a1n l11 0 0 ··· 0 u11 u12 u13 · · · u1n
 a21 a22 a23 · · · a2n   l21 l22 0 · · · 0   0 u22 u23 · · · u2n 
    
 a31 a32 a33 · · · a3n   l31 l32 l33 · · · 0  0 0 u33 · · · u3n 
 =  
 .. .. .. .. ..   .. .. .. . . ..   .. .. .. .. .. 
 . . . . .   . . . . .  . . . . . 
an1 an2 an3 · · · ann ln1 ln2 ln3 · · · lnn 0 0 0 · · · unn
15
LU Decomposition

Example 3
Obtain the LU decomposition and then obtain the solution x.
    
6 −2 2 4 x1 16
 12 −8 6 10  x2   26 
   =  
 3 −13 9 3  x3  −19
−6 4 1 −18 x4 −34
    
6 −2 2 4 l11 0 0 0 u11 u12 u13 u14
 12 −8 6 10  l21 l22 0 0   0 u22 u23
  u24 
 = 
 3 −13 9 3  l31 l32 l33 0   0 0 u33 u34 
−6 4 1 −18 l41 l42 l43 l44 0 0 0 u44
As per LU Decomposition
16
LU Decomposition
Comparing the first entry of LU with A, we obtain that l11 u11 = 6. Now, we can
choose any real number l11 such that l11 u11 = 6. For the sake of convenience,
we always choose lii = aii , then we obtain that l11 = 6 and u11 = 1.
l11 u11 = 6, l11 = 6 =⇒ u11 = 1
−1
l11 u12 = −2 =⇒ u12 =
3
1
l11 u13 = 2 =⇒ u13 =
3
2
l11 u14 = 4 =⇒ u14 =
3
Since u11 is known, we can easily compute the first column of L by the
following formula.
ai1
li1 = , i>1
u11 17
LU Decomposition

    
6 −2 2 4 6 0 0 0 1 −1/3 1/3 2/3
 12 −8 6 10   12 l22 0 0  0 u22 u23 u24 
 =  
 3 −13 9 3   3 l32 l33 0  0 0 u33 u34 
−6 4 1 −18 −6 l42 l43 l44 0 0 0 u44

Therefore, the formula to compute u2j is given by

a2j − l21 u1j

u2j = , j>2
l22
Since u22 and u12 are known, we can compute the second column of L by the
following formula
ai2 − li1 u12
li2 = , i>2
u22 18
LU Decomposition
Similarly the formula to compute u2j is given by

a3j − l31 u1j − l32 u2j

u3j = ,j > 3
l33
Since u22 and u12 are known, we can compute the second column of L by the
following formula
ai3 − li1 u13 − li2 u23
li3 = , i>3
u33
    
6 −2 2 4 6 0 0 0 1 −1/3 1/3 2/3
 12 −8 6 10   12 −8 0 0  0 1/2 −1/4 −1/4
 
 =
 3 −13 9 3   3 −24 9 0  0 0 2/9 −5/9
−6 4 1 −18 −6 4 18 −18 0 0 0 1/6

19
LU Decomposition
Now, let us consider the system Ly = b
    
6 0 0 0 y1 16
 12 −8 0 0  y2  =  26 
   

 3 −24 9 0  y3   −19
−6 4 18 −18 y4 −34
Using forward substitution, we obtain that
 
16 8 8 3
y1 = = =⇒ 12 − 8y2 = 26 =⇒ y2 =
6 3 3 4
   
8 3
=⇒ 3 − 24 + 9y3 = −19 =⇒ y3 = −1
3 4
   
8 3 1
=⇒ −6 +4 + 18(1) − 18y4 = −34 =⇒ y4 =
3 4 6
20
LU Decomposition
Now, let us consider the system Ly = b
   
y1 8/3
y2  3/4
 = 
y3   −1 
y4 1/6

Now, consider the system U x = y

    
1 −1/3 1/3 2/3 x1 8/3
0 1/2 −1/4 −1/4 x2  3/4
   =  
0 0 2/9 −5/9 x3   −1 
0 0 0 1/6 x4 1/6

21
LU Decomposition
Using backward substitution, we obtain that

1/6 2 5
x4 = = 1 =⇒ x3 − = −1 =⇒ x3 = −2
1/6 9 9
       
1 −1 −1 3
=⇒ x2 + (−2) + (1) = =⇒ x2 = 1
2 4 4 4
   
1 1 2 8
=⇒ x1 − −2 + = =⇒ x1 = 3
3 3 3 3

22
LU Decomposition
For a general LU decomposition, we have first guess lkk ̸= 0 value or ukk ̸= 0
values and use the following formula for j > k and i > k
k−1
X
akj − lks usj
s=1
ukj = (3)
lkk
k−1
X
aik − lis usk
s=1
lik = (4)
ukk

From the above two equations, it is easy to note that lkk ̸= 0 and ukk ̸= 0 are
necessary.

23
LU Decomposition
Theorem 4
If all n leading principal minors of A are nonsingular, then A has an LU-
decomposition.

Definition 5 (Diagonally Dominant)

A square matrix A = (aij ) is said to be diagonally dominant if it satisfies the
following inequality:
n
X
|aii | > |aij | 1≤i≤n
j=1,j̸=i

Theorem 6
Every diagonally dominant matrix is non-singular and has an LU-
decomposition. 24
Crout, Doolittle, LDLT ,
Cholesky

25
Crout’s Decomposition
In LU-decomposition one of the condition specified is that lkk ̸= 0 and ukk ̸= 0
are necessary. If we choose, ukk = 1, then U is an upper triangular matrix with
its diagonal entries as 1. This decomposition is called Crout’s decomposition.
    
a11 a12 a13 · · · a1n l11 0 0 ··· 0 1 u12 u13 · · · u1n
 a21 a22 a23 · · · a2n   l21 u22 0 · · · 0  0 1 u23 · · · u2n 
    
 a31 a32 a33 · · · a3n   l31 l32 u33 · · · 0  0 0 1 · · · u3n 
 =  
 .. .. .. . . ..   .. .. .. . . ..   .. .. .. . . .. 
 . . . . .   . . . . .  . . . . . 
an1 an2 an3 · · · ann ln1 ln2 ln3 · · · lnn 0 0 0 ··· 1

26
Doolittle’s Decomposition
If we choose, lkk = 1, then L is a lower triangular matrix with its diagonal
entries as 1. This decomposition is called Doolittle’s decomposition.
    
a11 a12 a13 · · · a1n 1 0 0 ··· 0 u11 u12 u13 · · · u1n
 a21 a22 a23 · · · a2n   l21 1 0 · · · 0  0 u22 u23
  · · · u2n 
   
 a31 a32 a33 · · · a3n   l31 l32 1 · · · 0  0 0 u33 · · · u3n 
 =  
 .. .. .. . .. .
..   ..
  . .
.. .
.. . . .
. . ..   ..
  .. .. .. .. 
 . . . . . . . 
an1 an2 an3 · · · ann ln1 ln2 ln3 · · · 1 0 0 0 · · · unn

27
LDLT Decomposition
Observe that when Doolittle’s decomposition is applied, the first row of U and
A are same. Similarly, when Crout’s decomposition is applied, then the first
column of L and A are same. In fact we can prove uniqueness of LU
decomposition in case of Doolittle and Crout assumptions as follows:

Theorem 7
Every nonsingular matrix has a unique LU decomposition.
Suppose
A = L1 U1 = L2 U2 =⇒ L−1 −1
1 L2 = U1 U2

Since left side is lower triangular and right side is upper triangular, it should
be a diagonal matrix. Since diag(L−1 1 L2 ) = I (for Doolittle) or
−1
diag(U1 U2 ) = I (for Crout), it leads to L1 = L2 and U1 = U2 . Using
Gauss-Jordan method it can be obtained easily.
28
LDLT Decomposition
If A is a symmetric matrix and A has an LU decomposition, then it has an
LDLT decomposition. For,
A = LU =⇒ AT = U T LT
A = AT =⇒ LU = U T LT
Since L is unit lower triangular, it is invertible, we can write
U = L−1 U T LT =⇒ U (LT )−1 = L−1 U T
Transpose of an upper triangular matrix is a lower triangular and vice versa.
Since L is lower triangular, L−1 is also a lower triangular and hence L−1 U T is
a lower triangular matrix. On the other side, we have U (LT )−1 is an upper
triangular matrix. Therefore,
U (LT )−1 = L−1 U T = D =⇒ U = DLT =⇒ A = LDLT
29
Cholesky Decomposition
When A is symmetric and positive definite and U = LT , then we obtain that

A = LLT

Theorem 8
If A is real, symmetric and positive definite, then it has a unique factorization

A = LLT

in which L is lower triangular with positive diagonal.

Since A > 0, A = AT by LDLT decomposition, we can obtain

A = LDLT

A > 0 =⇒ D > 0 =⇒ L̃ ≡ LD1/2 =⇒ A = L̃L̃T

√
where the entries of D1/2 are dii . 30
 Thanks
Doubts and Suggestions
[email protected]

31
 MA206L-Numerical Methods
Lecture 20: Numerical Linear Algebra - LU Decomposition

Panchatcharam Mariappan1

1 Associate Professor

Department of Mathematics and Statistics

IIT Tirupati, Tirupati

October 7, 2024

31