MA206L-Numerical Methods

Lecture 21: Numerical Linear Algebra - LU Decomposition

Panchatcharam Mariappan1

1 Associate Professor

Department of Mathematics and Statistics

IIT Tirupati, Tirupati

October 9, 2024
LU Decomposition

1
LU Decomposition
If we can decompose a square matrix A by in the form of

A = LU

then one can easily obtain the solution of the linear system

Ax = b

Ax = b =⇒ LU x = b =⇒ Ly = b where U x = y
Since Ly = b is a lower triangular system, by the forward substitution, we can
solve for y. Using y in U x = y, which is an upper triangular system, and the
backward substitution, we can obtain x.

2
LU Decomposition
The following steps are used to solve the linear system Ax = b.
1. Find L and U such that A = LU
2. Using forward substitution, Solve for y from Ly = b
3. Use y in U x = y
4. Using backward substitution, Solve for x from U x = y
As per the LU decomposition, we can write A as
    
a11 a12 a13 · · · a1n l11 0 0 ··· 0 u11 u12 u13 · · · u1n
 a21 a22 a23 · · · a2n   l21 l22 0 · · · 0   0 u22 u23 · · · u2n 
    
 a31 a32 a33 · · · a3n   l31 l32 l33 · · · 0  0 0 u33 · · · u3n 
 =  
 .. .. .. .. ..   .. .. .. . . ..   .. .. .. .. .. 
 . . . . .   . . . . .  . . . . . 
an1 an2 an3 · · · ann ln1 ln2 ln3 · · · lnn 0 0 0 · · · unn
3
LU Decomposition

Example 1
Obtain the LU decomposition and then obtain the solution x.
    
6 −2 2 4 x1 16
 12 −8 6 10  x2   26 
   =  
 3 −13 9 3  x3  −19
−6 4 1 −18 x4 −34
    
6 −2 2 4 l11 0 0 0 u11 u12 u13 u14
 12 −8 6 10  l21 l22 0 0   0 u22 u23
  u24 
 = 
 3 −13 9 3  l31 l32 l33 0   0 0 u33 u34 
−6 4 1 −18 l41 l42 l43 l44 0 0 0 u44

4
LU Decomposition
Comparing the first entry of LU with A, we obtain that l11 u11 = 6. Now, we can
choose any real number l11 such that l11 u11 = 6. For the sake of convenience,
we always choose lii = aii , then we obtain that l11 = 6 and u11 = 1.
l11 u11 = 6, l11 = 6 =⇒ u11 = 1
−1
l11 u12 = −2 =⇒ u12 =
3
1
l11 u13 = 2 =⇒ u13 =
3
2
l11 u14 = 4 =⇒ u14 =
3
Since u11 is known, we can easily compute the first column of L by the
following formula.
ai1
li1 = , i>1
u11 5
LU Decomposition

    
6 −2 2 4 6 0 0 0 1 −1/3 1/3 2/3
 12 −8 6 10   12 l22 0 0  0 u22 u23 u24 
 =  
 3 −13 9 3   3 l32 l33 0  0 0 u33 u34 
−6 4 1 −18 −6 l42 l43 l44 0 0 0 u44

Therefore, the formula to compute u2j is given by

a2j − l21 u1j

u2j = , j>2
l22
Since u22 and u12 are known, we can compute the second column of L by the
following formula
ai2 − li1 u12
li2 = , i>2
u22 6
LU Decomposition
Similarly the formula to compute u2j is given by

a3j − l31 u1j − l32 u2j

u3j = ,j > 3
l33
Since u22 and u12 are known, we can compute the second column of L by the
following formula
ai3 − li1 u13 − li2 u23
li3 = , i>3
u33
    
6 −2 2 4 6 0 0 0 1 −1/3 1/3 2/3
 12 −8 6 10   12 −8 0 0  0 1/2 −1/4 −1/4
 
 =
 3 −13 9 3   3 −24 9 0  0 0 2/9 −5/9
−6 4 1 −18 −6 4 18 −18 0 0 0 1/6

7
LU Decomposition
Now, let us consider the system Ly = b
    
6 0 0 0 y1 16
 12 −8 0 0  y2  =  26 
   

 3 −24 9 0  y3   −19
−6 4 18 −18 y4 −34
Using forward substitution, we obtain that
 
16 8 8 3
y1 = = =⇒ 12 − 8y2 = 26 =⇒ y2 =
6 3 3 4
   
8 3
=⇒ 3 − 24 + 9y3 = −19 =⇒ y3 = −1
3 4
   
8 3 1
=⇒ −6 +4 + 18(1) − 18y4 = −34 =⇒ y4 =
3 4 6
8
LU Decomposition
Now, let us consider the system Ly = b
   
y1 8/3
y2  3/4
 = 
y3   −1 
y4 1/6

Now, consider the system U x = y

    
1 −1/3 1/3 2/3 x1 8/3
0 1/2 −1/4 −1/4 x2  3/4
   =  
0 0 2/9 −5/9 x3   −1 
0 0 0 1/6 x4 1/6

9
LU Decomposition
Using backward substitution, we obtain that

1/6 2 5
x4 = = 1 =⇒ x3 − = −1 =⇒ x3 = −2
1/6 9 9
       
1 −1 −1 3
=⇒ x2 + (−2) + (1) = =⇒ x2 = 1
2 4 4 4
   
1 1 2 8
=⇒ x1 − −2 + = =⇒ x1 = 3
3 3 3 3

10
LU Decomposition
For a general LU decomposition, we have first guess lkk ̸= 0 value or ukk ̸= 0
values and use the following formula for j > k and i > k
k−1
X
akj − lks usj
s=1
ukj = (1)
lkk
k−1
X
aik − lis usk
s=1
lik = (2)
ukk

From the above two equations, it is easy to note that lkk ̸= 0 and ukk ̸= 0 are
necessary.

11
LU Decomposition

Theorem 2
If all n leading principal minors of A are nonsingular, then A has an LU-
decomposition.

Definition 3 (Diagonally Dominant)

A square matrix A = (aij ) is said to be diagonally dominant if it satisfies the
following inequality:
n
X
|aii | > |aij | 1≤i≤n
j=1,j̸=i

Theorem 4
Every diagonally dominant matrix is non-singular and has an LU-
decomposition.
12
Crout, Doolittle, LDLT ,
Cholesky

13
Crout’s Decomposition
In LU-decomposition one of the condition specified is that lkk ̸= 0 and ukk ̸= 0
are necessary. If we choose, ukk = 1, then U is an upper triangular matrix with
its diagonal entries as 1. This decomposition is called Crout’s decomposition.
    
a11 a12 a13 · · · a1n l11 0 0 ··· 0 1 u12 u13 · · · u1n
 a21 a22 a23 · · · a2n   l21 u22 0 · · · 0  0 1 u23 · · · u2n 
    
 a31 a32 a33 · · · a3n   l31 l32 u33 · · · 0  0 0 1 · · · u3n 
 =  
 .. .. .. . . ..   .. .. .. . . ..   .. .. .. . . .. 
 . . . . .   . . . . .  . . . . . 
an1 an2 an3 · · · ann ln1 ln2 ln3 · · · lnn 0 0 0 ··· 1

14
Doolittle’s Decomposition
If we choose, lkk = 1, then L is a lower triangular matrix with its diagonal
entries as 1. This decomposition is called Doolittle’s decomposition.
    
a11 a12 a13 · · · a1n 1 0 0 ··· 0 u11 u12 u13 · · · u1n
 a21 a22 a23 · · · a2n   l21 1 0 · · · 0  0 u22 u23
  · · · u2n 
   
 a31 a32 a33 · · · a3n   l31 l32 1 · · · 0  0 0 u33 · · · u3n 
 =  
 .. .. .. . .. .
..   ..
  . .
.. .
.. . . .
. . ..   ..
  .. .. .. .. 
 . . . . . . . 
an1 an2 an3 · · · ann ln1 ln2 ln3 · · · 1 0 0 0 · · · unn

15
Doolittle Decomposition

Example 5
Find the Doolittle decomposition
 
1 1 1
A =  1 2 −2
−2 1 1

and the solve for Ax = (1, 0, 3)T

16
Doolittle Decomposition
    
1 1 1 1 0 0 u11 u12 u13
1 2 −2 = l21
  1 0   0 u22 u23 
−2 1 1 l31 l32 1 0 0 u33
    
1 1 1 1 0 0 1 1 1
1 2 −2 = l21 1 0 0 u22 u23 
−2 1 1 l31 l32 1 0 0 u33
    
1 1 1 1 0 0 1 1 1
1 2 −2 =  1 1 0 0 u22 u23 
−2 1 1 −2 l32 1 0 0 u33

17
Doolittle Decomposition
    
1 1 1 1 0 0 1 1 1
 1 2 −2 =  1 1 0 0 1 −3 
−2 1 1 −2 l32 1 0 0 u33
    
1 1 1 1 0 0 1 1 1
 1 2 −2 =  1 1 0 0 1 −3
−2 1 1 −2 3 1 0 0 12
        
1 0 0 y1 1 y1 1
 1 1 0 y2  = 0 =⇒ y2  = −1
−2 3 1 y3 3 y3 8
        
1 1 1 x1 1 x1 −2/3
0 1 −3 x2  = −1 =⇒ x2  =  1 
0 0 12 x3 8 x3 2/3
18
Doolittle Decomposition

Example 6
Show that the following matrix is invertible but has no LU decomposition
 
2 2 1
A = 1 1 1
3 2 1

|A| = 2(1 − 2) − 2(1 − 3) + 1(2 − 3) = 2

19
Doolittle Decomposition
Suppose it has an LU decomposition
    
2 2 1 1 0 0 u11 u12 u13
1 1 1 = l21 1 0  0 u22 u23 
3 2 1 l31 l32 1 0 0 u33
    
2 2 1 1 0 0 2 2 1
1 1 1 = l21 1 0 0 u22 u23 
3 2 1 l31 l32 1 0 0 u33
    
2 2 1 1 0 0 2 2 1
1 1 1 = 1/2 1 0 0 u22 u23 
3 2 1 l31 l32 1 0 0 u33

20
Doolittle Decomposition
Suppose it has an LU decomposition
    
2 2 1 1 0 0 2 2 1
1 1 1 = 1/2 1 0 0 0 u23 
3 2 1 l31 l32 1 0 0 u33
We know that |AB| = |A||B| and the determinant of a triangular matrix is the
product of its diagonal entries. Therefore, |U | = 0 =⇒ |LU | = |L||U | = 0.
Hence, no LU decomposition is possible.

21
LDLT Decomposition
Observe that when Doolittle’s decomposition is applied, the first row of U and
A are same. Similarly, when Crout’s decomposition is applied, then the first
column of L and A are same. In fact we can prove uniqueness of LU
decomposition in case of Doolittle and Crout assumptions as follows:

Theorem 7
Every nonsingular matrix has a unique LU decomposition.
Suppose
A = L1 U1 = L2 U2 =⇒ L−1 −1
1 L2 = U1 U2

Since left side is lower triangular and right side is upper triangular, it should
be a diagonal matrix. Since diag(L−1 1 L2 ) = I (for Doolittle) or
−1
diag(U1 U2 ) = I (for Crout), it leads to L1 = L2 and U1 = U2 . Using
Gauss-Jordan method it can be obtained easily.
22
LDLT Decomposition
If A is a symmetric matrix and A has an LU decomposition, then it has an
LDLT decomposition. For,
A = LU =⇒ AT = U T LT
A = AT =⇒ LU = U T LT
Since L is unit lower triangular, it is invertible, we can write
U = L−1 U T LT =⇒ U (LT )−1 = L−1 U T
Transpose of an upper triangular matrix is a lower triangular and vice versa.
Since L is lower triangular, L−1 is also a lower triangular and hence L−1 U T is
a lower triangular matrix. On the other side, we have U (LT )−1 is an upper
triangular matrix. Therefore,
U (LT )−1 = L−1 U T = D =⇒ U = DLT =⇒ A = LDLT
23
Cholesky Decomposition
When A is symmetric and positive definite and U = LT , then we obtain that

A = LLT

Theorem 8
If A is real, symmetric and positive definite, then it has a unique factorization

A = LLT

in which L is lower triangular with positive diagonal.

Since A > 0, A = AT by LDLT decomposition, we can obtain

A = LDLT

A > 0 =⇒ D > 0 =⇒ L̃ ≡ LD1/2 =⇒ A = L̃L̃T

√
where the entries of D1/2 are dii . 24
Cholesky Decomposition
Find the Cholesky Decomposition and then solve the following system
    
9.0000 0.6000 −0.3000 1.5000 x1 3
 0.6000 16.0400 1.1800 −1.5000 x2  4
   =  
−0.3000 1.1800 4.1000 −0.5700 x3  2
1.5000 −1.5000 −0.5700 25.4500 x4 1

25
Cholesky Decomposition

    
9 0.6 −0.3 1.5 l11 0 0 0 l11 l21 l31 l41
 0.6 16.04 1.18 −1.5  l21 l22 0
  0   0 l22 l32
  l42 

−0.3 1.180 = 
4.1 −0.57 l31 l32 l33 0   0 0 l33 l43 
1.5 −1.5 −0.57 25.45 l41 l42 l43 l44 0 0 0 l44
2
l11 = 9 =⇒ l11 = 3
    
9 0.6 −0.3 1.5 3 0 0 0 3 l21 l31 l41
 0.6 16.04 1.18
 −1.5  l21
= l22 0 0 0 l22 l32
 l42 

−0.3 1.180 4.1 −0.57 l31 l32 l33 0  0 0 l33 l43 
1.5 −1.5 −0.57 25.45 l41 l42 l43 l44 0 0 0 l44

26
Cholesky Decomposition

    
9 0.6 −0.3 1.5 3 0 0 0 3 0.2 −0.1 0.5
 0.6 16.04 1.18
 −1.5  =  0.2 l22 0
 0  0 l22
 l32 l42 

−0.3 1.180 4.1 −0.57   −0.1 l32 l33 0   0 0 l33 l43 
1.5 −1.5 −0.57 25.45 0.5 l42 l43 l44 0 0 0 l44
2
0.04 + l22 = 16.04 =⇒ l22 = 4

    
9 0.6 −0.3 1.5 3 0 0 0 3 0.2 −0.1 0.5
 0.6 16.04 1.18
 −1.5  =  0.2
 4 0 0  0 4
 l32 l42 

−0.3 1.180 4.1 −0.57  −0.1 l32 l33 0  0 0 l33 l43 
1.5 −1.5 −0.57 25.45 0.5 l42 l43 l44 0 0 0 l44
27
Cholesky Decomposition

    
9 0.6 −0.3 1.5 3 0 0 0 3 0.2 −0.1 0.5
 0.6 16.04 1.18
 −1.5  =  0.2
 4 0 0  0 4
 0.3 −0.4
−0.3 1.180 4.1 −0.57   −0.1 0.3 l33 0   0 0 l33 l43 
1.5 −1.5 −0.57 25.45 0.5 −0.4 l43 l44 0 0 0 l44
2
0.01 + 0.09 + l33 = 4.1 =⇒ l33 = 2
    
9 0.6 −0.3 1.5 3 0 0 0 3 0.2 −0.1 0.5
 0.6 16.04 1.18
 −1.5  =  0.2
 4 0 0 0 4
 0.3 −0.4
−0.3 1.180 4.1 −0.57 −0.1 0.3 2 0  0 0 2 l43 
1.5 −1.5 −0.57 25.45 0.5 −0.4 l43 l44 0 0 0 l44

28
Cholesky Decomposition

    
9 0.6 −0.3 1.5 3 0 0 0 3 0.2 −0.1 0.5
 0.6 16.04 1.18
 −1.5  =  0.2
 4 0 0  0 4
 0.3 −0.4 
−0.3 1.18 4.1 −0.57   −0.1 0.3 2 0  0 0 2 −0.2
1.5 −1.5 −0.57 25.45 0.5 −0.4 −0.2 l44 0 0 0 l44
2
0.25 + 0.16 + 0.04 + l44 = 25.45 =⇒ l44 = 25
    
9 0.6 −0.3 1.5 3 0 0 0 3 0.2 −0.1 0.5
 0.6 16.04 1.18
 −1.5   0.2
= 4 0 0 0 4
 0.3 −0.4
−0.3 1.18 4.1 −0.57 −0.1 0.3 2 0 0 0 2 −0.2
1.5 −1.5 −0.57 25.45 0.5 −0.4 −0.2 5 0 0 0 5
Solving part is exercise
Answer: x = (0.3277, 0.2074, 0.4580, 0.0425)′ 29
 Thanks
Doubts and Suggestions
[email protected]

30
 MA206L-Numerical Methods
Lecture 21: Numerical Linear Algebra - LU Decomposition

Panchatcharam Mariappan1

1 Associate Professor

Department of Mathematics and Statistics

IIT Tirupati, Tirupati

October 9, 2024

30