Numerical Methods

LU decomposition

Thai Minh Quan - Numerical Methods 1

Contents
1. Introduction
2. Roots of Non-linear equations
3. Systems of linear equations
4. LU decomposition
5. Linear Programming
6. Numerical Differentiation and Integration

Thai Minh Quan - Numerical Methods 2

LU decomposition

 Topics
 Eigenvalues and eigenvectors

 LU decomposition

 Decomposition with pivot

Thai Minh Quan - Numerical Methods 3

LU decomposition Eigenvalues and eigenvectors

 Objectives
Understanding the mathematical definition of eigenvalues and
eigenvectors

Understanding the physical interpretation of eigenvalues and

eigenvectors within the context of engineering systems that vibrate or
oscillate

Thai Minh Quan - Numerical Methods 4

LU decomposition Eigenvalues and eigenvectors

 MATHEMATICAL BACKGROUND

a homogeneous linear algebraic system has a right-hand side equal to zero

eigenvalue problems associated with engineering are typically of the general form

where the parameter λ is the eigenvalue

for nontrivial solutions to be possible, the determinant of the matrix must equal zero

Expanding the determinant yields a polynomial in λ, which is called the characteristic

polynomial. The roots of this polynomial are the solutions for the eigenvalues

Thai Minh Quan - Numerical Methods 5

LU decomposition Eigenvalues and eigenvectors

 Case Study

Two-dimensional matrix example

Taking the determinant to find characteristic polynomial of A

Setting the characteristic polynomial equal to zero, it has roots at λ = 1 and λ = 3,

which are the two eigenvalues of A.

Thai Minh Quan - Numerical Methods 6

LU decomposition Eigenvalues and eigenvectors

 Case Study

For λ = 1, we have that equation

Any non-zero vector with v1 = −v2 solves this equation. Therefore,

is an eigenvector of A corresponding to λ = 1, as is any scalar multiple of this vector

For λ = 3, we have that equation

is an eigenvector of A corresponding to λ = 3, as is any scalar multiple of this vector

Thai Minh Quan - Numerical Methods 7
LU decomposition

 Topics
 Eigenvalues and eigenvectors

 LU decomposition (or LU factorization)

 Decomposition with pivot

Thai Minh Quan - Numerical Methods 8

When is LU Decomposition better than
Gaussian Elimination?
To solve [A][X] = [B]
Table. Time taken by methods

Gaussian Elimination LU Decomposition

 8n3 4n   8n3 4n 
T   12n 2   T   12n 2  
 3 3   3 3 

where T = clock cycle time and n = size of the matrix

So both methods are equally efficient.

Thai Minh Quan - Numerical

Methods 9
To find inverse of [A]
Time taken by Gaussian Elimination Time taken by LU Decomposition
 nCT |FE CT |BS   CT |LU  n  CT |FS  n  CT |BS
 8n 4 4n 2   32n 3 20n 
 T   12n 
3
  T   12n 2  
 3 3   3 3 

Table 1 Comparing computational times of finding inverse of a

matrix using LU decomposition and Gaussian elimination.

n 10 100 1000 10000

CT|inverse GE / CT|inverse LU 3.28 25.83 250.8 2501

Thai Minh Quan - Numerical

Methods
10
LU decomposition

 Objectives
Another way of solving a system of equations is by using a
factorization technique for matrices called LU decomposition
Understanding that LU factorization involves decomposing the
coefficient matrix into two triangular matrices that can then be used
to efficiently evaluate different right-hand-side vectors.
Knowing how to express Gauss elimination as an LU factorization

 Applications
 Solving linear equations
 Inverting a matrix
 Computing the determinant

Thai Minh Quan - Numerical Methods 11

LU decomposition

 Definition
 In numerical analysis and linear algebra, LU decomposition (where
'LU' stands for „lower upper‟, and also called LU factorization) factors
a matrix as the product of a lower triangular matrix and an upper
triangular matrix

Lower triangular Upper triangular

matrix matrix
Thai Minh Quan - Numerical Methods 12
LU decomposition
 LU decomposition was originally derived as a decomposition of
quadratic and bilinear forms. Lagrange, in the very first paper in his
collected works (1759) derives the algorithm we call Gaussian
elimination. Later Turing introduced the LU decomposition of a matrix in
1948 that is used to solve the system of linear equation
 Let A be a m × m with nonsingular square matrix. Then there exists two
matrices L and U such that, where L is a lower triangular matrix and U is
an upper triangular matrix.
u11 u12  u1m   l11 0  0 
0 u l  0 
 u 2 m  l 22
U  22
and L   21
          
   
0 0  u mm  l m1 lm2  l mm 

Where, A  LU

Thai Minh Quan - Numerical Methods 13

LU decomposition
 Systems of linear algebraic equations

 Equation can be rearranged to give

 Suppose that Equation could be expressed as an upper triangular system. For

example, for a 3 × 3 system

 Equation can also be expressed in matrix notation and rearranged to give

Thai Minh Quan - Numerical Methods 14

LU decomposition
(*)

Lower diagonal matrix with 1‟s on the diagonal

where

 when Eq. (*) is premultiplied by [L]

 it follows from the rules for matrix multiplication that

and

Thai Minh Quan - Numerical Methods 15

LU decomposition
 A two-step strategy for obtaining solutions
1. LU factorization step
[A] is factored or “decomposed” into lower [L] and upper [U]
triangular matrices

2. Substitution step
[L] and [U] are used to determine a solution {x} for a right-hand side
{b}. This step itself consists of two steps.
a. First, generate an intermediate vector {d} by forward substitution
b. Then, the result is substituted into equation which can be solved
by back substitution for {x}

Thai Minh Quan - Numerical Methods 16

LU decomposition
 The steps in LU factorization

1. LU decomposition step

2. Substitution step

Thai Minh Quan - Numerical Methods 17

LU decomposition
 Case study
Factorize the following 2-by-2 matrix

[A] = [L] and [U] ?

One way to find the LU decomposition of this simple matrix would be to simply
solve the linear equations by inspection. Expanding the matrix multiplication
gives

The forward-substitution phase is implemented

Thai Minh Quan - Numerical Methods 18

LU decomposition
 Case study

the lower triangular matrix L is a unit triangular matrix. Then the system of
equations has the following solution

Substituting these values into the LU decomposition above yields

Thai Minh Quan - Numerical Methods 19

Method: [A] Decompose to [L] and [U]

 1 0 0 u11 u12 u13 

A  L  U    21 1 0

 0 u 22 u 23 


 31  32 1
 0 0 u33 

[U] is the same as the coefficient matrix at the end of the forward elimination step.
[L] is obtained using the multipliers that were used in the forward elimination
process

Thai Minh Quan - Numerical

20
Methods
 Finding the [U] matrix
Using the Forward Elimination Procedure of Gauss Elimination

 25 5 1
 64 8 1
 

144 12 1

 25 5 1 
Row 2  Row12.56    
Step 1: 64
 2.56;  0  4.8  1.56 
25

144 12 1  
25 5 1 
Row3  Row15.76    
144
 5.76;  0  4.8  1.56 
25

 0  16.8  4.76
Thai Minh Quan - Numerical
21
Methods
Finding the [U] Matrix

25 5 1 
Matrix after Step 1: 0  4.8  1.56 
 

0  16.8  4.76

25 5 1 
 16.8
Step 2:  3.5; Row3  Row23.5  
 0  4.8  1.56 

 4.8

0 0 0.7  

25 5 1 
U    0  4.8  1.56
 0 0 0.7 
Thai Minh Quan - Numerical
22
Methods
Finding the [L] matrix

 1 0 0
 1 0
 21 

 31  32 1


Using the multipliers used during the Forward Elimination Procedure

a 21 64
From the first step of  25 5 1   21    2.56
forward elimination  64 8 1
a11 25
   31 
a31

144
 5.76

144 12 1
 a11 25

Thai Minh Quan - Numerical

23
Methods
Finding the [L] Matrix

From the second step of 25 5 1 

0 a32  16.8
forward elimination
  4.8  1.56 

 32    3.5
a 22  4.8

0  16.8  4.76

 1 0 0
L  2.56 1 0


5.76 3.5 1


Thai Minh Quan - Numerical

24
Methods
Does [L][U] = [A]?

 1 0 0 25 5 1 
LU   2.56 1 0  0  4.8  1.56 

?
5.76 3.5 1
0 0 0.7  

Thai Minh Quan - Numerical

25
Methods
Using LU Decomposition to solve SLEs

Solve the following set of linear  25 5 1  x1  106.8 

equations using LU Decomposition  64 8 1  x   177.2 
   2  

144 12 1
 
 x3 
 
279.2

Using the procedure for finding the [L] and [U] matrices

 1 0 0 25 5 1 
A  LU   2.56 1 0

0  4.8  1.56


5.76 3.5 1
0 0 0.7 

Thai Minh Quan - Numerical

26
Methods
Example

Set [L][Z] = [C]  1 0 0   z1  106.8 

2.56 1 0  z   177.2 
  2   

5.76 3.5 1 z3 
 
279.2

Solve for [Z] z1  10

2.56 z1  z 2  177.2
5.76 z1  3.5 z 2  z 3  279.2

Thai Minh Quan - Numerical

27
Methods
Example

Complete the forward substitution to solve for [Z]

z1  106.8
z 2  177.2  2.56 z1  z1   106.8 
 177.2  2.56106.8
 96.2
Z    z2    96.21
z3  279.2  5.76 z1  3.5 z 2 
 z3    0.735 
 279.2  5.76106.8  3.5 96.21
 0.735

Thai Minh Quan - Numerical

28
Methods
Example

Set [U][X] = [Z]

25 5 1   x1   106.8 
 0  4.8  1.56  x    96.21
   2  

0 0 0.7  
 x3 
  0.735 


Solve for [X] The 3 equations become

25a1  5a2  a3  106.8

 4.8a2  1.56a3  96.21
0.7 a3  0.735
Thai Minh Quan - Numerical
29
Methods
Example

Substituting in a3 and using the second

From the 3rd equation equation

0.7 a3  0.735  4.8a2  1.56a3  96.21

a3 
0.735  96.21  1.56a3
a2 
0.7  4.8
a3  1.050  96.21  1.561.050 
a2 
 4.8
a2  19.70

Thai Minh Quan - Numerical

30
Methods
Example

Substituting in a3 and a2 using the first Hence the Solution Vector is:
equation
25a1  5a2  a3  106.8  a1  0.2900
106.8  5a2  a3 a    19.70 
a1 
25
 2  
106.8  519.70   1.050

 a3 
 
 1.050 


25
 0.2900

Thai Minh Quan - Numerical

31
Methods
LU decomposition
 Case study
use LU to solve a set of linear algebraic equations

Thai Minh Quan - Numerical Methods 32

LU decomposition
 Case study
use LU to solve a set of linear algebraic equations

the forward-elimination phase of conventional Gauss elimination resulted in

[U]

The forward-substitution phase is implemented to determine [L]

Thai Minh Quan - Numerical Methods 33

LU decomposition
 Case study forward-elimination

The forward-substitution phase is implemented

[L]

Thai Minh Quan - Numerical Methods 34

LU decomposition
 Case study
Consequently, the LU factorization is

This result can be verified by performing the multiplication of [L][U] to give

[A] =

where the minor discrepancies are due to roundoff

Thai Minh Quan - Numerical Methods 35

LU decomposition
 Case study
We can solve the first equation for d1= 7.85

which can be substituted into the second

equation to solve for

Both d1 and d2 can be substituted into the third equation to give

Thus This result can then be substituted into Eq

Thai Minh Quan - Numerical Methods 36

Finding the inverse of a square matrix

The inverse [B] of a square matrix [A] is defined as

[A][B] = [I] = [B][A]

Thai Minh Quan - Numerical Methods 37

Finding the inverse of a square matrix
[A][B] = [I] = [B][A]

How can LU Decomposition be used to find the inverse?

Assume the first column of [B] to be [b11 b12 … bn1]T
Using this and the definition of matrix multiplication

First column of [B] Second column of [B]

 b11  1   b12  0
b  0  b  1
A  21     A  22    
         
       
bn1  0  n 2  0
b

The remaining columns in [B] can be found in the same manner

Thai Minh Quan - Numerical Methods 38

Example: Inverse of a Matrix
Find the inverse of a square matrix [A]
 25 5 1
A   64 8 1


144 12 1

Using the decomposition procedure, the [L] and [U] matrices are found to be

 1 0 0 25 5 1 
A  LU   2.56 1 0  0  4.8  1.56
5.76 3.5 1  0 0 0.7 

Thai Minh Quan - Numerical Methods 39

Example: Inverse of a Matrix

Solving for the each column of [B] requires two steps

1) Solve [L] [Z] = [C] for [Z]
2) Solve [U] [X] = [Z] for [X]
 1 0 0  z1  1
Step 1: LZ   C   2.56 1 0  z2   0

5.76 3.5 1
 z3   0 

This generates the equations:
z1  1
2.56 z1  z 2  0
5.76 z1  3.5 z 2  z3  0
Thai Minh Quan - Numerical Methods 40
 Example: Inverse of a Matrix

Solving for [Z]

z1  1
 z1   1 
z 2  0  2.56 z1
 0  2.561
Z    z2    2.56
 2.56 
 z3    3.2  
z3  0  5.76 z1  3.5 z 2
 0  5.761  3.5 2.56 
 3.2

Thai Minh Quan - Numerical Methods 41

Example: Inverse of a Matrix

25 5 1  b11   1 
Solving [U][X] = [Z] for [X] 0  4.8  1.56 b    2.56
   21   

0 0 0.7  
b31 
 
 3.2 

25b11  5b21  b31  1

 4.8b21  1.56b31  2.56
0.7b31  3.2

Thai Minh Quan - Numerical Methods 42

Example: Inverse of a Matrix

Using Backward Substitution

3.2
b31   4.571 So the first column of the
0.7
inverse of [A] is:
2.56  1.560b31
b21 
4.8 b11   0.04762 
2.56  1.5604.571 b    0.9524
  0.9524
4.8  21   
b11 
1  5b21  b31 
b31 
 
 4.571  
25
1  5 0.9524   4.571
  0.04762
25

Thai Minh Quan - Numerical Methods 43

Example: Inverse of a Matrix
Repeating for the second and third columns of the inverse

Second Column Third Column

 25 5 1 b12  0  25 5 1  b13  0
 64 1 b   1  64
 8   22    8 1 b23   0
    
 1
b32 
 
0   1
144  b33  1
12
144 12 
b12   0.08333 b13   0.03571 
b    1.417  b    0.4643
 22     23   

b32 
 
  5.000 
 
b33 
 
 1.429  

Thai Minh Quan - Numerical Methods 44

Example: Inverse of a Matrix

The inverse of [A] is

 0.04762  0.08333 0.03571 
A1 
 0.9524 1.417  0.4643


 4.571  5.000 1.429 

To check your work do the following operation

[A][A]-1 = [I] = [A]-1[A]

Thai Minh Quan - Numerical Methods 45

LU decomposition
 LU decomposition with pivoting
 Just as for standard Gauss elimination, partial pivoting is necessary
to obtain reliable solutions with LU factorization
 One way to do this involves using a permutation matrix. The
approach consists of the following steps:
 Step 1- Elimination. The LU factorization with pivoting of a matrix
[A] can be represented in matrix form as
[P][A]=[L][U]

The upper triangular matrix, [U], is generated by elimination with

partial pivoting, while storing the multiplier factors in [L] and
employing the permutation matrix, [P], to keep track of the row
switches
Thai Minh Quan - Numerical Methods 46
LU decomposition
 LU decomposition with pivoting
 Step 2 - Forward substitution
The matrices [L] and [P] are used to perform the elimination step with
pivoting on {b} in order to generate the intermediate right-hand-side
vector, {d}
This step can be represented concisely as the solution of the following
matrix formulation:

[L]{d}=[P]{b}

Thai Minh Quan - Numerical Methods 47

LU decomposition
 LU decomposition with pivoting
 Step 3 – Back substitution
The final solution is generated in the same fashion as done previously
for Gauss elimination. This step can also be represented concisely as
the solution of the matrix formulation:

[U]{x}={d}

Thai Minh Quan - Numerical Methods 48

LU decomposition
 Case study
used LU to solve a set of linear algebraic equations

Before elimination, we set up the initial permutation matrix

We immediately see that pivoting is necessary, so prior to elimination

we switch the rows

[A] =

Thai Minh Quan - Numerical Methods 49

LU decomposition
 Case study
use LU to solve a set of linear algebraic equations

At the same time, we keep track of the pivot by switching the rows of
the permutation matrix:

the elimination step is complete with the result

Thai Minh Quan - Numerical Methods 50

LU decomposition
 Case study
Before implementing forward substitution, the permutation matrix is used
to reorder the right-hand-side vector to reflect the pivots as in

Then, forward substitution is applied as in

which can be solved for d1 = 1 and d2 = 2.0001 − 0.0003(1) = 1.9998.

Thai Minh Quan - Numerical Methods 51

LU decomposition
 Case study

• The LU factorization algorithm requires the same total flops as for

Gauss elimination

• The only difference is that a little less effort is expended in the

factorization phase since the operations are not applied to the right-
hand side

• Conversely, the substitution phase takes a little more effort

Which is better, Gauss Elimination

or LU Decomposition?
Thai Minh Quan - Numerical Methods 52
When is LU Decomposition better than
Gaussian Elimination?
To solve [A][X] = [B]
Table. Time taken by methods

Gaussian Elimination LU Decomposition

 8n3 4n   8n3 4n 
T   12n 2   T   12n 2  
 3 3   3 3 

where T = clock cycle time and n = size of the matrix

So both methods are equally efficient.

Thai Minh Quan - Numerical

Methods 53
To find inverse of [A]
Time taken by Gaussian Elimination Time taken by LU Decomposition
 nCT |FE CT |BS   CT |LU  n  CT |FS  n  CT |BS
 8n 4 4n 2   32n 3 20n 
 T   12n 
3
  T   12n 2  
 3 3   3 3 

Table 1 Comparing computational times of finding inverse of a

matrix using LU decomposition and Gaussian elimination.

n 10 100 1000 10000

CT|inverse GE / CT|inverse LU 3.28 25.83 250.8 2501

Thai Minh Quan - Numerical

Methods
54
LU decomposition

Exercises
Use the LU factorization to find the solution

Thai Minh Quan - Numerical Methods 55

LU decomposition

Exercises
Use the LU factorization to find the solution

[L]= [U]=

[x]= {1 2 -1}

Thai Minh Quan - Numerical Methods 56

Thai Minh Quan - Numerical Methods 57
LU decomposition

Matlab

Thai Minh Quan - Numerical Methods 58

LU decomposition Matlab

Eigenvalues and Eigenvectors With MATLAB

Use MATLAB to determine all the eigenvalues and eigenvectors for the system

The matrix can be entered as

>> A = [40 -20 0;-20 40 -20;0 -20 40];

If we just desire the eigenvalues, we can enter

>> e = eig(A) e=
11.7157
40.0000
68.2843
Thai Minh Quan - Numerical Methods 59
LU decomposition Matlab

Eigenvalues and Eigenvectors With MATLAB

Use MATLAB to determine all the eigenvalues and eigenvectors for the system

If we want both the eigenvalues and eigenvectors, we can enter

>> [v,d] = eig(A)

full matrix whose columns are the corresponding eigenvectors

Thai Minh Quan - Numerical Methods 60
LU decomposition Matlab

SOLVING LINEAR ALGEBRAIC EQUATIONS WITH MATLAB

Use MATLAB to compute the LU factorization and find the solution

The coefficient matrix and the right-hand-side vector can be entered in standard fashion as

>> A = [3 -.1 -.2;.1 7 -.3;.3 -.2 10];

>> b = [7.85; -19.3; 71.4];

Next, the LU factorization can be computed with

>> [L,U] = lu(A)

We can test that it is correct by computing the original matrix as >> L*U
Thai Minh Quan - Numerical Methods 61
LU decomposition Matlab

To generate the solution, we first compute

>> d = L\b

x= 3.0000
And then use this result to compute the solution -2.5000
>> x = U\d 7.0000

Thai Minh Quan - Numerical Methods 62

 LU decomposition Matlab
Exercises
function [L,U,P] = mylup(A) for
i = j+1:n
[n,m] = size(A); t = C(i,j)/C(j,j);
C = zeros(n,n+1); C(i,j) = t;
% initialize C(i,j+1:n) = C(i,j+1:n)
C(1:n,1:n) = A; - t*C(j, j+1:n);
C(1:n,n+1) = (1:n)'; end
% main algorithm end
for j=1:n-1 % refine the results
if (sum(C(1:n,j))==0) PP = C(1:n,n+1);
error('mylup:input','The solution C = C(1:n,1:n);
is not determined.'); L = tril(C,-1);
end U = triu(C);
[~,i]= max(abs(C(1:n,j))); P = zeros(n,n);
if (i~= j) for i=1:n
tt = C(i,1:n+1); L(i,i)=1;
C(i,1:n+1) = C(j,1:n+1); P(i, PP(i))=1;
C(j,1:n+1) = tt; end
end end
Thai Minh Quan - Numerical Methods 63