Eigenvalues and Eigenvectors

1
Aim:
To construct strong knowledge in theory of
Matrix and its application of matrices

Objectives:

Introducing the basic concepts of

Eigenvalues and Eigenvectors

2
 Prerequisites:
Students should have basic
knowledge in the concepts and
applications of matrices.

Learning Outcome:
Students should be able to do basic
computation of Eigenvalues and
Eigenvectors 3
 Recall
• Matrix
• Order of a Matrix
• Determinant
• Transpose of a Matrix
• Identity Matrix
• Multiplication of two Matrices
• Inverse of a Matrix
• Symmetric and Non-symmetric Matrix
• Singular and Non-singular Matrix
4
 Outline

Eigenvalues and Eigenvectors

• Introduction
• Definition of Matrix and various terms
• Characteristic Equation of a matrix
• Properties of Eigenvalues
• Finding Eigenvalues and Eigenvectors

5
 Eigenvalues and Eigenvectors
Definition: Matrix
A system of mn numbers(elements) arranged in a rectangular arrangement along m rows
and n columns and bounded by the brackets [ ] or ( ) is called an m by n matrix, which is
written as m  n matrix
 a11 a12 .. .. a1n 
a a .. .. a 
 21 22 2 n 
A =  ... .. .. .. .. 
 
 .. .. .. .. .. 
a m1 a m 2 .. .. a mn 
Characteristic polynomial
The determinant |A- I | when expanded will give a polynomial, which we call as
characteristic polynomial of matrix A.

Definition: Eigenvalues
A = [ a ij ] be a square matrix.
The characteristic equation of A is |A- I | = 0.
The roots of the characteristic equation are called Eigenvalues of A.
6
Definition: Eigenvectors
A = [ aij ] be a square matrix of order ‘n’
 x1 
x 
 2
If there exist a non zero vector X =  . 
 
.
 xn 
such that AX = X , then the vector X is called an Eigenvector of A corresponding to the
Eigenvalue  .

Method of finding characteristic equation of a 3x3 matrix and 2x2 matrix

The characteristic equation of a 3x3 matrix is 3  S12  S 2   S3  0
Where, S1= sum of main diagonal elements.
S2 = sum of minor of main diagonal elements.
S3 = Det (A) =|A|
The characteristic equation of a 2x2 matrix is  2  S1  S2  0
Where, S1 = sum of main diagonal elements.
S2 = Det (A) =|A|

7
 1 2
1. Find the characteristic equation of the matrix  
0 2
Solution:
The characteristic equation is  2  S 1   S 2  0
S1 = sum of main diagonal elements
= 1+2=3
S2 = Det (A) =|A|
1 2
=
0 2
S2 = 2-0 = 2
The characteristic equation is  2  3   2  0 .

 2  3 1 
 
2. Find the characteristic equation of  3 1 3 
 5 2  4 

Solution:
The characteristic equation is  3  S 1  2  S 2   S 3  0
Where, S1 = sum of the main diagonal elements
= 2+1-4 = -1
S2 = sum of minor of main diagonal elements
1 3 2 1 2  3
= + +
2  4  5  4 3 1
= (-4-6)+(-8+5)+(2+9) = -10+(-3)+11 = -2
S3 = Det (A) =|A|
2  3 1
= 3 1 3
 5 2  4
= 2(-4-6)-(-3)(-12+15)+1(6+5)
= 2(-10) +3(3) +1(11) = -20+9+11= 0
The characteristic equation is  3   2  2  0
8
  1  1 0
 
3. Find the Eigenvalues of   1 2 1
 0 1 1
 
Solution:
The characteristic equation is 3  S12  S 2   S3  0
S1 = 1+2+1=4
2 1 1 0 1 1
S2 = + +
1 1 0 1 1 2
= (2-1)+(1-0)+(2-1) = 3
1 1 0
S3 =  1 2 1 = 1(2-1)+(-1-0)+0 = 0
0 1 1
Therefore the characteristic equation is 3  42  3  0  0
To find the Eigenvalues
3  42  3  0  0
(2 -4+3) = 0
= 0, (2 -4+3) = 0

9
 (-1)(-3) = 0
 = 1, 3
The Eigen values are 1, 3, and 0.

Properties of Eigenvalues.

i. The sum of the Eigenvalues of a matrix is the sum of the elements of main
diagonal
ii. The product of the Eigenvalues is equal to the determinant of the matrix.
iii. The Eigen values of the triangular matrix are just the diagonal element of the
matrix
iv. If  is an Eigenvalue of a matrix A, then 1 , (0) is Eigen value of A-1

v. If 1, 2, ………n are the Eigenvalues of a matrix A, then Am has a Eigenvalues
1m , m2 ,.......mn

10
 2 1 2 
1. Find the sum & product of the Eigenvalues of the matrix A=  1 3 1 
 2 1  6 
Solution:
Sum of the Eigen values = sum of the main diagonal elements
= 2+3-6=-1
Product of the Eigen value = |A|
3 1 1 1 1 3
2 1 2
1 6 2 6 2 1
= 2(-18-1)-1(-6-2)+2(1-6) = - 40
Sum = -1 and Product = - 40

 6 2 2 
 
2. The product of two Eigenvalues of the matrix A=   2 3  1  is 16. Find the third
 2 1 3 

Eigenvalue of A.
Solution:
Given: The product of two Eigen values of A is 16
(i.e) 12 =16
By property, Product of Eigen values = |A|
11
 123 = |A|
6 2 2
163 =  2 3  1
2 1 3
= 6(9-1)+2(-6+2)+2(2-6)= 32
32
3 = =2
16
The third Eigen value is 2.
 8 6 2 
 
3. Two Eigenvalues of the matrix A =  6 7 4  are 3 and 0. What is the third
 2 4 3 
 
eigenvalue?
What is the product of the eigenvalues of A?
Solution:
Given: If 1 = 3, 2 = 0, and 3 =?
By property, Sum of the Eigenvalues = sum of the main diagonals.
1+2+3 = 8+7+3= 18
3+0+3 = 18
3 = 18 - 3 = 15

12
By property, Product of the Eigen values = |A|
(3)(0)(15) = |A|
|A| = 0
The third eigenvalue is 15, The product of the eigenvalues of A is 0.
 8 6 2 
 
4. If 3 and 15 are two Eigenvalues of the matrix A =  6 7 4  then find the third
 2 4 3 
 
eigenvalue and hence = |A|
Solution:
Given: If 1 = 3, 2 = 15, and 3 =?
By property, Sum of the Eigenvalues = sum of the main diagonals.
1+2+3 = 8+7+3= 18
3+15+3 = 18
3 = 18 - 18 = 0
By property, Product of the Eigenvalues = |A|
(3)(15)(0) = |A|
|A| = 0
The third eigenvalue is 0, The product of the eigenvalues of A is 0.

13
 Non-Symmetric Matrix With Non-Repeated Eigenvalues
 1 1 4 
 
1. Find all the Eigenvalues and Eigenvectors of the matrix  3 2  1
 2 1  1
 
Solution:
 1 1 4 
 
Given: A=  3 2  1
 2 1  1
 
To find the characteristic equation of A
Formula: The characteristic equation of A is 3  S12  S 2   S 3  0
Where, S1= sum of main diagonal
=1+2-1=2
S2 = sum of minor of main diagonal elements
2 1 1 4 1 1
= + +
1 1 2 1 3 2
= (-2+1)+(-1-8)+(2+3) = -1-9+5 = -5

14
 S3 = Det (A)=|A|
1 1 4
= 3 2  1 =1(-2+1)+1(-3+2)+4(3-4)
2 1 1
=1(-1)+1(-1)+4(-1) = -1-1-4= - 6
Hence the characteristic equation is 3  22 5  6  0
To solve the characteristic equation:
If =1 By synthetic division 1 1 -2 -5 6
0 1 -1 -6
1 -1 -6 0
Therefore the =1 and other roots are given by 2    6  0
(+2)(-3) = 0
 = -2, 3
Therefore Eigenvalues are 1,-2, 3

To find the Eigenvectors:

To get the Eigenvectors solve: (A-λI)X=0

15
 1 1 4  1 0 0    x1   0 
      
 3 2  1    0 1 0  x 2    0 
  2 1  1  0
 0 1    x 3   0 
 1   1 4    x1   0 
     
 3 2    1  x 2    0 
  2 1  1      x 3   0 
(1-λ)x1-x2+4x3=0
3x1+(2-λ)x2-x3=0 ….(1)
2x1+x2+(-1-λ)x3=0
Case 1: Substitute λ=1 in, (1) we get
0x1-x2+4x3=0 …. (2)
3x1+x2-x3=0 …. (3)
2x1+x2-2x3=0 …. (4)
Solving (2) and (3) by cross multiplication rule, we get
x1 x2 x3
-1 4 0 -1
1 -1 3 1
x1 x2 x3
 
1 4 12  0 0  3
x1 x2 x3
  
3 12 3
16
 x1 x2 x3
  
1 4 1
  1
 
Therefore X1=  4 
 1 
 
Case 2: Substitute λ= -2 in (1), we get
3x1-x2+4x3=0 ….(5)
3x1+4x2-x3=0 ….(6)
2x1+x2+x3=0 ….(7)
Solving (5) and (6) by cross multiplication rule we get

x1 x2 x3
-1 4 3 -1
4 -1 3 4
x1 x2 x3
 
1  16 12  3 12  3
x1 x x
  2  3
 15 15 15
x1 x2 x3
  
1 1 1
  1
 
Therefore X2=  1 
 1 
 

17
Case 3: Substitute λ=3 in (1) we get
-2x1-x2+4x3=0 …. (8)
3x1-x2-x3=0 …. (9)
2x1+x2-4x3=0 …. (10)
Solving (8) and (9) by cross multiplication rule we get
x1 x2 x3
-1 4 -2 -1
-1 -1 3 -1
x1 x2 x3
 
1  4 12  2 23
x1 x 2 x3
  
5 10 5
x1 x2 x3
  
1 2 1
1
 
Therefore X3=  2 
1
 
  1   1 1
     
Result: The Eigen values of A are 1,-2, 3 and the Eigenvectors are  4  ,  1  ,  2
1 1 1
     

18
 Non-Symmetric Matrix With Repeated Eigenvalues
  2 2  3
 
1. Find all the Eigenvalues and Eigenvectors of the matrix  2 1  6
 1  2 0 
 
Solution:
 2 2  3
Given: A=  2 1  6

 1 2 0 

To find the characteristic equation of A
Formula: The characteristic equation of A is  3  S 1  2  S 2   S 3  0
Where, S1 = sum of main diagonal
= -2+1+0=-1
S2 = sum of minor of main diagonal elements
1 6 2 3 2 2
= + +
2 0 1 0 2 1
= (0-12)+(0-3)+(-2-4) = -12-3-6=-21
S3 = Det (A)=|A|
  2 2  3
 
= 2 1  6  = -2(0-12)-2(0-6)-3(-4+1) =45
 1  2 0 
 
Hence the characteristic equation is 3  2  21  45  0
19
 if λ=1; 1+1-21-45 ≠0
if λ=-1; -1+1-21-45 ≠0
if λ=2; 8+4-42-45 ≠0
if λ=-2; -8+4+42-45≠0
if λ=3; 27+9-63-45≠0
if λ=-3; -27+9+63-45≠0
Therefore λ=-3 is a root
By synthetic division -3 1 1 -21 -45
0 -3 6 45
1 -2 -15 0
Therefore the = -3 and other roots are given by 2  2  15  0
(-5)(+3) = 0
= 5,-3,-3
Therefore Eigenvalues are 5, -3,-3 and Here the Eigenvalues are repeated.
To find the Eigenvectors:
To get the Eigenvectors solve (A-λI)X=0
  2 2  3    0 0  x1   0 
      
 2 1  6 
  0  0  x 2    0 
  1  2 0   0 0   x3   0 

20
21
 Since (5),(6),(7) are all same, So we considered only one equation
x1+2x2-3x3=0
Put x1=0
2x2-3x3=0
 2x2=3x3
x2 x
 3
3 2
0
 
Therefore Eigenvector is X2=  3 
2
 
Put x2=0
x1-3x3=0
 x1=3x3
x1 x
 3
3 1
3
 
Therefore Eigenvector is X3=  0 
1
 

 1 0 3

     
Result: The Eigenvalues are-3,-3,5 and Eigenvectors are   2  ,  3  ,0
 1  2 1
     

22
 Symmetric Matrix With Non-Repeated Eigenvalues
1 1 3
 
1. Find all the Eigen values and Eigen vectors of the matrix  1 5 1 
3 1 1
 
Solution:
1 1 3
 
Given: A=  1 5 1 
3 1 1
 
To find the characteristic equation of A
Formula: The characteristic equation of A is  3  S 1  2  S 2   S 3  0
Where, S1 = sum of main diagonal
=1+5+1=7
S2 = sum of minor of main diagonal elements
5 1 1 3 1 1
= + + = (5-1)+(1-9)+(5-1)=0
1 1 3 1 1 5
1 1 3
S3 = |A|= 1 5 1 =1(5-1)-1(1-3)+3(1-15) = -36
3 1 1
Hence the characteristic equation is   7  0  36  0
3 2

if λ=1; 1-7+0+36 ≠0
if λ=-1; -1-7+0+36 ≠0
if λ=2; 8-24+0+36 ≠0
if λ=-2; -8-24+0+36=0
λ= -2 is a root

23
To solve the characteristic equation:
if =-2 By synthetic division -2 1 -7 0 36
0 -2 18 -36
1 -9 18 0
Therefore the =-2 and other roots are given by  2  9  18  0
(-6)(-3) = 0
= 3,6
Therefore Eigenvalues are-2, 3, 6
To find the Eigenvectors:
To get the Eigenvectors solve (A-λI)X=0
 1 1 3    0 0   x1   0 
      
 1 5 1 
  0  0  x 2    0 
 3 1 1   0 0    x 3   0 

(1-λ)x1+x2+3x3=0
x1+(5-λ)x2+x3=0 …(1)
3x1+x2+(1-λ)x3=0
Case 1: Substitute λ=-2 in (1), we get
3x1+x2+3x3=0 …(2)
x1+7x2+x3=0 …(3)
3x1+x2+3x3=0 …(4)

24
 Since (2) and (4) are same we consider, solving (2)and (3) by cross multiplication
rule we get
x1 x2 x3
1 3 3 1
7 1 1 7
x1 x2 x3
 
1  21 33 21  1
x1 x x x1 x x
  2  3   2  3
 20 0 20  10 0 10
  1
 
Therefore X1 =  0 
 1 
 
Case 2: Substitute λ=3 in (1), we get
-2x1+x2+3x3=0 … (5)
x1+2x2+x3=0 …(6)
3x1+x2-2x3=0 … (7)
Solving (5) and (6) by cross multiplication rule we get
x1 x2 x3
1 3 -2 1
2 1 1 2
x1 x2 x3
 
1 6 23  4 1
x1 x2 x3 x1 x2 x3
     
5 5 5 1 1 1
25
   1
 
Therefore X2=  1 
  1
 
Case 3: Substitute λ=6 in (1), we get
-5x1+x2+3x3=0 … (8)
x1-x2+x3=0 … (9)
3x1+x2-5x3=0 …(10)

x1 x2 x3
1 3 -5 1
-1 1 1 -1
x1 x2 x3
 
1 3 35 5 1
x1 x2 x3 x1 x2 x3
     
4 8 4 1 2 1
1
 
Therefore X3=  2 
1
 
Therefore the Eigenvalues of A are 6,-2, 3
  1   1  1 
     
Result: The Eigenvalues of A are 6,-2, 3 and the Eigenvectors are  0  ,  1  ,  2 
 1    1  1 
     
26
 Symmetric Matrix With Repeated Eigenvalues

 6 2 2 
 
1. Find all the Eigenvalues and Eigenvectors of   2 3  1
 2 1 3 

Solution:
 6 2 2 
 
Given: A=   2 3  1
 2 1 3 

To find the characteristic equation of A
The characteristic equation of A is  3  S 1  2  S 2   S 3  0
Where, S1= sum of main diagonal
=6+3+3=12
S2 = sum of minor of main diagonal elements
3 1 6 2 6 2
= + +
1 3 2 3 2 3
= (9-1)+(18-4)+(18-4) = 8+14+14 = 36

S3 = Det (A)=|A|
6 2 2
=2 3  1 =32
2 1 3

27
 Hence the characteristic equation is 3 122  36  32  0
if λ=1; 1-12+36-32 ≠0
if λ=-1; -1-12-36-32≠0
if λ=2;8-42+72-32=0

By synthetic division 2 1 -12 36 -32

0 2 -20 32
1 -10 16 0
Therefore the =2 is a root
and other roots are given by  2  10   16  0
(-8)(-2) = 0
 = 8, 2
Therefore Eigenvalues are 8, 2, 2.

To find the Eigenvectors:

To get the Eigenvectors solve (A-λI) X=0
6   2 2   x1   0 
 2 3  1   x 2    0  ….. (A)

 2  1 3     x 3   0 
Case (1): If λ = 8, then the equation (A) becomes
  2  2 2   x1   0 
  2  5  1   x   0 
  2   
 2  1  5   x 3   0 

28
 (i.e)-2x1-2x2+2x3=0 .... (1)
-2x1-5x2-x3=0 ….. (2)

2x1-x2-5x3=0 ….. (3)

Solving (1) and (2) by rule of cross multiplication, we get
x1 x2 x3
-2 2 -2 -2
-5 -1 -2 -5
x1 x2 x3
 
2  10  4  2 10  4
x x x
 1  2  3
12 6 6
x x x
 1  2  3
2 1 1
 2 
Hence the corresponding Eigenvector is X1=   1 
 
 1 

Case (2): If λ = 2 then the equation (A) becomes

 4 2 2   x1   0 
 2 1  1   x   0 
  2   
 2 1 1   x 3   0 
(i.e) 4x1-2x2+2x3=0 …. (4)
-
2x1+x2-x3=0 …. (5)
2x1-x2+x3=0 …. (6)

29
Here (4), (5), (6) represents the same equation,
2x1-x2+x3=0
If x1=0 we get -x2+x3=0
-x2 =- x3
x2 = x3
x x
(i.e) 2  3
1 1
0
Hence the corresponding eigenvector is X2= 1 
 
1 
l 
Let X3=  m  as x3 is orthogonal to x1 and x2 since the given matrix is symmetric
 
 n 
l 
[2 -1 1] m   0 or 2l-m+n=0 …. (7)
 
 n 
l 
[0 1 1] m   0 or 0l+m+n=0 …. (8)
 
 n 
30
 Solving (7) and (8) by rule of cross multiplication, we get
l m n
-1 1 2 -1
1 1 0 1
l m n
 
1  1 0  2 2  0
l m n
  
2 2 2
1
Hence the corresponding Eigenvector is X3=  1 
 
1
 2  0  1 
Result: The Eigenvalues are 8, 2, 2 and the Eigenvectors are  1 , 1 ,  1 
     
 1  1 1

31
 References:
Eigenvalues and Eigenvectors

Video Link: https://youtu.be/T8dPpuc8YN8

Book List:
1. A. K. Ghatak, I. C. Goyal and A. J. Chua, Mathematical Physics (McMillan, New
Delhi 1995).
2. P. K. Chattopadhyay, Mathematical Physics (Wiley, Eastern, New Delhi, 1990)
3. Sathyaprakash, Mathematical Physics
4. H. K. Dass – Mathematical Physics
5. B. D. Gupta – Mathematical Physics

32
Thank you

33
Solution: Let A = ( ) which is a non-symmetric matrix

To find the characteristic equation:

The characteristic equation of A is where

,

= 1(-1) – 1(3) = - 4

Therefore, the characteristic equation is i.e., or

Therefore, the eigen values are 2, -2

A is a non-symmetric matrix with non- repeated eigen values

To find the eigen vectors:

[( ) ( )] [ ] [ ] [( ) ( )] [ ] [ ]

[ ][ ] [ ]--------------- (1)

Case 1: If [ ][ ] [ ] From (1)]

i.e., [ ][ ] [ ]

i.e.,

i.e., we get only one equation

Therefore [ ]

Case 2: If [ ][ ] [ ] From (1)]

i.e., [ ][ ] [ ]

i.e.,

i.e., we get only one equation

Hence, [ ]

2. Find the eigen values and eigen vectors of [ ]

Solution: Let A = [ ] which is a non-symmetric matrix

To find the characteristic equation:

Its characteristic equation can be written as where

| | | | | |

= 2(4)-2(1)+1(-1) = 5

Therefore, the characteristic equation of A is

1 -6 5 0

√ √
Therefore, the eigen values are 1, 1, and 5

A is a non-symmetric matrix with repeated eigen values

To find the eigen vectors:

[ ][ ] [ ]

Case 1: If [ ][ ] [ ]

i.e., [ ][ ] [ ]

--------- (1)

------------- (2)

------------ (3)

Considering equations (1) and (2) and using method of cross-multiplication, we get,

2 1 -3 2

-2 1 1 -2

Therefore, [ ]

Case 2: If ,[ ][ ] [ ]
i.e., [ ][ ] [ ]

All the three equations are one and the same. Therefore,

Put

Therefore, [ ]

Put

Therefore, [ ]

3. Find the eigen values and eigen vectors of [ ]

Solution: Let A = [ ] which is a non-symmetric matrix

To find the characteristic equation:

Its characteristic equation can be written as where

,

| | | | | |

,
= 2(-4)+2(-2)+2(2) = - 8 – 4 + 4 = - 8
Therefore, the characteristic equation of A is
 1 0 -4 0

Therefore, the eigen values are 2, 2, and -2

A is a non-symmetric matrix with repeated eigen values

To find the eigen vectors:

[ ][ ] [ ]

Case 1: If [ ][ ] [ ]

i.e., [ ][ ] [ ]

--------- (1)

------------- (2)

------------ (3) . Equations (2) and (3) are one and the same.

Considering equations (1) and (2) and using method of cross-multiplication, we get,

-1 1 2 -1

3 1 1 3

Therefore, [ ]
Case 2: If ,[ ][ ] [ ]

i.e., [ ][ ] [ ]

---------- (1)

---------------- (2)

------------ (3)

Considering equations (1) and (2) and using method of cross-multiplication, we get,

-2 2 0 -2

-1 1 1 -1

Therefore, [ ]

We get one eigen vector corresponding to the repeated root

4. Find the eigen values and eigen vectors of [ ]

Solution: Let A =[ ] which is a symmetric matrix

To find the characteristic equation:

Its characteristic equation can be written as where

 ,

| | | | | |

,
= 1(4)-1(-2)+3(-14) = - 4 + 2-42 = - 36

Therefore, the characteristic equation of A is

1 -9 18 0

√ √

Therefore, the eigen values are -2, 3, and 6

A is a symmetric matrix with non- repeated eigen values

To find the eigen vectors:

[ ][ ] [ ]

Case 1: If [ ][ ] [ ]

i.e., [ ][ ] [ ]

--------- (1)

------------- (2)

------------ (3)
Considering equations (1) and (2) and using method of cross-multiplication, we get,

1 3 3 1

7 1 1 7

Therefore, [ ]

Case 2: If ,[ ][ ] [ ]

i.e., [ ][ ] [ ]

---------- (1)

---------------- (2)

3 ------------ (3)

Considering equations (1) and (2) and using method of cross-multiplication, we get,

1 3 -2 1

2 1 1 2

Therefore, [ ]

Case 3: If ,[ ][ ] [ ]
i.e., [ ][ ] [ ]

---------- (1)

---------------- (2)

3 ------------ (3)

Considering equations (1) and (2) and using method of cross-multiplication, we get,

1 3 -5 1

-1 1 1 -1

Therefore, [ ]

5. Find the eigen values and eigen vectors of the matrix[ ]. Determine the

algebraic and geometric multiplicity

Solution: Let A =[ ] which is a symmetric matrix

To find the characteristic equation:

Its characteristic equation can be written as where

,

| | | | | |

,
= 0 -1(-1)+ 1(1) = 0 + 1 + 1 = 2

Therefore, the characteristic equation of A is

 1 -1 -2 0

√ √

Therefore, the eigen values are 2, -1, and -1

A is a symmetric matrix with repeated eigen values. The algebraic multiplicity of is 2

To find the eigen vectors:

[ ][ ] [ ]

Case 1: If [ ][ ] [ ]

i.e., [ ][ ] [ ]

--------- (1)

------------- (2)

------------ (3)

Considering equations (1) and (2) and using method of cross-multiplication, we get,

1 1 -2 1

-2 1 1 -2
Therefore, [ ]

Case 2: If ,[ ][ ] [ ]

i.e., [ ][ ] [ ]

---------- (1)

---------------- (2)

------------ (3). All the three equations are one and the same.

Therefore, . Put

Therefore, [ ]

Since the given matrix is symmetric and the eigen values are repeated, let [ ]. is

orthogonal to

][ ] ------------ (1)

][ ] -------- (2)

Solving (1) and (2) by method of cross-multiplication, we get,

l m

1 1 1 1

1 -1 0 1
 . Therefore, [ ]

Thus, for the repeated eigen value there corresponds two linearly independent eigen
vectors . So, the geometric multiplicity of eigen value is 2

Problems under properties of eigen values and eigen vectors.

1. Find the sum and product of the eigen values of the matrix [ ]

Solution: Sum of the eigen values = Sum of the main diagonal elements = -3

Product of the eigen values = │A│ = -1 (1 – 1) -1(-1 – 1) + 1(1- (-1)) = 2 + 2 = 4

2. Product of two eigen values of the matrix A = [ ] is 16. Find the third eigen

value
Solution: Let the eigen values of the matrix be .

Given

We know that (Since product of the eigen values is equal to the determinant of
the matrix)

| | = 6(9-1)+2(-6+2) +2(2-6) = 48-8-8 = 32

3. Find the sum and product of the eigen values of the matrix A = ( ) without

finding the roots of the characteristic equation

Solution:We know that the sum of the eigen values = Trace of A = a + d

Product of the eigen values = │A│ = ad – bc

4. If 3 and 15 are the two eigen values of A = [ ], find │A│, without

expanding the determinant

Solution:Given

We know that sum of the eigen values = Sum of the main diagonal elements

5. If 2, 2, 3 are the eigen values of A = [ ], find the eigen values of

Solution:By the property “A square matrix A and its transpose have the same eigen
values”, the eigen values of

6. Find the eigen values of A = [ ]

Solution:Given A = [ ] .Clearly, A is a lower triangular matrix. Hence, by the

property “the characteristic roots of a triangular matrix are just the diagonal elements of the
matrix”, the eigen values of A are 2, 3, 4

7. Two of the eigen values of A = [ ] are 3 and 6. Find the eigen values of

Solution:Sum of the eigen values = Sum of the main diagonal elements = 3 +5+3 = 11

Given 3,6 are two eigen values of A. Let the third eigen value be k.
 Then, 3 + 6 + k = 11

Therefore, the eigen values of A are 3, 6, 2

By the property “If the eigen values of A are , then the eigen values of
are ”, the eigen values of are

8. Find the eigen values of the matrix[ ]. Hence, form the matrix whose eigen

values are

Solution: Let A =[ ]. The characteristic equation of the given matrix is

where and

√
Therefore, the characteristic equation is =

Therefore, the eigen values of A are 6, -1

Hence, the matrix whose eigen values are is

│A│ = 4 -10 = - 6; adj A = [ ]

Therefore, [ ]

9. Find the eigen values of the inverse of the matrix A = [ ]

Solution:We know that A is an upper triangular matrix. Therefore, the eigen values of A are
2, 3, 4. Hence, by using the property “If the eigen values of A are , then the eigen
values of are ”, the eigen values of

10. Find the eigen values of given A = [ ]

Solution:Given A = [ ]. A is an upper triangular matrix. Hence, the eigen values of

A are 1, 2, 3

Therefore, the eigen values of are i.e., 1,8,27

11. If 1 and 2 are the eigen values of a 2 x 2 matrix A, what are the eigen values of
?

Solution:Given 1 and 2 are the eigen values of A.

Therefore, i.e., 1 and 4 are the eigen values of and 1 and are the eigen

values of

12. If 1,1,5 are the eigen values of A = [ ], find the eigen values of 5A

Solution:By the property “If are the eigen values of A, then are the
eigen values of kA, the eigen values of 5A are 5(1), 5(1), 5(5) ie., 5,5,25

13. Find the eigen values of A, if A = [ ]

Solution:Given A = [ ]. A is an upper triangular matrix. Hence, the eigen values of A are

2, 5

The eigen values of are i.e., 4, 25

The eigen values of are i.e., 8, 125

The eigen values of are i.e., 16, 625

The eigen values of 3A are 3(2), 3(5) i.e., 6, 15

The eigen values of are

A–I=[ ]-[ ]=[ ]

Since A - I is an upper triangular matrix, the eigen values of A- I are its main diagonal
elements i.e., 1,4

Eigen values of where

First eigen value =

= 3(2)3+ 5(2)2 - 6(2) + 2 = 24 + 20 -12 + 2=34

Second eigen value =

= 3(5)3+ 5(5)2- 6(5) + 2

= 375+ 125-30 + 2 = 472

14. Find the eigen values of adj A if A = [ ]

Solution:Given A =[ ]. A is an upper triangular matrix. Hence, the eigen values of A

are 3, 4, 1

We know that

Adj A = │A│

The eigen values of are

│A│=Product of the eigen values = 12

Therefore, the eigen values of adj A is equal to the eigen values of 12 i.e., i.e.,

4, 3, 12

Note: A =[ ] [ ] [ ]. Here, A is an upper triangular matrix,

B is a lower triangular matrix and C is a diagonal matrix. In all the cases, the elements in the
main diagonal are the eigen values. Hence, the eigen values of A, B and C are 1, 4, 6
 15. Two eigen values of A = [ ] are equal and they are times the third. Find

them

Solution:Let the third eigen value be

We know that 2+3+2 = 7

Given =

[ ]

Therefore, and hence the eigen values of A are 1,1, 5

16. If 2, 3 are the eigen values of [ ]

Solution:Let A =[ ]. Let the eigen values of A be 2, 3, k

We know that the sum of the eigen values = sum of the main diagonal elements

Therefore, 2 +3 +k = 2+ 2+2 = 6

We know that product of the eigen values = │A│

2(3)(k) = │A│

| |

17. Prove that the eigen vectors of the real symmetric matrix A = [ ] are

orthogonal in pairs
Solution:The characteristic equation of A is
where ;

| |

The characteristic equation of A is

3 1-7 0 36

0 3 -12 -36

1 -4 -12 0

√
Therefore,

Therefore, the eigen values of A are -2, 3, 6

To find the eigen vectors:

Case 1: When [ ][ ] [ ]

-------- (1)

---------- (2)

--------- (3)

Solving (1) and (2) by rule of cross-multiplication, we get,

1 3 3 1

7 1 1 7

[ ]
Case 2:When [ ][ ] [ ]

--------- (1)

--------- (2)

--------- (3)

Solving (1) and (2) by rule of cross-multiplication, we get,

1 3 -2 1

2 1 1 2

[ ]

Case 3:When [ ][ ] [ ]

--------- (1)

--------- (2)

--------- (3)

Solving (1) and (2) by rule of cross-multiplication, we get,

1 3 -5 1

-1 1 1 -1

[ ]

Therefore, [ ], [ ] [ ]

To prove that:

][ ]
 ][ ]

][ ]

Hence, the eigen vectors are orthogonal in pairs

18. Find the sum and product of all the eigen values of the matrix A = [ ].Is the

matrix singular?

Solution:Sum of the eigen values = Sum of the main diagonal elements =Trace of the matrix

Therefore, the sum of the eigen values = 1+2+7=10

Product of the eigen values = │A│ = 1(14 - 8) -2(14 - 4) + 3(4 - 2) = 6-20+ 6= - 8

│A│≠0. Hence the matrix is non-singular.

19. Find the product of the eigen values of A = [ ]

Solution:Product of the eigen values of A = │A│=| |

ORTHOGONAL TRANSFORMATION OF A SYMMETRIC MATRIX TODIAGONAL FORM:

Orthogonal matrices:

A square matrix A (with real elements) is said to be orthogonal if or

Problems:

1. Check whether the matrix B is orthogonal. Justify. B = [ ]

Solution: Condition for orthogonality is

To prove that:

B=[ ]; [ ]
 [ ][ ]

[ ] [ ]

Similarly,

[ ][ ]

[ ] [ ]

Therefore, B is an orthogonal matrix

2. Show that the matrix P = [ ] is orthogonal

Solution:To prove that:

[ ]; [ ]

[ ] [ ]= I

[ ][ ]

[ ] [ ]

Therefore, P is an orthogonal matrix

WORKING RULE FOR DIAGONALIZATION

[ORTHOGONAL TRANSFORMATION]:

Step 1: To find the characteristic equation

Step 2: To solve the characteristic equation

Step 3:To find the eigen vectors

Step 4: If the eigen vectors are orthogonal, then form a normalized matrix N

Step 5: Find

Step 6: Calculate AN

Step 7: Calculate D =
Problems:

1. Diagonalize the matrix [ ]

Solution: Let A = [ ]

The characteristic equation is where

Therefore, the characteristic equation is

2 1 -11 36 -36

0 2 -18 36

1 -9 18 0

√ √
=

Hence, the eigen values of A are 2, 3, 6

To find the eigen vectors:

[ ][ ] [ ]

Case 1: When [ ][ ] [ ]

---------- (1)

-------- (2)

--------- (3)

Solving (1) and (2) by rule of cross-multiplication,

-1 1 1 -1

3 -1 -1 3

[ ]

Case 2: When [ ][ ] [ ]

---------- (1)

-------- (2)

--------- (3)

Solving (1) and (2) by rule of cross-multiplication,

-1 1 0 -1

2 -1 -1 2

[ ]

Case 3: When [ ][ ] [ ]

---------- (1)

-------- (2)

--------- (3)

Solving (1) and (2) by rule of cross-multiplication,

-1 1 -3 -1

-1 -1 -1 -1

[ ]

][ ]

][ ] =0

][ ]

Hence, the eigen vectors are orthogonal to each other

√ √ √ √ √ √
The Normalized matrix N = √ √ √
; √ √ √

[√ √ √ ] [√ √ √ ]

√ √ √ √ √ √
AN = [ ] √ √ √ √ √ √

[√ √ √ ] [√ √ √ ]

√ √ √ √ √ √ √ √
[ ]
√ √ √ √ √ √ √ √

[√ √ √ ] [√ √ √ ] [√ √ ]

i.e., [ ]

The diagonal elements are the eigen values of A

 2. Diagonalize the matrix [ ]

Solution: Let A = [ ]

The characteristic equation is where

Therefore, the characteristic equation is i.e.,

√ √

Hence, the eigen values of A are 0, 3, 15

To find the eigen vectors:

[ ][ ] [ ]

Case 1: When [ ][ ] [ ]

---------- (1)

-------- (2)

--------- (3)

Solving (1) and (2) by rule of cross-multiplication,

-6 2 8 -6

7 -4 -6 7
 [ ]

Case 2: When [ ][ ] [ ]

-------- (1)

------- (2)

--------- (3)

Solving (1) and (2) by rule of cross-multiplication,

-6 2 5 -6

4 -4 -6 4

[ ]

Case 3: When [ ][ ] [ ]

---------- (1)

---------- (2)

---------- (3)

Solving (1) and (2) by rule of cross-multiplication,

-6 2 -7 -6

-8 -4 -6 -8
 [ ]

][ ]

][ ] 4-2-2 = 0

][ ]

Hence, the eigen vectors are orthogonal to each other

The Normalized matrix N = [ ]

[ ]

[ ]

[ ]

AN =

[ ] [ ] [ ][ ]

[ ] [ ] [ ]

[ ][ ] [ ]

[ ] [ ]

i.e., [ ]

The diagonal elements are the eigen values of A

118 Matrix Algebra

2.7 LU-Factorization15

The solution to a system Ax = b of linear equations can be solved quickly if A can be factored as A = LU
where L and U are of a particularly nice form. In this section we show that gaussian elimination can be
used to find such factorizations.

Triangular Matrices

 
As for square matrices, if A = ai j is an m × n matrix, the elements a11 , a22 , a33 , . . . form the main
diagonal of A. Then A is called upper triangular if every entry below and to the left of the main diagonal
is zero. Every row-echelon matrix is upper triangular, as are the matrices
 
    1 1 1
1 −1 0 3 0 2 1 0 5  0 −1 1 
 0 2 1 1   0 0 0 3 1    0

0 0 
0 0 −3 0 0 0 1 0 1
0 0 0
By analogy, a matrix A is called lower triangular if its transpose is upper triangular, that is if each entry
above and to the right of the main diagonal is zero. A matrix is called triangular if it is upper or lower
triangular.

Example 2.7.1
Solve the system
x1 + 2x2 − 3x3 − x4 + 5x5 = 3
5x3 + x4 + x5 = 8
2x5 = 6
where the coefficient matrix is upper triangular.

Solution. As in gaussian elimination, let the “non-leading” variables be parameters: x2 = s and

x4 = t. Then solve for x5 , x3 , and x1 in that order as follows. The last equation gives
6
x5 = 2 =3

Substitution into the second last equation gives

x3 = 1 − 15 t

Finally, substitution of both x5 and x3 into the first equation gives

x1 = −9 − 2s + 52 t

The method used in Example 2.7.1 is called back substitution because later variables are substituted
into earlier equations. It works because the coefficient matrix is upper triangular. Similarly, if the coeffi-
15 This section is not used later and so may be omitted with no loss of continuity.
 2.7. LU-Factorization 119

cient matrix is lower triangular the system can be solved by forward substitution where earlier variables
are substituted into later equations. As observed in Section 1.2, these procedures are more numerically
efficient than gaussian elimination.
Now consider a system Ax = b where A can be factored as A = LU where L is lower triangular and U
is upper triangular. Then the system Ax = b can be solved in two stages as follows:

1. First solve Ly = b for y by forward substitution.

2. Then solve U x = y for x by back substitution.

Then x is a solution to Ax = b because Ax = LU x = Ly = b. Moreover, every solution x arises this way

(take y = U x). Furthermore the method adapts easily for use in a computer.
This focuses attention on efficiently obtaining such factorizations A = LU . The following result will
be needed; the proof is straightforward and is left as Exercises 2.7.7 and 2.7.8.

Lemma 2.7.1
Let A and B denote matrices.

1. If A and B are both lower (upper) triangular, the same is true of AB.

2. If A is n × n and lower (upper) triangular, then A is invertible if and only if every main
diagonal entry is nonzero. In this case A−1 is also lower (upper) triangular.

LU-Factorization

Let A be an m × n matrix. Then A can be carried to a row-echelon matrix U (that is, upper triangular). As
in Section 2.5, the reduction is

A → E1 A → E2 E1 A → E3 E2 E1 A → · · · → Ek Ek−1 · · · E2 E1 A = U

where E1 , E2 , . . . , Ek are elementary matrices corresponding to the row operations used. Hence

A = LU

where L = (Ek Ek−1 · · · E2 E1 )−1 = E1−1 E2−1 · · · Ek−1

−1 −1
Ek . If we do not insist that U is reduced then, except
for row interchanges, none of these row operations involve adding a row to a row above it. Thus, if no
row interchanges are used, all the Ei are lower triangular, and so L is lower triangular (and invertible) by
Lemma 2.7.1. This proves the following theorem. For convenience, let us say that A can be lower reduced
if it can be carried to row-echelon form using no row interchanges.
120 Matrix Algebra

Theorem 2.7.1
If A can be lower reduced to a row-echelon matrix U , then

A = LU

where L is lower triangular and invertible and U is upper triangular and row-echelon.

Definition 2.14 LU-factorization

A factorization A = LU as in Theorem 2.7.1 is called an LU-factorization of A.

Such a factorization may not exist (Exercise 2.7.4) because A cannot be carried to row-echelon form
using no row interchange. A procedure for dealing with this situation will be outlined later. However, if
an LU-factorization A = LU does exist, then the gaussian algorithm gives U and also leads to a procedure
for finding L. Example 2.7.2 provides an illustration. For convenience, the first nonzero column from the
left in a matrix A is called the leading column of A.

Example 2.7.2
 
0 2 −6 −2 4
Find an LU-factorization of A =  0 −1 3 3 2 .
0 −1 3 7 10

Solution. We lower reduce A to row-echelon form as follows:

     
0 2 −6 −2 4 0 1 −3 −1 2 0 1 −3 −1 2
A =  0 −1 3 3 2 → 0 0 0 2 4 → 0 0 0 1 2  =U
0 −1 3 7 10 0 0 0 6 12 0 0 0 0 0

The circled columns are determined as follows: The first is the leading column of A, and is used
(by lower reduction) to create the first leading 1 and create zeros below it. This completes the work
on row 1, and we repeat the procedure on the matrix consisting of the remaining rows. Thus the
second circled column is the leading column of this smaller matrix, which we use to create the
second leading 1 and the zeros below it. As the remaining row is zero here, we are finished. Then
A = LU where  
2 0 0
L =  −1 2 0 
−1 6 1
This matrix L is obtained from I3 by replacing the bottom of the first two columns by the circled
columns in the reduction. Note that the rank of A is 2 here, and this is the number of circled
columns.

The calculation in Example 2.7.2 works in general. There is no need to calculate the elementary
 2.7. LU-Factorization 121

matrices Ei , and the method is suitable for use in a computer because the circled columns can be stored in
memory as they are created. The procedure can be formally stated as follows:

LU-Algorithm
Let A be an m × n matrix of rank r, and suppose that A can be lower reduced to a row-echelon
matrix U . Then A = LU where the lower triangular, invertible matrix L is constructed as follows:

1. If A = 0, take L = Im and U = 0.

2. If A 6= 0, write A1 = A and let c1 be the leading column of A1 . Use c1 to create the first
leading 1 and create zeros below it (using lower reduction). When this is completed, let A2
denote the matrix consisting of rows 2 to m of the matrix just created.

3. If A2 6= 0, let c2 be the leading column of A2 and repeat Step 2 on A2 to create A3 .

4. Continue in this way until U is reached, where all rows below the last leading 1 consist of
zeros. This will happen after r steps.

5. Create L by placing c1 , c2 , . . . , cr at the bottom of the first r columns of Im .

A proof of the LU-algorithm is given at the end of this section.

LU-factorization is particularly important if, as often happens in business and industry, a series of
equations Ax = B1 , Ax = B2 , . . . , Ax = Bk , must be solved, each with the same coefficient matrix A. It is
very efficient to solve the first system by gaussian elimination, simultaneously creating an LU-factorization
of A, and then using the factorization to solve the remaining systems by forward and back substitution.

Example 2.7.3
 
5 −5 10 0 5
 −3 3 2 2 1 
Find an LU-factorization for A = 
 −2
.
2 0 −1 0 
1 −1 10 2 5

Solution. The reduction to row-echelon form is

122 Matrix Algebra

   
5 −5 10 0 5 1 −1 2 0 1
 −3 3 2 2 1   2 4 
 → 0 0 8 
 −2 2 0 −1 0   0 0 4 −1 2 
1 −1 10 2 5 0 0 8 2 4
 
1 −1 2 0 1
 
 1 1 
 0 0 1 4 2

→



 0 0 0 −2 0 
 
0 0 0 0 0
 
1 −1 2 0 1
 
 1 1 
 0 0 1 4 2 

→ =U

 0 0 0 1 0 
 
0 0 0 0 0

If U denotes this row-echelon matrix, then A = LU , where

 
5 0 0 0
 −3 8 0 0 
L=  −2 4 −2 0


1 8 0 1

The next example deals with a case where no row of zeros is present in U (in fact, A is invertible).

Example 2.7.4
 
2 4 2
Find an LU-factorization for A =  1 1 2 .
−1 0 2

Solution. The reduction to row-echelon form is

       
2 4 2 1 2 1 1 2 1 1 2 1
 1 1 2  →  0 −1 1  →  0 1 −1  →  0 1 −1  = U
−1 0 2 0 2 3 0 0 5 0 0 1

 
2 0 0
Hence A = LU where L =  1 −1 0 .
−1 2 5
LU DECOMPOSITION METHOD

The Gauss Elimination Method has the disadvantage that all right-hand sides (i.e. all the
b vectors of interest for a given problem) must be known in advance for the elimination
step to proceed. The LU Decomposition Method outlined here has the property that the
matrix modification (or decomposition) step can be performed independent of the right
hand side vector. This feature is quite useful in practice - therefore, the LU
Decomposition Method is usually the Direct Scheme of choice in most applications.
To develop the basic method, let's break the coefficient matrix into a product of two
matrices,
A=LU (3.12)
where L is a lower triangular matrix and U is an upper triangular matrix.
Now, the original system of equations,
Ax =b (3.13)
becomes
LU x = b (3.14)
This expression can be broken into two problems,
Ly = b and Ux=b (3.15)
The rationale behind this approach is that the two systems given in eqn. (3.15) are both
easy to solve; one by forward substitution and the other by back substitution. In
particular, because L is a lower diagonal matrix, the expression, Ly = b , can be solved
with a simple forward substitution step. Similarly, since U has upper triangular form, U x
= b can be evaluated with a simple back substitution algorithm.
Thus the key to this method is the ability to find two matrices, L and U, that satisfy eqn.
(3.12). Doing this is referred to as the Decomposition Step and there are a variety of
algorithms available. Three specific approaches are as follows:
Doolittle Decomposition:

(3.16)
Because of the specific structure of the matrices, a systematic set of formulae for the
components of L and U results.
Crout Decomposition:

(3.17)
The evaluation of the components of L and U is done in a similar fashion as above.
Cholesky Factorization:
For symmetric, positive definite matrices, where
A = AT and x TA x > 0 for x ≠ 0 (3.18)
then,
U = LT and A = L LT (3.19)
and a simple set of expressions for the elements of L can be obtained (as above).
Once the elements of L and U are available (usually stored in a single NxN matrix), the
solution step for the unknown vector x is a simple process [as outlined above in eqn.
(3.15)].

A procedure for decomposing an matrix A into a product of a ower triangular

matrix L and an upper triangular matrix ,

Written explicitly for a matrix the decomposition is

(3.20)
This gives three types of equations

(3.21)
(3.22)
(3.23)

This gives equations for unknown (the decomposition is not unique), and can
be solved using either using Doolittle or Crout's method.

Doolittle Method : Here l ii = 1, i = 1 to N. In this case , equation (3.20) gives

u 1j = a 1j j = 1 to N

l i1 = a i1 / a 11 , i = 2 to N

u2 j = a2j – l21 . u1 j , j=2 to N

l i2 = ( a i2 – l i1 u 12 ) / u22 , i = 3 to N,

and so on

Crout’s Method : Here u ii = 1, i = 1 to N . In this case , we get

l i1 = a i1 , i=1 to N

u1j = a1j / a11 , j= 2 to N

l i2 = a i2 – l i1 u 12, i = 2 to N

u 2j = ( a 2j – l21 u 1j ) / l22, j= 3 to N,

and so on

Example 7: Given the following system of linear equations, determine the value of each
of the variables using the LU decomposition method.

6x1 - 2x2 = 14
9x1 - x2 + x3= 21 (3.24)
3x1 - 7x2 + 5x3= 9

Solution :
 Upper Lower
Explanation of Step
Triangular Triangular

<--- Beginning Matrix

Matrix Storing Elementary Row Operations --->
In order to force a value of 1 at position (1,1), we must
multiply row 1 by 1/6. Thus storing its reciprocal, 6, in
position (1,1) in the lower matrix.
Introducing zeros to positions (2,1) and (3,1) require
multiplications by -9 and -3 respectively. So we will store
the opposite of these numbers in their respective
locations.
On to the next position in the main diagonal, (2,2). To
replace the value in this position with a 1, multiply row 2
by 1/2, thus storing a 2 (the reciprocal) in position (2,2)
in the lower triangular matrix.
Replacing the position under the leading 1, position
(3,2), with a zero can be done with a multiplication of -8.
We will then store 8, the opposite of -8, in the lower
matrix at that position.
Only a multiplication of 1 is necessary to introduce a 1 to
the next diagonal position. In fact nothing is being done
to the upper triangular matrix, but we need the 1 in the
lower matrix to show that.

If a matrix A can be decomposed into an LU representation, then A is equal to the

product of the lower and upper triangular matrices. This can be shown with one matrix
multiplication.

= (3.25)

Solving Systems of Equations using the LU decomposition.

Systems of linear equations can be represented in a number of ways. In the Gauss-Jordan
elimination method, the system was represented as an augmented matrix. In this method,
we will represent the system as a matrix equation.

1. Rewrite the system Ax = b using the LU representation for A. Making the system
LUx = b.
 =

2. Define a new column matrix y so that Ux = y.

3. Rewrite step one with the substitution from step two yielding Ly = b.

4. Solve step three for y using forward substitution.

y1 = 7/3, y2 = 29/6, y3 = 33/2

5. Using the results from step four, solve for x in step two using back substitution.

x1=43/36, x2=-41/12, x3=33/2

 LU Decomposition
INGE4035 Numerical Methods Applied to Engineering
Dr. Marco A Arocha
October 2, 2014
Motivation
• The motivation for an LU decomposition is based on the observation
that systems of equations involving triangular coefficient matrices are
easier to deal with.
• Indeed, the whole point of Gaussian Elimination is to replace the
coefficient matrix with one that is triangular.
• The LU decomposition is another approach designed to exploit
triangular systems.
Introduction
• Matrices can be decomposed or factored into the product of two
other matrices in many number of ways.
• One way to decompose A is:
A = LU

• where L is a lower triangular and U an upper triangular matrices.

• Specifying the diagonal elements of either L or U makes factoring
unique.
• There are three methods: Doolittle, Crout, Cholesky
Introduction
• We assume that we can write
A = LU
• Our aim is to find L and U and once we have done so we have found
an LU decomposition of A.
• An LU decomposition of a matrix A is the product of a lower triangular
matrix and an upper triangular matrix that is equal to A. Off course
the multiplication process follows the rules of linear algebra.
There are three factorization methods:

• Crout Method:
• diag (U) = 1; uii=1
• Doolittle Method:
• diag (L) = 1; lii=1
• Choleski Method:
• diag (U) = diag (L); uii= lii
Doolittle
1 0 0 0
𝑥 1 0 0
L = x= any number; one’s in diagonal
𝑥 𝑥 1 0
𝑥 𝑥 𝑥 1
Crout
1 𝑥 𝑥 𝑥
0 1 𝑥 𝑥
U = x= any number; one’s in diagonal
0 0 1 𝑥
0 0 0 1
Cholesky
• Lii = Uii diagonal elements are equal
Example (Crout)

𝐿11 0 0 1 𝑈12 𝑈13

where 𝐿 = 𝐿21 𝐿22 0 𝑈= 0 1 𝑈23
𝐿31 𝐿32 𝐿33 0 0 1
Example (Crout)
Multiplying out LU and setting the answer equal to A gives

𝐿11 𝐿11 𝑈12 𝐿11 𝑈13 1 2 4

𝐿21 𝐿21 𝑈12 + 𝐿22 𝐿21 𝑈13 + 𝐿22 𝑈23 = 3 8 14
𝐿31 𝐿31 𝑈12 + 𝐿32 𝐿31 𝑈13 + 𝐿32 𝑈23 + 𝐿33 2 6 13

Matrix multiplication is reviewed at the end of the presentation

Example (Crout)
𝐿11 𝐿11 𝑈12 𝐿11 𝑈13 1 2 4
𝐿21 𝐿21 𝑈12 + 𝐿22 𝐿21 𝑈13 + 𝐿22 𝑈23 = 3 8 14
𝐿31 𝐿31 𝑈12 + 𝐿32 𝐿31 𝑈13 + 𝐿32 𝑈23 + 𝐿33 2 6 13

We begin by running along the first column:

𝐿11 = 1; 𝐿21 = 3; 𝐿31 = 2

For the second column:

𝐿11 𝑈12 = 2, then 𝑈12 = 2
𝐿21 𝑈12 + 𝐿22 = 8 then 𝐿22 = 2
𝐿31 𝑈12 + 𝐿32 = 6 then 𝐿32 = 2
For the third column:
𝐿11 𝑈13 = 4, then 𝑈13 = 4
𝐿21 𝑈13 + 𝐿22 𝑈23 = 14 then 𝑈23 = 1
𝐿31 𝑈13 + 𝐿32 𝑈23 + 𝐿33 = 13 then 𝐿33 = 3
Example (Crout)

1 0 0 1 2 4
where 𝐿 = 3 2 0 𝑈= 0 1 1
2 2 3 0 0 1
Example (Doolittle)
Example (Doolittle)

Matrix multiplication is reviewed at the end of the presentation

Example (Doolittle)
Example (Doolittle)
 function [L, U] = CroutLU(A)

matlab Crout LU [R, C] = size(A);

for i = 1:R
L(i, 1) = A(i, 1);
% Crout LU Decomposition U(i, i) = 1;
end
clc, clear for j = 2:R
A=[1,2,4; 3,8,14;2,6,13] U(1, j) = A(1, j) / L(1, 1);
end
[L, U] = CroutLU(A) for i = 2:R
for j = 2:i
L(i, j) = A(i, j) - L(i, 1:j - 1) * U(1:j - 1, j);
end

for j = i - 1:R
U(i, j) = (A(i, j) - L(i, 1:i - 1) * U(1:i - 1, j)) / L(i, i);
end
end
end

https://rosettacode.org/wiki/LU_decomposition#Creating_a_MATLAB_function
matlab Crout LU: output
 A=

matlab lu function 1
3
2
8 14
4

2 6 13
% LU Decomposition
% lu is a library matlab function
L=
clc, clear
A=[1,2,4; 3,8,14;2,6,13] 0.3333 -1.0000 1.0000
not a triangular
1.0000 0 0
matrix
[L,U]=lu(A) 0.6667 1.0000 0

U=

3.0000 8.0000 14.0000

0 0.6667 3.6667
0 0 3.0000
Using LU decomposition to solve systems of
equations
• Once a matrix A has been decomposed into lower and upper triangular
parts it is possible to obtain the solution to AX = B in a direct way.

• The procedure can be summarized as follows

• Given A, find L and U so that
• A = LU.
• AX=B, hence LUX = B.
• Let D = UX so that LD = B. Solve this triangular system for D .
• Finally solve the triangular system UX = D for X.

• The benefit of this approach is that we only ever need to solve triangular
systems. The cost is that we have to solve two of them.
Consider the following system of equation:
The key components can be expressed as:

coefficient matrix: right-hand side vector:

1st Step: Decomposing A matrix in LU yields:

Crout
The steps in finding the
solution vector with LU
Decomposition
2nd Step: Find D vector
Solve by {D}:

Expressing each equation and solving for {D}, i.e.,

d1, d2 and d3:

1d1=15

2d1-2d2+0d3=22

3d1-5d2+3.5d3=39
 1d1=15

2d1-2d2+0d3=22

3d1-5d2+3.5d3=39

which produces 𝑑1 =15, 𝑑2 =4, and 𝑑3 =4

𝑑1 15
𝐷 = 𝑑2 = 4
𝑑3 4
3rd Step: Find the solution vector x

Third step:

𝑈 𝑋 = 𝐷
𝑑1 15
𝐷 = 𝑑2 = 4 1 3 2 𝑥1 15
𝑑3 4
0 1 0.5 𝑥2 = 4
0 0 1 𝑥3 4
 𝑈 𝑋 = 𝐷

1 3 2 𝑥1 15
0 1 0.5 𝑥2 = 4
0 0 1 𝑥3 4

1𝑥3 = 4
1𝑥2 + 0.5𝑥3 = 4 which produces 𝑥3 = 4, 𝑥2 = 2 , 𝑥1 = 1
1𝑥1 + 3𝑥2 + 2𝑥3 = 15 𝑥1 1
𝑋 = 𝑥2 = 2
𝑥3 4
 A=
1 3 2
2 4 3

Crout LU-matlab B=
3 4 7

15
22
% Linear System Solution with LU 39
% Decomposition with CroutLU function
L=

clc, clear 1.0000 0 0

2.0000 -2.0000 0
A=[1,3,2; 2,4,3;3,4,7] 3.0000 -5.0000 3.5000
B=[15;22;39]
U=
1.0000 3.0000 2.0000
[L,U] = CroutLU(A) 0 1.0000 0.5000
0 0 1.0000
D = L\B
X = U\D D=
15
4
4

X=
1
2
4
 A=
1 3 2
2 4 3

lu function (matlab) B=
3 4 7

15
22
% Linear System Solution with LU 39
% Decomposition with lu matlab
% function L=
• 0.3333 1.0000 0
clc, clear 0.6667 0.8000 1.0000
1.0000 0 0
A=[1,3,2; 2,4,3;3,4,7]
B=[15;22;39] U=
3.0000 4.0000 7.0000
0 1.6667 -0.3333
[L,U] = lu(A) 0 0 -1.4000
D = L\B
D=
X = U\D
39.0000
2.0000
-5.6000

X=
1.0000
2.0000
4.0000
Matrix Multiplication Review
Matrix Multiplication
• Compute the dot
products of each row in
A with each column in B
• Each dot-prod result
becomes an element
with row equal to A and
column equal to B in A*B
the resulting matrix A B

m
AB  cij   aik bkj
k 1
( n x m) * ( m x p )  n x p

No commutative: AB≠BA 34
Matrix Multiplication
Math Syntax: AB
MATLAB Syntax: A*B (NO DOT)

>> A=[1 3 5; 2 4 6] >> B=[-2 4; 3 8; 12 -2] >> A*B

A= B= ans =
-2 4
1 3 5
3 8 67 18
2 4 6 12 -2 80 28

Sample calculation:
The dot product of row-1 of A and column-1 of B:
(1*-2)+(3*3)+(5*12)=67
35
Matrix Multiplication
Math Syntax: AB
MATLAB Syntax: A*B %(NO DOT)

>> A=[1 3 5; 2 4 6] >> B=[-2 4; 3 8; 12 -2] >> matriX(A,B)

A= B= ans =
-2 4
1 3 5
3 8 67 18
2 4 6 12 -2 80 28

Sample calculation:
The dot product of row-1 of A and column-1 of B:
(1*-2)+(3*3)+(5*12)=67
36
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

Singular Value Decomposition (SVD) —

Working Example
Roshan Joe Vincent · Follow
Published in Intuition
6 min read · Jul 29, 2021

Listen Share

Recently, I started looking into recommender systems and collaborative filtering in

particular in which the input matrix of users-ratings is broken down into 3 matrices
of user-features, features-features and item-features matrices using a technique
called Singular Value Decomposition (SVD). Before, I applied this, I wanted to get an
intuitive understanding of the math behind and thus started my week-long journey
in the world of matrix decomposition. Well, I was not one of those math heads at
school so I ended up lost in a lot of the tutorials for SVD because they missed some
of the steps like computing row-echelon form, null space of a matrix. etc. So, I set out
learning and diving deeper into how SVD actually works and how these three
matrices are formed a.k.a how is a given input matrix decomposed to these three
matrices. In this story, I will be working through an example of SVD and breakdown
the entire process mathematically. So, let’s go!

According to the formula for SVD,

SVD Formula

1. A is the input matrix

https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 1/24
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

2. U are the left singular vectors,

3. sigma are the diagonal/eigenvalues

4. V are the right singular vectors.

The shape of these three matrices will be

1. A — m x n matrix

2. U — m x k matrix

3. Sigma — k x k matrix

4. V — n x k matrix

Step 1
So, as the first step, we need to find eigenvalues (watch the video provided below to get
an understanding of eigenvalues and eigenvectors) of matrix A and as A can be a
rectangular matrix, we need to convert it to a square matrix by multiplying A with
its transpose. Here, for easier computation I have taken A as a 2 x 2 matrix.

Step 2
https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 2/24
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

Now, that we have a square matrix, we can calculate the eigenvalues of A(transpose)
A. We, can do so by calculating the determinant of A(transpose)A — (lambda)I
where lambda are the two eigenvalues.

Solving the equation, we get

https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 3/24
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

Eigenvalues

Step 3
Once we have calculated the eigenvalues, it’s time to calculate the two eigenvectors
for each eigenvalue. So, let’s start by calculating the eigenvector for 10.

Step 3.1

We plug the value of lambda in the A(transpose)A — (lambda)I matrix.

https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 4/24
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

In order to find the eigenvector, we need to find the null space of a matrix where AB
= 0. In other words,

Null Space Formula

Next, we need to reduce this matrix to the Row-Echelon Form so that we can easily
solve the equation. Let’s talk about Row-Echelon for a moment here.

Row-Echelon Form

A matrix is said to be in row-echelon form if the following rules are satisfied.

1. All the leading entries in each row of the matrix is 1

2. If a column contains a leading entry then all the entries below the leading entry
should be zero

https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 5/24
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

3. If any two consecutive non-zero rows, the leading entry in the upper row should
occur to the left of the leading entry in the lower row.

4. All rows which consist only of zeros should occur in the bottom of the matrix

We need to perform some operations on the rows to reduce the matrix. These
operations are called elementary row operations and there are a certain rules to
follow for these operations as given below,

https://en.wikibooks.org/wiki/Linear_Algebra/Row_Reduction_and_Echelon_Forms

Armed with the above rules, lets start reducing the matrix in Step 3.1 to row-
echelon form.

Row-Echelon Form for matrix mentioned in Step 3.1

https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 6/24
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

Now, we can solve for null space as below to find the eigenvector for eigenvalue 10

Once we get this vector, we need to convert it to a unit vector . The way we do that is
by taking the columnar values and dividing them by taking the square root of the
sum of squares of the values. So, in this case we do the following,

Denominator

https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 7/24
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

So the final eigenvector for eigenvalue is

We do the similar steps to get the eigenvector for eigenvalue 40

Matrix for computation

https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 8/24
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

Row-Echelon Form
Open in app Sign up Sign In

https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 9/24
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

EigenVector for 40

Now that we have got both the eigenvectors, let’s put it together.

https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 10/24
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

Note that the diagonal values in sigma are always in the descending order and so the
vectors are also placed in that corresponding order. If you are familiar with PCA, the
principal components correspond to the top k diagonal element which captures the
most variance. The higher the value, the more important the component is and the
more variance they describe.

Step 4
Now that we have our V and Sigma matrices, now it’s time to find U. We can just
multiply the equation by sigma(inverse) and V on both sides to get the equation for
U. In this case, as V is an orthogonal matrix, the transpose and inverse of V are the
same, therefore, V(transpose) multiplied by V becomes an identity matrix. Same
goes, for the diagonal matrix as well.

Note: On the left hand side, it is sigma(inverse) and not transpose as mentioned in the slide
below

https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 11/24
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

Computing A x V

Next up, we need to convert this to unit vectors using the steps described above.

https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 12/24
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

AV as unit vectors

Next up we multiply this matrix with Sigma (transpose) which is sigma in itself
because its a diagonal matrix.

https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 13/24
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

AV x sigma (transpose)

Now, we need to convert this to unit vectors to get the final U matrix.

https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 14/24
9/14/23, 5:07 PM Singular Value Decomposition (SVD) — Working Example | by Roshan Joe Vincent | Intuition | Medium

So, there you go, we have calculated U, sigma and V and decomposed the matrix A
into three matrices as given below.

SVD

You can verify this by going to this nice little tool and performing the matrix
multiplications.
https://medium.com/intuition/singular-value-decomposition-svd-working-example-c2b6135673b5 15/24
 SVD computation example
 
T 3 2 2
Example: Find the SVD of A, U ΣV , where A = .
2 3 −2

First we compute the singular values σi by finding the eigenvalues of AAT .

 
T 17 8
AA = .
8 17
T
The characteristic polynomial √is det(AA − λI)√ = λ2 − 34λ + 225 = (λ − 25)(λ − 9), so
the singular values are σ1 = 25 = 5 and σ2 = 9 = 3.
Now we find the right singular vectors (the columns of V ) by finding an orthonormal
set of eigenvectors of AT A. It is also possible to proceed by finding the left singular
vectors (columns of U ) instead. The eigenvalues of AT A are 25, 9, and 0, and since
AT A is symmetric we know that the eigenvectors will be orthogonal.
For λ = 25, we have
 
−12 12 2
AT A − 25I =  12 −12 −2 
2 −2 −17
 
1 −1 0
which row-reduces to  0 0 1 . A unit-length vector in the kernel of that matrix
0 0 0
 √ 
1/√2
is v1 =  1/ 2 .
0    
4 12 2 1 0 − 14
1 
For λ = 9 we have AT A−9I =  12 4 −2  which row-reduces to  0 1 4
.
2 −2 −1 0 0 0
 √ 
1/ √18
A unit-length vector in the kernel is v2 =  −1/√ 18 .
4/ 18
T
For the last eigenvector, we could compute the kernel of A A or find a unit vector
a
perpendicular to v1 and v2 . To be perpendicular to v1 =  b  we need −a = b.
c
T
√ √
Thenthe condition
 that v2 v3 = 0 becomes 2a/ 18 + 4c/ 18 = 0 or  −a =  2c. So
a 2/3
v3 =  −a  and for it to be unit-length we need a = 2/3 so v3 =  −2/3 .
−a/2 −1/3
1
2

So at this point we know that

 √ √ 
  1/√ 2 1/√ 2 √ 0
5 0 0 
A = U ΣV T = U 1/ 18 −1/ 18 4/ 18  .
0 3 0
2/3 −2/3 −1/3
Finally, we can compute U by the formula σui = Avi , or ui = σ1 Avi . This gives
 √ √ 
1/√2 1/ √2
U= . So in its full glory the SVD is:
1/ 2 −1/ 2
√ √
 √ √ 
 
 1/√ 2 1/√ 2 √ 0
1/√2 1/ √2 5 0 0 
A = U ΣV T = 1/ 18 −1/ 18 4/ 18  .
1/ 2 −1/ 2 0 3 0
2/3 −2/3 −1/3
Section 7.4 Notes (The SVD)
The Singular Value Decomposition: Let A be any m × n matrix. Then there are
orthogonal matrices U , V and a diagonal matrix Σ such that

A = U ΣV T

Specifically:

• The ordering of the vectors comes from the ordering of the singular values (largest to
smallest).

• The columns of U are the eigenvectors of AAT

• The columns of V are the eigenvectors of AT A.

√
• The diagonal elements of Σ are the singular values, σi = λi

• There are relationships between vi and ui (with normalization):

Avi = σi ui AT ui = σi vi

The scaling factor comes from kAvi k = σi = kAT ui k.

• If the matrix A is already a symmetric matrix, then U = V and we get the decompo-
sition from 7.1.

Numerical Example
 
−3 1
(Exercise 11) Find the SVD of the matrix A =  6 −2 
  6 −2
81 −27
First, we’ll work with AT A = . The eigenvalues are λ = 0, 90.
−27
 9   
1 −1/3 1 1
For λ = 0, the reduced matrix is , so v = √10
0 0  3
1 3 −3
For λ = 90, the reduced matrix is , so v = √110
0 0 1
Now, we can find the reduced SVD right away since
 
1
1 1
u1 = Av1 =  −2 
σ1 3
−2

1
We now need a basis for the null space of
       
10 −20 −20 1 −2 −2  2 2 
AAT =  −20 40 40  ∼  0 0 0  ⇒  1 , 0 
−40 40 40 0 0 0 0 1
 

Now the full SVD is given by:

√ √ 
1/3 2/√5 2/ 5  √ √ √ T
  
−3 1 
3 10 0 −3/√ 10 1/√10
A =  6 −2  =  −2/3 1/ 5 √0

0 0 1/ 10 3/ 10
6 −2 −2/3 0 1/ 5

The SVD and the Four Subspaces

• If the columns of V are ordered by the magnitude of the singular values (largest to
smallest), and the rank of A is r, then

Null(A) = span {vr+1 , vr+2 , . . . vn }

is an orthonormal basis for the null space of AT A (and thus also of A)- Note that the
eigenspace E0 for AT A is the null space of A.

• The row space of A therefore has the basis:

Row(A) = span {v1 , v2 , . . . vr }

• The column space of A is therefore r dimensional:

Col(A) = span {u1 , u2 , . . . ur }

• The null space of AT is given by:

Null(AT ) = span {ur+1 , ur+2 , . . . um }

The Reduced SVD

Again, let A be m × n with rank r.
If we’re not interested in getting the null spaces, we can get what’s called the reduced
SVD:
A = Û Σ̂V̂ T
where Û is m × r with o.n. columns (ordered by the singular values), Σ̂ is r × r with non-zero
values along the diagonal, and where V̂ is n × r with o.n. columns.
When there is no room for confusion, the hats are sometimes left off.

2
Singular Value
Decomposition
Motivatation
• The diagonalization theorem play a part in many interesting
applications.
• Unfortunately not all matrices can be factored as 𝐴 = 𝑃𝐷𝑃−1
• However a factorization 𝐴 = 𝑄𝐷𝑃−1 is possible for any 𝑚 × 𝑛 matrix
A.
• A special factorization of this type, called singular value
decomposition, is one of the most useful matrix factorizations in
applied linear algebra.
• As we will see one application is finding minimum value of Ax for
solving Ax=0.
 Singular Value Decomposition
• Let 𝐴 be a 𝑚 × 𝑛 matrix of rank 𝑟. We cannot find eigenvalues and eigenvectors
for non-square matrices.
• However, 𝐴 can be diagonalized in such a way that
𝐴𝒗1 = 𝜎1 𝒖1
𝐴𝒗2 = 𝜎2 𝒖2
⋮
𝐴𝒗𝑟 = 𝜎𝑟 𝒖𝑟
• The singular vectors 𝒗1 … 𝒗𝑟 are orthogonal and are the basis for the row space
of 𝐴.
• The output vectors 𝒖1 … 𝒖𝑟 are orthogonal and in column space of 𝐴.
• The singular values 𝜎1 … 𝜎𝑟 are all positive numbers in non-increasing order.

3
Singular value decomposition
• 𝐴𝒗𝑖 = 𝜎𝑖 𝒖𝑖 leads to 𝐴𝑉 = 𝑈Σ:
𝜎1
𝐴 𝒗1 … 𝒗𝑟 = 𝒖1 … 𝒖𝑟 ⋱
𝜎𝑟
• Note that in above equation the dimensions are
𝑚 × 𝑛 𝑛 × 𝑟 = (𝑚 × 𝑟)(𝑟 × 𝑟)
• We always can find orthogonal basis for 𝑁𝑢𝑙(𝐴) and 𝑁𝑢𝑙(𝐴𝑇 ) and
augment those vectors in matrices 𝑈 and 𝑉, respectively to make them
into square orthogonal matrices. We can fill zeroes in the rest of diagonal
elements in matrix Σ. ( row space is orthogonal to null space)
• Then the matrices dimension in the equation 𝐴𝑉 = 𝑈Σ become:
𝑚 × 𝑛 𝑛 × 𝑛 = (𝑚 × 𝑚)(𝑚 × 𝑛)
4
Singular value decomposition
• After adding null space vectors the equation 𝐴𝑉 = 𝑈Σ becomes:
𝜎1
• 𝐴 𝒗1 … 𝒗𝑚 = 𝒖1 … 𝒖𝑛 ⋱
𝜎𝑟
0
• How to find 𝑈 and 𝑉?
• 𝐴𝑉 = 𝑈Σ → 𝐴 = 𝑈Σ𝑉 −1 = 𝑈Σ𝑉 𝑇 , since 𝑉 is orthogonal.
• 𝐴𝐴𝑇 = 𝑈Σ𝑉 𝑇 (𝑈Σ𝑉 𝑇 )𝑇 = 𝑈Σ𝑉 𝑇 𝑉Σ 𝑇 𝑈 𝑇 = 𝑈Σ 2 𝑈 𝑇 , since 𝑉 𝑇 𝑉 = 𝐼 and
ΣΣ 𝑇 = Σ 2
• 𝐴𝑇 𝐴 = (𝑈Σ𝑉 𝑇 )𝑇 𝑈Σ𝑉 𝑇 = 𝑉Σ 𝑇 𝑈 𝑇 𝑈Σ𝑉 𝑇 = 𝑉Σ 2 𝑉 𝑇 , since 𝑈 𝑇 𝑈 = 𝐼 and
ΣΣ 𝑇 = Σ 2
5
Singular value decomposition
• 𝐴 = 𝑈Σ𝑉 𝑇
• 𝐴𝐴𝑇 = 𝑈Σ 2 𝑈 𝑇
• 𝐴𝑇 𝐴 = 𝑉Σ 2 𝑉 𝑇
• Recall that 𝐴𝐴𝑇 and 𝐴𝑇 𝐴 are symmetric matrices where their sizes are
𝑚 × 𝑚 and 𝑛 × 𝑛, respectively.
• Recall that symmetric matrices are diagonalizable and their
eigenvector matrix are orthogonal.
• The eigenvalues of 𝐴𝐴𝑇 are the same as eigenvalues of 𝐴𝑇 𝐴 since the
eigenvalues of 𝐴𝐵 are the same as eigenvalues of 𝐵𝐴.
• In other words, Σ 2 are the eigenvalues and columns of 𝑉 are the
eigenvectors of 𝐴𝑇 𝐴, and 𝑈are the eigenvectors of 𝐴𝐴𝑇
6
Singular value decomposition
3 1 1
• Example: Find matrix 𝐴 = to its equivalent singular
−1 3 1
vector decomposition.

• Find 𝐴𝐴𝑇 = 𝑈𝐷𝑈 𝑇 ,

3 −1
3 1 1 11 1
• 𝐴𝐴𝑇 = 1 3 =
−1 3 1 1 11
1 1
11 − 𝜆 1
• = 0, 11 − 𝜆 2 − 1 = 0 → 𝜆 − 10 𝜆 − 12 = 0
1 11 − 𝜆

• 𝜆1 = 12, 𝜆2 = 10.

7
Singular value decomposition
• Example Cont’d:

11 − 12 1 −1 1 1
• 𝒖1 = 𝟎 → 𝒖1 = 𝟎 → 𝒖1 =
1 11 − 12 1 −1 1

11 − 10 1 1 1 1
• 𝒖2 = 𝟎 → 𝒖2 = 𝟎 → 𝒖2 =
1 11 − 10 1 1 −1
1 1
2 2 12 0
• 𝑈= 1 1 ,𝐷 =
− 0 10
2 2

8
Singular value decomposition
• Example Cont’d:

• Find 𝐴𝑇 𝐴 = 𝑉𝐷𝑉 𝑇 ,

3 −1 10 0 −2
3 1 1
• 𝐴𝐴𝑇 = 1 3 = 0 10 4
−1 3 1
1 1 −2 4 2
10 − 𝜆 0 −2
• 0 10 − 𝜆 4 = 0, → 𝜆 𝜆 − 10 𝜆 − 12 = 0
−2 4 2−𝜆

• 𝜆1 = 12, 𝜆2 = 10, 𝜆3 = 0.
9
Singular value decomposition
• Example Cont’d:

10 − 12 0 −2 −2 0 −2 1
• 0 10 − 12 4 𝒗1 = 𝟎 → 0 −2 4 𝒗1 = 𝟎 → 𝒗1 = −2
−2 4 2 − 12 −2 4 −10 −1

10 − 10 0 −2 0 0 −2 2
• 0 10 − 10 4 𝒗2 = 𝟎 → 0 0 4 𝒗2 = 𝟎 → 𝒗2 = −1
−2 4 2 − 10 −2 4 −8 0

10 − 0 0 −2 10 0 −2 1
• 0 10 − 0 4 𝒗3 = 𝟎 → 0 10 4 𝒗3 = 𝟎 → 𝒗3 = −2
−2 4 2−0 −2 4 2 5
10
Singular value decomposition
• Example Cont’d:

1 2 1
6 5 30
2 1 2
12 0 0
• 𝑉= −
6
−
5
−
30
,𝐷 = 0 10 0
1 1 5 0 0 0
−
6 5 30 1 2 1
1 1
− −
6 6 6
3 1 1 2 2 12 0 0 2 1 1
• 𝐴= 𝑈Σ𝑉 𝑇 → = 1 1
−
−1 3 1 − 0 10 0 5 5 5
2 2 1 2 5
−
30 30 30

11
Geometric interpretation
• Here is a geometric interpretation
of SVD for a 2 × 2 matrix 𝑀.
• 𝑉 𝑇 (in figure 𝑉 ∗ ) rotates unit
vectors.
• Σ scales the vectors
• 𝑈 perform the final rotation.

12
Singular value decomposition
• Note that we can assume SVD of matrix 𝐴 as:
𝑟

𝐴= 𝜎𝑖 𝒖𝑖 𝒗𝑖 𝑇
𝑖=1
Where 𝑟 is the rank of the matrix.
• Size of each 𝜎𝑖 𝒖𝑖 𝒗𝑖 𝑇 is 𝑚 × 𝑛. The greater 𝜎𝑖 the greater values
added to reconstruct matrix 𝐴.

𝑇 𝑇
𝐴 = 𝜎1 𝒖1 𝒗1 + … + 𝜎𝑟 𝒖𝑟 𝒗𝑟
13
Solve Ax = 0
• What minimizes Ax. And why?
• As defined previous we can write A as
𝜎1
𝒗1
𝐴 = 𝑈Σ𝑉 𝑇 = 𝒖1 … 𝒖𝑛 ⋱
⋮
𝜎𝑟
𝒗𝑟
0
Where 𝜎𝑟 is the smallest eigenvalue. Now
𝑟
we have
Ax = 𝜎𝑖 𝒖𝑖 𝒗𝑖 𝑇 x
𝑖=1
If we choose 𝑥 = 𝒗𝑟 . Then since all 𝑣𝑖𝑇 𝑣𝑟 = 0 𝑖 ≠ 𝑟 we have
𝐴𝑥 = 𝐴𝑣𝑟 = 𝜎𝑟 𝒖𝑟 𝒗𝑟 𝑇 𝑣𝑟 = 𝜎𝑟 𝒖𝑟
Smallest value of Ax associate with 𝑥 = 𝒗𝑟 . And if 𝜎𝑟 = 0 then 𝒗𝑟 is an answer of
Ax=0
Pseudo inverse
• Assume 𝐴𝑥 = 𝑏, Then by singular value decomposition of A we have
𝐴𝑥 = 𝑏 ⇒ 𝑈Σ𝑉 𝑇 𝑥 = 𝑏
Σ𝑉 𝑇 𝑥 = 𝑈 𝑇 𝑏
𝑉 𝑇 𝑥 = Σ∗𝑈𝑇 𝑏
𝑥 = 𝑉Σ ∗ 𝑈 𝑇 𝑏
• 𝑉Σ ∗ 𝑈 𝑇 called the pseudo inverse of A . It is useful for finding inverse
of non-square matrices.
Applications
• One of the applications of SVD is dimensionality reduction.
• A 𝑚 × 𝑛 matrix can be thought of gray level of a digital image.
• If the image 𝐴 is decomposed to its singular values and vectors, we
can pick only the most significant 𝒖𝑖 ’s, 𝜎𝑖 ’s and 𝒗𝑖 ’s.
• By doing this we can compress the information of the image.
• Suppose the image 𝐴 is 𝑚 × 𝑛.
•𝐴= 𝐾 𝜎 𝒖
𝑖=1 𝑖 𝑖 𝑖𝒗 𝑇

• In next slide you will see the original image and its compressed up to
K most significant singular values.
16
Image compression using SVD
K=8 K=32

K=128 K=512

Original
17
550 CHAPTER 10 NUMERICAL METHODS

10.3 POWER METHOD FOR APPROXIMATING EIGENVALUES

In Chapter 7 we saw that the eigenvalues of an n 3 n matrix A are obtained by solving its
characteristic equation
ln 1 cn21ln21 1 cn22ln22 1 . . . 1 c0 5 0.
For large values of n, polynomial equations like this one are difficult and time-consuming
to solve. Moreover, numerical techniques for approximating roots of polynomial equations
of high degree are sensitive to rounding errors. In this section we look at an alternative
method for approximating eigenvalues. As presented here, the method can be used only to
find the eigenvalue of A that is largest in absolute value—we call this eigenvalue the
dominant eigenvalue of A. Although this restriction may seem severe, dominant eigenval-
ues are of primary interest in many physical applications.

Definition of Dominant Let l1, l2, . . . , and ln be the eigenvalues of an n 3 n matrix A. l1 is called the
Eigenvalue and dominant eigenvalue of A if

Dominant Eigenvector |l | > |l |,

1 i i 5 2, . . . , n.
The eigenvectors corresponding to l1 are called dominant eigenvectors of A.

Not every matrix has a dominant eigenvalue. For instance, the matrix

30 4
1 0
A5
21
(with eigenvalues of l1 5 1 and l2 5 21) has no dominant eigenvalue. Similarly, the
matrix

3 4
2 0 0
A5 0 2 0
0 0 1
(with eigenvalues of l1 5 2, l2 5 2, and l3 5 1) has no dominant eigenvalue.

EXAMPLE 1 Finding a Dominant Eigenvalue

Find the dominant eigenvalue and corresponding eigenvectors of the matrix

2 212
A5 31 25
. 4
Solution From Example 4 of Section 7.1 we know that the characteristic polynomial of A is
l2 1 3l 1 2 5 (l 1 1)(l 1 2). Therefore the eigenvalues of A are l1 5 21 and
l2 5 22, of which the dominant one is l2 5 22. From the same example we know that
the dominant eigenvectors of A (those corresponding to l2 5 22) are of the form

3 14,
3
x5t t Þ 0.
 SECTION 10.3 POWER METHOD FOR APPROXIMATING EIGENVALUES 551

The Power Method

Like the Jacobi and Gauss-Seidel methods, the power method for approximating eigenval-
ues is iterative. First we assume that the matrix A has a dominant eigenvalue with corre-
sponding dominant eigenvectors. Then we choose an initial approximation x0 of one of the
dominant eigenvectors of A. This initial approximation must be a nonzero vector in Rn.
Finally we form the sequence given by
x1 5 Ax0
x2 5 Ax1 5 A(Ax0) 5 A2x0
x3 5 Ax2 5 A(A2x0) 5 A3x0

...
xk 5 Axk21 5 A(Ak21x0) 5 Akx0.
For large powers of k, and by properly scaling this sequence, we will see that we obtain
a good approximation of the dominant eigenvector of A. This procedure is illustrated in
Example 2.

EXAMPLE 2 Approximating a Dominant Eigenvector by the Power Method

Complete six iterations of the power method to approximate a dominant eigenvector of

2 212
A5 31 25
. 4
Solution We begin with an initial nonzero approximation of

34
1
x0 5 .
1
We then obtain the following approximations.
Iteration Approximation
2 212 1 210
31 43 4 3 4 31.004
2.50
x1 5 Ax0 5 5 24
25 1 24
2 212 210
31 43 4 3 4 31.004
28 2.80
x2 5 Ax1 5 5 10
25 24 10
2 212 28 264
31 2543104 5 32224 31.004
2.91
x3 5 Ax2 5 222

2 212 264
53
1 25432224 3 464
463
1.004
136 2.96
x4 5 Ax3 5

2 212 136 2280

53
1 2543 464 3 2944
2943
1.004
2.98
x5 5 Ax4 5

2 212 2280
53 43 4 53 4 1903
1.004
568 2.99
x6 5 Ax5
1 25 294 190
552 CHAPTER 10 NUMERICAL METHODS

Note that the approximations in Example 2 appear to be approaching scalar multiples of

314,
3

which we know from Example 1 is a dominant eigenvector of the matrix

2 212
A5 31 25
. 4
In Example 2 the power method was used to approximate a dominant eigenvector of the
matrix A. In that example we already knew that the dominant eigenvalue of A was l 5 22.
For the sake of demonstration, however, let us assume that we do not know the dominant
eigenvalue of A. The following theorem provides a formula for determining the eigenvalue
corresponding to a given eigenvector. This theorem is credited to the English physicist John
William Rayleigh (1842–1919).

Theorem 10.2 If x is an eigenvector of a matrix A, then its corresponding eigenvalue is given by

Ax ? x
Determining an Eigenvalue l5 .
x?x
from an Eigenvector
This quotient is called the Rayleigh quotient.

Proof Since x is an eigenvector of A, we know that Ax 5 lx, and we can write

Ax ? x lx ? x l(x ? x)
5 5 5 l.
x?x x?x x?x

In cases for which the power method generates a good approximation of a dominant
eigenvector, the Rayleigh quotient provides a correspondingly good approximation of the
dominant eigenvalue. The use of the Rayleigh quotient is demonstrated in Example 3.

EXAMPLE 3 Approximating a Dominant Eigenvalue

Use the result of Example 2 to approximate the dominant eigenvalue of the matrix
2 212
A5 31 25
. 4
Solution After the sixth iteration of the power method in Example 2, we had obtained.

3 4 3 4
568 2.99
x6 5 < 190 .
190 1.00

With x 5 (2.99, 1) as our approximation of a dominant eigenvector of A, we use the

Rayleigh quotient to obtain an approximation of the dominant eigenvalue of A. First we
compute the product Ax.
 SECTION 10.3 POWER METHOD FOR APPROXIMATING EIGENVALUES 553

2 212 26.02
31 4 31.004 5 322.014
2.99
Ax 5
25
Then, since
Ax ? x 5 (26.02)(2.99) 1 (22.01)(1) < 220.0
and
x ? x 5 (2.99)(2.99) 1 (1)(1) < 9.94,
we compute the Rayleigh quotient to be
Ax ? x 220.0
l5 < < 22.01,
x?x 9.94
which is a good approximation of the dominant eigenvalue l 5 22.

From Example 2 we can see that the power method tends to produce approximations
with large entries. In practice it is best to “scale down” each approximation before pro-
ceeding to the next iteration. One way to accomplish this scaling is to determine the com-
ponent of Axi that has the largest absolute value and multiply the vector Axi by the
reciprocal of this component. The resulting vector will then have components whose
absolute values are less than or equal to 1. (Other scaling techniques are possible. For
examples, see Exercises 27 and 28.

EXAMPLE 4 The Power Method with Scaling

Calculate seven iterations of the power method with scaling to approximate a dominant
eigenvector of the matrix

3 4
1 2 0
A 5 22 1 2 .
1 3 1
Use x0 5 (1, 1, 1) as the initial approximation.

Solution One iteration of the power method produces

3 43 4 3 4
1 2 0 1 3
Ax0 5 22 1 2 1 5 1 ,
1 3 1 1 5
and by scaling we obtain the approximation

34 3 4
3 0.60
x1 5 15 1 5 0.20 .
5 1.00
554 CHAPTER 10 NUMERICAL METHODS

A second iteration yields

3 43 4 3 4
1 2 0 0.60 1.00
Ax1 5 22 1 2 0.20 5 1.00
1 3 1 1.00 2.20
and

3 4 3 4
1.00 0.45
1
x2 5 1.00 5 0.45 .
2.20
2.20 1.00
Continuing this process, we obtain the sequence of approximations shown in Table 10.6.
TABLE 10.6

x0 x1 x2 x3 x4 x5 x6 x7

3 4 3 4 3 4 3 4 3 4 3 4 3 4 3 4
1.00 0.60 0.45 0.48 0.51 0.50 0.50 0.50
1.00 0.20 0.45 0.55 0.51 0.49 0.50 0.50
1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00

From Table 10.6 we approximate a dominant eigenvector of A to be

3 4
0.50
x 5 0.50 .
1.00
Using the Rayleigh quotient, we approximate the dominant eigenvalue of A to be l 5 3.
(For this example you can check that the approximations of x and l are exact.)

REMARK: Note that the scaling factors used to obtain the vectors in Table 10.6,
x1 x2 x3 x4 x5 x6 x7
↓ ↓ ↓ ↓ ↓ ↓ ↓

5.00 2.20 2.82 3.13 3.02 2.99 3.00,

are approaching the dominant eigenvalue l 5 3.

In Example 4 the power method with scaling converges to a dominant eigenvector. The
following theorem tells us that a sufficient condition for convergence of the power method
is that the matrix A be diagonalizable (and have a dominant eigenvalue).

Theorem 10.3 If A is an n 3 n diagonalizable matrix with a dominant eigenvalue, then there exists a
nonzero vector x0 such that the sequence of vectors given by
Convergence of the
Ax0, A2x0, A3x0, A4x0, . . . , Akx0, . . .
Power Method
approaches a multiple of the dominant eigenvector of A.
 SECTION 10.3 POWER METHOD FOR APPROXIMATING EIGENVALUES 555

Proof Since A is diagonalizable, we know from Theorem 7.5 that it has n linearly independent
eigenvectors x1, x2, . . . , xn with corresponding eigenvalues of l1, l2, . . . , ln. We
assume that these eigenvalues are ordered so that l1 is the dominant eigenvalue (with a cor-
responding eigenvector of x1). Because the n eigenvectors x1, x2, . . . , xn are linearly
independent, they must form a basis for Rn. For the initial approximation x0, we choose a
nonzero vector such that the linear combination
x0 5 c1x1 1 c2x2 1 . . . 1 cnxn
has nonzero leading coefficients. (If c1 5 0, the power method may not converge, and a dif-
ferent x0 must be used as the initial approximation. See Exercises 21 and 22.) Now, multi-
plying both sides of this equation by A produces

Ax0 5 A(c1x1 1 c2x2 1 . . . 1 cnxn)

5 c1(Ax1) 1 c2(Ax2) 1 . . . 1 cn(Axn)
5 c1(l1x1) 1 c2(l2x2) 1 . . . 1 cn(lnxn).
Repeated multiplication of both sides of this equation by A produces
Akx0 5 c1(l1kx1) 1 c2(l2kx2) 1 . . . 1 cn(lnkxn),
which implies that

l2 ln
3 1l 2 x 1l 2 x 4.
k k
Akx0 5 l1k c1x1 1 c2 2 1 . . . 1 cn n
1 1

Now, from our original assumption that l1 is larger in absolute value than the other eigen-
values it follows that each of the fractions
l2 l3 ln
, , ...,
l1 l1 l1
is less than 1 in absolute value. Therefore each of the factors
l2 k l3 k ln
1 2 1 2 1 2
k
, , ...,
l1 l1 l1
must approach 0 as k approaches infinity. This implies that the approximation
Akx0 < l1kc1x1, c1 Þ 0

improves as k increases. Since x1 is a dominant eigenvector, it follows that any scalar

multiple of x1 is also a dominant eigenvector. Thus we have shown that Akx0 approaches a
multiple of the dominant eigenvector of A.

The proof of Theorem 10.3 provides some insight into the rate of convergence of the
power method. That is, if the eigenvalues of A are ordered so that

|l | . |l | $ |l | $ . . . $ |l |,
1 2 3 n
556 CHAPTER 10 NUMERICAL METHODS

then the power method will converge quickly if l 2 y l 1 is small, and slowly if
| | | |
l 2 y l 1 is close to 1. This principle is illustrated in Example 5.
| | | |
EXAMPLE 5 The Rate of Convergence of the Power Method

(a) The matrix

36 4
4 5
A5
5
has eigenvalues of l1 5 10 and l2 5 21. Thus the ratio l 2 y l 1 is 0.1. For this
| | | |
matrix, only four iterations are required to obtain successive approximations that agree
when rounded to three significant digits. (See Table 10.7.)
TABLE 10.7

x0 x1 x2 x3 x4

31.0004 31.0004 31.0004 31.0004 31.0004

1.000 0.818 0.835 0.833 0.833

(b) The matrix

24
3 4
10
A5
7 5
has eigenvalues of l1 5 10 and l2 5 29. For this matrix, the ratio l 2 y l 1 is 0.9,
| | | |
and the power method does not produce successive approximations that agree to three
significant digits until sixty-eight iterations have been performed, as shown in Table 10.8.
TABLE 10.8

x0 x1 x2 x66 x67 x68

...
31.0004 31.0004 31.0004 31.0004 31.0004 31.0004
1.000 0.500 0.941 0.715 0.714 0.714
...

In this section we have discussed the use of the power method to approximate the
dominant eigenvalue of a matrix. This method can be modified to approximate other eigen-
values through use of a procedure called deflation. Moreover, the power method is only
one of several techniques that can be used to approximate the eigenvalues of a matrix.
Another popular method is called the QR algorithm.
This is the method used in most computer programs and calculators for finding eigen-
values and eigenvectors. The algorithm uses the QR–factorization of the matrix, as pre-
sented in Chapter 5. Discussions of the deflation method and the QR algorithm can be
found in most texts on numerical methods.
 SECTION 10.3 EXERCISES 557

SECTION 10.3 ❑ EXERCISES

21 26

3 4 3 4
In Exercises 1–6, use the techniques presented in Chapter 7 to find 0 0 6 0
the eigenvalues of the given matrix A. If A has a dominant eigen- 17. A 5 2 7 0 18. A 5 0 24 0
value, find a corresponding dominant eigenvector. 1 2 21 2 1 1
23
30 4 3 4
2 1 0
1. A 5 2. A 5 C In Exercises 19 and 20, the given matrix A does not have a domi-
24 1 3
nant eigenvalue. Apply the power method with scaling, starting
25 25 with x0 5 s1, 1, 1d, and observe the results of the first four iterations.
3 4 3 4
1 4
3. A 5 4. A 5
23 21 2 23
22

3 4 3 4
1 1 0 1 2
25 19. A 5 3 21 20. A 5 22 22

3 4 3 4
2 3 1 0 0 0 5
5. A 5 0 21 2 6. A 5 3 7 0 0 0 22 26 6 23
0 0 3 4 22 3 21. (a) Find the eigenvalues and corresponding eigenvectors of
21
322 4
In Exercises 7–10, use the Rayleigh quotient to compute the eigen- 3
A5 .
value l of A corresponding to the given eigenvector x. 4
25 23
32 4 34 31 4 3 4
4 5 2 3 21. (b) Calculate two iterations of the power method with scaling,
7. A 5 ,x5 8. A 5 ,x5
23 2 4 1 starting with x0 5 s1, 1d.

22 21. (c) Explain why the method does not seem to converge to a

3 4 34
1 2 1
9. A 5 22 5 22 , x 5 1 dominant eigenvector.
26 6 23 3 22. Repeat Exercise 21 using x0 5 s1, 1, 1d, for the matrix
23

3 4 34
3 2 3 23

3 4
0 2
10. A 5 23 24 9 ,x5 0 A5 0 21 0 .
21 22 5 1 0 1 22
C In Exercises 11–14, use the power method with scaling to approxi- C 23. The matrix
mate a dominant eigenvector of the matrix A. Start with
2 212
x0 5 s1, 1d and calculate five iterations. Then use x5 to approxi-
mate the dominant eigenvalue of A.
A5 31 25 4
21 has a dominant eigenvalue of l 5 22. Observe that Ax 5 lx
30 4 3 4
2 1 0
11. A 5 12. A 5 implies that
27 1 6
1
24 23 A21x 5 x.
322 4 322 4 l
1 6
13. A 5 14. A 5
8 1 Apply five iterations of the power method (with scaling) on
C In Exercises 15–18, use the power method with scaling to approxi- A21 to compute the eigenvalue of A with the smallest magni-
mate a dominant eigenvector of the matrix A. Start with tude.
x0 5 s1, 1, 1d and calculate four iterations. Then use x4 to approxi- C 24. Repeat Exercise 23 for the matrix
mate the dominant eigenvalue of A.

3 4
2 3 1

3 4 3 4
3 0 0 1 2 0 A5 0 21 2 .
15. A 5 1 21 0 16. A 5 0 27 1 0 0 3
0 2 8 0 0 0
 558 CHAPTER 10 NUMERICAL METHODS

C 25. (a) Compute the eigenvalues of C In Exercises 27 and 28, apply four iterations of the power method
(with scaling) to approximate the dominant eigenvalue of the given
matrix. After each iteration, scale the approximation by dividing by
3 4 3 4
2 1 2 3
A5 and B 5 . its length so that the resulting approximation will be a unit vector.
1 2 1 4
24

3 4
25. (b) Apply four iterations of the power method with scaling to 7 2
3 4
5 6
each matrix in part (a), starting with x0 5 s21, 2d. 27. A 5 28. A 5 16 29 6
4 3
25. (c) Compute the ratios l2yl1 for A and B. For which do you 8 24 5
expect faster convergence?
26. Use the proof of Theorem 10.3 to show that
AsAkx 0d < l1sAkx 0d
for large values of k. That is, show that the scale factors
obtained in the power method approach the dominant eigen-
value.

10.4 APPLICATIONS OF NUMERICAL METHODS

Applications of Gaussian Elimination with Pivoting
In Section 2.5 we used least squares regression analysis to find linear mathematical models
that best fit a set of n points in the plane. This procedure can be extended to cover poly-
nomial models of any degree as follows.

Regression Analysis The least squares regression polynomial of degree m for the points {(x1, y1), (x2, y2),
. . . , (xn, yn)} is given by
for Polynomials
y 5 amxm 1 am21xm21 1 . . . 1 a2x2 1 a1x 1 a0,
where the coefficients are determined by the following system of m 1 1 linear equa-
tions.
na0 1 sS xida1 1
sS xi2da2 1 . . . 1 sS ximdam 5 S yi
sS xida0 1 sS xi2da1 1 sS xi3da2 1 . . . 1 sS xim11dam 5 S xiyi
sS xi2da0 1 sS xi3da1 1 sS xi4da2 1 .. . . 1 sS xim12dam 5 S xi2yi
.
.
.
sS xi da0 1 sS xi da1 1 sS xi da2 1 . . . 1 sS xi dam 5 S ximyi
m m11 m12 2m

Note that if m 5 1 this system of equations reduces to

na0 1 2 sS xida1 5 S yi
sS xida0 1 sS xi2da1 5 S xiyi ,
9/15/23, 9:52 AM Power Method for finding dominant eigenvalue Example [[2,3],[4,10]]

[ ]
2
4 10
3

Solution:
2 3
A=
4 10

1
x0 =
1

1 st Iteration

2 3 1 5
Ax 0 = =
4 10 1 14

and by scaling we obtain the approximation

1 5 0.35714
x1 = =
14 14 1

2 nd Iteration

2 3 0.35714 3.71429
Ax 1 = =
4 10 1 11.42857

and by scaling we obtain the approximation

1 3.71429 0.325
x2 = =
11.42857 11.42857 1

3 rd Iteration

2 3 0.325 3.65
Ax 2 = =
4 10 1 11.3

and by scaling we obtain the approximation

1 3.65 0.32301
x3 = =
11.3 11.3 1

4 th Iteration

2 3 0.32301 3.64602
Ax 3 = =
4 10 1 11.29204

and by scaling we obtain the approximation

1 3.64602 0.32288
x4 = =
11.29204 11.29204 1

∴ The dominant eigenvalue λ = 11.29204

and the dominant eigenvector is :

0.32288
=
1

( Enter your problem )

This material is intended as a summary. Use your textbook for detail explanation.
Any bug, improvement, feedback then Submit Here

https://atozmath.com/example/MatrixEv.aspx?q=powermethod&q1=E1 2/3
9/15/23, 9:54 AM Power Method for finding dominant eigenvalue Example [[8,-6,2],[-6,7,-4],[2,-4,3]]

Power Method for finding dominant eigenvalue ...

[ ]
8 -6 2
-6 7 -4
2 -4 3

Solution:
8 -6 2
A= -6 7 -4
2 -4 3

1
x0 = 1
1

1 st Iteration

8 -6 2 1 4
Ax 0 = -6 7 -4 1 = -3
2 -4 3 1 1

and by scaling we obtain the approximation

4 1
1
x1 = -3 = -0.75
4
1 0.25

2 nd Iteration

8 -6 2 1 13
Ax 1 = -6 7 -4 -0.75 = -12.25
2 -4 3 0.25 5.75

and by scaling we obtain the approximation

13 1
1
x2 = -12.25 = -0.94231
13
5.75 0.44231

3 rd Iteration

8 -6 2 1 14.53846
Ax 2 = -6 7 -4 -0.94231 = -14.36538
2 -4 3 0.44231 7.09615

and by scaling we obtain the approximation

14.53846 1
1
x3 = -14.36538 = -0.9881
14.53846
7.09615 0.4881

4 th Iteration

8 -6 2 1 14.90476
Ax 3 = -6 7 -4 -0.9881 = -14.86905 ( Enter your problem )
2 -4 3 0.4881 7.41667

and by scaling we obtain the approximation

x4 = 1 14.90476 = 1
14.90476 -14.86905 -0.9976

https://atozmath.com/example/MatrixEv.aspx?q=powermethod&q1=E2 2/4
9/15/23, 9:54 AM Power Method for finding dominant eigenvalue Example [[8,-6,2],[-6,7,-4],[2,-4,3]]

7.41667 0.4976

5 th Iteration

8 -6 2 1 14.98083
Ax 4 = -6 7 -4 -0.9976 = -14.97364
2 -4 3 0.4976 7.48323

and by scaling we obtain the approximation

14.98083 1
1
x5 = -14.97364 = -0.99952
14.98083
7.48323 0.49952

6 th Iteration

8 -6 2 1 14.99616
Ax 5 = -6 7 -4 -0.99952 = -14.99472
2 -4 3 0.49952 7.49664

and by scaling we obtain the approximation

14.99616 1
1
x6 = -14.99472 = -0.9999
14.99616
7.49664 0.4999

∴ The dominant eigenvalue λ = 14.99616

and the dominant eigenvector is :

1
= -0.9999
0.4999

This material is intended as a summary. Use your textbook for detail explanation.
Any bug, improvement, feedback then Submit Here

1. Example
[ ]
2
4 10
3
3. Ex

(Previous example)

Share this solution or page with your friends.

Home What's new College Algebra Games Fee

Copyright © 2023. All rights reserved. Terms, Privacy

( Enter your problem )

NOW
PLAYING

https://atozmath.com/example/MatrixEv.aspx?q=powermethod&q1=E2 3/4