Mathematics for Computing 2

Similar Matrices
Two square matrices A and B are said to be similar if there exist
an invertible matrix P such that
B = PAP-1

• If A and B are similar, then they represent the same linear transformation under
different bases
• If two matrices are similar then they have same rank, trace, determinant,
characteristic polynomial and eigenvalues
• If A and B are similar, then An and Bn are also similar
Diagonalization
Diagonalization requires full set of eigenvectors . Hence some matrix are not diagonalizable
Let 1 , 2 , 3 are three distint eigenvalues of 3  3 matrix
Let x1 , x2 , x3 be corresponding eigenvectors.
| |
A x1 = 1 x1
| |
| |
A x2 = 2 x2
| |
| |
A x3 = 3 x3
| |

 1 0 0
 |   |   | 
x3   0 2 0
| | | | | |
A  x1 x2 x3  =  1 x1 2 x2 3 x3  =  x1 x2
 | |   |   | | 
 
| | | |
0 0 3 
S S



AS = S 
A = S S −1
If A=SS−1
A 2 = = ( SS−1 )( SS−1 ) = S S−1S S−1 = S 2S−1
I

 A n = S nS−1
A n x = S nS−1 x
 A1000 x can be computed quite easily
What is S−1 x ?.
Let S−1 x = y
x = Sy = Linear comination of x in terms of eigenvectors of A
y is linear combination coefficient vector
If A is an m  m matrix with distinct eigenvalues
the set of eigenvectors are independent and they form a
basis set for R m
 X1 X2
1 1 
2 2.5 
T
x 1 1 1
x T
2 2.5 x2

3 4 
2
T
x 3 4
3
 
x T
4 4.2  4 4.2 
4
A=
x T
−1 −1.5  −1 −1.5 
 
5

x 6
T
−2 −2.8  −2 −2.8 
x1
x 7
T
−3 −3.5  −3 −3.5 
x T
−4 −3.9  
8
 −4 −3.9 

Here each row represent a point in the X 1 X 2 plane

This matrix can be decomposed into three matrices using SVD

What is the sum of the squares of the lengths of all the row
vectors (here each data point can be considered as a vector).

It is the sum of squares of all the elements A

It is same as the sum of the squares of X1 and X2 coordinates

Based on Matrix A how will be represent these sums??

But a 2-D vector can be decomposed into two components in any two orthogonal directions.

We are again using Pythagoras theorem and since our data vector length do not change, we get
the same sum T
 Full Picture of A = U V T
Basis for Row space

1
2 r

r n-r

r m-r

mm mn nn

U  V T
Basis for column space
Basis for left null space Basis for Right null space
 Reduced Picture of A = U V T Basis for Row space

1
2 r

r

mr rr rn

r

U  V T
Basis for column space
 SVD for two column matrix
 a11 a21   u11 u21 u31   1 0
a  v11 v12 
a22  = u12 u22 u32   0 
2   
 12     v v22 
 a13 a23  u13 u23 u33   0 0  21

VT
A U 
 Every column of A is a linear combination of
columns of U
 a11 a21   u11 u21 u31   1 0 
a       v11 v12 
 12 a22  = u12 u22 u32   0  2   
a23  u13 u23 u33   0 0   21 22 
v v
 a13
VT
A U 

 a11 a21    u11   u21   Columns of U forms orthogonal basis

a         v11 v12 
 12 a22  =  1  u12   2  u22    
for column space of A
 a13 a23    u13   u   v21 v22 
 23  
 a11 a21    u11   u21   u11   u21  
     
a
 12 a22  =  1v11  u12  +  2 v21  u22   1v12  u12  +  2v22  u22  
a23          
 a13  u13   u23   u13   u23  
 a11   u11   u21   a21   u11   u21 
           
a
 12  =  v u
1 11  12  +  v u
2 21  22  , a = v
 22  1 12  12  u +  2 22  u 22 
v
a  u  u  a  u  u 
 13   13   23   23   13   23 
Every row of A is a linear combination
of rows of VT
 a11 a21   u11 u21 u31    1 0 
a  v11 v12 
a22  = u12 u32   0 2 
 v22 
u22
 12     v21 

 a13 a23 
 
 u13 u23 u33 
 0 0  VT
A U 

 a11 a21   u11 u21 u31    1 ( v11 v12 ) 

a   Rows of VT forms orthogonal basis
a22  = u12 u22 u32   2 ( v21 v22 ) 
 12    for row space of A

 a13 a23 
 
 u13 u23 
u33  0 ( 0 0) 

 ( a11 a21 )   u11 1 ( v11 v12 ) + u21 2 ( v21 v22 ) 

   
( a12 a22 )  = u12 1 ( v11 v12 ) + u22 2 ( v21 v22 ) 
 ( a13
 a23 ) 
  u13 1 ( v11
 v12 ) + u23 2 ( v21 v22 ) 


( a11 a21 ) = u11 1 ( v11 v12 ) + u21 2 ( v21 v22 )

( a12 a22 ) = u12 1 ( v11 v12 ) + u22 2 ( v21 v22 )
( a13 a23 ) = u13 1 ( v11 v12 ) + u23 2 ( v21 v22 )
SVD - Visualization
  x1 y1   u11
1
  u11 
   1   
 . .   u12   u12 
 . .   .   . 
     
 . .   .   . 
 . .   .   . 
     
 . .   v11   .   .  | | |
A=  ; A  =   = 1   =  1 u1 ; A v1 =  1 u1

. .
  v12   .   .  | | |

 . .  v1  .   . 
 .    
 .   .   . 
 . .   .   . 
     
 . .   .   . 
x y500   u1  u 
 500  1,500   1,500 
  x1 y1   u121   u21 
   1   
 . .   u22   u22 
 . .   .   . 
     
 . .   .   . 
 . .   .   . 
     
 . .   v21   .   .  | | |
A=  ; A  =   = 2   =  2 u2 ; A v2 =  2 u2

. .
  v22   .   .  | | |

 . .  v2  .   . 
 .    
 .   .   . 
 . .   .   . 
     
 . .   .   . 
x y500   u1  u 
 500  2,500   2,500 
  x1 y1   u11 u21 
   
 . .   u12 u22 
 . .   . . 
   
 . .   . . 
 . .   . . 
   
.    1 0   v11
T
 . .   . v21 
A= =   
. .  . .   0  2   v12 v22 
   
 . .   . .   VT
 .  
 .   . . 
 . .   . . 
   
 . .   . . 
x y500  u1,500 u2,500 
 500
U
Why we say eigenvector correspong to l arg est eigen value of AT A
po int in the direction of max imum var iation of row_wise data points ?
Answer :
Consider 500  2 matrix A whose row vectors are spread as shown in figure
Each row vector is a point in R 2
Let v be 2 1 unit norm vector. vT v = 1
Let us project each row vector in the direction of v.
It result in a vector of 500 elements since we have 500 row vectors. Let it be u.
u = Av.
Spread of elements (variation =  ui2 ) of u assuming its mean as 0 is uT u
i

But u u = ( Av ) Av = v T AT Av
T T

We want to know which direction the spread is maximum

We can cast the problem as an optimization problem
argmax v T AT Av
v

with the constraint v T v = 1

on Taking Lagrangian
L ( v ,  ) = v T AT Av −  ( v T v − 1)
L
= 2 AT Av − 2 v = 0 vector  AT Av =  v
v
To get maximum variation, choose v , that is eigen vector of AT A
Which eigen vector?.
AT Av =  v  vT AT Av =  v T v =  (variation= )
That is  represent measure of variation
Therefore choose v that correspond to maximum eigenvalue
Nonzero eigenvalues of AT A and AAT are same.

Note : AT A is n  n matrix, AAT is m  m matrix. number of eigen values are n and m respectively

Let  and v be a generic eigenvalue and corresponding eigenvector of AT A

 AT Av =  v
 AAT Av =  Av
let Av = u
Then AAT u = u implies  is also an eigenvalue of AAT
Let 1 , 2 ,..., n be eigenvalues and v1 ,v2 ,.., vn be orthonormal eigenvectors of AT A

 i = A
2
Pr ove That Total variation = F
i =1 to n

Variation along direction v = ( Av ) Av = v A Av = v v = v v = 

T T T T T

 Total variation =  i = Trace( AT A) = A

2
F
i =1 to n

Note : Trace(A A) = Trace(AA ) = A F = sum of nonzero eigenvalues

T T 2
What is the relation between eigenvalues  of AT A and sin gular values of A

Answer . Let A be m  n matrix

Consider Av , where v  R n is eigen vecor of AT A for the eigenvalue 
Av is a vector  R m
Let Av = u where u is a unit norm vector in R m ,   R +
(We can show that u is eigenvector of matrix AAT for the same  )
Av = u  ( Av ) Av = ( u )  u
T T

v T AT Av =  2 uT u
=v =1

vT ( v ) =  2
 vT v =  2   = 
=1

 is singular value of A
 SVD
and Pseudo inverse
A = U V T
If A is invertible square matrix
A = (U V )
−1 T −1

= (V )
−1
T
 −1U −1
= V  −1U T ( since U and V are orthonormal matrices)
 SVD and inverse
A = U V T What if A is a rectangular matrix

A = (U V
†
)
T †
; †
stands for Pseudo Inverse

= (V T †
) †U †
= V †U T ( since U and V are orthonormal matrices)
What is † ?.

1 0 0  1 0 0
1 0 0 0   1 0 0 0
  0 2 0 †  
  0 1/ 2 0 If  43 =  ;  34 =  0 1/ 2 0 0 
If 34 =  0 2 0 0  , † 43 = 
0 0 3
0 0 3 0  0 0 1/ 3   0 0 1/ 3 0 
       
 0 0 0  0 0 0
 Psedoinverse
If Ax = b has infinite solution, then b is column space and
Columns of A are dependent.
But x = pinv( A)  b gives a unique solution.
What is the speciality of this solution?.

Answer in the next slide

Let b be in Column space of A
Let columns of A are dependent
VTx = y
A = U V T  VV T x = Vy .
I
U mm  mnVnTn xn1 = bm1
Note : Since x = Vy and V is orthonormal, x = y
V xn1 = U b = b
T T

Let y=V x T x = Vy. But y=†b

y=b x = Vy = V  †b = V  † U T b
b
Example
= V †U T b = A†b
 y1 
a 0 0 0 0     b1'  A†
  y2  
0 b 0 0 0     b2' 
 y3  = ;
0 0 c 0 0     b3' 
   y4   ' 
0 0 0 0 0     b4 
 y5 
y4 and y5 can take any value. This leads into infinite solution
y =  † b
1/ a 0 0 0 '  y1 = b1' / a 
  b   
 0 1/ b 0 0   1'   y2 = b2' / b 
b
y =  † b =  0 0 1/ c 0   2'  =  y3 = b3' / c  → least norm solution for y
   b3   
 0 0 0 0   '   y4 = 0 
 0 0   4   y5 = 0 
b
 0 0
 3. If (  0, v) is eigenvalue-eigenvector pair of A A , then
T

( , Av) is eigenvalue-eigenvector pair of AA T

4. Eigenvectors of AT A corresponding to different eigenvalues are orthogonal

T
5. Eigenvectors of AA corresponding to different eigenvalues are orthogonal
1. rank ( A) = rank ( A A) T

To prove rank ( A) = rank ( A A) T

T
we will show that Rightnullspace(A)=Rightnullspace(A A)

Let x  0 be a vector in rightnullspace of A

Then Ax = 0
But Ax = 0  A Ax = 0
T

 every rightnullspace vector of A is also rightnullspace vector of AT A ---(1)

Let x  0 be a vector in rightnullspace of AT A
Then AT Ax = 0
But AT Ax = 0  xT AT Ax = xT 0 = 0 (scalar)
xT AT Ax = 0  ( Ax ) ( Ax ) = 0
T

 Ax = 0
 every rightnullspace vector of AT A is also rightnullspace vector of A − − − (2)
(1)and (2)  rightnullspace( A) = rightnullspace( AT A) − − − − − (3)
(3)  rowspace( A) = rowspace( AT A)  rowspacedimension( A) = rowspacedimension( AT A)
 rank ( A) = rank ( AT A)
Let (   0, v ) be eigenvalue-eigenvector pair of AT
A
It implies AT Av =  v
AT Av =  v  AAT ( Av ) =  ( Av )
 (  , Av ) is eigenvalue-eigenvector pair of AAT
Let v =1 If v = 1, Av =   u where u = 1
Av need not be 1.
We will show Av =   u where u = 1

We have AT Av =  v
 vT AT Av =  vT v =  . because vT v = 1
 vT AT Av = 
 ( Av ) ( Av ) = 
T

 Av = 
Av  Av Av
Av = = Av =  u , u =1
Av Av
( Av ) is eigenvector of AAT  u is eigenvector of AAT
 A special case.
A be m  n and rank be n
If rank of Amn is n, Then
rank ( A) = n  all columns are independent
1) A A is n  n
T

AT Av1 = 1v1
2) all n eigenvalues AT A are nonzero and positive
3) Matrix V formed by unity norm AT Av2 = 2 v2
eigenvectors is orthonormal ...
4)V T V = I nn = VV T AT Avn = n vn
 Av1 = 1 u1
Av2 = 2 u2
...
Avn = n un
A ( v1 v2 vn ) = ( 1 u1 2 u2 n un )
 1 0  1 
 
AV = ( u1 u2 un )   = ( u1 u2 un )  
 
    n 
 0 n  
AV = US
A = USV T since VV T = I
SVD Applications
SVD based Image Watermarking
 Watermarking is the technique of embedding or
hiding digital information called watermark into
multimedia data for later retrieval and detection.

extracted
watermarked image watermark
watermark

cover image
 Applications
• Copy right protection
• Data monitoring
• Authentication
• Secret communication
• Protection against unauthorized copying an d distribution
 SVD based Watermarking

A → cover image

W→ watermark image

% compute SVD of A (cover image)

A = U  AV T
% Embed watermark (W) into the singular values of
A
Sn =  A + W

 → Scaling parameter to determine

embedding strength

% Obtain watermarked image (AW)

AW = USnV T
• Watermarking extraction → is the process of
recovering or obtaining an estimate of the original
watermark (Ŵ ) from the watermarked image (AW)

ˆ 1
W = ( Sn − )

SVD based Image Compression