Prof. T.-W. Yoon, Korea Univ.

Eigenvectors, SVD & Pseudo inverse

• Definitions of eigenvalues and eigenvectors:

Given A 2 Rn£n,
Av = ∏v, v 6= 0
∏ 2 C (complex number): eigenvalue of A
v 2 Cn (complex vector): eigenvector of A
corresponding to eigenvalue ∏
• Theorem
– ∏ is an eigenvalue of A , det(A ° ∏I) = 0 or det(∏I ° A) = 0
– v is an eigenvector of A corresponding to ∏
, (A ° ∏I)v = 0 or (∏I ° A)v = 0, v 6= 0
(v 2 N (A ° ∏I) = N (∏I ° A))
Prof. T.-W. Yoon, Korea Univ. 2

• Characteristic equation
det(∏I ° A) = ∏n + c1∏n°1 + · · · + cn = 0 where c1, · · · , cn 2 R
Number of eigenvalues = n (including multiplicity)
Number of independent eigenvectors ∑ n
• Ex)
" # " #
0 1 ∏ °1
–A= , ∏I ° A =
°2 3 2 ∏°3

det(∏I ° A) = (∏ ° 2)(∏ ° 1) = 0 , ∏ = ∏1 = 1 or ∏ = ∏2 = 2
(∏1I ° A)v1 = 0, (∏2I ° A)v2 = 0
" # " # " # " #
1 °1 0 2 °1 0
v1 = , v2 =
2 °2 0 2 °1 0
" # " #
1 1
v1 = k1 (k1 6= 0), v2 = k2 (k2 6= 0)
1 2
Prof. T.-W. Yoon, Korea Univ. 3

v2 x Ax

v1

x = c1v1 + c2v2
Ax = c1v1 + 2c2v2 = (∏1c1)v1 + (∏2c2)v2
) Interesting to use eigenvectors as basis vectors
" # " #
0 1 ∏ °1
–A= , ∏I ° A =
°2 2 2 ∏°2

det(∏I ° A) = ∏2 ° 2∏ + 2 = 0, ∏ = 1 ± i
Eigenvectors are also complex (2 C2)
Prof. T.-W. Yoon, Korea Univ. 4

" # " #
1 0 ∏°1 0
–A= , ∏I ° A =
0 1 0 ∏°1

det(∏I ° A) = (∏ ° 1)2 = 0, ∏ = 1, 1
(∏I ° A)v = 0 , 0v = 0
Any two independent vectors (2 R2) are eigenvectors
" # " #
1 1 ∏ ° 1 °1
–A= , ∏I ° A =
0 1 0 ∏°1

det(∏I ° A) = (∏ ° 1)2 = 0, ∏ = 1, 1
(∏I ° A)v = 0
" # " #
0 °1 1
v=0,v=k
0 0 0

Note: Depending on dim N (∏I ° A),

the number of independent eigenvectors varies
Prof. T.-W. Yoon, Korea Univ. 5

• Theorem
All eigenvectors corresponding to different eigenvalues are independent
• Corollary
A has n distinct eigenvalues ) Corresponding n eigenvectors are independent
hProof of the theoremi
· Avi = ∏ivi, vi 6= 0, ∏i 6= ∏j for i 6= j, i, j = 1, 2, · · · , k
· Consider c1v1 + c2v2 + · · · + ck vk = 0
Multiply boths side by (A ° ∏2I)(A ° ∏3I) · · · (A ° ∏k I)
) c1(∏1 ° ∏2)(∏1 ° ∏3) · · · (∏1 ° ∏k )v1 = 0 , c1 = 0
* (A ° ∏j I)vi = (∏i ° ∏j )vi
(A ° ∏k I)(c1v1 + c2v2 + · · · + ck vk )
= c1(∏1 ° ∏k )v1 + · · · + ck°1(∏k°1 ° ∏k )vk°1
In general, multiply boths side by
(A ° ∏1I) · · · (A ° ∏i°1I)(A ° ∏i+1I) · · · (A ° ∏k I) )
ci(∏i ° ∏1) · · · (∏i ° ∏i°1)(∏2 ° ∏i+1) · · · (∏2 ° ∏k )vi = 0 , ci = 0
· v1, · · · , vk independent !
Prof. T.-W. Yoon, Korea Univ. 6

• Corollary
Number of independent eigenvectors < n
) 9 multiple eigenvalues
• Theorem A 2 Rn£n
If ∏ = Æ + jØ (Ø 6= 0) is an eigenvalue of A
¯ = Æ ° jØ
) so is ∏
hProofi
Av = ∏v
Av = ∏v , Av = ∏v , Av = ∏v
• Theorem
The eigenvalues of a triangular matrix are the diagonal elements
Prof. T.-W. Yoon, Korea Univ. 7

Ex) 2 3
1 2
4 3
60 7 7
5 6
6 7
A=6 7
40 9 5
0 8
0 0
10 0
2 3
∏ ° 1 °2 °3 °4
6 0 ∏ ° 5 °6 °7 7
6 7
(∏I ° A)v = 6 7v = 0 (§)
4 0 0 ∏ ° 8 °9 5
0 0 0 ∏ ° 10
· ∏ 6= 1 and ∏ 6= 5 and ∏ 6= 8 and ∏ 6= 10
, v = 0 (* @free variable)
· ∏ = 1 or 5 or 8 or 10 , 9 free variable(s)
, 9 nontrivial solution for (§)
) 1, 5, 8, 10 are eigenvalues
Prof. T.-W. Yoon, Korea Univ. 8

• Theorem
A = AT 2 Rn£n ) All eigenvalues of A are real
hProofi
Av = ∏v ) vT Av = ∏vT v (§)
Av = ∏v , vT AT = ∏vT , vT A = ∏vT
) vT Av = ∏vT v (§§)
From (§§) and (§), (∏ ° ∏)vT v = 0
(∏ ° ∏)||v||2 = 0
∏ = ∏ , ∏ : real
§ Note that ||x||2 = xT x in Rn
||x||2 = xT x in Cn
Prof. T.-W. Yoon, Korea Univ. 9

• Theorem
A = AT 2 Rn£n has distinct eigenvalues
) corresponding eigenvectors are orthogonal
hProofi
Avi = ∏ivi, Avj = ∏j vj (i 6= j, ∏i 6= ∏j )
viT Avj = ∏iviT vj (§)
viT Avj = ∏j viT vj (§§)
(§) ° (§§) ) (∏i ° ∏j )viT vj = 0 , viT vj = 0
• Theorem
A = AT 2 Rn£n
) 9n orthogonal eigenvectors
Prof. T.-W. Yoon, Korea Univ. 10

• Similarity transformation
– A and B are similar
def
() 9 nonsingular M s.t.
B = M°1AM or A = MBM°1 or MB = AM
() The eigenvalues of A and B are the same
– Meaning of similarity transformation:
Consider y = Ax
Change basis to {q1, q2, · · · , qn}
Rewrite x and y as
x = x1q1 + x2q2 + · · · + xnqn
y = y 1q1 + y 2q2 + · · · + y nqn
i.e. 2 3
x1
h i6 x 7
6 7
x = q1 q2 · · · qn 6 2. 7 = Mx
4 .5
xn
Prof. T.-W. Yoon, Korea Univ. 11

2 3
y1
h i6 y 7
6 7
y = q1 q2 · · · qn 6 2. 7 = My
4 .5
yn
where h i
M = q1 q2 · · · qn
2 3 2 3
x1 y1
6x 7 6y 7
6 27 6 7
x = 6 . 7, y = 6 2. 7
4 .5 4 .5
xn yn
x : representation of x w.r.t. {q1, q2, · · · , qn}
y : representation of y w.r.t. {q1, q2, · · · , qn}

Hence
y = Ax , My = AMx
, y = M°1AMx = Bx
Prof. T.-W. Yoon, Korea Univ. 12

• Diagonalization for y = Ax
– Assumption:
9 n independent eigenvectors v1, · · · , vn of A 2 Rn£n
( n distinct eigenvalues
– Changing basis to this set of eigenvectors {v1, · · · , vn}
) x = Mx, y = My
where M = [ v1 v2 · · · vn ]
Then
y = Λx
where
2 3
∏1 0 0 ··· 0
6 7
6 0 ∏2 0 ··· 0 7
6 7
Λ = M AM = 6
°1
6 0 0 ∏3 ··· 0 7 7
6 .. .. .. ... .. 7
4 5
0 0 0 ··· ∏n
Prof. T.-W. Yoon, Korea Univ. 13

hProof (1)i
Avi = ∏ivi
AM = [Av1 Av2 · · · Avn] = [∏1v1 ∏2v2 · · · ∏nvn]
= [v1 v2 · · · vn] diag{∏1, ∏2, · · · , ∏n}
= MΛ , Λ = M°1AM
hProof (2)i
x = x1v1 + x2v2 + · · · + xnvn
y = y 1v1 + y 2v2 + · · · + y nvn
y = Ax
y = y 1v1 + y 2v2 + · · · + y nvn
= A(x1v1 + x2v2 + · · · + xnvn)
= ∏1x1v1 + ∏2x2v2 + · · · + ∏nxnvn
y i = ∏i xi
Prof. T.-W. Yoon, Korea Univ. 14

• Diagonalization of A = AT
Λ = MT AM
where M contains orthonormal eigenvectors of A as its columns
(* MT M = I)
Note: 9 n orthonormal eigenvector always
even when 9 multiple eigenvalues
• Theorem
∏ : eigenvalue of A, v : eigenvector of A
) ∏k : eigenvalue of Ak , v : eigenvector of Ak
hProofi
Av = ∏v, A2v = ∏Av = ∏2v
..
Ak v = ∏k v
Prof. T.-W. Yoon, Korea Univ. 15

• Theorem
f (A) = Æ0I + Æ1A + · · · + ÆnAn (Æi : scalar)
∏ : eigenvalue of A, v : eigenvector of A
) f (∏) : eigenvalue of f (A), v : eigenvector of f (A)
hProofi
f (A)v = (Æ0I + Æ1A + · · · + ÆnAn)v
= (Æ0 + Æ1∏ + · · · + Æn∏n)v = f (∏)v
• Theorem 2 3
∏1 0
6 ∏2 7
6 7
9S s.t. S°1AS = 6 ... 7=Λ
4 5
0 ∏n
(i.e. A is diagonalizable)
Prof. T.-W. Yoon, Korea Univ. 16

2 3
∏k1 0
6 ∏k2 7
6 7
) S°1Ak S = Λk = 6 ... 7
4 5
0 ∏kn

(i.e. Ak is diagonalizable)

In general 2 3
f (∏1) 0
6 f (∏2) 7
6 7
S°1f (A)S = f (Λ) = 6 ... 7
4 5
0 f (∏n)
(i.e. f (A) diagonalizable)

hProofi
From Λk = (S°1AS)(S°1AS) · · · = S°1Ak S
Prof. T.-W. Yoon, Korea Univ. 17

• Theorem
Given two diagonalizable matrices A and B,
A and B share the same eigenvectors , AB = BA
hProofi
())
Λ1 = S°1AS, Λ2 = S°1BS
A = SΛ1S°1, B = SΛ2S°1
AB = SΛ1Λ2S°1, BA = SΛ2Λ1S°1
AB = BA (* Λ1Λ2 = Λ2Λ1)
(() Partial proof for the case where A and B have distinct eigenvalues
AB = BA
Consider Av = ∏v (v : eigenvector of A)
ABv = BAv = B(∏v)
i.e. A(Bv) = ∏(Bv) ) Bv is also an eigenvector of A
9k s.t. Bv = kv
v is also an eigenvector of B
Prof. T.-W. Yoon, Korea Univ. 18

• Theorem (problems 5.18 and 5.19)

(1) det(A) = Πni=1∏i
Pn Pn
(2) trace(A) , i=1 aii = i=1 ∏i
where ∏i: eigenvalue of Adistinct eigenvalues
hProofi
det(∏I ° A) = ∏n + Æ1∏n°1 + · · · + Æn°1∏ + Æ
= Πni=1(∏ ° ∏i)
Pn
= ∏ ° i aii∏n°1 + · · ·
n

(1) det(A ° ∏I) = Πni=1(∏i ° ∏) ) det(A) = Πni=1∏i

Pn Pn
(2) i=1 ∏i = i=1 aii
Prof. T.-W. Yoon, Korea Univ. 19

• Positive (semi) definite matrices

– f (x) : Rn ! R is a positive definite function
def
() f (x) > 0 8 x 6= 0, and f (0) = 0
– f (x) : Rn ! R is a positive semi-definite function
def
() f (x) ∏ 0 8 x
– P = PT 2 Rn£n is a positive definite matrix
def
() xT Px is a positive definite function
() All eigenvalues of P are positive
– P = PT 2 Rn£n is a positive semi-definite matrix
def
() xT Px is a positive semi-definite function
() All eigenvalues of P are nonnegative
– Note that P is assumed to be symmetric without
≥ loss of generality
¥
T
(= xT Px = 12 (xT Px + (xT Px)T ) = xT P+P 2 x =: xT
Px
P = PT
Prof. T.-W. Yoon, Korea Univ. 20

• Theorem (Parts of 6B & 6D)

(1) P is positive definite , All eigenvalues of P are positive
(2) P is positive semi-definite , All eigenvalues of P are nonnegative
(3) ∏min(P)kxk2 ∑ xT Px ∑ ∏max(P)kxk2
hProof of (1)i (Proof of (2) can be given similarly)
(1) ()) Suppose P is positive definite and Px = ∏x (x 6= 0)
) xT Px = ∏xT x > 0 ) ∏ > 0
(() Suppose Pxi = ∏ixi and ∏i > 0 (1 ∑ i ∑ n),
and write x = c1x1 + · · · + cnxn (xTi xj = 0 for i 6= j)
) xT Px = (c1xT1 + · · · + cnxTn )(c1∏1x1 + · · · + cn∏nxn)
= c21∏1 + · · · + c2n∏n > 0 unless x = 0 (c1 = · · · = cn = 0)
(3) (min1∑k∑n ∏k )(c21 + · · · + c2n) ∑ xT Px ∑ (max1∑k∑n ∏k )(c21 + · · · + c2n)
• Matrices of the form MT M or MMT
– positive semi-definite (( xT MT Mx = kMxk2, xT MMT x = kMT xk2)
– positive definite if they have full rank (MT M with M being full column
rank or MMT with M being full row rank)
Prof. T.-W. Yoon, Korea Univ. 21

• Singular values of A 2 Rm£n

– æi(A)2 = eigenvalues of AT A or AAT , i = 1, 2, · · · , min(m, n)
æ1(A) ∏ æ2(A) ∏ · · · ∏ æmin(m,n)(A) ∏ 0
– Note
· The eigenvalues of AT A and AAT are all non-negative real numbers
· The nonzero (positive) eigenvalues of AT A and AAT are the same
( ∏n det(∏Im ° AAT ) = ∏m det(∏In ° AT A)
· AAT = U diag{æ12, æ22, · · · , æm 2
}UT (UT U = UUT = Im)
· AT A = V diag{æ11, æ22, · · · , æn2 }VT (VT V = VVT = In)
· æ1(A) ∏ æ2(A) ∏ · · · ∏ ær (A) > 0, æk (A) = 0 for k > r
where r = rank(A) = rank(AT ) = rank(AAT ) = rank(AT A)
– SVD (Singular Value Decomposition)
8 A 2 Rm£n, 9 U, V and Σ s.t. A = UΣVT
UT U = UUT = Im, VT V = VVT = In and
Σ: m £ n matrix with the singular values on the diagonals
Prof. T.-W. Yoon, Korea Univ. 22

• Remarks on SVD
– AV = UΣ, i.e. Avk = æk uk (AT Avk = æk2vk , AAT uk = æk2uk )
uk = æ1 Avk (k = 1, · · · , r) ) AAT uk = æ1 AAT Avk = æk2uk
k k
– rank(A) = rank(Σ) = number of nonzero (positive) singular values (r)
2 32 T 3
æ1 0 0 · · · 0 v1
6 0 æ 0 · · · 0 7 6 vT 7 X r
6 2 76 2 7 T
– A = [u1 u2 · · · um] 6 . . . . 7 6 7 = æ u v
4 . . . . . .. 5 4 .. 5
k k k
k=1
T
0 0 0 ··· § vn
" #" #
h i Σ 0 VrT
r T
= Ur Un = U r Σr Vr
0 0 VnT
– First r columns of U : Orthogonal basis for R(A)
Last m ° r columns of U : Orthogonal basis for N (AT )
First r columns of V : Orthogonal basis for R(AT )
Last n ° r columns of V : Orthogonal basis for N (A)
i.e. R(A) = R(Ur ), N (AT ) = R(Un), R(AT ) = R(Vr ), N (A) = R(Vn)
Prof. T.-W. Yoon, Korea Univ. 23

Ex)
" #
4 4 p p
A= , æ1 = 4 2, æ2 = 3 2
°3 3
" # " # " #
p1 1
p
T 25 7 2 2
A A= , v1 = ± 1 , v2 = ± 1
7 25 p
2
° p2
" # " # " #
T 32 0 1 0
AA = , u1 = ± , u2 = ±
0 18 0 1
p p
Av1 = 4 2 u1 , Av2 = 3 2 u2
" # " #
1
p °p 1
1 0
U= , V = 12 1
2
0 1 p
2
p
2
" #" p #" #T
1
p °p 1
1 0 4 2 0
A = UΣVT = p 2 2
0 1 0 3 2 p1 p1
2 2
Prof. T.-W. Yoon, Korea Univ. 24

Ex)
2 3
1 °1
6 7 p
A = 4 °1 1 5 , æ1 = 6, æ2 = 0
1 °1
" # " # " #
3 °3 1 1
AT A = , v1 = ± p12 , v2 = ± p12
°3 3 °1 1
2 3 2 3 2 3 2 3
2 °2 2 1 1 1
6 7 6 7 6 7 6 7
AAT = 4 °2 2 °2 5, u1 = ± p13 4 °1 5, u2 = ± p16 4 °1 5, u3 = ± p12 4 1 5
2 °2 2 1 °2 0
p
Av1 = 6 u1
2 3 2 32 p 3
p1 1
p p 1 " # p1 1
p p 1
6 0 " #T
6 13 16 12 7 1
p p 1 6 13 16 12 76 1 1
6 7 2 2 6 7 7 p2 p2
U = 4 ° 3 ° 6 2 5, V =
p p p
1 p1 , A = 4
° 3 ° 6 2 54 0 0 5
p p p
° p ° p1 p1
p1 ° p2 0 2 2 p1 ° p2 0 0 0 2 2
3 6 3 6
Prof. T.-W. Yoon, Korea Univ. 25

• SVD and image compression

r
X
– Original rank-r (monochrome) image: A = æk uk vkT
k=1
(3 matrices are used for a color image)
X̀
– Approximating the image with a rank-` matrix (` ∑ r): A = æk uk vkT
k=1
Prof. T.-W. Yoon, Korea Univ. 26

• Pseudo inverse, A+(2 Rn£m), of A(2 Rm£n)

– Definition 1
· AA+A = A
· A+AA+ = A+
· (A+A)T = A+A
· (AA+)T = AA+
– Definition 2
· A+Axr = xr 8xr 2 R(AT )
· A+yn = 0 8yn 2 N (AT )
– Uniqueness of the pseudo inverse (Definition 1)
* Suppose B and C are two pseudo inverses of A
AB = ACAB = CT AT BT AT = CT (ABA)T = CT AT = AC
BA = BACA = AT BT AT CT = (ABA)T CT = AT CT = CA
) B = BAB = BAC = CAC = C
Prof. T.-W. Yoon, Korea Univ. 27

• Equivalence of the two defintions

– Definition 1 ) Definition 2
A = AA+A, A+ = A+AA+, (A+A)T = A+A, (AA+)T = AA+
· AT = (A+A)T AT = A+AAT, 8xr 2 R(AT ), 9b s.t. xr = AT b
) A+Axr = A+AAT b = AT b = xr
· A+ = A+(AA+)T = A+A+T AT, AT yn = 0 8yn 2 N (AT )
) A+yn = A+A+T AT yn = 0
– Definition 2 ) Definition 1 (A = UΣVT = Ur Σr VrT )
· 8yn 2 N (AT ), 9d 2 Rm°r s.t. yn = Und
Pr
A yn = A Und = 0 ) A = [c1 · · · cr ]Ur (= i=1 ciuTi )
+ + + T

· 8xr 2 R(AT ), 9e 2 Rr s.t. xr = Vr e

A+Axr = xr ) A+Ur Σr VrT Vr e = Vr e 8e, A+Ur Σr = Vr
[c1 · · · cr ]Σr = Vr , ci = æ1i vi, i = 1, · · · r
" #" #
h i Σ°1 0 UTr
+ °1 T r + T
A = Vr Σr Ur = Vr Vn T
=: VΣ U
0 0 Un
) A+ satisfies Definition 1
Prof. T.-W. Yoon, Korea Univ. 28

• The pseudo inverse of A

A+ = VΣ+UT where Σ+: n£m matrix with æk°1 (1 ∑ k ∑ r)
on the diagonals
• Properties
– AA+ is the (orthogonal) projection onto R(A)
* (AA+)T = AA+, (AA+)2 = AA+
R(AA+) Ω R(A), but rank(AA+) ∏ rank(A) (( A = (AA+)A)
) R(AA+) = R(A) (rank(AA+) = rank(A))
– A+A is the (orthogonal) projection onto R(AT )
– x+ := A+b is the minimum length solution to AT Ax = AT b
i.e. Ax = b for b 2 R(A)
* x = A+b + (I ° A+A)w =: x+ + xn satisfies AT Ax = AT b 8w
(( AT Ax = AT AA+b + AT (A ° AA+A)w = AT b)
kxk2 = kx+k2 + kxnk2 (( x+ 2 R(AT ), xn 2 N (A))