Scribd
Upload
59 views

CSE/MATH 6643: Numerical Linear Algebra: Haesun Park

This document describes solving least squares problems using singular value decomposition (SVD). It explains that for a matrix A, decomposing it into its SVD A = UΣV^T allows writing the least squares problem as minimizing ||ΣV^Tx - U^Tb||_2. The solution is given by x=V(Σ^-1U^Tb, z) where z can be any vector. It also discusses solving least squares for rank deficient matrices using QR decomposition with column pivoting.

Uploaded by

Muhammad Rizwan
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
59 views

CSE/MATH 6643: Numerical Linear Algebra: Haesun Park

This document describes solving least squares problems using singular value decomposition (SVD). It explains that for a matrix A, decomposing it into its SVD A = UΣV^T allows writing the least squares problem as minimizing ||ΣV^Tx - U^Tb||_2. The solution is given by x=V(Σ^-1U^Tb, z) where z can be any vector. It also discusses solving least squares for rank deficient matrices using QR decomposition with column pivoting.

Uploaded by

Muhammad Rizwan
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 13

CSE/MATH 6643: Numerical Linear Algebra

Haesun Park
[email protected]

School of Computational Science and Engineering


College of Computing
Georgia Institute of Technology
Atlanta, GA 30332, USA

Lecture 14

Least Square Problem SVD


7.4.4

Solving LS

min kAx bk2 , A Rmn , b Rm1 , m n.


x

Let the SVD of A be

A = U V

U1

U2

"

1
0

0
0

V1

V2

where U1 Rmr , U2 Rm(mr) , 1 = diag(1 , , r ), 1 r > 0,


and V1 Rnr , V2 Rn(nr) ,i.e. rank(A) = r.

CSE/MATH 6643: Numerical Linear Algebra p.1/12

Least Square Problem SVD


kAx bk2

=
=

T
T
U V x b = U (U V x b)
2
2

T
T
V x U b
2

8
>
>
<
>
>
:

V1T
V2T

Letting V T x =
T

U b=

"

1

0
0

U1T
U2T

#"

y
z

b=

"

x=

nr

mr

#
# "

c 1 y c

d
d

CSE/MATH 6643: Numerical Linear Algebra p.2/12

Least Square Problem SVD


Since 1 Rrr , non-singular, we can find the unique solution for
1 y c = 0 1 y = c y = 1
1 c.
Letting r(x) = kAx bk2 , the residual r(xLS ) = kdk where xLS is the LS
solution.
The solution is
xLS = V

"

y
z

where y = 1
1 c and z can be anything. (if rank(A) = n, z is null).
!

y
= V1 y + V2 z
xLS = V1 V2
z
Note V2 z null(A).

When z = 0, xLS = V

"

1
1 c
0

is called minimum-norm solution.

CSE/MATH 6643: Numerical Linear Algebra p.3/12

Least Square Problem Rank Decision


7.4.5

Example

A R64 , 4 (A) = 1016 , 3 (A) = 1014 .


2
1
6
0.5
6
6
6
1014
6
T
A = U V = U 6
6
1016
6
6
4

7
7
7
7
7 T
7V
7
7
7
5

Depending on rank decision (2 or 3 or 4?), we obtain very different solutions.


If we consider
the tolerance 3s.t. 1016
32
2 < , and rank(A)
2
1
1
76
6
7
6
76
7,y = 6
1 = 6
2
0.5
54
4
5
4
14
14
10
10

= 3,
3
x
7
x 7
5.
x

If we consider the tolerance s.t. 1014 < , and rank(A) = 2,


"
#
"
#"
#
1
1
x
1 =
,y =
.
0.5
2
x

CSE/MATH 6643: Numerical Linear Algebra p.4/12

Least Square Problem Rank Decision


Min-norm solution:
the first case, xLS

2 2

6 6
6 4
=V 6
4

the second case, xLS

2 "

6
=V 4

1014
0
#"

32

3 3

x
76
7 7
54 x 5 7
7
5
x
x
x

# 3
7
5

0
Determine numerical rank can be difficult. Usually we find a large gap.
2
3
1
6
7
101
6
7
6
7
102
6
7 no gap?
6
7
3
6
7
10
4
5
..
.

CSE/MATH 6643: Numerical Linear Algebra p.5/12

Least Square Problem


In some applications, just need to compute the size of the residual vector.
= Can be done WITHOUT computing the solution vector x.
r = AxLS b
krk2 = kAxLS bk2
r

=
=
=
=

krk2 = kU

0
d

U V T xLS b
!
!
1 0
y
U
U
0 0
z
!
1 y c
U
d
!
0
U
d

k2 = kdk2 .
CSE/MATH 6643: Numerical Linear Algebra p.6/12

c
d

Least Square Problem


Full Rank case
h

Theorem. If A Rmn , rank(A) = n, A = a1 an ,


i
i h
h
Q = q1 qm = Q1 Q2 where Q1 Rnn , Q2 R(mn)n ,
"
#
R
A=Q
,
0
then
span {a1 , , ak } = span {q1 , , qk } (1 k n)

CSE/MATH 6643: Numerical Linear Algebra p.7/12

Least Square Problem Rank Deficient


Proof: A = Q

a1

ak =

k
P

"

an

R
0
i

#
h

q1

Q1

Q2

qn

"
2

i6
6
4

r11

= Q1 R

..
.

r1n
.. 7
7
. 5
rnm

qi rik span {q1 , , qk } = span {a1 , , ak } span {q1 , , qk }

i=1

As the dimensions of both span {a1 , , ak } and span {q1 , , qk } are k,


span {a1 , , ak } = span {q1 , , qk }.
Range(A) = R(A) = span {a1 , , ak } = span {q1 , , qk } = R(Q1 )

m1 T
R (A) = z R
|z w = 0 for w R(A)

CSE/MATH 6643: Numerical Linear Algebra p.8/12

Least Square Problem Rank Deficient


QRD with Column Pivoting for Deficient Rank
If A = QR and rank(A)
i 1 , , qk },
i {a1 , h , ak } = span {q
h = n, then span

1 k n where A =

a1

an

,Q=

q1

qn

Why QRD with Column Pivoting?


2

6
Eg. A = 6
4

Consider QRD of A

7
1 7
5.rank(A) = 2.
1
2

6
A = QR = 6
4

32

1
1
1

76
76
54

7
1 7
5
1

Although rank(A) = 2, we dont have Range(A) = span {qi , qj : i 6= j}. QRD with
C.P. can help us to maintain span {a1 , , ak } = span {q1 , , qk } in rank deficient
case.

CSE/MATH 6643: Numerical Linear Algebra p.9/12

Least Square Problem Rank Deficient


For any A Rmn , QRD with Column Pivoting computes
!
R
A = Q
0
, where
0

B
R=@

R11
0
|{z}

(n r) r

R12
0
|{z}

(n r) (n r)

1
C
A

where R11 Rrr is upper-triangular matrix, R12 Rrnr ,


r = rank(A) = rank(R) = rank(R11 ).
Q Rmm , orthogonal; Rnn , permutation.

CSE/MATH 6643: Numerical Linear Algebra p.10/12

Least Square Problem Rank Deficient


Solving LS
A = Q

"

R
0

6
= Q6
4

R11

T x =

y
z

"

11

T
x b =

R12

R12

nr

R11
6
6
kAx bk2 =
4 0

R12

Letting

"

R11

R12

"

R11

kAx bk2 = Q

, QT b =

R12
0
0

6
7
7 A = Q 6
4
5

"

We can choose

mr

"
# "
#

7
y
c
=
7

z
d

kAx bk22 = kR11 y + R12 z ck22 + kdk22


!

7 T
7 .
5
#

T
T
x Q b

,
!

R11 y + R12 z c

so that

z
R11 y + R12 z c = 0 R11 y = c R12 z

CSE/MATH 6643: Numerical Linear Algebra p.11/12

Least Square Problem Rank Deficient


z can be anything, and y can be chosen so that R11 y = c R12 z. Can set
z = 0, then y satisfies R11 y = c.
"
#
1
R11 (c R12 z)
x=
where z is free.
z
#
"
1
R11 c
.
If we set z = 0, we get basic solution x =
0
#
"
R11
and R11 = R.
When rank(A) = n, A = Q
0

CSE/MATH 6643: Numerical Linear Algebra p.12/12

You might also like