Scribd
Upload
33 views

Least Squares & Pseudo Inverse

This document discusses least squares and pseudoinverse methods for line fitting problems with multiple data points. It explains that the least squares method finds the line that minimizes the sum of the squared residuals by using the pseudoinverse. The pseudoinverse (Y') of a matrix Y is defined as (YtY)-1Yt, which allows solving overdetermined systems where the number of equations exceeds the number of unknowns. MATLAB's polyfit function demonstrates solving a linear regression problem using these methods.

Uploaded by

misscoma
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
33 views

Least Squares & Pseudo Inverse

This document discusses least squares and pseudoinverse methods for line fitting problems with multiple data points. It explains that the least squares method finds the line that minimizes the sum of the squared residuals by using the pseudoinverse. The pseudoinverse (Y') of a matrix Y is defined as (YtY)-1Yt, which allows solving overdetermined systems where the number of equations exceeds the number of unknowns. MATLAB's polyfit function demonstrates solving a linear regression problem using these methods.

Uploaded by

misscoma
Copyright
© Attribution Non-Commercial (BY-NC)
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 12

Least Squares

& Pseudo Inverse

1
Contents

1 Line Fitting Problem

2 Least Squares

3 Pseudoinverse

4 Matlab Examples

2
Line Fitting

Find line equation passing through two points

( x1 , y1 )  x1 1 a   y1 
     
 x2 1 b   y2 
( x2 , y 2 ) y  ax  b
a 1  1  1 y1 
     , x1  x2
 b  x1  x2   x2 x1  y2 

It is easy to find
Because the number of unknown variables equals to one of data pairs.
Line Fitting

If we know more equations than unknowns,


How can we determine the straight line equation ?

1. No exact solution exists !!


( x1 , y1 )
( x3 , y3 ) 2. No inversion of matrix exists !!
( x2 , y 2 ) 1
 a   x1 1  y1 
      
 b   x2 1  y2 
1
 x1 1  y1 
a    
    x2 1  y2 
b   y 
 x3 1   3
Least Squares

 y10 y11  y1d   b1 


  a0   
 y20 y21  y2 d    b1 
   a1    or Ya=b
    
    
        
y  ad    a : weight vector to find
 n0 yn1  ynd   bn  b : constant vector

If Y were nonsingular (nd) a=Y-1b

If Y is rectangular (n≠d)

Find a weight vector a that minimizes Ya-b


Least Squares

1. define the error vector

e=Ya-b

2. define the squared length of the error vector

n
Js(a)=||Ya-b||2 =  i i
( a
i 1
t
y  b ) 2

3. find minimum condition by a gradient search

n
Js =  yi  bi ) yi
2( a
i 1
t
= 2Yt(Ya-b)

 YtYa=Ytb
Pseudoinverse

YtYa=Ytb

YtY : square and often nonsingular

If YtY is nonsingular,

a = (YtY)-1Ytb = Yb

Y≡(YtY)-1Yt

: pseudoinverse of Y
Pseudoinverse

matrix equation :

MB=A

find M, when A and B are given

e   || an  Mbn || 2
n

optimal solution :

M=ABT (BBT) -1
Pseudoinverse
differentiation of matrix
d
Trace ( AB)  B T
dA
| e n |   en  en   en en
2 T
d
Trace ( AMB )  AT B T
dM
|| M ||2  Trace ( M T M )
d
Trace ( E T E )  Trace (( A  MB )T ( A  MB )) || M || 2  2 M
dM
 Trace ( AT A  B T M T A  AT MB  B T M TMB )
Trace ( AT A)  0
Trace ( B T M T A)   ABT
Trace ( AT MB )   ABT
Trace ( B T M TMB )  2( B T M T )T B T
d
Trace ( E T E )   ABT  ABT  ( B T M T )T BT  ( B T M T )T B T
dM
d
Trace ( E T E )  0, then ABT  MBBT
dM
 M  ABT ( BBT ) 1
Matlab Examples

Line Fitting : polyfit(x,y,N)

x = [1 2 3 4 5 6 7 8 9];

y = [5 6 10 20 28 33 34 36 42];

x1=[x;ones(1,9)] '

polyfit(x,y,1) pinv(x1)*y'
Conclusion

The method to find a vector a satisfying

Ya=b

if Y is nonsingular
a=Y-1b

if Y is rectangular
a = (YtY)-1Ytb = Yb
Y≡(YtY)-1Yt : pseudoinverse
References

• Richard O. Duda, Peter E. Hart, David G. Stork,“Pattern Classification”, A Wiley-Interscience


Publication, 2nd Edition, pp. 240 ~ 241

• Cleve Moler,“Numerical Computing with MATLAB”, Chapter 5 Least Squares,


(http://www.mathworks.com/moler/leastsquares.pdf)

You might also like