Mathematics

of least
squares

Walter
Mudzimbabwe

Decomposition

Mathematics of least squares

Walter Mudzimbabwe
 Least square problem

Mathematics
of least
squares

Walter
Mudzimbabwe Consider the least squares (LS) problem:
Decomposition
Find x ∈ Rn such that Ax = b, A ∈ Rm×n , (m > n) and b ∈ Rm .

The system is overdetermined when there are more equations

than unknowns, m > n.
Usually there is no solution for overdertemined systems.
So we rather solve the problem:

min ||Ax − b||2 , where A ∈ Rm×n , b ∈ Rm .

x∈Rn
 Important properties of solution

Mathematics
of least
squares

Walter
Mudzimbabwe
1.) If x∗ ∈ Rn solves
Decomposition

min ||Ax − b||2

x∈Rn

i.e., ||Ax∗ − b||2 is the minimum then

AT (b − Ax∗ ) = 0.

2.) x∗ ∈ Rn is not necessarily unique.

 Exercise

Mathematics
of least
Verify that
squares AT (b − Ax∗ ) = 0
Walter
Mudzimbabwe where x ∗ solves the least square problem for the matrices
Decomposition
 
a11 a12    
x 1 b
A = a21 ∗
a22  , x = , b= 1
x2 b2
a31 a32
Solution: we consider the minimum using calculus of the
function
1
f (x) = ||b − Ax||22 .
2
Therefore
1
f (x∗ ) = [(b1 − a11 x1 − a12 x2 )2
2
+ (b2 − a21 x1 − a22 x2 )2 + (b3 − a31 x1 − a32 x2 )2 ]
 Exercise

Mathematics
of least
squares
Therefore
Walter
Mudzimbabwe  ∂f 
 ∂x 
5f (x) =  1
Decomposition  ∂f 
∂x2
 
−a11 (b1 − a11 x1 + a12 x2 ) − a21 (b2 − a21 x1 − a22 x2 ) − a31 (b3 − a31 x1 − a32 x2 )
=
−a12 (b1 − a11 x1 − a22 x2 ) − a22 (b2 − a21 x1 − a22 x2 ) − a32 (b3 − a31 x1 − a32 x2 )
 
 (b − a11 x1 + a12 x2 )
a31  1

a a21
= − 11 (b2 − a21 x1 + a22 x2 )
a12 a22 a32
(b3 − a31 x1 + a32 x2 )
T ∗
= −A (b − Ax )

We know that from calculus to miminise a f (x) we equate it’s derivative to zero and solve for x, i.e.,

T ∗
A (b − Ax ) = 0.

The system of equations AT (b − Ax∗ ) = 0 is known as normal equations.

 Solving LS using SVD

Mathematics
of least
squares r
σj uj vjT where r = rank(A) and
P
Walter
Let A =
Mudzimbabwe j=1

U = [u1 , u2 , · · · , · · · , um ] ∈ Rm×m
Decomposition

V = [v1 , v2 , · · · , vn ] ∈ Rn×n

be SVD of A ∈ Rm×n , m ≥ n. If b ∈ Rm then

r
X 1 T
xLS = u bvj .
σj j
j=1

So if you know the SVD of A in an LS problem then you can

solve the LS by using elements in the SVD decomposition.
 Example: Solving LS using SVD

Mathematics
of least
squares

Walter Use SVD method to solve LS problem with

Mudzimbabwe
   
Decomposition
1 2 1
A= 1  4 ,
 b = 1

1 6 1

given that the SVD of A is

 
−0.28 0.87 0.41  
−0.21 0.98
U = −0.54 0.21 −0.82 , V=
−0.98 −0.21
−0.79 −0.45 0.41

and singular values are 7.65 and 0.64.

 Example: Solving LS using SVD

Mathematics
of least
Solution:
squares Note that rank(A) = 2.
Walter
Mudzimbabwe
The solution of the LS is
2
Decomposition
∗
X 1 T 1 T 1 T
x = u bvj = u bv1 + u bv2
σj j 7.65 1 0.64 2
j=1
 
1  
1 −0.21
= [−0.28, −0.54, −0.79] 1
7.65 −0.98
1
 
1  
1 −0.98
+ [0.87, 0.21, −0.45] 1
 
0.64 −0.21
1
 
−0.27
= .
−1.71