Chapter 6: Application: The Least-Square Solution and

the least-squares error.

1 Best Approximation
In this section we shall show how orthogonal projections can be used to solve a
certain approximation problems.

Theorem: Best Approximation Theorem If W is a finite-dimensional

subspace of an inner product space V , and if u is a vector in V , then projW u
is the best approximation to u from W in the sense that

ku − projW uk < ku − wk

for every vector w in W that is different from projW u.

Proof: For every vector w in W we can write

u − w = (u − projW u) + (projW u − w) (∗)

But projW u − w is a difference of vectors in W , is in W ; and u − projW u is

orthogonal to W , so the two terms on the right hand side of (*) are orthogonal.
Thus,

ku − wk2 = k(u − projW u) + (projW u − w)k2

= ku − projW uk2 + 2 < (u − projW u), (projW u − w) >
+ k(projW u − w)k2
= ku − projW uk2 + kprojW u − wk2 (∗∗)

If w 6= projW u, then the second term in the sum (**) will be positive, so
that
ku − wk2 > ku − projW uk2
or equivalently,
ku − wk > ku − projW uk

1
 P

u u − projW u

Q
projW u
W

Figure 1: Q is the point in W closest to P .

P

u−w
u

u − projW u
O w

Q
projW u
W

Figure 2: ku − wk minimised by w = projW u

2 The Least-Square Solution and the least-squares error.

So far we have been concerned with consistent systems of linear equations.
However, inconsistent linear systems are also important in physical applications.
In such situation one looks for a value of x that comes “as close a possible”
to being a solution in the sense that it mininises the value of kAx − bk with
respect to the Euclidean inner product.
The quantity kAx − bk can be viewed as a measure of the “error” that results
from regarding x as an approximate solution of the linear system Ax = b.
Least Squares Problem. Given a linear system Ax = b of m equations in
n unknowns, find a vector x, if possible, that minimizes kAx − bk with respect
to the Euclidean inner product on Rm . Such a vector is called a least squares
solution of Ax = b.
Remark: To understand the origin of the term least squares

2
 Let e = Ax − b, the error vector that results from the approximation x.

If e = (e1, e2 , . . . , em ), then a least squares solution minimises kek = (e21 +

e22 + . . . + e2m )1/2; hence it minimises kek2 = e21 + e22 + . . . + e2m , hence, the term
least squares.
To solve the least squares problem, let W be the column space of A.

For each n×1 matrix x, the product Ax is a linear combination of the column
vectors of A. Hence as x varies over Rn the vector Ax varies over the entire
column space W . Geometrically, solving the least squares problems is simply
to find a vector x in Rn such that Ax is the closest vector in W to b.

It follows from the Best Approximation Theorem that the closest vector in
W to b is the orthogonal projection of b on W . Thus, for a vector x to be a
least squares solution of Ax = b, this vector must satisfy

Ax = projW b (∗)

Ax
W

Figure 3: A least squares solution x produces the vector Ax in W closet to b.

One could try to find least squares solutions of Ax = b by first calculating

the vector projW b and then solving for x.

There is a better approach. It follows from Projection Theorem that

b − Ax = b − projW b

is orthogonal to W . Therefore, a least squares solution of Ax = b must satisfy

AT (b − Ax) = 0

3
or equivalently,

AT Ax = AT b
This is called the normal system associated with Ax = b, and the individual
equations are called the normal equations associated with Ax = b.

Thus, the problem of finding a least squares solution of Ax = b has been

reduced to problem of finding an exact solution of the associated normal system.
The previous discussion is summarised into the following theorem:

Theorem: Least-Squares Solution

The Least-Square Solution of an inconsistent linear system given by

Ax = b,

is the solution to the system of the so-called normal equations

AT Ax̂ = AT b,

Further, if x̂ is the least-squares solution, then the least-squares error is

given by the formula
kb − Ax̂k.

Example: Find a least-square solution of the inconsistent system

4x1 = 2
2x2 = 0
x1 + x2 = 11

Note that, in matrix form, the above system can be represented as

   
4 0   2
 0 2  x1 =  0 
x2
1 1 11
   
4 0   2
x1
or Ax = b, where A =  0 2 , x = , and b =  0 .
x2
1 1 11

4
 Given Ax = b, as above, we first form a new system, the so-called system of
normal equations:
AT Ax̂ = AT b.
 
4 0  
T 4 0 1
Since A =  0 2 , we see that A = , hence
0 2 1
1 1
   
  4 0     2
4 0 1  17 1 4 0 1 
AT A = 0 2 = , and AT b = 0  =
0 2 1 1 5 0 2 1
1 1 11
 
19
.
11
So the system of normal equations AT Ax̂ = AT b in our case is
    
17 1 x̂1 19 17x̂1 + x̂2 = 19
= or .
1 5 x̂2 11 x̂1 + 5x̂2 = 11

One can solve this system by using the techniques learnt in the first half of
the semester and obtain x̂1 = 1 and x̂2 = 2, i.e., the least-squares solution to
the original system is    
x̂1 1
x̂ = = .
x̂2 2

To find the least-squares error we need to compute

kb − Ax̂k.
   
2 4 0  
1
In our case, b =  0 , A =  0 2 , and x̂ = . Hence Ax̂ =
2
11 1 1
         
4 0   4 2 4 −2
 0 2  1 =  4 , b − Ax̂ =  0  −  4  =  −4  and
2
1 1 3 11 3 8
p √
kb − Ax̂k = (−2)2 + (−4)2 + 82 = 84.

5
3 The Least-Squares Lines.
Suppose you are given the following data points:
(2, 1), (5, 2), (7, 3), (8, 3)
.
If you plot these 4 points on a graph, a straight-line graph is suggested, but
the four points do not perfectly fit on a single straight line.
Our aim now is to find the best straight line that fits the data.
A straight line is given by the equation y = a + bx. Once we know a and b,
we have our line. So let us find the “best” a and b for the data given above:
The first point says that, if x = 2, then y = 1, giving the equation
1 = a + b(2) = a + 2b.
The rest of the equations are
2 = a + 5b, 3 = a + 7b, & 3 = a + 8b
, so we have the following system of linear equations:
a + 2b = 1
a + 5b = 2
a + 7b = 3
a + 8b = 3
and we want to solve for a and b.
If the data points are not collinear, the it is impossible to find the coefficients
a and b that satisfy the system of liner equations exactly. This is an inconsistent
system.
In this case we shall look for a least squares solution.
   
1 2   1
1 5
, x = a , and b =  2 , then in matrix form the
 
If we let A = 
1 7 b 3
1 8 3
system is the now familiar
Ax = b

6
 and we have to solve the system of normal equations

AT Ax̂ = AT b
 
  1 2  
1 1 1 1  1 5  4 22
as we did before: AT A =  = ,
2 5 7 8 1 7 22 142
1 8
 
  1  
1 1 1 1  2  9
AT b =  = and the system of normal equations is
2 5 7 8 3 57
3
    
4 22 â 9
= .
22 142 b̂ 57

We obtain the least-squares solution

 
2/7
x̂ = ,
5/14

i.e., â = 2/7 and b̂ = 5/14 are the best choices for a and b, and the least-squares
line is
2 5
y = + x.
7 14

Suppose we are also asked to “predict” or “estimate” or “approximate” the

value of y when x = 10.
We simply plug in the value of 10 for x in the equation we obtained above
to get:
2 5 27
y = + (10) = .
7 14 7