Bachelor in Statistics and Business Mathematical Methods II

Universidad Carlos III de Madrid Marı́a Barbero Liñán

Homework sheet 5: ORTHOGONALITY AND LEAST-SQUARES

PROBLEM
Year 2011-2012

  
    3 6
−1 4
1. Let u = , v = , w =  −1  , x =  −2  . Compute the
2 6
−5 3
x·w 1 x·w
following quantities: w · w, x · w, , u, x, ||x||.
w·w u·u x·x
 
  30/49
1 −1/5 
Solution: Respectively: 35; 5; , , −10/49.
7 2/5
15/49

 
−6  
8/3
2. Let v =  4 , w=
 . Find a unit vector in the direction of each given
2
−3
vector.
√
   
−6 −6  
1   61   4/5
Solution: Respectively: √ 4 = 4 , .
61 −3 61 3/5
−3

   
0 −4
3. Find the distance between x = −5 and z = −1.
2 8
√
Solution:dist(u, z) = ku − zk = 2 17.

4. Let u = [5, −6, 7]T and W be the subset of all vectors x in IR3 such that u · x = 0.
Give a geometric interpretation of W and show that W is a subspace.
Solution: W is a plane in R3 that contains the origin with normal vector u. Hence
it is a subspace. Another way to prove it is to check that 0 ∈ W , x + y ∈ W for
x, y ∈ W , λx ∈ W for λ ∈ R and x ∈ W .

5. Let u = (u1 , u2 , u3 ). Explain why u · u ≥ 0. When is u · u = 0?

Solution:u · u = u21 + u22 + u23 . As u · u is equal to the sum of positive numbers,
u · u ≥ 0. The inner product u · u = 0 if and only if the three positive numbers in
the addition vanish, that is, u21 = u22 = u23 = 0. Equivalently, u1 = u2 = u3 = 0.
Thus u · u = 0 if and only is u is the zero vector.
   
a x
6. Let v = . Describe the set H of vectors that are orthogonal to v.
b y

1
  
−b
Solution:H = span .
a
7. Determine which set of vectors are orthogonal:
     
      5 −4 3 
1 0 −5 

 
−4   1   3 
   
S1 =  −2  ,  1  ,  −2  , S2 =   0  ,  −3
,
  5  .

1 2 1
  
 
3 8 −1
 

Solution: S1 is orthogonal, S2 is not orthogonal.

     
3 2 1
8. Show that u1 =  −3  , u2 =  2  , u3 =  1  is an orthogonal basis
0 −1 4
for R3 . Then express the vector x = [5, −3, 1]T as a linear combination of the u0 s.
Solution:It needs to be proved that u1 · u2 = u1 · u3 = u2 · u3 = 0. As they
are three orthogonal vectors two by two in R3 , which are different from the zero
vector, they are linearly independent.
 
5
x · u1 x · u2 x · u3 4 1 1
x = −3 =
 u1 + u2 + u3 = u1 + u2 + u3 .
u1 · u1 u2 · u2 u3 · u3 3 3 3
1

   
1 −1
9. Compute the orthogonal projection of y = onto the line through
−1 3
and the origin.
 
y·v 2/5
Solution:projv y = v= .
v·v −6/5
   
2 7
10. Let y = and u = . Write y as the sum of a vector in span{u} and a
6 1
vector orthogonal to u.
 
−1
Solution:An orthogonal vector to u is v = . Then
7

y·u y·v 2 4
y= u+ v = u + v.
u·u v·v 5 5

   
−3 1
11. Let y = , u= . Compute the distance from y to the line through u
9 2
and the origin.

 distance is given by the norm/length of the following vectory −

Solution:The
−6 √
proju y = . Hence, the distance is 3 5.
3

2
12. Determine which of the following sets of vectors are orthonormal. If a set is only
orthogonal, normalize the vectors to produce an orthonormal set.
2
  
 1 
    
 − 
 0 0   13   3 

 
  
S1 =  1 , −1
   , S2 =   ,   ,
   2 

0 0
 

  32  3 


 0 
3 
1
 
2 
 1  
 √ − 
√

18
 
 4   2   3 

    
1

S3 =  √  ,  0  ,   .
   
  18   1    32 


  1  −√ 

 √ −
 
2
 
3

18

Solution:S1 is not orthogonal, then it is not orthonormal. S2 is orthogonal, but

the second vector
isnot a unit
 vector.
√ After
 normalizing, we obtain the following
 −2/3 √ 5/5 
orthonormal set:  1/3  , 2 5/5 . S3 is orthonormal.
2/3 0
 

13. Let {v1 , v2 } be an orthogonal set of nonzero vectors and let c1 , c2 be any nonzero
scalars. Show that {c1 v1 , c2 v2 } is also an orthogonal set.
Solution: Using the properties of the inner product, we have (c1 u1 ) · (c2 u2 ) =
(c1 c2 )(u1 · u2 ) = 0.

14. Let W be the subspace spanned by the vectors u0 s. Write y as the sum of a vector
in W and a vector orthogonal to W .
     
−1 1 0
a) y =  4  , u1 =  1  , u2 =  4 
3 1 −1
       
3 1 2 1
 4   1   1   0 
b) y = 
 5  , u1 =  0  , u2 =  1  , u3 =  1 
      

6 −1 0 −2
Solution:We first have to compute projW y, then y − projW y to obtain a vector
orthogonal to W .
   
3/2 −5/2
a) y = 7/2 +  1/2 ;
1 2
   
5 −2
2  2 
b) y = 
3 +  2 .
  

6 0

3
15. Let W be the subspace spanned by the vectors u1 and u2 . Find the closest point
to y in the subspace W .
     
3 1 −3
 −1   −2   −1 
y=  1  , u1 =  −1  , u2 =  −1
    

13 2 5

 
−1
−5
Solution:projW y =  −3.


9
√ √
16. Let y = [7, 9]T , u1 = [1/ 10, −3/ 10]T , and W = Gen{u1 }.

a) Let U be the 2 × 1 matrix whose only column is u1 . Compute U T U and

UUT .
b) Compute proyW y and (U U T )y.
   
T T 1/10 −3/10 T −2
Solution: U U = 1, U U = , projW y = (U U )y = .
−3/10 9/10 6

     
1 5 0
17. Let u1 =  1 , u2 = −1 , u3 = 0. Show that u1 and u2 are orthogonal
   
−2 2 1
but that u3 is not orthogonal to u1 or u2 . Show that u3 is not in the subspace W
spanned by u1 and u2 . Use this fact to construct a nonzero vector v in R3 that is
orthogonal to u1 and u2 .
Solution: It must be proved that u1 · u2 = 0, u3 · u1 = −2 6= 0, u3 · u2 = 2 6= 0.
To check that u3 is not in the vector space spanned by u1 and u2 we compute the
determinant of the matrix given by the those three vectors. As the determinant
is different from zero, the three vectors are linearly independent. Thus, u3 is not
in the subspace W spanned by u1 and u2 . A nonzero orthogonal vector to u1 and
u2 is obtained as follows:
 
0
u3 − projW u3 = 2/5 .

1/5

18. Use the Gram-Schmidt process to produce an orthonormal basis for the subspace
W spanned by the following vectors:
   
        3 −5
0 5 3 −3 −1  9 
(a) 4 ,  6  (b) −4 ,  14  (c)   2  , −9 .
  
2 −7 5 −7
−1 3

4
 Solution:
     √   
0√ 5 3/(5 √2) 1
1   1  
(a) 2/√5 , √
 4 , (b) 
√ 2) , √6 2 ,
−4/(5 
1/ 5 105 −8 1
   2
5/
3 4
1  −1  1 6
(c) √  , √  .
15  2  61 −3
−1 0

19. Use the Gram-Schmidt process to find an orthogonal basis for the column space
of each of the following matrices:
 
  1 3 5
−1 6 6  −1 −3 1 
 3 −8 3   
(a)   (b)  0 2 3  .
1 −2 6  
 1 5 2 
1 −4 −3
1 5 8
      

 −1 3 −1 
 1   1   1  
3  1  , √ −1 .

Solution:(a) √  , √
 1  2 3  1  2 3  3 
2 3

 
1 −1 −1

      

 1 −1 1  
 1 −1 1  1  1  1 

      


(b)  0 , √  2 ,  0  .
     
 2  1  2 2  1  2 −1

 

 
1 1 1
 

20. Find a least-squares solution of Ax = b by constructing the normal equations

for x̂ and solving for x̂. In the first two exercises compute the least-squares error
associated with the least-squares solution found.
   
2 1 −5
a) A =  −2 0  , b =  8 
2 3 1
   
1 −2 3
 −1 2 
 , b= 1 
 
b) A =   0  −4 
3 
2 5 2
   
1 1 0 1
 1 1 0   3 
c) A =   1 0 1  , b= 8 
  

1 0 1 2

5
  
   
√ 5 − z
−4 4/3
Solution: (a) , zero error. (b) , the error is 2 5. (c) −3 + z .
3 −1/3
z
21. Find the orthogonal projection of b onto the column space of A and find a least-
squares solution of Ax = b.
   
1 5 4
a) A =  3 1  , b =  −2 
−2 4 −3
   
1 2 3
b) A =  −1 4  , b =  −1 
1 2 5
   
1 1 0 2
 1 0 −1   5 
c) A =   0 1
 , b= 
1   6 
−1 1 −1 6
   
1   4  
2/7 3
Solution:(a) projColA b = 1 , x
  b= . (b) projColA b = −1 , x
  b= .
1/7 1/2
0 4
 
5  
2 1/3
(c) projColA b =   b =  14/3 .
3, x
−5/3
6
  
x+y =2 x
22. Describe all least-squares solutions of the system: . Solution: =
x+y =4 y
 
3−y
.
y

23. Find the equation y = β0 + β1 x of the least-squares line that best fits the given
data points:

a) (0, 1), (1, 1), (2, 2), (3, 2).

b) (2, 3), (3, 2), (5, 1), (6, 0).
9 2 43 7
Solution:(a) y = + x. (b) y = − x.
10 5 10 10
24. Let (x1 , y1 ), (x2 , y2 ), (x3 , y3 ) be the data that must be fit by a least-squares line.
Show that the normal equations have a unique solution if and only if the data
include at least two data points with different x-coordinates.

Solution: Find the system of linear equations imposing thatthe straight

 line
1 x1  
β
y = β0 + β1 x must contain the following points three points: 1 x2  0 =
β1
1 x3

6
  
y1
y2 .
y3
The augmented matrix of the system has the following echelon form:
 
1 x1 y1
0 x2 − x1 y2 − y1  .
0 x3 − x1 y3 − y1

The system has a unique solution if the last column is not a pivot column. That is,
x2 − x1 6= 0 or x3 − x1 6= 0, these two equations could be satisfied simultaneously.

25. A certain experiment produces the data (1,1.8), (2,2.7), (3,3.4), (4,3.8), (5,3.9).
Describe the model that produces a least-squares fit of these points by a function
of the form y = β1 x + β2 x2 .
Solution:    
1 1 1,8
2 4    2,7
3 9  β1 = 3,4
   
  β2  
4 16 3,8
5 25 3,9

Additional exercises: D. C. Lay “Linear algebra and its applications”,

2012.

Sections 6.1-6.6.