Orthogonal Projections and Reflections (with exercises)

by Dan Klain
October 16, 2018
Corrections and comments are welcome.

Orthogonal Projections

Let X1 , . . . , Xk be a family of linearly independent (column) vectors in Rn , and let

W = Span( X1 , . . . , Xk ).
In other words, the vectors X1 , . . . , Xk form a basis for the k-dimensional subspace W
of Rn .
Suppose we are given another vector Y ∈ Rn . How can we project Y onto W
orthogonally? In other words, can we find a vector Ŷ ∈ W so that Y − Ŷ is orthogonal
(perpendicular) to all of W? See Figure 1.
To begin, translate this question into the language of matrices and dot products.
We need to find a vector Ŷ ∈ W such that
(1) (Y − Ŷ ) ⊥ Z, for all vectors Z ∈ W.
Actually, it’s enough to know that Y − Ŷ is perpendicular to the vectors X1 , . . . , Xk
that span W. This would imply that (1) holds. (Why?)
Expressing this using dot products, we need to find Ŷ ∈ W so that
(2) XiT (Y − Ŷ ) = 0, for all i = 1, 2, . . . , k.
This condition involves taking k dot products, one for each Xi . We can do them all at
once by setting up a matrix A using the Xi as the columns of A, that is, let
 

A =  X1 X2 · · · Xk  .

Note that each vector Xi ∈ Rn has n coordinates, so that A is an n × k matrix. The set
of conditions listed in (2) can now be re-written:
A T (Y − Ŷ ) = 0,
which is equivalent to
(3) A T Y = A T Ŷ.

Meanwhile, we need the projected vector Ŷ to be a vector in W, since we are pro-

jecting onto W. This means that Ŷ lies in the span of the vectors X1 , . . . , Xk . In other
1
2

X1
Ŷ
X2
W o

Figure 1. Projection of a vector onto a subspace.

words,
c1
 
 c2 
Ŷ = c1 X1 + c2 X2 + · · · + ck Xk = A 
 ...  = AC.


ck
where C is a k-dimensional column vector. On combining this with the matrix equa-
tion (3) we have
A T Y = A T AC.
If we knew what C was then we would also know Ŷ, since we were given the columns
Xi of A, and Ŷ = AC. To solve for C just invert the k × k matrix A T A to get
(4) ( A T A)−1 A T Y = C.
How do we know that ( A T A)−1 exists? Let’s assume it does for now, and then address
this question later on.
Now finally we can find our projected vector Ŷ. Since Ŷ = AC, multiply both sides
of (4) to obtain
A( A T A)−1 A T Y = AC = Ŷ.
The matrix
Q = A ( A T A ) −1 A T
is called the projection matrix for the subspace W. According to our derivation above, the
projection matrix Q maps a vector Y ∈ Rn to its orthogonal projection (i.e. its shadow)
QY = Ŷ in the subspace W.
It is easy to check that Q has the following nice properties:

(1) Q T = Q.
(2) Q2 = Q.

One can show that any matrix satisfying these two properties is in fact a projection
matrix for its own column space. You can prove this using the hints given in the
exercises.
 3

There remains one problem. At a crucial step in the derivation above we took the
inverse of the k × k matrix A T A. But how do we know this matrix is invertible? It
is invertible, because the columns of A, the vectors X1 , . . . , Xk , were assumed to be
linearly independent. But this claim of invertibility needs a proof.
Lemma 1. Suppose A is an n × k matrix, where k ≤ n, such that the columns of A are
linearly independent. Then the k × k matrix A T A is invertible.

Proof of Lemma 1: Suppose that A T A is not invertible. In this case, there exists a vector
X 6= 0 such that A T AX = 0. It then follows that

( AX ) · ( AX ) = ( AX )T AX = X T A T AX = X T 0 = 0,
p
so that the length k AX k = ( AX ) · ( AX ) = 0. In other words, the length of AX is
zero, so that AX = 0. Since X 6= 0, this implies that the columns of A are linearly
dependent. Therefore, if the columns of A are linearly independent, then A T A must
be invertible. 

Example: Compute the projection matrix Q for the 2-dimensional subspace W of R4

spanned by the vectors (1, 1, 0, 2) and (−1, 0, 0, 1). What is the orthogonal projection
of the vector (0, 2, 5, −1) onto W?
Solution: Let
 
1 −1
 1 0 
A=
 0
.
0 
2 1
Then
2 −1
   
T 6 1 T −1 11 11
A A= and ( A A) = −1 6 ,
1 2 11 11
so that the projection matrix Q is given by
10 3 −1
   
1 −1  11 0
11 11
2 −1 3 2 3
  
 1 0  1 1 0 2 0
Q = A ( A T A ) −1 A T = 
 
 11 11 = 11 11 11
−
 
 0 0  11 111 6 −1 0 0 1  0 0 0 0 
2 1 −1 3 10
11 11 0 11

We can now compute the orthogonal projection of the vector (0, 2, 5, −1) onto W. This
is
−1
 10 3    7 
11 11 0 11 0 11
 3 2 0 3  2   1 
 11 11 11    =  11 
 0 0 0 0  5   0 
−1 3 10 −1 −4
11 11 0 11 11
4

v̂
X1
u
X2
W o
Refl(v)

Figure 2. Reflection of the vector v across the plane W.

Reflections

We have seen earlier in the course that reflections of space across (i.e. through)
a plane is linear transformation. Like rotations, a reflection preserves lengths and
angles, although, unlike rotations, a reflection reverses orientation (“handedness").
Once we have projection matrices it is easy to compute the matrix of a reflection.
Let W denote a plane passing through the origin, and suppose we want to reflect a
vector v across this plane, as in Figure 2.
Let u denote a unit vector along W ⊥ , that is, let u be a normal to the plane W. We
will think of u and v as column vectors. The projection of v along the line through u
is then given by:
v̂ = Proju (v) = u(u T u)−1 u T v.
But since we chose u to be a unit vector, u T u = u · u = 1, so that
v̂ = Proju (v) = uu T v.
Let Qu denote the matrix uu T , so that v̂ = Qu v.
What is the reflection of v across W? It is the vector ReflW (v) that lies on the other
side of W from v, exactly the same distance from W as is v, and having the same
projection into W as v. See Figure 2. The distance between v and its reflection is
exactly twice the distance of v to W, and the difference between v and its reflection
is perpendicular to W. That is, the difference between v and its reflection is exactly
twice the projection of v along the unit normal u to W. This observation yields the
equation:
v − ReflW (v) = 2Qu v,
so that
ReflW (v) = v − 2Qu v = Iv − 2Qu v = ( I − 2uu T )v.
The matrix HW = I − 2uu T is called the reflection matrix for the plane W, and is also
sometimes called a Householder matrix.
 5

Example: Compute the reflection of the vector v = (−1, 3, −4) across the plane
2x − y + 7z = 0.
Solution: The vector w = (2, −1, 7) is normal to the plane, and w T w = 22 + (−1)2 +
72 = 54, so a unit normal will be
w 1
u= = √ (2, −1, 7).
|w| 54
The reflection matrix is then given by

2 
2 1 
H = I − 2uu T = I − ww T = I −

−1  2 −1 7 = · · ·
54 27 7
   
1 0 0 4 −2 14
1 
··· =  0 1 0 − −2 1 −7  ,
0 0 1 27 14 −7 49
so that  
23/27 2/27 −14/27
H= 2/27 26/27 7/27 
−14/27 7/27 −22/27
The reflection of v across W is then given by
    
23/27 2/27 −14/27 −1 39/27
Hv =  2/27 26/27 7/27   3  =  48/27 
−14/27 7/27 −22/27 −4 123/27

Reflections and Projections

Notice in Figure 2 that the projection of v into W is the midpoint of the vector v and
its reflection Hv = ReflW (v); that is,
1
Qv = (v + Hv) or, equivalently Hv = 2Qv − v,
2
where Q = QW denotes the projection onto W. (This is not the same as Qu in the
previous section, which was projection onto the normal of W.)
Let I denote the identity matrix, so that v = Iv for all vectors v ∈ Rn . The identities
above can now be expressed as matrix identities:
1
Q = ( I + H ) and H = 2Q − I,
2
So once you have computed the either the projection or reflection matrix for a sub-
space of Rn , the other is quite easy to obtain.
6

Exercises

1. Suppose that M is an n × n matrix such that M T = M = M2 . Let W denote the

column space of M.
(a) Suppose that Y ∈ W. (This means that Y = MX for some X.) Prove that MY = Y.
(b) Suppose that v is a vector in Rn . Why is Mv ∈ W?
(c) If Y ∈ W, why is v − Mv ⊥ Y?
(d) Conclude that Mv is the projection of v into W.
2. Compute the projection of the vector v = (1, 1, 0) onto the plane x + y − z = 0.
3. Compute the projection matrix Q for the subspace W of R4 spanned by the vectors
(1, 2, 0, 0) and (1, 0, 1, 1).
4. Compute the orthogonal projection of the vector z = (1, −2, 2, 2) onto the subspace
W of Problem 3. above. What does your answer tell you about the relationship
between the vector z and the subspace W?
5. Recall that a square matrix P is said to be an orthogonal matrix if P T P = I. Show that
Householder matrices are always orthogonal matrices; that is, show that H T H = I.
6. Compute the Householder matrix for reflection across the plane x + y − z = 0.
7. Compute the reflection of the vector v = (1, 1, 0) across the plane x + y − z = 0.
What happens when you add v to its reflection? How does this sum compare to your
answer from Exercise 2? Draw a sketch to explain this phenomenon.
8. Compute the reflection of the vector v = (1, 1) across the line ` in R2 spanned by
the vector (2, 3). Sketch the vector v, the line ` and the reflection of v across `. (Do
not confuse the spanning vector for ` with the normal vector to `.)
9. Compute the Householder matrix H for reflection across the hyperplane
x1 + 2x2 − x3 − 3x4 = 0 in R4 . Then compute the projection matrix Q for this hyper-
plane.
10. Compute the Householder matrix for reflection across the plane z = 0 in R3 .
Sketch the reflection involved. Your answer should not be too surprising!
 7

Selected Solutions to Exercises:

2. We describe two ways to solve this problem.
Solution 1: Pick a basis for the plane. Since the plane is 2-dimensional, any two
independent vectors in the plane will do, say, (1, −1, 0) and (0, 1, 1). Set
 
1 0  
T 2 − 1
A =  −1 1  and A A =
−1 2
0 1

The projection matrix Q for the plane is

   
1 0    2/3 −1/3 1/3
2/3 1/3 1 −1 0
Q = A ( A T A ) −1 A T =  − 1 1  =  −1/3 2/3 1/3 
1/3 2/3 0 1 1
0 1 1/3 1/3 2/3

We can now project any vector onto the plane by multiplying by Q:

    
2/3 −1/3 1/3 1 1/3
Projection(v) = Qv =  −1/3 2/3 1/3   1  =  1/3 
1/3 1/3 2/3 0 2/3

Solution 2: First, project v onto the normal vector n = (1, 1, −1) to the plane:
v·n
y = Projn (v) = n = (2/3, 2/3, −2/3).
n·n
Since y is the component of v orthogonal to the plane, the vector v − y is the or-
thogonal projection of v onto the plane. The solution (given in row vector notation)
is
v − y = (1, 1, 0) − (2/3, 2/3, −2/3) = (1/3, 1/3, 2/3),
as in the previous solution.
Note: Which method is better? The second way is shorter for hyperplanes (sub-
spaces of Rn having dimension n − 1), but finding the projection matrix Q is needed
if you are projecting from Rn to some intermediate dimension k, where you no longer
have a single normal vector to work with. For example, a 2-subspace in R4 has a
2-dimensional orthogonal complement as well, so one must compute a projection ma-
trix in order to project to either component of R4 , as in the next problem.
3. Set
 
1 1  
 2 0  T 5 1
A=
 0
 so that A A =
1  1 3
0 1
8

The projection matrix Q for the plane is

   6 4 4 4 
1 1 " # 14 14 14 14
3 1 4 12 2 2
− − 14 − 14
 
T −1 T
 2 0  14 14 1 2 0 0 14 14

Q = A( A A) A =  =
  
 0 1  −1 5 1 0 1 1 4 2
− 14 5 5 
14 14 14 14 14
 
0 1 4 2 5 5
14 − 14 14 14

5. Here is a hint: Use the fact that H = I − 2uu T , where I is the identity matrix and u
is a unit column vector. What is H T =? What is H T H =?
6. The vector v = (1, 1, −1) is normal to the plane x + y − z = 0, so the vector
u = √1 (1, 1, −1) = √1 v is a unit normal. Expressing u and v as column vectors we
3 3
find that  
1 
2
I − 2uu T = I − (2/3)vv T = I −  1  1 1 −1

3 −1
  1 2 2

3 −3 3
  
1 0 0 1 1 −1
2
=  0 1 0 −  1 1 −1  =  − 23 1 2 

3 −1 −1 3 3 
0 0 1 1 2 2 1
3 3 3