Chapter 2

Projection Matrices
2.1

Definition

Definition 2.1 Let x E n = V W . Then x can be uniquely decomposed

into
x = x1 + x2 (where x1 V and x2 W ).
The transformation that maps x into x1 is called the projection matrix (or
simply projector) onto V along W and is denoted as . This is a linear
transformation; that is,
(a1 y 1 + a2 y 2 ) = a1 (y 1 ) + a2 (y 2 )

(2.1)

for any y 1 , y 2 E n . This implies that it can be represented by a matrix.

This matrix is called a projection matrix and is denoted by P V W . The vector transformed by P V W (that is, x1 = P V W x) is called the projection (or
the projection vector) of x onto V along W .
Theorem 2.1 The necessary and sufficient condition for a square matrix
P of order n to be the projection matrix onto V = Sp(P ) along W = Ker(P )
is given by
P2 = P.
(2.2)
We need the following lemma to prove the theorem above.
Lemma 2.1 Let P be a square matrix of order n, and assume that (2.2)
holds. Then
E n = Sp(P ) Ker(P )
(2.3)
H. Yanai et al., Projection Matrices, Generalized Inverse Matrices, and Singular Value Decomposition,
Statistics for Social and Behavioral Sciences, DOI 10.1007/978-1-4419-9887-3_2,
Springer Science+Business Media, LLC 2011

25

26

CHAPTER 2. PROJECTION MATRICES

and
Ker(P ) = Sp(I n P ).

(2.4)

Proof of Lemma 2.1. (2.3): Let x Sp(P ) and y Ker(P ). From

x = P a, we have P x = P 2 a = P a = x and P y = 0. Hence, from
x+y = 0 P x+P y = 0, we obtain P x = x = 0 y = 0. Thus, Sp(P )
Ker(P ) = {0}. On the other hand, from dim(Sp(P )) + dim(Ker(P )) =
rank(P ) + (n rank(P )) = n, we have E n = Sp(P ) Ker(P ).
(2.4): We have P x = 0 x = (I n P )x Ker(P ) Sp(I n P ) on
the one hand and P (I n P ) Sp(I n P ) Ker(P ) on the other. Thus,
Q.E.D.
Ker(P ) = Sp(I n P ).
Note When (2.4) holds, P (I n P ) = O P 2 = P . Thus, (2.2) is the necessary
and sufficient condition for (2.4).

Proof of Theorem 2.1. (Necessity) For x E n , y = P x V . Noting

that y = y + 0, we obtain
P (P x) = P y = y = P x = P 2 x = P x = P 2 = P .
(Sufficiency) Let V = {y|y = P x, x E n } and W = {y|y = (I n
P )x, x E n }. From Lemma 2.1, V and W are disjoint. Then, an arbitrary
x E n can be uniquely decomposed into x = P x + (I n P )x = x1 + x2
(where x1 V and x2 W ). From Definition 2.1, P is the projection matrix
onto V = Sp(P ) along W = Ker(P ).
Q.E.D.
Let E n = V W , and let x = x1 + x2 , where x1 V and x2 W . Let
P W V denote the projector that transforms x into x2 . Then,
P V W x + P W V x = (P V W + P W V )x.

(2.5)

Because the equation above has to hold for any x E n , it must hold that
I n = P V W + P W V .
Let a square matrix P be the projection matrix onto V along W . Then,
Q = I n P satisfies Q2 = (I n P )2 = I n 2P + P 2 = I n P = Q,
indicating that Q is the projection matrix onto W along V . We also have
P Q = P (I n P ) = P P 2 = O,

(2.6)

2.1. DEFINITION

27

implying that Sp(Q) constitutes the null space of P (i.e., Sp(Q) = Ker(P )).
Similarly, QP = O, implying that Sp(P ) constitutes the null space of Q
(i.e., Sp(P ) = Ker(Q)).
Theorem 2.2 Let E n = V W . The necessary and sufficient conditions
for a square matrix P of order n to be the projection matrix onto V along
W are:
(i) P x = x for x V, (ii) P x = 0 for x W.

(2.7)

Proof. (Sufficiency) Let P V W and P W V denote the projection matrices

onto V along W and onto W along V , respectively. Premultiplying (2.5) by
P , we obtain P (P V W x) = P V W x, where P P W V x = 0 because of (i) and
(ii) above, and P V W x V and P W V x W . Since P x = P V W x holds
for any x, it must hold that P = P V W .
(Necessity) For any x V , we have x = x+0. Thus, P x = x. Similarly,
for any y W , we have y = 0+y, so that P y = 0.
Q.E.D.

Example 2.1 In Figure 2.1, OA indicates the projection of z onto Sp(x)

along Sp(y) (that is, OA= P Sp(x)Sp(y ) z), where P Sp(x)Sp(y ) indicates the

projection matrix onto Sp(x) along Sp(y). Clearly, OB= (I 2 P Sp(y )Sp(x) )
z.
Sp(y) = {y}
z

Sp(x) = {x}

O P {x}{y} z A

Figure 2.1: Projection onto Sp(x) = {x} along Sp(y) = {y}.

Example 2.2 In Figure 2.2, OA indicates the projection of z onto V =

{x|x = 1 x1 +2 x2 } along Sp(y) (that is, OA= P V Sp(y ) z), where P V Sp(y )
indicates the projection matrix onto V along Sp(y).

28

CHAPTER 2. PROJECTION MATRICES

Sp(y) = {y}
z

V = {1 x1 + 2 x2 }
O P V {y} z A
Figure 2.2: Projection onto a two-dimensional space V along Sp(y) = {y}.
Theorem 2.3 The necessary and sufficient condition for a square matrix
P of order n to be a projector onto V of dimensionality r (dim(V ) = r) is
given by
P = T r T 1 ,
(2.8)
where T is a square nonsingular matrix of order n and

1 0 0
. .
. . ... ... . . .
..

0 1 0

r =
0 0 0

. . .
.. . .
.
. .. .. . .
0 0 0

0
..
.
0
0
..
.

(There are r unities on the leading diagonals, 1 r n.)

Proof. (Necessity) Let E n = V W , and let A = [a1 , a2 , , ar ] and
B = [b1 , b2 , bnr ] be matrices of linearly independent basis vectors spanning V and W , respectively. Let T = [A, B]. Then T is nonsingular,
since rank(A) + rank(B) = rank(T ). Hence, x V and y W can be
expressed as

=T
,
x = A = [A, B]
0
0

y = A = [A, B]

=T

2.1. DEFINITION

29

Thus, we obtain

P x = x = P T

P y = 0 = P T

=T

0
0

= T r

= T r

Adding the two equations above, we obtain

PT

Since

= T r

is an arbitrary vector in the n-dimensional space E n , it follows

that
P T = T r = P = T r T 1 .
Furthermore, T can be an arbitrary nonsingular matrix since V = Sp(A)
and W = Sp(B) such that E n = V W can be chosen arbitrarily.
(Sufficiency) P is a projection matrix, since P 2 = P , and rank(P ) = r
from Theorem 2.1. (Theorem 2.2 can also be used to prove the theorem
above.)
Q.E.D.
Lemma 2.2 Let P be a projection matrix. Then,
rank(P ) = tr(P ).

(2.9)

Proof. rank(P ) = rank(T r T 1 ) = rank(r ) = tr(T T 1 ) = tr(P ).

Q.E.D.
The following theorem holds.
Theorem 2.4 Let P be a square matrix of order n. Then the following
three statements are equivalent.
P2 = P,

(2.10)

rank(P ) + rank(I n P ) = n,

(2.11)

E n = Sp(P ) Sp(I n P ).

(2.12)

Proof. (2.10) (2.11): It is clear from rank(P ) = tr(P ).

30

CHAPTER 2. PROJECTION MATRICES

(2.11) (2.12): Let V = Sp(P ) and W = Sp(I n P ). Then, dim(V +

W ) = dim(V ) + dim(W ) dim(V W ). Since x = P x + (I n P )x
for an arbitrary n-component vector x, we have E n = V + W . Hence,
dim(V W ) = 0 = V W = {0}, establishing (2.12).
(2.12) (2.10): Postmultiplying I n = P + (I n P ) by P , we obtain
P = P 2 + (I n P )P , which implies P (I n P ) = (I n P )P . On the other
hand, we have P (I n P ) = O and (I n P )P = O because Sp(P (I n P ))
Sp(P ) and Sp((I n P )P ) Sp(I n P ).
Q.E.D.
Corollary
P 2 = P Ker(P ) = Sp(I n P ).
Proof. (): It is clear from Lemma 2.1.
(): Ker(P ) = Sp(I n P ) P (I n P ) = O P 2 = P .

2.2

(2.13)
Q.E.D.

Orthogonal Projection Matrices

Suppose we specify a subspace V in E n . There are in general infinitely many

ways to choose its complement subspace V c = W . We will discuss some of
them in Chapter 4. In this section, we consider the case in which V and W
are orthogonal, that is, W = V .
Let x, y E n , and let x and y be decomposed as x = x1 + x2 and
y = y 1 +y 2 , where x1 , y 1 V and x2 , y 2 W . Let P denote the projection
matrix onto V along V . Then, x1 = P x and y 1 = P y. Since (x2 , P y) =
(y 2 , P x) = 0, it must hold that
(x, P y) = (P x + x2 , P y) = (P x, P y)
= (P x, P y + y 2 ) = (P x, y) = (x, P 0 y)
for any x and y, implying
P0 = P.

(2.14)

Theorem 2.5 The necessary and sufficient condition for a square matrix
P of order n to be an orthogonal projection matrix (an orthogonal projector)
is given by
(i) P 2 = P and (ii) P 0 = P .
Proof. (Necessity) That P 2 = P is clear from the definition of a projection
matrix. That P 0 = P is as shown above.
(Sufficiency) Let x = P Sp(P ). Then, P x = P 2 = P = x. Let
y Sp(P ) . Then, P y = 0 since (P x, y) = x0 P 0 y = x0 P y = 0 must

2.2. ORTHOGONAL PROJECTION MATRICES

31

hold for an arbitrary x. From Theorem 2.2, P is the projection matrix

onto Sp(P ) along Sp(P ) ; that is, the orthogonal projection matrix onto
Sp(P ).
Q.E.D.
Definition 2.2 A projection matrix P such that P 2 = P and P 0 = P is
called an orthogonal projection matrix (projector). Furthermore, the vector
P x is called the orthogonal projection of x. The orthogonal projector P
is in fact the projection matrix onto Sp(P ) along Sp(P ) , but it is usually
referred to as the orthogonal projector onto Sp(P ). See Figure 2.3.

Qy (where Q = I P )

Sp(A) = Sp(Q)
Sp(A) = Sp(P )

Py
Figure 2.3: Orthogonal projection.
Note A projection matrix that does not satisfy P 0 = P is called an oblique projector as opposed to an orthogonal projector.

Theorem 2.6 Let A = [a1 , a2 , , am ], where a1 , a2 , , am are linearly

independent. Then the orthogonal projector onto V = Sp(A) spanned by
a1 , a2 , , am is given by
P = A(A0 A)1 A0 .

(2.15)

Proof. Let x1 Sp(A). From x1 = A, we obtain P x1 = x1 = A =

A(A0 A)1 A0 x1 . On the other hand, let x2 Sp(A) . Then, A0 x2 = 0 =
A(A0 A)1 A0 x2 = 0. Let x = x1 + x2 . From P x2 = 0, we obtain P x =
A(A0 A)1 A0 x, and (2.15) follows because x is arbitrary.

32

CHAPTER 2. PROJECTION MATRICES

Let Q = I n P . Then Q is the orthogonal projector onto Sp(A) , the

ortho-complement subspace of Sp(A).
Example 2.3 Let 1n = (1, 1, , 1)0 (the vector with n ones). Let P M
denote the orthogonal projector onto VM = Sp(1n ). Then,
1

..
.

P M = 1n (10n 1n )1 10n = ...

1
n

1
n

.. .
.

(2.16)

1
n

= Sp(1 ) , the ortho-complement subThe orthogonal projector onto VM

n
space of Sp(1n ), is given by

In P M =

1
n1
..
.
n1

1
n

n1
1
..
.

1
n

n1

n1

n1
..
..
.
.
1 n1

(2.17)

Let
QM = I n P M .

(2.18)

Clearly, P M and QM are both symmetric, and the following relation holds:
P 2M = P M , Q2M = QM , and P M QM = QM P M = O.

(2.19)

Note The matrix QM in (2.18) is sometimes written as P

M.

Example 2.4 Let

xR =

x1
x2
..
.
xn

, x =

x1 x

x2 x
..
.

1X
, where x

=
xj .

n j=1

xn x

Then,
x = QM xR ,
and so

n
X

(xj x
)2 = ||x||2 = x0 x = x0R QM xR .

j=1

The proof is omitted.

(2.20)

2.3. SUBSPACES AND PROJECTION MATRICES

2.3

33

Subspaces and Projection Matrices

In this section, we consider the relationships between subspaces and projectors when the n-dimensional space E n is decomposed into the sum of several
subspaces.

2.3.1

Decomposition into a direct-sum of disjoint subspaces

Lemma 2.3 When there exist two distinct ways of decomposing E n ,

E n = V1 W1 = V2 W2 ,

(2.21)

and if V1 W2 or V2 W1 , the following relation holds:

E n = (V1 V2 ) (W1 W2 ).

(2.22)

Proof. When V1 W2 , Theorem 1.5 leads to the following relation:

V1 + (W1 W2 ) = (V1 + W1 ) W2 = E n W2 = W2 .
Also from V1 (W1 W2 ) = (V1 W1 ) W2 = {0}, we have W2 = V1
(W1 W2 ). Hence the following relation holds:
E n = V2 W2 = V2 V1 (W1 W2 ) = (V1 V2 ) (W1 W2 ).
When V2 W2 , the same result follows by using W1 = V2 (W1
W2 ).
Q.E.D.
Corollary When V1 V2 or W2 W1 ,
E n = (V1 W2 ) (V2 W1 ).
Proof. In the proof of Lemma 2.3, exchange the roles of W2 and V2 .

(2.23)
Q.E.D.

Theorem 2.7 Let P 1 and P 2 denote the projection matrices onto V1 along
W1 and onto V2 along W2 , respectively. Then the following three statements
are equivalent:
(i) P 1 + P 2 is the projector onto V1 V2 along W1 W2 .
(ii) P 1 P 2 = P 2 P 1 = O.
(iii) V1 W2 and V2 W1 . (In this case, V1 and V2 are disjoint spaces.)
Proof. (i) (ii): From (P 1 +P 2 )2 = P 1 +P 2 , P 21 = P 1 , and P 22 = P 2 , we

34

CHAPTER 2. PROJECTION MATRICES

have P 1 P 2 = P 2 P 1 . Pre- and postmutiplying both sides by P 1 , we obtain

P 1 P 2 = P 1 P 2 P 1 and P 1 P 2 P 1 = P 2 P 1 , respectively, which imply
P 1 P 2 = P 2 P 1 . This and P 1 P 2 = P 2 P 1 lead to P 1 P 2 = P 2 P 1 = O.
(ii) (iii): For an arbitrary vector x V1 , P 1 x = x because P 1 x V1 .
Hence, P 2 P 1 x = P 2 x = 0, which implies x W2 , and so V1 W2 . On
the other hand, when x V2 , it follows that P 2 x V2 , and so P 1 P 2 x =
P 1 x = 0, implying x W2 . We thus have V2 W2 .
(iii) (ii): For x E n , P 1 x V1 , which implies (I n P 2 )P 1 x = P 1 x,
which holds for any x. Thus, (I n P 2 )P 1 = P 1 , implying P 1 P 2 = O.
We also have x E n P 2 x V2 (I n P 1 )P 2 x = P 2 x, which again
holds for any x, which implies (I n P 1 )P 2 = P 2 P 1 P 2 = O. Similarly,
P 2 P 1 = O.
(ii) (i): An arbitrary vector x (V1 V2 ) can be decomposed into
x = x1 + x2 , where x1 V1 and x2 V2 . From P 1 x2 = P 1 P 2 x = 0
and P 2 x1 = P 2 P 1 x = 0, we have (P 1 + P 2 )x = (P 1 + P 2 )(x1 + x2 ) =
P 1 x1 + P 2 x2 = x1 + x2 = x. On the other hand, by noting that P 1 =
P 1 (I n P 2 ) and P 2 = P 2 (I n P 1 ) for any x (W1 W2 ), we have
(P 1 + P 2 )x = P 1 (I n P 2 )x + P 2 (I n P 1 )x = 0. Since V1 W2 and
V2 W1 , the decomposition on the right-hand side of (2.22) holds. Hence,
we know P 1 + P 2 is the projector onto V1 V2 along W1 W2 by Theorem
2.2.
Q.E.D.
Note In the theorem above, P 1 P 2 = O in (ii) does not imply P 2 P 1 = O.
P 1 P 2 = O corresponds with V2 W1 , and P 2 P 1 = O with V1 W2 in (iii). It
should be clear that V1 W2 V2 W1 does not hold.

Theorem 2.8 Given the decompositions of E n in (2.21), the following three

statements are equivalent:
(i) P 2 P 1 is the projector onto V2 W1 along V1 W2 .
(ii) P 1 P 2 = P 2 P 1 = P 1 .
(iii) V1 V2 and W2 W1 .
Proof. (i) (ii): (P 2 P 1 )2 = P 2 P 1 implies 2P 1 = P 1 P 2 + P 2 P 1 .
Pre- and postmultiplying both sides by P 2 , we obtain P 2 P 1 = P 2 P 1 P 2
and P 1 P 2 = P 2 P 1 P 2 , respectively, which imply P 1 P 2 = P 2 P 1 = P 1 .
(ii) (iii): For x E n , P 1 x V1 , which implies P 1 x = P 2 P 1 x V2 ,
which in turn implies V1 V2 . Let Qj = I n P j (j = 1, 2). Then,
P 1 P 2 = P 1 implies Q1 Q2 = Q2 , and so Q2 x W2 , which implies Q2 x =
Q1 Q2 x W1 , which in turn implies W2 W1 .

2.3. SUBSPACES AND PROJECTION MATRICES

35

(iii) (ii): From V1 V2 , for x E n , P 1 x V1 V2 P 2 (P 1 x) =

P 1 x P 2 P 1 = P 1 . On the other hand, from W2 W1 , Q2 x W2 W1
for x E n Q1 Q2 x = Q2 x Q1 Q2 Q2 (I n P 1 )(I n P 2 ) =
(I n P 2 ) P 1 P 2 = P 1 .
(ii) (i): For x (V2 W1 ), it holds that (P 2 P 1 )x = Q1 P 2 x =
Q1 x = x. On the other hand, let x = y + z, where y V1 and z W2 .
Then, (P 2 P 1 )x = (P 2 P 1 )y + (P 2 P 1 )z = P 2 Q1 y + Q1 P 2 z = 0.
Q.E.D.
Hence, P 2 P 1 is the projector onto V2 W1 along V1 W2 .
Note As in Theorem 2.7, P 1 P 2 = P 1 does not necessarily imply P 2 P 1 = P 1 .
Note that P 1 P 2 = P 1 W2 W1 , and P 2 P 1 = P 1 V1 V2 .

Theorem 2.9 When the decompositions in (2.21) and (2.22) hold, and if
P 1P 2 = P 2P 1,

(2.24)

then P 1 P 2 (or P 2 P 1 ) is the projector onto V1 V2 along W1 + W2 .

Proof. P 1 P 2 = P 2 P 1 implies (P 1 P 2 )2 = P 1 P 2 P 1 P 2 = P 21 P 22 = P 1 P 2 ,
indicating that P 1 P 2 is a projection matrix. On the other hand, let x
V1 V2 . Then, P 1 (P 2 x) = P 1 x = x. Furthermore, let x W1 + W2
and x = x1 + x2 , where x1 W1 and x2 W2 . Then, P 1 P 2 x =
P 1 P 2 x1 +P 1 P 2 x2 = P 2 P 1 x1 +0 = 0. Since E n = (V1 V2 )(W1 W2 ) by
the corollary to Lemma 2.3, we know that P 1 P 2 is the projector onto V1 V2
along W1 W2 .
Q.E.D.
Note Using the theorem above, (ii) (i) in Theorem 2.7 can also be proved as
follows: From P 1 P 2 = O
Q1 Q2 = (I n P 1 )(I n P 2 ) = I n P 1 P 2 = Q2 Q1 .
Hence, Q1 Q2 is the projector onto W1 W2 along V1 V2 , and P 1 +P 2 = I n Q1 Q2
is the projector onto V1 V2 along W1 W2 .

If we take W1 = V1 and W2 = V2 in the theorem above, P 1 and P 2

become orthogonal projectors.

36

CHAPTER 2. PROJECTION MATRICES

Theorem 2.10 Let P 1 and P 2 be the orthogonal projectors onto V1 and

V2 , respectively. Then the following three statements are equivalent:

(i) P 1 + P 2 is the orthogonal projector onto V1 V2 .

(ii) P 1 P 2 = P 2 P 1 = O.
(iii) V1 and V2 are orthogonal.
Theorem 2.11 The following three statements are equivalent:
(i) P 2 P 1 is the orthogonal projector onto V2 V1 .
(ii) P 1 P 2 = P 2 P 1 = P 1 .
(iii) V1 V2 .

V2

The two theorems above can be proved by setting W1 = V1 and W2 =

in Theorems 2.7 and 2.8.

Theorem 2.12 The necessary and sufficient condition for P 1 P 2 to be the

orthogonal projector onto V1 V2 is (2.24).
Proof. Sufficiency is clear from Theorem 2.9. Necessity follows from P 1 P 2
= (P 1 P 2 )0 , which implies P 1 P 2 = P 2 P 1 since P 1 P 2 is an orthogonal projector.
Q.E.D.
We next present a theorem concerning projection matrices when E n is
expressed as a direct-sum of m subspaces, namely
E n = V1 V2 Vm .

(2.25)

Theorem 2.13 Let P i (i = 1, , m) be square matrices that satisfy

P 1 + P 2 + + P m = I n.

(2.26)

Then the following three statements are equivalent:

P i P j = O (i 6= j).

(2.27)

P 2i = P i (i = 1, m).

(2.28)

rank(P 1 ) + rank(P 2 ) + + rank(P m ) = n.

(2.29)

2.3. SUBSPACES AND PROJECTION MATRICES

37

Proof. (i) (ii): Multiply (2.26) by P i .

(ii) (iii): Use rank(P i ) = tr(P i ) when P 2i = P i . Then,
m
X
i=1

rank(P i ) =

m
X

tr(P i ) = tr

i=1

m
X

Pi

= tr(I n ) = n.

i=1

(iii) (i), (ii): Let Vi = Sp(P i ). From rank(P i ) = dim(Vi ), we obtain

dim(V1 ) + dim(V2 ) + dim(Vm ) = n; that is, E n is decomposed into the
sum of m disjoint subspaces as in (2.26). By postmultiplying (2.26) by P i ,
we obtain
P 1 P i + P 2 P i + + P i (P i I n ) + + P m P i = O.
Since Sp(P 1 ), Sp(P 2 ), , Sp(P m ) are disjoint, (2.27) and (2.28) hold from
Theorem 1.4.
Q.E.D.
Note P i in Theorem 2.13 is a projection matrix. Let E n = V1 Vr , and let
V(i) = V1 Vi1 Vi+1 Vr .

(2.30)

Then, E n = Vi V(i) . Let P i(i) denote the projector onto Vi along V(i) . This matrix
coincides with the P i that satisfies the four equations given in (2.26) through (2.29).

The following relations hold.

Corollary 1
P 1(1) + P 2(2) + + P m(m) = I n ,

(2.31)

P 2i(i) = P i(i) (i = 1, , m),

(2.32)

P i(i) P j(j) = O (i 6= j).

(2.33)

Corollary 2 Let P (i)i denote the projector onto V(i) along Vi . Then the
following relation holds:
P (i)i = P 1(1) + + P i1(i1) + P i+1(i+1) + + P m(m) .

(2.34)

Proof. The proof is straightforward by noting P i(i) +P (i)i = I n .

Q.E.D.

38

CHAPTER 2. PROJECTION MATRICES

Note The projection matrix P i(i) onto Vi along V(i) is uniquely determined. Assume that there are two possible representations, P i(i) and P i(i) . Then,
P 1(1) + P 2(2) + + P m(m) = P 1(1) + P 2(2) + + P m(m) ,
from which
(P 1(1) P 1(1) ) + (P 2(2) P 2(2) ) + + (P m(m) P m(m) ) = O.
Each term in the equation above belongs to one of the respective subspaces V1 , V2 ,
, Vm , which are mutually disjoint. Hence, from Theorem 1.4, we obtain P i(i) =
P i(i) . This indicates that when a direct-sum of E n is given, an identity matrix I n
of order n is decomposed accordingly, and the projection matrices that constitute
the decomposition are uniquely determined.

The following theorem due to Khatri (1968) generalizes Theorem 2.13.

Theorem 2.14 Let P i denote a square matrix of order n such that
P = P 1 + P 2 + + P m.

(2.35)

Consider the following four propositions:

(i) P 2i = P i
(ii) P i P j = O

(i = 1, m),
(i 6= j), and rank(P 2i ) = rank(P i ),

(iii) P 2 = P ,
(iv) rank(P ) = rank(P 1 ) + + rank(P m ).
All other propositions can be derived from any two of (i), (ii), and (iii), and
(i) and (ii) can be derived from (iii) and (iv).
Proof. That (i) and (ii) imply (iii) is obvious. To show that (ii) and (iii)
imply (iv), we may use
P 2 = P 21 + P 22 + + P 2m and P 2 = P ,
which follow from (2.35).
(ii), (iii) (i): Postmultiplying (2.35) by P i , we obtain P P i = P 2i , from
which it follows that P 3i = P 2i . On the other hand, rank(P 2i ) = rank(P i )
implies that there exists W such that P 2i W i = P i . Hence, P 3i = P 2i
P 3i W i = P 2i W i P i (P 2i W i ) = P 2i W P 2i = P i .

2.3. SUBSPACES AND PROJECTION MATRICES

39

(iii), (iv) (i), (ii): We have Sp(P ) Sp(I n P ) = E n from P 2 = P .

Hence, by postmultiplying the identity
P 1 + P 2 + + P m + (I n P ) = I n
by P , we obtain P 2i = P i , and P i P j = O (i 6= j).

Q.E.D.

Next we consider the case in which subspaces have inclusion relationships

like the following.
Theorem 2.15 Let
E n = Vk Vk1 V2 V1 = {0},
and let Wi denote a complement subspace of Vi . Let P i be the orthogonal
projector onto Vi along Wi , and let P i = P i P i1 , where P 0 = O and
P k = I n . Then the following relations hold:
(i) I n = P 1 + P 2 + + P k .
(ii) (P i )2 = P i .
(iii) P i P j = P j P i = O (i 6= j).
(iv) P i is the projector onto Vi Wi1 along Vi1 Wi .
Proof. (i): Obvious. (ii): Use P i P i1 = P i1 P i = P i1 . (iii): It
follows from (P i )2 = P i that rank(P i ) = tr(P i ) = tr(P i P i1 ) =
P
tr(P i ) tr(P i1 ). Hence, ki=1 rank(P i ) = tr(P k ) tr(P 0 ) = n, from
which P i P j = O follows by Theorem 2.13. (iv): Clear from Theorem
2.8(i).
Q.E.D.
Note The theorem above does not presuppose that P i is an orthogonal projector. However, if Wi = Vi , P i and P i are orthogonal projectors. The latter, in

particular, is the orthogonal projector onto Vi Vi1

.

2.3.2

Decomposition into nondisjoint subspaces

In this section, we present several theorems indicating how projectors are

decomposed when the corresponding subspaces are not necessarily disjoint.
We elucidate their meaning in connection with the commutativity of projectors.

40

CHAPTER 2. PROJECTION MATRICES

We first consider the case in which there are two direct-sum decompositions of E n , namely
E n = V1 W1 = V2 W2 ,
as given in (2.21). Let V12 = V1 V2 denote the product space between V1
and V2 , and let V3 denote a complement subspace to V1 + V2 in E n . Furthermore, let P 1+2 denote the projection matrix onto V1+2 = V1 + V2 along
V3 , and let P j (j = 1, 2) represent the projection matrix onto Vj (j = 1, 2)
along Wj (j = 1, 2). Then the following theorem holds.
Theorem 2.16 (i) The necessary and sufficient condition for P 1+2 = P 1 +
P 2 P 1 P 2 is
(V1+2 W2 ) (V1 V3 ).
(2.36)
(ii) The necessary and sufficient condition for P 1+2 = P 1 + P 2 P 2 P 1 is
(V1+2 W1 ) (V2 V3 ).

(2.37)

Proof. (i): Since V1+2 V1 and V1+2 V2 , P 1+2 P 1 is the projector

onto V1+2 W1 along V1 V3 by Theorem 2.8. Hence, P 1+2 P 1 = P 1 and
P 1+2 P 2 = P 2 . Similarly, P 1+2 P 2 is the projector onto V1+2 W2 along
V2 V3 . Hence, by Theorem 2.8,
P 1+2 P 1 P 2 + P 1 P 2 = O (P 1+2 P 1 )(P 1+2 P 2 ) = O.
Furthermore,
(P 1+2 P 1 )(P 1+2 P 2 ) = O (V1+2 W2 ) (V1 V3 ).
(ii): Similarly, P 1+2 P 1 P 2 + P 2 P 1 = O (P 1+2 P 2 )(P 1+2
Q.E.D.
P 1 ) = O (V1+2 W1 ) (V2 V3 ).
Corollary Assume that the decomposition (2.21) holds. The necessary and
sufficient condition for P 1 P 2 = P 2 P 1 is that both (2.36) and (2.37) hold.
The following theorem can readily be derived from the theorem above.
Theorem 2.17 Let E n = (V1 +V2 )V3 , V1 = V11 V12 , and V2 = V22 V12 ,
where V12 = V1 V2 . Let P 1+2 denote the projection matrix onto V1 + V2
along V3 , and let P 1 and P 2 denote the projectors onto V1 along V3 V22
and onto V2 along V3 V11 , respectively. Then,
P 1 P 2 = P 2 P 1

(2.38)

2.3. SUBSPACES AND PROJECTION MATRICES

41

and
P 1+2 = P 1 + P 2 P 1 P 2 .

(2.39)

Proof. Since V11 V1 and V22 V2 , we obtain

V1+2 W2 = V11 (V1 V3 ) and V1+2 W1 = V22 (V2 V3 )
by setting W1 = V22 V3 and W2 = V11 V3 in Theorem 2.16.
Another proof. Let y = y 1 + y 2 + y 12 + y 3 E n , where y 1 V11 ,
y 2 V22 , y 12 V12 , and y 3 V3 . Then it suffices to show that (P 1 P 2 )y =
(P 2 P 1 )y.
Q.E.D.
Let P j (j = 1, 2) denote the projection matrix onto Vj along Wj . Assume that E n = V1 W1 V3 = V2 W2 V3 and V1 + V2 = V11 V22 V12
hold. However, W1 = V22 may not hold, even if V1 = V11 V12 . That is,
(2.38) and (2.39) hold only when we set W1 = V22 and W2 = V11 .
Theorem 2.18 Let P 1 and P 2 be the orthogonal projectors onto V1 and
V2 , respectively, and let P 1+2 denote the orthogonal projector onto V1+2 .
Let V12 = V1 V2 . Then the following three statements are equivalent:
(i) P 1 P 2 = P 2 P 1 .
(ii) P 1+2 = P 1 + P 2 P 1 P 2 .
and V

(iii) V11 = V1 V12

22 = V2 V12 are orthogonal.

Proof. (i) (ii): Obvious from Theorem 2.16.

(ii) (iii): P 1+2 = P 1 + P 2 P 1 P 2 (P 1+2 P 1 )(P 1+2 P 2 ) =
(P 1+2 P 2 )(P 1+2 P 1 ) = O V11 and V22 are orthogonal.
(iii) (i): Set V3 = (V1 + V2 ) in Theorem 2.17. Since V11 and V22 , and
V1 and V22 , are orthogonal, the result follows.
Q.E.D.
When P 1 , P 2 , and P 1+2 are orthogonal projectors, the following corollary holds.
Corollary P 1+2 = P 1 + P 2 P 1 P 2 P 1 P 2 = P 2 P 1 .

2.3.3

Commutative projectors

In this section, we focus on orthogonal projectors and discuss the meaning

of Theorem 2.18 and its corollary. We also generalize the results to the case
in which there are three or more subspaces.

42

CHAPTER 2. PROJECTION MATRICES

Theorem 2.19 Let P j denote the orthogonal projector onto Vj . If P 1 P 2 =

P 2 P 1 , P 1 P 3 = P 3 P 1 , and P 2 P 3 = P 3 P 2 , the following relations hold:
V1 + (V2 V3 ) = (V1 + V2 ) (V1 + V3 ),

(2.40)

V2 + (V1 V3 ) = (V1 + V2 ) (V2 + V3 ),

(2.41)

V3 + (V1 V2 ) = (V1 + V3 ) (V2 + V3 ).

(2.42)

Proof. Let P 1+(23) denote the orthogonal projector onto V1 + (V2 V3 ).

Then the orthogonal projector onto V2 V3 is given by P 2 P 3 (or by P 3 P 2 ).
Since P 1 P 2 = P 2 P 1 P 1 P 2 P 3 = P 2 P 3 P 1 , we obtain
P 1+(23) = P 1 + P 2 P 3 P 1 P 2 P 3
by Theorem 2.18. On the other hand, from P 1 P 2 = P 2 P 1 and P 1 P 3 =
P 3 P 1 , the orthogonal projectors onto V1 + V2 and V1 + V3 are given by
P 1+2 = P 1 + P 2 P 1 P 2 and P 1+3 = P 1 + P 3 P 1 P 3 ,
respectively, and so P 1+2 P 1+3 = P 1+3 P 1+2 holds. Hence, the orthogonal
projector onto (V1 + V2 ) (V1 + V3 ) is given by
(P 1 + P 2 P 1 P 2 )(P 1 + P 3 P 1 P 3 ) = P 1 + P 2 P 3 P 1 P 2 P 3 ,
which implies P 1+(23) = P 1+2 P 1+3 . Since there is a one-to-one correspondence between projectors and subspaces, (2.40) holds.
Relations (2.41) and (2.42) can be similarly proven by noting that (P 1 +
P 2 P 1 P 2 )(P 2 + P 3 P 2 P 3 ) = P 2 + P 1 P 3 P 1 P 2 P 3 and (P 1 + P 3
P 1 P 3 )(P 2 + P 3 P 2 P 3 ) = P 3 + P 1 P 2 P 1 P 2 P 3 , respectively.
Q.E.D.
The three identities from (2.40) to (2.42) indicate the distributive law of
subspaces, which holds only if the commutativity of orthogonal projectors
holds.
We now present a theorem on the decomposition of the orthogonal projectors defined on the sum space V1 + V2 + V3 of V1 , V2 , and V3 .
Theorem 2.20 Let P 1+2+3 denote the orthogonal projector onto V1 + V2 +
V3 , and let P 1 , P 2 , and P 3 denote the orthogonal projectors onto V1 , V2 ,
and V3 , respectively. Then a sufficient condition for the decomposition
P 1+2+3 = P 1 + P 2 + P 3 P 1 P 2 P 2 P 3 P 3 P 1 + P 1 P 2 P 3

(2.43)

2.3. SUBSPACES AND PROJECTION MATRICES

43

to hold is
P 1 P 2 = P 2 P 1 , P 2 P 3 = P 3 P 2 , and P 1 P 3 = P 3 P 1 .

(2.44)

Proof. P 1 P 2 = P 2 P 1 P 1+2 = P 1 + P 2 P 1 P 2 and P 2 P 3 = P 3 P 2

P 2+3 = P 2 + P 3 P 2 P 3 . We therefore have P 1+2 P 2+3 = P 2+3 P 1+2 . We
also have P 1+2+3 = P (1+2)+(1+3) , from which it follows that
P 1+2+3 = P (1+2)+(1+3) = P 1+2 + P 1+3 P 1+2 P 1+3
= (P 1 + P 2 P 1 P 2 ) + (P 1 + P 3 P 1 P 3 )
(P 2 P 3 + P 1 P 1 P 2 P 3 )
= P 1 + P 2 + P 3 P 1P 2 P 2P 3 P 1P 3 + P 1P 2P 3.
An alternative proof. From P 1 P 2+3 = P 2+3 P 1 , we have P 1+2+3 = P 1 +
P 2+3 P 1 P 2+3 . If we substitute P 2+3 = P 2 +P 3 P 2 P 3 into this equation,
we obtain (2.43).
Q.E.D.
Assume that (2.44) holds, and let
P 1 = P 1 P 1 P 2 P 1 P 3 + P 1 P 2 P 3 ,
P 2 = P 2 P 2 P 3 P 1 P 2 + P 1 P 2 P 3 ,
P 3 = P 3 P 1 P 3 P 2 P 3 + P 1 P 2 P 3 ,
P 12(3) = P 1 P 2 P 1 P 2 P 3 ,
P13(2) = P 1 P 3 P 1 P 2 P 3 ,
P 23(1) = P 2 P 3 P 1 P 2 P 3 ,
and
P 123 = P 1 P 2 P 3 .
Then,
P 1+2+3 = P 1 + P 2 + P 3 + P 12(3) + P 13(2) + P 23(1) + P 123 .

(2.45)

Additionally, all matrices on the right-hand side of (2.45) are orthogonal

projectors, which are also all mutually orthogonal.
Note Since P 1 = P 1 (I n P 2+3 ), P 2 = P 2 (I n P 1+3 ), P 3 = P 3 (I P 1+2 ),
P 12(3) = P 1 P 2 (I n P 3 ), P 13(2) = P 1 P 3 (I n P 2 ), and P 23(1) = P 2 P 3 (I n P 1 ),

44

CHAPTER 2. PROJECTION MATRICES

the decomposition of the projector P 123 corresponds with the decomposition of

the subspace V1 + V2 + V3

V1 + V2 + V3 = V1 V2 V3 V12(3) V13(2) V23(1) V123 ,

(2.46)

where V1 = V1 (V2 + V3 ) , V2 = V2 (V1 + V3 ) , V3 = V3 (V1 + V2 ) ,

V12(3) = V1 V2 V3 , V13(2) = V1 V2 V3 , V23(1) = V1 V2 V3 , and
V123 = V1 V2 V3 .

Theorem 2.20 can be generalized as follows.

Corollary Let V = V1 +V2 + +Vs (s 2). Let P V denote the orthogonal
projector onto V , and let P j denote the orthogonal projector onto Vj . A
sufficient condition for
PV =

s
X

Pj

j=1

X
i<j

P iP j +

P i P j P k + + (1)s1 P 1 P 2 P 3 P s

i<j<k

(2.47)
to hold is
P i P j = P j P i (i 6= j).

2.3.4

(2.48)

Noncommutative projectors

We now consider the case in which two subspaces V1 and V2 and the corresponding projectors P 1 and P 2 are given but P 1 P 2 = P 2 P 1 does not
necessarily hold. Let Qj = I n P j (j = 1, 2). Then the following lemma
holds.
Lemma 2.4
V1 + V2 = Sp(P 1 ) Sp(Q1 P 2 )

(2.49)

= Sp(Q2 P 1 ) Sp(P 2 ).

(2.50)

Proof. [P 1 , Q1 P 2 ] and [Q2 P 1 , P 2 ] can be expressed as

"

[P 1 , Q1 P 2 ] = [P 1 , P 2 ]
and

"

[Q2 P 1 , P 2 ] = [P 1 , P 2 ]

I n P 2
O
In
In
O
P 1 I n

= [P 1 , P 2 ]S
#

= [P 1 , P 2 ]T .

2.3. SUBSPACES AND PROJECTION MATRICES

45

Since S and T are nonsingular, we have

rank(P 1 , P 2 ) = rank(P 1 , Q1 P 2 ) = rank(Q2 P 1 , P 1 ),
which implies
V1 + V2 = Sp(P 1 , Q1 P 2 ) = Sp(Q2 P 1 , P 2 ).
Furthermore, let P 1 x + Q1 P 2 y = 0. Premultiplying both sides by P 1 ,
we obtain P 1 x = 0 (since P 1 Q1 = O), which implies Q1 P 2 y = 0. Hence,
Sp(P 1 ) and Sp(Q1 P 2 ) give a direct-sum decomposition of V1 + V2 , and so
Q.E.D.
do Sp(Q2 P 1 ) and Sp(P 2 ).
The following theorem follows from Lemma 2.4.
Theorem 2.21 Let E n = (V1 + V2 ) W . Furthermore, let
V2[1] = {x|x = Q1 y, y V2 }

(2.51)

V1[2] = {x|x = Q2 y, y V1 }.

(2.52)

and
Let Qj = I n P j (j = 1, 2), where P j is the orthogonal projector onto
Vj , and let P , P 1 , P 2 , P 1[2] , and P 2[1] denote the projectors onto V1 + V2
along W , onto V1 along V2[1] W , onto V2 along V1[2] W , onto V1[2] along
V2 W , and onto V2[1] along V1 W , respectively. Then,
P = P 1 + P 2[1]

(2.53)

P = P 1[2] + P 2

(2.54)

or
holds.
Note When W = (V1 + V2 ) , P j is the orthogonal projector onto Vj , while P j[i]
is the orthogonal projector onto Vj [i].

Corollary Let P denote the orthogonal projector onto V = V1 V2 , and

let P j (j = 1, 2) be the orthogonal projectors onto Vj . If Vi and Vj are
orthogonal, the following equation holds:
P = P 1 + P 2.

(2.55)

46

2.4

CHAPTER 2. PROJECTION MATRICES

Norm of Projection Vectors

We now present theorems concerning the norm of the projection vector P x

(x E n ) obtained by projecting x onto Sp(P ) along Ker(P ) by P .
Lemma 2.5 P 0 = P and P 2 = P P 0 P = P .
(The proof is trivial and hence omitted.)
Theorem 2.22 Let P denote a projection matrix (i.e., P 2 = P ). The
necessary and sufficient condition to have
||P x|| ||x||

(2.56)

P0 = P.

(2.57)

for an arbitrary vector x is

Proof. (Sufficiency) Let x be decomposed as x = P x + (I n P )x. We
have (P x)0 (I n P )x = x0 (P 0 P 0 P )x = 0 because P 0 = P P 0 P = P 0
from Lemma 2.5. Hence,
||x||2 = ||P x||2 + ||(I n P )x||2 ||P x||2 .
(Necessity) By assumption, we have x0 (I n P 0 P )x 0, which implies
I n P 0 P is nnd with all nonnegative eigenvalues. Let 1 , 2 , , n denote
the eigenvalues of P 0 P . Then, 1 j 0 or 0 j 1 (j = 1, , n).
P
P
Hence, nj=1 2j nj=1 j , which implies tr(P 0 P )2 tr(P 0 P ).
On the other hand, we have
(tr(P 0 P ))2 = (tr(P P 0 P ))2 tr(P 0 P )tr(P 0 P )2
from the generalized Schwarz inequality (set A0 = P and B = P 0 P in
(1.19)) and P 2 = P . Hence, tr(P 0 P ) tr(P 0 P )2 tr(P 0 P ) = tr(P 0 P )2 ,
from which it follows that tr{(P P 0 P )0 (P P 0 P )} = tr{P 0 P P 0 P
P 0 P + (P 0 P )2 } = tr{P 0 P (P 0 P )2 } = 0. Thus, P = P 0 P P 0 =
P.
Q.E.D.
Corollary Let M be a symmetric pd matrix, and define the (squared) norm
of x by
||x||2M = x0 M x.
(2.58)

2.4. NORM OF PROJECTION VECTORS

47

The necessary and sufficient condition for a projection matrix P (satisfying

P 2 = P ) to satisfy
(2.59)
||P x||2M ||x||2M
for an arbitrary n-component vector x is given by
(M P )0 = M P .

(2.60)

Proof. Let M = U 2 U 0 be the spectral decomposition of M , and let

=
M 1/2 = U 0 . Then, M 1/2 = U 1 . Define y = M 1/2 x, and let P
2
1/2
1/2

M PM
. Then, P = P , and (2.58) can be rewritten as ||P y||2
2
||y|| . By Theorem 2.22, the necessary and sufficient condition for (2.59) to
hold is given by
= (M 1/2 P M 1/2 )0 = M 1/2 P M 1/2 ,
2 = P
P
leading to (2.60).

(2.61)
Q.E.D.

Note The theorem above implies that with an oblique projector P (P 2 = P , but
P 0 6= P ) it is possible to have ||P x|| ||x||. For example, let

1 1
1
P =
and x =
.
0 0
1

Then, ||P x|| = 2 and ||x|| = 2.

Theorem 2.23 Let P 1 and P 2 denote the orthogonal projectors onto V1

and V2 , respectively. Then, for an arbitrary x E n , the following relations
hold:
(2.62)
||P 2 P 1 x|| ||P 1 x|| ||x||
and, if V2 V1 ,
||P 2 x|| ||P 1 x||.

(2.63)

Proof. (2.62): Replace x by P 1 x in Theorem 2.22.

(2.63): By Theorem 2.11, we have P 1 P 2 = P 2 , from which (2.63) follows immediately.
Let x1 , x2 , , xp represent p n-component vectors in E n , and define
X = [x1 , x2 , , xp ]. From (1.15) and P = P 0 P , the following identity
holds:
(2.64)
||P x1 ||2 + ||P x2 ||2 + + ||P xp ||2 = tr(X 0 P X).

48

CHAPTER 2. PROJECTION MATRICES

The above identity and Theorem 2.23 lead to the following corollary.
Corollary
(i) If V2 V1 , tr(X 0 P 2 X) tr(X 0 P 1 X) tr(X 0 X).
(ii) Let P denote an orthogonal projector onto an arbitrary subspace in E n .
If V1 V2 ,
tr(P 1 P ) tr(P 2 P ).
Proof. (i): Obvious from Theorem 2.23. (ii): We have tr(P j P ) = tr(P j P 2 )
= tr(P P j P ) (j = 1, 2), and (P 1 P 2 )2 = P 1 P 2 , so that
tr(P P 1 P ) tr(P P 2 P ) = tr(SS 0 ) 0,
where S = (P 1 P 2 )P . It follows that tr(P 1 P ) tr(P 2 P ).
Q.E.D.
We next present a theorem on the trace of two orthogonal projectors.
Theorem 2.24 Let P 1 and P 2 be orthogonal projectors of order n. Then
the following relations hold:
tr(P 1 P 2 ) = tr(P 2 P 1 ) min(tr(P 1 ), tr(P 2 )).

(2.65)

Proof. We have tr(P 1 ) tr(P 1 P 2 ) = tr(P 1 (I n P 2 )) = tr(P 1 Q2 ) =

tr(P 1 Q2 P 1 ) = tr(S 0 S) 0, where S = Q2 P 1 , establishing tr(P 1 )
tr(P 1 P 2 ). Similarly, (2.65) follows from tr(P 2 ) tr(P 1 P 2 ) = tr(P 2 P 1 ).
Q.E.D.
Note From (1.19), we obtain
tr(P 1 P 2 )

p
tr(P 1 )tr(P 2 ).

However, (2.65) is more general than (2.66) because

tr(P 2 )).

(2.66)

p
tr(P 1 )tr(P 2 ) min(tr(P 1 ),

2.5. MATRIX NORM AND PROJECTION MATRICES

2.5

49

Matrix Norm and Projection Matrices

Let A = [aij ] be an n by p matrix. We define its Euclidean norm (also called

the Frobenius norm) by
v
uX
p
u n X
a2ij .
||A|| = {tr(A0 A)}1/2 = t

(2.67)

i=1 j=1

Then the following four relations hold.

Lemma 2.6
||A|| 0.

(2.68)

||CA|| ||C|| ||A||,

(2.69)

Let both A and B be n by p matrices. Then,

||A + B|| ||A|| + ||B||.

(2.70)

Let U and V be orthogonal matrices of orders n and p, respectively. Then

||U AV || = ||A||.

(2.71)

Proof. Relations (2.68) and (2.69) are trivial. Relation (2.70) follows immediately from (1.20). Relation (2.71) is obvious from
tr(V 0 A0 U 0 U AV ) = tr(A0 AV V 0 ) = tr(A0 A).
Q.E.D.
Note Let M be a symmetric nnd matrix of order n. Then the norm defined in
(2.67) can be generalized as
||A||M = {tr(A0 M A)}1/2 .

(2.72)

This is called the norm of A with respect to M (sometimes called a metric matrix).
Properties analogous to those given in Lemma 2.6 hold for this generalized norm.
There are other possible definitions of the norm of A. For example,
Pn
(i) ||A||1 = maxj i=1 |aij |,
(ii) ||A||2 = 1 (A), where 1 (A) is the largest singular value of A (see Chapter 5),
and
Pp
(iii) ||A||3 = maxi j=1 |aij |.
All of these norms satisfy (2.68), (2.69), and (2.70). (However, only ||A||2 satisfies
(2.71).)

50

CHAPTER 2. PROJECTION MATRICES

denote orthogonal projectors of orders n and p,

Lemma 2.7 Let P and P
respectively. Then,
||P A|| ||A||
(2.73)
(the equality holds if and only if P A = A) and
|| ||A||
||AP

(2.74)

= A).
(the equality holds if and only if AP
Proof. (2.73): Square both sides and subtract the right-hand side from the
left. Then,
tr(A0 A) tr(A0 P A) = tr{A0 (I n P )A}
= tr(A0 QA) = tr(QA)0 (QA) 0 (where Q = I n P ).
The equality holds when QA = O P A = A.
A0 AP
)
||2 = tr(P
(2.74): This can be proven similarly by noting that ||AP
0
0
0
0
2
A || . The equality holds when QA

A =
A ) = ||P
= O P
= tr(AP
0

Q.E.D.
A AP = A, where Q = I n P .
The two lemmas above lead to the following theorem.
Theorem 2.25 Let A be an n by p matrix, B and Y n by r matrices, and
C and X r by p matrices. Then,
||A BX|| ||(I n P B )A||,

(2.75)

where P B is the orthogonal projector onto Sp(B). The equality holds if and
only if BX = P B A. We also have
||A Y C|| ||A(I p P C 0 )||,

(2.76)

where P C 0 is the orthogonal projector onto Sp(C 0 ). The equality holds if

and only if Y C = AP C 0 . We also have
||A BX Y C|| ||(I n P B )A(I p P C 0 )||.

(2.77)

The equality holds if and only if

P B (A Y C) = BX and (I n P B )AP C 0 = (I n P B )Y C

(2.78)

2.5. MATRIX NORM AND PROJECTION MATRICES

51

or
(A BX)P C 0 = Y C and P B A(I p P C 0 ) = BX(I n P C 0 ).

(2.79)

Proof. (2.75): We have (I n P B )(A BX) = A BX P B A +

BX = (I n P B )A. Since I n P B is an orthogonal projector, we have
||A BX|| ||(I n P B )(A BX)|| = ||(I n P B )A|| by (2.73) in Lemma
2.7. The equality holds when (I n P B )(A BX) = A BX, namely
P B A = BX.
(2.76): It suffices to use (A Y C)(I p P C 0 ) = A(I p P C 0 ) and (2.74)
in Lemma 2.7. The equality holds when (A Y C)(I p P C 0 ) = A Y C
holds, which implies Y C = AP C 0 .
(2.77): ||ABX Y C|| ||(I n P B )(AY C)|| ||(I n P B )A(I p
P C 0 )|| or ||A BX Y C|| ||(A BX)(I p P C 0 )|| ||(I p P B )A(I p
P C 0 )||. The first equality condition (2.78) follows from the first relation
above, and the second equality condition (2.79) follows from the second relation above.
Q.E.D.
Note Relations (2.75), (2.76), and (2.77) can also be shown by the least squares
method. Here we show this only for (2.77). We have
||A BX Y C||2 = tr{(A BX Y C)0 (A BX Y C)}
= tr(A Y C)0 (A Y C) 2tr(BX)0 (A Y C) + tr(BX)0 (BX)
to be minimized. Differentiating the criterion above by X and setting the result to zero, we obtain B 0 (A Y C) = B 0 BX. Premultiplying this equation by
B(B 0 B)1 , we obtain P B (A Y C) = BX. Furthermore, we may expand the
criterion above as
tr(A BX)0 (A BX) 2tr(Y C(A BX)0 ) + tr(Y C)(Y C)0 .
Differentiating this criterion with respect to Y and setting the result equal to zero,
we obtain C(A BX) = CC 0 Y 0 or (A BX)C 0 = Y CC 0 . Postmultiplying
the latter by (CC 0 )1 C 0 , we obtain (A BX)P C 0 = Y C. Substituting this into
P B (A Y C) = BX, we obtain P B A(I p P C 0 ) = BX(I p P C 0 ) after some
simplification. If, on the other hand, BX = P B (A Y C) is substituted into
(A BX)P C 0 = Y C, we obtain (I n P B )AP C 0 = (I n P B )Y C. (In the
derivation above, the regular inverses can be replaced by the respective generalized
inverses. See the next chapter.)

52

CHAPTER 2. PROJECTION MATRICES

2.6

General Form of Projection Matrices

The projectors we have been discussing so far are based on Definition 2.1,
namely square matrices that satisfy P 2 = P (idempotency). In this section,
we introduce a generalized form of projection matrices that do not necessarily satisfy P 2 = P , based on Rao (1974) and Rao and Yanai (1979).
Definition 2.3 Let V E n (but V 6= E n ) be decomposed as a direct-sum
of m subspaces, namely V = V1 V2 Vm . A square matrix P j of
order n that maps an arbitrary vector y in V into Vj is called the projection
matrix onto Vj along V(j) = V1 Vj1 Vj+1 Vm if and only if
P j x = x x Vj (j = 1, , m)

(2.80)

P j x = 0 x V(j) (j = 1, , m).

(2.81)

and

Let xj Vj . Then any x V can be expressed as

x = x1 + x2 + + xm = (P 1 + P 2 + P m )x.
Premultiplying the equation above by P j , we obtain
P i P j x = 0 (i 6= j) and (P j )2 x = P j x (i = 1, , m)

(2.82)

since Sp(P 1 ), Sp(P 2 ), , Sp(P m ) are mutually disjoint. However, V does

not cover the entire E n (x V 6= E n ), so (2.82) does not imply (P j )2 = P j
or P i P j = O (i 6= j).
Let V1 and V2 E 3 denote the subspaces spanned by e1 = (0, 0, 1)0 and
e2 = (0, 1, 0)0 , respectively. Suppose

a 0 0

P = b 0 0 .
c 0 1
Then, P e1 = e1 and P e2 = 0, so that P is the projector onto V1 along V2
according to Definition 2.3. However, (P )2 6= P except when a = b = 0,
or a = 1 and c = 0. That is, when V does not cover the entire space E n ,
the projector P j in the sense of Definition 2.3 is not idempotent. However,
by specifying a complement subspace of V , we can construct an idempotent
matrix from P j as follows.

2.7. EXERCISES FOR CHAPTER 2

53

Theorem 2.26 Let P j (j = 1, , m) denote the projector in the sense of

Definition 2.3, and let P denote the projector onto V along Vm+1 , where
V = V1 V2 Vm is a subspace in E n and where Vm+1 is a complement
subspace to V . Then,
P j = P j P (j = 1, m) and P m+1 = I n P

(2.83)

are projectors (in the sense of Definition 2.1) onto Vj (j = 1, , m + 1)

= V V
along V(j)
1
j1 Vj+1 Vm Vm+1 .
Proof. Let x V . If x Vj (j = 1, , m), we have P j P x = P j x = x.
On the other hand, if x Vi (i 6= j, i = 1, , m), we have P j P x = P j x =
0. Furthermore, if x Vm+1 , we have P j P x = 0 (j = 1, , m). On the
other hand, if x V , we have P m+1 x = (I n P )x = x x = 0, and if
x Vm+1 , P m+1 x = (I n P )x = x 0 = x. Hence, by Theorem 2.2, P j
(j = 1, , m + 1) is the projector onto Vj along V(j) .
Q.E.D.

2.7

Exercises for Chapter 2

=
1. Let A

A1
O

O
A2

and A =

A1
. Show that P A P A = P A .
A2

2. Let P A and P B denote the orthogonal projectors onto Sp(A) and Sp(B), respectively. Show that the necessary and sufficient condition for Sp(A) = {Sp(A)

Sp(B)} {Sp(A) Sp(B) } is P A P B = P B P A .

3. Let P be a square matrix of order n such that P 2 = P , and suppose
||P x|| = ||x||
for any n-component vector x. Show the following:
(i) When x (Ker(P )) , P x = x.
(ii) P 0 = P .

4. Let Sp(A) = Sp(A1 ) Sp(Am ), and let P j (j = 1, , m) denote the

projector onto Sp(Aj ). For x E n :
(i) Show that
(2.84)
||x||2 ||P 1 x||2 + ||P 2 x||2 + + ||P m x||2 .
(Also, show that the equality holds if and only if Sp(A) = E n .)
(ii) Show that Sp(Ai ) and Sp(Aj ) (i 6= j) are orthogonal if Sp(A) = Sp(A1 )
Sp(A2 ) Sp(Am ) and the inequality in (i) above holds.
(iii) Let P [j] = P 1 + P 2 + + P j . Show that
||P [m] x|| ||P [m1] x|| ||P [2] x|| ||P [1] x||.

54

CHAPTER 2. PROJECTION MATRICES

5. Let E n = V1 W1 = V2 W2 = V3 W3 , and let P j denote the projector onto

Vj (j = 1, 2, 3) along Wj . Show the following:
(i) Let P i P j = O for i 6= j. Then, P 1 + P 2 + P 3 is the projector onto V1 + V2 + V3
along W1 W2 W3 .
(ii) Let P 1 P 2 = P 2 P 1 , P 1 P 3 = P 3 P 1 , and P 2 P 3 = P 3 P 2 . Then P 1 P 2 P 3 is
the projector onto V1 V2 V3 along W1 + W2 + W3 .
(iii) Suppose that the three identities in (ii) hold, and let P 1+2+3 denote the projection matrix onto V1 + V2 + V3 along W1 W2 W3 . Show that
P 1+2+3 = P 1 + P 2 + P 3 P 1 P 2 P 2 P 3 P 1 P 3 + P 1 P 2 P 3 .
6. Show that
Q[A,B] = QA QQA B ,
where Q[A,B] , QA , and QQA B are the orthogonal projectors onto the null space of
[A, B], onto the null space of A, and onto the null space of QA B, respectively.
7. (a) Show that
P X = P XA + P X(X 0 X)1 B ,
where P X , P XA , and P X(X 0 X)1 B are the orthogonal projectors onto Sp(X),
Sp(XA), and Sp(X(X 0 X)1 B), respectively, and A and B are such that Ker(A0 )
= Sp(B).
(b) Use the decomposition above to show that
P [X1 ,X2 ] = P X1 + P QX1 X2 ,
where X = [X 1 , X 2 ], P QX1 X2 is the orthogonal projector onto Sp(QX1 X 2 ), and
QX1 = I X 1 (X 01 X 1 )1 X 01 .
8. Let E n = V1 W1 = V2 W2 , and let P 1 = P V1 W1 and P 2 = P V2 W2 be two
projectors (not necessarily orthogonal) of the same size. Show the following:
(a) The necessary and sufficient condition for P 1 P 2 to be a projector is V12
V2 (W1 W2 ), where V12 = Sp(P 1 P 2 ) (Brown and Page, 1970).
(b) The condition in (a) is equivalent to V2 V1 (W1 V2 ) (W1 W2 ) (Werner,
1992).
9. Let A and B be n by a (n a) and n by b (n b) matrices, respectively. Let
P A and P B be the orthogonal projectors defined by A and B, and let QA and
QB be their orthogonal complements. Show that the following six statements are
equivalent: (1) P A P B = P B P A , (2) A0 B = A0 P B P A B, (3) (P A P B )2 = P A P B ,
(4) P [A,B] = P A + P B P A P B , (5) A0 QB QA B = O, and (6) rank(QA B) =
rank(B) rank(A0 B).

http://www.springer.com/978-1-4419-9886-6