Scribd
Upload
58 views3 pages

Orthogonal Projections Onto Higher-Dimensional Sub-Spaces

The document discusses orthogonal projections of vectors onto subspaces. It explains that projecting a vector x onto a subspace U involves: 1) Determining a basis B = (b1, ..., bm) for U and finding the coordinates λ that minimize the distance between x and its projection using the normal equation. 2) Computing the projection as πU(x) = Bλ. 3) The projection matrix is Pπ = B(B^T B)^-1 B^T. It notes that projections can be used to find approximate solutions to unsolvable linear systems by projecting the target vector b onto the column space of A, known as the least squares solution.

Uploaded by

Prashant
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
58 views3 pages

Orthogonal Projections Onto Higher-Dimensional Sub-Spaces

The document discusses orthogonal projections of vectors onto subspaces. It explains that projecting a vector x onto a subspace U involves: 1) Determining a basis B = (b1, ..., bm) for U and finding the coordinates λ that minimize the distance between x and its projection using the normal equation. 2) Computing the projection as πU(x) = Bλ. 3) The projection matrix is Pπ = B(B^T B)^-1 B^T. It notes that projections can be used to find approximate solutions to unsolvable linear systems by projecting the target vector b onto the column space of A, known as the least squares solution.

Uploaded by

Prashant
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 3

Figure 1: Projection onto a two-dimensional subspace U with basis b1 , b2 .

The
projection πU (x) of x ∈ R3 onto U can be expressed as a linear combination of
b1 , b2 and the displacement vector x − πU (x) is orthogonal to both b1 and b2 .
x

x − πU (x)

U
b2

πU (x)

0 b1

Orthogonal Projections onto Higher-Dimensional Sub-


spaces
In the following, we will look at orthogonal projections of vectors x ∈ Rn onto
higher-dimensional subspaces U ⊆ Rn with dim(U ) = m ≥ 1. An illustration is
given in Figure 1.
Assume that (b1 , . . . , bm ) is an ordered basis of U . Projections πU (x) onto U If U is given by
are elements of U . Therefore, they can be representedP as linear combinations a set of span-
m
of the basis vectors b1 , . . . , bm of U , such that πU (x) = i=1 λi bi . ning vectors,
As in the 1D case, we follow a three-step procedure to find the projection which are not
p = πU (x) and the projection matrix P π : a basis, make
sure you deter-
1. Find the coordinates λ1 , . . . , λm of the projection (with respect to the basis mine a basis
of U ), such that the linear combination b1 , . . . , bm
m before pro-
ceeding.
X
πU (x) = p = λi bi = Bλ , (1)
i=1 The basis vec-
B = (b1 | . . . |bm ) ∈ Rn×m , λ = [λ1 , . . . , λm ]> ∈ Rm , (2) tors form the
columns of
is closest to x ∈ Rn . As in the 1D case, “closest” means “minimum dis- B ∈ Rn×m ,
tance”, which implies that the vector connecting p ∈ U and x ∈ Rn must where B =
be orthogonal to all basis vectors of U . Therefore, we obtain m simulta- (b1 |...|bm ).
neous conditions (assuming the dot product as the inner product)

hb1 , x − pi = b>
1 (x − p) = 0 (3)

1
..
. (4)
hbm , x − pi = b>
m (x − p) = 0 (5)

which, with p = Bλ, can be written as

b>
1 (x − Bλ) = 0 (6)
..
. (7)
b>
m (x − Bλ) = 0 (8)

such that we obtain a homogeneous linear equation system


 > 
b1 
 ..   >
 .  x − Bλ = 0 ⇐⇒ B (x − Bλ) = 0 (9)
b>m

⇐⇒ B > Bλ = B > x . (10)

The last expression is called normal equation. Since b1 , . . . , bm are a ba- normal equa-
sis of U and, therefore, linearly independent, B > B ∈ Rm×m is regular tion
and can be inverted. This allows us to solve for the optimal coefficients/
coordinates

λ = (B > B)−1 B > x . (11)

The matrix (B > B)−1 B > is also called the pseudo-inverse of B, which can pseudo-
be computed for non-square matrices B. It only requires that B > B is inverse
positive definite, which is the case if B is full rank.1
2. Find the projection πU (x) = p ∈ U . We already established that p = Bλ.
Therefore, with (11)

p = B(B > B)−1 B > x . (12)

3. Find the projection matrix P π . From (12) we can immediately see that
the projection matrix that solves P π x = p must be

P π = B(B > B)−1 B > . (13)

Remark. Comparing the solutions for projecting onto a one-dimensional sub-


space and the general case, we see that the general case includes the 1D case as
a special case: If dim(U ) = 1 then B > B ∈ R is a scalar and we can rewrite the
BB >
projection matrix in (13) P π = B(B > B)−1 B > as P π = B > B , which is exactly

the projection matrix for the one-dimensional case.


1 In practical applications (e.g., linear regression), we often add a “jitter term” I to B > B

to guarantee increase numerical stability and positive definiteness. This “ridge” can be rigorously
derived using Bayesian inference.

2
Remark. The projections πU (x) are still vectors in Rn although they lie in an
m-dimensional subspace U ⊆ Rn . However, to represent a projected vector
we only need the m coordinates λ1 , . . . , λm with respect to the basis vectors
b1 , . . . , bm of U .
Remark. In vector spaces with general inner products, we have to pay attention
when computing angles and distances, which are defined by means of the inner
product.
We can find
Projections allow us to look at situations where we have a linear system approximate
Ax = b without a solution. Recall that this means that b does not lie in the solutions to
span of A, i.e., the vector b does not lie in the subspace spanned by the columns unsolvable lin-
of A. Given that the linear equation cannot be solved exactly, we can find an ear equation
approximate solution. The idea is to find the vector in the subspace spanned by systems using
the columns of A that is closest to b, i.e., we compute the orthogonal projection projections.
of b onto the subspace spanned by the columns of A. This problem arises often
in practice, and the solution is called the least squares solution (assuming the least squares
dot product as the inner product) of an overdetermined system. solution

You might also like