Linear Algebra (part 5): Bilinear and Quadratic Forms (by Evan Dummit, 2022, v. 2.

00)

Contents
5 Bilinear and Quadratic Forms 1
5.1 Bilinear Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

5.1.1 Denition, Associated Matrices, Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . 1

5.1.2 Symmetric Bilinear Forms and Diagonalization . . . . . . . . . . . . . . . . . . . . . . . . . . 3

5.2 Quadratic Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

5.2.1 Denition and Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

5.2.2 Quadratic Forms Over Rn : Diagonalization of Quadratic Varieties . . . . . . . . . . . . . . . 7

5.2.3 Deniteness of Real Quadratic Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

5.2.4 Quadratic Forms Over Rn : The Second Derivatives Test . . . . . . . . . . . . . . . . . . . . . 12

5.2.5 Quadratic Forms Over Rn : Sylvester's Law of Inertia . . . . . . . . . . . . . . . . . . . . . . . 13

5.3 Singular Values and Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5.3.1 Singular Values and Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . 15

5.3.2 The Moore-Penrose Pseudoinverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

5 Bilinear and Quadratic Forms

In this chapter, we will discuss bilinear and quadratic forms. Bilinear forms are simply linear transformations that
are linear in more than one variable, and they will allow us to extend our study of linear phenomena. They are
closely related to quadratic forms, which are (classically speaking) homogeneous quadratic polynomials in multiple
variables. Despite the fact that quadratic forms are not linear, we can (perhaps surprisingly) still use many of the
tools of linear algebra to study them. We will discuss the basic properties of bilinear and quadratic forms and in
particular highlighting the notions of positive deniteness and positive semideniteness, along with some of their
applications in linear algebra, calculus, and geometry. We nish with a discussion of singular value decomposition,
which ties together many of the threads we have discussed.

5.1 Bilinear Forms

• We begin by discussing basic properties of bilinear forms on an arbitrary vector space.

5.1.1 Denition, Associated Matrices, Basic Properties

• Let V be a vector space over the eld F.

• Denition: A function Φ : V ×V → F is a bilinear form on V if it is linear in each variable when the

other variable is xed. Explicitly, this means Φ(v1 + αv2 , y) = Φ(v1 , w) + αΦ(v2 , w) and Φ(v, w1 + αw2 ) =
Φ(v, w1 ) + αΦ(v, w2 ) for arbitrary vi , wi ∈ V and α ∈ F .

◦ It is easy to see that the set of all bilinear forms on V forms a vector space under componentwise addition
and scalar multiplication.

1
 ◦ Example: An inner product on a real vector space is a bilinear form, but an inner product on a complex
vector space is not, since it is conjugate-linear in the second component rather than (actually) linear.

◦ Example: If V = F [x] and a, b ∈ F , then Φ(p, q) = p(a)q(b) is a bilinear form on V.

´b
◦ Example: If V = C[a, b] is the space of continuous functions on [a, b], then Φ(f, g) = a
f (x)g(x) dx is a
bilinear form on V .

• A large class of examples of bilinear forms arise as follows: if V = F n, then for any matrix A ∈ Mn×n (F ),
T
the map ΦA (v, w) = v Aw is a bilinear form on V.
        
1 2 x1 x2 1 2 x2
◦ Example: The matrix A= yields the bilinear form ΦA , = [x1 y1 ] =
3 4 y1 y2 3 4 y2
x1 x2 + 2x1 y2 + 3x2 y1 + 4y1 y2 .

• Indeed, if V is nite-dimensional, then by choosing a basis of V we can see that every bilinear form arises in
the manner described above:

• Denition: If V is a nite-dimensional vector space, β = {β1 , . . . , βn } is a basis of V , and Φ is a bilinear form

on V, the associated matrix of Φ with respect to β is the matrix [Φ]β ∈ Mn×n (F ) whose (i, j)-entry is the
value Φ(βi , βj ).

◦ This is the natural analogue of the matrix associated to a linear transformation, for bilinear forms.

◦ We will remark that we could give a more general formulation of an associated matrix with respect to
two dierent bases β in the rst coordinate and γ in the second coordinate, but we will not need this
general formulation at any point.

• Example: For the bilinear form Φ((a, b), (c, d)) = 2ac + 4ad − bc on F 2, nd [Φ]β for the standard basis
β = {(1, 0), (0, 1)}.

◦ We simply calculate the four values Φ(βi , βj ) for i, j ∈ {1, 2}, where β1 = (1, 0) and β2 = (0, 1).
◦ This yields Φ(β1 , β1 ) = 2, Φ(β1 , β2 ) = 4, Φ(β2 , β1 ) = −1, and Φ(β2 , β2 ) = 0.
 
2 4
◦ Thus, the associated matrix is [Φ]β = .
−1 0

• Example: For the bilinear form Φ((a, b), (c, d)) = 2ac + 4ad − bc on F 2, nd [Φ]γ for the basis γ =
{(2, 1), (−1, 4)}.

◦ We simply calculate the four values Φ(γi , γj ) for i, j ∈ {1, 2}, where γ1 = (2, 1) and γ2 = (−1, 4).
◦ This yields Φ(γ1 , γ1 ) = 14, Φ(γ1 , γ2 ) = 29, Φ(γ2 , γ1 ) = −16, and Φ(γ2 , γ2 ) = −10.
 
14 29
◦ Thus, the associated matrix is [Φ]γ = .
−16 −10

´1
• Example: For the bilinear form Φ(p, q) = 0
p(x)q(x) dx on P2 (R), nd [Φ]β for the basis β = {1, x, x2 }.

◦ We simply calculate the nine values Φ(βi , βj ) for i, j ∈ {1, 2, 3}, where β1 = 1, β2 = x, β3 = x2 .
´1 1 ´1 1
◦ For example, Φ(β1 , β3 ) = 0 1 · x2 dx = and Φ(β3 , β2 ) = 0 x2 · x dx = .
3 4
 
1 1/2 1/3
◦ The resulting associated matrix is [Φ]β =  1/2 1/3 1/4  .
1/3 1/4 1/5

• Like with the matrices associated with linear transformations, we can describe how the matrices associated
to bilinear forms relate to coordinate vectors, how they transform under a change of basis, and how to use
them to translate back and forth between bilinear forms and matrices:

2
 • Proposition (Associated Matrices): Suppose that V is a nite-dimensional vector space, β = {β1 , . . . , βn } is a
basis of V, and Φ is a bilinear form on V. Then the following hold:

1. If v and w are any vectors in V, then Φ(v, w) = [v]Tβ [Φ]β [w]β .

◦ Proof: If v = βi and w = βj then the result follows immediately from the denition of matrix
multiplication and the matrix [Φ]β . The result for arbitrary v and w then follows by linearity.

2. The association Φ 7→ [Φ]β of a bilinear form with its matrix representation yields an isomorphism of the
space B(V ) of bilinear forms on V with Mn×n (F ). In particular, dimF B(V ) = n2 .
◦ Proof: The inverse map is dened as follows: given a matrix A ∈ Mn×n (F ), dene a bilinear form
ΦA via ΦA (v, w) = [v]Tβ A[w]β .
◦ It is easy to verify that this map is a well-dened linear transformation and that it is inverse to the
map given above. The dimension calculation is immediate.

T
3. If Φ is the reverse form of Φ dened via ΦT (v, w) = Φ(w, v), then [ΦT ]β = [Φ]Tβ .
◦ Proof: By denition we have ΦT (v, w) = [w]Tβ [Φ]β [v]β . Since the matrix product on the right is a
T T
scalar, it is equal to its transpose, which is [v]β [Φ]β [w]β .

◦ This means [ΦT ]β and [Φ]Tβ agree, as bilinear forms, on all pairs of vectors [v]β and [w]β in F n, so
they are equal.

4. If γ is another basis of V and Q = [I]γβ is the change-of-basis matrix from β to γ, then [Φ]γ = QT [Φ]β Q.
◦ Proof: By denition, [v]γ = Q[v]β . Hence [v]Tβ QT [Φ]β Q[w]β = [v]Tγ [Φ]β [w]γ .
◦ This means that QT [Φ]β Q and [Φ]γ agree, as bilinear forms, on all pairs of vectors [v]β and [w]β in
n
F , so they are equal.

• The last result of the proposition above tells us how bilinear forms behave under change of basis: rather than
the more familiar conjugation relation B = QAQ−1 , we instead have a slightly dierent relation B = QT AQ.

• We record one other property of bilinear forms that we will need to make use of later:

• Denition: If Φ is a bilinear form on V and there exists a nonzero vector x ∈ V such that Φ(x, v) = 0 for all
v ∈V, we say Φ is degenerate. Otherwise, if there is no such x, we say Φ is nondegenerate.

5.1.2 Symmetric Bilinear Forms and Diagonalization

• In the same way that we classied the linear operators on a vector space that can be diagonalized, we would
also like to classify the diagonalizable bilinear forms.

• Denition: If V is nite-dimensional, a bilinear form Φ on V is diagonalizable if there exists a basis β of V

such that [Φ]β is a diagonal matrix.

◦ The matrix formulation of this question is as follows: we say that matrices B and C are congruent if
there exists an invertible matrix Q such that C = QT BQ.
◦ Then the matrices B and C are congruent if and only if they represent the same bilinear form in dierent
bases (the translation being B = [Φ]β and C = [Φ]γ , with Q = [I]γβ being the corresponding change-of-
basis matrix).

◦ Warning: Although we use the same word, diagonalizability for bilinear forms is not the same as diagonal-
izability for linear transformations! Make sure to keep straight the dierence between the corresponding
matrix versions: two matrices are similar when we can write B = Q−1 AQ, whereas they are congruent
T
when we can write B = Q AQ.

• It turns out that when char(F ) 6= 2, there is an easy criterion for diagonalizability.

• Denition: A bilinear form Φ on V is symmetric if Φ(v, w) = Φ(w, v) for all v, w ∈ V .

◦ Notice that Φ is symmetric if and only if it equals its reverse form ΦT .

3
 ◦ By taking associated matrices, we see immediately that if V is nite-dimensional with basis β, then Φ
is a symmetric bilinear form if and only if [Φ]β is equal to its transpose, which is to say, when it is a
symmetric matrix.

◦ Now observe that if Φ is diagonalizable, then [Φ]β is a diagonal matrix hence symmetric, and thus Φ
must be symmetric.

• When the characteristic of F is not equal to 2, the converse holds also:

• Theorem (Diagonalization of Bilinear Forms): Let V be a nite-dimensional vector space over a eld F of
characteristic not equal to 2. Then a bilinear form on V is diagonalizable if and only if it is symmetric.

◦ Proof: The forward direction was established above. For the reverse, we show the result by induction on
n = dimF V . The base case n = 1 is trivial, so suppose the result holds for all spaces of dimension less
than n, and let Φ be symmetric on V .
◦ If Φ is the zero form, then clearly Φ is diagonalizable. Otherwise, suppose Φ is not identically zero: we
claim there exists a vector x with Φ(x, x) 6= 0.
◦ By hypothesis, Φ is not identically zero so suppose that Φ(v, w) 6= 0. If Φ(v, v) 6= 0 or Φ(w, w) 6= 0 we
may take x = v or x = w. Otherwise, we have Φ(v + w, v + w) = Φ(v, v) + 2Φ(v, w) + Φ(w, w) =
2Φ(v, w) 6= 0 by the assumption that Φ(v, w) 6= 0 and 2 6= 0 in F (here is where we require the
characteristic not to equal 2), and so we may take x = v + w.

◦ Now consider the linear functional T : V → F given by T (v) = Φ(x, v). Since T (x) = Φ(x, x) 6= 0, we
see that im(T ) = F , so dimF ker(T ) = n − 1 by the nullity-rank theorem.
◦ Then the restriction of Φ to ker(T ) is clearly a symmetric bilinear form on ker(T ), so by induction, there
exists a basis {v1 , . . . , vn−1 } of ker(T ) such that the restriction of Φ is diagonalized by this basis, which
is to say, Φ(vi , vj ) = 0 for i 6= j .
◦ Now observe that since x 6∈ ker(T ), the set β = {v1 , . . . , vn−1 , x} is a basis of V . Since Φ(x, vi ) =
Φ(vi , x) = 0 for all i < n by denition of T , we conclude that β diagonalizes Φ, as required.

• We will note that the assumption that char(F ) 6= 2 in the theorem above cannot be removed.

◦ Explicitly, if
 F = F2
 is the eld with 2 elements and Φ is the bilinear form on F2 with associated matrix
0 1
A= , then Φ is symmetric but cannot be diagonalized.
1 0
       
a b T a c 0 1 a b 0 ad + bc
◦ Explicitly, suppose Q= : then Q AQ = = ,
c d b d 1 0 c d ad + bc 0
so the only possible diagonalization of Φ would be the zero matrix, but that is impossible because Φ is
not the zero form.

◦ In this example we can see that

1 Φ(x, x) = 0 for all x ∈ F 2, which causes the inductive argument to fail.

• As an immediate corollary, we see that every symmetric matrix is congruent to a diagonal matrix in charac-
teristic 6= 2:
• Corollary: If char(F ) 6= 2, then every symmetric matrix over F is congruent to a diagonal matrix.

◦ Proof: The result follows immediately by diagonalizing the corresponding bilinear form.

• We can give an explicit procedure for writing a symmetric matrix S in the form D = QT SQ that is similar
to the algorithm for computing the inverse of a matrix.

◦ Recall that if E is an elementary row matrix (obtained by performing an elementary row operation on
the identity matrix), then EA is the matrix obtained by performing that elementary row operation on
A.
◦ Likewise, if C is an elementary column matrix, then AC is the matrix obtained by performing that
elementary column operation on A.
1 This follows by noting that Φ(β , β ) = 0 for each basis element β . Then if v = a β + · · · + a β , expanding Φ(v, v) linearly and
i i i 1 1 n n
applying symmetry shows that every term ai aj Φ(βi , βj ) for i 6= j has a coecient of 2, so Φ(v, v) = 0 for all v.

4
 ◦ Hence if E is an elementary row matrix, then EAE T is the matrix obtained by performing the elementary
row operation on A (given by E) and then the corresponding elementary column operation (given by
E T ).
◦ Since the invertible matrix Q is a product E1 · · · Ed of elementary row matrices, we see that QT SQ =
EdT · · · E1T SE1 · · · Ed is obtained from S by performing a sequence of these paired row-column operations.

◦ Our result on diagonalization above ensures that there is a sequence of these operations that will yield
a diagonal matrix.

◦ We may nd the proper sequence of operations by performing these paired operations using a method
similar to row-reduction: using the (1,1)-entry, we apply row operations to clear out all the entries in
the rst column below it. (If this entry is zero, we add an appropriate multiple of another row to the
top row to make it nonzero.)

◦ This will also clear out the column entries to the right of the (1,1)-entry, yielding a matrix whose rst
row and column are now diagonalized. We then restrict attention to the smaller (n − 1) × (n − 1) matrix
excluding the rst row and column, and repeat the procedure recursively until the matrix is diagonalized.

◦ Then we may obtain the matrix QT = EdT · · · E1T I by applying all of the elementary row operations (in
the same order) starting with the identity matrix.

• We may keep track of these operations using a double matrix as in the algorithm for computing the inverse
of a matrix: on the left we start with the symmetric matrix S, and on the right we start with the identity
matrix I.

◦ At each step, we select a row operation and perform it, and its corresponding column operation, on the
left matrix. We also perform the row operation (but only the row operation!) on the right matrix.

◦ When we are nished, we will have transformed the double-matrix [S|I] into the double-matrix [D|QT ],
T
and we will have Q SQ = D.
 
1 3
• Example: For S= , nd an invertible matrix Q and diagonal matrix D such that QT SQ = D.
3 −4

◦ We set up the double matrix and perform row/column operations as listed (to emphasize again, we
perform the row and then the corresponding column operation on the left side, but only the row operation
on the right side):
   
1 3 1 0 R2 −3R1 1 0 1 0
−→
3 −4 0 1 C2 −3C1 0 −13 −3 1
◦ The matrix on the left is now diagonal.
     
1 0 T 1 0 1 −3
◦ Thus, we may take D = with Q = and thus Q = . Indeed, one may
0 −13 −3 1 0 1
T
double-check that Q SQ = D , as claimed.
 
1 2 3
• Example: For S =  2 1 0 , nd an invertible matrix Q and diagonal matrix D such that QT SQ = D.
3 0 2

◦ We set up the double matrix and perform row/column operations as listed:

   
1 2 3 1 0 0 1 0 3 1
0 0
R −2R
 2 1 0 0 1 0  2−→ 1  0 −3 −6 −2
1 0 
C2 −2C1
3 0 2 0 0 1 3 −6 2 0
0 1
   
1 0 0 1 0 0 1 0 0 1 0 0
R3 −3R1 R −2R
−→  0 −3 −6 −2 1 0  3−→ 2  0 −3 0 −2 1 0 
C3 −3C1 C3 −2C2
0 −6 −7 −3 0 1 0 0 5 1 −2 1
◦ The matrix on the left is now diagonal.
     
1 0 0 1 0 0 1 −2 1
◦ Thus, we may take D= 0 −3 0  with QT =  −2 1 0  and thus Q= 0 1 −2 .
0 0 5 1 −2 1 0 0 1
Indeed, one may double-check that QT SQ = D, as claimed.

5
5.2 Quadratic Forms

• In the proof that symmetric bilinear forms are diagonalizable, the existence of a vector x ∈ V such that
Φ(x, x) 6= 0 played a central role. We now examine this non-linear function Φ(x, x) more closely.

5.2.1 Denition and Basic Properties

• Denition: If Φ is a symmetric bilinear form on V, the function Q:V →F given by Q(v) = Φ(v, v) is called
the quadratic form associated to Φ.
 
2 1 3
◦ Example: If Φ is the symmetric bilinear form with matrix A = over F , then the corresponding
3 4
 
x
quadratic form has Q( ) = x2 + 6xy + 4y 2 . (The fact that this is a homogeneous2 quadratic function
y
of the entries of the input vector is the reason for the name quadratic form.)

◦ Example: If Φ is an inner product h·, ·i on a real vector space, then the associated quadratic form is
2
Q(v) = ||v|| , the square of the norm of v.

• Clearly, Q is uniquely determined by Φ. When char(F ) 6= 2, the reverse holds as well.

◦ Q(v + w) = Φ(v + w, v + w) = Q(v) + 2Φ(v, w) + Q(w), we can write Φ(v, w) =

Explicitly, since
1
[Q(v + w) − Q(v) − Q(w)], and so we may recover Φ from Q.
2
◦ Also, observe that for any scalar α ∈ F , we have Q(αv) = Φ(αv, αv) = α2 Φ(v, v) = α2 Q(v).
◦ This last two relations provide us a way to dene a quadratic form without explicit reference to the
underlying symmetric bilinear form.

• Denition: If V is a vector space, a quadratic form is a function Q:V →F such that Q(αv) = α2 Q(v) for
all α ∈ F, and the function Q(v + w) − Q(v) − Q(w) is a bilinear form in v and w.

◦ By setting α=0 we see Q(0) = 0, and by setting α = −1 we see Q(−v) = Q(v).

◦ Like with bilinear forms, the set of all quadratic forms on V forms a vector space.

• Example (again): Show that the function Q[(x, y)] = x2 + 6xy + 4y 2 is a quadratic form on F 2.

◦ First observe that Q[α(x, y)] = (αx)2 + 6(αx)(αy) + 4(αy)2 = α2 (x2 + 6xy + 4y 2 ) = α2 Q(x, y).
◦ We also see that Q[(x1 , y1 ) + (x2 , y2 )] − Q[(x1 , y1 )] − Q[(x2 , y2 )] = 2x1 x2 + 6x1 y2 + 6x2 y1 + 8y1 y2 . It is
straightforward to verify that this is a bilinear form by checking the linearity explicitly.

◦ Alternatively (at least when char(F ) 6= 2) we can write down the associated bilinear form Φ((a, b), (c, d)) =
1
[Q[(a + c, b + d)] − Q[(a, c)] − Q[(b, d)]] = ac + 3ad + 3bc + 4bd, and this is the bilinear form associated
2  
1 3
to the matrix , as indeed we saw earlier.
3 4
´b
• Example: If V = C[a, b], show that the function Q(f ) = a
f (x)2 dx is a quadratic form on V.
´b ´b
◦ First, we have [αf (x)]2 dx = α2 a f (x)2 dx = α2 Q(f ).
Q(αf ) = a
´b ´b ´b ´b
◦ Also, we have Q(f +g)−Q(f )−Q(g) = a [f (x)+g(x)]2 dx− a f (x)2 dx− a g(x)2 dx = a 2f (x)g(x) dx,
and this is indeed a bilinear form in f and g .

1
• If char(F ) 6= 2, then the function [Q(v + w) − Q(v) − Q(w)] is the bilinear pairing associated to Q. It is not
2
hard to see that we obtain a correspondence between quadratic forms and bilinear pairings in this case, since
we may recover a bilinear pairing from each quadratic form and a quadratic form from each bilinear pairing.

2 A polynomial in several variables is called homogeneous if all of the terms have the same total degree. For example, x3 + 3xy 2 − 2y 3
and 2xyz are both homogeneous of degree 3, while x4 z + 2w2 y3 − p5 is homogeneous of degree 5. But x2 + y and x3 y + y4 − x2 are not
homogeneous because they both have terms of dierent degrees.

6
 ◦ In particular, any homogeneous quadratic function on Fn (i.e., any polynomial function all of whose
n
terms have total degree 2) is a quadratic form on F : for variables x1 , . . . , x n , such a function has the
P 3
general form 1≤i≤j≤n ai,j xi xj .
◦ Then we can see that the associated matrix
( A for the corresponding bilinear form has entries ai,j =
ai,i for i=j
aj,i = ; this of course requires char(F ) 6= 2 in order to be able to divide by 2.
ai,j /2 for i 6= j
2 2
◦ Example: The function
 1 − 4x1 x2 + 3x2
Q(x1 , x2 ) = 7x is a quadratic form on F 2. The matrix for the
7 −2
associated symmetric bilinear form is .
−2 3
◦ Example: The function Q(x1 , x2 , x3 ) = x21 + 2x1 x3 − 3x2 x3 + 4x23 is a quadratic form on
  F 3. When
1 0 1
char(F ) 6= 2, the matrix for the associated symmetric bilinear form is  0 0 − 23 .
1 − 32 4
◦ Example: The function Q(x1 , . . . , xn ) = x21 + 2x22 + 3x23 + · · · + nx2n is a quadratic form on F n. Its
associated matrix is the diagonal matrix with entries 1, 2, . . . , n.

5.2.2 Quadratic Forms Over Rn : Diagonalization of Quadratic Varieties

• In the event that V is a nite-dimensional vector space over F = R, quadratic forms are particularly pleasant.
By choosing a basis we may assume that V = Rn for concreteness.

◦ Then, per the real spectral theorem, any real symmetric matrix is orthogonally diagonalizable, meaning
that if S is any real symmetric matrix, then there exists an orthogonal matrix Q (with QT = Q−1 ) such
that QSQ−1 = D is diagonal.
◦ But since QT = Q−1 , if we take R = QT then this condition is the same as saying RT SR = D is diagonal.
This is precisely the condition we require in order to diagonalize a symmetric bilinear form.

◦ Hence: we may diagonalize a symmetric bilinear form over R by computing the (regular) diagonaliza-
tion of the corresponding matrix: this is quite ecient as it only requires nding the eigenvalues and
eigenvectors.

◦ The corresponding diagonalization represents completing the square in the quadratic form via a change
of variables that is orthogonal (i.e., arises from an orthonormal basis), which corresponds geometrically
to a rotation of the standard coordinate axes, possibly also with a reection.

• Example: Find the matrix associated to the quadratic form Q(x, y) = 4x2 − 4xy + 7y 2 , and also nd an
orthonormal basis of R2 that diagonalizes Q.
 
4 −2
◦ We can read o the associated matrix from the coecients as A= .
−2 7
◦ To diagonalize Q, we diagonalize A by nding the eigenvalues and eigenvectors of A.
2
◦ The characteristic polynomial is p(t) = (t − 4)(t − 7) − 4 = t − 11t + 24 = (t − 3)(t − 8), so the eigenvalues
are λ = 3, 8.
◦ We can compute eigenvectors (2, 1) and(−1, 2) for λ = 3, 8 respectively, so upon normalizing these
n o
eigenvectors, we see that we can take γ = √15 (2, 1), √15 (−1, 2)
   
1 2 −1 3 0
◦ Explicitly, with Q= √ , we have [Φ]γ = QT [Φ]β Q = .
5 1 2 0 8
◦ In terms of the quadratic form, this says for x0 =
√1 (2x + y) and y 0 = √1 (−x + 2y), we have Q(x, y) =
5 5
4x2 − 4xy + 7y 2 = 3(x0 )2 + 8(y 0 )2 . Note that by changing basis in this manner, we have eliminated the
cross-term −4xy in the original quadratic form Q.
3 Itcan also be veried directly from the denition that this is a quadratic form via some mild calculations; this also shows the
statement is true even when char(F ) = 2.

7
• Example: Find an orthonormal change of basis that diagonalizes the quadratic form Q(x, y, z) = 5x2 + 4xy +
2 2 3
6y + 4yz + 7z over R .

 
5 2 0
◦ We simply diagonalize the matrix for the corresponding bilinear form, which is A =  2 6 2  . The
0 2 7
3 2
characteristic polynomial is p(t) = det(tI3 − A) = t − 18t + 99t − 162 = (t − 3)(t − 6)(t − 9), so the
eigenvalues are λ = 3, 6, 9.

◦ Computing a basis for each eigenspace yields eigenvectors (2, −2, 1), (−2, −1, 2), and (1, 2, 2) for λ =
3, 6, 9 respectively.
 
2 −2 1
1
◦ Hence, after normalizing, we may take Q =  −2 −1 2 , so that QT = Q−1 and QAQ−1 =
3
  1 2 2
3 0 0
 0 6 0  = D.
0 0 9
1 1 1
◦ Therefore the desired change of basis is x0 = (2x − 2y + z), y 0 = (−2x − y + 2z), z 0 = (x + 2y + 2z),
3 3 3
0 2 0 2 0 2
and with this change of basis it is not hard to verify that, indeed, Q(x, y, z) = 3(x ) + 6(y ) + 9(z ) .

• One application of the existence of such a diagonalization is to classify the conic sections in R2 , and the
quadric surfaces in R3 . These curves (in R2 ) and surfaces (in R3 )

◦ For conics in R2 , the general equation is Ax2 + Bxy + Cy 2 + Dx + Ey + F = 0. By diagonalizing, we

may eliminate the xy term, and so the quadratic term can be put into the form Ax2 + Cy 2 . We then
have various cases depending on the signs of A and C.
◦ If A and C are both zero then the conic degenerates to a line. If one is zero and the other is not,
then by rescaling and swapping variables we may assume A=1 and C = 0, in which case the equation
x2 + Dx + Ey + F = 0 yields a parabola upon solving for y .

◦ If both A, C are nonzero, then we may complete the square to eliminate the linear terms, and then rescale
so that F = −1. The resulting equation then has the form A0 x2 + C 0 y 2 = 1. If A0 , C 0 have the same
0 0
sign, then the curve is an ellipse, while if A , C have the opposite sign, the curve is a hyperbola.

◦ For quadric surfaces in R3 we may likewise eliminate cross-terms by diagonalizing, which yields a reduced
equation Ax + By + Cz 2 + Dx + Ey + F z + G = 0.
2 2

◦ We can then perform a similar analysis (based on how many of A, B, C are zero and the relative signs of
the nonzero coecients and the linear terms) to obtain all of the possible quadric surfaces in R3 .
◦ In addition to the degenerate surfaces (e.g., a point, a plane, two planes), after rescaling the variables,
x2 + y 2 + z 2 = 1), the elliptic, parabolic,
one obtains 9 dierent quadric surfaces: the ellipsoid (e.g.,
2 2 2 2 2
and hyperbolic cylinders (e.g., x + y = 1, y = x , and x − y = 1), the hyperboloid of one sheet
2 2 2 2 2 2
(e.g., z − x − y = 1), the elliptical cone (e.g., z = x + y ), the hyperboloid of two sheets (e.g.,
x2 + y 2 − z 2 = 1), the elliptic paraboloid (e.g., z = x2 + y 2 ), and the hyperbolic paraboloid (e.g.,
z = x2 − y 2 ).
◦ Seven of the quadric surfaces are plotted in Figure 1 (the parabolic and hyperbolic cylinders are omitted).

• All of the conics and quadric surfaces are examples of algebraic varieties, which are the solution sets of
polynomial equations in several variables.

◦ If we have a general quadratic variety (i.e., a quadratic polynomial equation in n variables), we can make
an appropriate translation and rescaling to convert it to the form Q(x1 , . . . , xn ) = 1 or 0, where Q is a
quadratic form.

◦ By diagonalizing the corresponding quadratic form using an orthonormal change of basis (which corre-
sponds to a rotation of the coordinates axes and possibly also a reection), we can then determine the
shape of the variety's graph in Rn .

8
Figure 1: (Top row) Ellipsoid, Circular Cylinder, Hyperboloid of One Sheet, Cone,
(Bottom row) Hyperboloid of Two Sheets, Elliptic Paraboloid, Hyperbolic Paraboloid

• Example: Diagonalize the quadratic form Q(x, y) = 2x2 − 4xy − y 2 . Use the result to describe the shape of
the conic section Q(x, y) = 1 in R2 .
 
2 −2
◦ The matrix associated to the corresponding bilinear form is A= .
−2 −1
◦ The characteristic polynomial is p(t) = det(tI2 − A) = t3 − t + 6 with eigenvalues λ = 3, −2.
◦ We need to diagonalize A using an orthonormal basis of eigenvectors. Since the eigenvalues are distinct,
we simply compute a basis for each eigenspace: doing so yields eigenvectors (−2, 1) and (1, 2) for λ = 3, −2
respectively.
 
1 −2 1
◦ Thus, we may diagonalize A via the orthogonal matrix Q= √ , and the resulting diagonal-
5 1 2
ization is Q(x, y, z) = 3(x0 )2 − 2(y 0 )2 .
◦ In particular, since the change of basis is orthonormal, in the new coordinate system the equation
Q(x, y, z) = 1 reads simply as 3(x0 )2 − 2(y 0 )2 = 1.
√ √
◦ By rescaling again, with x00 = 3x0 , y 00 = 2y 0 , this is equivalent to (x00 )2 − (y 00 )2 = 1, which is a

hyperbola .

• Example: Diagonalize the quadratic form Q(x, y, z) = 2x2 + 4xy − 20xz + 11y 2 + 16yz + 5z 2 . Use the result
3
to describe the shape of the quadric surface Q(x, y, z) = 1 in R .
 
2 2 −10
◦ The matrix associated to the corresponding bilinear form is A =  2 11 8 .
−10 8 5
◦ The characteristic polynomial is p(t) = det(tI3 − A) = t3 − 18t2 − 81t + 1458 = (t + 9)(t − 9)(t − 18) with
eigenvalues λ = 9, 18, −9.
◦ We need to diagonalize A using an orthonormal basis of eigenvectors. Since the eigenvalues are distinct,
we simply compute a basis for each eigenspace: doing so yields eigenvectors (−2, −2, 1), (−1, 2, 2),
(2, −1, 2), for λ = 9, 18, −9 respectively.

9
  
−2 −1 2
1
◦ Thus, we may diagonalize A via the orthogonal matrix Q = −2 2 −1 , and the resulting
3
1 2 2
diagonalization is Q(x, y, z) = 9(x0 )2 + 18(y 0 )2 − 9(z 0 )2 .
◦ In particular, since the change of basis is orthonormal, in the new coordinate system the equation
Q(x, y, z) = 1 reads simply as 9(x0 )2 + 18(y 0 )2 − 9(z 0 )2 = 1.
√
◦ By rescaling again, with x00 = 3x0 , y 00 = 3 2y 0 , z 00 = 3z 0 , this is equivalent to (x00 )2 + (y 00 )2 − (z 00 )2 = 1,
which is a hyperboloid of one sheet .

5.2.3 Deniteness of Real Quadratic Forms

• One of the main properties of a real quadratic form that characterizes its behavior is whether it takes positive
values, negative values, or both:

• Denition: If V is a real vector space, a quadratic form Q on V is positive denite if Q(v) > 0 for every
nonzero vector v ∈ V , it is negative denite if Q(v) < 0 for every nonzero vector v ∈ V , and it is indenite if
Q takes both positive and negative values.

2
◦ Example: If V is a real inner product space, then the square of the norm function ||v|| = hv, vi is a
positive-denite quadratic form on V. Indeed, it is not hard to see that the underlying bilinear pairing
Φ associated with Q is an inner product precisely when Q is a positive-denite quadratic form.
2 2
◦ Example: The quadratic form Q(x, y) = x + 2y is positive denite, since Q(x, y) > 0 for all (x, y) 6=
(0, 0).
◦ Example: The quadratic form Q(x, y, z) = −2x2 − 2xy − 5y 2 = −(x − y)2 − (x + 2y)2 is negative denite,
since the second expression shows that Q(x, y) < 0 for all (x, y) 6= (0, 0).

◦ Example: The quadratic form Q(x, y) = xy is indenite, since Q(1, 1) = 1 and Q(−1, 1) = −1, so Q
takes both positive and negative values.

◦ There are also useful weaker versions of these conditions: we say Q is positive semidenite if Q(v) ≥ 0
for all v∈V and negative semidenite if Q(v) ≤ 0 for all v ∈V.
◦ Example: The quadratic form Q(x, y) = x2 is positive semidenite, since Q(x, y) ≥ 0 for all (x, y), but
it is not positive denite because Q(0, 1) = 0.
◦ It is easy to see that Q is positive (semi)denite if and only if −Q is negative (semi)denite, so for
example by the above we see that Q(x, y) = −x2 is negative semidenite.

◦ Notice that every nonzero quadratic form lies in exactly one of the following ve classes: positive-denite,
positive-semidenite but not positive-denite, indenite, negative-semidenite but not negative denite,
negative denite.

• By diagonalizing a quadratic form, we can easily determine its deniteness:

• Proposition (Deniteness and Eigenvalues): If Q is a quadratic form on a nite-dimensional real vector space
V with associated matrix A, then Q is positive denite if and only if all eigenvalues of A are positive, Q is
positive semidenite if and only if all eigenvalues of A are nonnegative, Q is negative denite if and only all
eigenvalues of A are negative, Q is negative semidenite if and only if all eigenvalues of A are nonpositive,
and Q is indenite if and only if it has both a positive and a negative eigenvalue.

◦ Note that all of the eigenvalues of A are real, by the spectral theorem.

◦ Proof: Observe that deniteness is unaected by changing basis, because each of the deniteness condi-
tions Q(v) > 0, Q(v) ≥ 0, Q(v) < 0, Q(v) ≤ 0 are statements about all vectors v in the vector space
V.
◦ Therefore, we may diagonalize Q without aecting its deniteness. After diagonalizing, we have an
expression of the form Q(x1 , x2 , . . . , xn ) = a1 x21 + a2 x22 + · · · + an x2n .
◦ If any of the coecients are negative, then Q necessarily takes negative values (specically, if ai < 0,
then Q evaluated at the ith standard unit vector will be ai ).

10
 ◦ Likewise, if any of the coecients are positive then Q necessarily takes positive values, and if any
coecients are zero or have opposite signs then Q takes the value 0 at some nonzero vector.

◦ Assuming we use an orthogonal diagonalization, then since the coecients ai are simply the eigenvalues
of A, all of the claimed statements then follow immediately.

◦ Explicitly, if Q takes only positive values on nonzero vectors then no coecients ai can be zero or negative
(so they are all positive), if Q takes only nonnegative values then no coecients ai can be negative, and
likewise in the other two cases.

• Example: Determine the deniteness of the quadratic form Q(x, y) = 2x2 + 4xy + 5y 2 on R2 .
 
2 2
◦ The associated matrix is , whose characteristic polynomial is p(t) = (t − 2)(t − 5) − (2)(2) =
2 5
t2 − 7t + 6 = (t − 6)(t − 1), so its eigenvalues are λ = 1, 6.
◦ Since both eigenvalues are positive, Q is positive denite .

• Example: Determine the deniteness of the quadratic form Q(x, y) = 3x2 − 6xy − 5y 2 on R2 .
 
3 −3
◦ The associated matrix is , whose characteristic polynomial is p(t) = (t−3)(t+5)−(−3)(−3) =
−3 −5
t2 − 2t − 24 = (t + 6)(t − 4), so its eigenvalues are λ = −6, 4.
◦ Since one eigenvalue is positive and the other is negative, Q is indenite .

• Example: Determine the deniteness of the quadratic form Q(x, y) = 3x2 − 2xy + 4xz + 3y 2 − 4yz + 2z 2 on
3
R .
 
3 −1 2
◦ The associated matrix is  −1 3 −2 , whose characteristic polynomial is p(t) = t3 − 8t2 + 12t =
2 −2 2
t(t − 2)(t − 6), so its eigenvalues are λ = 0, 2, 6.
◦ Since one eigenvalue is zero and the others are positive, Q is positive semidenite .

• We can also view deniteness as a property of symmetric matrices themselves by considering the deniteness of
the associated quadratic form. In this lens, we can give another way to identify deniteness using determinants:

• Theorem (Sylvester's Criterion): Suppose A is ann × n symmetric real matrix. For each 1 ≤ k ≤ n, dene
the k th principal minor Ak to be the upper-left k × k corner of A. Then A is positive denite if and only if
det(Ak ) > 0 for all k .

◦ There is also a version for checking positive semideniteness, but it is more complicated: one requires
det(Ak ) ≥ 0 for every k×k symmetric submatrix of A, rather than just the principal minors.

◦ Also, since A is positive denite if and only if −A is negative denite, one can also use Sylvester's criterion
to identify negative denite matrices.

◦ We will not prove Sylvester's criterion, although it is not hard to see that the given condition is necessary,
since if A is positive denite we must have xT Ax > 0 for all vectors x = [x1 x2 · · · xk 0 · · · 0]: this
means the matrix Ak must also be positive denite and therefore must have positive determinant.
 
2 −1 2
• Example: Determine the deniteness of the matrix A =  −1 4 2 .
2 2 5
 
  2 −1 2
2 −1
◦ The principal minors are [2], , and  −1 4 2 , whose determinants respectively are
−1 4
2 2 5
2, 7, and 3.

◦ Since all of the principal minors have positive determinants, the given matrix is positive denite .

◦ Remark: To four decimal places, the eigenvalues are 6.8004, 4.0917, and 0.1078, so we see A is indeed
positive denite.

11
5.2.4 Quadratic Forms Over Rn : The Second Derivatives Test
• We can also use quadratic forms and an analysis of deniteness to establish the famous second derivatives
test from multivariable calculus:

• Theorem (Second Derivatives Test in Rn ): Suppose f is a function of n variables x1 , . . . , xn that is twice-

dierentiable and P is a critical point of f , so that fxi (P ) = 0 for each i. Let H be the Hessian matrix, whose
(i, j)-entry is the second-order partial derivative fxi xj (P ). If all eigenvalues of H are positive then f has a
local minimum at P , if all eigenvalues of H are negative then f has a local maximum at P , if H has at least
one eigenvalue of each sign then f has a saddle point at P , and in all other cases (where H has at least one
zero eigenvalue and does not have one of each sign) the test is inconclusive.

◦ Proof (outline): By translating appropriately, assume for simplicity that P is at the origin.
2
◦ Then by the multivariable version of Taylor's theorem in R , the function xn ) − f (P ) will be
f (x1 , . . . , P
closely approximated by its degree-2 Taylor polynomial T , which has the form T = 1≤i≤j≤n ai,j xi xi ,
(
fxi ,xi (P )/2 for i=j
where ai,j = .
fxi ,xj (P ) for i 6= j
f (x1 , . . . , xn ) − T − f (P )
◦ Specically, Taylor's theorem says that lim = 0, which we can write
(x1 ,...,xn )→P x21 + · · · + x2n
more compactly as f (x1 , . . . , xn ) − f (P ) = T + O(x21 + · · · + x2n ).
◦ Now observe T is a quadratic form whose associated bilinear form has matrix H/2. By using an or-
thonormal change of basis, we may diagonalize this quadratic form, and the entries on the diagonal of
the diagonalization are the eigenvalues of H/2.
0 0 1 0 2 1 0 2
◦ If x1 , . . . , xn is the new coordinate system, this means f (x1 , . . . , xn )−f (P ) = λ1 (x1 ) +· · ·+ λn (xn ) +
2 2
0 2 0 2
O[(x1 ) + · · · + (xn ) ].
1 0 2 1 0 2
◦ If all of the λi are positive (i.e., when λ1 (x1 ) + · · · + λn (xn ) is positive denite) then the error term
2 2
is smaller than the remaining terms, and so we see that f (x1 , . . . , xn ) − f (P ) > 0 suciently close to P,
meaning that P is a local minimum.
1 0 2 1 0 2
◦ Likewise, if all of the λi 2 λ1 (x1 ) + · · · + 2 λn (xn ) is negative denite) then the
are negative (i.e., when
error term is smaller than the remaining terms, and so we see that f (x1 , . . . , xn ) − f (P ) < 0 suciently
close to P, meaning that P is a local maximum.

◦ λi and one negative eigenvalue λj (i.e., when 21 λ1 (x01 )2 + · · · +

If there is at least one positive eigenvalue
1 0 2 0
2 λn (xn ) is indenite) then approaching P along the direction xi yields values of f less than P , while
0
approaching P along the direction xj yields values of f greater than P , so P is a saddle point.

1 0 2 1 0 2
◦ The other cases (i.e., when
2 λ1 (x1 ) +· · ·+ 2 λn (xn ) is positive or negative semidenite) are inconclusive:
0
if λ1 = 0 then as we approach P along the direction x1 , the quadratic form T is constant, and so the sign
of f (x1 , . . . , xn ) − f (P ) along that path will be determined by the error term. Except in the case where
the function is known to take both positive and negative values (guaranteeing a saddle point), any of the
other possible behaviors not ruled out by the existence of positive or negative eigenvalues could occur.

◦ As explicit examples, consider the functions f1 = x21 + x42

f2 = x21 − x42 : for both functions H has
and
a single positive eigenvalue and a zero eigenvalue, but at (0, 0) f1 has a local minimum while f2 has a
4 4 4 4 4 4
saddle point. Likewise, for g1 = x1 + x2 , g2 = x1 − x2 , and g3 = −x1 − x2 , we see that for all three
functions H has both eigenvalues equal to zero, but at (0, 0) g1 has a local minimum, g2 has a saddle
point, and g3 has a local maximum.

• Example: Classify the critical point at (0, 0) for the function f (x, y) = 2x2 + xy + 4y 2 .

◦ We compute the Hessian matrix: we have

  fxx = 4, fxy = fyx = 1, and fyy = 8, so evaluating these at
4 1
(0, 0) yields H= .
1 8
√
◦ The characteristic polynomial of H is p(t) = det(tI2 − H) = t2 − 12t + 31, whose roots are λ=6± 5.
◦ Since the eigenvalues are both positive, the critical point is a local minimum .

12
 • Example: Classify the critical point at (0, 0) for the function f (x, y) = x2 + 3xy − 6y 2 + x5 y 3 .
3 3
◦ We compute the Hessian matrix: we have
 fxx = 2+20x
 y , fxy = fyx = 3+15x4 y 2 , and fyy = −6+6x5 y ,
2 3
so evaluating these at (0, 0) yields H= .
3 −6
◦ The characteristic polynomial of H is p(t) = det(tI2 − H) = t2 − 4t − 21 = (t − 7)(t + 3).
◦ Since the eigenvalues are −7 and 3, there is an eigenvalue of each sign, so the critical point is a
saddle point .

• Example: Classify the critical point at (0, 0, 0) for the function f (x, y, z) = 3x2 + 2xy − xz + y 2 − yz + z 2 .

◦ We compute the Hessian matrix: we have

 fxx = 6, fxy = fyx = 2, fxz = fzx = −1, fyy = 2, fyz = fzy =
6 2 −1
−1, and fzz = 2, so H= 2 2 −1 .
−1 −1 2
◦ The characteristic polynomial of
√ H is p(t) = det(tI2 − H) = t3 − 10t2 + 22t − 12 = (t − 2)(t2 − 8t + 6),
whose roots are λ = 2, 4 ± 10.
◦ Since the eigenvalues are all positive, the critical point is a local minimum .

5.2.5 Quadratic Forms Over Rn : Sylvester's Law of Inertia

• We now discuss another fundamental result (which was, in fact, somewhat implicit in our earlier discussion
of conics) regarding the possible diagonal entries for the diagonalization of a real quadratic form.

◦ By making dierent choices for the matrix P (e.g., by rescaling it or selecting dierent row operations),
we may obtain dierent diagonalizations of a given real quadratic form.

◦ For example, with the quadratic form Q(x, y) = x2 + 2y 2 , which is already diagonal, if we change basis
0 0
to x = x/2, y = y/3, then we obtain Q(x, y) = 4(x0 )2 + 18(y 0 )2 .
◦ Indeed, it is clear that given any diagonalization, if we scale the ith row of the diagonalizing matrix by
the scalar α, then the coecient of the ith variable will be scaled by α2 .
◦ Hence, by rescaling, we may change any positive coecient to an arbitrary positive value and any negative
coecient to an arbitrary negative value.

◦ It turns out that this is essentially the only possible change we may make to the diagonalization over R.

• Theorem (Sylvester's Law of Inertia): Suppose V is a nite-dimensional real vector space and Q is a quadratic
form on V. Then the numbers of positive diagonal entries, zero diagonal entries, and negative diagonal entries
in any diagonalization of Q is independent of the choice of diagonalization.

◦ The idea of this result is that we may decompose V as a direct sum of three spaces: one on which Q acts
as a positive-denite quadratic form (corresponding to the positive diagonal entries), one on which Q
acts as the zero map (corresponding to the zero entries), and one on which Q acts as a negative-denite
quadratic form (corresponding to the negative diagonal entries).

◦ Since this decomposition of V depends only on Q, these three spaces (and thus their dimensions) are
independent of the choice of diagonalizing basis, and so the number of positive, zero, and negative
diagonal entries in any diagonalization is necessarily xed.

◦ Proof: Since we are over a eld of characteristic not 2, we may equivalently work with the symmetric
bilinear form Φ associated to Q.
◦ Let V0 be the subspace of V given by V0 = {v0 ∈ V : Φ(v0 , v) = 0 for all v ∈ V }: then Φ acts as the
zero map on V0 . Now write V = V0 ⊕ V1 : we claim that Φ is nondegenerate on V1 .
◦ To see this, suppose y ∈ V1 has Φ(y, v1 ) = 0 for all v1 ∈ V : then for any w ∈ V we may write
w = v0 + v1 for vi ∈ Vi , in which case Φ(y, w) = Φ(y, v0 ) + Φ(y, v1 ) = 0. But this would imply y ∈ V0
whence y = 0.

13
 ◦ Now we will show that if Φ is nondegenerate on V1 , then V1 decomposes as a direct sum V1 = V+ ⊕ V− ,
where Φ is positive-denite on V+ and negative-denite on V− .
◦ Let V+ be the maximal subspace of V1 on which Φ is positive-denite (since the condition is dened
only on individual vectors, this subspace is well-dened), and dene V− = {w ∈ V : Φ(v+ , w) =
0 for all v+ ∈ V+ }. Then by an application of Gram-Schmidt4 (via Φ, rather than an inner product), we
see that V1 = V+ ⊕ V− .
◦ It remains to show that Φ is negative-denite on V− , so let z ∈ V− be nonzero. Then by assumption, Φ
is not positive-denite on V+ ⊕ hzi, so there exists some nonzero v = v+ + αz with v+ ∈ V+ and α ∈ R
such that Φ(v, v) ≤ 0.
◦ We cannot haveα = 0 since then positive-deniteness would imply v+ = 0. Since Φ(v, v) = Φ(v+ , v+ ) +
1
2αΦ(v+ , z) + α Φ(z, z) = Φ(v+ , v+ ) + α2 Φ(z, z), we have Φ(z, z) = 2 [Φ(v, v) − Φ(v+ , v+ )].
2
α
◦ Then both terms are less than or equal to zero, and both cannot be zero. Hence Φ(z, z) < 0 for all
nonzero z ∈ V− and so Φ is negative-denite on V− .

◦ The desired result then follows from the direct sum decomposition V = V0 ⊕ V+ ⊕ V− : if we select any
diagonalization, then the restriction to the subspace generated by the basis vectors with diagonal entries
0, positive, negative (respectively) is trivial, positive-denite, negative-denite (respectively), and thus
the number of diagonal elements is at least dim(V0 ), dim(V+ ), dim(V− ) (respectively). But since the
total number of diagonal elements is dim(V ) = dim(V0 ) + dim(V+ ) + dim(V− ), we must have equality
everywhere.

◦ Hence the numbers of positive diagonal entries, zero diagonal entries, and negative diagonal entries in
any diagonalization of Q is independent of the choice of diagonalization, as claimed.

• We will also mention that there is some classical terminology associated with this result: the index of a
quadratic form is the number of positive diagonal entries (in any diagonalization) and the signature is the
dierence between the number of positive and negative diagonal entries.

◦ Equivalently, by our discussion of the spectral theorem, the index is equal to the number of positive
eigenvalues of the matrix associated to the symmetric bilinear form, while the signature is the dierence
between the number of positive eigenvalues and the number of negative eigenvalues.

◦ Remark: Some authors instead refer to the triple (dim V+ , dim V− , dim V0 ), or some appropriate permu-
tation, as the signature of the quadratic form. These three values themselves are called the invariants
of the form, and the value of any two of them (along with the dimension of the ambient space V) is
sucient to nd the value of the other one.

◦ For nondegenerate forms, where there are no 0 entries (so dim V0 = 0), the dimension of the space along
with the value of dim V+ − dim V− is sucient to recover the two values.

◦ Example: The quadratic form Q(x, y, z) = x2 − y 2 − z 2 on R3 has index 1 and signature −1.
2 2 3
◦ Example: The quadratic form Q(x, y, z) = x − z on R has index 1 and signature 0.
2 2 2 3
◦ Example: The quadratic form Q(x, y, z) = 5x + 4xy + 6y + 4yz + 7z on R has index 3 and signature
3, since we computed its diagonalization to have diagonal entries 3, 6, 9.

• Example: Find the index and signature of the quadratic form Q(x, y, z) = −x2 − 8xy + 4xz − y 2 + 4yz + 2z 2 .
 
−1 −4 2
◦ The matrix associated to the corresponding bilinear form is A =  −4 −1 2 .
2 2 2
◦ The characteristic polynomial is p(t) = det(tI3 − A) = t3 − 27t + 54 = (t − 3)2 (t + 6).
◦ Thus, since the eigenvalues are λ = 3, 3, −6, we see that the diagonalization will have two positive
diagonal entries and one negative diagonal entry.

◦ This means that the index is 2 and the signature is 1 .

4 The argument here is the same as for showing that dim(W ) + dim(W ⊥ ) = dim(V ) for an inner product. The Gram-Schmidt
algorithm does not use the positive-deniteness of the inner product (it requires only linearity and symmetry), so the same argument
also works for any bilinear form.

14
 • As a corollary of Sylvester's law of inertia, we can read o the shape of a conic section or quadric surface (in
all nondegenerate cases, and also in many degenerate cases) simply by examining the signs of the eigenvalues
of the underlying quadratic form.

• Example: Determine the shape of the quadric surface 13x2 − 4xy + 10y 2 − 8xz + 4yz + 13z 2 = 1.
 
13 −2 −4
◦ If Q(x, y, z) is the quadratic form above, the bilinear form has associated matrix A =  −2 10 2 .
−4 2 13
◦ The characteristic polynomial is p(t) = det(tI3 − A) = t3 − 144t2 + 6480t − 93312 = (t − 36)2 (t − 72).
◦ This means, upon diagonalizing Q(x, y, z), we will obtain the equation 36(x0 )2 + 36(y 0 )2 + 72(z 0 )2 = 1.
This is the equation of an ellipsoid .

◦ Note that the only information we needed here was the fact that all three eigenvalues were positive to
make this observation: the quadric surfaces Q(x, y, z) = 1 that are ellipsoids are precisely those for which
Q(x, y, z) is a positive-denite quadratic form.

• We will close our discussion by observing that the study of quadratic forms touches on nearly every branch of
mathematics: we have already examined some of its ties to linear algebra (in the guise of bilinear forms and
diagonalization), analysis (in the classication of critical points), and geometry (in the analysis of quadratic
varieties and the action of matrices on quadratic forms).

◦ We will not discuss it much here, since the requisite tools do not really belong to linear algebra, but the
study of quadratic forms over Q turns out to be intimately tied with many topics in number theory.

◦ A very classical problem in elementary number theory is to characterize, in as much detail as possible,
the integers represented by a particular quadratic form. For example: which integers are represented by
the quadratic form Q(x, y) = x2 + y 2 (i.e., which integers can be written as the sum of two squares)?

◦ This family of problems, while seemingly quite simple, is actually intimately related to a number of very
deep results in modern number theory, and (historically speaking) was a major motivating force in the
development of a branch of algebraic number theory known as class eld theory.

5.3 Singular Values and Singular Value Decomposition

• Diagonalization is a very useful tool, but it suers from two main drawbacks: rst, not all linear transfor-
mations T : V → V are diagonalizable, and second, we cannot diagonalize general linear transformations
T : V → W when V and W are dierent.

◦ The Jordan canonical form allows us to give a near diagonalization for non-diagonalizable linear trans-
formations T :V →V. However, it still cannot be used to describe general transformations T : V → W.

• We will now discuss a decomposition that is in some sense a modied diagonalization, called singular value decomposition,
that extends naturally to transformations T :V →W when V and W are dierent.

5.3.1 Singular Values and Singular Value Decomposition

• The main idea of singular value decomposition is as follows: if we have a linear transformation T :V →W
where V and W are nite-dimensional inner product spaces, then we may construct orthonormal bases β of V
and γ of W such that the associated matrix A = [T ]γβ has its only nonzero entries in the diagonal positions
ai,i .

◦ This procedure combines the ideas of diagonalization, in that we obtain a representation of T by an

essentially diagonal matrix (up to not being square), and the QR factorization, in that we perform
orthonormal changes of basis to simplify the form of the transformation.

◦ The main idea is to note that if T :V →W is linear, then T ∗T : V → V is Hermitian, since (T ∗ T )∗ =

∗ ∗∗ ∗
T T =T T again.

15
 2
◦ Moreover, the quadratic form QT ∗ T (v) = hT ∗ T v, vi = hT v, T vi = ||T v|| on V is positive semidenite
2
since ||T v|| ≥ 0 for all v. Thus, from our characterization of the deniteness of quadratic forms, we see
that all of the eigenvalues of T ∗T are nonnegative.
∗
◦ Therefore, since T T is a Hermitian operator with nonnegative eigenvalues, it can be orthogonally diag-
onalized with respect to an orthonormal basis β, and the diagonal entries of its diagonalization D are
σ12 , σ22 , . . . , σn2 for some nonnegative real numbers σ1 ≥ σ2 ≥ · · · ≥ σn ≥ 0.
◦ We can then use the orthonormal basis β of V to write down an appropriate orthonormal basis γ for W
so that [T ]γβ will have its only nonzero entries in the positions ai,i .

• To make this precise, we introduce some terminology:

• Denition: If V and W are nite-dimensional inner product spaces and T : V → W is linear, the singular values
of T are the nonnegative real numbers σ1 ≥ σ2 ≥ · · · ≥ σn ≥ 0 such that σ12 , σ22 , . . . , σn2 are the eigenvalues of
T ∗T .

◦ We have previously shown that ker(T ∗ T ) = ker(T ), so the rank of T ∗T equals the rank of T . Thus, if
T has rank r, the singular values σ1 , . . . , σr will be positive and the remaining ones σr+1 , . . . , σn will be
zero.

◦ It is a matter of convention whether to include the zero eigenvalues on the list of singular values.

◦ For a matrix A, A∗ A. If β is an orthonormal basis of V and γ

the singular values are the eigenvalues of
γ
is an orthonormal basis of W , then the singular values of T are the singular values of [T ]β . (Note that
∗
if we make an orthonormal change of basis in V or W , the eigenvalues of A A are not changed.)

 
2 2
 2 2 
• Example: Find the singular values of the matrix A =   −1 1 .


1 −1
 
  2 2  
∗ 2 2 −1 1  2 2  10 6
◦ We have A A =   = with characteristic polynomial p(t) =
2 2 1 −1  −1 1  6 10
1 −1
det(tI3 − A) = (t − 10)(t − 10) − (6)(6) = t2 − 20t + 64 = (t − 4)(t − 16).
◦ Since the eigenvalues of A∗ A are λ = 16, 4, we see that the singular values of A are 4, 2 .

 
1−i 3i
• Example: Find the singular values of the matrix A= 2+i i .
2+i 1 − 2i
 
 1−i 3i   
∗ 1+i 2−i 2−i −2  = 12
◦ We have A A =  2+i i with characteristic polyno-
−3i −i 1 + 2i 15 −2
2 + i 1 − 2i
2
mial p(t) = det(tI2 − A) = (t − 12)(t − 15) − (−2) = (t − 11)(t − 16).
√
◦ Since the eigenvalues of A∗ A are λ = 16, 11, we see that the singular values of A are 4, 11 .

 
1+i 3 1 i
• Example: Find the singular values of the matrix A= .
1+i 1 2 −i

  
1+i 1+i   4 4 − 4i 3 − 3i 0
∗
 3 1  1+i 3 1 i  4 + 4i 10 5 2i 
◦ We have A A =   =  with charac-
 1 2  1 + i 1 2 −i  3 + 3i 5 5 −i 
i −i 0 −2i i 2
4 3 2 2
teristic polynomial p(t) = det(tI4 − A) = t − 21t + 68t = t (t − 4)(t − 17).
√
◦ Since the eigenvalues of A∗ A are λ = 17, 4, 0, 0, we see that the singular values of A are 17, 2, 0, 0 .

16
• Our general result is that we can use the singular values of a matrix to write down a matrix associated to T
with respect to orthonormal bases that is particularly nice:

• Theorem (Singular Value Bases): V and W are nite-dimensional inner product spaces and T :
Suppose
V →W is a linear transformation of rank r with singular values σ1 ≥ σ2 ≥ · · · ≥ σr > 0. Then there exist
orthonormal bases {v1 , . . . , vn } of V and {w1 , . . . , wm } of W such that T (v1 ) = σ1 w1 , T (v2 ) = σ2 w2 , ... ,
T (vr ) = σr wr , and T (vr+1 ) = T (vr+2 ) = · · · = T (vn ) = 0.

◦ Proof: Recall that we proved in our discussion of least squares that T ∗T and T have the same rank, so
∗
T T also has rank r.
◦ Additionally, as noted above, T ∗ T is a Hermitian operator on V , so by the spectral theorem, there exists
∗
an orthonormal basis {v1 , . . . , vn } of V consisting of eigenvectors of T T , where the basis is ordered the
2 2 2
associated eigenvalues are σ1 , σ2 , . . . , σr , 0, . . . , 0.

◦ Then for 6 j , we have hT (vi ), T (vj )i = hvi , T ∗ T vj i = σi2 hvi , vj i = 0

i= since vi , vj are orthogonal, so
T (vi ) and T (vj ) are also orthogonal.
2
◦ Additionally, for 1 ≤ i ≤ r we have ||T (vi )|| = hT (vi ), T (vi )i = hvi , T ∗ T vi i = σi2 hvi , vi i = σi2 , so since
σi > 0 we see ||T (vi )|| = σi .
◦ Therefore, if we dene wi = T (vi )/σi for each 1 ≤ i ≤ r, we see that {w1 , . . . , wr } is an orthonormal
set. By extending this set to an orthonormal basis {w1 , . . . , wm } of W , we obtain the required result.

• To calculate a singular value decomposition, we simply need to compute an orthonormal basis of eigenvectors
{v1 , . . . , vn } for T ∗T , and then compute their images under T and divide by the singular values (potentially
then extending to a basis of W using Gram-Schmidt) to get the basis {w1 , . . . , wm }.

◦ To compute T ∗, A = [T ]δα where α and δ are arbitrary orthonormal bases for

if we set V and W
∗ α ∗
(respectively) then [T ]δ is simply the adjoint matrix A .

• Example: Find singular value bases for the map T : P1 (R) → P2 (R) with T (p) = p(0), under the inner product
´1
hf, gi = 0
f (x)g(x) dx on both P1 (R) and P2 (R).
∗
√
◦ √ T . Using√Gram-Schmidt
First, we must compute we construct orthonormal bases α = {1, 3(2x − 1)}
for P1 (R) and δ = {1, 3(2x − 1), 5(6x2 − 6x + 1)} for P2 (R).
 √ 
√ √ 1 − 3
◦ Then since T (1) = 1 and T ( 3(2x − 1)) = − 3, we obtain [T ]δα =  0 0 , hence because α and
0 0
 
1
√ 0 0
δ are orthonormal bases, we have [T ∗ ]α
δ = .
− 3 0 0
◦ To nd the desired singular value bases, we compute an eigenbasis for
√ T ∗T , whose associated matrix is
√
 
  1 − 3  
1
√ 0 0 1
√ − 3
[T ∗ T ]α ∗ α δ
α = [T ]δ [T ]α =
 0 0 = .
− 3 0 0 − 3 3
0 0
 
1 1
√
◦ The eigenvalues are λ = 4, 0 with corresponding orthonormal eigenvectors [v1 ]α = and
2 − 3
 √
√ √ √

1 3 1  1 √ 
[v2 ]α = , whence v1 = −1 + 3 · 3(2x − 1) = 3x − 2 and v2 = 3 + 1 · 3(2x − 1) =
2 1 2 2
√
x 3.
√
◦ The nonzero singular value is σ1 = 4 = 2, so that w1 = T (v1 )/σ1 = −1, and then using Gram-Schmidt
we can extend this to an orthonormal basis of W (indeed, we can just reuse the other vectors from δ
above).
√ √ √
◦ Thus we obtain bases {v1 , v2 } = {3x − 2, x 3} and {w1 , w2 , w3 } = {−1, 3(2x − 1), 5(6x2 − 6x + 1)} .

• We can recast the theorem above in terms of matrices, as follows:

17
• Theorem (Singular Value Decomposition): Suppose F = R or C and that A ∈ Mm×n (F ) has rank r . If
σ1 ≥ σ2 ≥ · · · ≥ σr > 0 are the nonzero singular values of A, then A can be written as a matrix product
A = U ΣV ∗ where U ∈ Mn×n (F ) and V ∈ Mm×m (F ) are unitary and Σ ∈ Mn×m (F ) is the matrix whose rst
r diagonal entries are σ1 , . . . , σr and whose remaining entries are 0.

◦ Proof: Let T :V →W be the linear transformation with T (v) = Av.

◦ By the theorem above, we have orthonormal bases β = {v1 , . . . , vn } of F n and γ = {w1 , . . . , wm } of F m
such that T (v1 ) = σ1 w1 , T (v2 ) = σ2 w2 , ... , T (vr ) = σr wr , and T (vr+1 ) = T (vr+2 ) = · · · = T (vn ) = 0.

◦ This means the associated matrix [T ]γβ is the n × m matrix Σ whose rst r diagonal entries are σ1 , . . . , σ r
and whose remaining entries are 0.

◦ Now let α be the standard basis of Fn and δ be the standard basis of Fm and note that [T ]δα = A.
Furthermore, since β is orthonormal the change-of-basis matrix V = [I]α
β is unitary so [I]βα = V −1 = V ∗ ,
and since γ is orthonormal the change-of-basis matrix U = [I]δγ is also unitary.

◦ Then A= [T ]δα = [I]δγ [T ]γβ [I]βα = U ΣV ∗

, as claimed.

 
0 6
• Example: Find the singular values, and a singular value decomposition, of A= .
6 5
 
36 30
◦ We have A∗ A = with characteristic polynomial p(t) = (t − 36)(t − 61) − (30)(30) = t −
2
30 61
√ √
97t + 1296 = (t − 16)(t − 81), so the singular values of A are σ1 = 81 = 9 and σ2 = 16 = 4.
 
∗ 2
◦ We can then nd a basis for the 81-eigenspace of A A as and a basis for the 16-eigenspace as
   3  
−3 1 2 1 −3
, so after normalizing we can take v1 = √ and v2 = √ .
2 13 3 13 2
   
1 1 2 1 1 3
◦ We also have w1 = Av1 = √ and w2 = Av2 = √ ; as expected we see that
9 13 3 4 13 −2
{w1 , w2 } is an orthonormal set (and in fact an orthonormal basis) of R2 .
     
1 2 3 9 0 ∗ 1 2 −3
◦ Putting all of this together, we get U = √ , Σ = , and V = √ .
13 3 −2 0 4 13 3 2
 
2 2
 2 2 
• Example: Find a singular value decomposition of A =  −1
.
1 
1 −1
 
10 6
◦ We previously found that the eigenvalues of A∗ A = are λ = 16, 4 and so the singular values
6 10
of A are σ1 = 4 and σ2 = 2.
 
∗ 1
◦ We can then calculate a basis for the 16-eigenspace of A A as and a basis for the 4-eigenspace
1
     
−1 1 1 1 −1
of A∗ A as , so after normalizing we can take v1 = √ and v2 = √ .
1 2 1 2 1
   
1 0
1 1  1  1 1  0 
◦ We also have w1 = Av1 = √   and w2 = Av2 = √  ; as expected we see that
4 2 0  2 2  1 
0 −1
{w1 , w2 } is an orthonormal set.

◦ By using Gram-Schmidt, we can extend

    {w1 , w2 } to an orthonormal basis {w1 , w2 , w3 , w4 } of R4 with
1 0
1  −1  1  0 
w3 = √   and w4 = √  .
2  0  2 1 

0 1

18
    
1 0 1 0 4 0  
1  1 0 −1 0   0 2  1 1 −1
◦ Thus, we get U= √  ,Σ=   , and V∗ = √ .
2 0 1 0 1   0 0  2 1 1
0 −1 0 1 0 0
◦ Remark: Note that the singular value decomposition is not unique here, since we could choose other
vectors w3 , w4 to complete the orthonormal basis of R4 .
 
1+i 3 1 i
• Example: Find a singular value decomposition of the matrix A= .
1+i 1 2 −i

◦ We previously calculated the eigenvalues of

√ A∗ A as 17, 4, 0, and 0 so that the positive singular values
are σ1 = 17 and σ2 = 2.
◦ We can then calculate an orthonormal basis of eigenvectors with
    λ = 17, 4, 0,0 respectively
  as
5 + 5i −1 − i 1+i −2 + 2i
1  11i  1  3i
, v3 = 1  −i , v4 = √1 
    −1 
v1 = √  , v2 = √  .
221  7i  52  −4i  2  0  34  4 
1 5 1 −3i
   
1 1 3i 1 1 2i
◦ We also have w1 = √ Av1 = √ and w2 = Av2 = √ ; as expected we see that
17 13 2i 2 13 −3i
{w1 , w2 } is an orthonormal set.
   √ 
1 3i 2i 17 0 0 0
◦ Putting all of this together, we get U = √ , Σ = , and V∗ =
13 2i−3i 0 2 0 0
 √ √ √ 
(5 + 5i)/
√ 221 (−1 − √
i)/ 52 (1 + i)/2 (−2 + 2i)/
√ 34
 11i/ 221
 √ 3i/ √52 −i/2 −1/√ 34 
.
 7i/√ 221 −4i/√ 52 0 4/ √34 
1/ 221 5/ 52 1/2 −3i/ 34
◦ Remark: Note that the singular value decomposition is not unique here, since for example we could have
chosen any other orthonormal basis v3 , v4 of the 0-eigenspace of A∗ A.

• The singular value basis and associated decomposition have a convenient geometric interpretation in terms of
the action of the transformation T :V →W on the unit sphere ||v|| = 1 in V.
2 2
◦ 
To illustrate, consider the linear transformation T : R → R
 with associated standard matrix A =
0 6
from the rst example above.
6 5
◦ The image of the unit circle ||v|| = 1 (i.e., x2 + y 2 = 1) under T is an ellipse, shown below:

19
 ◦ We can see quite clearly from the second picture that the vectors Av1 and Av2 give the principal axes
of the ellipse.

◦ This observation can be veried algebraically from the facts that β = {v1 , v2 } and γ = {w1 , w2 } are

2 γ 9 0
orthonormal bases of R and the fact that [T ]β is the matrix : then the image 9aw1 + 4bw2 of
0 4
2 2 2 2
any linear combination av1 + bv2 on the unit circle (i.e., with a + b = 1) has norm 81a + 16b , and
the norm is clearly maximized when b = 0 and minimized when a = 0.

◦ This means that the major axis of the ellipse is parallel to w1 and has length σ1 , while the minor axis
of the ellipse is parallel to w2 and has length σ2 .

◦ It is not hard to see that analogous results hold in higher dimensions, for the same reasons: in general,
n m
the image of the unit sphere ||v|| = 1 under a linear transformation T : R → R of rank r will be an
r-dimensional ellipsoid whose principal axes are the vectors σ1 w1 , σ2 w2 , ... , σr wr .
• This geometric interpretation of singular value decomposition has many practical applications, such as per-
forming principal component analysis and doing data compression.

◦ The main idea is that for an m×n matrix A with singular values σ1 , . . . , σr and corresponding orthonormal
{v1 , . . . , vn } and {w1 , . . . , wm }, if we multiply out the matrix product A = U ΣV ∗ , we can rephrase
bases
∗ ∗ ∗
the singular value decomposition as giving a sum A = σ1 v1 w1 + σ2 v2 w2 + · · · + σr vr wr of a total of r
∗
m × n matrices vi wi each of which has rank 1.
◦ Therefore, if we want to approximate A by a matrix of rank less than r, the best approximation will
be given by deleting the terms of the sum above that have the smallest norm, which are the terms with
smallest σi .
◦ 5
In other words , the best approximation to A by a matrix of rank d is obtained by taking the initial
terms of the singular value sum above: σ1 v1 w1∗ + σ2 v2 w2∗ + · · · + σd vd wd∗ .
◦ In the situation where we have a set of data that is high-dimensional (i.e., lies inside Fn where n is
large), this gives an explicit procedure for projecting onto a smaller-dimensional subspace that loses as
little information as possible.

• We can illustrate these ideas by calculating the singular value decomposition of a matrix representing the 2-
dimensional grid of color intensity from a black-and-white photograph (taken from the standard set of sample
data included with Mathematica).

◦ The photograph used here is 512 pixels by 512 pixels, corresponding to a 512 × 512 matrix A.
◦ We can then give compressed versions of the image by taking the initial terms of the singular value sum.

◦ Below are the image reconstructions using various numbers of singular values:

5 The formal statement is the Eckart-Young-Mirsky theorem, as follows: let A ∈ M

m×n (F ) for F = R or C have rank r and singular
values σ1 ≥ σ2 ≥ · · · ≥ σr > 0, with corresponding singular value bases {v1 , . . . , vn } of F n and {w1 , . . . , wm } of F m . Then for any
1 ≤ d ≤ r, the best rank-d approximation to A is the sum Ad = σ1 v1 w1∗ + σ2 v2 w2∗ + · · · + σd vd wd∗ , in the sense that if B : V → W
is any rank-d transformation, then ||Ad − A|| ≤ ||B − A|| with equality only for B = Ad , where the norm is the Frobenius norm on
Mm×n (F ).

20
 ◦ The total amount of data required to store the full image is the equivalent to 5122 data points (one per
pixel). To store the decomposition with k singular values, on the other hand, requires only storing about
2k · 512 data points (each singular value matrix vi wi∗ requires just the values of the vectors vi and wi ).
◦ So, for example, to store and reconstruct the compressed image using 20 singular values only requires
about 40/512 ≈ 8% of the total amount of uncompressed data in the original image.

◦ The reason this sort of procedure works is because most of the information in the image is carried by the
rst few singular values, which are much larger than the later ones. For this image, the rst ten singular
values are 66679, 10490, 5904, 4144, 3501, 2853, 2664, 2420, 2384, and 2188, with most of the remaining
values being smaller:

◦ Therefore, taking just the rst few singular values will capture the vast majority of information contained
in the data set.

5.3.2 The Moore-Penrose Pseudoinverse

• To conclude our discussion, we will study the Moore-Penrose pseudoinverse of a linear transformation T :
V → W, which has pleasant applications to computing orthogonal projections and to solving systems of
linear equations.

◦ The main idea is to construct a linear transformation T† : W → V that captures as much of the spirit
of an inverse transformation as possible, even when T is not necessarily invertible or even one-to-one.

◦ A natural approach is simply to restrict attention to the piece of T that is invertible: clearly, T is not
invertible on ker(T ) (since it is the zero map there), so there is nothing to be gained by considering T
on ker(T ).
◦ However, if we take any subspace V1 such that V = ker(T ) ⊕ V1 (we may nd such a subspace simply
by extending a basis of ker(T ) to a basis of V ), then ker(T |V1 ) = ker(T ) ∩ V1 = {0}, so T is one-to-one
when restricted to the subspace V1 .
◦ It is easy to see that im(T |V1 ) = im(T ), so since T |V1 : V1 → im(T ) is one-to-one, it has an inverse
map S : im(T ) → V . We can then extend this inverse map to be dened on all of W (rather than just
im(T )) by selecting a subspace W1 of W with W = im(T ) ⊕ W1 (we may nd such a subspace simply by
extending a basis of im(T ) to a basis of W ) and declaring that S maps W1 to zero, so that ker(S) = W1 .

◦ The resulting map S : W → V then behaves like a partial inverse of T : V → W , in the sense that for
any w ∈ im(T ) and any v ∈ V1 , we have S(T (v)) = v and T (S(w)) = w, and also for any ṽ ∈ ker(T )
and w̃ ∈ W1 we have T (S(T (v)) = 0 = T (v) and S(T (S(w)) = 0 = S(w).
◦ In particular, although S andT are not full inverses of one another, the calculations above show that
they do satisfy the relations ST S = S and T ST = T .
◦ Of course, we have made two arbitrary choices above; namely, we have chosen the complement subspace
V1 to ker(T ) in V arbitrarily, and we have also chosen the complement subspace W1 to im(T ) in W
arbitrarily.

◦ When V and W are nite-dimensional inner product spaces, there are natural choices for these comple-
ments: specically, we could take V1 to be the orthogonal complement of ker(T ) in V and W1 to be the
orthogonal complement of im(T ) inW.
◦ The resulting construction yields the Moore-Penrose pseudoinverse of T:

21
• Denition: Suppose that V and W are nite-dimensional inner product spaces and T : V → W is linear, and
let S : ker(T )⊥ → im(T ) be the restriction of T to ker(T )⊥ . The (Moore-Penrose) pseudoinverse of T is the
† † −1
linear transformation T : W → V dened so that T (w) = S (w) for all w ∈ im(T ) and T † (w) = 0 for all
⊥
w ∈ im(T ) .

◦ We extend the denition of the pseudoinverse to matrices in the natural way: if A ∈ Mm×n (F ) where
F =R or C, then the pseudoinverse of A is the associated matrix (with respect to the standard bases)
of the pseudoinverse T† of the linear transformation with T (v) = Av.
◦ By the discussion above, the pseudoinverse of T is well dened, since S is one-to-one (hence has a valid
inverse) and the union of bases of im(T ) and im(T )⊥ yields a basis for W , so the two conditions on T †
characterize the value of T† on all of W .

◦ If T is an isomorphism, then the pseudoinverse of T is merely T −1 , since in that case ker(T )⊥ = V so

that S = T, and im(T ) = W , so that T † (w) = S −1 (w) = T −1 (w) for all w ∈ W .
◦ If T is merely one-to-one, then T† is obtained by extending the domain of T −1 : im(T ) → V to all of W,
where the extra vectors in W (namely, im(T )⊥ ) are sent to zero.

• In principle, we can use the denition of the pseudoinverse to compute it directly, although this requires
computing bases for ker(T )⊥ , im(T ), and im(T )⊥ and then evaluating T −1 on im(T ):
• Example: Find the pseudoinverse of the linear transformation T : R3 → R2 given by T (x, y, z) = (x, 0), where
3 2
the inner products on R and R are the dot product.

◦ We have the obvious basis {(0, 1, 0), (0, 0, 1)} for ker(T ), and ker(T )⊥ is easily seen to have basis {(1, 0, 0)}.
◦ Likewise, we have the obvious basis {(1, 0)} for im(T ), and im(T )⊥ is easily seen to have basis {(0, 1)}.
† −1 †
◦ Therefore, since S(1, 0, 0) = (1, 0), we see that T (1, 0) = S (1, 0) = (1, 0, 0), and also T (0, 1) =
(0, 0, 0).
◦ Thus, the map T † : R2 → R3 has T † (x, y) = (x, 0, 0) .

• Example: Find the pseudoinverse of the linear transformation T : R3 → R2 given by T (x, y, z) = (x + y −

3 2
z, x + z), where the inner products on R and R are the dot product.

◦ We can compute the basis {(−1, 2, 1)} for ker(T ), and ker(T )⊥ is easily seen to have basis {(1, 1, −1), (1, 0, 1)}.
◦ Likewise, we have the obvious basis {(1, 0), (0, 1)} for im(T ), while im(T )⊥ has empty basis.

◦ Therefore, with S = T |ker(T )⊥ , we compute S(1, 1, −1) = (3, 0) and S(1, 0, 1) = (0, 2), so S(1/3, 1/3, −1/3) =
(1, 0) and S(1/2, 0, 1/2) = (0, 1).
◦ Thus, we have T † (1, 0) = S −1 (1, 0) = (1/3, 1/3, −1/3), and also T † (0, 1) = (1/2, 0, 1/2).
◦ This means the map T † : R2 → R3 has T † (x, y) = ( 13 x + 12 y, 13 x, − 13 x + 12 y) .

• Using singular value decomposition, we can give another procedure for computing pseudoinverses.

◦ Explicitly, for T : V → W suppose we have computed the singular value bases {v1 , . . . , vn } of V and
{w1 , . . . , wm } of W such that T (vi ) = σi wi for 1 ≤ i ≤ r and T (vi ) = 0 for i > r .

◦ Then {vr+1 , . . . , vn } is a basis for ker(T ) so {v1 , . . . , vr } is a basis for ker(T )⊥ , and {w1 , . . . , wr } is a
⊥
basis for im(T ) so {wr+1 , . . . , wm } is a basis for im(T ) .

◦ We then see S(vi ) = σi wi and so T † (wi ) = S −1 (wi ) = vi /σi for each 1 ≤ i ≤ r, and also T † (wi ) = 0
†
for i > r, which gives the values of T on a basis for W .

• The above calculations in fact yield a singular value decomposition for the pseudoinverse of a matrix:

• Proposition (Pseudoinverses and SVDs): Suppose F = R or C and that A ∈ Mm×n (F ) has rank r and
a singular value decomposition A = U ΣV ∗ where U ∈ Mn×n (F ) and V ∈ Mm×m (F ) are unitary and
Σ ∈ Mn×m (F ) is the matrix whose rst r diagonal entries are σ1 , . . . , σr and whose remaining entries are 0.
† −1 −1
If Σ ∈ Mm×n (F ) is the matrix whose rst r diagonal entries are σ1 , . . . , σr and whose remaining entries
† † † ∗
are 0, then the pseudoinverse A has a singular value decomposition A = V Σ U .

22
 ◦ We will remark that the nonzero diagonal entries of Σ† are in increasing order rather than decreasing
order; one may put the singular values in the usual decreasing order simply by reversing the order of the
corresponding basis elements.

◦ Proof: Let T : F n → F m have T (v) = Av, and suppose we have singular value bases β = {v1 , . . . , vn }
n m
of V = F and γ = {w1 , . . . , wm } of W = F such that T (vi ) = σi wi for 1 ≤ i ≤ r and T (vi ) = 0 for
i > r.
◦ Then as calculated above, we have T † (wi ) = S −1 (wi ) = vi /σi for each 1 ≤ i ≤ r, and also T † (wi ) = 0
for i > r.
◦ Thus, the associated matrix [T † ]βγ is the m×n rectangular diagonal matrix Σ† .
◦ α be the standard basis of F n and δ be the standard basis of F m , then [T ]δα = A, V = [I]α
Now, if we let β,
γ −1
δ
and U = [I]γ so that [I]δ = U = U ∗.
† † α α † β γ † ∗
◦ Then the pseudoinverse A = [T ]δ = [I]β [T ]γ [I]δ = V Σ U , as claimed, and V and U are unitary and

Σ is a rectangular diagonal matrix, this is in fact a singular value decomposition of A† .

†



2 2
 2 2 
• Example: Find the pseudoinverse of the matrix A=
 −1 1 .


1 −1
◦ We previously calculated a singular value decomposition
    A = U ΣV ∗ with
1 0 1 0 4 0  
1  1 0 −1 0   0 2  ∗ 1 1 −1
U=√  , Σ =  , and V =√ .
2 0
 1 0 1   0 0  2 1 1
0 −1 0 1 0 0
 
   1 1 0 0  
† 1 † ∗ 1 1 1/4 0 0 0 1  0 0 = 1 1
1 −1  1 −2 2
◦ Then A = V Σ U = √ √  .
2 −1 1 0 1/2 0 0 2 1
 −1 0 0  8 1 1 2 −2
0 0 1 1
 
1+i 3 1 i
• Example: Find the pseudoinverse of the matrix A= .
1+i 1 2 −i
 
1 3i 2i ∗
◦ We previously calculated a singular value decomposition A = U ΣV with U = √ ,
 √ √ 13 2i −3i √ 
 √  √ 221 (−1 − √
(5 + 5i)/ i)/ 52 (1 + i)/2 (−2 + 2i)/
√ 34
17 0 0 0 ∗
 11i/√ 221 3i/ √52 −i/2 −1/√ 34 
Σ= , and V =  .
0 2 0 0  7i/√ 221 −4i/√ 52 0 4/ √34 
1/ 221 5/ 52 1/2 −3i/ 34
 √   
1/ 17 0   2 − 2i 7 − 7i
†
 0 1/2  1 −3i −2i 1  18 −5 
◦ Then A = V  √ =  .
 0 0  13 −2i 3i 68  −4 20 
0 0 −14i 19i

• Example: Find the pseudoinverse of the linear transformation T : R3 → R2 given by T (x, y, z) = (x + y −

3 2
z, x + z), where the inner products on R and R are the dot product.
 
1 1 −1
◦ We use the singular value decomposition formula on the associated matrix A= .
1 0 1
 
2 1 0
◦ We compute A∗ A =  1 1 −1  with eigenvalues λ = 3, 2, 0 and corresponding unit eigenvectors
  0 −1 2  
1 1 −1 √ √
1 1 1
v1 = √  1 , v2 = √  0 , v3 = √  2 . The singular values are σ1 = 3, σ2 = 2, and
3 −1 2 1 6
  1 
1 0
then w1 = Av1 /σ1 = and w1 = Av2 /σ2 = .
0 1

23
 √ √ √ 
 √

1/√3 1/ 2 −1/√ 6
 
1 0 3 √0 0 
◦ The SVD for A is A = 1/ √3 0√ 2/√6  so the SVD for A† is
0 1 0 2 0
−1/ 3 1/ 2 1/ 6
 √ √ √  √   
1/√3 1/ 3 −1/√ 3 1/ 3 0√   1/3 1/2
1 0
A† =  1/ √2 0√ 1/√2   0 1/ 2  =  1/3 0 .
0 1
−1/ 6 2/ 6 1/ 6 0 0 −1/3 1/2

◦ This means the map T † : R2 → R3 has T † (x, y) = ( 13 x + 12 y, 13 x, − 13 x + 12 y) , just as calculated before.

• The pseudoinverse is dened in terms of orthogonal complements, and as such we can use it to calculate
orthogonal projections and thereby also compute least-squares solutions to inconsistent systems. Explicitly:

• Proposition (Pseudoinverses and Projections): Suppose that V and W are nite-dimensional inner product
spaces, T :V →W is linear, and F =R or C. Then the following hold:

1. The composition T †T is the orthogonal projection of V onto ker(T )⊥ .

◦ V onto ker(T )⊥ is the linear transformation that
Proof: By denition, the orthogonal projection of
⊥ ⊥ ⊥
 
is the identity on ker(T ) and zero on ker(T ) = ker(T ).
◦ But for v ∈ ker(T ) we have T T (v) = T (0) = 0, while for v ∈ ker(T )⊥ we have T † T (v) = v since
† †

T † is the inverse of T when restricted to im(T ). Thus, T † T is the orthogonal projection of V onto
ker(T )⊥ .
2. The composition TT† is the orthogonal projection of W onto im(T ).
◦ W
Proof: By denition, the orthogonal projection of onto im(T ) is the linear transformation that is
im(T ) and zero on im(T )⊥ .
the identity on

◦ But for w ∈ im(T ) we have T T † (w) = w since T † is the inverse of T when restricted to im(T ),
⊥ †
while for w ∈ im(T ) we have T T (w) = T (0) = 0 since T† is zero on im(T )⊥ . Thus, T T † is the
orthogonal projection of W onto im(T ).

3. We have T T †T = T and T †T T † = T †.
◦ Proof: For v ∈ ker(T ) we have T T † T (v) = 0 = T (v) and for v ∈ ker(T )⊥ we have T T † T (v) = T (v)
†
since T T is the identity on v by (1). Thus T T † T and T agree on ker(T ) and ker(T )⊥ , hence are
equal.

◦ w ∈ im(T )⊥ we have T † T T † (w) = 0 = T (w) and

Similarly, forw ∈ im(T ) we have T † T T † (w) =
T (w) since T T † is the identity on w by (2). Thus T † T T †
† †
and T agree on im(T ) and im(T ) ,
⊥

hence are equal.

4. For anyA ∈ Mm×n (F ), the product A† A represents orthogonal projection onto the row space of A, while
†
AA represents orthogonal projection onto the column space of A.
◦ Proof: This follows immediately from (1) and (2) by taking associated matrices.

5. For any A ∈ Mm×n (F ) and c ∈ F m, if the system Ax = c is consistent, then the vector x̂ = A† c is
n
the unique solution of minimum norm, meaning that any other solution y∈F has ||x̂|| ≤ ||y|| with
equality only for y = x̂.
◦ T : F n → F m have T (v) = Av. If Ax = c is consistent then c ∈ im(T ) and then
Proof: Let
†
Ax̂ = AA c = c by (2): thus x̂ is a solution to the system.
◦ For the second part suppose that Ay = c: then A† Ay = A† c = x̂, and so by (1), x̂ is the orthogonal
⊥
projection of y onto ker(T ) .
2 2 2
◦ Then for y⊥ = y−x̂ ∈ [ker(T )⊥ ]⊥ = ker(T ) we have the Pythagorean relation ||y|| = ||x̂|| + y⊥ ,
which immediately yields ||x̂|| ≤ ||y|| with equality only for y = x̂.

6. For any A ∈ Mm×n (F ) and c ∈ F m , the vector x̂ = A† c is the unique least-squares solution of minimum
n
norm to the system Ax = c, meaning that for any y ∈ F it is true that ||Ax̂ − c|| ≤ ||Ay − c|| with
equality only if Ax̂ = Ay, and in that case ||x̂|| ≤ ||y|| with equality only for y = x̂.

◦ Proof: Let T : Fn → Fm have T (v) = Av. By our discussion of least squares, the unique vector
w ∈ im(T ) minimizing the distance ||w − c|| will be the orthogonal projection c into im(T ), which
†
by (1) is w = AA c = Ax̂.

24
 ◦ In other words, for any y ∈ F n we see that ||Ax̂ − c|| ≤ ||Ay − c|| with equality only if Ax̂ = Ay.
0 † 0 † † † †
◦ For the second part, let Ax̂ = Ay = c . Then by (3) we have A c = A Ax̂ = A AA c = A c = x̂,
0
and so by applying (5) to the system Ax = c we have ||x̂|| ≤ ||y|| with equality only for y = x̂.

• Example: Find the matrix associated to orthogonal projection onto the subspace of C4 spanned by the vectors
(1 + i, 3, 1, i) and (1 + i, 1, 2, −i), with respect to the standard basis under the standard inner product.

 
1+i 3 1 i
◦ For A= , we wish to compute the orthogonal projection onto the row space of A,
1+i 1 2 −i
which by (4) of the theorem above is given by the product A† A.
   
2 − 2i 7 − 7i 18 13 − 13i 16 − 16i −5 − 5i
† 1 
 18 −5  † 1  13 + 13i 49 8 23i 
◦ Since A = , the projection is A A =   .
68  −4 20  68  16 + 16i 8 36 −24i 
−14i 19i −5 + 5i −23i 24i 33

• Example: Find the solution of minimal norm to the system x + y − z = 6, x + z = 2.

 
† 1 1 −1
◦ By (5) of the theorem above, the unique solution of minimum norm is x̂ = A c for A=
  1 0 1
6
and c= .
2
     
1/3 1/2 1/3 1/2   3
6
◦ Since A† =  1/3 0 , the solution is x̂ = A† c =  1/3 0  =  2  .
2
−1/3 1/2 −1/3 1/2 −1
◦ Remark: Solving the system with row-reduction yields (x, y, z) = (2 − t, 4 + 2t, t) which as a vector has
squared norm (2 − t)2 + (4 + 2t)2 + t2 = 6t2 + 12t + 18 = 6(t + 1)2 + 12, so the minimum norm indeed
occurs when t = −1, as claimed.

• Example: Find the least-squares solution of minimal norm to the inconsistent system 2x+2y = 5, 2x+2y = 3,
−x + y = 3, x − y = −5.
   
2 2 5
 2 2   3 
◦ By (6) of the theorem above, for A = 
 −1
and c = 
 3  the least-squares
 solution of
1 
1 −1 −5
 
1 1 1 −2 2
minimal norm is x̂ = A† c. Earlier we found
†
A = , so the desired solution is
  8 1 1 2 −2
  5  
1 1 1 −2 2  3  −1
x̂ = A† c =  = .
8 1 1 2 −2  3  3
−5

Well, you're at the end of my handout. Hope it was helpful.

Copyright notice: This material is copyright Evan Dummit, 2019-2022. You may not reproduce or distribute this
material without my express permission.

25