MATH10212 • Linear Algebra • Brief lecture notes 57

Gram–Schmidt Process enables us to find an orthogonal basis of a sub-

space. Let ~u1 , . . . , ~uk be a basis of a subspace V of Rn . We begin the process
of finding a new, orthogonal, basis ~v1 , . . . , ~vk by setting ~v1 = ~u1 .
Then we seek ~v2 = ~u2 + c1 ~u1 with unknown coeff. c1 ; we need ~v2 · ~v1 = 0,
that is, ~u2 · ~u1 + c1 (~u1 · ~u1 ) = 0, whence c1 = − ~~uu21 ·~
u1
u1 (note that ~
·~ u1 · ~u1 6= 0 as
~u1 6= ~0 being a vector in a basis).
Next we seek ~v3 = ~u3 + d1~v1 + d2~v2 with unknown coeff. d1 , d2 ; we need
~v3 · ~v1 = 0, that is, ~u3 · ~v1 + c1 (~v1 · ~v1 ) = 0 (the term with ~v2 · ~v1 = 0 vanishes),
whence d1 = − ~u~v13·~
·~
v1
v1 ; also we need ~ v3 · ~v2 = 0, that is, ~u3 · ~v2 + c1 (~v2 · ~v2 ) = 0
(the term with ~v1 · ~v2 = 0 vanishes), whence d2 = − ~u~v23·~ ·~
v2
v2 .
...And so on, at each step we seek ~vj = ~uj + linear comb. of preceding
~v1 , . . . , ~vj−1 with unknown coeff., which are easily found.

Orthogonal Diagonalization of Symmetric Matrices

Definition. A square matrix A is orthogonally diagonalizable if there ex-
ists an orthogonal matrix Q such that QT AQ = D is a diagonal matrix.

Remarks. Since QT = Q−1 for orthogonal Q, the equality QT AQ = D is

the same as Q−1 AQ = D, so A ∼ D, so this a special case of diagonaliza-
tion: the diagonal entries of D are eigenvalues of A, and the columns of
Q are corresponding eigenvectors. The only difference is the additional re-
quirement that Q be orthogonal, which is equivalent to the fact that those
eigenvectors – columns of Q – form an orthonormal basis of Rn .

Theorem 5.17. If a matrix is orthogonally diagonalizable, then it is sym-

metric.

Proof. We have QT AQ = D; times Q on the left, and QT on the right

gives A = QDQT (since QT = Q−1 ). Then AT = (QDQT )T = (QT )T DT QT =
QDQT = A, so A is symmetric.

Theorem 5.18. All eigenvalues (all roots of the characteristic polynomial)

of a symmetric matrix are real.

Theorem 5.19. Eigenvectors of a symmetric matrix corresponding to dif-

ferent eigenvalues are orthogonal.

Proof. Let AT = A have eigenvectors ~v1 and ~v2 for eigenvalues λ1 6= λ2 .

We compute the dot product (A~v1 ) ·~v2 = (λ1~v1 ) ·~v2 = λ1 (~v1 ·~v2 ). On the other
hand, the left-hand side can be written as a matrix product: (A~v1 ) · ~v2 =
MATH10212 • Linear Algebra • Brief lecture notes 58

(A~v1 )T ~v2 = ~v1T AT ~v2 = ~v1T (A~v2 ) = ~v1 · (λ2~v2 ) = λ2 (~v1 · ~v2 ). Thus, λ1 (~v1 · ~v2 ) =
λ2 (~v1 · ~v2 ). Since λ1 6= λ2 by hypothesis, we must have ~v1 · ~v2 = 0.

Theorem 5.20. Every symmetric matrix is orthogonally diagonalizable.

Method for orthogonal diagonalization of a symmetric matrix. Find

eigenvalues of A. Find the eigenspace for each eigenvalue. For repeated
eigenvalues (when the dimension of the eigenspace is greater than 1) ap-
ply Gram–Schmidt orthogonalization to find an orthogonal basis. Together,
these orthogonal bases of eigenspaces form an orthogonal basis of Rn . Nor-
malize, dividing each vector of the basis by its length. The resulting or-
thonormal basis can be taken as columns of Q such that QT AQ = Q−1 AQ =
D, where D is diagonal with the eigenvalues of A on the diagonal, in the
order corresponding to the order of their eigenvectors as columns of Q.

 
1 2 2
Example. For A = 2 1 2 the characteristic polynomial is
¯ 2 2 1 ¯
¯1 − λ 2 2 ¯¯
¯
det(A − λI) = ¯¯ 2 1−λ 2 ¯¯ =
¯ 2 2 1 − λ¯
(1 − λ)3 + 8 + 8 − 4(1 − λ) − 4(1 − λ) − 4(1 − λ) = · · · = −(λ − 5)(λ + 1)2 . Thus,
eigenvalues are 5 and −1.     
2 2 2 x1 0
Eigenspace E−1 : (A−(−1)I)~x = ~0; 2 2 2 x2  = 0; x1 = −x2 −x3 ,
 2 2  2 x3  0
 −s − t 
where x2 , x3 are free var.; E−1 =  s  | s, t ∈ R ;
 
    t 
 −1 −1 
a basis of E−1 : ~u1 =  1  , ~u2 =  0  .
 
0 1
Apply Gram–Schmidt to orthogonalize: ~v1 = ~u1 ; seek ~v2 = ~u2 + c~ v1 ; 
−1/2
for ~v2 · ~v1 = 0 obtain c = − ~u~v12·~
·~
v1
= − 1
, thus, ~
v 2 = ~
u 2 − (1/2)~v1 =  −1/2.
v1 2
     1
−4 2 2 x1 0
Eigenspace E5 : (A − 5I)~x = ~0;  2 −4 2  x2  = 0; solve this
2 2 −4 x3  0 

 t 
system....: x1 = x2 = x3 , where x3 is a free var.; E5 = t | t ∈ R ;
 
  t
 1 
a basis of E5 : 1 .
 
1
(Note, E5 is automatically orthogonal to E−1 .)
MATH10212 • Linear Algebra • Brief lecture notes 59

Together, we have an orthogonal basis of R3 consisting of eigenvectors:

     
−1 −1/2 1
 1  , −1/2 , 1 .
0 1 1

Normalize:  √   √   √ 
−1/√ 2 −1/√6 1/√3
 1/ 2  , −1/ 6 , 1/ 3 .
p √
0 2/3 1/ 3
 √ √ √ 
−1/√ 2 −1/√6 1/√3
Let Q =  1/ 2 −1/ 
p 6 1/√3 , which is an orthogonal matrix;
0 2/3 1/ 3
−1 0 0
then QT AQ =  0 −1 0.
0 0 5

Orthogonal Complements and Orthogonal Projections

Definition Let W be a subspace of Rn . We say that a vector ~v in Rn

is orthogonal to W if ~v is orthogonal to every vector in W . The set of all
vectors that are orthogonal to W is called the orthogonal complement of W ,
denoted W ⊥ . That is,

W ⊥ = {~v in Rn | ~v · w
~ = 0 for all w
~ in W }

Theorem 5.9 Let W be a subspace of Rn .

a. W ⊥ is a subspace of Rn .
b. (W ⊥ )⊥ = W .

c. W ∩ W ⊥ = {~0}.
d. If W = span(w ~ k ), then ~v is in W ⊥ if and only if ~v · w
~ 1, . . . , w ~ i = 0 for all
i = 1, . . . , k.

Theorem 5.10 Let A be an m × n matrix. Then the orthogonal comple-

ment of the row space of A is the null space of A, and the orthogonal com-
plement of the column space of A is the null space of AT :
(row(A))⊥ =null(A) and (col(A))⊥ =null(AT )
MATH10212 • Linear Algebra • Brief lecture notes 60

Orthogonal Projections

Theorem 5.11 The Orthogonal Decomposition Theorem

Let V be a subspace of Rn and let ~u be a vector in Rn . Then there are
~ ∈ V ⊥ such that ~u = ~v + w.
unique vectors ~v ∈ V and w ~
In fact: if ~v1 , . . . , ~vr is an orthogonal basis of V , then an orthogonal basis
~vr+1 , . . . , ~vn of V ⊥ must have n−r vectors, and together these n vectors form
an orthogonal basis ~v1 , . . . , ~vr , ~vr+1 , . . . , ~vn of Rn

Definition In Theorem 5.11, the vector ~v is called the orthogonal projec-

tion of ~u onto V , and the length of w
~ is called the distance from ~u to V .

Method for finding the orthogonal projection and the distance (given
a subspace V of Rn and some vector ~u ∈ Rn ).
Choose an orthogonal basis ~v1 , . . . , ~vr of V (if V is given as a span of some
non-orthogonal vectors, apply Gram–Schmidt first to obtain an orthogonal
basis of V ); we know that there is an orthogonal basis ~vr+1 , . . . , ~vn of V ⊥
such that
~v1 , . . . , ~vr , ~vr+1 , . . . , ~vn
is an orthogonal basis of Rn (but we do not really need these ~vr+1 , . . . , ~vn !).
Then
r
X n
X
~u = ai~vi + bj ~vj .
i=1 j=r+1

We now find the coefficients ai . For that, take dot product with ~vi0 for each
i0 = 1, . . . , r: on the right only one term does not vanish, since the ~vk are
orthogonal to each other:

~u · ~vi0 = ai0 (~vi0 · ~vi0 ),

whence
~u · ~vi0
a i0 = .
~vi0 · ~vi0

Having found these ai0 for each i0 = 1, . . . , r, we now have the orthogonal
projection of ~u onto V :
Xr
~v = ai~vi .
i=1

The orthogonal component is found by subtracting:

~ = ~u − ~v ∈ V ⊥ ,
w

and the distance from ~u to V is kwk.

~
MATH10212 • Linear Algebra • Brief lecture notes 61

Corollary 5.12 If W is a subspace of Rn , then

(W ⊥ )⊥ = W

Theorem 5.13 If W is a subspace of Rn , then

dim W + dim W ⊥ = n

Corollary 5.14 The Rank Theorem

If A is an m × n matrix, then

rank (A) + nullity (A) = n

Vector Spaces and Subspaces

Definition Let V be a set on which two operations, called addition and
scalar multiplication, have been defined. If ~u and ~v are defined in V , the
sum of ~u and ~v is denoted by ~u + ~v , and if c is a scalar, the scalar multiple of
~u is denoted by c~u. If the following axioms hold for all ~u, ~v and w ~ in V and
for all scalars c and d, then V is called a vector space and its elements are
called vectors.

1. ~u + ~v is in V . Closure under addition

2. ~u + ~v = ~v + ~u Commutativity
3. (~u + ~v ) + w
~ = ~u + (~v + w)
~ Associativity

4. There exists an element ~0 in V , called a zero vector such that ~u +~0 = ~u.

5. For each ~u in V , there is an element −~u in V such that ~u + (−~u) = ~0.

6. c~u is in V . Closure under scalar multiplication

7. c(~u + ~v ) = c~u + c~v Distributivity

8. (c + d)~u = c~u + d~u Distributivity
9. c(d~u) = (cd)~u
10. 1~u = ~u
MATH10212 • Linear Algebra • Brief lecture notes 62

Theorem 6.1 Let V be a vector space, ~u a vector in V , and c a scalar.

a. 0~u = ~0
b. c~0 = ~0
c. (−1)~u = −~u
d. If c~u = ~0, then c = 0 or ~u = ~0.

Subspaces
Definition A subset W of a vector space V is called a subspace of V if W
is itself a vector space with the same scalars, addition and scalar multipli-
cation as V .

Theorem 6.2 Let V be a vector space and let W be a nonempty subset of

V . Then W is a subspace of V if and only if the following conditions hold:

a. If ~u and ~v are in W , then ~u + ~v is in W .

b. If ~u is in W and c is a scalar, then c~u is in W .

Example 6.14 If V is a vector space, then V is clearly a subspace of it-

self. The set {~0}, consisting of only the zero vector, is also a subspace of V ,
called the zero subspace. To show this, we simply note that the two closure
conditions of Theorem 6.2 are satisfied:
~0 + ~0 = ~0 and c~0 = ~0 for any scalar c

The subspaces {~0} and V are called the trivial subspaces of V .

Spanning Sets

Definition If
S = {~v1 , ~v2 , . . . , ~vk }
is a set of vectors in a vector space V , then the set of all linear combinations
of
~v1 , ~v2 , . . . , ~vk
is called the span of
~v1 , ~v2 , . . . , ~vk
and is denoted by
span(~v1 , ~v2 , . . . , ~vk )
or span(S). If V =span(S), then S is called a spanning set for V and V is
said to be spanned by S.
MATH10212 • Linear Algebra • Brief lecture notes 63

Theorem 6.3 Let ~v1 , ~v2 , . . . , ~vk be vectors in a vector space V .

a. span(~v1 , ~v2 , . . . , ~vk ) is a subspace of V .

b. span(~v1 , ~v2 , . . . , ~vk ) is a smallest subspace of V that contains ~v1 , ~v2 , . . . , ~vk .

Linear Independence, Basis and Dimension

Linear Independence

Definition A set of vectors S = {~v1 , ~v2 , . . . , ~vk } in a vector space V is lin-

early dependent if there are scalars c1 , c2 , . . . , ck at least one of which is not
zero, such that
c1~v1 + c2~v2 + · · · + ck~vk = ~0
A set of vectors that is not linearly dependent is said to be linearly indepen-
dent.

Theorem 6.4 A set of vectors S = {~v1 , ~v2 , . . . , ~vk } in a vector space V is

linearly dependent if and only if at least one of the vectors can be expressed
as a linear combination of the others.

Bases

Definition A subset B of a vector space V is a basis for V if

1. B spans V and
2. B is linearly independent.