MTH5112-5212 Linear Algebra, 2019–2020

Lectures given by Dr Ivan Tomašić,

based on notes by Dr Oscar Bandtlow

December 4, 2019
ii
Contents

I Revision from Vectors & Matrices v

1 Systems of Linear Equations 1
1.1 Basic terminology and examples . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Gaussian elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Special classes of linear systems . . . . . . . . . . . . . . . . . . . . . . . . 7

2 Matrix Algebra 9
2.1 Revision from Geometry I . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Transpose of a matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 Special types of square matrices . . . . . . . . . . . . . . . . . . . . . . . . 13
2.4 Linear systems in matrix notation . . . . . . . . . . . . . . . . . . . . . . . 14
2.5 Elementary matrices and the Invertible Matrix Theorem . . . . . . . . . . 16
2.6 Gauss-Jordan inversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3 Determinants 23
3.1 General definition of determinants . . . . . . . . . . . . . . . . . . . . . . . 23
3.2 Properties of determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.3 Cramer’s Rule and a formula for A−1 . . . . . . . . . . . . . . . . . . . . . 31

II Linear Algebra 35
4 Vector Spaces 37
4.1 Definition and examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2 Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.3 The span of a set of vectors . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.4 Linear independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.5 Linear independence test . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.6 Basis and dimension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.7 Coordinates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.8 Row space and column space . . . . . . . . . . . . . . . . . . . . . . . . . . 58

5 Linear Transformations 61
5.1 Definition and examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5.2 Basic properties of linear transformations . . . . . . . . . . . . . . . . . . . 63
5.3 Linear transformations between standard vector spaces . . . . . . . . . . . 63
5.4 Linear transformations on general vector spaces . . . . . . . . . . . . . . . 64
5.5 Defining linear maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.6 Forming new linear maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5.7 Isomorphisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

iii
iv CONTENTS

5.8 Image and Kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

5.9 Linear maps, coordinatisation and matrices . . . . . . . . . . . . . . . . . . 72
5.10 Change of basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.11 Hom spaces and End algebras . . . . . . . . . . . . . . . . . . . . . . . . . 77

6 Eigenvalues and Eigenvectors 79

6.1 Definition and examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.2 Diagonalisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6.3 Interlude: complex vector spaces and matrices . . . . . . . . . . . . . . . . 87

7 Orthogonality 91
7.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
7.2 Orthogonal complements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
7.3 Orthogonal sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
7.4 Orthonormal sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7.5 Orthogonal projections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
7.6 Gram Schmidt process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.7 Least squares problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
7.8 Spectral Theorem for Symmetric Matrices . . . . . . . . . . . . . . . . . . 102
 Part I

Revision from Vectors & Matrices

v
Chapter 1

Systems of Linear Equations

Systems of linear equations arise frequently in many areas of the sciences, including
physics, engineering, business, economics, and sociology. Their systematic study also
provided part of the motivation for the development of modern linear algebra at the end
of the 19th century. Linear equations are extremely important and in particular in higher
dimensions, one aims to have a systematic and efficient way to solve them.
The material in this chapter will be familiar from Geometry I, where systems of linear
equations have already been discussed in some detail. As this chapter is fundamental for
what is to follow, it is recommended to carefully recall the basic terminology and methods
for linear equations. This module will lead to a more general formalism motivated by linear
equations.

1.1 Basic terminology and examples

A linear equation in n unknowns is an equation of the form

a1 x1 + a2 x2 + · · · + an xn = b ,

where a1 , . . . , an and b are given real numbers and x1 , . . . , xn are variables.

A system of m linear equations in n unknowns is a collection of equations of the
form
a11 x1 + a12 x2 + · · · + a1n xn = b1
a21 x1 + a22 x2 + · · · + a2n xn = b2
..
.
am1 x1 + am2 x2 + · · · + amn xn = bm
where the aij ’s and bi ’s are all real numbers. We also call such systems m × n systems.

Example 1.1.

x1 − x2 = 0
2x1 + x2 = 4 x + x2 − x3 = 3
(a) (b) 1 (c) x1 + x2 = 3 .
3x1 + 2x2 = 7 2x1 − x2 + x3 = 6
x2 = 1

(a) is a 2 × 2 system, (b) is a 2 × 3 system, and (c) is a 3 × 2 system.

A solution of an m × n system is an ordered n-tuple (x1 , x2 , . . . , xn ) that satisfies all

equations of the system.

1
2 CHAPTER 1. SYSTEMS OF LINEAR EQUATIONS

Example 1.2. (1, 2) is a solution of Example 1.1 (a).

For each α ∈ R, the 3-tuple (3, α, α) is a solution of Example 1.1 (b) (CHECK!).
Example 1.1 (c) has no solution, since, on the one hand x2 = 1 by the last equation,
but the first equation implies x1 = 1, while the second equation implies x1 = 2, which is
impossible.
A system with no solution is called inconsistent, while a system with at least one
solution is called consistent.
The set of all solutions of a system is called its solution set, which may be empty if
the system is inconsistent.
The basic problem we want to address in this section is the following: given an ar-
bitrary m × n system, determine its solution set. Later on, we will discuss a procedure
that provides a complete and practical solution to this problem (the so-called ‘Gaussian
algorithm’). Before we encounter this procedure, we require a bit more terminology.
Definition 1.3. Two m × n systems are said to be equivalent, if they have the same
solution set.
Example 1.4. Consider the two systems
5x1 − x2 + 2x3 = −3 5x1 − x2 + 2x3 = −3
(a) x2 = 2 (b) −5x1 + 2x2 − 2x3 = 5 .
3x3 = 6 5x1 − x2 + 5x3 = 3
(a) is easy to solve: looking at the last equation we find first that x3 = 2; the second from
the bottom implies x2 = 2; and finally the first one yields x1 = (−3 + x2 − 2x3 )/5 = −1.
So the solution set of this system is {(−1, 2, 2)}.
To find the solution of (b), add the first and the second equation. Then x2 = 2,
while subtracting the first from the third equation gives 3x3 = 6, that is x3 = 2. Finally,
the first equation now gives x1 = (−3 + x2 − 2x3 )/5 = −1, so the solution set is again
{(−1, 2, 2)}.
Thus the systems (a) and (b) are equivalent.
In solving system (b) above we have implicitly used the following important observa-
tion:
Lemma 1.5. The following operations do not change the solution set of a linear system:
(i) interchanging two equations;
(ii) multiplying an equation by a non-zero scalar;
(iii) adding a multiple of one equation to another.
Proof. (i) and (ii) are obvious. (iii) is a simple consequence of the linearity of the equa-
tions.
We shall see shortly how to use the above operations systematically to obtain the
solution set of any given linear system. Before doing so, however, we introduce a useful
short-hand.
Given an m × n linear system
a11 x1 + · · · + a1n xn = b1
..
.
am1 x1 + · · · + amn xn = bm
1.2. GAUSSIAN ELIMINATION 3

we call the array  

a11 ··· a1n b1
 .. .. .. 
 . . . 
am1 · · · amn bm
the augmented matrix of the linear system, and the m × n matrix
 
a11 · · · a1n
 .. .. 
 . . 
am1 · · · amn

the coefficient matrix of the linear system.

Example 1.6.
 
3x1 + 2x2 − x3 = 5 3 2 −1 5
system: augmented matrix: .
2x1 + x3 = −1 2 0 1 −1

A system can be solved by performing operations on the augmented matrix. Corre-

sponding to the three operations given in Lemma 1.5 we have the following three opera-
tions that can be applied to the augmented matrix, called elementary row operations.

Definition 1.7 (Elementary row operations).

Type I interchanging two rows;
Type II multiplying a row by a non-zero scalar;
Type III adding a multiple of one row to another row.

1.2 Gaussian elimination

Gaussian elimination is a systematic procedure to determine the solution set of a given
linear system. The basic idea is to perform elementary row operations on the correspond-
ing augmented matrix bringing it to a simpler form from which the solution set is readily
obtained.
The simple form alluded to above is given in the following definition.

Definition 1.8. A matrix is said to be in row echelon form if it satisfies the following
three conditions:

(i) All zero rows (consisting entirely of zeros) are at the bottom.

(ii) The first non-zero entry from the left in each nonzero row is a 1, called the leading
1 for that row.

(iii) Each leading 1 is to the right of all leading 1’s in the rows above it.

A row echelon matrix is said to be in reduced row echelon form if, in addition it
satisfies the following condition:

(iv) Each leading 1 is the only nonzero entry in its column

Roughly speaking, a matrix is in row echelon form if the leading 1’s form an echelon
(that is, a ‘steplike’) pattern.
4 CHAPTER 1. SYSTEMS OF LINEAR EQUATIONS

Example 1.9. Matrices in row echelon form:

   
1 4 2 1 3 1 0  
0 1 3 , 0 0 1 3 , 0 1 2
.
0 0 1
0 0 1 0 0 0 0

Matrices in reduced row echelon form:

     
1 2 0 1 0 1 5 0 2 1 0 3
0 0 1 2 0 , 0 0 1 −1 , 0 1 2 .
0 0 0 0 1 0 0 0 0 0 0 0

The variables corresponding to the leading 1’s of the augmented matrix in row ech-
elon form will be referred to as the leading variables, the remaining ones as the free
variables.
Example 1.10.
 
1 2 3 −4 6
(a) .
0 0 1 2 3
Leading variables: x1 and x3 ; free variables: x2 and x4 .
 
1 0 5
(b) .
0 1 3
Leading variables: x1 and x2 ; no free variables.
Note that if the augmented matrix of a system is in row echelon form, the solution set
is easily obtained.
Example 1.11. Determine the solution set of the systems given by the following aug-
mented matrices in row echelon form:
 
  1 −2 0 1 2
1 3 0 2
(a) , (b)  0 0 1 −2 1  .
0 0 0 1
0 0 0 0 0

Solution. (a) The corresponding system is

x1 + 3x2 = 2
0 = 1

so the system is inconsistent and the solution set is empty.

(b) The corresponding system is

x1 − 2x2 + x4 = 2
x3 − 2x4 = 1
0 = 0

We can express the leading variables in terms of the free variables x2 and x4 . So set
x2 = α and x4 = β, where α and β are arbitrary real numbers. The second line now tells
us that x3 = 1 + 2x4 = 1 + 2β, and then the first line that x1 = 2 + 2x2 − x4 = 2 + 2α − β.
Thus the solution set is { (2 + 2α − β, α, 1 + 2β, β) | α, β ∈ R }.
It turns out that every matrix can be brought into row echelon form using only ele-
mentary row operations. The procedure is known as the
1.2. GAUSSIAN ELIMINATION 5

Gaussian algorithm:

Step 1 If the matrix consists entirely of zeros, stop — it is already in row echelon form.

Step 2 Otherwise, find the first column from the left containing a non-zero entry (call it
a), and move the row containing that entry to the top position.

Step 3 Now multiply that row by 1/a to create a leading 1.

Step 4 By subtracting multiples of that row from rows below it, make each entry below
the leading 1 zero.

This completes the first row. All further operations are carried out on the other rows.

Step 5 Repeat steps 1-4 on the matrix consisting of the remaining rows

The process stops when either no rows remain at Step 5 or the remaining rows consist of
zeros.

Example 1.12. Solve the following system using the Gaussian algorithm:

x2 + 6x3 =4
3x1 − 3x2 + 9x3 = −3
2x1 + 2x2 + 18x3 = 8

Solution. Performing the Gaussian algorithm on the augmented matrix gives:

    1  
0 1 6 4 3 −3 9 −3 R1 1 −1 3 −1
 3 −3 9 −3  ∼ R1 ↔ R2  0 1 6 4  ∼
3
0 1 6 4 
2 2 18 8 2 2 18 8 2 2 18 8

     
1 −1 3 −1 1 −1 3 −1 1 −1 3 −1
∼ 0 1 6 4  ∼ 0 1 6 4 ∼ 0 1 6 4  ,
R3 − 2R1 0 4 12 10 R3 − 4R2 0 0 −12 −6 − 12 R3 0 0 1 12
1

where the last matrix is now in row echelon form. The corresponding system reads:

x1 − x2 + 3x3 = −1
x2 + 6x3 = 4
x3 = 12

Leading variables are x1 , x2 and x3 ; there are no free variables. The last equation now
implies x3 = 12 ; the second equation from bottom yields x2 = 4 − 6x3= 1 and finally the
first equation yields x1 = −1 + x2 − 3x3 = − 23 . Thus the solution is (− 32 , 1, 21 ) .

A variant of the Gauss algorithm is the Gauss-Jordan algorithm, which brings a matrix
to reduced row echelon form:
6 CHAPTER 1. SYSTEMS OF LINEAR EQUATIONS

Gauss-Jordan algorithm

Step 1 Bring matrix to row echelon form using the Gaussian algorithm.

Step 2 Find the row containing the first leading 1 from the right, and add suitable mul-
tiples of this row to the rows above it to make each entry above the leading 1
zero.

This completes the first non-zero row from the bottom. All further operations are carried
out on the rows above it.

Step 3 Repeat steps 1-2 on the matrix consisting of the remaining rows.

Example 1.13. Solve the following system using the Gauss-Jordan algorithm:

x1 + x2 + x3 + x4 + x5 = 4
x1 + x2 + x3 + 2x4 + 2x5 = 5
x1 + x2 + x3 + 2x4 + 3x5 = 7

Solution. Performing the Gauss-Jordan algorithm on the augmented matrix gives:

     
1 1 1 1 1 4 1 1 1 1 1 4 1 1 1 1 1 4
 1 1 1 2 2 5  ∼ R2 − R1  0 0 0 1 1 1  ∼ 0 0 0 1 1 1
1 1 1 2 3 7 R3 − R1 0 0 0 1 2 3 R3 − R2 0 0 0 0 1 2
   
R1 − R3 1 1 1 1 0 2 R1 − R2 1 1 1 0 0 3
∼ R2 − R3  0 0 0 1 0 −1  ∼  0 0 0 1 0 −1  ,
0 0 0 0 1 2 0 0 0 0 1 2
where the last matrix is now in reduced row echelon form. The corresponding system
reads:
x1 + x2 + x3 = 3
x4 = −1
x5 = 2
Leading variables are x1 , x4 , and x5 ; free variables x2 and x3 . Now set x2 = α and
x3 = β, and solve for the leading variables starting from the last equation. This yields
x5 = 2, x4 = −1, and finally x1 = 3 − x2 − x3 = 3 − α − β. Thus the solution set is
{ (3 − α − β, α, β, −1, 2) | α, β ∈ R }.
We have just seen that any matrix can be brought to (reduced) row echelon form
using only elementary row operations, and moreover that there is an explicit procedure
to achieve this (namely the Gaussian and Gauss-Jordan algorithm). We record this im-
portant insight for later use.
Theorem 1.14.

(a) Every matrix can be brought to row echelon form by a series of elementary row
operations.

(b) Every matrix can be brought to reduced row echelon form by a series of elementary
row operations.
Proof. For (a):apply the Gaussian algorithm; for (b): apply the Gauss-Jordan algorithm.
1.3. SPECIAL CLASSES OF LINEAR SYSTEMS 7

Remark 1.15. It can be shown (but not in this module) that the reduced row echelon
form of a matrix is unique. On the contrary, this is not the case for just the row echelon
form.

The remark above implies that if a matrix is brought to reduced row echelon form by
any sequence of elementary row operations (that is, not necessarily by those prescribed
by the Gauss-Jordan algorithm) the leading ones will nevertheless always appear in the
same positions. As a consequence, the following definition makes sense.

Definition 1.16. A pivot position in a matrix A is a location that corresponds to a

leading 1 in the reduced row echelon form of A. A pivot column is a column of A that
contains a pivot position.

Example 1.17. Let  

1 1 1 1 1 4
A = 1 1 1 2 2 5 .
1 1 1 2 3 7
By Example 1.13 the reduced row echelon form of A is
 
1 1 1 0 0 3
0 0 0 1 0 −1 ,
0 0 0 0 1 2

Thus the pivot positions of A are the (1, 1)-entry, the (2, 4)-entry, and the (3, 5)-entry
and the pivot columns of A are columns 1, 4, and 5.

The notion of a pivot position and a pivot column will come in handy later in the
module.

1.3 Special classes of linear systems

In this last section of our first chapter we’ll have a look at a number of special types of
linear systems and derive the first important consequences of the fact that every matrix
can be brought to row echelon form by a series of elementary row operations.
We start with the following classification of linear systems:

Definition 1.18. An m × n linear system is said to be

• overdetermined if it has more equations than unknowns (i.e. m > n);

• underdetermined if it has fewer equations than unknowns (i.e. m < n).

Note that overdetermined systems are usually (but not necessarily) inconsistent. Un-
derdetermined systems may or may not be consistent. However, if they are consistent,
then they necessarily have infinitely many solutions:

Theorem 1.19. If an underdetermined system is consistent, it must have infinitely many

solutions.

Proof. Note that the row echelon form of the augmented matrix of the system has r ≤ m
non-zero rows. Thus there are r leading variables, and consequently n − r ≥ n − m > 0
free variables.
8 CHAPTER 1. SYSTEMS OF LINEAR EQUATIONS

Another useful classification of linear systems is the following:

Definition 1.20. A linear system
a11 x1 + a12 x2 + · · · + a1n xn = b1
a21 x1 + a22 x2 + · · · + a2n xn = b2
.. (1.1)
.
am1 x1 + am2 x2 + · · · + amn xn = bm
is said to be homogeneous if bi = 0 for all i. Otherwise it is said to be inhomogeneous.
Given an inhomogeneous system (1.1), call the system obtained by setting all bi ’s to
zero, the associated homogeneous system .
Example 1.21.
3x1 + 2x2 + 5x3 = 2 3x1 + 2x2 + 5x3 = 0
2x1 − x2 + x3 = 5 2x1 − x2 + x3 = 0
| {z } | {z }
inhomogeneous system associated homogeneous system

The first observation about homogeneous systems is that they always have a solution,
the so-called trivial or zero solution: (0, 0, . . . , 0).
For later use we record the following useful consequence of the previous theorem on
consistent homogeneous systems:
Theorem 1.22. An underdetermined homogeneous system always has non-trivial solu-
tions.
Proof. We just observed that a homogeneous systems is consistent. Thus, if the system
is underdetermined and homogeneous, it must have infinitely many solutions by Theo-
rem 1.19, hence, in particular, it must have a non-zero solution.
Our final result in this section is devoted to the special case of n × n systems. For such
systems there is a delightful characterisation of the existence and uniqueness of solutions
of a given system in terms of the associated homogeneous systems. At the same time, the
proof of this result serves as another illustration of the usefulness of the row echelon form
for theoretical purposes.
Theorem 1.23. An n × n system is consistent and has a unique solution, if and only if
the only solution of the associated homogeneous system is the zero solution.
Proof. Follows from the following two observations:
• The same sequence of elementary row operations that brings the augmented matrix
of a system to row echelon form, also brings the augmented matrix of the associated
homogeneous system to row echelon form, and vice versa.
• An n × n system in row echelon form has a unique solution precisely if there are n
leading variables.
Thus, if an n × n system is consistent and has a unique solution, the corresponding
homogeneous system must have a unique solution, which is necessarily the zero solution.
Conversely, if the associated homogeneous system of a given system has the zero
solution as its unique solution, then the original inhomogeneous system must have a
solution, and this solution must be unique.
Chapter 2

Matrix Algebra

In this chapter we first repeat basic rules and definitions that are necessary for doing
calculations with matrices in an efficient way. Most of this will already be familiar from
Geometry I. We will then consider the inverse of a matrix, the transpose of a matrix,
and what is meant by the concept of a symmetric matrix. A first highlight in the later
sections is the Invertible Matrix Theorem.

2.1 Revision from Geometry I

Recall that an m × n matrix A is a rectangular array of scalars (real numbers)
 
a11 · · · a1n
 .. ..  .
 . . 
am1 · · · amn

We write A = (aij )m×n or simply A = (aij ) to denote an m × n matrix whose (i, j)-entry
is aij , i.e. aij is the i-th row and in the j-th column.
If A = (aij )m×n we say that A has size m × n. An n × n matrix is said to be square.

Example 2.1. If  
1 3 2
A= ,
−2 4 0
then A is a matrix of size 2 × 3. The (1, 2)-entry of A is 3 and the (2, 3)-entry of A is 0.

Definition 2.2 (Equality). Two matrices A and B are equal and we write A = B if they
have the same size and aij = bij where A = (aij ) and B = (bij ).

Definition 2.3 (Scalar multiplication). If A = (aij )m×n and α is a scalar, then αA (the
scalar product of α and A) is the m × n matrix whose (i, j)-entry is αaij .

Definition 2.4 (Addition). If A = (aij )m×n and B = (bij )m×n then the sum A + B of A
and B is the m × n matrix whose (i, j)-entry is aij + bij .

Example 2.5. Let    

2 3 0 1
A = −1 2 and B =  2 3 .
4 0 −2 1

9
10 CHAPTER 2. MATRIX ALGEBRA

Then      
6 9 0 2 6 11
3A + 2B = −3 6 +  4 6 = 1 12 .
12 0 −4 2 8 2
Definition 2.6 (Zero matrix). We write Om×n or simply O (if the size is clear from the
context) for the m × n matrix all of whose entries are zero, and call it a zero matrix.
Scalar multiplication and addition of matrices satisfy the following rules proved in
Geometry I:
Theorem 2.7. Let A, B and C be matrices of the same size, and let α and β be scalars.
Then:
(a) A + B = B + A;
(b) A + (B + C) = (A + B) + C;
(c) A + O = A;
(d) A + (−A) = O, where −A = (−1)A;
(e) α(A + B) = αA + αB;
(f ) (α + β)A = αA + βA;
(g) (αβ)A = α(βA);
(h) 1A = A.
Example 2.8. Simplify 2(A + 3B) − 3(C + 2B), where A, B, and C are matrices with
the same size.
Solution.
2(A + 3B) − 3(C + 2B) = 2A + 2 · 3B − 3C − 3 · 2B = 2A + 6B − 3C − 6B = 2A − 3C .

Definition 2.9 (Matrix multplication). If A = (aij ) is an m × n matrix and B = (bij ) is

an n × p matrix then the product AB of A and B is the m × p matrix C = (cij ) with
n
X
cij = aik bkj .
k=1

Example 2.10. Compute the (1, 3)-entry and the (2, 4)-entry of AB, where
 
  2 1 6 0
3 −1 2
A= and B =  0 2 3 4 .
0 1 4
−1 0 5 8
Solution.
(1, 3)-entry: 3 · 6 + (−1) · 3 + 2 · 5 = 25;
(2, 4)-entry: 0 · 0 + 1 · 4 + 4 · 8 = 36.
Definition 2.11 (Identity matrix). An identity matrix I is a square matrix with 1’s
on the diagonal and zeros elsewhere. If we want to emphasise its size we write In for the
n × n identity matrix.
2.1. REVISION FROM GEOMETRY I 11

Matrix multiplication satisfies the following rules proved in Geometry I:

Theorem 2.12. Assume that α is a scalar and that A, B, and C are matrices so that
the indicated operations can be performed. Then:

(a) IA = A and BI = B;

(b) A(BC) = (AB)C;

(c) A(B + C) = AB + AC;

(d) (B + C)A = BA + CA;

(e) α(AB) = (αA)B = A(αB).

Notation 2.13.

• Since A(BC) = (AB)C, we can omit the brackets and simply write ABC and
similarly for products of more than three factors.

• If A is a square matrix we write Ak = AA · · · A} for the k-th power of A.

| {z
k factors

Warning: In general AB 6= BA, even if AB and BA have the same size!

Example 2.14.     
1 0 0 1 0 1
=
0 0 0 0 0 0
but     
0 1 1 0 0 0
=
0 0 0 0 0 0
Definition 2.15. If A and B are two matrices with AB = BA, then A and B are said
to commute.

Finally we recall the notion of an inverse of a matrix.

Definition 2.16. If A is a square matrix, a matrix B is called an inverse of A if

AB = I and BA = I .

A matrix that has an inverse is called invertible.

Note that not every matrix is invertible. For example the matrix
 
1 0
A=
0 0

cannot have an inverse since for any 2 × 2 matrix B = (bij ) we have

    
1 0 b11 b12 b11 b12
AB = = 6= I2 .
0 0 b21 b22 0 0

Later on in this chapter we shall discuss an algorithm that lets us decide whether a
matrix is invertible and at the same furnishes an inverse if the matrix is invertible.
12 CHAPTER 2. MATRIX ALGEBRA

It turns out that if a matrix is invertible its inverse is uniquely determined:

Theorem 2.17. If B and C are both inverses of A, then B = C.

Proof. Since B and C are inverses of A we have AB = I and CA = I. Thus

B = IB = (CA)B = C(AB) = CI = C .

If A is an invertible matrix, the unique inverse of A is denoted by A−1 . Hence A−1 (if
it exists!) is a square matrix of the same size as A with the property that

AA−1 = A−1 A = I .

Note that the above equality implies that if A is invertible, then its inverse A−1 is also
invertible with inverse A, that is,

(A−1 )−1 = A .

Slightly deeper is the following result:

Theorem 2.18. If A and B are invertible matrices of the same size, then AB is invertible
and
(AB)−1 = B −1 A−1 .

Proof. Observe that

(AB)(B −1 A−1 ) = A(BB −1 )A−1 = AIA−1 = AA−1 = I ,

(B −1 A−1 )(AB) = B −1 (A−1 A)B = B −1 IB = B −1 B = I .

Thus, by definition of invertibility, AB is invertible with inverse B −1 A−1 .

2.2 Transpose of a matrix

The first new concept we encounter is the following:

Definition 2.19. The transpose of an m × n matrix A = (aij ) is the n × m matrix

B = (bij ) given by
bij = aji
The transpose of A is denoted by AT .

Example 2.20.  
  1 4
1 2 3
(a) A = ⇒ AT = 2 5
4 5 6
3 6
   
1 2 T 1 3
(b) B = ⇒B =
3 −1 2 −1
2.3. SPECIAL TYPES OF SQUARE MATRICES 13

Matrix transposition satisfies the following rules:

Theorem 2.21. Assume that α is a scalar and that A, B, and C are matrices so that
the indicated operations can be performed. Then:

(a) (AT )T = A;

(b) (αA)T = α(AT );

(c) (A + B)T = AT + B T ;

(d) (AB)T = B T AT .

Proof. (a) is obvious while (b) and (c) are proved as Exercise 6 in Coursework 2. For
the proof of (d) assume A = (aij )m×n and B = (bij )n×p and write AT = (ãij )n×m and
B T = (b̃ij )p×n where
ãij = aji and b̃ij = bji .
Notice that (AB)T and B T AT have the same size, so it suffices to show that they have
the same entries. Now, the (i, j)-entry of B T AT is
n
X n
X n
X
b̃ik ãkj = bki ajk = ajk bki ,
k=1 k=1 k=1

which is the (j, i)-entry of AB, that is, the (i, j)-entry of (AB)T . Thus B T AT = (AB)T .

Transposition ties in nicely with invertibility:

Theorem 2.22. Let A be invertible. Then AT is invertible and

(AT )−1 = (A−1 )T .

Proof. See Exercise 8 in Coursework 2.

2.3 Special types of square matrices

In this section we briefly introduce a number of special classes of matrices which will be
studied in more detail later in this course.

Definition 2.23. A matrix is said to be symmetric if AT = A.

Note that a symmetric matrix is necessarily square.

Example 2.24.  
1 2 4  
5 2
symmetric: 2 −1 3 , .
2 −1
4 3 0
 
2 2 4  
1 1 1
not symmetric: 2 2 3 .
1 1 1
1 3 5
14 CHAPTER 2. MATRIX ALGEBRA

Symmetric matrices play an important role in many parts of pure and applied Math-
ematics as well as in some other areas of science, for example in quantum physics. Some
of the reasons for this will become clearer towards the end of this course, when we shall
study symmetric matrices in much more detail.
Some other useful classes of square matrices are the triangular ones, which will also
play a role later on in the course.

Definition 2.25. A square matrix A = (aij ) is said to be

upper triangular if aij = 0 for i > j;
strictly upper triangular if aij = 0 for i ≥ j;
lower triangular if aij = 0 for i < j;
strictly lower triangular if aij = 0 for i ≤ j;
diagonal if aij = 0 for i 6= j.

If A = (aij ) is a square matrix of size n × n, we call a11 , a22 , . . . , ann the diagonal
entries of A. So, informally speaking, a matrix is upper triangular if all the entries
below the diagonal entries are zero, and it is strictly upper triangular if all entries below
the diagonal entries and the diagonal entries itself are zero. Similarly for (strictly) lower
triangular matrices.

Example 2.26.  
  1 0 0 0
1 2 0 3 0 0
upper triangular: , diagonal:  
0 3 0 0 5 0
0 0 0 3
 
0 0 0
strictly lower triangular: −1 0 0 .
2 3 0

We close this section with the following two observations:

Theorem 2.27. The sum and product of two upper triangular matrices of the same size
is upper triangular.

Proof. See Exercise 5, Coursework 2.

2.4 Linear systems in matrix notation

We shall now have another look at systems of linear equations. The added spice in this
discussion will be that we now use the language of matrices to study them. More precisely,
we shall now introduce two equivalent ways of writing systems of linear equations. Both
reformulations will in their own way shed some light on both linear systems and matrices.
Before discussing these reformulations let us recall from Geometry I that an n × 1
matrix  
a1
 a2 
 
 .. 
.
an
2.4. LINEAR SYSTEMS IN MATRIX NOTATION 15

is called a column vector of dimension n, or simply an n-vector. The collection of

all n-vectors is denoted by Rn . Thus:
  

 a 1 

  a2 
 

n
R =  ..  a1 , a2 , · · · , an ∈ R .
 

 . 


 a 

n

Suppose now that we are given an m × n linear system

a11 x1 + a12 x2 + · · · + a1n xn = b1

a21 x1 + a22 x2 + · · · + a2n xn = b2
.. . (2.1)
.
am1 x1 + am2 x2 + · · · + amn xn = bm

The first reformulation is based on the observation that we can write this system more
succinctly as a single matrix equation

Ax = b , (2.2)

where
     
a11 ··· a1n x1 b1
 .. ..  ,  ..   .. 
A= . .  x =  .  ∈ R , and b =  .  ∈ Rm ,
n

am1 · · · amn xn bm

and where Ax is interpreted as the matrix product of A and x.

Example 2.28. Using matrix notation the system

2x1 − 3x2 + x3 = 2
3x1 − x3 = −1

can be written  
  x1  
2 −3 1   2
x2 = .
3 0 −1 −1
| {z } x 3 | {z }
=A | {z } =b
=x

Apart from obvious notational economy, writing (2.1) in the form (2.2) has a number
of other advantages which will become clearer shortly.
The other useful way of writing (2.1) is the following: with A and x as before we have
     
a11 x1 + · · · + a1n xn a11 a1n
..  .   . 
Ax =   = x1  ..  + · · · + xn  ..  ,
 
.
am1 x1 + · · · + amn xn am1 amn
| {z } | {z }
=a1 =an

where aj ∈ Rm is the j-th column of A.

Thus the linear system (2.1) can also be represented as a matrix (or vector) equation
of the form
x1 a1 + · · · + xn an = b . (2.3)
16 CHAPTER 2. MATRIX ALGEBRA

Example 2.29. The linear system

2x1 − 3x2 − 2x3 = 5

5x1 − 4x2 + 2x3 = 6

can also be written as

       
2 −3 −2 5
x1 + x2 + x3 = . (2.4)
5 −4 2 6

Sums such as the left-hand side of (2.3) or (2.4) will turn up time and again in this
course, so it will be convenient to introduce the following terminology

Definition 2.30. If a1 , . . . , an are vectors in in Rm and α1 , . . . , αn are scalars, a sum of

the form
α1 a1 + · · · + αn an
is called a linear combination of the vectors a1 , . . . , an with weights α1 , . . . , αn .

Summarising the previous discussion, we now have the following characterisation of

consistency:

Theorem 2.31 (Consistency Theorem for Linear Systems). A linear system Ax = b is

consistent if and only if b can be written as a linear combination of the column vectors
of A.

2.5 Elementary matrices and the Invertible Matrix

Theorem
Using the reformulation of linear systems discussed in the previous section we shall now
have another look at the process of solving them. Instead of performing elementary row
operations we shall now view this process in terms of matrix multiplication. This will
shed some light on both matrices and linear systems and will be useful for formulating
and proving the main result of this chapter, the Invertible Matrix Theorem, which will be
presented towards the end of this section. Before doing so, however, we shall consider the
effect of multiplying both sides of a linear system in matrix form by an invertible matrix.

Lemma 2.32. Let A be an m × n matrix and let b ∈ Rm . Suppose that M is an invertible

m × m matrix. The following two systems are equivalent:

Ax = b (2.5)

M Ax = M b (2.6)

Proof. Note that if x satisfies (2.5), then it clearly satisfies (2.6). Conversely, suppose
that x satisfies (2.6), that is,
M Ax = M b .
Since M is invertible, we may multiply both sides of the above equation by M −1 from the
left to obtain
M −1 M Ax = M −1 M b ,
so IAx = Ib, and hence Ax = b, that is, x satisfies (2.5).
2.5. ELEMENTARY MATRICES AND THE INVERTIBLE MATRIX THEOREM 17

We now come back to the idea outlined at the beginning of this section. It turns out
that we can ‘algebraize’ the process of applying an elementary row operation to a matrix
A by left-multiplying A by a certain type of matrix, defined as follows:
Definition 2.33. An elementary matrix of type I (respectively, type II, type III) is
a matrix obtained by applying an elementary row operation of type I (respectively, type
II, type III) to an identity matrix.
Example 2.34.
 
0 1 0
type I: E1 = 1 0 0 (take I3 and swap rows 1 and 2)
0 0 1
 
1 0 0
type II: E2 = 0
 1 0 (take I3 and multiply row 3 by 4)
0 0 4
 
1 0 2
type III: E3 = 0
 1 0 (take I3 and add 2 times row 3 to row 1)
0 0 1

Let us now consider the effect of left-multiplying an arbitrary 3 × 3 matrix A in turn

by each of the three elementary matrices given in the previous example.
Example 2.35. Let A = (aij )3×3 and let El (l = 1, 2, 3) be defined as in the previous
example. Then
    
0 1 0 a11 a12 a13 a21 a22 a23
E 1 A = 1 0 0 a21 a22 a23  = a11 a12 a13  ,
0 0 1 a31 a32 a33 a31 a32 a33
    
1 0 0 a11 a12 a13 a11 a12 a13
E2 A = 0 1 0 a21 a22 a23  =  a21 a22 a23  ,
0 0 4 a31 a32 a33 4a31 4a32 4a33
    
1 0 2 a11 a12 a13 a11 + 2a31 a12 + 2a32 a13 + 2a33
E3 A = 0 1 0  a21 a22 a23  =  a21 a22 a23 .
0 0 1 a31 a32 a33 a31 a32 a33

You should now pause and marvel at the following observation: interchanging rows 1 and
2 of A produces E1 A, multiplying row 3 of A by 4 produces E2 A, and adding 2 times row
3 to row 1 of A produces E3 A.
This example should convince you of the truth of the following theorem, the proof
of which will be omitted as it is straightforward, slightly lengthy and not particularly
instructive.
Theorem 2.36. If E is an m × m elementary matrix obtained from I by an elementary
row operation, then left-multiplying an m × n matrix A by E has the effect of performing
that same row operation on A.
Slightly deeper is the following:
Theorem 2.37. If E is an elementary matrix, then E is invertible and E −1 is an ele-
mentary matrix of the same type.
18 CHAPTER 2. MATRIX ALGEBRA

Proof. The assertion follows from the previous theorem and the observation that an ele-
mentary row operation can be reversed by an elementary row operation of the same type.
More precisely,
• if two rows of a matrix are interchanged, then interchanging them again restores
the original matrix;
• if a row is multiplied by α 6= 0, then multiplying the same row by 1/α restores the
original matrix;
• if α times row q has been added to row r, then adding −α times row q to row r
restores the original matrix.
Now, suppose that E was obtained from I by a certain row operation. Then, as we just
observed, there is another row operation of the same type that changes E back to I. Thus
there is an elementary matrix F of the same type as E such that F E = I. A moment’s
thought shows that EF = I as well, since E and F correspond to reverse operations. All
in all, we have now shown that E is invertible and its inverse E −1 = F is an elementary
matrix of the same type.
Example 2.38. Determine the inverses of the elementary matrices E1 , E2 , and E3 in
Example 2.34.
Solution. In order to transform E1 into I we need to swap rows 1 and 2 of E1 . The
elementary matrix that performs this feat is
 
0 1 0
E1−1 = 1 0 0 .
0 0 1

Similarly, in order to transform E2 into I we need to multiply row 3 of E2 by 41 . Thus

 
1 0 0
−1
E2 = 0  1 0 .
0 0 14
Finally, in order to transform E3 into I we need to add −2 times row 3 to row 1, and so
 
1 0 −2
E3−1 = 0 1 0  .
0 0 1

Before we come to the main result of this chapter we need some more terminology:
Definition 2.39. A matrix B is row equivalent to a matrix A if there exists a finite
sequence E1 , E2 , . . . , Ek of elementary matrices such that

B = Ek Ek−1 · · · E1 A .

In other words, B is row equivalent to A if and only if B can be obtained from A by a

finite number of row operations. In particular, two augmented matrices (A|b) and (B|c)
are row equivalent if and only if Ax = b and Bx = c are equivalent systems.
The following properties of row equivalent matrices are easily established:
2.5. ELEMENTARY MATRICES AND THE INVERTIBLE MATRIX THEOREM 19

1
Fact 2.40.

(a) A is row equivalent to itself;

(b) if A is row equivalent to B, then B is row equivalent to A;

(c) if A is row equivalent to B, and B is row equivalent to C, then A is row equivalent

to C.

Property (b) follows from Theorem 2.37. Details of the proof of (a), (b), and (c) are
left as an exercise.
We are now able to formulate and prove the first highlight of this module, a truly de-
lightful characterisation of invertibility of matrices. More precisely, the following theorem
provides three equivalent conditions for a matrix to be invertible. Later on in this course,
we will encounter further equivalent conditions.
Before stating the theorem we recall that the zero vector, denoted by 0, is the column
vector all of whose entries are zero.

Theorem 2.41 (Invertible Matrix Theorem). Let A be a square n × n matrix. The

following are equivalent:

(a) A is invertible;

(b) Ax = 0 has only the trivial solution;

(c) A is row equivalent to I;

(d) A is a product of elementary matrices.

Proof. We shall prove this theorem using a cyclic argument: we shall first show that (a)
implies (b), then (b) implies (c), then (c) implies (d), and finally that (d) implies (a).
This is a frequently used trick to show the logical equivalence of a list of assertions.
(a) ⇒ (b): Suppose that A is invertible. If x satisfies Ax = 0, then

x = Ix = (A−1 A)x = A−1 0 = 0 ,

so the only solution of Ax = 0 is the trivial solution.

(b) ⇒ (c): Use elementary row operations to bring the system Ax = 0 to the form
U x = 0, where U is in row echelon form. Since, by hypothesis, the solution of Ax = 0
and hence the solution of U x = 0 is unique, there must be exactly n leading variables.
Thus U is upper triangular with 1’s on the diagonal, and hence, the reduced row echelon
form of U is I. Thus A is row equivalent to I.
(c) ⇒ (d): If A is row equivalent to I, then there is a sequence E1 , . . . , Ek of elementary
matrices such that
A = Ek Ek−1 · · · E1 I = Ek Ek−1 · · · E1 ,
that is, A is a product of elementary matrices.
(d) ⇒ (a). If A is a product of elementary matrices, then A must be invertible, since
elementary matrices are invertible by Theorem 2.37 and since the product of invertible
matrices is invertible by Theorem 2.18.
1
In the language of MTH4104 (Introduction to Algebra) which some of you will have taken, these
statements mean that ‘row equivalence’ is an equivalence relation.
20 CHAPTER 2. MATRIX ALGEBRA

An immediate consequence of the previous theorem is the following perhaps surprising

result:
Corollary 2.42. Let A and C be square matrices such that CA = I, then also AC = I;
in particular both A and C are invertible with C = A−1 and A = C −1 .
Proof. By Exercise 1 on Coursework 3 it follows that A is invertible. But then C is
invertible since
C = CI = CAA−1 = IA−1 = A−1 .
Furthermore, C −1 = (A−1 )−1 = A and AC = C −1 C = I.
What is surprising about this result is the following: suppose we are given a square
matrix A. If we want to check that A is invertible, then, by the definition of invertibility,
we need to produce a matrix B such that AB = I and BA = I. The above corollary
tells us that if we have a candidate C for an inverse of A it is enough to check that either
AC = I or CA = I in order to guarantee that A is invertible with inverse C. This is a
non-trivial fact about matrices, which is often useful.

2.6 Gauss-Jordan inversion

The Invertible Matrix Theorem provides a simple method for inverting matrices. Recall
that the theorem states (amongst other things) that if A is invertible, then A is row
equivalent to I. Thus there is a sequence E1 , . . . Ek of elementary matrices such that
Ek Ek−1 · · · E1 A = I .
Multiplying both sides of the above equation by A−1 from the right yields
Ek Ek−1 · · · E1 = A−1 ,
that is,
Ek Ek−1 · · · E1 I = A−1 .
Thus, the same sequence of elementary row operations that brings an invertible matrix
to I, will bring I to A−1 . This gives a practical algorithm for inverting matrices, known
as Gauss-Jordan inversion.
Note that in the following we use a slight generalisation of the augmented matrix
notation. Given an m × n matrix A and an m-vector b we currently use (A|b) to denote
the m × (n + 1) matrix consisting of A with b attached as an extra column to the right
of A, and a vertical line in between them. Suppose now that B is an m × r matrix then
we write (A|B) for the m × (n + r) matrix consisting of A with B attached to the right
of A, and a vertical line separating them.
Gauss-Jordan inversion
Bring the augmented matrix (A|I) to reduced row echelon form. If A is row equivalent
to I, then (A|I) is row equivalent to (I|A−1 ). Otherwise, A does not have an inverse.
Example 2.43. Show that  
1 2 0
A = 2 5 3
0 3 8
is invertible and compute A−1 .
2.6. GAUSS-JORDAN INVERSION 21

Solution. Using Gauss-Jordan inversion we find

   
1 2 0 1 0 0 1 2 0 1 0 0
 2 5 3 0 1 0  ∼ R2 − 2R1  0 1 3 −2 1 0
0 3 8 0 0 1 0 3 8 0 0 1
   
1 2 0 1 0 0 1 2 0 1 0 0
∼  0 1 3 −2 1 0 ∼   0 1 3 −2 1 0 
R3 − 3R2 0 0 −1 6 −3 1 (−1)R3 0 0 1 −6 3 −1
   
1 2 0 1 0 0 R1 − 2R2 1 0 0 −31 16 −6
∼ R2 − 3R3  0 1 0 16 −8 3  ∼  0 1 0 16 −8 3  .
0 0 1 −6 3 −1 0 0 1 −6 3 −1

Thus A is invertible (because it is row equivalent to I3 ) and

 
−31 16 −6
A−1 =  16 −8 3  .
−6 3 −1
22 CHAPTER 2. MATRIX ALGEBRA
Chapter 3

Determinants

We will define the important concept of a determinant, which is a useful invariant for
general n × n matrices. We will discuss the most important properties of determinants,
and illustrate what they are good for and how calculations involving determinants can be
simplified.

3.1 General definition of determinants

Let A = (aij ) be a 2 × 2 matrix. Recall that the determinant of A was defined by
a11 a12
det(A) = = a11 a22 − a21 a12 . (3.1)
a21 a22
In other words, with every 2 × 2 matrix A it is possible to associate a scalar, called the
determinant of A, which is given by a certain sum of products of the entries of A. The
following fact was proved in Geometry I by a brute force calculation:
Fact 3.1. If A and B are 2 × 2 matrices then
(a) det(A) 6= 0 if and only if A is invertible;
(b) det(AB) = det(A) det(B).
This fact reveals one of the main motivations to introduce this somewhat non-intuitive
object: the determinant of a matrix allows us to decide whether a matrix is invertible or
not.
In this chapter we introduce determinants for arbitrary square matrices, study some of
their properties, and then prove the generalisation of the above fact for arbitrary square
matrices.
Before giving the general definition of the determinant of an n × n matrix, let us recall
the definition of 3 × 3 determinants given in Geometry I:
If A = (aij ) is a 3 × 3 matrix, then its determinant is defined by
a11 a12 a13
a a a a a a
det(A) = a21 a22 a23 = a11 22 23 − a21 12 13 + a31 12 13 (3.2)
a32 a33 a32 a33 a22 a23
a31 a32 a33
= a11 a22 a33 − a11 a32 a23 − a21 a12 a33 + a21 a32 a13 + a31 a12 a23 − a31 a22 a13 .
Notice that the determinant of a 3 × 3 matrix A is given in terms of the determinants
of certain 2 × 2 submatrices of A. In general, we shall see that the determinant of a 4 × 4
matrix is given in terms of the determinants of 3 × 3 submatrices, and so forth. Before
stating the general definition we introduce a convenient short-hand:

23
24 CHAPTER 3. DETERMINANTS

Notation 3.2. For any square matrix A, let Aij denote the submatrix formed by deleting
the i-th row and the j-th column of A.
Example 3.3. If  
3 2 5 −1
−2 9 0 6
A=
 7 −2 −3 1  ,


4 −5 8 −4
then  
3 2 −1
A23 = 7 −2 1  .
4 −5 −4
If we now define the determinant of a 1 × 1 matrix A = (aij ) by det(A) = a11 , we can
recast (3.1) and (3.2) as follows:
• if A = (aij )2×2 then

det(A) = a11 det(A11 ) − a21 det(A21 ) ;

• if A = (aij )3×3 then

det(A) = a11 det(A11 ) − a21 det(A21 ) + a31 det(A31 ) .

This observation motivates the following recursive definition:

Definition 3.4. Let A = (aij ) be an n × n matrix. The determinant of A, written
det(A), is defined as follows:
• If n = 1, then det(A) = a11 .
• If n > 1 then det(A) is the sum of n terms of the form ±ai1 det(Ai1 ), with plus and
minus signs alternating, and where the entries a11 , a21 , . . . , an1 are from the first
column of A. In symbols:

det(A) = a11 det(A11 ) − a21 det(A21 ) + · · · + (−1)n+1 an1 det(An1 )

Xn
= (−1)i+1 ai1 det(Ai1 ) .
i=1

Example 3.5. Compute the determinant of

 
0 0 7 −5
−2 9 6 −8
A= .
0 0 −3 2 
0 3 −1 4
Solution.
0 0 7 −5
0 7 −5
−2 9 6 −8 7 −5
= −(−2) 0 −3 2 = 2 · 3 = 2 · 3 · [7 · 2 − (−3) · (−5)] = −6 .
0 0 −3 2 −3 2
3 −1 4
0 3 −1 4
3.1. GENERAL DEFINITION OF DETERMINANTS 25

To state the next theorem, it will be convenient to write the definition of det(A) in a
slightly different form.

Definition 3.6. Given a square matrix A = (aij ), the (i, j)-cofactor of A is the number
Cij defined by
Cij = (−1)i+j det(Aij ) .

Thus, the definition of det(A) reads

det(A) = a11 C11 + a21 C21 + · · · + an1 Cn1 .

This is called the cofactor expansion down the first column of A. There is nothing
special about the first column, as the next theorem shows:

Theorem 3.7 (Cofactor Expansion Theorem). The determinant of an n × n matrix A

can be computed by a cofactor expansion across any column or row. The expansion down
the j-th column is
det(A) = a1j C1j + a2j C2j + · · · + anj Cnj
and the cofactor expansion across the i-th row is

det(A) = ai1 Ci1 + ai2 Ci2 + · · · + ain Cin .

Although this theorem is fundamental for the development of determinants, we shall

not prove it here, as it would lead to a rather lengthy workout.
Before moving on, notice that the plus or minus sign in the (i, j)-cofactor depends on
the position of aij in the matrix, regardless of aij itself. The factor (−1)i+j determines
the following checkerboard pattern of signs
 
+ − + ···
− + − 
.
 
+ − +
 
.. ...
.

Example 3.8. Use a cofactor expansion across the second row to compute det(A), where
 
4 −1 3
A = 0 0 2  .
1 0 7

Solution.

det(A) = a21 C21 + a22 C22 + a23 C23

= (−1)2+1 a21 det(A21 ) + (−1)2+2 a22 det(A22 ) + (−1)2+3 a23 det(A23 )
−1 3 4 3 4 −1
= −0 +0 −2
0 7 1 7 1 0
= −2[4 · 0 − 1 · (−1)] = −2 .
26 CHAPTER 3. DETERMINANTS

Example 3.9. Compute det(A), where

 
3 0 0 0 0
−2 5 0 0 0
 
A=  9 −6 4 −1 3 .

2 4 0 0 2
8 3 1 0 7
Solution. Notice that all entries but the first of row 1 are 0. Thus it will shorten our
labours if we expand across the first row:
5 0 0 0
−6 4 −1 3
det(A) = 3 .
4 0 0 2
3 1 0 7
Again it is advantageous to expand this 4 × 4 determinant across the first row:
4 −1 3
det(A) = 3 · 5 · 0 0 2 .
1 0 7
We have already computed the value of the above 3 × 3 determinant in the previous
example and found it to be equal to −2. Thus det(A) = 3 · 5 · (−2) = −30.
Notice that the matrix in the previous example was almost lower triangular. The
method of this example is easily generalised to prove the following theorem:
Theorem 3.10. If A is either an upper or a lower triangular matrix, then det(A) is the
product of the diagonal entries of A.

3.2 Properties of determinants

We saw time and again in this module that elementary row operations play a fundamental
role in matrix theory. It is only natural to enquire how det(A) behaves when an elementary
row operation is applied to A.
Theorem 3.11. Let A be a square matrix.
(a) If two rows of A are interchanged to produce B, then det(B) = − det(A).
(b) If one row of A is multiplied by α to produce B, then det(B) = α det(A).
(c) If a multiple of one row of A is added to another row to produce a matrix B then
det(B) = det(A).
Proof. These assertions follow from a slightly stronger result to be proved later in this
chapter (see Theorem 3.21).
Example 3.12.
1 2 3 4 5 6
(a) 4 5 6 = − 1 2 3 by (a) of the previous theorem.
7 8 9 7 8 9
3.2. PROPERTIES OF DETERMINANTS 27

0 1 2 0 1 2
(b) 3 12 9 = 3 1 4 3 by (b) of the previous theorem.
1 2 1 1 2 1

3 1 0 3 1 0
(c) 4 2 9 = 7 3 9 by (c) of the previous theorem.
0 −2 1 0 −2 1

The following examples show how to use the previous theorem for the effective com-
putation of determinants:

Example 3.13. Compute

3 −1 2 −5
0 5 −3 −6
.
−6 7 −7 4
−5 −8 0 9
Solution. Perhaps the easiest way to compute this determinant is to spot that when
adding two times row 1 to row 3 we get two identical rows, which, by another application
of the previous theorem, implies that the determinant is zero:

3 −1 2 −5 3 −1 2 −5
0 5 −3 −6 0 5 −3 −6
=
−6 7 −7 4 R3 + 2R1 0 5 −3 −6
−5 −8 0 9 −5 −8 0 9
3 −1 2 −5
0 5 −3 −6
= = 0,
R3 − R2 0 0 0 0
−5 −8 0 9

by a cofactor expansion across the third row.

Example 3.14. Compute det(A), where

 
0 1 2 −1
2 5 −7 3 
A= 0
.
3 6 2
−2 −5 4 −2

Solution. Here we see that the first column already has two zero entries. Using the
previous theorem we can introduce another zero in this column by adding row 2 to row
4. Thus
0 1 2 −1 0 1 2 −1
2 5 −7 3 2 5 −7 3
det(A) = = .
0 3 6 2 0 3 6 2
−2 −5 4 −2 0 0 −3 1
If we now expand down the first column we see that

1 2 −1
det(A) = −2 3 6 2 .
0 −3 1
28 CHAPTER 3. DETERMINANTS

The 3 × 3 determinant above can be further simplified by subtracting 3 times row 1 from
row 2. Thus
1 2 −1
det(A) = −2 0 0 5 .
0 −3 1
Finally we notice that the above determinant can be brought to triangular form by swap-
ping row 2 and row 3, which changes the sign of the determinant by the previous theorem.
Thus
1 2 −1
det(A) = (−2) · (−1) 0 −3 1 = (−2) · (−1) · 1 · (−3) · 5 = −30 ,
0 0 5
by Theorem 3.10.
We are now able to prove the first important result about determinants. It allows us
to decide whether a matrix is invertible or not by computing its determinant. It will play
an important role in later chapters.

Theorem 3.15. A matrix A is invertible if and only if det(A) 6= 0.

Proof. Bring A to row echelon form U (which is then necessarily upper triangular). Since
we can achieve this using elementary row operations, and since, in the process we only
ever multiply a row by a non-zero scalar det(A) = γ det(U ) for some γ with γ 6= 0, by
Theorem 3.11. If A is invertible, then det(U ) = 1, since U is upper triangular with 1’s on
the diagonal, and hence det(A) = γ det(U ) 6= 0. Otherwise, at least one diagonal entry of
U is zero, so det(U ) = 0, and hence det(A) = γ det(U ) = 0.

Definition 3.16. A square matrix A is called singular if det(A) = 0. Otherwise it is

said to be nonsingular.

Corollary 3.17. A matrix is invertible if and only if it is nonsingular

Our next result shows what effect transposing a matrix has on its determinant:

Theorem 3.18. If A is an n × n matrix, then det(A) = det(AT ).

Proof. The proof is by induction on n (that is, the size of A).1 The theorem is obvious for
n = 1. Suppose now that it has already been proved for k × k matrices for some integer
k. Our aim now is to show that the assertion of the theorem is true for (k + 1) × (k + 1)
matrices as well. Let A be a (k + 1) × (k + 1) matrix. Note that the (i, j)-cofactor of A
equals the (i, j)-cofactor of AT , because the cofactors involve k × k determinants only, for
which we assumed that the assertion of the theorem holds. Hence

cofactor expansion of det(A) across first row

=cofactor expansion of det(AT ) down first column

so det(A) = det(AT ).
Let’s summarise: the theorem is true for 1 × 1 matrices, and the truth of the theorem
for k × k matrices for some k implies the truth of the theorem for (k + 1) × (k + 1)
1
If you have never encountered this method of proof, don’t despair! Simply read through the following
argument. The last paragraph explains the underlying idea of this method.
3.2. PROPERTIES OF DETERMINANTS 29

matrices. Thus, the theorem must be true for 2 × 2 matrices (choose k = 1); but since
we now know that it is true for 2 × 2 matrices, it must be true for 3 × 3 matrices as well
(choose k = 2); continuing with this process, we see that the theorem must be true for
matrices of arbitrary size.
By the previous theorem, each statement of the theorem on the behaviour of determi-
nants under row operations (Theorem 3.11) is also true if the word ‘row’ is replaced by
‘column’, since a row operation on AT amounts to a column operation on A.
Theorem 3.19. Let A be a square matrix.
(a) If two columns of A are interchanged to produce B, then det(B) = − det(A).
(b) If one column of A is multiplied by α to produce B, then det(B) = α det(A).
(c) If a multiple of one column of A is added to another column to produce a matrix B
then det(B) = det(A).
Example 3.20. Find det(A) where
 
1 3 4 8
−1 2 1 9
A=
2
.
5 7 0
3 −4 −1 5
Solution. Adding column 1 to column 2 gives
1 3 4 8 1 4 4 8
−1 2 1 9 −1 1 1 9
det(A) = = .
2 5 7 0 2 7 7 0
3 −4 −1 5 3 −1 −1 5
Now subtracting column 3 from column 2 the determinant is seen to vanish by a cofactor
expansion down column 2.
1 0 4 8
−1 0 1 9
det(A) = = 0.
2 0 7 0
3 0 −1 5

Our next aim is to prove that determinants are multiplicative, that is, det(AB) =
det(A) det(B) for any two square matrices A and B of the same size. We start by estab-
lishing a baby-version of this result, which, at the same time, proves the theorem on the
behaviour of determinants under row operations stated earlier (see Theorem 3.11).
Theorem 3.21. If A is an n × n matrix and E an elementary n × n matrix, then
det(EA) = det(E) det(A)
with

−1
 if E is of type I (interchanging two rows)
det(E) = α if E is of type II (multiplying a row by α) .

1 if E is of type III (adding a multiple of one row to another)

30 CHAPTER 3. DETERMINANTS

Proof. By induction on the size of A. The case where A is a 2 × 2 matrix is an easy

exercise (see Exercise 1, Coursework 4). Suppose now that the theorem has been verified
for determinants of k ×k matrices for some k with k ≥ 2. Let A be (k +1)×(k +1) matrix
and write B = EA. Expand det(EA) across a row that is unaffected by the action of E
on A, say, row i. Note that Bij is obtained from Aij by the same type of elementary row
operation that E performs on A. But since these matrices are only k × k, our hypothesis
implies that
det(Bij ) = r det(Aij ) ,
where r = −1, α, 1 depending on the nature of E.
Now by a cofactor expansion across row i
k+1
X
det(EA) = det(B) = aij (−1)i+j det(Bij )
j=1
k+1
X
= aij (−1)i+j r det(Aij )
j=1

= r det(A) .

In particular, taking A = Ik+1 we see that det(E) = −1, α, 1 depending on the nature of
E.
To summarise: the theorem is true for 2 × 2 matrices and the truth of the theorem
for k × k matrices for some k ≥ 2 implies the truth of the theorem for (k + 1) × (k + 1)
matrices. By the principle of induction the theorem is true for matrices of any size.

Using the previous theorem we are now able to prove the second important result of
this chapter:

Theorem 3.22. If A and B are square matrices of the same size, then

det(AB) = det(A) det(B) .

Proof. Case I: If A is not invertible, then neither is AB (for otherwise A(B(AB)−1 ) = I,

which by the corollary to the Invertible Matrix Theorem would force A to be invertible).
Thus, by Theorem 3.15,

det(AB) = 0 = 0 · det(B) = det(A) det(B) .

Case II: If A is invertible, then by the Invertible Matrix Theorem A is a product of

elementary matrices, that is, there exist elementary matrices E1 , . . . , Ek , such that

A = Ek Ek−1 · · · E1 .

For brevity, write |A| for det(A). Then, by the previous theorem,

|AB| = |Ek · · · E1 B| = |Ek ||Ek−1 · · · E1 B| = . . .

= |Ek | · · · |E1 ||B| = . . . = |Ek · · · E1 ||B|
= |A||B| .
3.3. CRAMER’S RULE AND A FORMULA FOR A−1 31

3.3 Cramer’s Rule and a formula for A−1

In the following, we use a shorthand to identify a matrix by its columns. Let A be an
m × n matrix. If aj ∈ Rm is the j-th column vector of A, then we write

A = (a1 . . . an ) .

Note that if B is an l × m matrix then, by the definition of matrix multiplication,

BA = (Ba1 . . . Ban ) ,

that is, the j-th column of BA is Baj .

Cramer’s Rule is a curious formula that allows us to write down the solution for certain
n × n systems in terms of quotients of two determinants. Before stating it we need some
more notation.
For any n × n matrix A and any b ∈ Rn write Ai (b) for the matrix obtained from A
by replacing column i by b, that is,

Ai (b) = (a1 . . . b . . . an ) .
col i

Theorem 3.23 (Cramer’s Rule). Let A be an invertible n × n matrix. For any b ∈ Rn ,

the unique solution x of Ax = b has entries given by
det(Ai (b))
xi = for i = 1, . . . , n .
det(A)
Proof. Let a1 . . . , an be the columns of A, and e1 , . . . , en the columns of the n × n identity
matrix I. Then

AIi (x) = A(e1 . . . x . . . en )

= (Ae1 . . . Ax . . . Aen )
= (a1 . . . b . . . an )
= Ai (b) ,

and hence
det(A) det(Ii (x)) = det(Ai (b)) .
But det(Ii (x)) = xi by a cofactor expansion across row i, so

det(Ai (b))
xi = ,
det(A)

since det(A) 6= 0.

Example 3.24. Use Cramer’s Rule to solve the system

3x1 − 2x2 = 6
.
−5x1 + 4x2 = 8

Solution. Write the system as Ax = b, where

   
3 −2 6
A= , b= .
−5 4 8
32 CHAPTER 3. DETERMINANTS

Then    
6 −2 3 6
A1 (b) = , A2 (b) = ,
8 4 −5 8
and Cramer’s Rule gives
det(A1 (b)) 40
x1 = = = 20 ,
det(A) 2
det(A2 (b)) 54
x2 = = = 27 .
det(A) 2

Cramer’s Rule is not really useful for practical purposes (except for very small sys-
tems), since evaluation of determinants is time consuming when the system is large. For
3 × 3 systems and larger, you are better off using Gaussian elimination. Apart from its
intrinsic beauty, its main strength is as a theoretical tool. For example, it allows you to
study how sensitive the solution of Ax = b is to a change in an entry in A or b.
As an application of Cramer’s Rule, we shall now derive an explicit formula for the
inverse of a matrix. Before doing so we shall have another look at the process of inverting
a matrix. Again, denote the columns of In by e1 , . . . , en . The Gauss-Jordan inversion
process bringing (A|I) to (I|A−1 ) can be viewed as solving the n systems

Ax = e1 , Ax = e2 , ... Ax = en .

Thus the j-th column of A−1 is the solution of

Ax = ej ,

and the i-th entry of x is the (i, j)-entry of A−1 . By Cramer’s rule
det(Ai (ej ))
(i, j)-entry of A−1 = xi = . (3.3)
det(A)
A cofactor expansion down column i of Ai (ej )) shows that

det(Ai (ej )) = (−1)i+j det(Aji ) = Cji ,

where Cji is the (j, i)-cofactor of A. Thus, by (3.3), the (i, j)-entry of A−1 is the cofactor
Cji divided by det(A) (note that the order of the indices is reversed!). Thus
 
C11 C21 · · · Cn1
1  C12
 C22 · · · Cn2 
A−1 = ..  . (3.4)

 . .. ..
det(A)  .. . . . 
C1n C2n · · · Cn
| {z }
= adj (A)

The matrix of cofactors on the right of (3.4) is called the adjugate of A, and is denoted
by adj (A). The following theorem is simply a restatement of (3.4):
Theorem 3.25 (Inverse Formula). Let A be an invertible matrix. Then
1
A−1 = adj (A) .
det(A)
3.3. CRAMER’S RULE AND A FORMULA FOR A−1 33

Example 3.26. Find the inverse of the following matrix using the Inverse Formula
 
1 3 −1
A = −2 −6 0  .
1 4 −3

Proof. First we need to calculate the 9 cofactors of A:

−6 0 −2 0 −2 −6
C11 = + = 18 , C12 = − = −6 , C13 = + = −2 ,
4 −3 1 −3 1 4
3 −1 1 −1 1 3
C21 = − = 5, C22 = + = −2 , C23 = − = −1 ,
4 −3 1 −3 1 4
3 −1 1 −1 1 3
C31 = + = −6 , C32 = − = 2, C33 = + = 0.
−6 0 −2 0 −2 −6

Thus  
18 5 −6
adj (A) = −6 −2 2  ,
−2 −1 0
and since det(A) = 2, we have
5
 
9 2
−3
A−1 = −3 −1 1  .
−1 − 12 0

Note that the above calculations are just as laborious as if we had used the Gauss-
Jordan inversion process to compute A−1 . As with Cramer’s Rule, the deceptively neat
formula for the inverse is not useful if you want to invert larger matrices. As a rule, for
matrices larger than 3 × 3 the Gauss-Jordan inversion algorithm is much faster.
34 CHAPTER 3. DETERMINANTS
 Part II

Linear Algebra

35
Chapter 4

Vector Spaces

In this chapter, we will study abstract vector spaces. Roughly speaking a vector space is
a mathematical structure on which an operation of addition and an operation of scalar
multiplication is defined, and we require these operations to obey a number of algebraic
rules. We will introduce important general concepts such as linear independence, basis,
dimension, coordinates and discuss their usefulness.

4.1 Definition and examples

We have already encountered examples of vector spaces in this module. Recall that Rn is
the collection of all n-vectors. On Rn two operations were defined:
• addition: if    
x1 y1
x =  ...  ∈ Rn ,
 .. 
and y =  .  ∈ Rn ,
 
xn yn
then x + y is the n-vector given by
 
x1 + y1
x + y =  ...  .
 
xn + yn

• scalar multiplication: if

x1
x =  ...  ∈ Rn , and α is a scalar
 
xn
then αx is the n-vector given by
 
αx1
αx =  ...  .
 
αxn

After these operations were defined, it turned out that they satisfy a number of rules (see
Theorem 2.7). We are now going to turn this process on its head. That is, we start from a
set on which two operations are defined, we postulate that these operations satisfy certain
rules, and we call the resulting structure a ‘vector space’:

37
38 CHAPTER 4. VECTOR SPACES

Definition 4.1. A vector space is a non-empty set V on which are defined two oper-
ations, called addition and scalar multiplication, such that the following axioms hold for
all u, v, w in V and all scalars α, β:

(C1) the sum of u and v, denoted by u + v, is in V ;

(C2) the scalar multiple of u by α, denoted by αu, is in V ;

(A1) u + v = v + u;

(A2) u + (v + w) = (u + v) + w;

(A3) there is an element 0 in V such that u + 0 = u;

(A4) for each u in V there is an element −u in V such that u + (−u) = 0;

(A5) α(u + v) = αu + αv;

(A6) (α + β)u = αu + βu;

(A7) (αβ)u = α(βu);

(A8) 1u = u.

We will refer to V as the universal set for the vector space. Its elements are called
vectors, and we usually write them using bold letters u, v, w, etc.
The term ‘scalar’ will usually refer to a real number, although later on we will some-
times allow scalars to be complex numbers. To distinguish these cases we will use the
term real vector space (if the scalars are real numbers) or complex vector space (if
the scalars are complex numbers). For the moment, however, we will only consider real
vector spaces.
Note that in the above definition the axioms (C1) and (C2), known as closure axioms,
simply state that the two operations produce values in V . The other eight axioms, also
known as the classical vector space axioms, stipulate how the two operations interact.
Let’s have a look at some examples:

Example 4.2. Let Rm×n denote the set of all m × n matrices. Define addition and scalar
multiplication of matrices in the usual way. Then Rm×n is a vector space by Theorem 2.7.

Example 4.3. Let Pn denote the set of all polynomials with real coefficients of degree
less or equal than n. Thus, an element p in Pn is of the form

p(t) = a0 + a1 t + a2 t2 + · · · + an tn ,

where the coefficients a0 , . . . , an and the variable t are real numbers.

Define addition and scalar multiplication on Pn as follows: if q ∈ Pn is given by

q(t) = b0 + b1 t + b2 t2 + · · · + bn tn ,

p is as above and α a scalar, then

• p + q is the polynomial

(p + q)(t) = (a0 + b0 ) + (a1 + b1 )t + · · · + (an + bn )tn

4.1. DEFINITION AND EXAMPLES 39

• αp is the polynomial

(αp)(t) = (αa0 ) + (αa1 )t + · · · + (αan )tn .

Note that (C1) and (C2) clearly hold, since if p, q ∈ Pn and α is a scalar, then p + q and
αp are again polynomials of degree less than n. Axiom (A1) holds since if p and q are
as above, then

(p + q)(t) = (a0 + b0 ) + (a1 + b1 )t + · · · + (an + bn )tn

= (b0 + a0 ) + (b1 + a1 )t + · · · + (bn + an )tn
= (q + p)(t)

so p + q = q + p. A similar calculation shows that (A2) holds. Axiom (A3) holds if we

let 0 be the zero polynomial, that is

0(t) = 0 + 0 · t + · · · + 0 · tn ,

since then (p + 0)(t) = p(t), that is, p + 0 = p. Axiom (A4) holds if, given p ∈ Pn we
set −p = (−1)p, since then

(p + (−p))(t) = (a0 − a0 ) + (a1 − a1 )t + · · · + (an − an )tn = 0(t) ,

that is p + (−p) = 0. The remaining axioms are easily verified as well, using familiar
properties of real numbers.
Example 4.4. Let C[a, b] denote the set of all real-valued functions that are defined and
continuous on the closed interval [a, b]. For f , g ∈ C[a, b] and α a scalar, define f + g and
αf pointwise, that is, by

(f + g)(t) = f (t) + g(t) for all t ∈ [a, b]

(αf )(t) = αf (t) for all t ∈ [a, b]

Equipped with these operations, C[a, b] is a vector space. The closure axiom (C1) holds
because the sum of two continuous functions on [a, b] is continuous on [a, b], and (C2)
holds because a constant times a continuous function on [a, b] is again continuous on
[a, b]. Axiom (A1) holds as well, since for all t ∈ [a, b]

(f + g)(t) = f (t) + g(t) = g(t) + f (t) = (g + f )(t) ,

so f + g = g + f . Axiom (A3) is satisfied if we let 0 be the zero function,

0(t) = 0 for all t ∈ [a, b] ,

since then
(f + 0)(t) = f (t) + 0(t) = f (t) + 0 = f (t) ,
so f + 0 = f . Axiom (A4) holds if, given f ∈ C[a, b], we let −f be the function

(−f )(t) = −f (t) for all t ∈ [a, b],

since then
(f + (−f ))(t) = f (t) + (−f )(t) = f (t) − f (t) = 0 = 0(t) ,
that is, f + (−f ) = 0. We leave it as an exercise to verify the remaining axioms.
40 CHAPTER 4. VECTOR SPACES

Let us list a number of elementary properties of vector spaces.

Theorem 4.5. If V is a vector space and u and v are elements in V , then
(a) 0u = 0;

(b) if u + v = 0 then v = −u;1

(c) (−1)u = −u.

The proof consists of a rather mechanical successive application of vector space axioms,
we omit it here to have more time for other, more interesting things.

4.2 Subspaces
Given a vector space V , a ‘subspace’ of V is, roughly speaking, a subset of V that inherits
the vector space structure from V , and can thus be considered as a vector space in its own
right. One of the main motivations to consider such ‘substructures’ of vector spaces, is
the following. As you might have noticed, it can be frightfully tedious to check whether a
given set, call it H, is a vector space. Suppose that we know that H is a subset of a larger
set V equipped with two operations (addition and scalar multiplication), for which we
have already checked that the vector space axioms are satisfied. Now, in order for H to
be the universal set of a vector space equipped with the operations of addition and scalar
multiplication inherited from V , the set H should certainly be closed under addition and
scalar multiplication (so that (C1) and (C2) are satisfied). Checking these two axioms is
enough in order for H to be a vector space in its own right, as we shall see shortly. To
summarise: if H is a subset of a vector space V , and if H is closed under addition and
scalar multiplication, then H is a vector space in its own right. So instead of having to
check 10 axioms, we only need to check two in this case. Let’s cast these observations
into the following definition:
Definition 4.6. Let H be a nonempty subset of a vector space V . Suppose that H
satisfies the following two conditions:
(i) if u, v ∈ H, then u + v ∈ H;

(ii) if u ∈ H and α is a scalar, then αu ∈ H.

Then H is said to be a subspace of V .
Theorem 4.7. Let H be a subspace of a vector space V . Then H with addition and scalar
multiplication inherited from V is a vector space in its own right.
Proof. Clearly, by definition of a subspace, (C1) and (C2) are satisfied. Axioms (A3)
and (A4) follow from Theorem 4.5 and condition (ii) of the definition of a subspace. The
remaining axioms are valid for any elements in V , so, in particular, they are valid for any
elements in H as well.
Remark 4.8. If V is a vector space, then {0} and V are clearly subspaces of V . All
other subspaces are said to be proper subspaces of V . We call {0} the zero subspace
of V .
1
In the language of MTH4104 (Introduction to Algebra) this statement says that the additive inverse
is unique.
 4.2. SUBSPACES 41

Let’s have a look at some more concrete examples:

Example 4.9. Show that the following are subspaces of R3 :


(a) L = (r, s, t, )T r, s, t ∈ R and r = s = t ;2

(b) P = (r, s, t, )T r, s, t ∈ R and r − s + 3t = 0 .

Solution. (a) Notice that an arbitrary element in L is of the form r(1, 1, 1)T for some real
number r. Thus, in particular, L is not empty, since (0, 0, 0)T ∈ L. In order to check that
L is a subspace of R3 we need to check that conditions (i) and (ii) of Definition 4.6 are
satisfied.
We start with condition (i). Let x1 and x2 belong to L. Then x1 = r1 (1, 1, 1)T and
x2 = r2 (1, 1, 1)T for some real numbers r1 and r2 , so
     
1 1 1
x1 + x2 = r1 1 + r2 1 = (r1 + r2 ) 1 ∈ L .
1 1 1

Thus condition (i) holds.

We now check condition (ii). Let x ∈ L and let α be a real number. Then x =
r(1, 1, 1)T for some real number r ∈ R, so
 
1
αx = αr 1 ∈ L .

1

Thus condition (ii) holds.

Let’s summarise: the non-empty set L satisfies conditions (i) and (ii), that is, it is
closed under addition and scalar multiplication, hence L is a subspace of R3 as claimed.
(b) In order to see that P is a subspace of R3 we first note that (0, 0, 0)T ∈ P , so P is not
empty.
Next we check condition (i). Let x1 = (r1 , s1 , t1 )T ∈ P and x2 = (r2 , s2 , t2 )T ∈ P .
Then r1 − s1 + 3t1 = 0 and r2 − s2 + 3t2 = 0, so
 
r1 + r2
x 1 + x2 = s 1 + s 2  ∈ P ,
t1 + t2

since (r1 + r2 ) − (s1 + s2 ) + 3(t1 + t2 ) = (r1 − s1 + 3t1 ) + (r2 − s2 + 3t2 ) = 0 + 0 = 0. Thus

condition (i) holds.
We now check condition (ii). Let x = (r, s, t)T ∈ P and let α be a scalar. Then
r − s + 3t = 0 and  
αr
αx = αs ∈ P

αt
2
In order to save paper, hence trees and thus do our bit to prevent man-made climate change, we shall
sometimes write n-vectors x ∈ Rn in the form (x1 , . . . , xn )T . So, for example,
 
2
(2, 3, 1)T = 3 .
1
42 CHAPTER 4. VECTOR SPACES

since αr − αs + 3αt = α(r − s + 3t) = 0. Thus condition (ii) holds as well.

As P is closed under addition and scalar multiplication, P is a subspace of R3 as
claimed.
Remark 4.10. In the example above the two subspaces L and P of R3 can also be
thought of as geometric objects. More precisely, L can be interpreted geometrically as a
line through the origin with direction vector (1, 1, 1)T , while P can be interpreted as a
plane through the origin with normal vector (1, −1, 3)T .
More generally, all proper subspaces of R3 can be interpreted geometrically as either
lines or planes through the origin. Similarly, all proper subspaces of R2 can be interpreted
geometrically as lines through the origin.

Example 4.11. H = (r2 , s, r)T r, s ∈ R is not a subspace of R3 , since
     
1 1 2
0 ∈ H, but 2 0 = 0 6∈ H .
1 1 2

Example 4.12. The set

  
a b 2×2
H= ∈R a, b ∈ R ,
0 1

is not a subspace of R2×2 . In order to see this, note that every subspace must contain the
zero vector. However,
O2×2 6∈ H .
Example 4.13. Let H = { f ∈ C[−2, 2] | f (1) = 0 }. Then H is a subspace of C[−2, 2].
First observe that the zero function is in H, so H is not empty. Next we check that the
closure properties are satisfied.
Let f , g ∈ H . Then f (1) = 0 and g(1) = 0, so

(f + g)(1) = f (1) + g(1) = 0 + 0 = 0 ,

so f + g ∈ H. Thus H is closed under addition.

Let f ∈ H and α be a a real number. Then f (1) = 0 and

(αf )(1) = αf (1) = α · 0 = 0 ,

so αf ∈ H. Thus H is closed under scalar multiplication.

Since H is closed under addition and scalar multiplication it is a subspace of C[−2, 2]
as claimed.
A class of subspaces we have already encountered (but didn’t think about them in
this way) are the solution sets of homogeneous systems. More precisely, if A ∈ Rm×n is
the coefficient matrix of such a system, then the solution set can be thought of as the
collection of all x ∈ Rn with Ax = 0, and the collection of all such x is a subspace of Rn .
Before convincing us of this fact, we introduce some convenient terminology:
Definition 4.14. Let A ∈ Rm×n . Then

N (A) = { x ∈ Rn | Ax = 0 }

is called the nullspace of A.

4.3. THE SPAN OF A SET OF VECTORS 43

Theorem 4.15. If A ∈ Rm×n , then N (A) is a subspace of Rn .

Proof. Clearly 0 ∈ N (A), so N (A) is not empty.
If x, y ∈ N (A) then Ax = 0 and Ay = 0, so

A(x + y) = Ax + Ay = 0 + 0 = 0 ,

and hence x + y ∈ N (A).

Furthermore, if x ∈ N (A) and α is a real number then Ax = 0 and

A(αx) = α(Ax) = α0 = 0 ,

so αx ∈ N (A).
Thus N (A) is a subspace of Rn as claimed.
Example 4.16. Determine N (A) for
 
−3 6 −1 1 −7
A =  1 −2 2 3 −1 .
2 −4 5 8 −4

Solution. We need to find the solution set of Ax = 0. To do this you can use your favourite
method to solve linear systems. Perhaps the fastest one is to bring the augmented matrix
(A|0) to reduced row echelon form and write the leading variables in terms of the free
variables. In our case, we have
   
−3 6 −1 1 −7 0 1 −2 0 −1 3 0
 1 −2 2 3 −1 0  ∼ · · · ∼  0 0 1 2 −2 0  .
2 −4 5 8 −4 0 0 0 0 0 0 0

The leading variables are x1 and x3 , and the free variables are x2 , x4 and x5 . Now setting
x2 = α, x4 = β and x5 = γ we find x3 = −2x4 +2x5 = −2β +2γ and x1 = 2x2 +x4 −3x5 =
2α + β − 3γ. Thus
         
x1 2α + β − 3γ 2 1 −3
 x2   α  1 0 0
         
x3  =  −2β + 2γ  = α 0 + β −2 + γ  2  ,
         
 x4   β  0 1 0
x5 γ 0 0 1

hence        

 2 1 −3 

 1 0 0

       

    
N (A) = α 0 + β −2 + γ  2  α, β, γ ∈ R .
     


 0 1 0 


 
0 0 1
 

4.3 The span of a set of vectors

In this section we shall have a look at a way to construct subspaces from a collection of
vectors.
44 CHAPTER 4. VECTOR SPACES

Definition 4.17. Let v1 , . . . , vn be vectors in a vector space. The set of all linear combi-
nations of v1 , . . . , vn is called the span of v1 , . . . , vn and is denoted by Span (v1 , . . . , vn ),
that is,
Span (v1 , . . . , vn ) = { α1 v1 + · · · + αn vn | α1 , . . . , αn ∈ R } .
Example 4.18. Let e1 , e2 , e3 ∈ R3 be given by
     
1 0 0
e1 = 0 , e2 = 1 ,
   e3 = 0 .

0 0 1

Determine Span (e1 , e2 ) and Span (e1 , e2 , e3 ).

Solution. Since
   
α1 α1
α1 e1 + α2 e2 = α2  , while α1 e1 + α2 e2 + α3 e3 = α2  ,
0 α3

we see that
  
 x1 
Span (e1 , e2 ) = x2  ∈ R3 x3 = 0 , while Span (e1 , e2 , e3 ) = R3 .
x3
 

Notice that in the above example Span (e1 , e2 ) can be interpreted geometrically as
the x1 , x2 plane, that is, the plane containing the x1 - and the x2 -axis. In particular,
Span (e1 , e2 ) is a subspace of R3 . This is true more generally:
Example 4.19. Given vectors v1 and v2 in a vector space V , show that H = Span (v1 , v2 )
is a subspace of V .
Solution. Notice that 0 ∈ H (since 0 = 0v1 + 0v2 ), so H is not empty. In order to show
that H is closed under addition, let u and w be arbitrary vectors in H. Then there are
scalars α1 , α2 and β1 , β2 , such that

u = α 1 v 1 + α 2 v2 ,
w = β1 v1 + β2 v2 .

Now by axioms (A1), (A2) and (A6)

u + w = (α1 v1 + α2 v2 ) + (β1 v1 + β2 v2 ) = (α1 + β1 )v1 + (α2 + β2 )v2 ,

so u + w ∈ H.
In order to show that H is closed under scalar multiplication, let u ∈ H, say, u =
α1 v1 + α2 v2 , and let γ be a scalar. Then, by axioms (A5) and (A7)

γu = γ(α1 v1 + α2 v2 ) = (γα1 )v1 + (γα2 )v2 ,

so γu ∈ H.
More generally, using exactly the same method of proof, it is possible to show the
following:
4.3. THE SPAN OF A SET OF VECTORS 45

Theorem 4.20. Let v1 , . . . , vn be vectors in a vector space V . Then Span (v1 , . . . , vn ) is

a subspace of V .

We have just seen that the span of a collection of vectors in a vector space V is a
subspace of V . As we saw in Example 4.18, the span may be a proper subspace of V , or
it may be equal to all of V . The latter is sufficiently interesting a case to merit its own
definition:

Definition 4.21. Let V be a vector space, and let v1 , . . . , vn ∈ V . We say that the set
{v1 , . . . , vn } is a spanning set for V if

Span (v1 , . . . , vn ) = V .

If {v1 , . . . , vn } is a spanning set for V , we shall also say that {v1 , . . . , vn } spans V , that
v1 , . . . , vn span V or that V is spanned by v1 , . . . , vn .

Notice that the above definition can be rephrased as follows. A set {v1 , . . . , vn } is a
spanning set for V , if and only if every vector in V can be written as a linear combination
of v1 , . . . , vn .

Example 4.22. Which of the following sets are spanning sets for R3 ?

(a) (1, 0, 0)T , (0, 1, 0)T , (0, 0, 1)T , (1, 2, 4)T (b) (1, 1, 1)T , (1, 1, 0)T , (1, 0, 0)T
 

(c) (1, 0, 1)T , (0, 1, 0)T (d) (1, 2, 4)T , (2, 1, 3)T , (4, −1, 1)T
 

Solution. (only example (a) treated in the lectures, the other three examples kept here
for illustration) (a) Let (a, b, c)T be an arbitrary vector in R3 . Clearly
         
a 1 0 0 1
 b  = a 0 + b 1 + c 0 + 0 2 ,
c 0 0 1 4

so the set is a spanning set for R3 .

(b) Let (a, b, c)T be an arbitrary vector in R3 . We need to determine whether it is possible
to find constants α1 , α2 , α3 such that
       
1 1 1 a
α1 1 + α2 1 + α3 0 =  b  .
1 0 0 c

This means that the weights α1 , α2 and α3 have to satisfy the system

α1 + α2 + α3 = a
α1 + α2 = b
α1 = c

Since the coefficient matrix of the system is nonsingular, the system has a unique solution.
In fact, using back substitution we find
   
α1 c
α2  =  b − c  .
α3 a−b
46 CHAPTER 4. VECTOR SPACES

Thus        
a 1 1 1
 b  = c 1 + (b − c) 1 + (a − b) 0 ,
c 1 0 0
so the set is a spanning set for R3 .
(c) Noting that      
1 0 α1
α1 0 + α2 1 = α2  ,
1 0 α1
we see that a vector of the form (a, b, c)T with a 6= c cannot be in the span of the two
vectors. Thus the set is not a spanning set for R3 .
(d) Proceeding as in (b), we let (a, b, c)T be an arbitrary vector in R3 . Again, we need to
determine whether it is possible to find constants α1 , α2 , α3 such that
       
1 2 4 a
α1 2 + α2 1 + α3 −1 =  b  .
4 3 1 c
This means that the weights α1 , α2 and α3 have to satisfy the system
α1 + 2α2 + 4α3 = a
2α1 + α2 − α3 = b
4α1 + 3α2 + α3 = c
A short calculation shows that the coefficient matrix of the system is singular, from which
we could conclude that the system cannot have a solution for all a, b, c ∈ R. In other words,
the vectors cannot span R3 . It is however instructive to reach the same conclusion by a
slightly different route: using Gaussian elimination we see that the system is equivalent
to the following
α1 + 2α2 + 4α3 = a
2a−b
α2 + 3α3 = 3
0 = 2a + 5b − 3c
It follows that the system is consistent if and only if

2a + 5b − 3c = 0 .

Thus a vector (a, b, c)T in R3 belongs to the span of the vectors (1, 2, 4)T , (2, 1, 3)T , and
(4, −1, 1)T if and only if 2a + 5b − 3c = 0. In other words, not every vector in R3 can be
written as a linear combination of the vectors (1, 2, 4)T , (2, 1, 3)T , and (4, −1, 1)T , so in
particular these vectors cannot span R3 .
Example 4.23. Show that {p1 , p2 , p3 } is a spanning set for P2 , where

p1 (x) = 2 + 3x + x2 , p2 (x) = 4 − x , p3 (x) = −1 .

Solution. Let p be an arbitrary polynomial in P2 , say, p(x) = a + bx + cx2 . We need to

show that it is possible to find weights α1 , α2 and α3 such that

α1 p1 + α2 p2 + α3 p3 = p ,

that is
α1 (2 + 3x + x2 ) + α2 (4 − x) − α3 = a + bx + cx2 .
4.4. LINEAR INDEPENDENCE 47

Comparing coefficients we find that the weights have to satisfy the system

2α1 + 4α2 − α3 = a
3α1 − α2 = b
α1 = c

The coefficient matrix is nonsingular, so the system must have a unique solution for all
choices of a, b, c. In fact, using back substitution yields α1 = c, α2 = 3c − b, α3 =
14c − 4b − a. Thus {p1 , p2 , p3 } is a spanning set for P2 .

Example 4.24. Find a spanning set for N (A), where

 
−3 6 −1 1 −7
A =  1 −2 2 3 −1 .
2 −4 5 8 −4

Proof. We have already calculated N (A) for this matrix in Example 4.16, and found that
       

 2 1 −3 

 1 0 0

       

    
 0   −2
N (A) = α   + β   + γ   α, β, γ ∈ R .
  2 


 0 1 0 


 
0 0 1
 


Thus, (2, 1, 0, 0, 0)T , (1, 0, −2, 1, 0)T , (−3, 0, 2, 0, 1)T is a spanning set for N (A).

4.4 Linear independence

The notion of linear independence plays a fundamental role in the theory of vector spaces.
Roughly speaking, it is a certain minimality property a collection of vectors in a vector
space may or may not have. To motivate it consider the following vectors in R3 :
     
−3 2 −5
x1 =  1  , x2 = −1 , x3 =  1  . (4.1)
2 1 8

Let’s ask the question: what is Span (x1 , x2 , x3 )?

Notice that
x3 = 3x1 + 2x2 . (4.2)
Thus any linear combination of x1 , x2 , x3 can be written as a linear combination of x1
and x2 alone, because

α1 x1 + α2 x2 + α3 x3 = α1 x1 + α2 x2 + α3 (3x1 + 2x2 )
= (α1 + 3α3 )x1 + (α2 + 2α3 )x2 .

Hence
Span (x1 , x2 , x3 ) = Span (x1 , x2 ) .
Observing that equation (4.2) can be written as

3x1 + 2x2 − x3 = 0 , (4.3)

48 CHAPTER 4. VECTOR SPACES

we see that any of the three vectors can be expressed as a linear combination of the other
two, so
Span (x1 , x2 , x3 ) = Span (x1 , x2 ) = Span (x1 , x3 ) = Span (x2 , x3 ) .
In other words, because of the dependence relation (4.3), the span of x1 , x2 , x3 can be
written as the span of only two of the given vectors. Or, put yet differently, we can throw
away one of the three vectors without changing their span. So the three vectors are not
the most economic way to express their span, because two of them suffice.
On the other hand, no dependency of the form (4.3) exists between x1 and x2 , so we
cannot further reduce the number of vectors to express Span (x1 , x2 , x3 ) = Span (x1 , x2 ).
This discussion motivates the following definitions:

Definition 4.25. The vectors v1 , . . . , vn in a vector space V are said to be linearly

dependent if there exist scalars c1 , . . . , cn , not all zero, such that

c1 v 1 + · · · + cn v n = 0 .

Example 4.26. The three vectors x1 , x2 , x3 defined in (4.1) are linearly dependent.

Definition 4.27. The vectors v1 , . . . , vn in a vector space V are said to be linearly

independent if they are not linearly dependent, that is, if

c1 v 1 + · · · + cn v n = 0 ,

forces all scalars c1 , . . . , cn to be 0.

   
2 1
Example 4.28. The vectors , ∈ R2 are linearly independent. In order to see
1 1
this, suppose that      
2 1 0
c1 + c2 = .
1 1 0
Then c1 and c2 must satisfy the 2 × 2 system

2c1 + c2 = 0
c1 + c2 = 0

However, as is easily seen, the only solution of this system is c1 = c2 = 0. Thus, the two
vectors are indeed linearly independent as claimed.

Example 4.29. Let p1 , p2 ∈ P1 be given by

p1 (t) = 2 + t , p2 (t) = 1 + t .

Then p1 and p2 are linearly independent. In order to see this, suppose that

c1 p1 + c2 p2 = 0 .

Then, for all t

c1 (2 + t) + c2 (1 + t) = 0 ,
so, for all t
(2c1 + c2 ) + (c1 + c2 )t = 0 .
4.5. LINEAR INDEPENDENCE TEST 49

Notice that the polynomial on the left-hand side of the above equation will be the zero
polynomial if and only if its coefficients vanish, so c1 and c2 must satisfy the 2 × 2 system

2c1 + c2 = 0
c1 + c2 = 0

However, as in the previous example, the only solution of this system is c1 = c2 = 0. Thus
p1 and p2 are indeed linearly independent as claimed.

The following result will become important later in this chapter, when we discuss
coordinate systems.

Theorem 4.30. Let v1 , . . . , vn be vectors in a vector space V . A vector v ∈ Span (v1 , . . . , vn )

can be written uniquely as a linear combination of v1 , . . . , vn if and only if v1 , . . . , vn are
linearly independent.

Proof. If v ∈ Span (v1 , . . . , vn ) then v can be written

v = α 1 v1 + · · · + α n vn , (4.4)

for some scalars α1 , . . . , αn . Suppose that v can also be written in the form

v = β1 v1 + · · · + βn vn , (4.5)

for some scalars β1 , . . . , βn . We start by showing that if v1 , . . . , vn are linearly indepen-

dent, then αi = βi for every i = 1, . . . , n (that is, the representation (4.4) is unique). To
see this, suppose that v1 , . . . , vn are linearly independent. Then subtracting (4.5) from
(4.4) gives
(α1 − β1 )v1 + · · · + (αn − βn )vn = 0 , (4.6)
which forces αi = βi for every i = 1, . . . , n as desired.
Conversely, if the representation (4.4) is not unique, then there must be a representa-
tion of the form (4.5) where αi 6= βi for some i between 1 and n. But then (4.6) means
that there exists a non-trivial linear dependence between v1 , . . . , vn , so these vectors are
linearly dependent.

4.5 Linear independence test

In order to show linear independence in an abstract vector space, we usually test the
condition from Definition 4.27. When working in a standard vector space of column
vectors, the general definition specialises to the following practical tests.
Linear independence test
To test whether a1 , . . . , an ∈ Rm are linearly independent:

• consider the matrix A ∈ Rm×n with columns a1 , . . . , an ;

• write augmented matrix (A|0) corresponding to the system Ax = 0;

• bring (A|0) to REF using Gauss-Jordan and identify leading and free variables:

– if there are no free variables, then the vectors are independent;

– if there are free variables, then the vectors are dependent.
50 CHAPTER 4. VECTOR SPACES
    
1 2 5
 2  0 −4
Example 4.31. Are vectors −3 , 4 ,  14  linearly
     independent?
1 1 −3
Using the linear independence test,
   
1 2 5 0 1 0 0 0
 2 0 −4 0  0 1 0 0
 −3 4 14 0  ∼ · · · ∼  0 0 1
   .
0
1 1 −3 0 0 0 0 0

We see that there are no free variables, so yes!

     
2 1 5
1 2 1
Example 4.32. Are vectors  2 , 2 , 4 linearly independent?
    

1 0 3
Using the linear independence test,
   
2 1 5 0 1 0 3 0
1 2 1 0  0 1 −1 0 
2 2 4 0 ∼ · · · ∼ 0 0 0 0 .
   

1 0 3 0 0 0 0 0

We see that x3 is a free variables, so no!

Theorem 4.33. Let x1 , . . . , xn be n vectors in Rn and let X ∈ Rn×n be the matrix whose
j-th column is xj . Then the vectors x1 , . . . , xn are linearly dependent if and only if X is
singular (i.e., its determinant is 0).

Proof. The equation

c1 x 1 + · · · + cn x n = 0
can be written as  
c1
 .. 
Xc = 0 , where c =  .  .
cn
This system has a non-trivial solution c 6= 0 if and only X is singular.

Example 4.34. Determine whether the following three vectors in R3 are linearly inde-
pendent:      
−1 5 4
 3  , 2 , 5 .
1 5 6

Solution. Since
−1 5 4 −1 3 1 R1 + R2 4 5 6 R1 − R3 0 0 0
3 2 5 = 5 2 5 = 5 2 5 = 5 2 5 = 0,
1 5 6 4 5 6 4 5 6 4 5 6

the vectors are linearly dependent.

4.6. BASIS AND DIMENSION 51

4.6 Basis and dimension

The concept of a basis and the related notion of dimension are among the key ideas in
the theory of vector spaces, of immense practical and theoretical importance. Let’s start
with the definition of a basis, delaying the discussion of its interpretation for a bit:

Definition 4.35. A set {v1 , . . . , vn } of vectors forms a basis for a vector space V if

(i) v1 , . . . , vn are linearly independent;

(ii) Span (v1 , . . . , vn ) = V .

In other words, a basis for a vector space is a ‘minimal’ spanning set, in the sense
that it contains no superfluous vectors: every vector in V can be written as a linear
combination of the basis vectors (because of property (ii)), and there is no redundancy in
the sense that no basis vector can be expressed as a linear combination of the other basis
vectors (by property (i)). Let’s look at some examples:

Example 4.36. Let

     
1 0 0
e1 = 0 ,
 e2 = 1 ,
 e3 = 0 .

0 0 1

Then {e1 , e2 , e3 } is a basis for R3 , called the standard basis.

Indeed, as is easily seen, every vector in R3 can be written as a linear combination of
e1 , e2 , e3 and, moreover, the vectors e1 , e2 , e3 are linearly independent.

Example 4.37.       
 1 1 1 
0 , 1 , 1
0 0 1
 

is a basis for R3 .
To see this note that the vectors are linearly independent, because

1 1 1
0 1 1 = 1 6= 0 .
0 0 1

Moreover, the vectors span R3 since, if (a, b, c)T is an arbitrary vector in R3 , then
       
a 1 1 1
 b  = (a − b) 0 + (b − c) 1 + c 1 .
c 0 0 1

The previous two examples show that a vector space may have more than one basis.
This is not a nuisance, but, quite to the contrary, a blessing, as we shall see later in
this module. For the moment, you should only note that both bases consist of exactly
three elements. We will revisit and expand this observation shortly, when we discuss the
dimension of a vector space.
52 CHAPTER 4. VECTOR SPACES

Example 4.38. Let

       
1 0 0 1 0 0 0 0
E11 = , E12 = , E21 = , E22 = .
0 0 0 0 1 0 0 1

Then {E11 , E12 , E21 , E22 } is a basis for R2×2 , because the four vectors span R2×2 and
they are linearly independent. To see this, suppose that

c1 E11 + c2 E12 + c3 E21 + c4 E22 = O2×2 .

Then    
c1 c2 0 0
= ,
c3 c4 0 0
so c1 = c2 = c3 = c4 = 0.

Most of the vector spaces we have encountered so far have particularly simple bases,
termed ‘standard bases’:

Example 4.39 (Standard bases for Rn , Rm×n and Pn ).

Rn : The n columns of In form the standard basis of Rn , usually denoted by {e1 , e2 , . . . , en }.
Rm×n : A canonical basis can be constructed as follows. For i = 1, . . . , m and j =
1, . . . , n let Eij ∈ Rm×n be the matrix whose (i, j)-entry is 1, and all other entries are 0.
Then { Eij | i = 1, . . . , m , j = 1, . . . , n } is the standard basis for Rm×n .
Pn : The standard basis is the collection {p0 , . . . , pn } of all monomials of degree less
or equal than n, that is,
pk (t) = tk , for k = 0, . . . , n.
If this is not clear to you, you should check that it really is a basis!

Going back to Examples 4.36 and 4.37, recall the observation that both bases of R3
contained exactly three elements. This is not pure coincidence, but has a deeper reason.
In fact, as we shall see shortly, any basis of a vector space must contain the same number
of vectors.

Theorem 4.40 (Steinitz). If u1 , . . . , um are linearly independent in a vector space V =

Span (v1 , . . . , vn ), then m ≤ n.

Proof. By mathematical induction on k ∈ {0, . . . , m}, we prove that k ≤ n, and, after a

possible reordering of the vi , the set

{u1 , . . . , uk , vk+1 , . . . , vn }

spans V .
Base case. For k = 0, there is nothing to prove.
Induction step. Suppose that the above claim holds for some k < m. Thus, we have
that
uk+1 ∈ V = Span (u1 , . . . , uk , vk+1 , . . . , vn ),
so we can write
k
X n
X
uk+1 = αi ui + α j vj ,
i=1 j=k+1

for some α1 , . . . , αn ∈ R.
4.6. BASIS AND DIMENSION 53

We note that at least one of αk+1 , . . . , αn must be nonzero, otherwise we would get that
uk+1 ∈ Span (u1 , . . . , uk ), which would contradict the linear independence of u1 , . . . , un .
By reordering the terms αk+1 vk+1 , . . . , αn vn , we may assume that αk+1 6= 0, and we can
write
k n
!
1 X X
vk+1 = uk+1 − αi ui + αj vj ∈ Span (u1 , . . . , uk+1 , vk+2 , . . . , vn ),
αk+1 i=1 j=k+2

and we deduce that

insert vk+1
Span (u1 , . . . , uk+1 , vk+2 , . . . , vn ) = Span (u1 , . . . , uk+1 , vk+1 , vk+2 , . . . , vn )
omit uk+1
⊇ Span (u1 , . . . , uk , vk+1 , vk+2 , . . . , vn ) = V,

so u1 , . . . , uk+1 , vk+2 , . . . , vn spans V , which is the claim for k + 1 and the induction is
complete.
We are now able to prove the observation alluded to earlier:
Corollary 4.41. If a vector space V has a basis of n vectors, then every basis of V must
have exactly n vectors.
Proof. Suppose that {v1 , . . . , vn } and {u1 , . . . , um } are both bases for V . We shall show
that m = n. In order to see this, notice that, since Span (v1 , . . . , vn ) = V and u1 , . . . , um
are linearly independent it follows by the previous theorem that m ≤ n. By the same
reasoning, since Span (u1 , . . . , um ) = V and v1 , . . . , vn are linearly independent, we must
have n ≤ m. So, all in all, we have n = m, that is, the two bases have the same number
of elements.
In view of this corollary it now makes sense to talk about the number of elements of
a basis, and give it a special name:
Definition 4.42. Let V be a vector space. If V has a basis consisting of n vectors, we
say that V has dimension n, and write dim V = n.
The vector space {0} is said to have dimension 0.
Definition 4.43. A vector space V is said to be finite dimensional if there is a finite
set of vectors spanning V ; otherwise it is said to be infinite dimensional .
The following theorem and corollary justify the definition of ‘finite dimensional’, show-
ing that the dimension of a finite dimensional vector space is indeed a natural number.
Theorem 4.44. If S is a finite spanning set for a vector space V , then S contains a basis
for V .
Proof. Let S = {v1 , . . . , vn } be a spanning set for V . If S is already linearly independent,
then S is already a basis.
Otherwise, S is linearly dependent, so there exists a vector, say, vk ∈ S which is a
linear combination of other vectors in S. Let S1 = S \ {vk }. By Lemma ??, Span (S1 ) =
Span (S) = V , and we can repeat the reasoning from the start of the proof with S1 in
place of S.
We obtain a sequence S ) S1 ) S2 ) · · · of spanning sets which terminates in finitely
many steps (at most n − 1), and the last spanning set must also be linearly independent,
hence a basis.
54 CHAPTER 4. VECTOR SPACES

Corollary 4.45. Any finite dimensional vector space has a (finite) basis, hence its di-
mension is a natural number.
Remark 4.46. In fact, any vector space has a basis, but the proof is beyond the scope
of this module; it uses some sophisticated techniques of Set Theory, such as the Zorn’s
Lemma.
Example 4.47. By Example 4.39 the vector spaces Rn , Rm×n and Pn are finite dimen-
sional with dimensions

dim Rn = n , dim Rm×n = mn , dim Pn = n + 1 .

Example 4.48. As an example of an infinite dimensional (i.e., not finite dimensional)

vector space, consider the set of all polynomials with real coefficients, and call it P .
In order to see that P is a vector space when equipped with the usual addition and
scalar multiplication, notice that P is a subset of the vector space C[−1, 1] of continuous
functions on [−1, 1] (in fact, it is a subset of C[a, b] for any a, b ∈ R with a < b), which is
closed under addition and scalar multiplication. Thus P is a vector space. Note that any
finite collection of monomials is linearly independent, so P must be infinite dimensional.
For the same reason, C[a, b] and C 1 [a, b] are infinite dimensional vector spaces. While
infinite dimensional vector spaces play an important role in many parts of contemporary
applied and pure mathematics, we shall be mainly concerned with finite dimensional
vector spaces for the rest of this module.
Example 4.49. Geometric interpretation of subspaces of R3 :
• 0-dimensional subspaces. Only the zero subspace {0}.

• 1-dimensional subspaces. Any subspace spanned by a nonzero vector, that is all

lines through the origin.

• 2-dimensional subspaces. Any subspace spanned by two linearly independent vec-

tors, that is all planes through the origin.

• 3-dimensional subspaces. Only R3 .

We close this section with the following result, which is often useful when trying to
decide whether a collection of vectors forms a basis of a vector space:
Theorem 4.50. If V is a vector space with dim V = n, then:
(a) any set consisting of n linearly independent vectors spans V ;

(b) any n vectors that span V are linearly independent.

Proof. (a) Let v1 , . . . , vn be a linearly independent. Pick v ∈ V . Since dim V = n, the
n + 1 vectors v, v1 , . . . , vn must be linearly dependent by Theorem 4.40. Thus

c0 v + c1 v 1 + · · · + cn v n = 0 , (4.7)

where c0 , c1 , . . . , cn are not all 0. But c0 6= 0 (for otherwise (4.7) would imply that the
vectors v1 , . . . , vn are linearly dependent), hence
   
c1 cn
v= − v1 + · · · + − vn ,
c0 c0
4.7. COORDINATES 55

so v ∈ Span (v1 , . . . , vn ). But v was arbitrary, so Span (v1 , . . . , vn ) = V .

(b) Suppose Span (v1 , . . . , vn ) = V . In order to show that v1 , . . . , vn are linearly inde-
pendent we argue by contradiction: suppose to the contrary that v1 , . . . , vn are linearly
dependent. Then one of the vi ’s, say vn can be written as a linear combination of the
other vectors. So v1 , . . . , vn−1 also span V . If v1 , . . . , vn−1 are still linearly dependent we
can eliminate another vector and still have a spanning set. We can continue this process
until we have found a spanning set with k linearly independent vectors where k < n.
This, however, contradicts the fact that dim V = n.
Remark 4.51. The above theorem provides a convenient tool to check whether a set
of vectors forms a basis. The theorem tells us that n linearly independent vectors in an
n-dimensional vector space are automatically spanning, so these vectors are a basis for
the vector space. This is often useful in situations where linear independence is easier to
check than the spanning property.
Remark 4.52. The above theorem also provides two perspectives on a basis of a vector
space:
(
a spanning set that is as small as possible;
a basis is
a linearly independent collection of vectors that is as large as possible.
So, for example:
                   
 1 2   1 2 4   1 2 4 7 
0 , 3 , 0 , 3 , 5 , 0 , 3 , 5 , 8 .
0 0 0 0 6 0 0 6 9
     
linearly independent, basis for R3 spans R3 ,
but doesn’t span R3 but not linearly independent

The following result completes our picture of the relationship between the concepts of
linearly independent sets, spanning sets and bases.
Theorem 4.53. In a finite dimensional vector space V , any linearly independent set can
be completed to a basis.
Proof. Suppose that {u1 , . . . , un } is a linearly independent set in V . Using 4.44, take
an arbitrary basis {v1 , . . . , vm } for V . In particular, v1 , . . . , vm span V , so the proof of
Theorem 4.40 shows that, up to reordering of the vj , we may arrange so that
{u1 , . . . , un , vn+1 , . . . , vm }
is a spanning set. By 4.50, this is also a basis.

4.7 Coordinates
In this short section we shall discuss an important application of the notion of a basis.
In essence, a basis allows us to view a vector space of dimension n as if it were Rn . This
is a tremendously useful idea, with many practical and theoretical applications, many of
which you will see in the following chapters.
The basic idea is the following. Suppose that {b1 , . . . , bn } is a basis for a vector space
V . Since the basis vectors are spanning, given v ∈ V , there are scalars c1 , . . . , cn such
that
v = c1 b1 + · · · + cn bn .
56 CHAPTER 4. VECTOR SPACES

Moreover, since the basis vectors b1 , . . . , bn are linearly independent, the scalars c1 , . . . , cn
are uniquely determined by Theorem 4.30. Thus, the vector v in the vector space V , can
be uniquely represented as an n-vector (c1 , . . . , cn )T in Rn . This motivates the following
definition:

Definition 4.54. Suppose that B = {b1 , . . . , bn } is a basis for a vector space V . If v ∈ V

then the uniquely determined scalars c1 , . . . , cn such that

v = c1 b1 + · · · + cn bn ,

are called the coordinates of v relative to B . The n-vector (c1 , . . . , cn )T ∈ Rn is called

the B-coordinate vector of v, or the coordinate vector of v relative to B, and is
denoted by [v]B .

Example 4.55. Consider the basis B = {b1 , b2 } for R2 , where

   
1 1
b1 = , b2 = .
0 2

Suppose that x ∈ R2 has B-coordinate vector [x]B = (−2, 3)T . Find x.

Solution.      
1 1 1
x = −2b1 + 3b2 = (−2) +3 = .
0 2 6

 
1
Example 4.56. The entries of x = are the coordinates of x relative to the standard
6
basis E = {e1 , e2 }, since
     
1 1 0
=1 +6 = 1e1 + 6e2 .
6 0 1

Thus, x = [x]E .

Theorem 4.57. Let B = {b1 , . . . , bn } be a basis for Rn . There is an invertible n × n

matrix PB such that for any x ∈ Rn

x = PB [x]B .

In fact, the matrix PB is the matrix whose j-th column is bj .

Proof. Let [x]B = (c1 , . . . , cn )T . Then

x = c1 b1 + · · · + cn bn ,

so  
c1
 .. 
x = (b1 · · · bn )  .  .
| {z }
=PB cn
Moreover, by Theorem 4.33, the matrix PB is invertible since its columns are linearly
independent.
4.7. COORDINATES 57

Since a vector x ∈ Rn is equal to its coordinate vector relative to the standard basis,
the matrix PB given in the theorem above is called the transition matrix from B to
the standard basis.

Corollary 4.58. Let B = {b1 , . . . , bn } be a basis for Rn . For any x ∈ Rn

[x]B = PB−1 x .
     
2 −1 4
Example 4.59. Let b1 = , b2 = and x = . Let B = {b1 , b2 } be the
1 1 5
corresponding basis for R2 . Find the B-coordinates of x.

Solution. By the previous corollary

[x]B = PB−1 x .

Now  
2 −1
PB = ,
1 1
so  
1 1 1
PB−1 = .
3 −1 2
Thus     
1 1 1 4 3
[x]B = = .
3 −1 2 5 2

Theorem 4.60. Let B and D be two bases for Rn . If x is any vector in Rn , then

[x]B = PB−1 PD [x]D .

Proof. Clear, since PB is invertible and

PB [x]B = x = PD [x]D .

The n × n matrix PB−1 PD given in the theorem above is called the transition matrix
from D to B.

Example 4.61. Let B = {b1 , b2 } be the basis given in Example 4.59, let D = {d1 , d2 },
where    
1 1
d1 = , d2 = ,
0 2
and let x ∈ R2 . If the D-coordinates of x are (−3, 2)T , what are the B-coordinates of x?

Solution.      
1 1 1 1 1 −3 1
[x]B = PB−1 PD [x]D = = .
3 −1 2 0 2 2 3
58 CHAPTER 4. VECTOR SPACES

4.8 Row space and column space

In this final section of this rather long chapter on vector spaces, we shall briefly discuss
a number of naturally arising vector spaces associated with matrices. We have already
encountered one such space, the nullspace of a matrix.
Definition 4.62. Let A ∈ Rm×n .
• The subspace of R1×n spanned by the row vectors of A is called the row space of
A and is denoted by row(A).
• The subspace of Rm×1 spanned by the column vectors of A is called the column
space of A and is denoted by col(A).
 
1 0 0
Example 4.63. Let A = .
0 1 0
• Since



α 1 0 0 +β 0 1 0 = α β 0
row(A) is a 2-dimensional subspace of R1×3 .
• Since        
1 0 0 α
α +β +γ =
0 1 0 β
col(A) is a 2-dimensional subspace of R2×1 .
Notice that the row space and column space of a matrix are generally distinct objects.
Indeed, one is a subspace of R1×n the other a subspace of Rm×1 . However, in the example
above, both spaces have the same dimension (namely 2). We shall see shortly, that, rather
surprisingly, this is always the case. Before exploring this topic further we introduce the
following important concept:
Definition 4.64. The rank of a matrix, denoted by rank A, is the dimension of the row
space.
How does one calculate the rank of a matrix? The next result provides the clue:
Theorem 4.65. Two row equivalent matrices have the same row space, so, in particular,
have the same rank.
Proof. Let A and B be two row equivalent matrices. Since B is row equivalent to A,
the matrix B can be obtained from A by a finite sequence of elementary row operations.
Thus the rows of B are a linear combination of the rows of A. Consequently, row(B) is a
subspace of row(A). Exchanging the roles of A and B it follows, using the same argument,
that row(A) is also a subspace of row(B), so row(A) = row(B).
Combining the previous theorem with the observation that the nonzero rows of a
matrix in row echelon form are linearly independent, we obtain the following recipe for
calculating a basis for the row space and the rank of a matrix:

In order to calculate a basis for the row space and the rank of a matrix A:

• bring matrix to row echelon form U ;

• the nonzero rows of U will form a basis for row(A);

• the number of nonzero rows of U equals rank A.

4.8. ROW SPACE AND COLUMN SPACE 59

Example 4.66. Let  

1 −3 2
A = 1 −2 1 .
2 −5 3
Then      
1 −3 2 1 −3 2 1 −3 2
1 −2 1 ∼ 0 1 −1 ∼ 0 1 −1 .
2 −5 3 0 1 −1 0 0 0
Thus 


1 −3 2 , 0 1 −1
is a basis for row(A), and rank A = 2.
It turns out that the rank of a matrix A is intimately connected with the dimension of
its nullspace N (A). Before formulating this relation, we require some more terminology:
Definition 4.67. If A ∈ Rm×n , then dim N (A) is called the nullity of A, and is denoted
by nul A.
Example 4.68. Find the nullity of the matrix
 
−3 6 −1 1 −7
A= 1 −2 2 3 −1 .
2 −4 5 8 −4

Solution. We have already calculated the nullspace N (A) of this matrix in Example 4.16
by bringing A to row echelon form U and then using back substitution to solve U x = 0,
giving
N (A) = { αx1 + βx2 + γx3 | α, β, γ ∈ R } ,
where      
2 1 −3
1 0 0
     
0 ,
x1 =  −2 ,
x2 =  2.
x3 = 
  
0 1 0
0 0 1
It is not difficult to see that x1 , x2 , x3 are linearly independent, so {x1 , x2 , x3 } is a basis
for N (A). Thus, nul A = 3.
Notice that in the above example the nullity of A is equal to the number of free
variables of the system Ax = 0. This is no coincidence, but true in general.
The connection between the rank and nullity of a matrix, alluded to above, is the
content of the following beautiful theorem with an ugly name:
Theorem 4.69 (Rank-Nullity Theorem). If A ∈ Rm×n , then

rank A + nul A = n .

Proof. Bring A to row echelon form U . Write r = rank A. Now observe that U has r
non-zero rows, hence U x = 0 has n − r free variables, so nul A = n − r.
We now return to the perhaps rather surprising connection between the dimensions of
the row space and the column space of a matrix.
60 CHAPTER 4. VECTOR SPACES

Theorem 4.70. Let A ∈ Rm×n . Then

dim col(A) = dim row(A) .

Proof. Ommitted here.

How to find a basis for the column space of a matrix:

In order to find a basis for the column space of a matrix A:

• bring A to row echelon form and identify the leading variables;

• the columns of A containing the leading variables form a basis for col(A).

Example 4.71. Let  

1 −1 3 2 1
A = 1 0 1 4 1 .
2 −1 4 7 4
Then the row echelon form of A is
 
1 −1 3 2 1
0 1 −2 2 0 .
0 0 0 1 2

The leading variables are in columns 1,2, and 4. Thus a basis for col(A) is given by
      
 1 −1 2 
1 ,  0  , 4 .
2 −1 7
 
Chapter 5

Linear Transformations

Some linear transformations have already been introduced in Geometry I, but the concept
is much more general and can be extended to general vector spaces. In fact, every linear
transformation between finite-dimensional vector spaces can be viewed as a matrix: there
is a matrix representation of a given linear transformation. But we won’t go into much
detail on this topic. Roughly speaking a linear transformation is a mapping between two
vector spaces that preserves the linear structure of the underlying spaces.

5.1 Definition and examples

Definition 5.1. Let V and W be two vector spaces. A mapping L : V → W (that is, a
mapping from V to W ) is said to be a linear transformation or a linear mapping if
it satisfies the following two conditions:

(i) L(v + w) = L(v) + L(w) for all v and w in V ;

(ii) L(αv) = αL(v) for all v in V and all scalars α.

Example 5.2. Let V be a vector space and let id : V → V denote the identity trans-
formation (or identity for short) on V , that is,

id(v) = v for all v ∈ V .

The transformation Id is linear, since, if v, w ∈ V and α is a scalar, then

(i) id(v + w) = v + w = id(v) + id(w);

(ii) id(αv) = αv = αid(v).

Example 5.3. Let L : R2 → R2 be defined by

L(x) = 2x .

Then L is linear since, if x and y are arbitrary vectors in R2 and α is an arbitrary real
number, then

(i) L(x + y) = 2(x + y) = 2x + 2y = L(x) + L(y);

(ii) L(αx) = 2(αx) = α(2x) = αL(x).

61
62 CHAPTER 5. LINEAR TRANSFORMATIONS

Example 5.4. Let L : R2 → R2 be defined by

 
x1
L(x) = x1 e1 , where x= .
x2

Then L is linear. In order to see this suppose that x and y are arbitrary vectors in R2
with    
x1 y
x= , y= 1 .
x2 y2
Notice that, if α is an arbitrary real number, then
   
x1 + y1 αx1
x+y = and αx = .
x2 + y2 αx2

Thus

(i) L(x + y) = (x1 + y1 )e1 = x1 e1 + y1 e1 = L(x) + L(y);

(ii) L(αx) = (αx1 )e1 = α(x1 e1 ) = αL(x).

Hence L is linear, as claimed.

In order to shorten statements of theorems and examples let us introduce the following
convention:

If x is a vector in Rn , we shall henceforth denote its i-th entry by xi , and similarly for
vectors in Rn denoted by other bold symbols. So, for example, if y = (1, 4, 2, 7)T ∈ R4 ,
then y3 = 2.

Example 5.5. Let L : R2 → R2 be given by

 
−x2
L(x) = .
x1

L is linear, since, if x, y ∈ R2 and α ∈ R, then

     
−(x2 + y2 ) −x2 −y2
(i) L(x + y) = = + = L(x) + L(y);
x1 + y 1 x1 y1
   
−αx2 −x2
(ii) L(αx) = =α = αL(x).
αx1 x1

Example 5.6. The mapping M : R2 → R1 defined by

q
M (x) = x21 + x22
√
is not linear. Inporder to see this note that M ((1, 0)T ) = 12 = 1 while M (−(1, 0)T ) =
M ((−1, 0)T ) = (−1)2 = 1. Thus

M (−(1, 0)T ) = 1 6= −1 = −M ((1, 0)T ) .

5.2. BASIC PROPERTIES OF LINEAR TRANSFORMATIONS 63

5.2 Basic properties of linear transformations

Theorem 5.7. If V and W are vector spaces and L : V → W is a linear transformation
then:
(a) L(0) = 0;

(b) L(−v) = −L(v) for any v ∈ V ;

(c) L( ni=1 αi vi ) = ni=1 αi L(vi ) for any vi ∈ V and any scalars αi where i = 1, . . . , n.
P P

Proof.

(a) L(0) = L(00) = 0L(0) = 0;

(b) L(−v) = L((−1)v) = (−1)L(v) = −L(v);

(c) follows by repeated application of the defining properties (i) and (ii) of linear trans-
formations.
Note that we have used Theorem 4.5 for the proof of (a) and (b).

5.3 Linear transformations between standard vector

spaces
In this section we describe how linear transformations from Rn to Rm are directly linked
to m × n matrices.
Remark 5.8. Any m × n matrix A induces a linear transformation LA : Rn → Rm given
by
LA (x) = Ax for each x ∈ Rn .
The transformation LA is linear, since, if x, y ∈ Rn and α ∈ R, then
(i) LA (x + y) = A(x + y) = Ax + Ay = LA (x) + LA (y);

(ii) LA (αx) = A(αx) = αAx = αLA (x).

In other words, every m × n matrix gives rise to a linear transformation from Rn to
Rm . Conversely, every linear transformation from Rn to Rm arises from an m × n matrix
as follows.
Theorem 5.9. If L : Rn → Rm is a linear transformation, then there exists a matrix
A ∈ Rm×n such that L = LA , i.e., for all x ∈ Rn ,

L(x) = Ax.

Proof. Let (e1 , . . . , en ) be the standard basis for Rn , and let A ∈ Rm×n be the matrix
with columns L(e1 ), . . . , L(en ). Then
Xn n
X
L(x) = L( xi e i ) = xi L(ei ) = Ax,
i=1 i=1

where the last equality follows by definition of matrix multiplication.

64 CHAPTER 5. LINEAR TRANSFORMATIONS

Example 5.10. Let L : R3 → R2 be given by the formula

 
x1  
x 1 − 3x 2 + 5x 3
L  x2  = .
−7x2 + 2x3
x3

Writing (e1 , e2 , e3 ) for the standard basis for R3 , we evaluate

     
1 −3 5
L(e1 ) = , L(e2 ) = , L(e3 ) = ,
0 −7 2

and the matrix associated to L is formed from these columns:

 
1 −3 5
.
0 −7 2

5.4 Linear transformations on general vector spaces

So far we have only considered linear transformations from Rn to Rm . In this short section,
we shall have a look at some examples of linear transformations on abstract vector spaces,
which should convince you that linear transformations arise naturally in other areas of
Mathematics.

Example 5.11. Let L : C[a, b] → R1 be defined by

Z b
L(f ) = f (t) dt .
a

L is linear since, if f , g ∈ C[a, b] and α ∈ R, then

Z b Z b Z b
(i) L(f + g) = (f (t) + g(t)) dt = f (t) dt + g(t) dt = L(f ) + L(g);
a a a
Z b Z b
(ii) L(αf ) = (αf (t)) dt = α f (t) dt = αL(f ).
a a

In other words, integration is a linear transformation.

Example 5.12. Let D : C 1 [a, b] → C[a, b] be defined to be the transformation that sends
an f ∈ C 1 [a, b] to its derivative f 0 ∈ C[a, b], that is,

D(f ) = f 0 .

Then D is linear since, if f , g ∈ C 1 [a, b] and α ∈ R, then

(i) D(f + g) = (f + g)0 = f 0 + g0 = D(f ) + D(g);

(ii) D(αf ) = (αf )0 = αf 0 = αD(f ).

In other words, differentiation is a linear transformation.

Example 5.13. Let Pn denote the vector space of polynomials of degree at most n.
5.5. DEFINING LINEAR MAPS 65

• We have a derivation operator

D : Pn → Pn−1 , D(p) = p0 .
More explicitly,
n
X n
X
i
D( αi t ) = i αi ti−1 .
i=0 i=1

• We have an integration operator

Z t
I : Pn → Pn+1 , (I(p))(t) = p(x)dx.
0

More explicitly,
n n
X
i
X αi i+1
I( αi t ) = t .
i=0 i=0
i+1

Both D and I are linear transformations, and we invite the reader to check this, either
using the rules for differentiation and integration, or directly, using the explicit formulae.
Example 5.14. Let S : C(R) → C(R) be the ‘shift operator’, defined by
(S(f ))(x) = f (x + 1), for x ∈ R.
Then S is linear.
On the other hand, the map T : C(R) → C(R) given by
(T (f ))(x) = f (x) + 1, for x ∈ R
is not linear because T (0) 6= 0.

5.5 Defining linear maps

A linear operator is uniquely determined by its action on a basis of a vector space. More
precisely, we have the following.
Proposition 5.15. Let (v1 , . . . , vn ) be basis for a vector space V , and let (w1 , . . . , wn )
be an n-tuple of vectors from a vector space W . Then there exists a unique linear map
L:V →W
such that L(vi ) = wi for i = 1, . . . , n.
Proof. Any v ∈ V can be uniquely written as a linear combination of v1 , . . . , vn , so we
can define a function L : V → W by the rule
n
X n
X
L( αi vi ) = αi wi ∈ W.
i=1 i=1

We invite the reader to check that L is in fact a linear map, which clearly has the required
property, so we proved existence.
To see that it is unique, if M : V → W is linear with M (vi ) = wi , then
n
X n
X n
X Xn
M( αi vi ) = αi M (vi ) = αi wi = L( αi vi ),
i=1 i=1 i=1 i=1

for all choices of αi , so we conclude that M = L.

66 CHAPTER 5. LINEAR TRANSFORMATIONS

5.6 Forming new linear maps

Definition 5.16. Let L, L0 : U → V , M : V → W be linear maps, and let α ∈ R. We
define maps

1. L + L0 : U → V , given by (L + L0 )(u) = L(u) + L0 (u), for u ∈ U ;

2. αL : U → V , given by (αL)(u) = αL(u), for u ∈ U ;

3. M ◦ L : U → W , as the composite of M and L, given by (M ◦ L)(u) = M (L(u))

for u ∈ U .

Lemma 5.17. With notation from the above definition, L + L0 , αL and M ◦ L are linear
maps.

Proof. To see that L + L is linear, we proceed as follows.

1. L + L0 is additive since, for all u, v ∈ U ,

(L + L0 )(u + v) = L(u + v) + L0 (u + v) = (L and L0 are linear)

= L(u) + L(v) + L0 (u) + L0 (v) = (L(u) + L0 (u)) + (L(v) + L0 (v))
= (L + L0 )(u) + (L + L0 )(v).

2. L + L0 is homogeneous since for all u ∈ U , λ ∈ R,

(L + L0 )(λu) = L(λu) + L0 (λu) = (L and L0 are linear)

= λL(u) + λL0 (u) = λ(L(u) + L0 (u)) = λ(L + L0 )(u).

Showing that αL is linear is similar and we leave it as an exercise for the reader.
Let us verify that M ◦ L is linear.

1. M ◦ L is additive since, for all u, v ∈ U ,

(M ◦ L)(u + v) = M (L(u + v)) = (L is linear) = M (L(u) + L(v)) = (M is linear)

= M (L(u)) + M (L(v)) = (M ◦ L)(u) + (M ◦ L)(v).

2. M ◦ L is homogeneous since, for all u ∈ U and λ ∈ R,

(M ◦ L)(λu) = M (L(λu)) = (L is linear) = M (λL(u)) = (M is linear)

= λM (L(u)) = λ(M ◦ L)(u).

Example 5.18. Let A, A0 ∈ Rm×n , B ∈ Rn×p and α ∈ R. Then, using the notation for
linear operators associated with matrices from 5.8, we have that

1. LA + LA0 = LA+A0 ;

2. αLA = LαA ;

3. LA ◦ LB = LAB .
5.7. ISOMORPHISMS 67

5.7 Isomorphisms
Definition 5.19. A linear map L : V → W is an isomorphism, if there exists a linear
map M : W → V such that

M ◦ L = idV and L ◦ M = idW ,

i.e., if L has a linear inverse.

Definition 5.20. We say that vector spaces V and W are isomorphic, and write

V ∼
= W,

if there exists an isomorphism L : V → W .

Remark 5.21. For students familiar with equivalence relations, it can be shown that ∼ =
is an equivalence relation between vector spaces.
If two vector spaces are isomorphic, we think of them as being essentially the same,
because an isomorphism between them gives us a way of identifying their elements in
a way compatible with the vector space structure. In the rest of this section, we shall
illustrate this point of view further.

Lemma 5.22. A linear map L : V → W is an isomorphism if and only if it is bijective.

Proof. If L is an isomorphism, then L has a linear inverse, so in particular it has an

inverse as a function and thus it is bijective.
Conversely, suppose that L is a bijective linear map. Since L is in particular a bijective
function, it has an inverse function M : W → V . We need to show that M is linear.
Indeed, for any w, w0 ∈ W ,

L(M (w + w0 )) = w + w0 = L(M (w)) + L(M (w0 )) = (L is linear) = L(M (w) + M (w0 )),

and, since L is injective, we conclude that M (w + w0 ) = M (w) + M (w0 ). Using a similar

trick, we can show that for all α ∈ R and w ∈ W , we have M (αw) = αM (w), and we
are done.

Proposition 5.23. Let L : V → W be an isomorphism, and let S be a subset of V . Then

1. S is linearly independent in V if and only if L(S) is linearly independent in W ;

2. S spans V if and only if L(S) spans W ;

3. S is a basis for V if and only if L(S) is a basis for W .

Proof. Statements 1. and 2. are an easy exercise using the fact that L is a bijective linear
map, and 3. follows from 1. and 2.

Corollary 5.24. Let V and W be vector spaces. Then V ∼

= W if and only if dim(V ) =
dim(W ).

Proof. If L : V → W is an isomorphism, then Proposition 5.23 shows that L takes a basis

V to a basis of W , so dim(V ) = dim(W ).
Conversely, if V and W have the same dimension n, say, let (v1 , . . . , vn ) be a basis
for V , and let (w1 , . . . , wn ) be a basis for W . By Proposition 5.15, there exist unique
68 CHAPTER 5. LINEAR TRANSFORMATIONS

linear maps L : V → W and M : W → V such that L(vi ) = wj and M (wi ) = vi for

i = 1, . . . , n. Thus, for all i,

(M ◦ L)(vi ) = vi = idV (vi ), and (L ◦ M )(wi ) = wi = idW (wi ).

Using the uniqueness from 5.15 again, we get that M ◦ L = idV and L ◦ M = idW , so we
conclude that L is an isomorphism.

Corollary 5.25. If dim(V ) = n, then V ∼

= Rn .

Hence, any finite dimensional vector space can be identified with a standard vector
space of column vectors, which is our preferred ambient where all the calculations become
explicit.

Remark 5.26. Let B = (b1 , . . . , bn ) be a basis for a vector space V . The coordinatisation
map
[ ]B : V → Rn
that results from Definition 4.54 is an isomorphism, given that is is bijective and linear,
i.e., for all u, v ∈ V and α ∈ R, we have

1. [u + v]B = [u]B + [v]B ;

2. [αu]B = α[u]B .

This is a directly constructed isomorphism between V and Rn , and we can think of it as

an effective version of the previous corollary.

Example 5.27. By the above corollary, since dim(Rm×n ) = mn = Rmn , and dim(Pn ) =
n + 1 = dim(Rn+1 ), we have isomorphisms of vector spaces

Rm×n ∼
= Rmn , and Pn ∼
= Rn+1 .

Example 5.28. Let B = (E11 , E12 , E21 , E22 ) be the standard basis for R2×2 . The coor-
dinatisation map with respect to B is the map
 
  a
a b  b
[ ]B : R2×2 → R4 , = .
c d B c
d

We invite the reader to verify that this is an isomorphism.

Example 5.29. Let B = (1, t, t2 , t3 ) be the standard basis for P3 . The coordinatisation
map with respect to B is the map
 
d
 c
[ ]B : P3 → R4 , [at3 + bt2 + ct + d]B = b .


The reader should verify that this is an isomorphism for exercise.

5.8. IMAGE AND KERNEL 69

5.8 Image and Kernel

We shall now discuss two useful notions, namely that of the ‘image’ and that of the
‘kernel’ of a linear transformation, that help to generalise two notions that you have
already encountered in connection with matrices.
Definition 5.30. Let V and W be vector spaces, let L : V → W be a linear transforma-
tion, let H be a subspace of V and K a subspace of W .
1. The preimage of K under L is the subset of V defined as

L−1 (K) = {v ∈ V | L(v) ∈ K}.

2. The kernel of L, denoted by ker(L), is the subset of V defined as the preimage of

the zero subspace {0} under L. More explicitly,

ker(L) = L−1 ({0}) = { v ∈ V | L(v) = 0 } .

3. The image of H under L is the subset of W defined as

L(H) = { w ∈ W | w = L(v) for some v ∈ H }

= {L(v) | v ∈ H}.

4. The image of L is the image of the entire vector space V under L,

im(L) = L(V ) = { w ∈ W | w = L(v) for some v ∈ V }

= {L(v) | v ∈ V }.

Example 5.31. If A ∈ Rm×n and LA is the corresponding linear transformation from Rn

to Rm . Then
ker(LA ) = N (A) ,
that is, the kernel of LA is the nullspace of A. Moreover,

im(LA ) = LA (Rn ) = col(A) ,

that is, the image of LA is the column space of A.

The previous example shows that the kernel of a linear transformation is the natural
generalisation of the nullspace of a matrix and that that the image of a linear transfor-
mation is the natural generalisation of the column space of a matrix.
We saw previously that the nullspace and the column space of an m × n matrix
are subspaces of Rn and Rm respectively. The same is true for the abstract analogues
introduced above.
Theorem 5.32. Let V and W and be vector spaces, L : V → W a linear transformation,
H is a subspace of V and K a subspace of W . Then
1. L−1 (K) is a subspace of V ;

2. ker(L) is a subspace of V ;

3. L(H) is a subspace of W ;
70 CHAPTER 5. LINEAR TRANSFORMATIONS

4. im(L) is a subspace of W .
Proof. Let us verify that L−1 (K) is a subspace of V , and we leave the rest as exercise.
To see that L−1 (K) is closed under addition, take u, v ∈ L−1 (K). That means that
L(u) ∈ K and L(v) ∈ K. Since K is a subspace, we have that L(u) + L(v) ∈ K, and,
since L is linear, we obtain that L(u + v) ∈ K, i.e., that u + v ∈ L−1 (K).
To see that L−1 (K) is closed under scalar multiplication, take u ∈ L−1 (K) and α ∈ R.
Then L(u) ∈ K, so, since K is a subspace, we have that αL(u) ∈ K. Since L is linear,
this entails that L(αu) ∈ K, i.e., that αu ∈ L−1 (K).
Definition 5.33. Let L : V → W be a linear map.
1. The rank of L is the number

rank (L) = dim(im(L)).

2. The nullity of L is the number

nul (L) = dim(ker(L)).

Theorem 5.34 (Rank-Nullity Theorem for Linear Transformations). Let L : V → W be

a linear transformation. Then

rank (L) + nul (L) = dim(V ).

Proof. Let
(u1 , . . . , ud ) be a basis for ker(L). (*)
Since u1 , . . . , ud are linearly independent in V , by 4.53, we can find v1 , . . . , vr ∈ V so
that
(u1 , . . . , ud , v1 , . . . , vr ) is a basis for V. (**)
Since V = Span (u1 , . . . , ud , v1 , . . . , vr ), we have that

im(L) = L(V ) = Span (L(u1 ), . . . , L(ud ), L(v1 ), . . . , L(vr ))

= Span (0, . . . , 0, L(v1 ), . . . , L(vr )) = Span (L(v1 ), . . . , L(vr )).

We claim that L(v1 ), . . . , L(vr ) are linearly independent, hence

(L(v1 ), . . . , L(vr )) is a basis for im(L). (***)

Indeed, suppose that α1 L(v1 ) + · · · + αr L(vr ) = 0. Since L is linear, this means that
L(α1 v1 + · · · + αr vr ) = 0, so α1 v1 + · · · + αr vr ∈ ker(L) = Span (u1 , . . . , ud ). Hence, there
exist β1 , . . . , βd ∈ R such that α1 v1 + · · · + αr vr = β1 u1 + · · · + βd ud , i.e.,

α1 v1 + · · · + αr vr + (−β1 )u1 + · · · + (−βd )ud = 0.

By (**), u1 , . . . , ud , v1 , . . . , vr are linearly independent, so it follows that α1 = · · · = αr =

0 (and β1 = · · · = βd = 0), so we have shown linear independence.
From (*), (***) and (**), we have

dim(ker(L)) = d, dim(im(L)) = r and dim(V ) = d + r,

as required.
5.8. IMAGE AND KERNEL 71

Remark 5.35. If L : Rn → Rm is linear, by 5.9, there is a matrix A ∈ Rm×n so that

L = LA . As observed in Example 5.31, ker(L) = N (A) and im(L) = col(A).
Hence, using the Rank-Nullity Theorem for matrices, we obtain

rank (L) + nul (L) = dim(im(L)) + dim(ker(L))

= dim(col(A)) + dim(N (A)) = rank (A) + nul (A) = n = dim(Rn ),

so the Rank-Nullity Theorem for linear maps between standard vector spaces easily follows
from the Rank-Nullity Theorem for their associated matrices.

Example 5.36. Let D : P3 → P2 be the derivation operator, given by D(p) = p0 . Then

ker(D) = {p ∈ P3 : p0 = 0} = (the set of constant polynomials) = Span (1),

so the basis for ker(D) is (1) and nul (D) = dim(ker(D)) = 1.

If we choose a basis, say, (1, t, t2 , t3 ) of P3 , then im(D) is spanned by the images by D
of the basis vectors, i.e.,

im(D) = Span (D(1), D(t), D(t2 ), D(t3 )) = Span (0, 1, 2t, 3t2 ) = Span (1, t, t2 ) = P2 ,

considered as a subspace of P3 . The basis for im(D) is therefore (1, t, t2 ) and rank (D) =
dim(im(D)) = 3.
Hence, we computed that

rank (D) + nul (D) = 3 + 1 = 4 = dim(P3 ),

in accordance with the Rank-Nullity Theorem.

Example 5.37. Let L : R2×2 → R2×2 be the linear operator L(A) = A − AT .

Then

ker(L) = {A ∈ R2×2 : L(A) = O} = {A ∈ R2×2 : A = AT }

        
a b 1 0 0 1 0 0
= : a, b, c ∈ R = Span , , ,
b c 0 0 1 0 0 1

the set to symmetric 2 × 2 matrices. Thus the basis for ker(L) consists of these three
matrices so nul (L) = 3.        
1 0 0 1 0 0 0 0
To find the basis for im(L), let us fix the standard basis , , ,
0 0 0 0 1 0 0 1
2×2
of R , and then im(L) is spanned by the images of the four basis vectors by L, i.e.,
        
1 0 0 1 0 0 0 0
im(L) = Span L ,L ,L ,L
0 0 0 0 1 0 0 1
       
0 1 0 −1 0 1
= Span O, , , O = Span ,
−1 0 1 0 −1 0

the set of antisymmetric 2 × 2 matrices. Thus the basis for im(L) consists of that one
matrix, so rank (L) = 1. Rank-Nullity Theorem in this special case is verified as

rank (L) + nul (L) = 1 + 3 = 4 = dim(R2×2 ).

72 CHAPTER 5. LINEAR TRANSFORMATIONS

5.9 Linear maps, coordinatisation and matrices

In this section we use the notation for the coordinatisation introduced in 4.54 and for the
resulting coordinatisation isomorphism discussed in 5.26.
Definition 5.38. Let
• B = (b1 , . . . , bn ) be a basis for a vector space U .

• C = (c1 , . . . , cm ) be a basis for a vector space V ,

• L : U → V be a linear map.
The matrix associated to L with respect to the pair of bases (B, C) is the matrix

[L]B
C ∈ R
m×n

with columns [L(b1 )]C , . . . [L(bP

n )]C .
More explicitly, if L(bj ) = m i=1 αij ci for j = 1, . . . , n, then
   
α1j α11 · · · α1n
 ..  m B  .. ..  .
[L(bj )]C =  .  ∈ R and [L]C =  . . 
αmj αm1 · · · αmn

Remark 5.39. Using the fact that a linear map is uniquely determined by its action on
the basis (Proposition 5.15), the above linear map L is uniquely determined by the n-tuple
of vectors (L(b1 ), . . . , L(bn )) in V . Each of these vectors is in turn uniquely determined
by its coordinate vector with respect to the basis C, so we conclude that L is uniquely
determined by the n-tuple of column vectors ([L(b1 )]C , . . . [L(bn )]C ) from Rm , i.e., by its
associated matrix [L]B C.

Proposition 5.40. With the above notation, for each u ∈ U ,

[L(u)]C = [L]B
C · [u]B ,

where the operation on the right is matrix multiplication.

Proof. If u = ni=1 αi bi , then L(u) = L( ni=1 αi bi ) = ni=1 αi L(bi ). Thus, using the
P P P
fact that [ ]C is a linear map, as noted in 5.26, we obtain
" n # n
X X
[L(u)]C = αi L(bi ) = αi [L(bi )]C .
i=1 C i=1

Given that [L(bi )]C are columns of the matrix [L]B

C associated to L and the definition of
matrix multiplication, the last term in the above equals
 
α1
B  .. 
[L]C  .  = [L]B
C [u]B .
αn

Remark 5.41. The statement of the above proposition is equivalent to saying that the
diagram
5.9. LINEAR MAPS, COORDINATISATION AND MATRICES 73

[ ]B
U Rn
L L[L]B
C
[ ]C
V Rm

is commutative, i.e., that the composites of all maps along directed paths between a
given domain and a codomain are equal. In this case, the statement is that

[ ]C ◦ L = L[L]BC ◦ [ ]B .

This is equivalent to the statement of the proposition, since, by evaluatingon any u ∈U ,
the left hand side gives ([ ]C ◦L)(u) = [L(u)]C , and the right hand side gives L[L]BC ◦ [ ]B (u) =
L[L]BC ([u]B ) = [L]B
C [u]B .

Proposition 5.42. Let B be a basis of a vector space U of dimension n, and let C be a

basis of a vector space V of dimension m. Assume L, L0 : U → V are linear maps, i.e.,
L, L0 ∈ Hom(U, V ). Then

1. [L + L0 ]B B 0 B
C = [L]C + [L ]C ;

2. [αL]B B
C = α[L]C .

In words, the main idea behind this correspondence is that, by using coordinatisation,
addition of linear operators corresponds to addition of associated matrices, and a scalar
multiple of an operator corresponds to a scalar multiple of the associated matrix.
The proof follows directly from Definition 5.38, so it is left as an exercise.

Corollary 5.43. Let

• B be a basis for a vector space U of dimension n;

• C be a basis for a vector space V of dimension m;

• D be a basis for a vector space W of dimension p;

• L : U → V be a linear map;

• M : V → W be a linear map.

Then
[M ◦ L]B C B
D = [M ]D · [L]C .

Proof. Before proceeding with the essence of the proof, let us verify that the formats of
the above matrices are compatible (even though it is automatic from the proof). Indeed,
Definition 5.38 tells us that [L]B
C ∈ R
m×n
, [M ]C
D ∈ R
p×m
, so the product [M ]C B
D ·[L]C ∈ R
p×n
.
B p×n
On the other hand, [M ◦ L]D ∈ R too, so it makes sense to ask whether these matrices
are equal.
Coordinatisation with respect to given bases gives the diagram
74 CHAPTER 5. LINEAR TRANSFORMATIONS

[ ]B
U Rn
L L[L]B
C
[ ]C
V Rm
M L[M ]C
D
[ ]D
W Rp

Hence, M ◦ L corresponds to L[M ]CD ◦ L[L]BC = L[M ]CD ·[L]BC , and the claim follows.
We now consider a special case of matrices associated to endomorphisms of a vector
spaces.

Definition 5.44. Let B be a basis for a vector space V of dimension n, and let L : V → V
be a linear map. We define
[L]B = [L]B B ∈ R
n×n
.

Corollary 5.45. Given linear maps L, L0 , M from V to V and α ∈ R, we have

[L + L0 ]B = [L]B + [L0 ]B ;
[αL]B = α[L]B ;
[M ◦ L]B = [M ]B · [L]B .

Proof. The first two properties follow from 5.42 applied to the special case of U = V and
[ ]B = [ ]B
B , and the third property follows from 5.43 applied to the special case U = V = W ,
where we choose the same basis B for all spaces.

Example 5.46. Consider the derivation operator

D : P3 → P2 ,

and choose the basis B = (1, t, t2 , t3 ) for P3 and the basis C = (1, t, t2 ) for P2 .
Then [D]B 2 3
C has columns [D(1)]C , [D(t)]C , [D(t )]C , [D(t )]C , i.e., the columns are
       
0 1 0 0
2
[0]C = 0 , [1]C = 0 , [2t]C = 2 , [3t ]C = 0 ,
      
0 0 0 3

so we assemble them into the matrix

 
0 1 0 0
[D]B
C =
0 0 2 0 .
0 0 0 3

In Example 5.36, we computed the rank and nullity of D by first principles, but the above
material on associated matrices tells us that we could also obtain them as

rank (D) = rank ([D]B B

C ) = 3, and nul (D) = nul ([D]C ) = 1.

Example 5.47. Consider the linear map

L : R2×2 → R2×2 , L(A) = A − AT .

5.10. CHANGE OF BASIS 75
        
1 0 0 1 0 0 0 0
Let B be the standard basis E11 = , E12 = , E21 = , E22 =
0 0 0 0 1 0 0 1
of R2×2 .
Then [L]B = [L]B
B has columns [L(E11 )]B , [L(E12 )]B ,[L(E21 )]B ,[L(E22 )]B , i.e., the
columns
       
0   0   0 0
0 0 1 1 0 −1 −1 0
[O]B = 
0 , −1 0
 =
−1 , 1 0
 =  1  , [O]B = 0 ,
  
B B
0 0 0 0
and we assemble them into the matrix
 
0 0 0 0
0 1 −1 0
[L]B = 
0 −1 1 0 .


0 0 0 0

5.10 Change of basis

Coordinatisation maps depend on the choice of relevant bases we make. In this section,
we investigate how the associate matrix of a linear operator changes when we choose
different bases for its domain and codomain.
Definition 5.48. Let B and B 0 be bases for a vector space U of dimension n. The matrix

PB,B 0 = [idU ]B
B0 ∈ R
n×n

is called the transition matrix from B to B 0 .

Remark 5.49. By 5.38, the matrix PB,B 0 has columns

[b1 ]B 0 , . . . , [bn ]B 0 ,

where B = (b1 , . . . , bn ), and, by 5.40, it satisfies the property

[u]B 0 = PB,B 0 [u]B ,

for u ∈ U . This justifies the name of ‘transition matrix from B to B 0 ’, because the
coordinates with respect to B 0 are easily calculated using PB,B 0 from coordinates with
respect to B.
Lemma 5.50. Let B, B 0 , B 00 be bases of U . Then
1. PB,B = In ;
2. PB 0 ,B 00 PB,B 0 = PB,B 00 ;
−1
3. PB,B 0 is invertible and PB,B 0 = PB 0 ,B .

Proof. Property (1) is obvious from the definition, and (2) follows from 5.43, since
0
[id]B B B B
B 00 = [id ◦ id]B 00 = [id]B 00 [id]B 0 .

Property (3) follows by applying (2) to the special case where B 00 = B, and then using
(1).
76 CHAPTER 5. LINEAR TRANSFORMATIONS

Theorem 5.51 (Change of basis formula). Let

• B, B 0 be bases for a vector space U of dimension n;
• C, C 0 be bases for a vector space V of dimension m;
• L : U → V be a linear map.
Then
0 0
−1
[L]B C B B B
C 0 = [idV ]C 0 [L]C [idU ]B = PC,C 0 [L]C PB,B 0 .

Proof. Putting together the diagrams from 5.41 for linear maps idU , idV and L with
respect to relevant combinations of bases B, B 0 , C, C 0 , we obtain the following diagram.
[ ]B 0
U Rn
idU
L
[ ]B L[id]B L[L]B0
U Rn B0 C0

[ ]C 0
L
V Rm
idV L[L]B
C
L[id]C
[ ]C C0
V Rm

From the right face of this diagram, we see that

L[L]B00 ◦ L[idU ]B0 = L[idV ]C 0 ◦ L[L]BC .
C B C

Bearing in mind that the composite of linear transformations associated to matrices cor-
responds to the linear transformation associated to the product of those matrices, we can
rewrite this as
0
[L]B B C B
C 0 · [idU ]B 0 = [idV ]C 0 · [L]C ,
0 B −1


and the statement follows by multiplying both sides by [idU ]B B = [idU ] B 0 .
Corollary 5.52. If C and C 0 are bases for V , and M : V → V is a linear map, then
−1
[M ]C 0 = PC,C 0 [M ]C PC,C 0.

   
1 0
Example 5.53. Let B = , be the standard basis for R2 and let B 0 =
    0 1
−1 0
, be another basis for R2 . Assume L : R2 → R2 has the associated matrix
1 2
with respect to B  
1 −1
[L]B = .
2 3
What is [L]B 0 ?
Since B is standard, the transition matrix PB 0 ,B is the matrix whose columns are
simply the vectors of B 0 , i.e.,  
−1 0
PB 0 ,B = ,
1 2
and now, by the change of basis formula,
     
−1 −1 0 1 −1 −1 0 2 2
[L]B 0 = PB,B 0 [L]B PB,B 0 = PB−10 ,B [L]B PB 0 ,B = = .
1/2 1/2 2 3 1 2 −1/2 2
5.11. HOM SPACES AND END ALGEBRAS 77

5.11 Hom spaces and End algebras

This section is not examinable. It shows how the previous results on linear maps fit
together into a bigger picture, and reflects the way modern mathematicians think of
linear algebra.
Definition 5.54. Let U and V be vector spaces. We write

Hom(U, V )

for the set of all linear transformations from U to V .

Proposition 5.55. With the above notation, Hom(U, V ) is a vector space.
Proof. By Lemma 5.17, Hom(U, V ) is closed under the operations:
1. addition of linear transformations;
2. multiplication of a linear transformation by a scalar.
It is straightforward to verify that Hom(U, V ), endowed with these operations, is actually
a vector space.
Remark 5.56. In the case of standard vector spaces, the Hom space is far from myste-
rious; as shown by Example 5.18,

Hom(Rn , Rm ) = Rm×n ,

the vector space of m × n matrices.

Definition 5.57. Let V be a vector space. We write

End(V ) = Hom(V, V )

for the set of all linear transformations from V to V (also called endomorphisms of V ).
Proposition 5.58. With the above notation, End(V ) is an algebra over the field R.
Proof. We only sketch the proof, given that we have not defined the notion of ‘algebra’.
By Lemma 5.17, End(V ) is closed under the operations:
1. addition of linear transformations;
2. multiplication of a linear transformation by a scalar;
3. composition of linear transformations,
and hence constitutes a mathematical structure called an algebra over the field R.
We will not delve into a detailed study of algebras, but students should think of them
as vector spaces equipped with an additional operation of multiplication of vectors, com-
patible with the vector space structure. For students who took Introduction to Algebra,
we point out that, not only is End(V ) a vector space, but also (End(V ), +, ◦) is a ring.
Remark 5.59. In the case of standard vector spaces, Example 5.18 shows that

End(Rn ) = Rn×n ,

the algebra of n × n square matrices, with the operations

78 CHAPTER 5. LINEAR TRANSFORMATIONS

1. matrix addition;

2. scalar multiplication;

3. matrix multiplication.

In the sequel, we will show that every abstract Hom space can be identified with a
space of matrices, and every abstract End algebra can be identified with an algebra of
square matrices.

Definition 5.60. Let V be a vector space, and let

Aut(V ) = {L ∈ End(V ) : L is an isomorphism}

denote the set of invertible linear maps from V to itself.

Remark 5.61. The set Aut(V ) is a group under the operation of composition of linear
operators. Students familiar with the concept from Introduction to Algebra module will be
able to prove this using the fact that the composite of two isomorphisms is an isomorphism
and that the identity map on V is an isomorphism.

Remark 5.62. Definition 5.38 gives rise to the map

[ ]B
C : Hom(U, V ) → R
m×n
,

which associates to every linear transformation from U to V its matrix with respect to
bases B, C. Remark 5.39 shows that this map is bijective, and Proposition 5.42 shows
that it is linear, so it is in fact an isomorphism of vector spaces.

Remark 5.63. Corollary 5.43 shows that the composite of linear maps corresponds,
via coordinatisation, to the product of associated matrices. This can be viewed as the
following diagram:

[ ]C B
D × [ ]C
Hom(V, W ) × Hom(U, V ) Rp×m × Rm×n
◦ ·
[ ]B
D
Hom(U, W ) Rp×n

Remark 5.64. Corollary 5.45 shows that we have an isomorphism of algebras

[ ]B : End(V ) → Rn×n .

The restriction of this map to Aut(V ) ⊆ End(V ) gives a group isomorphism

Aut(V ) → GLn (R)

with the general linear group consisting of n×n invertible matrices, with the operation
of matrix product.
Chapter 6

Eigenvalues and Eigenvectors

In this last chapter of our exploration of Linear Algebra we will revisit eigenvalues and
eigenvectors of matrices, concepts that were already introduced in Geometry I and possibly
also used in other modules. After a short repetition of the basic facts we will arrive at
our main result, a spectral theorem for symmetric matrices.

6.1 Definition and examples

If A is a square n × n matrix, we may regard it as a linear transformation from Rn to Rn .
This transformation sends a vector x ∈ Rn to the vector Ax. For certain vectors, this
action can be very simple.

Example 6.1. Let

     
1 2 1 2
A= , u= , w= .
−1 4 1 1

Then
    
1 2 1 3
Au = = = 3u
−1 4 1 3
    
1 2 2 4
Aw = = = 2w
−1 4 1 2

so the action of A on u and w is very easy to picture: it simply amounts to a stretching

by 3 and 2, respectively.

Definition 6.2. An eigenvector of an n × n matrix A is a nonzero vector x such that

Ax = λx ,

for some scalar λ. A scalar λ is called an eigenvalue of A if there is a non-trivial solution

x to Ax = λx, in which case we say that x is an eigenvector corresponding to the
eigenvalue λ.

Remark 6.3. Note that if x is an eigenvector of a matrix A with eigenvalue λ, then any
nonzero multiple of x is also an eigenvector corresponding to λ, since

A(αx) = αAx = αλx = λ(αx) .

79
80 CHAPTER 6. EIGENVALUES AND EIGENVECTORS

We shall now investigate how to determine all the eigenvalues and eigenvectors of an
n×n matrix A. We start by observing that the defining equation Ax = λx can be written

(A − λI)x = 0 . (6.1)

Thus λ is an eigenvalue of A if and only if (6.1) has a non-trivial solution. The set of
solutions of (6.1) is N (A − λI), that is, the nullspace of A − λI, which is a subspace of
Rn .Thus, λ is an eigenvalue of A if and only if

N (A − λI) 6= {0} ,

and any nonzero vector in N (A − λI) is an eigenvector belonging to λ. Moreover, by

the Invertible Matrix Theorem, (6.1) has a non-trivial solution if and only if the matrix
A − λI is singular, or equivalently

det(A − λI) = 0 . (6.2)

Notice now that if the determinant in (6.2) is expanded we obtain a polynomial of degree
n in the variable λ,
p(λ) = det(A − λI) ,
called the characteristic polynomial of A , and equation (6.2) is called the charac-
teristic equation of A. So, in other words, the roots of the characteristic polynomial
of A are exactly the eigenvalues of A. The following theorem summarises our findings so
far:
Theorem 6.4. Let A be an n × n matrix and λ a scalar. The following statements are
equivalent:
(a) λ is an eigenvalue of A;

(b) (A − λI)x = 0 has a non-trivial solution;

(c) N (A − λI) 6= {0};

(d) A − λI is singular;

(e) det(A − λI) = 0.

In view of the above theorem the following concept arises naturally:
Definition 6.5. If A is a square matrix and λ an eigenvalue of A, then N (A − λI) is
called the eigenspace corresponding to λ.
Note that if λ is an eigenvalue of a matrix A, then every nonzero vector in the cor-
responding eigenspace N (A − λI) is an eigenvector corresponding to λ, and conversely,
the set of all eigenvectors corresponding to λ together with the zero vector forms the
corresponding eigenspace N (A − λI).
We shall now see how to use (e) in the theorem above to determine the eigenvalues
and eigenvectors of a given matrix.
Example 6.6. Find all eigenvalues and eigenvectors of the matrix
 
−7 −6
A= .
9 8
6.1. DEFINITION AND EXAMPLES 81

Proof. First we calculate the characteristic polynomial of A

−7 − λ −6
det(A − λI) = = (−7 − λ)(8 − λ) − (−6) · 9
9 8−λ
= −56 + 7λ − 8λ + λ2 + 54 = λ2 − λ − 2 = (λ + 1)(λ − 2) .

Thus the characteristic equation is

(λ + 1)(λ − 2) = 0 ,

so the eigenvalues of the matrix are λ1 = −1 and λ2 = 2.

In order to find the eigenvectors belonging to λ1 = −1 we must determine the nullspace
of A − λ1 I = A + I. Or, put differently, we need to determine all solutions of the system
(A + I)x = 0. This can be done using your favourite method, but, for reasons which will
become clear in the next example, I strongly recommend Gaussian elimination, that is,
we bring the augmented matrix (A + I|0) to row echelon form:

       
−7 + 1 −6 0 −6 −6 0 1 1 0 1 1 0
(A + I|0) = = ∼ ∼ ,
9 8+1 0 9 9 0 9 9 0 0 0 0

so setting x2 = α we find x1 = −x2 = −α. Thus every vector in N (A + I) is of the form

   
−α −1
=α ,
α 1

so the eigenspace corresponding to the eigenvalue −1 is

   
−1
α α∈R ,
1
 
−1
and any nonzero multiple of is an eigenvector corresponding to the eigenvalue −1.
1
Similarly, in order to find the eigenvectors belonging to λ2 = 2 we bring (A − λ2 I|0) =
(A − 2I|0) to row echelon form:

1 23 0 1 23 0
       
−7 − 2 −6 0 −9 −6 0
(A − 2I|0) = = ∼ ∼ ,
9 8−2 0 9 6 0 9 6 0 0 0 0

so setting x2 = α we find x1 = − 32 x2 = − 23 α. Thus every vector in N (A − 2I) is of the

form  2   
−3α α −2
= ,
α 3 3
so the eigenspace corresponding to the eigenvalue 2 is
   
−2
α α∈R ,
3
 
−2
and any nonzero multiple of is an eigenvector corresponding to the eigenvalue
3
2.
82 CHAPTER 6. EIGENVALUES AND EIGENVECTORS

Before we continue with another example you might want to have another look at the
above calculations of eigenspaces. Observe that since we need to solve a homogeneous
linear system there is no need to write down the right-most column of the augmented
matrix (since it consists only of zeros); we simply perform elementary row operations
on the coefficient matrix, keeping in mind that the right-most column of the augmented
matrix will remain the zero column. We shall use this short-cut in all the following
calculations of eigenspaces.
Example 6.7. Let  
2 −3 1
A = 1 −2 1 .
1 −3 2
Find the eigenvalues and corresponding eigenspaces.
Solution. A slightly tedious calculation using repeated cofactor expansions shows that the
characteristic polynomial of A is
2−λ −3 1
det(A − λI) = 1 −2 − λ 1 = −λ(λ − 1)2 ,
1 −3 2−λ
so the eigenvalues of A are λ1 = 0 and λ2 = 1.
In order to find the eigenspace corresponding to λ1 we find the nullspace of A−λ1 I = A
using Gaussian elimination:
   
2 −3 1 1 0 −1
A = 1 −2 1 ∼ · · · ∼ 0 1 −1 ,
1 −3 2 0 0 0

so setting x3 = α we find x2 = 0 − (−1)x3 = α and x1 = 0 − (−1)x3 = α. Thus, every

vector in N (A) is of the form    
α 1
α = α 1 ,
α 1
so the eigenspace corresponding to the eigenvalue 0 is
   
 1 
α 1 α ∈ R .
1
 

In order to find the eigenspace corresponding to λ2 we find the nullspace of A − λ2 I =

A − I, again using Gaussian elimination:
     
2−1 −3 1 1 −3 1 1 −3 1
A−I = 1 −2 − 1 1  = 1 −3 1 ∼ 0 0 0 ,
1 −3 2−1 1 −3 1 0 0 0
so setting x2 = α and x3 = β we find x1 = 3x2 − x3 = 3α − β. Thus every vector in
N (A − I) is of the form
     
3α − β 3 −1
 α  = α 1 + β  0  ,
β 0 1
6.1. DEFINITION AND EXAMPLES 83

and so the eigenspace corresponding to the eigenvalue 1 is

     
 3 −1 
α 1 + β  0  α, β ∈ R .
0 1
 

Example 6.8. Find the eigenvalues of the matrix

 
1 2 3
A = 0 4 5 .
0 0 6

Solution. Using the fact that the determinant of a triangular matrix is the product of the
diagonal entries we find

1−λ 2 3
det(A − λI) = 0 4−λ 5 = (1 − λ)(4 − λ)(6 − λ) ,
0 0 6−λ

so the eigenvalues of A are 1, 4, and 6.

The above example and its method of solution are easily generalised:

Theorem 6.9. The eigenvalues of a triangular matrix are precisely the diagonal entries
of the matrix.

The next theorem gives an important sufficient (but not necessary) condition for two
matrices to have the same eigenvalues. It also serves as the foundation for many numerical
procedures to approximate eigenvalues of matrices, some of which you will encounter if
you take the module MTH5110, Introduction to Numerical Computing.

Theorem 6.10. Let A and B be two n×n matrices and suppose that A and B are similar,
that is, there is an invertible matrix S ∈ Rn×n such that B = S −1 AS. Then A and B
have the same characteristic polynomial, and, consequently, have the same eigenvalues.

Proof. If B = S −1 AS, then

B − λI = S −1 AS − λI = S −1 AS − λS −1 S = S −1 (AS − λS) = S −1 (A − λI)S .

Thus, using the multiplicativity of determinants,

det(B − λI) = det(S −1 ) det(A − λI) det(S) = det(A − λI) ,

because det(S −1 ) det(S) = det(S −1 S) = det(I) = 1.

We will revisit this theorem from a different perspective in the next section.
84 CHAPTER 6. EIGENVALUES AND EIGENVECTORS

6.2 Diagonalisation
In many applications of Linear Algebra one is faced with the following problem: given
a square matrix A, find the k-th power Ak of A for large values of k. In general, this
can be a very time-consuming task. For certain matrices, however, evaluating powers is
spectacularly easy:

Example 6.11. Let D ∈ R2×2 be given by

 
2 0
D= .
0 3

Then     2 
2 2 0 2 0 2 0
D = =
0 3 0 3 0 32
and   2   3 
3 2 02 2 0 2 0
D = DD = = .
0 3 0 32 0 33
In general,  
k 2k 0
D = ,
0 3k
for k ≥ 1.

After having had another look at the example above, you should convince yourself that
if D is a diagonal n × n matrix with diagonal entries d1 , . . . , dn , then Dk is the diagonal
matrix whose diagonal entries are dk1 , . . . , dkn . The moral of this is that calculating powers
for diagonal matrices is easy. What if the matrix is not diagonal? The next best situation
arises if the matrix is similar to a diagonal matrix. In this case, calculating powers is
almost as easy as calculating powers of diagonal matrices, as we shall see shortly. We
shall now single out matrices with this property and give them a special name:

Definition 6.12. An n × n matrix A is said to be diagonalisable if it is similar to a

diagonal matrix, that is, if there is an invertible matrix P ∈ Rn×n such that

P −1 AP = D ,

where D is a diagonal matrix. In this case we say that P diagonalises A.

Note that if A is a matrix which is diagonalised by P , that is, P −1 AP = D with D

diagonal, then

A = P DP −1 ,
A2 = P DP −1 P DP −1 = P D2 P −1 ,
A3 = AA2 = P DP −1 P D2 P −1 = P D3 P −1 ,

and in general
Ak = P Dk P −1 ,
for any k ≥ 1. Thus powers of A are easily computed, as claimed.
In the following, we list (without proof) a few useful theorems relating eigenvalues,
eigenvectors, and the linear independence property:
6.2. DIAGONALISATION 85

Theorem 6.13. If v1 , . . . , vr are eigenvectors that correspond to distinct eigenvalues

λ1 , . . . , λr of an n × n matrix A, then the vectors v1 , . . . , vr are linearly independent.
Theorem 6.14 (Diagonalisation Theorem). An n × n matrix A is diagonalisable if and
only if A has n linearly independent eigenvectors.
In fact, P −1 AP = D, with D a diagonal matrix, if and only if the columns of P
are n linearly independent eigenvectors of A. In this case, the diagonal entries of D are
eigenvalues of A that correspond, respectively, to the eigenvectors in P .
The proofs are not difficult, but rather than doing that here we want to concentrate
onto some practical examples:
Example 6.15. Diagonalise the following matrix, if possible:
 
−7 3 −3
A = −9 5 −3 .
9 −3 5
Solution. A slightly tedious calculation shows that the characteristic polynomial is given
by
−7 − λ 3 −3
p(λ) = det(A − λI) = −9 5 − λ −3 = −λ3 + 3λ2 − 4 .
9 −3 5 − λ
The cubic p above can be factored by spotting that −1 is a root. Polynomial division
then yields
p(λ) = −(λ + 1)(λ2 − 4λ + 4) = −(λ + 1)(λ − 2)2 ,
so the distinct eigenvalues of A are 2 and −1.
The usual methods (see Examples 6.6 and 6.7) now produce a basis for each of the
two eigenspaces and it turns out that
   
1 −1
N (A − 2I) = Span (v1 , v2 ), where v1 = 3 , v2 =
   0,
0 3
 
−1
N (A + I) = Span (v3 ), where v3 = −1 .
1
You may now want to confirm, using your favourite method, that the three vectors
v1 , v2 , v3 are linearly independent. As we shall see shortly, this is not really necessary:
the union of basis vectors for eigenspaces always produces linearly independent vectors
(see the proof of Corollary 6.17 below).
Thus, A is diagonalisable, since it has 3 linearly independent eigenvectors. In order to
find the diagonalising matrix P we recall that defining
 

1 −1 −1
P = v1 v2 v3 = 3 0 −1
0 3 1
does the trick, that is, P −1 AP = D, where D is the diagonal matrix whose entries are
the eigenvalues of A and where the order of the eigenvalues matches the order chosen for
the eigenvectors in P , that is,  
2 0 0
D = 0 2 0  .
0 0 −1
86 CHAPTER 6. EIGENVALUES AND EIGENVECTORS

It is good practice to check that P and D really do the job they are supposed to do:
    
−7 3 −3 1 −1 −1 2 −2 1
AP = −9 5 −3 3 0 −1 = 6 0 1,
9 −3 5 0 3 1 0 6 −1
    
1 −1 −1 2 0 0 2 −2 1
P D = 3 0 −1 0 2 0  = 6 0 1,
0 3 1 0 0 −1 0 6 −1
so AP = P D, and hence P −1 AP = D as required.

Example 6.16. Diagonalise the following matrix, if possible:

 
−6 3 −2
A = −7 5 −1 .
8 −3 4

Solution. The characteristic polynomial of A turns out to be exactly the same as in the
previous example:

det(A − λI) = −λ3 + 3λ2 − 4 = −(λ + 1)(λ − 2)2 .

Thus the eigenvalues of A are 2 and −1. However, in this case it turns out that both
eigenspaces are 1-dimensional:
 
1
N (A − 2I) = Span (v1 ) where v1 =  2,
−1
 
−1
N (A + I) = Span (v2 ) where v2 = −1 .
1
Since A has only 2 linearly independent eigenvectors, the Diagonalisation Theorem implies
that A is not diagonalisable.

Put differently, the Diagonalisation Theorem states that a matrix A ∈ Rn×n is diag-
onalisable if and only if A has enough eigenvectors to form a basis of Rn . The following
corollary makes this restatement even more precise:

Corollary 6.17. Let A ∈ Rn×n and let λ1 , . . . , λr be the (distinct) eigenvalues of A. Then
A is diagonalisable if and only if

dim N (A − λ1 I) + · · · + dim N (A − λr I) = n .

A very useful special case of the Diagonalisation Theorem is the following:

Theorem 6.18. An n × n matrix with n distinct eigenvalues is diagonalisable.

Proof. Let v1 , . . . , vn be eigenvectors corresponding to the n distinct eigenvalues of A.

Then the n vectors v1 , . . . , vn are linearly independent by Theorem 6.13. Hence A is
diagonalisable by the Diagonalisation Theorem.
6.3. INTERLUDE: COMPLEX VECTOR SPACES AND MATRICES 87

Remark 6.19. Note that the above condition for diagonalisability is sufficient but not
necessary: an n × n matrix which does not have n distinct eigenvalues may or may not
be diagonalisable (see Examples 6.15 and 6.16).

Example 6.20. The matrix  

1 −1 5
A = 0 2 6 
0 0 3
is diagonalisable, since it has three distinct eigenvalues 1, 2, and 3.

6.3 Interlude: complex vector spaces and matrices

Consider the matrix  
0 −1
A= .
1 0
What are the eigenvalues of A? Notice that

−λ −1
det(A − λI) = = λ2 + 1 ,
1 −λ

so the characteristic polynomial does not have any real roots, and hence A does not have
any real eigenvalues. However, since

λ2 + 1 = λ2 − (−1) = λ − i2 = (λ − i)(λ + i) ,

the characteristic polynomial has two complex roots, namely i and −i. Thus it makes
sense to say that A has two complex eigenvalues i and −i. What are the corresponding
eigenvectors? Solving
(A − iI)x = 0
leads to the system
−ix1 − x2 = 0
x1 − ix2 = 0
 
1
Both equations yield the condition x2 = −ix1 , so is an eigenvector corresponding
−i
to the eigenvalue i. Indeed
        
0 −1 1 i i 1
= = =i .
1 0 −i 1 −i2 −i
 
1
Similarly, we see that is an eigenvector corresponding to the eigenvalue −i. Indeed
i
        
0 −1 1 −i −i 1
= = = −i .
1 0 i 1 −i2 i

The moral of this example is the following: on the one hand, we could just say that
the matrix A has no real eigenvalues and stop the discussion right here. On the other
hand, we just saw that it makes sense to say that A has two complex eigenvalues with
corresponding complex eigenvectors.
88 CHAPTER 6. EIGENVALUES AND EIGENVECTORS

This leads to the idea of leaving our current real set-up, and enter a complex realm
instead. As it turns out, this is an immensely powerful idea. However, as our time is
limited, we shall only cover the bare necessities, allowing us to prove the main result of
the next section.
Let Cn denote the set of all n-vectors with complex entries, that is,
  

 z 1 

n  .. 
C =  .  z1 , . . . , zn ∈ C .

 z 

n

Just as in Rn , we add vectors in Cn by adding their entries, and we can multiply a vector
in Cn by a complex number, by multiplying each entry.
Example 6.21. Let z, w ∈ C3 and α ∈ C, with
   
1+i −2 + 3i
z =  2i  , w =  1  , α = (1 + 2i) .
3 2+i

Then    
(1 + i) + (−2 + 3i) −1 + 4i
z+w = 2i + 1  =  1 + 2i 
3 + (2 + i) 5+i
     
(1 + 2i)(1 + i) 1 + 2i + i + 2i2 −1 + 3i
αz =  (1 + 2i)(2i)  =  2i + (2i)2  = −4 + 2i
(1 + 2i) · 3 3 + 6i 3 + 6i
If addition and scalar multiplication is defined in this way (now allowing scalars to
be in C), then Cn satisfies all the axioms of a vector space. Similarly, we can introduce
the set of all m × n matrices with complex entries, call it Cm×n , and define addition and
scalar multiplication (again allowing complex scalars) entry-wise just as in Rm×n . Again,
Cm×n satisfies all the axioms of a vector space.
Fact 6.22. All the results in Chapters 1–5, and all the results from the beginning of this
chapter hold verbatim, if ‘scalar’ is taken to mean ‘complex number’.
Since ‘scalars’ are now allowed to be complex numbers, Cn and Cm×n are known as
complex vector spaces.
The reason for allowing this more general set-up is that, in a certain sense, complex
numbers are much nicer than real numbers. More precisely, we have the following result:
Theorem 6.23 (Fundamental Theorem of Algebra). If p is a complex polynomial of
degree n ≥ 1, that is,
p(z) = cn z n + · · · + c1 z + c0 ,
where c0 , c1 , . . . , cn ∈ C, then p has at least one (possibly complex) root.
Corollary 6.24. Every matrix A ∈ Cn×n has at least one (possibly complex) eigenvalue
and a corresponding eigenvector z ∈ Cn .
Proof. Since λ is an eigenvalue of A if and only if det(A − λI) = 0 and since p(λ) =
det(A − λI) is a polynomial with complex coefficients of degree n, the assertion follows
from the Fundamental Theorem of Algebra.
6.3. INTERLUDE: COMPLEX VECTOR SPACES AND MATRICES 89

The corollary above is the main reason why complex vector spaces are considered. We
are guaranteed that every matrix has at least one eigenvalue, and we may then use the
powerful tools developed in the earlier parts of this chapter to analyse matrices through
their eigenvalues and eigenvectors.
90 CHAPTER 6. EIGENVALUES AND EIGENVECTORS
Chapter 7

Orthogonality

In this chapter we will return to the concrete vector space Rn and add a new concept
that will reveal new aspects of it. The added spice in the discussion is the notion of
‘orthogonality’. This concept extends our intuitive notion of perpendicularity in R2 and
R3 to Rn . This new concept turns out to be a rather powerful device, as we shall see
shortly.

7.1 Definition
We start by revisiting a concept that you have already encountered in Geometry I. Before
stating it recall that a vector x in Rn is, by definition, an n × 1 matrix. Given another
vector y in Rn , we may then form the matrix product xT y of the 1 × n matrix xT and the
n × 1 matrix y. Notice that by the rules of matrix multiplication xT y is a 1 × 1 matrix,
which we can simply think of as a real number.

Definition 7.1. Let x and y be two vectors in Rn . The scalar xT y is called the scalar
product or dot product of x and y, and is often written x·y. Thus, if
   
x1 y1
 x2   y2 
x =  ..  , y =  ..  ,
   
. .
xn yn

then
 
y1

 y2 

x·y = xT y = x1 x2 · · · xn  ..  = x1 y1 + x2 y2 + · · · + xn yn .

.
yn

Example 7.2. If
   
2 4
x = −3 and y = 5 ,
1 6

91
92 CHAPTER 7. ORTHOGONALITY

then
 
4
T


x·y = x y = 2 −3 1  5 = 2 · 4 + (−3) · 5 + 1 · 6 = 8 − 15 + 6 = −1 ,
6
 

2
y·x = yT x = 4 5 6 −3 = 4 · 2 + 5 · (−3) + 6 · 1 = 8 − 15 + 6 = −1 .
1

Having had a second look at the example above it should be clear why x·y = y·x. In
fact, this is true in general. The following further properties of the dot product follow
easily from properties of the transpose operation:

Theorem 7.3. Let x, y and z be vectors in Rn , and let α be a scalar. Then

(a) x·y = y·x;

(b) (x + y)·z = x·z + y·z;

(c) (αx)·y = α(x·y) = x·(αy);

(d) x·x ≥ 0, and x·x = 0 if and only if x = 0.

Definition 7.4. If x = (x1 , . . . , xn )T ∈ Rn , the length or norm of x is the nonnegative

scalar kxk defined by
√ q
kxk = x·x = x21 + · · · + x2n .
A vector whose length is 1 is called a unit vector .

Example 7.5. If x = (a, b)T ∈ R2 , then

√
kxk = a2 + b2 .

The above example should convince you that in R2 and R3 the definition of the length
of a vector x coincides with the standard notion of the length of the line segment from
the origin to x.
Note that if x ∈ Rn and α ∈ R then

kαxk = |α| kxk ,

because kαxk2 = (αx)·(αx) = α2 (x·x) = α2 kxk2 . Thus, if x 6= 0, we can always find a

unit vector y in the same direction as x by setting
1
y= x.
kxk

The process of creating a unit vector y from x is called normalising x.

Definition 7.6. For x and y in Rn , the distance between x and y, written dist(x, y),
is the length of x − y, that is,

dist(x, y) = kx − yk .

Definition 7.7. Two vectors x and y in Rn are orthogonal (to each other) if x·y = 0.
7.2. ORTHOGONAL COMPLEMENTS 93

Note that the zero vector is orthogonal to every other vector in Rn .

The following theorem is an old acquaintance in new clothing, and, at the same time,
contains a key fact about orthogonal vectors:

Theorem 7.8 (Pythagorean Theorem). Two vectors x and y in Rn are orthogonal if and
only if
kx + yk2 = kxk2 + kyk2 .

Proof. See Exercise 6 on Coursework 8.

7.2 Orthogonal complements

In this short section we introduce an important concept that will form the basis of sub-
sequent developments.

Definition 7.9. Let Y be a subspace of Rn . A vector x ∈ Rn is said to be orthogonal to

Y if x is orthogonal to every vector in Y . The set of all vectors in Rn that are orthogonal
to Y is called the orthogonal complement of Y and is denoted by Y ⊥ (pronounced ‘Y
perpendicular’ or ‘Y perp’ for short). Thus

Y ⊥ = { x ∈ Rn | x·y = 0 for all y ∈ Y } .

Example 7.10. Let W be a plane through the origin in R3 and let L be the line through
the origin and perpendicular to W . By construction, each vector in W is orthogonal to
every vector in L, and each vector in L is orthogonal to every vector in W . Hence

L⊥ = W and W⊥ = L.

The following theorem collects some useful facts about orthogonal complements.

Theorem 7.11. Let Y be a subspace of Rn . Then:

(a) Y ⊥ is a subspace of Rn .

(b) A vector x belongs to Y ⊥ if and only if x is orthogonal to every vector in a set that
spans Y .

Proof. See Exercises 7 and 9 on Coursework 8.

We finish this section with an application of the concepts introduced so far to the
column space and the nullspace of a matrix. These subspaces are sometimes called the
fundamental subspaces of a matrix. The next theorem, a veritable gem of Linear
Algebra, shows that the fundamental subspaces of a matrix and that of its transpose are
intimately related via orthogonality:

Theorem 7.12 (Fundamental Subspace Theorem). Let A ∈ Rm×n . Then:

(a) N (A) = col(AT )⊥ .

(b) N (AT ) = col(A)⊥ .

Proof. In this proof we shall identify the rows of A (which are strictly speaking 1 × n
matrices) with vectors in Rn .
94 CHAPTER 7. ORTHOGONALITY

(a) Let x ∈ Rn . Then

x ∈ N (A) ⇐⇒ Ax = 0
⇐⇒ x is orthogonal to every row of A
⇐⇒ x is orthogonal to every column of AT
⇐⇒ x ∈ col(AT )⊥ ,

so N (A) = col(AT )⊥ .
(b) Apply (a) to AT .

7.3 Orthogonal sets

In this section we shall investigate collections of vectors with the property that each vector
is orthogonal to every other vector in the set. As we shall see, these sets have a number
of astonishing properties.
Definition 7.13. A set of vectors {u1 , . . . , ur } in Rn is said to be an orthogonal set if
each pair of distinct vectors is orthogonal, that is, if

ui ·uj = 0 whenever i 6= j .

Example 7.14. If
    
3 −1 −1
u1 = 1 ,
 u2 =  2  , u3 = −4 ,
1 1 7

then {u1 , u2 , u3 } is an orthogonal set since

u1 ·u2 = 3 · (−1) + 1 · 2 + 1 · 1 = 0
u1 ·u3 = 3 · (−1) + 1 · (−4) + 1 · 7 = 0
u2 ·u3 = (−1) · (−1) + 2 · (−4) + 1 · 7 = 0

The next theorem contains the first, perhaps surprising, property of orthogonal sets:
Theorem 7.15. If {u1 , . . . , ur } is an orthogonal set of nonzero vectors, then the vectors
u1 , . . . , ur are linearly independent.
Proof. Suppose that
c1 u1 + c2 u2 + · · · + cr ur = 0 .
Then

0 = 0·u1
= (c1 u1 + c2 u2 + · · · + cr ur )·u1
= c1 (u1 ·u1 ) + c2 (u2 · u1 ) + · · · + cr (ur ·u1 )
= c1 (u1 ·u1 ) ,

since u1 is orthogonal to u2 , . . . , ur . But since u1 is nonzero, u1 ·u1 is nonzero, so c1 = 0.

Similarly, c2 , . . . , cr must be zero, and the assertion follows.
7.4. ORTHONORMAL SETS 95

Definition 7.16. An orthogonal basis for a subspace H of Rn is a basis of H that is

also an orthogonal set.

The following theorem reveals why orthogonal bases are much ‘nicer’ than other bases
in that the coordinates of a vector with respect to an orthogonal basis are easy to compute:

Theorem 7.17. Let {u1 , . . . , ur } be an orthogonal basis for a subspace H of Rn and let
y ∈ H. If c1 , . . . , cr are the coordinates of y with respect to {u1 , . . . , ur }, that is,

y = c1 u1 + · · · + cr ur ,

then
y·uj
cj = for each j = 1, . . . , r.
uj ·uj
Proof. As in the proof of the preceding theorem, the orthogonality of {u1 , . . . , ur } implies
that
y·u1 = (c1 u1 + · · · + cr ur )·u1 = c1 (u1 ·u1 ) .
Since u1 ·u1 is not zero, we can solve for c1 in the above equation and find the stated
expression. In order to find cj for j = 2, . . . , r, compute y·uj and solve for cj .

Example 7.18. Show that the set {u1 , u2 , u3 } in Example 7.14 is an orthogonal basis
for R3 and express the vector y = (6, 1, −8)T as a linear combination of the basis vectors.

Solution. Note that by Theorem 7.15 the vectors in the orthogonal set {u1 , u2 , u3 } are
linearly independent, so must form a basis for R3 , since dim R3 = 3.
Now

y·u1 = 11 , y·u2 = −12 , y·u3 = −66 ,

u1 ·u1 = 11 , u2 ·u2 = 6 , u3 ·u3 = 66 ,

so
y·u1 y·u2 y·u3 11 −12 −66
y= u1 + u2 + u3 = u1 + u2 + u3 = u1 − 2u2 − u3 .
u1 ·u1 u2 ·u2 u3 ·u3 11 6 66

7.4 Orthonormal sets

Definition 7.19. A set {u1 , . . . , ur } of vectors in Rn is said to be an orthonormal set
if it is an orthogonal set of unit vectors. Thus {u1 , . . . , ur } is an orthonormal set if and
only if
ui ·uj = δij for i, j = 1, . . . , r ,
where (
1 if i = j
δij = .
0 if i 6= j
If H is a subspace of Rn spanned by {u1 , . . . , ur }, then {u1 , . . . , ur } is said to be an
orthonormal basis for H.
96 CHAPTER 7. ORTHOGONALITY

Example 7.20. The standard basis {e1 , . . . , en } of Rn is an orthonormal set (and also
an orthonormal basis for Rn ). Moreover, any nonempty subset of {e1 , . . . , en } is an
orthonormal set.
Here is a less trivial example:
Example 7.21. If
 √   √   
2/√6 −1/√ 3 0√
u1 = 1/√6 , u2 =  1/√3  , u3 = −1/√ 2 ,
1/ 6 1/ 3 1/ 2

then {u1 , u2 , u3 } is an orthonormal set, since

√ √ √
u1 ·u2 = −2/ 18 + 1/ 18 + 1/ 18 = 0
√ √ √
u1 ·u3 = 0/ 12 − 1/ 12 + 1/ 12 = 0
√ √ √
u2 ·u3 = 0/ 6 − 1/ 6 + 1/ 6 = 0

and

u1 ·u1 = 4/6 + 1/6 + 1/6 = 1

u2 ·u2 = 1/3 + 1/3 + 1/3 = 1
u3 ·u3 = 0/2 + 1/2 + 1/2 = 1

Moreover, since by Theorem 7.15 the vectors u1 , u2 , u3 are linearly independent and
dim R3 = 3, the set {u1 , u2 , u3 } is a basis for R3 . Thus {u1 , u2 , u3 } is an orthonormal
basis for R3 .
Matrices whose columns form an orthonormal set are important in applications, in
particular in computational algorithms. We are now going to explore some of their prop-
erties.
Theorem 7.22. An m × n matrix U has orthonormal columns if and only if U T U = I.
Proof. As an illustration of the general idea, suppose for the moment that U has only
three columns, each a vector in Rm . Write


U = u1 u2 u3 .

Then  T  T 
T T
u1 u 1 u 1 u 1 u 2 u 1 u 3
U T U = uT2  u1 u2 u3 = uT2 u1 uT2 u2 uT2 u3 

uT3 uT3 u1 uT3 u2 uT3 u3

so the (i, j)-entry of U T U is just ui ·uj and the assertion follows from the definition of
orthonormality.
The proof for the general case is exactly the same, once you have convinced yourself
that the (i, j)-entry of U T U is the dot product of the i-th column of U with the j-th
column of U .
The following theorem is a simple consequence:
Theorem 7.23. Let U ∈ Rm×n have orthonormal columns, and let x and y in Rn . Then:
7.5. ORTHOGONAL PROJECTIONS 97

(a) (U x)·(U y) = x·y;

(b) kU xk = kxk;

(c) (U x)·(U y) = 0 if and only if x·y = 0.

Proof. See Exercise 1 on Coursework 9.

Here is a rephrasing of the theorem above in the language of linear transformations.

Let U ∈ Rm×n be a matrix with orthonormal columns and let LU be the corresponding
linear transformation from Rn to Rm . Property (b) says that the mapping LU preserves
the lengths of vectors, and (c) says that LU preserves orthogonality. These properties are
important for many computer algorithms.
Before concluding this section we mention a class of matrices that fits naturally in the
present context and which will play an important role in the next chapter:

Definition 7.24. A square matrix Q is said to be orthogonal if QT Q = I.

The above considerations show that every square matrix with orthonormal columns
is an orthogonal matrix. Two other interesting properties of orthogonal matrices are
contained in the following theorem.

Theorem 7.25. Let Q ∈ Rn×n be an orthogonal matrix. Then:

(a) Q is invertible and Q−1 = QT ;

(b) if {v1 , . . . , vn } is an orthonormal basis for Rn , then {Qv1 , . . . , Qvn } is an orthonor-

mal basis for Rn .

Proof. See Exercise 2 on Coursework 9.

7.5 Orthogonal projections

In this section we shall study a particularly nice way of decomposing an arbitrary vector
in Rn . More precisely, if H is a subspace of Rn and y any vector in Rn then, as we shall
see, we can write y = ŷ + z where ŷ is in H, and z is orthogonal to H. This is a very
useful technique which has a number of interesting consequences, some of which you will
see later in this chapter.

Theorem 7.26 (Orthogonal Decomposition Theorem). Let H be a subspace of Rn . Then

each y in Rn can be written uniquely in the form

y = ŷ + z , (7.1)

where ŷ ∈ H and z ∈ H ⊥ . In fact, if {u1 , . . . , ur } is an orthogonal basis for H, then

y·u1 y·ur
ŷ = u1 + · · · + ur , (7.2)
u1 ·u1 ur ·ur

and z = y − ŷ.
98 CHAPTER 7. ORTHOGONALITY

Proof. Let ŷ be given by (7.2). Since ŷ is a linear combination of the vectors u1 , . . . , ur ,

the vector ŷ must belong to H. Let z = y − ŷ. Then
z·u1 = (y − ŷ)·u1
 
y·u1
= y·u1 − (u1 ·u1 ) − 0 − · · · − 0
u1 ·u1
= y·u1 − y·u1
= 0,
so z is orthogonal to u1 . Similarly, we see that z is orthogonal to uj for j = 2, . . . , r, so
z ∈ H ⊥ by Theorem 7.11 (b).
In order to see that the decomposition (7.1) is unique, suppose that y can also be
written as y = yˆ1 + z1 , where ŷ1 ∈ H and z1 ∈ H ⊥ . Thus ŷ + z = ŷ1 + z1 , so
ŷ − ŷ1 = z1 − z .
The above equality shows that the vector v = ŷ − ŷ1 belongs to both H and H ⊥ . Thus
v·v = 0, which implies v = 0. Therefore ŷ = ŷ1 and z = z1 , so the decomposition (7.1)
is unique.
The vector ŷ in (7.1) is called the orthogonal projection of y onto H , and is
written projH y, that is,
ŷ = projH y .
One of the reasons why orthogonal projections play an important role in Linear Alge-
bra, and indeed in other branches of Mathematics, is made plain in the following theorem:
Theorem 7.27 (Best Approximation Theorem). Let H be a subspace of Rn , y any vector
in Rn , and ŷ = projH y. Then ŷ is the closest point in H to y, in the sense that
ky − ŷk < ky − vk (7.3)
for all v ∈ H distinct from ŷ.
Proof. Take v ∈ H distinct from ŷ. Then ŷ − v ∈ H. By the Orthogonal Decomposition
Theorem, y − ŷ is orthogonal to H, so y − ŷ is orthogonal to ŷ − v.
Since
y − v = (y − ŷ) + (ŷ − v) ,
the Pythagorean Theorem (Theorem 7.8) gives
ky − vk2 = ky − ŷk2 + kŷ − vk2 .
But kŷ − vk2 > 0, since ŷ 6= v, so the desired inequality (7.3) holds.
The theorem above is the reason why the orthogonal projection of y onto H is often
called the best approximation of y by elements in H.
We conclude this section with the following consequence of the Orthogonal Decompo-
sition Theorem:
Theorem 7.28. Let H be a subspace of Rn . Then
(a) (H ⊥ )⊥ = H;
(b) dim H + dim H ⊥ = n.
Proof. See exercise on Coursework 9.
7.6. GRAM SCHMIDT PROCESS 99

7.6 Gram Schmidt process

In the previous sections we saw on a number of occasions how useful orthogonal bases
of subspaces can be. Witness, for example, the explicit expression for the orthogonal
projection onto a subspace given in the Orthogonal Decomposition Theorem. So far we
have not addressed the problem of how to manufacture an orthogonal basis. It turns out
that there is a simple algorithm that does just that, namely producing an orthogonal basis
for any nonzero subspace of Rn :
Theorem 7.29 (Gram Schmidt process). Given a basis {x1 , . . . , xr } of a subspace H of
Rn define
v1 = x1
x2 ·v1
v2 = x2 − v1
v1 ·v1
x3 ·v1 x3 ·v2
v3 = x3 − v1 − v2
v1 ·v1 v2 ·v2
..
.
xr ·v1 xr ·v2 xr ·vr−1
vr = x r − v1 − v2 − · · · − vr−1
v1 ·v1 v2 ·v2 vr−1 ·vr−1
Then {v1 , . . . , vr } is an orthogonal basis for H. In addition
Span (v1 , . . . , vk ) = Span (x1 , . . . , xk ) for 1 ≤ k ≤ r .
Proof. Write Hk = Span (x1 , . . . , xk ). Set v1 = x1 , so that Span (v1 ) = Span (x1 ). Sup-
pose that for some k < r we have already constructed v1 , . . . , vk so that {v1 , . . . , vk } is
an orthogonal basis for Hk . Define
vk+1 = xk+1 − projHk xk+1 .
By the Orthogonal Decomposition Theorem, vk+1 is orthogonal to Hk . Now the orthogo-
nal projection projHk xk+1 belongs to Hk , which in turn is a subset of Hk+1 , so vk+1 ∈ Hk+1 .
Moreover, vk+1 6= 0, since xk+1 6∈ Hk . Thus {v1 , . . . , vk+1 } is an orthogonal set of nonzero
vectors in Hk+1 . But dim Hk+1 = k + 1, so Hk+1 = Span (v1 , . . . , vk+1 ).
Remark 7.30. As with almost all the other results and techniques presented in this
module, the best way to remember the Gram Schmidt process is to understand the proof.
Here is the idea in a nut-shell: the Gram Schmidt process is an iterative procedure;
if, at some stage, the orthogonal vectors v1 , . . . , vk have already been constructed, the
next vector vk+1 is obtained by subtracting the orthogonal projection of xk+1 onto Hk =
Span (v1 , . . . vk ) from xk+1 , that is,
vk+1 = xk+1 − projHk xk+1 ,
as this makes the vector vk+1 orthogonal Hk , and thus in particular, orthogonal to all
previously constructed vectors v1 , . . . , vk .
Example 7.31. Let H = Span (x1 , x2 , x3 ) where
     
1 0 0
1 1 0
x1 = 
1 , x2 = 1 ,
   x3 = 
2 .


1 2 6
Clearly {x1 , x2 , x3 } is a basis of H. Construct an orthogonal basis of H.
100 CHAPTER 7. ORTHOGONALITY

Solution. We start by setting  

1
1
v1 = x1 = 
1 .


1
The vector v2 is constructed by subtracting the orthogonal projection of x2 onto Span (v1 )
from x2 , that is,  
−1
x2 ·v1 4 0
v2 = x2 − v1 = x2 − v1 = 0.

v1 ·v1 4
1
The vector v3 is constructed by subtracting the orthogonal projection of x3 onto Span (v1 , v2 )
from x3 , that is,
 
1
x3 ·v1 x3 ·v2 8 6 −2
v3 = x3 − v1 − v2 = x3 − v 1 − v2 =  0,

v1 ·v1 v2 ·v2 4 2
1

producing the orthogonal basis {v1 , v2 , v3 } for H.

7.7 Least squares problems

A type of problem that often arises in applications of Linear Algebra is to make sense of
an overdetermined system
Ax = b , (7.4)
where A ∈ Rm×n with m > n and b ∈ Rm . Clearly, the system (7.4) will not have a
solution for every b ∈ Rm ; in fact, as you will recall, the system has a solution if and only
if b ∈ col(A).
What do we do if we still demand a solution? The idea is to find an x ∈ Rn that makes
Ax as close as possible to b. In other words, in cases where no exact solution exists, we
think of Ax as an approximation to b. The smaller the distance between b and Ax, given
by kb − Axk, the better the approximation.
The general least squares problem is to find x ∈ Rn that makes kb − Axk as small
as possible. Here, ‘least squares’ refers to the fact that kb − Axk is the square root of a
sum of squares.

Definition 7.32. Let A ∈ Rm×n and b ∈ Rm . A least squares solution of Ax = b is

an x̂ ∈ Rn such that

kb − Ax̂k ≤ kb − Axk for all x ∈ Rn .

How do we find least squares solutions of a given system Ax = b? To motivate the

result to follow, let
b̂ = projcol(A) b .
Since b̂ is in col(A), the equation Ax = b̂ is consistent, and there is an x̂ ∈ Rn such that

Ax̂ = b̂ .
7.7. LEAST SQUARES PROBLEMS 101

By the Orthogonal Decomposition Theorem, b − b̂ is orthogonal to col(A), so

b − Ax̂ ∈ col(A)⊥ .

But by the Fundamental Subspace Theorem col(A)⊥ = N (AT ), so

b − Ax̂ ∈ N (AT ) .

Thus
AT (b − Ax̂) = 0 ,
and hence
AT Ax̂ = AT b .
To summarise what we have just said: a least squares solution of Ax = b satisfies

AT Ax = AT b . (7.5)

The matrix equation (7.5) represents a system of equations called the normal equations
for Ax = b.

Theorem 7.33. Let A ∈ Rm×n and b ∈ Rm . The set of least squares solutions of Ax = b
coincides with the non-empty solution set of the normal equations

AT Ax = AT b . (7.6)

Proof. We have just seen that a least squares solution x̂ must satisfy the normal equations.
It turns out that the argument outlined also works in the reverse direction. To be precise,
suppose that x̂ satisfies the normal equations, that is

AT Ax̂ = AT b .

Then AT (b − Ax̂) = 0, so b − Ax̂ ∈ N (AT ) = col(A)⊥ . Thus b − Ax̂ is orthogonal to

col(A). Hence the equation
b = Ax̂ + (b − Ax̂)
is a decomposition of b into a sum of a vector in col(A) and a vector orthogonal to it. By
the uniqueness of the orthogonal decomposition, Ax̂ must be the orthogonal projection
of b onto col(A). Thus, Ax̂ = b̂, and x̂ is a least squares solution.

Example 7.34. Find the least squares solution of the inconsistent system Ax = b, where
   
4 0 2
A = 0 2  b =  0  .
1 1 11

Solution. Compute  
  4 0  
T 4 0 1  17 1
A A= 0 2 = ,
0 2 1 1 5
1 1
 
  2  
T 4 0 1   19
A b= 0 = .
0 2 1 11
11
102 CHAPTER 7. ORTHOGONALITY

Thus the normal equations AT Ax = AT b are

    
17 1 x1 19
= .
1 5 x2 11
This system can (and should, in general!) be solved by Gaussian elimination. In our case,
however, it is quicker to spot that the coefficient matrix is invertible with inverse
 
T −1 1 5 −1
(A A) = ,
84 −1 17
so AT Ax = AT b can be solved my multiplying both sides with (AT A)−1 from the left,
giving the least squares solution
    
T −1 T 1 5 −1 19 1
x̂ = (A A) A b = = .
84 −1 17 11 2

Often (but not always!) the matrix AT A is invertible, and the method shown in the
example above can be used. In general, the least squares solution need not be unique, and
Gaussian elimination has to used to solve the normal equations. The following theorem
gives necessary and sufficient conditions for AT A to be invertible.
Theorem 7.35. Let A ∈ Rm×n and b ∈ Rm . The matrix AT A is invertible if and only
if the columns of A are linearly independent. In this case, Ax = b has only one least
squares solution x̂, given by
x̂ = (AT A)−1 AT b .
Proof. See Exercise 1 on Coursework 10 for the first part. The remaining assertion follows
as in the previous example.

7.8 Spectral Theorem for Symmetric Matrices

This last section of the last chapter is devoted to one of the gems of Linear Algebra:
the Spectral Theorem. This result, which has many applications, a number of which you
will see in other modules, is concerned with the diagonalisability of symmetric matrices.
Recall that a matrix A ∈ Rn×n is said to be diagonalisable if there is an invertible matrix
P ∈ Rn×n such that P −1 AP is diagonal. We already know that A is diagonalisable if and
only if A has n linearly independent eigenvectors. However, this condition is difficult to
check in practice. It may thus come as a surprise that there is a sufficiently rich class of
matrices that are always diagonalisable, and moreover, that the diagonalising matrix P
is of a special form. This is the content of the Spectral Theorem for Symmetric Matrices,
or Spectral Theorem for short:1
Theorem 7.36 (Spectral Theorem for Symmetric Matrices). Let A ∈ Rn×n be symmetric.
Then there is an orthogonal matrix Q ∈ Rn×n such that
QT AQ = D,
where D ∈ Rn×n is diagonal.
Or put differently: every symmetric matrix can be diagonalised by an orthogonal ma-
trix.
1
There are other, more general versions of the Spectral Theorem. In this course, we will only consider
the symmetric case.
7.8. SPECTRAL THEOREM FOR SYMMETRIC MATRICES 103

Reminder: A is symmetric means AT = A.

Q is orthogonal means QT Q = I.
The proof of the above Theorem is omitted here. Rather, we end this course on Linear
Algebra with an important

Corollary 7.37. The eigenvalues of a symmetric matrix A are real, and eigenvectors
corresponding to distinct eigenvalues are orthogonal.

Example 7.38. Consider the symmetric matrix

 
0 2 −1
A= 2 3 −2 .
−1 −2 0

Find an orthogonal matrix Q that diagonalises A.

Solution. The characteristic polynomial of A is

det(A − λI) = −λ3 + 3λ2 + 9λ + 5 = (1 + λ)2 (5 − λ) ,

so the eigenvalues of A are −1 and 5. Computing N (A + I) in the usual way shows that
{x1 , x1 } is a basis for N (A + I) where
   
1 −2
x1 = 0 , x2 =
   1.
1 0

Similarly, we find that the eigenspace N (A − 5I) corresponding to the eigenvalue 5 is

1-dimensional with basis  
−1
x3 = −2 .
1
In order to construct the diagonalising orthogonal matrix for A it suffices to find or-
thonormal bases for each of the eigenspaces, since, by the previous corollary, eigenvectors
corresponding to distinct eigenvalues are orthogonal.
In order to find an orthonormal basis for N (A+I) we apply the Gram Schmidt process
to the basis {x1 , x2 } to produce the orthogonal set {v1 , v2 }:
 
1
v1 = x1 = 0 ,

1
 
−1
x2 ·v1
v2 = x2 − v1 =  1.
v1 ·v1
1

Now {v1 , v2 , x3 } is an orthogonal basis of R3 consisting of eigenvectors of A, so nor-

malising them to produce
 √   √   √ 
1/ 2 −1/√ 3 −1/√6
1 1 1
u1 = v1 =  0√  , u2 = v2 =  1/√3  , u3 = x3 = −2/√ 6 ,
kv1 k kv2 k kx3 k
1/ 2 1/ 3 1/ 6
104 CHAPTER 7. ORTHOGONALITY

allows us to write down the orthogonal matrix

 √ √ √ 

1/ 2 −1/
√ 3 −1/√6
u
Q= 1 2 3 =u u  0√ 1/√3 −2/√ 6
1/ 2 1/ 3 1/ 6

which diagonalises A, that is,

 
−1 0 0
QT AQ =  0 −1 0 .
0 0 5