Math 225 Linear Algebra II Lecture Notes

John C. Bowman
University of Alberta
Edmonton, Canada

April 7, 2016
 c 2010
John C. Bowman
ALL RIGHTS RESERVED

Reproduction of these lecture notes in any form, in whole or in part, is permitted only for
nonprofit, educational use.
Contents
1 Vectors 4

2 Linear Equations 6

3 Matrix Algebra 8

4 Determinants 11

5 Eigenvalues and Eigenvectors 13

6 Linear Transformations 16

7 Dimension 17

8 Similarity and Diagonalizability 18

9 Complex Numbers 23

10 Projection Theorem 28

11 Gram-Schmidt Orthonormalization 29

12 QR Factorization 31

13 Least Squares Approximation 32

14 Orthogonal (Unitary) Diagonalizability 34

15 Systems of Differential Equations 40

16 Quadratic Forms 43

17 Vector Spaces: 46

18 Inner Product Spaces 48

19 General linear transformations 51

20 Singular Value Decomposition 53

21 The Pseudoinverse 57

Index 58
1 Vectors
Vectors in Rn :
u1 v1 0
. . .
u = .. , v = .. , 0 = .. .

un vn 0

Parallelogram law:
u1 + v1
..
u+v = . .
un + vn

Multiplication by scalar c R:

cv1
.
cv = .. .
cvn

Dot (inner) product:

uv = u1 v1 + + un vn .

Length (norm):
q
|v|= vv = v12 + + vn2 .

Unit vector:
1
v.
|v|

Distance between (endpoints of) u and v (positioned at 0):

d(u, v) = |u v|.

Law of cosines:
|u v|2 = |u|2 +|v|2 2|u||v|cos .

Angle between u and v:

1 1
|u||v|cos = (|u|2 +|v|2 |u v|2 ) = (|u|2 +|v|2 [u v][u v])
2 2
1
= (|u|2 +|v|2 [uu 2uv + vv])
2
= uv.

4
 Orthogonal vectors u and v:
uv = 0.

Pythagoras theorem for orthogonal vectors u and v:

|u + v|2 = |u|2 +|v|2 .

Cauchy-Schwarz inequality (|cos | 1):

|uv| |u||v|.

Triangle inequality:
p p
|u + v| = (u + v)(u + v) = |u|2 +2uv + |v|2
p p p
|u|2 +2|uv|+|v|2 |u|2 +2|u||v|+|v|2 = (|u|+|v|)2 = |u|+|v|.

Component of v in direction u:
u
v = |v|cos .
|u|

Equation of line:
Ax + By = C.

Parametric equation of line through p and q:

v = (1 t)p + tq.

Equation of plane with unit normal (A, B, C) a distance D from origin:

Ax + By + Cz = D.

Equation of plane with normal (A, B, C) through (x0 , y0 , z0 ):

A(x x0 ) + B(y y0 ) + C(z z0 ) = 0.

Parametric equation of plane spanned by u and w through v0 :

v = v0 + tu + sw.

5
2 Linear Equations
Linear equation:
a1 x1 + a2 x2 + + an xn = b.

Homogeneous linear equation:

a1 x1 + a2 x2 + + an xn = 0.

System of linear equations:

a11 x1 + a12 x2 + + a1n xn = b1

a21 x1 + a22 x2 + + a2n xn = b2
.. ..
. .
am1 x1 + am2 x2 + + amn xn = bm .

Systems of linear equations may have 0, 1, or an infinite number of solutions [Anton

& Busby, p. 41].

Matrix formulation:
a11 a12 a1n x1 b1

a21 a22 a2n x2 b2

.
.. .. ... .. .. = .. .
. . . .
a am2 amn xn bm
| m1 {z }
nm coefficient matrix

Augmented matrix:
a11 a12 a1n b1

a21 a22 a2n b2
.
.. .. .. .. .. .
. . . .
am1 am2 amn bm .

Elementary row operations:

Multiply a row by a nonzero constant.

Interchange two rows.

Add a multiple of one row to another.

6
Remark: Elementary row operations do not change the solution!

Row echelon form (Gaussian elimination):

1
0 0 1
. . .
.. .. .. . .
..
0 0 0 1

Reduced row echelon form (GaussJordan elimination):

1 0 0
0 0 1 0
. . .
.. .. .. . .
..
0 0 0 1

For example:

2x1 + x2 = 5,
7x1 + 4x2 = 17.

A diagonal matrix is a square matrix whose nonzero values appear only as entries aii
along the diagonal. For example, the following matrix is diagonal:

1 0 0 0
0 1 0 0
0 0 0 0 .

0 0 0 1

An upper triangular matrix has zero entries everywhere below the diagonal (aij = 0
for i > j).

A lower triangular matrix has zero entries everywhere above the diagonal (aij = 0
for i < j).

Problem 2.1: Show that the product of two upper triangular matrices of the same
size is an upper triangle matrix.

Problem 2.2: If L is a lower triangular matrix, and U is an upper triangular matrix,

show that LU is a lower triangular matrix and UL is an upper triangular matrix.

7
3 Matrix Algebra
Element aij appears in row i and column j of the mn matrix A = [aij ]mn .

Addition of matrices of the same size:

[aij ]mn + [bij ]mn = [aij + bij ]mn .

Multiplication of a matrix by a scalar c R:

c[aij ]mn = [caij ]mn .

Multiplication of matrices:
" n
#
X
[aij ]mn [bjk ]n` = aij bjk .
j=1 m`

Matrix multiplication is linear:

A(B + C) = AB + AC

for all scalars and .

Two matrices A and B are said to commute if AB = BA.

Problem 3.1: Give examples of 2 2 matrices that commute and ones that dont.

Transpose of a matrix:
[aij ]Tmn = [aji ]nm .

A symmetric matrix A is equal to its transpose.

Problem 3.2: Does a matrix necessarily compute with its transpose? Prove or
provide a counterexample.

Problem 3.3: Let A be an m n matrix and B be an n ` matrix. Prove that

(AB)T = BT AT .

8
Problem 3.4: Show that the dot product can be expressed as a matrix multiplication:
uv = uT v. Also, since uv = vu, we see equivalently that uv = v T u.

Trace of a square matrix:

n
X
Tr [aij ]nn = aii .
i=1

Problem 3.5: Let A and B be square matrices of the same size. Prove that

Tr(AB) = Tr(BA).

Identity matrix:
In = [ij ]nn .

Inverse A1 of an invertible matrix A:

AA1 = A1 A = I.

Problem 3.6: If Ax = b is a linear system of n equations, and the coefficient

matrix A is invertible, prove that the system has the unique solution

x = A1 b.

Problem 3.7: Prove that

(AB)1 = B1 A1 .

Inverse of a 2 2 matrix:
    
a b d b 1 0
= (ad bc)
c d c a 0 1

 1  
a b 1 d b
= exists if and only if ad bc 6= 0.
c d ad bc c a

Determinant of a 2 2 matrix:
   
a b a b
 = ad bc.
det = 
c d c d

9
 An elementary matrix is obtained on applying a single elementary row operation
to the identity matrix.

Two matrices are row equivalent if one can be obtained from the other by elementary
row operations.

An elementary matrix is row equivalent to the identity matrix.

A subspace is closed under scalar multiplication and addition.

span{v1 , v2 , . . . , vn } is the subspace formed by the set of linear combinations

c1 v1 + c2 v2 + + cn vn generated by all possible scalar multipliers c1 , c2 , . . . , cn .

The subspaces in R3 are the origin, lines, planes, and all of R3 .

The solution space of the homogeneous linear system Ax = 0 with n unknowns is

always a subspace of Rn .

A set of vectors {v1 , v2 , . . . , vn } in Rn are linearly independent if the equation

c1 v1 + c2 v2 + + cn vn = 0
has only the trivial solution c1 = c2 = = cn = 0. If this were not the case, we
say that the set is linearly dependent; for n 2, this means we can express at least
one of the vectors in the set as a linear combination of the others.

The following are equivalent for an n n matrix A:

(a) A is invertible;
(b) The reduced row echelon form of A is In ;
(c) A can be expressed as a product of elementary matrices;
(d) A is row equivalent to In ;
(e) Ax = 0 has only the trivial solution;
(f) Ax = b has exactly one solution for all b Rn ;
(g) The rows of A are linearly independent;
(h) The columns of A are linearly independent;
(i) det A 6= 0;
(j) 0 is not an eigenvalue of A.

10
 The elementary row operations that reduce an invertible matrix A to the identity
matrix can be applied directly to the identity matrix to yield A1 .

Remark: If BA = I then the system of equations Ax = 0 has the unique solution

x = Ix = BAx = B(Ax) = B0 = 0. Hence A is invertible and B = A1 .

4 Determinants
Determinant of a 2 2 matrix:
 
a b
 c d  = ad bc.
 

Determinant of a 3 3 matrix:
 
a b c       
  e f d f  d e 
 h i  b g i  + c  g h  .
 d e f  = a     
 
g h i 

Determinant of a square matrix A = [aij ]nn can be formally defined as:

X
det A = sgn(j1 , j2 , . . . , jn ) a1j1 a2j2 . . . anjn .
permutations
(j1 , j2 , . . . , jn ) of (1, 2, . . . , n)

Properties of Determinants:
Multiplying a single row or column by a scalar c scales the determinant by c.
Exchanging two rows or columns swaps the sign of the determinant.
Adding a multiple of one row (column) to another row (column) leaves the
determinant invariant.

Given an nn matrix A = [aij ], the minor Mij of the element aij is the determinant
of the submatrix obtained by deleting the ith row and jth column from A. The
signed minor (1)i+j Mij is called the cofactor of the element aij .

Determinants can be evaluated by cofactor expansion (signed minors), either along

some row i or some column j:
n
X n
X
det[aij ]nn = aik Mik = akj Mkj .
k=1 k=1

11
 In general, evaluating a determinant by cofactor expansion is inefficient. However,
cofactor expansion allows us to see easily that the determinant of a triangular
matrix is simply the product of the diagonal elements!

Problem 4.1: Prove that det AT = det A.

Problem 4.2: Let A and B be square matrices of the same size. Prove that
det(AB) = det(A) det(B).
.
1
Problem 4.3: Prove for an invertible matrix A that det(A1 ) = .
det A
Determinants can be used to solve linear systems of equations like
    
a b x u
= .
c d y v

Cramers rule:    
u b a u
   
v d c v
x =  , y =  .
a b a b
 
c d c d

For matrices larger than 3 3, row reduction is more efficient than Cramers rule.

An upper (or lower) triangular system of equations can be solved directly by back
substitution:
2x1 + x2 = 5,
4x2 = 17.

The transpose of the matrix of cofactors, adj A = [(1)i+j Mji ], is called the adjugate
(formerly sometimes known as the adjoint) of A.

The inverse A1 of a matrix A may be expressed compactly in terms of its adjugate

matrix:
1
A1 = adj A.
det A
However, this is an inefficient way to compute the inverse of matrices larger than
3 3. A better way to compute the inverse of a matrix is to row reduce [A|I] to
[I|A1 ].

12
 The cross product of two vectors u = (ux , uy , uz ) and v = (vx , vy , vz ) in R3 can be
expressed as the determinant
 
 i j k
 
uv =  ux uy uz  = (uy vz uz vy )i + (uz vx ux vz )j + (ux vy uy vx )k.
 vx vy vz 

The magnitude |uv|= |u||v|sin (where is the angle between the vectors u and
v) represents the area of the parallelogram formed by u and v.

5 Eigenvalues and Eigenvectors

A fixed point x of a matrix A satisfies Ax = x.

The zero vector is a fixed point of any matrix.

An eigenvector x of a matrix A is a nonzero vector x that satisfies Ax = x for

a particular value of , known as an eigenvalue (characteristic value).

An eigenvector x is a nontrivial solution to (AI)x = Axx = 0. This requires

that (A I) be noninvertible; that is, must be a root of the characteristic
polynomial det(I A): it must satisfy the characteristic equation det(I A) = 0.

If [aij ]nn is a triangular matrix, each of the n diagonal entries aii are eigenvalues
of A:
det(I A) = ( a11 )( a22 ) . . . ( ann ).

If A has eigenvalues 1 , 2 , . . ., n then det(A I) = (1 )(2 ) . . . (n ).

On setting = 0, we see that det A is the product 1 2 . . . n of the eigenvalues
of A.

Given an n n matrix A, the terms in det(I A) that involve n and n1 must

come from the main diagonal:
( a11 )( a22 ) . . . ( ann ) = n (a11 + a22 + . . . + ann )n1 + . . . .
But
det(IA) = (1 )(2 ) . . . (n ) = n (1 +2 +. . .+n )n1 +. . .+(1)n 1 2 . . . n .
Thus, Tr A is the sum 1 +2 +. . .+n of the eigenvalues of A and the characteristic
polynomial of A always has the form
n Tr An1 + . . . + (1)n det A.

13
Problem 5.1: Show that the eigenvalues and corresponding eigenvectors of the ma-
trix  
1 2
A=
3 2
are 1, with eigenvector [1, 1], and 4, with eigenvector [2, 3]. Note that the trace
(3) equals the sum of the eigenvalues and the determinant (4) equals the product
of eigenvalues. (Any nonzero multiple of the given eigenvectors are also acceptable
eigenvectors.)

Problem 5.2: Compute Av where A is the matrix in Problem 5.1 and v = [7, 3]T ,
both directly and by expressing v as a linear combination of the eigenvectors of A:
[7, 3] = 3[1, 1] + 2[2, 3].

Problem 5.3: Show that the trace and determinant of a 5 5 matrix whose charac-
teristic polynomial is det(I A) = 5 + 24 + 3 + 4 + 5 + 6 are given by 2
and 6, respectively.

The algebraic multiplicity of an eigenvalue 0 of a matrix A is the number of factors

of ( 0 ) in the characteristic polynomial det(I A).

The eigenspace of A corresponding to an eigenvalue is the linear space spanned

by all eigenvectors of A associated with . This is the null space of A I.

The geometric multiplicity of an eigenvalue of a matrix A is the dimension of the

eigenspace of A corresponding to .

The sum of the algebraic multiplicities of all eigenvalues of an n n matrix is equal

to n.

The geometric multiplicity of an eigenvalue is always less than or equal to its

algebraic multiplicity.

Problem 5.4: Show that the 2 2 identity matrix

 
1 0
0 1

has a double eigenvalue of 1 associated with two linearly independent eigenvectors,

say [1, 0]T and [0, 1]T . The eigenvalue 1 thus has algebraic multiplicity two and
geometric multiplicity two.

14
Problem 5.5: Show that the matrix
 
1 1
0 1

has a double eigenvalue of 1 but only a single eigenvector [1, 0]T . The eigenvalue 1
thus has algebraic multiplicity two, while its geometric multiplicity is only one.

Problem 5.6: Show that the matrix

2 0 0
1 3 0
3 5 3

has [Anton and Busby, p. 458]:

(a) an eigenvalue 2 with algebraic and geometric multiplicity one;

(b) an eigenvalue 3 with algebraic multiplicity two and geometric multiplicity one.

15
6 Linear Transformations
A mapping T : Rn Rm is a linear transformation from Rn to Rm if for all vectors
u and v in Rn and all scalars c:
(a) T (u + v) = T (u) + T (v);

(b) T (cu) = cT (u).

If m = n, we say that T is a linear operator on Rn .

The action of a linear transformation on a vector can be represented by matrix

multiplication: T (x) = Ax.

An orthogonal linear operator T preserves lengths: |T (x)|= |x| for all vectors xb.

1
A square matrix A with real entries is orthogonal if AT A = I; that is, A = AT .
This implies that det A = 1 and preserves lengths:

|Ax|2 = AxAx = (Ax)T Ax = xT AT Ax = xT AT Ax = xT x = xx = |x|2 .

Since the columns of an orthogonal matrix are unit vectors in addition to being
mutually orthogonal, they can also be called orthonormal matrices.

The matrix that describes rotation about an angle is orthogonal:

 
cos sin
.
sin cos

The kernel ker T of a linear transformation T is the subspace consisting of all

vectors u that are mapped to 0 by T .

The null space null A (also known as the kernel ker A) of a matrix A is the subspace
consisting of all vectors u such that Au = 0.

Linear transformations map subspaces to subspaces.

The range ran T of a linear transformation T : Rn Rm is the image of the

domain Rn under T .

A linear transformation T is one-to-one if T (u) = T (v) u = v.

16
Problem 6.1: Show that a linear transformation is one-to-one if and only if ker T = {0}.

A linear operator T on Rn is one-to-one if and only if it is onto.

A set {v1 , v2 , . . . , vn } of linearly independent vectors is called a basis for the n-

dimensional subspace

span{v1 , v2 , . . . , vn } = {c1 v1 + c2 v2 + + cn vn : (c1 , c2 , . . . , cn ) Rn }.

If w = a1 v1 + a2 v2 + . . . + vn is a vector in span{v1 , v2 , . . . , vn } with basis B =

{v1 , v2 , . . . , vn } we say that [a1 , a2 , . . . , an ]T are the coordinates [w]B of the vector w
with respect to the basis B.

If w is a vector in Rn the transition matrix PB 0 B that changes coordinates [w]B

with respect to the basis B = {v1 , v2 , . . . , vn } for Rn to coordinates [w]B 0 =
PB 0 B [w]B with respect to the basis B 0 = {v10 , v20 , . . . , vn0 } for Rn is

PB 0 B = [[v1 ]B 0 [v2 ]B 0 [vn ]B 0 ].

An easy way to find the transition matrix PB 0 B from a basis B to B 0 is to use

elementary row operations to reduce the matrix [B 0 |B] to [I|PB 0 B ].

Note that the matrices PB 0 B and PBB 0 are inverses of each other.

If T : Rn Rn is a linear operator and if B = {v1 , v2 , . . . , vn } and B 0 =

{v10 , v20 , . . . , vn0 } are bases for Rn then the matrix representation [T ]B with respect
to the bases B is related to the matrix representation [T ]B 0 with respect to the
bases B 0 according to
[T ]B 0 = PB 0 B [T ]B PBB 0 .

7 Dimension
The number of linearly independent rows (or columns) of a matrix A is known
as its rank and written rank A. That is, the rank of a matrix is the dimension
dim(row(A)) of the span of its rows. Equivalently, the rank of a matrix is the
dimension dim(col(A)) of the span of its columns.

The dimension of the null space of a matrix A is known as its nullity and written
dim(null(A)) or nullity A.

17
 The dimension theorem states that if A is an m n matrix then

rank A + nullity A = n.

If A is an m n matrix then

dim(row(A)) = rank A dim(null(A)) = n rank A,

dim(col(A)) = rank A dim(null(AT )) = m rank A.

Problem 7.1: Find dim(row) and dim(null) for both A and AT :

1 2 3 4
A= 5 6
7 8 .
9 10 11 12

While the row space of a matrix is invariant to (unchanged by) elementary row
operations, the column space is not. For example, consider the row-equivalent
matrices    
0 0 1 0
A= and B= .
1 0 0 0
Note that row(A) = span([1, 0]) = row(B). However col(A) = span([0, 1]T ) but
col(B) = span([1, 0]T ).

The following are equivalent for an m n matrix A:

(a) A has rank k;

(b) A has nullity n k;

(c) Every row echelon form of A has k nonzero rows and m k zero rows;

(d) The homogeneous system Ax = 0 has k pivot variables and n k free variables.

8 Similarity and Diagonalizability

If A and B are square matrices of the same size and B = P1 AP for some invertible
matrix P, we say that B is similar to A.

Problem 8.1: If B is similar to A, show that A is also similar to B.

18
Problem 8.2: Suppose B = P1 AP. Multiplying A by the matrix P1 on the left
certainly cannot increase its rank (the number of linearly independent rows or
columns), so rank P1 A rank A. Likewise rank B = rank P1 AP rank P1 A
rank A. But since we also know that A is similar to B, we see that rank A rank B.
Hence rank A = rank B.

Problem 8.3: Show that similar matrices also have the same nullity.

Similar matrices represent the same linear transformation under the change of bases
described by the matrix P.

Problem 8.4: Prove that similar matrices have the following similarity invariants:

(a) rank;

(b) nullity;

(c) characteristic polynomial and eigenvalues, including their algebraic and geometric
multiplicities;

(d) determinant;

(e) trace.

Proof of (c): Suppose B = P1 AP. We first note that

I B = I P1 AP = P1 IP P1 AP = P1 (I A)P.

The matrices I B and I A are thus similar and share the same nullity (and hence
geometric multiplicity) for each eigenvalue . Moreover, this implies that the matrices
A and B share the same characteristic polynomial (and hence eigenvalues and algebraic
multiplicities):
1
det(I B) = det(P1 ) det(I A) det(P) = det(I A) det(P) = det(I A).
det(P)

A matrix is said to be diagonalizable if it is similar to a diagonal matrix.

19
 An n n matrix A is diagonalizable if and only if A has n linearly independent
eigenvectors. Suppose that P1 AP = D, where

p11 p12 p1n 1 0 0

p21 p22 p2n 0 2 0
P= ... .. .. .. and D= ... .. .. . .
. . . . . ..
pn1 pn2 pnn 0 0 n

Since P is invertible, its columns are nonzero and linearly independent. Moreover,

1 p11 2 p12 n p1n

1 p21 2 p22 n p2n
AP = PD = ... .. .. .. .
. . .
1 pn1 2 pn2 n pnn

That is, for j = 1, 2, . . . , n:

p1j p1j

p2j p2j
A
... = j ... .

pnj pnj

This says, for each j, that the nonzero column vector [p1j , p2j , . . . , pnj ]T is an eigen-
vector of A with eigenvalue j . Moreover, as mentioned above, these n column
vectors are linearly independent. This established one direction of the claim. On
reversing this argument, we see that if A has n linearly independent eigenvectors,
we can use them to form an eigenvector matrix P such that AP = PD.

Equivalently, an n n matrix is diagonalizable if and only if the geometric multi-

plicity of each eigenvalue is the same as its algebraic multiplicity (so that the sum
of the geometric multiplicities of all eigenvalues is n).

Problem 8.5: Show that the matrix

 
1
0

has a double eigenvalue of but only one eigenvector [1, 0]T (i.e. the eigenvalue
has algebraic multiplicity two but geometric multiplicity one) and consequently is
not diagonalizable.

20
 Eigenvectors of a matrix A associated with distinct eigenvalues are linearly inde-
pendent. If not, then one of them would be expressible as a linear combination of
the others. Let us order the eigenvectors so that vk+1 is the first eigenvector that
is expressible as a linear combination of the others:
vk+1 = c1 v1 + c2 v2 + + ck vk , (1)
where the coefficients ci are not all zero. The condition that vk+1 is the first such
vector guarantees that the vectors on the right-hand side are linearly independent.
If each eigenvector vi corresponds to the eigenvalue i , then on multiplying by A
on the left, we find that
k+1 vk+1 = Avk+1 = c1 Av1 + c2 Av2 + + ck Avk = c1 1 v1 + c2 2 v2 + + ck k vk .
If we multiply Eq.(2) by k+1 we obtain:
k+1 vk+1 = c1 k+1 v1 + c2 k+1 v2 + + ck k+1 vk ,
The difference of the previous two equations yields
0 = c1 (1 k+1 )v1 + (2 k+1 )c2 v2 + + (k k+1 )ck vk ,
Since the vectors on the right-hand side are linearly independent, we know that
each of the coefficients ci (i k+1 ) must vanish. But this is not possible since the
coefficients ci are nonzero and eigenvalues are distinct. The only way to escape this
glaring contradiction is that all of the eigenvectors of A corresponding to distinct
eigenvalues must in fact be independent!

An n n matrix A with n distinct eigenvalues is diagonalizable.

However, the converse is not true (consider the identity matrix). In the words
of Anton & Busby, the key to diagonalizability rests with the dimensions of the
eigenspaces, not with the distinctness of the eigenvalues.

Problem 8.6: Show that the matrix

 
1 2
A=
3 2
from Problem 5.1 is diagonalizable by evaluating
   
1 3 2 1 2 1 2
.
5 1 1 3 2 1 3
What do you notice about the resulting product and its diagonal elements? What
is the significance of each of the above matrices and the factor 1/5 in front? Show
that A may be decomposed as
    
1 2 1 0 1 3 2
A= .
1 3 0 4 5 1 1

21
Problem 8.7: Show that the characteristic polynomial of the matrix that describes
rotation in the xy plane by = 90 :
 
0 1
A=
1 0

is 2 + 1 = 0. This equation has no real roots, so A is not diagonalizable over the

real numbers. However, it is in fact diagonalizable over a broader number system
known as the complex numbers C: we will see that the characteristic equation
2 + 1 = 0 admits two distinct complex eigenvalues, namely i and i, where i is
an imaginary number that satisfies i2 + 1 = 0.

Remark: An efficient way to compute high powers and fractional (or even negative
powers) of a matrix A is to first diagonalize it; that is, using the eigenvector matrix
P to express A = PDP1 where D is the diagonal matrix of eigenvalues.

Problem 8.8: To find the 5th power of A note that

A5 = PDP1 PDP1 PDP1 PDP1 PDP1

= PDP1 PDP1 PDP1 PDP1 PDP1

5 1
= PD
 P    
1 2 (1)5 0 1 3 2
=
1 3 0 45 5 1 1
   
1 1 2 1 0 3 2
=
5 1 3 0 1024 1 1
  
1 1 2048 3 2
=
5 1 3072 1 1
 
1 2045 2050
=
5 3075 3070
 
409 410
= .
615 614

Problem 8.9: Check the result of the previous problem by manually computing the
product A5 . Which way is easier for computing high powers of a matrix?

22
 1
Remark: We can use the same technique to find the square root A 2 of A (i.e. a
matrix B such that B2 = A, we will again need to work in the complex number
system C, where i2 = 1:
1 1
1
A 2 = PD
 2P   
1 1 2 i 0 3 2
=
5 1 3 0 2 1 1
  
1 i 4 3 2
=
5 i 6 1 1
 
1 4 + 3i 4 2i
= .
5 6 3i 6 + 2i

Then
1
 1
1
 1
 1 1
1 1
A A = PD P
2 2
2 PD P
2 = PD 2 D 2 P1 = PDP1 = A,

as desired.

Problem 8.10: Verify explicitly that

    
1 4 + 3i 4 2i 4 + 3i 4 2i 1 2
= = A.
25 6 3i 6 + 2i 6 3i 6 + 2i 3 2

9 Complex Numbers
In order to diagonalize a matrix A with linearly independent eigenvectors (such as a
matrix with distinct eigenvalues), we first need to solve for the roots of the charac-
teristic polynomial det(I A) = 0. These roots form the n elements of the diagonal
matrix that A is similar to. However, as we have already seen, a polynomial of
degree n does not necessarily have n real roots.
Recall that z is a root of the polynomial P (x) if P (z) = 0.

Q. Do all polynomials have at least one root z R?

A. No, consider P (x) = x2 + 1. It has no real roots: P (x) 1 for all x.

The complex numbers C are introduced precisely to circumvent this problem. If we

replace z R by z C, we can answer the above question affirmatively.
The complex numbers consist of ordered pairs (x, y) together with the usual
component-by-component addition rule (e.g. which one has in a vector space)

(x, y) + (u, v) = (x + u, y + v),

23
but with the unusual multiplication rule

(x, y) (u, v) = (xu yv, xv + yu).

Note that this multiplication rule is associative, commutative, and distributive. Since

(x, 0) + (u, 0) = (x + u, 0) and (x, 0) (u, 0) = (xu, 0),

we see that (x, 0) and (u, 0) behave just like the real numbers x and u. In fact, we
can map (x, 0) C to x R:
(x, 0) x.
Hence R C.

Remark: We see that the complex number z = (0, 1) satisfies the equation z 2 +1 = 0.
That is, (0, 1) (0, 1) = (1, 0).

Denote (0, 1) by the letter i. Then any complex number (x, y) can be represented
as (x, 0) + (0, 1)(y, 0) = x + iy.

Remark: Unlike R, the set C = {(x, y) : x R, y R} is not ordered; there is no

notion of positive and negative (greater than or less than) on the complex plane.
For example, if i were positive or zero, then i2 = 1 would have to be positive
or zero. If i were negative, then i would be positive, which would imply that
(i)2 = i2 = 1 is positive. It is thus not possible to divide the complex numbers
into three classes of negative, zero, and positive numbers.

Remark: The frequently appearingnotation 1 for i is misleading and should be

avoided, because the rule xy = x y (which one might anticipate) does not hold
for negative x and y, as the following contradiction illustrates:
p
1 = 1 = (1)(1) = 1 1 = i2 = 1.

Furthermore, by definition x 0, but one cannot write i 0 since C is not
ordered.

Remark: We may write (x, 0) = x + i0 = x since i0 = (0, 1) (0, 0) = (0, 0) = 0.

The complex conjugate (x, y) of (x, y) is (x, y). That is,

x + iy = x iy.

24
 p
The complex modulus |z| of z = x + iy is given by x2 + y 2 .

Remark: If z R then |z| = z 2 is just the absolute value of z.

We now establish some important properties of the complex conjugate. Let z =

x + iy and w = u + iv be elements of C. Then

(i)
zz = (x, y)(x, y) = (x2 + y 2 , yx xy) = (x2 + y 2 , 0) = x2 + y 2 = |z|2 ,

(ii)
z + w = z + w,

(iii)
zw = z w.

Problem 9.1: Prove properties (ii) and (iii).

Remark: Property (i) provides an easy way to compute reciprocals of complex num-
bers:
1 z z
= = 2.
z zz |z|

Remark: Properties (i) and (iii) imply that

|zw|2 = zwzw = zzww = |z|2 |w|2 .

Thus |zw| = |z| |w| for all z, w C.

The dot product of vectors u = [u1 , u2 , . . . , un ] and v = [v1 , v2 , . . . , vn ] in Cn is

given by
uv = uT v = u1 v1 + + un vn .

Remark: Note that this definition of the dot product implies that uv = vu = v T u.

Remark: Note however that uu can be written either as uT u or as uT u.

Problem 9.2: Prove for k C that (ku)v = k(uv) but u(kv) = k(uv).

25
 In view of the definition of the complex modulus, it is sensible to define the length
of a vector v = [v1 , v2 , . . . , vn ] in Cn to be
p
|v|= vv = v T v = v1 v1 + + vn vn = |v1 |2 + + |vn |2 0.

Lemma 1 (Complex Conjugate Roots): Let P be a polynomial with real coefficients.

If z is a root of P , then so is z.

Proof: Suppose P (z) = nk=0 ak z k = 0, where each of the coefficients ak are real.
P
Then
n
X n
X n
X n
X
k
P (z) = ak (z) = ak zk = ak zk = ak z k = P (z) = 0 = 0.
k=0 k=0 k=0 k=0

Thus, complex roots of real polynomials occur in conjugate pairs, z and z.

Remark: Lemma 1 implies that eigenvalues of a matrix A with real coefficients also
occur in conjugate pairs: if is an eigenvalue of A, then so if . Moreover, if x is
an eigenvector corresponding to , then x is an eigenvector corresponding to :

Ax = x Ax = x
Ax = x
Ax = x.

A real matrix consists only of real entries.

Problem 9.3: Find the eigenvalues and eigenvectors of the real matrix [Anton &
Busby, p. 528]  
2 1
.
5 2

If A is a real symmetric matrix, it must have real eigenvalues. This follows from
the fact that an eigenvalue and its eigenvector x 6= 0 must satisfy

Ax = x xT Ax = xT x = xx = |x|2
xT Ax (AT x)T x (Ax)T x (x)T x xx
= = = = = = .
|x|2 |x|2 |x|2 |x|2 |x|2

26
Remark: There is a remarkable similarity between the complex multiplication rule

(x, y) (u, v) = (xu yv, xv + yu)

and the trigonometric angle sum formulae. Notice that

(cos , sin ) (cos , sin ) = (cos cos sin sin , cos sin + sin cos )
= (cos( + ), sin( + )).

That is, multiplication of 2 complex numbers on the unit circle x2 + y 2 = 1 cor-

responds to addition of their angles of inclination to the x axis. In particular, the
mapping f (z) = z 2 doubles the angle of z = (x, y) and f (z) = z n multiplies the
angle of z by n.

These statements hold even if z lies on a circle of radius r 6= 1,

(r cos , r sin )n = rn (cos n, sin n);

this is known as deMoivres Theorem.

The power of complex numbers comes from the following important theorem.

Theorem 1 (Fundamental Theorem of Algebra): Any non-constant polynomial P (z)

with complex coefficients has a root in C.

Lemma 2 (Polynomial Factors): If z0 is a root of a polynomial P (z) then P (z) is

divisible by (z z0 ).
Pn k
Proof: Suppose z0 is a root of a polynomial P (z) = k=0 ak z of degree n.
Consider
n
X n
X n
X
P (z) = P (z) P (z0 ) = ak z k ak z0k = ak (z k z0k )
k=0 k=0 k=0
n
X
= ak (z z0 )(z k1 + z k2 z0 + . . . + z0k1 ) = (z z0 )Q(z),
k=0
Pn
where Q(z) = k=0 ak (z k1 + z k2 z0 + . . . + z0k1 ) is a polynomial of degree n 1.

Corollary 1.1 (Polynomial Factorization): Every complex polynomial P (z) of degree

n 0 has exactly n complex roots z1 , z2 , . . ., zn and can be factorized as P (z) =
A(z z1 )(z z2 ) . . . (z zn ), where A C.

Proof: Apply Theorem 1 and Lemma 2 recursively n times. (It is conventional to

define the degree of the zero polynomial, which has infinitely many roots, to be .)

27
10 Projection Theorem
The component of a vector v in the direction u
is given by

= |v|cos .
vu

The orthogonal projection (parallel component) vk of v onto a unit vector u

is
vu
u
vk = (vu) = u.
|u|2

The perpendicular component of v relative to the direction u

is

v = v (vu)
u.

The decomposition v = v + vk is unique. Notice that v u vu

= vu = 0.

In fact, for a general n-dimensional subspace W of Rm , every vector v has a unique

decomposition as a sum of a vector vk in W and a vector v in W (this is the set
of vectors that are perpendicular to each of the vectors in W ). First, find a basis
for W . We can write these basis vectors as the columns of an m n matrix A
with full column rank (linearly independent columns). Then W is the null space
for AT . The projection vk of v = vk + v onto W satisfies vk = Ax for some
vector x and hence
0 = AT v = AT (v Ax).
The vector x thus satisfies the normal equation
AT Ax = AT v, (2)

so that x = (AT A)1 AT v. We know from Question 4 of Assignment 1 that AT A

shares the same null space as A. But the null space of A is {0} since the columns of
A, being basis vectors, are linearly independent. Hence the square matrix AT A is
invertible, and there is indeed a unique solution for x. The orthogonal projection vk
of v onto the subspace W is thus given by the formula

vk = A(AT A)1 AT v.

Remark: This complicated projection formula becomes much easier if we first go to

the trouble to construct an orthonormal basis for W . This is a basis of unit vectors
that are mutual orthogonal to one another, so that AT A = I. In this case the
projection of v onto W reduces to multiplication by the symmetric matrix AAT :

vk = AAT v.

28
Problem 10.1: Find the orthogonal projection of v = [1, 1, 1] onto the plane W
spanned by the orthonormal vectors [0, 1, 0] and 45 , 0, 35 .
Compute

45 
16
0 12
4
0  1
25 25 1 25
0 1 0
vk = AAT v = 1 0 4 3 1 = 0 1 0 1 = 1 .
3 50 5 12 9 3
0 5 1 25 0 5 1 25

As desired, we note that v vk = 25 , 0, 28

 21  7
3 4

25 = 25 [3, 0, 4] is parallel to the normal 5 , 0, 5
to the plane computed from the cross product of the given orthonormal vectors.

11 Gram-Schmidt Orthonormalization
The Gram-Schmidt process provides a systematic procedure for constructing an or-
thonormal basis for an n-dimensional subspace span{a1 , a2 , . . . , an } of Rm :

1. Start with the first vector: q1 = a1 .

2. Remove the component of the second vector parallel to the first:

a2 q1
q2 = a2 q1 .
|q1 |2

3. Remove the components of the third vector parallel to the first and second:
a3 q1 a3 q2
q3 = a3 2
q1 q2 .
|q1 | |q2 |2

4. Continue this procedure by successively defining, for k = 4, 5, . . . , n:

ak q1 ak q2 ak qk1
qk = ak q 1 q 2 qk1 . (3)
|q1 |2 |q2 |2 |qk1 |2

5. Finally normalize each of the vectors to obtain the orthogonal basis {q1 , q2 , . . . , qn }.

Remark: Notice that q2 is orthogonal to q1 :

a2 q1
q2 q1 = a2 q1 |q1 |2 = 0.
|q1 |2

29
Remark: At the kth stage of the orthonormalization, if all of the previously created
vectors were orthogonal to each other, then so is the newly created vector:
ak qj
qk qj = ak qj |qj |2 = 0, j = 1, 2, . . . k 1.
|qj |2

Remark: The last two remarks imply that, at each stage of the orthonormalization,
all of the vectors created so far are orthogonal to each other!

Remark: Notice that q1 = a1 is a linear combination of the original a1 , a2 , . . . , an .

Remark: At the kth stage of the orthonormalization, if all of the previously created
vectors were a linear combination of the original vectors a1 , a2 , . . . , an , then by
Eq. (4), we see that qk is as well.

Remark: The last two remarks imply that, at each stage of the orthonormalization,
all of the vectors created so far are orthogonal to each vector qk can be written as a
linear combination of the original basis vectors a1 , a2 , . . . , an . Since these vectors
are given to be linearly independent, we thus know that qk 6= 0. This is what
allows us to normalize qk in Step 5. It also implies that {q1 , q2 , . . . , qn } spans the
same space as {a1 , a2 , . . . , an }.

Problem 11.1: What would happen if we were to apply the Gram-Schmidt procedure
to a set of vectors that is not linearly independent?

Problem 11.2: Use the Gram-Schmidt process to find an orthonormal basis for the
plane x + y + z = 0.
In terms of the two parameters (say) y = s and z = t, we see that each point on the
plane can be expressed as
x = s t, y = s, z = t.
The parameter values (s, t) = (1, 0) and (s, t) = (0, 1) then yield two linearly independent
vectors on the plane:
a1 = [1, 1, 0] and a2 = [1, 0, 1].
The Gram-Schmidt process then yields
q1 = [1, 1, 0],
[1, 0, 1][1, 1, 0]
q2 = [1, 0, 1] [1, 1, 0]
|[1, 1, 0]|2
1
= [1, 0, 1] [1, 1, 0]
 2
1 1
= , ,1 .
2 2
We note that q2 q1 = 0, as desired. Finally, we normalize the vectors to obtain q1 =
[ 12 , 12 , 0] and q2 = [ 16 , 16 , 26 ].

30
12 QR Factorization
We may rewrite the kth step (Eq. 4) of the Gram-Schmidt orthonormalization
process more simply in terms of the unit normal vectors qj :
qk = ak (ak q1 )q1 (ak q2 )q2 (ak qk1 )qk1 . (4)

On taking the dot product with qk we find, using the orthogonality of the vectors qj ,

0 6= qk qk = ak qk (5)
so that qk = (qk qk )qk = (ak qk )qk . Equation 5 may then be rewritten as

ak = q1 (ak q1 ) + q2 (ak q2 ) + + qk1 (ak qk1 ) + qk (ak qk ).

On varying k from 1 to n we obtain the following set of n equations:

a1 q1 a2 q1 an q1

0 a2 q2 an q2
[ a1 a2 . . . an ] = [ q1 q2 . . . qn ]
... .. .. .. .
. . .
0 0 an qn
If we denote the upper triangular n n matrix on the right-hand-side by R, and
the matrices composed of the column vectors ak and qk by the m n matrices A
and Q, respectively, we can express this result as the so-called QR factorization
of A:
A = QR. (6)

Remark: Every matrix m n matrix A with full column rank (linearly independent
columns) thus has a QR factorization.

Remark: Equation 6 implies that each of the diagonal elements of R is nonzero. This
guarantees that R is invertible.

Remark: If the columns of Q are orthonormal, then QT Q = I. An efficient way to

find R is to premultiply both sides of Eq. 7 by QT :

QT A = R.

Problem 12.1: Find the QR factorization of [Anton & Busby, p. 418]

1 0 0
1 1 0 .
1 1 1

31
13 Least Squares Approximation
Suppose that a set of data (xi , yi ) for i = 1, 2, . . . n is measured in an experiment and
that we wish to test how well it fits an affine relationship of the form

y = + x.

Here and are parameters that we wish to vary to achieve the best fit.
If each data point (xi , yi ) happens to fall on the line y = +x, then the unknown
parameters and can be determined from the matrix equation
1 x1 y1

y2
 
1 x2
.. .. = ... . (7)
. .

1 xn yn

If the xi s are distinct, then the n 2 coefficient matrix A on the left-hand side, known
as the design matrix, has full column rank (two). Also note in the case of exact
agreement between the experimental data and the theoretical model y = + x that
the vector b on the right-hand side is in the column space of A.

Recall that if b is in the column space of A, the linear system of equations Ax = b

is consistent, that is, it has at least one solution x. Recall that this solution is
unique iff A has full column rank.

In practical applications, however, there may be measurement errors in the entries

of A and b so that b does not lie exactly in the column space of A.

The least squares approximation to an inconsistent system of equations Ax = b is

the solution to
Ax = bk
where bk denotes the projection of b onto the column space of A. From Eq. (3) we
see that the solution vector x satisfies the normal equation

AT Ax = AT b.

Remark: For the normal equation Eq. (8), we see that

1 x1

   P 
1 1 1 x2 n
T
A A= .. .. = P
P x2i
x 1 xn . . xi xi
1 xn

32
 and
y

 1
y2
  P 
T 1 1 yi
A b= . = P
,
x1 xn .. xi y i
yn
where the sums are computed from i = 1 to n. Thus the solution to the least-
squares fit of Eq. (8) is given by
   P 1  P 
n x i y i
= P P 2 P .
xi xi xi y i

Problem 13.1: Show that the least squares line of best fit to the measured data
(0, 1), (1, 3), (2, 4), and (3, 4) is y = 1.5 + x.

The difference b = b bk is called the least squares error vector.

The least squares method is sometimes called linear regression and the line y =
+ x can either be called the least squares line of best fit or the regression line.

For each data pair (xi , yi ), the difference yi ( + xi ) is called the residual.

Remark: Since the least-squares solution x = [, ] minimizes the least squares error
vector
b = b Ax = [y1 ( + x1 ), . . . , yn ( + xn )],
Xn
the method effectively minimizes the sum [yi ( + xi )]2 of the squares of the
i=1
residuals.

A numerically robust way to solve the normal equation AT Ax = AT b is to factor

A = QR:
RT QT QRx = RT QT b.
which, since RT is invertible, simplifies to
x = R1 QT b.

Problem 13.2: Show that for the same measured data as before, (0, 1), (1, 3), (2, 4),
and (3, 4), that  
2 3
R=
0 5
and from this that the least squares solution again is y = 1.5 + x.

33
Remark: The least squares method can be extended to fit a polynomial

y = a0 + a1 x + + am x m

as close as possible to n given data points (xi , yi ):

1 x1 x21 xm a0 y1

1
2 m
1 x2 x2 x2 a1 y2
. . .. . . .. . = . .
.. .. . . . .. ..

1 xn x2n xm n am yn

Here, the design matrix takes the form of a Vandermonde matrix, which has special
properties. For example, in the case where m = n 1, its determinant is given
Yn
by (xi xj ). If the n xi values are distinct, this determinant is nonzero. The
i,j=1
i>j
system of equations is consistent and has a unique solution, known as the Lagrange
interpolating polynomial. This polynomial passes exactly through the given data
points. However, if m < n 1, it will usually not be possible to find a polynomial
that passes through the given data points and we must find the least squares
solution by solving the normal equation for this problem [cf. Example 7 on p. 403
of Anton & Busby].

14 Orthogonal (Unitary) Diagonalizability

A matrix is said to be Hermitian if AT = A.

For real matrices, being Hermitian is equivalent to being symmetric.

On defining the Hermitian transpose A = AT , we see that a Hermitian matrix A

obeys A = A.

For real matrices, the Hermitian transpose is equivalent to the usual transpose.

A complex matrix P is orthogonal (or orthonormal) if P P = I. (Most books use

the term unitary here but in view of how the dot product of two complex vectors
is defined, there is really no need to introduce new terminology.)

For real matrices, an orthogonal matrix A obeys AT A = I.

34
 If A and B are square matrices of the same size and B = P AP for some orthogonal
matrix P, we say that B is orthogonally similar to A.

Problem 14.1: Show that two orthogonally similar matrices A and B are similar.

Orthogonally similar matrices represent the same linear transformation under the
orthogonal change of bases described by the matrix P.

A matrix is said to be orthogonally diagonalizable if it is orthogonally similar to a

diagonal matrix.

Remark: An n n matrix A is orthogonally diagonalizable if and only if it has an

orthonormal set of n eigenvectors. But which matrices have this property? The
following definition will help us answer this important question.

A matrix A that commutes with its Hermitian transpose, so that A A = AA , is

said to be normal.

Problem 14.2: Show that every diagonal matrix is normal.

Problem 14.3: Show that every Hermitian matrix is normal.

An orthogonally diagonalizable matrix must be normal. Suppose

D = P AP

for some diagonal matrix D and orthogonal matrix P. Then

A = PDP ,

and
A = (PDP ) = PD P .
Then AA = PDP PD P = PDD P and A A = PD P PDP = PD DP . But
D D = DD since D is diagonal. Therefore A A = AA .

If A is a normal matrix and is orthogonally similar to U, then U is also normal:

U U = P AP P AP = P A PP AP = P A AP = P AA P = P APP A P = UU .

35
 if A is an orthogonally diagonalizable matrix with real eigenvalues, it must be
Hermitian: if A = PDP , then

A = (PDP ) = PD P = PDP = A.

Moreover, if the entries in A are also real, then A must be symmetric.

Q. Are all normal matrices orthogonally diagonalizable?

A. Yes. The following important matrix factorization guarantees this.

An n n matrix A has a Schur factorization

A = PUP ,

where P is an orthogonal matrix and U is an upper triangle matrix. This can be

easily seen from the fact that the characteristic polynomial always has at least one
complex root, so that A has at least one eigenvalue 1 corresponding to some unit
eigenvector x1 . Now use the Gram-Schmidt process to construct some orthonormal
basis {x1 , b2 , . . . , bn } for Rn . Note here that the first basis vector is x1 itself. In
this coordinate system, the eigenvector x1 has the components [1, 0, . . . , 0]T , so
that
1 1
0 0
A
... = ... .

0 0
This requires that A have the form

1

0
. . .. .
.. .. . .. .
. . .. .. ..
.. .. . . .
0

We can repeat this argument for the n 1 n 1 submatrix obtained by deleting

the first row and column from A. This submatrix must have an eigenvalue 2 corre-
sponding to some eigenvector x2 . Now construct an orthogonal basis {x2 , b3 , . . . , bn }
for Rn1 . In this new coordinate system x2 = [2 , 0, . . . , 0]T and the first column
of the submatrix appears as
2
0
. .
..
0

36
 This means that in the coordinate system {x1 , x2 , b3 , . . . , bn }, A has the form

1

0 2
. .. .. .
.. . . .. .
. .. .. .. ..
.. . . . .
0 0

Continuing in this manner we thus construct an orthogonal basis {x1 , x2 , x3 , . . . , xn }

in which A appears as an upper triangle matrix U whose diagonal values are just the
n (not necessarily distinct) eigenvalues of A. Therefore, A is orthogonally similar
to an upper triangle matrix, as claimed.

Remark: Given a normal matrix A with Schur factorization A = PUP , we have

seen that U is also normal.

Problem 14.4: Show that every normal nn upper triangular matrix U is a diagonal
matrix. Hint: letting Uij for i j be the nonzero elements of U, we can write out for

each i = 1, . . . , n the diagonal elements ik=1 Uik Uki = ik=1 Uki Uki of U U = UU :
P P

i
X n
X
|Uki |2 = |Uik |2 .
k=1 k=i

Thus A is orthogonally diagonalizable if and only if it is normal.

If A is a Hermitian matrix, it must have real eigenvalues. This follows from the
fact that an eigenvalue and its eigenvector x 6= 0 must satisfy

Ax = x x Ax = x x = xx = |x|2
x Ax (A x) x (Ax) x (x) x xx
= 2
= 2
= 2
= 2
= = .
|x| |x| |x| |x| |x|2

Problem 14.5: Show that the Hermitian matrix

 
0 i
i 0

is similar to a diagonal matrix with real eigenvalues and find the eigenvalues.

37
Problem 14.6: Show that the anti-Hermitian matrix
 
0 i
i 0
is similar to a diagonal matrix with complex eigenvalues and find the eigenvalues.
2
If A is an n n normal matrix, then |Ax|2 = x A Ax = x AA x = A x for all


vectors x Rn .
Problem 14.7: If A is a normal matrix, show that A I is also normal.
If A is a normal matrix and Ax = x, then 0 = |(A I)x|= |(A I)x|, so that
A x = x.
If A is a normal matrix, the eigenvectors associated with distinct eigenvalues are
orthogonal: if Ax = x and Ay = y, then
0 = (x Ay)T x Ay = y T A xx Ay = y T xx y = x yx y = ()xy,
so that xy = 0 whenever 6= .
An n n normal matrix A can therefore be orthogonally diagonalized by applying
the Gram-Schmidt process to each of its distinct eigenspaces to obtain a set of n
mutually orthogonal eigenvectors that form the columns of an orthonormal ma-
trix P such that AP = PD, where the diagonal matrix D contains the eigenvalues
of A.
Problem 14.8: Find a matrix P that orthogonally diagonalizes

4 2 2
2 4 2 .
2 2 4

Remark: If a normal matrix has eigenvalues 1 , 2 , . . . , n corresponding to the or-

thonormal matrix [ x1 x2 xn ] then A has the spectral decomposition

1 0 0 x1

0 2 0 x2
A = [ x 1 x 2 x n ]
... .. . . . .
. . .. ..
0 0 n xn

x1

x2
= [ 1 x1 2 x2 n xn ]
...

xn
= 1 x1 x1 + 2 x2 x2 + + n xn xn .

38
 As we have seen earlier, powers of an orthogonally diagonalizable matrix A = PDP
are easy to compute once P and U are known.

The Caley-Hamilton Theorem makes a remarkable connection between the powers

of an nn matrix A and its characteristic equation n +cn1 n1 +. . . c1 +c0 = 0.
Specifically, the matrix A satisfies its characteristic equation in the sense that
An + cn1 An1 + . . . + c1 A + c0 I = 0. (8)

Problem 14.9: The characteristic polynomial of

 
1 2
A=
3 2
is 2 3 4 = 0. Verify that the Caley-Hamilton theorem A2 3A 4I = 0 holds
by showing that A2 = 3A + 4I. Then use the Caley-Hamilton theorem to show that
 
3 2 2 25 26
A = A A = (3A + 4I)A = 3A + 4A = 3(3A + 4I) + 4A = 13A + 12I = .
39 38

Remark: The Caley-Hamilton Theorem can also be used to compute negative powers
(such as the inverse) of a matrix. For example, we can rewrite 9 as
 
1 n1 cn1 n2 c1
A A A . . . A = I.
c0 c0 c0

We can also easily compute other functions of diagonalizable matrices. Give an

analytic function f with a Taylor series
x2 x3
f (x) = f (0) + f 0 (0)x + f 00 (0) + f 000 (0) + . . . ,
2! 3!
1
and a diagonalizable matrix A = PDP , where D is diagonal and P is invertible,
one defines
A2 A3
f (A) = f (0)I + f 0 (0)A + f 00 (0) + f 000 (0) + .... (9)
2! 3!
1
On recalling that An = PDn P and A0 = I, one can rewrite this relation as
1 1
0 1
0 1 PD2 P
00 000 PD3 P
f (A) = f (0)PD P + f (0)PDP + f (0) + f (0) + ...
3! 3!
 2 3 
(10)
D D 1
= P f (0)D0 + f 0 (0)D + f 00 (0) + f 000 (0) + ... P
3! 3!
1
= Pf (D)P .
Here f (D) is simply the diagonal matrix with elements f (Dii ).

39
Problem 14.10: Compute eA for the diagonalizable matrix
 
1 2
A= .
3 2

15 Systems of Differential Equations

Consider the first-order linear differential equation
y 0 = ay,
where y = y(t), a is a constant, and y 0 denotes the derivative of y with respect to t.
For any constant c the function
y = ceat
is a solution to the differential equation. If we specify an initial condition
y(0) = y0 ,
then the appropriate value of c is y0 .

Now consider a system of first-order linear differential equations:

y10 = a11 y1 + a12 y2 + + a1n yn ,
y20 = a21 y1 + a22 y2 + + a2n yn ,
.. .. ..
. . .
0
yn = an1 y1 + an2 y2 + + ann yn ,

which in matrix notation appears as

0
y1 a11 a12 a1n y1

0
y2 a21 a22
. = . a2n y2
. ,
.. .. .. .. ..
. . . ..

0 an1 an2 ann
yn yn
or equivalently as
y 0 = Ay.

Problem 15.1: Solve the system [Anton & Busby p. 543]:

0
y1 3 0 0 y1
y20 = 0 2 0 y2
y30 0 0 5 y3
for the initial conditions y1 (0) = 1, y2 (0) = 4, y3 (0) = 2.

40
Remark: A linear combination of solutions to an ordinary differential equation is
also a solution.

If A is an n n matrix, then y 0 = Ay has a set of n linearly independent solutions.

All other solutions can be expressed as linear combinations of such a fundamental
set of solutions.

If x is an eigenvector of A corresponding to the eigenvalue , then y = t x is a

solution of y 0 = Ay:
d t
y0 = e x = et x = et Ax = Aet x = Ay.
dt

If x1 , x2 , . . ., xk , are linearly independent eigenvectors of A corresponding to the

(not necessarily distinct) eigenvalues 1 , 2 , . . ., k , then

y = e1 t x1 , y = e2 t x2 , ..., y = ek t xk

are linearly independent solutions of y 0 = Ay: if

0 = c1 e1 t x1 + c2 e2 t x2 + . . . + ck ek t xk

then at t = 0 we find
0 = c1 x1 + c2 x2 + . . . + ck xk ;
the linear independence of the k eigenvectors then implies that

c1 = c2 = . . . = ck = 0.

If x1 , x2 , . . ., xn , are linearly independent eigenvectors of an n n matrix A

corresponding to the (not necessarily distinct) eigenvalues 1 , 2 , . . ., n , then
every solution to y 0 = Ay can be expressed in the form

y = c1 e1 t x1 + c2 e2 t x2 + + cn en t xn ,

which is known as the general solution to y 0 = Ay.

On introducing the matrix of eigenvectors P = [ x1 x2 xn ], we may write

the general solution as
1 t
c1

e 0 0
2 t
0 e 0 c2

y = c1 e1 t x1 + c2 e2 t x2 + + cn en t xn = P
... .
.. . .. .. .. .
. .
0 0 en t cn

41
 Moreover, if y = y0 at t = 0 we find that
c1

c2
y0 = P
... .

cn
If the n eigenvectors x1 , x2 , . . ., xn are linearly independent, then
c1

c2
. = P1 y0 .
..
cn
The solution to y 0 = Ay for the initial condition y0 may thus be expressed as
1 t
e 0 0
0 e2 t 0
y = P
... .. .. .. P1 y0 = eAt y0 ,
. . .
0 0 en t
on making use of Eq. (11) with f (x) = ex and A replaced by At.

Problem 15.2: Find the solution to y 0 = Ay corresponding to the initial condition

y0 = [1, 1], where  
1 2
A= .
3 2

Remark: The solution y = eAt y0 to the system of ordinary differential equations

y 0 = Ay y(0) = y0

actually holds even when the matrix A isnt diagonalizable (that is, when A doesnt
have a set of n linearly independent eigenvectors). In this case we cannot use
Eq. (11) to find eAt . Instead we must find eAt from the infinite series in Eq. (10):

A2 t2 A3 t3
eAt = I + At + + + ....
2! 3!

Problem 15.3: Show for the matrix

 
0 1
A=
0 1

that the series for eAt simplifies to I + At.

42
16 Quadratic Forms
Linear combinations of x1 , x2 , . . ., xn such as
n
X
a1 x 1 + a2 x 2 + . . . + an x n = ai x i ,
i=1

where a1 , a2 , . . ., an are fixed but arbitrary constants, are called linear forms on Rn .

Expressions of the form

n
X
aij xi xj ,
i,j=1

where the aij are fixed but arbitrary constants, are called quadratic forms on Rn .

Without loss of generality, we restrict the constants aij to be the components of a

symmetric matrix A and define x = [x1 , x2 . . . xn ], so that
X
xT Ax = xi aij xj
i,j
X X X
= aij xi xj + aij xi xj + aij xi xj
i<j i=j i>j
X X X
= aij xi xj + aij xi xj + aji xj xi
i=j i<j i<j
X X
= aii x2i + 2 aij xi xj .
i=j i<j

Remark: Note that xT Ax = xAx = Axx.

(Principle Axes Theorem) Consider the change of variable x = Py, where P or-
thogonally diagonalizes A. Then in terms of the eigenvalues 1 , 2 , . . ., n of A:
xT Ax = (Py)T APy = y T (PT AP)y = 1 y12 + 2 y22 + . . . + n yn2 .

(Constrained Extremum Theorem) For a symmetric n n matrix A, the minimum

(maximum) value of
{xT Ax : |x|2 = 1}
is given by the smallest (largest) eigenvalue of A and occurs at the corresponding
unit eigenvector. To see this, let x = Py, where P orthogonally diagonalizes A,
and list the eigenvalues in ascending order: 1 2 . . . n . Then
1 = 1 (y12 +y22 +. . .+yn2 ) 1 y12 + 2 y22 + . . . + n yn2 n (y12 +y22 +. . .+yn2 ) = n .
| {z }
T
x Ax

43
Problem 16.1: Find the minimum and maximum values of the quadratic form

z = 5x2 + 4xy + 5y 2

subject to the constraint x2 + y 2 = 1 [Anton & Busby p. 498].

Definition: A symmetric matrix A is said to be

positive definite if xT Ax > 0 for x 6= 0;

negative definite if xT Ax < 0 for x 6= 0;
indefinite if xT Ax exhibits positive and negative values for various x 6= 0.

Remark: If A is a symmetric matrix then xT Ax

is positive definite if and only if all eigenvalues of A are positive;

is negative definite if and only if all eigenvalues of A are negative;
is indefinite if and only if A has at least one positive eigenvalue and at least one
negative eigenvalue.

The kth principal submatrix of an n n matrix A is the submatrix composed of

the first k rows and columns of A.

(Sylvesters Criterion) If A is a symmetric matrix then A is positive definite if and

only if the determinant of every principal submatrix of A is positive.

Remark: Sylvesters criterion is easily verified for a 2 2 symmetric matrix

 
a b
A= ;
b d

if a > 0 and ad > b2 then d > 0, so both the trace a + d and determinant are
positive, ensuring in turn that both eigenvalues of A are positive.

Remark: If A is a symmetric matrix then the following are equivalent:

(a) A is positive definite;

(b) there is a symmetric positive definite matrix B such that A = B2 ;
(c) there is an invertible matrix C such that A = CT C.

44
Remark: If A is a symmetric 2 2 matrix then the equation xT Ax = 1 can be
expressed in the orthogonal coordinate system y = PT x as 1 y12 + 2 y22 = 1. Thus
xT Ax represents

an ellipse if A is positive definite;

no graph if A is negative definite;

a hyperbola if A is indefinite.

A critical point of a function f is a point in the domain of f where either f is not

differentiable or its derivative is 0.

The Hessian H(x0 , y0 ) of a twice-differentiable function f is the matrix of second

partial derivatives  
fxx (x0 , y0 ) fxy (x0 , y0 )
.
fyx (x0 , y0 ) fyy (x0 , y0 )

Remark: If f has continuous second-order partial derivatives at a point (x0 , y0 ) then

fxy (x0 , y0 ) = fyx (x0 , y0 ), so that the Hessian is a symmetric matrix.

Remark (Second Derivative Test): If f has continuous second-order partial deriva-

tives near a critical point (x0 , y0 ) of f then

f has a relative minimum at (x0 , y0 ) if H is positive definite.

f has a relative maximum at (x0 , y0 ) if H is negative definite.

f has a saddle point at (x0 , y0 ) if H is indefinite.

anything can happen at (x0 , y0 ) if det H(x0 , y0 ) = 0.

Problem 16.2: Determine the relative minima, maxima, and saddle points of the
function [Anton & Busby p. 498]
1
f (x, y) = x3 + xy 2 8xy + 3.
3

45
17 Vector Spaces:
A vector space V over R (C) is a set containing an element 0 that is closed under a
vector addition and scalar multiplication operation such that for all vectors u, v,
w in V and scalars c, d in R (C) the following axioms hold:
(A1) (u + v) + w = u + (v + w) (associative);

(A2) u + v = v + u (commutative);

(A3) u + 0 = 0 + u = u (additive identity);

(A4) there exists an element (u) V such that u + (u) = (u) + u = 0

(additive inverse);

(A5) 1u = u (multiplicative identity);

(A6) c(du) = (cd)u (scalar multiplication);

(A7) (c + d)u = cu + du (distributive);

(A8) c(u + v) = cu + cv (distributive).

Problem 17.1: Show that axiom (A2) in fact follows from the other seven axioms.

Remark: In addition to Rn , other vector spaces can also be constructed:

Since it satisfies axioms A1A8, the set of m n matrices is in fact a vector space.

The set of n n symmetric matrices is a vector space.

The set Pn of polynomials a0 + a1 x + . . . + an xn of degree less than or equal to n

(where a0 , a1 , . . . , an are real numbers) is a vector space.

The set P of all polynomials is an infinite-dimensional vector space.

The set F (R) of functions defined on R form an infinite-dimensional vector space

if we define vector addition by

(f + g)(x) = f (x) + g(x) for all x R.

and scalar multiplication by

(cf )(x) = cf (x) for all x R.

46
 The set C(R) of continuous functions on R is a vector space.

The set of differentiable functions on R is a vector space.

The set C 1 (R) of functions with continuous first derivatives on R is a vector space.

The set C m (R) of functions with continuous mth derivatives on R is a vector space.

The set C (R) of functions with continuous derivatives of all orders on R is a vector
space.

Definition: A nonempty subset of a vector space V that is itself a vector space under
the same vector addition and scalar multiplication operations is called a subspace
of V .

A nonempty subset of a vector space V is a subspace of V iff it is closed under

vector addition and scalar multiplication.

Remark: The concept of linear independence can be carried over to infinite-dimensional

vector spaces.

If the Wronskian
 f1 (x) f2 (x) fn (x) 
 
 f10 (x) f20 (x) fn0 (x) 
 
W (x) =  .. .. .. .. 
 . . . . 

 f (n1) (x) f (n1) (x) f (n1) (x) 
1 2 n

of functions f1 (x), f2 (x), . . ., fn (x) in C n1 (x) is nonzero for some x R, then

these n functions are linearly independent on R. This follows on observing that if
there were constants c1 , c2 , . . ., cn such that the function
g(x) = c1 f1 (x) + c2 f2 (x) + . . . + cn fn (x)
is zero for all x R, so that g (k) (x) = 0 for k = 0, 1, 2, . . . n 1 for all x R, then
the linear system
f1 (x) f2 (x) fn (x) c1

0
f10 (x) f20 (x) fn0 (x) c2 0
.. .. ... .. . = .
.. ..

. . .
(n1) (n1) (n1)
f1 (x) f2 (x) fn (x) cn 0
would have a nontrivial solution for all x R.

47
Problem 17.2: Show that the functions f1 (x) = 1, f2 (x) = ex , and f3 (x) = e2x are
linearly independent on R.

18 Inner Product Spaces

An inner product on a vector space V over R (C) is a function that maps each pair
of vectors u and v in V to a unique real (complex) number hu, vi such that, for
all vectors u, v, and w in V and scalars c in R (C):
(I1) hu, vi = hv, ui (conjugate symmetry);
(I2) hu + v, wi = hu, wi + hv, wi (additivity);
(I3) hcu, vi = c hu, vi (homogeneity);
(I4) hv, vi 0 and hv, vi = 0 iff v = 0 (positivity).

The usual Euclidean dot product hu, vi = uv = uT v provides an inner product

on Cn .

Problem 18.1: If w1 , w2 , . . ., wn are positive numbers, show that

hu, vi = w1 u1 v1 + + wn un vn
provides an inner product on the vector space Rn .

Every inner product on Rn can be uniquely expressed as hu, vi = uT Av for some

positive definite symmetric matrix A, so that hv, vi = v T Av > 0 for 6= 0.

In a vector
p space V with inner product hu, vi, the norm |v| p of a vector v is
given by hv, vi, the distance d(u, v) is given by |u v|= hu v, u vi,
and the angle between two vectors u and v satisfies cos = hu, vi. Two vec-
tors u and v are orthogonal if hu, vi = 0. Analogues of the Pythagoras theorem,
Cauchy-Schwarz inequality, and triangle inequality follow directly from (I1)(I4).

Problem 18.2: Show that

Z b
hf, gi = f (x)g(x) dx
a

defines an inner product on the vector space C[a, b] of continuous functions on [a, b],
with norm s
Z b
|f |= f 2 (x) dx.
a

48
 The Fourier theorem states than an orthonormal basis for the infinite-dimensional
vector space
R of differentiable periodic functions on [, ] with inner product
c
hf, gi = f (x)g(x) dx is given by {un }n=0 = { 2 , c1 , s1 , c2 , s2 , . . .}, where
0

1
cn (x) = cos nx

and
1
sn (x) = sin nx.

The orthonormality of this basis follows from the trigonometric addition formulae

sin(nx mx) = sin nx cos mx cos nx sin mx

and
cos(nx mx) = cos nx cos mx sin nx sin mx,
from which we see that
sin(nx + mx) sin(nx mx) = 2 cos nx sin mx, (11)

cos(nx + mx) + cos(nx mx) = 2 cos nx cos mx (12)

cos(nx mx) cos(nx + mx) = 2 sin nx sin mx (13)

Integration of Eq. (12) from to for distinct non-negative integers n and m

yields
Z  
cos(nx + mx) cos(nx mx)
2 cos nx sin mx dx = + = 0,
n+m nm

since the cosine function is periodic with period 2. When n = m > 0 we find
Z  
cos(2nx)
2 cos nx sin nx dx = = 0,
2n

Likewise, for distinct non-negative integers n and m, Eq. (13) leads to

Z  
sin(nx + mx) sin(nx mx)
2 cos nx cos mx dx = + = 0,
n+m nm

but when n = m > 0 we find, since cos(nx nx) = 1,

Z  
2 sin(2nx)
2 cos nx dx = +x = 2.
2n
R
Note when n = m = 0 that cos2 nx dx = 2.

49
 Similarly, Eq. (14) yields for distinct non-negative integers n and m,
Z
2 sin nx sin mx dx = 0,

but Z  
2 sin(2nx)
2 sin nx dx = x = 2.
2n

A differentiable periodic function f on [, ] may thus be represented exactly by

its infinite Fourier series:

X
f (x) = hf, un i un
n=0
 
c0 c
0 +
X
= f, [hf, cn i cn + hf, sn i sn ]
2 2 n=1

a0 X
= + [an cos nx + bn sin nx],
2 n=1

in terms of the Fourier coefficients

1
Z
1
an = hf, cn i = f (x) cos nx dx (n = 0, 1, 2, . . .),

and Z
1 1
bn = hf, sn i = f (x) sin nx dx (n = 1, 2, . . .).

Problem 18.3: By using integration by parts, show that the Fourier series for f (x) =
x on [, ] is

X (1)n+1 sin nx
2 .
n=1
n
For x (, ) this series is guaranteed to converge to x. For example, at x = /2,
we find
2m+2
 
(1) sin (2m + 1)
(1)m
2
X X
4 =4 = .
m=0
2m + 1 m=0
2m + 1

50
Problem 18.4: By using integration by parts, show that the Fourier series for f (x) =
|x| on [, ] is

4 X cos(2m 1)x
.
2 m=1 (2m 1)2
This series can be shown to converge to |x| for all x [, ]. For example, at
x = 0, we find

X 1 2
2
= .
m=1
(2m 1) 8

Remark: Many other concepts for Euclidean vector spaces can be generalized to
function spaces.

The functions cos nx and sin nx in a Fourier series can be thought of as the
eigenfunctions of the differential operator d2 /dx2 :

d2
y = n2 y.
dx2

19 General linear transformations

A linear transformation T : V W from one vector space to another is a mapping
that for all vectors u, v in V and all scalars satisfies c

(a) T (cu) = cT (u);

(b) T (u + v) = T (u) + T (v);

Problem 19.1: Show that every linear transformation T satisfies

(a) T (0) = 0;

(b) T (u v) = T (u) T (v);

Problem 19.2: Show that the transformation that maps each n n matrix to its
trace is a linear transformation.

Problem 19.3: Show that the transformation that maps each n n matrix to its
determinant is not a linear transformation.

51
 The kernel of a linear transformation T : V W is the set of all vectors in V that
are mapped by T to the zero vector.

The range of a linear transformation T is the set of all vectors in W that are the
image under T of at least one vector in V .

Problem 19.4: Given an inner product space V containing a fixed nonzero vector u,
let T (x) = hx, ui for x V . Show that the kernel of T is the set of vectors that
are orthogonal to u and that the range of T is R.

Problem 19.5: Show that the derivative operator, which maps continuously differ-
entiable functions f (x) to continuous functions f 0 (x), is a linear transformation
from C 1 (R) to C(R).

Problem 19.6: Show that the antiderivative operator,R which maps continuous func-
x
tions f (x) to continuously differentiable functions 0 f (t) dt, is a linear transfor-
mation from C(R) to C 1 (R).

Problem 19.7: Show that the kernel of the derivative operator on C 1 (R) is the set
of constant functions on R and that its range is C(R).

Problem 19.8: Let T be a linear transformation T . Show that T is one-to-one if

and only if ker T = {0}.

Definition: A linear transformation T : V W is called an isomorphism if it is

one-to-one and onto. If such a transformation exists from V to W we say that V
and W are isomorphic.

Every real n-dimensional vector space V is isomorphic to Rn : given a basis {u1 , u2 , . . . , un },

we may uniquely express v = c1 u1 + c2 u2 + . . . + cn un ; the linear mapping
T (v) = (c1 , c2 , . . . , cn ) from V to Rn is then easily shown to be one-to-one and
onto. This means V differs from Rn only in the notation used to represent vectors.

For example, the set Pn of polynomials of degree n is isomorphic to Rn+1 .

52
 Let T : Rn Rn be a linear operator and B = {v1 , v2 , . . . , vn } be a basis for Rn .
The matrix
A = [[T (v1 )]B [T (v2 )]B [T (vn )]B ]
is called the matrix for T with respect to B, with

[T (x)]B = A[x]B

for all x Rn . In the case where B is the standard Cartesian basis for Rn , the ma-
trix A is called the standard matrix for the linear transformation T . Furthermore,
if B 0 = {v10 , v20 , . . . , vn0 } is any basis for Rn , then

[T (x)]B 0 = PB 0 B [T (x)]B PBB 0 .

20 Singular Value Decomposition

Given an m n matrix A, consider the square Hermitian n n matrix A A. Each
eigenvalue associated with an eigenvector v of A A must be real and non-negative
since

0 |Av|2 = AvAv = (Av) Av = v A Av = v v = vv = |v|2 .

Problem 20.1: Show that null(A A) = null(A) and conclude from the dimension theorem
that rank(A A) = rank(A).

Since A A is Hermitian, it can be orthogonally diagonalized as A A = VDV ,

where the diagonal matrix D contains the n (non-negative) eigenvalues j2 of A A
listed in decreasing order (taking j 0) and the column vectors of the matrix
V = [ v1 v2 . . . vn ] are a set of orthonormal eigenvectors for A A. Now if A has
rank k, so does D since it is similar to A A, which we have just seen has the same
rank as A. In other words 1 2 k > 0 but k+1 = k+2 = = n = 0.
Since
Avi Avj = (Avi ) Avj = vi A Avj = vi j2 vj = j2 vi vj ,
we see that the vectors
Avj Avj
uj = = , j = 1, 2, . . . , k
|Avj | j

are also orthonormal, so that the matrix

[ u1 u2 . . . uk ]

53
 is orthogonal. We also see that Avj = 0 for k < j n. We can then extend this set
of vectors to an orthonormal basis for Rm , which we write as the column vectors
of an m m matrix

U = [ u1 u2 . . . uk . . . um ].

The product U and the m n matrix

1 0 0 0 0

0 2 0 0 0
. .. . . . .. ..
.. . . .. . .

= 0 0 k 0 0 ,
0 0 0 0
. .. .. .. .. .. ..
.. . . . . . .
0 0 0 0 0
is the m n matrix
U = [ 1 u1 2 u2 k uk 0 0 ]
= [ Av1 Av2 Avk Avk+1 Avn ]
= AV.

Since the n n matrix V is an orthogonal matrix, we then obtain a singular value

decomposition of the m n matrix A:

A = UV .

The positive values j for j = 1, 2, . . . k are known as the singular values of A.

Remark: By allowing the orthogonal matrices U and V to be different, the concept

of orthogonal diagonalization can thus be replaced by a more general concept that
applies even to nonnormal matrices.

Remark: The singular value decomposition plays an important role in numerical

linear algebra, image compression, and statistics.

Problem 20.2: Show that the singular values of a positive-definite Hermitian matrix
are the same as its eigenvalues.

Problem 20.3: Find a singular value decomposition of [Anton & Busby p. 505]
 
3 2
A= .
0 3

54
Problem 20.4: Find a singular value decomposition of [Anton & Busby p. 509]

1 1
A = 0 1 .
1 0

We find
  1 1  
1 0 1 2 1
AT A = 0 1 = ,
1 1 0 1 2
1 0
which has characteristic polynomial (2 )2 1 = 2 4 + 3 = ( 3)( 1). The matrix
AT A has eigenvalues 3 and 1; their respective eigenvectors
" 1 # " 1 #

2 2
and
1 12
2

form the columns of V.

The singular values of A are 1 = 3 and 2 = 1 and
2

1 1 " 1 # 3
1
u1 = 0 1 12 = 16 ,

3 1
0 2 1
6
0

1 1 " 1
#
1
u2 = 0 1 2
= 2 .
12 1
1 0
2
We nowlook for a third vector
which is perpendicular to scaled versions of u1 and u2 ,
namely 6u1 = [2, 1, 1] and 2u2 = [0, 1, 1]. That is, we want to find a vector in the null
space of the matrix  
2 1 1
,
0 1 1
or equivalently, after row reduction,
 
1 0 1
.
0 1 1
We then see that [1, 1, 1] lies in the null space of this matrix. On normalizing this vector,
we find 1
3
1
u3 = 3 .
1
3
A singular value decomposition of A is thus given by

2 0 1
1 1 3 3 3 0 " 1 1
#
1 1 1
0 1 = 0 1 12 2
.
12
6 2 3

1 0 1 1 1 0 0 2
6 2 3

55
Remark: An alternative way of computing a singular value decomposition of the
nonsquare matrix in Problem 20.4 is to take the transpose of a singular value
decomposition for  
T 1 0 1
A = .
1 1 0

The first k columns of the matrix U form an orthonormal basis for col(A), whereas
the remaining m k columns form an orthonormal basis for col(A) = null(AT ).

The first k columns of the matrix V form an orthonormal basis for row(A), whereas
the remaining n k columns form an orthonormal basis for row(A) = null(A).

If A is a nonzero matrix, the singular value decomposition may be expressed more

efficiently as A = U1 1 V1 , in terms of the m k matrix
U1 = [ u1 u2 . . . uk ],
the diagonal k k matrix
1 0 0

0 2 0
1 =
... .. . . . .. ,
. .
0 0 k
and the n k matrix
V1 = [ v1 v2 . . . vk ].
This reduced singular value decomposition avoids the superfluous multiplications by
zero in the product UV .

Problem 20.5: Show that the matrix 1 is always invertible.

Problem 20.6: Find a reduced singular value decomposition for the matrix

1 1
A = 0 1
1 0
considered previously in Problem 20.4.

The singular value decomposition can equivalently be expressed as the singular

value expansion
A = 1 u1 v1 + 2 u2 v2 + + k uk vk .
In contrast to the spectral decomposition, which applies only to Hermitian matri-
ces, the singular value expansion applies to all matrices.

56
 If A is an n n matrix of rank k, then A has the polar decomposition

A = UV = UU UV = PQ,

where P = UU is a positive semidefinite matrix of rank k and Q = UV is an

n n orthogonal matrix.

21 The Pseudoinverse
If A is an n n matrix of rank n, then we know it is invertible. In this case,
the singular value decomposition and the reduced singular value decomposition
coincide:
A = U1 1 V1 ,
where the diagonal matrix 1 contains the n positive singular values of A. More-
over, the orthogonal matrices U1 and V1 are square, so they are invertible. This
yields an interesting expression for the inverse of A:

A1 = V1 1
1 U1 ,

where 1
1 contains the reciprocals of the n positive singular values of A along its
diagonal.

Definition: For a general nonzero m n matrix A of rank k, the n m matrix

A+ = V1 1
1 U1

is known as the pseudoinverse of A. It is also convenient to define 0+ = 0, so that

the pseudoinverse exists for all matrices. In the case where A is invertible, then
A1 = A+ .

Remark: Equivalently, if we define + to be the m n matrix obtained by replacing

all nonzero entries of with their reciprocals, we can define the pseudoinverse
directly in terms of the unreduced singular value decompostion:

A+ = V+ U .

Problem 21.1: Show that the pseudoinverse of the matrix in Problem 20.4 is [cf.
Anton & Busby p. 520]: 1 1 2

3
3 3
.
1 2 1
3 3
3

57
Problem 21.2: Prove that the pseuodinverse A+ of an m n matrix A satisfies the
following properties:

(a) AA+ A = A;

(b) A+ AA+ = A+ ;

(c) A AA+ = A ;

(d) A+ A and AA+ are Hermitian;

(e) (A+ ) = (A )+ .

Remark: If A has full column rank, then A A is invertible and property (c) in
Problem 21.2 provides an alternative way of computing the pseudoinverse:

1
A+ = A A A .

An important application of the pseudoinverse of a matrix A is in solving the

least-squares problem
A Ax = A b
since the solution x of minimum norm can be conveniently expressed in terms of
the pseudoinverse:
x = A+ b.

58
Index
C, 23 design matrix, 32
span, 10 Determinant, 11
QR factorization, 31 diagonal matrix, 7
diagonalizable, 19
Addition of matrices, 8 dimension theorem, 18
additive identity, 46 Distance, 4
additive inverse, 46 distance, 48
additivity, 48 distributive, 46
adjoint, 13 domain, 16
adjugate, 13 Dot, 4
affine, 32 dot product, 25
Angle, 4
angle, 48 eigenfunctions, 51
associative, 46 eigenspace, 15
Augmented matrix, 6 eigenvalue, 13
axioms, 46 eigenvector, 13
eigenvector matrix, 20
basis, 17 elementary matrix, 10
Caley-Hamilton Theorem, 39 Elementary row operations, 7
Cauchy-Schwarz inequality, 5 Equation of line, 5
change of bases, 19 Equation of plane, 5
characteristic equation, 14 first-order linear differential equation, 40
characteristic polynomial, 14 fixed point, 13
cofactor, 12 Fourier coefficients, 50
commutative, 46 Fourier series, 50
commute, 8 Fourier theorem, 49
Complex Conjugate Roots, 26 full column rank, 28
complex modulus, 25 fundamental set of solutions, 41
complex numbers, 22, 23 Fundamental Theorem of Algebra, 27
Component, 5
component, 28 GaussJordan elimination, 7
conjugate pairs, 26 Gaussian elimination, 7
conjugate symmetry, 48 general solution, 41
consistent, 32 Gram-Schmidt process, 29
coordinates, 17
Cramers rule, 12 Hermitian, 34
critical point, 45 Hermitian transpose, 34
cross product, 13 Hessian, 45
homogeneity, 48
deMoivres Theorem, 27 Homogeneous linear equation, 6

59
Identity, 9 null space, 16
imaginary number, 22 nullity, 17
indefinite, 44
initial condition, 40 one-to-one, 16, 17
inner, 4 onto, 17
inner product, 48 ordered, 24
Inverse, 9 orthogonal, 16, 34, 48
invertible, 9 orthogonal change of bases, 35
isomorphic, 52 orthogonal projection, 28
isomorphism, 52 Orthogonal vectors, 5
orthogonally diagonalizable, 35
kernel, 16, 52 orthogonally similar, 35
orthonormal, 16, 34
Lagrange interpolating polynomial, 34
orthonormal basis, 28
Law of cosines, 4
least squares, 32 parallel component, 28
least squares error vector, 33 Parallelogram law, 4
least squares line of best fit, 33 Parametric equation of line, 5
Length, 4 Parametric equation of plane, 6
length, 26 perpendicular component, 28
linear, 8 polar decomposition, 57
Linear equation, 6 Polynomial Factorization, 27
linear forms, 43 positive definite, 44
linear independence, 47 positivity, 48
linear operator, 16 pseudoinverse, 57
linear regression, 33 Pythagoras theorem, 5
linear transformation, 16, 51
linearly independent, 10 quadratic forms, 43
lower triangular matrix, 7
range, 16, 52
Matrix, 6 rank, 17
matrix, 8 real, 26
matrix for T with respect to B, 53 reciprocals, 25
matrix representation, 17 Reduced row echelon form, 7
minor, 12 reduced singular value decomposition, 56
Multiplication by scalar, 4 regression line, 33
Multiplication of a matrix by a scalar, 8 relative maximum, 45
Multiplication of matrices, 8 relative minimum, 45
multiplicative identity, 46 residual, 33
root, 23
negative definite, 44 rotation, 16
norm, 4, 48 Row echelon form, 7
normal, 35 row equivalent, 10
normal equation, 28

60
saddle point, 45
scalar multiplication, 46
Schur factorization, 36
similar, 18
similarity invariants, 19
singular value decomposition, 54
singular value expansion, 56
singular values, 54
spectral decomposition, 38
standard matrix, 53
subspace, 10, 47
symmetric matrix, 8
System of linear equations, 6

Trace, 9
transition matrix, 17
Transpose, 8
Triangle inequality, 5
triangular system, 13

Unit vector, 4
unitary, 34
upper triangular matrix, 7

Vandermonde, 34
vector space, 46
Vectors, 4

Wronskian, 47

61