MA1522 Notes

compiled by:
m. zaidan
qianbo dong

for:
Academic Year 2023-2024, Semester 2

Contents

1 Linear Systems 4
1.1 Elementary Row Operations . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Matrices 5
2.1 Matrix multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Representation of linear system . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Inverses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3.1 Properties of invertible matrices . . . . . . . . . . . . . . . . . . . 6
2.3.2 Finding the inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4 LU Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.5 Partial pivoting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.6 Elementary matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.7 Determinant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 Vector Spaces 11
3.1 Linear span . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2 Solution space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.3 Linear independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.4 Basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.5 Transition matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.6 Row & column space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1
 3.7 Finding a basis/extending basis . . . . . . . . . . . . . . . . . . . . . . . 14
3.8 Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.9 Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.10 Nullspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.10.1 Dimension theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4 Orthogonality 18
4.1 Orthogonality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.2 Projection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.3 Gram-Schmidt Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.4 Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.5 QR Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.6 Orthogonal Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

5 Eigenvalues, eigenvectors & eigenspaces 30

5.1 Eigenvalues & eigenvectors . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.2 Characteristic equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.3 Eigenspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

6 Diagonalization 33
6.1 Diagonalizable matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
6.2 Algebraic and geometric multiplicity . . . . . . . . . . . . . . . . . . . . . 33
6.3 Criterion of diagonalizability . . . . . . . . . . . . . . . . . . . . . . . . . . 33
6.4 Algorithm to find diagonalization . . . . . . . . . . . . . . . . . . . . . . . 34

7 Orthogonal Diagonalization 35
7.1 P −1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
7.2 Orthogonally diagonalizable . . . . . . . . . . . . . . . . . . . . . . . . . . 35
7.3 Algorithm to find orthogonal diagonalization . . . . . . . . . . . . . . . . 36
7.4 Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . 36

2
8 Linear Transformations 38
8.1 Linearity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
8.2 Change of Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
8.3 Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
8.4 Ranges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
8.5 Kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

3
1 Linear Systems

1.1 Elementary Row Operations

Definition
1. Type I: Multiply a row by non-zero constant
2. Type II: Interchange two rows
3. Type III: Add a constant multiple of a row to another row

Inversing EROs

1. Type I: Divide scalar number

2. Type II: Interchange
3. Type III: Minus

E E
A −→
1
· → ... −−→
k
B
E −1 E −1
A ←−1−− · ← ... ←−k−− B

Ek .....E1 A = B

A = E1−1 ...Ek−1 B

4
2 Matrices

2.1 Matrix multiplication

Definition
AB is the m × n matrix, such that its (i, j)−entry is

∑
p
ai1 b1j + ... + aip bpj = aik bkj
k=1

To find the entry

1. Extract ith row of A

2. Extract j th column of b
3. Multiply componentwise
4. Add products

AB: pre-multiplication of A to B.
BA: post-multiplication of A to B.

2.2 Representation of linear system

Ax = b

A: coefficient matrix.
x: variable matrix (solution)
b: constant matrix

5
2.3 Inverses

Definition
Given a square matrix of order n A,
if there exists square matrix B of order n

AB = BA = In

A is invertible, B is an inverse of A. If A is not invertible, it is singular.

2.3.1 Properties of invertible matrices

A square matrix A of order n is invertible ⇐⇒

1. A is row equivalent to In
2. A has n pivot columns (in REF)
3. Ax = 0 has only the trivial solution x = 0
4. Columns of A are linearly independent
5. x → Ax is one-to-one:
Ax = Ay =⇒ x = y
6. For each column vector b ∈ Rn , Ax = b has an unique solution.
7. Columns of A span Rn
8. x → Ax is surjection:
∀T : Rn → Rm , all elements in Rm can be expressed as Ax in Rn
9. There exists square matrix of order n B such that BA = In
10. There exists square matrix of order n B such that AB = In
11. AT is invertible
12. Columns of A are a basis for Rn
13. Column space of A = Rn
14. Dimension of column space of A = n
15. Rank of A = 0

6
 16. Null space of A = {0}
17. 0 is not an eigenvalue of A.
18. det(A) 6= 0
19. Row space of A is Rn
20. c 6= 0, (cA)−1 = 1c A−1

2.3.2 Finding the inverse

Gauss-Jordan Elimination
1. Write matrix in form A|I
2. RREF to get I|A−1

Adjoint Matrix

1
A−1 = adj(A)
det(A)

2.4 LU Decomposition

Definition
If exclusively Type III operations are used in Gaussian elimination for matrix A,
there exists
A = LU

L: lower-triangular matrix L with diagonal entries 1. U : row-echelon form of A

Finding matrices L and U

1. A = LU
2. Find U by getting the REF of A.
3. L is the inverse EROs (A = E1−1 ...Ek−1 U, L = E1−1 ...Ek−1 )

7
 Solving linear systems with LU decomposition

If A = LU , to solve Ax = b

1. We get LU x = b
2. Substituting y = U x, we get Ly = b, where Ly is lower triangular
3. Solve for y, by solving (L | b)
4. Solve for x, by solving (U | y)

2.5 Partial pivoting

Definition
If interchanging of rows is needed (Type III operations),

E −1 E E −1
A ←−1−− · ← ...· ←−
i
· ← ...· ←−k−− U, Ei : Ri ↔ Rj

A = E1−1 ...Ei ...Ek−1 U

Ei A = (Ei E1−1 ...)Ei ...Ek−1 U

P = Ei , L = Ei E1−1 ...Ei ...Ek−1

For every matrix A, there exists,

• A permutation matrix P : Product of type II elementary matrices

The matrix of all the Type II EROs
• L: lower-triangular matrix L with diagonal entries 1.
• U : row-echelon form of A

such that P A = LU

8
2.6 Elementary matrices

Definition
A matrix obtained by doing an ERO on an identity matrix.

2.7 Determinant

2 by 2 matrix
( )
a b
Let A =
c d
det(A) = |A| = ad − bc

3 by 3 matrix
 
a b c
 
Let A = 
d e f 

g h i

det(A) = |A| = a(ei − f h) − b(di − f g) + c(dh − eg)

Elementary matrix of order 3

Let A be a square matrix of order 3, and E be an elementary matrix of order 3.

det(EA) = det(E)det(A)

If A is invertible, then R = I, det(R) = 1,

det(A) = det(E1−1 )....det(Ek−1 )

Cofactor expansion

Let Aij denote (i, j)−cofactor of A.

det(A) = ai1 Ai1 + ..... + ain Ain

9
General strategy:

Let A = (aij )n×n .

• If A has a zero row/zero column, then det(A) = 0.

• If A is triangular, det(A) = aii ...ann .
• Otherwise:

– If n = 2, det(A) = a11 a22 − a12 a21

– If row or column has many 0 entries, use cofactor.
– Use Gaussian elimination:
det(A) = (−1)t det(R), t is number of type II operations.

10
3 Vector Spaces

3.1 Linear span

Definition
Let S = {v1 ..., vk } be a subset of Rn .
The linear span is the set of all linear combinations

span(S) = {c1 v1 + ....ck vk | c1 , ...ck ∈ R}

3.2 Solution space

Definition
The solution space of a homogenous linear system of n variables is the solu-
tion set.

3.3 Linear independence

Definition
Let S = {v1 ..., vk } be a subset of Rn .
Given the equation c1 v1 + ... + ck vk = 0:

• Non-trivial solution exists

– S is a linearly dependent set

– v1 , ..., vk are linearly dependent.

• Non-trivial solution does not exist

– S is a linearly independent set

– v1 , ..., vk are linearly independent.

• S is linearly dependent ⇐⇒ some vi is a linear combination of other vectors in

S

11
 • S is linearly independent ⇐⇒ no vector in S can be written as a linear combi-
nation of other vectors

3.4 Basis

Definition
Let S be a subset of vector space V .
S is a basis if it is the minimal subset of V where span(S) = V

To check if it is a basis
It is sufficient to show two of these conditions to prove S is a basis for V :

1. S is linearly independent
2. span(S) = V
3. dim(V ) = |S|

3.5 Transition matrices

Definition
Coordinate vector, (v)S is a row vector that represents
the scalar coefficient of the basis, S, of a vector space V to represent vector v
relative to basis, S.

v = c1 v1 + c2 v2 + ... + ck vk

(v)S = (c1 , c2 , ..., ck )

Coordinate vector, [v]S is the column vector form of (v)S

i.e. [v]S = (v)TS
Suppose the column vectors of A forms a basis for vector space V , then
A[v]S = v

12
Definition
Transition matrix, P is a matrix that transitions a coordinator vector, [v]S , rel-
ative to basis S to a coordinator vector, [v]T , relative to basis T.
Essentially, Switching the representation of a point in a vector space V from a
particular basis S to another particular basis T.
Suppose S and T are basis for vector space V, P [w]S = [w]T

Properties of Transition Matrix

Transition matrix is transitive.

Suppose S1 , S2 , S3 are bases for vector space V and P and Q be the transitive
matrix from S1 to S2 and from S2 to S3 respectively. Then,

P Q
[v]S1 → [v]S2 → [v]S3

[v]S3 = QP [v]S1

P −1 is the transition matrix from S2 tp S1

P P −1
[v]S1 → [v]S2 → [v]S1

Transition matrix from S to T

Used when span(S) = span(T ) = vector space, V

Let S = {u1 , u2 , u3 } and T = {v1 , v2 , v3 }

Gauss−Jordan
(T | S) = (v1 v2 v3 | u1 | u2 | u3 ) −−−−−−−−−→ (I|P )
Elimination

13
3.6 Row & column space

Definition
Row space is the vector space spanned by the rows.
Let r1 , .... be the rows of A.

Row space = span{r1 , ...}

Column space is the vector space spanned by the cols.

Let c1 , .... be the column of A.

Column space = span{c1 , ...}

• row space of A = column space of AT

• column space of A = row space of AT

3.7 Finding a basis/extending basis

Using row vectors given a set of vectors

Let V = span{v1 , ...., vk }

1. View each vi as a row vector

 
v
 1

2. Form a matrix  ... 

vk
3. Perform Gaussian elimination to get REF R
4. Non-zero rows of REF are the basis of the vector space

14
Using column vectors given a set of vectors

Let V = span{v1 , ...., vk }

1. View each vi as a column vector

( )
2. Form a matrix v1 ... vk
3. Perform Gaussian elimination to get REF R
4. Columns in V that correspond to the pivot columns in R are a basis
Note: The pivot columns in R form a basis for the column space of R

Extending a basis

Used when span(S) is smaller than Rn

Let S = {v1 , ..., vk }

1. Gaussian elimination on R to reduce it to row-echelon form.

• Remove vectors that are linearly dependent

• Note columns that are not pivot

2. Add rows to R such that columns are all pivot

15
3.8 Consistency

Consistency

Let b ∈ Rm

Ax = b is consistent ⇐⇒ Av = b for some v ∈ Rn

⇐⇒ b is in the column space of A

Using dimension,
let R be a row echelon form of A.
Thus, row echelon form of (A | b) is of the form (R | b′ )

Ax = b is consistent ⇐⇒ b′ is non pivot in (R | b′ )

⇐⇒ rank(R) = rank(R | b′ )
⇐⇒ rank(A) = rank(A | b)

3.9 Rank

Definition
Let R be a row echelon form of A.

dim(row space of A) = number of nonzero rows of R

dim(column space of A) = number of pivot columns of R

rank(A) = dim(column space of A) = dim(row space of A)

Let A be a m × n matrix.

• rank(A) = rank(AT )
• rank(A) = 0 =⇒ A = 0
• rank(A) ≤ min(m, n)
• rank(A) = min(m, n) =⇒ A is full rank

16
 • A full rank ⇐⇒ A invertible

3.10 Nullspace

Definition
Let A be a m × n matrix. The nullspace of A is the solution space of Ax = 0.

{v ∈ Rn | Av = 0}

nullity(A) = dim(nullspace of A)

Let R be a row-echelon form of A.

nullity(A) = nullity(R)
= the number of non-pivot columns of R

3.10.1 Dimension theorem

Dimension Theorem
Let A be a m × n matrix.

rank(A) + nullity(A) = n

Properties of product of matrix multiplication

Let A be a m × n matrix and B be an n × p matrix.

• column space of AB ⊆ column space of A

• row space of AB ⊆ row space of A
• rank(AB) ≤ min(rank(A), rank(B))

17
4 Orthogonality

4.1 Orthogonality

Definition
Dot Product (inner product) of u and v is defined as:
( )
( ) v
1
u · v = u1 v 1 + u2 v 2 = u1 u2
v2

= uvT when u, v are viewed as row vectors

= uT v when u, v are viewed as column vectors

When u · v is 0, u and v are perpendicular/orthogonal, denoted by u ⊥ v.
(Special Case) When u = 0, 0 · v = 0 ⇒ 0 is orthogonal to every vector v ∈ Rn
Angle between u and v is given by
u·v
cos θ = , where u, v 6= 0
||u||||v||

Length of a vector, u is given as:

√ √
||u|| = u21 + ... + u2n = u · u, u ∈ Rn

When ||u|| = 1, u is a unit vector.

18
Properties of dot product

• Commutative Law: u · v = v · u
• Distributive law: (u + v) · w = u · w + v · w = w · (u + v)
• Scalar: (cu) · v = u · (cv) = c(u · v), c ∈ R
• ||cv|| = |c|||v||
• u·u≥0
• u·u=0⇔u=0
• |u · v| ≤ ||u|| ||v|| (Cauchy-Schwaz ineqality)
• ||u + v|| ≤ ||u|| + ||v||
• d(u, w) ≤ d(u, v) + d(v, w) where d(s, e) represents distance from s to e

Definition
Let S = {v1 , ..., vk } be a subset of Rn
S is orthogonal if every pair of distinct vectors in S are orthogonal:
vi · vj = 0 , ∀i 6= j
S is orthonormal if every distinct vector in S is pairwise orthogonal and a unit
vector 
0 i 6= j
vi · vj =
1 i=j

• S orthonormal ⇒ S orthogonal
• S orthogonal ⇒ T ⊆ S is orthogonal
• S orthonormal ⇒ T ⊆ S is orthonormal
• S orthogonal ⇒ S ∪ 0 is orthogonal
• S orthonormal ⇒ 0 ∈
/S

19
Definition
Normalizing is the process of converting an orthogonal set of nonzero vectors
in Rn , S = u1 , .., uk to an orthonormal set of vectors, T = v1 , .., vk .

ui
ui 7→ vi = , ui ∈ S, vi ∈ T
||ui ||

Check if a set of vectors is orthogonal/orthonormal

Let A ={v1 , v2 , ..., vk } be a subset of Rn .

diagonal matrix if A is orthogonal
AT A =
I if A is orthonormal
k

Linear Independance in orthogonal sets

• An orthogonal set of nonzero vectors is linearly independent.

• An orthonormal set of vectors is linearly independent

20
 Orthogonal sets as basis

Let V = span(S), where S is a set of nonzero vectors in Rn

• S is orthogonal set ⇒ S is a orthogonal basis for V

• S is orthonormal set ⇒ S is a orthonormal basis for V

Checking if S is a basis for V is made simpler as now you have to only check
one of the following:

• |S| = dim(V )
• span(S) = V

The condition that S is linearly indepedent is covered as S is either a nonzero

orthogonal set of vectors or a orthonormal set of vectors.

Let S = {u1 , ..., uk } be a basis for a vector space V.

∀w ∈ V, w = c1 u1 + ... + ck uk

(w)S = (c1 , ..., ck ), the coordinate vector of w relative to S

w · ui w · ui
ci = =
ui · ui ||ui ||2
Thus, finding the solution, [w]S can be found easily.

4.2 Projection

Definition
Projection is the vector that is imposed onto a vector v by another vector u.

v v u·v v u·v
The project vector, p = ||p|| = ||u||cosθ = ||u|| = v
||v|| ||v|| ||u||||v|| ||v|| ||v||2

If v is a unit vector, then p = (u · v)v

21
Projection on orthogonal basis

Suppose S = v1 , ..., vk be an orthonormal basis for vector space V

The projection of w on V is

(w · v1 )v1 + (w · v2 )v2 + ... + (w · vk )vk

Suppose S = u1 , ..., uk be an orthogonal basis for vector space V

The projection of w on V is
u1 u2 uk
(w · )u1 + (w · )u2 + ... + (w · )uk
||u1 || 2 ||u2 || 2 ||uk ||2

Projection on vector space

Suppse Ax = b where the column space of A is V

Method 1:

1. Find a set S such that V = span(S) ⇒ S = column vectors of A

2. Use Gram-Schmidt process to convert S to an orthogonal basis for V
3. Find projection p of b onto V

Method 2:

1. Let A be a matrix whose column space is V

2. Find a least squared solution u to Ax = b using AT Ax = AT b
3. The projection p = Au where u is the solution found in the previous step

22
4.3 Gram-Schmidt Process

Definition
Gram-Schmidt Process allows us to find an orthogonal basis from a given
basis for a vector space.

Let u1 , u2 , ..., uk be a basis for a vector space V

v 1 = u1
u2 · v 1
v 2 = u2 − v1
||v1 ||2
..
.
u2 · v 1 uk · vk−1
v k = uk − v1 − ... − vk−1
||v1 || 2 ||vk−1 ||2
{v1 , v2 , ..., vk } is an orthogonal basis for V.

After normalizing, vi 7→ wi = v1
||v1 ||
{w1 , w2 , ..., wk } is an orthonormal basis for V.

23
4.4 Least Squares

Least Squared Solutions

If Ax = b is inconsistent, its least squared solution, u, is Au = p where p is the

projection of b onto V

Method 1:
Steps:

1. Set the columns space of A as V

2. Use Gram-Schmidt process to find the orthogonal basis for V
3. Find the projection of b onto V
4. Solve for Ax = p, going to RREF

Method 2:
Solve the system AT Ax = AT b
The method is guaranteed to get a solution regardless of consistency of sys-
tem.

Remark: From Method 2: x = (AT A)−1 (AT b).

This can be used to find the projection of w on a vector space V given the basis of
V , which would be A.
Remark: Given two least-squared solutions v1 , v2 , if A has an eigenvalue 0,

AT Av1 =AT b
AT Av2 =AT b
AT A(v1 − v2 ) =AT (b − b)
AT A(v1 − v2 ) =0

v1 − v2 is thus in the nullspace of AT A, making it an eigenvector associated to 0.

Remark: Least-squared solutions are non-unique, unless the matrix A is linearly
independent.

24
4.5 QR Decomposition

QR Decomposition

Let A be a m × n matrx whose column are linearly independent. Then there

exist:

• an m × n matrix Q whose column form a orthonormal set

• an invertiable upper triangluar matrix R of order n

Steps:

1. Use Gram-Schmidt process to obTain an orthonormal basis

{w1 , w2 , ..., wn } for the column space of A
( )
2. Q = w1 w2 ... wn
 
w 1 · u1 w 1 · u2 . . . w 1 · un
 
 0 w · u . . . w · u 
 2 2 2 n 
3. R = . .. .. .. 
 .. . . . 
 
0 0 . . . w n · un

Application:
Suppose A has linearly independent columns with QR decomposition QR

Ax = b ⇔ QRx = b ⇔ Rx = QT b

where u is the least squares solution to Rx = QT b which is also the least squares
solution to Ax = b

25
Full QR Decomposition

Used when the basis may not be independent.

Suppose V = span(S), S = {u1 , u2 , ..., uk } where S may not be independent.

⇒ Gram-Schmidt process generates an orthonormal basis by ignoring the zero
vector, 0.

v 1 = u1
u2 · v 1
v 2 = u2 − v1 = 0, Vector is ignored
||v1 ||2
u3 · v 1
v 3 = u3 − v1
||v1 ||2
..
.

After normalizing, vi 7→ wi = v1
||v1 ||
{w1 , w3 , ...} is an orthonormal basis.
Since v2 is a zero vector, we need to extend the orthonormal basis by choosing
a arbituary vector (preferably elementary vectors such as e1 = (1, 0, 0)) and
process it through the above Gram-Schmidt process to obtain a hopefully non-
zero orthogonal basis and normalise it to obtain the extended orthonormal
basis.

Let A be an m × n matrix with m ≥ n. Then there exist

• An orthogonal matrix Q of order m

• An invertible upper triangular matrix R′ of order n
( )
R‘
such that A = QR, where R = is an m × n matrix
0

26
Let {u1 , u2 , ..., um } be the columns of A

1. Use Gram-Schmidt process to find an orthogonal basis.

2. Extend it to an orthonormal basis {w1 , ..., wm } for Rm if necessary.
( )
3. Q = w1 . . . wm
 
w 1 · u1 w 1 · u2 . . . w 1 · un
 
 0 w 2 · u2 . . . w 2 · un 
 
4. R‘ = . . . .  a upper triangular matrix.
 .. .. .. .. 
 
0 0 . . . w n · un

27
4.6 Orthogonal Matrices

Definition

Orthogonal Matrix is a square matrix where AT A = I or equivalently A−1 = AT

Properties:
Let A and B be orthogonal matricies of the same order

(AB)T (AB) = B T AT AB = I

AB is a orthogonal matrix ( )
Let S = {v1 , ..., vk } ⊆ Rn and A = v1 . . . vk

AT A = Ik ⇔ S is an orthonormal set

If k = n, then A is a square matrix and |S| = dim(Rn ) ⇒ A is orthogonal

⇔ AT A = In ⇔ S is an orthonormal basis for Rn

Let A now be a m × n matrix

AT A = In ⇔ columns of A form an orthonormal set in Rm
AAT = In ⇔ rows of A form an orthonormal set in Rn

Specialised to a orthogonal matrix, A, of order n:

⇔ columns of A form an orthonormal set in Rn
⇔ rows of A form an orthonormal set in Rn

28
Premultiplication of Orthogonal matrix with another matrix
( )
Let A = v1 . . . vk where v1 to vk represent the column vectors of A,
( )
P A = P v1 . . . P v k
Then, (P A)T (P A) = AT P T P A = I ⇒ {P u1 , ..., P uk } is an orthonormal set

If A consist of orthonormal set of column vectors, then P A is also a orthonor-

mal set of vectors
Essentially, the pre-multiplication of an orthogonal matrix converts an or-
thonormal set to another orthonormal set.

Transition matrix of orthogonal bases

Let A = {u1 , u2 , u3 } and B = {v1 , v2 , v3 } be orthonormal bases for V. Let w ∈ V

w = A[w]S = B[w]T

[w]T = B T B[w]T = B T A[w]S ⇒ P = B T A is transition matrix from S to T.

[w]S = AT A[w]S = AT B[w]T ⇒ Q = AT B is transition matrix from T to S.

P −1 = Q Both P and Q are orthogonal matrix.

∴ The transition matrix between orthonormal bases is an orthogonal matrix

29
5 Eigenvalues, eigenvectors & eigenspaces

5.1 Eigenvalues & eigenvectors

Definition
Let A be a square matrix of order n.
Suppose that for some λ ∈ R and nonzero v ∈ Rn :

Av = λv

Then,

• λ is an eigenvalue of A.
• v is an eigenvector of A associated to the eigenvalue λ.

5.2 Characteristic equation

λ ∈ R is an eigenvalue of A ⇐⇒ Av = λv for some 0 6= v ∈ Rn

⇐⇒ λIv − Av = 0 for some 0 6= v ∈ Rn
⇐⇒ (λI − A)v = 0 for some 0 6= v ∈ Rn
⇐⇒ λI − A is singular
⇐⇒ det(λI − A) = 0

Additionally,

0 is not an eigenvalue of A ⇐⇒ A is an invertible matrix

Remark: Note that AT A has the same nullspace as A.

Thus, if A has an eigenvalue of 0, the eigenvector v associated to eigenvalue 0 is in
the nullspace of both AT A and A.

30
 Main Theorem for Invertible Matrices
A is an invertible matrix, if and only if any (and hence, all) of the following hold:

• columns of A form a basis for Rn

• rank(A) = n
• nullity(A) = 0
• 0 is not an eigenvalue of A.

Definition
The characteristic polynomial of A:

det(λI − A)

The characteristic equation of A:

det(λI − A) = 0

Eigenvalues are the roots of the characteristic equation.

Special case for triangular matrices

Eigenvalues of a triangular matrix are its diagonal entries.

5.3 Eigenspace

Let A be a square matrix of order n, and let λ be an eigenvalue of A.

For any nonzero vector v ∈ Rn ,

v is an eigenvector of A associated to λ ⇐⇒ Av = λv
⇐⇒ (λI − A)v = 0
⇐⇒ v ∈ nullspace of (λI − A)

31
Definition
Let λ be an eigenvalue of a square matrix A.

The eigenspace of A associated to λ, denoted EA,λ or Eλ , is the nullspace of

(λI − A).
Note: (λI − A) is singular ∴ dim(Eλ ) ≥ 1

Eigenvectors

Let λ be an eigenvalue of a square matrix A.

The eigenvectors of A associated to the eigenvalue λ are precisely all nonzero

vectors in the eigenspace Eλ .

32
6 Diagonalization

6.1 Diagonalizable matrices

Definition
A square matrix A is diagonalizable if
there is an invertible matrix P where P −1 AP is a diagonal matrix.

6.2 Algebraic and geometric multiplicity

Definition
Algebraic multiplicity of λ, denoted a(λ) is its multiplicity as a root of the
characteristic polynomial of A.
e.g: If the characteristic equation is (λ − 1)3 = 0, a(1) = 3

Geometric multiplicity of λ, denoted g(λ) is the dimension of Eλ .

Algebraic & geometric multiplicity

Let A be a square matrix, λ be an eigenvalue of A. Then,

1 ≤ g(λ) ≤ a(λ)

6.3 Criterion of diagonalizability

Diagonalization Theorem

Let A be a square matrix of order n.

A is diagonalizable ⇐⇒ A has n linearly independent eigenvectors.

⇐⇒ sum of geometric multiplicities of eigenvalues of A = n
⇐⇒ sum of algebraic multiplicities of eigenvalues of A = n
and for each eigenvalue λ, g(λ) = a(λ).

33
If a square matrix A has n distinct eigenvalues, then A is diagonalizable.
Remark: The inverse is not true. If A is diagonalizable, A might not have n distinct
eigenvalues.

Remark: The diagonal matrix D is not unique unless A only has one eigenvalue.

Remark: For this course, diagonalizable = diagonalizable over R

6.4 Algorithm to find diagonalization

Algorithm

Let A be a square matrix of order n.

1. det(λI − A) cannot be completely factorized in R

• A is not diagonalizable.

2. det(λI − A) can be completely factorized in R

• Let λ1 , ...λk be distinct eigenvalues of A.

• Find basis Si for eigenspace Eλi .
(a) g(λi ) < a(λi ) for some i
– A is not diagonalizable.
(b) g(λi ) = a(λi ) for all i
– A is diagonalizable.

34
7 Orthogonal Diagonalization

7.1 P −1

Given P −1 AP = D, the inverse of P , P −1 is required for computations such as:

• Gauss-Jordan elimination: (P |I) → (I|P −1 )

• Adjoint matrix: P −1 = 1
det(P ) adj(P )

Remark: P is orthogonal =⇒ P −1 = P T

7.2 Orthogonally diagonalizable

Definition
A square matrix A is orthogonally diagonalizable if there exists an orthogonal
matrix P such that P T AP is a diagonal matrix.

All orthogonally diagonalizable matrices are symmetric

Let A be a square matrix.

A is orthogonally diagonalizable =⇒ A symmetric

P T AP = D = DT = (P T AP )T = P T AT P

Remark: Given that A is a symmetric matrix with all real entries,

• eigenvalues are all real

• for each eigenvalue λ of A, g(λ) = a(λ) = dim(Eλ )
• eigenvectors associated to distinct eigenvalues are orthogonal.

35
7.3 Algorithm to find orthogonal diagonalization

Algorithm

Let A be a symmetric matrix of order n.

1. Find all eigenvalues of A by solving det(λI − A) = 0

2. For each eigenvalue λi of A,

(a) Solve (λi I = A)x = 0 to find basis Si for eigenspace Eλi

(b) Use Gram-Schmidt to convert Si to orthonormal basis Ti .

3. Ti ∪ .. ∪ Tk = {w1 , ..., wn } is an orthonormal basis for Rn .

• P = {w1 , ..., wn } orthogonally diagonalizes A.

7.4 Singular Value Decomposition

Non-negativity of eigenvalues of AT A

If λ was an eigenvalue of AT A, and v was an associated eigenvector:

AT Av = λv =⇒ v T AT Av = v T λv =⇒ ||Av||2 = λ||v||2 ≥ 0

Thus,
let A be a m × n matrix.

The eigenvalues of AT A are nonnegative.

Definition
Let A be a m × n matrix, and λ1 , ... be its eigenvalues.
√
σ1 = λ1 ......

These values are the singular values of A.

36
Algorithm: Singular Value Decomposition

Let A be an m × n matrix.

1. Find eigenvalues of AT A λ1 > ... > λr + 1

2. Find corresponding orthonormal set of eigenvectors of AT A {v1 ....vn }.
√
3. Let σi = λi , and ui = σ1i Avi for i = 1, ..., r
4. Extend {ui , ..., ur } to orthonormal basis {ui , ..., um } for Rm

Let U = {u1 ...um }, V = {v1 ....vn }. U, V are orthogonal matrices.

 
σ1 ... 0 ... 0
 
 ... ... ... ... 0
∑   

Let =  0 ... σr ... 0, a rectangular diagonal matrix.
 
 ... ... ... ... 0
 
0 ... 0 ... 0
∑
Then, A = U V T is the singular value decomposition of A.

37
8 Linear Transformations

Definition
Linear tranformation can be seen as a mapping f , from domain of Rn to a
codomain of Rm

A mapping T : Rn 7→ R can be defined by:

f (x1 , x2 , ..., xn ) = a1 x1 + a2 x2 + . . . + an xn

is a linear transformation from Rn to R

A linear transformation can be viewed in the matrix form:

T (x) = Ax, x ∈ Rn

A is called the standard matrix for T

Remark: T is called a linear operator on Rn if dimension of domain = dimension of

codomain

Linear operator

Identity operator is the mapping T : Rn to Rn where I(x) = x for x ∈ Rn

I(x) = x = In x ⇒ In is the standard matrix for I(x)

Zero transformation is the mapping T : Rn to Rm where )(x) = 0 for x ∈ Rn

O(x) = O = 0m×n 0 ⇒ 0m×n is the standard matrix for O(x)

Remark: T : Rn to Rn has a unique standard transformation

To show that a function T : Rn to Rm is a linear transformation, it suffices to find an

m × n matrix A such that T (x) = Ax ∀x ∈ Rn

38
8.1 Linearity

Linearity

Suppose T : Rn to Rm is a linear transformation and A is the standard matrix of

T , i.e T (x) = Ax.

• T (0) = A0 = 0
• T (cv) = A(cv) = c(Av) = cT (v)
• T (u + v) = A(u + v) = Au + Av = T (u) + T (v)
• For any v1 , ..., vk ∈ Rn and c1 , ..., ck ∈ R,

T (c1 v1 + ... + ck vk ) = A(c1 v1 + ... + ck vk )

= A(c1 v1 ) + ... + A(ck vk )

= c1 (Av1 ) + ... + ck (Avk )

= c1 T (v1 ) + ... + ck T (vk )

which is the linear transformation of images

T (c1 v1 + ... + ck vk ) is completely determined by T (v1 ), ..., T (vk )

Determine mapping T is NOT a linear transformation

To show that a mapping T is not a linear transformation

• Show that T (0) 6= 0

• Find v ∈ Rn and c ∈ R such that T (cv) 6= cT (v)
• Find v ∈ Rn and c ∈ R such that T (u + v) 6= T (u) + T (v)

39
8.2 Change of Bases

Change of basis

Suppose S = {v1 , .., vn } be a basis for Rn

∀v ∈ Rn , (v)S = (c1 , ...,(cn ) )
v = c1 v1 + ... + cn vn = v1 . . . vn [v]S = P [v]S
Suppose T : Rn to Rm is a linear transformation

T (v) = c1 T (v1 ) + ... + cn T (vn )

( )
= T (v1 ) . . . T (vn ) [v]S

= B[v]S

= AP [v]S

B = AP ⇔ A = BP −1

To find the standard matrix of T :

Gauss−Jordan
(P |I) −−−−−−−−−→ (I|P −1 )
Elimination
The standard matrix for T would be BP −1

40
Suppose T : Rn to Rm and S1 = {v1 , .., vn } and S2 = {u1 , .., un } are bases for Rn
( )
T (w) = B[w]S1 = T (v1 ) . . . T (vn ) [w]S1
( )
= C[w]S2 = T (u1 ) . . . T (un ) [w]S2

Let P be the transition matrix from S1 tp S2

P [w]S1 = [w]S2 ⇒ CP [w]S1 = C[w]S2 = T (w) = B[w]S1

B = CP

If S2(= E is the standard

) basis,
P = u1 . . . un and P −1 is the transition matrix from E to S
C = A which is the standard matrix of A
B = AP ⇔ A = BP −1 ( )
Suppose T : Rn 7→ Rn and S = {v1 , .., vn } represented by P as v1 . . . vn . P
is invertible.
v = P [v]S

T (v) = P [T (v)]S and Av = AP [v]S

P [T (v)]S = A[v]S ⇒ [T (v)]S = P −1 AP [v]S

T can be represented by [v]S 7→ B[v]S where B = P −1 AP . A and B are similar.

A square matrix is diagonalisable ⇔ it is similar to a diagonal matrix.

41
8.3 Composition

Definition

Let S: Rn 7→ Rm and T : Rm 7→ Rk be linear transformations

Let T ◦ S: Rn 7→ Rk ,
Composition of T with S: (T ◦ S)(u) = T (S(u)) ∀u ∈ Rn
Remark: T ◦ S 6= S ◦ T
Let A and B be the standard matrix for the above mapping S and T respec-
tively.
The standard matrix of T ◦ S = BA

8.4 Ranges

Definition
Let T : Rn 7→ Rm
The range of T is the set of all images of T : R(T ) = {T (v)|v ∈ Rn } ⊆ Rm

R(T ) = span(T (v)) = span(T (v1 ), ..., T (vn )) where v1 , ..., vn is any basis for Rn
( )
Since T has a standard matrix A = T (e1 ) . . . T (en )

• R(T ) =(column space of A)

• rank(T ) = dim(R(T )) = rank(A)

42
8.5 Kernel

Definition
Let T : Rn 7→ Rm with standard matrix A
Kernel of T is the set of all vectors in Rn whose image is 0 ∈ Rm

Ker(T ) ={v ∈ Rn | T (v) = 0}

={v ∈ Rn | Av = 0}
=nullspace of A

Remark: Since T (0) = 0, (0 ∈ Rn ) ∈ Ker(T )

The nullity of T , nullity(T ) = dim(Ker(T )) = nullity(A)

Dimension Theorem for Linear Transformations

Let T : Rn 7→ Rm with standard matrix A

R(T ) = column space of A

rank(T ) = rank(A)

Ker(T ) = nullspace of A

nullity(T ) = nullity(A)

∴ rank(T ) + nullity(T ) = n

43