MATH 257

Linear Algebra with Computational Applications

ILLINOIS
Department of Mathematics
 Table of Contents.
Module 1: Introduction to Linear Systems Module 16: Subspaces of Rn
Module 2: Matrices and Linear Systems Module 17: Column spaces and Nullspaces
Module 3: Echelon forms of matrices Module 18: Abstract vector spaces
Module 4: Gaussian elimination Module 19: Linear independence
Module 5: Linear combinations Module 20: Basis and Dimension
Module 6: Matrix vector multiplication Module 21: The four fundamental subspaces
Module 7: Matrix multiplication Module 22: Graphs and adjacency matrices
Module 8: Properties of matrix multiplication Module 23: Orthogonal complements
Module 9: Elementary matrices Module 24: Coordinates
Module 10: Inverse of a matrix Module 25: Orthonormal basis
Module 11: Computing an inverse Module 26: Linear Transformations
Module 12: LU decomposition Module 27: Coordinate matrix
Module 13: Solving using LU decomposition Module 28: Determinants
Module 14: Spring-mass systems Module 29: Cofactor expansion
Module 15: Inner products and orthogonality Module 30: Eigenvectors and eigenvalues
 Table of Contents .
(ctd)

Module 31: Computing eigenvalues

Module 32: Properties of eigenvectors
Module 33: Markov matrices
Module 34: Diagonalization
Module 35: Powers of Matrices
Module 36: Matrix exponential
Module 37: Linear Differential Equations
Module 38: Projections onto lines
Module 39: Projections onto subspaces
Module 40: Least squares solutions
Module 41: Linear Regression
Module 42: Gram-Schmidt Method
Module 43: Spectral theorem
Module 44: Singular Value Decomposition
Module 45: Low rank approximations
 LINEAR ALGEBRA
Introduction to Linear Systems

ILLINOIS
Department of Mathematics
Definition. A linear equation is a equation of the form
a1 x1 + . . . + an xn = b
where a1 , ..., an , b are numbers and x1 , ..., xn are variables.
Example. Which of the following equations are linear equations (or can be rearranged to
become linear equations)?
Solution.

4x1 − 5x2 + 2 = x1
√
x2 = 2( 6 − x1 ) + x3
4x1 − 6x2 = x1 x2
√
x2 = 2 x1 − 7

Definition. A linear system is a collection of one or more linear equations involving the
same set of variables, say, x1 , x2 , ..., xn .
A solution of a linear system is a list (s1 , s2 , ..., sn ) of numbers that makes each equation in
the system true when the values s1 , s2 , ..., sn are substituted for x1 , x2 , ..., xn , respectively.
Example. Two equations in two variables:
x1 + x2 = 1 (I)
−x1 + x2 = 0. (II)
What is a solution for this system of linear equations?
Solution.
Example. Does every system of linear equation have a solution?
x1 − 2x2 = −3 (III)
2x1 − 4x2 = 8. (IV)
Solution.
Example. How many solutions are there to the following system?
x1 + x2 = 3 (V)
−2x1 − 2x2 = −6 (VI)
Solution.
Theorem 1. A linear system has either
one unique solution or no solution or infinitely many solutions.
Definition. The solution set of a linear system is the set of all solutions of the linear system.
Two linear systems are equivalent if they have the same solution set.
Example. Consider
x1 − 3x2 = 1 (VII)
−x1 + 5x2 = 3 (VIII)
Transform this linear system into another easier equivalent system.
Solution.
 LINEAR ALGEBRA
Matrices and Linear Systems

ILLINOIS
Department of Mathematics
Definition. An m × n matrix is a rectangular array of numbers with m rows and n columns.
Example. Let’s give a few examples.
Solution.

In terms of the entries of A:

 
a11 a12 ··· a1n
 a21 a22 ··· a2n 
A= . aij is in the ith row and jth column
 
.. .. .. 
 .. . . . 
am1 am2 · · · amn
Definition. For a linear system, we define the coefficient and augmented matrix as follows:
linear system coefficient matrix augmented matrix
   
a11 x1 + a12 x2 + · · · + a1n xn = b1 a11 a12 · · · a1n a11 a12 · · · a1n b1
 a21 a22 · · · a2n b2 
a21 x1 + a22 x2 + · · · + a2n xn = b2  a21 a22 · · · a2n 
 
 
 .. .. .. ..   .. .. .. .. .. 
..  . . . .   . . . . . 
.
am1 am2 · · · amn am1 am2 · · · amn bm
am1 x1 + am2 x2 + · · · + amn xn = bm
Example. Determine the coefficient matrix and augmented matrix of the linear system
x1 − 3x2 = 1
−x1 + 5x2 = 3.
Solution.
Definition. An elementary row operation is one of the following
(Replacement) Add a multiple of one row to another row: Ri → Ri + cRj , where i 6= j,
(Interchange) Interchange two rows: Ri ↔ Rj ,
(Scaling) Multiply all entries in a row by a nonzero constant: Ri → cRi , where c 6= 0.
Example. Give several examples of elementary row operations.
Solution.
Example. Consider the elementary row operation R3 → R3 + 3R1 . Is there an elementary row
operation that reverse this row operation?
Solution.

Remark. Indeed, every row operation is reversible. We already saw how to reverse the
replacement operator. The scaling operatior R2 → cR2 is reversed by the scaling operator
R2 → c1 R2 . Row interchange R1 ↔ R2 is reversible by performing it twice.
Definition. Two matrices are row equivalent, if one matrix can be transformed into the
other matrix by a sequence of elementary row operations.
Theorem 2. If the augmented matrices of two linear systems are row equivalent, then the
two systems have the same solution set.
 LINEAR ALGEBRA
Echelon forms of matrices

ILLINOIS
Department of Mathematics
Definition. A matrix is in echelon form (or row echelon form) if
1. All nonzero rows (rows with at least one nonzero element) are above any rows of all zeros.
2. the leading entry (the first nonzero number from the left) of a nonzero row is always
strictly to the right of the leading entry of the row above it.
Example. Are the following matrices in echelon form? Circle the leading entries.
   
3 1 2 0 5 0 2 0 1 4
0 2 0 1 4 3 1 2 0 5
a) 
0
 b)  
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
   
2 −2 3 0 1 √3
c) 0 5 0  d) 0 0 2
0 0 52 0 0 0
Definition. A matrix is in row reduced echelon form (or: reduced echelon form, or:
RREF) if it is in echelon form and
3. The leading entry in each nonzero row is 1.
4. Each leading entry is the only nonzero entry in its column.
Example.
 Are the following matrices in reduced
 echelon form?
0 1 3 0 0 2 5 0 0 6
1 0 1 12 0 0
 
0 0
 0 −2  1 0 −2 3 2 −24
a) 
0 0 0 0 1 −3 4 0 0 5  b) 0 1 −2 2 0 −7 
0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 4
0 0 0 0 0 0 0 0 1 1

Theorem 3. Each matrix is row-equivalent to one and only one matrix in reduced echelon
form.
Definition. We say a matrix B is the reduced echelon form (or: the RREF) of a matrix A
if A and B are row-equivalent and B is in reduced echelon form.
Question. Is each matrix also row-equivalent to one and only one matrix in echelon from?
Solution.

Example. Find the reduced echelon form of the matrix

 
3 −9 12 −9 6 15
3 −7 8 −5 8 9
Solution.
Definition. A pivot position is the position of a leading entry in an echelon form of a
matrix. A pivot column is a column that contains a pivot position.
Example. Locate the pivot columns of the following matrix.
 
0 −3 −6 4 9
A = −1 −2 −1 3 1
1 4 5 −9 −7
Solution.
Definition. A basic variable (or pivot variable) is a variable that corresponds to a pivot
column in the coefficient matrix of a linear system. A free variable is variable that is not a
pivot variable.
Example. Consider the following system of linear equations:
 
1 6 0 3 0 0 x1 +6x2 +3x4 =0
0 0 1 −8 0 5  x3 −8x4 =5
0 0 0 0 1 7 x5 = 7

Determine the basic variables and free variables of this system.

Solution.
 LINEAR ALGEBRA
Gaussian Elimination

ILLINOIS
Department of Mathematics
Goal. Solve linear systems for the pivot variables in terms of the free variables (if any) in the
equation.
Algorithm. (Gaussian Elimination) Given a linear system,
(1) Write down the augmented matrix.
(2) Find the reduced echelon form of the matrix.
(3) Write down the equations corresponding to the reduced echelon form.
(4) Express pivot variables in terms of free variables.
Example. Find the general solution of
3x1 −7x2 +8x3 −5x4 +8x5 = 9
C.F. Gauß (1777–1855)
3x1 −9x2 +12x3 −9x4 +6x5 = 15
Solution.
Solution.
 
1 0 −2 3 5 −4
RREF of the augmented matrix:
0 1 −2 2 1 −3
Example. Is the following linear system consistent? Is the solution unique?
3x2 − 6x3 + 6x4 + 4x5 = −5
3x1 − 7x2 + 8x3 − 5x4 + 8x5 = 9
3x1 − 9x2 + 12x3 − 9x4 + 6x5 = 15
Solution.
   
0 3 −6 6 4 −5 3 −9 12 −9 6 15
Row reduce augmented matrix: 3 −7 8 −5 8 9  ; 0 2 −4 4 2 −6 
3 −9 12 −9 6 15 0 0 0 0 1 4
Theorem 4. A linear system is consistent if and only if an echelon form of the augmented
matrix has no row of the form 0 ... 0 b , where b is nonzero. If a linear system is
consistent, then the linear system has
○ a unique solution (when there are no free variables) or
○ infinitely many solutions (when there is at least one free variable).
 
3 4 −3
Example. Consider the linear system whose augmented matrix is 3 4 −3 . What can
6 8 −5
you say about the number of solutions of this system?
Solution.
 LINEAR ALGEBRA
Linear combinations

ILLINOIS
Department of Mathematics
    
a11 a12 ··· a1n b11 b12 ··· b1n
 a21 a22 ··· a2n   b21 b22 ··· b2n 
Definition. Consider m × n-matrices A = 
 .. .. .. .. ,

and B = 
 .. .. .. .. .

 . . . .   . . . . 
am1 am2 · · · amn bm1 bm2 · · · bmn

a) The sum of A + B is b)
 The product cA fora scalar c is
ca11 ca12 · · · ca1n
 
a11 + b11 a12 + b12 · · · a1n + b1n
 a21 + b21 a22 + b22 · · · a2n + b2n   ca21 ca22 · · · ca2n 
 
 .. .. .. 
 
 .. .. .. ..  ..
 . . . .   . . . . 
am1 + bm1 am2 + bm2 · · · amn + bmn cam1 cam2 · · · camn
Example. Calculate
     
1 0 2 3 2 1 0
+ = 5 =
5 2 3 1 3 1 −1

Warning. Addition is only defined if A has the same number of columns as B and the same
number of rows as B.
Definition. A column vector is an m × 1-matrix. A row vector is a 1 × n-matrix.
Example. Give a few ex-
amples of column and row
vectors.

Definition. If A is m × n, the transpose of A is the n × m matrix, denoted by AT , whose

columns are formed from the corresponding rows of A. In terms of matrix elements:
(AT )ij = Aji .
Example. What is the
1 2
transpose of 3 4?
5 6

Remark. The transpose of a column vector is a row vector and vice versa.
Definition. The linear combination of m × n-matrices A1 , A2 , . . . , Ap with coefficients
c1 , c2 , . . . , cp is defined as
c1 A1 + c2 A2 + · · · + cp Ap .
Example. Consider m × n-matrices A1 and A2 . Give examples of linear combinations of these
two matrices.
Solution.

Definition. The span (A1 , . . . , Ap ) is defined as the set of all linear combinations of
A1 , . . . , Ap . Stated another way:
span (A1 , . . . , Ap ) := {c1 A1 + c2 A2 + · · · + cp Ap : c1 , . . . , cp scalars}.
Definition. We denote the set of all column vectors of length m by Rm .
     
1 4 −1
Example. Let a1 = 0 , a2 = 2 and b = 8 . Is b a linear combination of a1 , a2 ?
    
3 14 −5
Solution.
Key observation from the previous example: Solving linear systems is the same as finding
linear combinations!
Theorem 5. A vector equation

x1 a1 + x2 a2 + · · · + xn an = b

has the same solution set as the linear system whose augmented matrix is
 
a1 a2 · · · an | b

In particular, b can be generated by a linear combination of a1 , a2 , . . . , an if and only if there is

a solution to the linear system corresponding to the augmented matrix.
Notation. We often define a matrix in terms of its columns or its rows:
 
R1
 R2 
   
A := a1 a2 · · · an or A :=  . 
 .. 
Rm
 LINEAR ALGEBRA
Matrix vector multiplication

ILLINOIS
Department of Mathematics
Definition. Let x be a vector in Rn and A = a1 . . . an an m × n-matrix. We define the
 

product Ax by
Ax = x1 a1 + x2 a2 + . . . + xn an .

Remark.
○ Ax is a linear combination of the columns of A using the entries in x as coefficients.
○ Ax is only defined if the number of entries of x is equal to the number of columns of A.
 
  1 2  
2 0 2
Example. Consider A = ,B= 0 1 ,x=
  . Determine Ax and Bx.
1 1 3
3 5
Solution.
Example. Consider the following vector equation
     
1 3 0
x1 + x2 = .
2 4 2

Find a 2 × 2 matrix A such that (x1 , x2 ) is a solution to the above equation if and only if
   
x 0
A 1 = ?
x2 2

Solution.
Theorem 6. Let A = a1 . . . an be an m × n-matrix and b in Rm . Then the following
 

are equivalent
○ (x1 , x2 , . . . , xn ) is a solution of the vector equation x1 a1 + x2 a2 + · · · + xn an = b
 
x1
 .. 
○  .  is a solution to the matrix equation Ax = b.
xn
 
○ (x1 , x2 , . . . , xn ) is a solution of the system with augmented matrix A b
 
Notation. We will write Ax = b for the system of equations with augmented matrix A b .

A matrix as machine. Let A be a m × n matrix.

○ Input: n-component vector x ∈ Rn .
○ Output: m-component vector b = Ax ∈ Rm .

x Ax
A
  
0 1
Example. Consider the matrix A = . What does this machine do?
1 0
Solution.
  
1 0
Example. Consider the matrix B = . What does this machine do?
0 0
Solution.
Composition of machines. Let A be an m × n matrix and B be an k × l matrix.Now we can
compose the two machines:

x Ax B(Ax)
A B

Question. This composition only works for some k, l, m, n. For which?

Solution.
    
0 1 1 0
Example. Let A= ,B = be as before. Is A(Bx) = B(Ax)?
1 0 0 0
Solution.
 LINEAR ALGEBRA
Matrix multiplication

ILLINOIS
Department of Mathematics
  
Definition. Let A be an m × n-matrix and let B = b1 . . . bp be an n × p-matrix. We
define
AB := [Ab1 Ab2 · · · Abp ]
 
4 −2  
2 −3
Example. Compute AB where A = 3 −5 and B =
  .
6 −7
0 1
Solution.

Remark. Ab1 is a linear combination of the columns of A and Ab2 is a linear combination
of the columns of A. Each column of AB is a linear combination of the columns of A using
coefficients from the corresponding columns of B.
Definition. Let A be an m × n-matrix and let B be an n × p-matrix. We define

AB := [Ab1 Ab2 · · · Abp ]

Question. If C is 4 × 3 and D is 3 × 2, are CD and DC defined? What are their sizes?

Solution.
We already learnt that we can think of matrices as machines.
○ Let B be n × p: input x ∈ Rp , output c = Bx ∈ Rn .
○ Let A be m × n: input y ∈ Rn , output b = Ay ∈ Rm .
We already saw that these machines can be composed.

x Bx A(Bx)
B A

Question. How does that compare to the machine given by AB?

x (AB)x
AB
Example. Consider      
2 0 1 2 x
A= , B= , x= 1
1 1 0 1 x2
Compute (AB)x and A(B(x)). Are these the same?
Solution.

Theorem 7. Let A be an m × n matrix and B be an n × p matrix. Then for every x ∈ Rp

A(Bx) = (AB)x.
Row-Column Rule for computing AB. Let A be m × n and B be n × p such that
 
R1
 ..   
A =  .  , and B = C1 . . . Cp .
Rm
Then  
R1 C1 . . . R1 Cp
AB =  R2 C1 . . . R2 Cp  and (AB)ij = Ri Cj = ai1 b1j + ai2 b2j + · · · + ain bnj .
Rm C1 . . . Rm Cp
 
  2 −3
2 3 6
Example. A = ,B= 0 1  . Compute AB, if it is defined.
−1 0 1
4 −7
Solution.
Outer Product Rule for computing AB. Let A be m × n and B be n × p such that
 
R1
   .. 
A = C1 · · · Cn , and B =  .  .
Rn
Then
AB = C1 R1 + · · · + · · · + Cn Rn
 
  2 −3
2 3 6
Example. A = ,B= 0 1  . Compute AB, if it is defined.
−1 0 1
4 −7
Solution.
 LINEAR ALGEBRA
Properties of matrix multiplication

ILLINOIS
Department of Mathematics
Definition. The identity matrix In of size n is defined as
 
1 0 ... 0
0 1 . . . 0
In =  . .. . . ..  .
 
 .. . . .
0 0 ... 1

Theorem 8. Let A be an m × n matrix and let B and C be matrices for which the indicated
sums and products are defined.
(a) A (BC ) = (AB)C (associative law of multiplication)
(b) A (B + C ) = AB + AC , (B + C ) A = BA + CA (distributive laws)
(c) r (AB) = (rA)B = A(rB) for every scalar r ,
(d) A (rB + sC ) = rAB + sAC for every scalars r , s (linearity of matrix multiplication)
(e) Im A = A = AIn (identity for matrix multiplication)
Warning. Properties above are analogous to properties of real numbers. But NOT ALL
properties of real number also hold for matrices.
   
1 1 1 0
Example. Let A = ,B= . Determine AB and BA. Are these matrices the
0 1 1 1
same?
Solution.
  
  1 2
1 2 0
Example. Let A = ,B= 0 1  . Compute (AB)T , AT B T and B T AT .
3 0 1
Solution. −2 4
 
  1 2
1 2 0 
AB = 0 1 =
3 0 1
−2 4
(AB)T =
   
1 3   7 3 10
T T 1 0 −2
A B = 2
 0 = 2 0 −4
2 1 4
0 1 2 1 4
 
  1 3
T T 1 0 −2 
B A = 2 0 =
2 1 4
0 1

Theorem 9. The transpose of a product is the product of transposes in opposite order:

(AB)T = B T AT
Definition. We write Ak for A · · · A, k-times; that is Ak is obtained by multiplying A k-times
with itself.
Question. For which matrices A does Ak make sense? If A is m × n what can m and n be?
Solution.

 3
1 0
Example. Determine .
3 2
Solution.
 LINEAR ALGEBRA
Elementary matrices

ILLINOIS
Department of Mathematics
Definition. An elementary matrix is one that is obtained by performing a single elementary
row operation on an identity matrix.
A permutation matrix is one that is obtained by performing row exchanges on an identity
matrix.      
1 0 0 1 0 0 1 0 0
Example. Consider E1 =  0 3 0 , E2 =  0 0 1 , E3 =  0 1 0 . Are these
0 0 1 0 1 0 2 0 1
matrices elementary matrices?
Solution.
Question. Let A be a 3 × 3-matrix. What happens to A if you multiply it by one of E1 , E2
and E3 ?
Solution.
  
1 0 0 a11 a12 a13
E1 A = 0 2 0
  a21 a22 a23  =
0 0 1 a31 a32 a33
  
1 0 0 a11 a12 a13
E2 A = 0 0 1
  a21 a22 a23  =
0 1 0 a31 a32 a33
  
1 0 0 a11 a12 a13
E3 A = 0 1 0
  a21 a22 a23  =
3 0 1 a31 a32 a33

Theorem 10. If an elementary row operation is performed on an m × n-matrix A, the

resulting matrix can be written as EA, where the m × m-matrix E is created by performing the
same row operations on Im .
Theorem 11. Let A, B be two m × n-matrices and row-equivalent. Then there is a sequence
m × m-elementary matrices E1 , . . . , E` such that
E` . . . E1 A = B.
   
0 1 1 2
Example. Consider A = 1 2 and B = 0 1. Find two elementary matrices E1 , E2 such
2 4 0 0
that E2 E1 A = B.
Solution.
Example. From the previous example, we know
     
1 2 1 0 0 0 1 0 0 1
0 1 =  0 1 0 1 0 0 1 2 .
0 0 −2 0 1 0 0 1 2 4
| {z } | {z }
B A

Find elementary matrices E1−1 and E2−1 such that A = E1−1 E2−1 B.
Solution.
 LINEAR ALGEBRA
Inverse of a matrix

ILLINOIS
Department of Mathematics
The inverse of a real number a is denoted by a−1 . For example, 7−1 = 1/7 and

7 · 7−1 = 7−1 · 7 = 1.

Remark. Not all real numbers have inverse. 0−1 is not well defined, since there is no real
number b such that 0 · b = 1.
Remember that the identity matrix In is the n × n-matrix
 
1 0 ··· 0
0 1 · · · 0
In =  . . .
 
. . . .. 
. . . .
0 0 ··· 1

Definition. An n × n matrix A is said to be invertible if there is an n × n matrix C satisfying

CA = AC = In

where In is the n × n identity matrix. We call C the inverse of A.

  
1 0 0
Example. What is the inverse of 5 1 0?
0 0 1
Solution.

Theorem 12. Let A be an invertible matrix, then its inverse C is unique.

Proof. Assume B and C are both inverses of A. Then
Theorem 13. Suppose A and B are invertible. Then:

−1
(a) A−1 is invertible and A−1 = A (i.e. A is the inverse of A−1 ).
(b) AB is invertible and (AB)−1 = B −1 A−1 .

−1
T
(c) AT is invertible and AT = A−1 .
Proof.
○ We will write A−1 for the inverse of A. Multiplying by A−1 is like “dividing by A.”
○ Do not write BA . Why?

○ Fact: if AB = I , then A−1 = B and so BA = I . (Not so easy to show at this stage.)

○ Not all n × n matrices are invertible. For example, the 2 × 2 matrix
 
0 1
0 0
is not invertible. Try to find an inverse!
Theorem 14. Let A be an invertible n × n matrix. Then for each b in Rn , the equation
Ax = b has the unique solution x = A−1 b.
Proof.

Question. If A is an invertible n × n matrix, then A has how many pivot columns?

 LINEAR ALGEBRA
Computing the inverse

ILLINOIS
Department of Mathematics
  
Question. When is the 1 × 1 matrix a invertible?
Solution.

 
a b
Theorem 15. Let A = . If ad − bc 6= 0, then A is invertible and
c d
 
−1 1 d −b
A = .
ad − bc −c a

If ad − bc = 0, then A is not invertible.

Proof.
Calculate
    
1 d −b a b 1
=
ad − bc −c a c d ad − bc
Theorem 16. Let A be an n × n-matrix. The following are equivalent:
○ A is invertible.
○ the reduced echelon form of A is In .
Proof.

Theorem 17. Suppose A is invertible. The every sequence of elementary row operations
that reduces A to In will also transform In to A−1 .
Algorithm.
○ Place A and I side-by-side to form an augmented matrix [ A | I ].
This is an n × 2n matrix (Big Augmented Matrix), instead of n × (n + 1).
○ Perform row operations on this matrix (which will produce identical operations on A and
I ).
○ By Theorem:
[ A | I ] will row reduce to I | A−1
 

or A is not invertible.
 
2 0 0
Example. Find the inverse of A = −3 0 1, if it exists.
0 1 0
Solution.
1
   
  2 0 0 1 0 0 1 0 0 2 0 0
A I = −3 0 1 0 1 0  ∼ · · · ∼ 0 1 0 0 0 1 
3
0 1 0 0 0 1 0 0 1 2 1 0
Example. Let’s do the previous example step by step.
Solution.
 
  2 0 0 1 0 0
A I = −3 0 1 0 1 0 
0 1 0 0 0 1
 LINEAR ALGEBRA
LU decomposition

ILLINOIS
Department of Mathematics
Definition. An n × n matrix A is called  
? ? ? ? ?
0 ? ? ? ?
 
0
upper triangular if it is of the form  0 ? ? ? ,
. .
0 . . .. 

0 0
0 0 0 0 ∗
 
? 0 0 0 0
? ? 0 0 0
 
?
lower triangular if it is of the form  ? ? 0 0 .
 . . .. 
? ? ? . .
? ? ? ? ?
Example. Give a few examples of upper and lower triangular matrices!
Solution.
Theorem 18. The product of two lower (upper) triangular matrices is lower (upper)
triangular.
Example. Consider the row operation Ri → Ri + cRj where j < i. What can you say about
the elementary matrix E corresponding this row operation? What about E −1 ?
Solution.

Remark. The inverse of a lower (upper) triangular matrix (if it exists) is again lower (upper)
triangular.
Definition. A matrix A has an LU decomposition if there is a lower triangular matrix L and
a upper triangular matrix U such that A = LU.
Theorem 19. Let A be an n × n-matrix. If A can be brought to echelon form just using row
operations of the form Ri → Ri + cRj where j < i, then A has an LU-decomposition.
Proof.

Remark. Not every matrix has an LU decomposition.

The LU decomposition of a matrix is not unique.
  
2 1 1
Example. Let A =  4 −6 0 . Determine its LU decomposition.
−2 7 2
Solution.
 
2 1 1
A =  4 −6 0
−2 7 2
  
1 0 0
Remark. In previous example L = `21 1 0 where `ij is the factor between pivot and
`31 `32 1
the entry you want to make zero in the elimination process. This works in general if you do the
row operations in the right order.
Step 1:    
2 1 1 2 1 1
A =  4 −6 0 ; = 0 −8 −2
−2 7 2 R2 →R2 −2 R1 ,R3 →R3 + 1 R1 0 8 3
Then set `21 = 2 and `31 = −1.
Step 2:    
2 1 1 2 1 1
0 −8 −2 ; = 0 −8 −2 .
0 8 3 R3 →R3 + 1 ·R2 0 0 1
     
2 1 1 1 0 0 1 0 0
Set `32 = −1,‘ U := 0 −8 −2 , and L :=  `21 1 0 = 2 1 0 .
0 0 1 `31 `32 1 −1 −1 1
  
1 2 2
Example. Determine the LU-decomposition of 4 4 4.
4 4 8
Solution.
 
1 2 2
4 4 4
4 4 8

Remark. It is important that you do the row operations in the right order!
 LINEAR ALGEBRA
Solving linear systems using LU decomposition

ILLINOIS
Department of Mathematics
Theorem 20. Let A be an n × n-matrix such that A = LU, where L is a lower triangular
matrix and U is a upper triangular matrix, and let b ∈ Rn . In order to find a solution of the
linear system
Ax = b,
it is enough to find a solution of the linear system

Ux = c,

where c satisfies Lc = b.
Proof.
     
2 1 1 x1 5
Example. Find a solution to the linear system  4 −6 0 x2  = −2.
−2 7 2 x3 9
Solution.
    
2 1 1 1 0 0 2 1 1
Recall the LU decomposition  4 −6 0 =  2 1 0 0 −8 −2 .
−2 7 2 −1 −1 1 0 0 1
Question. Why do we care about LU decomposition if we already have Gaussian elimination?
Solution.

Theorem 21. Let A be n × n matrix. Then there is a permutation matrix P such that PA
has an LU-decomposition.
Proof.
  
0 0 1
Example. Let A = 1 1 0. Find a permutation matrix P such that PA has a LU
2 1 0
decomposition.
Solution.
 LINEAR ALGEBRA
Spring-mass systems

ILLINOIS
Department of Mathematics
Example. Consider the following spring-mass system, consisting of five masses and six
springs fixed between two walls.

m1 m2 m3 m4 m5

Goal.
○ Add (steady) applied forces f1 , f2 , f3 , f4 , f5 a (steady) applied force on mass i.
○ Compute the displacements u1 , u2 , u3 , u4 , u5 of the fives masses.

u1 u2 u3 u4 u5

m1 m2 m3 m4 m5

f1 f2 f3 f4 f5
Equilibrium. In the equilibrium, the forces at each mass add up to 0.

Hooke’s law. The force F needed to extend or compress a spring by some distance u is
proportional to that distance; that is F = −ku, where k is a constant factor characteristic of
the spring, called its stiffness. Let ki be the stiffness of the i-th spring.

Springs:
○ u1 > 0 ; spring 1 is extended.
○ u2 − u1 > 0 ; spring 2 is extended.
u1 u2 Forces
u3 at m1 : u4 u5
f1 f2 ○ applied forces f1 (if positive, pushes m1 to the right).
m1 m2 ○m spring 1: −k1 u1 (since
m4 u1 > 0, pulls m m15 to the left).
3
○ spring 2: k2 (u2 − u1 ) (since u2 − u1 > 0, pulls m1 to
f3
the right). f4 f5
−k1 u1 k2 (u2 − u1 )
Equilibrium:

f1 − k1 u1 + k2 (u2 − u1 ) = 0 ; (k1 + k2 )u1 − k2 u2 = f1

 Springs:
○ u2 − u1 > 0 ; spring 2 is extended.
○ u3 − u2 > 0 ; spring 3 is extended.
Forces at m2 :
u1 u2 u3 u4 u5
f1 f2 ○ applied forces f2 (if positive, pushes m2 to the right).

m1 m2 m3 f3 ○ spring m
2: −k2 (u2 − u1 ) (since (u2 − u1 ) > 0, pulls
m5
4 left).
m2 to the
○ spring 3: k3 (u3f4− u2 ) (since u3 − fu52 > 0, pulls m2 to
−k2 (u2 − u1 ) k3 (u3 − u2 ) the right).
Equilibrium at m2 :

f2 − k2 (u2 − u1 ) + k3 (u3 − u2 ) = 0
; −k2 u1 + (k2 + k3 )u2 − k3 u3 = f2
 Springs:
○ u5 − u4 > 0 ; spring 5 is extended.
○ u5 > 0 ; spring 6 is compressed.
Forces at m2 :
u4 u5
f4 f5 ○ applied forces f5 (if positive, pushes m5 to the right).
○ spring 5: −k5 (u5 − u4 ) (since (u5 − u4 ) > 0, pulls
m4 m5 m5 to the left).
○ spring 6: k3 (u3 − u2 ) (since u5 > 0, pushes m2 to
−k5 (u5 − u4 ) −k6 u5 the left).
Equilibrium at m5 :

f5 − k5 (u5 − u4 ) − k6 u5 = 0
; (k5 + k6 )u5 − k5 u4 = f5
Equilibrium equations.
 
(k1 + k2 )u1 − k2 u2 = f1 k1 + k2 −k2 0 0 0 f1
−k2 u1 + (k2 + k3 )u2 − k3 u3 = f2

 −k2 k2 + k3 −k3 0 0 f2 

 0 −k3 k3 + k4 −k4 0 f3 
−k3 u2 + (k3 + k4 )u3 − k4 u4 = f3  
 0 0 −k4 k4 + k5 −k5 f4 
−k4 u3 + (k4 + k5 )u4 − k5 u5 = f4 0 0 0 −k5 k5 + k6 f5
−k5 u4 + (k5 + k6 )u5 = f5 .
Remark.
○ The purpose of this example is not to find the precise solutions to these equations, but
rather to show you that linear equations appear naturally in engineering.
○ Finite element method: object is broken up into many small parts with connections
between the different parts ; gigantic spring-mass system where the forces from spring
correspond to the interaction between the different parts.
○ In practice: millions of equations, not just five!
○ Solve not for a single force vector f, but for many different vectors. Thus the coefficient
matrix stays the same ; LU-decomposition can make a difference in how quickly such
systems can be solved.
 LINEAR ALGEBRA
Inner Product and Orthogonality

ILLINOIS
Department of Mathematics
Definition. The inner product of v, w ∈ Rn is

v · w = vT w.
   
v1 w1
 ..   .. 
Example. If v =  .  and w =  . , then v · w is ...
vn wn
Solution.

Question. Why is v · w = w · v?
Solution.
Question. Why is v · v always larger or equal to 0? For which v is v · v = 0?
Solution.

Theorem 22. Let u, v and w be vectors in Rn , and let c be any scalar. Then
(a) u · v = v · u
(b) (u + v) · w = u · w + v · w
(c) (cu) · v =c (u · v) = u · (cv)
(d) u · u ≥ 0, and u · u = 0 if and only if u = 0.
Definition. Let v, w ∈ Rn . x2
The norm (or length) of v is
√ q v−w
kvk = v · v = v12 + · · · + vn2 . w v

The distance between v and w is kvk

dist (v, w) = kv − wk. x1

     
2 2 0
Example. Compute k−1k and dist(0 , 1).
1 1 0
Solution.
Definition. Let v, w ∈ Rn . We say v and w are orthogonal if v · w = 0.
Theorem 23. Let v, w ∈ Rn be non-zero. Then v and w are orthogonal if and only if they
are perpendicular (ie. if they form a right angle).
Proof.

v−w

w
v
        
1 1 1 1
Example. Are , orthogonal? Are , orthogonal?
1 −1 1 −2
Solution.

Definition. A set of vectors in Rn is pairwise orthogonal if each pairing of them is

orthogonal. We call such a set an orthogonal set.
  
1 1
3
Example. Find a non-zero v ∈ R such that 1 , −1 and v form an orthogonal set?
  
Solution. 0 0
Definition. A unit vector in Rn is vector of length 1.
 
1 x2
Example. Let v = . Is v a unit vector?
2
Solution.
x1

Definition. A set of vectors form an orthonormal set if this set is an orthogonal set and all
vectors in the set are unit vectors.
   
1 1
Example. Let v1 = , v2 = . Do v1 and v2 form an orthonormal set?
1 −1
Solution.
 LINEAR ALGEBRA
Subspaces of Rn

ILLINOIS
Department of Mathematics
Definition. A non-empty subset H of Rn is a subspace of Rn if it satisfies the following two
conditions:
○ If u, v ∈ H, then the sum u + v ∈ H. (H is closed under vector addition).
○ If u ∈ H and c is a scalar, then cu ∈ H. (H is closed under scalar multiplication.)
Theorem 24. Let v1 , v2 , . . . , vm ∈ Rn . Then Span (v1 , v2 , . . . , vm ) is a subspace of Rn .
Proof.
   
x
Example. Is H = : x ∈ R a subspace of R2 ?
x
Solution.
x2

x1

 
0
Example. Let Z = . Is Z a subspace of R2 ?
0
Solution.
   
x
Example. Let H = : x ∈ R . Is H a subspace of R2 ?
x +1
Solution.
x2 H

x1

  
x
Example. Is U = ∈ R2 : x 2 + y 2 < 1 a subspace of R2 ?
y
Solution.
x2

U
x1
   
x 2
Example. Consider V = ∈ R : xy ≥ 0 .
y
Solution.
x2
V

x1

  
x 2
Question. Is W = ∈ R : xy = 0 a subspace?
y
Solution.
 LINEAR ALGEBRA
Column spaces and Nullspaces

ILLINOIS
Department of Mathematics
Definition. The column space, written as Col(A),  of an m × n matrix A is the set of all
linear combinations of the columns of A. If A = a1 a2 · · · an , then
Col(A) = span (a1 , a2 , . . . , an ).
 
1 0
Example. Describe the column space of A = .
0 0
Solution.

Theorem 25. Let A be an m × n matrix. Then Col(A) is a subspace of Rm .

Proof.
Theorem 26. Let A be an m × n matrix and b ∈ Rm . Then b is in Col(A) if and only if the
linear system Ax = b has a solution.
Proof.

Question. Let A and B be two row-equivalent matrices. Is Col(A) = Col(B)?

Solution.
Definition. The nullspace of an m × n matrix A, written as Nul(A), is the set of all solutions
to the homogeneous equation Ax = 0; that is, Nul(A) = {v ∈ Rn : Av = 0}.
Theorem 27. The nullspace of an m × n matrix A is a subspace of Rn .
Proof.

 
v1
Example. Let H := {v2  : v1 + v2 − v3 = 0}. Find a matrix A such that H = Nul(A).
v3
Solution.
  
Example. Let A = 1 1 −1 . Find two vectors v, w such that Nul(A) = span(v, w).
Solution.

Nul(A)
x2
v
w x1
x3

Question. Is there a matrix B such that Nul(A) = Col(B)?

Solution.
Theorem 28. Let A be an m × n matrix, let b ∈ Rm , and let w ∈ Rn such that Aw = b.
Then {v ∈ Rn : Av = b} = w + Nul(A).
Proof.

   T
Example. Let A = 1 1 −1 and let b = 1. Observe that A 1 0 0 = b. Use this to
describe {v ∈ Rn : Av = b}. x2
Solution. Nul(A) w + Nul(A)

x1
x3
 LINEAR ALGEBRA
Abstract vector spaces

ILLINOIS
Department of Mathematics
○ The most important property of column vectors in Rn is that you can take linear
combinations of them.
○ There are many mathematical objects X , Y , . . . for which a linear combination cX + dY
make sense, and have the usual properties of linear combination in Rn .
○ We are going to define a vector space in general as a collection of objects for which linear
combinations make sense.The objects of such a set are called vectors.

x2 f (x) = x + 1
3 (f + g )(x) = sin x + x
2
1 1 1
2· + 2·
x1
−2−1 1 2 g (x) = sin x − 1
−1
2g (x) = 2 sin x − 2

=
Definition. A vector space is a non-empty set V of objects, called vectors, for which linear
combinations make sense. More precisely: on V there are defined two operations, called
addition and multiplication by scalars (real numbers),satisfying the following axioms for all
u, v , w ∈ V and for all scalars c, d ∈ R:
○ u + v is in V . (V is “closed under addition”.)
○ u + v = v + u.
○ (u + v) + w = u + (v + w).
○ There is a vector (called the zero vector) 0V in V such that u + 0V = u.
○ For each u in V , there is a vector −u in V satisfying u + (−u) = 0V .
○ cu is in V . (V is “closed under scalar multiplication”.)
○ c(u + v) = cu + cv.
○ (c + d)u = cu + du.
○ (cd)u = c(du).
○ 1u = u.
In particular, we may talk about linear combinations and span within a vector space, e.g.
3u + 2v or span(u, v).
Example. Explain how the set of functions R → R is a vector space.
Solution.

Example. Explain how the set of all 2 × 2 matrices is a vector space.

Solution.
Definition. Let V be a vector space. A non-empty subset W ⊆ V is a subspace of V if
○ u + v for all u, v ∈ U (closed under addition)
○ cu for all u ∈ U and c ∈ R (closed under scalar multiplication)
Example. Explain why the set of all symmetric 2 × 2 matrices is a subspace of the vector
space of 2 × 2 matrices?
Solution.

Question. Is the set of all invertible 2 × 2 matrices a subspace of the vector space of 2 × 2
matrices?
Solution.
Example. Let Pn be the set of all polynomials of degree at most n, that is

Pn = {a0 + a1 t + a2 t 2 + · · · + an t n : a0 , . . . , an ∈ R}.

Explain why it is a vector space. Can you think of it as a subspace of the vector space of all
functions R → R?
Solution.
 LINEAR ALGEBRA
Linear Independence

ILLINOIS
Department of Mathematics
Definition. Vectors v1 , . . . , vp are said to be linearly independent if the equation

x1 v1 + x2 v2 + · · · + xp vp = 0

has only the trivial solution (namely, x1 = x2 = · · · = xp = 0).

We say the vectors
 are linearly
 dependent if they are not linearly independent.
1 2
Example. Are , linearly independent?
1 2
Solution.

Theorem 29. Vectors v1 , . . . , vp are linear dependent if and only if there is i ∈ {1, . . . , p}
such that vi ∈ span(v1 , . . . , vi−1 , vi+1 , . . . , vp ).
Question. A single non-zero vector v1 is always linearly independent. Why?
Solution.

Question. Two vectors v1 , v2 are linearly independent if and only if neither of the vectors is a
multiple of the other. Why?
Solution.

Question. Vectors v1 , . . . , vp containing the zero vector are linearly dependent.

Solution.
      
1 1 −1
Example. Consider 1 , 2 , 1  . Are these vectors linearly independent?
    
1 3 3
Solution.
Question. In the previous example we determined the following linear dependence:
     
1 1 −1
3 1 − 2 2 + 1  = 0.
    
1 3 3
Can you write this as a matrix equation?
Solution.

Idea. The Null space determines linear (in)dependence!

Theorem 30. Let A be an m × n matrix. The following are equivalent:
○ The columns of A are linearly independent.
○ Ax = 0 has only the solution x = 0.
○ A has n pivots.
○ there are no free variables for Ax = 0.
Question. Let v1 , . . . , vn ∈ Rm . If n > m, then v1 , . . . vn are linearly dependent. Why?
Solution.

Question. Consider an m × n-matrix A in echelon form. The pivot columns of A are linearly
independent. Why?
Solution.
 LINEAR ALGEBRA
Basis and Dimension

ILLINOIS
Department of Mathematics
Definition. Let V be a vector space. A sequence of vectors (v1 , . . . , vp ) in V is a basis of V
if
○ V = span (v1 , . . . , vp ) , and
○ (v1 , . . . , vp ) are linearly independent.
       
1 0 1 1
Example. Check that both , and , are bases of R2 .
0 1 1 −1
Solution.
Theorem 31. Every two bases in a vector space V contain the same numbers of vectors.
Definition. The number of vectors in a basis of V is the dimension of V .
Example. What is the dimension of Rn ?
Solution.
Theorem 32. Suppose that V has dimension d.
○ A sequence of d vectors in V are a basis if they span V .
○ A sequence of d vectors in V are a basis if they are linearly independent.
Proof.
      
1 0 1
Example. Is 2 , 1 , 0 a basis of R3 ?
   
0 1 3
Solution.
Theorem 33. A basis is a minimal spanning set of V ; that is the elements of the basis span
V but you cannot delete any of these elements and still get all of V .
Example. Produce a basis of R2 from the vectors
     
1 1 −.5 x2
v1 = , v2 = , v3 = . v1
2 1 −2
v2
Solution.
x1
v3

 
2
Example. Produce a basis of R2 from the vector .
1
Solution.
 LINEAR ALGEBRA
Bases and dimensions of the four fundamental
subspaces

ILLINOIS
Department of Mathematics
Algorithm. To find a basis for Nul(A):
○ Find the parametric form of the solutions to Ax = 0.
○ Express solutions x as a linear combination of vectors with the free variables as coefficients.
○ Use these vectors as a basis of Nul(A).
 
3 6 6 3 9
Example. Find a basis for Nul(A) where A = .
6 12 15 0 3
Solution.
   
3 6 6 3 9 1 2 0 5 13
6 12 15 0 3 RREF 0 0 1 −2 −5
Definition. The rank of a matrix is the number of pivots it has.
Theorem 34. Let A be an m × n matrix with rank r . Then dim Nul(A) = n − r .
   
Remark. Let A = a1 . . . an and let U = u1 . . . un be an echelon form of A. Explain
why
x1 u1 + · · · + xn un = 0 ⇐⇒ x1 a1 + · · · + xn an = 0.
Solution.

Theorem 35. Let A be an m × n matrix with rank r . The pivot columns of A form a basis
of Col(A). In particular, dim Col(A) = r .
Proof.
  
1 2 0 4
2 4 −1 3 
Example. Find a basis for Col(A) where A = 
3
.
6 2 22
4 8 0 16
Solution.
   
1 2 0 4 1 2 0 4
2 4 −1 3  0 0 1 5
   
3 6 2 22 RREF 0 0 0 0
4 8 0 16 0 0 0 0
    
1 3 1 3
Example. Let A = . The RREF of A is U = . Is Col(A) = Col(U)? Is
2 6 0 0
Col(AT ) = Col(U T )?
Solution.

Theorem 36. Let A, B be two row-equivalent matrices. Then Col(AT ) = Col(B T ).

Theorem 37. Let A be m × n matrix with rank r. Then the non-zero rows in an echelon
form of A form a basis of Col(AT ), and thus dim(AT ) = r .
  
1 2 0 4
2 4 −1 3 
Example. Find a basis for Col(AT ) where A = 
3
.
6 2 22
4 8 0 16
Solution.
   
1 2 0 4 1 2 0 4
2 4 −1 3  0 0 1 5
   
3 6 2 22 RREF 0 0 0 0
4 8 0 16 0 0 0 0

Theorem 38. Let A be an m × n matrix with rank r . Then

○ dim Col(A) = dim Col(AT ) = r .
○ dim Nul(A) = n − r .
Question. What is dim Nul(AT )? What is dim Col(A) + dim Nul(A)?
Solution.
 LINEAR ALGEBRA
Graphs

ILLINOIS
Department of Mathematics
Definition. A graph is a set of nodes (or: vertices) that are connected through edges.
Example. A graph with 4 nodes and 5 edges:

1 2  
0 1 1 0
1 1 1 0
 
1 1 0 1
0 0 1 0
3 4

Definition. Let G be a graph with n nodes. The adjacency matrix of G is the n × n-matrix
A = (aij ) such that
(
1 if there is an edge between node i and node j
aij =
0 otherwise .
Definition. A walk of length k on a graph of is a sequence of k + 1 vertices and k edges
between two nodes (including the start and end) that may repeat.A path is walk in which all
vertices are distinct.
Example. Count the number of walks of length 2 from node 2 to node 3 and the number of
walks of length 3 from node 3 back to node 3:

1 2

○ Node 2 to Node 3: 2 walks of length 2

○ Node 3 to Node 3: 3 walks of length 3

3 4

Definition. A graph is connected if for every pair of nodes i and j there is a walk from node
i to node j. A graph is disconnected if it is not connected.
Theorem 39. Let G be a graph and let A be its adjacency matrix. Then the entry in the
i-th row and j-th column of A` is the number of walks of length ` from node j to node i on G.
Proof.
Definition. A directed graph is a set of vertices connected by edges, where the edges have a
direction associated with them.
Example. A graph with 4 nodes and 5 edges:
1
1 2  
0 0 0 0
1 0 1 0
3  
2 4 1 0 0 0
0 1 1 0
5
3 4
We can also talk about adjacency matrices of directed graphs. We use the following
convention:
Definition. Let G be a directed graph with m edges and n nodes. The adjacency matrix of
G is the n × n matrix A = (ai,j )i,j with

1, if there is a directed edge from node j to node i
ai,j =
0, otherwise
Directed graphs have another important associated matrix:
Definition. Let G be a directed graph with m edges and n nodes. The edge-node incidence
matrix of G is the m × n matrix A = (ai,j )i,j with

 −1, if edge i leaves node j
ai,j = +1, if edge i enters node j
0, otherwise


Example. A graph with 4 nodes and 5 edges:

1
1 2
 
−1 1 0 0
−1 0 1 0
 
3 0 1 −1 0
2 4 
 0 −1 0

1
0 0 −1 1
5
3 4
Definition. A connected component of an undirected graph is a part in which any two
vertices are connected to each other by paths, and which is connected to no additional vertices
in the rest of the graph. The connected components of a directed graph are those of its
underlying undirected graph. A graph is connected if only has one connected component.
Example.

A graph with one connected component: A graph with two connected components:
1 1 2
1 2

3 1 2
2 4

5 3 4
3 4
Theorem 40. Let G be a directed graph and let A be its edge-node incidence matrix. Then
dim Nul(A) is equal to the number of connected components of G.
 1 2
Example. Find a basis of the null space of the edge-node
incidence matrix of the following graph: 1 2

3 4
Solution.
Definition. A cycle in an undirected graph is a path in which all edges are distinct and the
only repeated vertices are the first and last vertices. By cycles of a directed graph we mean
those of its underlying undirected graph.
Example. Find cycles in the following graph.
Solution.
Definition. The span of all cycle vectors of a graph G is called the cycle space.
Theorem 41. Let G be a directed graph and let A be its edge-node incidence matrix. Then
the cycle space of G is equal to Nul(AT ).
Example. We explain the idea in the case of the following graph G. Let A be its edge-node
incidence matrix, a 5 × 4-matrix. Note that Nul(AT ) ⊆ R5 .
 
1 y1
1 2 y2 
3 Think of y = y3  as assigning a flow to each edge. If y ∈ Nul(AT ):
 
2 4 y 
4
5 y
3 4 5 
   
0  y1  
−1 −1 0 0 0   −y1 − y2
0  y
 2 

−1 1 0 0

0 = AT y =  1
   0 1 −1 0   y3  = y1 + y3 − y4 

−1 0 1 0
 
0
0 1 −1 0 −1  y4 
 y2 − y3 − y5 
0 0 0 1 1 y4 + y5
 
0
 1 −1 0  0 y5
 0 −1 0 1
; y ∈ Nul(AT ) if and only if the inflow equals the outflow at each node.
0 0 −1 1
What is the simplest way to balance flow? Assign flow around cycles.
Example (ctd.).
Let’s solve AT y = 0. 1
1 2
   
−1 −1 0 0 0 1 0 1 0 1
1 0 1 −1 0 0 1 −1 0 −1 3
T  ;  2 4

A = 
0 1 −1 0 −1 RREF 0 0 0 1 1
0 0 0 1 1 0 0 0 0 0 5
3 4
     
−y3 − y5 −1 −1
 y3 + y5  1 1
y ∈ Nul(AT ) ; y1 = −y3 −y5 , y2 = y3 +y5 , y4 = −y5 ; y = 
     
 y 3
 = y3  1  +y5  0 
    
 −y5  0 −1
y5 0 1
Cycle1 Cycle3

; Cycle1 and Cycle3 form a basis of the cycle space of G.

 LINEAR ALGEBRA
Orthogonal complements

ILLINOIS
Department of Mathematics
  
1 2
Example. Let A = 2 4. Find bases of Nul(A) and Col(AT ).
3 6
Solution.
x2
Nul(A)
Col(AT )

x1

Definition. Let W be a subspace of Rn . The orthogonal complement of W is the

subspace W ⊥ of all vectors that are orthogonal to W ; that is
W ⊥ := {v ∈ Rn : v · w = 0 for all w ∈ W }.

Remark. Observe that (W ⊥ )⊥ = W .

Theorem 42. Let A be an m × n matrix. Then Nul(A) is the orthogonal complement of
Col(AT ); that is Nul(A) = Col(AT )⊥ .
Proof.

Remark. It follows that

○ Nul(A)⊥ = Col(AT ).
○ Nul(AT ) = Col(A)⊥ .
Theorem 43. Let V be a subspace of Rn . Then dim V + dim V ⊥ = n.
Proof.

   
1 0
Example. Find a basis of the orthogonal complement of span 0 , 1.
Solution. 0 1
 LINEAR ALGEBRA
Coordinates

ILLINOIS
Department of Mathematics
Theorem 44. Let (v1 , . . . , vp ) be a basis of V . Then every vector w in V can be expressed
uniquely as
w = c1 v1 + · · · + cp vp .
Proof.

Definition. Let B = (v1 , v2 , . . . , vp ) be an (ordered) basis of V , and let w ∈ V .The

coordinate vector wB of w with respect to the basis B is
 
c1
c2 
 
wB = c3  , if w = c1 v1 + c2 v2 + · · · + cp vp .
 
 .. 
.
cp
Example. Let V = R2 , and consider the bases
         
1 1 1 0
B := b1 = , b2 = , E := e1 = , e2 = .
1 −1 0 1
 
3
Let w = . Determine wB and wE .
−1
Solution.

x2 x2

b1 e2
x1
x1 e1
b2
Definition. In Rn let ei denote the vector with a 1 in the i-th coordinate and 0’s elsewhere.
The standard basis of Rn is the ordered basis En := (e1 , . . . , en ).
Question. For all v ∈ Rn , we have v = vEn . Why?
Solution.

    
1 1
Example. Consider the basis B := b1 = , b2 = of R2 . Let v ∈ R2 be such that
1 −1
 
2
vB = . Can you determine v?
1
Solution.
Definition. Let B and C be two bases of Rn . The change of basis matrix IC,B is the matrix
such that for all v ∈ Rn
IC,B vB = vC

Theorem 45. Let B = (b1 , . . . , bn ) be a basis of Rn . Then

 
IEn ,B = b1 . . . bn

That is, for all v ∈ Rn ,  

v = b1 . . . bn vB .
Proof.
Question. Let B = (b1 , . . . , bn ) be a basis of Rn . How do you compute IB,En ?
Solution.

Question. Let B and C be two bases of Rn . How do you compute IB,C ?

Solution.

Remark. Another way to compute IC,B :

 
IC,B = (b1 )C . . . (bn )C
 LINEAR ALGEBRA
Orthogonal and Orthonormal bases

ILLINOIS
Department of Mathematics
Theorem 46. Let v1 , . . . , vm ∈ Rn be non-zero and pairwise orthogonal.Then v1 , . . . , vm are
linearly independent.
Proof.

Remark. The theorem implies that a set of n orthonormal vectors in Rn is a basis of Rn .

Definition. An orthogonal basis (an orthonormal basis) is an orthogonal set of vectors (an
orthonormal set of vectors) that forms a basis.
Theorem 47. Let B := (b1 , b2 , . . . , bn ) be an orthogonal basis of Rn , and let v ∈ Rn . Then
v · b1 v · bn
v= b1 + . . . + bn .
b1 · b1 bn · bn
Solution.

Remark. When B is orthonormal, then bi · bi = 1 for i = 1, . . . , n.

     
1 1 1 −1
Example. Let U be the orthonormal basis u1 = 2 √ , u2 = 2
√ of R2 . Let
1 1
 
2
v= . Determine vU using the formula from the previous theorem!
3
Solution.

Example. With U as above, compute the change of basis matrix IU ,E2 ?

Solution.
Theorem 48. Let U = (u1 , . . . , un ) be an orthonormal basis of Rn . Then
 T
IU ,En = u1 . . . un .
Proof.

Definition. An n × n-matrix Q is orthogonal if Q −1 = Q T .

Remark. The columns of an orthogonal matrix form an orthonormal basis. Why?
Solution.
 LINEAR ALGEBRA
Linear Transformation

ILLINOIS
Department of Mathematics
Definition. Let V and W be vector spaces. A map T : V → W is a linear transformation if

T (av + bw) = aT (v) + bT (w)

for all v, w ∈ V and all a, b ∈ R.

Example. Let V = R, W = R. Then the map f (x) = 3x is linear, g (x) = 2x − 2 is not.
Why?
Solution.

Remark. T (0V ) = T (0 · 0V ) = 0 · T (0V ) = 0W ; T (0V ) = 0W

Question. Let A be an m × n matrix, and consider the map T : Rn → Rm given by
T (v) := Av. Is this a linear transformation?
Solution.

Example. Let Pn be the vector space of all polynomials of degree at most n. Consider the
d
map T : Pn → Pn−1 given by T (p(t)) := dt p(t). This map is linear! Why?
Solution.
Theorem 49. Let V , W be two vector spaces, let T : V → W be a linear transformation
and let (v1 , . . . , vn ) be a basis of V . Then T is completely determined by the values
T (v1 ), . . . , T (vn ).
Solution.
  
  1
2 3 1
Example. Let T : R → R be a linear transformation with T = 2 and

0
3
 
  0  
0 1
T =  0 . What is T ?
1 2
−2
Solution.
Theorem 50. Let T : Rn → Rm be a linear transformation. Then there is a m × n matrix A
such that
○ T (v) = Av, for all v ∈ Rn .
○ A = T (e1 ) T (e2 ) . . . T (en ) , where (e1 , e2 , . . . , en ) is the standard basis of Rn .
 

Proof.

Remark. We call this A the coordinate matrix of T with respect to the standard bases - we
write TEm ,En .
Example. Let Tα : R2 → R2 be the “rotation over α radians (counterclockwise)” map, that
is Tα (v) is the vector obtained by rotating v over angle α. Find the 2 × 2 matrix Aα such that
Tα (v) = Aα v for all v ∈ R2 .
Solution.

x2

e2
Tα (e2 ) Tα (e1 )
e1 x1
 LINEAR ALGEBRA
Coordinate matrices of linear transformations

ILLINOIS
Department of Mathematics
Last time: Linear transformation is matrix multiplication. For every linear transformation
T : Rn → Rm , there is an m × n matrix A such that T (v) = Av.
Today: The same in abstract vector spaces.
Theorem 51. Let V , W be two vector space, let B = (b1 , . . . , bn ) be a basis of V and
C = (c1 , . . . , cm ) be a basis of W , and let T : V → W be a linear transformation. Then there
is a m × n matrix TC,B such that
○ T (v)C = TC,B vB , for all v ∈ V .
 
○ TC,B = T (b1 )C T (b2 )C . . . T (bn )C .

apply T
v : vector in V / vector in W : T (v)

write in coordinates wrt B write in coordinates wrt C

 multiply by TC,B 
vB : coordinate vector in Rn / coordinate vector in Rm : TC,B v
 d
Example. Let D : P2 → P 1 be given by D(p(t)) = dt p(t). Consider the bases B = (1, t, t 2 )
and C = (1, t) of P2 and P1 . Determine DC,B .
Solution.

Question. Consider p(t) = 2 − t + 3t 2 in P2 . Compute D(p(t))C and DC,B p(t)B .

Solution.
  
3 1
Example. Let T : R2→ R2be such that T (v) = v. Consider
1 3
    
1 1
B := b1 = , b2 = . Compute TB,B . Let v = b1 + b2 . Use TB,B to compute T (v).
−1 1
Solution.
Theorem 52. Let T : Rm → Rn be a linear transformation and A and B be two bases of
Rm and C, D be two bases of Rn . Then

TC,A = IC,D TD,B IB,A .

Proof.

apply TC,A
(Rm , A) / (Rn , C)
O
IB,A IC,D
 apply TD,B
(Rm , B) / (Rn , D)
        
1 0 1 1
Example. Consider E := { , } and B := { , } as before. Let T : R2 → R2 be
0 1 −1 1
 
3 1
again the linear transformation that v 7→ v. Determine TB,B .
1 3
Solution.
 LINEAR ALGEBRA
Determinants

ILLINOIS
Department of Mathematics
Definition. The determinant
 of
a b
○ a 2 × 2 matrix is det = ad − bc,
c d
○ a 1 × 1 matrix is det([a]) = a.
Remark. Recall that  −1  
a b 1 d −b
= .
c d ad − bc −c a

Goal. Define the determinant of an n × n-matrix such that

A is invertible ⇐⇒ det(A) 6= 0.

 
a b a b
Notation. We will write both det and for the determinant.
c d c d
Definition. The determinant is the operation that assigns to each n × n-matrix a number
and satisfies the following conditions:
○ (Normalization) det In = 1,
○ It is affected by elementary row operations as follows:
o (Replacement) Adding a multiple of one row to another row does not change the
determinant.
o (Interchange) Interchanging two different rows reverses the sign of the determinant.
o (Scaling) Multiplying all entries in a row by s, multiplies the determinant by s.
 
2 3 3
Example. Compute det 0 1 2 .
0 0 6
Solution.
Theorem 53. The determinant of a triangular matrix is the product of the diagonal entries.
1 2 0
Example. Compute 3 −1 2 .
2 0 1
Solution.
 a b
Example. (Re)Discover the formula for .
c d
Solution.

Example. Suppose A is a 3 × 3 matrix with det(A) = 5. What is det(2A)?

Solution.
Theorem 54. Let A be an n × n-matrix. Then det(A) = 0 if and only if A is not invertible.
Proof.

Theorem 55. Let A, B be two n × n-matrices. Then det(AB) = det(A) det(B).

Example. If A is invertible, then det(A−1 ) = 1
det(A) . Why?
Solution.
Theorem 56. Let A be an n × n-matrix. Then det(AT ) = det(A).
Proof.

Remark. det(AT ) = det(A) means that everything you know about determinants in terms of
rows of A is also true for the columns of A. In particular:
○ If you exchange two columns in a determinant, the determinant changes by a factor of −1.
○ You can add a multiple of a column to another column without changing the determinant.
○ Multipying each entry of a column by a scalar s, change the determinant by a factor of s.
 LINEAR ALGEBRA
Cofactor expansion

ILLINOIS
Department of Mathematics
Notation. Let A be an n × n-matrix. We denote by Aij the matrix obtained from matrix A by
deleting the i-th row and j-th column of A.
 
1 2 3 4
5 6 7 8
Example. Let A =   9 10 11 12. Find A23 and A43 .


13 14 15 16
Solution.
Definition. Let A be an n × n-matrix. The (i, j)-cofactor of A is the scalar Cij defined by
Cij = (−1)i+j det Aij .
Theorem 57. Let A be an n × n-matrix. Then for every i, j ∈ {1, . . . , n}
det A = ai1 Ci1 + ai2 Ci2 + · · · + ain Cin (expansion across row i)
= a1j C1j + a2j C2j + · · · + anj Cnj (expansion down column j)
1 2 0
Example. Compute 3 −1 2 by cofactor expansion across row 1.
2 0 1
Solution.
 1 2 0
Example. Compute 3 −1 2 by cofactor expansion down column 2 and by cofactor
2 0 1
expansion down column 3.
Solution.
Question. Why is the method of cofactor expansion not practical for large n?
Solution.

P.-S. Laplace (1749–1827)

Remark. The cofactor expansion is also called Laplace expansion, because it is due
Pierre-Simon Laplace.
 LINEAR ALGEBRA
Eigenvectors and Eigenvalues

ILLINOIS
Department of Mathematics
Definition. Let A be an n × n matrix. An eigenvector of A is a nonzero v ∈ Rn such that
Av = λv,

where λ is a scalar, known

  as the eigenvalue associated
 with v.  
1 0 −2 0
Example. Verify that is an eigenvector of A = . Is an eigenvector?
1 −4 2 1
Solution.

x2

v2 v1

x1
  
0 1
Example. Find the eigenvectors and the eigenvalues of A = .
1 0
Solution.

x2

x1
  
1 0
Example. Find the eigenvectors and the eigenvalues of B = .
0 0
Solution.

x2

x1
Definition. Let λ be an eigenvalue of A. The eigenspace of A associated with λ is the set

Eigλ (A) := {v : Av = λv}.

It consists of all the eigenvectors of A with eigenvalue λ and the zero vector.
   
0 1 1 0
Example. Draw the eigenspaces of the two matrices A = and B = .
1 0 0 0
Solution.

x2 x2

x1 x1
 LINEAR ALGEBRA
Computing Eigenvalues and Eigenvectors

ILLINOIS
Department of Mathematics
Theorem 58. Let A be an n × n matrix and λ be a scalar. Then λ is an eigenvalue of A if
and only if det(A − λI ) = 0.
Proof.

Theorem 59. Let A be an n × n matrix. Then pA (t) := det(A − tI ) is a polynomial of

degree n. Thus A has at most n eigenvalues.
Definition. We call pA (t) the characteristic polynomial of A.
  
3 1
Example. Find the eigenvalues of A = .
1 3
Solution.
     
3 1 1 0 3−λ 1
A − λI = −λ =
1 3 0 1 1 3−λ

3−λ 1
det(A − λI ) = = (3 − λ)2 − 1
1 3−λ

= λ2 − 6λ + 8 = 0 ; λ1 = 2, λ2 = 4

Theorem 60. Let A be n × n matrix and let λ be eigenvalue of A. Then

Eigλ (A) = Nul(A − λI ).

 
3 1
Example. Determine the eigenspaces of A = .
1 3
Solution.

Recall that the eigenvalues of A are λ1 = 2, λ2 = 4.

  
3 2 3
Example. Find the eigenvalues and eigenspaces of A = 0 6 10 .
0 0 2
Solution.
 LINEAR ALGEBRA
Properties of Eigenvectors and Eigenvalues

ILLINOIS
Department of Mathematics
Definition. Let A be an n × n matrix and let λ be an eigenvalue of A.
○ The algebraic multiplicity of λ is its multiplicity as a root of the characteristic
polynomial, that is, the largest integer k such that (t − λ)k divides pA (t).
○ The geometric multiplicity of λ is the dimension of the eigenspace Eigλ (A) of λ.
 
1 1
Example. Find the eigenvalues of A = and determine their algebraic and geometric
0 1
multiplicities.
Solution.
Theorem 61. Let A be an n × n matrix and let v1 , . . . , vm be eigenvectors of A
corresponding to different eigenvalues. Then v1 , . . . , vm are linearly independent.
Proof.
  
a11 a12 . . . a1n
a21 a22 . . . a2n 
Definition. Let A = 
 ..
 .

... . . . ... 

. The trace of A is the sum of the diagonal entries of A;
an1 an2 . . . ann
that is
Tr(A) = a11 + a22 + · · · + ann .

Theorem 62. Let A be n × n matrix with eigenvalues λ1 , . . . , λn . Then

○ Tr(A) = λ1 + λ2 + · · · + λn .
○ det(A) = λ1 · λ2 · · · · · λn .
Theorem 63. Let A be a 2 × 2-matrix. Then the characteristic polynomial of A is
p(λ) = λ2 − Tr(A)λ + det(A).

Solution.
 LINEAR ALGEBRA
Markov matrices

ILLINOIS
Department of Mathematics
Definition. An n × n matrix A is a Markov matrix (or: stochastic
matrix) if it has only non-negative entries, and the entries in each column
add up to 1.
A vector in Rn is a probability vector (or: stochastic vector) if has only
non-negative entries, and the entries add up to 1.

Example. Give examples of Markov matrices and probability vectors.

A. Markov (1856–1922)
Solution.
Question. Let A be n × n Markov matrix and v ∈ Rn be a probability vector. Is Av also a
probability vector?
Solution.

Theorem 64. Let A be a Markov matrix.Then

○ 1 is an eigenvalue of A and every other eigenvalue λ of A satisfies |λ| ≤ 1.
○ If A has only positive entries, then any other eigenvalue satisfies |λ| < 1.
Definition. A stationary probability vector of a Markov matrix is probability vector v that is
an eigenvector of A corresponding to the eigenvaule 1.
Theorem 65. Let A be an n × n-Markov matrix with only positive entries and let z ∈ Rn be
a probability vector. Then
z∞ := lim Ak z exists,
k→∞

and z∞ is a stationary probability vector of A (ie. Az∞ = z∞ ).

Proof.
Example. Consider a fixed population of people with or without job. Suppose that each year,
50% of those unemployed find a job while 10% of those employed lose their job.
What is the unemployment rate in the long term equilibrium?
Solution.
 LINEAR ALGEBRA
Diagonalization

ILLINOIS
Department of Mathematics
Definition. A square matrix A is said to be diagonalizable if there is a invertible matrix P
and a diagonal matrix D such that A = PDP −1 .
Theorem 66. Let A be an n × n matrix that has n linearly independent eigenvectors
v1 , v2 , . . . , vn with associated eigenvalues λ1 , . . . , λn . Then A is diagonalizable as PDP −1 ,
where  
λ1
 
P = v1 . . . vn and D = 
 .. 
. 
λn
Proof.
  
6 −1
Example. Diagonalize A = .
2 3
Solution.
Definition. Vectors v1 , . . . , vn form an eigenbasis of n × n matrix A if v1 , . . . , vn form a
basis of Rn and v1 , . . . , vn are all eigenvectors of A.
Theorem 67. The following are equivalent for n × n matrix A:
○ A has an eigenbasis.
○ A is diagonalizable.
○ The geometric multiplicities of all eigenvalues of A sums up to n.
Theorem 68. Let A be n × n matrix and let B = (v1 , . . . , vn ) be an eigenbasis of A. Then
there is a diagonal matrix D such that
A = IEn ,B DIB,En .
Proof.
Question. Let A be an n × n with n distinct eigenvalues. Is A diagonalizable?
Solution.

 
0 1
Question. Is A = diagonalizable?
0 0
Solution.
 LINEAR ALGEBRA
Powers of Matrices

ILLINOIS
Department of Mathematics
Idea. If A has an eigenbasis, then we can raise A to large powers easily!
Theorem 69. If A = PDP −1 , where D is a diagonal matrix, then for any m,
Am = PD m P −1
Proof.

Remark. Finding D m is easy!

m
(λ1 )m
  
λ1
m
D =
 .. 
=
 .. 
.  . 
λn (λn )m
 1
 
20 0
Example. Let A = − 21 1 6. The matrix has the following eigenvectors:
0 0 2
     
1 0 0
1
v1 = 1 with λ1 = , v2 = 1 with λ2 = 1, v3 = 6 with λ3 = 2.
2
0 0 1
Find A100 .
Solution.
Example.
  Let A be a 2 × 2 matrix such that  
1 1 1
is an eigenvector with eigenvalue 2 , is an eigenvector with eigenvalue 1.
1 0
 
2
Let v = . Graphically determine how An v behaves as n → ∞.
1
Solution.
 LINEAR ALGEBRA
Matrix exponential

ILLINOIS
Department of Mathematics
Definition. Let A be an n × n-matrix. We define the matrix exponential e At as
(At)2 (At)3
e At = I + At + + + ...
2! 3!
 
2 0
Example. Compute e At for A = .
0 1
Solution.

Theorem 70. Let A be an n × n matrix. Then

○ The series in the definition of e At always converges, ○ e At e As = e A(t+s) ,
○ e At e −At = In , ○ d At
dt (e ) = Ae At .
Theorem 71. Let A be a n × n matrix such that A = PDP −1 for some invertible matrix P
and some diagonal matrix D. Then

e At = Pe Dt P −1 .
Proof.
      −1
−2 1 1 1 −1 0 1 1
Example. Let A = = . Compute e At .
1 −2 1 −1 0 −3 1 −1
Solution.
 LINEAR ALGEBRA
Linear Differential Equations

ILLINOIS
Department of Mathematics
Definition. A linear (first order) differential equation is an equation of the form

du
= Au
dt
where u is function from R to Rn . A further condition u(0) = v, for some v in Rn is called a
initial condition.
Theorem 72. Let A be an n × n matrix and v ∈ Rn . The solution of the differential
equation du At
dt = Au with initial condition u(0) = v is u(t) = e v.
du
Example. Let n = 1. Find the solution of the differential equation dt = −u and u(0) = 1.
Solution.
Theorem 73. Let A be an n × n-matrix, and let v ∈ Rn be an eigenvector of A with
eigenvalue λ. Then e At v = e λt v.
Proof.
Theorem 74. Let A be an n × n-matrix and (v1 , . . . , vn ) be an eigenbasis of A with
eigenvalues λ1 , . . . , λn . If v = c1 v1 + · · · + cn vn ,then the unique solution to the differential
equation du
dt = Au with initial condition u(0) = v is

e At v = c1 e λ1 t v1 + · · · + cn e λn t vn .

Proof.
  
0 1 du
Example. Let A = . Solve the differential equation dt = Au with initial condition
1 0
 
1
u(0) = .
0
Solution.
  
1 2 du
Example. Let A = . Solve the differential equation dt = Au with initial condition
2 1
 
0
u(0) = .
1
Solution.
 LINEAR ALGEBRA
Orthogonal projection onto a line

ILLINOIS
Department of Mathematics
Definition. Let v, w ∈ Rn . The orthogonal projection of v onto the line spanned by w is
w·v
projw (v) := w.
w·w

Theorem 75. Let v, w ∈ Rn . Then projw (v) is the point in span(w) closest to v; that is
dist(v, projw (v)) = min dist(v, u).
u∈span(w)

Proof.
x2
w
v
projw (v)
x1
Remark. Note that v − projw (v) (called the error term) is in span(w)⊥ .
v = projw (v) + v − projw (v)
| {z } | {z }
∈span(w) ∈span(w)⊥

  
−2 3
Example. Find the orthogonal projection of v = onto the line spanned by w = .
1 1
Solution.

x2

v w

x1
Theorem 76. Let w ∈ Rn . Then for all v ∈ Rn
 
1 T
projw (v) = ww v.
w·w
Proof.

1 T


Remark. Note that w·w ww is an n × n matrix, we call the orthogonal projection
matrix onto span(w).
  
1
Example. Let w = . Find the orthogonal projection matrix P onto span(w). Use it to
1
     
1 1 1
calculate the projections of , , onto span(w).
0 1 −1
Solution.
 LINEAR ALGEBRA
Orthogonal projection onto a subspace

ILLINOIS
Department of Mathematics
Theorem 77. Let W be a subspace of Rn and v ∈ Rn . Then v can be written uniquely as
v = |{z} v⊥
v̂ + |{z}
in W in W ⊥
Proof.

w3
v W
v⊥
w1
w2 v̂

Definition. We say v̂ is the orthogonal projection of v onto W - written projW (v).

Remark. If (w1 , . . . , wm ) is an orthogonal basis of W , then
   
v · w1 v · wm
projW (v) = w1 + . . . + wm .
w1 · w 1 wm · wm
      
 3 0  0
Example. Let W = span 0 , 1
   and v = 3 . Compute projW (v).

1 0 10
|{z} |{z}
=:w1 =:w2
Solution.
Theorem 78. The projection map projW : Rn → Rn that sends v to projW (v) is linear.
Definition. The matrix PW is the matrix (projW )En ,En that represents projW with respect to
the standard basis. We call PW the orthogonal projection matrix onto W .
   
3 0
Example. Compute PW for W = span 0 , 1.
Solution. 1 0
Question. Let PW be the orthogonal projection matrix onto W in Rn , and let v ∈ Rn .
○ If PW v = v, what can you say about v? ○ If PW v = 0, what can you say about v?
Solution.

Question. What is the orthogonal projection matrix PW ⊥ for projecting onto W ⊥ ?

Solution.
 LINEAR ALGEBRA
Least squares solutions

ILLINOIS
Department of Mathematics
Goal. Suppose Ax = b is inconsistent. Can we still find something like a best solution?
Definition. Let A be an m × n matrix and b ∈ Rm . A least squares solution (short: LSQ
solution) of the system Ax = b is a vector x̂ ∈ Rn such that

dist(Ax̂, b) = minn dist(Ax, b).

x∈R

Theorem 79. Let A be an m × n matrix and b ∈ Rm . Then x̂ is an LSQ solution to Ax = b

if and only if Ax̂ = projCol(A) (b).
Proof.
    
1 1 2
Example. Let A = −1 1 and b = 1. Find an LSQ solution of Ax = b.
0 0 1
Solution.
Theorem 80. Let A be an m × n matrix and b ∈ Rm . Then x̂ is an LSQ solution to Ax = b
if and only if AT Ax̂ = AT b.
Proof.

Theorem 81. Let A be an m × n matrix with linearly independent columns and b ∈ Rm .

Then projCol(A) (b) = A(AT A)−1 AT b.
Proof.
    
4 0 2
Example. Let A = 0 2 and b = 0 . Find a LSQ solution of Ax = b.
  
1 1 11
Solution.
 LINEAR ALGEBRA
Linear Regression

ILLINOIS
Department of Mathematics
Example. Are there β1 , β2 ∈ R such y
that the data points × ×
×
(x1 , y1 ) = (2, 1), (x2 , y2 ) = (5, 2),
(x3 , y3 ) = (7, 3), (x4 , y4 ) = (8, 3) ×

all lie on the line y = β1 + β2 x. x

Solution.
 y
Example (ctd). Find β1 , β2 such that × ×
the line y = β1 +β2 x best fits the data
points: ×
(x1 , y1 ) = (2, 1), (x2 , y2 ) = (5, 2), ×
(x3 , y3 ) = (7, 3), (x4 , y4 ) = (8, 3) x

Solution.
Example. A scientist tries to find the relation between the y
mysterious quantities x and y . She measures the following values: × ×

x −2 −1 0 1 2
y 5 2.5 2.25 2 5 × ×
×

Find β1 , β2 , β3 such that y = β1 + β2 x + β3 x 2 best fits the data.

x
Solution.
Question. Suppose you are given a data Question. Suppose you are given a data
set (ui , vi , yi ), i = 1, . . . , n. You expect set (xi , yi ), i = 1, . . . , n. You expect that y
that y depends linearly on u and v . Which depends linearly on x and sin(x). Which lin-
linear system do you have to solve to deter- ear system to you have to solve to determine
mine this dependency? this dependency?
Solution. Solution.
 LINEAR ALGEBRA
Gram-Schmidt Method

ILLINOIS
Department of Mathematics
Theorem 82. Every subspace of Rn has an orthonormal basis.
Algorithm. (Gram-Schmidt orthonormalization) Given a basis a1 , . . . , am , produce an
orthogonal basis b1 , . . . , bm and an orthonormal basis q1 , . . . , qm .
b1
b1 = a1 , q1 = kb1 k
b2 q3
b2 = a2 − projspan(q1 ) (a2 ), q2 = kb2 k
| {z } a3 span(a1 , a2 )
=(a2 ·q1 )q1
b3 q1
b3 = a3 − projspan(q1 ,q2 ) (a3 ) q3 = kb3 k
| {z } q2
(a3 ·q1 )q1 +(a3 ·q2 )q2
··· ···
Remark.
○ span(q1 , . . . , qi ) = span(a1 , . . . , ai ) for i = 1, . . . , m.
○ qj ∈
/ span(a1 , . . . , ai ) for all j > i.
    
2 0
Example. Let V = span  1 , 0. Use the Gram-Schmidt method to find an
 
2 3
orthonormal basis of V .
Solution.
Theorem 83. (QR decomposition) Let A be an m × n matrix of rank n. There is is an
m × n-matrix Q with orthonormal columns and an upper triangular n × n invertible matrix R
such that A = QR.
Proof.
  
1 2 4
Example. Find the QR decomposition of A = 0 0 5.
0 3 6
Solution.
 LINEAR ALGEBRA
Spectral Theorem

ILLINOIS
Department of Mathematics
Theorem 84. Let A be a symmetric n × n matrix. Then A has an orthonormal basis of
eigenvectors.
Proof.
Theorem 85. Let A be a symmetric n × n matrix. Then there is a diagonal matrix D and a
matrix Q with orthonormal columns such that A = QDQ T .
Proof.

Question. If A is an n × n matrix with an orthonormal eigenbasis, is it symmetric?

Solution.
  
3 1
Example. Let A = . Write A as QDQ T , where D is diagonal and Q has orthonormal
1 3
columns.
Solution.
         
3 1 1 1 1 2 0 √1 1 −1 √1
1
Example. Recall that A = = 2
√ , and q1 =
1 3 −1 1 0 4 2 1 1 2 −1
 
1
and q2 = √12 . Suppose v = −q1 + 12 q2 . Graphically, determine Av.
1
Solution.

x2 Aq2 x2

q2 q2

x1 x1
q1 q1
Aq1
 LINEAR ALGEBRA
Singular Value Decomposition

ILLINOIS
Department of Mathematics
Definition. Let A be an m × n matrix. A singular value decomposition of A is a
decomposition A = UΣV T where
○ U is an m × m matrix with orthonormal columns,
○ Σ is an m × n rectangular diagonal matrix with non-negative numbers on the diagonal,
○ V is an n × n matrix with orthonormal columns.
Remark. The diagonal entries σi = Σii which are positive are called the singular values of A.
We usually arrange them in decreasing order, that is σ1 ≥ σ2 ≥ . . .
Question. Let A be an m × n matrix with rank r . Recall why
○ Nul(AT A) = Nul(A) and
○ AT A is symmetric and has rank r .
Nul(AAT ) = Nul(AT ).
Solution.
Algorithm. Let A be an m × n matrix with rank r .
○ Find orthonormal eigenbasis (v1 , . . . , vn ) of AT A with eigenvalues
λ1 ≥ · · · ≥ λr > λr +1 = 0 = · · · = λn .
√
○ Set σi = λi for i = 1, . . . , n.
○ Set u1 = σ11 Av1 , . . ., ur = σ1r Avr . (Magic: orthonormal!)
○ Find ur +1 , . . . , um ∈ Rm such that (u1 , . . . , um ) is an orthonormal basis of Rm .
○ Set  
σ1

U = u1 . . . um ,

Σ=
 .. 
 , V = v1 . . . vn
 
.
σmin{m,n}

Proof that A = UΣV T .

  
−1 1 0
Example. Compute the SVD of A = .
0 −1 1
Solution.
  
Theorem 86.  Let A be an m × n matrix with rank r , and let U = u1 . . . um ,
V = v1 . . . vn , Σ be such that A = UΣV T is a SVD of A. Then


○ (u1 , . . . , ur ) is a basis of Col(A). ○ (v1 , . . . , vr ) is a basis of Col(AT ).

○ (ur +1 , . . . , um ) is a basis of Nul(AT ). ○ (vr +1 , . . . , vn ) is a basis of Nul(A).
Proof.
 LINEAR ALGEBRA
Low rank approximation via SVD

ILLINOIS
Department of Mathematics
  
Theorem
 87.  Let A be an m × n matrix with rank r , and let U = u1 . . . um ,
V = v1 . . . vn be matrices with with orthonormal columns and Σ be a rectangular diagonal
m × n matrix such that A = UΣV T is an SVD of A. Then

A = σ1 u1 v1T + σ2 u2 v2T + · · · + σr ur vrT

 
σ1

= u1 . . . ur 
 .. 
 v1 . . . vr
T
.
σr
Proof.
Example. Use
 1
− √26 √1

" # √  √6
− √12 √1 6
 
−1 1 0 3 0 0  √1 √1 
= √1 √1
2
− 2 0
0 −1 1 0 1 0 1
2
2 2 √ √1 √1
3 3 3
 
−1 1 0
to write as a sum of rank 1 matrices.
0 −1 1
Solution.
Definition. For k ≤ r , define

Ak = σ1 u1 v1T + σ2 u2 v2T + · · · + σk uk vkT .

Idea. If σ1  σ2  . . . ..., then Ak is a good approximation of A.

Definition. Let A be an m × n matrix with rank r . A compact singular value
decomposition of A is a decomposition A = Uc Σc VcT where
 
○ Uc = u1 . . . ur is an m × r matrix with orthonormal columns,
○ Σc is an r × r diagonal matrix with positive diagonal elements,
 
○ Vc = v1 . . . vr is an n × r matrix with orthonormal columns.
Question. What is Col(Vc )? What is Col(Uc )?
Solution.
Example. Use
 1
− √26 √1

" # √  √6
− √12 √1 6
 
−1 1 0 3 0 0  √1 √1 
= √1 √1
2
− 2 0
0 −1 1 0 1 0 1
2
2 2 √ √1 √1
3 3 3
 
−1 1 0
to find a compact SVD of .
0 −1 1
Solution.
 LINEAR ALGEBRA
The Pseudo-Inverse

ILLINOIS
Department of Mathematics
Definition. Let A be an m × n matrix with rank r . Given the compact singular value
decomposition T
 A = UcΣc Vc where
○ Uc = u1 . . . ur is an m × r matrix with orthonormal columns,
○ Σc is an r × r diagonal
 matrix with positive diagonal elements,
○ Vc = v1 . . . vr is an n × r matrix with orthonormal columns,
we define the pseudoinverse A+ of A as Vc Σ−1 T
c Uc .
Example. Recall that
T
√1 − √12

 " √1 √1
# √
− 6
 
−1 1 0 3 0 − √2
A := = √1 2 √1
2 0   .
0 −1 1 0 1 6

2 2 √1 √1
6 2
Determine A+ .
Solution.
Theorem 88. Let v ∈ Col(AT ) and w ∈ Col(A). Then A+ Av = v and AA+ w = w.
Proof.
Remark. If A is n × n and invertible, then Col(A) = Rn . Thus A−1 = A+ .
Question. Let v ∈ Rn such that vr + vn = v . What is A+ Av?
|{z} |{z}
∈Col(AT ) ∈Nul(A)

Solution.

Theorem 89. Let A be an m × n matrix and let b ∈ Rm . Then A+ b is the LSQ solution of
Ax = b (with minimum length).
Proof.

Remark. This is particularly useful, when solving many different LSQ problems of the form
Ax = b, where A stays the same, but b varies.
 LINEAR ALGEBRA
PCA

ILLINOIS
Department of Mathematics
Setup.
○ Given m objects, we measure the same n variables.
○ Thus m samples of n-dimensional data ; m × n matrix (each row is a sample)
○ Analyse this matrix to understand what drives the variance in the data.
 T
Definition. Let X = a1 . . . am be an m × n matrix. We define the column average
µ(X ) of X as
1
µ(X ) := (a1 + · · · + am ).
m
We say X is centered if µ(X ) = 0.
1
For X centered, we define the covariance matrix cov(X ) of X as m−1 XTX.
Remark.
 T
○ Not centered, replace X by a1 − µ(X ) . . . am − µ(X ) .
○ If the columns of X are orthogonal, then cov(X ) is a diagonal matrix ; each variable is
independent.
○ What if not? Idea: Pick an eigenbasis of X T X .
Principal component analysis
○ Input: centered m × n-matrix X .
○ Compute cov(X ).
○ Since cov(X ) is symmetric, we can find an orthonormal eigenbasis v1 , v2 , . . . , vn of cov(X )
with eigenvalues λ1 ≥ · · · ≥ λn ≥ 0.
○ Write cov(X ) as a sum of rank 1 matrices:

cov(X ) = λ1 v1 v1T + · · · + λn vn vnT .

○ Each principal component vi explains part of the variance of the data. The larger λi , the
more of the variances is explained by vi .
Example. Let X be a centered 30 × 2-matrix. We
plot the data and see that
 
7.515 20.863
cov(X ) = .
20.863 63.795 x2

An orthonormal eigenbasis is

0.314
 
0.95
 v1
v1 = , v2 =
0.95 −0.314 x1
v2
with λ1 = 70.685 and λ2 = 0.625. Thus
   
6.97 21.085 0.564 −0.186
cov(X ) = + .
21.085 63.793 −0.186 0.0616
| {z } | {z }
λ1 v1 v1T λ2 v2 v2 T
PCA using SVD.
○ Let X be a centered data matrix. Observe that X T X = (m − 1) cov(X ).
○ To find orthonormal eigenbasis of cov(X ), it is enough to find orthonormal eigenbasis of
XTX.
○ Compute the SVD of X ; X = UΣV T .
 
○ The columns of V = v1 . . . vn are the desired orthonormal eigenbasis.
○ If σi is the singular value for vi , then

σi2
λi =
m−1
 LINEAR ALGEBRA
Review of Complex Numbers

ILLINOIS
Department of Mathematics
Definition.
√ C = {x + iy | x, y ∈ R} where
2
i = −1, or i = −1.

○ The real part of z, denoted <(z) is de-

fined by <(z) = x. y

○ The imaginary part of z, denoted Im(z) z = x + iy

is defined by Im(z) = y . Im z = y ◦
○ The complex conjugate of z, denoted z, |z|
is defined by z = x − iy . <z = x
x
○ The absolute value, or magnitude of z,
denoted |z| or kzk, is given by
Im z = −y ◦
p z = x − iy
|z| = x 2 + y 2 .

Any point in 2
x

R can be viewed as a complex
number: y ↔ x + iy
Definition. Given z = x + iy , w = u + iv , Example. Compute i(x + iy ), i 2 (x + iy ) and
we define i 3 (x + iy ) and i 4 (x + iy ).
z + w = (x + u) + i(y + v ) Solution.

zw = (x + iy )(u + iv )
= xu + x(iv ) + (iy )u + (iy )(iv )
= (xu − yv ) + i(xv + yu)

Example. Compute (3 + 2i) + (4 − i) and

(3 + 2i)(4 − i).
Solution.
 =
Theorem 90. Let z ∈ C. z = x + iy
○ z =z ◦
○ |z|2 = zz
<
○ |z| = |z| ◦
z = x − iy
Proof.
 LINEAR ALGEBRA
Complex Linear Algebra

ILLINOIS
Department of Mathematics
Goal. Use complex numbers (instead of real numbers) as scalars.
 
z1
z2 
Definition. The (complex) vector space Cn is of all complex column vectors z =  . ,
 
 .. 
zn
where z1 , z2 , . . . , zn are complex numbers.
○ Now multiplication by a complex scalar makes sense.
○ We can define subspaces, span, independence, basis, dimension for Cn in the usual way.
○ We can multiply complex vectors by complex matrices. Column space and Null space still
make sense.
○ The only difference is the dot product, you need to use the complex conjugate to get a
good notion of length:
   
z1 w1
 ..   .. 
 .  ·  .  = z1 w̄1 + z2 w̄2 + . . . zn w̄n .
zn wn
  
0 −1
Example. Find the complex eigenvectors and eigenvalues of A = .
1 0
Solution.
Definition. Let A be an m × n-matrix. The conjugate matrix A of A is obtained from A by
taking the complex conjugate of each entry of A.
Theorem 91. Let A be a matrix with real entries and λ is an eigenvalue of A. Then λ̄ is
also a eigenvalue. Furthermore, if v is an eigenvector with eigenvalue λ, then v̄ is an
eigenvector with eigenvalue λ̄.
Proof.

T
Definition. Let A be an m × n-matrix. The conjugate transpose AH of A is defined as A .
We say the matrix A is Hermitian if A = AH .
  
1 1
Example. Find the eigenvectors and eigenvalues of A = . Can complex numbers help
0 1
you find an eigenbasis of A?

○ We can not find an eigenbasis for this matrix.

This kind of problem cannot really be fixed.
We have to lower our expectations and look for generalized eigenvectors.
These are solutions to (A − λI )2 x = 0, (A − λI )3 x = 0, . . .