Matrix

3. Matrices
a rectangular array of numbers, for example
 
0 1 −2.3 0.1
A =  1.3 4 −0.1 0 
• notation and terminology 4.1 −1 0 1.7

• matrix operations
• numbers in array are the elements (components, entries, coefficients)
• linear and affine functions
• Aij is the i, j element of A; i is its row index, j the column index
• complexity
• size of the matrix is (#rows) × (#columns), e.g., A is a 3 × 4 matrix

• set of m × n matrices with real elements is written Rm×n

• set of m × n matrices with complex elements is written Cm×n

3-1 Matrices 3-2

Other conventions Matrix shapes

Scalar: we don’t distinguish between a 1 × 1 matrix and a scalar

• many authors use parentheses as delimiters:
  Vector: we don’t distinguish between an n × 1 matrix and an n-vector
0 1 −2.3 0.1
A=  1.3 4 −0.1 0 
4.1 −1 0 1.7 Row and column vectors
• a 1 × n-matrix is called a row vector
• often aij is used to denote the i, j element of A
• an n × 1-matrix is called a column vector (or just vector )

Tall, wide, square matrices: an m × n-matrix is

• tall if m > n
• wide if m < n
• square if m = n

Matrices 3-3 Matrices 3-4

Block matrix Rows and columns

• a block matrix is a rectangular array of matrices a matrix can be viewed as a block matrix with row/column vector blocks
• elements in the array are the blocks or submatrices of the block matrix
• m × n matrix A as 1 × n block matrix
Example    
B C A= a1 a2 · · · an
A=
D E
each aj is an m-vector (the jth column of A)
is a 2 × 2 block matrix; if the blocks are
    • m × n matrix A as m × 1 block matrix
2 0 2 3    
B= , C= , D= 1 , E= −1 6 0  
1 5 4 7 b1
 b2 
 
then   A= . 
2 0 2 3  .. 
A= 1 5 4 7  bm
1 −1 6 0
each bi is a 1 × n row vector (the ith row of A)
Note: dimensions of the blocks must be compatible!

Matrices 3-5 Matrices 3-6

 Special matrices Symmetric and Hermitian matrices

Zero matrix Symmetric matrix: square with Aij = Aji

• matrix with Aij = 0 for all i, j    
4 3 −2 4 + 3j 3 − 2j 0
• notation: 0 (usually) or 0m×n (if dimension is not clear from context)  3 −1 5  ,  3 − 2j −j −2j 
−2 5 0 0 −2j 3

Identity matrix
• square matrix with Aij = 1 if i = j and Aij = 0 if i 6= j
Hermitian matrix: square with Aij = Āji (complex conjugate of Aij )
• notation: I (usually) or In (if dimension is not clear from context)
 
• columns of In are unit vectors e1, e2, . . . , en; for example, 4 3 − 2j −1 + j
 3 + 2j −1 2j 
 
1 0 0 −1 − j −2j 3
 
I3 =  0 1 0  = e1 e2 e3
0 0 1 note: diagonal elements are real (since Aii = Āii)

Matrices 3-7 Matrices 3-8

Structured matrices Sparse matrices

a matrix is sparse if most (almost all) of its elements are zero
matrices with special patterns or structure arise in many applications
• sparse matrix storage formats and algorithms exploit sparsity
• diagonal matrix: square with Aij = 0 for i 6= j • efficiency depends on number of nonzeros and their positions
    • positions of nonzeros are visualized in a ‘spy plot’
−1 0 0 −1 0 0
 0 2 0 ,  0 0 0 
0 0 −5 0 0 −5

Example
• lower triangular matrix: square with Aij = 0 for i < j
• 2,987,012 rows and columns
 
4 0 0 • 26,621,983 nonzeros
 3 −1 0 
0 5 −2 (Freescale/FullChip matrix from Univ.
of Florida Sparse Matrix Collection)

• upper triangular matrix: square with Aij = 0 for i > j

Matrices 3-9 Matrices 3-10

Outline Scalar-matrix multiplication and addition

Scalar-matrix multiplication:
scalar-matrix product of m × n matrix A with scalar β
 
β A11 β A12 · · · β A1n
• notation and terminology  β A21 β A22 · · · β A2n 
 
βA =  .. .. .. .. 
• matrix operations  . . . . 
β Am1 β Am2 · · · β Amn
• linear and affine functions
A and β can be real or complex
• complexity Addition: sum of two m × n matrices A and B (real or complex)
 
A11 + B11 A12 + B12 · · · A1n + B1n
 A21 + B21 A22 + B22 · · · A2n + B2n 
 
A+B = .. .. .. .. 
 . . . . 
Am1 + Bm1 Am2 + Bm2 · · · Amn + Bmn

Matrices 3-11
 Transpose Conjugate transpose

the transpose of an m × n matrix A is the n × m matrix the conjugate transpose of an m × n matrix A is the n × m matrix
   
A11 A21 · · · Am1 Ā11 Ā21 · · · Ām1
 A12 A22 · · · Am2   Ā12 Ā22 · · · Ām2 
   
AT =  . .. ..  AH = . .. .. 
 .. . .   .. . . 
A1n A2n · · · Amn Ā1n Ā2n · · · Āmn

• (AT )T = A (Āij is complex conjugate of Aij )

• a symmetric matrix satisfies A = AT • AH = AT if A is a real matrix

• A may be complex, but transpose of complex matrices is rarely needed • a Hermitian matrix satisfies A = AH

• transpose of matrix-scalar product and matrix sum • conjugate transpose of matrix-scalar product and matrix sum

(βA)T = βAT , (A + B)T = AT + B T (βA)H = β̄AH , (A + B)H = AH + B H

Matrices 3-12 Matrices 3-13

Matrix-matrix product Exercise: paths in directed graph

product of m × n matrix A and n × p matrix B (A, B real or complex) directed graph with n = 5 vertices matrix representation
 
2 3 0 1 0 0 1
C = AB  1 0 1 0 0 
 
A= 0 0 0 1 1 

is the m × p matrix with elements 5  1 0 0 0 0 
0 0 0 1 0
Cij = Ai1B1j + Ai2B2j + · · · + AinBnj
1 4 Aij = 1 indicates an edge j → i

dimensions must be compatible: #columns in A = #rows in B Question: give a graph interpretation of A2 = AA, A3 = AAA,. . .
   
1 0 1 1 0 1 1 0 1 2
 0 1 0 1 2   2 0 1 2 0 
   
2 
A = 1 0 0 1 0 , 3 
A = 1 1 0 0 1 , ...
 
 0 1 0 0 1   1 0 1 1 0 
1 0 0 1 0 0 1 0 0 1

Matrices 3-14 Matrices 3-15

Properties of matrix-matrix product Notation for vector inner product

• associative: (AB)C = A(BC) so we write ABC • inner product of a, b ∈ Rn (see page 1-18):
• associative with scalar-matrix multiplication: (γA)B = γ(AB) = γAB  T  
b1 a1
• distributes with sum:  b2   a2 
   
b T a = b 1 a 1 + b2 a 2 + · · · + bn a n =  ..   .. 
 .   . 
A(B + C) = AB + AC, (A + B)C = AC + BC
bn an

• transpose and conjugate transpose of product: product of the transpose of column vector b and column vector a

(AB)T = B T AT , (AB)H = B H AH • inner product of a, b ∈ Cn (see page 1-25):


H  
b1 a1
• not commutative: AB 6= BA in general; for example,  b2   a 2 
H    
      b a = b̄1a1 + b̄2a2 + · · · + b̄nan =  .   . 
−1 0 0 1 0 1 −1 0  ..   .. 
6 =
0 1 1 0 1 0 0 1 bn an

there are exceptions, e.g., AI = IA for square A product of conjugate transpose of column vector b and column vector a

Matrices 3-16 Matrices 3-17

 Matrix-matrix product and block matrices Outline

block-matrices can be multiplied as regular matrices

• notation and terminology

Example: product of two 2 × 2 block matrices
• matrix operations
    
A B W Y AW + BX AY + BZ
= • linear and affine functions
C D X Z CW + DX CY + DZ

if the dimensions of the blocks are compatible • complexity

Matrices 3-18

Matrix-vector product Linear function

product of m × n matrix A with n-vector (or n × 1 matrix) x
a function f : Rn → Rm is linear if superposition holds:
 
A11x1 + A12x2 + · · · + A1nxn
 A21x1 + A22x2 + · · · + A2nxn  f (αx + βy) = αf (x) + βf (y)
 
Ax =  .. 
 . 
for all n-vectors x, y and all scalars α, β
Am1x1 + Am2x2 + · · · + Amnxn

• dimensions must be compatible: #columns of A = length of x

• Ax is a linear combination of the columns of A:
Extension: if f is linear, superposition holds for any linear combination:
 
x1
 
 x 2

 f (α1u1 + α2u2 + · · · + αpup) = α1f (u1) + α2f (u2) + · · · + αpf (up)
Ax = a1 a2 · · · an  .  = x1a1 + x2a2 + · · · + xnan
 .. 
xn for all scalars, α1, . . . , αp and all n-vectors u1, . . . , up

each ai is an m-vector (ith column of A)

Matrices 3-19 Matrices 3-20

Matrix-vector product function Input-output (operator) interpretation

for fixed A ∈ Rm×n, define a function f : Rn → Rm as think of a function f : Rn → Rm in terms of its effect on x

f (x) = Ax

• any function of this type is linear: A(αx + βy) = α(Ax) + β(Ay) x y = f (x) = Ax
A
• every linear function can be written as a matrix-vector product function:

f (x) = f (x1e1 + x2e2 + · · · + xnen)

= x1f (e1) + x2f (e2) + · · · + xnf (en)
  • signal processing/control interpretation: n inputs xi, m outputs yi
x1
  . 
= f (e1) · · · f (en)  ..  • f is linear if we can represent its action on x as a a product f (x) = Ax
xn
 
hence, f (x) = Ax with A = f (e1) f (e2) · · · f (en)

Matrices 3-21 Matrices 3-22

 Examples (f : R3 → R3) Operator interpretation of matrix-matrix product
• f reverses the order of the components of x
explains why in general AB 6= BA
a linear function: f (x) = Ax with
 
0 0 1
A= 0 1 0  x B A y = ABx x A B y = BAx
1 0 0

• f sorts the components of x in decreasing order: not linear Example    

• f scales x1 by a given number d1, x2 by d2, x3 by d3 −1 0 0 1
A= , B=
0 1 1 0
a linear function: f (x) = Ax with
 
d1 0 0 • f (x) = ABx swaps the elements of x; then changes sign of first element
A=  0 d2 0 
• f (x) = BAx changes sign of 1st element; then swaps the two elements
0 0 d3

• f replaces each xi by its absolute value |xi|: not linear

Matrices 3-23 Matrices 3-24

Reversal and circular shift Permutation

Reversal matrix Permutation matrix

 
0 0 ··· 0 1 • a square 0-1 matrix with one element 1 per row and one 1 per column
 0 0 ··· 1 0 
 
 .. ..  , • equivalently, an identity matrix with columns reordered
A =  ... ..
. ..
.
. .  Ax = (xn, xn−1, . . . , x2, x1)
 
 0 1 ··· 0 0  • equivalently, an identity matrix with rows reordered
1 0 ··· 0 0
Ax is a permutation of the elements of x
Circular shift matrix
  Example
0 0 ··· 0 1
 1 0 ··· 0 0   
  0 1 0 0
 0 1 ··· 0 0   0 0
A= , Ax = (xn, x1, x2, . . . , xn−1) 0 1 
 .. .. . . .. ..  A=
 1 0
, Ax = (x2, x4, x1, x3)
 . . . . .  0 0 
0 0 ··· 1 0 0 0 1 0

Matrices 3-25 Matrices 3-26

Rotation in a plane Projection on line and reflection

x a
 
cos θ − sin θ y
A=
sin θ cos θ
z

Ax is x rotated counterclockwise over an angle θ

Ax • projection on line through a (see page 2-11):

aT x 1
y= a = Ax with A= aaT
kak2 kak2
x
θ
• reflection with respect to line through a
2
z = x + 2(y − x) = Bx, with B= aaT − I
kak2

Matrices 3-27 Matrices 3-28

 Node-arc incidence matrix Kirchhoff’s current law

• directed graph with m vertices, n arcs (directed edges) n-vector x = (x1, x2, . . . , xn) with xj the current through arc j

• incidence matrix is m × n matrix A with X X

(Ax)i = xj − xj
 arc j enters arc j leaves
 1 if arc j enters node i node i node i
Aij = −1 if arc j leaves node i = total current arriving at node i

0 otherwise

x3
3 2 3
2 3
 
  −x1 − x2 + x4
−1 −1 0 1 0 x1 x4 x5  x − x 
1 4 5  1 0 −1 0 0  Ax =  1 3
 x3 − x4 − x5 

A=
 0

0 1 −1 −1  x2 + x5
0 1 0 0 1 x2
2 1 4
1 4

Matrices 3-29 Matrices 3-30

Kirchhoff’s voltage law Vandermonde matrix

m-vector y = (y1, y2, . . . , ym) with yi the potential at node i • polynomial of degree n − 1 or less with coefficients x1, x2, . . . , xn:

(AT y)j = yk − y l if edge j goes from node l to k p(t) = x1 + x2t + x3t2 + · · · + xntn−1

= negative of voltage across arc j

• values of p(t) at m points t1, . . . , tm:
    
p(t1) 1 t1 · · · tn−1
1 x1
y2 y3  p(t2)   1 t2 · · ·   x2 
2
3
3    tn−1
2  
   ..  =  . . ..   .. 
y2 − y1  .   .. .. .  . 
 y4 − y1  p(tm) 1 tm · · · tn−1
m xn
 
1 4 5 AT y = 
 y3 − y2

 = Ax
 y1 − y3 
y4 − y3 the matrix A is called a Vandermonde matrix
2
1 4
y1 y4 • f (x) = Ax maps coefficients of polynomial to function values

Matrices 3-31 Matrices 3-32

Discrete Fourier transform Affine function

the DFT maps a complex n-vector (x1, x2, . . . , xn) to the complex n-vector a function f : Rn → Rm is affine if it satisfies
    
y1 1 1 1 ··· 1 x1
   1   f (αx + βy) = αf (x) + βf (y)
 y2   ω −1 ω −2 ··· ω −(n−1)  x2 
    
 y3  =  1 ω −2 ω −4 ··· ω −2(n−1)  x3  for all n-vectors x, y and all scalars α, β with α + β = 1
 ..   .. .. .. ..  .. 
 .   . . . .  . 
yn 1 ω −(n−1)
ω −2(n−1)
··· ω −(n−1)(n−1) xn
= Wx Extension: if f is affine, then

√
where ω = e2πj/n (and j = −1) f (α1u1 + α2u2 + · · · + αmum) = α1f (u1) + α2f (u2) + · · · + αmf (um)

• DFT matrix W ∈ Cn×n has k, l element Wkl = ω −(k−1)(l−1) for all n-vectors u1, . . . , um and all scalars α1, . . . , αm with

• a Vandermonde matrix with m = n and

α1 + α2 + · · · + αm = 1
t1 = 1, t2 = ω −1, t3 = ω −2, ..., tn = ω −(n−1)

Matrices 3-33 Matrices 3-34

 Affine functions and matrix-vector product Affine approximation

for fixed A ∈ Rm×n, b ∈ Rm, define a function f : Rn → Rm by first-order Taylor approximation of differentiable f : Rn → Rm around z:

∂fi ∂fi
f (x) = Ax + b fbi(x) = fi(z) + (z)(x1 − z1) + · · · + (z)(xn − zn), i = 1, . . . , m
∂x1 ∂xn
i.e., a matrix-vector product plus a constant
in matrix-vector notation: fb(x) = f (z) + Df (z)(x − z) where
• any function of this type is affine: if α + β = 1 then  ∂f1 ∂f1 ∂f1
  
∂x1 (z) ∂x2 (z) ··· ∂xn (z) ∇f1(z)T
   
 ∂f2 ∂f2 ∂f2   ∇f2(z)T
A(αx + βy) + b = α(Ax + b) + β(Ax + b)  ∂x1 (z) ∂x2 (z) ··· ∂xn (z)   

Df (z) = 
 .. .. ..
=
  .. 
 . . .   . 
  
• every affine function can be written as f (x) = Ax + b with: ∂fm ∂fm ∂fm
∂x1 (z) ∂x2 (z) ··· ∂xn (z)
∇fm(z)T
 
A= f (e1) − f (0) f (e2) − f (0) · · · f (en) − f (0)
• Df (z) is called the derivative matrix or Jacobian matrix of f at z
and b = f (0) • fb is a local affine approximation of f around z

Matrices 3-35 Matrices 3-36

Example Outline

   
f1(x) e2x1+x2 − x1
f (x) = =
f2(x) x21 − x2

• notation and terminology

• derivative matrix
• matrix operations
 
2e2x1+x2 − 1 e2x1+x2
Df (x) =
2x1 −1 • linear and affine functions

• complexity
• first order approximation of f around z = 0:
" #     
fb1(x) 1 1 1 x1
fb(x) = = +
fb2(x) 0 0 −1 x2

Matrices 3-37

Matrix-vector product Matrix-matrix product

matrix-vector multiplication of m × n matrix A and n-vector x: product of m × n matrix A and n × p matrix B:

y = Ax C = AB

requires (2n − 1)m flops requires mp(2n − 1) flops

• m elements in y; each element requires an inner product of length n • mp elements in C; each element requires an inner product of length n
• approximately 2mn for large n
• approximately 2mnp for large n
Special cases: flop count is lower for structured matrices

• A diagonal: n flops
• A lower triangular: n2 flops
• A sparse: #flops ≪ 2mn

Matrices 3-38 Matrices 3-39

 Exercises

1. evaluate y = ABx two ways (A and B are n × n, x is a vector)

• y = (AB)x (first make product C = AB, then multiply C with x)

• y = A(Bx) (first make product y = Bx, then multiply A with y)
both methods give the same answer, but which method is faster?

2. evaluate y = (I + uv T )x where u, v, x are n-vectors

• A = I + uv T followed by y = Ax
in MATLAB: y = (eye(n) + u*v’) * x
• w = (v T x)u followed by y = x + w
in MATLAB: y = x + (v’*x) * u

Matrices 3-40
 Left and right inverse
4. Matrix inverses
AB 6= BA in general, so we have to distinguish two types of inverses

Left inverse: X is a left inverse of A if

• left and right inverse
XA = I
• linear independence

A is left invertible if it has at least one left inverse

• nonsingular matrices

• matrices with linearly independent columns

Right inverse: X is a right inverse of A if

• matrices with linearly independent rows

AX = I

A is right invertible if it has at least one right inverse

4-1 Matrix inverses 4-2

Examples Some immediate properties

  Dimensions
−3 −4  
1 0 1
A= 4 6 , B= a left or right inverse of an m × n matrix must have size n × m
0 1 1
1 1

• A is left invertible; the following matrices are left inverses:

   
1 −11 −10 16 0 −1/2 3 Left and right inverse of (conjugate) transpose
,
9 7 8 −11 0 1/2 −2
• X is a left inverse of A if and only if X T is a right inverse of AT

AT X T = (XA)T = I
• B is right invertible; the following matrices are right inverses:
      • X is a left inverse of A if and only if X H is a right inverse of AH
1 −1 1 0 1 −1
1
−1 1 ,  0 1 ,  0 0 
2 AH X H = (XA)H = I
1 1 0 0 0 1

Matrix inverses 4-3 Matrix inverses 4-4

Inverse Linear equations

if A has a left and a right inverse, then they are equal and unique: set of m linear equations in n variables

XA = I, AY = I =⇒ X = X(AY ) = (XA)Y = Y A11x1 + A12x2 + · · · + A1nxn = b1

A21x1 + A22x2 + · · · + A2nxn = b2
• we call X = Y the inverse of A (notation: A−1) ..
.
• A is invertible if its inverse exists
Am1x1 + Am2x2 + · · · + Amnxn = bm

• in matrix form: Ax = b
Example
    • may have no solution, a unique solution, infinitely many solutions
−1 1 −3 2 4 1
1
A= 1 −1 1 , A−1 = 0 −2 1 
4
2 2 2 −2 −2 0

Matrix inverses 4-5 Matrix inverses 4-6

 Linear equations and matrix inverse Outline

Left invertible matrix: if X is a left inverse of A, then

Ax = b =⇒ x = XAx = Xb
• left and right inverse
there is at most one solution (if there is a solution, it must be equal to Xb)

• linear independence
Right invertible matrix: if X is a right inverse of A, then
• nonsingular matrices
x = Xb =⇒ Ax = AXb = b
• matrices with linearly independent columns
there is at least one solution (namely, x = Xb)
• matrices with linearly independent rows
Invertible matrix: if A is invertible, then

Ax = b ⇐⇒ x = A−1b
there is a unique solution

Matrix inverses 4-7

Linear combination Linear dependence

a linear combination of vectors a1, . . . , an is a sum of scalar-vector products a collection of vectors a1, a2, . . . , an is linearly dependent if

x1 a1 + x2 a 2 + · · · + xn an x1 a1 + x2 a2 + · · · + xn an = 0

• the scalars xi are the coefficients of the linear combination for some scalars x1, . . . , xn, not all zero

• can be written as a matrix-vector product

• vector 0 can be written as a nontrivial linear combination of a1, . . . , an
 
x1
• equivalently, at least one vector ai is a linear combination of the rest:
 
 x2 

x1 a1 + x2 a2 + · · · + xn an = a1 a2 · · · an  .. 
 .  x1 xi−1 xi+1 xn
ai = − a1 − · · · − ai−1 − ai+1 − · · · − an
xn xi xi xi xi

if xi 6= 0
• the trivial linear combination has coefficients x1 = · · · = xn = 0

(same definition holds for real and complex vectors/scalars)

Matrix inverses 4-8 Matrix inverses 4-9

Example Linear independence

the vectors
vectors a1, . . . , an are linearly independent if they are not linearly dependent
     
0.2 −0.1 0
a1 =  −7  , a2 =  2  , a3 =  −1  • the zero vector cannot be written as a nontrivial linear combination:
8.6 −1 2.2
x1 a1 + x2 a2 + · · · + xn an = 0 =⇒ x1 = x2 = · · · = xn = 0
are linearly dependent

• none of the vectors ai is a linear combination of the other vectors

• 0 can be expressed as a nontrivial linear combination of a1, a2, a3:

0 = a1 + 2a2 − 3a3
Matrix with linearly independent columns
• a1 can be expressed as a linear combination of a2, a3:  
A= a1 a2 · · · an
a1 = −2a1 + 3a3
has linearly independent columns if Ax = 0 only for x = 0
(and similarly a2 and a3)

Matrix inverses 4-10 Matrix inverses 4-11

 Example Dimension inequality

the vectors if n vectors a1, a2, . . . , an of length m are linearly independent, then
     
1 −1 0 n≤m
a1 =  −2  , a2 =  0  , a3 =  1 
0 1 1
(proof is in textbook)
are linearly independent:
  • if an m × n matrix has linearly independent columns then m ≥ n
x1 − x2
x1 a1 + x2 a2 + x3 a3 =  −2x 1 + x 3
=0
• if an m × n matrix has linearly independent rows then m ≤ n
x2 + x3

only if x1 = x2 = x3 = 0

Matrix inverses 4-12 Matrix inverses 4-13

Outline Nonsingular matrix

for a square matrix A the following four properties are equivalent

1. A is left invertible
• left and right inverse 2. the columns of A are linearly independent

• linear independence 3. A is right invertible

4. the rows of A are linearly independent

• nonsingular matrices
a square matrix with these properties is called nonsingular
• matrices with linearly independent columns

• matrices with linearly independent rows Nonsingular = invertible

• if properties 1 and 3 hold, then A is invertible (page 4-5)

• if A is invertible, properties 1 and 3 hold (by definition of invertibility)

Matrix inverses 4-14

Proof Part a: suppose A is left invertible

(a) • if B is a left inverse of A (satisfies BA = I ), then

left invertible linearly independent columns
Ax = 0 =⇒ BAx = 0
(b’) (b)
=⇒ x = 0
(a’)
linearly independent rows right invertible
• this means that the columns of A are linearly independent: if
 
A= a1 a2 · · · an
• we show that (a) holds in general
then
• we show that (b) holds for square matrices x1 a1 + x2 a2 + · · · + xn an = 0

holds only for the trivial linear combination x1 = x2 = · · · = xn = 0

• (a’) and (b’) follow from (a) and (b) applied to AT

Matrix inverses 4-15 Matrix inverses 4-16

Part b: suppose A is square with linearly independent columns a1, . . . , an Examples
• for every n-vector b the vectors a1, . . . , an, b are linearly dependent  
  1 −1 1 −1
(from dimension inequality on page 4-13) 1 −1 1  −1 
1 −1 1
A =  −1 1 1 , B=
 1

1 −1 −1 
• hence for every b there exists a nontrivial linear combination 1 1 −1
−1 −1 1 1
x1a1 + x2a2 + · · · + xnan + xn+1b = 0

we must have xn+1 6= 0 because a1, . . . , an are linearly independent • A is nonsingular because its columns are linearly independent:

• hence every b can be written as a linear combination of a1, . . . , an

x1 − x2 + x3 = 0, −x1 + x2 + x3 = 0, x1 + x 2 − x 3 = 0
• in particular, there exist n-vectors c1, . . . , cn such that
is only possible if x1 = x2 = x3 = 0
Ac1 = e1, Ac2 = e2, ..., Acn = en,
• B is singular because its columns are linearly dependent:
 
• the matrix C = c1 c2 · · · cn is a right inverse of A:
Bx = 0 for x = (1, 1, 1, 1)
   
A c1 c2 · · · cn = e1 e2 · · · en =I

Matrix inverses 4-17 Matrix inverses 4-18

Example: Vandermonde matrix Inverse of transpose and product

 
1 t1 t21 · · · tn−1
1 Transpose and conjugate transpose
 1 t2 t22 · · · tn−1 
 2  if A is nonsingular, then AT and AH are nonsingular and
A= . . .. . . ..  with ti 6= tj for i 6= j
 .. .. . . . 
1 tn t2n · · · tnn−1 (AT )−1 = (A−1)T , (AH )−1 = (A−1)H ,

we show that A is nonsingular by showing that Ax = 0 only if x = 0 we write these as A−T and A−H

• Ax = 0 means p(t1) = p(t2) = · · · = p(tn) = 0 where

p(t) = x1 + x2t + x3t2 + · · · + xntn−1 Product

if A and B are nonsingular and of equal size, then AB is nonsingular with

p(t) is a polynomial of degree n − 1 or less

• if x 6= 0, then p(t) can not have more than n − 1 distinct real roots (AB)−1 = B −1A−1

• therefore p(t1) = · · · = p(tn) = 0 is only possible if x = 0

Matrix inverses 4-19 Matrix inverses 4-20

Outline Gram matrix

the Gram matrix associated with a matrix

 
A= a1 a2 · · · an

• left and right inverse is the matrix of column inner products

• linear independence • for real matrices:

 
aT1 a1 aT1 a2 · · · aT1 an
• nonsingular matrices  aT2 a1 aT2 a2 · · · aT2 an 
 
AT A =  . .. .. 
 .. . . 
• matrices with linearly independent columns
aTn a1 aTn a2 · · · aTn an
• matrices with linearly independent rows
• for complex matrices
 
aH H
1 a1 a1 a2 · · · aH
1 an
 aH H
aH 
H  2 a1 a2 a2 · · · 2 an 
A A= . .. .. 
 .. . . 
H H
an a1 an a2 · · · aH
n an
Matrix inverses 4-21
 Nonsingular Gram matrix Pseudo-inverse
m×n
the Gram matrix is nonsingular if only if A has linearly independent columns • suppose A ∈ R has linearly independent columns

• this implies that A is tall or square (m ≥ n); see page 4-13

• suppose A ∈ Rm×n has linearly independent columns:

AT Ax = 0 =⇒ xT AT Ax = (Ax)T (Ax) = kAxk2 = 0 the pseudo-inverse of A is defined as

=⇒ Ax = 0
=⇒ x = 0 A† = (AT A)−1AT

therefore AT A is nonsingular • this matrix exists, because the Gram matrix AT A is nonsingular
• A† is a left inverse of A:
• suppose the columns of A ∈ Rm×n are linearly dependent
A†A = (AT A)−1(AT A) = I
∃x 6= 0, Ax = 0 =⇒ ∃x 6= 0, AT Ax = 0

therefore AT A is singular
(for complex A with linearly independent columns, A† = (AH A)−1AH )
(for A ∈ Cm×n, replace AT with AH and xT with xH )
Matrix inverses 4-22 Matrix inverses 4-23

Summary Outline

the following three properties are equivalent

1. A is left invertible

2. the columns of A are linearly independent • left and right inverse

3. AT A is nonsingular • linear independence

• nonsingular matrices
• 1 ⇒ 2 was already proved on page 4-16
• matrices with linearly independent columns
• 2 ⇒ 1: we have seen that the pseudo-inverse is a left inverse
• 2 ⇔ 3: proved on page 4-22 • matrices with linearly independent rows

• a matrix with these properties must be tall or square

Matrix inverses 4-24

Pseudo-inverse Summary

• suppose A ∈ Rm×n has linearly independent rows

the following three properties are equivalent
• this implies that A is wide or square (m ≤ n); see page 4-13
1. A is right invertible

2. the rows of A are linearly independent

the pseudo-inverse of A is defined as
3. AAT is nonsingular
A† = AT (AAT )−1

• AT has linearly independent columns

• hence its Gram matrix AAT is nonsingular, so A† exists • 1 ⇒ 2 and 2 ⇔ 3: by transposing result on page 4-24
• A† is a right inverse of A: • 2 ⇒ 1: we have seen that the pseudo-inverse is a right inverse
• a matrix with these properties must be wide or square
AA† = (AAT )(AAT )−1 = I

(for complex A with linearly independent rows, A† = AH (AAH )−1)

Matrix inverses 4-25 Matrix inverses 4-26