Vector spaces and linear mappings

Sudhir A. Shah

Delhi School of Economics

September 23, 2019

Outline
Preliminaries
Vector space
Span, (In)dependence, Dimension
Vector subspaces
Linear transformations
Projectors
Inverse mappings
Linear equations
Inner product, orthogonality, adjoints
Orthogonal projectors
Roots of a matrix and invariant spaces
Quadratic forms and diagonalisation
Why we do what we do?

‘Does it contain any abstract reasoning concerning quantity or

number?
No.
Does it contain any experimental reasoning concerning matter
of fact and existence?
No.
Commit it then to the flames: for it can contain nothing but
sophistry and illusion.’

David Hume, An Enquiry Concerning Human Understanding

(1748)
Vector space
Definition
A collection V := {V , F ; +, ., ⊕, } is called a vector space over
the field F if
1. {V ; ⊕} is a commutative group (of ‘vectors’) with identity
element θ, Definition
2. F := {F ; +, .} is a field (of ‘scalars’, usually F = < or
F = C) with additive identity element 0 and multiplicative
identity element 1, and Definition
3. for all λ, µ ∈ F and x, y ∈ V , we have λ  x ∈ V and
3.1 (λ + µ)  x = (λ  x) ⊕ (µ  x),
3.2 λ  (x ⊕ y ) = (λ  x) ⊕ (λ  y ),
3.3 (λ.µ)  x = λ  (µ  x), and
3.4 1  x = x. Example

Notation: + is scalar addition; . is scalar multiplication; ⊕ is

vector addition;  is the multiplication of a vector by a scalar.
Conventions

1. Instead of referring to the vector space V, we shall refer

simply to V as the ‘vector space’, taking the other data
{F ; +, ., ⊕, } as given and understood.
2. We use x + y to denote x ⊕ y and use αx to denote α  x.
3. We denote the additive identity element of V by 0 instead
of θ.
4. Whenever symbols do double-duty, the intended meaning
should be clear from the context.
5. A vector space with F = < is called a real vector space. A
vector space with F = C is called a complex vector space.
6. Scalars of these two types are adequate for most
applications.
Span of a set of vectors

Definition
Given a vector space
n
PnV , n ∈ N , {x1 , . . . , xn } ⊂ V , and
(α1 , . . . , αn ) ∈ F , i=1 αi xi is called a linear combination of
x1 , . . . , xn .

Definition
Consider a vector space V and X ⊂ V .
1. For n ∈ N , let Xn be the collection of subsets of X with n
elements.
2. The spanPof Y := {y1 , . . . , yn } ∈ Xn is
n n .
[Y ] := i=1 αi yi | (α1 , . . . , αn ) ∈ F
3. The span of X is [X ] := ∪n∈N ∪Y ∈Xn [Y ].
4. [∅] := {0}.
Linearly (in)dependent sets of vectors

Definition
Consider a vector space V and X ⊂ V .
1. y ∈ V is said to be linearly dependent on X if y ∈ [X ].
2. X is said to be a linearly dependent set if there exists
y ∈ X such that y ∈ [X \ {y }].
3. A set X ⊂ V is said to be a linearly independent set if it is
not linearly dependent.
Properties of (in)dependent sets

Theorem
Consider a vector space V and sets X ⊂ V and Y ⊂ V .

Pnis independent iff. for every finite set {x1 , . . . , xn } ⊂ X ,

1. X
i=1 αi xi = 0 implies α1 = . . . = αn = 0.
Proof

2. If X is independent and Y ⊂ X , then Y is independent.

Proof

3. If Y is dependent and Y ⊂ X , then X is dependent. Proof

4. If 0 ∈ X , then X is dependent. Proof

5. If x ∈ V \ {0}, then {x} is independent. Proof

6. If X is independent and y ∈ V , then X ∪ {y } is

independent iff. y 6∈ [X \ {y }]. Proof
7. {x1 , . . . , xn } ⊂ V \ {0} is dependent iff. xm ∈ [x1 , . . . , xm−1 ]
for some m ≤ n. Proof
Basis of a vector space V
Definition
1. X ⊂ V is said to span V if V ⊂ [X ].
2. If V has a finite subset that spans V , then V is said to be
finite dimensional. Otherwise, V is said to be infinite
dimensional.
3. A basis for a vector space V is an independent set X ⊂ V
that spans V . Example

Theorem
1. Every vector space has a basis.
Proof uses Zorn’s lemma. For infinite dimensional spaces,
this basis is called a Hamel basis in order to distinguish it
from other notions of bases such as Schauder bases.
2. A finite dimensional vector space has a finite basis. Proof
Working assumptions

1. Henceforth, we deal with finite dimensional real vector

spaces only. This qualification is to be read into all
subsequent statements.
2. Many of the results can also be proved more generally for
complex or infinite dimensional vector spaces, often with
almost no extra effort, but we shall eschew the generality in
the interest of simplicity of exposition.
3. Real vector spaces are adequate for many, but not for all,
purposes. For instance, the area of dynamics and stability
uses differential equations, whose study is not possible
without complex vector spaces.
Dimension of a vector space V

Theorem
Every basis of V has the same number of vectors. Proof

Definition
The number of vectors in a basis of V is called the dimension of
V , denoted by dim V .

Theorem
Let dim V = n.
1. If X ⊂ V is independent, then X can be extended to a
basis for V . Proof
2. {x1 , . . . , xn } ⊂ V spans V iff. {x1 , . . . , xn } is a basis for V .
Proof

3. {x1 , . . . , xn } ⊂ V is independent iff. {x1 , . . . , xn } is a basis

for V . Proof
Coordinate representation of vectors

Theorem
If {x1 , . . . , xn } ⊂ V is a basis for the vector space V , then for
every x ∈ V there exists a unique Pn-tuple of scalars
(α1 , . . . , αn ) ∈ F n such that x = ni=1 αi xi .
The scalars {α1 , . . . , αn } are called the coordinates of x with
respect to the basis {x1 , . . . , xn }.
Subspaces of a vector space V

Definition
Let {V , F ; +, ., ⊕, } be a vector space and let S ⊂ V . If
{S, F ; +, ., ⊕, } is a vector space, then {S, F ; +, ., ⊕, } is
called a subspace of {V , F ; +, ., ⊕, }.

Theorem
If S ⊂ V , then S is a subspace of V iff.
1. x, y ∈ S implies x + y ∈ S, and
2. x ∈ S and α ∈ F implies αx ∈ S. Proof left as exercise.

Theorem
If S ⊂ V , then {[S], F ; +, ., ⊕, } is a subspace of V .
Proof left as exercise.
Sums of subspaces

Definition
Let S and T be subspaces of a vector space V . Example

1. The sum of S and T is S + T = {x + y | x ∈ S ∧ y ∈ T }.

The intersection of S and T is S ∩ T .
2. If V = S + T and S ∩ T = [0], then V is called the direct
sum of S and T , denoted by V = S ⊕ T .

Theorem
Let S and T be subspaces of a vector space V .
1. V = S ⊕ T iff. every v ∈ V has a unique representation
v = x + y for some x ∈ S and some y ∈ T . Proof
2. dim(S + T ) + dim(S ∩ T ) = dim S + dim T . Proof
Linear transformations

Definition
Given vector spaces V and W , a function A : V → W is called
a linear transformation if A(αx + βy ) = αA(x) + βA(y ) for all
x, y ∈ V and α, β ∈ F . The space of all linear transformations
from V to W is denoted by L(V , W ).
Terminology. Linear transformations may be called linear
mappings, or homomorphisms, or in certain contexts, they may
also be called linear operators.
Definition
The range space of A ∈ L(V , W ) is R(A) = {Ax ∈ W | x ∈ V }.
The null space of A is N (A) = {x ∈ V | Ax = 0}.
It is easy to check that R(A) and N (A) are subspaces of W
and V respectively.
Canonical n-dimensional vector space

Definition
A bijection A ∈ L(V , W ) is called an isomorphism.

Definition
Given the field of scalars F , we can define {F n , F ; +, ., ⊕, }.
F n as the vector space of n-tuples of scalars.

Theorem
If V is a vector space with dim V = n, then it is isomorphic to
F n . Proof
Rank and nullity

Definition
Let V and W be vector spaces and A ∈ L(V , W ). The
dimension of R(A) is called the rank of A, denoted by ρ(A). The
dimension of N (A) is called the nullity of A, denoted by ν(A).

Theorem
Let V and W be vector spaces, and A ∈ L(V , W ).
1. ρ(A) + ν(A) = dim V . Proof

2. If ν(A) = 0, then A is injective. Proof

3. If ρ(A) = dim W , then A is surjective. Proof

4. If ν(A) = 0 and ρ(A) = dim W , then A is an isomorphism.

Proof
Rank, nullity and composition mappings

Theorem
Let U, V and W be vector spaces, A ∈ L(V , W ) and
B ∈ L(U, V ).
1. AB ∈ L(U, W ). Proof

2. R(AB) ⊂ R(A) and N (B) ⊂ N (AB). Proof

3. ρ(AB) ≤ min{ρ(A), ρ(B)} and ν(B) ≤ ν(AB). Proof

Matrix representation of a linear transformation

Theorem
Consider a vector space V with basis {v1 , . . . , vn }, a vector
space W with basis {w1 , . . . , wm }, and a linear transformation
F ∈ L(V , W ). Then, there exists a unique m × n matrix of
scalars A such that, if v ∈ V has coordinates b = (b1 , . . . , bn )
with respect to basis {v1 , . . . , vn }, then F (v ) has coordinates
(c1 , . . . , cm ) = Ab with respect to basis {w1 , . . . , wm }. Proof
Projection mappings
Definition
Let U and W be subspaces of V such that V = U ⊕ W .
P : V → V is said to project V on U along W if P(u + w) = u
for all u ∈ U and w ∈ W ; then, P is called a projector. Example

Theorem
Let V be a vector space and P : V → V .
1. If P is a projector, then P ∈ L(V , V ), V = R(P) ⊕ N (P),
and P projects V on R(P) along N (P). Proof
2. If P is a projector, then P 2 = P and R(I − P) = N (P). Proof

3. If P ∈ L(V , V ) is idempotent, then it is a projector. Proof

4. If P ∈ L(V , V ), U is a vector space, and X ∈ L(U, V ) with

R(P) = R(X ), then P is a projector iff. PX = X . Proof
5. P is a projector iff. I − P is a projector, where I is the
identity mapping. Proof
Inverses: Definitions

Definition
Let A ∈ L(V , W ).
1. AL ∈ L(W , V ) is a left inverse of A if AL A = I ∈ L(V , V ).
Example

2. AR ∈ L(W , V ) is a right inverse of A if AAR = I ∈ L(W , W ).

3. A−1 ∈ L(W , V ) is a regular inverse of A if
A−1 A = I ∈ L(V , V ) and AA−1 = I ∈ L(W , W ). If A has a
regular inverse, then A is called non-singular.
Inverses

Theorem
Let A ∈ L(V , W ).
1. A left inverse of A exists iff. ν(A) = 0. Proof

2. A right inverse of A exists iff. ρ(A) = dim W . Proof

3. A regular inverse of A exists iff. ρ(A) = dim W and

ν(A) = 0. Consequently, if a regular inverse of A exists,
then dim V = dim W . Proof
4. If a regular inverse of A exists, then it is unique. Proof

5. Let B ∈ L(U, V ). If A and B are non-singular, then AB is

non-singular and (AB)−1 = B −1 A−1 . Proof
Generalised inverse: definition and existence

Definition
Let A ∈ L(V , W ). A− ∈ L(W , V ) is said to be a
generalised-inverse (or g-inverse) of A if AA− A = A.

Theorem
Every A ∈ L(V , W ) has a g-inverse.
Proof omitted. Many proofs are available in the literature.
Properties of g-inverses
Theorem
Let A ∈ L(V , W ) and B ∈ L(W , U).
1. ρ(A− ) ≥ ρ(A). Proof

2. ρ(AA− ) = ρ(A− A) = ρ(A). Proof

3. ν(AA− ) = ν(A− A) = ν(A). Proof

4. AA− projects W on R(A) and A− A projects V on R(A− A).

Proof

5. If ρ(BA) = ρ(B), then A(BA)− = B − . Proof

6. If ρ(BA) = ρ(A), then (BA)− B = A− . Proof

7. If ρ(BA) = ρ(A), then A(BA)− B projects W on R(A). Proof

8. If ν(A) = 0, then A− = AL . Proof

9. If ρ(A) = dim W , then A− = AR . Proof

10. If ν(A) = 0 and ρ(A) = dim W , then A− = A−1 . Proof

Linear equations

Definition
1. Given A ∈ L(V , W ) and y ∈ W , Ax = y is called a
non-homogeneous equation.
2. The equation Ax = 0 is called the homogeneous part of
Ax = y .
3. A solution of the equation Ax = y is an x0 ∈ V such that
Ax0 = y .
4. If the equation Ax = y has a solution, then it is called
consistent.

Theorem
If A ∈ L(V , W ), then the solution set of the equation Ax = 0 is
N (A) = R(I − A− A) = (I − A− A)V . Proof
Linear equations

Theorem
Let A ∈ L(V , W ).
1. Given y ∈ W , the equation Ax = y is consistent iff.
AA− y = y . Proof
2. Suppose the equation Ax = y is consistent for every
y ∈ W . Then, it has a unique solution x = AL y for every
y ∈ W iff. ν(A) = 0. Proof
3. If ν(A) = 0 and ρ(A) = dim W , then the equation Ax = y is
consistent and has a unique solution x = A−1 y for every
y ∈ W . Proof
4. The solution set of the consistent equation Ax = y is
A− y + (I − A− A)V . Proof
Inner product
Definition
Given the vector space V , a function h., .i : V × V → < is called
an inner product on V if for all x, y , z ∈ V and λ, µ ∈ F ,
1. hx, y i = hy , xi,
2. hλx + µy , zi = λhx, zi + µhy , zi,
3. hx, xi ≥ 0, and
4. hx, xi = 0 iff. x = 0. Example

(V , h., .i) is called an inner product space. Let kxk := hx, xi1/2 .

Theorem
(Cauchy-Schwartz) Consider an inner product space (V , h., .i).
If x, y ∈ V , then
1. hx, y i ≤ hx, xi1/2 hy , y i1/2 Proof

2. Equality holds in (1) iff. y = 0 or x = λy for some λ ∈ <.

Proof
Orthogonality

Definition
Consider an inner product space (V , h., .i).
1. x, y ∈ V are said to be orthogonal if hx, y i = 0.
2. x ∈ V is said to be orthogonal to a subspace W of V if
hx, y i = 0 for every y ∈ W .
3. Subspaces U and W of V are said to be orthogonal
subspaces if every x ∈ U is orthogonal to W .
4. Given a subspace W of V ,

W ⊥ = [∩x∈W {y ∈ V | hx, y i = 0}]

is called the orthogonal complement of W . Example

Orthonormal sets

Definition
Consider an inner product space (V , h., .i).
1. A set {c1 , . . . , c(r } ⊂ V is called an orthonormal set if
1, i = j
hci , cj i = δij =
0, i 6= j
2. If an orthonormal set {c1 , . . . , cr } is a basis for V , then it is
called an orthonormal basis of V .
Properties of orthonormal sets
Consider an inner product space (V , h., .i).
Theorem
1. An orthonormal subset of V is linearly independent. Proof

2. If {c1P
, . . . , cr } is an orthonormal basis for V , then
y = ri=1 hy , ci ici for every y ∈ V . Proof

Lemma
If {x1 , . . . , xk } ⊂ V is independent and {y1 , . . . , yk −1 } ⊂ V is an
orthonormal set with [{y1 , . . . , yk −1 }] = [{x1 , . . . , xk −1 }], then
there exists yk ∈ V such that {y1 , . . . , yk } is an orthonormal set
with [{y1 , . . . , yk }] = [{x1 , . . . , xk }]. Proof

Theorem
(Gram-Schmidt) If (V , h., .i) is finite dimensional, then it has an
orthonormal basis. Proof
Orthogonal decomposition

Theorem
Consider an inner product space (V , h., .i). If W is a subspace
of V , then W ⊕ W ⊥ = V . Proof
Adjoint transformations I

Definition
Consider inner product spaces (V , h., .iV ) and (W , h., .iW ).
1. If A ∈ L(V , W ), then A∗ ∈ L(W , V ) is called the adjoint of A
if hx, A∗ y iV = hAx, y iW for every x ∈ V and y ∈ W . Example
2. If A ∈ L(V , V ) and A = A∗ , then A is called self-adjoint.
Adjoint transformations II

Theorem
Consider inner product spaces (V , h., .iV ) and (W , h., .iW ), and
A ∈ L(V , W ).
1. A∗ exists and is unique. Proof

2. (A∗ )∗ = A. Proof

3. (A∗ A)∗ = A∗ A. Proof

4. N (A) = R(A∗ )⊥ and R(A) = N (A∗ )⊥ . Proof

5. ρ(A) = ρ(A∗ ). Proof

6. R(A∗ A) = R(A∗ ) and N (A∗ A) = N (A). Proof

Adjoint transformations III
Theorem
Consider inner product spaces (U, h., .iU ), (V , h., .iV ), and
(W , h., .iW ), and mappings A, B ∈ L(V , W ) and C ∈ L(U, V ).
1. (AC)∗ = C ∗ A∗ Proof

2. (A + B)∗ = A∗ + B ∗ Proof

3. (λA)∗ = λA∗ Proof

4. If W = V , then I ∗ = I and 0∗ = 0. Proof

5. If A is nonsingular, then (AA−1 )∗ = (A−1 )∗ A∗ = I and

(A−1 )∗ = (A∗ )−1 . Proof
6. If AL exists, then (AL A)∗ = A∗ (AL )∗ = I and (AL )∗ = (A∗ )R .
Proof

7. If AR exists, then (AAR )∗ = (AR )∗ A∗ = I and (AR )∗ = (A∗ )L .

Proof

8. (AA− A)∗ = A∗ (A− )∗ A∗ = A∗ and (A− )∗ = (A∗ )− . Proof

Orthogonal projection and its characterisation

Definition
Given a vector space V , a projector P ∈ L(V , V ) is called an
orthogonal projector of V if R(P) = N (P)⊥ ; in this case, P is
said to orthogonally project V on R(P) along N (P).

Theorem
1. P ∈ L(V , V ) is an orthogonal projector of V iff. it is
self-adjoint and idempotent, i.e., P = P ∗ = P 2 . Proof
2. P is an orthogonal projector of V iff. P ∗ (I − P) = 0. Proof
Distance minimisation

Theorem
Let W be a subspace of an inner product space (V , h., .i) and
let P : V → W . Then, kPx − xk ≤ kw − xk for all x ∈ V and
w ∈ W iff. P is an orthogonal projector of V on W . Proof

Theorem
Consider vector spaces V and U, and transformations
P ∈ L(V , V ) and X ∈ L(U, V ). If P is an orthogonal projector
and R(P) = R(X ), then P = X (X ∗ X )− X ∗ . Proof
Application to linear estimation

1. Suppose we postulate a linear model x 7→ hβ, xi, where

x ∈ <n is the independent variable and hβ, xi is the
predicted dependent variable in <, but β is unknown.
2. Consider the data {(xi , yi ) ∈ <n × < | i = 1, . . . , m}. Let
y = (y1 , . . . , ym ). Let X be the m × n matrix with xi as the
i-th row. Given β ∈ <n , y − X β ∈ <m is the vector of errors.
3. Choose β ∈ <n to minimise ky − X βk. As X β ∈ R(X ) for
every β, the problem is: minimise ky − zk over z ∈ R(X ).
4. For y ∈ <m , let Py ∈ R(X ) solve the problem. Using Ref ,
y 7→ Py is an orthogonal projector of <m on R(X ).
5. Using Ref , P = X (X ∗ X )− X ∗ . So,
ky − X (X ∗ X )− X ∗ y k ≤ ky − zk for ever z ∈ R(X ).
Moreover, X β̂ = X (X ∗ X )− X ∗ y .
Roots and invariant subspaces of real matrices

Throughout the sequel, we use the unitary inner product, i.e.,

hx, y i = x T Iy .
Definition
Let A be an n × n real matrix.
1. A subspace V ⊂ <n is called an invariant subspace with
respect to A if AV ⊂ V .
2. If x ∈ <n \ {0} and λ ∈ < are such that Ax = λx, then x is
called a characteristic vector of A corresponding to the
characteristic root λ.
3. Given a root λ, {x ∈ <n | Ax = λx} is called the
characteristic subspace corresponding to λ. The
dimension of the characteristic subspace is called the
geometric multiplicity of λ.
Roots and invariant subspaces

Theorem
If A is a real symmetric n × n matrix and V ⊂ <n is an invariant
subspace with respect to A, then:
1. There exists λ ≥ 0 and x ∈ V \ {0} such that A2 x = λx.
Proof

2. There exists µ ∈ < and y ∈ V \ {0} such that Ay = µy .

Proof
Roots and invariant subspaces II

Theorem
Suppose A is a real symmetric n × n matrix. Then:
1. The characteristic subspaces of <n corresponding to
distinct roots of A are orthogonal. Proof
2. If λ1 , . . . , λr are the distinct roots of A with corresponding
characteristic subspaces L1 , . . . , Lr , then
<n = L1 ⊕ . . . ⊕ Lr is an orthogonal decomposition. Proof
3. <n has an orthonormal basis consisting of characteristic
vectors of A. Proof
(Semi)definite matrices and roots

Theorem
Suppose A is a real symmetric n × n matrix. Then:
1. hAx, xi ≥ 0 for every x ∈ <n iff. all the roots of A are
non-negative.
2. hAx, xi > 0 for every x ∈ <n \ {0} iff. all the roots of A are
positive.
3. hAx, xi ≤ 0 for every x ∈ <n iff. all the roots of A are
non-positive.
4. hAx, xi < 0 for every x ∈ <n \ {0} iff. all the roots of A are
negative. Proof
Diagonalisation

Theorem
If A is a real symmetric n × n matrix, then there exists an n × n
non-singular matrix C such that A = CΛC T and Λ = C T AC,
where Λ is a diagonal matrix with the characteristic roots of A
on the diagonal. Proof

Corollary
det A = det Λ.
Diagonalisation of semidefinite matrices

Corollary
If A is positive semidefinite, then there exists a matrix E such
that  
T Ir 0
E AE = J =
0 0
where r is the number of non-zero roots of A. Proof

Corollary
If A is positive definite, then there exists a non-singular matrix E
such that E T AE = I. Furthermore, A = (E T )−1 E −1 and
A−1 = EE T . Proof
Semidefinite matrices

Theorem
A positive semidefinite matrix A is positive definite if and only if
A is nonsingular. Proof

Theorem
A is positive definite if and only if A−1 is symmetric and positive
definite. Proof
Definitions

1. Given a set F and an operation ∗ : F × F → F , {F ; ∗} is

said to be a commutative (or Abelian) group if Back
1.1 ∗ is commutative and associative, i.e., λ ∗ µ = µ ∗ λ and
(λ ∗ µ) ∗ ν = λ ∗ (µ ∗ ν) for all λ, µ, ν ∈ F ,
1.2 there exists ω ∈ F (called the identity element with respect
to ∗) such that λ ∗ ω = λ for every λ ∈ F , and
1.3 for every λ ∈ F , there exists λ∗ ∈ F (called the inverse of λ
with respect to ∗) such that λ ∗ λ∗ = ω.
2. Given a set F and operations + : F × F → F (addition) and
. : F × F → F (multiplication), {F ; +, .} is said to be a field
if Back
2.1 {F ; +} is a commutative group with identity element ω ∈ F ,
2.2 {F \ {ω}; .} is a commutative group, and
2.3 the operation . is distributive over +, i.e.,
λ.(µ ∗ ν) = (λ.µ) ∗ (λ.ν).
Vector space example

1. F = < Back

2. + and . have the ‘usual’ meaning of adding and multiplying

real numbers respectively (If you know about the
construction of real numbers, these notions are quite
non-trivial!)
3. V = <2
4. (x1 , x2 ) ⊕ (y1 , y2 ) := (x1 + y1 , x2 + y2 ) for
(x1 , x2 ), (y1 , y2 ) ∈ <2
5. θ := (0, 0)
6. λ  (x1 , x2 ) := (λx1 , λx2 ) for (x1 , x2 ) ∈ <2 and λ ∈ <
7. We can easily generalise these definitions to very general
function spaces.
Example of independence, spanning, and basis

1. Let V = <2 and X = {(1, 0), (0, 1)}. Back

2. X is independent as (0, 0) = α(1, 0) + β(0, 1) = (α, β)

implies α = β = 0.
3. [X ] = <2 as (x1 , x2 ) = x1 (1, 0) + x2 (0, 1) for every
(x1 , x2 ) ∈ <2 .
4. So, X is a basis for <2 .
5. Later, we shall see that this means <2 is 2-dimensional.
Example of subspaces and sums

1. S = [{(1, 0)}] and T = [{(0, 1)}] are subspaces of <2 . Back

2. S + T = <2 <2
as every (x1 , x2 ) ∈ can be written as
(x1 , x2 ) = (x1 , 0) + (0, x2 ) ∈ S + T .
3. S ∩ T = {(0, 0)} = [(0, 0)].
4. So, <2 = S ⊕ T .
Example of projector
1. S = [{(1, 1)}] and T = [{(1, −1)}] are subspaces of <2 .
Back

2. S + T = <2 as every (x1 , x2 ) ∈ <2 can be written as

x1 −x2
(x1 , x2 ) = x1 +x
2 (1, 1) +
2
2 (1, −1) ∈ S + T .
3. S ∩ T = {(0, 0)} = [(0, 0)].
4. So, <2 = S ⊕ T .
5. Define P : <2 → <2 by
x1 −x2
P(x1 , x2 ) = P( x1 +x
2 (1, 1) +
2
2 (1, −1)) = x1 +x2
2 (1, 1).
6. For (x1 , x2 ) ∈ S, i.e., (x1 , x2 ) = α(1, 1) for some α ∈ <,
P(x1 , x2 ) = αP(1, 1) = α(1, 1) = I(x1 , x2 ). So, P’s
restriction to S is the identity mapping on S.
7. For (x1 , x2 ) ∈ T , i.e., (x1 , x2 ) = α(1, −1) for some α ∈ <,
P(x1 , x2 ) = αP(1, −1) = α(0, 0) = (0, 0). So, P’s restriction
to T is the zero mapping on T .
8. So, P ∈ L(<2 , <2 ) projects <2 on S along T .
Examples of inverses

1. Define A ∈ L(<, <2 ) by Ax = (x, −x); define B : <2 → < by

B(x1 , x2 ) = x1 . Back
1.1 Then, BA(x) = B(x, −x) = x for every x ∈ <.
1.2 So, we may set AL = B and B R = A.
2. Define A ∈ L(<2 , <2 ) by A(x1 , x2 ) = (x1 , −x2 ); define
B : <2 → <2 by B(x1 , x2 ) = (x1 , −x2 ).
3. Then, BA(x1 , x2 ) = B(x1 , −x2 ) = (x1 , x2 ) = A(x1 , −x2 ) =
AB(x1 , x2 ).
4. So, A−1 = B.
Inner product examples

1. Let A be an n × n positive definite matrix. Define

h., .i : <n × <n → < by hx, y i := x T Ay . Back
2. Clearly, hx, xi = x T Ax ≥ 0 and hx, xi = x T Ax = 0 iff.
x = 0. Moreover, hλx + µy , zi = (λx + µy )T Az =
λx T Az + µy T Az = λhx, zi + µhy , zi.
3. So, h., .i is an inner product on <n .
4. Setting A = I in the above example yields the unitary inner
product.
Orthogonality examples
1. Consider V := <2 , the unitary inner product hx, y i = x T y
on <2 , and subspaces U := [{(1, 1)}] and
W := [{(1, −1)}]. Back
2. Consider (x, x) ∈ U and (y , −y ) ∈ W . Then,
h(x, x), (y , −y )i = (x, x)T (y , −y ) = xy − xy = 0.
3. So, U and W are orthogonal complements of each other.
4. Orthogonality depends critically on the inner product.
2
5. Suppose the inner product on  < is generated by the
2 0
positive definite matrix A = .
0 1
6. Then, h(1, 1), (1, −1)i = (1, 1)T A(1, −1) 6= 0. So, given the
inner product generated by A, subspaces U and V are not
orthogonal to each other.
7. However, with the new inner product, U and
W 0 := [{(1, −2)}] are orthogonal to each other.
Adjoint examples

1. Consider inner product spaces (<n , h., .in ) and (<m , h., .im ),
where h., .in and h., .im are the unitary inner products on <n
and <m respectively. Back
2. Let the m × n matrix A generate the linear mapping
x 7→ Ax from <n to <m . Let the n × m matrix AT generate
the linear mapping y 7→ AT y from <m to <n .
3. For every x ∈ <n and y ∈ <m ,
hx, AT y in = x T AT y = (Ax)T y = hAx, y im .
7 AT y represents the adjoint transformation of
4. Thus, y →
x 7→ Ax .
5. This characterisation depends on the unitary inner
products.
(In)dependent proofs I
Proof.
1. Suppose there exist sets {xP 1 , . . . , xn } ⊂ X and
{α1 , . . . , αn } ⊂ F such that ni=1 αi xi = 0 and α1 6= 0. Back

Pn
1.1 As x1 = − i=2 (αi /α1 )xi , we have
x1 ∈ [{x2 , . . . , xn }] ⊂ [X \ {x1 }]. So, X is dependent.
1.2 Conversely, suppose X is dependent. Then, there exists
y ∈ V such that y ∈ [X \ {y }]. So, there exists
{x1 , . . . , xn } ⊂ X \ {y } and {α1 , . . . , αn } ⊂ F such that
Pn Pn+1
y = i=1 αi xi . Setting xn+1 = y , we have i=1 αi xi = 0
with αn+1 = −1.
2. Trivial. Back

3. Trivial. Back

4. Trivial. Back

5. Trivial. Back
(In)dependent proofs II

Proof.
6. If X ∪ {y } is independent, then
y 6∈ [(X ∪ {y }) \ {y }] = [X \ {y }], as required. Back

6.1 Conversely, suppose y 6∈ [X \ {y }].

6.2 Consider
Pn {x1 , . . . , xn } ⊂ X ∪ {y } and the equation
i=1 α i x i = 0. We need to show that α1 = . . . = αn = 0.
6.3 If {x1 , . . . , xn } ⊂ X , then α1 = . . . = αn = 0 as X is
independent.
6.4 If {x1 , . . . , xn } 6⊂ X , then there is some xi , say Pxn1 , such that
x1 6∈ X . Then, x1 = y . If α1 6= 0, then y = − i=2 (αi /α1 )xi ,
which means y P ∈ [X \ {y }], a contradiction.
n
So, α1 = 0 and i=2 αi xi = 0, whose only solution is
α2 = . . . = αn = 0 as X is independent.
(In)dependent proofs III

Proof.
7. If {x1 , . . . , xn } is dependent, then there
P exist
α1 , . . . , αn ∈ F , not all 0, such that ni=1 αi xi = 0. Let
m = max{i Pm−1 | αi 6= 0}. Then,
xm = − i=1 (αi /αm )xi ∈ [{x1 , . . . , xm−1 }], as required.
The converse is trivial. Back
Lemma

Lemma
If B = {y1 , . . . , yn } is a basis for V and x ∈ V \ {0}, then there
is a set B 0 ⊂ B, B 0 6= B, such that {x} ∪ B 0 is a basis for V .

Proof.
1. As B is a basis for V , {x} ∪ B is dependent and spans V .
2. Then, there exists yj such that yj ∈ [{x, y1 , . . . , yj−1 }]. Ref

3. Let B1 = B \ {yj }. Clearly, [{x} ∪ B1 ] = [{x} ∪ B] = V .

4. If {x} ∪ B1 is independent, then set B 0 = B1 .
5. If {x} ∪ B1 is dependent, then implement Steps 2 and 3 on
{x} ∪ B1 .
6. In k ≤ n steps, we have Bk ⊂ B, Bk 6= B, such that
[{x} ∪ Bk ] = V and {x} ∪ Bk is independent. Set B 0 = Bk .
Dimension proofs
Proof.
1. Let X1 = {x1 , . . . , xn } and Y1 = {y1 , . . . , ym } be bases of V .
Back

2. By Lemma Ref , there exists X2 ⊂ X1 , X2 6= X1 , such that

{y1 } ∪ X2 is a basis for V . By Lemma Ref , there exists
X3 ⊂ X2 , X3 6= X2 , such that {y1 , y2 } ∪ X3 is a basis for V .
3. Iterate this process on the bases of the form
{y1 , . . . , yk } ∪ Xk +1 , stopping at k = k ∗ when either k ∗ = m
or Xk ∗ +1 = ∅.
4. If k ∗ < m, then Xk ∗ +1 = ∅, which contradicts the hypothesis
that Y1 is a basis.
5. So, k ∗ = m. Since each iteration removes at least one
vector from Y1 , m = k ∗ ≤ n.
6. Reversing the roles of X1 and Y1 , we have n ≤ m.
Bases proofs I

Proof.
1. If [X ] = V , the result holds. Suppose [X ] 6= V . Back

1.1 Let B be a basis for V ; note that |B| = n. If B ⊂ [X ], then

V = [B] ⊂ [X ] ⊂ V , a contradiction.
1.2 Suppose B 6⊂ [X ] and b1 ∈ B \ [X ]. Consider X ∪ {b1 },
Pk
{x1 , . . . , xk } ⊂ X , and the equation β1 b1 + i=1 αi xi = 0. If
β1 = 0, then the independence of X implies
Pk
α1 = . . . = αn = 0. If β1 6= 0, then b1 = − i=1 (αi /β1 )xi ,
which contradicts b1 6∈ [X ]. So, X ∪ {b1 } is independent.
1.3 Suppose B 0 ⊂ B and X ∪ B 0 is independent. If B ⊂ [X ∪ B 0 ],
then [X ∪ B 0 ] = V . If B 6⊂ [X ∪ B 0 ], then there exists
b0 ∈ B \ [X ∪ B 0 ] such that X ∪ B 0 ∪ {b0 } is independent.
1.4 For some B 0 ⊂ B, X ∪ B 0 is independent and spans V as
V = [B] ⊂ [X ∪ B] ⊂ V .
Bases proofs II

Proof.
2. If {x1 , . . . , xn } is a basis for V , then it spans V by definition.
Back

2.1 Conversely, suppose {x1 , . . . , xn } spans V . If {x1 , . . . , xn } is

independent, then it is a basis for V .
2.2 If {x1 , . . . , xn } is dependent, then we can remove a
dependent vector from it and the remaining set with n − 1
vectors will still span V .
2.3 In no more than n iterations of this procedure, we will find a
proper subset of {x1 , . . . , xn } that is independent and spans
V.
2.4 Such a set with less than n vectors is a basis for V , which
contradicts the fact that every basis has the same number
of vectors.
Bases proofs III

Proof.
3. If {x1 , . . . , xn } is a basis for V , then it is independent by
definition. Back
3.1 Conversely, suppose X := {x1 , . . . , xn } is independent.
3.2 If [X ] 6= V , then there exists y ∈ V \ [X ]. Since [X ∪ {y }] is
independent and every basis must have the same number
of vectors, the number of vectors in a basis for V must be at
least n + 1, which contradicts dim V = n.
Direct sum proofs I

Proof.
1. Let V = S ⊕ T and x ∈ V . Suppose x = y1 + z1 = y2 + z2 ,
where y1 , y2 ∈ S and z1 , z2 ∈ T . Then,
y1 − y2 = z2 − z1 ∈ S ∩ T = [0]. Consequently, y1 = y2 and
z1 = z2 .
The converse is trivial. Back
2. Let X := {x1 , . . . , xk } be a basis for S ∩ T . Since X ⊂ S is
independent, it can be extended to a basis
Y := X ∪ {y1 , . . . , yl } for S. Similarly, there is a basis
Z := X ∪ {z1 , . . . , zm } for T .
So, dim S = k + l, dim T = k + m, and dim(S ∩ T ) = k .
We show that Y ∪ {z1 , . . . , zm } is a basis for S + T , i.e.,
dim(S + T ) = k + l + m, as required. Back
Direct sum proofs II

Proof.
3. Let X := {x1 , . . . , xk } be a basis for S ∩ T . As X is
independent, it is extendable to a basis
Y := X ∪ {y1 , . . . , yl } for S. Similarly, there is a basis
Z := X ∪ {z1 , . . . , zm } for T . It suffices to show that
Y ∪ {z1 , . . . , zm } is a basis for S + T . This set clearly spans
S + T . So, it suffices to show that it is independent. Back
3.1 Consider the equation
Pk Pl Pm
i=1 αi xi + j=1 βj yj + h=1 γh zh = 0. It follows that
Pm
h=1 γh zh ∈ S ∩ T .
Pm Pk
3.2 Thus, h=1 γh zh = i=1 δi xi for some δ1 , . . . , δk .
3.3 As Z is independent, γ1 = . . . = γm = 0 = δ1 = . . . = δk .
Pk Pl
Thus, i=1 αi xi + j=1 βj yj = 0.
3.4 As Y is independent, α1 = · · · = αk = 0 = β1 = . . . = βl .
Isomorphism proofs

Proof.
1. Given a basis {x1 , . . . , xn } for V and y ∈ V , there is a
n
Pnn-tuple (α1 (y ), . . . , αn (y )) ∈ F such that
unique
y = i=1 αi (y )xi . Back
2. It is easy to check that y 7→ (α1 (y ), . . . , αn (y )) is an
isomorphism (i.e., a linear bijection) from V to F n .
Rank and nullity proofs I

Proof.
1. Let ν(A) = k and let {x1 , . . . , xk } be a basis for N (A). Back

1.1 Use Theorem Ref to extend {x1 , . . . , xk } to a basis

{x1 , . . . , xk , xk +1 , . . . , xn } for V . We show that
{Axk +1 , . . . , Axn } is a basis for R(A).
1.2 If y ∈PR(A), then y = Ax for some Pnx ∈ V . Let
n
x = i=1 αi xi . Then, y = Ax = i=k +1 αi Axi . So,
[{Axk +1 , . . . , Axn }] = R(A).
1.3 Suppose {Axk +1 , . . . , Axn } is dependent. Then, there exist
n−k
(αk +1P, .n. . , αn ) ∈ F \P{0} such that
n
0 = i=k +1 αi Axi = A i=k +1 αi xi . Therefore,
Pn
i=k +1 αi xi ∈ N (A). Consequently, there exists
Pn Pk
(β1 , . . . , βk ) ∈ F k such that i=k +1 αi xi = i=1 βi xi , which
contradicts the fact that {x1 , . . . , xn } is independent. So,
{Axk +1 , . . . , Axn } is independent.
Rank and nullity proofs II

Proof.
2. If A is not injective, then there exist x, y ∈ V , x 6= y , such
that A(x − y ) = Ax − Ay = 0. Thus, x − y 6= 0,
x − y ∈ N (A), and ν(A) > 0. Back
3. As R(A) ⊂ W and dim R(A) = dim W , we have R(A) = W .
Back

4. Combine 2 and 3. Back

Rank, nullity, and composition mapping proofs

Proof.
1. Clearly, ABx = A(Bx) ∈ W . Consider x, y ∈ U and
α, β ∈ F . Then,
AB(αx + βy ) = A(αBx + βBy ) = αABx + βABy . Back
2. Clearly, R(AB) = AR(B) ⊂ AV = R(A). As A0 = 0, we
have N (B) ⊂ N (AB). Back
3. As R(AB) ⊂ R(A), we have ρ(AB) ≤ ρ(A). As
ρ(AB) = ρ(B) − ν(AR(B) ) ≤ ρ(B). As N (B) ⊂ N (AB), we
have ν(B) ≤ ν(AB). Back
Matrix representation of mapping
Proof.
1. Let v = nj=1 bj vj be the representation of v in terms of the
P

basis {v1 , . . . , vn }. Let Fvj = m

P
i=1 aij wi , for j = 1, . . . , n, be
the representation of Fvj in terms of the basis
{w1 , . . . , wm }. Back
2. Then, Fv = nj=1 bj Fvj = nj=1 bj m
P P P
Pm Pn i=1 aij wi =
i=1 ( j=1 bj aij )wi .
3. So, Fv = m
P Pn
i=1 ci wi , where ci = j=1 bj aij for i = 1, . . . , m.
4. The equations determining (c1 , . . . , cm ) can be written as
    
c1 a11 . . . a1n b1
 ..   .. .. ..   .. 
 . = . . .  . 
cm am1 . . . amn bn
Projector proofs I
Proof.
1. There exist subspaces U and W such that V = U ⊕ W and
P projects U along W . Let v1 , v2 ∈ V and α1 , α2 ∈ F .
Then, there exist unique u1 , u2 ∈ U and w1 , w2 ∈ W such
that v1 = u1 + w1 and v2 = u2 + w2 . Therefore,
P(α1 v1 + α2 v2 ) = P((α1 u1 + α2 u2 ) + (α1 w1 + α2 w2 )) =
α1 u1 + α2 u2 = α1 Pv1 + α2 Pv2 . Clearly, R(P) = U and
N (P) = W . Back
2. Suppose P is a projector.
2.1 Consider z = x + y , where x ∈ R(P) and y ∈ N (P). So,
P 2 z = PPz = PP(x + y ) = Px = x = P(x + y ) = Pz. Back
2.2 If x ∈ R(I − P), then x = y − Py for some y ∈ V .
Therefore, Px = Py − P 2 y = 0. So, R(I − P) ⊂ N (P).
2.3 If x ∈ N (P), then Px = 0. Consequently,
(I − P)x = x − Px = x. So, R(I − P) ⊃ N (P).
Projector proofs II

Proof.
3. Suppose P ∈ L(V , V ) is idempotent. Back

3.1 If x ∈ N (P), then x = Px + (I − P)x = (I − P)x. So,

N (P) ⊂ R(I − P).
3.2 If y ∈ R(I − P), then there exists x ∈ V such that
(I − P)x = y and Py = P(I − P)x = (P − P 2 )x = 0. So,
N (P) ⊃ R(I − P).
3.3 As x = Px + (I − P)x for every x ∈ V , we have
V = R(P) + R(I − P) = R(P) + N (P). If y ∈ R(P) ∩ N (P),
then for some x ∈ V , y = Px = P 2 x = Py = 0. So,
V = R(P) ⊕ N (P).
3.4 If u ∈ R(P) and w ∈ N (P), then u = Px for some x ∈ V .
So, P(u + w) = Pu = P 2 x = Px = u and P is a projector.
Projector proofs III
Proof.
4. Suppose P is a projector. Back

4.1 If y ∈ U, then Xy ∈ R(X ) = R(P), so that PXy = Xy . Thus,

PX = X .
4.2 Conversely, suppose PX = X . Given y ∈ V ,
Py ∈ R(P) = R(X ) and there exists z ∈ U such that
Py = Xz. Therefore, P 2 y = PXz = Xz = Py . Using Ref , P
is a projector.
5. Suppose P is a projector. Back

5.1 Then, P, I − P ∈ L(V , V ). Using Ref , P 2 = P. So,

(I − P)2 = I − P − P + P 2 = I − P. Using Theorem Ref ,
I − P is a projector.
5.2 Conversely, suppose I − P is a projector. Then,
I − P, P ∈ L(V , V ). Using Ref , (I − P)2 = I − P. So,
P 2 = (I − P)2 − I + 2P = I − P − I + 2P = P.Using
Theorem Ref , P is a projector.
Inverse mapping proofs I
Proof.
1. Suppose B is a left inverse of A. Back

1.1 If x ∈ N (A), then x = Ix = BAx = B0 = 0. Thus,

N (A) = [0], i.e. ν(A) = 0.
1.2 Conversely, let ν(A) = 0. Then N (A) = [0]. Consider
x, y ∈ V such that x 6= y . Then x − y 6= 0. Consequently,
Ax − Ay = A(x − y ) 6= 0. Therefore, A is injective and has a
function inverse f : R(A) → V .
1.3 In order to define a linear transformation on a vector space,
it suffices to define it over a basis for that vector space.
Consider( a basis for W and let y belong to this basis. Let
f (y ), y ∈ R(A)
By =
0, y 6∈ R(A)
By construction, BAx = f (Ax) = x for every x ∈ V . Note
that, in general, B is not unique.
Inverse mapping proofs II

Proof.
2. Suppose B is a right inverse of A. Back

2.1 By definition, R(A) ⊂ W . Consider y ∈ W . By definition,

y = Iy = ABy . As By ∈ V , y ∈ R(A). Thus, W ⊂ R(A).
Therefore, W = R(A). Consequently, ρ(A) = dim W .
2.2 Conversely, let ρ(A) = dim W . Then W = R(A). Therefore,
A is surjective and there exists a function f : W → V such
that Af (y ) = y for every y ∈ W . Consider a basis for W and
let y belong to this basis. Let By = f (y ). By construction,
ABy = Af (y ) = y for every y ∈ W . Note that, in general, B
is not unique.
3. Combine 1 and 2. Back
Inverse mapping proofs III
Proof.
4. Suppose B and C are regular inverses. Then
B = BI = BAC = IC = C. Back
5. Suppose A and B are non-singular. Back

5.1 Using Theorem Ref , ρ(A) = dim W and ρ(B) = dim V .

Therefore, R(A) = W and R(B) = V . So,
R(AB) = AR(B) = AV = R(A) = W .
5.2 Using Theorem Ref , ν(A) = 0 and ν(B) = 0. Therefore,
N (A) = [0] and N (B) = [0]. If x ∈ U and x 6= 0, then
Bx 6= 0, and therefore, ABx 6= 0. Consequently,
N (AB) = [0] and ν(AB) = 0.
5.3 Therefore, AB is non-singular; using Ref , it has a unique
regular inverse (AB)−1 .
5.4 As AB(AB)−1 = IW = AA−1 = AIV A−1 = ABB −1 A−1 and
the regular inverse is unique, we have (AB)−1 = B −1 A−1 .
g-inverse proofs I

Proof.
1. Using Theorem Ref , ρ(A) = ρ(AA− A) ≤
min{ρ(AA− ), ρ(A)} ≤ min{min{ρ(A), ρ(A− )}, ρ(A)} ≤ ρ(A− )
Back

2. As R(AA− ) = AR(A− ) ⊂ AV = R(A), we have

ρ(AA− ) ≤ ρ(A).
Also, applying Theorem Ref to the restriction of A− to
R(A), we have ρ(A− A) = ρ(A) − ν(A− A) ≤ ρ(A). Back
2.1 Using Theorem Ref , ρ(AA− ) ≤ ρ(A) = ρ(AA− A) ≤
min{ρ(AA− ), ρ(A)} = ρ(AA− ). Therefore, ρ(A) = ρ(AA− ).
2.2 Using Theorem Ref , ρ(A− A) ≤ ρ(A) = ρ(AA− A) ≤
min{ρ(A), ρ(A− A)} = ρ(A− A). Therefore, ρ(A) = ρ(A− A).
3. Follows from 2. Back
g-inverse proofs II
Proof.
4. We first show that AA− projects W on R(A). Back

4.1 AA− is idempotent as AA− AA− = AA− .

4.2 Using Theorem Ref , AA− projects W on R(AA− ).
4.3 As R(AA− ) ⊂ R(A) and ρ(AA− ) = ρ(A), we have
R(AA− ) = R(A). Thus, AA− projects W on R(A).
Next, we show that A− A projects V on R(A− A).
4.1 A− A is idempotent as A− AA− A = A− A.
4.2 Using Theorem Ref , A− A projects V on R(A− A).
5. We need to show that BA(BA)− B = B. Back

5.1 P := BA(BA)− is idempotent. Using Theorem Ref , P

projects U on R(BA).
5.2 R(BA) ⊂ R(B) and ρ(BA) = ρ(B). So, R(BA) = R(B).
5.3 Therefore, P projects U on R(B).
5.4 Using Theorem Ref , B = PB = BA(BA)− B.
g-inverse proofs III

Proof.
6. We need to show that A(BA)− BA = A. Back

6.1 We have BA(BA)− BA = BA.

6.2 If ρ(BA) = ρ(A), then the restriction of B to R(A) is injective.
6.3 Therefore, we have A(BA)− BA = A.
7. Combine Theorems Ref and Ref . Back

8. As AA− A = A, we have ρ(AA− A) = ρ(A). Back

8.1 If ν(A) = 0, then ρ(A) = dim V .

8.2 As A is injective, ρ(A− A) = ρ(AA− A) = ρ(A) = dim V .
Therefore, R(A− A) = V .
8.3 Using Theorem Ref , A− A projects V on R(A− A). Thus,
A− A projects V on V , i.e., A− A = I. This implies A− = AL .
g-inverse proofs IV

Proof.
9. Using Theorem Ref , AA− projects W on R(A). Back

9.1 If ρ(A) = dim W , then R(A) = W .

9.2 Thus, AA− projects W on W , i.e., AA− = I. Therefore,
A− = AR .
10. Combine Theorems Ref and Ref . Back
Homogeneous equation proofs

Proof.
1. We have to show that R(I − A− A) = N (A). Back

1.1 Since A(I − A− A) = 0, we have R(I − A− A) ⊂ N (A).

1.2 Consider x ∈ N (A). Then, Ax = 0. As
(I − A− A)x = x − A− Ax = x − A− 0 = x, we have
x ∈ R(I − A− A). So, R(I − A− A) ⊃ N (A).
1.3 Therefore, R(I − A− A) = N (A).
Non-homogeneous equation proofs I

Proof.
1. If Ax = y is consistent, then y ∈ R(A). We know that AA−
projects W on R(A). Therefore, AA− y = y .
The converse is trivial. Back
2. Suppose Ax = y has a unique solution for every y ∈ W . If
ν(A) > 0, then there exists x ∈ N (A) such that x 6= 0. This
violates the uniqueness assumption. Conversely, if
ν(A) = 0, then AL exists. If Ax = y is consistent, then
y ∈ R(A). Thus, there exists x ∈ V such that Ax = y . This
means AL y = AL Ax = Ix = x, i.e. AAL y = y . Back
3. It follows from the assumptions that A−1 exists. Therefore,
the equation Ax = y has a unique solution A−1 y for every
y ∈ W . Back
Non-homogeneous equation proofs II

Proof.
4. Let x and x 0 be solutions of Ax = y . Back

4.1 Then Ax = y and Ax 0 = y . Consequently,

0 = Ax − Ax 0 = A(x − x 0 ), i.e., x − x 0 ∈ N (A).
4.2 Thus, the difference between any two solutions of the
equation Ax = y is a vector in N (A).
4.3 As Ax = y is consistent, Theorem Ref implies that
AA− y = y . So, A− y is a solution of Ax = y . Then, using
Theorem Ref , the set of all solutions of Ax = y must be
A− y + N (A) = A− y + (I − A− A)V .
Cauchy-Schwartz inequality proof

Proof.
1. If y = 0, then hy , y i = 0 and
hx, y i = hx, 0i = hx, c − ci = hx, ci − hx, ci = 0, where
c ∈ V . So, the inequality holds. Back
1.1 Suppose y 6= 0. Then, hy , y i > 0.
1.2 For every λ ∈ <, we have
0 ≤ hx − λy , x − λy i = hx, xi + λ2 hy , y i − 2λhx, y i.
1.3 Setting λ = hx, y i/hy , y i, we have
0 ≤ hx, xi + hx, y i2 /hy , y i − 2hx, y i2 /hy , y i, which yields the
result.
2. If y = 0 or x = λy for some λ ∈ <, then the equality is
satisfied. Back
2.1 Conversely, suppose y 6= 0 and x 6= λy for every λ ∈ <.
Then, the above argument holds with a strict inequality.
Orthonormal sets proofs I

Proof.
1. Consider P an orthonormal set {c1 , . . . , cr } ⊂ V and the
r
equation
Pr i = 0. For every j
∈
i=1 αi cP P{1, . . . , r },
r r
αj = i=1 αi δij = i=1 αi hci , cj i = i=1 αi ci , cj =
h0, cj i = 0. Back
2. Let y ∈ V . As {c1 , . P. . , cr } is a basis for V, y has a
representation y = ri=1 αi ci . Then, as {c1 , . . . , cr } is an
orthonormal
Prset,  Pr Pr
hy , cj i = i=1 αi ci , cj = i=1 αi hci , cj i = i=1 αi δij = αj .
Back
Gram-Schmidt Lemma proof
Proof.
Pk −1
1. Let zk := xk − j=1 hyj , xk iyj . For every i ∈ {1, . . . , k − 1},
Pk −1
hzk , yi i = hxk , yi i − j=1 hyj , xk ihyj , yi i =
hxk , yi i − hyi , xk i = 0. Back
P −1
2. If zk = 0, then xk = kj=1 hyj , xk iyj , i.e.,
xk ∈ [{y1 , . . . , yk −1 }] = [{x1 , . . . , xk −1 }], a contradiction.
3. As zk 6= 0, let yk := zk /kzk k. Then, kyk k = 1, and by Step
1, kzk kyk = zk ∈ [{y1 , . . . , yk −1 }]⊥ . Thus, {y1 , . . . , yk } is
orthonormal.
P −1
4. Note that xk = kj=1 hyj , xk iyj + kzk kyk .
5. As [{x1 , . . . , xk −1 }] = [{y1 , . . . ,P
yk −1 }], there exists
−1 P −1
(α1 , . . . , αk −1 ) ∈ F n such that kj=1 hyj , xk iyj = kj=1 αj xj .
Pk −1
So, kzk kyk = xk − j=1 αj xj .
6. Thus, [{x1 , . . . , xk }] ⊂ [{y1 , . . . , yk }] ⊂ [{x1 , . . . , xk }].
Gram-Schmidt proof

Proof.
1. Let {x1 , . . . , xn } be a basis for V . Then, every xi 6= 0. Back

2. Let y1 = x1 /kx1 k. Then, ky1 k = 1. So, {y1 } is an

orthonormal set and [{y1 }] = [{x1 }].
3. For k ∈ {2, . . . , n}, suppose {y1 , . . . , yk −1 } is an
orthonormal set and [{y1 , . . . , yk −1 }] = [{x1 , . . . , xk −1 }].
4. Using the Lemma Ref , there exists yk ∈ V such that
{y1 , . . . , yk } is an orthonormal set with
[{y1 , . . . , yk }] = [{x1 , . . . , xk }].
5. Proceed inductively until k = n. Since orthonormal sets
are independent by Theorem Ref , {y1 , . . . , yn } is
independent and [{y1 , . . . , yn }] = [{x1 , . . . , xn }] = V . Thus,
{y1 , . . . , yn } is the required orthonormal basis.
Orthogonal decomposition proof
Proof.
1. Let {c1 , . . . , cr } be an orthonormal basis for W . Using the
Gram-Schmidt construction, this basis can be extended to
an orthonormal basis {c1 , . . . , cr , cr +1 , . . . , cn } for V . Back
2. So, y = ni=1 hy , ci ici for every y ∈ V .
P

3. Given y ∈ V , let p = ri=1 hy , ci ici and q = ni=r +1 hy , ci ici .

P P
Thus, y = p + q. Clearly, p ∈ W .
4. Any z ∈ W has a unique representation z = ri=1 hz, ci ici .
P
As {c1 , . . . , cn } is an orthonormal set and q is a linear
combination of {cr +1 , . . . , cn }, it follows that hz, qi = 0.
Thus, q ∈ W ⊥ .
5. It follows that V = W + W ⊥ .
6. Let y ∈ W ∩ W ⊥ . This implies hy , y i = 0. Thus, y = 0.
Consequently, W ∩ W ⊥ = [0] and W ⊕ W ⊥ = V .
Adjoint proof I

Proof.
1. Let y ∈ W . Using Ref , V has an orthonormal basis
{e1 , ..., en }. Back
1.1 If B ∈ L(W , V ) is an adjoint of A, then hei , By i = hAei , y i for
i = 1, P. . . , n. Consequently,
n Pn
By = i=1 hei , By iei = i=1 hAei , y iei (1)
Since every adjoint of A satisfies (1), it is unique if it exists.
1.2 We now verify that B given Pnby (1) is an adjoint of A.
1.3 Using
Pn (1), he j , By i = he j , i=1 hAei , y iei i =
i=1 hAei , y ihej , ei i = hAe Pny i for every ej and y ∈ W .
j ,
1.4 Consider x ∈P V . As x = i=1 hx,P ei iei ,
n n
hAx,
Pn y i = hA i=1 hx, ei ie
Pn i , y i = i=1 hx, ei ihAei , y i =
i=1 hx, ei ihei , By i = h i=1 hx, ei iei , By i = hx, By i. So,
A∗ = B.
Adjoint proofs II

Proof.
2. hy , (A∗ )∗ xiW = hA∗ y , xiV = hy , AxiW for all x ∈ V and
y ∈ W . Thus, hy , (A∗ )∗ x − AxiW = 0 for all x ∈ V and
y ∈ W . So, (A∗ )∗ x = Ax for every x ∈ V , i.e., (A∗ )∗ = A.
Back

3. Apply Theorem Ref . Back

4. Suppose x ∈ N (A). Back

4.1 Then hx, A∗ y iV = hAx, y iW = h0, y iW = 0 for every y ∈ W ,

i.e., x ∈ R(A∗ )⊥ . So, N (A) ⊂ R(A∗ )⊥ .
4.2 Conversely, if x ∈ R(A∗ )⊥ , then 0 = hx, A∗ y iV = hAx, y iW
for every y ∈ W . Then, Ax = 0, i.e., x ∈ N (A).
4.3 To simplify notation, let B := A∗ . Then, R(A) = R((A∗ )∗ ) =
R(B ∗ ) = (R(B ∗ )⊥ )⊥ = (N (B))⊥ = (N (A∗ ))⊥ .
Adjoint proofs III

Proof.
5. Using Theorem Ref , V = N (A) ⊕ N (A)⊥ . Using Theorem
Ref , dim V = ν(A) + dim N (A)⊥ . Back

5.1 Using Theorem Ref , R(A∗ )⊥ = N (A).

5.2 Therefore, R(A∗ ) = (R(A∗ )⊥ )⊥ = N (A)⊥ .
5.3 So, ρ(A∗ ) = dim N (A)⊥ = dim V − ν(A) = ρ(A).
6. Using Theorem Ref , R(A) = N (A∗ )⊥ . Using Theorem Ref ,
W = N (A∗ ) ⊕ N (A∗ )⊥ = N (A∗ ) ⊕ R(A). Back
6.1 Therefore, R(A∗ ) = A∗ W = A∗ [N (A∗ ) ⊕ R(A)] =
A∗ N (A∗ ) ⊕ A∗ R(A) = [0] ⊕ R(A∗ A) = R(A∗ A).
6.2 Clearly, N (A) ⊂ N (A∗ A). Using Theorems Ref and Ref ,
ν(A∗ A) = dim V − ρ(A∗ A) = dim V − ρ(A∗ ) =
dim V − ρ(A) = ν(A).
Adjoint proofs IV
Proof.
1. For all u ∈ U and w ∈ W , as
hu, (AC)∗ wi = hACu, wi = hCu, A∗ wi = hu, C ∗ A∗ wi, we
have hu, (AC)∗ w − C ∗ A∗ wi = 0. Back
2. For all v ∈ V and w ∈ W , as
h(A + B)∗ w, v i = hw, (A + B)v i = hw, Av i + hw, Bv i =
hA∗ w, v i + hB ∗ w, v i = hA∗ w + B ∗ w, v i, we have
h(A + B)∗ w − (A∗ + B ∗ )w, v i = 0. Back
3. For all v ∈ V and w ∈ W , as hv , (λA)∗ wi = hλAv , wi =
λhAv , wi = λhv , A∗ wi = hv , λA∗ wi, we have
hv , (λA)∗ w − λA∗ wi = 0. Back
4. For all v ∈ V and w ∈ W , as
hv , I ∗ wi = hIv , wi = hv , wi = hv , Iwi, we have
hv , I ∗ w − Iwi = 0. Back
Adjoint proofs V

Proof.
The following proofs use Theorems Ref and Ref .
5. I = I ∗ = (AA−1 )∗ = (A−1 )∗ A∗ and
I = I ∗ = (A−1 A)∗ = A∗ (A−1 )∗ . Back
6. If AL exists, then I = I ∗ = (AL A)∗ = A∗ (AL )∗ . So,
(A∗ )R = (AL )∗ . Back
7. If AR exists, then I = I ∗ = (AAR )∗ = (AR )∗ A∗ . So,
(A∗ )L = (AR )∗ . Back
8. Using the definition of A− ,
A∗ = (AA− A)∗ = A∗ (AA− )∗ = A∗ (A− )∗ A∗ . By the definition
of (A∗ )− , we have (A∗ )− = (A− )∗ . Back
Orthogonal projector proof I

Proof.
1. Suppose P is an orthogonal projector. Back

1.1 As P is a projector, it is idempotent by Theorem Ref .

1.2 Consider x, y ∈ V . As P is a projector, V = R(P) ⊕ N (P).
Therefore, x = u1 + v1 and y = u2 + v2 , where
u1 , u2 ∈ R(P) and v1 , v2 ∈ N (P). As R(P) = N (P)⊥ ,
hx, Py i = hu1 + v1 , P(u2 + v2 )i = hu1 , u2 i =
hP(u1 + v1 ), u2 + v2 i = hPx, y i. Thus, P is self-adjoint.
1.3 Conversely, suppose P is idempotent and self-adjoint.
Then, by Theorem Ref , P is a projector on R(P).
1.4 Let x ∈ R(P) and y ∈ N (P). Then, x = Pu for some u ∈ V .
As P is self-adjoint, hx, y i = hPu, y i = hu, Py i = hu, 0i = 0.
Thus, R(P) = N (P)⊥ . So, P is an orthogonal projector.
Orthogonal projector proof II

Proof.
2. Suppose P is an orthogonal projector. Back

2.1 Using Theorem Ref , P = P ∗ = P 2 . Consequently,

P ∗ (I − P) = P ∗ − P ∗ P = P − P 2 = 0.
2.2 Conversely, suppose P ∗ (I − P) = 0. Then, P ∗ = P ∗ P.
2.3 Using Theorem Ref , P = (P ∗ )∗ = (P ∗ P)∗ = P ∗ P = P ∗ , i.e.,
P is self-adjoint. Moreover, P 2 = PP = P ∗ P = P, i.e., P is
idempotent. Thus, P is an orthogonal projector.
Orthogonal projector minimises distance proof

Proof.
1. Suppose P is an orthogonal projector of V on W . Let
x ∈ V and w ∈ W . Back
1.1 Then, w − Px ∈ W = R(P) and
x − Px ∈ R(I − P) = N (P) = W ⊥ .
1.2 Consequently, hx − Px, w − Pxi = 0.
1.3 As kx − wk2 = k(x − Px) + (Px − w)k2 =
kx − Pxk2 + kPx − wk2 + 2hx − Px, Px − wi, we have

kx − wk2 − kx − Pxk2 = kPx − wk2 + 2hx − Px, Px − wi

= kPx − wk2
≥ 0

1.4 So, Px minimises the distance between x and W .

Minimiser orthogonal characterisation proof I

Proof.
2. Conversely, suppose Px ∈ W and kPx − xk ≤ kw − xk for
all x ∈ V and w ∈ W .
3. We first show that x − Px ∈ W ⊥ . Moreover, if x ∈ W ⊥ ,
then Px = 0.
3.1 If w, w 0 ∈ W and t ∈ (0, 1), then
0 ≥ kx − Pxk2 − kx − (tw 0 + (1 − t)Px)k2 = kx − Pxk2 − k(x −
Px) − t(w 0 − Px)k2 = 2thx − Px, w 0 − Pxi − t 2 kw 0 − Pxk2 .
Therefore, 0 ≥ 2hx − Px, w 0 − Pxi − tkw 0 − Pxk2 .
Letting t ↓ 0, we have 0 ≥ hx − Px, w 0 − Pxi.
3.2 If w 0 = Px + w, then 0 ≥ hx − Px, wi. If w 0 = Px − w, then
0 ≤ hx − Px, wi. Thus, hx − Px, wi = 0, i.e., x − Px ∈ W ⊥ .
3.3 If x ∈ W ⊥ , then kx − wk2 = kxk2 + kwk2 ≥ kx − 0k2 , for
every w ∈ W . Thus, Px = 0.
Linearity of distance minimiser I
Proof.
4. We now show that P ∈ L(V , W ), i.e., P is linear.
4.1 kw 0 − (x + w)k = k(w 0 − w) − xk for w, w 0 ∈ W and x ∈ V .
Therefore, kw 00 − (x + w)k ≥ kw 0 − (x + w)k iff.
k(w 00 − w) − xk ≥ k(w 0 − w) − xk.
4.2 So, w 0 = P(x + w) iff. w 0 − w = P(x). Thus,
P(x + w) = P(x) + w.
4.3 For x ∈ V , let x ⊥ := x − Px. For x, y ∈ V ,
P(x + y ) = P(x ⊥ + Px + y ⊥ + Py ). As Px + Py ∈ W , we
have P(x + y ) = P(x ⊥ + y ⊥ ) + Px + Py .
4.4 By Lemma Ref , x ⊥ , y ⊥ ∈ W ⊥ . So, x ⊥ + y ⊥ ∈ W ⊥ . By
Lemma Ref , P(x ⊥ + y ⊥ ) = 0. Thus, P(x + y ) = Px + Py .
4.5 Let α 6= 0 and x ∈ V . If w ∈ W , then
kw − αxk = |α|kw/α − xk. So, if w, w 0 ∈ W , then
kw − αxk ≤ kw 0 − αxk iff. kw/α − xk ≤ kw 0 /α − xk.
Therefore, if w = P(αx), then w/α = Px. So,
P(αx) = αPx.
Linearity of distance minimiser II

Proof.
5. Using Steps Ref and Ref , R(P) = W and W ⊥ ⊂ N (P).
We finally show that W ⊥ ⊃ N (P), and consequently, P is
an orthogonal projector.
5.1 Let x ∈ N (P).
5.2 As V = W ⊕ W ⊥ , we have x = w + w ⊥ with w ∈ W and
w ⊥ ∈ W ⊥.
5.3 Then, 0 = Px = Pw + Pw ⊥ = Pw. So, w = 0 and
x = w ⊥ ∈ W ⊥ . Thus, W ⊥ ⊃ N (P).
Characterisation of orthogonal projector
Proof.
1. Let y ∈ V . As P is an orthogonal projector, P(I − P)y = 0
and (I − P)y ∈ N (P) = R(P)⊥ . Back
2. As R(X ) = R(P), Py = Xb for some b ∈ U. Therefore,
y − Xb = y − Py ∈ R(P)⊥ = R(X )⊥ .
3. So, 0 = hy − Xb, Xzi = hX ∗ y − X ∗ Xb, zi for every z ∈ U.
Thus, X ∗ y = X ∗ Xb. Using Ref ,
b = (X ∗ X )− X ∗ y + [I − (X ∗ X )− X ∗ X ]u for some u ∈ U.
4. Using Ref and Ref , ρ(X ∗ X ) = ρ(X ∗ ) = ρ(X ). Using Ref ,
X (X ∗ X )− X ∗ projects U on R(X ).
5. Thus, X [I − (X ∗ X )− X ∗ X ]u = [X − X (X ∗ X )− X ∗ X ]u =
[I − X (X ∗ X )− X ∗ ]Xu = 0.
6. So, Py = Xb = X (X ∗ X )− X ∗ y + X [I − (X ∗ X )− X ∗ X ]u =
X (X ∗ X )− X ∗ y for every y ∈ V . Thus, P = X (X ∗ X )− X ∗ .
Existence of roots and invariant subspaces I

Proof.
1. By Weierstrass’ theorem, there exists x ∈ V such that
kxk = 1 and kAxk ≥ kAy k for all y ∈ V with ky k = 1. Back
1.1 So, for every y ∈ V \ {0}, as kA(y /ky k)k ≤ kAxk, we have
kAy k ≤ kAxkky k.
1.2 Using the symmetry of A, the Cauchy-Schwartz inequality
Ref , and Step (1.1),

kAxk2 = hAx, Axi = hA2 x, xi ≤ hA2 x, A2 xi1/2 hx, xi1/2 =

kA2 xk = kA(Ax)k ≤ kAxkkAxk = kAxk2 .
1.3 So, hA2 x, xi = hA2 x, A2 xi1/2 hx, xi1/2 .
1.4 By Cauchy-Schwartz Ref , there exists λ ∈ < such that
A2 x = λx.
1.5 As 0 ≤ hAx, Axi = hA2 x, xi = λhx, xi, x 6= 0, and hx, xi > 0,
we have λ ≥ 0.
Existence of roots and invariant subspaces II

Proof.
2. By Theorem Ref , there exist λ ≥ 0 and x ∈ V \ {0} such
that √ √
0 = A2 x − λx = (A2 − λI)x = (A − λI)(A + λI)x. Back
√ √
2.1 If z =
√(A + λI)x =6 0, then (A − λI)z = 0. Set y = z and
µ = λ. √ √
(A + λI)x = 0, then Ax = − λx. Set y = x and
2.2 If z = √
µ = − λ.
Roots and orthogonal characteristic subspaces I

Proof.
1. Let λ and µ be distinct roots of A, and let Ax = λx and
Ay = µy .
1.1 Then,
λhx, y i = hλx, y i = hAx, y i = hx, Ay i = hx, µy i = µhx, y i.
Back

1.2 As λ 6= µ, we have hx, y i = 0.

Roots and orthogonal characteristic subspaces II

Proof.
2. Suppose x ∈ Li ∩ Lj for some i 6= j. Back

2.1 Then, λi x = Ax = λj x, i.e., (λi − λj )x = 0. As λi 6= λj , we

have x = 0. Thus, Li ∩ Lj = [0] for i 6= j.
2.2 For i 6= j, the orthogonality of Li and Lj follows from
Theorem Ref .
2.3 Suppose L := L1 + . . . + Lr 6= <n . By Theorem Ref ,
L ⊕ L ⊥ = <n .
2.4 Consider y ∈ L⊥ . If x ∈ Li , then hx, y i = 0. Furthermore,
hx, Ay i = hAx, y i = λi hx, y i = 0. Therefore, Ay ∈ L⊥ . This
means L⊥ is an invariant subspace with respect to A.
2.5 By Theorem Ref , there exists λ ∈ < and z ∈ L⊥ \ {0}, such
that Az = λz. But, then z ∈ L, i.e., z ∈ L ∩ L⊥ = [0]. Thus,
z = 0, a contradiction.
Roots and orthogonal characteristic subspaces III

Proof.
3. By Theorem Ref , <n = L1 ⊕ . . . ⊕ Lr . Back

3.1 By Theorem Ref , each Li has an orthonormal basis Bi .

3.2 As distinct Li and Lj are orthogonal, B := ∪ri=1 Bi is an
orthonormal set in <n . Therefore, B is linearly P
independent.
r
3.3 Each x ∈ <n has a unique representation x = i=1 xi
where xi ∈ Li .
3.4 As each xi ∈ [Bi ], we have x ∈ [B]. Thus, <n = [B].
3.5 It follows that B is a basis for <n .
3.6 By construction, each b ∈ B is a characteristic vector of A.
(Semi)definite matrices and roots Proofs

Proof.
1. By Theorem Ref , <n has an orthonormal basis {c1 , . . . , cn }
where each ci is a characteristic
Pn vector of A. Then, x ∈ <n
has a representation x = i=1 αi ci . Back
2. As Aci =P λi ci for λ
i ∈ <Pand i = 1, . . P . , n,
n n n
Ax = A α
i=1 i ic = α
i=1 i Ac i = i=1 αi λi ci .
DP E
n P n
3. Then, hAx, xi = α λ
i=1 i i i c , c
j=1 j j =
α
Pn Pn Pn 2
i=1 j=1 αi αj λi hci , cj i = i=1 αi λi .
4. The results follow from this equation.
Diagonalisation Proof

Proof.
1. By Theorem Ref , <n has an orthonormal basis {c1 , . . . , cn }
where each ci is a characteristic vector of A. Let C be the
matrix with ci as the i-th column. Back
2. By definition, AC = CΛ. By construction, C T C = I. So,
C T AC = C T CΛ = Λ.
3. As {c1 , . . . , cn } is orthonormal, it is independent.
Therefore, C is non-singular.
4. Consequently, C T = C −1 . Therefore, CΛC T = ACC T = A.
Diagonalisation of semidefinite matrices Proofs

Proof.
1. By Theorem Ref , Λ = C T AC. Back

2. Order the roots in Λ and the columns of C such that the

first r entries correspond to the r non-zero roots of A.
−1/2
3. Let D be the diagonal matrix with dii = λi for i ≤ r and
dii = 0 for i > r .
4. Then, J = D T ΛD = D T C T ACD = E T AE, where E = CD.
Diagonalisation of definite matrices Proofs

Proof.
1. By Theorem Ref , there exists a non-singular matrix C such
that C T AC = Λ. Back
2. As all the roots of A are positive by Theorems, D is a
diagonal matrix with positive diagonal entries.
3. So, D is non-singular.
4. C is non-singular by construction as it consists of
orthonormal vectors.
5. Therefore, E is non-singular. The other claims follow from
trivial manipulations.
Semidefinite matrices Proofs

Proof.
1. Consider a positive semidefinite matrix A. Back

2. If A is singular, then there exists x ∈ <n \ {0} such that

Ax = 0. So, hAx, xi = 0 and A cannot be positive definite.
3. Conversely, if A is nonsingular, then Ax = 0 does not have
a non-trivial solution, i.e., 0 is not a characteristic root of A.
4. By Theorem Ref , as A is positive semidefinite, all its roots
are non-negative. Since they are non-zero in this instance,
they are all positive. By Theorem Ref , A is positive definite.
Semidefinite matrices Proofs II

Proof.
1. Suppose A is positive definite. Back

2. By Theorem Ref , A−1 exists.

3. Symmetry of A−1 follows from Theorem Ref as
A−1 = EE T = (A−1 )T .
4. So, x T A−1 x = x T EE T x = (E T x)T (E T x) > 0 for every
x 6= 0 as E is non-singular by Theorem Ref .
5. The converse follows from the above argument and the
fact that A = (A−1 )−1 .