Mathematical

Background:
Linear Algebra

Prof. Daniel Cremers

Chapter 1
Mathematical Background:
Linear Algebra Vector Spaces

Linear Transformations
and Matrices

Multiple View Geometry Properties of Matrices

Summer 2019 Singular Value

Decomposition

Prof. Daniel Cremers

Chair for Computer Vision and Artificial Intelligence
Departments of Informatics & Mathematics
Technical University of Munich
updated May 6, 2019 1/28
 Mathematical
Overview Background:
Linear Algebra

Prof. Daniel Cremers

1 Vector Spaces
Vector Spaces

Linear Transformations
and Matrices

Properties of Matrices
2 Linear Transformations and Matrices
Singular Value
Decomposition

3 Properties of Matrices

4 Singular Value Decomposition

updated May 6, 2019 2/28

 Mathematical
Vector Space (Vektorraum) Background:
Linear Algebra

Prof. Daniel Cremers

A set V is called a linear space or a vector space over the field
R if it is closed under vector summation

+: V ×V →V
Vector Spaces

and under scalar multiplication Linear Transformations

and Matrices

Properties of Matrices
· : R × V → V, Singular Value
Decomposition

i.e. αv1 + βv2 ∈ V ∀v1 , v2 ∈ V , ∀α, β ∈ R. With respect to

addition (+) it forms a commutative group (existence of neutral
element 0, inverse element −v ). Scalar multiplication respects
the structure of R: α(βu) = (αβ)u. Multiplication and addition
respect the distributive law:
(α + β)v = αv + βv and α(v + u) = αv + αu.

Example: V = Rn , v = (x1 , . . . , xn )> .

A subset W ⊂ V of a vector space V is called subspace if

0 ∈ W and W is closed under + and · (for all α ∈ R). updated May 6, 2019 3/28
 Mathematical
Linear Independence and Basis Background:
Linear Algebra

The spanned subspace of a set of vectors Prof. Daniel Cremers

S = {v1 , . . . , vk } ⊂ V is the subspace formed by all linear

combinations of these vectors:
( k
)
 X
span(S) = v ∈ V  v = αi vi
 Vector Spaces

Linear Transformations
i=1 and Matrices

Properties of Matrices

The set S is called linearly independent if: Singular Value

Decomposition

k
X
αi vi = 0 ⇒ αi = 0 ∀i,
i=1

in other words: if none of the vectors can be expressed as a

linear combination of the remaining vectors. Otherwise the set
is called linearly dependent.

A set of vectors B = {v1 , . . . , vn } is called a basis of V if it is

linearly independent and if it spans the vector space V . A basis
is a maximal set of linearly independent vectors.
updated May 6, 2019 4/28
 Mathematical
Properties of a Basis Background:
Linear Algebra
Let B and B 0 be two bases of a linear space V . Prof. Daniel Cremers

1 B and B 0 contain the same number of vectors. This

number n is called the dimension of the space V .
2 Any vector v ∈ V can be uniquely expressed as a linear
combination of the basis vectors in B = {b1 , . . . , bn }: Vector Spaces

Linear Transformations
and Matrices
n
X Properties of Matrices
v= αi bi .
Singular Value
i=1 Decomposition

3 In particular, all vectors of B can be expressed as linear

combinations of vectors of another basis bi0 ∈ B 0 :
n
X
bi0 = αji bj
j=1

The coefficients αji for this basis transform can be

combined in a matrix A. Setting B ≡ (b1 , . . . , bn ) and
B 0 ≡ (b10 , . . . , bn0 ) as the matrices of basis vectors, we can
write: B 0 = BA ⇔ B = B 0 A−1 .
updated May 6, 2019 5/28
 Mathematical
Inner Product Background:
Linear Algebra
On a vector space one can define an inner product (dot
Prof. Daniel Cremers
product, dt.: Skalarprodukt 6= skalare Multiplikation):

h·, ·i : V × V → R

which is defined by three properties: Vector Spaces

Linear Transformations
and Matrices
1 hu, αv + βwi = αhu, v i + βhu, wi (linear)
Properties of Matrices
2 hu, v i = hv , ui (symmetric) Singular Value
Decomposition
3 hv , v i ≥ 0 and hv , v i = 0 ⇔ v = 0 (positive definite)
The scalar product induces a norm
p
| · | : V → R, |v | = hv , v i

and a metric
p
d : V × V → R, d(v , w) = |v − w| = hv − w, v − wi

for measuring lengths and distances, making V a metric space.

Since the metric is induced by a scalar product V is called a
Hilbert space. updated May 6, 2019 6/28
 Mathematical
Canonical and Induced Inner Product Background:
Linear Algebra

On V = Rn , one can define the canonical inner product for the Prof. Daniel Cremers

canonical basis B = In as
n
X
hx, y i = x > y = xi yi
i=1 Vector Spaces

Linear Transformations
which induces the standard L2 -norm or Euclidean norm and Matrices

Properties of Matrices
√ q Singular Value
|x|2 = x > x = x12 + · · · + xn2 Decomposition

With a basis transform A to the new basis B 0 given by

I = B 0 A−1 the canonical inner product in the new coordinates
x 0 , y 0 is given by:

hx, y i = x > y = (Ax 0 )> (Ay 0 ) = x 0> A> A y 0 ≡ hx 0 , y 0 iA> A

The latter product is called the induced inner product from the
matrix A.

Two vectors v and w are orthogonal iff hv , wi = 0.

updated May 6, 2019 7/28
 Mathematical
Kronecker Product and Stack of a Matrix Background:
Linear Algebra

Given two matrices A ∈ Rm×n and B ∈ Rk ×l , one can define Prof. Daniel Cremers

their Kronecker product A ⊗ B by:

 
a11 B · · · a1n B
A⊗B ≡ .. .. .. mk ×nl
 ∈R .
 
. . . Vector Spaces

am1 B ··· amn B Linear Transformations

and Matrices

Properties of Matrices

In Matlab this can be implemented by C=kron(A,B). Singular Value

Decomposition
m×n s
Given a matrix A ∈ R , its stack A is obtained by stacking
its n column vectors a1 , . . . , an ∈ Rm :
 
a1
As ≡  ...  ∈ Rmn .
 

an

These notations allow to rewrite algebraic expressions, for

example:
u > A v = (v ⊗ u)> As .
updated May 6, 2019 8/28
 Mathematical
Linear Transformations and Matrices Background:
Linear Algebra
Linear algebra studies the properties of linear transformations Prof. Daniel Cremers
between linear spaces. Since these can be represented by
matrices, linear algebra studies the properties of matrices.
A linear transformation L between two linear spaces V and W
is a map L : V → W such that:
Vector Spaces

• L(x + y ) = L(x) + L(y ) ∀x, y ∈ V Linear Transformations

and Matrices

• L(αx) = αL(x) ∀x ∈ V , α ∈ R. Properties of Matrices

Singular Value
Due to the linearity, the action of L on the space V is uniquely Decomposition

defined by its action on the basis vectors of V . In the canonical

basis {e1 , . . . , en } we have:

L(x) = Ax ∀x ∈ V ,

where
A = (L(e1 ), . . . , L(en )) ∈ Rm×n .
The set of all real m × n-matrices is denoted by M(m, n). In
the case that m = n, the set M(m, n) ≡ M(n) forms a ring
over the field R, i.e. it is closed under matrix multiplication and
summation. updated May 6, 2019 9/28
 Mathematical
The Linear Groups GL(n) and SL(n) Background:
Linear Algebra

There exist certain sets of linear transformations which form a Prof. Daniel Cremers

group.
A group is a set G with an operation ◦ : G × G → G such that:
1 closed: g1 ◦ g2 ∈ G ∀g1 , g2 ∈ G, Vector Spaces

2 assoc.: (g1 ◦ g2 ) ◦ g3 = g1 ◦ (g2 ◦ g3 ) ∀g1 , g2 , g3 ∈ G, Linear Transformations

and Matrices

3 neutral: ∃e ∈ G : e ◦ g = g ◦ e = g ∀g ∈ G, Properties of Matrices

Singular Value

4 inverse: ∃g −1 ∈ G : g ◦ g −1 = g −1 ◦ g = e ∀g ∈ G. Decomposition

Example: All invertible (non-singular) real n × n-matrices form

a group with respect to matrix multiplication. This group is
called the general linear group GL(n). It consists of all
A ∈ M(n) for which
det(A) 6= 0
All matrices A ∈ GL(n) for which det(A) = 1 form a group
called the special linear group SL(n). The inverse of A is also
in this group, as det(A−1 ) = det(A)−1
updated May 6, 2019 10/28
 Mathematical
Matrix Representation of Groups Background:
Linear Algebra

A group G has a matrix representation (dt.: Darstellung) or can Prof. Daniel Cremers

be realized as a matrix group if there exists an injective

transformation:
R : G → GL(n)
Vector Spaces
which preserves the group structure of G, that is inverse and
Linear Transformations
composition are preserved by the map: and Matrices

Properties of Matrices

R(e) = In×n , R(g ◦ h) = R(g)R(h) ∀g, h ∈ G Singular Value

Decomposition

Such a map R is called a group homomorphism (dt.

Homomorphismus).
The idea of matrix representations of a group is that they allow
to analyze more abstract groups by looking at the properties of
the respective matrix group. Example: The rotations of an
object form a group, as there exists a neutral element (no
rotation) and an inverse (the inverse rotation) and any
concatenation of rotations is again a rotation (around a
different axis). Studying the properties of the rotation group is
easier if rotations are represented by respective matrices.
updated May 6, 2019 11/28
 Mathematical
The Affine Group A(n) Background:
Linear Algebra

Prof. Daniel Cremers

n n
An affine transformation L : R → R is defined by a matrix
A ∈ GL(n) and a vector b ∈ Rn such that:

L(x) = Ax + b Vector Spaces

Linear Transformations
The set of all such affine transformations is called the affine and Matrices

group of dimension n, denoted by A(n). Properties of Matrices

Singular Value
L defined above is not a linear map unless b = 0. By Decomposition

introducing homogeneous coordinates to represent x ∈ Rn by

x n+1


1 ∈R , L becomes a linear mapping from
    
n+1 n+1 x A b x
L:R →R ; 7→ .
1 0 1 1

A matrix A0 1b with A ∈ GL(n) and b ∈ Rn is called an affine

matrix. It is an element of GL(n + 1). The affine matrices form

a subgroup of GL(n + 1). Why?

updated May 6, 2019 12/28

 Mathematical
The Orthogonal Group O(n) Background:
Linear Algebra
A matrix A ∈ M(n) is called orthogonal if it preserves the inner Prof. Daniel Cremers

product, i.e.:

hAx, Ay i = hx, y i, ∀x, y ∈ Rn .

The set of all orthogonal matrices forms the orthogonal group Vector Spaces

Linear Transformations
O(n), which is a subgroup of GL(n). For an orthogonal matrix and Matrices
R we have Properties of Matrices

Singular Value

hRx, Ry i = x > R > Ry = x > y , ∀x, y ∈ Rn Decomposition

Therefore we must have R > R = RR > = I, in other words:

O(n) = {R ∈ GL(n) | R > R = I}

The above identity shows that for any orthogonal matrix R, we

have det(R > R) = (det(R))2 = det(I) = 1, such that
det(R) ∈ {±1}.
The subgroup of O(n) with det(R) = +1 is called the special
orthogonal group SO(n). SO(n) = O(n) ∩ SL(n). In particular,
SO(3) is the group of all 3-dimensional rotation matrices.
updated May 6, 2019 13/28
 Mathematical
The Euclidean Group E(n) Background:
Linear Algebra

A Euclidean transformation L from Rn to Rn is defined by an Prof. Daniel Cremers

orthogonal matrix R ∈ O(n) and a vector T ∈ Rn :

L : Rn → Rn ; x 7→ Rx + T .
Vector Spaces
The set of all such transformations is called the Euclidean Linear Transformations
group E(n). It is a subgroup of the affine group A(n). and Matrices

Embedded by homogeneous coordinates, we get: Properties of Matrices

Singular Value
(   ) Decomposition

R T n
E(n) =  R ∈ O(n), T ∈ R .

0 1 

If R ∈ SO(n) (i.e. det(R) = 1), then we have the special

Euclidean group SE(n). In particular, SE(3) represents the
rigid-body motions (dt.: Starrkörpertransformationen) in R3 .

In summary:
SO(n) ⊂ O(n) ⊂ GL(n), SE(n) ⊂ E(n) ⊂ A(n) ⊂ GL(n + 1).

updated May 6, 2019 14/28

 Mathematical
Range, Span, Null Space and Kernel Background:
Linear Algebra
Let A ∈ Rm×n be a matrix defining a linear map from Rn to Rm . Prof. Daniel Cremers
The range or span of A (dt.: Bild) is defined as the subspace of
Rm which can be ‘reached’ by A:

range(A) = {y ∈ Rm  ∃x ∈ Rn : Ax = y }.

Vector Spaces

The range of a matrix A is given by the span of its column Linear Transformations
and Matrices
vectors. Properties of Matrices
The null space or kernel of a matrix A (dt.: Kern) is given by the Singular Value

subset of vectors x ∈ Rn which are mapped to zero: Decomposition

null(A) ≡ ker(A) = {x ∈ Rn  Ax = 0}.



The null space of a matrix A is given by the vectors orthogonal

to its row vectors. Matlab: Z=null(A).
The concepts of range and null space are useful when studying
the solution of linear equations. The system Ax = b will have a
solution x ∈ Rn if and only if b ∈ range(A). Moreover, this
solution will be unique only if ker(A) = {0}. Indeed, if xs is a
solution of Ax = b and xo ∈ ker(A), then xs +xo is also a
solution: A(xs +xo ) = Axs + Axo = b.
updated May 6, 2019 15/28
 Mathematical
Rank of a Matrix Background:
Linear Algebra
The rank of a matrix (dt. Rang) is the dimension of its range:
Prof. Daniel Cremers

rank(A) = dim (range(A)) .

The rank of a matrix A ∈ Rm×n has the following properties:

Vector Spaces
1 rank(A) = n − dim(ker(A)). Linear Transformations
and Matrices

2 0 ≤ rank(A) ≤ min{m, n}. Properties of Matrices

Singular Value
3 rank(A) is equal to the maximum number of linearly Decomposition

independent row (or column) vectors of A.

4 rank(A) is the highest order of a nonzero minor of A,
where a minor of order k is the determinant of a k × k
submatrix of A.
5 Sylvester’s inequality: Let B ∈ Rn×k .
Then AB ∈ Rm×k and
rank(A) + rank(B) − n ≤ rank(AB) ≤ min{rank(A), rank(B)}.
6 For any nonsingular matrices C ∈ Rm×m and D ∈ Rn×n ,
we have: rank(A) = rank(CAD). Matlab: d=rank(A).
updated May 6, 2019 16/28
 Mathematical
Eigenvalues and Eigenvectors Background:
Linear Algebra

Prof. Daniel Cremers

Let A ∈ Cn×n be a complex matrix. A non-zero vector v ∈ Cn is

called a (right) eigenvector of A if:
Vector Spaces
Av = λv , with λ ∈ C. Linear Transformations
and Matrices

λ is called an eigenvalue of A. Similarly v is called a left Properties of Matrices

eigenvector of A, if v > A = λv > for some λ ∈ C. Singular Value

Decomposition
The spectrum σ(A) of a matrix A is the set of all its eigenvalues.
Matlab:

[V,D]=eig(A);

where D is a diagonal matrix containing the eigenvalues and V

is a matrix whose columns are the corresponding eigenvectors,
such that AV=VD.

updated May 6, 2019 17/28

 Mathematical
Properties of Eigenvalues and Eigenvectors Background:
Linear Algebra

Let A ∈ Rn×n be a square matrix. Then: Prof. Daniel Cremers

1 If Av = λv for some λ ∈ R, then there also exists a

left-eigenvector η ∈ Rn : η > A = λη > . Hence
σ(A) = σ(A> ). Vector Spaces

Linear Transformations
and Matrices
2 The eigenvectors of a matrix A associated with different Properties of Matrices
eigenvalues are linearly independent. Singular Value
Decomposition

3 All eigenvalues σ(A) are the roots of the characteristic

polynomial det(λI − A) = 0. Therefore det(A) is equal to
the product of all eigenvalues (some of which may appear
multiple times).

4 If B = PAP −1 for some nonsingular matrix P, then

σ(B) = σ(A).

5 If λ ∈ C is an eigenvalues, then its conjugate λ is also an

eigenvalue. Thus σ(A) = σ(A) for real matrices A.
updated May 6, 2019 18/28
 Mathematical
Symmetric Matrices Background:
Linear Algebra
A matrix S ∈ Rn×n is called symmetric if S > = S. A symmetric
Prof. Daniel Cremers
matrix S is called positive semi-definite (denoted by S ≥ 0 or
S  0) if x > Sx ≥ 0. S is called positive definite (denoted by
S > 0 or S  0) if x > Sx > 0 ∀x 6= 0.
Let S ∈ Rn×n be a real symmetric matrix. Then:
Vector Spaces

1 All eigenvalues of S are real, i.e. σ(S) ⊂ R. Linear Transformations

and Matrices

2 Eigenvectors vi and vj of S corresponding to distinct Properties of Matrices

eigenvalues λi 6= λj are orthogonal. Singular Value

Decomposition

3 There always exist n orthonormal eigenvectors of S which

form a basis of Rn . Let V = (v1 , . . . , vn ) ∈ O(n) be the
orthogonal matrix of these eigenvectors, and
Λ = diag{λ1 , . . . , λn } the diagonal matrix of eigenvalues.
Then we have S = V Λ V > .
4 S is positive (semi-)definite, if all eigenvalues are positive
(nonnegative).
5 Let S be positive semi-definite and λ1 , λn the largest and
smallest eigenvalue. Then λ1 = max|x|=1 hx, Sxi and
λn = min|x|=1 hx, Sxi.
updated May 6, 2019 19/28
 Mathematical
Norms of Matrices Background:
Linear Algebra
There are many ways to define norms on the space of matrices Prof. Daniel Cremers

A ∈ Rm×n . They can be defined based on norms on the

domain or codomain spaces on which A operates. In particular,
the induced 2-norm of a matrix A is defined as
q
Vector Spaces
||A||2 ≡ max |Ax|2 = max hx, A> Axi. Linear Transformations
|x|2 =1 |x|2 =1 and Matrices

Properties of Matrices
Alternatively, one can define the Frobenius norm of A as: Singular Value
Decomposition
sX q
||A||f ≡ aij2 = trace(A> A).
i,j

Note that these norms are in general not the same. Since the
matrix A> A is symmetric and pos.
 semi-definite, we can
diagonalize it as: A> A = V diag σ12 , . . . , σn2 V > with

σ12 ≥ σi2 ≥ 0. This leads to:

q q
||A||2 = σ1 , and ||A||f = trace(A> A) = σ12 + . . . + σn2 .

updated May 6, 2019 20/28

 Mathematical
Skew-symmetric Matrices Background:
Linear Algebra

Prof. Daniel Cremers

A matrix A ∈ Rn×n is called skew-symmetric or anti-symmetric

(dt. schiefsymmetrisch) if A> = −A.
If A is a real skew-symmetric matrix, then:
Vector Spaces
1 All eigenvalues of A are either zero or purely imaginary,
Linear Transformations
i.e. of the form iω with i 2 = −1, ω ∈ R. and Matrices

Properties of Matrices
2 There exists an orthogonal matrix V such that Singular Value
Decomposition
>
A = V ΛV ,

where Λ is a block-diagonal matrix

Λ = diag{A1 , . . . , Am , 0, . . . , 0}, with real skew-symmetric
matrices Ai of the form:
 
0 ai
Ai = ∈ R2×2 , i = 1, . . . , m.
−ai 0

In particular, the rank of any skew-symmetric matrix is even.

updated May 6, 2019 21/28

 Mathematical
Examples of Skew-symmetric Matrices Background:
Linear Algebra

Prof. Daniel Cremers

In Computer Vision, a common skew-symmetric matrix is given

by the hat operator of a vector u ∈ R3 is:
 
0 −u3 u2 Vector Spaces

b =  u3
u 0 −u1  ∈ R3×3 . Linear Transformations
and Matrices
−u2 u1 0 Properties of Matrices

Singular Value
Decomposition
This is a linear operator from the space of vectors R3 to the
space of skew-symmetric matrices in R3×3 .
In particular, the matrix u
b has the property that

bv = u × v ,
u

where × denotes the standard vector cross product in R3 . For

u 6= 0, we have rank(u
b) = 2 and the null space of u
b is spanned
by u, because ubu = u > u
b = 0.

updated May 6, 2019 22/28

 Mathematical
The Singular Value Decomposition (SVD) Background:
Linear Algebra

Prof. Daniel Cremers

In the last slides, we have studied many properties of matrices,

such as rank, range, null space, and induced norms of
Vector Spaces
matrices. Many of these properties can be captured by the
Linear Transformations
so-called singular value decomposition (SVD). and Matrices

Properties of Matrices
SVD can be seen as a generalization of eigenvalues and Singular Value
eigenvectors to non-square matrices. The computation of SVD Decomposition

is numerically well-conditioned. It is very useful for solving

linear-algebraic problems such as matrix inversion, rank
computation, linear least-squares estimation, projections, and
fixed-rank approximations.
In practice, both singular value decomposition and eigenvalue
decompositions are used quite extensively.

updated May 6, 2019 23/28

 Mathematical
Algebraic Derivation of SVD Background:
Linear Algebra
Let A ∈ Rm×n with m ≥ n be a matrix of rank(A) = p. Then
Prof. Daniel Cremers
there exist

• U ∈ Rm×p whose columns are orthonormal

• V ∈ Rn×p whose columns are orthonormal, and Vector Spaces

Linear Transformations
• Σ ∈ Rp×p , Σ = diag{σ1 , . . . , σp }, with σ1 ≥ . . . ≥ σp , and Matrices

Properties of Matrices

such that Singular Value

Decomposition
A = U ΣV > .
Note that this generalizes the eigenvalue decomposition. While
the latter decomposes a symmetric square matrix A with an
orthogonal transformation V as:

A = V Λ V >, with V ∈ O(n), Λ = diag{λ1 , . . . , λn },

SVD allows to decompose an arbitrary (non-square) matrix A

of rank p with two transformations U and V with orthonormal
columns as shown above. Nevertheless, we will see that SVD
is based on the eigenvalue decomposition of symmetric square
matrices. updated May 6, 2019 24/28
 Mathematical
Proof of SVD Decomposition 1 Background:
Linear Algebra
Given a matrix A ∈ Rm×n with m ≥ n and rank(A) = p, the Prof. Daniel Cremers
matrix
A> A ∈ Rn×n
is symmetric and positive semi-definite. Therefore it can be
decomposed with non-negative eigenvalues σ12 ≥ . . . ≥ σn2 ≥ 0 Vector Spaces

with orthonormal eigenvectors v1 , . . . , vn . The σi are called Linear Transformations

and Matrices
singular values. Since Properties of Matrices

Singular Value
> > >
ker(A A) = ker(A) and range(A A) = range(A ), Decomposition

we have span{v1 , . . . , vp } = range(A> ) and

span{vp+1 , . . . , vn } = ker(A). Let

1
ui ≡ Avi ⇔ Avi = σi ui , i = 1, . . . , p
σi
then the ui ∈ Rm are orthonormal:
1 1
hui , uj i = hAvi , Avj i = hvi , A> Avj i = δij .
σi σj σi σj
updated May 6, 2019 25/28
 Mathematical
Proof of SVD Decomposition 2 Background:
Linear Algebra

Complete {ui }pi=1 to a basis {ui }m

Prof. Daniel Cremers
i=1 of Rm . Since Avi = σi ui , we
have
0 ···
 
σ1 0 0
 . .. 0 .. 
 0 . 0  Vector Spaces
 
A (v1 , . . . , vn ) = (u1 , . . . , um )  0 · · · σp
 .. 
,
Linear Transformations
and Matrices
. 0 
  Properties of Matrices
 . .
 .. · · · · · · ..

0  Singular Value
Decomposition
0 ··· ··· 0 0

which is of the form AṼ = Ũ Σ̃, thus

A = Ũ Σ̃ Ṽ > .

Now simply delete all columns of Ũ and the rows of Ṽ > which
are multiplied by zero singular values and we obtain the form
A = U Σ V > , with U ∈ Rm×p and V ∈ Rn×p .

In Matlab: [U,S,V] = svd(A).

updated May 6, 2019 26/28
 Mathematical
A Geometric Interpretation of SVD Background:
Linear Algebra
For A ∈ Rn×n , the singular value decomposition A = U Σ V > is
Prof. Daniel Cremers
such that the columns U = (u1 , . . . , un ) and V = (v1 , . . . , vn )
form orthonormal bases of Rn . If a point x ∈ Rn is mapped to a
point y ∈ Rn by the transformation A, then the coordinates of y
in basis U are related to the coordinates of x in basis V by the
diagonal matrix Σ: each coordinate is merely scaled by the Vector Spaces

Linear Transformations
corresponding singular value: and Matrices

Properties of Matrices
y = Ax = U Σ V > x ⇔ U > y = Σ V > x. Singular Value
Decomposition

The matrix A maps the unit sphere into an ellipsoid with

semi-axes σi ui . To see this, we call α ≡ V > x the coefficients of
the point x in the basis V and those of y in basis UP shall be
called β ≡ U > y . All points of the circle fulfill |x|22 = i αi2 = 1.
The above statement says that βi = σi αi . Thus for the points
on the sphere we have
X X
αi2 = βi2 /σi2 = 1,
i i

which states that the transformed points lie on an ellipsoid

oriented along the axes of the basis U. updated May 6, 2019 27/28
 Mathematical
The Generalized (Moore Penrose) Inverse Background:
Linear Algebra

For certain quadratic matrices one can define an inverse Prof. Daniel Cremers

matrix, if det(A) 6= 0. The set of all invertible matrices forms the

group GL(n). One can also define a (generalized) inverse (also
called pseudo inverse) (dt.: Pseudoinverse) for an arbitrary
(non-quadratic) matrix A ∈ Rm×n . If its SVD is A = U Σ V > the Vector Spaces
pseudo inverse is defined as: Linear Transformations
and Matrices
 −1 
Σ1 0 Properties of Matrices
A† = V Σ† U > , where Σ† = , Singular Value
0 0 n×m Decomposition

where Σ1 is the diagonal matrix of non-zero singular values. In

Matlab: X=pinv(A). In particular, the pseudo inverse can be
employed in a similar fashion as the inverse of quadratic
invertible matrices:

AA† A = A, A† AA† = A† .

The linear system Ax = b with A ∈ Rm×n of rank r ≤ min(m, n)

can have multiple or no solutions. xmin = A† b is among all
2
minimizers of |Ax − b| the one with the smallest norm |x|.
updated May 6, 2019 28/28