R Reevviieew W:: L Liinneeaarr A Allggeebbrraa
R Reevviieew W:: L Liinneeaarr A Allggeebbrraa
1.1.1 Numbers
Bou rkaki: b u ild all of math ematics from set th eory.
(1) N = {0, 1, 2, ....} : Natu ral nu mb ers
(2) Z = {0, 1, −1, 2, −2 , ....} : I ntegers
(3) Q = { ab | a, b ∈ Z, b = 0} : Ration als
(4) R = {dr · · · d1 d0 .d−1 · · · | di ∈ {0 , 1, ..., 9}}: Reals
√
Exercise: Why is 2 irration al?
√
Proof. Supp ose 2 is rational. Th at mean s it can b e written as th e ratio of two integers a and b
√ a
2 = (1)
b
wh ere we may assu me th at a and b have no common factors . S qu arin g in equ ation (1) on b oth sid es
gives
a2
2 = 2 (2)
b
wh ich imp lies
a2 = 2b 2 (3)
Thu s a is even . Th e on ly way th is can b e tru e is th at a itself is even . Bu t th en a2 is actu ally d ivisib le by 4.
2
Hen ce b2 an d th erefore b mu st b e even . √
S o a and b are b oth even wh ich is a contrad iction to th e assu mp tion
th at th ey h ave n o common factors. S o 2 is irration al.
1.2 Function
D efinition. Let S and T b e sets. A function or map f : S → T is given by associatin g each element s ∈ S
to an element f (s ) ∈ T . I n ord er to ch eck th at a fu n ction is well defined, on e mu st ch eck th at
(1) If s ∈ S th en f (s ) ∈ T .
(2) If s1 = s2 th en f (s 1 ) = f (s 2 ).
A fu n ction f : S → T map s every el ement of S to an unique el ement of T .
(Th e examp les in th e lectu re2 P.11/42: (b ) an d (c) are n ot a fu n ction .)
D efinition. A sequence is a fu n ction f : N → X . Write xn = f (n ) for all n ∈ N.
1
Figure 1 : Th e examp les in th e lectu re2
2 Topology
D efinition. Given a set X , a topology T on X is a collection of su b sets of X su ch th at
(1) ∅ ∈ T , X ∈ T
(2) arb itrary u n ion of elements of T is in T
(3) fi n ite intersection of elements of T is in T
D efinition. Topological space is a set X togeth er with a collection of su b sets of X satisfyin g th e ab ove
axioms.
D efinition. Th e comp lement set of a op en set is referred to as closed sets .
Examples:
X = {1 , 2, 3, 4} an d collection τ = {{∅}, {1, 2, 3, 4}} of on ly th e two su b sets of X requ ired by th e axioms
form a top ology.
X = {1, 2, 3, 4} an d collection τ = {{∅}, {2}, {1, 2} , {2 , 3}, {1, 2, 3}, {1, 2, 3, 4}} of six su b sets of X form
an oth er top ology.
D efinition. Let S b e a su b set of a top ological sp ace X . A p oint x in X is a limit point of S i f every
n eighb orh ood of x contain s at least on e p oint of S d ifferent from x itself.
2
2.1 Metric Spaces
D efinition. Metric Spaces : A metric sp ace is a set X with a d istan ce fu n ction d : X × X → R su ch th at
(1) non- negativity: ∀x , y ∈ X, d(x , y) ≥ 0
(2) definiteness: ∀x , y ∈ X, d(x , y) = 0 ⇔ x = y
(3) symme try: ∀x , y ∈ X, d(x , y) = d(y , x)
(4) trian g l e in eq ual ity: ∀x , y ∈ X, x + y ≤ x + y
D efinition. Open Ball: Let M = (X , d) b e a metric sp ace, x ∈ X , and ∈ R>0 , th e op en -b all of x in M
is d efi n ed as: B (x ) = {y ∈ X | d(x , y) ≤ }.
2.3 Compactness
D efinition. Compactness: A top ological sp ace is comp act if every op en cover h as a fi n ite su b co ver.
Exp licitly, th is mean s th at for every arb itrary collection {Uα }α ∈I of op en su b sets of X su ch th at X = Uα ,
U . (some typ o existin g in th e slice)
α ∈I
th ere is a fi n ite su b set of J of A su ch th at X = i
i∈ J
For metric sp aces, closed an d b ou n d ed n ess imp lies comp actn ess.
3 Vector Spaces
A vector space over reals is a n on -emp ty set X with p rop erties b elow, ad d ition :
(1) closu re: Performan ce of ad d ition on memb ers of th e set always p ro d u ces a memb er of th e same set.
(2) commu tative law: Ch an gin g th e ord er of th e op eran d s d oes n ot ch an ge th e resu lt.
(3) associative law: With in an exp ression contain in g two or more occu rren ces in a row of th e same associa-
tive op erator, th e ord er in wh ich th e op eration s are p erformed d oes n ot matter as lon g as th e sequ en ce
of th e op eran d s is n ot ch an ged .
(4) ad d itive id entity: Ad d a nu mb er by 0, th e resu lt is th at origin al nu mb er.
(5) ad d itive inverse: Wh en a nu mb er is ad d ed by its op p osite, th e resu lt is 0.
3
(4) d istrib u tive law I I : Given a set S , two b in ary op erators: ad d ition + an d mu ltip lication × on S , and
any elements x, y, and z of S , (x + y ) × z = x × z + y × z.
(5) mu ltip licative id entity: Mu ltip ly a nu mb er by 1, th e resu lt, or p rod u ct, is th at origin al nu mb er.
x p = (
d |x |p ) , ∀x ∈ Rd
1
p
i
i= 1
4
√
x ∞ ≤ x 2 ≤ dx ∞
Proof.
D efinition. Cauchy Sequence: Given a metric sp ace (X, d), a sequ en ce is Cau chy, if for every p ositive
real nu mb er > 0 th ere is a p ositive integer N su ch th at for all p ositive integers m, n > N , th e d istan ce
d(x m
, xn ) < .
D efinition. Hilbert Space: A vector sp ace with an in n er p rod u ct is called an in n er-p rod u ct sp ace. A
Hilb ert sp ace is a real or comp lex in n er p rod u ct sp ace th at is also a complete metric sp ace with resp ect
to th e d istan ce fu n ction in d u ced by th e in n er p rod u ct (A comp lete metric sp ace is a metric sp ace in wh ich
every Cau chy sequ en ce is convergent.)
5
4 Subspace
D efinition. Vector Subpace: A su b sp ace of X is a su b set W ⊂ X th at forms a vector sp ace.
D efinition. Span: Given a su b set U ⊆ X , th e sp an of U is d efi n ed to b e th e set of all fi n ite lin ear
comb in ation s of elements of U :
span(U
) = {
n c x |n ∈ N, x ∈ X, c ∈ R}
k k k k
k =1
4.1 Basis
A su b set U ⊆ X , is lin early in d ep en d ent if wh en ever x1 , ..., xn is a fi n ite collection of vectors form U and
c1 , ..., cn ∈ R,
n c x = 0 implies c = 0, for all k
k k k
k =1
5 Matrices
Rank of a matrix X ∈ Rk×d is th e size of th e largest set of lin early in d ep en d ent colu mn s or rows.
Prop erties:
(1) Th e ran k of an m × n matrix is a n on n egative integer an d can n ot b e greater th an eith er m or n.
(2) I f A is a squ are matrix (i.e., m = n), A is invertib le if an d on ly if A h as ran k n (i.e., A h as fu ll ran k).
(3) Assu me A is any m-by-n matrix an d B is any n -by-k matrix, th en
ran k(A
B ) ≤ min (ran k(A), ran k(B
))
(4) If B h as fu ll row ran k, th en
ran k(A
B ) = ran k(A
)
6
5.1 Matrix Norms
Operator Norm :
X p = sup Xwwp p
x=
0
I f th ere exist two set of vector u, v (wh ich are lin early in d ep en d ent) an d sin gu lar valu e σ wh ich satisfi es
X v = σ u , X T u = σ v
th en σ is a d egen erate sin gu lar valu e. I f X h as d egen erate sin gu lar valu es, th en its sin gu lar valu e d ecomp o-
sition is n ot u n iqu e; oth erwise if all sin gu lar valu es of X are n on -d egen erate an d n on -zero, th en its sin gu lar
valu e d ecomp osition is u n iqu e u p to mu ltip lication of a p air of corresp on d in g colu mn s of U an d V by (-1).
S in ce U an d V are orth ogon al matrices, we h ave
X X T = U ΣV T V ΣU T = U Σ2 U T , X T X = V Σ2 V T
S o given a S VD d ecomp osition of X, we cou ld ob tain th e eigenvalu e d ecomp osition s of matrix X X T and
X T X . Th e colu mn s of U are eigenvectors of X X T ; th e colu mn s of V are eigenvectors of X T X .
Wh en X is a p ositive semi-d efi n ite matrix, th e eigenvalu e d ecomp osition is also a sin gu lar valu e d ecomp osi-
tion.
With sin gu lar valu e d ecomp osition , we can recon stru ct X as follows:
r
X = uk σk vkT
k =1
References
[1] Wikip ed ia contrib u tors.“Top ological sp ace.” Wikip ed ia, Th e Free En cyclop ed ia. Wikip ed ia, Th e Free
En cyclop ed ia, 4 S ep . 2015. Web . 10 S ep . 2015.
[2] Ben gio, Yoshu a an d Cou rville, Aaron an d Vin cent, Pascal, “Rep resentation learn in g: A review an d n ew
p ersp ectives”,C
amb ridg e Un iv e rsity Pre ss, 2006.