DDA3020 Lecture 02 Linear Algebra
DDA3020 Lecture 02 Linear Algebra
Jicong Fan
School of Data Science, CUHK-SZ
2 −2 1
Illustrations of three two-dimensional vectors, a = ,b = ,c = are
3 5 0
given in Figure 1 below.
(1)
a 2 a 2
a = (2) = , or more commonly, a = 1 =
a 3 a2 3
Note:
The notation x(j) should not be confused with the power operator, such as the 2
in x2 (squared) or 3 in x3 (cubed).
If we want to apply a power operator, say square, to an indexed attribute of a
vector, we write like this: (x(j) )2 .
Operations on Vectors:
x1 y1 x1 + y1
x+y = + =
x2 y2 x2 + y2
x1 y1 x1 − y1
x−y = − =
x2 y2 x2 − y2
x ax1
ax = a 1 =
x2 ax2
1
1 1 x1 x
x= = a1 1
a a x2 a x2
x · y = x> y
y1
= x1 x2
y2
= x1 y1 + x2 y2
Matrix-Vector Product
x1,1 x1,2 x1,3 w1
Xw = x2,1 x2,2 x2,3 w2
x3,1 x3,2 x3,3 w3
x1,1 w1 + x1,2 w2 + x1,3 w3
= x2,1 w1 + x2,2 w2 + x2,3 w3
x3,1 w1 + x3,2 w2 + x3,3 w3
Matrix-Vector Product
w1,1 w1,2 w1,3
x> W = x1 x2 x3 w2,1
w2,2 w2,3
w3,1 w3,2 w3,3
= [(w1,1 x1 + w2,1 x2 + w3,1 x3 ) (w1,2 x1 + w2,2 x2 + w3,2 x3 ) ]
(w1,3 x1 + w2,3 x2 + w3,3 x3 )
Matrix-Matrix Product
x1,1 . . . x1,d w1,1 . . . w1,h
XW = ... .. .. .. .. ..
. . . . .
xm,1 . . . xm,d wd,1 . . . wd,h
(x1,1 w1,1 + . . . + x1,d wd,1 ) . . . (x1,1 w1,h + . . . + x1,d wd,h )
=
.. .. ..
. . .
(xm,1 w1,1 + . . . + xm,d wd,1 ) . . . (xm,1 w1,h + . . . + xm,d wd,h )
Pd Pd
i=1 x1,i wi,1 ... i=1 x1,i wi,h
=
.. .. ..
. . .
Pd Pd
x
i=1 m,i i,1w . . . x
i=1 m,i i,h w
Matrix Inverse
Definition:
A d × d square matrix A is called invertible (also nonsingular) if there
exists a d × d square matrix B such that AB = BA = I (Identity matrix)
given by
1 0 ... 0 0
0 1 . . . 0 0
I = ... . . . . . . . . . ... of d by d dimension
0 0 . . . 1 0
0 0 ... 0 1
Determinant computation
Example: 2 × 2 matrix
a b
det(A) = |A| = = ad − bc (1)
c d
Determinant computation
Example: 3 × 3 matrix
a b c
|A| = d e f = a
e f − b d
f + c d
e
g h i h i g i g h
e f d f d e
= a − b
+ c
h i g i g h
= a(ei − f h) − b(di − f g) + c(dh − eg)
β1 x 1 + . . . + βm x m = 0
Xw = y,
where
x1,1 ... x1,d w1 y1
.. .. ,
X= . .. w = ... , y = ... .
. .
xm,1 ... xm,d wd ym
Note: X is of m × d dimension.
If X is invertible (or X−1 X = I), then pre-multiply both sides by X−1 , we have
X−1 Xw = X−1 y
⇒ w = X−1 y
1 1 w1 4
=
1 −2 w2 1
X w y
w = X−1 y
1 1 −1 4 3
= =
1 −2 1 1
X† Xw = X† y
⇒ w = X† y
Definition: a matrix B that satisfies BA=I (identity matrix) is called a
left-inverse of A. (Note: A is m-by-d and B is d-by-m.
Note: The left-inverse can be computed as X† = (X> X)−1 X> given X> X
is invertible.
1 1 1
w1
1 −1 = 0
w2
1 0 2
X w y
XX† y = y
⇒y=y
Definition: a matrix B that satisfies AB=I (identity matrix) is called a
right-inverse of A. (Note: A is m-by-d and B is d-by-m).
Note: The right-inverse can be computed as X† = X> (XX> )−1 given
XX> is invertible.
Derivation:
Under-determined system: m < d in Xw = y, X ∈ Rm×d
(i.e., there are more unknowns than equations ⇒ infinite number of solu-
tions ⇒ but a unique solution is yet possible by constraining the search
using w = X> a !)
If XX> is invertible, let w = X> a, then
XX> a = y
⇒ a = (XX> )−1 y
w = X> a = X> (XX> )−1 y
| {z }
X†
w1
1 2 3 w2 2
=
1 −2 3 1
w3
X w y
> −1
w= X> (XX
) y Here XX
>
is invertible.
1 1 0.15
14 6 −1 2
= 2 −2 = 0.25 (Constrained solution)
6 14 1
3 3 0.45
Jicong Fan School of Data Science, CUHK-SZ
DDA3020 Machine Learning Lecture 02 Linear Algebra
September 14, 2022 36 / 37
Systems of linear equations