EE2211 Introduction to Machine

Learning
mm humpylinalgimport inv matrixrank

Lecture 4
Semester 2
2021/2022

Helen Juan Zhou

[email protected]

Electrical and Computer Engineering Department

National University of Singapore

Acknowledgement:
EE2211 development team
(Kar-Ann Toh, Thomas Yeo, Chen Khong, Helen Zhou, Vincent Tan, Robby Tan and
Haizhou Li)

© Copyright EE, NUS. All Rights Reserved.

 Course Contents
• Introduction and Preliminaries (Haizhou)
– Introduction
– Data Engineering
– Introduction to Probability, Statistics, and Matrix
• Fundamental Machine Learning Algorithms I (Helen)
– Systems of linear equations
– Least squares, Linear regression
– Ridge regression, Polynomial regression
• Fundamental Machine Learning Algorithms II (Thomas)
– Over-fitting, bias/variance trade-off
– Optimization, Gradient descent
– Decision Trees, Random Forest
• Performance and More Algorithms (Haizhou)
– Performance Issues
– K-means Clustering
– Neural Networks

2
© Copyright EE, NUS. All Rights Reserved.
 Systems of Linear Equations

Module II Contents
• Operations on Vectors and Matrices
• Systems of Linear Equations
• Set and Functions
• Derivative and Gradient
• Least Squares, Linear Regression
• Linear Regression with Multiple Outputs
• Linear Regression for Classification
• Ridge Regression
• Polynomial Regression

3
© Copyright EE, NUS. All Rights Reserved.
 Fundamental ML Algorithms:
Linear Regression
References for Lectures 4-6:
Main
• [Book1] Andriy Burkov, “The Hundred-Page Machine Learning Book”, 2019.
(read first, buy later: http://themlbook.com/wiki/doku.php)
• [Book2] Andreas C. Muller and Sarah Guido, “Introduction to Machine
Learning with Python: A Guide for Data Scientists”, O’Reilly Media, Inc., 2017

Supplementary
• [Book3] Jeff Leek, “The Elements of Data Analytic Style: A guide for people
who want to analyze data”, Lean Publishing, 2015.
• [Book4] Stephen Boyd and Lieven Vandenberghe, “Introduction to Applied
Linear Algebra”, Cambridge University Press, 2018 (available online)
http://vmls-book.stanford.edu/
• [Ref 5] Professor Vincent Tan’s notes (chapters 4-6): (useful)
https://vyftan.github.io/papers/ee2211book.pdf

4
© Copyright EE, NUS. All Rights Reserved.
 Recap on Notations, Vectors, Matrices
Scalar Numerical value 15, -3.5
Variable Take scalar values x or a

Vector An ordered list of scalar values x or #

%& 2
Attributes of a vector #= % = 2D
' 3
a ofvectorwithsamedimension
collection

Matrix A rectangular array of numbers q 2 4

*=
arranged in rows and columns 21 −6

Capital Sigma ∑2
01& 30 = 3& + 3' + … + 325& + 32

Capital Pi ∏2
01& 30 = 3& · 3' ·…· 325& · 32
5
© Copyright EE, NUS. All Rights Reserved.
 Operations on Vectors and Matrices

Operations on Vectors: summation and subtraction

3& :& 3 + :&

8+9= 3 + : = &
' ' 3' + :'

3& :& 3& − :&

8−9= 3 − : = 3 −:
' ' ' '

6
© Copyright EE, NUS. All Rights Reserved.
 Operations on Vectors and Matrices

Operations on Vectors: scalar

3& %3&
% 8 = % 3 = %3
' '

&
3& 3
& & ; &
;
8 = ; 3' = &
3
; '

7
© Copyright EE, NUS. All Rights Reserved.
 Operations on Vectors and Matrices
ComputeRankofMatrix

Matrix or Vector Transpose: fromnumpylinalgimportmatrixrank

X uparray
3&
8= 3 , 8< = 3& 3'
'
3&,& 3&,' 3&,> 3&,& 3',& 3>,&
* = 3',& 3',' 3',> , * < = 3&,' 3',' 3>,'
3>,& 3>,' 3>,> 3&,> 3',> 3>,>
importhumpy as up
fromnumpylinalgimportmatrixrank
X nparray I11.4.33104,2 3 2,9
printmatrix rank X 3
print X
print XT Python demo 1
I Transpose

8
© Copyright EE, NUS. All Rights Reserved.
 Operations on Vectors and Matrices
dotproductofvector
at nparray I 1,2 3
Dot Product or Inner Product of Vectors:
b nparray E 2,31
D 8 · 9 = 8<9 c a dot b
:&
ix2
2
Bc a b8
= 3& 3' :
'
= 3& :& + 3':' scalar

Geometric definition: A
8 · 9 = 8 9 cosA 9
8 cosA

where A is the angle between 8 and 9,

and 8 = 8 ⋅ 8 is the Euclidean length of vector 8
2 1
C. E. # = ,F = è # · F = 2*1 + 3 *0 = 2
3 0
9
© Copyright EE, NUS. All Rights Reserved.
 Operations on Vectors and Matrices

cow x
Matrix-Vector Product or

2 3 3 1 C W dot x
I&,& I&,' I&,> 3&
H8 = I',& I',' I',> 3'
3>

I&,&3& + I&,'3' + I&,>3>

= I',&3& + I','3' + I',>3> vector

10
© Copyright EE, NUS. All Rights Reserved.
 Operations on Vectors and Matrices
c x W
or
Vector-Matrix Product
I
2 3 C X.da W
2
I&,& I&,' I&,>
8 < H = 3& 3' I',& I',' I',>
= (3& I&,& + 3' I',& ) (3& I&,' + 3' I',' ) (3& I&,> + 3' I',> )

I 3

11
© Copyright EE, NUS. All Rights Reserved.
 Operations on Vectors and Matrices

Matrix-Matrix Product D C X W
or
3&,& … 3&,M I&,& … I&,P C X daw
*H = ⁞ ⋱ ⁞ ⁞ ⋱ ⁞
32,& … 32,M IM,& … IM,P

(3&,& I&,& + ⋯ + 3&,M IM,& ) … (3&,& I&,P + ⋯ + 3&,M IM,P )

= ⁞ ⋱ ⁞
(32,& I&,& + ⋯ + 32,M IM,& ) … (32,& I&,P + ⋯ + 32,M IM,P )

∑M01& 3&,0 I0,& … ∑M01& 3&,0 I0,P

= ⁞ ⋱ ⁞
∑M01& 32,0 I0,& … M
∑01& 32,0 I0,P

If X is m x d and W is d x h, then the outcome is a m x h matrix

12
© Copyright EE, NUS. All Rights Reserved.
 Operations on Vectors and Matrices

Matrix inverse
Ows columns are
linearlyindependent
n
determinantsof A A to y p
Definition: gonon squarematrix
y
A d-by-d square matrix A is invertible (also nonsingular) A isinvertib
if there exists a d-by-d square matrix B such that
RS = SR = T (identity matrix)
B is aninverseof A
1 0…0 0 B At
0 1 0 0
T= ⁞ ⋱ ⁞ d-by-d dimension Squarematrix A is invertible if
0 0 1 0
0 0…0 1 A exists
Ref: https://en.wikipedia.org/wiki/Invertible_matrix

13
© Copyright EE, NUS. All Rights Reserved.
 Operations on Vectors and Matrices
Matrix inverse computation
5&
1
R = adj(R)
det R
• det R is the determinant of R scalar
• adj(R) is the adjugate or adjoint of R matrix

Determinant computation
Example: 2x2 matrix
% Z
R=
[ \

% Z
det R = |R| = = %\ − Z[
[ \
Ref: https://en.wikipedia.org/wiki/Invertible_matrix

14
© Copyright EE, NUS. All Rights Reserved.
 Operations on Vectors and Matrices
• adj(R) is the adjugate or adjoint of R
• adj(R) is the transpose of the cofactor matrix ^ of R à adj(A)= CT
• Minor of an element in a matrix R is defined as the determinant
obtained by deleting the row and column in which that element lies
%11 %12 %13 % %23
A= %21 %22 %23 Minor of a12 is f12 = 21
%31 %33
%31 %32 %33
ftp.Immimmmi deAin anmn anmntaisM
• The `, a entry of the cofactor matrix ^ is the minor of `, a
element times a sign factor +
Cofactor Cij = −1 ` a f`a

• The determinant of R can also be defined by minors as

det(A)= ∑kj1& = %`aCij = −1 ` + a% f
Mmm `a `a

I AijCi
Ref: https://en.wikipedia.org/wiki/Invertible_matrix

15
© Copyright EE, NUS. All Rights Reserved.
 Operations on Vectors and Matrices
%21 %23
Minor of a12 is f12 = adj(A)= CT
%31 %33
Cofactor Cij = −1 ` + af det(A)= ∑kj1& −1 ` + a%
`a `af`a

% Z
• E.g. R =
[ \
\ −[
^= def A ad t bl c ad be
−Z %
< \ −Z
• adj R = ^ = det R = |R| = %\ − Z[
−[ %
& & \ −Z
R5& = adj(R) =
lmn R ;M5op −[ %
Ref: https://en.wikipedia.org/wiki/Invertible_matrix

16
© Copyright EE, NUS. All Rights Reserved.
 Operations on Vectors and Matrices

Determinant computation det(A)= ∑kj1& −1 ` + a%

`af`a
Example: 3x3 matrix, use the first row (i = 1)
just onerow

U
importnumpyasup
fromnumpylinalgimportinv
x uparray El43,1047,13 4
y uparray I30.5.4
= a(ei - fh) – b(di - fg) + c(dh - eg) print inv x

Python demo 2 Ref: https://en.wikipedia.org/wiki/Determinant

17
© Copyright EE, NUS. All Rights Reserved.
 I
Operations on Vectors and Matrices

won
%'' %'>
The minor of %&& = %
>' %>>

Ref: https://en.wikipedia.org/wiki/Determinant

18
© Copyright EE, NUS. All Rights Reserved.
 Operations on Vectors and Matrices
Example
1 2 3 diet A 24 215 31 4
Find the cofactor matrix of R given that R = 0 4 5 .
1 0 6
Solution:
4 5 0 5 0 4
%&& ⇒ = 24, %&' ⇒ − = 5, %&> ⇒ = −4,
0 6 1 6 1 0
2 3 1 3 1 2
%'& ⇒ − = −12, %'' ⇒ = 3, %'> ⇒ − = 2,
0 6 1 6 1 0
2 3 1 3 1 2
%>& ⇒ = −2, %>' ⇒ − = −5, %>> ⇒ = 4,
4 5 0 5 0 4
24 5 −4
The cofactor matrix C is thus −12 3 2 .
−2 − 5 4

Ref: https://www.mathwords.com/c/cofactor_matrix.htm

23
© Copyright EE, NUS. All Rights Reserved.
 Systems of Linear Equations

Module II Contents
• Operations on Vectors and Matrices
• Systems of Linear Equations
• Set and Functions
• Derivative and Gradient
• Least Squares, Linear Regression
• Linear Regression with Multiple Outputs
• Linear Regression for Classification
• Ridge Regression
• Polynomial Regression

24
© Copyright EE, NUS. All Rights Reserved.
 Systems of Linear Equations

• Consider a system of m linear equations with d

variables or unknowns I1, … , I\ :

3&,& I& + 3&,' I' + ⋯ + 3&,M IM = :&

3',& I& + 3',' I' + ⋯ + 3',M IM = :'
⁞
32,& I& + 32,' I' + ⋯ + 32,M IM = :2 .

Ref: [Book4] Stephen Boyd and Lieven Vandenberghe, “Introduction to

Applied Linear Algebra”, Cambridge University Press, 2018 (Chp8.3)

25
© Copyright EE, NUS. All Rights Reserved.
 Systems of Linear Equations
These equations can be written compactly in matrix-vector
notation:
*s = 9
Where Supervised ML

datamatrix 3&,& 3&,' … 3&,M unknown I& targetvector :&

*= ⁞ ⁞ ⋱ ⁞ , s= ⁞ , 9= ⁞ .
32,& 32,' … 32,M MLProblem IM :2

Note:
inputdata
• The data matrix * ∈ x2×M and the target vector 9 ∈ x2 are given
• The unknown vector of parameters s ∈ xM is to be learnt

Ref: [Book4] Stephen Boyd and Lieven Vandenberghe, “Introduction to

Applied Linear Algebra”, Cambridge University Press, 2018 (Chp8.3)

26
© Copyright EE, NUS. All Rights Reserved.
 Xis invertible it det X 0
Systems of Linear Equations
A set of linear equations can have no solution, one
solution, or multiple solutions:
*s = 9
Where
3&,& 3&,' … 3&,M I& :&
*= ⁞ ⁞ ⋱ ⁞ , s= ⁞ , 9= ⁞ .
32,& 32,' … 32,M IM :2

* is Square Even-determined m=d Equal number of equations

and unknowns
* is Tall Over-determined m>d More number of equations
data features than unknowns leftinverse
* is Wide Under-determined m<d Fewer number of equations
datafeatures than unknowns Rightinverse
Ref: [Book4] Stephen Boyd and Lieven Vandenberghe, “Introduction to
Applied Linear Algebra”, (Chp8.3 & 11) & [Ref 5] Tan’s notes, (Chp 4)

27
© Copyright EE, NUS. All Rights Reserved.
 Systems of Linear Equations
*s = 9, * ∈ x2×M , s ∈ xM×&, 9 ∈ x2×&
1. Square or even-determined system: ‚ = ƒ
- Equal number of equations and unknowns, i.e., * ∈ xM×M
- One unique solution if * is invertible or all rows/columns of * are
linearly independent
- If all rows or columns of * are linearly independent, then * is
invertible.

Solution:
If * is invertible (or * 5& * = T ), then pre-multiply both sides by * 5&
* 5& * s = * 5& 9
⇒ Ñ = * 5& 9
s
(Note: we use a hat on top of s to indicate that it is a specific point in the space of s)

Ref: [Book4] Stephen Boyd and Lieven Vandenberghe, “Introduction to

Applied Linear Algebra”, Cambridge University Press, 2018 (Chp11)

28
© Copyright EE, NUS. All Rights Reserved.
 Systems of Linear Equations
Example 1 I& + I' = 4 (1) Two unknowns
I& − 2I' = 1 (2) Two equations

* s 9 importhumpyasup
fromnumpy Iinal import in
1 1 I& 4 import g as
plot pit
= matplotsibDy

1 − 2 I' 1 X nparray 11 I E 27 1
y i nparray14 1 s
Ñ
Ñ = * 5&9
s W in x y
1 1 5& 4
=
1 −2 1

5& −2 −1 4 3
= =
> −1 1 1 1 Python demo 3
1 \ −Z
R5& = adj(R) adj R = ^< = det R = %\ − Z[
det R −[ %
29
© Copyright EE, NUS. All Rights Reserved.
 Systems of Linear Equations
*s = 9, * ∈ x2×M , s ∈ xM×&, 9 ∈ x2×&
2. Over-determined system: ‚ > ƒ
– More equations than unknowns
– * is non-square (tall) and hence not invertible
– Has no exact solution in general *
– An approximated solution is available using the left inverse
If the left-inverse of * exists such that * Ü* = T, then pre-multiply both
sides by * Ü results in
* Ü* s = * Ü9
⇒s Ñ = * Ü9
Definition:
A matrix B that satisfies ‡ƒ ˆ ‚‰‚ ˆ ƒ = T is called a left-inverse of R.
The left-inverse of *: * Ü= (* < *)5‹ * < given * < * is invertible.
Note: * exception: when rank(*) = rank([*,9]), there is a solution. approximatesolution rankx rankXY
exists rankX rank
solution
Ref: [Book4] Stephen Boyd and Lieven Vandenberghe, “Introduction to Applied Linear Algebra”, Cambridge University Press, 2018 Xy
(Chp11.1-11.2, 11.5)

30
© Copyright EE, NUS. All Rights Reserved.
 makeathup.vstack importnumpyasnp
X.Ty XY from
Systems of Linear Equations numpy.linalgimportin x
np.arrayleei.is i t
1,0
Y np.arraylliio.rs
wainuxet x X Y
Example 2 I& + I' = 1 (1) Two unknowns
I& − I' = 0 (2) Three equations
* s 9 I& = 2 (3)
1 1 I 1
&
1 −1 I = 0
'
1 0 2
s
Ñ
No exact solution
Approximated solution

Ñ = * Ü 9 = (* < *)5‹ * < 9

s
3 0 5&
1 1 1 1 1
= 0=
0 2 1 −1 0 2 0.5 Python demo 4
* < * is invertible detX'X 0
31
© Copyright EE, NUS. All Rights Reserved.
 Systems of Linear Equations
*s = 9, * ∈ x2×M , s ∈ xM×&, 9 ∈ x2×&
3. Under-determined system: ‚ < ƒ
– More unknowns than equations
– Infinite number of solutions in general *

If the right-inverse of * exists such that ** Ü = T, then the \-vector

s = * Ü9 (one of the infinite cases) satisfies the equation *s = 9, i.e.,
*s = 9 ⇒ ** Ü9 = 9
⇒ T9 = 9
Definition:
A matrix B that satisfies R ‚ ˆ ƒSƒ ˆ ‚ = T is called a right-inverse of R.
The right-inverse of *: * Ü= * < (** < )5& given ** < is invertible.
If * is right−invertible, we can find a unique constrained solution.

Note: * exception: no solution if the system is inconsistent rank(*) < rank([*,9]

bothXXT XTX arenotinvertible
Ref: [Book4] Stephen Boyd and Lieven Vandenberghe, “Introduction to Applied Linear Algebra”, Cambridge University Press, 2018 (Chp11.1-11.2, 11.5)

32
© Copyright EE, NUS. All Rights Reserved.
 Systems of Linear Equations
3. Under-determined system: ‚ < ƒ
Derivation:
*s = 9, * ∈ x2×M , s ∈ xM×&, 9 ∈ x2×&

A unique solution is yet possible by constraining the search using

s = *< #

If ** < is invertible, let s = * < #, then

** < # = 9
⇒ #é = (** < )5& 9
⇒sÑ = * < #é = * < (** < )5& 9

*Ü
right-inverse

33
© Copyright EE, NUS. All Rights Reserved.
 Systems of Linear Equations
Example 3 I& + 2I' + 3I> = 2 (1) Three unknowns
I& − 2I' + 3I> = 1 (2) Two equations
* s 9
I
1 2 3 I& 2
' =
1 −2 3 I> 1

Infinitely many solutions along

the intersection line
Here ** < is invertible

Ñ = * < (** < )5&9

s
5&
1 1 14 6 0.15
2
= 2 −2 = 0.25 Constrained solution
1
3 3 6 14 0.45

34
© Copyright EE, NUS. All Rights Reserved.
 Systems of Linear Equations

Example 4 I& + 2I' + 3I> = 2 (1) Three unknowns

3I& + 6I' + 9I> = 1 (2) Two equations

* s 9
I
1 2 3 I& 2
' =
3 6 9 I> 1

Both ** < and * < * are not invertible!

There is no solution for the system
rank X rank X y

35
© Copyright EE, NUS. All Rights Reserved.
 Systems of Linear Equations

Module II Contents
• Operations on Vectors and Matrices
• Systems of Linear Equations
• Set and Functions
• Derivative and Gradient
• Least Squares, Linear Regression
• Linear Regression with Multiple Outputs
• Linear Regression for Classification
• Ridge Regression
• Polynomial Regression

36
© Copyright EE, NUS. All Rights Reserved.
 Notations: Set
• A set is an unordered collection of unique elements
– Denoted as a calligraphic capital character e.g., •, x, ‘ etc
– When an element 3 belongs to a set ’, we write 3 ∈ •
• A set of numbers can be finite - include a fixed amount of values
– Denoted using accolades, e.g. {1, 3, 18, 23, 235} or {3& , 3' , 3> , 3ì , . . . , 3M }
• A set can be infinite and include all values in some interval
– If a set of real numbers includes all values between a and b, including a and
b, it is denoted using square brackets as [a, b]
– If the set does not include the values a and b, it is denoted using
parentheses as (a, b)
• Examples:
– The special set denoted by x includes all real numbers from minus infinity
to plus infinity
– The set [0, 1] includes values like 0, 0.0001, 0.25, 0.9995, and 1.0

Ref: [Book1] Andriy Burkov, “The Hundred-Page Machine Learning Book”, 2019 (p4 of chp2).

37
© Copyright EE, NUS. All Rights Reserved.
 Notations: Set operations

• Intersection of two sets:

•> ← •& ∩ •'
Example: {1,3,5,8} ∩ {1,8,4} = {1,8}

• Union of two sets:

•> ← •& ∪ •'
Example: {1,3,5,8} ∪ {1,8,4} = {1,3,4,5,8}

Ref: [Book1] Andriy Burkov, “The Hundred-Page Machine Learning Book”, 2019 (p4 of chp2).

38
© Copyright EE, NUS. All Rights Reserved.
 Functions
• A function is a relation that associates each element 3 of a set —,
the domain of the function, to a single element : of another set ˜,
the codomain of the function
• If the function is called f, this relation is denoted : = ™(3)
– The element 3 is the argument or input of the function
– : is the value of the function or the output
• The symbol used for representing the input is the variable of the
function
– ™(3) ™ is a function of the variable 3; ™(3, I) ™ is a function of the variable 3 and w

— ˜
1
1 2 Range
2 3 (or Image)
3 4 {3,4,5,6}
4 5
6
{1,2,3,4} domain codomain {1,2,3,4,5,6}
Ref: [Book1] Andriy Burkov, “The Hundred-Page Machine Learning Book”, 2019 (p6 of chp2). 39
© Copyright EE, NUS. All Rights Reserved.
 Functions

• A scalar function can have vector argument

– E.g. : = ™(8) = 3& + 3' +23>
• A vector function, denoted as 9 = š(8) is a function
that returns a vector 9
– Input argument can be a vector 9 = š(8) or a scalar 9 = š(3)
:& −3&
– E.g. : = 3 vectorinputs
' '
:& −23&
– E.g. : = scalarinputs
' 33&

Ref: [Book1] Andriy Burkov, “The Hundred-Page Machine Learning Book”, 2019 (p7 of chp2).

40
© Copyright EE, NUS. All Rights Reserved.
 Functions inputvectors
real vectors

g r
real scalar
• The notation ™: xM → x means that ™ is a function that
number

maps real d-vectors to real numbers

– i.e., ™ is a scalar-valued function of d-vectors
• If 8 is a d-vector argument, then ™ 8 denotes the value
of the function ™ at 8
– i.e., ™ 8 = ™ 3& , 3' , … , 3M , 8 ∈ xM , ™ 8 ∈ x

• Example: we can define a function ™: xì → x by

™ 8 = 3& + 3' − 3ì'

at I 1 I

Ref: [Book4] Stephen Boyd and Lieven Vandenberghe, “Introduction to Applied Linear Algebra”, 2018 (Ch 2, p29)

41
© Copyright EE, NUS. All Rights Reserved.
 in ML
Functions a is treated as 5
x isdatafunction X
y
The inner product function
• Suppose œ is a d-vector. We can define a scalar valued function ™ of d-
vectors, given by
™ 8 = œ< 8 = %& 3& + %' 3' + ⋯ %M 3M (1)
for any d-vector 8
• The inner product of its d-vector argument 8 with some (fixed) d-vector œ
• We can also think of ™ as forming a weighted sum of the elements of 8;
the elements of œ give the weights œ

A
8
œ cosA
Ref: [Book4] Stephen Boyd and Lieven Vandenberghe, “Introduction to Applied Linear Algebra”, Cambridge University Press, 2018 (p30)

42
© Copyright EE, NUS. All Rights Reserved.
 Functions

Linear Functions

A function ™: xM → x is linear if it satisfies the following two properties:

• Homogeneity
• For any d-vector 8 and any scalar •, ™ •8 = •™ 8
• Scaling the (vector) argument is the same as scaling the
function value

• Additivity
• For any d-vectors 8 and 9, ™ 8 + 9 = ™ 8 + ™ 9
• Adding (vector) arguments is the same as adding the function
values

Ref: [Book4] Stephen Boyd and Lieven Vandenberghe, “Introduction to Applied Linear Algebra”, Cambridge University Press, 2018 (p31)

43
© Copyright EE, NUS. All Rights Reserved.
 Functions

Linear Functions
Superposition and linearity
• The inner product function ™ 8 = œ< 8 defined in equation (1)
(slide 9) satisfies the property
™ •8 + ž9 = œ< •8 + ž9
= œ< •8) + œ< (ž9
= • œ< 8) + ž(œ< 9
= •™ 8) + ž™(9
for all d-vectors 8, 9, and all scalars •, ž.

• This property is called superposition, which consists of

homogeneity and additivity
• A function that satisfies the superposition property is called linear

Ref: [Book4] Stephen Boyd and Lieven Vandenberghe, “Introduction to Applied Linear Algebra”, Cambridge University Press, 2018 (p30)

44
© Copyright EE, NUS. All Rights Reserved.
 Functions

Linear Functions

• If a function ™ is linear, superposition extends to linear

combinations of any number of vectors:
™ •& 8& + ⋯ + •k 8k = •& ™ 8& ) + ⋯ + •k ™(8k
for any d vectors 8& + ⋯ + 8k , and any scalars
•& + ⋯ + •k .

Ref: [Book4] Stephen Boyd and Lieven Vandenberghe, “Introduction to Applied Linear Algebra”, Cambridge University Press, 2018 (p30)

45
© Copyright EE, NUS. All Rights Reserved.
 Functions

Linear and Affine Functions

A linear function plus a constant is called an affine function

A linear function ™: xM → x is affine if and only if it can be

expressed as ™ 8 = œ< 8 + Z for some d-vector œ and scalar Z,
which is called the offset (or bias)

Example:
™ 8 = 2.3 − 23& + 1.33' − 3>

This function is affine, with Z = 2.3, œ< = [−2, 1.3, −1].

Ref: [Book4] Stephen Boyd and Lieven Vandenberghe, “Introduction to Applied Linear Algebra”, Cambridge University Press, 2018 (p32)

46
© Copyright EE, NUS. All Rights Reserved.
 Functions

Ref: [Book4] Stephen Boyd and Lieven Vandenberghe, “Introduction to Applied Linear Algebra”, Cambridge University Press, 2018 (p33)

47
© Copyright EE, NUS. All Rights Reserved.
 Summary
• Operations on Vectors and Matrices Assignment 1 (week 6)
• Dot-product, matrix inverse Tutorial 4
• Systems of Linear Equations *s = 9
• Matrix-vector notation, linear dependency, invertible
• Even-, over-, under-determined linear systems
• Set and Functions
* is Even- m = d One unique solution in general Ñ = * 5& 9
s
Square determined
* is Over- m > d No exact solution in general; Ñ = (* < *)5‹ * < 9
s
Tall determined An approximated solution Left-inverse
* is Under- m < d Infinite number of solutions in general; sÑ = * < (** < )5& 9
Wide determined Unique constrained solution Right-inverse

• Scalar and vector functions

• Inner product function python package numpy
Inverse: numpy.linalg.inv(X)
• Linear and affine functions
Transpose: X.T

48
© Copyright EE, NUS. All Rights Reserved.