Foundations of Machine Learning

DSA 5102X • Lecture 2

Soufiane Hayou
Department of Mathematics
Consultation, Homework, Project
Consultation
We have three TAs for this class
• Wang Shida
• Jiang Haotian
• Wang Weixi

Each would have set up a 3x15min consultation slot per week.

Please use LumiNUS (consultation tab) to sign up.

I will also address common questions during the lectures.

Homework
Regression problem on the UCI Concrete Compressive
Strength dataset

• Instructions are in DSA5102X-homework1.ipynb

• Due: 4th Sept 2021 (2 weeks)
• Submission: Luminus submission folder under Files
• Late submission policy
• To ensure fairness, -20% of homework total grade every day
late, down to 0
• Example: Actual grade 8/10, 1 day late, Obtained grade:
6/10.
Project
Instructions are found in the Project Instructions folder on
Luminus

Use this homework as a starting point for the project

Due date: End of reading week before exam week

Last time
From linear models to linear basis models via feature maps

From this, we can derive the least squares formula etc.

Today, we will focus on the role of feature maps and their

relationship with kernels
Interpreting Feature Maps
What do feature maps do really?

𝜙3

𝜙1 𝜙2
Another view of feature maps
One can also view feature maps as implicitly defining some sort
of similarity measure

Consider two vectors and . Then, measures how similar they are

𝑢 𝑣 𝑢 𝑢𝑣

𝑣 Increasing

Increasing similarity
 Feature maps defines a similarity between two samples by
computing the dot product in feature space

𝜙 ( 𝑥) 𝜙 ( 𝑥′) 𝜙 ( 𝑥) 𝜙 ( 𝑥′)
𝜙 ( 𝑥)

𝜙 ( 𝑥′)
Increasing

Increasing similarity in feature space

Least Squares Revisited
Let us revisit the linear basis hypothesis space

The (regularized) least squares problem is

Recall:

This is known as ridge regression.

Solution of ridge regression:

Making new predictions

Two observations:
• The dataset is memorized by and is not needed for new
predictions
• For each new prediction, we have operations
Reformulation of Ridge Regression
Let us now write ridge regression solution another way…

Two observations:
• The input data participates in theGram matrix
predictions
• For each new prediction, we have operations
 Original Solution Reformulated Solution

What did we gain with this reformulation?

• They are exactly the same function, but…
• Left side requires and right side requires
• Most importantly, the right side only depends on the
feature maps through the dot product
 Reformulated Solution

So why not just specify it and

Only need “similarity” measure
forget about !

Kernel Ridge Regression

Kernel
Can we choose any we want?
Observe that cannot be arbitrary, since we need
• (Symmetry)

• (Non-negativity)

• (Positive Semi-definiteness)

for all

If satisfies these conditions, it is called Symmetric Positive

Definite (SPD). Are these conditions all we need?
Symmetric Positive Definite Kernels
For a kernel to represent a valid feature map, we define the
notion of Symmetric Positive Definite (SPD) kernels. These
satisfy

1. Symmetry:

2. Positive Semi-definiteness: For any and , the Gram matrix

is positive semi-definite
(Recall: a matrix is positive semi-definite if for any vector )
Mercer’s Theorem (1909)
Suppose is a SPD kernel. Then, there exists a feature space and a
feature map such that

In fact,

where are eigenvalues/eigenfunctions of the linear integral operator

Feature map SPD kernel

and

SPD kernel Feature map

Examples of SPD kernels
• Linear kernel

• Polynomial kernel

• Gaussian (Radial Basis Function, RBF) kernel

• Many more…
Flexibility of using kernels
For example, consider the RBF kernel in 1 input dimension

Taylor expanding gives

where

The feature space is infinite-dimensional!

Kernel ridge regression with different
types of kernels
 Express solution in
1 terms of similarity
(dot product)
Key Idea of
Kernel
Methods
Replace them with
2 kernels
Support Vector Machines
Linear (Affine) Functions and
Hyperplanes
Linear functions

What are hyperplanes?

In two dimension: a line

In general:

Hyperplanes are solutions of a linear equation

Classification using linear functions
Binary classification: ,
Linear decision function:
Linear separability assumption:
There exists a linear decision function such that
for all
Margin
Margin
There can be many possible decision functions!
Maximum Margin Solution
Mathematically, the margin of a decision function

is

The goal of support vector machines (SVM) is to find the

maximum margin solution

Why?
 1
min |𝑤 𝑥𝑖 +𝑏| subject ¿ 𝑦 𝑖 ( 𝑤 𝑥 𝑖 +𝑏 ) >0 ∀ 𝑖
𝑇 𝑇
max
𝑤 ,𝑏 ‖𝑤‖ 𝑖=1 ,… , 𝑁

Reformulated as a constrained
convex optimization problem

1 2
min ‖𝑤‖ subject ¿ 𝑦 𝑖 ( 𝑤 𝑥 𝑖 +𝑏 ) ≥1 ∀ 𝑖
𝑇

𝑤, 𝑏 2
The Method of Lagrange Multipliers
Minimizing a function can be found by . What if there are
constraints?
First example:

𝐹 (𝑧) 𝐹 (𝑧) ( ^
𝑧 )
𝐹
𝑧∇
^
∇𝐹
𝑎
𝑎

𝑇 𝑇
𝑧 𝑎=0 𝑧 𝑎=0
What about general equality constraints?

parallel at a local optimum point, i.e.

The gradient plays the role of so we must have and

𝐹 (𝑧)

^
𝑧 ∇ 𝐹( 𝑧^ )
∇ 𝐺( 𝑧^ )

𝐺 ( 𝑧 )=0
What about general inequality constraints?

Constraint Inactive Constraint Active

𝐹 (𝑧) 𝐹 (𝑧)

^
𝑧 ∇ 𝐹( 𝑧^ )
^
𝑧 ∇ 𝐺( 𝑧^ )
𝐺 ( 𝑧 )≤ 0 𝐺 ( 𝑧 )≤ 0

𝛻 𝐹(^ ^ ) <0
𝑧 ) =0 𝐺 ( 𝑧
 Inactive Case Active Case

Define the Lagrangian

Then these conditions can be combined in the following

conditions:

The variable is called a Lagrange multiplier

The most general case has inequality constraints (why no
equality constraints?)
Karush-Kuhn-Tucker (KKT) Conditions
Define the Lagrangian

Then, under technical conditions, for each locally optimal , there

exists Lagrange multipliers
1. (Stationarity)
2. (Primal Feasibility)
3. (Dual Feasibility)
4. (Complementary Slackness)
Dual Problem
Moreover, under some additional conditions, we can find the
multipliers via the dual problem

Choice (Primal) Price (Dual)

Back to the SVM…

1 2
min ‖𝑤‖ subject ¿ 𝑦 𝑖 ( 𝑤 𝑥 𝑖 +𝑏 ) ≥1 ∀ 𝑖
𝑇

𝑤, 𝑏 2

We apply KKT conditions with

We obtain the following
1. From Stationarity

2. From Dual Feasibility

3. From Complementary Slackness

4. The multipliers can be found by the dual problem

Dual Formulation of SVM

Decision function:
Complementary slackness:
Crucial Observations
From the dual formulation, we
observe the following
1. Only vectors closest to the
decision boundary matters
in predictions. These are
called support vectors.
2. The dual formulation of
the problem depends on
the inputs only through the
dot product support vectors
Kernel Support Vector Machines

Decision function:

As before, only support vectors satisfying

matter for predictions. This is a sparse kernel method.
Summary
The essence of kernel methods
• Write solution only in terms of dot products (this usually
involves going to a “dual” formulation)
• Go nonlinear by using kernels to replace dot products

Support vector machines

• Maximum margins solution
• Example of sparse kernel method: only some points
(support vectors) are used for prediction