1

Unsupervised learning
Announcements

▪ Last year exam posted on Moodle front page

▪ Quiz 1 grades posted.

▪ Next week: 22.11.2023, Quiz 2 - same format as last week

• Covers topics covered during 23.10-06.11
 3
Outline

▪ Unsupervised learning

• Dimensionality reduction: Principal Component Analysis (PCA)

• Clustering: k-Means
 4
Introduction Linear regression Logistic regression

Feature engineering Data statistics Naive Bayes

KNN Clustering Dimensionality reduction

Neural networks Convolutional neural Decision-trees

networks
 Review of data statistics, ex: Quiz 1 grades
Mean: 7.63, Mode: 8, Median: 8, Standard deviation: 1.85
Brief review of last lecture
 Introduction
Unsupervised learning

Unsupervised learning is a type of machine learning that looks for previously

undetected patterns in a data set with no pre-existing labels and with a minimum of
human supervision.

in the next techniques, we don’t use labels anymore!

Dimensionality reduction
 Motivation
Intuition

We have samples described by a series of features

We want to find a smaller set of new

x2 features that explain our sample because:

Less features is easier to visualize

Feature 2

Some of the current feature can be

redundant
Some of the current features are not very
useful to describe our samples
Feature 1 x1
 Principal component analysis (PCA)
Approach to dimensionality reduction

How to find this smaller set of new features ?

PCA : Find the best linear combination of
x2
features to create new features that explain our
samples better
Feature 2

Feature 1 x1
 PCA
Projection of points onto a lower dimensional subspace

How to find this smaller set of new features ?

x2

w1x1 + w2x2
The new feature
Length

A mix of length and weight that describe

our samples better

Weight x1
 Projection of a point onto a subspace
Subspace

Distance of a point to a subspace

Projection of a point onto a subspace

 Find a subspace to minimize average distance of data to it

Sum of distances of all data points to a subspace

PCA chooses a subspace to minimize

 Formulation of PCA objective using the data matrix

Standardize data:

Frobenius norm of a matrix:

PCA objective:
Eigenvalue decomposition of a symmetric matrix
Solution to PCA using eigenvalue decomposition
 Principal components
Example: projection of data onto 2 principal components
PCA
Pseudo-code
 PCA example - distinguishing texts
Defining features

Each data sample is a document

There are d unique words in all the documents

Feature is positive for document if word is in document

 TFIDF feature definition for documents

Term frequency of word … in document …

Document frequency of word ….

TFIDF for each word ….

 PCA example
Based on “Principal Component Analysis” lecture of Stanford EE104

Distinguishing text: The Critique of Pure Reason by Immanuel Kant and

The Problems of Philosophy by Bertrand Russell
Dimensionality reduction
Other techniques

There are several other techniques for dimensionality reduction :

Linear discriminant analysis (LDA)

Generalized discriminant analysis (GDA)

T-distributed Stochastic Neighbor Embedding (t-SNE)

Autoencoders
 Autoencoder
Introduction

▪ An autoencoder is a type of neural network often used for dimensionality

reduction.
▪ Autoencoders are trained in an unsupervised manner, by minimizing the
N
L (x , x ̂ )
i i
∑
reconstruction error / loss
i=1

▪ Example: Squared error

 Autoencoder
Autoencoder vs. PCA

Top: Some examples of the original MNIST

test samples

Middle: Reconstructed output from an auto-

encoder with a latent space of 8 dimensions
This auto-encoder uses convolutional layers, and
was trained on the MNIST training set

Bottom: Reconstructed output from PCA with

8 reduced dimensions

Image credit: F. Fleuret, Deep Learning (EPFL)

 Summary - dimensionality reduction
Used for
• Exploratory data analysis
• Visualizing data
• Help reduce overfitting by reducing feature dimension

PCA: an approach to dimensionality reduction

▪ Projects data onto a linear subspace
▪ Useful in case there is approximately linear dependence
between different features
▪ Easy to compute
▪ Connection to singular value decomposition (see Problem set 2)
 Clustering
ML

Supervised
S
& - unsupervised
classification dimensionale

I 1)
regression ,
recluchen clostery
2X :
lin logesha
regar K .
NN
negression
>
·

PCA
-
K-
means
.
"
NN
-

Naive
Bayes
 Clustering - definition and motivation

Goal: group data points as a first step to understand the data set

Examples: (see 4.4 and 4.5 in LinAlgebra book

- Clustering music genres: grouping music based on the similarities in their
audio features (beat per minute, duration, loudness, amount of words, etc)
 Clustering introduction

▪ Toy example:
• Each data sample is a point in 2D

cluster 1
cluster 2

cluster 3 cluster 4
k-means approach to clustering

How to cluster data without labels?

Try to find similarity between groups of points

k-means: Group points based on their proximity

(in terms of distance in the feature space)
k-means
Introduction

▪ Given a set of unlabelled input samples, group the samples into k

clusters (k ∈ ℕ) [xi). ,

▪ k-Means idea:
• Identify&
k cluster of data points given N samples.
-

• Find prototype points μ1, μ2, . . . , μk representing the center of each

-

&
cluster and add the other data points to the nearest cluster center.

pank
representative
 k-Means
Preliminaries

▪ A single representative point for data:

 k-Means
▪ Choose k clusters to represent data

▪ Determining the cluster a single point … belongs to

▪ Determining the cluster centres to minimize the distance of each point to its
assigned cluster
 k-Means objective function

Algorithm - heuristic
1. Initialize {μ1, μ2, . . . , μk} (e.g., randomly)
2. While not converged
1. Assign each point …. to the nearest center
2. Update each center μj based on the points assigned to it
 k-Means
Algorithm - Details
▪ Step 2.1: Assign each point…. to the nearest center
• For each point …, compute the Euclidean distance to every center
{μ1, μ2, . . . , μk}
• Find the smallest distance
• The point is said to be assigned to the corresponding cluster (note that each
point is assigned to a single cluster)
▪ Step 2.2: Update each center μj based on the points assigned to it

• Recompute each center μj as the mean of the points that were assigned to it
 k-Means
Algorithm - Convergence
▪ Step 2 is repeated while k-Means has not converged
• What criteria to stop iterating?
• Fixed number of iterations? It’s arbitrary and a too small number can lead to
bad results
• The difference in assignments or center locations between two iterations can
be used as criteria to stop the algorithm

▪ K-Means does not always converge to the best solution

•
 k-Means
Example
▪ Use the Palmer Penguins dataset
▪ With Flipper length against Bill length

With label Without label

 k-Means
Example
▪ Centroid initialisation
 k-Means
Example
▪ First assignment
 k-Means
Example
▪ Next assignment
 k-Means
Example
▪ Next assignment
 k-Means
Example
▪ Final assignment
 Summary - Clustering
Used for understanding data
Examples:
Topic discovery in a large set of documents
Recommendation engines
Guessing missing entries

k-means: an approach to clustering

Easy to implement and to interpret
k-means algorithm converges. However, non-convex optimization: solution depends on
initial condition
 Further reading

PCA
importance of Standardisation:
PCA on StatQuest!!! k-Means: chapter 4 of LinAlgebra book

k-means: Chapter 4 of LinAlgebra book

▪ For coding of PCA/k-means, see SciKit Learn