Lesson 4.1 - Unsupervised Learning Partitioning Methods PDF
Lesson 4.1 - Unsupervised Learning Partitioning Methods PDF
December 3, 2018
1 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Roadmap
1 Basic Concepts
2 K-Means
3 K-Medoids
2 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Cluster Analysis
Principle
Maximizing intra-class similarity & minimizing interclass similarity
Typical Applications
WWW, Social networks, Marketing, Biology, Library, etc.
3 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Partitioning Methods
Given
A data set of n objects
K the number of clusters to form
Organize the objects into k partitions (k ≤ n) where each partition
represents a cluster
The clusters are formed to optimize an objective partitioning
criterion
Objects within a cluster are similar
Objects of different clusters are dissimilar
4 / 41
Unsupervised Learning: Nearest-Neighbor Classification
1 Eager learning
Given a set of training set, constructs a classification model before
receiving new (e.g., test) data to classify
e.g. decision tree induction, Bayesian classification, rule-based
classification
2 Lazy learning
Simply stores training data (or only minor processing) and waits
until it is given a new instance
Lazy learners take less time in training but more time in predicting
e.g., k-nearest-neighbor classifiers, case-based reasoning classifiers
5 / 41
Unsupervised Learning: Nearest-Neighbor Classification
6 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Lazy Learning
7 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Roadmap
1 Basic Concepts
2 K-Means
3 K-Medoids
8 / 41
Unsupervised Learning: Nearest-Neighbor Classification
K-Means
Goal
Create 3 clusters (partitions)
9 / 41
Unsupervised Learning: Nearest-Neighbor Classification
K-Means
Goal
Create 3 clusters (partitions)
4 Recompute Clusters
10 / 41
Unsupervised Learning: Nearest-Neighbor Classification
K-Means Algorithm
Input
K : the number of clusters
D : a data set containing n objects
Output: A set of k clusters
Method:
1 Arbitrary choose k objects from D as in initial cluster centers
2 Repeat
3 Reassign each object to the most similar cluster based on the mean
value of the objects in the cluster
4 Update the cluster means
5 Until no change
11 / 41
Unsupervised Learning: Nearest-Neighbor Classification
K-Means Properties
E : the sum of the squared error for all objects in the dataset
P : the data point in the space representing an object
mi : is the mean of cluster Ci
It works well when the clusters are compact clouds that are rather
well separated from one another
12 / 41
Unsupervised Learning: Nearest-Neighbor Classification
K-Means: Advantages
13 / 41
Unsupervised Learning: Nearest-Neighbor Classification
K-Means: Disadvantages
14 / 41
Unsupervised Learning: Nearest-Neighbor Classification
K-Means demo
Demo
15 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Variations of K-Means
16 / 41
Unsupervised Learning: Nearest-Neighbor Classification
k-Nearest-Neighbor Classifiers
17 / 41
Unsupervised Learning: Nearest-Neighbor Classification
k-Nearest-Neighbor Classifiers
Example:
We are interested in classifying the type of drug a patient should
be prescribed
Based on the age of the patient and the patient’s
sodium/potassium ratio (Na/K)
Dataset includes 200 patients
18 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Scatter plot
19 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Main questions:
How many neighbors should we consider? That is, what is k?
How do we measure distance?
Should all points be weighted equally, or should some points have
more influence than others?
20 / 41
Unsupervised Learning: Nearest-Neighbor Classification
k-Nearest-Neighbor Classifiers
21 / 41
Unsupervised Learning: Nearest-Neighbor Classification
k-Nearest-Neighbor Classifiers
Min-max normalization:
all attribute values lie between 0 and 1
For more information on normalization methods refer to data
preprocessing section
22 / 41
Unsupervised Learning: Nearest-Neighbor Classification
k-Nearest-Neighbor Classifiers
23 / 41
Unsupervised Learning: Nearest-Neighbor Classification
k-Nearest-Neighbor Classifiers
24 / 41
Unsupervised Learning: Nearest-Neighbor Classification
k-Nearest-Neighbor Classifiers
25 / 41
Unsupervised Learning: Nearest-Neighbor Classification
k-Nearest-Neighbor Classifiers
26 / 41
Unsupervised Learning: Nearest-Neighbor Classification
27 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Roadmap
1 Basic Concepts
2 K-Means
3 K-Medoids
28 / 41
Unsupervised Learning: Nearest-Neighbor Classification
K-Medoids Method
E : the sum of absolute error for all objects in the data set
P : the data point in the space representing an object
Oi : is the representative object of cluster Ci
29 / 41
Unsupervised Learning: Nearest-Neighbor Classification
30 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Data Objects
A1 A2
O1 2 6
O2 3 4
O3 3 8
O4 4 7
O5 6 2
O6 6 4
O7 7 3
O8 7 4
O9 8 5
O10 7 6 Goal: create two clusters
Choose randmly two medoids
O2 = (3, 4)
O8 = (7, 4)
31 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Data Objects
A1 A2
O1 2 6
O2 3 4
O3 3 8
O4 4 7
O5 6 2
O6 6 4
O7 7 3
O8 7 4
O9 8 5 Assign each object to the closest representative object
O10 7 6
Using L1 Metric (Manhattan), we form the following
clusters
Cluster1 = {O1 , O2 , O3 , O4 }
Cluster2 = {O5 , O6 , O7 , O8 , O9 , O10 }
32 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Data Objects
A1 A2
O1 2 6
O2 3 4
O3 3 8
O4 4 7
O5 6 2
O6 6 4 Compute the absolute error criterion [for the set of Medoids
O7 7 3 (O2 ,O8 )]
O8 7 4
O9 8 5 k X
X
O10 7 6 E= |p − oi |
i =1 p∈Ci
= |O1 − O2 | + |O3 − O2 | + |O4 − O2 | + |O5 − O8 |+
|O6 − O8 | + |O7 − O8 | + |O9 − O8 | + |O10 − O8 |
= (3 + 4 + 4) + (3 + 1 + 1 + 2 + 2) = 20
33 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Data Objects
A1 A2
O1 2 6
O2 3 4
O3 3 8
O4 4 7
O5 6 2
O6 6 4
O7 7 3
O8 7 4
O9 8 5
O10 7 6 Choose a random object O7
Swap O8 and O7
Compute the absolute error criterion [for the set of
Medoids (O2 , O7 )]
34 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Data Objects
A1 A2
O1 2 6
O2 3 4
O3 3 8
O4 4 7
O5 6 2
O6 6 4
O7 7 3
O8 7 4
O9 8 5 Compute the cost function
O10 7 6 Absolute error [for O2 ,O7 ] - Absolute error [O2 ,O8 ]
S = 22 − 20
35 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Data Objects
A1 A2
O1 2 6
O2 3 4
O3 3 8
O4 4 7
O5 6 2
O6 6 4
O7 7 3
O8 7 4
O9 8 5
O10 7 6 In this example, changing the medoid of cluster 2 did
not change the assignments of objects to clusters.
What are the possible cases when we replace a medoid
by another object?
36 / 41
Unsupervised Learning: Nearest-Neighbor Classification
K-Medoids
First case
Currently P assigned to A
The assignment of P to A does not
change
Second case
Currently P assigned to B
P is reassigned to A
37 / 41
Unsupervised Learning: Nearest-Neighbor Classification
K-Medoids
Third case
Currently P assigned to B
P is reassigned to the new B
Fourth case
Currently P assigned to A
P is reassigned to A
38 / 41
Unsupervised Learning: Nearest-Neighbor Classification
Input
K : the number of clusters
D : a data set containing n objects
Output: A set of k clusters
Method:
1 Arbitrary choose k objects from D as representative objects (seeds)
2 Repeat
3 Assign each remaining object to the cluster with the nearest
representative object
4 For each representative object Oj
5 Randomly select a non representative object Orandom
6 Compute the total cost S of swapping representative object Oj with
Orandom
7 if S < 0 then replace Oj with Orandom
8 Until no change
39 / 41
Unsupervised Learning: Nearest-Neighbor Classification
PAM Properties
40 / 41
Unsupervised Learning: Nearest-Neighbor Classification
References
41 / 41