K- means and Hierarchical Clustering

What is Clustering ?

● Clustering is the classification of objects into different

groups, or more precisely, the partitioning of a data set
into subsets (clusters), so that the data in each subset
(ideally) share some common trait - often according to
some defined distance measure.

2
K – Means Clustering

3
K – means Clustering:
● K-means clustering is one of the simplest and popular
unsupervised machine learning algorithms.

● The objective of K-means is simple: group similar data points

together and discover underlying patterns. To achieve this
objective, K-means looks for a fixed number (k) of clusters in
a dataset.

● In other words, the K-means algorithm identifies k number

of centroids, and then allocates every data point to the nearest
cluster, while keeping the centroids as small as possible.
The ‘means’ in the K-means refers to averaging of the data;
that is, finding the centroid.
4
How K – means Clustering algorithm works?

5
K – means clustering Algorithm with example:
Given Dataset:
K= {2,3,4,10,11,12,20,25,30}
K=2

6
Weakness of K – means Clustering:

● When the numbers of data are not so many, initial grouping will
determine the cluster significantly.

● The number of cluster, K, must be determined before hand. Its

disadvantage is that it does not yield the same result with each run, since
the resulting clusters depend on the initial random assignments.

● We never know the real cluster, using the same data, because if it is
inputted in a different order it may produce different cluster if the number
of data is few.

● It is sensitive to initial condition. Different initial condition may produce

different result of cluster. The algorithm may be trapped in the local
optimum.

7
Applications of K – means Clustering:

● It is relatively efficient and fast. It computes result at O(tkn),

where n is number of objects or points, k is number of
clusters and t is number of iterations.

● k-means clustering can be applied to machine learning or

data mining.

● Used on acoustic data in speech understanding to convert

waveforms into one of k categories (known as Vector
Quantization or Image Segmentation).

● Also used for choosing color palettes on old fashioned

graphical display devices and Image Quantization.
8
Hierarchical Clustering

9
Hierarchical Clustering

● The hierarchical clustering Technique is one of the

popular Clustering techniques in Machine Learning.
● Hierarchical clustering is one of the easy to understand

clustering technique. This clustering technique is divided into

two types:
Agglomerative
Divisive

10
Hierarchical Clustering

● Agglomerative approach
Initialization:
Each object is a cluster
Iteration:
a
ab Merge two clusters which are
b abcde most similar to each other;
Until all objects are merged
c
cde into a single cluster
d
de
e

Step 0 Step 1 Step 2 Step 3 Step 4 bottom-up

11
Hierarchical Clustering

● Divisive Approaches Initialization:

All objects stay in one cluster
Iteration:
a Select a cluster and split it into
ab
two sub clusters
b abcde Until each leaf cluster contains
c only one object
cde
d
de
e

Step 4 Step 3 Step 2 Step 1 Step 0 Top-down

12
Dendrogram
● A binary tree that shows how clusters are
merged/split hierarchically
● Each node on the tree is a cluster; each leaf node is a
singleton cluster

13
Hierarchical Agglomerative Clustering-Linkage
Method
● The single linkage method is based on minimum distance,
or the nearest neighbor rule.
● The complete linkage method is based on the maximum
distance or the furthest neighbor approach.
● The average linkage method the distance between two
clusters is defined as the average of the distances between all
pairs of objects.
Centroid Method
● In the centroid methods, the distance between two clusters is
the distance between their centroids.

14
 Single Linkage

Minimum Distance

Cluster 2
Cluster 1
Complete Linkage

Maximum
Distance

Cluster 1 Cluster 2
Average Linkage

Average Distance
15
Cluster 1 Cluster 2
Centroid Method

16
How to Merge Clusters?

● How to measure the distance between clusters?

● Single-link
● Complete-link
Distance?
● Average-link
● Centroid distance

Hint: Distance between clusters is

usually defined on the basis of distance
between objects.

17
How to Define Inter-Cluster Distance

● Single-link
● Complete-link
● Average-link The distance between two
● Centroid distance clusters is represented by the
distance of the closest pair of
data objects belonging to
different clusters.
18
How to Define Inter-Cluster Distance

● Single-link
● Complete-link
● Average-link The distance between two
● Centroid distance clusters is represented by the
distance of the farthest pair of
data objects belonging to
different clusters.
19
How to Define Inter-Cluster Distance

● Single-link
● Complete-link
● Average-link
The distance between two
● Centroid distance
clusters is represented by the
average distance of all pairs of
data objects belonging to
different clusters.
20
How to Define Inter-Cluster Distance

× ×

● Single-link
● Complete-link
● Average-link The distance between two
● Centroid distance clusters is represented by the
distance between the means of
the cluters.

21
An Example of the Agglomerative Hierarchical
Clustering Algorithm

● For the following data set, we will get different

clustering results with the single-link and
complete-link algorithms.

1 5

3 4
2 6

22
Result of the Single-Link algorithm

1 5

3 4
2 6
1 3 4 5 2 6

Result of the Complete-Link algorithm

1 5

3 4
2 6
1 3 2 4 5 6

23
Hierarchical Clustering: Comparison
Single-link Complete-link
5
1 4 1
3
2 5
5 5
2 1 2
2 3 6 3 6
3
1
4 4
4

Average-link Centroid distance

5
1 5 4 1
2
5 2
2 5
2
3 6 3
3 6
4 1 1
4 4
3
24
Compare Dendrograms
Single-link Complete-link

1 2 5 3 6 4 1 2 5 3 6 4

Average-link Centroid distance

2 5 3 6 4 1 25
1 2 5 3 6 4
Strength of Single-link

Original Points Two Clusters

• Can handle non-global shapes

26
 Limitations of Single-Link

Original Points
Two Clusters

• Sensitive to noise and outliers

27
Strength of Complete-link

Original Points Two Clusters

• Less susceptible to noise and outliers

28
Which Distance Measure is Better?
● Each method has both advantages and disadvantages;
application-dependent, single-link and complete-link
are the most common methods
● Single-link
● Can find irregular-shaped clusters
● Sensitive to outliers, suffers the so-called chaining effects
● Complete-link, Average-link, and Centroid distance
● Robust to outliers
● Tend to break large clusters
● Prefer spherical clusters

29
30