Demystifying

Clustering: KMeans,
Agglomerative, and
DBSCAN
Welcome to this lecture on clustering techniques! Clustering
is a fundamental concept in machine learning, focusing on
grouping similar data points together. Today, we'll explore
three popular methods: KMeans, Agglomerative Clustering,
and DBSCAN. Each offers unique advantages for different
datasets and problems. Let's dive in and discover how these
algorithms can unlock valuable insights from your data.

by Props
KMeans Clustering: An Overview
KMeans partitions data into \(k\) clusters, aiming to minimize the sum of squared
distances between data points and their respective cluster centroids. This
optimization is represented mathematically as:

\[F = \sum_{i=1}^{k}\sum_{x_{j} \in S_{i}}\left \| x_{j} - \mu_{i} \right \|^{2}\]

KMeans assumes that clusters are spherical and equally sized, which makes it
very fast. However it's sensitive to initialization and the selection of \(k\). It
works best when these assumptions are met.

Advantages
• Fast
• Easy to Implement

Disadvantages
• Assumes spherical Clusters
• Sensitive to initial Centroids
• Requires pre-defined K clusters
KMeans vs. Agglomerative vs. DBSCAN
KMeans Agglomerative DBSCAN

A centroid-based approach, A hierarchical method that A density-based algorithm

which works best on data with doesn’t require a fixed \(k\). capable of detecting clusters
spherical clusters. It's Instead, it builds a with arbitrary shapes and
computationally efficient but dendrogram visualization to identifying noise points or
requires predefining the represent the merging of outliers. Great for messy data
number of clusters \(k\). clusters at different levels of with unpredicatable
Initialization is very important similarity. Flexible in terms of relationships.
to avoid local optima. cluster shape.
Introduction to
Agglomerative Clustering
Agglomerative clustering takes a "bottom-up" approach, starting with
each data point as its own cluster. It then iteratively merges the closest
pairs of clusters until only one cluster remains. This hierarchical process
can be visualized using a dendrogram, a tree-like diagram showing the
sequence of merges. Agglomerative clustering does not require to
specify the number of clusters beforehand, which is an advantage over
KMeans.

Bottom-Up Approach Iterative Merging

Each data point starts as its Closest clusters are merged

own cluster. based on similarity.

Dendrogram Visualization

The merging process is represented as a tree.

How Agglomerative Clustering Works
The key to agglomerative clustering lies in how it measures the similarity between clusters. Different linkage methods exist, each
with its own approach:

Single Linkage: Uses the shortest distance between any two points in the clusters.
Complete Linkage: Uses the longest distance between any two points in the clusters.
Average Linkage: Uses the average distance between all pairs of points in the clusters.
Ward’s Method: Minimizes the variance within clusters.

Agglomerative clustering is particularly useful for smaller datasets or when a hierarchical structure is expected in the data.

Ward's
1
Average
2

Complete
3

Single
4
DBSCAN: Density-Based
Spatial Clustering
DBSCAN forms clusters based on data density, grouping together
points that are closely packed together while marking as outliers
points that lie alone in low-density regions. The algorithm relies on
two key parameters: \(\epsilon\) (eps) and \(min\_samples\). Core
points have at least \(min\_samples\) within a radius of \(\epsilon\),
while border points are within \(\epsilon\) of a core point but do not
meet the density threshold themselves. Points that are neither core
nor border points are considered noise or outliers.

Core Points Border Points

Meet Density Threshold Near core but not dense

Noise Points

Outliers
How DBSCAN Works
Let's delve deeper into the workings of DBSCAN. First, the algorithm selects an unvisited data point and checks its neighborhood within the \(\epsilon\) radius. If the
neighborhood contains at least \(min\_samples\) data points, a new cluster is formed. The algorithm then expands the cluster by recursively finding all connected data points
that meet the density requirement. If the initial point does not meet the density threshold, it is marked as noise, at least until it is included in another point radius.

Select Point

Choose an unvisited point.

Check Density

Examine epsilon neighborhood.

Form Cluster

Expand if density threshold is met.

Mark Noise

If unexpandable.
Use Cases: Agglomerative & DBSCAN
Agglomerative Clustering DBSCAN

Gene expression analysis: Identify hierarchical Geographic data: Clustered cities based on
relationships between genes. population density, and noise isolates
Customer segmentation in marketing: Group Image processing: Segmenting complex
customers based on purchasing behavior and textures in satellite imagery or medical scans.
demographics. Anomaly Detection: Detecting unusual behavior
Document clustering: Identify topics based on in network traffic or financial transactions.
textual analysis.
Dendrograms Explained
A dendrogram serves as a visual tool for understanding the hierarchical structure produced by agglomerative clustering.
It displays the sequence of cluster merges, with the height of each branch indicating the distance between the merged
clusters. By cutting the dendrogram at a certain height, you can determine the optimal number of clusters for your
data. A higher cut leads to fewer, larger clusters, while a lower cut results in more, smaller clusters.

Branches

2 Show merging sequence

Nodes
Represent data points or clusters 1

Height
3
Indicates distance between clusters
Conclusion & Choosing Techniques
Choosing the right clustering technique depends on your data and goals. KMeans is fast and suitable for
spherical data with a known number of clusters, but sensitive to initialization. Agglomerative clustering is
valuable when a hierarchical structure is expected, particularly useful for small datasets, but can be
computationally expensive. DBSCAN excels at handling arbitrary shapes and identifying noise, making it ideal
for data with complex relationships.

When deciding which technique to use, always experiment and compare results. Each dataset has its own
story to tell. By understanding the strengths and weaknesses of each algorithm, you can unlock valuable
insights and make informed decisions.

DBSCAN
Agglomerative
Arbitrary shapes, handles noise
KMeans
Hierarchical needs, small datasets
Fast, spherical data, fixed \(k\)