Fuzzy Clustering

Clustering is an unsupervised machine learning technique that divides the given data into different clusters
based on their distances (similarity) from each other.
The unsupervised k-means clustering algorithm gives the values of any point lying in some particular cluster
to be either as 0 or 1 i.e., either true or false. But the fuzzy logic gives the fuzzy values of any particular data
point to be lying in either of the clusters.
In Fuzzy c-means clustering, we find out the centroid of the data points and then calculate the distance of
each data point from the given centroids until the clusters formed become constant.
Suppose the given data points are {(1, 3), (2, 5), (6, 8), (7, 9)}
Fuzzy Clustering is a type of clustering algorithm in machine learning that allows a data point to belong to
more than one cluster with different degrees of membership. Unlike other clustering algorithms, such as k-
means or hierarchical clustering, which assign each data point to a single cluster, fuzzy clustering assigns a
membership degree between 0 and 1 for each data point for each cluster.

Applications in several fields of Fuzzy clustering :

1. Image segmentation: Fuzzy clustering can be used to segment images by grouping pixels with similar
properties together, such as color or texture.
2. Pattern recognition: Fuzzy clustering can be used to identify patterns in large datasets by grouping
similar data points together.
3. Marketing: Fuzzy clustering can be used to segment customers based on their preferences and
purchasing behavior, allowing for more targeted marketing campaigns.
4. Medical diagnosis: Fuzzy clustering can be used to diagnose diseases by grouping patients with similar
symptoms together.
5. Environmental monitoring: Fuzzy clustering can be used to identify areas of environmental concern by
grouping together areas with similar pollution levels or other environmental indicators.
6. Traffic flow analysis: Fuzzy clustering can be used to analyze traffic flow patterns by grouping similar
traffic patterns together, allowing for better traffic management and planning.
7. Risk assessment: Fuzzy clustering can be used to identify and quantify risks in various fields, such as
finance, insurance, and engineering.

Advantages of Fuzzy Clustering:

1. Flexibility: Fuzzy clustering allows for overlapping clusters, which can be useful when the data has a
complex structure or when there are ambiguous or overlapping class boundaries.
2. Robustness: Fuzzy clustering can be more robust to outliers and noise in the data, as it allows for a more
gradual transition from one cluster to another.
3. Interpretability: Fuzzy clustering provides a more nuanced understanding of the structure of the data, as it
allows for a more detailed representation of the relationships between data points and clusters.

Disadvantages of Fuzzy Clustering:

1. Complexity: Fuzzy clustering algorithms can be computationally more expensive than traditional
clustering algorithms, as they require optimization over multiple membership degrees.

2. Model selection: Choosing the right number of clusters and membership functions can be challenging,
and may require expert knowledge or trial and error.

The steps to perform the algorithm are:

Step 1: Initialize the data points into the desired number of clusters randomly.
Let us assume there are 2 clusters in which the data is to be divided, initializing the data point randomly.
Each data point lies in both clusters with some membership value which can be assumed anything in the
initial state.
The table below represents the values of the data points along with their membership (gamma) in each
cluster.

Cluster (1, 3) (2, 5) (4, 8) (7, 9)

1) 0.8 0.7 0.2 0.1
2) 0.2 0.3 0.8 0.9
Step 2: Find out the centroid.
The formula for finding out the centroid (V) is:

Where, µ is fuzzy membership value of the data point, m is the fuzziness parameter (generally taken as
2), and xk is the data point.
Here,
V11 = (0.8^2 *1 + 0.7^2 * 2 + 0.2^2 * 4 + 0.1^2 * 7) / ( (0.8^2 + 0.7^2 + 0.2^2 +
0.1^2 ) = 1.568
V12 = (0.8^2 *3 + 0.7^2 * 5 + 0.2^2 * 8 + 0.1^2 * 9) / ( (0.8^2 + 0.7^2 + 0.2^2 +
0.1^2 ) = 4.051
V21 = (0.2^2 *1 + 0.3^2 * 2 + 0.8^2 * 4 + 0.9^2 * 7) / ( (0.2^2 + 0.3^2 + 0.8^2 +
0.9^2 ) = 5.35
V22 = (0.2^2 *3 + 0.3^2 * 5 + 0.8^2 * 8 + 0.9^2 * 9) / ( (0.2^2 + 0.3^2 + 0.8^2 +
0.9^2 ) = 8.215
Centroids are: (1.568, 4.051) and (5.35, 8.215)
Step 3: Find out the distance of each point from the centroid.
D11 = ((1 - 1.568)2 + (3 - 4.051)2)0.5 = 1.2
D12 = ((1 - 5.35)2 + (3 - 8.215)2)0.5 = 6.79
Similarly, the distance of all other points is computed from both the centroids.
Step 4: Updating membership values.

For point 1 new membership values are:

= [{ [(1.2)2 / (1.2)2] + [(1.2)2 / (6.79)2]} ^ {(1 / (2 – 1))} ] -1 = 0.96
= [{ [(6.79)2 / (6.79)2] + [(6.79)2 / (1.2)2]} ^ {(1 / (2 – 1))} ] -1 = 0.04
Alternatively,

Similarly, compute all other membership values, and update the matrix.
Step 5: Repeat the steps(2-4) until the constant values are obtained for the membership values or the
difference is less than the tolerance value (a small value up to which the difference in values of two
consequent updations is accepted).
Step 6: Defuzzify the obtained membership values.
Implementation: The fuzzy scikit learn library has a pre-defined function for fuzzy c-means which can be
used in Python. For using fuzzy c-means you need to install the skfuzzy library.
pip install sklearn
pip install skfuzzy

OPTICS Clustering Explanation

OPTICS Clustering stands for Ordering Points To Identify Cluster Structure. It draws inspiration from the
DBSCAN clustering algorithm. It adds two more terms to the concepts of DBSCAN clustering.
OPTICS (Ordering Points To Identify the Clustering Structure) is a density-based clustering algorithm, similar
to DBSCAN (Density-Based Spatial Clustering of Applications with Noise), but it can extract clusters of
varying densities and shapes. It is useful for identifying clusters of different densities in large, high-
dimensional datasets.
The main idea behind OPTICS is to extract the clustering structure of a dataset by identifying the density-
connected points. The algorithm builds a density-based representation of the data by creating an ordered list
of points called the reachability plot. Each point in the list is associated with a reachability distance, which is
a measure of how easy it is to reach that point from other points in the dataset. Points with similar
reachability distances are likely to be in the same cluster.
 The OPTICS algorithm follows these main steps:
Define a density threshold parameter, Eps, which controls the minimum density of clusters.
For each point in the dataset, calculate the distance to its k-nearest neighbors.
Starting with an arbitrary point, calculate the reachability distance of each point in the dataset, based on the
density of its neighbors.
Order the points based on their reachability distance and create the reachability plot.
Extract clusters from the reachability plot by grouping points that are close to each other and have similar
reachability distances.
One of the main advantage of OPTICS over DBSCAN, is that it does not require to set the number of
clusters in advance, instead, it extracts the clustering structure of the data and produces the reachability
plot. This allows the user to have more flexibility in selecting the number of clusters, by cutting the
reachability plot at a certain point.
Also, unlike other density-based clustering algorithms like DBSCAN, It can handle clusters of different
densities and shapes and can identify hierarchical structure.
OPTICS is implemented in Python using the sklearn.cluster.OPTICS class in the scikit-learn library. It takes
several parameters including the minimum density threshold (Eps), the number of nearest neighbors to
consider (min_samples), and a reachability distance cutoff (xi).
They are:-
1. Core Distance: It is the minimum value of radius required to classify a given point as a core point. If the
given point is not a Core point, then it’s Core Distance is undefined.
2. Reachability Distance: It is defined with respect to another data point q(Let). The Reachability distance
between a point p and q is the maximum of the Core Distance of p and the Euclidean Distance(or some
other distance metric) between p and q. Note that The Reachability Distance is not defined if q is not a
Core point.

This clustering technique is different from other clustering techniques in the sense that this technique does
not explicitly segment the data into clusters. Instead, it produces a visualization of Reachability distances
and uses this visualization to cluster the data.

OPTICS Clustering v/s DBSCAN Clustering:

1. Memory Cost : The OPTICS clustering technique requires more memory as it maintains a priority queue
(Min Heap) to determine the next data point which is closest to the point currently being processed in
terms of Reachability Distance. It also requires more computational power because the nearest neighbour
queries are more complicated than radius queries in DBSCAN.
2. Fewer Parameters : The OPTICS clustering technique does not need to maintain the epsilon parameter
and is only given in the above pseudo-code to reduce the time taken. This leads to the reduction of the
analytical process of parameter tuning.
 3. This technique does not segregate the given data into clusters. It merely produces a Reachability
distance plot and it is upon the interpretation of the programmer to cluster the points accordingly.
4. Handling varying densities: DBSCAN clustering can struggle to handle datasets with varying densities,
as it requires a single value of epsilon to define the neighborhood size for all points. In contrast, OPTICS
can handle varying densities by using the concept of reachability distance, which adapts to the local
density of the data. This means that OPTICS can identify clusters of different sizes and shapes more
effectively than DBSCAN in datasets with varying densities.
5. Cluster extraction: While both OPTICS and DBSCAN can identify clusters, OPTICS produces a
reachability distance plot that can be used to extract clusters at different levels of granularity. This allows
for more flexible clustering and can reveal clusters that may not be apparent with a fixed epsilon value in
DBSCAN. However, this also requires more manual interpretation and decision-making on the part of the
programmer.
6. Noise handling: DBSCAN explicitly distinguishes between core points, boundary points, and noise
points, while OPTICS does not explicitly identify noise points. Instead, points with high reachability
distances can be considered as potential noise points. However, this also means that OPTICS may be
less effective at identifying small clusters that are surrounded by noise points, as these clusters may be
merged with the noise points in the reachability distance plot.
7. Runtime complexity: The runtime complexity of OPTICS is generally higher than that of DBSCAN, due
to the use of a priority queue to maintain the reachability distances. However, recent research has
proposed optimizations to reduce the computational complexity of OPTICS, making it more scalable for
large datasets.

Difference Between Agglomerative clustering and Divisive

clustering


Hierarchical clustering is a popular unsupervised machine learning technique used to group

similar data points into clusters based on their similarity or dissimilarity. It is called “hierarchical”
because it creates a tree-like hierarchy of clusters, where each node represents a cluster that can
be further divided into smaller sub-clusters.
There are two types of hierarchical clustering techniques:
1. Agglomerative and
2. Divisive clustering

Agglomerative Clustering

Agglomerative clustering is a type of hierarchical clustering algorithm that merges the most similar pairs of
data points or clusters, building a hierarchy of clusters until all the data points belong to a single cluster. It
starts with each data point as its own cluster and then iteratively merges the most similar pairs of clusters
until all data points belong to a single cluster

Divisive Clustering

Divisive Clustering is the technique that starts with all data points in a single cluster and recursively splits the
clusters into smaller sub-clusters based on their dissimilarity. It is also known as, “top-down” clustering. It
starts with all data points in a single cluster, and then recursively splits the clusters into smaller sub-clusters
based on their dissimilarity.
Unlike agglomerative clustering, which starts with each data point as its own cluster and iteratively merges
the most similar pairs of clusters, divisive clustering is a “divide and conquer” approach that breaks a large
cluster into smaller sub-clusters
Difference between agglomerative clustering and Divisive clustering :
S.No. Parameters Agglomerative Clustering Divisive Clustering

1. Category Bottom-up approach Top-down approach

each data point starts in its all data points start in a

own cluster, and the algorithm single cluster, and the
recursively merges the closest algorithm recursively splits
2. Approach
pairs of clusters until a single the cluster into smaller sub-
cluster containing all the data clusters until each data
points is obtained. point is in its own cluster.

Agglomerative clustering is
generally more Comparatively less
computationally expensive, expensive as divisive
especially for large datasets as clustering only requires the
3. Complexity level this approach requires the calculation of distances
calculation of all pairwise between sub-clusters, which
distances between data can reduce the
points, which can be computational burden.
computationally expensive.

Agglomerative clustering can divisive clustering may

handle outliers better than create sub-clusters around
4. Outliers divisive clustering since outliers, leading to
outliers can be absorbed into suboptimal clustering
larger clusters results.

5. Interpretability Agglomerative clustering divisive clustering can be

tends to produce more more difficult to interpret
interpretable results since the since the dendrogram shows
dendrogram shows the the splitting process of the
merging process of the clusters, and the user must
clusters, and the user can choose a stopping criterion
choose the number of clusters to determine the number of
based on the desired level of clusters.
S.No. Parameters Agglomerative Clustering Divisive Clustering

granularity.

Scikit-learn provides multiple

linkage methods for
divisive clustering is not
6. Implementation agglomerative clustering, such
currently implemented in
as “ward,” “complete,”
Scikit-learn.
“average,” and “single,”

Here are some of the

Here are some of the
applications in which
applications in which
Agglomerative Clustering is
Divisive Clustering is used :
used :
7. Example Market segmentation,
Image segmentation, Customer
Anomaly detection,
segmentation, Social network
Biological classification,
analysis, Document clustering,
Natural language processing,
Genetics, genomics, etc., and
etc.
many more.