Density-based Approaches

 Why Density-Based Clustering methods?

 Discover clusters of arbitrary shape.
 Clusters – Dense regions of objects separated by
regions of low density
 DBSCAN – the first density based clustering
 OPTICS – density based cluster-ordering
 DENCLUE – a general density-based description
of cluster and clustering
Source: https://cse.buffalo.edu/faculty/azhang/cse601/density-based
.ppt
 DBSCAN: Density Based Spatial
Clustering of Applications with Noise

Proposed by Ester, Kriegel, Sander, and Xu

(KDD96)
Relies on a density-based notion of cluster: A
cluster is defined as a maximal set of density-
connected points.
Discovers clusters of arbitrary shape in spatial
databases with noise
 Density-Based Clustering
Basic Idea:
Clusters are dense regions in the
data space, separated by regions of
lower object density

Why Density-Based Clustering?

Results of a k-medoid

algorithm for k=4

Different density-based approaches exist (see Textbook & Papers)

Here we discuss the ideas underlying the DBSCAN algorithm
 Density Based Clustering: Basic Concept

Intuition for the formalization of the basic idea

For any point in a cluster, the local point density
around that point has to exceed some threshold
The set of points from one cluster is spatially
connected
Local point density at a point p defined by two
parameters
– radius for the neighborhood of point p:
N (p) := {q in data set D | dist(p, q)  }
MinPts – minimum number of points in the given
neighbourhood N(p)
 -Neighborhood
 -Neighborhood – Objects within a radius of 
from an object.
N  ( p ) : {q | d ( p, q )  }
 “High density” - ε-Neighborhood of an object
contains at least MinPts of objects.

ε-Neighborhood of p
ε ε ε-Neighborhood of q
qq pp
Density of p is “high” (MinPts = 4)
Density of q is “low” (MinPts = 4)
 Core, Border & Outlier
Outlier Given  and MinPts,
categorize the objects into
Border three exclusive groups.

A point is a core point if it has more than

Core a specified number of points (MinPts)
within Eps These are points that are at
the interior of a cluster.

A border point has fewer than

 = 1unit, MinPts = 5 MinPts within Eps, but is in the
neighborhood of a core point.

A noise point is any point that is not

a core point nor a border point.
 Example

M, P, O, and R are core objects since each is

in an Eps neighborhood containing at least 3
points

Minpts = 3
Eps=radius
of the
circles
 Density-Reachability
 Directly density-reachable
An object q is directly density-reachable from
object p if p is a core object and q is in p’s -
neighborhood.

 q is directly density-reachable from p

ε ε  p is not directly density- reachable
qq pp from q?
 Density-reachability is asymmetric.

MinPts = 4
 Density-reachability
 Density-Reachable (directly and
indirectly):
 A point p is directly density-reachable from p2;
 p2 is directly density-reachable from p1;
 p1 is directly density-reachable from q;
 pp2p1q form a chain.
p  p is (indirectly) density-reachable
p2 from q
p1  q is not density- reachable from p?
q
MinPts = 7
 Density-Connectivity
Density-reachable is not symmetric
 not good enough to describe clusters

Density-Connected
A pair of points p and q are density-connected
if they are commonly density-reachable from a
point o.
 Density-connectivity is
symmetric
p q

o
 Formal Description of Cluster

 Given a data set D, parameter  and threshold

MinPts.
 A cluster C is a subset of objects satisfying
two criteria:
Connected: ∀ p,q ∈ C: p and q are density-
connected.
Maximal: ∀ p,q: if p ∈ C and q is density-
reachable from p, then q ∈ C. (avoid redundancy)

P is a core object.
 Review of Concepts
Is an object o in a cluster Are objects p and q in the
or an outlier? same cluster?

Are p and q density-

Is o a core object?
connected?

Is o density-reachable by Are p and q density-

some core object? reachable by some
object o?

Directly density- Indirectly density-

reachable reachable through a chain
 DBSCAN Algorithm
Input: The data set D
Parameter: , MinPts
For each object p in D
if p is a core object and not processed then
C = retrieve all objects density-reachable from p
mark all objects in C as processed
report C as a cluster
else mark p as outlier
end if
End For

DBScan Algorithm
DBSCAN: The Algorithm
 Arbitrary select a point p

 Retrieve all points density-reachable from p wrt Eps and

MinPts.
 If p is a core point, a cluster is formed.

 If p is a border point, no points are density-reachable from p

and DBSCAN visits the next point of the database.
 Continue the process until all of the points have been
processed.
DBSCAN Algorithm: Example
Parameter
  = 2 cm
 MinPts = 3

for each o  D do
if o is not yet classified then
if o is a core-object then
collect all objects density-reachable from o
and assign them to a new cluster.
else
assign o to NOISE
DBSCAN Algorithm: Example

Parameter
  = 2 cm
 MinPts = 3

for each o  D do
if o is not yet classified then
if o is a core-object then
collect all objects density-reachable from o
and assign them to a new cluster.
else
assign o to NOISE
 DBSCAN Algorithm: Example

Parameter
  = 2 cm
 MinPts = 3

for each o  D do
if o is not yet classified then
if o is a core-object then
collect all objects density-reachable from o
and assign them to a new cluster.
else
assign o to NOISE
 DBSCAN Algorithm
Eliminate noise points
Perform clustering on the remaining points

18
 MinPts = 5


P1

 C1 P
P C1
C1

1. Check the - 1. Check the unprocessed

neighborhood of p; objects in C
2. If p has less than MinPts 2. If no core object, return C
neighbors then mark p
3. Otherwise, randomly pick
as outlier and continue
up one core object p1,
with the next object
mark p1 as processed,
3. Otherwise mark p as and put all unprocessed
processed and put all neighbors of p1 in cluster
the neighbors in cluster C
C
 


C1
C1





C1 C1
C1
 Example

Original Points Point types: core,

border and outliers

 = 10, MinPts = 4
 When DBSCAN Works Well

Original Points Clusters

• Resistant to Noise
• Can handle clusters of different shapes and sizes
 When DBSCAN Does NOT Work Well

(MinPts=4, Eps=9.92).

Original Points

• Cannot handle Varying

densities
• sensitive to parameters

(MinPts=4, Eps=9.75)
DBSCAN: Sensitive to Parameters
 Determining the Parameters 
and MinPts
 Cluster: Point density higher than specified by  and MinPts
 Idea: use the point density of the least dense cluster in the data
set as parameters – but how to determine this?
 Heuristic: look at the distances to the k-nearest neighbors

p 3-distance(p) :

q 3-distance(q) :

 Function k-distance(p): distance from p to the its k-nearest

neighbor
 k-distance plot: k-distances of all objects, sorted in decreasing
order
DBSCAN: Heuristics for determining
EPS and MinPts

 Idea is that for points in a cluster, their kth nearest

neighbors are at roughly the same distance
 Noise points have the kth nearest neighbor at farther
distance
 So, plot sorted distance of every point to its kth nearest
neighbor (e.g., k=4)

Thus, eps=10

26
 Determining the Parameters 
and MinPts
 Example k-distance plot

3-distance
first „valley“

Objects

„border object“
 Heuristic method:
 Fix a value for MinPts (default: 2  d –1)
 User selects “border object” o from the MinPts-distance plot;
 is set to MinPts-distance(o)
 Determining the Parameters 
and MinPts
Problematic example

A C
F A, B, C
E

B, D, E

3-Distance
G
B‘, D‘, F, G
G1
G3 D1, D2,
D G2 G1, G2, G3
B D’
B’ D1
D2 Objects
 Density Based Clustering:
Discussion
Advantages
Clusters can have arbitrary shape and size
Number of clusters is determined automatically
Can separate clusters from surrounding noise
Can be supported by spatial index structures
Disadvantages
Input parameters may be difficult to determine
In some situations very sensitive to input parameter
setting
 OPTICS: Ordering Points To Identify
the Clustering Structure
DBSCAN
Input parameter – hard to determine.
Algorithm very sensitive to input parameters.
OPTICS – Ankerst, Breunig, Kriegel, and
Sander (SIGMOD’99)
 Based on DBSCAN.
 Does not produce clusters explicitly.
 Rather generate an ordering of data objects
representing density-based clustering structure.
 OPTICS con’t
Produces a special order of the database wrt its
density-based clustering structure
This cluster-ordering contains info equiv to the
density-based clusterings corresponding to a broad
range of parameter settings
Good for both automatic and interactive cluster
analysis, including finding intrinsic clustering
structure
Can be represented graphically or using
visualization techniques
 Density-Based Hierarchical
Clustering
 Observation: Dense clusters are completely contained
by less dense clusters
C D
C1 C2

 Idea: Process objects in the “right” order and keep track of point
density in their neighborhood
C MinPts = 3
C1 C2
2 1
 Core- and Reachability Distance
Parameters: “generating” distance  fixed value
MinPts

core-distance,MinPts(o)
“smallest distance such that o is a core object”
(if that distance is  “?”otherwise)
MinPts = 5
reachability-distance,MinPts(p, o)
“smallest distance such that p is p
directly density-reachable from o” q o 
(if that distance is  “?”otherwise)

core-distance(o)
reachability-distance(p,o)
reachability-distance(q,o)
 OPTICS: Extension of
DBSCAN
 Order points by shortest reachability distance to
guarantee that clusters w.r.t. higher density are finished
first. (for a constant MinPts, higher
density requires lower ε)
 The Algorithm OPTICS
Basic data structure: controlList
Memorize shortest reachability distances seen so far
(“distance of a jump to that point”)
Visit each point
Make always a shortest jump
Output:
order of points
core-distance of points
reachability-distance of points
 The Algorithm OPTICS
ControlList ordered by reachability-distance.
ControlList cluster-ordered
foreach o  Database  file
// initially, o.processed = false for all objects o
if o.processed = false;
insert (o, “?”) into ControlList;
while ControlList is not empty
database
select first element (o, r-dist) from ControlList;
retrieve N(o) and determine c_dist= core-distance(o);
set o.processed = true;
write (o, r_dist, c_dist) to file;
if o is a core object at any distance 
foreach p  N(o) not yet processed;
determine r_distp = reachability-distance(p, o);
if (p, _)  ControlList
insert (p, r_distp) in ControlList;
else if (p, old_r_dist)  ControlList and r_distp  old_r_dist
update (p, r_distp) in ControlList;
 OPTICS: Properties
 “Flat” density-based clusters wrt. * andMinPts afterwards:
 Starts with an object o where c-dist(o)  * and r-dist(o) > *
 Continues while r-dist  *

1
2 17
3 16 18 34
1
4 2 16 17

18

Core-distance
 Performance: approx. Reachability-distance
runtime( DBSCAN( , MinPts) )
 O( n * runtime(-neighborhood-query) )
 without spatial index support (worst case): O( n2 )
 e.g. tree-based spatial index support: O( n  log(n) )
 OPTICS: The Reachability Plot
represents the density-based clustering
structure
easy to analyze
independent of the dimension of the data
reachability distance

reachability distance

cluster ordering cluster ordering

 OPTICS: Parameter Sensitivity
Relatively insensitive to parameter settings
Good result if parameters are just 1 3

“large enough”
2

MinPts = 10,  = 10 MinPts = 10,  = 5 MinPts = 2,  = 10

1 2 3
1 2 3
1 2 3
 An Example of OPTICS
neighboring objects stay close to each other in a linear
sequence.

Reachability-
distance


undefined


 ‘

Cluster-order of the objects

 DBSCAN VS OPTICS

DBSCAN OPTICS
Density Boolean value Numerical value
(high/low) (core distance)
Density- Boolean value Numerical value
connected (yes/no) (reachability distance)
Searching random greedy
strategy
 When OPTICS Works Well

Cluster-order of the objects

 When OPTICS Does NOT
Work Well




Cluster-order of the objects

 DENCLUE: using density
functions
 DENsity-based CLUstEring by Hinneburg & Keim
(KDD’98)
 Major features
 Solid mathematical foundation
 Good for data sets with large amounts of noise
 Allows a compact mathematical description of arbitrarily
shaped clusters in high-dimensional data sets
 Significantly faster than existing algorithm (faster than
DBSCAN by a factor of up to 45)
 But needs a large number of parameters
 Denclue: Technical Essence
 Model density by the notion of influence
 Each data object exert influence on its neighborhood.
 The influence decreases with distance
 Example:
 Consider each object is a radio, the closer you are to the
object, the louder the noise
 Key: Influence is represented by mathematical
function
 Denclue: Technical Essence
Influence functions: (influence of y on x,  is a user given constant)
Square : f ysquare(x) = 0, if dist(x,y) > ,
1, otherwise

Guassian:

d ( x, y )2

y 2 2
f Gaussian ( x) e
 Density Function

 Density Definition is defined as the sum of the

influence functions of all data points.

d ( x , xi ) 2

( x )  i 1 e
D N
2 2
f Gaussian
Gradient: The steepness of a
slope
Example
d ( x , y )2

f Gaussian ( x , y ) e 2 2

d ( x , xi ) 2

( x )  i 1 e
D N
2 2
f Gaussian
d ( x , xi ) 2

( x, xi )  i 1 ( xi  x) e
D N
2 2
f Gaussian
Denclue: Technical Essence
Clusters can be determined mathematically
by identifying density attractors.
Density attractors are local maximum of the
overall density function.
Density Attractor
 Cluster Definition

 Center-defined cluster
 A subset of objects attracted by an attractor x
 density(x) ≥ 
 Arbitrary-shape cluster
 A group of center-defined clusters which are
connected by a path P
 For each object x on P, density(x) ≥ .
Center-Defined and Arbitrary
DENCLUE: How to find the
clusters
Divide the space into grids, with size 2
Consider only grids that are highly
populated
For each object, calculate its density
attractor using hill climbing technique
Tricks can be applied to avoid calculating
density attractor of all points
Density attractors form basis of all clusters
 Features of DENCLUE
 Major features
 Solid mathematical foundation
 Compact definition for density and cluster
 Flexible for both center-defined clusters and arbitrary-
shape clusters
 But needs a large number of parameters
 : parameter to calculate density
 : density threshold
 : parameter to calculate attractor