1

MS (Data Science)
Fall 2024 Semester

CT-530 DATA MINING

Dr. Sohail Abdul Sattar

Ex Professor & Co-chairman
Department of Computer Science & Information
Technology
NED University of Engineering & Technology
 2

Course Teacher

Dr. Sohail Abdul Sattar

Ex Professor & Co-Chairman
Department of Computer Science & Information
Technology
NED University

Professor
Ex Dean & Chairman
Nazeer Hussain University, Karachi

PhD Computer Vision NED + UCF (Orlando, US)

MS Comp. Science NED
MCS Comp. Science KU
BE Mech. Engg. NED
 3

Books
• “Introduction to Data Mining” by Tan, Steinbach, Kumar.

• Mining Massive Datasets by Anand Rajaraman, Jeff Ullman, and Jure Leskovec. Free online
book. Includes slides from the course

• “Data Mining: Concepts and Techniques”, by Jiawei Han and Micheline Kambe

• “Data Mining: Practical Machine Learning Tools and Techniques”

by Ian H. Witten
 4

Thanks to the owner of these slides !!!

 5

DATA MINING
SIMILARITY & DISTANCE
Similarity and Distance
Recommender Systems
 6

SIMILARITY AND DISTANCE

Thanks to:
Tan, Steinbach, and Kumar, “Introduction to Data Mining”
Rajaraman and Ullman, “Mining Massive Datasets”
 7

Similarity and Distance

• For many different problems we need to quantify how close two objects
are.
• Examples:
• For an item bought by a customer, find other similar items
• Group together the customers of a site so that similar customers are shown the
same ad.
• Group together web documents so that you can separate the ones that talk about
politics and the ones that talk about sports.
• Find all the near-duplicate mirrored web documents.
• Find credit card transactions that are very different from previous transactions.
• To solve these problems we need a definition of similarity, or distance.
• The definition depends on the type of data that we have
 8

Similarity
• Numerical measure of how alike two data objects are.
• A function that maps pairs of objects to real values
• Higher when objects are more alike.
• Often falls in the range [0,1], sometimes in [-1,1]

• Desirable properties for similarity

1. s(p, q) = 1 (or maximum similarity) only if p = q. (Identity)
2. s(p, q) = s(q, p) for all p and q. (Symmetry)
 9

Similarity between sets

• Consider the following documents

apple apple new

releases releases apple pie
new ipod new ipad recipe

• Which ones are more similar?

• How would you quantify their similarity?

 10

Similarity: Intersection
• Number of words in common

apple apple new

releases releases apple pie
new ipod new ipad recipe

• Sim(D,D) = 3, Sim(D,D) = Sim(D,D) =2

• What about this document?

Vefa releases new book

with apple pie recipes
• Sim(D,D) = Sim(D,D) =
 11

Similarity: Intersection
• Number of words in common

apple apple new

releases releases apple pie
new ipod new ipad recipe

• Sim(D,D) = 3, Sim(D,D) = Sim(D,D) =2

• What about this document?

Vefa releases new book

with apple pie recipes
• Sim(D,D) = Sim(D,D) = 3
 12

Jaccard Similarity
• The Jaccard similarity (Jaccard coefficient) of two sets S1, S2 is
the size of their intersection divided by the size of their union.
• JSim (S1, S2) = |S1S2| / |S1S2|.

? in intersection.
? in union.
Jaccard similarity
= ?/?

• Extreme behavior:
• Jsim(X,Y) = 1, iff X ? Y
• Jsim(X,Y) = 0 iff X,Y have ? elements in common
• JSim is symmetric
 13

Jaccard Similarity
• The Jaccard similarity (Jaccard coefficient) of two sets S1, S2 is
the size of their intersection divided by the size of their union.
• JSim (S1, S2) = |S1S2| / |S1S2|.

3 in intersection.
8 in union.
Jaccard similarity
= 3/8

• Extreme behavior:
• Jsim(X,Y) = 1, iff X ? Y
• Jsim(X,Y) = 0 iff X,Y have ? elements in common
• JSim is symmetric
 14

Jaccard Similarity
• The Jaccard similarity (Jaccard coefficient) of two sets S1, S2 is
the size of their intersection divided by the size of their union.
• JSim (S1, S2) = |S1S2| / |S1S2|.

3 in intersection.
8 in union.
Jaccard similarity
= 3/8

• Extreme behavior:
• Jsim(X,Y) = 1, iff X = Y
• Jsim(X,Y) = 0 iff X,Y have no elements in common
• JSim is symmetric
 15

Jaccard Similarity between sets

• The distance for the documents

apple apple new Vefa releases

releases releases apple pie new book with
new ipod new ipad recipe apple pie
recipes

• JSim(D,D) = ?
• JSim(D,D) = JSim(D,D) = ?
• JSim(D,D) = JSim(D,D) = ?
 16

Jaccard Similarity between sets

• The distance for the documents

apple apple new Vefa releases

releases releases apple pie new book with
new ipod new ipad recipe apple pie
recipes

• JSim(D,D) = 3/5
• JSim(D,D) = JSim(D,D) = 2/6
• JSim(D,D) = JSim(D,D) = 3/9
 17

Similarity between vectors

Documents (and sets in general) can also be represented as vectors
document Apple Microsoft Obama Election
D1 10 20 0 0
D2 30 60 0 0
D3 60 30 0 0
D4 0 0 10 20

How do we measure the similarity of two vectors?

• We could view them as sets of words. Jaccard Similarity will show that
D4 is different form the rest
• But all pairs of the other three documents are equally similar

We want to capture how well the two vectors are aligned

 18

Example
document Apple Microsoft Obama Election
D1 10 20 0 0
D2 30 60 0 0
D3 60 30 0 0
D4 0 0 10 20

apple

microsoft

{Obama, election}
 19

Example
document Apple Microsoft Obama Election
D1 10 20 0 0
D2 30 60 0 0
D3 60 30 0 0
D4 0 0 10 20

apple
Documents D1, D2 are in the “same direction”

Document D3 is on the same plane as D1, D2

microsoft
Document D4 is orthogonal to the rest
{Obama, election}
 20

Example
document Apple Microsoft Obama Election
D1 10 20 0 0
D2 30 60 0 0
D3 60 30 0 0
D4 0 0 10 20

apple
Documents D1, D2 are in the “same direction”

Document D3 is on the same plane as D1, D2

microsoft
Document D4 is orthogonal to the rest
{Obama, election}
 21

Cosine Similarity

• Sim(X,Y) = cos(X,Y)
• The cosine of the angle between X and Y

• If the vectors are aligned (correlated) angle is zero degrees and

cos(X,Y)=1
• If the vectors are orthogonal (no common coordinates) angle is 90
degrees and cos(X,Y) = 0

• Cosine is commonly used for comparing documents, where we

assume that the vectors are normalized by the document length.
 22

Cosine Similarity - math

• If d and d are two vectors, then
1 2

cos( d1, d2 ) = (d1  d2) / ||d1|| ||d2|| ,

where  indicates vector dot product and || d || is the length of vector d.

• Example:
Note: We only need to
d1 = 3 2 0 5 0 0 0 2 0 0 consider the non-zero
d2 = 1 0 0 0 0 0 0 1 0 2 entries of the vectors

d1  d2= 3*1 + 2*0 + 0*0 + 5*0 + 0*0 + 0*0 + 0*0 + 2*1 + 0*0 + 0*2 = 5

||d1|| = (3*3+2*2+0*0+5*5+0*0+0*0+0*0+2*2+0*0+0*0)0.5 = (42) 0.5 = 6.481

||d2|| = (1*1+0*0+0*0+0*0+0*0+0*0+0*0+1*1+0*0+2*2) 0.5 = (6) 0.5 = 2.245

cos( d1, d2 ) = .3150

 23

Example
document Apple Microsoft Obama Election
D1 10 20 0 0
D2 30 60 0 0
D3 60 30 0 0
D4 0 0 10 20

apple

microsoft

{Obama, election}
 24

Example
document Apple Microsoft Obama Election
D1 10 20 0 0
D2 30 60 0 0
D3 60 30 0 0
D4 0 0 10 20

apple
Cos(D1,D2) = ?

Cos (D3,D1) = Cos(D3,D2) = ?

microsoft
Cos(D4,D1) = Cos(D4,D2) = Cos(D4,D3) = ?
{Obama, election}
 25

Example
document Apple Microsoft Obama Election
D1 10 20 0 0
D2 30 60 0 0
D3 60 30 0 0
D4 0 0 10 20

apple
Cos(D1,D2) = 1

Cos (D3,D1) = Cos(D3,D2) = 4/5

microsoft
Cos(D4,D1) = Cos(D4,D2) = Cos(D4,D3) = 0
{Obama, election}
 26

Correlation Coefficient
• The correlation coefficient measures correlation between two random
variables.
• If we have observations (vectors) and is defined as

• This is essentially the cosine similarity between the normalized

vectors (where from each entry we remove the mean value of the
vector.
• The correlation coefficient takes values in [-1,1]
• -1 negative correlation, +1 positive correlation, 0 no correlation.
 27

Example
document Apple Microsoft Obama Election
D1 10 20 0 0
D2 30 60 0 0
D3 60 30 0 0
D4 0 0 10 20

apple

microsoft

{Obama, election}
 28

Correlation Coefficient
Normalized vectors
document Apple Microsoft Obama Election
D1 -5 +5 0 0
D2 -15 +15 0 0
D3 +15 -15 0 0
D4 0 0 -5 +5

CorrCoeff(D1,D2) = ?
CorrCoeff(D1,D3) = CorrCoeff(D2,D3) = ?
CorrCoeff(D1,D4) = CorrCoeff(D2,D4) = CorrCoeff(D3,D4) = ?
 29

Correlation Coefficient
Normalized vectors
document Apple Microsoft Obama Election
D1 -5 +5 0 0
D2 -15 +15 0 0
D3 +15 -15 0 0
D4 0 0 -5 +5

CorrCoeff(D1,D2) = 1
CorrCoeff(D1,D3) = CorrCoeff(D2,D3) = -1
CorrCoeff(D1,D4) = CorrCoeff(D2,D4) = CorrCoeff(D3,D4) = 0
 30

Distance
• Numerical measure of how different two data objects are
• A function that maps pairs of objects to real values
• Lower when objects are more alike
• Higher when two objects are different
• Minimum distance is 0, when comparing an object with itself.
• Upper limit varies
 31

Distance Metric
• A distance function d is a distance metric if it is a function from
pairs of objects to real numbers such that:
1. . (non-negativity)
2. iff . (identity)
3. . (symmetry)
4. (triangle inequality ).
 32

Triangle Inequality
• Triangle inequality guarantees that the distance function is well-
behaved.
• The direct connection is the shortest distance

• It is useful also for proving useful properties about the data.

 33

Distances for real vectors

• Vectors and
Lp norms are known to be distance metrics
• -norms or Minkowski distance:

• -norm: Euclidean distance:

• -norm: Manhattan distance:

• -norm:

• The limit of as p goes to infinity.

 34

Example of Distances
y = (9,8)
-norm:

5 3
-norm:
4
x = (5,5)

-norm:
 35

Example

𝑥=(𝑥 1 , … , 𝑥 𝑛 )

Green: All points y at distance from point

Blue: All points y at distance from point

Red: All points y at distance from point

 36

Similarities into distances

• Jaccard distance:

• Jaccard Distance is a metric

• Cosine distance:

• Cosine distance is a metric

 38

Hamming Distance
• Hamming distance is the number of positions in which bit-vectors
differ.
• Example:
• p1 = 10101
• p2 = 10011.
• because the bit-vectors differ in the 3 rd and 4th positions.

• Hamming distance between two vectors of categorical attributes is

the number of positions in which they differ.
• Example:
• x = (married, low income, cheat)
• y = (single, low income, not cheat)
 39

Why Hamming Distance Is a Distance Metric

• d(x,x) = 0 since no positions differ.

• d(x,y) = d(y,x) by symmetry of “different from.”
• d(x,y) > 0 since strings cannot differ in a negative number of
positions.
• Triangle inequality: changing x to z and then to y is one way to
change x to y.
 40

Distance between strings

• How do we define similarity between strings?

weird wierd
intelligent unintelligent
Athena Athina

• Important for recognizing and correcting typing errors and

analyzing DNA sequences.
 41

Edit Distance for strings

• The edit distance of two strings is the number of inserts and
deletes of characters needed to turn one into the other.
• Example: x = abcde ; y = bcduve.
• Turn x into y by deleting a, then inserting u and v after d.
• Edit distance = 3.
• Minimum number of operations can be computed using
dynamic programming
• Common distance measure for comparing DNA sequences
 42

Why Edit Distance Is a Distance Metric

• d(x,x) = 0 because 0 edits suffice.

• d(x,y) = d(y,x) because insert/delete are inverses of each other.
• d(x,y) > 0: no notion of negative edits.
• Triangle inequality: changing x to z and then to y is one way to
change x to y. The minimum is no more than that
 43

Variant Edit Distances

• Allow insert, delete, and mutate.

• Change one character into another.
• Minimum number of inserts, deletes, and mutates also forms a
distance measure.

• Same for any set of operations on strings.

• Example: substring reversal or block transposition OK for DNA sequences
• Example: character transposition is used for spelling
 44

Distance between sets of points

How do we measure the distance between the two sets?
 45

Distance between sets of points

How do we measure the distance between the two sets?

Minimum distance over all pairs

 46

Distance between sets of points

How do we measure the distance between the two sets?

Minimum distance over all pairs

Maximum distance over all pairs
 47

Distance between sets of points

How do we measure the distance between the two sets?

Minimum distance over all pairs

Maximum distance over all pairs
Average distance over all pairs
 48

Distance between sets of points

How do we measure the distance between the two sets?

Minimum distance over all pairs

Maximum distance over all pairs
Average distance over all pairs
Distance between averages
 49

Distance between sets of points

How do we measure the distance between the two sets?

Minimum distance over all pairs

Maximum distance over all pairs
Average distance over all pairs
Distance between averages
 50

Distance between sets of points

How do we measure the distance between the two sets?

Minimum distance over all pairs

Maximum distance over all pairs
Average distance over all pairs
Distance between averages
 51

Distance between sets of points

How do we measure the distance between the two sets?

Minimum distance over all pairs

Maximum distance over all pairs
Average distance over all pairs
Distance between averages
 52

Distances between distributions

• Some times data can be represented as a distribution (e.g., a
document is a distribution over the words)
document Apple Microsoft Obama Election
D1 0.35 0.5 0.1 0.05
D2 0.4 0.4 0.1 0.1
D3 0.05 0.05 0.6 0.3

• How do we measure distance between distributions?

 53

Variational distance
• Variational distance: The distance between the distribution vectors
document Apple Microsoft Obama Election
Dist(D1,D2) = 0.05+0.1+0.05 = 0.2
D1 0.35 0.5 0.1 0.05
D2 0.4 0.4 0.1 0.1 Dist(D2,D3) = 0.35+0.35+0.5+ 0.2 = 1.4
D3 0.05 0.05 0.6 0.3
Dist(D1,D3) = 0.3+0.45+0.5+ 0.25 = 1.5
0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
Apple Microsoft Obama Election

D1 D2 D3
 54

APPLICATIONS OF SIMILARITY:
RECOMMENDATION SYSTEMS
 55

An important problem
• Recommendation systems
• When a user buys an item (initially books) we want to recommend other
items that the user may like
• When a user rates a movie, we want to recommend movies that the user
may like
• When a user likes a song, we want to recommend other songs that they
may like

• A big success of data mining

• E.g. Amazon, Netflix
• How Into Thin Air made Touching the Void popular
 56

How Into Thin Air made Touching the Void popular

 57

How Into Thin Air made Touching the Void popular

 58

How Into Thin Air made Touching the Void popular

 59

Utility (Preference) Matrix

Harry Harry Harry Twilight Star Star Star

Potter 1 Potter 2 Potter 3 Wars 1 Wars 2 Wars 3
A 4 5 1
B 5 5 4
C 2 4 5
D 3 3

Rows: Users
Columns: Movies (in general Items)
Values: The rating of the user for the movie

How can we fill the empty entries of the matrix?

 60

Recommendation Systems
• Content-based:
• Represent the items into a feature space and recommend items to customer
C similar to previous items rated highly by C
• Movie recommendations: recommend movies with same actor(s), director, genre, …
• Websites, blogs, news: recommend other sites with “similar” content
 61

Content-based prediction

Harry Harry Harry Twilight Star Star Star

Potter 1 Potter 2 Potter 3 Wars 1 Wars 2 Wars 3
A 4 5 1
B 5 5 4
C 2 4 5
D 3 3

Someone who likes one of the Harry Potter (or Star Wars)
movies is likely to like the rest
• Same actors, similar story, same genre
 62

Intuition
Item profiles

likes

build
recommend

match Red
Circles
Triangles
User profile
 63

Approach
• Map items into a feature space:
• For movies:
• Actors, directors, genre, rating, year,…
• Challenge: make all features compatible.
• For documents?

• To compare items with users we need to map users to the same feature
space. How?
• Take all the movies that the user has seen and take the average vector
• Other aggregation functions are also possible.

• Recommend to user C the most similar item i computing similarity in the

common feature space
• Distributional distance measures also work well.
 64

Limitations of content-based approach

• Finding the appropriate features
• e.g., images, movies, music
• Embeddings and deep learning can help
• Overspecialization
• Never recommends items outside user’s content profile
• People might have multiple interests
• Recommendations for new users
• How to build a profile?
 65

Collaborative filtering

Harry Harry Harry Twilight Star Star Star

Potter 1 Potter 2 Potter 3 Wars 1 Wars 2 Wars 3
A 4 5 1
B 5 5 4
C 2 4 5
D 3 3

Two users are similar if they rate the same items in a similar way

Recommend to user C, the items

liked by many of the most similar users.
 66

User Similarity

Harry Harry Harry Twilight Star Star Star

Potter 1 Potter 2 Potter 3 Wars 1 Wars 2 Wars 3
A 4 5 1
B 5 5 4
C 2 4 5
D 3 3

Which pair of users do you consider as the most similar?

What is the right definition of similarity?

 67

User Similarity

Harry Harry Harry Twilight Star Star Star

Potter 1 Potter 2 Potter 3 Wars 1 Wars 2 Wars 3
A 1 1 1
B 1 1 1
C 1 1 1
D 1 1

Jaccard Similarity: users are sets of movies

Disregards the ratings.
Jsim(A,B) = 1/5
Jsim(A,C) = 1/2
Jsim(B,D) = 1/4
 68

User Similarity

Harry Harry Harry Twilight Star Star Star

Potter 1 Potter 2 Potter 3 Wars 1 Wars 2 Wars 3
A 4 5 1
B 5 5 4
C 2 4 5
D 3 3

Cosine Similarity:
Assumes zero entries are negatives:
Cos(A,B) = 0.38
Cos(A,C) = 0.32
 69

The Netflix Challenge

• 1M prize to improve the prediction accuracy by 10%
 70

Similarity Measures in Data Science

• Introduction to Similarity Measures

• `Definition: Similarity measures are metrics used to quantify how

similar two objects or data points are.

• Importance in Data Science:

Used in clustering, classification, recommendation systems, etc.
Helps to understand the relationships between data points in
different spaces (e.g., Euclidean, vector).
 71

Types of Similarity Measures

• We will cover three major types of similarity measures:
• Distance-based: Euclidean, Manhattan, Minkowski
• Set-based: Jaccard, Cosine
• Correlation-based: Pearson correlation
 72

Distance-Based Similarity Measures

Euclidean Distance
Formula:
 73

• Example:
Consider two users in a movie recommendation system where their
ratings of 5 movies are recorded. The ratings are as follows:
User A: [5, 3, 4, 4, 2]
User B: [4, 2, 5, 3, 1]
We want to calculate the Euclidean distance between these two
users.
 74

• Solution:
 75

Manhattan Distance
Formula:
 76

• Example:
• Consider two users in a movie recommendation system where
their ratings of 5 movies are recorded. The ratings are as follows:
• User A: [5, 3, 4, 4, 2]
• User B: [4, 2, 5, 3, 1]
• We want to calculate the Manhatan distance between these two
users.
 77

• Solution:
 78

Minkowski Distance
Formula:
`
79
 80

• Example:
• Consider two users in a movie recommendation system where
their ratings of 5 movies are recorded. The ratings are as follows:
• User A: [5, 3, 4, 4, 2]
• User B: [4, 2, 5, 3, 1]
• We want to calculate the Minskowisky distance between these
two users.
81
 82

Set-Based Similarity Measures

Jaccard Similarity
Formula:
83
 84

• Example:
• Let's calculate the Jaccard similarity between two sets of movie
genres watched by two users:
85
86
 87

Cosine Similarity
Formula:
88
89
90
91
 92

Correlation-Based Similarity Measures

Pearson Correlation
Formula:
93
 94

• Example:
• Now, let’s calculate the Pearson correlation for two users’ movie
ratings over 4 movies:
• User A’s ratings: [4, 3, 5, 1]
• User B’s ratings: [5, 2, 4, 1]
• Solution:
95
96
 97

• Distance-based measures (Euclidean, Manhattan) work well with

numerical data.
• Set-based measures (Jaccard, Cosine) are effective with binary or
vector data.
• Pearson correlation measures the strength of linear relationships.
 98

Euclidean and Vector Spaces

• Euclidean Space
• Definition: A Euclidean space is a geometric space that extends
the intuitive concepts of distance and angles to more dimensions. It
generalizes 2D and 3D spaces to n-dimensions.
• Key Characteristics:
• Contains points, lines, and planes.
• Distances and angles between points are well-defined using the Euclidean
distance formula.
• It obeys familiar geometric principles, such as the Pythagorean theorem.
99
 100

• Properties:
• The norm (magnitude) of a vector in Euclidean space is given by
the square root of the sum of the squares of its components.
• Angles between vectors can be calculated using geometric
concepts like the dot product.
 101

• Vector Space
• Definition: A vector space is a more abstract algebraic structure
that consists of vectors and operations like vector addition and
scalar multiplication. Vector spaces do not inherently have the
notions of distance or angles; they rely on algebraic rules instead.
• Key Characteristics:
• Defined by vectors (elements that can be added and multiplied by scalars).
• No direct geometric interpretation unless a metric is defined (e.g., through
inner products or norms).
• Can exist in any dimension, even infinite.
102
103
 104

• Relation Between Euclidean and Vector Spaces

• Euclidean spaces are a specific type of vector space with
additional structure (distance and angles). In Euclidean space,
vectors represent points in space, and geometric concepts like
lengths and angles are well-defined.
• A vector space can be generalized and doesn't require these
geometric interpretations unless an inner product or norm is
introduced, which can transform it into a Euclidean-like structure.
 105

• Example of Both Concepts: