CS246: Mining Massive Datasets

Jure Leskovec, Stanford University

http://cs246.stanford.edu
High dim.
data
Locality
sensitive
hashing
Clustering
Dimensional
ity
reduction
Graph
data
PageRank,
SimRank
Network
Analysis
Spam
Detection
Infinite
data
Filtering
data
streams
Web
advertising
Queries on
streams
Machine
learning
SVM
Decision
Trees
Perceptron,
kNN
Apps
Recommen
der systems
Association
Rules
Duplicate
document
detection
1/13/2014 Jure Leskovec, Stanford CS246: Mining Massive Datasets, http://cs246.stanford.edu 2
1/13/2014 3 Jure Leskovec, Stanford C246: Mining Massive Datasets
[Hays and Efros, SIGGRAPH 2007]
1/13/2014 4 Jure Leskovec, Stanford C246: Mining Massive Datasets
[Hays and Efros, SIGGRAPH 2007]
10 nearest neighbors from a collection of 20,000 images
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 5
[Hays and Efros, SIGGRAPH 2007]
10 nearest neighbors from a collection of 2 million images
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 6
[Hays and Efros, SIGGRAPH 2007]
Many problems can be expressed as
finding similar sets:
Find near-neighbors in high-dimensional space
Examples:
Pages with similar words
For duplicate detection, classification by topic
Customers who purchased similar products
Products with similar customer sets
Images with similar features
Users who visited similar websites
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 7
Given: High dimensional data points

,
For example: Image is a long vector of pixel colors
1 2 1
0 2 1
0 1 0
[1 2 1 0 2 1 0 1 0]
And some distance function (

)
Which quantifies the distance between

and

Goal: Find all pairs of data points (

) that are
within some distance threshold

Note: Nave solution would take

where is the number of data points
MAGIC: This can be done in !! How?
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 8
Last time: Finding frequent pairs
1/14/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 9
I
t
e
m
s

1

N

Items 1N
Count of pair {i,j}
in the data
Nave solution:
Single pass but requires
space quadratic in the
number of items
I
t
e
m
s

1

K

Items 1K
Count of pair {i,j}
in the data
A-Priori:
First pass: Find frequent singletons
For a pair to be a frequent pair
candidate, its singletons have to be
frequent!
Second pass:
Count only candidate pairs!
N number of distinct items
K number of items with support s
Last time: Finding frequent pairs
Further improvement: PCY
Pass 1:
Count exact frequency of each item:
Take pairs of items {i,j}, hash them into B buckets and
count of the number of pairs that hashed to each bucket:

1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 10
Items 1N
Basket 1: {1,2,3}
Pairs: {1,2}{1,3}{2,3}
Buckets 1B
2 1
Last time: Finding frequent pairs
Further improvement: PCY
Pass 1:
Count exact frequency of each item:
Take pairs of items {i,j}, hash them into B buckets and
count of the number of pairs that hashed to each bucket:
Pass 2:
For a pair {i,j} to be a candidate for
a frequent pair, its singletons {i}, {j}
have to be frequent and the pair
has to hash to a frequent bucket!
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 11
Items 1N
Basket 1: {1,2,3}
Pairs: {1,2}{1,3}{2,3}
Basket 2: {1,2,4}
Pairs: {1,2}{1,4}{2,4}
Buckets 1B
3 1 2
Last time: Finding frequent pairs
Further improvement: PCY
Pass 1:
Count exact frequency of each item:
Take pairs of items {i,j}, hash them into B buckets and
count of the number of pairs that hashed to each bucket:
Pass 2:
For a pair {i,j} to be a candidate for
a frequent pair, its singletons have
to be frequent and its has to hash
to a frequent bucket!
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 12
Items 1N
Basket 1: {1,2,3}
Pairs: {1,2}{1,3}{2,3}
Basket 2: {1,2,4}
Pairs: {1,2}{1,4}{2,4}
Buckets 1B
3 1 2
Previous lecture: A-Priori
Main idea: Candidates
Instead of keeping a count of each pair, only keep a count
of candidate pairs!
Todays lecture: Find pairs of similar docs
Main idea: Candidates
-- Pass 1: Take documents and hash them to buckets such that
documents that are similar hash to the same bucket
-- Pass 2: Only compare documents that are candidates
(i.e., they hashed to a same bucket)
Benefits: Instead of O(N
2
) comparisons, we need O(N)
comparisons to find similar documents
Goal: Find near-neighbors in high-dim. space
We formally define near neighbors as
points that are a small distance apart
For each application, we first need to define
what distance means
Today: Jaccard distance/similarity
The Jaccard similarity of two sets is the size of their
intersection divided by the size of their union:
sim(C
1
, C
2
) = |C
1
C
2
|/|C
1
C
2
|
Jaccard distance: d(C
1
, C
2
) = 1 - |C
1
C
2
|/|C
1
C
2
|


1/13/2014 14 Jure Leskovec, Stanford C246: Mining Massive Datasets
3 in intersection
8 in union
J accard similarity= 3/8
J accard distance = 5/8
Goal: Given a large number ( in the millions or
billions) of documents, find near duplicate pairs
Applications:
Mirror websites, or approximate mirrors
Dont want to show both in search results
Similar news articles at many news sites
Cluster articles by same story
Problems:
Many small pieces of one document can appear
out of order in another
Too many documents to compare all pairs
Documents are so large or so many that they cannot
fit in main memory


1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 15
1. Shingling: Convert documents to sets

2. Min-Hashing: Convert large sets to short
signatures, while preserving similarity

3. Locality-Sensitive Hashing: Focus on
pairs of signatures likely to be from
similar documents
Candidate pairs!
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 16
17
Docu-
ment
The set
of strings
of length k
that appear
in the doc-
ument
Signatures:
short integer
vectors that
represent the
sets, and
reflect their
similarity
Locality-
Sensitive
Hashing
Candidate
pairs:
those pairs
of signatures
that we need
to test for
similarity
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets
Step 1: Shingling: Convert documents to sets
Docu-
ment
The set
of strings
of length k
that appear
in the doc-
ument
Step 1: Shingling: Convert documents to sets

Simple approaches:
Document = set of words appearing in document
Document = set of important words
Dont work well for this application. Why?

Need to account for ordering of words!
A different way: Shingles!

1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 19
A k-shingle (or k-gram) for a document is a
sequence of k tokens that appears in the doc
Tokens can be characters, words or something
else, depending on the application
Assume tokens = characters for examples

Example: k=2; document D
1
= abcab
Set of 2-shingles: S(D
1
) = {ab, bc, ca}
Option: Shingles as a bag (multiset), count ab
twice: S(D
1
) = {ab, bc, ca, ab}

1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 20
To compress long shingles, we can hash them
to (say) 4 bytes
Represent a document by the set of hash
values of its k-shingles
Idea: Two documents could (rarely) appear to have
shingles in common, when in fact only the hash-
values were shared
Example: k=2; document D
1
= abcab
Set of 2-shingles: S(D
1
) = {ab, bc, ca}
Hash the singles: h(D
1
) = {1, 5, 7}
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 21
Document D
1
is a set of its k-shingles C
1
=S(D
1
)
Equivalently, each document is a
0/1 vector in the space of k-shingles
Each unique shingle is a dimension
Vectors are very sparse
A natural similarity measure is the
Jaccard similarity:
sim(D
1
, D
2
) = |C
1
C
2
|/|C
1
C
2
|
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 22
Documents that have lots of shingles in
common have similar text, even if the text
appears in different order

Caveat: You must pick k large enough, or most
documents will have most shingles
k = 5 is OK for short documents
k = 10 is better for long documents
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 23
Suppose we need to find near-duplicate
documents among = million documents

Navely, we would have to compute pairwise
Jaccard similarities for every pair of docs
()/ 5*10
11
comparisons
At 10
5
secs/day and 10
6
comparisons/sec,
it would take 5 days

For = million, it takes more than a year
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 24
Step 2: Minhashing: Convert large sets to
short signatures, while preserving similarity

Docu-
ment
The set
of strings
of length k
that appear
in the doc-
ument
Signatures:
short integer
vectors that
represent the
sets, and
reflect their
similarity
Many similarity problems can be
formalized as finding subsets that
have significant intersection
Encode sets using 0/1 (bit, boolean) vectors
One dimension per element in the universal set
Interpret set intersection as bitwise AND, and
set union as bitwise OR

Example: C
1
= 10111; C
2
= 10011
Size of intersection = 3; size of union = 4,
Jaccard similarity (not distance) = 3/4
Distance: d(C
1
,C
2
) = 1 (Jaccard similarity) = 1/4
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 26
Rows = elements (shingles)
Columns = sets (documents)
1 in row e and column s if and only
if e is a member of s
Column similarity is the Jaccard
similarity of the corresponding
sets (rows with value 1)
Typical matrix is sparse!
Each document is a column:
Example: sim(C
1
,C
2
) = ?
Size of intersection = 3; size of union = 6,
Jaccard similarity (not distance) = 3/6
d(C
1
,C
2
) = 1 (Jaccard similarity) = 3/6

27 1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets
0 1 0 1
0 1 1 1
1 0 0 1
1 0 0 0
1 0 1 0
1 0 1 1
0 1 1 1
Documents
S
h
i
n
g
l
e
s

So far:
Documents Sets of shingles
Represent sets as boolean vectors in a matrix
Next goal: Find similar columns while
computing small signatures
Similarity of columns == similarity of signatures
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 28
Next Goal: Find similar columns, Small signatures
Nave approach:
1) Signatures of columns: small summaries of columns
2) Examine pairs of signatures to find similar columns
Essential: Similarities of signatures and columns are related
3) Optional: Check that columns with similar signatures
are really similar
Warnings:
Comparing all pairs may take too much time: Job for LSH
These methods can produce false negatives, and even false
positives (if the optional check is not made)
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 29
Key idea: hash each column C to a small
signature h(C), such that:
(1) h(C) is small enough that the signature fits in RAM
(2) sim(C
1
, C
2
) is the same as the similarity of
signatures h(C
1
) and h(C
2
)

Goal: Find a hash function h() such that:
If sim(C
1
,C
2
) is high, then with high prob. h(C
1
) = h(C
2
)
If sim(C
1
,C
2
) is low, then with high prob. h(C
1
) h(C
2
)

Hash docs into buckets. Expect that most pairs
of near duplicate docs hash into the same bucket!
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 30
Goal: Find a hash function h() such that:
if sim(C
1
,C
2
) is high, then with high prob. h(C
1
) = h(C
2
)
if sim(C
1
,C
2
) is low, then with high prob. h(C
1
) h(C
2
)

Clearly, the hash function depends on
the similarity metric:
Not all similarity metrics have a suitable
hash function
There is a suitable hash function for
the Jaccard similarity: It is called Min-Hashing
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 31
32
Imagine the rows of the boolean matrix
permuted under random permutation

Define a hash function h

(C) = the index of

the first (in the permuted order ) row in
which column C has value 1:
h

(C) = min

(C)

Use several (e.g., 100) independent hash
functions (that is, permutations) to create a
signature of a column
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets
33
3
4
7
2
6
1
5
Signature matrix M
1 2 1 2
5
7
6
3
1
2
4
1 4 1 2
4
5
1
6
7
3
2
2 1 2 1
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets
2
nd
element of the permutation
is the first to map to a 1
4
th
element of the permutation
is the first to map to a 1
0 1 0 1
0 1 0 1
1 0 1 0
1 0 1 0
1 0 1 0
1 0 0 1
0 1 0 1
Input matrix (Shingles x Documents) Permutation
Note: Another (equivalent) way is to
store row indexes:



1 5 1 5
2 3 1 3
6 4 6 4
Choose a random permutation
Claim: Pr[h

(C
1
) = h

(C
2
)] = sim(C
1
, C
2
)
Why?
Let X be a doc (set of shingles), y X is a shingle
Then: Pr[(y) = min((X))] = 1/|X|
It is equally likely that any y X is mapped to the min element
Let y be s.t. (y) = min((C
1
C
2
))
Then either: (y) = min((C
1
)) if y C
1
, or

(y) = min((C
2
)) if y C
2
So the prob. that both are true is the prob. y C
1
C
2
Pr[min((C
1
))=min((C
2
))]=|C
1
C
2
|/|C
1
C
2
|= sim(C
1
, C
2
)
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 34
0 1
1 0
0 0
1 1
0 0
0 0
One of the two
cols had to have
1 at position y
36
We know: Pr[h

(C
1
) = h

(C
2
)] = sim(C
1
, C
2
)
Now generalize to multiple hash functions

The similarity of two signatures is the
fraction of the hash functions in which they
agree

Note: Because of the Min-Hash property, the
similarity of columns is the same as the
expected similarity of their signatures

1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets
37 1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets
Similarities:
1-3 2-4 1-2 3-4
Col/Col 0.75 0.75 0 0
Sig/Sig 0.67 1.00 0 0
Signature matrix M
1 2 1 2
5
7
6
3
1
2
4
1 4 1 2
4
5
1
6
7
3
2
2 1 2 1
0 1 0 1
0 1 0 1
1 0 1 0
1 0 1 0
1 0 1 0
1 0 0 1
0 1 0 1
Input matrix (Shingles x Documents)
3
4
7
2
6
1
5
Permutation
Pick K=100 random permutations of the rows
Think of sig(C) as a column vector
sig(C)[i] = according to the i-th permutation, the
index of the first row that has a 1 in column C
sig(C)[i] = min (
i
(C))
Note: The sketch (signature) of document C is
small ~ bytes!

We achieved our goal! We compressed
long bit vectors into short signatures
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 38
Permuting rows even once is prohibitive
Row hashing!
Pick K = 100 hash functions k
i

Ordering under k
i
gives a random row permutation!
One-pass implementation
For each column C and hash-func. k
i
keep a slot for
the min-hash value
Initialize all sig(C)[i] =
Scan rows looking for 1s
Suppose row j has 1 in column C

Then for each k
i
:
If k
i
(j) < sig(C)[i], then sig(C)[i] k
i
(j)
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 39
How to pick a random
hash function h(x)?
Universal hashing:
h
a,b
(x)=((ax+b) mod p) mod N
where:
a,b random integers
p prime number (p >N)

Step 3: Locality-Sensitive Hashing:
Focus on pairs of signatures likely to be from
similar documents
Docu-
ment
The set
of strings
of length k
that appear
in the doc-
ument
Signatures:
short integer
vectors that
represent the
sets, and
reflect their
similarity
Locality-
Sensitive
Hashing
Candidate
pairs:
those pairs
of signatures
that we need
to test for
similarity
Goal: Find documents with Jaccard similarity at
least s (for some similarity threshold, e.g., s=0.8)

LSH General idea: Use a function f(x,y) that
tells whether x and y is a candidate pair: a pair
of elements whose similarity must be evaluated

For Min-Hash matrices:
Hash columns of signature matrix M to many buckets
Each pair of documents that hashes into the
same bucket is a candidate pair
1/13/2014 41 Jure Leskovec, Stanford C246: Mining Massive Datasets
1 2 1 2
1 4 1 2
2 1 2 1
Pick a similarity threshold s (0 < s < 1)

Columns x and y of M are a candidate pair if
their signatures agree on at least fraction s of
their rows:
M (i, x) = M (i, y) for at least frac. s values of i
We expect documents x and y to have the same
(Jaccard) similarity as their signatures

1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 42
1 2 1 2
1 4 1 2
2 1 2 1
Big idea: Hash columns of
signature matrix M several times

Arrange that (only) similar columns are
likely to hash to the same bucket, with
high probability

Candidate pairs are those that hash to
the same bucket
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 43
1 2 1 2
1 4 1 2
2 1 2 1
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 44
Signature matrix M
r rows
per band
b bands
One
signature
1 2 1 2
1 4 1 2
2 1 2 1
Divide matrix M into b bands of r rows

For each band, hash its portion of each
column to a hash table with k buckets
Make k as large as possible

Candidate column pairs are those that hash
to the same bucket for 1 band

Tune b and r to catch most similar pairs,
but few non-similar pairs
45 1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets
Matrix M
r rows
b bands
Buckets
Columns 2 and 6
are probably identical
(candidate pair)
Columns 6 and 7 are
surely different.
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 46
There are enough buckets that columns are
unlikely to hash to the same bucket unless
they are identical in a particular band

Hereafter, we assume that same bucket
means identical in that band

Assumption needed only to simplify analysis,
not for correctness of algorithm
1/13/2014 47 Jure Leskovec, Stanford C246: Mining Massive Datasets
Assume the following case:
Suppose 100,000 columns of M (100k docs)
Signatures of 100 integers (rows)
Therefore, signatures take 40Mb
Choose b = 20 bands of r = 5 integers/band

Goal: Find pairs of documents that
are at least s = 0.8 similar
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 48
1 2 1 2
1 4 1 2
2 1 2 1
Find pairs of s=0.8 similarity, set b=20, r=5
Assume: sim(C
1
, C
2
) = 0.8
Since sim(C
1
, C
2
) s, we want C
1
, C
2
to be a candidate
pair: We want them to hash to at least 1 common bucket
(at least one band is identical)
Probability C
1
, C
2
identical in one particular
band: (0.8)
5
= 0.328
Probability C
1
, C
2
are not similar in all of the 20
bands: (1-0.328)
20
= 0.00035
i.e., about 1/3000th of the 80%-similar column pairs
are false negatives (we miss them)
We would find 99.965% pairs of truly similar documents
1/13/2014 49 Jure Leskovec, Stanford C246: Mining Massive Datasets
1 2 1 2
1 4 1 2
2 1 2 1
Find pairs of s=0.8 similarity, set b=20, r=5
Assume: sim(C
1
, C
2
) = 0.3
Since sim(C
1
, C
2
) < s we want C
1
, C
2
to hash to NO
common buckets (all bands should be different)
Probability C
1
, C
2
identical in one particular
band: (0.3)
5
= 0.00243
Probability C
1
, C
2
identical in at least 1 of 20
bands: 1 - (1 - 0.00243)
20
= 0.0474
In other words, approximately 4.74% pairs of docs
with similarity 0.3% end up becoming candidate pairs
They are false positives since we will have to examine them
(they are candidate pairs) but then it will turn out their
similarity is below threshold s
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 50
1 2 1 2
1 4 1 2
2 1 2 1
Pick:
The number of Min-Hashes (rows of M)
The number of bands b, and
The number of rows r per band
to balance false positives/negatives

Example: If we had only 15 bands of 5
rows, the number of false positives would
go down, but the number of false negatives
would go up
1/13/2014 51 Jure Leskovec, Stanford C246: Mining Massive Datasets
1 2 1 2
1 4 1 2
2 1 2 1
Similarity t =sim(C
1
, C
2
) of two sets
Probability
of sharing
a bucket
S
i
m
i
l
a
r
i
t
y

t
h
r
e
s
h
o
l
d

s

No chance
if t < s
Probability = 1
if t > s
1/13/2014 52 Jure Leskovec, Stanford C246: Mining Massive Datasets
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 53
Remember:
Probability of
equal hash-values
= similarity
Similarity t =sim(C
1
, C
2
) of two sets
Probability
of sharing
a bucket
Columns C
1
and C
2
have similarity t
Pick any band (r rows)
Prob. that all rows in band equal = t
r

Prob. that some row in band unequal = 1 - t
r


Prob. that no band identical = (1 - t
r
)
b

Prob. that at least 1 band identical =
1 - (1 - t
r
)
b


1/13/2014 54 Jure Leskovec, Stanford C246: Mining Massive Datasets
t
r
All rows
of a band
are equal
1 -
Some row
of a band
unequal
( )
b

No bands
identical
1 -
At least
one band
identical
s ~ (1/b)
1/r
1/13/2014 55 Jure Leskovec, Stanford C246: Mining Massive Datasets
Similarity t=sim(C
1
, C
2
) of two sets
Probability
of sharing
a bucket
Similarity threshold s
Prob. that at least 1 band is identical:
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 56
s 1-(1-s
r
)
b
.2 .006
.3 .047
.4 .186
.5 .470
.6 .802
.7 .975
.8 .9996
Picking r and b to get the best S-curve
50 hash-functions (r=5, b=10)
1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 57
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Blue area: False Negative rate
Green area: False Positive rate
Similarity
P
r
o
b
.

s
h
a
r
i
n
g

a

b
u
c
k
e
t

Tune M, b, r to get almost all pairs with
similar signatures, but eliminate most pairs
that do not have similar signatures

Check in main memory that candidate pairs
really do have similar signatures

Optional: In another pass through data,
check that the remaining candidate pairs
really represent similar documents
1/13/2014 58 Jure Leskovec, Stanford C246: Mining Massive Datasets
Shingling: Convert documents to sets
We used hashing to assign each shingle an ID
Min-Hashing: Convert large sets to short
signatures, while preserving similarity
We used similarity preserving hashing to generate
signatures with property Pr[h

(C
1
) = h

(C
2
)] = sim(C
1
, C
2
)
We used hashing to get around generating random
permutations
Locality-Sensitive Hashing: Focus on pairs of
signatures likely to be from similar documents
We used hashing to find candidate pairs of similarity s

1/13/2014 Jure Leskovec, Stanford C246: Mining Massive Datasets 59