CP610 Data Analysis

- Mining Frequent Patterns and

Associations: Basic Concepts
Mining Frequent Patterns, Association and Correlations:
Basic Concepts and Methods

• Basic Concepts

• Frequent Itemset Mining Methods

• Which Patterns Are Interesting?—Pattern Evaluation

Methods

• Summary

2
What Is Frequent Pattern
Analysis?
• Frequent pattern: a pattern (a set of items,
subsequences, substructures, etc.) that occurs
frequently in a data set
• First proposed by Agrawal, Imielinski, and Swami
[AIS93] in the context of frequent itemsets and
association rule mining
A simple example from market
basket analysis
• Five transactions in a supermarket:

• Given these transactions, how to find frequent

itemsets and association rules for potential
purchase behaviour?
• Suppose there is a set I = {𝑖! …, 𝑖" } of m items and
a set D = {𝑡! …, 𝑡# } with n transactions. Each
transaction has a unique identification (ID) number
and consists of a subset of the items in I.
• Supermarket example:
• Items I = {milk, bread, butter, beer}
• Transaction database D =
Definitions
• An association rule is defined as an implication X⇒Y,
where X , Y ⊆ I and X ∩ Y = ∅. X and Y are therefore sets of
items, so-called itemsets.
• An itemset that contains k items is called a k-itemset. X is
called the antecedent and Y the consequent.
• Supermarket example: one possible rule would be
{milk, bread} ⇒ {butter}.
Interpretation: if a customer buys milk and bread, he/she is
likely to also buy butter.
The set of all possible rules
• Power set of set I

• How to find interesting associations? Support and

Conference
Support
• The support count of an itemset X is the number of
transactions that contain the itemset X.
• The (relative) support of an itemset X, supp(X), is defined as
the proportion of transactions in the database that contain
the itemset X (the relative frequency of the itemset).
• The support of a rule supp(X ⇒ Y ) is the support of the joint
itemsets X and Y : supp(X ∪ Y ) (the relative frequency of the
transactions to which the rule can be applied (= P(X ∪ Y )).

Ex. Consider the rule {milk, bread} ⇒ {butter}.

Since the support count for
{milk, bread, butter} is 1 and |D| = 5, the
support of the rule is
supp({milk, bread, butter}) = 0.2
Confidence
• The confidence determines how frequently items in Y appear in
transactions that contain X. Confidence measures the strength of
a rule.
• The confidence of a rule X ⇒ Y is defined as

which is the relative frequency of transactions containing Y

that also contain X (= P(Y |X )).

Ex. Consider again the rule {milk, bread} ⇒ {butter}.

Since supp({milk, bread, butter}) = 0.2 and
supp({milk, bread}) = 0.4, conf({milk, bread} ⇒
{butter}) = 0.5.
Association analysis
• Association analysis is concerned with finding strong
association rules.
• Problem

• Generating strong rules

• Define a minimum support threshold s and a minimum
confidence threshold c.
• Find all itemsets having support ≥ s. These itemsets are called
frequent itemsets.
• From the frequent itemsets found in the previous step extract
all rules having confidence ≥ c. These rules are called strong
rules.
 Closed Patterns and Max-Patterns
• A long itemset contains a combinatorial number of sub-
itemsets, e.g., {a1, …, a100} contains (1001) + (1002) + … + (110000) =
2100 – 1 = 1.27*1030 sub-itemsets!
• Solution: Mine closed itemset and max-itemset instead
• An itemset X is closed if X is frequent and there exists no super-
itemset Y ‫ כ‬X, with the same support as X (proposed by Pasquier,
et al. @ ICDT’99)
• An itemset X is a max-itemset if X is frequent and there exists no
frequent super-itemset Y ‫ כ‬X and Y is frequent (proposed by
Bayardo @ SIGMOD’98)
• Closed itemsets is a lossless compression of freq. patterns
• Reducing the # of itemsets and rules
13
 Closed Patterns and Max-Patterns
• Exercise. DB = {<a1, …, a100>, < a1, …, a50>}
• Min_sup = 1.
• What is the set of closed itemset?
• <a1, …, a100>: 1
• < a1, …, a50>: 2
• What is the set of max-itemset?
• <a1, …, a100>: 1
• What is the set of all itemset?
• !!

14
Mining Frequent Patterns, Association and Correlations:
Basic Concepts and Methods

• Basic Concepts

• Frequent Itemset Mining Methods

• Which Patterns Are Interesting?—Pattern Evaluation

Methods

• Summary

15
Scalable Frequent Itemset Mining Methods

• Apriori: A Candidate Generation-and-Test Approach

• Improving the Efficiency of Apriori

• FPGrowth: A Frequent Pattern-Growth Approach

• ECLAT: Frequent Pattern Mining with Vertical Data

Format
16
The Downward Closure Property and Scalable
Mining Methods
• The downward closure property of frequent patterns
• Any subset of a frequent itemset must be frequent
• If {beer, diaper, nuts} is frequent, so is {beer, diaper}
• i.e., every transaction having {beer, diaper, nuts} also contains
{beer, diaper}
• Scalable mining methods: Three major approaches
• Apriori (Agrawal & Srikant@VLDB’94)
• Freq. pattern growth (FPgrowth—Han, Pei & Yin
@SIGMOD’00)
• Vertical data format approach (Charm—Zaki & Hsiao
@SDM’02)

17
Apriori: A Candidate Generation & Test Approach

• Apriori pruning principle: If there is any itemset which is

infrequent, its superset should not be generated/tested! (Agrawal
& Srikant @VLDB’94, Mannila, et al. @ KDD’ 94)
• Method:
• Initially, scan DB once to get frequent 1-itemset (k=1)
• Iteratively
• Generate length (k+1) candidate itemsets from length k frequent
itemsets
• Test the candidates against DB
• Terminate when no frequent or candidate set can be generated

18
 The Apriori Algorithm—An Example
Supmin = 2 Itemset sup
Itemset sup
Database TDB {A} 2
L1 {A} 2
Tid Items C1 {B} 3
{B} 3
10 A, C, D {C} 3
1st scan {C} 3
20 B, C, E {D} 1
{E} 3
30 A, B, C, E {E} 3
40 B, E
C2 Itemset sup C2 Itemset
{A, B} 1
L2 Itemset sup
{A, C} 2
2nd scan {A, B}
{A, C} 2 {A, C}
{A, E} 1
{B, C} 2
{B, C} 2 {A, E}
{B, E} 3
{B, E} 3 {B, C}
{C, E} 2
{C, E} 2 {B, E}
{C, E}
Itemset Itemset Sup
C3 C3
{A, B, C} {A, B, C} 1
3rd scan {A, C, E} 1
Itemset sup
{A, C, E} L3 {B, C, E} 2
19 {B, C, E} 2
{B, C, E}
 The Apriori Algorithm (Pseudo-Code)
Ck: Candidate itemset of size k
Lk : frequent itemset of size k

L1 = {frequent items};
for (k = 1; Lk !=Æ; k++) do begin
Ck+1 = candidates generated from Lk;
for each transaction t in database do
increment the count of all candidates in Ck+1 that are
contained in t
Lk+1 = candidates in Ck+1 with min_support
end
return Èk Lk;
20
 Implementation of Apriori
• How to generate candidates Ck+1 from Lk?
• Step 1: self-joining Lk
• Step 2: pruning
• Example of Candidate-generation
• L3={abc, abd, acd, ace, bcd}
• Self-joining: L3*L3
• abcd from abc and abd
• acde from acd and ace
• abce from abc and ace
• NO abcde !
• Pruning:
• acde is removed because ade is not in L3
• abce is removed because bce is not in L3
• C4 = {abcd}
21
Generating Association Rules
from Frequent Itemsets
• Strong association rules: min_support,
min_confident
• Steps
• For each frequent itemset l, generate all nonempty
subsets of l.
• For every nonempty subset s of l, output the rule
s⇒l–s
If min_confidence
Example
• X={I1,I2,I5}
• What are the association rules
generated from X?

• If min_conf=70%, then the 2nd ,

3rd, and 6th rules are strong
rules.
Scalable Frequent Itemset Mining Methods

• Apriori: A Candidate Generation-and-Test Approach

• Improving the Efficiency of Apriori

• FPGrowth: A Frequent Pattern-Growth Approach

• ECLAT: Frequent Pattern Mining with Vertical Data Format

• Mining Close Frequent Patterns and Maxpatterns

24
Further Improvement of the Apriori Method

• Major computational challenges

• Multiple scans of transaction database
• Huge number of candidates
• Tedious workload of support counting for candidates
• Improving Apriori: general ideas
• Reduce passes of transaction database scans
• Shrink number of candidates
• Facilitate support counting of candidates

25
 Partition
• Any itemset that is potentially frequent in DB must be frequent
in at least one of the partitions of DB
• Scan 1: partition database and find local frequent patterns
• Scan 2: consolidate global frequent patterns

DB1 + DB2 + + DBk = DB

sup1(i) < σDB1 sup2(i) < σDB2 supk(i) < σDBk sup(i) < σDB
 Dynamic Hashing and Pruning
(DHP): Reduce the Number of Candidates
• A k-itemset whose corresponding hashing bucket count is below the threshold
cannot be frequent
count itemsets
• Candidates: a, b, c, d, e {ab, ad, ae}
35
• Hash entries 88 {bd, be, de}

• {ab, ad, ae} . .

. .
• {bd, be, de} . .

• … 102 {yz, qs, wt}

• Frequent 1-itemset: a, b, d, e Hash Table

• ab is not a candidate 2-itemset if the sum of count of {ab, ad, ae} is below
support threshold

27
Sampling for Frequent Patterns

• Select a sample of original database, mine frequent patterns

within sample using Apriori
• Scan database once to verify frequent itemsets found in
sample
• Scan database again to find missed frequent patterns

30
 Dynamic item counting (DIC): Reduce Number of Scans
ABCD

• Once both A and D are determined frequent,

ABC ABD ACD BCD the counting of AD begins
• Once all length-2 subsets of BCD are
AB AC BC AD BD CD determined frequent, the counting of BCD
begins

Transactions
A B C D
1-itemsets
Apriori 2-itemsets
{}
…
Itemset lattice
1-itemsets
2-items
DIC 3-items

31
Scalable Frequent Itemset Mining Methods

• Apriori: A Candidate Generation-and-Test Approach

• Improving the Efficiency of Apriori

• FPGrowth: A Frequent Pattern-Growth Approach

• ECLAT: Frequent Pattern Mining with Vertical Data Format

• Mining Close Frequent Patterns and Maxpatterns

32
Pattern-Growth Approach: Mining Frequent Patterns
Without Candidate Generation

• Bottlenecks of the Apriori approach

• Breadth-first (i.e., level-wise) search
• Candidate generation and test
• Often generates a huge number of candidates
• The FPGrowth Approach (J. Han, J. Pei, and Y. Yin, SIGMOD’ 00)
• Depth-first search
• Avoid explicit candidate generation
• Major philosophy: Grow long patterns from short ones using local frequent
items only
• “abc” is a frequent pattern
• Get all transactions having “abc”, i.e., project DB on abc: DB|abc
• “d” is a local frequent item in DB|abc à abcd is a frequent pattern
33
Construct FP-tree from a Transaction Database

TID Items bought (ordered) frequent items

100 {f, a, c, d, g, i, m, p} {f, c, a, m, p}
200 {a, b, c, f, l, m, o} {f, c, a, b, m}
300 {b, f, h, j, o, w} {f, b} min_support = 3
400 {b, c, k, s, p} {c, b, p}
500 {a, f, c, e, l, p, m, n} {f, c, a, m, p}

1. Scan DB once, find frequent 1-itemset

(a,3),(b,3), (c,4),(f,4),(m,3)(p,3)
(single item pattern)
2. Sort frequent items in frequency
descending order, F-list
F-list = f-c-a-b-m-p
3. Scan DB again, construct FP-tree. (The
transactions are processed in the order of
the F-list)

34
FP Tree construction
• Root is represented as null
• Scan the data set one transaction at a time to
create the FP-tree. For each transaction:
• If it is a unique transaction, form a new path and set the
counter for each node to 1.
• If it shares a common prefix itemset then increment the
common itemset node counters and create new nodes if
needed.
• Continue this until each transaction has been
mapped unto the tree.
FP Tree with Header table and node-links
{}
Header Table

Item frequency head f:4 c:1

f 4
c 4 c:3 b:1 b:1
a 3
b 3 a:3 p:1
m 3
p 3
m:2 b:1

p:2 m:1
 Partition Patterns and Databases
• Frequent patterns can be partitioned into subsets
according to f-list
• F-list = f-c-a-b-m-p
• Patterns containing p - p’s conditional patternbase
• Patterns having m but no p - m’s conditional patternbase
• …
• Patterns having c but no a nor b, m, p - c’s conditional
patternbase
• Pattern f
• Completeness and non-redundency

37
Find Patterns Having P From P-conditional Database

• Starting at the frequent item header table in the FP-tree

• Traverse the FP-tree by following the link of each frequent item p
• Accumulate all of transformed prefix paths of item p to form p’s
conditional pattern base
{}
Header Table

Item frequency head f:4 c:1 Conditional

Conditional
f 4
c 4 c:3 b:1 b:1 pattern bases FP-tree
a 3 c f:3 c f:3
b 3 a:3 p:1
m 3 a fc:3 a fc:3
p 3 m:2 b:1 b fca:1, f:1, c:1 b ∅
m fca:2, fcab:1 m fca:3
p:2 m:1 p c:3
p fcam:2, cb:1
38
From Conditional Pattern-bases to Conditional FP-trees

• For each pattern-base

• Accumulate the count for each item in the base
• Construct the FP-tree for the frequent items of the pattern
base
{}
Header Table

Item frequency head f:4 c:1 Conditional

Conditional
f 4
c 4 c:3 b:1 b:1 pattern bases FP-tree
a 3 c f:3 c f:3
b 3 a:3 p:1
m 3 a fc:3 a fc:3
p 3 m:2 b:1 b fca:1, f:1, c:1 b ∅
m fca:2, fcab:1 m fca:3
p:2 m:1 p c:1
p fcam:2, cb:1
39
• Generate frequent patters
from Conditional FP-tree
All frequent
patterns relate to m
• If the conditional FP-tree has a {}
single path, then generate all m,
the combinations of the f:3 Ú fm, cm, am,
headers fcm, fam, cam,
c:3 fcam
• Otherwise, construct
conditional pattern base and a:3
mine its conditional FP-tree for m-conditional FP-tree
each frequent item-set
Conditional FP-tree for each frequent item-set

{}

{} Cond. pattern base of “am”: (fc:3) f:3

c:3
f:3
am-conditional FP-tree
c:3 {}
Cond. pattern base of “cm”: (f:3)
a:3 f:3
m-conditional FP-tree
cm-conditional FP-tree

{}

Cond. pattern base of “cam”: (f:3) f:3

cam-conditional FP-tree
41
 Benefits of the FP-tree Structure

• Completeness
• Preserve complete information for frequent pattern mining
• Never break a long pattern of any transaction
• Compactness
• Reduce irrelevant info—infrequent items are gone
• Items in frequency descending order: the more frequently
occurring, the more likely to be shared
• Never be larger than the original database (not count node-
links and the count field)

43
Scaling FP-growth by Database Projection

• What about if FP-tree cannot fit in memory?

• DB projection
• First partition a database into a set of projected DBs
• Then construct and mine FP-tree for each projected DB
• Partition projection
• Partition the DB based on the ordered frequent items
• Passing the unprocessed parts to the subsequent partitions

44
 Partition-Based Projection

Tran. DB
fcamp
fcabm
fb
cbp
fcamp

p-proj DB m-proj DB b-proj DB a-proj DB c-proj DB f-proj DB

fcam fcab f fc f …
cb fca cb … …
fcam fca …

am-proj DB cm-proj DB
fc f …
fc f
fc f
45
Performance of FPGrowth in Large Datasets

100

90 D1 FP-grow th runtime
D1 Apriori runtime
80

70
Run time(sec.)

60

50 Data set T25I20D10K

40

30
20

10

0
0 0.5 1 1.5 2 2.5 3
Support threshold(%)

FP-Growth vs. Apriori

46
Advantages of the Pattern Growth Approach

• Divide-and-conquer:
• Decompose both the mining task and DB according to the frequent
patterns obtained so far
• Lead to focused search of smaller databases
• Other factors
• No candidate generation, no candidate test
• Compressed database: FP-tree structure
• No repeated scan of entire database
• Basic ops: counting local freq items and building sub FP-tree, no pattern
search and matching

47
 Further Improvements of Mining Methods

• AFOPT (Liu, et al. @ KDD’03)

• A “push-right” method for mining condensed frequent pattern (CFP) tree

• Carpenter (Pan, et al. @ KDD’03)

• Mine data sets with small rows but numerous columns
• Construct a row-enumeration tree for efficient mining

• FPgrowth+ (Grahne and Zhu, FIMI’03)

• Efficiently Using Prefix-Trees in Mining Frequent Itemsets, Proc. ICDM'03
Int. Workshop on Frequent Itemset Mining Implementations (FIMI'03),
Melbourne, FL, Nov. 2003

• TD-Close (Liu, et al, SDM’06)

48
Extension of Pattern Growth Mining Methodology

• Mining closed frequent itemsets and max-patterns

• CLOSET (DMKD’00), FPclose, and FPMax (Grahne & Zhu, Fimi’03)
• Mining sequential patterns
• PrefixSpan (ICDE’01), CloSpan (SDM’03), BIDE (ICDE’04)
• Mining graph patterns
• gSpan (ICDM’02), CloseGraph (KDD’03)
• Constraint-based mining of frequent patterns
• Convertible constraints (ICDE’01), gPrune (PAKDD’03)
• Computing iceberg data cubes with complex measures
• H-tree, H-cubing, and Star-cubing (SIGMOD’01, VLDB’03)
• Pattern-growth-based Clustering
• MaPle (Pei, et al., ICDM’03)
• Pattern-Growth-Based Classification
• Mining frequent and discriminative patterns (Cheng, et al, ICDE’07)
49
Scalable Frequent Itemset Mining Methods

• Apriori: A Candidate Generation-and-Test Approach

• Improving the Efficiency of Apriori

• FPGrowth: A Frequent Pattern-Growth Approach

• ECLAT: Frequent Pattern Mining with Vertical Data Format

• Mining Close Frequent Patterns and Maxpatterns

50
ECLAT: Mining by Exploring Vertical Data Format

• Vertical format: t(AB) = {T11, T25, …}

• tid-list: list of trans.-ids containing an itemset
• Deriving frequent patterns based on vertical intersections
• t(X) = t(Y): X and Y always happen together
• t(X) Ì t(Y): transaction having X always has Y
• Using diffset to accelerate mining
• Only keep track of differences of tids
• t(X) = {T1, T2, T3}, t(XY) = {T1, T3}
• Diffset (XY, X) = {T2}
• Eclat (Zaki et al. @KDD’97)
• Mining Closed patterns using vertical format: CHARM (Zaki &
Hsiao@SDM’02)
51
Scalable Frequent Itemset Mining Methods

• Apriori: A Candidate Generation-and-Test Approach

• Improving the Efficiency of Apriori

• FPGrowth: A Frequent Pattern-Growth Approach

• ECLAT: Frequent Pattern Mining with Vertical Data Format

• Mining Close Frequent Patterns and Maxpatterns

52
CLOSET+: Mining Closed Itemsets by Pattern-Growth

• Pruning
• Itemset merging: if Y appears in every occurrence of X, then Y is
merged with X
• Sub-itemset pruning: if Y ‫ כ‬X, and sup(X) = sup(Y), X and all of X’s
descendants in the set enumeration tree can be pruned
• Checking
• superset checking
• Subset checking
Mining Frequent Patterns, Association and Correlations:
Basic Concepts and Methods

• Basic Concepts

• Frequent Itemset Mining Methods

• Which Patterns Are Interesting?—Pattern Evaluation

Methods

• Summary

55
 Interestingness Measure: Correlations (Lift)

• play basketball Þ eat cereal [40%, 66.7%]

• The overall % of students eating cereal is 75% > 66.7% !! Rue is misleading

• play basketball Þ not eat cereal [20%, 33.3%] is more accurate, although with
lower support and confidence

• Measure of dependent/correlated events: lift

P( AÈ B) Basketball Not basketball Sum (row)
lift = Cereal 2000 1750 3750
P( A) P( B)
Not cereal 1000 250 1250
2000 / 5000
lift ( B, C ) = = 0.89 Sum(col.) 3000 2000 5000
3000 / 5000 * 3750 / 5000
1000 / 5000
lift ( B, ¬C ) = = 1.33
3000 / 5000 *1250 / 5000

56
• “Buy walnuts Þ buy milk
[1%, 80%]” is misleading if
85% of customers buy milk

• Support and confidence

are not good to indicate
correlations

• Over 20 interestingness
measures have been
proposed (see Tan, Kumar,
Sritastava @KDD’02)

• Which are good ones?

57
Null-Invariant Measures

58
Mining Frequent Patterns, Association and Correlations:
Basic Concepts and Methods

• Basic Concepts

• Frequent Itemset Mining Methods

• Which Patterns Are Interesting?—Pattern Evaluation

Methods

• Summary

59
Summary
• Basic concepts: association rules, support-confident
framework, closed and max-patterns
• Scalable frequent pattern mining methods
• Apriori (Candidate generation & test)
• Projection-based (FPgrowth, CLOSET+, ...)
• Vertical format approach (ECLAT, ...)

§ Which patterns are interesting?

§ Pattern evaluation methods

60
Assignment 1 & Midterm
• A1 is posted in the dropbox.
• Due on Feb 2nd at 5pm
• Submit to dropbox
• Only ipynb

• Midterm Date: 4:00pm – 5:20pm, Feb 14th