Data Mining:

Concepts and Techniques

— Chapter 5 —

Jiawei Han
Department of Computer Science
University of Illinois at Urbana-Champaign
www.cs.uiuc.edu/~hanj
©2006 Jiawei Han and Micheline Kamber, All rights reserved

February 16, 2022 1

 Chapter 5: Mining Frequent Patterns,
Association and Correlations

 Basic concepts and a road map

 Efficient and scalable frequent itemset mining
methods
 Mining various kinds of association rules
 From association mining to correlation
analysis
 Constraint-based association mining
 Summary

February 16, 2022 2

 What Is Frequent Pattern Analysis?
 Frequent pattern: a pattern (a set of items, subsequences, substructures,
etc.) that occurs frequently in a data set
 First proposed by Agrawal, Imielinski, and Swami [AIS93] in the context of
frequent itemsets and association rule mining
 Motivation: Finding inherent regularities in data
 What products were often purchased together?— Beer and diapers?!
 What are the subsequent purchases after buying a PC?
 What kinds of DNA are sensitive to this new drug?
 Can we automatically classify web documents?
 Applications
 Basket data analysis, cross-marketing, catalog design, sale campaign
analysis, Web log (click stream) analysis, and DNA sequence analysis.

February 16, 2022 3

 Why Is Freq. Pattern Mining Important?

 Discloses an intrinsic and important property of data sets

 Forms the foundation for many essential data mining tasks
 Association, correlation, and causality analysis
 Sequential, structural (e.g., sub-graph) patterns
 Pattern analysis in spatiotemporal, multimedia, time-
series, and stream data
 Classification: associative classification
 Cluster analysis: frequent pattern-based clustering
 Data warehousing: iceberg cube and cube-gradient
 Semantic data compression: fascicles
 Broad applications
February 16, 2022 4
 Basic Concepts: Frequent Patterns and
Association Rules
Transaction-id Items bought  Itemset X = {x1, …, xk}
10 A, B, D  Find all the rules X  Y with minimum
20 A, C, D support and confidence
30 A, D, E  support, s, probability that a
40 B, E, F
transaction contains X  Y
50 B, C, D, E, F
 confidence, c, conditional
Customer
buys both
Customer probability that a transaction
buys diaper
having X also contains Y
Let supmin = 50%, confmin = 50%
Freq. Pat.: {A:3, B:3, D:4, E:3, AD:3}
Association rules:
Customer
A  D (60%, 100%)
buys beer
D  A (60%, 75%)
February 16, 2022 5
 Chapter 5: Mining Frequent Patterns,
Association and Correlations
 Basic concepts and a road map
 Efficient and scalable frequent itemset mining
methods
 Mining various kinds of association rules
 From association mining to correlation
analysis
 Constraint-based association mining
 Summary

February 16, 2022 6

Scalable Methods for Mining Frequent Patterns
 The downward closure property of frequent patterns
 Any subset of a frequent itemset must be frequent

 If {beer, diaper, nuts} is frequent, so is {beer,

diaper}
 i.e., every transaction having {beer, diaper, nuts} also

contains {beer, diaper}

 Scalable mining methods: Three major approaches
 Apriori (Agrawal & Srikant@VLDB’94)

 Freq. pattern growth (FPgrowth—Han, Pei & Yin

@SIGMOD’00)
 Vertical data format approach (Charm—Zaki & Hsiao

@SDM’02)
February 16, 2022 7
 Apriori: A Candidate Generation-and-Test Approach

 Apriori pruning principle: If there is any itemset which is

infrequent, its superset should not be generated/tested!
(Agrawal & Srikant @VLDB’94, Mannila, et al. @ KDD’ 94)
 Method:
 Initially, scan DB once to get frequent 1-itemset
 Generate length (k+1) candidate itemsets from length k
frequent itemsets
 Test the candidates against DB
 Terminate when no frequent or candidate set can be
generated
February 16, 2022 8
 The Apriori Algorithm—An Example
Supmin = 2 Itemset sup
Itemset sup
Database TDB {A} 2
L1 {A} 2
Tid Items C1 {B} 3
{B} 3
10 A, C, D {C} 3
20 B, C, E 1st scan {D} 1
{C} 3
{E} 3
30 A, B, C, E {E} 3
40 B, E
C2 Itemset sup C2 Itemset
{A, B} 1
L2 Itemset sup 2nd scan {A, B}
{A, C} 2
{A, C} 2 {A, C}
{A, E} 1
{B, C} 2 {A, E}
{B, C} 2
{B, E} 3
{B, E} 3 {B, C}
{C, E} 2
{C, E} 2 {B, E}
{C, E}

C3 Itemset
3rd scan L3 Itemset sup
{B, C, E} {B, C, E} 2
February 16, 2022 9
 The Apriori Algorithm
 Pseudo-code:
Ck: Candidate itemset of size k
Lk : frequent itemset of size k

L1 = {frequent items};
for (k = 1; Lk !=; k++) do begin
Ck+1 = candidates generated from Lk;
for each transaction t in database do
increment the count of all candidates in Ck+1
that are contained in t
Lk+1 = candidates in Ck+1 with min_support
end
return k Lk;
February 16, 2022 10
 Important Details of Apriori
 How to generate candidates?
 Step 1: self-joining Lk
 Step 2: pruning
 How to count supports of candidates?
 Example of Candidate-generation
 L3={abc, abd, acd, ace, bcd}
 Self-joining: L3*L3
 abcd from abc and abd
 acde from acd and ace
 Pruning:
 acde is removed because ade is not in L3
 C4={abcd}

February 16, 2022 11

 Chapter 5: Mining Frequent Patterns,
Association and Correlations
 Basic concepts and a road map
 Efficient and scalable frequent itemset mining
methods
 Mining various kinds of association rules
 From association mining to correlation
analysis
 Constraint-based association mining
 Summary

February 16, 2022 12

 Mining Various Kinds of Association Rules

 Mining multilevel association

 Miming multidimensional association

 Mining quantitative association

 Mining interesting correlation patterns

February 16, 2022 13

 Mining Multiple-Level Association Rules
 Items often form hierarchies
 Flexible support settings
 Items at the lower level are expected to have lower

support
 Exploration of shared multi-level mining (Agrawal &
Srikant@VLB’95, Han & Fu@VLDB’95)

uniform support reduced support

Level 1
Milk Level 1
min_sup = 5%
[support = 10%] min_sup = 5%

Level 2 2% Milk Skim Milk Level 2

min_sup = 5% [support = 6%] [support = 4%] min_sup = 3%

February 16, 2022 14

 Multi-level Association: Redundancy Filtering

 Some rules may be redundant due to “ancestor”

relationships between items.
 Example
 milk  wheat bread [support = 8%, confidence = 70%]
 2% milk  wheat bread [support = 2%, confidence = 72%]
 We say the first rule is an ancestor of the second rule.
 A rule is redundant if its support is close to the
“expected” value, based on the rule’s ancestor.

February 16, 2022 15

 Mining Multi-Dimensional Association
 Single-dimensional rules:
buys(X, “milk”)  buys(X, “bread”)
 Multi-dimensional rules:  2 dimensions or predicates
 Inter-dimension assoc. rules (no repeated predicates)
age(X,”19-25”)  occupation(X,“student”)  buys(X, “coke”)
 hybrid-dimension assoc. rules (repeated predicates)
age(X,”19-25”)  buys(X, “popcorn”)  buys(X, “coke”)
 Categorical Attributes: finite number of possible values, no
ordering among values—data cube approach
 Quantitative Attributes: numeric, implicit ordering among
values—discretization, clustering, and gradient approaches

February 16, 2022 16

 Mining Quantitative Associations

 Techniques can be categorized by how numerical

attributes, such as age or salary are treated
1. Static discretization based on predefined concept
hierarchies (data cube methods)
2. Dynamic discretization based on data distribution
(quantitative rules, e.g., Agrawal & Srikant@SIGMOD96)
3. Clustering: Distance-based association (e.g., Yang &
Miller@SIGMOD97)
 one dimensional clustering then association
4. Deviation: (such as Aumann and Lindell@KDD99)
Sex = female => Wage: mean=$7/hr (overall mean = $9)

February 16, 2022 17

 Static Discretization of Quantitative Attributes

 Discretized prior to mining using concept hierarchy.

 Numeric values are replaced by ranges.
 In relational database, finding all frequent k-predicate sets
will require k or k+1 table scans.
()
 Data cube is well suited for mining.
 The cells of an n-dimensional (age) (income) (buys)
cuboid correspond to the
predicate sets.
(age, income) (age,buys) (income,buys)
 Mining from data cubes
can be much faster. (age,income,buys)
February 16, 2022 18
 Quantitative Association Rules
 Proposed by Lent, Swami and Widom ICDE’97
 Numeric attributes are dynamically discretized
 Such that the confidence or compactness of the rules

mined is maximized
 2-D quantitative association rules: Aquan1  Aquan2  Acat
 Cluster adjacent
association rules
to form general
rules using a 2-D grid
 Example
age(X,”34-35”)  income(X,”30-50K”)
 buys(X,”high resolution TV”)

February 16, 2022 19

 Mining Other Interesting Patterns

 Flexible support constraints (Wang et al. @ VLDB’02)

 Some items (e.g., diamond) may occur rarely but are
valuable
 Customized supmin specification and application
 Top-K closed frequent patterns (Han, et al. @ ICDM’02)
 Hard to specify supmin, but top-k with lengthmin is more
desirable
 Dynamically raise supmin in FP-tree construction and
mining, and select most promising path to mine

February 16, 2022 20

 Chapter 5: Mining Frequent Patterns,
Association and Correlations
 Basic concepts and a road map
 Efficient and scalable frequent itemset mining
methods
 Mining various kinds of association rules
 From association mining to correlation analysis
 Constraint-based association mining
 Summary

February 16, 2022 21

 Chapter 5: Mining Frequent Patterns,
Association and Correlations
 Basic concepts and a road map
 Efficient and scalable frequent itemset mining
methods
 Mining various kinds of association rules
 From association mining to correlation analysis
 Constraint-based association mining
 Summary

February 16, 2022 22

 Constraint-based (Query-Directed) Mining

 Finding all the patterns in a database autonomously? —

unrealistic!
 The patterns could be too many but not focused!
 Data mining should be an interactive process
 User directs what to be mined using a data mining
query language (or a graphical user interface)
 Constraint-based mining
 User flexibility: provides constraints on what to be
mined
 System optimization: explores such constraints for
efficient mining—constraint-based mining
February 16, 2022 23
 Constraints in Data Mining

 Knowledge type constraint:

 classification, association, etc.

 Data constraint — using SQL-like queries

 find product pairs sold together in stores in Chicago in

Dec.’02
 Dimension/level constraint
 in relevance to region, price, brand, customer category

 Rule (or pattern) constraint

 small sales (price < $10) triggers big sales (sum >

$200)
 Interestingness constraint
 strong rules: min_support  3%, min_confidence 

60%
February 16, 2022 24
 Constrained Mining vs. Constraint-Based Search

 Constrained mining vs. constraint-based search/reasoning

 Both are aimed at reducing search space

 Finding all patterns satisfying constraints vs. finding

some (or one) answer in constraint-based search in AI

 Constraint-pushing vs. heuristic search

 It is an interesting research problem on how to

integrate them
 Constrained mining vs. query processing in DBMS
 Database query processing requires to find all

 Constrained pattern mining shares a similar philosophy

as pushing selections deeply in query processing

February 16, 2022 25

 Chapter 5: Mining Frequent Patterns,
Association and Correlations
 Basic concepts and a road map
 Efficient and scalable frequent itemset mining
methods
 Mining various kinds of association rules
 From association mining to correlation analysis
 Constraint-based association mining
 Summary

February 16, 2022 26

 Frequent-Pattern Mining: Summary

 Frequent pattern mining—an important task in data mining

 Scalable frequent pattern mining methods
 Apriori (Candidate generation & test)
 Projection-based (FPgrowth, CLOSET+, ...)
 Vertical format approach (CHARM, ...)
 Mining a variety of rules and interesting patterns
 Constraint-based mining
 Mining sequential and structured patterns
 Extensions and applications
February 16, 2022 27
 Frequent-Pattern Mining: Research Problems

 Mining fault-tolerant frequent, sequential and structured

patterns
 Patterns allows limited faults (insertion, deletion,
mutation)
 Mining truly interesting patterns
 Surprising, novel, concise, …
 Application exploration
 E.g., DNA sequence analysis and bio-pattern
classification
 “Invisible” data mining

February 16, 2022 28

 Ref: Basic Concepts of Frequent Pattern Mining

 (Association Rules) R. Agrawal, T. Imielinski, and A. Swami. Mining

association rules between sets of items in large databases.
SIGMOD'93.
 (Max-pattern) R. J. Bayardo. Efficiently mining long patterns from
databases. SIGMOD'98.
 (Closed-pattern) N. Pasquier, Y. Bastide, R. Taouil, and L. Lakhal.
Discovering frequent closed itemsets for association rules. ICDT'99.
 (Sequential pattern) R. Agrawal and R. Srikant. Mining sequential
patterns. ICDE'95

February 16, 2022 29

 Ref: Apriori and Its Improvements

 R. Agrawal and R. Srikant. Fast algorithms for mining association rules.

VLDB'94.
 H. Mannila, H. Toivonen, and A. I. Verkamo. Efficient algorithms for
discovering association rules. KDD'94.
 A. Savasere, E. Omiecinski, and S. Navathe. An efficient algorithm for
mining association rules in large databases. VLDB'95.
 J. S. Park, M. S. Chen, and P. S. Yu. An effective hash-based algorithm for
mining association rules. SIGMOD'95.
 H. Toivonen. Sampling large databases for association rules. VLDB'96.
 S. Brin, R. Motwani, J. D. Ullman, and S. Tsur. Dynamic itemset counting
and implication rules for market basket analysis. SIGMOD'97.
 S. Sarawagi, S. Thomas, and R. Agrawal. Integrating association rule
mining with relational database systems: Alternatives and implications.
SIGMOD'98.
February 16, 2022 30
 Ref: Depth-First, Projection-Based FP Mining

 R. Agarwal, C. Aggarwal, and V. V. V. Prasad. A tree projection

algorithm for generation of frequent itemsets. J. Parallel and
Distributed Computing:02.
 J. Han, J. Pei, and Y. Yin. Mining frequent patterns without candidate
generation. SIGMOD’ 00.
 J. Pei, J. Han, and R. Mao. CLOSET: An Efficient Algorithm for Mining
Frequent Closed Itemsets. DMKD'00.
 J. Liu, Y. Pan, K. Wang, and J. Han. Mining Frequent Item Sets by
Opportunistic Projection. KDD'02.
 J. Han, J. Wang, Y. Lu, and P. Tzvetkov. Mining Top-K Frequent Closed
Patterns without Minimum Support. ICDM'02.
 J. Wang, J. Han, and J. Pei. CLOSET+: Searching for the Best
Strategies for Mining Frequent Closed Itemsets. KDD'03.
 G. Liu, H. Lu, W. Lou, J. X. Yu. On Computing, Storing and Querying
Frequent Patterns. KDD'03.
February 16, 2022 31
 Ref: Vertical Format and Row Enumeration Methods

 M. J. Zaki, S. Parthasarathy, M. Ogihara, and W. Li. Parallel algorithm

for discovery of association rules. DAMI:97.
 Zaki and Hsiao. CHARM: An Efficient Algorithm for Closed Itemset
Mining, SDM'02.
 C. Bucila, J. Gehrke, D. Kifer, and W. White. DualMiner: A Dual-
Pruning Algorithm for Itemsets with Constraints. KDD’02.
 F. Pan, G. Cong, A. K. H. Tung, J. Yang, and M. Zaki , CARPENTER:
Finding Closed Patterns in Long Biological Datasets. KDD'03.

February 16, 2022 32

 Ref: Mining Multi-Level and Quantitative Rules

 R. Srikant and R. Agrawal. Mining generalized association rules.

VLDB'95.
 J. Han and Y. Fu. Discovery of multiple-level association rules from
large databases. VLDB'95.
 R. Srikant and R. Agrawal. Mining quantitative association rules in
large relational tables. SIGMOD'96.
 T. Fukuda, Y. Morimoto, S. Morishita, and T. Tokuyama. Data mining
using two-dimensional optimized association rules: Scheme,
algorithms, and visualization. SIGMOD'96.
 K. Yoda, T. Fukuda, Y. Morimoto, S. Morishita, and T. Tokuyama.
Computing optimized rectilinear regions for association rules. KDD'97.
 R.J. Miller and Y. Yang. Association rules over interval data.
SIGMOD'97.
 Y. Aumann and Y. Lindell. A Statistical Theory for Quantitative
Association Rules KDD'99.
February 16, 2022 33
 Ref: Mining Correlations and Interesting Rules

 M. Klemettinen, H. Mannila, P. Ronkainen, H. Toivonen, and A. I.

Verkamo. Finding interesting rules from large sets of discovered
association rules. CIKM'94.
 S. Brin, R. Motwani, and C. Silverstein. Beyond market basket:
Generalizing association rules to correlations. SIGMOD'97.
 C. Silverstein, S. Brin, R. Motwani, and J. Ullman. Scalable
techniques for mining causal structures. VLDB'98.
 P.-N. Tan, V. Kumar, and J. Srivastava. Selecting the Right
Interestingness Measure for Association Patterns. KDD'02.
 E. Omiecinski. Alternative Interest Measures for Mining
Associations. TKDE’03.
 Y. K. Lee, W.Y. Kim, Y. D. Cai, and J. Han. CoMine: Efficient Mining
of Correlated Patterns. ICDM’03.
February 16, 2022 34
 Ref: Mining Other Kinds of Rules

 R. Meo, G. Psaila, and S. Ceri. A new SQL-like operator for mining

association rules. VLDB'96.
 B. Lent, A. Swami, and J. Widom. Clustering association rules.
ICDE'97.
 A. Savasere, E. Omiecinski, and S. Navathe. Mining for strong
negative associations in a large database of customer transactions.
ICDE'98.
 D. Tsur, J. D. Ullman, S. Abitboul, C. Clifton, R. Motwani, and S.
Nestorov. Query flocks: A generalization of association-rule mining.
SIGMOD'98.
 F. Korn, A. Labrinidis, Y. Kotidis, and C. Faloutsos. Ratio rules: A new
paradigm for fast, quantifiable data mining. VLDB'98.
 K. Wang, S. Zhou, J. Han. Profit Mining: From Patterns to Actions.
EDBT’02.
February 16, 2022 35
 Ref: Constraint-Based Pattern Mining

 R. Srikant, Q. Vu, and R. Agrawal. Mining association rules with item

constraints. KDD'97.
 R. Ng, L.V.S. Lakshmanan, J. Han & A. Pang. Exploratory mining and
pruning optimizations of constrained association rules. SIGMOD’98.
 M.N. Garofalakis, R. Rastogi, K. Shim: SPIRIT: Sequential Pattern
Mining with Regular Expression Constraints. VLDB’99.
 G. Grahne, L. Lakshmanan, and X. Wang. Efficient mining of
constrained correlated sets. ICDE'00.
 J. Pei, J. Han, and L. V. S. Lakshmanan. Mining Frequent Itemsets
with Convertible Constraints. ICDE'01.
 J. Pei, J. Han, and W. Wang, Mining Sequential Patterns with
Constraints in Large Databases, CIKM'02.

February 16, 2022 36

 Ref: Mining Sequential and Structured Patterns

 R. Srikant and R. Agrawal. Mining sequential patterns: Generalizations

and performance improvements. EDBT’96.
 H. Mannila, H Toivonen, and A. I. Verkamo. Discovery of frequent
episodes in event sequences. DAMI:97.
 M. Zaki. SPADE: An Efficient Algorithm for Mining Frequent Sequences.
Machine Learning:01.
 J. Pei, J. Han, H. Pinto, Q. Chen, U. Dayal, and M.-C. Hsu. PrefixSpan:
Mining Sequential Patterns Efficiently by Prefix-Projected Pattern Growth.
ICDE'01.
 M. Kuramochi and G. Karypis. Frequent Subgraph Discovery. ICDM'01.
 X. Yan, J. Han, and R. Afshar. CloSpan: Mining Closed Sequential
Patterns in Large Datasets. SDM'03.
 X. Yan and J. Han. CloseGraph: Mining Closed Frequent Graph Patterns.
KDD'03.
February 16, 2022 37
 Ref: Mining Spatial, Multimedia, and Web Data

 K. Koperski and J. Han, Discovery of Spatial Association Rules in

Geographic Information Databases, SSD’95.
 O. R. Zaiane, M. Xin, J. Han, Discovering Web Access Patterns and
Trends by Applying OLAP and Data Mining Technology on Web Logs.
ADL'98.
 O. R. Zaiane, J. Han, and H. Zhu, Mining Recurrent Items in
Multimedia with Progressive Resolution Refinement. ICDE'00.
 D. Gunopulos and I. Tsoukatos. Efficient Mining of Spatiotemporal
Patterns. SSTD'01.

February 16, 2022 38

 Ref: Mining Frequent Patterns in Time-Series Data

 B. Ozden, S. Ramaswamy, and A. Silberschatz. Cyclic association rules.

ICDE'98.
 J. Han, G. Dong and Y. Yin, Efficient Mining of Partial Periodic Patterns in
Time Series Database, ICDE'99.
 H. Lu, L. Feng, and J. Han. Beyond Intra-Transaction Association
Analysis: Mining Multi-Dimensional Inter-Transaction Association Rules.
TOIS:00.
 B.-K. Yi, N. Sidiropoulos, T. Johnson, H. V. Jagadish, C. Faloutsos, and A.
Biliris. Online Data Mining for Co-Evolving Time Sequences. ICDE'00.
 W. Wang, J. Yang, R. Muntz. TAR: Temporal Association Rules on
Evolving Numerical Attributes. ICDE’01.
 J. Yang, W. Wang, P. S. Yu. Mining Asynchronous Periodic Patterns in
Time Series Data. TKDE’03.
February 16, 2022 39
 Ref: FP for Classification and Clustering
 G. Dong and J. Li. Efficient mining of emerging patterns:
Discovering trends and differences. KDD'99.
 B. Liu, W. Hsu, Y. Ma. Integrating Classification and Association
Rule Mining. KDD’98.
 W. Li, J. Han, and J. Pei. CMAR: Accurate and Efficient
Classification Based on Multiple Class-Association Rules. ICDM'01.
 H. Wang, W. Wang, J. Yang, and P.S. Yu. Clustering by pattern
similarity in large data sets. SIGMOD’ 02.
 J. Yang and W. Wang. CLUSEQ: efficient and effective sequence
clustering. ICDE’03.
 B. Fung, K. Wang, and M. Ester. Large Hierarchical Document
Clustering Using Frequent Itemset. SDM’03.
 X. Yin and J. Han. CPAR: Classification based on Predictive
Association Rules. SDM'03.
February 16, 2022 40
 Ref: Stream and Privacy-Preserving FP Mining

 A. Evfimievski, R. Srikant, R. Agrawal, J. Gehrke. Privacy Preserving

Mining of Association Rules. KDD’02.
 J. Vaidya and C. Clifton. Privacy Preserving Association Rule Mining
in Vertically Partitioned Data. KDD’02.
 G. Manku and R. Motwani. Approximate Frequency Counts over
Data Streams. VLDB’02.
 Y. Chen, G. Dong, J. Han, B. W. Wah, and J. Wang. Multi-
Dimensional Regression Analysis of Time-Series Data Streams.
VLDB'02.
 C. Giannella, J. Han, J. Pei, X. Yan and P. S. Yu. Mining Frequent
Patterns in Data Streams at Multiple Time Granularities, Next
Generation Data Mining:03.
 A. Evfimievski, J. Gehrke, and R. Srikant. Limiting Privacy Breaches
in Privacy Preserving Data Mining. PODS’03.
February 16, 2022 41
 Ref: Other Freq. Pattern Mining Applications

 Y. Huhtala, J. Kärkkäinen, P. Porkka, H. Toivonen. Efficient

Discovery of Functional and Approximate Dependencies Using
Partitions. ICDE’98.
 H. V. Jagadish, J. Madar, and R. Ng. Semantic Compression and
Pattern Extraction with Fascicles. VLDB'99.
 T. Dasu, T. Johnson, S. Muthukrishnan, and V. Shkapenyuk.
Mining Database Structure; or How to Build a Data Quality
Browser. SIGMOD'02.

February 16, 2022 42