0% found this document useful (0 votes)
69 views24 pages

Lecture 4 - Data Pre-Processing: Fall 2010 Dr. Tariq MAHMOOD Nuces (Fast) - Khi

The document discusses various techniques for data pre-processing including data reduction, parametric and non-parametric methods, linear and log-linear regression models, histograms, clustering, sampling, discretization, and concept hierarchy generation. The goal of data pre-processing is to prepare raw data for further analysis by cleaning, transforming, reducing dimensionality, and handling missing values. Common techniques include feature selection, normalization, aggregation, and converting data into appropriate formats.
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
69 views24 pages

Lecture 4 - Data Pre-Processing: Fall 2010 Dr. Tariq MAHMOOD Nuces (Fast) - Khi

The document discusses various techniques for data pre-processing including data reduction, parametric and non-parametric methods, linear and log-linear regression models, histograms, clustering, sampling, discretization, and concept hierarchy generation. The goal of data pre-processing is to prepare raw data for further analysis by cleaning, transforming, reducing dimensionality, and handling missing values. Common techniques include feature selection, normalization, aggregation, and converting data into appropriate formats.
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPT, PDF, TXT or read online on Scribd
You are on page 1/ 24

Lecture 4 – Data Pre-processing

Fall 2010

Dr. Tariq MAHMOOD


NUCES (FAST) – KHI
1
 Reduce data volume by choosing alternative,
smaller forms of data representation
 Parametric methods
 Assume the data fits some model, estimate
model parameters, store only the parameters,
and discard the data (except possible outliers)
 Example: Log-linear models—obtain value at
a point in m-D space as the product on
appropriate marginal subspaces
 Non-parametric methods
 Do not assume models
 Major families: histograms, clustering,
sampling.
December 8, 2021 Data Mining: Concepts and Techniques 2
 Linear regression: Data are modeled to fit a straight
line

 Often uses the least-square method to fit the line


 Multiple regression: allows a response variable Y to
be modeled as a linear function of multidimensional
feature vector
 Log-linear model: approximates discrete
multidimensional probability distributions

December 8, 2021 Data Mining: Concepts and Techniques 3


 Linear regression: Y = w X + b
 Two regression coefficients, w and b, specify
the line and are to be estimated by using the
data at hand
 Using the least squares criterion to the known
values of Y1, Y2, …, X1, X2, ….
 Multiple regression: Y = b0 + b1 X1 + b2 X2.
 Many nonlinear functions can be transformed
into the above
 Log-linear models:
 The multi-way table of joint probabilities is
approximated by a product of lower-order
tables
 Probability: p(a, b, c, d) = ab acad bcd
 Divide data into buckets and store average (sum) for each
bucket
 Partitioning rules:
 Equal-width: equal bucket range
 Equal-frequency (or equal-depth)
 V-optimal: with the least histogram variance (weighted
sum of the original values that each bucket represents)
 MaxDiff: set bucket boundary between each pair for pairs
have the β–1 largest differences

December 8, 2021 Data Mining: Concepts and Techniques 5


 Partition data set into clusters based on similarity, and
store cluster representation (e.g., centroid and diameter)
only
 Can be very effective if data is clustered but not if data is
“smeared”
 Can have hierarchical clustering and be stored in multi-
dimensional index tree structures
 There are many choices of clustering definitions and
clustering algorithms
 Cluster analysis will be studied in depth in Chapter 7

December 8, 2021 Data Mining: Concepts and Techniques 6


 Sampling: obtaining a small sample s to
represent the whole data set N
 Allow a mining algorithm to run in complexity
that is potentially sub-linear to the size of the
data
 Choose a representative subset of the data
 Simple random sampling may have very poor
performance in the presence of skew
 Develop adaptive sampling methods
 Stratified sampling:
 Approximate the percentage of each class (or
subpopulation of interest) in the overall
database
 Used in conjunction with skewed data.

December 8, 2021 Data Mining: Concepts and Techniques 7


Sampling: With or Without
Replacement

W O R
SRS le random
i m p h ou t
( s e wi t
l
samp ment)
pl a c e
re

SRSW
R

Raw Data
December 8, 2021 Data Mining: Concepts and Techniques 8
Raw Data Cluster/Stratified Sample

December 8, 2021 Data Mining: Concepts and Techniques 9


 Why preprocess the data?
 Data cleaning
 Data integration and transformation
 Data reduction
 Discretization and concept hierarchy generation
 Summary

December 8, 2021 Data Mining: Concepts and Techniques 10


 Three types of attributes:
 Nominal — values from an unordered set, e.g., color,
profession
 Ordinal — values from an ordered set, e.g., military or
academic rank
 Continuous — real numbers, e.g., integer or real numbers
 Discretization:
 Divide the range of a continuous attribute into intervals
 Some classification algorithms only accept categorical
attributes
 Reduce data size by discretization
 Prepare for further analysis

December 8, 2021 Data Mining: Concepts and Techniques 11


 Discretization
 Reduce the number of values for a given continuous attribute
by dividing the range of the attribute into intervals
 Interval labels can then be used to replace actual data values
 Supervised vs. unsupervised
 Split (top-down) vs. merge (bottom-up)
 Discretization can be performed recursively on an attribute
 Concept hierarchy formation
 Recursively reduce the data by collecting and replacing low
level concepts (such as numeric values for age) by higher
level concepts (such as young, middle-aged, or senior)

December 8, 2021 Data Mining: Concepts and Techniques 12


 Typical methods: All the methods can be applied recursively
 Binning (covered above)
 Top-down split, unsupervised,
 Histogram analysis (covered above)
 Top-down split, unsupervised
 Clustering analysis (covered above)
 Either top-down split or bottom-up merge, unsupervised
 Entropy-based discretization: supervised, top-down split
 Interval merging by 2 Analysis: unsupervised, bottom-up
merge
 Segmentation by natural partitioning: top-down split,
unsupervised

December 8, 2021 Data Mining: Concepts and Techniques 13


 Given a set of samples S, if S is partitioned into two intervals
S1 and S2 using boundary T, the information gain after
partitioning is | S1 | | S2 |
I (S , T )  Entropy ( S 1)  Entropy ( S 2)
|S| |S|
 Entropy is calculated based on class distribution of the
samples in the set. Given m classes,
m
the entropy of S1 is
Entropy ( S1 )   pi log 2 ( pi )
i 1

where pi is the probability of class i in S1


 The boundary that minimizes the entropy function over all
possible boundaries is selected as a binary discretization
 The process is recursively applied to partitions obtained until
some stopping criterion is met
 Such a boundary may reduce data size and improve
classification accuracy
December 8, 2021 Data Mining: Concepts and Techniques 14
 Merging-based (bottom-up) vs. splitting-based methods
 Merge: Find the best neighboring intervals and merge them
to form larger intervals recursively
 ChiMerge [Kerber AAAI 1992, See also Liu et al. DMKD 2002]
 Initially, each distinct value of a numerical attr. A is
considered to be one interval
 2 tests are performed for every pair of adjacent intervals
 Adjacent intervals with the least 2 values are merged
together, since low 2 values for a pair indicate similar class
distributions
 This merge process proceeds recursively until a predefined
stopping criterion is met (such as significance level)

December 8, 2021 Data Mining: Concepts and Techniques 15


 A simply 3-4-5 rule can be used to segment numeric
data into relatively uniform, “natural” intervals.
 If an interval covers 3, 6, 7 or 9 distinct values at
the most significant digit, partition the range into 3
equi-width intervals
 If it covers 2, 4, or 8 distinct values at the most
significant digit, partition the range into 4 intervals
 If it covers 1, 5, or 10 distinct values at the most
significant digit, partition the range into 5 intervals

December 8, 2021 Data Mining: Concepts and Techniques 16


count

Step 1: -$351 -$159 profit $1,838 $4,700


Min Low (i.e, 5%-tile) High(i.e, 95%-0 tile) Max
Step 2: msd=1,000 Low=-$1,000 High=$2,000

(-$1,000 - $2,000)
Step 3:

(-$1,000 - 0) (0 -$ 1,000) ($1,000 - $2,000)

(-$400 -$5,000)
Step 4:

(-$400 - 0) ($2,000 - $5, 000)


(0 - $1,000) ($1,000 - $2, 000)

(-$400 - 0) (0 -
($1,000($1,000
- - $2, 000)
(-$400 - $200) (0 - $1,000)
$1,200) ($2,000 -
-$300) (0 -
($200 - ($1,000 - $3,000)
(-$400 - $200) ($1,200 -
-$300) $400) $1,200)
(-$300 - $1,400)
-$200) ($200 - ($3,000 -
($1,200 -
(-$300 - $400)
($400 - ($1,400 - $4,000)
$1,400)
(-$200 -
-$200) $600) $1,600) ($4,000 -
-$100) ($400 - ($600 - ($1,400 - $5,000)
($1,600 -
(-$200 - $600) $800) $1,600) ($1,800 -
($800 - $1,800)
-$100)(-$100 - $2,000)
($600 - $1,000) ($1,600 -
0) $800) ($800 - ($1,800 -
$1,800)
(-$100 - $1,000) $2,000)
0)
December 8, 2021 Data Mining: Concepts and Techniques 17
 Specification of a partial/total ordering of attributes
explicitly at the schema level by users or experts
 street < city < state < country
 Specification of a hierarchy for a set of values by
explicit data grouping
 {Urbana, Champaign, Chicago} < Illinois
 Specification of only a partial set of attributes
 E.g., only street < city, not others
 Automatic generation of hierarchies (or attribute
levels) by the analysis of the number of distinct values
 E.g., for a set of attributes: {street, city, state,
country}
December 8, 2021 Data Mining: Concepts and Techniques 18
 Some hierarchies can be automatically
generated based on the analysis of the number
of distinct values per attribute in the data set
 The attribute with the most distinct values is
placed at the lowest level of the hierarchy
 Exceptions, e.g., weekday, month, quarter,
year

country 15 distinct
15 distinct values
values

province_or_ state 365 365 distinct


distinct values
values

city 35673567 distinct


distinct values
values

street 674,339
674,339 distinct
distinct values
values
December 8, 2021 Data Mining: Concepts and Techniques 19
 Why preprocess the data?
 Data cleaning
 Data integration and transformation
 Data reduction
 Discretization and concept hierarchy
generation
 Summary

December 8, 2021 Data Mining: Concepts and Techniques 20


 Data preparation or preprocessing is a big issue
for both data warehousing and data mining
 Descriptive data summarization is needed for
quality data preprocessing
 Data preparation includes
 Data cleaning and data integration
 Data reduction and feature selection
 Discretization
 A lot a methods have been developed but data
preprocessing still an active area of research.
December 8, 2021 Data Mining: Concepts and Techniques 21
 D. P. Ballou and G. K. Tayi. Enhancing data quality in data warehouse environments.
Communications of ACM, 42:73-78, 1999
 T. Dasu and T. Johnson. Exploratory Data Mining and Data Cleaning. John Wiley & Sons,
2003
 T. Dasu, T. Johnson, S. Muthukrishnan, V. Shkapenyuk. 
Mining Database Structure; Or, How to Build a Data Quality Browser. SIGMOD’02. 
 H.V. Jagadish et al., Special Issue on Data Reduction Techniques. Bulletin of the
Technical Committee on Data Engineering, 20(4), December 1997
 D. Pyle. Data Preparation for Data Mining. Morgan Kaufmann, 1999
 E. Rahm and H. H. Do. Data Cleaning: Problems and Current Approaches. IEEE Bulletin
of the Technical Committee on Data Engineering. Vol.23, No.4
 V. Raman and J. Hellerstein. Potters Wheel: An Interactive Framework for Data Cleaning
and Transformation, VLDB’2001
 T. Redman. Data Quality: Management and Technology. Bantam Books, 1992
 Y. Wand and R. Wang. Anchoring data quality dimensions ontological foundations.
Communications of ACM, 39:86-95, 1996
 R. Wang, V. Storey, and C. Firth. A framework for analysis of data quality research. IEEE
Trans. Knowledge and Data Engineering, 7:623-640, 1995

December 8, 2021 Data Mining: Concepts and Techniques 22


23
1. What is meant by symmetric and skewed data [5]

2. Describe techniques for smoothing out data [10]

3. Why is it important to carry out descriptive data


summarization? Justify your response through a
fictitious quantile-quantile plot [5]

4. Why is it necessary to carry out co-relation analysis?


[5]

5. Describe “data cube aggregation” and its


advantages [5]
6. Can you suggest some change(s) to the state-of-
the-art data pre-processing activity? [10]
24

You might also like