Data Preprocessing 1

Outline
Part 1
o Why preprocess the
data?
o Data cleaning
o Incomplete
o Noise
o Inconsistent

Part 2 (Next Week)

2
 Why preprocess data?

incomplete
Dirty
data
• lacking attribute values,
lacking certain attributes
of interest, or containing
inconsistent only aggregate data
• e.g., occupation=“ ”
• containing discrepancies in codes (missing data)
or
names, e.g., nois
• Age=“42” Birthday=“03/07/1997” e
• Was rating “1,2,3”, now rating “A,
B, C” • containing noise, errors, or
• discrepancy between duplicate outliers
records • e.g., Salary=“−10” (an error)

7
Why Is Data Dirty?
o Incomplete data may come from
 “Not applicable” data value when collected
 Different considerations between the time when the data
was collected and when it is analyzed.
 Human/hardware/software problems
o N oisy data (incorrect values) may come from
 Faulty data collection instruments
 Human or computer error at data entry
 Errors in data transmission
o Inconsistent data may come from
 Different data sources
 Functional dependency violation (e.g., modify some linked
data)
o Duplicate records also need data cleaning

8
Why Preprocessing is Important?
o No quality data, no quality mining results!
 Q uality decisions must be based on quality data
 e.g., duplicate or missing data may cause incorrect or even

misleading statistics.
 Data warehouse needs consistent integration of quality data

o Data extraction, cleaning, and transformation comprises the

9
Outline
o General data
characteristics
o Why preprocess the
data?
 Data cleaning
o Data integration
o Data transformation
o Data reduction
o Summary

10
Data Cleaning
o To clean data from:
 Incomplete /missing data
 Noisy data
 Inconsistent /outliers
o Importance
 “Data cleaning is one of the three biggest
problems in data warehousing”— Ralph Kimball
 “Data cleaning is the number one problem in data
warehousing”—DCI survey
o Data cleaning tasks
 Fill in missing values
 Identify outliers and smooth out noisy data
 Correct inconsistent data
 Resolve redundancy caused by data integration

11
Missing Data
o Data is not always available
 E.g., many tuples have no recorded value for several
attributes, such as
customer income in sales data
o Missing data may be due to
 equipment malfunction
 inconsistent with other recorded data and thus deleted
 data not entered due to misunderstanding
 certain data may not be considered important at the time of
entry
 not register history or changes of the data
o Missing data may need to be inferred

12
How to Handle Missing Data?
o Ignore the tuple: usually done when class label is missing
(when doing classification)— not effective when the % of
missing values per attribute varies considerably
o Fill in the missing value manually: tedious + infeasible?
o Fill in it automatically with
 a global constant : e.g., “unknown”, a new class?!
 the attribute mean
 the attribute mean for all samples belonging to the same
class: smarter
 the most probable value: inference-based such as Bayesian
formula or
decision tree

13
Noisy Data
o Noise: random error or variance in a
measured variable
o Incorrect attribute values may due to
 faulty data collection instruments
 data entry problems
 data transmission problems
 technology limitation
 inconsistency in naming convention
o Other data problems which requires data
cleaning
 duplicate records
 incomplete data
 inconsistent data

18
How to Handle Noisy Data?
o Binning
 first sort data and partition into (equal-frequency) bins
 then one can smooth by bin means, smooth by bin median,
smooth by bin boundaries, etc.
o Regression
 smooth by fitting the data into regression functions
o Clustering
 detect and remove outliers
o Combined computer and human inspection
 detect suspicious values and check by human (e.g., deal
with possible outliers)

19
Simple Discretization Methods - Binning
o Equal-width (distance) partitioning
 Divides the range into N intervals of equal size: uniform grid
 if A and B are the lowest and highest values of the attribute,
the width of intervals will be:W = (B – A)/N.
 The most straightforward, but outliers may dominate
presentation
 Skewed data is not handled well
o Equal-depth (frequency) partitioning
 Divides the range into N intervals, each containing
approximately same number of samples
 Good data scaling
 Managing categorical attributes can be tricky

20
Binning Methods for Data Smoothing
o Sorted data for price (in dollars):
4, 8, 9, 15, 21, 21, 24, 25, 26,
28, 29, 34
o Partition into (equi-depth) 4 bins:
 Bin 1: 4, 8, 9, 15
 Bin 2: 21, 21, 24, 25
 Bin 3: 26, 28, 29, 34
o Smoothing by bin means:
 Bin 1: 9, 9, 9, 9
 Bin 2: 23, 23, 23, 23
 Bin 3: 29, 29, 29, 29
o Smoothing by bin boundaries:
 Bin 1: 4, 4, 4, 15
 Bin 2: 21, 21, 25, 25
 Bin 3: 26, 26, 26, 34
21
Regression

Y1

Y1’ y=x+
1

X x
1

22
Cluster Analysis

23