CS F415: Data Mining

Yashvardhan Sharma

1/30/24 C1S F415 1

Today’s Outline
• Data Preprocessing
• Data Quality
• Steps in Data Preprocessing
• Dimensionality Reduction - PCA

1/30/24 C1S F415 2

 Major Tasks in Data Preprocessing

• Data cleaning outliers=exceptions!

• Fill in missing values, smooth noisy data, identify or remove

outliers, and resolve inconsistencies
• Data integration
• Integration of multiple databases, data cubes, or files
• Data transformation
• Normalization and aggregation
• Data reduction
• Obtains reduced representation in volume but produces the same or
similar analytical results
• Data discretization
• Part of data reduction but with particular importance, especially for
1/30/24
numerical data CS F415 3
Forms of data preprocessing

1/30/24 CS F415 4
 Data Quality

• What kinds of data quality problems?

• How can we detect problems with the data?
• What can we do about these problems?

• Examples of data quality problems:

• Noise and outliers
• missing values
• duplicate data

1/30/24 CS F415 5
Data Cleaning
• Importance
• “Data cleaning is one of the three biggest problems in data warehousing”—
Ralph Kimball
• “Data cleaning is the number one problem in data warehousing”—DCI survey
• Data cleaning tasks
• Fill in missing values
• Identify outliers and smooth out noisy data
• Correct inconsistent data
• Resolve redundancy caused by data integration

1/30/24 CS F415 6
 Missing Data

• Data is not always available

• E.g., many tuples have no recorded value for several attributes, such as
customer income in sales data
• Missing data may be due to
• equipment malfunction
• inconsistent with other recorded data and thus deleted
• data not entered due to misunderstanding
• certain data may not be considered important at the time of entry
• not register history or changes of the data
• Missing data may need to be inferred.

1/30/24 CS F415 7
 Missing Values
• Reasons for missing values
• Information is not collected
(e.g., people decline to give their age and weight)
• Attributes may not be applicable to all cases
(e.g., annual income is not applicable to children)

• Handling missing values

• Eliminate Data Objects
• Estimate Missing Values
• Ignore the Missing Value During Analysis
• Replace with all possible values (weighted by their probabilities)
1/30/24 CS F415 8
 How to Handle Missing Data?
• Ignore the tuple: usually done when class label is missing (assuming the
tasks in classification—not effective when the percentage of missing
values per attribute varies considerably.
• Fill in the missing value manually: tedious + infeasible?
• Fill in it automatically with
• a global constant : e.g., “unknown”, a new class?!
• the attribute mean
• the attribute mean for all samples belonging to the same class: smarter
• the most probable value: inference-based such as Bayesian formula or decision
tree

1/30/24 CS F415 9
 Noisy Data
• Noise: random error or variance in a measured variable
• Incorrect attribute values may due to
• faulty data collection instruments
• data entry problems
• data transmission problems
• technology limitation
• inconsistency in naming convention
• Other data problems which requires data cleaning
• duplicate records
• incomplete data
• inconsistent data

1/30/24 CS F415 10
 Noise
• Noise refers to modification of original values
• Examples: distortion of a person’s voice when talking on a poor phone

Two Sine Waves Two Sine Waves + Noise

1/30/24 CS F415 11
How to Handle Noisy Data?
• Binning method:
• first sort data and partition into (equi-depth) bins
• then one can smooth by bin means, smooth by bin median, smooth by bin
boundaries, etc.
• Clustering
• detect and remove outliers
• Combined computer and human inspection
• detect suspicious values and check by human (e.g., deal with possible
outliers)
• Regression
• smooth by fitting the data into regression functions

1/30/24 CS F415 12
Simple Discretization Methods: Binning
• Equal-width (distance) partitioning:
• Divides the range into N intervals of equal size: uniform grid
• if A and B are the lowest and highest values of the attribute, the width of
intervals will be: W = (B –A)/N.
• The most straightforward, but outliers may dominate presentation
• Skewed data is not handled well.
• Equal-depth (frequency) partitioning:
• Divides the range into N intervals, each containing approximately same
number of samples
• Good data scaling
• Managing categorical attributes can be tricky.

1/30/24 CS F415 13
 Binning Methods for Data Smoothing
• Sorted data (e.g., by price)
• 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34
• Partition into (equi-depth) bins:
• Bin 1: 4, 8, 9, 15
• Bin 2: 21, 21, 24, 25
• Bin 3: 26, 28, 29, 34
• Smoothing by bin means:
• Bin 1: 9, 9, 9, 9
• Bin 2: 23, 23, 23, 23
• Bin 3: 29, 29, 29, 29
• Smoothing by bin boundaries:
• Bin 1: 4, 4, 4, 15
• Bin 2: 21, 21, 25, 25
• Bin 3: 26, 26, 26, 34

1/30/24 CS F415 14
Cluster Analysis

1/30/24 CS F415 15
Regression
y

Y1

Y1’ y=x+1

X1 x

1/30/24 CS F415 16
 Outliers
• Outliers are data objects with characteristics that are
considerably different than most of the other data objects in
the data set

1/30/24 CS F415 17
 Duplicate Data
• Data set may include data objects that are duplicates, or
almost duplicates of one another
• Major issue when merging data from heterogeneous sources

• Examples:
• Same person with multiple email addresses

• Data cleaning
• Process of dealing with duplicate data issues

1/30/24 CS F415 18
 Data Preprocessing
• Aggregation
• Sampling
• Dimensionality Reduction
• Feature subset selection
• Feature creation
• Discretization and Binarization
• Attribute Transformation

1/30/24 CS F415 19
 Data Reduction Strategies

• A data warehouse may store terabytes of data

• Complex data analysis/mining may take a very long time to run on the complete
data set
• Data reduction
• Obtain a reduced representation of the data set that is much smaller in volume
but yet produce the same (or almost the same) analytical results
• Data reduction strategies
• Data cube aggregation
• Dimensionality reduction — remove unimportant attributes
• Data Compression
• Discretization and concept hierarchy generation

1/30/24 CS F415 20
 Aggregation
• Combining two or more attributes (or objects) into a single
attribute (or object)
• Purpose
• Data reduction
• Reduce the number of attributes or objects
• Change of scale
• Cities aggregated into regions, states, countries, etc.
• More “stable” data
• Aggregated data tends to have less variability

1/30/24 CS F415 21
Data Cube Aggregation
• The lowest level of a data cube
• the aggregated data for an individual entity of interest
• e.g., a customer in a phone calling data warehouse.
• Multiple levels of aggregation in data cubes
• Further reduce the size of data to deal with
• Reference appropriate levels
• Use the smallest representation which is enough to solve the task
• Queries regarding aggregated information should be answered using
data cube, when possible

1/30/24 CS F415 22
SAMPLE CUBE
Total annual sales
Date of TV in U.S.A.
1Qtr 2Qtr 3Qtr 4Qtr sum

t
TV
uc
Total annual sales
od PC ofU.S.A
PC in U.S.A.
Pr
VCR Total salesTotal annual sales
sum Q1 sales
Total In U.S.Aof VCR in U.S.A.

Country
In U.S.A Canada
Total sales
Total Q1 sales
In Canada In Canada Mexico
Total Q1 sales Total sales
In Mexico In Mexico sum
Total Q2 sales
Total Q1 sales
In all countries TOTAL SALES
In all countries

1/30/24 BITS Pilani 23

Aggregation
Variation of Precipitation in Australia

Standard Deviation of Standard Deviation of

Average Monthly Average Yearly
1/30/24 Precipitation CS F415 Precipitation 24
 Sampling
• Sampling is the main technique employed for data selection.
• It is often used for both the preliminary investigation of the data and the final data
analysis.

• Statisticians sample because obtaining the entire set of data of interest is

too expensive or time consuming.

• Sampling is used in data mining because processing the entire set of data
of interest is too expensive or time consuming.

1/30/24 CS F415 25
Sampling …
• The key principle for effective sampling is the following:
• using a sample will work almost as well as using the entire data sets, if the
sample is representative

• A sample is representative if it has approximately the same property (of

interest) as the original set of data

1/30/24 CS F415 26
 Types of Sampling
• Simple Random Sampling
• There is an equal probability of selecting any particular item
• Sampling without replacement
• As each item is selected, it is removed from the population
• Sampling with replacement
• Objects are not removed from the population as they are selected for the
sample.
• In sampling with replacement, the same object can be picked up more than once
• Stratified sampling
• Split the data into several partitions; then draw random samples from each
partition

1/30/24 CS F415 27
 Sampling

• Allow a mining algorithm to run in complexity that is potentially sub-

linear to the size of the data
• Choose a representative subset of the data
• Simple random sampling may have very poor performance in the presence of
skew
• Develop adaptive sampling methods
• Stratified sampling:
• Approximate the percentage of each class (or subpopulation of interest) in the overall
database
• Used in conjunction with skewed data
• Sampling may not reduce database I/Os (page at a time).

1/30/24 CS F415 28
Sampling

W O R
SRS le random
i m p hout
(s e wi t
p l
sam ment)
p l a ce
re

SRSW
R

Raw Data
1/30/24 CS F415 29
Sampling
Raw Data Cluster/Stratified Sample

1/30/24 CS F415 30
 Sample Size

8000 points 2000 Points 500 Points

1/30/24 CS F415 31
 Sample Size
• What sample size is necessary to get at least one
object from each of 10 groups.

1/30/24 CS F415 32
 Data Dimensionality
• From a theoretical point of view, increasing the number of
features should lead to better performance.

• In practice, the inclusion

of more features leads to
worse performance (i.e., curse
of dimensionality).

• The number of training examples required

increases exponentially with dimensionality.
Dimensionality Reduction
• Significant improvements can be achieved by first mapping
the data into a lower-dimensional space.

• Dimensionality can be reduced by:

− Combining features using a linear or non-linear transformations.
− Selecting a subset of features (i.e., feature selection).

34
 Dimensionality Reduction
• Purpose:
• Avoid curse of dimensionality
• Reduce amount of time and memory required by data mining algorithms
• Allow data to be more easily visualized
• May help to eliminate irrelevant features or reduce noise

• Techniques
• Principle Component Analysis
• Singular Value Decomposition
• Others: supervised and non-linear techniques

1/30/24 CS F415 35
Example of
Decision Tree Induction
Initial attribute set:
{A1, A2, A3, A4, A5, A6}
A4 ?

A1? A6?

Class 1 Class 2 Class 1 Class 2

> Reduced attribute set: {A1, A4, A6}

1/30/24 CS F415 36
Curse of Dimensionality

• When dimensionality increases,

data becomes increasingly sparse
in the space that it occupies

• Definitions of density and distance

between points, which is critical
for clustering and outlier
detection, become less meaningful

• Randomly generate 500 points

• Compute difference between max and min
distance between any pair of points
Dimensionality Reduction (cont’d)
• Linear combinations are particularly attractive because they are simple to
compute and analytically tractable.

• Given x ϵ RN, the goal is to find an N x K matrix U such that:

y = UTx ϵ RK where K<<N

UT

38
Dimensionality Reduction (cont’d)
• Idea: represent data in terms of basis vectors in a lower dimensional space
(embedded within the original space).

(1) Higher-dimensional space representation:

(2) Lower-dimensional sub-space representation:

39
Principal Component Analysis
• Given N data vectors from k-dimensions, find c ≤ k orthogonal
vectors that can be best used to represent data
• The original data set is reduced to one consisting of N data vectors on c
principal components (reduced dimensions)
• Each data vector is a linear combination of the c principal component
vectors
• Works for numeric data only
• Used when the number of dimensions is large

1/30/24 CS F415 40
Principal Component Analysis
X2

Y1
Y2

X1

1/30/24 CS F415 41
 Dimensionality Reduction: PCA
• Goal is to find a projection that captures the largest amount of
variation in data
x2

x1

1/30/24 CS F415 42
 Dimensionality Reduction: PCA
• Find the eigenvectors of the covariance matrix
• The eigenvectors define the new space

x2

x1
1/30/24 CS F415 43
Dimensionality Reduction: PCA
Dimensions = 206
Dimensions
Dimensions==160
120
10
40
80

1/30/24 CS F415 44
PCA: Motivation
• Choose directions such that a total variance of data
will be maximum
• Maximize Total Variance

• Choose directions that are orthogonal

• Minimize correlation

• Choose k<d orthogonal directions which maximize

total variance

45
Principal Component Analysis (PCA)
• Dimensionality reduction implies information loss; PCA
preserves as much information as possible by minimizing
the reconstruction error:

• How should we determine the “best” lower dimensional

space?

The “best” low-dimensional space can be determined by the

“best” eigenvectors of the covariance matrix of the data (i.e.,
the eigenvectors corresponding to the “largest” eigenvalues –
also called “principal components”).
46
PCA - Steps

− Suppose x1, x2, ..., xM are N x 1 vectors

(i.e., center at zero)

1
M

47
PCA – Steps (cont’d)

an orthogonal basis

( x  x).ui
where bi 
(ui .ui )

48
PCA – Linear Transformation
( x  x).ui
If ui has unit length: bi   ( x  x).ui
(ui .ui )

• The linear transformation RN  RK that performs the

dimensionality reduction is:

49
 Geometric interpretation
• PCA projects the data along the directions where the data varies
most.
• These directions are determined by the eigenvectors of the
covariance matrix corresponding to the largest eigenvalues.
• The magnitude of the eigenvalues corresponds to the variance of
the data along the eigenvector directions.

50
How to choose K?

• Choose K using the following criterion : Proportion of

Variance (PoV)

• In this case, we say that we “preserve” 90% or 95% of the

information (variance) in the data.
• If K=N, then we “preserve” 100% of the information in the
data.
• Scree graph plots of PoV vs k, stop at “elbow”
51
52
Error due to dimensionality reduction

• The original vector x can be reconstructed using its

principal components:

• PCA minimizes the reconstruction error:

• It can be shown that the reconstruction error is:

53