Scribd
Upload
13 views25 pages

Dimensonality Reduction

sdfgh

Uploaded by

jugal.chhatriwala.spam
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
13 views25 pages

Dimensonality Reduction

sdfgh

Uploaded by

jugal.chhatriwala.spam
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 25

Dimensionality Reduction

• Most of the real-world datasets are having


thousands, or millions of dimensions.
• Problems of having high dimensional data
• The error increases with he increase in the
Curse of number of features
• The computational cost of data
Dimensionality mining/machine learning techniques increases
exponentially.
• The data becomes very sparse in high
dimensional dataset, making the machine
learning/data mining algorithms in effective.
• Overfitting problem in the predictive models.
Dimensionality
Reduction
• Usually, the data can be
described with fewer
dimensions, without losing
much of the meaning of the
data.
• The data reside in a
space of lower
dimensionality
• Visualization: Projection of high dimensional data
onto 2D or 3D.
Why to • Data Compression: Efficient storage and retrieval.
• Noise Removal: Positive effect on accuracy of the
Reduce built model.
• Remove Redundant Features: Positive effect on the
Dimension? performance of the model.
• Hidden Correlations: May find hidden correlations
among features.
• Variance: measure of the deviation from the mean
for points in one dimension e.g., heights
• Covariance: measure of how much each of the
dimensions vary from the mean with respect to
each other.
Covariance • Covariance is measure between two dimension to
see if there is a relationship between the 2
dimensions e.g., number of hours studied, and
marks obtained.
• The covariance between one dimension and itself
is the variance.
Covariance Matrix

• Diagonal is the variances of x, y and z.


• cov (x,y) = cov (y,x) hence matrix is symmetrical about the
diagonal.
• N-dimensional data will result in N x N covariance matrix.
Covariance
Examples
• A positive value of covariance indicates both
dimensions increase or decrease together, e.g. as
the number of hour studied increases, the marks in
that subject increase.
Covariance • A negative value indicates while one increases the
other decreases, or vice versa.
• If covariance is zero: the two dimensions are
independent of each other e.g., height of students
vs marks obtained in a subject.
• PCA is a technique to reduce the dimension of a
dataset without affecting the information.

Principal
• It is a linear transformation that chooses a new
coordinate system for the dataset such that:
• The greatest variance by any projection of
Component the data set comes to lie on first axis (called
the first principal component)
Analysis • The second greatest variance on the second
axis and so on.
• PCA can be used for reducing dimensionality by
eliminating the later principal components.
Geometrical Interpretation
View each point in 3D space.

In this example, all the points happen to belong to a line: a 1D


subspace of the original 3D space.
Geometrical Interpretation
Consider a new coordinate system where one of the axes is along the
direction of the line.

Here every point has only one non-zero coordinate.


PCA-Concept
• Given a set of points, how do we know if they can be compressed like in the previous
example?
• We have to look into the correlation between the points
• By finding the eigenvalues and eigenvectors of the covariance matrix, we find that
the eigenvectors with the largest eigenvalues correspond to the dimensions that the
strongest correlation in the dataset.
• This is the principal component.
PCA-Theorem
PCA-Theorem
PCA-Theorem

Generally:
1. Q is square
2. Q is symmetric
3. Q is the covariance matrix
PCA-Theorem
Each can be written as:

Where, are the n eigenvectors of Q with non-zero eigenvalues.


Note:

1. The eigenvectors span an eigenspace.


2. These are N x 1 orthogonal vectors (directions in N-Dimensional space.
3. The scalars are the coordinates of in the space.
Using PCA to Compress Data
• Expressing x in terms of has not changed the size of the data.
• If the points are highly correlated many of the coordinates of x will be
zero or close to zero.
• Sort the eigenvectors according to their eigenvalue.
PCA Example-Step 1
PCA Example-Step 2
Calculate the covariance matrix.

Since the non-diagonal elements in this covariance


matrix are positive, we should expect that both x
and y variable increase together.
PCA Example-Step 3
Calculate the eigenvectors and eigenvalues of the covariance matrix.
PCA Example-Step 3

• Eigenvectors are plotted as diagonal


dotted lines on the plot.
• They are perpendicular to each
other.
• One of the eigenvectors goes
through the middle of the points,
like drawing a line of best fit.
PCA Example-Step 4
PCA Example-Step 5
PCA Example-Step 5
Thank You

You might also like