Scribd
Upload
13 views39 pages

MLSP-6 Dimensionality Reduction

The document discusses dimensionality reduction techniques in machine learning, focusing on the curse of dimensionality and methods to overcome it, such as incorporating prior knowledge and reducing dimensionality. It explains Linear Discriminant Analysis (LDA) and Principal Component Analysis (PCA) as key techniques for transforming high-dimensional data into lower-dimensional representations while preserving essential information. The document also outlines the steps involved in the PCA algorithm and provides examples of its application.

Uploaded by

pranayjn14
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
13 views39 pages

MLSP-6 Dimensionality Reduction

The document discusses dimensionality reduction techniques in machine learning, focusing on the curse of dimensionality and methods to overcome it, such as incorporating prior knowledge and reducing dimensionality. It explains Linear Discriminant Analysis (LDA) and Principal Component Analysis (PCA) as key techniques for transforming high-dimensional data into lower-dimensional representations while preserving essential information. The document also outlines the steps involved in the PCA algorithm and provides examples of its application.

Uploaded by

pranayjn14
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 39

Machine learning for Signal

Processing (MLSP)
Dimensionality Reduction
Course Instructor
Prof. Jyotsna Singh
curse of dimensionality
• In practice, the curse of dimensionality means that, for a
given sample size, there is a maximum number of features
above which the performance of our classifier will degrade
rather than improve
In most cases, the additional information that is lost by
discarding some features is compensated by a more accurate
mapping in the lower dimensional space
• How do we beat the curse of dimensionality?
• By incorporating prior knowledge
• By providing increasing smoothness of the
target function
• By reducing the dimensionality
Dimensionality Reduction
• In pattern recognition, Dimension Reduction is
defined as-
• It is a process of converting a data set having
vast dimensions into a data set with lesser
dimensions.
• It ensures that the converted data set conveys
similar information concisely.
Dimensionality Reduction
Dimensionality Reduction
Signal representation versus classification
Linear Discriminant Analysis
• LDA is a dimensionality reduction technique used
as a pre-processing step for Pattern recognition or
Machine Learning Application.
• LDA is similar to PCA, But LDA in addition finds the
axis that maximises the separation between
multiple classes.
• It projects the N dimensional feature space onto a
smaller subspace M (M<=N), While maintaining
the class discriminatory Information.
Linear Discriminant Analysis
Linear Discriminant Analysis, two-classes
Fisher linear discriminant
Linear Discriminant Analysis, two-classes
Linear Discriminant Analysis, two-classes
Fisher LDA for C classes
Fisher LDA for C classes
Fisher LDA for C classes
PCA Algorithm

Prof. Jyotsna Singh


How does PCA achieve dimension
reduction?
• 1.Principal Component analysis reduces high dimensions
into low dimension subspace by creating a new set of
components that carry most of the essential information
of all the features.
• 2.These new components are a linear combination of all
the features and the components thus formed are nothing
but the eigenvectors which are now called the Principal
components.
• 3.The eigenvalue corresponding to each of these
eigenvectors will tell us about how much variation in the
data has been captured by that particular Principal
component.
• 4. Principal components are orthogonal to each other
and are uncorrelated that increases maximum variance.
As they are uncorrelated it solves the problem
of multicollinearity.
The steps involved in PCA Algorithm are as follows-

• Step-01: Get data.


• Step-02: Compute the mean vector (µ).
• Step-03: Subtract mean from the given data.
• Step-04: Calculate the covariance matrix.
• Step-05: Calculate the eigen vectors and eigen values
of the covariance matrix.
• Step-06: Choosing components and forming a feature
vector.
• Step-07: Deriving the new data set.
• Problem-01:

• Consider the two dimensional patterns (2, 1),


(3, 5), (4, 3), (5, 6), (6, 7), (7, 8).
• Compute the principal component using
PCA Algorithm.
• Solution-
•
• We use the above discussed PCA Algorithm-

• Step-01:

• Get data. The given feature vectors are-


• x1 = (2, 1)
• x2 = (3, 5)
• x3 = (4, 3)
• x4 = (5, 6)
• x5 = (6, 7)
• x6 = (7, 8)
• Step-02:

• Calculate the mean vector (µ).


• Mean vector (µ)
= ((2 + 3 + 4 + 5 + 6 + 7) / 6, (1 + 5 + 3 + 6 +
7 + 8) / 6)
= (4.5, 5)
• Step-03:
•
• Subtract mean vector (µ) from the given feature
vectors.
• x1 – µ = (2 – 4.5, 1 – 5) = (-2.5, -4)
• x2 – µ = (3 – 4.5, 5 – 5) = (-1.5, 0)
• x3 – µ = (4 – 4.5, 3 – 5) = (-0.5, -2)
• x4 – µ = (5 – 4.5, 6 – 5) = (0.5, 1)
• x5 – µ = (6 – 4.5, 7 – 5) = (1.5, 2)
• x6 – µ = (7 – 4.5, 8 – 5) = (2.5, 3)
• Covariance matrix is given by-

• Now, Covariance matrix


= (m1 + m2 + m3 + m4 + m5 + m6) / 6
• The covariance matrix will be

• Step-05:
• Calculate the eigen values and eigen vectors of
the covariance matrix.
• λ is an eigen value for a matrix M if it is a
solution of the characteristic equation |M – λI| =
0.
• So, we have-
• From here,
• (2.92 – λ)(5.67 – λ) – (3.67 x 3.67) = 0
• 16.56 – 2.92λ – 5.67λ + λ2 – 13.47 = 0
• λ2 – 8.59λ + 3.09 = 0
• Solving this quadratic equation, we get λ = 8.22, 0.38
• Thus, two eigen values are λ1 = 8.22 and λ2 = 0.38.
•
• Clearly, the second eigen value is very small compared to
the first eigen value.
• So, the second eigen vector can be left out.
•
• Eigen vector corresponding to the greatest eigen value is
the principal component for the given data set.
• So. we find the eigen vector corresponding to eigen value
λ1.
• We use the following equation to find the eigen
vector-
• MX = λX
• where-
• M = Covariance Matrix
• X = Eigen vector
• λ = Eigen value
•
• Substituting the values in the above equation, we
get-
• Solving these, we get-
• 2.92X1 + 3.67X2 = 8.22X1
• 3.67X1 + 5.67X2 = 8.22X2
• On simplification, we get-
• 5.3X1 = 3.67X2 ………(1)
• 3.67X1 = 2.55X2 ………(2)

• From (1) and (2), X1 = 0.69X2


• From (2), the eigen vector is-
• Lastly, we project the data points onto the new
subspace as-
• Problem-02:
• Use PCA Algorithm to transform the pattern (2, 1) onto the
eigen vector in the previous question.
• Solution-
• The given feature vector is (2, 1).
• The feature vector gets transformed to
= Transpose of Eigen vector x (Feature Vector – Mean Vector)

You might also like