Scribd Logo
Upload

Welcome to Scribd!

  • 6 views

    Dimensionality Reduction

    Uploaded by

    Yashwanth Yashu
    This document discusses dimensionality reduction and principal component analysis (PCA). It provides the following key points: 1) Dimensionality reduction is used to simplify complex high-dimensional data by summarizing it with a lower dimensional real-valued vector, with minimal loss of information. 2) PCA finds the directions (principal components) along which the data has the most variance. It projects the data onto these principal components to reduce dimensionality while preserving as much information as possible. 3) PCA works by computing the eigenvalues and eigenvectors of the covariance matrix of the data. The principal components are the eigenvectors corresponding to the largest eigenvalues. Projecting the data onto these principal components reduces dimensionality.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd

    Dimensionality Reduction

    Uploaded by

    Yashwanth Yashu
    6 views33 pages
    This document discusses dimensionality reduction and principal component analysis (PCA). It provides the following key points: 1) Dimensionality reduction is used to simplify complex high-dimensional data by summarizing it with a lower dimensional real-valued vector, with minimal loss of information. 2) PCA finds the directions (principal components) along which the data has the most variance. It projects the data onto these principal components to reduce dimensionality while preserving as much information as possible. 3) PCA works by computing the eigenvalues and eigenvectors of the covariance matrix of the data. The principal components are the eigenvectors corresponding to the largest eigenvalues. Projecting the data onto these principal components reduces dimensionality.

    Original Title

    Pca

    Copyright

    © © All Rights Reserved

    Available Formats

    PPTX, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    This document discusses dimensionality reduction and principal component analysis (PCA). It provides the following key points: 1) Dimensionality reduction is used to simplify complex high-dimensional data by summarizing it with a lower dimensional real-valued vector, with minimal loss of information. 2) PCA finds the directions (principal components) along which the data has the most variance. It projects the data onto these principal components to reduce dimensionality while preserving as much information as possible. 3) PCA works by computing the eigenvalues and eigenvectors of the covariance matrix of the data. The principal components are the eigenvectors corresponding to the largest eigenvalues. Projecting the data onto these principal components reduces dimensionality.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Download as pptx, pdf, or txt
    6 views33 pages

    Dimensionality Reduction

    Uploaded by

    Yashwanth Yashu
    This document discusses dimensionality reduction and principal component analysis (PCA). It provides the following key points: 1) Dimensionality reduction is used to simplify complex high-dimensional data by summarizing it with a lower dimensional real-valued vector, with minimal loss of information. 2) PCA finds the directions (principal components) along which the data has the most variance. It projects the data onto these principal components to reduce dimensionality while preserving as much information as possible. 3) PCA works by computing the eigenvalues and eigenvectors of the covariance matrix of the data. The principal components are the eigenvectors corresponding to the largest eigenvalues. Projecting the data onto these principal components reduces dimensionality.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Download as pptx, pdf, or txt
    You are on page 1of 33

    Dimensionality Reduction

    Motivation
    • Clustering
    • One way to summarize a complex real-valued data point with a single
    categorical variable
    • Dimensionality reduction
    • Another way to simplify complex high-dimensional data
    • Summarize data with a lower dimensional real valued vector
    Motivation
    • Clustering
    • One way to summarize a complex real-valued data point with a single
    categorical variable
    • Dimensionality reduction
    • Another way to simplify complex high-dimensional data
    • Summarize data with a lower dimensional real valued vector

    • Given data points in d dimensions


    • Convert them to data points in r<d dimensions
    • With minimal loss of information
    Data Compression
    Reduce data from
    (inches)

    2D to 1D

    (cm)
    Data Compression
    Reduce data from
    (inches)

    2D to 1D

    (cm)

    Andrew Ng
    Data Compression
    Reduce data from 3D to 2D
    Principal Component Analysis (PCA) problem formulation

    Reduce from 2-dimension to 1-dimension: Find a direction (a vector )


    onto which to project the data so as to minimize the projection error.
    Reduce from n-dimension to k-dimension: Find vectors
    onto which to project the data, so as to minimize the projection error.

    Andrew Ng
    Covariance
    • Variance and Covariance:
    • Measure of the “spread” of a set of points around their center of mass(mean)
    • Variance:
    • Measure of the deviation from the mean for points in one dimension
    • Covariance:
    • Measure of how much each of the dimensions vary from the mean with
    respect to each other

    • Covariance is measured between two dimensions


    • Covariance sees if there is a relation between two dimensions
    • Covariance between one dimension is the variance
    Positive: Both dimensions increase or decrease together Negative: While one increase the other decrease
    Covariance

    • Used to find relationships between dimensions in high dimensional


    data sets

    The Sample mean


    Eigenvector and Eigenvalue
    Ax = λx
    A: Square Matirx
    λ: Eigenvector or characteristic vector
    X: Eigenvalue or characteristic value
    • The zero vector can not be an eigenvector
    • The value zero can be eigenvalue
    Eigenvector and Eigenvalue
    Ax = λx
    A: Square Matirx
    λ: Eigenvector or characteristic vector
    X: Eigenvalue or characteristic value

    Example
    Eigenvector and Eigenvalue
    Ax - λx = 0
    Ax = λx
    (A – λI)x = 0

    If we define a new matrix B:


    B = A – λI
    Bx = 0
    BUT! an eigenvector
    If B has an inverse: x=B 0=0-1
    cannot be zero!!

    x will be an eigenvector of A if and only if B does


    not have an inverse, or equivalently det(B)=0 :
    det(A – λI) = 0
    Eigenvector and Eigenvalue
    Example 1: Find the eigenvalues of
    2  12
    A 
     1  5 
      2 12
    I  A   (  2)(  5)  12
    1   5
     2  3  2  (  1)(  2)
    two eigenvalues: 1,  2
    Note: The roots of the characteristic equation can be repeated. That is, λ1 = λ2 =…= λk. If that happens, the
    eigenvalue is said to be of multiplicity k.
    2 1 0
    Example 2: Find the eigenvalues of
    A  0 2 0
     2 1 0 0 0 2
    I  A  0  2 0  (  2)3  0
    0 0  2
    Principal Component Analysis
    Input:

    Set of basis vectors:

    Summarize a D dimensional vector X with K dimensional


    feature vector h(x)
    Principal Component Analysis

    Basis vectors are orthonormal

    New data representation h(x)


    Principal Component Analysis

    New data representation h(x)

    Empirical mean of the data


    SIFT feature visualization

    • The top three principal components of SIFT descriptors from a set of images are computed
    • Map these principal components to the principal components of the RGB space
    • pixels with similar colors share similar structures
    Application: Image compression

    Original Image

    • Divide the original 372x492 image into patches:


    • Each patch is an instance that contains 12x12 pixels on a grid
    • View each as a 144-D vector
    PCA compression: 144D  60D
    PCA compression: 144D  16D
    16 most important eigenvectors
    2 2 2 2
    4 4 4 4
    6 6 6 6
    8 8 8 8
    10 10 10 10
    12 12 12 12
    2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12

    2 2 2 2
    4 4 4 4
    6 6 6 6
    8 8 8 8
    10 10 10 10
    12 12 12 12
    2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12

    2 2 2 2
    4 4 4 4
    6 6 6 6
    8 8 8 8
    10 10 10 10
    12 12 12 12
    2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12

    2 2 2 2
    4 4 4 4
    6 6 6 6
    8 8 8 8
    10 10 10 10
    12 12 12 12
    2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12
    PCA compression: 144D ) 6D
    6 most important eigenvectors

    2 2 2
    4 4 4
    6 6 6
    8 8 8
    10 10 10
    12 12 12
    2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12

    2 2 2
    4 4 4
    6 6 6
    8 8 8
    10 10 10
    12 12 12
    2 4 6 8 10 12 2 4 6 8 10 12 2 4 6 8 10 12
    PCA compression: 144D 
    3D
    3 most important eigenvectors
    2 2

    4 4

    6 6

    8 8

    10 10

    12 12
    2 4 6 8 10 12 2 4 6 8 10 12

    10

    12
    2 4 6 8 10 12
    PCA compression: 144D 
    1D
    60 most important eigenvectors

    Looks like the discrete cosine bases of JPG!...

    You might also like