Principal components analysis (PCA)

PREPARED BY:
MISRAK BIRHANU
DAWIT TESEMA
SAMRAWIT ABERA
Outline
•Introduction
•Why Principal Component Analysis (PCA)?
•Mathematical Background & Example of PCA
•How it Works of PCA?
•Application of Principal component analysis ?
•References
What is Principal Component Analysis (PCA)
PCA is a method of extracting important variables (in form of components) from a large
set of variables available in a data set.

It extracts low dimensional set of features from a high dimensional data set with a motive
to capture as much information as possible.

PCA is used to extract the important information from a multivariate data table and to
express this information as a set of few new variables called principal components.

◦ These new variables correspond to a linear combination of the originals. The number of
principal components is less than or equal to the number of original variables.
 What…
PCA is a technique that can be used to simplify a dataset
PCA seeks a linear combination of variables such that the maximum variance is
extracted from the variables.
◦ It then removes this variance and seeks a second linear combination which explains
the maximum proportion of the remaining variance, and so on. This is called the
principal axis method and results in orthogonal (uncorrelated) factors. PCA analyzes
total (common and unique) variance.
Why Principal Component Analysis (PCA)?
The main goal of PCA is:
1. To Identify hidden Patterns in the dataset
2. To reduce the dimensionality of the data by removing the noise and redundancy in
the data.
3. To identify correlated variables
 PCA method is particular useful when the variables with in the data set are highly
correlated.
 Correlation indicates that there is redundancy in the data. Due to redundancy, PCA
can be used to reduce the original variables into a smaller number of new variable is
called Principal component.
 Applications of Principal Component
Analysis
 Computer vision:-matrix techniques in computer vision, we must consider representation of
images. A square, N by N image can be expressed as an dimensional vector.

 To find patterns:- create an image vector and put all the images together in one big image-
matrix in order to analyze. Eg. Face recognition system.

 Image compression:- new image is reproduced with lost of some information.

Mathematical Background of PCA
Standard Deviation
1. Assume a data set

2. Calculate Mean

 Notice the symbol (said “X bar”) to indicate the mean of the set . All this formula says is “Add up
all the numbers and then divide by how many there are”.

Unfortunately, the mean doesn’t tell us a lot about the data except for a sort of middle point. For example,
these two data sets have exactly the same mean, but are obviously quite different:
 Mathematical Background of PCA
Cont..
So what is different about these two sets? It is the spread of the data.

3. Calculate Standard Deviation

The Standard Deviation (SD) of a data set is a measure of how spread out the data is.

The average distance from the mean of the data set to a point”. The way to calculate it is to
compute the squares of the distance from each data point to the mean of the set, add them all
up, divide by n-1, and take the positive square root. As a formula:
 Mathematical Background of PCA
Cont..
Dataset 1 Dataset 2

 And so, as expected, the first set has a much larger standard deviation due to the fact that the
data is much more spread out from the mean
 also has a mean of 10, but its standard deviation is 0, because all the numbers are the same.
None of them deviate from the mean.
Mathematical Background of PCA
Cont..
Variance is another measure of the spread of data in a data set. In fact it is almost
identical to the standard deviation. The formula is this:

Standard deviation and variance only operate on 1 dimension, so that you could
only calculate the standard deviation for each dimension of the data set independently
of the other dimensions.

Covariance is always measured between 2 dimensions. If you calculate the covariance

between one dimension and itself, you get the variance.
Mathematical Background & example
Cont..
Covariance
N(number of dataset)=12
Mathematical Background …Matrices
The mathematical concept of a matrix refers to a set of numbers, variables or functions ordered
in rows and columns. Then it can be manipulated as a whole according to some basic
mathematical rules.

Two matrices can be multiplied together if the number of columns in the first matrix [A] must
be equal to the number of rows in the second matrix [B]. The resulting matrix [C] will have the
same number of rows as [A] and the same number of columns as [B].
 Mathematical Background …Matrices
Let A be an nXn matrix. A scalar  is called an eigenvalue of A if there is a nonzero vector
such that A = . Such a vector is called an eigenvector of A corresponding to 

 2 3 2 
Show that =   is an eigenvector of A=   corresponding to =4
1 3  2

3 2   2  2
     
 3  2  1 1

Note: X-bar is Eigenvector

How it works PCA
Step 2: PCA…cont.
2. Calculate the covariance matrix: used to compute two dimension dataset
using the formula
 Step 3: PCA…cont.
3. Calculate the (unit) eigenvectors and eigenvalues of the covariance matrix:

Steps to execute: 1. compute I (multiply lamda with identity matrix )
 
2. Compute A- I (Subtract the given matrix A from product of I)
3. Compute determinant and find eigenvalue
Eigenvectors : from PCA
reflect both common and 4. using eigenvalue compute eigenvector with matrices reduction technique
unique variance of variables 5. Check A = 
Eigenvalues : measure the
amount of variation in the
total sample.
Step 4: PCA…cont.
4. Once eigenvectors are found from the covariance matrix, the next step is to order them by
eigenvalue, highest to lowest. This gives you the components in order of significance.

 Ignore the components of lesser significance and we might lose some information, but if the
eigenvalues are small, we don’t lose much information. If we leave out some components, the
final data set will have less dimensions than the original.

 To be precise, if we originally have N dimensions in our data and so we calculate N

eigenvectors and eigenvalues, and then we choose only the first p eigenvectors, then the final
data set has only p dimensions.
Step 5: PCA…cont.
5. Choosing components and forming a feature vector
 Construct new feature vector by taking the eigenvectors that you want to keep from the list
of eigenvectors, and forming a matrix with these eigenvectors in the columns.

  .677873399  .735178956 
FeatureVector1   
  .735178956 .677873399  With in the example : we have two
choices. We can either form a feature
or reduced dimension feature vector : vector with both of the eigenvectors or, we
can choose to leave out the smaller, less
significant component and only have a
  .677873399  single column.
FeatureVector 2   
  .735178956 
Step 6: PCA…cont.
6. Derive the new data set : This the final step in PCA.
 Once we have chosen the components (eigenvectors) that we wish to keep in our data and
formed a feature vector, we simply take the transpose of the vector and multiply it on the left
of the original data set, transposed.

Row Feature Vector is the matrix with the eigenvectors in the columns transposed so that the
eigenvectors are now in the rows, with the most significant eigenvector at the top.

Row Data Adjust is the mean-adjusted data transposed, ie. the data items are in each column, with
each row holding a separate dimension.
 Step 6: PCA…cont.
TransformedData = RowFeatureVector  RowDataAdjust

RowFeatureVector 2   .677873399  .735178956

 .69  1.31 .39 .09 1.29 .49 .19  .81  .31  .71 
RowDataAdjust   
 .49  1.21 .99 .29 1.09 .79  .31  .81  .31  1.01
Final result of original data restored using
only a single eigenvector
This single eigenvector are
plotted as diagonal doted lines
on the plot. Which gives us
important information and
shows patters in the data.
References
 A tutorial on Principal Components Analysis Lindsay I Smith February 26, 2002
 Basic matrices concepts
 Finding Egin Value and Egin vector Video tutorials
 PRINCIPAL COMPONENT ANALYSIS IN IMAGE PROCESSING M. Mudrov´a, A.
Proch´azka
 A Tutorial on Principal Component Analysis Jonathon Shlens
Thank You !