FEM 2063 - Data Analytics

CHAPTER 5
At the end of this chapter
students should be able to
understand:
Dimensionality
Reduction Methods

1
 OVERVIEW

➢Singular Value Decomposition (SVD)

➢Principal Components Analysis (PCA)

2
Unsupervised Learning – Dimension Reduction
 Unsupervised Learning
Datasets in the form of matrices
We are given n objects and p features describing the
objects.

Dataset
An n-by-p matrix A
n rows representing n objects
Each object has p numeric values describing it.

Goal
1. Understand the structure of the data, e.g., the
underlying process generating the data.
2. Reduce the number of features representing the data
 Unsupervised Learning
Example - Market basket matrices

p products (e.g., milk, bread, rice, etc.)

 
 --------------------------------------- 
-------------------------------------------- Aij= quantity of j-th product
n customers purchased by the i-th
 A  customer
 
 
Aim: find a subset of the products that characterize customer behavior
 5 Unsupervised Learning
Dimensionality reduction method:
• Singular Value Decomposition (SVD)
• Principal Components Analysis (PCA)
• Canonical Correlation Analysis (CCA)
• Multi-dimensional scaling (MDS)
• Independent component analysis (ICA)
 Overview

➢Singular Value Decomposition (SVD)

➢Principal Components Analysis (PCA)

7
 SVD – general overview
The singular value decomposition (SVD) provides another way to factorize a
matrix, into singular vectors and singular values. The SVD is used widely both
in the calculation of other matrix operations, such as matrix inverse, but also as a
data reduction method in machine learning. Data matrices have n rows (one for
each object) and p columns (one for each feature).

Data
Matrix

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 8 8
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright
Internalowner.
 SVD – general overview
Singular Value Decomposition (SVD) is a widely used technique to decompose
a matrix into several component matrices, exposing many of the useful and
interesting properties of the original matrix.

n x n matrix
m x n matrix *Rows of VT = Right Singular Value
Data matrix *Column of V = Orthonormal
eigenvector ATA
n
m x m matrix m x n diagonal matrix
*Column of U = Left singular *Nonzero value in diagonal = Singular
n
value Value
*Orthonormal eigenvector *Diagonal matrix = square roots of
n
m AAT eigenvalue of U and V in descending
order VT
m
m

S
U
© 2021 UNIVERSITI TEKNOLOGI PETRONAS
All rights reserved. 9 9
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright
Internalowner.
 SVD – general overview

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 10 10
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright
Internalowner.
SVD – general overview


1
SVD - Singular What are singular values in a matrix?

values The singular values are the diagonal entries of

the S matrix and are arranged in descending
order. The singular values are always real
numbers. If the matrix A is a real matrix, then U and
V are also real. The values of x1 and x2 are chosen
such that the elements of the S are the square roots
of the eigenvalues.

1: measures how much of the data variance is

explained by the first singular vector.
1st (right) singular vector:
direction of maximal variance
2: measures how much of the data variance is
explained by the second singular vector.
2nd (right) singular vector:
direction of maximal variance, after removing the
projection of the data along the first singular vector.
Why SVD
 SVD – step by step

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 14
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 SVD – Example

Note:
5 – most preferred
© 2021 UNIVERSITI TEKNOLOGI PETRONAS 0 – not preferred
All rights reserved. 15
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 SVD – Example

Represent 53.44%
User 1 of the dataset
User 2 Represent 40.95%
of the dataset
User 3
User 4
User 5
Represent 5.61% of
User 6 the dataset
User 7

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 16
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 SVD – Example (Users-to-Movies)

Represent 90% of
User 1 the dataset
User 2
User 3
User 4
User 5
User 6
User 7

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 17
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 SVD – Example (Users-to-Movies)

Conclusion: It is observed that the amount percentage variance explained (explained variance ratio) by each principal component
© 2021 UNIVERSITI TEKNOLOGI PETRONAS
18
All rights reserved.
has different value: 53.44% of the variance is explained by the 1st component, 40.95% by the 2nd component, 15.46%
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 • SVD – Example (Users to Movies)

What can you

observe
between
original matrix
and matrix after
SVD?

Original Matrix Matrix after SVD

© 2021 UNIVERSITI TEKNOLOGI PETRONAS
All rights reserved. 19
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 SVD Applications
➢Using SVD in computation rather than A has the advantage of being
more robust to numerical error
➢SVD usually found by iterative methods
➢Other applications:
➢Inverse of matrix A
➢Conditions of matrix
➢Image compression
➢Solve Ax=b for all cases (unique, many, no solutions)
➢Rank determination, matrix approximation
Note: Rank of a matrix is defined as
(a) the maximum number of linearly independent column vectors in the matrix or
(b) the maximum number of linearly independent row vectors in the matrix. Both
definitions are equivalent.
© 2021 UNIVERSITI TEKNOLOGI PETRONAS
For an r x c matrix, If r is less than c, then the maximum rank of the matrix is r.
All rights reserved. 20
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 SVD Applications and Example

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 21
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 How SVD is used in image compression?
SVD Applications and Example In this method, digital image is given to
SVD. SVD refactors the given digital
image into three matrices. Singular values
are used to refactor the image and at the end
of this process, image is represented with
smaller set of values, hence reducing the
storage space required by the image.

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 22
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 SVD Example

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 23
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 SVD
Example computation:
 SVD
Example computation:
 5.2 SVD - Advantages
➢SVD is stable, small change in the input results small change in the
singular matrix
➢Compression speed in SVD is also high
➢Decomposition provides low rank approximation to A
➢There exist efficient, stable algorithms to compute the SVD

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 26
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 Overview

➢Singular Value Decomposition (SVD)

➢Principal Components Analysis (PCA)

27
 Principal Component Analysis (PCA)
Principal Component Analysis, PCA is a “dimensionality reduction” method. It
reduces the number of variables that are correlated to each other into fewer
independent variables without losing the essence of these variables. It
provides an overview of linear relationships between inputs and variables.

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 28 28
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright
Internalowner.
 Principal Component Analysis (PCA)
PCA is a statistical procedure that How is PCA used?
converts a set of observations of possibly Principal Component Analysis (PCA) is
correlated variables into a set of values of used to explain the variance-covariance
linearly uncorrelated variables called structure of a set of variables through
principal components. PCA is often linear combinations.
used to simplify data, reduce noise,
and find unmeasured “latent variables”. What is PC1 and PC2 in PCA?
It finds directions of maximal variance Principal components are created in
of data, directions that are mutually order of the amount of variation they
orthogonal. The relationship between cover: PC1 captures the most
variance and information is the larger the variation, PC2, the second most, and
variance carried by a line, the larger the so on. Each of them contributes some
dispersion of the data points along it, information of the data, and in a PCA,
and the larger the dispersion along a there are as many principal components
line, the more the information it has. as there are characteristics.
© 2021 UNIVERSITI TEKNOLOGI PETRONAS
All rights reserved. 29
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright
Internalowner.
How PCA works?
PC2 is oriented such
PCA creates a that it reflects the
visualization of data that second largest source
minimizes residual of variation in the data
variance in the least while being orthogonal
squares sense and to the first PC. PC2 also
maximizes the variance passes through the
of the projection average point.
coordinates.

Two PCs form a

PC1 is the line that best plane. This plane
accounts for the shape of is a window into
the point swarm. It the
represents the maximum multidimensional
variance direction in the space, which can
data. Each observation be visualized
(yellow dot) may be graphically. Each
projected onto this line in observation may
order to get a coordinate be projected onto
value along the PC-line. This this plane, giving
value is known as a score. a score for each.
 SVD vs PCA
What are the differences/similarities between SVD and PCA?
• SVD and PCA are two eigenvalue methods used to reduce a high-dimensional
data set into fewer dimensions while retaining important information.
As PCA uses the SVD in its calculation, clearly there is some 'extra' analysis
done.
• SVD gives you the whole nine-yard of diagonalizing a matrix into special
matrices that are easy to manipulate and to analyze. It lay down the foundation
to untangle data into independent components. PCA skips less significant
components.
 SVD vs PCA

Now you
may
compare the
scores.
Any idea?
 SVD vs PCA

PCA scores are

-mean centred
- uncorrelated
 5.3c How to construct PCA
Steps:
1. Scale / Normalize the Data (A)
2. Calculate covariance matrix from the dataset.
3. Find the eigenvalue and eigenvectors from the covariance matrix.
4. Find the PC in the covariance matrix using SVD. A=USV’ >>> AV=US
=PC, Principal Components
5. Compare the variance of each PC in the dataset.
6. The highest variance indicated the suitable PC to perform in the
modeling.

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 34
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 PCA – Assumptions
Assumptions of PCA
1. Independent variables are highly correlated to each other. The correlation
coefficient, r, tells us about the strength and direction of the linear
relationships between variables. In the case of more than two variables,
use the correlation matrix.
2. Variables included are metric level or nominal level.
3. Features are low dimensional in nature.
4. Independent variables are numeric in nature.

When to use PCA?

• Whenever we want to ensure that variables in data are independent to
each other.
• When we want to reduce the number of variables in a data set with many
variables in it.
• When we want to interpret data and variable selection out of it.
© 2021 UNIVERSITI TEKNOLOGI PETRONAS
All rights reserved. 35
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 How to construct PCA? STEP 1: Scale / Normalize the Data (A)

Do I need to scale before PCA?

• PCA is sensitive to scale
Yes, it is necessary to normalize data before
• PCA should be applied on
performing PCA. The PCA calculates a new projection
data that have approximately
of data set. After normalizing the data, all variables have
the same scale in each
the same standard deviation, thus all variables have the
variable.
same weight and the PCA calculates relevant axis

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 36
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 How to construct PCA? STEP 2: Covariance Matrix Computation
The aim of this step is to understand how the variables of the input data set
are varying from the mean with respect to each other. Variables are highly
correlated in such a way that they contain redundant information. So, in
order to identify these correlations, we compute the covariance matrix.
• Can variance tell the data is in difference orientation? No.
• The data orientation can be answered by covariance.
• The covariance can show the positive and negative correlation.

Covariance” indicates
the direction of the
linear relationship
between variables.

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 37
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 How to construct PCA?
STEP 3: Compute the Eigenvectors and Eigenvalues of the Covariance
Matrix by Identify the Principal Components
Eigenvectors and eigenvalues are computed from the covariance matrix in order
to determine the principal components of the data.
Information in principal components will allow to reduce dimensionality without
losing much information, and this by discarding the components with low
information and considering the remaining components as your new
variables.

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 38
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 PCA - Example
Understanding USArrests data using PCA by Hemang Goswami

This data set contains arrests per 100,000 residents for assault, murder,
and rape in each of the 50 US states in 1973. Also given is the percent
of the population living in urban areas.
A data frame with 50 observations on 4 variables.
•Murder numeric Murder arrests (per 100,000)
•Assault numeric Assault arrests (per 100,000)
•UrbanPop numeric Percent urban population
•Rape numeric Rape arrests (per 100,000)
The size of the matrix is 50 states x 4 variables – which is very
large to be processed if it is not reduced.
Link: https://rstudio-pubs-static.s3.amazonaws.com/377338_75ed92a8463d482a80045abcae0e395d.html
© 2021 UNIVERSITI TEKNOLOGI PETRONAS
All rights reserved. 39
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 PCA - Example
Data values are not in the same scales and the means,
Data Structure medians and variances are not in the same range

Data Summary

Link: https://rstudio-pubs-static.s3.amazonaws.com/377338_75ed92a8463d482a80045abcae0e395d.html

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 40
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 PCA - Example
Correlation Matrix Correlation Matrix

Before Scaling After Scaling

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 41
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 PCA - Example
Principal components
The rotation matrix provides the principal component loadings vector.
Mean Variance

We create the principal components for the four variables to explain the variance vectors
in the dataset without including the correlation between variables (PC1,PC2,PC 3, and
PC 4).The rotation matrix provides the principal component loadings vector.

The amount of variance explained by each principal component:

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 42
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 PCA - Choosing the number of required Principal components
The percentage of variance explained by each principal component:
62% of the variance is explained by the first principal component, 25% by the second
principal component, 9% by the third principal component and the remaining 4% by
the last principal component.
Hence a large proportion of the variance is explained by the first 2 principal components.
The points after which the variation explained starts to drop off is called as the elbow
point. A fair amount of variance is explained by the first two principal components, and
that there is an elbow after the second component. Third principal component explained
less than 10% variance and the last was almost negligible. Hence, we decide to go with
two principal components

Elbow point

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 43
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 PCA - Example
Creating 2 principal components

PC values here represents

vector for the components

Checking the weights of 2 principal components, we see that:

•The first loading vector places approximately equal weight on Assault, Murder, and
Rape, with much less weight on UrbanPop. Hence this component roughly
corresponds to a measure of overall rates of serious crimes.
•The second loading vector places most of its weight on UrbanPop and much less
weight on the other three features. Hence, this component roughly corresponds to
the level of urbanization of the state.
© 2021 UNIVERSITI TEKNOLOGI PETRONAS
All rights reserved. 44
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 PCA - Example
The biplot shows that 50 states mapped to the 2
principal components. The vectors of the PCA for 4
variables are also plotted.

•The large positive scores on the first component, such

as California, Nevada and Florida, have high crime
rates, while states like North Dakota, with negative
scores on the first component, have low crime rates.
•California also has a high score on the second
component, indicating a high level of urbanization,
while the opposite is true for states like Mississippi.
•States close to zero on both components, such as
Indiana, have approximately average levels of both
crime and urbanization.
© 2021 UNIVERSITI TEKNOLOGI PETRONAS
All rights reserved. 45
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 PCA - Example Checking the principal components scores
vector for all 50 states: basis vector multiply
with original data (scaled data)

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 46
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 PCA - Example
Modeling a Data Set
Data set of foods commonly consumed in
different European countries. Figure below
displays the score plot of the first two principal
components called t1 and t2. The score plot is
a map of 16 countries. Countries close to each
other have similar food consumption profiles,
whereas those far from each other are
dissimilar. The Nordic countries (Finland,
Norway, Denmark and Sweden) are located
together in the upper right-hand corner, thus
representing a group of nations with some The PCA score plot of the first two PCs of a data set
similarity in food consumption. about food consumption profiles. This provides a map of
Belgium and Germany are close to the center how the countries relate to each other. The first
(origin) of the plot, which indicates they have component explains 32% of the variation, and the
second component 19%. Colored by geographic
average properties. location (latitude) of the respective capital city.
 PCA - Example
How to Interpret the Score Plot
In a PCA model with two components, which variables (food provisions) are responsible
for the patterns seen among the observations (countries)? We would like to know which
variables are influential, and how the variables are correlated. Such knowledge is given
by the principal component loadings (graph below) called p1 and p2.
The figure below displays the relationships between all 20 variables at the same time.
Variables contributing similar information are grouped together, that is, they are
correlated. Crisp bread (crips_br) and frozen fish (Fro_Fish) are examples of two
variables that are positively correlated. When the numerical value of one variable
increases or decreases, the numerical value of the other variable tends to change in the
same way.
When variables are negatively (“inversely”) correlated, they are positioned on opposite
sides of the plot origin, in diagonally opposed quadrants. For instance, the variables
garlic and sweetener are inversely correlated, meaning that when garlic increases,
sweetener decreases, and vice versa.
 PCA - Example

The distance to the origin also conveys information. The further away from the plot origin a variable
lies, the stronger the impact that variable has on the model. This means, for instance, that the
variables crisp bread (Crisp_br), frozen fish (Fro_Fish), frozen vegetables (Fro_Veg) and garlic
(Garlic) separate the four Nordic countries from the others. The four Nordic countries are
characterized as having high values (high consumption) of the former three provisions, and low
consumption of garlic. Moreover, the model interpretation suggests that countries like Italy, Portugal,
Spain and to some extent, Austria have high consumption of garlic, and low consumption of
sweetener, tinned soup (Ti_soup) and tinned fruit (Ti_Fruit).
Example PCA – simple demonstration
Example PCA
 PCA - Applications
1. Image compression. Image can be resized as per the requirement and patterns
can be determined.
2. Customer profiling based on demographics as well as their intellect in the
purchase.
3. Widely used by researchers in the food science field.
4. Banking field in many areas like applicants applied for loans, credit cards, etc.
5. Customer Perception towards brands.
6. Finance field to analyze stocks quantitatively, forecasting portfolio returns, also
in the interest rate implantation.
7.Healthcare industries in multiple areas like patient insurance data where there
are multiple sources of data and with a huge number of variables

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 52
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright owner.
 Limitations of PCA
➢PCA should be used mainly for variables which are strongly correlated. If the
relationship is weak between variables, PCA does not work well to reduce
data.
➢How many principal components should we use?
- If we use principal components as a summary of data,
how many components are sufficient?
• No simple answer to this question, as cross-validation is not available for
this purpose.
• The “scree plot" on the previous slide can be used as a guide: we look for
an “elbow".

© 2021 UNIVERSITI TEKNOLOGI PETRONAS

All rights reserved. 53
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright
Internalowner.
 Limitations of PCA
➢ If the data does not follow a
multidimensional normal
(Gaussian) distribution, PCA
may not give the best principal
components
➢ Cannot fit data that is not linear
➢ The direction of maximum
variance is not always good for
classification
➢ if the data is a set of strings.
• (1,0,0,0,...), (0,1,0,0...),
...,(0,0,0,...,1) then the
eigenvalues do not fall off as PCA
requires.
© 2021 UNIVERSITI TEKNOLOGI PETRONAS
All rights reserved. 54
No part of this document may be reproduced, stored in a retrieval system or transmitted in any form or by any means (electronic, mechanical, photocopying, recording or otherwise)
without the permission of the copyright
Internalowner.
55
56