Chapter five

Introduction Principal Component Analysis(PCA)

Supervised and Unsupervised learning
• Machine learning is a field of computer science that gives computers
the ability to learn without being explicitly programmed.
• The two main types of machine learning are:
Supervised learning and
unsupervised learning
supervised learning
• In supervised learning ,the machine is trained on a set of labeled data,
which means that the input data is paired with the desired output.
• The machine then learns to predict the output for new input data.
Supervised learning is often used for tasks such as
• classification,
• regression
unsupervised learning
• In unsupervised learning, the machine is trained on a set of unlabeled
data, which means that the input data is not paired with the desired
output. The machine then learns to find patterns and relationships in
the data.
• Unsupervised learning is often used for tasks such as
• clustering
• dimensionality reduction,
Cont..
Understanding Dimensionality reduction
• Dimensionality reduction is the process of reducing the number of
features (or dimensions) in a dataset while retaining as much
information as possible.
• Reasons for Dimensionality reduction are:
to reduce the complexity of a model,
 to improve the performance of a learning algorithm, or
to make it easier to visualize the data.
Approaches to dimensionality reduction
• There are two main approaches to dimensionality reduction:
feature selection and
feature extraction.
Feature Selection:
Feature selection involves selecting a subset of the original features
that are most relevant to the problem at hand.
The goal is to reduce the dimensionality of the dataset while retaining
the most important features.
Cont..
• There are several methods for feature selection,
 filter methods: Filter methods rank the features based on their
relevance to the target variable.
wrapper methods: use the model performance as the criteria for
selecting features.
 embedded methods. embedded methods combine feature selection
with the model training process.
Cont…
• Feature Extraction:
Feature extraction involves creating new features by combining or
transforming the original features. The goal is to create a set of
features that captures the essence of the original data in a lower-
dimensional space.
• There are several methods for feature extraction
principal component analysis (PCA).
linear discriminant analysis (LDA), and
t-distributed stochastic neighbor embedding (t-SNE).
Understanding PCA
• In many explanatory studies the number of variables under
consideration are too large to handle. A way of reducing the number
of variables to be treated is the discord the linear combination which
have small variance and study only those with large variance.
• In other words to examine the relationship among a set of correlated
variables, it may be useful to transform the original set of variables to
a new set of a variable which are uncorrelated with each other and
called principal component analysis.
Cont..
• Principal component analysis, or PCA, is a dimensionality reduction
method that is often used to reduce the dimensionality of large data
set by transforming a large set of variables into a smaller one that still
contains most of the information in the large set.
• Reducing the number of variables of a data set naturally comes at the
expense of accuracy, but the trick in dimensionality reduction is to
trade a little accuracy for simplicity.
• Because smaller data sets are easier to explore and visualize, and thus
make analyzing data points much easier and faster for machine
learning algorithms without extraneous variables to process.
Steps for PCA
• Principal component analysis can be broken down into five steps.
Standardize the range of continuous initial variables
Compute the covariance matrix to identify correlations
Compute the eigenvectors and eigenvalues of the covariance matrix
to identify the principal components
Create a feature vector to decide which principal components to
keep
Recast the data along the principal components axes
Applications of PCA in Machine Learning

PCA is used to visualize multidimensional data.

It is used to reduce the number of dimensions in healthcare data.
PCA can help resize an image.
It can be used in finance to analyze stock data and forecast returns.
PCA helps to find patterns in the high-dimensional datasets.
Advantages of PCA
Dimensionality reduction: By determining the most crucial features
or components, PCA reduces the dimensionality of the data, which is
one of its primary benefits. This can be helpful when the initial data
contains a lot of variables and is therefore challenging to visualize or
analyze.
Feature Extraction: PCA can also be used to derive new features or
elements from the original data that might be more insightful or
understandable than the original features. This is particularly helpful
when the initial features are correlated or noisy.
Cont..
Data visualization: By projecting the data onto the first few principal
components, PCA can be used to visualize high-dimensional data in
two or three dimensions.
Noise Reduction: By locating the underlying signal or pattern in the
data, PCA can also be used to lessen the impacts of noise or
measurement errors in the data.
Multicollinearity: When two or more variables are strongly
correlated, there is multicollinearity in the data, which PCA can
handle.
 Disadvantages of PCA
Interpretability: Although principal component analysis (PCA) is
effective at reducing the dimensionality of data and spotting patterns,
the resulting principal components are not always simple to
understand or describe in terms of the original features.
Information loss: PCA involves choosing a subset of the most crucial
features or components in order to reduce the dimensionality of the
data.
Outliers: Because PCA is susceptible to anomalies in the data, the
resulting principal components may be significantly impacted. The
covariance matrix can be distorted by outliers, which can make it
harder to identify the most crucial characteristics.
Cont..
Scaling: PCA makes the assumption that the data is scaled and
centralized, which can be a drawback in some circumstances.
Computing complexity: For big datasets, it may be costly to compute
the eigenvectors and eigenvalues of the covariance matrix.
 How Does Principal Component Analysis
Work?
1) Normalize the Data:Standardize the data before performing PCA.
This will ensure that each feature has a mean = 0 and variance = 1.
2. Build the Covariance Matrix
• Construct a square matrix to express the correlation between two or
more features in a multidimensional dataset.
3.Find the Eigenvectors and Eigenvalues
• Calculate the eigenvectors/unit vectors and eigenvalues. Eigenvalues
are scalars by which we multiply the eigenvector of the covariance
matrix.

4. Sort the Eigenvectors in Highest to Lowest Order and Select the Number of
Principal Components.
Example
Solution
Cont..
Cont..
Cont..
Cont..
Cont..
Cont..
Cont..
Cont
Cont..
Cont..
Cont..