Yangon Technological University

Department of Electronic Engineering

Biomedical Signal and Image Processing II

EcE-52005

Assignment I

Name - Moe Myint Kyaw Lwin

Roll Number- VEC (1)

Date –June 20, 2024

 Feature extraction methods

Principle component analysis (PCA) and linear discriminant analysis (LDA)

Feature extraction step is one of the most important steps in pattern recognition system.
Feature extraction is a process in machine learning and data analysis that involves identifying
and extracting relevant features from raw data. These features are later used for further analysis,
such as classification, recognition, or detection tasks. It aims to reduce the dimensionality of the
data and helps to improve the performance and efficiency of the model.

Two types of feature extraction methods:

 Principle component analysis (PCA)

 Linear discriminant analysis (LDA)

Principle component analysis (PCA)

PCA is an unsupervised method that tries to find the directions of maximum variance in
the data. It projects the data onto a newset of othogonal axes, called principle components, that
capture as much of the variability as possible. The variation reflects differences in data. The first
principle component is the one that explains the most variance, the second one is the one that
explains the most variance after removing the effect of the first one and so on. The feature vector
corresponding to the largest eigenvalue contains the most massive amount of information. If a
few eigenvalues of some features are small, it means that there is little information in these
features.

PCA was invented in 1901 by Karl Pearson as an analogue of the principle axis theorem
in mechanics. It was later independently developed and named by Harold hotelling in 1930. The
main goal of the PCA analysis is to identify patterns in data. It is basically used to reduce the
dimension of data set. PCA aims to detect the correlation between variables.
Mathematical Foundation
Variance and Covariance

 Variance measures how much the data points deviate from the mean.
 Covariance measures how two variables change together. Positive covariance indicates
that the variables increase together, while negative covariance indicates an inverse
relationship.

Eigenvalues and Eigenvectors

 Eigenvalues represent the magitude of principle components.

 Eigenvectors represent the direction of principle components.
 The principle components are derived from the eigenvectors of the covarianvce matrix of
the data, ordered by their corresponding eigenvalues.

Steps in PCA
(1) Standardized the data

Standardizing the data ensures that each feature contributes equally to the analysis. This involves
centering the data which is subtracting the mean and scaling to unit variance.

(2) Compute the covariance matrix

The covariance matrix captures the relationships between the features.

(3) Perform Eigen Decomposition

The eigenvalues and eigenvectors of the convariance matrix are computed. These provide the
principle components and their corresponding variance.

(4) Select Principle Components

Principle components are selected based on the eigenvalues, with the top k components capturing
the most variance.

(5) Transform the data

The original data is projected onto the selected principle components, reducing the
dimensionality while preserving most of the variance.
Application of PCA
 Data compression: Reducing the number of features while retaining most of the
information.
 Visualization: Plotting high dimensional data in 2D or 3D.
 Noise reduction: Removing less significant components (with smaller eigenvalues)
 Feature extraction: Creating new features that summarize the original dataset.

Conclusion
Principle component analysis is a valuable tool for simplifying complex datasets,
uncovering hidden patterns and enhancing data visualization. By transforming the data into a set
of uncorrelated principle components, PCA facilitates more efficient data analysis and
interpretation.

Linear Discriminant Analysis (LDA)

LDA is a supervised method that tries to find the directions of maximun separation
between the classes in the data. It assumes that the data within each class follow the multivariate
normal distribution with a common covariance matrix. It projects the data onto a new set of axes,
called linear discriminants that maximize the ratio of the between-class variance to the within-
calss variance. The first linear discriminant is the one that separates the classes the most, the
second one is the one that separates the classes the most after removing the effect of the first one
and so on. Unlike PCA, which tries to maintain data information as much as possible, LDA is to
make the data points more distinguishable after dimension reduction.

Steps in LDA
(1) Compute the mean vectors

Compute the mean vector for each class and overall mean vector for the entire dataset.

(2) Compute the scatter matrices

Calculate the within class scatter matrix and between class scatter matrix.

(3) Solve the generalized eigenvalue problem

Solve the eigenvalue problems and eigenvectors corresponding to the largest eigenvalues from
the linear discriminants.

(4) Select linear discriminants

Select the top k eigenvectors to reduce the dimensionality of the data.

 (5) Transform the data

Project the original data onto the new lower dimensional space defined by the selected linear
discriminants.

Application of LDA
 Classification: Distinguishing between different classes in supervised learning.
 Dimensionality Reduction: Reducing the number of features while retaining class-
discriminatory information.
 Pattern Recognition: Recognizing patterns and making decisions based on the linear
combination of features.

Conclusion
Linear Discriminant Analysis is a valuable tool for both classification and dimensionality
reduction. By focusing on maximizing the separation between classes, LDA provides a robust
method for analyzing and visualizing high-dimensional data. Its effectiveness is grounded in its
ability to project data onto a space where class distinctions are clearer, facilitating more accurate
and insightful analyses.