Dimensionality Reduction With PCA For large

Scale Image Databases.

Hariprasad Boddepalli
School of Computerscience and
Engineering
Lovely Proffesional University
Punjab , India
[email protected]

Keywords—Computer Vision , Dimensionality Reduction , PCA is then used to reduce the dimensionality of the dataset
Algorithms , PCA , Standard Scalar , Heatmaps .
by retaining only the components that capture 95% of the
I. ABSTRACT total variance, significantly reducing the computational
burden while preserving the critical information required for
The rapid growth of image databases in modern applications tasks such as image recognition. The results of this study
poses challenges in terms of data storage, processing demonstrate that PCA can effectively reduce the
efficiency, and computational complexity. This paper dimensionality of large image datasets, leading to faster
presents an application of Principal Component Analysis processing, lower storage requirements, and improved
(PCA) for dimensionality reduction in large-scale image efficiency in machine learning tasks.
datasets, using the MNIST database as a case study. PCA, a
widely-used statistical technique, transforms high-
dimensional image data into a lower-dimensional space, III. METHODOLOGIES
retaining the most significant features while reducing the
number of dimensions. This not only improves computational A. DATASET SOURCE
performance but also enhances the efficiency of downstream I had taken this data set from open ml kaggle.[1]this data
tasks such as image recognition and classification. set is about images of handwritten digits , which consists of
image data and pixels of 784.
II. INTRODUCTION
This paper aims to explore the use of PCA for dimensionality
reduction in large-scale image datasets. We analyze its B. DATA ACQUISITION
effectiveness in reducing the number of features while The first step in this methodology involves acquiring the
maintaining image quality and examine its impact on dataset to be used for dimensionality reduction. In this study,
computational performance across various image databases. we utilize the MNIST dataset, which is a widely known and
Furthermore, we investigate how PCA can enhance the publicly available dataset consisting of handwritten digit
efficiency of machine learning models by minimizing data images. Each image in the dataset is represented as a 28x28
complexity while preserving essential information. Through pixel grid, resulting in 784 features 28 multiplied by 28 per
a series of experiments and case studies, we demonstrate the image. These features represent the pixel intensity values that
benefits and limitations of PCA as a tool for managing large describe the visual characteristics of each image. The MNIST
image repositories, providing insights into its practical dataset includes 70,000 images, of which 60,000 are used for
applications in the real world. training and 10,000 for testing. This dataset is ideal for
analyzing dimensionality reduction techniques like Principal
Component Analysis PCA due to its size and the high
The increasing reliance on image data in fields such as dimensionality of the images.
computer vision, medical imaging, and machine learning has
led to the development of massive image databases. These The dataset is loaded using the fetch open ml function from
datasets, while rich in information, present significant the sklearn datasets library, which fetches the dataset directly
challenges in terms of data storage, computational speed, and from the Open ML repository. This step ensures that the
processing complexity. Each image, often represented by a images are downloaded as numerical data (pixels as
large matrix of pixel values, translates into high-dimensional numerical features) and are stored in arrays where each row
data. For instance, in the popular MNIST dataset of corresponds to an individual image and each column
handwritten digits, each image consists of 28x28 pixels, represents a pixel feature. Along with the image data X, the
resulting in 784 features per image. As databases scale up, corresponding labels y are also extracted, which represent the
handling such high-dimensional data can become digits 0-9 depicted in each image. This labeled data will later
computationally expensive and inefficient. help in evaluating the effectiveness of the dimensionality
reduction process.

XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE

 IV. PREPROCESSING components that capture the most important characteristics of
A. STANDARD SCALER the images.
Once the MNIST dataset is acquired, the next critical step
involves data preprocessing, which is an essential stage
before applying any machine learning algorithm, especially
Principal Component Analysis (PCA). Since PCA relies on
the variance of the features to compute the principal
components, the input data must be standardized.
Standardization ensures that each feature (in this case, each
pixel) has the same scale and contributes equally to the
analysis.

standardization to transform the dataset such that each feature

has a mean of 0 and a standard deviation of 1. This is achieved
using the Standard Scaler function from the sklearn
preprocessing module. Standardization is necessary because
the pixel values in the images vary between 0 and 255, which
means that some features might have larger scales than
others. Without scaling, PCA would be biased towards
features with larger variances, potentially skewing the results.
The StandardScaler() , fit transform X function is used to fit
the scaler to the data and transform the entire dataset at once,
ensuring that the transformation is applied uniformly across
all images. B. EXPLAINED VARIANCE AND PRINCIPAL
COMPONENT
the MNIST data is now standardized and ready for the
application of PCA. Each feature has been adjusted so that it
contributes proportionally to the computation of principal One of the most important steps in PCA is determining how
components, preventing any feature from dominating due to many principal components should be retained. While PCA
its original scale. transforms the data into a lower-dimensional space, the
number of components can vary based on how much of the
original dataset's variance we wish to preserve. In this
V. APPLYING PRINCIPAL COMPONENT ANALYSIS
methodology, we aim to retain enough principal components
Once the data has been standardized, the core of this to explain 95% of the variance in the data. Retaining this level
methodology, Principal Component Analysis (PCA), is of variance ensures that the reduced dataset retains the
applied. PCA is a powerful dimensionality reduction essential information from the original data while reducing
technique that transforms high-dimensional data into a lower- computational complexity.
dimensional space by identifying new variables, known as
principal components. These components are linear
combinations of the original features and are designed to To identify the number of components needed to capture 95%
capture the maximum amount of variance in the data. In of the variance, the cumulative explained variance is
essence, PCA identifies the directions in which the data varies calculated. The explained variance for each component
the most and reduces the dataset along those directions, indicates how much information that component retains from
thereby preserving the most significant patterns. the original data. A plot of cumulative explained variance is
generated using np cumsum (pca explained variance ratio),
PCA is applied to the standardized data without initially where the sum of the explained variance is plotted against the
specifying the number of components. This allows the number of components. The number of components required
algorithm to compute all possible components, so that we can to retain 95% of the variance is determined.
later determine how much variance each component explains.
The PCA().fit transform(X scaled) function is used here,
where X scaled represents the standardized dataset. This step By selecting the number of components that retain 95% of the
computes the principal components and transforms the data variance, we ensure that the dimensionality of the dataset is
into a new coordinate system based on those components, significantly reduced while still preserving the essential
reducing the dimensionality of the original dataset while structure of the images. In this case, the reduced dataset can
retaining as much of the variance as possible. be effectively analyzed and processed with far fewer
dimensions than the original dataset.
This transformation allows us to investigate how PCA can be
used to compress the MNIST dataset from its original 784
dimensions (one for each pixel) into a smaller set of
C. DIMENSIONALITY REDUCTION original 28x28 pixel format for display. This reshaping is
crucial, as it reconstructs the visual structure of the images,
After determining the optimal number of components, we allowing viewers to see the actual handwritten digits. By
apply PCA again, this time specifying the number of plotting these images in a systematic grid layout, we can
components to be retained. The goal is to reduce the dataset visually inspect how different examples of handwritten digits
to a lower-dimensional space, where only the principal appear, identify any potential outliers, and assess the general
components that account for 95% of the total variance are quality of the data. This initial visualization step is critical for
kept. This significantly reduces the size of the data and allows ensuring that the dataset is well-prepared for subsequent
for more efficient storage and computational processing analysis and that the data is representative of the target
without a substantial loss of information. classes.

This step transforms the dataset from 784 dimensions down Once PCA has been applied to the dataset and the principal
to the selected number of principal components. The reduced components have been extracted, it becomes essential to
dataset retains the most critical patterns and features from the visualize these components to interpret the transformation
original images, but in a compressed form, which is that has occurred. The principal components represent new
especially useful for tasks that involve large-scale image feature axes in the transformed space, capturing the directions
databases, such as image classification or recognition. of maximum variance within the data. Visualizing these
components helps elucidate the underlying patterns and
In many real-world applications, data can contain hundreds relationships in the dataset that PCA has uncovered.
or even thousands of variables, which can present challenges
in terms of both computational efficiency and interpretability. To visualize the principal components, heatmaps are
High-dimensional data often suffers from problems such as employed. Each principal component can be reshaped back
overfitting, the curse of dimensionality, and increased into its original image dimensions (28x28 pixels) and plotted
computational cost. Dimensionality reduction techniques as a grayscale heatmap. Each pixel in the heatmap
address these issues by transforming the data into a lower- corresponds to a weight assigned to that pixel by the principal
dimensional space, while retaining as much of the original component, where lighter shades indicate higher weights and
information as possible. Among various dimensionality darker shades indicate lower weights. By examining these
reduction methods, Principal Component Analysis PCA is heatmaps, we gain valuable insights into how each principal
one of the most widely used techniques due to its component captures distinct features of the dataset.
effectiveness and simplicity.
the first principal component often captures the most
dimensionality reduction using PCA is a highly effective significant overall features of the images, such as the general
approach for handling large image databases. It reduces the structure of the digits. Subsequent components capture finer
complexity of the data by transforming high-dimensional variations, such as slants, thicknesses, or other stylistic
datasets into a lower-dimensional space, while retaining the details. The visual representation of these components
most important patterns and structures. This not only reveals how PCA prioritizes certain features over others,
facilitates efficient storage and computation but also which is vital for understanding the impact of dimensionality
enhances the performance of machine learning models by reduction on the data.
eliminating redundant features and focusing on the most
informative components. The ability to explain a significant
portion of the variance with a reduced number of components
makes PCA an indispensable tool in the field of data science
and image processing, particularly when working with large-
scale image datasets that require efficient and scalable
analysis.

D. VISUALIZATION OF IMAGE AND PRINCIPAL

COMPONENTS

One of the initial steps in the PCA workflow is to visualize a

selection of original images from the dataset. This allows
researchers to familiarize themselves with the data they are
working with, assess the quality and diversity of the images, E. VISUALIZING THE FIRST FOUR PRINCIPAL
and understand the visual patterns present in the dataset. For COMPONENTS
instance, in the case of the MNIST dataset, which consists of
handwritten digits, displaying a subset of randomly selected
images provides insight into the variability in handwriting The line pca = PCA (n components=4) signifies the
styles, thickness, and other attributes inherent to the dataset. initialization of a PCA (Principal Component Analysis)
object, which is a critical step in the process of dimensionality
we utilize Python's Matplotlib library to generate a grid of reduction. PCA is a widely used technique in data analysis
and machine learning that transforms high-dimensional data
random images, reshaping each 1D pixel array back into its
into a lower-dimensional form while preserving as much By analyzing the heatmaps, researchers can identify the
variance as possible. characteristics that define each principal component, leading
to a better understanding of the underlying patterns within the
By specifying n components=4, we are indicating that we data. This visual representation not only enhances
wish to reduce our dataset to only four principal components. interpretability but also serves as a powerful communication
Each principal component is a new feature that is constructed tool when presenting PCA results in research papers.
as a linear combination of the original features in the dataset.
These combinations are designed to capture the maximum the heatmap of principal components serves as an essential
variance present in the data. The decision to keep only four visualization tool in the analysis of high-dimensional
components can be driven by various factors, such as the need datasets. By employing PCA, we can reduce the complexity
for simplicity in analysis, the computational efficiency of the data while retaining the most important features.
required for downstream tasks, or the desire to visualize the Heatmaps facilitate the interpretation of these features,
data in a more interpretable format. revealing their contributions to the principal components.
The integration of heatmaps into the PCA workflow enriches
The choice of the number of components to retain is crucial; the analysis, offering a clear and intuitive means of
it often involves a trade-off between maintaining sufficient understanding complex datasets, ultimately contributing to
information from the original dataset and reducing more informed decision-making and further research
complexity. Retaining too few components can result in exploration.
significant loss of information, while retaining too many may
lead to overfitting, where the model learns the noise rather
than the underlying patterns. By setting the number of
components to four, we strike a balance that allows us to
capture the most important features of the data while
simplifying it to a manageable size.

VI. CONCLUSION

PCA offers numerous advantages, it is essential to recognize

its limitations. The linear nature of PCA means that it may
not capture complex, non-linear relationships present in the
data. Future research could explore hybrid approaches that
integrate PCA with non-linear dimensionality reduction
techniques, thereby enhancing its applicability to diverse
datasets .PCA stands out as a robust method for
dimensionality reduction in large image databases. Its ability
F. HEATMAPS OF PRINCIPAL COMPONENT to streamline data processing while preserving essential
information makes it an invaluable tool for researchers and
The heatmaps generated from the principal components practitioners alike. As the demand for efficient data analysis
provide valuable insights into the structure of the data. Each continues to rise, further advancements in PCA and its
heatmap reveals the contribution of the corresponding integration with other methodologies will undoubtedly pave
component to the overall dataset variance. In the context of the way for more sophisticated approaches to handling the
image datasets, such as MNIST, the heatmaps can illustrate complexities of modern image data. By harnessing the power
which pixels contribute significantly to the variance captured of PCA, we can unlock new possibilities for innovation and
by each principal component. For instance, a heatmap of the discovery in the realm of image analysis, ultimately
first principal component may show that certain regions of contributing to the advancement of artificial intelligence and
the digit images like the center or edges are highlighted, machine learning.
indicating that these features are crucial for distinguishing
between different digits.
8. REFERENCES Engineering (IJSCE) ISSN: 2231-2307, Vol.-I Issue-II pp.33-36.

[9] Arunasakthi. K and Kamatchipriya. L(2014), ‘A Review On Linear

[1] Lee, D.-H.; Kim, H.-J. A fast content-based indexing and retrieval And Non-Linear Dimensionality Reduction Techniques’, Machine
technique by the shape information in large image database. Learning And Applications: An Int. J. (Mlaij), Vol.1, No.1, Pp.65-76.
J. Syst. Softw. 2001, 56, 165–182.
[10] Shereena V.B. and Julie M. David (2014), ‘Content Based Image
Retrieval : Classification Using Neural Networks’, The Int. J. of
[2] Shi, Q.; Li, H.; Shen, C. Rapid face recognition using hashing. In Multimedia & Its Applications (IJMA) Vol.6, No.5. DOI
Proceedings of the 2010 IEEE Computer Society Conference on ;10.5121/ijma.2014.6503 31.
Computer Vision and Pattern Recognition, San Francisco, CA, USA,
13–18 June 2010; pp. 2753–2760.
[11] G. Sasikala , R. Kowsalya and Dr. M. Punithavalli (2010), ‘A
Comparative Study Of Dimension Reduction Techniques For Content-
[3] Comer, D. Ubiquitous b-tree. ACM Comput. Surv. (CSUR) 1979, 11, Based Image Retrieval’, The Int. J. of Multimedia & Its
121–137.
Applications, Vol.2, No.3, Doi : 10.5121/Ijma.2010.2303 40.

[12] Aravind Nagathan, Animozhi and Jithendra Mungara (2014), ‘Content-

[3] Cai, H.; Wang, X.; Wang, Y. Compact and robust fisher descriptors for Based Image Retrieval System using Feed-Forward Backpropagation
large-scale image retrieval. In Proceedings of the IEEEInternational Neural Network’,IJCSNS International Journal of Computer
Workshop on Machine Learning for Signal Processing (MLSP’2011), Science and Network Security, Vol.14 No.6.pp 70-77.
Santander, Spain, 18–21 September 2011;
pp. 1–6.
[13] Valgren, C.; Lilienthal, A.J. Sift, surf and seasons: Long-term outdoor
localization using local features. In Proceedings of the 3rd
European Conference on Mobile Robots (EMCR), Freiburg,
[4] White, D.A.; Jain, R. Similarity indexing with the ss-tree. In Germany, 19–21 September 2007.
Proceedings of the Twelfth International Conference on
DataEngineering, New Orleans, LA, USA, 26 February–1 March 1996;
pp. 516–523. [14] Cools, A.; Belarbi, M.A.; Belarbi, M.A.; Mahmoudi, S.A. A
Comparative Study of Reduction Methods Applied on a Convolutional
Neural Network. Electronics 2022, 11, 1422.
[5] [8] Rafael Gonzalez and Richard E. Woods, ‘Digital Image
processing’, Addison Wesley, 2nd Edn.,2002.
[15] M. Belkin, P. Niyogi, “Laplacian eigenmaps for dimensionality
reduction and data representation,” Neural computation, vol.
15(6), 2003, pp. 1373-1396.
[6] Manimala Singha and K. Hemachandran (2012) ‘Content based image
retrieval using color and texture’, Signal & Image Processing : An Int.
J. (SIPIJ), Vol.3, No.1, pp.39-57.

[7] S. Mangijao Singh , K. Hemachandran (2012) “Content-Based Image

Retrieval using Color Moment and Gabor Texture Feature”, Int.J. of .
Computer Science Issues,Vol. 9, Issue 5, No 1, pp.299-309.

[8] Wasim Khan, Shiv Kumar. Neetesh Gupta and Nilofar Khan(2011), ‘A
.
Proposed Method for ImageRetrieval using Histogram values and
Texture Descriptor Analysis’, Int. J.of Soft Computing and