Autoencoders

By
Dr Nisarg Gandhewar
 Overview of Syllabus

Unit 3: Autoencoders

•Auto encoders: Auto encoders and relation to PCA,

•Regularization in auto encoders,
•Denoising auto encoders,
•Sparse auto encoders,
https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

•Contractive auto encoders

Autoencoders

you can actually keep these balls in smaller boxes(in size) with
force and they will acquire their original shape back
(approximately) once the box is opened.

The idea behind autoencoders is very similar.

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
 Autoencoder
•Autoencoder is an unsupervised neural network that tries to reconstruct the output
layer as similar as the input layer.

•Autoencoder is a type of neural network where the output layer has the same dimensionality
as the input layer.

•In simpler words, the number of output units in the output layer is equal to the number of
input units in the input layer.
https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

•Autoencoders are special neural networks that learn how to recreate the given information.

•An autoencoder replicates the data from the input to the output in an unsupervised manner
and is therefore sometimes referred to as a replicator neural network.

•Autoencoder is an unsupervised algorithm.

•We can define autoencoder as feature extraction algorithm.

Autoencoder

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
 Autoencoder

• Autoencoders are neural network models primarily used for unsupervised learning
tasks such as dimensionality reduction, data compression, and feature extraction.

•They learn to reconstruct the input data and capture its essential patterns, making
them useful for anomaly detection and image-denoising tasks.

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
Autoencoder

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
Components / Layers of Autoencoder

Loss Function: A loss function measures the difference between the input
and the reconstructed output, guiding the auto encoder’s training process.

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
Example of Autoencoder

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
Mathematical Formulation of Autoencoder

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
 Autoencoder

•Design an simple autoencoder network to regenerate image of size 28 * 28 using

tesnsorflow.

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
 Application of Autoencoder

•Image and Audio Compression: Autoencoders can compress huge images or audio files
while maintaining most of the vital information.

• Dimensionality Reduction: Autoencoders can lower the dimensionality of high-

dimensional datasets.

•Data Generation: Employ autoencoders to generate new data similar to the training data.
https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
•Denoising: One can utilize autoencoders to reduce noise from data. We can accomplish
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

this by teaching an autoencoder to recover the original data from a noisy version.

•Anomaly Detection: Any input that the autoencoder cannot accurately reconstruct is called
an anomaly.
 Autoencoder vs PCA

• Autoencoders and PCA serve similar purposes in reducing the dimensionality of

data,

•The choice between them depends on the specific characteristics of the data and
the goals of the analysis.

•Autoencoders are more flexible and powerful in capturing non-linear relationships

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
but may require more computational resources and training data.
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

•PCA, being a linear technique, is computationally efficient and interpretable but may
not perform as well on highly non-linear data.
Autoencoder vs PCA

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
 Autoencoder vs PCA

•Linearity:
PCA: It is essentially a linear transformation, finds axes with maximum variance, assumes
linear relationships in data.
Autoencoders: Can be non-linear, learn complex relationships, suitable for non-linear data.
It is capable of capturing non-linear relationships within the data due to their neural network
architecture.

•Interpretability: https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

PCA: Principal components are interpretable as linear combinations of original features.

Autoencoders: More complex, interpretability can be challenging due to hidden layers.

Computation:
PCA: Fast and computationally efficient.
Autoencoders: Can be computationally expensive, especially with complex architectures.
PCA is faster and computationally cheaper than autoencoders.
 Autoencoder vs PCA

• PCA is easier to implement and understand, and can give you a good idea of the
variance and structure of the data.

• Autoencoders are better suited for exploring more complex and nonlinear
relationships, or for tasks that require a neural network approach.

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
 Autoencoder vs PCA

When to use each:

PCA: Good for interpretable, linear dimensionality reduction, high-dimensional

data, fast processing.

Autoencoders: Ideal for capturing non-linear relationships, potentially better data

compression, tasks like anomaly detection, data generation.
https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

•Autoencoder is prone to overfitting due to high number of parameters.

•If the features have non-linear relationship with each other then autoencoder will be
able to compress the information better into low dimensional latent space
 Regularization in Auto Encoders

•Regularization in autoencoders is a technique used to prevent overfitting and

improve the generalization performance of the model.

•Overfitting occurs when the autoencoder learns the training data too well, capturing
noise or specific details that do not generalize well to new, unseen data.

•Regularization methods help to control the complexity of the autoencoder, making it

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/

more robust and preventing it from memorizing the training data.

https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
 Regularization in Auto Encoders

Some common regularization techniques used in autoencoders are

•L1 regularization (Lasso): Adds the sum of absolute values of weights to the loss function. This
encourages sparsity, where many weights become zero, forcing the model to rely on a smaller subset
of features.

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

•L2 regularization (Ridge): Adds the sum of squared weights to the loss function. This encourages
small weights, preventing individual features from having too much influence and promoting stability.
 Regularization in Auto Encoders

•Dropout: Randomly drops out a fraction of units in each layer during training. This prevents units
from co-adapting excessively and encourages features to be independently useful.

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

•Early Stopping:
Monitors the model's performance on a validation set. Stops training when performance on the
validation set starts to deteriorate, preventing overfitting to the training data.
 Regularization in Auto Encoders

The best regularization technique for your autoencoder depends on your specific
data and goals. Consider factors like:

Data size and complexity: Large datasets benefit from stronger regularization like
L1, while smaller datasets might require weaker methods like L2.

Desired feature properties: L2 promotes stability.

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

Computational resources: Techniques like dropout are efficient, while L1/L2 require
more computation.
 Types of Auto Encoders
Over the years, different types of Autoencoders have been developed:

•Undercomplete Autoencoder

•Overcomplete Autoencoder

•Sparse Autoencoder
https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
•Contractive Autoencoder
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

•Denoising Autoencoder

•Variational Autoencoder

•Convolutional Autoencoder
Undercomplete Autoencoder

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
Overcomplete Autoencoder

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
 Sparse Autoencoder

•It is similar to Undercomplete Autoencoder (UAE) since it encodes

information by squeezing it into fewer dimensions.

• However, instead of relying on fewer neurons, SAE uses

regularisation to enforce sparsity.

•By sparsity, we mean that fewer neurons can be activated at the

same time, creating an information bottleneck similar to that of
Undercomplete AE. https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

•only some percentage of nodes can be active in a hidden layer. The

neurons with output close to 1 are active, whereas the neurons close
to 0 are in-active neurons.

•Sparse autoencoders can be beneficial in various applications,

including feature learning, data compression, and denoising.
Sparse Autoencoder

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
 Contractive Autoencoder

•Contractive Autoencoders are a variant of traditional

autoencoders that introduce a regularization term to the loss
function.

•This term penalizes the model not only for reconstruction

errors but also for the sensitivity of the learned representations
to the input data.

•Like a basic autoencoder, a CAE consists of two main

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/

components: an encoder and a decoder.

https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

•The distinction lies in the loss function, where the CAE

incorporates a contractive penalty.

• This autoencoder is used for Image Processing, Anomaly

Detection, Data Compression.
 Denoising Autoencoder
•Denoising autoencoders (DAEs) are a type of artificial
neural network that is trained to reconstruct clean input data
from corrupted or noisy versions of that data.

•Denoising Autoencoders are neural network models that

remove noise from corrupted or noisy data by learning to
reconstruct the initial data from its noisy counterpart.

•The primary purpose of denoising autoencoders is to learn

a robust and meaningful representation of the input data by
https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

forcing the model to remove noise and recover the original,

clean information.

•Denoising autoencoders create a corrupted copy of the

input by introducing some noise.

•This helps to avoid the autoencoders to copy the input to

the output without learning features about the data.
 Denoising Autoencoder

•These autoencoders take a partially corrupted input while

training to recover the original undistorted input.

•The model learns a vector field for mapping the input data
towards a lower dimensional manifold which describes the
natural data to cancel out the added noise.

•DAEs are particularly useful for tasks where the input data
https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
is subject to noise, such as image denoising, speech
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

recognition, and other signal processing applications.

•Denoising autoencoders offer a powerful approach for

learning robust representations from noisy data, and they
have proven effective in a variety of applications where
noise is a common challenge.
 Variational Autoencoder
•A variational autoencoder (VAE) is a generative AI
algorithm that uses deep learning to generate new
content, detect anomalies and remove noise.

•Its a type of generative model that combines elements

of autoencoders with probabilistic modeling.

•VAEs are designed to learn a probabilistic mapping

between the input data and a latent space, allowing for
the generation of new data pointshttps://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
that resemble the
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph

training data.

•Unlike traditional autoencoders, VAEs model the latent

space as a probability distribution. Instead of encoding
input data into a fixed-point in the latent space,

•VAEs map input data to a probability distribution over

the latent space, typically assuming a Gaussian
distribution.
 Variational Autoencoder

Applications

•Image Generation
•Text Generation
•Recommender Systems
•Anomaly Detection
•Image Denoising
https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph
 Convolutional Autoencoder

•Convolutional Autoencoders (CAEs) are a type of autoencoder architecture that leverages

convolutional layers for both the encoding and decoding processes.

•They are particularly well-suited for tasks involving image data and spatial relationships.

https://www.linkedin.com/pulse/how-does-machine-learning-work-rohit-jayale/
https://www.freepik.com/free-vector/isometric-people-working-with-technology_5083803.htm#query=DeFi&position=6&from_view=search&track=sph