Auto-encoders

• Auto-encoders have emerged as one of the technologies

and techniques that enable computer systems to solve
data compression problems more efficiently.

• They became a popular solution for reducing noisy data.

• Simple auto-encoders provide outputs that are the same

or similar to the input data—only compressed.

• What is an auto-encoder?

• An auto-encoder is a type of artificial neural network used

to learn data encodings in an unsupervised manner.
 Auto-encoders
• The aim of an auto-encoder is to learn a lower-
dimensional representation (encoding) for a
higher-dimensional data, typically for
dimensionality reduction, by training the
network to capture the most important parts of
the input image.
The architecture of auto-encoders
 The architecture of auto-encoders
• Auto-encoders consist of 3 parts:

• 1. Encoder: A module that compresses the train-validate-

test set input data into an encoded representation that is
typically represented in several orders of magnitude
smaller than the input data.

• 2. Bottleneck: A module that contains the compressed

knowledge representations and is therefore the most
important part of the network.
 The architecture of auto-encoders
• 3. Decoder: A module that helps the network
“decompress” the knowledge representations and
reconstructs the data back from its encoded form.

• The output is then compared with a ground truth.

• The architecture as a whole is shown in Figure below:

The architecture of auto-encoders
 The architecture of auto-encoders
• The relationship between the Encoder, Bottleneck, and

Decoder

• Encoder

• The encoder is a set of convolutional blocks followed by

pooling modules that compress the input to the model into

a compact section called the bottleneck.

• The bottleneck is followed by the decoder that consists of a

series of upsampling modules to bring the compressed

feature back into the form of an image.

 The architecture of auto-encoders
• In case of simple auto-encoders, the output is expected to
be the same as the input data with reduced noise.

• However, for variational autoencoders it is a completely

new image, formed with information that is provided as
input.

• Bottleneck

• The most important part of the neural network, and

ironically the smallest one, is the bottleneck.
 The architecture of auto-encoders
• The bottleneck exists to restrict the flow of information to
the decoder from the encoder, thus, allowing only the most
vital information to pass through.

• Since the bottleneck is designed in such a way that the

maximum information possessed by an image is captured
in it, we can say that the bottleneck helps us form
a knowledge-representation of the input.

• Thus, the encoder-decoder structure helps us to extract

the most from an image in the form of data and establish
useful correlations between various inputs within the
network.
 The architecture of auto-encoders
• A bottleneck as a compressed representation of the input that
further prevents the neural network from memorising the
input and overfitting on the data.

• As a rule of thumb, remember this: The smaller the bottleneck,

the lower the risk of overfitting.

• Decoder

• Finally, the decoder is a set of upsampling and convolutional

blocks that reconstructs the bottleneck's output.

• Since the input to the decoder is a compressed knowledge

representation, the decoder serves as a “decompressor” and
builds back the image from its latent attributes.
 Example
• Consider an auto-encoder trained on image data for
compression:

1. Input:

• An image of size 28x28 pixels, resulting in 784 input

features.

2. Encoder:

• The encoder compresses the 784 input features into a

latent representation of size 32.

3. Decoder:

• The decoder reconstructs the 784 original features from

the 32 latent features.
 How to train autoencoders?
• To train auto-encoder a set 4 hyper-parameters is needed
before training :

1. Code size: The code size or the size of the bottleneck is

the most important hyper-parameter used to tune the auto-
encoder. The bottleneck size decides how much the data has
to be compressed. This can also act as a regularisation
term.

2. Number of layers: Like all neural networks, an important

hyper-parameter to tune auto-encoders is the depth of the
encoder and the decoder. While a higher depth increases
model complexity, a lower depth is faster to process.
 How to train auto-encoders ?
3. Number of nodes per layer: The number of nodes per
layer defines the weights we use per layer.

• Typically, the number of nodes decreases with each sub-

sequent layer in the auto-encoder as the input to each of
these layers becomes smaller across the layers.

4. Reconstruction Loss: The loss function we use to train the

auto-encoder is highly dependent on the type of input and
output we want the auto-encoder to adapt to.

• If we are working with image data, the most popular loss

functions for reconstruction are MSE Loss and L1 Loss.
 How to train auto-encoders ?
• In case the inputs and outputs are within the range [0,1],
as in MNIST, we can also make use of Binary Cross
Entropy as the reconstruction loss.

• Types of auto-encoders

• The idea of auto-encoders for neural networks is not new.

• The first applications initially used for dimensionality

reduction and feature learning.

• An auto-encoder concept has evolved over the years and

is now widely used for learning generative models of
data.
 Types of auto-encoders
• The five popular auto-encoders are:

1. Undercomplete auto-encoders

2. Sparse auto-encoders

3. Contractive auto-encoders

4. Denoising auto-encoders

5. Variational Auto-encoders (for generative modelling)

 Types of auto-encoders
1. Undercomplete auto-encoders
• An undercomplete auto-encoder is one of the simplest
types of auto-encoders.
• The way it works is very straightforward—
• The Undercomplete auto-encoder takes in an image and
tries to predict the same image as output, thus
reconstructing the image from the compressed bottleneck
region.
• Undercomplete auto-encoders are truly unsupervised as
they do not take any form of label, the target being the
same as the input.
 Types of auto-encoders
• The primary use of auto-encoders is the generation of the
latent space or the bottleneck, which forms a compressed
substitute of the input data and can be easily
decompressed back with the help of the network when
needed.

• This form of compression in the data can be modelled as a

form of dimensionality reduction.
 Types of auto-encoders

• When we think of dimensionality reduction, we tend to think

of methods like PCA (Principal Component Analysis) that
form a lower-dimensional hyperplane to represent data in a
higher-dimensional form without losing information.
 Types of auto-encoders
• However—PCA can only build linear relationships. As a result,
it is put at a dis-advantage compared with methods like
undercomplete autoencoders that can learn non-linear
relationships and, therefore, perform better in dimensionality
reduction.

• This form of non-linear dimensionality reduction where the

auto-encoder learns a non-linear manifold is also termed
as manifold learning.

• The loss function used to train an undercomplete auto-encoder

is called reconstruction loss, as it is a check of how well the
image has been reconstructed from the input data.
 Types of auto-encoders
• The reconstruction loss can be anything depending on the
input and output, we will use an L1 loss to depict the term
(also called the norm loss) represented by:

• Where represents the predicted output

and x represents the ground truth.
 Types of auto-encoders
• 2. Sparse auto-encoders

• A sparse autoencoder is a type of neural network that aims

to learn a sparse representation of the input data.

• Sparse autoencoders are a variant of the traditional

autoencoder designed to learn efficient, sparse
representations of the input data.

• The sparsity constraint encourages the autoencoder to use a

limited number of active neurons in the hidden layers at any
given time, which can lead to more interpretable features
and improved performance on various tasks.
Types of auto-encoders
 Types of auto-encoders
• Key Concepts of Sparse Autoencoders

1. Encoder and Decoder Structure:

• Similar to traditional autoencoders, sparse

autoencoders consist of an encoder that compresses the
input into a lower-dimensional representation and a
decoder that reconstructs the original input from this
representation.

2. Sparsity Constraint:

• A sparsity constraint is applied to the hidden layers to

ensure that only a few neurons are active at once.
 Types of auto-encoders
• This can be implemented using a penalty term in the loss
function, such as L1 regularization or Kullback-Leibler
(KL) divergence, to enforce the sparsity constraint.

3. Loss Function:

• The loss function of a sparse autoencoder typically consists

of two components:

i. Reconstruction Loss: Measures the difference between

the input and the reconstructed output (e.g., Mean
Squared Error).
 Types of auto-encoders
ii. Sparsity Penalty: Enforces the sparsity constraint on

the hidden layers.

• The overall loss function can be represented as :

Loss = Reconstruction Loss + 𝜆 × Sparsity Penalty

• where 𝜆 is a hyper-parameter that controls the weight of the

sparsity penalty.

• Benefits of Sparse Autoencoders

1. Efficient Representation:

2. Feature Learning:

3. Robustness to Noise:
 Types of auto-encoders
• 3. Contractive auto-encoders

• A Contractive Autoencoder is a type of autoencoder designed to

learn a robust representation of the input data by encouraging
the model to be invariant to small changes in the input.

• This is achieved by adding a regularization term to the loss

function that penalizes the sensitivity of the encoder to the
input.

• Similar to other auto-encoders, contractive auto-encoders

perform the task of learning a representation of the image while
passing it through a bottleneck and reconstructing it in the
decoder.
 Types of auto-encoders
• Contractive Auto-encoders introduce a regularization
term to the objective function of the auto-encoder.

• This regularization term penalizes the sensitivity of

the encoding to variations in the input data.

• The key idea is to encourage the learned

representations to be robust to small changes in the
input, making the representations more invariant and
useful for downstream tasks.
 Types of auto-encoders
• Contractive autoencoders work on the basis that similar inputs
should have similar encodings and a similar latent space
representation.

• To train a model that works along with this constraint.

• We have to ensure that the derivatives of the hidden layer

activations are small with respect to the input data.

• Mathematically:

𝛿ℎ/𝛿𝑥

• Where h represents the hidden layer and x represents the

input.
 Types of auto-encoders
• Loss Function

• The loss function for a Contractive Auto-encoder is given by:

• 𝐿𝐶𝐴𝐸 = 𝐿𝑟𝑒𝑐𝑜𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑜𝑛 + 𝜆 𝐿𝑐𝑜𝑛𝑡𝑟𝑎𝑐𝑡𝑖𝑣𝑒

where:

• Lreconstruction is the standard reconstruction loss, typically

the mean squared error:

𝐿𝑟𝑒𝑐𝑜𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑜𝑛 = ∥𝑥 − 𝑥^∥2

• The reconstruction loss wants the model to tell differences

between two inputs and observe variations in the data.
 Types of auto-encoders
• The total loss function can be mathematically expressed as:

• Where h > is the hidden layer for which a gradient is

calculated and represented with respect to the input x as

• Advantages of Contractive Autoencoders

• Robustness to Perturbations:

• Regularization:
 Types of auto-encoders
• 4. Denoising autoencoders

• Denoising autoencoders, as the name suggests, are

autoencoders that remove noise from an image.

• In denoising autoencoders, we feed a noisy version of the

image, where noise has been added via digital alterations.

• The noisy image is fed to the encoder-decoder architecture,

and the output is compared with the ground truth image.

• The denoising autoencoder gets rid of noise by learning a

representation of the input where the noise can be filtered
out easily.
 Types of auto-encoders

• The autoencoder maps the input data into a lower-dimensional

manifold (like in undercomplete autoencoders), where filtering
of noise which becomes much easier.
 Types of auto-encoders
• Essentially, denoising autoencoders work with the help of
non-linear dimensionality reduction and the loss function
generally used in these types of networks is L2 or L1 loss.

• 5. Variational autoencoders

• Variational Autoencoders (VAEs) are a type of generative

model that combines concepts from Bayesian inference
and deep learning.

• VAEs are particularly useful for generating new data

samples that are similar to the training data, and for
learning latent representations of the data.
 Types of auto-encoders
• For example—

• This is what a variational autoencoder would learn from

the input:
 Types of auto-encoders
• These attributes explain the image and can be used
in reconstructing the image from the compressed
latent space, and they do not allow the latent
attributes to be expressed in a probabilistic fashion.
• Variational auto-encoders deal with this specific topic
and express their latent attributes as a probability
distribution, leading to the formation of a continuous
latent space that can be easily sampled and
interpolated.
 Types of auto-encoders
• Key Concepts of Variational Autoencoders
• 1. Latent Variables: VAEs introduce latent variables that
capture the underlying structure of the data.
• These latent variables are typically assumed to follow a
prior distribution, such as a standard normal distribution.
• 2. Encoder Network: The encoder network maps the input
data to a distribution over the latent variables.
• Instead of producing a single point estimate, the encoder
outputs the parameters of a probability distribution
(usually a Gaussian), represented by mean (𝜇) and
standard deviation (𝜎).
 Types of auto-encoders
• 3. Decoder Network: The decoder network (also known as
the generative network) maps the latent variables back to
the data space, generating new data samples.

• The decoder takes samples from the latent distribution

and produces reconstructed data.

• 4. Reparameterization Trick: To enable backpropagation

through the stochastic sampling of latent variables, the
reparameterization trick is used.

• This involves expressing the latent variables as a

deterministic function of the encoder outputs and some
additional noise.
 Types of auto-encoders
• For a Gaussian distribution, this can be written as:

𝑧=𝜇+𝜎⋅𝜖

• where 𝜖 is sampled from a standard normal distribution.

5. Loss Function: The loss function in VAEs consists of two

parts:

• Reconstruction Loss: Measures how well the decoded

samples match the input data.

• This is typically done using a loss function like Mean

Squared Error (MSE) or Binary Cross-Entropy.
 Types of auto-encoders
• KL Divergence: Measures how much the learned latent variable
distribution deviates from the prior distribution.

• This term encourages the latent space to follow the prior

distribution, usually a standard normal distribution.

• The combined loss function can be written as:

𝐿=Reconstruction Loss+KL Divergence

• Applications of VAEs

• Image Generation: VAEs can generate new images similar to

the training data.

• Anomaly Detection: By learning the normal data distribution,

VAEs can help detect anomalies.
 Types of auto-encoders
• Data Imputation: VAEs can be used to fill in missing data by
sampling from the learned distribution.

• Representation Learning: VAEs provide meaningful low-

dimensional representations that can be used for downstream
tasks such as clustering or classification.
 Applications of auto-encoders
1. Dimensionality reduction

• Undercomplete autoencoders are those that are used for

dimensionality reduction.

• These can be used as a pre-processing step for

dimensionality reduction as they can perform fast and
accurate dimensionality reductions without losing much
information.

• Furthermore, while dimensionality reduction procedures

like PCA can only perform linear dimensionality
reductions.
 Applications of auto-encoders
• Undercomplete autoencoders can perform large-scale non-
linear dimensionality reductions.

• 2. Image denoising

• Autoencoders like the denoising autoencoder can be used

for performing efficient and highly accurate image
denoising.

• Unlike traditional methods of denoising, rather than

searching for noise, autoencoders learn a representation of
the noisy data that has been fed to them.
 Applications of auto-encoders
• The representation is then decompressed to form a noise-
free image.

• Denoising autoencoders thus can denoise complex images

that cannot be denoised via traditional methods.

3. Generation of image and time series data

• Variational Autoencoders can be used to generate both

image and time series data.

• VAEs are designed to learn a compressed representation

(latent space) of the input data, which can then be used to
generate new data points.
 Applications of auto-encoders
• VAE’s particularly effective for generate both image and time
series data.

• The key components and processes involved are:

• 1. Encoder: 2. Latent Space: 3. Decoder:

• VAEs can also be adapted to handle time series data, such as

music or stock prices.

• The process is similar to image generation but with a focus

on sequential data:
 Applications of auto-encoders
a) The encoder compresses the sequence into a latent
space.

b) The decoder reconstructs the sequence from the

latent space.

c) New sequences can be generated by sampling from

the latent space and decoding these samples.
 Applications of auto-encoders
• 4. Anomaly detection

• Undercomplete autoencoders can also be used for anomaly

detection.

• For example—consider an autoencoder that has been

trained on a specific dataset P.

• For any image sampled for the training dataset, the

autoencoder is bound to give a low reconstruction loss and

is supposed to reconstruct the image as is.

 Applications of auto-encoders
• For any image which is not present in the training
dataset, however, the autoencoder cannot perform the
reconstruction, as the latent attributes are not adapted for
the specific image that has never been seen by the
network.

• As a result, the outlier image gives off a very high

reconstruction loss and can easily be identified as an
anomaly with the help of a proper threshold.