Autoencoders

Presented By:

Parag Choudhury (210710007032)

Chao Asheem Gogoi (210710007012)
Jintu Nath (210710007019)

1
What is Autoencoders?

★ unsupervised machine learning

algorithm.

★compress the data and reduce

its dimensionality.

2
Components of Autoencoders

★ Encoder.

★ Code/Bottleneck.

★ Decoder

3
Basic architecture of autoencoders 4
Properties of Autoencoders

Unsupervised
Data-specific

Lossy

5
Autoencoders and PCA
★ Both PCA and autoencoders are used for
dimensionality reduction.

★ PCA reduces dimensionality by projecting data

onto a new axis.

★ Autoencoders reduce dimensionality by

useing an encoder-decoder network.

★ If an autoencoder has only one hidden layer, a

linear activation function and mean squared error
(MSE) loss, the Autoencoder become equivalent
to PCA.
Regularization in autoencoders
Regularization -> prevent overfitting and improve generalization

Overfitting :
Training data - high accuracy

Test data - low accuracy

Generalization : How well a model performs on new unseen data.

7
8
Remove certain neuron -> increase linearity

9
Two main types of Regularization

L1 Regularization(Lasso Regression)

L2 Regularization (Ridge Regression)

10
L1 Regularization(Lasso Regression):

Adds “absolute value of magnitude”of the coefficient

as a penalty term to the loss function
number of featres

regularization
actual target value
parameter
number of examples coefficient
predicted target value 11
L2 Regularization (Ridge Regression):

Adds “square magnitude” of the coefficient as a

penalty term to the loss function
number of features

number of examples
regularization coefficient
actual target value
parameter
12
predicted target value
Denoising autoencoders
Remove noise from data.

Create a corrupted copy of the input by intoroducing some noise .

Helps to avoid the autoencoders to copy the input to the output, without learning
features about the data .

13
Denoising Autoencoders

14
15
Sparse Autoencoder
A Sparse Autoencoder (SAE) is a type of autoencoder that enforces
sparsity on the hidden layer.

Instead of reducing dimensions like traditional autoencoders, it limits the

number of active neurons per input.

Most neurons are inactive, Only a few neurons “light up” at a time.

Think of it like: A team where only the specialists work on a task, not
everyone
16
Here is the diagram of SAE
where only blue color
neurons are activate and
rest others are in neutral
state for a particular
training dataset

17
Advantages of Sparse Autoencoder
Improved Generalization

Reduced Computational Complexity

Lower Risk of Overfitting

Feature Extraction

18
How Do We Make It Sparse
Add a rule (called an L1 penalty) to the training process.

What it does: Encourages most neuron outputs to be zero.

Result: Only a few neurons are active for each input.

Analogy: Like paying a small fine every time too many team members join
in—keeps the group small.

19
 Contractive Autoencoder
Encoder
compresses
noisy image into
a lower-
dimensional
representation.
Decoder
reconstructs
original image
from compressed
representation. 20
Working of CAE
Additional regularization constraints that enforce robustness.

Contractive penalty is applied here to the neurons respond to small

changes in input.

Ensures that each neuron does not overreact to slight input variations,
making the encoding more stable and robust

21
Sensitivity

here,
hi --> output image
xj ---> input image

22
Loss Function

Loss

Reconstruction Error
coefficient penalty factor
for sensitivity and
stability

23
24
Thank You

25