Autoencoders

Autoencoders, Extensions, and Applications
Piyush Rai
IIT Kanpur
Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 1

Outline
Introduction to Autoencoders
Autoencoder Variants and Extensions
Some Applications of Autoencoders
Autoencoders for Recommender Systems

Autoencoder
Similar to the standard feedforward neural network with a key difference:

Unsupervised. No “label” at the output layer; Output layer simply tries to “recreate” the input

Autoencoder

Defined by two (possibly nonlinear) mapping functions: Encoding function f , Decoding function g

Autoencoder

h = f (x) denotes an encoding (possibly nonlinear) for the input x

Autoencoder

x̂ = g (h) = g (f (x)) denotes the reconstruction (or the “decoding”) for the input x

Autoencoder

x̂ = g (h) = g (f (x)) denotes the reconstruction (or the “decoding”) for the input x
For an Autoencoder, f and g are learned with a goal to minimize the difference between x̂ and x

Autoencoder for Feature Learning
The learned code h = f (x) can be used as a new feature representation of the input x
Therefore autoencoders can also be used for “feature learning”

Autoencoder for Feature Learning
The learned code h = f (x) can be used as a new feature representation of the input x
Therefore autoencoders can also be used for “feature learning”

Note: Size of the hidden units (encoding) can also be larger than the input

A Simple Autoencoder
Let’s assume a D × 1 input x ∈ RD , and a single hidden layer with K × 1 code h ∈ RK

We can then define a simple linear autoencoder as
h = f (x) = Wx + b
x̂ = g (h) = W∗ h + c
where f is defined by W ∈ RK ×D and b ∈ RK ×1


h = f (x) = Wx + b
x̂ = g (h) = W∗ h + c
where f is defined by W ∈ RK ×D and b ∈ RK ×1 , g is defined by W∗ ∈ RD×K and c ∈ RD×1


h = f (x) = Wx + b
x̂ = g (h) = W∗ h + c


h = f (x) = Wx + b
x̂ = g (h) = W∗ h + c
Note: If we learn f , g to minimize the squared error ||x̂ − x||2 then the linear autoencoder with
W∗ = W> is optimal


h = f (x) = Wx + b
x̂ = g (h) = W∗ h + c
Note: If we learn f , g to minimize the squared error ||x̂ − x||2 then the linear autoencoder with
W∗ = W> is optimal, and is equivalent to Principal Component Analysis (PCA)

Autoencoder: Zooming in..

W: K × D matrix of weights of edges between input and hidden layer


Wkd is the weight of edge connecting input layer node d to hidden layer node k


W∗ : D × K matrix of weights of edges between hidden and output layer



∗
Wdk is the weight of edge connecting hidden layer node k to output layer node d



∗
Wdk is the weight of edge connecting hidden layer node k to output layer node d
If W∗ = W> , the autoencoder architecture is said to have “tied weights”

Nonlinear Autoencoders
The hidden nodes can also be nonlinear transforms of the inputs, e.g.,
Can define h as a linear transform of x followed by a nonlinearity (e.g., sigmoid, ReLU)
h = sigmoid(Wx + b)
1
where the nonlinearity sigmoid(z) = 1+exp(−z)
squashes the real-valued z to lie between 0 and 1

h = sigmoid(Wx + b)
1
Most commonly used autoencoders use such nonlinear transforms

h = sigmoid(Wx + b)
1
Most commonly used autoencoders use such nonlinear transforms

Note: If inputs x ∈ {0, 1}D are binary, it may be more appropriate to also define x̂ as
x̂ = sigmoid(W∗ h + c)

What’s Learned by an Autoencoder?
Figure below: The K × D matrix W learned on digits data. Each tiny block visualizes a row of W

Thus W captures the possible “patterns” in the training data (akin to the K basis vectors in PCA)

Thus W captures the possible “patterns” in the training data (akin to the K basis vectors in PCA)
For any input x, the encoding h tells us how much each of these K features in present in x

Training the Autoencoder
To train the autoencoder, we need to define a loss function `(x̂, x)

The loss function (a function of parameters W, b, W∗ , c) can be defined using various ways

In general, it is defined in terms of the difference between x̂ and x (reconstruction error)

For a single input x = [x1 , . . . , xD ] and its reconstruction x̂ = [x̂1 , . . . , x̂D ]
D
X
`(x̂, x) = (x̂d − xd )2 (squared loss; used if input are real-valued)
d=1

D
X
d=1
D
X
`(x̂, x) = − [xd log(x̂d ) + (1 − xd ) log(1 − x̂d )] (cross-entropy loss; used if input are binary)
d=1

D
X
d=1
D
X
d=1
We find (W, b, W∗ , c) by minimizing the reconstruction error (summed over all training data)

D
X
d=1
D
X
d=1
We find (W, b, W∗ , c) by minimizing the reconstruction error (summed over all training data)
This can be done using backpropagation
Undercomplete, Overcomplete, and Need for Regularization

In both cases, it is important to control the capacity of encoder and decoder


Undercomplete: Imagine K = 1 and very powerful f and g . Can achieve very small
reconstruction error but the learned code will not capture any interesting properties in the data


Overcomplete: Imagine K ≥ D and trivial (identity) functions f and g . Can achieve even zero


Overcomplete: Imagine K ≥ D and trivial (identity) functions f and g . Can achieve even zero
It is therefore important to regularize the functions as well as the learned code, and not just focus
on minimizing the reconstruction error
Regularized Autoencoders
Several ways to regularize the model, e.g.

Make the learned code sparse (Sparse Autoencoders)
Make the model robust against noisy/incomplete inputs (Denoising Dutoencoders)

Regularized Autoencoders
Several ways to regularize the model, e.g.

Make the learned code sparse (Sparse Autoencoders)
Make the model robust against noisy/incomplete inputs (Denoising Dutoencoders)
Make the model robust against small changes in the input (Contractive Autoencoders)

Sparse Autoencoders
Make the learned code sparse (Sparse Autoencoders). Done by adding a sparsity penalty on h
Loss Function: `(x̂, x) + Ω(h)

PK
where Ω(h) = k=1 |hk | is the `1 norm of h

Sparse Autoencoders
Make the learned code sparse (Sparse Autoencoders). Done by adding a sparsity penalty on h
Loss Function: `(x̂, x) + Ω(h)

PK
where Ω(h) = k=1 |hk | is the `1 norm of h
Sparse autoencoder is learned by minimizing the above regularized loss function

Denoising Autoencoders
First add some noise (e.g., Gaussian noise) to the original input x
Let’s denote x̃ as the corrupted version of x
The encoder f operates on x̃, i.e., h = f (x̃)

However, we still want to reconstruction x̂ to be close to the original uncorrupted input x

However, we still want to reconstruction x̂ to be close to the original uncorrupted input x

Since the corruption is stochastic, we minimize the expected loss: Ex̃∼p(x̃|x) [`(x̂, x̃)] + Ω(h)

Deep/Stacked Autoencoders
Most autoencoders can be extended to have more than one hidden layer

Stochastic Autoencoders
Can also define the encoder and decoder functions using probability distributions
pencoder (h|x)
pdecoder (x|h)

pencoder (h|x)
pdecoder (x|h)
The choice of distributions depends on the type of data being modeled and of the encodings

pencoder (h|x)
pdecoder (x|h)
This gives a probabilistic approach for designing autoencoders

pencoder (h|x)
pdecoder (x|h)

Negative log-likelihood − log pdecoder (x|h) is equivalent to the reconstruction error

pencoder (h|x)
pdecoder (x|h)

Can also use a prior distribution p(h) on the encodings (equivalent to regularizer)

pencoder (h|x)
pdecoder (x|h)

Such ideas have been used to design generative models for autoencoders

pencoder (h|x)
pdecoder (x|h)

Variational Autoencoder (VAE) is a popular example of such a model

pencoder (h|x)
pdecoder (x|h)

Variational Autoencoder (VAE) is a popular example of such a model
Generative models like VAE can be used to “generate” new data using a random code h

Variational Autoencoders (VAE)
Learns a distribution (e.g., a Gaussian) on the encoding1
1 http://www.birving.com/presentations/autoencoders/
Variational Autoencoders (VAE)
Learns a distribution (e.g., a Gaussian) on the encoding1
Unlike standard AE, a VAE model learns to generate plausible data from random encodings
1 http://www.birving.com/presentations/autoencoders/
Some Applications of Autoencoders
(Unsupervised) Feature learning and Dimensionality reduction

Denoising and inpainting
Pre-training of deep neural networks
Recommender systems applications

Feature learning and Dimensionality Reduction
Example: A deep AE for low-dim feature learning for 784-dimensional MNIST images2
2 Figure credit: Hinton and Salakhutdinov

Feature learning and Dimensionality Reduction
Example: Low-dim feature learning for 2000-dimensional bag-of-words documents

Denoising and Inpainting

Denoising and Inpainting

Applications in Recommender Systems

Recommender Systems
Assume we are given a partially known N × M ratings matrix R of N users on M items (movies)
Denote by r (u) the (partially known) M × 1 ratings vector of user u
Denote by r (i) the (partially known) N × 1 ratings vector of item i

Recommender Systems
Assume we are given a partially known N × M ratings matrix R of N users on M items (movies)
Denote by r (u) the (partially known) M × 1 ratings vector of user u
Denote by r (i) the (partially known) N × 1 ratings vector of item i

How can we use this data to build a recommender system?

Recommender Systems via Matrix Completion
An idea: If the predicted value of a user’s rating for a movie is high, then we should ideally
recommend this movie to the user

Recommender Systems via Matrix Completion
An idea: If the predicted value of a user’s rating for a movie is high, then we should ideally
recommend this movie to the user
Thus if we can “reconstruct” the missing entries in R, we can use this method to recommend
movies to users. Using an autoencoders can help us do this!

An Autoencoder based Approach
Using the rating vectors {r (u) }N

u=1 of all users, can learn an autoencoder
Note: During backprop, only update weights in W that are connected to the observed ratings3
Once learned, the model can predict (reconstruct) the missing ratings
3 AutoRec: Autoencoders Meet Collaborative Filtering (Sedhain et al, WWW 2015)

Another Autoencoder based Approach
Another approach is to combine (denoising) autoencoders with a matrix factorization model4
4 Deep Collaborative Filtering via Marginalized Denoising Auto-encoder (Li et al, CIKM 2015)
Idea: Rating of a user u on an item i can be defined using the inner-product based similarity of
>
their features learned via an autoencoder: Rui = f (h (u) h (i) ) where f is some compatibity function
Idea: Rating of a user u on an item i can be defined using the inner-product based similarity of
>
their features learned via an autoencoder: Rui = f (h (u) h (i) ) where f is some compatibity function
Denoting {h (u) }N (i) M

u=1 = U and {h }i=1 = V, we can write R = UV
>
Other Approaches on Autoencoders for Recommender Systems
Several recent papers on similar autoencoder based ideas

Collaborative Denoising Auto-Encoders for Top-N Recommender Systems (Wu et al, WSDM 2016)
Collaborative Deep Learning for Recommender Systems (Wang et al, KDD 2015)
Also possible to incorporate side information about the users and/or items (Wang et al, KDD 2015)

Autoencoders: Summary
Simple and powerful for (nonlinear) feature learning
Learned features are able to capture salient properties of data

Several extensions (sparse, denoising, stochastic, etc.)
Can also be stacked to create “deep” autoencoders
Recent focus on autoencoders that are based on generative models of data
Example: Variational Autoencoders

Autoencoders

Uploaded by

Document Informationclick to expand document information

Copyright:

Available Formats

Autoencoders

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Autoencoders

Uploaded by

Copyright:

Available Formats

Autoencoders, Extensions, and Applications

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 1

Autoencoders for Recommender Systems

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 2

Similar to the standard feedforward neural network with a key difference:

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 3

Similar to the standard feedforward neural network with a key difference:

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 3

Similar to the standard feedforward neural network with a key difference:

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 3

Similar to the standard feedforward neural network with a key difference:

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 3

Similar to the standard feedforward neural network with a key difference:

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 3

Therefore autoencoders can also be used for “feature learning”

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 4

Therefore autoencoders can also be used for “feature learning”

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 4

Let’s assume a D × 1 input x ∈ RD , and a single hidden layer with K × 1 code h ∈ RK

where f is defined by W ∈ RK ×D and b ∈ RK ×1

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 5

Let’s assume a D × 1 input x ∈ RD , and a single hidden layer with K × 1 code h ∈ RK

where f is defined by W ∈ RK ×D and b ∈ RK ×1 , g is defined by W∗ ∈ RD×K and c ∈ RD×1

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 5

Let’s assume a D × 1 input x ∈ RD , and a single hidden layer with K × 1 code h ∈ RK

where f is defined by W ∈ RK ×D and b ∈ RK ×1 , g is defined by W∗ ∈ RD×K and c ∈ RD×1

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 5

Let’s assume a D × 1 input x ∈ RD , and a single hidden layer with K × 1 code h ∈ RK

where f is defined by W ∈ RK ×D and b ∈ RK ×1 , g is defined by W∗ ∈ RD×K and c ∈ RD×1

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 5

Let’s assume a D × 1 input x ∈ RD , and a single hidden layer with K × 1 code h ∈ RK

where f is defined by W ∈ RK ×D and b ∈ RK ×1 , g is defined by W∗ ∈ RD×K and c ∈ RD×1

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 5

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 6

W: K × D matrix of weights of edges between input and hidden layer

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 6

W: K × D matrix of weights of edges between input and hidden layer

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 6

W: K × D matrix of weights of edges between input and hidden layer

W∗ : D × K matrix of weights of edges between hidden and output layer

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 6

W: K × D matrix of weights of edges between input and hidden layer

W∗ : D × K matrix of weights of edges between hidden and output layer

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 6

W: K × D matrix of weights of edges between input and hidden layer

W∗ : D × K matrix of weights of edges between hidden and output layer

If W∗ = W> , the autoencoder architecture is said to have “tied weights”

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 6

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 7

Most commonly used autoencoders use such nonlinear transforms

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 7

Most commonly used autoencoders use such nonlinear transforms

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 7

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 8

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 8

Piyush Rai (IIT Kanpur) Autoencoders, Extensions, and Applications 8

To train the autoencoder, we need to define a loss function `(x̂, x)