Scribd
Upload
4 views

7.Variational Autoencoders

Variational autoencoders (VAEs) are directed models that utilize learned approximate inference and can be trained with gradient-based methods. They generate samples by drawing from a code distribution and using a differentiable generator network, while training involves maximizing a variational lower bound. VAEs are versatile, extendable to various architectures, and achieve state-of-the-art results in generative modeling, though they may produce somewhat blurry images.

Uploaded by

Kavitha
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
4 views

7.Variational Autoencoders

Variational autoencoders (VAEs) are directed models that utilize learned approximate inference and can be trained with gradient-based methods. They generate samples by drawing from a code distribution and using a differentiable generator network, while training involves maximizing a variational lower bound. VAEs are versatile, extendable to various architectures, and achieve state-of-the-art results in generative modeling, though they may produce somewhat blurry images.

Uploaded by

Kavitha
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 4

7.

VARIATIONAL AUTOENCODERS

 The variational autoencoder or VAE is a directed model that uses


learned approximate inference and can be trained purely with gradient-based
methods.

 To generate a sample from the model, the VAE first draws a sample z from
the code distribution pmodel (z).

 The sample is then run through a differentiable generator network g(z ).


Finally, x is sampled from a distribution pmodel (x; g (z)) = pmodel (x |
z).

 However, during training, the approximate inference network (or encoder)


q(z | x) is used to obtain z and pmodel(x | z) is then viewed as a decoder
network.

 The key insight behind variational autoencoders is that they may be


trained by maximizing the variational lower bound L(q) associated with
data point x:

L(q) = Ez∼q(z|x) log pmodel(z, x) + H(q(z (1)


| x)) (2)
= Ez∼q(z|x) log pmodel(x | z) − DKL(q(z | x)||
pmodel(z))
≤ log p model(x). (3)

 When q is chosen to be a Gaussian distribution, with noise added to a


predicted mean value, maximizing this entropy term encourages increasing the
standard deviation of this noise.

 More generally, this entropy term encourages the variational posterior to place
high probability mass on many z values that could have generated x.

 The second term tries to make the approximate posterior distribution q(z |
x) and the model prior pmodel(z) approach each other.

1
 Traditional approaches to variational inference and learning infer q via an
opti- mization algorithm.

 These approaches are slow and often require the ability to compute Ez∼q
log p model(z, x) in closed form.

 The main idea behind the variational autoencoder is to train a parametric


encoder (also sometimes called an inference network or recognition
model) that produces the parameters of q.

 So long as z is a continuous variable, we can then back-propagate through


samples of z drawn from q(z | x) = q (z; f(x; θ)) in order to obtain a
gradient with respect to θ.

 Learning then consists solely of maximizing L with respect to the


parameters of the encoder and decoder.

 All of the expectations in L may be approximated by Monte Carlo


sampling.

 The variational autoencoder approach is elegant, theoretically pleasing,


and simple to implement.

 It also obtains excellent results and is among the state of the art approaches
to generative modeling.

 Its main drawback is that samples from variational autoencoders trained on


images tend to be somewhat blurry.

 The causes of this phenomenon are not yet known. One possibility is that
the blurriness is an intrinsic effect of maximum likelihood, which minimizes
DKL(pdatapmodel).

2
VAE Framework

 The VAE framework is very straightforward to extend to a wide range of


model architectures.

 This is a key advantage over Boltzmann machines, which require extremely


careful model design to maintain tractability.

 VAEs work very well with a diverse family of differentiable operators.


One particularly sophisticated VAE is the deep recurrent attention
writer or DRAW model.

 DRAW uses a recurrent encoder and recurrent decoder combined with an


attention mechanism.

 The generation process for the DRAW model consists of sequentially


visiting different small image patches and drawing the values of the pixels at
those points.

 VAEs can also be extended to generate sequences by defining variational


RNNs by using a recurrent encoder and decoder within the VAE framework.

 Generating a sample from a traditional RNN involves only non-deterministic


operations at the output space.

 Variational RNNs also have random variability at the potentially more


abstract level captured by the VAE latent variables.

3
4

You might also like