1

Lesson 12
Generative models

2
Outline
• Introduction to generative models
• Autoencoder
• GANs

3
Introduction to generative models

4
Which face is real?
• http://www.whichfaceisreal.com/

5
Supervised vs. unsupervised learning
Supervised Unsupervised
• Data sample: (x, y) • Data sample: x
x are input features, y is x are input features, no
label label!
• Objective: Learn the • Objective: Learn
mapping function x → y hidden structure and
• Eg.: Classification, relationships from
regression, object inputs
recognition, semantic • Eg: clustering,
segmentation, machine association rules,
translation,... dimension reduction,…

6
Generative models
• Deep generative models (DGM) are neural networks
with many hidden layers trained to approximate
complicated, high-dimensional probability distributions
using samples.

• How to learn pmodel(x) which should be like pdata(x)?

7
Why generative models?
• Because it can uncover underlying hidden information
in data

• Use learned latent distribution to generate more

8
Why generative models? (2)
• Outlier detection: How to detect a rare occurrence of a
new event?
• Use the generative model to learn the distribution of the data,
thereby identifying outliers based on the learned distribution.

9
Autoencoder

10
Autoencoder
• It is an unsupervised learning model that allows to
learn the feature representation with a smaller number
of dimensions from the unlabeled training set
• “Encoder” learns mapping from data x into hidden
space z of lower dimension

11
Autoencoder (2)
• How to learn hidden space?
• Train the model to use the hidden feature z to restore
the original data
• “Decoder” maps the hidden feature z back to recover
the input data information

12
Autoencoder (3)
• How to learn hidden space?
• Train the model to use the hidden feature z to restore
the original data
• The objective function doesn't need a label!

13
The number of hidden dimensions
affects the recovery quality
• Autoencoder is a type of data compression.
• The smaller the number of hidden dimensions, the
larger the training bottleneck

14
Autoencoders for learning representation
• Encoder
• Hidden layers are bottlenecks that force the network to
compress data representation
• Decoder
• The objective function for reconstruction forces the hidden
representation to encode as much information from the input
as possible
• Autoencoding = Automatically encoding data

15
Variational Autoencoders (VAEs)

16
Reviewing Autoencoders
• Each dimension in the encoding vector contains a
single value that represents a hidden attribute from the
input data.

17
Variational Autoencoders
• Each hidden attribute is a probability distribution

18
Decoder in VAEs
• When decoding from the hidden state, take a random
sample from the distribution of each hidden attribute to
generate the input vector for the decoder.

19
VAE optimization

20
VAE optimization

21
Reconstruction loss

22
Regularization term (KL)

23
Hidden space representation
• Only reconstruction loss
• uneven distribution, can not learn a smooth representation of
hidden state
• Only KL divergence loss
• Does not learn the representation of the original data, only
memorize the input data

24
VAE implementation
• With the assumption that the hidden state distribution is a
Gaussian distribution
• The output of the Encoder is two vectors describing the mean and
the variance of the hidden state distribution
• Decoder takes random samples from hidden state and decodes

25
VAE computation graph
• Problem: Cannot back-propagate through the sampling
layer!

26
Reparameterization trick
• Randomly sample ε from a unit Gaussian, and then
shift the randomly sampled ε by the latent distribution's
mean μ and scale it by the latent distribution's
variance σ.

27
Reparameterization trick (2)

28
Reparameterization trick (3)

29
VAEs: hidden space transformation
• Slowly increase or decrease one hidden variable,
keeping other variables fixed
• Each dimension of z encodes meaningful hidden
features

30
VAEs: hidden space transformation (2)
• Ideally, the hidden
variables are
completely
independent of each
other (uncorrelated).
• It is possible to add a
cross-matrix
constraint to force the
hidden variables to be
independent of each
other (the correlation
coefficient between
different hidden
variables is
approximately 0)
• Disentanglement
31
VAEs: Summary
1. Reproducible objective function allows training
without labels (unsupervised)
2. Using re-parameterization to enable end-to-end
training
3. Interpret hidden variables by varying their values and
observing
4. Generate new data

32
Generative adversarial
networks (GANs)

33
Generative adversarial networks
• GAN is a generative model containing two opposing
neural networks
• The generator network turns a noise vector into an
imitation of the data to fool the discriminator.
• The classifier network tries to distinguish between the
real data and the fake data generated by the generator

34
Initial intuition about GANs
• Generator generates fake data from noise

35
Initial intuition about GANs (2)
• Discriminator looks at real and fake data

36
Initial intuition about GANs (3)
• Discriminator looks at real and fake data

37
Initial intuition about GANs (4)
• Discriminator predict what's real and what's fake

38
Initial intuition about GANs (5)
• Discriminator predict what's real and what's fake

39
Initial intuition about GANs (6)
• Discriminator predict what's real and what's fake

40
Initial intuition about GANs (7)
• Discriminator predict what's real and what's fake

41
Initial intuition about GANs (8)
• Generator strives to improve the quality of fake data

42
Initial intuition about GANs (9)
• Generator strives to improve the quality of fake data

43
Initial intuition about GANs (10)
• Generator strives to improve the quality of fake data

44
Initial intuition about GANs (11)
• Discriminator predict what's real and what's fake

45
Initial intuition about GANs (12)
• Discriminator predict what's real and what's fake

46
Initial intuition about GANs (13)
• Discriminator predict what's real and what's fake

47
Initial intuition about GANs (14)
• Discriminator predict what's real and what's fake

48
Initial intuition about GANs (15)
• Generator strives to improve the quality of fake data

49
Initial intuition about GANs (16)
• Generator strives to improve the quality of fake data

50
Initial intuition about GANs (17)
• Generator strives to improve the quality of fake data

51
Initial intuition about GANs (18)
• Discriminator tries to distinguish real and fake data
• Generator tries to improve the quality of fake data to
trick discriminator

52
Training GANs
• Discriminator tries to distinguish real and fake data
• Generator tries to improve the quality of fake data to
trick discriminator

53
Training GANs

54
Image-to-image transformation

58
pix2pix
• Training a conditional GAN to map edges→photo.
• The discriminator, D, learns to classify between fake
(synthesized by the generator) and real {edge, photo}
tuples.
• The generator, G, learns to fool the discriminator.
• Unlike an unconditional GAN, both the generator and
discriminator observe the input edge map

59
Conditional GAN

60
CycleGAN

61
Progressive growing of GANs
(NVIDIA)

62
Progressive growing of GANs
(NVIDIA)

63
Style-based generator

64
Style-based transfer

65
Summary

66
References
1. MIT Deep Learning 6.S191:
http://introtodeeplearning.com/
2. Stanford cs231n :
http://cs231n.stanford.edu/slides/2020/lecture_11.pdf

67
 Thank you
for your
attention!!!

68