CS294-158 Deep Unsupervised Learning

Lecture 5 Implicit Models -- Generative Adversarial Networks (GANs)

Pieter Abbeel, Wilson Yan, Kevin Frans, Philipp Wu

 Outline
■ Motivation
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 2
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 3
 Motivation: GAN Progress

- Ian Goodfellow is first-author on the first GAN paper

- GAN is most prominent of Implicit Models

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 4
 Motivation: BigGAN

[BigGAN, Brock, Donahue, Simonyan, 2018]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 5
 Motivation: GAN Art

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 6
 So far...
■ Autoregressive models
■ MADE, PixelRNN/CNN, Gated PixelCNN, PixelSNAIL
■ Flow models
■ Autoregressive Flows, NICE, RealNVP, Glow, Flow++
■ Latent Variable Models
■ VAE, IWAE, VQ-VAE

■ Common aspect: Likelihood-based models

■ exact (autoregressive and flows)
■ approximate (VAE)
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 7
 Generative Models
■ Sample
■ Evaluate likelihood
■ Train
■ Representation

→ What if all we care about is sampling?

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 8
 Building a sampler
■ How about this sampler?
import glob, cv2, numpy as np
files = glob.glob(‘*.jpg’)
def _sample():
idx = np.random.randint(len(files))
return cv2.imread(files[idx])
def sample(*, n_samples):
samples = np.array([_sample() for _ in range(n_samples)])
return samples

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 9
 Building a sampler
■ You don’t just want to sample the exact data points you have.
■ You want to build a generative model that can understand the
underlying distribution of data points and
■ smoothly interpolate across the training samples
■ output samples similar but not the same as training data samples
■ output samples representative of the underlying factors of variation in
the training distribution.
■ Example: digits with unseen strokes, faces with unseen poses, etc.

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 10
 Implicit Models
■ Sample z from a fixed noise source distribution
(uniform or gaussian).
■ Pass the noise through a deep neural network
to obtain a sample x.
■ Sounds familiar? Right:
■ Flow Models
■ VAE
■ What’s going to be different here?
■ Learning the deep neural network without explicit
density estimation

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 11
 Implicit Models
■

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 12
 Departure from maximum likelihood
■

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 13
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 14
Generative Adversarial Networks

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 15
 Generative Adversarial Networks

■ Two player minimax game between generator (G) and discriminator (D)
■ (D) tries to maximize the log-likelihood for the binary classification problem
- data: real (1)
- generated: fake (0)
■ (G) tries to minimize the log-probability of its samples being classified as “fake” by
the discriminator (D)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 16
Generative Adversarial Networks

Figure from NeurIPS 2016

GAN Tutorial (Goodfellow)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 17
 GANs - Pseudocode

[Goodfellow et al 2014]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 18
 GAN
See it in action: https://poloclub.github.io/ganlab/

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 19
 GAN samples from 2014

Figure from Goodfellow et al 2014

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 20
 How to evaluate?
■ Evaluation for GANs is still an open problem
■ Unlike density models, you cannot report explicit
likelihood estimates on test sets.

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 21
 Parzen-Window density estimator
■ Also known as Kernel Density Estimator (KDE)
■ An estimator with kernel K and bandwidth h:

■ In generative model evaluation, K is usually density function

of standard Normal distribution

Bishop 2006

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 22
 Parzen-Window density estimator
■ Bandwidth h matters
■ Bandwidth h chosen according to validation set

Bishop 2006

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 23
 Evaluation

Parzen Window density estimates (Goodfellow et al, 2014)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 24
 Parzen-Window density estimator
■ Parzen Window estimator can be unreliable

[A note on the evaluation of generative models (Theis, Van den Oord, Bethge 2015)]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 25
 Inception Score
■ Can we side-step high-dim density estimation?
■ One idea: good generators generate samples that are semantically diverse
■ Semantics predictor: trained Inception Network v3
■ p(y|x), y is one of the 1000 ImageNet classes
■ Considerations:
■ each image x should have distinctly recognizable object -> p(y|x) should have
low entropy
■ there should be as many classes generated as possible -> p(y) should have
high entropy

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 26
 Inception Score
■ Inception model:
■ Marginal label distribution:
■ Inception Score:

[Improved GAN: Salimans et al 2016]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 27
 Inception Score

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 28
 Fréchet Inception Distance
■ Inception Score doesn’t sufficiently measure diversity: a list of 1000 images (one of
each class) can obtain perfect Inception Score
■ FID was proposed to capture more nuances
■ Embed image x into some feature space (2048-dimensional activations of the
Inception-v3 pool3 layer), then compare mean (m) & covariance (C) of those
random features

[Heusel et al, 2017]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 29
 Fréchet Inception Distance

[Heusel et al, 2017]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 30
 Fréchet Inception Distance

[Heusel et al, 2017]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 31
 Generative Adversarial Networks
■ Key pieces of GAN
■ Fast sampling
■ Notion of optimizing directly for what you care about - perceptual
samples
■ No inference, i.e. no evaluation of p(sample) or p(data)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 32
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding
Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 33
GAN: Bayes-Optimal Discriminator
Discriminator Data distribution

Model / Generator distribution

[Figure Source: Goodfellow

NeurIPS 2016 Tutorial on GANs]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 34
 GAN: Bayes-Optimal Discriminator
■ What’s the optimal discriminator given generated and true distributions?

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 35
GAN: Generator Objective under Bayes-Optimal Discriminator D* ?

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 36
 KL, reverse KL, JSD

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 37
Mode covering vs Mode seeking: Tradeoffs
■ For compression, one would prefer to ensure all points in the data distribution are
assigned probability mass.
■ For generating good samples, blurring across modes spoils perceptual quality
because regions outside the data manifold are assigned non-zero probability mass.
■ Picking one mode without assigning probability mass on points outside can
produce “better-looking” samples.
■ Caveat: More expressive density models can place probability mass more
accurately.
Example: Using mixture of Gaussians as opposed to a single isotropic gaussian.

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 38
 Mode Collapse

Standard GAN training collapses when the true distribution is a mixture of

gaussians (Figure from Metz et al 2016)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 39
 Back to our GAN objective
Recall

Mini-Exercise
■ Is it feasible to run the inner optimization to completion?
■ For this specific objective, would it create problems if we were able to do so?

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 40
 Discriminator Saturation
■ Generator samples confidently classified as fake by the discriminator receive no
gradient for the generator update.

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 41
Avoiding Discriminator Saturation: (1) Alternating Optimization
■ Alternate gradient steps on discriminator and generator objectives

■ Balancing these two updates is hard for the zero-sum game

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 42
Avoiding Discriminator Saturation: (2) Non Saturating Formulation

Not zero-sum

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 43
Avoiding Discriminator Saturation: (2) Non Saturating Formulation
■ ORIGINAL ISSUE: Generator samples confidently classified as fake by the
discriminator receive no gradient for the generator update.
■ FIX: non-saturating loss for when discriminator confident about fake

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 44
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 45
Deep Convolutional GAN (DCGAN)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 46
 Deep Convolutional GAN (DCGAN)

[Radford et al 2016]
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 47
 DCGAN - Architecture Design
■ Supervised Learning CNNs not directly usable
■ Remove max-pooling and mean-pooling
■ Upsample using transposed convolutions in the generator
■ Downsample with strided convolutions and average pooling
■ Non-Linearity: ReLU for generator, Leaky-ReLU (0.2) for discriminator
■ Output Non-Linearity: tanh for Generator, sigmoid for discriminator
■ Batch Normalization used to prevent mode collapse
■ Batch Normalization is not applied at the output of G and input of D
■ Optimization details
■ Adam: small LR - 2e-4; small momentum: 0.5, batch-size: 128
[Radford et al 2016]
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 48
 DCGAN Batch Norm

Chintala 2016

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 49
 DCGAN - Key Results
■ Good samples on datasets with 3M images (Faces, Bedrooms) for the first
time

[Radford et al 2016]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 50
 DCGAN - Key Results

[Radford et al 2016]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 51
 DCGAN - Key Results
■ Smooth interpolations in high dimensions

[Radford et al 2016]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 52
 DCGAN - Key Results
■ Imagenet samples
(32x32)

[Radford et al 2016]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 53
 DCGAN - Key Results
■ Vector Arithmetic

[Radford et al 2016]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 54
 DCGAN - Key Results

[Radford et al 2016]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 55
 DCGAN - Key Results

[Radford et al 2016]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 56
 DCGAN - Key Results
Representation Learning

[Radford et al 2016]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 57
 DCGAN - Conclusions
■ Incredible samples for any generative model
■ GANs could be made to work well with architecture details
■ Perceptually good samples and interpolations
■ Representation Learning
■ Problems to address:
■ Unstable training
■ Brittle architecture / hyperparameters

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 58
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 59
 Improved training of GANs
■ Minibatch discrimination

■ Feature Matching

■ Historical Averaging

■ Virtual batch normalization

Salimans 2016

■ One-sided label smoothing

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 60
 Improved training of GANs
■ Minibatch discrimination

Salimans 2016

Allows to incorporate side information from other samples and is superior to feature matching in the unconditional setting.
Helps addressing mode collapse by allowing discriminator to detect if the generated samples are too close to each other.
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 61
 Improved training of GANs
■ Feature Matching

Add this to Generator objective

Salimans 2016

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 62
 Improved training of GANs
■ Historical Averaging

Salimans 2016

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 63
 Improved training of GANs
■ Virtual Batch Normalization

Figure source:
NeurIPS tutorial
Goodfellow 2016

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 64
 Improved training of GANs
■ Virtual Batch Normalization
■ Use a reference batch (fixed) to compute normalization statistics
■ Construct a batch containing the sample and reference batch

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 65
 Improved training of GANs
■ One-sided label smoothing

Figure source:
NeurIPS tutorial
Goodfellow 2016

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 66
 Improved training of GANs
■ Semi-Supervised Learning
■ Predict labels in addition to fake/real in the discriminator
■ Approximate way of modeling p(x,y)
■ Generator doesn’t have to be made conditional p(x|y)
■ Use a deeper architecture for the discriminator compared to
generator

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 67
 Improved training of GANs
■ Inception Score

■ Approximate way of modeling p(x,y)

■ Correlates with human judgement
■ Captures some necessity for diversity

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 68
 Improved training of GANs

Salimans 2016

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 69
 fGAN - variational divergence

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 70
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 71
 Wasserstein Distance
■

■ Another distance measure inspired from Optimal Transport is the Earth Mover
(EM) distance

■ Goal: Design a GAN objective function such that the generator minimizes the Earth
Mover / Wasserstein distance between data and generated distributions.

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 72
 Kantorovich Rubinstein Duality
■

■ Intractable to estimate

■ Search over joint distributions is now a search over 1-Lipschitz functions

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 73
 Wasserstein GAN

Another distance measure inspired from Optimal Transport is the Earth Mover
(EM) distance

■ Supremum over linear (function space) expectations => search over K-Lipschitz
gives you K times the Wasserstein distance.

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 74
 Wasserstein GAN - Pseudocode

(Arjovsky et al 2017)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 75
Wasserstein GAN - Training critic to converge

(Arjovsky et al 2017)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 76
Wasserstein distance correlates with sample quality
Wasserstein Estimate JSD Estimate

(Arjovsky et al 2017)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 77
 WGAN Samples on par with DCGAN

(Arjovsky et al 2017)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 78
 WGAN robust to architecture choices

(Arjovsky et al 2017)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 79
 WGAN robust to architecture choices

(Arjovsky et al 2017)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 80
 WGAN Summary
Standard GAN

Wasserstein GAN

(Arjovsky et al 2017)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 81
 WGAN Summary
■ New divergence measure for optimizing the generator
■ Addresses instabilities with JSD version (sigmoid cross entropy)
■ Robust to architectural choices
■ Progress on mode collapse and stability of derivative wrt input
■ Introduces the idea of using lipschitzness to stabilize GAN training
■ Negative:

(Arjovsky et al 2017)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 82
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 83
WGAN-GP: Gradient Penalty for Lipschitzness

Gulrajani et al 2017

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 84
WGAN-GP: Gradient Penalty for Lipschitzness

Gulrajani et al 2017
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 85
 WGAN-GP: Pseudocode

Gulrajani et al 2017

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 86
 WGAN-GP: BatchNorm

Gulrajani et al 2017
Since then no-one uses batchnorm in D anymore
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 87
WGAN-GP: Robustness to architectures

Gulrajani et al 2017

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 88
WGAN-GP: Robustness to architectures

Gulrajani et al 2017

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 89
 WGAN-GP: High quality samples

Gulrajani et al 2017

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 90
 WGAN-GP: High quality samples

Gulrajani et al 2017

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 91
 WGAN-GP: Summary
■ Robust to architectural choices

■ Became a very popular GAN model - 10,000+ citations, has been used in
NVIDIA’s Progressive GANs, StyleGAN, etc - biggest GAN successes

■ Residual architecture widely adopted

■ Possible negative: slow wall clock time due to gradient penalty

■ Gradient penalty applied on a heuristic distribution of samples from

current generator. Could be unstable when learning rates are high

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 92
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 93
 Progressive growing of GANs

Karras et al 2017
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 94
 Progressive growing of GANs

First super-high quality samples from any generative models Karras et al 2017
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 95
 Progressive growing of GANs

Karras et al 2017

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 96
 Progressive growing of GANs

Karras et al 2017

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 97
 Progressive growing of GANs

K
a
r
r
a
s
e
t
a
l
2
0
1
7

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 98
 Progressive growing of GANs

Karras et al 2017

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 99
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 100
Spectral Normalization GAN (SNGAN)

Miyato et al 2017

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 101
Spectral Normalization GAN (SNGAN)

Miyato et al 2017
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 102
 Spectral Normalization GAN (SNGAN)
■ Key idea: Connecting Lipschitzness of discriminator to spectral norm
of each layer.

Miyato et al 2017

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 103
 Spectral Normalization GAN (SNGAN)

In practice: sigma = max of norm over all data passed through the layer Miyato et al 2017

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 104
Spectral Normalization GAN (SNGAN)

Miyato et al 2017

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 105
Spectral Normalization GAN (SNGAN)

Miyato et al 2017

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 106
 Projection Discriminator

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 107
 SNGAN: Summary

■ High quality class conditional samples at Imagenet scale

■ First GAN to work on full Imagenet (million image dataset)

■ Computational benefits over WGAN-GP (single power

iteration and no need of a backward pass)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 108
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 109
 Self Attention GAN (SAGAN)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 110
 Self Attention GAN (SAGAN)

Zhang et al 2018

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 111
 Self Attention GAN (SAGAN)

Zhang et al 2018

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 112
 Self Attention GAN (SAGAN)
■ Salient bits:
■ Applies spectral normalization to both the generator and
discriminator weight matrices
■ This is counter-intuitive to popular belief that you only have to
mathematically condition the discriminator
■ Uses self-attention in both the generator and discriminator
■ Hinge Loss
■ First GAN to produce “good” unconditional full Imagenet samples
■ Conditional models
■ Conditional BN for G, Projection Discriminator for D

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 113
 Self Attention GAN (SAGAN)

Zhang et al 2018
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 114
 Self Attention GAN (SAGAN)

Zhang et al 2018

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 115
 Self Attention GAN (SAGAN)

Zhang et al 2018

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 116
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 117
 BigGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 118
 BigGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 119
 BigGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 120
 BigGAN and BigGAN-deep

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 121
 BigGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 122
 BigGAN-deep

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 123
 BigGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 124
 BigGAN
■ Salient bits
■ Increase your batch size (as much as you can)
■ Use Cross-Replica (Sync) Batch Norm
■ Increase your model size
■ Wider helps as much as deeper
■ Fuse class information at all levels
■ Hinge Loss
■ Orthonormal regularization & Truncation Trick (on z)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 125
 BigGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 126
 BigGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 127
 BigGAN - Truncation Trick

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 128
 BigGAN - Sampling

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 129
 BigGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 130
 BigGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 131
 StyleGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 132
 StyleGAN - Adaptive Instance Norm

Instance normalization = normalize per channel per sample

Adaptive ___________ = addition of learned y inputs above
Note: layer normalization = normalize per sample
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 133
 StyleGAN - Style Transfer

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 134
 StyleGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 135
StyleGAN - Effect of adding noise

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 136
 StyleGAN-v2

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 137
 StyleGAN-v2

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 138
 StyleGAN-v2

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 139
 Information Bottleneck

Variational Information Bottleneck [Alemi et al., 2016]

Variational Information Bottleneck GAN [Peng et al, 2019]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 140
 Information Bottleneck

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 141
 Information Bottleneck

Variational Information Bottleneck (VIB)

[Alemi et al., 2016]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 142
Variational Information Bottleneck

Real Fake

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 143
Variational Information Bottleneck
Instance Noise
[Salimans et al. 2016; Sønderby et al.
2016; Arjovsky and Bottou 2017]

Real Fake

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 144
Variational Information Bottleneck
Instance Noise
[Salimans et al. 2016; Sønderby et al.
2016; Arjovsky and Bottou 2017]

Real Fake

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 145
Variational Information Bottleneck
Instance Noise
[Salimans et al. 2016; Sønderby et al.
2016; Arjovsky and Bottou 2017]

Real Fake

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 146
Variational Information Bottleneck
Instance Noise
[Salimans et al. 2016; Sønderby et al.
2016; Arjovsky and Bottou 2017]

Real Fake

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 147
Variational Information Bottleneck
Instance Noise
[Salimans et al. 2016; Sønderby et al.
2016; Arjovsky and Bottou 2017]

Real Fake

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 148
Variational Information Bottleneck
Variational Information Bottleneck
[Alemi et al., 2016]

Real Fake

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 149
Variational Information Bottleneck
Variational Information Bottleneck
[Alemi et al., 2016]

Real Fake

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 150
Variational Information Bottleneck
Variational Information Bottleneck
[Alemi et al., 2016]

Real Fake

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 151
Variational Information Bottleneck
Variational Information Bottleneck
[Alemi et al., 2016]

Real Fake

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 152
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 153
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 154
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 155
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 156
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 157
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 158
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 159
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 160
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 161
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 162
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 163
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 164
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 165
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 166
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 167
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 168
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 169
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 170
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 171
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 172
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 173
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 174
 Conditional GANs / pix2pix

Slide: Phillip Isola

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 175
 Video2Video (NVIDIA)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 176
 Everybody Dance Now

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 177
NVIDIA GauGAN: sketch->photorealistic image

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 178
 Learning to paint (GANs + RL)
https://learning-to-paint.github.io/

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 179
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 180
 GANs for unsupervised feature learning
■ InfoGAN (Information Maximizing GAN)
■ BiGAN (Bidirectional Generative Adversarial Networks)
■ BigBiGAN (Big Bidirectional Generative Adversarial Networks)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 181
 InfoGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 182
 InfoGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 183
 InfoGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 184
 InfoGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 185
 InfoGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 186
 InfoGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 187
 InfoGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 188
 InfoGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 189
 InfoGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 190
 InfoGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 191
 InfoGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 192
Unsupervised Category Discovery - BigGAN

Slide: Jeff Donahue

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 193
Unsupervised Category Discovery - BigGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 194
 Big Bidirectional GAN

Slide: Jeff Donahue

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 195
BigBiGAN:Unconditional Image Generation

Slide: Jeff Donahue

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 196
BigBiGAN - Unconditional Image Generation

Slide: Jeff Donahue

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 197
 BigBiGAN

Slide: Jeff Donahue

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 198
 BigBiGAN

Slide: Jeff Donahue

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 199
 BigBiGAN

Slide: Jeff Donahue

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 200
 BigBiGAN

Slide: Jeff Donahue

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 201
 BigBiGAN: Representation Learning

Slide: Jeff Donahue

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 202
 BigBiGAN: Latent Space NNs

Slide: Jeff Donahue

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 203
 BigBiGAN Reconstructions

Slide: Jeff Donahue

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 204
 BigBiGAN Reconstructions

Slide: Jeff Donahue

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 205
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 206
 Energy Models
■ Assign energy E(x) to every possible x

■ Low energy makes for high probability x

■ High energy makes for low probability x
■ Practical challenge: domain of x usually very large
■ E.g. all possible 28x28x3 images = 256^(28*28*3) ~ 10^16464

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 207
 Energy Models

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 208
 Energy Models -- Definition
■ Assign energy E(x) to every possible x

■ Low energy makes for high probability x

■ High energy makes for low probability x
■ Practical challenge: domain of x usually very large
■ E.g. all possible 28x28x3 images = 256^(28*28*3) ~ 10^16464
→ Z impractical to compute
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 209
 Energy Models -- Maximum Likelihood

...

Can we just ignore Z and maximize first part of the objective?

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 210
Variational Lower Bound for log Z

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 211
Energy Models -- Maximum Likelihood

...

= entropy regularized W-GAN objective

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 212
 GANs and Energy Models

- Energy Model = W-GAN with Entropy Regularization

- Discriminator D = - energy E
- Generator G = q_phi
- Note: W-GAN also had regularizer (lifschitz) on D (= -E)
- Is Entropy easy to compute?
- Generally, no… but can be done for some models, worth investigating!
- E.g. q_phi auto-regressive or VAE or flow (why not just log-likelihood objective for those?)
- Could in principle also add entropy regularization to those models when training w/ log-likelihood

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 213
 GANs and Energy Models
Inspiring, clean comprehensive mathematical write-up
- John Schulman
Notes On GANs, Energy-Based Models, and Saddle Points
http://joschu.net/docs/gan-notes.pdf (2016)

Early papers:
- Taesup Kim and Yoshua Bengio
Deep Directed Generative Models with Energy-Based Probability Estimation
https://arxiv.org/abs/1606.03439
- Junbo Zhao, Michael Mathieu, Yann LeCun
Energy-based Generative Adversarial Network
https://arxiv.org/abs/1609.03126

Recent related papers:

- Adji B. Dieng, Francisco J. R. Ruiz, David M. Blei, Michalis K. Titsias
Prescribed Generative Adversarial Networks
https://arxiv.org/abs/1910.04302
- Aditya Grover, Manik Dhar, Stefano Ermon
Flow-GAN: Combining Maximum Likelihood and Adversarial Learning in Generative Models
https://arxiv.org/abs/1705.08868

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 214
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 215
 Recall: Wasserstein Distance
■ Earth Mover (EM) distance

■ W-GAN optimizes the dual

■ How about optimizing the primal directly?

→ Optimal Transport GANs

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 216
 GANs and Optimal Transport
■ Improving GANs using Optimal Transport
Tim Salimans, Han Zhang, Alec Radford, Dimitris Metaxas
ICLR 2018
https://arxiv.org/abs/1803.05573

■ Sinkhorn AutoDiff GAN: Learning Generative Models with Sinkhorn Divergences

Aude Genevay, Gabriel Peyre, Marco Cuturi
AISTATS 2018
https://arxiv.org/abs/1706.00292

■ Cramer GAN: The Cramer Distance as a Solution to Biased Wasserstein Gradients

Marc Bellemare, Ivo Danihelka, W Dabney, S Mohamed, B Laskhiminarayanan, S Hoyer, R Munos
https://arxiv.org/abs/1705.10743

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 217
 Implicit Likelihood Models

→ every training data point needs to have a closeby generated neighbor

- Implicit Maximum Likelihood Estimation

Ke Li, Jitendra Malik
https://arxiv.org/abs/1809.09087
- Diverse Image Synthesis from Semantic Layouts via Conditional IMLE
Ke Li, Tianhao Zhang, Jitendra Malik
https://arxiv.org/abs/1811.12373

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 218
 Moment Matching
■ Key idea: Match the moments of the data and model distributions to bring
them closer
■ Called the two-sample test in hypothesis testing

■ Not feasible to compute higher order moments in high dimensions

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 219
 Moment Matching
■ Kernel trick

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 220
 Moment Matching
■

■ Refer to Gretton et al 2007, 2012 - Mathematical Treatment

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 221
Generative Moment Matching Networks

Li, Swersky, Zemel

(2015)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 222
Generative Moment Matching Networks

Li, Swersky, Zemel

(2015)

Li, Swersky, Zemel

(2015)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 223
Generative Moment Matching Networks
■ Need a good kernel for the mean discrepancy measure

■ Not shown to scale well beyond MNIST and TFD (and some variants
on CIFAR 10 later) - needs autoencoding, large minibatch and
mixture of kernels with different bandwidths

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 224
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 225
Other Uses of Adversarial Loss -- Transfer
- Deep Domain Confusion: Maximizing for Domain Invariance
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, Trevor Darrell
https://arxiv.org/abs/1412.3474

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 226
Other Uses of Adversarial Loss -- Fairness

... ...

...

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 227
 Outline
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, BigGAN-Deep, StyleGAN, StyleGAN-v2, VIB-GAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport, Implicit Likelihood Models, Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 228
 GANs and Imitation Learning: GAIL
Imitation learning as a GAN problem:

- Discriminator tries to distinguish trajectories (s,a) from demonstrator vs. from learned
imitation policy pi
- Learned policy pi tries to make itself indistinguishable from demonstrator
- Note: matches Energy-Based Model GAN formulation

GAIL: Generative Adversarial Imitation Learning

Jonathan Ho, Stefano Ermon
NeurIPS 2016
https://arxiv.org/pdf/1606.03476.pdf

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 229
 GANs and Imitation Learning: GAIL

GAIL: Generative Adversarial Imitation Learning

Jonathan Ho, Stefano Ermon
NeurIPS 2016
https://arxiv.org/pdf/1606.03476.pdf

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 230
 GANs and Imitation Learning: GAIL

GAIL: Generative Adversarial Imitation Learning

Jonathan Ho, Stefano Ermon
NeurIPS 2016
https://arxiv.org/pdf/1606.03476.pdf

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 231
 GANs and Imitation Learning: VAIL
- Recall: Variational Discriminator Bottleneck GAN

Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, Sergey Levine
ICLR 2019
https://arxiv.org/pdf/1810.00821.pdf

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 232
 VAIL

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 233
 VAIL

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 234
 VAIL

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 235
 VAIL

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 236
 VAIL

VAIL [Merel et al 2017]

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 237
 VAIL

Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, Sergey Levine
ICLR 2019
https://arxiv.org/pdf/1810.00821.pdf

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 238
 Summary
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, StyleGAN
■ More GANs: BigGAN-Deep, StyleGAN-v2, VIB-GAN, LOGAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport
■ Implicit Likelihood Models
■ Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning
■ Scaling up: GigaGAN, StyleGAN-XL
■ GAN losses for improved quantized tokenization: VQGAN, ViT-VQGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 239
 StyleGAN-XL
StyleGAN struggles when scaling to large diverse datasets and higher
resolutions (ie ImageNet)
First to generate quality images at 1024x1024x scale

StyleGAN3 StyleGAN-XL
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 240
 StyleGAN-XL
● StyleGAN3 base
● Leverage Projected GAN ideas, which enabled more stable
training
● Design a progressive growing strategy
● Leverage classifier guidance for additional class conditioning
● Enables 3x parameters of StyleGAN3
● Matches prior SOTA

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 241
 StyleGAN-XL
Projected GAN

Projected GAN does not work well with StyleGAN

● Reducing latent space helps 512-64 -> enables lower FID and stable
training
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 242
 StyleGAN-XL
● Pretrained class embeddings
○ Extract mean of lowest
resolution feature per class from
an Efficientnet-lite0
● When growing, remove 2
layers, then add 7
○ Reduce aliasing
○ Discriminator fixed (upsampled
to match the max resolution)
○ Lower resolution layers are fixed

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 243
 StyleGAN-XL
● Pretrained class embeddings
○ Extract mean of lowest
resolution feature per class from
an Efficientnet-lite0
● When growing, remove 2
layers, then add 7
○ Reduce aliasing
○ Discriminator fixed (upsampled
to match the max resolution)
○ Lower resolution layers are fixed
● Combine multiple different feature
extractors in Projection GAN loss
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 244
 StyleGAN-XL

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 245
 StyleGAN-XL
ImageNet Synthesis

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 246
 GigaGAN
What:
● Scale GANs to 1B parameters
● Text to Image GAN with Lower
FID than Stable Diffusion
● Fast Generation (0.13s for 512px)
● Disentangled Prompt Mixing and
Interpolation
● High Res Upscaling

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 247
 GigaGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 248
 GigaGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 249
 GigaGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 250
 GigaGAN
What: How:
● Scale GANs to 1B parameters ● Sample-adaptive kernel
● Text to Image GAN with Lower selection
FID than Stable Diffusion ● Self-attention layers
● Fast Generation (0.13s for 512px) ● Multi-scale generator and
● Disentangled Prompt Mixing and discriminator
Interpolation ● Additional loss
● High Res Upscaling

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 251
 GigaGAN - Generator
Sample-adaptive kernel selection
Goal: increase model capacity
● Bank of N input features

● Style vector + affine layer

● Aggregated filter

● Then StyleGAN2 weight modulation conv

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 252
 GigaGAN - Generator
Attention Layers
● Directly adding attention layers results in training collapse
● Dot-product self attention is NOT lipschitz
● Use L2-distance instead of dot-product distance
● Querry and Key weights have to be tied (sufficient condition)

Standard dot product attention Use L2 distances instead

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 253
 GigaGAN - Generator
Text-Conditioning

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 254
 GigaGAN - Generator
Text-Conditioning

Used in cross attention

layers

Used in computation of the

style vector for adaptive
kernel convs and an
additional token in self
attention
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 255
 GigaGAN - Generator
Multi scale image generation
● Predict images a multiple
resolutions
● Images at each level

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 256
 GigaGAN - Generator
Summary
● StyleGANv2
Base architecture
● Text condition the style
vector
● Introduce adaptive kernels based on
● L2 Lipschitz attention layers
● Apply cross attention to the local text features
● Predict images a multiple resolutions
● Important to keep other design details from StyleGANv2
● Turn off style mixing and path regularization as in StyleGAN-XL
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 257
 GigaGAN - Discriminator
Multiscale feature extractor

Is a subset of

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 258
 GigaGAN - Discriminator
Multiscale feature extractor

Multiscale input / output Discriminator

: 4 layer 1x1 conv modulated by

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 259
 GigaGAN - Discriminator
Loss:

Original non saturating GAN loss

Matching-aware loss

Enforce discriminator to incorporate

conditioning on (a randomly
sampled caption from the dataset)
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 260
 GigaGAN - Discriminator
Loss:

Original non saturating GAN loss

Matching-aware loss

Enforce discriminator to incorporate

conditioning on (a randomly
sampled caption from the dataset)
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 261
 GigaGAN - Additional Losses

● CLIP contrastive loss

○ and are frozen CLIP
models
● Vision aided GAN:
○ Freeze CLIP image encoder
○ Extract intermediate features and learn a
simple discriminator
○ Text condition through modulation
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 262
 GigaGAN Experiments

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 263
 GigaGAN Experiments

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 264
 GigaGAN Experiments
● Disentangled latent space

● Start with: “a X on
tabletop”, and generate
style w
● Re compute style w and t
for second half with
prompt “a X with the
texture of Y on tabletop”

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 265
 GigaGAN Experiments
● Prompt
interpolation
● Fix noise
vector Z
● Change the
caption c

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 266
 GigaGAN Experiments
Failure
modes

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 267
 GigaGAN - Upsampler
Trains a text conditioned
superresolution model
Generator becomes an asymmetric
U-Net
● Input 64x64 image
● 3 downsampling blocks, 6
upsampling blocks
● Remove Vision-aided GAN loss
● Add LPIPS Perceptual Loss
● All other losses are the same

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 268
 Summary
■ Motivation & Definition of Implicit Models
■ Original GAN (Goodfellow et al, 2014)
■ Evaluation: Parzen, Inception, Frechet
■ Some Theory: Bayes-optimal Discriminator; Jensen-Shannon Divergence; Mode Collapse; Avoiding Saturation
■ GAN Progression:
■ DC GAN (Radford et al, 2016)
■ Improved Training of GANs (Salimans et al, 2016)
■ WGAN, WGAN-GP, Progressive GAN, SN-GAN, SAGAN
■ BigGAN, StyleGAN
■ More GANs: BigGAN-Deep, StyleGAN-v2, VIB-GAN, LOGAN
■ Creative Conditional GANs
■ GANs and Representations
■ GANs as Energy Models
■ GANs and Optimal Transport
■ Implicit Likelihood Models
■ Moment Matching
■ Other uses of Adversarial Loss: Transfer Learning, Fairness
■ GANs and Imitation Learning
■ Scaling up: GigaGAN, StyleGAN-XL
■ GAN losses for improved quantized tokenization: VQGAN, ViT-VQGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 269
 VQGAN
● Recall VQ-VAE
● VAE with a discrete
latent code
○ Initially assumed
uniform prior over z
○ After encoder is trained,
train an autoregressive
to learn the prior (ie.
pixel CNN)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 270
 VQGAN

● VQVAE has blurry smoothed out details

● GANs have been successful at achieving

crisp and realistic looking images

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 271
 VQGAN

3 main innovations

● Add perceptual loss (LPIPS)

● Add patch GAN loss
● Transformer to learn the
prior over latent codes

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 272
 VQGAN
Loss function:

Patch GAN
Adaptive weights:

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 273
 VQGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 274
 VQGAN
● Preserves high frequency details and
textures
● Enables higher compression ratios
● Less tokens in the latent space
makes is easier to learn a prior with
a transformer

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 275
 VQGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 276
 ViT-VQGAN

Improved image
quantization process
● Transformer encoder
/ decoder
● Adversarial loss
● Logit laplace loss
● Factorized Codebook

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 277
 ViT-VQGAN

● Use StyleGANv2 Discriminator instead of PatchGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 278
 ViT-VQGAN

● Use StyleGANv2 Discriminator instead of PatchGAN

● Logit-Laplace loss - consider modeling a pdf over (0, 1)

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 279
 ViT-VQGAN

● Use StyleGANv2 Discriminator instead of PatchGAN

● Logit-Laplace loss - consider modeling a pdf over (0, 1)

= 0, b=1 = 1, b=1
UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 280
 ViT-VQGAN
Factorized Codebook
● Vanilla VQVAE results in many codes being
rarely used or unused
● Introduce a linear projection from the
feature to the code index
● Apply l2 normalization on encoded latents
and codebook latents
○ Maps latents onto a sphere - improves
stability and reconstruction quality

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 281
 ViT-VQGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 282
 ViT-VQGAN

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 283
 ViT-VQGAN
● a better image quantizer with respect to both computational
efficiency and reconstruction quality
● faster quantizer quantizer can speed up stage 2 training
(where augmentations are applied)
● Better quantizer can reduce information loss, important for
image understanding

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 284
 Applications

UC Berkeley -- Spring 2024 -- Deep Unsupervised Learning -- Pieter Abbeel, Kevin Frans, Philipp Wu, Wilson Yan -- L5 Implicit Models / GANs 285