Copyright Notice

These slides are distributed under the Creative Commons License.

DeepLearning.AI makes these slides available for educational purposes. You may not use or
distribute these slides for commercial purposes. You may make copies of these slides and
use or distribute them for educational purposes as long as you cite DeepLearning.AI as the
source of the slides.

For the rest of the details of the license, see

https://creativecommons.org/licenses/by-sa/2.0/legalcode
 GAN
Improvements
deeplearning.ai
Outline
● How GANs have improved

● State of the art methods for improving GANs performance

GANs Over Time
Main Improvements: (1) Stability

High standard Low standard

deviation deviation

Use batch standard deviation to

encourage diversity
Main Improvements: (1) Stability : gradient

Improve stability by enforcing

1-Lipschitz continuity

E.g. WGAN-GP and Spectral

Normalization
Main Improvements: (1) Stability

Take average
Generator of the weights
Generator Generator
Generator
Generator
Generator

Generator at
different iterations
 Main Improvements: (1) Stability
No averaging

Exponential
averaging
Use moving average for
smoother results

Available from: https://arxiv.org/abs/1806.04498v2

 Main Improvements: (1) Stability

Progressive growing gradually trains

GAN at increasing resolutions

Available from: https://arxiv.org/abs/1710.10196

Main Improvements: (2) Capacity

Larger models can use

higher resolution images
 Main Improvements: (3) Diversity

Available from: https://github.com/NVlabs/stylegan

Summary
● GANs have improved because of:

○ Stability - longer training and better images

○ Capacity - larger models and higher resolution images

○ Diversity - increasing variety in generated images

 StyleGAN
Overview
deeplearning.ai
Outline
● StyleGAN achievements

● What styles are

● Introduction to StyleGAN architecture and components

StyleGAN Goals
1. Greater fidelity on high-resolution images
2. Increased diversity of outputs
3. More control over image features
 Greater Fidelity

Not fooling anyone I’m shook

(Left) Available from: https://arxiv.org/abs/1406.2661
(Right) Available from: https://github.com/NVlabs/stylegan
 Increased Diversity

Available from: https://arxiv.org/abs/1812.04948

Increased Diversity
 More Feature Control

Hair color/style →

← Glasses

Available from: https://arxiv.org/abs/1812.04948

 Style in GANs
Style = variation in an
image

Early styles are coarser like

face shape

Later styles are finer like

hair wisps

Available from: https://arxiv.org/abs/1812.04948

The Style-Based Generator Noise vector: z

Noise vector: z Mapping Random noise

Generator Network

Intermediate StyleGAN
noise: w Generator

Traditional architecture
StyleGAN architecture
 Progressive Growing

Available from: https://arxiv.org/abs/1710.10196

Summary
● StyleGAN’s goals:
○ Greater fidelity, increased diversity, improved control over features
● Style is any variation in the image
● Main components of StyleGAN:
○ Progressive growing
○ Noise mapping network
○ Adaptive instance normalization (AdaIN)
 Progressive
Growing
deeplearning.ai
Outline
● Progressive growing intuition and motivation

● How to implement it
 Progressive Growing
Latent Latent Latent
G 4x4 4x4 4x4
8x8

1024x1024

Reals Reals
... Reals
D 1024x1024

4x4
4x4 8x8 4x4
Training progress
Based on: https://arxiv.org/abs/1710.10196
 Progressive Growing in Action

Available from: https://www.gwern.net/images/gan/2019-03-16-stylegan-facestraining.mp4

 Progressive Growing: Generator
Conv 16x16

Upsample 2x

99% 1%
Conv 32x32

Based on: https://arxiv.org/abs/1710.10196

 Progressive Growing: Generator
Conv 16x16

Upsample 2x

50% 50%
Conv 32x32

Based on: https://arxiv.org/abs/1710.10196

 Progressive Growing: Generator
Conv 16x16

Upsample 2x

1% 99%
Conv 32x32

Based on: https://arxiv.org/abs/1710.10196

 Progressive Growing: Generator
Conv 16x16

Upsample 2x

Conv 32x32

Based on: https://arxiv.org/abs/1710.10196

 Progressive Growing: Discriminator

Downsample 0.5x

Conv 32x32

Conv 16x16

...
Real/Fake
Based on: https://arxiv.org/abs/1710.10196
 Progressive Growing: Discriminator

Downsample 0.5x

Conv 32x32

Conv 16x16

...
Real/Fake
Based on: https://arxiv.org/abs/1710.10196
 Progressive Growing in Context

4x4
8x8
16x16

...

1024x1024

Based on: https://arxiv.org/abs/1812.04948

 Progressive Growing in Context

4x4 Upsample

8x8 Convolution
16x16 Convolution

...

1024x1024

Based on: https://arxiv.org/abs/1812.04948

Summary
● Progressive growing gradually doubles image resolution

● Helps with faster, more stable training for higher resolutions

 Noise Mapping
Network
deeplearning.ai
Outline
● Noise mapping network structure

● Motivation behind the noise mapping network

● Where its output goes

 Noise Mapping Network

FC

Mapping Network
FC

FC

FC

FC

FC

FC

FC

Intermediate noise vector

Based on: https://arxiv.org/abs/1812.04948

Remember: Z-Space Entanglement
Glasses
Beard
Noise vector Output
Hair
features
Eyes
Age

Not possible to control single output features

W-Space: Less Entangled
Glasses
Intermediate Beard
noise vector Output
Hair
features
Eyes
Age

More possible to control single output features

 Mapping Network in Context

Based on: https://arxiv.org/abs/1812.04948

 Mapping Network in Context

Based on: https://arxiv.org/abs/1812.04948

Summary
● Noise mapping allows for a more disentangled noise space

● The intermediate noise vector is used as input to the generator

 Adaptive Instance
Normalization
(AdaIN)
deeplearning.ai
Outline
● Instance Normalization

● Adaptive Instance Normalization (AdaIN)

● Where and why AdaIN is used

 AdaIN in Context

Based on: https://arxiv.org/abs/1812.04948

 AdaIN in Context

Based on: https://arxiv.org/abs/1812.04948

 AdaIN
Upsample

Convolution

AdaIN

Convolution

AdaIN

Based on: https://arxiv.org/abs/1812.04948

 AdaIN
Upsample

Convolution

AdaIN

Convolution

AdaIN

Step 1: Normalize convolution outputs

Based on: https://arxiv.org/abs/1812.04948

 AdaIN
Upsample

Convolution

AdaIN

Convolution

AdaIN

Step 1: Normalize convolution outputs using Instance Normalization

Based on: https://arxiv.org/abs/1812.04948

 AdaIN
Batch norm
Upsample

Convolution

AdaIN

Convolution

AdaIN

Step 1: Normalize convolution outputs using Instance Normalization

(Left) Available from: https://medium.com/syncedreview/facebook-ai-proposes-group-normalization-alternative-to-batch-normalization-fb0699bffae7

(Right) Based on: https://arxiv.org/abs/1812.04948
 AdaIN
Batch norm Instance norm
Upsample

Convolution

AdaIN

Convolution

AdaIN

Step 1: Normalize convolution outputs using Instance Normalization

(Left) Available from: https://medium.com/syncedreview/facebook-ai-proposes-group-normalization-alternative-to-batch-normalization-fb0699bffae7

(Right) Based on: https://arxiv.org/abs/1812.04948
 AdaIN
Batch norm Instance norm
Upsample

Convolution

AdaIN

Convolution

AdaIN

Step 1: Normalize convolution outputs using Instance Normalization

(Left) Available from: https://medium.com/syncedreview/facebook-ai-proposes-group-normalization-alternative-to-batch-normalization-fb0699bffae7

(Right) Based on: https://arxiv.org/abs/1812.04948
 AdaIN
Upsample

Convolution

AdaIN

Convolution

AdaIN

Step 2: Apply adaptive styles

Based on: https://arxiv.org/abs/1812.04948

 AdaIN
Upsample

Convolution
Mapping
AdaIN
Network
Convolution

AdaIN

Step 2: Apply adaptive styles using the intermediate noise vector

Based on: https://arxiv.org/abs/1812.04948

 AdaIN
Upsample

Convolution

AdaIN

Convolution

AdaIN

Step 2: Apply adaptive styles using the intermediate noise vector

Based on: https://arxiv.org/abs/1812.04948

 AdaIN
Upsample

Convolution

AdaIN

Convolution

AdaIN

Step 2: Apply adaptive styles using the intermediate noise vector

Based on: https://arxiv.org/abs/1812.04948

 AdaIN
Upsample

Convolution

AdaIN

Convolution
FC FC
AdaIN

Step 2: Apply adaptive styles using the intermediate noise vector

Based on: https://arxiv.org/abs/1812.04948

 AdaIN
Upsample

Convolution

AdaIN

Convolution
FC FC
AdaIN

Step 2: Apply adaptive styles using the intermediate noise vector

Based on: https://arxiv.org/abs/1812.04948

 AdaIN
Upsample

Convolution

AdaIN

Convolution

AdaIN

Step 2: Apply adaptive styles using the intermediate noise vector

Based on: https://arxiv.org/abs/1812.04948

AdaIN

Step 1: Instance normalization

AdaIN

Step 2: Adaptive styles

 AdaIN in Context

Based on: https://arxiv.org/abs/1812.04948

Summary
● AdaIN transfers style information onto the generated image from the
intermediate noise vector

● Instance Normalization is used to normalize individual examples before

apply style statistics from
 Style Mixing &
Stochastic
Noise
deeplearning.ai
Outline
● Controlling coarse and fine styles with StyleGAN

● Style mixing for increased diversity during training/inference

● Stochastic noise for additional variation

Style Mixing

Tabby
Cat

Tuxedo
Cat
Style Mixing

Tabby
Cat

Tuxedo
Cat
 Style Mixing in Context

Based on: https://arxiv.org/abs/1812.04948

 Style Mixing in Context

Based on: https://arxiv.org/abs/1812.04948

 Style Mixing in Context

Based on: https://arxiv.org/abs/1812.04948

 Style Mixing in Context

Based on: https://arxiv.org/abs/1812.04948

 Style Mixing

Source B
Source A

Coarse styles from B

Middle styles from B

Fine styles from B

Available from: https://arxiv.org/abs/1812.04948

 Stochastic Variation
Fine
layers

Coarse
layers

Available from: https://arxiv.org/abs/1812.04948

 Stochastic Noise in Context

Based on: https://arxiv.org/abs/1812.04948

 Stochastic Noise in Context

1. Sample noise from

Normal distribution

Based on: https://arxiv.org/abs/1812.04948

 Stochastic Noise in Context

1. Sample noise from

Normal distribution
2. Concatenate noise to x,
before AdaIN
Based on: https://arxiv.org/abs/1812.04948
 Stochastic Variation
Small details: hair strands,
wrinkles, etc.

Different extra noise values

create stochastic variation

Available from: https://arxiv.org/abs/1812.04948

Summary
● Style mixing increases diversity that the model sees during training

● Stochastic noise causes small variations to output

● Coarse or fineness depends where in the network style or noise is added

○ Earlier for coarser variation
○ Later for finer variation
 Putting It All
Together
deeplearning.ai
Outline
● Putting all the StyleGAN components together!
 StyleGAN Architecture: Progressive Growing

Based on: https://arxiv.org/abs/1812.04948

 StyleGAN Architecture: Noise Mapping Network

Based on: https://arxiv.org/abs/1812.04948

 StyleGAN Architecture: AdaIN

Based on: https://arxiv.org/abs/1812.04948

 StyleGAN Architecture: Style Mixing

Based on: https://arxiv.org/abs/1812.04948

 StyleGAN Architecture: Stochastic Noise

Based on: https://arxiv.org/abs/1812.04948

 StyleGAN Architecture: That’s a Wrap!

Based on: https://arxiv.org/abs/1812.04948

Summary
● Main components of StyleGAN:
○ Progressive Growing
○ Noise Mapping Network
○ AdaIN
○ Style Mixing
○ Stochastic Noise