Copyright Notice

These slides are distributed under the Creative Commons License.

DeepLearning.AI makes these slides available for educational purposes. You may not use or
distribute these slides for commercial purposes. You may make copies of these slides and
use or distribute them for educational purposes as long as you cite DeepLearning.AI as the
source of the slides.

For the rest of the details of the license, see

https://creativecommons.org/licenses/by-sa/2.0/legalcode
 Unpaired
Image-to-Image
Translation
deeplearning.ai
Outline
● Paired vs. unpaired image-to-image translation
● Unpaired image-to-image translation
○ Mapping between two piles of image styles
○ Finding commonalities and differences
 Image-to-Image Translation

Edges to photo

Paired images

Available from: https://arxiv.org/abs/1611.07004

 Image-to-Image Translation

Edges to photo Monet to photo

Paired images Unpaired images

Available from: https://arxiv.org/abs/1611.07004

 Image-to-Image Translation

Paired
images

Available from: https://arxiv.org/abs/1703.10593

 Image-to-Image Translation

Paired Unpaired
images images

Available from: https://arxiv.org/abs/1703.10593

 Unpaired Image-to-Image Translation

Available from: https://arxiv.org/abs/1703.10593

 Mapping Between Two Piles

(Center) Images available from: https://arxiv.org/abs/1703.10593

(Side) Images available from: https://github.com/togheppi/CycleGAN
 Mapping Between Two Piles

(Center) Images available from: https://arxiv.org/abs/1703.10593

(Sides) Images available from: https://github.com/togheppi/CycleGAN
 Mapping Between Two Piles
Content = common elements
Style = unique elements

(Center) Images available from: https://arxiv.org/abs/1703.10593

(Sides) Images available from: https://github.com/togheppi/CycleGAN
Summary

● Unpaired image-to-image translation:

○ Learns a mapping between two piles of images
○ Examines common elements of the two piles (content) and unique
elements of each pile (style)

● Unlike paired image-to-image translation,

this method is unsupervised
 CycleGAN
Overview
deeplearning.ai
Outline
● Overview of CycleGAN
○ The “Cycle” in CycleGAN
○ Two GANs!
 Cycle Consistency

Images available from: https://github.com/togheppi/CycleGAN

 Cycle Consistency

Real

Fake

Images available from: https://github.com/togheppi/CycleGAN

 Cycle Consistency

Real

Fake

Fake

Images available from: https://github.com/togheppi/CycleGAN

 Cycle Consistency

Real

Should be the Fake

same

Fake

Images available from: https://github.com/togheppi/CycleGAN

 Cycle Consistency

Cycle consistency

Images available from: https://github.com/togheppi/CycleGAN

 Two GANs

GAN Z → H

GAN H → Z

Images available from: https://github.com/togheppi/CycleGAN

 Two GANs
Real
GAN Z → H
Fake

GAN H → Z

Fake

Images available from: https://github.com/togheppi/CycleGAN

 Two GANs
Real Real
GAN Z → H GAN H→ Z
Fake Fake

GAN H → Z GAN Z → H

Fake Fake

Images available from: https://github.com/togheppi/CycleGAN

 CycleGAN

Discriminator:
PatchGAN

0.8

Image available from: https://github.com/togheppi/CycleGAN

 CycleGAN

Generator ≈
U-Net

Available from: https://arxiv.org/abs/1611.07004

 CycleGAN

Generator ≈
U-Net + DCGAN
generator

Available from: https://arxiv.org/abs/1611.07004

 CycleGAN

Generator ≈
U-Net + DCGAN
generator

Additional skip connections

Available from: https://arxiv.org/abs/1611.07004

Summary

● CycleGAN uses two GANs for unpaired image-to-image translation

● The discriminators are PatchGAN’s

● The generators are similar to a U-Net and DCGAN generator with

additional skip connections
 CycleGAN: Two
GANs
deeplearning.ai
Outline
● Two GANs, four components
○ Two generators
○ Two discriminators
 CycleGAN Components

GAN Z → H

Images available from: https://github.com/togheppi/CycleGAN

 CycleGAN Components

GAN Z → H

GAN H → Z

Images available from: https://github.com/togheppi/CycleGAN

 CycleGAN Components
Generator Z → H
Discriminator H

GAN Z → H

GAN H → Z

Images available from: https://github.com/togheppi/CycleGAN

 CycleGAN Components
Generator Z → H
Discriminator H

GAN Z → H

GAN H → Z

Generator H → Z
Discriminator Z
Images available from: https://github.com/togheppi/CycleGAN
 GAN Z → H
Real Fake

Generator
Z→H

Images available from: https://github.com/togheppi/CycleGAN

 GAN Z → H
Real Fake

Generator Discriminator
Z→H H

Real Discriminator
H

Images available from: https://github.com/togheppi/CycleGAN

 GAN Z → H
Classification
Real Fake matrix

Generator Discriminator
Z→H H

Classification
matrix

Real Discriminator
H

Images available from: https://github.com/togheppi/CycleGAN

 GAN H → Z
Real Fake

Generator
H→Z

Images available from: https://github.com/togheppi/CycleGAN

 GAN H → Z
Classification
Real Fake matrix

Generator Discriminator
H→Z Z

Classification
matrix

Real Discriminator
Z

Images available from: https://github.com/togheppi/CycleGAN

Summary

● CycleGAN has four components:

○ Two generators
○ Two discriminators

● The inputs to the generators and discriminators

are similar to Pix2Pix, except:
○ There are no real target outputs
○ Each discriminator is in charge of one pile of images
 CycleGAN:
Cycle
Consistency
deeplearning.ai
Outline
● Encouraging cycle consistency
○ Cycle Consistency Loss term
● Loss with cycle consistency for each of two GANs
● How cycle consistency helps
 Cycle Consistency Loss
Real
Generator Fake
Z→H

Images available from: https://github.com/togheppi/CycleGAN

 Cycle Consistency Loss
Real
Generator Fake
Z→H

Generator
H→Z

Fake
Images available from: https://github.com/togheppi/CycleGAN
 Cycle Consistency Loss
Real
Generator Fake
Z→H
Take pixel
difference for
Z → H→ Z

Generator
H→Z

Fake
Images available from: https://github.com/togheppi/CycleGAN
 Cycle Consistency Loss
Real
Generator Fake
H→Z
Take pixel
difference for
H→ Z → H

Generator
Z→H

Fake
Images available from: https://github.com/togheppi/CycleGAN
 Cycle Consistency Loss

+
Z→H→Z H→Z→H

Cycle Consistency Loss is the

sum of both directions
Images available from: https://github.com/togheppi/CycleGAN
 Cycle Consistency Loss

+
Adversarial Loss +
Z→H→Z H→Z→H

Images available from: https://github.com/togheppi/CycleGAN

Cycle Consistency Loss

Adversarial Loss + Cycle Consistency Loss

Cycle Consistency Loss

Adversarial Loss + λ * Cycle Consistency Loss

 Ablation Studies
Input Ground truth Cycle alone

Available from: https://arxiv.org/abs/1703.10593

 Ablation Studies
Input Ground truth Cycle alone

Without Adversarial
GAN Loss, outputs are
not realistic

Available from: https://arxiv.org/abs/1703.10593

 Ablation Studies
Input Ground truth GANs alone

Available from: https://arxiv.org/abs/1703.10593

 Ablation Studies
Input Ground truth GANs alone

Without Cycle Consistency

Loss, outputs show signs of
mode collapse

Available from: https://arxiv.org/abs/1703.10593

 Ablation Studies
Input Ground truth GANs + 1-way cycle

Available from: https://arxiv.org/abs/1703.10593

 Ablation Studies
Input Ground truth GANs + 1-way cycle

Without full Cycle

Consistency Loss, outputs
see mode collapse too

Available from: https://arxiv.org/abs/1703.10593

 Ablation Studies
Input Ground truth CycleGAN

Available from: https://arxiv.org/abs/1703.10593

 Ablation Studies
Input Ground truth CycleGAN

CycleGAN uses both

Adversarial Loss and
Cycle Consistency Loss

Available from: https://arxiv.org/abs/1703.10593

Summary

● Cycle consistency helps transfer uncommon style elements between

the two GANs, while maintaining common content
● Add an extra loss term to each generator to softly encourage cycle
consistency
● Cycle consistency is used in both directions
 CycleGAN:
Least Squares
Loss
deeplearning.ai
Outline
● Least squares in statistics
● Least Squares Loss in GANs
○ Discriminator
○ Generator
Least Squares Loss: Another GAN Loss Function
● Came out when training stability was a big problem in GANS
○ Similar time to WGAN-GP
Least Squares Loss: Another GAN Loss Function
● Came out when training stability was a big problem in GANS
○ Similar time to WGAN-GP

● Helps with vanishing gradients and mode collapse

Least Squares Loss: Another GAN Loss Function
● Came out when training stability was a big problem in GANS
○ Similar time to WGAN-GP

● Helps with vanishing gradients and mode collapse

● GAN loss functions are chosen empirically

Least Squares
6

4 A B
3

1
C

0 1 2 3 4 5 6 7 8 9 10 11
Least Squares
6

4 A B
3

1
C

0 1 2 3 4 5 6 7 8 9 10 11
Least Squares
6

4 A B
3
1
2 2 4

1
C

0 1 2 3 4 5 6 7 8 9 10 11
Least Squares
6

4 A B
3
1 1
2 2² = 4 2 4² = 16 4

1
C

0 1 2 3 4 5 6 7 8 9 10 11
Least Squares Sum of squares is 4 + 1 + 16 = 21
6

4 A B
3
1 1
2 2² = 4 2 4² = 16 4

1
C

0 1 2 3 4 5 6 7 8 9 10 11
Least Squares Sum of squares is 4 + 1 + 16 = 21
6

4 A B
Minimize
3
sum of squares 1 1
2 2² = 4 2 4² = 16 4

1
C

0 1 2 3 4 5 6 7 8 9 10 11
Least Squares Loss: Discriminator

Discriminator classification
of real image x
Least Squares Loss: Discriminator
Least Squares Loss: Discriminator

Discriminator classification
of fake image G(z)
Least Squares Loss: Discriminator
Least Squares Loss: Discriminator
Least Squares Loss: Generator
Least Squares Loss

Discriminator
Loss
Least Squares Loss

Discriminator
Loss

Generator
Loss
Least Squares Loss

Discriminator
Loss

Generator
Loss

Reduces vanishing gradient problem

Least Squares Loss

Discriminator
Loss

Generator
Loss

Also known as Mean Squared Error!

Context of Least Squares Loss

Adversarial Loss + λ * Cycle Consistency Loss

Least Squares Loss

Summary

● Least squares fits a line from several points

● Least Squares Loss is used as the Adversarial Loss function in CycleGAN

● More stable than BCELoss, since the gradient is only flat when
prediction is exactly correct
 CycleGAN:
Identity Loss
deeplearning.ai
Outline
● Identity Loss
○ How it works
○ Impact on outputs
 Identity Loss
Real input Ideal output

Generator
Z→H
“Opposite”
generator

Images available from: https://github.com/togheppi/CycleGAN

 Identity Loss
Real input Ideal output

Generator
Z→H
“Opposite”
generator

Pixel distance

Images available from: https://github.com/togheppi/CycleGAN

 Identity Loss
Real input Ideal output

Generator
Z→H
“Opposite”
generator

This ideal example has an

Pixel distance Identity Loss of zero
Images available from: https://github.com/togheppi/CycleGAN
 Identity Loss
Real input Discouraged output

Generator
Z→H
“Opposite”
generator

Discourage mapping Z → H
Pixel distance to distort colors
Images available from: https://github.com/togheppi/CycleGAN
Context of Identity Loss

Adversarial Loss + λ * Cycle Consistency Loss

 Context of Identity Loss

Adversarial Loss + λ * Cycle Consistency Loss

+ Generator
Z→H + Generator
H→Z

Pixel distance Pixel distance

Images available from: https://github.com/togheppi/CycleGAN

Context of Identity Loss

Adversarial Loss + λ * Cycle Consistency Loss

+ Identity Loss
Context of Identity Loss

Adversarial Loss + λ * Cycle Consistency Loss

1

+ λ * Identity Loss
2
 Identity Loss Example: Photo → Monet
Input No Identity Loss With Identity Loss

Identity Loss helps preserve

original photo color

Available from: https://arxiv.org/abs/1703.10593

Summary
● Identity Loss takes a real image in domain B and inputs it into Generator:
A → B, expecting an identity mapping
○ An identity mapping means the output is the same as the input

● Pixel distance is used

○ Ideally, no difference between input and output!

● Identity Loss is optionally added to help

with color preservation
 CycleGAN:
Putting It All
Together
deeplearning.ai
Outline
● Putting CycleGAN together!
○ Two GANs
○ Cycle Consistency Loss
○ Least Squares Adversarial Loss
○ Identity Loss (optional)
 CycleGAN
Real Fake

Generator
Z→H

Images available from: https://github.com/togheppi/CycleGAN

 CycleGAN
Real Fake Classification matrix

Generator Discriminator
Z→H H

Discriminator
H

Classification matrix

Real

Images available from: https://github.com/togheppi/CycleGAN

 CycleGAN
Real Fake Classification matrix

Generator Discriminator
Z→H H

Least Squares
Loss

Discriminator
H

Classification matrix

Real

Images available from: https://github.com/togheppi/CycleGAN

 CycleGAN
Real Fake Classification matrix

Generator Discriminator
Z→H H

Least Squares
Loss

Discriminator
H
Generator
H→Z
Classification matrix

Fake Real

Images available from: https://github.com/togheppi/CycleGAN

 CycleGAN
Real Fake Classification matrix

Generator Discriminator
Z→H H

Cycle Least Squares

Consistency Loss
Loss
Discriminator
H
Generator
H→Z
Classification matrix

Fake Real

Images available from: https://github.com/togheppi/CycleGAN

 CycleGAN
Real Fake Classification matrix

Generator Discriminator
H→Z Z

Cycle Least Squares

Consistency Loss
Loss
Discriminator
Z
Generator
Z→H
Classification matrix

Fake Real

Images available from: https://github.com/togheppi/CycleGAN

 CycleGAN
Real Fake

Generator
H→Z

Identity Loss

Images available from: https://github.com/togheppi/CycleGAN

 CycleGAN
Real Fake

Generator
Z→H

Identity Loss

Images available from: https://github.com/togheppi/CycleGAN

CycleGAN Loss

Least Squares Cycle Consistency

Generator Loss = Adversarial Loss
H→ Z
+ Loss
H→Z→H
+ Identity Loss
H→Z

Least Squares Cycle Consistency

+ Adversarial Loss
Z→H
+ Loss
Z→H→Z
+ Identity Loss
Z→H
Summary

● CycleGAN is composed of two GANs

● Generators have 6 loss terms in total, 3 each:

○ Least Squares Adversarial Loss
○ Cycle Consistency Loss
○ Identity Loss

● Discriminator is simpler, with BCELoss using PatchGAN

 CycleGAN
Applications &
Variants
deeplearning.ai
Outline
● Overview of some CycleGAN applications
● Some variants of unpaired image-to-image translation
 Applications

Available from: https://arxiv.org/abs/1611.07004

 Applications

Available from: https://arxiv.org/abs/1805.08841

 Applications

CT Expert CycleGAN Standard

Augmentation

Available from: https://www.nature.com/articles/s41598-019-52737-x.pdf

 Variant: UNIT

Available from: https://github.com/mingyuliutw/UNIT

 Variant: Multimodal UNIT (MUNIT)

Available from: https://github.com/NVlabs/MUNIT

 Variant: Multimodal UNIT (MUNIT)

Available from: https://github.com/NVlabs/MUNIT

Summary

● Various applications of CycleGAN including:

○ Democratized art and style transfer
○ Medical data augmentation
○ Creating paired data

● UNIT and MUNIT are other models for unpaired

(unsupervised) image-to-image translation