Séquence 12

“Les Variational Autoencoder (VAE),

ou comment jouer avec les espaces latents”

v2.07
 Course materials (pdf)

Practical work environment

https://fidle.cnrs.fr v3.0.9
Powered by CNRS CRIC, Corrected notebooks
and UGA DGDSI of Grenoble, thanks !

Videos (YouTube)

Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) (*) Procedure via Docket or pip
https://creativecommons.org/licenses/by-nc-nd/4.0/ Remember to get the latest version !
Questions and answers :
https://fidle.cnrs.fr/q2a

Accompanied by :
IA Support (dream) Team of IDRIS
Directed by :
Agathe, Baptiste et Yanis - UGA/DAPI
Thibaut, Kamel - IDRIS
 https://fidle.cnrs.fr/listeinfo
Fidle information list
New !
http://fidle.cnrs.fr/agoria
AI exchange list
[email protected]

(*) ESR is Enseignement Supérieur et Recherche, french universities and public academic research organizations
 https://listes.services.cnrs.fr/wws/info/devlog
List of ESR* « Software developers » group

https://listes.math.cnrs.fr/wws/info/calcul
List of ESR* « Calcul » group

(*) ESR is Enseignement Supérieur et Recherche, french universities and public academic research organizations
Previously on Fidle !

– Unsupervised learning is great !

– The world of convolutions is vast
– Don't forget 1D convolutions
– Keras 3 is good for you !
– PyTorch is good for you too ! :-)
– Multi-input models
– Multi-output models
– Inception models
– GPUS is good for you
– Think clean, structured programming !
– (and to document them!)
VAE network
➔ Concepts and architecture
➔ Gaussian / probabilistic projections
➔ Kullback-Leibler divergence
➔ Morphing in latent space
Applied example
➔ Deeper in programming !
➔ VAE using Functional API (MNIST)
➔ VAE using Model subclass (MNIST)
VAE network
➔ Concepts and architecture
➔ Gaussian / probabilistic projections
➔ Kullback-Leibler divergence
➔ Morphing in latent space
Applied example
➔ Deeper in programming !
➔ VAE using Functional API (MNIST)
➔ VAE using Model subclass (MNIST)
Concepts & architecture
We have seen that an autoencoder network is
trained to minimize a reconstruction error :

Input Encoder Latent space Decoder Output

10
Autoencoder

An autoencoders network have 2 parts :

inputs z

Encoder
z = encoder(inputs)

11
Autoencoder

An autoencoders network have 2 parts :

inputs z outputs

Decoder
outputs = decoder(z)

12
Autoencoder

An autoencoders network have 2 parts :

inputs z outputs

z = encoder(inputs)
ae = keras.Model(inputs, outputs)
outputs = decoder(z)
13
Autoencoder

An autoencoders network have 2 parts :

inputs z outputs

z = encoder(inputs)
ae = keras.Model(inputs, outputs)
outputs = decoder(z)

Keras functional API 14

Autoencoder
3
An autoencoders network have 2 parts :

inputs outputs

Latent space
z

15
Autoencoder
Region
Example of of the « 1 » Region
MNIST of the « 0 » Clusters
dataset appear, but
distribution many of them
in its latent are nested or
space : very spread out

How can
Only two we make our
dimensions network better
are represented separate the
in abscissa and different
ordinate clusters?

z=encoder(inputs) Region
of the « 6 »

See : notebook [AE3]

16
Autoencoder (AE)
z vector in latent space An autoencoder performs a direct
projection into the latent space and an
upsampling from this latent space.

Loss

Binary cross entropy

Measures the difference
between input and output
17
Variational Autoencoder (VAE) The stochastic approach
μ is a mean encourages the network
σ is a variance to generate relevant
z vector in latent space output throughout the
distribution space.
This implies that the
input data are
statistically distributed.

Diederik P Kingma; Welling, Max (2013). "Auto-Encoding Variational Bayes".

https://arxiv.org/abs/1312.6114
18
Variational Autoencoder (VAE)
Kullback-Leibler divergence*

KL divergence between two

(probability) distributions
measures how much they
diverge from each other
KL loss

Reconstruction loss Total loss

Binary cross entropy totalloss = k1.rloss + k2.klloss
Measures the The trick is to find a nice
difference between compromise between
input and output these two components of
the loss function. 19
(*) Special case for a standard normal distribution
Variational Autoencoder (VAE)
totalloss = k1.rloss + k2.klloss

20
Variational Autoencoder (VAE) Easy !

k1 is too high k2 is too high

Reconstruction totalloss = k1.rloss + k2.klloss Agglomeration
is privileged. is privileged.

21
Variational Autoencoder
Variational Autoencoder (VAE)(VAE) Nice !

VAE2, with : VAE2, with :

k1, k2 = [1, 5e-4] totalloss = k1.rloss + k2.klloss k1, k2 = [1, 1e-3]

22
Variational Autoencoder (VAE)

Tha’s great ! We can generate data

in profusion !!! 23
VAE network
➔ Concepts and architecture
➔ Gaussian / probabilistic projections
➔ Kullback-Leibler divergence
➔ Morphing in latent space
Applied example
➔ Deeper in programming !
➔ VAE using Functional API (MNIST)
➔ VAE using Model subclass (MNIST)
VAE network
➔ Concepts and architecture
➔ Gaussian / probabilistic projections
➔ Kullback-Leibler divergence
➔ Morphing in latent space
Applied example
➔ Deeper in programming !
➔ VAE using Functional API (MNIST)
➔ VAE using Model subclass (MNIST)
Our first VAE
Notebook : [VAE1/3]

Objectives :
implementating a VAE, using Keras 3 functional API
and model subclass, using real PyTorch !
Dataset :
MNIST

28
 VAE1 | VAE2 |VAE3
#1 Using MNIST
module
ImagesCallBack
module
20 ‘ on a CPU
Custom Layer BestModelCallBack
module With scale=1

MNIST

Import Retrieve the

and init dataset Train
START

END
Parameters Build
models Training
Objectives : latent_dim = 2
loss_weights = [1,.001] review
VAE using scale = .1

Keras seed = 123

SamplingLayer
functional API module

and custom VariationalLossLayer

module

layer 29
VAE1 | VAE2 |VAE3

The classical loss functions use only

the inputs and the outputs tensors.
The calculation of the loss is here
more complex and will be done via a
custom layer.
So we have two custom layers in this
example :

SamplingLayer

VariationalLossLayer

30
 VAE1 | VAE2 |VAE3
#1 Using MNIST ImagesCallBack
module
module
20 ‘ on a CPU
Custom Layer BestModelCallBack
module With scale=1

MNIST

Import Retrieve the

and init dataset Train
START

END
Parameters Build
models Training
Objectives : latent_dim = 2
loss_weights = [1,.001] review
VAE using scale = .1

Keras seed = 123

SamplingLayer
functional API module

and custom VariationalLossLayer

module

layer 31
 VAE1 | VAE2 |VAE3

#2 Using MNIST ImagesCallBack

module
module 20 ‘ on a CPU
Custom model BestModelCallBack With scale=1
module

MNIST

Import Retrieve the

and init dataset Train Model evaluation

START

END
Parameters Build Training
latent_dim = 2
model review
Objectives : loss_weights = [1,.001]

VAE using scale

seed
= .1
= 123
Keras subclass SamplingLayer
API and MNIST module

dataset VAE
32
module
VAE1 | VAE2 |VAE3

The encoder and decoder are always

built via the functional API and a custom
layer.

The model is defined by a custom class in

real Python :-)

The gradient descent and the the loss

function are defined via a method.

We have one custom layer and one

custom model class :

SamplingLayer
VAE

33
 VAE1 | VAE2 |VAE3

#2 Using MNIST ImagesCallBack

module
module 20 ‘ on a CPU
Custom model BestModelCallBack With scale=1
module

MNIST

Import Retrieve the

and init dataset Train Model evaluation

START

END
Parameters Build Training
latent_dim = 2
model review
Objectives : loss_weights = [1,.001]

VAE using scale

seed
= .1
= 123
Keras subclass SamplingLayer
API and MNIST module

dataset VAE
34
module
 VAE1 | VAE2 | VAE3

MNIST
module
Few seconds !

MNIST
Import
and init Retrieve the Images Generate fom
dataset reconstruction latent space
START

Reload END
Parameters Visualizing
scale = .1
model latent space
seed = 123
Objectives :
Reload a
saved model SamplingLayer
and visualize module

latent space VAE

35
module
Little things and concepts to keep in mind

– Unsupervised learning is great!

– The use cases of the VAE are very wide
– Anomaly detection
– Clustering
– Semi-supervised classification
– ...
– VAEs are often coupled with other tools
– Balancing the loss function is not always easy to find
– GPUs are great !
– Other (and better?) generative networks exist
– ...

36
 Jeudi

Next, on Fidle :
21
Mars
à 14h00
L’IA
comme
un outil

Réalisation, technique et mise en scène :

Agathe, Baptiste et Yanis - UGA/DAPI
Thibaut, Kamel, Léo - IDRIS 37