CAP6412

Advanced Computer Vision

Mubarak Shah
[email protected]
HEC-245
Lecture-4: Diffusion Models

1/23/2023 CAP6412 - Lecture 1 Introduction 1

 Diffusion models in vision: A survey
https://arxiv.org/pdf/2209.04747.pdf

Alin Croitoru Vlad Hondru Radu Tudor Ionescu Mubarak Shah

University of Bucharest, University of Bucharest, University of Bucharest, University of Central
Romania Romania Romania Florida, US
[email protected] [email protected] [email protected] [email protected]
Agenda

1. Motivation
2. High-level overview
3. Denoising diffusion probabilistic models
4. Noise Conditioned Score Network
5. Conditional Generation
6. Stochastic Differential Equations
7. Applications
8. Research directions
Outline

1. Motivation
2. High-level overview
3. Denoising diffusion probabilistic models
4. Noise Conditioned Score Network
5. Conditional Generation
6. Stochastic Differential Equations
7. Applicatons
8. Research directions
Motivation
Motivation
 Motivation

A hedgehog using a A corgi wearing a red A transparent sculpture- of

calculator. bowtie and a purple a duck made out of glass.
party hat.

A photo of a Corgi dog riding a Pomeranian king Zebras roaming

bike in Times Square. It is wearing with tiger soldiers. in the field.
sunglasses and a beach hat.
Outline

1. Motivation
2. High-level overview
3. Denoising diffusion probabilistic models
4. Noise Conditioned Score Network
5. Conditional Generation
6. Stochastic Differential Equations
7. Research directions
High-level overview
• Diffusion models are probabilistic models used for image generation
• They involve reversing the process of gradually degrading the data
• Consist of two processes:
 The forward process: data is progressively destroyed by adding noise across
multiple time steps
 The reverse process: using a neural network, noise is sequentially removed
to obtain the original data

Standard Gaussian
Data distribution

reverse

forward
High-level overview

• Three categories:

 Denoising Diffusion Probabilistic Models (DDPM)

 Noise Conditioned Score Networks (NCSN)

 Stochastic Differential Equations (SDE)

Outline

1. Motivation
2. High-level overview
3. Denoising diffusion probabilistic models
4. Noise Conditioned Score Network
5. Conditional Generation
6. Stochastic Differential Equations
7. Research directions
Notations
𝑝 𝑥 - data distribution

𝒩 𝑥; 𝜇, 𝜎 ⋅ 𝐼 - Gaussian distribution

Random Variable (image) Mean Vector Covariance matrix. 𝐼 is the identity matrix

𝑥= 𝜇+ 𝜎 ⋅ 𝑧, 𝑧~𝒩(0, 𝐼)

Sample from this distribution

Denoising Diffusion Probabilistic Models (DDPMs)

Forward process
𝑥 𝑥

… …

𝑥 ~𝑝(𝑥 ) 𝑥 ~𝒩(0, 𝐼)
Denoising Diffusion Probabilistic Models (DDPMs)

𝑥 𝑥

… …

𝑥 ~𝑝(𝑥 ) Reverse process 𝑥 ~𝒩(0, 𝐼)

 Denoising Diffusion Probabilistic Models (DDPMs)

Forward process (Iterative) The image is

replaced with
noise
𝑥 ~𝑝 𝑥 𝑥 = 𝒩(𝑥 ; 1 − 𝛽 𝑥 , 𝛽 I) 𝛽 ≪ 1 , 𝑡 = 1, 𝑇

… …
𝑥 𝑥 𝑥 𝑥
 Denoising Diffusion Probabilistic Models (DDPMs)

Forward process. Ancestral sampling (One Shot) Notations:

𝛽 = 𝛼
𝑥 ~𝑝 𝑥 𝑥 = 𝒩(𝑥 ; 𝛽 ⋅ 𝑥 , 1 − 𝛽 I) 𝛼 =1 − 𝛽

… …
𝑥 𝑥 𝑥 𝑥
 DDPMs. Properties of

1. 𝛽 ≪ 1 , 𝑡 = 1, 𝑇
𝑥 ~𝑝 𝑥 𝑥 = 𝒩(𝑥 ; 1 − 𝛽 𝑥 , 𝛽 I)

𝑥 is created with a small step modeled by 𝛽

𝑡−1 𝑡

𝑥 comes from region close to 𝑥 ,

therefore we can model with Gaussian

𝑥 ~𝑝 𝑥 𝑥 = 𝒩(𝑥 ; 𝜇 𝑥 ,𝑡 ,Σ 𝑥 ,𝑡 )
DDPMs. Properties of

1. 𝛽 ≪ 1 , 𝑡 = 1, 𝑇

𝑥 ~𝑝 𝑥 𝑥 = 𝒩(𝑥 ; 1 − 𝛽 𝑥 , 𝛽 I)

𝑡−1 𝑡

?
Less certain where was the 𝑥 , because we could have
reached 𝑥 from many more regions.
 DDPMs. Properties of

1. 𝛽 ≪ 1, 𝑡 = 1, 𝑇 ⟹ 2. 𝑇 𝑖𝑠 𝑙𝑎𝑟𝑔𝑒
𝑥 𝑖𝑠 𝑝𝑢𝑟𝑒 𝑛𝑜𝑖𝑠𝑒

𝑥 𝑥

𝑇 𝑖𝑡𝑒𝑟𝑎𝑡𝑖𝑜𝑛𝑠
 DDPMs. Training objective
Remember that:

𝑥 𝑥 𝑥 𝑥

… …
𝑝 𝑥 𝑥 ≈𝑝 𝑥 𝑥 = 𝒩(𝑥 ;𝜇 𝑥 ,𝑡 ,Σ 𝑥 ,𝑡 )
Reverse process

Neural network Approximated by

weights a neural network
 DDPMs. Training objective
Simplification:

𝑥 𝑥 𝑥 𝑥

… …
𝑝 𝑥 𝑥 ≈𝑝 𝑥 𝑥 = 𝒩(𝑥 ; 𝜇 𝑥 , 𝑡 , 𝜎 I)
Reverse process

Neural network Approximated by

Fix the variance instead of learning, and predict/learn the mean weights a neural network
DDPMs. Training objective
UNet-like neural network

𝜇 (𝑥 , 𝑡)

~𝒩 𝑥 , 𝜇 (𝑥 , 𝑡), 𝜎 I

𝑥
U-Net
U-Net
U-Net
Slide from:
Denoising Diffusion-based Generative Modeling:
Foundations and Applications
Karsten Kreis Ruiqi Gao Arash Vahdat
DDPMs. Training Objective
02 Attention and Transformers

Cross Entropy and KL (Kullback-Leibler) divergence

• Entropy: E(P) = - ΣiP(i)logP(i)

• Cross Entropy: C(P) = - ΣiP(i) log Q(i)
• KL divergence: DKL(P || Q) = ΣiP(i)log[P(i)/Q(i)] = ΣiP(i)[logP(i) – logQ(i)]

Slides from Ming Li, University of Waterloo, CS 886 Deep Learning and NLP
DDPMs. Training Objective

min 𝔼 ~ ( ) − log 𝑝 𝑥 𝑥 + 𝐾𝐿(𝑝(𝑥 |𝑥 )||𝑝 𝑥 ) + 𝐾𝐿(𝑝 𝑥 𝑥 , 𝑥 ||𝑝 𝑥 𝑥 )

At each time step t, 𝑝 𝑥 𝑥 is as close

This term can be ignored because 𝑝 𝑥 is 𝒩 0, Ι as possible to the true posterior of the
and does not depend on 𝜃. forward process when conditioned on the
original image.
DDPMs. Training Objective. Simplifications
min 𝔼 ~ ( ) − log 𝑝 𝑥 𝑥 + 𝐾𝐿(𝑝 𝑥 𝑥 , 𝑥 ||𝑝 𝑥 𝑥 )

• The KL divergences of 2 gaussians is L2 distance between their means

• The first term measures the reconstruction error and can be addressed
with an independent decoder.

• DDPMs paper introduced two simplifications that led to a much simple

objective that is based on the noise in the image.
 DDPMs. Training Objective. Simplifications
min 𝔼 ~ ( ) − log 𝑝 𝑥 𝑥 + 𝐾𝐿(𝑝 𝑥 𝑥 , 𝑥 ||𝑝 𝑥 𝑥 )

Notations:
𝑝 𝑥 𝑥 ,𝑥 = 𝒩 𝑥 ; 𝜇 𝑥 ,𝑥 ,𝛽 𝐼
𝛽 = 𝛼
1 1 − 𝛼
𝜇 𝑥 ,𝑥 = 𝑥 − 𝑧 , 𝑧 ~𝒩(0, I) 𝛼 =1 − 𝛽
𝛼
1−𝛽
1−𝛽
𝛽 = ⋅𝛽
1−𝛽
 DDPMs. Training Objective. Simplifications
min 𝔼 ~ ( ) − log 𝑝 𝑥 𝑥 + 𝐾𝐿(𝑝 𝑥 𝑥 , 𝑥 ||𝑝 𝑥 𝑥 )

Tractable posterior: Notations:

𝑝 𝑥 𝑥 ,𝑥 = 𝒩 𝑥 ; 𝜇 𝑥 ,𝑥 ,𝛽 𝐼 𝛽 = 𝛼

𝛼 =1 − 𝛽
1 1 − 𝛼
𝜇 𝑥 ,𝑥 = 𝑥 − 𝑧 , 𝑧 ~𝒩(0, I) 1−𝛽
𝛼 𝛽 = ⋅𝛽
1−𝛽 1−𝛽
1 1 − 𝛼 ⟹
𝜇 𝑥 ,𝑥 = 𝑥 − 𝑧 (𝑥 , 𝑡)
𝛼
1−𝛽

𝛽
⟹ 𝐾𝐿(𝑝 𝑥 𝑥 , 𝑥 |𝑝 𝑥 𝑥 = 𝔼 ~𝒩( , ) 𝑧 − 𝑧 (𝑥 , 𝑡)
2𝜎 𝛼 (1 − 𝛽 )
 DDPMs. Training Objective. Simplifications
min 𝔼 ~ ( ) − log 𝑝 𝑥 𝑥 + 𝐾𝐿(𝑝 𝑥 𝑥 , 𝑥 ||𝑝 𝑥 𝑥 )

Tractable posterior: Notations:

𝑝 𝑥 𝑥 ,𝑥 = 𝒩 𝑥 ; 𝜇 𝑥 ,𝑥 ,𝛽 𝐼 𝛽 = 𝛼

𝛼 =1 − 𝛽
1 1 − 𝛼
𝜇 𝑥 ,𝑥 = 𝑥 − 𝑧 , 𝑧 ~𝒩(0, I) 1−𝛽
𝛼 𝛽 = ⋅𝛽
1−𝛽 1−𝛽
1 1 − 𝛼 ⟹
𝜇 𝑥 ,𝑥 = 𝑥 − 𝑧 (𝑥 , 𝑡)
𝛼
1−𝛽
Ignored

𝛽
⟹ 𝐾𝐿(𝑝 𝑥 𝑥 , 𝑥 |𝑝 𝑥 𝑥 = 𝔼 ~𝒩( , ) 𝑧 − 𝑧 (𝑥 , 𝑡)
2𝜎 𝛼 (1 − 𝛽 )
DDPMs. Training Algorithm

1
min 𝔼 ~ , ~𝒩 , 𝑧 − 𝑧 (𝑥 , 𝑡)
𝑇

Training algorithm:

Repeat 𝛽 = 𝛼
𝑥 ~𝑝 𝑥
𝑡~𝒰 1, … , 𝑇
𝑧 ~𝒩(0, I)
𝑥 = 𝛽 ⋅𝑥 + 1−𝛽 𝑧
𝜃 = 𝜃 − 𝑙𝑟 ⋅ ∇ ℒ
Until convergence
 DDPMs. Sampling

𝑥
𝑧 (𝑥 , 𝑡)

• Pass the current noisy image along with t to the neural network

• With the resultant compute the mean of the gaussian distribution

 DDPMs. Sampling

𝑥
𝑧 (𝑥 , 𝑡)

Sample the image for the next iteration

𝜇 (𝑥 , 𝑡)

1 1 − 𝛼
~𝒩 𝑥 , 𝑥 − 𝑧 𝑥 ,𝑡 ,𝜎 I
𝛼
1−𝛽

𝑥
Thank You