Scribd
Upload
10 views30 pages

Mit Diffusion

The document outlines a course on Generative AI focusing on flow and diffusion models, emphasizing their applications in generating images, videos, and proteins. It details the course structure, including lectures on theory, implementation, and advanced topics, while also introducing key concepts such as data distribution and conditional generation. The course aims to equip students with the necessary mathematical foundations and practical skills to implement generative algorithms.

Uploaded by

marcus
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
10 views30 pages

Mit Diffusion

The document outlines a course on Generative AI focusing on flow and diffusion models, emphasizing their applications in generating images, videos, and proteins. It details the course structure, including lectures on theory, implementation, and advanced topics, while also introducing key concepts such as data distribution and conditional generation. The course aims to equip students with the necessary mathematical foundations and practical skills to implement generative algorithms.

Uploaded by

marcus
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 30

Generative AI with Stochastic

Differential Equations
An introduction to flow and diffusion models

MIT IAP 2025 | Jan 21, 2025

Peter Holderrieth and Ezra Erives

Sponsor: Tommi Jaakkola


Welcome to class 6.S184/6.S975!
Course Instructors Sponsor

Peter Ezra Tommi


Generative AI - A new generation of AI systems

Artistic Images Realistic Videos Draft Texts

These systems are “creative”: they generate new objects.

This class teaches you algorithms to generate objects.


Flow and Diffusion Models: state-of-the-art models for
generating images, videos, proteins!

Stable Diffusion OpenAI Sora AlphaFold3


DALEE Meta MovieGen RFDiffusion

Most SOTA generative AI models for


images/videos/proteins/robotics: Diffusion and Flow Models
This Class: Theory and Practice of Flow/Diffusion Models

Theory Ordinary/stochastic differential


equations
Flow and Diffusion
Models
Practice How to implement and apply these
machine learning models

The goal of this class to teach you:


1. Flow and diffusion models from first principles.
2. The minimal but necessary amount of mathematics for 1.
3. How to implement and apply these algorithms.
Class Overview
● Lecture 1 (today):
○ From Generation to Sampling: Formalize “generating an image/etc.”
○ Construct Flow and Diffusion Models
● Lecture 2 - Marginal Vector Field and Score: Derive training targets.
● Lecture 3 - Flow Matching and Score Matching: Define training algorithm.
● Lecture 4 - Build Image Generators: Network architectures + conditioning
● Lecture 5 - Advanced Topics: Alignment, complex data types, distillation
● Lecture 6 - Guest lecture:
○ Jason Yim (MIT): Protein design
○ Benjamin Burchfiel (Toyota Research Institute): Robotics
Section 1:
From Generation to Sampling

Goal: Formalize what it means to “generate” something.


We represent images/videos/protein as vectors
Images: Videos: Molecular structures:

- Height H and Width W - T time frames - N atoms


- 3 color channels (RBG) - Each frame is image - Each atom has 3 coordinate

We represent the objects we want to generate as vectors:


What does it mean to successfully generate something?
Prompt: “A picture of a dog”

Useless < Bad < Wrong animal < Great!

These are subjective statements - Can we formalize this?


Data Distribution: How “likely” are we to find this picture in
the internet? Prompt: “A picture of a dog”

Impossible < Rare < Unlikely < Very likely

How good an image is ~= How likely it is under the


data distribution
Generation as sampling from the data distribution
Data distribution: Distribution of objects that we want to generate:
Probability density:
Note: We
don’t know the
probability
density!

Generation means sampling the data distribution:


A Dataset consists of samples from the data distribution

Training requires datasets: To train our algorithms, we need a dataset.

Examples:

● Images: Publicly available images from the internet


● Videos: YouTube
● Protein structures: Scientific data (e.g. Protein Data Bank)

A dataset consists of a finite number of samples from the data


distribution:
Conditional Generation allows us to condition on prompts
Data distribution Conditional data distribution
Fixed prompt Condition variable: y
“Dog” y=“Dog” y=“Cat” y=“Landscape”

Conditional generation means sampling the conditional data


distribution:

We will first focus on unconditional generation and then learn how to


translate an unconditional model to a conditional one.
Generative Models generate samples from data distribution
Initial distribution: Default:

A generative model converts samples from a initial distribution (e.g.


Gaussian) into samples from the data distribution:

Generative
Model
Summary
● Objects to Generate: We focus on vectors z representing data objects (e.g., images, videos)
● Data distribution: Distribution that places higher probability to objects that we consider
“good”.
● Generation as sampling: generate an object = sampling from the data distribution
● Dataset: Finite number of samples from the data distribution used for training
● Conditional Generation: Condition on label y and sample from the conditional data
distribution
● Generative Model: Train a model to transform samples from a simple (e.g., Gaussian)
distribution into the data distribution.
Section 2:
Flow and Diffusion Models

Goal: Understand differential equations and how we can build


generative models with them.
Flow - Example
Existence and Uniqueness Theorem ODEs
Theorem (Picard–Lindelöf theorem): If the vector field , is
continuously differentiable with bounded derivatives, then a unique
solution to the ODE

exists. In other words, a flow map exists. More generally, this is true if
the vector field is Lipschitz.
Key takeaway: In the cases of practical interest for machine learning,
unique solutions to ODE/flows exist.

Math class: Construct solutions via Picard-Iteration


Example: Linear ODE
Simple vector field:

Claim: Flow is given by

Proof:

1. Initial condition:

2. ODE:
Numerical ODE simulation - Euler method

Small step into direction of vector field

Return trajectory
Toy example

Figure credit:
Yaron Lipman
Toy Flow
Model

Figure credit:
Yaron Lipman
How to generate objects with a Flow Model

Random initialization!

Return final point


Brownian Motion
Existence and Uniqueness Theorem SDEs
Theorem: If the vector field , is continuously differentiable with bounded
derivatives and the diffusion coeff. is continuous, then a unique solution to the
SDE

exists. More generally, this is true if the vector field is Lipschitz.

Key takeaway: In the cases of practical interest for machine learning,


unique solutions to SDE.

Stochastic calculus class: Construct solutions via stochastic integrals and


Ito-Riemann sums
Numerical SDE simulation (Euler-Maruyama method)

Add additional noise with var=h


scaled by diffusion coefficient
Ornstein-Uhlenbeck Process

Increasing diffusion coefficient


Logistics Website: https://diffusion.csail.mit.edu/

How to pass this class:

1. Come to lecture
2. Do the labs (necessary to pass)!

Support:

1. Use the lecture notes (self-contained)


2. Come to office hours

Lab 1 is out today (see website)! We recommend doing it as


soon possible for you!
Next class:
Wednesday (Tomorrow), 11am-12:30pm

E25-111 (same room)

Office hours: Wednesday (tomorrow), 3pm-4:30pm in 37-212

You might also like