Introduction

To some extent, we've 'solved' image generation (ie, specify any number of bits about an image, and they'll be rendered ~exactly, with the rest filled in plausibly). For a while I've wanted to know how image generation got so good, so I spent a few days talking to GPT and wrote up what I learned:)

Why is generating images hard?

'Generating images' willy nilly is a much harder problem than it sounds, mainly because there are so many possibilities. Imagine we have a dataset of full-detail face images x0∈Rdx_0\in\mathbb{R}^d (d=3HWd=3HW for an H×WH\times W RGB image), each paired with a blocked-in sketch y∈Rdy\in\mathbb{R}^d of the same face (rough head shape, eye/nose placement, hairstyle, etc). Given paired (y,x0)(y,x_0) data and a continuous codomain, the obvious approach is regression: train a deterministic model fθ:Rd→Rdf_\theta:\mathbb{R}^d\to\mathbb{R}^d with L2 (MSE) to map each yy to a single x^0=fθ(y)\hat x_0=f_\theta(y)⁠.

However, this model will almost always produce blurry, washed-out faces, no matter how big your model or dataset. Through an information theory lens, yy specifies some bits about x0x_0⁠, but not enough to fully pin down a single x0x_0 (imagine filling in yy to get some x0x_0⁠, then making an eyelash longer, adding stubble, or changing the skin texture entirely). Each yy defines a multimodal probability distribution p(x0∣y)p(x_0\mid y) over image space (similar images are close together in image space, so probability mass clusters into 'modes' around each plausible x0x_0⁠), and the x0x_0 that produced yy is a single sample from it. Across a large dataset, similar yys are paired with vastly different x0x_0s, but neural networks like fθf_\theta are Lipschitz continuous by construction (or at least locally), so similar yys must produce similar x^0\hat x_0s. So how does fθf_\theta resolve conflicting x0x_0s? L2 implicitly treats p(x0∣y)p(x_0\mid y) as a single bell curve (minimizing L2 = maximizing likelihood under a fixed-variance Gaussian centered at x^0\hat x_0⁠), and the best fixed-variance bell curve for any distribution is centered at its mean, meaning fθf_\theta learns to mode average, or predict the probability-weighted mean over all modes: fθ(y)≈E[x0∣y]f_\theta(y)\approx\mathbb{E}[x_0\mid y]⁠.

To generate a high-quality image, fθf_\theta must commit to a single plausible x0x_0 (equivalently, a single mode of p(x0∣y)p(x_0\mid y)⁠), but the hand (multimodal p(x0∣y)p(x_0\mid y)⁠) just doesn't fit the glove (single fixed-variance Gaussian L2). What if we change the learning objective?

A GAN concurrently trains a generator G:Rk→RdG:\mathbb{R}^k\to\mathbb{R}^d to turn kk-dimensional noise vectors zz into images x^0=G(z)\hat x_0=G(z)⁠, and a discriminator D:Rd→[0,1]D:\mathbb{R}^d\to[0,1] to estimate the probability that an image is real. An ideal DD pushes GG to generate diverse x^0\hat x_0s that fall into any mode of p(x0)p(x_0)⁠. Since GG's only source of variety is zz⁠, it learns to assign 'meaning' to directions in zz⁠; at generation time, sampling zz fully determines x^0\hat x_0⁠, and GG is essentially a renderer. Unfortunately, the adversarial objective is unstable (a constant tug of war) and unanchored (loss values say nothing about quality and are incomparable across runs), so GG often mode-collapses and doesn't scale predictably.

Over a discrete set of outputs, a model with a softmax head creates a distribution pθp_\theta⁠, and the expected cross-entropy Ex0∼p[−log⁡pθ(x0)]\mathbb{E}_{x_0\sim p}\big[-\log p_\theta(x_0)\big] is smallest when pθp_\theta matches the true distribution pp⁠. However, this requires reframing our regression problem as classification: represent x0x_0 as a sequence of LL one-hot tokens over KK values (eg, a flattened discrete autoencoder bottleneck, where token position ↔\leftrightarrow image position, token value ↔\leftrightarrow which of KK learned patch types the region looks like) and train a model to generate new x^0\hat x_0 sequences via autoregressive NTP. For rendering pixels diffusion simply outperforms: AR images can only be as detailed as their KK patch types, require at least LL model calls each (with L∝L\propto resolution), and must follow an arbitrary top-left-to-bottom-right raster order. That said, autoregressive models (LLMs) have proven extremely useful for the 'text' parts of 'text-to-image'.

To directly answer our guiding question: image generation got good by 1. finding a workable inductive bias (diffusion), and 2. continuously pulling high-leverage levers within it (sampling, scaling, conditioning). Rather than fighting L2, diffusion uses a few clever tricks to narrow the modes of p(x0∣⋅)p(x_0\mid\cdot) until E[x0∣⋅]\mathbb{E}[x_0\mid\cdot] is a sharp, specific, high-quality x^0\hat x_0⁠. The rest of this post is about how it works, and how it went from good to great.

DDPM

Diffusion Probabilistic Models (DPMs), proposed in 2015, model p(x0)p(x_0) by iteratively adding noise to images and learning to reverse the process. Denoising DPMs (DDPMs) clean up their objective and make them competitive with GANs.

A noisy image can be written as x=x0+σεx=x_0+\sigma\varepsilon⁠, with clean image x0x_0⁠, Gaussian noise ε∼N(0,I)\varepsilon\sim\mathcal{N}(0,I)⁠, and noise level σ\sigma⁠. DDPM technically uses the variance-preserving form of the same equation, x=x0αˉ+ε1−αˉx=x_0\sqrt{\bar\alpha}+\varepsilon\sqrt{1-\bar\alpha}⁠, but dividing by αˉ\sqrt{\bar\alpha} gives the equivalent, easier-to-interpret variance-exploding form (σ=(1−αˉ)/αˉ\sigma=\sqrt{(1-\bar\alpha)/\bar\alpha}⁠). Like our blocked-in sketches, xx defines a multimodal distribution p(x0∣x)p(x_0\mid x)⁠: an image feature of contrast aa and area nn only stays visible if signal an≳an\gtrsim noise σn\sigma\sqrt{n} (ie, as σ\sigma goes from σmax⁡\sigma_{\max} to 00⁠, more and more bits of x0x_0 become specified).

DDPM has two parts: a U-Net εθ(x,σ)\varepsilon_\theta(x,\sigma) trained on L2 to predict ε\varepsilon⁠, and a sampler that denoises σmax⁡ε\sigma_{\max}\varepsilon into a specific x^0\hat x_0 over many steps, each decreasing σ\sigma by Δσi=σi−σi−1\Delta\sigma_i=\sigma_i-\sigma_{i-1}⁠. A single step from x=σmax⁡εx=\sigma_{\max}\varepsilon to x^0\hat x_0 gives the dataset average, x^0=x−σmax⁡εθ(x,σmax⁡)≈E[x0]\hat x_0=x-\sigma_{\max}\varepsilon_\theta(x,\sigma_{\max})\approx\mathbb{E}[x_0]⁠, since p(x0∣σmax⁡ε)≈p(x0)p(x_0\mid\sigma_{\max}\varepsilon)\approx p(x_0)⁠, whereas committing to a single xi−1∼p(xi−1∣xi)x_{i-1}\sim p(x_{i-1}\mid x_i) at each step iteratively narrows p(x0∣xi)p(x_0\mid x_i)⁠.

So how do we sample from p(xi−1∣xi)p(x_{i-1}\mid x_i)⁠? By Gaussian conjugacy, p(xi−1∣xi,x0)p(x_{i-1}\mid x_i,x_0) is Gaussian with mean μ~i(x0)=σi−12σi2xi+(1−σi−12σi2)x0\tilde\mu_i(x_0)=\frac{\sigma_{i-1}^2}{\sigma_i^2}x_i+\Big(1-\frac{\sigma_{i-1}^2}{\sigma_i^2}\Big)x_0 and standard deviation σ~i=σi−11−σi−12/σi2\tilde\sigma_i=\sigma_{i-1}\sqrt{1-\sigma_{i-1}^2/\sigma_i^2}⁠. By the law of total probability, p(xi−1∣xi)=Ex0∼p(x0∣xi)[p(xi−1∣xi,x0)]p(x_{i-1}\mid x_i)=\mathbb{E}_{x_0\sim p(x_0\mid x_i)}\big[p(x_{i-1}\mid x_i,x_0)\big]⁠, and a weighted average of Gaussians sharing σ~i\tilde\sigma_i is approximately a Gaussian when ∥μ~i(x0)−μ~i(x0′)∥≪σ~i\|\tilde\mu_i(x_0)-\tilde\mu_i(x_0')\|\ll\tilde\sigma_i⁠. For small steps (Δσi≪σi\Delta\sigma_i\ll\sigma_i⁠), σ~i∝Δσi\tilde\sigma_i\propto\sqrt{\Delta\sigma_i} and μ~i(x0)−μ~i(x0′)≈2Δσiσi(x0−x0′)∝Δσi\tilde\mu_i(x_0)-\tilde\mu_i(x_0')\approx\frac{2\Delta\sigma_i}{\sigma_i}(x_0-x_0')\propto\Delta\sigma_i⁠, so p(xi−1∣xi)p(x_{i-1}\mid x_i) is approximately Gaussian!

Since μ~i\tilde\mu_i is linear in x0x_0⁠, linearity of expectation gives the mean of p(xi−1∣xi)p(x_{i-1}\mid x_i) as μ~i(E[x0∣xi])\tilde\mu_i(\mathbb{E}[x_0\mid x_i])⁠. Mode averaging gives us E[x0∣xi]≈xi−σiεθ(xi,σi)\mathbb{E}[x_0\mid x_i]\approx x_i-\sigma_i\varepsilon_\theta(x_i,\sigma_i)⁠, so with the reparametrization trick we can sample xi−1=μ~i(xi−σiεθ(xi,σi))+σ~iξx_{i-1}=\tilde\mu_i\big(x_i-\sigma_i\varepsilon_\theta(x_i,\sigma_i)\big)+\tilde\sigma_i\xi⁠, where ξ∼N(0,I)\xi\sim\mathcal{N}(0,I)⁠. Expanding and simplifying, we get the final DDPM update rule: xi−1=xi−σi2−σi−12σiεθ(xi,σi)+σ~iξ.x_{i-1}=x_i-\frac{\sigma_i^2-\sigma_{i-1}^2}{\sigma_i}\varepsilon_\theta(x_i,\sigma_i)+\tilde\sigma_i\xi.

DDIM

To sample from p(xi−1∣xi)p(x_{i-1}\mid x_i)⁠, DDPM is forced to take small, noisy steps (essentially a random walk down −log⁡p(x)-\log p(x)⁠). Instead, DDIM deterministically computes xi−1x_{i-1}⁠, generating comparable images in 10–50 steps instead of ~1,000.

Differentiating x=x0+σεx=x_0+\sigma\varepsilon wrt σ\sigma (holding x0x_0 and ε\varepsilon fixed) gives ∂x∂σ=ε\frac{\partial x}{\partial\sigma}=\varepsilon⁠. Substituting DDPM's εθ\varepsilon_\theta gives the ODE dxdσ=εθ(x,σ),\frac{dx}{d\sigma}=\varepsilon_\theta(x,\sigma), a velocity field over image space, which we can follow using Euler's method: xi−1=xi−Δσiεθ(xi,σi).x_{i-1}=x_i-\Delta\sigma_i\varepsilon_\theta(x_i,\sigma_i). As σi→0\sigma_i\to0⁠, xix_i moves along a denoising trajectory x(σ)x(\sigma) toward x^0\hat x_0 (aka: a particular solution to the ODE on [0,σmax⁡][0,\sigma_{\max}] with initial condition x(σmax⁡)=σmax⁡εx(\sigma_{\max})=\sigma_{\max}\varepsilon⁠, an integral curve of εθ\varepsilon_\theta⁠, or a flow line of εθ\varepsilon_\theta⁠). Empirically, x(σ)x(\sigma)s are also gently curved, allowing Euler's method to take large Δσi\Delta\sigma_is.

For any σmin⁡>0\sigma_{\min}>0⁠, εθ\varepsilon_\theta is Lipschitz continuous on [σmin⁡,σmax⁡][\sigma_{\min},\sigma_{\max}]⁠, so Picard–Lindelöf guarantees that each σmax⁡ε\sigma_{\max}\varepsilon has a unique trajectory x(σ)x(\sigma)⁠. That means DDIM can step an x0x_0 back up in σ\sigma to recover the σmax⁡ε\sigma_{\max}\varepsilon that generates it (DDIM inversion), which can be used to reconstruct or semantically interpolate between x0x_0s. Inverted images drift slightly, though, because inversion is only exact in the continuous limit, Euler steps add error, and Lipschitz continuity breaks down at σ=0\sigma=0⁠.

Other later works (Consistency Models) also take advantage of Picard–Lindelöf by training every point on x(σ)x(\sigma) to predict its endpoint x^0\hat x_0⁠. Consistency Models work particularly well for distilling diffusion models into few-step samplers.

ADM

ADM ablates the DDPM/DDIM recipe, finding significant gains from scaling εθ\varepsilon_\theta (adding self-attention layers & heads, residual blocks, and AdaGN). ADM also introduces Classifier Guidance, which allows steering denoising toward a chosen class yy (unconditional diffusion chooses from ImageNet's ~1,000 classes at random).

First, a classifier (a randomly initialized clone of the downsampling half of εθ\varepsilon_\theta plus an output head) is trained to predict p(y∣x)p(y\mid x) from (x,σ)(x,\sigma)⁠, then its gradient ∇xilog⁡p(y∣xi)\nabla_{x_i}\log p(y\mid x_i) is computed by backpropagating log⁡p(y∣xi)\log p(y\mid x_i) to xix_i (rather than to classifier weights). During each DDIM step, this gradient is scaled by σi\sigma_i and a strength constant ww⁠, then subtracted from εθ(xi,σi)\varepsilon_\theta(x_i,\sigma_i)⁠: εguided=εθ(xi,σi)−wσi∇xilog⁡p(y∣xi).\varepsilon_{\text{guided}}=\varepsilon_\theta(x_i,\sigma_i)-w\sigma_i\nabla_{x_i}\log p(y\mid x_i). Geometrically, Classifier Guidance superimposes the classifier's gradient field onto εθ\varepsilon_\theta at generation time. Functionally, p(x0∣xi,y)p(x_0\mid x_i,y) is narrower than p(x0∣xi)p(x_0\mid x_i)⁠.

One question I had was, why do we scale ∇xilog⁡p(y∣xi)\nabla_{x_i}\log p(y\mid x_i) by σi\sigma_i⁠? Classifier Guidance's goal is to find a class-conditional version of Tweedie's formula, εcond=−σi∇xilog⁡p(xi∣y)\varepsilon_{\text{cond}}=-\sigma_i\nabla_{x_i}\log p(x_i\mid y)⁠, but we don't know ∇xilog⁡p(xi∣y)\nabla_{x_i}\log p(x_i\mid y)⁠. Taking the ∇xilog⁡\nabla_{x_i}\log of Bayes' theorem gives ∇xilog⁡p(xi∣y)=∇xilog⁡p(xi)+∇xilog⁡p(y∣xi)−∇xilog⁡p(y)\nabla_{x_i}\log p(x_i\mid y)=\nabla_{x_i}\log p(x_i)+\nabla_{x_i}\log p(y\mid x_i)-\nabla_{x_i}\log p(y)⁠. Zeroing out ∇xilog⁡p(y)\nabla_{x_i}\log p(y) (since p(y)p(y) doesn't depend on xix_i⁠), multiplying by −σi-\sigma_i⁠, substituting εθ(xi,σi)≈−σi∇xilog⁡p(xi)\varepsilon_\theta(x_i,\sigma_i)\approx-\sigma_i\nabla_{x_i}\log p(x_i)⁠, and scaling the classifier term by ww⁠, we get: εguided=εθ(xi,σi)−wσi∇xilog⁡p(y∣xi).\varepsilon_{\text{guided}}=\varepsilon_\theta(x_i,\sigma_i)-w\sigma_i\nabla_{x_i}\log p(y\mid x_i).

One objection you might have is, why not just feed a class embedding into εθ\varepsilon_\theta⁠? ADM also does this! Ranked by FID (measuring quality and diversity): embedding + CG (w=1w=1⁠) > embedding + CG (w=10w=10⁠) > embedding alone > CG (w=10w=10⁠) > CG (w=1w=1⁠). Notably, embedding + CG (w=10w=10⁠) produces the highest-quality images at some cost to diversity, while embedding provides richer signal at fixed, insufficient strength.

Classifier-Free Guidance

Classifier-Free Guidance's goal is to compute ∇xilog⁡p(y∣xi)\nabla_{x_i}\log p(y\mid x_i) from the embedding signal. To do this, CFG trains one εθ\varepsilon_\theta to denoise with and without yy⁠. Rearranging our ∇log⁡\nabla\log Bayes identity from ADM, we get ∇xilog⁡p(y∣xi)=∇xilog⁡p(xi∣y)−∇xilog⁡p(xi)\nabla_{x_i}\log p(y\mid x_i)=\nabla_{x_i}\log p(x_i\mid y)-\nabla_{x_i}\log p(x_i) (ie, push toward y=push toward real y images−push toward real images\text{push toward }y=\text{push toward real }y\text{ images}-\text{push toward real images}⁠). By Tweedie's formula, εθ(xi,σi,y)≈−σi∇xilog⁡p(xi∣y)\varepsilon_\theta(x_i,\sigma_i,y)\approx-\sigma_i\nabla_{x_i}\log p(x_i\mid y) and εθ(xi,σi,∅)≈−σi∇xilog⁡p(xi)\varepsilon_\theta(x_i,\sigma_i,\varnothing)\approx-\sigma_i\nabla_{x_i}\log p(x_i)⁠; scaling the modified Bayes identity by −σi-\sigma_i and substituting, we get −σi∇xilog⁡p(y∣xi)≈εθ(xi,σi,y)−εθ(xi,σi,∅).-\sigma_i\nabla_{x_i}\log p(y\mid x_i)\approx\varepsilon_\theta(x_i,\sigma_i,y)-\varepsilon_\theta(x_i,\sigma_i,\varnothing).

Substituting into εguided\varepsilon_{\text{guided}} from ADM: εguided=εθ(xi,σi,∅)+w(εθ(xi,σi,y)−εθ(xi,σi,∅)).\varepsilon_{\text{guided}}=\varepsilon_\theta(x_i,\sigma_i,\varnothing)+w\big(\varepsilon_\theta(x_i,\sigma_i,y)-\varepsilon_\theta(x_i,\sigma_i,\varnothing)\big).

Latent Diffusion

Up to this point, the diffusion recipe has been driven by clever theoretical reformulations, but from Latent Diffusion onwards, progress mostly comes from subtle changes and extreme attention to detail. I'll focus on the headline changes, but be aware that exact subtleties are extremely important from here on out. If you want to deeply understand what makes diffusion work, please please please read the papers/code (and check the PSET at the end 🙂).

One way to improve image quality is to increase resolution, but we'd quickly become memory-bound. The first convolution in εθ\varepsilon_\theta expands a 256×256×3256\times256\times3 image to 256 channels, producing a ~34 MB feature map (fp16). Backprop stores many intermediate activations, totaling ~1.5 GB per image, so training at batch size 256 would need ~384 GB of VRAM (ADM takes 22,000 V100-hours to train with data parallelism, ~2 months on 16 GPUs). Activation memory scales as M∝B×H×WM\propto B\times H\times W⁠, so 512×512512\times512 needs 4×4\times more VRAM (~1.5 TB, ~8 months), 1024×10241024\times1024 needs 16×16\times more (~6 TB, ~2.7 years), and so on.

Latent Diffusion trains an autoencoder (essentially a lightly regularized VAE) to downsample each image by a factor rr into a latent with cc channels, usually r=8r=8⁠, c=4c=4 ("f8c4"): 256×256×3→32×32×4256\times256\times3\to32\times32\times4⁠, around 48×48\times fewer values for ~the same image. Then, diffusion is run in the autoencoder's latent space, and only the final x^0\hat x_0 is decoded back to pixels. The idea that raw images are highly compressible is already well known (manifold hypothesis, JPEG, etc); Latent Diffusion's contribution is finding a recipe that works: too little downsampling (r≤2r\le2⁠) barely saves compute and too much (r=32r=32⁠) loses detail diffusion can't recover (r=4r=4–1616 being the sweet spot). Training the autoencoder and diffusion model separately also keeps training simpler and more stable, and lets one autoencoder serve many εθ\varepsilon_\thetas.

Latent Diffusion also adds cross-attention, letting each image region attend to specific text tokens instead of one shared conditioning vector.

Imagen

Image generation is getting good, but models still struggle to follow instructions (objects go missing, attributes mix, etc) and render subtle details. Part of the problem is that models are weak (eg, Latent Diffusion with small cc physically can't represent fine letters or numbers), but Imagen's main finding is that scaling the text encoder pays off far more than scaling εθ\varepsilon_\theta⁠.

This makes sense: prompts can be nuanced, and small encoders trained jointly with diffusion can't make "a red cube on a blue cube" and "a blue cube on a red cube" meaningfully different embeddings. Instead, Imagen uses a frozen T5 (an early text-to-text LM) to enrich the text tokens that εθ\varepsilon_\theta cross-attends to. This sets a nice precedent: as LLMs improve, image models can ride on their success! Imagen also tweaks the pixel-space recipe (mostly irrelevant once Latent Diffusion takes over).

EDM / DPM-Solver

If generating an image is ~solving an ODE, using better solvers has high leverage! EDM and DPM-Solver both use higher-order solvers to get more accuracy per εθ\varepsilon_\theta call.

Taylor-expanding x(σ)x(\sigma) over one step, we get x(σi−1)=xi−Δσix′+Δσi22x′′−…x(\sigma_{i-1})=x_i-\Delta\sigma_ix'+\frac{\Delta\sigma_i^2}{2}x''-\dots⁠. Euler's method only keeps the first two terms, implicitly assuming x′x' is constant over a step. EDM uses Heun's method, which captures one more Taylor term by assuming x′x' changes linearly: take an Euler step to xi−1trialx_{i-1}^{\text{trial}}⁠, then step from xix_i using the average slope, xi−1=xi−Δσiεθ(xi,σi)+εθ(xi−1trial,σi−1)2.x_{i-1}=x_i-\Delta\sigma_i\frac{\varepsilon_\theta(x_i,\sigma_i)+\varepsilon_\theta(x_{i-1}^{\text{trial}},\sigma_{i-1})}{2}.

Geometrically, on a steadily curving path (constant x′′x''⁠), the slopes at the start and end of a step miss the direction to the true next point by Δσi2x′′\frac{\Delta\sigma_i}{2}x'' in opposite directions, so averaging them cancels the Δσi2\Delta\sigma_i^2 error term. Heun's per-step error scales as Δσi3\Delta\sigma_i^3 instead of Euler's Δσi2\Delta\sigma_i^2⁠, at the cost of two εθ\varepsilon_\theta calls per step (four with CFG).

DPM-Solver captures the same extra term by assuming εθ\varepsilon_\theta changes linearly in log⁡σ\log\sigma (every halving of σ\sigma changes εθ\varepsilon_\theta by about the same amount). Heuristically, this makes sense: natural images have power spectra that roughly follow 1/ω21/\omega^2 (spatial frequency ω\omega⁠), so each halving of σ\sigma uncovers another octave of frequencies, and each octave carries a similar amount of image content. Under the linear-in-log⁡σ\log\sigma assumption, the prediction at the step's log-midpoint σi−12=σiσi−1\sigma_{i-\frac12}=\sqrt{\sigma_i\sigma_{i-1}} is nearly the step's average. So DPM-Solver takes an Euler step to xi−12=xi−(σi−σi−12)εθ(xi,σi)x_{i-\frac12}=x_i-(\sigma_i-\sigma_{i-\frac12})\varepsilon_\theta(x_i,\sigma_i) and uses that midpoint prediction for the whole step: xi−1=xi−Δσiεθ(xi−12,σi−12).x_{i-1}=x_i-\Delta\sigma_i\varepsilon_\theta(x_{i-\frac12},\sigma_{i-\frac12}).

EDM also 1. samples training σ\sigmas from a log-normal distribution, focusing εθ\varepsilon_\theta's training on mid-level σ\sigmas where it actually commits features, 2. introduces a noise schedule parametrized by ρ\rho⁠, which controls how strongly steps bunch at low σ\sigmas (ρ=1\rho=1⁠: evenly spaced in σ\sigma⁠; ρ→∞\rho\to\infty⁠: evenly spaced in log⁡σ\log\sigma⁠), and 3. adds preconditioning, which uses σ\sigma-dependent scalings and a skip connection around εθ\varepsilon_\theta to keep its inputs and outputs similarly scaled at any σ\sigma⁠.

Stable Diffusion

Stable Diffusion, a ~860M-parameter εθ\varepsilon_\theta trained on 512×512512\times512 images, conditioned using a frozen CLIP-L text encoder, essentially just scales up Latent Diffusion. Its significance mostly comes from being open-weights and running on consumer GPUs, which creates a massive ecosystem of samplers, finetunes, and tools. One of the most popular samplers is DPM-Solver++ 2M on EDM's noise schedule at ρ=7\rho=7 ("Karras").

Rectified Flow / Flow Matching

Instead of using better solvers, Rectified Flow constructs a better ODE. RF reparametrizes the noise corruption formula x=x0+σεx=x_0+\sigma\varepsilon into x=(1−t)x0+tεx=(1-t)x_0+t\varepsilon⁠, where t∈[0,1]t\in[0,1]⁠. Both describe the same path (up to rescaling, with σ=t/(1−t)\sigma=t/(1-t)⁠), but RF changes the prediction target: differentiating wrt tt (holding x0x_0 and ε\varepsilon fixed) gives ∂x∂t=ε−x0,\frac{\partial x}{\partial t}=\varepsilon-x_0, so each (x0,ε)(x_0,\varepsilon) pair moves at a constant velocity v=ε−x0v=\varepsilon-x_0⁠. Training vθ(x,t)v_\theta(x,t) to predict vv gives the ODE dxdt=vθ(x,t)\frac{dx}{dt}=v_\theta(x,t) with denoising trajectory x(t)x(t)⁠. Euler's method and CFG apply as before.

Predicting vv also avoids a few problems with predicting ε\varepsilon⁠. In DDPM's variance-preserving form, reconstructing x^0=(x−εθ(x,σ)1−αˉ)/αˉ\hat x_0=(x-\varepsilon_\theta(x,\sigma)\sqrt{1-\bar\alpha})/\sqrt{\bar\alpha} amplifies the model's mistakes by 1/αˉ1/\sqrt{\bar\alpha} (about 160×160\times at the noisiest step, where αˉ≈0.0064\sqrt{\bar\alpha}\approx0.0064⁠) and is impossible at pure noise (αˉ=0\sqrt{\bar\alpha}=0⁠). This also creates a subtle train-test mismatch: its noisiest training step keeps ~0.6% of x0x_0⁠, while generation starts from 0%. Flow Matching generally shows that a 'diffusion model' can be trained on any path from x0x_0 to ε\varepsilon⁠, but RF's path is the most common. (FM/RF also have Jacobian interpretations, which I skip.)

One reason I find machine learning so fascinating is that you can just impose structure and train away constraints you'd have classically been limited by. RF also introduces reflow, an optional step to make x(t)x(t)s straighter: train vθv_\theta as usual, generate (x^0,ε)(\hat x_0,\varepsilon) pairs by sampling ε\varepsilon and running the ODE, then keep training vθv_\theta on those pairs. Randomly paired (x0,ε)(x_0,\varepsilon)s give each (x,t)(x,t) many possible x0x_0s, which leads to mode averaging and bent trajectories; generated trajectories can't cross (Picard–Lindelöf), so each ε\varepsilon maps to exactly one x^0\hat x_0⁠, and training on straight lines between these pairs produces straighter x(t)x(t)s. Reflow can be repeated (with diminishing returns) and produces usable images in 1–2 steps :o!

DiT

Following ADM and Latent Diffusion, U-Nets have increasingly become transformer-like (50–75% of SD1's U-Net is transformer blocks). A Diffusion Transformer drops the U-Net entirely and uses a ViT-style transformer as εθ\varepsilon_\theta⁠. It patchifies the latent into a sequence of patches, linearly embeds and adds a positional encoding to each one, runs the sequence through standard transformer blocks, and assembles εθ(x,σ,y)\varepsilon_\theta(x,\sigma,y)⁠. Each DiT block is conditioned on σ\sigma and an ImageNet class label yy via adaLN-Zero (no text prompts to keep the architecture study simple).

DiTs scale smoothly and predictably with forward-pass compute (GFLOPs). U-Nets generally improve with scale, but lack predictability (partly because they have many axes to scale: channels, blocks, attention placement, etc, whereas a transformer has depth, width, and token count). Using transformers also lets us take advantage of the rapidly advancing LLM training stack (FlashAttention, FSDP, kernels, mixed precision, etc).

DALL·E 3

DALL·E 1 and 2 focus on autoregression and diffusion respectively, but DALL·E 3 is all about data! Most text-to-image datasets pair web images with short, vague, and incomplete scraped alt-text, which (besides affecting instruction following) does little to narrow p(x0∣y)p(x_0\mid y) relative to p(x0)p(x_0)⁠.

DALL·E 3's authors finetune a captioner (a language model conditioned on a CLIP image embedding) on a small set of ultra-detailed captions, use it to recaption their dataset, and train a diffusion model on a mix of 95% synthetic and 5% original alt-text. At generation time, GPT-4 "upsamples" the user's prompt into the synthetic captions' style and detail.

It's extremely cool to me how new models unlock new capabilities (eg, forming an image by describing it with words, splitting a song into its audio tracks, etc), which create more data, which creates even more new capabilities, and so on! I looooove synthetic data sooooooo much it's so cool!

SD3

SD3 is another massive ablation paper. Its best model is an 8B MMDiT vθv_\theta trained with RF (no reflow) on the latents of an r=8r=8⁠, c=16c=16 autoencoder. Notably, an MMDiT conditions on text using joint attention (text and image tokens are concatenated into one token sequence but keep separate weights), which improves prompt adherence, attribute binding, and text rendering since information flows bidirectionally (text ↔\leftrightarrow image). Prompts go through three frozen encoders (CLIP-L and OpenCLIP-bigG for image semantics, T5-XXL for language understanding/text rendering); pooled CLIP embeddings plus a tt embedding condition every block via adaLN, and all three encoders' token embeddings are combined to form text tokens.

SD3 is trained on a 50/50 mix of synthetically labeled captions and original alt-text, with tt drawn from a logit-normal distribution (EDM's log-normal σ\sigma sampling, adapted to RF), starting at 256×256256\times256 and going up to 1024×10241024\times1024⁠. SD3 also introduces post-training with Diffusion-DPO (another trick from the LLM playbook!). SD3 samples with plain Euler + CFG, and its validation loss falls predictably with size while tracking human preference!

Many of SD3's authors went on to found Black Forest Labs, whose FLUX models mostly continue scaling SD3 and curating data. One cool idea they popularize is guidance distillation: CFG's two passes only produce vguided=vθ(x,t,∅)+w(vθ(x,t,y)−vθ(x,t,∅))v_{\text{guided}}=v_\theta(x,t,\varnothing)+w\big(v_\theta(x,t,y)-v_\theta(x,t,\varnothing)\big)⁠, a fixed function of (x,t,y,w)(x,t,y,w)⁠, so a student can be trained to predict it directly. They also swap absolute 2D sinusoidal positional encodings with 2D RoPE, which handles new aspect ratios more gracefully.

Qwen-Image

Qwen-Image scales SD3's recipe to ~20B parameters and trains on more text-image pairs with heavier curation. It also takes advantage of Alibaba's ecosystem, encoding prompts with a frozen Qwen2.5-VL (a stronger encoder w/ aligned text-image representations; also lets prompts include images) and compressing images with Wan 2.1's autoencoder. Notably, Qwen-Image adds MSRoPE (better multimodal positional encodings) and further borrows from the LLM playbook with SFT + GRPO post-training.

One of Qwen-Image's main goals is to render all kinds of text well. Along with finetuning the autoencoder on text reconstruction, it trains on lots of programmatically rendered text (paragraphs on clean backgrounds, text composited into real scenes, small text on slides and UI templates, etc) and follows a curriculum from small to large images and from no text to text in natural images to full paragraphs. It's also good at rendering Chinese (non-Latin script with thousands of characters), which is insanely cool!! They also finetune Qwen-Image into Qwen-Image-Edit, which produces a final edited image from a source image and edit instruction. Qwen-Image-Edit is immensely popular, and has thousands of finetunes (relighting, camera-angle changes, style transfer, object removal, virtual try-on, etc).

Qwen-Image 2.0

Qwen-Image is a really good model, but its capabilities are scattered across checkpoints (generate, edit, multi-image, layers, etc). Qwen-Image 2.0 is an RF MMDiT with a stronger, lopsided r=16r=16⁠, c=64c=64 autoencoder (79M-parameter encoder, 259M decoder) and a Qwen3-VL text encoder. It builds on the Qwen-Image-Edit recipe (input images go through the VLM and in as VAE latents), but trains on both editing and generation, across many more editing task types, throughout pretraining and post-training.

Qwen-Image 2.0 is post-trained with SFT then GRPO against five reward models (aesthetics, alignment, portraits, instruction following, edit consistency). They also train a Qwen3.5-9B prompt enhancer based on the images it leads to (GRPO), and build a data flywheel to route user feedback from qwen.ai to RL, pretraining, post-training, or the prompt enhancer.

Closing notes

It's interesting how the core idea behind diffusion, narrowing p(x0∣x)p(x_0\mid x) until the L2 mode average is a sharp image, has endured. It's also cool how a lot of 'machine learning math' is focused much more on 'interpretation' than proofs or derivations for their own sake.

Sorry I didn't add many images; one might say I'm Hegel-maxxing (the 'interpretations' from AP Calc/AP Stats are actually quite useful!). More seriously, I think I tend to conceptualize in words, and I think most things aren't hard to understand when established properly. I've met many people who think this, but only a few who can actually grasp the anima of their readers (the inner learner not the feminine side if uknow what I mean).

You should read the code.

I worry I must warn you (the young machine learning researcher presumably reading this): architectural details and complex derivations are fascinating, but machine learning is almost entirely about 1. thinking very hard about your learning objective and its subtle implications, and 2. relentlessly cleaning, curating, understanding, and interpreting your data. In my eyes, training neural networks is much more engineering than 'science'-- and when engineering, the utmost priority is making the simplest thing that works. To me, good research reads almost like a mathematical proof: you start with the simplest baseline (linear/logistic regression), prove fundamental properties of your problem (axioms!), try a solution built around those properties, diagnose underlying failure modes, try fixes (eg, xxx -> oh hey, someone else found L1 empirically solves this exact problem with L2!), and so on. Nowadays, I get much more excited by simple, stupid ways to solve hard problems (like using logreg, decision trees, and OpenCV to interpret brain signals!) than I do by complex architectures and elaborate methods. IMO, when researching, it's important to never stray too far from what's 'known'; if your chain of logic is sound, the 'new' parts will fall into place:)

I think it's worth viewing diffusion's evolution in the context of compute available at each paper's time. Today, with massive language models and NVL72s to throw around willy nilly, it feels very obvious, for instance, that early diffusion was bottlenecked by language understanding; with V100s and language models nothing like today's, this wasn't obvious at all, and even if it had been, very little could've been done about it (apart from waiting). I wonder what we're bottlenecked by today without realizing it, whether by compute, data, language, attention span, or any of the countless constraints life, society, and our worldly infrastructure impose on us.

What's left/what's next (with image generation):

  • There's an extremely recognizable 'AI slop' aesthetic in AI images/writing. I think this is 1. a model ×\times society thing (anything that can be mass-generated will have a 'slop' middle, even if the long tails are really cool) and 2. an eval problem (where we don't have metrics that track human preference well). Midjourney mostly avoids this slop problem thanks to years of data curation and edge exploration, though there's also some selection bias here (MJ users are artistic, write unusually detailed prompts, etc). We should figure out how to push probability mass to the edges. MiniMax H3 Image (which just released) looks pretty good too.
  • Images don't look obviously wrong anymore, but they're still often subtly wrong: for example, diagrams might not make sense, vanishing points might not converge, maps might make up geography, etc. This 'reasoning gap' is, to me, the final leg of realism (using VLMs/prompt upsamplers only partially resolves the problem).
  • It's interesting how well diffusion works for continuous, spatial domains; it's also the backbone for audio and video generation (which then morphs into worldsim, VLA action heads, etc).
  • Something I'm realizing post-writing this post is how "not dead" autoregression is/how "cooked" diffusion is. I've mostly believed that diffusion is a cool intellectual toy, but complex and hard to adapt to tasks other than image/state generation, and increasingly it looks like the frontier (Reve, Nano Banana 1/2, GPT-Image-2, etc) is moving toward autoregressive planning (using diffusion only to render pixels). Diffusion's solution to mode averaging (committing in pieces) is suboptimal by construction, and there's very little image data at or above 4K resolution. Especially with the release of Fable 5.2, I'm beginning to seriously consider the possibility that diffusion is just a blip in time and that the big model writing code will just win.

I am #unemployed, so if you find my analysis interesting or cool or abhorrent feel free to reach out! I'm also looking for ways to 1. hear about/do cool things before and 2. meet cool people:)

Notation

SymbolMeaning
x0∈Rdx_0\in\mathbb{R}^dclean image; after Latent Diffusion, a latent
x0′x_0'another clean image, for comparison in the DDPM Gaussian-mixture argument
x^0\hat x_0predicted or generated clean image; after Latent Diffusion, a latent
dd⁠, HH⁠, WWimage dimensionality, height, width; d=3HWd=3HW for RGB pixels
θ\thetalearned model parameters (weights)
E[⋅]\mathbb{E}[\cdot]⁠, E[⋅∣⋅]\mathbb{E}[\cdot\mid\cdot]expectation, conditional expectation
∇x\nabla_xgradient with respect to xx
∂\partialpartial derivative
log⁡\lognatural logarithm
≈\approx⁠, ∝\proptoapproximate equality, proportionality
yycondition: sketch, class label, or text prompt
∅\varnothingnull condition (unconditional)
pp⁠, pθp_\thetadata distribution, model distribution
fθf_\thetadeterministic regression model
GG⁠, DD⁠, z∈Rkz\in\mathbb{R}^kGAN generator, discriminator, noise vector
kkGAN noise-vector dimensionality
LL⁠, KKtokens per image, token values / learned patch types (AR)
ε\varepsilonGaussian noise pattern, N(0,I)\mathcal{N}(0,I)
N(0,I)\mathcal{N}(0,I)⁠, IIstandard multivariate Gaussian distribution, identity covariance matrix
σ\sigmanoise level
σmin⁡\sigma_{\min}⁠, σmax⁡\sigma_{\max}minimum, maximum noise level
x=x0+σεx=x_0+\sigma\varepsilonnoisy image
αˉ\bar\alphaDDPM's variance-preserving coefficient, σ=(1−αˉ)/αˉ\sigma=\sqrt{(1-\bar\alpha)/\bar\alpha}
p(x)p(x)⁠, p(xi)p(x_i)noisy image distributions at the associated noise levels σ\sigma⁠, σi\sigma_i
p(x0∣x)p(x_0\mid x)distribution of clean images consistent with xx
p(xi−1∣xi)p(x_{i-1}\mid x_i)reverse-step distribution
p(xi−1∣xi,x0)p(x_{i-1}\mid x_i,x_0)reverse-step distribution given the underlying clean image
p(xi∣y)p(x_i\mid y)⁠, p(y∣xi)p(y\mid x_i)noisy image distribution conditioned on yy⁠, probability of yy given the noisy image
p(y)p(y)prior probability of condition yy
p(x0∣xi,y)p(x_0\mid x_i,y)distribution of clean images consistent with xix_i and yy
εθ(x,σ)\varepsilon_\theta(x,\sigma)noise-prediction network (U-Net or DiT)
εθ(x,σ,y)\varepsilon_\theta(x,\sigma,y)⁠, εθ(x,σ,∅)\varepsilon_\theta(x,\sigma,\varnothing)conditional and unconditional noise predictions from the same network
aa⁠, nncontrast and area of an image feature
xix_i⁠, σi\sigma_iimage and noise level at sampling step ii (counting down to x0x_0⁠)
iisampling-step index
Δσi=σi−σi−1\Delta\sigma_i=\sigma_i-\sigma_{i-1}step size
μ~i\tilde\mu_i⁠, σ~i\tilde\sigma_imean, standard deviation of p(xi−1∣xi,x0)p(x_{i-1}\mid x_i,x_0)
ξ\xifresh Gaussian noise added during DDPM sampling
x(σ)x(\sigma)⁠, x(t)x(t)denoising trajectories as functions of noise level or RF time (ODE solutions)
x′x'⁠, x′′x''first, second derivative of x(σ)x(\sigma) with respect to σ\sigma
wwguidance strength (00 unconditional, 11 conditional, >1>1 amplified)
εcond\varepsilon_{\text{cond}}⁠, εguided\varepsilon_{\text{guided}}class-conditional noise prediction, guided noise prediction
rr⁠, ccautoencoder downsampling factor, latent channels
MM⁠, BBactivation memory, batch size
xi−1trialx_{i-1}^{\text{trial}}Heun's Euler trial point
σi−12\sigma_{i-\frac12}⁠, xi−12x_{i-\frac12}DPM-Solver's log-midpoint noise level and image
ω\omegaspatial frequency
ρ\rhoEDM schedule parameter
ttRF time, x=(1−t)x0+tεx=(1-t)x_0+t\varepsilon
vv⁠, vθ(x,t)v_\theta(x,t)velocity ε−x0\varepsilon-x_0⁠, velocity-prediction network
vguidedv_{\text{guided}}CFG-guided velocity

PSET

As I alluded to earlier, I've been spending lots of time nowadays learning to think very mathematically. As such, I've been trying to make GPT generate good questions that'll help me internalize the things I read. Here are a few (optional) questions (new symbols defined within each question):

  1. What the exact L2 minimizer does on a finite training set. Suppose εθ\varepsilon_\theta reaches the exact global minimum of the DDPM L2 objective on a finite training set {x0(1),…,x0(N)}⊂Rd\{x_0^{(1)},\dots,x_0^{(N)}\}\subset\mathbb{R}^d⁠, treated as the empirical distribution.
    • (a) Derive E[x0∣x]\mathbb{E}[x_0\mid x] at noise level σ\sigma in closed form as a function of xx⁠, σ\sigma⁠, and the training images, and write the DDIM ODE dxdσ\frac{dx}{d\sigma} it induces.
    • (b) Predict what fraction of DDIM samples (starting from σmax⁡ε\sigma_{\max}\varepsilon⁠, integrated exactly to σ=0\sigma=0⁠) are not training images, and how that fraction changes as NN grows with the model still at the exact optimum.
    • (c) For N=2N=2 with xx at squared distance gg closer to x0(1)x_0^{(1)} than to x0(2)x_0^{(2)}⁠, find the σ\sigma below which x0(1)x_0^{(1)} gets posterior weight at least 1−δ1-\delta⁠.
    • (d) Real large-NN diffusion models do produce novel images. Name the exact link in the post's chain from "L2 regression" to "sharp novel x^0\hat x_0" that must be false for them, and say what quantity you would measure on a trained network to confirm it.
  2. What distribution CFG actually samples. Let p(x0)=N(0,1)p(x_0)=\mathcal{N}(0,1) and p(x0∣y)=N(0,τ2)p(x_0\mid y)=\mathcal{N}(0,\tau^2) with τ<1\tau<1⁠, in the variance-exploding form x=x0+σεx=x_0+\sigma\varepsilon⁠, with both εθ\varepsilon_\thetas given exactly by Tweedie's formula and plugged into εguided=εθ(x,σ,∅)+w(εθ(x,σ,y)−εθ(x,σ,∅))\varepsilon_{\text{guided}}=\varepsilon_\theta(x,\sigma,\varnothing)+w\big(\varepsilon_\theta(x,\sigma,y)-\varepsilon_\theta(x,\sigma,\varnothing)\big)⁠.
    • (a) Show that as σ→0\sigma\to0⁠, εguided\varepsilon_{\text{guided}} matches Tweedie's formula for the tilted distribution q∝p(x0∣y)wp(x0)1−wq\propto p(x_0\mid y)^wp(x_0)^{1-w} (ie, εguided/σ→−∇xlog⁡q(x)\varepsilon_{\text{guided}}/\sigma\to-\nabla_x\log q(x)⁠), and give Var⁡q\operatorname{Var}_q⁠.
    • (b) For τ2=14\tau^2=\tfrac14⁠, w=3w=3⁠, σ=1\sigma=1⁠, find the variance VV for which Tweedie's formula on N(0,V)\mathcal{N}(0,V) gives εguided\varepsilon_{\text{guided}}⁠, compute the variance of qq noised to σ=1\sigma=1⁠, and show that no V0V_0 makes V=V0+σ2V=V_0+\sigma^2 for all σ\sigma⁠.
    • (c) Integrate the guided DDIM ODE exactly from σmax⁡→∞\sigma_{\max}\to\infty to 00 and give the variance of the final samples as a closed-form function of τ\tau and ww⁠. Evaluate it at τ2=14\tau^2=\tfrac14⁠, w=3w=3 and order it against Var⁡q\operatorname{Var}_q and τ2\tau^2⁠.
    • (d) Name the exact step in the post's ADM->CFG derivation that holds at w=1w=1 and fails at w≠1w\neq1⁠. Predict whether more sampling steps or a higher-order solver shrinks the gap from (c).
  3. Straight paths versus optimal couplings. Take Rectified Flow with independent coupling in 1D, x0∼N(0,λ)x_0\sim\mathcal{N}(0,\lambda) and x=(1−t)x0+tεx=(1-t)x_0+t\varepsilon⁠.
    • (a) Derive the exact v(x,t)=E[ε−x0∣x]v(x,t)=\mathbb{E}[\varepsilon-x_0\mid x] at time tt and solve the ODE to get the map ε↦x^0\varepsilon\mapsto\hat x_0⁠. Prove it equals the Brenier (squared-cost optimal transport) map between N(0,1)\mathcal{N}(0,1) and N(0,λ)\mathcal{N}(0,\lambda)⁠.
    • (b) Show the trajectories x(t)x(t) are curved for every λ>0\lambda>0⁠, including λ=1\lambda=1⁠, where the map from (a) is the identity. At λ=4\lambda=4⁠, compute E∥ε−x0∥2\mathbb{E}\|\varepsilon-x_0\|^2 under the training coupling and W22W_2^2⁠. Using Benamou–Brenier, state whether the learned flow's kinetic energy ∫01E∥v∥2dt\int_0^1\mathbb{E}\|v\|^2dt equals, exceeds, or falls below W22W_2^2⁠, and whether equality holds for any λ>0\lambda>0⁠.
    • (c) Extend (a) to x0∼N(0,Σ)x_0\sim\mathcal{N}(0,\Sigma) in Rd\mathbb{R}^d and state exactly which structural property makes the map optimal.
    • (d) The post argues reflow straightens paths because generated trajectories can't cross (Picard–Lindelöf). Decide whether every coupling that is a fixed point of reflow is an optimal transport coupling. Prove it, or give an explicit 2D Gaussian counterexample with both transport costs computed. Then say which property of 1D makes "non-crossing" sufficient there and what the d≥2d\ge2 optimality condition needs beyond it.
  4. Why SD1 can't make a black image. Stable Diffusion 1.x trains on 64×64×464\times64\times4 latents up to a terminal noise level of σmax⁡≈14.6\sigma_{\max}\approx14.6 (its schedule ends at αˉ≈0.0047\bar\alpha\approx0.0047⁠), but starts sampling from pure noise. Observed: prompts like "a pitch-black room" or "a white void" reliably come out near mid-brightness, while local content follows the prompt fine.
    • (a) Name the assumption about the training distribution at the terminal noise level that must have failed.
    • (b) Apply the post's visibility criterion (signal an≳an\gtrsim noise σn\sigma\sqrt{n}⁠) at σmax⁡\sigma_{\max}⁠. For a per-channel spatial-mean deviation of a=0.5a=0.5 over the n=4096n=4096 positions of one channel, compute signal and noise at the terminal noise level, and do the same for a single-position feature (n=1n=1⁠).
    • (c) Find the smallest square region side length, in latent pixels, at which a contrast-0.50.5 feature is visible at the terminal noise level. Use that to explain why the failure shows up specifically as global brightness rather than as wrong textures or colors.
    • (d) For each fix, predict whether it removes the effect and give the mechanism in one line: more sampling steps; a higher-order solver; rescaling the schedule so terminal αˉ=0\bar\alpha=0 (with the prediction target changed so the pure-noise endpoint is well posed); adding a per-channel constant offset to ε\varepsilon during training only.
  5. What the loss plateau measures. Guo, Shamai, and Verdú (2005) proved that for Y=snrX+ZY=\sqrt{\mathrm{snr}}X+Z with standard Gaussian noise ZZ⁠, ddsnrI(X;Y)=12mmse(snr)\frac{d}{d\mathrm{snr}}I(X;Y)=\tfrac12\mathrm{mmse}(\mathrm{snr})⁠, where I(X;Y)I(X;Y) is mutual information and mmse=E∥X−E[X∣Y]∥2\mathrm{mmse}=\mathbb{E}\|X-\mathbb{E}[X\mid Y]\|^2⁠.
    • (a) Map x=x0+σεx=x_0+\sigma\varepsilon onto this channel. Then show that I(x0;xσlo)−I(x0;xσhi)=∫σloσhimmseε(σ)dlog⁡σI(x_0;x_{\sigma_{\text{lo}}})-I(x_0;x_{\sigma_{\text{hi}}})=\int_{\sigma_{\text{lo}}}^{\sigma_{\text{hi}}}\mathrm{mmse}_\varepsilon(\sigma)d\log\sigma⁠, where mmseε\mathrm{mmse}_\varepsilon is the optimal ε\varepsilon-prediction L2 loss.
    • (b) Check both sides exactly for x0∼N(0,s2)x_0\sim\mathcal{N}(0,s^2) in 1D.
    • (c) For x0x_0 uniform over NN well-separated images, evaluate ∫0∞mmseεdlog⁡σ\int_0^\infty\mathrm{mmse}_\varepsilon d\log\sigma⁠, and predict lim⁡σ→0mmseε\lim_{\sigma\to0}\mathrm{mmse}_\varepsilon⁠.
    • (d) For x0x_0 supported on a smooth mm-dimensional manifold in Rd\mathbb{R}^d⁠, predict lim⁡σ→0mmseε\lim_{\sigma\to0}\mathrm{mmse}_\varepsilon⁠. Then give the number of bits revealed per halving of σ\sigma in that limit, and state what a trained model's small-σ\sigma loss plateau measures.
  6. Where Picard–Lindelöf stops applying. Consider the claim that the exact DDIM flow map from σmax⁡\sigma_{\max} to 00 is a diffeomorphism, so DDIM inversion is well posed.
    • (a) For x0∼N(0,1)x_0\sim\mathcal{N}(0,1) in 1D with σmax⁡=10\sigma_{\max}=10⁠, compute the Jacobian ∂x(0)/∂x(σmax⁡)\partial x(0)/\partial x(\sigma_{\max}) exactly, and show that the flow map is a diffeomorphism.
    • (b) For two-point data x0∈{−1,+1}x_0\in\{-1,+1\} with equal weights, write εθ(x,σ)\varepsilon_\theta(x,\sigma) in closed form. Compute ∣∂εθ/∂x∣|\partial\varepsilon_\theta/\partial x| at x=0x=0 as a function of σ\sigma⁠, and show that its integral over [0,σmax⁡][0,\sigma_{\max}] diverges.
    • (c) Along the separatrix x=0x=0⁠, compute the factor by which an infinitesimal perturbation grows between σ=10\sigma=10 and σ=0.25\sigma=0.25⁠, to three significant figures. Then compute the Jacobian in the limit σ→0\sigma\to0 for a trajectory inside the +1+1 basin.
    • (d) State which hypothesis of Picard–Lindelöf fails here and at which endpoint. State what set of starting noises "the σmax⁡ε\sigma_{\max}\varepsilon that generates x0=+1x_0=+1" actually is.