Introduction
To some extent, we've 'solved' image generation (ie, specify any number of bits about an image, and they'll be rendered ~exactly, with the rest filled in plausibly). For a while I've wanted to know how image generation got so good, so I spent a few days talking to GPT and wrote up what I learned:)
Why is generating images hard?
'Generating images' willy nilly is a much harder problem than it sounds, mainly because there are so many possibilities. Imagine we have a dataset of full-detail face images ( for an RGB image), each paired with a blocked-in sketch of the same face (rough head shape, eye/nose placement, hairstyle, etc). Given paired data and a continuous codomain, the obvious approach is regression: train a deterministic model with L2 (MSE) to map each to a single .
However, this model will almost always produce blurry, washed-out faces, no matter how big your model or dataset. Through an information theory lens, specifies some bits about , but not enough to fully pin down a single (imagine filling in to get some , then making an eyelash longer, adding stubble, or changing the skin texture entirely). Each defines a multimodal probability distribution over image space (similar images are close together in image space, so probability mass clusters into 'modes' around each plausible ), and the that produced is a single sample from it. Across a large dataset, similar s are paired with vastly different s, but neural networks like are Lipschitz continuous by construction (or at least locally), so similar s must produce similar s. So how does resolve conflicting s? L2 implicitly treats as a single bell curve (minimizing L2 = maximizing likelihood under a fixed-variance Gaussian centered at ), and the best fixed-variance bell curve for any distribution is centered at its mean, meaning learns to mode average, or predict the probability-weighted mean over all modes: .
To generate a high-quality image, must commit to a single plausible (equivalently, a single mode of ), but the hand (multimodal ) just doesn't fit the glove (single fixed-variance Gaussian L2). What if we change the learning objective?
A GAN concurrently trains a generator to turn -dimensional noise vectors into images , and a discriminator to estimate the probability that an image is real. An ideal pushes to generate diverse s that fall into any mode of . Since 's only source of variety is , it learns to assign 'meaning' to directions in ; at generation time, sampling fully determines , and is essentially a renderer. Unfortunately, the adversarial objective is unstable (a constant tug of war) and unanchored (loss values say nothing about quality and are incomparable across runs), so often mode-collapses and doesn't scale predictably.
Over a discrete set of outputs, a model with a softmax head creates a distribution , and the expected cross-entropy is smallest when matches the true distribution . However, this requires reframing our regression problem as classification: represent as a sequence of one-hot tokens over values (eg, a flattened discrete autoencoder bottleneck, where token position image position, token value which of learned patch types the region looks like) and train a model to generate new sequences via autoregressive NTP. For rendering pixels diffusion simply outperforms: AR images can only be as detailed as their patch types, require at least model calls each (with resolution), and must follow an arbitrary top-left-to-bottom-right raster order. That said, autoregressive models (LLMs) have proven extremely useful for the 'text' parts of 'text-to-image'.
To directly answer our guiding question: image generation got good by 1. finding a workable inductive bias (diffusion), and 2. continuously pulling high-leverage levers within it (sampling, scaling, conditioning). Rather than fighting L2, diffusion uses a few clever tricks to narrow the modes of until is a sharp, specific, high-quality . The rest of this post is about how it works, and how it went from good to great.
DDPM
Diffusion Probabilistic Models (DPMs), proposed in 2015, model by iteratively adding noise to images and learning to reverse the process. Denoising DPMs (DDPMs) clean up their objective and make them competitive with GANs.
A noisy image can be written as , with clean image , Gaussian noise , and noise level . DDPM technically uses the variance-preserving form of the same equation, , but dividing by gives the equivalent, easier-to-interpret variance-exploding form (). Like our blocked-in sketches, defines a multimodal distribution : an image feature of contrast and area only stays visible if signal noise (ie, as goes from to , more and more bits of become specified).
DDPM has two parts: a U-Net trained on L2 to predict , and a sampler that denoises into a specific over many steps, each decreasing by . A single step from to gives the dataset average, , since , whereas committing to a single at each step iteratively narrows .
So how do we sample from ? By Gaussian conjugacy, is Gaussian with mean and standard deviation . By the law of total probability, , and a weighted average of Gaussians sharing is approximately a Gaussian when . For small steps (), and , so is approximately Gaussian!
Since is linear in , linearity of expectation gives the mean of as . Mode averaging gives us , so with the reparametrization trick we can sample , where . Expanding and simplifying, we get the final DDPM update rule:
DDIM
To sample from , DDPM is forced to take small, noisy steps (essentially a random walk down ). Instead, DDIM deterministically computes , generating comparable images in 10–50 steps instead of ~1,000.
Differentiating wrt (holding and fixed) gives . Substituting DDPM's gives the ODE a velocity field over image space, which we can follow using Euler's method: As , moves along a denoising trajectory toward (aka: a particular solution to the ODE on with initial condition , an integral curve of , or a flow line of ). Empirically, s are also gently curved, allowing Euler's method to take large s.
For any , is Lipschitz continuous on , so Picard–Lindelöf guarantees that each has a unique trajectory . That means DDIM can step an back up in to recover the that generates it (DDIM inversion), which can be used to reconstruct or semantically interpolate between s. Inverted images drift slightly, though, because inversion is only exact in the continuous limit, Euler steps add error, and Lipschitz continuity breaks down at .
Other later works (Consistency Models) also take advantage of Picard–Lindelöf by training every point on to predict its endpoint . Consistency Models work particularly well for distilling diffusion models into few-step samplers.
ADM
ADM ablates the DDPM/DDIM recipe, finding significant gains from scaling (adding self-attention layers & heads, residual blocks, and AdaGN). ADM also introduces Classifier Guidance, which allows steering denoising toward a chosen class (unconditional diffusion chooses from ImageNet's ~1,000 classes at random).
First, a classifier (a randomly initialized clone of the downsampling half of plus an output head) is trained to predict from , then its gradient is computed by backpropagating to (rather than to classifier weights). During each DDIM step, this gradient is scaled by and a strength constant , then subtracted from : Geometrically, Classifier Guidance superimposes the classifier's gradient field onto at generation time. Functionally, is narrower than .
One question I had was, why do we scale by ? Classifier Guidance's goal is to find a class-conditional version of Tweedie's formula, , but we don't know . Taking the of Bayes' theorem gives . Zeroing out (since doesn't depend on ), multiplying by , substituting , and scaling the classifier term by , we get:
One objection you might have is, why not just feed a class embedding into ? ADM also does this! Ranked by FID (measuring quality and diversity): embedding + CG () > embedding + CG () > embedding alone > CG () > CG (). Notably, embedding + CG () produces the highest-quality images at some cost to diversity, while embedding provides richer signal at fixed, insufficient strength.
Classifier-Free Guidance
Classifier-Free Guidance's goal is to compute from the embedding signal. To do this, CFG trains one to denoise with and without . Rearranging our Bayes identity from ADM, we get (ie, ). By Tweedie's formula, and ; scaling the modified Bayes identity by and substituting, we get
Substituting into from ADM:
Latent Diffusion
Up to this point, the diffusion recipe has been driven by clever theoretical reformulations, but from Latent Diffusion onwards, progress mostly comes from subtle changes and extreme attention to detail. I'll focus on the headline changes, but be aware that exact subtleties are extremely important from here on out. If you want to deeply understand what makes diffusion work, please please please read the papers/code (and check the PSET at the end 🙂).
One way to improve image quality is to increase resolution, but we'd quickly become memory-bound. The first convolution in expands a image to 256 channels, producing a ~34 MB feature map (fp16). Backprop stores many intermediate activations, totaling ~1.5 GB per image, so training at batch size 256 would need ~384 GB of VRAM (ADM takes 22,000 V100-hours to train with data parallelism, ~2 months on 16 GPUs). Activation memory scales as , so needs more VRAM (~1.5 TB, ~8 months), needs more (~6 TB, ~2.7 years), and so on.
Latent Diffusion trains an autoencoder (essentially a lightly regularized VAE) to downsample each image by a factor into a latent with channels, usually , ("f8c4"): , around fewer values for ~the same image. Then, diffusion is run in the autoencoder's latent space, and only the final is decoded back to pixels. The idea that raw images are highly compressible is already well known (manifold hypothesis, JPEG, etc); Latent Diffusion's contribution is finding a recipe that works: too little downsampling () barely saves compute and too much () loses detail diffusion can't recover (– being the sweet spot). Training the autoencoder and diffusion model separately also keeps training simpler and more stable, and lets one autoencoder serve many s.
Latent Diffusion also adds cross-attention, letting each image region attend to specific text tokens instead of one shared conditioning vector.
Imagen
Image generation is getting good, but models still struggle to follow instructions (objects go missing, attributes mix, etc) and render subtle details. Part of the problem is that models are weak (eg, Latent Diffusion with small physically can't represent fine letters or numbers), but Imagen's main finding is that scaling the text encoder pays off far more than scaling .
This makes sense: prompts can be nuanced, and small encoders trained jointly with diffusion can't make "a red cube on a blue cube" and "a blue cube on a red cube" meaningfully different embeddings. Instead, Imagen uses a frozen T5 (an early text-to-text LM) to enrich the text tokens that cross-attends to. This sets a nice precedent: as LLMs improve, image models can ride on their success! Imagen also tweaks the pixel-space recipe (mostly irrelevant once Latent Diffusion takes over).
EDM / DPM-Solver
If generating an image is ~solving an ODE, using better solvers has high leverage! EDM and DPM-Solver both use higher-order solvers to get more accuracy per call.
Taylor-expanding over one step, we get . Euler's method only keeps the first two terms, implicitly assuming is constant over a step. EDM uses Heun's method, which captures one more Taylor term by assuming changes linearly: take an Euler step to , then step from using the average slope,
Geometrically, on a steadily curving path (constant ), the slopes at the start and end of a step miss the direction to the true next point by in opposite directions, so averaging them cancels the error term. Heun's per-step error scales as instead of Euler's , at the cost of two calls per step (four with CFG).
DPM-Solver captures the same extra term by assuming changes linearly in (every halving of changes by about the same amount). Heuristically, this makes sense: natural images have power spectra that roughly follow (spatial frequency ), so each halving of uncovers another octave of frequencies, and each octave carries a similar amount of image content. Under the linear-in- assumption, the prediction at the step's log-midpoint is nearly the step's average. So DPM-Solver takes an Euler step to and uses that midpoint prediction for the whole step:
EDM also 1. samples training s from a log-normal distribution, focusing 's training on mid-level s where it actually commits features, 2. introduces a noise schedule parametrized by , which controls how strongly steps bunch at low s (: evenly spaced in ; : evenly spaced in ), and 3. adds preconditioning, which uses -dependent scalings and a skip connection around to keep its inputs and outputs similarly scaled at any .
Stable Diffusion
Stable Diffusion, a ~860M-parameter trained on images, conditioned using a frozen CLIP-L text encoder, essentially just scales up Latent Diffusion. Its significance mostly comes from being open-weights and running on consumer GPUs, which creates a massive ecosystem of samplers, finetunes, and tools. One of the most popular samplers is DPM-Solver++ 2M on EDM's noise schedule at ("Karras").
Rectified Flow / Flow Matching
Instead of using better solvers, Rectified Flow constructs a better ODE. RF reparametrizes the noise corruption formula into , where . Both describe the same path (up to rescaling, with ), but RF changes the prediction target: differentiating wrt (holding and fixed) gives so each pair moves at a constant velocity . Training to predict gives the ODE with denoising trajectory . Euler's method and CFG apply as before.
Predicting also avoids a few problems with predicting . In DDPM's variance-preserving form, reconstructing amplifies the model's mistakes by (about at the noisiest step, where ) and is impossible at pure noise (). This also creates a subtle train-test mismatch: its noisiest training step keeps ~0.6% of , while generation starts from 0%. Flow Matching generally shows that a 'diffusion model' can be trained on any path from to , but RF's path is the most common. (FM/RF also have Jacobian interpretations, which I skip.)
One reason I find machine learning so fascinating is that you can just impose structure and train away constraints you'd have classically been limited by. RF also introduces reflow, an optional step to make s straighter: train as usual, generate pairs by sampling and running the ODE, then keep training on those pairs. Randomly paired s give each many possible s, which leads to mode averaging and bent trajectories; generated trajectories can't cross (Picard–Lindelöf), so each maps to exactly one , and training on straight lines between these pairs produces straighter s. Reflow can be repeated (with diminishing returns) and produces usable images in 1–2 steps :o!
DiT
Following ADM and Latent Diffusion, U-Nets have increasingly become transformer-like (50–75% of SD1's U-Net is transformer blocks). A Diffusion Transformer drops the U-Net entirely and uses a ViT-style transformer as . It patchifies the latent into a sequence of patches, linearly embeds and adds a positional encoding to each one, runs the sequence through standard transformer blocks, and assembles . Each DiT block is conditioned on and an ImageNet class label via adaLN-Zero (no text prompts to keep the architecture study simple).
DiTs scale smoothly and predictably with forward-pass compute (GFLOPs). U-Nets generally improve with scale, but lack predictability (partly because they have many axes to scale: channels, blocks, attention placement, etc, whereas a transformer has depth, width, and token count). Using transformers also lets us take advantage of the rapidly advancing LLM training stack (FlashAttention, FSDP, kernels, mixed precision, etc).
DALL·E 3
DALL·E 1 and 2 focus on autoregression and diffusion respectively, but DALL·E 3 is all about data! Most text-to-image datasets pair web images with short, vague, and incomplete scraped alt-text, which (besides affecting instruction following) does little to narrow relative to .
DALL·E 3's authors finetune a captioner (a language model conditioned on a CLIP image embedding) on a small set of ultra-detailed captions, use it to recaption their dataset, and train a diffusion model on a mix of 95% synthetic and 5% original alt-text. At generation time, GPT-4 "upsamples" the user's prompt into the synthetic captions' style and detail.
It's extremely cool to me how new models unlock new capabilities (eg, forming an image by describing it with words, splitting a song into its audio tracks, etc), which create more data, which creates even more new capabilities, and so on! I looooove synthetic data sooooooo much it's so cool!
SD3
SD3 is another massive ablation paper. Its best model is an 8B MMDiT trained with RF (no reflow) on the latents of an , autoencoder. Notably, an MMDiT conditions on text using joint attention (text and image tokens are concatenated into one token sequence but keep separate weights), which improves prompt adherence, attribute binding, and text rendering since information flows bidirectionally (text image). Prompts go through three frozen encoders (CLIP-L and OpenCLIP-bigG for image semantics, T5-XXL for language understanding/text rendering); pooled CLIP embeddings plus a embedding condition every block via adaLN, and all three encoders' token embeddings are combined to form text tokens.
SD3 is trained on a 50/50 mix of synthetically labeled captions and original alt-text, with drawn from a logit-normal distribution (EDM's log-normal sampling, adapted to RF), starting at and going up to . SD3 also introduces post-training with Diffusion-DPO (another trick from the LLM playbook!). SD3 samples with plain Euler + CFG, and its validation loss falls predictably with size while tracking human preference!
Many of SD3's authors went on to found Black Forest Labs, whose FLUX models mostly continue scaling SD3 and curating data. One cool idea they popularize is guidance distillation: CFG's two passes only produce , a fixed function of , so a student can be trained to predict it directly. They also swap absolute 2D sinusoidal positional encodings with 2D RoPE, which handles new aspect ratios more gracefully.
Qwen-Image
Qwen-Image scales SD3's recipe to ~20B parameters and trains on more text-image pairs with heavier curation. It also takes advantage of Alibaba's ecosystem, encoding prompts with a frozen Qwen2.5-VL (a stronger encoder w/ aligned text-image representations; also lets prompts include images) and compressing images with Wan 2.1's autoencoder. Notably, Qwen-Image adds MSRoPE (better multimodal positional encodings) and further borrows from the LLM playbook with SFT + GRPO post-training.
One of Qwen-Image's main goals is to render all kinds of text well. Along with finetuning the autoencoder on text reconstruction, it trains on lots of programmatically rendered text (paragraphs on clean backgrounds, text composited into real scenes, small text on slides and UI templates, etc) and follows a curriculum from small to large images and from no text to text in natural images to full paragraphs. It's also good at rendering Chinese (non-Latin script with thousands of characters), which is insanely cool!! They also finetune Qwen-Image into Qwen-Image-Edit, which produces a final edited image from a source image and edit instruction. Qwen-Image-Edit is immensely popular, and has thousands of finetunes (relighting, camera-angle changes, style transfer, object removal, virtual try-on, etc).
Qwen-Image 2.0
Qwen-Image is a really good model, but its capabilities are scattered across checkpoints (generate, edit, multi-image, layers, etc). Qwen-Image 2.0 is an RF MMDiT with a stronger, lopsided , autoencoder (79M-parameter encoder, 259M decoder) and a Qwen3-VL text encoder. It builds on the Qwen-Image-Edit recipe (input images go through the VLM and in as VAE latents), but trains on both editing and generation, across many more editing task types, throughout pretraining and post-training.
Qwen-Image 2.0 is post-trained with SFT then GRPO against five reward models (aesthetics, alignment, portraits, instruction following, edit consistency). They also train a Qwen3.5-9B prompt enhancer based on the images it leads to (GRPO), and build a data flywheel to route user feedback from qwen.ai to RL, pretraining, post-training, or the prompt enhancer.
Closing notes
It's interesting how the core idea behind diffusion, narrowing until the L2 mode average is a sharp image, has endured. It's also cool how a lot of 'machine learning math' is focused much more on 'interpretation' than proofs or derivations for their own sake.
Sorry I didn't add many images; one might say I'm Hegel-maxxing (the 'interpretations' from AP Calc/AP Stats are actually quite useful!). More seriously, I think I tend to conceptualize in words, and I think most things aren't hard to understand when established properly. I've met many people who think this, but only a few who can actually grasp the anima of their readers (the inner learner not the feminine side if uknow what I mean).
You should read the code.
I worry I must warn you (the young machine learning researcher presumably reading this): architectural details and complex derivations are fascinating, but machine learning is almost entirely about 1. thinking very hard about your learning objective and its subtle implications, and 2. relentlessly cleaning, curating, understanding, and interpreting your data. In my eyes, training neural networks is much more engineering than 'science'-- and when engineering, the utmost priority is making the simplest thing that works. To me, good research reads almost like a mathematical proof: you start with the simplest baseline (linear/logistic regression), prove fundamental properties of your problem (axioms!), try a solution built around those properties, diagnose underlying failure modes, try fixes (eg, xxx -> oh hey, someone else found L1 empirically solves this exact problem with L2!), and so on. Nowadays, I get much more excited by simple, stupid ways to solve hard problems (like using logreg, decision trees, and OpenCV to interpret brain signals!) than I do by complex architectures and elaborate methods. IMO, when researching, it's important to never stray too far from what's 'known'; if your chain of logic is sound, the 'new' parts will fall into place:)
I think it's worth viewing diffusion's evolution in the context of compute available at each paper's time. Today, with massive language models and NVL72s to throw around willy nilly, it feels very obvious, for instance, that early diffusion was bottlenecked by language understanding; with V100s and language models nothing like today's, this wasn't obvious at all, and even if it had been, very little could've been done about it (apart from waiting). I wonder what we're bottlenecked by today without realizing it, whether by compute, data, language, attention span, or any of the countless constraints life, society, and our worldly infrastructure impose on us.
What's left/what's next (with image generation):
- There's an extremely recognizable 'AI slop' aesthetic in AI images/writing. I think this is 1. a model society thing (anything that can be mass-generated will have a 'slop' middle, even if the long tails are really cool) and 2. an eval problem (where we don't have metrics that track human preference well). Midjourney mostly avoids this slop problem thanks to years of data curation and edge exploration, though there's also some selection bias here (MJ users are artistic, write unusually detailed prompts, etc). We should figure out how to push probability mass to the edges. MiniMax H3 Image (which just released) looks pretty good too.
- Images don't look obviously wrong anymore, but they're still often subtly wrong: for example, diagrams might not make sense, vanishing points might not converge, maps might make up geography, etc. This 'reasoning gap' is, to me, the final leg of realism (using VLMs/prompt upsamplers only partially resolves the problem).
- It's interesting how well diffusion works for continuous, spatial domains; it's also the backbone for audio and video generation (which then morphs into worldsim, VLA action heads, etc).
- Something I'm realizing post-writing this post is how "not dead" autoregression is/how "cooked" diffusion is. I've mostly believed that diffusion is a cool intellectual toy, but complex and hard to adapt to tasks other than image/state generation, and increasingly it looks like the frontier (Reve, Nano Banana 1/2, GPT-Image-2, etc) is moving toward autoregressive planning (using diffusion only to render pixels). Diffusion's solution to mode averaging (committing in pieces) is suboptimal by construction, and there's very little image data at or above 4K resolution. Especially with the release of Fable 5.2, I'm beginning to seriously consider the possibility that diffusion is just a blip in time and that the big model writing code will just win.
I am #unemployed, so if you find my analysis interesting or cool or abhorrent feel free to reach out! I'm also looking for ways to 1. hear about/do cool things before and 2. meet cool people:)
Notation
| Symbol | Meaning |
|---|---|
| clean image; after Latent Diffusion, a latent | |
| another clean image, for comparison in the DDPM Gaussian-mixture argument | |
| predicted or generated clean image; after Latent Diffusion, a latent | |
| , , | image dimensionality, height, width; for RGB pixels |
| learned model parameters (weights) | |
| , | expectation, conditional expectation |
| gradient with respect to | |
| partial derivative | |
| natural logarithm | |
| , | approximate equality, proportionality |
| condition: sketch, class label, or text prompt | |
| null condition (unconditional) | |
| , | data distribution, model distribution |
| deterministic regression model | |
| , , | GAN generator, discriminator, noise vector |
| GAN noise-vector dimensionality | |
| , | tokens per image, token values / learned patch types (AR) |
| Gaussian noise pattern, | |
| , | standard multivariate Gaussian distribution, identity covariance matrix |
| noise level | |
| , | minimum, maximum noise level |
| noisy image | |
| DDPM's variance-preserving coefficient, | |
| , | noisy image distributions at the associated noise levels , |
| distribution of clean images consistent with | |
| reverse-step distribution | |
| reverse-step distribution given the underlying clean image | |
| , | noisy image distribution conditioned on , probability of given the noisy image |
| prior probability of condition | |
| distribution of clean images consistent with and | |
| noise-prediction network (U-Net or DiT) | |
| , | conditional and unconditional noise predictions from the same network |
| , | contrast and area of an image feature |
| , | image and noise level at sampling step (counting down to ) |
| sampling-step index | |
| step size | |
| , | mean, standard deviation of |
| fresh Gaussian noise added during DDPM sampling | |
| , | denoising trajectories as functions of noise level or RF time (ODE solutions) |
| , | first, second derivative of with respect to |
| guidance strength ( unconditional, conditional, amplified) | |
| , | class-conditional noise prediction, guided noise prediction |
| , | autoencoder downsampling factor, latent channels |
| , | activation memory, batch size |
| Heun's Euler trial point | |
| , | DPM-Solver's log-midpoint noise level and image |
| spatial frequency | |
| EDM schedule parameter | |
| RF time, | |
| , | velocity , velocity-prediction network |
| CFG-guided velocity |
PSET
As I alluded to earlier, I've been spending lots of time nowadays learning to think very mathematically. As such, I've been trying to make GPT generate good questions that'll help me internalize the things I read. Here are a few (optional) questions (new symbols defined within each question):
- What the exact L2 minimizer does on a finite training set. Suppose reaches the exact global minimum of the DDPM L2 objective on a finite training set , treated as the empirical distribution.
- (a) Derive at noise level in closed form as a function of , , and the training images, and write the DDIM ODE it induces.
- (b) Predict what fraction of DDIM samples (starting from , integrated exactly to ) are not training images, and how that fraction changes as grows with the model still at the exact optimum.
- (c) For with at squared distance closer to than to , find the below which gets posterior weight at least .
- (d) Real large- diffusion models do produce novel images. Name the exact link in the post's chain from "L2 regression" to "sharp novel " that must be false for them, and say what quantity you would measure on a trained network to confirm it.
- What distribution CFG actually samples. Let and with , in the variance-exploding form , with both s given exactly by Tweedie's formula and plugged into .
- (a) Show that as , matches Tweedie's formula for the tilted distribution (ie, ), and give .
- (b) For , , , find the variance for which Tweedie's formula on gives , compute the variance of noised to , and show that no makes for all .
- (c) Integrate the guided DDIM ODE exactly from to and give the variance of the final samples as a closed-form function of and . Evaluate it at , and order it against and .
- (d) Name the exact step in the post's ADM->CFG derivation that holds at and fails at . Predict whether more sampling steps or a higher-order solver shrinks the gap from (c).
- Straight paths versus optimal couplings. Take Rectified Flow with independent coupling in 1D, and .
- (a) Derive the exact at time and solve the ODE to get the map . Prove it equals the Brenier (squared-cost optimal transport) map between and .
- (b) Show the trajectories are curved for every , including , where the map from (a) is the identity. At , compute under the training coupling and . Using Benamou–Brenier, state whether the learned flow's kinetic energy equals, exceeds, or falls below , and whether equality holds for any .
- (c) Extend (a) to in and state exactly which structural property makes the map optimal.
- (d) The post argues reflow straightens paths because generated trajectories can't cross (Picard–Lindelöf). Decide whether every coupling that is a fixed point of reflow is an optimal transport coupling. Prove it, or give an explicit 2D Gaussian counterexample with both transport costs computed. Then say which property of 1D makes "non-crossing" sufficient there and what the optimality condition needs beyond it.
- Why SD1 can't make a black image. Stable Diffusion 1.x trains on latents up to a terminal noise level of (its schedule ends at ), but starts sampling from pure noise. Observed: prompts like "a pitch-black room" or "a white void" reliably come out near mid-brightness, while local content follows the prompt fine.
- (a) Name the assumption about the training distribution at the terminal noise level that must have failed.
- (b) Apply the post's visibility criterion (signal noise ) at . For a per-channel spatial-mean deviation of over the positions of one channel, compute signal and noise at the terminal noise level, and do the same for a single-position feature ().
- (c) Find the smallest square region side length, in latent pixels, at which a contrast- feature is visible at the terminal noise level. Use that to explain why the failure shows up specifically as global brightness rather than as wrong textures or colors.
- (d) For each fix, predict whether it removes the effect and give the mechanism in one line: more sampling steps; a higher-order solver; rescaling the schedule so terminal (with the prediction target changed so the pure-noise endpoint is well posed); adding a per-channel constant offset to during training only.
- What the loss plateau measures. Guo, Shamai, and Verdú (2005) proved that for with standard Gaussian noise , , where is mutual information and .
- (a) Map onto this channel. Then show that , where is the optimal -prediction L2 loss.
- (b) Check both sides exactly for in 1D.
- (c) For uniform over well-separated images, evaluate , and predict .
- (d) For supported on a smooth -dimensional manifold in , predict . Then give the number of bits revealed per halving of in that limit, and state what a trained model's small- loss plateau measures.
- Where Picard–Lindelöf stops applying. Consider the claim that the exact DDIM flow map from to is a diffeomorphism, so DDIM inversion is well posed.
- (a) For in 1D with , compute the Jacobian exactly, and show that the flow map is a diffeomorphism.
- (b) For two-point data with equal weights, write in closed form. Compute at as a function of , and show that its integral over diverges.
- (c) Along the separatrix , compute the factor by which an infinitesimal perturbation grows between and , to three significant figures. Then compute the Jacobian in the limit for a trajectory inside the basin.
- (d) State which hypothesis of Picard–Lindelöf fails here and at which endpoint. State what set of starting noises "the that generates " actually is.