Denoising Diffusion Probabilistic Models

The problem

By 2020 the generative landscape was a menu of compromises. GANs produced sharp images but trained unstably and covered the data distribution poorly, dropping whole modes. Likelihood models — VAEs, normalizing flows, autoregressive pixel models — were stable and covered the distribution but produced blurry or slow samples. Diffusion probabilistic models had been proposed by Sohl-Dickstein et al. in 2015 as a principled middle path, but they had never been made to generate competitively. DDPM is the paper that closed that gap, reaching a FID of 3.17 on CIFAR-10 — better than the strong GANs of its day — and in doing so it kicked off the modern diffusion era.

The forward process

The setup is a fixed, untrainable Markov chain that destroys structure. At each step it scales the current image down slightly and adds a little Gaussian noise, with the amount governed by a variance schedule $\beta_1, \dots, \beta_T$.

$$q(x_t \mid x_{t-1}) = \mathcal{N}\!\big(x_t;\, \sqrt{1 - \beta_t}\, x_{t-1},\; \beta_t I\big)$$

The $\sqrt{1 - \beta_t}$ factor shrinks the signal by exactly enough to keep the variance bounded as noise piles up, so after $T = 1000$ steps the image is indistinguishable from a draw from $\mathcal{N}(0, I)$. The decisive algebraic fact is that the composition of $t$ linear-Gaussian steps is itself Gaussian, so with $\alpha_t = 1 - \beta_t$ and $\bar\alpha_t = \prod_{s=1}^{t} \alpha_s$ the marginal has a closed form.

$$q(x_t \mid x_0) = \mathcal{N}\!\big(x_t;\, \sqrt{\bar\alpha_t}\, x_0,\; (1 - \bar\alpha_t) I\big)$$

This lets training jump directly to any timestep — sample one noise vector and form $x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon$ — instead of simulating the intervening hundreds of steps.

The reverse process and the ELBO

Generation is the reverse chain, which the model must learn because denoising is not free. It is parameterized as a sequence of Gaussians.

$$p_\theta(x_{t-1} \mid x_t) = \mathcal{N}\!\big(x_{t-1};\, \mu_\theta(x_t, t),\; \Sigma_\theta(x_t, t)\big)$$

A Gaussian is the right family because, in the limit of small $\beta_t$, the true reverse conditional is itself approximately Gaussian. Training maximizes the variational lower bound (ELBO) on the data log-likelihood, which decomposes into per-timestep KL terms comparing $p_\theta(x_{t-1} \mid x_t)$ against the forward posterior $q(x_{t-1} \mid x_t, x_0)$. That posterior is tractable — conditioning on $x_0$ turns the otherwise unknown reverse step into a Gaussian you can write down exactly — so each term reduces to matching the network's predicted mean to the posterior mean.

The simplified objective

The paper's signature move is the parameterization that makes this concrete. Rather than predicting the mean or $x_0$ directly, the network predicts the noise $\epsilon_\theta(x_t, t)$ that was added to produce $x_t$. Because $x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon$, knowing the noise is equivalent to knowing the clean image, and substituting it back gives the reverse-step mean in closed form.

$$\mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}}\!\left(x_t - \frac{\beta_t}{\sqrt{1 - \bar\alpha_t}}\, \epsilon_\theta(x_t, t)\right)$$

The full ELBO weights each timestep differently, but Ho et al. found that discarding those weights — regressing predicted noise onto true noise with a plain mean-squared error — both simplifies the loss and improves sample quality.

$$L_\text{simple} = \mathbb{E}_{t,\, x_0,\, \epsilon}\!\left[\, \big\| \epsilon - \epsilon_\theta(x_t, t) \big\|^2 \,\right]$$

This is denoising score matching indexed by $t$: at every noise level the network learns to name the noise, which is the same as learning the score of the noised distribution at that level. Each gradient step picks a random $t$, builds $x_t$ from a clean image and a single noise draw, and never touches the chain — which is what makes training tractable at scale.

Architecture

The function computing $\epsilon_\theta$ is a U-Net: an encoder–decoder with skip connections, well suited to denoising because the problem lives at two scales at once — the noise is high-frequency and local, but deciding what the underlying image is needs global context. Ho et al. add residual blocks throughout and self-attention at the lower resolutions. The timestep $t$ is turned into a sinusoidal embedding, projected through a small MLP, and added into each residual block, so the same weights behave differently depending on how much noise they must remove. This additive timestep conditioning is worth flagging precisely: the adaptive group normalization (AdaGN) that later became standard is not from DDPM — it was introduced the following year by Dhariwal and Nichol. DDPM's conditioning is the simpler additive embedding.

Results and why it matters

DDPM reached FID 3.17 on CIFAR-10 unconditional generation and produced high-quality 256×256 LSUN samples, demonstrating for the first time that diffusion models were competitive with — and in coverage terms better than — GANs. The deeper contribution is the recipe itself: a fixed forward process, a noise-prediction objective that is trivial to implement, and a stable training signal with none of the adversarial dynamics that made GANs temperamental.

The one glaring weakness is sampling cost. Generating an image means running the reverse chain all the way down — one full U-Net forward pass for each of the thousand steps — which is structural, a consequence of the chain being Markovian and each step looking only one step back. Nearly every paper that followed attacks some part of this. DDIM breaks the Markov assumption to sample in tens of steps; Dhariwal and Nichol's Diffusion Models Beat GANs introduces AdaGN and classifier guidance to push quality past GANs on ImageNet; latent diffusion moves the whole process into a compressed space; and DiT swaps the U-Net for a transformer. All of them build on the forward/reverse/noise-prediction skeleton that DDPM established here.