Diffusion Models Beat GANs on Image Synthesis

The problem

DDPM showed diffusion models could generate well and cover the data distribution, but on the competitive benchmark — ImageNet at high resolution — GANs like BigGAN still held the FID lead. Two things were missing. First, the diffusion U-Net had been adopted largely as DDPM left it, with little systematic effort to find the architecture that diffusion actually wants. Second, GANs had a lever diffusion lacked: the truncation trick, which trades sample diversity for fidelity by sampling nearer the mode of the latent distribution. Diffusion had no equivalent knob. Dhariwal and Nichol supplied both, and in doing so claimed the headline result that gives the paper its name.

Architecture ablations and AdaGN

The first half of the paper is a careful sweep over the denoiser. Holding compute roughly fixed, the authors vary depth versus width, the number of attention heads, and the resolutions at which attention is applied — finding that more attention heads, attention at multiple resolutions (32×32, 16×16, and 8×8 rather than only 16×16), and the BigGAN-style residual block for up- and downsampling all help.

The most consequential change is how conditioning enters the network. DDPM injected the timestep embedding additively into each residual block. This paper replaces that with adaptive group normalization (AdaGN): the timestep embedding $t$ and class embedding $y$ are combined and used to predict a per-channel scale and shift that modulate the group-normalized activations directly.

$$\mathrm{AdaGN}(h, y, t) = y_s \,\big(\mathrm{GroupNorm}(h)\big) + y_b$$

Here $(y_s, y_b)$ are produced from the conditioning by a learned projection. This is a strictly more expressive way to feed timestep and class information into the network than addition, and it measurably improved FID. It is worth stating plainly because the attribution is routinely garbled: AdaGN comes from this paper, not from DDPM. It is the direct ancestor of the adaptive-LayerNorm conditioning that DiT later used to feed timestep and class signals into a transformer.

Classifier guidance

The second contribution is the diversity-versus-fidelity knob diffusion was missing. It starts from Bayes' rule applied to the score. Diffusion's learned object at noise level $t$ is the score $\nabla_{x_t} \log p(x_t)$, and the conditional score decomposes cleanly.

$$\nabla_{x_t} \log p(x_t \mid y) = \nabla_{x_t} \log p(x_t) + \nabla_{x_t} \log p(y \mid x_t)$$

The first term is what the unconditional diffusion model already estimates. The second is the gradient of a classifier that reads the noisy image $x_t$ and reports how class-$y$ it looks. So you train such a classifier separately — it must operate across every noise level, not an off-the-shelf ImageNet network — and add its gradient, scaled by a guidance weight $s$, to the reverse-process drift.

$$\hat\epsilon_\theta(x_t \mid y) = \epsilon_\theta(x_t) - \sqrt{1 - \bar\alpha_t}\; s\, \nabla_{x_t} \log p_\phi(y \mid x_t)$$

Pushing $s$ above one over-counts the classifier term, steering each step harder toward regions the classifier is confident belong to $y$. The effect mirrors GAN truncation: higher $s$ yields samples that adhere far more tightly to the class at the cost of diversity, sweeping out a fidelity–diversity frontier the authors can trace by varying a single number.

Results

With the improved architecture and classifier guidance, the model reached FID 4.59 on ImageNet 256×256 and 7.72 on 512×512 — beating BigGAN-deep, the strongest GAN of the time, on both fidelity and the distribution-coverage metrics where GANs had always been weak. It was the first clear demonstration that diffusion was not merely a stable alternative to GANs but a better one on the hardest standard benchmark.

Why it matters and what followed

Two threads run forward from this paper. The architectural thread — AdaGN conditioning, multi-resolution attention, the tuned residual block — became the default diffusion U-Net and, generalized, the adaptive-normalization conditioning at the heart of DiT. The guidance thread proved that a controllable fidelity knob was both possible and essential for state-of-the-art samples, but it left an awkward dependency: a separate noisy-image classifier that has to be retrained for every new conditioning modality. That cost is exactly what classifier-free guidance eliminated the next year, by extracting the same steering signal from a single model trained with conditioning dropout — making this paper the direct setup for the guidance technique that every text-to-image system now uses.