Classifier-Free Diffusion Guidance

The problem

Classifier guidance, from Dhariwal and Nichol, gave diffusion the fidelity knob it had been missing — but at an awkward price. It required a separate classifier, trained from scratch to read noisy images across every noise level, and a new one for every kind of conditioning you wanted: class labels today, text embeddings tomorrow, segmentation maps after that. Worse, steering generation by following a classifier's gradient is essentially an adversarial attack on that classifier, which raised the worry that guidance was boosting metrics by exploiting the classifier rather than by genuinely improving images. Ho and Salimans asked whether the same steering could be had with no auxiliary model at all.

The key observation

The trick is to notice that the classifier gradient is already implicit in quantities the diffusion model can produce by itself. Rearranging the Bayes decomposition that classifier guidance starts from,

$$\nabla_{x_t} \log p(y \mid x_t) = \nabla_{x_t} \log p(x_t \mid y) - \nabla_{x_t} \log p(x_t),$$

the implicit classifier gradient is just the conditional score minus the unconditional one. If a single network can evaluate both scores, there is no separate classifier to train — the difference of its own two predictions is the guidance signal.

The method

You arrange for one network to produce both scores by conditioning dropout during training. With some probability — around 10% works well — the real conditioning $c$ (a class label, a caption embedding) is replaced by a fixed null token $\varnothing$. The same weights therefore learn both the conditional noise prediction $\epsilon_\theta(x_t, c)$ and the unconditional $\epsilon_\theta(x_t, \varnothing)$. At sampling time you run the network twice and extrapolate along the line from unconditional to conditional.

$$\tilde{\epsilon}(x_t, c) = \epsilon_\theta(x_t, \varnothing) + w \, \big( \epsilon_\theta(x_t, c) - \epsilon_\theta(x_t, \varnothing) \big)$$

The bracket is the direction the conditioning adds to the prediction, and $w$ controls how far past the plain conditional model you travel along it. Reading off $w$ gives the whole behavior: at $w = 0$ you sample unconditionally; at $w = 1$ the expression collapses to the ordinary conditional model with no sharpening; and at $w > 1$ you extrapolate beyond the conditional, amplifying whatever the conditioning contributed and suppressing what the unconditional model would have produced on its own.

Results and the tradeoff

On class-conditional ImageNet the method matched or beat classifier guidance on the fidelity–diversity frontier — without any classifier. The central empirical finding is that the same single knob trades diversity for fidelity: raising $w$ pulls samples toward the mode of the conditional distribution, sharpening prompt adherence and Inception/FID-style fidelity while collapsing the variety of images a single condition produces. Push $w$ too far and samples saturate and distort, because nothing in the objective guarantees the sharpened distribution stays on the manifold of real images. There is no setting that escapes the exchange — the sharpening that creates fidelity is the same operation that reduces diversity.

Why it matters

Classifier-free guidance is, quietly, one of the most consequential techniques in generative modeling. By removing the auxiliary classifier it made guidance trivial to apply to any conditioning that a single model can ingest — which is precisely what text-to-image needed, where the condition is a free-form caption rather than a fixed label set. Every major text-to-image system runs it: Stable Diffusion, DALL·E 2, and Imagen all sample with a guidance scale, and the familiar "CFG scale" slider in image-generation tools is this $w$.

It also generalizes past a single null token in a way everyone now uses. Replace $\varnothing$ with a negative prompt embedding $c_\text{neg}$ and the same extrapolation points away from $c_\text{neg}$ and toward the positive prompt — which is exactly negative prompting, the standard way to scrub artifacts and unwanted styles. The technique carried forward unchanged into transformer-backbone diffusion: DiT trains with conditioning dropout for precisely this reason, and its headline ImageNet results are reported at a tuned guidance scale. The method is short to state and trivial to implement, which is part of why it became universal.