High-Resolution Image Synthesis with Latent Diffusion Models

The problem

DDIM cut the number of denoising steps, but each step remained a full U-Net pass over a full-resolution image — and at 512×512 that resolution is the dominant cost. A 512×512 RGB image is $512 \times 512 \times 3 = 786{,}432$ dimensions, and the U-Net processes a tensor of that spatial size at every step of every sample. Yet almost none of those dimensions do generative work: adjacent pixels are heavily correlated, photographic images live on a thin manifold inside that huge space, and the visual system barely registers much of the high-frequency detail the pixels encode. Pixel-space diffusion spends the entire process modeling perceptual redundancy at enormous expense.

The method: separate perception from generation

Latent diffusion splits the two jobs pixel-space diffusion conflated. First train a perceptual autoencoder that compresses images into a small latent space; then run the whole diffusion process in that space. A 512×512 image compressed by a factor of 8 per spatial dimension becomes a $64 \times 64 \times 4$ latent — $16{,}384$ numbers instead of three-quarters of a million — and the U-Net now denoises a tensor two orders of magnitude smaller, with no change to the diffusion math.

The autoencoder is trained first and on its own, with an objective built to preserve what the eye cares about. An encoder $E$ maps an image to a latent $z$ and a decoder $D$ maps it back, under a loss that combines a reconstruction term, a perceptual term matching features of a pretrained VGG network rather than raw pixels, an adversarial term from a patch discriminator that pushes reconstructions to look real, and a light KL penalty keeping the latent distribution close to a standard Gaussian.

$$L_\text{AE} = L_\text{rec} + \lambda_\text{perc}\, L_\text{perc} + \lambda_\text{adv}\, L_\text{adv} + \lambda_\text{KL}\, \mathrm{KL}\!\big(q(z \mid x)\,\|\,\mathcal{N}(0, I)\big)$$

The KL weight is deliberately tiny — this is an autoencoder that reconstructs, regularized just enough to be Gaussian-ish, not a VAE that prioritizes a clean prior over sharp output. Once trained, its weights freeze, and stage two trains the diffusion U-Net entirely on latents $z = E(x)$ with the familiar noise-prediction loss carried over unchanged from DDPM.

Splitting training in two is the reason the method works. Perceptual compression and generative modeling are different problems — the autoencoder decides what information is worth keeping (a question about human perception), while the diffusion model learns the distribution of valid latents (a question about the data). Forcing one network to do both at full resolution wastes capacity on redundancy the autoencoder handles once, cheaply.

Cross-attention conditioning

Working in latent space also turned out to be the natural place to introduce general conditioning, and this is where the paper stopped being a compute optimization and became the template for text-to-image. The U-Net is augmented with cross-attention layers: a conditioning input $y$ — a text prompt, a class label, a segmentation map — is run through a domain-specific encoder $\tau_\theta$ to produce an embedding, and at each spatial resolution the U-Net's features attend to it. Queries come from the spatial feature map; keys and values from the conditioning embedding.

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{Q K^\top}{\sqrt{d}}\right) V, \qquad Q = W_Q\, \varphi(z_t),\;\; K = W_K\, \tau_\theta(y),\;\; V = W_V\, \tau_\theta(y)$$

Because conditioning enters through attention rather than being baked into the architecture, the same mechanism accepts any signal you can encode into a sequence of vectors — exactly the flexibility text-to-image needs.

Results and Stable Diffusion

Latent diffusion runs roughly 4–8× faster than pixel-space diffusion at matched quality, and it set or matched the state of the art across unconditional generation, inpainting, super-resolution, and text-to-image on standard benchmarks while training on far less compute than comparable pixel-space models. Its most famous instantiation is Stable Diffusion: a latent diffusion model with a CLIP text encoder as $\tau_\theta$, trained on the LAION-5B image–text dataset — the same design as the LDM paper, distinguished mainly by the scale of the training data. Everything downstream of the latent — the U-Net, the noise schedule, the DDIM sampler — is the machinery of the earlier diffusion work, simply run in a smaller space.

Why it matters and the catch

The compression is what made open, runnable text-to-image possible: by shrinking the tensor the U-Net operates on, latent diffusion brought high-resolution generation within reach of a single consumer GPU, which is largely why Stable Diffusion sparked the open-source generative-image ecosystem. The catch is the autoencoder bottleneck: anything the encoder discards is gone for good, so at aggressive compression ratios fine texture, small regular patterns, and crisp text degrade in ways no amount of diffusion capacity can recover. Pixel-space systems like DALL·E 2 and Imagen avoid that ceiling and pay for it in compute. The latent-space design carried forward: when DiT replaced the U-Net with a transformer it kept operating on exactly this kind of VAE latent, and the latest systems (Stable Diffusion 3, Flux) remain latent-space diffusion models with transformer backbones.