Series
Inference Systems for LLMs
Prefill vs decode, continuous batching, memory management, and serving at scale — the engineering behind fast and efficient LLM inference.
- 1
Prefill and Decode: The Two Phases of LLM Inference
LLM inference has two fundamentally different compute phases. Prefill processes the prompt in parallel and is compute-bound. Decode generates tokens one at a time and is memory-bandwidth-bound. Understanding both determines how you optimize.
2024-06-19
- 1
Prefill and Decode: The Two Phases of LLM Inference
LLM inference has two fundamentally different compute phases. Prefill processes the prompt in parallel and is compute-bound. Decode generates tokens one at a time and is memory-bandwidth-bound. Understanding both determines how you optimize.
2024-06-19
- 2
Continuous Batching: How vLLM Serves Thousands of Requests
Static batching wastes GPU capacity whenever sequences finish at different times. Continuous batching fixes this by treating the decode loop as a queue — adding new requests the moment a slot opens up.
2024-06-19
- 2
Continuous Batching: How vLLM Serves Thousands of Requests
Static batching wastes GPU capacity whenever sequences finish at different times. Continuous batching fixes this by treating the decode loop as a queue — adding new requests the moment a slot opens up.
2024-06-19
- 3
KV Cache Memory: Quantization, Eviction, and the Long-Context Problem
The KV cache is the memory bottleneck in LLM inference. As context length grows, it dominates GPU memory. Here's how quantization, eviction policies, and architectural changes manage it.
2024-06-19
- 3
KV Cache Memory: Quantization, Eviction, and the Long-Context Problem
The KV cache is the memory bottleneck in LLM inference. As context length grows, it dominates GPU memory. Here's how quantization, eviction policies, and architectural changes manage it.
2024-06-19
- 4
LLM Serving at Scale: Throughput, Latency, and System Design
A single GPU is the easy part. Serving LLMs at production scale involves tensor parallelism, pipeline parallelism, load balancing, SLO enforcement, and hardware heterogeneity. Here's how it fits together.
2024-06-19
- 4
LLM Serving at Scale: Throughput, Latency, and System Design
A single GPU is the easy part. Serving LLMs at production scale involves tensor parallelism, pipeline parallelism, load balancing, SLO enforcement, and hardware heterogeneity. Here's how it fits together.
2024-06-19
- 5
GPU Architecture for LLM Inference
LLM inference is shaped by GPU hardware: HBM bandwidth, SRAM per SM, tensor core throughput, and the roofline that connects them. This post maps the memory hierarchy from HBM to tensor core, shows where decode and prefill sit on the roofline, and explains why FlashAttention exists.
2024-06-19
- 5
GPU Architecture for LLM Inference
LLM inference is shaped by GPU hardware: HBM bandwidth, SRAM per SM, tensor core throughput, and the roofline that connects them. This post maps the memory hierarchy from HBM to tensor core, shows where decode and prefill sit on the roofline, and explains why FlashAttention exists.
2024-06-19
- 6
PagedAttention: Virtual Memory for the KV Cache
Contiguous KV cache allocation wastes GPU memory through fragmentation and over-reservation. PagedAttention fixes this by treating the KV cache as paged virtual memory — small fixed-size blocks assigned on demand, freed immediately, and reused without copying.
2024-06-19
- 6
PagedAttention: Virtual Memory for the KV Cache
Contiguous KV cache allocation wastes GPU memory through fragmentation and over-reservation. PagedAttention fixes this by treating the KV cache as paged virtual memory — small fixed-size blocks assigned on demand, freed immediately, and reused without copying.
2024-06-19
- 7
Memory-Bound vs Compute-Bound: Where LLM Inference Really Spends Its Time
Every LLM operation is either limited by how fast you can move bytes or how fast you can multiply. The roofline model tells you which — and understanding it explains why decode is slow, why batching helps, why prefill is fast, and why Flash Attention exists.
2025-01-17
- 7
Memory-Bound vs Compute-Bound: Where LLM Inference Really Spends Its Time
Every LLM operation is either limited by how fast you can move bytes or how fast you can multiply. The roofline model tells you which — and understanding it explains why decode is slow, why batching helps, why prefill is fast, and why Flash Attention exists.
2025-01-17
- 8
vLLM Cache Metrics: KV Cache Usage, Prefix Cache Hit Rate, and the Block Pool
Two numbers determine whether a vLLM deployment is healthy: KV cache usage and prefix cache hit rate. This post explains what they measure, how vLLM computes them from its block pool, and what the LRU evictor does when memory runs out.
2025-01-24
- 8
vLLM Cache Metrics: KV Cache Usage, Prefix Cache Hit Rate, and the Block Pool
Two numbers determine whether a vLLM deployment is healthy: KV cache usage and prefix cache hit rate. This post explains what they measure, how vLLM computes them from its block pool, and what the LRU evictor does when memory runs out.
2025-01-24