Blog Post··4 min read
ML System Design: LLM Systems
Serving LLMs (KV cache, continuous batching, speculative decoding), building enterprise RAG (chunking, hybrid retrieval, reranking), and the fine-tuning pipeline.
Tag: kv-cache
Blog Post··7 min read
Why Long-Context Mid-Training Is Its Own Stage: RoPE Scaling, Attention Entropy, and Lost-in-the-Middle
Training a model to handle 128K context isn't just running inference on longer sequences — it requires a dedicated mid-training phase because positional encoding, attention entropy, and information retrieval all break in distinct ways beyond the training window.
Blog Post··7 min read
Why Long-Context Mid-Training Is Its Own Stage: RoPE Scaling, Attention Entropy, and Lost-in-the-Middle
Training a model to handle 128K context isn't just running inference on longer sequences — it requires a dedicated mid-training phase because positional encoding, attention entropy, and information retrieval all break in distinct ways beyond the training window.
Blog Post··7 min read
vLLM Cache Metrics: KV Cache Usage, Prefix Cache Hit Rate, and the Block Pool
Two numbers determine whether a vLLM deployment is healthy: KV cache usage and prefix cache hit rate. This post explains what they measure, how vLLM computes them from its block pool, and what the LRU evictor does when memory runs out.
Blog Post··7 min read
vLLM Cache Metrics: KV Cache Usage, Prefix Cache Hit Rate, and the Block Pool
Two numbers determine whether a vLLM deployment is healthy: KV cache usage and prefix cache hit rate. This post explains what they measure, how vLLM computes them from its block pool, and what the LRU evictor does when memory runs out.
Blog Post··3 min read
The Architecture Playground: What a Transformer Config Actually Buys You
An interactive research blog. Drag the config of a decoder-only transformer — hidden size, head counts, FFN type — and watch the parameter count, KV cache, and mixture-of-experts routing recompute live.
Blog Post··5 min read
KV Cache Memory: Quantization, Eviction, and the Long-Context Problem
The KV cache is the memory bottleneck in LLM inference. As context length grows, it dominates GPU memory. Here's how quantization, eviction policies, and architectural changes manage it.
Blog Post··8 min read
PagedAttention: Virtual Memory for the KV Cache
Contiguous KV cache allocation wastes GPU memory through fragmentation and over-reservation. PagedAttention fixes this by treating the KV cache as paged virtual memory — small fixed-size blocks assigned on demand, freed immediately, and reused without copying.
Blog Post··5 min read
Prefill and Decode: The Two Phases of LLM Inference
LLM inference has two fundamentally different compute phases. Prefill processes the prompt in parallel and is compute-bound. Decode generates tokens one at a time and is memory-bandwidth-bound. Understanding both determines how you optimize.
Blog Post··3 min read
The Architecture Playground: What a Transformer Config Actually Buys You
An interactive research blog. Drag the config of a decoder-only transformer — hidden size, head counts, FFN type — and watch the parameter count, KV cache, and mixture-of-experts routing recompute live.
Blog Post··5 min read
KV Cache Memory: Quantization, Eviction, and the Long-Context Problem
The KV cache is the memory bottleneck in LLM inference. As context length grows, it dominates GPU memory. Here's how quantization, eviction policies, and architectural changes manage it.
Blog Post··8 min read
PagedAttention: Virtual Memory for the KV Cache
Contiguous KV cache allocation wastes GPU memory through fragmentation and over-reservation. PagedAttention fixes this by treating the KV cache as paged virtual memory — small fixed-size blocks assigned on demand, freed immediately, and reused without copying.
Blog Post··5 min read
Prefill and Decode: The Two Phases of LLM Inference
LLM inference has two fundamentally different compute phases. Prefill processes the prompt in parallel and is compute-bound. Decode generates tokens one at a time and is memory-bandwidth-bound. Understanding both determines how you optimize.