A neutral technical blog about AI systems, evaluation, tooling, and software practice.
Blog Post · 2026-07-17 · Revised 2026-07-17 · 7 min read
MT-Bench's frontier models cluster above 9.0 — the scale is out of room. AlpacaEval 2.0's length-controlled win rate is now saturated above 95% for top models. LMArena Elo keeps separating models as long as votes keep coming in — and they do.
Blog Post · 2026-07-17 · Revised 2026-07-17 · 6 min read
EQ-Bench CW v3 rubric scores are already saturated at the top — a 0.35-point spread across 10 models. Elo still discriminates. Here's what that gap reveals about how we evaluate creative writing.
Blog Post · 2026-07-17 · Revised 2026-07-17 · 6 min read
IFEval's top-5 span fewer than 2 percentage points — frontier models have converged on its constraint set. IFBench exposes a 29pp gap between Grok and Claude on out-of-distribution constraints. And SOB shows that JSON schema compliance is not the same as correct field values.
Blog Post · 2026-07-17 · Revised 2026-07-17 · 5 min read
IFEval's top-10 spread is 2.9pp — six of the top 12 spots go to Qwen3.5 variants. IFBench then shows a 29pp gap between reasoning models and standard instruction-tuned ones on constraints none of them trained against.
Blog Post · 2026-07-17 · Revised 2026-07-17 · 6 min read
RewardBench v1's top-6 spread is 5.7 points — small specialist models now dominate it. RewardBench 2 drops scores by 20 points and actually correlates with downstream RLHF. RM-Bench finds that style bias can push SOTA models below random performance.
Blog Post · 2026-07-10 · Revised 2026-07-10 · 4 min read
Five applied systems — content moderation, ETA, visual search, document extraction, multimodal search — each reduced to its one interesting design decision.
Blog Post · 2026-07-10 · Revised 2026-07-10 · 4 min read
The universal seven-step framework for any ML system design problem — and the specific mistakes that make interviewers fail strong candidates.
Blog Post · 2026-07-10 · Revised 2026-07-10 · 4 min read
Feature stores (online/offline duality), data vs model parallelism for distributed training, and why A/B testing ML models is harder than product A/B tests.
Blog Post · 2026-07-10 · Revised 2026-07-10 · 4 min read
Serving LLMs (KV cache, continuous batching, speculative decoding), building enterprise RAG (chunking, hybrid retrieval, reranking), and the fine-tuning pipeline.
Blog Post · 2026-07-10 · Revised 2026-07-10 · 3 min read
All 20 canonical ML system design questions as an interview prep set — what makes each hard, the anchoring design decision, and what a strong answer includes. Prompts, not solutions.
Blog Post · 2026-07-10 · Revised 2026-07-10 · 4 min read
Fraud, anomaly detection, and real-time bidding share three enemies: a tight latency SLA, concept drift, and extreme class imbalance. How production systems handle all three at once.
Blog Post · 2026-07-10 · Revised 2026-07-10 · 5 min read
Candidate generation, ranking, and reranking — the three-stage funnel behind YouTube and Spotify. Two-tower retrieval, why the funnel exists, and how cold start is actually solved.
Blog Post · 2026-07-10 · Revised 2026-07-10 · 4 min read
Learning to rank — pointwise, pairwise, listwise — plus query understanding with BERT and what feed ranking adds: engagement prediction and diversity.
Blog Post · 2026-07-02 · Revised 2026-07-02 · 5 min read
The determinant measures how much a matrix stretches or squishes space — and whether it flips orientation. When it's zero, information is lost and the matrix can't be inverted.
Blog Post · 2026-07-02 · Revised 2026-07-02 · 7 min read
Most vectors get rotated and scaled when multiplied by a matrix. Eigenvectors are the special directions that only get scaled — and their scaling factors, the eigenvalues, reveal everything about a matrix's long-term behavior.
Blog Post · 2026-07-02 · Revised 2026-07-02 · 9 min read
What does it mean for two vectors to be similar? Inner products measure alignment between vectors — and cosine similarity is just the dot product with magnitudes divided out.
Blog Post · 2026-07-02 · Revised 2026-07-02 · 8 min read
A matrix is not just a grid of numbers — it's a function that transforms space. This post builds the geometric intuition for matrix-vector multiplication as rotation, scaling, and shearing.
Blog Post · 2026-07-02 · Revised 2026-07-02 · 5 min read
Every layer of a neural network is a matrix multiplication followed by a nonlinearity. Understanding what these matrices do geometrically — how they stretch, rotate, and project — explains why deep learning works.
Blog Post · 2026-07-02 · Revised 2026-07-02 · 5 min read
Positive definite matrices define 'bowl-shaped' quadratic forms with a unique minimum. They show up everywhere optimization problems have unique solutions — from least squares to neural network loss landscapes.
Blog Post · 2026-07-02 · Revised 2026-07-02 · 6 min read
A tour through the whole series: how vectors, matrices, eigendecomposition, SVD, and least squares combine to explain the mathematical machinery inside modern ML systems — from PCA to attention to gradient descent.
Blog Post · 2026-07-02 · Revised 2026-07-02 · 4 min read
Symmetric matrices can always be diagonalized by an orthogonal matrix — their eigenvectors form a natural coordinate system for the data. This is the spectral theorem, and it underlies PCA, kernel methods, and graph Laplacians.
Blog Post · 2026-07-02 · Revised 2026-07-02 · 8 min read
Vectors are the atoms of linear algebra — everything else is built on them. This post builds intuition for what a vector is, how addition and scaling work geometrically, and why norms give us a way to measure the world.
Blog Post · 2026-06-20 · Revised 2026-06-20 · 10 min read
Chatbot Arena ranks LLMs with ELO, InstructGPT trains a reward model on pairwise preferences, and chess has rated players for seventy years. All three rest on the same one-line probabilistic model — Bradley–Terry — which turns out to be logistic regression over comparisons.
Blog Post · 2026-06-20 · Revised 2026-06-20 · 10 min read
Eckart–Young says the truncated SVD is the optimal low-rank approximation. This post turns that theorem into engineering — factorized layers, truncated-SVD compression, LoRA and its descendants QLoRA, DoRA, and GaLore, and nuclear-norm minimization for matrix completion.
Blog Post · 2026-06-20 · Revised 2026-06-20 · 9 min read
Every matrix, no matter how ugly, factors into a rotation, a scaling, and another rotation. That single fact — the singular value decomposition — is the engine behind PCA, image compression, latent semantic analysis, and the low-rank tricks that let us fine-tune giant models cheaply.
Blog Post · 2026-06-20 · Revised 2026-06-20 · 10 min read
Chatbot Arena ranks LLMs with ELO, InstructGPT trains a reward model on pairwise preferences, and chess has rated players for seventy years. All three rest on the same one-line probabilistic model — Bradley–Terry — which turns out to be logistic regression over comparisons.
Blog Post · 2026-06-20 · Revised 2026-06-20 · 10 min read
Eckart–Young says the truncated SVD is the optimal low-rank approximation. This post turns that theorem into engineering — factorized layers, truncated-SVD compression, LoRA and its descendants QLoRA, DoRA, and GaLore, and nuclear-norm minimization for matrix completion.
Blog Post · 2026-06-20 · Revised 2026-06-20 · 9 min read
Every matrix, no matter how ugly, factors into a rotation, a scaling, and another rotation. That single fact — the singular value decomposition — is the engine behind PCA, image compression, latent semantic analysis, and the low-rank tricks that let us fine-tune giant models cheaply.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 13 min read
A systematic comparison of GRPO and its descendants — CLIP-DAPO, CISCO, DAPO, Dr. GRPO, GDPO, REINFORCE++ — what each fixes, what trade-offs each makes, and when to use which.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 11 min read
What each attribution method actually computes, where they agree, where they fail, and whether gradient-based and perturbation-based approaches are still relevant for LLMs.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 12 min read
How to identify the minimal subgraph of attention heads and MLP layers that implements a specific behavior — and what we've learned from the indirect object identification circuit in GPT-2.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 8 min read
Applying the unembedding matrix at intermediate layers to watch how a transformer's prediction evolves — and what direct logit attribution tells us about which components matter.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 12 min read
Faithfulness vs. plausibility, scaling to frontier models, the composition problem, automated interpretability, and what it would take to actually understand a large language model.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 11 min read
The linear representation hypothesis, superposition, polysemanticity, and why transformer activations are more structured than they look.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 11 min read
Dictionary learning for neural networks — how sparse autoencoders recover monosemantic features from polysemantic activations, and what Anthropic's scaling monosemanticity work found in Claude.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 9 min read
Attention heads as information-routing circuits, MLP layers as key-value memories, and the residual stream as a shared communication bus.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 20 min read
What actually happens at each stage of training a large language model — what data, what objective, what the model learns, and why the stages are ordered the way they are.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 7 min read
Training a model to handle 128K context isn't just running inference on longer sequences — it requires a dedicated mid-training phase because positional encoding, attention entropy, and information retrieval all break in distinct ways beyond the training window.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 13 min read
A systematic comparison of GRPO and its descendants — CLIP-DAPO, CISCO, DAPO, Dr. GRPO, GDPO, REINFORCE++ — what each fixes, what trade-offs each makes, and when to use which.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 11 min read
What each attribution method actually computes, where they agree, where they fail, and whether gradient-based and perturbation-based approaches are still relevant for LLMs.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 12 min read
How to identify the minimal subgraph of attention heads and MLP layers that implements a specific behavior — and what we've learned from the indirect object identification circuit in GPT-2.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 8 min read
Applying the unembedding matrix at intermediate layers to watch how a transformer's prediction evolves — and what direct logit attribution tells us about which components matter.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 12 min read
Faithfulness vs. plausibility, scaling to frontier models, the composition problem, automated interpretability, and what it would take to actually understand a large language model.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 11 min read
The linear representation hypothesis, superposition, polysemanticity, and why transformer activations are more structured than they look.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 11 min read
Dictionary learning for neural networks — how sparse autoencoders recover monosemantic features from polysemantic activations, and what Anthropic's scaling monosemanticity work found in Claude.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 9 min read
Attention heads as information-routing circuits, MLP layers as key-value memories, and the residual stream as a shared communication bus.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 20 min read
What actually happens at each stage of training a large language model — what data, what objective, what the model learns, and why the stages are ordered the way they are.
Blog Post · 2025-06-20 · Revised 2025-06-20 · 7 min read
Training a model to handle 128K context isn't just running inference on longer sequences — it requires a dedicated mid-training phase because positional encoding, attention entropy, and information retrieval all break in distinct ways beyond the training window.
Blog Post · 2025-06-01 · Revised 2025-06-01 · 13 min read
Activation norms, gradient norms, weight norms — what they tell you, why they explode or vanish, and every practical cause from hyperparameters to data bugs.
Blog Post · 2025-06-01 · Revised 2025-06-01 · 13 min read
Activation norms, gradient norms, weight norms — what they tell you, why they explode or vanish, and every practical cause from hyperparameters to data bugs.
Blog Post · 2025-02-02 · Revised 2025-02-02 · 4 min read
A practical introduction to activation steering, why it is promising for AI control, and where scaling challenges still remain.
Blog Post · 2025-02-02 · Revised 2025-02-02 · 4 min read
A practical introduction to activation steering, why it is promising for AI control, and where scaling challenges still remain.
Blog Post · 2025-01-24 · Revised 2025-01-24 · 7 min read
Two numbers determine whether a vLLM deployment is healthy: KV cache usage and prefix cache hit rate. This post explains what they measure, how vLLM computes them from its block pool, and what the LRU evictor does when memory runs out.
Blog Post · 2025-01-24 · Revised 2025-01-24 · 7 min read
Two numbers determine whether a vLLM deployment is healthy: KV cache usage and prefix cache hit rate. This post explains what they measure, how vLLM computes them from its block pool, and what the LRU evictor does when memory runs out.
Blog Post · 2025-01-17 · Revised 2025-01-17 · 8 min read
Every LLM operation is either limited by how fast you can move bytes or how fast you can multiply. The roofline model tells you which — and understanding it explains why decode is slow, why batching helps, why prefill is fast, and why Flash Attention exists.
Blog Post · 2025-01-17 · Revised 2025-01-17 · 8 min read
Every LLM operation is either limited by how fast you can move bytes or how fast you can multiply. The roofline model tells you which — and understanding it explains why decode is slow, why batching helps, why prefill is fast, and why Flash Attention exists.
Blog Post · 2025-01-10 · Revised 2025-01-10 · 3 min read
Five LLM architectures — GPT-2, Qwen3-8B, DeepSeek-V3, DeepSeek-R1, GPT-OSS-20B/120B — shown as interactive block diagrams. Click any block to expand equations and parameters. Each model is sourced from its official HF config.json.
Blog Post · 2025-01-10 · Revised 2025-01-10 · 6 min read
Every equation in scaled dot-product attention and multi-head attention annotated term-by-term — the scaling, the softmax, the heads, RoPE, and KV cache — with links to the posts explaining each design choice.
Blog Post · 2025-01-10 · Revised 2025-01-10 · 10 min read
The full forward pass, written out as equations, for GPT-2, Qwen3-8B, DeepSeek-V3, and GPT-OSS. Every matrix, every norm, every residual — in the order the computation actually happens.
Blog Post · 2025-01-10 · Revised 2025-01-10 · 6 min read
PPO and GRPO loss functions annotated term-by-term — the clipped surrogate, GAE, value loss, entropy bonus, and group-normalised advantages — with links to the posts explaining each design choice.
Blog Post · 2025-01-10 · Revised 2025-01-10 · 6 min read
Perplexity, negative log-likelihood, entropy, KL divergence, and cross-entropy annotated term-by-term — what each measures, how they relate, and what values to expect during training.
Blog Post · 2025-01-10 · Revised 2025-01-10 · 3 min read
Five LLM architectures — GPT-2, Qwen3-8B, DeepSeek-V3, DeepSeek-R1, GPT-OSS-20B/120B — shown as interactive block diagrams. Click any block to expand equations and parameters. Each model is sourced from its official HF config.json.
Blog Post · 2025-01-10 · Revised 2025-01-10 · 6 min read
Every equation in scaled dot-product attention and multi-head attention annotated term-by-term — the scaling, the softmax, the heads, RoPE, and KV cache — with links to the posts explaining each design choice.
Blog Post · 2025-01-10 · Revised 2025-01-10 · 10 min read
The full forward pass, written out as equations, for GPT-2, Qwen3-8B, DeepSeek-V3, and GPT-OSS. Every matrix, every norm, every residual — in the order the computation actually happens.
Blog Post · 2025-01-10 · Revised 2025-01-10 · 6 min read
PPO and GRPO loss functions annotated term-by-term — the clipped surrogate, GAE, value loss, entropy bonus, and group-normalised advantages — with links to the posts explaining each design choice.
Blog Post · 2025-01-10 · Revised 2025-01-10 · 6 min read
Perplexity, negative log-likelihood, entropy, KL divergence, and cross-entropy annotated term-by-term — what each measures, how they relate, and what values to expect during training.
Blog Post · 2024-06-20 · Revised 2024-06-20 · 9 min read
CUDA exposes GPU parallelism through a three-level thread hierarchy: grid, block, and warp. Understanding how these map to hardware — SMs, register files, shared memory — is the prerequisite for writing fast kernels.
Blog Post · 2024-06-20 · Revised 2024-06-20 · 8 min read
Kernel fusion eliminates the HBM round-trips between chained operations. Triton makes this practical in Python. This post builds a fused online softmax from scratch, then extends it to a fused RMSNorm + linear projection — the kind of kernel that actually speeds up LLM inference.
Blog Post · 2024-06-20 · Revised 2024-06-20 · 9 min read
CUDA exposes GPU parallelism through a three-level thread hierarchy: grid, block, and warp. Understanding how these map to hardware — SMs, register files, shared memory — is the prerequisite for writing fast kernels.
Blog Post · 2024-06-20 · Revised 2024-06-20 · 8 min read
Kernel fusion eliminates the HBM round-trips between chained operations. Triton makes this practical in Python. This post builds a fused online softmax from scratch, then extends it to a fused RMSNorm + linear projection — the kind of kernel that actually speeds up LLM inference.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 3 min read
An interactive research blog. Drag the config of a decoder-only transformer — hidden size, head counts, FFN type — and watch the parameter count, KV cache, and mixture-of-experts routing recompute live.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Conditioning a diffusion model on text gives you text-to-image generation. Classifier-free guidance makes that conditioning much stronger — at the cost of some diversity.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
DDPM needs 1000 steps to generate a sample. DDIM reframes the reverse process as an ODE and gets the same quality in 50. The model weights are identical — only the sampling procedure changes.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
DDPM defines a fixed forward process that gradually destroys an image into noise, then trains a neural network to reverse it. The math is tractable because each step is Gaussian.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
DDPM, DDIM, and latent diffusion all use a U-Net backbone. DiT replaces it with a transformer — and finds that diffusion scales with model size the same way language models do.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Diffusion models learn to reverse a specific noise process. Flow matching learns to transport any source distribution to any target distribution along straight paths — simpler math, faster sampling, and better training signal.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Running DDPM in pixel space at 512×512 is expensive. Latent diffusion compresses the image into a small latent space first, runs the diffusion process there, and decodes back. The same quality, a fraction of the compute.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Diffusion models learn to reverse a noise process. The key insight is that you don't need to know the data distribution — you only need to learn its score function, the gradient of the log-density.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 4 min read
BERT used dropout everywhere. LLaMA uses none. The reason isn't that regularization stopped mattering — it's that at trillion-token scale, data diversity IS the regularizer.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
GRPO achieves competitive alignment results without a value function. Here's exactly what changes in the math and implementation, and why that matters for training efficiency and stability.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 4 min read
Static batching wastes GPU capacity whenever sequences finish at different times. Continuous batching fixes this by treating the decode loop as a queue — adding new requests the moment a slot opens up.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 8 min read
LLM inference is shaped by GPU hardware: HBM bandwidth, SRAM per SM, tensor core throughput, and the roofline that connects them. This post maps the memory hierarchy from HBM to tensor core, shows where decode and prefill sit on the roofline, and explains why FlashAttention exists.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
The KV cache is the memory bottleneck in LLM inference. As context length grows, it dominates GPU memory. Here's how quantization, eviction policies, and architectural changes manage it.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 8 min read
Contiguous KV cache allocation wastes GPU memory through fragmentation and over-reservation. PagedAttention fixes this by treating the KV cache as paged virtual memory — small fixed-size blocks assigned on demand, freed immediately, and reused without copying.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
LLM inference has two fundamentally different compute phases. Prefill processes the prompt in parallel and is compute-bound. Decode generates tokens one at a time and is memory-bandwidth-bound. Understanding both determines how you optimize.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
A single GPU is the easy part. Serving LLMs at production scale involves tensor parallelism, pipeline parallelism, load balancing, SLO enforcement, and hardware heterogeneity. Here's how it fits together.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
A request-level walkthrough of how the KV cache is populated, grown, and read during LLM inference — covering prefill, decode, memory layout, and why decode is memory-bandwidth-bound.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 9 min read
GPT-4, Gemini, LLaMA, Mistral, DeepSeek, Qwen — they all build on the same transformer skeleton. But the architectural choices diverge sharply. Here's a systematic comparison across model families.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 8 min read
Training a helpful, harmless, honest LLM requires three sequential stages that each build on the previous one. Here's how SFT, reward modeling, and RL fit together as a system — and where each stage can fail.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 18 min read
Probability, likelihood, log-likelihood, NLL, expectation, entropy, cross-entropy, KL divergence, and perplexity are constantly confused for one another. They are not the same thing — but they are all the same thing seen from different angles. This is the definitive deep-dive that names each one precisely and shows exactly how they connect for LLMs.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
Without nonlinearity, stacking layers collapses to a single matrix multiplication. Activation functions break that linearity — and the choice of which one determines expressivity, gradient flow, and training efficiency.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 8 min read
How do you measure whether a model is actually good? The answer is a set of metrics — precision, recall, F1, perplexity, calibration, confidence intervals — each measuring something different and failing in a different way.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Information theory gives precise answers to questions like: how much does the context tell you about the next token? What information is preserved in a representation? Why does compression and prediction point to the same objective?
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Every forward pass is a sequence of matrix multiplications. Understanding what those matrices do — rotate, scale, project — is the foundation for understanding why transformers work.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
The loss function is the specification. Everything the model learns is in service of minimizing it. Here's the math behind every major loss used in LLM training and fine-tuning.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Training a neural network is an optimization problem: minimize a loss function over billions of parameters. The journey from vanilla gradient descent to Adam reveals why each step was necessary.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
A language model is a probability distribution over sequences. Training it means pushing that distribution toward the data distribution. The math of how you measure and minimize that gap is what this post covers.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
A single training step involves linear algebra, probability, information theory, optimization, and statistical estimation — all at once. Here's how the pieces fit.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
Language models are probabilistic systems. Understanding the statistical machinery behind maximum likelihood estimation, Bayesian inference, and sampling algorithms clarifies why training and decoding work the way they do.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 8 min read
LoRA works because weight matrices are empirically low-rank. SVD explains why, and the math of low-rank approximation explains how LoRA exploits it to fine-tune billion-parameter models on a laptop.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Padding wastes GPU compute. Sequence packing eliminates it — but introduces cross-document attention contamination unless you explicitly mask it. Here's what the attention mask actually looks like.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
ppo-loss-per-token covered the clipped surrogate objective. This post covers what surrounds it: how the value function is trained, where the advantage estimates come from, and why the entropy bonus exists.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Most explanations of PPO stay at the algorithm level. This post goes one level deeper: how the surrogate loss is actually computed token by token for a language model response.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
LLMs memorize training data. Under the right prompts, they reproduce it. Here's how memorization works, how to measure it, and the specific privacy risks in code generation models.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Human red-teaming finds attacks automated evals miss. Automated evals achieve scale humans can't. Here's how to combine them, and what each can and can't tell you.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Training-time alignment is not enough. Production AI systems need runtime layers that detect, intercept, and respond to harmful inputs and outputs. Here's how to build them.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Most safety benchmarks are gameable, distribution-shifted, or measure the wrong thing. Here's what separates a rigorous safety evaluation from a checkbox.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 10 min read
Single-turn RL teaches a model to produce good responses. Agentic RL teaches it to complete multi-step tasks in an environment — with delayed rewards, partial observability, and real consequences.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
RLHF is three steps: supervised fine-tuning, reward model training, and policy optimization. Each step has a specific failure mode. Here's the full picture.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
The policy gradient theorem lets you differentiate through a reward signal you can't backprop through. Here's the derivation and why it works.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
The reward signal determines what the model learns to do. Swap the reward, swap the capability. Here's how RL elicits reasoning, code generation, math, and tool use.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Supervised fine-tuning teaches a model to imitate. Reinforcement learning teaches it to optimize. The difference turns out to matter enormously.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
Contrastive learning teaches a model that two views of the same image should be close in representation space, and views of different images should be far apart. The details of how you enforce this determine everything.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 4 min read
I-JEPA applies the JEPA idea to images: predict the representations of target patches from a context region, without any view-level augmentations. The result transfers better to semantic tasks than pixel-level methods.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
MAE predicts pixels. Contrastive methods match views. JEPA predicts representations of target regions from context regions — in an abstract space where irrelevant details have already been discarded.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Mask 75% of an image's patches. Train a model to reconstruct them. The result is a rich visual representation — and the recipe works because pixels are redundant and structure is not.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 4 min read
V-JEPA extends JEPA to video: predict the representations of future or masked frames from context frames. No pixel reconstruction, no contrastive loss — just abstract prediction across time.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Supervised learning requires labels. Labels require humans. At scale, that's the bottleneck. Self-supervised learning sidesteps it by constructing supervision from the data itself.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
JEPA is a learning architecture. World models are the goal it points toward — internal simulators that can predict the consequences of actions and support planning without interacting with the real world.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
Human annotation doesn't scale to the data volumes modern alignment requires. Synthetic data — generated by LLMs, filtered, and refined — has become the dominant approach. Here's how it's done and where it breaks down.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 8 min read
GPT-2 established the decoder-only transformer as the dominant paradigm. What followed was six years of systematic improvements — in scale, efficiency, alignment, and reasoning. Here's the arc.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
The FFN block consumes most of a transformer's parameters. The choices made there — activation function, gating, expert routing — account for much of the quality gap between model families.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Multi-head attention was the original. Multi-query attention was the efficient approximation. Grouped-query attention is the synthesis that modern LLMs converged on — and the reason is bandwidth, not FLOPs.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 4 min read
Up to 30% of GPU compute can vanish into padding tokens that contribute nothing to learning. Here's how modern pretraining pipelines eliminate that waste.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 13 min read
Most training failures leave signatures in the metrics before they fully manifest. Here's how to read loss curves, gradient norms, learning rate schedules, and activation statistics to diagnose what's going wrong.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 4 min read
Post-LN dominated the original transformer. Pre-LN dominates everything since GPT-2. The reason comes down to gradient flow — and the math is clean enough to be worth understanding.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Adam is the default optimizer for language model training, but using it correctly — the right β values, weight decay, learning rate schedule — makes a larger difference than most people expect.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 10 min read
Attention is permutation-invariant. Positional encodings break that symmetry. The choice of encoding method determines whether your model can generalize to longer sequences than it trained on.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 3 min read
An interactive research blog. Drag the config of a decoder-only transformer — hidden size, head counts, FFN type — and watch the parameter count, KV cache, and mixture-of-experts routing recompute live.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Conditioning a diffusion model on text gives you text-to-image generation. Classifier-free guidance makes that conditioning much stronger — at the cost of some diversity.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
DDPM needs 1000 steps to generate a sample. DDIM reframes the reverse process as an ODE and gets the same quality in 50. The model weights are identical — only the sampling procedure changes.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
DDPM defines a fixed forward process that gradually destroys an image into noise, then trains a neural network to reverse it. The math is tractable because each step is Gaussian.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
DDPM, DDIM, and latent diffusion all use a U-Net backbone. DiT replaces it with a transformer — and finds that diffusion scales with model size the same way language models do.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Diffusion models learn to reverse a specific noise process. Flow matching learns to transport any source distribution to any target distribution along straight paths — simpler math, faster sampling, and better training signal.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Running DDPM in pixel space at 512×512 is expensive. Latent diffusion compresses the image into a small latent space first, runs the diffusion process there, and decodes back. The same quality, a fraction of the compute.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Diffusion models learn to reverse a noise process. The key insight is that you don't need to know the data distribution — you only need to learn its score function, the gradient of the log-density.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 4 min read
BERT used dropout everywhere. LLaMA uses none. The reason isn't that regularization stopped mattering — it's that at trillion-token scale, data diversity IS the regularizer.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
GRPO achieves competitive alignment results without a value function. Here's exactly what changes in the math and implementation, and why that matters for training efficiency and stability.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 4 min read
Static batching wastes GPU capacity whenever sequences finish at different times. Continuous batching fixes this by treating the decode loop as a queue — adding new requests the moment a slot opens up.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 8 min read
LLM inference is shaped by GPU hardware: HBM bandwidth, SRAM per SM, tensor core throughput, and the roofline that connects them. This post maps the memory hierarchy from HBM to tensor core, shows where decode and prefill sit on the roofline, and explains why FlashAttention exists.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
The KV cache is the memory bottleneck in LLM inference. As context length grows, it dominates GPU memory. Here's how quantization, eviction policies, and architectural changes manage it.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 8 min read
Contiguous KV cache allocation wastes GPU memory through fragmentation and over-reservation. PagedAttention fixes this by treating the KV cache as paged virtual memory — small fixed-size blocks assigned on demand, freed immediately, and reused without copying.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
LLM inference has two fundamentally different compute phases. Prefill processes the prompt in parallel and is compute-bound. Decode generates tokens one at a time and is memory-bandwidth-bound. Understanding both determines how you optimize.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
A single GPU is the easy part. Serving LLMs at production scale involves tensor parallelism, pipeline parallelism, load balancing, SLO enforcement, and hardware heterogeneity. Here's how it fits together.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
A request-level walkthrough of how the KV cache is populated, grown, and read during LLM inference — covering prefill, decode, memory layout, and why decode is memory-bandwidth-bound.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 9 min read
GPT-4, Gemini, LLaMA, Mistral, DeepSeek, Qwen — they all build on the same transformer skeleton. But the architectural choices diverge sharply. Here's a systematic comparison across model families.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 8 min read
Training a helpful, harmless, honest LLM requires three sequential stages that each build on the previous one. Here's how SFT, reward modeling, and RL fit together as a system — and where each stage can fail.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 18 min read
Probability, likelihood, log-likelihood, NLL, expectation, entropy, cross-entropy, KL divergence, and perplexity are constantly confused for one another. They are not the same thing — but they are all the same thing seen from different angles. This is the definitive deep-dive that names each one precisely and shows exactly how they connect for LLMs.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
Without nonlinearity, stacking layers collapses to a single matrix multiplication. Activation functions break that linearity — and the choice of which one determines expressivity, gradient flow, and training efficiency.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 8 min read
How do you measure whether a model is actually good? The answer is a set of metrics — precision, recall, F1, perplexity, calibration, confidence intervals — each measuring something different and failing in a different way.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Information theory gives precise answers to questions like: how much does the context tell you about the next token? What information is preserved in a representation? Why does compression and prediction point to the same objective?
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Every forward pass is a sequence of matrix multiplications. Understanding what those matrices do — rotate, scale, project — is the foundation for understanding why transformers work.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
The loss function is the specification. Everything the model learns is in service of minimizing it. Here's the math behind every major loss used in LLM training and fine-tuning.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Training a neural network is an optimization problem: minimize a loss function over billions of parameters. The journey from vanilla gradient descent to Adam reveals why each step was necessary.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
A language model is a probability distribution over sequences. Training it means pushing that distribution toward the data distribution. The math of how you measure and minimize that gap is what this post covers.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
A single training step involves linear algebra, probability, information theory, optimization, and statistical estimation — all at once. Here's how the pieces fit.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
Language models are probabilistic systems. Understanding the statistical machinery behind maximum likelihood estimation, Bayesian inference, and sampling algorithms clarifies why training and decoding work the way they do.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 8 min read
LoRA works because weight matrices are empirically low-rank. SVD explains why, and the math of low-rank approximation explains how LoRA exploits it to fine-tune billion-parameter models on a laptop.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Padding wastes GPU compute. Sequence packing eliminates it — but introduces cross-document attention contamination unless you explicitly mask it. Here's what the attention mask actually looks like.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
ppo-loss-per-token covered the clipped surrogate objective. This post covers what surrounds it: how the value function is trained, where the advantage estimates come from, and why the entropy bonus exists.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Most explanations of PPO stay at the algorithm level. This post goes one level deeper: how the surrogate loss is actually computed token by token for a language model response.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
LLMs memorize training data. Under the right prompts, they reproduce it. Here's how memorization works, how to measure it, and the specific privacy risks in code generation models.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Human red-teaming finds attacks automated evals miss. Automated evals achieve scale humans can't. Here's how to combine them, and what each can and can't tell you.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Training-time alignment is not enough. Production AI systems need runtime layers that detect, intercept, and respond to harmful inputs and outputs. Here's how to build them.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Most safety benchmarks are gameable, distribution-shifted, or measure the wrong thing. Here's what separates a rigorous safety evaluation from a checkbox.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 10 min read
Single-turn RL teaches a model to produce good responses. Agentic RL teaches it to complete multi-step tasks in an environment — with delayed rewards, partial observability, and real consequences.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
RLHF is three steps: supervised fine-tuning, reward model training, and policy optimization. Each step has a specific failure mode. Here's the full picture.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
The policy gradient theorem lets you differentiate through a reward signal you can't backprop through. Here's the derivation and why it works.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
The reward signal determines what the model learns to do. Swap the reward, swap the capability. Here's how RL elicits reasoning, code generation, math, and tool use.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Supervised fine-tuning teaches a model to imitate. Reinforcement learning teaches it to optimize. The difference turns out to matter enormously.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
Contrastive learning teaches a model that two views of the same image should be close in representation space, and views of different images should be far apart. The details of how you enforce this determine everything.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 4 min read
I-JEPA applies the JEPA idea to images: predict the representations of target patches from a context region, without any view-level augmentations. The result transfers better to semantic tasks than pixel-level methods.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
MAE predicts pixels. Contrastive methods match views. JEPA predicts representations of target regions from context regions — in an abstract space where irrelevant details have already been discarded.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Mask 75% of an image's patches. Train a model to reconstruct them. The result is a rich visual representation — and the recipe works because pixels are redundant and structure is not.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 4 min read
V-JEPA extends JEPA to video: predict the representations of future or masked frames from context frames. No pixel reconstruction, no contrastive loss — just abstract prediction across time.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Supervised learning requires labels. Labels require humans. At scale, that's the bottleneck. Self-supervised learning sidesteps it by constructing supervision from the data itself.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
JEPA is a learning architecture. World models are the goal it points toward — internal simulators that can predict the consequences of actions and support planning without interacting with the real world.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 7 min read
Human annotation doesn't scale to the data volumes modern alignment requires. Synthetic data — generated by LLMs, filtered, and refined — has become the dominant approach. Here's how it's done and where it breaks down.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 8 min read
GPT-2 established the decoder-only transformer as the dominant paradigm. What followed was six years of systematic improvements — in scale, efficiency, alignment, and reasoning. Here's the arc.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
The FFN block consumes most of a transformer's parameters. The choices made there — activation function, gating, expert routing — account for much of the quality gap between model families.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 6 min read
Multi-head attention was the original. Multi-query attention was the efficient approximation. Grouped-query attention is the synthesis that modern LLMs converged on — and the reason is bandwidth, not FLOPs.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 4 min read
Up to 30% of GPU compute can vanish into padding tokens that contribute nothing to learning. Here's how modern pretraining pipelines eliminate that waste.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 13 min read
Most training failures leave signatures in the metrics before they fully manifest. Here's how to read loss curves, gradient norms, learning rate schedules, and activation statistics to diagnose what's going wrong.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 4 min read
Post-LN dominated the original transformer. Pre-LN dominates everything since GPT-2. The reason comes down to gradient flow — and the math is clean enough to be worth understanding.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 5 min read
Adam is the default optimizer for language model training, but using it correctly — the right β values, weight decay, learning rate schedule — makes a larger difference than most people expect.
Blog Post · 2024-06-19 · Revised 2024-06-19 · 10 min read
Attention is permutation-invariant. Positional encodings break that symmetry. The choice of encoding method determines whether your model can generalize to longer sequences than it trained on.