Series
Transformer Training Internals
The engineering details that determine whether your model converges, generalizes, and runs efficiently — from normalization choices to optimizer internals.
- 1
Normalization in Transformers: Why Pre-LN Became the Default
Post-LN dominated the original transformer. Pre-LN dominates everything since GPT-2. The reason comes down to gradient flow — and the math is clean enough to be worth understanding.
2024-06-19
- 1
Normalization in Transformers: Why Pre-LN Became the Default
Post-LN dominated the original transformer. Pre-LN dominates everything since GPT-2. The reason comes down to gradient flow — and the math is clean enough to be worth understanding.
2024-06-19
- 2
Optimizers for LLMs: Adam, Weight Decay, and Why Learning Rate Matters More Than You Think
Adam is the default optimizer for language model training, but using it correctly — the right β values, weight decay, learning rate schedule — makes a larger difference than most people expect.
2024-06-19
- 2
Optimizers for LLMs: Adam, Weight Decay, and Why Learning Rate Matters More Than You Think
Adam is the default optimizer for language model training, but using it correctly — the right β values, weight decay, learning rate schedule — makes a larger difference than most people expect.
2024-06-19
- 3
Data Efficiency in Pretraining: Packing, Batching, and What Gets Wasted
Up to 30% of GPU compute can vanish into padding tokens that contribute nothing to learning. Here's how modern pretraining pipelines eliminate that waste.
2024-06-19
- 3
Data Efficiency in Pretraining: Packing, Batching, and What Gets Wasted
Up to 30% of GPU compute can vanish into padding tokens that contribute nothing to learning. Here's how modern pretraining pipelines eliminate that waste.
2024-06-19
- 4
Positional Encodings: From Sinusoids to RoPE
Attention is permutation-invariant. Positional encodings break that symmetry. The choice of encoding method determines whether your model can generalize to longer sequences than it trained on.
2024-06-19
- 4
Positional Encodings: From Sinusoids to RoPE
Attention is permutation-invariant. Positional encodings break that symmetry. The choice of encoding method determines whether your model can generalize to longer sequences than it trained on.
2024-06-19
- 5
Attention Variants: MHA, MQA, GQA, and the Memory Math Behind Them
Multi-head attention was the original. Multi-query attention was the efficient approximation. Grouped-query attention is the synthesis that modern LLMs converged on — and the reason is bandwidth, not FLOPs.
2024-06-19
- 5
Attention Variants: MHA, MQA, GQA, and the Memory Math Behind Them
Multi-head attention was the original. Multi-query attention was the efficient approximation. Grouped-query attention is the synthesis that modern LLMs converged on — and the reason is bandwidth, not FLOPs.
2024-06-19
- 6
Inside the FFN: MoE, SwiGLU, and the Architectural Details That Scale
The FFN block consumes most of a transformer's parameters. The choices made there — activation function, gating, expert routing — account for much of the quality gap between model families.
2024-06-19
- 6
Inside the FFN: MoE, SwiGLU, and the Architectural Details That Scale
The FFN block consumes most of a transformer's parameters. The choices made there — activation function, gating, expert routing — account for much of the quality gap between model families.
2024-06-19
- 7
Debugging Transformer Training Runs: Reading the Curves
Most training failures leave signatures in the metrics before they fully manifest. Here's how to read loss curves, gradient norms, learning rate schedules, and activation statistics to diagnose what's going wrong.
2024-06-19
- 7
Debugging Transformer Training Runs: Reading the Curves
Most training failures leave signatures in the metrics before they fully manifest. Here's how to read loss curves, gradient norms, learning rate schedules, and activation statistics to diagnose what's going wrong.
2024-06-19