Series
Reinforcement Learning for LLMs
From policy gradients to RLHF, GRPO, and agentic systems — a ground-up series on how RL shapes modern language models.
- 1
Why Language Models Need Reinforcement Learning
Supervised fine-tuning teaches a model to imitate. Reinforcement learning teaches it to optimize. The difference turns out to matter enormously.
2024-06-19
- 1
Why Language Models Need Reinforcement Learning
Supervised fine-tuning teaches a model to imitate. Reinforcement learning teaches it to optimize. The difference turns out to matter enormously.
2024-06-19
- 2
Policy Gradients: The Math Behind RLHF
The policy gradient theorem lets you differentiate through a reward signal you can't backprop through. Here's the derivation and why it works.
2024-06-19
- 2
Policy Gradients: The Math Behind RLHF
The policy gradient theorem lets you differentiate through a reward signal you can't backprop through. Here's the derivation and why it works.
2024-06-19
- 3
Learning from Feedback: RLHF, RLAIF, and Beyond
RLHF is three steps: supervised fine-tuning, reward model training, and policy optimization. Each step has a specific failure mode. Here's the full picture.
2024-06-19
- 3
Learning from Feedback: RLHF, RLAIF, and Beyond
RLHF is three steps: supervised fine-tuning, reward model training, and policy optimization. Each step has a specific failure mode. Here's the full picture.
2024-06-19
- 4
RL as a Skill Acquisition Engine
The reward signal determines what the model learns to do. Swap the reward, swap the capability. Here's how RL elicits reasoning, code generation, math, and tool use.
2024-06-19
- 4
RL as a Skill Acquisition Engine
The reward signal determines what the model learns to do. Swap the reward, swap the capability. Here's how RL elicits reasoning, code generation, math, and tool use.
2024-06-19
- 5
RL for Agentic Systems
Single-turn RL teaches a model to produce good responses. Agentic RL teaches it to complete multi-step tasks in an environment — with delayed rewards, partial observability, and real consequences.
2024-06-19
- 5
RL for Agentic Systems
Single-turn RL teaches a model to produce good responses. Agentic RL teaches it to complete multi-step tasks in an environment — with delayed rewards, partial observability, and real consequences.
2024-06-19
- 6
GRPO Variants: From GRPO to DAPO, Dr. GRPO, and Beyond
A systematic comparison of GRPO and its descendants — CLIP-DAPO, CISCO, DAPO, Dr. GRPO, GDPO, REINFORCE++ — what each fixes, what trade-offs each makes, and when to use which.
2025-06-20
- 6
GRPO Variants: From GRPO to DAPO, Dr. GRPO, and Beyond
A systematic comparison of GRPO and its descendants — CLIP-DAPO, CISCO, DAPO, Dr. GRPO, GDPO, REINFORCE++ — what each fixes, what trade-offs each makes, and when to use which.
2025-06-20