Blog Post··6 min read
Cheatsheet: RL Loss Functions
PPO and GRPO loss functions annotated term-by-term — the clipped surrogate, GAE, value loss, entropy bonus, and group-normalised advantages — with links to the posts explaining each design choice.
Tag: rl
Blog Post··6 min read
Cheatsheet: RL Loss Functions
PPO and GRPO loss functions annotated term-by-term — the clipped surrogate, GAE, value loss, entropy bonus, and group-normalised advantages — with links to the posts explaining each design choice.
Blog Post··7 min read
GRPO vs PPO: Why Removing the Value Head Changes Everything
GRPO achieves competitive alignment results without a value function. Here's exactly what changes in the math and implementation, and why that matters for training efficiency and stability.
Blog Post··7 min read
The Full PPO Training Step: Value Loss, Entropy, and GAE
ppo-loss-per-token covered the clipped surrogate objective. This post covers what surrounds it: how the value function is trained, where the advantage estimates come from, and why the entropy bonus exists.
Blog Post··6 min read
How PPO Computes Loss Over a Language Model Output
Most explanations of PPO stay at the algorithm level. This post goes one level deeper: how the surrogate loss is actually computed token by token for a language model response.
Blog Post··10 min read
RL for Agentic Systems
Single-turn RL teaches a model to produce good responses. Agentic RL teaches it to complete multi-step tasks in an environment — with delayed rewards, partial observability, and real consequences.
Blog Post··7 min read
Learning from Feedback: RLHF, RLAIF, and Beyond
RLHF is three steps: supervised fine-tuning, reward model training, and policy optimization. Each step has a specific failure mode. Here's the full picture.
Blog Post··6 min read
Policy Gradients: The Math Behind RLHF
The policy gradient theorem lets you differentiate through a reward signal you can't backprop through. Here's the derivation and why it works.
Blog Post··7 min read
RL as a Skill Acquisition Engine
The reward signal determines what the model learns to do. Swap the reward, swap the capability. Here's how RL elicits reasoning, code generation, math, and tool use.
Blog Post··5 min read
Why Language Models Need Reinforcement Learning
Supervised fine-tuning teaches a model to imitate. Reinforcement learning teaches it to optimize. The difference turns out to matter enormously.
Blog Post··7 min read
GRPO vs PPO: Why Removing the Value Head Changes Everything
GRPO achieves competitive alignment results without a value function. Here's exactly what changes in the math and implementation, and why that matters for training efficiency and stability.
Blog Post··7 min read
The Full PPO Training Step: Value Loss, Entropy, and GAE
ppo-loss-per-token covered the clipped surrogate objective. This post covers what surrounds it: how the value function is trained, where the advantage estimates come from, and why the entropy bonus exists.
Blog Post··6 min read
How PPO Computes Loss Over a Language Model Output
Most explanations of PPO stay at the algorithm level. This post goes one level deeper: how the surrogate loss is actually computed token by token for a language model response.
Blog Post··10 min read
RL for Agentic Systems
Single-turn RL teaches a model to produce good responses. Agentic RL teaches it to complete multi-step tasks in an environment — with delayed rewards, partial observability, and real consequences.
Blog Post··7 min read
Learning from Feedback: RLHF, RLAIF, and Beyond
RLHF is three steps: supervised fine-tuning, reward model training, and policy optimization. Each step has a specific failure mode. Here's the full picture.
Blog Post··6 min read
Policy Gradients: The Math Behind RLHF
The policy gradient theorem lets you differentiate through a reward signal you can't backprop through. Here's the derivation and why it works.
Blog Post··7 min read
RL as a Skill Acquisition Engine
The reward signal determines what the model learns to do. Swap the reward, swap the capability. Here's how RL elicits reasoning, code generation, math, and tool use.
Blog Post··5 min read
Why Language Models Need Reinforcement Learning
Supervised fine-tuning teaches a model to imitate. Reinforcement learning teaches it to optimize. The difference turns out to matter enormously.