DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

The critic is the bottleneck

PPO needs a baseline to turn raw returns into advantages, and that baseline comes from a learned value function $V_\phi(s)$. In RLHF the value network is a second transformer, typically the same size as the policy, trained alongside it. So a 70B policy implies a 70B critic, and both must hold parameters, gradients, and optimizer state in GPU memory at once. The value network is pure training overhead — it is discarded at inference — yet it roughly doubles the memory footprint of the run and adds its own forward and backward pass to every step. At the scale where reasoning models become interesting, that second copy is often what you cannot afford.

Estimating advantage from a group

GRPO removes the critic by replacing the learned baseline with an empirical one. For a prompt $q$, sample a group of $G$ completions $\{o_1, \dots, o_G\}$ from the current policy and score each with the reward model to get $\{r_1, \dots, r_G\}$. The advantage of a completion is just its reward standardized within the group:

$$\hat{A}_i = \frac{r_i - \operatorname{mean}(r_1, \dots, r_G)}{\operatorname{std}(r_1, \dots, r_G)}$$

Every token in completion $o_i$ inherits this same scalar $\hat{A}_i$, so a completion that beats its siblings gets a positive signal across all of its tokens and one that trails them gets a negative one — with no value network anywhere in the computation.

The objective

GRPO keeps PPO's clipped surrogate verbatim; only the source of $\hat{A}$ changes. Writing $r_{i,t}(\theta) = \pi_\theta(o_{i,t} \mid q, o_{i,<t}) / \pi_{\theta_\text{old}}(o_{i,t} \mid q, o_{i,<t})$ for the per-token probability ratio, the loss averages the clipped objective over the tokens of each completion and the completions of each group, with an explicit KL penalty pulling the policy toward a reference:

$$\mathcal{L}^{\text{GRPO}}(\theta) = -\,\mathbb{E}\!\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\!\Big(r_{i,t}(\theta)\,\hat{A}_i,\; \operatorname{clip}\big(r_{i,t}(\theta),\, 1-\epsilon,\, 1+\epsilon\big)\,\hat{A}_i\Big)\right] + \beta\, D_{\text{KL}}\!\big(\pi_\theta \,\|\, \pi_\text{ref}\big)$$

The clip term plays exactly the role it does in PPO — it caps how far each update can move the ratio from 1 — while the $\beta$ term keeps the policy from drifting away from the reference model and degenerating, a job PPO usually folds into the reward instead.

Why the group mean is a valid baseline

A baseline reduces the variance of a policy gradient without biasing it, as long as it does not depend on the action being scored. The value function $V(s)$ is the optimal such baseline: it is the expected reward over actions from state $s$. The group mean $\operatorname{mean}(r_1, \dots, r_G)$ is a Monte Carlo estimate of exactly that expectation — average reward over $G$ completions drawn from the policy at prompt $q$ — so subtracting it answers "was this completion better or worse than what this policy typically produces here?" Dividing by the group standard deviation rescales that comparison to unit variance, which keeps the gradient magnitude stable across prompts of wildly different difficulty.

What you give up

The group baseline buys its memory savings with resolution. Because every token of $o_i$ shares one scalar $\hat{A}_i$, GRPO supplies an outcome-level advantage: it can say a whole completion was good but not which step inside it earned the credit. A PPO critic, or a process reward model that scores intermediate steps, can hand back a different $\hat{A}_t$ at each token and assign blame to the specific line where a derivation went wrong. That finer signal matters when the reward is dense. But for the tasks GRPO targets — a math answer is right or wrong, a program passes its tests or fails them — the reward is a single bit delivered at the end, and there is no intermediate truth for a token-level estimator to recover. When the reward itself is outcome-level, an outcome-level advantage loses nothing.

Choosing G

The group size $G$ trades estimator quality against inference cost. The group mean and standard deviation are sample statistics over $G$ draws, so their error shrinks like $1/\sqrt{G}$ — a larger group gives a more accurate baseline and a less noisy advantage — but it also multiplies the number of completions you must generate per prompt, and generation, not the gradient step, dominates the wall-clock of RL on language models. In practice $G = 8$ or $G = 16$ sits at the knee of that curve: enough samples for the within-group statistics to be meaningful, few enough that sampling stays affordable.