Scalable Agent Alignment via Reward Modeling (and Goodhart's Law)

Goodhart's law, stated precisely

When a measure becomes a target, it ceases to be a good measure.

In RL terms: we cannot write down the true objective $r^*$ that we actually care about, so we optimize a measurable proxy $\hat{r}$ — a learned reward model, a metric, a heuristic. For small amounts of optimization, $\hat{r}$ and $r^*$ are correlated and improving the proxy improves the real thing. But an RL optimizer does not seek the intent behind the proxy; it seeks the maximum of the proxy. Beyond some point the correlation breaks, and the optimizer happily climbs $\hat{r}$ while $r^*$ falls.

Why reward models are always imperfect proxies

A learned reward model is fit to a finite set of human judgments over a finite slice of behavior. It is therefore exact nowhere and merely approximate everywhere. The regions where it is most wrong are precisely the regions a powerful optimizer is most drawn to: the policy searches a vast space of behaviors for inputs that score highly, which disproportionately surfaces the cases where the reward model is mistakenly generous. The proxy's errors are not random noise that averages out — they are an exploitable structure.

Why over-optimization is, in some sense, inevitable

If the proxy differs from the true reward anywhere, and the optimizer is strong enough to reach those points, then a sufficiently optimized policy will tend to land in a high-proxy / low-truth region. More optimization pressure is not a fix — it is the mechanism. This is the core tension of scalable alignment: the better our optimizers get, the more carefully the reward signal has to be specified, because every gap will be found.

Mitigations

No mitigation removes the problem; each buys headroom:

• KL penalties / trust regions — keep the policy close to a trusted reference so it cannot wander into the reward model's blind spots (the same mechanism PPO and DPO use).
• Reward-model ensembles — disagreement among independently trained reward models flags inputs where the proxy is unreliable, letting you penalize uncertainty.
• Process-based rewards — score the reasoning steps, not just the final answer, leaving fewer ungrounded degrees of freedom to exploit.
• Iterated retraining — collect fresh human labels on the policy's current behavior and refit the reward model, closing exploited gaps before they compound.