Training Language Models to Follow Instructions with Human Feedback

The gap between a language model and a useful assistant

GPT-3 was trained to do one thing: predict the next token on web text. That objective makes it a remarkable autocomplete engine, but it is misaligned with what people actually want from it. Asked to follow an instruction, a raw language model often continues the prompt in style rather than obeying it, makes things up, or produces toxic or unhelpful text — because none of those failures cost it anything during pretraining. InstructGPT is the paper that turned "predict the next token" into "do what the user asked," and the pipeline it introduced became the template for essentially every aligned chat model that followed.

The three-stage pipeline

The method has three stages, run in sequence.

1. Supervised fine-tuning (SFT). Human contractors write high-quality demonstrations of the desired behavior — given a prompt, what a good response looks like. The base model is fine-tuned on these demonstrations. This teaches the model the format of helpful, instruction-following answers and gives the later stages a reasonable starting policy.

2. Reward model (RM) training. For a set of prompts, the SFT model samples several candidate responses, and human labelers rank them from best to worst. A reward model — a language model with a scalar output head — is trained to predict these human preferences. The loss is a pairwise ranking loss: for a pair where response $y_w$ was preferred over $y_l$,

$$L_\text{RM} = -\,\log \sigma\!\big(r_\theta(x, y_w) - r_\theta(x, y_l)\big),$$

so the model learns to assign a higher scalar to the response humans liked. Ranking is used instead of absolute scores because people are far more consistent at saying which is better than at assigning a number.

3. RL with PPO. The SFT policy is now optimized against the reward model using Proximal Policy Optimization. The model generates a response, the RM scores it, and PPO nudges the policy toward higher-reward outputs. A crucial detail is the per-token KL penalty that keeps the policy close to the SFT model:

$$\text{objective} = \mathbb{E}\big[r_\theta(x, y)\big] - \beta\, D_\text{KL}\!\big(\pi_\text{RL} \,\|\, \pi_\text{SFT}\big).$$

Without it, the policy quickly learns to exploit quirks of the frozen reward model, drifting into degenerate text that scores well but reads badly — the canonical reward-hacking failure. The KL term anchors the policy to coherent language while still letting it climb the reward.

The headline result

The finding that made the paper famous: on the OpenAI API prompt distribution, human labelers prefer the 1.3B-parameter InstructGPT to the 175B-parameter GPT-3, despite the aligned model being more than 100× smaller. Alignment, in other words, bought more usable quality than two orders of magnitude of scale. InstructGPT also produces more truthful answers (roughly double the rate on TruthfulQA), generates less toxic output when asked to be respectful, and follows explicit constraints in prompts far more reliably.

It does all of this with only small regressions on standard NLP benchmarks — the so-called "alignment tax" — and the paper shows that mixing pretraining gradients into the RL stage (PPO-ptx) largely closes even that gap.

Why it matters

Before InstructGPT, "alignment" was mostly a research aspiration; after it, it was a recipe. The SFT → RM → PPO pipeline is the direct ancestor of ChatGPT and of the RLHF stage in nearly every subsequent instruction-tuned model. Two ideas in particular proved durable: that human preferences are easier to collect as rankings than as demonstrations or scores, and that a learned reward model lets you optimize against human judgment at a scale no human could label directly. The per-token KL-to-SFT regularizer is still the standard defense against reward hacking, and it reappears almost verbatim in later methods — it is the same anchoring role the reference model plays in DPO and the $\beta$ term plays in GRPO.

Limitations

The pipeline is expensive and operationally heavy: it depends on a continuous stream of human comparison labels, trains and serves multiple large models at once (policy, reference, reward, and a value head), and inherits PPO's hyperparameter sensitivity. The alignment target is only as good as the labelers — InstructGPT is aligned to the preferences of a specific contractor pool and the researchers' instructions, not to any universal notion of helpfulness, and the paper is explicit that "whose values" is an unsolved question. The reward model remains a proxy that can be gamed, so the model is helpful and more truthful but still hallucinates and can be steered into harmful output. These costs are exactly what the later DPO and RLAIF lines of work set out to reduce.