DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

The question the paper asks

Every strong reasoning model before R1 was taught to reason by imitation: collect human or model-written chains of thought, fine-tune on them, then maybe polish with reinforcement learning. DeepSeek-R1 asks whether the imitation step is necessary at all. Can a base language model discover how to reason — backtracking, checking its own work, spending more tokens on harder problems — when the only thing it is ever told is whether its final answer was right?

The paper answers in two parts. R1-Zero is the clean experiment: RL straight from the base model with no supervised fine-tuning. R1 is the practical system: a small amount of cold-start data plus a multi-stage pipeline that keeps R1-Zero's reasoning but fixes its rough edges.

R1-Zero: reasoning from pure RL

R1-Zero starts from DeepSeek-V3-Base and applies reinforcement learning directly, with no SFT warmup. The RL algorithm is GRPO (Group Relative Policy Optimization), which drops the value network used by PPO and estimates each completion's advantage from the spread of rewards within a sampled group of answers to the same prompt. The reward is deliberately crude: a rule-based signal that checks whether the final answer is correct (for math, the boxed result must match; for code, the program must pass its tests) plus a format reward that asks the model to put its thinking between <think> tags and its answer between <answer> tags.

No reward model scores the reasoning. No human writes an example chain of thought. The model is only ever told right or wrong at the end. And under that pressure it begins, on its own, to produce longer and longer chains of thought — the average response length climbs steadily over training — and to exhibit behaviors nobody put in a training set: it re-examines earlier steps, tries alternative approaches, and in one widely-quoted transcript pauses with an "aha moment" to reconsider a problem. On AIME 2024 the model's pass@1 rises from 15.6% to 71.0% over the RL run, and to 86.7% with majority voting — into the range of OpenAI's o1.

The lesson is that the capacity to reason is already latent in the base model; RL with a verifiable reward is enough to incentivize it, without any demonstration of what good reasoning looks like.

What R1-Zero gets wrong

R1-Zero is a proof of concept, not a product. Its chains of thought are effective but unpleasant to read: they mix languages mid-derivation, ramble, and ignore formatting conventions. Pure outcome-reward RL optimizes for being right, and nothing in that signal rewards a human-legible explanation. So the reasoning works but the presentation does not.

R1: cold start, then RL, then polish

R1 keeps R1-Zero's engine and wraps it in a four-stage pipeline that trades a little purity for a lot of usability:

1. Cold-start SFT. Fine-tune the base model on a few thousand curated long chain-of-thought examples. This is a small dataset whose only job is to give the model a readable starting format before RL begins, so reasoning does not have to be discovered from a blank slate.
2. Reasoning-oriented RL. Run GRPO as in R1-Zero, now with an added language-consistency reward that penalizes the language-mixing that plagued R1-Zero.
3. Rejection sampling + SFT. Sample many completions from the RL checkpoint, keep the good ones (roughly 600K reasoning samples plus 200K general samples), and fine-tune on them to broaden the model beyond math and code into writing, QA, and other tasks.
4. RL for all scenarios. A final RL stage that combines rule-based rewards for reasoning with preference-style rewards for helpfulness and harmlessness, aligning the model for general use.

The result, DeepSeek-R1, matches OpenAI-o1-1217 on a range of reasoning benchmarks (for example 79.8% on AIME 2024 and 97.3% on MATH-500) while producing clean, readable answers.

Distillation: the reasoning transfers

A striking secondary result is that the reasoning ability can be distilled. Fine-tuning small dense models (Qwen and Llama variants from 1.5B to 70B) on R1's outputs produces models that reason far better than their size suggests — the distilled 32B and 70B models beat much larger non-reasoning baselines. Importantly, the paper finds that distilling from R1 works better than running RL directly on a small model: the large model discovers reasoning patterns through RL that a small model cannot find on its own but can readily imitate once shown.

Why it matters

DeepSeek-R1 reframes how reasoning is taught. The dominant assumption had been that you need high-quality reasoning demonstrations to fine-tune on; R1-Zero shows you can grow reasoning from a verifiable reward alone, and R1 shows the same recipe scales to a polished, general-purpose model. Because the core signal is rule-based correctness rather than a learned reward model, the approach sidesteps much of the reward-hacking fragility of preference-based RLHF, at least on tasks where correctness is checkable. The full open release of weights and the distilled models also made strong reasoning broadly reproducible.

Limitations

The recipe leans on verifiable rewards, so it is most natural on math, code, and other domains with a ground-truth check; extending it to open-ended tasks where correctness is a matter of judgment is harder and still relies on preference rewards with their usual hacking risks. R1 can be sensitive to prompting (few-shot prompts can hurt it), its language-mixing is reduced but not eliminated, and the long chains of thought cost real inference compute. And R1-Zero, for all its conceptual importance, remains too unpolished to deploy on its own — the cold-start and post-training stages are what make the method usable, which is a reminder that pure RL discovers capability but not presentation.