Self-Instruct: Aligning Language Models with Self-Generated Instructions

The instruction-data bottleneck

Instruction tuning — fine-tuning a language model on (instruction, output) pairs — is what makes a raw model follow directions. But the recipe has an expensive ingredient: the instruction data. Human-written instruction datasets are limited in quantity, narrow in the variety of tasks they cover, and costly to expand, because every new example needs a person to invent a task and write a correct answer. Self-Instruct asks whether a language model can generate this data for itself, turning a small human-written seed into a large, diverse instruction set with almost no further human effort.

Bootstrapping from 175 seeds

The pipeline starts with a tiny seed pool: 175 tasks, each one human-written, consisting of an instruction and one example instance. From there it loops, using the model (the original GPT-3) as both the author and the answerer.

1. Instruction generation. Sample a handful of existing instructions from the pool (a mix of seed tasks and previously generated ones) as few-shot examples, and prompt the model to write a new instruction. This generates fresh tasks, not just fresh answers — the model is inventing new things to do.

2. Classification detection. The model decides whether the new instruction is a classification task, because classification needs a different instance-generation strategy (generate the label first, then an input that fits it) to avoid the model only ever producing one easy class.

3. Instance generation. For each new instruction, the model generates the input/output instance(s) — the actual training data. For non-classification tasks it uses an input-first approach; for classification it uses output-first.

4. Filtering and deduplication. New instructions are added to the pool only if they are sufficiently novel — specifically, an instruction is kept only when its ROUGE-L overlap with every existing instruction is below 0.7. Instructions that are too similar to something already in the pool are discarded. Additional heuristics drop instances that are too long, too short, or obviously degenerate.

Run this loop and 175 seed tasks expand into roughly 52,000 instructions with 82,000 instances. Fine-tuning the original GPT-3 on this self-generated data yields GPT-3$_\text{Self-Inst}$.

The results

On SuperNI (Super-NaturalInstructions), Self-Instruct improves the base GPT-3 by about 33 absolute points, bringing it to near-parity with InstructGPT-001 — even though InstructGPT was trained on private human-written data and human feedback, and Self-Instruct used neither. On a set of novel expert-written instructions, GPT-3$_\text{Self-Inst}$ dramatically outperforms the base model and trails InstructGPT-001 by only a small margin. The deduplication threshold matters: the ROUGE-L filter is what keeps the generated pool diverse instead of collapsing into thousands of paraphrases of the same few tasks.

Why it matters

Self-Instruct showed that the data needed to align a model can largely be produced by a model, with human effort reduced to a couple hundred seed examples and a filtering rule. That idea is the foundation of the synthetic-data alignment wave that followed: Alpaca applied essentially this recipe to a stronger teacher (using text-davinci-003 to generate 52K instructions) and produced a capable instruction-following model from LLaMA for a few hundred dollars, and the broader practice of distilling instruction data from a strong model into a weaker one descends directly from here. The two durable lessons are that generation is cheap but diversity is the scarce resource — hence the explicit deduplication step — and that a surprising fraction of alignment quality comes from the breadth of tasks, not from human-written answers specifically.

Limitations

Self-generated data inherits the generating model's weaknesses: factual errors, biases, and blind spots are reproduced and sometimes amplified, and the model cannot create competence it does not already have — Self-Instruct surfaces and organizes latent ability rather than adding new knowledge. The generated instances are often noisy, with incorrect or low-quality outputs that the filters only partly catch. The diversity of generated tasks, while large, still skews toward what the model finds easy to imagine. And when the recipe is used to distill from a stronger teacher (as Alpaca does), the student is capped by the teacher and can pick up its mistakes wholesale — imitation transfers style and format more reliably than it transfers correctness.