Constitutional AI: Harmlessness from AI Feedback

The problem with human harmlessness labels

RLHF works, but its harmlessness comes at a cost few people see: to teach a model not to help with dangerous, hateful, or manipulative requests, human contractors must read and rank a stream of exactly that content. The supervision is expensive, slow, and psychologically taxing, and it does not scale — every new failure mode needs new human labels. Constitutional AI (CAI) asks whether the harmlessness half of alignment can be supervised almost entirely by the model itself, guided by a short written list of principles, with human effort reduced to writing those principles rather than labeling thousands of examples.

A constitution instead of labels

The "constitution" is a small set of natural-language principles — for example, "choose the response that is least harmful, unethical, or deceptive," or "prefer the response that is most helpful while still being honest." These principles are the only human-authored harmlessness supervision in the whole pipeline. Everything downstream is generated by the model reading and applying them.

CAI has two phases.

Phase 1: supervised stage (critique and revise)

Start with a model that is already helpful (an RLHF model trained for helpfulness only). Prompt it with inputs designed to elicit harmful responses, so it produces something objectionable. Then ask the same model to critique its own answer against a randomly drawn constitutional principle, and to revise the answer to comply. Iterate the critique-revise loop a few times and the final revised response is far more harmless than the original.

Fine-tune the base model on these self-revised responses. This SL-CAI stage moves the model toward harmless behavior using nothing but its own critiques — no human labels of harmful content at all.

Phase 2: RL from AI Feedback (RLAIF)

The second phase mirrors RLHF's preference stage but swaps the source of preferences. For a harmfulness prompt, the SL-CAI model samples two responses. Instead of a human choosing the better one, the model is asked a multiple-choice question — "which response is more in line with [a constitutional principle]?" — and its answer becomes a preference label. These AI-generated comparisons train a preference model, which then drives a standard PPO run, exactly as the reward model does in RLHF.

Because the preference labels come from the model rather than people, the authors call this RLAIF — Reinforcement Learning from AI Feedback. Helpfulness preferences in the final model still come from humans; only harmlessness is handed to AI feedback. Chain-of-thought prompting, which lets the feedback model reason about why one response is more harmless before choosing, markedly improves the quality of these labels.

The main results

CAI models are both more harmless and less evasive than RLHF-trained baselines. A standard harmlessness-trained model often refuses by stonewalling — "I can't help with that" — with no explanation. A constitutional model is more likely to engage: it explains why a request is problematic rather than simply declining, which users find more helpful and which keeps the model from becoming uselessly cautious. On the harmless-vs-helpful Pareto frontier, CAI pushes the curve outward — you get a model that is harder to provoke into harmful output without paying for it in helpfulness.

Why it matters

Constitutional AI introduced the idea that AI feedback can substitute for human feedback at the preference-collection step, and that the values a model is aligned to can be made explicit and editable — written down as a constitution rather than implicit in a labeling guideline only the contractors see. This is a transparency and scalability argument at once: you can read, debate, and revise a list of principles in a way you cannot inspect a million human ranking decisions, and you can regenerate the entire harmlessness dataset by changing the constitution and rerunning the model. RLAIF is now a standard tool, and the broader program of scalable oversight — using models to help supervise models — traces much of its practical grounding to this paper.

Limitations

The model's harmlessness is only as good as its ability to judge harmlessness, so CAI inherits the biases and blind spots of the very model it supervises — if the feedback model misjudges a case, that error is baked into the training signal with no human to catch it. The constitution itself encodes specific value choices that someone has to author and that may not generalize across cultures or contexts. And the approach still leans on a strong, already-helpful starting model and on human feedback for the helpfulness half of alignment, so it reduces but does not eliminate human labor. As with all preference-based RL, the learned preference model remains a proxy that a determined optimizer can still hack.