Deep dive

Constitutional AI explained

Training an assistant with a written set of principles and AI feedback.

Updated

Constitutional AI (CAI) is a method developed at Anthropic for training AI assistants to be helpful and harmless using a written set of principles. It was described in the December 2022 paper Constitutional AI: Harmlessness from AI Feedback.

The problem it addresses

Reinforcement learning from human feedback (RLHF) needs large numbers of human judgments, including judgments about harmful content that people would rather not label. It also leaves the model's values implicit in thousands of individual rankings. CAI tries to make the guiding principles explicit and reduce the amount of human labelling needed for harmlessness.

How it works

  1. Supervised phase. The model answers prompts, critiques its own answers against a randomly chosen principle, and revises them. The model is then fine-tuned on the revised answers.
  2. Reinforcement phase. The model compares pairs of responses and picks the one that better follows the principles. These AI-generated preferences train a preference model, which is used for reinforcement learning. This is often called RLAIF (reinforcement learning from AI feedback).

Where the principles come from

The paper drew its principles from several sources, including the Universal Declaration of Human Rights and other published guidelines. Anthropic has since written a longer constitution for Claude describing the values and priorities it wants its models to hold.

Why it matters

Authorship note: Constitutional AI is an Anthropic paper. It reflects the organisation Dario Amodei leads.