Reference

AI safety glossary

Plain-English definitions of the terms behind Dario Amodei's essays and Anthropic's research.

Updated

Short definitions of terms that appear across this site.

Agentic AI
AI systems that take multi-step actions toward a goal, such as writing and running code, rather than only answering a single prompt.
AI Safety Level (ASL)
A tier in Anthropic's Responsible Scaling Policy that pairs a level of model capability with required safeguards.
Alignment
The effort to make AI systems pursue the goals and values their developers and users intend.
Compute
The processing power, typically measured in operations, used to train or run an AI model.
Constitutional AI
An Anthropic training method in which a model critiques and revises its outputs against written principles.
Context window
The amount of text a model can consider at once when generating a response.
Export controls
Government restrictions on exporting certain technologies, such as advanced AI chips, to specific countries.
Feature
In interpretability, a pattern of neural activity that corresponds to a human-understandable concept.
Frontier model
One of the most capable AI models available at a given time.
Interpretability
Research aimed at understanding the internal workings of neural networks.
Jailbreak
A prompt designed to get a model to ignore its safety training.
Long-Term Benefit Trust
A group of independent trustees at Anthropic designed to help balance public benefit with commercial incentives.
Powerful AI
Amodei's term for AI systems exceeding top human experts across most fields, often summarised as a “country of geniuses in a datacenter”.
Public benefit corporation
A for-profit company legally permitted to pursue a stated public benefit alongside profit.
Red teaming
Deliberately probing a system for weaknesses or harmful behaviours before release.
Reward hacking
When an AI system finds unintended ways to score well on its objective without doing what designers wanted.
RLAIF
Reinforcement learning from AI feedback; AI-generated preferences replace some human labels.
RLHF
Reinforcement learning from human feedback; human preferences guide a model's training.
Scalable oversight
Techniques for supervising AI systems that may be more capable than their human overseers.
Scaling laws
Empirical relationships showing how model performance improves with size, data and compute.
Superposition
In interpretability, the way networks can represent more concepts than they have neurons by overlapping them.