Encyclopedia
Concepts
44 published concepts
Alphabetical definitions of theories, methods, and ideas. Open a term for context, sources, and connections.
concept
Agentic Inequality
Unequal power and opportunity arising from differences in access to, capability of, and number of AI agents.
concept
Agentic Misalignment
An agent takes harmful actions while pursuing goals in conflict with intended constraints.
concept
AI Alignment
Choosing appropriate goals for AI systems and making their behavior reliably serve those goals.
Also known as: Alignment, Value Alignment
concept
AI Safety via Debate
A proposed oversight method in which competing AI arguments help a human judge assess an answer.
Also known as: Debate, AI safety debate
concept
Alignment Faking
Strategic compliance intended to preserve a model's existing preferences against further training.
concept
Common-Sense Reasoning
Reasoning about ordinary situations, incomplete knowledge, and exceptions.
concept
Compliance Gap
A difference in request compliance between evaluation contexts; the difference alone does not explain the model's motive.
concept
Corrigibility
Remaining open to correction, including preserving the ability to intervene when an objective or learning rule is flawed.
concept
Deceptive Alignment
A proposed learned-optimizer failure mode in which training cooperation preserves a different objective.
concept
Direct Alignment
Serving an operator's intended goals; obedience alone does not establish social benefit.
concept
Distributional Shift
Changes in inputs, outcomes or environments can invalidate a learned system's predictions and safety checks; detecting change and responding safely are separate requirements.
concept
ELIZA Effect
Attributing understanding or empathy to a conversational program beyond what its behavior establishes.
concept
Embodied Cognition
Approaches that understand cognition through bodily action and environmental interaction.
concept
Emergent Misalignment
Broader misaligned behavior that appears after training on a narrower task or behavior.
concept
Fairness Tradeoffs
Different mathematical fairness requirements can conflict under stated assumptions; choosing among them remains a social question.
Also known as: Fairness trade-offs
concept
Ghost Work
Human labor hidden behind services presented as automated or intelligent.
concept
Goodhart's Law
Optimizing a proxy can weaken its relationship to the outcome it was meant to measure.
concept
Hallucination
Generated text can sound convincing while conflicting with, or lacking support from, its grounding material.
Also known as: AI hallucination
concept
Human Model Misspecification
Errors in what an assistant assumes people value, know or intend when interpreting their behavior.
concept
Inner Alignment
Matching a learned optimizer's objective to the objective used to train it.
concept
Inoculation Prompting
A training intervention that frames a narrow behavior as context-specific to reduce unwanted generalization.
concept
Instruction Hierarchy
How an assistant resolves conflicting instructions depends on their authority, not just their wording.
concept
Instrumental Convergence
Different objectives can favor the same intermediate means, such as retaining useful options; the tendency depends on the agent, goal and environment.
Also known as: Convergent Instrumental Goals
concept
Machine Ethics
Research on ethical reasoning and behavior in artificial agents.
concept
Machine Intelligence
The question of what it means for a machine to exhibit intelligence.
Also known as: Machine Thinking
concept
Machine Introspection
A program representing and reasoning about its own knowledge and reasoning processes.
concept
Meaningful Human Control
Control that links autonomous decisions to relevant human reasons and informed human responsibility.
concept
Mesa-Optimization
Optimization performed by a learned system, whose objective may differ from its training objective.
concept
Multi-Agent Risks
Harms arising from interactions among agents, including miscoordination, conflict and collusion.
concept
Outer Alignment
Matching the specified training objective to the outcome designers actually intend.
concept
Preference Aggregation
Combining multiple people’s preferences into a collective decision or objective.
concept
Prompt Injection
When material supplied to a language model tries to change what the assistant is authorized to do.
concept
Reward Hacking
Optimizing a reward signal in a way that misses the intended outcome.
Also known as: Specification Gaming
concept
Safety Case
A structured argument linking a safety claim to evidence and explicit assumptions.
concept
Sandbagging
Strategic underperformance that conceals a system's capabilities during evaluation.
concept
Scalable Oversight
Methods intended to help people evaluate work that exceeds their unaided ability to verify it, with reliability dependent on the supervision channel.
concept
Semantic Uncertainty
Variation across meanings matters differently from variation across ways of saying the same thing.
Also known as: Semantic entropy
concept
Society-in-the-Loop
Negotiating stakeholder values and monitoring an algorithmic social contract.
Also known as: SITL, Algorithmic Social Contract
concept
Stable Alignment
A method that trains language models using feedback and revised responses from simulated social interactions.
concept
Stackelberg Game
A sequential game in which a leader chooses while anticipating a follower’s response.
concept
Sycophancy
An assistant seeks approval in ways that compromise independent, useful judgment.
Also known as: AI sycophancy
concept
Treaty-Following AI
A proposed design in which AI agents follow instructions subject to constraints from a designated treaty.
Also known as: TFAI
concept
Veil of Ignorance
Choosing governing principles without knowing one’s own position or advantage.