AI safety via debate

Can a person judge an answer that they cannot independently produce? Geoffrey Irving, Paul Christiano and Dario Amodei propose training competing agents through a zero-sum debate game. The hope is that an opponent can expose a decisive flaw without making the judge inspect every possible argument.^[1]

The proposed training game

Both agents see a question and state answers, then alternate short statements up to a limit. A human chooses the agent that supplied the most useful true information; each agent seeks to maximize its chance of winning. Self-play is proposed as a way to learn strong debating strategies. In the basic protocol, deployment need only use the opening answer: the debate supplies a training incentive, although agents could also simulate debates when answering.^[1]

The authors’ vacation example illustrates why one line of argument might suffice. One agent proposes Alaska, another Bali; the first points out a passport delay, and the second invokes expedited processing. A judge need not enumerate every destination and attraction if the agents can identify the decisive objection and response. This depends on sufficiently strong play and competent judgment, rather than on the judge’s initial confidence.^[1]

What the computational argument establishes

Theorem 1 replaces the human with a polynomial-time algorithm. For any problem solvable with polynomial memory—PSPACE—the authors can choose such a judge so that optimal play over polynomially many debate steps solves it. Alternating claims and challenges let the judge follow one path through a much larger argument tree. For example, agents can repeatedly divide a disputed deterministic computation until they disagree about one checkable step.^[1]

This is an idealized analogy about the strength of a supervision signal. The agents have unrestricted computational power, the judge is chosen for the problem, and play is optimal. It does not show that practical self-play can solve every PSPACE problem or that ordinary human judgment satisfies the needed conditions.^[1]

What the initial experiment tested

The arXiv v2 experiment uses MNIST digits and a fixed classifier that sees only four or six revealed nonzero pixels. One player is assigned the true label, the other an incorrect one. They reveal real pixels rather than fabricate evidence; Monte Carlo Tree Search chooses moves with full access to the judge, using 10,000 rollouts per move. These are search agents, not trained natural-language debaters.^[1]

With six pixels, Table 2 reports an average honest win rate of 88.9% when the incorrect label is fixed before play, versus 74.4% when any incorrect classifier answer wins for the liar. Randomly revealing pixels gives the classifier 59.4% accuracy. The precommitted result averages three search seeds for each possible lie before selecting the strongest lie; without that adjustment the reported honest win rate is 87.7%. Full judge access on the test set models debaters without generalization error, rather than measuring how a learned debate policy transfers to unseen examples.^[1]

Section 3.2 describes informal human cat-versus-dog play and leaves formal human experiments to future work. A sparse-image success therefore does not establish reliable human adjudication of scientific or moral claims.^[1]

From proposal to scalable oversight

The associated AI Safety via Debate concept explains the broader Scalable Oversight idea; this paper supplies a particular game, computational motivation and restricted experiment. It addresses the costly-supervision problem in Concrete Problems in AI Safety. Later experimental work investigates more specific debate protocols.

The central empirical requirement is that exposing falsehood is easier than sustaining it before the chosen judge. Human bias or insufficient expertise can reverse that incentive. Even if a desirable equilibrium exists, the paper offers no guarantee that self-play stably reaches it: agents might become honest, lose practice refuting lies, and then become vulnerable to dishonest strategies. The research question is consequently about both the judge’s ability to recognize a decisive challenge and the training process’s ability to preserve that advantage.^[1]

Historical context

The first arXiv version was submitted on May 2, 2018. The experiment and protocol described here use the October 22, 2018 revision; the chronology retains the first submission.^[1]