Writeup: Progress on AI Safety via Debate

Beth Barnes and Paul Christiano report OpenAI’s Reflection-Humans work from late 2019, developing AI Safety via Debate for non-expert oversight.^[1]

Human experiments

Using tricky physics questions, expert debaters defended a correct answer and an inferior alternative. The questions came from Thinking Physics: conceptually tricky problems with clear answers that the debaters understood but judges often did not. Early debates remained unreliable even with motivated judges spending about an hour. The later target—over 90% correct judgments within ten minutes—was an objective, not an achieved result.^[1]

Protocol changes

Dishonest debaters could evade questions or gain control of the explanation’s framing. To reduce these advantages, the structured protocol gave both sides simultaneous constructive arguments and objections. A challenged claim–objection pair became the next round’s dispute, with a reduced depth limit; the eventual decision turned on that narrower disagreement. The experimental rules conservatively let the dishonest debater choose which challenged objection to pursue.^[1]

Recursion exposed a further problem: a debater could reinterpret an ambiguous statement after seeing the objection. In the writeup’s prime-number example, counting primes “between” adjacent intervals can double-count 5. Cross-examination first makes the debater specify the interval; that answer can then expose either a false subclaim or a faulty decomposition. The example illustrates a protocol vulnerability, rather than a measured error rate.^[1]

The ideal cross-examination queries noncommunicating copies of an opponent at the time of an earlier statement. Questions and answers can be gathered outside the main transcript, with a selected exchange shown to the judge. Human approximations use backtracking until answers become predictable or separate team members; the paired-team implementation was untested. These are attempts to enforce consistent commitments while economizing on judge attention.^[1]

Why finding a flaw matters

The writeup’s “Long computation problem” distinguishes knowing an answer is wrong from finding a small error a judge can assess. If neither debater understands a long supporting argument well enough to isolate its flaw, a dishonest debater may force a draw. Recursion alone does not supply the missing ability.^[1]

The “Current concerns” section also questions how judges should update after objections are conceded. Debaters choose objections in light of their winning chances, so concessions do not straightforwardly certify truth. The authors worry that rewarding completely honest play might fail to provide a useful training gradient toward less dishonesty. These are unresolved protocol and learning concerns, not demonstrated failures of trained superhuman models.^[1]

Significance and limits

The writeup develops the 2018 proposal toward practical Scalable Oversight. Unresolved issues include judging conceded objections, training dynamics and arguments too complex for either debater to locate an error. Human experiments do not establish reliable superhuman oversight.^[1]

Historical context

Released February 5, 2020; the research occurred in Q3–Q4 2019.^[1]