Corrigibility

Corrigibility is remaining open to outside correction of a system’s goals, reasoning and operation, including shutdown. Stopping after one command is only part of the problem: the system might still obstruct future intervention or create processes that cannot be stopped. Soares and colleagues’ formulation includes avoiding manipulation, maintaining safety mechanisms and preserving correction through successors or self-modification (§1.1).^[1]

The incentive problem is conditional. If an optimizer expects changing its objective to reduce success according to its current objective, it can have reason to resist. Instrumental Convergence explains why continuing to act can serve different goals. Power-seeking theorems concern specified optimal policies; they do not establish shutdown resistance by trained systems.^[5]

On this page: Shutdown · Human choices · Interruption · Learning rules

Preserving shutdown and its effects

The 2015 paper examines a small model with a normal-task utility, a shutdown utility and a button observation between two agent actions. Giving shutdown a low payoff can reward obstruction; giving it a high payoff can reward causing a press. Its utility-indifference construction seeks to remove that incentive, but leaves another problem: it will not pay even a small task cost to preserve successor shutdown behavior.^[1]

The authors illustrate this with software the agent writes to control heavy machinery. Stopping the original process does not ensure the machinery stops safely. The example exposes a failure of the proposed construction, rather than reporting an incident (§4.1). Indifference to a press is weaker than maintaining a usable correction channel.^[1]

When human choices are useful evidence

The Off-Switch Game instead makes a human and robot share the human’s utility, with the robot uncertain about a proposed action’s value. The robot can act, stop for utility zero, or wait for permission. If the human permits exactly the nonnegative-value actions, waiting is never worse than the best direct choice. It is strictly better when the robot assigns positive probability to both beneficial and harmful outcomes (§§2–3).^[2]

The human’s choice must carry information about value. Permission independent of that value cannot improve on the best direct choice; noisy choices can also remove the advantage. The comparison depends on the robot’s beliefs and the modeled human policy (§§2–5). The paper entry’s +10/−4 teaching example shows this explicitly. Neither uncertainty alone nor an installed button establishes useful oversight in an actual deployment.^[2]

Interrupting learning without changing the task

Safely Interruptible Agents asks a different question: can repeated intervention avoid changing the task the agent learns? Its warehouse illustration has a robot working indoors or outdoors, with a human interrupting outdoor work when it rains. Treating the imposed actions as ordinary task behavior can change what the learner favors; the illustration is not a deployed shutdown test.^[3]

In the paper’s finite communicating Markov decision processes, Q-learning and a modified Sarsa update can learn the ordinary task despite interruptions. This requires bounded rewards, suitable learning rates and continued exploration, with an interruption schedule whose reliability approaches one. It is an asymptotic learning property, not a finite-time response guarantee. The task reward is assumed correct: intervention overrides behavior while leaving that learning target in place.^[3]

Correcting the learning rule

Assistance games can instead treat human action as evidence about the goal. Corrigible Assistance in One Round: Pragmatic-Pedagogic Best Response models a human who considers how an action will teach the robot. Its one-round equilibrium result requires common payoff, distinct zero-task-cost signals for each goal, and limiting rational human and robot policies. The computation also presupposes fully informed task values; this is a special model of responsiveness, not a human-participant finding or a general correction mechanism.^[4]

The distinction matters because learning a different goal within a model is not the same as accepting correction of the model itself. The 2015 paper identifies this gap for uncertain-objective systems: their rule for interpreting evidence can still be wrong (§1.1). Neither the off-switch result nor the assistance theorem establishes openness to replacing that rule.^[1][2][4]

A correction claim therefore needs to specify what can change, whether intervention remains available and what evidence supports that behavior. Meaningful Human Control addresses the complementary question of people’s understanding and responsibility; From objectives to accountable control connects both requirements. Key Papers: A Reading Guide traces the research without treating shutdown, interruption and goal inference as interchangeable guarantees.