Should Robots be Obedient?

Smitha Milli, Dylan Hadfield-Menell, Anca Dragan and Stuart Russell model a robot learning shared reward parameters from a human’s orders. The chronology marks first arXiv submission; this entry reviews the IJCAI 2017 proceedings original.^[1]

Orders as evidence

The supervision model gives the human and robot the same reward, but only the human knows its static parameters. The robot observes an order and then chooses its own action. Rewards are linear combinations of known features; human choices follow a specified policy that may be noisy. Most analysis then restricts the general model to independent rounds: an action cannot change later states, so exploration and delayed consequences disappear.^[1]

The authors define obedience as the probability of executing the order, and autonomy advantage as the expected reward difference between the robot’s action and that order. Under the correct model, maximizing reward for the posterior mean of the reward parameters gives a nonnegative expected advantage on each round. A positive advantage requires sometimes departing from the order. With a strongly consistent parameter estimate, asymptotic behavior approaches reward-optimal action selection and obeys when the human orders that action. These are claims about the modeled shared reward, rather than a general license to override people.^[1]

Why the estimator matters

The paper compares posterior-mean, maximum-a-posteriori and maximum-likelihood estimates. An order is undominated if some allowed reward parameters make it optimal. Under the paper’s noisy-rational model, a maximum-likelihood learner obeys an undominated first order; other estimators can reject it because of their priors. The result concerns initial behavior, not obedience to every later instruction.^[1]

Maximum-likelihood action selection is also invariant to using the wrong positive noise-temperature parameter under the theorem’s assumptions: rescaling inferred rewards preserves action rankings. The posterior-mean policy lacks that robustness. This does not establish robustness to arbitrary errors in a human model.^[1]

Missing reasons to obey

In the simulated linear-feature setting, omitting reward features can make the learner both less obedient and worse than following orders. Adding irrelevant features instead makes it more obedient in the reported comparison, while still allowing convergence to the maximum advantage. The contrast matters because a learner can become confident within an inadequate space of possible preferences.^[1]

The authors exploit the first-order result to build a restricted diagnostic. During an initial period the robot actually obeys, while checking whether separate first-order maximum-likelihood fits would reproduce those orders. If too few do, it continues obeying; otherwise it switches to inferred-reward optimization. Figure 5 uses ten initial orders and reports improved simulated reward when features are missing. This test depends on the estimator’s first-order property and the feature model; it does not detect every omitted human concern.^[1]

Historical context and limits

The authors connect their analysis to The Off-Switch Game and Safely Interruptible Agents. The distinction is between treating instructions as evidence about a goal and ensuring an intervention does not distort learning. Human Model Misspecification explains why reward learning alone does not establish Corrigibility: a model can omit precisely the reason for a correction. The paper’s formal results and simulations assume a shared, static, linear reward; they do not settle authority, rights or disagreement among affected people.^[1]