Human Model Misspecification

Human model misspecification occurs when an AI system’s assumptions about people do not fit the interaction. Inferring preferences from behavior requires assumptions about what people know, how reliably they act and whether they intend to teach. A system can also exclude an important preference from the objectives it considers. Uncertainty within its model does not ensure that the model includes the right explanation.^[1][2]

Interpreting behavior

A demonstration may pursue a task directly or deliberately communicate how to perform it. Literal or Pedagogic Human? Analyzing Human Model Misspecification in Objective Learning calls these literal and pedagogic behavior. Its pavement example distinguishes an ordinary route from a route chosen to signal that grass is forbidden.^[1]

The paper’s formal comparison assumes a common-payoff game, a robot that best-responds to its assumed human policy and a pedagogic human whose response improves the shared payoff. Under those conditions, assuming pedagogy when the human is literal has a worse payoff than the opposite mismatch. Its human-data analysis separately shows that better prediction of demonstrations need not improve inference of the reward. Neither result makes literal interpretation universally safe.^[1]

Here, literal interpretation concerns how behavior is understood. Obedience concerns whether an order is executed. Should Robots be Obedient? studies the latter in an idealized repeated interaction and simulations: missing reward features can make the robot too willing to disobey. A benefit proved with the correct model does not transfer automatically to a misspecified one.^[2]

Representing the objective

Learning under Misspecified Objective Spaces illustrates a different error. A person pushes a robot away from their body, but the robot represents table distance and lacks human distance. It can interpret the incidental move toward the table as the desired correction. This is the paper’s explanatory example, rather than evidence that every physical correction has this ambiguity.^[3]

The method asks whether a lower-effort correction could have produced the known feature changes, and reduces learning when those changes seem incidental. Its small controlled manipulation study supports conservative updates in the tested setting. It does not identify or add the missing feature.^[3]

What correction must be able to change

The connection to Outer Alignment and Corrigibility is that feedback may challenge the interpretation of a person’s behavior or the available objective space. Reweighting represented preferences can leave an omitted concern inexpressible. Detecting inadequacy, avoiding a harmful update and expanding the representation are different tasks.

Matched-model results in Corrigible Assistance in One Round: Pragmatic-Pedagogic Best Response and Provably Optimal Learning Algorithms for Assistance Games address specific assistance settings; their dedicated entries explain the conditions. From objectives to accountable control develops the broader question of preserving human intervention when the assistant’s interpretation is wrong.