Literal or Pedagogic Human? Analyzing Human Model Misspecification in Objective Learning

Smitha Milli and Anca D. Dragan examine whether a learner should assume a person is performing a task or teaching it. This entry reviews the PMLR proceedings original for UAI 2019, published in the 2020 volume; the chronology marks the first arXiv submission.^[1]

The mismatch

In Figure 1, walking on pavement might simply be a convenient route. A learner assuming deliberate teaching can instead infer that grass is forbidden. A literal learner treats behavior as task performance; a pedagogic learner assumes the person chose actions to inform it. Neither label describes obedience to commands.^[1]

In a common-payoff game, Claim 2.1 ranks four pairings under recursive robot best responses and human improving responses. A pedagogic learner paired with a literal human has no higher payoff than a literal learner paired with a pedagogic human. The correctly matched pedagogic pair can still do best: this is an asymmetry of mismatch, not universal superiority of literal inference (§2).^[1]

Evidence from human demonstrations

The authors reanalyze an earlier experiment with 60 Mechanical Turk participants. Each gridworld has a goal worth 10 points and three tile colors, each either safe (0 points) or dangerous (−2 points): eight possible reward functions. Participants receive a two-cent bonus per point. In the literal condition, only their own score counts; in the teaching condition, they are told that another person’s score after learning from their demonstration will also count. Teaching participants more often visit multiple safe colors or loop over safe tiles (§3.1; Figure 2).^[1]

The learner starts with a uniform prior over the eight reward functions and is scored on whether its most probable inferred reward is correct. This measures reward identification, rather than the return of an autonomous robot deployed afterward. The literal human model favors actions with high task value. The pedagogic model adds a reward for increasing the literal learner’s belief in the true reward; both allow noisy choices (§§3.2–3.3).^[1]

The pedagogic learner paired with literal demonstrations performs significantly worse than the literal learner paired with teaching demonstrations. Even with participants instructed to teach, the literal learner slightly outperforms the pedagogic learner, but that difference is not statistically significant. The empirical ordering therefore differs from the idealized best-response ordering; instructing someone to teach does not make their behavior match the assumed teaching model (Figure 3).^[1]

Predicting behavior is not the same as recovering a goal

The paper tests mixtures at two levels: a demonstrator can follow one model for an entire demonstration, or mix literal and pedagogic choices action by action. These fit the teaching participants better than either pure model, with substantial variation between individuals. In simulations using the action mixture, the pedagogic learner achieves 97% reward-inference accuracy with the best population-wide mixture, but 90% when demonstrators use individually fitted mixtures. These are simulated demonstrations based on fits to the human data, not new participant success rates (§4.2; Figure 5).^[1]

Mixture behavior explains some of the mismatch, but not all of the observed gap. Giving the learner the more predictive action-mixture model does not improve its inference in this experiment. Claim 5.1 also constructs a finite counterexample: a model can assign greater likelihood to the observed behavior while assigning less posterior probability to the true hidden objective. Bayesian inference compares competing explanations of an action, so predicting that action well in isolation is insufficient (§§4.3–5).^[1]

Historical context

The paper builds on Cooperative Inverse Reinforcement Learning. It sharpens Human Model Misspecification: learning a goal shifts some specification work into the interpretation of people. The 60-person gridworld study does not establish robustness across other tasks, cultures or deployed assistants. The practical design question is which interpretations remain useful when a person falls between the models, and whether a simpler learner would be easier for that person to teach.^[1]