Goodhart's Law

Goodhart’s law describes ways that optimizing a proxy can weaken its relationship to the intended outcome. A score is a measurement or incentive; success on it need not mean success at the task. Reward Hacking documents this distinction in AI systems.

Four mechanisms

Manheim and Garrabrant’s taxonomy distinguishes four mechanisms, which can overlap:^[1]

• Regressional: Selecting high values of a noisy proxy also selects favorable error. In the paper’s additive-noise model, the true outcome can improve while improving less than the score suggests.
• Extremal: Selection reaches a region where the old relationship fails, through an inadequate approximation or a change in the underlying regime.
• Causal: An intervention changes the relationship between proxy and outcome. Merely selecting on a mistaken causal model is insufficient for this category.
• Adversarial: Another actor’s different objective or response to incentives undermines the proxy, potentially exploiting any of the other mechanisms.

The initial formalization uses a single scalar goal, a scalar proxy and selection above a threshold. Intervention and interacting actors introduce additional assumptions. This is a taxonomy, not a theorem that every metric fails or that further optimization always worsens outcomes.^[1]

Understanding a failure

An investigation needs evidence about measurement error, the range where the relationship holds, available interventions and responding actors. These questions guide interpretation; the taxonomy alone does not identify a unique cause or establish Deceptive Alignment.

The paper writeup retains a worked selection example. Outer Alignment explains how a proxy becomes a training objective; Inner Alignment concerns a learned optimizer’s own criterion, if such an optimizer exists.