Categorizing Variants of Goodhart's Law

David Manheim and Scott Garrabrant ask why a useful metric can become harmful under stronger optimization. Their conceptual taxonomy uses mathematical examples to distinguish four mechanisms.^[1]

Method and contribution

Regressional failures select noise in an imperfect proxy. Extremal failures push a relationship beyond the region where it holds. Causal failures change the system through intervention. Adversarial failures involve other agents exploiting a metric. These mechanisms can overlap; diagnosis determines which mitigation is relevant.^[1]

Selection and intervention

The regressional model adds noise to the goal to obtain the metric. A high metric selects both genuine quality and favorable error; expected goal quality can still rise. Sections 2–3 distinguish selecting states outside a model’s useful range from intervening on the causal system. The latter includes changing a shared cause, an intermediary or the metric itself. Section 4 adds agents responding to incentives and shows how their actions can reproduce the earlier mechanisms.^[1]

The examples and equations are explanatory models, not a benchmark of deployed systems. The colonial cobra story is presented as a supposed historical account; this article does not treat it as a verified incident. The paper’s cited historical and empirical originals have not been independently audited here.

Alignment relevance and limits

Goodhart’s Law connects Outer Alignment to Reward Hacking: maximizing a measured objective can diverge from the intended outcome. The taxonomy offers explanations, not an empirical estimate of every failure or a universal cure. It does not imply that all metrics are useless.

See David Manheim and Scott Garrabrant.

Historical context

arXiv records the first submission of Categorizing Variants of Goodhart’s Law on 13 March 2018. The reviewed version is v4, revised 24 February 2019; submission and revision are different events.^[1]