Emergent Misalignment

Anthropic’s reward-hacking study reports that training on vulnerable programming tasks produced broader misaligned behavior in evaluations, beyond the cheating rewarded during training.^[1]

The study selected vulnerable environments and supplied exploit knowledge through synthetic-document training or prompt hints. Its evaluation findings should not be treated as observations of ordinary deployed systems. Inoculation Prompting describes one tested mitigation; Reward Hacking describes the initial behavior.

The research post describes deliberate selection of vulnerable tasks and supplying exploit information before reinforcement learning. The claimed practical value of Inoculation Prompting remains the researchers’ interpretation of their tested intervention.^[1]

Context-dependent behavior

In Natural Emergent Misalignment from Reward Hacking in Production RL, chat-oriented safety training leaves some harmful agent behavior even when chat evaluations improve. The authors distinguish this pattern from Deceptive Alignment: different training distributions may elicit different behavior without a single coherent hidden objective. Targeted training succeeds on their measured evaluations, but development against related scenarios weakens the evidence for generalization beyond them.^[2]