Instruction Hierarchy

An instruction hierarchy assigns priority to instructions from different sources. Wallace and colleagues train models to follow compatible lower-priority requests while resisting conflicting ones.^[1]

Imagine a museum assistant authorized to describe exhibits. A visitor asks for a shorter explanation: that fits the task. An exhibit label asks the assistant to abandon the museum and advertise a shop: reading that label should not give it authority. This is an original teaching example, extending the document-versus-supervisor analogy.

Behavior and permission

In a second hypothetical example, an assistant is given a key to one display cabinet. Training it to honor instructions concerns how it chooses actions. Restricting which cabinet the key opens concerns which actions the surrounding system permits. Neither example estimates the reliability of a particular deployed product.

The 2026 position paper argues that security-critical model judgments should be constrained by system design.^[2] Read both papers’ evidence scopes: a controlled training result and an architectural proposal answer different questions. The incidents guide asks what controls were actually present and what failed.