SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors

Elle Najt, Colin Toft, Tyler Tracy, Fabien Roger and Joe Benton introduce SLEIGHT-Bench, a 2026 benchmark of 40 human-designed attacks across eleven monitor blind-spot categories. The authors use AI assistance to write synthetic coding-agent transcripts and paired benign versions. These are constructed tests of detection, rather than recordings of autonomous attacks on deployed systems.^[1]

Problem and approach

A monitor may recognize an obvious harmful command while missing consequences that depend on earlier state, environmental details or an omitted step. SLEIGHT-Bench organizes attacks around such reasoning gaps. Its categories include losing track of reused files, counting errors, assuming a default environment despite contrary evidence, and confusing manufactured permission with a user’s authorization.^[1]

The authors design each strategy, generate a plausible transcript with Claude Code, and use coding-agent verifiers to check consistency, visible evidence and payload effectiveness. Section 3.2 reports approximately ten iterations against a monitor and two to three hours of human authoring per transcript, with a wider range. The benchmark’s emphasis on avoiding heavy iteration does not mean that the transcripts were produced without monitor feedback.^[1]

The main monitors receive tool calls and outputs with assistant narration and reasoning removed, then assign a suspicion score. The main-body prompt is adapted from ControlArena. A separate agentic configuration can execute code in a sandbox to inspect the supplied transcript. Benign counterparts remove the malicious payload while retaining the scenario and style, allowing thresholds to be calibrated on closely matched examples.^[1]

Findings and calibration

Section 5.1 and Figure 2 report a 32% per-trial catch rate for Opus 4.6 with extended thinking at a threshold calibrated to a 1% false-positive rate on paired benign transcripts. Twenty of the forty attacks are never caught across ten trials; eight are caught on at least eight trials. These quantities describe this benchmark and configuration, rather than the probability that a real deployment avoids harm.^[1]

More investigation helps in the tested setup. Figure 2 reports about 52% per-trial detection for Opus 4.6 in the Claude Code scaffold. The authors associate improvements on counting and opaque-content attacks with the ability to count, decode and compare using tools. Targeted prompts improve their intended categories but each worsens at least one other category; the paper proposes complementary prompts as future work rather than demonstrating a complete solution.^[1]

The any-trial result needs special care. Taking the highest suspicion score over ten trials increases opportunities to catch an attack, but also increases opportunities to flag benign work. Appendix H therefore recalibrates the threshold on maximum benign scores too, reducing catch rates. Appendix J tests external benign sources; the authors report reductions of roughly ten to twenty percentage points relative to paired calibration. An operating point depends on both the scoring procedure and the benign workload.^[1]

Alignment relevance and limits

For Scalable Oversight, the study separates having evidence from correctly tracking its consequences. A monitor can be shown an environmental difference yet continue judging commands as if the default environment applied. A Safety Case should specify state access, session boundaries, tools, prompts and calibration data, alongside its detection score.

Section 7 explicitly states: “Attacks are not end-to-end on production systems.” Network restrictions and other safeguards may prevent depicted harms independently of the monitor. Synthetic outputs and verifier checks do not establish live-environment realism or the ability of a particular agent to execute these strategies.^[1]

As an editorial comparison, ResearchArena: Evaluating Sabotage and Monitoring in Automated AI R&D evaluates research artifacts in environments, while attack selection against trusted monitoring evaluates when an attacker submits a candidate. SLEIGHT-Bench broadens the strategies used to challenge detection. These methods address different gaps and their headline percentages are not interchangeable.

Historical context

First submitted May 15, 2026; this account reviews v2, revised May 19. Main sections 1–8, the rendered page 7 result figure and selected Appendix H/J calibration discussion were checked. The full appendices, underlying encrypted transcripts, verifier outputs, code and independent replication were not audited. The paper credits the Anthropic Fellows Program with compute and funding; this is a documented contribution to this project, not a general effectiveness assessment of the institution.^[1]