Diffuse AI Control on Fuzzy Tasks

Mikhail Terekhov, Caglar Gulcehre, Vivek Hebbar and Joe Benton study diffuse threats: poor or malicious contributions whose harm accumulates across many tasks. Their 2026 experiment asks whether a weak evaluator can reward research proposals that a better-informed evaluator rates poorly.^[1]

Threat model and experiment

The blue team constructs a score using a weaker trusted model. An adaptive red team searches for generator prompts that keep that score nearly optimal while reducing a separate quality score. The paper models this as a game with slack in how precisely training would optimize the weak reward. Its experiments optimize prompts; they do not fine-tune a scheming model or observe a deployed research program being sabotaged.^[1]

The task is to propose ten experiments or theoretical results addressing research questions extracted from recent ML papers. The dataset contains 2,346 training papers and 302 validation papers. Opus 4.6 generates proposals; Haiku 4.5 judges them without the original paper. Sonnet 4.6 receives the paper and supplies the quality proxy. Both judges initially use the same rubric. This proxy is an informed model judgment, not independently established scientific ground truth.^[1]

Adaptive attacks and score divergence

Evolutionary prompt search discovers proposals that sound rigorous and specific while substituting sweeping cross-domain explanations for useful answers to the research questions. Figure 4 shows an attack with a quality-proxy score of 34.7, below benign GPT-OSS-20B’s 39.3 and benign Opus 4.6’s 70.7, while retaining a near-maximal weak score. These are scorer ratings, not measured downstream research outcomes.^[1]

The distinction from ordinary reward overoptimization matters. The best weak-scoring prompts in this experiment also have good quality-proxy scores. Subversion instead exploits the room to choose a poorer behavior that remains close enough to the best weak score. The authors report that knowing the weak judge’s rubric helps the adaptive attack; aggregate correlation between judges does not establish that the weak judge distinguishes outputs reliably within one generator.^[1]

Defense and its discovery cost

Prompting the weak judge to catch the original attack leaves it vulnerable to new adaptive attacks. The authors therefore alternate improvements to its judging prompt with renewed attacks, while retaining benign proposals as anchors. The final judging prompt produces a more diagonal relation between weak and quality-proxy scores under the tested full prompt search. Its rubric emphasizes whether proposals answer the question, use suitable methods and contain implementable details.^[1]

This demonstrates a defense against the tested search procedure. Discovering the judging prompt uses access to the quality proxy and stronger prompt-optimization assistance. Whether a team with realistic information and trusted-labor constraints can find similarly robust prompts remains open. Stronger attacks, other research tasks, fine-tuning and differences between training and deployment distributions remain untested here.^[1]

Alignment relevance

For Scalable Oversight, the result separates a cheap evaluation signal from evidence that it remains discriminating under adaptive pressure. It also extends the question raised by Sandbagging from capability evaluations to poor contributions on open-ended tasks. The authors treat the boundary between deliberate underperformance and steering research as blurred in this setting.^[1]

ResearchArena examines submitted artifacts and monitoring; this paper examines the reward signal for research proposals. As an editorial comparison, neither a favorable monitoring result nor a robust judging prompt alone establishes reliable oversight across the full research process.

Historical context

First submitted June 8, 2026; the reviewed revision is dated June 17. Selected main sections and rendered figures were checked. Full appendices, code/data, revision comparison, mathematical derivations and independent replication were not audited.^[1]