Ensemble Monitoring for AI Control: Diverse Signals Outweigh More Compute

Eugene Koran, Yejun Yun, Samantha Tetef, Benjamin Arnav and Pablo Bernabeu-Pérez test whether multiple monitors built from one trusted base model supply complementary evidence about malicious code. The work connects detection with Scalable Oversight: false positives consume the human review budget, so performance at low false-positive rates matters.^[1]

Approach and threat model

The authors construct twelve GPT-4.1-Mini monitors using prompting and fine-tuning. Monitors inspect the problem statement and candidate code, rather than the untrusted agent’s reasoning trace. Their suspiciousness scores are averaged; thresholds for operating-point analysis are calibrated on validation data. Diversity is measured by pairwise score correlation on attack samples, not by counting models or assuming independent errors.^[1]

The filtered APPS data contain honest solutions and deliberately backdoored solutions that pass standard tests but fail on attack inputs. Comments are stripped to remove accidental disclosures of backdoor logic. The APPS test set contains 1,375 samples, including 172 attacks; the paper also tests transfer to BigCodeBench-Sabotage with different backdoor types. This is short-horizon code detection, not an evaluation of an entire deployed agent-control system.^[1]

Findings and selection limits

The main metric is normalized partial area under the ROC curve at false-positive rates up to 10%. Table 1 reports 0.8160 for the best three-monitor combination, compared with 0.7226 for one baseline monitor and 0.7621 for three repeated baseline monitors. The advertised roughly 2.4-fold advantage refers to the gain over the single-monitor baseline, not a 2.4-fold detection rate or safety probability.^[1]

For ensemble-composition analysis, combinations are ranked by APPS test-set performance. These best-combination numbers are therefore optimistic selection results, not untouched estimates of a prospectively fixed ensemble. The paper separately reports validation-selected operating points and applies APPS-selected combinations to BigCodeBench-Sabotage, where the selected ensembles retain advantages over the single baseline and median ensemble. Transfer evidence strengthens the finding but does not erase the original selection procedure.^[1]

Low correlation alone is insufficient: selecting the most diverse combinations performs worse than median selection on APPS when weak monitors are included. Filtering poor individual monitors before selecting for diversity improves results. Fine-tuned monitors appear in the top-performing combinations, suggesting complementary signals; the proposed mechanism behind those signals remains a hypothesis.^[1]

Alignment relevance and limits

The study supports evaluating both individual quality and complementary failure patterns when allocating monitoring effort. Its evidence does not establish that adding monitors always improves safety or that architectural diversity has been tested: the main twelve monitors share one base model and differ in prompts or fine-tuning.^[1]

Attackers are treated as unaware of ensemble composition. The authors do not test an adaptive adversary optimized against it, and explicitly leave shared-model prompt injection, long-horizon attacks and broader deployment choices unresolved. Thresholds, audit rates and intervention design still determine how detection contributes to a Safety Case. Noticing the Watcher: LLM Agents Can Infer CoT Monitoring from Blocking Feedback supplies a complementary reason to examine what repeated intervention feedback reveals to an agent, without showing a successful attack on these ensembles.^[1]

Historical context

First arXiv submission is May 14, 2026; the reviewed v2 revision is May 18. Its first page also prints May 19 as a preprint date; the chronology uses the repository submission record, and that internal date is not treated as an independently verified release. Main sections 1–7, Tables 1–2 and the selection/limitations passages were checked. Entire appendices, code, data provenance and independent replication were not audited. The chronology records first submission rather than a separate revision milestone.^[1]

Complementary research questions

How does information access affect LLM monitors’ ability to detect sabotage? tests a separate evidence-selection stage before judging.^[2] Editorially, ensemble composition and context selection should be compared as separate uses of a monitoring budget. The studies do not test their combination or establish that either transfers unchanged to a new task.