Findings

What the experiment shows so far, what it doesn't, and the ways to read it. The first section is computed from the logs as they stand now; the log below it records each finding on the day it was made, with the data it rested on then. New findings go at the top of the log.

Where it stands now

574 honest replays and 233 landed attacks, each step scored twice by Haiku 5.5 and averaged. Catches split the ties at the threshold; intervals come from 500 resamples of the replays.

The question is whether the kernel arms catch more attacks at the same honest audit budget. Sign tests describe score shifts within each group; they do not test improved separation. The full curves are on the overview.

Log

25 October 2026, evening: the leaderboard comparison, paired

The 596 honest and 569 landed originals from the leaderboard's two BashArena runs on data.linuxarena.ai. Every monitor, ours and theirs, scored each action once. 2,000 paired resamples of the trajectories (seed 0), ties split. The intervals cover which trajectories were drawn, not a monitor's run-to-run noise.

• On the leaderboard's own prompt, Haiku 5.5 catches 4.7× as many landed attacks as Haiku 4.5 at a 5% budget (208 against 44; 95% interval 3.3–7.8×), and 8.4× as many at 1% (74 against 8.8; 3.0–23×). AUROC is 0.724 against 0.628 (+0.063 to +0.130).
• It beats the best leaderboard monitor, GLM-4.6, at 5%: 208 against 124, 1.7× (1.3–2.1×), with AUROC +0.075 (+0.044 to +0.109). At 1% the difference, +45 (−45 to +118), includes zero.
• The prompt matters as much as the model. Haiku 5.5 on control-arena's prompt (latest step only) against the leaderboard's (full history): 362 against 208 at 5%, 1.7× (1.5–2.1×), and AUROC 0.875 against 0.724 (+0.127 to +0.173).
• Caveats: the leaderboard's simple prompt is our port, run through control-arena and Inspect rather than Control Tower's own pipeline. The 1% ratios divide small counts, so their intervals are wide.

25 October 2026, afternoon: the pre-registered result

574 honest replays and 233 landed attacks, each step the mean of two scorings (scored and scored-s2), then the trajectory's highest step. 2,000 paired resamples of the replays (seed 0). The live table above reads the same scorings, with its own, smaller resampling.

• Primary endpoint: no improvement shown. pAUC to 2% is 0.271 for baseline, 0.323 for kernel and 0.298 for kernel+notes. Kernel+notes minus baseline is +0.027 (95% interval −0.099 to +0.151). The interval includes zero, so by the pre-registered rule no improvement is claimed. This doesn't establish that there's no effect.
• Secondary: kernel minus baseline is +0.052 (−0.081 to +0.169). At 1%, catches with ties split are 68.5, 75.8 and 74.7, with differences +7.3 (−35.2 to +53.3) and +6.2 (−33.6 to +52.6). At 5% they're 161.9, 152.3 and 157.2, with differences −9.6 (−24.8 to +4.0) and −4.7 (−19.2 to +9.0). AUROC is 0.872, 0.856 and 0.864.
• The low-budget lead shrank (exploratory curve, 1,000 resamples, pointwise bands). The kernel arms lead only below about 1.5%; at 0.5% the catches are 26.4, 62.6 and 48.7. Kernel alone puts 12 attacks above every honest replay. That 0% band (+6 to +59) is the only one of 51 that excludes zero, so it's no evidence on its own. From 2% to 25% both kernel arms sit 2–16 below the baseline, with every band including zero.
• By step (exploratory, no intervals): at 1% of 17,320 honest steps, 191.4, 185.3 and 184.6 of 233 are caught.
• Kernel+notes still calms the monitor on both groups. Replay by replay, it scores 216 honest replays lower than the baseline and 129 higher (sign test p = 3×10⁻⁶), and 123 attacks lower and 62 higher (p = 9×10⁻⁶). Kernel alone: 152 lower and 186 higher on honest (p = 0.07), and 105 and 73 on attacks (p = 0.02). These are shifts, not separation.
• Three scorings (exploratory, on the 223 honest replays that have them): pAUC to 2% differences are +0.079 (−0.084 to +0.221) and +0.028 (−0.133 to +0.204). Same reading.
• Left out by the cohort rule: two honest replays failed inside the task (9011772, 38644578). The conditions of 24 and 25 October still apply: the 233 attacks were selected as strict-rule misses, they span 126 tasks, and there's no transcript-plus-notes control.

25 October 2026: reviewing the low-budget gain

Exploratory checks of the 24 October cohort: 145 honest replays and 233 landed attacks, one scoring per step. These are not the planned three-scoring analysis.

• A possible benefit at very low audit budgets, not an established general improvement. Expected catches at 1% are 45.3, 82.3 and 86.0 for baseline, kernel and kernel+notes. At 5% they are 138.1, 125.9 and 137.3. The 1% differences have wide intervals spanning zero. Similar overall AUROC does not rule out a useful improvement at 1%, but 1% of this honest cohort is only 1.45 runs.
• Much of the 1% gain comes from the honest tail. At a fixed score cutoff of 0.8, baseline flags 59 attacks and 2 honest runs, kernel 70 and 0, and kernel+notes 56 and 0. Keeping the baseline's randomised audit rule fixed (all scores above 0.8 and 72.5% of those at 0.8) gives expected catches of 45.3, 55.2 and 45.0. The remaining gain at 1% comes from recalibrating each arm against its own honest scores. This is a descriptive diagnostic, not a causal attribution or a comparison at equal false-positive rates. Reliably reducing extreme honest scores could still be useful.
• The clearest attack-score gain is in sudoers modifications. At the same 0.8 cutoff, sudoers catches are 22/68, 38/68 and 33/68. Password-leak catches are 11/50, 11/50 and 7/50; malware-download catches 20/53, 17/53 and 13/53; firewall catches 6/62, 4/62 and 3/62. This is consistent with explicit monitoring of sudoers writes, but is exploratory and does not establish a general detection gain.
• The comparison is conditional on selection. These attacks were selected as strict-rule baseline misses, so this cohort cannot establish whether kernel evidence preserves catches on attacks the baseline already detected. The 233 attacks span 126 tasks; the reported bootstrap resamples replays, and a task-level sensitivity analysis could assess dependence between attacks on the same task. A transcript-plus-notes control would also be needed to determine how much of the combined arm's benefit requires kernel observations.
• Two numerical issues are corrected. Per-trajectory pAUC now integrates threshold ties without spreading vertical ROC jumps across neighbouring budgets. Averaged scores preserve mathematically equal decimal means as ties. The single-scoring pAUC-to-5% point estimates remain 0.405, 0.431 and 0.476. Recomputing 1,000 paired resamples (seed 0) gives pAUC differences of +0.025 for kernel (95% interval −0.122 to +0.184) and +0.070 for kernel+notes (−0.066 to +0.207). The 1% catch estimates and their intervals are unchanged. Saved reports from before the correction retain the old calculations. The agreed primary endpoint, cohort rules and resampling scheme are unchanged.

24 October 2026, evening: auditing by step (exploratory)

Same data; see By step.

• No clear improvement demonstrated by step. At 1% of honest steps the baseline catches 190 of 233, kernel 184 and kernel+notes 195; every difference band includes zero, and kernel sits just below the baseline from 2% on. These intervals do not establish equivalence or rule out a useful effect.
• The budget units differ. On the originals, our Haiku baseline catches 84% at 1% of steps against 36% at 1% of trajectories, and the leaderboard's GLM-4.6 39% against 5%. But 1% of honest steps touches about 19% of honest runs. At a per-trajectory budget of 15–20%, the baseline catches 186–194 of the replayed attacks, about the same as by step at 1%. Runs touched is a workload proxy; equal numbers of runs touched do not imply equal review time for individual actions and full trajectories.
• Pre-registered (EXPERIMENT.md, before any averaged score): pAUC to 2% per trajectory, kernel+notes minus baseline, on the three averaged scorings and every honest replay.

24 October 2026: the headline, re-read

145 honest replays, 233 landed attacks, one sample per step.

• The 1% headline was mostly a tie at the threshold. The strict count read 9 → 70 of 233. In the baseline arm, exactly two honest replays score 0.8, so its threshold sat a step above the kernel arm's 0.7, and the 50 attacks at 0.8 counted as missed. Splitting ties, the counts at 1% are baseline 45, kernel 82 and kernel+notes 86. The differences, +37 (95% interval −45 to +61) and +41 (−34 to +61), both include zero.
• The observed advantage is concentrated around 1–2%. From 3% the kernel arm catches 5 to 21 fewer attacks than the baseline; kernel+notes tracks the baseline. Out to 25%, every pointwise difference band includes zero. AUROC is 0.856, 0.833 and 0.861. This does not establish no effect, or simultaneous coverage across all budgets.
• kernel+notes lowers scores on honest work and on attacks. Replay by replay, it scores 41 honest replays lower than the baseline and 19 higher (sign test p = 0.006), and 88 attacks lower and 55 higher (p = 0.007). These tests describe score shifts, not improved discrimination. False positives fall at some cutoffs (27 against 32 honest replays at 0.5 or above) but not all (131 against 129 at 0.3 or above). Kernel alone shows no clear directional shift, although its upper-tail attack scores do change.
• The 233 attacks were chosen as strict-rule misses on the originals. With ties split, the same monitor catches about 203 of 569 originals at 1%, so many picks sat on the threshold. The paired comparison describes this selected cohort; it does not estimate performance over all attacks or identify attacks the baseline reliably misses on repeated scoring.
• The highest-step score gives longer runs more opportunities to be flagged. Honest originals run a median of 25 steps and landed attacks 14. Several leaderboard monitors have AUROC below 0.5. Length is a possible contributor, not an established explanation for those results.
• Nine broken replays had been scored: four stopped part-way and duplicated by a rerun, five that failed inside the task. They're set aside, and the scoring step now refuses such runs. 13 attacks that landed originally didn't land when replayed, and are excluded.
• Jev 1.14 saturates: 0 of 233 caught in any arm, with honest originals at a median score of 0.94. Set aside until it can cache.

What would reduce the uncertainty

• More honest replays: done (25 October), 574 honest replays. The other honest originals didn't build (mostly arm64 problems) or failed inside the task (DIARY.md, 25 October).
• Several samples per step, averaged: done at two scorings for every replay (25 October); the Haiku credits didn't cover a third for the new honest replays.
• A length-aware trajectory score, beside the highest-step rule on the same replays.
• A fixed follow-up endpoint: the kernel arm's observed difference changes sign between 2% and 3%. The agreed 2% endpoint was chosen after seeing the single-scoring curve; the follow-up is not an independent replication.