EVIDENCE OVERVIEW / OCTOBER 15, 2026
Can an agent obey a rule
and still lose the
safe outcome?
A record of narrow model studies, reproduced software behavior and the questions that remain open. Research direction by Linh Ngo; implementation and analysis produced with AI coding agents.
This update consolidates existing records through October 8. It adds no new model trials or findings.
What we investigate
An agent may respect an authorization boundary yet miss a safer useful option. A security operation may work as documented while disrupting recovery. A safeguard may enforce action bounds while leaving other hazards unchecked.
MurderBench asks which of these boundaries can support a useful evaluation, and which proposed experiments should be rejected. The work so far does not establish a novel alignment failure, general model safety or a validated production repair.
Would calling an unchanged partial guard a “safety approver” change executed behavior? Both comparison arms disclose identical coverage facts. A separate repair would explicitly state uncovered hazards. Prior work, fresh confirmation cases and protocol freeze remain gates before querying models.
Inspect the proposed comparison →The completed proposal study
The first study recorded 2,304 calls across eight hosted model routes, with 2,223 valid outputs and 81 technical unknowns. It paired unchanged and changed permission or safety conditions across fixed synthetic histories.
| Verified state | Complete proposals | Valid outputs |
|---|---|---|
| Permission becomes invalid | 0 | 1,098 |
| Permission remains valid | 1,093 | 1,125 |
All valid changed-state outputs paused or requested clarification in this export. That narrow result does not prove alignment. Scripted histories, repeated calls and compound contexts are dependent; ordinary instruction following and an easy diagnostic remain plausible explanations. Filters, unknown finish reasons and truncation were not scored as safe.
Full results and limitations → Counts, route identities and denominators ↓Deadline pilots: planning failures and a negative comparison
A twelve-episode native development pilot used in-memory tools on two hosted proprietary routes. One selected episode missed an available way to preserve a verification window; another used a protected allocation despite a safer alternative. Separately implemented event folds checked the reported outcomes.
These were selected synthetic realizations, with one run per case and route. They did not isolate a safety-instruction effect, estimate prevalence or establish novelty. Earlier interface failures remain disclosed.
The later matched pilot attempted 125 of 192 planned episodes: 123 completed, 2 stopped prefixes and 67 unattempted episodes. Recorded omission was 0/60 completed need-positive episodes; authority breach was 0/123 completed episodes. Fully observed matched-arm comparisons had zero outcome discordance.
A perfect read-only trusted alternative made that comparison too easy to establish its intended contribution. The funding and novelty pitch was rejected. Missing cases do not support a population null effect.
Development observations → Native pilot report ↓ Matched pilot and rejection → Replay audit and missing cases ↓Remediation and recovery: a mechanism, not a model finding
Vault v1.20.4 was exercised on loopback with synthetic tokens, checkpoints and recovery jobs. Across two disclosed versions, 58 of 59 scripted reference attempts completed; 1 stopped unscored.
In the dependent-worker v2 topology, revoking the token tree denied both old identities in 4/4 cases but recovery succeeded in 0/4. A handoff before revocation preserved recovery and eventual denial in 4/4. The credentials remained live during handoff; security during that interval was not measured.
No model was queried. The mechanism and repair overlapped known operational work, and no surprising model result or validated operator decision was established. The candidate stopped at the contribution gate. This limited audit does not prove that every narrower recovery-aware evaluation is duplicate.
Reproduction, amendment and rejection → Attempt and outcome audit ↓What is implemented for the next question?
The restricted motion-guard adapter matched a pinned Inspect Robots implementation in 11,025 one-step comparisons and 224 scripted traces, with 0 differential and 0 trace mismatches. A separately implemented geometry verifier replays events, and a separate search labels safe-delivery feasibility.
The optional local Phoenix bridge matched 672 native evaluator calls over 224 scripted traces. It is an interoperability example, with no hosted service, customer implementation or measured model effect.
These controls use symbolic geometry, dependent rotations and some duplicate observations. They do not validate real robot dynamics or injury risk. The wording experiment, information-changing repair comparison and fresh confirmation study remain planned.
Controls, supported subset and reproduction ↓ Native comparisons and traces ↓ Phoenix interface and limitations ↗How the evidence is produced
Across these studies, the evaluation designs distinguish proposals from executed simulator effects, feasible tasks from impossible situations, and valid behavior from technical failures. Correct and deliberately faulty controls test the scoring boundary. Frozen manifests, source identities, disclosed amendments and retained missing denominators make the experiments inspectable.
“Independent replay” means a separately implemented verifier. It does not mean independent human review. Public logs can be replayed; authenticating model receipts also requires the private raw ledger. API-returned model labels do not establish checkpoint equivalence. No physical actuators, real subjects or production credentials were used in these experiments.
Source, tests, registrations and reports ↗ Exact public-reuse scope →Research direction and implementation
Linh Ngo initiated MurderBench and leads it through XP.COM, LLC dba Xenocom. The human headcount is one. He set the research goals, model priorities, sandbox boundaries and budget; challenged reassuring interpretations; and required novelty audits, useful controls and publication of negative findings and rejected candidates.
AI coding agents drafted literature audits and designs, wrote the implementation and analysis code, ran authorized experiments, and produced the reports, figures and website. Other AI agents supplied preparation critiques. Linh's contribution is the research direction, constraints and critique described above; the implementation and analysis artifacts were generated with coding agents. Independent human annotation remains pending.
The work remains preliminary. External scientific review, replication, a novel contribution and a concrete production decision changed by the evaluation are unestablished. No award or lab endorsement was verified in the last outreach check on October 8; this publication is not a new inbox check.
About Linh Ngo and the initiative → Challenge a method or propose a bounded collaboration →