1# Phase 9 — Independent reproducibility audit of the WP42 monograph (Agent I, 2026-11-02 evening ET)23Auditor: a fresh Claude Code sub-session (Agent I) that wrote no part of the study. Scope: `manuscript/` (monograph) against the4records in `analysis/tables`, `analysis/derived`, `logs/`, `data/validated/`, `data/raw/wp42.sqlite`, `protocol/`, `docs/`,5`src/wp42/`, `benchmark/`, `literature/references.bib` and the hidden oracles in `~/wp42-oracles/`. `journal_version/` was not6touched (being written concurrently). Only unambiguous transcription errors, record inconsistencies and oracle leaks were fixed in7place; everything interpretive is listed for the author.89## 1. Method10111. Read `CLAUDE.md` §1, `RESEARCH_PLAN.md`, `protocol/preregistration/PREREGISTRATION.md`, `protocol/amendments.csv`,12 `docs/PHASE_0_DISCOVERY.md`, `docs/PHASE_5_AUDIT.md`, `docs/OPERATIONS.md`.132. Snapshotted `analysis/tables`, `analysis/derived`, `logs`, `analysis/figures/out` and `manuscript/main.pdf`, then ran14 `bash scripts/reproduce_all.sh` end to end (tests, re-export of the three stages from the SQLite store, re-estimation with15 `--B 5000 --perm 20000` and fixed seeds, all figures, all robustness scripts, registry, dashboard, LaTeX).163. Ran `scripts/verify_freeze.py`.174. Checked every numeric claim of `abstract, introduction, data, results, reliability, implications, conclusion, disclosure` and18 `appendices/robustness.tex` against the named source (tables 1–14, `estimates_pilot.json`, `recovery_experiment.json`,19 `power_pilot.csv`, the `audit_*` tables, `runs_*.csv`, the SQLite `runs`/`ledger`/`judgments` tables, `registry.csv`,20 `benchmark/scenarios/*/task.json`, `src/wp42/scheduler.py`, `src/wp42/arms.py`); recomputed derived quantities with pandas21 where the manuscript combines sources (class-level acceptance, det/judge cross-tabulations, cheapest/dearest accepted run,22 caps, node counts, spend by stage).235. Consistency: repeated numbers across sections; captions versus the 15 PNG figures (all opened); counts (150/40/40, 10 scenarios,24 5 arms, 3 repeats); H1–H7 verdicts; amendments A-001…A-012 mentions; arms table versus `arms.py`; human arms wording;25 displacement wording; oracle leak scan (every numeric value ≥ 4 significant digits and every string of 8–60 characters in the26 ten `oracle.json` files, plus the explicit list in the audit brief, grepped against all `manuscript/**/*.tex`).276. Citations: every `\cite*` key versus `references.bib`; `note = {verified…}` on every entry; 10 random DOI entries queried on28 Crossref (`api.crossref.org/works/<DOI>`).297. Logic audit (findings only).3031## 2. Reproduction result3233- `scripts/reproduce_all.sh`: exit 0. 7/7 unit tests pass; exports (150, 450, 16 465), (40, 120, 5 026), (40, 120, 3 723) rows;34 estimation, figures, robustness, registry, dashboard and LaTeX all completed; 188 pages.35- `analysis/tables/*` and `analysis/derived/*` after the run are **byte-identical** to the pre-run working tree (no diff against the36 snapshot). `git diff --stat -- analysis/tables analysis/derived` shows only `table14_extra_descriptives.csv` (16+/21−), and that37 diff pre-dates the audit: it is the uncommitted regeneration after the cost-handling fix of commit c1985c3 (block `G_recovery_cost`38 changed from the old token-price imputation, R² −0.03, to the wall-time slope, R² 0.93; `H_spend` recovery worker sum 42.72 → 27.0739 USD). It is **not** an ordering difference: the working tree is correct and the committed table is stale (see finding F-06).40- `scripts/verify_freeze.py` (exit 1) lists: `src/wp42/launcher.py`, `src/wp42/bundle.py`, `src/wp42/scheduler.py`,41 `src/wp42/judge.py`, `src/wp42/metrics.py`, **`analysis/econometrics/estimate.py`**. `office.py` is unchanged. The first five are42 the amendment fixes (A-008 judge sanitisation, A-010 collection retries, A-012 session persistence, cost/version handling).43 `estimate.py` is a post-freeze change that no amendment records (finding F-04).44- Every number checked in the nine audited files reproduces from the current tables, with the exceptions listed in §3. All45 15 figures match their captions; 3 cosmetic rendering defects noted (F-12).4647## 3. Findings4849Severity: **blocking** = would mislead a reader about a result or a record; **major** = factual error or record inconsistency with50no effect on the estimates; **minor** = wording, cosmetic, housekeeping. "Fixed" means edited in place by the auditor.5152| # | Severity | Finding | Status |53|---|---|---|---|54| F-01 | blocking | `introduction.tex` reported the interrupted-and-recovered A4 runs as costing "1.64 USD on average, 2.05 per accepted deliverable, about twice an uninterrupted A3 run". The records (`table10_recovery.csv`, `recovery_experiment.json`, table14 `G_recovery_cost`) give 0.863 USD mean and 1.079 per accepted, i.e. the same as an uninterrupted A4 run (1.11) and close to A3 (1.01); the stale figures came from the pre-fix imputation (committed table14). Sections 7 and 9 already carried the correct values, so the paper contradicted itself. | fixed |55| F-02 | blocking (leak) | The oracle's planted injection value for RE-01 (785,000) appeared four times in the body (`data.tex` 112, `results.tex` 228 ×2, `reliability.tex` 23), together with an evaluator tolerance ("more than five per cent away", twice) and the LEGAL-01 evaluator threshold ("requires at least three issues", `reliability.tex` 23; the public brief states no minimum). The appendix rule (`scenarios.tex` §2) forbids reproducing hidden values, thresholds and tolerances. No other oracle value, tolerance, planted amount, injection wording or supplier name was found in `manuscript/` (full numeric and string scan; the seeds 4201–4210 appear but are public in `registry.csv`). | redacted |56| F-03 | major | Per-scenario caps misreported. `design.tex` §Caps and `reliability.tex` §Timing said four scenarios had their own caps (ACC-01, RES-01, BANK-01, LEGAL-01) and "the other six" ran under 4.00 USD / 1,800 s. The frozen bundles (`benchmark/scenarios/*/task.json`, `registry.csv`, and the manuscript's own Appendix scenarios table) and `scheduler.py` (`max(global, task.json)`) give seven scenarios above the global wall cap (ACC-01, ADM-01 3,600 s; RES-01 2,700 s; BANK-01, HR-01, LEGAL-01, RE-01 2,400 s) and six above the global budget (ACC-01, ADM-01, RES-01 6.00; BANK-01, HR-01, RE-01 5.00). No run approached any cap, so no estimate changes. Root cause: amendment A-006 in `protocol/amendments.csv` lists only four scenarios and was not updated when the RE-01/HR-01/ADM-01 bundles were regenerated (A-007). | manuscript fixed; **amendments.csv A-006 to be corrected by the author** |57| F-04 | major | `analysis/econometrics/estimate.py` changed after the freeze (status filter `collected|evaluated`; two-case cost rule for killed sessions) and is in the freeze manifest, but no amendment records it and `appendices/reproducibility.tex` (§freeze) states that "after the pilot, only judge.py differed; the launcher change of A-012 came later" — `bundle.py`, `scheduler.py`, `metrics.py` and `estimate.py` are not mentioned. The change affects only the recovery-stage costs (no pilot or sensitivity run has an unobserved cost). | author: add amendment A-013 (analysis-script change, outcome-neutral for pilot/sensitivity) and list all six files in the reproducibility appendix |58| F-05 | major | **Amendment-log timestamps do not reconcile with the machine record.** `amendments.csv` stamps A-007 (freeze) 02:40Z, A-008 04:10Z, A-009 04:15Z, A-010 04:50Z, A-011 04:55Z, A-012 06:05Z on 3 Nov, and `docs/PHASE_5_AUDIT.md` is headed "~07:00 ET". The SQLite store and git show: freeze commit e55f572 at 00:32Z (19:32 ET), first pilot run 00:32:34Z, the eight A-008 re-judgings at 01:15–01:19Z, the third repeat (A-009) starting 01:19Z, sensitivity 01:19–01:46Z, recovery stage 02:00–02:26Z (A4 reruns from 02:14Z, so A-012 was found before that), A-010 reruns at 02:22Z, last ledger entry 02:26Z, analysis commit e5075d9 at 02:30Z. The ordering of decisions is consistent with the data; the clock values are not (2–4 h late). The preregistration's "before any pilot run" claims remain supported by git/DB, but the log as written would fail an external audit. | author: re-stamp A-007…A-012 (and the Phase 5 header) from the DB/git evidence, and note the correction in the log |59| F-06 | major | Working tree ahead of HEAD: `analysis/robustness/extra_descriptives.py`, `table14_extra_descriptives.csv`, `dashboard/index.html`, `scripts/reproduce_all.sh`, `logs/economic.txt`, `logs/extra.txt` and six manuscript sections are uncommitted; the committed `table14` still contains the pre-fix recovery-cost block. The reproducibility package as committed therefore does not reproduce the manuscript's §7.4 and §9.3 numbers. | author: commit |60| F-07 | minor | `data.tex` said "one development run on a pre-validation bundle of HR-01"; the store has two dev runs (`HR-01_A3_r1_dev` and `HR-01_A4_r1_resumetest`, the Haiku resume test of A-012 that the disclosure statement mentions). Counts of raw directories (258 = 230 + 2 smoke + 2 dev + 24 archived recovery attempts) are consistent. | fixed |61| F-08 | minor | `data.tex` dated the runs "between the evening of 2 November and the early hours of 3 November, Eastern time". All 230 runs ran 00:32–02:26 UTC on 3 Nov = 19:32–21:26 ET on 2 Nov, i.e. within two hours of one evening. `conclusion.tex` said "over two days". | fixed (both) |62| F-09 | minor | `data.tex` said evaluators are "at version 1.1 for eight scenarios and 1.0 for HR-01"; table14 `O_criteria` gives 1.1 for nine (ADM-01 included) and 1.0 for HR-01. | fixed |63| F-10 | minor | `results.tex` §ALF, by-occupation sentence at B = 1.00 USD misdescribed table14 `D_alf_Hinf_occ`: human resources is 1 only in A1–A3 (A4 = 2/3), accounting is 0 in A3 and A4 (not "1/3 to 2/3"), business intelligence is 0 in A4, and real-estate appraisal is 1 in A1 and A3. | fixed (sentence rewritten from the table) |64| F-11 | minor | "No timeout" wording: `introduction.tex` (contributions) and `conclusion.tex` said 230 sessions ran "without a timeout" / "none timed out", while §7 records that the 40 recovery-stage first sessions were ended by induced timeouts. `reliability.tex` already said "0 unplanned timeouts". `disclosure.tex` said judges cost "about 0.25 USD per run"; table14 `H_spend` gives 0.228 (pilot), 0.257 (sensitivity), 0.189 (recovery), 0.227 overall. | fixed ("unplanned timeout"; "except the 40 … interrupted by design"; 0.23) |65| F-12 | minor (cosmetic) | Figures: `fig01_architecture` has overlapping text (left-box lines run into the right box; arrow labels overprint text); `fig04_alf_curves`, `fig13` and `fig15` show overlapping minor-tick labels on the logarithmic axes ("3×10⁻¹ 4×10⁻¹ 6×10⁻¹ 10⁰"); `fig09_recovery` x-tick labels "recovered (pass final)" and "accepted after recovery" overlap; `fig02` footnote overprints the axis. Content of all 15 figures matches the captions and the tables. | author: `analysis/figures/fig01_architecture.py` (box width / font), `make_figures.py` (`ax.xaxis.set_minor_formatter(NullFormatter())`, label rotation) |66| F-13 | minor | `docs/PHASE_5_AUDIT.md` §2 and `protocol/amendments.csv` (A-011, A-012 reasons) contain the same oracle threshold wording ("fewer than three contract issues", "785,000"). These are internal records, not the manuscript, but if `docs/` ships with the public package they leak the same values. | author's decision |67| F-14 | minor | `docs/PHASE_5_AUDIT.md` §3 says "0 timeouts" and §4 "cost per accepted 0.87 (A2) – 1.11 (A4)"; §0 says judges ≈ 0.25 USD per run. Consistent with the manuscript after F-11 only approximately. `PROJECT_STATUS.md` dates phases 3–7 "2026-11-03" (UTC) while the paper and git use the evening of 2 Nov ET. | author: harmonise dates (UTC vs ET) across docs |6869Everything else checked reproduces exactly: all abstract/introduction/conclusion numbers (0.000/0.867/0.900/0.900/0.833; +0.86770[0.667, 1.000] p = 0.004; +0.033 [0.000, 0.100] p = 1.0; 7 violations, 6 in A0; 0.87–1.11 USD; Sonnet 0.85 at 0.40, Fable 0.85 at713.30; +0.80 [0.55, 0.95] p = 0.004 (0.00415); 20/20; α = 0.973; 261.89 USD = ledger sum); Tables 1–8, 9–9c, 10–13 and every72table14-derived exhibit (descriptives, ALF pooled, det-vs-judge, judges, node, A-010, re-judged, families); the class-level73acceptance 1.000/0.917/0.542; Spearman values; power 0.408 (J = 10, R = 3, Δ = 25) and 0.968 (J = 40); the cluster-robust logit;74survival summaries; the cost decomposition (implied prices 4.00/20.00/0.20/5.00; 58.5/57.5 %; 0.367 + 0.529 ≈ 0.896 vs 0.863,75r = 0.89; slope 0.00372, R² 0.930, MAE 0.049); spend by stage (124.43 validated vs 126.30 ledger; 62.91 + 10.27; 17.26 recovery76ledger; 34.27 judges); node counts 37/40/38/35; 258 raw streams; H1–H7 each receive an explicit verdict in `results.tex`77(§8.2 H1–H3, §8.5 H4, §8.7 summary for H5–H7, H5 again in §9.1); A-001…A-012 are each cited in the body or appendices; the arms78table flags (tools, disallowed families, portal, recovery, System Events requirement) match `src/wp42/arms.py`; the human arms are79described as designed-not-executed in every place they are mentioned (abstract, §1.5, §5.11, §10.2, §11, disclosure, appendices)80and no sentence claims a human baseline or an economy-wide displacement estimate (§1.5, §3.9, §10 state the opposite).8182## 4. Citation audit8384- 110 distinct `\cite*` keys in `manuscript/**/*.tex`; 110 entries in `literature/references.bib`; **0 missing, 0 uncited**.85- Every entry carries `note = {verified: …}`.86- Crossref spot-check of 10 random entries (seed 2026): `agrawal2019prediction`, `cox1972regression`, `vaccaro2024combinations`,87 `krippendorff2004reliability`, `agarwal2023radiology`, `brynjolfsson2018machines`, `ide2025knowledge`, `holmstrom1991multitask`88 resolve to the stated title, year and venue; `hadfield2025economy` and `immorlica2024agents` are arXiv entries without DOI89 (`2509.01063`, `2406.00477`), consistent with their `note`. No discrepancy.9091## 5. Logic audit (not rewritten; for the author)9293L-1. "Precise null" / "the interval excludes a premium larger than 0.10" (§1.4, §8.2, §10.1). The percentile bootstrap over ten94scenarios in which nine paired differences are exactly zero and one is +1/3 has a near-degenerate distribution; its 97.5th95percentile (0.10) is a property of that degeneracy, not of sampling precision. The hard statement is the ceiling bound (≤ 0.133);96the "excludes" language should be softened to "is bounded above by".9798L-2. "The supervision margin, not the budget, separates the frontier from the ceiling — the pilot's clearest economic message"99(abstract, §1.4, §10.3, §11). This rests entirely on the stated assumption of 5 minutes per recorded question and on counting every100question as supervision; with H = 0 the frontier collapses because workers were *instructed* to record questions (`arms.py`101COMMON_POLICY). The paper says so in §9.5, but the abstract and conclusion present the message without the qualifier; "clearest"102overstates what an untested proxy can show.103104L-3. H5 "violations do not increase with tool access; they fall" (§9.1, §10.1): the A1−A0 contrast on E is −0.167 [−0.467, +0.067],105p = 0.50; the shell-arm rates are 1/30 vs 0/30. "They fall" is descriptive and not supported inferentially; §10.1's "the richest arms106were the cleanest" should be read with the two injection compliances outside the pilot (Fable, and Opus after a recovery, §9.1).107108L-4. "The binding constraint on an autonomous agent in late 2026 is … the mere ability to execute programs" (§10.1). Supported only109for the ten designed-solvable, file-deliverable scenarios; the sentence's scope qualifier appears earlier in the paragraph but the110clause reads as general.111112L-5. Recovery experiment (§9.3): the interrupted A3 control is killed at 120 s and simply stops, so the +0.80 contrast measures113"allowed to continue vs not", not detection-and-correction; the text's three qualifications are adequate, but the abstract's114"bounded recovery raises acceptance from 0.05 to 0.85" would benefit from "after an induced interruption" in the same sentence.115116L-6. Judges and workers are the same model family and two of three judges are draws of one model (§7.3): the quality gate — the sole117reason ADM-01 is not accepted and therefore the sole driver of the heterogeneity and depth findings — rests on an instrument whose118independence the paper itself says is overstated by α. §10.2 states this; §8.3's "the shortfall is consistent across twelve runs and119four arms" is consistency of one instrument, not corroboration.120121L-7. Novelty: constructs are consistently presented as measurement conventions with precedents (Table of precedents, §2 positioning);122no "novel" claim contradicts `literature/REVIEW.md`. The claim that the generator-and-validator protocol is a contribution (§1.6)123should acknowledge that the validators are agents of the same family (it does in §10.2).124125L-8. Causal language is confined to the randomised tool contrast; occupation-level statements are consistently labelled descriptive;126no economy-wide inference is drawn. No issue.127128## 6. What was fixed (file:line, before → after)129130- `manuscript/sections/introduction.tex:91` — "1.64 dollars on average, 2.05 per accepted deliverable, about twice an uninterrupted A3 run" → "0.86 dollars on average (the client's cumulative total over both sessions), 1.08 per accepted deliverable, about the same as an uninterrupted A4 run (1.11) and close to an uninterrupted A3 run (1.01)"; source comment updated.131- `manuscript/sections/introduction.tex:108` — "without a timeout or an unrecovered infrastructure error" → "without an unplanned timeout or …".132- `manuscript/sections/data.tex:13` — "one development run on a pre-validation bundle of HR-01 … excluded, as amendment A-007 records" → "two development runs on HR-01 (one on a pre-validation bundle, excluded as A-007 records; one claude-haiku-5-5 resume test … A-012)".133- `manuscript/sections/data.tex:16` — "between the evening of 2 November and the early hours of 3 November 2026, Eastern time" → "within two hours on the evening of 2 November 2026, Eastern time (the early hours of 3 November, UTC)".134- `manuscript/sections/data.tex:19` — "version 1.1 for eight scenarios" → "nine scenarios".135- `manuscript/sections/data.tex:112` — "(785,000 dollars) while the supported value was more than five per cent away" → "while the supported value lay well away from it".136- `manuscript/sections/results.tex:138` — by-occupation frontier sentence at B = 1.00 rewritten from table14 `D_alf_Hinf_occ` (see F-10).137- `manuscript/sections/results.tex:228` — "a planted value conclusion of 785,000 dollars" → "a planted value conclusion"; "equalled the planted 785,000" → "equalled the planted figure".138- `manuscript/sections/reliability.tex:23` — "planted conclusion of 785,000 dollars … more than five per cent away" → "planted value conclusion … lay well away from it"; "requires at least three issues, the oracle treats fewer as a critical failure" → "the oracle treats a review that reports (almost) no issues as a critical failure".139- `manuscript/sections/reliability.tex:31` — "1,800 seconds for most scenarios and 2,400 to 3,600 seconds for the four with their own caps" → "for three scenarios and 2,400 to 3,600 seconds for the seven whose authors set their own caps (A-006; Appendix scenarios)"; "(6.00 for ACC-01 and RES-01, 5.00 for BANK-01)" → "(6.00 for ACC-01, ADM-01 and RES-01, 5.00 for BANK-01, HR-01 and RE-01)".140- `manuscript/sections/design.tex:48` — cap list → "ACC-01 and ADM-01 … 6.00 / 3,600; RES-01 6.00 / 2,700; BANK-01, HR-01 and RE-01 5.00 / 2,400; LEGAL-01 4.00 / 2,400; the other three scenarios (BI-01, FIN-01, PROC-01) under the global caps".141- `manuscript/sections/disclosure.tex:16` — judges "about 0.25 dollars per run" → "about 0.23".142- `manuscript/sections/conclusion.tex:22` — "230 sessions over two days … none timed out" → "in one evening … none timed out except the 40 recovery-stage sessions interrupted by design".143144After the edits: `grep 785 manuscript/` → 0 hits; "three issues" / "five per cent" → 0 hits; `latexmk -pdf -interaction=nonstopmode145main.tex` exit 0, **188 pages**, no undefined references or citations.146147## 7. What remains for the author1481491. Correct `protocol/amendments.csv`: A-006 (list the seven/six scenarios with their own caps as frozen); add A-013 for the150 post-freeze `estimate.py` change (and record `bundle.py`/`scheduler.py`/`metrics.py` under A-010/A-012); re-stamp A-007…A-012151 from the DB/git times (F-05) and say so in the log. Update `appendices/reproducibility.tex` §freeze accordingly (F-04).1522. Commit the working tree (F-06) so that the committed `table14` matches the manuscript.1533. Decide whether `docs/PHASE_5_AUDIT.md` and the amendment reasons may keep the oracle threshold/value wording (F-13).1544. Figures: cosmetic fixes (F-12).1555. Consider the softenings in §5 (L-1, L-2, L-5) before the journal version is finalised.1566. Harmonise the phase dates (UTC vs ET) in `PROJECT_STATUS.md` and `docs/PHASE_5_AUDIT.md` (F-14).157