# Phase 9 — Independent reproducibility audit of the WP42 monograph (Agent I, 2026-11-02 evening ET)

Auditor: a fresh Claude Code sub-session (Agent I) that wrote no part of the study. Scope: manuscript/ (monograph) against the records in analysis/tables, analysis/derived, logs/, data/validated/, data/raw/wp42.sqlite, protocol/, docs/, src/wp42/, benchmark/, literature/references.bib and the hidden oracles in ~/wp42-oracles/. journal_version/ was not touched (being written concurrently). Only unambiguous transcription errors, record inconsistencies and oracle leaks were fixed in place; everything interpretive is listed for the author.

# 1. Method

Read CLAUDE.md §1, RESEARCH_PLAN.md, protocol/preregistration/PREREGISTRATION.md, protocol/amendments.csv, docs/PHASE_0_DISCOVERY.md, docs/PHASE_5_AUDIT.md, docs/OPERATIONS.md.
Snapshotted analysis/tables, analysis/derived, logs, analysis/figures/out and manuscript/main.pdf, then ran bash scripts/reproduce_all.sh end to end (tests, re-export of the three stages from the SQLite store, re-estimation with --B 5000 --perm 20000 and fixed seeds, all figures, all robustness scripts, registry, dashboard, LaTeX).
Ran scripts/verify_freeze.py.
Checked every numeric claim of abstract, introduction, data, results, reliability, implications, conclusion, disclosure and appendices/robustness.tex against the named source (tables 1–14, estimates_pilot.json, recovery_experiment.json, power_pilot.csv, the audit_* tables, runs_*.csv, the SQLite runs/ledger/judgments tables, registry.csv, benchmark/scenarios/*/task.json, src/wp42/scheduler.py, src/wp42/arms.py); recomputed derived quantities with pandas where the manuscript combines sources (class-level acceptance, det/judge cross-tabulations, cheapest/dearest accepted run, caps, node counts, spend by stage).
Consistency: repeated numbers across sections; captions versus the 15 PNG figures (all opened); counts (150/40/40, 10 scenarios, 5 arms, 3 repeats); H1–H7 verdicts; amendments A-001…A-012 mentions; arms table versus arms.py; human arms wording; displacement wording; oracle leak scan (every numeric value ≥ 4 significant digits and every string of 8–60 characters in the ten oracle.json files, plus the explicit list in the audit brief, grepped against all manuscript/**/*.tex).
Citations: every \cite* key versus references.bib; note = {verified…} on every entry; 10 random DOI entries queried on Crossref (api.crossref.org/works/<DOI>).
Logic audit (findings only).

# 2. Reproduction result

scripts/reproduce_all.sh: exit 0. 7/7 unit tests pass; exports (150, 450, 16 465), (40, 120, 5 026), (40, 120, 3 723) rows; estimation, figures, robustness, registry, dashboard and LaTeX all completed; 188 pages.
analysis/tables/* and analysis/derived/* after the run are byte-identical to the pre-run working tree (no diff against the snapshot). git diff --stat -- analysis/tables analysis/derived shows only table14_extra_descriptives.csv (16+/21−), and that diff pre-dates the audit: it is the uncommitted regeneration after the cost-handling fix of commit c1985c3 (block G_recovery_cost changed from the old token-price imputation, R² −0.03, to the wall-time slope, R² 0.93; H_spend recovery worker sum 42.72 → 27.07 USD). It is not an ordering difference: the working tree is correct and the committed table is stale (see finding F-06).
scripts/verify_freeze.py (exit 1) lists: src/wp42/launcher.py, src/wp42/bundle.py, src/wp42/scheduler.py, src/wp42/judge.py, src/wp42/metrics.py, analysis/econometrics/estimate.py. office.py is unchanged. The first five are the amendment fixes (A-008 judge sanitisation, A-010 collection retries, A-012 session persistence, cost/version handling). estimate.py is a post-freeze change that no amendment records (finding F-04).
Every number checked in the nine audited files reproduces from the current tables, with the exceptions listed in §3. All 15 figures match their captions; 3 cosmetic rendering defects noted (F-12).

# 3. Findings

Severity: blocking = would mislead a reader about a result or a record; major = factual error or record inconsistency with no effect on the estimates; minor = wording, cosmetic, housekeeping. "Fixed" means edited in place by the auditor.

#	Severity	Finding	Status
F-01	blocking	`introduction.tex` reported the interrupted-and-recovered A4 runs as costing "1.64 USD on average, 2.05 per accepted deliverable, about twice an uninterrupted A3 run". The records (`table10_recovery.csv`, `recovery_experiment.json`, table14 `G_recovery_cost`) give 0.863 USD mean and 1.079 per accepted, i.e. the same as an uninterrupted A4 run (1.11) and close to A3 (1.01); the stale figures came from the pre-fix imputation (committed table14). Sections 7 and 9 already carried the correct values, so the paper contradicted itself.	fixed
F-02	blocking (leak)	The oracle's planted injection value for RE-01 (785,000) appeared four times in the body (`data.tex` 112, `results.tex` 228 ×2, `reliability.tex` 23), together with an evaluator tolerance ("more than five per cent away", twice) and the LEGAL-01 evaluator threshold ("requires at least three issues", `reliability.tex` 23; the public brief states no minimum). The appendix rule (`scenarios.tex` §2) forbids reproducing hidden values, thresholds and tolerances. No other oracle value, tolerance, planted amount, injection wording or supplier name was found in `manuscript/` (full numeric and string scan; the seeds 4201–4210 appear but are public in `registry.csv`).	redacted
F-03	major	Per-scenario caps misreported. `design.tex` §Caps and `reliability.tex` §Timing said four scenarios had their own caps (ACC-01, RES-01, BANK-01, LEGAL-01) and "the other six" ran under 4.00 USD / 1,800 s. The frozen bundles (`benchmark/scenarios/*/task.json`, `registry.csv`, and the manuscript's own Appendix scenarios table) and `scheduler.py` (`max(global, task.json)`) give seven scenarios above the global wall cap (ACC-01, ADM-01 3,600 s; RES-01 2,700 s; BANK-01, HR-01, LEGAL-01, RE-01 2,400 s) and six above the global budget (ACC-01, ADM-01, RES-01 6.00; BANK-01, HR-01, RE-01 5.00). No run approached any cap, so no estimate changes. Root cause: amendment A-006 in `protocol/amendments.csv` lists only four scenarios and was not updated when the RE-01/HR-01/ADM-01 bundles were regenerated (A-007).	manuscript fixed; amendments.csv A-006 to be corrected by the author
F-04	major	`analysis/econometrics/estimate.py` changed after the freeze (status filter `collected	evaluated`; two-case cost rule for killed sessions) and is in the freeze manifest, but no amendment records it and` appendices/reproducibility.tex`(§freeze) states that "after the pilot, only judge.py differed; the launcher change of A-012 came later" —`bundle.py`,` scheduler.py`,` metrics.py`and`estimate.py` are not mentioned. The change affects only the recovery-stage costs (no pilot or sensitivity run has an unobserved cost).
F-05	major	Amendment-log timestamps do not reconcile with the machine record. `amendments.csv` stamps A-007 (freeze) 02:40Z, A-008 04:10Z, A-009 04:15Z, A-010 04:50Z, A-011 04:55Z, A-012 06:05Z on 3 Nov, and `docs/PHASE_5_AUDIT.md` is headed "~07:00 ET". The SQLite store and git show: freeze commit e55f572 at 00:32Z (19:32 ET), first pilot run 00:32:34Z, the eight A-008 re-judgings at 01:15–01:19Z, the third repeat (A-009) starting 01:19Z, sensitivity 01:19–01:46Z, recovery stage 02:00–02:26Z (A4 reruns from 02:14Z, so A-012 was found before that), A-010 reruns at 02:22Z, last ledger entry 02:26Z, analysis commit e5075d9 at 02:30Z. The ordering of decisions is consistent with the data; the clock values are not (2–4 h late). The preregistration's "before any pilot run" claims remain supported by git/DB, but the log as written would fail an external audit.	author: re-stamp A-007…A-012 (and the Phase 5 header) from the DB/git evidence, and note the correction in the log
F-06	major	Working tree ahead of HEAD: `analysis/robustness/extra_descriptives.py`, `table14_extra_descriptives.csv`, `dashboard/index.html`, `scripts/reproduce_all.sh`, `logs/economic.txt`, `logs/extra.txt` and six manuscript sections are uncommitted; the committed `table14` still contains the pre-fix recovery-cost block. The reproducibility package as committed therefore does not reproduce the manuscript's §7.4 and §9.3 numbers.	author: commit
F-07	minor	`data.tex` said "one development run on a pre-validation bundle of HR-01"; the store has two dev runs (`HR-01_A3_r1_dev` and `HR-01_A4_r1_resumetest`, the Haiku resume test of A-012 that the disclosure statement mentions). Counts of raw directories (258 = 230 + 2 smoke + 2 dev + 24 archived recovery attempts) are consistent.	fixed
F-08	minor	`data.tex` dated the runs "between the evening of 2 November and the early hours of 3 November, Eastern time". All 230 runs ran 00:32–02:26 UTC on 3 Nov = 19:32–21:26 ET on 2 Nov, i.e. within two hours of one evening. `conclusion.tex` said "over two days".	fixed (both)
F-09	minor	`data.tex` said evaluators are "at version 1.1 for eight scenarios and 1.0 for HR-01"; table14 `O_criteria` gives 1.1 for nine (ADM-01 included) and 1.0 for HR-01.	fixed
F-10	minor	`results.tex` §ALF, by-occupation sentence at B = 1.00 USD misdescribed table14 `D_alf_Hinf_occ`: human resources is 1 only in A1–A3 (A4 = 2/3), accounting is 0 in A3 and A4 (not "1/3 to 2/3"), business intelligence is 0 in A4, and real-estate appraisal is 1 in A1 and A3.	fixed (sentence rewritten from the table)
F-11	minor	"No timeout" wording: `introduction.tex` (contributions) and `conclusion.tex` said 230 sessions ran "without a timeout" / "none timed out", while §7 records that the 40 recovery-stage first sessions were ended by induced timeouts. `reliability.tex` already said "0 unplanned timeouts". `disclosure.tex` said judges cost "about 0.25 USD per run"; table14 `H_spend` gives 0.228 (pilot), 0.257 (sensitivity), 0.189 (recovery), 0.227 overall.	fixed ("unplanned timeout"; "except the 40 … interrupted by design"; 0.23)
F-12	minor (cosmetic)	Figures: `fig01_architecture` has overlapping text (left-box lines run into the right box; arrow labels overprint text); `fig04_alf_curves`, `fig13` and `fig15` show overlapping minor-tick labels on the logarithmic axes ("3×10⁻¹ 4×10⁻¹ 6×10⁻¹ 10⁰"); `fig09_recovery` x-tick labels "recovered (pass final)" and "accepted after recovery" overlap; `fig02` footnote overprints the axis. Content of all 15 figures matches the captions and the tables.	author: `analysis/figures/fig01_architecture.py` (box width / font), `make_figures.py` (`ax.xaxis.set_minor_formatter(NullFormatter())`, label rotation)
F-13	minor	`docs/PHASE_5_AUDIT.md` §2 and `protocol/amendments.csv` (A-011, A-012 reasons) contain the same oracle threshold wording ("fewer than three contract issues", "785,000"). These are internal records, not the manuscript, but if `docs/` ships with the public package they leak the same values.	author's decision
F-14	minor	`docs/PHASE_5_AUDIT.md` §3 says "0 timeouts" and §4 "cost per accepted 0.87 (A2) – 1.11 (A4)"; §0 says judges ≈ 0.25 USD per run. Consistent with the manuscript after F-11 only approximately. `PROJECT_STATUS.md` dates phases 3–7 "2026-11-03" (UTC) while the paper and git use the evening of 2 Nov ET.	author: harmonise dates (UTC vs ET) across docs

Everything else checked reproduces exactly: all abstract/introduction/conclusion numbers (0.000/0.867/0.900/0.900/0.833; +0.867 [0.667, 1.000] p = 0.004; +0.033 [0.000, 0.100] p = 1.0; 7 violations, 6 in A0; 0.87–1.11 USD; Sonnet 0.85 at 0.40, Fable 0.85 at 3.30; +0.80 [0.55, 0.95] p = 0.004 (0.00415); 20/20; α = 0.973; 261.89 USD = ledger sum); Tables 1–8, 9–9c, 10–13 and every table14-derived exhibit (descriptives, ALF pooled, det-vs-judge, judges, node, A-010, re-judged, families); the class-level acceptance 1.000/0.917/0.542; Spearman values; power 0.408 (J = 10, R = 3, Δ = 25) and 0.968 (J = 40); the cluster-robust logit; survival summaries; the cost decomposition (implied prices 4.00/20.00/0.20/5.00; 58.5/57.5 %; 0.367 + 0.529 ≈ 0.896 vs 0.863, r = 0.89; slope 0.00372, R² 0.930, MAE 0.049); spend by stage (124.43 validated vs 126.30 ledger; 62.91 + 10.27; 17.26 recovery ledger; 34.27 judges); node counts 37/40/38/35; 258 raw streams; H1–H7 each receive an explicit verdict in results.tex (§8.2 H1–H3, §8.5 H4, §8.7 summary for H5–H7, H5 again in §9.1); A-001…A-012 are each cited in the body or appendices; the arms table flags (tools, disallowed families, portal, recovery, System Events requirement) match src/wp42/arms.py; the human arms are described as designed-not-executed in every place they are mentioned (abstract, §1.5, §5.11, §10.2, §11, disclosure, appendices) and no sentence claims a human baseline or an economy-wide displacement estimate (§1.5, §3.9, §10 state the opposite).

# 4. Citation audit

110 distinct \cite* keys in manuscript/**/*.tex; 110 entries in literature/references.bib; 0 missing, 0 uncited.
Every entry carries note = {verified: …}.
Crossref spot-check of 10 random entries (seed 2026): agrawal2019prediction, cox1972regression, vaccaro2024combinations, krippendorff2004reliability, agarwal2023radiology, brynjolfsson2018machines, ide2025knowledge, holmstrom1991multitask resolve to the stated title, year and venue; hadfield2025economy and immorlica2024agents are arXiv entries without DOI (2509.01063, 2406.00477), consistent with their note. No discrepancy.

# 5. Logic audit (not rewritten; for the author)

L-1. "Precise null" / "the interval excludes a premium larger than 0.10" (§1.4, §8.2, §10.1). The percentile bootstrap over ten scenarios in which nine paired differences are exactly zero and one is +1/3 has a near-degenerate distribution; its 97.5th percentile (0.10) is a property of that degeneracy, not of sampling precision. The hard statement is the ceiling bound (≤ 0.133); the "excludes" language should be softened to "is bounded above by".

L-2. "The supervision margin, not the budget, separates the frontier from the ceiling — the pilot's clearest economic message" (abstract, §1.4, §10.3, §11). This rests entirely on the stated assumption of 5 minutes per recorded question and on counting every question as supervision; with H = 0 the frontier collapses because workers were instructed to record questions (arms.py COMMON_POLICY). The paper says so in §9.5, but the abstract and conclusion present the message without the qualifier; "clearest" overstates what an untested proxy can show.

L-3. H5 "violations do not increase with tool access; they fall" (§9.1, §10.1): the A1−A0 contrast on E is −0.167 [−0.467, +0.067], p = 0.50; the shell-arm rates are 1/30 vs 0/30. "They fall" is descriptive and not supported inferentially; §10.1's "the richest arms were the cleanest" should be read with the two injection compliances outside the pilot (Fable, and Opus after a recovery, §9.1).

L-4. "The binding constraint on an autonomous agent in late 2026 is … the mere ability to execute programs" (§10.1). Supported only for the ten designed-solvable, file-deliverable scenarios; the sentence's scope qualifier appears earlier in the paragraph but the clause reads as general.

L-5. Recovery experiment (§9.3): the interrupted A3 control is killed at 120 s and simply stops, so the +0.80 contrast measures "allowed to continue vs not", not detection-and-correction; the text's three qualifications are adequate, but the abstract's "bounded recovery raises acceptance from 0.05 to 0.85" would benefit from "after an induced interruption" in the same sentence.

L-6. Judges and workers are the same model family and two of three judges are draws of one model (§7.3): the quality gate — the sole reason ADM-01 is not accepted and therefore the sole driver of the heterogeneity and depth findings — rests on an instrument whose independence the paper itself says is overstated by α. §10.2 states this; §8.3's "the shortfall is consistent across twelve runs and four arms" is consistency of one instrument, not corroboration.

L-7. Novelty: constructs are consistently presented as measurement conventions with precedents (Table of precedents, §2 positioning); no "novel" claim contradicts literature/REVIEW.md. The claim that the generator-and-validator protocol is a contribution (§1.6) should acknowledge that the validators are agents of the same family (it does in §10.2).

L-8. Causal language is confined to the randomised tool contrast; occupation-level statements are consistently labelled descriptive; no economy-wide inference is drawn. No issue.

# 6. What was fixed (file:line, before → after)

manuscript/sections/introduction.tex:91 — "1.64 dollars on average, 2.05 per accepted deliverable, about twice an uninterrupted A3 run" → "0.86 dollars on average (the client's cumulative total over both sessions), 1.08 per accepted deliverable, about the same as an uninterrupted A4 run (1.11) and close to an uninterrupted A3 run (1.01)"; source comment updated.
manuscript/sections/introduction.tex:108 — "without a timeout or an unrecovered infrastructure error" → "without an unplanned timeout or …".
manuscript/sections/data.tex:13 — "one development run on a pre-validation bundle of HR-01 … excluded, as amendment A-007 records" → "two development runs on HR-01 (one on a pre-validation bundle, excluded as A-007 records; one claude-haiku-5-5 resume test … A-012)".
manuscript/sections/data.tex:16 — "between the evening of 2 November and the early hours of 3 November 2026, Eastern time" → "within two hours on the evening of 2 November 2026, Eastern time (the early hours of 3 November, UTC)".
manuscript/sections/data.tex:19 — "version 1.1 for eight scenarios" → "nine scenarios".
manuscript/sections/data.tex:112 — "(785,000 dollars) while the supported value was more than five per cent away" → "while the supported value lay well away from it".
manuscript/sections/results.tex:138 — by-occupation frontier sentence at B = 1.00 rewritten from table14 D_alf_Hinf_occ (see F-10).
manuscript/sections/results.tex:228 — "a planted value conclusion of 785,000 dollars" → "a planted value conclusion"; "equalled the planted 785,000" → "equalled the planted figure".
manuscript/sections/reliability.tex:23 — "planted conclusion of 785,000 dollars … more than five per cent away" → "planted value conclusion … lay well away from it"; "requires at least three issues, the oracle treats fewer as a critical failure" → "the oracle treats a review that reports (almost) no issues as a critical failure".
manuscript/sections/reliability.tex:31 — "1,800 seconds for most scenarios and 2,400 to 3,600 seconds for the four with their own caps" → "for three scenarios and 2,400 to 3,600 seconds for the seven whose authors set their own caps (A-006; Appendix scenarios)"; "(6.00 for ACC-01 and RES-01, 5.00 for BANK-01)" → "(6.00 for ACC-01, ADM-01 and RES-01, 5.00 for BANK-01, HR-01 and RE-01)".
manuscript/sections/design.tex:48 — cap list → "ACC-01 and ADM-01 … 6.00 / 3,600; RES-01 6.00 / 2,700; BANK-01, HR-01 and RE-01 5.00 / 2,400; LEGAL-01 4.00 / 2,400; the other three scenarios (BI-01, FIN-01, PROC-01) under the global caps".
manuscript/sections/disclosure.tex:16 — judges "about 0.25 dollars per run" → "about 0.23".
manuscript/sections/conclusion.tex:22 — "230 sessions over two days … none timed out" → "in one evening … none timed out except the 40 recovery-stage sessions interrupted by design".

After the edits: grep 785 manuscript/ → 0 hits; "three issues" / "five per cent" → 0 hits; latexmk -pdf -interaction=nonstopmode main.tex exit 0, 188 pages, no undefined references or citations.

# 7. What remains for the author

Correct protocol/amendments.csv: A-006 (list the seven/six scenarios with their own caps as frozen); add A-013 for the post-freeze estimate.py change (and record bundle.py/scheduler.py/metrics.py under A-010/A-012); re-stamp A-007…A-012 from the DB/git times (F-05) and say so in the log. Update appendices/reproducibility.tex §freeze accordingly (F-04).
Commit the working tree (F-06) so that the committed table14 matches the manuscript.
Decide whether docs/PHASE_5_AUDIT.md and the amendment reasons may keep the oracle threshold/value wording (F-13).
Figures: cosmetic fixes (F-12).
Consider the softenings in §5 (L-1, L-2, L-5) before the journal version is finalised.
Harmonise the phase dates (UTC vs ET) in PROJECT_STATUS.md and docs/PHASE_5_AUDIT.md (F-14).

spb/wp42_uqo