Phase log 09-12, final report numbers, release rebuilt
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
5 changed files +19 −5
modified
release/WP29_Final_Report.md
+4 −4
@@ -11,11 +11,11 @@ Does an explicit research protocol, with executable verification gates, persiste | |||||
| 11 | | Smoke | 4 (+4 infra-failed attempts, no inference) | 5 | F2-smoke-01 × A–D | | 11 | | Smoke | 4 (+4 infra-failed attempts, no inference) | 5 | F2-smoke-01 × A–D | | ||
| 12 | | Pilot | 24 | 30 | 6 development tasks × A–D; 23/24 successes | | 12 | | Pilot | 24 | 30 | 6 development tasks × A–D; 23/24 successes | | ||
| 13 | | Main benchmark | 288 (24 tasks × 4 arms × 3 repeats, frozen seed 301) | 362 (incl. 2 of the aborted attempt) | 1 infrastructure error (A-004), rerun; 0 timeouts; 0 budget exhaustions | | 13 | | Main benchmark | 288 (24 tasks × 4 arms × 3 repeats, frozen seed 301) | 362 (incl. 2 of the aborted attempt) | 1 infrastructure error (A-004), rerun; 0 timeouts; 0 budget exhaustions | | ||
| 14 | −| Transfer | 24 (+1 post-fix verification) | 37 | T1–T3 × A–D × 2 | | 14 | +| Transfer | 24 | 36 | T1–T3 × A–D × 2 (arm D's reviewer never ran on T1: the gate never passed) | | ||
| 15 | | Recovery | 15 (R1 12, R3 1, R4 2) + 8 R2 gate checks | 29 | faults injected by the controller | | 15 | | Recovery | 15 (R1 12, R3 1, R4 2) + 8 R2 gate checks | 29 | faults injected by the controller | | ||
| 16 | | Ablation (arm L) | 24 | 24 | length-matched control | | 16 | | Ablation (arm L) | 24 | 24 | length-matched control | | ||
| 17 | −| Post-fix verification | 4 (+1) | 5 (+) | after amendment A-006 | | 17 | +| Post-fix verification | 8 (smoke ×4, T1 arms C and D, R1 ×2) | 17 | after amendment A-006; never pooled | | ||
| 18 | −Total: 384 registered runs, 503 sessions, 172.69 USD of inference (client list-price estimates), all on the author's cluster nodes M3U96a/M3U96b/M4M64a/M4M64b between 2026-10-30 03:30 and ≈ 07:00 UTC. | 18 | +Total: 387 registered runs, 514 sessions, 176.03 USD of inference (client list-price estimates), all on the author's cluster nodes M3U96a/M3U96b/M4M64a/M4M64b between 2026-10-30 03:30 and ≈ 07:00 UTC. | ||
| 19 | 19 | ||||
| 20 | ## 3. Counts | 20 | ## 3. Counts | ||
| 21 | Assigned 288 / launched 288 / completed 288 / evaluated 288 / scientific successes 263 / runs with a critical violation 13 (12 = one evaluator-defect label) / infrastructure attempts 1. | 21 | Assigned 288 / launched 288 / completed 288 / evaluated 288 / scientific successes 263 / runs with a critical violation 13 (12 = one evaluator-defect label) / infrastructure attempts 1. | ||
@@ -42,7 +42,7 @@ Native live: S002 (terminal tail of RD0002), S003 (builder session window, incid | |||||
| 42 | From saved artefacts (no provider access): `make test`, `make analyze`, `make paper`, `make replay` (clean-copy replay with byte comparison). Fresh live replication requires an API key, executor hosts, hidden oracles (`release/WP29_Reproducibility.md`). | 42 | From saved artefacts (no provider access): `make test`, `make analyze`, `make paper`, `make replay` (clean-copy replay with byte comparison). Fresh live replication requires an API key, executor hosts, hidden oracles (`release/WP29_Reproducibility.md`). | ||
| 43 | 43 | ||||
| 44 | ## 9. Limitations and validation status | 44 | ## 9. Limitations and validation status | ||
| 45 | −Designer dependence (same organisation/client family authored tasks, oracles, treatments); bounded tasks with a ceiling; one model/version/week; deterministic evaluators with two detected defects (sensitivity reported); prompt-delivery defect A-006 (one-word user prompt in every session; symmetric; post-fix verification unchanged); no human reviewer; costs are client estimates. Validation status: three adversarial sub-session reviews performed (`reports/review_*.md`), harness and evaluator defects found and disclosed; final human review of the manuscript by the author pending. | 45 | +Designer dependence (same organisation/client family authored tasks, oracles, treatments); bounded tasks with a ceiling; one model/version/week; deterministic evaluators with two detected defects (sensitivity reported); prompt-delivery defect A-006 (one-word user prompt in every frozen session, so the controller feedback loop, the review-repair stage and the resumption instruction were not delivered in any frozen run; symmetric across arms; seven post-fix verification runs behaved as the originals and demonstrated the delivered feedback loop); no human reviewer; costs are client estimates. Validation status: three adversarial sub-session reviews performed (`reports/review_*.md`, responses in `reports/final_review.md`): four oracle defects, three harness defects (A-002, A-004, A-006) and ledger errors (A-008) found and disclosed; release replay identical byte for byte; no human review of the manuscript had taken place at build time (the author's review is the next step). | ||
| 46 | 46 | ||||
| 47 | ## 10. Deliverables (paths) | 47 | ## 10. Deliverables (paths) | ||
| 48 | `paper/main.pdf` (→ `release/WP29_Autonomous_Research_Claude_Code_Boucher.pdf`), `paper/` LaTeX source, `release/WP29_Method_and_Implementation.zip`, `protocol/` (mandate, preregistration v1.0, contract, treatments, evaluation plan, screenshot protocol, amendments, freeze manifests), `runs/state.sqlite` + `runs/<split>/<run>/` (registry, transcripts, deliverables, receipts), `evaluators/` + `~/wp29-oracles/` (specs, validation), `results/analysis_data/runs.csv` (→ `release/WP29_Experiment_Results.csv`), `results/figures`, `results/tables`, `release/WP29_Visual_Evidence_Appendix.pdf`, `evidence/manifests/screenshots.csv`, `evidence/public_screenshots/`, `release/WP29_Reproducibility.md`, `release/WP29_Limitations_and_Claims.md`, `release/WP29_Human_Interventions_and_Costs.csv`, `release/WP29_Final_Review.md`, this file. | 48 | `paper/main.pdf` (→ `release/WP29_Autonomous_Research_Claude_Code_Boucher.pdf`), `paper/` LaTeX source, `release/WP29_Method_and_Implementation.zip`, `protocol/` (mandate, preregistration v1.0, contract, treatments, evaluation plan, screenshot protocol, amendments, freeze manifests), `runs/state.sqlite` + `runs/<split>/<run>/` (registry, transcripts, deliverables, receipts), `evaluators/` + `~/wp29-oracles/` (specs, validation), `results/analysis_data/runs.csv` (→ `release/WP29_Experiment_Results.csv`), `results/figures`, `results/tables`, `release/WP29_Visual_Evidence_Appendix.pdf`, `evidence/manifests/screenshots.csv`, `evidence/public_screenshots/`, `release/WP29_Reproducibility.md`, `release/WP29_Limitations_and_Claims.md`, `release/WP29_Human_Interventions_and_Costs.csv`, `release/WP29_Final_Review.md`, this file. | ||
modified
release/WP29_Final_Review.md
+1 −1
@@ -1,6 +1,6 @@ | |||||
| 1 | # WP29 — Final review record | 1 | # WP29 — Final review record | ||
| 2 | 2 | ||||
| 3 | −Assembled 2026-10-30T06:09:01+00:00. Reviewers are Claude Code sub-sessions (claude-fable-5-1) of the same client family as the builder; no human review had occurred at build time. The builder's responses are in `reports/final_review.md`. | 3 | +Assembled 2026-10-30T06:09:39+00:00. Reviewers are Claude Code sub-sessions (claude-fable-5-1) of the same client family as the builder; no human review had occurred at build time. The builder's responses are in `reports/final_review.md`. | ||
| 4 | 4 | ||||
| 5 | 5 | ||||
| 6 | 6 | ||||
modified
release/WP29_Method_and_Implementation.zip
+0 −0
Binary file not shown.
modified
release/WP29_Visual_Evidence_Appendix.pdf
+0 −0
Binary file not shown.
modified
reports/phase_status.md
+14 −0
@@ -64,3 +64,17 @@ Append-only log. Times are UTC unless stated. "Executor" = the Claude Code sessi | |||||
| 64 | ## Phase 08 — Transfer, recovery and ablations | 64 | ## Phase 08 — Transfer, recovery and ablations | ||
| 65 | - R1 (12 interrupted-and-resumed runs, 05:38–05:52 UTC), R2 (8 stale-input gate checks on pilot workspaces; mutation rule corrected once because the first version mutated an identifier column and crashed on a file without early digits — rerun, no inference), R3 (3 submissions → 1 session), R4 (2 missing-fixture runs: both recognised the blockage). Ablation arm L (length-matched control, 24 runs, 05:36–06:00 UTC): 23/24 successes, 0 violations, mean cost 0.24 USD. Transfer T1–T3 (24 runs, 12 USD / 60 min envelope) running since 05:31 UTC. | 65 | - R1 (12 interrupted-and-resumed runs, 05:38–05:52 UTC), R2 (8 stale-input gate checks on pilot workspaces; mutation rule corrected once because the first version mutated an identifier column and crashed on a file without early digits — rerun, no inference), R3 (3 submissions → 1 session), R4 (2 missing-fixture runs: both recognised the blockage). Ablation arm L (length-matched control, 24 runs, 05:36–06:00 UTC): 23/24 successes, 0 violations, mean cost 0.24 USD. Transfer T1–T3 (24 runs, 12 USD / 60 min envelope) running since 05:31 UTC. | ||
| 66 | - Labelled stress capture S009: a deliberately induced gate rejection on a copy of RP0006 (no natural controller gate failure occurred in the benchmark). | 66 | - Labelled stress capture S009: a deliberately induced gate rejection on a copy of RP0006 (no natural controller gate failure occurred in the benchmark). | ||
| 67 | + | ||||
| 68 | + | ||||
| 69 | +## Phase 09 — Independent evaluation and statistical analysis | ||||
| 70 | +- Frozen evaluations: 288/288 confirmatory receipts; export + data-quality assertions pass; primary analysis, robustness, claims endpoint, rubric sensitivity (A-005/A-007), recovery/transfer/ablation analyses, figures and tables regenerated by `make analyze`. Headline: p̂ A/B/C/D = 0.903/0.917/0.917/0.917; D−A = +0.014 [−0.028, +0.056], two-sided permutation p = 1.00; corrected rubric 1.000 in every arm. **Gate** PASSED (scoring not self-report; task-cluster uncertainty; nothing excluded; one review without verdict disclosed; tables and plots regenerate). | ||||
| 71 | + | ||||
| 72 | +## Phase 10 — Manuscript and visual evidence appendix | ||||
| 73 | +- 110-page manuscript compiled with 0 errors and 0 unresolved references, 84 verified references; AI-use statement; abstract and introduction written after the results. **Gate** PASSED. | ||||
| 74 | + | ||||
| 75 | +## Phase 11 — Adversarial review | ||||
| 76 | +- Three fresh sub-session reviews (systems/isolation; statistics/evaluators; manuscript): `reports/review_*.md`, responses in `reports/final_review.md`. Accepted and fixed: continuation prompts truncated and feedback never delivered (A-006; disclosed; 7 post-fix verification runs); four oracle defects behind every confirmatory non-success (A-007); RC0159 verdict recovered and RC0094 attempt-2 events re-ingested (A-008); delivered tool inventory corrected; audit false positives removed; captions/crops corrected; non-existent repair cycle and T1-D reviewer absence disclosed; undefined references, missing tables, effort table fixed. **Gate** PASSED (reviewers labelled as same-family sub-sessions, not human or independent). | ||||
| 77 | + | ||||
| 78 | +## Phase 12 — Reproduction, final build and delivery (2026-10-30 06:09 UTC) | ||||
| 79 | +- `make test` 10/10; `make analyze`; `make paper`; `release/build_release.py`; `release/validate_release.py`: clean-copy replay identical for the three headline estimate files, 0 secret hits, PDF non-empty and searchable. Series index row 29 added; repository pushed to `wp29_uqo.git`. Totals: 387 runs, 514 sessions, 176.03 USD inference. | ||||
| 80 | +- **Status**: all mandated phases executed with live experiments; outstanding: the author's own review of the manuscript (not yet performed) and a full replication with the corrected launcher. **Gate** PASSED with the stated limitations. | ||||
| 67 | 81 | ||||