DELIVERY REPORT — WP45 / JoF-250 (2026-11-03)
Executor: Claude Code 2.1.317 (claude-fable-5-1) operating autonomously on the author's MacBook Pro, under the mandate in CLAUDE.md. All times America/Toronto. Everything below was verified by running the commands listed; nothing is projected.
1. Completed features
| Phase | Deliverable | Evidence |
|---|---|---|
| 1 Inventory | 278 files scanned, 276 included, 2 excluded (index.csv; AFA 2021 program mislabelled as article), 0 duplicates, 276/276 Crossref-verified, frozen manifest corpus_v1_frozen.json |
data/manifests/corpus_audit.md, jof250 corpus audit |
| 2 Extraction | PyMuPDF heuristic parser v1.3; 276/276 parsed, mean confidence 0.951; QA vs pdftotext regex (26-doc sample): sections 85% exact (26/26 after adjudication), tables ±1 96%, figures 92%, references ±10% 92%; title vs Crossref ≥ 90: 98.9% | data/derived/parser_qa.md, scripts/02_parser_qa.py |
| 3 Ontology | 12-part schema v1.0 (strict JSON schema), 276/276 valid annotations (228 first pass, 38 deterministic repairs, 10 uncached retries); deterministic cross-checks; cost 22.1 USD | src/jof250/rhetoric.py, data/annotations/ |
| 4 Profiles | 8 YAML profiles (train split 221 docs) with medians/IQR/bootstrap CIs, move prevalence, contribution placement; 4 figures | profiles/, figures/ |
| 5 Retrieval + engine | 9 642-unit hybrid index (BM25 + MiniLM, provenance); engine modes outline/write/upgrade, conditions A–E, deterministic audits, referee pass, proposal+report+diff contract; 5 review roles; evidence lookup | src/jof250/{embeddings,retrieval,generation,auditing}.py |
| 6 Claude Code | 10 slash commands, 7 agents, 1 skill (project + user scope); registration verified in a live session (skill + commands listed; jof-* agent types available); /jof-audit executed headlessly from a separate sample project |
.claude/, experiments/outputs/demo/separate_project_test.md |
| 7 Experiment | Preregistration frozen (commit 87c73cd) before runs; 199/199 runs, 33 blind judge sets; preregistered analysis with cluster bootstrap + Holm | experiments/preregistration.md, experiments/analysis/results_heldout_v1.json |
| 8 Paper | LaTeX WP45 (UQO template), 11 sections + appendices A–H, 64 verified references, all numbers from numbers.tex macros; compiled PDF |
paper/main.pdf |
| 9 Tests | 28 pytest tests (inventory, hashes, duplicates, parsing, schema, segmentation cap, auditing, overlap, CLI, Claude Code files, provenance, budget cap, retrieval) — all pass | make test |
2. Commands actually run (chronological, abridged)
jof250 corpus audit (×4, fixes in between) · jof250 corpus parse --force (×3) · python -I scripts/02_parser_qa.py · python -I scripts/03_make_splits.py · jof250 corpus annotate --workers 4 (+ repair/retry pass) · jof250 profiles build (×3) · jof250 index (×2) · jof250 write --paper …/wp41_uqo --section introduction --mode upgrade --output experiments/outputs/demo/wp41_intro_upgrade_E.tex · jof250 review --paper …/autonomous-research-claude-code --role referee --output experiments/outputs/demo/wp29_referee.md · jof250 install-claude-code --scope project|user · claude -p "/jof-audit /tmp/sample-paper" · python -I scripts/04_build_fixtures.py · jof250 benchmark --dry-run · jof250 benchmark --config config/evaluation.yaml --workers 8 · python -I scripts/05_analyze_benchmark.py heldout_v1 · python -I scripts/07_make_numbers.py · make test · cd paper && latexmk -pdf main.tex.
3. Actual measurements
- Corpus: 276 docs (2017: 63, 2018: 65, 2019: 74, 2020: 74); 263 articles, 4 presidential addresses, 8 corrigenda/replications, 1 retracted article; 11 735 pages; median 43 pages, 10 965 body words, 8 tables, 3 figures, 10 numbered equations, 52 references, 101-word abstract.
- Profiles (empirical, n = 173): abstracts state results in 100% and a magnitude in 29%; 90% of introductions open with a motivation move; contribution paragraph median relative position 0.63; roadmap in 65%; median causal-language strength 3/5 (theory papers 1/5); economic significance discussed in 93%; conclusions bounded in 83%.
- Benchmark (finance held-out, n = 24 paired): H1 not supported (D−B argument logic +0.33 [−0.22, 0.66]); H2 not supported (D−C ≈ 0); H3 supported (E−D rigor +1.33 [1.00, 1.84], Holm p = 0.0003; fidelity +0.62 [0.31, 1.03], p = 0.004); ablation: referee pass carries the effect (+1.63 rigor); new numbers absent from source E−A −0.88 [−1.94, −0.03]; overlap with non-source corpus documents max 0.006; judge second-pass Spearman 0.60–0.91; E costs 0.20 USD and 80 s per task vs 0.10 USD and 38 s for D.
- API spend: 50.9 USD total (annotation 22.1, benchmark ≈ 29, demos ≈ 0.6) under the 90 USD envelope (A-001).
4. Uncompleted work and blockers
- No human evaluation (no qualified raters available in session): H1 as stated in human-rated terms untested; protocol and blinded outputs prepared.
- Pilot scale: 2 held-out finance documents + 1 out-of-domain manuscript; 1–2 repetitions.
- H5 metric mis-specified in the preregistration (self-overlap); refined post hoc and both values reported.
- Cost-matched comparison not run (E uses two calls).
- Human gold standard for the parser and human inter-rater reliability for the ontology not available; automated cross-checks reported instead.
financial_econometricsprofile has only 8 documents (engine falls back toempirical_all).- spbgit: repository
wp45_uqo.gitcreated and pushed (see git log); SSRN not submitted.
5. Amendments
A-001 budget envelope 60 → 90 USD; A-002 corpus scope = volumes 72–75 (277 → 276 documents); A-003 global Claude Code install on explicit request.