DELIVERY REPORT — WP45 / JoF-250 (2026-11-03)
Executor: Claude Code 2.1.317 (claude-fable-5-1) operating autonomously on the author's MacBook Pro, under the mandate in CLAUDE.md. All times America/Toronto. Everything below was verified by running the commands listed; nothing is projected.
1. Completed features
| Phase | Deliverable | Evidence |
|---|---|---|
| 1 Inventory | 278 files scanned, 276 included, 2 excluded (index.csv; AFA 2021 program mislabelled as article), 0 duplicates, 276/276 Crossref-verified, frozen manifest corpus_v1_frozen.json |
data/manifests/corpus_audit.md, jof250 corpus audit |
| 2 Extraction | PyMuPDF heuristic parser v1.3; 276/276 parsed, mean confidence 0.951; QA vs pdftotext regex (26-doc sample): sections 85% exact (26/26 after adjudication), tables ±1 96%, figures 92%, references ±10% 92%; title vs Crossref ≥ 90: 98.9% | data/derived/parser_qa.md, scripts/02_parser_qa.py |
| 3 Ontology | 12-part schema v1.0 (strict JSON schema), 276/276 valid annotations (228 first pass, 38 deterministic repairs, 10 uncached retries); deterministic cross-checks; cost 22.1 USD | src/jof250/rhetoric.py, data/annotations/ |
| 4 Profiles | 8 YAML profiles (train split 221 docs) with medians/IQR/bootstrap CIs, move prevalence, contribution placement; 4 figures | profiles/, figures/ |
| 5 Retrieval + engine | 9 642-unit hybrid index (BM25 + MiniLM, provenance); engine modes outline/write/upgrade, conditions A–E, deterministic audits, referee pass, proposal+report+diff contract; 5 review roles; evidence lookup | src/jof250/{embeddings,retrieval,generation,auditing}.py |
| 6 Claude Code | 10 slash commands, 7 agents, 1 skill (project + user scope); registration verified in a live session (skill + commands listed; jof-* agent types available); /jof-audit executed headlessly from a separate sample project |
.claude/, experiments/outputs/demo/separate_project_test.md |
| 7 Experiment | Preregistration frozen (commit 87c73cd) before runs; run 1 discarded after two defects (A-004, archived); run 2: 199/199 runs, 33 blind judge sets; preregistered analysis with cluster bootstrap + Holm | experiments/preregistration.md, experiments/analysis/results_heldout_v1.json |
| 8 Paper | LaTeX WP45 (UQO template), 11 sections + appendices A–H, 64 verified references, all numbers from numbers.tex macros; compiled PDF |
paper/main.pdf |
| 9 Tests | 29 pytest tests (inventory, hashes, duplicates, parsing, schema, segmentation cap, auditing, overlap, CLI, Claude Code files, provenance, budget cap, retrieval, Markdown loader regression) — all pass | make test |
2. Commands actually run (chronological, abridged)
jof250 corpus audit (×4, fixes in between) · jof250 corpus parse --force (×3) · python -I scripts/02_parser_qa.py · python -I scripts/03_make_splits.py · jof250 corpus annotate --workers 4 (+ repair/retry pass) · jof250 profiles build (×3) · jof250 index (×2) · jof250 write --paper …/wp41_uqo --section introduction --mode upgrade --output experiments/outputs/demo/wp41_intro_upgrade_E.tex · jof250 review --paper …/autonomous-research-claude-code --role referee --output experiments/outputs/demo/wp29_referee.md · jof250 install-claude-code --scope project|user · claude -p "/jof-audit /tmp/sample-paper" · python -I scripts/04_build_fixtures.py · jof250 benchmark --dry-run · jof250 benchmark --config config/evaluation.yaml --workers 8 · python -I scripts/05_analyze_benchmark.py heldout_v1 · python -I scripts/07_make_numbers.py · make test · cd paper && latexmk -pdf main.tex.
3. Actual measurements
- Corpus: 276 docs (2017: 63, 2018: 65, 2019: 74, 2020: 74); 263 articles, 4 presidential addresses, 8 corrigenda/replications, 1 retracted article; 11 735 pages; median 43 pages, 10 965 body words, 8 tables, 3 figures, 10 numbered equations, 52 references, 101-word abstract.
- Profiles (empirical, n = 173): abstracts state results in 100% and a magnitude in 29%; 90% of introductions open with a motivation move; contribution paragraph median relative position 0.63; roadmap in 65%; median causal-language strength 3/5 (theory papers 1/5); economic significance discussed in 93%; conclusions bounded in 83%.
- Benchmark run 2 (finance held-out, n = 24 paired): H1 not supported (D−B argument logic +0.04 [−0.16, 0.22]); H2 not supported (D−C |Δ| ≤ 0.12); H3 supported (E−D rigor +0.96 [0.44, 1.47], Holm p = 0.0085; fidelity +0.46 [0.03, 0.69], p = 0.024); usefulness E−D −0.33 [−0.78, 0.06] n.s.; ablation: referee pass carries the effect (+1.25 rigor, +0.56 fidelity); unsupported claims E−C −1.00 [−1.59, −0.16]; unverifiable citations ≈ 0 in all conditions; overlap with non-source corpus documents max 0.007; judge second-pass Spearman 0.41–0.84; E costs 0.32 USD and 98 s per task vs 0.15 USD and 49 s for D.
- API spend: 93.2 USD total (annotation 22.1, benchmark run 1 ≈ 29 — discarded, run 2 ≈ 42, demos ≈ 0.6) under the 110 USD envelope (A-001, A-005).
4. Uncompleted work and blockers
- No human evaluation (no qualified raters available in session): H1 as stated in human-rated terms untested; protocol and blinded outputs prepared.
- Pilot scale: 2 held-out finance documents + 1 out-of-domain manuscript; 1–2 repetitions.
- H5 metric mis-specified in the preregistration (self-overlap); refined post hoc and both values reported.
- Run 1 discarded (A-004): Markdown loader dropped most fixture material; two tasks mis-framed; found by the results-writing agent from judge justifications; fixed, regression-tested, re-run.
- Cost-matched comparison not run (E uses two calls).
- Human gold standard for the parser and human inter-rater reliability for the ontology not available; automated cross-checks reported instead.
financial_econometricsprofile has only 8 documents (engine falls back toempirical_all).- spbgit: repository
wp45_uqo.gitcreated and pushed (see git log); SSRN not submitted.
5. Amendments
A-001 budget envelope 60 → 90 USD; A-002 corpus scope = volumes 72–75 (277 → 276 documents); A-003 global Claude Code install on explicit request; A-004/A-004b benchmark re-run after loader/material/task-framing defects; A-005 envelope 90 → 110 USD.