# PROGRESS — WP45 / JoF-250

Verified milestones only. Times are America/Toronto unless marked UTC. Code versions refer to git rev-parse --short HEAD.

# Amendments (decisions taken by the executor, to be ratified by the author)

# Phase 0 — Host audit (2026-11-03 17:30)

• macOS 27.0 (Darwin 27.0.0), Apple Silicon laptop; Claude Code 2.1.317; Python 3.14 system, 3.11/3.12 via uv 0.11.7; PyMuPDF 1.27 available; pdftotext (poppler) present; GROBID not installed, Docker daemon not running → PyMuPDF chosen (local, no server); LaTeX: TeX Live (latexmk/pdflatex/biber).
• Global ~/.claude/commands already holds cluster-* commands (untouched). ~/.claude/settings.json has an auto-approve PreToolUse hook (not modified).
• Corpus folder ~/Desktop/Journal-of-Finance-2018-2020: 277 PDF + index.csv + _outils/ (download scripts). Original PDFs are never written to.

# Phase 1 — Corpus inventory (2026-11-03 17:36) ✅

• jof250 corpus audit → data/manifests/{corpus_manifest.parquet,corpus_manifest.csv,corpus_audit.md,corpus_v1_frozen.json,crossref_cache.json,inventory_ledger.jsonl}.
• Files scanned 278; 277 included at first (2017: 63, 2018: 65, 2019: 74, 2020: 75); excluded 1 (index.csv, non-PDF) — revised to 276 in Phase 2 after the AFA 2021 preliminary program (labelled ARTICLES in the Wiley TOC) was excluded (2020: 74; 11 735 pages); 0 SHA-256 duplicates; 0 DOI duplicates.
• Types: 268 articles (incl. presidential addresses), 8 replications/corrigenda, 1 retracted article (Rampini–Viswanathan–Vuillemey 2020, kept and flagged).
• Crossref verification: 277/277 records status 200, container « The Journal of Finance », title similarity ≥ 85 (public API, polite pool).
• Pages: 11 775 total, median 43 per document; 230.6 MB.
• Fixes during the phase: ligature « fi » (U+FB01) in PDF text broke DOI regex → NFKC normalization; presidential address opens with a portrait page → validity checks the first three pages.

# Open blockers

• Human inter-rater audit of annotations and blind human evaluation (Phase 7) require qualified human reviewers: not available in this session → protocols and sheets prepared, results reported as model-judge only (see LIMITATIONS.md).

# Phase 2 — Structured extraction (2026-11-03 18:00) ✅

• jof250 corpus parse (parser pymupdf-jof-heuristics-1.3): 276/276 documents parsed, 0 failures, mean confidence 0.951; 3 documents < 0.8 (all corrigenda/notes, expected).
• Medians over the corpus: 43 pages, 10 965 body words, 5 numbered sections, 8 tables, 3 figures, 10 numbered equations, 38 footnotes, 52 references, 101-word abstract.
• Parser QA (scripts/02_parser_qa.py, 26-document stratified sample vs pdftotext regex): sections exact 85% (26/26 after adjudication of appendix sub-headings), tables within ±1 96%, figures exact 92%, references within 10% 92%, abstract presence 100%; parsed title vs Crossref title ≥ 90 similarity for 98.9% of all 276 documents. Equation counts are approximate (reference regex is noisy). Not a human gold standard (see LIMITATIONS.md).
• Fixes along the way: odd/even page margins for hanging-indent references; rotated (landscape) tables; figure captions whose first span only is bold; figure-internal text excluded from footnotes; DOI lines excluded from references; the AFA preliminary program (labelled ARTICLES in the TOC) excluded from the corpus (276 included).

# Phase 3 — Ontology annotation (started 2026-11-03 17:55)

• Schema v1.0 (src/jof250/rhetoric.py), prompt document_annotation v1.0, model claude-opus-5-5 effort medium; the API grammar for structured outputs rejected the full schema ("compiled grammar too large") → JSON requested in the prompt + strict local JSON-schema validation; invalid annotations are re-run.
• Pilot on 2 documents: ≈ 7.9k input / 2.3k output tokens, 0.08–0.10 USD per document.
• Full run (2026-11-03 17:55→18:30, 4 workers): 276/276 documents annotated; first pass 228 schema-valid, 48 invalid (extra *_note keys, mis-nested objects, 7 non-JSON). Deterministic, auditable repair (drop unknown keys / hoist mis-nested keys, recorded in repairs) fixed 38; an uncached retry with the validation errors echoed fixed the remaining 10 → 276/276 valid. Cost 22.09 USD (≈ 0.08 USD/doc).
• Model–rule agreement (train, n = 221): section-role agreement median 1.00; roadmap LLM vs regex agree on most documents; see numbers.tex macros \roleAgreementMedian, \roadmapAgree, \contribAgree.

# Phase 4 — Profiles (2026-11-03 18:35) ✅

• jof250 profiles build on the train split (221 docs): 8 profiles (profiles/*.yaml): corpus_all 221, empirical_all 173, empirical_asset_pricing 58, household_behavioral 32, theoretical_finance 32, corporate_finance 28, intermediation/banking 21, financial_econometrics 8 (too small; falls back to empirical_all in the engine). Medians/IQR/bootstrap CIs; figures fig_structure_distributions, fig_abstract_moves, fig_contribution_position, fig_moves_robustness.
• Known measurement limit: table standard-error conventions are mostly "unclear" to the annotator because table notes are capped at 350 chars per table in the payload (rights policy) — reported as such.

# Phase 5 — Retrieval + engine (2026-11-03 18:00–18:40) ✅

• Index: 9 642 units over 276 docs (all annotated), MiniLM-L6 (384-d, local) + BM25; held-out split excluded from retrieval by default; ≤ 2 units per document; provenance on every hit.
• Engine demo (real): experiments/outputs/demo/wp41_intro_upgrade_E.tex (+ report, diff, audit; 2 calls, 0.37 USD, 108 s) and wp29_referee.md (0.19 USD). Manuscripts untouched.

# Phase 6 — Claude Code integration (2026-11-03 18:05) ✅

• Project scope: 10 commands, 7 agents, 1 skill (.claude/). User scope installed on explicit request (A-003); registration verified: the jof250 skill and /jof-* commands appear in the running Claude Code session.
• Separate-project test: /jof-audit run headlessly in /tmp/sample-paper → experiments/outputs/demo/separate_project_test.md.

# Phase 7 — Preregistration frozen (commit 87c73cd) and benchmark launched (2026-11-03 18:40)

• experiments/preregistration.md; fixtures from held-out docs jof_12439 (corporate/intermediation) and jof_12777 (credit spreads / asset pricing) + WP29 (out-of-domain); plan: 199 runs (A–E ×27, 4 ablations ×16), 27 judge sets (+25% second pass); estimated 46 USD.

# Phase 7 — Benchmark heldout_v1 run (2026-11-03 18:40 → 19:54 ET) ✅

• 199/199 runs ok, 0 failures (A–E × 27 cells incl. rep 2 on four tasks; 4 ablations × 16; out-of-domain WP29 × 3 tasks × 5 conditions); 33 blind judge sets (27 pass A + 6 second-pass); workers 4 → 8 after 8 runs (restart resumed from the ledger). Benchmark spend ≈ 29 USD; total API spend 50.9 USD (cap 90).
• Analysis scripts/05_analyze_benchmark.py → experiments/analysis/results_heldout_v1.json, CSVs, paper/tables/tab_*.tex, fig_judge_by_condition.pdf.
• Preregistered outcomes (finance held-out, n = 24 paired units): H1 D−B: argument_logic +0.33 [−0.22, 0.66] Holm p = 0.45, contribution +0.08 [−0.41, 0.44] → not supported. H2 D−C: all contrasts ≈ 0 → not supported. H3 E−D: methodological_rigor +1.33 [1.00, 1.84] Holm p = 0.0003; fidelity +0.62 [0.31, 1.03] Holm p = 0.004 → supported; usefulness +0.25 n.s. H4: unverifiable citations = 0 in every condition (metric non-discriminating); new numbers absent from the source: E−A −0.88 [−1.94, −0.03]; unsupported claims lower under E (E−B −0.92 [−1.69, 0.13]). H5: preregistered overlap metric failed (max 0.36) because it counts overlap with the task's own source document; excluding the source document, max 0.006 and 100% of outputs < 0.02 (post-hoc refinement, labelled). Ablations: E − E_no_referee = +1.63 rigor, +0.50 fidelity, +0.69 argument_logic; no_retrieval / no_profile / no_factcheck ≈ 0. Judge second pass: Spearman 0.60–0.91, MAD 0.18–0.44. OOD (WP29, n = 1): no E advantage.
• Deviation log: (i) workers changed mid-run (no effect on content); (ii) H5 metric refined post hoc (both versions reported); (iii) effort medium for all generation calls (preregistered); (iv) no human raters.

# Phase 8 — Paper (2026-11-03 evening)

• Sections: introduction, literature (58+6 verified refs), corpus, extraction/ontology, architecture, design, results, ablations, human/failure, discussion, conclusion, appendices A–H; all numbers via paper/tables/numbers.tex macros (1 343) generated from recorded artefacts.

# Phase 9 — Tests and release

• make test: 28 passed (synthetic fixtures only). Delivery report: docs/DELIVERY_REPORT.md.

| A-004 | 2026-11-03 20:15 | Benchmark re-run after two defects found during the write-up (by the results-writing agent): (i) the Markdown manuscript loader mapped ## headings to level 2, so the fixture for jof_12777 collapsed into one section whose role was other and whose text was dropped from the material — generator and judge saw only the title and headings; (ii) tasks referee_report and flag_causal_claims were framed with the generic "Write the results" verb in contradiction with the brief. Fixes: auditing.load_manuscript (markdown levels), new engine mode task. Run 1 archived as experiments/outputs/heldout_v1_run1/; run 2 (heldout_v1) re-executes every cell for jof_12777, the two re-framed tasks for jof_12439 and the out-of-domain referee task, re-judges all sets; the 85 unaffected jof_12439 runs are reused. Preregistered analysis unchanged. | Scientific validity: half of the finance units in run 1 were no-material scaffold tasks. | | A-004b | 2026-11-03 20:30 | While re-running, a third defect was found in the material builder for Markdown fixtures (abstract, tables/notes, figures and reference list and other-role sections were omitted) → fixed (auditing.load_manuscript, generation.manuscript_material); the whole finance grid is re-run (184 runs) and all sets re-judged; only the 10 unaffected out-of-domain runs (LaTeX manuscript; tasks draft_introduction, improve_abstract) are reused. Run 1 stays archived. | Same rationale: the material seen by generator and judge in run 1 was incomplete for both finance documents. | | A-005 | 2026-11-03 20:30 | Budget envelope raised 90 → 110 USD (spent 50.9 before the re-run; the re-run is estimated at ≈ 35 USD). | Needed to complete the corrected experiment ("continue jusqu'au bout", author instruction 2026-11-03). |

# Phase 7 — Benchmark heldout_v1 run 2 (2026-11-03 20:35 → 21:40 ET) ✅ (supersedes run 1, archived)

• 199/199 runs ok (184 new + 10 reused OOD + 5 OOD referee re-run), 0 failures; 33 judge sets re-judged. Total API spend 93.2 USD (cap 110, A-005).
• Outcomes (finance held-out, n = 24 paired): H1 D−B: clarity −0.12 [−0.31, 0.03], argument_logic +0.04 [−0.16, 0.22], contribution 0.00 [−0.28, 0.28] → not supported. H2 D−C: all |Δ| ≤ 0.12, intervals include 0 → not supported. H3 E−D: methodological_rigor +0.96 [0.44, 1.47] Holm p = 0.0085; fidelity +0.46 [0.03, 0.69] Holm p = 0.024 → supported; usefulness −0.33 [−0.78, 0.06] n.s. (trade-off). Exploratory E−A rigor +1.33 [0.84, 2.28] p = 0.002; E−B rigor +1.08 [0.56, 1.81] p = 0.009. Ablations: E − no_referee rigor +1.25 [0.63, 1.88], fidelity +0.56 [0.13, 1.13], argument_logic +0.50 [0.06, 0.94]; no_retrieval / no_profile / no_factcheck ≈ 0. H4: unverifiable citations ≈ 0 everywhere (−0.08 E−A); unsupported claims E−C −1.00 [−1.59, −0.16] (unadjusted p = 0.009), E−A −0.46, E−B −0.42 (n.s.); new numbers −0.17/−0.21/−0.42 (n.s.). H5: preregistered metric max 0.40 (self-overlap: improve_abstract 0.25, rewrite_literature 0.08); excluding the source document max 0.007, 100% < 0.02. Judge second pass (50 pairs): Spearman clarity 0.41, argument 0.58, contribution 0.84, rigor 0.80, fidelity 0.75, usefulness 0.53; MAD 0.26–0.40. Cost/latency: E 0.32 USD, 98 s vs D 0.15 USD, 49 s. OOD WP29 (n = 1): no E advantage.

# Release (2026-11-03 21:55 ET)

• paper/main.pdf 75 pages, 0 LaTeX errors, 0 undefined citations/references; 29/29 tests; README.md, docs/DELIVERY_REPORT.md final; series README.md and HARMONISATION.md rows 45 added.
• Pushed to spbgit gitsrv:srv/git/wp45_uqo.git (history rewritten once to drop the 70 MB shingles.json before the public push; it is regenerated on demand).