# PROGRESS — WP45 / JoF-250

Verified milestones only. Times are America/Toronto unless marked UTC. Code versions refer to `git rev-parse --short HEAD`.

## Amendments (decisions taken by the executor, to be ratified by the author)

| ID | Date | Decision | Rationale |
|---|---|---|---|
| A-001 | 2026-11-03 | API budget envelope fixed at **90 USD** (initially 60, raised before the annotation run once the per-document cost was measured at ≈0.09 USD on Opus 5.5) (config/models.yaml `budget_usd_cap`) for annotation + pilot experiments, `claude-sonnet-5-5`, key from `~/Desktop/Cluster/secrets/wp42/.env` | Mandate forbids spending without authorization; the author's standing practice on WP26/WP29/WP42/WP44 is an executor-fixed envelope logged as an amendment. Every call is logged in `data/annotations/api_ledger.jsonl`. |
| A-002 | 2026-11-03 | Corpus scope = the 4 volumes present in the folder (72–75, 2017–2020), 277 documents, not the « ~250 of 2018–2020 » of the mandate | The folder contains 2017 (downloaded on request the same day); all pre-ChatGPT; year is a stratification variable. |
| A-003 | 2026-11-03 | Global install of the Claude Code commands/skill/agents under `~/.claude/` (prefix `jof-`) | Explicit user request « accessible à toutes les sessions Claude Code ». |

## Phase 0 — Host audit (2026-11-03 17:30)
- macOS 27.0 (Darwin 27.0.0), Apple Silicon laptop; Claude Code **2.1.317**; Python 3.14 system, 3.11/3.12 via uv 0.11.7; PyMuPDF 1.27 available; `pdftotext` (poppler) present; GROBID not installed, Docker daemon not running → **PyMuPDF chosen** (local, no server); LaTeX: TeX Live (latexmk/pdflatex/biber).
- Global `~/.claude/commands` already holds `cluster-*` commands (untouched). `~/.claude/settings.json` has an auto-approve PreToolUse hook (not modified).
- Corpus folder `~/Desktop/Journal-of-Finance-2018-2020`: 277 PDF + `index.csv` + `_outils/` (download scripts). Original PDFs are never written to.

## Phase 1 — Corpus inventory (2026-11-03 17:36) ✅
- `jof250 corpus audit` → `data/manifests/{corpus_manifest.parquet,corpus_manifest.csv,corpus_audit.md,corpus_v1_frozen.json,crossref_cache.json,inventory_ledger.jsonl}`.
- Files scanned 278; **277 included at first** (2017: 63, 2018: 65, 2019: 74, 2020: 75); excluded 1 (`index.csv`, non-PDF) — **revised to 276 in Phase 2** after the AFA 2021 preliminary program (labelled ARTICLES in the Wiley TOC) was excluded (2020: 74; 11 735 pages); 0 SHA-256 duplicates; 0 DOI duplicates.
- Types: 268 articles (incl. presidential addresses), 8 replications/corrigenda, 1 retracted article (Rampini–Viswanathan–Vuillemey 2020, kept and flagged).
- **Crossref verification: 277/277** records status 200, container « The Journal of Finance », title similarity ≥ 85 (public API, polite pool).
- Pages: 11 775 total, median 43 per document; 230.6 MB.
- Fixes during the phase: ligature « fi » (U+FB01) in PDF text broke DOI regex → NFKC normalization; presidential address opens with a portrait page → validity checks the first three pages.

## Open blockers
- Human inter-rater audit of annotations and blind human evaluation (Phase 7) require qualified human reviewers: **not available in this session** → protocols and sheets prepared, results reported as model-judge only (see LIMITATIONS.md).

## Phase 2 — Structured extraction (2026-11-03 18:00) ✅
- `jof250 corpus parse` (parser `pymupdf-jof-heuristics-1.3`): 276/276 documents parsed, 0 failures, mean confidence 0.951; 3 documents < 0.8 (all corrigenda/notes, expected).
- Medians over the corpus: 43 pages, 10 965 body words, 5 numbered sections, 8 tables, 3 figures, 10 numbered equations, 38 footnotes, 52 references, 101-word abstract.
- Parser QA (`scripts/02_parser_qa.py`, 26-document stratified sample vs `pdftotext` regex): sections exact 85% (26/26 after adjudication of appendix sub-headings), tables within ±1 96%, figures exact 92%, references within 10% 92%, abstract presence 100%; parsed title vs Crossref title ≥ 90 similarity for 98.9% of all 276 documents. Equation counts are approximate (reference regex is noisy). Not a human gold standard (see LIMITATIONS.md).
- Fixes along the way: odd/even page margins for hanging-indent references; rotated (landscape) tables; figure captions whose first span only is bold; figure-internal text excluded from footnotes; DOI lines excluded from references; the AFA preliminary program (labelled ARTICLES in the TOC) excluded from the corpus (276 included).

## Phase 3 — Ontology annotation (started 2026-11-03 17:55)
- Schema v1.0 (`src/jof250/rhetoric.py`), prompt `document_annotation` v1.0, model claude-opus-5-5 effort medium; the API grammar for structured outputs rejected the full schema ("compiled grammar too large") → JSON requested in the prompt + strict local JSON-schema validation; invalid annotations are re-run.
- Pilot on 2 documents: ≈ 7.9k input / 2.3k output tokens, 0.08–0.10 USD per document.
- Full run (2026-11-03 17:55→18:30, 4 workers): 276/276 documents annotated; first pass 228 schema-valid, 48 invalid (extra `*_note` keys, mis-nested objects, 7 non-JSON). Deterministic, auditable **repair** (drop unknown keys / hoist mis-nested keys, recorded in `repairs`) fixed 38; an uncached retry with the validation errors echoed fixed the remaining 10 → **276/276 valid**. Cost 22.09 USD (≈ 0.08 USD/doc).
- Model–rule agreement (train, n = 221): section-role agreement median 1.00; roadmap LLM vs regex agree on most documents; see `numbers.tex` macros `\roleAgreementMedian`, `\roadmapAgree`, `\contribAgree`.

## Phase 4 — Profiles (2026-11-03 18:35) ✅
- `jof250 profiles build` on the train split (221 docs): 8 profiles (`profiles/*.yaml`): corpus_all 221, empirical_all 173, empirical_asset_pricing 58, household_behavioral 32, theoretical_finance 32, corporate_finance 28, intermediation/banking 21, financial_econometrics 8 (too small; falls back to empirical_all in the engine). Medians/IQR/bootstrap CIs; figures `fig_structure_distributions`, `fig_abstract_moves`, `fig_contribution_position`, `fig_moves_robustness`.
- Known measurement limit: table standard-error conventions are mostly "unclear" to the annotator because table notes are capped at 350 chars per table in the payload (rights policy) — reported as such.

## Phase 5 — Retrieval + engine (2026-11-03 18:00–18:40) ✅
- Index: 9 642 units over 276 docs (all annotated), MiniLM-L6 (384-d, local) + BM25; held-out split excluded from retrieval by default; ≤ 2 units per document; provenance on every hit.
- Engine demo (real): `experiments/outputs/demo/wp41_intro_upgrade_E.tex` (+ report, diff, audit; 2 calls, 0.37 USD, 108 s) and `wp29_referee.md` (0.19 USD). Manuscripts untouched.

## Phase 6 — Claude Code integration (2026-11-03 18:05) ✅
- Project scope: 10 commands, 7 agents, 1 skill (`.claude/`). User scope installed on explicit request (A-003); registration verified: the `jof250` skill and `/jof-*` commands appear in the running Claude Code session.
- Separate-project test: `/jof-audit` run headlessly in `/tmp/sample-paper` → `experiments/outputs/demo/separate_project_test.md`.

## Phase 7 — Preregistration frozen (commit 7ef3b9a) and benchmark launched (2026-11-03 18:40)
- `experiments/preregistration.md`; fixtures from held-out docs jof_12439 (corporate/intermediation) and jof_12777 (credit spreads / asset pricing) + WP29 (out-of-domain); plan: 199 runs (A–E ×27, 4 ablations ×16), 27 judge sets (+25% second pass); estimated 46 USD.

## Phase 7 — Benchmark heldout_v1 run (2026-11-03 18:40 → 19:54 ET) ✅
- 199/199 runs ok, 0 failures (A–E × 27 cells incl. rep 2 on four tasks; 4 ablations × 16; out-of-domain WP29 × 3 tasks × 5 conditions); 33 blind judge sets (27 pass A + 6 second-pass); workers 4 → 8 after 8 runs (restart resumed from the ledger). Benchmark spend ≈ 29 USD; total API spend 50.9 USD (cap 90).
- Analysis `scripts/05_analyze_benchmark.py` → `experiments/analysis/results_heldout_v1.json`, CSVs, `paper/tables/tab_*.tex`, `fig_judge_by_condition.pdf`.
- **Preregistered outcomes** (finance held-out, n = 24 paired units): H1 D−B: argument_logic +0.33 [−0.22, 0.66] Holm p = 0.45, contribution +0.08 [−0.41, 0.44] → **not supported**. H2 D−C: all contrasts ≈ 0 → **not supported**. **H3 E−D: methodological_rigor +1.33 [1.00, 1.84] Holm p = 0.0003; fidelity +0.62 [0.31, 1.03] Holm p = 0.004 → supported**; usefulness +0.25 n.s. H4: unverifiable citations = 0 in every condition (metric non-discriminating); new numbers absent from the source: E−A −0.88 [−1.94, −0.03]; unsupported claims lower under E (E−B −0.92 [−1.69, 0.13]). H5: preregistered overlap metric failed (max 0.36) because it counts overlap with the task's own source document; excluding the source document, max 0.006 and 100% of outputs < 0.02 (post-hoc refinement, labelled). Ablations: E − E_no_referee = +1.63 rigor, +0.50 fidelity, +0.69 argument_logic; no_retrieval / no_profile / no_factcheck ≈ 0. Judge second pass: Spearman 0.60–0.91, MAD 0.18–0.44. OOD (WP29, n = 1): no E advantage.
- Deviation log: (i) workers changed mid-run (no effect on content); (ii) H5 metric refined post hoc (both versions reported); (iii) effort `medium` for all generation calls (preregistered); (iv) no human raters.

## Phase 8 — Paper (2026-11-03 evening)
- Sections: introduction, literature (58+6 verified refs), corpus, extraction/ontology, architecture, design, results, ablations, human/failure, discussion, conclusion, appendices A–H; all numbers via `paper/tables/numbers.tex` macros (1 343) generated from recorded artefacts.

## Phase 9 — Tests and release
- `make test`: 28 passed (synthetic fixtures only). Delivery report: `docs/DELIVERY_REPORT.md`.

| A-004 | 2026-11-03 20:15 | **Benchmark re-run after two defects found during the write-up** (by the results-writing agent): (i) the Markdown manuscript loader mapped `##` headings to level 2, so the fixture for jof_12777 collapsed into one section whose role was `other` and whose text was dropped from the material — generator and judge saw only the title and headings; (ii) tasks `referee_report` and `flag_causal_claims` were framed with the generic "Write the results" verb in contradiction with the brief. Fixes: `auditing.load_manuscript` (markdown levels), new engine mode `task`. Run 1 archived as `experiments/outputs/heldout_v1_run1/`; run 2 (`heldout_v1`) re-executes every cell for jof_12777, the two re-framed tasks for jof_12439 and the out-of-domain referee task, re-judges all sets; the 85 unaffected jof_12439 runs are reused. Preregistered analysis unchanged. | Scientific validity: half of the finance units in run 1 were no-material scaffold tasks. |
| A-004b | 2026-11-03 20:30 | While re-running, a third defect was found in the material builder for Markdown fixtures (abstract, tables/notes, figures and reference list and `other`-role sections were omitted) → fixed (`auditing.load_manuscript`, `generation.manuscript_material`); **the whole finance grid is re-run** (184 runs) and all sets re-judged; only the 10 unaffected out-of-domain runs (LaTeX manuscript; tasks draft_introduction, improve_abstract) are reused. Run 1 stays archived. | Same rationale: the material seen by generator and judge in run 1 was incomplete for both finance documents. |
| A-005 | 2026-11-03 20:30 | Budget envelope raised 90 → **110 USD** (spent 50.9 before the re-run; the re-run is estimated at ≈ 35 USD). | Needed to complete the corrected experiment ("continue jusqu'au bout", author instruction 2026-11-03). |

## Phase 7 — Benchmark heldout_v1 **run 2** (2026-11-03 20:35 → 21:40 ET) ✅ (supersedes run 1, archived)
- 199/199 runs ok (184 new + 10 reused OOD + 5 OOD referee re-run), 0 failures; 33 judge sets re-judged. Total API spend **93.2 USD** (cap 110, A-005).
- **Outcomes** (finance held-out, n = 24 paired): H1 D−B: clarity −0.12 [−0.31, 0.03], argument_logic +0.04 [−0.16, 0.22], contribution 0.00 [−0.28, 0.28] → **not supported**. H2 D−C: all |Δ| ≤ 0.12, intervals include 0 → **not supported**. **H3 E−D: methodological_rigor +0.96 [0.44, 1.47] Holm p = 0.0085; fidelity +0.46 [0.03, 0.69] Holm p = 0.024 → supported**; usefulness −0.33 [−0.78, 0.06] n.s. (trade-off). Exploratory E−A rigor +1.33 [0.84, 2.28] p = 0.002; E−B rigor +1.08 [0.56, 1.81] p = 0.009. Ablations: E − no_referee rigor +1.25 [0.63, 1.88], fidelity +0.56 [0.13, 1.13], argument_logic +0.50 [0.06, 0.94]; no_retrieval / no_profile / no_factcheck ≈ 0. H4: unverifiable citations ≈ 0 everywhere (−0.08 E−A); unsupported claims E−C −1.00 [−1.59, −0.16] (unadjusted p = 0.009), E−A −0.46, E−B −0.42 (n.s.); new numbers −0.17/−0.21/−0.42 (n.s.). H5: preregistered metric max 0.40 (self-overlap: improve_abstract 0.25, rewrite_literature 0.08); excluding the source document max 0.007, 100% < 0.02. Judge second pass (50 pairs): Spearman clarity 0.41, argument 0.58, contribution 0.84, rigor 0.80, fidelity 0.75, usefulness 0.53; MAD 0.26–0.40. Cost/latency: E 0.32 USD, 98 s vs D 0.15 USD, 49 s. OOD WP29 (n = 1): no E advantage.

## Release (2026-11-03 21:55 ET)
- `paper/main.pdf` 75 pages, 0 LaTeX errors, 0 undefined citations/references; 29/29 tests; `README.md`, `docs/DELIVERY_REPORT.md` final; series `README.md` and `HARMONISATION.md` rows 45 added.
- Pushed to spbgit `gitsrv:srv/git/wp45_uqo.git` (history rewritten once to drop the 70 MB `shingles.json` before the public push; it is regenerated on demand).