# Preregistration — WP45 / JoF-250 evaluation protocol `heldout_v1`

**Frozen on 2026-11-03 (America/Toronto), before any benchmark run.** Code version at freeze: see `git log` entry "preregistration frozen". Any later change is an amendment logged in `docs/PROGRESS.md`.

## Research question
Do abstract, domain-specific writing conventions extracted from a pre-ChatGPT finance corpus (JoF 2017–2020) improve LLM-assisted scholarly writing and review beyond standard prompting and conventional retrieval, while maintaining originality, factual accuracy and reproducibility?

## Hypotheses (directional, tested as paired contrasts within task × document)
- **H1** D (structured profiles + rhetorical retrieval) > B (strong prompt) on judge-rated *argument clarity* and *contribution positioning*.
- **H2** D > C (conventional RAG over raw excerpts) on the finance writing tasks (same two outcomes + *usefulness*).
- **H3** E (D + verification + specialist referee) > D on *methodological rigor* and *fidelity*, with no loss in numeric preservation (preserved_rate_E ≥ preserved_rate_D).
- **H4** E has fewer *unverifiable citations* (author–year strings absent from the source reference list) and fewer *unsupported-claim sentences* than A, B and C.
- **H5** Any D/E advantage persists on held-out documents and on the out-of-domain manuscript, and is not explained by overlap: max 8-gram overlap share with any corpus document < 0.02 for every output.

## Design
- **Units**: task × source document × condition × repetition. Source documents are drawn from the frozen held-out split (`data/splits/heldout_v1.json`, 55 documents never used for profiles or prompt tuning): two empirical JoF articles of different subfields (selected by the fixture script with seed 45 among held-out empirical articles with ≥ 6 tables), plus one **out-of-domain** manuscript: UQO WP29 (methods/computer-science flavoured, LaTeX, author's own verified results).
- **Conditions**: A baseline (minimal instruction); B strong academic-writing prompt; C = B + conventional RAG (6 raw ≤400-char excerpts by hybrid similarity, train split only); D = B + measured profile summary + 6 structured rhetorical units with provenance; E = D + deterministic fact-check (numeric preservation, citation check, unsupported claims, overlap) + specialist-referee revision pass. **Ablations of E**: no_retrieval, no_profile, no_referee, no_factcheck.
- **Tasks** (8, from the mandate): draft_introduction (from a verified brief: abstract, headings, table/figure captions; the published introduction is withheld), rewrite_literature (input = a degraded "laundry-list" positioning built deterministically from the paper's own literature paragraphs), explain_specification, interpret_results (fixed table notes + the paper's own results paragraph as the fixed facts), improve_table (caption + notes), flag_causal_claims, referee_report, improve_abstract (findings must be unchanged).
- **Controls**: same model (`claude-opus-5-5`), same effort (`medium`) and `max_tokens`, same manuscript material and brief for every condition; temperature is not configurable on this model (adaptive thinking); nondeterminism is handled by repetitions (rep 1 for the full grid; rep 2 for the main grid A–E on four tasks: draft_introduction, improve_abstract, referee_report, interpret_results). Costs, tokens, latency, retries, prompt versions and model ids are logged per call (`data/annotations/api_ledger.jsonl`, `experiments/outputs/heldout_v1/runs.jsonl`).
- **Judge**: `claude-sonnet-5-5` (different family member from the generator to limit self-preference), blind to condition labels, sees all condition outputs of one (task, document, rep) in random order under random letters, scores each 1–7 on six primary outcomes (clarity, argument_logic, contribution_positioning, methodological_rigor, fidelity, usefulness) with a calibrated rubric (`prompts/evaluation/judge_rubric_v1.md`), and gives a one-line justification. Judge consistency is measured by re-judging a 25% random subset with a second random order (agreement = mean absolute score difference and rank correlation).
- **Deterministic secondary outcomes** per output: preserved_rate and new_rate of numbers vs the source material, count of unverifiable author–year citations, count of unsupported-claim sentences, max 8-gram overlap share with the corpus, latency, cost, tokens.
- **Human evaluation**: not available in this session; protocol and rating sheets prepared (`experiments/protocols/human_rating_sheet.md`); H1 as stated in the mandate (human-rated) therefore cannot be tested here and is reported as *judge-rated* only.

## Analysis plan (fixed)
- Primary: for each hypothesis contrast and outcome, the paired difference within (task, document, rep), averaged, with a 95% cluster bootstrap CI (clusters = task × document; 2,000 resamples, seed 45) and a paired Wilcoxon signed-rank test. Holm correction across the six primary outcomes within each hypothesis. Effect size = mean paired difference on the 1–7 scale and Cliff's delta.
- Secondary outcomes: same paired bootstrap; counts reported with rates.
- Ablations: each ablation vs full E, same procedure.
- Out-of-domain: reported separately (n = 1 document), descriptively only.
- Significance is declared only at Holm-adjusted p < 0.05 **and** CI excluding 0; the sample is small (2 finance documents × 8 tasks) and the study is explicitly a pilot of the protocol; this is stated in the paper.
- No results are written before the runs complete; failed/refused runs are reported, not dropped silently.

## Contamination statement
The generator may have seen the held-out JoF articles during pretraining. The design mitigates but does not remove this: the published introduction is withheld from the brief, outputs are checked for overlap with the corpus, and author–year citations are verified against the reference list. Results are interpreted as conditional on this caveat.