spb/wpai0002_uqo

1# Pre-registration — WP-AI-0002 (v1.0, FROZEN 2026-11-09 17:59 UTC)23Status: **v1.0, frozen 2026-11-09 17:59 UTC, pre-specified in-session by the operating assistant** (the SESOI values are the v0.1 proposals; the author had not yet confirmed them at freeze time — this is stated in the paper as *pre-specified in-session*, not as author-reviewed pre-registration).4Version 1.0 must be frozen **before any confirmatory split is scored**. The confirmatory splits are `test`,5`heldout_domain`, `heldout_period`, held-out prompt and held-out model.67## 1. Units, systems, data89- **Unit of analysis.** A validated pair: an LLM rewrite as source and the original passage as target, both in10  placeholder form.11- **Clustering.** All inference is clustered by document (`doc_id`), which covers the multiple variants and12  samples per passage.13- **Systems.**14  - B0 identity and B1 generic paraphrase (`evaluation/baselines.py`).15  - The best HTR model, M\*. M\* is selected on **validation only**, among M1/M2/M3 and the ablations, and is16    fixed in the freeze record (run id + checkpoint sha256).17  - B2, if implemented, is a secondary comparator.18- **Primary test set.** Document-safe, author-group-safe `test` split of the corpus that is current at freeze19  time. Its build summary and splits hash are recorded at freeze.20- **Decoding.** Greedy, unless beam width is chosen on validation and recorded at freeze.2122## 2. Confirmatory hypotheses2324| ID | Hypothesis | Primary metric | SESOI (proposed) | Test |25|---|---|---|---|---|26| H1 | M\* reconstructs the original better than B0 **and** better than B1 | chrF++ (sacrebleu `CHRF(word_order=2)`, sentence level, mean over items) | +2.0 chrF++ points vs each baseline | one-sided: the 95 % CI lower bound of the paired difference must be > 0; practical relevance: the point estimate must be ≥ SESOI |27| H2 | Same comparison as H1 | normalised Levenshtein similarity (`edit_sim`) | +0.02 | as H1 |28| H3 | M\* does not lose numerical facts relative to B0 (non-inferiority) | number recall vs the original (`metrics.factual`) | margin −0.01 | the lower CI bound of (M\* − B0) must be > −0.01 |29| H4 | Reconstructions are distributionally closer to the originals than the rewrites are | `W1_mean` over 18 standardised stylometric features (`stylometry.distribution_distances`) | ≥ 10 % relative reduction vs W1(rewrite, original) | document-clustered bootstrap of the W1 difference: resample documents, then recompute both W1 values |30| H5 | The H1 gain over B0 persists for held-out prompts (H1/H2) and held-out models (Sonnet 5.5, Haiku 5.5) | chrF++ gain vs B0 on each held-out set | +1.0 | one-sided, per held-out set |31| H6 | The dual-latent model (M3) beats token-loss M1 at the same size, data and step budget (mean of 3 seeds) | chrF++ on test | +1.0 | paired bootstrap on seed-averaged item scores |3233**Multiplicity.**3435- Holm–Bonferroni at family-wise α = 0.05 over the primary family {H1-vs-B0, H1-vs-B1, H2-vs-B0, H2-vs-B1,36  H3, H4}.37- H5 is a separate family with Holm across the four held-out sets.38- H6 is tested alone.3940**Interpretation rules.**4142- A hypothesis is "supported" only if it is significant after Holm **and** its point estimate reaches the SESOI.43- A significant result below the SESOI is reported as "statistically detectable but practically small".44- Non-significant results are reported with their CIs as null findings.4546## 3. Statistical procedure4748- **Bootstrap.** Document-clustered paired bootstrap (`evaluation/bootstrap.py`). Documents are resampled with49  replacement, and every item of a selected document enters the resample.50  - Confirmatory analyses use **10 000** resamples with seed 0. The current code default in `runner.py` is51    1 000, so it must be raised before freeze.52- **p-values.** Two-sided p = 2·min(P(d̄\* ≤ 0), P(d̄\* > 0)). One-sided decisions use the CI bounds above.53- **Reporting.** Means, 95 % percentile CIs, the number of items and the number of documents.5455## 4. Exploratory (not confirmatory) analyses5657These analyses are reported clearly labelled as exploratory:5859- **Feature shifts.** RQ1 per-feature shifts, with Holm over 18 features as a descriptive guard.60- **Heterogeneity (RQ3).** Breakdowns by discipline, era, section role, variant, rewrite strength and input61  length.62- **Other metrics.** Exact match, BLEU, ROUGE-L, placeholder exactness, negation and hedge deltas.63- **Diagnostics.** The ablations (E5), the scaling curves (E10, E11), calibration (E18) and the error taxonomy64  (E17).65- **Style probe.** The style probe's AUC and mean P(original) on reconstructions are **exploratory only**. They66  are never evidence of human authorship. The probe must be fitted on a split that is disjoint from the67  evaluated items.68- **RQ6 analyses.** Non-identifiability and memorisation are descriptive.6970## 5. Freeze procedure71721. The PREREGISTRATION v1.0 must state:73   - the final SESOIs;74   - the M\* selection rule;75   - the decoding settings;76   - the corpus and splits hashes;77   - the held-out sets.782. Commit v1.0, record the **commit hash** in `docs/DECISION_LOG.md` and in a new section, "§6 Freeze record",79   of this file, and push it to spbgit. Optionally, also deposit it on OSF.803. Until the freeze commit exists:81   - only `train` and `val` may be scored;82   - `scripts/predict.py` / `evaluate.py` may not be run on confirmatory splits, except B0/B1 sanity checks that83     report **no** model comparison.844. After freeze, run the confirmatory scoring **once**. Any deviation is reported as a dated amendment with its85   reason. Results obtained after a deviation are labelled exploratory.8687## 6. Freeze record (v1.0)8889- Date: 2026-11-09 17:59 UTC90- M\* selection rule: the candidate run (same size, data and step budget) with the lowest validation token cross-entropy, recorded with its checkpoint sha256 before any confirmatory scoring91- Decoding: greedy, KV-cached, placeholder protocol of the pair's protect_version, max_new_tokens = 1.6 x input + 1692- Confirmatory bootstrap: 10 000 resamples, seed 0 (`scripts/evaluate.py --n-boot 10000 --confirmatory`)93- Corpus: `data/parsed/pilot_pmc` · splits: `data/splits/pilot_pmc/splits.jsonl` · pairs: `data/pairs/pilot_opus/validated/pairs.jsonl`94- sha256 corpus build_summary.json: `d028b5aec44b10c3db5c5ae9a2249b6e64b3c5043175257a2e3c22884b0a3dcd`95- sha256 documents.jsonl: `9a244f007974226557c08d61f5dd6ececf1969a9ec883838d2ea158dcbc1232d`96- sha256 splits.jsonl: `f0088cccbd5e94a6926e81fc2dd4e342a60fe9cc2d23245d62973ed871d6e6e0`97- sha256 validated pairs.jsonl: `695fefdf9f6b1e755b90a63e7ed830dd9bc00f79781de891521092d5436f1b8f`98- sha256 held-out pairs pilot_heldout_models: `3eebdbdbdbf453867cf11dcd294a4a52b2346617c84d91158333f5e87b591d20`99- sha256 held-out pairs pilot_heldout_prompts: `49592652ce4a94085bf6752498b422627224d7f3ef2f4350b1ca34c450b39b36`100- Held-out sets: `test`, `heldout_domain` (chemistry as labelled at split time), `heldout_period` (pub_year < 1950; 4 passages only), held-out prompts H1/H2, held-out models Sonnet 5.5 / Haiku 5.5101- Commit hash: `e74b71266d5c0e8c20f181a50df139f52c771d0f` (the freeze commit; this line was added by the following commit)102103## 7. Amendments104105- **A1 (2026-11-09, before any confirmatory scoring).** Implementation correction of H4: `runner.py` now standardises every stylometric W1 comparison with ONE common scale (pooled SD of originals ∪ rewrites, floored), so W1(recon, orig) and W1(rewrite, orig) are comparable. The hypothesis, metric and SESOI are unchanged. Reason: the paper-writing review found that the two distances previously used different scales.106