spb/wpai0002_uqo

1# Experiment matrix23Version 0.1 · 2026-11-09.45**Systems:**67- B0: identity.8- B1: rule-based generic paraphrase.9- B2: pretrained encoder–decoder fine-tune, not implemented.10- M1, M2, M3: see `docs/architecture/RESEARCH_PLAN.md` §5.1112**Splits** (`htr.data.splits`):1314- `train`, `val`, `test`: document-safe, near-duplicate-safe and author-group-safe, with seed 20261109.15- `heldout_domain`: whole disciplines.16- `heldout_period`: documents with `pub_year` < the cut-off. The smoke run used 1990.17- Held-out prompts (H1, H2) and held-out models (Sonnet 5.5, Haiku 5.5) are applied to **test** passages only.1819**Metrics.** Primary metrics follow `docs/PREREGISTRATION.md`. Defaults are 3 training seeds (1, 2, 3) and20document-clustered bootstrap CIs.2122**Status:**2324- *done*: evidence exists.25- *smoke*: zero-cost mock data only.26- *ready*: code exists, run not done.27- *planned*: code missing.2829| ID | RQ | Design | Systems | Splits | Metrics | Seeds | Status | Evidence |30|---|---|---|---|---|---|---|---|---|31| E0 | — | Vertical slice: corpus → mock rewrites → validation → tokenizer → M1-nano → B0/B1 predictions → metrics | B0, B1, M1-nano | smoke splits | all `PRIMARY` | 1 | smoke. M1 overfit CE 5.28 → 0.0022 in 150 steps; 300-step run best val CE 4.93. The M1 predictions and metrics files are not yet present. | `artifacts/runs/vs_smoke-m1-nano/`, `artifacts/predictions/vs_smoke/` |32| E1 | RQ1 | Paired feature shift original → rewrite, per variant A–E, on Opus 5.5 pilot pairs | rewrites (no model) | all pilot pairs | per-feature Δ, W1 (rewrite, orig), Holm over 18 features | — | ready (needs E2 pairs) | `artifacts/metrics/rq1_*` |33| E2 | — (Gate B) | Pilot rewrite of ~1 000 stratified passages × 6 generations (A–D ×1, E ×2), Message Batches, ceiling 30 USD; validation; human review of ≥ 100 stratified pairs | Opus 5.5 | train/val/test (rewriting happens after the splits) | acceptance per variant, failure reasons, spend | selection seed 1 | ready | `data/pairs/<pilot>/{ledger,manifest}.json`, `validated/validation_summary.json` |34| E3 | RQ2 | Baselines on document-safe test | B0, B1 (B2 when implemented) | test | chrF++, edit_sim, number recall, W1 | — | ready (B2 planned) | `artifacts/metrics/baselines_*` |35| E4 | RQ2/RQ5 | Main comparison at tiny size: M1 vs M2 vs M3 | M1, M2, M3 (tiny) | train → val (development) → test (once) | H1–H4 metrics | 1, 2, 3 | ready | `artifacts/runs/*-tiny-*` |36| E5 | RQ5 | Ablations of the loss and style components, using the `scripts/train.py` presets `m3_tokenonly`, `m3_nostyle`, `m3_noadv`, `m3_nofact`, `m3_nocon`, plus `--no-copy`, plus the M1 metadata control `--text-meta-prefix` | M3 variants, M1 + prefix | train/val | val chrF++, number recall, W1; adversary accuracy and post-hoc probe on frozen c | 1, 2, 3 | ready | `artifacts/runs/*-abl-*` |37| E6 | RQ4 | Held-out prompt: test passages rewritten with H1 and H2 (never used in training) | best M*, B0 | test | chrF++ gain vs B0 (H5) | 1 | ready (needs the `--variants H1,H2` run; prompts exist) | `data/pairs/<heldout_prompt>/` |38| E7 | RQ4 | Held-out model: test passages rewritten by Sonnet 5.5 and Haiku 5.5 (variants A–E) | best M*, B0 | test | chrF++ gain vs B0 (H5), factual metrics | 1 | ready (`--model` override) | `data/pairs/<heldout_model>/` |39| E8 | RQ4 | Held-out variant: train on A–D, test on E (`--variants A,B,C,D`) | M1, M3 | train/test | chrF++ gain | 1, 2, 3 | ready | — |40| E9 | RQ3 | Held-out discipline and held-out period | best M*, B0 | `heldout_domain`, `heldout_period` | H1–H4 metrics by stratum | 1 | ready | — |41| E10 | RQ5 | Scaling with model size: nano / tiny / small / base (large only if measured feasible) at fixed data | M1 or M3 | train/val | val CE, chrF++; FLOPs (6ND) | 1–3 | planned (needs data at Medium scale) | `artifacts/runs/*-scale-*` |42| E11 | RQ5 | Scaling with data: {¼, ½, 1} × pilot pairs; later Medium (20 000 passages and up) | tiny, small | train/val | val chrF++ vs pairs and tokens | 1–3 | planned | — |43| E12 | RQ2/RQ3 | Rewrite strength: gains as a function of edit_sim(rewrite, orig) deciles and of variant (C = strongest compression) | best M*, B0 | test | chrF++ gain by decile | — | ready (analysis) | — |44| E13 | RQ3 | Input length: gains by source length bucket (60–400 words) and paragraph vs section (hierarchical chunking is not implemented) | best M*, B0 | test | chrF++, number recall by bucket | — | ready (bucketing); planned (sections) | — |45| E14 | RQ6 | Non-identifiability: consistency across the 6 rewrites of one original; identification accuracy among confusable originals; oracle ceiling | best M* | test | ambiguity rate, identification accuracy | — | planned (code ready (`evaluation/identifiability.py`)) | — |46| E15 | RQ6 | Memorisation: n-gram and longest-common-span overlap of reconstructions with training originals; synthetic canaries; for B2, contamination audit | M*, B2 | train vs test | excess overlap, canary exposure | — | planned (code ready (`evaluation/memorization.py`)) | — |47| E16 | RQ2/RQ6 | **Human evaluation** (see protocol below) | B0, B1, best M*, original | test subset | fidelity, fluency, historical-style authenticity, factual safety | — | planned | `artifacts/human_eval/` |48| E17 | RQ6 | Error taxonomy on ≥ 200 quarantined and worst-scoring pairs (numbers, citations, negation, hedging, hallucinated content) | rewrites, M* | test | category rates | — | planned | — |49| E18 | — | Confidence calibration: token log-prob vs instance chrF++ (reliability curve) | M* | test | ECE-style binning | — | planned | — |50| E19 | RQ2 | Style probe (exploratory): logistic probe original-vs-rewrite, applied to reconstructions; must be refitted on a disjoint probe split | all | probe-train split ≠ eval | AUC, mean P(original) | — | ready (code), requires a disjoint split | — |5152**E16 human evaluation protocol.**5354- **Raters:** 3 domain-literate raters.55- **Items:** 120 test passages, stratified by discipline × variant.56- **Conditions:** 4 per item: original, B0, B1 and the best model.57- **Latin square:** each rater sees each item in exactly one condition, and each condition equally often.58- **Blinding:** all texts are shown in placeholder-restored form. Raters are not told which text is the original.59- **Scales:** 5-point scales for60  - fidelity to the source meaning (the reference shown is the rewrite);61  - fluency;62  - historical-style authenticity ("plausibly written before 2018");63  - factual safety: any changed number, citation, negation or hedge, flagged by the rater and also checked64    automatically.65- **Analysis:** agreement by Krippendorff's α; condition effects by an ordinal mixed model with rater and item66  random effects.67- **Pilot:** 20 items for rater calibration.68- **Approvals:** ethics and recruitment status to be confirmed with UQO before E16 runs.69