Experiment matrix
Version 0.1 · 2026-11-09.
Systems:
- B0: identity.
- B1: rule-based generic paraphrase.
- B2: pretrained encoder–decoder fine-tune, not implemented.
- M1, M2, M3: see
docs/architecture/RESEARCH_PLAN.md§5.
Splits (htr.data.splits):
train,val,test: document-safe, near-duplicate-safe and author-group-safe, with seed 20261109.heldout_domain: whole disciplines.heldout_period: documents withpub_year< the cut-off. The smoke run used 1990.- Held-out prompts (H1, H2) and held-out models (Sonnet 5.5, Haiku 5.5) are applied to test passages only.
Metrics. Primary metrics follow docs/PREREGISTRATION.md. Defaults are 3 training seeds (1, 2, 3) and
document-clustered bootstrap CIs.
Status:
- done: evidence exists.
- smoke: zero-cost mock data only.
- ready: code exists, run not done.
- planned: code missing.
| ID | RQ | Design | Systems | Splits | Metrics | Seeds | Status | Evidence |
|---|---|---|---|---|---|---|---|---|
| E0 | — | Vertical slice: corpus → mock rewrites → validation → tokenizer → M1-nano → B0/B1 predictions → metrics | B0, B1, M1-nano | smoke splits | all PRIMARY |
1 | smoke. M1 overfit CE 5.28 → 0.0022 in 150 steps; 300-step run best val CE 4.93. The M1 predictions and metrics files are not yet present. | artifacts/runs/vs_smoke-m1-nano/, artifacts/predictions/vs_smoke/ |
| E1 | RQ1 | Paired feature shift original → rewrite, per variant A–E, on Opus 5.5 pilot pairs | rewrites (no model) | all pilot pairs | per-feature Δ, W1 (rewrite, orig), Holm over 18 features | — | ready (needs E2 pairs) | artifacts/metrics/rq1_* |
| E2 | — (Gate B) | Pilot rewrite of ~1 000 stratified passages × 6 generations (A–D ×1, E ×2), Message Batches, ceiling 30 USD; validation; human review of ≥ 100 stratified pairs | Opus 5.5 | train/val/test (rewriting happens after the splits) | acceptance per variant, failure reasons, spend | selection seed 1 | ready | data/pairs/<pilot>/{ledger,manifest}.json, validated/validation_summary.json |
| E3 | RQ2 | Baselines on document-safe test | B0, B1 (B2 when implemented) | test | chrF++, edit_sim, number recall, W1 | — | ready (B2 planned) | artifacts/metrics/baselines_* |
| E4 | RQ2/RQ5 | Main comparison at tiny size: M1 vs M2 vs M3 | M1, M2, M3 (tiny) | train → val (development) → test (once) | H1–H4 metrics | 1, 2, 3 | ready | artifacts/runs/*-tiny-* |
| E5 | RQ5 | Ablations of the loss and style components, using the scripts/train.py presets m3_tokenonly, m3_nostyle, m3_noadv, m3_nofact, m3_nocon, plus --no-copy, plus the M1 metadata control --text-meta-prefix |
M3 variants, M1 + prefix | train/val | val chrF++, number recall, W1; adversary accuracy and post-hoc probe on frozen c | 1, 2, 3 | ready | artifacts/runs/*-abl-* |
| E6 | RQ4 | Held-out prompt: test passages rewritten with H1 and H2 (never used in training) | best M*, B0 | test | chrF++ gain vs B0 (H5) | 1 | ready (needs the --variants H1,H2 run; prompts exist) |
data/pairs/<heldout_prompt>/ |
| E7 | RQ4 | Held-out model: test passages rewritten by Sonnet 5.5 and Haiku 5.5 (variants A–E) | best M*, B0 | test | chrF++ gain vs B0 (H5), factual metrics | 1 | ready (--model override) |
data/pairs/<heldout_model>/ |
| E8 | RQ4 | Held-out variant: train on A–D, test on E (--variants A,B,C,D) |
M1, M3 | train/test | chrF++ gain | 1, 2, 3 | ready | — |
| E9 | RQ3 | Held-out discipline and held-out period | best M*, B0 | heldout_domain, heldout_period |
H1–H4 metrics by stratum | 1 | ready | — |
| E10 | RQ5 | Scaling with model size: nano / tiny / small / base (large only if measured feasible) at fixed data | M1 or M3 | train/val | val CE, chrF++; FLOPs (6ND) | 1–3 | planned (needs data at Medium scale) | artifacts/runs/*-scale-* |
| E11 | RQ5 | Scaling with data: {¼, ½, 1} × pilot pairs; later Medium (20 000 passages and up) | tiny, small | train/val | val chrF++ vs pairs and tokens | 1–3 | planned | — |
| E12 | RQ2/RQ3 | Rewrite strength: gains as a function of edit_sim(rewrite, orig) deciles and of variant (C = strongest compression) | best M*, B0 | test | chrF++ gain by decile | — | ready (analysis) | — |
| E13 | RQ3 | Input length: gains by source length bucket (60–400 words) and paragraph vs section (hierarchical chunking is not implemented) | best M*, B0 | test | chrF++, number recall by bucket | — | ready (bucketing); planned (sections) | — |
| E14 | RQ6 | Non-identifiability: consistency across the 6 rewrites of one original; identification accuracy among confusable originals; oracle ceiling | best M* | test | ambiguity rate, identification accuracy | — | planned (code ready (evaluation/identifiability.py)) |
— |
| E15 | RQ6 | Memorisation: n-gram and longest-common-span overlap of reconstructions with training originals; synthetic canaries; for B2, contamination audit | M*, B2 | train vs test | excess overlap, canary exposure | — | planned (code ready (evaluation/memorization.py)) |
— |
| E16 | RQ2/RQ6 | Human evaluation (see protocol below) | B0, B1, best M*, original | test subset | fidelity, fluency, historical-style authenticity, factual safety | — | planned | artifacts/human_eval/ |
| E17 | RQ6 | Error taxonomy on ≥ 200 quarantined and worst-scoring pairs (numbers, citations, negation, hedging, hallucinated content) | rewrites, M* | test | category rates | — | planned | — |
| E18 | — | Confidence calibration: token log-prob vs instance chrF++ (reliability curve) | M* | test | ECE-style binning | — | planned | — |
| E19 | RQ2 | Style probe (exploratory): logistic probe original-vs-rewrite, applied to reconstructions; must be refitted on a disjoint probe split | all | probe-train split ≠ eval | AUC, mean P(original) | — | ready (code), requires a disjoint split | — |
E16 human evaluation protocol.
- Raters: 3 domain-literate raters.
- Items: 120 test passages, stratified by discipline × variant.
- Conditions: 4 per item: original, B0, B1 and the best model.
- Latin square: each rater sees each item in exactly one condition, and each condition equally often.
- Blinding: all texts are shown in placeholder-restored form. Raters are not told which text is the original.
- Scales: 5-point scales for
- fidelity to the source meaning (the reference shown is the rewrite);
- fluency;
- historical-style authenticity ("plausibly written before 2018");
- factual safety: any changed number, citation, negation or hedge, flagged by the rater and also checked automatically.
- Analysis: agreement by Krippendorff's α; condition effects by an ordinal mixed model with rater and item random effects.
- Pilot: 20 items for rater calibration.
- Approvals: ethics and recruitment status to be confirmed with UQO before E16 runs.