Gate D — Working M1/M2/M3 training, one-batch overfit, reproducible pilot results
Date: 2026-11-09/10 · Decision: GO with reservations (the training stack works and is reproducible; the pilot pair
set is too small for from-scratch models, and the dual-latent objective needed two corrections) · Evidence: artifacts/runs/*/
(manifest, train_log, train_summary, checkpoints), artifacts/metrics/pilot_val_exploratory/report.json, docs/DECISION_LOG.md
D17–D19, docs/INCIDENTS.md I1.
What was trained (pilot pairs: 5 383 validated, ≈ 3 900 train / 500 val; tokenizer 16 k; tiny = 27.4 M params at 16 k vocab)
| Run | Arch / objective | Best val CE | Steps | Device |
|---|---|---|---|---|
pilot_m1_tiny (seeds 1/2/3) |
M1 token loss | 3.69 / 4.09 / 4.31 | 2 250 each (early stop, patience 6) | laptop MPS / M4M64a |
pilot_m2_tiny (seeds 1/2/3) |
M2 metadata-conditioned | 3.72 / 5.03 / 3.71 | 2 250 | M4M64a |
pilot_m3_tokenonly_tiny (seeds 1/2/3) |
M3 architecture, token loss only | 3.80 / 3.74 / 3.58 | 2 250 | M4M64a |
pilot_m3_tiny (full objective, original code) |
M3 + style/adv/con/sem/orth/fact | 5.43 (step 250, then worse) | 1 750 | laptop |
diagnostics m1_fact, m3_factonly, m3_cononly, m3_semonly |
single auxiliary terms | 5.41 / 5.45 / 5.21 / 4.08 | ≤ 1 500 | laptop |
v2 (L_fact re-normalised, D18) m3, nostyle, noadv, nocon, nofact |
5.30 / 4.82 / 5.30 / 5.29 / 5.30 | ≤ 4 000 | laptop | |
v3 (D18 + masked InfoNCE, D19) m3, nostyle, noadv, nocon, nofact |
5.31 / 4.82 / 5.30 / 5.29 / 5.30 (unchanged: only 1.3 % of batch pairs share a passage) | ≤ 4 000 | laptop | |
denoise_m1_tiny_pilot → pilot_m1_tiny_denoiseinit (seeds 1, 2) |
denoising pretraining (12 000 steps on 17 k train passages, val CE 1.58) then pair fine-tuning | 1.61 / 1.61 | 1 750 | M3U96a, laptop |
→ pilot_m3light_tiny_denoiseinit, pilot_m3tokenonly_tiny_denoiseinit |
same init, M3-light objective / M3 token-only (runs cut by MPS OOM at best 1.61 / 1.60) | 1.61 / 1.60 | ≈ 1 500 | laptop |
pilot_b2_t5small |
B2: T5-small (60.5 M) fine-tune, 3 000 steps | 1.01 (T5 vocab; not comparable) | M3U96b |
One-batch overfit test: CE 5.28 → 0.002 in 150 steps (artifacts/runs/vs_smoke-m1-nano/overfit_test.json). Every run has a manifest with
git commit, seeds, library versions, hardware, token counts, accelerator seconds and 6ND FLOPs.
Exploratory validation results (502 pairs; CIs = 95 % document-clustered bootstrap, 2 000 resamples)
| System | chrF++ | edit sim | ROUGE-L | number recall | placeholders exact | W1 to originals ↓ |
|---|---|---|---|---|---|---|
| B0 identity | 71.2 [69.7, 72.7] | 0.666 | 0.690 | 1.000 | 1.000 | 0.170 |
| B1 generic paraphrase | 68.4 [66.9, 70.0] | 0.561 | 0.612 | 1.000 | 1.000 | 0.281 |
| B2 T5-small fine-tune | 68.8 [67.7, 69.9] | 0.638 | 0.671 | 0.945 | 0.691 | 0.112 |
| HTR denoiser only (no pair training) | 58.1 [55.2, 60.9] | 0.456 | 0.493 | 0.856 | 0.741 | 0.215 |
| HTR-tiny denoise-init + pair fine-tune | 63.6 [62.2, 64.9] | 0.578 | 0.600 | 0.931 | 0.884 | 0.150 |
| HTR-tiny denoise-init, seed 2 | 64.1 [62.5, 65.5] | 0.582 | 0.603 | |||
| HTR-tiny denoise-init, M3-light objective | 64.0 [62.6, 65.3] | 0.584 | 0.606 | 0.941 | ||
| HTR-tiny denoise-init, M3 token-only | 64.1 [62.2, 65.6] | 0.584 | 0.606 | 0.952 | ||
| HTR-tiny from scratch | 32.9 [30.9, 34.5] | 0.287 | 0.241 | 0.571 | 0.633 | 1.699 |
Paired differences vs B0 (chrF++): B2 −2.4 [−3.3, −1.6]; denoise-init −7.6 [−8.6, −6.7]. No system beats identity on lexical metrics at pilot scale (≈ 1.7 M target tokens). B2 and denoise-init HTR move the stylometric distribution closer to the originals than the rewrites are (W1 0.112 and 0.150 vs 0.170) — the "distribution matching ≠ verbatim reconstruction" distinction the mandate asks us to keep separate.
Findings and corrections
- From-scratch sequence models cannot learn the inverse map from ≈ 3 900 pairs; denoising pretraining on unlabelled originals (zero API cost) recovers most of the gap (chrF 32.9 → 63.6) — D17.
- The factual-span loss as first implemented (normalised by the number of protected tokens) destroyed generalisation for every model that used it, including M1 — D18 (fixed: normalised by all tokens).
- The contrastive term treated the other rewrites of the same original as negatives — D19 (fixed: same-passage masking).
- After D18 alone, M3 variants still generalise worse than token-only (v2 table); the style/adversarial branch is the remaining cause (v3 confirms: masking changed nothing). At pilot scale the dual-latent objective has no demonstrated benefit; this is reported as a negative/undetermined finding, not hidden.
- Seed variance is large at this scale (M1 3.69–4.31; M2 3.71–5.03): single-seed comparisons are not conclusive.
- Incident I1: two cluster nodes froze under Medium-scale pretraining (MPS memory near the limit); mitigations recorded.
Decision
GO to Gate E conditional on (a) the Medium-stage rewrite budget (author approval; configs/rewriting/medium.json), and (b)
Medium-scale denoising pretraining completing on a node with memory caps. The confirmatory test scoring (pre-registration v1.0)
has not been run; M* will be selected on validation among the Medium-stage candidates.