spb/wpai0002_uqo

1# Gate D — Working M1/M2/M3 training, one-batch overfit, reproducible pilot results23**Date:** 2026-11-09/10 · **Decision: GO with reservations** (the training stack works and is reproducible; the pilot pair4set is too small for from-scratch models, and the dual-latent objective needed two corrections) · Evidence: `artifacts/runs/*/`5(manifest, train_log, train_summary, checkpoints), `artifacts/metrics/pilot_val_exploratory/report.json`, `docs/DECISION_LOG.md`6D17–D19, `docs/INCIDENTS.md` I1.78## What was trained (pilot pairs: 5 383 validated, ≈ 3 900 train / 500 val; tokenizer 16 k; tiny = 27.4 M params at 16 k vocab)910| Run | Arch / objective | Best val CE | Steps | Device |11|---|---|---|---|---|12| `pilot_m1_tiny` (seeds 1/2/3) | M1 token loss | **3.69** / 4.09 / 4.31 | 2 250 each (early stop, patience 6) | laptop MPS / M4M64a |13| `pilot_m2_tiny` (seeds 1/2/3) | M2 metadata-conditioned | 3.72 / 5.03 / 3.71 | 2 250 | M4M64a |14| `pilot_m3_tokenonly_tiny` (seeds 1/2/3) | M3 architecture, token loss only | 3.80 / 3.74 / **3.58** | 2 250 | M4M64a |15| `pilot_m3_tiny` (full objective, original code) | M3 + style/adv/con/sem/orth/fact | 5.43 (step 250, then worse) | 1 750 | laptop |16| diagnostics `m1_fact`, `m3_factonly`, `m3_cononly`, `m3_semonly` | single auxiliary terms | 5.41 / 5.45 / 5.21 / **4.08** | ≤ 1 500 | laptop |17| v2 (L_fact re-normalised, D18) `m3`, `nostyle`, `noadv`, `nocon`, `nofact` | | 5.30 / **4.82** / 5.30 / 5.29 / 5.30 | ≤ 4 000 | laptop |18| v3 (D18 + masked InfoNCE, D19) `m3`, `nostyle`, `noadv`, `nocon`, `nofact` | | 5.31 / 4.82 / 5.30 / 5.29 / 5.30 (unchanged: only 1.3 % of batch pairs share a passage) | ≤ 4 000 | laptop |19| `denoise_m1_tiny_pilot` → `pilot_m1_tiny_denoiseinit` (seeds 1, 2) | denoising pretraining (12 000 steps on 17 k train passages, val CE 1.58) then pair fine-tuning | **1.61 / 1.61** | 1 750 | M3U96a, laptop |20| → `pilot_m3light_tiny_denoiseinit`, `pilot_m3tokenonly_tiny_denoiseinit` | same init, M3-light objective / M3 token-only (runs cut by MPS OOM at best 1.61 / 1.60) | 1.61 / 1.60 | ≈ 1 500 | laptop |21| `pilot_b2_t5small` | B2: T5-small (60.5 M) fine-tune, 3 000 steps | 1.01 (T5 vocab; not comparable) | | M3U96b |2223One-batch overfit test: CE 5.28 → 0.002 in 150 steps (`artifacts/runs/vs_smoke-m1-nano/overfit_test.json`). Every run has a manifest with24git commit, seeds, library versions, hardware, token counts, accelerator seconds and 6ND FLOPs.2526## Exploratory validation results (502 pairs; CIs = 95 % document-clustered bootstrap, 2 000 resamples)2728| System | chrF++ | edit sim | ROUGE-L | number recall | placeholders exact | W1 to originals ↓ |29|---|---|---|---|---|---|---|30| B0 identity | **71.2** [69.7, 72.7] | **0.666** | **0.690** | 1.000 | 1.000 | 0.170 |31| B1 generic paraphrase | 68.4 [66.9, 70.0] | 0.561 | 0.612 | 1.000 | 1.000 | 0.281 |32| B2 T5-small fine-tune | 68.8 [67.7, 69.9] | 0.638 | 0.671 | 0.945 | 0.691 | **0.112** |33| HTR denoiser only (no pair training) | 58.1 [55.2, 60.9] | 0.456 | 0.493 | 0.856 | 0.741 | 0.215 |34| HTR-tiny denoise-init + pair fine-tune | 63.6 [62.2, 64.9] | 0.578 | 0.600 | 0.931 | 0.884 | 0.150 |35| HTR-tiny denoise-init, seed 2 | 64.1 [62.5, 65.5] | 0.582 | 0.603 | | | |36| HTR-tiny denoise-init, M3-light objective | 64.0 [62.6, 65.3] | 0.584 | 0.606 | 0.941 | | |37| HTR-tiny denoise-init, M3 token-only | 64.1 [62.2, 65.6] | 0.584 | 0.606 | 0.952 | | |38| HTR-tiny from scratch | 32.9 [30.9, 34.5] | 0.287 | 0.241 | 0.571 | 0.633 | 1.699 |3940Paired differences vs B0 (chrF++): B2 −2.4 [−3.3, −1.6]; denoise-init −7.6 [−8.6, −6.7]. No system beats identity on lexical41metrics at pilot scale (≈ 1.7 M target tokens). B2 and denoise-init HTR move the stylometric distribution closer to the originals42than the rewrites are (W1 0.112 and 0.150 vs 0.170) — the "distribution matching ≠ verbatim reconstruction" distinction the mandate43asks us to keep separate.4445## Findings and corrections461. From-scratch sequence models cannot learn the inverse map from ≈ 3 900 pairs; denoising pretraining on unlabelled47   originals (zero API cost) recovers most of the gap (chrF 32.9 → 63.6) — D17.482. The factual-span loss as first implemented (normalised by the number of protected tokens) destroyed generalisation for every49   model that used it, including M1 — D18 (fixed: normalised by all tokens).503. The contrastive term treated the other rewrites of the same original as negatives — D19 (fixed: same-passage masking).514. After D18 alone, M3 variants still generalise worse than token-only (v2 table); the style/adversarial branch is the remaining52   cause (v3 confirms: masking changed nothing). At pilot scale the dual-latent objective has **no demonstrated benefit**; this is reported53   as a negative/undetermined finding, not hidden.545. Seed variance is large at this scale (M1 3.69–4.31; M2 3.71–5.03): single-seed comparisons are not conclusive.556. Incident I1: two cluster nodes froze under Medium-scale pretraining (MPS memory near the limit); mitigations recorded.5657## Decision58GO to Gate E **conditional on** (a) the Medium-stage rewrite budget (author approval; `configs/rewriting/medium.json`), and (b)59Medium-scale denoising pretraining completing on a node with memory caps. The confirmatory test scoring (pre-registration v1.0)60has **not** been run; M\* will be selected on validation among the Medium-stage candidates.61