Gate D report (GO with reservations; v3 pending)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 changed file +56 −0
added
docs/gates/GATE_D.md
+56 −0
@@ -0,0 +1,56 @@ | ||
| 1 | +# Gate D — Working M1/M2/M3 training, one-batch overfit, reproducible pilot results | |
| 2 | + | |
| 3 | +**Date:** 2026-11-09/10 · **Decision: GO with reservations** (the training stack works and is reproducible; the pilot pair | |
| 4 | +set is too small for from-scratch models, and the dual-latent objective needed two corrections) · Evidence: `artifacts/runs/*/` | |
| 5 | +(manifest, train_log, train_summary, checkpoints), `artifacts/metrics/pilot_val_exploratory/report.json`, `docs/DECISION_LOG.md` | |
| 6 | +D17–D19, `docs/INCIDENTS.md` I1. | |
| 7 | + | |
| 8 | +## What was trained (pilot pairs: 5 383 validated, ≈ 3 900 train / 500 val; tokenizer 16 k; tiny = 27.4 M params at 16 k vocab) | |
| 9 | + | |
| 10 | +| Run | Arch / objective | Best val CE | Steps | Device | | |
| 11 | +|---|---|---|---|---| | |
| 12 | +| `pilot_m1_tiny` (seeds 1/2/3) | M1 token loss | **3.69** / 4.09 / 4.31 | 2 250 each (early stop, patience 6) | laptop MPS / M4M64a | | |
| 13 | +| `pilot_m2_tiny` (seeds 1/2/3) | M2 metadata-conditioned | 3.72 / 5.03 / 3.71 | 2 250 | M4M64a | | |
| 14 | +| `pilot_m3_tokenonly_tiny` (seeds 1/2/3) | M3 architecture, token loss only | 3.80 / 3.74 / **3.58** | 2 250 | M4M64a | | |
| 15 | +| `pilot_m3_tiny` (full objective, original code) | M3 + style/adv/con/sem/orth/fact | 5.43 (step 250, then worse) | 1 750 | laptop | | |
| 16 | +| diagnostics `m1_fact`, `m3_factonly`, `m3_cononly`, `m3_semonly` | single auxiliary terms | 5.41 / 5.45 / 5.21 / **4.08** | ≤ 1 500 | laptop | | |
| 17 | +| v2 (L_fact re-normalised, D18) `m3`, `nostyle`, `noadv`, `nocon`, `nofact` | | 5.30 / **4.82** / 5.30 / 5.29 / 5.30 | ≤ 4 000 | laptop | | |
| 18 | +| v3 (D18 + masked InfoNCE, D19) | | pending (running) | | laptop | | |
| 19 | +| `denoise_m1_tiny_pilot` → `pilot_m1_tiny_denoiseinit` | denoising pretraining (12 000 steps on 17 k train passages, val CE 1.58) then pair fine-tuning | **1.61** | 1 750 | M3U96a | | |
| 20 | +| `pilot_b2_t5small` | B2: T5-small (60.5 M) fine-tune, 3 000 steps | 1.01 (T5 vocab; not comparable) | | M3U96b | | |
| 21 | + | |
| 22 | +One-batch overfit test: CE 5.28 → 0.002 in 150 steps (`artifacts/runs/vs_smoke-m1-nano/overfit_test.json`). Every run has a manifest with | |
| 23 | +git commit, seeds, library versions, hardware, token counts, accelerator seconds and 6ND FLOPs. | |
| 24 | + | |
| 25 | +## Exploratory validation results (502 pairs; CIs = 95 % document-clustered bootstrap, 2 000 resamples) | |
| 26 | + | |
| 27 | +| System | chrF++ | edit sim | ROUGE-L | number recall | placeholders exact | W1 to originals ↓ | | |
| 28 | +|---|---|---|---|---|---|---| | |
| 29 | +| B0 identity | **71.2** [69.7, 72.7] | **0.666** | **0.690** | 1.000 | 1.000 | 0.170 | | |
| 30 | +| B1 generic paraphrase | 68.4 [66.9, 70.0] | 0.561 | 0.612 | 1.000 | 1.000 | 0.281 | | |
| 31 | +| B2 T5-small fine-tune | 68.8 [67.7, 69.9] | 0.638 | 0.671 | 0.945 | 0.691 | **0.112** | | |
| 32 | +| HTR denoiser only (no pair training) | 58.1 [55.2, 60.9] | 0.456 | 0.493 | 0.856 | 0.741 | 0.215 | | |
| 33 | +| HTR-tiny denoise-init + pair fine-tune | 63.6 [62.2, 64.9] | 0.578 | 0.600 | 0.931 | 0.884 | 0.150 | | |
| 34 | +| HTR-tiny from scratch | 32.9 [30.9, 34.5] | 0.287 | 0.241 | 0.571 | 0.633 | 1.699 | | |
| 35 | + | |
| 36 | +Paired differences vs B0 (chrF++): B2 −2.4 [−3.3, −1.6]; denoise-init −7.6 [−8.6, −6.7]. No system beats identity on lexical | |
| 37 | +metrics at pilot scale (≈ 1.7 M target tokens). B2 and denoise-init HTR move the stylometric distribution closer to the originals | |
| 38 | +than the rewrites are (W1 0.112 and 0.150 vs 0.170) — the "distribution matching ≠ verbatim reconstruction" distinction the mandate | |
| 39 | +asks us to keep separate. | |
| 40 | + | |
| 41 | +## Findings and corrections | |
| 42 | +1. From-scratch sequence models cannot learn the inverse map from ≈ 3 900 pairs; denoising pretraining on unlabelled | |
| 43 | + originals (zero API cost) recovers most of the gap (chrF 32.9 → 63.6) — D17. | |
| 44 | +2. The factual-span loss as first implemented (normalised by the number of protected tokens) destroyed generalisation for every | |
| 45 | + model that used it, including M1 — D18 (fixed: normalised by all tokens). | |
| 46 | +3. The contrastive term treated the other rewrites of the same original as negatives — D19 (fixed: same-passage masking). | |
| 47 | +4. After D18 alone, M3 variants still generalise worse than token-only (v2 table); the style/adversarial branch is the remaining | |
| 48 | + suspect; v3 (D18 + D19) results pending. At pilot scale the dual-latent objective has **no demonstrated benefit**; this is reported | |
| 49 | + as a negative/undetermined finding, not hidden. | |
| 50 | +5. Seed variance is large at this scale (M1 3.69–4.31; M2 3.71–5.03): single-seed comparisons are not conclusive. | |
| 51 | +6. Incident I1: two cluster nodes froze under Medium-scale pretraining (MPS memory near the limit); mitigations recorded. | |
| 52 | + | |
| 53 | +## Decision | |
| 54 | +GO to Gate E **conditional on** (a) the Medium-stage rewrite budget (author approval; `configs/rewriting/medium.json`), and (b) | |
| 55 | +Medium-scale denoising pretraining completing on a node with memory caps. The confirmatory test scoring (pre-registration v1.0) | |
| 56 | +has **not** been run; M\* will be selected on validation among the Medium-stage candidates. | |
| 57 | ||