Gate C report, D17 (denoising pretraining), STATUS update
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
3 changed files +47 −2
modified
docs/DECISION_LOG.md
+4 −0
@@ -33,3 +33,7 @@ Each entry records the decision, its rationale and the evidence. Entries D1–D1 | ||
| 33 | 33 | ## D-FREEZE — 2026-11-09 17:59 UTC |
| 34 | 34 | |
| 35 | 35 | PREREGISTRATION v1.0 frozen at commit `e74b71266d5c0e8c20f181a50df139f52c771d0f` (pre-specified in-session; SESOIs = v0.1 proposals, author confirmation pending). No confirmatory split had been scored with a model before this commit; only B0/B1 sanity checks on the smoke corpus. |
| 36 | + | |
| 37 | +## D17 — 2026-11-09 (after the exploratory validation of the M1 pilot) | |
| 38 | + | |
| 39 | +**Decision.** Add a denoising pretraining stage on unlabelled, rights-cleared original passages (train split only; BART-style span masking, word dropout, local shuffling, rule-based paraphrase) before fine-tuning on (rewrite → original) pairs, and block repeated 4-grams at decoding. **Why.** The from-scratch M1-tiny scored chrF 32.9 vs 71.2 for identity on the validation split with degenerate repetitions: 3 900 pairs (≈ 1.7 M target tokens) cannot train a 27 M sequence model from random weights. Pretraining uses only data we already hold (no API spend) and is standard practice; it is reported as part of the model recipe, and token-loss-only fine-tuning without pretraining remains the documented control. Exploratory, before any confirmatory scoring. | |
modified
docs/STATUS.md
+2 −2
@@ -5,8 +5,8 @@ | ||
| 5 | 5 | |---|---|---| |
| 6 | 6 | | A — repository, governance, 100-paper smoke, quality audit | **GO** | `docs/gates/GATE_A.md` | |
| 7 | 7 | | B — 500–1 000 validated Claude pairs, spend, human review | **GO** (974 passages, 5 383 validated pairs, 29.68 USD total spend) | `docs/gates/GATE_B.md` | |
| 8 | −| C — leakage-safe splits, reproducible baselines, benchmark + statistical plan | in progress (splits done; B0/B1 implemented; B2 pending; pre-registration v0.1 drafted, v1.0 freeze pending) | `data/splits/pilot_pmc/`, `docs/PREREGISTRATION.md` | | |
| 9 | −| D — M1/M2/M3 training, one-batch overfit, pilot results | in progress (overfit test passed on smoke data: CE 5.28 → 0.002 in 150 steps; `pilot_m1_tiny` training on the laptop, `pilot_m2_tiny` on M4M64a) | `artifacts/runs/` | | |
| 8 | +| C — leakage-safe splits, reproducible baselines, benchmark + statistical plan | **GO** (splits frozen; B0/B1 run on val; pre-registration v1.0 frozen at `e74b7126` + A1) | `docs/gates/GATE_C.md` | | |
| 9 | +| D — M1/M2/M3 training, one-batch overfit, pilot results | in progress — exploratory val: B0 chrF 71.2, B1 68.4, **M1-tiny 32.9** (from scratch, overfits after step 750, degenerate repetitions); M2 best val CE 3.72 vs M1 3.69; M3 full objective worse (val CE 5.9 at step 1000); M3 token-only, B2 T5-small, M1 seed 2 running; **denoising pretraining (D17) started on M3U96a** | `artifacts/metrics/pilot_val_exploratory/`, `artifacts/runs/` | | |
| 10 | 10 | | E — larger training | not started (depends on D) | — | |
| 11 | 11 | | F — release + paper | scaffolding only (release exporter, CLI, demo, model-card template, paper skeleton compiles: 5 pages of placeholders) | `src/htr/release`, `src/htr/cli.py`, `demo/`, `paper/` | |
| 12 | 12 | |
added
docs/gates/GATE_C.md
+41 −0
@@ -0,0 +1,41 @@ | ||
| 1 | +# Gate C — Leakage-safe splits, reproducible baselines, initial benchmark and statistical plan | |
| 2 | + | |
| 3 | +**Date:** 2026-11-09 · **Decision: GO** · Evidence: `data/splits/pilot_pmc/splits_summary.json`, `docs/PREREGISTRATION.md` | |
| 4 | +(v1.0 frozen at commit `e74b7126`, amendment A1), `artifacts/metrics/pilot_val_exploratory/report.json`, `src/htr/evaluation/*`, | |
| 5 | +`tests/unit/test_splits_and_dedup.py`, `tests/unit/test_eval_audits.py`. | |
| 6 | + | |
| 7 | +## Splits | |
| 8 | +- Unit = document; leakage groups = union of near-duplicate cluster (MinHash Jaccard ≥ 0.8) and normalised first-author | |
| 9 | + group; deterministic hash of the group id → train 80 / val 10 / test 10; whole disciplines and periods carved out. | |
| 10 | +- Pilot (1 180 PMC docs): train 856 · val 98 · test 100 · heldout_domain 47 (chemistry, pre-fix label) · heldout_period 79 | |
| 11 | + (pub_year < 1950) · 0 group leaks (checked by `check_no_group_leak`). | |
| 12 | +- Assignments of rewritten documents are **frozen**: `make_splits.py --fixed` carries them into every later corpus build. | |
| 13 | + | |
| 14 | +## Baselines (implemented, run on the validation split — exploratory, pre-freeze scoring of val is allowed) | |
| 15 | +| System | chrF++ | edit sim | ROUGE-L | number recall | placeholders exact | W1 to originals | | |
| 16 | +|---|---|---|---|---|---|---| | |
| 17 | +| B0 identity (copy the rewrite) | 71.24 [69.69, 72.65] | 0.666 | 0.690 | 1.000 | 1.000 | 0.170 | | |
| 18 | +| B1 generic paraphrase (rule-based) | 68.42 [66.90, 69.99] | 0.561 | 0.612 | 1.000 | 1.000 | 0.281 | | |
| 19 | +| M1-tiny from scratch (27.4 M, 3 900 pairs, best val CE 3.69) | 32.86 [30.87, 34.47] | 0.287 | 0.241 | 0.571 | 0.633 | 1.699 | | |
| 20 | +| B2 T5-small fine-tune | running on M3U96b | | | | | | | |
| 21 | + | |
| 22 | +(n = 502 validation pairs; 95 % document-clustered bootstrap CIs, 2 000 resamples.) **Reading:** on this pilot the identity | |
| 23 | +baseline is strong (chrF 71) and the from-scratch tiny model is far below it with degenerate repetitions — the expected | |
| 24 | +failure of a 27 M-parameter sequence model trained from random initialisation on ≈ 1.7 M target tokens. This is a | |
| 25 | +genuine pilot finding, not a bug: the copy mechanism alone does not make identity easy to learn in 2 250 steps. | |
| 26 | + | |
| 27 | +## Benchmark and statistical plan | |
| 28 | +- Primary metrics, hypotheses H1–H6, SESOIs, Holm families, document-clustered paired bootstrap (10 000 resamples for | |
| 29 | + confirmatory runs), M\* selection on validation only, single confirmatory scoring: `docs/PREREGISTRATION.md` v1.0 | |
| 30 | + (pre-specified in-session; SESOIs await author confirmation) + A1 (common stylometric scale). | |
| 31 | +- Benchmark artefacts: pairs (placeholder form) + metadata + splits + validation records; release subset policy in | |
| 32 | + `docs/data_governance/DATA_CARD.md`. | |
| 33 | +- Evaluation code: lexical/factual metrics, stylometric W1 and exploratory probe, Holm, identifiability and memorisation | |
| 34 | + audits — all unit-tested. | |
| 35 | + | |
| 36 | +## Consequences for Gate D | |
| 37 | +1. Add a **denoising pretraining stage** on unlabelled original passages (zero API cost; `scripts/pretrain_denoise.py`, | |
| 38 | + `--init-from`), then fine-tune on pairs — decision D17. | |
| 39 | +2. Add n-gram repetition blocking at decoding (`no_repeat_ngram=4`) — affects all systems equally. | |
| 40 | +3. Treat B2 (pretrained T5) as the strongest comparator; the Medium-stage pair set (≥ 20 000 passages) is required for a | |
| 41 | + fair from-scratch comparison and needs budget approval. | |
| 42 | ||