Gate C report, D17 (denoising pretraining), STATUS update
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
3 changed files +47 −2
modified
docs/DECISION_LOG.md
+4 −0
@@ -33,3 +33,7 @@ Each entry records the decision, its rationale and the evidence. Entries D1–D1 | |||||
| 33 | ## D-FREEZE — 2026-11-09 17:59 UTC | 33 | ## D-FREEZE — 2026-11-09 17:59 UTC | ||
| 34 | 34 | ||||
| 35 | PREREGISTRATION v1.0 frozen at commit `e74b71266d5c0e8c20f181a50df139f52c771d0f` (pre-specified in-session; SESOIs = v0.1 proposals, author confirmation pending). No confirmatory split had been scored with a model before this commit; only B0/B1 sanity checks on the smoke corpus. | 35 | PREREGISTRATION v1.0 frozen at commit `e74b71266d5c0e8c20f181a50df139f52c771d0f` (pre-specified in-session; SESOIs = v0.1 proposals, author confirmation pending). No confirmatory split had been scored with a model before this commit; only B0/B1 sanity checks on the smoke corpus. | ||
| 36 | + | ||||
| 37 | +## D17 — 2026-11-09 (after the exploratory validation of the M1 pilot) | ||||
| 38 | + | ||||
| 39 | +**Decision.** Add a denoising pretraining stage on unlabelled, rights-cleared original passages (train split only; BART-style span masking, word dropout, local shuffling, rule-based paraphrase) before fine-tuning on (rewrite → original) pairs, and block repeated 4-grams at decoding. **Why.** The from-scratch M1-tiny scored chrF 32.9 vs 71.2 for identity on the validation split with degenerate repetitions: 3 900 pairs (≈ 1.7 M target tokens) cannot train a 27 M sequence model from random weights. Pretraining uses only data we already hold (no API spend) and is standard practice; it is reported as part of the model recipe, and token-loss-only fine-tuning without pretraining remains the documented control. Exploratory, before any confirmatory scoring. | ||||
modified
docs/STATUS.md
+2 −2
@@ -5,8 +5,8 @@ | |||||
| 5 | |---|---|---| | 5 | |---|---|---| | ||
| 6 | | A — repository, governance, 100-paper smoke, quality audit | **GO** | `docs/gates/GATE_A.md` | | 6 | | A — repository, governance, 100-paper smoke, quality audit | **GO** | `docs/gates/GATE_A.md` | | ||
| 7 | | B — 500–1 000 validated Claude pairs, spend, human review | **GO** (974 passages, 5 383 validated pairs, 29.68 USD total spend) | `docs/gates/GATE_B.md` | | 7 | | B — 500–1 000 validated Claude pairs, spend, human review | **GO** (974 passages, 5 383 validated pairs, 29.68 USD total spend) | `docs/gates/GATE_B.md` | | ||
| 8 | −| C — leakage-safe splits, reproducible baselines, benchmark + statistical plan | in progress (splits done; B0/B1 implemented; B2 pending; pre-registration v0.1 drafted, v1.0 freeze pending) | `data/splits/pilot_pmc/`, `docs/PREREGISTRATION.md` | | 8 | +| C — leakage-safe splits, reproducible baselines, benchmark + statistical plan | **GO** (splits frozen; B0/B1 run on val; pre-registration v1.0 frozen at `e74b7126` + A1) | `docs/gates/GATE_C.md` | | ||
| 9 | −| D — M1/M2/M3 training, one-batch overfit, pilot results | in progress (overfit test passed on smoke data: CE 5.28 → 0.002 in 150 steps; `pilot_m1_tiny` training on the laptop, `pilot_m2_tiny` on M4M64a) | `artifacts/runs/` | | 9 | +| D — M1/M2/M3 training, one-batch overfit, pilot results | in progress — exploratory val: B0 chrF 71.2, B1 68.4, **M1-tiny 32.9** (from scratch, overfits after step 750, degenerate repetitions); M2 best val CE 3.72 vs M1 3.69; M3 full objective worse (val CE 5.9 at step 1000); M3 token-only, B2 T5-small, M1 seed 2 running; **denoising pretraining (D17) started on M3U96a** | `artifacts/metrics/pilot_val_exploratory/`, `artifacts/runs/` | | ||
| 10 | | E — larger training | not started (depends on D) | — | | 10 | | E — larger training | not started (depends on D) | — | | ||
| 11 | | F — release + paper | scaffolding only (release exporter, CLI, demo, model-card template, paper skeleton compiles: 5 pages of placeholders) | `src/htr/release`, `src/htr/cli.py`, `demo/`, `paper/` | | 11 | | F — release + paper | scaffolding only (release exporter, CLI, demo, model-card template, paper skeleton compiles: 5 pages of placeholders) | `src/htr/release`, `src/htr/cli.py`, `demo/`, `paper/` | | ||
| 12 | 12 | ||||
added
docs/gates/GATE_C.md
+41 −0
@@ -0,0 +1,41 @@ | |||||
| 1 | +# Gate C — Leakage-safe splits, reproducible baselines, initial benchmark and statistical plan | ||||
| 2 | + | ||||
| 3 | +**Date:** 2026-11-09 · **Decision: GO** · Evidence: `data/splits/pilot_pmc/splits_summary.json`, `docs/PREREGISTRATION.md` | ||||
| 4 | +(v1.0 frozen at commit `e74b7126`, amendment A1), `artifacts/metrics/pilot_val_exploratory/report.json`, `src/htr/evaluation/*`, | ||||
| 5 | +`tests/unit/test_splits_and_dedup.py`, `tests/unit/test_eval_audits.py`. | ||||
| 6 | + | ||||
| 7 | +## Splits | ||||
| 8 | +- Unit = document; leakage groups = union of near-duplicate cluster (MinHash Jaccard ≥ 0.8) and normalised first-author | ||||
| 9 | + group; deterministic hash of the group id → train 80 / val 10 / test 10; whole disciplines and periods carved out. | ||||
| 10 | +- Pilot (1 180 PMC docs): train 856 · val 98 · test 100 · heldout_domain 47 (chemistry, pre-fix label) · heldout_period 79 | ||||
| 11 | + (pub_year < 1950) · 0 group leaks (checked by `check_no_group_leak`). | ||||
| 12 | +- Assignments of rewritten documents are **frozen**: `make_splits.py --fixed` carries them into every later corpus build. | ||||
| 13 | + | ||||
| 14 | +## Baselines (implemented, run on the validation split — exploratory, pre-freeze scoring of val is allowed) | ||||
| 15 | +| System | chrF++ | edit sim | ROUGE-L | number recall | placeholders exact | W1 to originals | | ||||
| 16 | +|---|---|---|---|---|---|---| | ||||
| 17 | +| B0 identity (copy the rewrite) | 71.24 [69.69, 72.65] | 0.666 | 0.690 | 1.000 | 1.000 | 0.170 | | ||||
| 18 | +| B1 generic paraphrase (rule-based) | 68.42 [66.90, 69.99] | 0.561 | 0.612 | 1.000 | 1.000 | 0.281 | | ||||
| 19 | +| M1-tiny from scratch (27.4 M, 3 900 pairs, best val CE 3.69) | 32.86 [30.87, 34.47] | 0.287 | 0.241 | 0.571 | 0.633 | 1.699 | | ||||
| 20 | +| B2 T5-small fine-tune | running on M3U96b | | | | | | | ||||
| 21 | + | ||||
| 22 | +(n = 502 validation pairs; 95 % document-clustered bootstrap CIs, 2 000 resamples.) **Reading:** on this pilot the identity | ||||
| 23 | +baseline is strong (chrF 71) and the from-scratch tiny model is far below it with degenerate repetitions — the expected | ||||
| 24 | +failure of a 27 M-parameter sequence model trained from random initialisation on ≈ 1.7 M target tokens. This is a | ||||
| 25 | +genuine pilot finding, not a bug: the copy mechanism alone does not make identity easy to learn in 2 250 steps. | ||||
| 26 | + | ||||
| 27 | +## Benchmark and statistical plan | ||||
| 28 | +- Primary metrics, hypotheses H1–H6, SESOIs, Holm families, document-clustered paired bootstrap (10 000 resamples for | ||||
| 29 | + confirmatory runs), M\* selection on validation only, single confirmatory scoring: `docs/PREREGISTRATION.md` v1.0 | ||||
| 30 | + (pre-specified in-session; SESOIs await author confirmation) + A1 (common stylometric scale). | ||||
| 31 | +- Benchmark artefacts: pairs (placeholder form) + metadata + splits + validation records; release subset policy in | ||||
| 32 | + `docs/data_governance/DATA_CARD.md`. | ||||
| 33 | +- Evaluation code: lexical/factual metrics, stylometric W1 and exploratory probe, Holm, identifiability and memorisation | ||||
| 34 | + audits — all unit-tested. | ||||
| 35 | + | ||||
| 36 | +## Consequences for Gate D | ||||
| 37 | +1. Add a **denoising pretraining stage** on unlabelled original passages (zero API cost; `scripts/pretrain_denoise.py`, | ||||
| 38 | + `--init-from`), then fine-tune on pairs — decision D17. | ||||
| 39 | +2. Add n-gram repetition blocking at decoding (`no_repeat_ngram=4`) — affects all systems equally. | ||||
| 40 | +3. Treat B2 (pretrained T5) as the strongest comparator; the Medium-stage pair set (≥ 20 000 passages) is required for a | ||||
| 41 | + fair from-scratch comparison and needs budget approval. | ||||
| 42 | |||||