Gate F — Release candidate and paper
Date: 2026-11-11 · Decision: GO (paper v1.0 compiled with genuine figures, tables and citations; release
candidates load independently; the items the author must still decide are listed explicitly below and none of them
is hidden in the paper) · Evidence: paper/main.pdf (v1.0, 2026-11-11), paper/sections/htr2.tex,
artifacts/metrics/medium_test_htr2_exploratory/{report.json,summary.md}, artifacts/metrics/htr2_split_deltas/,
artifacts/metrics/pangram_test*/, artifacts/metrics/audit_*_val/, artifacts/releases/, docs/gates/GATE_E.md,
docs/DECISION_LOG.md D22, docs/INCIDENTS.md I3–I4, docs/reproducibility/COMPUTE_BUDGET.md.
What Gate F requires (mandate) and what exists
| Requirement | State | Evidence |
|---|---|---|
| Independent inference from a clean environment | done for HTR-Tiny 0.3.0 and HTR-Base 0.1.0 (SafeTensors + tokenizer + config + MANIFEST sha256 + LICENSE note load in python -I; htr CLI reconstructs; demo answers /health, /api/reconstruct); the fine-tuned backbones load through transformers from artifacts/releases/flan-t5-xl-htr and the platform's model directory |
scripts/export_release.py, src/htr/cli.py, demo/, artifacts/releases/ |
| Full model card | done for the six served models (HTR-Tiny 0.3.0, HTR-Base 0.1.0, flan-t5-large-HTR, flan-t5-xl-HTR, Qwen2.5-1.5B-HTR, Qwen2.5-3B-HTR): confirmatory and exploratory numbers, preservation warnings, Pangram block, licence note (non-commercial research use: training pairs include CC BY-NC sources) | https://htr.spboucher.ai/model-card, artifacts/releases/*/README.md, model_card.json |
| Audited release candidate | partial: licence matrix, memorisation audit (no overlap with training originals for Tiny, Base, flan-t5-large, Qwen 1.5B), retraction check at acquisition. Not done: the Anthropic-terms question (LICENSE_MATRIX §4) and the public upload itself (author decisions, below) | docs/data_governance/LICENSE_MATRIX.md, artifacts/metrics/audit_* |
| Compiled paper | done: paper/main.pdf v1.0, all \pending{} markers resolved, figures/tables generated from artefacts (scripts/build_figures.py, scripts/build_htr2_figures.py), 12 new verified references, no undefined references or citations |
paper/, paper/figures/FIGURE_PROVENANCE.md |
Final numbers (paper v1.0)
Confirmatory (Gate E, unchanged; test n = 15 649, 10 000 document-clustered resamples, Holm): M* = HTR-Tiny M1 seed 3 (33.5 M). H1 vs B0 +0.44 chrF++ [+0.30, +0.58] (detectable, below the +2.0 SESOI); H1 vs B1 +3.13 (supported); H2 vs B0 +0.0024 [+0.0011, +0.0036] (below SESOI); H2 vs B1 +0.115 (supported); H3 number recall −0.0087 [−0.0104, −0.0072] (non-inferiority not established); H4 W1 reduction 32.5 % [29.6, 37.6] (supported); H5 not supported (lexical gain transfers only to Opus rewrites; style gain holds on every set).
Exploratory, post hoc (D22; test n = 15 649, 2 000 resamples; val 3 000 / domain 2 493 / period 2 574, ± 1.96 SE):
| System | val | test [95 % CI] | domain | period | numbers / placeholders | Pangram Human |
|---|---|---|---|---|---|---|
| HTR-Tiny M* (confirmatory) | +0.42 | +0.44 [+0.30, +0.58] | −0.10 | −0.52 | 0.991 / 0.992 | 54/54 |
| HTR-Base 335 M, H100 (seed 2) | +0.83 (+0.67) | +0.69 [+0.56, +0.83] (+0.63) | +0.29 (+0.32) | −0.12 (−0.07) | 0.992 / 0.993 | 53/54 |
| HTR-Large 1.07 B, stage-2 denoiser (lr 1.5e-5 from stage 3) | −0.64 (−0.60) | −0.71 (−0.79) | −1.15 (−1.05) | −1.62 (−1.10) | 0.988 / 0.987 | — |
| HTR-Small v2 | −0.13 | −0.09 | — | — | 0.988 / 0.989 | — |
| HTR-Edit tagger | −2.27 | −2.20 | — | — | 1.000 / 0.999 | — |
| flan-T5-large 783 M | +2.04 | +1.79 [+1.62, +1.96] | +1.27 | +1.09 | 0.987 / 0.992 | 54/54 |
| flan-T5-xl 2.85 B | +2.56 | +2.39 [+2.20, +2.57] | +1.78 | +2.12 | 0.986 / 0.992 | 54/54 |
| Qwen2.5-1.5B SFT (seed 2) | +2.19 (+2.17) | +1.97 [+1.82, +2.10] (+1.97) | +1.78 (+1.75) | +1.30 (+1.28) | 0.998 / 0.998 | 48/54 |
| Qwen2.5-3B SFT | +2.42 | +2.27 [+2.13, +2.40] | +2.05 | +1.91 | 0.998 / 0.999 | 49/54 |
T1 decoding (Tiny, 1 000 val, K = 8): greedy +0.54 → MBR +0.66 → MBR + source anchor +0.70; best-of-9 oracle +1.74; single sample −1.23. T2 MRT (Tiny): 68.96 → 68.80–68.86, negative. T4 discriminator RoBERTa-large: val AUROC 0.994, acc 0.976 (reward variant implemented, not run). HTR-Edit perfect-edit oracle: 81.57 vs identity 68.43 (+13). Stylometric W1 to the originals (test; rewrite 0.154): flan-t5-xl 0.081, flan-t5-large 0.083, Qwen 3B 0.100, Tiny 0.104, Base 0.108, Qwen 1.5B 0.123 (final report 2026-11-11T01:31Z, 27 systems).
Reading: the pretrained English prior is the lever (+2.3–2.4 on test, transfers to held-out disciplines and pre-1900 books); the in-house architecture scales modestly (Tiny +0.44 → Base +0.69) and Large is unstable; decoding/MRT/discriminator add ≤ 0.2 or nothing; Qwen = lexical closeness + fact preservation, flan-T5 = stylometric distribution + 0 % detector flags. All of this is exploratory and so stated in the paper.
Spend: rewriting 43.20 USD · Pangram 57.17 USD · cloud GPU ≈ 158 USD (to confirm against the Lambda invoice) · total ≈ 258 USD.
What the author must still do (explicit; the paper says so where relevant)
- Confirm the SESOIs (H1 +2.0, H2 +0.02, H3 −0.01, H4 10 %, H5/H6 +1.0): they were pre-specified in-session before the freeze and never author-reviewed (paper §Threats, §RQ). If the author changes them, the verdicts are re-stated, not re-run.
Decide the weight releaseDecided 2026-11-11 (D23): the six served models are public on the Hugging Face Hub (spboucher04/htr-tiny,htr-base,flan-t5-large-htr,flan-t5-xl-htr,qwen2.5-1.5b-htr,qwen2.5-3b-htr; collection https://huggingface.co/collections/spboucher04/htr-human-text-reconstruction-wp-ai-0002-6af3ec2d756c4d246fc7c498), fine-tuned weights under CC BY-NC 4.0 (Qwen-3B also under the Qwen Research Licence), cards with the synthetic-output and no-detection clauses. The pairs stay unreleased. A UQO/legal check of the provider-terms question remains advisable.- Decide the default model of the public platform (currently Qwen2.5-1.5B-HTR): Qwen = best wording/fact preservation but 11 % Pangram "Mixed"; flan-T5-xl = best stylometric match and 0 % flags but T5 character coverage; the paper reports both without ranking.
- Independent blind rater study (E16): protocol and sheet generator exist; no rater recruited, no ethics application filed. Future work, stated in the paper (§Human Evaluation, §Limitations).
- Optional before submission: confirmatory replication of the backbone result under a fresh pre-registration;
re-score the flan-T5 models with
protect_unk_chars; author review of the 100-pair stratified sample that the mandate asks for (not done; stated in §Rewrite Protocol); confirm the Lambda invoice.
Issues carried forward
- Run manifests from the H100 instances carry no git commit (code synchronised by rsync); the runs are identified by directory name, args and checkpoint hashes.
- The flan-T5 scores pre-date the
<unk>-character fix (slightly under-stated). - The Holm adjustment in the exploratory report runs across systems within each metric, not over a pre-registered family (stated in the paper).