spb/wpai0002_uqo

1# Gate F — Release candidate and paper23**Date:** 2026-11-11 · **Decision: GO** (paper v1.0 compiled with genuine figures, tables and citations; release4candidates load independently; the items the author must still decide are listed explicitly below and none of them5is hidden in the paper) · Evidence: `paper/main.pdf` (v1.0, 2026-11-11), `paper/sections/htr2.tex`,6`artifacts/metrics/medium_test_htr2_exploratory/{report.json,summary.md}`, `artifacts/metrics/htr2_split_deltas/`,7`artifacts/metrics/pangram_test*/`, `artifacts/metrics/audit_*_val/`, `artifacts/releases/`, `docs/gates/GATE_E.md`,8`docs/DECISION_LOG.md` D22, `docs/INCIDENTS.md` I3–I4, `docs/reproducibility/COMPUTE_BUDGET.md`.910## What Gate F requires (mandate) and what exists1112| Requirement | State | Evidence |13|---|---|---|14| Independent inference from a clean environment | **done** for HTR-Tiny 0.3.0 and HTR-Base 0.1.0 (SafeTensors + tokenizer + config + MANIFEST sha256 + LICENSE note load in `python -I`; `htr` CLI reconstructs; demo answers `/health`, `/api/reconstruct`); the fine-tuned backbones load through `transformers` from `artifacts/releases/flan-t5-xl-htr` and the platform's model directory | `scripts/export_release.py`, `src/htr/cli.py`, `demo/`, `artifacts/releases/` |15| Full model card | **done** for the six served models (HTR-Tiny 0.3.0, HTR-Base 0.1.0, flan-t5-large-HTR, flan-t5-xl-HTR, Qwen2.5-1.5B-HTR, Qwen2.5-3B-HTR): confirmatory and exploratory numbers, preservation warnings, Pangram block, licence note (non-commercial research use: training pairs include CC BY-NC sources) | https://htr.spboucher.ai/model-card, `artifacts/releases/*/README.md`, `model_card.json` |16| Audited release candidate | **partial**: licence matrix, memorisation audit (no overlap with training originals for Tiny, Base, flan-t5-large, Qwen 1.5B), retraction check at acquisition. **Not done**: the Anthropic-terms question (LICENSE_MATRIX §4) and the public upload itself (author decisions, below) | `docs/data_governance/LICENSE_MATRIX.md`, `artifacts/metrics/audit_*` |17| Compiled paper | **done**: `paper/main.pdf` v1.0, all `\pending{}` markers resolved, figures/tables generated from artefacts (`scripts/build_figures.py`, `scripts/build_htr2_figures.py`), 12 new verified references, no undefined references or citations | `paper/`, `paper/figures/FIGURE_PROVENANCE.md` |1819## Final numbers (paper v1.0)2021Confirmatory (Gate E, unchanged; test n = 15 649, 10 000 document-clustered resamples, Holm): M* = HTR-Tiny M1 seed 322(33.5 M). H1 vs B0 **+0.44 chrF++ [+0.30, +0.58]** (detectable, below the +2.0 SESOI); H1 vs B1 +3.13 (supported);23H2 vs B0 +0.0024 [+0.0011, +0.0036] (below SESOI); H2 vs B1 +0.115 (supported); H3 number recall −0.008724[−0.0104, −0.0072] (non-inferiority not established); H4 W1 reduction **32.5 % [29.6, 37.6]** (supported); H5 not25supported (lexical gain transfers only to Opus rewrites; style gain holds on every set).2627Exploratory, post hoc (D22; test n = 15 649, 2 000 resamples; val 3 000 / domain 2 493 / period 2 574, ± 1.96 SE):2829| System | val | test [95 % CI] | domain | period | numbers / placeholders | Pangram Human |30|---|---|---|---|---|---|---|31| HTR-Tiny M* (confirmatory) | +0.42 | +0.44 [+0.30, +0.58] | −0.10 | −0.52 | 0.991 / 0.992 | 54/54 |32| HTR-Base 335 M, H100 (seed 2) | +0.83 (+0.67) | +0.69 [+0.56, +0.83] (+0.63) | +0.29 (+0.32) | −0.12 (−0.07) | 0.992 / 0.993 | 53/54 |33| HTR-Large 1.07 B, stage-2 denoiser (lr 1.5e-5 from stage 3) | −0.64 (−0.60) | −0.71 (−0.79) | −1.15 (−1.05) | −1.62 (−1.10) | 0.988 / 0.987 | — |34| HTR-Small v2 | −0.13 | −0.09 | — | — | 0.988 / 0.989 | — |35| HTR-Edit tagger | −2.27 | −2.20 | — | — | 1.000 / 0.999 | — |36| flan-T5-large 783 M | +2.04 | +1.79 [+1.62, +1.96] | +1.27 | +1.09 | 0.987 / 0.992 | 54/54 |37| **flan-T5-xl 2.85 B** | +2.56 | **+2.39** [+2.20, +2.57] | +1.78 | **+2.12** | 0.986 / 0.992 | 54/54 |38| Qwen2.5-1.5B SFT (seed 2) | +2.19 (+2.17) | +1.97 [+1.82, +2.10] (+1.97) | +1.78 (+1.75) | +1.30 (+1.28) | 0.998 / 0.998 | 48/54 |39| **Qwen2.5-3B SFT** | +2.42 | **+2.27** [+2.13, +2.40] | **+2.05** | +1.91 | **0.998 / 0.999** | 49/54 |4041T1 decoding (Tiny, 1 000 val, K = 8): greedy +0.54 → MBR +0.66 → MBR + source anchor +0.70; best-of-9 oracle +1.74;42single sample −1.23. T2 MRT (Tiny): 68.96 → 68.80–68.86, negative. T4 discriminator RoBERTa-large: val AUROC 0.994,43acc 0.976 (reward variant implemented, not run). HTR-Edit perfect-edit oracle: 81.57 vs identity 68.43 (+13).44Stylometric W1 to the originals (test; rewrite 0.154): flan-t5-xl 0.081, flan-t5-large 0.083, Qwen 3B 0.100, Tiny 0.104, Base 0.108, Qwen 1.5B 0.123 (final report 2026-11-11T01:31Z, 27 systems).4546Reading: the pretrained English prior is the lever (+2.3–2.4 on test, transfers to held-out disciplines and47pre-1900 books); the in-house architecture scales modestly (Tiny +0.44 → Base +0.69) and Large is unstable;48decoding/MRT/discriminator add ≤ 0.2 or nothing; Qwen = lexical closeness + fact preservation, flan-T5 =49stylometric distribution + 0 % detector flags. All of this is exploratory and so stated in the paper.5051Spend: rewriting 43.20 USD · Pangram 57.17 USD · cloud GPU ≈ 158 USD (to confirm against the Lambda invoice) ·52total ≈ 258 USD.5354## What the author must still do (explicit; the paper says so where relevant)55561. **Confirm the SESOIs** (H1 +2.0, H2 +0.02, H3 −0.01, H4 10 %, H5/H6 +1.0): they were pre-specified in-session57   before the freeze and never author-reviewed (paper §Threats, §RQ). If the author changes them, the verdicts are58   re-stated, not re-run.592. ~~Decide the weight release~~ **Decided 2026-11-11 (D23): the six served models are public on the Hugging Face Hub**60   (`spboucher04/htr-tiny`, `htr-base`, `flan-t5-large-htr`, `flan-t5-xl-htr`, `qwen2.5-1.5b-htr`, `qwen2.5-3b-htr`;61   collection https://huggingface.co/collections/spboucher04/htr-human-text-reconstruction-wp-ai-0002-6af3ec2d756c4d246fc7c498),62   fine-tuned weights under CC BY-NC 4.0 (Qwen-3B also under the Qwen Research Licence), cards with the synthetic-output and63   no-detection clauses. The pairs stay unreleased. A UQO/legal check of the provider-terms question remains advisable.643. **Decide the default model of the public platform** (currently Qwen2.5-1.5B-HTR): Qwen = best wording/fact65   preservation but 11 % Pangram "Mixed"; flan-T5-xl = best stylometric match and 0 % flags but T5 character66   coverage; the paper reports both without ranking.674. **Independent blind rater study** (E16): protocol and sheet generator exist; no rater recruited, no ethics68   application filed. Future work, stated in the paper (§Human Evaluation, §Limitations).695. Optional before submission: confirmatory replication of the backbone result under a fresh pre-registration;70   re-score the flan-T5 models with `protect_unk_chars`; author review of the 100-pair stratified sample that the71   mandate asks for (not done; stated in §Rewrite Protocol); confirm the Lambda invoice.7273## Issues carried forward7475- Run manifests from the H100 instances carry no git commit (code synchronised by rsync); the runs are identified by76  directory name, args and checkpoint hashes.77- The flan-T5 scores pre-date the `<unk>`-character fix (slightly under-stated).78- The Holm adjustment in the exploratory report runs across systems within each metric, not over a pre-registered79  family (stated in the paper).80