spb/wpai0002_uqo

1# HTR-Edit — a non-generative text-editing reconstruction model23Status: exploratory (validation only). Run `artifacts/runs/medium_edit_tiny` (M4M64a, 2026-11-10). Test-split4predictions exist (`artifacts/predictions/medium_test/outputs_htr-edit-tiny.jsonl`) but are **not scored** here; the5confirmatory scoring is run once by the coordinator.67## 1. Why a non-generative model89Author request (2026-11-10): a second model family built on the same principle (LLM rewrite → pre-2018 human10prose) that is **not generative**. HTR-Edit is a LaserTagger/Felix-style *tagger*:1112- **No free generation.** The output consists only of (i) source tokens that are kept, in their original order, and13  (ii) phrases from a closed vocabulary of 4 000 frequent inserted token sequences (1–4 SentencePiece tokens) built14  on the training split. It cannot write a word sequence that is neither in its input nor in that list.15- **Numbers and citations cannot be invented or lost.** Phrases containing a placeholder (`⟦C1⟧`, …) or a digit are16  excluded from the vocabulary, and at inference every protected source piece is forced to KEEP, together with the17  pieces glued to it (see §3). Measured on validation: number recall 1.000 and placeholders exact 1.000.18- **Untouched spans are preserved exactly** (byte-identical up to SentencePiece NFKC normalisation).19- **Deterministic, one forward pass**: argmax per position, no decoding loop, no beam, no sampling. 15 649 test20  passages are edited in 19 s on one M4 Max (MPS).2122## 2. Edit scheme (`src/htr/training/edits.py`)2324Source (`source_text`, placeholder form) and target (`target_text`) are tokenised with the 32 k SentencePiece model25(`artifacts/tokenizers/medium_v0`) and aligned with `difflib.SequenceMatcher(autojunk=False)` on token ids.26Labels per source of length *n*: `tags[i] ∈ {KEEP, DELETE}` (n labels) and `inserts[k]` = phrase id inserted before27source position *k* (k = 0..n, slot n = after the last token; 0 = `<none>`).2829| opcode | labels |30|---|---|31| equal | KEEP |32| delete | DELETE |33| insert | the inserted sequence must be one in-vocabulary phrase in slot i1 |34| replace src[i1:i2] → tgt[j1:j2] | DELETE the k = i2−i1 source tokens; segment the inserted sequence (minimum pieces, DP) into ≤ k phrases placed in slots i1, i1+1, … |3536Unrepresentable edits (OOV, more pieces than slots, protected token inside): a replace falls back to KEEP on its37source span (locally the identity), an insert to `<none>`. Realisation:38`out = ins[0] (src[0] if KEEP) ins[1] … (src[n-1] if KEEP) ins[n]`.3940**Phrase vocabulary** (train split only): every inserted sequence of length ≤ 4 counts once as a whole; each token of41a longer inserted sequence counts once as a unigram (so long rewrites can be segmented into frequent units). Top424 000 admissible sequences kept (+ `<none>`); persisted as `phrases.json` with the run and every checkpoint.43222 813 candidate sequences, 214 995 admissible; the 4 000 kept carry 70.3 % of the admissible count mass. The most44frequent phrases are function words and punctuation (`▁the`, `▁of`, `,`, `▁in`, `▁to`, `▁is`, `▁and`, …).4546### Coverage (`artifacts/runs/medium_edit_tiny/coverage.json`)4748| | train (72 698 pairs) | val (first 3 000 pairs) |49|---|---|---|50| exact coverage (labels reproduce the target ids) | **0.32 %** | **0.27 %** |51| pairs with ≥ 1 unrepresentable insertion | 99.68 % | 99.73 % |52| edit spans / unrepresentable | 1 960 672 / 645 128 (32.9 %) | 77 824 / 25 912 (33.3 %) |53| source tokens deleted by the gold labels | 19.4 % | 19.4 % |54| insertion slots used by the gold labels | 10.5 % | 10.6 % |55| pairs dropped (source > 510 tokens) | 100 | 0 |5657Oracle (gold labels realised, 3 000 val pairs): **chrF++ 81.57** vs 68.43 for the identity after the same tokenizer58round trip. So the scheme cannot reproduce full passages (Haiku rewrites change ~⅓ of edit spans in ways the594 000-phrase vocabulary cannot express), but a perfect tagger would still sit ~13 chrF above identity: the60headroom exists, the question is whether a tagger can find it.6162## 3. Model (`src/htr/models/tagger.py`)6364- Input `[BOS] src [EOS]`; encoder = the project's pre-LN `EncoderLayer` stack (RoPE self-attention, SwiGLU), tiny65  configuration (d 384, 6 heads, d_ff 1 536, 4 layers), parameter names `emb`, `enc`, `enc_ln` identical to66  `HTRModel`. **23.6 M parameters.**67- Initialised from the Medium denoiser encoder (`artifacts/runs/denoise_m1_tiny_medium_c/best`): **47/47 encoder68  tensors loaded**; new: `tag_head` (Linear d→2) and `ins_head` (MLP 2d→d→4 001 on `concat(h_k, h_{k+1})`, i.e. the69  states on both sides of slot k; slot 0 uses BOS, slot n uses the EOS state = the learned end-of-sequence slot).70- Loss = CE(tag) + weighted CE(insert): weight 1 on slots with a gold insertion, 0.1 on `<none>` slots (≈ 90 % of71  slots are `<none>`; without down-weighting the insertion head collapses to `<none>`).72- **Inference-time logit adjustment.** With class weights w_c the weighted-CE optimum is q_c ∝ w_c p_c; the editor73  adds log(w_edit / w_none) = log 10 = 2.303 to the `<none>` logit before the argmax to undo the weighting (stored as74  `none_logit_bias` in `config.json`; `--none-bias` overrides it). This is a fixed correction, not tuned on validation.75- **Protection plan** (`protection_plan`): protected pieces (placeholders, any digit) are kept; a protected piece with76  no leading `▁` (second piece of a multi-piece number, number after a bare `▁` or `(`) also keeps its left neighbour77  and blocks the insertion slot before it; a non-protected piece glued to the right of a protected one (`%`, `.5`,78  `mm`) is kept and its slot blocked. The unit test with a random model showed that keeping digit pieces alone is not79  enough (an insertion between `▁3` and `7` corrupts "37").8081Checkpoints: `model.safetensors` + `config.json` (`model_type: "htr_edit"`) + `phrases.json` + `tokenizer.model` +82`release_meta.json`. API: `HTREditor.from_pretrained(dir).edit(text, metadata)` / `edit_many(...)` returns a83`ReconstructionResult` (same fields as the generative pipeline; `metadata_used.edits` counts deletions/insertions).8485## 4. Training (`scripts/train_tagger.py`)8687AdamW (lr 3e-4, betas 0.9/0.98, wd 0.01), 500 warmup steps + cosine to 10 %, clip 1.0, token-budget buckets of 4 09688tokens (≤ 64 pairs), eval every 1 000 steps on 2 000 validation pairs, early stopping on validation loss (patience 4),89MPS cache release every 20 steps, `PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.7 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.5`, seed 1.9091```92PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.7 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.5 .venv/bin/python scripts/train_tagger.py \93  --pairs data/pairs/medium_v1_haiku/validated/pairs.jsonl --splits data/splits/medium_v1/splits.jsonl \94  --tokenizer artifacts/tokenizers/medium_v0 --init-from artifacts/runs/denoise_m1_tiny_medium_c/best \95  --run medium_edit_tiny --max-steps 15000 --eval-every 1000 --max-tokens 4096 --max-batch 64 --patience 4 --seed 1 --device mps96.venv/bin/python scripts/predict_tagger.py --pairs … --splits … --split val --limit 3000 \97  --checkpoint artifacts/runs/medium_edit_tiny/best --system htr-edit-tiny --out artifacts/predictions/medium_val98```99100Run (M4M64a, Apple M4 Max, torch 2.12.1): 72 598 training pairs, budget 15 000 steps, **early-stopped at 10 000**101(≈ 4 epochs, 39.4 M source tokens, 1 088 s); **best checkpoint at step 6 000**, validation loss 3.094.102103| at best step (2 000 val pairs) | value |104|---|---|105| tag accuracy (KEEP/DELETE) | 0.826 |106| insertion-slot accuracy (incl. `<none>`) | 0.892 |107| insertion recall (exact phrase on gold insertion slots) | 0.082 |108| deletion recall / precision | 0.354 / 0.589 |109| exact match of all labels / of the realised target | 0.000 / 0.000 |110111Validation loss reached its minimum at step 6 000 and rose afterwards while training loss kept falling (overfitting112of the insertion head).113114## 5. Results (validation, exploratory; 3 000 pairs, 284 document clusters, 2 000 bootstrap resamples)115116From `artifacts/metrics/medium_val_exploratory/report.json` (report rewritten with all systems, 2026-11-10).117118| system | chrF++ | edit sim | number recall | placeholders exact | W1 to originals ↓ | paired chrF vs B0 |119|---|---|---|---|---|---|---|120| B0 identity | 68.42 [67.60, 69.23] | 0.678 | 1.000 | 1.000 | 0.164 | — |121| **HTR-Edit tiny** (`htr-edit-tiny`) | **66.15 [65.27, 67.03]** | 0.671 [0.663, 0.679] | **1.000** | **1.000** | 0.157 | **−2.27 [−2.54, −2.03]**, p_Holm < 0.001 |122| M\* generative HTR-Tiny M1 s3 (`htr-medium_m1_tiny_mediumdenoiseinit_s3`) | 68.84 [68.03, 69.62] | 0.679 | 0.991 | 0.995 | 0.100 | +0.42 [+0.20, +0.66], p_Holm 0.003 |123124Other paired differences of HTR-Edit vs B0: edit similarity −0.0075 [−0.0096, −0.0055]; ROUGE-L **+0.018**125[+0.016, +0.019]; number recall and placeholders exactly 0 (identical to identity; the runner's p_Holm of 0.0 on a126zero-variance difference is an artefact, not a significant effect). Stylometric probe P(original) mean 0.51 (B0 0.42,127M\* 0.52). Predicted edits on validation: 11.6 % of source tokens deleted, 2.24 insertions per 100 source tokens,12863/3 000 outputs unchanged.129130**Reading.** The non-generative editor meets its design guarantees (perfect number and placeholder preservation,131determinism, speed) but is **worse than identity on chrF++ (−2.3)** and far from the generative M\* on the132stylometric distance (W1 0.157 vs 0.100). The cause is visible in the training metrics and outputs: the tag head133learns *where* the rewrite departs from the original (deletion precision 0.59) but the insertion head recovers the134replacement phrase only 8 % of the time, so most replacements degenerate into bare deletions ("Sharpe (1994) ⟦C1⟧135~~applies~~ the Treynor …", "~~Using~~ the same methodology"); outputs are 11 % shorter than the source in words while136originals are 4 % longer. ROUGE-L rises (longest-common-subsequence rewards deleting rewrite-only words) while chrF++137and edit similarity fall. The oracle (81.6 chrF) shows the limitation is the tagger's prediction, not only the scheme.138139## 6. Limitations and next steps140141- Coverage is ~0.3 % exact: the Haiku rewrites are lexical paraphrases, not local edits; a 4 000-phrase vocabulary142  of ≤ 4 tokens cannot express a third of the edit spans (they fall back to KEEP in training).143- Independent per-position argmax: deletions and their replacement insertions are not coupled. Candidate fixes, all144  untested here: decode a replace only when its insertion is predicted (joint DELETE+INSERT constraint), a145  Felix-style pointer/reordering + masked insertion model, a larger phrase vocabulary, or a tag/insert confidence146  threshold chosen on a separate tuning slice (not on the validation pairs used for model selection).147- Model selection on validation loss; the same validation split is used for the exploratory scoring (as for the148  generative models).149- The node mirror `~/wpai0002` is not a git checkout: the manifest's `git.commit` is `null`; the code that produced150  the run is the files listed in §2–4 (uncommitted at run time).151- Test predictions (15 649 pairs) are written but deliberately unscored.152