# HTR-Edit — a non-generative text-editing reconstruction model

Status: exploratory (validation only). Run artifacts/runs/medium_edit_tiny (M4M64a, 2026-11-10). Test-split predictions exist (artifacts/predictions/medium_test/outputs_htr-edit-tiny.jsonl) but are not scored here; the confirmatory scoring is run once by the coordinator.

# 1. Why a non-generative model

Author request (2026-11-10): a second model family built on the same principle (LLM rewrite → pre-2018 human prose) that is not generative. HTR-Edit is a LaserTagger/Felix-style tagger:

• No free generation. The output consists only of (i) source tokens that are kept, in their original order, and (ii) phrases from a closed vocabulary of 4 000 frequent inserted token sequences (1–4 SentencePiece tokens) built on the training split. It cannot write a word sequence that is neither in its input nor in that list.
• Numbers and citations cannot be invented or lost. Phrases containing a placeholder (⟦C1⟧, …) or a digit are excluded from the vocabulary, and at inference every protected source piece is forced to KEEP, together with the pieces glued to it (see §3). Measured on validation: number recall 1.000 and placeholders exact 1.000.
• Untouched spans are preserved exactly (byte-identical up to SentencePiece NFKC normalisation).
• Deterministic, one forward pass: argmax per position, no decoding loop, no beam, no sampling. 15 649 test passages are edited in 19 s on one M4 Max (MPS).

# 2. Edit scheme (src/htr/training/edits.py)

Source (source_text, placeholder form) and target (target_text) are tokenised with the 32 k SentencePiece model (artifacts/tokenizers/medium_v0) and aligned with difflib.SequenceMatcher(autojunk=False) on token ids. Labels per source of length n: tags[i] ∈ {KEEP, DELETE} (n labels) and inserts[k] = phrase id inserted before source position k (k = 0..n, slot n = after the last token; 0 = <none>).

Unrepresentable edits (OOV, more pieces than slots, protected token inside): a replace falls back to KEEP on its source span (locally the identity), an insert to <none>. Realisation: out = ins[0] (src[0] if KEEP) ins[1] … (src[n-1] if KEEP) ins[n].

Phrase vocabulary (train split only): every inserted sequence of length ≤ 4 counts once as a whole; each token of a longer inserted sequence counts once as a unigram (so long rewrites can be segmented into frequent units). Top 4 000 admissible sequences kept (+ <none>); persisted as phrases.json with the run and every checkpoint. 222 813 candidate sequences, 214 995 admissible; the 4 000 kept carry 70.3 % of the admissible count mass. The most frequent phrases are function words and punctuation (▁the, ▁of, ,, ▁in, ▁to, ▁is, ▁and, …).

# Coverage (artifacts/runs/medium_edit_tiny/coverage.json)

Oracle (gold labels realised, 3 000 val pairs): chrF++ 81.57 vs 68.43 for the identity after the same tokenizer round trip. So the scheme cannot reproduce full passages (Haiku rewrites change ~⅓ of edit spans in ways the 4 000-phrase vocabulary cannot express), but a perfect tagger would still sit ~13 chrF above identity: the headroom exists, the question is whether a tagger can find it.

# 3. Model (src/htr/models/tagger.py)

• Input [BOS] src [EOS]; encoder = the project's pre-LN EncoderLayer stack (RoPE self-attention, SwiGLU), tiny configuration (d 384, 6 heads, d_ff 1 536, 4 layers), parameter names emb, enc, enc_ln identical to HTRModel. 23.6 M parameters.
• Initialised from the Medium denoiser encoder (artifacts/runs/denoise_m1_tiny_medium_c/best): 47/47 encoder tensors loaded; new: tag_head (Linear d→2) and ins_head (MLP 2d→d→4 001 on concat(h_k, h_{k+1}), i.e. the states on both sides of slot k; slot 0 uses BOS, slot n uses the EOS state = the learned end-of-sequence slot).
• Loss = CE(tag) + weighted CE(insert): weight 1 on slots with a gold insertion, 0.1 on <none> slots (≈ 90 % of slots are <none>; without down-weighting the insertion head collapses to <none>).
• Inference-time logit adjustment. With class weights w_c the weighted-CE optimum is q_c ∝ w_c p_c; the editor adds log(w_edit / w_none) = log 10 = 2.303 to the <none> logit before the argmax to undo the weighting (stored as none_logit_bias in config.json; --none-bias overrides it). This is a fixed correction, not tuned on validation.
• Protection plan (protection_plan): protected pieces (placeholders, any digit) are kept; a protected piece with no leading ▁ (second piece of a multi-piece number, number after a bare ▁ or () also keeps its left neighbour and blocks the insertion slot before it; a non-protected piece glued to the right of a protected one (%, .5, mm) is kept and its slot blocked. The unit test with a random model showed that keeping digit pieces alone is not enough (an insertion between ▁3 and 7 corrupts "37").

Checkpoints: model.safetensors + config.json (model_type: "htr_edit") + phrases.json + tokenizer.model + release_meta.json. API: HTREditor.from_pretrained(dir).edit(text, metadata) / edit_many(...) returns a ReconstructionResult (same fields as the generative pipeline; metadata_used.edits counts deletions/insertions).

# 4. Training (scripts/train_tagger.py)

AdamW (lr 3e-4, betas 0.9/0.98, wd 0.01), 500 warmup steps + cosine to 10 %, clip 1.0, token-budget buckets of 4 096 tokens (≤ 64 pairs), eval every 1 000 steps on 2 000 validation pairs, early stopping on validation loss (patience 4), MPS cache release every 20 steps, PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.7 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.5, seed 1.

PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.7 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.5 .venv/bin/python scripts/train_tagger.py \
  --pairs data/pairs/medium_v1_haiku/validated/pairs.jsonl --splits data/splits/medium_v1/splits.jsonl \
  --tokenizer artifacts/tokenizers/medium_v0 --init-from artifacts/runs/denoise_m1_tiny_medium_c/best \
  --run medium_edit_tiny --max-steps 15000 --eval-every 1000 --max-tokens 4096 --max-batch 64 --patience 4 --seed 1 --device mps
.venv/bin/python scripts/predict_tagger.py --pairs … --splits … --split val --limit 3000 \
  --checkpoint artifacts/runs/medium_edit_tiny/best --system htr-edit-tiny --out artifacts/predictions/medium_val

Run (M4M64a, Apple M4 Max, torch 2.12.1): 72 598 training pairs, budget 15 000 steps, early-stopped at 10 000 (≈ 4 epochs, 39.4 M source tokens, 1 088 s); best checkpoint at step 6 000, validation loss 3.094.

Validation loss reached its minimum at step 6 000 and rose afterwards while training loss kept falling (overfitting of the insertion head).

# 5. Results (validation, exploratory; 3 000 pairs, 284 document clusters, 2 000 bootstrap resamples)

From artifacts/metrics/medium_val_exploratory/report.json (report rewritten with all systems, 2026-11-10).

Other paired differences of HTR-Edit vs B0: edit similarity −0.0075 [−0.0096, −0.0055]; ROUGE-L +0.018 [+0.016, +0.019]; number recall and placeholders exactly 0 (identical to identity; the runner's p_Holm of 0.0 on a zero-variance difference is an artefact, not a significant effect). Stylometric probe P(original) mean 0.51 (B0 0.42, M* 0.52). Predicted edits on validation: 11.6 % of source tokens deleted, 2.24 insertions per 100 source tokens, 63/3 000 outputs unchanged.

Reading. The non-generative editor meets its design guarantees (perfect number and placeholder preservation, determinism, speed) but is worse than identity on chrF++ (−2.3) and far from the generative M* on the stylometric distance (W1 0.157 vs 0.100). The cause is visible in the training metrics and outputs: the tag head learns where the rewrite departs from the original (deletion precision 0.59) but the insertion head recovers the replacement phrase only 8 % of the time, so most replacements degenerate into bare deletions ("Sharpe (1994) ⟦C1⟧ applies the Treynor …", "Using the same methodology"); outputs are 11 % shorter than the source in words while originals are 4 % longer. ROUGE-L rises (longest-common-subsequence rewards deleting rewrite-only words) while chrF++ and edit similarity fall. The oracle (81.6 chrF) shows the limitation is the tagger's prediction, not only the scheme.

# 6. Limitations and next steps

• Coverage is ~0.3 % exact: the Haiku rewrites are lexical paraphrases, not local edits; a 4 000-phrase vocabulary of ≤ 4 tokens cannot express a third of the edit spans (they fall back to KEEP in training).
• Independent per-position argmax: deletions and their replacement insertions are not coupled. Candidate fixes, all untested here: decode a replace only when its insertion is predicted (joint DELETE+INSERT constraint), a Felix-style pointer/reordering + masked insertion model, a larger phrase vocabulary, or a tag/insert confidence threshold chosen on a separate tuning slice (not on the validation pairs used for model selection).
• Model selection on validation loss; the same validation split is used for the exploratory scoring (as for the generative models).
• The node mirror ~/wpai0002 is not a git checkout: the manifest's git.commit is null; the code that produced the run is the files listed in §2–4 (uncommitted at run time).
• Test predictions (15 649 pairs) are written but deliberately unscored.