spb/wpai0002_uqo

1# WP-AI-0002 — Research plan23*Human Text Reconstruction: Learning the Inverse of LLM-Induced Transformations in Scientific Writing*4Version 0.1 · 2026-11-09 · binds to `docs/MANDATE.md` (phases 0–8, gates A–F, RQ1–RQ6).5Every number below comes from a file in the repository. A number that has not been measured yet is written6"pending (run id …)".78## 1. Mission and narrowed contribution910**Mission.** Build a reproducible research programme and a model family that is usable after release. The11programme asks whether a Transformer can reconstruct characteristics of pre-2018 human-authored scientific prose12from LLM-rewritten versions of the same passages. Two deliverables are mandatory: a paper and an independently13loadable model.1415**What is not new.** The direct antecedents are Rivera Soto et al.:1617- Rivera Soto et al. (2024, `riverasoto2024paraphrase`) introduced *paraphrase inversion*. They trained18  paraphrase→original models on synthetic pairs and showed that these models generalise to paraphrasers unseen19  in training.20- Rivera Soto et al. (2026, `riverasoto2026unsupervised`) trained a style encoder that reconstructs human text21  from its machine paraphrase while a semantic encoder is frozen. This is architecturally close to our M3.2223WP-AI-0002 therefore **does not claim to be the first to invert LLM paraphrases** (`docs/RESEARCH_GAP.md` §5).2425**What the contribution is.** Our claim is restricted to four points:26271. **Reconstruction fidelity is the object of study.** Prior work used inversion as a means to detection.28   We measure fidelity against identity (B0) and generic-paraphrase (B1) baselines.292. **The originals are rights-verified, pre-2018 scientific prose.** Each passage has controlled rewrite30   variants A–E, which allows paired, per-passage measurement for RQ1.313. **Factual integrity, non-identifiability and memorisation are evaluated.** This covers numbers,32   citations, equations, negation and hedging, plus a test of how many distinct originals a rewrite is33   compatible with.344. **Results are broken down by discipline, era and section role, and generalisation is tested.** Held-out35   prompts and held-out models are a primary research question (RQ4), not a robustness footnote.3637Whether Rivera Soto et al. stratify by discipline or era is marked *to check in full text*.3839## 2. Research questions → hypotheses → metrics → tests4041Confirmatory hypotheses H1–H6 are pre-registered in `docs/PREREGISTRATION.md`. The others are exploratory.4243| RQ | Hypothesis (direction) | Primary metric | Test |44|---|---|---|---|45| RQ1 feature shift | Rewrites move stylometric features away from the originals (paired, per variant) | per-feature paired Δ (rewrite − original) on 18 features of `stylometry.FEATURES`; W1 (rewrite, original) | document-clustered paired bootstrap per feature, Holm over 18 features (exploratory-descriptive) |46| RQ2 recovery | **H1** best HTR > B0 and > B1 on chrF++; **H2** same on edit similarity; **H3** number recall is non-inferior to B0; **H4** W1(recon, orig) < W1(rewrite, orig) | chrF++ (sacrebleu `CHRF(word_order=2)`), normalised Levenshtein similarity, number recall, mean standardised W1 | clustered paired bootstrap; Holm across H1–H4 |47| RQ3 heterogeneity | Gains differ by discipline, era, section role and variant | same metrics, by stratum (`runner._agg`) | stratum CIs; interaction reported descriptively |48| RQ4 generalisation | **H5** the gain over B0 persists for held-out prompts H1/H2 and held-out models (Sonnet 5.5, Haiku 5.5) | chrF++ gain vs B0 on the held-out sets | clustered paired bootstrap, one-sided |49| RQ5 scale / conditioning | **H6** M3 > M1 at matched size and steps; gains grow with data and capacity | chrF++ on validation (development) and test (confirmatory, final configuration only) | paired bootstrap; scaling curves with 3 seeds |50| RQ6 losses / risks | Report factual-distortion rates, memorisation and non-identifiability | placeholder exactness, number precision and recall, negation and hedge deltas; nearest-neighbour overlap; ambiguity rate | descriptive with CIs |5152Lower lexical overlap is **never** read as failure to recover style (mandate §6). Style results are53distributional (W1). The original-vs-rewrite probe is exploratory only.5455## 3. Data pipeline5657```mermaid58flowchart LR59  subgraph Sources[Official channels, metadata first]60    P[PMC OA · AWS bucket pmc-oa-opendata<br/>JSON license_code + JATS]61    X[arXiv OAI-PMH arXivRaw + e-prints<br/>CC licence, v1 date · in progress]62    C[Crossref CC BY econ/finance journals<br/>in progress]63    B[Books: Gutenberg PD · NCBI Bookshelf BITS ·<br/>OpenStax-IA only if dated]64  end65  Sources --> S[raw file + meta.json stub<br/>licence evidence, sha256, provider md5]66  S --> PA[parse: JATS/BITS · LaTeX · txt/PDF<br/>pub-date verified < 2018-01-01]67  PA --> L{licence policy<br/>htr.licenses.policy}68  L -- tier C/X --> M[metadata only]69  L -- training set --> SEG[paragraphs → passages<br/>60–400 words, ≥2 sentences,<br/>en ≥ 0.9, quality ≥ 0.6]70  SEG --> DD[MinHash dedup<br/>5-word shingles, 128 perm, J ≥ 0.8]71  DD --> SP[splits BEFORE rewriting<br/>cluster ∪ author group (∪ venue);<br/>held-out discipline / period]72  SP --> RW[protect: citations/math/URLs → ⟦C/M/U n⟧<br/>Claude rewrite A–E (+H1/H2 held out)]73  RW --> V[validate val-1.0<br/>pass → pairs · fail → quarantine]74  V --> T[tokenizer (train split only) → M1/M2/M3]75  T --> E[evaluation · release]76```7778**Measured so far.**7980| Run | Measured result | Evidence |81|---|---|---|82| Smoke corpus (`smoke100`) | 100 PMC docs, 2 964 paragraphs, 2 013 eligible passages | `data/parsed/smoke/build_summary.json` |83| PMC pilot (`pilot_pmc`) | 1 200 fetched, 1 180 kept, 34 695 paragraphs, 23 658 eligible passages. Exclusions: 5 not pre-2018, 3 licence, 12 exact duplicates | `data/parsed/pilot_pmc/build_summary.json` |84| Gutenberg books (`books_smoke`) | 11 kept, 2 039 eligible passages | `data/parsed/books_smoke/` |85| NCBI Bookshelf (`books_bookshelf_trial`) | 17 chapters, 299 eligible passages | `data/parsed/books_bookshelf_trial/` |86| Economics and finance | Only 4 PMC documents labelled economics; finance not yet reached | `pilot_pmc` build summary |8788The economics and finance strata depend on the arXiv q-fin/econ adapter and the Crossref adapter, both in89progress.9091**Scale target (author, 2026-11-09).** Medium stage: ≥ 10 000 documents and ≥ 200 000 eligible passages,92with no single archive dominating.9394## 4. Rewrite protocol (`src/htr/rewrite/`, prompt version `rw-1.0`)9596**Prompts.**9798- **System prompt.** Forbids invented information. Requires each placeholder `⟦C n⟧`, `⟦M n⟧`, `⟦U n⟧` to be99  copied verbatim, exactly once per occurrence.100- **PRESERVE block.** Requires preservation of numbers, units, entities, citations, math, URLs, negation,101  hedges and uncertainty. Output must be a single English paragraph.102103**Variants.** A neutral paraphrase · B polished/standardised · C concise (~70 % length) · D104structure-preserving (same sentence count and order) · E diverse sampling (2 samples).105106**Held-out prompts.** H1 (plain-language expert) and H2 (journal copy-editor) are never used in training.107108**Placeholder protocol.**109110- `protect()` replaces math, URLs/DOIs and numeric or author–year citations by indexed placeholders before111  any LLM call.112- Both rewrite and target are stored in placeholder form.113- Numbers stay as text. They are checked, not masked (D9).114115**Provider (`anthropic_provider.py`).**116117- Model ids are pinned: Opus 5.5 (`claude-opus-5-5`) is the primary rewriter. Sonnet 5.5 and Haiku 5.5 are118  held-out models for RQ4 (`configs/rewriting/pilot.json`).119- `output_config.effort = "low"`. Thinking is adaptive (provider default). No temperature is set.120- Each response logs model, request id, usage, stop reason, cost and latency.121- Retries use exponential backoff.122- Message Batches are billed at 50 %.123- Request ids are deterministic (passage, variant, sample, provider, model, prompt version). This deduplicates124  requests and makes runs resumable. The output is an append-only `rewrites.jsonl` plus `ledger.json`.125- The default provider is the zero-cost mock. Real calls require `--live`. The budget ceiling is enforced on126  recorded spend (pilot: 30 USD).127128**Validation (`validate/checks.py`, `val-1.0`).** A pair passes only if all of the following hold:129130- length ratio within [0.5, 1.8] (0.4 for C);131- normalised edit similarity within [0.05, 0.95] (above 0.95 counts as a near-verbatim copy);132- language is English;133- the placeholder multiset is identical;134- no numbers are missing and at most 1 is extra;135- proxy entity recall ≥ 0.6;136- |Δ negation| ≤ 2 and |Δ hedge| ≤ 4;137- for D, sentence count is within ±1;138- there are no meta-text artefacts;139- optionally, MiniLM cosine ≥ 0.70.140141Failed pairs are quarantined, not dropped. Bidirectional entailment (mandate Phase 3) is **not yet142implemented**.143144## 5. Model family (`src/htr/models/`)145146**Shared backbone.** A pre-LN encoder–decoder with RoPE, SwiGLU feed-forward and tied embeddings. The tokenizer147is a from-scratch SentencePiece unigram model trained on train-split text only, with atomic control and148placeholder tokens.149150**M1** is the plain backbone with an optional copy gate.151152**M2** adds learned discipline, era, section-role and source embeddings to the encoder and decoder inputs, plus153one `<meta>` memory token.154155**M3** (dual-latent) is defined as follows:156157- **Content and style.** E_c is the encoder. The content vector is c = W_c · mean-pool(H). The style branch158  E_s = MLP([mean;max](H) + meta) predicts a distribution over K style classes. K = observed159  (discipline | era) buckets with ≥ 20 training pairs, plus "other".160- **Style memory.** Style memory = Σ_k p_k · prototype_k. It contributes 4 memory tokens, which are prepended161  to the cross-attention memory.162- **Adversary.** A gradient-reversal adversary predicts the style class from c.163- **Copy gate.** p(w) = (1−g_t)·softmax(W h_t)_w + g_t · Σ_{i: x_i=w} a_{t,i}, where a is the head-averaged164  last-layer cross-attention over source positions, renormalised, and g_t = σ(w_g·h_t).165- **No target at inference.** The original target is never an input at inference time. Metadata, if used, is166  supplied by the caller.167168| Design point (`config.py`) | d_model | heads | d_ff | enc/dec layers | Mandate label | Parameters at init, vocab 32 000, M1 / M2 / M3 (K = 24) |169|---|---|---|---|---|---|---|170| nano (tests) | 256 | 4 | 1 024 | 3/3 | — | 15.3 M / 15.4 M / 15.7 M |171| tiny | 384 | 6 | 1 536 | 4/4 | HTR-Tiny ~30 M | 33.5 M (M1, vocab 32 k; 27.4 M measured at vocab 16 k) |172| small | 768 | 12 | 3 072 | 5/5 | HTR-Small ~125 M | 130.8 M (M1, vocab 32 k) |173| base | 1 024 | 16 | 4 096 | 8/8 | HTR-Base ~350 M (release target) | 334.8 M (M1, vocab 32 k) |174| large | 1 536 | 24 | 6 144 | 12/12 | HTR-Large ~1 B (conditional) | 1 068.6 M (M1, vocab 32 k) |175176**How the counts were obtained.** Parameter counts come from instantiating `HTRModel` on the meta device. Trained177checkpoint counts are logged in `manifest.json` and are pending, except for the smoke run:178`vs_smoke-m1-nano` has 8 326 401 parameters at tokenizer vocab 4 842.179180**Mismatch with the mandate.** The design points exceed the mandate's approximate sizes by 40–60 %. Either181resize them or relabel them before Gate D.182183## 6. Objective (`src/htr/models/losses.py`)184185**Notation.**186187| Symbol | Meaning |188|---|---|189| x | rewrite tokens |190| y_{1..T} | original tokens (placeholder form) |191| N | number of non-pad target tokens |192| V | vocabulary size |193| p | copy-augmented output distribution |194| f_t | 1 if target piece t contains `⟦` or a digit, else 0 |195| k | style class |196| sg | stop-gradient |197| c | content vector of the rewrite |198| c′ | content vector of the original, from the same encoder under `no_grad` |199200**Full objective.**201202L = λ_ce L_token + λ_fact L_factual + λ_style L_style + λ_adv L_adv + λ_con L_contrastive + λ_sem L_semantic + λ_orth L_orth203204**Terms.**205206| Term | Definition |207|---|---|208| L_token | (1/N) Σ_t [(1−ε)(−log p(y_t)) + ε · (1/V) Σ_v (−log p(v))], with ε = 0.1 (D8; ε = 0 in the overfit test) |209| L_factual | Σ_t f_t (−log p(y_t)) / max(1, Σ_t f_t). Unsmoothed NLL on factual tokens. |210| L_style | CE(E_s logits, k) |211| L_adv | CE(A(GRL_λ(c)), k). GRL is the identity forward and −λ·∇ backward; λ = `grl_lambda` = 0.1. |212| L_contrastive | ½[CE(S, I) + CE(Sᵀ, I)], with S_ij = ĉ_i · sg(ĉ′_j)/τ, τ = 0.07, ĉ L2-normalised; the other items in the batch are the negatives |213| L_semantic | 1 − mean_b cos(c_b, sg(c′_b)) |214| L_orth | ‖C̃ᵀ S̃ /(B−1)‖²_F / (d·d), with C̃ and S̃ the batch-centred content and style vectors |215216**Differentiability.**217218- All terms are differentiable with respect to model parameters.219- f_t and k are constants.220- The copy mixture is differentiable through the gate and the attention weights.221- c′ receives no gradient.222- Through the GRL, the adversary minimises L_adv while the encoder receives −λ times its gradient.223- Logged accuracies (argmax) are not in the graph.224225**Where the terms apply.** L_style, L_adv, L_con, L_sem and L_orth are active only for `arch = m3`. The original226is encoded only inside the loss (training).227228**Weights.** Preset `m3`: ce 1, fact 0.5, style 0.3, adv 0.3, con 0.2, sem 0.1, orth 0.1. Weights are tuned on229validation only.230231**Controls.** M1 and M2 are token-loss only (ce 1). The ablation presets are `m3_tokenonly`, `m3_nostyle`,232`m3_noadv`, `m3_nofact` and `m3_nocon`.233234**Disentanglement claims.** Chance-level adversary accuracy is **not** taken as evidence of disentanglement235(`elazar2018adversarial`). A post-hoc probe on frozen c is required.236237## 7. Baselines238239| Baseline | Description | Status |240|---|---|---|241| **B0** | Identity: output = rewrite | `evaluation/baselines.py` |242| **B1** | "Generic paraphrase": the deterministic rule-based mock paraphraser (variant A) applied to the rewrite | Implemented. Weak, because it does not learn. A minimally trained rewrite baseline (e.g. M1 trained on rewrite→rewrite′) is planned. |243| **B2** | Fine-tuning a pretrained encoder–decoder (T5/BART family), subject to weight licences | The docstring references `htr.training.hf_finetune`, which **does not exist yet** |244245All trained systems are compared at matched data and with FLOPs reported.246247## 8. Evaluation plan248249**Instance level** (`metrics.py`):250251- exact match, normalised edit similarity, chrF++, sentence BLEU, ROUGE-L, length ratio;252- factual: placeholder exactness and recall, number recall and precision, negation, hedge and sentence-count253  deltas.254255All instance metrics are reported next to B0.256257**Corpus level.**258259- Document-clustered bootstrap CIs (`bootstrap.py`), by variant, discipline, era and role (`runner.py`).260- Paired differences vs B0.261262**Stylometric fidelity.**263264- 18 features: sentence length mean/sd, TTR, MTLD, word length, long-word, comma, semicolon, parenthesis,265  discourse, passive, nominalisation, first person, subordination, hedge, negation, digit rates, Flesch.266- W1 is the 1-D Wasserstein distance per feature, standardised by the reference spread, then averaged267  (`W1_mean`).268- Comparisons: rewrite vs original, and each system vs original.269270**Exploratory probe.** A logistic regression (original = 1 vs rewrite = 0), with a 70/30 split. We report AUC and271mean P(original) on reconstructions. It is never used as evidence of origin.272273- Current limitation: the probe is fitted on the originals and rewrites of the evaluated pairs. A disjoint274  probe-training split is required before any reported probe number.275276**Non-identifiability (planned, not implemented).** Three measurements:277278- (a) Consistency: for the six rewrites of one original, compare the agreement among their reconstructions with279  their distance to the original.280- (b) Confusable originals: take near-neighbour originals from different documents (embedding and MinHash281  neighbours). Measure identification accuracy, i.e. whether the reconstruction is closer to its own original.282- (c) An "oracle" ceiling: the chrF++ of the best available rewrite.283284The ambiguity rate goes to RQ6.285286**Memorisation audit (planned).** Four checks:287288- longest common token span and 13-gram overlap between each reconstruction and the training originals;289- the excess overlap of training neighbours over the test original;290- synthetic canaries inserted into training, then measured for exposure (lawful because they are synthetic);291- for B2, a contamination audit of the pretraining data where it is documented.292293**Human evaluation (planned).** See `docs/experiments/EXPERIMENT_MATRIX.md` (E16).294295## 9. Gates (acceptance criteria → evidence)296297| Gate | Acceptance criteria | Evidence | Status |298|---|---|---|---|299| A | Repository, licence governance, 100-paper smoke ingestion, parsed-quality audit | `docs/gates/GATE_A.md`, `data/manifests/discover_pmc_seed1.json`, `data/parsed/smoke/build_summary.json` | **GO** (2026-11-09) |300| B | 500–1 000 validated Claude pairs; spend ≤ 30 USD recorded in the ledger; acceptance rate per variant; stratified human review (≥ 100 pairs) | `data/pairs/<pilot>/ledger.json`, `validated/validation_summary.json`, `docs/gates/GATE_B.md` | pending |301| C | Splits frozen and leak-checked on the pilot+books+arXiv corpus; B0/B1 (± B2) evaluated; `PREREGISTRATION.md` v1.0 frozen with commit hash | `data/splits/<corpus>/splits_summary.json`, `artifacts/metrics/<run>/report.json` | pending |302| D | M1/M2/M3 one-batch overfit; pilot runs × 3 seeds; validation results with CIs | `artifacts/runs/<run>/{overfit_test,train_summary,manifest}.json` | pending (smoke M1-nano overfit only: CE 5.28 → 0.0022 in 150 steps, `vs_smoke-m1-nano`) |303| E | Scale only if some model beats B0 on **validation** chrF++ with a CI that excludes 0 and costs are approved; all negative results published | gate report + approved budget line | pending |304| F | Clean-environment install and inference; model card; release audit (licence, privacy, memorisation); compiled `paper/main.pdf` | `docs/gates/GATE_F.md`, `model_cards/`, `paper/main.pdf` | pending |305306## 10. Roadmap and decision points3073081. **Corpus expansion** (this week): arXiv q-fin/econ/math/cs/stat/physics, Crossref econ/finance, books.309   Rebuild the splits with venue-safe and author-safe options.310   - Decision point DP1: is the mix balanced, i.e. no archive above roughly 50 %?3112. **Gate B pilot:** estimate the cost, then submit batches.312   - DP2: is acceptance ≥ 70 % per variant? If not, revise the prompts (`rw-1.1`) before scaling.3133. **Freeze the pre-registration** (DP3), then run baselines.3144. **M1/M2/M3 at tiny size** on the pilot pairs.315   - DP4: does any model beat B0 on validation? If not, report the null result. Scale only to test RQ5.3165. **Medium-scale rewriting** (20 000 passages and up). Requires an explicit budget approval317   (`docs/reproducibility/COMPUTE_BUDGET.md`).318   - DP5: small/base on M4M64a, M3U96a/b or the laptop, depending on throughput measured at tiny size.3196. **Final confirmatory scoring.** Run once on the test and held-out sets.3207. **Release candidate:** audit, model card, paper.321322## 11. Release plan323324Planned layout of the Hugging Face-compatible model directory:325326| File | Content |327|---|---|328| `model.safetensors` | weights (`save_checkpoint` already writes it) |329| `config.json` | `model_type = htr` |330| `tokenizer.model` + `tokenizer_config.json` | tokenizer |331| `release_meta.json` | style classes, metadata-prefix flag |332| `README.md` | model card: data, licences, metrics with CIs, limitations, intended research uses |333| `NOTICE` | attribution |334335**API and CLI.** The usable API is `HTRReconstructor.from_pretrained(dir).reconstruct(text, metadata)`336(`inference/pipeline.py`). Decoding is deterministic, greedy or beam. Placeholders are restored. Preservation337warnings are produced, along with a notice that the output is synthetic. The CLI entry point `htr = htr.cli:main`338is declared in `pyproject.toml`, but `htr/cli.py` **does not exist yet**.339340**Demo.** A small demo showing the input, the reconstruction, a diff and the warnings. `demo/` is empty.341342**Release conditions.** Weights are released only for models trained on the release set, or after legal review343of NC/SA inheritance (`LICENSE_MATRIX.md`). Nothing is uploaded and the namespace is not chosen without the344author's approval.345346## 12. Honest-claims policy347348- Outputs are **synthetic reconstructions**. They do not restore authorship, they are not evidence of human349  authorship, and they are not an AI detector.350- Verbatim reconstruction, stylistic distribution matching and factual fidelity are reported separately.351- Every gain is stated against B0 and B1, with document-clustered CIs.352- Null and negative findings are reported in the paper and the gate reports.353- Probe and classifier scores are exploratory.354- Only rights-verified data and verified citations are used, and every number traces to a run manifest.355