spb/wpai0002_uqo

1# Gate B — Validated Claude pairs with documented spend and human review23**Date:** 2026-11-09 · **Decision: GO** · Evidence: `data/pairs/pilot_opus/{ledger.json,manifest.json,batches/}`,4`data/pairs/pilot_opus/validated/validation_summary.json`, `data/pairs/pilot_heldout_models/validated/validation_summary.json`,5`data/pairs/pilot_heldout_prompts/validated/validation_summary.json`, `data/manifests/discover_pmc_seed2.json`,6`data/parsed/pilot_pmc/build_summary.json`, `data/splits/pilot_pmc/splits_summary.json`.78## Corpus and splits used for the pilot910| Item | Value |11|---|---|12| PMC candidates sampled (seed 2, stratified id range) → metadata found → fetched | 4 187 → 1 820 → 1 200 (TDM 405, UNKNOWN 137, ND 74 excluded; not-OA / manuscripts skipped) |13| Parsed / kept (pre-2018 verified from JATS, licence in training set, English, not duplicate) | 1 200 / 1 180 (12 exact duplicates, 5 dated 2018, 3 licence-excluded) |14| Licences of kept documents | CC-BY 833, CC-BY-NC 198, CC-BY-NC-SA 124, PUBLIC-DOMAIN 19, CC0 6 |15| Passages / eligible | 34 695 / 23 658 |16| Splits (seed 20261109, author-group + near-duplicate safe; held-out discipline = chemistry *as labelled at split time*, held-out period = pub_year < 1950) | train 856 · val 98 · test 100 · heldout_domain 47 · heldout_period 79 documents; 0 group leaks |17| Discipline labels | relabelled after the split with the corrected taxonomy (`relabel_log.json`: 165/1 180 changed; e.g. "Physiology" had matched *physics*). The held-out-domain group therefore corresponds to the *pre-fix* chemistry label and is reported as such. |1819## Rewrite job `pilot_opus` (Claude Opus 5.5, `claude-opus-5-5`, effort low, prompt `rw-1.0`, placeholder protocol 1.0, Message Batches)2021| Item | Value |22|---|---|23| Passages selected (stratified by discipline × section role, per split) | train 700 · val 90 · test 150 · heldout_domain 30 · heldout_period 4 (only 4 eligible passages exist in pre-1950 OCR documents) = 974 |24| Requests (variants A, B, C, D ×1, E ×2) | 5 844 in 7 batches, all `ended`; 5 801 ok, **43 refused** (0.7 %, `stop_reason=refusal`; e.g. HBV/CRISPR cccDNA passages), 0 errors, 0 truncated |25| Mean tokens per request (measured) | input 691, output 323 (pre-flight estimate had assumed 492 / 259) |26| **Spend** (ledger, batch prices) | **26.85 USD** (ceiling 30 USD) |27| Automated validation (`val-1.0`: language, length ratio, near-copy, placeholder multiset, numbers, entities, negation/hedging deltas, structure for D, formatting artefacts, MiniLM semantic similarity ≥ 0.70) | **5 383 / 5 844 passed (92.1 %)**; by variant A 896/974, B 884/974, C 894/974, D 905/974, E 1 804/1 948 |28| Quarantine reasons (top) | numbers missing ≥ 1: 318 · refused: 43 · entity recall < 0.6: 39 · extra numbers: 31 · markdown/meta artefacts: 21 · negation delta: 11 |2930Held-out generalisation sets (test passages only, never used in training):3132| Job | Model / prompts | Requests | ok / refused | Spend | Validation pass |33|---|---|---|---|---|---|34| `pilot_heldout_models` | claude-sonnet-5-5 and claude-haiku-5-5, variants A, B | 600 | 600 / 0 | 1.43 USD | 531 / 600 (88.5 %) |35| `pilot_heldout_prompts` | claude-opus-5-5, held-out prompts H1 (plain expert), H2 (copy-editor) | 300 | 297 / 3 | 1.40 USD | 266 / 300 (88.7 %) |3637**Total API spend to date: 29.68 USD** (all jobs, batch pricing, from the ledgers).3839## Human review (author-side review by the operating assistant, 2026-11-09; not an independent rater study)4041A stratified sample of 12 validated pairs (2 per variant, 3 for E; disciplines biomedicine, chemistry, physics,42computer science, statistics, economics labels as then assigned) was read side by side.4344- Fidelity: all 12 rewrites preserved the propositions, numbers, units, named entities and placeholders; none added45  or removed a claim. Variant C compressed by 18–21 % without dropping facts; variant D kept sentence counts.46- Style: rewrites are uniformly smoother and more nominal; hedges were preserved; British/American spelling was47  sometimes normalised ("archeology" → "archaeology").48- Artefacts: one variant-B rewrite wrapped species names in Markdown italics (`*Trypanosoma cruzi*`); the validator49  now rejects single-asterisk/underscore italics and headings (21 pairs quarantined for formatting after the fix).50- Validator calibration from the quarantine audit: (i) entity-recall false positives were caused by apostrophe51  variants, possessives, hyphenated compounds and footnote marks → tokens are now normalised and matched as52  substrings; (ii) numbers glued to units ("0.13mm") were not extracted → fixed; (iii) a stricter citation pattern53  introduced after the batches were generated changed the placeholder set → the placeholder protocol is now54  **versioned** (`protect_version`), and each pair is validated with the version used at request time. The pilot pairs55  are protocol 1.0; new jobs use 1.1.56- Residual quarantine is dominated by genuine number drops (e.g. a model omitting one of several p-values in a long57  results paragraph), which is exactly the failure mode the factual-integrity checks exist to catch.5859## Issues carried forward60611. The pre-1950 held-out period has only 4 eligible PMC passages (OCR noise); historical style variation must come from62   the books stratum (Gutenberg/Bookshelf) at scale.632. Economics/finance pairs are nearly absent in the PMC pilot (3 + 1 documents); the Crossref (1 730 eligible64   articles) and arXiv (q-fin/econ) adapters supply them for the next stage.653. Entity and number checks are heuristics; spelled-out numbers ("twelve") are counted as missing. Reported as a66   validator limitation, not corrected post hoc.674. 0.7 % refusals (bio safety classifier) are recorded as abstentions; passages are not resubmitted.6869## Next action7071Gate C: leakage-safe splits are in place; train tokenizer (done, 16 k unigram on train split) and baselines B0/B1/B272and the M1 pilot; freeze PREREGISTRATION v1.0 before scoring any confirmatory split.73