Gate B — Validated Claude pairs with documented spend and human review
Date: 2026-11-09 · Decision: GO · Evidence: data/pairs/pilot_opus/{ledger.json,manifest.json,batches/},
data/pairs/pilot_opus/validated/validation_summary.json, data/pairs/pilot_heldout_models/validated/validation_summary.json,
data/pairs/pilot_heldout_prompts/validated/validation_summary.json, data/manifests/discover_pmc_seed2.json,
data/parsed/pilot_pmc/build_summary.json, data/splits/pilot_pmc/splits_summary.json.
Corpus and splits used for the pilot
| Item | Value |
|---|---|
| PMC candidates sampled (seed 2, stratified id range) → metadata found → fetched | 4 187 → 1 820 → 1 200 (TDM 405, UNKNOWN 137, ND 74 excluded; not-OA / manuscripts skipped) |
| Parsed / kept (pre-2018 verified from JATS, licence in training set, English, not duplicate) | 1 200 / 1 180 (12 exact duplicates, 5 dated 2018, 3 licence-excluded) |
| Licences of kept documents | CC-BY 833, CC-BY-NC 198, CC-BY-NC-SA 124, PUBLIC-DOMAIN 19, CC0 6 |
| Passages / eligible | 34 695 / 23 658 |
| Splits (seed 20261109, author-group + near-duplicate safe; held-out discipline = chemistry as labelled at split time, held-out period = pub_year < 1950) | train 856 · val 98 · test 100 · heldout_domain 47 · heldout_period 79 documents; 0 group leaks |
| Discipline labels | relabelled after the split with the corrected taxonomy (relabel_log.json: 165/1 180 changed; e.g. "Physiology" had matched physics). The held-out-domain group therefore corresponds to the pre-fix chemistry label and is reported as such. |
Rewrite job pilot_opus (Claude Opus 5.5, claude-opus-5-5, effort low, prompt rw-1.0, placeholder protocol 1.0, Message Batches)
| Item | Value |
|---|---|
| Passages selected (stratified by discipline × section role, per split) | train 700 · val 90 · test 150 · heldout_domain 30 · heldout_period 4 (only 4 eligible passages exist in pre-1950 OCR documents) = 974 |
| Requests (variants A, B, C, D ×1, E ×2) | 5 844 in 7 batches, all ended; 5 801 ok, 43 refused (0.7 %, stop_reason=refusal; e.g. HBV/CRISPR cccDNA passages), 0 errors, 0 truncated |
| Mean tokens per request (measured) | input 691, output 323 (pre-flight estimate had assumed 492 / 259) |
| Spend (ledger, batch prices) | 26.85 USD (ceiling 30 USD) |
Automated validation (val-1.0: language, length ratio, near-copy, placeholder multiset, numbers, entities, negation/hedging deltas, structure for D, formatting artefacts, MiniLM semantic similarity ≥ 0.70) |
5 383 / 5 844 passed (92.1 %); by variant A 896/974, B 884/974, C 894/974, D 905/974, E 1 804/1 948 |
| Quarantine reasons (top) | numbers missing ≥ 1: 318 · refused: 43 · entity recall < 0.6: 39 · extra numbers: 31 · markdown/meta artefacts: 21 · negation delta: 11 |
Held-out generalisation sets (test passages only, never used in training):
| Job | Model / prompts | Requests | ok / refused | Spend | Validation pass |
|---|---|---|---|---|---|
pilot_heldout_models |
claude-sonnet-5-5 and claude-haiku-5-5, variants A, B | 600 | 600 / 0 | 1.43 USD | 531 / 600 (88.5 %) |
pilot_heldout_prompts |
claude-opus-5-5, held-out prompts H1 (plain expert), H2 (copy-editor) | 300 | 297 / 3 | 1.40 USD | 266 / 300 (88.7 %) |
Total API spend to date: 29.68 USD (all jobs, batch pricing, from the ledgers).
Human review (author-side review by the operating assistant, 2026-11-09; not an independent rater study)
A stratified sample of 12 validated pairs (2 per variant, 3 for E; disciplines biomedicine, chemistry, physics, computer science, statistics, economics labels as then assigned) was read side by side.
- Fidelity: all 12 rewrites preserved the propositions, numbers, units, named entities and placeholders; none added or removed a claim. Variant C compressed by 18–21 % without dropping facts; variant D kept sentence counts.
- Style: rewrites are uniformly smoother and more nominal; hedges were preserved; British/American spelling was sometimes normalised ("archeology" → "archaeology").
- Artefacts: one variant-B rewrite wrapped species names in Markdown italics (
*Trypanosoma cruzi*); the validator now rejects single-asterisk/underscore italics and headings (21 pairs quarantined for formatting after the fix). - Validator calibration from the quarantine audit: (i) entity-recall false positives were caused by apostrophe
variants, possessives, hyphenated compounds and footnote marks → tokens are now normalised and matched as
substrings; (ii) numbers glued to units ("0.13mm") were not extracted → fixed; (iii) a stricter citation pattern
introduced after the batches were generated changed the placeholder set → the placeholder protocol is now
versioned (
protect_version), and each pair is validated with the version used at request time. The pilot pairs are protocol 1.0; new jobs use 1.1. - Residual quarantine is dominated by genuine number drops (e.g. a model omitting one of several p-values in a long results paragraph), which is exactly the failure mode the factual-integrity checks exist to catch.
Issues carried forward
- The pre-1950 held-out period has only 4 eligible PMC passages (OCR noise); historical style variation must come from the books stratum (Gutenberg/Bookshelf) at scale.
- Economics/finance pairs are nearly absent in the PMC pilot (3 + 1 documents); the Crossref (1 730 eligible articles) and arXiv (q-fin/econ) adapters supply them for the next stage.
- Entity and number checks are heuristics; spelled-out numbers ("twelve") are counted as missing. Reported as a validator limitation, not corrected post hoc.
- 0.7 % refusals (bio safety classifier) are recorded as abstentions; passages are not resubmitted.
Next action
Gate C: leakage-safe splits are in place; train tokenizer (done, 16 k unigram on train split) and baselines B0/B1/B2 and the M1 pilot; freeze PREREGISTRATION v1.0 before scoring any confirmatory split.