spb/wpai0002_uqo

1# Data card — HTR corpus and HTR-Benchmark (draft, 2026-11-09)23## Summary4Paired data: **source** = a passage rewritten by a large language model (Claude Opus 5.5 primary; Sonnet 5.5 / Haiku 5.55held-out), **target** = the original English passage from a document published before 2018-01-01 whose licence permits6the operations we perform. Both sides are stored in *placeholder form* (citations ⟦Cn⟧, math ⟦Mn⟧, links ⟦Un⟧; see7`src/htr/parse/protect.py`, protocol version recorded per pair). Metadata per pair: discipline, publication year/era,8section role, source archive, licence code, venue, author group, rewriting model, prompt version, protocol version.910## Sources (all acquired through official channels, rate-limited, rights recorded per document)11| Source | Access route | Date evidence | Licence evidence | Scale observed |12|---|---|---|---|---|13| PMC Open Access Subset | AWS Open Data bucket `pmc-oa-opendata` (per-article JSON + JATS XML; MD5 verified) | JATS `<pub-date>` (ppub → collection → epub) | JSON `license_code` cross-checked with JATS `<license xlink:href>` | pilot 1 180 docs; Medium ingestion ~10 000 |14| arXiv (q-fin, econ; others possible) | OAI-PMH `arXivRaw` metadata + e-print source of the last pre-2018 version (3 s between requests) | v1 submission date | OAI `license` URI (only CC licences kept; the arXiv non-exclusive licence is excluded) | 162 eligible q-fin papers 2008–2017 |15| CC BY economics & finance journals (16 verified: MDPI Econometrics, JRFM, IJFS, Risks, Economies, Games; SpringerOpen Financial Innovation, J. Economic Structures, IZA journals, LAER, SJES, JIE) | Crossref REST (polite pool) + publisher PDF via Crossref text-mining links | Crossref `published-print` → `published-online` → `issued` | Crossref `license[]` (content-version vor) | 1 730 eligible articles |16| Project Gutenberg (reference / scholarly non-fiction, LoCC Q R T H B G L Z) | catalog feed + official mirror `gutenberg.pglaf.org` | front matter year (imprint / copyright / roman numerals); else author death year as an **upper bound** (flagged) | public domain in the US (catalog + licence header) | pool 7 799; 500 requested |17| NCBI Bookshelf open-access books | FTP `pub/litarch` (BITS XML) | BITS `<pub-date>` | BITS `<permissions>` (CC BY / public domain NIH works; UNKNOWN rejected) | 3 941 pre-2018 books; 300 requested |18| OpenStax / CNX pre-2018 editions | Internet Archive items; date and CC statement read **inside the PDF** | PDF copyright page | PDF licence statement | 5 trial items |1920Excluded by policy: `TDM` author manuscripts, `CC-BY-ND` / `CC-BY-NC-ND`, `UNKNOWN`/`OTHER-OA`, the arXiv non-exclusive21licence, retracted articles, non-English text, near-duplicates (MinHash Jaccard ≥ 0.8), references/acknowledgements/22funding/ethics sections, captions and list items (kept in the parsed store, never eligible).2324## Processing25JATS/BITS, LaTeX, PDF and plain-text parsers → paragraphs with offsets into the normalised document text → eligibility26(60–400 words, ≥ 2 sentences, English ≥ 0.9, quality ≥ 0.6) → protected-span placeholders → LLM rewrite (variants A–E,27prompts `rw-1.0`) → automated validation (`val-1.0`: language, length ratio, near-copy, placeholder multiset, numbers,28entities, negation/hedging deltas, structure for D, formatting artefacts, MiniLM semantic similarity) → quarantine of29failures (kept, never deleted) → leakage-safe splits (document + near-duplicate cluster + author group; held-out discipline30and period; pilot assignments frozen).3132## Release policy (HTR-Benchmark)33- **Public subset**: pairs whose document licence is `CC0`, `PUBLIC-DOMAIN` or `CC-BY` — redistributed with attribution34  (title, authors, venue, DOI/PMCID/arXiv id/Gutenberg id, licence URI, "modified: segmented and placeholder-protected").35- **Research subset** (`CC-BY-SA`, `CC-BY-NC`, `CC-BY-NC-SA`): metadata + identifiers + scripts to rebuild locally; text36  shared only under the matching licence terms (SA → CC BY-SA; NC → non-commercial research).37- **Never released**: documents of tiers C/X, quarantined pairs' rewrites of non-releasable originals, provider request38  payloads beyond what the schema records.39- LLM rewrites are our outputs under Anthropic's commercial terms; their redistribution and use for training are recorded as a40  compliance item to confirm before publication (`docs/RISK_REGISTER.md`).41- Takedown: a retraction or licence change at the source triggers removal from the next release and a dated note in42  `docs/DECISION_LOG.md`.4344## Known biases and limitations45- Biomedicine dominates PMC; economics/finance, mathematics, physics and books are deliberate strata but smaller.46- Discipline labels are heuristic (journal title / subject categories / arXiv categories / LoCC classes).47- Pre-1950 PMC material is OCR and rarely passes quality; historical variation comes mostly from books, whose dates may48  be upper bounds.49- Validation heuristics: spelled-out numbers count as missing; entity matching is a proxy.50- The rewrites reflect a specific model generation (Claude 5.5 family, late 2026) and six prompt variants.5152## Citation and acknowledgement53Acknowledge NLM/NCBI (PMC Article Datasets on AWS, accessed 2026-11), arXiv, Crossref, the publishers of the CC BY54journals, Project Gutenberg, the Internet Archive and OpenStax (Rice University). See `LICENSE_MATRIX.md`.55