Gate A — Repository, licence governance, 100-paper ingestion smoke test, parsed quality audit
Date: 2026-11-09 · Decision: GO (with the caveats listed below) · Evidence: data/manifests/discover_pmc_seed1.json,
data/parsed/smoke/build_summary.json, data/splits/smoke/splits_summary.json, data/pairs/mock_smoke/validated/validation_summary.json,
unit tests (.venv/bin/python -m pytest tests/unit -q).
Measured
| Item | Value |
|---|---|
| PMCID candidates sampled (stratified over PMC10000–PMC5760000) | 347 |
| Metadata objects found (404 = id unused/removed) | 147 |
| Licence distribution of found objects | {'CC-BY-NC-ND': 4, 'CC-BY': 75, 'CC-BY-NC-SA': 11, 'UNKNOWN': 12, 'TDM': 31, 'CC0': 1, 'CC-BY-NC': 13} |
| Rejected before the licence filter (not in the OA subset: 30, author manuscript: 1) | 31 |
| Excluded by licence policy (ND / unknown / other) | 16 |
| Documents fetched (JATS XML, MD5 verified against provider digest) | 100 in 39.4 s |
| Parsed without error | 100 / 100 |
Publication date verified from JATS pub-date and < 2018-01-01 |
100 / 100 (0 undated, 0 post-2017) |
| Licences of kept documents | {'CC-BY': 75, 'CC-BY-NC-SA': 11, 'CC-BY-NC': 13, 'CC0': 1} |
| Discipline (coarse, from journal/subject hints) | {'biomedicine': 75, 'life_sciences': 9, 'social_sciences': 7, 'computer_science': 4, 'chemistry': 4, 'physics': 1} |
| Paragraphs extracted / eligible passages (60–400 words, ≥ 2 sentences, English ≥ 0.9, quality ≥ 0.6) | 2964 / 2013 |
| Eligible passages by section role | {'abstract': 102, 'background': 74, 'results': 475, 'discussion': 389, 'conclusion': 36, 'methods': 367, 'other': 473, 'data': 50, 'theory': 21, 'related_work': 4, 'limitations': 7, 'introduction': 15} |
| Near-duplicate clusters (MinHash, Jaccard ≥ 0.8) | exact 0, near 0 |
| Leakage-safe split of the 100 docs (seed 20261109, author-group safe, physics + pre-1990 held out) | {'heldout_domain': 1, 'heldout_period': 12, 'test': 12, 'train': 69, 'val': 6}, group leaks: [] |
| Mock rewrite run (zero cost) 60 passages × 6 generations | 360 rewrites, acceptance 0.7917 (fails = near-verbatim mock outputs) |
Quality audit (manual, 2026-11-09)
- Citations: numeric xrefs are normalised to
[n]; author–year citations are kept as text; both are detected byprotect()and replaced by⟦Cn⟧placeholders before any LLM call. A doubled-bracket bug ([[16]]) was found and fixed in this gate. - Section roles: an "Introduction" → "other" bug caused by the numbering-stripping regex (it ate the leading "I") was found by auditing the 40 most frequent titles of "other" passages, and fixed. Remaining "other" passages are untitled sections or topical headings (e.g. "The Follow-up Period"), which is the expected residue.
- Historical back-scanned articles (1877–1986, OCR text) are present in PMC OA under CC licences; they pass the quality gate rarely (2 eligible passages from 6 such documents), so OCR noise is not a dominant risk but must be re-checked at scale.
- Captions and list items are extracted but never eligible; references, acknowledgements, funding, contributions, ethics, supplementary sections are excluded.
Issues / risks carried forward
- The PMC FTP bulk packages and the OA Web Service were retired in August 2026; the only official bulk channel is the
AWS Open Data bucket
pmc-oa-opendata(per-article JSON + JATS). The id-range sampling prior must be re-validated when scaling (date is verified per document anyway). - Discipline labels for PMC come from journal/subject keywords — coarse. Non-biomedical strata (economics, finance, mathematics, physics, computer science, books) come from the other adapters (arXiv, Crossref journals, Gutenberg/Bookshelf).
TDMauthor manuscripts andCC-BY-NDarticles are ingested as metadata only (31 + 4 of 147 found objects).
Next action
Gate B: pilot rewrite of ~1 000 validated passages with Claude (budget ceiling 30 USD, Message Batches), human review of a stratified sample, acceptance-rate report.