spb/wpai0002_uqo

1# Licence matrix23Version 0.1 · 2026-11-09. The single source of truth is `src/htr/licenses/policy.py`. This document describes4that policy; if the two ever disagree, the code wins and this file must be corrected. Open access ≠ open5licence. The policy is conservative: a document whose licence is unclear is kept as metadata only.67## 1. Canonical codes and permitted operations89| Code | Tier | Operations allowed by `POLICY` | Conditions (from `policy.py`) | Training (default) | Release (default) |10|---|---|---|---|---|---|11| `CC0` | A | ingest, parse, train, evaluate, release_source_text, release_derivative_text, commercial_use | attribution recommended | yes | yes |12| `PUBLIC-DOMAIN` | A | all | verify public-domain status per jurisdiction; US works only unless stated | yes | yes |13| `CC-BY` | A | all | attribution: title, authors, source, licence URI, changes stated | yes | yes |14| `CC-BY-SA` | B | all (commercial allowed) | derivative text must be released under CC BY-SA; attribution | yes | no (ShareAlike release only under a separate SA bundle) |15| `CC-BY-NC` | B | all except commercial_use | non-commercial research use only; derivative text released NC; attribution | yes | no |16| `CC-BY-NC-SA` | B | all except commercial_use | non-commercial; derivatives CC BY-NC-SA; attribution | yes | no |17| `CC-BY-ND` | C | ingest, parse, evaluate | no derivative distribution; training disabled by default | no | no |18| `CC-BY-NC-ND` | C | ingest, parse, evaluate | no derivative distribution; non-commercial; training disabled by default | no | no |19| `TDM` | C | ingest, parse, evaluate | PMC author manuscripts: text mining only; no redistribution | no | no |20| `ARXIV-NONEXCLUSIVE` | C | ingest, parse, evaluate | the arXiv licence grants distribution rights to arXiv, not to us; metadata and internal analysis only | no | no |21| `OTHER-OA` | X | ingest (metadata) | no machine-readable licence | no | no |22| `UNKNOWN` | X | ingest (metadata) | licence not determined | no | no |2324**Defaults.** The training set is `TRAINING_CODES_DEFAULT` = CC0, PUBLIC-DOMAIN, CC-BY, CC-BY-SA, CC-BY-NC,25CC-BY-NC-SA. The release set is `RELEASE_CODES_DEFAULT` = CC0, PUBLIC-DOMAIN, CC-BY.2627**Eligibility rules.**2829- `is_training_eligible` requires the code to be in the configured set **and** to allow `train`.30- `is_release_eligible` requires the code to be in the release set **and** to allow `release_derivative_text`.31- "Evaluate" for tier C means internal analysis only. Tier-C text never enters a released benchmark.3233**Consequences for the trained model and the benchmark.** Weights trained on NC or SA material inherit34uncertainty about the licence terms that apply to the weights. Two releases are planned:3536- **Release-only weights:** a model trained only on CC0, PUBLIC-DOMAIN and CC-BY data, publishable after37  review.38- **Research weights:** a model trained on the full training set, released only after the author and legal39  review decide.4041The benchmark redistributes only source and derivative text whose code is in the release set. Rewrites of NC/SA42originals are derivatives and follow their source's terms.4344## 2. How licence evidence is captured (per source)4546`normalize_license(code, uri)` maps the evidence to a canonical code. When both a URI and a free-text code are47available, the URI wins. Each document stores `license{code, uri, evidence, raw, tier, permitted}`. If the48document's own statement disagrees with the provider field, the document statement is used and the mismatch is49written to `notes` (`ingest/corpus.py`).5051| Source | Evidence 1 | Evidence 2 / date evidence | Status |52|---|---|---|---|53| PMC OA (AWS `pmc-oa-opendata`) | per-article JSON `license_code`, `is_pmc_openaccess`, `is_retracted`, `is_manuscript` (`evidence = pmc_json.license_code`) | JATS `<permissions><license xlink:href>` or `license_ref`, or a public-domain `copyright-statement`; date from JATS `<pub-date>` (ppub → collection → epub …), never from deposit dates | implemented |54| arXiv | OAI-PMH `arXivRaw` `<license>` URI (absent → UNKNOWN; default licence → ARXIV-NONEXCLUSIVE, excluded) | v1 submission date < 2018-01-01; the e-print of the latest version submitted before 2018 is requested | in progress (`ingest/arxiv_oai.py`) |55| Crossref (econ/finance journals) | `license[].URL` with `content-version` (VoR vs AM) and `start` date; planned rule: the licence must apply to the VoR (or be unrestricted by content-version) | issued / published-print date (planned) | in progress (`ingest/crossref.py`, `configs/data/crossref_journals.json` not yet present) |56| Project Gutenberg | catalog `pg_catalog.csv` (sha256 recorded) + public-domain header check (`gutenberg_header_status`) | front-matter year (imprint, copyright, published, roman numerals); else the latest contributor death year (upper bound, flagged) | implemented (`ingest/books.py`) |57| NCBI Bookshelf | BITS `<permissions>` per book; per the archive's `license.txt`, a book with no licence and no copyright statement is public domain | `file_list.txt` year + XML date | implemented (trial) |58| OpenStax/CNX via Internet Archive | CC statement inside the PDF | in-file date only ("PDF generated", "revised", ©); IA upload time is never used as date evidence | implemented; 0 of 8 accepted in the trial (3 licence unverified, 1 undated, 4 too large) |5960**Policy follow-up.** In `_CODE_PATTERNS`, the free-text pattern `public domain` maps to `CC0`. Only the exact61strings `PUBLIC-DOMAIN` and `PUBLIC DOMAIN` map to `PUBLIC-DOMAIN`. Both codes are tier A, so permissions are62unaffected, but provenance reporting is imprecise. Fix this in `policy.py`.6364## 3. Observed licence distributions (PMC discovery)6566`by_license` counts **every metadata object found**, before the open-access, retraction, manuscript and licence67filters are applied (`ingest/pmc_s3.py`).6869| Manifest | Candidates | Found | CC-BY | CC-BY-NC | CC-BY-NC-SA | CC0 | TDM | UNKNOWN | CC-BY-NC-ND | CC-BY-ND | Not OA | Manuscript | Retracted | Licence excluded | Fetched |70|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|71| `discover_pmc_seed1.json` | 347 | 147 | 75 | 13 | 11 | 1 | 31 | 12 | 4 | 0 | 30 | 1 | 0 | 16 | 100 |72| `discover_pmc_seed2.json` | 4 187 | 1 820 | 842 | 210 | 126 | 26 | 405 | 137 | 73 | 1 | 395 | 14 | 1 | 210 | 1 200 |7374**Seed 2 shares of found objects.**7576| Licence group | Share |77|---|---|78| CC-BY | 46.3 % |79| TDM | 22.3 % |80| CC-BY-NC | 11.5 % |81| UNKNOWN | 7.5 % |82| CC-BY-NC-SA | 6.9 % |83| CC-BY-NC-ND | 4.0 % |84| CC0 | 1.4 % |85| CC-BY-ND | 0.05 % |86| **Training-eligible** | **66.2 %** |87| **Release-eligible** | **47.7 %** |8889**Kept after parsing.** The pilot (`data/parsed/pilot_pmc/build_summary.json`, 1 180 documents) kept the90following licences:9192| Licence | Documents |93|---|---|94| CC-BY | 833 |95| CC-BY-NC | 198 |96| CC-BY-NC-SA | 124 |97| PUBLIC-DOMAIN | 19 |98| CC0 | 6 |99100Release-eligible documents: 858 of 1 180 (72.7 %).101102**Where the build disagrees with the provider.** The 19 PUBLIC-DOMAIN documents and the 3 build-time licence103exclusions come from JATS statements that override the provider code.104105**Books.** Gutenberg smoke: 11 PUBLIC-DOMAIN. Bookshelf trial: 12 PUBLIC-DOMAIN and 5 CC-BY chapters; 1 book106with an UNKNOWN licence was excluded.107108## 4. Provider and platform terms109110- **Anthropic outputs (rewrites).** Under Anthropic's commercial terms as we understand them, the customer owns111  the outputs. Anthropic's usage policy also restricts using outputs to develop models that compete with112  Anthropic. **Risk to confirm, not a conclusion:** whether training a small, task-specific reconstruction model113  on Claude rewrites, and publishing that model, falls within this restriction must be checked against the114  current terms and with UQO before any public weight release. Until then:115  - rewrites stay internal (they are never committed, per `.gitignore`);116  - no rewrite text is redistributed;117  - the release decision is recorded in `docs/DECISION_LOG.md`.118119  RQ4 cross-provider tests use only outputs whose terms permit this use (mandate Phase 6).120- **NLM/PMC.** Use the official bulk channel, the AWS Open Data bucket. The FTP packages and the OA Web Service121  were retired in August 2026, according to the docstring of `ingest/pmc_s3.py`. Acknowledge NLM/PMC as the122  source. Redistribute only articles whose own licence permits it. Respect the per-article licence, not "open123  access" status.124- **arXiv.** Use OAI-PMH and e-print access within arXiv's bulk-access guidance. The default arXiv licence is125  excluded.126- **Gutenberg / NCBI / Internet Archive.** Use the catalog feeds and mirror (not the website), the FTP bulk127  channel (the HTTPS robots file disallows) and in-file dates. Keep request intervals ≥ 2 s per host128  (`ingest/books.py`).129130## 5. Attribution procedure131132Every released item (benchmark row or model card data statement) carries:133134- title, authors, venue, year, DOI or source id, source URL;135- the licence code and URI;136- the statement "modified: segmented; rewritten by an LLM; reconstructed by HTR", as applicable.137138The benchmark ships an `ATTRIBUTION.tsv` generated from `documents.jsonl`, plus the NLM/PMC acknowledgement. NC139and SA items, if ever released, ship in separate bundles under their own terms.140141## 6. Takedown and retraction procedure1421431. Takedown requests go to the corresponding author (address in the model card).1442. Within 5 working days: mark the `doc_id` as withdrawn in a committed `data/manifests/withdrawn.json`, remove145   its passages and pairs from every release bundle, and record the action in the decision log.1463. Retractions: the PMC `is_retracted` flag is checked at acquisition. Before each release, all PMC ids are147   re-queried and retracted items are withdrawn the same way.1484. If a withdrawn document was in a released model's training data, the model card's next version lists it. A149   retrain is considered if memorisation tests show verbatim recall of the document.150