Licence matrix
Version 0.1 · 2026-11-09. The single source of truth is src/htr/licenses/policy.py. This document describes
that policy; if the two ever disagree, the code wins and this file must be corrected. Open access ≠ open
licence. The policy is conservative: a document whose licence is unclear is kept as metadata only.
1. Canonical codes and permitted operations
| Code | Tier | Operations allowed by POLICY |
Conditions (from policy.py) |
Training (default) | Release (default) |
|---|---|---|---|---|---|
CC0 |
A | ingest, parse, train, evaluate, release_source_text, release_derivative_text, commercial_use | attribution recommended | yes | yes |
PUBLIC-DOMAIN |
A | all | verify public-domain status per jurisdiction; US works only unless stated | yes | yes |
CC-BY |
A | all | attribution: title, authors, source, licence URI, changes stated | yes | yes |
CC-BY-SA |
B | all (commercial allowed) | derivative text must be released under CC BY-SA; attribution | yes | no (ShareAlike release only under a separate SA bundle) |
CC-BY-NC |
B | all except commercial_use | non-commercial research use only; derivative text released NC; attribution | yes | no |
CC-BY-NC-SA |
B | all except commercial_use | non-commercial; derivatives CC BY-NC-SA; attribution | yes | no |
CC-BY-ND |
C | ingest, parse, evaluate | no derivative distribution; training disabled by default | no | no |
CC-BY-NC-ND |
C | ingest, parse, evaluate | no derivative distribution; non-commercial; training disabled by default | no | no |
TDM |
C | ingest, parse, evaluate | PMC author manuscripts: text mining only; no redistribution | no | no |
ARXIV-NONEXCLUSIVE |
C | ingest, parse, evaluate | the arXiv licence grants distribution rights to arXiv, not to us; metadata and internal analysis only | no | no |
OTHER-OA |
X | ingest (metadata) | no machine-readable licence | no | no |
UNKNOWN |
X | ingest (metadata) | licence not determined | no | no |
Defaults. The training set is TRAINING_CODES_DEFAULT = CC0, PUBLIC-DOMAIN, CC-BY, CC-BY-SA, CC-BY-NC,
CC-BY-NC-SA. The release set is RELEASE_CODES_DEFAULT = CC0, PUBLIC-DOMAIN, CC-BY.
Eligibility rules.
is_training_eligiblerequires the code to be in the configured set and to allowtrain.is_release_eligiblerequires the code to be in the release set and to allowrelease_derivative_text.- "Evaluate" for tier C means internal analysis only. Tier-C text never enters a released benchmark.
Consequences for the trained model and the benchmark. Weights trained on NC or SA material inherit uncertainty about the licence terms that apply to the weights. Two releases are planned:
- Release-only weights: a model trained only on CC0, PUBLIC-DOMAIN and CC-BY data, publishable after review.
- Research weights: a model trained on the full training set, released only after the author and legal review decide.
The benchmark redistributes only source and derivative text whose code is in the release set. Rewrites of NC/SA originals are derivatives and follow their source's terms.
2. How licence evidence is captured (per source)
normalize_license(code, uri) maps the evidence to a canonical code. When both a URI and a free-text code are
available, the URI wins. Each document stores license{code, uri, evidence, raw, tier, permitted}. If the
document's own statement disagrees with the provider field, the document statement is used and the mismatch is
written to notes (ingest/corpus.py).
| Source | Evidence 1 | Evidence 2 / date evidence | Status |
|---|---|---|---|
PMC OA (AWS pmc-oa-opendata) |
per-article JSON license_code, is_pmc_openaccess, is_retracted, is_manuscript (evidence = pmc_json.license_code) |
JATS <permissions><license xlink:href> or license_ref, or a public-domain copyright-statement; date from JATS <pub-date> (ppub → collection → epub …), never from deposit dates |
implemented |
| arXiv | OAI-PMH arXivRaw <license> URI (absent → UNKNOWN; default licence → ARXIV-NONEXCLUSIVE, excluded) |
v1 submission date < 2018-01-01; the e-print of the latest version submitted before 2018 is requested | in progress (ingest/arxiv_oai.py) |
| Crossref (econ/finance journals) | license[].URL with content-version (VoR vs AM) and start date; planned rule: the licence must apply to the VoR (or be unrestricted by content-version) |
issued / published-print date (planned) | in progress (ingest/crossref.py, configs/data/crossref_journals.json not yet present) |
| Project Gutenberg | catalog pg_catalog.csv (sha256 recorded) + public-domain header check (gutenberg_header_status) |
front-matter year (imprint, copyright, published, roman numerals); else the latest contributor death year (upper bound, flagged) | implemented (ingest/books.py) |
| NCBI Bookshelf | BITS <permissions> per book; per the archive's license.txt, a book with no licence and no copyright statement is public domain |
file_list.txt year + XML date |
implemented (trial) |
| OpenStax/CNX via Internet Archive | CC statement inside the PDF | in-file date only ("PDF generated", "revised", ©); IA upload time is never used as date evidence | implemented; 0 of 8 accepted in the trial (3 licence unverified, 1 undated, 4 too large) |
Policy follow-up. In _CODE_PATTERNS, the free-text pattern public domain maps to CC0. Only the exact
strings PUBLIC-DOMAIN and PUBLIC DOMAIN map to PUBLIC-DOMAIN. Both codes are tier A, so permissions are
unaffected, but provenance reporting is imprecise. Fix this in policy.py.
3. Observed licence distributions (PMC discovery)
by_license counts every metadata object found, before the open-access, retraction, manuscript and licence
filters are applied (ingest/pmc_s3.py).
| Manifest | Candidates | Found | CC-BY | CC-BY-NC | CC-BY-NC-SA | CC0 | TDM | UNKNOWN | CC-BY-NC-ND | CC-BY-ND | Not OA | Manuscript | Retracted | Licence excluded | Fetched |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
discover_pmc_seed1.json |
347 | 147 | 75 | 13 | 11 | 1 | 31 | 12 | 4 | 0 | 30 | 1 | 0 | 16 | 100 |
discover_pmc_seed2.json |
4 187 | 1 820 | 842 | 210 | 126 | 26 | 405 | 137 | 73 | 1 | 395 | 14 | 1 | 210 | 1 200 |
Seed 2 shares of found objects.
| Licence group | Share |
|---|---|
| CC-BY | 46.3 % |
| TDM | 22.3 % |
| CC-BY-NC | 11.5 % |
| UNKNOWN | 7.5 % |
| CC-BY-NC-SA | 6.9 % |
| CC-BY-NC-ND | 4.0 % |
| CC0 | 1.4 % |
| CC-BY-ND | 0.05 % |
| Training-eligible | 66.2 % |
| Release-eligible | 47.7 % |
Kept after parsing. The pilot (data/parsed/pilot_pmc/build_summary.json, 1 180 documents) kept the
following licences:
| Licence | Documents |
|---|---|
| CC-BY | 833 |
| CC-BY-NC | 198 |
| CC-BY-NC-SA | 124 |
| PUBLIC-DOMAIN | 19 |
| CC0 | 6 |
Release-eligible documents: 858 of 1 180 (72.7 %).
Where the build disagrees with the provider. The 19 PUBLIC-DOMAIN documents and the 3 build-time licence exclusions come from JATS statements that override the provider code.
Books. Gutenberg smoke: 11 PUBLIC-DOMAIN. Bookshelf trial: 12 PUBLIC-DOMAIN and 5 CC-BY chapters; 1 book with an UNKNOWN licence was excluded.
4. Provider and platform terms
-
Anthropic outputs (rewrites). Under Anthropic's commercial terms as we understand them, the customer owns the outputs. Anthropic's usage policy also restricts using outputs to develop models that compete with Anthropic. Risk to confirm, not a conclusion: whether training a small, task-specific reconstruction model on Claude rewrites, and publishing that model, falls within this restriction must be checked against the current terms and with UQO before any public weight release. Until then:
- rewrites stay internal (they are never committed, per
.gitignore); - no rewrite text is redistributed;
- the release decision is recorded in
docs/DECISION_LOG.md.
RQ4 cross-provider tests use only outputs whose terms permit this use (mandate Phase 6).
- rewrites stay internal (they are never committed, per
-
NLM/PMC. Use the official bulk channel, the AWS Open Data bucket. The FTP packages and the OA Web Service were retired in August 2026, according to the docstring of
ingest/pmc_s3.py. Acknowledge NLM/PMC as the source. Redistribute only articles whose own licence permits it. Respect the per-article licence, not "open access" status. -
arXiv. Use OAI-PMH and e-print access within arXiv's bulk-access guidance. The default arXiv licence is excluded.
-
Gutenberg / NCBI / Internet Archive. Use the catalog feeds and mirror (not the website), the FTP bulk channel (the HTTPS robots file disallows) and in-file dates. Keep request intervals ≥ 2 s per host (
ingest/books.py).
5. Attribution procedure
Every released item (benchmark row or model card data statement) carries:
- title, authors, venue, year, DOI or source id, source URL;
- the licence code and URI;
- the statement "modified: segmented; rewritten by an LLM; reconstructed by HTR", as applicable.
The benchmark ships an ATTRIBUTION.tsv generated from documents.jsonl, plus the NLM/PMC acknowledgement. NC
and SA items, if ever released, ship in separate bundles under their own terms.
6. Takedown and retraction procedure
- Takedown requests go to the corresponding author (address in the model card).
- Within 5 working days: mark the
doc_idas withdrawn in a committeddata/manifests/withdrawn.json, remove its passages and pairs from every release bundle, and record the action in the decision log. - Retractions: the PMC
is_retractedflag is checked at acquisition. Before each release, all PMC ids are re-queried and retracted items are withdrawn the same way. - If a withdrawn document was in a released model's training data, the model card's next version lists it. A retrain is considered if memorisation tests show verbatim recall of the document.