| OpenAlex works snapshot, release 2026-10-14 |
data/jsonl/works/ 1,898 part files, 464,765,485 works, 675.3 GB gz |
2026-11-02 21:19 → 23:05 ET, HTTPS from openalex.s3.amazonaws.com, 8 parallel streams |
filtered CS subset works/cs/*.jsonl.zst 20 GB (15,490,159 records), slim table works/slim/*.parquet 9.3 GB (all works) |
manifest (works_manifest_2026-10-14.json), per-file content length and SHA-256 in works/ledger.sqlite (1,898/1,898 ok, 0 parse errors, record total = manifest total) |
| OpenAlex works snapshot, second pass (id set) |
the same 1,898 part files re-scanned for the 1,116,606 OpenAlex ids of DBLP-/arXiv-linked works outside the CS-topic filter (data/metadata/needed_oa_ids.csv) |
2026-11-03 15:55 → 17:55 ET (interrupted twice by node freezes, completed on M5M36 with 10 workers) |
works_pass2/cs/*.jsonl.zst, 1,102,807 records (98.8 % of the ids; the remainder are ids merged or deleted between the slim table and the record scan) |
works_pass2/ledger.sqlite (1,898/1,898 ok, SHA-256 per file) |
| OpenAlex entity snapshots |
topics 4,516; subfields 252; fields 26; domains 4; sources 268,629; institutions 143,455; publishers 10,713; funders 41,799; concepts 65,026 |
2026-11-02 |
0.6 GB |
entities_ledger.json (SHA-256 per file) |
| OpenAlex API increment |
works with any CS topic, publication_date 2026-10-01 → 2026-11-02, sliced by day × (CS primary / non-CS primary), basic paging ≤ 10,000 per slice |
2026-11-03 (public budget $0.10/day → several days) |
raw JSON pages api_increment/pages/ |
api_increment/ledger.json (counts, cost) |
| DBLP |
dblp.xml.gz (2026-11-02 22:39 UTC), dblp.dtd, .md5 |
2026-11-02 |
1.12 GB |
MD5 ff1c424c3b4cef36582f647c36df27c0 verified; parse report dblp_parse_report.json (8,840,866 records) |
| arXiv OAI-PMH |
set cs, metadataPrefix arXiv, datestamp 2018-01-01 → 2026-11-02, monthly windows, raw XML pages kept |
2026-11-02/03 |
758 pages, 915,115 unique records |
harvest_state.json, arxiv_parse_report.json |
| ACL Anthology |
anthology+abstracts.bib.gz |
2026-11-02 |
46.7 MB, 138,796 entries |
SHA-256 44c3f7e907f221b740bea09069818a3d7c0dfd88b2da3b750d9f02955efc58b3 |