WP44: pass 2 complete (1,102,807 records), provenance note
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 changed file +1 −0
modified
data/raw/README.md
+1 −0
@@ -3,6 +3,7 @@ | ||
| 3 | 3 | | Source | What | Obtained | Size | Integrity | |
| 4 | 4 | |---|---|---|---|---| |
| 5 | 5 | | OpenAlex works snapshot, release 2026-10-14 | `data/jsonl/works/` 1,898 part files, 464,765,485 works, 675.3 GB gz | 2026-11-02 21:19 → 23:05 ET, HTTPS from `openalex.s3.amazonaws.com`, 8 parallel streams | filtered CS subset `works/cs/*.jsonl.zst` 20 GB (15,490,159 records), slim table `works/slim/*.parquet` 9.3 GB (all works) | manifest (`works_manifest_2026-10-14.json`), per-file content length and SHA-256 in `works/ledger.sqlite` (1,898/1,898 ok, 0 parse errors, record total = manifest total) | |
| 6 | +| OpenAlex works snapshot, second pass (id set) | the same 1,898 part files re-scanned for the 1,116,606 OpenAlex ids of DBLP-/arXiv-linked works outside the CS-topic filter (`data/metadata/needed_oa_ids.csv`) | 2026-11-03 15:55 → 17:55 ET (interrupted twice by node freezes, completed on M5M36 with 10 workers) | `works_pass2/cs/*.jsonl.zst`, 1,102,807 records (98.8 % of the ids; the remainder are ids merged or deleted between the slim table and the record scan) | `works_pass2/ledger.sqlite` (1,898/1,898 ok, SHA-256 per file) | | |
| 6 | 7 | | OpenAlex entity snapshots | topics 4,516; subfields 252; fields 26; domains 4; sources 268,629; institutions 143,455; publishers 10,713; funders 41,799; concepts 65,026 | 2026-11-02 | 0.6 GB | `entities_ledger.json` (SHA-256 per file) | |
| 7 | 8 | | OpenAlex API increment | works with any CS topic, publication_date 2026-10-01 → 2026-11-02, sliced by day × (CS primary / non-CS primary), basic paging ≤ 10,000 per slice | 2026-11-03 (public budget $0.10/day → several days) | raw JSON pages `api_increment/pages/` | `api_increment/ledger.json` (counts, cost) | |
| 8 | 9 | | DBLP | `dblp.xml.gz` (2026-11-02 22:39 UTC), `dblp.dtd`, `.md5` | 2026-11-02 | 1.12 GB | MD5 `ff1c424c3b4cef36582f647c36df27c0` verified; parse report `dblp_parse_report.json` (8,840,866 records) | |
| 9 | 10 | |