Phase 0 — Feasibility audit and source verification (2026-11-02, ET evening)
Executing agent: Claude Code (Fable 5.1) acting as Agent A. All numbers below were measured in this session; nothing is assumed.
1. Infrastructure discovered
| Node | Role for WP44 | Measured |
|---|---|---|
| M3U96b (Mac Studio M3 Ultra, 28 cores, 96 GB) | primary ingestion/storage/analytics (~/wp44_uqo) |
375 GB free; load ≈ 2; Python 3.14 system → uv venv Python 3.12; PostgreSQL 16/17 installed; S3 single stream 71 MB/s, 8 streams ≈ 112 MB/s aggregate |
| M3U96a (M3 Ultra, 96 GB) | spare for embeddings / graph | 238 GB free; hosts other apps (14 PM2 processes) |
| M4M64a / M4M64b / M4M64c (M4 Max, 64 GB) | embedding batches | 540 / 52 / 318 GB free; a/b run WP42 executors |
| M2U64, M5M36 and 15 reachable minis/laptops | embarrassingly parallel batches if needed | 8–845 GB free; Python 3.9 system (uv available for venvs) |
| Unreachable at audit time | — | m4ma, m4mb, m4mc, M2M32, M2M32b, M4M36, m2m16 (SSH timeout) |
Decision: run ingestion and database work on M3U96b only (network-bound; distribution would not improve throughput because all nodes share the same WAN uplink, measured ≈ 12 MB/s to dblp.org and ≈ 110 MB/s to AWS S3).
2. Sources verified
| Source | Verification performed | Result |
|---|---|---|
| OpenAlex API | GET /works?filter=primary_topic.field.id:17,publication_year:2018-2026&group_by=publication_year |
6,128,071 works with primary field CS 2018–2026 (2018: 533,455 … 2025: 904,351; 2026: 1,104,438 incl. forward-dated records); 6,550,652 works with any CS topic from 2022-11-30 to 2026-11-02. No API key: $0.10/day shared budget; $1/day with a free key (requires the PI to click an e-mail link — not done without instruction). List+filter costs $0.10 per 1,000 calls; single-record lookups are free. |
| OpenAlex snapshot | data/jsonl/works/manifest.json |
release 2026-10-14, 1,898 files, 675.3 GB gz, 464,765,485 works; CC0 licence (LICENSE.txt); entity manifests: topics 4,516, subfields 252, fields 26, sources 268,629, institutions 143,455, funders 41,799, publishers 10,713, concepts 65,026, keywords 1,881,723 |
| DBLP | dblp.xml.gz 1,116,270,104 bytes dated 2026-11-02 22:39 UTC; MD5 ff1c424c3b4cef36582f647c36df27c0 matches dblp.xml.gz.md5; parsed with lxml + DTD |
8,840,866 publication records (article 4,500,501; inproceedings 3,995,530; phdthesis 158,398; …); 2018–2026: 4,372,487 records (2018 378,942 → 2025 605,696; 2026 partial 446,274); 3,570 venue streams with ≥ 100 records; journals/corr (arXiv) 851,800 |
| arXiv OAI-PMH | Identify, ListSets, ListRecords&set=cs&metadataPrefix=arXiv |
operational at https://oaipmh.arxiv.org/oai; harvest 2018-01 → 2026-11 running at 3.5 s/request with Retry-After handling; raw pages archived |
| ACL Anthology | anthology+abstracts.bib.gz 46,723,382 bytes, SHA-256 44c3f7e9…58b3 |
138,796 entries (inproceedings 132,105; article 3,453; proceedings 3,016; book 222) |
| Crossref REST | works?filter=from-pub-date:2022-11-30&rows=0 |
operational, 38,510,092 works since 2022-11-30 (all fields); used for verification only |
| Semantic Scholar | graph/v1/paper/arXiv:2303.08774 anonymous |
HTTP 429 "apply for a key"; datasets endpoint requires a key → unavailable, recorded |
| Kaggle arXiv snapshot | credentials check | none configured → not used (OAI-PMH suffices) |
| AWS CLI | which aws |
absent; S3 accessed over HTTPS with curl (public bucket) |
3. Scale check against the mandate's planning targets
- Candidate records: OpenAlex CS 2018–2026 (≈ 8–10 M any-topic) + DBLP 4.37 M + arXiv cs (≈ 0.8 M) + ACL 0.14 M → well above 1,000,000 before deduplication.
- Deep classification (≥ 100 k): abstracts are available for most OpenAlex records (test batch: 55,702 of 57,465); feasible for the full corpus.
- High-priority full text (≥ 10 k): arXiv versions cover most highly cited post-2022 CS works; lawful download at 1 request / 3 s ≈ 8–9 h for 10 k papers.
- Storage: filtered CS JSONL ≈ 10–15 GB; slim table of all works ≈ 10 GB; normalised Parquet ≈ 15 GB; embeddings (768-d float16) for 10 M records ≈ 15 GB — all within M3U96b's free space.
4. Pipelines implemented and running (Phase 1 start)
ingestion/openalex_snapshot.py (resumable, SHA-256 per file, slim table of all works, CS filter), openalex_entities.py,
openalex_topics_flat.py, dblp_parse.py, dblp_streams.py, arxiv_oai.py, acl_parse.py, build_corpus.py
(normalised Parquet tables: works, authorships, work_topics, work_keywords, references, abstracts, locations, grants,
counts_by_year; measured 57,465 works in 2.2 s with 4 workers). Logs in M3U96b:~/wp44_uqo/logs/.
5. Risks identified
- Right-truncation: 2026 records are incomplete (OpenAlex indexing lag; DBLP 2026 = 446 k vs 606 k in 2025) → all analyses end the "complete" window in 2026-06 and report 2026-07→11 separately with lag-corrected estimates.
- OpenAlex topic classifier noise and "secondary CS topic" over-inclusion → core/penumbra design (PROTOCOL P2).
- Forward-dated records (publication_year 2026 count > 2025 in the API) →
publication_datesanity filters. - No Semantic Scholar → no citation-intent/influential-citation signals; the knowledge graph relies on OpenAlex references and LLM extraction with provenance.
- LLM-based classification/extraction errors → validated samples with agreement statistics and PI checkpoints.