1# Phase 0 — Feasibility audit and source verification (2026-11-02, ET evening)23Executing agent: Claude Code (Fable 5.1) acting as Agent A. All numbers below were measured in this session; nothing is4assumed.56## 1. Infrastructure discovered78| Node | Role for WP44 | Measured |9|---|---|---|10| M3U96b (Mac Studio M3 Ultra, 28 cores, 96 GB) | primary ingestion/storage/analytics (`~/wp44_uqo`) | 375 GB free; load ≈ 2; Python 3.14 system → uv venv Python 3.12; PostgreSQL 16/17 installed; S3 single stream 71 MB/s, 8 streams ≈ 112 MB/s aggregate |11| M3U96a (M3 Ultra, 96 GB) | spare for embeddings / graph | 238 GB free; hosts other apps (14 PM2 processes) |12| M4M64a / M4M64b / M4M64c (M4 Max, 64 GB) | embedding batches | 540 / 52 / 318 GB free; a/b run WP42 executors |13| M2U64, M5M36 and 15 reachable minis/laptops | embarrassingly parallel batches if needed | 8–845 GB free; Python 3.9 system (uv available for venvs) |14| Unreachable at audit time | — | m4ma, m4mb, m4mc, M2M32, M2M32b, M4M36, m2m16 (SSH timeout) |1516Decision: run ingestion and database work on M3U96b only (network-bound; distribution would not improve throughput17because all nodes share the same WAN uplink, measured ≈ 12 MB/s to dblp.org and ≈ 110 MB/s to AWS S3).1819## 2. Sources verified2021| Source | Verification performed | Result |22|---|---|---|23| OpenAlex API | `GET /works?filter=primary_topic.field.id:17,publication_year:2018-2026&group_by=publication_year` | 6,128,071 works with primary field CS 2018–2026 (2018: 533,455 … 2025: 904,351; 2026: 1,104,438 incl. forward-dated records); 6,550,652 works with *any* CS topic from 2022-11-30 to 2026-11-02. No API key: $0.10/day shared budget; $1/day with a free key (requires the PI to click an e-mail link — not done without instruction). List+filter costs $0.10 per 1,000 calls; single-record lookups are free. |24| OpenAlex snapshot | `data/jsonl/works/manifest.json` | release 2026-10-14, 1,898 files, 675.3 GB gz, 464,765,485 works; CC0 licence (LICENSE.txt); entity manifests: topics 4,516, subfields 252, fields 26, sources 268,629, institutions 143,455, funders 41,799, publishers 10,713, concepts 65,026, keywords 1,881,723 |25| DBLP | `dblp.xml.gz` 1,116,270,104 bytes dated 2026-11-02 22:39 UTC; MD5 `ff1c424c3b4cef36582f647c36df27c0` matches `dblp.xml.gz.md5`; parsed with lxml + DTD | 8,840,866 publication records (article 4,500,501; inproceedings 3,995,530; phdthesis 158,398; …); 2018–2026: 4,372,487 records (2018 378,942 → 2025 605,696; 2026 partial 446,274); 3,570 venue streams with ≥ 100 records; `journals/corr` (arXiv) 851,800 |26| arXiv OAI-PMH | `Identify`, `ListSets`, `ListRecords&set=cs&metadataPrefix=arXiv` | operational at `https://oaipmh.arxiv.org/oai`; harvest 2018-01 → 2026-11 running at 3.5 s/request with Retry-After handling; raw pages archived |27| ACL Anthology | `anthology+abstracts.bib.gz` 46,723,382 bytes, SHA-256 `44c3f7e9…58b3` | 138,796 entries (inproceedings 132,105; article 3,453; proceedings 3,016; book 222) |28| Crossref REST | `works?filter=from-pub-date:2022-11-30&rows=0` | operational, 38,510,092 works since 2022-11-30 (all fields); used for verification only |29| Semantic Scholar | `graph/v1/paper/arXiv:2303.08774` anonymous | HTTP 429 "apply for a key"; datasets endpoint requires a key → **unavailable**, recorded |30| Kaggle arXiv snapshot | credentials check | none configured → not used (OAI-PMH suffices) |31| AWS CLI | `which aws` | absent; S3 accessed over HTTPS with curl (public bucket) |3233## 3. Scale check against the mandate's planning targets3435- Candidate records: OpenAlex CS 2018–2026 (≈ 8–10 M any-topic) + DBLP 4.37 M + arXiv cs (≈ 0.8 M) + ACL 0.14 M → well above 1,000,000 before deduplication.36- Deep classification (≥ 100 k): abstracts are available for most OpenAlex records (test batch: 55,702 of 57,465); feasible for the full corpus.37- High-priority full text (≥ 10 k): arXiv versions cover most highly cited post-2022 CS works; lawful download at 1 request / 3 s ≈ 8–9 h for 10 k papers.38- Storage: filtered CS JSONL ≈ 10–15 GB; slim table of all works ≈ 10 GB; normalised Parquet ≈ 15 GB; embeddings (768-d float16) for 10 M records ≈ 15 GB — all within M3U96b's free space.3940## 4. Pipelines implemented and running (Phase 1 start)4142`ingestion/openalex_snapshot.py` (resumable, SHA-256 per file, slim table of all works, CS filter), `openalex_entities.py`,43`openalex_topics_flat.py`, `dblp_parse.py`, `dblp_streams.py`, `arxiv_oai.py`, `acl_parse.py`, `build_corpus.py`44(normalised Parquet tables: works, authorships, work_topics, work_keywords, references, abstracts, locations, grants,45counts_by_year; measured 57,465 works in 2.2 s with 4 workers). Logs in `M3U96b:~/wp44_uqo/logs/`.4647## 5. Risks identified48491. Right-truncation: 2026 records are incomplete (OpenAlex indexing lag; DBLP 2026 = 446 k vs 606 k in 2025) → all50 analyses end the "complete" window in 2026-06 and report 2026-07→11 separately with lag-corrected estimates.512. OpenAlex topic classifier noise and "secondary CS topic" over-inclusion → core/penumbra design (PROTOCOL P2).523. Forward-dated records (publication_year 2026 count > 2025 in the API) → `publication_date` sanity filters.534. No Semantic Scholar → no citation-intent/influential-citation signals; the knowledge graph relies on OpenAlex references and LLM extraction with provenance.545. LLM-based classification/extraction errors → validated samples with agreement statistics and PI checkpoints.55