WP44: Phase 1 corpus report (v2)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2 changed files +44 −2
modified
PROJECT_STATUS.md
+2 −2
@@ -3,8 +3,8 @@ | ||
| 3 | 3 | | Phase | Status | Evidence | |
| 4 | 4 | |---|---|---| |
| 5 | 5 | | 0 Feasibility & protocol | done 2026-11-02 (PI checkpoint H-0 pending) | `reports/PHASE0_FEASIBILITY.md`, `RESEARCH_PLAN.md`, `protocol/PROTOCOL.md`, `protocol/amendments.csv` (A-000…A-004) | |
| 6 | −| 1 Bibliographic corpus | v1 done (6.40 M eligible core works from pass 1); pass 2 (1.12 M DBLP/arXiv-linked ids) ≈ 75 % when M3U96b froze (incident I-4) — resumes automatically, then `ingestion/run_pass2_chain.sh` builds corpus v2 | `data/metadata/prisma_counts.json`, `linkage_report.json`, `reports/INCIDENTS.md` | | |
| 7 | −| 2 Taxonomy | ontology v1.0 delivered (Agent D); classifier tested (9 ms/work); sharded run armed on M3U96b + M3U96a after corpus v2; validation raters ready (A-003) | `taxonomy/ontology.yaml`, `taxonomy/ONTOLOGY.md`, `taxonomy/VALIDATION.md`, `tests/` (7/7) | | |
| 6 | +| 1 Bibliographic corpus | **done 2026-11-03 18:00** — corpus v2: 16.59 M OpenAlex records loaded, 7,110,289 eligible CS-core works (3.40 M baseline / 3.70 M post), DBLP linked 86.8 %, arXiv linked 97.1 % | `reports/PHASE1_CORPUS.md`, `data/metadata/prisma_counts.json`, `linkage_report.json` | | |
| 7 | +| 2 Taxonomy | ontology v1.0 delivered (Agent D); classification of 14.36 M works running on M3U96b (26 processes, started 18:03); validation raters ready (A-003) | `taxonomy/ontology.yaml`, `taxonomy/ONTOLOGY.md`, `taxonomy/VALIDATION.md`, `tests/` (7/7) | | |
| 8 | 8 | | 3 Knowledge graph | citation graph scripts tested on v1 (165 M references; internal/external edges, field flows); SPECTER2 embeddings running (4.87 M abstracts, 98 shards; M3U96a + M4M64c active, M3U96b/M4M64a shares to resume); semantic k-NN edges and LLM extraction scripts ready (A-004) | `knowledge_graph/01–04`, `embeddings/` | |
| 9 | 9 | | 4 Statistics | volume, collaboration, citation-window, venue, country tables and 12 figures generated on v1; structural-change toolkit tested (ITS, placebo, Bai–Perron, PELT, BOCPD) | `results/`, `figures/`, `econometrics/` | |
| 10 | 10 | | 5 Literature synthesis | 581-entry verified bibliography delivered (Agent B-lit); specialist briefs ready; landmark selection script ready | `manuscript/references.bib`, `literature_synthesis/REFERENCE_MAP.md`, `agents/briefs/` | |
added
reports/PHASE1_CORPUS.md
+42 −0
@@ -0,0 +1,42 @@ | ||
| 1 | +# Phase 1 — Bibliographic corpus (v2, 2026-11-03 18:00 ET) | |
| 2 | + | |
| 3 | +Generated from `data/metadata/prisma_counts.json` and `linkage_report.json` (corpus v2 = snapshot pass 1 + id-set pass 2). | |
| 4 | + | |
| 5 | +## Identification | |
| 6 | +| Source | Records | | |
| 7 | +|---|---| | |
| 8 | +| OpenAlex works loaded (CS-topic filter 2018–2026 + DBLP/arXiv-linked second pass) | 16,592,966 | | |
| 9 | +| DBLP 2018–2026 | 4,347,365 | | |
| 10 | +| arXiv cs created 2018–2026 | 890,852 | | |
| 11 | +| ACL Anthology 2018–2026 | 85,625 | | |
| 12 | + | |
| 13 | +## Linkage | |
| 14 | +| Link | Count | | |
| 15 | +|---|---| | |
| 16 | +| DBLP → OpenAlex by DOI | 3,639,848 | | |
| 17 | +| DBLP → OpenAlex by title+year (+ first author) | 141,062 | | |
| 18 | +| DBLP records linked (distinct) | 3,774,184 (86.8 %) | | |
| 19 | +| DBLP records without any OpenAlex record | 573,181 | | |
| 20 | +| arXiv → OpenAlex (id / DOI / title) | 914,406 / 783 / 1,768 → 864,677 distinct (97.1 %) | | |
| 21 | +| ACL → OpenAlex | 73,054 | | |
| 22 | + | |
| 23 | +## Eligibility (sequential) | |
| 24 | +| Stage | Records | | |
| 25 | +|---|---| | |
| 26 | +| e1_outside_window | 59,453 | | |
| 27 | +| e2_type | 3,994,472 | | |
| 28 | +| e3_xpac | 1,430,851 | | |
| 29 | +| e4_no_title | 1,193 | | |
| 30 | +| e5_duplicate | 114,421 | | |
| 31 | +| e7_generic_deposit | 473,549 | | |
| 32 | +| After exclusions | 10,519,027 | | |
| 33 | +| **CS core — mapping corpus** | **7,110,289** | | |
| 34 | +| CS penumbra (sensitivity) | 3,402,475 | | |
| 35 | +| Strict core (DBLP ∧ primary CS) | 1,832,623 | | |
| 36 | + | |
| 37 | +Periods: baseline 3,404,397, post 3,704,028. Date precision: month 5,703,718, year-only 1,406,571. | |
| 38 | +Membership of the core: DBLP 3,608,016, primary CS 5,322,535, arXiv-linked 802,308, abstracts 5,385,235, with references 4,614,417, retracted (flagged) 17,094. | |
| 39 | +Preprint→published merges: {'arxiv_id': 13175, 'title_year': 84776}. | |
| 40 | +By year: {'2018': 642661, '2019': 695209, '2020': 708854, '2021': 710953, '2022': 699308, '2023': 784990, '2024': 901406, '2025': 1065569, '2026': 901339}. | |
| 41 | + | |
| 42 | +Known limitations: 2026 incomplete (snapshot 2026-10-14; API increment to 2026-11-02 ingested separately); 573 k DBLP records have no OpenAlex record (counted in `dblp_only_counts.csv`, not in the corpus); year-only dates (YYYY-01-01) excluded from monthly series; OpenAlex `created_date` unusable for indexing lag. | |
| 43 | ||