WP44: phase 0 (feasibility, plan, protocol) + phase 1 ingestion pipelines

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 6 h ago (Nov 3, 2026) Browse files

28 changed files +10,294 −0

added .gitignore +54 −0

		`@@ -0,0 +1,54 @@`
	1	+# Raw & processed data — never committed (see data/raw/README.md for provenance)
	2	+data/raw/*
	3	+!data/raw/README.md
	4	+data/processed/*
	5	+!data/processed/.gitkeep
	6	+
	7	+# Python
	8	+__pycache__/
	9	+*.pyc
	10	+.pytest_cache/
	11	+.venv/
	12	+*.egg-info/
	13	+
	14	+# LaTeX build artifacts (paper/main.pdf IS committed)
	15	+paper/*.aux
	16	+paper/*.log
	17	+paper/*.out
	18	+paper/*.toc
	19	+paper/*.lof
	20	+paper/*.lot
	21	+paper/*.fls
	22	+paper/*.fdb_latexmk
	23	+paper/*.bbl
	24	+paper/*.blg
	25	+paper/*.spl
	26	+paper/*.synctex.gz
	27	+paper/*.run.xml
	28	+paper/*.bcf
	29	+
	30	+# Logs
	31	+logs/
	32	+*.log
	33	+
	34	+# macOS
	35	+.DS_Store
	36	+
	37	+# WP44 specifics
	38	+data/metadata/*.sqlite
	39	+logs/
	40	+*.jsonl.zst
	41	+*.parquet
	42	+!data/metadata/*/.parquet
	43	+manuscript/*.aux
	44	+manuscript/*.log
	45	+manuscript/*.out
	46	+manuscript/*.toc
	47	+manuscript/*.lof
	48	+manuscript/*.lot
	49	+manuscript/*.fls
	50	+manuscript/*.fdb_latexmk
	51	+manuscript/*.bbl
	52	+manuscript/*.blg
	53	+manuscript/*.synctex.gz
	54	+.venv/

added CLAUDE.md +527 −0

		`@@ -0,0 +1,527 @@`
	1	+# CLAUDE.md — UQO Working Paper 44
	2	+
	3	+> Mandate pasted verbatim by the principal investigator on 2026-11-02 ("fait ce projet rigoureusement"). The paste was
	4	+> truncated by the terminal inside §14 (after "The corpus is documented and reproduc"); the executing agent completed §14
	5	+> from the acceptance criteria implied by §1–§13 and recorded the completion in `protocol/amendments.csv` (A-000).
	6	+> Operational decisions are in `RESEARCH_PLAN.md`; progress in `PROJECT_STATUS.md`.
	7	+
	8	+## Project Identity
	9	+
	10	+Working Paper: WP44 — UQO Working Paper Series
	11	+
	12	+Proposed Title: The Post-ChatGPT Computer Science Revolution: A Comprehensive Systematic Mapping, Bibliometric Analysis, and Knowledge Graph of Computer Science Research, 2022–2026
	13	+
	14	+Research period: November 30, 2022 – November 2, 2026
	15	+
	16	+Historical baseline: January 1, 2018 – November 29, 2022
	17	+
	18	+Primary language: English
	19	+
	20	+Target manuscript: 300–500 pages, including extensive technical appendices.
	21	+
	22	+Scientific objective: Produce one of the most comprehensive, rigorously documented, reproducible computational mappings of contemporary computer science research.
	23	+
	24	+The ambition is to build an original scientific contribution, not merely a long literature summary.
	25	+
	26	+Never claim that this is the largest review ever conducted without independently establishing that claim.
	27	+
	28	+---
	29	+
	30	+# 1. NON-NEGOTIABLE SCIENTIFIC REQUIREMENTS
	31	+
	32	+The project must:
	33	+
	34	+1. Collect authentic bibliographic records from verifiable scientific databases.
	35	+2. Build a unified, deduplicated corpus with persistent identifiers.
	36	+3. Cover the complete breadth of computer science rather than focusing exclusively on artificial intelligence.
	37	+4. Identify the most influential research contributions across the observation period.
	38	+5. Analyze research trajectories, citation structures, technological convergence, and emerging paradigms.
	39	+6. Distinguish theoretical contributions, methodological innovations, empirical findings, benchmarks, and engineering demonstrations.
	40	+7. Critically evaluate conflicting findings and limitations.
	41	+8. Produce an evidence-linked scientific manuscript.
	42	+9. Make the methodology reproducible.
	43	+10. Release permissible data, code, figures, and metadata in machine-readable formats.
	44	+
	45	+Strict integrity rules:
	46	+
	47	+- Never fabricate references, DOI identifiers, citation counts, experiments, or results.
	48	+- Never invent paper content when only metadata is available.
	49	+- Never treat an abstract as equivalent to a full-text review.
	50	+- Never classify unverified model-generated summaries as scientific evidence.
	51	+- Never interpret temporal association as causation.
	52	+- Never describe an unpublished preprint as peer-reviewed.
	53	+- Never fabricate statistical significance.
	54	+- Record unavailable or inaccessible data explicitly.
	55	+- Maintain human-review checkpoints for scientific conclusions.
	56	+
	57	+# 2. PROJECT SCALE
	58	+
	59	+Initial engineering targets:
	60	+
	61	+- 1,000,000+ candidate bibliographic records, subject to actual availability.
	62	+- 100,000+ deeply classified research records.
	63	+- 10,000+ high-priority papers for detailed evidence extraction, subject to lawful full-text access.
	64	+- 1,000+ landmark papers for intensive critical synthesis.
	65	+- 30+ computer science research categories.
	66	+- 100+ publication-quality figures.
	67	+- 100+ analytical tables.
	68	+- 300–500 manuscript pages including appendices.
	69	+
	70	+These are planning targets, not guaranteed outcomes.
	71	+
	72	+Corpus size must never take precedence over validity, relevance, or quality.
	73	+
	74	+# 3. SCIENTIFIC DATA SOURCES
	75	+
	76	+Prioritize:
	77	+
	78	+- DBLP
	79	+- OpenAlex
	80	+- Semantic Scholar Academic Graph
	81	+- arXiv
	82	+- Crossref
	83	+- ACL Anthology
	84	+- Open-access conference proceedings
	85	+- Institutional repositories
	86	+- Publicly available benchmark repositories
	87	+
	88	+Before ingestion, verify current API specifications, rate limits, licensing, authentication, bulk-access policies, and redistribution permissions.
	89	+
	90	+Prefer official snapshots and documented bulk datasets for large-scale ingestion.
	91	+
	92	+Never bypass access controls or publisher restrictions.
	93	+
	94	+# 4. RESEARCH TAXONOMY
	95	+
	96	+Build a hierarchical taxonomy including:
	97	+
	98	+1. Large Language Models
	99	+2. Foundation Models
	100	+3. Multimodal AI
	101	+4. Generative AI
	102	+5. AI Agents and Autonomous Systems
	103	+6. Reinforcement Learning
	104	+7. Machine Learning Theory
	105	+8. Deep Learning Architectures
	106	+9. Computer Vision
	107	+10. Natural Language Processing
	108	+11. Speech and Audio Computing
	109	+12. Robotics
	110	+13. Human-Computer Interaction
	111	+14. Software Engineering
	112	+15. Programming Languages
	113	+16. Compilers
	114	+17. Formal Verification
	115	+18. Algorithms and Data Structures
	116	+19. Computational Complexity
	117	+20. Database Systems
	118	+21. Information Retrieval
	119	+22. Distributed Computing
	120	+23. Cloud Computing
	121	+24. Edge Computing
	122	+25. Computer Networks
	123	+26. Cybersecurity
	124	+27. Applied Cryptography
	125	+28. Blockchain Systems
	126	+29. Computer Architecture
	127	+30. Accelerators and AI Hardware
	128	+31. Operating Systems
	129	+32. Quantum Computing
	130	+33. Scientific Computing
	131	+34. Computational Biology
	132	+35. Computer Graphics
	133	+36. Privacy-Preserving Computation
	134	+37. Federated Learning
	135	+38. AI Safety and Alignment
	136	+39. Sustainable Computing
	137	+40. Autonomous Scientific Discovery
	138	+
	139	+Allow papers to belong to multiple categories.
	140	+
	141	+Construct an extensible hierarchical ontology rather than a flat keyword classification.
	142	+
	143	+# 5. SCIENTIFIC QUESTIONS
	144	+
	145	+The paper must investigate:
	146	+
	147	+RQ1: How has the volume and composition of computer science research evolved since November 2022?
	148	+
	149	+RQ2: Which subfields have experienced the strongest acceleration or structural change?
	150	+
	151	+RQ3: Which publications introduced the most influential methodological or theoretical contributions?
	152	+
	153	+RQ4: How have citation networks and interdisciplinary knowledge flows evolved?
	154	+
	155	+RQ5: Has the emergence of foundation models changed the relationship between previously distinct computer science disciplines?
	156	+
	157	+RQ6: How have research institutions, countries, and collaboration networks changed?
	158	+
	159	+RQ7: Which research areas demonstrate genuine breakthroughs versus incremental benchmark improvements?
	160	+
	161	+RQ8: Which scientific problems remain unresolved despite rapid publication growth?
	162	+
	163	+RQ9: How have reproducibility, open-source availability, and evaluation standards evolved?
	164	+
	165	+RQ10: Can computational models detect emerging scientific paradigms before they become dominant?
	166	+
	167	+# 6. RESEARCH METHODOLOGY
	168	+
	169	+Implement multiple complementary analytical approaches.
	170	+
	171	+## 6.1 Systematic Mapping
	172	+
	173	+Follow a preregistered protocol where practical.
	174	+
	175	+Use PRISMA 2020 reporting principles where applicable.
	176	+
	177	+Document:
	178	+
	179	+- Search strategies
	180	+- Database coverage
	181	+- Eligibility criteria
	182	+- Deduplication
	183	+- Screening decisions
	184	+- Full-text availability
	185	+- Exclusion reasons
	186	+- Quality assessment
	187	+- Final synthesis populations
	188	+
	189	+Separate broad bibliometric mapping from focused systematic reviews.
	190	+
	191	+## 6.2 Bibliometric Analysis
	192	+
	193	+Estimate:
	194	+
	195	+- Publication growth
	196	+- Citation growth
	197	+- Field-normalized impact
	198	+- Citation velocity
	199	+- Collaboration networks
	200	+- Institutional influence
	201	+- Country-level participation
	202	+- Venue concentration
	203	+- Citation inequality
	204	+- Research topic diversification
	205	+
	206	+Account for citation-age bias, database coverage, and preprint duplication.
	207	+
	208	+## 6.3 Semantic Research Mapping
	209	+
	210	+Use scientific document embeddings, topic modeling, hierarchical clustering, and temporal semantic analysis.
	211	+
	212	+Compare multiple embedding approaches and validate topic stability.
	213	+
	214	+Construct a dynamic semantic atlas of computer science.
	215	+
	216	+## 6.4 Scientific Knowledge Graph
	217	+
	218	+Create nodes for:
	219	+
	220	+- Papers
	221	+- Authors
	222	+- Institutions
	223	+- Venues
	224	+- Research topics
	225	+- Algorithms
	226	+- Models
	227	+- Datasets
	228	+- Benchmarks
	229	+- Scientific claims
	230	+
	231	+Create evidence-supported relationships including:
	232	+
	233	+- Cites
	234	+- Extends
	235	+- Evaluates
	236	+- Contradicts
	237	+- Reproduces
	238	+- Uses dataset
	239	+- Introduces method
	240	+- Addresses research problem
	241	+
	242	+Every inferred relationship must include provenance, confidence, and verification status.
	243	+
	244	+## 6.5 Structural Change Analysis
	245	+
	246	+Use interrupted time-series models, changepoint detection, dynamic network models, and appropriate robustness checks.
	247	+
	248	+The November 2022 boundary is a historical reference point, not proof of a causal ChatGPT effect.
	249	+
	250	+Include pre-period trend diagnostics, placebo breakpoints, alternative event dates, and sensitivity analysis.
	251	+
	252	+## 6.6 Emerging Research Detection
	253	+
	254	+Develop an original computational framework for identifying emerging scientific fields.
	255	+
	256	+Candidate signals:
	257	+
	258	+- Abnormal publication growth
	259	+- Semantic novelty
	260	+- Citation acceleration
	261	+- Interdisciplinary connectivity
	262	+- Methodological adoption
	263	+- Benchmark proliferation
	264	+- Open-source ecosystem activity
	265	+
	266	+Evaluate the framework retrospectively using temporally separated training and test periods.
	267	+
	268	+Avoid look-ahead bias.
	269	+
	270	+# 7. AUTONOMOUS RESEARCH AGENTS
	271	+
	272	+Use Claude Code subagents with explicit responsibilities.
	273	+
	274	+Agent A — Research Director: Coordinates the scientific program and validates deliverables.
	275	+
	276	+Agent B — Bibliographic Acquisition: Collects metadata through legitimate APIs and snapshots.
	277	+
	278	+Agent C — Data Engineering: Builds normalized databases and reproducible ingestion pipelines.
	279	+
	280	+Agent D — Taxonomy Architect: Constructs the computer science ontology.
	281	+
	282	+Agent E — AI Research Specialist: Reviews AI, machine learning, foundation models, and agents.
	283	+
	284	+Agent F — Systems Specialist: Reviews distributed systems, operating systems, databases, and hardware.
	285	+
	286	+Agent G — Theory Specialist: Reviews algorithms, complexity, formal methods, and programming languages.
	287	+
	288	+Agent H — Security Specialist: Reviews cybersecurity, privacy, and cryptography.
	289	+
	290	+Agent I — Bibliometrician: Conducts citation and collaboration analyses.
	291	+
	292	+Agent J — Econometrician: Designs temporal models, statistical inference, and robustness tests.
	293	+
	294	+Agent K — Scientific Critic: Challenges unsupported claims and methodological weaknesses.
	295	+
	296	+Agent L — Academic Writer: Produces publication-quality scientific prose.
	297	+
	298	+Agent M — LaTeX Engineer: Manages equations, references, tables, figures, and compilation.
	299	+
	300	+Agent N — Reproducibility Auditor: Independently validates computations and evidence.
	301	+
	302	+Agent O — Visualization Specialist: Creates publication-quality research graphics.
	303	+
	304	+Agents must exchange structured artifacts, not undocumented conversational summaries.
	305	+
	306	+No agent may approve its own unverified scientific findings.
	307	+
	308	+# 8. MACLUSTR INFRASTRUCTURE
	309	+
	310	+Discover the available compute nodes, CPU, GPU, memory, storage, networking, and existing services before implementation.
	311	+
	312	+Use distributed execution only when it materially improves throughput.
	313	+
	314	+Suggested stack:
	315	+
	316	+- Python
	317	+- DuckDB
	318	+- PostgreSQL
	319	+- Parquet and Apache Arrow
	320	+- Polars
	321	+- NetworkX or igraph
	322	+- Scientific embedding models
	323	+- Vector search
	324	+- PyTorch where appropriate
	325	+- LaTeX
	326	+- Git
	327	+- Containerized reproducible pipelines
	328	+
	329	+Use incremental ingestion, checkpointing, resumable processing, and content hashes.
	330	+
	331	+Preserve immutable raw-source snapshots.
	332	+
	333	+Implement job-level provenance, retry logic, and resource monitoring.
	334	+
	335	+Do not assume access to credentials or services before testing them.
	336	+
	337	+# 9. MANUSCRIPT STRUCTURE
	338	+
	339	+Produce the following major parts.
	340	+
	341	+Part I — Introduction and Scientific Motivation
	342	+
	343	+Explain the research problem, historical context, contributions, and scope.
	344	+
	345	+Part II — The Evolution of Computer Science
	346	+
	347	+Provide the conceptual and historical background needed to interpret post-2022 developments.
	348	+
	349	+Part III — Data and Systematic Review Protocol
	350	+
	351	+Explain corpus construction, source coverage, deduplication, eligibility, and research integrity.
	352	+
	353	+Part IV — Computational Methodology
	354	+
	355	+Develop the mathematical framework for bibliometrics, semantic mapping, networks, and structural-change analysis.
	356	+
	357	+Part V — The Artificial Intelligence Transformation
	358	+
	359	+Critically synthesize foundation models, multimodal learning, agents, reasoning, efficiency, evaluation, and safety.
	360	+
	361	+Part VI — Systems and Computing Infrastructure
	362	+
	363	+Review hardware, distributed systems, databases, networks, and operating systems.
	364	+
	365	+Part VII — Theory, Algorithms, and Security
	366	+
	367	+Review theoretical computer science, formal methods, cryptography, and cybersecurity.
	368	+
	369	+Part VIII — Scientific Knowledge Networks
	370	+
	371	+Analyze authors, institutions, citations, interdisciplinarity, and knowledge diffusion.
	372	+
	373	+Part IX — Empirical Results
	374	+
	375	+Present all verified quantitative findings with publication-quality figures and tables.
	376	+
	377	+Part X — The Next Research Frontiers
	378	+
	379	+Identify unresolved scientific problems, competing hypotheses, and testable research directions.
	380	+
	381	+Part XI — Discussion and Conclusion
	382	+
	383	+Explain contributions, limitations, generalizability, and implications for computer science.
	384	+
	385	+Appendices A–Z
	386	+
	387	+Include detailed taxonomy, search strategies, mathematical derivations, robustness checks, data dictionaries, supplementary tables, figure documentation, and reproducibility instructions.
	388	+
	389	+Do not pad the manuscript to meet page targets.
	390	+
	391	+# 10. ACADEMIC WRITING STANDARD
	392	+
	393	+Write in formal, precise, technically sophisticated academic English.
	394	+
	395	+Target the scholarly standards of leading computer science journals and major survey venues.
	396	+
	397	+The writing must:
	398	+
	399	+- Critically compare competing approaches.
	400	+- Distinguish established findings from speculation.
	401	+- Present meaningful mathematical notation.
	402	+- Explain assumptions and limitations.
	403	+- Integrate citations into the scientific argument.
	404	+- Avoid promotional or exaggerated language.
	405	+- Avoid repetitive AI-generated prose.
	406	+- Include genuine original synthesis.
	407	+
	408	+All references must be resolved to verified bibliographic records.
	409	+
	410	+# 11. FIGURES AND TABLES
	411	+
	412	+Create publication-quality visualizations including:
	413	+
	414	+- Annual and monthly publication trends
	415	+- Research domain growth
	416	+- Dynamic topic maps
	417	+- Citation networks
	418	+- Scientific influence distributions
	419	+- Institutional collaboration networks
	420	+- Country-level research activity
	421	+- Topic emergence and decline
	422	+- Interdisciplinary knowledge flows
	423	+- Methodological genealogy diagrams
	424	+- Structural-break estimates
	425	+- Benchmark evolution
	426	+- Research reproducibility indicators
	427	+- Uncertainty and robustness visualizations
	428	+
	429	+All figures must be generated from reproducible scripts.
	430	+
	431	+Use vector formats when appropriate.
	432	+
	433	+No decorative or fabricated empirical graphics.
	434	+
	435	+# 12. OUTPUT STRUCTURE
	436	+
	437	+Create:
	438	+
	439	+```text
	440	+WP44/
	441	+├── CLAUDE.md
	442	+├── README.md
	443	+├── manuscript/
	444	+│ ├── main.tex
	445	+│ ├── sections/
	446	+│ ├── appendices/
	447	+│ └── references.bib
	448	+├── data/
	449	+│ ├── raw/
	450	+│ ├── processed/
	451	+│ └── metadata/
	452	+├── ingestion/
	453	+├── taxonomy/
	454	+├── embeddings/
	455	+├── knowledge_graph/
	456	+├── bibliometrics/
	457	+├── econometrics/
	458	+├── systematic_review/
	459	+├── literature_synthesis/
	460	+├── figures/
	461	+├── tables/
	462	+├── experiments/
	463	+├── agents/
	464	+├── tests/
	465	+├── reports/
	466	+└── reproducibility/
	467	+```
	468	+
	469	+# 13. EXECUTION PHASES
	470	+
	471	+Phase 0 — Feasibility and Scientific Protocol
	472	+
	473	+Audit available infrastructure, verify data sources, define inclusion criteria, and prepare the research protocol.
	474	+
	475	+Phase 1 — Bibliographic Corpus
	476	+
	477	+Build ingestion pipelines and create the deduplicated research database.
	478	+
	479	+Phase 2 — Scientific Taxonomy
	480	+
	481	+Classify the corpus and validate the ontology using manually reviewed samples.
	482	+
	483	+Phase 3 — Knowledge Graph
	484	+
	485	+Construct citation, semantic, institutional, and methodological relationships.
	486	+
	487	+Phase 4 — Statistical Research
	488	+
	489	+Estimate bibliometric trends, network measures, and temporal models.
	490	+
	491	+Phase 5 — Critical Literature Synthesis
	492	+
	493	+Generate evidence-grounded disciplinary reviews and reconcile conflicting findings.
	494	+
	495	+Phase 6 — Original Scientific Contribution
	496	+
	497	+Develop and validate the emerging-research detection framework.
	498	+
	499	+Phase 7 — Manuscript Construction
	500	+
	501	+Write the full LaTeX paper and comprehensive appendices.
	502	+
	503	+Phase 8 — Independent Scientific Audit
	504	+
	505	+Use independent review agents to verify citations, statistical results, claims, equations, and reproducibility.
	506	+
	507	+Phase 9 — Final Research Release
	508	+
	509	+Compile the final PDF, archive reproducibility artifacts, prepare the dataset documentation, and produce a release-ready research repository.
	510	+
	511	+Do not advance to the next phase until required validation checks pass.
	512	+
	513	+# 14. ACCEPTANCE CRITERIA
	514	+
	515	+The project is complete only when:
	516	+
	517	+- The corpus is documented and reproduc[ible — paste truncated here. Completed by the executing agent (A-000):]
	518	+- The corpus is documented and reproducible from the immutable raw snapshots, with record counts at every PRISMA stage.
	519	+- Every quantitative claim in the manuscript traces to a script, an input artefact and a result file.
	520	+- Every reference in `references.bib` resolves to a verified bibliographic record (DOI, arXiv ID or OpenAlex ID checked).
	521	+- The taxonomy has been validated on a manually reviewed sample with reported agreement statistics.
	522	+- Structural-change results include pre-trend diagnostics, placebo breakpoints and sensitivity analyses.
	523	+- The emerging-research detection framework has been evaluated out-of-time without look-ahead.
	524	+- Independent audit agents have verified citations, statistics, equations and reproducibility, and their reports are archived.
	525	+- The LaTeX manuscript compiles without unresolved references, and the release repository contains code, metadata,
	526	+ figures, tables and documentation under an explicit licence.
	527	+- Human-review checkpoints for scientific conclusions are recorded and any unreviewed conclusion is flagged as such.

added PROJECT_STATUS.md +14 −0

		`@@ -0,0 +1,14 @@`
	1	+# PROJECT_STATUS — WP44 (updated by the executing agent; times ET)
	2	+
	3	+\| Phase \| Status \| Evidence \|
	4	+\|---\|---\|---\|
	5	+\| 0 Feasibility & protocol \| done 2026-11-02 (pending PI checkpoint H-0) \| `reports/PHASE0_FEASIBILITY.md`, `RESEARCH_PLAN.md`, `protocol/PROTOCOL.md` \|
	6	+\| 1 Bibliographic corpus \| in progress — OpenAlex snapshot ingestion running on M3U96b; DBLP parsed; ACL parsed; arXiv harvest running \| `logs/` on M3U96b, `data/metadata/` \|
	7	+\| 2 Taxonomy \| in progress — Agent D building `taxonomy/ontology.yaml` \| `agents/briefs/AGENT_D_TAXONOMY.md` \|
	8	+\| 3 Knowledge graph \| pending \| \|
	9	+\| 4 Statistics \| pending \| \|
	10	+\| 5 Literature synthesis \| pending \| \|
	11	+\| 6 Emerging-research detection \| pending \| \|
	12	+\| 7 Manuscript \| pending \| \|
	13	+\| 8 Independent audit \| pending \| \|
	14	+\| 9 Release \| pending \| \|

added README.md +75 −0

		`@@ -0,0 +1,75 @@`
	1	+<!-- Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca -->
	2	+# 🧭 The Post-ChatGPT Computer Science Revolution
	3	+
	4	+UQO Working Paper No. 44 — A Comprehensive Systematic Mapping, Bibliometric Analysis, and Knowledge Graph of Computer Science Research, 2022–2026
	5	+
	6	+[![Paper](https://img.shields.io/badge/paper-in%20preparation-lightgrey?logo=latex&logoColor=white)](manuscript/)
	7	+[![Python](https://img.shields.io/badge/python-3.12-3776AB?logo=python&logoColor=white)](requirements.txt)
	8	+[![Data](https://img.shields.io/badge/data-OpenAlex%202026--10--14%20·%20DBLP%20·%20arXiv%20·%20ACL-4051b5)](data/raw/README.md)
	9	+[![Reproducible](https://img.shields.io/badge/reproducible-ledgers%20%2B%20hashes-2e7d32)](reproducibility/)
	10	+[![Institution](https://img.shields.io/badge/UQO-D%C3%A9pt.%20des%20sciences%20administratives-16365c)](https://uqo.ca)
	11	+[![Author](https://img.shields.io/badge/author-Simon--Pierre%20Boucher-16365c)](mailto:simon-pierre.boucher@uqo.ca)
	12	+
	13	+> TL;DR — Field-level systematic mapping of computer-science research from 2018-01-01 to 2026-11-02 (baseline
	14	+> 2018 → 2022-11-29; post-ChatGPT window 2022-11-30 → 2026-11-02) built from the full OpenAlex public snapshot
	15	+> (release 2026-10-14; 464.8 M works scanned, 15.49 M with a computer-science topic in 2018–2026), the DBLP dump of
	16	+> 2026-11-02 (8.84 M records), the arXiv `cs` OAI-PMH harvest (915 k records) and the ACL Anthology (139 k entries),
	17	+> linked, deduplicated and classified into a 40-category ontology; bibliometric, network, semantic, structural-change and
	18	+> emerging-topic analyses; knowledge graph with provenance; preregistered protocol (`protocol/PROTOCOL.md`).
	19	+> Status: Phase 1 (corpus) in progress — see `PROJECT_STATUS.md`.
	20	+
	21	+---
	22	+
	23	+## 📌 Status
	24	+
	25	+\| Item \| Value \|
	26	+\|---\|---\|
	27	+\| Version \| 0.1 (November 2026, in preparation) \|
	28	+\| SSRN \| not yet submitted \|
	29	+\| Journal \| — \|
	30	+\| Code & data \| reproducible from `ingestion/` → `bibliometrics/` → `manuscript/` (see Quick start); raw snapshots not redistributed (manifests, ledgers and SHA-256 hashes are) \|
	31	+
	32	+## 🎯 What this paper does
	33	+
	34	+Mandate: `CLAUDE.md` (verbatim). Design: `RESEARCH_PLAN.md`. Protocol: `protocol/PROTOCOL.md` (frozen 2026-11-02;
	35	+amendments in `protocol/amendments.csv`). Feasibility audit: `reports/PHASE0_FEASIBILITY.md`.
	36	+
	37	+## 🗂 Repository layout
	38	+
	39	+```
	40	+wp44/ (spbgit: wp44_uqo.git)
	41	+├── CLAUDE.md · RESEARCH_PLAN.md · PROJECT_STATUS.md · protocol/{PROTOCOL.md,amendments.csv}
	42	+├── ingestion/ openalex_snapshot.py (S3 snapshot → CS JSONL + slim table, SHA-256 ledger), openalex_entities.py,
	43	+│ openalex_api_increment.py, dblp_parse.py, dblp_streams.py, arxiv_oai.py, acl_parse.py,
	44	+│ build_corpus.py (normalised Parquet tables), link_sources.py (record linkage, DuckDB)
	45	+├── taxonomy/ ontology.yaml (40 leaves, 8 areas), topic_map.csv, ONTOLOGY.md, src/ (mapping data), build_ontology.py
	46	+├── embeddings/ · knowledge_graph/ · bibliometrics/ · econometrics/ · systematic_review/ · literature_synthesis/
	47	+├── experiments/ emerging-research detection framework
	48	+├── agents/ briefs/ (one per agent), REGISTER.md (artefacts exchanged, who verified what)
	49	+├── data/ raw/README.md (provenance), metadata/ (manifests, ledgers, PRISMA counts, linkage report), processed/ (not in git)
	50	+├── figures/ · tables/ · reports/ · tests/ · reproducibility/
	51	+└── manuscript/ main.tex, sections/, appendices/, references.bib, figures/, tables/ (mandate §12 layout)
	52	+```
	53	+
	54	+## 🚀 Quick start (compute node M3U96b, `~/wp44_uqo`)
	55	+
	56	+```bash
	57	+uv venv --python 3.12 .venv && . .venv/bin/activate && uv pip install -r requirements.txt
	58	+python ingestion/openalex_snapshot.py --out data/raw/openalex/works --manifest data/raw/openalex/works_manifest_2026-10-14.json --workers 8
	59	+python ingestion/openalex_entities.py --out data/raw/openalex
	60	+python ingestion/dblp_parse.py --xml data/raw/dblp/dblp.xml.gz --out data/raw/dblp
	61	+python ingestion/arxiv_oai.py --out data/raw/arxiv --from 2018-01-01 --until 2026-11-02
	62	+python ingestion/acl_parse.py --bib data/raw/acl/anthology+abstracts.bib.gz --out data/raw/acl
	63	+python ingestion/build_corpus.py --cs-dir data/raw/openalex/works/cs --out data/processed --workers 12
	64	+python ingestion/link_sources.py --root ~/wp44_uqo
	65	+```
	66	+
	67	+## 🗃 Data
	68	+
	69	+See `data/raw/README.md` and `reports/PHASE0_FEASIBILITY.md`. Licences: OpenAlex CC0; DBLP ODC-BY 1.0; arXiv metadata
	70	+(CC0 terms of the OAI-PMH feed); ACL Anthology CC BY 4.0. Semantic Scholar was not used (API key required; recorded).
	71	+
	72	+## 📝 Notes (deviations from the UQO working-paper standard)
	73	+
	74	+- The manuscript lives in `manuscript/` (mandate §12) instead of `paper/`; the title page and preamble follow the UQO template.
	75	+- Phase status and human-review checkpoints: `PROJECT_STATUS.md`, `protocol/PROTOCOL.md` §P7.

added RESEARCH_PLAN.md +131 −0

		`@@ -0,0 +1,131 @@`
	1	+# WP44 — Research plan (operational design, v1.0, 2026-11-02)
	2	+
	3	+This document turns the mandate (`CLAUDE.md`) into the concrete design implemented by the code, the protocol and the
	4	+manuscript. Decisions the mandate left open are listed in §10 with their rationale; they are the executing agent's until
	5	+the principal investigator (PI) confirms them (human-review checkpoint H-0).
	6	+
	7	+## 1. Scientific object and estimands
	8	+
	9	+Population: the computer-science research literature indexed by at least one of DBLP, OpenAlex (any topic in field 17,
	10	+Computer Science), arXiv (set `cs`), or the ACL Anthology, first published between 2018-01-01 and 2026-11-02.
	11	+Two windows: baseline B = [2018-01-01, 2022-11-29] and post P = [2022-11-30, 2026-11-02]. The boundary is the public
	12	+release of ChatGPT; it is a reference date for description and for interrupted-time-series designs, not a causal
	13	+treatment (CLAUDE.md §6.5). Monthly series are the primary temporal unit; years are secondary.
	14	+
	15	+Primary descriptive estimands (RQ1, RQ2): monthly counts of works per category c, N_{c,t}; composition shares
	16	+s_{c,t} = N_{c,t}/N_t; growth rates and their pre/post structural-break statistics; citation-based influence (field- and
	17	+age-normalised); collaboration/interdisciplinarity indices (RQ4–RQ6). Inferential estimands (RQ2, RQ10): breakpoint
	18	+locations and magnitudes with placebo distributions; out-of-time precision/recall of the emerging-topic detector.
	19	+
	20	+## 2. Sources, access and what each contributes (verified 2026-11-02, see `reports/PHASE0_FEASIBILITY.md`)
	21	+
	22	+\| Source \| Access used \| Licence \| Role in the corpus \|
	23	+\|---\|---\|---\|---\|
	24	+\| OpenAlex public snapshot, release 2026-10-14 \| S3 public bucket, JSONL, 1,898 works files (675 GB), ~110 MB/s from M3U96b \| CC0 \| backbone: works, authorships + institutions + countries, venues, topics (field/subfield/topic), `referenced_works` (citation graph), abstracts (inverted index), OA status, FWCI, counts_by_year \|
	25	+\| OpenAlex API (no key, $0.10/day shared budget) \| REST, polite `mailto` \| CC0 \| incremental works 2026-10-14 → 2026-11-02; spot checks; free single-record lookups for reference verification \|
	26	+\| DBLP XML dump (2026-11-02, 1.12 GB gz, MD5 verified) \| HTTPS \| ODC-BY 1.0 \| CS-scope gold standard, venue streams, author PIDs, DOI links; defines "CS venue" \|
	27	+\| arXiv OAI-PMH, set `cs`, 2018-01 → 2026-11 \| OAI-PMH, 3.5 s/request, raw XML kept \| arXiv metadata terms (CC0 for metadata) \| preprints, categories (cs.XX, cross-lists), versions, licences; preprint–publication linking \|
	28	+\| ACL Anthology BibTeX (anthology+abstracts.bib, 138,796 entries) \| HTTPS \| CC BY 4.0 \| NLP venue coverage check, abstracts for ACL papers \|
	29	+\| Crossref REST API (polite pool) \| REST \| open metadata \| reference verification, `is-referenced-by-count` cross-check sample, publisher/type cross-check \|
	30	+\| Semantic Scholar Academic Graph \| not available: API key required for bulk/dataset access; anonymous calls return HTTP 429 \| — \| recorded as unavailable (CLAUDE.md §1: record inaccessible data explicitly); influential-citation and citation-intent signals are therefore not used \|
	31	+\| Papers with Code / benchmark repositories, GitHub \| HTTPS, public dumps where available \| varies \| RQ7/RQ9 (benchmark proliferation, code availability) — scoped in Phase 3 \|
	32	+\| Full texts \| arXiv e-prints (rate-limited, licence-respecting), OA locations from OpenAlex/Unpaywall \| per-paper licence \| ≤ 10,000 high-priority papers; never through paywalls \|
	33	+
	34	+No credential was assumed: the OpenAlex key, Semantic Scholar key and Kaggle credentials were tested as absent.
	35	+
	36	+## 3. Corpus construction (Phase 1) — PRISMA-style stages
	37	+
	38	+S0 Identification: all OpenAlex works with any topic in field 17 and `publication_year` 2018–2026 (full records kept),
	39	+plus DBLP records 2018–2026, arXiv `cs` records created 2018–2026, ACL entries 2018–2026. A slim table of every
	40	+OpenAlex work (465 M rows: id, DOI, year, type, primary topic/subfield/field/domain, cited_by_count, flags) is kept to resolve
	41	+cited works outside CS.
	42	+S1 Record linkage: OpenAlex ↔ DBLP by DOI (lower-cased), else by arXiv ID (DBLP `ee` → arxiv.org/abs), else by normalised
	43	+title + year + first-author surname (blocking on title hash); OpenAlex ↔ arXiv by `locations[].landing_page_url`/`ids`
	44	+and DOI (10.48550/arXiv.*); ACL by DOI/URL. Linkage precision is estimated on a 500-pair manually reviewed sample (H-1).
	45	+S2 Deduplication: one node per work; preprint and published version are merged into a single node with
	46	+`preprint_date`, `publication_date`, `version_relationship` kept (CLAUDE.md §6.2: preprint duplication).
	47	+S3 Eligibility: (i) `publication_year` in window; (ii) type in {article, preprint, book-chapter, proceedings-article,
	48	+dissertation, report, book} — exclude paratext, editorials, errata, retracted (kept with flag `is_retracted`), XPAC
	49	+expansion records (`is_xpac`); (iii) CS membership: DBLP-indexed or OpenAlex primary field = CS or (secondary CS
	50	+topic and arXiv primary category cs.). Secondary-only CS topics without DBLP/arXiv support form a penumbra* set
	51	+reported separately. Sensitivity: results re-estimated on the strict core (DBLP ∩ OpenAlex primary CS).
	52	+S4 Classification: 40-category hierarchical ontology (`taxonomy/ontology.yaml`), multi-label, assigned by rules over
	53	+OpenAlex topics/subfields, arXiv categories, DBLP venue streams and keyword patterns; validated on manually reviewed
	54	+samples (Phase 2, H-2) and by an LLM-based second rater with reported agreement.
	55	+S5 Synthesis populations: (a) mapping corpus (all eligible); (b) deeply classified set (≥ 100 k: abstracts + embeddings +
	56	+full feature vectors); (c) high-priority set (≥ 10 k: citation-ranked within category × year with OA full text);
	57	+(d) landmark set (≥ 1 k: top field-normalised impact, reviewed by specialist agents and flagged for PI review, H-3).
	58	+
	59	+Every stage writes counts to `data/metadata/prisma_counts.json`, rendered as the PRISMA flow figure.
	60	+
	61	+## 4. Taxonomy (Phase 2)
	62	+
	63	+`taxonomy/ontology.yaml`: 40 leaf categories under 8 top-level areas (AI & ML; Vision, Language & Perception; Systems &
	64	+Infrastructure; Theory & Foundations; Security, Privacy & Cryptography; Software & Human Factors; Emerging Computing
	65	+Paradigms; Computing & Science). Each leaf has: definition, parent, OpenAlex topic IDs (weighted), subfield IDs, arXiv
	66	+categories, DBLP streams, keyword regexes with weights, exclusions, and provenance of the mapping. Assignment score per
	67	+category = weighted sum of matched signals; multi-label with threshold; `primary_category` = argmax. Validation: 2,000
	68	+stratified works (50 per category) rated by two independent LLM raters with the rubric in `taxonomy/VALIDATION.md`,
	69	+Cohen's κ and per-category precision/recall; a 200-work subsample is flagged for PI review (H-2).
	70	+
	71	+## 5. Knowledge graph (Phase 3)
	72	+
	73	+Nodes: works, authors (OpenAlex A-ids), institutions, countries, venues (sources), topics, taxonomy categories, arXiv
	74	+categories; method/model/dataset/benchmark and claim nodes only for the high-priority set, extracted from abstracts
	75	+(and full text where lawfully available) with a schema-constrained LLM extractor; each extracted edge carries
	76	+`provenance` (work id + text span), `confidence`, `verification_status` ∈ {unverified, machine-verified, human-verified}.
	77	+Citation edges come from `referenced_works` (verified by construction). Semantic edges from embeddings (k-NN, cosine).
	78	+Storage: Parquet tables + DuckDB views; graph analytics with igraph (python-igraph) on M3U96b (96 GB RAM).
	79	+
	80	+## 6. Statistics (Phase 4) and structural change
	81	+
	82	+- Bibliometrics: monthly/annual counts; shares; CAGR; field-normalised citation score FNCS = c / E[c \| field, year, type];
	83	+ citation velocity (citations within 12/24 months, computed from `counts_by_year`); Gini/Hirsch-type inequality; venue
	84	+ concentration (HHI); author/institution/country collaboration networks; Rao–Stirling and Leinster–Cobbold diversity.
	85	+- Citation-age bias: all cross-period comparisons use fixed citation windows and year-normalisation; FWCI used as a
	86	+ secondary measure with its known limits.
	87	+- Structural change: interrupted time series on log monthly counts with seasonality (OLS-HAC, Newey–West), Bai–Perron
	88	+ multiple-breakpoint estimation, PELT/binary-segmentation changepoints, Bayesian online changepoint detection; pre-trend
	89	+ diagnostics (placebo breaks at every month of the baseline), alternative event dates (2020-05 GPT-3, 2022-04 DALL·E 2,
	90	+ 2022-11 ChatGPT, 2023-03 GPT-4, 2024-09 o1), donut windows; sensitivity to OpenAlex indexing lag (right-truncation
	91	+ corrected by indexing-delay profiles estimated from `created_date`).
	92	+- Dynamic networks: yearly topic co-occurrence and citation-flow matrices between taxonomy categories; modularity and
	93	+ cross-area flow shares; ERGM-style descriptives are out of scope.
	94	+
	95	+## 7. Emerging-research detection (Phase 6, original contribution)
	96	+
	97	+Unit: topic candidates = (i) OpenAlex topics, (ii) semantic clusters (HDBSCAN on embeddings, re-estimated per
	98	+training cutoff without look-ahead), (iii) keyword n-grams. Signals at month t using data ≤ t only: growth anomaly,
	99	+semantic novelty (distance to pre-t centroid), citation acceleration, interdisciplinary connectivity, methodological
	100	+adoption (share of works citing a method node), benchmark proliferation, open-source activity. Label at t+h: top-decile
	101	+growth and citation share at horizon h ∈ {12, 24} months. Training cutoffs 2019–2022; test cutoffs 2023–2024 with
	102	+outcomes observed up to 2026-11. Models: regularised logistic, gradient boosting, and a simple z-score baseline;
	103	+metrics: AUC, precision@k, lead time. Known-emergence validation set: LLMs, diffusion models, NeRF, agents, RAG,
	104	+state-space models, vision-language models, mechanistic interpretability (defined before estimation, listed in
	105	+`protocol/PROTOCOL.md`).
	106	+
	107	+## 8. Agents and artefacts
	108	+
	109	+The executing session is Agent A (Research Director). Subagents are spawned with written briefs (`agents/briefs/`),
	110	+return structured artefacts (JSON/CSV/Markdown with declared schema) and never approve their own findings: every
	111	+scientific artefact is checked by a different agent (K or N) before it enters the manuscript (`agents/REGISTER.md`).
	112	+
	113	+## 9. Infrastructure
	114	+
	115	+Primary compute/storage: M3U96b (M3 Ultra 28 c, 96 GB, 375 GB free; `~/wp44_uqo/`, venv Python 3.12 via uv:
	116	+orjson, pyarrow, polars, duckdb, zstandard, lxml, requests). Secondary: M4M64a (540 GB free) for embeddings; other LAN
	117	+nodes for embarrassingly parallel embedding/extraction batches when it materially helps. Laptop = controller and git
	118	+repository (`Recherche-UQO/informatique/wp44`, origin `gitsrv:srv/git/wp44_uqo.git`). Data are not in git; ledgers,
	119	+manifests, hashes and PRISMA counts are.
	120	+
	121	+## 10. Decisions taken by the executing agent (pending PI confirmation, H-0)
	122	+
	123	+\| # \| Decision \| Rationale \|
	124	+\|---\|---\|---\|
	125	+\| D1 \| Snapshot over API for OpenAlex \| mandate §3 prefers snapshots; the API budget without a key is $0.10/day (≈ 200 k works/day) \|
	126	+\| D2 \| Keep a slim table of all 465 M works \| required for interdisciplinary citation flows (RQ4) and cited-work normalisation \|
	127	+\| D3 \| Semantic Scholar not used \| key required; recorded as unavailable rather than scraped \|
	128	+\| D4 \| CS membership rule (§3 S3) with strict-core sensitivity \| avoids both AI-only bias and over-inclusion from secondary topics \|
	129	+\| D5 \| Manuscript directory `manuscript/` (mandate §12) instead of the UQO `paper/` convention \| mandate is explicit; deviation noted in README \|
	130	+\| D6 \| Historical raw snapshot kept as filtered subset + per-file SHA-256 + manifest, not the 675 GB bucket copy \| public versioned release; hashes allow verification; storage \|
	131	+\| D7 \| LLM raters (Claude) used for taxonomy validation and knowledge-graph extraction, with PI review checkpoints \| mandate §1 requires human checkpoints; PI is the human \|

added agents/briefs/AGENT_D_TAXONOMY.md +77 −0

		`@@ -0,0 +1,77 @@`
	1	+# Brief — Agent D (Taxonomy Architect), WP44
	2	+
	3	+Goal: produce the extensible hierarchical ontology of computer-science research used to classify every work of the WP44
	4	+corpus into the 40 leaf categories of CLAUDE.md §4 (multi-label), with a deterministic mapping from the signals available
	5	+in the corpus. The mapping is data, not code: it must be complete, explicit and justified so that another agent can audit it.
	6	+
	7	+Inputs (read-only):
	8	+- `taxonomy/openalex_topics_2026-10-14.csv` — the 4,516 OpenAlex topics (topic_id, display_name, subfield, field, domain,
	9	+ works_count, keywords, description). 302 topics are in field 17 (Computer Science); relevant topics also exist in
	10	+ Mathematics (26), Engineering (22), Decision Sciences (18), Physics (31: quantum), Biochemistry (13: bioinformatics), etc.
	11	+- `taxonomy/dblp_streams_2018_2026.csv` — 3,570 DBLP venue streams (journal/conference series keys) with ≥ 100 records
	12	+ in 2018–2026, counts and names. `journals/corr` = arXiv CoRR (not a venue signal).
	13	+- arXiv categories: cs.AI cs.AR cs.CC cs.CE cs.CG cs.CL cs.CR cs.CV cs.CY cs.DB cs.DC cs.DL cs.DM cs.DS cs.ET cs.FL cs.GL
	14	+ cs.GR cs.GT cs.HC cs.IR cs.IT cs.LG cs.LO cs.MA cs.MM cs.MS cs.NA cs.NE cs.NI cs.OH cs.OS cs.PF cs.PL cs.RO cs.SC cs.SD
	15	+ cs.SE cs.SI cs.SY, plus cross-listed stat.ML, eess.AS, eess.IV, eess.SP, eess.SY, quant-ph, math.OC, math.IT, math.ST,
	16	+ q-bio.QM, q-bio.GN, physics.comp-ph.
	17	+
	18	+Deliverables (write these files; nothing else):
	19	+1. `taxonomy/ontology.yaml` with this schema:
	20	+ ```yaml
	21	+ version: "1.0"
	22	+ created: "2026-11-02"
	23	+ areas: # 8 top-level areas (choose names; every leaf belongs to exactly one area)
	24	+ - id: ai_ml
	25	+ name: ...
	26	+ leaves: [llm, foundation_models, ...]
	27	+ categories: # exactly 40 leaves, ids snake_case, in the order of CLAUDE.md §4
	28	+ - id: llm
	29	+ number: 1
	30	+ name: Large Language Models
	31	+ area: ai_ml
	32	+ definition: one or two precise sentences (what is in, what is out)
	33	+ openalex_topics: # list of {id: T10028, weight: 1.0, note: ...}; weight in (0,1]; include topics from any field
	34	+ openalex_subfields: # list of {id: 1702, weight: 0.3} — coarse fallback only
	35	+ arxiv_categories: # list of {id: cs.CL, weight: 0.5}
	36	+ dblp_streams: # list of {key: conf/acl, weight: 1.0}
	37	+ keywords: # list of {pattern: "(?i)\\blarge language model", weight: 1.0, field: title\|abstract\|both}
	38	+ exclusions: # optional list of regex patterns that veto the category
	39	+ related: [foundation_models, nlp] # ontology edges (is-related-to); also use `parent_of`/`child_of` where a leaf is a specialisation of another (e.g. llm child_of foundation_models; federated_learning child_of privacy_preserving_computation)
	40	+ provenance: short note on how the mapping was built
	41	+ assignment_rule:
	42	+ score: "sum of weights of matched signals, each signal type capped at 1.0, then sum over signal types"
	43	+ threshold: 1.0 # a work belongs to a category if score >= threshold
	44	+ primary: "argmax of score; ties broken by the more specific category (child before parent)"
	45	+ ```
	46	+2. `taxonomy/ONTOLOGY.md` — human-readable description: the 8 areas, the 40 leaves with definitions, the hierarchy
	47	+ (parent/child edges), coverage statistics you can compute from the inputs (how many OpenAlex CS topics are mapped,
	48	+ unmapped CS topics listed explicitly with a reason), and the known ambiguities (e.g. LLM vs foundation model vs
	49	+ generative AI; cloud vs distributed vs edge; cryptography vs security vs privacy).
	50	+3. `taxonomy/topic_map.csv` — one row per (openalex topic_id, category_id, weight) used in the YAML (for auditing).
	51	+
	52	+Constraints:
	53	+- Every one of the 302 OpenAlex CS topics must appear in `topic_map.csv` or in the "unmapped" list of ONTOLOGY.md.
	54	+- The six AI categories that OpenAlex topics cannot separate well (LLM, foundation models, multimodal, generative AI,
	55	+ agents, AI safety) and the small categories (federated learning, sustainable computing, autonomous scientific discovery,
	56	+ privacy-preserving computation, blockchain, quantum, edge) MUST have keyword patterns (title and abstract) strong enough
	57	+ to reach the threshold on their own; the broad categories (CV, NLP, SE, networks, ...) can rely on topics/venues/arXiv.
	58	+- Keyword regexes must be valid Python `re` patterns (double-escape backslashes in YAML strings) and should be
	59	+ conservative (word boundaries, avoid single ambiguous tokens such as "agent" or "model").
	60	+- Map at least the 600 largest DBLP streams; the mapping may give a stream to several categories.
	61	+- Do not invent topic ids: use only ids present in the CSV. Validate the YAML by loading it with Python (`yaml.safe_load`)
	62	+ and compiling every regex before you finish; report counts (topics mapped per category, keywords per category).
	63	+- Write nothing in the manuscript. Report back a short summary (≤ 30 lines) with the counts and the open ambiguities.
	64	+
	65	+## Addendum (2026-11-03, after a first attempt failed on output size)
	66	+
	67	+The first attempt tried to emit `ontology.yaml` in one response and exceeded the 64,000-token output limit. Work
	68	+incrementally and keep every single file write under ~6,000 tokens:
	69	+- Put the mapping data in Python modules, one per area: `taxonomy/src/area_ai_ml.py`, `taxonomy/src/area_perception.py`,
	70	+ … each defining `CATEGORIES = [ {...}, ... ]` dicts with the schema of the brief (ids, weights, patterns as raw strings).
	71	+ Write each module with its own Write call; if an area is large, split it into two modules.
	72	+- Write `taxonomy/build_ontology.py` that imports the modules, validates (40 leaves, unique ids, topic ids exist in the
	73	+ CSV, regexes compile, weights in (0,1]), and generates `taxonomy/ontology.yaml`, `taxonomy/topic_map.csv` and the
	74	+ coverage statistics section of `taxonomy/ONTOLOGY.md` (the prose part of ONTOLOGY.md you write by hand, briefly).
	75	+- Run `python3 taxonomy/build_ontology.py` (pyyaml: `/opt/homebrew/bin/python3 -m pip install --user pyyaml` or use
	76	+ `uv pip install` in a venv) and fix errors until it passes. Never print whole CSVs to your context: use grep/awk/python
	77	+ to look up topics by keyword.

added data/raw/README.md +10 −0

		`@@ -0,0 +1,10 @@`
	1	+# data/raw — provenance (nothing here is committed except this file; copies live on M3U96b:~/wp44_uqo/data/raw)
	2	+
	3	+\| Source \| What \| Obtained \| Size \| Integrity \|
	4	+\|---\|---\|---\|---\|---\|
	5	+\| OpenAlex works snapshot, release 2026-10-14 \| `data/jsonl/works/` 1,898 part files, 464,765,485 works, 675.3 GB gz \| 2026-11-02 21:19 → 23:05 ET, HTTPS from `openalex.s3.amazonaws.com`, 8 parallel streams \| filtered CS subset `works/cs/.jsonl.zst` 20 GB (15,490,159 records), slim table `works/slim/.parquet` 9.3 GB (all works) \| manifest (`works_manifest_2026-10-14.json`), per-file content length and SHA-256 in `works/ledger.sqlite` (1,898/1,898 ok, 0 parse errors, record total = manifest total) \|
	6	+\| OpenAlex entity snapshots \| topics 4,516; subfields 252; fields 26; domains 4; sources 268,629; institutions 143,455; publishers 10,713; funders 41,799; concepts 65,026 \| 2026-11-02 \| 0.6 GB \| `entities_ledger.json` (SHA-256 per file) \|
	7	+\| OpenAlex API increment \| works with any CS topic, publication_date 2026-10-01 → 2026-11-02, sliced by day × (CS primary / non-CS primary), basic paging ≤ 10,000 per slice \| 2026-11-03 (public budget $0.10/day → several days) \| raw JSON pages `api_increment/pages/` \| `api_increment/ledger.json` (counts, cost) \|
	8	+\| DBLP \| `dblp.xml.gz` (2026-11-02 22:39 UTC), `dblp.dtd`, `.md5` \| 2026-11-02 \| 1.12 GB \| MD5 `ff1c424c3b4cef36582f647c36df27c0` verified; parse report `dblp_parse_report.json` (8,840,866 records) \|
	9	+\| arXiv OAI-PMH \| set `cs`, metadataPrefix `arXiv`, datestamp 2018-01-01 → 2026-11-02, monthly windows, raw XML pages kept \| 2026-11-02/03 \| 758 pages, 915,115 unique records \| `harvest_state.json`, `arxiv_parse_report.json` \|
	10	+\| ACL Anthology \| `anthology+abstracts.bib.gz` \| 2026-11-02 \| 46.7 MB, 138,796 entries \| SHA-256 `44c3f7e907f221b740bea09069818a3d7c0dfd88b2da3b750d9f02955efc58b3` \|

added ingestion/acl_parse.py +83 −0

		`@@ -0,0 +1,83 @@`
	1	+#!/usr/bin/env python3
	2	+"""WP44 — parse the ACL Anthology BibTeX export (anthology+abstracts.bib.gz) into Parquet."""
	3	+import argparse, gzip, hashlib, json, os, re, datetime as dt
	4	+import pyarrow as pa, pyarrow.parquet as pq
	5	+
	6	+FIELDS = ["title", "author", "editor", "booktitle", "journal", "month", "year", "address", "publisher", "url", "doi", "pages", "abstract", "volume", "number"]
	7	+SCHEMA = pa.schema([("acl_id", pa.string()), ("entry_type", pa.string())] + [(f, pa.string()) for f in FIELDS])
	8	+
	9	+
	10	+def entries(txt):
	11	+ i = 0; n = len(txt)
	12	+ while True:
	13	+ j = txt.find("@", i)
	14	+ if j < 0:
	15	+ return
	16	+ k = txt.find("{", j)
	17	+ et = txt[j + 1:k].strip().lower()
	18	+ depth = 0; p = k
	19	+ while p < n:
	20	+ c = txt[p]
	21	+ if c == "{":
	22	+ depth += 1
	23	+ elif c == "}":
	24	+ depth -= 1
	25	+ if depth == 0:
	26	+ break
	27	+ p += 1
	28	+ body = txt[k + 1:p]; i = p + 1
	29	+ if et in ("comment", "string", "preamble"):
	30	+ continue
	31	+ yield et, body
	32	+
	33	+
	34	+def parse_body(body):
	35	+ key, _, rest = body.partition(",")
	36	+ out = {"acl_id": key.strip()}
	37	+ # fields of the form name = {value} or name = "value", values may contain nested braces
	38	+ pos = 0
	39	+ while True:
	40	+ m = re.search(r"(\w+)\s=\s", rest[pos:])
	41	+ if not m:
	42	+ break
	43	+ name = m.group(1).lower(); p = pos + m.end()
	44	+ if p >= len(rest):
	45	+ break
	46	+ if rest[p] == "{":
	47	+ depth = 0; q = p
	48	+ while q < len(rest):
	49	+ if rest[q] == "{": depth += 1
	50	+ elif rest[q] == "}":
	51	+ depth -= 1
	52	+ if depth == 0: break
	53	+ q += 1
	54	+ val = rest[p + 1:q]; pos = q + 1
	55	+ elif rest[p] == '"':
	56	+ q = rest.find('"', p + 1); val = rest[p + 1:q]; pos = q + 1
	57	+ else:
	58	+ q = rest.find(",", p); q = len(rest) if q < 0 else q; val = rest[p:q]; pos = q
	59	+ val = re.sub(r"\s+", " ", val).strip()
	60	+ val = val.replace("{", "").replace("}", "")
	61	+ if name in FIELDS:
	62	+ out[name] = val
	63	+ return out
	64	+
	65	+
	66	+def main():
	67	+ ap = argparse.ArgumentParser(); ap.add_argument("--bib", required=True); ap.add_argument("--out", required=True)
	68	+ a = ap.parse_args(); os.makedirs(a.out, exist_ok=True)
	69	+ raw = open(a.bib, "rb").read(); sha = hashlib.sha256(raw).hexdigest()
	70	+ txt = gzip.decompress(raw).decode("utf-8", "replace")
	71	+ rows = {n: [] for n in SCHEMA.names}; n = 0; types = {}
	72	+ for et, body in entries(txt):
	73	+ d = parse_body(body); d["entry_type"] = et; types[et] = types.get(et, 0) + 1
	74	+ for k in SCHEMA.names:
	75	+ rows[k].append(d.get(k))
	76	+ n += 1
	77	+ pq.write_table(pa.Table.from_pydict(rows, schema=SCHEMA), os.path.join(a.out, "acl_anthology.parquet"), compression="zstd")
	78	+ rep = {"input": a.bib, "sha256": sha, "bytes": len(raw), "entries": n, "types": types, "parsed_at": dt.datetime.now(dt.UTC).isoformat()}
	79	+ json.dump(rep, open(os.path.join(a.out, "acl_parse_report.json"), "w"), indent=1); print(json.dumps(rep, indent=1))
	80	+
	81	+
	82	+if __name__ == "__main__":
	83	+ main()

added ingestion/arxiv_oai.py +134 −0

		`@@ -0,0 +1,134 @@`
	1	+#!/usr/bin/env python3
	2	+"""WP44 — harvest arXiv metadata (set=cs) through OAI-PMH (metadataPrefix=arXiv).
	3	+
	4	+Harvests by monthly datestamp windows from --from to --until; each window's raw
	5	+XML pages are stored verbatim (immutable raw snapshot) under <out>/raw/<window>/
	6	+page_NNNN.xml and parsed into <out>/arxiv_cs.parquet at the end (``--parse-only``
	7	+re-parses). Politeness: >= 3 s between requests, honours Retry-After on 503.
	8	+Note: OAI ``from``/``until`` filter on the record datestamp (last update), so a
	9	+window contains every record updated in that window; ``created`` gives the first
	10	+submission date. Duplicates across windows are resolved at parse time (latest
	11	+datestamp wins).
	12	+"""
	13	+import argparse, glob, json, os, re, time, datetime as dt, calendar
	14	+import requests
	15	+from lxml import etree
	16	+import pyarrow as pa, pyarrow.parquet as pq
	17	+
	18	+BASE = "https://oaipmh.arxiv.org/oai"
	19	+NS = {"oai": "http://www.openarchives.org/OAI/2.0/", "ax": "http://arxiv.org/OAI/arXiv/"}
	20	+UA = "WP44-UQO-systematic-mapping/0.1 (academic research; contact simon-pierre.boucher@uqo.ca)"
	21	+SCHEMA = pa.schema([
	22	+ ("arxiv_id", pa.string()), ("datestamp", pa.string()), ("created", pa.string()), ("updated", pa.string()),
	23	+ ("title", pa.string()), ("abstract", pa.string()), ("authors", pa.list_(pa.string())), ("categories", pa.list_(pa.string())),
	24	+ ("primary_category", pa.string()), ("comments", pa.string()), ("journal_ref", pa.string()), ("doi", pa.string()),
	25	+ ("license", pa.string()), ("report_no", pa.string()), ("msc_class", pa.string()), ("acm_class", pa.string()), ("set_specs", pa.list_(pa.string())),
	26	+])
	27	+
	28	+
	29	+def month_windows(start, end):
	30	+ y, m = start.year, start.month
	31	+ while (y, m) <= (end.year, end.month):
	32	+ last = calendar.monthrange(y, m)[1]
	33	+ a = dt.date(y, m, 1); b = min(dt.date(y, m, last), end)
	34	+ yield a, b
	35	+ m += 1
	36	+ if m == 13:
	37	+ y, m = y + 1, 1
	38	+
	39	+
	40	+def fetch(session, params, log):
	41	+ for attempt in range(12):
	42	+ try:
	43	+ r = session.get(BASE, params=params, timeout=180, headers={"User-Agent": UA})
	44	+ except requests.RequestException as e:
	45	+ log(f" network error {e}; sleep 30"); time.sleep(30); continue
	46	+ if r.status_code == 503:
	47	+ wait = int(r.headers.get("Retry-After", "30")); log(f" 503 retry-after {wait}"); time.sleep(wait + 1); continue
	48	+ if r.status_code != 200:
	49	+ log(f" HTTP {r.status_code}; sleep 30"); time.sleep(30); continue
	50	+ return r.text
	51	+ raise RuntimeError("too many failures")
	52	+
	53	+
	54	+def harvest(a, log):
	55	+ s = requests.Session()
	56	+ state_path = os.path.join(a.out, "harvest_state.json")
	57	+ state = json.load(open(state_path)) if os.path.exists(state_path) else {"windows": {}}
	58	+ for w0, w1 in month_windows(dt.date.fromisoformat(a.frm), dt.date.fromisoformat(a.until)):
	59	+ key = f"{w0}_{w1}"
	60	+ if state["windows"].get(key, {}).get("status") == "done":
	61	+ continue
	62	+ d = os.path.join(a.out, "raw", key); os.makedirs(d, exist_ok=True)
	63	+ params = {"verb": "ListRecords", "metadataPrefix": "arXiv", "set": a.set, "from": str(w0), "until": str(w1)}
	64	+ page = 0; nrec = 0; t0 = time.time()
	65	+ while True:
	66	+ xml = fetch(s, params, log)
	67	+ with open(os.path.join(d, f"page_{page:04d}.xml"), "w") as fh:
	68	+ fh.write(xml)
	69	+ root = etree.fromstring(xml.encode())
	70	+ err = root.find("oai:error", NS)
	71	+ if err is not None:
	72	+ if err.get("code") == "noRecordsMatch":
	73	+ break
	74	+ raise RuntimeError(f"OAI error {err.get('code')}: {err.text}")
	75	+ nrec += len(root.findall(".//oai:record", NS))
	76	+ tok = root.find(".//oai:resumptionToken", NS)
	77	+ page += 1
	78	+ if tok is None or not (tok.text or "").strip():
	79	+ break
	80	+ params = {"verb": "ListRecords", "resumptionToken": tok.text.strip()}
	81	+ time.sleep(a.delay)
	82	+ state["windows"][key] = {"status": "done", "pages": page, "records": nrec, "seconds": round(time.time() - t0),
	83	+ "when": dt.datetime.now(dt.UTC).isoformat()}
	84	+ json.dump(state, open(state_path, "w"), indent=1)
	85	+ log(f"window {key}: {page} pages, {nrec} records, {time.time()-t0:.0f}s")
	86	+ time.sleep(a.delay)
	87	+
	88	+
	89	+def parse(a, log):
	90	+ recs = {}
	91	+ files = sorted(glob.glob(os.path.join(a.out, "raw", "", "page_.xml")))
	92	+ for f in files:
	93	+ root = etree.parse(f).getroot()
	94	+ for rec in root.findall(".//oai:record", NS):
	95	+ h = rec.find("oai:header", NS)
	96	+ if h.get("status") == "deleted":
	97	+ continue
	98	+ m = rec.find("oai:metadata/ax:arXiv", NS)
	99	+ if m is None:
	100	+ continue
	101	+ g = lambda t: (m.findtext(f"ax:{t}", namespaces=NS) or "").strip() or None
	102	+ aid = g("id"); ds = h.findtext("oai:datestamp", namespaces=NS)
	103	+ if aid in recs and recs[aid]["datestamp"] >= ds:
	104	+ continue
	105	+ cats = (g("categories") or "").split()
	106	+ authors = []
	107	+ for au in m.findall("ax:authors/ax:author", NS):
	108	+ kn = au.findtext("ax:keyname", namespaces=NS) or ""; fn = au.findtext("ax:forenames", namespaces=NS) or ""
	109	+ authors.append((fn + " " + kn).strip())
	110	+ recs[aid] = dict(arxiv_id=aid, datestamp=ds, created=g("created"), updated=g("updated"),
	111	+ title=re.sub(r"\s+", " ", g("title") or ""), abstract=re.sub(r"\s+", " ", g("abstract") or ""),
	112	+ authors=authors, categories=cats, primary_category=cats[0] if cats else None,
	113	+ comments=g("comments"), journal_ref=g("journal-ref"), doi=(g("doi") or "").lower() or None, license=g("license"),
	114	+ report_no=g("report-no"), msc_class=g("msc-class"), acm_class=g("acm-class"),
	115	+ set_specs=[x.text for x in h.findall("oai:setSpec", NS)])
	116	+ cols = {n: [r[n] for r in recs.values()] for n in SCHEMA.names}
	117	+ pq.write_table(pa.Table.from_pydict(cols, schema=SCHEMA), os.path.join(a.out, "arxiv_cs.parquet"), compression="zstd")
	118	+ rep = {"files": len(files), "records": len(recs), "parsed_at": dt.datetime.now(dt.UTC).isoformat(), "set": a.set, "from": a.frm, "until": a.until}
	119	+ json.dump(rep, open(os.path.join(a.out, "arxiv_parse_report.json"), "w"), indent=1); log(json.dumps(rep))
	120	+
	121	+
	122	+def main():
	123	+ ap = argparse.ArgumentParser(); ap.add_argument("--out", required=True); ap.add_argument("--set", default="cs")
	124	+ ap.add_argument("--from", dest="frm", default="2018-01-01"); ap.add_argument("--until", default="2026-11-02")
	125	+ ap.add_argument("--delay", type=float, default=3.5); ap.add_argument("--parse-only", action="store_true")
	126	+ a = ap.parse_args(); os.makedirs(a.out, exist_ok=True)
	127	+ log = lambda s: print(f"[{dt.datetime.now():%H:%M:%S}] {s}", flush=True)
	128	+ if not a.parse_only:
	129	+ harvest(a, log)
	130	+ parse(a, log)
	131	+
	132	+
	133	+if __name__ == "__main__":
	134	+ main()

added ingestion/build_corpus.py +197 −0

		`@@ -0,0 +1,197 @@`
	1	+#!/usr/bin/env python3
	2	+"""WP44 — normalise the filtered OpenAlex CS records (cs/*.jsonl.zst) into Parquet tables.
	3	+
	4	+Tables (one Parquet file per input part under <out>/tables/<table>/<part>.parquet, resumable):
	5	+ works, authorships, work_topics, work_keywords, references, abstracts, locations, grants, counts_by_year
	6	+Run on the compute node after (or while) ingestion/openalex_snapshot.py runs; parts whose outputs exist are skipped.
	7	+"""
	8	+import argparse, glob, io, multiprocessing as mp, os, re, time, datetime as dt
	9	+import orjson, zstandard as zstd
	10	+import pyarrow as pa, pyarrow.parquet as pq
	11	+
	12	+ARXIV_RE = re.compile(r"arxiv\.org/(?:abs\|pdf)/([0-9]{4}\.[0-9]{4,5}(?:v\d+)?\|[a-z\-]+(?:\.[A-Z]{2})?/[0-9]{7}(?:v\d+)?)", re.I)
	13	+ARXIV_DOI_RE = re.compile(r"^10\.48550/arxiv\.(.+)$", re.I)
	14	+
	15	+S = {
	16	+ "works": pa.schema([("id", pa.int64()), ("doi", pa.string()), ("title", pa.string()), ("publication_year", pa.int32()),
	17	+ ("publication_date", pa.string()), ("created_date", pa.string()), ("updated_date", pa.string()), ("type", pa.string()),
	18	+ ("type_crossref", pa.string()), ("language", pa.string()), ("cited_by_count", pa.int32()), ("fwci", pa.float32()),
	19	+ ("citation_percentile", pa.float32()), ("is_top1", pa.bool_()), ("is_top10", pa.bool_()), ("is_oa", pa.bool_()), ("oa_status", pa.string()),
	20	+ ("license", pa.string()), ("is_retracted", pa.bool_()), ("is_paratext", pa.bool_()), ("is_xpac", pa.bool_()), ("n_refs", pa.int32()),
	21	+ ("n_authors", pa.int32()), ("n_institutions", pa.int32()), ("n_countries", pa.int32()), ("countries", pa.list_(pa.string())),
	22	+ ("institution_types", pa.list_(pa.string())), ("primary_topic_id", pa.int32()), ("primary_subfield_id", pa.int32()),
	23	+ ("primary_field_id", pa.int32()), ("primary_domain_id", pa.int32()), ("cs_primary", pa.bool_()), ("cs_any", pa.bool_()),
	24	+ ("source_id", pa.int64()), ("source_name", pa.string()), ("source_type", pa.string()), ("host_organization", pa.string()),
	25	+ ("is_in_doaj", pa.bool_()), ("arxiv_id", pa.string()), ("pmid", pa.string()), ("mag", pa.string()), ("has_abstract", pa.bool_()),
	26	+ ("has_fulltext", pa.bool_()), ("n_locations", pa.int32()), ("indexed_in", pa.list_(pa.string())), ("referenced_works_count", pa.int32()),
	27	+ ("n_grants", pa.int32()), ("apc_paid_usd", pa.int32()), ("is_authors_truncated", pa.bool_()), ("part", pa.string())]),
	28	+ "authorships": pa.schema([("work_id", pa.int64()), ("position", pa.int32()), ("author_position", pa.string()), ("author_id", pa.int64()),
	29	+ ("author_name", pa.string()), ("orcid", pa.string()), ("is_corresponding", pa.bool_()), ("institution_ids", pa.list_(pa.int64())),
	30	+ ("institution_names", pa.list_(pa.string())), ("institution_types", pa.list_(pa.string())), ("countries", pa.list_(pa.string())),
	31	+ ("n_raw_affiliations", pa.int32())]),
	32	+ "work_topics": pa.schema([("work_id", pa.int64()), ("rank", pa.int16()), ("topic_id", pa.int32()), ("score", pa.float32()),
	33	+ ("subfield_id", pa.int32()), ("field_id", pa.int32()), ("domain_id", pa.int32())]),
	34	+ "work_keywords": pa.schema([("work_id", pa.int64()), ("keyword", pa.string()), ("score", pa.float32())]),
	35	+ "references": pa.schema([("work_id", pa.int64()), ("ref_id", pa.int64())]),
	36	+ "abstracts": pa.schema([("work_id", pa.int64()), ("abstract", pa.string()), ("n_words", pa.int32())]),
	37	+ "locations": pa.schema([("work_id", pa.int64()), ("rank", pa.int16()), ("is_primary", pa.bool_()), ("is_best_oa", pa.bool_()), ("source_id", pa.int64()),
	38	+ ("source_name", pa.string()), ("source_type", pa.string()), ("landing_page_url", pa.string()), ("pdf_url", pa.string()), ("is_oa", pa.bool_()),
	39	+ ("license", pa.string()), ("version", pa.string()), ("is_accepted", pa.bool_()), ("is_published", pa.bool_())]),
	40	+ "grants": pa.schema([("work_id", pa.int64()), ("funder_id", pa.int64()), ("funder_name", pa.string()), ("award_id", pa.string())]),
	41	+ "counts_by_year": pa.schema([("work_id", pa.int64()), ("year", pa.int16()), ("cited_by_count", pa.int32())]),
	42	+}
	43	+
	44	+
	45	+def oid(u):
	46	+ if not u:
	47	+ return None
	48	+ t = u.rsplit("/", 1)[-1]
	49	+ i = 0
	50	+ while i < len(t) and not t[i].isdigit():
	51	+ i += 1
	52	+ try:
	53	+ return int(t[i:])
	54	+ except ValueError:
	55	+ return None
	56	+
	57	+
	58	+def abstract_text(inv):
	59	+ if not inv:
	60	+ return None
	61	+ pos = []
	62	+ for w, ps in inv.items():
	63	+ for p in ps:
	64	+ pos.append((p, w))
	65	+ pos.sort()
	66	+ return " ".join(w for _, w in pos)
	67	+
	68	+
	69	+def arxiv_from(w):
	70	+ ids = w.get("ids") or {}
	71	+ if ids.get("arxiv"):
	72	+ return str(ids["arxiv"]).rsplit("/", 1)[-1]
	73	+ doi = w.get("doi") or ""
	74	+ m = ARXIV_DOI_RE.match(doi.replace("https://doi.org/", ""))
	75	+ if m:
	76	+ return m.group(1)
	77	+ for loc in (w.get("locations") or []):
	78	+ for k in ("landing_page_url", "pdf_url"):
	79	+ u = loc.get(k) or ""
	80	+ m = ARXIV_RE.search(u)
	81	+ if m:
	82	+ return re.sub(r"v\d+$", "", m.group(1))
	83	+ return None
	84	+
	85	+
	86	+def process(args):
	87	+ path, out, cs_field = args
	88	+ part = os.path.basename(path).replace(".jsonl.zst", "")
	89	+ outs = {t: os.path.join(out, "tables", t, part + ".parquet") for t in S}
	90	+ if all(os.path.exists(p) for p in outs.values()):
	91	+ return part, "skip", 0
	92	+ for p in outs.values():
	93	+ os.makedirs(os.path.dirname(p), exist_ok=True)
	94	+ rows = {t: {c: [] for c in S[t].names} for t in S}
	95	+ n = 0
	96	+ with open(path, "rb") as fh, zstd.ZstdDecompressor().stream_reader(fh) as r:
	97	+ for line in io.TextIOWrapper(r, encoding="utf-8"):
	98	+ if not line.strip():
	99	+ continue
	100	+ w = orjson.loads(line); n += 1
	101	+ wid = oid(w["id"])
	102	+ doi = (w.get("doi") or "").replace("https://doi.org/", "").lower() or None
	103	+ pt = w.get("primary_topic") or {}
	104	+ topics = w.get("topics") or []
	105	+ cs_any = any(((t.get("field") or {}).get("id") == cs_field) for t in topics) or ((pt.get("field") or {}).get("id") == cs_field)
	106	+ loc = w.get("primary_location") or {}
	107	+ src = loc.get("source") or {}
	108	+ best = w.get("best_oa_location") or {}
	109	+ oa = w.get("open_access") or {}
	110	+ auths = w.get("authorships") or []
	111	+ inst_ids = set(); countries = set(); itypes = []
	112	+ for i, a in enumerate(auths):
	113	+ ins = a.get("institutions") or []
	114	+ ii = [oid(x.get("id")) for x in ins]
	115	+ inst_ids.update(x for x in ii if x)
	116	+ cc = [c for c in (a.get("countries") or []) if c]
	117	+ countries.update(cc)
	118	+ itypes.extend(x.get("type") for x in ins if x.get("type"))
	119	+ au = a.get("author") or {}
	120	+ R = rows["authorships"]
	121	+ R["work_id"].append(wid); R["position"].append(i); R["author_position"].append(a.get("author_position"))
	122	+ R["author_id"].append(oid(au.get("id"))); R["author_name"].append(au.get("display_name")); R["orcid"].append(au.get("orcid"))
	123	+ R["is_corresponding"].append(bool(a.get("is_corresponding"))); R["institution_ids"].append(ii)
	124	+ R["institution_names"].append([x.get("display_name") for x in ins]); R["institution_types"].append([x.get("type") for x in ins])
	125	+ R["countries"].append(cc); R["n_raw_affiliations"].append(len(a.get("raw_affiliation_strings") or []))
	126	+ for k, t in enumerate(topics):
	127	+ R = rows["work_topics"]
	128	+ R["work_id"].append(wid); R["rank"].append(k); R["topic_id"].append(oid(t.get("id"))); R["score"].append(t.get("score"))
	129	+ R["subfield_id"].append(oid((t.get("subfield") or {}).get("id"))); R["field_id"].append(oid((t.get("field") or {}).get("id")))
	130	+ R["domain_id"].append(oid((t.get("domain") or {}).get("id")))
	131	+ for kw in (w.get("keywords") or []):
	132	+ R = rows["work_keywords"]; R["work_id"].append(wid); R["keyword"].append(kw.get("display_name")); R["score"].append(kw.get("score"))
	133	+ refs = w.get("referenced_works") or []
	134	+ for rr in refs:
	135	+ rows["references"]["work_id"].append(wid); rows["references"]["ref_id"].append(oid(rr))
	136	+ ab = abstract_text(w.get("abstract_inverted_index"))
	137	+ if ab:
	138	+ R = rows["abstracts"]; R["work_id"].append(wid); R["abstract"].append(ab); R["n_words"].append(len(ab.split()))
	139	+ locs = w.get("locations") or []
	140	+ best_url = best.get("landing_page_url") if best else None
	141	+ for k, L in enumerate(locs):
	142	+ s_ = L.get("source") or {}
	143	+ R = rows["locations"]
	144	+ R["work_id"].append(wid); R["rank"].append(k); R["is_primary"].append(L.get("landing_page_url") == loc.get("landing_page_url") and L.get("pdf_url") == loc.get("pdf_url"))
	145	+ R["is_best_oa"].append(bool(best_url) and L.get("landing_page_url") == best_url)
	146	+ R["source_id"].append(oid(s_.get("id"))); R["source_name"].append(s_.get("display_name")); R["source_type"].append(s_.get("type"))
	147	+ R["landing_page_url"].append(L.get("landing_page_url")); R["pdf_url"].append(L.get("pdf_url")); R["is_oa"].append(bool(L.get("is_oa")))
	148	+ R["license"].append(L.get("license")); R["version"].append(L.get("version")); R["is_accepted"].append(bool(L.get("is_accepted"))); R["is_published"].append(bool(L.get("is_published")))
	149	+ for g in (w.get("grants") or []):
	150	+ R = rows["grants"]; R["work_id"].append(wid); R["funder_id"].append(oid(g.get("funder"))); R["funder_name"].append(g.get("funder_display_name")); R["award_id"].append(g.get("award_id"))
	151	+ for c in (w.get("counts_by_year") or []):
	152	+ R = rows["counts_by_year"]; R["work_id"].append(wid); R["year"].append(c.get("year")); R["cited_by_count"].append(c.get("cited_by_count"))
	153	+ cnp = w.get("citation_normalized_percentile") or {}
	154	+ ids = w.get("ids") or {}
	155	+ apc = (w.get("apc_paid") or {}).get("value_usd") if isinstance(w.get("apc_paid"), dict) else None
	156	+ R = rows["works"]
	157	+ vals = dict(id=wid, doi=doi, title=w.get("title") or w.get("display_name"), publication_year=w.get("publication_year"), publication_date=w.get("publication_date"),
	158	+ created_date=w.get("created_date"), updated_date=(w.get("updated_date") or "")[:10] or None, type=w.get("type"), type_crossref=w.get("type_crossref"),
	159	+ language=w.get("language"), cited_by_count=w.get("cited_by_count"), fwci=w.get("fwci"), citation_percentile=cnp.get("value"),
	160	+ is_top1=bool(cnp.get("is_in_top_1_percent")), is_top10=bool(cnp.get("is_in_top_10_percent")), is_oa=bool(oa.get("is_oa")), oa_status=oa.get("oa_status"),
	161	+ license=(best.get("license") if best else None) or loc.get("license"), is_retracted=bool(w.get("is_retracted")), is_paratext=bool(w.get("is_paratext")),
	162	+ is_xpac=bool(w.get("is_xpac")), n_refs=len(refs), n_authors=w.get("authors_count") if w.get("authors_count") is not None else len(auths),
	163	+ n_institutions=len(inst_ids), n_countries=len(countries), countries=sorted(countries), institution_types=sorted(set(itypes)),
	164	+ primary_topic_id=oid(pt.get("id")), primary_subfield_id=oid((pt.get("subfield") or {}).get("id")), primary_field_id=oid((pt.get("field") or {}).get("id")),
	165	+ primary_domain_id=oid((pt.get("domain") or {}).get("id")), cs_primary=(pt.get("field") or {}).get("id") == cs_field, cs_any=bool(cs_any),
	166	+ source_id=oid(src.get("id")), source_name=src.get("display_name"), source_type=src.get("type"), host_organization=src.get("host_organization_name"),
	167	+ is_in_doaj=bool(src.get("is_in_doaj")), arxiv_id=arxiv_from(w), pmid=(ids.get("pmid") or "").rsplit("/", 1)[-1] or None, mag=str(ids.get("mag")) if ids.get("mag") else None,
	168	+ has_abstract=ab is not None, has_fulltext=bool(w.get("has_fulltext") or w.get("has_content")), n_locations=len(locs), indexed_in=w.get("indexed_in") or [],
	169	+ referenced_works_count=w.get("referenced_works_count") if w.get("referenced_works_count") is not None else len(refs), n_grants=len(w.get("grants") or []),
	170	+ apc_paid_usd=apc, is_authors_truncated=bool(w.get("is_authors_truncated")), part=part)
	171	+ for k, v in vals.items():
	172	+ R[k].append(v)
	173	+ for t, sch in S.items():
	174	+ tbl = pa.Table.from_pydict(rows[t], schema=sch)
	175	+ tmp = outs[t] + ".part"
	176	+ pq.write_table(tbl, tmp, compression="zstd"); os.replace(tmp, outs[t])
	177	+ return part, "done", n
	178	+
	179	+
	180	+def main():
	181	+ ap = argparse.ArgumentParser(); ap.add_argument("--cs-dir", required=True); ap.add_argument("--out", required=True)
	182	+ ap.add_argument("--workers", type=int, default=8); ap.add_argument("--limit", type=int, default=0)
	183	+ a = ap.parse_args()
	184	+ files = sorted(glob.glob(os.path.join(a.cs_dir, "*.jsonl.zst")))
	185	+ if a.limit:
	186	+ files = files[: a.limit]
	187	+ t0 = time.time(); tot = 0; done = 0
	188	+ with mp.Pool(a.workers, maxtasksperchild=40) as pool:
	189	+ for part, st, n in pool.imap_unordered(process, [(f, a.out, "https://openalex.org/fields/17") for f in files]):
	190	+ tot += n; done += 1
	191	+ if st == "done" or done % 100 == 0:
	192	+ print(f"[{dt.datetime.now():%H:%M:%S}] {st} {part} n={n} \| {done}/{len(files)} files, {tot} works, {time.time()-t0:.0f}s", flush=True)
	193	+ print("finished", done, "files", tot, "works")
	194	+
	195	+
	196	+if __name__ == "__main__":
	197	+ main()

added ingestion/dblp_parse.py +85 −0

		`@@ -0,0 +1,85 @@`
	1	+#!/usr/bin/env python3
	2	+"""WP44 — parse the DBLP XML dump (dblp.xml.gz + dblp.dtd) into Parquet.
	3	+
	4	+Output: <out>/dblp_publications.parquet with one row per publication record
	5	+(article, inproceedings, proceedings, book, incollection, phdthesis, mastersthesis,
	6	+www is skipped) and <out>/dblp_parse_report.json (counts, md5 of input, timings).
	7	+All years are kept (the historical baseline needs 2018–2022; counts per year are
	8	+reported for the whole archive).
	9	+"""
	10	+import argparse, gzip, hashlib, json, os, time, datetime as dt
	11	+from lxml import etree
	12	+import pyarrow as pa, pyarrow.parquet as pq
	13	+
	14	+TYPES = {"article", "inproceedings", "proceedings", "book", "incollection", "phdthesis", "mastersthesis", "data"}
	15	+SCHEMA = pa.schema([
	16	+ ("key", pa.string()), ("rec_type", pa.string()), ("mdate", pa.string()), ("publtype", pa.string()),
	17	+ ("title", pa.string()), ("year", pa.int32()), ("authors", pa.list_(pa.string())), ("author_pids", pa.list_(pa.string())),
	18	+ ("editors", pa.list_(pa.string())), ("journal", pa.string()), ("booktitle", pa.string()), ("volume", pa.string()),
	19	+ ("number", pa.string()), ("pages", pa.string()), ("publisher", pa.string()), ("series", pa.string()),
	20	+ ("doi", pa.string()), ("ee", pa.list_(pa.string())), ("url", pa.string()), ("crossref", pa.string()),
	21	+ ("stream", pa.string()),
	22	+])
	23	+
	24	+
	25	+def text(el):
	26	+ return "".join(el.itertext()).strip() if el is not None else None
	27	+
	28	+
	29	+def main():
	30	+ ap = argparse.ArgumentParser(); ap.add_argument("--xml", required=True); ap.add_argument("--out", required=True)
	31	+ ap.add_argument("--min-year", type=int, default=0)
	32	+ a = ap.parse_args(); a.xml = os.path.abspath(a.xml); a.out = os.path.abspath(a.out); os.makedirs(a.out, exist_ok=True)
	33	+ t0 = time.time()
	34	+ md5 = hashlib.md5(open(a.xml, "rb").read(1 << 30)).hexdigest() # first GB hash (file is 1.1 GB) + size recorded
	35	+ size = os.path.getsize(a.xml)
	36	+ # lxml needs the DTD next to the XML for entity resolution; parse the gz stream
	37	+ parser = etree.XMLParser(load_dtd=True, resolve_entities=True, huge_tree=True)
	38	+ counts = {}; years = {}
	39	+ rows = {n: [] for n in SCHEMA.names}; n = 0
	40	+ writer = pq.ParquetWriter(os.path.join(a.out, "dblp_publications.parquet"), SCHEMA, compression="zstd")
	41	+
	42	+ def flush():
	43	+ nonlocal rows
	44	+ if rows["key"]:
	45	+ writer.write_table(pa.Table.from_pydict(rows, schema=SCHEMA)); rows = {k: [] for k in SCHEMA.names}
	46	+
	47	+ # iterparse over the gzip stream; the DTD is referenced relatively (dblp.dtd) -> chdir
	48	+ os.chdir(os.path.dirname(os.path.abspath(a.xml)))
	49	+ with gzip.open(a.xml, "rb") as fh:
	50	+ ctx = etree.iterparse(fh, events=("end",), tag=list(TYPES \| {"www"}), load_dtd=True, resolve_entities=True, huge_tree=True)
	51	+ for _, el in ctx:
	52	+ t = el.tag
	53	+ if t in TYPES:
	54	+ counts[t] = counts.get(t, 0) + 1
	55	+ yr = el.findtext("year"); yr = int(yr) if yr and yr.isdigit() else None
	56	+ years[yr] = years.get(yr, 0) + 1
	57	+ if yr is not None and yr >= a.min_year:
	58	+ ees = [e.text for e in el.findall("ee") if e.text]
	59	+ doi = next((e[16:].lower() for e in ees if e.startswith("https://doi.org/")), None)
	60	+ stream = el.get("key", "").rsplit("/", 1)[0] if "/" in el.get("key", "") else None
	61	+ vals = dict(key=el.get("key"), rec_type=t, mdate=el.get("mdate"), publtype=el.get("publtype"),
	62	+ title=text(el.find("title")), year=yr,
	63	+ authors=[text(x) for x in el.findall("author")], author_pids=[x.get("pid") for x in el.findall("author")],
	64	+ editors=[text(x) for x in el.findall("editor")], journal=el.findtext("journal"), booktitle=el.findtext("booktitle"),
	65	+ volume=el.findtext("volume"), number=el.findtext("number"), pages=el.findtext("pages"),
	66	+ publisher=el.findtext("publisher"), series=text(el.find("series")), doi=doi, ee=ees,
	67	+ url=el.findtext("url"), crossref=el.findtext("crossref"), stream=stream)
	68	+ for k, v in vals.items():
	69	+ rows[k].append(v)
	70	+ n += 1
	71	+ if n % 200000 == 0:
	72	+ flush(); print(f"{n} rows, {time.time()-t0:.0f}s", flush=True)
	73	+ el.clear()
	74	+ while el.getprevious() is not None:
	75	+ del el.getparent()[0]
	76	+ flush(); writer.close()
	77	+ rep = {"input": os.path.abspath(a.xml), "input_bytes": size, "input_md5_first_GiB": md5, "parsed_at": dt.datetime.now(dt.UTC).isoformat(),
	78	+ "rows_written": n, "min_year": a.min_year, "counts_by_type": counts, "counts_by_year": {str(k): v for k, v in sorted(years.items(), key=lambda kv: (kv[0] is None, kv[0]))},
	79	+ "seconds": round(time.time() - t0)}
	80	+ json.dump(rep, open(os.path.join(a.out, "dblp_parse_report.json"), "w"), indent=1)
	81	+ print(json.dumps({k: v for k, v in rep.items() if k != "counts_by_year"}, indent=1))
	82	+
	83	+
	84	+if __name__ == "__main__":
	85	+ main()

added ingestion/dblp_streams.py +9 −0

		`@@ -0,0 +1,9 @@`
	1	+#!/usr/bin/env python3
	2	+"""Venue streams (journals / conference series) of DBLP 2018-2026 with counts, for the taxonomy mapping."""
	3	+import duckdb, sys
	4	+src, out = sys.argv[1], sys.argv[2]
	5	+con = duckdb.connect()
	6	+con.sql(f"""copy (select stream, rec_type, count(*) n, min(year) y0, max(year) y1, any_value(journal) journal, any_value(booktitle) booktitle
	7	+ from read_parquet('{src}') where year between 2018 and 2026 and rec_type in ('article','inproceedings')
	8	+ group by 1,2 having n >= 100 order by n desc) to '{out}' (header)""")
	9	+print(con.sql(f"select count() from read_csv_auto('{out}')").fetchall(), con.sql(f"select rec_type, count() from read_parquet('{src}') where year between 2018 and 2026 group by 1 order by 2 desc").fetchall())

added ingestion/link_sources.py +121 −0

		`@@ -0,0 +1,121 @@`
	1	+#!/usr/bin/env python3
	2	+"""WP44 — Stage S1 record linkage: OpenAlex CS records ↔ DBLP ↔ arXiv ↔ ACL Anthology, in DuckDB.
	3	+
	4	+Creates/refreshes <db> with views over the Parquet tables and the linkage tables
	5	+ link_dblp_oa(dblp_key, oa_id, method), link_arxiv_oa(arxiv_id, oa_id, method), link_acl_oa(acl_id, oa_id, method)
	6	+and writes data/metadata/linkage_report.json. Also writes the list of OpenAlex ids of DBLP-indexed works (2018–2026)
	7	+that are in the slim table but NOT in the CS-topic subset (needed for the second snapshot pass).
	8	+"""
	9	+import argparse, json, os, time, datetime as dt
	10	+import duckdb
	11	+
	12	+
	13	+def main():
	14	+ ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); ap.add_argument("--db", default=None)
	15	+ a = ap.parse_args(); R = a.root; db = a.db or os.path.join(R, "data/processed/wp44.duckdb")
	16	+ con = duckdb.connect(db); con.execute("SET threads TO 24; SET memory_limit='60GB'; SET preserve_insertion_order=false;")
	17	+ T = os.path.join(R, "data/processed/tables"); t0 = time.time(); rep = {"created": dt.datetime.now(dt.UTC).isoformat()}
	18	+ def log(m): print(f"[{time.time()-t0:7.0f}s] {m}", flush=True)
	19	+ for t in ["works", "authorships", "work_topics", "work_keywords", "references", "abstracts", "locations", "grants", "counts_by_year"]:
	20	+ con.execute(f"CREATE OR REPLACE VIEW oa_{t} AS SELECT * FROM read_parquet('{T}/{t}/*.parquet')")
	21	+ con.execute(f"CREATE OR REPLACE VIEW oa_slim AS SELECT * FROM read_parquet('{R}/data/raw/openalex/works/slim/*.parquet')")
	22	+ con.execute(f"CREATE OR REPLACE VIEW arxiv_raw AS SELECT * FROM read_parquet('{R}/data/raw/arxiv/arxiv_cs.parquet')")
	23	+ con.execute(f"CREATE OR REPLACE VIEW acl_raw AS SELECT * FROM read_parquet('{R}/data/raw/acl/acl_anthology.parquet')")
	24	+ con.execute(f"CREATE OR REPLACE VIEW dblp_raw AS SELECT * FROM read_parquet('{R}/data/raw/dblp/dblp_publications.parquet')")
	25	+ log("views created")
	26	+ # --- normalised keys ---
	27	+ con.execute("""CREATE OR REPLACE TABLE oa_keys AS
	28	+ SELECT id AS oa_id, doi, lower(regexp_replace(arxiv_id, 'v[0-9]+$', '')) AS arxiv_id, publication_year AS year,
	29	+ regexp_replace(lower(title), '[^a-z0-9]+', '', 'g') AS ntitle, length(title) AS tlen, type
	30	+ FROM oa_works""")
	31	+ con.execute("""CREATE OR REPLACE TABLE dblp_keys AS
	32	+ SELECT key AS dblp_key, rec_type, year, doi,
	33	+ lower(regexp_replace(regexp_extract(array_to_string(ee, ' '), 'arxiv\\.org/abs/([^ ]+)', 1), 'v[0-9]+$', '')) AS arxiv_id,
	34	+ regexp_replace(lower(title), '[^a-z0-9]+', '', 'g') AS ntitle, length(title) AS tlen, stream
	35	+ FROM dblp_raw WHERE year BETWEEN 2018 AND 2026 AND rec_type IN ('article','inproceedings','incollection','book','phdthesis','data')""")
	36	+ con.execute("UPDATE dblp_keys SET arxiv_id = NULL WHERE arxiv_id = ''")
	37	+ con.execute("""CREATE OR REPLACE TABLE arxiv_keys AS
	38	+ SELECT lower(arxiv_id) AS arxiv_id, created, substr(created,1,4)::INT AS created_year, doi, primary_category, categories,
	39	+ regexp_replace(lower(title), '[^a-z0-9]+', '', 'g') AS ntitle FROM arxiv_raw""")
	40	+ con.execute("""CREATE OR REPLACE TABLE acl_keys AS
	41	+ SELECT acl_id, try_cast(year AS INT) AS year, lower(doi) AS doi, url, regexp_replace(lower(title), '[^a-z0-9]+', '', 'g') AS ntitle,
	42	+ entry_type FROM acl_raw WHERE entry_type IN ('inproceedings','article')""")
	43	+ log("keys built")
	44	+ # --- DBLP ↔ OpenAlex ---
	45	+ con.execute("""CREATE OR REPLACE TABLE link_dblp_oa AS
	46	+ SELECT d.dblp_key, o.oa_id, 'doi' AS method FROM dblp_keys d JOIN oa_keys o ON d.doi = o.doi WHERE d.doi IS NOT NULL""")
	47	+ con.execute("""INSERT INTO link_dblp_oa
	48	+ SELECT d.dblp_key, o.oa_id, 'arxiv' FROM dblp_keys d JOIN oa_keys o ON d.arxiv_id = o.arxiv_id
	49	+ WHERE d.arxiv_id IS NOT NULL AND o.arxiv_id IS NOT NULL AND d.dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa)
	50	+ AND o.oa_id NOT IN (SELECT oa_id FROM link_dblp_oa)""")
	51	+ # title+year: unique normalised titles on both sides, length >= 25, \|year diff\| <= 1
	52	+ con.execute("""CREATE OR REPLACE TEMP TABLE d_un AS SELECT ntitle, any_value(dblp_key) dblp_key, any_value(year) year FROM dblp_keys
	53	+ WHERE tlen >= 25 AND dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa) GROUP BY ntitle HAVING count(*) = 1""")
	54	+ con.execute("""CREATE OR REPLACE TEMP TABLE o_un AS SELECT ntitle, any_value(oa_id) oa_id, any_value(year) year FROM oa_keys
	55	+ WHERE tlen >= 25 AND oa_id NOT IN (SELECT oa_id FROM link_dblp_oa) GROUP BY ntitle HAVING count(*) = 1""")
	56	+ con.execute("""INSERT INTO link_dblp_oa SELECT d.dblp_key, o.oa_id, 'title_year' FROM d_un d JOIN o_un o ON d.ntitle = o.ntitle
	57	+ WHERE abs(d.year - o.year) <= 1""")
	58	+ log("dblp linked")
	59	+ # --- arXiv ↔ OpenAlex ---
	60	+ con.execute("""CREATE OR REPLACE TABLE link_arxiv_oa AS
	61	+ SELECT x.arxiv_id, o.oa_id, 'arxiv_id' AS method FROM arxiv_keys x JOIN oa_keys o ON x.arxiv_id = o.arxiv_id""")
	62	+ con.execute("""INSERT INTO link_arxiv_oa SELECT x.arxiv_id, o.oa_id, 'doi' FROM arxiv_keys x JOIN oa_keys o ON x.doi = o.doi
	63	+ WHERE x.doi IS NOT NULL AND x.arxiv_id NOT IN (SELECT arxiv_id FROM link_arxiv_oa)""")
	64	+ con.execute("""CREATE OR REPLACE TEMP TABLE x_un AS SELECT ntitle, any_value(arxiv_id) arxiv_id, any_value(created_year) year FROM arxiv_keys
	65	+ WHERE length(ntitle) >= 25 AND arxiv_id NOT IN (SELECT arxiv_id FROM link_arxiv_oa) GROUP BY ntitle HAVING count(*) = 1""")
	66	+ con.execute("""CREATE OR REPLACE TEMP TABLE o_un2 AS SELECT ntitle, any_value(oa_id) oa_id, any_value(year) year FROM oa_keys
	67	+ WHERE tlen >= 25 GROUP BY ntitle HAVING count(*) = 1""")
	68	+ con.execute("""INSERT INTO link_arxiv_oa SELECT x.arxiv_id, o.oa_id, 'title_year' FROM x_un x JOIN o_un2 o ON x.ntitle = o.ntitle
	69	+ WHERE abs(x.year - o.year) <= 2 AND o.oa_id NOT IN (SELECT oa_id FROM link_arxiv_oa)""")
	70	+ log("arxiv linked")
	71	+ # --- ACL ↔ OpenAlex ---
	72	+ con.execute("""CREATE OR REPLACE TABLE link_acl_oa AS
	73	+ SELECT c.acl_id, o.oa_id, 'doi' AS method FROM acl_keys c JOIN oa_keys o ON c.doi = o.doi WHERE c.doi IS NOT NULL""")
	74	+ con.execute("""INSERT INTO link_acl_oa SELECT c.acl_id, l.work_id, 'url' FROM acl_keys c
	75	+ JOIN (SELECT DISTINCT work_id, regexp_extract(landing_page_url, 'aclanthology\\.org/([^/ ]+)', 1) AS aid FROM oa_locations
	76	+ WHERE landing_page_url LIKE '%aclanthology.org/%') l ON l.aid = c.acl_id OR l.aid = c.acl_id \|\| '/'
	77	+ WHERE c.acl_id NOT IN (SELECT acl_id FROM link_acl_oa)""")
	78	+ con.execute("""INSERT INTO link_acl_oa SELECT c.acl_id, o.oa_id, 'title_year' FROM
	79	+ (SELECT ntitle, any_value(acl_id) acl_id, any_value(year) year FROM acl_keys WHERE length(ntitle) >= 25 AND acl_id NOT IN (SELECT acl_id FROM link_acl_oa) GROUP BY ntitle HAVING count(*)=1) c
	80	+ JOIN o_un2 o ON c.ntitle = o.ntitle WHERE abs(c.year - o.year) <= 1 AND o.oa_id NOT IN (SELECT oa_id FROM link_acl_oa)""")
	81	+ log("acl linked")
	82	+ # --- report ---
	83	+ q = lambda s: con.execute(s).fetchall()
	84	+ rep["oa_cs_records"] = q("SELECT count(*) FROM oa_keys")[0][0]
	85	+ rep["oa_cs_distinct_ids"] = q("SELECT count(DISTINCT oa_id) FROM oa_keys")[0][0]
	86	+ rep["dblp_2018_2026"] = q("SELECT count(*) FROM dblp_keys")[0][0]
	87	+ rep["dblp_with_doi"] = q("SELECT count(*) FROM dblp_keys WHERE doi IS NOT NULL")[0][0]
	88	+ rep["dblp_with_arxiv"] = q("SELECT count(*) FROM dblp_keys WHERE arxiv_id IS NOT NULL")[0][0]
	89	+ rep["link_dblp_oa"] = dict(q("SELECT method, count(*) FROM link_dblp_oa GROUP BY 1"))
	90	+ rep["dblp_linked_distinct"] = q("SELECT count(DISTINCT dblp_key) FROM link_dblp_oa")[0][0]
	91	+ rep["dblp_unlinked"] = rep["dblp_2018_2026"] - rep["dblp_linked_distinct"]
	92	+ rep["dblp_unlinked_by_type"] = dict(q("SELECT rec_type, count(*) FROM dblp_keys WHERE dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa) GROUP BY 1"))
	93	+ rep["dblp_unlinked_with_doi_in_slim"] = q("""SELECT count(*) FROM dblp_keys d JOIN oa_slim s ON d.doi = s.doi
	94	+ WHERE d.dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa)""")[0][0]
	95	+ rep["arxiv_cs_records"] = q("SELECT count(*) FROM arxiv_keys")[0][0]
	96	+ rep["arxiv_created_2018_2026"] = q("SELECT count(*) FROM arxiv_keys WHERE created_year BETWEEN 2018 AND 2026")[0][0]
	97	+ rep["link_arxiv_oa"] = dict(q("SELECT method, count(*) FROM link_arxiv_oa GROUP BY 1"))
	98	+ rep["arxiv_linked_distinct"] = q("SELECT count(DISTINCT arxiv_id) FROM link_arxiv_oa")[0][0]
	99	+ rep["acl_records"] = q("SELECT count(*) FROM acl_keys")[0][0]
	100	+ rep["acl_2018_2026"] = q("SELECT count(*) FROM acl_keys WHERE year BETWEEN 2018 AND 2026")[0][0]
	101	+ rep["link_acl_oa"] = dict(q("SELECT method, count(*) FROM link_acl_oa GROUP BY 1"))
	102	+ rep["acl_linked_distinct_2018_2026"] = q("SELECT count(DISTINCT l.acl_id) FROM link_acl_oa l JOIN acl_keys c USING(acl_id) WHERE c.year BETWEEN 2018 AND 2026")[0][0]
	103	+ # ids needed for the second snapshot pass: DBLP-indexed (2018–2026) works in slim but not in the CS subset; plus arXiv cs works in slim via DOI not in subset
	104	+ con.execute("""CREATE OR REPLACE TABLE needed_oa_ids AS
	105	+ SELECT DISTINCT s.id AS oa_id, 'dblp_doi' AS why FROM dblp_keys d JOIN oa_slim s ON d.doi = s.doi
	106	+ WHERE d.dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa) AND s.id NOT IN (SELECT oa_id FROM oa_keys)
	107	+ UNION
	108	+ SELECT DISTINCT s.id, 'arxiv_doi' FROM arxiv_keys x JOIN oa_slim s ON x.doi = s.doi
	109	+ WHERE x.doi IS NOT NULL AND x.arxiv_id NOT IN (SELECT arxiv_id FROM link_arxiv_oa) AND s.id NOT IN (SELECT oa_id FROM oa_keys)""")
	110	+ rep["needed_oa_ids_second_pass"] = dict(q("SELECT why, count(*) FROM needed_oa_ids GROUP BY 1"))
	111	+ rep["slim_total"] = q("SELECT count(*) FROM oa_slim")[0][0]
	112	+ rep["slim_cs_any_2018_2026"] = q("SELECT count(*) FROM oa_slim WHERE cs_any_topic AND publication_year BETWEEN 2018 AND 2026")[0][0]
	113	+ rep["seconds"] = round(time.time() - t0)
	114	+ os.makedirs(os.path.join(R, "data/metadata"), exist_ok=True)
	115	+ json.dump(rep, open(os.path.join(R, "data/metadata/linkage_report.json"), "w"), indent=1, default=str)
	116	+ con.execute(f"COPY (SELECT oa_id FROM needed_oa_ids) TO '{R}/data/metadata/needed_oa_ids.csv' (HEADER)")
	117	+ print(json.dumps(rep, indent=1, default=str))
	118	+
	119	+
	120	+if __name__ == "__main__":
	121	+ main()

added ingestion/openalex_api_increment.py +66 −0

Large diff (66 lines) is not loaded by default to keep the page fast.Load diff

added ingestion/openalex_entities.py +41 −0

Large diff (41 lines) is not loaded by default to keep the page fast.Load diff

added ingestion/openalex_snapshot.py +245 −0

Large diff (245 lines) is not loaded by default to keep the page fast.Load diff

added ingestion/openalex_topics_flat.py +17 −0

Large diff (17 lines) is not loaded by default to keep the page fast.Load diff

added manuscript/.latexmkrc +5 −0

Large diff (5 lines) is not loaded by default to keep the page fast.Load diff

added manuscript/Makefile +11 −0

Large diff (11 lines) is not loaded by default to keep the page fast.Load diff

added manuscript/sections/titlepage.tex +77 −0

Large diff (77 lines) is not loaded by default to keep the page fast.Load diff

added manuscript/sections/titlepage_anon.tex +53 −0

Large diff (53 lines) is not loaded by default to keep the page fast.Load diff

added manuscript/uq_logo.jpg +0 −0

Binary file not shown.

added protocol/PROTOCOL.md +96 −0

Large diff (96 lines) is not loaded by default to keep the page fast.Load diff

added protocol/amendments.csv +2 −0

Large diff (2 lines) is not loaded by default to keep the page fast.Load diff

added reports/PHASE0_FEASIBILITY.md +54 −0

Large diff (54 lines) is not loaded by default to keep the page fast.Load diff

added requirements.txt +22 −0

Large diff (22 lines) is not loaded by default to keep the page fast.Load diff

added taxonomy/dblp_streams_2018_2026.csv +3571 −0

Large diff (3,571 lines) is not loaded by default to keep the page fast.Load diff

added taxonomy/openalex_topics_2026-10-14.csv +4517 −0

Large diff (4,518 lines) is not loaded by default to keep the page fast.Load diff

		`@@ -0,0 +1,14 @@`
	1	+# PROJECT_STATUS — WP44 (updated by the executing agent; times ET)
	2	+
	3	+\| Phase \| Status \| Evidence \|
	4	+\|---\|---\|---\|
	5	+\| 0 Feasibility & protocol \| done 2026-11-02 (pending PI checkpoint H-0) \| `reports/PHASE0_FEASIBILITY.md`, `RESEARCH_PLAN.md`, `protocol/PROTOCOL.md` \|
	6	+\| 1 Bibliographic corpus \| in progress — OpenAlex snapshot ingestion running on M3U96b; DBLP parsed; ACL parsed; arXiv harvest running \| `logs/` on M3U96b, `data/metadata/` \|
	7	+\| 2 Taxonomy \| in progress — Agent D building `taxonomy/ontology.yaml` \| `agents/briefs/AGENT_D_TAXONOMY.md` \|
	8	+\| 3 Knowledge graph \| pending \| \|
	9	+\| 4 Statistics \| pending \| \|
	10	+\| 5 Literature synthesis \| pending \| \|
	11	+\| 6 Emerging-research detection \| pending \| \|
	12	+\| 7 Manuscript \| pending \| \|
	13	+\| 8 Independent audit \| pending \| \|
	14	+\| 9 Release \| pending \| \|

		`@@ -0,0 +1,10 @@`
	1	+# data/raw — provenance (nothing here is committed except this file; copies live on M3U96b:~/wp44_uqo/data/raw)
	2	+
	3	+\| Source \| What \| Obtained \| Size \| Integrity \|
	4	+\|---\|---\|---\|---\|---\|
	5	+\| OpenAlex works snapshot, release 2026-10-14 \| `data/jsonl/works/` 1,898 part files, 464,765,485 works, 675.3 GB gz \| 2026-11-02 21:19 → 23:05 ET, HTTPS from `openalex.s3.amazonaws.com`, 8 parallel streams \| filtered CS subset `works/cs/.jsonl.zst` 20 GB (15,490,159 records), slim table `works/slim/.parquet` 9.3 GB (all works) \| manifest (`works_manifest_2026-10-14.json`), per-file content length and SHA-256 in `works/ledger.sqlite` (1,898/1,898 ok, 0 parse errors, record total = manifest total) \|
	6	+\| OpenAlex entity snapshots \| topics 4,516; subfields 252; fields 26; domains 4; sources 268,629; institutions 143,455; publishers 10,713; funders 41,799; concepts 65,026 \| 2026-11-02 \| 0.6 GB \| `entities_ledger.json` (SHA-256 per file) \|
	7	+\| OpenAlex API increment \| works with any CS topic, publication_date 2026-10-01 → 2026-11-02, sliced by day × (CS primary / non-CS primary), basic paging ≤ 10,000 per slice \| 2026-11-03 (public budget $0.10/day → several days) \| raw JSON pages `api_increment/pages/` \| `api_increment/ledger.json` (counts, cost) \|
	8	+\| DBLP \| `dblp.xml.gz` (2026-11-02 22:39 UTC), `dblp.dtd`, `.md5` \| 2026-11-02 \| 1.12 GB \| MD5 `ff1c424c3b4cef36582f647c36df27c0` verified; parse report `dblp_parse_report.json` (8,840,866 records) \|
	9	+\| arXiv OAI-PMH \| set `cs`, metadataPrefix `arXiv`, datestamp 2018-01-01 → 2026-11-02, monthly windows, raw XML pages kept \| 2026-11-02/03 \| 758 pages, 915,115 unique records \| `harvest_state.json`, `arxiv_parse_report.json` \|
	10	+\| ACL Anthology \| `anthology+abstracts.bib.gz` \| 2026-11-02 \| 46.7 MB, 138,796 entries \| SHA-256 `44c3f7e907f221b740bea09069818a3d7c0dfd88b2da3b750d9f02955efc58b3` \|

spb/wp44_uqo

WP44: phase 0 (feasibility, plan, protocol) + phase 1 ingestion pipelines

Theme