SPB Git
spboucher.ai

spb/wp44_uqo

Public

WP44: phase 0 (feasibility, plan, protocol) + phase 1 ingestion pipelines

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 3, 2026) Browse files

28 changed files +10,294 −0

28 changed files
Load all 28 files Unified Split
added .gitignore +54 −0
@@ -0,0 +1,54 @@
1+# Raw & processed data — never committed (see data/raw/README.md for provenance)
2+data/raw/*
3+!data/raw/README.md
4+data/processed/*
5+!data/processed/.gitkeep
6+
7+# Python
8+__pycache__/
9+*.pyc
10+.pytest_cache/
11+.venv/
12+*.egg-info/
13+
14+# LaTeX build artifacts (paper/main.pdf IS committed)
15+paper/*.aux
16+paper/*.log
17+paper/*.out
18+paper/*.toc
19+paper/*.lof
20+paper/*.lot
21+paper/*.fls
22+paper/*.fdb_latexmk
23+paper/*.bbl
24+paper/*.blg
25+paper/*.spl
26+paper/*.synctex.gz
27+paper/*.run.xml
28+paper/*.bcf
29+
30+# Logs
31+logs/
32+*.log
33+
34+# macOS
35+.DS_Store
36+
37+# WP44 specifics
38+data/metadata/*.sqlite
39+logs/
40+*.jsonl.zst
41+*.parquet
42+!data/metadata/**/*.parquet
43+manuscript/*.aux
44+manuscript/*.log
45+manuscript/*.out
46+manuscript/*.toc
47+manuscript/*.lof
48+manuscript/*.lot
49+manuscript/*.fls
50+manuscript/*.fdb_latexmk
51+manuscript/*.bbl
52+manuscript/*.blg
53+manuscript/*.synctex.gz
54+.venv/
added CLAUDE.md +527 −0
@@ -0,0 +1,527 @@
1+# CLAUDE.md — UQO Working Paper 44
2+
3+> Mandate pasted verbatim by the principal investigator on 2026-11-02 ("fait ce projet rigoureusement"). The paste was
4+> truncated by the terminal inside §14 (after "The corpus is documented and reproduc"); the executing agent completed §14
5+> from the acceptance criteria implied by §1–§13 and recorded the completion in `protocol/amendments.csv` (A-000).
6+> Operational decisions are in `RESEARCH_PLAN.md`; progress in `PROJECT_STATUS.md`.
7+
8+## Project Identity
9+
10+**Working Paper:** WP44 — UQO Working Paper Series
11+
12+**Proposed Title:** *The Post-ChatGPT Computer Science Revolution: A Comprehensive Systematic Mapping, Bibliometric Analysis, and Knowledge Graph of Computer Science Research, 2022–2026*
13+
14+**Research period:** November 30, 2022 – November 2, 2026
15+
16+**Historical baseline:** January 1, 2018 – November 29, 2022
17+
18+**Primary language:** English
19+
20+**Target manuscript:** 300–500 pages, including extensive technical appendices.
21+
22+**Scientific objective:** Produce one of the most comprehensive, rigorously documented, reproducible computational mappings of contemporary computer science research.
23+
24+The ambition is to build an original scientific contribution, not merely a long literature summary.
25+
26+Never claim that this is the largest review ever conducted without independently establishing that claim.
27+
28+---
29+
30+# 1. NON-NEGOTIABLE SCIENTIFIC REQUIREMENTS
31+
32+The project must:
33+
34+1. Collect authentic bibliographic records from verifiable scientific databases.
35+2. Build a unified, deduplicated corpus with persistent identifiers.
36+3. Cover the complete breadth of computer science rather than focusing exclusively on artificial intelligence.
37+4. Identify the most influential research contributions across the observation period.
38+5. Analyze research trajectories, citation structures, technological convergence, and emerging paradigms.
39+6. Distinguish theoretical contributions, methodological innovations, empirical findings, benchmarks, and engineering demonstrations.
40+7. Critically evaluate conflicting findings and limitations.
41+8. Produce an evidence-linked scientific manuscript.
42+9. Make the methodology reproducible.
43+10. Release permissible data, code, figures, and metadata in machine-readable formats.
44+
45+**Strict integrity rules:**
46+
47+- Never fabricate references, DOI identifiers, citation counts, experiments, or results.
48+- Never invent paper content when only metadata is available.
49+- Never treat an abstract as equivalent to a full-text review.
50+- Never classify unverified model-generated summaries as scientific evidence.
51+- Never interpret temporal association as causation.
52+- Never describe an unpublished preprint as peer-reviewed.
53+- Never fabricate statistical significance.
54+- Record unavailable or inaccessible data explicitly.
55+- Maintain human-review checkpoints for scientific conclusions.
56+
57+# 2. PROJECT SCALE
58+
59+Initial engineering targets:
60+
61+- 1,000,000+ candidate bibliographic records, subject to actual availability.
62+- 100,000+ deeply classified research records.
63+- 10,000+ high-priority papers for detailed evidence extraction, subject to lawful full-text access.
64+- 1,000+ landmark papers for intensive critical synthesis.
65+- 30+ computer science research categories.
66+- 100+ publication-quality figures.
67+- 100+ analytical tables.
68+- 300–500 manuscript pages including appendices.
69+
70+These are planning targets, not guaranteed outcomes.
71+
72+Corpus size must never take precedence over validity, relevance, or quality.
73+
74+# 3. SCIENTIFIC DATA SOURCES
75+
76+Prioritize:
77+
78+- DBLP
79+- OpenAlex
80+- Semantic Scholar Academic Graph
81+- arXiv
82+- Crossref
83+- ACL Anthology
84+- Open-access conference proceedings
85+- Institutional repositories
86+- Publicly available benchmark repositories
87+
88+Before ingestion, verify current API specifications, rate limits, licensing, authentication, bulk-access policies, and redistribution permissions.
89+
90+Prefer official snapshots and documented bulk datasets for large-scale ingestion.
91+
92+Never bypass access controls or publisher restrictions.
93+
94+# 4. RESEARCH TAXONOMY
95+
96+Build a hierarchical taxonomy including:
97+
98+1. Large Language Models
99+2. Foundation Models
100+3. Multimodal AI
101+4. Generative AI
102+5. AI Agents and Autonomous Systems
103+6. Reinforcement Learning
104+7. Machine Learning Theory
105+8. Deep Learning Architectures
106+9. Computer Vision
107+10. Natural Language Processing
108+11. Speech and Audio Computing
109+12. Robotics
110+13. Human-Computer Interaction
111+14. Software Engineering
112+15. Programming Languages
113+16. Compilers
114+17. Formal Verification
115+18. Algorithms and Data Structures
116+19. Computational Complexity
117+20. Database Systems
118+21. Information Retrieval
119+22. Distributed Computing
120+23. Cloud Computing
121+24. Edge Computing
122+25. Computer Networks
123+26. Cybersecurity
124+27. Applied Cryptography
125+28. Blockchain Systems
126+29. Computer Architecture
127+30. Accelerators and AI Hardware
128+31. Operating Systems
129+32. Quantum Computing
130+33. Scientific Computing
131+34. Computational Biology
132+35. Computer Graphics
133+36. Privacy-Preserving Computation
134+37. Federated Learning
135+38. AI Safety and Alignment
136+39. Sustainable Computing
137+40. Autonomous Scientific Discovery
138+
139+Allow papers to belong to multiple categories.
140+
141+Construct an extensible hierarchical ontology rather than a flat keyword classification.
142+
143+# 5. SCIENTIFIC QUESTIONS
144+
145+The paper must investigate:
146+
147+**RQ1:** How has the volume and composition of computer science research evolved since November 2022?
148+
149+**RQ2:** Which subfields have experienced the strongest acceleration or structural change?
150+
151+**RQ3:** Which publications introduced the most influential methodological or theoretical contributions?
152+
153+**RQ4:** How have citation networks and interdisciplinary knowledge flows evolved?
154+
155+**RQ5:** Has the emergence of foundation models changed the relationship between previously distinct computer science disciplines?
156+
157+**RQ6:** How have research institutions, countries, and collaboration networks changed?
158+
159+**RQ7:** Which research areas demonstrate genuine breakthroughs versus incremental benchmark improvements?
160+
161+**RQ8:** Which scientific problems remain unresolved despite rapid publication growth?
162+
163+**RQ9:** How have reproducibility, open-source availability, and evaluation standards evolved?
164+
165+**RQ10:** Can computational models detect emerging scientific paradigms before they become dominant?
166+
167+# 6. RESEARCH METHODOLOGY
168+
169+Implement multiple complementary analytical approaches.
170+
171+## 6.1 Systematic Mapping
172+
173+Follow a preregistered protocol where practical.
174+
175+Use PRISMA 2020 reporting principles where applicable.
176+
177+Document:
178+
179+- Search strategies
180+- Database coverage
181+- Eligibility criteria
182+- Deduplication
183+- Screening decisions
184+- Full-text availability
185+- Exclusion reasons
186+- Quality assessment
187+- Final synthesis populations
188+
189+Separate broad bibliometric mapping from focused systematic reviews.
190+
191+## 6.2 Bibliometric Analysis
192+
193+Estimate:
194+
195+- Publication growth
196+- Citation growth
197+- Field-normalized impact
198+- Citation velocity
199+- Collaboration networks
200+- Institutional influence
201+- Country-level participation
202+- Venue concentration
203+- Citation inequality
204+- Research topic diversification
205+
206+Account for citation-age bias, database coverage, and preprint duplication.
207+
208+## 6.3 Semantic Research Mapping
209+
210+Use scientific document embeddings, topic modeling, hierarchical clustering, and temporal semantic analysis.
211+
212+Compare multiple embedding approaches and validate topic stability.
213+
214+Construct a dynamic semantic atlas of computer science.
215+
216+## 6.4 Scientific Knowledge Graph
217+
218+Create nodes for:
219+
220+- Papers
221+- Authors
222+- Institutions
223+- Venues
224+- Research topics
225+- Algorithms
226+- Models
227+- Datasets
228+- Benchmarks
229+- Scientific claims
230+
231+Create evidence-supported relationships including:
232+
233+- Cites
234+- Extends
235+- Evaluates
236+- Contradicts
237+- Reproduces
238+- Uses dataset
239+- Introduces method
240+- Addresses research problem
241+
242+Every inferred relationship must include provenance, confidence, and verification status.
243+
244+## 6.5 Structural Change Analysis
245+
246+Use interrupted time-series models, changepoint detection, dynamic network models, and appropriate robustness checks.
247+
248+The November 2022 boundary is a historical reference point, not proof of a causal ChatGPT effect.
249+
250+Include pre-period trend diagnostics, placebo breakpoints, alternative event dates, and sensitivity analysis.
251+
252+## 6.6 Emerging Research Detection
253+
254+Develop an original computational framework for identifying emerging scientific fields.
255+
256+Candidate signals:
257+
258+- Abnormal publication growth
259+- Semantic novelty
260+- Citation acceleration
261+- Interdisciplinary connectivity
262+- Methodological adoption
263+- Benchmark proliferation
264+- Open-source ecosystem activity
265+
266+Evaluate the framework retrospectively using temporally separated training and test periods.
267+
268+Avoid look-ahead bias.
269+
270+# 7. AUTONOMOUS RESEARCH AGENTS
271+
272+Use Claude Code subagents with explicit responsibilities.
273+
274+**Agent A — Research Director:** Coordinates the scientific program and validates deliverables.
275+
276+**Agent B — Bibliographic Acquisition:** Collects metadata through legitimate APIs and snapshots.
277+
278+**Agent C — Data Engineering:** Builds normalized databases and reproducible ingestion pipelines.
279+
280+**Agent D — Taxonomy Architect:** Constructs the computer science ontology.
281+
282+**Agent E — AI Research Specialist:** Reviews AI, machine learning, foundation models, and agents.
283+
284+**Agent F — Systems Specialist:** Reviews distributed systems, operating systems, databases, and hardware.
285+
286+**Agent G — Theory Specialist:** Reviews algorithms, complexity, formal methods, and programming languages.
287+
288+**Agent H — Security Specialist:** Reviews cybersecurity, privacy, and cryptography.
289+
290+**Agent I — Bibliometrician:** Conducts citation and collaboration analyses.
291+
292+**Agent J — Econometrician:** Designs temporal models, statistical inference, and robustness tests.
293+
294+**Agent K — Scientific Critic:** Challenges unsupported claims and methodological weaknesses.
295+
296+**Agent L — Academic Writer:** Produces publication-quality scientific prose.
297+
298+**Agent M — LaTeX Engineer:** Manages equations, references, tables, figures, and compilation.
299+
300+**Agent N — Reproducibility Auditor:** Independently validates computations and evidence.
301+
302+**Agent O — Visualization Specialist:** Creates publication-quality research graphics.
303+
304+Agents must exchange structured artifacts, not undocumented conversational summaries.
305+
306+No agent may approve its own unverified scientific findings.
307+
308+# 8. MACLUSTR INFRASTRUCTURE
309+
310+Discover the available compute nodes, CPU, GPU, memory, storage, networking, and existing services before implementation.
311+
312+Use distributed execution only when it materially improves throughput.
313+
314+Suggested stack:
315+
316+- Python
317+- DuckDB
318+- PostgreSQL
319+- Parquet and Apache Arrow
320+- Polars
321+- NetworkX or igraph
322+- Scientific embedding models
323+- Vector search
324+- PyTorch where appropriate
325+- LaTeX
326+- Git
327+- Containerized reproducible pipelines
328+
329+Use incremental ingestion, checkpointing, resumable processing, and content hashes.
330+
331+Preserve immutable raw-source snapshots.
332+
333+Implement job-level provenance, retry logic, and resource monitoring.
334+
335+Do not assume access to credentials or services before testing them.
336+
337+# 9. MANUSCRIPT STRUCTURE
338+
339+Produce the following major parts.
340+
341+**Part I — Introduction and Scientific Motivation**
342+
343+Explain the research problem, historical context, contributions, and scope.
344+
345+**Part II — The Evolution of Computer Science**
346+
347+Provide the conceptual and historical background needed to interpret post-2022 developments.
348+
349+**Part III — Data and Systematic Review Protocol**
350+
351+Explain corpus construction, source coverage, deduplication, eligibility, and research integrity.
352+
353+**Part IV — Computational Methodology**
354+
355+Develop the mathematical framework for bibliometrics, semantic mapping, networks, and structural-change analysis.
356+
357+**Part V — The Artificial Intelligence Transformation**
358+
359+Critically synthesize foundation models, multimodal learning, agents, reasoning, efficiency, evaluation, and safety.
360+
361+**Part VI — Systems and Computing Infrastructure**
362+
363+Review hardware, distributed systems, databases, networks, and operating systems.
364+
365+**Part VII — Theory, Algorithms, and Security**
366+
367+Review theoretical computer science, formal methods, cryptography, and cybersecurity.
368+
369+**Part VIII — Scientific Knowledge Networks**
370+
371+Analyze authors, institutions, citations, interdisciplinarity, and knowledge diffusion.
372+
373+**Part IX — Empirical Results**
374+
375+Present all verified quantitative findings with publication-quality figures and tables.
376+
377+**Part X — The Next Research Frontiers**
378+
379+Identify unresolved scientific problems, competing hypotheses, and testable research directions.
380+
381+**Part XI — Discussion and Conclusion**
382+
383+Explain contributions, limitations, generalizability, and implications for computer science.
384+
385+**Appendices A–Z**
386+
387+Include detailed taxonomy, search strategies, mathematical derivations, robustness checks, data dictionaries, supplementary tables, figure documentation, and reproducibility instructions.
388+
389+Do not pad the manuscript to meet page targets.
390+
391+# 10. ACADEMIC WRITING STANDARD
392+
393+Write in formal, precise, technically sophisticated academic English.
394+
395+Target the scholarly standards of leading computer science journals and major survey venues.
396+
397+The writing must:
398+
399+- Critically compare competing approaches.
400+- Distinguish established findings from speculation.
401+- Present meaningful mathematical notation.
402+- Explain assumptions and limitations.
403+- Integrate citations into the scientific argument.
404+- Avoid promotional or exaggerated language.
405+- Avoid repetitive AI-generated prose.
406+- Include genuine original synthesis.
407+
408+All references must be resolved to verified bibliographic records.
409+
410+# 11. FIGURES AND TABLES
411+
412+Create publication-quality visualizations including:
413+
414+- Annual and monthly publication trends
415+- Research domain growth
416+- Dynamic topic maps
417+- Citation networks
418+- Scientific influence distributions
419+- Institutional collaboration networks
420+- Country-level research activity
421+- Topic emergence and decline
422+- Interdisciplinary knowledge flows
423+- Methodological genealogy diagrams
424+- Structural-break estimates
425+- Benchmark evolution
426+- Research reproducibility indicators
427+- Uncertainty and robustness visualizations
428+
429+All figures must be generated from reproducible scripts.
430+
431+Use vector formats when appropriate.
432+
433+No decorative or fabricated empirical graphics.
434+
435+# 12. OUTPUT STRUCTURE
436+
437+Create:
438+
439+```text
440+WP44/
441+├── CLAUDE.md
442+├── README.md
443+├── manuscript/
444+│ ├── main.tex
445+│ ├── sections/
446+│ ├── appendices/
447+│ └── references.bib
448+├── data/
449+│ ├── raw/
450+│ ├── processed/
451+│ └── metadata/
452+├── ingestion/
453+├── taxonomy/
454+├── embeddings/
455+├── knowledge_graph/
456+├── bibliometrics/
457+├── econometrics/
458+├── systematic_review/
459+├── literature_synthesis/
460+├── figures/
461+├── tables/
462+├── experiments/
463+├── agents/
464+├── tests/
465+├── reports/
466+└── reproducibility/
467+```
468+
469+# 13. EXECUTION PHASES
470+
471+**Phase 0 — Feasibility and Scientific Protocol**
472+
473+Audit available infrastructure, verify data sources, define inclusion criteria, and prepare the research protocol.
474+
475+**Phase 1 — Bibliographic Corpus**
476+
477+Build ingestion pipelines and create the deduplicated research database.
478+
479+**Phase 2 — Scientific Taxonomy**
480+
481+Classify the corpus and validate the ontology using manually reviewed samples.
482+
483+**Phase 3 — Knowledge Graph**
484+
485+Construct citation, semantic, institutional, and methodological relationships.
486+
487+**Phase 4 — Statistical Research**
488+
489+Estimate bibliometric trends, network measures, and temporal models.
490+
491+**Phase 5 — Critical Literature Synthesis**
492+
493+Generate evidence-grounded disciplinary reviews and reconcile conflicting findings.
494+
495+**Phase 6 — Original Scientific Contribution**
496+
497+Develop and validate the emerging-research detection framework.
498+
499+**Phase 7 — Manuscript Construction**
500+
501+Write the full LaTeX paper and comprehensive appendices.
502+
503+**Phase 8 — Independent Scientific Audit**
504+
505+Use independent review agents to verify citations, statistical results, claims, equations, and reproducibility.
506+
507+**Phase 9 — Final Research Release**
508+
509+Compile the final PDF, archive reproducibility artifacts, prepare the dataset documentation, and produce a release-ready research repository.
510+
511+Do not advance to the next phase until required validation checks pass.
512+
513+# 14. ACCEPTANCE CRITERIA
514+
515+The project is complete only when:
516+
517+- The corpus is documented and reproduc[ible — paste truncated here. Completed by the executing agent (A-000):]
518+- The corpus is documented and reproducible from the immutable raw snapshots, with record counts at every PRISMA stage.
519+- Every quantitative claim in the manuscript traces to a script, an input artefact and a result file.
520+- Every reference in `references.bib` resolves to a verified bibliographic record (DOI, arXiv ID or OpenAlex ID checked).
521+- The taxonomy has been validated on a manually reviewed sample with reported agreement statistics.
522+- Structural-change results include pre-trend diagnostics, placebo breakpoints and sensitivity analyses.
523+- The emerging-research detection framework has been evaluated out-of-time without look-ahead.
524+- Independent audit agents have verified citations, statistics, equations and reproducibility, and their reports are archived.
525+- The LaTeX manuscript compiles without unresolved references, and the release repository contains code, metadata,
526+ figures, tables and documentation under an explicit licence.
527+- Human-review checkpoints for scientific conclusions are recorded and any unreviewed conclusion is flagged as such.
added PROJECT_STATUS.md +14 −0
@@ -0,0 +1,14 @@
1+# PROJECT_STATUS — WP44 (updated by the executing agent; times ET)
2+
3+| Phase | Status | Evidence |
4+|---|---|---|
5+| 0 Feasibility & protocol | done 2026-11-02 (pending PI checkpoint H-0) | `reports/PHASE0_FEASIBILITY.md`, `RESEARCH_PLAN.md`, `protocol/PROTOCOL.md` |
6+| 1 Bibliographic corpus | in progress — OpenAlex snapshot ingestion running on M3U96b; DBLP parsed; ACL parsed; arXiv harvest running | `logs/` on M3U96b, `data/metadata/` |
7+| 2 Taxonomy | in progress — Agent D building `taxonomy/ontology.yaml` | `agents/briefs/AGENT_D_TAXONOMY.md` |
8+| 3 Knowledge graph | pending | |
9+| 4 Statistics | pending | |
10+| 5 Literature synthesis | pending | |
11+| 6 Emerging-research detection | pending | |
12+| 7 Manuscript | pending | |
13+| 8 Independent audit | pending | |
14+| 9 Release | pending | |
added README.md +75 −0
@@ -0,0 +1,75 @@
1+<!-- Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca -->
2+# 🧭 The Post-ChatGPT Computer Science Revolution
3+
4+**UQO Working Paper No. 44** — *A Comprehensive Systematic Mapping, Bibliometric Analysis, and Knowledge Graph of Computer Science Research, 2022–2026*
5+
6+[![Paper](https://img.shields.io/badge/paper-in%20preparation-lightgrey?logo=latex&logoColor=white)](manuscript/)
7+[![Python](https://img.shields.io/badge/python-3.12-3776AB?logo=python&logoColor=white)](requirements.txt)
8+[![Data](https://img.shields.io/badge/data-OpenAlex%202026--10--14%20·%20DBLP%20·%20arXiv%20·%20ACL-4051b5)](data/raw/README.md)
9+[![Reproducible](https://img.shields.io/badge/reproducible-ledgers%20%2B%20hashes-2e7d32)](reproducibility/)
10+[![Institution](https://img.shields.io/badge/UQO-D%C3%A9pt.%20des%20sciences%20administratives-16365c)](https://uqo.ca)
11+[![Author](https://img.shields.io/badge/author-Simon--Pierre%20Boucher-16365c)](mailto:simon-pierre.boucher@uqo.ca)
12+
13+> **TL;DR** — Field-level systematic mapping of computer-science research from 2018-01-01 to 2026-11-02 (baseline
14+> 2018 → 2022-11-29; post-ChatGPT window 2022-11-30 → 2026-11-02) built from the full OpenAlex public snapshot
15+> (release 2026-10-14; 464.8 M works scanned, 15.49 M with a computer-science topic in 2018–2026), the DBLP dump of
16+> 2026-11-02 (8.84 M records), the arXiv `cs` OAI-PMH harvest (915 k records) and the ACL Anthology (139 k entries),
17+> linked, deduplicated and classified into a 40-category ontology; bibliometric, network, semantic, structural-change and
18+> emerging-topic analyses; knowledge graph with provenance; preregistered protocol (`protocol/PROTOCOL.md`).
19+> Status: Phase 1 (corpus) in progress — see `PROJECT_STATUS.md`.
20+
21+---
22+
23+## 📌 Status
24+
25+| Item | Value |
26+|---|---|
27+| Version | 0.1 (November 2026, in preparation) |
28+| SSRN | not yet submitted |
29+| Journal | — |
30+| Code & data | reproducible from `ingestion/` → `bibliometrics/` → `manuscript/` (see Quick start); raw snapshots not redistributed (manifests, ledgers and SHA-256 hashes are) |
31+
32+## 🎯 What this paper does
33+
34+Mandate: `CLAUDE.md` (verbatim). Design: `RESEARCH_PLAN.md`. Protocol: `protocol/PROTOCOL.md` (frozen 2026-11-02;
35+amendments in `protocol/amendments.csv`). Feasibility audit: `reports/PHASE0_FEASIBILITY.md`.
36+
37+## 🗂 Repository layout
38+
39+```
40+wp44/ (spbgit: wp44_uqo.git)
41+├── CLAUDE.md · RESEARCH_PLAN.md · PROJECT_STATUS.md · protocol/{PROTOCOL.md,amendments.csv}
42+├── ingestion/ openalex_snapshot.py (S3 snapshot → CS JSONL + slim table, SHA-256 ledger), openalex_entities.py,
43+│ openalex_api_increment.py, dblp_parse.py, dblp_streams.py, arxiv_oai.py, acl_parse.py,
44+│ build_corpus.py (normalised Parquet tables), link_sources.py (record linkage, DuckDB)
45+├── taxonomy/ ontology.yaml (40 leaves, 8 areas), topic_map.csv, ONTOLOGY.md, src/ (mapping data), build_ontology.py
46+├── embeddings/ · knowledge_graph/ · bibliometrics/ · econometrics/ · systematic_review/ · literature_synthesis/
47+├── experiments/ emerging-research detection framework
48+├── agents/ briefs/ (one per agent), REGISTER.md (artefacts exchanged, who verified what)
49+├── data/ raw/README.md (provenance), metadata/ (manifests, ledgers, PRISMA counts, linkage report), processed/ (not in git)
50+├── figures/ · tables/ · reports/ · tests/ · reproducibility/
51+└── manuscript/ main.tex, sections/, appendices/, references.bib, figures/, tables/ (mandate §12 layout)
52+```
53+
54+## 🚀 Quick start (compute node M3U96b, `~/wp44_uqo`)
55+
56+```bash
57+uv venv --python 3.12 .venv && . .venv/bin/activate && uv pip install -r requirements.txt
58+python ingestion/openalex_snapshot.py --out data/raw/openalex/works --manifest data/raw/openalex/works_manifest_2026-10-14.json --workers 8
59+python ingestion/openalex_entities.py --out data/raw/openalex
60+python ingestion/dblp_parse.py --xml data/raw/dblp/dblp.xml.gz --out data/raw/dblp
61+python ingestion/arxiv_oai.py --out data/raw/arxiv --from 2018-01-01 --until 2026-11-02
62+python ingestion/acl_parse.py --bib data/raw/acl/anthology+abstracts.bib.gz --out data/raw/acl
63+python ingestion/build_corpus.py --cs-dir data/raw/openalex/works/cs --out data/processed --workers 12
64+python ingestion/link_sources.py --root ~/wp44_uqo
65+```
66+
67+## 🗃 Data
68+
69+See `data/raw/README.md` and `reports/PHASE0_FEASIBILITY.md`. Licences: OpenAlex CC0; DBLP ODC-BY 1.0; arXiv metadata
70+(CC0 terms of the OAI-PMH feed); ACL Anthology CC BY 4.0. Semantic Scholar was not used (API key required; recorded).
71+
72+## 📝 Notes (deviations from the UQO working-paper standard)
73+
74+- The manuscript lives in `manuscript/` (mandate §12) instead of `paper/`; the title page and preamble follow the UQO template.
75+- Phase status and human-review checkpoints: `PROJECT_STATUS.md`, `protocol/PROTOCOL.md` §P7.
added RESEARCH_PLAN.md +131 −0
@@ -0,0 +1,131 @@
1+# WP44 — Research plan (operational design, v1.0, 2026-11-02)
2+
3+This document turns the mandate (`CLAUDE.md`) into the concrete design implemented by the code, the protocol and the
4+manuscript. Decisions the mandate left open are listed in §10 with their rationale; they are the executing agent's until
5+the principal investigator (PI) confirms them (human-review checkpoint H-0).
6+
7+## 1. Scientific object and estimands
8+
9+Population: the computer-science research literature indexed by at least one of DBLP, OpenAlex (any topic in field 17,
10+Computer Science), arXiv (set `cs`), or the ACL Anthology, first published between 2018-01-01 and 2026-11-02.
11+Two windows: **baseline** B = [2018-01-01, 2022-11-29] and **post** P = [2022-11-30, 2026-11-02]. The boundary is the public
12+release of ChatGPT; it is a reference date for description and for interrupted-time-series designs, *not* a causal
13+treatment (CLAUDE.md §6.5). Monthly series are the primary temporal unit; years are secondary.
14+
15+Primary descriptive estimands (RQ1, RQ2): monthly counts of works per category c, N_{c,t}; composition shares
16+s_{c,t} = N_{c,t}/N_t; growth rates and their pre/post structural-break statistics; citation-based influence (field- and
17+age-normalised); collaboration/interdisciplinarity indices (RQ4–RQ6). Inferential estimands (RQ2, RQ10): breakpoint
18+locations and magnitudes with placebo distributions; out-of-time precision/recall of the emerging-topic detector.
19+
20+## 2. Sources, access and what each contributes (verified 2026-11-02, see `reports/PHASE0_FEASIBILITY.md`)
21+
22+| Source | Access used | Licence | Role in the corpus |
23+|---|---|---|---|
24+| OpenAlex public snapshot, release 2026-10-14 | S3 public bucket, JSONL, 1,898 works files (675 GB), ~110 MB/s from M3U96b | CC0 | backbone: works, authorships + institutions + countries, venues, topics (field/subfield/topic), `referenced_works` (citation graph), abstracts (inverted index), OA status, FWCI, counts_by_year |
25+| OpenAlex API (no key, $0.10/day shared budget) | REST, polite `mailto` | CC0 | incremental works 2026-10-14 → 2026-11-02; spot checks; free single-record lookups for reference verification |
26+| DBLP XML dump (2026-11-02, 1.12 GB gz, MD5 verified) | HTTPS | ODC-BY 1.0 | CS-scope gold standard, venue streams, author PIDs, DOI links; defines "CS venue" |
27+| arXiv OAI-PMH, set `cs`, 2018-01 → 2026-11 | OAI-PMH, 3.5 s/request, raw XML kept | arXiv metadata terms (CC0 for metadata) | preprints, categories (cs.XX, cross-lists), versions, licences; preprint–publication linking |
28+| ACL Anthology BibTeX (anthology+abstracts.bib, 138,796 entries) | HTTPS | CC BY 4.0 | NLP venue coverage check, abstracts for ACL papers |
29+| Crossref REST API (polite pool) | REST | open metadata | reference verification, `is-referenced-by-count` cross-check sample, publisher/type cross-check |
30+| Semantic Scholar Academic Graph | **not available**: API key required for bulk/dataset access; anonymous calls return HTTP 429 | — | recorded as unavailable (CLAUDE.md §1: record inaccessible data explicitly); influential-citation and citation-intent signals are therefore not used |
31+| Papers with Code / benchmark repositories, GitHub | HTTPS, public dumps where available | varies | RQ7/RQ9 (benchmark proliferation, code availability) — scoped in Phase 3 |
32+| Full texts | arXiv e-prints (rate-limited, licence-respecting), OA locations from OpenAlex/Unpaywall | per-paper licence | ≤ 10,000 high-priority papers; never through paywalls |
33+
34+No credential was assumed: the OpenAlex key, Semantic Scholar key and Kaggle credentials were tested as absent.
35+
36+## 3. Corpus construction (Phase 1) — PRISMA-style stages
37+
38+S0 Identification: all OpenAlex works with any topic in field 17 and `publication_year` 2018–2026 (full records kept),
39+plus DBLP records 2018–2026, arXiv `cs` records created 2018–2026, ACL entries 2018–2026. A *slim* table of **every**
40+OpenAlex work (465 M rows: id, DOI, year, type, primary topic/subfield/field/domain, cited_by_count, flags) is kept to resolve
41+cited works outside CS.
42+S1 Record linkage: OpenAlex ↔ DBLP by DOI (lower-cased), else by arXiv ID (DBLP `ee` → arxiv.org/abs), else by normalised
43+title + year + first-author surname (blocking on title hash); OpenAlex ↔ arXiv by `locations[].landing_page_url`/`ids`
44+and DOI (10.48550/arXiv.*); ACL by DOI/URL. Linkage precision is estimated on a 500-pair manually reviewed sample (H-1).
45+S2 Deduplication: one node per *work*; preprint and published version are merged into a single node with
46+`preprint_date`, `publication_date`, `version_relationship` kept (CLAUDE.md §6.2: preprint duplication).
47+S3 Eligibility: (i) `publication_year` in window; (ii) type in {article, preprint, book-chapter, proceedings-article,
48+dissertation, report, book} — exclude paratext, editorials, errata, retracted (kept with flag `is_retracted`), XPAC
49+expansion records (`is_xpac`); (iii) CS membership: DBLP-indexed **or** OpenAlex primary field = CS **or** (secondary CS
50+topic **and** arXiv primary category cs.*). Secondary-only CS topics without DBLP/arXiv support form a *penumbra* set
51+reported separately. Sensitivity: results re-estimated on the strict core (DBLP ∩ OpenAlex primary CS).
52+S4 Classification: 40-category hierarchical ontology (`taxonomy/ontology.yaml`), multi-label, assigned by rules over
53+OpenAlex topics/subfields, arXiv categories, DBLP venue streams and keyword patterns; validated on manually reviewed
54+samples (Phase 2, H-2) and by an LLM-based second rater with reported agreement.
55+S5 Synthesis populations: (a) mapping corpus (all eligible); (b) deeply classified set (≥ 100 k: abstracts + embeddings +
56+full feature vectors); (c) high-priority set (≥ 10 k: citation-ranked within category × year with OA full text);
57+(d) landmark set (≥ 1 k: top field-normalised impact, reviewed by specialist agents and flagged for PI review, H-3).
58+
59+Every stage writes counts to `data/metadata/prisma_counts.json`, rendered as the PRISMA flow figure.
60+
61+## 4. Taxonomy (Phase 2)
62+
63+`taxonomy/ontology.yaml`: 40 leaf categories under 8 top-level areas (AI & ML; Vision, Language & Perception; Systems &
64+Infrastructure; Theory & Foundations; Security, Privacy & Cryptography; Software & Human Factors; Emerging Computing
65+Paradigms; Computing & Science). Each leaf has: definition, parent, OpenAlex topic IDs (weighted), subfield IDs, arXiv
66+categories, DBLP streams, keyword regexes with weights, exclusions, and provenance of the mapping. Assignment score per
67+category = weighted sum of matched signals; multi-label with threshold; `primary_category` = argmax. Validation: 2,000
68+stratified works (50 per category) rated by two independent LLM raters with the rubric in `taxonomy/VALIDATION.md`,
69+Cohen's κ and per-category precision/recall; a 200-work subsample is flagged for PI review (H-2).
70+
71+## 5. Knowledge graph (Phase 3)
72+
73+Nodes: works, authors (OpenAlex A-ids), institutions, countries, venues (sources), topics, taxonomy categories, arXiv
74+categories; *method/model/dataset/benchmark* and *claim* nodes only for the high-priority set, extracted from abstracts
75+(and full text where lawfully available) with a schema-constrained LLM extractor; each extracted edge carries
76+`provenance` (work id + text span), `confidence`, `verification_status` ∈ {unverified, machine-verified, human-verified}.
77+Citation edges come from `referenced_works` (verified by construction). Semantic edges from embeddings (k-NN, cosine).
78+Storage: Parquet tables + DuckDB views; graph analytics with igraph (python-igraph) on M3U96b (96 GB RAM).
79+
80+## 6. Statistics (Phase 4) and structural change
81+
82+- Bibliometrics: monthly/annual counts; shares; CAGR; field-normalised citation score FNCS = c / E[c | field, year, type];
83+ citation velocity (citations within 12/24 months, computed from `counts_by_year`); Gini/Hirsch-type inequality; venue
84+ concentration (HHI); author/institution/country collaboration networks; Rao–Stirling and Leinster–Cobbold diversity.
85+- Citation-age bias: all cross-period comparisons use fixed citation windows and year-normalisation; FWCI used as a
86+ secondary measure with its known limits.
87+- Structural change: interrupted time series on log monthly counts with seasonality (OLS-HAC, Newey–West), Bai–Perron
88+ multiple-breakpoint estimation, PELT/binary-segmentation changepoints, Bayesian online changepoint detection; pre-trend
89+ diagnostics (placebo breaks at every month of the baseline), alternative event dates (2020-05 GPT-3, 2022-04 DALL·E 2,
90+ 2022-11 ChatGPT, 2023-03 GPT-4, 2024-09 o1), donut windows; sensitivity to OpenAlex indexing lag (right-truncation
91+ corrected by indexing-delay profiles estimated from `created_date`).
92+- Dynamic networks: yearly topic co-occurrence and citation-flow matrices between taxonomy categories; modularity and
93+ cross-area flow shares; ERGM-style descriptives are out of scope.
94+
95+## 7. Emerging-research detection (Phase 6, original contribution)
96+
97+Unit: topic candidates = (i) OpenAlex topics, (ii) semantic clusters (HDBSCAN on embeddings, re-estimated per
98+training cutoff without look-ahead), (iii) keyword n-grams. Signals at month t using data ≤ t only: growth anomaly,
99+semantic novelty (distance to pre-t centroid), citation acceleration, interdisciplinary connectivity, methodological
100+adoption (share of works citing a method node), benchmark proliferation, open-source activity. Label at t+h: top-decile
101+growth and citation share at horizon h ∈ {12, 24} months. Training cutoffs 2019–2022; test cutoffs 2023–2024 with
102+outcomes observed up to 2026-11. Models: regularised logistic, gradient boosting, and a simple z-score baseline;
103+metrics: AUC, precision@k, lead time. Known-emergence validation set: LLMs, diffusion models, NeRF, agents, RAG,
104+state-space models, vision-language models, mechanistic interpretability (defined before estimation, listed in
105+`protocol/PROTOCOL.md`).
106+
107+## 8. Agents and artefacts
108+
109+The executing session is Agent A (Research Director). Subagents are spawned with written briefs (`agents/briefs/`),
110+return structured artefacts (JSON/CSV/Markdown with declared schema) and never approve their own findings: every
111+scientific artefact is checked by a different agent (K or N) before it enters the manuscript (`agents/REGISTER.md`).
112+
113+## 9. Infrastructure
114+
115+Primary compute/storage: **M3U96b** (M3 Ultra 28 c, 96 GB, 375 GB free; `~/wp44_uqo/`, venv Python 3.12 via uv:
116+orjson, pyarrow, polars, duckdb, zstandard, lxml, requests). Secondary: M4M64a (540 GB free) for embeddings; other LAN
117+nodes for embarrassingly parallel embedding/extraction batches when it materially helps. Laptop = controller and git
118+repository (`Recherche-UQO/informatique/wp44`, origin `gitsrv:srv/git/wp44_uqo.git`). Data are not in git; ledgers,
119+manifests, hashes and PRISMA counts are.
120+
121+## 10. Decisions taken by the executing agent (pending PI confirmation, H-0)
122+
123+| # | Decision | Rationale |
124+|---|---|---|
125+| D1 | Snapshot over API for OpenAlex | mandate §3 prefers snapshots; the API budget without a key is $0.10/day (≈ 200 k works/day) |
126+| D2 | Keep a slim table of all 465 M works | required for interdisciplinary citation flows (RQ4) and cited-work normalisation |
127+| D3 | Semantic Scholar not used | key required; recorded as unavailable rather than scraped |
128+| D4 | CS membership rule (§3 S3) with strict-core sensitivity | avoids both AI-only bias and over-inclusion from secondary topics |
129+| D5 | Manuscript directory `manuscript/` (mandate §12) instead of the UQO `paper/` convention | mandate is explicit; deviation noted in README |
130+| D6 | Historical raw snapshot kept as filtered subset + per-file SHA-256 + manifest, not the 675 GB bucket copy | public versioned release; hashes allow verification; storage |
131+| D7 | LLM raters (Claude) used for taxonomy validation and knowledge-graph extraction, with PI review checkpoints | mandate §1 requires human checkpoints; PI is the human |
added agents/briefs/AGENT_D_TAXONOMY.md +77 −0
@@ -0,0 +1,77 @@
1+# Brief — Agent D (Taxonomy Architect), WP44
2+
3+Goal: produce the extensible hierarchical ontology of computer-science research used to classify every work of the WP44
4+corpus into the 40 leaf categories of CLAUDE.md §4 (multi-label), with a deterministic mapping from the signals available
5+in the corpus. The mapping is *data*, not code: it must be complete, explicit and justified so that another agent can audit it.
6+
7+Inputs (read-only):
8+- `taxonomy/openalex_topics_2026-10-14.csv` — the 4,516 OpenAlex topics (topic_id, display_name, subfield, field, domain,
9+ works_count, keywords, description). 302 topics are in field 17 (Computer Science); relevant topics also exist in
10+ Mathematics (26), Engineering (22), Decision Sciences (18), Physics (31: quantum), Biochemistry (13: bioinformatics), etc.
11+- `taxonomy/dblp_streams_2018_2026.csv` — 3,570 DBLP venue streams (journal/conference series keys) with ≥ 100 records
12+ in 2018–2026, counts and names. `journals/corr` = arXiv CoRR (not a venue signal).
13+- arXiv categories: cs.AI cs.AR cs.CC cs.CE cs.CG cs.CL cs.CR cs.CV cs.CY cs.DB cs.DC cs.DL cs.DM cs.DS cs.ET cs.FL cs.GL
14+ cs.GR cs.GT cs.HC cs.IR cs.IT cs.LG cs.LO cs.MA cs.MM cs.MS cs.NA cs.NE cs.NI cs.OH cs.OS cs.PF cs.PL cs.RO cs.SC cs.SD
15+ cs.SE cs.SI cs.SY, plus cross-listed stat.ML, eess.AS, eess.IV, eess.SP, eess.SY, quant-ph, math.OC, math.IT, math.ST,
16+ q-bio.QM, q-bio.GN, physics.comp-ph.
17+
18+Deliverables (write these files; nothing else):
19+1. `taxonomy/ontology.yaml` with this schema:
20+ ```yaml
21+ version: "1.0"
22+ created: "2026-11-02"
23+ areas: # 8 top-level areas (choose names; every leaf belongs to exactly one area)
24+ - id: ai_ml
25+ name: ...
26+ leaves: [llm, foundation_models, ...]
27+ categories: # exactly 40 leaves, ids snake_case, in the order of CLAUDE.md §4
28+ - id: llm
29+ number: 1
30+ name: Large Language Models
31+ area: ai_ml
32+ definition: one or two precise sentences (what is in, what is out)
33+ openalex_topics: # list of {id: T10028, weight: 1.0, note: ...}; weight in (0,1]; include topics from any field
34+ openalex_subfields: # list of {id: 1702, weight: 0.3} — coarse fallback only
35+ arxiv_categories: # list of {id: cs.CL, weight: 0.5}
36+ dblp_streams: # list of {key: conf/acl, weight: 1.0}
37+ keywords: # list of {pattern: "(?i)\\blarge language model", weight: 1.0, field: title|abstract|both}
38+ exclusions: # optional list of regex patterns that veto the category
39+ related: [foundation_models, nlp] # ontology edges (is-related-to); also use `parent_of`/`child_of` where a leaf is a specialisation of another (e.g. llm child_of foundation_models; federated_learning child_of privacy_preserving_computation)
40+ provenance: short note on how the mapping was built
41+ assignment_rule:
42+ score: "sum of weights of matched signals, each signal type capped at 1.0, then sum over signal types"
43+ threshold: 1.0 # a work belongs to a category if score >= threshold
44+ primary: "argmax of score; ties broken by the more specific category (child before parent)"
45+ ```
46+2. `taxonomy/ONTOLOGY.md` — human-readable description: the 8 areas, the 40 leaves with definitions, the hierarchy
47+ (parent/child edges), coverage statistics you can compute from the inputs (how many OpenAlex CS topics are mapped,
48+ unmapped CS topics listed explicitly with a reason), and the known ambiguities (e.g. LLM vs foundation model vs
49+ generative AI; cloud vs distributed vs edge; cryptography vs security vs privacy).
50+3. `taxonomy/topic_map.csv` — one row per (openalex topic_id, category_id, weight) used in the YAML (for auditing).
51+
52+Constraints:
53+- Every one of the 302 OpenAlex CS topics must appear in `topic_map.csv` or in the "unmapped" list of ONTOLOGY.md.
54+- The six AI categories that OpenAlex topics cannot separate well (LLM, foundation models, multimodal, generative AI,
55+ agents, AI safety) and the small categories (federated learning, sustainable computing, autonomous scientific discovery,
56+ privacy-preserving computation, blockchain, quantum, edge) MUST have keyword patterns (title and abstract) strong enough
57+ to reach the threshold on their own; the broad categories (CV, NLP, SE, networks, ...) can rely on topics/venues/arXiv.
58+- Keyword regexes must be valid Python `re` patterns (double-escape backslashes in YAML strings) and should be
59+ conservative (word boundaries, avoid single ambiguous tokens such as "agent" or "model").
60+- Map at least the 600 largest DBLP streams; the mapping may give a stream to several categories.
61+- Do not invent topic ids: use only ids present in the CSV. Validate the YAML by loading it with Python (`yaml.safe_load`)
62+ and compiling every regex before you finish; report counts (topics mapped per category, keywords per category).
63+- Write nothing in the manuscript. Report back a short summary (≤ 30 lines) with the counts and the open ambiguities.
64+
65+## Addendum (2026-11-03, after a first attempt failed on output size)
66+
67+The first attempt tried to emit `ontology.yaml` in one response and exceeded the 64,000-token output limit. Work
68+incrementally and keep every single file write under ~6,000 tokens:
69+- Put the mapping data in Python modules, one per area: `taxonomy/src/area_ai_ml.py`, `taxonomy/src/area_perception.py`,
70+ … each defining `CATEGORIES = [ {...}, ... ]` dicts with the schema of the brief (ids, weights, patterns as raw strings).
71+ Write each module with its own Write call; if an area is large, split it into two modules.
72+- Write `taxonomy/build_ontology.py` that imports the modules, validates (40 leaves, unique ids, topic ids exist in the
73+ CSV, regexes compile, weights in (0,1]), and generates `taxonomy/ontology.yaml`, `taxonomy/topic_map.csv` and the
74+ coverage statistics section of `taxonomy/ONTOLOGY.md` (the prose part of ONTOLOGY.md you write by hand, briefly).
75+- Run `python3 taxonomy/build_ontology.py` (pyyaml: `/opt/homebrew/bin/python3 -m pip install --user pyyaml` or use
76+ `uv pip install` in a venv) and fix errors until it passes. Never print whole CSVs to your context: use grep/awk/python
77+ to look up topics by keyword.
added data/raw/README.md +10 −0
@@ -0,0 +1,10 @@
1+# data/raw — provenance (nothing here is committed except this file; copies live on M3U96b:~/wp44_uqo/data/raw)
2+
3+| Source | What | Obtained | Size | Integrity |
4+|---|---|---|---|---|
5+| OpenAlex works snapshot, release 2026-10-14 | `data/jsonl/works/` 1,898 part files, 464,765,485 works, 675.3 GB gz | 2026-11-02 21:19 → 23:05 ET, HTTPS from `openalex.s3.amazonaws.com`, 8 parallel streams | filtered CS subset `works/cs/*.jsonl.zst` 20 GB (15,490,159 records), slim table `works/slim/*.parquet` 9.3 GB (all works) | manifest (`works_manifest_2026-10-14.json`), per-file content length and SHA-256 in `works/ledger.sqlite` (1,898/1,898 ok, 0 parse errors, record total = manifest total) |
6+| OpenAlex entity snapshots | topics 4,516; subfields 252; fields 26; domains 4; sources 268,629; institutions 143,455; publishers 10,713; funders 41,799; concepts 65,026 | 2026-11-02 | 0.6 GB | `entities_ledger.json` (SHA-256 per file) |
7+| OpenAlex API increment | works with any CS topic, publication_date 2026-10-01 → 2026-11-02, sliced by day × (CS primary / non-CS primary), basic paging ≤ 10,000 per slice | 2026-11-03 (public budget $0.10/day → several days) | raw JSON pages `api_increment/pages/` | `api_increment/ledger.json` (counts, cost) |
8+| DBLP | `dblp.xml.gz` (2026-11-02 22:39 UTC), `dblp.dtd`, `.md5` | 2026-11-02 | 1.12 GB | MD5 `ff1c424c3b4cef36582f647c36df27c0` verified; parse report `dblp_parse_report.json` (8,840,866 records) |
9+| arXiv OAI-PMH | set `cs`, metadataPrefix `arXiv`, datestamp 2018-01-01 → 2026-11-02, monthly windows, raw XML pages kept | 2026-11-02/03 | 758 pages, 915,115 unique records | `harvest_state.json`, `arxiv_parse_report.json` |
10+| ACL Anthology | `anthology+abstracts.bib.gz` | 2026-11-02 | 46.7 MB, 138,796 entries | SHA-256 `44c3f7e907f221b740bea09069818a3d7c0dfd88b2da3b750d9f02955efc58b3` |
added ingestion/acl_parse.py +83 −0
@@ -0,0 +1,83 @@
1+#!/usr/bin/env python3
2+"""WP44 — parse the ACL Anthology BibTeX export (anthology+abstracts.bib.gz) into Parquet."""
3+import argparse, gzip, hashlib, json, os, re, datetime as dt
4+import pyarrow as pa, pyarrow.parquet as pq
5+
6+FIELDS = ["title", "author", "editor", "booktitle", "journal", "month", "year", "address", "publisher", "url", "doi", "pages", "abstract", "volume", "number"]
7+SCHEMA = pa.schema([("acl_id", pa.string()), ("entry_type", pa.string())] + [(f, pa.string()) for f in FIELDS])
8+
9+
10+def entries(txt):
11+ i = 0; n = len(txt)
12+ while True:
13+ j = txt.find("@", i)
14+ if j < 0:
15+ return
16+ k = txt.find("{", j)
17+ et = txt[j + 1:k].strip().lower()
18+ depth = 0; p = k
19+ while p < n:
20+ c = txt[p]
21+ if c == "{":
22+ depth += 1
23+ elif c == "}":
24+ depth -= 1
25+ if depth == 0:
26+ break
27+ p += 1
28+ body = txt[k + 1:p]; i = p + 1
29+ if et in ("comment", "string", "preamble"):
30+ continue
31+ yield et, body
32+
33+
34+def parse_body(body):
35+ key, _, rest = body.partition(",")
36+ out = {"acl_id": key.strip()}
37+ # fields of the form name = {value} or name = "value", values may contain nested braces
38+ pos = 0
39+ while True:
40+ m = re.search(r"(\w+)\s*=\s*", rest[pos:])
41+ if not m:
42+ break
43+ name = m.group(1).lower(); p = pos + m.end()
44+ if p >= len(rest):
45+ break
46+ if rest[p] == "{":
47+ depth = 0; q = p
48+ while q < len(rest):
49+ if rest[q] == "{": depth += 1
50+ elif rest[q] == "}":
51+ depth -= 1
52+ if depth == 0: break
53+ q += 1
54+ val = rest[p + 1:q]; pos = q + 1
55+ elif rest[p] == '"':
56+ q = rest.find('"', p + 1); val = rest[p + 1:q]; pos = q + 1
57+ else:
58+ q = rest.find(",", p); q = len(rest) if q < 0 else q; val = rest[p:q]; pos = q
59+ val = re.sub(r"\s+", " ", val).strip()
60+ val = val.replace("{", "").replace("}", "")
61+ if name in FIELDS:
62+ out[name] = val
63+ return out
64+
65+
66+def main():
67+ ap = argparse.ArgumentParser(); ap.add_argument("--bib", required=True); ap.add_argument("--out", required=True)
68+ a = ap.parse_args(); os.makedirs(a.out, exist_ok=True)
69+ raw = open(a.bib, "rb").read(); sha = hashlib.sha256(raw).hexdigest()
70+ txt = gzip.decompress(raw).decode("utf-8", "replace")
71+ rows = {n: [] for n in SCHEMA.names}; n = 0; types = {}
72+ for et, body in entries(txt):
73+ d = parse_body(body); d["entry_type"] = et; types[et] = types.get(et, 0) + 1
74+ for k in SCHEMA.names:
75+ rows[k].append(d.get(k))
76+ n += 1
77+ pq.write_table(pa.Table.from_pydict(rows, schema=SCHEMA), os.path.join(a.out, "acl_anthology.parquet"), compression="zstd")
78+ rep = {"input": a.bib, "sha256": sha, "bytes": len(raw), "entries": n, "types": types, "parsed_at": dt.datetime.now(dt.UTC).isoformat()}
79+ json.dump(rep, open(os.path.join(a.out, "acl_parse_report.json"), "w"), indent=1); print(json.dumps(rep, indent=1))
80+
81+
82+if __name__ == "__main__":
83+ main()
added ingestion/arxiv_oai.py +134 −0
@@ -0,0 +1,134 @@
1+#!/usr/bin/env python3
2+"""WP44 — harvest arXiv metadata (set=cs) through OAI-PMH (metadataPrefix=arXiv).
3+
4+Harvests by monthly datestamp windows from --from to --until; each window's raw
5+XML pages are stored verbatim (immutable raw snapshot) under <out>/raw/<window>/
6+page_NNNN.xml and parsed into <out>/arxiv_cs.parquet at the end (``--parse-only``
7+re-parses). Politeness: >= 3 s between requests, honours Retry-After on 503.
8+Note: OAI ``from``/``until`` filter on the record *datestamp* (last update), so a
9+window contains every record updated in that window; ``created`` gives the first
10+submission date. Duplicates across windows are resolved at parse time (latest
11+datestamp wins).
12+"""
13+import argparse, glob, json, os, re, time, datetime as dt, calendar
14+import requests
15+from lxml import etree
16+import pyarrow as pa, pyarrow.parquet as pq
17+
18+BASE = "https://oaipmh.arxiv.org/oai"
19+NS = {"oai": "http://www.openarchives.org/OAI/2.0/", "ax": "http://arxiv.org/OAI/arXiv/"}
20+UA = "WP44-UQO-systematic-mapping/0.1 (academic research; contact simon-pierre.boucher@uqo.ca)"
21+SCHEMA = pa.schema([
22+ ("arxiv_id", pa.string()), ("datestamp", pa.string()), ("created", pa.string()), ("updated", pa.string()),
23+ ("title", pa.string()), ("abstract", pa.string()), ("authors", pa.list_(pa.string())), ("categories", pa.list_(pa.string())),
24+ ("primary_category", pa.string()), ("comments", pa.string()), ("journal_ref", pa.string()), ("doi", pa.string()),
25+ ("license", pa.string()), ("report_no", pa.string()), ("msc_class", pa.string()), ("acm_class", pa.string()), ("set_specs", pa.list_(pa.string())),
26+])
27+
28+
29+def month_windows(start, end):
30+ y, m = start.year, start.month
31+ while (y, m) <= (end.year, end.month):
32+ last = calendar.monthrange(y, m)[1]
33+ a = dt.date(y, m, 1); b = min(dt.date(y, m, last), end)
34+ yield a, b
35+ m += 1
36+ if m == 13:
37+ y, m = y + 1, 1
38+
39+
40+def fetch(session, params, log):
41+ for attempt in range(12):
42+ try:
43+ r = session.get(BASE, params=params, timeout=180, headers={"User-Agent": UA})
44+ except requests.RequestException as e:
45+ log(f" network error {e}; sleep 30"); time.sleep(30); continue
46+ if r.status_code == 503:
47+ wait = int(r.headers.get("Retry-After", "30")); log(f" 503 retry-after {wait}"); time.sleep(wait + 1); continue
48+ if r.status_code != 200:
49+ log(f" HTTP {r.status_code}; sleep 30"); time.sleep(30); continue
50+ return r.text
51+ raise RuntimeError("too many failures")
52+
53+
54+def harvest(a, log):
55+ s = requests.Session()
56+ state_path = os.path.join(a.out, "harvest_state.json")
57+ state = json.load(open(state_path)) if os.path.exists(state_path) else {"windows": {}}
58+ for w0, w1 in month_windows(dt.date.fromisoformat(a.frm), dt.date.fromisoformat(a.until)):
59+ key = f"{w0}_{w1}"
60+ if state["windows"].get(key, {}).get("status") == "done":
61+ continue
62+ d = os.path.join(a.out, "raw", key); os.makedirs(d, exist_ok=True)
63+ params = {"verb": "ListRecords", "metadataPrefix": "arXiv", "set": a.set, "from": str(w0), "until": str(w1)}
64+ page = 0; nrec = 0; t0 = time.time()
65+ while True:
66+ xml = fetch(s, params, log)
67+ with open(os.path.join(d, f"page_{page:04d}.xml"), "w") as fh:
68+ fh.write(xml)
69+ root = etree.fromstring(xml.encode())
70+ err = root.find("oai:error", NS)
71+ if err is not None:
72+ if err.get("code") == "noRecordsMatch":
73+ break
74+ raise RuntimeError(f"OAI error {err.get('code')}: {err.text}")
75+ nrec += len(root.findall(".//oai:record", NS))
76+ tok = root.find(".//oai:resumptionToken", NS)
77+ page += 1
78+ if tok is None or not (tok.text or "").strip():
79+ break
80+ params = {"verb": "ListRecords", "resumptionToken": tok.text.strip()}
81+ time.sleep(a.delay)
82+ state["windows"][key] = {"status": "done", "pages": page, "records": nrec, "seconds": round(time.time() - t0),
83+ "when": dt.datetime.now(dt.UTC).isoformat()}
84+ json.dump(state, open(state_path, "w"), indent=1)
85+ log(f"window {key}: {page} pages, {nrec} records, {time.time()-t0:.0f}s")
86+ time.sleep(a.delay)
87+
88+
89+def parse(a, log):
90+ recs = {}
91+ files = sorted(glob.glob(os.path.join(a.out, "raw", "*", "page_*.xml")))
92+ for f in files:
93+ root = etree.parse(f).getroot()
94+ for rec in root.findall(".//oai:record", NS):
95+ h = rec.find("oai:header", NS)
96+ if h.get("status") == "deleted":
97+ continue
98+ m = rec.find("oai:metadata/ax:arXiv", NS)
99+ if m is None:
100+ continue
101+ g = lambda t: (m.findtext(f"ax:{t}", namespaces=NS) or "").strip() or None
102+ aid = g("id"); ds = h.findtext("oai:datestamp", namespaces=NS)
103+ if aid in recs and recs[aid]["datestamp"] >= ds:
104+ continue
105+ cats = (g("categories") or "").split()
106+ authors = []
107+ for au in m.findall("ax:authors/ax:author", NS):
108+ kn = au.findtext("ax:keyname", namespaces=NS) or ""; fn = au.findtext("ax:forenames", namespaces=NS) or ""
109+ authors.append((fn + " " + kn).strip())
110+ recs[aid] = dict(arxiv_id=aid, datestamp=ds, created=g("created"), updated=g("updated"),
111+ title=re.sub(r"\s+", " ", g("title") or ""), abstract=re.sub(r"\s+", " ", g("abstract") or ""),
112+ authors=authors, categories=cats, primary_category=cats[0] if cats else None,
113+ comments=g("comments"), journal_ref=g("journal-ref"), doi=(g("doi") or "").lower() or None, license=g("license"),
114+ report_no=g("report-no"), msc_class=g("msc-class"), acm_class=g("acm-class"),
115+ set_specs=[x.text for x in h.findall("oai:setSpec", NS)])
116+ cols = {n: [r[n] for r in recs.values()] for n in SCHEMA.names}
117+ pq.write_table(pa.Table.from_pydict(cols, schema=SCHEMA), os.path.join(a.out, "arxiv_cs.parquet"), compression="zstd")
118+ rep = {"files": len(files), "records": len(recs), "parsed_at": dt.datetime.now(dt.UTC).isoformat(), "set": a.set, "from": a.frm, "until": a.until}
119+ json.dump(rep, open(os.path.join(a.out, "arxiv_parse_report.json"), "w"), indent=1); log(json.dumps(rep))
120+
121+
122+def main():
123+ ap = argparse.ArgumentParser(); ap.add_argument("--out", required=True); ap.add_argument("--set", default="cs")
124+ ap.add_argument("--from", dest="frm", default="2018-01-01"); ap.add_argument("--until", default="2026-11-02")
125+ ap.add_argument("--delay", type=float, default=3.5); ap.add_argument("--parse-only", action="store_true")
126+ a = ap.parse_args(); os.makedirs(a.out, exist_ok=True)
127+ log = lambda s: print(f"[{dt.datetime.now():%H:%M:%S}] {s}", flush=True)
128+ if not a.parse_only:
129+ harvest(a, log)
130+ parse(a, log)
131+
132+
133+if __name__ == "__main__":
134+ main()
added ingestion/build_corpus.py +197 −0
@@ -0,0 +1,197 @@
1+#!/usr/bin/env python3
2+"""WP44 — normalise the filtered OpenAlex CS records (cs/*.jsonl.zst) into Parquet tables.
3+
4+Tables (one Parquet file per input part under <out>/tables/<table>/<part>.parquet, resumable):
5+ works, authorships, work_topics, work_keywords, references, abstracts, locations, grants, counts_by_year
6+Run on the compute node after (or while) ingestion/openalex_snapshot.py runs; parts whose outputs exist are skipped.
7+"""
8+import argparse, glob, io, multiprocessing as mp, os, re, time, datetime as dt
9+import orjson, zstandard as zstd
10+import pyarrow as pa, pyarrow.parquet as pq
11+
12+ARXIV_RE = re.compile(r"arxiv\.org/(?:abs|pdf)/([0-9]{4}\.[0-9]{4,5}(?:v\d+)?|[a-z\-]+(?:\.[A-Z]{2})?/[0-9]{7}(?:v\d+)?)", re.I)
13+ARXIV_DOI_RE = re.compile(r"^10\.48550/arxiv\.(.+)$", re.I)
14+
15+S = {
16+ "works": pa.schema([("id", pa.int64()), ("doi", pa.string()), ("title", pa.string()), ("publication_year", pa.int32()),
17+ ("publication_date", pa.string()), ("created_date", pa.string()), ("updated_date", pa.string()), ("type", pa.string()),
18+ ("type_crossref", pa.string()), ("language", pa.string()), ("cited_by_count", pa.int32()), ("fwci", pa.float32()),
19+ ("citation_percentile", pa.float32()), ("is_top1", pa.bool_()), ("is_top10", pa.bool_()), ("is_oa", pa.bool_()), ("oa_status", pa.string()),
20+ ("license", pa.string()), ("is_retracted", pa.bool_()), ("is_paratext", pa.bool_()), ("is_xpac", pa.bool_()), ("n_refs", pa.int32()),
21+ ("n_authors", pa.int32()), ("n_institutions", pa.int32()), ("n_countries", pa.int32()), ("countries", pa.list_(pa.string())),
22+ ("institution_types", pa.list_(pa.string())), ("primary_topic_id", pa.int32()), ("primary_subfield_id", pa.int32()),
23+ ("primary_field_id", pa.int32()), ("primary_domain_id", pa.int32()), ("cs_primary", pa.bool_()), ("cs_any", pa.bool_()),
24+ ("source_id", pa.int64()), ("source_name", pa.string()), ("source_type", pa.string()), ("host_organization", pa.string()),
25+ ("is_in_doaj", pa.bool_()), ("arxiv_id", pa.string()), ("pmid", pa.string()), ("mag", pa.string()), ("has_abstract", pa.bool_()),
26+ ("has_fulltext", pa.bool_()), ("n_locations", pa.int32()), ("indexed_in", pa.list_(pa.string())), ("referenced_works_count", pa.int32()),
27+ ("n_grants", pa.int32()), ("apc_paid_usd", pa.int32()), ("is_authors_truncated", pa.bool_()), ("part", pa.string())]),
28+ "authorships": pa.schema([("work_id", pa.int64()), ("position", pa.int32()), ("author_position", pa.string()), ("author_id", pa.int64()),
29+ ("author_name", pa.string()), ("orcid", pa.string()), ("is_corresponding", pa.bool_()), ("institution_ids", pa.list_(pa.int64())),
30+ ("institution_names", pa.list_(pa.string())), ("institution_types", pa.list_(pa.string())), ("countries", pa.list_(pa.string())),
31+ ("n_raw_affiliations", pa.int32())]),
32+ "work_topics": pa.schema([("work_id", pa.int64()), ("rank", pa.int16()), ("topic_id", pa.int32()), ("score", pa.float32()),
33+ ("subfield_id", pa.int32()), ("field_id", pa.int32()), ("domain_id", pa.int32())]),
34+ "work_keywords": pa.schema([("work_id", pa.int64()), ("keyword", pa.string()), ("score", pa.float32())]),
35+ "references": pa.schema([("work_id", pa.int64()), ("ref_id", pa.int64())]),
36+ "abstracts": pa.schema([("work_id", pa.int64()), ("abstract", pa.string()), ("n_words", pa.int32())]),
37+ "locations": pa.schema([("work_id", pa.int64()), ("rank", pa.int16()), ("is_primary", pa.bool_()), ("is_best_oa", pa.bool_()), ("source_id", pa.int64()),
38+ ("source_name", pa.string()), ("source_type", pa.string()), ("landing_page_url", pa.string()), ("pdf_url", pa.string()), ("is_oa", pa.bool_()),
39+ ("license", pa.string()), ("version", pa.string()), ("is_accepted", pa.bool_()), ("is_published", pa.bool_())]),
40+ "grants": pa.schema([("work_id", pa.int64()), ("funder_id", pa.int64()), ("funder_name", pa.string()), ("award_id", pa.string())]),
41+ "counts_by_year": pa.schema([("work_id", pa.int64()), ("year", pa.int16()), ("cited_by_count", pa.int32())]),
42+}
43+
44+
45+def oid(u):
46+ if not u:
47+ return None
48+ t = u.rsplit("/", 1)[-1]
49+ i = 0
50+ while i < len(t) and not t[i].isdigit():
51+ i += 1
52+ try:
53+ return int(t[i:])
54+ except ValueError:
55+ return None
56+
57+
58+def abstract_text(inv):
59+ if not inv:
60+ return None
61+ pos = []
62+ for w, ps in inv.items():
63+ for p in ps:
64+ pos.append((p, w))
65+ pos.sort()
66+ return " ".join(w for _, w in pos)
67+
68+
69+def arxiv_from(w):
70+ ids = w.get("ids") or {}
71+ if ids.get("arxiv"):
72+ return str(ids["arxiv"]).rsplit("/", 1)[-1]
73+ doi = w.get("doi") or ""
74+ m = ARXIV_DOI_RE.match(doi.replace("https://doi.org/", ""))
75+ if m:
76+ return m.group(1)
77+ for loc in (w.get("locations") or []):
78+ for k in ("landing_page_url", "pdf_url"):
79+ u = loc.get(k) or ""
80+ m = ARXIV_RE.search(u)
81+ if m:
82+ return re.sub(r"v\d+$", "", m.group(1))
83+ return None
84+
85+
86+def process(args):
87+ path, out, cs_field = args
88+ part = os.path.basename(path).replace(".jsonl.zst", "")
89+ outs = {t: os.path.join(out, "tables", t, part + ".parquet") for t in S}
90+ if all(os.path.exists(p) for p in outs.values()):
91+ return part, "skip", 0
92+ for p in outs.values():
93+ os.makedirs(os.path.dirname(p), exist_ok=True)
94+ rows = {t: {c: [] for c in S[t].names} for t in S}
95+ n = 0
96+ with open(path, "rb") as fh, zstd.ZstdDecompressor().stream_reader(fh) as r:
97+ for line in io.TextIOWrapper(r, encoding="utf-8"):
98+ if not line.strip():
99+ continue
100+ w = orjson.loads(line); n += 1
101+ wid = oid(w["id"])
102+ doi = (w.get("doi") or "").replace("https://doi.org/", "").lower() or None
103+ pt = w.get("primary_topic") or {}
104+ topics = w.get("topics") or []
105+ cs_any = any(((t.get("field") or {}).get("id") == cs_field) for t in topics) or ((pt.get("field") or {}).get("id") == cs_field)
106+ loc = w.get("primary_location") or {}
107+ src = loc.get("source") or {}
108+ best = w.get("best_oa_location") or {}
109+ oa = w.get("open_access") or {}
110+ auths = w.get("authorships") or []
111+ inst_ids = set(); countries = set(); itypes = []
112+ for i, a in enumerate(auths):
113+ ins = a.get("institutions") or []
114+ ii = [oid(x.get("id")) for x in ins]
115+ inst_ids.update(x for x in ii if x)
116+ cc = [c for c in (a.get("countries") or []) if c]
117+ countries.update(cc)
118+ itypes.extend(x.get("type") for x in ins if x.get("type"))
119+ au = a.get("author") or {}
120+ R = rows["authorships"]
121+ R["work_id"].append(wid); R["position"].append(i); R["author_position"].append(a.get("author_position"))
122+ R["author_id"].append(oid(au.get("id"))); R["author_name"].append(au.get("display_name")); R["orcid"].append(au.get("orcid"))
123+ R["is_corresponding"].append(bool(a.get("is_corresponding"))); R["institution_ids"].append(ii)
124+ R["institution_names"].append([x.get("display_name") for x in ins]); R["institution_types"].append([x.get("type") for x in ins])
125+ R["countries"].append(cc); R["n_raw_affiliations"].append(len(a.get("raw_affiliation_strings") or []))
126+ for k, t in enumerate(topics):
127+ R = rows["work_topics"]
128+ R["work_id"].append(wid); R["rank"].append(k); R["topic_id"].append(oid(t.get("id"))); R["score"].append(t.get("score"))
129+ R["subfield_id"].append(oid((t.get("subfield") or {}).get("id"))); R["field_id"].append(oid((t.get("field") or {}).get("id")))
130+ R["domain_id"].append(oid((t.get("domain") or {}).get("id")))
131+ for kw in (w.get("keywords") or []):
132+ R = rows["work_keywords"]; R["work_id"].append(wid); R["keyword"].append(kw.get("display_name")); R["score"].append(kw.get("score"))
133+ refs = w.get("referenced_works") or []
134+ for rr in refs:
135+ rows["references"]["work_id"].append(wid); rows["references"]["ref_id"].append(oid(rr))
136+ ab = abstract_text(w.get("abstract_inverted_index"))
137+ if ab:
138+ R = rows["abstracts"]; R["work_id"].append(wid); R["abstract"].append(ab); R["n_words"].append(len(ab.split()))
139+ locs = w.get("locations") or []
140+ best_url = best.get("landing_page_url") if best else None
141+ for k, L in enumerate(locs):
142+ s_ = L.get("source") or {}
143+ R = rows["locations"]
144+ R["work_id"].append(wid); R["rank"].append(k); R["is_primary"].append(L.get("landing_page_url") == loc.get("landing_page_url") and L.get("pdf_url") == loc.get("pdf_url"))
145+ R["is_best_oa"].append(bool(best_url) and L.get("landing_page_url") == best_url)
146+ R["source_id"].append(oid(s_.get("id"))); R["source_name"].append(s_.get("display_name")); R["source_type"].append(s_.get("type"))
147+ R["landing_page_url"].append(L.get("landing_page_url")); R["pdf_url"].append(L.get("pdf_url")); R["is_oa"].append(bool(L.get("is_oa")))
148+ R["license"].append(L.get("license")); R["version"].append(L.get("version")); R["is_accepted"].append(bool(L.get("is_accepted"))); R["is_published"].append(bool(L.get("is_published")))
149+ for g in (w.get("grants") or []):
150+ R = rows["grants"]; R["work_id"].append(wid); R["funder_id"].append(oid(g.get("funder"))); R["funder_name"].append(g.get("funder_display_name")); R["award_id"].append(g.get("award_id"))
151+ for c in (w.get("counts_by_year") or []):
152+ R = rows["counts_by_year"]; R["work_id"].append(wid); R["year"].append(c.get("year")); R["cited_by_count"].append(c.get("cited_by_count"))
153+ cnp = w.get("citation_normalized_percentile") or {}
154+ ids = w.get("ids") or {}
155+ apc = (w.get("apc_paid") or {}).get("value_usd") if isinstance(w.get("apc_paid"), dict) else None
156+ R = rows["works"]
157+ vals = dict(id=wid, doi=doi, title=w.get("title") or w.get("display_name"), publication_year=w.get("publication_year"), publication_date=w.get("publication_date"),
158+ created_date=w.get("created_date"), updated_date=(w.get("updated_date") or "")[:10] or None, type=w.get("type"), type_crossref=w.get("type_crossref"),
159+ language=w.get("language"), cited_by_count=w.get("cited_by_count"), fwci=w.get("fwci"), citation_percentile=cnp.get("value"),
160+ is_top1=bool(cnp.get("is_in_top_1_percent")), is_top10=bool(cnp.get("is_in_top_10_percent")), is_oa=bool(oa.get("is_oa")), oa_status=oa.get("oa_status"),
161+ license=(best.get("license") if best else None) or loc.get("license"), is_retracted=bool(w.get("is_retracted")), is_paratext=bool(w.get("is_paratext")),
162+ is_xpac=bool(w.get("is_xpac")), n_refs=len(refs), n_authors=w.get("authors_count") if w.get("authors_count") is not None else len(auths),
163+ n_institutions=len(inst_ids), n_countries=len(countries), countries=sorted(countries), institution_types=sorted(set(itypes)),
164+ primary_topic_id=oid(pt.get("id")), primary_subfield_id=oid((pt.get("subfield") or {}).get("id")), primary_field_id=oid((pt.get("field") or {}).get("id")),
165+ primary_domain_id=oid((pt.get("domain") or {}).get("id")), cs_primary=(pt.get("field") or {}).get("id") == cs_field, cs_any=bool(cs_any),
166+ source_id=oid(src.get("id")), source_name=src.get("display_name"), source_type=src.get("type"), host_organization=src.get("host_organization_name"),
167+ is_in_doaj=bool(src.get("is_in_doaj")), arxiv_id=arxiv_from(w), pmid=(ids.get("pmid") or "").rsplit("/", 1)[-1] or None, mag=str(ids.get("mag")) if ids.get("mag") else None,
168+ has_abstract=ab is not None, has_fulltext=bool(w.get("has_fulltext") or w.get("has_content")), n_locations=len(locs), indexed_in=w.get("indexed_in") or [],
169+ referenced_works_count=w.get("referenced_works_count") if w.get("referenced_works_count") is not None else len(refs), n_grants=len(w.get("grants") or []),
170+ apc_paid_usd=apc, is_authors_truncated=bool(w.get("is_authors_truncated")), part=part)
171+ for k, v in vals.items():
172+ R[k].append(v)
173+ for t, sch in S.items():
174+ tbl = pa.Table.from_pydict(rows[t], schema=sch)
175+ tmp = outs[t] + ".part"
176+ pq.write_table(tbl, tmp, compression="zstd"); os.replace(tmp, outs[t])
177+ return part, "done", n
178+
179+
180+def main():
181+ ap = argparse.ArgumentParser(); ap.add_argument("--cs-dir", required=True); ap.add_argument("--out", required=True)
182+ ap.add_argument("--workers", type=int, default=8); ap.add_argument("--limit", type=int, default=0)
183+ a = ap.parse_args()
184+ files = sorted(glob.glob(os.path.join(a.cs_dir, "*.jsonl.zst")))
185+ if a.limit:
186+ files = files[: a.limit]
187+ t0 = time.time(); tot = 0; done = 0
188+ with mp.Pool(a.workers, maxtasksperchild=40) as pool:
189+ for part, st, n in pool.imap_unordered(process, [(f, a.out, "https://openalex.org/fields/17") for f in files]):
190+ tot += n; done += 1
191+ if st == "done" or done % 100 == 0:
192+ print(f"[{dt.datetime.now():%H:%M:%S}] {st} {part} n={n} | {done}/{len(files)} files, {tot} works, {time.time()-t0:.0f}s", flush=True)
193+ print("finished", done, "files", tot, "works")
194+
195+
196+if __name__ == "__main__":
197+ main()
added ingestion/dblp_parse.py +85 −0
@@ -0,0 +1,85 @@
1+#!/usr/bin/env python3
2+"""WP44 — parse the DBLP XML dump (dblp.xml.gz + dblp.dtd) into Parquet.
3+
4+Output: <out>/dblp_publications.parquet with one row per publication record
5+(article, inproceedings, proceedings, book, incollection, phdthesis, mastersthesis,
6+www is skipped) and <out>/dblp_parse_report.json (counts, md5 of input, timings).
7+All years are kept (the historical baseline needs 2018–2022; counts per year are
8+reported for the whole archive).
9+"""
10+import argparse, gzip, hashlib, json, os, time, datetime as dt
11+from lxml import etree
12+import pyarrow as pa, pyarrow.parquet as pq
13+
14+TYPES = {"article", "inproceedings", "proceedings", "book", "incollection", "phdthesis", "mastersthesis", "data"}
15+SCHEMA = pa.schema([
16+ ("key", pa.string()), ("rec_type", pa.string()), ("mdate", pa.string()), ("publtype", pa.string()),
17+ ("title", pa.string()), ("year", pa.int32()), ("authors", pa.list_(pa.string())), ("author_pids", pa.list_(pa.string())),
18+ ("editors", pa.list_(pa.string())), ("journal", pa.string()), ("booktitle", pa.string()), ("volume", pa.string()),
19+ ("number", pa.string()), ("pages", pa.string()), ("publisher", pa.string()), ("series", pa.string()),
20+ ("doi", pa.string()), ("ee", pa.list_(pa.string())), ("url", pa.string()), ("crossref", pa.string()),
21+ ("stream", pa.string()),
22+])
23+
24+
25+def text(el):
26+ return "".join(el.itertext()).strip() if el is not None else None
27+
28+
29+def main():
30+ ap = argparse.ArgumentParser(); ap.add_argument("--xml", required=True); ap.add_argument("--out", required=True)
31+ ap.add_argument("--min-year", type=int, default=0)
32+ a = ap.parse_args(); a.xml = os.path.abspath(a.xml); a.out = os.path.abspath(a.out); os.makedirs(a.out, exist_ok=True)
33+ t0 = time.time()
34+ md5 = hashlib.md5(open(a.xml, "rb").read(1 << 30)).hexdigest() # first GB hash (file is 1.1 GB) + size recorded
35+ size = os.path.getsize(a.xml)
36+ # lxml needs the DTD next to the XML for entity resolution; parse the gz stream
37+ parser = etree.XMLParser(load_dtd=True, resolve_entities=True, huge_tree=True)
38+ counts = {}; years = {}
39+ rows = {n: [] for n in SCHEMA.names}; n = 0
40+ writer = pq.ParquetWriter(os.path.join(a.out, "dblp_publications.parquet"), SCHEMA, compression="zstd")
41+
42+ def flush():
43+ nonlocal rows
44+ if rows["key"]:
45+ writer.write_table(pa.Table.from_pydict(rows, schema=SCHEMA)); rows = {k: [] for k in SCHEMA.names}
46+
47+ # iterparse over the gzip stream; the DTD is referenced relatively (dblp.dtd) -> chdir
48+ os.chdir(os.path.dirname(os.path.abspath(a.xml)))
49+ with gzip.open(a.xml, "rb") as fh:
50+ ctx = etree.iterparse(fh, events=("end",), tag=list(TYPES | {"www"}), load_dtd=True, resolve_entities=True, huge_tree=True)
51+ for _, el in ctx:
52+ t = el.tag
53+ if t in TYPES:
54+ counts[t] = counts.get(t, 0) + 1
55+ yr = el.findtext("year"); yr = int(yr) if yr and yr.isdigit() else None
56+ years[yr] = years.get(yr, 0) + 1
57+ if yr is not None and yr >= a.min_year:
58+ ees = [e.text for e in el.findall("ee") if e.text]
59+ doi = next((e[16:].lower() for e in ees if e.startswith("https://doi.org/")), None)
60+ stream = el.get("key", "").rsplit("/", 1)[0] if "/" in el.get("key", "") else None
61+ vals = dict(key=el.get("key"), rec_type=t, mdate=el.get("mdate"), publtype=el.get("publtype"),
62+ title=text(el.find("title")), year=yr,
63+ authors=[text(x) for x in el.findall("author")], author_pids=[x.get("pid") for x in el.findall("author")],
64+ editors=[text(x) for x in el.findall("editor")], journal=el.findtext("journal"), booktitle=el.findtext("booktitle"),
65+ volume=el.findtext("volume"), number=el.findtext("number"), pages=el.findtext("pages"),
66+ publisher=el.findtext("publisher"), series=text(el.find("series")), doi=doi, ee=ees,
67+ url=el.findtext("url"), crossref=el.findtext("crossref"), stream=stream)
68+ for k, v in vals.items():
69+ rows[k].append(v)
70+ n += 1
71+ if n % 200000 == 0:
72+ flush(); print(f"{n} rows, {time.time()-t0:.0f}s", flush=True)
73+ el.clear()
74+ while el.getprevious() is not None:
75+ del el.getparent()[0]
76+ flush(); writer.close()
77+ rep = {"input": os.path.abspath(a.xml), "input_bytes": size, "input_md5_first_GiB": md5, "parsed_at": dt.datetime.now(dt.UTC).isoformat(),
78+ "rows_written": n, "min_year": a.min_year, "counts_by_type": counts, "counts_by_year": {str(k): v for k, v in sorted(years.items(), key=lambda kv: (kv[0] is None, kv[0]))},
79+ "seconds": round(time.time() - t0)}
80+ json.dump(rep, open(os.path.join(a.out, "dblp_parse_report.json"), "w"), indent=1)
81+ print(json.dumps({k: v for k, v in rep.items() if k != "counts_by_year"}, indent=1))
82+
83+
84+if __name__ == "__main__":
85+ main()
added ingestion/dblp_streams.py +9 −0
@@ -0,0 +1,9 @@
1+#!/usr/bin/env python3
2+"""Venue streams (journals / conference series) of DBLP 2018-2026 with counts, for the taxonomy mapping."""
3+import duckdb, sys
4+src, out = sys.argv[1], sys.argv[2]
5+con = duckdb.connect()
6+con.sql(f"""copy (select stream, rec_type, count(*) n, min(year) y0, max(year) y1, any_value(journal) journal, any_value(booktitle) booktitle
7+ from read_parquet('{src}') where year between 2018 and 2026 and rec_type in ('article','inproceedings')
8+ group by 1,2 having n >= 100 order by n desc) to '{out}' (header)""")
9+print(con.sql(f"select count(*) from read_csv_auto('{out}')").fetchall(), con.sql(f"select rec_type, count(*) from read_parquet('{src}') where year between 2018 and 2026 group by 1 order by 2 desc").fetchall())
added ingestion/link_sources.py +121 −0
@@ -0,0 +1,121 @@
1+#!/usr/bin/env python3
2+"""WP44 — Stage S1 record linkage: OpenAlex CS records ↔ DBLP ↔ arXiv ↔ ACL Anthology, in DuckDB.
3+
4+Creates/refreshes <db> with views over the Parquet tables and the linkage tables
5+ link_dblp_oa(dblp_key, oa_id, method), link_arxiv_oa(arxiv_id, oa_id, method), link_acl_oa(acl_id, oa_id, method)
6+and writes data/metadata/linkage_report.json. Also writes the list of OpenAlex ids of DBLP-indexed works (2018–2026)
7+that are in the slim table but NOT in the CS-topic subset (needed for the second snapshot pass).
8+"""
9+import argparse, json, os, time, datetime as dt
10+import duckdb
11+
12+
13+def main():
14+ ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); ap.add_argument("--db", default=None)
15+ a = ap.parse_args(); R = a.root; db = a.db or os.path.join(R, "data/processed/wp44.duckdb")
16+ con = duckdb.connect(db); con.execute("SET threads TO 24; SET memory_limit='60GB'; SET preserve_insertion_order=false;")
17+ T = os.path.join(R, "data/processed/tables"); t0 = time.time(); rep = {"created": dt.datetime.now(dt.UTC).isoformat()}
18+ def log(m): print(f"[{time.time()-t0:7.0f}s] {m}", flush=True)
19+ for t in ["works", "authorships", "work_topics", "work_keywords", "references", "abstracts", "locations", "grants", "counts_by_year"]:
20+ con.execute(f"CREATE OR REPLACE VIEW oa_{t} AS SELECT * FROM read_parquet('{T}/{t}/*.parquet')")
21+ con.execute(f"CREATE OR REPLACE VIEW oa_slim AS SELECT * FROM read_parquet('{R}/data/raw/openalex/works/slim/*.parquet')")
22+ con.execute(f"CREATE OR REPLACE VIEW arxiv_raw AS SELECT * FROM read_parquet('{R}/data/raw/arxiv/arxiv_cs.parquet')")
23+ con.execute(f"CREATE OR REPLACE VIEW acl_raw AS SELECT * FROM read_parquet('{R}/data/raw/acl/acl_anthology.parquet')")
24+ con.execute(f"CREATE OR REPLACE VIEW dblp_raw AS SELECT * FROM read_parquet('{R}/data/raw/dblp/dblp_publications.parquet')")
25+ log("views created")
26+ # --- normalised keys ---
27+ con.execute("""CREATE OR REPLACE TABLE oa_keys AS
28+ SELECT id AS oa_id, doi, lower(regexp_replace(arxiv_id, 'v[0-9]+$', '')) AS arxiv_id, publication_year AS year,
29+ regexp_replace(lower(title), '[^a-z0-9]+', '', 'g') AS ntitle, length(title) AS tlen, type
30+ FROM oa_works""")
31+ con.execute("""CREATE OR REPLACE TABLE dblp_keys AS
32+ SELECT key AS dblp_key, rec_type, year, doi,
33+ lower(regexp_replace(regexp_extract(array_to_string(ee, ' '), 'arxiv\\.org/abs/([^ ]+)', 1), 'v[0-9]+$', '')) AS arxiv_id,
34+ regexp_replace(lower(title), '[^a-z0-9]+', '', 'g') AS ntitle, length(title) AS tlen, stream
35+ FROM dblp_raw WHERE year BETWEEN 2018 AND 2026 AND rec_type IN ('article','inproceedings','incollection','book','phdthesis','data')""")
36+ con.execute("UPDATE dblp_keys SET arxiv_id = NULL WHERE arxiv_id = ''")
37+ con.execute("""CREATE OR REPLACE TABLE arxiv_keys AS
38+ SELECT lower(arxiv_id) AS arxiv_id, created, substr(created,1,4)::INT AS created_year, doi, primary_category, categories,
39+ regexp_replace(lower(title), '[^a-z0-9]+', '', 'g') AS ntitle FROM arxiv_raw""")
40+ con.execute("""CREATE OR REPLACE TABLE acl_keys AS
41+ SELECT acl_id, try_cast(year AS INT) AS year, lower(doi) AS doi, url, regexp_replace(lower(title), '[^a-z0-9]+', '', 'g') AS ntitle,
42+ entry_type FROM acl_raw WHERE entry_type IN ('inproceedings','article')""")
43+ log("keys built")
44+ # --- DBLP ↔ OpenAlex ---
45+ con.execute("""CREATE OR REPLACE TABLE link_dblp_oa AS
46+ SELECT d.dblp_key, o.oa_id, 'doi' AS method FROM dblp_keys d JOIN oa_keys o ON d.doi = o.doi WHERE d.doi IS NOT NULL""")
47+ con.execute("""INSERT INTO link_dblp_oa
48+ SELECT d.dblp_key, o.oa_id, 'arxiv' FROM dblp_keys d JOIN oa_keys o ON d.arxiv_id = o.arxiv_id
49+ WHERE d.arxiv_id IS NOT NULL AND o.arxiv_id IS NOT NULL AND d.dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa)
50+ AND o.oa_id NOT IN (SELECT oa_id FROM link_dblp_oa)""")
51+ # title+year: unique normalised titles on both sides, length >= 25, |year diff| <= 1
52+ con.execute("""CREATE OR REPLACE TEMP TABLE d_un AS SELECT ntitle, any_value(dblp_key) dblp_key, any_value(year) year FROM dblp_keys
53+ WHERE tlen >= 25 AND dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa) GROUP BY ntitle HAVING count(*) = 1""")
54+ con.execute("""CREATE OR REPLACE TEMP TABLE o_un AS SELECT ntitle, any_value(oa_id) oa_id, any_value(year) year FROM oa_keys
55+ WHERE tlen >= 25 AND oa_id NOT IN (SELECT oa_id FROM link_dblp_oa) GROUP BY ntitle HAVING count(*) = 1""")
56+ con.execute("""INSERT INTO link_dblp_oa SELECT d.dblp_key, o.oa_id, 'title_year' FROM d_un d JOIN o_un o ON d.ntitle = o.ntitle
57+ WHERE abs(d.year - o.year) <= 1""")
58+ log("dblp linked")
59+ # --- arXiv ↔ OpenAlex ---
60+ con.execute("""CREATE OR REPLACE TABLE link_arxiv_oa AS
61+ SELECT x.arxiv_id, o.oa_id, 'arxiv_id' AS method FROM arxiv_keys x JOIN oa_keys o ON x.arxiv_id = o.arxiv_id""")
62+ con.execute("""INSERT INTO link_arxiv_oa SELECT x.arxiv_id, o.oa_id, 'doi' FROM arxiv_keys x JOIN oa_keys o ON x.doi = o.doi
63+ WHERE x.doi IS NOT NULL AND x.arxiv_id NOT IN (SELECT arxiv_id FROM link_arxiv_oa)""")
64+ con.execute("""CREATE OR REPLACE TEMP TABLE x_un AS SELECT ntitle, any_value(arxiv_id) arxiv_id, any_value(created_year) year FROM arxiv_keys
65+ WHERE length(ntitle) >= 25 AND arxiv_id NOT IN (SELECT arxiv_id FROM link_arxiv_oa) GROUP BY ntitle HAVING count(*) = 1""")
66+ con.execute("""CREATE OR REPLACE TEMP TABLE o_un2 AS SELECT ntitle, any_value(oa_id) oa_id, any_value(year) year FROM oa_keys
67+ WHERE tlen >= 25 GROUP BY ntitle HAVING count(*) = 1""")
68+ con.execute("""INSERT INTO link_arxiv_oa SELECT x.arxiv_id, o.oa_id, 'title_year' FROM x_un x JOIN o_un2 o ON x.ntitle = o.ntitle
69+ WHERE abs(x.year - o.year) <= 2 AND o.oa_id NOT IN (SELECT oa_id FROM link_arxiv_oa)""")
70+ log("arxiv linked")
71+ # --- ACL ↔ OpenAlex ---
72+ con.execute("""CREATE OR REPLACE TABLE link_acl_oa AS
73+ SELECT c.acl_id, o.oa_id, 'doi' AS method FROM acl_keys c JOIN oa_keys o ON c.doi = o.doi WHERE c.doi IS NOT NULL""")
74+ con.execute("""INSERT INTO link_acl_oa SELECT c.acl_id, l.work_id, 'url' FROM acl_keys c
75+ JOIN (SELECT DISTINCT work_id, regexp_extract(landing_page_url, 'aclanthology\\.org/([^/ ]+)', 1) AS aid FROM oa_locations
76+ WHERE landing_page_url LIKE '%aclanthology.org/%') l ON l.aid = c.acl_id OR l.aid = c.acl_id || '/'
77+ WHERE c.acl_id NOT IN (SELECT acl_id FROM link_acl_oa)""")
78+ con.execute("""INSERT INTO link_acl_oa SELECT c.acl_id, o.oa_id, 'title_year' FROM
79+ (SELECT ntitle, any_value(acl_id) acl_id, any_value(year) year FROM acl_keys WHERE length(ntitle) >= 25 AND acl_id NOT IN (SELECT acl_id FROM link_acl_oa) GROUP BY ntitle HAVING count(*)=1) c
80+ JOIN o_un2 o ON c.ntitle = o.ntitle WHERE abs(c.year - o.year) <= 1 AND o.oa_id NOT IN (SELECT oa_id FROM link_acl_oa)""")
81+ log("acl linked")
82+ # --- report ---
83+ q = lambda s: con.execute(s).fetchall()
84+ rep["oa_cs_records"] = q("SELECT count(*) FROM oa_keys")[0][0]
85+ rep["oa_cs_distinct_ids"] = q("SELECT count(DISTINCT oa_id) FROM oa_keys")[0][0]
86+ rep["dblp_2018_2026"] = q("SELECT count(*) FROM dblp_keys")[0][0]
87+ rep["dblp_with_doi"] = q("SELECT count(*) FROM dblp_keys WHERE doi IS NOT NULL")[0][0]
88+ rep["dblp_with_arxiv"] = q("SELECT count(*) FROM dblp_keys WHERE arxiv_id IS NOT NULL")[0][0]
89+ rep["link_dblp_oa"] = dict(q("SELECT method, count(*) FROM link_dblp_oa GROUP BY 1"))
90+ rep["dblp_linked_distinct"] = q("SELECT count(DISTINCT dblp_key) FROM link_dblp_oa")[0][0]
91+ rep["dblp_unlinked"] = rep["dblp_2018_2026"] - rep["dblp_linked_distinct"]
92+ rep["dblp_unlinked_by_type"] = dict(q("SELECT rec_type, count(*) FROM dblp_keys WHERE dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa) GROUP BY 1"))
93+ rep["dblp_unlinked_with_doi_in_slim"] = q("""SELECT count(*) FROM dblp_keys d JOIN oa_slim s ON d.doi = s.doi
94+ WHERE d.dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa)""")[0][0]
95+ rep["arxiv_cs_records"] = q("SELECT count(*) FROM arxiv_keys")[0][0]
96+ rep["arxiv_created_2018_2026"] = q("SELECT count(*) FROM arxiv_keys WHERE created_year BETWEEN 2018 AND 2026")[0][0]
97+ rep["link_arxiv_oa"] = dict(q("SELECT method, count(*) FROM link_arxiv_oa GROUP BY 1"))
98+ rep["arxiv_linked_distinct"] = q("SELECT count(DISTINCT arxiv_id) FROM link_arxiv_oa")[0][0]
99+ rep["acl_records"] = q("SELECT count(*) FROM acl_keys")[0][0]
100+ rep["acl_2018_2026"] = q("SELECT count(*) FROM acl_keys WHERE year BETWEEN 2018 AND 2026")[0][0]
101+ rep["link_acl_oa"] = dict(q("SELECT method, count(*) FROM link_acl_oa GROUP BY 1"))
102+ rep["acl_linked_distinct_2018_2026"] = q("SELECT count(DISTINCT l.acl_id) FROM link_acl_oa l JOIN acl_keys c USING(acl_id) WHERE c.year BETWEEN 2018 AND 2026")[0][0]
103+ # ids needed for the second snapshot pass: DBLP-indexed (2018–2026) works in slim but not in the CS subset; plus arXiv cs works in slim via DOI not in subset
104+ con.execute("""CREATE OR REPLACE TABLE needed_oa_ids AS
105+ SELECT DISTINCT s.id AS oa_id, 'dblp_doi' AS why FROM dblp_keys d JOIN oa_slim s ON d.doi = s.doi
106+ WHERE d.dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa) AND s.id NOT IN (SELECT oa_id FROM oa_keys)
107+ UNION
108+ SELECT DISTINCT s.id, 'arxiv_doi' FROM arxiv_keys x JOIN oa_slim s ON x.doi = s.doi
109+ WHERE x.doi IS NOT NULL AND x.arxiv_id NOT IN (SELECT arxiv_id FROM link_arxiv_oa) AND s.id NOT IN (SELECT oa_id FROM oa_keys)""")
110+ rep["needed_oa_ids_second_pass"] = dict(q("SELECT why, count(*) FROM needed_oa_ids GROUP BY 1"))
111+ rep["slim_total"] = q("SELECT count(*) FROM oa_slim")[0][0]
112+ rep["slim_cs_any_2018_2026"] = q("SELECT count(*) FROM oa_slim WHERE cs_any_topic AND publication_year BETWEEN 2018 AND 2026")[0][0]
113+ rep["seconds"] = round(time.time() - t0)
114+ os.makedirs(os.path.join(R, "data/metadata"), exist_ok=True)
115+ json.dump(rep, open(os.path.join(R, "data/metadata/linkage_report.json"), "w"), indent=1, default=str)
116+ con.execute(f"COPY (SELECT oa_id FROM needed_oa_ids) TO '{R}/data/metadata/needed_oa_ids.csv' (HEADER)")
117+ print(json.dumps(rep, indent=1, default=str))
118+
119+
120+if __name__ == "__main__":
121+ main()
added ingestion/openalex_api_increment.py +66 −0
Large diff (66 lines) is not loaded by default to keep the page fast.Load diff
added ingestion/openalex_entities.py +41 −0
Large diff (41 lines) is not loaded by default to keep the page fast.Load diff
added ingestion/openalex_snapshot.py +245 −0
Large diff (245 lines) is not loaded by default to keep the page fast.Load diff
added ingestion/openalex_topics_flat.py +17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added manuscript/.latexmkrc +5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added manuscript/Makefile +11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added manuscript/sections/titlepage.tex +77 −0
Large diff (77 lines) is not loaded by default to keep the page fast.Load diff
added manuscript/sections/titlepage_anon.tex +53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added manuscript/uq_logo.jpg +0 −0

Binary file not shown.

added protocol/PROTOCOL.md +96 −0
Large diff (96 lines) is not loaded by default to keep the page fast.Load diff
added protocol/amendments.csv +2 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added reports/PHASE0_FEASIBILITY.md +54 −0
Large diff (54 lines) is not loaded by default to keep the page fast.Load diff
added requirements.txt +22 −0
Large diff (22 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/dblp_streams_2018_2026.csv +3571 −0
Large diff (3,571 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/openalex_topics_2026-10-14.csv +4517 −0
Large diff (4,518 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere