SPB Git

spb/wp44_uqo

Public
7.6 KB

# WP44 — Preregistered protocol (v1.0, frozen 2026-11-02 before any analysis of the corpus)

Systematic mapping, bibliometric analysis and knowledge graph of computer-science research, 2022-11-30 → 2026-11-02, with the 2018-01-01 → 2022-11-29 baseline. This protocol follows PRISMA 2020 reporting items where they apply to a mapping study (items on risk of bias and effect measures are replaced by bibliometric quality indicators). Amendments after the freeze are logged in protocol/amendments.csv with date and reason; analyses added after the freeze are labelled exploratory in the manuscript.

# P1. Research questions and pre-specified hypotheses

RQ1 volume/composition; RQ2 acceleration/structural change; RQ3 influential contributions; RQ4 citation networks and interdisciplinary flows; RQ5 foundation models and disciplinary boundaries; RQ6 institutions, countries, collaboration; RQ7 breakthroughs vs incremental benchmark gains; RQ8 unresolved problems; RQ9 reproducibility/open-source/evaluation standards; RQ10 early detection of emerging paradigms (CLAUDE.md §5).

Pre-specified, falsifiable hypotheses (tested in Phase 4; the direction is stated so that null or opposite results are reported as such):

  • H1 (volume): the monthly growth rate of CS works is higher in P than in B, but the level shift at 2022-11 is not distinguishable from placebo breaks in B once seasonality and OpenAlex indexing lag are modelled. (Two-sided; we expect growth differences to be concentrated in a few categories, not uniform.)
  • H2 (composition): the share of AI-related categories (1–8, 38) in CS output increases in P; the share of at least one non-AI area (theory, systems, networks) declines in relative terms while growing in absolute terms.
  • H3 (structural break): Bai–Perron tests on category-level monthly series locate at least one break in [2022-10, 2023-06] for LLM, generative AI, agents and AI safety; for formal verification, complexity, operating systems and computer architecture no break is detected in that window at the 5 % level after placebo correction.
  • H4 (convergence): the share of citations crossing area boundaries (e.g. systems → AI, security → AI, HCI → AI) rises in P relative to B; the citation-flow network's modularity (by area) decreases.
  • H5 (concentration): citation inequality (Gini of 24-month citations within year) is higher in P than in B for AI categories; venue concentration (HHI by source) is stable or lower because of preprint growth.
  • H6 (geography/collaboration): the share of works with industry affiliations and with China-affiliated authors rises in P in AI categories; international co-authorship share does not rise.
  • H7 (preprints and openness): the share of works with an arXiv version and the share with linked code rise in P.
  • H8 (detection): the emerging-topic detector trained on cutoffs ≤ 2022 achieves AUC > 0.70 and precision@50 > 0.40 on test cutoffs 2023–2024 against the baseline-growth z-score rule; the pre-registered known-emergence set (large language models; diffusion models; neural radiance fields; LLM agents; retrieval-augmented generation; state-space sequence models; vision–language models; mechanistic interpretability; speculative decoding; mixture of experts at scale; test-time compute / reasoning models; AI for code generation) is detected with a median lead time ≥ 6 months before the topic's peak growth month.

# P2. Eligibility criteria (PRISMA items 5–6)

Inclusion: works first published (preprint or version of record, earliest date) between 2018-01-01 and 2026-11-02, in computer science by the rule: DBLP-indexed OR OpenAlex primary field = Computer Science (17) OR (any OpenAlex CS topic AND arXiv primary category cs.*). Types: article, preprint, proceedings-article/inproceedings, book-chapter, book, dissertation, report. Any language. Exclusion (with reason codes, counted): E1 outside window; E2 paratext/editorial/erratum/letter/peer-review/other non-research types; E3 XPAC expansion records (is_xpac); E4 no title; E5 duplicate version (merged); E6 retracted (kept in a flagged side table, excluded from impact analyses); E7 CS penumbra (secondary CS topic only, no DBLP/arXiv support) — reported separately and used in sensitivity analyses only.

# P3. Information sources and search (items 6–7)

See RESEARCH_PLAN.md §2 and reports/PHASE0_FEASIBILITY.md. There is no keyword search: the corpus is defined by database scope filters (DBLP full dump; OpenAlex topic field 17; arXiv set cs; ACL Anthology full), which is the appropriate strategy for a field-level mapping. Dates of extraction: DBLP dump 2026-11-02 (file dated 2026-11-02 22:39 UTC); OpenAlex release 2026-10-14 + API increment to 2026-11-02; arXiv OAI harvest 2026-11-02/03; ACL 2026-11-02. Semantic Scholar: unavailable (API key), recorded.

# P4. Selection, data collection and items (items 8–10)

Automated stages with counts; record linkage precision estimated on a manually reviewed sample of 500 pairs; a 200-pair subsample is reserved for PI review (checkpoint H-1). Data items per work: identifiers (OpenAlex W, DOI, arXiv, DBLP key, ACL id), dates (preprint, publication, OpenAlex created), type, title, abstract (where available), language, venue (source id, type, name, DBLP stream), authorships (author id, position, institutions with country and type), topics (OpenAlex, arXiv categories), references (OpenAlex ids), citation counts (total; per year from counts_by_year), OA status and licence, FWCI and percentile, taxonomy categories with scores, code/benchmark links (Phase 3).

# P5. Quality assessment (item 11, adapted)

Bibliometric quality indicators instead of risk of bias: metadata completeness (abstract, references, affiliations), source coverage overlap (DBLP∩OpenAlex, arXiv∩OpenAlex), indexing lag profiles, duplicate rate, link precision, retraction rate. Reported per year and per category in the appendix.

# P6. Synthesis methods (items 13–14)

Descriptive mapping (counts, shares, growth, concentration); bibliometric indicators as defined in RESEARCH_PLAN.md §6; structural change (ITS with HAC errors, Bai–Perron, PELT, BOCPD; placebo breaks; alternative dates; indexing-lag correction); network analysis (co-authorship, institution, country, citation-flow by category; modularity, cross-area flows, Rao–Stirling diversity); semantic mapping (SPECTER2-class and a second embedding family; UMAP/HDBSCAN; topic stability by bootstrap ARI); knowledge-graph extraction for the high-priority set with provenance/confidence/verification; emerging-topic detection evaluated strictly out-of-time. All uncertainty reported with bootstrap or HAC intervals. Known limitations declared before analysis: right-truncation of 2026 (incomplete indexing), citation-age bias, OpenAlex topic classifier errors, DBLP coverage bias toward established venues, abstracts missing for a share of works, no full-text access for closed works, LLM extraction errors (quantified on validated samples).

# P7. Human-review checkpoints

H-0 plan and protocol (this file + RESEARCH_PLAN §10); H-1 linkage sample; H-2 taxonomy validation sample; H-3 landmark list and specialist reviews; H-4 structural-change results and their interpretation; H-5 manuscript conclusions. Until the PI signs a checkpoint, the manuscript labels the corresponding conclusions "pending human review".

# P8. Data and code availability

Code, ontology, ledgers, manifests, hashes, PRISMA counts, aggregated tables and figures are released with the paper (repository wp44_uqo). Record-level derived data are released where the source licence permits (OpenAlex CC0, DBLP ODC-BY, arXiv metadata, ACL CC BY); full texts are never redistributed.

Files · 7d8c8f8592599c79aba715f8bff62d0f0aa710b4
Menu

Theme