SPB Git

spb/wp44_uqo

Public
7.6 KB · 97 lines markdown
1# WP44 — Preregistered protocol (v1.0, frozen 2026-11-02 before any analysis of the corpus)23Systematic mapping, bibliometric analysis and knowledge graph of computer-science research, 2022-11-30 → 2026-11-02,4with the 2018-01-01 → 2022-11-29 baseline. This protocol follows PRISMA 2020 reporting items where they apply to a5mapping study (items on risk of bias and effect measures are replaced by bibliometric quality indicators). Amendments6after the freeze are logged in `protocol/amendments.csv` with date and reason; analyses added after the freeze are labelled7*exploratory* in the manuscript.89## P1. Research questions and pre-specified hypotheses1011RQ1 volume/composition; RQ2 acceleration/structural change; RQ3 influential contributions; RQ4 citation networks and12interdisciplinary flows; RQ5 foundation models and disciplinary boundaries; RQ6 institutions, countries, collaboration;13RQ7 breakthroughs vs incremental benchmark gains; RQ8 unresolved problems; RQ9 reproducibility/open-source/evaluation14standards; RQ10 early detection of emerging paradigms (CLAUDE.md §5).1516Pre-specified, falsifiable hypotheses (tested in Phase 4; the direction is stated so that null or opposite results are reported as such):1718- H1 (volume): the monthly growth rate of CS works is higher in P than in B, but the *level* shift at 2022-11 is not19  distinguishable from placebo breaks in B once seasonality and OpenAlex indexing lag are modelled. (Two-sided; we expect20  growth differences to be concentrated in a few categories, not uniform.)21- H2 (composition): the share of AI-related categories (1–8, 38) in CS output increases in P; the share of at least one22  non-AI area (theory, systems, networks) declines in relative terms while growing in absolute terms.23- H3 (structural break): Bai–Perron tests on category-level monthly series locate at least one break in [2022-10, 2023-06]24  for LLM, generative AI, agents and AI safety; for formal verification, complexity, operating systems and computer25  architecture no break is detected in that window at the 5 % level after placebo correction.26- H4 (convergence): the share of citations crossing area boundaries (e.g. systems → AI, security → AI, HCI → AI)27  rises in P relative to B; the citation-flow network's modularity (by area) decreases.28- H5 (concentration): citation inequality (Gini of 24-month citations within year) is higher in P than in B for AI29  categories; venue concentration (HHI by source) is stable or lower because of preprint growth.30- H6 (geography/collaboration): the share of works with industry affiliations and with China-affiliated authors rises in31  P in AI categories; international co-authorship share does not rise.32- H7 (preprints and openness): the share of works with an arXiv version and the share with linked code rise in P.33- H8 (detection): the emerging-topic detector trained on cutoffs ≤ 2022 achieves AUC > 0.70 and precision@50 > 0.40 on34  test cutoffs 2023–2024 against the baseline-growth z-score rule; the pre-registered known-emergence set35  (large language models; diffusion models; neural radiance fields; LLM agents; retrieval-augmented generation;36  state-space sequence models; vision–language models; mechanistic interpretability; speculative decoding; mixture of37  experts at scale; test-time compute / reasoning models; AI for code generation) is detected with a median lead time ≥ 638  months before the topic's peak growth month.3940## P2. Eligibility criteria (PRISMA items 5–6)4142Inclusion: works first published (preprint or version of record, earliest date) between 2018-01-01 and 2026-11-02,43in computer science by the rule: DBLP-indexed OR OpenAlex primary field = Computer Science (17) OR (any OpenAlex CS topic44AND arXiv primary category cs.*). Types: article, preprint, proceedings-article/inproceedings, book-chapter, book,45dissertation, report. Any language.46Exclusion (with reason codes, counted): E1 outside window; E2 paratext/editorial/erratum/letter/peer-review/other47non-research types; E3 XPAC expansion records (`is_xpac`); E4 no title; E5 duplicate version (merged); E6 retracted48(kept in a flagged side table, excluded from impact analyses); E7 CS penumbra (secondary CS topic only, no DBLP/arXiv49support) — reported separately and used in sensitivity analyses only.5051## P3. Information sources and search (items 6–7)5253See `RESEARCH_PLAN.md` §2 and `reports/PHASE0_FEASIBILITY.md`. There is no keyword search: the corpus is defined by54database scope filters (DBLP full dump; OpenAlex topic field 17; arXiv set cs; ACL Anthology full), which is the55appropriate strategy for a field-level mapping. Dates of extraction: DBLP dump 2026-11-02 (file dated 2026-11-02 22:39 UTC);56OpenAlex release 2026-10-14 + API increment to 2026-11-02; arXiv OAI harvest 2026-11-02/03; ACL 2026-11-02.57Semantic Scholar: unavailable (API key), recorded.5859## P4. Selection, data collection and items (items 8–10)6061Automated stages with counts; record linkage precision estimated on a manually reviewed sample of 500 pairs; a 200-pair62subsample is reserved for PI review (checkpoint H-1). Data items per work: identifiers (OpenAlex W, DOI, arXiv, DBLP key,63ACL id), dates (preprint, publication, OpenAlex created), type, title, abstract (where available), language, venue64(source id, type, name, DBLP stream), authorships (author id, position, institutions with country and type), topics65(OpenAlex, arXiv categories), references (OpenAlex ids), citation counts (total; per year from `counts_by_year`), OA66status and licence, FWCI and percentile, taxonomy categories with scores, code/benchmark links (Phase 3).6768## P5. Quality assessment (item 11, adapted)6970Bibliometric quality indicators instead of risk of bias: metadata completeness (abstract, references, affiliations),71source coverage overlap (DBLP∩OpenAlex, arXiv∩OpenAlex), indexing lag profiles, duplicate rate, link precision,72retraction rate. Reported per year and per category in the appendix.7374## P6. Synthesis methods (items 13–14)7576Descriptive mapping (counts, shares, growth, concentration); bibliometric indicators as defined in `RESEARCH_PLAN.md` §6;77structural change (ITS with HAC errors, Bai–Perron, PELT, BOCPD; placebo breaks; alternative dates; indexing-lag78correction); network analysis (co-authorship, institution, country, citation-flow by category; modularity, cross-area79flows, Rao–Stirling diversity); semantic mapping (SPECTER2-class and a second embedding family; UMAP/HDBSCAN; topic80stability by bootstrap ARI); knowledge-graph extraction for the high-priority set with provenance/confidence/verification;81emerging-topic detection evaluated strictly out-of-time. All uncertainty reported with bootstrap or HAC intervals.82Known limitations declared before analysis: right-truncation of 2026 (incomplete indexing), citation-age bias,83OpenAlex topic classifier errors, DBLP coverage bias toward established venues, abstracts missing for a share of works,84no full-text access for closed works, LLM extraction errors (quantified on validated samples).8586## P7. Human-review checkpoints8788H-0 plan and protocol (this file + RESEARCH_PLAN §10); H-1 linkage sample; H-2 taxonomy validation sample; H-3 landmark89list and specialist reviews; H-4 structural-change results and their interpretation; H-5 manuscript conclusions.90Until the PI signs a checkpoint, the manuscript labels the corresponding conclusions "pending human review".9192## P8. Data and code availability9394Code, ontology, ledgers, manifests, hashes, PRISMA counts, aggregated tables and figures are released with the paper95(repository `wp44_uqo`). Record-level derived data are released where the source licence permits (OpenAlex CC0, DBLP96ODC-BY, arXiv metadata, ACL CC BY); full texts are never redistributed.97
Files · 37cde265fb11386038d259d622596446eb28bf77
Menu

Theme