SPB Git
spboucher.ai

spb/wp44_uqo

Public

WP44: corpus definition (A-001, A-002), structural-change toolkit + tests, bibliometrics/graph/embedding scripts, manuscript skeleton

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 3, 2026) parent 05d5bd4 Browse files

55 changed files +2,503 −14

55 changed files
Load all 55 files Unified Split
added agents/REGISTER.md +8 −0
@@ -0,0 +1,8 @@
1+# Agent register — WP44 (who produced what, who verified it)
2+
3+| Date | Agent | Brief | Artefact(s) | Verified by | Status |
4+|---|---|---|---|---|---|
5+| 2026-11-02 | A (director, this session) | CLAUDE.md | RESEARCH_PLAN.md, protocol/PROTOCOL.md, reports/PHASE0_FEASIBILITY.md | PI checkpoint H-0 pending | delivered |
6+| 2026-11-02/03 | B+C (acquisition & data engineering, this session) | RESEARCH_PLAN §2–3 | ingestion/*.py, ledgers on M3U96b, data/metadata/linkage_report.json | N (audit, Phase 8) pending | delivered |
7+| 2026-11-03 | D (taxonomy architect, subagent, 2nd attempt) | agents/briefs/AGENT_D_TAXONOMY.md | taxonomy/ontology.yaml, topic_map.csv, ONTOLOGY.md, src/ | A + K (validation sample, Phase 2) | running |
8+| 2026-11-03 | B-lit (reference acquisition, subagent) | agents/briefs/AGENT_LIT_REFERENCES.md | manuscript/references.bib, references_verification.csv, literature_synthesis/REFERENCE_MAP.md | M (compile) + N (re-verify sample) | running |
added agents/briefs/AGENT_LIT_REFERENCES.md +78 −0
@@ -0,0 +1,78 @@
1+# Brief — reference acquisition for the WP44 manuscript (Agent B-lit, 2026-11-03)
2+
3+Goal: build the *verified* bibliography that the manuscript's methodological and historical parts need, BEFORE any
4+analysis result exists. Every entry must be resolved to a real bibliographic record; nothing may be invented.
5+
6+Output files (only these):
7+1. `manuscript/references.bib` — BibTeX, keys `surnameYYYYkeyword` (lower-case ASCII), one entry per work, fields
8+ title/author/year/journal-or-booktitle/volume/pages/doi (or `eprint`+`archiveprefix={arXiv}` for preprints, or `url`
9+ for software/datasets/reports), `note = {verified: <method> <date>}`.
10+2. `manuscript/references_verification.csv` — columns: key, doi_or_arxiv, verified_via (crossref|openalex|arxiv|datacite|url),
11+ verified_title (as returned by the service), match (exact|minor|FAILED), note. Entries that FAIL verification must be
12+ removed from references.bib and kept in the CSV with match=FAILED.
13+3. `literature_synthesis/REFERENCE_MAP.md` — for each manuscript part (I–XI, appendices), the list of keys it should cite
14+ and one line on why (what the reference supports).
15+
16+Verification method (mandatory, per entry): `curl -s "https://api.crossref.org/works/<DOI>?mailto=spbou4@icloud.com"` or
17+`curl -s "https://api.openalex.org/works/https://doi.org/<DOI>"` (single lookups are free) or
18+`curl -s "https://export.arxiv.org/api/query?id_list=<id>"`; compare the returned title with yours. Do not use web search
19+results as verification. If you cannot verify an item, drop it.
20+
21+Scope (target 120–180 verified entries), grouped:
22+A. Scientometric data sources: OpenAlex (Priem, Piwowar, Orr 2022 arXiv:2205.01833), DBLP (Ley 2002/2009), arXiv
23+ (Ginsparg), Semantic Scholar (Kinney et al. 2023 S2AG), Crossref, ACL Anthology, Microsoft Academic Graph (Wang et al.
24+ 2020), CiteSeerX, Papers with Code; OpenAlex topic classification (OpenAlex topic model documentation, 2024).
25+B. Systematic mapping / review methodology: PRISMA 2020 (Page et al. 2021 BMJ), PRISMA-S, Kitchenham & Charters 2007
26+ guidelines (SE), Petersen et al. 2008/2015 systematic mapping, Tricco scoping reviews, living reviews, automation of
27+ screening (e.g., ASReview van de Schoot 2021), LLM-assisted screening evaluations 2023–2025.
28+C. Bibliometric indicators: Waltman 2016 review; field-normalised indicators (Waltman & van Eck 2019; MNCS; FWCI);
29+ citation windows and age bias (Wang 2013 "Citation time window choice"); h-index (Hirsch 2005); Gini/citation
30+ inequality (Nielsen & Andersen 2021); interdisciplinarity (Rao–Stirling: Stirling 2007; Leinster & Cobbold 2012;
31+ Porter & Rafols 2009; Leydesdorff); co-authorship networks (Newman 2001, 2004); country/institution analyses (Wagner
32+ 2017 international collaboration); preprints in CS (Lin et al. 2020 arXiv; Fraser 2021 bioRxiv analogies); citation
33+ advantage of OA (Piwowar 2018).
34+D. Science mapping & embeddings: VOSviewer (van Eck & Waltman 2010), Small 1973 co-citation, Kessler bibliographic
35+ coupling, Boyack & Klavans 2010 accuracy of clustering, Leiden algorithm (Traag 2019), Louvain (Blondel 2008), SPECTER
36+ (Cohan et al. 2020), SPECTER2 (Singh et al. 2023 SciRepEval), SciNCL (Ostendorff 2022), SciBERT (Beltagy 2019),
37+ sentence-BERT (Reimers 2019), UMAP (McInnes 2018), HDBSCAN (Campello 2013; McInnes 2017), BERTopic (Grootendorst 2022),
38+ LDA (Blei 2003), dynamic topic models (Blei & Lafferty 2006), topic stability (Greene 2014).
39+E. Structural change & time series: Bai & Perron 1998, 2003; Andrews 1993; Killick, Fearnhead & Eckley 2012 (PELT);
40+ Adams & MacKay 2007 (BOCPD); Truong, Oudre & Vayatis 2020 (ruptures review); Newey & West 1987; interrupted time series
41+ (Bernal, Cummins, Gasparrini 2017); synthetic control (Abadie 2010); event-study pitfalls.
42+F. Emerging topic / technology detection: Rotolo, Hicks & Martin 2015; Small, Boyack & Klavans 2014; Cozzens et al.
43+ 2010; Wang Q. 2018 "A bibliometric model for identifying emerging research topics"; Klavans & Boyack 2017 research
44+ fronts; sleeping beauties (Ke, Ferrara, Radicchi, Flammini 2015); disruption index (Funk & Owen-Smith 2017; Wu, Wang,
45+ Evans 2019; Park, Leahey, Funk 2023 "Papers and patents are becoming less disruptive"); novelty (Uzzi et al. 2013);
46+ burst detection (Kleinberg 2003).
47+G. Historical/contextual works on the AI transformation (only landmark, verified): Transformer (Vaswani 2017), BERT
48+ (Devlin 2019), GPT-3 (Brown 2020), scaling laws (Kaplan 2020; Hoffmann 2022), InstructGPT (Ouyang 2022), ChatGPT
49+ release (OpenAI blog 2022-11-30 — cite as web page with access date), GPT-4 technical report (2023), Llama 1/2/3
50+ (Touvron 2023; Dubey 2024), diffusion (Ho 2020; Rombach 2022), CLIP (Radford 2021), ViT (Dosovitskiy 2021), NeRF
51+ (Mildenhall 2020), RLHF (Christiano 2017), chain-of-thought (Wei 2022), RAG (Lewis 2020), ReAct (Yao 2023), Toolformer
52+ (Schick 2023), Mamba (Gu & Dao 2023), MoE Switch (Fedus 2022), DeepSeek-R1 (2025), o1 system card (2024), AlphaFold 2
53+ (Jumper 2021), AlphaGo/AlphaZero, foundation-model report (Bommasani 2021), "Sparks of AGI" (Bubeck 2023), stochastic
54+ parrots (Bender 2021), Stanford AI Index 2024/2025 reports, Epoch AI compute trends (Sevilla 2022).
55+H. Evidence on LLM effects on science: Liang et al. 2024 (LLM-modified content in reviews/papers), Kobak et al. 2024/2025
56+ ("delving"), Geng & Trotta 2024, Bin-Nashwan / Dwivedi 2023 "So what if ChatGPT wrote it?", Birhane 2023, Messeri &
57+ Crockett 2024 (illusions of understanding), Nature surveys on AI use by researchers (Van Noorden 2023), arXiv
58+ submission growth analyses, "AI research is growing faster than…" (Krenn et al. 2023 Nature Machine Intelligence
59+ forecasting AI research), Fortunato et al. 2018 Science of science, Wang & Barabási 2021 book.
60+I. Reproducibility/open science in CS: Gundersen & Kjensmo 2018; Pineau et al. 2021 (ML reproducibility program);
61+ Kapoor & Narayanan 2023 (leakage); Papers with Code / Hugging Face ecosystem studies; benchmark saturation (Ott et al.
62+ 2022 "Mapping global dynamics of benchmark creation and saturation"; Dehghani 2021 benchmark lottery; Raji 2021 AI and
63+ the everything in the whole wide world benchmark); Sculley 2018; ACM artifact badging policy (web).
64+J. Other CS areas' survey anchors (one or two verified surveys each, 2022–2026 preferred): quantum computing (Preskill
65+ 2018 NISQ; a 2023–2025 survey), blockchain, federated learning (McMahan 2017; Kairouz 2021), differential privacy
66+ (Dwork 2006/2014), post-quantum cryptography (NIST FIPS 203/204/205 2024), formal verification (Lean mathlib / Rust
67+ verification surveys), computer architecture (Hennessy & Patterson 2019 Turing lecture; "golden age"), AI accelerators
68+ surveys, sustainable computing (Patterson 2021 carbon; Strubell 2019; Luccioni 2023), edge computing (Shi 2016), robotics
69+ foundation models (RT-2 2023; survey), AI for science (Wang H. et al. 2023 Nature "Scientific discovery in the age of
70+ AI"), AI safety/alignment surveys (Hendrycks 2021; Ji et al. 2023 alignment survey), agents surveys (Wang L. et al. 2024;
71+ Xi et al. 2023), LLM surveys (Zhao et al. 2023), multimodal surveys, speech (Whisper 2023), information retrieval
72+ (dense retrieval Karpukhin 2020), databases (vector databases survey 2024), compilers (MLIR Lattner 2021), programming
73+ languages (Rust safety), operating systems (eBPF / unikernels / LLM-OS), networks (programmable networks / P4), security
74+ (LLM security surveys 2024; prompt injection Greshake 2023), cryptography (FHE surveys), computational biology
75+ (AlphaFold 3 2024; ESM-2 2023), graphics (Gaussian splatting Kerbl 2023), HCI (Generative AI & HCI 2023–2024 surveys).
76+
77+Rules: work through the groups in order; verify as you go; keep each file write small (append to the .bib in batches of
78+~20 entries with `cat >>`). Report back ≤ 25 lines: counts per group, failures, and anything you could not find.
added bibliometrics/01_volume_series.py +36 −0
@@ -0,0 +1,36 @@
1+#!/usr/bin/env python3
2+"""WP44 — RQ1 volume series from corpus_works (eligible, CS core) with indexing-lag diagnostics.
3+Outputs (results/bibliometrics/): volume_monthly.csv (first_month × type × membership), volume_yearly.csv,
4+indexing_lag.csv (publication month → distribution of created_date − publication_date), penumbra_monthly.csv,
5+dblp_only_yearly.csv. All tables are regenerated from the DuckDB database; nothing is hand-edited."""
6+import argparse, os, duckdb
7+
8+
9+def main():
10+ ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
11+ out = os.path.join(R, "results/bibliometrics"); os.makedirs(out, exist_ok=True)
12+ con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb"), read_only=True); con.execute("SET threads TO 16;")
13+ con.execute(f"""COPY (SELECT first_month, period, type, in_dblp, cs_primary, (arxiv_id_linked IS NOT NULL) AS has_arxiv, count(*) AS n,
14+ sum(has_abstract::int) AS n_abstract, sum((n_refs>0)::int) AS n_with_refs, sum(is_oa::int) AS n_oa
15+ FROM corpus_works WHERE eligible GROUP BY ALL ORDER BY 1,2,3) TO '{out}/volume_monthly.csv' (HEADER)""")
16+ con.execute(f"""COPY (SELECT publication_year AS year, count(*) AS n, sum(in_dblp::int) AS n_dblp, sum(cs_primary::int) AS n_cs_primary,
17+ sum((arxiv_id_linked IS NOT NULL)::int) AS n_arxiv, sum(in_acl::int) AS n_acl, sum(has_abstract::int) AS n_abstract,
18+ sum(is_oa::int) AS n_oa, sum((type='preprint')::int) AS n_preprint, sum((type='conference-paper')::int) AS n_conf,
19+ sum((type='article')::int) AS n_article, sum(e6_retracted::int) AS n_retracted, sum(xpac_kept_dblp::int) AS n_xpac_dblp,
20+ round(avg(n_authors),2) AS mean_authors, round(avg(n_refs),2) AS mean_refs, round(avg(n_countries),3) AS mean_countries
21+ FROM corpus_works WHERE eligible GROUP BY 1 ORDER BY 1) TO '{out}/volume_yearly.csv' (HEADER)""")
22+ con.execute(f"""COPY (SELECT substr(publication_date,1,7) AS pub_month, count(*) AS n,
23+ quantile_cont(datediff('day', publication_date::DATE, created_date::DATE), 0.5) AS lag_days_p50,
24+ quantile_cont(datediff('day', publication_date::DATE, created_date::DATE), 0.9) AS lag_days_p90,
25+ sum((datediff('day', publication_date::DATE, created_date::DATE) <= 90)::int) AS n_indexed_within_90d,
26+ sum((created_date::DATE > publication_date::DATE + INTERVAL 180 DAY)::int) AS n_indexed_after_180d
27+ FROM corpus_works WHERE eligible AND created_date IS NOT NULL AND publication_date IS NOT NULL GROUP BY 1 ORDER BY 1) TO '{out}/indexing_lag.csv' (HEADER)""")
28+ con.execute(f"""COPY (SELECT first_month, count(*) AS n_penumbra FROM corpus_works WHERE cs_penumbra AND NOT e1_outside_window AND NOT e2_type AND NOT e3_xpac AND NOT e4_no_title AND NOT e5_duplicate
29+ GROUP BY 1 ORDER BY 1) TO '{out}/penumbra_monthly.csv' (HEADER)""")
30+ con.execute(f"COPY (SELECT * FROM dblp_only_counts) TO '{out}/dblp_only_yearly.csv' (HEADER)")
31+ for f in ["volume_yearly.csv", "indexing_lag.csv"]:
32+ print(f, open(os.path.join(out, f)).read()[:1500])
33+
34+
35+if __name__ == "__main__":
36+ main()
added bibliometrics/02_collaboration.py +42 −0
@@ -0,0 +1,42 @@
1+#!/usr/bin/env python3
2+"""WP44 — RQ6 collaboration, institutions, countries (eligible corpus), yearly and by period.
3+Outputs (results/bibliometrics/): team_size_yearly.csv, countries_yearly.csv (top-N country output + share, with
4+fractional counting), international_collab_yearly.csv, institution_types_yearly.csv (education/company/government/…),
5+top_institutions_by_period.csv (full and fractional counts), country_pairs_by_period.csv (edges for the collaboration
6+network, ≥ 50 joint works), top_sources_by_period.csv (venue concentration inputs), venue_hhi_yearly.csv."""
7+import argparse, os, duckdb
8+
9+
10+def main():
11+ ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
12+ out = os.path.join(R, "results/bibliometrics"); os.makedirs(out, exist_ok=True)
13+ con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb"), read_only=True); con.execute("SET threads TO 20; SET memory_limit='60GB';")
14+ con.execute("CREATE OR REPLACE TEMP VIEW E AS SELECT * FROM corpus_works WHERE eligible")
15+ con.execute(f"""COPY (SELECT publication_year AS year, period, count(*) n, round(avg(n_authors),3) mean_authors, quantile_cont(n_authors,0.5) med_authors,
16+ sum((n_authors=1)::int) single, sum((n_authors>=5)::int) five_plus, sum((n_authors>=10)::int) ten_plus, sum((n_countries>=2)::int) intl,
17+ sum((n_institutions>=2)::int) multi_inst, sum((n_countries=0)::int) no_country FROM E GROUP BY 1,2 ORDER BY 1) TO '{out}/team_size_yearly.csv' (HEADER)""")
18+ # country output: full counting (work counted once per country) and fractional (1/n_countries)
19+ con.execute("""CREATE OR REPLACE TEMP TABLE wc AS SELECT w.id, w.publication_year AS year, w.period, unnest(w.countries) AS country, 1.0/len(w.countries) AS frac FROM E w WHERE len(w.countries) > 0""")
20+ con.execute(f"""COPY (SELECT year, country, count(*) n_full, round(sum(frac),1) n_frac FROM wc GROUP BY 1,2 ORDER BY 1, 3 DESC) TO '{out}/countries_yearly.csv' (HEADER)""")
21+ con.execute(f"""COPY (SELECT year, count(*) n, sum((n_countries>=2)::int) n_intl, round(sum((n_countries>=2)::int)*1.0/count(*),4) share_intl FROM E WHERE n_countries>0 GROUP BY 1 ORDER BY 1) TO '{out}/international_collab_yearly.csv' (HEADER)""")
22+ con.execute("""CREATE OR REPLACE TEMP TABLE wi AS SELECT w.id, w.publication_year AS year, w.period, unnest(w.institution_types) AS itype FROM E w WHERE len(w.institution_types) > 0""")
23+ con.execute(f"""COPY (SELECT year, itype, count(*) n FROM wi GROUP BY 1,2 ORDER BY 1,3 DESC) TO '{out}/institution_types_yearly.csv' (HEADER)""")
24+ con.execute(f"""COPY (SELECT year, count(DISTINCT id) n_with_inst, count(DISTINCT CASE WHEN itype='company' THEN id END) n_company,
25+ round(count(DISTINCT CASE WHEN itype='company' THEN id END)*1.0/count(DISTINCT id),4) share_company FROM wi GROUP BY 1 ORDER BY 1) TO '{out}/industry_share_yearly.csv' (HEADER)""")
26+ # institutions (from authorships): full and fractional (by distinct institutions per work)
27+ con.execute("""CREATE OR REPLACE TEMP TABLE winst AS
28+ SELECT DISTINCT a.work_id, unnest(a.institution_ids) AS inst_id FROM oa_authorships a WHERE a.work_id IN (SELECT id FROM E) AND len(a.institution_ids) > 0""")
29+ con.execute("""CREATE OR REPLACE TEMP TABLE winst2 AS SELECT w.work_id, w.inst_id, 1.0/count(*) OVER (PARTITION BY w.work_id) AS frac, e.period, e.publication_year AS year FROM winst w JOIN E e ON e.id = w.work_id""")
30+ con.execute(f"""COPY (SELECT period, inst_id, count(*) n_full, round(sum(frac),1) n_frac FROM winst2 GROUP BY 1,2 HAVING count(*) >= 200 ORDER BY 1, 3 DESC) TO '{out}/top_institutions_by_period.csv' (HEADER)""")
31+ con.execute(f"""COPY (SELECT a.period, a.country AS c1, b.country AS c2, count(DISTINCT a.id) n FROM wc a JOIN wc b ON a.id = b.id AND a.country < b.country
32+ GROUP BY 1,2,3 HAVING count(DISTINCT a.id) >= 50 ORDER BY 1, 4 DESC) TO '{out}/country_pairs_by_period.csv' (HEADER)""")
33+ con.execute(f"""COPY (SELECT period, source_id, any_value(source_name) source_name, any_value(source_type) source_type, count(*) n FROM E WHERE source_id IS NOT NULL
34+ GROUP BY 1,2 HAVING count(*) >= 100 ORDER BY 1, 5 DESC) TO '{out}/top_sources_by_period.csv' (HEADER)""")
35+ con.execute(f"""COPY (SELECT year, count(*) n, count(DISTINCT source_id) n_sources, round(sum(s*s),6) hhi, round(max(s),5) top1_share FROM
36+ (SELECT publication_year AS year, source_id, count(*)*1.0/sum(count(*)) OVER (PARTITION BY publication_year) AS s FROM E WHERE source_id IS NOT NULL GROUP BY 1,2)
37+ GROUP BY 1 ORDER BY 1) TO '{out}/venue_hhi_yearly.csv' (HEADER)""")
38+ print(open(f"{out}/team_size_yearly.csv").read()[:800]); print(open(f"{out}/industry_share_yearly.csv").read())
39+
40+
41+if __name__ == "__main__":
42+ main()
added bibliometrics/03_citation_impact.py +44 −0
@@ -0,0 +1,44 @@
1+#!/usr/bin/env python3
2+"""WP44 — RQ3/RQ4 citation impact with explicit citation-window control.
3+
4+From oa_counts_by_year (citations received per calendar year) we build fixed-window counts:
5+ c12 = citations received in the publication year and the following year (≈ 0–24 months), c36 = pub year + 2 years.
6+Normalisation: NCS_k = c_k / mean(c_k | same publication year, same type class, same primary subfield) over the eligible
7+corpus (field-normalised within OpenAlex subfield; category-level normalisation is added in 05_category_impact.py).
8+Outputs: results/bibliometrics/citation_windows_yearly.csv (means, medians, shares uncited, Gini of c36 by year),
9+work_citation_windows.parquet (work-level c12/c36/NCS for downstream scripts), top_cited_by_year.csv (top 50 per year by c36),
10+fwci_summary_yearly.csv (OpenAlex FWCI as a secondary indicator)."""
11+import argparse, os, duckdb
12+
13+
14+def main():
15+ ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
16+ out = os.path.join(R, "results/bibliometrics"); os.makedirs(out, exist_ok=True)
17+ con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb")); con.execute("SET threads TO 20; SET memory_limit='60GB';")
18+ con.execute("""CREATE OR REPLACE TABLE work_citation_windows AS
19+ WITH E AS (SELECT id, publication_year AS y, type, primary_subfield_id AS sf, cited_by_count, fwci FROM corpus_works WHERE eligible),
20+ C AS (SELECT c.work_id, sum(CASE WHEN c.year <= e.y + 1 THEN c.cited_by_count ELSE 0 END) AS c12,
21+ sum(CASE WHEN c.year <= e.y + 2 THEN c.cited_by_count ELSE 0 END) AS c36,
22+ sum(CASE WHEN c.year <= e.y + 3 THEN c.cited_by_count ELSE 0 END) AS c48
23+ FROM oa_counts_by_year c JOIN E e ON e.id = c.work_id GROUP BY 1)
24+ SELECT e.id AS work_id, e.y AS year, e.type, e.sf, e.cited_by_count, e.fwci, coalesce(c.c12,0) c12, coalesce(c.c36,0) c36, coalesce(c.c48,0) c48
25+ FROM E e LEFT JOIN C c ON c.work_id = e.id""")
26+ con.execute("""CREATE OR REPLACE TABLE work_citation_windows2 AS
27+ SELECT w.*, CASE WHEN m12 > 0 THEN c12 / m12 END AS ncs12, CASE WHEN m36 > 0 THEN c36 / m36 END AS ncs36
28+ FROM work_citation_windows w JOIN (SELECT year, type, sf, avg(c12) m12, avg(c36) m36, count(*) n_ref FROM work_citation_windows GROUP BY 1,2,3) r
29+ USING (year, type, sf)""")
30+ con.execute("DROP TABLE work_citation_windows; ALTER TABLE work_citation_windows2 RENAME TO work_citation_windows")
31+ con.execute(f"COPY work_citation_windows TO '{R}/data/processed/corpus/work_citation_windows.parquet' (FORMAT PARQUET, COMPRESSION ZSTD)")
32+ con.execute(f"""COPY (SELECT year, count(*) n, round(avg(c12),3) mean_c12, round(avg(c36),3) mean_c36, quantile_cont(c36,0.5) med_c36,
33+ round(sum((c36=0)::int)*1.0/count(*),4) share_uncited_36, round(sum((c12=0)::int)*1.0/count(*),4) share_uncited_12,
34+ quantile_cont(c36,0.99) p99_c36, max(c36) max_c36,
35+ round((2*sum(c36*rk) / (count(*)*sum(c36)) - (count(*)+1.0)/count(*)),4) gini_c36,
36+ round(avg(fwci),3) mean_fwci, round(quantile_cont(fwci,0.5),3) med_fwci
37+ FROM (SELECT *, row_number() OVER (PARTITION BY year ORDER BY c36) rk FROM work_citation_windows) GROUP BY 1 ORDER BY 1) TO '{out}/citation_windows_yearly.csv' (HEADER)""")
38+ con.execute(f"""COPY (SELECT * FROM (SELECT w.year, w.work_id, c.title, c.source_name, c.type, w.c12, w.c36, w.cited_by_count, w.fwci, w.ncs36,
39+ row_number() OVER (PARTITION BY w.year ORDER BY w.c36 DESC, w.cited_by_count DESC) rk FROM work_citation_windows w JOIN corpus_works c ON c.id = w.work_id) WHERE rk <= 50 ORDER BY year, rk) TO '{out}/top_cited_by_year.csv' (HEADER)""")
40+ print(open(f"{out}/citation_windows_yearly.csv").read())
41+
42+
43+if __name__ == "__main__":
44+ main()
added data/metadata/linkage_report.json +45 −0
@@ -0,0 +1,45 @@
1+{
2+ "created": "2026-11-03T20:50:56.182276+00:00",
3+ "oa_cs_records": 15490159,
4+ "oa_cs_distinct_ids": 15490159,
5+ "dblp_2018_2026": 4347365,
6+ "dblp_with_doi": 4052326,
7+ "dblp_with_arxiv": 0,
8+ "link_dblp_oa": {
9+ "title_year": 179990,
10+ "doi": 2562431
11+ },
12+ "dblp_linked_distinct": 2736903,
13+ "dblp_unlinked": 1610462,
14+ "dblp_unlinked_by_type": {
15+ "phdthesis": 26841,
16+ "incollection": 3362,
17+ "inproceedings": 477235,
18+ "book": 1398,
19+ "article": 1084640,
20+ "data": 16986
21+ },
22+ "dblp_unlinked_with_doi_in_slim": 1077407,
23+ "arxiv_cs_records": 915115,
24+ "arxiv_created_2018_2026": 890852,
25+ "link_arxiv_oa": {
26+ "arxiv_id": 792672,
27+ "doi": 520,
28+ "title_year": 2143
29+ },
30+ "arxiv_linked_distinct": 744680,
31+ "acl_records": 135558,
32+ "acl_2018_2026": 85625,
33+ "link_acl_oa": {
34+ "doi": 57474,
35+ "title_year": 14904
36+ },
37+ "acl_linked_distinct_2018_2026": 72223,
38+ "needed_oa_ids_second_pass": {
39+ "dblp_doi": 1077406,
40+ "arxiv_doi": 39200
41+ },
42+ "slim_total": 464765485,
43+ "slim_cs_any_2018_2026": 15490159,
44+ "seconds": 10
45+}
\ No newline at end of file
added econometrics/structural_change.py +135 −0
@@ -0,0 +1,135 @@
1+#!/usr/bin/env python3
2+"""WP44 — structural-change toolkit for monthly publication series (RQ2, hypotheses H1/H3).
3+
4+Functions (all pure, tested in tests/test_structural_change.py):
5+ its_fit(y, dates, event, lags) interrupted time series on log counts with month-of-year seasonality,
6+ level and slope change at `event`, Newey–West HAC standard errors
7+ placebo_breaks(y, dates, window) distribution of the ITS level/slope t-statistics over every candidate month of the
8+ baseline (placebo), returns the empirical rank/p-value of the true event
9+ bai_perron(y, dates, max_breaks) multiple structural breaks in mean + trend by dynamic programming (BIC), with
10+ minimum segment length h (Bai & Perron 1998/2003, pure-python DP on SSR)
11+ pelt(y, pen) PELT changepoints (ruptures, l2 cost) on log counts, penalty by BIC
12+ bocpd(y, hazard, mu0, kappa0, ...) Bayesian online changepoint detection (Adams & MacKay 2007), Student-t predictive,
13+ returns run-length posterior and P(changepoint at t)
14+Right-truncation: callers must stop the series at the last complete month (see bibliometrics/01_volume_series.py).
15+"""
16+from __future__ import annotations
17+import numpy as np, pandas as pd
18+import statsmodels.api as sm
19+
20+
21+def _design(dates: pd.DatetimeIndex, event: pd.Timestamp, seasonal=True):
22+ t = np.arange(len(dates), dtype=float)
23+ post = (dates >= event).astype(float)
24+ t_post = np.where(post == 1, t - t[post == 1][0] if post.sum() else 0, 0.0)
25+ X = {"const": np.ones(len(t)), "t": t, "post": post, "t_post": t_post}
26+ if seasonal:
27+ for m in range(2, 13):
28+ X[f"m{m}"] = (dates.month == m).astype(float)
29+ return pd.DataFrame(X, index=dates)
30+
31+
32+def its_fit(y, dates, event, lags=6, seasonal=True, log=True):
33+ dates = pd.DatetimeIndex(dates); event = pd.Timestamp(event)
34+ yy = np.log(np.asarray(y, dtype=float)) if log else np.asarray(y, dtype=float)
35+ X = _design(dates, event, seasonal)
36+ res = sm.OLS(yy, X).fit(cov_type="HAC", cov_kwds={"maxlags": lags})
37+ out = {"level_change": res.params["post"], "level_se": res.bse["post"], "level_t": res.tvalues["post"], "level_p": res.pvalues["post"],
38+ "slope_change": res.params["t_post"], "slope_se": res.bse["t_post"], "slope_t": res.tvalues["t_post"], "slope_p": res.pvalues["t_post"],
39+ "pre_slope": res.params["t"], "pre_slope_se": res.bse["t"], "n": int(res.nobs), "r2": res.rsquared, "event": str(event.date())}
40+ return out, res
41+
42+
43+def placebo_breaks(y, dates, true_event, min_pre=24, min_post=12, lags=6, seasonal=True):
44+ dates = pd.DatetimeIndex(dates)
45+ rows = []
46+ for i in range(min_pre, len(dates) - min_post):
47+ ev = dates[i]
48+ o, _ = its_fit(y, dates, ev, lags, seasonal)
49+ rows.append({"event": ev, "level_t": o["level_t"], "slope_t": o["slope_t"], "level_change": o["level_change"], "slope_change": o["slope_change"]})
50+ df = pd.DataFrame(rows)
51+ tv = df.loc[df.event == pd.Timestamp(true_event)]
52+ out = {"n_placebo": len(df)}
53+ if len(tv):
54+ for k in ("level_t", "slope_t"):
55+ v = float(tv[k].iloc[0]); out[f"{k}_true"] = v
56+ out[f"{k}_placebo_p"] = float((np.abs(df[k]) >= abs(v)).mean())
57+ out[f"{k}_rank"] = int((np.abs(df[k]) >= abs(v)).sum())
58+ return out, df
59+
60+
61+def bai_perron(y, dates=None, max_breaks=3, h=None, trend=True):
62+ """Global minimiser of SSR with m breaks (Bai & Perron 2003 dynamic programming) for a segment-wise linear trend
63+ (or mean) model; model selection by BIC and by the LWZ criterion. Returns dict with breaks (indices and dates)."""
64+ y = np.asarray(y, dtype=float); n = len(y); h = h or max(8, int(0.15 * n))
65+ t = np.arange(n, dtype=float)
66+ def ssr(i, j): # segment [i, j)
67+ yy = y[i:j]
68+ if trend:
69+ X = np.column_stack([np.ones(j - i), t[i:j]])
70+ beta, *_ = np.linalg.lstsq(X, yy, rcond=None); r = yy - X @ beta
71+ else:
72+ r = yy - yy.mean()
73+ return float(r @ r)
74+ S = np.full((n + 1, n + 1), np.inf)
75+ for i in range(n):
76+ for j in range(i + h, n + 1):
77+ S[i, j] = ssr(i, j)
78+ best = {}
79+ k_params = 2 if trend else 1
80+ # DP: C[m][j] = min SSR of first j obs with m breaks
81+ C = {0: S[0, :].copy()}; B = {0: {j: [] for j in range(n + 1)}}
82+ for m in range(1, max_breaks + 1):
83+ C[m] = np.full(n + 1, np.inf); B[m] = {}
84+ for j in range((m + 1) * h, n + 1):
85+ cands = [(C[m - 1][i] + S[i, j], i) for i in range(m * h, j - h + 1) if np.isfinite(C[m - 1][i]) and np.isfinite(S[i, j])]
86+ if cands:
87+ v, i = min(cands); C[m][j] = v; B[m][j] = B[m - 1][i] + [i]
88+ for m in range(0, max_breaks + 1):
89+ tot = C[m][n]
90+ if not np.isfinite(tot):
91+ continue
92+ p = (m + 1) * k_params + m
93+ bic = n * np.log(tot / n) + p * np.log(n)
94+ lwz = n * np.log(tot / (n - p)) + p * 0.299 * np.log(n) ** 2.1
95+ best[m] = {"ssr": tot, "bic": bic, "lwz": lwz, "breaks": B[m].get(n, [])}
96+ m_bic = min(best, key=lambda m: best[m]["bic"]); m_lwz = min(best, key=lambda m: best[m]["lwz"])
97+ res = {"by_m": best, "m_bic": m_bic, "m_lwz": m_lwz, "breaks_bic": best[m_bic]["breaks"], "h": h}
98+ if dates is not None:
99+ d = pd.DatetimeIndex(dates); res["break_dates_bic"] = [str(d[i].date()) for i in best[m_bic]["breaks"]]
100+ res["break_dates_lwz"] = [str(d[i].date()) for i in best[m_lwz]["breaks"]]
101+ return res
102+
103+
104+def pelt(y, pen=None, min_size=6, model="l2"):
105+ import ruptures as rpt
106+ y = np.asarray(y, dtype=float); n = len(y)
107+ pen = pen if pen is not None else 2 * np.log(n) * np.var(y) # BIC-type penalty for l2 cost
108+ algo = rpt.Pelt(model=model, min_size=min_size, jump=1).fit(y.reshape(-1, 1))
109+ return algo.predict(pen=pen)[:-1]
110+
111+
112+def bocpd(y, hazard=1 / 60.0, mu0=None, kappa0=1.0, alpha0=1.0, beta0=None):
113+ """Adams & MacKay (2007) with Normal–Inverse-Gamma conjugacy (Student-t predictive). Returns (R, cp_prob, map_rl):
114+ R[t, r] is the run-length posterior after observing y[:t]; with a constant hazard the mass on r = 0 equals the hazard
115+ by construction, so the detection statistics are cp_prob[t] = P(r_t <= 3 | y[:t+1]) (a changepoint in the last
116+ months) and map_rl[t] = argmax_r R[t+1, r] (the MAP run length, which resets after a change)."""
117+ from scipy.stats import t as student
118+ y = np.asarray(y, dtype=float); n = len(y)
119+ mu0 = y[:6].mean() if mu0 is None else mu0; beta0 = y[:6].var() + 1e-6 if beta0 is None else beta0
120+ R = np.zeros((n + 1, n + 1)); R[0, 0] = 1.0
121+ mu = np.array([mu0]); kappa = np.array([kappa0]); alpha = np.array([alpha0]); beta = np.array([beta0])
122+ cp = np.zeros(n); map_rl = np.zeros(n, dtype=int)
123+ for t_ in range(n):
124+ x = y[t_]
125+ df = 2 * alpha; scale = np.sqrt(beta * (kappa + 1) / (alpha * kappa))
126+ pred = student.pdf(x, df, loc=mu, scale=scale)
127+ growth = R[t_, : t_ + 1] * pred * (1 - hazard)
128+ change = (R[t_, : t_ + 1] * pred * hazard).sum()
129+ R[t_ + 1, 1 : t_ + 2] = growth; R[t_ + 1, 0] = change
130+ R[t_ + 1] /= R[t_ + 1].sum() + 1e-300
131+ cp[t_] = R[t_ + 1, :4].sum(); map_rl[t_] = int(np.argmax(R[t_ + 1, : t_ + 2]))
132+ mu_new = (kappa * mu + x) / (kappa + 1); kappa_new = kappa + 1; alpha_new = alpha + 0.5
133+ beta_new = beta + kappa * (x - mu) ** 2 / (2 * (kappa + 1))
134+ mu = np.concatenate([[mu0], mu_new]); kappa = np.concatenate([[kappa0], kappa_new]); alpha = np.concatenate([[alpha0], alpha_new]); beta = np.concatenate([[beta0], beta_new])
135+ return R, cp, map_rl
added embeddings/bench_models.py +21 −0
@@ -0,0 +1,21 @@
1+#!/usr/bin/env python3
2+"""Benchmark candidate scientific embedding models on Apple Silicon (MPS): throughput and a quick sanity check.
3+Models: allenai/specter2_base (SPECTER2 base, no adapter), malteos/scincl, BAAI/bge-base-en-v1.5, sentence-transformers/all-MiniLM-L6-v2."""
4+import sys, time, json, torch, duckdb
5+from sentence_transformers import SentenceTransformer
6+R = sys.argv[1]; N = int(sys.argv[2]) if len(sys.argv) > 2 else 2000
7+con = duckdb.connect(f"{R}/data/processed/wp44.duckdb", read_only=True)
8+rows = con.execute(f"SELECT w.title, a.abstract FROM oa_works w JOIN oa_abstracts a ON a.work_id=w.id WHERE w.publication_year=2024 LIMIT {N}").fetchall()
9+texts = [f"{t}. {ab[:1500]}" for t, ab in rows]
10+dev = "mps" if torch.backends.mps.is_available() else "cpu"; print("device", dev, "texts", len(texts), flush=True)
11+res = {}
12+for m in ["allenai/specter2_base", "malteos/scincl", "BAAI/bge-base-en-v1.5", "sentence-transformers/all-MiniLM-L6-v2"]:
13+ try:
14+ model = SentenceTransformer(m, device=dev); model.max_seq_length = 256
15+ _ = model.encode(texts[:64], batch_size=64)
16+ t0 = time.time(); E = model.encode(texts, batch_size=128, convert_to_numpy=True, normalize_embeddings=True); dt_ = time.time() - t0
17+ res[m] = {"dim": int(E.shape[1]), "texts_per_s": round(len(texts) / dt_, 1), "seconds": round(dt_, 1)}
18+ print(m, res[m], flush=True)
19+ except Exception as e:
20+ res[m] = {"error": str(e)[:300]}; print(m, "ERROR", str(e)[:200], flush=True)
21+json.dump(res, open(f"{R}/data/metadata/embedding_benchmark.json", "w"), indent=1)
added embeddings/embed_corpus.py +63 −0
@@ -0,0 +1,63 @@
1+#!/usr/bin/env python3
2+"""WP44 — embed title + abstract of corpus works with a scientific sentence-embedding model, sharded for multi-node runs.
3+
4+ --prepare : DuckDB exports data/processed/embeddings/input/shard_NNN.parquet (work_id, text) for eligible works with an
5+ abstract (or title only when --title-only-fallback), N shards of --shard-rows rows.
6+ --run : encode shards [--shard-from, --shard-to) with --model on MPS/CPU → data/processed/embeddings/<model_tag>/
7+ shard_NNN.npy (float16, L2-normalised) + shard_NNN.ids.npy. Resumable (existing shards skipped).
8+Provenance: data/processed/embeddings/<model_tag>/manifest.json (model, revision, max_seq_length, host, timings).
9+"""
10+import argparse, glob, json, os, socket, time, datetime as dt
11+import numpy as np
12+
13+
14+def prepare(root, shard_rows, title_only_fallback):
15+ import duckdb
16+ con = duckdb.connect(os.path.join(root, "data/processed/wp44.duckdb"), read_only=True); con.execute("SET threads TO 16;")
17+ out = os.path.join(root, "data/processed/embeddings/input"); os.makedirs(out, exist_ok=True)
18+ fb = " OR TRUE" if title_only_fallback else ""
19+ con.execute(f"""CREATE OR REPLACE TEMP TABLE emb_in AS
20+ SELECT w.id AS work_id, w.title || '. ' || coalesce(substr(a.abstract, 1, 2000), '') AS text
21+ FROM corpus_works w LEFT JOIN oa_abstracts a ON a.work_id = w.id
22+ WHERE w.eligible AND (a.abstract IS NOT NULL{fb}) ORDER BY w.id""")
23+ n = con.execute("SELECT count(*) FROM emb_in").fetchone()[0]
24+ con.execute(f"COPY (SELECT *, (row_number() OVER (ORDER BY work_id) - 1) // {shard_rows} AS shard FROM emb_in) TO '{out}' (FORMAT PARQUET, PARTITION_BY (shard), OVERWRITE_OR_IGNORE)")
25+ print("prepared", n, "texts in", len(glob.glob(out + "/shard=*")), "shards")
26+
27+
28+def run(root, model_name, shard_from, shard_to, batch, max_len):
29+ import torch, pyarrow.parquet as pq
30+ from sentence_transformers import SentenceTransformer
31+ tag = model_name.replace("/", "__"); out = os.path.join(root, "data/processed/embeddings", tag); os.makedirs(out, exist_ok=True)
32+ dev = "mps" if torch.backends.mps.is_available() else "cpu"
33+ model = SentenceTransformer(model_name, device=dev); model.max_seq_length = max_len
34+ shards = sorted(glob.glob(os.path.join(root, "data/processed/embeddings/input", "shard=*", "*.parquet")))
35+ man_p = os.path.join(out, f"manifest_{socket.gethostname()}.json"); man = json.load(open(man_p)) if os.path.exists(man_p) else {"model": model_name, "max_seq_length": max_len, "device": dev, "host": socket.gethostname(), "shards": {}}
36+ for p in shards:
37+ k = int(os.path.basename(os.path.dirname(p)).split("=")[1])
38+ if not (shard_from <= k < shard_to):
39+ continue
40+ o = os.path.join(out, f"shard_{k:04d}.npy")
41+ if os.path.exists(o):
42+ continue
43+ t = pq.read_table(p).to_pydict(); t0 = time.time()
44+ E = model.encode(t["text"], batch_size=batch, convert_to_numpy=True, normalize_embeddings=True, show_progress_bar=False).astype(np.float16)
45+ np.save(o + ".part.npy", E); os.replace(o + ".part.npy", o); np.save(o.replace(".npy", ".ids.npy"), np.asarray(t["work_id"], dtype=np.int64))
46+ man["shards"][str(k)] = {"n": len(t["text"]), "seconds": round(time.time() - t0, 1), "when": dt.datetime.now(dt.UTC).isoformat()}
47+ json.dump(man, open(man_p, "w"), indent=1)
48+ print(f"shard {k}: {len(t['text'])} texts in {time.time()-t0:.0f}s ({len(t['text'])/(time.time()-t0):.0f}/s)", flush=True)
49+
50+
51+def main():
52+ ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); ap.add_argument("--prepare", action="store_true"); ap.add_argument("--run", action="store_true")
53+ ap.add_argument("--model", default="allenai/specter2_base"); ap.add_argument("--shard-rows", type=int, default=50000); ap.add_argument("--shard-from", type=int, default=0)
54+ ap.add_argument("--shard-to", type=int, default=10**6); ap.add_argument("--batch", type=int, default=128); ap.add_argument("--max-len", type=int, default=256); ap.add_argument("--title-only-fallback", action="store_true")
55+ a = ap.parse_args()
56+ if a.prepare:
57+ prepare(a.root, a.shard_rows, a.title_only_fallback)
58+ if a.run:
59+ run(a.root, a.model, a.shard_from, a.shard_to, a.batch, a.max_len)
60+
61+
62+if __name__ == "__main__":
63+ main()
added ingestion/compact_tables.py +20 −0
@@ -0,0 +1,20 @@
1+#!/usr/bin/env python3
2+"""Compact the per-part Parquet tables (1,898 files per table) into one Parquet file per table
3+(data/processed/compact/<table>.parquet, zstd, ~1M-row row groups) so that downstream DuckDB work does not
4+hit the macOS open-file limit. Also compacts the slim table of all OpenAlex works."""
5+import duckdb, os, sys, time
6+R = sys.argv[1]; suffix = sys.argv[2] if len(sys.argv) > 2 else "" # e.g. "_p2" for the second snapshot pass
7+tables_dir = os.path.join(R, "data/processed/tables" + suffix); out = os.path.join(R, "data/processed/compact"); os.makedirs(out, exist_ok=True)
8+con = duckdb.connect(); con.execute("SET threads TO 24; SET memory_limit='70GB'; SET preserve_insertion_order=false;")
9+jobs = [(t + suffix, f"{tables_dir}/{t}/*.parquet") for t in ["works", "authorships", "work_topics", "work_keywords", "references", "abstracts", "locations", "grants", "counts_by_year"]]
10+if not suffix:
11+ jobs.append(("slim_all_works", f"{R}/data/raw/openalex/works/slim/*.parquet"))
12+for t, src in jobs:
13+ dst = f"{out}/{t}.parquet"
14+ if os.path.exists(dst):
15+ print("skip", t); continue
16+ t0 = time.time()
17+ con.execute(f"COPY (SELECT * FROM read_parquet('{src}')) TO '{dst}.part' (FORMAT PARQUET, COMPRESSION ZSTD, ROW_GROUP_SIZE 1000000)")
18+ os.replace(dst + ".part", dst)
19+ n = con.execute(f"SELECT count(*) FROM read_parquet('{dst}')").fetchone()[0]
20+ print(f"{t}: {n:,} rows, {os.path.getsize(dst)/1e9:.2f} GB, {time.time()-t0:.0f}s", flush=True)
modified ingestion/link_sources.py +10 −7
@@ -17,8 +17,11 @@ def main():
17 T = os.path.join(R, "data/processed/tables"); t0 = time.time(); rep = {"created": dt.datetime.now(dt.UTC).isoformat()} 17 T = os.path.join(R, "data/processed/tables"); t0 = time.time(); rep = {"created": dt.datetime.now(dt.UTC).isoformat()}
18 def log(m): print(f"[{time.time()-t0:7.0f}s] {m}", flush=True) 18 def log(m): print(f"[{time.time()-t0:7.0f}s] {m}", flush=True)
19 for t in ["works", "authorships", "work_topics", "work_keywords", "references", "abstracts", "locations", "grants", "counts_by_year"]: 19 for t in ["works", "authorships", "work_topics", "work_keywords", "references", "abstracts", "locations", "grants", "counts_by_year"]:
20− con.execute(f"CREATE OR REPLACE VIEW oa_{t} AS SELECT * FROM read_parquet('{T}/{t}/*.parquet')") 20+ srcs = [f"{R}/data/processed/compact/{t}.parquet"]
21− con.execute(f"CREATE OR REPLACE VIEW oa_slim AS SELECT * FROM read_parquet('{R}/data/raw/openalex/works/slim/*.parquet')") 21+ if os.path.exists(f"{R}/data/processed/compact/{t}_p2.parquet"):
22+ srcs.append(f"{R}/data/processed/compact/{t}_p2.parquet")
23+ con.execute(f"CREATE OR REPLACE VIEW oa_{t} AS SELECT * FROM read_parquet({srcs!r}, union_by_name=true)")
24+ con.execute(f"CREATE OR REPLACE VIEW oa_slim AS SELECT * FROM read_parquet('{R}/data/processed/compact/slim_all_works.parquet')")
22 con.execute(f"CREATE OR REPLACE VIEW arxiv_raw AS SELECT * FROM read_parquet('{R}/data/raw/arxiv/arxiv_cs.parquet')") 25 con.execute(f"CREATE OR REPLACE VIEW arxiv_raw AS SELECT * FROM read_parquet('{R}/data/raw/arxiv/arxiv_cs.parquet')")
23 con.execute(f"CREATE OR REPLACE VIEW acl_raw AS SELECT * FROM read_parquet('{R}/data/raw/acl/acl_anthology.parquet')") 26 con.execute(f"CREATE OR REPLACE VIEW acl_raw AS SELECT * FROM read_parquet('{R}/data/raw/acl/acl_anthology.parquet')")
24 con.execute(f"CREATE OR REPLACE VIEW dblp_raw AS SELECT * FROM read_parquet('{R}/data/raw/dblp/dblp_publications.parquet')") 27 con.execute(f"CREATE OR REPLACE VIEW dblp_raw AS SELECT * FROM read_parquet('{R}/data/raw/dblp/dblp_publications.parquet')")
@@ -49,9 +52,9 @@ def main():
49 WHERE d.arxiv_id IS NOT NULL AND o.arxiv_id IS NOT NULL AND d.dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa) 52 WHERE d.arxiv_id IS NOT NULL AND o.arxiv_id IS NOT NULL AND d.dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa)
50 AND o.oa_id NOT IN (SELECT oa_id FROM link_dblp_oa)""") 53 AND o.oa_id NOT IN (SELECT oa_id FROM link_dblp_oa)""")
51 # title+year: unique normalised titles on both sides, length >= 25, |year diff| <= 1 54 # title+year: unique normalised titles on both sides, length >= 25, |year diff| <= 1
52− con.execute("""CREATE OR REPLACE TEMP TABLE d_un AS SELECT ntitle, any_value(dblp_key) dblp_key, any_value(year) year FROM dblp_keys 55+ con.execute("""CREATE OR REPLACE TEMP TABLE d_un AS SELECT ntitle, any_value(dblp_key) AS dblp_key, any_value(year) AS year FROM dblp_keys
53 WHERE tlen >= 25 AND dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa) GROUP BY ntitle HAVING count(*) = 1""") 56 WHERE tlen >= 25 AND dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa) GROUP BY ntitle HAVING count(*) = 1""")
54− con.execute("""CREATE OR REPLACE TEMP TABLE o_un AS SELECT ntitle, any_value(oa_id) oa_id, any_value(year) year FROM oa_keys 57+ con.execute("""CREATE OR REPLACE TEMP TABLE o_un AS SELECT ntitle, any_value(oa_id) AS oa_id, any_value(year) AS year FROM oa_keys
55 WHERE tlen >= 25 AND oa_id NOT IN (SELECT oa_id FROM link_dblp_oa) GROUP BY ntitle HAVING count(*) = 1""") 58 WHERE tlen >= 25 AND oa_id NOT IN (SELECT oa_id FROM link_dblp_oa) GROUP BY ntitle HAVING count(*) = 1""")
56 con.execute("""INSERT INTO link_dblp_oa SELECT d.dblp_key, o.oa_id, 'title_year' FROM d_un d JOIN o_un o ON d.ntitle = o.ntitle 59 con.execute("""INSERT INTO link_dblp_oa SELECT d.dblp_key, o.oa_id, 'title_year' FROM d_un d JOIN o_un o ON d.ntitle = o.ntitle
57 WHERE abs(d.year - o.year) <= 1""") 60 WHERE abs(d.year - o.year) <= 1""")
@@ -61,9 +64,9 @@ def main():
61 SELECT x.arxiv_id, o.oa_id, 'arxiv_id' AS method FROM arxiv_keys x JOIN oa_keys o ON x.arxiv_id = o.arxiv_id""") 64 SELECT x.arxiv_id, o.oa_id, 'arxiv_id' AS method FROM arxiv_keys x JOIN oa_keys o ON x.arxiv_id = o.arxiv_id""")
62 con.execute("""INSERT INTO link_arxiv_oa SELECT x.arxiv_id, o.oa_id, 'doi' FROM arxiv_keys x JOIN oa_keys o ON x.doi = o.doi 65 con.execute("""INSERT INTO link_arxiv_oa SELECT x.arxiv_id, o.oa_id, 'doi' FROM arxiv_keys x JOIN oa_keys o ON x.doi = o.doi
63 WHERE x.doi IS NOT NULL AND x.arxiv_id NOT IN (SELECT arxiv_id FROM link_arxiv_oa)""") 66 WHERE x.doi IS NOT NULL AND x.arxiv_id NOT IN (SELECT arxiv_id FROM link_arxiv_oa)""")
64− con.execute("""CREATE OR REPLACE TEMP TABLE x_un AS SELECT ntitle, any_value(arxiv_id) arxiv_id, any_value(created_year) year FROM arxiv_keys 67+ con.execute("""CREATE OR REPLACE TEMP TABLE x_un AS SELECT ntitle, any_value(arxiv_id) AS arxiv_id, any_value(created_year) AS year FROM arxiv_keys
65 WHERE length(ntitle) >= 25 AND arxiv_id NOT IN (SELECT arxiv_id FROM link_arxiv_oa) GROUP BY ntitle HAVING count(*) = 1""") 68 WHERE length(ntitle) >= 25 AND arxiv_id NOT IN (SELECT arxiv_id FROM link_arxiv_oa) GROUP BY ntitle HAVING count(*) = 1""")
66− con.execute("""CREATE OR REPLACE TEMP TABLE o_un2 AS SELECT ntitle, any_value(oa_id) oa_id, any_value(year) year FROM oa_keys 69+ con.execute("""CREATE OR REPLACE TEMP TABLE o_un2 AS SELECT ntitle, any_value(oa_id) AS oa_id, any_value(year) AS year FROM oa_keys
67 WHERE tlen >= 25 GROUP BY ntitle HAVING count(*) = 1""") 70 WHERE tlen >= 25 GROUP BY ntitle HAVING count(*) = 1""")
68 con.execute("""INSERT INTO link_arxiv_oa SELECT x.arxiv_id, o.oa_id, 'title_year' FROM x_un x JOIN o_un2 o ON x.ntitle = o.ntitle 71 con.execute("""INSERT INTO link_arxiv_oa SELECT x.arxiv_id, o.oa_id, 'title_year' FROM x_un x JOIN o_un2 o ON x.ntitle = o.ntitle
69 WHERE abs(x.year - o.year) <= 2 AND o.oa_id NOT IN (SELECT oa_id FROM link_arxiv_oa)""") 72 WHERE abs(x.year - o.year) <= 2 AND o.oa_id NOT IN (SELECT oa_id FROM link_arxiv_oa)""")
@@ -76,7 +79,7 @@ def main():
76 WHERE landing_page_url LIKE '%aclanthology.org/%') l ON l.aid = c.acl_id OR l.aid = c.acl_id || '/' 79 WHERE landing_page_url LIKE '%aclanthology.org/%') l ON l.aid = c.acl_id OR l.aid = c.acl_id || '/'
77 WHERE c.acl_id NOT IN (SELECT acl_id FROM link_acl_oa)""") 80 WHERE c.acl_id NOT IN (SELECT acl_id FROM link_acl_oa)""")
78 con.execute("""INSERT INTO link_acl_oa SELECT c.acl_id, o.oa_id, 'title_year' FROM 81 con.execute("""INSERT INTO link_acl_oa SELECT c.acl_id, o.oa_id, 'title_year' FROM
79− (SELECT ntitle, any_value(acl_id) acl_id, any_value(year) year FROM acl_keys WHERE length(ntitle) >= 25 AND acl_id NOT IN (SELECT acl_id FROM link_acl_oa) GROUP BY ntitle HAVING count(*)=1) c 82+ (SELECT ntitle, any_value(acl_id) AS acl_id, any_value(year) AS year FROM acl_keys WHERE length(ntitle) >= 25 AND acl_id NOT IN (SELECT acl_id FROM link_acl_oa) GROUP BY ntitle HAVING count(*)=1) c
80 JOIN o_un2 o ON c.ntitle = o.ntitle WHERE abs(c.year - o.year) <= 1 AND o.oa_id NOT IN (SELECT oa_id FROM link_acl_oa)""") 83 JOIN o_un2 o ON c.ntitle = o.ntitle WHERE abs(c.year - o.year) <= 1 AND o.oa_id NOT IN (SELECT oa_id FROM link_acl_oa)""")
81 log("acl linked") 84 log("acl linked")
82 # --- report --- 85 # --- report ---
modified ingestion/openalex_snapshot.py +35 −7
@@ -39,6 +39,7 @@ import pyarrow.parquet as pq
39 import zstandard as zstd 39 import zstandard as zstd
40 40
41 S3_HTTP = "https://openalex.s3.amazonaws.com/" 41 S3_HTTP = "https://openalex.s3.amazonaws.com/"
42+ID_SET = None # optional: set of OpenAlex work ids (ints) to extract instead of the CS filter (second pass)
42 CS_FIELD = "https://openalex.org/fields/17" 43 CS_FIELD = "https://openalex.org/fields/17"
43 YEAR_MIN, YEAR_MAX = 2018, 2026 44 YEAR_MIN, YEAR_MAX = 2018, 2026
44 45
@@ -116,9 +117,21 @@ def slim_row(w: dict) -> tuple:
116 ), cs_any 117 ), cs_any
117 118
118 119
120+def _load_ids(path):
121+ with open(path) as fh:
122+ next(fh)
123+ return {int(x.strip().rsplit("W", 1)[-1]) for x in fh if x.strip()}
124+
125+
126+def _init_worker(id_list):
127+ """Pool initializer: macOS uses the spawn start method, so module globals must be rebuilt in each worker."""
128+ global ID_SET
129+ ID_SET = _load_ids(id_list) if id_list else None
130+
131+
119 def process_file(args): 132 def process_file(args):
120 """Download, hash, parse and filter one part file. Returns a result dict.""" 133 """Download, hash, parse and filter one part file. Returns a result dict."""
121− url, expected_len, out_dir, keep_raw = args 134+ url, expected_len, out_dir, keep_raw, mode = args
122 rel = url.replace("s3://openalex/data/jsonl/works/", "") 135 rel = url.replace("s3://openalex/data/jsonl/works/", "")
123 name = rel.replace("/", "__") # updated_date=X__part_0000.gz 136 name = rel.replace("/", "__") # updated_date=X__part_0000.gz
124 http = url.replace("s3://openalex/", S3_HTTP) 137 http = url.replace("s3://openalex/", S3_HTTP)
@@ -164,6 +177,11 @@ def process_file(args):
164 n_bad += 1 177 n_bad += 1
165 continue 178 continue
166 n_all += 1 179 n_all += 1
180+ if mode == "idset":
181+ if _oa_int(w.get("id")) in ID_SET:
182+ cs_w.write(line if line.endswith(b"\n") else line + b"\n")
183+ n_cs += 1
184+ continue
167 row, cs_any = slim_row(w) 185 row, cs_any = slim_row(w)
168 for i, v in enumerate(row): 186 for i, v in enumerate(row):
169 cols[i].append(v) 187 cols[i].append(v)
@@ -172,10 +190,13 @@ def process_file(args):
172 cs_w.write(line if line.endswith(b"\n") else line + b"\n") 190 cs_w.write(line if line.endswith(b"\n") else line + b"\n")
173 n_cs += 1 191 n_cs += 1
174 os.replace(cs_tmp, cs_tmp[:-5]) 192 os.replace(cs_tmp, cs_tmp[:-5])
175− table = pa.Table.from_arrays([pa.array(c, type=f.type) for c, f in zip(cols, SLIM_SCHEMA)], schema=SLIM_SCHEMA) 193+ if mode == "idset":
176− slim_tmp = os.path.join(slim_dir, name[:-3] + ".parquet.part") 194+ cols = None
177− pq.write_table(table, slim_tmp, compression="zstd") 195+ table = None if cols is None else pa.Table.from_arrays([pa.array(c, type=f.type) for c, f in zip(cols, SLIM_SCHEMA)], schema=SLIM_SCHEMA)
178− os.replace(slim_tmp, slim_tmp[:-5]) 196+ if table is not None:
197+ slim_tmp = os.path.join(slim_dir, name[:-3] + ".parquet.part")
198+ pq.write_table(table, slim_tmp, compression="zstd")
199+ os.replace(slim_tmp, slim_tmp[:-5])
179 if not keep_raw: 200 if not keep_raw:
180 os.remove(raw_path) 201 os.remove(raw_path)
181 res.update(status="done", sha256=sha, n_records=n_all, n_cs=n_cs, n_bad=n_bad, 202 res.update(status="done", sha256=sha, n_records=n_all, n_cs=n_cs, n_bad=n_bad,
@@ -193,7 +214,14 @@ def main():
193 ap.add_argument("--keep-raw", action="store_true") 214 ap.add_argument("--keep-raw", action="store_true")
194 ap.add_argument("--limit", type=int, default=0, help="process only N files (smoke test)") 215 ap.add_argument("--limit", type=int, default=0, help="process only N files (smoke test)")
195 ap.add_argument("--smallest-first", action="store_true") 216 ap.add_argument("--smallest-first", action="store_true")
217+ ap.add_argument("--id-list", default=None, help="CSV with a header and one OpenAlex work id per line: extract exactly these works (second pass)")
196 a = ap.parse_args() 218 a = ap.parse_args()
219+ global ID_SET
220+ mode = "cs"
221+ if a.id_list:
222+ ID_SET = _load_ids(a.id_list)
223+ mode = "idset"
224+ print(f"id-set mode: {len(ID_SET)} ids", flush=True)
197 os.makedirs(a.out, exist_ok=True) 225 os.makedirs(a.out, exist_ok=True)
198 man = json.load(open(a.manifest)) 226 man = json.load(open(a.manifest))
199 files = [(f["url"], f["meta"]["content_length"], f["meta"].get("record_count")) for f in man["files"]] 227 files = [(f["url"], f["meta"]["content_length"], f["meta"].get("record_count")) for f in man["files"]]
@@ -214,12 +242,12 @@ def main():
214 (url, url.replace("s3://openalex/data/jsonl/works/", "").replace("/", "__"), ln, rc)) 242 (url, url.replace("s3://openalex/data/jsonl/works/", "").replace("/", "__"), ln, rc))
215 db.commit() 243 db.commit()
216 done = {r[0] for r in db.execute("SELECT url FROM files WHERE status='done'")} 244 done = {r[0] for r in db.execute("SELECT url FROM files WHERE status='done'")}
217− todo = [(u, ln, a.out, a.keep_raw) for u, ln, _ in files if u not in done] 245+ todo = [(u, ln, a.out, a.keep_raw, mode) for u, ln, _ in files if u not in done]
218 total_bytes = sum(t[1] for t in todo) 246 total_bytes = sum(t[1] for t in todo)
219 print(f"[{dt.datetime.now():%H:%M:%S}] manifest {man.get('date')}: {len(files)} files, {len(done)} done, " 247 print(f"[{dt.datetime.now():%H:%M:%S}] manifest {man.get('date')}: {len(files)} files, {len(done)} done, "
220 f"{len(todo)} to do ({total_bytes/1e9:.1f} GB), workers={a.workers}", flush=True) 248 f"{len(todo)} to do ({total_bytes/1e9:.1f} GB), workers={a.workers}", flush=True)
221 t0 = time.time(); done_bytes = 0; n_ok = n_err = 0 249 t0 = time.time(); done_bytes = 0; n_ok = n_err = 0
222− with mp.Pool(a.workers, maxtasksperchild=50) as pool: 250+ with mp.Pool(a.workers, maxtasksperchild=50, initializer=_init_worker, initargs=(a.id_list,)) as pool:
223 for r in pool.imap_unordered(process_file, todo): 251 for r in pool.imap_unordered(process_file, todo):
224 now = dt.datetime.now(dt.UTC).isoformat() 252 now = dt.datetime.now(dt.UTC).isoformat()
225 db.execute("UPDATE files SET attempts=attempts+1 WHERE url=?", (r["url"],)) 253 db.execute("UPDATE files SET attempts=attempts+1 WHERE url=?", (r["url"],))
added ingestion/run_increment_daily.sh +8 −0
@@ -0,0 +1,8 @@
1+#!/bin/sh
2+# re-run the API increment once per day until all slices are done (public budget $0.10/day)
3+cd ~/wp44_uqo; . .venv/bin/activate
4+while ! grep -q "all slices done" logs/openalex_api_increment.log 2>/dev/null; do
5+ python ingestion/openalex_api_increment.py --out data/raw/openalex >> logs/openalex_api_increment.log 2>&1
6+ grep -q "all slices done" logs/openalex_api_increment.log && break
7+ sleep 21700
8+done
added knowledge_graph/01_citation_graph.py +41 −0
@@ -0,0 +1,41 @@
1+#!/usr/bin/env python3
2+"""WP44 — citation graph of the eligible corpus (Phase 3, structural layer).
3+
4+Builds, in DuckDB and Parquet (data/processed/graph/):
5+ cit_edges_internal.parquet (citing_id, cited_id, citing_year, cited_year) both ends eligible
6+ cit_edges_external.parquet (citing_id, cited_id, cited_field_id, cited_year, cited_type) cited work outside the corpus,
7+ resolved through the slim table of all 465 M OpenAlex works (field of the cited work)
8+ field_flows_yearly.csv citing year × cited primary field (OpenAlex field) counts — interdisciplinary inflows (RQ4)
9+ cited_age_yearly.csv distribution of citation age (citing year − cited year) by citing year
10+ reference_coverage_yearly.csv share of references resolvable (in OpenAlex), internal vs external
11+Category-level flows are produced by knowledge_graph/02_category_flows.py after classification."""
12+import argparse, os, duckdb
13+
14+
15+def main():
16+ ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
17+ G = os.path.join(R, "data/processed/graph"); out = os.path.join(R, "results/graph"); os.makedirs(G, exist_ok=True); os.makedirs(out, exist_ok=True)
18+ con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb")); con.execute("SET threads TO 24; SET memory_limit='70GB';")
19+ con.execute("CREATE OR REPLACE TEMP TABLE E AS SELECT id, publication_year AS y, type, primary_field_id FROM corpus_works WHERE eligible")
20+ con.execute("""CREATE OR REPLACE TABLE cit_edges_internal AS
21+ SELECT r.work_id AS citing_id, r.ref_id AS cited_id, a.y AS citing_year, b.y AS cited_year
22+ FROM oa_references r JOIN E a ON a.id = r.work_id JOIN E b ON b.id = r.ref_id""")
23+ con.execute("""CREATE OR REPLACE TABLE cit_edges_external AS
24+ SELECT r.work_id AS citing_id, r.ref_id AS cited_id, a.y AS citing_year, s.publication_year AS cited_year, s.primary_field_id AS cited_field_id, s.type AS cited_type, s.cs_any_topic AS cited_cs_any
25+ FROM oa_references r JOIN E a ON a.id = r.work_id LEFT JOIN E b ON b.id = r.ref_id
26+ JOIN oa_slim s ON s.id = r.ref_id WHERE b.id IS NULL""")
27+ for t in ("cit_edges_internal", "cit_edges_external"):
28+ con.execute(f"COPY {t} TO '{G}/{t}.parquet' (FORMAT PARQUET, COMPRESSION ZSTD)")
29+ con.execute(f"""COPY (SELECT citing_year, 'internal' AS kind, NULL AS cited_field_id, count(*) n FROM cit_edges_internal GROUP BY 1
30+ UNION ALL SELECT citing_year, 'external', cited_field_id, count(*) FROM cit_edges_external GROUP BY 1,3 ORDER BY 1,2,3) TO '{out}/field_flows_yearly.csv' (HEADER)""")
31+ con.execute(f"""COPY (SELECT citing_year, citing_year - cited_year AS age, count(*) n FROM (SELECT citing_year, cited_year FROM cit_edges_internal UNION ALL SELECT citing_year, cited_year FROM cit_edges_external)
32+ WHERE cited_year IS NOT NULL GROUP BY 1,2 ORDER BY 1,2) TO '{out}/cited_age_yearly.csv' (HEADER)""")
33+ con.execute(f"""COPY (SELECT e.y AS year, count(*) n_works, sum(n_refs) n_refs_declared,
34+ (SELECT count(*) FROM cit_edges_internal i WHERE i.citing_year = e.y) n_internal,
35+ (SELECT count(*) FROM cit_edges_external x WHERE x.citing_year = e.y) n_external
36+ FROM E e JOIN corpus_works c ON c.id = e.id GROUP BY 1 ORDER BY 1) TO '{out}/reference_coverage_yearly.csv' (HEADER)""")
37+ print(open(f"{out}/reference_coverage_yearly.csv").read())
38+
39+
40+if __name__ == "__main__":
41+ main()
added manuscript/appendices/appA_taxonomy.tex +1 −0
@@ -0,0 +1 @@
1+% appA_taxonomy — to be written (Phase 7)
added manuscript/appendices/appB_search_and_sources.tex +1 −0
@@ -0,0 +1 @@
1+% appB_search_and_sources — to be written (Phase 7)
added manuscript/appendices/appC_derivations.tex +1 −0
@@ -0,0 +1 @@
1+% appC_derivations — to be written (Phase 7)
added manuscript/appendices/appD_robustness.tex +1 −0
@@ -0,0 +1 @@
1+% appD_robustness — to be written (Phase 7)
added manuscript/appendices/appE_data_dictionary.tex +1 −0
@@ -0,0 +1 @@
1+% appE_data_dictionary — to be written (Phase 7)
added manuscript/appendices/appF_supplementary_tables.tex +1 −0
@@ -0,0 +1 @@
1+% appF_supplementary_tables — to be written (Phase 7)
added manuscript/appendices/appG_figure_documentation.tex +1 −0
@@ -0,0 +1 @@
1+% appG_figure_documentation — to be written (Phase 7)
added manuscript/appendices/appH_reproducibility.tex +1 −0
@@ -0,0 +1 @@
1+% appH_reproducibility — to be written (Phase 7)
added manuscript/main.tex +97 −0
@@ -0,0 +1,97 @@
1+% ============================================================================
2+% UQO Working Paper 44 — The Post-ChatGPT Computer Science Revolution
3+% Preamble follows the UQO working-paper template (_modele-working-paper/main.tex)
4+% ============================================================================
5+\documentclass[12pt,letterpaper]{article}
6+\usepackage[utf8]{inputenc}
7+\usepackage[T1]{fontenc}
8+\usepackage[english]{babel}
9+\usepackage[letterpaper,top=1in,bottom=1in,left=1in,right=1in]{geometry}
10+\usepackage{mathptmx}
11+\usepackage{microtype}
12+\usepackage{setspace}
13+\onehalfspacing
14+\usepackage{amsmath,amssymb,amsthm}
15+\usepackage{booktabs,tabularx,multirow,threeparttable,longtable,array}
16+\usepackage{siunitx}
17+\sisetup{input-symbols = {()}, table-format = -1.3, parse-numbers = false}
18+\usepackage{graphicx,subcaption,float}
19+\usepackage[dvipsnames]{xcolor}
20+\usepackage[colorlinks=true,linkcolor=NavyBlue,citecolor=NavyBlue,urlcolor=NavyBlue,breaklinks=true]{hyperref}
21+\usepackage[authoryear,round,semicolon]{natbib}
22+\bibliographystyle{aer}
23+\usepackage[font=small,labelfont=bf,labelsep=period,justification=justified,singlelinecheck=false]{caption}
24+\usepackage{fancyhdr}
25+\pagestyle{fancy}\fancyhf{}\renewcommand{\headrulewidth}{0pt}\fancyfoot[C]{\thepage}
26+\usepackage{titlesec}
27+\titleformat{\section}{\large\bfseries}{\thesection.}{0.5em}{}
28+\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection.}{0.5em}{}
29+\titleformat{\subsubsection}{\normalsize\itshape}{\thesubsubsection.}{0.5em}{}
30+\usepackage[toc,page]{appendix}
31+\usepackage{enumitem}
32+\setlist{nosep,leftmargin=*}
33+\newtheorem{definition}{Definition}
34+\newtheorem{proposition}{Proposition}
35+\newtheorem{hypothesis}{Hypothesis}
36+\newcommand{\E}{\mathbb{E}}
37+\newcommand{\R}{\mathbb{R}}
38+\newcommand{\pending}[1]{\textcolor{BrickRed}{[pending human review: #1]}}
39+\graphicspath{{figures/}}
40+
41+% ---- metadata ----
42+\newcommand{\WPnumber}{44}
43+\newcommand{\WPtitle}{The Post-ChatGPT Computer Science Revolution}
44+\newcommand{\WPsubtitle}{A Comprehensive Systematic Mapping, Bibliometric Analysis, and Knowledge Graph of Computer Science Research, 2022--2026}
45+\newcommand{\WPdate}{November 2026}
46+\newcommand{\WPversion}{0.1}
47+\input{tables/numbers} % machine-generated macros (counts, estimates) — never edited by hand
48+\newcommand{\WPabstract}{\input{sections/abstract}}
49+\newcommand{\WPkeywords}{systematic mapping, bibliometrics, computer science, large language models, structural change, knowledge graph, emerging topics, OpenAlex}
50+\newcommand{\WPjel}{C55, C88, O33, L86, I23}
51+\newcommand{\WPauthor}{Simon-Pierre Boucher}
52+\newcommand{\WPaffiliation}{Département des sciences administratives\\Université du Québec en Outaouais}
53+\newcommand{\WPemail}{simon-pierre.boucher@uqo.ca}
54+\newcommand{\WPaddress}{Gatineau -- Pavillon Alexandre-Taché\\283, boulevard Alexandre-Taché\\Gatineau, Québec, Canada J9A 1L8}
55+
56+\begin{document}
57+\input{sections/titlepage}
58+\tableofcontents
59+\clearpage
60+
61+\part{Introduction and Scientific Motivation}
62+\input{sections/part01_introduction}
63+\part{The Evolution of Computer Science}
64+\input{sections/part02_evolution}
65+\part{Data and Systematic Review Protocol}
66+\input{sections/part03_data}
67+\part{Computational Methodology}
68+\input{sections/part04_methodology}
69+\part{The Artificial Intelligence Transformation}
70+\input{sections/part05_ai}
71+\part{Systems and Computing Infrastructure}
72+\input{sections/part06_systems}
73+\part{Theory, Algorithms, and Security}
74+\input{sections/part07_theory_security}
75+\part{Scientific Knowledge Networks}
76+\input{sections/part08_networks}
77+\part{Empirical Results}
78+\input{sections/part09_results}
79+\part{The Next Research Frontiers}
80+\input{sections/part10_frontiers}
81+\part{Discussion and Conclusion}
82+\input{sections/part11_discussion}
83+
84+\clearpage
85+\bibliography{references}
86+
87+\begin{appendices}
88+\input{appendices/appA_taxonomy}
89+\input{appendices/appB_search_and_sources}
90+\input{appendices/appC_derivations}
91+\input{appendices/appD_robustness}
92+\input{appendices/appE_data_dictionary}
93+\input{appendices/appF_supplementary_tables}
94+\input{appendices/appG_figure_documentation}
95+\input{appendices/appH_reproducibility}
96+\end{appendices}
97+\end{document}
added manuscript/sections/abstract.tex +1 −0
@@ -0,0 +1 @@
1+% abstract — to be written (Phase 7)
added manuscript/sections/part01_introduction.tex +1 −0
@@ -0,0 +1 @@
1+% part01_introduction — to be written (Phase 7)
added manuscript/sections/part02_evolution.tex +1 −0
@@ -0,0 +1 @@
1+% part02_evolution — to be written (Phase 7)
added manuscript/sections/part03_data.tex +1 −0
@@ -0,0 +1 @@
1+% part03_data — to be written (Phase 7)
added manuscript/sections/part04_methodology.tex +1 −0
@@ -0,0 +1 @@
1+% part04_methodology — to be written (Phase 7)
added manuscript/sections/part05_ai.tex +1 −0
@@ -0,0 +1 @@
1+% part05_ai — to be written (Phase 7)
added manuscript/sections/part06_systems.tex +1 −0
@@ -0,0 +1 @@
1+% part06_systems — to be written (Phase 7)
added manuscript/sections/part07_theory_security.tex +1 −0
@@ -0,0 +1 @@
1+% part07_theory_security — to be written (Phase 7)
added manuscript/sections/part08_networks.tex +1 −0
@@ -0,0 +1 @@
1+% part08_networks — to be written (Phase 7)
added manuscript/sections/part09_results.tex +1 −0
@@ -0,0 +1 @@
1+% part09_results — to be written (Phase 7)
added manuscript/sections/part10_frontiers.tex +1 −0
@@ -0,0 +1 @@
1+% part10_frontiers — to be written (Phase 7)
added manuscript/sections/part11_discussion.tex +1 −0
@@ -0,0 +1 @@
1+% part11_discussion — to be written (Phase 7)
added manuscript/tables/numbers.tex +1 −0
@@ -0,0 +1 @@
1+% machine-generated numbers (tables/make_numbers.py) — placeholder until results exist
modified protocol/amendments.csv +2 −0
@@ -1,2 +1,4 @@
1 id,date,phase,change,reason,approved_by 1 id,date,phase,change,reason,approved_by
2 A-000,2026-11-02,0,"§14 of the mandate completed from §1–§13 (paste truncated)","terminal truncation of the pasted mandate",executing agent (pending PI) 2 A-000,2026-11-02,0,"§14 of the mandate completed from §1–§13 (paste truncated)","terminal truncation of the pasted mandate",executing agent (pending PI)
3+A-001,2026-11-03,1,"Eligibility type list corrected: OpenAlex (2026) types conference-paper and software-paper are research outputs and are kept; DBLP-linked records are kept whatever their OpenAlex type unless in a hard-exclusion list; XPAC records are kept when DBLP-linked (flag xpac_kept_dblp)","first run of define_corpus.py showed 2.5 M conference-paper records (1.09 M DBLP-linked) excluded by an outdated type list; DBLP is the CS gold standard",executing agent (pending PI)
4+A-002,2026-11-03,1,"Records whose primary source is a generic deposit platform (Zenodo, Figshare, IRDB, LA Referencia, Dialnet, DOAJ-as-source, OSTI) are excluded unless DBLP- or arXiv-linked (flag e7_generic_deposit); date_precision flag added (publication_date = YYYY-01-01 treated as year precision; monthly series use month-precision records only)","Zenodo alone contributes 138,533 eligible records in 2026 (20,141 in 2025; 1.3% with references) — deposits of slides, datasets and reports typed as articles; 18–28% of records carry a year-only date defaulting to 1 January",executing agent (pending PI)
added reports/HUMAN_REVIEW_CHECKPOINTS.md +12 −0
@@ -0,0 +1,12 @@
1+# Human-review checkpoints (protocol §P7) — items awaiting the principal investigator
2+
3+| Checkpoint | What to review | Where | Status |
4+|---|---|---|---|
5+| H-0 | Plan, protocol, decisions D1–D7 and amendments A-000–A-002 | `RESEARCH_PLAN.md` §10, `protocol/PROTOCOL.md`, `protocol/amendments.csv` | pending |
6+| H-1 | Record-linkage precision sample (500 DBLP↔OpenAlex pairs, 200 reserved for the PI) | `data/metadata/linkage_sample.csv` (to be generated by `systematic_review/linkage_sample.py`) | pending |
7+| H-2 | Taxonomy validation sample (2,000 works, two LLM raters; 200 for the PI) | `taxonomy/validation/` | pending |
8+| H-3 | Landmark list (≥ 1,000) and specialist reviews | `literature_synthesis/landmarks/` | pending |
9+| H-4 | Structural-change results and interpretation (no causal language) | `results/econometrics/` | pending |
10+| H-5 | Manuscript conclusions | `manuscript/` | pending |
11+
12+Conclusions that depend on an unsigned checkpoint are labelled "pending human review" in the manuscript.
added systematic_review/define_corpus.py +110 −0
@@ -0,0 +1,110 @@
1+#!/usr/bin/env python3
2+"""WP44 — Stages S2–S3: deduplicate (preprint ↔ published), apply eligibility, define the mapping corpus, write PRISMA counts.
3+
4+Inputs: DuckDB views/tables made by ingestion/link_sources.py (oa_* views, dblp_keys, arxiv_keys, link_* tables).
5+Outputs: table `corpus_works` (+ Parquet copy data/processed/corpus/corpus_works.parquet), `dblp_only_counts`,
6+data/metadata/prisma_counts.json. Rules: protocol/PROTOCOL.md §P2; decisions logged in RESEARCH_PLAN §10.
7+"""
8+import argparse, json, os, time, datetime as dt
9+import duckdb
10+
11+KEEP_TYPES = "('article','conference-paper','preprint','book-chapter','book','dissertation','report','review','software-paper')"
12+# hard exclusions even for DBLP-linked records (DBLP-linked records of any other type are kept: DBLP is the CS gold standard)
13+# generic deposit platforms (A-002): records whose primary source is one of these are kept only if DBLP- or arXiv-linked
14+GENERIC_DEPOSIT_SOURCES = "('Zenodo (CERN European Organization for Nuclear Research)','Figshare','Institutional Repositories DataBase (IRDB)','LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas)','Dialnet (Universidad de la Rioja)','DOAJ (DOAJ: Directory of Open Access Journals)','OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information)','Zenodo')"
15+HARD_EXCLUDE_TYPES = "('editorial','erratum','retraction','peer-review','paratext','conference-abstract','reference-entry','dataset','software','supplementary-materials','libguides','letter','grant','standard')"
16+WINDOW_START, BOUNDARY, WINDOW_END = "2018-01-01", "2022-11-30", "2026-11-02"
17+
18+
19+def main():
20+ ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
21+ con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb")); con.execute("SET threads TO 24; SET memory_limit='60GB';")
22+ t0 = time.time(); q = lambda s: con.execute(s).fetchall(); n = lambda s: q(s)[0][0]
23+ log = lambda m: print(f"[{time.time()-t0:6.0f}s] {m}", flush=True)
24+ # ---- S0/S1: base table with membership flags ----
25+ con.execute(f"""CREATE OR REPLACE TABLE cw0 AS
26+ SELECT w.*, regexp_replace(lower(w.title), '[^a-z0-9]+', '', 'g') AS ntitle,
27+ (w.id IN (SELECT oa_id FROM link_dblp_oa)) AS in_dblp,
28+ (w.id IN (SELECT oa_id FROM link_acl_oa)) AS in_acl,
29+ x.arxiv_id AS arxiv_id_linked, x.created AS arxiv_created, x.primary_category AS arxiv_primary,
30+ (x.primary_category LIKE 'cs.%') AS arxiv_cs_primary,
31+ (w.doi LIKE '10.48550/arxiv.%') AS is_arxiv_doi
32+ FROM oa_works w
33+ LEFT JOIN (SELECT l.oa_id, any_value(k.arxiv_id) arxiv_id, min(k.created) created, any_value(k.primary_category) primary_category
34+ FROM link_arxiv_oa l JOIN arxiv_keys k USING(arxiv_id) GROUP BY l.oa_id) x ON x.oa_id = w.id""")
35+ log(f"cw0 {n('SELECT count(*) FROM cw0'):,}")
36+ # ---- S2: preprint ↔ published duplicates (same normalised title, |year| <= 2, one preprint-like, one not) ----
37+ con.execute("""CREATE OR REPLACE TABLE dup_pairs AS
38+ WITH pre AS (SELECT id, ntitle, publication_year y, publication_date d FROM cw0 WHERE (type = 'preprint' OR is_arxiv_doi) AND length(ntitle) >= 25),
39+ pub AS (SELECT id, ntitle, publication_year y, publication_date d FROM cw0 WHERE type <> 'preprint' AND NOT is_arxiv_doi AND length(ntitle) >= 25)
40+ SELECT p.id AS preprint_id, b.id AS published_id, p.d AS preprint_date, b.d AS published_date,
41+ row_number() OVER (PARTITION BY b.id ORDER BY p.d, p.id) AS rn_pub, row_number() OVER (PARTITION BY p.id ORDER BY b.d, b.id) AS rn_pre
42+ FROM pre p JOIN pub b ON p.ntitle = b.ntitle AND abs(p.y - b.y) <= 2""")
43+ con.execute("DELETE FROM dup_pairs WHERE rn_pub > 1 OR rn_pre > 1")
44+ log(f"duplicate preprint→published pairs: {n('SELECT count(*) FROM dup_pairs'):,}")
45+ # ---- S3: eligibility flags ----
46+ con.execute(f"""CREATE OR REPLACE TABLE corpus_works AS
47+ SELECT c.*,
48+ dp.published_id AS duplicate_of,
49+ dp2.preprint_id AS preprint_work_id,
50+ least(c.publication_date, coalesce(dp2.preprint_date, c.publication_date), coalesce(c.arxiv_created, c.publication_date)) AS first_date,
51+ (c.publication_date IS NULL OR c.publication_date < '{WINDOW_START}' OR c.publication_date > '{WINDOW_END}') AS e1_outside_window,
52+ (c.type NOT IN {KEEP_TYPES} AND NOT (c.in_dblp AND c.type NOT IN {HARD_EXCLUDE_TYPES})) AS e2_type,
53+ (c.is_xpac AND NOT c.in_dblp) AS e3_xpac,
54+ (c.is_xpac AND c.in_dblp) AS xpac_kept_dblp,
55+ (c.source_name IN {GENERIC_DEPOSIT_SOURCES} AND NOT c.in_dblp AND c.arxiv_id_linked IS NULL) AS e7_generic_deposit,
56+ (CASE WHEN substr(c.publication_date, 6, 5) = '01-01' THEN 'year' ELSE 'month' END) AS date_precision,
57+ (c.title IS NULL OR length(trim(c.title)) = 0) AS e4_no_title,
58+ (dp.published_id IS NOT NULL) AS e5_duplicate,
59+ c.is_retracted AS e6_retracted,
60+ (c.in_dblp OR c.cs_primary OR (c.cs_any AND coalesce(c.arxiv_cs_primary, FALSE))) AS cs_core,
61+ (c.cs_any AND NOT (c.in_dblp OR c.cs_primary OR (c.cs_any AND coalesce(c.arxiv_cs_primary, FALSE)))) AS cs_penumbra
62+ FROM cw0 c
63+ LEFT JOIN dup_pairs dp ON dp.preprint_id = c.id
64+ LEFT JOIN dup_pairs dp2 ON dp2.published_id = c.id""")
65+ con.execute(f"""ALTER TABLE corpus_works ADD COLUMN eligible BOOLEAN; UPDATE corpus_works SET eligible =
66+ NOT e1_outside_window AND NOT e2_type AND NOT e3_xpac AND NOT e4_no_title AND NOT e5_duplicate AND NOT e7_generic_deposit AND cs_core""")
67+ con.execute(f"""ALTER TABLE corpus_works ADD COLUMN period VARCHAR; UPDATE corpus_works SET period = CASE
68+ WHEN first_date < '{WINDOW_START}' THEN 'pre' WHEN first_date < '{BOUNDARY}' THEN 'baseline' WHEN first_date <= '{WINDOW_END}' THEN 'post' ELSE 'future' END""")
69+ con.execute("ALTER TABLE corpus_works ADD COLUMN first_month VARCHAR; UPDATE corpus_works SET first_month = substr(first_date, 1, 7)")
70+ log("corpus_works built")
71+ # ---- DBLP-only records (no OpenAlex record after both passes) ----
72+ con.execute("""CREATE OR REPLACE TABLE dblp_only_counts AS
73+ SELECT year, rec_type, count(*) n, sum(CASE WHEN doi IS NOT NULL THEN 1 ELSE 0 END) with_doi FROM dblp_keys
74+ WHERE dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa) GROUP BY 1,2 ORDER BY 1,2""")
75+ # ---- PRISMA counts ----
76+ P = {"generated": dt.datetime.now(dt.UTC).isoformat(), "window": [WINDOW_START, WINDOW_END], "boundary": BOUNDARY}
77+ P["identified"] = {
78+ "openalex_cs_topic_2018_2026_pass1": n("SELECT count(*) FROM oa_keys"),
79+ "openalex_total_records_loaded": n("SELECT count(*) FROM cw0"),
80+ "dblp_2018_2026": n("SELECT count(*) FROM dblp_keys"), "arxiv_cs_2018_2026": n("SELECT count(*) FROM arxiv_keys WHERE created_year BETWEEN 2018 AND 2026"),
81+ "acl_2018_2026": n("SELECT count(*) FROM acl_keys WHERE year BETWEEN 2018 AND 2026")}
82+ P["linked"] = {"dblp_to_openalex": n("SELECT count(DISTINCT dblp_key) FROM link_dblp_oa"), "arxiv_to_openalex": n("SELECT count(DISTINCT arxiv_id) FROM link_arxiv_oa"),
83+ "acl_to_openalex": n("SELECT count(DISTINCT acl_id) FROM link_acl_oa"), "dblp_only_no_openalex_record": n("SELECT sum(n) FROM dblp_only_counts")}
84+ P["excluded"] = {k: n(f"SELECT count(*) FROM corpus_works WHERE {k}") for k in ["e1_outside_window", "e2_type", "e3_xpac", "e4_no_title", "e5_duplicate", "e7_generic_deposit", "e6_retracted", "xpac_kept_dblp"]}
85+ P["excluded_e2_by_type"] = dict(q("SELECT type, count(*) FROM corpus_works WHERE e2_type GROUP BY 1 ORDER BY 2 DESC"))
86+ P["excluded_sequential"] = {}
87+ cond = "TRUE"
88+ for k in ["e1_outside_window", "e2_type", "e3_xpac", "e4_no_title", "e5_duplicate", "e7_generic_deposit"]:
89+ P["excluded_sequential"][k] = n(f"SELECT count(*) FROM corpus_works WHERE {cond} AND {k}"); cond += f" AND NOT {k}"
90+ P["after_exclusions"] = n(f"SELECT count(*) FROM corpus_works WHERE {cond}")
91+ P["cs_core_eligible"] = n("SELECT count(*) FROM corpus_works WHERE eligible")
92+ P["cs_penumbra_eligible_otherwise"] = n(f"SELECT count(*) FROM corpus_works WHERE {cond} AND cs_penumbra")
93+ P["eligible_by_period"] = dict(q("SELECT period, count(*) FROM corpus_works WHERE eligible GROUP BY 1"))
94+ P["eligible_by_year"] = dict(q("SELECT publication_year, count(*) FROM corpus_works WHERE eligible GROUP BY 1 ORDER BY 1"))
95+ P["eligible_by_type"] = dict(q("SELECT type, count(*) FROM corpus_works WHERE eligible GROUP BY 1 ORDER BY 2 DESC"))
96+ P["eligible_membership"] = {"in_dblp": n("SELECT count(*) FROM corpus_works WHERE eligible AND in_dblp"), "cs_primary": n("SELECT count(*) FROM corpus_works WHERE eligible AND cs_primary"),
97+ "arxiv_linked": n("SELECT count(*) FROM corpus_works WHERE eligible AND arxiv_id_linked IS NOT NULL"), "in_acl": n("SELECT count(*) FROM corpus_works WHERE eligible AND in_acl"),
98+ "has_abstract": n("SELECT count(*) FROM corpus_works WHERE eligible AND has_abstract"), "has_references": n("SELECT count(*) FROM corpus_works WHERE eligible AND n_refs > 0"),
99+ "retracted_flagged": n("SELECT count(*) FROM corpus_works WHERE eligible AND e6_retracted")}
100+ P["eligible_date_precision"] = dict(q("SELECT date_precision, count(*) FROM corpus_works WHERE eligible GROUP BY 1"))
101+ P["strict_core_dblp_and_cs_primary"] = n("SELECT count(*) FROM corpus_works WHERE eligible AND in_dblp AND cs_primary")
102+ os.makedirs(os.path.join(R, "data/processed/corpus"), exist_ok=True); os.makedirs(os.path.join(R, "data/metadata"), exist_ok=True)
103+ con.execute(f"COPY corpus_works TO '{R}/data/processed/corpus/corpus_works.parquet' (FORMAT PARQUET, COMPRESSION ZSTD)")
104+ con.execute(f"COPY dblp_only_counts TO '{R}/data/metadata/dblp_only_counts.csv' (HEADER)")
105+ json.dump(P, open(os.path.join(R, "data/metadata/prisma_counts.json"), "w"), indent=1, default=str)
106+ print(json.dumps(P, indent=1, default=str)); log("done")
107+
108+
109+if __name__ == "__main__":
110+ main()
added systematic_review/inspect_2026.py +11 −0
@@ -0,0 +1,11 @@
1+import duckdb, sys
2+con = duckdb.connect(sys.argv[1] + "/data/processed/wp44.duckdb", read_only=True)
3+q = lambda s: con.execute(s).fetchall()
4+print("2026 eligible by first_month:", q("select first_month, count(*) from corpus_works where eligible and publication_year=2026 group by 1 order by 1"))
5+print("2026 by type:", q("select type, count(*) from corpus_works where eligible and publication_year=2026 group by 1 order by 2 desc limit 6"))
6+print("2026 top sources:", q("select source_name, count(*) from corpus_works where eligible and publication_year=2026 group by 1 order by 2 desc limit 10"))
7+print("2025 top sources:", q("select source_name, count(*) from corpus_works where eligible and publication_year=2025 group by 1 order by 2 desc limit 10"))
8+print("2026 pub_date day-01 share:", q("select sum((substr(publication_date,9,2)='01')::int)*1.0/count(*) from corpus_works where eligible and publication_year=2026"))
9+print("2025 pub_date day-01 share:", q("select sum((substr(publication_date,9,2)='01')::int)*1.0/count(*) from corpus_works where eligible and publication_year=2025"))
10+print("2026 created_date months:", q("select substr(created_date,1,7), count(*) from corpus_works where eligible and publication_year=2026 group by 1 order by 1"))
11+print("other-type dblp sample:", q("select id, doi, source_name, publication_year, n_refs from corpus_works where in_dblp and type='other' limit 4"))
added systematic_review/inspect_sources.py +10 −0
@@ -0,0 +1,10 @@
1+import duckdb, sys
2+con = duckdb.connect(sys.argv[1] + "/data/processed/wp44.duckdb", read_only=True)
3+q = lambda s: con.execute(s).fetchall()
4+print("eligible by source_type:", q("select source_type, count(*) n, sum(in_dblp::int) dblp, sum((arxiv_id_linked is not null)::int) arx, sum(has_abstract::int) ab, sum((n_refs>0)::int) refs from corpus_works where eligible group by 1 order by 2 desc"))
5+print("repository sources top 15:", q("select source_name, count(*) n, sum(in_dblp::int) dblp, sum((arxiv_id_linked is not null)::int) arx, sum((n_refs>0)::int) refs, sum(has_abstract::int) ab from corpus_works where eligible and source_type='repository' group by 1 order by 2 desc limit 15"))
6+print("Zenodo by year:", q("select publication_year, count(*), sum((n_refs>0)::int) refs, sum(has_abstract::int) ab, sum(in_dblp::int) dblp from corpus_works where eligible and source_name like 'Zenodo%' group by 1 order by 1"))
7+print("Zenodo 2026 sample:", q("select substr(title,1,70), type, n_refs, has_abstract, n_authors, language from corpus_works where eligible and source_name like 'Zenodo%' and publication_year=2026 using sample 8"))
8+print("NULL-source by year:", q("select publication_year, count(*), sum((n_refs>0)::int) refs, sum(has_abstract::int) ab, sum(in_dblp::int) dblp, sum((type='dissertation')::int) diss from corpus_works where eligible and source_id is null group by 1 order by 1"))
9+print("NULL-source 2025 sample:", q("select substr(title,1,60), type, n_refs, has_abstract, n_locations, indexed_in from corpus_works where eligible and source_id is null and publication_year=2025 using sample 8"))
10+print("day-01-01 share by year:", q("select publication_year, round(sum((substr(publication_date,6,5)='01-01')::int)*1.0/count(*),3) from corpus_works where eligible group by 1 order by 1"))
added taxonomy/classify.py +150 −0
@@ -0,0 +1,150 @@
1+#!/usr/bin/env python3
2+"""WP44 — Phase 2: assign the 40 ontology categories (multi-label) to every work of the corpus.
3+
4+Signals per work: OpenAlex topics (ids + subfield ids), arXiv categories (through link_arxiv_oa), DBLP venue stream
5+(through link_dblp_oa), keyword regexes on title and abstract. Scoring follows ontology.yaml `assignment_rule`:
6+per signal type the matched weights are summed and capped at 1.0, then summed over signal types; a category is assigned
7+if score >= threshold; primary = argmax (ties: child before parent, then lower category number).
8+
9+Step 1 (`--prepare`): DuckDB builds data/processed/classification/input/chunk_*.parquet (id, year, title, abstract,
10+topic_ids, subfield_ids, arxiv_cats, dblp_streams). Step 2 (`--run`): multiprocessing over chunks →
11+data/processed/classification/work_categories.parquet (work_id, category_id, score, s_topic, s_subfield, s_arxiv,
12+s_dblp, s_kw) and work_primary.parquet (work_id, primary_category, n_categories). Deterministic; version-stamped.
13+"""
14+import argparse, glob, json, multiprocessing as mp, os, re, time, datetime as dt
15+import yaml, duckdb
16+import pyarrow as pa, pyarrow.parquet as pq
17+
18+ONT = None
19+
20+
21+def load_ontology(path):
22+ o = yaml.safe_load(open(path))
23+ cats = o["categories"]
24+ comp = []
25+ for c in cats:
26+ kws = []
27+ for k in (c.get("keywords") or []):
28+ kws.append((re.compile(k["pattern"]), float(k["weight"]), k.get("field", "both")))
29+ exc = [re.compile(p if isinstance(p, str) else p["pattern"]) for p in (c.get("exclusions") or [])]
30+ comp.append(dict(
31+ id=c["id"], number=int(c["number"]), parent=c.get("child_of"),
32+ topics={str(t["id"]).lstrip("T"): float(t["weight"]) for t in (c.get("openalex_topics") or [])},
33+ subfields={str(t["id"]): float(t["weight"]) for t in (c.get("openalex_subfields") or [])},
34+ arxiv={t["id"]: float(t["weight"]) for t in (c.get("arxiv_categories") or [])},
35+ dblp={t["key"]: float(t["weight"]) for t in (c.get("dblp_streams") or [])},
36+ kws=kws, exc=exc))
37+ rule = o.get("assignment_rule", {})
38+ return dict(cats=comp, threshold=float(rule.get("threshold", 1.0)), version=o.get("version"))
39+
40+
41+def _init(path):
42+ global ONT
43+ ONT = load_ontology(path)
44+
45+
46+def score_row(title, abstract, topic_ids, subfield_ids, arxiv_cats, dblp_streams):
47+ title = title or ""; abstract = abstract or ""; both = title + "\n" + abstract
48+ out = []
49+ for c in ONT["cats"]:
50+ st = min(1.0, sum(c["topics"].get(str(t), 0.0) for t in (topic_ids or [])))
51+ ss = min(1.0, sum(c["subfields"].get(str(s), 0.0) for s in set(subfield_ids or [])))
52+ sa = min(1.0, sum(c["arxiv"].get(a, 0.0) for a in set(arxiv_cats or [])))
53+ sd = min(1.0, sum(c["dblp"].get(d, 0.0) for d in set(dblp_streams or [])))
54+ sk = 0.0
55+ for rx, w, field in c["kws"]:
56+ txt = title if field == "title" else abstract if field == "abstract" else both
57+ if txt and rx.search(txt):
58+ sk += w
59+ if sk >= 1.0:
60+ sk = 1.0; break
61+ s = st + ss + sa + sd + sk
62+ if s >= ONT["threshold"] and c["exc"] and any(rx.search(both) for rx in c["exc"]):
63+ s = 0.0
64+ if s >= ONT["threshold"]:
65+ out.append((c["id"], c["number"], c["parent"], round(s, 3), st, ss, sa, sd, sk))
66+ return out
67+
68+
69+def process_chunk(args):
70+ path, out_dir = args
71+ name = os.path.basename(path).replace(".parquet", "")
72+ o1 = os.path.join(out_dir, "cats", name + ".parquet"); o2 = os.path.join(out_dir, "primary", name + ".parquet")
73+ if os.path.exists(o1) and os.path.exists(o2):
74+ return name, 0, 0
75+ t = pq.read_table(path).to_pydict()
76+ cw, cc, cs, c1, c2, c3, c4, c5 = [], [], [], [], [], [], [], []
77+ pw, pc, pn, ps = [], [], [], []
78+ n = len(t["id"])
79+ for i in range(n):
80+ res = score_row(t["title"][i], t["abstract"][i], t["topic_ids"][i], t["subfield_ids"][i], t["arxiv_cats"][i], t["dblp_streams"][i])
81+ for (cid, num, parent, s, st, ss, sa, sd, sk) in res:
82+ cw.append(t["id"][i]); cc.append(cid); cs.append(s); c1.append(st); c2.append(ss); c3.append(sa); c4.append(sd); c5.append(sk)
83+ if res:
84+ parents = {r[2] for r in res if r[2]}
85+ # primary: highest score; tie → child before parent (a category that is someone's parent loses), then lower number
86+ best = sorted(res, key=lambda r: (-r[3], 1 if r[0] in parents else 0, r[1]))[0]
87+ pw.append(t["id"][i]); pc.append(best[0]); pn.append(len(res)); ps.append(best[3])
88+ os.makedirs(os.path.dirname(o1), exist_ok=True); os.makedirs(os.path.dirname(o2), exist_ok=True)
89+ pq.write_table(pa.table({"work_id": pa.array(cw, pa.int64()), "category_id": cc, "score": pa.array(cs, pa.float32()), "s_topic": pa.array(c1, pa.float32()),
90+ "s_subfield": pa.array(c2, pa.float32()), "s_arxiv": pa.array(c3, pa.float32()), "s_dblp": pa.array(c4, pa.float32()), "s_kw": pa.array(c5, pa.float32())}), o1 + ".part", compression="zstd")
91+ os.replace(o1 + ".part", o1)
92+ pq.write_table(pa.table({"work_id": pa.array(pw, pa.int64()), "primary_category": pc, "n_categories": pa.array(pn, pa.int16()), "primary_score": pa.array(ps, pa.float32())}), o2 + ".part", compression="zstd")
93+ os.replace(o2 + ".part", o2)
94+ return name, n, len(pw)
95+
96+
97+def prepare(root, db, out_dir, chunk_rows):
98+ con = duckdb.connect(db); con.execute("SET threads TO 24; SET memory_limit='60GB';")
99+ os.makedirs(os.path.join(out_dir, "input"), exist_ok=True)
100+ con.execute("""CREATE OR REPLACE TABLE cls_input AS
101+ SELECT w.id, w.publication_year AS year, w.title, a.abstract,
102+ (SELECT list(topic_id ORDER BY rank) FROM oa_work_topics t WHERE t.work_id = w.id) AS topic_ids,
103+ (SELECT list(subfield_id ORDER BY rank) FROM oa_work_topics t WHERE t.work_id = w.id) AS subfield_ids,
104+ (SELECT flatten(list(x.categories)) FROM link_arxiv_oa l JOIN arxiv_keys x USING(arxiv_id) WHERE l.oa_id = w.id) AS arxiv_cats,
105+ (SELECT list(DISTINCT d.stream) FROM link_dblp_oa l JOIN dblp_keys d USING(dblp_key) WHERE l.oa_id = w.id) AS dblp_streams
106+ FROM corpus_works w LEFT JOIN oa_abstracts a ON a.work_id = w.id""")
107+ n = con.execute("SELECT count(*) FROM cls_input").fetchone()[0]
108+ con.execute(f"""COPY (SELECT *, (row_number() OVER ()) // {chunk_rows} AS chunk FROM cls_input) TO '{out_dir}/input' (FORMAT PARQUET, PARTITION_BY (chunk), OVERWRITE_OR_IGNORE, COMPRESSION ZSTD)""")
109+ print("prepared", n, "works")
110+
111+
112+def main():
113+ ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); ap.add_argument("--ontology", required=True)
114+ ap.add_argument("--prepare", action="store_true"); ap.add_argument("--run", action="store_true"); ap.add_argument("--workers", type=int, default=20)
115+ ap.add_argument("--chunk-rows", type=int, default=100000); ap.add_argument("--limit", type=int, default=0)
116+ a = ap.parse_args(); out_dir = os.path.join(a.root, "data/processed/classification"); db = os.path.join(a.root, "data/processed/wp44.duckdb")
117+ if a.prepare:
118+ prepare(a.root, db, out_dir, a.chunk_rows)
119+ if a.run:
120+ files = sorted(glob.glob(os.path.join(out_dir, "input", "chunk=*", "*.parquet")))
121+ files = [(f, out_dir) for f in files]
122+ if a.limit:
123+ files = files[: a.limit]
124+ # chunk files share the name data_0.parquet inside chunk=N dirs → rename key
125+ files = [(f, out_dir) for f, _ in files]
126+ t0 = time.time(); tot = 0; cl = 0
127+ with mp.Pool(a.workers, initializer=_init, initargs=(a.ontology,)) as pool:
128+ for name, n, k in pool.imap_unordered(process_chunk_named, [(f, out_dir, os.path.basename(os.path.dirname(f))) for f, _ in files]):
129+ tot += n; cl += k
130+ print(f"[{time.time()-t0:6.0f}s] {name}: {n} works, {k} classified | total {tot} / classified {cl}", flush=True)
131+ json.dump({"ontology": a.ontology, "version": load_ontology(a.ontology)["version"], "works": tot, "classified": cl, "when": dt.datetime.now(dt.UTC).isoformat(),
132+ "seconds": round(time.time() - t0)}, open(os.path.join(out_dir, "classification_report.json"), "w"), indent=1)
133+
134+
135+def process_chunk_named(args):
136+ path, out_dir, chunk_name = args
137+ name = chunk_name.replace("=", "_")
138+ o1 = os.path.join(out_dir, "cats", name + ".parquet"); o2 = os.path.join(out_dir, "primary", name + ".parquet")
139+ if os.path.exists(o1) and os.path.exists(o2):
140+ return name, 0, 0
141+ # reuse process_chunk with a symlinked name
142+ tmp = os.path.join(out_dir, "input_named"); os.makedirs(tmp, exist_ok=True)
143+ link = os.path.join(tmp, name + ".parquet")
144+ if not os.path.exists(link):
145+ os.symlink(path, link)
146+ return process_chunk((link, out_dir))
147+
148+
149+if __name__ == "__main__":
150+ main()
added taxonomy/src/area_ai_ml_a.py +153 −0
@@ -0,0 +1,153 @@
1+"""Area ai_ml, part A: categories 1-4 (LLM, foundation models, multimodal AI, generative AI).
2+
3+Tuple conventions (converted to dicts by build_ontology.py):
4+ topics: (openalex_topic_id, weight, note)
5+ subfields:(subfield_id, weight)
6+ arxiv: (category, weight)
7+ streams: (dblp_stream_key, weight)
8+ keywords: (regex, weight, field) field in {"title", "abstract", "both"}
9+"""
10+
11+CATEGORIES = [
12+dict(
13+ id="llm", number=1, name="Large Language Models", area="ai_ml",
14+ definition=("Research whose object is a transformer-based language model trained at scale: pre-training, "
15+ "instruction tuning, prompting and in-context learning, reasoning, retrieval augmentation, efficient "
16+ "inference, evaluation and benchmarks of LLMs. Excludes classical NLP tasks solved with small task-specific "
17+ "models (nlp) and non-language foundation models (foundation_models)."),
18+ topics=[("T10028", 1.0, "OpenAlex cluster hosting BERT / pre-trained LM / RAG / QA work: the de-facto LLM topic"),
19+ ("T11714", 0.4, "multimodal LLMs and vision-language models"),
20+ ("T12031", 0.3, "dialogue systems, LLM chatbots"),
21+ ("T10181", 0.3, "NLP techniques incl. neural MT and NLG"),
22+ ("T12128", 0.3, "chatbots in service interactions"),
23+ ("T14414", 0.2, "AI in education: mostly ChatGPT studies post-2023")],
24+ subfields=[(1702, 0.2)],
25+ arxiv=[("cs.CL", 0.5), ("cs.AI", 0.2), ("cs.LG", 0.2)],
26+ streams=[("conf/acl", 0.5), ("conf/emnlp", 0.5), ("conf/naacl", 0.5), ("conf/eacl", 0.4), ("conf/coling", 0.4),
27+ ("journals/tacl", 0.5), ("conf/iclr", 0.3), ("conf/nips", 0.3), ("conf/icml", 0.3),
28+ ("journals/tmlr", 0.3), ("conf/aaai", 0.2), ("conf/nlpcc", 0.3)],
29+ keywords=[
30+ (r"(?i)\blarge language models?\b", 1.0, "both"),
31+ (r"(?i)\bLLMs?\b", 1.0, "both"),
32+ (r"(?i)\b(ChatGPT|GPT-?3(\.5)?|GPT-?4o?|GPT-?4\.\d|GPT-?5|LLaMA|Llama-? ?[234]|PaLM|InstructGPT|Qwen|Vicuna|Mixtral|DeepSeek-?(V\d|R1)|Claude ?[34]|Gemini ?(Pro|Ultra|1\.5|2))\b", 1.0, "both"),
33+ (r"(?i)\bpre-?trained language models?\b|\bPLMs?\b", 0.7, "both"),
34+ (r"(?i)\b(instruction[- ]tuning|instruction-?tuned|in-?context learning|chain[- ]of[- ]thought|retrieval[- ]augmented generation|RAG\b|prompt (engineering|tuning|learning)|prompting|RLHF|supervised fine-?tuning|SFT\b)\b", 0.8, "both"),
35+ (r"(?i)\b(masked|autoregressive|causal|decoder-?only|encoder-?decoder) language model", 0.7, "both"),
36+ (r"(?i)\b(language model|transformer) (pre-?training|scaling|scaling laws?)\b", 0.7, "both"),
37+ (r"(?i)\b(parameter-?efficient fine-?tuning|PEFT\b|LoRA\b|adapters? for (language|LLM))\b", 0.6, "both"),
38+ (r"(?i)\bBERT\b|\bRoBERTa\b|\bT5\b|\bBART\b", 0.6, "both"),
39+ (r"(?i)\bhallucinat(ion|ions|e|es|ing)\b", 0.5, "both"),
40+ (r"(?i)\blanguage models?\b", 0.4, "both"),
41+ (r"(?i)\b(tokeni[sz](er|ation)|context (window|length)|KV[- ]cache|speculative decoding|mixture[- ]of[- ]experts)\b", 0.4, "both"),
42+ ],
43+ exclusions=[],
44+ related=["foundation_models", "nlp", "generative_ai", "ai_agents", "ai_safety"],
45+ parent_of=[], child_of=["foundation_models"],
46+ provenance=("Topics chosen by inspecting OpenAlex keyword lists (T10028 lists BERT, pre-trained language models, "
47+ "retrieval-augmented generation). Venues: *CL conferences and ML venues at partial weight because LLM "
48+ "work is spread across them. Keyword patterns reach the threshold alone, as required for the six "
49+ "AI categories OpenAlex cannot separate."),
50+),
51+dict(
52+ id="foundation_models", number=2, name="Foundation Models", area="ai_ml",
53+ definition=("Large pre-trained models intended for adaptation to many downstream tasks in any modality: scaling "
54+ "laws, pre-training objectives, self-supervised representation learning at scale, transfer, fine-tuning "
55+ "and adapter methods, model releases and their evaluation. Language-specific work is llm (a child); "
56+ "vision-language work is multimodal_ai (a child)."),
57+ topics=[("T11307", 0.5, "domain adaptation, few-shot, parameter-efficient fine-tuning, continual learning"),
58+ ("T11714", 0.4, "vision-language models / CLIP"),
59+ ("T10028", 0.3, "pre-trained language models"),
60+ ("T10036", 0.2, "backbone networks, model compression"),
61+ ("T12422", 0.2, "foundation models in medical imaging / radiomics")],
62+ subfields=[(1702, 0.2)],
63+ arxiv=[("cs.LG", 0.4), ("cs.CV", 0.2), ("cs.CL", 0.2), ("cs.AI", 0.2)],
64+ streams=[("conf/nips", 0.3), ("conf/icml", 0.3), ("conf/iclr", 0.3), ("journals/tmlr", 0.3), ("conf/cvpr", 0.2),
65+ ("conf/iccv", 0.2), ("conf/eccv", 0.2), ("journals/natmi", 0.2)],
66+ keywords=[
67+ (r"(?i)\bfoundation models?\b|\bfoundational models?\b", 1.0, "both"),
68+ (r"(?i)\b(vision|visual|time[- ]series|protein|genomic|geospatial|medical|clinical|weather|climate|chemistry|molecular|tabular|graph|speech|audio|robotics?|world) foundation models?\b", 1.0, "both"),
69+ (r"(?i)\bscaling laws?\b", 0.8, "both"),
70+ (r"(?i)\b(self-?supervised|contrastive|masked) (pre-?training|representation learning|learning at scale)\b", 0.7, "both"),
71+ (r"(?i)\b(pre-?train(ed|ing)) (on|at) (web|internet|large|massive|billion)[- ]scale", 0.7, "both"),
72+ (r"(?i)\b(vision transformers?|ViT-?[BLHG]?|DINO(v2)?|segment anything|SAM ?2|masked auto-?encoders?|MAE\b|CLIP\b|SigLIP)\b", 0.6, "both"),
73+ (r"(?i)\b(general-?purpose|generalist|universal|all-?in-?one) (models?|agents?|representations?)\b", 0.6, "both"),
74+ (r"(?i)\b(transfer learning|fine-?tun(ing|ed?)|downstream tasks?|zero-?shot|few-?shot)\b", 0.4, "both"),
75+ (r"(?i)\b(adapters?|LoRA|parameter-?efficient|prompt tuning|emergent (abilities|capabilities))\b", 0.5, "both"),
76+ (r"(?i)\bworld models?\b", 0.5, "both"),
77+ ],
78+ exclusions=[],
79+ related=["llm", "multimodal_ai", "generative_ai", "deep_learning_architectures", "ml_theory"],
80+ parent_of=["llm", "multimodal_ai"], child_of=[],
81+ provenance=("No OpenAlex topic isolates foundation models; T11307 (PEFT, few-shot, continual) is the closest "
82+ "methodological host. Keyword patterns carry the category; the generic 'transfer learning/fine-tuning' "
83+ "terms are deliberately below threshold."),
84+),
85+dict(
86+ id="multimodal_ai", number=3, name="Multimodal AI", area="ai_ml",
87+ definition=("Models and systems that jointly process two or more modalities (vision-language, audio-visual, "
88+ "video-text, 3D-language, sensors with language): vision-language models and multimodal LLMs, "
89+ "captioning, visual question answering, cross-modal retrieval and grounding. Unimodal vision is "
90+ "computer_vision; pure speech is speech_audio."),
91+ topics=[("T11714", 1.0, "Multimodal Machine Learning Applications: VLMs, MLLMs, VQA, captioning"),
92+ ("T10627", 0.4, "image and video retrieval incl. cross-modal"),
93+ ("T11439", 0.2, "video analysis and summarization"),
94+ ("T10824", 0.2, "image retrieval and classification"),
95+ ("T13155", 0.2, "digital communication and language (multimodal discourse)")],
96+ subfields=[(1707, 0.1)],
97+ arxiv=[("cs.MM", 0.6), ("cs.CV", 0.3), ("cs.CL", 0.3), ("cs.LG", 0.2)],
98+ streams=[("conf/mm", 0.6), ("journals/tmm", 0.6), ("conf/icmi", 0.6), ("journals/tomccap", 0.5),
99+ ("conf/icmcs", 0.5), ("conf/mir", 0.5), ("journals/mms", 0.5), ("journals/mta", 0.3),
100+ ("conf/cvpr", 0.2), ("conf/iccv", 0.2), ("conf/eccv", 0.2), ("conf/acl", 0.2), ("conf/emnlp", 0.2)],
101+ keywords=[
102+ (r"(?i)\bmulti-?modal(ity)?\b", 0.7, "both"),
103+ (r"(?i)\b(vision[- ]language|vision[- ]and[- ]language|visual[- ]language|language[- ]vision|image[- ]text|video[- ]language|audio[- ]language|speech[- ]language|3D[- ]language|text[- ]to[- ]3D)[- ]?(models?|pre-?training|tasks?|alignment|understanding|learning|reasoning|navigation|action|retrieval|generation)?\b", 1.0, "both"),
104+ (r"(?i)\b(VLMs?|MLLMs?|LVLMs?|LMMs?|VLAs?)\b", 1.0, "both"),
105+ (r"(?i)\bvisual question answering\b|\bVQA\b|\bvideo question answering\b", 1.0, "both"),
106+ (r"(?i)\b(image|video|dense|audio) captioning\b", 1.0, "both"),
107+ (r"(?i)\bcross-?modal(ity)?\b", 0.8, "both"),
108+ (r"(?i)\b(audio-?visual|visual grounding|referring expression|visual dialog|visual reasoning|visual instruction tuning|LLaVA|BLIP-?2?|Flamingo|GPT-?4V|CLIP|ALIGN\b|CoCa)\b", 0.8, "both"),
109+ (r"(?i)\b(text-to-image|text-to-video|image-to-text|text-image|image-text) (retrieval|alignment|matching|generation|models?)\b", 0.8, "both"),
110+ (r"(?i)\b(modalit(y|ies) (fusion|alignment|gap)|(early|late|intermediate) fusion of|sensor fusion with language)\b", 0.7, "both"),
111+ ],
112+ exclusions=[r"(?i)\bmultimodal (therapy|analgesi|treatment|intervention|pain|anesthesi|anaesthesi|imaging biomarkers?|prehabilitation|transport)\b"],
113+ related=["computer_vision", "nlp", "llm", "speech_audio", "generative_ai"],
114+ parent_of=[], child_of=["foundation_models"],
115+ provenance=("T11714 is the only OpenAlex topic centred on vision-language learning. Multimedia venues at partial "
116+ "weight (they also host unimodal media work). Medical 'multimodal' senses are vetoed."),
117+),
118+dict(
119+ id="generative_ai", number=4, name="Generative AI", area="ai_ml",
120+ definition=("Models that synthesise new content (images, video, audio, 3D, text, code, molecules) and their training, "
121+ "sampling, evaluation, detection and misuse: GANs, VAEs, diffusion and flow models, autoregressive "
122+ "generators, text-to-X synthesis, deepfakes and AI-generated-content detection. LLM text generation as "
123+ "such is llm unless the contribution concerns the generative model or its outputs' authenticity."),
124+ topics=[("T10775", 1.0, "GANs, diffusion models, image/video generation"),
125+ ("T12357", 0.5, "digital media forensics: deepfake detection"),
126+ ("T11349", 0.2, "music technology (music generation)"),
127+ ("T10388", 0.2, "steganography / watermarking (of generated content)"),
128+ ("T14254", 0.2, "digital media and visual art (AI art)")],
129+ subfields=[],
130+ arxiv=[("cs.CV", 0.2), ("cs.LG", 0.3), ("cs.GR", 0.2), ("cs.SD", 0.1), ("cs.AI", 0.1), ("cs.MM", 0.1)],
131+ streams=[("conf/nips", 0.2), ("conf/icml", 0.2), ("conf/iclr", 0.2), ("conf/cvpr", 0.2), ("conf/iccv", 0.2),
132+ ("conf/eccv", 0.2), ("conf/siggraph", 0.2), ("conf/siggrapha", 0.2), ("journals/tog", 0.1)],
133+ keywords=[
134+ (r"(?i)\bgenerative (AI|artificial intelligence|models?|modell?ing|adversarial networks?|pre-?trained|design|art)\b|\bGenAI\b|\bAIGC\b", 1.0, "both"),
135+ (r"(?i)\bGANs?\b|\b(conditional|cycle|style|Wasserstein)GAN\b|\bStyleGAN\w*\b", 0.9, "both"),
136+ (r"(?i)\b(denoising diffusion|latent diffusion|stable diffusion|diffusion transformers?|DDPM|DDIM|score-?based generative|score matching|flow[- ]matching|rectified flow|normali[sz]ing flows?|consistency models?)\b", 1.0, "both"),
137+ (r"(?i)\bdiffusion models?\b", 0.8, "both"),
138+ (r"(?i)\btext-?to-?(image|video|3D|audio|speech|music|motion|SVG|scene)\b", 1.0, "both"),
139+ (r"(?i)\b(image|video|music|audio|speech|3D|motion|molecule|molecular|code|scene|face|avatar|talking[- ]head|layout|texture|graph) (generation|synthesis)\b", 0.7, "both"),
140+ (r"(?i)\bdeep-?fakes?\b|\bface swap(ping)?\b|\bsynthetic media\b", 1.0, "both"),
141+ (r"(?i)\bvariational auto-?encoders?\b|\bVAEs?\b|\bVQ-?VAE\b|\bVQGAN\b", 0.7, "both"),
142+ (r"(?i)\b(AI-?generated|machine-?generated|synthetic(ally)?[- ]generated|LLM-?generated) (content|images?|text|data|videos?|audio|art|code|faces)\b", 0.8, "both"),
143+ (r"(?i)\b(image (inpainting|outpainting|editing|to-image translation)|style transfer|super-?resolution|image restoration|controllable generation|ControlNet|DreamBooth|textual inversion|classifier-?free guidance|Midjourney|DALL-?E|Imagen|Sora)\b", 0.6, "both"),
144+ (r"(?i)\bautoregressive (models?|generation|image|video)\b", 0.5, "both"),
145+ ],
146+ exclusions=[r"(?is)^(?!.*(generative|GANs?\b|synthesis|generation|deepfake|VAE|AI-?generated)).*\b(drug|gas|molecular|thermal|anomalous|reaction-?|water|heat|ion|oxygen|particle|turbulent|surface|solute|vapou?r) diffusion\b",
147+ r"(?is)^(?!.*(generative|GANs?\b|synthesis|generation|deepfake|VAE|AI-?generated)).*\bdiffusion (coefficients?|tensor|MRI|weighted|equations?|process(es)?|of innovations?|limited)\b"],
148+ related=["llm", "multimodal_ai", "computer_vision", "computer_graphics", "foundation_models", "speech_audio"],
149+ parent_of=[], child_of=[],
150+ provenance=("T10775 keywords explicitly list GANs, diffusion models, Stable Diffusion, text-to-image. Physics/finance "
151+ "senses of 'diffusion' are vetoed only when no generative-modelling term is present."),
152+),
153+]
added taxonomy/src/area_ai_ml_b.py +205 −0
@@ -0,0 +1,205 @@
1+"""Area ai_ml, part B: categories 5-8 and 38 (agents, RL, ML theory, DL architectures, AI safety)."""
2+
3+CATEGORIES = [
4+dict(
5+ id="ai_agents", number=5, name="AI Agents and Autonomous Systems", area="ai_ml",
6+ definition=("Software agents that perceive, decide and act with some autonomy: LLM-based agents and tool use, "
7+ "multi-agent systems, planning and automated reasoning for acting, agent architectures, negotiation, "
8+ "agentic workflows and benchmarks, embodied agents in simulated environments. Physical robots are "
9+ "robotics; RL algorithms as such are reinforcement_learning."),
10+ topics=[("T10456", 1.0, "Multi-Agent Systems and Negotiation: BDI, agent architectures, argumentation"),
11+ ("T10906", 0.6, "AI-based problem solving and planning (PDDL, multi-agent planning)"),
12+ ("T11574", 0.4, "AI in games (game-playing agents)"),
13+ ("T10249", 0.3, "distributed control of multi-agent systems (control-theoretic MAS)"),
14+ ("T12128", 0.2, "chatbots and service agents")],
15+ subfields=[(1702, 0.1)],
16+ arxiv=[("cs.MA", 1.0), ("cs.AI", 0.4)],
17+ streams=[("conf/atal", 1.0), ("conf/ifaamas", 1.0), ("journals/aamas", 1.0), ("conf/icaart", 0.6),
18+ ("conf/aaai", 0.2), ("conf/ijcai", 0.2), ("journals/jair", 0.2), ("journals/ai", 0.2), ("conf/kr", 0.2)],
19+ keywords=[
20+ (r"(?i)\b(LLM|language[- ]model|AI|autonomous|intelligent|embodied|conversational|software|web|GUI|coding|browsing|tool-?using|generative|multi-?step|computer-?use|research|planning)[- ](based |powered |driven )?agents?\b", 1.0, "both"),
21+ (r"(?i)\bagentic\b|\bagent-?based (LLM|AI|systems?|frameworks?|workflows?)\b", 1.0, "both"),
22+ (r"(?i)\bmulti-?agent (systems?|reinforcement|collaboration|coordination|negotiation|planning|simulation|frameworks?|debate|learning|path finding|communication|environments?)\b|\bMAS\b|\bMARL\b", 1.0, "both"),
23+ (r"(?i)\b(tool (use|using|learning|calling|augmented|integration)|function calling|API calling|ReAct|AutoGPT|AutoGen|LangChain|LangGraph|CrewAI|MetaGPT|SWE-?agent|SWE-?bench|WebArena|AgentBench|OSWorld|Model Context Protocol|MCP servers?)\b", 1.0, "both"),
24+ (r"(?i)\b(BDI|belief-?desire-?intention) (agents?|architectures?|models?)\b|\bagent-?oriented (programming|software engineering)\b", 1.0, "both"),
25+ (r"(?i)\b(automated|classical|hierarchical|task|AI|temporal|epistemic) planning\b|\bPDDL\b|\btask (decomposition|and motion planning)\b", 0.7, "both"),
26+ (r"(?i)\bagent-?based (model(ing|s)?|simulations?)\b|\bABM\b", 0.6, "both"),
27+ (r"(?i)\b(autonomous systems?|autonomy|self-?driving|self-?improving|self-?evolving|long-?horizon (tasks?|planning)|memory (module|mechanism|augmented) (for|in) agents?)\b", 0.5, "both"),
28+ (r"(?i)\b(game[- ]playing|general game|AlphaGo|AlphaZero|MuZero|OpenAI Five|AlphaStar|CICERO|Diplomacy)\b", 0.6, "both"),
29+ ],
30+ exclusions=[r"(?i)\b(reagents?|chemical|contrast|therapeutic|antimicrobial|chelating|reducing|oxidi[sz]ing|pharmacological|anti-?cancer|infectious|causative|etiologic(al)?|biological|pathogenic|anesthetic|anaesthetic|alkylating|cross-?linking|blowing|surface-?active|wetting|coupling|cytotoxic|antiviral|antibacterial|antifungal|bleaching|dispersing|sweetening|flavou?ring|leavening|real estate|travel|insurance|customs|shipping|sports|literary|secret|double|free|special) agents?\b"],
31+ related=["llm", "reinforcement_learning", "robotics", "ai_safety", "autonomous_scientific_discovery"],
32+ parent_of=[], child_of=[],
33+ provenance=("T10456 is the classical MAS host; the LLM-agent wave has no OpenAlex topic yet, so keyword patterns on "
34+ "'X agent' (with the qualifier restricted to computing senses) and agent frameworks/benchmarks carry it. "
35+ "Chemical/medical/commercial senses of 'agent' are vetoed."),
36+),
37+dict(
38+ id="reinforcement_learning", number=6, name="Reinforcement Learning", area="ai_ml",
39+ definition=("Learning to act from reward signals: value- and policy-based methods, model-based RL, offline and "
40+ "inverse RL, multi-agent RL, bandits, exploration, reward design, RL from human feedback, and RL theory. "
41+ "Applications to robot control are shared with robotics."),
42+ topics=[("T10462", 1.0, "Reinforcement Learning in Robotics: Q-learning, PPO, offline RL, imitation learning"),
43+ ("T12794", 0.5, "adaptive dynamic programming control"),
44+ ("T11574", 0.3, "AI in games"),
45+ ("T10100", 0.1, "metaheuristic optimisation (RL-based hyper-heuristics)")],
46+ subfields=[],
47+ arxiv=[("cs.LG", 0.3), ("cs.AI", 0.2), ("cs.RO", 0.1), ("eess.SY", 0.1), ("cs.MA", 0.1)],
48+ streams=[("conf/nips", 0.2), ("conf/icml", 0.2), ("conf/iclr", 0.2), ("conf/corl", 0.3), ("conf/aaai", 0.1),
49+ ("journals/jmlr", 0.1), ("conf/atal", 0.2), ("journals/tnn", 0.1)],
50+ keywords=[
51+ (r"(?i)\breinforcement learning\b", 1.0, "both"),
52+ (r"(?i)\b(deep|multi-?agent|offline|online|inverse|model-?based|model-?free|safe|hierarchical|meta|batch|distributional|constrained|goal-?conditioned|preference-?based|federated|quantum)[- ]RL\b", 1.0, "both"),
53+ (r"(?i)\b(Q-?learning|DQN|double DQN|Rainbow DQN|PPO|proximal policy optimi[sz]ation|TRPO|actor-?critic|A3C|A2C|SAC\b|soft actor-?critic|TD3|DDPG|policy gradients?|REINFORCE|AlphaZero|MuZero|Dreamer(V\d)?|decision transformers?|world models? for (control|RL|planning))\b", 1.0, "both"),
54+ (r"(?i)\bMarkov decision process(es)?\b|\bMDPs?\b|\bPOMDPs?\b|\bsemi-?MDP\b|\bDec-?POMDP\b", 0.8, "both"),
55+ (r"(?i)\b(reward (functions?|shaping|models?|hacking|design|learning|signals?)|sparse rewards?|exploration[- ]exploitation|intrinsic (motivation|rewards?)|curiosity-?driven|multi-?armed bandits?|contextual bandits?|credit assignment|experience replay|sim-?to-?real)\b", 0.8, "both"),
56+ (r"(?i)\b(imitation learning|behavio(u)?r(al)? cloning|learning from demonstrations?|inverse reinforcement|apprenticeship learning|offline (policy|data) (evaluation|learning))\b", 0.8, "both"),
57+ (r"(?i)\b(RLHF|RLAIF|reinforcement learning from (human|AI) feedback|GRPO|DPO\b|direct preference optimi[sz]ation|RLVR|reasoning models? trained with|verifiable rewards?)\b", 0.8, "both"),
58+ (r"(?i)\b(temporal[- ]difference|value (functions?|iteration)|policy (iteration|evaluation|optimi[sz]ation|learning)|Bellman|dynamic programming for control|optimal control via learning|regret bounds? for (MDP|RL))\b", 0.6, "both"),
59+ (r"(?i)\b(agents?|policies|policy|environments?|episodes?|rollouts?)\b", 0.2, "both"),
60+ ],
61+ exclusions=[r"(?i)\breinforcement (of|in|for) (concrete|steel|soil|composites?|polymers?|fib(er|re)s?|masonry|beams?|slabs?|pavements?)\b(?!.*reinforcement learning)"],
62+ related=["ai_agents", "robotics", "ml_theory", "ai_safety", "llm"],
63+ parent_of=[], child_of=[],
64+ provenance="T10462 is the RL topic (its keywords list MARL, Q-learning, PPO, offline RL). Algorithm names reach the threshold alone.",
65+),
66+dict(
67+ id="ml_theory", number=7, name="Machine Learning Theory", area="ai_ml",
68+ definition=("Statistical and computational learning theory and principled ML methodology: generalisation and sample "
69+ "complexity, optimisation theory for learning, kernel and Bayesian methods, probabilistic models, causal "
70+ "inference, online learning, PAC learning, interpretability and fairness as formal problems. "
71+ "Architecture engineering is deep_learning_architectures."),
72+ topics=[("T12072", 1.0, "Machine Learning and Algorithms: PAC learning, generalisation bounds, VC dimension"),
73+ ("T11612", 0.8, "stochastic gradient optimisation, NTK, implicit regularisation"),
74+ ("T12814", 0.7, "Gaussian processes and Bayesian inference"),
75+ ("T11303", 0.6, "Bayesian modelling and causal inference"),
76+ ("T10845", 0.5, "advanced causal inference techniques"),
77+ ("T11901", 0.5, "Bayesian methods and mixture models"),
78+ ("T12535", 0.4, "ML and data classification (SVM, ensembles)"),
79+ ("T10637", 0.4, "clustering algorithms"),
80+ ("T12026", 0.4, "explainable AI"),
81+ ("T12536", 0.4, "topological and geometric data analysis"),
82+ ("T11652", 0.3, "imbalanced classification"),
83+ ("T11689", 0.3, "adversarial robustness (certified robustness)"),
84+ ("T10963", 0.3, "advanced optimisation algorithms"),
85+ ("T12761", 0.3, "data stream mining (concept drift)"),
86+ ("T12205", 0.3, "time series forecasting"),
87+ ("T13373", 0.2, "data mining and ML applications"),
88+ ("T12676", 0.2, "extreme learning machines")],
89+ subfields=[(1702, 0.3)],
90+ arxiv=[("stat.ML", 0.8), ("cs.LG", 0.4), ("math.ST", 0.3), ("math.OC", 0.2)],
91+ streams=[("conf/colt", 1.0), ("conf/alt", 1.0), ("journals/jmlr", 0.8), ("conf/aistats", 0.8), ("conf/uai", 0.8),
92+ ("journals/ml", 0.6), ("journals/tmlr", 0.4), ("conf/nips", 0.3), ("conf/icml", 0.3), ("conf/iclr", 0.2),
93+ ("journals/tnn", 0.3), ("journals/nn", 0.3), ("journals/pr", 0.2), ("conf/pkdd", 0.3), ("conf/kdd", 0.2),
94+ ("conf/icdm", 0.2), ("journals/pami", 0.2), ("journals/ijar", 0.4), ("journals/sac", 0.4),
95+ ("journals/csda", 0.3), ("journals/ml", 0.6), ("journals/cstat", 0.2), ("journals/eswa", 0.1),
96+ ("journals/kbs", 0.1), ("journals/isci", 0.1), ("journals/ijon", 0.2), ("journals/nca", 0.1),
97+ ("journals/mlc", 0.3), ("journals/apin", 0.2), ("journals/air", 0.2)],
98+ keywords=[
99+ (r"(?i)\b(generali[sz]ation (bounds?|error|gap|theory)|sample complexity|regret (bounds?|minimi[sz]ation|analysis)|minimax (rates?|optimal|lower)|PAC(-Bayes(ian)?)?( learning| bounds?)?|Rademacher|VC[- ]dimension|convergence (rates?|guarantees?|analysis)|statistical learning theory|computational learning theory|learnability|uniform convergence|excess risk)\b", 0.9, "both"),
100+ (r"(?i)\b(kernel (methods?|machines?|ridge|mean embedding)|reproducing kernel|RKHS|Gaussian process(es)?|variational inference|Bayesian (optimi[sz]ation|neural networks?|inference|deep learning|nonparametric)|probabilistic (models?|programming|graphical)|graphical models?|expectation[- ]maximi[sz]ation|Markov chain Monte Carlo|MCMC|Hamiltonian Monte Carlo|normali[sz]ing flows? for inference|conformal prediction|calibration)\b", 0.8, "both"),
101+ (r"(?i)\bcausal (inference|discovery|representation learning|effect estimation|structure learning|machine learning)\b|\bcounterfactual (reasoning|inference|fairness)\b|\bstructural causal models?\b|\btreatment effect\b", 0.8, "both"),
102+ (r"(?i)\b(online learning|bandits?|stochastic (gradient|optimi[sz]ation|approximation)|non-?convex optimi[sz]ation|convex optimi[sz]ation for learning|implicit (bias|regulari[sz]ation)|gradient (descent|flow) dynamics|saddle points?|loss landscapes?|mirror descent|optimal transport|Wasserstein)\b", 0.6, "both"),
103+ (r"(?i)\b(double descent|neural tangent kernel|NTK\b|lottery ticket|over-?parameteri[sz](ed|ation)|benign overfitting|grokking|feature learning (regime|theory)|mean-?field (limit|theory) of neural|infinite-?width|random features?)\b", 1.0, "both"),
104+ (r"(?i)\b(interpretab(le|ility)|explainab(le|ility)) (machine learning|AI|models?|methods?)\b|\bXAI\b|\bSHAP\b|\bLIME\b|\bfeature attribution\b|\bsaliency maps? for\b", 0.5, "both"),
105+ (r"(?i)\b(fairness (constraints?|metrics?|in machine learning)|algorithmic fairness|demographic parity|equali[sz]ed odds|fair (classification|representation))\b", 0.5, "both"),
106+ (r"(?i)\b(semi-?supervised|self-?supervised|unsupervised|active|transfer|meta|continual|lifelong|few-?shot|zero-?shot|multi-?task|weakly[- ]supervised|positive-?unlabeled|contrastive|representation|manifold|metric|dictionary|ensemble|boosting|bagging) learning\b", 0.4, "both"),
107+ (r"(?i)\b(distribution shift|out-?of-?distribution|domain generali[sz]ation|covariate shift|label noise|robust(ness)? (to|under|against) (noise|shift|corruption)|concept drift|anomaly detection|outlier detection|imbalanced (data|learning|classification)|dimensionality reduction|matrix (completion|factori[sz]ation)|tensor (decomposition|methods?)|sparse (coding|recovery)|compressed sensing)\b", 0.5, "both"),
108+ (r"(?i)\b(support vector machines?|SVMs?|random forests?|gradient boosting|XGBoost|LightGBM|k-?means|spectral clustering|Gaussian mixture|hidden Markov models?|HMMs?|logistic regression|decision trees?|nearest neighbou?rs?|kNN)\b", 0.5, "both"),
109+ (r"(?i)\bmachine learning\b|\bstatistical learning\b|\blearning theory\b", 0.4, "both"),
110+ ],
111+ exclusions=[],
112+ related=["deep_learning_architectures", "reinforcement_learning", "algorithms", "computational_complexity", "ai_safety", "foundation_models"],
113+ parent_of=[], child_of=[],
114+ provenance=("Hosts classical ML and its theory: OpenAlex splits them across T12072 (learning theory), T11612 (optimisation), "
115+ "T12814/T11303/T11901 (Bayesian), T12535/T10637 (classical methods). Venue weights follow venue purity "
116+ "(COLT/ALT/JMLR/AISTATS/UAI strong; NeurIPS/ICML partial)."),
117+),
118+dict(
119+ id="deep_learning_architectures", number=8, name="Deep Learning Architectures", area="ai_ml",
120+ definition=("Design, training and compression of neural network architectures as such: CNNs, RNNs, transformers and "
121+ "attention, graph neural networks, state-space models, normalisation and optimisers in practice, neural "
122+ "architecture search, pruning, quantisation and distillation, neuromorphic and spiking networks as models. "
123+ "Task-specific results go to the application category; hardware goes to ai_accelerators."),
124+ topics=[("T10320", 1.0, "Neural Networks and Applications: training, architectures, activation functions"),
125+ ("T11273", 1.0, "Advanced Graph Neural Networks"),
126+ ("T10036", 0.5, "Advanced Neural Network Applications: YOLO, lightweight networks, model compression"),
127+ ("T12676", 0.5, "extreme learning machines"),
128+ ("T12611", 0.4, "reservoir computing / echo state networks"),
129+ ("T11612", 0.3, "DNN training optimisation"),
130+ ("T11347", 0.3, "neural network stability and synchronisation"),
131+ ("T11307", 0.3, "PEFT and continual learning for deep nets"),
132+ ("T12205", 0.2, "time-series forecasting with deep nets"),
133+ ("T10502", 0.2, "neuromorphic computing (spiking models)"),
134+ ("T11206", 0.2, "model reduction and neural networks")],
135+ subfields=[(1702, 0.3)],
136+ arxiv=[("cs.NE", 1.0), ("cs.LG", 0.5), ("cs.CV", 0.1), ("cs.AI", 0.1)],
137+ streams=[("conf/iclr", 0.5), ("conf/nips", 0.4), ("conf/icml", 0.4), ("journals/tnn", 0.6), ("journals/nn", 0.6),
138+ ("conf/ijcnn", 0.6), ("journals/ijon", 0.5), ("journals/nca", 0.5), ("conf/icann", 0.6), ("conf/iconip", 0.5),
139+ ("journals/npl", 0.5), ("conf/esann", 0.5), ("journals/tmlr", 0.3), ("journals/pami", 0.2),
140+ ("journals/cin", 0.3), ("journals/tetci", 0.3), ("journals/tai", 0.3), ("journals/ficn", 0.2),
141+ ("journals/cogcom", 0.2), ("journals/aisy", 0.1), ("conf/aaai", 0.1)],
142+ keywords=[
143+ (r"(?i)\b(deep|convolutional|recurrent|graph|spiking|artificial|feed-?forward|residual|siamese|capsule|physics-?informed|liquid|binary|quanti[sz]ed|Hopfield|Kolmogorov-?Arnold) neural networks?\b", 0.8, "both"),
144+ (r"(?i)\b(CNNs?|RNNs?|LSTMs?|GRUs?|GNNs?|GCNs?|GATs?|SNNs?|MLPs?|DNNs?|ANNs?|KANs?|ResNets?|DenseNet|EfficientNet|MobileNet|U-?Net|ViTs?|Swin)\b", 0.7, "both"),
145+ (r"(?i)\btransformers?\b", 0.5, "both"),
146+ (r"(?i)\b(self-?|cross-?|multi-?head |sparse |linear |flash)?attention (mechanisms?|heads?|layers?|maps?|weights)\b|\battention is all\b", 0.7, "both"),
147+ (r"(?i)\bneural architecture search\b|\bNAS\b.*\b(architecture|search space|supernet)\b|\bAutoML\b|\bdifferentiable architecture\b", 0.9, "both"),
148+ (r"(?i)\b(model compression|knowledge distillation|network pruning|structured pruning|unstructured pruning|post-?training quanti[sz]ation|quanti[sz]ation-?aware training|low-?rank (factori[sz]ation|approximation) of (weights|layers)|weight sharing|parameter sharing|lightweight (networks?|architectures?|models?)|efficient (architectures?|networks?|transformers?))\b", 0.8, "both"),
149+ (r"(?i)\b(batch|layer|group|instance|weight|spectral) normali[sz]ation\b|\bdropout\b|\bresidual (connections?|learning|blocks?)\b|\bskip connections?\b|\bactivation functions?\b|\bReLU\b|\bGELU\b|\bSwiGLU\b|\bsoftmax\b", 0.7, "both"),
150+ (r"(?i)\b(mixture[- ]of[- ]experts|MoE\b|state[- ]space models?|SSMs?\b|Mamba|S4\b|RWKV|Hyena|linear (RNNs?|transformers?)|positional (encodings?|embeddings?)|RoPE|rotary|hypernetworks?|neural ODEs?|implicit (layers|neural representations?)|capsule networks?|equivariant (networks?|neural)|geometric deep learning|message passing (neural|networks?)|graph (convolution|attention|transformers?|pooling)|over-?smoothing)\b", 0.9, "both"),
151+ (r"(?i)\b(backpropagation|back-?prop|gradient (vanishing|exploding)|learning rate (schedul|warm-?up)|Adam(W)?\b|optimi[sz]ers? for (deep|neural)|weight (decay|initiali[sz]ation)|training (dynamics|instabilit)|loss functions? for|curriculum learning|data augmentation|mixup|CutMix|label smoothing|early stopping|hyper-?parameter (tuning|optimi[sz]ation))\b", 0.7, "both"),
152+ (r"(?i)\bdeep learning\b|\bneural networks?\b|\bneural nets?\b", 0.5, "both"),
153+ (r"(?i)\b(spiking|neuromorphic|reservoir computing|echo state|extreme learning machines?|ELM\b)\b", 0.5, "both"),
154+ ],
155+ exclusions=[r"(?is)^(?!.*(neural|deep learning|attention|language model|vision|encoder|decoder)).*\b(power|distribution|current|voltage|instrument|high-?voltage|dry-?type|oil-?immersed|pulse|flyback|isolation|phase-?shifting|step-?(up|down)) transformers?\b"],
156+ related=["ml_theory", "computer_vision", "nlp", "ai_accelerators", "foundation_models", "llm"],
157+ parent_of=[], child_of=[],
158+ provenance=("T10320 and T11273 are architecture-centred OpenAlex topics; cs.NE is the arXiv home of neural and "
159+ "evolutionary computation. 'Transformer' alone is weak and electrical senses are vetoed when no "
160+ "learning term co-occurs."),
161+),
162+dict(
163+ id="ai_safety", number=38, name="AI Safety and Alignment", area="ai_ml",
164+ definition=("Making AI systems safe, aligned, robust, fair, interpretable and governable: alignment and preference "
165+ "learning framed as a safety problem, red-teaming and jailbreaks, adversarial robustness and backdoors, "
166+ "hallucination and truthfulness, bias and fairness, mechanistic interpretability, evaluation of dangerous "
167+ "capabilities, AI governance and regulation. Generic system security is cybersecurity; XAI as a statistical "
168+ "method only is ml_theory."),
169+ topics=[("T10883", 1.0, "Ethics and Social Impacts of AI: AI safety, alignment, fairness, governance"),
170+ ("T11689", 0.6, "adversarial robustness, backdoors, prompt injection"),
171+ ("T12026", 0.5, "explainable AI / mechanistic interpretability"),
172+ ("T13851", 0.4, "law, AI and intellectual property"),
173+ ("T12262", 0.3, "hate speech and cyberbullying detection (harm mitigation)"),
174+ ("T11147", 0.3, "misinformation and its impacts"),
175+ ("T10525", 0.2, "human-automation interaction and safety"),
176+ ("T14307", 0.1, "bioethics and human rights")],
177+ subfields=[],
178+ arxiv=[("cs.CY", 0.5), ("cs.AI", 0.2), ("cs.LG", 0.1), ("cs.CL", 0.1), ("cs.CR", 0.1)],
179+ streams=[("conf/fat", 1.0), ("conf/aies", 1.0), ("journals/aiethics", 1.0), ("conf/satml", 1.0),
180+ ("journals/ais", 0.5), ("journals/natmi", 0.2), ("conf/nips", 0.1), ("conf/iclr", 0.1), ("conf/icml", 0.1),
181+ ("journals/tai", 0.2), ("journals/dai", 0.1), ("conf/sp", 0.1), ("conf/ccs", 0.1)],
182+ keywords=[
183+ (r"(?i)\bAI (safety|alignment|governance|regulation|ethics|risks?|act|auditing|accountability|harms?|misuse|incidents?|oversight|assurance|red[- ]teaming)\b", 1.0, "both"),
184+ (r"(?i)\b(aligned|alignment of|align(ing)?|misaligned|misalignment of) (large )?(language models?|LLMs?|AI (systems?|models?)|foundation models?|agents?)\b|\bsuper-?alignment\b|\bvalue alignment\b|\bhuman values?\b", 1.0, "both"),
185+ (r"(?i)\b(jail-?break(s|ing)?|red[- ]teaming|prompt injection|backdoor (attacks?|defenses?)|data poisoning|model (stealing|extraction|inversion)|membership inference|adversarial (prompts?|suffixes?)|refusal|safety (training|fine-?tuning|guardrails?|alignment|benchmarks?|evaluations?)|guardrails?|harmful (content|outputs?|responses?|requests?)|unsafe (outputs?|behavio(u)?rs?))\b", 1.0, "both"),
186+ (r"(?i)\badversarial (examples?|attacks?|robustness|training|perturbations?|defenses?|vulnerabilit)\b|\bcertified robustness\b|\brobustness (certification|verification)\b", 0.8, "both"),
187+ (r"(?i)\b(algorithmic|AI|model|machine learning|ML|LLM|gender|racial|demographic|social) (fairness|bias(es)?|discrimination|transparency|accountability|audits?|auditing)\b|\bbias (mitigation|amplification|in (LLMs?|language models?|AI))\b|\bdebiasing\b", 1.0, "both"),
188+ (r"(?i)\b(responsible|trustworthy|ethical|safe|harmless|honest|helpful|human-?centered|human-?compatible|beneficial) (AI|artificial intelligence|machine learning|LLMs?|agents?)\b", 1.0, "both"),
189+ (r"(?i)\b(mechanistic interpretability|sparse auto-?encoders? (for|on) (LLM|language|features)|activation (steering|patching)|circuit discovery|probing classifiers?|representation engineering|superposition hypothesis|induction heads|feature visuali[sz]ation|concept (bottleneck|activation vectors?)|model editing|knowledge editing)\b", 1.0, "both"),
190+ (r"(?i)\b(hallucinations?|truthfulness|factuality|sycophancy|deceptive alignment|deception|reward hacking|specification gaming|goal misgeneralization|scalable oversight|weak-?to-?strong|constitutional AI|RLHF|preference (learning|optimi[sz]ation)|human feedback|over-?optimi[sz]ation|situational awareness|sandbagging|power-?seeking|instrumental convergence|corrigib)\b", 0.8, "both"),
191+ (r"(?i)\b(existential risks?|catastrophic risks?|x-?risk|frontier (AI|models?)|dangerous capabilit(y|ies)|misuse of AI|dual-?use|AGI (safety|risk|alignment)|superintelligen|loss of control|AI (doom|pause)|compute governance|responsible scaling|model (cards?|evaluations? for safety)|capability evaluations?|system cards?)\b", 1.0, "both"),
192+ (r"(?i)\b(toxic(ity)?|hate speech|harmful|offensive|abusive) (language|content|speech|text|generation|detection)\b|\bcontent moderation\b|\b(misinformation|disinformation|fake news) (detection|generation|and LLMs?)\b", 0.5, "both"),
193+ (r"(?i)\b(watermark(ing)?|detect(ing|ion) of|detecting) (LLM|AI|machine)-?generated (text|content|images?)\b|\bAI-?text detection\b", 0.8, "both"),
194+ (r"(?i)\b(machine unlearning|right to be forgotten|differential privacy for (LLM|language)|memori[sz]ation (in|of) (LLM|language models?)|training data extraction)\b", 0.5, "both"),
195+ (r"(?i)\b(uncertainty (quantification|estimation|calibration)|out-?of-?distribution detection|distribution shift|safe (reinforcement learning|RL|exploration)|runtime monitoring of (neural|ML)|verification of neural networks?)\b", 0.4, "both"),
196+ (r"(?i)\b(ethics|ethical|societal (impact|implications)|social (impact|implications)|regulation|policy|governance|accountability|transparency|explainability|interpretability|fairness|bias|safety|risk)\b", 0.2, "both"),
197+ ],
198+ exclusions=[r"(?i)\b(multiple sequence|read|genome|sequence|image|spinal|wheel|dental|teeth|occlusal|fib(er|re)|beam|optical|antenna|shaft|laser) alignment\b(?!.*(AI|LLM|language model|RLHF|human values))"],
199+ related=["llm", "ml_theory", "cybersecurity", "privacy_preserving_computation", "ai_agents", "hci", "reinforcement_learning"],
200+ parent_of=[], child_of=[],
201+ provenance=("T10883 (field 33, Social Sciences) explicitly lists 'AI safety', 'AI alignment', 'algorithmic fairness'; "
202+ "T11689 and T12026 host the technical robustness/interpretability halves. FAccT, AIES, SaTML and AI & Ethics "
203+ "are the dedicated venues; the generic ethics/governance words are kept far below threshold."),
204+),
205+]
added taxonomy/src/area_data_a.py +126 −0
@@ -0,0 +1,126 @@
1+"""Area data_distributed, part A: categories 20-22 (databases, information retrieval, distributed computing)."""
2+
3+CATEGORIES = [
4+dict(
5+ id="database_systems", number=20, name="Database Systems", area="data_distributed",
6+ definition=("Storage, querying and management of structured and semi-structured data: query processing and optimisation, "
7+ "transactions and concurrency control, indexing and storage engines, data integration, cleaning and provenance, "
8+ "big-data processing frameworks, data mining as a data-management problem, learned database components, "
9+ "vector and graph databases. Ranking of unstructured documents is information_retrieval."),
10+ topics=[("T10317", 1.0, "advanced database systems and queries"), ("T12761", 0.6, "data stream mining"),
11+ ("T11719", 0.6, "data quality and management"), ("T14280", 0.6, "big data technologies and applications"),
12+ ("T10538", 0.5, "data mining algorithms and applications"), ("T11181", 0.5, "advanced data storage technologies"),
13+ ("T12601", 0.5, "web applications and data management"), ("T11106", 0.5, "data management and algorithms"),
14+ ("T10772", 0.4, "distributed systems and fault tolerance (distributed databases, transactions)"),
15+ ("T13373", 0.4, "data mining and ML applications"), ("T11891", 0.4, "big data and business intelligence"),
16+ ("T10215", 0.4, "semantic web and ontologies (RDF, SPARQL)"), ("T12016", 0.4, "web data mining"),
17+ ("T14435", 0.4, "information retrieval and data mining"), ("T11937", 0.3, "research data management"),
18+ ("T13647", 0.3, "AI and big data applications"), ("T14347", 0.2, "big data and digital economy"),
19+ ("T10101", 0.2, "cloud computing (MapReduce, Hadoop)"), ("T10757", 0.2, "geographic information systems"),
20+ ("T13832", 0.2, "advanced decision-making techniques (DSS)"), ("T14484", 0.2, "technology and data analysis")],
21+ subfields=[(1710, 0.1)],
22+ arxiv=[("cs.DB", 1.0)],
23+ streams=[("conf/sigmod", 1.0), ("journals/pvldb", 1.0), ("journals/pacmmod", 1.0), ("conf/icde", 1.0), ("conf/edbt", 1.0),
24+ ("conf/pods", 1.0), ("journals/tods", 1.0), ("journals/vldb", 1.0), ("conf/dasfaa", 1.0), ("conf/vldb", 0.8),
25+ ("journals/tkde", 0.8), ("journals/tbd", 0.6), ("conf/apweb", 0.6), ("conf/icdm", 0.5), ("conf/pakdd", 0.5),
26+ ("journals/tkdd", 0.5), ("journals/kais", 0.5), ("conf/bigdataconf", 0.5), ("journals/jbd", 0.5), ("conf/gis", 0.5),
27+ ("conf/cikm", 0.4), ("conf/kdd", 0.4), ("journals/ijdsa", 0.4), ("journals/data", 0.4), ("conf/pkdd", 0.3),
28+ ("journals/www", 0.3), ("journals/tgis", 0.3), ("journals/gis", 0.3), ("journals/biodb", 0.3),
29+ ("conf/semweb", 0.3), ("conf/esws", 0.3), ("journals/dss", 0.2), ("journals/isf", 0.1), ("journals/tist", 0.2),
30+ ("journals/kbs", 0.1), ("conf/ksem", 0.2)],
31+ keywords=[
32+ (r"(?i)\bdatabases?\b|\bDBMSs?\b|\bRDBMSs?\b|\bdatabase (systems?|management|engines?|design|theory|queries|tuning|research)\b", 0.8, "both"),
33+ (r"(?i)\b(query (processing|optimi[sz]ation|execution|languages?|engines?|compilation|plans?|rewriting|answering over)|SQL\b|NoSQL|NewSQL|relational (model|algebra|databases?|queries|tables?)|transaction(al)? (processing|management|systems?)|ACID\b|OLTP|OLAP|HTAP|index(ing|es) (structures?|for|techniques)|B\+?-?trees?|learned (index(es)?|cardinality|query optimi[sz]ers?|database)|key-?value stores?|column(ar)? (stores?|storage|formats?)|data (warehous(e|es|ing)|lakes?|lakehouses?|integration|cleaning|quality|provenance|lineage|wrangling|catalogs?|cubes?|partitioning|versioning|discovery|marketplaces?)|schema (matching|evolution|mapping|inference)|entity (resolution|matching)|record linkage|join (algorithms?|processing|ordering|optimi[sz]ation)|cardinality estimation|selectivity estimation|data management|graph databases?|property graphs?|spatial (databases?|indexing|queries|joins?)|spatio-?temporal (databases?|queries|indexing)|time[- ]series databases?|vector (databases?|search engines?|indexes?)|approximate nearest neighbou?r search|similarity search|top-?k queries|skyline queries|query by example|data lakes?|metadata management|Datalog engines?)\b", 1.0, "both"),
34+ (r"(?i)\b(data mining|frequent (patterns?|itemsets?|subgraphs?)|association rules?|knowledge discovery in databases|KDD\b|sequential pattern mining|stream (processing|mining)|data streams?|sliding windows?|big data (analytics|processing|systems?|platforms?|frameworks?|management)|Hadoop|HDFS|Apache Spark|Spark SQL|MapReduce|Flink|Hive|Presto|Trino|Kafka|ETL\b|ELT\b|text-?to-?SQL|NL2SQL|natural language interfaces? to databases?|NLIDB|LLMs? for (data|databases?|SQL|tables?)|table (question answering|understanding|representation))\b", 0.7, "both"),
35+ (r"(?i)\b(PostgreSQL|Postgres|MySQL|SQLite|DuckDB|MongoDB|Cassandra|RocksDB|LevelDB|Neo4j|ClickHouse|Snowflake|BigQuery|Redshift|TiDB|CockroachDB|Spanner|DynamoDB|LSM-?trees?|storage engines?|buffer (pool|management)|concurrency control|MVCC|multi-?version|crash recovery|write-?ahead log(ging)?|WAL\b|checkpointing in databases|isolation levels?|serializab(le|ility)|snapshot isolation|two-?phase (locking|commit)|optimistic concurrency|deterministic databases?|in-?memory databases?|main-?memory databases?|disaggregated (storage|memory) for databases?|cloud(-?native)? databases?|database-?as-?a-?service|DBaaS|multi-?model databases?|time-?travel queries|temporal databases?)\b", 1.0, "both"),
36+ (r"(?i)\b(data|queries|query|tables?|records?|tuples?|schemas?|indexes|indices)\b", 0.2, "both"),
37+ ],
38+ exclusions=[r"(?is)^(?!.*(query processing|SQL|transaction|index(ing)?|storage engine|DBMS|data management|data integration|schema|NoSQL|query optimi)).*\b(PubMed|Embase|Scopus|Web of Science|Cochrane|MEDLINE|CINAHL|PsycINFO) (databases?|were searched|search)\b"],
39+ related=["information_retrieval", "distributed_computing", "cloud_computing", "algorithms", "ml_theory"],
40+ parent_of=[], child_of=[],
41+ provenance=("T10317 is the OpenAlex database topic; data-mining and big-data topics (1710, 1711, 18xx) shared at 0.4-0.6. "
42+ "SIGMOD/PVLDB/PACMMOD/ICDE/EDBT/PODS/TODS/VLDBJ pure; CIKM/KDD shared with IR and ML. Systematic-review "
43+ "mentions of bibliographic databases are vetoed when no DB-systems term co-occurs."),
44+),
45+dict(
46+ id="information_retrieval", number=21, name="Information Retrieval", area="data_distributed",
47+ definition=("Finding and ranking relevant information for a user need: search engines and ranking models, dense and "
48+ "sparse retrieval, recommender systems, query understanding, evaluation and test collections, web search and "
49+ "mining, scholarly and digital-library retrieval, retrieval components of RAG systems. Structured query "
50+ "processing is database_systems; text understanding tasks are nlp."),
51+ topics=[("T10286", 1.0, "information retrieval and search behavior"), ("T10203", 1.0, "recommender systems"),
52+ ("T13274", 0.8, "expert finding and Q&A systems"), ("T14435", 0.6, "information retrieval and data mining"),
53+ ("T13976", 0.5, "web visibility and informetrics"), ("T12016", 0.5, "web data mining and analysis"),
54+ ("T10627", 0.4, "advanced image and video retrieval"), ("T10824", 0.3, "image retrieval and classification"),
55+ ("T10028", 0.3, "topic modeling / RAG / sentence retrieval"), ("T10215", 0.3, "semantic web and ontologies"),
56+ ("T13673", 0.2, "library science and information"), ("T11937", 0.2, "research data management"),
57+ ("T11813", 0.2, "library collection development and digital resources"), ("T12863", 0.2, "web and library services"),
58+ ("T10664", 0.2, "sentiment analysis and opinion mining"), ("T11644", 0.2, "spam and phishing detection"),
59+ ("T14330", 0.2, "library science and information systems"), ("T13166", 0.2, "information science and libraries"),
60+ ("T13874", 0.2, "information architecture and usability"), ("T11122", 0.2, "online learning and analytics (recommendation)")],
61+ subfields=[(1710, 0.1)],
62+ arxiv=[("cs.IR", 1.0), ("cs.DL", 0.6), ("cs.SI", 0.3)],
63+ streams=[("conf/sigir", 1.0), ("conf/wsdm", 1.0), ("conf/ecir", 1.0), ("conf/recsys", 1.0), ("journals/tois", 1.0),
64+ ("journals/ipm", 0.8), ("conf/clef", 0.8), ("conf/jcdl", 0.8), ("conf/um", 0.8), ("conf/cikm", 0.6), ("conf/www", 0.6),
65+ ("journals/www", 0.6), ("journals/jasis", 0.6), ("conf/asist", 0.6), ("journals/scientometrics", 0.5),
66+ ("conf/semweb", 0.5), ("conf/esws", 0.5), ("conf/webi", 0.5), ("journals/kais", 0.3), ("conf/kdd", 0.3),
67+ ("conf/icwsm", 0.3), ("journals/tist", 0.3), ("journals/tkde", 0.2), ("journals/snam", 0.2), ("conf/asunam", 0.2),
68+ ("journals/tcss", 0.1), ("journals/ijinfoman", 0.2), ("journals/jkm", 0.1), ("journals/iam", 0.1)],
69+ keywords=[
70+ (r"(?i)\binformation retrieval\b|\bIR (systems?|models?|evaluation|research|community)\b|\bad[- ]hoc retrieval\b|\bretrieval (models?|systems?|effectiveness|augmented)\b", 1.0, "both"),
71+ (r"(?i)\b(search engines?|web search|enterprise search|product search|query (expansion|reformulation|understanding|suggestion|logs?|auto-?completion|intent)|ranking (models?|functions?|losses?|metrics)|learning[- ]to[- ]rank|LTR\b|dense (retrieval|passage retrieval|retrievers?)|sparse retrieval|lexical retrieval|hybrid retrieval|BM25|TF-?IDF|neural (retrieval|rankers?|IR|re-?ranking)|re-?rank(ing|ers?)|cross-?encoders?|bi-?encoders?|dual encoders?|ColBERT|late interaction|SPLADE|DPR\b|document (ranking|retrieval|expansion)|passage (ranking|retrieval)|relevance (judgments?|feedback|assessment|modeling)|click (models?|logs?|through)|conversational search|interactive (IR|retrieval)|session search|evaluation (campaigns?|measures?)|nDCG|MRR\b|mean reciprocal rank|MAP@|test collections?|TREC\b|MS ?MARCO|BEIR|pooling bias|query performance prediction|retrieval-?augmented|RAG (systems?|pipelines?)|retriever|generative (retrieval|recommendation))\b", 1.0, "both"),
72+ (r"(?i)\brecommend(er|ation) (systems?|models?|algorithms?|engines?|tasks?|lists?|quality)\b|\brecommenders?\b|\bcollaborative filtering\b|\b(sequential|session-?based|conversational|cross-?domain|cold-?start|news|music|point-?of-?interest|POI|job|group|bundle|LLM-?based|explainable|fair|multi-?stakeholder) recommendation\b|\bclick-?through rate\b|\bCTR prediction\b|\bimplicit feedback\b|\buser (modeling|preferences?|interests?) (for|in) recommendation\b|\bmatrix factori[sz]ation\b|\bitem (embeddings?|cold start)\b|\bMovieLens\b|\bNetflix Prize\b", 1.0, "both"),
73+ (r"(?i)\b(knowledge graphs?|entity linking|entity search|semantic web|linked (open )?data|SPARQL|RDF\b|ontolog(y|ies) (matching|alignment|engineering)|schema\.org|Wikidata|DBpedia|question answering over knowledge)\b", 0.5, "both"),
74+ (r"(?i)\b(digital librar(y|ies)|scholarly (search|recommendation|data|documents?|knowledge graphs?)|citation (analysis|networks?|recommendation|counts?|indexing)|bibliometric|scientometric|altmetric|h-?index|peer[- ]review (process|quality)|web (crawl(ing|ers?)|archiv(es?|ing)|mining)|social media (mining|analytics|retrieval)|hashtags?|microblog|user-?generated content|community question answering|CQA\b|expert finding|e-?commerce search|sponsored search|computational advertising|ad (auctions?|ranking))\b", 0.6, "both"),
75+ (r"(?i)\b(retrieval|search|ranking|recommendation|relevance|queries|query|users?' (information|search))\b", 0.3, "both"),
76+ ],
77+ exclusions=[],
78+ related=["nlp", "database_systems", "llm", "hci", "ml_theory"],
79+ parent_of=[], child_of=[],
80+ provenance=("T10286 (search behaviour) and T10203 (recommenders) are the OpenAlex IR hosts; library/information-science "
81+ "topics of 1710 included at 0.2 because they share venues (JASIST, ASIS&T). SIGIR/WSDM/ECIR/RecSys/TOIS pure."),
82+),
83+dict(
84+ id="distributed_computing", number=22, name="Distributed Computing", area="data_distributed",
85+ definition=("Computation across multiple processors or machines: distributed algorithms and consensus, fault tolerance "
86+ "and replication, parallel programming and HPC systems software, cluster and grid resource management, "
87+ "distributed storage and middleware, distributed and parallel ML training systems. Cloud service models are "
88+ "cloud_computing and edge placement is edge_computing (both children)."),
89+ topics=[("T10715", 1.0, "distributed and parallel computing systems (grid, cluster, scheduling)"),
90+ ("T10772", 1.0, "distributed systems and fault tolerance (consensus, replication, transactions)"),
91+ ("T10742", 0.8, "peer-to-peer network technologies"), ("T10054", 0.6, "parallel computing (HPC, MPI, GPU)"),
92+ ("T10829", 0.5, "interconnection networks and systems"), ("T11181", 0.5, "advanced data storage technologies"),
93+ ("T10101", 0.4, "cloud computing and resource management"), ("T12127", 0.4, "software system performance and reliability"),
94+ ("T10679", 0.3, "service-oriented architecture"), ("T10273", 0.3, "IoT and edge/fog computing"),
95+ ("T10249", 0.3, "distributed control multi-agent systems"), ("T12203", 0.3, "mobile agent-based network management"),
96+ ("T11986", 0.3, "scientific workflows"), ("T12879", 0.2, "distributed sensor networks and detection"),
97+ ("T12216", 0.2, "network time synchronization"), ("T11478", 0.2, "caching and content delivery")],
98+ subfields=[(1705, 0.1)],
99+ arxiv=[("cs.DC", 1.0), ("cs.PF", 0.5), ("cs.OS", 0.2), ("cs.NI", 0.1)],
100+ streams=[("journals/tpds", 1.0), ("journals/jpdc", 1.0), ("conf/ipps", 1.0), ("conf/podc", 1.0), ("conf/spaa", 1.0),
101+ ("conf/opodis", 1.0), ("conf/europar", 1.0), ("conf/ics", 1.0), ("conf/hpdc", 1.0), ("conf/icdcs", 1.0),
102+ ("conf/middleware", 1.0), ("conf/cluster", 1.0), ("conf/icpads", 1.0), ("conf/ica3pp", 1.0), ("conf/ispa", 1.0),
103+ ("conf/sc", 0.8), ("conf/ccgrid", 0.8), ("conf/hpcc", 0.8), ("conf/ppopp", 0.8), ("journals/concurrency", 0.8),
104+ ("journals/tjs", 0.8), ("journals/cluster", 0.8), ("journals/ijhpca", 0.8), ("journals/scpe", 0.8),
105+ ("journals/fgcs", 0.6), ("conf/eurosys", 0.6), ("journals/ppna", 0.6), ("conf/aina", 0.6), ("conf/nsdi", 0.5),
106+ ("conf/dsn", 0.5), ("conf/socc", 0.5), ("conf/cloud", 0.5), ("journals/tos", 0.5), ("conf/fast", 0.5),
107+ ("conf/osdi", 0.4), ("conf/sosp", 0.4), ("journals/tc", 0.4), ("journals/sp", 0.4), ("conf/mlsys", 0.4),
108+ ("conf/ucc", 0.4), ("journals/tcc", 0.3), ("journals/jsa", 0.3), ("journals/computing", 0.3),
109+ ("journals/tnsm", 0.2), ("conf/xsede", 0.4), ("conf/pact", 0.2), ("journals/taco", 0.1), ("journals/tnse", 0.2),
110+ ("journals/jnca", 0.2), ("conf/isca", 0.1), ("journals/ieicetd", 0.1)],
111+ keywords=[
112+ (r"(?i)\b(distributed|parallel|concurrent) (systems?|computing|algorithms?|programming|processing|storage|file systems?|transactions?|consensus|ledgers?|training|inference|machine learning|deep learning|graph processing|databases?|data processing|simulation|applications?|execution|runtime|scheduling|snapshots?|shared memory|memory|protocols?|optimi[sz]ation|SGD|stochastic)\b", 1.0, "both"),
113+ (r"(?i)\b(consensus (protocols?|algorithms?)|Paxos|Raft\b|Byzantine|fault[- ]toleran(t|ce)|replication|state machine replication|linearizab(le|ility)|serializab|eventual consistency|strong consistency|consistency models?|CRDTs?|conflict-?free replicated|vector clocks?|logical clocks?|leader election|gossip protocols?|epidemic protocols?|distributed (hash tables?|locks?|snapshots?|tracing|ledger|shared)|two-?phase commit|atomic (broadcast|commit)|failure detectors?|self-?stabili[sz]|population protocols?|LOCAL model|CONGEST|message complexity|round complexity|asynchronous (systems?|networks?|model))\b", 1.0, "both"),
114+ (r"(?i)\b(high[- ]performance computing|HPC\b|supercomput\w+|exascale|petascale|MPI\b|OpenMP|message passing|load balancing|job scheduling|workflow (scheduling|management|engines?)|resource (allocation|management|scheduling) (in|for|on|across) (clusters?|grids?|data ?cent(er|re)s?|HPC|supercomputers?)|scalab(le|ility) (to|on|across) \d[\d,]* (nodes|cores|GPUs|processors)|multi-?GPU|GPU clusters?|data[- ]parallel(ism)?|model[- ]parallel(ism)?|pipeline[- ]parallel(ism)?|tensor[- ]parallel(ism)?|expert parallelism|all-?reduce|all-?to-?all|parameter servers?|collective communication|NCCL|ZeRO\b|FSDP|DeepSpeed|Megatron|Horovod|gradient (synchroni[sz]ation|accumulation across)|stragglers?|checkpoint(ing)? (for|of|in) (training|HPC)|large-?scale training|training (at scale|on \d+ GPUs)|distributed (LLM|DNN) (training|inference|serving)|cluster schedul(ers?|ing)|Slurm|Kubernetes schedul|Borg|YARN|Mesos)\b", 1.0, "both"),
115+ (r"(?i)\b(middleware|publish[- ]subscribe|pub/sub|message (queues?|brokers?)|Kafka|RabbitMQ|actor model|service mesh|stream processing (engines?|systems?|frameworks?)|Apache (Spark|Flink|Storm|Beam|Hadoop)|MapReduce|Ray\b|Dask|dataflow (systems?|engines?)|serverless (workflows?|computing)|microservices? (architecture|performance|systems?)|RPC\b|remote procedure|gRPC|RDMA|interconnects?|InfiniBand|network-?on-?chip)\b", 0.7, "both"),
116+ (r"(?i)\b(peer-?to-?peer|P2P\b|overlay networks?|volunteer computing|grid computing|cluster computing|utility computing|desktop grid|BOINC|BitTorrent|DHTs?\b|Chord|Kademlia|gossip-?based)\b", 0.8, "both"),
117+ (r"(?i)\b(multi-?core|many-?core|multi-?threaded|multithreading|lock-?free|wait-?free|non-?blocking (data structures?|algorithms?)|transactional memory|synchroni[sz]ation primitives|mutex(es)?|spin-?locks?|barriers?|work[- ]stealing|parallel(i[sz]ation|ism|izing)|task (parallel|graphs?)|speedup|Amdahl|strong scaling|weak scaling|parallel efficiency|SIMD parallel|GPU (computing|programming|acceleration)|CUDA\b|OpenCL|heterogeneous computing)\b", 0.8, "both"),
118+ (r"(?i)\b(nodes?|clusters?|scalab(le|ility)|parallel|distributed|concurrency|replicas?|throughput|latency)\b", 0.2, "both"),
119+ ],
120+ exclusions=[r"(?is)^(?!.*(comput|system|algorithm|protocol|network|server|cluster|node|GPU|processor|storage|training|software)).*\b(distributed (generation|energy resources?|lag|fibre|fiber|acoustic|temperature|parameter|load|element|order|delay|control of (power|microgrid|vehicles?|robots?)|leadership|cognition|practice|manufacturing))\b"],
121+ related=["cloud_computing", "edge_computing", "computer_networks", "database_systems", "operating_systems", "scientific_computing", "blockchain", "federated_learning"],
122+ parent_of=["cloud_computing", "edge_computing"], child_of=[],
123+ provenance=("T10715 and T10772 (subfield 1705) are the OpenAlex hosts; T10054 (parallel/HPC) is shared with architecture and "
124+ "scientific computing. TPDS/JPDC/IPDPS/PODC/SPAA/Euro-Par/ICS/HPDC/ICDCS/Middleware are the pure venues; cs.DC suffices."),
125+),
126+]
added taxonomy/src/area_data_b.py +140 −0
Large diff (140 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/area_hardware_a.py +124 −0
Large diff (124 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/area_perception_a.py +137 −0
Large diff (137 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/area_perception_b.py +131 −0
Large diff (131 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/area_security_a.py +89 −0
Large diff (89 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/area_security_b.py +95 −0
Large diff (95 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/area_software_pl.py +138 −0
Large diff (138 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/area_theory.py +97 −0
Large diff (97 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/areas.py +40 −0
Large diff (40 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_structural_change.py +29 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere