SPB Git
spboucher.ai

spb/wp44_uqo

Public

WP44: corpus definition (A-001, A-002), structural-change toolkit + tests, bibliometrics/graph/embedding scripts, manuscript skeleton

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 3, 2026) parent 05d5bd4 Browse files

55 changed files +2,503 −14

55 changed files
Load all 55 files Unified Split
added agents/REGISTER.md +8 −0
@@ -0,0 +1,8 @@
1 +# Agent register — WP44 (who produced what, who verified it)
2 +
3 +| Date | Agent | Brief | Artefact(s) | Verified by | Status |
4 +|---|---|---|---|---|---|
5 +| 2026-11-02 | A (director, this session) | CLAUDE.md | RESEARCH_PLAN.md, protocol/PROTOCOL.md, reports/PHASE0_FEASIBILITY.md | PI checkpoint H-0 pending | delivered |
6 +| 2026-11-02/03 | B+C (acquisition & data engineering, this session) | RESEARCH_PLAN §2–3 | ingestion/*.py, ledgers on M3U96b, data/metadata/linkage_report.json | N (audit, Phase 8) pending | delivered |
7 +| 2026-11-03 | D (taxonomy architect, subagent, 2nd attempt) | agents/briefs/AGENT_D_TAXONOMY.md | taxonomy/ontology.yaml, topic_map.csv, ONTOLOGY.md, src/ | A + K (validation sample, Phase 2) | running |
8 +| 2026-11-03 | B-lit (reference acquisition, subagent) | agents/briefs/AGENT_LIT_REFERENCES.md | manuscript/references.bib, references_verification.csv, literature_synthesis/REFERENCE_MAP.md | M (compile) + N (re-verify sample) | running |
added agents/briefs/AGENT_LIT_REFERENCES.md +78 −0
@@ -0,0 +1,78 @@
1 +# Brief — reference acquisition for the WP44 manuscript (Agent B-lit, 2026-11-03)
2 +
3 +Goal: build the *verified* bibliography that the manuscript's methodological and historical parts need, BEFORE any
4 +analysis result exists. Every entry must be resolved to a real bibliographic record; nothing may be invented.
5 +
6 +Output files (only these):
7 +1. `manuscript/references.bib` — BibTeX, keys `surnameYYYYkeyword` (lower-case ASCII), one entry per work, fields
8 + title/author/year/journal-or-booktitle/volume/pages/doi (or `eprint`+`archiveprefix={arXiv}` for preprints, or `url`
9 + for software/datasets/reports), `note = {verified: <method> <date>}`.
10 +2. `manuscript/references_verification.csv` — columns: key, doi_or_arxiv, verified_via (crossref|openalex|arxiv|datacite|url),
11 + verified_title (as returned by the service), match (exact|minor|FAILED), note. Entries that FAIL verification must be
12 + removed from references.bib and kept in the CSV with match=FAILED.
13 +3. `literature_synthesis/REFERENCE_MAP.md` — for each manuscript part (I–XI, appendices), the list of keys it should cite
14 + and one line on why (what the reference supports).
15 +
16 +Verification method (mandatory, per entry): `curl -s "https://api.crossref.org/works/<DOI>?mailto=spbou4@icloud.com"` or
17 +`curl -s "https://api.openalex.org/works/https://doi.org/<DOI>"` (single lookups are free) or
18 +`curl -s "https://export.arxiv.org/api/query?id_list=<id>"`; compare the returned title with yours. Do not use web search
19 +results as verification. If you cannot verify an item, drop it.
20 +
21 +Scope (target 120–180 verified entries), grouped:
22 +A. Scientometric data sources: OpenAlex (Priem, Piwowar, Orr 2022 arXiv:2205.01833), DBLP (Ley 2002/2009), arXiv
23 + (Ginsparg), Semantic Scholar (Kinney et al. 2023 S2AG), Crossref, ACL Anthology, Microsoft Academic Graph (Wang et al.
24 + 2020), CiteSeerX, Papers with Code; OpenAlex topic classification (OpenAlex topic model documentation, 2024).
25 +B. Systematic mapping / review methodology: PRISMA 2020 (Page et al. 2021 BMJ), PRISMA-S, Kitchenham & Charters 2007
26 + guidelines (SE), Petersen et al. 2008/2015 systematic mapping, Tricco scoping reviews, living reviews, automation of
27 + screening (e.g., ASReview van de Schoot 2021), LLM-assisted screening evaluations 2023–2025.
28 +C. Bibliometric indicators: Waltman 2016 review; field-normalised indicators (Waltman & van Eck 2019; MNCS; FWCI);
29 + citation windows and age bias (Wang 2013 "Citation time window choice"); h-index (Hirsch 2005); Gini/citation
30 + inequality (Nielsen & Andersen 2021); interdisciplinarity (Rao–Stirling: Stirling 2007; Leinster & Cobbold 2012;
31 + Porter & Rafols 2009; Leydesdorff); co-authorship networks (Newman 2001, 2004); country/institution analyses (Wagner
32 + 2017 international collaboration); preprints in CS (Lin et al. 2020 arXiv; Fraser 2021 bioRxiv analogies); citation
33 + advantage of OA (Piwowar 2018).
34 +D. Science mapping & embeddings: VOSviewer (van Eck & Waltman 2010), Small 1973 co-citation, Kessler bibliographic
35 + coupling, Boyack & Klavans 2010 accuracy of clustering, Leiden algorithm (Traag 2019), Louvain (Blondel 2008), SPECTER
36 + (Cohan et al. 2020), SPECTER2 (Singh et al. 2023 SciRepEval), SciNCL (Ostendorff 2022), SciBERT (Beltagy 2019),
37 + sentence-BERT (Reimers 2019), UMAP (McInnes 2018), HDBSCAN (Campello 2013; McInnes 2017), BERTopic (Grootendorst 2022),
38 + LDA (Blei 2003), dynamic topic models (Blei & Lafferty 2006), topic stability (Greene 2014).
39 +E. Structural change & time series: Bai & Perron 1998, 2003; Andrews 1993; Killick, Fearnhead & Eckley 2012 (PELT);
40 + Adams & MacKay 2007 (BOCPD); Truong, Oudre & Vayatis 2020 (ruptures review); Newey & West 1987; interrupted time series
41 + (Bernal, Cummins, Gasparrini 2017); synthetic control (Abadie 2010); event-study pitfalls.
42 +F. Emerging topic / technology detection: Rotolo, Hicks & Martin 2015; Small, Boyack & Klavans 2014; Cozzens et al.
43 + 2010; Wang Q. 2018 "A bibliometric model for identifying emerging research topics"; Klavans & Boyack 2017 research
44 + fronts; sleeping beauties (Ke, Ferrara, Radicchi, Flammini 2015); disruption index (Funk & Owen-Smith 2017; Wu, Wang,
45 + Evans 2019; Park, Leahey, Funk 2023 "Papers and patents are becoming less disruptive"); novelty (Uzzi et al. 2013);
46 + burst detection (Kleinberg 2003).
47 +G. Historical/contextual works on the AI transformation (only landmark, verified): Transformer (Vaswani 2017), BERT
48 + (Devlin 2019), GPT-3 (Brown 2020), scaling laws (Kaplan 2020; Hoffmann 2022), InstructGPT (Ouyang 2022), ChatGPT
49 + release (OpenAI blog 2022-11-30 — cite as web page with access date), GPT-4 technical report (2023), Llama 1/2/3
50 + (Touvron 2023; Dubey 2024), diffusion (Ho 2020; Rombach 2022), CLIP (Radford 2021), ViT (Dosovitskiy 2021), NeRF
51 + (Mildenhall 2020), RLHF (Christiano 2017), chain-of-thought (Wei 2022), RAG (Lewis 2020), ReAct (Yao 2023), Toolformer
52 + (Schick 2023), Mamba (Gu & Dao 2023), MoE Switch (Fedus 2022), DeepSeek-R1 (2025), o1 system card (2024), AlphaFold 2
53 + (Jumper 2021), AlphaGo/AlphaZero, foundation-model report (Bommasani 2021), "Sparks of AGI" (Bubeck 2023), stochastic
54 + parrots (Bender 2021), Stanford AI Index 2024/2025 reports, Epoch AI compute trends (Sevilla 2022).
55 +H. Evidence on LLM effects on science: Liang et al. 2024 (LLM-modified content in reviews/papers), Kobak et al. 2024/2025
56 + ("delving"), Geng & Trotta 2024, Bin-Nashwan / Dwivedi 2023 "So what if ChatGPT wrote it?", Birhane 2023, Messeri &
57 + Crockett 2024 (illusions of understanding), Nature surveys on AI use by researchers (Van Noorden 2023), arXiv
58 + submission growth analyses, "AI research is growing faster than…" (Krenn et al. 2023 Nature Machine Intelligence
59 + forecasting AI research), Fortunato et al. 2018 Science of science, Wang & Barabási 2021 book.
60 +I. Reproducibility/open science in CS: Gundersen & Kjensmo 2018; Pineau et al. 2021 (ML reproducibility program);
61 + Kapoor & Narayanan 2023 (leakage); Papers with Code / Hugging Face ecosystem studies; benchmark saturation (Ott et al.
62 + 2022 "Mapping global dynamics of benchmark creation and saturation"; Dehghani 2021 benchmark lottery; Raji 2021 AI and
63 + the everything in the whole wide world benchmark); Sculley 2018; ACM artifact badging policy (web).
64 +J. Other CS areas' survey anchors (one or two verified surveys each, 2022–2026 preferred): quantum computing (Preskill
65 + 2018 NISQ; a 2023–2025 survey), blockchain, federated learning (McMahan 2017; Kairouz 2021), differential privacy
66 + (Dwork 2006/2014), post-quantum cryptography (NIST FIPS 203/204/205 2024), formal verification (Lean mathlib / Rust
67 + verification surveys), computer architecture (Hennessy & Patterson 2019 Turing lecture; "golden age"), AI accelerators
68 + surveys, sustainable computing (Patterson 2021 carbon; Strubell 2019; Luccioni 2023), edge computing (Shi 2016), robotics
69 + foundation models (RT-2 2023; survey), AI for science (Wang H. et al. 2023 Nature "Scientific discovery in the age of
70 + AI"), AI safety/alignment surveys (Hendrycks 2021; Ji et al. 2023 alignment survey), agents surveys (Wang L. et al. 2024;
71 + Xi et al. 2023), LLM surveys (Zhao et al. 2023), multimodal surveys, speech (Whisper 2023), information retrieval
72 + (dense retrieval Karpukhin 2020), databases (vector databases survey 2024), compilers (MLIR Lattner 2021), programming
73 + languages (Rust safety), operating systems (eBPF / unikernels / LLM-OS), networks (programmable networks / P4), security
74 + (LLM security surveys 2024; prompt injection Greshake 2023), cryptography (FHE surveys), computational biology
75 + (AlphaFold 3 2024; ESM-2 2023), graphics (Gaussian splatting Kerbl 2023), HCI (Generative AI & HCI 2023–2024 surveys).
76 +
77 +Rules: work through the groups in order; verify as you go; keep each file write small (append to the .bib in batches of
78 +~20 entries with `cat >>`). Report back ≤ 25 lines: counts per group, failures, and anything you could not find.
added bibliometrics/01_volume_series.py +36 −0
@@ -0,0 +1,36 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — RQ1 volume series from corpus_works (eligible, CS core) with indexing-lag diagnostics.
3 +Outputs (results/bibliometrics/): volume_monthly.csv (first_month × type × membership), volume_yearly.csv,
4 +indexing_lag.csv (publication month → distribution of created_date − publication_date), penumbra_monthly.csv,
5 +dblp_only_yearly.csv. All tables are regenerated from the DuckDB database; nothing is hand-edited."""
6 +import argparse, os, duckdb
7 +
8 +
9 +def main():
10 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
11 + out = os.path.join(R, "results/bibliometrics"); os.makedirs(out, exist_ok=True)
12 + con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb"), read_only=True); con.execute("SET threads TO 16;")
13 + con.execute(f"""COPY (SELECT first_month, period, type, in_dblp, cs_primary, (arxiv_id_linked IS NOT NULL) AS has_arxiv, count(*) AS n,
14 + sum(has_abstract::int) AS n_abstract, sum((n_refs>0)::int) AS n_with_refs, sum(is_oa::int) AS n_oa
15 + FROM corpus_works WHERE eligible GROUP BY ALL ORDER BY 1,2,3) TO '{out}/volume_monthly.csv' (HEADER)""")
16 + con.execute(f"""COPY (SELECT publication_year AS year, count(*) AS n, sum(in_dblp::int) AS n_dblp, sum(cs_primary::int) AS n_cs_primary,
17 + sum((arxiv_id_linked IS NOT NULL)::int) AS n_arxiv, sum(in_acl::int) AS n_acl, sum(has_abstract::int) AS n_abstract,
18 + sum(is_oa::int) AS n_oa, sum((type='preprint')::int) AS n_preprint, sum((type='conference-paper')::int) AS n_conf,
19 + sum((type='article')::int) AS n_article, sum(e6_retracted::int) AS n_retracted, sum(xpac_kept_dblp::int) AS n_xpac_dblp,
20 + round(avg(n_authors),2) AS mean_authors, round(avg(n_refs),2) AS mean_refs, round(avg(n_countries),3) AS mean_countries
21 + FROM corpus_works WHERE eligible GROUP BY 1 ORDER BY 1) TO '{out}/volume_yearly.csv' (HEADER)""")
22 + con.execute(f"""COPY (SELECT substr(publication_date,1,7) AS pub_month, count(*) AS n,
23 + quantile_cont(datediff('day', publication_date::DATE, created_date::DATE), 0.5) AS lag_days_p50,
24 + quantile_cont(datediff('day', publication_date::DATE, created_date::DATE), 0.9) AS lag_days_p90,
25 + sum((datediff('day', publication_date::DATE, created_date::DATE) <= 90)::int) AS n_indexed_within_90d,
26 + sum((created_date::DATE > publication_date::DATE + INTERVAL 180 DAY)::int) AS n_indexed_after_180d
27 + FROM corpus_works WHERE eligible AND created_date IS NOT NULL AND publication_date IS NOT NULL GROUP BY 1 ORDER BY 1) TO '{out}/indexing_lag.csv' (HEADER)""")
28 + con.execute(f"""COPY (SELECT first_month, count(*) AS n_penumbra FROM corpus_works WHERE cs_penumbra AND NOT e1_outside_window AND NOT e2_type AND NOT e3_xpac AND NOT e4_no_title AND NOT e5_duplicate
29 + GROUP BY 1 ORDER BY 1) TO '{out}/penumbra_monthly.csv' (HEADER)""")
30 + con.execute(f"COPY (SELECT * FROM dblp_only_counts) TO '{out}/dblp_only_yearly.csv' (HEADER)")
31 + for f in ["volume_yearly.csv", "indexing_lag.csv"]:
32 + print(f, open(os.path.join(out, f)).read()[:1500])
33 +
34 +
35 +if __name__ == "__main__":
36 + main()
added bibliometrics/02_collaboration.py +42 −0
@@ -0,0 +1,42 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — RQ6 collaboration, institutions, countries (eligible corpus), yearly and by period.
3 +Outputs (results/bibliometrics/): team_size_yearly.csv, countries_yearly.csv (top-N country output + share, with
4 +fractional counting), international_collab_yearly.csv, institution_types_yearly.csv (education/company/government/…),
5 +top_institutions_by_period.csv (full and fractional counts), country_pairs_by_period.csv (edges for the collaboration
6 +network, ≥ 50 joint works), top_sources_by_period.csv (venue concentration inputs), venue_hhi_yearly.csv."""
7 +import argparse, os, duckdb
8 +
9 +
10 +def main():
11 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
12 + out = os.path.join(R, "results/bibliometrics"); os.makedirs(out, exist_ok=True)
13 + con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb"), read_only=True); con.execute("SET threads TO 20; SET memory_limit='60GB';")
14 + con.execute("CREATE OR REPLACE TEMP VIEW E AS SELECT * FROM corpus_works WHERE eligible")
15 + con.execute(f"""COPY (SELECT publication_year AS year, period, count(*) n, round(avg(n_authors),3) mean_authors, quantile_cont(n_authors,0.5) med_authors,
16 + sum((n_authors=1)::int) single, sum((n_authors>=5)::int) five_plus, sum((n_authors>=10)::int) ten_plus, sum((n_countries>=2)::int) intl,
17 + sum((n_institutions>=2)::int) multi_inst, sum((n_countries=0)::int) no_country FROM E GROUP BY 1,2 ORDER BY 1) TO '{out}/team_size_yearly.csv' (HEADER)""")
18 + # country output: full counting (work counted once per country) and fractional (1/n_countries)
19 + con.execute("""CREATE OR REPLACE TEMP TABLE wc AS SELECT w.id, w.publication_year AS year, w.period, unnest(w.countries) AS country, 1.0/len(w.countries) AS frac FROM E w WHERE len(w.countries) > 0""")
20 + con.execute(f"""COPY (SELECT year, country, count(*) n_full, round(sum(frac),1) n_frac FROM wc GROUP BY 1,2 ORDER BY 1, 3 DESC) TO '{out}/countries_yearly.csv' (HEADER)""")
21 + con.execute(f"""COPY (SELECT year, count(*) n, sum((n_countries>=2)::int) n_intl, round(sum((n_countries>=2)::int)*1.0/count(*),4) share_intl FROM E WHERE n_countries>0 GROUP BY 1 ORDER BY 1) TO '{out}/international_collab_yearly.csv' (HEADER)""")
22 + con.execute("""CREATE OR REPLACE TEMP TABLE wi AS SELECT w.id, w.publication_year AS year, w.period, unnest(w.institution_types) AS itype FROM E w WHERE len(w.institution_types) > 0""")
23 + con.execute(f"""COPY (SELECT year, itype, count(*) n FROM wi GROUP BY 1,2 ORDER BY 1,3 DESC) TO '{out}/institution_types_yearly.csv' (HEADER)""")
24 + con.execute(f"""COPY (SELECT year, count(DISTINCT id) n_with_inst, count(DISTINCT CASE WHEN itype='company' THEN id END) n_company,
25 + round(count(DISTINCT CASE WHEN itype='company' THEN id END)*1.0/count(DISTINCT id),4) share_company FROM wi GROUP BY 1 ORDER BY 1) TO '{out}/industry_share_yearly.csv' (HEADER)""")
26 + # institutions (from authorships): full and fractional (by distinct institutions per work)
27 + con.execute("""CREATE OR REPLACE TEMP TABLE winst AS
28 + SELECT DISTINCT a.work_id, unnest(a.institution_ids) AS inst_id FROM oa_authorships a WHERE a.work_id IN (SELECT id FROM E) AND len(a.institution_ids) > 0""")
29 + con.execute("""CREATE OR REPLACE TEMP TABLE winst2 AS SELECT w.work_id, w.inst_id, 1.0/count(*) OVER (PARTITION BY w.work_id) AS frac, e.period, e.publication_year AS year FROM winst w JOIN E e ON e.id = w.work_id""")
30 + con.execute(f"""COPY (SELECT period, inst_id, count(*) n_full, round(sum(frac),1) n_frac FROM winst2 GROUP BY 1,2 HAVING count(*) >= 200 ORDER BY 1, 3 DESC) TO '{out}/top_institutions_by_period.csv' (HEADER)""")
31 + con.execute(f"""COPY (SELECT a.period, a.country AS c1, b.country AS c2, count(DISTINCT a.id) n FROM wc a JOIN wc b ON a.id = b.id AND a.country < b.country
32 + GROUP BY 1,2,3 HAVING count(DISTINCT a.id) >= 50 ORDER BY 1, 4 DESC) TO '{out}/country_pairs_by_period.csv' (HEADER)""")
33 + con.execute(f"""COPY (SELECT period, source_id, any_value(source_name) source_name, any_value(source_type) source_type, count(*) n FROM E WHERE source_id IS NOT NULL
34 + GROUP BY 1,2 HAVING count(*) >= 100 ORDER BY 1, 5 DESC) TO '{out}/top_sources_by_period.csv' (HEADER)""")
35 + con.execute(f"""COPY (SELECT year, count(*) n, count(DISTINCT source_id) n_sources, round(sum(s*s),6) hhi, round(max(s),5) top1_share FROM
36 + (SELECT publication_year AS year, source_id, count(*)*1.0/sum(count(*)) OVER (PARTITION BY publication_year) AS s FROM E WHERE source_id IS NOT NULL GROUP BY 1,2)
37 + GROUP BY 1 ORDER BY 1) TO '{out}/venue_hhi_yearly.csv' (HEADER)""")
38 + print(open(f"{out}/team_size_yearly.csv").read()[:800]); print(open(f"{out}/industry_share_yearly.csv").read())
39 +
40 +
41 +if __name__ == "__main__":
42 + main()
added bibliometrics/03_citation_impact.py +44 −0
@@ -0,0 +1,44 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — RQ3/RQ4 citation impact with explicit citation-window control.
3 +
4 +From oa_counts_by_year (citations received per calendar year) we build fixed-window counts:
5 + c12 = citations received in the publication year and the following year (≈ 0–24 months), c36 = pub year + 2 years.
6 +Normalisation: NCS_k = c_k / mean(c_k | same publication year, same type class, same primary subfield) over the eligible
7 +corpus (field-normalised within OpenAlex subfield; category-level normalisation is added in 05_category_impact.py).
8 +Outputs: results/bibliometrics/citation_windows_yearly.csv (means, medians, shares uncited, Gini of c36 by year),
9 +work_citation_windows.parquet (work-level c12/c36/NCS for downstream scripts), top_cited_by_year.csv (top 50 per year by c36),
10 +fwci_summary_yearly.csv (OpenAlex FWCI as a secondary indicator)."""
11 +import argparse, os, duckdb
12 +
13 +
14 +def main():
15 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
16 + out = os.path.join(R, "results/bibliometrics"); os.makedirs(out, exist_ok=True)
17 + con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb")); con.execute("SET threads TO 20; SET memory_limit='60GB';")
18 + con.execute("""CREATE OR REPLACE TABLE work_citation_windows AS
19 + WITH E AS (SELECT id, publication_year AS y, type, primary_subfield_id AS sf, cited_by_count, fwci FROM corpus_works WHERE eligible),
20 + C AS (SELECT c.work_id, sum(CASE WHEN c.year <= e.y + 1 THEN c.cited_by_count ELSE 0 END) AS c12,
21 + sum(CASE WHEN c.year <= e.y + 2 THEN c.cited_by_count ELSE 0 END) AS c36,
22 + sum(CASE WHEN c.year <= e.y + 3 THEN c.cited_by_count ELSE 0 END) AS c48
23 + FROM oa_counts_by_year c JOIN E e ON e.id = c.work_id GROUP BY 1)
24 + SELECT e.id AS work_id, e.y AS year, e.type, e.sf, e.cited_by_count, e.fwci, coalesce(c.c12,0) c12, coalesce(c.c36,0) c36, coalesce(c.c48,0) c48
25 + FROM E e LEFT JOIN C c ON c.work_id = e.id""")
26 + con.execute("""CREATE OR REPLACE TABLE work_citation_windows2 AS
27 + SELECT w.*, CASE WHEN m12 > 0 THEN c12 / m12 END AS ncs12, CASE WHEN m36 > 0 THEN c36 / m36 END AS ncs36
28 + FROM work_citation_windows w JOIN (SELECT year, type, sf, avg(c12) m12, avg(c36) m36, count(*) n_ref FROM work_citation_windows GROUP BY 1,2,3) r
29 + USING (year, type, sf)""")
30 + con.execute("DROP TABLE work_citation_windows; ALTER TABLE work_citation_windows2 RENAME TO work_citation_windows")
31 + con.execute(f"COPY work_citation_windows TO '{R}/data/processed/corpus/work_citation_windows.parquet' (FORMAT PARQUET, COMPRESSION ZSTD)")
32 + con.execute(f"""COPY (SELECT year, count(*) n, round(avg(c12),3) mean_c12, round(avg(c36),3) mean_c36, quantile_cont(c36,0.5) med_c36,
33 + round(sum((c36=0)::int)*1.0/count(*),4) share_uncited_36, round(sum((c12=0)::int)*1.0/count(*),4) share_uncited_12,
34 + quantile_cont(c36,0.99) p99_c36, max(c36) max_c36,
35 + round((2*sum(c36*rk) / (count(*)*sum(c36)) - (count(*)+1.0)/count(*)),4) gini_c36,
36 + round(avg(fwci),3) mean_fwci, round(quantile_cont(fwci,0.5),3) med_fwci
37 + FROM (SELECT *, row_number() OVER (PARTITION BY year ORDER BY c36) rk FROM work_citation_windows) GROUP BY 1 ORDER BY 1) TO '{out}/citation_windows_yearly.csv' (HEADER)""")
38 + con.execute(f"""COPY (SELECT * FROM (SELECT w.year, w.work_id, c.title, c.source_name, c.type, w.c12, w.c36, w.cited_by_count, w.fwci, w.ncs36,
39 + row_number() OVER (PARTITION BY w.year ORDER BY w.c36 DESC, w.cited_by_count DESC) rk FROM work_citation_windows w JOIN corpus_works c ON c.id = w.work_id) WHERE rk <= 50 ORDER BY year, rk) TO '{out}/top_cited_by_year.csv' (HEADER)""")
40 + print(open(f"{out}/citation_windows_yearly.csv").read())
41 +
42 +
43 +if __name__ == "__main__":
44 + main()
added data/metadata/linkage_report.json +45 −0
@@ -0,0 +1,45 @@
1 +{
2 + "created": "2026-11-03T20:50:56.182276+00:00",
3 + "oa_cs_records": 15490159,
4 + "oa_cs_distinct_ids": 15490159,
5 + "dblp_2018_2026": 4347365,
6 + "dblp_with_doi": 4052326,
7 + "dblp_with_arxiv": 0,
8 + "link_dblp_oa": {
9 + "title_year": 179990,
10 + "doi": 2562431
11 + },
12 + "dblp_linked_distinct": 2736903,
13 + "dblp_unlinked": 1610462,
14 + "dblp_unlinked_by_type": {
15 + "phdthesis": 26841,
16 + "incollection": 3362,
17 + "inproceedings": 477235,
18 + "book": 1398,
19 + "article": 1084640,
20 + "data": 16986
21 + },
22 + "dblp_unlinked_with_doi_in_slim": 1077407,
23 + "arxiv_cs_records": 915115,
24 + "arxiv_created_2018_2026": 890852,
25 + "link_arxiv_oa": {
26 + "arxiv_id": 792672,
27 + "doi": 520,
28 + "title_year": 2143
29 + },
30 + "arxiv_linked_distinct": 744680,
31 + "acl_records": 135558,
32 + "acl_2018_2026": 85625,
33 + "link_acl_oa": {
34 + "doi": 57474,
35 + "title_year": 14904
36 + },
37 + "acl_linked_distinct_2018_2026": 72223,
38 + "needed_oa_ids_second_pass": {
39 + "dblp_doi": 1077406,
40 + "arxiv_doi": 39200
41 + },
42 + "slim_total": 464765485,
43 + "slim_cs_any_2018_2026": 15490159,
44 + "seconds": 10
45 +}
\ No newline at end of file
added econometrics/structural_change.py +135 −0
@@ -0,0 +1,135 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — structural-change toolkit for monthly publication series (RQ2, hypotheses H1/H3).
3 +
4 +Functions (all pure, tested in tests/test_structural_change.py):
5 + its_fit(y, dates, event, lags) interrupted time series on log counts with month-of-year seasonality,
6 + level and slope change at `event`, Newey–West HAC standard errors
7 + placebo_breaks(y, dates, window) distribution of the ITS level/slope t-statistics over every candidate month of the
8 + baseline (placebo), returns the empirical rank/p-value of the true event
9 + bai_perron(y, dates, max_breaks) multiple structural breaks in mean + trend by dynamic programming (BIC), with
10 + minimum segment length h (Bai & Perron 1998/2003, pure-python DP on SSR)
11 + pelt(y, pen) PELT changepoints (ruptures, l2 cost) on log counts, penalty by BIC
12 + bocpd(y, hazard, mu0, kappa0, ...) Bayesian online changepoint detection (Adams & MacKay 2007), Student-t predictive,
13 + returns run-length posterior and P(changepoint at t)
14 +Right-truncation: callers must stop the series at the last complete month (see bibliometrics/01_volume_series.py).
15 +"""
16 +from __future__ import annotations
17 +import numpy as np, pandas as pd
18 +import statsmodels.api as sm
19 +
20 +
21 +def _design(dates: pd.DatetimeIndex, event: pd.Timestamp, seasonal=True):
22 + t = np.arange(len(dates), dtype=float)
23 + post = (dates >= event).astype(float)
24 + t_post = np.where(post == 1, t - t[post == 1][0] if post.sum() else 0, 0.0)
25 + X = {"const": np.ones(len(t)), "t": t, "post": post, "t_post": t_post}
26 + if seasonal:
27 + for m in range(2, 13):
28 + X[f"m{m}"] = (dates.month == m).astype(float)
29 + return pd.DataFrame(X, index=dates)
30 +
31 +
32 +def its_fit(y, dates, event, lags=6, seasonal=True, log=True):
33 + dates = pd.DatetimeIndex(dates); event = pd.Timestamp(event)
34 + yy = np.log(np.asarray(y, dtype=float)) if log else np.asarray(y, dtype=float)
35 + X = _design(dates, event, seasonal)
36 + res = sm.OLS(yy, X).fit(cov_type="HAC", cov_kwds={"maxlags": lags})
37 + out = {"level_change": res.params["post"], "level_se": res.bse["post"], "level_t": res.tvalues["post"], "level_p": res.pvalues["post"],
38 + "slope_change": res.params["t_post"], "slope_se": res.bse["t_post"], "slope_t": res.tvalues["t_post"], "slope_p": res.pvalues["t_post"],
39 + "pre_slope": res.params["t"], "pre_slope_se": res.bse["t"], "n": int(res.nobs), "r2": res.rsquared, "event": str(event.date())}
40 + return out, res
41 +
42 +
43 +def placebo_breaks(y, dates, true_event, min_pre=24, min_post=12, lags=6, seasonal=True):
44 + dates = pd.DatetimeIndex(dates)
45 + rows = []
46 + for i in range(min_pre, len(dates) - min_post):
47 + ev = dates[i]
48 + o, _ = its_fit(y, dates, ev, lags, seasonal)
49 + rows.append({"event": ev, "level_t": o["level_t"], "slope_t": o["slope_t"], "level_change": o["level_change"], "slope_change": o["slope_change"]})
50 + df = pd.DataFrame(rows)
51 + tv = df.loc[df.event == pd.Timestamp(true_event)]
52 + out = {"n_placebo": len(df)}
53 + if len(tv):
54 + for k in ("level_t", "slope_t"):
55 + v = float(tv[k].iloc[0]); out[f"{k}_true"] = v
56 + out[f"{k}_placebo_p"] = float((np.abs(df[k]) >= abs(v)).mean())
57 + out[f"{k}_rank"] = int((np.abs(df[k]) >= abs(v)).sum())
58 + return out, df
59 +
60 +
61 +def bai_perron(y, dates=None, max_breaks=3, h=None, trend=True):
62 + """Global minimiser of SSR with m breaks (Bai & Perron 2003 dynamic programming) for a segment-wise linear trend
63 + (or mean) model; model selection by BIC and by the LWZ criterion. Returns dict with breaks (indices and dates)."""
64 + y = np.asarray(y, dtype=float); n = len(y); h = h or max(8, int(0.15 * n))
65 + t = np.arange(n, dtype=float)
66 + def ssr(i, j): # segment [i, j)
67 + yy = y[i:j]
68 + if trend:
69 + X = np.column_stack([np.ones(j - i), t[i:j]])
70 + beta, *_ = np.linalg.lstsq(X, yy, rcond=None); r = yy - X @ beta
71 + else:
72 + r = yy - yy.mean()
73 + return float(r @ r)
74 + S = np.full((n + 1, n + 1), np.inf)
75 + for i in range(n):
76 + for j in range(i + h, n + 1):
77 + S[i, j] = ssr(i, j)
78 + best = {}
79 + k_params = 2 if trend else 1
80 + # DP: C[m][j] = min SSR of first j obs with m breaks
81 + C = {0: S[0, :].copy()}; B = {0: {j: [] for j in range(n + 1)}}
82 + for m in range(1, max_breaks + 1):
83 + C[m] = np.full(n + 1, np.inf); B[m] = {}
84 + for j in range((m + 1) * h, n + 1):
85 + cands = [(C[m - 1][i] + S[i, j], i) for i in range(m * h, j - h + 1) if np.isfinite(C[m - 1][i]) and np.isfinite(S[i, j])]
86 + if cands:
87 + v, i = min(cands); C[m][j] = v; B[m][j] = B[m - 1][i] + [i]
88 + for m in range(0, max_breaks + 1):
89 + tot = C[m][n]
90 + if not np.isfinite(tot):
91 + continue
92 + p = (m + 1) * k_params + m
93 + bic = n * np.log(tot / n) + p * np.log(n)
94 + lwz = n * np.log(tot / (n - p)) + p * 0.299 * np.log(n) ** 2.1
95 + best[m] = {"ssr": tot, "bic": bic, "lwz": lwz, "breaks": B[m].get(n, [])}
96 + m_bic = min(best, key=lambda m: best[m]["bic"]); m_lwz = min(best, key=lambda m: best[m]["lwz"])
97 + res = {"by_m": best, "m_bic": m_bic, "m_lwz": m_lwz, "breaks_bic": best[m_bic]["breaks"], "h": h}
98 + if dates is not None:
99 + d = pd.DatetimeIndex(dates); res["break_dates_bic"] = [str(d[i].date()) for i in best[m_bic]["breaks"]]
100 + res["break_dates_lwz"] = [str(d[i].date()) for i in best[m_lwz]["breaks"]]
101 + return res
102 +
103 +
104 +def pelt(y, pen=None, min_size=6, model="l2"):
105 + import ruptures as rpt
106 + y = np.asarray(y, dtype=float); n = len(y)
107 + pen = pen if pen is not None else 2 * np.log(n) * np.var(y) # BIC-type penalty for l2 cost
108 + algo = rpt.Pelt(model=model, min_size=min_size, jump=1).fit(y.reshape(-1, 1))
109 + return algo.predict(pen=pen)[:-1]
110 +
111 +
112 +def bocpd(y, hazard=1 / 60.0, mu0=None, kappa0=1.0, alpha0=1.0, beta0=None):
113 + """Adams & MacKay (2007) with Normal–Inverse-Gamma conjugacy (Student-t predictive). Returns (R, cp_prob, map_rl):
114 + R[t, r] is the run-length posterior after observing y[:t]; with a constant hazard the mass on r = 0 equals the hazard
115 + by construction, so the detection statistics are cp_prob[t] = P(r_t <= 3 | y[:t+1]) (a changepoint in the last
116 + months) and map_rl[t] = argmax_r R[t+1, r] (the MAP run length, which resets after a change)."""
117 + from scipy.stats import t as student
118 + y = np.asarray(y, dtype=float); n = len(y)
119 + mu0 = y[:6].mean() if mu0 is None else mu0; beta0 = y[:6].var() + 1e-6 if beta0 is None else beta0
120 + R = np.zeros((n + 1, n + 1)); R[0, 0] = 1.0
121 + mu = np.array([mu0]); kappa = np.array([kappa0]); alpha = np.array([alpha0]); beta = np.array([beta0])
122 + cp = np.zeros(n); map_rl = np.zeros(n, dtype=int)
123 + for t_ in range(n):
124 + x = y[t_]
125 + df = 2 * alpha; scale = np.sqrt(beta * (kappa + 1) / (alpha * kappa))
126 + pred = student.pdf(x, df, loc=mu, scale=scale)
127 + growth = R[t_, : t_ + 1] * pred * (1 - hazard)
128 + change = (R[t_, : t_ + 1] * pred * hazard).sum()
129 + R[t_ + 1, 1 : t_ + 2] = growth; R[t_ + 1, 0] = change
130 + R[t_ + 1] /= R[t_ + 1].sum() + 1e-300
131 + cp[t_] = R[t_ + 1, :4].sum(); map_rl[t_] = int(np.argmax(R[t_ + 1, : t_ + 2]))
132 + mu_new = (kappa * mu + x) / (kappa + 1); kappa_new = kappa + 1; alpha_new = alpha + 0.5
133 + beta_new = beta + kappa * (x - mu) ** 2 / (2 * (kappa + 1))
134 + mu = np.concatenate([[mu0], mu_new]); kappa = np.concatenate([[kappa0], kappa_new]); alpha = np.concatenate([[alpha0], alpha_new]); beta = np.concatenate([[beta0], beta_new])
135 + return R, cp, map_rl
added embeddings/bench_models.py +21 −0
@@ -0,0 +1,21 @@
1 +#!/usr/bin/env python3
2 +"""Benchmark candidate scientific embedding models on Apple Silicon (MPS): throughput and a quick sanity check.
3 +Models: allenai/specter2_base (SPECTER2 base, no adapter), malteos/scincl, BAAI/bge-base-en-v1.5, sentence-transformers/all-MiniLM-L6-v2."""
4 +import sys, time, json, torch, duckdb
5 +from sentence_transformers import SentenceTransformer
6 +R = sys.argv[1]; N = int(sys.argv[2]) if len(sys.argv) > 2 else 2000
7 +con = duckdb.connect(f"{R}/data/processed/wp44.duckdb", read_only=True)
8 +rows = con.execute(f"SELECT w.title, a.abstract FROM oa_works w JOIN oa_abstracts a ON a.work_id=w.id WHERE w.publication_year=2024 LIMIT {N}").fetchall()
9 +texts = [f"{t}. {ab[:1500]}" for t, ab in rows]
10 +dev = "mps" if torch.backends.mps.is_available() else "cpu"; print("device", dev, "texts", len(texts), flush=True)
11 +res = {}
12 +for m in ["allenai/specter2_base", "malteos/scincl", "BAAI/bge-base-en-v1.5", "sentence-transformers/all-MiniLM-L6-v2"]:
13 + try:
14 + model = SentenceTransformer(m, device=dev); model.max_seq_length = 256
15 + _ = model.encode(texts[:64], batch_size=64)
16 + t0 = time.time(); E = model.encode(texts, batch_size=128, convert_to_numpy=True, normalize_embeddings=True); dt_ = time.time() - t0
17 + res[m] = {"dim": int(E.shape[1]), "texts_per_s": round(len(texts) / dt_, 1), "seconds": round(dt_, 1)}
18 + print(m, res[m], flush=True)
19 + except Exception as e:
20 + res[m] = {"error": str(e)[:300]}; print(m, "ERROR", str(e)[:200], flush=True)
21 +json.dump(res, open(f"{R}/data/metadata/embedding_benchmark.json", "w"), indent=1)
added embeddings/embed_corpus.py +63 −0
@@ -0,0 +1,63 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — embed title + abstract of corpus works with a scientific sentence-embedding model, sharded for multi-node runs.
3 +
4 + --prepare : DuckDB exports data/processed/embeddings/input/shard_NNN.parquet (work_id, text) for eligible works with an
5 + abstract (or title only when --title-only-fallback), N shards of --shard-rows rows.
6 + --run : encode shards [--shard-from, --shard-to) with --model on MPS/CPU → data/processed/embeddings/<model_tag>/
7 + shard_NNN.npy (float16, L2-normalised) + shard_NNN.ids.npy. Resumable (existing shards skipped).
8 +Provenance: data/processed/embeddings/<model_tag>/manifest.json (model, revision, max_seq_length, host, timings).
9 +"""
10 +import argparse, glob, json, os, socket, time, datetime as dt
11 +import numpy as np
12 +
13 +
14 +def prepare(root, shard_rows, title_only_fallback):
15 + import duckdb
16 + con = duckdb.connect(os.path.join(root, "data/processed/wp44.duckdb"), read_only=True); con.execute("SET threads TO 16;")
17 + out = os.path.join(root, "data/processed/embeddings/input"); os.makedirs(out, exist_ok=True)
18 + fb = " OR TRUE" if title_only_fallback else ""
19 + con.execute(f"""CREATE OR REPLACE TEMP TABLE emb_in AS
20 + SELECT w.id AS work_id, w.title || '. ' || coalesce(substr(a.abstract, 1, 2000), '') AS text
21 + FROM corpus_works w LEFT JOIN oa_abstracts a ON a.work_id = w.id
22 + WHERE w.eligible AND (a.abstract IS NOT NULL{fb}) ORDER BY w.id""")
23 + n = con.execute("SELECT count(*) FROM emb_in").fetchone()[0]
24 + con.execute(f"COPY (SELECT *, (row_number() OVER (ORDER BY work_id) - 1) // {shard_rows} AS shard FROM emb_in) TO '{out}' (FORMAT PARQUET, PARTITION_BY (shard), OVERWRITE_OR_IGNORE)")
25 + print("prepared", n, "texts in", len(glob.glob(out + "/shard=*")), "shards")
26 +
27 +
28 +def run(root, model_name, shard_from, shard_to, batch, max_len):
29 + import torch, pyarrow.parquet as pq
30 + from sentence_transformers import SentenceTransformer
31 + tag = model_name.replace("/", "__"); out = os.path.join(root, "data/processed/embeddings", tag); os.makedirs(out, exist_ok=True)
32 + dev = "mps" if torch.backends.mps.is_available() else "cpu"
33 + model = SentenceTransformer(model_name, device=dev); model.max_seq_length = max_len
34 + shards = sorted(glob.glob(os.path.join(root, "data/processed/embeddings/input", "shard=*", "*.parquet")))
35 + man_p = os.path.join(out, f"manifest_{socket.gethostname()}.json"); man = json.load(open(man_p)) if os.path.exists(man_p) else {"model": model_name, "max_seq_length": max_len, "device": dev, "host": socket.gethostname(), "shards": {}}
36 + for p in shards:
37 + k = int(os.path.basename(os.path.dirname(p)).split("=")[1])
38 + if not (shard_from <= k < shard_to):
39 + continue
40 + o = os.path.join(out, f"shard_{k:04d}.npy")
41 + if os.path.exists(o):
42 + continue
43 + t = pq.read_table(p).to_pydict(); t0 = time.time()
44 + E = model.encode(t["text"], batch_size=batch, convert_to_numpy=True, normalize_embeddings=True, show_progress_bar=False).astype(np.float16)
45 + np.save(o + ".part.npy", E); os.replace(o + ".part.npy", o); np.save(o.replace(".npy", ".ids.npy"), np.asarray(t["work_id"], dtype=np.int64))
46 + man["shards"][str(k)] = {"n": len(t["text"]), "seconds": round(time.time() - t0, 1), "when": dt.datetime.now(dt.UTC).isoformat()}
47 + json.dump(man, open(man_p, "w"), indent=1)
48 + print(f"shard {k}: {len(t['text'])} texts in {time.time()-t0:.0f}s ({len(t['text'])/(time.time()-t0):.0f}/s)", flush=True)
49 +
50 +
51 +def main():
52 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); ap.add_argument("--prepare", action="store_true"); ap.add_argument("--run", action="store_true")
53 + ap.add_argument("--model", default="allenai/specter2_base"); ap.add_argument("--shard-rows", type=int, default=50000); ap.add_argument("--shard-from", type=int, default=0)
54 + ap.add_argument("--shard-to", type=int, default=10**6); ap.add_argument("--batch", type=int, default=128); ap.add_argument("--max-len", type=int, default=256); ap.add_argument("--title-only-fallback", action="store_true")
55 + a = ap.parse_args()
56 + if a.prepare:
57 + prepare(a.root, a.shard_rows, a.title_only_fallback)
58 + if a.run:
59 + run(a.root, a.model, a.shard_from, a.shard_to, a.batch, a.max_len)
60 +
61 +
62 +if __name__ == "__main__":
63 + main()
added ingestion/compact_tables.py +20 −0
@@ -0,0 +1,20 @@
1 +#!/usr/bin/env python3
2 +"""Compact the per-part Parquet tables (1,898 files per table) into one Parquet file per table
3 +(data/processed/compact/<table>.parquet, zstd, ~1M-row row groups) so that downstream DuckDB work does not
4 +hit the macOS open-file limit. Also compacts the slim table of all OpenAlex works."""
5 +import duckdb, os, sys, time
6 +R = sys.argv[1]; suffix = sys.argv[2] if len(sys.argv) > 2 else "" # e.g. "_p2" for the second snapshot pass
7 +tables_dir = os.path.join(R, "data/processed/tables" + suffix); out = os.path.join(R, "data/processed/compact"); os.makedirs(out, exist_ok=True)
8 +con = duckdb.connect(); con.execute("SET threads TO 24; SET memory_limit='70GB'; SET preserve_insertion_order=false;")
9 +jobs = [(t + suffix, f"{tables_dir}/{t}/*.parquet") for t in ["works", "authorships", "work_topics", "work_keywords", "references", "abstracts", "locations", "grants", "counts_by_year"]]
10 +if not suffix:
11 + jobs.append(("slim_all_works", f"{R}/data/raw/openalex/works/slim/*.parquet"))
12 +for t, src in jobs:
13 + dst = f"{out}/{t}.parquet"
14 + if os.path.exists(dst):
15 + print("skip", t); continue
16 + t0 = time.time()
17 + con.execute(f"COPY (SELECT * FROM read_parquet('{src}')) TO '{dst}.part' (FORMAT PARQUET, COMPRESSION ZSTD, ROW_GROUP_SIZE 1000000)")
18 + os.replace(dst + ".part", dst)
19 + n = con.execute(f"SELECT count(*) FROM read_parquet('{dst}')").fetchone()[0]
20 + print(f"{t}: {n:,} rows, {os.path.getsize(dst)/1e9:.2f} GB, {time.time()-t0:.0f}s", flush=True)
modified ingestion/link_sources.py +10 −7
@@ -17,8 +17,11 @@ def main():
17 17 T = os.path.join(R, "data/processed/tables"); t0 = time.time(); rep = {"created": dt.datetime.now(dt.UTC).isoformat()}
18 18 def log(m): print(f"[{time.time()-t0:7.0f}s] {m}", flush=True)
19 19 for t in ["works", "authorships", "work_topics", "work_keywords", "references", "abstracts", "locations", "grants", "counts_by_year"]:
20 − con.execute(f"CREATE OR REPLACE VIEW oa_{t} AS SELECT * FROM read_parquet('{T}/{t}/*.parquet')")
21 − con.execute(f"CREATE OR REPLACE VIEW oa_slim AS SELECT * FROM read_parquet('{R}/data/raw/openalex/works/slim/*.parquet')")
20 + srcs = [f"{R}/data/processed/compact/{t}.parquet"]
21 + if os.path.exists(f"{R}/data/processed/compact/{t}_p2.parquet"):
22 + srcs.append(f"{R}/data/processed/compact/{t}_p2.parquet")
23 + con.execute(f"CREATE OR REPLACE VIEW oa_{t} AS SELECT * FROM read_parquet({srcs!r}, union_by_name=true)")
24 + con.execute(f"CREATE OR REPLACE VIEW oa_slim AS SELECT * FROM read_parquet('{R}/data/processed/compact/slim_all_works.parquet')")
22 25 con.execute(f"CREATE OR REPLACE VIEW arxiv_raw AS SELECT * FROM read_parquet('{R}/data/raw/arxiv/arxiv_cs.parquet')")
23 26 con.execute(f"CREATE OR REPLACE VIEW acl_raw AS SELECT * FROM read_parquet('{R}/data/raw/acl/acl_anthology.parquet')")
24 27 con.execute(f"CREATE OR REPLACE VIEW dblp_raw AS SELECT * FROM read_parquet('{R}/data/raw/dblp/dblp_publications.parquet')")
@@ -49,9 +52,9 @@ def main():
49 52 WHERE d.arxiv_id IS NOT NULL AND o.arxiv_id IS NOT NULL AND d.dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa)
50 53 AND o.oa_id NOT IN (SELECT oa_id FROM link_dblp_oa)""")
51 54 # title+year: unique normalised titles on both sides, length >= 25, |year diff| <= 1
52 − con.execute("""CREATE OR REPLACE TEMP TABLE d_un AS SELECT ntitle, any_value(dblp_key) dblp_key, any_value(year) year FROM dblp_keys
55 + con.execute("""CREATE OR REPLACE TEMP TABLE d_un AS SELECT ntitle, any_value(dblp_key) AS dblp_key, any_value(year) AS year FROM dblp_keys
53 56 WHERE tlen >= 25 AND dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa) GROUP BY ntitle HAVING count(*) = 1""")
54 − con.execute("""CREATE OR REPLACE TEMP TABLE o_un AS SELECT ntitle, any_value(oa_id) oa_id, any_value(year) year FROM oa_keys
57 + con.execute("""CREATE OR REPLACE TEMP TABLE o_un AS SELECT ntitle, any_value(oa_id) AS oa_id, any_value(year) AS year FROM oa_keys
55 58 WHERE tlen >= 25 AND oa_id NOT IN (SELECT oa_id FROM link_dblp_oa) GROUP BY ntitle HAVING count(*) = 1""")
56 59 con.execute("""INSERT INTO link_dblp_oa SELECT d.dblp_key, o.oa_id, 'title_year' FROM d_un d JOIN o_un o ON d.ntitle = o.ntitle
57 60 WHERE abs(d.year - o.year) <= 1""")
@@ -61,9 +64,9 @@ def main():
61 64 SELECT x.arxiv_id, o.oa_id, 'arxiv_id' AS method FROM arxiv_keys x JOIN oa_keys o ON x.arxiv_id = o.arxiv_id""")
62 65 con.execute("""INSERT INTO link_arxiv_oa SELECT x.arxiv_id, o.oa_id, 'doi' FROM arxiv_keys x JOIN oa_keys o ON x.doi = o.doi
63 66 WHERE x.doi IS NOT NULL AND x.arxiv_id NOT IN (SELECT arxiv_id FROM link_arxiv_oa)""")
64 − con.execute("""CREATE OR REPLACE TEMP TABLE x_un AS SELECT ntitle, any_value(arxiv_id) arxiv_id, any_value(created_year) year FROM arxiv_keys
67 + con.execute("""CREATE OR REPLACE TEMP TABLE x_un AS SELECT ntitle, any_value(arxiv_id) AS arxiv_id, any_value(created_year) AS year FROM arxiv_keys
65 68 WHERE length(ntitle) >= 25 AND arxiv_id NOT IN (SELECT arxiv_id FROM link_arxiv_oa) GROUP BY ntitle HAVING count(*) = 1""")
66 − con.execute("""CREATE OR REPLACE TEMP TABLE o_un2 AS SELECT ntitle, any_value(oa_id) oa_id, any_value(year) year FROM oa_keys
69 + con.execute("""CREATE OR REPLACE TEMP TABLE o_un2 AS SELECT ntitle, any_value(oa_id) AS oa_id, any_value(year) AS year FROM oa_keys
67 70 WHERE tlen >= 25 GROUP BY ntitle HAVING count(*) = 1""")
68 71 con.execute("""INSERT INTO link_arxiv_oa SELECT x.arxiv_id, o.oa_id, 'title_year' FROM x_un x JOIN o_un2 o ON x.ntitle = o.ntitle
69 72 WHERE abs(x.year - o.year) <= 2 AND o.oa_id NOT IN (SELECT oa_id FROM link_arxiv_oa)""")
@@ -76,7 +79,7 @@ def main():
76 79 WHERE landing_page_url LIKE '%aclanthology.org/%') l ON l.aid = c.acl_id OR l.aid = c.acl_id || '/'
77 80 WHERE c.acl_id NOT IN (SELECT acl_id FROM link_acl_oa)""")
78 81 con.execute("""INSERT INTO link_acl_oa SELECT c.acl_id, o.oa_id, 'title_year' FROM
79 − (SELECT ntitle, any_value(acl_id) acl_id, any_value(year) year FROM acl_keys WHERE length(ntitle) >= 25 AND acl_id NOT IN (SELECT acl_id FROM link_acl_oa) GROUP BY ntitle HAVING count(*)=1) c
82 + (SELECT ntitle, any_value(acl_id) AS acl_id, any_value(year) AS year FROM acl_keys WHERE length(ntitle) >= 25 AND acl_id NOT IN (SELECT acl_id FROM link_acl_oa) GROUP BY ntitle HAVING count(*)=1) c
80 83 JOIN o_un2 o ON c.ntitle = o.ntitle WHERE abs(c.year - o.year) <= 1 AND o.oa_id NOT IN (SELECT oa_id FROM link_acl_oa)""")
81 84 log("acl linked")
82 85 # --- report ---
modified ingestion/openalex_snapshot.py +35 −7
@@ -39,6 +39,7 @@ import pyarrow.parquet as pq
39 39 import zstandard as zstd
40 40
41 41 S3_HTTP = "https://openalex.s3.amazonaws.com/"
42 +ID_SET = None # optional: set of OpenAlex work ids (ints) to extract instead of the CS filter (second pass)
42 43 CS_FIELD = "https://openalex.org/fields/17"
43 44 YEAR_MIN, YEAR_MAX = 2018, 2026
44 45
@@ -116,9 +117,21 @@ def slim_row(w: dict) -> tuple:
116 117 ), cs_any
117 118
118 119
120 +def _load_ids(path):
121 + with open(path) as fh:
122 + next(fh)
123 + return {int(x.strip().rsplit("W", 1)[-1]) for x in fh if x.strip()}
124 +
125 +
126 +def _init_worker(id_list):
127 + """Pool initializer: macOS uses the spawn start method, so module globals must be rebuilt in each worker."""
128 + global ID_SET
129 + ID_SET = _load_ids(id_list) if id_list else None
130 +
131 +
119 132 def process_file(args):
120 133 """Download, hash, parse and filter one part file. Returns a result dict."""
121 − url, expected_len, out_dir, keep_raw = args
134 + url, expected_len, out_dir, keep_raw, mode = args
122 135 rel = url.replace("s3://openalex/data/jsonl/works/", "")
123 136 name = rel.replace("/", "__") # updated_date=X__part_0000.gz
124 137 http = url.replace("s3://openalex/", S3_HTTP)
@@ -164,6 +177,11 @@ def process_file(args):
164 177 n_bad += 1
165 178 continue
166 179 n_all += 1
180 + if mode == "idset":
181 + if _oa_int(w.get("id")) in ID_SET:
182 + cs_w.write(line if line.endswith(b"\n") else line + b"\n")
183 + n_cs += 1
184 + continue
167 185 row, cs_any = slim_row(w)
168 186 for i, v in enumerate(row):
169 187 cols[i].append(v)
@@ -172,10 +190,13 @@ def process_file(args):
172 190 cs_w.write(line if line.endswith(b"\n") else line + b"\n")
173 191 n_cs += 1
174 192 os.replace(cs_tmp, cs_tmp[:-5])
175 − table = pa.Table.from_arrays([pa.array(c, type=f.type) for c, f in zip(cols, SLIM_SCHEMA)], schema=SLIM_SCHEMA)
176 − slim_tmp = os.path.join(slim_dir, name[:-3] + ".parquet.part")
177 − pq.write_table(table, slim_tmp, compression="zstd")
178 − os.replace(slim_tmp, slim_tmp[:-5])
193 + if mode == "idset":
194 + cols = None
195 + table = None if cols is None else pa.Table.from_arrays([pa.array(c, type=f.type) for c, f in zip(cols, SLIM_SCHEMA)], schema=SLIM_SCHEMA)
196 + if table is not None:
197 + slim_tmp = os.path.join(slim_dir, name[:-3] + ".parquet.part")
198 + pq.write_table(table, slim_tmp, compression="zstd")
199 + os.replace(slim_tmp, slim_tmp[:-5])
179 200 if not keep_raw:
180 201 os.remove(raw_path)
181 202 res.update(status="done", sha256=sha, n_records=n_all, n_cs=n_cs, n_bad=n_bad,
@@ -193,7 +214,14 @@ def main():
193 214 ap.add_argument("--keep-raw", action="store_true")
194 215 ap.add_argument("--limit", type=int, default=0, help="process only N files (smoke test)")
195 216 ap.add_argument("--smallest-first", action="store_true")
217 + ap.add_argument("--id-list", default=None, help="CSV with a header and one OpenAlex work id per line: extract exactly these works (second pass)")
196 218 a = ap.parse_args()
219 + global ID_SET
220 + mode = "cs"
221 + if a.id_list:
222 + ID_SET = _load_ids(a.id_list)
223 + mode = "idset"
224 + print(f"id-set mode: {len(ID_SET)} ids", flush=True)
197 225 os.makedirs(a.out, exist_ok=True)
198 226 man = json.load(open(a.manifest))
199 227 files = [(f["url"], f["meta"]["content_length"], f["meta"].get("record_count")) for f in man["files"]]
@@ -214,12 +242,12 @@ def main():
214 242 (url, url.replace("s3://openalex/data/jsonl/works/", "").replace("/", "__"), ln, rc))
215 243 db.commit()
216 244 done = {r[0] for r in db.execute("SELECT url FROM files WHERE status='done'")}
217 − todo = [(u, ln, a.out, a.keep_raw) for u, ln, _ in files if u not in done]
245 + todo = [(u, ln, a.out, a.keep_raw, mode) for u, ln, _ in files if u not in done]
218 246 total_bytes = sum(t[1] for t in todo)
219 247 print(f"[{dt.datetime.now():%H:%M:%S}] manifest {man.get('date')}: {len(files)} files, {len(done)} done, "
220 248 f"{len(todo)} to do ({total_bytes/1e9:.1f} GB), workers={a.workers}", flush=True)
221 249 t0 = time.time(); done_bytes = 0; n_ok = n_err = 0
222 − with mp.Pool(a.workers, maxtasksperchild=50) as pool:
250 + with mp.Pool(a.workers, maxtasksperchild=50, initializer=_init_worker, initargs=(a.id_list,)) as pool:
223 251 for r in pool.imap_unordered(process_file, todo):
224 252 now = dt.datetime.now(dt.UTC).isoformat()
225 253 db.execute("UPDATE files SET attempts=attempts+1 WHERE url=?", (r["url"],))
added ingestion/run_increment_daily.sh +8 −0
@@ -0,0 +1,8 @@
1 +#!/bin/sh
2 +# re-run the API increment once per day until all slices are done (public budget $0.10/day)
3 +cd ~/wp44_uqo; . .venv/bin/activate
4 +while ! grep -q "all slices done" logs/openalex_api_increment.log 2>/dev/null; do
5 + python ingestion/openalex_api_increment.py --out data/raw/openalex >> logs/openalex_api_increment.log 2>&1
6 + grep -q "all slices done" logs/openalex_api_increment.log && break
7 + sleep 21700
8 +done
added knowledge_graph/01_citation_graph.py +41 −0
@@ -0,0 +1,41 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — citation graph of the eligible corpus (Phase 3, structural layer).
3 +
4 +Builds, in DuckDB and Parquet (data/processed/graph/):
5 + cit_edges_internal.parquet (citing_id, cited_id, citing_year, cited_year) both ends eligible
6 + cit_edges_external.parquet (citing_id, cited_id, cited_field_id, cited_year, cited_type) cited work outside the corpus,
7 + resolved through the slim table of all 465 M OpenAlex works (field of the cited work)
8 + field_flows_yearly.csv citing year × cited primary field (OpenAlex field) counts — interdisciplinary inflows (RQ4)
9 + cited_age_yearly.csv distribution of citation age (citing year − cited year) by citing year
10 + reference_coverage_yearly.csv share of references resolvable (in OpenAlex), internal vs external
11 +Category-level flows are produced by knowledge_graph/02_category_flows.py after classification."""
12 +import argparse, os, duckdb
13 +
14 +
15 +def main():
16 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
17 + G = os.path.join(R, "data/processed/graph"); out = os.path.join(R, "results/graph"); os.makedirs(G, exist_ok=True); os.makedirs(out, exist_ok=True)
18 + con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb")); con.execute("SET threads TO 24; SET memory_limit='70GB';")
19 + con.execute("CREATE OR REPLACE TEMP TABLE E AS SELECT id, publication_year AS y, type, primary_field_id FROM corpus_works WHERE eligible")
20 + con.execute("""CREATE OR REPLACE TABLE cit_edges_internal AS
21 + SELECT r.work_id AS citing_id, r.ref_id AS cited_id, a.y AS citing_year, b.y AS cited_year
22 + FROM oa_references r JOIN E a ON a.id = r.work_id JOIN E b ON b.id = r.ref_id""")
23 + con.execute("""CREATE OR REPLACE TABLE cit_edges_external AS
24 + SELECT r.work_id AS citing_id, r.ref_id AS cited_id, a.y AS citing_year, s.publication_year AS cited_year, s.primary_field_id AS cited_field_id, s.type AS cited_type, s.cs_any_topic AS cited_cs_any
25 + FROM oa_references r JOIN E a ON a.id = r.work_id LEFT JOIN E b ON b.id = r.ref_id
26 + JOIN oa_slim s ON s.id = r.ref_id WHERE b.id IS NULL""")
27 + for t in ("cit_edges_internal", "cit_edges_external"):
28 + con.execute(f"COPY {t} TO '{G}/{t}.parquet' (FORMAT PARQUET, COMPRESSION ZSTD)")
29 + con.execute(f"""COPY (SELECT citing_year, 'internal' AS kind, NULL AS cited_field_id, count(*) n FROM cit_edges_internal GROUP BY 1
30 + UNION ALL SELECT citing_year, 'external', cited_field_id, count(*) FROM cit_edges_external GROUP BY 1,3 ORDER BY 1,2,3) TO '{out}/field_flows_yearly.csv' (HEADER)""")
31 + con.execute(f"""COPY (SELECT citing_year, citing_year - cited_year AS age, count(*) n FROM (SELECT citing_year, cited_year FROM cit_edges_internal UNION ALL SELECT citing_year, cited_year FROM cit_edges_external)
32 + WHERE cited_year IS NOT NULL GROUP BY 1,2 ORDER BY 1,2) TO '{out}/cited_age_yearly.csv' (HEADER)""")
33 + con.execute(f"""COPY (SELECT e.y AS year, count(*) n_works, sum(n_refs) n_refs_declared,
34 + (SELECT count(*) FROM cit_edges_internal i WHERE i.citing_year = e.y) n_internal,
35 + (SELECT count(*) FROM cit_edges_external x WHERE x.citing_year = e.y) n_external
36 + FROM E e JOIN corpus_works c ON c.id = e.id GROUP BY 1 ORDER BY 1) TO '{out}/reference_coverage_yearly.csv' (HEADER)""")
37 + print(open(f"{out}/reference_coverage_yearly.csv").read())
38 +
39 +
40 +if __name__ == "__main__":
41 + main()
added manuscript/appendices/appA_taxonomy.tex +1 −0
@@ -0,0 +1 @@
1 +% appA_taxonomy — to be written (Phase 7)
added manuscript/appendices/appB_search_and_sources.tex +1 −0
@@ -0,0 +1 @@
1 +% appB_search_and_sources — to be written (Phase 7)
added manuscript/appendices/appC_derivations.tex +1 −0
@@ -0,0 +1 @@
1 +% appC_derivations — to be written (Phase 7)
added manuscript/appendices/appD_robustness.tex +1 −0
@@ -0,0 +1 @@
1 +% appD_robustness — to be written (Phase 7)
added manuscript/appendices/appE_data_dictionary.tex +1 −0
@@ -0,0 +1 @@
1 +% appE_data_dictionary — to be written (Phase 7)
added manuscript/appendices/appF_supplementary_tables.tex +1 −0
@@ -0,0 +1 @@
1 +% appF_supplementary_tables — to be written (Phase 7)
added manuscript/appendices/appG_figure_documentation.tex +1 −0
@@ -0,0 +1 @@
1 +% appG_figure_documentation — to be written (Phase 7)
added manuscript/appendices/appH_reproducibility.tex +1 −0
@@ -0,0 +1 @@
1 +% appH_reproducibility — to be written (Phase 7)
added manuscript/main.tex +97 −0
@@ -0,0 +1,97 @@
1 +% ============================================================================
2 +% UQO Working Paper 44 — The Post-ChatGPT Computer Science Revolution
3 +% Preamble follows the UQO working-paper template (_modele-working-paper/main.tex)
4 +% ============================================================================
5 +\documentclass[12pt,letterpaper]{article}
6 +\usepackage[utf8]{inputenc}
7 +\usepackage[T1]{fontenc}
8 +\usepackage[english]{babel}
9 +\usepackage[letterpaper,top=1in,bottom=1in,left=1in,right=1in]{geometry}
10 +\usepackage{mathptmx}
11 +\usepackage{microtype}
12 +\usepackage{setspace}
13 +\onehalfspacing
14 +\usepackage{amsmath,amssymb,amsthm}
15 +\usepackage{booktabs,tabularx,multirow,threeparttable,longtable,array}
16 +\usepackage{siunitx}
17 +\sisetup{input-symbols = {()}, table-format = -1.3, parse-numbers = false}
18 +\usepackage{graphicx,subcaption,float}
19 +\usepackage[dvipsnames]{xcolor}
20 +\usepackage[colorlinks=true,linkcolor=NavyBlue,citecolor=NavyBlue,urlcolor=NavyBlue,breaklinks=true]{hyperref}
21 +\usepackage[authoryear,round,semicolon]{natbib}
22 +\bibliographystyle{aer}
23 +\usepackage[font=small,labelfont=bf,labelsep=period,justification=justified,singlelinecheck=false]{caption}
24 +\usepackage{fancyhdr}
25 +\pagestyle{fancy}\fancyhf{}\renewcommand{\headrulewidth}{0pt}\fancyfoot[C]{\thepage}
26 +\usepackage{titlesec}
27 +\titleformat{\section}{\large\bfseries}{\thesection.}{0.5em}{}
28 +\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection.}{0.5em}{}
29 +\titleformat{\subsubsection}{\normalsize\itshape}{\thesubsubsection.}{0.5em}{}
30 +\usepackage[toc,page]{appendix}
31 +\usepackage{enumitem}
32 +\setlist{nosep,leftmargin=*}
33 +\newtheorem{definition}{Definition}
34 +\newtheorem{proposition}{Proposition}
35 +\newtheorem{hypothesis}{Hypothesis}
36 +\newcommand{\E}{\mathbb{E}}
37 +\newcommand{\R}{\mathbb{R}}
38 +\newcommand{\pending}[1]{\textcolor{BrickRed}{[pending human review: #1]}}
39 +\graphicspath{{figures/}}
40 +
41 +% ---- metadata ----
42 +\newcommand{\WPnumber}{44}
43 +\newcommand{\WPtitle}{The Post-ChatGPT Computer Science Revolution}
44 +\newcommand{\WPsubtitle}{A Comprehensive Systematic Mapping, Bibliometric Analysis, and Knowledge Graph of Computer Science Research, 2022--2026}
45 +\newcommand{\WPdate}{November 2026}
46 +\newcommand{\WPversion}{0.1}
47 +\input{tables/numbers} % machine-generated macros (counts, estimates) — never edited by hand
48 +\newcommand{\WPabstract}{\input{sections/abstract}}
49 +\newcommand{\WPkeywords}{systematic mapping, bibliometrics, computer science, large language models, structural change, knowledge graph, emerging topics, OpenAlex}
50 +\newcommand{\WPjel}{C55, C88, O33, L86, I23}
51 +\newcommand{\WPauthor}{Simon-Pierre Boucher}
52 +\newcommand{\WPaffiliation}{Département des sciences administratives\\Université du Québec en Outaouais}
53 +\newcommand{\WPemail}{simon-pierre.boucher@uqo.ca}
54 +\newcommand{\WPaddress}{Gatineau -- Pavillon Alexandre-Taché\\283, boulevard Alexandre-Taché\\Gatineau, Québec, Canada J9A 1L8}
55 +
56 +\begin{document}
57 +\input{sections/titlepage}
58 +\tableofcontents
59 +\clearpage
60 +
61 +\part{Introduction and Scientific Motivation}
62 +\input{sections/part01_introduction}
63 +\part{The Evolution of Computer Science}
64 +\input{sections/part02_evolution}
65 +\part{Data and Systematic Review Protocol}
66 +\input{sections/part03_data}
67 +\part{Computational Methodology}
68 +\input{sections/part04_methodology}
69 +\part{The Artificial Intelligence Transformation}
70 +\input{sections/part05_ai}
71 +\part{Systems and Computing Infrastructure}
72 +\input{sections/part06_systems}
73 +\part{Theory, Algorithms, and Security}
74 +\input{sections/part07_theory_security}
75 +\part{Scientific Knowledge Networks}
76 +\input{sections/part08_networks}
77 +\part{Empirical Results}
78 +\input{sections/part09_results}
79 +\part{The Next Research Frontiers}
80 +\input{sections/part10_frontiers}
81 +\part{Discussion and Conclusion}
82 +\input{sections/part11_discussion}
83 +
84 +\clearpage
85 +\bibliography{references}
86 +
87 +\begin{appendices}
88 +\input{appendices/appA_taxonomy}
89 +\input{appendices/appB_search_and_sources}
90 +\input{appendices/appC_derivations}
91 +\input{appendices/appD_robustness}
92 +\input{appendices/appE_data_dictionary}
93 +\input{appendices/appF_supplementary_tables}
94 +\input{appendices/appG_figure_documentation}
95 +\input{appendices/appH_reproducibility}
96 +\end{appendices}
97 +\end{document}
added manuscript/sections/abstract.tex +1 −0
@@ -0,0 +1 @@
1 +% abstract — to be written (Phase 7)
added manuscript/sections/part01_introduction.tex +1 −0
@@ -0,0 +1 @@
1 +% part01_introduction — to be written (Phase 7)
added manuscript/sections/part02_evolution.tex +1 −0
@@ -0,0 +1 @@
1 +% part02_evolution — to be written (Phase 7)
added manuscript/sections/part03_data.tex +1 −0
@@ -0,0 +1 @@
1 +% part03_data — to be written (Phase 7)
added manuscript/sections/part04_methodology.tex +1 −0
@@ -0,0 +1 @@
1 +% part04_methodology — to be written (Phase 7)
added manuscript/sections/part05_ai.tex +1 −0
@@ -0,0 +1 @@
1 +% part05_ai — to be written (Phase 7)
added manuscript/sections/part06_systems.tex +1 −0
@@ -0,0 +1 @@
1 +% part06_systems — to be written (Phase 7)
added manuscript/sections/part07_theory_security.tex +1 −0
@@ -0,0 +1 @@
1 +% part07_theory_security — to be written (Phase 7)
added manuscript/sections/part08_networks.tex +1 −0
@@ -0,0 +1 @@
1 +% part08_networks — to be written (Phase 7)
added manuscript/sections/part09_results.tex +1 −0
@@ -0,0 +1 @@
1 +% part09_results — to be written (Phase 7)
added manuscript/sections/part10_frontiers.tex +1 −0
@@ -0,0 +1 @@
1 +% part10_frontiers — to be written (Phase 7)
added manuscript/sections/part11_discussion.tex +1 −0
@@ -0,0 +1 @@
1 +% part11_discussion — to be written (Phase 7)
added manuscript/tables/numbers.tex +1 −0
@@ -0,0 +1 @@
1 +% machine-generated numbers (tables/make_numbers.py) — placeholder until results exist
modified protocol/amendments.csv +2 −0
@@ -1,2 +1,4 @@
1 1 id,date,phase,change,reason,approved_by
2 2 A-000,2026-11-02,0,"§14 of the mandate completed from §1–§13 (paste truncated)","terminal truncation of the pasted mandate",executing agent (pending PI)
3 +A-001,2026-11-03,1,"Eligibility type list corrected: OpenAlex (2026) types conference-paper and software-paper are research outputs and are kept; DBLP-linked records are kept whatever their OpenAlex type unless in a hard-exclusion list; XPAC records are kept when DBLP-linked (flag xpac_kept_dblp)","first run of define_corpus.py showed 2.5 M conference-paper records (1.09 M DBLP-linked) excluded by an outdated type list; DBLP is the CS gold standard",executing agent (pending PI)
4 +A-002,2026-11-03,1,"Records whose primary source is a generic deposit platform (Zenodo, Figshare, IRDB, LA Referencia, Dialnet, DOAJ-as-source, OSTI) are excluded unless DBLP- or arXiv-linked (flag e7_generic_deposit); date_precision flag added (publication_date = YYYY-01-01 treated as year precision; monthly series use month-precision records only)","Zenodo alone contributes 138,533 eligible records in 2026 (20,141 in 2025; 1.3% with references) — deposits of slides, datasets and reports typed as articles; 18–28% of records carry a year-only date defaulting to 1 January",executing agent (pending PI)
added reports/HUMAN_REVIEW_CHECKPOINTS.md +12 −0
@@ -0,0 +1,12 @@
1 +# Human-review checkpoints (protocol §P7) — items awaiting the principal investigator
2 +
3 +| Checkpoint | What to review | Where | Status |
4 +|---|---|---|---|
5 +| H-0 | Plan, protocol, decisions D1–D7 and amendments A-000–A-002 | `RESEARCH_PLAN.md` §10, `protocol/PROTOCOL.md`, `protocol/amendments.csv` | pending |
6 +| H-1 | Record-linkage precision sample (500 DBLP↔OpenAlex pairs, 200 reserved for the PI) | `data/metadata/linkage_sample.csv` (to be generated by `systematic_review/linkage_sample.py`) | pending |
7 +| H-2 | Taxonomy validation sample (2,000 works, two LLM raters; 200 for the PI) | `taxonomy/validation/` | pending |
8 +| H-3 | Landmark list (≥ 1,000) and specialist reviews | `literature_synthesis/landmarks/` | pending |
9 +| H-4 | Structural-change results and interpretation (no causal language) | `results/econometrics/` | pending |
10 +| H-5 | Manuscript conclusions | `manuscript/` | pending |
11 +
12 +Conclusions that depend on an unsigned checkpoint are labelled "pending human review" in the manuscript.
added systematic_review/define_corpus.py +110 −0
@@ -0,0 +1,110 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — Stages S2–S3: deduplicate (preprint ↔ published), apply eligibility, define the mapping corpus, write PRISMA counts.
3 +
4 +Inputs: DuckDB views/tables made by ingestion/link_sources.py (oa_* views, dblp_keys, arxiv_keys, link_* tables).
5 +Outputs: table `corpus_works` (+ Parquet copy data/processed/corpus/corpus_works.parquet), `dblp_only_counts`,
6 +data/metadata/prisma_counts.json. Rules: protocol/PROTOCOL.md §P2; decisions logged in RESEARCH_PLAN §10.
7 +"""
8 +import argparse, json, os, time, datetime as dt
9 +import duckdb
10 +
11 +KEEP_TYPES = "('article','conference-paper','preprint','book-chapter','book','dissertation','report','review','software-paper')"
12 +# hard exclusions even for DBLP-linked records (DBLP-linked records of any other type are kept: DBLP is the CS gold standard)
13 +# generic deposit platforms (A-002): records whose primary source is one of these are kept only if DBLP- or arXiv-linked
14 +GENERIC_DEPOSIT_SOURCES = "('Zenodo (CERN European Organization for Nuclear Research)','Figshare','Institutional Repositories DataBase (IRDB)','LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas)','Dialnet (Universidad de la Rioja)','DOAJ (DOAJ: Directory of Open Access Journals)','OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information)','Zenodo')"
15 +HARD_EXCLUDE_TYPES = "('editorial','erratum','retraction','peer-review','paratext','conference-abstract','reference-entry','dataset','software','supplementary-materials','libguides','letter','grant','standard')"
16 +WINDOW_START, BOUNDARY, WINDOW_END = "2018-01-01", "2022-11-30", "2026-11-02"
17 +
18 +
19 +def main():
20 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
21 + con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb")); con.execute("SET threads TO 24; SET memory_limit='60GB';")
22 + t0 = time.time(); q = lambda s: con.execute(s).fetchall(); n = lambda s: q(s)[0][0]
23 + log = lambda m: print(f"[{time.time()-t0:6.0f}s] {m}", flush=True)
24 + # ---- S0/S1: base table with membership flags ----
25 + con.execute(f"""CREATE OR REPLACE TABLE cw0 AS
26 + SELECT w.*, regexp_replace(lower(w.title), '[^a-z0-9]+', '', 'g') AS ntitle,
27 + (w.id IN (SELECT oa_id FROM link_dblp_oa)) AS in_dblp,
28 + (w.id IN (SELECT oa_id FROM link_acl_oa)) AS in_acl,
29 + x.arxiv_id AS arxiv_id_linked, x.created AS arxiv_created, x.primary_category AS arxiv_primary,
30 + (x.primary_category LIKE 'cs.%') AS arxiv_cs_primary,
31 + (w.doi LIKE '10.48550/arxiv.%') AS is_arxiv_doi
32 + FROM oa_works w
33 + LEFT JOIN (SELECT l.oa_id, any_value(k.arxiv_id) arxiv_id, min(k.created) created, any_value(k.primary_category) primary_category
34 + FROM link_arxiv_oa l JOIN arxiv_keys k USING(arxiv_id) GROUP BY l.oa_id) x ON x.oa_id = w.id""")
35 + log(f"cw0 {n('SELECT count(*) FROM cw0'):,}")
36 + # ---- S2: preprint ↔ published duplicates (same normalised title, |year| <= 2, one preprint-like, one not) ----
37 + con.execute("""CREATE OR REPLACE TABLE dup_pairs AS
38 + WITH pre AS (SELECT id, ntitle, publication_year y, publication_date d FROM cw0 WHERE (type = 'preprint' OR is_arxiv_doi) AND length(ntitle) >= 25),
39 + pub AS (SELECT id, ntitle, publication_year y, publication_date d FROM cw0 WHERE type <> 'preprint' AND NOT is_arxiv_doi AND length(ntitle) >= 25)
40 + SELECT p.id AS preprint_id, b.id AS published_id, p.d AS preprint_date, b.d AS published_date,
41 + row_number() OVER (PARTITION BY b.id ORDER BY p.d, p.id) AS rn_pub, row_number() OVER (PARTITION BY p.id ORDER BY b.d, b.id) AS rn_pre
42 + FROM pre p JOIN pub b ON p.ntitle = b.ntitle AND abs(p.y - b.y) <= 2""")
43 + con.execute("DELETE FROM dup_pairs WHERE rn_pub > 1 OR rn_pre > 1")
44 + log(f"duplicate preprint→published pairs: {n('SELECT count(*) FROM dup_pairs'):,}")
45 + # ---- S3: eligibility flags ----
46 + con.execute(f"""CREATE OR REPLACE TABLE corpus_works AS
47 + SELECT c.*,
48 + dp.published_id AS duplicate_of,
49 + dp2.preprint_id AS preprint_work_id,
50 + least(c.publication_date, coalesce(dp2.preprint_date, c.publication_date), coalesce(c.arxiv_created, c.publication_date)) AS first_date,
51 + (c.publication_date IS NULL OR c.publication_date < '{WINDOW_START}' OR c.publication_date > '{WINDOW_END}') AS e1_outside_window,
52 + (c.type NOT IN {KEEP_TYPES} AND NOT (c.in_dblp AND c.type NOT IN {HARD_EXCLUDE_TYPES})) AS e2_type,
53 + (c.is_xpac AND NOT c.in_dblp) AS e3_xpac,
54 + (c.is_xpac AND c.in_dblp) AS xpac_kept_dblp,
55 + (c.source_name IN {GENERIC_DEPOSIT_SOURCES} AND NOT c.in_dblp AND c.arxiv_id_linked IS NULL) AS e7_generic_deposit,
56 + (CASE WHEN substr(c.publication_date, 6, 5) = '01-01' THEN 'year' ELSE 'month' END) AS date_precision,
57 + (c.title IS NULL OR length(trim(c.title)) = 0) AS e4_no_title,
58 + (dp.published_id IS NOT NULL) AS e5_duplicate,
59 + c.is_retracted AS e6_retracted,
60 + (c.in_dblp OR c.cs_primary OR (c.cs_any AND coalesce(c.arxiv_cs_primary, FALSE))) AS cs_core,
61 + (c.cs_any AND NOT (c.in_dblp OR c.cs_primary OR (c.cs_any AND coalesce(c.arxiv_cs_primary, FALSE)))) AS cs_penumbra
62 + FROM cw0 c
63 + LEFT JOIN dup_pairs dp ON dp.preprint_id = c.id
64 + LEFT JOIN dup_pairs dp2 ON dp2.published_id = c.id""")
65 + con.execute(f"""ALTER TABLE corpus_works ADD COLUMN eligible BOOLEAN; UPDATE corpus_works SET eligible =
66 + NOT e1_outside_window AND NOT e2_type AND NOT e3_xpac AND NOT e4_no_title AND NOT e5_duplicate AND NOT e7_generic_deposit AND cs_core""")
67 + con.execute(f"""ALTER TABLE corpus_works ADD COLUMN period VARCHAR; UPDATE corpus_works SET period = CASE
68 + WHEN first_date < '{WINDOW_START}' THEN 'pre' WHEN first_date < '{BOUNDARY}' THEN 'baseline' WHEN first_date <= '{WINDOW_END}' THEN 'post' ELSE 'future' END""")
69 + con.execute("ALTER TABLE corpus_works ADD COLUMN first_month VARCHAR; UPDATE corpus_works SET first_month = substr(first_date, 1, 7)")
70 + log("corpus_works built")
71 + # ---- DBLP-only records (no OpenAlex record after both passes) ----
72 + con.execute("""CREATE OR REPLACE TABLE dblp_only_counts AS
73 + SELECT year, rec_type, count(*) n, sum(CASE WHEN doi IS NOT NULL THEN 1 ELSE 0 END) with_doi FROM dblp_keys
74 + WHERE dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa) GROUP BY 1,2 ORDER BY 1,2""")
75 + # ---- PRISMA counts ----
76 + P = {"generated": dt.datetime.now(dt.UTC).isoformat(), "window": [WINDOW_START, WINDOW_END], "boundary": BOUNDARY}
77 + P["identified"] = {
78 + "openalex_cs_topic_2018_2026_pass1": n("SELECT count(*) FROM oa_keys"),
79 + "openalex_total_records_loaded": n("SELECT count(*) FROM cw0"),
80 + "dblp_2018_2026": n("SELECT count(*) FROM dblp_keys"), "arxiv_cs_2018_2026": n("SELECT count(*) FROM arxiv_keys WHERE created_year BETWEEN 2018 AND 2026"),
81 + "acl_2018_2026": n("SELECT count(*) FROM acl_keys WHERE year BETWEEN 2018 AND 2026")}
82 + P["linked"] = {"dblp_to_openalex": n("SELECT count(DISTINCT dblp_key) FROM link_dblp_oa"), "arxiv_to_openalex": n("SELECT count(DISTINCT arxiv_id) FROM link_arxiv_oa"),
83 + "acl_to_openalex": n("SELECT count(DISTINCT acl_id) FROM link_acl_oa"), "dblp_only_no_openalex_record": n("SELECT sum(n) FROM dblp_only_counts")}
84 + P["excluded"] = {k: n(f"SELECT count(*) FROM corpus_works WHERE {k}") for k in ["e1_outside_window", "e2_type", "e3_xpac", "e4_no_title", "e5_duplicate", "e7_generic_deposit", "e6_retracted", "xpac_kept_dblp"]}
85 + P["excluded_e2_by_type"] = dict(q("SELECT type, count(*) FROM corpus_works WHERE e2_type GROUP BY 1 ORDER BY 2 DESC"))
86 + P["excluded_sequential"] = {}
87 + cond = "TRUE"
88 + for k in ["e1_outside_window", "e2_type", "e3_xpac", "e4_no_title", "e5_duplicate", "e7_generic_deposit"]:
89 + P["excluded_sequential"][k] = n(f"SELECT count(*) FROM corpus_works WHERE {cond} AND {k}"); cond += f" AND NOT {k}"
90 + P["after_exclusions"] = n(f"SELECT count(*) FROM corpus_works WHERE {cond}")
91 + P["cs_core_eligible"] = n("SELECT count(*) FROM corpus_works WHERE eligible")
92 + P["cs_penumbra_eligible_otherwise"] = n(f"SELECT count(*) FROM corpus_works WHERE {cond} AND cs_penumbra")
93 + P["eligible_by_period"] = dict(q("SELECT period, count(*) FROM corpus_works WHERE eligible GROUP BY 1"))
94 + P["eligible_by_year"] = dict(q("SELECT publication_year, count(*) FROM corpus_works WHERE eligible GROUP BY 1 ORDER BY 1"))
95 + P["eligible_by_type"] = dict(q("SELECT type, count(*) FROM corpus_works WHERE eligible GROUP BY 1 ORDER BY 2 DESC"))
96 + P["eligible_membership"] = {"in_dblp": n("SELECT count(*) FROM corpus_works WHERE eligible AND in_dblp"), "cs_primary": n("SELECT count(*) FROM corpus_works WHERE eligible AND cs_primary"),
97 + "arxiv_linked": n("SELECT count(*) FROM corpus_works WHERE eligible AND arxiv_id_linked IS NOT NULL"), "in_acl": n("SELECT count(*) FROM corpus_works WHERE eligible AND in_acl"),
98 + "has_abstract": n("SELECT count(*) FROM corpus_works WHERE eligible AND has_abstract"), "has_references": n("SELECT count(*) FROM corpus_works WHERE eligible AND n_refs > 0"),
99 + "retracted_flagged": n("SELECT count(*) FROM corpus_works WHERE eligible AND e6_retracted")}
100 + P["eligible_date_precision"] = dict(q("SELECT date_precision, count(*) FROM corpus_works WHERE eligible GROUP BY 1"))
101 + P["strict_core_dblp_and_cs_primary"] = n("SELECT count(*) FROM corpus_works WHERE eligible AND in_dblp AND cs_primary")
102 + os.makedirs(os.path.join(R, "data/processed/corpus"), exist_ok=True); os.makedirs(os.path.join(R, "data/metadata"), exist_ok=True)
103 + con.execute(f"COPY corpus_works TO '{R}/data/processed/corpus/corpus_works.parquet' (FORMAT PARQUET, COMPRESSION ZSTD)")
104 + con.execute(f"COPY dblp_only_counts TO '{R}/data/metadata/dblp_only_counts.csv' (HEADER)")
105 + json.dump(P, open(os.path.join(R, "data/metadata/prisma_counts.json"), "w"), indent=1, default=str)
106 + print(json.dumps(P, indent=1, default=str)); log("done")
107 +
108 +
109 +if __name__ == "__main__":
110 + main()
added systematic_review/inspect_2026.py +11 −0
@@ -0,0 +1,11 @@
1 +import duckdb, sys
2 +con = duckdb.connect(sys.argv[1] + "/data/processed/wp44.duckdb", read_only=True)
3 +q = lambda s: con.execute(s).fetchall()
4 +print("2026 eligible by first_month:", q("select first_month, count(*) from corpus_works where eligible and publication_year=2026 group by 1 order by 1"))
5 +print("2026 by type:", q("select type, count(*) from corpus_works where eligible and publication_year=2026 group by 1 order by 2 desc limit 6"))
6 +print("2026 top sources:", q("select source_name, count(*) from corpus_works where eligible and publication_year=2026 group by 1 order by 2 desc limit 10"))
7 +print("2025 top sources:", q("select source_name, count(*) from corpus_works where eligible and publication_year=2025 group by 1 order by 2 desc limit 10"))
8 +print("2026 pub_date day-01 share:", q("select sum((substr(publication_date,9,2)='01')::int)*1.0/count(*) from corpus_works where eligible and publication_year=2026"))
9 +print("2025 pub_date day-01 share:", q("select sum((substr(publication_date,9,2)='01')::int)*1.0/count(*) from corpus_works where eligible and publication_year=2025"))
10 +print("2026 created_date months:", q("select substr(created_date,1,7), count(*) from corpus_works where eligible and publication_year=2026 group by 1 order by 1"))
11 +print("other-type dblp sample:", q("select id, doi, source_name, publication_year, n_refs from corpus_works where in_dblp and type='other' limit 4"))
added systematic_review/inspect_sources.py +10 −0
@@ -0,0 +1,10 @@
1 +import duckdb, sys
2 +con = duckdb.connect(sys.argv[1] + "/data/processed/wp44.duckdb", read_only=True)
3 +q = lambda s: con.execute(s).fetchall()
4 +print("eligible by source_type:", q("select source_type, count(*) n, sum(in_dblp::int) dblp, sum((arxiv_id_linked is not null)::int) arx, sum(has_abstract::int) ab, sum((n_refs>0)::int) refs from corpus_works where eligible group by 1 order by 2 desc"))
5 +print("repository sources top 15:", q("select source_name, count(*) n, sum(in_dblp::int) dblp, sum((arxiv_id_linked is not null)::int) arx, sum((n_refs>0)::int) refs, sum(has_abstract::int) ab from corpus_works where eligible and source_type='repository' group by 1 order by 2 desc limit 15"))
6 +print("Zenodo by year:", q("select publication_year, count(*), sum((n_refs>0)::int) refs, sum(has_abstract::int) ab, sum(in_dblp::int) dblp from corpus_works where eligible and source_name like 'Zenodo%' group by 1 order by 1"))
7 +print("Zenodo 2026 sample:", q("select substr(title,1,70), type, n_refs, has_abstract, n_authors, language from corpus_works where eligible and source_name like 'Zenodo%' and publication_year=2026 using sample 8"))
8 +print("NULL-source by year:", q("select publication_year, count(*), sum((n_refs>0)::int) refs, sum(has_abstract::int) ab, sum(in_dblp::int) dblp, sum((type='dissertation')::int) diss from corpus_works where eligible and source_id is null group by 1 order by 1"))
9 +print("NULL-source 2025 sample:", q("select substr(title,1,60), type, n_refs, has_abstract, n_locations, indexed_in from corpus_works where eligible and source_id is null and publication_year=2025 using sample 8"))
10 +print("day-01-01 share by year:", q("select publication_year, round(sum((substr(publication_date,6,5)='01-01')::int)*1.0/count(*),3) from corpus_works where eligible group by 1 order by 1"))
added taxonomy/classify.py +150 −0
@@ -0,0 +1,150 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — Phase 2: assign the 40 ontology categories (multi-label) to every work of the corpus.
3 +
4 +Signals per work: OpenAlex topics (ids + subfield ids), arXiv categories (through link_arxiv_oa), DBLP venue stream
5 +(through link_dblp_oa), keyword regexes on title and abstract. Scoring follows ontology.yaml `assignment_rule`:
6 +per signal type the matched weights are summed and capped at 1.0, then summed over signal types; a category is assigned
7 +if score >= threshold; primary = argmax (ties: child before parent, then lower category number).
8 +
9 +Step 1 (`--prepare`): DuckDB builds data/processed/classification/input/chunk_*.parquet (id, year, title, abstract,
10 +topic_ids, subfield_ids, arxiv_cats, dblp_streams). Step 2 (`--run`): multiprocessing over chunks →
11 +data/processed/classification/work_categories.parquet (work_id, category_id, score, s_topic, s_subfield, s_arxiv,
12 +s_dblp, s_kw) and work_primary.parquet (work_id, primary_category, n_categories). Deterministic; version-stamped.
13 +"""
14 +import argparse, glob, json, multiprocessing as mp, os, re, time, datetime as dt
15 +import yaml, duckdb
16 +import pyarrow as pa, pyarrow.parquet as pq
17 +
18 +ONT = None
19 +
20 +
21 +def load_ontology(path):
22 + o = yaml.safe_load(open(path))
23 + cats = o["categories"]
24 + comp = []
25 + for c in cats:
26 + kws = []
27 + for k in (c.get("keywords") or []):
28 + kws.append((re.compile(k["pattern"]), float(k["weight"]), k.get("field", "both")))
29 + exc = [re.compile(p if isinstance(p, str) else p["pattern"]) for p in (c.get("exclusions") or [])]
30 + comp.append(dict(
31 + id=c["id"], number=int(c["number"]), parent=c.get("child_of"),
32 + topics={str(t["id"]).lstrip("T"): float(t["weight"]) for t in (c.get("openalex_topics") or [])},
33 + subfields={str(t["id"]): float(t["weight"]) for t in (c.get("openalex_subfields") or [])},
34 + arxiv={t["id"]: float(t["weight"]) for t in (c.get("arxiv_categories") or [])},
35 + dblp={t["key"]: float(t["weight"]) for t in (c.get("dblp_streams") or [])},
36 + kws=kws, exc=exc))
37 + rule = o.get("assignment_rule", {})
38 + return dict(cats=comp, threshold=float(rule.get("threshold", 1.0)), version=o.get("version"))
39 +
40 +
41 +def _init(path):
42 + global ONT
43 + ONT = load_ontology(path)
44 +
45 +
46 +def score_row(title, abstract, topic_ids, subfield_ids, arxiv_cats, dblp_streams):
47 + title = title or ""; abstract = abstract or ""; both = title + "\n" + abstract
48 + out = []
49 + for c in ONT["cats"]:
50 + st = min(1.0, sum(c["topics"].get(str(t), 0.0) for t in (topic_ids or [])))
51 + ss = min(1.0, sum(c["subfields"].get(str(s), 0.0) for s in set(subfield_ids or [])))
52 + sa = min(1.0, sum(c["arxiv"].get(a, 0.0) for a in set(arxiv_cats or [])))
53 + sd = min(1.0, sum(c["dblp"].get(d, 0.0) for d in set(dblp_streams or [])))
54 + sk = 0.0
55 + for rx, w, field in c["kws"]:
56 + txt = title if field == "title" else abstract if field == "abstract" else both
57 + if txt and rx.search(txt):
58 + sk += w
59 + if sk >= 1.0:
60 + sk = 1.0; break
61 + s = st + ss + sa + sd + sk
62 + if s >= ONT["threshold"] and c["exc"] and any(rx.search(both) for rx in c["exc"]):
63 + s = 0.0
64 + if s >= ONT["threshold"]:
65 + out.append((c["id"], c["number"], c["parent"], round(s, 3), st, ss, sa, sd, sk))
66 + return out
67 +
68 +
69 +def process_chunk(args):
70 + path, out_dir = args
71 + name = os.path.basename(path).replace(".parquet", "")
72 + o1 = os.path.join(out_dir, "cats", name + ".parquet"); o2 = os.path.join(out_dir, "primary", name + ".parquet")
73 + if os.path.exists(o1) and os.path.exists(o2):
74 + return name, 0, 0
75 + t = pq.read_table(path).to_pydict()
76 + cw, cc, cs, c1, c2, c3, c4, c5 = [], [], [], [], [], [], [], []
77 + pw, pc, pn, ps = [], [], [], []
78 + n = len(t["id"])
79 + for i in range(n):
80 + res = score_row(t["title"][i], t["abstract"][i], t["topic_ids"][i], t["subfield_ids"][i], t["arxiv_cats"][i], t["dblp_streams"][i])
81 + for (cid, num, parent, s, st, ss, sa, sd, sk) in res:
82 + cw.append(t["id"][i]); cc.append(cid); cs.append(s); c1.append(st); c2.append(ss); c3.append(sa); c4.append(sd); c5.append(sk)
83 + if res:
84 + parents = {r[2] for r in res if r[2]}
85 + # primary: highest score; tie → child before parent (a category that is someone's parent loses), then lower number
86 + best = sorted(res, key=lambda r: (-r[3], 1 if r[0] in parents else 0, r[1]))[0]
87 + pw.append(t["id"][i]); pc.append(best[0]); pn.append(len(res)); ps.append(best[3])
88 + os.makedirs(os.path.dirname(o1), exist_ok=True); os.makedirs(os.path.dirname(o2), exist_ok=True)
89 + pq.write_table(pa.table({"work_id": pa.array(cw, pa.int64()), "category_id": cc, "score": pa.array(cs, pa.float32()), "s_topic": pa.array(c1, pa.float32()),
90 + "s_subfield": pa.array(c2, pa.float32()), "s_arxiv": pa.array(c3, pa.float32()), "s_dblp": pa.array(c4, pa.float32()), "s_kw": pa.array(c5, pa.float32())}), o1 + ".part", compression="zstd")
91 + os.replace(o1 + ".part", o1)
92 + pq.write_table(pa.table({"work_id": pa.array(pw, pa.int64()), "primary_category": pc, "n_categories": pa.array(pn, pa.int16()), "primary_score": pa.array(ps, pa.float32())}), o2 + ".part", compression="zstd")
93 + os.replace(o2 + ".part", o2)
94 + return name, n, len(pw)
95 +
96 +
97 +def prepare(root, db, out_dir, chunk_rows):
98 + con = duckdb.connect(db); con.execute("SET threads TO 24; SET memory_limit='60GB';")
99 + os.makedirs(os.path.join(out_dir, "input"), exist_ok=True)
100 + con.execute("""CREATE OR REPLACE TABLE cls_input AS
101 + SELECT w.id, w.publication_year AS year, w.title, a.abstract,
102 + (SELECT list(topic_id ORDER BY rank) FROM oa_work_topics t WHERE t.work_id = w.id) AS topic_ids,
103 + (SELECT list(subfield_id ORDER BY rank) FROM oa_work_topics t WHERE t.work_id = w.id) AS subfield_ids,
104 + (SELECT flatten(list(x.categories)) FROM link_arxiv_oa l JOIN arxiv_keys x USING(arxiv_id) WHERE l.oa_id = w.id) AS arxiv_cats,
105 + (SELECT list(DISTINCT d.stream) FROM link_dblp_oa l JOIN dblp_keys d USING(dblp_key) WHERE l.oa_id = w.id) AS dblp_streams
106 + FROM corpus_works w LEFT JOIN oa_abstracts a ON a.work_id = w.id""")
107 + n = con.execute("SELECT count(*) FROM cls_input").fetchone()[0]
108 + con.execute(f"""COPY (SELECT *, (row_number() OVER ()) // {chunk_rows} AS chunk FROM cls_input) TO '{out_dir}/input' (FORMAT PARQUET, PARTITION_BY (chunk), OVERWRITE_OR_IGNORE, COMPRESSION ZSTD)""")
109 + print("prepared", n, "works")
110 +
111 +
112 +def main():
113 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); ap.add_argument("--ontology", required=True)
114 + ap.add_argument("--prepare", action="store_true"); ap.add_argument("--run", action="store_true"); ap.add_argument("--workers", type=int, default=20)
115 + ap.add_argument("--chunk-rows", type=int, default=100000); ap.add_argument("--limit", type=int, default=0)
116 + a = ap.parse_args(); out_dir = os.path.join(a.root, "data/processed/classification"); db = os.path.join(a.root, "data/processed/wp44.duckdb")
117 + if a.prepare:
118 + prepare(a.root, db, out_dir, a.chunk_rows)
119 + if a.run:
120 + files = sorted(glob.glob(os.path.join(out_dir, "input", "chunk=*", "*.parquet")))
121 + files = [(f, out_dir) for f in files]
122 + if a.limit:
123 + files = files[: a.limit]
124 + # chunk files share the name data_0.parquet inside chunk=N dirs → rename key
125 + files = [(f, out_dir) for f, _ in files]
126 + t0 = time.time(); tot = 0; cl = 0
127 + with mp.Pool(a.workers, initializer=_init, initargs=(a.ontology,)) as pool:
128 + for name, n, k in pool.imap_unordered(process_chunk_named, [(f, out_dir, os.path.basename(os.path.dirname(f))) for f, _ in files]):
129 + tot += n; cl += k
130 + print(f"[{time.time()-t0:6.0f}s] {name}: {n} works, {k} classified | total {tot} / classified {cl}", flush=True)
131 + json.dump({"ontology": a.ontology, "version": load_ontology(a.ontology)["version"], "works": tot, "classified": cl, "when": dt.datetime.now(dt.UTC).isoformat(),
132 + "seconds": round(time.time() - t0)}, open(os.path.join(out_dir, "classification_report.json"), "w"), indent=1)
133 +
134 +
135 +def process_chunk_named(args):
136 + path, out_dir, chunk_name = args
137 + name = chunk_name.replace("=", "_")
138 + o1 = os.path.join(out_dir, "cats", name + ".parquet"); o2 = os.path.join(out_dir, "primary", name + ".parquet")
139 + if os.path.exists(o1) and os.path.exists(o2):
140 + return name, 0, 0
141 + # reuse process_chunk with a symlinked name
142 + tmp = os.path.join(out_dir, "input_named"); os.makedirs(tmp, exist_ok=True)
143 + link = os.path.join(tmp, name + ".parquet")
144 + if not os.path.exists(link):
145 + os.symlink(path, link)
146 + return process_chunk((link, out_dir))
147 +
148 +
149 +if __name__ == "__main__":
150 + main()
added taxonomy/src/area_ai_ml_a.py +153 −0
@@ -0,0 +1,153 @@
1 +"""Area ai_ml, part A: categories 1-4 (LLM, foundation models, multimodal AI, generative AI).
2 +
3 +Tuple conventions (converted to dicts by build_ontology.py):
4 + topics: (openalex_topic_id, weight, note)
5 + subfields:(subfield_id, weight)
6 + arxiv: (category, weight)
7 + streams: (dblp_stream_key, weight)
8 + keywords: (regex, weight, field) field in {"title", "abstract", "both"}
9 +"""
10 +
11 +CATEGORIES = [
12 +dict(
13 + id="llm", number=1, name="Large Language Models", area="ai_ml",
14 + definition=("Research whose object is a transformer-based language model trained at scale: pre-training, "
15 + "instruction tuning, prompting and in-context learning, reasoning, retrieval augmentation, efficient "
16 + "inference, evaluation and benchmarks of LLMs. Excludes classical NLP tasks solved with small task-specific "
17 + "models (nlp) and non-language foundation models (foundation_models)."),
18 + topics=[("T10028", 1.0, "OpenAlex cluster hosting BERT / pre-trained LM / RAG / QA work: the de-facto LLM topic"),
19 + ("T11714", 0.4, "multimodal LLMs and vision-language models"),
20 + ("T12031", 0.3, "dialogue systems, LLM chatbots"),
21 + ("T10181", 0.3, "NLP techniques incl. neural MT and NLG"),
22 + ("T12128", 0.3, "chatbots in service interactions"),
23 + ("T14414", 0.2, "AI in education: mostly ChatGPT studies post-2023")],
24 + subfields=[(1702, 0.2)],
25 + arxiv=[("cs.CL", 0.5), ("cs.AI", 0.2), ("cs.LG", 0.2)],
26 + streams=[("conf/acl", 0.5), ("conf/emnlp", 0.5), ("conf/naacl", 0.5), ("conf/eacl", 0.4), ("conf/coling", 0.4),
27 + ("journals/tacl", 0.5), ("conf/iclr", 0.3), ("conf/nips", 0.3), ("conf/icml", 0.3),
28 + ("journals/tmlr", 0.3), ("conf/aaai", 0.2), ("conf/nlpcc", 0.3)],
29 + keywords=[
30 + (r"(?i)\blarge language models?\b", 1.0, "both"),
31 + (r"(?i)\bLLMs?\b", 1.0, "both"),
32 + (r"(?i)\b(ChatGPT|GPT-?3(\.5)?|GPT-?4o?|GPT-?4\.\d|GPT-?5|LLaMA|Llama-? ?[234]|PaLM|InstructGPT|Qwen|Vicuna|Mixtral|DeepSeek-?(V\d|R1)|Claude ?[34]|Gemini ?(Pro|Ultra|1\.5|2))\b", 1.0, "both"),
33 + (r"(?i)\bpre-?trained language models?\b|\bPLMs?\b", 0.7, "both"),
34 + (r"(?i)\b(instruction[- ]tuning|instruction-?tuned|in-?context learning|chain[- ]of[- ]thought|retrieval[- ]augmented generation|RAG\b|prompt (engineering|tuning|learning)|prompting|RLHF|supervised fine-?tuning|SFT\b)\b", 0.8, "both"),
35 + (r"(?i)\b(masked|autoregressive|causal|decoder-?only|encoder-?decoder) language model", 0.7, "both"),
36 + (r"(?i)\b(language model|transformer) (pre-?training|scaling|scaling laws?)\b", 0.7, "both"),
37 + (r"(?i)\b(parameter-?efficient fine-?tuning|PEFT\b|LoRA\b|adapters? for (language|LLM))\b", 0.6, "both"),
38 + (r"(?i)\bBERT\b|\bRoBERTa\b|\bT5\b|\bBART\b", 0.6, "both"),
39 + (r"(?i)\bhallucinat(ion|ions|e|es|ing)\b", 0.5, "both"),
40 + (r"(?i)\blanguage models?\b", 0.4, "both"),
41 + (r"(?i)\b(tokeni[sz](er|ation)|context (window|length)|KV[- ]cache|speculative decoding|mixture[- ]of[- ]experts)\b", 0.4, "both"),
42 + ],
43 + exclusions=[],
44 + related=["foundation_models", "nlp", "generative_ai", "ai_agents", "ai_safety"],
45 + parent_of=[], child_of=["foundation_models"],
46 + provenance=("Topics chosen by inspecting OpenAlex keyword lists (T10028 lists BERT, pre-trained language models, "
47 + "retrieval-augmented generation). Venues: *CL conferences and ML venues at partial weight because LLM "
48 + "work is spread across them. Keyword patterns reach the threshold alone, as required for the six "
49 + "AI categories OpenAlex cannot separate."),
50 +),
51 +dict(
52 + id="foundation_models", number=2, name="Foundation Models", area="ai_ml",
53 + definition=("Large pre-trained models intended for adaptation to many downstream tasks in any modality: scaling "
54 + "laws, pre-training objectives, self-supervised representation learning at scale, transfer, fine-tuning "
55 + "and adapter methods, model releases and their evaluation. Language-specific work is llm (a child); "
56 + "vision-language work is multimodal_ai (a child)."),
57 + topics=[("T11307", 0.5, "domain adaptation, few-shot, parameter-efficient fine-tuning, continual learning"),
58 + ("T11714", 0.4, "vision-language models / CLIP"),
59 + ("T10028", 0.3, "pre-trained language models"),
60 + ("T10036", 0.2, "backbone networks, model compression"),
61 + ("T12422", 0.2, "foundation models in medical imaging / radiomics")],
62 + subfields=[(1702, 0.2)],
63 + arxiv=[("cs.LG", 0.4), ("cs.CV", 0.2), ("cs.CL", 0.2), ("cs.AI", 0.2)],
64 + streams=[("conf/nips", 0.3), ("conf/icml", 0.3), ("conf/iclr", 0.3), ("journals/tmlr", 0.3), ("conf/cvpr", 0.2),
65 + ("conf/iccv", 0.2), ("conf/eccv", 0.2), ("journals/natmi", 0.2)],
66 + keywords=[
67 + (r"(?i)\bfoundation models?\b|\bfoundational models?\b", 1.0, "both"),
68 + (r"(?i)\b(vision|visual|time[- ]series|protein|genomic|geospatial|medical|clinical|weather|climate|chemistry|molecular|tabular|graph|speech|audio|robotics?|world) foundation models?\b", 1.0, "both"),
69 + (r"(?i)\bscaling laws?\b", 0.8, "both"),
70 + (r"(?i)\b(self-?supervised|contrastive|masked) (pre-?training|representation learning|learning at scale)\b", 0.7, "both"),
71 + (r"(?i)\b(pre-?train(ed|ing)) (on|at) (web|internet|large|massive|billion)[- ]scale", 0.7, "both"),
72 + (r"(?i)\b(vision transformers?|ViT-?[BLHG]?|DINO(v2)?|segment anything|SAM ?2|masked auto-?encoders?|MAE\b|CLIP\b|SigLIP)\b", 0.6, "both"),
73 + (r"(?i)\b(general-?purpose|generalist|universal|all-?in-?one) (models?|agents?|representations?)\b", 0.6, "both"),
74 + (r"(?i)\b(transfer learning|fine-?tun(ing|ed?)|downstream tasks?|zero-?shot|few-?shot)\b", 0.4, "both"),
75 + (r"(?i)\b(adapters?|LoRA|parameter-?efficient|prompt tuning|emergent (abilities|capabilities))\b", 0.5, "both"),
76 + (r"(?i)\bworld models?\b", 0.5, "both"),
77 + ],
78 + exclusions=[],
79 + related=["llm", "multimodal_ai", "generative_ai", "deep_learning_architectures", "ml_theory"],
80 + parent_of=["llm", "multimodal_ai"], child_of=[],
81 + provenance=("No OpenAlex topic isolates foundation models; T11307 (PEFT, few-shot, continual) is the closest "
82 + "methodological host. Keyword patterns carry the category; the generic 'transfer learning/fine-tuning' "
83 + "terms are deliberately below threshold."),
84 +),
85 +dict(
86 + id="multimodal_ai", number=3, name="Multimodal AI", area="ai_ml",
87 + definition=("Models and systems that jointly process two or more modalities (vision-language, audio-visual, "
88 + "video-text, 3D-language, sensors with language): vision-language models and multimodal LLMs, "
89 + "captioning, visual question answering, cross-modal retrieval and grounding. Unimodal vision is "
90 + "computer_vision; pure speech is speech_audio."),
91 + topics=[("T11714", 1.0, "Multimodal Machine Learning Applications: VLMs, MLLMs, VQA, captioning"),
92 + ("T10627", 0.4, "image and video retrieval incl. cross-modal"),
93 + ("T11439", 0.2, "video analysis and summarization"),
94 + ("T10824", 0.2, "image retrieval and classification"),
95 + ("T13155", 0.2, "digital communication and language (multimodal discourse)")],
96 + subfields=[(1707, 0.1)],
97 + arxiv=[("cs.MM", 0.6), ("cs.CV", 0.3), ("cs.CL", 0.3), ("cs.LG", 0.2)],
98 + streams=[("conf/mm", 0.6), ("journals/tmm", 0.6), ("conf/icmi", 0.6), ("journals/tomccap", 0.5),
99 + ("conf/icmcs", 0.5), ("conf/mir", 0.5), ("journals/mms", 0.5), ("journals/mta", 0.3),
100 + ("conf/cvpr", 0.2), ("conf/iccv", 0.2), ("conf/eccv", 0.2), ("conf/acl", 0.2), ("conf/emnlp", 0.2)],
101 + keywords=[
102 + (r"(?i)\bmulti-?modal(ity)?\b", 0.7, "both"),
103 + (r"(?i)\b(vision[- ]language|vision[- ]and[- ]language|visual[- ]language|language[- ]vision|image[- ]text|video[- ]language|audio[- ]language|speech[- ]language|3D[- ]language|text[- ]to[- ]3D)[- ]?(models?|pre-?training|tasks?|alignment|understanding|learning|reasoning|navigation|action|retrieval|generation)?\b", 1.0, "both"),
104 + (r"(?i)\b(VLMs?|MLLMs?|LVLMs?|LMMs?|VLAs?)\b", 1.0, "both"),
105 + (r"(?i)\bvisual question answering\b|\bVQA\b|\bvideo question answering\b", 1.0, "both"),
106 + (r"(?i)\b(image|video|dense|audio) captioning\b", 1.0, "both"),
107 + (r"(?i)\bcross-?modal(ity)?\b", 0.8, "both"),
108 + (r"(?i)\b(audio-?visual|visual grounding|referring expression|visual dialog|visual reasoning|visual instruction tuning|LLaVA|BLIP-?2?|Flamingo|GPT-?4V|CLIP|ALIGN\b|CoCa)\b", 0.8, "both"),
109 + (r"(?i)\b(text-to-image|text-to-video|image-to-text|text-image|image-text) (retrieval|alignment|matching|generation|models?)\b", 0.8, "both"),
110 + (r"(?i)\b(modalit(y|ies) (fusion|alignment|gap)|(early|late|intermediate) fusion of|sensor fusion with language)\b", 0.7, "both"),
111 + ],
112 + exclusions=[r"(?i)\bmultimodal (therapy|analgesi|treatment|intervention|pain|anesthesi|anaesthesi|imaging biomarkers?|prehabilitation|transport)\b"],
113 + related=["computer_vision", "nlp", "llm", "speech_audio", "generative_ai"],
114 + parent_of=[], child_of=["foundation_models"],
115 + provenance=("T11714 is the only OpenAlex topic centred on vision-language learning. Multimedia venues at partial "
116 + "weight (they also host unimodal media work). Medical 'multimodal' senses are vetoed."),
117 +),
118 +dict(
119 + id="generative_ai", number=4, name="Generative AI", area="ai_ml",
120 + definition=("Models that synthesise new content (images, video, audio, 3D, text, code, molecules) and their training, "
121 + "sampling, evaluation, detection and misuse: GANs, VAEs, diffusion and flow models, autoregressive "
122 + "generators, text-to-X synthesis, deepfakes and AI-generated-content detection. LLM text generation as "
123 + "such is llm unless the contribution concerns the generative model or its outputs' authenticity."),
124 + topics=[("T10775", 1.0, "GANs, diffusion models, image/video generation"),
125 + ("T12357", 0.5, "digital media forensics: deepfake detection"),
126 + ("T11349", 0.2, "music technology (music generation)"),
127 + ("T10388", 0.2, "steganography / watermarking (of generated content)"),
128 + ("T14254", 0.2, "digital media and visual art (AI art)")],
129 + subfields=[],
130 + arxiv=[("cs.CV", 0.2), ("cs.LG", 0.3), ("cs.GR", 0.2), ("cs.SD", 0.1), ("cs.AI", 0.1), ("cs.MM", 0.1)],
131 + streams=[("conf/nips", 0.2), ("conf/icml", 0.2), ("conf/iclr", 0.2), ("conf/cvpr", 0.2), ("conf/iccv", 0.2),
132 + ("conf/eccv", 0.2), ("conf/siggraph", 0.2), ("conf/siggrapha", 0.2), ("journals/tog", 0.1)],
133 + keywords=[
134 + (r"(?i)\bgenerative (AI|artificial intelligence|models?|modell?ing|adversarial networks?|pre-?trained|design|art)\b|\bGenAI\b|\bAIGC\b", 1.0, "both"),
135 + (r"(?i)\bGANs?\b|\b(conditional|cycle|style|Wasserstein)GAN\b|\bStyleGAN\w*\b", 0.9, "both"),
136 + (r"(?i)\b(denoising diffusion|latent diffusion|stable diffusion|diffusion transformers?|DDPM|DDIM|score-?based generative|score matching|flow[- ]matching|rectified flow|normali[sz]ing flows?|consistency models?)\b", 1.0, "both"),
137 + (r"(?i)\bdiffusion models?\b", 0.8, "both"),
138 + (r"(?i)\btext-?to-?(image|video|3D|audio|speech|music|motion|SVG|scene)\b", 1.0, "both"),
139 + (r"(?i)\b(image|video|music|audio|speech|3D|motion|molecule|molecular|code|scene|face|avatar|talking[- ]head|layout|texture|graph) (generation|synthesis)\b", 0.7, "both"),
140 + (r"(?i)\bdeep-?fakes?\b|\bface swap(ping)?\b|\bsynthetic media\b", 1.0, "both"),
141 + (r"(?i)\bvariational auto-?encoders?\b|\bVAEs?\b|\bVQ-?VAE\b|\bVQGAN\b", 0.7, "both"),
142 + (r"(?i)\b(AI-?generated|machine-?generated|synthetic(ally)?[- ]generated|LLM-?generated) (content|images?|text|data|videos?|audio|art|code|faces)\b", 0.8, "both"),
143 + (r"(?i)\b(image (inpainting|outpainting|editing|to-image translation)|style transfer|super-?resolution|image restoration|controllable generation|ControlNet|DreamBooth|textual inversion|classifier-?free guidance|Midjourney|DALL-?E|Imagen|Sora)\b", 0.6, "both"),
144 + (r"(?i)\bautoregressive (models?|generation|image|video)\b", 0.5, "both"),
145 + ],
146 + exclusions=[r"(?is)^(?!.*(generative|GANs?\b|synthesis|generation|deepfake|VAE|AI-?generated)).*\b(drug|gas|molecular|thermal|anomalous|reaction-?|water|heat|ion|oxygen|particle|turbulent|surface|solute|vapou?r) diffusion\b",
147 + r"(?is)^(?!.*(generative|GANs?\b|synthesis|generation|deepfake|VAE|AI-?generated)).*\bdiffusion (coefficients?|tensor|MRI|weighted|equations?|process(es)?|of innovations?|limited)\b"],
148 + related=["llm", "multimodal_ai", "computer_vision", "computer_graphics", "foundation_models", "speech_audio"],
149 + parent_of=[], child_of=[],
150 + provenance=("T10775 keywords explicitly list GANs, diffusion models, Stable Diffusion, text-to-image. Physics/finance "
151 + "senses of 'diffusion' are vetoed only when no generative-modelling term is present."),
152 +),
153 +]
added taxonomy/src/area_ai_ml_b.py +205 −0
@@ -0,0 +1,205 @@
1 +"""Area ai_ml, part B: categories 5-8 and 38 (agents, RL, ML theory, DL architectures, AI safety)."""
2 +
3 +CATEGORIES = [
4 +dict(
5 + id="ai_agents", number=5, name="AI Agents and Autonomous Systems", area="ai_ml",
6 + definition=("Software agents that perceive, decide and act with some autonomy: LLM-based agents and tool use, "
7 + "multi-agent systems, planning and automated reasoning for acting, agent architectures, negotiation, "
8 + "agentic workflows and benchmarks, embodied agents in simulated environments. Physical robots are "
9 + "robotics; RL algorithms as such are reinforcement_learning."),
10 + topics=[("T10456", 1.0, "Multi-Agent Systems and Negotiation: BDI, agent architectures, argumentation"),
11 + ("T10906", 0.6, "AI-based problem solving and planning (PDDL, multi-agent planning)"),
12 + ("T11574", 0.4, "AI in games (game-playing agents)"),
13 + ("T10249", 0.3, "distributed control of multi-agent systems (control-theoretic MAS)"),
14 + ("T12128", 0.2, "chatbots and service agents")],
15 + subfields=[(1702, 0.1)],
16 + arxiv=[("cs.MA", 1.0), ("cs.AI", 0.4)],
17 + streams=[("conf/atal", 1.0), ("conf/ifaamas", 1.0), ("journals/aamas", 1.0), ("conf/icaart", 0.6),
18 + ("conf/aaai", 0.2), ("conf/ijcai", 0.2), ("journals/jair", 0.2), ("journals/ai", 0.2), ("conf/kr", 0.2)],
19 + keywords=[
20 + (r"(?i)\b(LLM|language[- ]model|AI|autonomous|intelligent|embodied|conversational|software|web|GUI|coding|browsing|tool-?using|generative|multi-?step|computer-?use|research|planning)[- ](based |powered |driven )?agents?\b", 1.0, "both"),
21 + (r"(?i)\bagentic\b|\bagent-?based (LLM|AI|systems?|frameworks?|workflows?)\b", 1.0, "both"),
22 + (r"(?i)\bmulti-?agent (systems?|reinforcement|collaboration|coordination|negotiation|planning|simulation|frameworks?|debate|learning|path finding|communication|environments?)\b|\bMAS\b|\bMARL\b", 1.0, "both"),
23 + (r"(?i)\b(tool (use|using|learning|calling|augmented|integration)|function calling|API calling|ReAct|AutoGPT|AutoGen|LangChain|LangGraph|CrewAI|MetaGPT|SWE-?agent|SWE-?bench|WebArena|AgentBench|OSWorld|Model Context Protocol|MCP servers?)\b", 1.0, "both"),
24 + (r"(?i)\b(BDI|belief-?desire-?intention) (agents?|architectures?|models?)\b|\bagent-?oriented (programming|software engineering)\b", 1.0, "both"),
25 + (r"(?i)\b(automated|classical|hierarchical|task|AI|temporal|epistemic) planning\b|\bPDDL\b|\btask (decomposition|and motion planning)\b", 0.7, "both"),
26 + (r"(?i)\bagent-?based (model(ing|s)?|simulations?)\b|\bABM\b", 0.6, "both"),
27 + (r"(?i)\b(autonomous systems?|autonomy|self-?driving|self-?improving|self-?evolving|long-?horizon (tasks?|planning)|memory (module|mechanism|augmented) (for|in) agents?)\b", 0.5, "both"),
28 + (r"(?i)\b(game[- ]playing|general game|AlphaGo|AlphaZero|MuZero|OpenAI Five|AlphaStar|CICERO|Diplomacy)\b", 0.6, "both"),
29 + ],
30 + exclusions=[r"(?i)\b(reagents?|chemical|contrast|therapeutic|antimicrobial|chelating|reducing|oxidi[sz]ing|pharmacological|anti-?cancer|infectious|causative|etiologic(al)?|biological|pathogenic|anesthetic|anaesthetic|alkylating|cross-?linking|blowing|surface-?active|wetting|coupling|cytotoxic|antiviral|antibacterial|antifungal|bleaching|dispersing|sweetening|flavou?ring|leavening|real estate|travel|insurance|customs|shipping|sports|literary|secret|double|free|special) agents?\b"],
31 + related=["llm", "reinforcement_learning", "robotics", "ai_safety", "autonomous_scientific_discovery"],
32 + parent_of=[], child_of=[],
33 + provenance=("T10456 is the classical MAS host; the LLM-agent wave has no OpenAlex topic yet, so keyword patterns on "
34 + "'X agent' (with the qualifier restricted to computing senses) and agent frameworks/benchmarks carry it. "
35 + "Chemical/medical/commercial senses of 'agent' are vetoed."),
36 +),
37 +dict(
38 + id="reinforcement_learning", number=6, name="Reinforcement Learning", area="ai_ml",
39 + definition=("Learning to act from reward signals: value- and policy-based methods, model-based RL, offline and "
40 + "inverse RL, multi-agent RL, bandits, exploration, reward design, RL from human feedback, and RL theory. "
41 + "Applications to robot control are shared with robotics."),
42 + topics=[("T10462", 1.0, "Reinforcement Learning in Robotics: Q-learning, PPO, offline RL, imitation learning"),
43 + ("T12794", 0.5, "adaptive dynamic programming control"),
44 + ("T11574", 0.3, "AI in games"),
45 + ("T10100", 0.1, "metaheuristic optimisation (RL-based hyper-heuristics)")],
46 + subfields=[],
47 + arxiv=[("cs.LG", 0.3), ("cs.AI", 0.2), ("cs.RO", 0.1), ("eess.SY", 0.1), ("cs.MA", 0.1)],
48 + streams=[("conf/nips", 0.2), ("conf/icml", 0.2), ("conf/iclr", 0.2), ("conf/corl", 0.3), ("conf/aaai", 0.1),
49 + ("journals/jmlr", 0.1), ("conf/atal", 0.2), ("journals/tnn", 0.1)],
50 + keywords=[
51 + (r"(?i)\breinforcement learning\b", 1.0, "both"),
52 + (r"(?i)\b(deep|multi-?agent|offline|online|inverse|model-?based|model-?free|safe|hierarchical|meta|batch|distributional|constrained|goal-?conditioned|preference-?based|federated|quantum)[- ]RL\b", 1.0, "both"),
53 + (r"(?i)\b(Q-?learning|DQN|double DQN|Rainbow DQN|PPO|proximal policy optimi[sz]ation|TRPO|actor-?critic|A3C|A2C|SAC\b|soft actor-?critic|TD3|DDPG|policy gradients?|REINFORCE|AlphaZero|MuZero|Dreamer(V\d)?|decision transformers?|world models? for (control|RL|planning))\b", 1.0, "both"),
54 + (r"(?i)\bMarkov decision process(es)?\b|\bMDPs?\b|\bPOMDPs?\b|\bsemi-?MDP\b|\bDec-?POMDP\b", 0.8, "both"),
55 + (r"(?i)\b(reward (functions?|shaping|models?|hacking|design|learning|signals?)|sparse rewards?|exploration[- ]exploitation|intrinsic (motivation|rewards?)|curiosity-?driven|multi-?armed bandits?|contextual bandits?|credit assignment|experience replay|sim-?to-?real)\b", 0.8, "both"),
56 + (r"(?i)\b(imitation learning|behavio(u)?r(al)? cloning|learning from demonstrations?|inverse reinforcement|apprenticeship learning|offline (policy|data) (evaluation|learning))\b", 0.8, "both"),
57 + (r"(?i)\b(RLHF|RLAIF|reinforcement learning from (human|AI) feedback|GRPO|DPO\b|direct preference optimi[sz]ation|RLVR|reasoning models? trained with|verifiable rewards?)\b", 0.8, "both"),
58 + (r"(?i)\b(temporal[- ]difference|value (functions?|iteration)|policy (iteration|evaluation|optimi[sz]ation|learning)|Bellman|dynamic programming for control|optimal control via learning|regret bounds? for (MDP|RL))\b", 0.6, "both"),
59 + (r"(?i)\b(agents?|policies|policy|environments?|episodes?|rollouts?)\b", 0.2, "both"),
60 + ],
61 + exclusions=[r"(?i)\breinforcement (of|in|for) (concrete|steel|soil|composites?|polymers?|fib(er|re)s?|masonry|beams?|slabs?|pavements?)\b(?!.*reinforcement learning)"],
62 + related=["ai_agents", "robotics", "ml_theory", "ai_safety", "llm"],
63 + parent_of=[], child_of=[],
64 + provenance="T10462 is the RL topic (its keywords list MARL, Q-learning, PPO, offline RL). Algorithm names reach the threshold alone.",
65 +),
66 +dict(
67 + id="ml_theory", number=7, name="Machine Learning Theory", area="ai_ml",
68 + definition=("Statistical and computational learning theory and principled ML methodology: generalisation and sample "
69 + "complexity, optimisation theory for learning, kernel and Bayesian methods, probabilistic models, causal "
70 + "inference, online learning, PAC learning, interpretability and fairness as formal problems. "
71 + "Architecture engineering is deep_learning_architectures."),
72 + topics=[("T12072", 1.0, "Machine Learning and Algorithms: PAC learning, generalisation bounds, VC dimension"),
73 + ("T11612", 0.8, "stochastic gradient optimisation, NTK, implicit regularisation"),
74 + ("T12814", 0.7, "Gaussian processes and Bayesian inference"),
75 + ("T11303", 0.6, "Bayesian modelling and causal inference"),
76 + ("T10845", 0.5, "advanced causal inference techniques"),
77 + ("T11901", 0.5, "Bayesian methods and mixture models"),
78 + ("T12535", 0.4, "ML and data classification (SVM, ensembles)"),
79 + ("T10637", 0.4, "clustering algorithms"),
80 + ("T12026", 0.4, "explainable AI"),
81 + ("T12536", 0.4, "topological and geometric data analysis"),
82 + ("T11652", 0.3, "imbalanced classification"),
83 + ("T11689", 0.3, "adversarial robustness (certified robustness)"),
84 + ("T10963", 0.3, "advanced optimisation algorithms"),
85 + ("T12761", 0.3, "data stream mining (concept drift)"),
86 + ("T12205", 0.3, "time series forecasting"),
87 + ("T13373", 0.2, "data mining and ML applications"),
88 + ("T12676", 0.2, "extreme learning machines")],
89 + subfields=[(1702, 0.3)],
90 + arxiv=[("stat.ML", 0.8), ("cs.LG", 0.4), ("math.ST", 0.3), ("math.OC", 0.2)],
91 + streams=[("conf/colt", 1.0), ("conf/alt", 1.0), ("journals/jmlr", 0.8), ("conf/aistats", 0.8), ("conf/uai", 0.8),
92 + ("journals/ml", 0.6), ("journals/tmlr", 0.4), ("conf/nips", 0.3), ("conf/icml", 0.3), ("conf/iclr", 0.2),
93 + ("journals/tnn", 0.3), ("journals/nn", 0.3), ("journals/pr", 0.2), ("conf/pkdd", 0.3), ("conf/kdd", 0.2),
94 + ("conf/icdm", 0.2), ("journals/pami", 0.2), ("journals/ijar", 0.4), ("journals/sac", 0.4),
95 + ("journals/csda", 0.3), ("journals/ml", 0.6), ("journals/cstat", 0.2), ("journals/eswa", 0.1),
96 + ("journals/kbs", 0.1), ("journals/isci", 0.1), ("journals/ijon", 0.2), ("journals/nca", 0.1),
97 + ("journals/mlc", 0.3), ("journals/apin", 0.2), ("journals/air", 0.2)],
98 + keywords=[
99 + (r"(?i)\b(generali[sz]ation (bounds?|error|gap|theory)|sample complexity|regret (bounds?|minimi[sz]ation|analysis)|minimax (rates?|optimal|lower)|PAC(-Bayes(ian)?)?( learning| bounds?)?|Rademacher|VC[- ]dimension|convergence (rates?|guarantees?|analysis)|statistical learning theory|computational learning theory|learnability|uniform convergence|excess risk)\b", 0.9, "both"),
100 + (r"(?i)\b(kernel (methods?|machines?|ridge|mean embedding)|reproducing kernel|RKHS|Gaussian process(es)?|variational inference|Bayesian (optimi[sz]ation|neural networks?|inference|deep learning|nonparametric)|probabilistic (models?|programming|graphical)|graphical models?|expectation[- ]maximi[sz]ation|Markov chain Monte Carlo|MCMC|Hamiltonian Monte Carlo|normali[sz]ing flows? for inference|conformal prediction|calibration)\b", 0.8, "both"),
101 + (r"(?i)\bcausal (inference|discovery|representation learning|effect estimation|structure learning|machine learning)\b|\bcounterfactual (reasoning|inference|fairness)\b|\bstructural causal models?\b|\btreatment effect\b", 0.8, "both"),
102 + (r"(?i)\b(online learning|bandits?|stochastic (gradient|optimi[sz]ation|approximation)|non-?convex optimi[sz]ation|convex optimi[sz]ation for learning|implicit (bias|regulari[sz]ation)|gradient (descent|flow) dynamics|saddle points?|loss landscapes?|mirror descent|optimal transport|Wasserstein)\b", 0.6, "both"),
103 + (r"(?i)\b(double descent|neural tangent kernel|NTK\b|lottery ticket|over-?parameteri[sz](ed|ation)|benign overfitting|grokking|feature learning (regime|theory)|mean-?field (limit|theory) of neural|infinite-?width|random features?)\b", 1.0, "both"),
104 + (r"(?i)\b(interpretab(le|ility)|explainab(le|ility)) (machine learning|AI|models?|methods?)\b|\bXAI\b|\bSHAP\b|\bLIME\b|\bfeature attribution\b|\bsaliency maps? for\b", 0.5, "both"),
105 + (r"(?i)\b(fairness (constraints?|metrics?|in machine learning)|algorithmic fairness|demographic parity|equali[sz]ed odds|fair (classification|representation))\b", 0.5, "both"),
106 + (r"(?i)\b(semi-?supervised|self-?supervised|unsupervised|active|transfer|meta|continual|lifelong|few-?shot|zero-?shot|multi-?task|weakly[- ]supervised|positive-?unlabeled|contrastive|representation|manifold|metric|dictionary|ensemble|boosting|bagging) learning\b", 0.4, "both"),
107 + (r"(?i)\b(distribution shift|out-?of-?distribution|domain generali[sz]ation|covariate shift|label noise|robust(ness)? (to|under|against) (noise|shift|corruption)|concept drift|anomaly detection|outlier detection|imbalanced (data|learning|classification)|dimensionality reduction|matrix (completion|factori[sz]ation)|tensor (decomposition|methods?)|sparse (coding|recovery)|compressed sensing)\b", 0.5, "both"),
108 + (r"(?i)\b(support vector machines?|SVMs?|random forests?|gradient boosting|XGBoost|LightGBM|k-?means|spectral clustering|Gaussian mixture|hidden Markov models?|HMMs?|logistic regression|decision trees?|nearest neighbou?rs?|kNN)\b", 0.5, "both"),
109 + (r"(?i)\bmachine learning\b|\bstatistical learning\b|\blearning theory\b", 0.4, "both"),
110 + ],
111 + exclusions=[],
112 + related=["deep_learning_architectures", "reinforcement_learning", "algorithms", "computational_complexity", "ai_safety", "foundation_models"],
113 + parent_of=[], child_of=[],
114 + provenance=("Hosts classical ML and its theory: OpenAlex splits them across T12072 (learning theory), T11612 (optimisation), "
115 + "T12814/T11303/T11901 (Bayesian), T12535/T10637 (classical methods). Venue weights follow venue purity "
116 + "(COLT/ALT/JMLR/AISTATS/UAI strong; NeurIPS/ICML partial)."),
117 +),
118 +dict(
119 + id="deep_learning_architectures", number=8, name="Deep Learning Architectures", area="ai_ml",
120 + definition=("Design, training and compression of neural network architectures as such: CNNs, RNNs, transformers and "
121 + "attention, graph neural networks, state-space models, normalisation and optimisers in practice, neural "
122 + "architecture search, pruning, quantisation and distillation, neuromorphic and spiking networks as models. "
123 + "Task-specific results go to the application category; hardware goes to ai_accelerators."),
124 + topics=[("T10320", 1.0, "Neural Networks and Applications: training, architectures, activation functions"),
125 + ("T11273", 1.0, "Advanced Graph Neural Networks"),
126 + ("T10036", 0.5, "Advanced Neural Network Applications: YOLO, lightweight networks, model compression"),
127 + ("T12676", 0.5, "extreme learning machines"),
128 + ("T12611", 0.4, "reservoir computing / echo state networks"),
129 + ("T11612", 0.3, "DNN training optimisation"),
130 + ("T11347", 0.3, "neural network stability and synchronisation"),
131 + ("T11307", 0.3, "PEFT and continual learning for deep nets"),
132 + ("T12205", 0.2, "time-series forecasting with deep nets"),
133 + ("T10502", 0.2, "neuromorphic computing (spiking models)"),
134 + ("T11206", 0.2, "model reduction and neural networks")],
135 + subfields=[(1702, 0.3)],
136 + arxiv=[("cs.NE", 1.0), ("cs.LG", 0.5), ("cs.CV", 0.1), ("cs.AI", 0.1)],
137 + streams=[("conf/iclr", 0.5), ("conf/nips", 0.4), ("conf/icml", 0.4), ("journals/tnn", 0.6), ("journals/nn", 0.6),
138 + ("conf/ijcnn", 0.6), ("journals/ijon", 0.5), ("journals/nca", 0.5), ("conf/icann", 0.6), ("conf/iconip", 0.5),
139 + ("journals/npl", 0.5), ("conf/esann", 0.5), ("journals/tmlr", 0.3), ("journals/pami", 0.2),
140 + ("journals/cin", 0.3), ("journals/tetci", 0.3), ("journals/tai", 0.3), ("journals/ficn", 0.2),
141 + ("journals/cogcom", 0.2), ("journals/aisy", 0.1), ("conf/aaai", 0.1)],
142 + keywords=[
143 + (r"(?i)\b(deep|convolutional|recurrent|graph|spiking|artificial|feed-?forward|residual|siamese|capsule|physics-?informed|liquid|binary|quanti[sz]ed|Hopfield|Kolmogorov-?Arnold) neural networks?\b", 0.8, "both"),
144 + (r"(?i)\b(CNNs?|RNNs?|LSTMs?|GRUs?|GNNs?|GCNs?|GATs?|SNNs?|MLPs?|DNNs?|ANNs?|KANs?|ResNets?|DenseNet|EfficientNet|MobileNet|U-?Net|ViTs?|Swin)\b", 0.7, "both"),
145 + (r"(?i)\btransformers?\b", 0.5, "both"),
146 + (r"(?i)\b(self-?|cross-?|multi-?head |sparse |linear |flash)?attention (mechanisms?|heads?|layers?|maps?|weights)\b|\battention is all\b", 0.7, "both"),
147 + (r"(?i)\bneural architecture search\b|\bNAS\b.*\b(architecture|search space|supernet)\b|\bAutoML\b|\bdifferentiable architecture\b", 0.9, "both"),
148 + (r"(?i)\b(model compression|knowledge distillation|network pruning|structured pruning|unstructured pruning|post-?training quanti[sz]ation|quanti[sz]ation-?aware training|low-?rank (factori[sz]ation|approximation) of (weights|layers)|weight sharing|parameter sharing|lightweight (networks?|architectures?|models?)|efficient (architectures?|networks?|transformers?))\b", 0.8, "both"),
149 + (r"(?i)\b(batch|layer|group|instance|weight|spectral) normali[sz]ation\b|\bdropout\b|\bresidual (connections?|learning|blocks?)\b|\bskip connections?\b|\bactivation functions?\b|\bReLU\b|\bGELU\b|\bSwiGLU\b|\bsoftmax\b", 0.7, "both"),
150 + (r"(?i)\b(mixture[- ]of[- ]experts|MoE\b|state[- ]space models?|SSMs?\b|Mamba|S4\b|RWKV|Hyena|linear (RNNs?|transformers?)|positional (encodings?|embeddings?)|RoPE|rotary|hypernetworks?|neural ODEs?|implicit (layers|neural representations?)|capsule networks?|equivariant (networks?|neural)|geometric deep learning|message passing (neural|networks?)|graph (convolution|attention|transformers?|pooling)|over-?smoothing)\b", 0.9, "both"),
151 + (r"(?i)\b(backpropagation|back-?prop|gradient (vanishing|exploding)|learning rate (schedul|warm-?up)|Adam(W)?\b|optimi[sz]ers? for (deep|neural)|weight (decay|initiali[sz]ation)|training (dynamics|instabilit)|loss functions? for|curriculum learning|data augmentation|mixup|CutMix|label smoothing|early stopping|hyper-?parameter (tuning|optimi[sz]ation))\b", 0.7, "both"),
152 + (r"(?i)\bdeep learning\b|\bneural networks?\b|\bneural nets?\b", 0.5, "both"),
153 + (r"(?i)\b(spiking|neuromorphic|reservoir computing|echo state|extreme learning machines?|ELM\b)\b", 0.5, "both"),
154 + ],
155 + exclusions=[r"(?is)^(?!.*(neural|deep learning|attention|language model|vision|encoder|decoder)).*\b(power|distribution|current|voltage|instrument|high-?voltage|dry-?type|oil-?immersed|pulse|flyback|isolation|phase-?shifting|step-?(up|down)) transformers?\b"],
156 + related=["ml_theory", "computer_vision", "nlp", "ai_accelerators", "foundation_models", "llm"],
157 + parent_of=[], child_of=[],
158 + provenance=("T10320 and T11273 are architecture-centred OpenAlex topics; cs.NE is the arXiv home of neural and "
159 + "evolutionary computation. 'Transformer' alone is weak and electrical senses are vetoed when no "
160 + "learning term co-occurs."),
161 +),
162 +dict(
163 + id="ai_safety", number=38, name="AI Safety and Alignment", area="ai_ml",
164 + definition=("Making AI systems safe, aligned, robust, fair, interpretable and governable: alignment and preference "
165 + "learning framed as a safety problem, red-teaming and jailbreaks, adversarial robustness and backdoors, "
166 + "hallucination and truthfulness, bias and fairness, mechanistic interpretability, evaluation of dangerous "
167 + "capabilities, AI governance and regulation. Generic system security is cybersecurity; XAI as a statistical "
168 + "method only is ml_theory."),
169 + topics=[("T10883", 1.0, "Ethics and Social Impacts of AI: AI safety, alignment, fairness, governance"),
170 + ("T11689", 0.6, "adversarial robustness, backdoors, prompt injection"),
171 + ("T12026", 0.5, "explainable AI / mechanistic interpretability"),
172 + ("T13851", 0.4, "law, AI and intellectual property"),
173 + ("T12262", 0.3, "hate speech and cyberbullying detection (harm mitigation)"),
174 + ("T11147", 0.3, "misinformation and its impacts"),
175 + ("T10525", 0.2, "human-automation interaction and safety"),
176 + ("T14307", 0.1, "bioethics and human rights")],
177 + subfields=[],
178 + arxiv=[("cs.CY", 0.5), ("cs.AI", 0.2), ("cs.LG", 0.1), ("cs.CL", 0.1), ("cs.CR", 0.1)],
179 + streams=[("conf/fat", 1.0), ("conf/aies", 1.0), ("journals/aiethics", 1.0), ("conf/satml", 1.0),
180 + ("journals/ais", 0.5), ("journals/natmi", 0.2), ("conf/nips", 0.1), ("conf/iclr", 0.1), ("conf/icml", 0.1),
181 + ("journals/tai", 0.2), ("journals/dai", 0.1), ("conf/sp", 0.1), ("conf/ccs", 0.1)],
182 + keywords=[
183 + (r"(?i)\bAI (safety|alignment|governance|regulation|ethics|risks?|act|auditing|accountability|harms?|misuse|incidents?|oversight|assurance|red[- ]teaming)\b", 1.0, "both"),
184 + (r"(?i)\b(aligned|alignment of|align(ing)?|misaligned|misalignment of) (large )?(language models?|LLMs?|AI (systems?|models?)|foundation models?|agents?)\b|\bsuper-?alignment\b|\bvalue alignment\b|\bhuman values?\b", 1.0, "both"),
185 + (r"(?i)\b(jail-?break(s|ing)?|red[- ]teaming|prompt injection|backdoor (attacks?|defenses?)|data poisoning|model (stealing|extraction|inversion)|membership inference|adversarial (prompts?|suffixes?)|refusal|safety (training|fine-?tuning|guardrails?|alignment|benchmarks?|evaluations?)|guardrails?|harmful (content|outputs?|responses?|requests?)|unsafe (outputs?|behavio(u)?rs?))\b", 1.0, "both"),
186 + (r"(?i)\badversarial (examples?|attacks?|robustness|training|perturbations?|defenses?|vulnerabilit)\b|\bcertified robustness\b|\brobustness (certification|verification)\b", 0.8, "both"),
187 + (r"(?i)\b(algorithmic|AI|model|machine learning|ML|LLM|gender|racial|demographic|social) (fairness|bias(es)?|discrimination|transparency|accountability|audits?|auditing)\b|\bbias (mitigation|amplification|in (LLMs?|language models?|AI))\b|\bdebiasing\b", 1.0, "both"),
188 + (r"(?i)\b(responsible|trustworthy|ethical|safe|harmless|honest|helpful|human-?centered|human-?compatible|beneficial) (AI|artificial intelligence|machine learning|LLMs?|agents?)\b", 1.0, "both"),
189 + (r"(?i)\b(mechanistic interpretability|sparse auto-?encoders? (for|on) (LLM|language|features)|activation (steering|patching)|circuit discovery|probing classifiers?|representation engineering|superposition hypothesis|induction heads|feature visuali[sz]ation|concept (bottleneck|activation vectors?)|model editing|knowledge editing)\b", 1.0, "both"),
190 + (r"(?i)\b(hallucinations?|truthfulness|factuality|sycophancy|deceptive alignment|deception|reward hacking|specification gaming|goal misgeneralization|scalable oversight|weak-?to-?strong|constitutional AI|RLHF|preference (learning|optimi[sz]ation)|human feedback|over-?optimi[sz]ation|situational awareness|sandbagging|power-?seeking|instrumental convergence|corrigib)\b", 0.8, "both"),
191 + (r"(?i)\b(existential risks?|catastrophic risks?|x-?risk|frontier (AI|models?)|dangerous capabilit(y|ies)|misuse of AI|dual-?use|AGI (safety|risk|alignment)|superintelligen|loss of control|AI (doom|pause)|compute governance|responsible scaling|model (cards?|evaluations? for safety)|capability evaluations?|system cards?)\b", 1.0, "both"),
192 + (r"(?i)\b(toxic(ity)?|hate speech|harmful|offensive|abusive) (language|content|speech|text|generation|detection)\b|\bcontent moderation\b|\b(misinformation|disinformation|fake news) (detection|generation|and LLMs?)\b", 0.5, "both"),
193 + (r"(?i)\b(watermark(ing)?|detect(ing|ion) of|detecting) (LLM|AI|machine)-?generated (text|content|images?)\b|\bAI-?text detection\b", 0.8, "both"),
194 + (r"(?i)\b(machine unlearning|right to be forgotten|differential privacy for (LLM|language)|memori[sz]ation (in|of) (LLM|language models?)|training data extraction)\b", 0.5, "both"),
195 + (r"(?i)\b(uncertainty (quantification|estimation|calibration)|out-?of-?distribution detection|distribution shift|safe (reinforcement learning|RL|exploration)|runtime monitoring of (neural|ML)|verification of neural networks?)\b", 0.4, "both"),
196 + (r"(?i)\b(ethics|ethical|societal (impact|implications)|social (impact|implications)|regulation|policy|governance|accountability|transparency|explainability|interpretability|fairness|bias|safety|risk)\b", 0.2, "both"),
197 + ],
198 + exclusions=[r"(?i)\b(multiple sequence|read|genome|sequence|image|spinal|wheel|dental|teeth|occlusal|fib(er|re)|beam|optical|antenna|shaft|laser) alignment\b(?!.*(AI|LLM|language model|RLHF|human values))"],
199 + related=["llm", "ml_theory", "cybersecurity", "privacy_preserving_computation", "ai_agents", "hci", "reinforcement_learning"],
200 + parent_of=[], child_of=[],
201 + provenance=("T10883 (field 33, Social Sciences) explicitly lists 'AI safety', 'AI alignment', 'algorithmic fairness'; "
202 + "T11689 and T12026 host the technical robustness/interpretability halves. FAccT, AIES, SaTML and AI & Ethics "
203 + "are the dedicated venues; the generic ethics/governance words are kept far below threshold."),
204 +),
205 +]
added taxonomy/src/area_data_a.py +126 −0
@@ -0,0 +1,126 @@
1 +"""Area data_distributed, part A: categories 20-22 (databases, information retrieval, distributed computing)."""
2 +
3 +CATEGORIES = [
4 +dict(
5 + id="database_systems", number=20, name="Database Systems", area="data_distributed",
6 + definition=("Storage, querying and management of structured and semi-structured data: query processing and optimisation, "
7 + "transactions and concurrency control, indexing and storage engines, data integration, cleaning and provenance, "
8 + "big-data processing frameworks, data mining as a data-management problem, learned database components, "
9 + "vector and graph databases. Ranking of unstructured documents is information_retrieval."),
10 + topics=[("T10317", 1.0, "advanced database systems and queries"), ("T12761", 0.6, "data stream mining"),
11 + ("T11719", 0.6, "data quality and management"), ("T14280", 0.6, "big data technologies and applications"),
12 + ("T10538", 0.5, "data mining algorithms and applications"), ("T11181", 0.5, "advanced data storage technologies"),
13 + ("T12601", 0.5, "web applications and data management"), ("T11106", 0.5, "data management and algorithms"),
14 + ("T10772", 0.4, "distributed systems and fault tolerance (distributed databases, transactions)"),
15 + ("T13373", 0.4, "data mining and ML applications"), ("T11891", 0.4, "big data and business intelligence"),
16 + ("T10215", 0.4, "semantic web and ontologies (RDF, SPARQL)"), ("T12016", 0.4, "web data mining"),
17 + ("T14435", 0.4, "information retrieval and data mining"), ("T11937", 0.3, "research data management"),
18 + ("T13647", 0.3, "AI and big data applications"), ("T14347", 0.2, "big data and digital economy"),
19 + ("T10101", 0.2, "cloud computing (MapReduce, Hadoop)"), ("T10757", 0.2, "geographic information systems"),
20 + ("T13832", 0.2, "advanced decision-making techniques (DSS)"), ("T14484", 0.2, "technology and data analysis")],
21 + subfields=[(1710, 0.1)],
22 + arxiv=[("cs.DB", 1.0)],
23 + streams=[("conf/sigmod", 1.0), ("journals/pvldb", 1.0), ("journals/pacmmod", 1.0), ("conf/icde", 1.0), ("conf/edbt", 1.0),
24 + ("conf/pods", 1.0), ("journals/tods", 1.0), ("journals/vldb", 1.0), ("conf/dasfaa", 1.0), ("conf/vldb", 0.8),
25 + ("journals/tkde", 0.8), ("journals/tbd", 0.6), ("conf/apweb", 0.6), ("conf/icdm", 0.5), ("conf/pakdd", 0.5),
26 + ("journals/tkdd", 0.5), ("journals/kais", 0.5), ("conf/bigdataconf", 0.5), ("journals/jbd", 0.5), ("conf/gis", 0.5),
27 + ("conf/cikm", 0.4), ("conf/kdd", 0.4), ("journals/ijdsa", 0.4), ("journals/data", 0.4), ("conf/pkdd", 0.3),
28 + ("journals/www", 0.3), ("journals/tgis", 0.3), ("journals/gis", 0.3), ("journals/biodb", 0.3),
29 + ("conf/semweb", 0.3), ("conf/esws", 0.3), ("journals/dss", 0.2), ("journals/isf", 0.1), ("journals/tist", 0.2),
30 + ("journals/kbs", 0.1), ("conf/ksem", 0.2)],
31 + keywords=[
32 + (r"(?i)\bdatabases?\b|\bDBMSs?\b|\bRDBMSs?\b|\bdatabase (systems?|management|engines?|design|theory|queries|tuning|research)\b", 0.8, "both"),
33 + (r"(?i)\b(query (processing|optimi[sz]ation|execution|languages?|engines?|compilation|plans?|rewriting|answering over)|SQL\b|NoSQL|NewSQL|relational (model|algebra|databases?|queries|tables?)|transaction(al)? (processing|management|systems?)|ACID\b|OLTP|OLAP|HTAP|index(ing|es) (structures?|for|techniques)|B\+?-?trees?|learned (index(es)?|cardinality|query optimi[sz]ers?|database)|key-?value stores?|column(ar)? (stores?|storage|formats?)|data (warehous(e|es|ing)|lakes?|lakehouses?|integration|cleaning|quality|provenance|lineage|wrangling|catalogs?|cubes?|partitioning|versioning|discovery|marketplaces?)|schema (matching|evolution|mapping|inference)|entity (resolution|matching)|record linkage|join (algorithms?|processing|ordering|optimi[sz]ation)|cardinality estimation|selectivity estimation|data management|graph databases?|property graphs?|spatial (databases?|indexing|queries|joins?)|spatio-?temporal (databases?|queries|indexing)|time[- ]series databases?|vector (databases?|search engines?|indexes?)|approximate nearest neighbou?r search|similarity search|top-?k queries|skyline queries|query by example|data lakes?|metadata management|Datalog engines?)\b", 1.0, "both"),
34 + (r"(?i)\b(data mining|frequent (patterns?|itemsets?|subgraphs?)|association rules?|knowledge discovery in databases|KDD\b|sequential pattern mining|stream (processing|mining)|data streams?|sliding windows?|big data (analytics|processing|systems?|platforms?|frameworks?|management)|Hadoop|HDFS|Apache Spark|Spark SQL|MapReduce|Flink|Hive|Presto|Trino|Kafka|ETL\b|ELT\b|text-?to-?SQL|NL2SQL|natural language interfaces? to databases?|NLIDB|LLMs? for (data|databases?|SQL|tables?)|table (question answering|understanding|representation))\b", 0.7, "both"),
35 + (r"(?i)\b(PostgreSQL|Postgres|MySQL|SQLite|DuckDB|MongoDB|Cassandra|RocksDB|LevelDB|Neo4j|ClickHouse|Snowflake|BigQuery|Redshift|TiDB|CockroachDB|Spanner|DynamoDB|LSM-?trees?|storage engines?|buffer (pool|management)|concurrency control|MVCC|multi-?version|crash recovery|write-?ahead log(ging)?|WAL\b|checkpointing in databases|isolation levels?|serializab(le|ility)|snapshot isolation|two-?phase (locking|commit)|optimistic concurrency|deterministic databases?|in-?memory databases?|main-?memory databases?|disaggregated (storage|memory) for databases?|cloud(-?native)? databases?|database-?as-?a-?service|DBaaS|multi-?model databases?|time-?travel queries|temporal databases?)\b", 1.0, "both"),
36 + (r"(?i)\b(data|queries|query|tables?|records?|tuples?|schemas?|indexes|indices)\b", 0.2, "both"),
37 + ],
38 + exclusions=[r"(?is)^(?!.*(query processing|SQL|transaction|index(ing)?|storage engine|DBMS|data management|data integration|schema|NoSQL|query optimi)).*\b(PubMed|Embase|Scopus|Web of Science|Cochrane|MEDLINE|CINAHL|PsycINFO) (databases?|were searched|search)\b"],
39 + related=["information_retrieval", "distributed_computing", "cloud_computing", "algorithms", "ml_theory"],
40 + parent_of=[], child_of=[],
41 + provenance=("T10317 is the OpenAlex database topic; data-mining and big-data topics (1710, 1711, 18xx) shared at 0.4-0.6. "
42 + "SIGMOD/PVLDB/PACMMOD/ICDE/EDBT/PODS/TODS/VLDBJ pure; CIKM/KDD shared with IR and ML. Systematic-review "
43 + "mentions of bibliographic databases are vetoed when no DB-systems term co-occurs."),
44 +),
45 +dict(
46 + id="information_retrieval", number=21, name="Information Retrieval", area="data_distributed",
47 + definition=("Finding and ranking relevant information for a user need: search engines and ranking models, dense and "
48 + "sparse retrieval, recommender systems, query understanding, evaluation and test collections, web search and "
49 + "mining, scholarly and digital-library retrieval, retrieval components of RAG systems. Structured query "
50 + "processing is database_systems; text understanding tasks are nlp."),
51 + topics=[("T10286", 1.0, "information retrieval and search behavior"), ("T10203", 1.0, "recommender systems"),
52 + ("T13274", 0.8, "expert finding and Q&A systems"), ("T14435", 0.6, "information retrieval and data mining"),
53 + ("T13976", 0.5, "web visibility and informetrics"), ("T12016", 0.5, "web data mining and analysis"),
54 + ("T10627", 0.4, "advanced image and video retrieval"), ("T10824", 0.3, "image retrieval and classification"),
55 + ("T10028", 0.3, "topic modeling / RAG / sentence retrieval"), ("T10215", 0.3, "semantic web and ontologies"),
56 + ("T13673", 0.2, "library science and information"), ("T11937", 0.2, "research data management"),
57 + ("T11813", 0.2, "library collection development and digital resources"), ("T12863", 0.2, "web and library services"),
58 + ("T10664", 0.2, "sentiment analysis and opinion mining"), ("T11644", 0.2, "spam and phishing detection"),
59 + ("T14330", 0.2, "library science and information systems"), ("T13166", 0.2, "information science and libraries"),
60 + ("T13874", 0.2, "information architecture and usability"), ("T11122", 0.2, "online learning and analytics (recommendation)")],
61 + subfields=[(1710, 0.1)],
62 + arxiv=[("cs.IR", 1.0), ("cs.DL", 0.6), ("cs.SI", 0.3)],
63 + streams=[("conf/sigir", 1.0), ("conf/wsdm", 1.0), ("conf/ecir", 1.0), ("conf/recsys", 1.0), ("journals/tois", 1.0),
64 + ("journals/ipm", 0.8), ("conf/clef", 0.8), ("conf/jcdl", 0.8), ("conf/um", 0.8), ("conf/cikm", 0.6), ("conf/www", 0.6),
65 + ("journals/www", 0.6), ("journals/jasis", 0.6), ("conf/asist", 0.6), ("journals/scientometrics", 0.5),
66 + ("conf/semweb", 0.5), ("conf/esws", 0.5), ("conf/webi", 0.5), ("journals/kais", 0.3), ("conf/kdd", 0.3),
67 + ("conf/icwsm", 0.3), ("journals/tist", 0.3), ("journals/tkde", 0.2), ("journals/snam", 0.2), ("conf/asunam", 0.2),
68 + ("journals/tcss", 0.1), ("journals/ijinfoman", 0.2), ("journals/jkm", 0.1), ("journals/iam", 0.1)],
69 + keywords=[
70 + (r"(?i)\binformation retrieval\b|\bIR (systems?|models?|evaluation|research|community)\b|\bad[- ]hoc retrieval\b|\bretrieval (models?|systems?|effectiveness|augmented)\b", 1.0, "both"),
71 + (r"(?i)\b(search engines?|web search|enterprise search|product search|query (expansion|reformulation|understanding|suggestion|logs?|auto-?completion|intent)|ranking (models?|functions?|losses?|metrics)|learning[- ]to[- ]rank|LTR\b|dense (retrieval|passage retrieval|retrievers?)|sparse retrieval|lexical retrieval|hybrid retrieval|BM25|TF-?IDF|neural (retrieval|rankers?|IR|re-?ranking)|re-?rank(ing|ers?)|cross-?encoders?|bi-?encoders?|dual encoders?|ColBERT|late interaction|SPLADE|DPR\b|document (ranking|retrieval|expansion)|passage (ranking|retrieval)|relevance (judgments?|feedback|assessment|modeling)|click (models?|logs?|through)|conversational search|interactive (IR|retrieval)|session search|evaluation (campaigns?|measures?)|nDCG|MRR\b|mean reciprocal rank|MAP@|test collections?|TREC\b|MS ?MARCO|BEIR|pooling bias|query performance prediction|retrieval-?augmented|RAG (systems?|pipelines?)|retriever|generative (retrieval|recommendation))\b", 1.0, "both"),
72 + (r"(?i)\brecommend(er|ation) (systems?|models?|algorithms?|engines?|tasks?|lists?|quality)\b|\brecommenders?\b|\bcollaborative filtering\b|\b(sequential|session-?based|conversational|cross-?domain|cold-?start|news|music|point-?of-?interest|POI|job|group|bundle|LLM-?based|explainable|fair|multi-?stakeholder) recommendation\b|\bclick-?through rate\b|\bCTR prediction\b|\bimplicit feedback\b|\buser (modeling|preferences?|interests?) (for|in) recommendation\b|\bmatrix factori[sz]ation\b|\bitem (embeddings?|cold start)\b|\bMovieLens\b|\bNetflix Prize\b", 1.0, "both"),
73 + (r"(?i)\b(knowledge graphs?|entity linking|entity search|semantic web|linked (open )?data|SPARQL|RDF\b|ontolog(y|ies) (matching|alignment|engineering)|schema\.org|Wikidata|DBpedia|question answering over knowledge)\b", 0.5, "both"),
74 + (r"(?i)\b(digital librar(y|ies)|scholarly (search|recommendation|data|documents?|knowledge graphs?)|citation (analysis|networks?|recommendation|counts?|indexing)|bibliometric|scientometric|altmetric|h-?index|peer[- ]review (process|quality)|web (crawl(ing|ers?)|archiv(es?|ing)|mining)|social media (mining|analytics|retrieval)|hashtags?|microblog|user-?generated content|community question answering|CQA\b|expert finding|e-?commerce search|sponsored search|computational advertising|ad (auctions?|ranking))\b", 0.6, "both"),
75 + (r"(?i)\b(retrieval|search|ranking|recommendation|relevance|queries|query|users?' (information|search))\b", 0.3, "both"),
76 + ],
77 + exclusions=[],
78 + related=["nlp", "database_systems", "llm", "hci", "ml_theory"],
79 + parent_of=[], child_of=[],
80 + provenance=("T10286 (search behaviour) and T10203 (recommenders) are the OpenAlex IR hosts; library/information-science "
81 + "topics of 1710 included at 0.2 because they share venues (JASIST, ASIS&T). SIGIR/WSDM/ECIR/RecSys/TOIS pure."),
82 +),
83 +dict(
84 + id="distributed_computing", number=22, name="Distributed Computing", area="data_distributed",
85 + definition=("Computation across multiple processors or machines: distributed algorithms and consensus, fault tolerance "
86 + "and replication, parallel programming and HPC systems software, cluster and grid resource management, "
87 + "distributed storage and middleware, distributed and parallel ML training systems. Cloud service models are "
88 + "cloud_computing and edge placement is edge_computing (both children)."),
89 + topics=[("T10715", 1.0, "distributed and parallel computing systems (grid, cluster, scheduling)"),
90 + ("T10772", 1.0, "distributed systems and fault tolerance (consensus, replication, transactions)"),
91 + ("T10742", 0.8, "peer-to-peer network technologies"), ("T10054", 0.6, "parallel computing (HPC, MPI, GPU)"),
92 + ("T10829", 0.5, "interconnection networks and systems"), ("T11181", 0.5, "advanced data storage technologies"),
93 + ("T10101", 0.4, "cloud computing and resource management"), ("T12127", 0.4, "software system performance and reliability"),
94 + ("T10679", 0.3, "service-oriented architecture"), ("T10273", 0.3, "IoT and edge/fog computing"),
95 + ("T10249", 0.3, "distributed control multi-agent systems"), ("T12203", 0.3, "mobile agent-based network management"),
96 + ("T11986", 0.3, "scientific workflows"), ("T12879", 0.2, "distributed sensor networks and detection"),
97 + ("T12216", 0.2, "network time synchronization"), ("T11478", 0.2, "caching and content delivery")],
98 + subfields=[(1705, 0.1)],
99 + arxiv=[("cs.DC", 1.0), ("cs.PF", 0.5), ("cs.OS", 0.2), ("cs.NI", 0.1)],
100 + streams=[("journals/tpds", 1.0), ("journals/jpdc", 1.0), ("conf/ipps", 1.0), ("conf/podc", 1.0), ("conf/spaa", 1.0),
101 + ("conf/opodis", 1.0), ("conf/europar", 1.0), ("conf/ics", 1.0), ("conf/hpdc", 1.0), ("conf/icdcs", 1.0),
102 + ("conf/middleware", 1.0), ("conf/cluster", 1.0), ("conf/icpads", 1.0), ("conf/ica3pp", 1.0), ("conf/ispa", 1.0),
103 + ("conf/sc", 0.8), ("conf/ccgrid", 0.8), ("conf/hpcc", 0.8), ("conf/ppopp", 0.8), ("journals/concurrency", 0.8),
104 + ("journals/tjs", 0.8), ("journals/cluster", 0.8), ("journals/ijhpca", 0.8), ("journals/scpe", 0.8),
105 + ("journals/fgcs", 0.6), ("conf/eurosys", 0.6), ("journals/ppna", 0.6), ("conf/aina", 0.6), ("conf/nsdi", 0.5),
106 + ("conf/dsn", 0.5), ("conf/socc", 0.5), ("conf/cloud", 0.5), ("journals/tos", 0.5), ("conf/fast", 0.5),
107 + ("conf/osdi", 0.4), ("conf/sosp", 0.4), ("journals/tc", 0.4), ("journals/sp", 0.4), ("conf/mlsys", 0.4),
108 + ("conf/ucc", 0.4), ("journals/tcc", 0.3), ("journals/jsa", 0.3), ("journals/computing", 0.3),
109 + ("journals/tnsm", 0.2), ("conf/xsede", 0.4), ("conf/pact", 0.2), ("journals/taco", 0.1), ("journals/tnse", 0.2),
110 + ("journals/jnca", 0.2), ("conf/isca", 0.1), ("journals/ieicetd", 0.1)],
111 + keywords=[
112 + (r"(?i)\b(distributed|parallel|concurrent) (systems?|computing|algorithms?|programming|processing|storage|file systems?|transactions?|consensus|ledgers?|training|inference|machine learning|deep learning|graph processing|databases?|data processing|simulation|applications?|execution|runtime|scheduling|snapshots?|shared memory|memory|protocols?|optimi[sz]ation|SGD|stochastic)\b", 1.0, "both"),
113 + (r"(?i)\b(consensus (protocols?|algorithms?)|Paxos|Raft\b|Byzantine|fault[- ]toleran(t|ce)|replication|state machine replication|linearizab(le|ility)|serializab|eventual consistency|strong consistency|consistency models?|CRDTs?|conflict-?free replicated|vector clocks?|logical clocks?|leader election|gossip protocols?|epidemic protocols?|distributed (hash tables?|locks?|snapshots?|tracing|ledger|shared)|two-?phase commit|atomic (broadcast|commit)|failure detectors?|self-?stabili[sz]|population protocols?|LOCAL model|CONGEST|message complexity|round complexity|asynchronous (systems?|networks?|model))\b", 1.0, "both"),
114 + (r"(?i)\b(high[- ]performance computing|HPC\b|supercomput\w+|exascale|petascale|MPI\b|OpenMP|message passing|load balancing|job scheduling|workflow (scheduling|management|engines?)|resource (allocation|management|scheduling) (in|for|on|across) (clusters?|grids?|data ?cent(er|re)s?|HPC|supercomputers?)|scalab(le|ility) (to|on|across) \d[\d,]* (nodes|cores|GPUs|processors)|multi-?GPU|GPU clusters?|data[- ]parallel(ism)?|model[- ]parallel(ism)?|pipeline[- ]parallel(ism)?|tensor[- ]parallel(ism)?|expert parallelism|all-?reduce|all-?to-?all|parameter servers?|collective communication|NCCL|ZeRO\b|FSDP|DeepSpeed|Megatron|Horovod|gradient (synchroni[sz]ation|accumulation across)|stragglers?|checkpoint(ing)? (for|of|in) (training|HPC)|large-?scale training|training (at scale|on \d+ GPUs)|distributed (LLM|DNN) (training|inference|serving)|cluster schedul(ers?|ing)|Slurm|Kubernetes schedul|Borg|YARN|Mesos)\b", 1.0, "both"),
115 + (r"(?i)\b(middleware|publish[- ]subscribe|pub/sub|message (queues?|brokers?)|Kafka|RabbitMQ|actor model|service mesh|stream processing (engines?|systems?|frameworks?)|Apache (Spark|Flink|Storm|Beam|Hadoop)|MapReduce|Ray\b|Dask|dataflow (systems?|engines?)|serverless (workflows?|computing)|microservices? (architecture|performance|systems?)|RPC\b|remote procedure|gRPC|RDMA|interconnects?|InfiniBand|network-?on-?chip)\b", 0.7, "both"),
116 + (r"(?i)\b(peer-?to-?peer|P2P\b|overlay networks?|volunteer computing|grid computing|cluster computing|utility computing|desktop grid|BOINC|BitTorrent|DHTs?\b|Chord|Kademlia|gossip-?based)\b", 0.8, "both"),
117 + (r"(?i)\b(multi-?core|many-?core|multi-?threaded|multithreading|lock-?free|wait-?free|non-?blocking (data structures?|algorithms?)|transactional memory|synchroni[sz]ation primitives|mutex(es)?|spin-?locks?|barriers?|work[- ]stealing|parallel(i[sz]ation|ism|izing)|task (parallel|graphs?)|speedup|Amdahl|strong scaling|weak scaling|parallel efficiency|SIMD parallel|GPU (computing|programming|acceleration)|CUDA\b|OpenCL|heterogeneous computing)\b", 0.8, "both"),
118 + (r"(?i)\b(nodes?|clusters?|scalab(le|ility)|parallel|distributed|concurrency|replicas?|throughput|latency)\b", 0.2, "both"),
119 + ],
120 + exclusions=[r"(?is)^(?!.*(comput|system|algorithm|protocol|network|server|cluster|node|GPU|processor|storage|training|software)).*\b(distributed (generation|energy resources?|lag|fibre|fiber|acoustic|temperature|parameter|load|element|order|delay|control of (power|microgrid|vehicles?|robots?)|leadership|cognition|practice|manufacturing))\b"],
121 + related=["cloud_computing", "edge_computing", "computer_networks", "database_systems", "operating_systems", "scientific_computing", "blockchain", "federated_learning"],
122 + parent_of=["cloud_computing", "edge_computing"], child_of=[],
123 + provenance=("T10715 and T10772 (subfield 1705) are the OpenAlex hosts; T10054 (parallel/HPC) is shared with architecture and "
124 + "scientific computing. TPDS/JPDC/IPDPS/PODC/SPAA/Euro-Par/ICS/HPDC/ICDCS/Middleware are the pure venues; cs.DC suffices."),
125 +),
126 +]
added taxonomy/src/area_data_b.py +140 −0
Large diff (140 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/area_hardware_a.py +124 −0
Large diff (124 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/area_perception_a.py +137 −0
Large diff (137 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/area_perception_b.py +131 −0
Large diff (131 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/area_security_a.py +89 −0
Large diff (89 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/area_security_b.py +95 −0
Large diff (95 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/area_software_pl.py +138 −0
Large diff (138 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/area_theory.py +97 −0
Large diff (97 lines) is not loaded by default to keep the page fast.Load diff
added taxonomy/src/areas.py +40 −0
Large diff (40 lines) is not loaded by default to keep the page fast.Load diff
added tests/test_structural_change.py +29 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere