SPB Git
spboucher.ai

spb/wp44_uqo

Public

WP44: corpus definition (A-001, A-002), structural-change toolkit + tests, bibliometrics/graph/embedding scripts, manuscript skeleton

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 3, 2026) parent 05d5bd4 Browse files

55 changed files +2,503 −14

55 changed files
Unified Split
added agents/REGISTER.md +8 −0
@@ -0,0 +1,8 @@
1 +# Agent register — WP44 (who produced what, who verified it)
2 +
3 +| Date | Agent | Brief | Artefact(s) | Verified by | Status |
4 +|---|---|---|---|---|---|
5 +| 2026-11-02 | A (director, this session) | CLAUDE.md | RESEARCH_PLAN.md, protocol/PROTOCOL.md, reports/PHASE0_FEASIBILITY.md | PI checkpoint H-0 pending | delivered |
6 +| 2026-11-02/03 | B+C (acquisition & data engineering, this session) | RESEARCH_PLAN §2–3 | ingestion/*.py, ledgers on M3U96b, data/metadata/linkage_report.json | N (audit, Phase 8) pending | delivered |
7 +| 2026-11-03 | D (taxonomy architect, subagent, 2nd attempt) | agents/briefs/AGENT_D_TAXONOMY.md | taxonomy/ontology.yaml, topic_map.csv, ONTOLOGY.md, src/ | A + K (validation sample, Phase 2) | running |
8 +| 2026-11-03 | B-lit (reference acquisition, subagent) | agents/briefs/AGENT_LIT_REFERENCES.md | manuscript/references.bib, references_verification.csv, literature_synthesis/REFERENCE_MAP.md | M (compile) + N (re-verify sample) | running |
added agents/briefs/AGENT_LIT_REFERENCES.md +78 −0
@@ -0,0 +1,78 @@
1 +# Brief — reference acquisition for the WP44 manuscript (Agent B-lit, 2026-11-03)
2 +
3 +Goal: build the *verified* bibliography that the manuscript's methodological and historical parts need, BEFORE any
4 +analysis result exists. Every entry must be resolved to a real bibliographic record; nothing may be invented.
5 +
6 +Output files (only these):
7 +1. `manuscript/references.bib` — BibTeX, keys `surnameYYYYkeyword` (lower-case ASCII), one entry per work, fields
8 + title/author/year/journal-or-booktitle/volume/pages/doi (or `eprint`+`archiveprefix={arXiv}` for preprints, or `url`
9 + for software/datasets/reports), `note = {verified: <method> <date>}`.
10 +2. `manuscript/references_verification.csv` — columns: key, doi_or_arxiv, verified_via (crossref|openalex|arxiv|datacite|url),
11 + verified_title (as returned by the service), match (exact|minor|FAILED), note. Entries that FAIL verification must be
12 + removed from references.bib and kept in the CSV with match=FAILED.
13 +3. `literature_synthesis/REFERENCE_MAP.md` — for each manuscript part (I–XI, appendices), the list of keys it should cite
14 + and one line on why (what the reference supports).
15 +
16 +Verification method (mandatory, per entry): `curl -s "https://api.crossref.org/works/<DOI>?mailto=spbou4@icloud.com"` or
17 +`curl -s "https://api.openalex.org/works/https://doi.org/<DOI>"` (single lookups are free) or
18 +`curl -s "https://export.arxiv.org/api/query?id_list=<id>"`; compare the returned title with yours. Do not use web search
19 +results as verification. If you cannot verify an item, drop it.
20 +
21 +Scope (target 120–180 verified entries), grouped:
22 +A. Scientometric data sources: OpenAlex (Priem, Piwowar, Orr 2022 arXiv:2205.01833), DBLP (Ley 2002/2009), arXiv
23 + (Ginsparg), Semantic Scholar (Kinney et al. 2023 S2AG), Crossref, ACL Anthology, Microsoft Academic Graph (Wang et al.
24 + 2020), CiteSeerX, Papers with Code; OpenAlex topic classification (OpenAlex topic model documentation, 2024).
25 +B. Systematic mapping / review methodology: PRISMA 2020 (Page et al. 2021 BMJ), PRISMA-S, Kitchenham & Charters 2007
26 + guidelines (SE), Petersen et al. 2008/2015 systematic mapping, Tricco scoping reviews, living reviews, automation of
27 + screening (e.g., ASReview van de Schoot 2021), LLM-assisted screening evaluations 2023–2025.
28 +C. Bibliometric indicators: Waltman 2016 review; field-normalised indicators (Waltman & van Eck 2019; MNCS; FWCI);
29 + citation windows and age bias (Wang 2013 "Citation time window choice"); h-index (Hirsch 2005); Gini/citation
30 + inequality (Nielsen & Andersen 2021); interdisciplinarity (Rao–Stirling: Stirling 2007; Leinster & Cobbold 2012;
31 + Porter & Rafols 2009; Leydesdorff); co-authorship networks (Newman 2001, 2004); country/institution analyses (Wagner
32 + 2017 international collaboration); preprints in CS (Lin et al. 2020 arXiv; Fraser 2021 bioRxiv analogies); citation
33 + advantage of OA (Piwowar 2018).
34 +D. Science mapping & embeddings: VOSviewer (van Eck & Waltman 2010), Small 1973 co-citation, Kessler bibliographic
35 + coupling, Boyack & Klavans 2010 accuracy of clustering, Leiden algorithm (Traag 2019), Louvain (Blondel 2008), SPECTER
36 + (Cohan et al. 2020), SPECTER2 (Singh et al. 2023 SciRepEval), SciNCL (Ostendorff 2022), SciBERT (Beltagy 2019),
37 + sentence-BERT (Reimers 2019), UMAP (McInnes 2018), HDBSCAN (Campello 2013; McInnes 2017), BERTopic (Grootendorst 2022),
38 + LDA (Blei 2003), dynamic topic models (Blei & Lafferty 2006), topic stability (Greene 2014).
39 +E. Structural change & time series: Bai & Perron 1998, 2003; Andrews 1993; Killick, Fearnhead & Eckley 2012 (PELT);
40 + Adams & MacKay 2007 (BOCPD); Truong, Oudre & Vayatis 2020 (ruptures review); Newey & West 1987; interrupted time series
41 + (Bernal, Cummins, Gasparrini 2017); synthetic control (Abadie 2010); event-study pitfalls.
42 +F. Emerging topic / technology detection: Rotolo, Hicks & Martin 2015; Small, Boyack & Klavans 2014; Cozzens et al.
43 + 2010; Wang Q. 2018 "A bibliometric model for identifying emerging research topics"; Klavans & Boyack 2017 research
44 + fronts; sleeping beauties (Ke, Ferrara, Radicchi, Flammini 2015); disruption index (Funk & Owen-Smith 2017; Wu, Wang,
45 + Evans 2019; Park, Leahey, Funk 2023 "Papers and patents are becoming less disruptive"); novelty (Uzzi et al. 2013);
46 + burst detection (Kleinberg 2003).
47 +G. Historical/contextual works on the AI transformation (only landmark, verified): Transformer (Vaswani 2017), BERT
48 + (Devlin 2019), GPT-3 (Brown 2020), scaling laws (Kaplan 2020; Hoffmann 2022), InstructGPT (Ouyang 2022), ChatGPT
49 + release (OpenAI blog 2022-11-30 — cite as web page with access date), GPT-4 technical report (2023), Llama 1/2/3
50 + (Touvron 2023; Dubey 2024), diffusion (Ho 2020; Rombach 2022), CLIP (Radford 2021), ViT (Dosovitskiy 2021), NeRF
51 + (Mildenhall 2020), RLHF (Christiano 2017), chain-of-thought (Wei 2022), RAG (Lewis 2020), ReAct (Yao 2023), Toolformer
52 + (Schick 2023), Mamba (Gu & Dao 2023), MoE Switch (Fedus 2022), DeepSeek-R1 (2025), o1 system card (2024), AlphaFold 2
53 + (Jumper 2021), AlphaGo/AlphaZero, foundation-model report (Bommasani 2021), "Sparks of AGI" (Bubeck 2023), stochastic
54 + parrots (Bender 2021), Stanford AI Index 2024/2025 reports, Epoch AI compute trends (Sevilla 2022).
55 +H. Evidence on LLM effects on science: Liang et al. 2024 (LLM-modified content in reviews/papers), Kobak et al. 2024/2025
56 + ("delving"), Geng & Trotta 2024, Bin-Nashwan / Dwivedi 2023 "So what if ChatGPT wrote it?", Birhane 2023, Messeri &
57 + Crockett 2024 (illusions of understanding), Nature surveys on AI use by researchers (Van Noorden 2023), arXiv
58 + submission growth analyses, "AI research is growing faster than…" (Krenn et al. 2023 Nature Machine Intelligence
59 + forecasting AI research), Fortunato et al. 2018 Science of science, Wang & Barabási 2021 book.
60 +I. Reproducibility/open science in CS: Gundersen & Kjensmo 2018; Pineau et al. 2021 (ML reproducibility program);
61 + Kapoor & Narayanan 2023 (leakage); Papers with Code / Hugging Face ecosystem studies; benchmark saturation (Ott et al.
62 + 2022 "Mapping global dynamics of benchmark creation and saturation"; Dehghani 2021 benchmark lottery; Raji 2021 AI and
63 + the everything in the whole wide world benchmark); Sculley 2018; ACM artifact badging policy (web).
64 +J. Other CS areas' survey anchors (one or two verified surveys each, 2022–2026 preferred): quantum computing (Preskill
65 + 2018 NISQ; a 2023–2025 survey), blockchain, federated learning (McMahan 2017; Kairouz 2021), differential privacy
66 + (Dwork 2006/2014), post-quantum cryptography (NIST FIPS 203/204/205 2024), formal verification (Lean mathlib / Rust
67 + verification surveys), computer architecture (Hennessy & Patterson 2019 Turing lecture; "golden age"), AI accelerators
68 + surveys, sustainable computing (Patterson 2021 carbon; Strubell 2019; Luccioni 2023), edge computing (Shi 2016), robotics
69 + foundation models (RT-2 2023; survey), AI for science (Wang H. et al. 2023 Nature "Scientific discovery in the age of
70 + AI"), AI safety/alignment surveys (Hendrycks 2021; Ji et al. 2023 alignment survey), agents surveys (Wang L. et al. 2024;
71 + Xi et al. 2023), LLM surveys (Zhao et al. 2023), multimodal surveys, speech (Whisper 2023), information retrieval
72 + (dense retrieval Karpukhin 2020), databases (vector databases survey 2024), compilers (MLIR Lattner 2021), programming
73 + languages (Rust safety), operating systems (eBPF / unikernels / LLM-OS), networks (programmable networks / P4), security
74 + (LLM security surveys 2024; prompt injection Greshake 2023), cryptography (FHE surveys), computational biology
75 + (AlphaFold 3 2024; ESM-2 2023), graphics (Gaussian splatting Kerbl 2023), HCI (Generative AI & HCI 2023–2024 surveys).
76 +
77 +Rules: work through the groups in order; verify as you go; keep each file write small (append to the .bib in batches of
78 +~20 entries with `cat >>`). Report back ≤ 25 lines: counts per group, failures, and anything you could not find.
added bibliometrics/01_volume_series.py +36 −0
@@ -0,0 +1,36 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — RQ1 volume series from corpus_works (eligible, CS core) with indexing-lag diagnostics.
3 +Outputs (results/bibliometrics/): volume_monthly.csv (first_month × type × membership), volume_yearly.csv,
4 +indexing_lag.csv (publication month → distribution of created_date − publication_date), penumbra_monthly.csv,
5 +dblp_only_yearly.csv. All tables are regenerated from the DuckDB database; nothing is hand-edited."""
6 +import argparse, os, duckdb
7 +
8 +
9 +def main():
10 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
11 + out = os.path.join(R, "results/bibliometrics"); os.makedirs(out, exist_ok=True)
12 + con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb"), read_only=True); con.execute("SET threads TO 16;")
13 + con.execute(f"""COPY (SELECT first_month, period, type, in_dblp, cs_primary, (arxiv_id_linked IS NOT NULL) AS has_arxiv, count(*) AS n,
14 + sum(has_abstract::int) AS n_abstract, sum((n_refs>0)::int) AS n_with_refs, sum(is_oa::int) AS n_oa
15 + FROM corpus_works WHERE eligible GROUP BY ALL ORDER BY 1,2,3) TO '{out}/volume_monthly.csv' (HEADER)""")
16 + con.execute(f"""COPY (SELECT publication_year AS year, count(*) AS n, sum(in_dblp::int) AS n_dblp, sum(cs_primary::int) AS n_cs_primary,
17 + sum((arxiv_id_linked IS NOT NULL)::int) AS n_arxiv, sum(in_acl::int) AS n_acl, sum(has_abstract::int) AS n_abstract,
18 + sum(is_oa::int) AS n_oa, sum((type='preprint')::int) AS n_preprint, sum((type='conference-paper')::int) AS n_conf,
19 + sum((type='article')::int) AS n_article, sum(e6_retracted::int) AS n_retracted, sum(xpac_kept_dblp::int) AS n_xpac_dblp,
20 + round(avg(n_authors),2) AS mean_authors, round(avg(n_refs),2) AS mean_refs, round(avg(n_countries),3) AS mean_countries
21 + FROM corpus_works WHERE eligible GROUP BY 1 ORDER BY 1) TO '{out}/volume_yearly.csv' (HEADER)""")
22 + con.execute(f"""COPY (SELECT substr(publication_date,1,7) AS pub_month, count(*) AS n,
23 + quantile_cont(datediff('day', publication_date::DATE, created_date::DATE), 0.5) AS lag_days_p50,
24 + quantile_cont(datediff('day', publication_date::DATE, created_date::DATE), 0.9) AS lag_days_p90,
25 + sum((datediff('day', publication_date::DATE, created_date::DATE) <= 90)::int) AS n_indexed_within_90d,
26 + sum((created_date::DATE > publication_date::DATE + INTERVAL 180 DAY)::int) AS n_indexed_after_180d
27 + FROM corpus_works WHERE eligible AND created_date IS NOT NULL AND publication_date IS NOT NULL GROUP BY 1 ORDER BY 1) TO '{out}/indexing_lag.csv' (HEADER)""")
28 + con.execute(f"""COPY (SELECT first_month, count(*) AS n_penumbra FROM corpus_works WHERE cs_penumbra AND NOT e1_outside_window AND NOT e2_type AND NOT e3_xpac AND NOT e4_no_title AND NOT e5_duplicate
29 + GROUP BY 1 ORDER BY 1) TO '{out}/penumbra_monthly.csv' (HEADER)""")
30 + con.execute(f"COPY (SELECT * FROM dblp_only_counts) TO '{out}/dblp_only_yearly.csv' (HEADER)")
31 + for f in ["volume_yearly.csv", "indexing_lag.csv"]:
32 + print(f, open(os.path.join(out, f)).read()[:1500])
33 +
34 +
35 +if __name__ == "__main__":
36 + main()
added bibliometrics/02_collaboration.py +42 −0
@@ -0,0 +1,42 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — RQ6 collaboration, institutions, countries (eligible corpus), yearly and by period.
3 +Outputs (results/bibliometrics/): team_size_yearly.csv, countries_yearly.csv (top-N country output + share, with
4 +fractional counting), international_collab_yearly.csv, institution_types_yearly.csv (education/company/government/…),
5 +top_institutions_by_period.csv (full and fractional counts), country_pairs_by_period.csv (edges for the collaboration
6 +network, ≥ 50 joint works), top_sources_by_period.csv (venue concentration inputs), venue_hhi_yearly.csv."""
7 +import argparse, os, duckdb
8 +
9 +
10 +def main():
11 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
12 + out = os.path.join(R, "results/bibliometrics"); os.makedirs(out, exist_ok=True)
13 + con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb"), read_only=True); con.execute("SET threads TO 20; SET memory_limit='60GB';")
14 + con.execute("CREATE OR REPLACE TEMP VIEW E AS SELECT * FROM corpus_works WHERE eligible")
15 + con.execute(f"""COPY (SELECT publication_year AS year, period, count(*) n, round(avg(n_authors),3) mean_authors, quantile_cont(n_authors,0.5) med_authors,
16 + sum((n_authors=1)::int) single, sum((n_authors>=5)::int) five_plus, sum((n_authors>=10)::int) ten_plus, sum((n_countries>=2)::int) intl,
17 + sum((n_institutions>=2)::int) multi_inst, sum((n_countries=0)::int) no_country FROM E GROUP BY 1,2 ORDER BY 1) TO '{out}/team_size_yearly.csv' (HEADER)""")
18 + # country output: full counting (work counted once per country) and fractional (1/n_countries)
19 + con.execute("""CREATE OR REPLACE TEMP TABLE wc AS SELECT w.id, w.publication_year AS year, w.period, unnest(w.countries) AS country, 1.0/len(w.countries) AS frac FROM E w WHERE len(w.countries) > 0""")
20 + con.execute(f"""COPY (SELECT year, country, count(*) n_full, round(sum(frac),1) n_frac FROM wc GROUP BY 1,2 ORDER BY 1, 3 DESC) TO '{out}/countries_yearly.csv' (HEADER)""")
21 + con.execute(f"""COPY (SELECT year, count(*) n, sum((n_countries>=2)::int) n_intl, round(sum((n_countries>=2)::int)*1.0/count(*),4) share_intl FROM E WHERE n_countries>0 GROUP BY 1 ORDER BY 1) TO '{out}/international_collab_yearly.csv' (HEADER)""")
22 + con.execute("""CREATE OR REPLACE TEMP TABLE wi AS SELECT w.id, w.publication_year AS year, w.period, unnest(w.institution_types) AS itype FROM E w WHERE len(w.institution_types) > 0""")
23 + con.execute(f"""COPY (SELECT year, itype, count(*) n FROM wi GROUP BY 1,2 ORDER BY 1,3 DESC) TO '{out}/institution_types_yearly.csv' (HEADER)""")
24 + con.execute(f"""COPY (SELECT year, count(DISTINCT id) n_with_inst, count(DISTINCT CASE WHEN itype='company' THEN id END) n_company,
25 + round(count(DISTINCT CASE WHEN itype='company' THEN id END)*1.0/count(DISTINCT id),4) share_company FROM wi GROUP BY 1 ORDER BY 1) TO '{out}/industry_share_yearly.csv' (HEADER)""")
26 + # institutions (from authorships): full and fractional (by distinct institutions per work)
27 + con.execute("""CREATE OR REPLACE TEMP TABLE winst AS
28 + SELECT DISTINCT a.work_id, unnest(a.institution_ids) AS inst_id FROM oa_authorships a WHERE a.work_id IN (SELECT id FROM E) AND len(a.institution_ids) > 0""")
29 + con.execute("""CREATE OR REPLACE TEMP TABLE winst2 AS SELECT w.work_id, w.inst_id, 1.0/count(*) OVER (PARTITION BY w.work_id) AS frac, e.period, e.publication_year AS year FROM winst w JOIN E e ON e.id = w.work_id""")
30 + con.execute(f"""COPY (SELECT period, inst_id, count(*) n_full, round(sum(frac),1) n_frac FROM winst2 GROUP BY 1,2 HAVING count(*) >= 200 ORDER BY 1, 3 DESC) TO '{out}/top_institutions_by_period.csv' (HEADER)""")
31 + con.execute(f"""COPY (SELECT a.period, a.country AS c1, b.country AS c2, count(DISTINCT a.id) n FROM wc a JOIN wc b ON a.id = b.id AND a.country < b.country
32 + GROUP BY 1,2,3 HAVING count(DISTINCT a.id) >= 50 ORDER BY 1, 4 DESC) TO '{out}/country_pairs_by_period.csv' (HEADER)""")
33 + con.execute(f"""COPY (SELECT period, source_id, any_value(source_name) source_name, any_value(source_type) source_type, count(*) n FROM E WHERE source_id IS NOT NULL
34 + GROUP BY 1,2 HAVING count(*) >= 100 ORDER BY 1, 5 DESC) TO '{out}/top_sources_by_period.csv' (HEADER)""")
35 + con.execute(f"""COPY (SELECT year, count(*) n, count(DISTINCT source_id) n_sources, round(sum(s*s),6) hhi, round(max(s),5) top1_share FROM
36 + (SELECT publication_year AS year, source_id, count(*)*1.0/sum(count(*)) OVER (PARTITION BY publication_year) AS s FROM E WHERE source_id IS NOT NULL GROUP BY 1,2)
37 + GROUP BY 1 ORDER BY 1) TO '{out}/venue_hhi_yearly.csv' (HEADER)""")
38 + print(open(f"{out}/team_size_yearly.csv").read()[:800]); print(open(f"{out}/industry_share_yearly.csv").read())
39 +
40 +
41 +if __name__ == "__main__":
42 + main()
added bibliometrics/03_citation_impact.py +44 −0
@@ -0,0 +1,44 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — RQ3/RQ4 citation impact with explicit citation-window control.
3 +
4 +From oa_counts_by_year (citations received per calendar year) we build fixed-window counts:
5 + c12 = citations received in the publication year and the following year (≈ 0–24 months), c36 = pub year + 2 years.
6 +Normalisation: NCS_k = c_k / mean(c_k | same publication year, same type class, same primary subfield) over the eligible
7 +corpus (field-normalised within OpenAlex subfield; category-level normalisation is added in 05_category_impact.py).
8 +Outputs: results/bibliometrics/citation_windows_yearly.csv (means, medians, shares uncited, Gini of c36 by year),
9 +work_citation_windows.parquet (work-level c12/c36/NCS for downstream scripts), top_cited_by_year.csv (top 50 per year by c36),
10 +fwci_summary_yearly.csv (OpenAlex FWCI as a secondary indicator)."""
11 +import argparse, os, duckdb
12 +
13 +
14 +def main():
15 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
16 + out = os.path.join(R, "results/bibliometrics"); os.makedirs(out, exist_ok=True)
17 + con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb")); con.execute("SET threads TO 20; SET memory_limit='60GB';")
18 + con.execute("""CREATE OR REPLACE TABLE work_citation_windows AS
19 + WITH E AS (SELECT id, publication_year AS y, type, primary_subfield_id AS sf, cited_by_count, fwci FROM corpus_works WHERE eligible),
20 + C AS (SELECT c.work_id, sum(CASE WHEN c.year <= e.y + 1 THEN c.cited_by_count ELSE 0 END) AS c12,
21 + sum(CASE WHEN c.year <= e.y + 2 THEN c.cited_by_count ELSE 0 END) AS c36,
22 + sum(CASE WHEN c.year <= e.y + 3 THEN c.cited_by_count ELSE 0 END) AS c48
23 + FROM oa_counts_by_year c JOIN E e ON e.id = c.work_id GROUP BY 1)
24 + SELECT e.id AS work_id, e.y AS year, e.type, e.sf, e.cited_by_count, e.fwci, coalesce(c.c12,0) c12, coalesce(c.c36,0) c36, coalesce(c.c48,0) c48
25 + FROM E e LEFT JOIN C c ON c.work_id = e.id""")
26 + con.execute("""CREATE OR REPLACE TABLE work_citation_windows2 AS
27 + SELECT w.*, CASE WHEN m12 > 0 THEN c12 / m12 END AS ncs12, CASE WHEN m36 > 0 THEN c36 / m36 END AS ncs36
28 + FROM work_citation_windows w JOIN (SELECT year, type, sf, avg(c12) m12, avg(c36) m36, count(*) n_ref FROM work_citation_windows GROUP BY 1,2,3) r
29 + USING (year, type, sf)""")
30 + con.execute("DROP TABLE work_citation_windows; ALTER TABLE work_citation_windows2 RENAME TO work_citation_windows")
31 + con.execute(f"COPY work_citation_windows TO '{R}/data/processed/corpus/work_citation_windows.parquet' (FORMAT PARQUET, COMPRESSION ZSTD)")
32 + con.execute(f"""COPY (SELECT year, count(*) n, round(avg(c12),3) mean_c12, round(avg(c36),3) mean_c36, quantile_cont(c36,0.5) med_c36,
33 + round(sum((c36=0)::int)*1.0/count(*),4) share_uncited_36, round(sum((c12=0)::int)*1.0/count(*),4) share_uncited_12,
34 + quantile_cont(c36,0.99) p99_c36, max(c36) max_c36,
35 + round((2*sum(c36*rk) / (count(*)*sum(c36)) - (count(*)+1.0)/count(*)),4) gini_c36,
36 + round(avg(fwci),3) mean_fwci, round(quantile_cont(fwci,0.5),3) med_fwci
37 + FROM (SELECT *, row_number() OVER (PARTITION BY year ORDER BY c36) rk FROM work_citation_windows) GROUP BY 1 ORDER BY 1) TO '{out}/citation_windows_yearly.csv' (HEADER)""")
38 + con.execute(f"""COPY (SELECT * FROM (SELECT w.year, w.work_id, c.title, c.source_name, c.type, w.c12, w.c36, w.cited_by_count, w.fwci, w.ncs36,
39 + row_number() OVER (PARTITION BY w.year ORDER BY w.c36 DESC, w.cited_by_count DESC) rk FROM work_citation_windows w JOIN corpus_works c ON c.id = w.work_id) WHERE rk <= 50 ORDER BY year, rk) TO '{out}/top_cited_by_year.csv' (HEADER)""")
40 + print(open(f"{out}/citation_windows_yearly.csv").read())
41 +
42 +
43 +if __name__ == "__main__":
44 + main()
added data/metadata/linkage_report.json +45 −0
@@ -0,0 +1,45 @@
1 +{
2 + "created": "2026-11-03T20:50:56.182276+00:00",
3 + "oa_cs_records": 15490159,
4 + "oa_cs_distinct_ids": 15490159,
5 + "dblp_2018_2026": 4347365,
6 + "dblp_with_doi": 4052326,
7 + "dblp_with_arxiv": 0,
8 + "link_dblp_oa": {
9 + "title_year": 179990,
10 + "doi": 2562431
11 + },
12 + "dblp_linked_distinct": 2736903,
13 + "dblp_unlinked": 1610462,
14 + "dblp_unlinked_by_type": {
15 + "phdthesis": 26841,
16 + "incollection": 3362,
17 + "inproceedings": 477235,
18 + "book": 1398,
19 + "article": 1084640,
20 + "data": 16986
21 + },
22 + "dblp_unlinked_with_doi_in_slim": 1077407,
23 + "arxiv_cs_records": 915115,
24 + "arxiv_created_2018_2026": 890852,
25 + "link_arxiv_oa": {
26 + "arxiv_id": 792672,
27 + "doi": 520,
28 + "title_year": 2143
29 + },
30 + "arxiv_linked_distinct": 744680,
31 + "acl_records": 135558,
32 + "acl_2018_2026": 85625,
33 + "link_acl_oa": {
34 + "doi": 57474,
35 + "title_year": 14904
36 + },
37 + "acl_linked_distinct_2018_2026": 72223,
38 + "needed_oa_ids_second_pass": {
39 + "dblp_doi": 1077406,
40 + "arxiv_doi": 39200
41 + },
42 + "slim_total": 464765485,
43 + "slim_cs_any_2018_2026": 15490159,
44 + "seconds": 10
45 +}
\ No newline at end of file
added econometrics/structural_change.py +135 −0
@@ -0,0 +1,135 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — structural-change toolkit for monthly publication series (RQ2, hypotheses H1/H3).
3 +
4 +Functions (all pure, tested in tests/test_structural_change.py):
5 + its_fit(y, dates, event, lags) interrupted time series on log counts with month-of-year seasonality,
6 + level and slope change at `event`, Newey–West HAC standard errors
7 + placebo_breaks(y, dates, window) distribution of the ITS level/slope t-statistics over every candidate month of the
8 + baseline (placebo), returns the empirical rank/p-value of the true event
9 + bai_perron(y, dates, max_breaks) multiple structural breaks in mean + trend by dynamic programming (BIC), with
10 + minimum segment length h (Bai & Perron 1998/2003, pure-python DP on SSR)
11 + pelt(y, pen) PELT changepoints (ruptures, l2 cost) on log counts, penalty by BIC
12 + bocpd(y, hazard, mu0, kappa0, ...) Bayesian online changepoint detection (Adams & MacKay 2007), Student-t predictive,
13 + returns run-length posterior and P(changepoint at t)
14 +Right-truncation: callers must stop the series at the last complete month (see bibliometrics/01_volume_series.py).
15 +"""
16 +from __future__ import annotations
17 +import numpy as np, pandas as pd
18 +import statsmodels.api as sm
19 +
20 +
21 +def _design(dates: pd.DatetimeIndex, event: pd.Timestamp, seasonal=True):
22 + t = np.arange(len(dates), dtype=float)
23 + post = (dates >= event).astype(float)
24 + t_post = np.where(post == 1, t - t[post == 1][0] if post.sum() else 0, 0.0)
25 + X = {"const": np.ones(len(t)), "t": t, "post": post, "t_post": t_post}
26 + if seasonal:
27 + for m in range(2, 13):
28 + X[f"m{m}"] = (dates.month == m).astype(float)
29 + return pd.DataFrame(X, index=dates)
30 +
31 +
32 +def its_fit(y, dates, event, lags=6, seasonal=True, log=True):
33 + dates = pd.DatetimeIndex(dates); event = pd.Timestamp(event)
34 + yy = np.log(np.asarray(y, dtype=float)) if log else np.asarray(y, dtype=float)
35 + X = _design(dates, event, seasonal)
36 + res = sm.OLS(yy, X).fit(cov_type="HAC", cov_kwds={"maxlags": lags})
37 + out = {"level_change": res.params["post"], "level_se": res.bse["post"], "level_t": res.tvalues["post"], "level_p": res.pvalues["post"],
38 + "slope_change": res.params["t_post"], "slope_se": res.bse["t_post"], "slope_t": res.tvalues["t_post"], "slope_p": res.pvalues["t_post"],
39 + "pre_slope": res.params["t"], "pre_slope_se": res.bse["t"], "n": int(res.nobs), "r2": res.rsquared, "event": str(event.date())}
40 + return out, res
41 +
42 +
43 +def placebo_breaks(y, dates, true_event, min_pre=24, min_post=12, lags=6, seasonal=True):
44 + dates = pd.DatetimeIndex(dates)
45 + rows = []
46 + for i in range(min_pre, len(dates) - min_post):
47 + ev = dates[i]
48 + o, _ = its_fit(y, dates, ev, lags, seasonal)
49 + rows.append({"event": ev, "level_t": o["level_t"], "slope_t": o["slope_t"], "level_change": o["level_change"], "slope_change": o["slope_change"]})
50 + df = pd.DataFrame(rows)
51 + tv = df.loc[df.event == pd.Timestamp(true_event)]
52 + out = {"n_placebo": len(df)}
53 + if len(tv):
54 + for k in ("level_t", "slope_t"):
55 + v = float(tv[k].iloc[0]); out[f"{k}_true"] = v
56 + out[f"{k}_placebo_p"] = float((np.abs(df[k]) >= abs(v)).mean())
57 + out[f"{k}_rank"] = int((np.abs(df[k]) >= abs(v)).sum())
58 + return out, df
59 +
60 +
61 +def bai_perron(y, dates=None, max_breaks=3, h=None, trend=True):
62 + """Global minimiser of SSR with m breaks (Bai & Perron 2003 dynamic programming) for a segment-wise linear trend
63 + (or mean) model; model selection by BIC and by the LWZ criterion. Returns dict with breaks (indices and dates)."""
64 + y = np.asarray(y, dtype=float); n = len(y); h = h or max(8, int(0.15 * n))
65 + t = np.arange(n, dtype=float)
66 + def ssr(i, j): # segment [i, j)
67 + yy = y[i:j]
68 + if trend:
69 + X = np.column_stack([np.ones(j - i), t[i:j]])
70 + beta, *_ = np.linalg.lstsq(X, yy, rcond=None); r = yy - X @ beta
71 + else:
72 + r = yy - yy.mean()
73 + return float(r @ r)
74 + S = np.full((n + 1, n + 1), np.inf)
75 + for i in range(n):
76 + for j in range(i + h, n + 1):
77 + S[i, j] = ssr(i, j)
78 + best = {}
79 + k_params = 2 if trend else 1
80 + # DP: C[m][j] = min SSR of first j obs with m breaks
81 + C = {0: S[0, :].copy()}; B = {0: {j: [] for j in range(n + 1)}}
82 + for m in range(1, max_breaks + 1):
83 + C[m] = np.full(n + 1, np.inf); B[m] = {}
84 + for j in range((m + 1) * h, n + 1):
85 + cands = [(C[m - 1][i] + S[i, j], i) for i in range(m * h, j - h + 1) if np.isfinite(C[m - 1][i]) and np.isfinite(S[i, j])]
86 + if cands:
87 + v, i = min(cands); C[m][j] = v; B[m][j] = B[m - 1][i] + [i]
88 + for m in range(0, max_breaks + 1):
89 + tot = C[m][n]
90 + if not np.isfinite(tot):
91 + continue
92 + p = (m + 1) * k_params + m
93 + bic = n * np.log(tot / n) + p * np.log(n)
94 + lwz = n * np.log(tot / (n - p)) + p * 0.299 * np.log(n) ** 2.1
95 + best[m] = {"ssr": tot, "bic": bic, "lwz": lwz, "breaks": B[m].get(n, [])}
96 + m_bic = min(best, key=lambda m: best[m]["bic"]); m_lwz = min(best, key=lambda m: best[m]["lwz"])
97 + res = {"by_m": best, "m_bic": m_bic, "m_lwz": m_lwz, "breaks_bic": best[m_bic]["breaks"], "h": h}
98 + if dates is not None:
99 + d = pd.DatetimeIndex(dates); res["break_dates_bic"] = [str(d[i].date()) for i in best[m_bic]["breaks"]]
100 + res["break_dates_lwz"] = [str(d[i].date()) for i in best[m_lwz]["breaks"]]
101 + return res
102 +
103 +
104 +def pelt(y, pen=None, min_size=6, model="l2"):
105 + import ruptures as rpt
106 + y = np.asarray(y, dtype=float); n = len(y)
107 + pen = pen if pen is not None else 2 * np.log(n) * np.var(y) # BIC-type penalty for l2 cost
108 + algo = rpt.Pelt(model=model, min_size=min_size, jump=1).fit(y.reshape(-1, 1))
109 + return algo.predict(pen=pen)[:-1]
110 +
111 +
112 +def bocpd(y, hazard=1 / 60.0, mu0=None, kappa0=1.0, alpha0=1.0, beta0=None):
113 + """Adams & MacKay (2007) with Normal–Inverse-Gamma conjugacy (Student-t predictive). Returns (R, cp_prob, map_rl):
114 + R[t, r] is the run-length posterior after observing y[:t]; with a constant hazard the mass on r = 0 equals the hazard
115 + by construction, so the detection statistics are cp_prob[t] = P(r_t <= 3 | y[:t+1]) (a changepoint in the last
116 + months) and map_rl[t] = argmax_r R[t+1, r] (the MAP run length, which resets after a change)."""
117 + from scipy.stats import t as student
118 + y = np.asarray(y, dtype=float); n = len(y)
119 + mu0 = y[:6].mean() if mu0 is None else mu0; beta0 = y[:6].var() + 1e-6 if beta0 is None else beta0
120 + R = np.zeros((n + 1, n + 1)); R[0, 0] = 1.0
121 + mu = np.array([mu0]); kappa = np.array([kappa0]); alpha = np.array([alpha0]); beta = np.array([beta0])
122 + cp = np.zeros(n); map_rl = np.zeros(n, dtype=int)
123 + for t_ in range(n):
124 + x = y[t_]
125 + df = 2 * alpha; scale = np.sqrt(beta * (kappa + 1) / (alpha * kappa))
126 + pred = student.pdf(x, df, loc=mu, scale=scale)
127 + growth = R[t_, : t_ + 1] * pred * (1 - hazard)
128 + change = (R[t_, : t_ + 1] * pred * hazard).sum()
129 + R[t_ + 1, 1 : t_ + 2] = growth; R[t_ + 1, 0] = change
130 + R[t_ + 1] /= R[t_ + 1].sum() + 1e-300
131 + cp[t_] = R[t_ + 1, :4].sum(); map_rl[t_] = int(np.argmax(R[t_ + 1, : t_ + 2]))
132 + mu_new = (kappa * mu + x) / (kappa + 1); kappa_new = kappa + 1; alpha_new = alpha + 0.5
133 + beta_new = beta + kappa * (x - mu) ** 2 / (2 * (kappa + 1))
134 + mu = np.concatenate([[mu0], mu_new]); kappa = np.concatenate([[kappa0], kappa_new]); alpha = np.concatenate([[alpha0], alpha_new]); beta = np.concatenate([[beta0], beta_new])
135 + return R, cp, map_rl
added embeddings/bench_models.py +21 −0
@@ -0,0 +1,21 @@
1 +#!/usr/bin/env python3
2 +"""Benchmark candidate scientific embedding models on Apple Silicon (MPS): throughput and a quick sanity check.
3 +Models: allenai/specter2_base (SPECTER2 base, no adapter), malteos/scincl, BAAI/bge-base-en-v1.5, sentence-transformers/all-MiniLM-L6-v2."""
4 +import sys, time, json, torch, duckdb
5 +from sentence_transformers import SentenceTransformer
6 +R = sys.argv[1]; N = int(sys.argv[2]) if len(sys.argv) > 2 else 2000
7 +con = duckdb.connect(f"{R}/data/processed/wp44.duckdb", read_only=True)
8 +rows = con.execute(f"SELECT w.title, a.abstract FROM oa_works w JOIN oa_abstracts a ON a.work_id=w.id WHERE w.publication_year=2024 LIMIT {N}").fetchall()
9 +texts = [f"{t}. {ab[:1500]}" for t, ab in rows]
10 +dev = "mps" if torch.backends.mps.is_available() else "cpu"; print("device", dev, "texts", len(texts), flush=True)
11 +res = {}
12 +for m in ["allenai/specter2_base", "malteos/scincl", "BAAI/bge-base-en-v1.5", "sentence-transformers/all-MiniLM-L6-v2"]:
13 + try:
14 + model = SentenceTransformer(m, device=dev); model.max_seq_length = 256
15 + _ = model.encode(texts[:64], batch_size=64)
16 + t0 = time.time(); E = model.encode(texts, batch_size=128, convert_to_numpy=True, normalize_embeddings=True); dt_ = time.time() - t0
17 + res[m] = {"dim": int(E.shape[1]), "texts_per_s": round(len(texts) / dt_, 1), "seconds": round(dt_, 1)}
18 + print(m, res[m], flush=True)
19 + except Exception as e:
20 + res[m] = {"error": str(e)[:300]}; print(m, "ERROR", str(e)[:200], flush=True)
21 +json.dump(res, open(f"{R}/data/metadata/embedding_benchmark.json", "w"), indent=1)
added embeddings/embed_corpus.py +63 −0
@@ -0,0 +1,63 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — embed title + abstract of corpus works with a scientific sentence-embedding model, sharded for multi-node runs.
3 +
4 + --prepare : DuckDB exports data/processed/embeddings/input/shard_NNN.parquet (work_id, text) for eligible works with an
5 + abstract (or title only when --title-only-fallback), N shards of --shard-rows rows.
6 + --run : encode shards [--shard-from, --shard-to) with --model on MPS/CPU → data/processed/embeddings/<model_tag>/
7 + shard_NNN.npy (float16, L2-normalised) + shard_NNN.ids.npy. Resumable (existing shards skipped).
8 +Provenance: data/processed/embeddings/<model_tag>/manifest.json (model, revision, max_seq_length, host, timings).
9 +"""
10 +import argparse, glob, json, os, socket, time, datetime as dt
11 +import numpy as np
12 +
13 +
14 +def prepare(root, shard_rows, title_only_fallback):
15 + import duckdb
16 + con = duckdb.connect(os.path.join(root, "data/processed/wp44.duckdb"), read_only=True); con.execute("SET threads TO 16;")
17 + out = os.path.join(root, "data/processed/embeddings/input"); os.makedirs(out, exist_ok=True)
18 + fb = " OR TRUE" if title_only_fallback else ""
19 + con.execute(f"""CREATE OR REPLACE TEMP TABLE emb_in AS
20 + SELECT w.id AS work_id, w.title || '. ' || coalesce(substr(a.abstract, 1, 2000), '') AS text
21 + FROM corpus_works w LEFT JOIN oa_abstracts a ON a.work_id = w.id
22 + WHERE w.eligible AND (a.abstract IS NOT NULL{fb}) ORDER BY w.id""")
23 + n = con.execute("SELECT count(*) FROM emb_in").fetchone()[0]
24 + con.execute(f"COPY (SELECT *, (row_number() OVER (ORDER BY work_id) - 1) // {shard_rows} AS shard FROM emb_in) TO '{out}' (FORMAT PARQUET, PARTITION_BY (shard), OVERWRITE_OR_IGNORE)")
25 + print("prepared", n, "texts in", len(glob.glob(out + "/shard=*")), "shards")
26 +
27 +
28 +def run(root, model_name, shard_from, shard_to, batch, max_len):
29 + import torch, pyarrow.parquet as pq
30 + from sentence_transformers import SentenceTransformer
31 + tag = model_name.replace("/", "__"); out = os.path.join(root, "data/processed/embeddings", tag); os.makedirs(out, exist_ok=True)
32 + dev = "mps" if torch.backends.mps.is_available() else "cpu"
33 + model = SentenceTransformer(model_name, device=dev); model.max_seq_length = max_len
34 + shards = sorted(glob.glob(os.path.join(root, "data/processed/embeddings/input", "shard=*", "*.parquet")))
35 + man_p = os.path.join(out, f"manifest_{socket.gethostname()}.json"); man = json.load(open(man_p)) if os.path.exists(man_p) else {"model": model_name, "max_seq_length": max_len, "device": dev, "host": socket.gethostname(), "shards": {}}
36 + for p in shards:
37 + k = int(os.path.basename(os.path.dirname(p)).split("=")[1])
38 + if not (shard_from <= k < shard_to):
39 + continue
40 + o = os.path.join(out, f"shard_{k:04d}.npy")
41 + if os.path.exists(o):
42 + continue
43 + t = pq.read_table(p).to_pydict(); t0 = time.time()
44 + E = model.encode(t["text"], batch_size=batch, convert_to_numpy=True, normalize_embeddings=True, show_progress_bar=False).astype(np.float16)
45 + np.save(o + ".part.npy", E); os.replace(o + ".part.npy", o); np.save(o.replace(".npy", ".ids.npy"), np.asarray(t["work_id"], dtype=np.int64))
46 + man["shards"][str(k)] = {"n": len(t["text"]), "seconds": round(time.time() - t0, 1), "when": dt.datetime.now(dt.UTC).isoformat()}
47 + json.dump(man, open(man_p, "w"), indent=1)
48 + print(f"shard {k}: {len(t['text'])} texts in {time.time()-t0:.0f}s ({len(t['text'])/(time.time()-t0):.0f}/s)", flush=True)
49 +
50 +
51 +def main():
52 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); ap.add_argument("--prepare", action="store_true"); ap.add_argument("--run", action="store_true")
53 + ap.add_argument("--model", default="allenai/specter2_base"); ap.add_argument("--shard-rows", type=int, default=50000); ap.add_argument("--shard-from", type=int, default=0)
54 + ap.add_argument("--shard-to", type=int, default=10**6); ap.add_argument("--batch", type=int, default=128); ap.add_argument("--max-len", type=int, default=256); ap.add_argument("--title-only-fallback", action="store_true")
55 + a = ap.parse_args()
56 + if a.prepare:
57 + prepare(a.root, a.shard_rows, a.title_only_fallback)
58 + if a.run:
59 + run(a.root, a.model, a.shard_from, a.shard_to, a.batch, a.max_len)
60 +
61 +
62 +if __name__ == "__main__":
63 + main()
added ingestion/compact_tables.py +20 −0
@@ -0,0 +1,20 @@
1 +#!/usr/bin/env python3
2 +"""Compact the per-part Parquet tables (1,898 files per table) into one Parquet file per table
3 +(data/processed/compact/<table>.parquet, zstd, ~1M-row row groups) so that downstream DuckDB work does not
4 +hit the macOS open-file limit. Also compacts the slim table of all OpenAlex works."""
5 +import duckdb, os, sys, time
6 +R = sys.argv[1]; suffix = sys.argv[2] if len(sys.argv) > 2 else "" # e.g. "_p2" for the second snapshot pass
7 +tables_dir = os.path.join(R, "data/processed/tables" + suffix); out = os.path.join(R, "data/processed/compact"); os.makedirs(out, exist_ok=True)
8 +con = duckdb.connect(); con.execute("SET threads TO 24; SET memory_limit='70GB'; SET preserve_insertion_order=false;")
9 +jobs = [(t + suffix, f"{tables_dir}/{t}/*.parquet") for t in ["works", "authorships", "work_topics", "work_keywords", "references", "abstracts", "locations", "grants", "counts_by_year"]]
10 +if not suffix:
11 + jobs.append(("slim_all_works", f"{R}/data/raw/openalex/works/slim/*.parquet"))
12 +for t, src in jobs:
13 + dst = f"{out}/{t}.parquet"
14 + if os.path.exists(dst):
15 + print("skip", t); continue
16 + t0 = time.time()
17 + con.execute(f"COPY (SELECT * FROM read_parquet('{src}')) TO '{dst}.part' (FORMAT PARQUET, COMPRESSION ZSTD, ROW_GROUP_SIZE 1000000)")
18 + os.replace(dst + ".part", dst)
19 + n = con.execute(f"SELECT count(*) FROM read_parquet('{dst}')").fetchone()[0]
20 + print(f"{t}: {n:,} rows, {os.path.getsize(dst)/1e9:.2f} GB, {time.time()-t0:.0f}s", flush=True)
modified ingestion/link_sources.py +10 −7
@@ -17,8 +17,11 @@ def main():
17 17 T = os.path.join(R, "data/processed/tables"); t0 = time.time(); rep = {"created": dt.datetime.now(dt.UTC).isoformat()}
18 18 def log(m): print(f"[{time.time()-t0:7.0f}s] {m}", flush=True)
19 19 for t in ["works", "authorships", "work_topics", "work_keywords", "references", "abstracts", "locations", "grants", "counts_by_year"]:
20 − con.execute(f"CREATE OR REPLACE VIEW oa_{t} AS SELECT * FROM read_parquet('{T}/{t}/*.parquet')")
21 − con.execute(f"CREATE OR REPLACE VIEW oa_slim AS SELECT * FROM read_parquet('{R}/data/raw/openalex/works/slim/*.parquet')")
20 + srcs = [f"{R}/data/processed/compact/{t}.parquet"]
21 + if os.path.exists(f"{R}/data/processed/compact/{t}_p2.parquet"):
22 + srcs.append(f"{R}/data/processed/compact/{t}_p2.parquet")
23 + con.execute(f"CREATE OR REPLACE VIEW oa_{t} AS SELECT * FROM read_parquet({srcs!r}, union_by_name=true)")
24 + con.execute(f"CREATE OR REPLACE VIEW oa_slim AS SELECT * FROM read_parquet('{R}/data/processed/compact/slim_all_works.parquet')")
22 25 con.execute(f"CREATE OR REPLACE VIEW arxiv_raw AS SELECT * FROM read_parquet('{R}/data/raw/arxiv/arxiv_cs.parquet')")
23 26 con.execute(f"CREATE OR REPLACE VIEW acl_raw AS SELECT * FROM read_parquet('{R}/data/raw/acl/acl_anthology.parquet')")
24 27 con.execute(f"CREATE OR REPLACE VIEW dblp_raw AS SELECT * FROM read_parquet('{R}/data/raw/dblp/dblp_publications.parquet')")
@@ -49,9 +52,9 @@ def main():
49 52 WHERE d.arxiv_id IS NOT NULL AND o.arxiv_id IS NOT NULL AND d.dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa)
50 53 AND o.oa_id NOT IN (SELECT oa_id FROM link_dblp_oa)""")
51 54 # title+year: unique normalised titles on both sides, length >= 25, |year diff| <= 1
52 − con.execute("""CREATE OR REPLACE TEMP TABLE d_un AS SELECT ntitle, any_value(dblp_key) dblp_key, any_value(year) year FROM dblp_keys
55 + con.execute("""CREATE OR REPLACE TEMP TABLE d_un AS SELECT ntitle, any_value(dblp_key) AS dblp_key, any_value(year) AS year FROM dblp_keys
53 56 WHERE tlen >= 25 AND dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa) GROUP BY ntitle HAVING count(*) = 1""")
54 − con.execute("""CREATE OR REPLACE TEMP TABLE o_un AS SELECT ntitle, any_value(oa_id) oa_id, any_value(year) year FROM oa_keys
57 + con.execute("""CREATE OR REPLACE TEMP TABLE o_un AS SELECT ntitle, any_value(oa_id) AS oa_id, any_value(year) AS year FROM oa_keys
55 58 WHERE tlen >= 25 AND oa_id NOT IN (SELECT oa_id FROM link_dblp_oa) GROUP BY ntitle HAVING count(*) = 1""")
56 59 con.execute("""INSERT INTO link_dblp_oa SELECT d.dblp_key, o.oa_id, 'title_year' FROM d_un d JOIN o_un o ON d.ntitle = o.ntitle
57 60 WHERE abs(d.year - o.year) <= 1""")
@@ -61,9 +64,9 @@ def main():
61 64 SELECT x.arxiv_id, o.oa_id, 'arxiv_id' AS method FROM arxiv_keys x JOIN oa_keys o ON x.arxiv_id = o.arxiv_id""")
62 65 con.execute("""INSERT INTO link_arxiv_oa SELECT x.arxiv_id, o.oa_id, 'doi' FROM arxiv_keys x JOIN oa_keys o ON x.doi = o.doi
63 66 WHERE x.doi IS NOT NULL AND x.arxiv_id NOT IN (SELECT arxiv_id FROM link_arxiv_oa)""")
64 − con.execute("""CREATE OR REPLACE TEMP TABLE x_un AS SELECT ntitle, any_value(arxiv_id) arxiv_id, any_value(created_year) year FROM arxiv_keys
67 + con.execute("""CREATE OR REPLACE TEMP TABLE x_un AS SELECT ntitle, any_value(arxiv_id) AS arxiv_id, any_value(created_year) AS year FROM arxiv_keys
65 68 WHERE length(ntitle) >= 25 AND arxiv_id NOT IN (SELECT arxiv_id FROM link_arxiv_oa) GROUP BY ntitle HAVING count(*) = 1""")
66 − con.execute("""CREATE OR REPLACE TEMP TABLE o_un2 AS SELECT ntitle, any_value(oa_id) oa_id, any_value(year) year FROM oa_keys
69 + con.execute("""CREATE OR REPLACE TEMP TABLE o_un2 AS SELECT ntitle, any_value(oa_id) AS oa_id, any_value(year) AS year FROM oa_keys
67 70 WHERE tlen >= 25 GROUP BY ntitle HAVING count(*) = 1""")
68 71 con.execute("""INSERT INTO link_arxiv_oa SELECT x.arxiv_id, o.oa_id, 'title_year' FROM x_un x JOIN o_un2 o ON x.ntitle = o.ntitle
69 72 WHERE abs(x.year - o.year) <= 2 AND o.oa_id NOT IN (SELECT oa_id FROM link_arxiv_oa)""")
@@ -76,7 +79,7 @@ def main():
76 79 WHERE landing_page_url LIKE '%aclanthology.org/%') l ON l.aid = c.acl_id OR l.aid = c.acl_id || '/'
77 80 WHERE c.acl_id NOT IN (SELECT acl_id FROM link_acl_oa)""")
78 81 con.execute("""INSERT INTO link_acl_oa SELECT c.acl_id, o.oa_id, 'title_year' FROM
79 − (SELECT ntitle, any_value(acl_id) acl_id, any_value(year) year FROM acl_keys WHERE length(ntitle) >= 25 AND acl_id NOT IN (SELECT acl_id FROM link_acl_oa) GROUP BY ntitle HAVING count(*)=1) c
82 + (SELECT ntitle, any_value(acl_id) AS acl_id, any_value(year) AS year FROM acl_keys WHERE length(ntitle) >= 25 AND acl_id NOT IN (SELECT acl_id FROM link_acl_oa) GROUP BY ntitle HAVING count(*)=1) c
80 83 JOIN o_un2 o ON c.ntitle = o.ntitle WHERE abs(c.year - o.year) <= 1 AND o.oa_id NOT IN (SELECT oa_id FROM link_acl_oa)""")
81 84 log("acl linked")
82 85 # --- report ---
modified ingestion/openalex_snapshot.py +35 −7
@@ -39,6 +39,7 @@ import pyarrow.parquet as pq
39 39 import zstandard as zstd
40 40
41 41 S3_HTTP = "https://openalex.s3.amazonaws.com/"
42 +ID_SET = None # optional: set of OpenAlex work ids (ints) to extract instead of the CS filter (second pass)
42 43 CS_FIELD = "https://openalex.org/fields/17"
43 44 YEAR_MIN, YEAR_MAX = 2018, 2026
44 45
@@ -116,9 +117,21 @@ def slim_row(w: dict) -> tuple:
116 117 ), cs_any
117 118
118 119
120 +def _load_ids(path):
121 + with open(path) as fh:
122 + next(fh)
123 + return {int(x.strip().rsplit("W", 1)[-1]) for x in fh if x.strip()}
124 +
125 +
126 +def _init_worker(id_list):
127 + """Pool initializer: macOS uses the spawn start method, so module globals must be rebuilt in each worker."""
128 + global ID_SET
129 + ID_SET = _load_ids(id_list) if id_list else None
130 +
131 +
119 132 def process_file(args):
120 133 """Download, hash, parse and filter one part file. Returns a result dict."""
121 − url, expected_len, out_dir, keep_raw = args
134 + url, expected_len, out_dir, keep_raw, mode = args
122 135 rel = url.replace("s3://openalex/data/jsonl/works/", "")
123 136 name = rel.replace("/", "__") # updated_date=X__part_0000.gz
124 137 http = url.replace("s3://openalex/", S3_HTTP)
@@ -164,6 +177,11 @@ def process_file(args):
164 177 n_bad += 1
165 178 continue
166 179 n_all += 1
180 + if mode == "idset":
181 + if _oa_int(w.get("id")) in ID_SET:
182 + cs_w.write(line if line.endswith(b"\n") else line + b"\n")
183 + n_cs += 1
184 + continue
167 185 row, cs_any = slim_row(w)
168 186 for i, v in enumerate(row):
169 187 cols[i].append(v)
@@ -172,10 +190,13 @@ def process_file(args):
172 190 cs_w.write(line if line.endswith(b"\n") else line + b"\n")
173 191 n_cs += 1
174 192 os.replace(cs_tmp, cs_tmp[:-5])
175 − table = pa.Table.from_arrays([pa.array(c, type=f.type) for c, f in zip(cols, SLIM_SCHEMA)], schema=SLIM_SCHEMA)
176 − slim_tmp = os.path.join(slim_dir, name[:-3] + ".parquet.part")
177 − pq.write_table(table, slim_tmp, compression="zstd")
178 − os.replace(slim_tmp, slim_tmp[:-5])
193 + if mode == "idset":
194 + cols = None
195 + table = None if cols is None else pa.Table.from_arrays([pa.array(c, type=f.type) for c, f in zip(cols, SLIM_SCHEMA)], schema=SLIM_SCHEMA)
196 + if table is not None:
197 + slim_tmp = os.path.join(slim_dir, name[:-3] + ".parquet.part")
198 + pq.write_table(table, slim_tmp, compression="zstd")
199 + os.replace(slim_tmp, slim_tmp[:-5])
179 200 if not keep_raw:
180 201 os.remove(raw_path)
181 202 res.update(status="done", sha256=sha, n_records=n_all, n_cs=n_cs, n_bad=n_bad,
@@ -193,7 +214,14 @@ def main():
193 214 ap.add_argument("--keep-raw", action="store_true")
194 215 ap.add_argument("--limit", type=int, default=0, help="process only N files (smoke test)")
195 216 ap.add_argument("--smallest-first", action="store_true")
217 + ap.add_argument("--id-list", default=None, help="CSV with a header and one OpenAlex work id per line: extract exactly these works (second pass)")
196 218 a = ap.parse_args()
219 + global ID_SET
220 + mode = "cs"
221 + if a.id_list:
222 + ID_SET = _load_ids(a.id_list)
223 + mode = "idset"
224 + print(f"id-set mode: {len(ID_SET)} ids", flush=True)
197 225 os.makedirs(a.out, exist_ok=True)
198 226 man = json.load(open(a.manifest))
199 227 files = [(f["url"], f["meta"]["content_length"], f["meta"].get("record_count")) for f in man["files"]]
@@ -214,12 +242,12 @@ def main():
214 242 (url, url.replace("s3://openalex/data/jsonl/works/", "").replace("/", "__"), ln, rc))
215 243 db.commit()
216 244 done = {r[0] for r in db.execute("SELECT url FROM files WHERE status='done'")}
217 − todo = [(u, ln, a.out, a.keep_raw) for u, ln, _ in files if u not in done]
245 + todo = [(u, ln, a.out, a.keep_raw, mode) for u, ln, _ in files if u not in done]
218 246 total_bytes = sum(t[1] for t in todo)
219 247 print(f"[{dt.datetime.now():%H:%M:%S}] manifest {man.get('date')}: {len(files)} files, {len(done)} done, "
220 248 f"{len(todo)} to do ({total_bytes/1e9:.1f} GB), workers={a.workers}", flush=True)
221 249 t0 = time.time(); done_bytes = 0; n_ok = n_err = 0
222 − with mp.Pool(a.workers, maxtasksperchild=50) as pool:
250 + with mp.Pool(a.workers, maxtasksperchild=50, initializer=_init_worker, initargs=(a.id_list,)) as pool:
223 251 for r in pool.imap_unordered(process_file, todo):
224 252 now = dt.datetime.now(dt.UTC).isoformat()
225 253 db.execute("UPDATE files SET attempts=attempts+1 WHERE url=?", (r["url"],))
added ingestion/run_increment_daily.sh +8 −0
@@ -0,0 +1,8 @@
1 +#!/bin/sh
2 +# re-run the API increment once per day until all slices are done (public budget $0.10/day)
3 +cd ~/wp44_uqo; . .venv/bin/activate
4 +while ! grep -q "all slices done" logs/openalex_api_increment.log 2>/dev/null; do
5 + python ingestion/openalex_api_increment.py --out data/raw/openalex >> logs/openalex_api_increment.log 2>&1
6 + grep -q "all slices done" logs/openalex_api_increment.log && break
7 + sleep 21700
8 +done
added knowledge_graph/01_citation_graph.py +41 −0
@@ -0,0 +1,41 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — citation graph of the eligible corpus (Phase 3, structural layer).
3 +
4 +Builds, in DuckDB and Parquet (data/processed/graph/):
5 + cit_edges_internal.parquet (citing_id, cited_id, citing_year, cited_year) both ends eligible
6 + cit_edges_external.parquet (citing_id, cited_id, cited_field_id, cited_year, cited_type) cited work outside the corpus,
7 + resolved through the slim table of all 465 M OpenAlex works (field of the cited work)
8 + field_flows_yearly.csv citing year × cited primary field (OpenAlex field) counts — interdisciplinary inflows (RQ4)
9 + cited_age_yearly.csv distribution of citation age (citing year − cited year) by citing year
10 + reference_coverage_yearly.csv share of references resolvable (in OpenAlex), internal vs external
11 +Category-level flows are produced by knowledge_graph/02_category_flows.py after classification."""
12 +import argparse, os, duckdb
13 +
14 +
15 +def main():
16 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
17 + G = os.path.join(R, "data/processed/graph"); out = os.path.join(R, "results/graph"); os.makedirs(G, exist_ok=True); os.makedirs(out, exist_ok=True)
18 + con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb")); con.execute("SET threads TO 24; SET memory_limit='70GB';")
19 + con.execute("CREATE OR REPLACE TEMP TABLE E AS SELECT id, publication_year AS y, type, primary_field_id FROM corpus_works WHERE eligible")
20 + con.execute("""CREATE OR REPLACE TABLE cit_edges_internal AS
21 + SELECT r.work_id AS citing_id, r.ref_id AS cited_id, a.y AS citing_year, b.y AS cited_year
22 + FROM oa_references r JOIN E a ON a.id = r.work_id JOIN E b ON b.id = r.ref_id""")
23 + con.execute("""CREATE OR REPLACE TABLE cit_edges_external AS
24 + SELECT r.work_id AS citing_id, r.ref_id AS cited_id, a.y AS citing_year, s.publication_year AS cited_year, s.primary_field_id AS cited_field_id, s.type AS cited_type, s.cs_any_topic AS cited_cs_any
25 + FROM oa_references r JOIN E a ON a.id = r.work_id LEFT JOIN E b ON b.id = r.ref_id
26 + JOIN oa_slim s ON s.id = r.ref_id WHERE b.id IS NULL""")
27 + for t in ("cit_edges_internal", "cit_edges_external"):
28 + con.execute(f"COPY {t} TO '{G}/{t}.parquet' (FORMAT PARQUET, COMPRESSION ZSTD)")
29 + con.execute(f"""COPY (SELECT citing_year, 'internal' AS kind, NULL AS cited_field_id, count(*) n FROM cit_edges_internal GROUP BY 1
30 + UNION ALL SELECT citing_year, 'external', cited_field_id, count(*) FROM cit_edges_external GROUP BY 1,3 ORDER BY 1,2,3) TO '{out}/field_flows_yearly.csv' (HEADER)""")
31 + con.execute(f"""COPY (SELECT citing_year, citing_year - cited_year AS age, count(*) n FROM (SELECT citing_year, cited_year FROM cit_edges_internal UNION ALL SELECT citing_year, cited_year FROM cit_edges_external)
32 + WHERE cited_year IS NOT NULL GROUP BY 1,2 ORDER BY 1,2) TO '{out}/cited_age_yearly.csv' (HEADER)""")
33 + con.execute(f"""COPY (SELECT e.y AS year, count(*) n_works, sum(n_refs) n_refs_declared,
34 + (SELECT count(*) FROM cit_edges_internal i WHERE i.citing_year = e.y) n_internal,
35 + (SELECT count(*) FROM cit_edges_external x WHERE x.citing_year = e.y) n_external
36 + FROM E e JOIN corpus_works c ON c.id = e.id GROUP BY 1 ORDER BY 1) TO '{out}/reference_coverage_yearly.csv' (HEADER)""")
37 + print(open(f"{out}/reference_coverage_yearly.csv").read())
38 +
39 +
40 +if __name__ == "__main__":
41 + main()
added manuscript/appendices/appA_taxonomy.tex +1 −0
@@ -0,0 +1 @@
1 +% appA_taxonomy — to be written (Phase 7)
added manuscript/appendices/appB_search_and_sources.tex +1 −0
@@ -0,0 +1 @@
1 +% appB_search_and_sources — to be written (Phase 7)
added manuscript/appendices/appC_derivations.tex +1 −0
@@ -0,0 +1 @@
1 +% appC_derivations — to be written (Phase 7)
added manuscript/appendices/appD_robustness.tex +1 −0
@@ -0,0 +1 @@
1 +% appD_robustness — to be written (Phase 7)
added manuscript/appendices/appE_data_dictionary.tex +1 −0
@@ -0,0 +1 @@
1 +% appE_data_dictionary — to be written (Phase 7)
added manuscript/appendices/appF_supplementary_tables.tex +1 −0
@@ -0,0 +1 @@
1 +% appF_supplementary_tables — to be written (Phase 7)
added manuscript/appendices/appG_figure_documentation.tex +1 −0
@@ -0,0 +1 @@
1 +% appG_figure_documentation — to be written (Phase 7)
added manuscript/appendices/appH_reproducibility.tex +1 −0
@@ -0,0 +1 @@
1 +% appH_reproducibility — to be written (Phase 7)
added manuscript/main.tex +97 −0
@@ -0,0 +1,97 @@
1 +% ============================================================================
2 +% UQO Working Paper 44 — The Post-ChatGPT Computer Science Revolution
3 +% Preamble follows the UQO working-paper template (_modele-working-paper/main.tex)
4 +% ============================================================================
5 +\documentclass[12pt,letterpaper]{article}
6 +\usepackage[utf8]{inputenc}
7 +\usepackage[T1]{fontenc}
8 +\usepackage[english]{babel}
9 +\usepackage[letterpaper,top=1in,bottom=1in,left=1in,right=1in]{geometry}
10 +\usepackage{mathptmx}
11 +\usepackage{microtype}
12 +\usepackage{setspace}
13 +\onehalfspacing
14 +\usepackage{amsmath,amssymb,amsthm}
15 +\usepackage{booktabs,tabularx,multirow,threeparttable,longtable,array}
16 +\usepackage{siunitx}
17 +\sisetup{input-symbols = {()}, table-format = -1.3, parse-numbers = false}
18 +\usepackage{graphicx,subcaption,float}
19 +\usepackage[dvipsnames]{xcolor}
20 +\usepackage[colorlinks=true,linkcolor=NavyBlue,citecolor=NavyBlue,urlcolor=NavyBlue,breaklinks=true]{hyperref}
21 +\usepackage[authoryear,round,semicolon]{natbib}
22 +\bibliographystyle{aer}
23 +\usepackage[font=small,labelfont=bf,labelsep=period,justification=justified,singlelinecheck=false]{caption}
24 +\usepackage{fancyhdr}
25 +\pagestyle{fancy}\fancyhf{}\renewcommand{\headrulewidth}{0pt}\fancyfoot[C]{\thepage}
26 +\usepackage{titlesec}
27 +\titleformat{\section}{\large\bfseries}{\thesection.}{0.5em}{}
28 +\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection.}{0.5em}{}
29 +\titleformat{\subsubsection}{\normalsize\itshape}{\thesubsubsection.}{0.5em}{}
30 +\usepackage[toc,page]{appendix}
31 +\usepackage{enumitem}
32 +\setlist{nosep,leftmargin=*}
33 +\newtheorem{definition}{Definition}
34 +\newtheorem{proposition}{Proposition}
35 +\newtheorem{hypothesis}{Hypothesis}
36 +\newcommand{\E}{\mathbb{E}}
37 +\newcommand{\R}{\mathbb{R}}
38 +\newcommand{\pending}[1]{\textcolor{BrickRed}{[pending human review: #1]}}
39 +\graphicspath{{figures/}}
40 +
41 +% ---- metadata ----
42 +\newcommand{\WPnumber}{44}
43 +\newcommand{\WPtitle}{The Post-ChatGPT Computer Science Revolution}
44 +\newcommand{\WPsubtitle}{A Comprehensive Systematic Mapping, Bibliometric Analysis, and Knowledge Graph of Computer Science Research, 2022--2026}
45 +\newcommand{\WPdate}{November 2026}
46 +\newcommand{\WPversion}{0.1}
47 +\input{tables/numbers} % machine-generated macros (counts, estimates) — never edited by hand
48 +\newcommand{\WPabstract}{\input{sections/abstract}}
49 +\newcommand{\WPkeywords}{systematic mapping, bibliometrics, computer science, large language models, structural change, knowledge graph, emerging topics, OpenAlex}
50 +\newcommand{\WPjel}{C55, C88, O33, L86, I23}
51 +\newcommand{\WPauthor}{Simon-Pierre Boucher}
52 +\newcommand{\WPaffiliation}{Département des sciences administratives\\Université du Québec en Outaouais}
53 +\newcommand{\WPemail}{simon-pierre.boucher@uqo.ca}
54 +\newcommand{\WPaddress}{Gatineau -- Pavillon Alexandre-Taché\\283, boulevard Alexandre-Taché\\Gatineau, Québec, Canada J9A 1L8}
55 +
56 +\begin{document}
57 +\input{sections/titlepage}
58 +\tableofcontents
59 +\clearpage
60 +
61 +\part{Introduction and Scientific Motivation}
62 +\input{sections/part01_introduction}
63 +\part{The Evolution of Computer Science}
64 +\input{sections/part02_evolution}
65 +\part{Data and Systematic Review Protocol}
66 +\input{sections/part03_data}
67 +\part{Computational Methodology}
68 +\input{sections/part04_methodology}
69 +\part{The Artificial Intelligence Transformation}
70 +\input{sections/part05_ai}
71 +\part{Systems and Computing Infrastructure}
72 +\input{sections/part06_systems}
73 +\part{Theory, Algorithms, and Security}
74 +\input{sections/part07_theory_security}
75 +\part{Scientific Knowledge Networks}
76 +\input{sections/part08_networks}
77 +\part{Empirical Results}
78 +\input{sections/part09_results}
79 +\part{The Next Research Frontiers}
80 +\input{sections/part10_frontiers}
81 +\part{Discussion and Conclusion}
82 +\input{sections/part11_discussion}
83 +
84 +\clearpage
85 +\bibliography{references}
86 +
87 +\begin{appendices}
88 +\input{appendices/appA_taxonomy}
89 +\input{appendices/appB_search_and_sources}
90 +\input{appendices/appC_derivations}
91 +\input{appendices/appD_robustness}
92 +\input{appendices/appE_data_dictionary}
93 +\input{appendices/appF_supplementary_tables}
94 +\input{appendices/appG_figure_documentation}
95 +\input{appendices/appH_reproducibility}
96 +\end{appendices}
97 +\end{document}
added manuscript/sections/abstract.tex +1 −0
@@ -0,0 +1 @@
1 +% abstract — to be written (Phase 7)
added manuscript/sections/part01_introduction.tex +1 −0
@@ -0,0 +1 @@
1 +% part01_introduction — to be written (Phase 7)
added manuscript/sections/part02_evolution.tex +1 −0
@@ -0,0 +1 @@
1 +% part02_evolution — to be written (Phase 7)
added manuscript/sections/part03_data.tex +1 −0
@@ -0,0 +1 @@
1 +% part03_data — to be written (Phase 7)
added manuscript/sections/part04_methodology.tex +1 −0
@@ -0,0 +1 @@
1 +% part04_methodology — to be written (Phase 7)
added manuscript/sections/part05_ai.tex +1 −0
@@ -0,0 +1 @@
1 +% part05_ai — to be written (Phase 7)
added manuscript/sections/part06_systems.tex +1 −0
@@ -0,0 +1 @@
1 +% part06_systems — to be written (Phase 7)
added manuscript/sections/part07_theory_security.tex +1 −0
@@ -0,0 +1 @@
1 +% part07_theory_security — to be written (Phase 7)
added manuscript/sections/part08_networks.tex +1 −0
@@ -0,0 +1 @@
1 +% part08_networks — to be written (Phase 7)
added manuscript/sections/part09_results.tex +1 −0
@@ -0,0 +1 @@
1 +% part09_results — to be written (Phase 7)
added manuscript/sections/part10_frontiers.tex +1 −0
@@ -0,0 +1 @@
1 +% part10_frontiers — to be written (Phase 7)
added manuscript/sections/part11_discussion.tex +1 −0
@@ -0,0 +1 @@
1 +% part11_discussion — to be written (Phase 7)
added manuscript/tables/numbers.tex +1 −0
@@ -0,0 +1 @@
1 +% machine-generated numbers (tables/make_numbers.py) — placeholder until results exist
modified protocol/amendments.csv +2 −0
@@ -1,2 +1,4 @@
1 1 id,date,phase,change,reason,approved_by
2 2 A-000,2026-11-02,0,"§14 of the mandate completed from §1–§13 (paste truncated)","terminal truncation of the pasted mandate",executing agent (pending PI)
3 +A-001,2026-11-03,1,"Eligibility type list corrected: OpenAlex (2026) types conference-paper and software-paper are research outputs and are kept; DBLP-linked records are kept whatever their OpenAlex type unless in a hard-exclusion list; XPAC records are kept when DBLP-linked (flag xpac_kept_dblp)","first run of define_corpus.py showed 2.5 M conference-paper records (1.09 M DBLP-linked) excluded by an outdated type list; DBLP is the CS gold standard",executing agent (pending PI)
4 +A-002,2026-11-03,1,"Records whose primary source is a generic deposit platform (Zenodo, Figshare, IRDB, LA Referencia, Dialnet, DOAJ-as-source, OSTI) are excluded unless DBLP- or arXiv-linked (flag e7_generic_deposit); date_precision flag added (publication_date = YYYY-01-01 treated as year precision; monthly series use month-precision records only)","Zenodo alone contributes 138,533 eligible records in 2026 (20,141 in 2025; 1.3% with references) — deposits of slides, datasets and reports typed as articles; 18–28% of records carry a year-only date defaulting to 1 January",executing agent (pending PI)
added reports/HUMAN_REVIEW_CHECKPOINTS.md +12 −0
@@ -0,0 +1,12 @@
1 +# Human-review checkpoints (protocol §P7) — items awaiting the principal investigator
2 +
3 +| Checkpoint | What to review | Where | Status |
4 +|---|---|---|---|
5 +| H-0 | Plan, protocol, decisions D1–D7 and amendments A-000–A-002 | `RESEARCH_PLAN.md` §10, `protocol/PROTOCOL.md`, `protocol/amendments.csv` | pending |
6 +| H-1 | Record-linkage precision sample (500 DBLP↔OpenAlex pairs, 200 reserved for the PI) | `data/metadata/linkage_sample.csv` (to be generated by `systematic_review/linkage_sample.py`) | pending |
7 +| H-2 | Taxonomy validation sample (2,000 works, two LLM raters; 200 for the PI) | `taxonomy/validation/` | pending |
8 +| H-3 | Landmark list (≥ 1,000) and specialist reviews | `literature_synthesis/landmarks/` | pending |
9 +| H-4 | Structural-change results and interpretation (no causal language) | `results/econometrics/` | pending |
10 +| H-5 | Manuscript conclusions | `manuscript/` | pending |
11 +
12 +Conclusions that depend on an unsigned checkpoint are labelled "pending human review" in the manuscript.
added systematic_review/define_corpus.py +110 −0
@@ -0,0 +1,110 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — Stages S2–S3: deduplicate (preprint ↔ published), apply eligibility, define the mapping corpus, write PRISMA counts.
3 +
4 +Inputs: DuckDB views/tables made by ingestion/link_sources.py (oa_* views, dblp_keys, arxiv_keys, link_* tables).
5 +Outputs: table `corpus_works` (+ Parquet copy data/processed/corpus/corpus_works.parquet), `dblp_only_counts`,
6 +data/metadata/prisma_counts.json. Rules: protocol/PROTOCOL.md §P2; decisions logged in RESEARCH_PLAN §10.
7 +"""
8 +import argparse, json, os, time, datetime as dt
9 +import duckdb
10 +
11 +KEEP_TYPES = "('article','conference-paper','preprint','book-chapter','book','dissertation','report','review','software-paper')"
12 +# hard exclusions even for DBLP-linked records (DBLP-linked records of any other type are kept: DBLP is the CS gold standard)
13 +# generic deposit platforms (A-002): records whose primary source is one of these are kept only if DBLP- or arXiv-linked
14 +GENERIC_DEPOSIT_SOURCES = "('Zenodo (CERN European Organization for Nuclear Research)','Figshare','Institutional Repositories DataBase (IRDB)','LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas)','Dialnet (Universidad de la Rioja)','DOAJ (DOAJ: Directory of Open Access Journals)','OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information)','Zenodo')"
15 +HARD_EXCLUDE_TYPES = "('editorial','erratum','retraction','peer-review','paratext','conference-abstract','reference-entry','dataset','software','supplementary-materials','libguides','letter','grant','standard')"
16 +WINDOW_START, BOUNDARY, WINDOW_END = "2018-01-01", "2022-11-30", "2026-11-02"
17 +
18 +
19 +def main():
20 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); R = a.root
21 + con = duckdb.connect(os.path.join(R, "data/processed/wp44.duckdb")); con.execute("SET threads TO 24; SET memory_limit='60GB';")
22 + t0 = time.time(); q = lambda s: con.execute(s).fetchall(); n = lambda s: q(s)[0][0]
23 + log = lambda m: print(f"[{time.time()-t0:6.0f}s] {m}", flush=True)
24 + # ---- S0/S1: base table with membership flags ----
25 + con.execute(f"""CREATE OR REPLACE TABLE cw0 AS
26 + SELECT w.*, regexp_replace(lower(w.title), '[^a-z0-9]+', '', 'g') AS ntitle,
27 + (w.id IN (SELECT oa_id FROM link_dblp_oa)) AS in_dblp,
28 + (w.id IN (SELECT oa_id FROM link_acl_oa)) AS in_acl,
29 + x.arxiv_id AS arxiv_id_linked, x.created AS arxiv_created, x.primary_category AS arxiv_primary,
30 + (x.primary_category LIKE 'cs.%') AS arxiv_cs_primary,
31 + (w.doi LIKE '10.48550/arxiv.%') AS is_arxiv_doi
32 + FROM oa_works w
33 + LEFT JOIN (SELECT l.oa_id, any_value(k.arxiv_id) arxiv_id, min(k.created) created, any_value(k.primary_category) primary_category
34 + FROM link_arxiv_oa l JOIN arxiv_keys k USING(arxiv_id) GROUP BY l.oa_id) x ON x.oa_id = w.id""")
35 + log(f"cw0 {n('SELECT count(*) FROM cw0'):,}")
36 + # ---- S2: preprint ↔ published duplicates (same normalised title, |year| <= 2, one preprint-like, one not) ----
37 + con.execute("""CREATE OR REPLACE TABLE dup_pairs AS
38 + WITH pre AS (SELECT id, ntitle, publication_year y, publication_date d FROM cw0 WHERE (type = 'preprint' OR is_arxiv_doi) AND length(ntitle) >= 25),
39 + pub AS (SELECT id, ntitle, publication_year y, publication_date d FROM cw0 WHERE type <> 'preprint' AND NOT is_arxiv_doi AND length(ntitle) >= 25)
40 + SELECT p.id AS preprint_id, b.id AS published_id, p.d AS preprint_date, b.d AS published_date,
41 + row_number() OVER (PARTITION BY b.id ORDER BY p.d, p.id) AS rn_pub, row_number() OVER (PARTITION BY p.id ORDER BY b.d, b.id) AS rn_pre
42 + FROM pre p JOIN pub b ON p.ntitle = b.ntitle AND abs(p.y - b.y) <= 2""")
43 + con.execute("DELETE FROM dup_pairs WHERE rn_pub > 1 OR rn_pre > 1")
44 + log(f"duplicate preprint→published pairs: {n('SELECT count(*) FROM dup_pairs'):,}")
45 + # ---- S3: eligibility flags ----
46 + con.execute(f"""CREATE OR REPLACE TABLE corpus_works AS
47 + SELECT c.*,
48 + dp.published_id AS duplicate_of,
49 + dp2.preprint_id AS preprint_work_id,
50 + least(c.publication_date, coalesce(dp2.preprint_date, c.publication_date), coalesce(c.arxiv_created, c.publication_date)) AS first_date,
51 + (c.publication_date IS NULL OR c.publication_date < '{WINDOW_START}' OR c.publication_date > '{WINDOW_END}') AS e1_outside_window,
52 + (c.type NOT IN {KEEP_TYPES} AND NOT (c.in_dblp AND c.type NOT IN {HARD_EXCLUDE_TYPES})) AS e2_type,
53 + (c.is_xpac AND NOT c.in_dblp) AS e3_xpac,
54 + (c.is_xpac AND c.in_dblp) AS xpac_kept_dblp,
55 + (c.source_name IN {GENERIC_DEPOSIT_SOURCES} AND NOT c.in_dblp AND c.arxiv_id_linked IS NULL) AS e7_generic_deposit,
56 + (CASE WHEN substr(c.publication_date, 6, 5) = '01-01' THEN 'year' ELSE 'month' END) AS date_precision,
57 + (c.title IS NULL OR length(trim(c.title)) = 0) AS e4_no_title,
58 + (dp.published_id IS NOT NULL) AS e5_duplicate,
59 + c.is_retracted AS e6_retracted,
60 + (c.in_dblp OR c.cs_primary OR (c.cs_any AND coalesce(c.arxiv_cs_primary, FALSE))) AS cs_core,
61 + (c.cs_any AND NOT (c.in_dblp OR c.cs_primary OR (c.cs_any AND coalesce(c.arxiv_cs_primary, FALSE)))) AS cs_penumbra
62 + FROM cw0 c
63 + LEFT JOIN dup_pairs dp ON dp.preprint_id = c.id
64 + LEFT JOIN dup_pairs dp2 ON dp2.published_id = c.id""")
65 + con.execute(f"""ALTER TABLE corpus_works ADD COLUMN eligible BOOLEAN; UPDATE corpus_works SET eligible =
66 + NOT e1_outside_window AND NOT e2_type AND NOT e3_xpac AND NOT e4_no_title AND NOT e5_duplicate AND NOT e7_generic_deposit AND cs_core""")
67 + con.execute(f"""ALTER TABLE corpus_works ADD COLUMN period VARCHAR; UPDATE corpus_works SET period = CASE
68 + WHEN first_date < '{WINDOW_START}' THEN 'pre' WHEN first_date < '{BOUNDARY}' THEN 'baseline' WHEN first_date <= '{WINDOW_END}' THEN 'post' ELSE 'future' END""")
69 + con.execute("ALTER TABLE corpus_works ADD COLUMN first_month VARCHAR; UPDATE corpus_works SET first_month = substr(first_date, 1, 7)")
70 + log("corpus_works built")
71 + # ---- DBLP-only records (no OpenAlex record after both passes) ----
72 + con.execute("""CREATE OR REPLACE TABLE dblp_only_counts AS
73 + SELECT year, rec_type, count(*) n, sum(CASE WHEN doi IS NOT NULL THEN 1 ELSE 0 END) with_doi FROM dblp_keys
74 + WHERE dblp_key NOT IN (SELECT dblp_key FROM link_dblp_oa) GROUP BY 1,2 ORDER BY 1,2""")
75 + # ---- PRISMA counts ----
76 + P = {"generated": dt.datetime.now(dt.UTC).isoformat(), "window": [WINDOW_START, WINDOW_END], "boundary": BOUNDARY}
77 + P["identified"] = {
78 + "openalex_cs_topic_2018_2026_pass1": n("SELECT count(*) FROM oa_keys"),
79 + "openalex_total_records_loaded": n("SELECT count(*) FROM cw0"),
80 + "dblp_2018_2026": n("SELECT count(*) FROM dblp_keys"), "arxiv_cs_2018_2026": n("SELECT count(*) FROM arxiv_keys WHERE created_year BETWEEN 2018 AND 2026"),
81 + "acl_2018_2026": n("SELECT count(*) FROM acl_keys WHERE year BETWEEN 2018 AND 2026")}
82 + P["linked"] = {"dblp_to_openalex": n("SELECT count(DISTINCT dblp_key) FROM link_dblp_oa"), "arxiv_to_openalex": n("SELECT count(DISTINCT arxiv_id) FROM link_arxiv_oa"),
83 + "acl_to_openalex": n("SELECT count(DISTINCT acl_id) FROM link_acl_oa"), "dblp_only_no_openalex_record": n("SELECT sum(n) FROM dblp_only_counts")}
84 + P["excluded"] = {k: n(f"SELECT count(*) FROM corpus_works WHERE {k}") for k in ["e1_outside_window", "e2_type", "e3_xpac", "e4_no_title", "e5_duplicate", "e7_generic_deposit", "e6_retracted", "xpac_kept_dblp"]}
85 + P["excluded_e2_by_type"] = dict(q("SELECT type, count(*) FROM corpus_works WHERE e2_type GROUP BY 1 ORDER BY 2 DESC"))
86 + P["excluded_sequential"] = {}
87 + cond = "TRUE"
88 + for k in ["e1_outside_window", "e2_type", "e3_xpac", "e4_no_title", "e5_duplicate", "e7_generic_deposit"]:
89 + P["excluded_sequential"][k] = n(f"SELECT count(*) FROM corpus_works WHERE {cond} AND {k}"); cond += f" AND NOT {k}"
90 + P["after_exclusions"] = n(f"SELECT count(*) FROM corpus_works WHERE {cond}")
91 + P["cs_core_eligible"] = n("SELECT count(*) FROM corpus_works WHERE eligible")
92 + P["cs_penumbra_eligible_otherwise"] = n(f"SELECT count(*) FROM corpus_works WHERE {cond} AND cs_penumbra")
93 + P["eligible_by_period"] = dict(q("SELECT period, count(*) FROM corpus_works WHERE eligible GROUP BY 1"))
94 + P["eligible_by_year"] = dict(q("SELECT publication_year, count(*) FROM corpus_works WHERE eligible GROUP BY 1 ORDER BY 1"))
95 + P["eligible_by_type"] = dict(q("SELECT type, count(*) FROM corpus_works WHERE eligible GROUP BY 1 ORDER BY 2 DESC"))
96 + P["eligible_membership"] = {"in_dblp": n("SELECT count(*) FROM corpus_works WHERE eligible AND in_dblp"), "cs_primary": n("SELECT count(*) FROM corpus_works WHERE eligible AND cs_primary"),
97 + "arxiv_linked": n("SELECT count(*) FROM corpus_works WHERE eligible AND arxiv_id_linked IS NOT NULL"), "in_acl": n("SELECT count(*) FROM corpus_works WHERE eligible AND in_acl"),
98 + "has_abstract": n("SELECT count(*) FROM corpus_works WHERE eligible AND has_abstract"), "has_references": n("SELECT count(*) FROM corpus_works WHERE eligible AND n_refs > 0"),
99 + "retracted_flagged": n("SELECT count(*) FROM corpus_works WHERE eligible AND e6_retracted")}
100 + P["eligible_date_precision"] = dict(q("SELECT date_precision, count(*) FROM corpus_works WHERE eligible GROUP BY 1"))
101 + P["strict_core_dblp_and_cs_primary"] = n("SELECT count(*) FROM corpus_works WHERE eligible AND in_dblp AND cs_primary")
102 + os.makedirs(os.path.join(R, "data/processed/corpus"), exist_ok=True); os.makedirs(os.path.join(R, "data/metadata"), exist_ok=True)
103 + con.execute(f"COPY corpus_works TO '{R}/data/processed/corpus/corpus_works.parquet' (FORMAT PARQUET, COMPRESSION ZSTD)")
104 + con.execute(f"COPY dblp_only_counts TO '{R}/data/metadata/dblp_only_counts.csv' (HEADER)")
105 + json.dump(P, open(os.path.join(R, "data/metadata/prisma_counts.json"), "w"), indent=1, default=str)
106 + print(json.dumps(P, indent=1, default=str)); log("done")
107 +
108 +
109 +if __name__ == "__main__":
110 + main()
added systematic_review/inspect_2026.py +11 −0
@@ -0,0 +1,11 @@
1 +import duckdb, sys
2 +con = duckdb.connect(sys.argv[1] + "/data/processed/wp44.duckdb", read_only=True)
3 +q = lambda s: con.execute(s).fetchall()
4 +print("2026 eligible by first_month:", q("select first_month, count(*) from corpus_works where eligible and publication_year=2026 group by 1 order by 1"))
5 +print("2026 by type:", q("select type, count(*) from corpus_works where eligible and publication_year=2026 group by 1 order by 2 desc limit 6"))
6 +print("2026 top sources:", q("select source_name, count(*) from corpus_works where eligible and publication_year=2026 group by 1 order by 2 desc limit 10"))
7 +print("2025 top sources:", q("select source_name, count(*) from corpus_works where eligible and publication_year=2025 group by 1 order by 2 desc limit 10"))
8 +print("2026 pub_date day-01 share:", q("select sum((substr(publication_date,9,2)='01')::int)*1.0/count(*) from corpus_works where eligible and publication_year=2026"))
9 +print("2025 pub_date day-01 share:", q("select sum((substr(publication_date,9,2)='01')::int)*1.0/count(*) from corpus_works where eligible and publication_year=2025"))
10 +print("2026 created_date months:", q("select substr(created_date,1,7), count(*) from corpus_works where eligible and publication_year=2026 group by 1 order by 1"))
11 +print("other-type dblp sample:", q("select id, doi, source_name, publication_year, n_refs from corpus_works where in_dblp and type='other' limit 4"))
added systematic_review/inspect_sources.py +10 −0
@@ -0,0 +1,10 @@
1 +import duckdb, sys
2 +con = duckdb.connect(sys.argv[1] + "/data/processed/wp44.duckdb", read_only=True)
3 +q = lambda s: con.execute(s).fetchall()
4 +print("eligible by source_type:", q("select source_type, count(*) n, sum(in_dblp::int) dblp, sum((arxiv_id_linked is not null)::int) arx, sum(has_abstract::int) ab, sum((n_refs>0)::int) refs from corpus_works where eligible group by 1 order by 2 desc"))
5 +print("repository sources top 15:", q("select source_name, count(*) n, sum(in_dblp::int) dblp, sum((arxiv_id_linked is not null)::int) arx, sum((n_refs>0)::int) refs, sum(has_abstract::int) ab from corpus_works where eligible and source_type='repository' group by 1 order by 2 desc limit 15"))
6 +print("Zenodo by year:", q("select publication_year, count(*), sum((n_refs>0)::int) refs, sum(has_abstract::int) ab, sum(in_dblp::int) dblp from corpus_works where eligible and source_name like 'Zenodo%' group by 1 order by 1"))
7 +print("Zenodo 2026 sample:", q("select substr(title,1,70), type, n_refs, has_abstract, n_authors, language from corpus_works where eligible and source_name like 'Zenodo%' and publication_year=2026 using sample 8"))
8 +print("NULL-source by year:", q("select publication_year, count(*), sum((n_refs>0)::int) refs, sum(has_abstract::int) ab, sum(in_dblp::int) dblp, sum((type='dissertation')::int) diss from corpus_works where eligible and source_id is null group by 1 order by 1"))
9 +print("NULL-source 2025 sample:", q("select substr(title,1,60), type, n_refs, has_abstract, n_locations, indexed_in from corpus_works where eligible and source_id is null and publication_year=2025 using sample 8"))
10 +print("day-01-01 share by year:", q("select publication_year, round(sum((substr(publication_date,6,5)='01-01')::int)*1.0/count(*),3) from corpus_works where eligible group by 1 order by 1"))
added taxonomy/classify.py +150 −0
@@ -0,0 +1,150 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — Phase 2: assign the 40 ontology categories (multi-label) to every work of the corpus.
3 +
4 +Signals per work: OpenAlex topics (ids + subfield ids), arXiv categories (through link_arxiv_oa), DBLP venue stream
5 +(through link_dblp_oa), keyword regexes on title and abstract. Scoring follows ontology.yaml `assignment_rule`:
6 +per signal type the matched weights are summed and capped at 1.0, then summed over signal types; a category is assigned
7 +if score >= threshold; primary = argmax (ties: child before parent, then lower category number).
8 +
9 +Step 1 (`--prepare`): DuckDB builds data/processed/classification/input/chunk_*.parquet (id, year, title, abstract,
10 +topic_ids, subfield_ids, arxiv_cats, dblp_streams). Step 2 (`--run`): multiprocessing over chunks →
11 +data/processed/classification/work_categories.parquet (work_id, category_id, score, s_topic, s_subfield, s_arxiv,
12 +s_dblp, s_kw) and work_primary.parquet (work_id, primary_category, n_categories). Deterministic; version-stamped.
13 +"""
14 +import argparse, glob, json, multiprocessing as mp, os, re, time, datetime as dt
15 +import yaml, duckdb
16 +import pyarrow as pa, pyarrow.parquet as pq
17 +
18 +ONT = None
19 +
20 +
21 +def load_ontology(path):
22 + o = yaml.safe_load(open(path))
23 + cats = o["categories"]
24 + comp = []
25 + for c in cats:
26 + kws = []
27 + for k in (c.get("keywords") or []):
28 + kws.append((re.compile(k["pattern"]), float(k["weight"]), k.get("field", "both")))
29 + exc = [re.compile(p if isinstance(p, str) else p["pattern"]) for p in (c.get("exclusions") or [])]
30 + comp.append(dict(
31 + id=c["id"], number=int(c["number"]), parent=c.get("child_of"),
32 + topics={str(t["id"]).lstrip("T"): float(t["weight"]) for t in (c.get("openalex_topics") or [])},
33 + subfields={str(t["id"]): float(t["weight"]) for t in (c.get("openalex_subfields") or [])},
34 + arxiv={t["id"]: float(t["weight"]) for t in (c.get("arxiv_categories") or [])},
35 + dblp={t["key"]: float(t["weight"]) for t in (c.get("dblp_streams") or [])},
36 + kws=kws, exc=exc))
37 + rule = o.get("assignment_rule", {})
38 + return dict(cats=comp, threshold=float(rule.get("threshold", 1.0)), version=o.get("version"))
39 +
40 +
41 +def _init(path):
42 + global ONT
43 + ONT = load_ontology(path)
44 +
45 +
46 +def score_row(title, abstract, topic_ids, subfield_ids, arxiv_cats, dblp_streams):
47 + title = title or ""; abstract = abstract or ""; both = title + "\n" + abstract
48 + out = []
49 + for c in ONT["cats"]:
50 + st = min(1.0, sum(c["topics"].get(str(t), 0.0) for t in (topic_ids or [])))
51 + ss = min(1.0, sum(c["subfields"].get(str(s), 0.0) for s in set(subfield_ids or [])))
52 + sa = min(1.0, sum(c["arxiv"].get(a, 0.0) for a in set(arxiv_cats or [])))
53 + sd = min(1.0, sum(c["dblp"].get(d, 0.0) for d in set(dblp_streams or [])))
54 + sk = 0.0
55 + for rx, w, field in c["kws"]:
56 + txt = title if field == "title" else abstract if field == "abstract" else both
57 + if txt and rx.search(txt):
58 + sk += w
59 + if sk >= 1.0:
60 + sk = 1.0; break
61 + s = st + ss + sa + sd + sk
62 + if s >= ONT["threshold"] and c["exc"] and any(rx.search(both) for rx in c["exc"]):
63 + s = 0.0
64 + if s >= ONT["threshold"]:
65 + out.append((c["id"], c["number"], c["parent"], round(s, 3), st, ss, sa, sd, sk))
66 + return out
67 +
68 +
69 +def process_chunk(args):
70 + path, out_dir = args
71 + name = os.path.basename(path).replace(".parquet", "")
72 + o1 = os.path.join(out_dir, "cats", name + ".parquet"); o2 = os.path.join(out_dir, "primary", name + ".parquet")
73 + if os.path.exists(o1) and os.path.exists(o2):
74 + return name, 0, 0
75 + t = pq.read_table(path).to_pydict()
76 + cw, cc, cs, c1, c2, c3, c4, c5 = [], [], [], [], [], [], [], []
77 + pw, pc, pn, ps = [], [], [], []
78 + n = len(t["id"])
79 + for i in range(n):
80 + res = score_row(t["title"][i], t["abstract"][i], t["topic_ids"][i], t["subfield_ids"][i], t["arxiv_cats"][i], t["dblp_streams"][i])
81 + for (cid, num, parent, s, st, ss, sa, sd, sk) in res:
82 + cw.append(t["id"][i]); cc.append(cid); cs.append(s); c1.append(st); c2.append(ss); c3.append(sa); c4.append(sd); c5.append(sk)
83 + if res:
84 + parents = {r[2] for r in res if r[2]}
85 + # primary: highest score; tie → child before parent (a category that is someone's parent loses), then lower number
86 + best = sorted(res, key=lambda r: (-r[3], 1 if r[0] in parents else 0, r[1]))[0]
87 + pw.append(t["id"][i]); pc.append(best[0]); pn.append(len(res)); ps.append(best[3])
88 + os.makedirs(os.path.dirname(o1), exist_ok=True); os.makedirs(os.path.dirname(o2), exist_ok=True)
89 + pq.write_table(pa.table({"work_id": pa.array(cw, pa.int64()), "category_id": cc, "score": pa.array(cs, pa.float32()), "s_topic": pa.array(c1, pa.float32()),
90 + "s_subfield": pa.array(c2, pa.float32()), "s_arxiv": pa.array(c3, pa.float32()), "s_dblp": pa.array(c4, pa.float32()), "s_kw": pa.array(c5, pa.float32())}), o1 + ".part", compression="zstd")
91 + os.replace(o1 + ".part", o1)
92 + pq.write_table(pa.table({"work_id": pa.array(pw, pa.int64()), "primary_category": pc, "n_categories": pa.array(pn, pa.int16()), "primary_score": pa.array(ps, pa.float32())}), o2 + ".part", compression="zstd")
93 + os.replace(o2 + ".part", o2)
94 + return name, n, len(pw)
95 +
96 +
97 +def prepare(root, db, out_dir, chunk_rows):
98 + con = duckdb.connect(db); con.execute("SET threads TO 24; SET memory_limit='60GB';")
99 + os.makedirs(os.path.join(out_dir, "input"), exist_ok=True)
100 + con.execute("""CREATE OR REPLACE TABLE cls_input AS
101 + SELECT w.id, w.publication_year AS year, w.title, a.abstract,
102 + (SELECT list(topic_id ORDER BY rank) FROM oa_work_topics t WHERE t.work_id = w.id) AS topic_ids,
103 + (SELECT list(subfield_id ORDER BY rank) FROM oa_work_topics t WHERE t.work_id = w.id) AS subfield_ids,
104 + (SELECT flatten(list(x.categories)) FROM link_arxiv_oa l JOIN arxiv_keys x USING(arxiv_id) WHERE l.oa_id = w.id) AS arxiv_cats,
105 + (SELECT list(DISTINCT d.stream) FROM link_dblp_oa l JOIN dblp_keys d USING(dblp_key) WHERE l.oa_id = w.id) AS dblp_streams
106 + FROM corpus_works w LEFT JOIN oa_abstracts a ON a.work_id = w.id""")
107 + n = con.execute("SELECT count(*) FROM cls_input").fetchone()[0]
108 + con.execute(f"""COPY (SELECT *, (row_number() OVER ()) // {chunk_rows} AS chunk FROM cls_input) TO '{out_dir}/input' (FORMAT PARQUET, PARTITION_BY (chunk), OVERWRITE_OR_IGNORE, COMPRESSION ZSTD)""")
109 + print("prepared", n, "works")
110 +
111 +
112 +def main():
113 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); ap.add_argument("--ontology", required=True)
114 + ap.add_argument("--prepare", action="store_true"); ap.add_argument("--run", action="store_true"); ap.add_argument("--workers", type=int, default=20)
115 + ap.add_argument("--chunk-rows", type=int, default=100000); ap.add_argument("--limit", type=int, default=0)
116 + a = ap.parse_args(); out_dir = os.path.join(a.root, "data/processed/classification"); db = os.path.join(a.root, "data/processed/wp44.duckdb")
117 + if a.prepare:
118 + prepare(a.root, db, out_dir, a.chunk_rows)
119 + if a.run:
120 + files = sorted(glob.glob(os.path.join(out_dir, "input", "chunk=*", "*.parquet")))
121 + files = [(f, out_dir) for f in files]
122 + if a.limit:
123 + files = files[: a.limit]
124 + # chunk files share the name data_0.parquet inside chunk=N dirs → rename key
125 + files = [(f, out_dir) for f, _ in files]
126 + t0 = time.time(); tot = 0; cl = 0
127 + with mp.Pool(a.workers, initializer=_init, initargs=(a.ontology,)) as pool:
128 + for name, n, k in pool.imap_unordered(process_chunk_named, [(f, out_dir, os.path.basename(os.path.dirname(f))) for f, _ in files]):
129 + tot += n; cl += k
130 + print(f"[{time.time()-t0:6.0f}s] {name}: {n} works, {k} classified | total {tot} / classified {cl}", flush=True)
131 + json.dump({"ontology": a.ontology, "version": load_ontology(a.ontology)["version"], "works": tot, "classified": cl, "when": dt.datetime.now(dt.UTC).isoformat(),
132 + "seconds": round(time.time() - t0)}, open(os.path.join(out_dir, "classification_report.json"), "w"), indent=1)
133 +
134 +
135 +def process_chunk_named(args):
136 + path, out_dir, chunk_name = args
137 + name = chunk_name.replace("=", "_")
138 + o1 = os.path.join(out_dir, "cats", name + ".parquet"); o2 = os.path.join(out_dir, "primary", name + ".parquet")
139 + if os.path.exists(o1) and os.path.exists(o2):
140 + return name, 0, 0
141 + # reuse process_chunk with a symlinked name
142 + tmp = os.path.join(out_dir, "input_named"); os.makedirs(tmp, exist_ok=True)
143 + link = os.path.join(tmp, name + ".parquet")
144 + if not os.path.exists(link):
145 + os.symlink(path, link)
146 + return process_chunk((link, out_dir))
147 +
148 +
149 +if __name__ == "__main__":
150 + main()
added taxonomy/src/area_ai_ml_a.py +153 −0
@@ -0,0 +1,153 @@
1 +"""Area ai_ml, part A: categories 1-4 (LLM, foundation models, multimodal AI, generative AI).
2 +
3 +Tuple conventions (converted to dicts by build_ontology.py):
4 + topics: (openalex_topic_id, weight, note)
5 + subfields:(subfield_id, weight)
6 + arxiv: (category, weight)
7 + streams: (dblp_stream_key, weight)
8 + keywords: (regex, weight, field) field in {"title", "abstract", "both"}
9 +"""
10 +
11 +CATEGORIES = [
12 +dict(
13 + id="llm", number=1, name="Large Language Models", area="ai_ml",
14 + definition=("Research whose object is a transformer-based language model trained at scale: pre-training, "
15 + "instruction tuning, prompting and in-context learning, reasoning, retrieval augmentation, efficient "
16 + "inference, evaluation and benchmarks of LLMs. Excludes classical NLP tasks solved with small task-specific "
17 + "models (nlp) and non-language foundation models (foundation_models)."),
18 + topics=[("T10028", 1.0, "OpenAlex cluster hosting BERT / pre-trained LM / RAG / QA work: the de-facto LLM topic"),
19 + ("T11714", 0.4, "multimodal LLMs and vision-language models"),
20 + ("T12031", 0.3, "dialogue systems, LLM chatbots"),
21 + ("T10181", 0.3, "NLP techniques incl. neural MT and NLG"),
22 + ("T12128", 0.3, "chatbots in service interactions"),
23 + ("T14414", 0.2, "AI in education: mostly ChatGPT studies post-2023")],
24 + subfields=[(1702, 0.2)],
25 + arxiv=[("cs.CL", 0.5), ("cs.AI", 0.2), ("cs.LG", 0.2)],
26 + streams=[("conf/acl", 0.5), ("conf/emnlp", 0.5), ("conf/naacl", 0.5), ("conf/eacl", 0.4), ("conf/coling", 0.4),
27 + ("journals/tacl", 0.5), ("conf/iclr", 0.3), ("conf/nips", 0.3), ("conf/icml", 0.3),
28 + ("journals/tmlr", 0.3), ("conf/aaai", 0.2), ("conf/nlpcc", 0.3)],
29 + keywords=[
30 + (r"(?i)\blarge language models?\b", 1.0, "both"),
31 + (r"(?i)\bLLMs?\b", 1.0, "both"),
32 + (r"(?i)\b(ChatGPT|GPT-?3(\.5)?|GPT-?4o?|GPT-?4\.\d|GPT-?5|LLaMA|Llama-? ?[234]|PaLM|InstructGPT|Qwen|Vicuna|Mixtral|DeepSeek-?(V\d|R1)|Claude ?[34]|Gemini ?(Pro|Ultra|1\.5|2))\b", 1.0, "both"),
33 + (r"(?i)\bpre-?trained language models?\b|\bPLMs?\b", 0.7, "both"),
34 + (r"(?i)\b(instruction[- ]tuning|instruction-?tuned|in-?context learning|chain[- ]of[- ]thought|retrieval[- ]augmented generation|RAG\b|prompt (engineering|tuning|learning)|prompting|RLHF|supervised fine-?tuning|SFT\b)\b", 0.8, "both"),
35 + (r"(?i)\b(masked|autoregressive|causal|decoder-?only|encoder-?decoder) language model", 0.7, "both"),
36 + (r"(?i)\b(language model|transformer) (pre-?training|scaling|scaling laws?)\b", 0.7, "both"),
37 + (r"(?i)\b(parameter-?efficient fine-?tuning|PEFT\b|LoRA\b|adapters? for (language|LLM))\b", 0.6, "both"),
38 + (r"(?i)\bBERT\b|\bRoBERTa\b|\bT5\b|\bBART\b", 0.6, "both"),
39 + (r"(?i)\bhallucinat(ion|ions|e|es|ing)\b", 0.5, "both"),
40 + (r"(?i)\blanguage models?\b", 0.4, "both"),
41 + (r"(?i)\b(tokeni[sz](er|ation)|context (window|length)|KV[- ]cache|speculative decoding|mixture[- ]of[- ]experts)\b", 0.4, "both"),
42 + ],
43 + exclusions=[],
44 + related=["foundation_models", "nlp", "generative_ai", "ai_agents", "ai_safety"],
45 + parent_of=[], child_of=["foundation_models"],
46 + provenance=("Topics chosen by inspecting OpenAlex keyword lists (T10028 lists BERT, pre-trained language models, "
47 + "retrieval-augmented generation). Venues: *CL conferences and ML venues at partial weight because LLM "
48 + "work is spread across them. Keyword patterns reach the threshold alone, as required for the six "
49 + "AI categories OpenAlex cannot separate."),
50 +),
51 +dict(
52 + id="foundation_models", number=2, name="Foundation Models", area="ai_ml",
53 + definition=("Large pre-trained models intended for adaptation to many downstream tasks in any modality: scaling "
54 + "laws, pre-training objectives, self-supervised representation learning at scale, transfer, fine-tuning "
55 + "and adapter methods, model releases and their evaluation. Language-specific work is llm (a child); "
56 + "vision-language work is multimodal_ai (a child)."),
57 + topics=[("T11307", 0.5, "domain adaptation, few-shot, parameter-efficient fine-tuning, continual learning"),
58 + ("T11714", 0.4, "vision-language models / CLIP"),
59 + ("T10028", 0.3, "pre-trained language models"),
60 + ("T10036", 0.2, "backbone networks, model compression"),
61 + ("T12422", 0.2, "foundation models in medical imaging / radiomics")],
62 + subfields=[(1702, 0.2)],
63 + arxiv=[("cs.LG", 0.4), ("cs.CV", 0.2), ("cs.CL", 0.2), ("cs.AI", 0.2)],
64 + streams=[("conf/nips", 0.3), ("conf/icml", 0.3), ("conf/iclr", 0.3), ("journals/tmlr", 0.3), ("conf/cvpr", 0.2),
65 + ("conf/iccv", 0.2), ("conf/eccv", 0.2), ("journals/natmi", 0.2)],
66 + keywords=[
67 + (r"(?i)\bfoundation models?\b|\bfoundational models?\b", 1.0, "both"),
68 + (r"(?i)\b(vision|visual|time[- ]series|protein|genomic|geospatial|medical|clinical|weather|climate|chemistry|molecular|tabular|graph|speech|audio|robotics?|world) foundation models?\b", 1.0, "both"),
69 + (r"(?i)\bscaling laws?\b", 0.8, "both"),
70 + (r"(?i)\b(self-?supervised|contrastive|masked) (pre-?training|representation learning|learning at scale)\b", 0.7, "both"),
71 + (r"(?i)\b(pre-?train(ed|ing)) (on|at) (web|internet|large|massive|billion)[- ]scale", 0.7, "both"),
72 + (r"(?i)\b(vision transformers?|ViT-?[BLHG]?|DINO(v2)?|segment anything|SAM ?2|masked auto-?encoders?|MAE\b|CLIP\b|SigLIP)\b", 0.6, "both"),
73 + (r"(?i)\b(general-?purpose|generalist|universal|all-?in-?one) (models?|agents?|representations?)\b", 0.6, "both"),
74 + (r"(?i)\b(transfer learning|fine-?tun(ing|ed?)|downstream tasks?|zero-?shot|few-?shot)\b", 0.4, "both"),
75 + (r"(?i)\b(adapters?|LoRA|parameter-?efficient|prompt tuning|emergent (abilities|capabilities))\b", 0.5, "both"),
76 + (r"(?i)\bworld models?\b", 0.5, "both"),
77 + ],
78 + exclusions=[],
79 + related=["llm", "multimodal_ai", "generative_ai", "deep_learning_architectures", "ml_theory"],
80 + parent_of=["llm", "multimodal_ai"], child_of=[],
81 + provenance=("No OpenAlex topic isolates foundation models; T11307 (PEFT, few-shot, continual) is the closest "
82 + "methodological host. Keyword patterns carry the category; the generic 'transfer learning/fine-tuning' "
83 + "terms are deliberately below threshold."),
84 +),
85 +dict(
86 + id="multimodal_ai", number=3, name="Multimodal AI", area="ai_ml",
87 + definition=("Models and systems that jointly process two or more modalities (vision-language, audio-visual, "
88 + "video-text, 3D-language, sensors with language): vision-language models and multimodal LLMs, "
89 + "captioning, visual question answering, cross-modal retrieval and grounding. Unimodal vision is "
90 + "computer_vision; pure speech is speech_audio."),
91 + topics=[("T11714", 1.0, "Multimodal Machine Learning Applications: VLMs, MLLMs, VQA, captioning"),
92 + ("T10627", 0.4, "image and video retrieval incl. cross-modal"),
93 + ("T11439", 0.2, "video analysis and summarization"),
94 + ("T10824", 0.2, "image retrieval and classification"),
95 + ("T13155", 0.2, "digital communication and language (multimodal discourse)")],
96 + subfields=[(1707, 0.1)],
97 + arxiv=[("cs.MM", 0.6), ("cs.CV", 0.3), ("cs.CL", 0.3), ("cs.LG", 0.2)],
98 + streams=[("conf/mm", 0.6), ("journals/tmm", 0.6), ("conf/icmi", 0.6), ("journals/tomccap", 0.5),
99 + ("conf/icmcs", 0.5), ("conf/mir", 0.5), ("journals/mms", 0.5), ("journals/mta", 0.3),
100 + ("conf/cvpr", 0.2), ("conf/iccv", 0.2), ("conf/eccv", 0.2), ("conf/acl", 0.2), ("conf/emnlp", 0.2)],
101 + keywords=[
102 + (r"(?i)\bmulti-?modal(ity)?\b", 0.7, "both"),
103 + (r"(?i)\b(vision[- ]language|vision[- ]and[- ]language|visual[- ]language|language[- ]vision|image[- ]text|video[- ]language|audio[- ]language|speech[- ]language|3D[- ]language|text[- ]to[- ]3D)[- ]?(models?|pre-?training|tasks?|alignment|understanding|learning|reasoning|navigation|action|retrieval|generation)?\b", 1.0, "both"),
104 + (r"(?i)\b(VLMs?|MLLMs?|LVLMs?|LMMs?|VLAs?)\b", 1.0, "both"),
105 + (r"(?i)\bvisual question answering\b|\bVQA\b|\bvideo question answering\b", 1.0, "both"),
106 + (r"(?i)\b(image|video|dense|audio) captioning\b", 1.0, "both"),
107 + (r"(?i)\bcross-?modal(ity)?\b", 0.8, "both"),
108 + (r"(?i)\b(audio-?visual|visual grounding|referring expression|visual dialog|visual reasoning|visual instruction tuning|LLaVA|BLIP-?2?|Flamingo|GPT-?4V|CLIP|ALIGN\b|CoCa)\b", 0.8, "both"),
109 + (r"(?i)\b(text-to-image|text-to-video|image-to-text|text-image|image-text) (retrieval|alignment|matching|generation|models?)\b", 0.8, "both"),
110 + (r"(?i)\b(modalit(y|ies) (fusion|alignment|gap)|(early|late|intermediate) fusion of|sensor fusion with language)\b", 0.7, "both"),
111 + ],
112 + exclusions=[r"(?i)\bmultimodal (therapy|analgesi|treatment|intervention|pain|anesthesi|anaesthesi|imaging biomarkers?|prehabilitation|transport)\b"],
113 + related=["computer_vision", "nlp", "llm", "speech_audio", "generative_ai"],
114 + parent_of=[], child_of=["foundation_models"],
115 + provenance=("T11714 is the only OpenAlex topic centred on vision-language learning. Multimedia venues at partial "
116 + "weight (they also host unimodal media work). Medical 'multimodal' senses are vetoed."),
117 +),
118 +dict(
119 + id="generative_ai", number=4, name="Generative AI", area="ai_ml",
120 + definition=("Models that synthesise new content (images, video, audio, 3D, text, code, molecules) and their training, "
121 + "sampling, evaluation, detection and misuse: GANs, VAEs, diffusion and flow models, autoregressive "
122 + "generators, text-to-X synthesis, deepfakes and AI-generated-content detection. LLM text generation as "
123 + "such is llm unless the contribution concerns the generative model or its outputs' authenticity."),
124 + topics=[("T10775", 1.0, "GANs, diffusion models, image/video generation"),
125 + ("T12357", 0.5, "digital media forensics: deepfake detection"),
126 + ("T11349", 0.2, "music technology (music generation)"),
127 + ("T10388", 0.2, "steganography / watermarking (of generated content)"),
128 + ("T14254", 0.2, "digital media and visual art (AI art)")],
129 + subfields=[],
130 + arxiv=[("cs.CV", 0.2), ("cs.LG", 0.3), ("cs.GR", 0.2), ("cs.SD", 0.1), ("cs.AI", 0.1), ("cs.MM", 0.1)],
131 + streams=[("conf/nips", 0.2), ("conf/icml", 0.2), ("conf/iclr", 0.2), ("conf/cvpr", 0.2), ("conf/iccv", 0.2),
132 + ("conf/eccv", 0.2), ("conf/siggraph", 0.2), ("conf/siggrapha", 0.2), ("journals/tog", 0.1)],
133 + keywords=[
134 + (r"(?i)\bgenerative (AI|artificial intelligence|models?|modell?ing|adversarial networks?|pre-?trained|design|art)\b|\bGenAI\b|\bAIGC\b", 1.0, "both"),
135 + (r"(?i)\bGANs?\b|\b(conditional|cycle|style|Wasserstein)GAN\b|\bStyleGAN\w*\b", 0.9, "both"),
136 + (r"(?i)\b(denoising diffusion|latent diffusion|stable diffusion|diffusion transformers?|DDPM|DDIM|score-?based generative|score matching|flow[- ]matching|rectified flow|normali[sz]ing flows?|consistency models?)\b", 1.0, "both"),
137 + (r"(?i)\bdiffusion models?\b", 0.8, "both"),
138 + (r"(?i)\btext-?to-?(image|video|3D|audio|speech|music|motion|SVG|scene)\b", 1.0, "both"),
139 + (r"(?i)\b(image|video|music|audio|speech|3D|motion|molecule|molecular|code|scene|face|avatar|talking[- ]head|layout|texture|graph) (generation|synthesis)\b", 0.7, "both"),
140 + (r"(?i)\bdeep-?fakes?\b|\bface swap(ping)?\b|\bsynthetic media\b", 1.0, "both"),
141 + (r"(?i)\bvariational auto-?encoders?\b|\bVAEs?\b|\bVQ-?VAE\b|\bVQGAN\b", 0.7, "both"),
142 + (r"(?i)\b(AI-?generated|machine-?generated|synthetic(ally)?[- ]generated|LLM-?generated) (content|images?|text|data|videos?|audio|art|code|faces)\b", 0.8, "both"),
143 + (r"(?i)\b(image (inpainting|outpainting|editing|to-image translation)|style transfer|super-?resolution|image restoration|controllable generation|ControlNet|DreamBooth|textual inversion|classifier-?free guidance|Midjourney|DALL-?E|Imagen|Sora)\b", 0.6, "both"),
144 + (r"(?i)\bautoregressive (models?|generation|image|video)\b", 0.5, "both"),
145 + ],
146 + exclusions=[r"(?is)^(?!.*(generative|GANs?\b|synthesis|generation|deepfake|VAE|AI-?generated)).*\b(drug|gas|molecular|thermal|anomalous|reaction-?|water|heat|ion|oxygen|particle|turbulent|surface|solute|vapou?r) diffusion\b",
147 + r"(?is)^(?!.*(generative|GANs?\b|synthesis|generation|deepfake|VAE|AI-?generated)).*\bdiffusion (coefficients?|tensor|MRI|weighted|equations?|process(es)?|of innovations?|limited)\b"],
148 + related=["llm", "multimodal_ai", "computer_vision", "computer_graphics", "foundation_models", "speech_audio"],
149 + parent_of=[], child_of=[],
150 + provenance=("T10775 keywords explicitly list GANs, diffusion models, Stable Diffusion, text-to-image. Physics/finance "
151 + "senses of 'diffusion' are vetoed only when no generative-modelling term is present."),
152 +),
153 +]
added taxonomy/src/area_ai_ml_b.py +205 −0
@@ -0,0 +1,205 @@
1 +"""Area ai_ml, part B: categories 5-8 and 38 (agents, RL, ML theory, DL architectures, AI safety)."""
2 +
3 +CATEGORIES = [
4 +dict(
5 + id="ai_agents", number=5, name="AI Agents and Autonomous Systems", area="ai_ml",
6 + definition=("Software agents that perceive, decide and act with some autonomy: LLM-based agents and tool use, "
7 + "multi-agent systems, planning and automated reasoning for acting, agent architectures, negotiation, "
8 + "agentic workflows and benchmarks, embodied agents in simulated environments. Physical robots are "
9 + "robotics; RL algorithms as such are reinforcement_learning."),
10 + topics=[("T10456", 1.0, "Multi-Agent Systems and Negotiation: BDI, agent architectures, argumentation"),
11 + ("T10906", 0.6, "AI-based problem solving and planning (PDDL, multi-agent planning)"),
12 + ("T11574", 0.4, "AI in games (game-playing agents)"),
13 + ("T10249", 0.3, "distributed control of multi-agent systems (control-theoretic MAS)"),
14 + ("T12128", 0.2, "chatbots and service agents")],
15 + subfields=[(1702, 0.1)],
16 + arxiv=[("cs.MA", 1.0), ("cs.AI", 0.4)],
17 + streams=[("conf/atal", 1.0), ("conf/ifaamas", 1.0), ("journals/aamas", 1.0), ("conf/icaart", 0.6),
18 + ("conf/aaai", 0.2), ("conf/ijcai", 0.2), ("journals/jair", 0.2), ("journals/ai", 0.2), ("conf/kr", 0.2)],
19 + keywords=[
20 + (r"(?i)\b(LLM|language[- ]model|AI|autonomous|intelligent|embodied|conversational|software|web|GUI|coding|browsing|tool-?using|generative|multi-?step|computer-?use|research|planning)[- ](based |powered |driven )?agents?\b", 1.0, "both"),
21 + (r"(?i)\bagentic\b|\bagent-?based (LLM|AI|systems?|frameworks?|workflows?)\b", 1.0, "both"),
22 + (r"(?i)\bmulti-?agent (systems?|reinforcement|collaboration|coordination|negotiation|planning|simulation|frameworks?|debate|learning|path finding|communication|environments?)\b|\bMAS\b|\bMARL\b", 1.0, "both"),
23 + (r"(?i)\b(tool (use|using|learning|calling|augmented|integration)|function calling|API calling|ReAct|AutoGPT|AutoGen|LangChain|LangGraph|CrewAI|MetaGPT|SWE-?agent|SWE-?bench|WebArena|AgentBench|OSWorld|Model Context Protocol|MCP servers?)\b", 1.0, "both"),
24 + (r"(?i)\b(BDI|belief-?desire-?intention) (agents?|architectures?|models?)\b|\bagent-?oriented (programming|software engineering)\b", 1.0, "both"),
25 + (r"(?i)\b(automated|classical|hierarchical|task|AI|temporal|epistemic) planning\b|\bPDDL\b|\btask (decomposition|and motion planning)\b", 0.7, "both"),
26 + (r"(?i)\bagent-?based (model(ing|s)?|simulations?)\b|\bABM\b", 0.6, "both"),
27 + (r"(?i)\b(autonomous systems?|autonomy|self-?driving|self-?improving|self-?evolving|long-?horizon (tasks?|planning)|memory (module|mechanism|augmented) (for|in) agents?)\b", 0.5, "both"),
28 + (r"(?i)\b(game[- ]playing|general game|AlphaGo|AlphaZero|MuZero|OpenAI Five|AlphaStar|CICERO|Diplomacy)\b", 0.6, "both"),
29 + ],
30 + exclusions=[r"(?i)\b(reagents?|chemical|contrast|therapeutic|antimicrobial|chelating|reducing|oxidi[sz]ing|pharmacological|anti-?cancer|infectious|causative|etiologic(al)?|biological|pathogenic|anesthetic|anaesthetic|alkylating|cross-?linking|blowing|surface-?active|wetting|coupling|cytotoxic|antiviral|antibacterial|antifungal|bleaching|dispersing|sweetening|flavou?ring|leavening|real estate|travel|insurance|customs|shipping|sports|literary|secret|double|free|special) agents?\b"],
31 + related=["llm", "reinforcement_learning", "robotics", "ai_safety", "autonomous_scientific_discovery"],
32 + parent_of=[], child_of=[],
33 + provenance=("T10456 is the classical MAS host; the LLM-agent wave has no OpenAlex topic yet, so keyword patterns on "
34 + "'X agent' (with the qualifier restricted to computing senses) and agent frameworks/benchmarks carry it. "
35 + "Chemical/medical/commercial senses of 'agent' are vetoed."),
36 +),
37 +dict(
38 + id="reinforcement_learning", number=6, name="Reinforcement Learning", area="ai_ml",
39 + definition=("Learning to act from reward signals: value- and policy-based methods, model-based RL, offline and "
40 + "inverse RL, multi-agent RL, bandits, exploration, reward design, RL from human feedback, and RL theory. "
41 + "Applications to robot control are shared with robotics."),
42 + topics=[("T10462", 1.0, "Reinforcement Learning in Robotics: Q-learning, PPO, offline RL, imitation learning"),
43 + ("T12794", 0.5, "adaptive dynamic programming control"),
44 + ("T11574", 0.3, "AI in games"),
45 + ("T10100", 0.1, "metaheuristic optimisation (RL-based hyper-heuristics)")],
46 + subfields=[],
47 + arxiv=[("cs.LG", 0.3), ("cs.AI", 0.2), ("cs.RO", 0.1), ("eess.SY", 0.1), ("cs.MA", 0.1)],
48 + streams=[("conf/nips", 0.2), ("conf/icml", 0.2), ("conf/iclr", 0.2), ("conf/corl", 0.3), ("conf/aaai", 0.1),
49 + ("journals/jmlr", 0.1), ("conf/atal", 0.2), ("journals/tnn", 0.1)],
50 + keywords=[
51 + (r"(?i)\breinforcement learning\b", 1.0, "both"),
52 + (r"(?i)\b(deep|multi-?agent|offline|online|inverse|model-?based|model-?free|safe|hierarchical|meta|batch|distributional|constrained|goal-?conditioned|preference-?based|federated|quantum)[- ]RL\b", 1.0, "both"),
53 + (r"(?i)\b(Q-?learning|DQN|double DQN|Rainbow DQN|PPO|proximal policy optimi[sz]ation|TRPO|actor-?critic|A3C|A2C|SAC\b|soft actor-?critic|TD3|DDPG|policy gradients?|REINFORCE|AlphaZero|MuZero|Dreamer(V\d)?|decision transformers?|world models? for (control|RL|planning))\b", 1.0, "both"),
54 + (r"(?i)\bMarkov decision process(es)?\b|\bMDPs?\b|\bPOMDPs?\b|\bsemi-?MDP\b|\bDec-?POMDP\b", 0.8, "both"),
55 + (r"(?i)\b(reward (functions?|shaping|models?|hacking|design|learning|signals?)|sparse rewards?|exploration[- ]exploitation|intrinsic (motivation|rewards?)|curiosity-?driven|multi-?armed bandits?|contextual bandits?|credit assignment|experience replay|sim-?to-?real)\b", 0.8, "both"),
56 + (r"(?i)\b(imitation learning|behavio(u)?r(al)? cloning|learning from demonstrations?|inverse reinforcement|apprenticeship learning|offline (policy|data) (evaluation|learning))\b", 0.8, "both"),
57 + (r"(?i)\b(RLHF|RLAIF|reinforcement learning from (human|AI) feedback|GRPO|DPO\b|direct preference optimi[sz]ation|RLVR|reasoning models? trained with|verifiable rewards?)\b", 0.8, "both"),
58 + (r"(?i)\b(temporal[- ]difference|value (functions?|iteration)|policy (iteration|evaluation|optimi[sz]ation|learning)|Bellman|dynamic programming for control|optimal control via learning|regret bounds? for (MDP|RL))\b", 0.6, "both"),
59 + (r"(?i)\b(agents?|policies|policy|environments?|episodes?|rollouts?)\b", 0.2, "both"),
60 + ],
61 + exclusions=[r"(?i)\breinforcement (of|in|for) (concrete|steel|soil|composites?|polymers?|fib(er|re)s?|masonry|beams?|slabs?|pavements?)\b(?!.*reinforcement learning)"],
62 + related=["ai_agents", "robotics", "ml_theory", "ai_safety", "llm"],
63 + parent_of=[], child_of=[],
64 + provenance="T10462 is the RL topic (its keywords list MARL, Q-learning, PPO, offline RL). Algorithm names reach the threshold alone.",
65 +),
66 +dict(
67 + id="ml_theory", number=7, name="Machine Learning Theory", area="ai_ml",
68 + definition=("Statistical and computational learning theory and principled ML methodology: generalisation and sample "
69 + "complexity, optimisation theory for learning, kernel and Bayesian methods, probabilistic models, causal "
70 + "inference, online learning, PAC learning, interpretability and fairness as formal problems. "
71 + "Architecture engineering is deep_learning_architectures."),
72 + topics=[("T12072", 1.0, "Machine Learning and Algorithms: PAC learning, generalisation bounds, VC dimension"),
73 + ("T11612", 0.8, "stochastic gradient optimisation, NTK, implicit regularisation"),
74 + ("T12814", 0.7, "Gaussian processes and Bayesian inference"),
75 + ("T11303", 0.6, "Bayesian modelling and causal inference"),
76 + ("T10845", 0.5, "advanced causal inference techniques"),
77 + ("T11901", 0.5, "Bayesian methods and mixture models"),
78 + ("T12535", 0.4, "ML and data classification (SVM, ensembles)"),
79 + ("T10637", 0.4, "clustering algorithms"),
80 + ("T12026", 0.4, "explainable AI"),
81 + ("T12536", 0.4, "topological and geometric data analysis"),
82 + ("T11652", 0.3, "imbalanced classification"),
83 + ("T11689", 0.3, "adversarial robustness (certified robustness)"),
84 + ("T10963", 0.3, "advanced optimisation algorithms"),
85 + ("T12761", 0.3, "data stream mining (concept drift)"),
86 + ("T12205", 0.3, "time series forecasting"),
87 + ("T13373", 0.2, "data mining and ML applications"),
88 + ("T12676", 0.2, "extreme learning machines")],
89 + subfields=[(1702, 0.3)],
90 + arxiv=[("stat.ML", 0.8), ("cs.LG", 0.4), ("math.ST", 0.3), ("math.OC", 0.2)],
91 + streams=[("conf/colt", 1.0), ("conf/alt", 1.0), ("journals/jmlr", 0.8), ("conf/aistats", 0.8), ("conf/uai", 0.8),
92 + ("journals/ml", 0.6), ("journals/tmlr", 0.4), ("conf/nips", 0.3), ("conf/icml", 0.3), ("conf/iclr", 0.2),
93 + ("journals/tnn", 0.3), ("journals/nn", 0.3), ("journals/pr", 0.2), ("conf/pkdd", 0.3), ("conf/kdd", 0.2),
94 + ("conf/icdm", 0.2), ("journals/pami", 0.2), ("journals/ijar", 0.4), ("journals/sac", 0.4),
95 + ("journals/csda", 0.3), ("journals/ml", 0.6), ("journals/cstat", 0.2), ("journals/eswa", 0.1),
96 + ("journals/kbs", 0.1), ("journals/isci", 0.1), ("journals/ijon", 0.2), ("journals/nca", 0.1),
97 + ("journals/mlc", 0.3), ("journals/apin", 0.2), ("journals/air", 0.2)],
98 + keywords=[
99 + (r"(?i)\b(generali[sz]ation (bounds?|error|gap|theory)|sample complexity|regret (bounds?|minimi[sz]ation|analysis)|minimax (rates?|optimal|lower)|PAC(-Bayes(ian)?)?( learning| bounds?)?|Rademacher|VC[- ]dimension|convergence (rates?|guarantees?|analysis)|statistical learning theory|computational learning theory|learnability|uniform convergence|excess risk)\b", 0.9, "both"),
100 + (r"(?i)\b(kernel (methods?|machines?|ridge|mean embedding)|reproducing kernel|RKHS|Gaussian process(es)?|variational inference|Bayesian (optimi[sz]ation|neural networks?|inference|deep learning|nonparametric)|probabilistic (models?|programming|graphical)|graphical models?|expectation[- ]maximi[sz]ation|Markov chain Monte Carlo|MCMC|Hamiltonian Monte Carlo|normali[sz]ing flows? for inference|conformal prediction|calibration)\b", 0.8, "both"),
101 + (r"(?i)\bcausal (inference|discovery|representation learning|effect estimation|structure learning|machine learning)\b|\bcounterfactual (reasoning|inference|fairness)\b|\bstructural causal models?\b|\btreatment effect\b", 0.8, "both"),
102 + (r"(?i)\b(online learning|bandits?|stochastic (gradient|optimi[sz]ation|approximation)|non-?convex optimi[sz]ation|convex optimi[sz]ation for learning|implicit (bias|regulari[sz]ation)|gradient (descent|flow) dynamics|saddle points?|loss landscapes?|mirror descent|optimal transport|Wasserstein)\b", 0.6, "both"),
103 + (r"(?i)\b(double descent|neural tangent kernel|NTK\b|lottery ticket|over-?parameteri[sz](ed|ation)|benign overfitting|grokking|feature learning (regime|theory)|mean-?field (limit|theory) of neural|infinite-?width|random features?)\b", 1.0, "both"),
104 + (r"(?i)\b(interpretab(le|ility)|explainab(le|ility)) (machine learning|AI|models?|methods?)\b|\bXAI\b|\bSHAP\b|\bLIME\b|\bfeature attribution\b|\bsaliency maps? for\b", 0.5, "both"),
105 + (r"(?i)\b(fairness (constraints?|metrics?|in machine learning)|algorithmic fairness|demographic parity|equali[sz]ed odds|fair (classification|representation))\b", 0.5, "both"),
106 + (r"(?i)\b(semi-?supervised|self-?supervised|unsupervised|active|transfer|meta|continual|lifelong|few-?shot|zero-?shot|multi-?task|weakly[- ]supervised|positive-?unlabeled|contrastive|representation|manifold|metric|dictionary|ensemble|boosting|bagging) learning\b", 0.4, "both"),
107 + (r"(?i)\b(distribution shift|out-?of-?distribution|domain generali[sz]ation|covariate shift|label noise|robust(ness)? (to|under|against) (noise|shift|corruption)|concept drift|anomaly detection|outlier detection|imbalanced (data|learning|classification)|dimensionality reduction|matrix (completion|factori[sz]ation)|tensor (decomposition|methods?)|sparse (coding|recovery)|compressed sensing)\b", 0.5, "both"),
108 + (r"(?i)\b(support vector machines?|SVMs?|random forests?|gradient boosting|XGBoost|LightGBM|k-?means|spectral clustering|Gaussian mixture|hidden Markov models?|HMMs?|logistic regression|decision trees?|nearest neighbou?rs?|kNN)\b", 0.5, "both"),
109 + (r"(?i)\bmachine learning\b|\bstatistical learning\b|\blearning theory\b", 0.4, "both"),
110 + ],
111 + exclusions=[],
112 + related=["deep_learning_architectures", "reinforcement_learning", "algorithms", "computational_complexity", "ai_safety", "foundation_models"],
113 + parent_of=[], child_of=[],
114 + provenance=("Hosts classical ML and its theory: OpenAlex splits them across T12072 (learning theory), T11612 (optimisation), "
115 + "T12814/T11303/T11901 (Bayesian), T12535/T10637 (classical methods). Venue weights follow venue purity "
116 + "(COLT/ALT/JMLR/AISTATS/UAI strong; NeurIPS/ICML partial)."),
117 +),
118 +dict(
119 + id="deep_learning_architectures", number=8, name="Deep Learning Architectures", area="ai_ml",
120 + definition=("Design, training and compression of neural network architectures as such: CNNs, RNNs, transformers and "
121 + "attention, graph neural networks, state-space models, normalisation and optimisers in practice, neural "
122 + "architecture search, pruning, quantisation and distillation, neuromorphic and spiking networks as models. "
123 + "Task-specific results go to the application category; hardware goes to ai_accelerators."),
124 + topics=[("T10320", 1.0, "Neural Networks and Applications: training, architectures, activation functions"),
125 + ("T11273", 1.0, "Advanced Graph Neural Networks"),
126 + ("T10036", 0.5, "Advanced Neural Network Applications: YOLO, lightweight networks, model compression"),
127 + ("T12676", 0.5, "extreme learning machines"),
128 + ("T12611", 0.4, "reservoir computing / echo state networks"),
129 + ("T11612", 0.3, "DNN training optimisation"),
130 + ("T11347", 0.3, "neural network stability and synchronisation"),
131 + ("T11307", 0.3, "PEFT and continual learning for deep nets"),
132 + ("T12205", 0.2, "time-series forecasting with deep nets"),
133 + ("T10502", 0.2, "neuromorphic computing (spiking models)"),
134 + ("T11206", 0.2, "model reduction and neural networks")],
135 + subfields=[(1702, 0.3)],
136 + arxiv=[("cs.NE", 1.0), ("cs.LG", 0.5), ("cs.CV", 0.1), ("cs.AI", 0.1)],
137 + streams=[("conf/iclr", 0.5), ("conf/nips", 0.4), ("conf/icml", 0.4), ("journals/tnn", 0.6), ("journals/nn", 0.6),
138 + ("conf/ijcnn", 0.6), ("journals/ijon", 0.5), ("journals/nca", 0.5), ("conf/icann", 0.6), ("conf/iconip", 0.5),
139 + ("journals/npl", 0.5), ("conf/esann", 0.5), ("journals/tmlr", 0.3), ("journals/pami", 0.2),
140 + ("journals/cin", 0.3), ("journals/tetci", 0.3), ("journals/tai", 0.3), ("journals/ficn", 0.2),
141 + ("journals/cogcom", 0.2), ("journals/aisy", 0.1), ("conf/aaai", 0.1)],
142 + keywords=[
143 + (r"(?i)\b(deep|convolutional|recurrent|graph|spiking|artificial|feed-?forward|residual|siamese|capsule|physics-?informed|liquid|binary|quanti[sz]ed|Hopfield|Kolmogorov-?Arnold) neural networks?\b", 0.8, "both"),
144 + (r"(?i)\b(CNNs?|RNNs?|LSTMs?|GRUs?|GNNs?|GCNs?|GATs?|SNNs?|MLPs?|DNNs?|ANNs?|KANs?|ResNets?|DenseNet|EfficientNet|MobileNet|U-?Net|ViTs?|Swin)\b", 0.7, "both"),
145 + (r"(?i)\btransformers?\b", 0.5, "both"),
146 + (r"(?i)\b(self-?|cross-?|multi-?head |sparse |linear |flash)?attention (mechanisms?|heads?|layers?|maps?|weights)\b|\battention is all\b", 0.7, "both"),
147 + (r"(?i)\bneural architecture search\b|\bNAS\b.*\b(architecture|search space|supernet)\b|\bAutoML\b|\bdifferentiable architecture\b", 0.9, "both"),
148 + (r"(?i)\b(model compression|knowledge distillation|network pruning|structured pruning|unstructured pruning|post-?training quanti[sz]ation|quanti[sz]ation-?aware training|low-?rank (factori[sz]ation|approximation) of (weights|layers)|weight sharing|parameter sharing|lightweight (networks?|architectures?|models?)|efficient (architectures?|networks?|transformers?))\b", 0.8, "both"),
149 + (r"(?i)\b(batch|layer|group|instance|weight|spectral) normali[sz]ation\b|\bdropout\b|\bresidual (connections?|learning|blocks?)\b|\bskip connections?\b|\bactivation functions?\b|\bReLU\b|\bGELU\b|\bSwiGLU\b|\bsoftmax\b", 0.7, "both"),
150 + (r"(?i)\b(mixture[- ]of[- ]experts|MoE\b|state[- ]space models?|SSMs?\b|Mamba|S4\b|RWKV|Hyena|linear (RNNs?|transformers?)|positional (encodings?|embeddings?)|RoPE|rotary|hypernetworks?|neural ODEs?|implicit (layers|neural representations?)|capsule networks?|equivariant (networks?|neural)|geometric deep learning|message passing (neural|networks?)|graph (convolution|attention|transformers?|pooling)|over-?smoothing)\b", 0.9, "both"),
151 + (r"(?i)\b(backpropagation|back-?prop|gradient (vanishing|exploding)|learning rate (schedul|warm-?up)|Adam(W)?\b|optimi[sz]ers? for (deep|neural)|weight (decay|initiali[sz]ation)|training (dynamics|instabilit)|loss functions? for|curriculum learning|data augmentation|mixup|CutMix|label smoothing|early stopping|hyper-?parameter (tuning|optimi[sz]ation))\b", 0.7, "both"),
152 + (r"(?i)\bdeep learning\b|\bneural networks?\b|\bneural nets?\b", 0.5, "both"),
153 + (r"(?i)\b(spiking|neuromorphic|reservoir computing|echo state|extreme learning machines?|ELM\b)\b", 0.5, "both"),
154 + ],
155 + exclusions=[r"(?is)^(?!.*(neural|deep learning|attention|language model|vision|encoder|decoder)).*\b(power|distribution|current|voltage|instrument|high-?voltage|dry-?type|oil-?immersed|pulse|flyback|isolation|phase-?shifting|step-?(up|down)) transformers?\b"],
156 + related=["ml_theory", "computer_vision", "nlp", "ai_accelerators", "foundation_models", "llm"],
157 + parent_of=[], child_of=[],
158 + provenance=("T10320 and T11273 are architecture-centred OpenAlex topics; cs.NE is the arXiv home of neural and "
159 + "evolutionary computation. 'Transformer' alone is weak and electrical senses are vetoed when no "
160 + "learning term co-occurs."),
161 +),
162 +dict(
163 + id="ai_safety", number=38, name="AI Safety and Alignment", area="ai_ml",
164 + definition=("Making AI systems safe, aligned, robust, fair, interpretable and governable: alignment and preference "
165 + "learning framed as a safety problem, red-teaming and jailbreaks, adversarial robustness and backdoors, "
166 + "hallucination and truthfulness, bias and fairness, mechanistic interpretability, evaluation of dangerous "
167 + "capabilities, AI governance and regulation. Generic system security is cybersecurity; XAI as a statistical "
168 + "method only is ml_theory."),
169 + topics=[("T10883", 1.0, "Ethics and Social Impacts of AI: AI safety, alignment, fairness, governance"),
170 + ("T11689", 0.6, "adversarial robustness, backdoors, prompt injection"),
171 + ("T12026", 0.5, "explainable AI / mechanistic interpretability"),
172 + ("T13851", 0.4, "law, AI and intellectual property"),
173 + ("T12262", 0.3, "hate speech and cyberbullying detection (harm mitigation)"),
174 + ("T11147", 0.3, "misinformation and its impacts"),
175 + ("T10525", 0.2, "human-automation interaction and safety"),
176 + ("T14307", 0.1, "bioethics and human rights")],
177 + subfields=[],
178 + arxiv=[("cs.CY", 0.5), ("cs.AI", 0.2), ("cs.LG", 0.1), ("cs.CL", 0.1), ("cs.CR", 0.1)],
179 + streams=[("conf/fat", 1.0), ("conf/aies", 1.0), ("journals/aiethics", 1.0), ("conf/satml", 1.0),
180 + ("journals/ais", 0.5), ("journals/natmi", 0.2), ("conf/nips", 0.1), ("conf/iclr", 0.1), ("conf/icml", 0.1),
181 + ("journals/tai", 0.2), ("journals/dai", 0.1), ("conf/sp", 0.1), ("conf/ccs", 0.1)],
182 + keywords=[
183 + (r"(?i)\bAI (safety|alignment|governance|regulation|ethics|risks?|act|auditing|accountability|harms?|misuse|incidents?|oversight|assurance|red[- ]teaming)\b", 1.0, "both"),
184 + (r"(?i)\b(aligned|alignment of|align(ing)?|misaligned|misalignment of) (large )?(language models?|LLMs?|AI (systems?|models?)|foundation models?|agents?)\b|\bsuper-?alignment\b|\bvalue alignment\b|\bhuman values?\b", 1.0, "both"),
185 + (r"(?i)\b(jail-?break(s|ing)?|red[- ]teaming|prompt injection|backdoor (attacks?|defenses?)|data poisoning|model (stealing|extraction|inversion)|membership inference|adversarial (prompts?|suffixes?)|refusal|safety (training|fine-?tuning|guardrails?|alignment|benchmarks?|evaluations?)|guardrails?|harmful (content|outputs?|responses?|requests?)|unsafe (outputs?|behavio(u)?rs?))\b", 1.0, "both"),
186 + (r"(?i)\badversarial (examples?|attacks?|robustness|training|perturbations?|defenses?|vulnerabilit)\b|\bcertified robustness\b|\brobustness (certification|verification)\b", 0.8, "both"),
187 + (r"(?i)\b(algorithmic|AI|model|machine learning|ML|LLM|gender|racial|demographic|social) (fairness|bias(es)?|discrimination|transparency|accountability|audits?|auditing)\b|\bbias (mitigation|amplification|in (LLMs?|language models?|AI))\b|\bdebiasing\b", 1.0, "both"),
188 + (r"(?i)\b(responsible|trustworthy|ethical|safe|harmless|honest|helpful|human-?centered|human-?compatible|beneficial) (AI|artificial intelligence|machine learning|LLMs?|agents?)\b", 1.0, "both"),
189 + (r"(?i)\b(mechanistic interpretability|sparse auto-?encoders? (for|on) (LLM|language|features)|activation (steering|patching)|circuit discovery|probing classifiers?|representation engineering|superposition hypothesis|induction heads|feature visuali[sz]ation|concept (bottleneck|activation vectors?)|model editing|knowledge editing)\b", 1.0, "both"),
190 + (r"(?i)\b(hallucinations?|truthfulness|factuality|sycophancy|deceptive alignment|deception|reward hacking|specification gaming|goal misgeneralization|scalable oversight|weak-?to-?strong|constitutional AI|RLHF|preference (learning|optimi[sz]ation)|human feedback|over-?optimi[sz]ation|situational awareness|sandbagging|power-?seeking|instrumental convergence|corrigib)\b", 0.8, "both"),
191 + (r"(?i)\b(existential risks?|catastrophic risks?|x-?risk|frontier (AI|models?)|dangerous capabilit(y|ies)|misuse of AI|dual-?use|AGI (safety|risk|alignment)|superintelligen|loss of control|AI (doom|pause)|compute governance|responsible scaling|model (cards?|evaluations? for safety)|capability evaluations?|system cards?)\b", 1.0, "both"),
192 + (r"(?i)\b(toxic(ity)?|hate speech|harmful|offensive|abusive) (language|content|speech|text|generation|detection)\b|\bcontent moderation\b|\b(misinformation|disinformation|fake news) (detection|generation|and LLMs?)\b", 0.5, "both"),
193 + (r"(?i)\b(watermark(ing)?|detect(ing|ion) of|detecting) (LLM|AI|machine)-?generated (text|content|images?)\b|\bAI-?text detection\b", 0.8, "both"),
194 + (r"(?i)\b(machine unlearning|right to be forgotten|differential privacy for (LLM|language)|memori[sz]ation (in|of) (LLM|language models?)|training data extraction)\b", 0.5, "both"),
195 + (r"(?i)\b(uncertainty (quantification|estimation|calibration)|out-?of-?distribution detection|distribution shift|safe (reinforcement learning|RL|exploration)|runtime monitoring of (neural|ML)|verification of neural networks?)\b", 0.4, "both"),
196 + (r"(?i)\b(ethics|ethical|societal (impact|implications)|social (impact|implications)|regulation|policy|governance|accountability|transparency|explainability|interpretability|fairness|bias|safety|risk)\b", 0.2, "both"),
197 + ],
198 + exclusions=[r"(?i)\b(multiple sequence|read|genome|sequence|image|spinal|wheel|dental|teeth|occlusal|fib(er|re)|beam|optical|antenna|shaft|laser) alignment\b(?!.*(AI|LLM|language model|RLHF|human values))"],
199 + related=["llm", "ml_theory", "cybersecurity", "privacy_preserving_computation", "ai_agents", "hci", "reinforcement_learning"],
200 + parent_of=[], child_of=[],
201 + provenance=("T10883 (field 33, Social Sciences) explicitly lists 'AI safety', 'AI alignment', 'algorithmic fairness'; "
202 + "T11689 and T12026 host the technical robustness/interpretability halves. FAccT, AIES, SaTML and AI & Ethics "
203 + "are the dedicated venues; the generic ethics/governance words are kept far below threshold."),
204 +),
205 +]
added taxonomy/src/area_data_a.py +126 −0
@@ -0,0 +1,126 @@
1 +"""Area data_distributed, part A: categories 20-22 (databases, information retrieval, distributed computing)."""
2 +
3 +CATEGORIES = [
4 +dict(
5 + id="database_systems", number=20, name="Database Systems", area="data_distributed",
6 + definition=("Storage, querying and management of structured and semi-structured data: query processing and optimisation, "
7 + "transactions and concurrency control, indexing and storage engines, data integration, cleaning and provenance, "
8 + "big-data processing frameworks, data mining as a data-management problem, learned database components, "
9 + "vector and graph databases. Ranking of unstructured documents is information_retrieval."),
10 + topics=[("T10317", 1.0, "advanced database systems and queries"), ("T12761", 0.6, "data stream mining"),
11 + ("T11719", 0.6, "data quality and management"), ("T14280", 0.6, "big data technologies and applications"),
12 + ("T10538", 0.5, "data mining algorithms and applications"), ("T11181", 0.5, "advanced data storage technologies"),
13 + ("T12601", 0.5, "web applications and data management"), ("T11106", 0.5, "data management and algorithms"),
14 + ("T10772", 0.4, "distributed systems and fault tolerance (distributed databases, transactions)"),
15 + ("T13373", 0.4, "data mining and ML applications"), ("T11891", 0.4, "big data and business intelligence"),
16 + ("T10215", 0.4, "semantic web and ontologies (RDF, SPARQL)"), ("T12016", 0.4, "web data mining"),
17 + ("T14435", 0.4, "information retrieval and data mining"), ("T11937", 0.3, "research data management"),
18 + ("T13647", 0.3, "AI and big data applications"), ("T14347", 0.2, "big data and digital economy"),
19 + ("T10101", 0.2, "cloud computing (MapReduce, Hadoop)"), ("T10757", 0.2, "geographic information systems"),
20 + ("T13832", 0.2, "advanced decision-making techniques (DSS)"), ("T14484", 0.2, "technology and data analysis")],
21 + subfields=[(1710, 0.1)],
22 + arxiv=[("cs.DB", 1.0)],
23 + streams=[("conf/sigmod", 1.0), ("journals/pvldb", 1.0), ("journals/pacmmod", 1.0), ("conf/icde", 1.0), ("conf/edbt", 1.0),
24 + ("conf/pods", 1.0), ("journals/tods", 1.0), ("journals/vldb", 1.0), ("conf/dasfaa", 1.0), ("conf/vldb", 0.8),
25 + ("journals/tkde", 0.8), ("journals/tbd", 0.6), ("conf/apweb", 0.6), ("conf/icdm", 0.5), ("conf/pakdd", 0.5),
26 + ("journals/tkdd", 0.5), ("journals/kais", 0.5), ("conf/bigdataconf", 0.5), ("journals/jbd", 0.5), ("conf/gis", 0.5),
27 + ("conf/cikm", 0.4), ("conf/kdd", 0.4), ("journals/ijdsa", 0.4), ("journals/data", 0.4), ("conf/pkdd", 0.3),
28 + ("journals/www", 0.3), ("journals/tgis", 0.3), ("journals/gis", 0.3), ("journals/biodb", 0.3),
29 + ("conf/semweb", 0.3), ("conf/esws", 0.3), ("journals/dss", 0.2), ("journals/isf", 0.1), ("journals/tist", 0.2),
30 + ("journals/kbs", 0.1), ("conf/ksem", 0.2)],
31 + keywords=[
32 + (r"(?i)\bdatabases?\b|\bDBMSs?\b|\bRDBMSs?\b|\bdatabase (systems?|management|engines?|design|theory|queries|tuning|research)\b", 0.8, "both"),
33 + (r"(?i)\b(query (processing|optimi[sz]ation|execution|languages?|engines?|compilation|plans?|rewriting|answering over)|SQL\b|NoSQL|NewSQL|relational (model|algebra|databases?|queries|tables?)|transaction(al)? (processing|management|systems?)|ACID\b|OLTP|OLAP|HTAP|index(ing|es) (structures?|for|techniques)|B\+?-?trees?|learned (index(es)?|cardinality|query optimi[sz]ers?|database)|key-?value stores?|column(ar)? (stores?|storage|formats?)|data (warehous(e|es|ing)|lakes?|lakehouses?|integration|cleaning|quality|provenance|lineage|wrangling|catalogs?|cubes?|partitioning|versioning|discovery|marketplaces?)|schema (matching|evolution|mapping|inference)|entity (resolution|matching)|record linkage|join (algorithms?|processing|ordering|optimi[sz]ation)|cardinality estimation|selectivity estimation|data management|graph databases?|property graphs?|spatial (databases?|indexing|queries|joins?)|spatio-?temporal (databases?|queries|indexing)|time[- ]series databases?|vector (databases?|search engines?|indexes?)|approximate nearest neighbou?r search|similarity search|top-?k queries|skyline queries|query by example|data lakes?|metadata management|Datalog engines?)\b", 1.0, "both"),
34 + (r"(?i)\b(data mining|frequent (patterns?|itemsets?|subgraphs?)|association rules?|knowledge discovery in databases|KDD\b|sequential pattern mining|stream (processing|mining)|data streams?|sliding windows?|big data (analytics|processing|systems?|platforms?|frameworks?|management)|Hadoop|HDFS|Apache Spark|Spark SQL|MapReduce|Flink|Hive|Presto|Trino|Kafka|ETL\b|ELT\b|text-?to-?SQL|NL2SQL|natural language interfaces? to databases?|NLIDB|LLMs? for (data|databases?|SQL|tables?)|table (question answering|understanding|representation))\b", 0.7, "both"),
35 + (r"(?i)\b(PostgreSQL|Postgres|MySQL|SQLite|DuckDB|MongoDB|Cassandra|RocksDB|LevelDB|Neo4j|ClickHouse|Snowflake|BigQuery|Redshift|TiDB|CockroachDB|Spanner|DynamoDB|LSM-?trees?|storage engines?|buffer (pool|management)|concurrency control|MVCC|multi-?version|crash recovery|write-?ahead log(ging)?|WAL\b|checkpointing in databases|isolation levels?|serializab(le|ility)|snapshot isolation|two-?phase (locking|commit)|optimistic concurrency|deterministic databases?|in-?memory databases?|main-?memory databases?|disaggregated (storage|memory) for databases?|cloud(-?native)? databases?|database-?as-?a-?service|DBaaS|multi-?model databases?|time-?travel queries|temporal databases?)\b", 1.0, "both"),
36 + (r"(?i)\b(data|queries|query|tables?|records?|tuples?|schemas?|indexes|indices)\b", 0.2, "both"),
37 + ],
38 + exclusions=[r"(?is)^(?!.*(query processing|SQL|transaction|index(ing)?|storage engine|DBMS|data management|data integration|schema|NoSQL|query optimi)).*\b(PubMed|Embase|Scopus|Web of Science|Cochrane|MEDLINE|CINAHL|PsycINFO) (databases?|were searched|search)\b"],
39 + related=["information_retrieval", "distributed_computing", "cloud_computing", "algorithms", "ml_theory"],
40 + parent_of=[], child_of=[],
41 + provenance=("T10317 is the OpenAlex database topic; data-mining and big-data topics (1710, 1711, 18xx) shared at 0.4-0.6. "
42 + "SIGMOD/PVLDB/PACMMOD/ICDE/EDBT/PODS/TODS/VLDBJ pure; CIKM/KDD shared with IR and ML. Systematic-review "
43 + "mentions of bibliographic databases are vetoed when no DB-systems term co-occurs."),
44 +),
45 +dict(
46 + id="information_retrieval", number=21, name="Information Retrieval", area="data_distributed",
47 + definition=("Finding and ranking relevant information for a user need: search engines and ranking models, dense and "
48 + "sparse retrieval, recommender systems, query understanding, evaluation and test collections, web search and "
49 + "mining, scholarly and digital-library retrieval, retrieval components of RAG systems. Structured query "
50 + "processing is database_systems; text understanding tasks are nlp."),
51 + topics=[("T10286", 1.0, "information retrieval and search behavior"), ("T10203", 1.0, "recommender systems"),
52 + ("T13274", 0.8, "expert finding and Q&A systems"), ("T14435", 0.6, "information retrieval and data mining"),
53 + ("T13976", 0.5, "web visibility and informetrics"), ("T12016", 0.5, "web data mining and analysis"),
54 + ("T10627", 0.4, "advanced image and video retrieval"), ("T10824", 0.3, "image retrieval and classification"),
55 + ("T10028", 0.3, "topic modeling / RAG / sentence retrieval"), ("T10215", 0.3, "semantic web and ontologies"),
56 + ("T13673", 0.2, "library science and information"), ("T11937", 0.2, "research data management"),
57 + ("T11813", 0.2, "library collection development and digital resources"), ("T12863", 0.2, "web and library services"),
58 + ("T10664", 0.2, "sentiment analysis and opinion mining"), ("T11644", 0.2, "spam and phishing detection"),
59 + ("T14330", 0.2, "library science and information systems"), ("T13166", 0.2, "information science and libraries"),
60 + ("T13874", 0.2, "information architecture and usability"), ("T11122", 0.2, "online learning and analytics (recommendation)")],
61 + subfields=[(1710, 0.1)],
62 + arxiv=[("cs.IR", 1.0), ("cs.DL", 0.6), ("cs.SI", 0.3)],
63 + streams=[("conf/sigir", 1.0), ("conf/wsdm", 1.0), ("conf/ecir", 1.0), ("conf/recsys", 1.0), ("journals/tois", 1.0),
64 + ("journals/ipm", 0.8), ("conf/clef", 0.8), ("conf/jcdl", 0.8), ("conf/um", 0.8), ("conf/cikm", 0.6), ("conf/www", 0.6),
65 + ("journals/www", 0.6), ("journals/jasis", 0.6), ("conf/asist", 0.6), ("journals/scientometrics", 0.5),
66 + ("conf/semweb", 0.5), ("conf/esws", 0.5), ("conf/webi", 0.5), ("journals/kais", 0.3), ("conf/kdd", 0.3),
67 + ("conf/icwsm", 0.3), ("journals/tist", 0.3), ("journals/tkde", 0.2), ("journals/snam", 0.2), ("conf/asunam", 0.2),
68 + ("journals/tcss", 0.1), ("journals/ijinfoman", 0.2), ("journals/jkm", 0.1), ("journals/iam", 0.1)],
69 + keywords=[
70 + (r"(?i)\binformation retrieval\b|\bIR (systems?|models?|evaluation|research|community)\b|\bad[- ]hoc retrieval\b|\bretrieval (models?|systems?|effectiveness|augmented)\b", 1.0, "both"),
71 + (r"(?i)\b(search engines?|web search|enterprise search|product search|query (expansion|reformulation|understanding|suggestion|logs?|auto-?completion|intent)|ranking (models?|functions?|losses?|metrics)|learning[- ]to[- ]rank|LTR\b|dense (retrieval|passage retrieval|retrievers?)|sparse retrieval|lexical retrieval|hybrid retrieval|BM25|TF-?IDF|neural (retrieval|rankers?|IR|re-?ranking)|re-?rank(ing|ers?)|cross-?encoders?|bi-?encoders?|dual encoders?|ColBERT|late interaction|SPLADE|DPR\b|document (ranking|retrieval|expansion)|passage (ranking|retrieval)|relevance (judgments?|feedback|assessment|modeling)|click (models?|logs?|through)|conversational search|interactive (IR|retrieval)|session search|evaluation (campaigns?|measures?)|nDCG|MRR\b|mean reciprocal rank|MAP@|test collections?|TREC\b|MS ?MARCO|BEIR|pooling bias|query performance prediction|retrieval-?augmented|RAG (systems?|pipelines?)|retriever|generative (retrieval|recommendation))\b", 1.0, "both"),
72 + (r"(?i)\brecommend(er|ation) (systems?|models?|algorithms?|engines?|tasks?|lists?|quality)\b|\brecommenders?\b|\bcollaborative filtering\b|\b(sequential|session-?based|conversational|cross-?domain|cold-?start|news|music|point-?of-?interest|POI|job|group|bundle|LLM-?based|explainable|fair|multi-?stakeholder) recommendation\b|\bclick-?through rate\b|\bCTR prediction\b|\bimplicit feedback\b|\buser (modeling|preferences?|interests?) (for|in) recommendation\b|\bmatrix factori[sz]ation\b|\bitem (embeddings?|cold start)\b|\bMovieLens\b|\bNetflix Prize\b", 1.0, "both"),
73 + (r"(?i)\b(knowledge graphs?|entity linking|entity search|semantic web|linked (open )?data|SPARQL|RDF\b|ontolog(y|ies) (matching|alignment|engineering)|schema\.org|Wikidata|DBpedia|question answering over knowledge)\b", 0.5, "both"),
74 + (r"(?i)\b(digital librar(y|ies)|scholarly (search|recommendation|data|documents?|knowledge graphs?)|citation (analysis|networks?|recommendation|counts?|indexing)|bibliometric|scientometric|altmetric|h-?index|peer[- ]review (process|quality)|web (crawl(ing|ers?)|archiv(es?|ing)|mining)|social media (mining|analytics|retrieval)|hashtags?|microblog|user-?generated content|community question answering|CQA\b|expert finding|e-?commerce search|sponsored search|computational advertising|ad (auctions?|ranking))\b", 0.6, "both"),
75 + (r"(?i)\b(retrieval|search|ranking|recommendation|relevance|queries|query|users?' (information|search))\b", 0.3, "both"),
76 + ],
77 + exclusions=[],
78 + related=["nlp", "database_systems", "llm", "hci", "ml_theory"],
79 + parent_of=[], child_of=[],
80 + provenance=("T10286 (search behaviour) and T10203 (recommenders) are the OpenAlex IR hosts; library/information-science "
81 + "topics of 1710 included at 0.2 because they share venues (JASIST, ASIS&T). SIGIR/WSDM/ECIR/RecSys/TOIS pure."),
82 +),
83 +dict(
84 + id="distributed_computing", number=22, name="Distributed Computing", area="data_distributed",
85 + definition=("Computation across multiple processors or machines: distributed algorithms and consensus, fault tolerance "
86 + "and replication, parallel programming and HPC systems software, cluster and grid resource management, "
87 + "distributed storage and middleware, distributed and parallel ML training systems. Cloud service models are "
88 + "cloud_computing and edge placement is edge_computing (both children)."),
89 + topics=[("T10715", 1.0, "distributed and parallel computing systems (grid, cluster, scheduling)"),
90 + ("T10772", 1.0, "distributed systems and fault tolerance (consensus, replication, transactions)"),
91 + ("T10742", 0.8, "peer-to-peer network technologies"), ("T10054", 0.6, "parallel computing (HPC, MPI, GPU)"),
92 + ("T10829", 0.5, "interconnection networks and systems"), ("T11181", 0.5, "advanced data storage technologies"),
93 + ("T10101", 0.4, "cloud computing and resource management"), ("T12127", 0.4, "software system performance and reliability"),
94 + ("T10679", 0.3, "service-oriented architecture"), ("T10273", 0.3, "IoT and edge/fog computing"),
95 + ("T10249", 0.3, "distributed control multi-agent systems"), ("T12203", 0.3, "mobile agent-based network management"),
96 + ("T11986", 0.3, "scientific workflows"), ("T12879", 0.2, "distributed sensor networks and detection"),
97 + ("T12216", 0.2, "network time synchronization"), ("T11478", 0.2, "caching and content delivery")],
98 + subfields=[(1705, 0.1)],
99 + arxiv=[("cs.DC", 1.0), ("cs.PF", 0.5), ("cs.OS", 0.2), ("cs.NI", 0.1)],
100 + streams=[("journals/tpds", 1.0), ("journals/jpdc", 1.0), ("conf/ipps", 1.0), ("conf/podc", 1.0), ("conf/spaa", 1.0),
101 + ("conf/opodis", 1.0), ("conf/europar", 1.0), ("conf/ics", 1.0), ("conf/hpdc", 1.0), ("conf/icdcs", 1.0),
102 + ("conf/middleware", 1.0), ("conf/cluster", 1.0), ("conf/icpads", 1.0), ("conf/ica3pp", 1.0), ("conf/ispa", 1.0),
103 + ("conf/sc", 0.8), ("conf/ccgrid", 0.8), ("conf/hpcc", 0.8), ("conf/ppopp", 0.8), ("journals/concurrency", 0.8),
104 + ("journals/tjs", 0.8), ("journals/cluster", 0.8), ("journals/ijhpca", 0.8), ("journals/scpe", 0.8),
105 + ("journals/fgcs", 0.6), ("conf/eurosys", 0.6), ("journals/ppna", 0.6), ("conf/aina", 0.6), ("conf/nsdi", 0.5),
106 + ("conf/dsn", 0.5), ("conf/socc", 0.5), ("conf/cloud", 0.5), ("journals/tos", 0.5), ("conf/fast", 0.5),
107 + ("conf/osdi", 0.4), ("conf/sosp", 0.4), ("journals/tc", 0.4), ("journals/sp", 0.4), ("conf/mlsys", 0.4),
108 + ("conf/ucc", 0.4), ("journals/tcc", 0.3), ("journals/jsa", 0.3), ("journals/computing", 0.3),
109 + ("journals/tnsm", 0.2), ("conf/xsede", 0.4), ("conf/pact", 0.2), ("journals/taco", 0.1), ("journals/tnse", 0.2),
110 + ("journals/jnca", 0.2), ("conf/isca", 0.1), ("journals/ieicetd", 0.1)],
111 + keywords=[
112 + (r"(?i)\b(distributed|parallel|concurrent) (systems?|computing|algorithms?|programming|processing|storage|file systems?|transactions?|consensus|ledgers?|training|inference|machine learning|deep learning|graph processing|databases?|data processing|simulation|applications?|execution|runtime|scheduling|snapshots?|shared memory|memory|protocols?|optimi[sz]ation|SGD|stochastic)\b", 1.0, "both"),
113 + (r"(?i)\b(consensus (protocols?|algorithms?)|Paxos|Raft\b|Byzantine|fault[- ]toleran(t|ce)|replication|state machine replication|linearizab(le|ility)|serializab|eventual consistency|strong consistency|consistency models?|CRDTs?|conflict-?free replicated|vector clocks?|logical clocks?|leader election|gossip protocols?|epidemic protocols?|distributed (hash tables?|locks?|snapshots?|tracing|ledger|shared)|two-?phase commit|atomic (broadcast|commit)|failure detectors?|self-?stabili[sz]|population protocols?|LOCAL model|CONGEST|message complexity|round complexity|asynchronous (systems?|networks?|model))\b", 1.0, "both"),
114 + (r"(?i)\b(high[- ]performance computing|HPC\b|supercomput\w+|exascale|petascale|MPI\b|OpenMP|message passing|load balancing|job scheduling|workflow (scheduling|management|engines?)|resource (allocation|management|scheduling) (in|for|on|across) (clusters?|grids?|data ?cent(er|re)s?|HPC|supercomputers?)|scalab(le|ility) (to|on|across) \d[\d,]* (nodes|cores|GPUs|processors)|multi-?GPU|GPU clusters?|data[- ]parallel(ism)?|model[- ]parallel(ism)?|pipeline[- ]parallel(ism)?|tensor[- ]parallel(ism)?|expert parallelism|all-?reduce|all-?to-?all|parameter servers?|collective communication|NCCL|ZeRO\b|FSDP|DeepSpeed|Megatron|Horovod|gradient (synchroni[sz]ation|accumulation across)|stragglers?|checkpoint(ing)? (for|of|in) (training|HPC)|large-?scale training|training (at scale|on \d+ GPUs)|distributed (LLM|DNN) (training|inference|serving)|cluster schedul(ers?|ing)|Slurm|Kubernetes schedul|Borg|YARN|Mesos)\b", 1.0, "both"),
115 + (r"(?i)\b(middleware|publish[- ]subscribe|pub/sub|message (queues?|brokers?)|Kafka|RabbitMQ|actor model|service mesh|stream processing (engines?|systems?|frameworks?)|Apache (Spark|Flink|Storm|Beam|Hadoop)|MapReduce|Ray\b|Dask|dataflow (systems?|engines?)|serverless (workflows?|computing)|microservices? (architecture|performance|systems?)|RPC\b|remote procedure|gRPC|RDMA|interconnects?|InfiniBand|network-?on-?chip)\b", 0.7, "both"),
116 + (r"(?i)\b(peer-?to-?peer|P2P\b|overlay networks?|volunteer computing|grid computing|cluster computing|utility computing|desktop grid|BOINC|BitTorrent|DHTs?\b|Chord|Kademlia|gossip-?based)\b", 0.8, "both"),
117 + (r"(?i)\b(multi-?core|many-?core|multi-?threaded|multithreading|lock-?free|wait-?free|non-?blocking (data structures?|algorithms?)|transactional memory|synchroni[sz]ation primitives|mutex(es)?|spin-?locks?|barriers?|work[- ]stealing|parallel(i[sz]ation|ism|izing)|task (parallel|graphs?)|speedup|Amdahl|strong scaling|weak scaling|parallel efficiency|SIMD parallel|GPU (computing|programming|acceleration)|CUDA\b|OpenCL|heterogeneous computing)\b", 0.8, "both"),
118 + (r"(?i)\b(nodes?|clusters?|scalab(le|ility)|parallel|distributed|concurrency|replicas?|throughput|latency)\b", 0.2, "both"),
119 + ],
120 + exclusions=[r"(?is)^(?!.*(comput|system|algorithm|protocol|network|server|cluster|node|GPU|processor|storage|training|software)).*\b(distributed (generation|energy resources?|lag|fibre|fiber|acoustic|temperature|parameter|load|element|order|delay|control of (power|microgrid|vehicles?|robots?)|leadership|cognition|practice|manufacturing))\b"],
121 + related=["cloud_computing", "edge_computing", "computer_networks", "database_systems", "operating_systems", "scientific_computing", "blockchain", "federated_learning"],
122 + parent_of=["cloud_computing", "edge_computing"], child_of=[],
123 + provenance=("T10715 and T10772 (subfield 1705) are the OpenAlex hosts; T10054 (parallel/HPC) is shared with architecture and "
124 + "scientific computing. TPDS/JPDC/IPDPS/PODC/SPAA/Euro-Par/ICS/HPDC/ICDCS/Middleware are the pure venues; cs.DC suffices."),
125 +),
126 +]
added taxonomy/src/area_data_b.py +140 −0
@@ -0,0 +1,140 @@
1 +"""Area data_distributed, part B: categories 23-25 (cloud computing, edge computing, computer networks)."""
2 +
3 +CATEGORIES = [
4 +dict(
5 + id="cloud_computing", number=23, name="Cloud Computing", area="data_distributed",
6 + definition=("Computing delivered as elastic, multi-tenant, on-demand services: IaaS/PaaS/SaaS and serverless, "
7 + "virtualisation and containers, resource provisioning and auto-scaling, SLAs and pricing, cloud-native "
8 + "architectures, data-centre resource management, cloud security and migration. A child of distributed_computing; "
9 + "placement at the network edge is edge_computing."),
10 + topics=[("T10101", 1.0, "cloud computing and resource management (VMs, serverless, data centres)"),
11 + ("T14067", 1.0, "cloud computing and remote desktop technologies"), ("T11614", 0.6, "cloud data security solutions"),
12 + ("T12127", 0.6, "software system performance (microservices, cloud-native, AIOps)"),
13 + ("T10679", 0.6, "service-oriented architecture and web services"), ("T10273", 0.4, "IoT and edge/fog (cloud-edge)"),
14 + ("T14098", 0.4, "energy efficiency in computing (data centres)"), ("T10715", 0.3, "distributed and parallel computing (grid)"),
15 + ("T11181", 0.2, "data storage technologies"), ("T12799", 0.2, "mobile and web applications"),
16 + ("T12238", 0.2, "green IT (green data centres)")],
17 + subfields=[],
18 + arxiv=[("cs.DC", 0.6), ("cs.NI", 0.1), ("cs.SE", 0.1), ("cs.PF", 0.2)],
19 + streams=[("journals/tcc", 1.0), ("conf/cloud", 1.0), ("conf/socc", 1.0), ("conf/ucc", 1.0), ("conf/ic2e", 1.0),
20 + ("journals/tsc", 0.8), ("conf/icws", 0.8), ("conf/ccgrid", 0.6), ("journals/fgcs", 0.5), ("journals/cluster", 0.5),
21 + ("journals/tnsm", 0.4), ("conf/eurosys", 0.4), ("conf/nsdi", 0.4), ("conf/usenix", 0.4), ("conf/middleware", 0.4),
22 + ("journals/jnca", 0.4), ("conf/osdi", 0.3), ("conf/hpcc", 0.3), ("journals/tjs", 0.3), ("journals/concurrency", 0.3),
23 + ("journals/scpe", 0.3), ("journals/tsusc", 0.3), ("journals/suscom", 0.3), ("journals/tpds", 0.2), ("journals/jsa", 0.2),
24 + ("conf/icdcs", 0.2), ("conf/sc", 0.1), ("journals/computing", 0.2), ("conf/aina", 0.3)],
25 + keywords=[
26 + (r"(?i)\bcloud(-?based|-?native| computing| services?| providers?| infrastructures?| platforms?| data ?cent(er|re)s?| storage| resources?| workloads?| environments?| applications?| deployments?| migration| security| gaming| robotics| servers?| tenants?| customers?| costs?| pricing| offloading| continuum| vendors?| APIs?| regions?| instances?| orchestration| bursting| federation| brokers?)\b", 1.0, "both"),
27 + (r"(?i)\b(serverless|function-?as-?a-?service|FaaS\b|infrastructure-?as-?a-?service|IaaS\b|platform-?as-?a-?service|PaaS\b|software-?as-?a-?service|SaaS\b|multi-?cloud|hybrid cloud|cloud-?edge|edge-?cloud|virtual machines?|VMs?\b|virtuali[sz]ation|hypervisors?|containers?|containeri[sz]ed|Docker|Kubernetes|K8s|container orchestration|auto-?scaling|autoscal\w+|elastic(ity)? (provisioning|scaling|resources?)|service[- ]level (agreements?|objectives?)|SLAs?\b|SLOs?\b|multi-?tenan(t|cy)|resource provisioning|VM (placement|consolidation|migration|scheduling|allocation)|live migration|workload (prediction|consolidation|characteri[sz]ation|placement)|spot instances?|cloud (cost|FinOps)|pay-?as-?you-?go|data ?cent(er|re) (scheduling|networks?|resource|management|operators?|utili[sz]ation))\b", 1.0, "both"),
28 + (r"(?i)\b(AWS|Amazon (EC2|S3|Lambda|Web Services)|Microsoft Azure|Google Cloud|GCP\b|OpenStack|CloudSim|web services?|service-?oriented (architecture|computing)|service (composition|selection|discovery|mesh)|SOA\b|microservices?|cloud-?computing|Infrastructure as Code|Terraform|DevOps for cloud|cloud-?native (applications?|systems?))\b", 0.7, "both"),
29 + (r"(?i)\b(data ?cent(er|re)s?|hyperscal(e|ers?)|warehouse-?scale)\b", 0.6, "both"),
30 + (r"(?i)\b(virtuali[sz]ed|hosted|on-?demand|elastic|scalable services?|tenants?)\b", 0.2, "both"),
31 + ],
32 + exclusions=[r"(?is)^(?!.*(computing|comput|server|data ?cent|virtual|service|storage|infrastructure|platform|deployment|workload|software)).*\b(point clouds?|cloud (cover|droplets?|condensation|physics|microphysics|radiative|albedo|forcing|seeding|chamber|water|base|top|height|fraction|layers?|types?|formation)|molecular cloud|electron cloud|Oort cloud|dust cloud|cloud(s)? (and|in) (climate|atmosphere|precipitation))\b"],
33 + related=["distributed_computing", "edge_computing", "operating_systems", "computer_networks", "sustainable_computing", "cybersecurity"],
34 + parent_of=[], child_of=["distributed_computing"],
35 + provenance=("T10101 keywords list cloud computing, virtualization, serverless, SLAs, data centers. TCC/SoCC/UCC/IC2E pure; "
36 + "systems venues partial. Meteorological and point-cloud senses of 'cloud' are vetoed when no computing term co-occurs."),
37 +),
38 +dict(
39 + id="edge_computing", number=24, name="Edge Computing", area="data_distributed",
40 + definition=("Computation placed near data sources and users: edge/fog/MEC architectures, computation offloading, "
41 + "on-device and TinyML inference, edge-cloud collaboration and split computing, IoT platforms as computing "
42 + "substrates, latency-critical edge services, edge caching and orchestration. Radio access and protocols are "
43 + "computer_networks; model compression as such is deep_learning_architectures or ai_accelerators."),
44 + topics=[("T10273", 1.0, "IoT and edge/fog computing (offloading, MEC, edge AI)"), ("T13038", 0.6, "Internet of Things and AI"),
45 + ("T12079", 0.4, "IoT networks and protocols"), ("T13420", 0.3, "Arduino and IoT applications"),
46 + ("T13535", 0.3, "wireless sensor networks and IoT"), ("T13541", 0.3, "IoT-based control systems"),
47 + ("T13924", 0.3, "IoT and social network interactions"), ("T10080", 0.3, "energy-efficient wireless sensor networks"),
48 + ("T10101", 0.2, "cloud computing (cloud-edge)"), ("T12406", 0.2, "IoT and GPS vehicle safety"),
49 + ("T10904", 0.1, "embedded systems design"), ("T13918", 0.3, "advanced data and IoT technologies"),
50 + ("T11704", 0.2, "mobile crowdsensing")],
51 + subfields=[],
52 + arxiv=[("cs.DC", 0.4), ("cs.NI", 0.4), ("eess.SP", 0.1), ("cs.AR", 0.1)],
53 + streams=[("conf/sec", 1.0), ("conf/iotdi", 0.8), ("journals/iotj", 0.6), ("journals/iot", 0.6), ("conf/percom", 0.6),
54 + ("conf/wf-iot", 0.6), ("conf/ithings", 0.5), ("conf/uic", 0.4), ("conf/mobisys", 0.4), ("conf/sensys", 0.4),
55 + ("conf/ipsn", 0.4), ("conf/ewsn", 0.4), ("journals/ijdsn", 0.4), ("conf/dasc", 0.3), ("journals/tmc", 0.3),
56 + ("journals/tcc", 0.3), ("journals/fgcs", 0.3), ("journals/ett", 0.3), ("journals/jnca", 0.3), ("journals/tecs", 0.3),
57 + ("conf/aina", 0.3), ("conf/infocom", 0.2), ("journals/tnsm", 0.2), ("journals/tii", 0.2), ("conf/icdcs", 0.2),
58 + ("conf/hpcc", 0.2), ("conf/ispa", 0.2), ("journals/tvt", 0.1), ("journals/jsac", 0.1), ("journals/ton", 0.1),
59 + ("conf/ieeesensors", 0.1), ("journals/sensors", 0.1), ("journals/tgcn", 0.2), ("journals/tccn", 0.1),
60 + ("journals/wc", 0.2), ("journals/network", 0.2), ("conf/metroi", 0.3), ("journals/tnse", 0.1)],
61 + keywords=[
62 + (r"(?i)\b(edge|fog|mist|mobile[- ]edge|multi-?access edge|edge-?cloud|cloud-?edge|device-?edge-?cloud|edge-?to-?cloud|far[- ]edge|cloudlet|vehicular edge|satellite edge|serverless edge) (computing|intelligence|AI|inference|servers?|nodes?|devices?|deployment|caching|offloading|learning|networks?|resources?|orchestration|analytics|clusters?|platforms?|services?|applications?|continuum|infrastructures?|data ?cent(er|re)s?|scheduling|placement|training|LLMs?)\b|\bMEC\b|\bcloudlets?\b", 1.0, "both"),
63 + (r"(?i)\b(computation(al)?|task|workload|service|DNN|inference|model|function) offloading\b|\boffloading (decisions?|strategies|policies|schemes?)\b", 1.0, "both"),
64 + (r"(?i)\b(on-?device (AI|inference|learning|training|machine learning|deep learning|LLMs?|language models?|processing)|TinyML|microcontrollers?|MCUs?\b|resource-?constrained (devices?|IoT|hardware|environments?)|embedded (AI|inference|deep learning|machine learning|neural)|IoT (devices?|edge|gateways?|platforms?|architectures?|applications?|systems?|analytics|middleware)|Internet of (Things|Vehicles|Medical Things|Everything)|IIoT|cyber-?physical systems?|CPS\b|digital twins?|smart (cities|homes?|factor(y|ies)|grids?|manufacturing|agriculture|buildings?|campus)|Industry 4\.0|AIoT)\b", 0.7, "both"),
65 + (r"(?i)\b(latency-?sensitive|latency-?critical|ultra-?low latency|edge-?native|edge (AI|ML)|split (computing|inference|learning)|early[- ]exit|model partitioning|DNN partitioning|collaborative (inference|edge)|device-?cloud (collaboration|synergy)|edge-?assisted|edge-?enabled|near-?user|in-?network computing|in-?situ processing|data locality at the edge|mobile cloud computing|MCC\b|cloudlets?|micro ?data ?cent(er|re)s?|service placement|application placement|container orchestration at the edge|KubeEdge|OpenFaaS|edge orchestrat)\b", 0.8, "both"),
66 + (r"(?i)\b(fog|edge|IoT|offload\w*|devices?|sensors?|gateways?)\b", 0.3, "both"),
67 + ],
68 + exclusions=[r"(?is)^(?!.*(comput|offload|inference|IoT|device|server|latency|cloud|network|processing|deployment)).*\b(edge (detection|states?|modes?|effects?|dislocations?|plasma|localized|of (the|a) (wafer|chaos|cliff|forest|field|table|sample|crack|blade|flange))|band edge|leading edge|trailing edge|cutting-?edge|knife-?edge|edge (finishing|trimming|cracks?))\b"],
69 + related=["cloud_computing", "computer_networks", "federated_learning", "ai_accelerators", "sustainable_computing", "robotics", "distributed_computing"],
70 + parent_of=[], child_of=["distributed_computing"],
71 + provenance=("T10273 keywords list edge computing, fog computing, MEC, task offloading, edge AI; IoT topics (1705, 1710, 2208) "
72 + "included at 0.3-0.6 because IoT papers often treat edge placement. SEC and IoTDI are the dedicated streams; IoT-J "
73 + "and PerCom partial. Geometric/physics senses of 'edge' vetoed when no computing term co-occurs."),
74 +),
75 +dict(
76 + id="computer_networks", number=25, name="Computer Networks", area="data_distributed",
77 + definition=("Design, protocols, measurement and performance of communication networks: Internet architecture and "
78 + "transport, routing, wireless and mobile networks, 5G/6G and software-defined networking, network management, "
79 + "sensor and vehicular networks, content delivery and caching, network coding and information theory applied "
80 + "to networking. Physical-layer signal processing is included only when framed as networking."),
81 + topics=[("T10138", 1.0, "network traffic and congestion control"), ("T10246", 1.0, "mobile ad hoc networks"),
82 + ("T10575", 1.0, "wireless communication networks research"), ("T10714", 1.0, "software-defined networks and 5G"),
83 + ("T11158", 1.0, "wireless networks and protocols"), ("T11478", 1.0, "caching and content delivery"),
84 + ("T11896", 1.0, "opportunistic and delay-tolerant networks"), ("T13553", 1.0, "age of information optimization"),
85 + ("T10761", 1.0, "vehicular ad hoc networks"), ("T12326", 0.9, "network packet processing"),
86 + ("T10651", 0.9, "IPv6, mobility, handover, networks, security"), ("T10080", 0.8, "energy-efficient wireless sensor networks"),
87 + ("T10579", 0.8, "cognitive radio networks and spectrum sensing"), ("T10796", 0.8, "cooperative communication and network coding"),
88 + ("T12801", 0.8, "Bluetooth and wireless technologies"), ("T11409", 0.8, "advanced wireless network optimization"),
89 + ("T12079", 0.8, "IoT networks and protocols"), ("T10232", 0.7, "optical network technologies"),
90 + ("T10847", 0.7, "advanced optical network technologies"), ("T12203", 0.6, "mobile agent-based network management"),
91 + ("T11498", 0.6, "security in wireless sensor networks"), ("T10125", 0.6, "advanced wireless communication techniques"),
92 + ("T11932", 0.6, "wireless body area networks"), ("T13535", 0.6, "wireless sensor networks and IoT"),
93 + ("T10400", 0.5, "network security and intrusion detection"), ("T10829", 0.5, "interconnection networks"),
94 + ("T10742", 0.5, "peer-to-peer networks"), ("T12879", 0.5, "distributed sensor networks and detection"),
95 + ("T14353", 0.5, "wireless sensor networks for data analysis"), ("T12216", 0.5, "network time synchronization"),
96 + ("T11598", 0.5, "Internet traffic analysis"), ("T12042", 0.5, "satellite communication systems"),
97 + ("T11321", 0.5, "error-correcting code techniques"), ("T11458", 0.5, "advanced wireless communication technologies"),
98 + ("T11392", 0.5, "energy harvesting in wireless networks"), ("T12791", 0.5, "full-duplex wireless"),
99 + ("T10964", 0.5, "wireless communication security"), ("T10148", 0.4, "advanced MIMO systems"),
100 + ("T10851", 0.4, "optical wireless communication"), ("T12131", 0.4, "wireless signal modulation classification"),
101 + ("T11130", 0.3, "coding theory and cryptography"), ("T10711", 0.3, "target tracking and data fusion in sensor networks"),
102 + ("T10273", 0.3, "IoT and edge/fog computing"), ("T11797", 0.3, "graph theory and CDMA systems"),
103 + ("T10767", 0.3, "advanced photonic communication systems")],
104 + subfields=[(1705, 0.3)],
105 + arxiv=[("cs.NI", 1.0), ("cs.IT", 0.5), ("math.IT", 0.5), ("eess.SP", 0.3)],
106 + streams=[("journals/ton", 1.0), ("journals/cn", 1.0), ("journals/tmc", 1.0), ("conf/infocom", 1.0), ("conf/sigcomm", 1.0),
107 + ("conf/conext", 1.0), ("conf/imc", 1.0), ("conf/mobihoc", 1.0), ("conf/wowmom", 1.0), ("conf/lcn", 1.0),
108 + ("conf/icnp", 1.0), ("conf/mobicom", 1.0), ("conf/ewsn", 1.0), ("conf/mass", 1.0), ("conf/icc", 1.0),
109 + ("conf/globecom", 1.0), ("conf/wcnc", 1.0), ("conf/pimrc", 1.0), ("journals/tnsm", 1.0), ("journals/comcom", 1.0),
110 + ("journals/winet", 1.0), ("journals/adhoc", 1.0), ("journals/wpc", 1.0), ("journals/network", 1.0), ("journals/cm", 1.0),
111 + ("journals/comsur", 1.0), ("journals/wc", 1.0), ("journals/ijcomsys", 1.0), ("journals/tccn", 1.0), ("journals/monet", 1.0),
112 + ("journals/ejwcn", 1.0), ("journals/telsys", 1.0), ("journals/wicomm", 1.0), ("conf/noms", 1.0), ("conf/iwcmc", 1.0),
113 + ("conf/ccnc", 1.0), ("conf/comsnets", 1.0), ("conf/icoin", 1.0), ("conf/icccn", 1.0), ("conf/iscc", 1.0),
114 + ("conf/eucnc", 1.0), ("journals/rfc", 1.0), ("conf/nsdi", 0.8), ("conf/ipsn", 0.8), ("conf/vtc", 0.8),
115 + ("journals/jsac", 0.8), ("journals/tcom", 0.8), ("journals/twc", 0.8), ("journals/jnca", 0.8), ("journals/icl", 0.8),
116 + ("journals/wcl", 0.8), ("journals/iet-com", 0.8), ("journals/ieicet", 0.8), ("journals/ett", 0.8), ("journals/phycomm", 0.8),
117 + ("journals/ojcs", 0.8), ("conf/wcsp", 0.8), ("journals/ieicetb", 0.8), ("journals/jocn", 0.8), ("conf/IEEEants", 0.8),
118 + ("conf/isncc", 0.8), ("conf/iccchina", 0.8), ("conf/mobisys", 0.7), ("conf/sensys", 0.7), ("journals/fi", 0.7),
119 + ("journals/ijdsn", 0.7), ("conf/milcom", 0.7), ("journals/tvt", 0.6), ("journals/iotj", 0.6), ("journals/tnse", 0.6),
120 + ("journals/tgcn", 0.6), ("conf/icton", 0.6), ("journals/ppna", 0.6), ("journals/iot", 0.6), ("conf/icct", 0.6),
121 + ("conf/bmsb", 0.6), ("conf/spawc", 0.6), ("journals/tit", 0.5), ("conf/isit", 0.5), ("conf/ofc", 0.5), ("conf/ecoc", 0.5),
122 + ("journals/mis", 0.5), ("conf/icccnt", 0.5), ("conf/ictc", 0.5), ("journals/itiis", 0.5), ("journals/sensors", 0.3),
123 + ("conf/acssc", 0.3), ("journals/tsp", 0.2), ("conf/itw", 0.5), ("journals/entropy", 0.1), ("conf/icact", 0.5),
124 + ("journals/tetc", 0.1), ("conf/percom", 0.3), ("conf/wf-iot", 0.4), ("journals/js", 0.3), ("conf/ieeesensors", 0.2)],
125 + keywords=[
126 + (r"(?i)\b(computer|wireless|mobile|cellular|vehicular|sensor|ad[- ]hoc|mesh|satellite|optical|software-?defined|data ?cent(er|re)|core|access|backbone|overlay|delay-?tolerant|body area|underwater|LoRa(WAN)?|5G|6G|NB-?IoT|LTE|Wi-?Fi|IP|content-?centric|named data|information-?centric|non-?terrestrial|integrated|heterogeneous|programmable|intent-?based|self-?driving|zero-?touch|open radio access|radio access|time-?sensitive|deterministic|in-?vehicle|home|campus|enterprise|wide area|local area|personal area|low-?power wide|multi-?hop|peer-?to-?peer) networks?\b|\bnetworking\b", 1.0, "both"),
127 + (r"(?i)\b(TCP|UDP|QUIC|BGP|OSPF|IPv[46]|MPTCP|routing (protocols?|algorithms?|tables?)|congestion control|packet (loss|scheduling|processing|classification|forwarding|switching|headers?)|MAC (protocols?|layer)|medium access control|network (protocols?|topolog(y|ies)|slicing|function virtuali[sz]ation|measurement|traffic|management|coding|calculus|simulation|emulation|performance|planning|operators?|infrastructure|stack|layers?|devices?|interfaces?|cards?|services?|automation|digital twins?|telemetry|monitoring|tomography|utility|economics)|NFV\b|SDN\b|OpenFlow|P4\b|programmable (switches|data planes?)|data plane|control plane|traffic (engineering|classification|prediction|shaping|analysis)|quality of service|QoS\b|QoE\b|throughput|end-?to-?end (delay|latency)|bandwidth allocation|spectrum (sharing|allocation|sensing|management)|channel (allocation|assignment|access)|beamforming|MIMO|massive MIMO|OFDM(A)?|NOMA|RIS-?(aided|assisted)|reconfigurable intelligent surfaces?|mmWave|millimeter[- ]wave|THz communications?|channel estimation|resource allocation (in|for) (wireless|cellular|5G|6G|networks?)|handover|handoff|base stations?|small cells|user equipment|O-?RAN|Open RAN|RAN\b|Wi-?Fi|802\.11|802\.15|Bluetooth|BLE\b|Zigbee|LTE|5G NR|NR\b|age of information|AoI\b|URLLC|eMBB|mMTC|network (densification|coverage)|cell-?free|D2D\b|device-?to-?device|V2X|V2V|vehicular communications?|CDN\b|content delivery|edge caching|multicast|anycast|DNS\b|CDNs?|middleboxes?|NAT\b|firewalls? (rules|placement)|DDoS mitigation at|interdomain|AS-?level|Internet (measurement|topology|routing|architecture|protocols?|traffic|outages?|censorship|of Things)|RTT\b|jitter|bufferbloat|active queue management|flow (scheduling|completion time|tables?)|load balancing in (data ?cent|networks?)|RDMA networks?|InfiniBand|optical (networks?|switching|transport|access)|PON\b|passive optical|wavelength (division|assignment)|WDM|free-?space optical|visible light communication|VLC\b|LiFi|underwater (acoustic|communication)|satellite (networks?|communications?|constellations?)|LEO (satellite|constellation)|UAV (networks?|communications?|relays?)|network (security|intrusion) in|wireless (security|jamming|eavesdropping)|physical layer security|cross-?layer|protocol (design|stack|verification))\b", 0.9, "both"),
128 + (r"(?i)\b(information theory|channel capacity|coding theorem|error[- ]correcting codes?|channel codes?|LDPC|polar codes?|turbo codes?|rate-?distortion|source coding|network coding|index coding|coded caching|compressed sensing for|Shannon|mutual information bounds|outage probability|ergodic capacity|secrecy capacity|wiretap)\b", 0.6, "both"),
129 + (r"(?i)\b(intrusion detection|DDoS|network security|anomaly detection in network|botnets?|traffic anomaly)\b", 0.4, "both"),
130 + (r"(?i)(?<!neural )(?<!Neural )(?<!social )(?<!Social )(?<!citation )(?<!graph neural )(?<!Bayesian )(?<!bayesian )(?<!regulatory )(?<!gene )(?<!road )(?<!power )(?<!supply )(?<!transport )(?<!transportation )(?<!water )(?<!brain )\b(networks?|Internet|wireless|protocols?|routing|packets?|latency|bandwidth|antennas?|transceivers?|links?|channels?|relays?)\b", 0.2, "both"),
131 + ],
132 + exclusions=[],
133 + related=["distributed_computing", "edge_computing", "cybersecurity", "cloud_computing", "applied_cryptography"],
134 + parent_of=[], child_of=[],
135 + provenance=("43 topics of OpenAlex subfield 1705 reviewed (networking ones at 0.5-1.0, non-networking ones such as T13745 "
136 + "media studies or T11187 nonlinear dynamics left unmapped) plus Engineering 2208 wireless/optical topics at "
137 + "0.3-0.8. ToN/CN/TMC/INFOCOM/SIGCOMM/CoNEXT/IMC/MobiHoc pure; ICC/GLOBECOM/WCNC/PIMRC counted as networking "
138 + "venues although they also carry physical-layer work."),
139 +),
140 +]
added taxonomy/src/area_hardware_a.py +124 −0
@@ -0,0 +1,124 @@
1 +"""Area hardware_emerging, part A: categories 29-31 (computer architecture, AI accelerators, operating systems)."""
2 +
3 +CATEGORIES = [
4 +dict(
5 + id="computer_architecture", number=29, name="Computer Architecture", area="hardware_emerging",
6 + definition=("Organisation and design of processors, memory systems and digital hardware: microarchitecture, caches and "
7 + "memory hierarchies, interconnects and networks-on-chip, FPGAs and reconfigurable computing, embedded and "
8 + "real-time systems, hardware/software co-design, electronic design automation, VLSI and low-power design, "
9 + "processing-in-memory. Accelerators specialised for ML are ai_accelerators (a child); kernels are operating_systems."),
10 + topics=[("T10904", 1.0, "embedded systems design: FPGA, HLS, SoC, co-design"), ("T11522", 1.0, "VLSI and FPGA design techniques"),
11 + ("T12941", 1.0, "embedded systems and FPGA design"), ("T13292", 1.0, "embedded systems and FPGA applications"),
12 + ("T10054", 0.8, "parallel computing and optimization (multi-core, GPU, processor architecture)"),
13 + ("T10363", 0.8, "low-power high-performance VLSI design"), ("T11527", 0.8, "3D IC and TSV technologies"),
14 + ("T11032", 0.6, "VLSI and analog circuit testing"), ("T10829", 0.6, "interconnection networks and systems"),
15 + ("T10502", 0.6, "advanced memory and neural computing (RRAM, in-memory computing)"), ("T13182", 0.6, "quantum-dot cellular automata"),
16 + ("T10933", 0.5, "real-time systems scheduling"), ("T12122", 0.5, "PUFs and hardware security"),
17 + ("T11181", 0.5, "advanced data storage technologies (SSD, NVM)"), ("T14098", 0.4, "energy efficiency in computing"),
18 + ("T10558", 0.4, "semiconductor devices and circuit design"), ("T14117", 0.4, "IC and semiconductor failure analysis"),
19 + ("T12326", 0.3, "network packet processing (NICs, hardware)"), ("T10323", 0.3, "analog and mixed-signal circuit design"),
20 + ("T12611", 0.3, "reservoir and optical computing hardware"), ("T10187", 0.1, "RF integrated circuit design")],
21 + subfields=[(1708, 0.6)],
22 + arxiv=[("cs.AR", 1.0), ("cs.ET", 0.3), ("cs.PF", 0.3), ("cs.DC", 0.1), ("cs.OS", 0.1)],
23 + streams=[("conf/isca", 1.0), ("conf/micro", 1.0), ("conf/hpca", 1.0), ("conf/asplos", 1.0), ("journals/taco", 1.0),
24 + ("conf/hotchips", 1.0), ("journals/tc", 0.8), ("journals/tcad", 0.8), ("journals/tvlsi", 0.8), ("conf/dac", 0.8),
25 + ("conf/date", 0.8), ("conf/iccad", 0.8), ("conf/iccad2", 0.8), ("conf/aspdac", 0.8), ("journals/jsa", 0.8),
26 + ("journals/mam", 0.8), ("journals/tecs", 0.8), ("conf/fccm", 0.8), ("conf/fpga", 0.8), ("conf/fpl", 0.8),
27 + ("conf/fpt", 0.8), ("journals/trets", 0.8), ("conf/islped", 0.8), ("conf/glvlsi", 0.8), ("conf/rtss", 0.7),
28 + ("conf/rtas", 0.7), ("journals/integration", 0.7), ("conf/isocc", 0.6), ("journals/jssc", 0.5), ("conf/isscc", 0.5),
29 + ("journals/jcsc", 0.5), ("journals/mj", 0.4), ("conf/cicc", 0.4), ("journals/tetc", 0.4), ("conf/mlsys", 0.4),
30 + ("conf/iscas", 0.3), ("journals/tcasI", 0.3), ("journals/tcasII", 0.3), ("journals/tcas", 0.3), ("conf/mwscas", 0.3),
31 + ("conf/newcas", 0.3), ("conf/irps", 0.3), ("journals/ieiceee", 0.3), ("conf/apccas", 0.3), ("conf/icecsys", 0.3),
32 + ("journals/tr", 0.2), ("journals/tbcas", 0.2), ("conf/sc", 0.2), ("conf/ipps", 0.2), ("journals/tpds", 0.2),
33 + ("conf/ppopp", 0.2), ("journals/ieicetd", 0.2), ("journals/ieiceta", 0.2), ("journals/tsusc", 0.1), ("conf/dsn", 0.2),
34 + ("journals/tetci", 0.1), ("conf/biocas", 0.2), ("journals/cee", 0.2), ("journals/ijcta", 0.2)],
35 + keywords=[
36 + (r"(?i)\b(computer|processor|micro-?|system|memory|cache|chip|hardware|GPU|CPU|accelerator|multi-?core|many-?core|heterogeneous|domain-?specific|RISC-?V|dataflow|von Neumann|non-?von-?Neumann|SIMD|vector|VLIW|superscalar|in-?order|out-?of-?order|chiplet|reconfigurable|FPGA|SoC|NoC|storage|SSD|DRAM|NVM|server|cluster|interconnect|network-?on-?chip|tiled|spatial|CGRA|TPU|NPU|neuromorphic|quantum computer|instruction[- ]set|ISA|microprocessor|embedded|real-?time|automotive|avionics|safety-?critical) architectures?\b|\bmicro-?architectur\w*\b|\barchitectural (support|simulation|exploration|design|vulnerabilit|side)\b", 1.0, "both"),
37 + (r"(?i)\b(branch predict(ion|ors?)|out-?of-?order (execution|cores?)|speculative execution|instruction[- ]level parallelism|ILP\b|SIMD\b|vector (processors?|extensions?|units?|ISAs?)|AVX-?\d*|SVE\b|pipelin(ing|ed) (processors?|datapaths?)|cache (hierarch(y|ies)|coherence|replacement|partitioning|misses?|side[- ]channels?|compression|prefetch)|prefetch(ing|ers?)|memory (hierarch(y|ies)|controllers?|bandwidth|subsystems?|consistency|wall|disaggregation|expansion|pooling|compression|encryption hardware|scheduling)|DRAM|HBM\d?\b|high-?bandwidth memory|NVM\b|non-?volatile memor(y|ies)|persistent memory|Optane|processing-?in-?memory|PIM\b|near-?(data|memory) (processing|computing)|in-?memory computing|compute-?in-?memory|CIM\b|network-?on-?chip|NoCs?\b|on-?chip (networks?|interconnects?|memory)|interconnects?|chiplets?|3D (ICs?|stacking|integration)|2\.5D|TSVs?\b|FPGAs?\b|field-?programmable gate|reconfigurable (computing|hardware|architectures?|logic)|CGRAs?\b|coarse-?grained reconfigurable|high-?level synthesis|HLS\b|hardware/software co-?design|hardware-?software co-?design|HW/SW|RTL\b|register-?transfer level|Verilog|SystemVerilog|VHDL|SystemC|Chisel|Bluespec|logic synthesis|physical design|placement and routing|place[- ]and[- ]route|floorplan|timing (analysis|closure)|clock (tree|domain|gating)|design space exploration|DSE\b|system-?on-?chip|SoCs?\b|MPSoCs?|ASICs?\b|VLSI|CMOS (design|circuits?|scaling)|low-?power (design|architectures?|processors?)|DVFS|dynamic voltage|power gating|thermal (management|throttling)|dark silicon|RISC-?V|x86|ARM (Cortex|processors?|cores?|architecture)|AArch64|ISA (extensions?|design)|instruction set|custom instructions?|gem5|SimpleScalar|cycle-?accurate|performance (counters?|modeling|evaluation|analysis) of (processors?|architectures?|hardware|GPUs?)|roofline|memory-?bound|compute-?bound|TLB\b|virtual memory hardware|page walks?|hardware (prefetch|transactional|accelerat|support for|implementation|design|description|security|trojans?|verification|emulation|simulation)|GPU (architecture|microarchitecture|warps?|SMs?|streaming multiprocessors?)|warp scheduling|tensor cores?|SRAM|eDRAM|scratchpad|systolic|dataflow (architectures?|accelerators?|execution)|spatial (architectures?|accelerators?)|wafer-?scale|photonic interconnects?|optical interconnects?|silicon photonics for computing|superconducting (logic|electronics|computing)|cryogenic (CMOS|computing)|stochastic computing|approximate (computing|arithmetic|adders?|multipliers?)|arithmetic (units?|circuits?)|floating-?point units?|FPUs?\b|multipliers?|adders?|ALUs?\b|bit-?serial|number (formats?|representations?) for hardware|posit arithmetic|bfloat16|logic (gates?|circuits?) design|fault-?tolerant (hardware|architectures?)|soft errors?|radiation-?hardened|ECC memory|chip (design|multiprocessors?|area|yield)|die (area|size)|tape-?out|foundry|EDA\b|electronic design automation|hardware (fuzzing|verification|testing)|design for test|DFT\b|scan chains?|built-?in self-?test|BIST\b|ATPG|hardware emulation|prototyping on FPGA)\b", 1.0, "both"),
38 + (r"(?i)\b(embedded systems?|real-?time (systems?|scheduling|operating|embedded|tasks?|guarantees)|worst-?case execution time|WCET|schedulability|cyber-?physical|automotive (software|systems|ECUs?|electronics)|avionics|mixed-?criticality|time-?triggered|AUTOSAR|microcontrollers?|MCUs?\b|SoC (design|integration)|firmware|bare-?metal|energy harvesting (devices|systems|computing)|intermittent computing|wearable (electronics|hardware)|sensor nodes? (hardware|design))\b", 0.8, "both"),
39 + (r"(?i)\b(hardware|chips?|processors?|CPUs?|GPUs?|silicon|circuits?|transistors?|semiconductors?|cores?|caches?|memory|latency|throughput|energy efficiency|power consumption|area overhead)\b", 0.3, "both"),
40 + ],
41 + exclusions=[r"(?is)^(?!.*(processor|chip|hardware|FPGA|VLSI|circuit|memory|cache|silicon|microarchitect|embedded|SoC|ASIC|CMOS|GPU|CPU|RISC|instruction)).*\b(software|enterprise|information|network|service-?oriented|microservices?|cloud|data|security|landscape|naval|urban|building|brand|organi[sz]ational|business|blockchain|neural network|transformer|model|protocol|Internet|web|application|system|solution|reference|deep learning|learning|algorithm|choice|financial|market) architectures?\b"],
42 + related=["ai_accelerators", "operating_systems", "compilers", "sustainable_computing", "quantum_computing", "distributed_computing", "cybersecurity"],
43 + parent_of=["ai_accelerators"], child_of=[],
44 + provenance=("OpenAlex 1708 (Hardware and Architecture) has 8 topics, all reviewed; architecture work also lives in Engineering "
45 + "2207/2208 (FPGA, VLSI, 3D IC) and T10054. ISCA/MICRO/HPCA/ASPLOS/TACO/Hot Chips pure; EDA venues 0.8; circuits "
46 + "venues 0.3-0.5. Non-hardware senses of 'architecture' are vetoed when no hardware term co-occurs."),
47 +),
48 +dict(
49 + id="ai_accelerators", number=30, name="Accelerators and AI Hardware", area="hardware_emerging",
50 + definition=("Hardware and hardware-aware software for efficient machine-learning computation: GPU/TPU/NPU design and "
51 + "programming, ASIC and FPGA DNN accelerators, neuromorphic and spiking hardware, in-memory and analog/photonic "
52 + "computing for AI, quantisation and sparsity exploited by hardware, efficient LLM inference and serving systems, "
53 + "ML compilers' hardware mapping, benchmarks such as MLPerf. A child of computer_architecture."),
54 + topics=[("T10502", 1.0, "advanced memory and neural computing: memristors, neuromorphic hardware, in-memory computing"),
55 + ("T12611", 0.8, "neural networks and reservoir computing: optical/photonic neural networks"),
56 + ("T10904", 0.5, "embedded systems design: hardware accelerators, HLS"), ("T10054", 0.3, "parallel computing: GPU computing, CUDA"),
57 + ("T10036", 0.3, "advanced neural network applications: model compression, lightweight networks"),
58 + ("T11522", 0.3, "VLSI and FPGA design"), ("T12941", 0.3, "embedded systems and FPGA design"),
59 + ("T13292", 0.3, "embedded systems and FPGA applications"), ("T10363", 0.3, "low-power VLSI design"),
60 + ("T13182", 0.2, "quantum-dot cellular automata"), ("T10320", 0.2, "neural networks (hardware implementation)")],
61 + subfields=[],
62 + arxiv=[("cs.AR", 0.6), ("cs.ET", 0.5), ("cs.NE", 0.2), ("cs.LG", 0.1), ("cs.DC", 0.1)],
63 + streams=[("conf/mlsys", 1.0), ("conf/hotchips", 0.6), ("conf/isca", 0.5), ("conf/micro", 0.5), ("conf/hpca", 0.5),
64 + ("conf/fccm", 0.5), ("conf/fpga", 0.5), ("conf/asplos", 0.4), ("conf/dac", 0.4), ("conf/iccad", 0.4), ("conf/iccad2", 0.4),
65 + ("conf/fpl", 0.4), ("journals/taco", 0.4), ("conf/isscc", 0.4), ("conf/islped", 0.4), ("journals/trets", 0.4),
66 + ("conf/date", 0.3), ("journals/tcad", 0.3), ("journals/tvlsi", 0.3), ("journals/tc", 0.3), ("journals/tecs", 0.3),
67 + ("journals/jsa", 0.3), ("journals/jssc", 0.3), ("conf/aspdac", 0.3), ("journals/mam", 0.3), ("journals/tetc", 0.3),
68 + ("conf/glvlsi", 0.3), ("conf/biocas", 0.3), ("conf/sc", 0.3), ("journals/tetci", 0.2), ("journals/tcasI", 0.2),
69 + ("journals/tbcas", 0.2), ("conf/iscas", 0.2), ("conf/ipps", 0.2), ("conf/ppopp", 0.2), ("journals/tpds", 0.2),
70 + ("journals/aisy", 0.1), ("journals/natmi", 0.1), ("journals/tnn", 0.1), ("journals/jpdc", 0.1), ("journals/ieiceee", 0.1),
71 + ("conf/isocc", 0.3), ("journals/integration", 0.2), ("journals/ncs", 0.1)],
72 + keywords=[
73 + (r"(?i)\b(AI|DNN|CNN|GNN|RNN|LSTM|neural[- ]network|deep[- ]learning|machine[- ]learning|ML|DL|inference|training|transformer|LLM|attention|tensor|matrix|GEMM|sparse|SNN|spiking|neuromorphic|photonic|optical|analog|in-?memory|PIM|RRAM|memristor|FPGA|ASIC|hardware|domain-?specific|systolic|dataflow|edge|energy-?efficient|low-?power|reconfigurable|custom|dedicated|specialized|GPU|convolution|graph processing|recommendation|embedding|vision|NLP|diffusion|generative) accelerat(or|ors|ion)\b|\baccelerator (design|architectures?|chips?)\b", 1.0, "both"),
74 + (r"(?i)\b(TPUs?\b|tensor processing units?|NPUs?\b|neural processing units?|IPUs?\b|systolic arrays?|tensor cores?|GPU (kernels?|programming|utili[sz]ation|clusters? for (training|inference)|scheduling for (DNN|ML|inference)|memory for (training|LLM))|CUDA (kernels?|cores?|programming|implementation|graphs?)|H100|H200|B200|A100|V100|GH200|GB200|NVIDIA (GPUs?|Hopper|Ampere|Blackwell|Volta|Jetson)|AMD (MI\d+|Instinct|ROCm)|Cerebras|Groq|Graphcore|Gaudi|Trainium|Inferentia|Apple Neural Engine|Jetson (Nano|Orin|Xavier)|Coral TPU|Edge TPU|Movidius|Loihi|TrueNorth|SpiNNaker|BrainScaleS|Tianjic|Eyeriss|DianNao|Cambricon|Hailo|Mythic|Lightmatter|Tenstorrent|SambaNova|Ascend 910|Kunlun|Hanguang)\b", 1.0, "both"),
75 + (r"(?i)\b(neuromorphic (computing|hardware|chips?|processors?|systems?|engineering|devices?|architectures?)|spiking neural networks? (hardware|accelerat|on-?chip|processors?|implementation)|SNN (hardware|accelerat|processors?)|memristive|memristors?|RRAM|ReRAM|PCM-?based (computing|synapses?)|FeFET|MRAM-?based computing|crossbar (arrays?|architectures?)|compute-?in-?memory|in-?memory computing|processing-?in-?memory|analog (computing|in-?memory|neural network hardware|AI chips?)|mixed-?signal (neural|AI)|photonic (computing|neural networks?|accelerat|processors?|tensor)|optical (neural networks?|computing|accelerat)|diffractive (neural|optical)|reservoir computing hardware|stochastic computing for neural|approximate computing for (neural|DNN)|quanti[sz]ation-?aware|mixed-?precision (training|inference|arithmetic)|FP8|FP4|INT8|INT4|W4A16|W8A8|4-?bit (quanti[sz]ation|weights)|bit-?serial|binary neural networks?|BNNs?\b|XNOR-?net|ternary (weights|networks)|sparsity (exploitation|acceleration|support)|structured sparsity|2:4 sparsity|pruning for hardware|hardware-?aware (NAS|neural architecture|pruning|quanti[sz]ation|training|design)|hardware-?efficient|model compression for (deployment|edge|hardware)|efficient (inference|transformers?|attention|LLM (inference|serving|deployment))|LLM (serving|inference) (systems?|engines?|optimi[sz]ation|efficiency|acceleration|on (edge|mobile|CPU|GPU))|inference (serving|engines?|latency|throughput|acceleration|optimi[sz]ation)|serving systems?|KV[- ]cache (compression|quanti[sz]ation|management|eviction)|FlashAttention|PagedAttention|vLLM|TensorRT-?LLM|SGLang|TGI\b|llama\.cpp|speculative decoding|continuous batching|weight-?only quanti[sz]ation|GPTQ|AWQ|SmoothQuant|LLM\.int8|QLoRA|kernel fusion|operator fusion|roofline|MLPerf|DNN (inference|training) (on|for) (FPGA|edge|mobile|GPU|ASIC)|FPGA-?based (CNN|DNN|neural|accelerat|inference)|ASIC (for|design for) (neural|DNN|CNN|AI)|DNN (accelerat|hardware|dataflow|mapping)|accelerator (mapping|scheduling|compilers?)|dataflow (mapping|optimi[sz]ation) for (DNN|CNN)|loop (tiling|unrolling) for (CNN|DNN|accelerat)|weight stationary|output stationary|on-?chip (buffers?|memory) for (DNN|CNN)|AI (chips?|hardware|silicon|processors?)|deep learning (hardware|chips?|processors?|accelerat)|machine learning (hardware|chips?|accelerat)|training (hardware|chips?|accelerat)|wafer-?scale (engine|AI)|AI (super)?computers?|GPU (clusters?|pods?) for (training|LLM)|multi-?GPU (training|inference)|tensor (parallel|slicing) on (GPUs?|accelerators?)|NVLink|NVSwitch|interconnects? for (AI|GPU|accelerators?)|HBM for (AI|accelerators?)|memory wall (in|for) (AI|DNN|LLM)|energy (per|efficiency of) inference|TOPS/W|TOPS\b|GOPS/W|pJ/MAC|MAC (units?|arrays?|utili[sz]ation)|multiply-?accumulate)\b", 1.0, "both"),
76 + (r"(?i)\b(energy efficiency|throughput|latency|FLOPS|utili[sz]ation|speedup) (of|for|on|over) (the )?(accelerators?|hardware|FPGAs?|GPUs?|chips?|inference|TPUs?|NPUs?|edge devices?)\b", 0.8, "both"),
77 + (r"(?i)\b(hardware|FPGA|ASIC|GPU|chip|silicon|accelerat\w+|on-?chip|CMOS|energy-?efficient|low-?power|edge devices?)\b", 0.3, "both"),
78 + ],
79 + exclusions=[r"(?is)^(?!.*(neural|DNN|CNN|deep learning|machine learning|AI\b|inference|LLM|transformer|training|GPU|FPGA|ASIC|hardware|chip|neuromorphic|memristor|in-?memory|photonic)).*\b(particle accelerators?|linear accelerators?|LINAC|accelerator (physics|beam|cavity|magnets?|pedal)|accelerated (aging|weathering|corrosion|degradation|testing|approval|rehabilitation|healing)|hardware (store|retail))\b"],
80 + related=["computer_architecture", "deep_learning_architectures", "compilers", "edge_computing", "sustainable_computing", "distributed_computing", "llm"],
81 + parent_of=[], child_of=["computer_architecture"],
82 + provenance=("No OpenAlex topic isolates ML accelerators; T10502 (memristors/neuromorphic/in-memory, Engineering 2208) and "
83 + "T12611 (photonic NNs) are the closest; GPU/FPGA accelerator work sits in T10904/T10054/T11522. MLSys is the only "
84 + "near-dedicated stream; architecture venues at 0.4-0.5. Keyword patterns on 'X accelerator', chip names and "
85 + "efficient-inference systems carry the category; particle-accelerator senses are vetoed."),
86 +),
87 +dict(
88 + id="operating_systems", number=31, name="Operating Systems", area="hardware_emerging",
89 + definition=("System software that manages hardware and provides execution abstractions: kernels and drivers, "
90 + "scheduling, memory management, file and storage systems, virtualisation and containers, OS security and "
91 + "isolation, I/O and networking stacks, real-time and embedded OS, systems support for new hardware "
92 + "(persistent memory, CXL, SmartNICs). Distributed coordination is distributed_computing; hardware design is "
93 + "computer_architecture."),
94 + topics=[("T11181", 0.8, "advanced data storage technologies: file systems, SSDs, I/O"), ("T10933", 0.8, "real-time systems scheduling"),
95 + ("T11424", 0.6, "security and verification in computing: OS security, hypervisors, TEEs"),
96 + ("T12326", 0.5, "network packet processing (kernel bypass)"), ("T14067", 0.4, "virtual desktops, remote desktop"),
97 + ("T10101", 0.3, "cloud computing: virtualization, VMs"), ("T12127", 0.3, "system performance and reliability, logs"),
98 + ("T11675", 0.3, "open source software (Linux)"), ("T10904", 0.2, "embedded systems design"),
99 + ("T10772", 0.2, "distributed systems and fault tolerance"), ("T12799", 0.2, "mobile applications (Android)"),
100 + ("T10054", 0.2, "parallel computing (multi-core runtimes)")],
101 + subfields=[],
102 + arxiv=[("cs.OS", 1.0), ("cs.DC", 0.2), ("cs.PF", 0.2), ("cs.AR", 0.1), ("cs.CR", 0.05)],
103 + streams=[("conf/osdi", 1.0), ("conf/sosp", 1.0), ("conf/eurosys", 1.0), ("conf/usenix", 1.0), ("conf/fast", 1.0),
104 + ("journals/tos", 1.0), ("conf/asplos", 0.6), ("conf/rtss", 0.6), ("conf/rtas", 0.6), ("conf/mobisys", 0.4),
105 + ("conf/middleware", 0.3), ("journals/tecs", 0.3), ("journals/jsa", 0.3), ("conf/nsdi", 0.3), ("conf/dsn", 0.3),
106 + ("journals/spe", 0.3), ("journals/mam", 0.3), ("conf/socc", 0.3), ("conf/cloud", 0.3), ("conf/hpdc", 0.2),
107 + ("journals/tc", 0.2), ("journals/tpds", 0.2), ("conf/mlsys", 0.2), ("journals/computing", 0.2), ("conf/icdcs", 0.1),
108 + ("journals/tcc", 0.1), ("conf/ipps", 0.1), ("conf/isca", 0.1), ("conf/micro", 0.1), ("conf/hpca", 0.1),
109 + ("journals/taco", 0.1), ("conf/sec", 0.1), ("conf/percom", 0.1), ("journals/jss", 0.1), ("conf/uss", 0.1),
110 + ("conf/sp", 0.05), ("conf/ccs", 0.05), ("journals/tdsc", 0.1), ("conf/sigcomm", 0.1)],
111 + keywords=[
112 + (r"(?i)\boperating systems?\b|\bOS (kernels?|design|support|level|services|scheduler|virtuali[sz]ation|security|abstractions?|research|noise|jitter|interfaces?)\b|\bOSes\b|\bsystems software\b", 1.0, "both"),
113 + (r"(?i)\b(Linux (kernel|scheduler|drivers?|namespaces|cgroups|systems?)|kernel (modules?|bypass|extensions?|development|schedulers?|space|drivers?|bugs?|fuzzing|vulnerabilit|hardening|isolation|concurrency|verification|crashes|panics|patches|APIs?|data structures)|eBPF|BPF programs?|device drivers?|system calls?|syscalls?|user ?space|microkernels?|unikernels?|monolithic kernels?|exokernel|library OS|seL4|Fuchsia|Zephyr|FreeRTOS|RTOS\b|real-?time operating|Android (framework|runtime|kernel|OS|binder)|Windows (kernel|drivers?)|macOS kernel|XNU|file systems?|filesystems?|FS\b|ext4|Btrfs|ZFS|F2FS|XFS|journaling|log-?structured|copy-?on-?write|page cache|buffer cache|virtual memory|page (tables?|faults?|replacement|migration)|TLB\b|memory (management|allocation|allocators?|mapping|paging|swapping|deduplication|tiering|disaggregation|compaction|reclamation|safety in kernels?|bandwidth management)|huge pages|transparent huge|NUMA (awareness|placement|balancing)|process (scheduling|isolation|management|migration)|thread scheduling|CPU scheduling|schedulers? (for|in) (Linux|kernels?|multicore|data ?cent|heterogeneous)|CFS\b|EEVDF|interrupt (handling|latency|coalescing)|I/O (stacks?|scheduling|virtuali[sz]ation|paths?|performance|schedulers?)|io_uring|NVMe (SSDs?|drivers?|over fabrics)|SSDs?\b|flash (storage|translation layer|memory management)|FTL\b|zoned (namespaces|storage)|ZNS|persistent memory (programming|file systems?|allocators?)|PMEM|CXL\b|compute express link|memory expanders?|far memory|virtual machine monitors?|VMMs?\b|hypervisors?|Xen|KVM|QEMU|VMware|Hyper-?V|Firecracker|gVisor|microVMs?|containers? (isolation|runtimes?|security|images?|startup)|namespaces and cgroups|cgroups|Docker (runtime|engine|containers?)|OCI (runtime|images?)|containerd|runc|sandbox(ing|es)|WebAssembly (runtime|system interface|WASI)|boot(ing|loaders?|strap)|firmware (updates?|interfaces?)|BIOS|UEFI|trusted (boot|computing base)|TCB\b|crash (consistency|recovery|dumps?)|fault (isolation|containment) in (OS|kernel)|Rust (for Linux|in the kernel|OS)|seccomp|capabilit(y|ies)-?based (OS|security|systems)|CHERI|address space (isolation|layout)|ASLR|kernel (integrity|protection|exploits?)|privilege (levels?|rings?)|ring 0|supervisor mode|system (daemons?|services) management|init systems?|systemd|power management (in|for) (OS|Linux|mobile)|energy-?aware scheduling|DVFS governors?|thermal throttling in|mobile OS|smartphone (OS|kernel|power)|wearable OS|IoT (OS|operating)|TinyOS|Contiki|RIOT OS|embedded (Linux|OS|runtimes?)|hardware (abstraction|virtuali[sz]ation)|paravirtuali[sz]|nested virtuali[sz]ation|live (migration|patching|update)|hot-?patching|checkpoint/restore|CRIU|fork\(\)|exec\(\)|POSIX|system (configuration|administration)|package managers?|Nix(OS)?|reproducible builds)\b", 1.0, "both"),
114 + (r"(?i)\b(storage systems?|distributed file systems?|key-?value stores?|object storage|block storage|storage (stacks?|engines?|disaggregation|tiering|class memory)|RDMA|DPDK|SR-?IOV|SmartNICs?|DPUs?\b|data processing units?|IPUs?\b|kernel-?bypass|user-?space networking|TCP (stack|offload)|network stacks?|serverless (runtimes?|platforms?|cold starts?)|cold starts?|function (isolation|sandboxes?)|lightweight (virtuali[sz]ation|isolation|VMs?)|confidential VMs?|enclaves? (OS|runtime)|TEE (OS|runtime)|GPU (drivers?|virtuali[sz]ation|scheduling in OS|memory management)|accelerator (OS|virtuali[sz]ation|management)|resource (isolation|contention|management in (OS|kernels?))|tail latency|performance isolation|noisy neighbou?rs?|QoS in (OS|kernels?)|workload (consolidation|co-?location|interference))\b", 0.6, "both"),
115 + (r"(?i)\b(kernels?|runtimes?|schedulers?|scheduling|virtuali[sz]ation|concurrency|threads?|locks?|synchroni[sz]ation|I/O|storage|drivers?|isolation|hypervisor|containers?|processes|memory)\b", 0.3, "both"),
116 + ],
117 + exclusions=[r"(?is)^(?!.*(operating system|kernel|Linux|file system|virtual|hypervisor|scheduler|driver|memory management|syscall|I/O|storage|container|runtime|boot|firmware)).*\b(kernel (methods?|machines?|ridge|density|trick|functions?|SVM|PCA|regression|smoothing|estimation|mean embedding|Hilbert|convolution kernels?|size|regularization|alignment)|convolution(al)? kernels?|GPU kernels? for (matrix|stencil)|wheat kernels?|corn kernels?|palm kernels?|heat kernel|Gaussian kernel)\b"],
118 + related=["computer_architecture", "distributed_computing", "cloud_computing", "cybersecurity", "programming_languages", "edge_computing"],
119 + parent_of=[], child_of=[],
120 + provenance=("No OpenAlex topic is OS-centred: storage (T11181), real-time scheduling (T10933), OS security/hypervisors (T11424) "
121 + "and virtualization (T10101) are the hosts, so each is partial. OSDI/SOSP/EuroSys/USENIX ATC/FAST/TOS are pure "
122 + "streams (cs.OS alone suffices on arXiv). Kernel-method senses of 'kernel' are vetoed when no OS term co-occurs."),
123 +),
124 +]
added taxonomy/src/area_perception_a.py +137 −0
@@ -0,0 +1,137 @@
1 +"""Area perception_interaction, part A: categories 9-11 (computer vision, NLP, speech and audio)."""
2 +
3 +CATEGORIES = [
4 +dict(
5 + id="computer_vision", number=9, name="Computer Vision", area="perception_interaction",
6 + definition=("Extracting information from images, video and 3D data: recognition, detection, segmentation, tracking, "
7 + "pose, depth and 3D reconstruction, image restoration and enhancement, medical and remote-sensing image "
8 + "analysis, visual representation learning. Image synthesis is generative_ai; vision-language is multimodal_ai; "
9 + "rendering is computer_graphics."),
10 + topics=[("T10036", 1.0, "object detection, segmentation, YOLO"), ("T10531", 1.0, "advanced vision and imaging"),
11 + ("T10331", 1.0, "video surveillance and tracking"), ("T11448", 1.0, "face recognition"),
12 + ("T10057", 1.0, "face and expression recognition"), ("T10812", 1.0, "human pose and action recognition"),
13 + ("T12549", 1.0, "image and object detection"), ("T11605", 1.0, "visual attention and saliency"),
14 + ("T14339", 1.0, "image processing and 3D reconstruction"), ("T10601", 0.8, "handwritten text recognition"),
15 + ("T10824", 0.8, "image retrieval and classification"), ("T11019", 0.8, "image enhancement"),
16 + ("T11105", 0.8, "advanced image processing"), ("T11439", 0.8, "video analysis and summarization"),
17 + ("T12923", 0.8, "digital image processing"), ("T13579", 0.8, "image and video stabilization"),
18 + ("T12111", 0.8, "industrial vision and defect detection"), ("T10052", 0.7, "medical image segmentation"),
19 + ("T10688", 0.7, "image and signal denoising"), ("T10627", 0.7, "image and video retrieval"),
20 + ("T11165", 0.7, "image and video quality assessment"), ("T13114", 0.7, "image processing techniques"),
21 + ("T10444", 0.6, "context-aware activity recognition"), ("T10689", 0.6, "remote-sensing image classification"),
22 + ("T11659", 0.6, "image fusion"), ("T12422", 0.5, "radiomics and ML in medical imaging"),
23 + ("T12983", 0.5, "satellite image processing and photogrammetry"), ("T12859", 0.5, "cell image analysis"),
24 + ("T10719", 0.5, "3D shape modeling and analysis"), ("T11398", 0.5, "hand gesture recognition"),
25 + ("T14319", 0.5, "currency recognition"), ("T10522", 0.4, "medical imaging techniques"),
26 + ("T12357", 0.4, "digital media forensics"), ("T11714", 0.4, "multimodal (vision-language)"),
27 + ("T11707", 0.4, "gaze tracking"), ("T10775", 0.3, "GANs and image synthesis"),
28 + ("T10888", 0.3, "augmented reality"), ("T10901", 0.3, "advanced data compression (image)"),
29 + ("T10741", 0.3, "video coding"), ("T10388", 0.3, "steganography and watermarking"),
30 + ("T11017", 0.2, "chaos-based image encryption"), ("T12874", 0.4, "digital imaging for blood diseases"),
31 + ("T10638", 0.2, "optical measurement and interference"), ("T13759", 0.3, "medical imaging and pathology")],
32 + subfields=[(1707, 0.5)],
33 + arxiv=[("cs.CV", 1.0), ("eess.IV", 0.6)],
34 + streams=[("conf/cvpr", 1.0), ("conf/iccv", 1.0), ("conf/eccv", 1.0), ("conf/wacv", 1.0), ("conf/bmvc", 1.0),
35 + ("conf/accv", 1.0), ("conf/iccvw", 1.0), ("conf/fgr", 1.0), ("conf/3dim", 1.0), ("journals/ijcv", 1.0),
36 + ("journals/cviu", 1.0), ("journals/ivc", 1.0), ("journals/mva", 1.0), ("journals/pami", 0.8),
37 + ("journals/tip", 0.8), ("conf/icip", 0.8), ("conf/icpr", 0.8), ("conf/icdar", 0.8), ("conf/visapp", 0.8),
38 + ("conf/prcv", 0.8), ("journals/tcsv", 0.7), ("journals/iet-ipr", 0.7), ("journals/jrtip", 0.7),
39 + ("journals/pr", 0.6), ("journals/prl", 0.6), ("journals/jvcir", 0.6), ("journals/sivp", 0.6),
40 + ("journals/jimaging", 0.6), ("conf/miccai", 0.6), ("journals/mia", 0.6), ("journals/tmi", 0.5),
41 + ("conf/isbi", 0.5), ("journals/paa", 0.5), ("journals/tci", 0.5), ("journals/imst", 0.5),
42 + ("journals/cmig", 0.5), ("journals/spic", 0.5), ("journals/jei", 0.5), ("journals/cars", 0.4),
43 + ("journals/bmcmi", 0.4), ("journals/ijprai", 0.5), ("journals/tgrs", 0.3), ("conf/igarss", 0.2),
44 + ("journals/remotesensing", 0.2), ("journals/lgrs", 0.3), ("journals/staeors", 0.2), ("conf/vcip", 0.6),
45 + ("conf/icmcs", 0.3), ("conf/mm", 0.2), ("journals/mta", 0.3), ("conf/ivs", 0.3), ("journals/tits", 0.1),
46 + ("journals/jdi", 0.3), ("journals/jmihi", 0.3), ("conf/bmei", 0.4), ("journals/displays", 0.2),
47 + ("conf/icpram", 0.5), ("journals/cg", 0.2)],
48 + keywords=[
49 + (r"(?i)\bcomputer vision\b|\bmachine vision\b|\bvision (models?|transformers?|tasks?|backbones?|systems?)\b", 1.0, "both"),
50 + (r"(?i)\b(object|face|facial|pedestrian|vehicle|action|activity|pose|gesture|scene|visual|image|video|person|text|lane|landmark|anomaly in video|salient object) (detection|recognition|tracking|re-?identification|classification|segmentation|understanding|retrieval|estimation|parsing|captioning|matting)\b", 0.8, "both"),
51 + (r"(?i)\b(semantic|instance|panoptic|medical image|image|video|3D|point[- ]cloud|interactive|few-?shot|open-?vocabulary) segmentation\b", 1.0, "both"),
52 + (r"(?i)\b(point clouds?|optical flow|depth (estimation|completion)|stereo (matching|vision)|3D reconstruction|structure[- ]from[- ]motion|SfM\b|multi-?view (stereo|geometry)|NeRF|neural radiance fields?|Gaussian splatting|visual (SLAM|odometry|localization)|camera (pose|calibration)|epipolar|homography|feature (matching|descriptors?)|SIFT|keypoints?)\b", 1.0, "both"),
53 + (r"(?i)\b(image|video) (processing|enhancement|denoising|restoration|deblurring|dehazing|super-?resolution|quality assessment|registration|retrieval|classification|synthesis|inpainting|compression|coding|forensics|fusion|stitching|matting|colori[sz]ation|harmoni[sz]ation)\b|\bvideo (surveillance|understanding|analysis|summari[sz]ation|prediction|anomaly)\b", 0.8, "both"),
54 + (r"(?i)\b(convolutional neural networks?|CNNs?|YOLO\w*|R-?CNN|Mask R-?CNN|DETR|U-?Net|ViTs?|vision transformers?|Swin|ResNet|VGG|backbone networks?|feature pyramid|anchor-?free|non-?maximum suppression|IoU\b|mAP\b|COCO|ImageNet|Pascal VOC|KITTI|nuScenes|Cityscapes|ADE20K)\b", 0.6, "both"),
55 + (r"(?i)\b(medical imag(e|es|ing)|radiomics|histopatholog|whole-?slide|retinal (image|fundus)|OCT (image|scan)|MRI|CT (scans?|images?)|X-?ray (images?|classification)|ultrasound (image|video)|mammogra|dermoscop|endoscop|colonoscop|microscopy images?)\b", 0.5, "both"),
56 + (r"(?i)\b(saliency|visual attention|OCR|optical character recognition|handwrit|scene text|document (image|layout|understanding)|license plate|biometrics?|iris recognition|fingerprint|gait recognition|crowd counting|human parsing|6-?DoF pose|hand pose|body pose|facial (landmarks?|expression|attributes?)|deepfake detection|face anti-?spoofing|remote sensing (images?|scene)|hyperspectral (image|classification)|SAR (images?|target)|change detection in|aerial imag|satellite imag|UAV imag)\b", 0.8, "both"),
57 + (r"(?i)\b(self-?supervised visual|contrastive visual|visual representation|visual features?|image embeddings?|visual recognition|visual perception|pixel-?wise|pixel-?level|bounding box(es)?|RGB-?D|LiDAR (point|perception|segmentation|detection)|BEV (perception|features?|segmentation)|bird'?s-?eye view|autonomous driving perception|3D (object detection|detection|perception|vision|human))\b", 0.8, "both"),
58 + (r"(?i)\b(images?|visual|pixels?|cameras?|videos?)\b", 0.2, "both"),
59 + ],
60 + exclusions=[],
61 + related=["multimodal_ai", "computer_graphics", "robotics", "deep_learning_architectures", "generative_ai", "computational_biology"],
62 + parent_of=[], child_of=[],
63 + provenance=("All 35 topics of OpenAlex subfield 1707 reviewed; those that are image/video-centred receive 0.5-1.0, "
64 + "media/compression/encryption topics are shared at low weight; relevant Engineering (22) remote-sensing "
65 + "topics and Medicine imaging topics added at partial weight. cs.CV is a sufficient signal."),
66 +),
67 +dict(
68 + id="nlp", number=10, name="Natural Language Processing", area="perception_interaction",
69 + definition=("Computational processing of human language: machine translation, parsing and tagging, information "
70 + "extraction, sentiment and text classification, summarisation, question answering, dialogue, text mining, "
71 + "computational linguistics, low-resource and multilingual NLP, NLP evaluation and resources. LLM-centred "
72 + "work is llm (also classified here when the task framing dominates)."),
73 + topics=[("T10181", 1.0, "NLP techniques: MT, parsing, NLU/NLG"), ("T10664", 1.0, "sentiment analysis and opinion mining"),
74 + ("T11550", 1.0, "text and document classification"), ("T13629", 1.0, "text readability and simplification"),
75 + ("T13083", 0.8, "advanced text analysis"), ("T12380", 0.8, "authorship attribution and profiling"),
76 + ("T11710", 0.8, "biomedical text mining and ontologies"), ("T12262", 0.7, "hate speech detection"),
77 + ("T10028", 0.6, "topic modeling / pre-trained LMs"), ("T12031", 0.6, "speech and dialogue systems"),
78 + ("T13910", 0.5, "computational and text analysis methods"), ("T13623", 0.4, "linguistic studies and language acquisition"),
79 + ("T10215", 0.4, "semantic web and ontologies"), ("T13274", 0.4, "expert finding and Q&A"),
80 + ("T11010", 0.3, "logic, reasoning and knowledge representation"), ("T10034", 0.3, "syntax, semantics, linguistic variation"),
81 + ("T11644", 0.3, "spam and phishing detection (text)"), ("T11147", 0.3, "misinformation"),
82 + ("T12016", 0.3, "web data mining and analysis"), ("T13155", 0.3, "digital communication and language"),
83 + ("T12353", 0.2, "lexicography and language studies"), ("T13310", 0.2, "subtitles and audiovisual media")],
84 + subfields=[(1702, 0.1)],
85 + arxiv=[("cs.CL", 1.0), ("cs.IR", 0.1)],
86 + streams=[("conf/acl", 1.0), ("conf/emnlp", 1.0), ("conf/naacl", 1.0), ("conf/eacl", 1.0), ("conf/coling", 1.0),
87 + ("journals/tacl", 1.0), ("journals/coling", 1.0), ("conf/lrec", 1.0), ("conf/semeval", 1.0), ("conf/wmt", 1.0),
88 + ("conf/nlpcc", 1.0), ("journals/talip", 1.0), ("conf/wassa", 1.0), ("conf/sepln", 0.8), ("journals/csl", 0.6),
89 + ("journals/taslp", 0.5), ("conf/clef", 0.5), ("journals/ipm", 0.3), ("conf/dihu", 0.3), ("conf/dhd", 0.3),
90 + ("conf/interspeech", 0.2), ("conf/aaai", 0.1), ("conf/ijcai", 0.1), ("journals/kbs", 0.1),
91 + ("journals/eswa", 0.1), ("conf/cikm", 0.2), ("conf/www", 0.1), ("journals/jasis", 0.2), ("conf/cogsci", 0.2)],
92 + keywords=[
93 + (r"(?i)\bnatural language (processing|understanding|generation|inference|interfaces?)\b|\bNLP\b|\bNLU\b|\bNLG\b|\bNLI\b", 1.0, "both"),
94 + (r"(?i)\b(machine translation|neural machine translation|NMT\b|named entity recognition|NER\b|part-?of-?speech|POS tagging|dependency parsing|constituency parsing|coreference resolution|word embeddings?|word2vec|GloVe|sentiment analysis|opinion mining|aspect-?based sentiment|text (classification|summari[sz]ation|generation|mining|simplification|normali[sz]ation|clustering|retrieval)|abstractive summari[sz]ation|extractive summari[sz]ation|question answering|reading comprehension|relation extraction|event extraction|information extraction|semantic (parsing|role labeling|similarity|textual similarity)|textual entailment|language identification|tokeni[sz]ation|lemmati[sz]ation|stemming|word sense disambiguation|grammatical error correction|spell(ing)? correction|discourse (parsing|analysis)|argument mining|keyphrase extraction|topic model(ing|s)|LDA\b|latent Dirichlet)\b", 1.0, "both"),
95 + (r"(?i)\b(computational linguistics|corpus linguistics|treebanks?|low-?resource languages?|multilingual|cross-?lingual|code-?switching|morpholog(y|ical) (analysis|segmentation)|lexical (semantics|resources?)|WordNet|language resources?|annotated corp(us|ora)|inter-?annotator)\b", 0.8, "both"),
96 + (r"(?i)\b(dialogue (systems?|state tracking|generation|management|acts?)|task-?oriented dialogue|open-?domain dialogue|chatbots?|conversational (AI|agents?|systems?|search))\b", 0.7, "both"),
97 + (r"(?i)\b(hate speech|fake news|misinformation|rumou?r|stance|sarcasm|irony|emotion|humou?r|offensive language|toxic(ity)?|propaganda|clickbait|authorship) (detection|classification|identification|recognition|attribution|analysis)\b", 0.8, "both"),
98 + (r"(?i)\b(language models?|BERT|GPT|transformers?|sequence-?to-?sequence|seq2seq|encoder-?decoder|attention)\b", 0.3, "both"),
99 + (r"(?i)\b(knowledge graph|ontolog(y|ies)|semantic web|entity linking|knowledge base) (completion|embedding|construction|population|question answering|alignment|learning)?\b", 0.4, "both"),
100 + (r"(?i)\b(text|texts|textual|linguistic|lexical|syntactic|semantic|sentences?|words?|documents?|corpus|corpora|language)\b", 0.2, "both"),
101 + ],
102 + exclusions=[],
103 + related=["llm", "speech_audio", "information_retrieval", "multimodal_ai", "computational_biology"],
104 + parent_of=[], child_of=[],
105 + provenance=("OpenAlex 1702 language topics (T10181, T10664, T11550, T13083, T12380, T13629, T12262) plus biomedical "
106 + "text mining (field 13) and social-science text topics. *CL venues are the strongest venue signal in the "
107 + "whole ontology (ACL Anthology coverage)."),
108 +),
109 +dict(
110 + id="speech_audio", number=11, name="Speech and Audio Computing", area="perception_interaction",
111 + definition=("Processing and generation of speech, music and general audio: speech recognition and synthesis, "
112 + "speaker and language recognition, enhancement and separation, spoken language understanding, music "
113 + "information retrieval and generation, audio event detection, neural audio codecs. Text-only dialogue is nlp."),
114 + topics=[("T10201", 1.0, "speech recognition and synthesis"), ("T10860", 1.0, "speech and audio processing"),
115 + ("T11309", 1.0, "music and audio processing"), ("T11349", 0.6, "music technology and sound studies"),
116 + ("T11447", 0.6, "blind source separation"), ("T12031", 0.5, "speech and dialogue systems"),
117 + ("T10863", 0.3, "voice and speech disorders"), ("T11034", 0.2, "digital filter design"),
118 + ("T10931", 0.2, "direction-of-arrival estimation (microphone arrays)"), ("T10941", 0.1, "musicology")],
119 + subfields=[(1711, 0.2)],
120 + arxiv=[("eess.AS", 1.0), ("cs.SD", 1.0), ("cs.CL", 0.1), ("eess.SP", 0.1)],
121 + streams=[("conf/interspeech", 1.0), ("journals/taslp", 1.0), ("journals/speech", 1.0), ("conf/slt", 1.0),
122 + ("conf/asru", 1.0), ("journals/csl", 0.8), ("conf/icassp", 0.5), ("conf/apsipa", 0.3), ("conf/eusipco", 0.3),
123 + ("journals/spl", 0.2), ("journals/sigpro", 0.2), ("journals/dsp", 0.2), ("journals/jstsp", 0.2),
124 + ("journals/cssp", 0.1), ("conf/mm", 0.1), ("journals/tmm", 0.1)],
125 + keywords=[
126 + (r"(?i)\bspeech (recognition|synthesis|enhancement|separation|processing|emotion|translation|coding|to text|to speech|language models?|foundation models?|representations?|technolog(y|ies)|signals?|corpus|corpora|data)\b|\btext-?to-?speech\b|\bTTS\b|\bASR\b|\bspeech-?to-?text\b", 1.0, "both"),
127 + (r"(?i)\b(automatic speech|spoken (language|dialogue|term|question)|speaker (recognition|verification|identification|diarization|embeddings?|adaptation)|voice (conversion|cloning|assistants?|activity detection|biometrics|synthesis)|keyword spotting|wake[- ]word|acoustic (model(s|ing)?|scene|event|features?|echo)|phonemes?|phonetic|prosod(y|ic)|intonation|lip[- ]reading|audio-?visual speech|dysarthri|pathological speech|whisper(ed)? speech)\b", 1.0, "both"),
128 + (r"(?i)\b(audio|music|sound|acoustic) (classification|generation|synthesis|tagging|retrieval|processing|understanding|event detection|source separation|representation|captioning|language models?|foundation models?|codecs?|signals?|features?|embeddings?)\b", 1.0, "both"),
129 + (r"(?i)\b(music information retrieval|MIR\b|music (transcription|recommendation|generation|genre|emotion|structure|similarity)|singing voice|beat tracking|chord (recognition|estimation)|melody extraction|symbolic music|MIDI|audio (signal|watermark|deepfake|anti-?spoofing)|microphone arrays?|beamforming for speech|hearing aids?|cochlear|room (impulse|acoustics)|reverberation|dereverberation|noise suppression|sound event|environmental sound|bird (song|call) (detection|classification)|bioacoustic|wav2vec|Whisper\b|HuBERT|WavLM|WaveNet|vocoders?|HiFi-?GAN|spectrograms?|mel[- ]spectrogram|MFCCs?|EnCodec|SoundStream)\b", 0.9, "both"),
130 + (r"(?i)\b(speech|audio|acoustic|speaker|music|voice|sound|auditory|phonological)\b", 0.3, "both"),
131 + ],
132 + exclusions=[],
133 + related=["nlp", "multimodal_ai", "generative_ai", "llm"],
134 + parent_of=[], child_of=[],
135 + provenance="OpenAlex T10201/T10860/T11309 and the eess.AS/cs.SD arXiv pair; INTERSPEECH/TASLP strong, ICASSP partial (shared with signal processing).",
136 +),
137 +]
added taxonomy/src/area_perception_b.py +131 −0
@@ -0,0 +1,131 @@
1 +"""Area perception_interaction, part B: categories 12, 13, 35 (robotics, HCI, computer graphics)."""
2 +
3 +CATEGORIES = [
4 +dict(
5 + id="robotics", number=12, name="Robotics", area="perception_interaction",
6 + definition=("Machines that sense and act in the physical world: manipulation, locomotion, motion and path planning, "
7 + "SLAM and localisation, robot learning, human-robot interaction, autonomous vehicles and drones as robotic "
8 + "systems, soft and swarm robotics, teleoperation. RL algorithms are reinforcement_learning; robotic "
9 + "process automation and surgical-outcome studies are excluded."),
10 + topics=[("T10586", 1.0, "robotic path planning"), ("T10653", 1.0, "robot manipulation and learning"),
11 + ("T10191", 1.0, "robotics and sensor-based localization"), ("T10709", 1.0, "social robot interaction and HRI"),
12 + ("T10879", 1.0, "robotic locomotion and control"), ("T11615", 1.0, "control and dynamics of mobile robots"),
13 + ("T10571", 0.8, "robotic mechanisms and dynamics"), ("T10868", 0.8, "soft robotics"),
14 + ("T13382", 0.8, "robotics and automated systems"), ("T12784", 0.8, "modular robots and swarm intelligence"),
15 + ("T11687", 0.7, "teleoperation and haptic systems"), ("T11099", 0.7, "autonomous vehicle technology and safety"),
16 + ("T11023", 0.6, "prosthetics and rehabilitation robotics"), ("T11133", 0.6, "UAV applications"),
17 + ("T10462", 0.5, "reinforcement learning in robotics"), ("T10249", 0.5, "distributed control multi-agent (multi-robot)"),
18 + ("T14335", 0.5, "educational robotics"), ("T11486", 0.5, "micro and nano robotics"),
19 + ("T11325", 0.4, "inertial sensor and navigation"), ("T12158", 0.3, "guidance and control systems"),
20 + ("T10805", 0.2, "vehicle dynamics and control"), ("T11192", 0.3, "underwater vehicles"),
21 + ("T10525", 0.2, "human-automation interaction")],
22 + subfields=[],
23 + arxiv=[("cs.RO", 1.0), ("eess.SY", 0.2), ("cs.SY", 0.2)],
24 + streams=[("conf/icra", 1.0), ("conf/iros", 1.0), ("journals/ral", 1.0), ("conf/rss", 1.0), ("conf/corl", 1.0),
25 + ("journals/trob", 1.0), ("journals/ijrr", 1.0), ("journals/arobots", 1.0), ("journals/ras", 1.0),
26 + ("journals/jirs", 1.0), ("journals/robotica", 1.0), ("journals/jfr", 1.0), ("conf/humanoids", 1.0),
27 + ("conf/robio", 1.0), ("conf/hri", 1.0), ("journals/ijsr", 1.0), ("conf/ro-man", 1.0), ("conf/icira", 1.0),
28 + ("journals/firai", 1.0), ("journals/finr", 1.0), ("journals/robotics", 1.0), ("journals/ar", 1.0),
29 + ("journals/jrm", 1.0), ("conf/rcar", 1.0), ("conf/icarm", 1.0), ("conf/biorob", 1.0), ("conf/robosoft", 1.0),
30 + ("conf/aimech", 0.8), ("conf/sii", 0.8), ("conf/case", 0.6), ("conf/icarcv", 0.6), ("journals/rcim", 0.6),
31 + ("conf/icinco", 0.6), ("journals/tiv", 0.5), ("conf/ivs", 0.5), ("journals/tase", 0.5), ("conf/itsc", 0.3),
32 + ("journals/tits", 0.2), ("conf/cdc", 0.1), ("journals/tcst", 0.1), ("conf/amcc", 0.1)],
33 + keywords=[
34 + (r"(?i)\brobot(s|ic|ics|ically)?\b|\bcobots?\b", 1.0, "both"),
35 + (r"(?i)\b(manipulat(ion|ors?)|grasp(ing|s)?|dexterous|motion planning|path planning|trajectory (planning|optimi[sz]ation)|SLAM\b|simultaneous locali[sz]ation|visual servoing|legged|quadrupeds?|bipeds?|humanoids?|exoskeletons?|end-?effectors?|teleoperation|telerobot|kinematics|inverse dynamics|whole-?body control|locomotion)\b", 0.8, "both"),
36 + (r"(?i)\b(autonomous (driving|vehicles?|navigation|systems?|mobile|underwater|aerial|ground)|self-?driving|UAVs?|UGVs?|AUVs?|drones?|quadrotors?|quadcopters?|unmanned (aerial|ground|surface|underwater)|multi-?robot|swarm robot)\b", 0.7, "both"),
37 + (r"(?i)\b(human-?robot|embodied (AI|agents?|intelligence|learning)|sim-?to-?real|imitation learning|learning from demonstration|vision-?language-?action|VLA models?|robot(ic)? foundation models?|RT-?[12]\b|Open-?X Embodiment|diffusion polic(y|ies)|affordance|tactile sensing|force control|compliance control|soft (actuators?|grippers?))\b", 0.8, "both"),
38 + ],
39 + exclusions=[r"(?i)\brobotic process automation\b|\bRPA\b",
40 + r"(?is)^(?!.*(algorithm|control|planning|learning|sensor|kinematic|autonom|simulation|software|system design)).*\brobot(ic)?-?assisted (surgery|prostatectomy|hysterectomy|laparoscop\w+|thoracic|radical|gastrectomy|colectomy|nephrectomy|cystectomy)\b"],
41 + related=["reinforcement_learning", "computer_vision", "ai_agents", "hci", "edge_computing"],
42 + parent_of=[], child_of=[],
43 + provenance=("OpenAlex spreads robotics over CS 1707 (path planning), Engineering 2202/2204/2207 (manipulation, locomotion, "
44 + "mobile robots) and Psychology 3207 (HRI); all reviewed. 'robot' is unambiguous enough to be a 1.0 keyword."),
45 +),
46 +dict(
47 + id="hci", number=13, name="Human-Computer Interaction", area="perception_interaction",
48 + definition=("Design, evaluation and understanding of interactive systems and their users: user studies and usability, "
49 + "interaction techniques, VR/AR/XR, accessibility and assistive technology, CSCW and social computing, "
50 + "human-AI interaction, ubiquitous and wearable computing, affective computing. Rendering is "
51 + "computer_graphics; physical robots are robotics."),
52 + topics=[("T10470", 1.0, "usability and user interface design"), ("T10648", 1.0, "virtual reality applications"),
53 + ("T10789", 1.0, "interactive and immersive displays"), ("T10803", 1.0, "innovative human-technology interaction"),
54 + ("T11707", 0.8, "gaze tracking and assistive technology"), ("T14074", 0.8, "persona design"),
55 + ("T10888", 0.8, "augmented reality applications"), ("T13874", 0.8, "information architecture and usability"),
56 + ("T11398", 0.7, "hand gesture recognition"), ("T10525", 0.7, "human-automation interaction and safety"),
57 + ("T10709", 0.5, "social robot interaction"), ("T11704", 0.5, "mobile crowdsensing and crowdsourcing"),
58 + ("T11687", 0.5, "teleoperation and haptics"), ("T10429", 0.5, "EEG and brain-computer interfaces"),
59 + ("T12128", 0.4, "AI in service interactions (chatbot UX)"), ("T13155", 0.4, "digital communication and language"),
60 + ("T10914", 0.3, "tactile and sensory interactions"), ("T11902", 0.3, "intelligent tutoring systems"),
61 + ("T10444", 0.3, "context-aware activity recognition (ubicomp)"), ("T12260", 0.2, "educational technology"),
62 + ("T11122", 0.2, "online learning and analytics"), ("T13807", 0.2, "cultural insights and digital impacts"),
63 + ("T12338", 0.2, "digital literacy"), ("T14114", 0.2, "digital media and philosophy")],
64 + subfields=[(1709, 0.6)],
65 + arxiv=[("cs.HC", 1.0), ("cs.CY", 0.2), ("cs.SI", 0.1)],
66 + streams=[("conf/chi", 1.0), ("conf/uist", 1.0), ("conf/cscw", 1.0), ("journals/pacmhci", 1.0), ("journals/tochi", 1.0),
67 + ("journals/ijmms", 1.0), ("journals/ijhci", 1.0), ("conf/iui", 1.0), ("conf/ACMdis", 1.0), ("conf/vr", 1.0),
68 + ("conf/ismar", 1.0), ("journals/vr", 1.0), ("conf/huc", 1.0), ("journals/imwut", 1.0), ("conf/assets", 1.0),
69 + ("conf/interact", 1.0), ("conf/mc", 1.0), ("conf/tei", 1.0), ("conf/acmidc", 1.0), ("conf/etra", 1.0),
70 + ("journals/behaviourIT", 1.0), ("journals/interactions", 1.0), ("conf/hci", 0.8), ("journals/chb", 0.6),
71 + ("conf/ahfe", 0.6), ("conf/icmi", 0.6), ("journals/taffco", 0.6), ("journals/displays", 0.5),
72 + ("journals/cbsn", 0.4), ("journals/nms", 0.3), ("conf/hri", 0.3), ("journals/ile", 0.3), ("journals/tvcg", 0.2),
73 + ("journals/ce", 0.2), ("journals/eait", 0.2), ("conf/um", 0.3), ("journals/entcom", 0.5), ("conf/cig", 0.3),
74 + ("journals/tele", 0.2), ("conf/icwsm", 0.3), ("journals/ijim", 0.3), ("journals/jaihc", 0.2), ("conf/percom", 0.3)],
75 + keywords=[
76 + (r"(?i)\bhuman-?computer interaction\b|\bHCI\b|\bCSCW\b|\bcomputer-?supported cooperative\b|\buser-?centered design\b|\bhuman factors\b", 1.0, "both"),
77 + (r"(?i)\b(user (study|studies|experience|interfaces?|engagement|perception|acceptance|needs|research|testing|feedback)|usability (study|testing|evaluation)?|UX\b|UI (design|components)|participatory design|co-?design|design (fiction|probes?|implications|space|guidelines)|think-?aloud|Wizard[- ]of[- ]Oz|formative (study|evaluation)|within-?subjects?|between-?subjects?|Likert|System Usability Scale|SUS\b|NASA-?TLX)\b", 0.8, "both"),
78 + (r"(?i)\b(virtual|augmented|mixed|extended) reality\b|\bVR\b|\bXR\b|\bhead-?mounted\b|\bimmersive\b|\bmetaverse\b|\bHMDs?\b|\btelepresence\b|\bcybersickness\b|\bpresence (in|and) VR\b", 0.8, "both"),
79 + (r"(?i)\b(accessibilit(y|ies)|assistive technolog(y|ies)|eye[- ]tracking|gaze(-based)?|gestures?|gestural|haptics?|tangible|wearables?|smart ?watch|brain-?computer interfaces?|BCI\b|affective computing|emotion recognition for interaction|screen readers?|blind (users|people)|visually impaired|older adults|children'?s? (interaction|technology)|neurodivergent)\b", 0.7, "both"),
80 + (r"(?i)\b(human-?AI (interaction|collaboration|teaming|co-?creation)|human-?in-?the-?loop|human-?centered AI|explanation interfaces?|conversational (user )?interfaces?|voice (user )?interfaces?|chatbot (design|user)|interactive machine learning|mixed-?initiative|AI-?assisted (writing|design|creativity)|trust in (AI|automation|robots))\b", 0.8, "both"),
81 + (r"(?i)\b(interaction (design|techniques?|paradigms?|modalit)|interactive (systems?|visuali[sz]ation|prototypes?|installations?)|user interfaces?|GUIs?\b|touch(screen| input| interaction)|pen input|mid-?air|gaze input|input (devices?|techniques?)|text entry|target acquisition|Fitts)\b", 0.8, "both"),
82 + (r"(?i)\b(ubiquitous computing|pervasive computing|ubicomp|context-?aware(ness)?|smart (homes?|environments?)|ambient (intelligence|assisted)|activity recognition (from|using|with) (wearable|smartphone|sensors?)|mobile (interaction|HCI|users?)|notifications?|digital well-?being|persuasive technolog|serious games?|gamification|player experience|game (design|user research))\b", 0.7, "both"),
83 + (r"(?i)\b(social computing|social media (use|users|platforms?)|online communit(y|ies)|crowdsourcing|crowd ?workers?|Mechanical Turk|content creators?|misinformation (on|and) (social|platforms)|online harassment|moderation practices|collaborative (work|writing|tools)|remote (work|collaboration)|computer-?mediated communication)\b", 0.6, "both"),
84 + (r"(?i)\b(qualitative (study|analysis|interviews?)|interview study|semi-?structured interviews?|thematic analysis|grounded theory|focus groups?|diary study|ethnograph|participants (reported|described|were recruited)|N\s?=\s?\d+ participants)\b", 0.5, "both"),
85 + (r"(?i)\b(users?|participants?|interaction|interactive|interfaces?|design)\b", 0.2, "both"),
86 + ],
87 + exclusions=[],
88 + related=["computer_graphics", "robotics", "ai_safety", "multimodal_ai", "software_engineering", "information_retrieval"],
89 + parent_of=[], child_of=[],
90 + provenance=("All 8 topics of OpenAlex subfield 1709 mapped; supporting topics from Psychology (human-automation), "
91 + "Neuroscience (BCI) and education technology at low weight. CHI/UIST/CSCW/PACM HCI/TOCHI are near-pure venues."),
92 +),
93 +dict(
94 + id="computer_graphics", number=35, name="Computer Graphics", area="perception_interaction",
95 + definition=("Synthesis and manipulation of visual content from models: rendering, geometry processing and modelling, "
96 + "animation and simulation for graphics, neural rendering and 3D representations, visualisation, "
97 + "procedural content, game and VR engines. Image understanding is computer_vision; pixel-space generative "
98 + "models are generative_ai; user studies of VR are hci."),
99 + topics=[("T10481", 1.0, "computer graphics and visualization techniques"), ("T12290", 1.0, "human motion and animation"),
100 + ("T10996", 0.8, "computational geometry and mesh generation"), ("T10799", 0.8, "data visualization and analytics"),
101 + ("T10719", 0.8, "3D shape modeling and analysis"), ("T14339", 0.4, "image processing and 3D reconstruction"),
102 + ("T10789", 0.3, "interactive and immersive displays"), ("T10888", 0.3, "augmented reality"),
103 + ("T10648", 0.3, "virtual reality"), ("T13518", 0.3, "architecture and computational design"),
104 + ("T12698", 0.3, "3D modeling in geospatial applications"), ("T14254", 0.3, "digital media and visual art"),
105 + ("T11897", 0.2, "digital holography and microscopy"), ("T11211", 0.2, "3D surveying and cultural heritage"),
106 + ("T10775", 0.2, "image synthesis")],
107 + subfields=[(1704, 0.6)],
108 + arxiv=[("cs.GR", 1.0), ("cs.CG", 0.5), ("cs.CV", 0.1), ("cs.HC", 0.1)],
109 + streams=[("journals/tog", 1.0), ("journals/cgf", 1.0), ("conf/siggraph", 1.0), ("conf/siggrapha", 1.0),
110 + ("journals/cg", 1.0), ("journals/vc", 1.0), ("conf/pg", 1.0), ("conf/visualization", 1.0),
111 + ("journals/tvcg", 0.8), ("conf/3dim", 0.5), ("conf/cccg", 0.4), ("journals/dcg", 0.3), ("conf/vr", 0.3),
112 + ("conf/ismar", 0.3), ("journals/jei", 0.2), ("journals/vr", 0.2), ("conf/cvpr", 0.1), ("journals/mta", 0.1),
113 + ("journals/entcom", 0.3), ("conf/cig", 0.3), ("journals/cacie", 0.1)],
114 + keywords=[
115 + (r"(?i)\bcomputer graphics\b|\bgraphics (hardware|pipeline|processing|applications?)\b|\bSIGGRAPH\b", 1.0, "both"),
116 + (r"(?i)\b(rendering|ray[- ]tracing|path[- ]tracing|rasteri[sz]ation|shading|shaders?|global illumination|radiance|radiosity|BRDFs?|BSDFs?|photo-?realistic|real-?time rendering|differentiable rendering|neural rendering|inverse rendering|Monte Carlo rendering|light transport|participating media|subsurface scattering|anti-?aliasing|level of detail|LOD\b|texture (synthesis|mapping|compression)|image-?based (rendering|lighting)|HDR (imaging|rendering)|tone mapping|appearance (modeling|capture))\b", 0.9, "both"),
117 + (r"(?i)\b(mesh(es|ing)?|geometry processing|subdivision surfaces?|NURBS|B-?splines?|signed distance (fields?|functions?)|SDFs?\b|implicit surfaces?|tessellation|remeshing|surface (parameteri[sz]ation|reconstruction)|point[- ]set surfaces|geometric modeling|solid modeling|CAD (models?|modeling)|boundary representation|B-?rep|constructive solid|isosurface|marching cubes|Laplacian (smoothing|editing)|discrete differential geometry|as-?rigid-?as-?possible|shape (analysis|matching|correspondence|deformation|interpolation|modeling|grammars?))\b", 0.8, "both"),
118 + (r"(?i)\b(character animation|skeletal animation|motion capture|mocap|motion (synthesis|retargeting|editing|matching)|physics-?based (animation|character)|cloth simulation|hair simulation|fluid (simulation|animation) (for|in) graphics|position-?based dynamics|material point method|skinning|rigging|inverse kinematics for animation|facial animation|crowd simulation|procedural animation|keyframe)\b", 1.0, "both"),
119 + (r"(?i)\b(3D (modeling|modelling|shapes?|content|assets?|scenes? (generation|synthesis)|Gaussians?|avatars?)|NeRFs?|neural radiance fields?|Gaussian splatting|3DGS|volume rendering|volumetric (rendering|video|capture)|neural (implicit|fields?|scene representations?)|novel view synthesis|view synthesis|free-?viewpoint|light fields?|digital humans?|avatars?|virtual (humans|characters|try-?on))\b", 0.8, "both"),
120 + (r"(?i)\b(scientific|information|data|volume|flow|medical|immersive) visuali[sz]ation\b|\bvisual analytics\b|\bvisuali[sz]ation (techniques?|systems?|design|tools?|research|pipeline)\b|\bInfoVis\b|\bSciVis\b|\bvisual encodings?\b|\bglyphs?\b|\btreemaps?\b|\bparallel coordinates\b", 0.8, "both"),
121 + (r"(?i)\b(procedural (generation|modeling|content|textures?)|game engines?|Unity|Unreal Engine|GPU (shaders?|rendering|ray)|Vulkan|OpenGL|DirectX|WebGL|WebGPU|real-?time graphics|interactive graphics|graphics APIs?)\b", 0.7, "both"),
122 + (r"(?i)\b(computational geometry|Voronoi|Delaunay|convex hull|triangulations?|mesh generation|geometric (algorithms?|computing)|computational (design|fabrication)|3D printing (design|geometry)|digital fabrication|architectural geometry)\b", 0.6, "both"),
123 + (r"(?i)\b(3D|geometry|geometric|shapes?|surfaces?|animation|visual)\b", 0.2, "both"),
124 + ],
125 + exclusions=[],
126 + related=["computer_vision", "hci", "generative_ai", "scientific_computing", "algorithms"],
127 + parent_of=[], child_of=[],
128 + provenance=("All 4 topics of OpenAlex subfield 1704 mapped (T13615 transportation safety is the unmapped one), plus "
129 + "animation (Engineering 2207) and visualization (1707). TOG/CGF/SIGGRAPH are near-pure; TVCG shared with HCI."),
130 +),
131 +]
added taxonomy/src/area_security_a.py +89 −0
@@ -0,0 +1,89 @@
1 +"""Area security_privacy, part A: categories 26-27 (cybersecurity, applied cryptography)."""
2 +
3 +CATEGORIES = [
4 +dict(
5 + id="cybersecurity", number=26, name="Cybersecurity", area="security_privacy",
6 + definition=("Protecting computing systems, networks, software and users from attack: vulnerability discovery and "
7 + "exploitation, malware and intrusion detection, authentication and access control, system and hardware "
8 + "security, web and mobile security, digital forensics, security of AI systems, security operations and "
9 + "human factors of security. Cryptographic primitives and protocols are applied_cryptography; data-protection "
10 + "techniques are privacy_preserving_computation."),
11 + topics=[("T10400", 1.0, "network security and intrusion detection"), ("T10734", 1.0, "information and cyber security"),
12 + ("T13983", 1.0, "cybersecurity and information systems"), ("T11241", 1.0, "advanced malware detection"),
13 + ("T11644", 1.0, "spam and phishing detection"), ("T12479", 1.0, "web application security vulnerabilities"),
14 + ("T12034", 1.0, "digital and cyber forensics"), ("T11800", 0.9, "user authentication and security systems"),
15 + ("T11424", 0.8, "security and verification in computing (TEEs, OS security)"), ("T12519", 0.8, "cybercrime and law enforcement"),
16 + ("T12122", 0.7, "PUFs and hardware security"), ("T11498", 0.7, "security in wireless sensor networks"),
17 + ("T12221", 0.7, "cybersecurity and cyber warfare studies"), ("T11504", 0.6, "advanced authentication protocols"),
18 + ("T11614", 0.6, "cloud data security"), ("T10388", 0.6, "steganography and watermarking"),
19 + ("T10917", 0.6, "smart grid security and resilience"), ("T10964", 0.6, "wireless communication security"),
20 + ("T13999", 0.6, "digital rights management"), ("T10828", 0.6, "biometric identification and security"),
21 + ("T10927", 0.6, "access control and trust"), ("T10237", 0.5, "cryptography and data security"),
22 + ("T12357", 0.5, "digital media forensic detection"), ("T14042", 0.5, "technology and security systems"),
23 + ("T11689", 0.4, "adversarial robustness (ML security)"), ("T11045", 0.4, "privacy, security and data protection"),
24 + ("T11598", 0.4, "Internet traffic analysis and secure e-voting"), ("T10951", 0.4, "cryptographic implementations (side channels)"),
25 + ("T11017", 0.3, "chaos-based image encryption"), ("T12943", 0.3, "COVID-19 contact tracing"),
26 + ("T13270", 0.2, "QR code applications"), ("T10270", 0.2, "blockchain security"),
27 + ("T12131", 0.2, "wireless signal modulation classification (spectrum security)")],
28 + subfields=[],
29 + arxiv=[("cs.CR", 1.0)],
30 + streams=[("conf/ccs", 1.0), ("conf/uss", 1.0), ("conf/sp", 1.0), ("conf/ndss", 1.0), ("conf/esorics", 1.0), ("conf/raid", 1.0),
31 + ("conf/acsac", 1.0), ("journals/tdsc", 1.0), ("journals/compsec", 1.0), ("journals/istr", 1.0), ("journals/ijisec", 1.0),
32 + ("journals/scn", 1.0), ("conf/IEEEares", 1.0), ("conf/eurosp", 1.0), ("journals/jcs", 1.0), ("conf/pst", 1.0),
33 + ("journals/tifs", 0.9), ("conf/trustcom", 0.9), ("journals/dud", 0.8), ("conf/icccsec", 0.6), ("conf/dsn", 0.5),
34 + ("journals/tches", 0.5), ("conf/fc", 0.5), ("journals/popets", 0.4), ("conf/satml", 0.4), ("journals/iacr", 0.3),
35 + ("journals/cr", 0.2), ("journals/tnsm", 0.1), ("journals/tr", 0.1)],
36 + keywords=[
37 + (r"(?i)\b(cyber-?security|information security|computer security|network security|systems? security|software security|web security|hardware security|mobile security|IoT security|cloud security|application security|security (vulnerabilit(y|ies)|analysis|evaluation|assessment|threats?|attacks?|flaws?|breaches?|incidents?|monitoring|operations?|awareness|polic(y|ies)|engineering|testing|protocols?|mechanisms?|guarantees|properties|controls?|risks?|requirements|patches?|advisories|researchers?|practitioners?|tools?|audits?|hardening|posture)|secure (systems?|software|coding|boot|enclaves?|hardware|communication|channels?|protocols?|computation|messaging|design|by design|aggregation|updates?|storage|execution))\b", 1.0, "both"),
38 + (r"(?i)\b(malware|ransomware|botnets?|phishing|spam (detection|filtering)|intrusion (detection|prevention|response)|IDS\b|IPS\b|NIDS|HIDS|DDoS|denial[- ]of[- ]service|exploits?|exploitation|zero-?day|vulnerabilit(y|ies)|CVE-?\d+|CWE\b|buffer overflows?|memory (safety|corruption|errors?) (bugs?|vulnerabilit)|use-?after-?free|ROP\b|return-?oriented|control-?flow (integrity|hijack)|CFI\b|side-?channel|Spectre|Meltdown|Rowhammer|cache (timing|attacks?)|fault injection attacks?|glitching|penetration testing|pentest|red team|blue team|threat (intelligence|hunting|model(ing|s)?|actors?|landscape)|attack (surface|graphs?|vectors?|trees?|scenarios?|detection|campaigns?)|APTs?\b|advanced persistent threats?|cyber-?attacks?|cyber ?threats?|cyber ?defen[cs]e|cyber ?warfare|cyber ?crime|cyber ?resilience|cyber ?range|forensics?|honeypots?|honeynets?|sandbox(ing)? (analysis|evasion)|binary (analysis|rewriting|hardening|lifting)|reverse engineering|firmware (analysis|security|rehosting)|supply[- ]chain (attacks?|security)|SBOM|software bill of materials|dependency confusion|typosquatting|code injection|SQL injection|XSS\b|cross-?site|CSRF|clickjacking|browser (security|fingerprinting)|web tracking|ad fraud|click fraud|credential (stuffing|theft)|password (cracking|managers?|policies|strength)|account (takeover|hijacking)|SIM swap|social engineering|security (information and event|operations center)|SIEM|SOC\b|EDR\b|XDR\b|log-?based detection|anomaly-?based detection|signature-?based|evasion attacks?|obfuscation|packers?|deobfuscation|anti-?virus|endpoint (protection|security)|patch(ing)? (management|latency)|bug bount(y|ies)|responsible disclosure|CTF\b|capture the flag|ethical hacking|vulnerability (scanning|assessment|management|disclosure|discovery|detection|prediction)|exploit (generation|mitigation)|ASLR|stack canaries|secure (compilation|compilers?)|moving target defense|deception technology|cyber (insurance|risk)|ICS security|SCADA security|OT security|automotive security|CAN bus (attacks?|intrusion)|drone (security|hijacking)|GPS spoofing|medical device security|smart contract (vulnerabilit|exploits?|attacks?)|DeFi (attacks?|exploits?)|rug pull|flash loan attacks?)\b", 1.0, "both"),
39 + (r"(?i)\b(authentication|access control|authori[sz]ation|identity (management|and access)|biometric (authentication|security|spoofing)|password(s|-based)?|multi-?factor|two-?factor|2FA|MFA\b|CAPTCHA|single sign-?on|SSO\b|OAuth|OpenID|FIDO2?|WebAuthn|passkeys?|zero[- ]trust|trusted execution environments?|TEEs?\b|SGX\b|TrustZone|SEV\b|TDX\b|confidential computing|remote attestation|PUFs?\b|physical(ly)? unclonable|hardware (trojans?|root of trust|security modules?)|HSMs?\b|TPM\b|secure (boot|element)|anti-?tamper|logic locking|IC (piracy|counterfeit)|RBAC|ABAC|capability-?based security|information flow (control|security)|non-?interference|sandboxing|isolation (mechanisms|boundaries)|privilege (escalation|separation)|least privilege|Android (permissions?|malware|security)|iOS security|app (permissions?|vetting|stores? security)|mobile (malware|banking trojans?))\b", 0.8, "both"),
40 + (r"(?i)\b(adversarial (attacks?|examples?|machine learning|ML)|backdoor(s|ed)?|trojan(ed|ing)? (models?|neural)|data poisoning|model (stealing|inversion|extraction|hijacking)|membership inference|prompt injection|jailbreak(s|ing)?|LLM (security|attacks?)|AI security|security of (LLMs?|AI|machine learning|ML models?|deep learning)|ML-?based (intrusion|malware|anomaly) detection|deep learning for (security|malware|intrusion)|adversarial malware|evasion of (classifiers|detectors)|deepfake (detection|attacks?)|AI-?generated (phishing|malware)|LLM-?generated attacks?|agent (security|hijacking)|MCP security|tool poisoning)\b", 0.8, "both"),
41 + (r"(?i)\b(steganograph(y|ic)|steganalysis|watermark(ing|s)?|digital forensics?|memory forensics|disk forensics|network forensics|mobile forensics|anti-?forensics?|tamper(ing)? detection|image (forgery|manipulation) detection|provenance of media|C2PA|content authenticity|e-?voting security|election security|secure voting)\b", 0.7, "both"),
42 + (r"(?i)\b(security|secure|privacy|attack(s|ers?)?|adversar(y|ies|ial)|threats?|vulnerab\w+|malicious|exploit\w*|defen[cs]es?|mitigations?|detection of (attacks|intrusions)|trust(ed|worthy)?)\b", 0.3, "both"),
43 + ],
44 + exclusions=[r"(?is)^(?!.*(cyber|computer|network|software|information security|system security|attack|malware|vulnerab|intrusion|authentication|cryptograph|encryption|access control|threat model)).*\b(food|energy|water|national|social|border|maritime|job|economic|financial|human|global|regional|international|homeland|tenure|livelihood|nutritional|health|supply|income|employment|housing|seed|climate|environmental|ontological|emotional|attachment|collective|state|public|private|military|defence|defense) security\b"],
45 + related=["applied_cryptography", "privacy_preserving_computation", "computer_networks", "ai_safety", "blockchain", "formal_verification", "operating_systems", "software_engineering"],
46 + parent_of=[], child_of=[],
47 + provenance=("33 OpenAlex topics mapped: security topics of 1702, 1705, 1710, 1711, 1708 plus Social Sciences cyber-warfare, "
48 + "access control and data-protection topics at reduced weight. CCS/USENIX Security/S&P/NDSS/ESORICS/RAID/ACSAC/TDSC/"
49 + "Computers & Security pure; TIFS 0.9 (also media forensics); IACR ePrint 0.3 (mostly cryptography). The 1.0 "
50 + "keyword patterns require a computing qualifier; bare 'security' is 0.3 and non-computing senses are vetoed when no "
51 + "computing term co-occurs."),
52 +),
53 +dict(
54 + id="applied_cryptography", number=27, name="Applied Cryptography", area="security_privacy",
55 + definition=("Design, analysis, implementation and deployment of cryptographic primitives and protocols: symmetric and "
56 + "public-key schemes, hash functions, signatures, key exchange, post-quantum cryptography, zero-knowledge proofs, "
57 + "secure multi-party computation and homomorphic encryption as primitives, cryptanalysis, side-channel-resistant "
58 + "implementations, cryptographic hardware, provable security, quantum key distribution. Deployment of these "
59 + "primitives for data protection is privacy_preserving_computation; ledgers are blockchain."),
60 + topics=[("T10237", 1.0, "cryptography and data security"), ("T10951", 1.0, "cryptographic implementations and security"),
61 + ("T11693", 1.0, "cryptography and residue arithmetic"), ("T11130", 0.9, "coding theory and cryptography"),
62 + ("T11017", 0.7, "chaos-based image/signal encryption"), ("T10020", 0.5, "quantum information and cryptography"),
63 + ("T11504", 0.5, "advanced authentication protocols"), ("T11614", 0.4, "cloud data security (encrypted outsourcing)"),
64 + ("T12122", 0.4, "PUFs and hardware security"), ("T10764", 0.3, "privacy-preserving technologies (SMPC, HE)"),
65 + ("T10270", 0.3, "blockchain technology"), ("T11800", 0.3, "user authentication"),
66 + ("T11321", 0.2, "error-correcting codes"), ("T11598", 0.2, "secure e-voting")],
67 + subfields=[],
68 + arxiv=[("cs.CR", 0.5), ("cs.IT", 0.2), ("math.IT", 0.2), ("quant-ph", 0.1)],
69 + streams=[("journals/iacr", 1.0), ("conf/crypto", 1.0), ("conf/eurocrypt", 1.0), ("conf/asiacrypt", 1.0), ("conf/tcc", 1.0),
70 + ("conf/pkc", 1.0), ("journals/joc", 1.0), ("journals/tches", 1.0), ("journals/dcc", 1.0), ("conf/fc", 0.6),
71 + ("conf/ccs", 0.4), ("conf/esorics", 0.4), ("conf/uss", 0.3), ("conf/sp", 0.3), ("conf/ndss", 0.3), ("journals/tifs", 0.3),
72 + ("journals/ffa", 0.3), ("journals/ijisec", 0.3), ("journals/scn", 0.3), ("journals/istr", 0.3), ("journals/tit", 0.2),
73 + ("journals/qip", 0.2), ("journals/compsec", 0.2), ("journals/cr", 0.1), ("conf/isit", 0.1), ("conf/eurosp", 0.3),
74 + ("journals/tdsc", 0.3), ("conf/IEEEares", 0.2), ("conf/trustcom", 0.2)],
75 + keywords=[
76 + (r"(?i)\bcryptograph(y|ic|ically|ers?)\b|\bcryptosystems?\b|\bcryptanaly(sis|tic|ses|ze|se)\b|\bcrypto(graphic)? (primitives?|schemes?|protocols?|libraries|implementations?|hardware|agility|standards?)\b", 1.0, "both"),
77 + (r"(?i)\b(encryption|decryption|encrypted|ciphers?|ciphertexts?|plaintexts?|digital signatures?|signature schemes?|public[- ]key|private[- ]key|secret[- ]key|symmetric[- ]key|asymmetric (encryption|cryptograph)|key (exchange|agreement|encapsulation|management|derivation|generation|distribution|recovery|schedule)|hash functions?|SHA-?(2|3|256|512)|BLAKE|AES\b|RSA\b|ECC\b|elliptic[- ]curves?|ECDSA|EdDSA|Ed25519|Curve25519|Diffie-?Hellman|discrete logarithm|lattice-?based|LWE\b|learning with errors|ring-?LWE|module-?LWE|NTRU|code-?based|McEliece|hash-?based signatures?|multivariate cryptograph|isogen(y|ies)|SIKE|CSIDH|post-?quantum|quantum-?resistant|quantum-?safe|PQC\b|Kyber|ML-?KEM|Dilithium|ML-?DSA|Falcon|SPHINCS|homomorphic|FHE\b|HE schemes?|zero-?knowledge|ZKPs?\b|zk-?SNARKs?|zk-?STARKs?|Bulletproofs|Groth16|PLONK|succinct (arguments?|proofs?)|secure multi-?party|MPC\b|secret sharing|Shamir|oblivious (transfer|RAM|PRF)|garbled circuits?|private set intersection|PSI\b|commitment schemes?|identity-?based encryption|IBE\b|attribute-?based encryption|ABE\b|functional encryption|searchable encryption|proxy re-?encryption|authenticated encryption|AEAD|GCM\b|ChaCha20|Poly1305|MACs?\b|message authentication codes?|HMAC|block ciphers?|stream ciphers?|S-?boxes?|Feistel|substitution-?permutation|differential cryptanalysis|linear cryptanalysis|algebraic attacks?|meet-?in-?the-?middle|side-?channel (resistant|resistance|countermeasures?|attacks? on)|masking (schemes?|countermeasures?)|constant-?time|timing attacks? on|power analysis|DPA\b|CPA\b|electromagnetic analysis|fault attacks?|random oracle|standard model|provable security|security (proofs?|reductions?)|IND-?CCA|IND-?CPA|EUF-?CMA|UC (framework|security)|universal composability|cryptographic (protocols?|hardware|accelerators?)|TLS (1\.3)?|SSL\b|Signal protocol|double ratchet|end-?to-?end encrypt(ed|ion)|E2EE|secure messaging|password-?authenticated|PAKE|threshold (signatures?|cryptography|ECDSA|encryption)|multi-?signatures?|blind signatures?|ring signatures?|group signatures?|verifiable (random functions?|delay functions?|computation|secret sharing|encryption)|VRFs?\b|VDFs?\b|PRFs?\b|pseudorandom (functions?|generators?|permutations?|number generators?)|TRNG|true random number|entropy source|key (escrow|rotation)|certificates?|PKI\b|certificate (transparency|authorities)|X\.509|HSM|white-?box cryptograph|lightweight cryptograph|ASCON|PRESENT cipher|SIMON|SPECK|cryptocurrency (wallets?|signatures?)|Schnorr|BLS signatures?|pairing-?based|bilinear pairings?|supersingular|format-?preserving|order-?preserving|deterministic encryption|tokeni[sz]ation of data)\b", 1.0, "both"),
78 + (r"(?i)\b(quantum key distribution|QKD\b|quantum cryptograph\w*|BB84|E91|quantum random number|quantum-?secure|device-?independent|quantum (money|signatures?|commitment|oblivious))\b", 0.8, "both"),
79 + (r"(?i)\b(finite fields?|Galois fields?|GF\(2\^|number[- ]theoretic|modular (arithmetic|exponentiation|multiplication|reduction)|Montgomery (multiplication|ladder|reduction)|residue number systems?|RNS\b|polynomial multiplication (for|in) (lattice|crypto|NTT)|NTT\b|number theoretic transform|Boolean functions? (nonlinearity|bent)|bent functions?|S-?box (design|construction)|coding theory|linear codes?|cyclic codes?|BCH|Reed-?Solomon|MDS codes?|secret sharing schemes?)\b", 0.5, "both"),
80 + (r"(?i)\b(chaotic|chaos-?based|hyper-?chaotic|DNA-?based) (image |signal |video |audio |data )?encryption\b|\bimage encryption\b|\bimage (scrambling|ciphering)\b", 1.0, "both"),
81 + (r"(?i)\b(encrypt\w*|decrypt\w*|keys?|signatures?|ciphers?|hashing|hashes|secrecy|confidentiality|integrity)\b", 0.2, "both"),
82 + ],
83 + exclusions=[],
84 + related=["cybersecurity", "privacy_preserving_computation", "blockchain", "quantum_computing", "computational_complexity", "computer_architecture"],
85 + parent_of=[], child_of=[],
86 + provenance=("T10237/T10951/T11693/T11130 are the OpenAlex cryptography hosts; chaos-based image encryption (T11017) is a large "
87 + "applied sub-literature. IACR ePrint, CRYPTO/EUROCRYPT/ASIACRYPT/TCC/PKC, JoC, TCHES and DCC are pure venues."),
88 +),
89 +]
added taxonomy/src/area_security_b.py +95 −0
@@ -0,0 +1,95 @@
1 +"""Area security_privacy, part B: categories 28, 36, 37 (blockchain, privacy-preserving computation, federated learning)."""
2 +
3 +CATEGORIES = [
4 +dict(
5 + id="blockchain", number=28, name="Blockchain Systems", area="security_privacy",
6 + definition=("Distributed ledgers and the systems built on them: consensus for permissioned and permissionless chains, "
7 + "smart contracts and their security, scalability (sharding, layer-2, rollups), cryptocurrencies and DeFi "
8 + "mechanisms, blockchain applications and governance, measurement of blockchain ecosystems. Generic BFT "
9 + "consensus theory is shared with distributed_computing; primitives are applied_cryptography."),
10 + topics=[("T10270", 1.0, "blockchain technology applications and security"), ("T13913", 0.8, "blockchain in education and learning"),
11 + ("T10742", 0.2, "peer-to-peer networks"), ("T10772", 0.2, "distributed systems and fault tolerance (BFT)"),
12 + ("T11598", 0.2, "secure e-voting"), ("T10764", 0.1, "privacy-preserving technologies")],
13 + subfields=[],
14 + arxiv=[("cs.CR", 0.2), ("cs.DC", 0.2), ("cs.GT", 0.1)],
15 + streams=[("conf/blockchain", 1.0), ("conf/icbc", 1.0), ("conf/aft", 1.0), ("conf/fc", 0.7), ("conf/bpm", 0.3),
16 + ("journals/jtaer", 0.2), ("journals/fi", 0.2), ("journals/ppna", 0.2), ("conf/ccs", 0.1), ("journals/tifs", 0.1),
17 + ("journals/tdsc", 0.1), ("journals/tnsm", 0.1), ("journals/tsc", 0.1), ("journals/tcss", 0.1), ("journals/compsec", 0.1),
18 + ("journals/itiis", 0.1), ("journals/iotj", 0.05), ("journals/tnse", 0.1), ("conf/trustcom", 0.2)],
19 + keywords=[
20 + (r"(?i)\bblock-?chains?\b|\bblockchain-?(based|enabled|powered)\b", 1.0, "both"),
21 + (r"(?i)\b(smart contracts?|distributed ledgers?|DLT\b|crypto-?currenc(y|ies)|Bitcoin|Ethereum|Solidity|EVM\b|Ethereum Virtual Machine|Hyperledger( Fabric)?|decentrali[sz]ed finance|DeFi\b|NFTs?\b|non-?fungible tokens?|proof[- ]of[- ](work|stake|authority|space|elapsed time|history|replication)|PoW\b|PoS\b|mining pools?|miners?|sharding (in|for|of) blockchain|layer[- ]2|rollups?|optimistic rollup|payment channels?|Lightning Network|cross-?chain|atomic swaps?|bridges? (between|across) (chains|blockchains)|decentrali[sz]ed (applications?|exchanges?|autonomous organi[sz]ations?|identity|storage networks?)|dApps?\b|DAOs?\b|Web3|tokenomics|token (economics|design)|MEV\b|maximal extractable value|front-?running|stablecoins?|central bank digital currenc|CBDCs?|zk-?rollups?|consensus (mechanisms?|protocols?|algorithms?) (for|in|of) (blockchain|permissioned|permissionless|ledgers?)|double-?spending|51% attacks?|selfish mining|transaction fees? (market|mechanism)|EIP-?1559|gas (fees?|optimi[sz]ation|costs?)|Merkle (trees?|proofs?) for ledgers?|UTXO|wallets?|private keys? management for|oracles? (problem|for blockchain|networks?)|Chainlink|IPFS|Filecoin|Solana|Cardano|Polkadot|Cosmos SDK|Tendermint|Avalanche consensus|Algorand|Libra|Diem|Monero|Zcash|privacy coins?)\b", 1.0, "both"),
22 + (r"(?i)\b(Byzantine (fault[- ]toleran(t|ce)|agreement|consensus|generals)|BFT\b|PBFT|HotStuff|Nakamoto consensus|longest-?chain|GHOST protocol|DAG-?based (consensus|ledgers?)|hashgraph|IOTA|Tangle)\b", 0.8, "both"),
23 + (r"(?i)\b(decentrali[sz]ed|decentrali[sz]ation|immutable (ledgers?|records?)|tamper-?proof|tamper-?evident|trustless|permissioned|permissionless|ledgers?|tokens?|crypto(-?assets?)?)\b", 0.4, "both"),
24 + ],
25 + exclusions=[r"(?is)^(?!.*(blockchain|ledger|smart contract|cryptocurrenc|Bitcoin|Ethereum|consensus|decentrali)).*\b(block chains? of|blocks? of (amino|residues|carbon|polymers?))\b"],
26 + related=["applied_cryptography", "distributed_computing", "cybersecurity", "privacy_preserving_computation", "computer_networks"],
27 + parent_of=[], child_of=[],
28 + provenance=("T10270 (168k works) is the OpenAlex blockchain topic and is sufficient alone; T13913 covers educational "
29 + "applications. BLOCKCHAIN/ICBC/AFT pure, FC 0.7; 'blockchain' is an unambiguous 1.0 keyword."),
30 +),
31 +dict(
32 + id="privacy_preserving_computation", number=36, name="Privacy-Preserving Computation", area="security_privacy",
33 + definition=("Computing on, releasing or sharing data while limiting what is learned about individuals: differential "
34 + "privacy, homomorphic encryption, secure multi-party computation and TEEs applied to data analysis and ML, "
35 + "anonymisation and synthetic data, private information retrieval, privacy attacks on models and their "
36 + "defences, machine unlearning, privacy engineering and regulation (GDPR). Federated learning is a child; "
37 + "cryptographic constructions as such are applied_cryptography."),
38 + topics=[("T10764", 1.0, "privacy-preserving technologies in data (federated learning, differential privacy, SMPC, k-anonymity)"),
39 + ("T11045", 0.6, "privacy, security and data protection"), ("T12943", 0.5, "COVID-19 digital contact tracing"),
40 + ("T11614", 0.4, "cloud data security (outsourced encrypted data)"), ("T10237", 0.3, "cryptography and data security"),
41 + ("T12921", 0.3, "European criminal justice and data protection"), ("T10927", 0.3, "access control and trust"),
42 + ("T11800", 0.2, "user authentication"), ("T13999", 0.2, "digital rights management"),
43 + ("T11693", 0.2, "cryptography and residue arithmetic (HE arithmetic)"), ("T11598", 0.2, "secure e-voting"),
44 + ("T10828", 0.2, "biometric identification (template protection)"), ("T11689", 0.2, "adversarial ML (membership inference)")],
45 + subfields=[],
46 + arxiv=[("cs.CR", 0.4), ("cs.LG", 0.1), ("cs.DB", 0.1), ("cs.CY", 0.1)],
47 + streams=[("journals/popets", 1.0), ("conf/pst", 0.6), ("journals/dud", 0.5), ("conf/satml", 0.4), ("conf/trustcom", 0.4),
48 + ("conf/ccs", 0.3), ("conf/uss", 0.3), ("conf/sp", 0.3), ("conf/ndss", 0.3), ("journals/tdsc", 0.3), ("conf/esorics", 0.3),
49 + ("conf/IEEEares", 0.3), ("journals/tifs", 0.2), ("journals/iacr", 0.2), ("journals/compsec", 0.2), ("journals/istr", 0.2),
50 + ("journals/ijisec", 0.2), ("conf/fat", 0.2), ("journals/tkde", 0.1), ("conf/sigmod", 0.1), ("journals/pvldb", 0.1),
51 + ("journals/cr", 0.2), ("conf/nips", 0.05), ("conf/icml", 0.05), ("journals/jamia", 0.1), ("journals/titb", 0.1)],
52 + keywords=[
53 + (r"(?i)\bprivacy-?preserving\b|\bprivacy-?enhancing\b|\bprivacy-?enhanced\b|\bPETs?\b|\bprivacy-?aware\b|\bprivacy-?friendly\b|\bprivacy-?protecting\b|\bprivacy-?first\b|\bprivate (computation|inference|training|learning|information retrieval|set intersection|aggregation|analytics|statistics|record linkage|data (analysis|release|sharing|publishing)|queries|counting|mean estimation|synthetic)\b", 1.0, "both"),
54 + (r"(?i)\b(differential(ly)?[- ]privac(y|ate)|DP-?SGD|DP-?FTRL|local differential privacy|LDP\b|central DP|shuffle (model|DP)|Rényi differential|Gaussian mechanism|Laplace mechanism|exponential mechanism|privacy (budget|amplification|accounting|accountant|loss|guarantees?|leakage|attacks?|risks?|threats?|auditing|audit|breaches?|violations?|harms?|threat models?|implications|preserving|protection|mechanisms?|filters?|definitions?)|\(ε|epsilon-?delta|ε-?differential|moments accountant)\b", 1.0, "both"),
55 + (r"(?i)\b(homomorphic encryption|fully homomorphic|somewhat homomorphic|FHE\b|HE-?based|CKKS|BFV\b|TFHE|SEAL library|secure multi-?party computation|secure (two|multi)-?party|MPC-?based|MPC protocols? for (machine learning|inference|training|data)|secret sharing(-| )based (computation|learning)|garbled circuits? for|oblivious (transfer|RAM|inference|shuffling)|ORAM|trusted execution environments? for (privacy|private)|SGX-?based (privacy|private)|private information retrieval|PIR\b|zero-?knowledge proofs? (for|of) (machine learning|inference|training|data)|zkML|k-?anonymity|l-?diversity|t-?closeness|anonymi[sz](ation|ed|ing)|pseudonymi[sz]|de-?anonymi[sz]ation|re-?identification (risks?|attacks?)|linkage attacks?|synthetic data (generation|release) (for|with) privacy|privacy-?preserving (machine learning|deep learning|data mining|data publishing|record linkage|federated|inference|training|analytics|blockchain|computation|aggregation|statistics|queries|authentication|recommendation|biometrics|smart grid|IoT|cloud|genomics)|PPML\b|PPDM\b|PPRL\b|membership inference|model inversion|attribute inference|property inference|gradient (leakage|inversion)|reconstruction attacks? (on|against) (models?|training data)|training data (extraction|leakage|memori[sz]ation)|data (leakage|minimi[sz]ation|protection by design)|machine unlearning|certified (unlearning|removal)|right to be forgotten|right to erasure|GDPR|CCPA|HIPAA (compliance|privacy)|data protection (regulation|laws?|impact assessment|authorities)|DPIA|consent management|cookie (consent|banners?)|privacy (polic(y|ies)|by design|engineering|compliance|regulation|laws?|labels?|nutrition labels?|dashboards?|settings|preferences|controls?)|purpose limitation|data (brokers?|sovereignty|trusts?)|personal data (processing|protection)|PII\b|personally identifiable)\b", 1.0, "both"),
56 + (r"(?i)\b(location privacy|trajectory privacy|genomic privacy|health data privacy|user privacy|data privacy|information privacy|individual privacy|privacy concerns?|privacy risks?|privacy (threat|attack) models?|privacy-?utility (trade-?off|tradeoff)|privacy (of|in|for) (users|individuals|patients|children|IoT|smart|mobile|location|voice|AR|VR|LLMs?|language models?|federated|blockchain|recommender|social)|inference attacks?|side-?channel (leakage|privacy)|website fingerprinting|traffic analysis attacks?|Tor\b|onion routing|anonymous (communication|credentials|networks?|messaging)|mix-?nets?|metadata privacy|unlinkability|unobservability|plausible deniability|contact tracing (apps?|privacy|protocols?)|DP-?3T|exposure notification)\b", 0.8, "both"),
57 + (r"(?i)\bprivacy\b|\bprivate\b|\bconfidential(ity)?\b|\banonymous\b|\bsensitive (data|attributes?|information)\b", 0.4, "both"),
58 + ],
59 + exclusions=[r"(?is)^(?!.*(comput|data|algorithm|protocol|model|system|encrypt|differential|anonym|learning|inference|database|app|online|digital|users?)).*\b(privacy (in|of) (nursing|hospital rooms?|the home|toilets?|bedrooms?|bathrooms?)|private (practice|sector|hospitals?|schools?|equity|insurance|property|lands?|forests?|firms?|companies|banks?|clinics?|households?))\b"],
60 + related=["applied_cryptography", "cybersecurity", "ai_safety", "database_systems", "blockchain", "federated_learning", "ml_theory"],
61 + parent_of=["federated_learning"], child_of=[],
62 + provenance=("T10764 (97k works) keywords list federated learning, differential privacy, SMPC, k-anonymity: it is the host of both "
63 + "this category and its child. PoPETs is the only pure venue; security venues partial. Non-computing senses of "
64 + "'private/privacy' are vetoed when no computing term co-occurs."),
65 +),
66 +dict(
67 + id="federated_learning", number=37, name="Federated Learning", area="security_privacy",
68 + definition=("Training models across multiple clients or institutions without centralising their raw data: federated "
69 + "averaging and optimisation under heterogeneity, communication efficiency, client selection, personalisation, "
70 + "secure aggregation and privacy of federated systems, robustness to malicious clients, vertical and "
71 + "cross-silo federated learning, federated fine-tuning of foundation models, decentralised/gossip learning. "
72 + "A child of privacy_preserving_computation; also linked to distributed_computing and edge_computing."),
73 + topics=[("T10764", 0.7, "privacy-preserving technologies (federated learning, non-IID data, communication efficiency)"),
74 + ("T10273", 0.1, "IoT and edge computing (federated learning at the edge)"),
75 + ("T11307", 0.1, "domain adaptation / personalisation"), ("T11689", 0.1, "adversarial robustness (Byzantine clients)")],
76 + subfields=[],
77 + arxiv=[("cs.LG", 0.2), ("cs.DC", 0.2), ("cs.CR", 0.1), ("cs.NI", 0.1)],
78 + streams=[("journals/tmc", 0.1), ("journals/iotj", 0.1), ("journals/tpds", 0.1), ("journals/tifs", 0.05), ("conf/satml", 0.1),
79 + ("journals/tnse", 0.1), ("journals/tnsm", 0.05), ("journals/twc", 0.05), ("journals/jsac", 0.05), ("conf/infocom", 0.05),
80 + ("journals/tnn", 0.05), ("journals/tcc", 0.05), ("conf/nips", 0.02), ("conf/icml", 0.02), ("conf/iclr", 0.02)],
81 + keywords=[
82 + (r"(?i)\bfederated (learning|optimi[sz]ation|training|averaging|analytics|transfer learning|reinforcement learning|graph (learning|neural)|fine-?tuning|LLMs?|foundation models?|unlearning|recommendation|recommender|distillation|clustering|evaluation|inference|systems?|settings?|frameworks?|clients?|servers?|rounds?|aggregation|edge|IoT|healthcare|medical|split|meta-?learning|continual|domain|representation|contrastive|self-?supervised|semi-?supervised|prompt|instruction|benchmarks?|datasets?|simulations?|deployments?|platforms?)\b", 1.0, "both"),
83 + (r"\bFL\b", 0.5, "both"),
84 + (r"(?i)\b(FedAvg|FedProx|FedSGD|SCAFFOLD|FedNova|FedOpt|FedAdam|FedAdagrad|FedYogi|FedBN|FedPer|FedProto|FedDyn|FedMA|MOON\b|Ditto|pFedMe|FedRep|FedALA|FedLoRA|FedIT|FedBERT|personali[sz]ed federated|cross-?silo|cross-?device|non-?IID (data|clients?|settings?|distributions?)|statistical heterogeneity|client (drift|heterogeneity|selection|sampling|participation|dropout|unavailability|stragglers?)|secure aggregation|SecAgg|split learning|SplitFed|decentrali[sz]ed (learning|training|SGD|optimi[sz]ation|federated)|gossip (learning|SGD|averaging)|collaborative learning (without|across) (sharing|institutions?|data silos?|hospitals?)|swarm learning|vertical federated|horizontal federated|VFL\b|HFL\b|federated-?learning|Flower framework|FATE\b|FedML|TensorFlow Federated|TFF\b|LEAF benchmark|FedScale|model poisoning in|Byzantine-?robust (aggregation|federated)|Krum|Trimmed mean|free-?riders? in federated|incentive mechanisms? for federated|federated (unlearning|forgetting))\b", 1.0, "both"),
85 + (r"(?i)\b(communication-?efficient (distributed|learning|training|federated)|gradient (compression|sparsification|quanti[sz]ation) (for|in) (distributed|federated)|model aggregation|aggregation server|local (updates?|epochs?|training|steps?) (on|at) (clients?|devices?|the edge)|global model (and|is) (local|aggregated)|parameter server (for|and) (clients|devices)|on-?device (training|learning) (without|with) (sharing|privacy))\b", 0.6, "both"),
86 + (r"(?i)\b(privacy-?preserving (distributed|collaborative|decentrali[sz]ed) (learning|training|machine learning|deep learning)|data silos?|clients?|edge devices? (train|collaborat)|without (sharing|centrali[sz]ing|uploading|exchanging) (raw |local |private |their )?data)\b", 0.6, "both"),
87 + ],
88 + exclusions=[r"(?is)^(?!.*(learning|training|model|client|aggregat|distributed|privacy)).*\b(federated (states?|republic|search|identity|database|cloud|governance|architecture|systems? of|network|information systems?|authentication|SSO|single sign)|federation of)\b"],
89 + related=["privacy_preserving_computation", "distributed_computing", "edge_computing", "ml_theory", "ai_safety", "foundation_models"],
90 + parent_of=[], child_of=["privacy_preserving_computation"],
91 + provenance=("No OpenAlex topic or DBLP stream isolates federated learning (it lives inside T10764 and in ML/networking venues), "
92 + "so 'federated <learning-term>' patterns and algorithm names (FedAvg, FedProx, SCAFFOLD...) carry the category at 1.0; "
93 + "bare 'FL' is 0.5 and case-sensitive. Non-ML senses of 'federated' are vetoed when no learning term co-occurs."),
94 +),
95 +]
added taxonomy/src/area_software_pl.py +138 −0
@@ -0,0 +1,138 @@
1 +"""Area software_pl: categories 14-17 (software engineering, programming languages, compilers, formal verification)."""
2 +
3 +CATEGORIES = [
4 +dict(
5 + id="software_engineering", number=14, name="Software Engineering", area="software_pl",
6 + definition=("How software is specified, built, tested, maintained and studied: requirements, architecture, testing "
7 + "and debugging, program repair, mining software repositories, empirical studies of developers, DevOps, "
8 + "model-driven engineering, open-source ecosystems, and AI-assisted software development. Language design "
9 + "is programming_languages; proofs of correctness are formal_verification."),
10 + topics=[("T10430", 1.0, "software engineering techniques and practices"), ("T10260", 1.0, "software engineering research"),
11 + ("T10639", 1.0, "advanced software engineering methodologies"), ("T10743", 1.0, "software testing and debugging"),
12 + ("T11450", 1.0, "model-driven software engineering"), ("T12423", 1.0, "software reliability and analysis"),
13 + ("T12490", 1.0, "software engineering and design patterns"), ("T11675", 0.8, "open source software"),
14 + ("T13197", 0.8, "spreadsheets and end-user computing"), ("T12127", 0.7, "software system performance, microservices, AIOps"),
15 + ("T10679", 0.7, "service-oriented architecture and web services"), ("T12799", 0.6, "mobile and web applications"),
16 + ("T12601", 0.5, "web applications and data management"), ("T10533", 0.4, "teaching and learning programming"),
17 + ("T11986", 0.4, "research software and scientific workflows"), ("T12479", 0.3, "web application security vulnerabilities"),
18 + ("T13650", 0.3, "Python scientific programming"), ("T10126", 0.2, "logic, programming and type systems"),
19 + ("T13034", 0.2, "information systems education and curriculum"), ("T13681", 0.1, "engineering and information technology")],
20 + subfields=[(1712, 0.5)],
21 + arxiv=[("cs.SE", 1.0)],
22 + streams=[("conf/icse", 1.0), ("conf/sigsoft", 1.0), ("conf/kbse", 1.0), ("conf/issta", 1.0), ("conf/msr", 1.0),
23 + ("conf/icst", 1.0), ("conf/saner", 1.0), ("journals/tse", 1.0), ("journals/tosem", 1.0), ("journals/ese", 1.0),
24 + ("journals/jss", 1.0), ("journals/infsof", 1.0), ("conf/re", 1.0), ("conf/models", 1.0), ("conf/issre", 1.0),
25 + ("conf/qrs", 1.0), ("journals/spe", 1.0), ("journals/software", 1.0), ("conf/caise", 0.6), ("conf/compsac", 0.6),
26 + ("journals/jossw", 0.5), ("journals/softx", 0.5), ("journals/aes", 0.5), ("journals/csse", 0.4),
27 + ("journals/tr", 0.3), ("conf/dsn", 0.3), ("journals/computing", 0.3), ("conf/sac", 0.2), ("journals/cacm", 0.1),
28 + ("journals/computer", 0.1), ("conf/icws", 0.3), ("journals/tsc", 0.3), ("conf/fedcsis", 0.2), ("conf/gi", 0.2),
29 + ("journals/ieicetd", 0.1), ("journals/jsjkx", 0.1), ("conf/iceis", 0.3), ("conf/sigcse", 0.2), ("conf/iticse", 0.2)],
30 + keywords=[
31 + (r"(?i)\bsoftware (engineering|development|testing|maintenance|evolution|quality|defects?|bugs?|repositor(y|ies)|architectures?|design|reuse|process(es)?|product lines?|projects?|metrics|vulnerabilit(y|ies)|ecosystems?|supply chains?|requirements?|documentation|analytics|practitioners?|developers?|teams?|artifacts?|systems? engineering|verification and validation|reliability|security practices|refactoring|modernization|migration|libraries|dependencies)\b", 1.0, "both"),
32 + (r"(?i)\b(code (review|generation|completion|smells?|clones?|search|summari[sz]ation|comprehension|quality|coverage|readability|translation|refactoring|LLMs?)|program (repair|comprehension|synthesis|analysis|slicing)|bug (locali[sz]ation|fixing|reports?|prediction|detection|triage)|defect prediction|fault locali[sz]ation|test (generation|cases?|suites?|oracles?|flakiness|automation|prioriti[sz]ation|amplification)|unit tests?|fuzz(ing|ers?)|mutation testing|regression testing|continuous (integration|delivery|deployment)|DevOps|CI/CD|technical debt|refactoring|static analysis|dynamic analysis|GitHub|open[- ]source (projects?|software|development|communities)|Stack Overflow|pull requests?|commit (messages?|histor)|issue (tracking|reports?)|mining software|API (usage|misuse|evolution|documentation|recommendation)|build (systems?|failures?)|configuration (errors?|management)|infrastructure as code|logging practices|AIOps|incident management)\b", 1.0, "both"),
33 + (r"(?i)\b(agile|Scrum|Kanban|requirements engineering|user stories|model-?driven|MDE\b|UML\b|SysML|domain-?specific modeling|microservices?|monolith|software-?defined|low-?code|no-?code|end-?user (programming|development)|spreadsheets?|pair programming|code ownership|onboarding|developer (productivity|experience|tools|surveys?)|programmers?' )\b", 0.8, "both"),
34 + (r"(?i)\b(LLM|AI|automated|neural|learning)-?(based|assisted|powered|driven)? (software engineering|code generation|programming|program repair|test generation|code review|debugging)\b|\b(code|coding) (LLMs?|assistants?|models?|agents?)\b|\bCopilot\b|\bvibe coding\b|\bLLMs? for (code|software|programming|testing)\b|\bAI pair programm\w+\b|\bCodex\b|\bStarCoder\b|\bCodeLlama\b|\bHumanEval\b|\bMBPP\b|\bSWE-?bench\b", 1.0, "both"),
35 + (r"(?i)\b(developers?|programmers?|software|codebase|source code|programs?|debugging|testing|maintainab)\b", 0.3, "both"),
36 + ],
37 + exclusions=[],
38 + related=["programming_languages", "formal_verification", "llm", "hci", "ai_agents", "operating_systems"],
39 + parent_of=[], child_of=[],
40 + provenance=("All 5 topics of OpenAlex subfield 1712 plus SE topics in 1702/1710 (T10639, T10430, T10260) and Social "
41 + "Sciences (T12490); ICSE/FSE/ASE/TSE/TOSEM/EMSE are the core venues; cs.SE suffices alone."),
42 +),
43 +dict(
44 + id="programming_languages", number=15, name="Programming Languages", area="software_pl",
45 + definition=("Design, semantics, type systems and implementation principles of programming languages: type theory, "
46 + "operational and denotational semantics, functional and logic programming, effects, concurrency models, "
47 + "memory safety, domain-specific languages, program analysis as a language problem. Code generation and "
48 + "optimisation pipelines are compilers (a child); machine-checked correctness is formal_verification."),
49 + topics=[("T10126", 1.0, "logic, programming and type systems"), ("T11727", 0.3, "advanced algebra and logic"),
50 + ("T11567", 0.2, "formal languages and automata"), ("T12002", 0.2, "computability and logic"),
51 + ("T10533", 0.2, "teaching and learning programming"), ("T13197", 0.2, "end-user programming")],
52 + subfields=[],
53 + arxiv=[("cs.PL", 1.0), ("cs.LO", 0.3), ("cs.FL", 0.2)],
54 + streams=[("journals/pacmpl", 1.0), ("conf/pldi", 1.0), ("conf/oopsla", 1.0), ("conf/esop", 1.0), ("journals/toplas", 1.0),
55 + ("journals/jfp", 1.0), ("conf/fscd", 0.8), ("conf/sas", 0.6), ("journals/mscs", 0.5), ("conf/ppopp", 0.5),
56 + ("conf/concur", 0.4), ("conf/csl", 0.3), ("conf/lics", 0.3), ("journals/lmcs", 0.3), ("conf/cgo", 0.3),
57 + ("conf/cc", 0.3), ("journals/acta", 0.3), ("conf/birthday", 0.1), ("journals/tcs", 0.1), ("journals/spe", 0.2),
58 + ("journals/computing", 0.1), ("conf/sigcse", 0.1)],
59 + keywords=[
60 + (r"(?i)\bprogramming languages?\b|\blanguage (design|semantics|implementation|features?|constructs?|runtimes?)\b|\bPL (research|community|theory)\b", 1.0, "both"),
61 + (r"(?i)\btype (systems?|theory|inference|checking|safety|soundness|classes|errors?|annotations?|directed)\b|\b(dependent|gradual|refinement|session|linear|affine|effect|ownership|intersection|union|algebraic data|polymorphic|higher-?kinded|substructural) types?\b|\b(statically|dynamically)[- ]typed\b|\btyped (lambda|functional|assembly|languages?)\b", 1.0, "both"),
62 + (r"(?i)\b(lambda[- ]calculus|pi-?calculus|operational semantics|denotational semantics|axiomatic semantics|program semantics|abstract interpretation|separation logic|Hoare logic|category theory|monads?|monadic|algebraic effects|effect handlers|continuations?|CPS\b|Curry-?Howard|homotopy type theory|cubical|normali[sz]ation by evaluation|bidirectional typing|subtyping|parametricity|logical relations|step-?indexed|coinduction|abstract machines?)\b", 1.0, "both"),
63 + (r"(?i)\b(functional programming|logic programming|constraint logic programming|answer set programming|probabilistic programming languages?|reactive programming|dataflow programming|array programming|Haskell|OCaml|Standard ML|Scheme|Racket|Clojure|Erlang|Elixir|Scala|Prolog|Datalog|Coq|Agda|Idris|Lean 4|Rust|WebAssembly|Wasm|Kotlin|Swift|Julia|domain-?specific languages?|DSLs?|embedded DSL|metaprogramming|macros?|staging|multi-?stage programming|garbage collect(ion|ors?)|memory safety|unsafe (code|Rust)|borrow checker|ownership (types?|model)|smart contracts? languages?|language (interoperability|workbench)|parser (combinators?|generators?)|grammars? (for|of) programming)\b", 0.9, "both"),
64 + (r"(?i)\b(program (synthesis|analysis|verification|transformation|equivalence|slicing|refinement|calculation)|static analysis|dataflow analysis|points?-?to analysis|alias analysis|taint analysis|symbolic execution|partial evaluation|interpreters?|virtual machines? for|bytecode|intermediate language|runtime systems?|concurrency (primitives|models|bugs)|memory models?|weak memory|data races?|deadlock (detection|freedom)|software transactional memory|actors? model|async/await|coroutines?|exceptions? handling|gradual verification)\b", 0.6, "both"),
65 + (r"(?i)\b(syntax|semantics|compil\w+|interpret\w+|languages?|programs?|programming)\b", 0.2, "both"),
66 + ],
67 + exclusions=[],
68 + related=["compilers", "formal_verification", "software_engineering", "computational_complexity"],
69 + parent_of=["compilers"], child_of=[],
70 + provenance=("T10126 is the single OpenAlex PL topic (types, logic, programming); PACMPL (POPL/OOPSLA/ICFP/PLDI) is the "
71 + "dominant venue stream in DBLP; conf/popl and conf/icfp have no separate stream over the window."),
72 +),
73 +dict(
74 + id="compilers", number=16, name="Compilers", area="software_pl",
75 + definition=("Translation and optimisation of programs for execution: compiler construction, intermediate representations, "
76 + "optimisation passes, code generation, JIT compilation, auto-vectorisation and parallelisation, polyhedral "
77 + "methods, high-level synthesis, and compilers for machine-learning workloads and accelerators. "
78 + "A child of programming_languages."),
79 + topics=[("T10054", 0.6, "parallel computing: compiler optimization, OpenMP, CUDA"), ("T10126", 0.4, "logic, programming and type systems"),
80 + ("T10904", 0.3, "embedded systems design: high-level synthesis"), ("T13650", 0.2, "Python scientific programming (JIT, NumPy)")],
81 + subfields=[],
82 + arxiv=[("cs.PL", 0.5), ("cs.DC", 0.1), ("cs.AR", 0.1), ("cs.MS", 0.1), ("cs.PF", 0.1)],
83 + streams=[("conf/cgo", 1.0), ("conf/cc", 1.0), ("conf/pact", 0.8), ("conf/pldi", 0.6), ("journals/taco", 0.6),
84 + ("conf/ppopp", 0.5), ("conf/asplos", 0.4), ("conf/mlsys", 0.4), ("journals/pacmpl", 0.3), ("journals/toplas", 0.3),
85 + ("conf/ics", 0.3), ("conf/sc", 0.2), ("conf/micro", 0.2), ("journals/tecs", 0.2), ("conf/europar", 0.2),
86 + ("journals/tpds", 0.1), ("conf/ipps", 0.1), ("journals/jpdc", 0.1), ("conf/esop", 0.2), ("conf/sas", 0.2),
87 + ("journals/spe", 0.2), ("conf/date", 0.1), ("conf/dac", 0.1), ("conf/fccm", 0.2), ("conf/fpga", 0.2)],
88 + keywords=[
89 + (r"(?i)\bcompilers?\b|\bcompilation\b|\bcompiler (optimi[sz]ation|design|infrastructure|passes|back-?ends?|front-?ends?|frameworks?|construction|bugs?|testing|fuzzing|verification|correctness)\b|\bcompiling\b", 1.0, "both"),
90 + (r"(?i)\b(LLVM|MLIR|GCC|Clang|TVM|XLA|Halide|TensorRT|polyhedral|auto-?tuning|auto-?vectori[sz]ation|vectori[sz]ation|loop (tiling|fusion|unrolling|interchange|transformations?|nests?|optimi[sz]ations?)|register allocation|instruction (scheduling|selection)|intermediate representations?|IR\b.*\b(lowering|optimi[sz]ation)|just-?in-?time|JIT\b|ahead-?of-?time|AOT\b|code generation (for|targeting)|binary (translation|rewriting|optimi[sz]ation)|decompil\w+|superoptimi[sz]\w+|peephole|tensor compilers?|kernel (generation|fusion|compilers?)|operator fusion|graph (compilers?|lowering|rewriting for)|schedule (search|space)|cost models? for (compilers?|code))\b", 1.0, "both"),
91 + (r"(?i)\b(high-?level synthesis|HLS\b|OpenMP|OpenCL|OpenACC|CUDA (kernels?|code|programming)|SYCL|parallelizing|auto-?parallel|program transformation|link-?time optimi[sz]ation|profile-?guided|whole-?program|inlining|dead code|constant (propagation|folding)|SSA\b|static single assignment|control-?flow graphs?|data-?flow (analysis|graphs?)|alias analysis|escape analysis|garbage collector (design|implementation)|managed runtimes?|interpreters? (design|performance)|bytecode (optimi[sz]ation|interpreters?)|WebAssembly compil|transpil\w+|source-?to-?source)\b", 0.7, "both"),
92 + (r"(?i)\b(machine learning|deep learning|ML|DL|AI|neural network|DNN|tensor|quantum|domain-?specific|hardware|FPGA|GPU|accelerator|verified|certified) compilers?\b|\bcompilers? for (machine learning|deep learning|neural networks?|accelerators?|GPUs?|FPGAs?|quantum)\b", 1.0, "both"),
93 + (r"(?i)\b(Triton|CUTLASS|cuDNN|oneDNN|XNNPACK|ONNX|TorchInductor|torch\.compile|TorchScript|Glow|nGraph|IREE|Tiramisu|AutoTVM|Ansor|CompCert|CakeML|Cranelift|V8|HotSpot|GraalVM|Numba|JAX (compilation|tracing)|Mojo)\b", 0.8, "both"),
94 + ],
95 + exclusions=[],
96 + related=["programming_languages", "ai_accelerators", "computer_architecture", "scientific_computing", "distributed_computing"],
97 + parent_of=[], child_of=["programming_languages"],
98 + provenance=("No dedicated OpenAlex topic: compiler optimisation sits inside T10054 (keywords list 'compiler optimization') "
99 + "and T10126. CGO/CC/PACT/TACO are the dedicated streams; keyword patterns (compiler, LLVM, MLIR, polyhedral...) carry the category."),
100 +),
101 +dict(
102 + id="formal_verification", number=17, name="Formal Verification", area="software_pl",
103 + definition=("Mathematically rigorous establishment of system properties: model checking, theorem proving and proof "
104 + "assistants, SAT/SMT solving and automated reasoning, deductive and runtime verification, verified software "
105 + "and hardware, formal semantics as a verification tool, verification of neural networks, smart contracts and "
106 + "protocols, automata and temporal logics. Language semantics as design is programming_languages."),
107 + topics=[("T10142", 1.0, "formal methods in verification"), ("T11125", 0.8, "Petri nets in system modeling"),
108 + ("T12002", 0.4, "computability, logic and AI algorithms"), ("T11010", 0.4, "logic, reasoning and knowledge"),
109 + ("T11596", 0.4, "constraint satisfaction and optimization (SAT/CSP)"), ("T11727", 0.3, "advanced algebra and logic"),
110 + ("T11567", 0.3, "automata theory"), ("T10126", 0.3, "logic, programming and type systems"),
111 + ("T11424", 0.2, "security and verification in computing"), ("T13286", 0.2, "modeling and simulation systems")],
112 + subfields=[],
113 + arxiv=[("cs.LO", 1.0), ("cs.FL", 0.6), ("cs.PL", 0.2), ("cs.SE", 0.1)],
114 + streams=[("conf/cav", 1.0), ("conf/tacas", 1.0), ("conf/fm", 1.0), ("conf/cade", 1.0), ("conf/itp", 1.0), ("conf/cpp", 1.0),
115 + ("conf/fmcad", 1.0), ("conf/vmcai", 1.0), ("journals/fmsd", 1.0), ("journals/jar", 1.0), ("journals/tocl", 0.8),
116 + ("conf/sat", 0.8), ("conf/lpar", 0.8), ("conf/lics", 0.6), ("conf/csl", 0.6), ("journals/lmcs", 0.6),
117 + ("conf/concur", 0.6), ("conf/cp", 0.5), ("conf/fossacs", 0.5), ("journals/constraints", 0.4), ("journals/acta", 0.4),
118 + ("conf/sas", 0.4), ("journals/apal", 0.3), ("journals/iandc", 0.3), ("conf/jelia", 0.3), ("conf/aiml", 0.3),
119 + ("conf/kr", 0.3), ("journals/mscs", 0.3), ("conf/fscd", 0.3), ("conf/dlt", 0.3), ("conf/birthday", 0.2),
120 + ("journals/tcs", 0.2), ("conf/dsn", 0.1), ("journals/tse", 0.1), ("conf/icse", 0.05), ("journals/pacmpl", 0.2),
121 + ("conf/pldi", 0.2), ("conf/esop", 0.3), ("conf/models", 0.2), ("conf/rtss", 0.1), ("journals/tcad", 0.1),
122 + ("conf/date", 0.1), ("conf/dac", 0.1)],
123 + keywords=[
124 + (r"(?i)\bformal (verification|methods?|specification|semantics|proofs?|analysis|modell?ing|reasoning|guarantees)\b|\bformally (verified|proved|proven|specified)\b", 1.0, "both"),
125 + (r"(?i)\b(model check(ing|ers?)|theorem prov(ing|ers?)|proof assistants?|interactive theorem|automated (reasoning|theorem|deduction)|SMT (solvers?|solving)|SAT (solvers?|solving)|satisfiability modulo|bounded model checking|symbolic model checking|abstraction refinement|CEGAR|IC3|PDR\b|k-?induction|temporal logics?|LTL\b|CTL\b|TCTL|mu-?calculus|Büchi|timed automata|hybrid automata|probabilistic model checking|PRISM model|refinement (calculus|checking)|TLA\+|Isabelle|Coq|Rocq|Lean (prover|4|theorem)|Agda|Dafny|Why3|Frama-?C|SPARK Ada|Z3|CVC[45]|Alloy|Event-?B|Petri nets?|process (algebras?|calculi)|bisimulation|rewriting logic|Maude|B method|VDM\b)\b", 1.0, "both"),
126 + (r"(?i)\b(verified (compilers?|software|implementations?|kernels?|systems?|cryptograph\w+|parsers?|libraries)|certified (programming|software|compilers?)|mechani[sz]ed (proofs?|semantics|metatheory|formali[sz]ation)|machine-?checked|proof of correctness|correct-?by-?construction|deductive verification|invariant (synthesis|generation|inference)|loop invariants?|runtime verification|runtime monitoring|contract-?based|design by contract|refinement types for verification|termination (analysis|proofs?)|verification conditions?|weakest precondition|proof (obligations?|engineering|repair|automation))\b", 1.0, "both"),
127 + (r"(?i)\b(neural network verification|verif(y|ying|ication of) (neural|deep|DNN|smart contracts?|cryptographic protocols?|distributed (protocols?|systems?)|hardware|RTL|hybrid systems?|cyber-?physical|autonomous|learning-?enabled)|safety verification|reachability analysis|abstract domains?|barrier certificates?|symbolic reasoning about|equivalence checking|hardware verification|formal hardware)\b", 1.0, "both"),
128 + (r"(?i)\b(automata|decidab(le|ility)|undecidab|satisfiab(le|ility)|unsatisfiable core|constraint solving|first-?order logic|higher-?order logic|modal logic|separation logic|Hoare logic|dynamic logic|description logics?|answer set|logic programming for verification|proof (search|systems?|theory|nets)|sequent calculus|resolution (proving|refutation)|superposition|E prover|Vampire)\b", 0.7, "both"),
129 + (r"(?i)\b(autoformali[sz]ation|formal (mathematics|theorem proving) (with|using|via) (LLMs?|language models?|neural)|neural theorem proving|LLM-?based (theorem|proof)|Lean (proofs?|theorems?|mathlib)|miniF2F|ProofNet|AlphaProof|DeepSeek-?Prover|proof (generation|synthesis) (with|using) (LLMs?|language models?))\b", 1.0, "both"),
130 + (r"(?i)\b(verification|verified|correctness|soundness|completeness|specification|logic|proofs?)\b", 0.2, "both"),
131 + ],
132 + exclusions=[],
133 + related=["programming_languages", "software_engineering", "computational_complexity", "applied_cryptography", "ai_safety", "computer_architecture"],
134 + parent_of=[], child_of=[],
135 + provenance=("T10142 is the OpenAlex formal-methods topic; logic/automata topics of 1703 (T12002, T11567, T11727) and Petri "
136 + "nets (T11125) support it. CAV/TACAS/FM/CADE/ITP/CPP/FMCAD/VMCAI/JAR/FMSD are dedicated streams; cs.LO suffices alone."),
137 +),
138 +]
added taxonomy/src/area_theory.py +97 −0
@@ -0,0 +1,97 @@
1 +"""Area theory: categories 18-19 (algorithms and data structures, computational complexity)."""
2 +
3 +CATEGORIES = [
4 +dict(
5 + id="algorithms", number=18, name="Algorithms and Data Structures", area="theory",
6 + definition=("Design and analysis of algorithms and data structures: graph, string, geometric, approximation, randomized, "
7 + "online, streaming, parallel and parameterized algorithms, combinatorial optimisation and metaheuristics, "
8 + "algorithmic game theory, algorithm engineering, and the combinatorics that underpins them. Lower bounds and "
9 + "hardness as such are computational_complexity; numerical algorithms are scientific_computing."),
10 + topics=[("T10720", 1.0, "complexity and algorithms in graphs (approximation, shortest paths, flows)"),
11 + ("T11269", 1.0, "algorithms and data compression (strings, sorting, succinct structures)"),
12 + ("T10374", 0.8, "advanced graph theory research"), ("T12292", 0.8, "graph theory and algorithms"),
13 + ("T10996", 0.7, "computational geometry and mesh generation"), ("T12541", 0.6, "graph labeling and dimension problems"),
14 + ("T12288", 0.6, "optimization and search problems"), ("T11596", 0.6, "constraint satisfaction and optimization"),
15 + ("T11476", 0.6, "graph theory and applications"), ("T11106", 0.6, "data management and algorithms"),
16 + ("T10100", 0.5, "metaheuristic optimization algorithms"), ("T11975", 0.5, "evolutionary algorithms"),
17 + ("T10948", 0.5, "advanced combinatorial mathematics"), ("T10551", 0.5, "scheduling and optimization algorithms"),
18 + ("T12176", 0.5, "optimization and packing problems"), ("T10792", 0.4, "matrix theory and algorithms"),
19 + ("T11329", 0.4, "limits and structures in graph theory"), ("T10963", 0.4, "advanced optimization algorithms"),
20 + ("T12401", 0.4, "scheduling and timetabling"), ("T10848", 0.3, "multi-objective optimization algorithms"),
21 + ("T10545", 0.3, "optimization and variational analysis"), ("T11697", 0.3, "numerical methods and algorithms"),
22 + ("T12162", 0.3, "cellular automata"), ("T12536", 0.3, "topological data analysis"),
23 + ("T10567", 0.3, "vehicle routing optimization"), ("T12709", 0.3, "optimization and mathematical programming"),
24 + ("T12303", 0.3, "tensor decomposition"), ("T11031", 0.3, "game theory and applications"),
25 + ("T10991", 0.3, "game theory and voting systems"), ("T11182", 0.2, "auction theory"),
26 + ("T10287", 0.2, "algebraic structures and combinatorial models"), ("T12056", 0.2, "Markov chains and Monte Carlo"),
27 + ("T11716", 0.2, "random matrices"), ("T12177", 0.2, "resource-constrained project scheduling"),
28 + ("T11252", 0.2, "evolutionary game theory and cooperation")],
29 + subfields=[(1703, 0.3)],
30 + arxiv=[("cs.DS", 1.0), ("cs.DM", 0.8), ("cs.CG", 0.8), ("cs.GT", 0.5), ("cs.CC", 0.2), ("math.OC", 0.3), ("cs.NE", 0.3)],
31 + streams=[("conf/soda", 1.0), ("conf/esa", 1.0), ("journals/algorithmica", 1.0), ("journals/talg", 1.0), ("conf/wads", 1.0),
32 + ("conf/isaac", 1.0), ("conf/swat", 1.0), ("conf/alenex", 1.0), ("conf/wea", 1.0), ("conf/icalp", 0.8),
33 + ("conf/latin", 0.8), ("journals/jco", 0.8), ("journals/algorithms", 0.8), ("journals/dmaa", 0.8), ("conf/cccg", 0.8),
34 + ("conf/stacs", 0.7), ("conf/mfcs", 0.7), ("journals/ipl", 0.7), ("journals/dam", 0.7), ("journals/dcg", 0.7),
35 + ("conf/approx", 0.7), ("conf/stoc", 0.6), ("conf/focs", 0.6), ("journals/siamcomp", 0.6), ("journals/tcs", 0.6),
36 + ("journals/siamdm", 0.6), ("conf/fsttcs", 0.6), ("conf/gecco", 0.6), ("conf/cec", 0.6), ("journals/tec", 0.6),
37 + ("journals/swevo", 0.6), ("journals/jcss", 0.5), ("conf/sofsem", 0.5), ("journals/jacm", 0.4), ("journals/dm", 0.4),
38 + ("journals/combinatorics", 0.4), ("journals/jgt", 0.4), ("journals/gc", 0.4), ("journals/cor", 0.4), ("journals/orl", 0.4),
39 + ("journals/mp", 0.4), ("conf/sigecom", 0.4), ("conf/cp", 0.4), ("journals/ejc", 0.3), ("journals/eor", 0.3),
40 + ("journals/siamjo", 0.3), ("journals/jgo", 0.3), ("journals/ol", 0.3), ("journals/jcta", 0.3), ("journals/jctb", 0.3),
41 + ("journals/jct", 0.3), ("journals/itor", 0.3), ("journals/rairo", 0.3), ("journals/jota", 0.2), ("journals/geb", 0.2),
42 + ("journals/anor", 0.2), ("journals/ior", 0.2), ("journals/jors", 0.2), ("journals/mansci", 0.1), ("journals/jet", 0.1),
43 + ("conf/sat", 0.2), ("journals/ffa", 0.2), ("journals/candie", 0.2), ("journals/asc", 0.2), ("journals/soco", 0.2),
44 + ("journals/evi", 0.4), ("journals/jossac", 0.2), ("journals/siamam", 0.1)],
45 + keywords=[
46 + (r"(?i)\b(approximation|randomi[sz]ed|online|streaming|sublinear|parameteri[sz]ed|exact|combinatorial|graph|geometric|parallel|distributed|dynamic|external[- ]memory|cache-?oblivious|greedy|divide-?and-?conquer|branch-?and-?bound|exponential-?time|polynomial-?time|efficient|fast|optimal|linear-?time|near-?linear|quantum|string|matching|sorting|search|enumeration|sampling|sketching|learning-?augmented) algorithms?\b", 1.0, "both"),
47 + (r"(?i)\bdata structures?\b|\balgorithm (design|analysis|engineering|theory)\b|\balgorithmic\b", 1.0, "both"),
48 + (r"(?i)\b(time complexity|running time|worst-?case|amorti[sz]ed|competitive ratio|approximation (ratio|factor|scheme)|PTAS|FPTAS|NP-?hard|NP-?complete|polynomial[- ]time|FPT\b|fixed-?parameter|kernelization|treewidth|lower bounds?|upper bounds?|O\(n (log|\^)|tight bounds?|conditional lower bounds?)\b", 0.7, "both"),
49 + (r"(?i)\b(shortest paths?|(maximum|max)[- ]flow|(minimum|min)[- ]cut|spanning trees?|vertex cover|independent sets?|dominating sets?|graph colou?ring|cliques?|Steiner trees?|knapsack|bin packing|TSP\b|travell?ing salesman|scheduling problems?|vehicle routing|facility location|set cover|matchings?|matroids?|submodular|(linear|integer|semidefinite) programming|cutting planes?|primal-?dual|LP (relaxation|rounding)|network flows?|spanners?|reachability|dynamic graphs?|distance oracles?)\b", 0.7, "both"),
50 + (r"(?i)\b(sorting|searching|hashing|hash (tables?|functions? for)|suffix (trees?|arrays?)|string (matching|algorithms?|indexing)|pattern matching|succinct|compressed (indexes?|data structures?)|sketch(es|ing)|locality[- ]sensitive hashing|Bloom filters?|priority queues?|heaps?|balanced (trees?|search)|B-?trees?|tries?|range (queries|searching|minimum)|nearest neighbou?r search|k-?d trees?|persistent data|dynamic (connectivity|trees)|union-?find|disjoint sets?|skip lists?|cuckoo hashing|streaming (algorithms?|model)|sketch(ing)? algorithms?|sublinear)\b", 0.8, "both"),
51 + (r"(?i)\b(metaheuristics?|evolutionary (algorithms?|computation)|genetic algorithms?|particle swarm|ant colony|simulated annealing|tabu search|local search|differential evolution|swarm intelligence|memetic|hyper-?heuristics?|combinatorial optimi[sz]ation|constraint (programming|satisfaction|propagation)|SAT-?based|MaxSAT|mixed-?integer|branch-?and-?(bound|cut|price)|column generation|Lagrangian relaxation|multi-?objective optimi[sz]ation|Pareto (front|optimal))\b", 0.6, "both"),
52 + (r"(?i)\b(computational geometry|Voronoi|Delaunay|convex hull|triangulations?|arrangements? of|range searching|point location|motion planning (complexity|algorithms)|polygon|visibility graphs?|geometric (optimi[sz]ation|intersection|approximation)|kinetic data)\b", 0.8, "both"),
53 + (r"(?i)\b(algorithmic game theory|mechanism design|price of anarchy|Nash equilibri(a|um)|equilibrium computation|fair division|cake cutting|matching markets|stable matching|auction (design|theory)|social choice|voting rules?|computational social choice|incentive compatib)\b", 0.7, "both"),
54 + (r"(?i)\b(graph theory|planar graphs?|chromatic|Hamiltonian|Ramsey|extremal (graph|combinatorics)|combinatorics|combinatorial (designs?|structures?)|enumerative|Latin squares?|permutations? (patterns?|statistics)|hypergraphs?|digraphs?|tournaments?|graph (minors?|decompositions?|embeddings?|homomorphisms?|isomorphism))\b", 0.6, "both"),
55 + (r"(?i)\balgorithms?\b|\bcomplexity\b|\bheuristics?\b|\boptimi[sz]ation\b|\bgraphs?\b", 0.3, "both"),
56 + ],
57 + exclusions=[],
58 + related=["computational_complexity", "ml_theory", "computer_graphics", "database_systems", "distributed_computing", "scientific_computing"],
59 + parent_of=[], child_of=[],
60 + provenance=("Core topics T10720/T11269 (1703, 1702) plus graph-theory, geometry, combinatorial-optimisation and metaheuristic "
61 + "topics of fields 17, 26 and 22; SODA/ESA/Algorithmica/TALG are the pure venues, STOC/FOCS shared with complexity."),
62 +),
63 +dict(
64 + id="computational_complexity", number=19, name="Computational Complexity", area="theory",
65 + definition=("The intrinsic difficulty of computational problems: complexity classes and their separations, hardness of "
66 + "approximation, circuit, proof, communication and query complexity, fine-grained and parameterized complexity, "
67 + "pseudorandomness and derandomization, computability and automata theory, algebraic complexity, and quantum "
68 + "complexity. Algorithm design is algorithms; cryptographic hardness assumptions are applied_cryptography."),
69 + topics=[("T12002", 0.8, "computability, logic and AI algorithms (Turing machines, Kolmogorov complexity)"),
70 + ("T11567", 0.7, "semigroups and automata theory"), ("T10720", 0.5, "complexity and algorithms in graphs (P vs NP, complexity classes)"),
71 + ("T11435", 0.5, "polynomial and algebraic computation"), ("T11727", 0.3, "advanced algebra and logic"),
72 + ("T11130", 0.2, "coding theory and cryptography"), ("T12162", 0.2, "cellular automata"),
73 + ("T10020", 0.1, "quantum information (quantum complexity)"), ("T12469", 0.1, "chaos, complexity and education")],
74 + subfields=[(1703, 0.2)],
75 + arxiv=[("cs.CC", 1.0), ("cs.FL", 0.5), ("cs.LO", 0.2), ("cs.DS", 0.2), ("quant-ph", 0.1), ("cs.IT", 0.1), ("math.IT", 0.1)],
76 + streams=[("journals/eccc", 1.0), ("journals/toct", 1.0), ("conf/stoc", 0.8), ("conf/focs", 0.8), ("conf/csr", 0.6),
77 + ("conf/dlt", 0.6), ("conf/icalp", 0.5), ("conf/stacs", 0.5), ("conf/mfcs", 0.5), ("journals/siamcomp", 0.5),
78 + ("journals/jcss", 0.5), ("journals/iandc", 0.5), ("journals/jacm", 0.4), ("conf/approx", 0.4), ("conf/fsttcs", 0.4),
79 + ("journals/tcs", 0.3), ("conf/sofsem", 0.3), ("journals/ipl", 0.2), ("conf/lics", 0.2), ("journals/lmcs", 0.2),
80 + ("conf/soda", 0.2), ("journals/apal", 0.2), ("journals/mscs", 0.2), ("journals/algorithmica", 0.1),
81 + ("journals/talg", 0.1), ("conf/csl", 0.1), ("journals/acta", 0.2), ("conf/sat", 0.2)],
82 + keywords=[
83 + (r"(?i)\b(computational|circuit|communication|proof|query|descriptive|parameteri[sz]ed|fine-?grained|average-?case|Kolmogorov|algorithmic|quantum|structural|space|time|arithmetic|algebraic|Boolean|counting|streaming|information) complexity\b", 1.0, "both"),
84 + (r"(?i)\bcomplexity (theory|class(es)?|lower bounds?|separations?|dichotom(y|ies)|hierarch(y|ies))\b|\bcomplexity-?theoretic\b", 1.0, "both"),
85 + (r"(?i)\b(P (vs\.?|versus|=|≠|!=) ?NP|NP-?complete(ness)?|NP-?hard(ness)?|coNP|PSPACE|EXPTIME|NEXP|#P\b|BPP\b|BQP\b|QMA\b|MIP\*?\b|PH\b|polynomial(-time)? hierarchy|AC\^?0|NC\^?[01]|TC\^?0|ACC\^?0|NL\b|L\b vs|P/poly|NP ∩ coNP|ZPP|RP\b|PP\b|PPAD|PLS\b|TFNP|W\[1\]|W\[2\]|XP\b|para-?NP)\b", 1.0, "both"),
86 + (r"(?i)\b(hardness of approximation|inapproximab|PCP theorem|probabilistically checkable|unique games|exponential[- ]time hypothesis|SETH\b|ETH\b|orthogonal vectors|3-?SUM|APSP (hardness|conjecture)|derandomi[sz]ation|pseudorandom(ness)? (generators?|functions?|for)|hardness (amplification|vs\.? randomness|magnification|escalation)|natural proofs|lifting theorems?|arithmetic circuits?|algebraic (complexity|circuits)|VP (vs|versus|≠) VNP|matrix multiplication exponent|omega \(ω\)|Boolean (circuits?|functions?) (lower|complexity|size)|circuit lower bounds?|formula (size|lower bounds?)|decision tree complexity|sensitivity conjecture|constraint satisfaction problems? dichotomy|CSP dichotomy|counting complexity|holographic algorithms|interactive proofs?|zero-?knowledge complexity|Arthur-?Merlin|relativi[sz]ation|oracle separations?|time-?space trade-?offs?|catalytic|meta-?complexity|MCSP|minimum circuit size)\b", 1.0, "both"),
87 + (r"(?i)\b(computability|decidability|undecidab|Turing machines?|halting problem|recursion theory|recursively enumerable|computably enumerable|Turing degrees|automata theory|formal languages?|regular languages?|context-?free (languages?|grammars?)|pushdown automata|finite automata|cellular automata|descriptional complexity|state complexity|Rice'?s theorem|reductions? (between|from|to)|Kolmogorov|algorithmic (randomness|information)|reverse mathematics|Büchi automata|tree automata|weighted automata|transducers?|ω-?regular)\b", 0.8, "both"),
88 + (r"(?i)\bproof complexity\b|\bresolution (lower bounds?|refutations?|width|size)\b|\bcutting planes proofs?\b|\bFrege\b|\bbounded arithmetic\b", 1.0, "both"),
89 + (r"(?i)\b(hardness|intractab(le|ility)|lower bounds?|reductions?|separations?)\b", 0.4, "both"),
90 + ],
91 + exclusions=[r"(?is)^(?!.*(complexity (class|theory)|NP|PSPACE|lower bound|hardness|computab|automata|Turing|circuit|reduction|decidab)).*\b(complexity of (care|disease|life|organi[sz]ations|the (world|environment|problem))|complex systems?|complexity science|chaos)\b"],
92 + related=["algorithms", "formal_verification", "quantum_computing", "applied_cryptography", "ml_theory"],
93 + parent_of=[], child_of=[],
94 + provenance=("OpenAlex has no pure complexity topic: it mixes classes into T10720 and computability into T12002/T11567. ECCC and "
95 + "ToCT are the pure streams; STOC/FOCS shared with algorithms; cs.CC alone suffices."),
96 +),
97 +]
added taxonomy/src/areas.py +40 −0
@@ -0,0 +1,40 @@
1 +"""WP44 ontology — the 8 top-level areas.
2 +
3 +Each leaf category (defined in the area_*.py modules) names exactly one of
4 +these area ids in its `area` field; build_ontology.py derives `leaves` lists.
5 +"""
6 +
7 +AREAS = [
8 + ("ai_ml", "Artificial Intelligence and Machine Learning",
9 + "Learning systems and their theory: large language and foundation models, multimodal and generative "
10 + "models, agents, reinforcement learning, learning theory, neural architectures, and AI safety."),
11 + ("perception_interaction", "Perception, Language, Robotics and Interaction",
12 + "Systems that sense, communicate and act in the physical and human world: vision, natural language, "
13 + "speech and audio, robotics, human-computer interaction, and computer graphics."),
14 + ("software_pl", "Software, Programming Languages and Formal Methods",
15 + "How software is built, expressed, translated and proved correct: software engineering, programming "
16 + "languages, compilers, and formal verification."),
17 + ("theory", "Theory of Computation",
18 + "Algorithms, data structures and the limits of efficient computation."),
19 + ("data_distributed", "Data, Information and Distributed Systems",
20 + "Storing, retrieving and processing data at scale: databases, information retrieval, distributed, "
21 + "cloud and edge computing, and computer networks."),
22 + ("security_privacy", "Security, Privacy and Trust",
23 + "Protecting systems, data and users: cybersecurity, applied cryptography, blockchain systems, "
24 + "privacy-preserving computation, and federated learning."),
25 + ("hardware_emerging", "Hardware, Architecture and Emerging Computing",
26 + "Physical substrates of computation: computer architecture, AI accelerators, operating systems, "
27 + "quantum computing, and sustainable computing."),
28 + ("computational_science", "Computational Science and Discovery",
29 + "Computing as an instrument of science: scientific computing, computational biology, and autonomous "
30 + "scientific discovery."),
31 +]
32 +
33 +# Signal-type identifiers used by the assignment rule (see build_ontology.py).
34 +SIGNAL_TYPES = ["openalex_topics", "openalex_subfields", "arxiv_categories", "dblp_streams", "keywords"]
35 +
36 +# arXiv categories admitted in the mapping (brief, input list).
37 +ARXIV_CATEGORIES = set("""cs.AI cs.AR cs.CC cs.CE cs.CG cs.CL cs.CR cs.CV cs.CY cs.DB cs.DC cs.DL cs.DM cs.DS cs.ET
38 +cs.FL cs.GL cs.GR cs.GT cs.HC cs.IR cs.IT cs.LG cs.LO cs.MA cs.MM cs.MS cs.NA cs.NE cs.NI cs.OH cs.OS cs.PF cs.PL
39 +cs.RO cs.SC cs.SD cs.SE cs.SI cs.SY stat.ML eess.AS eess.IV eess.SP eess.SY quant-ph math.OC math.IT math.ST
40 +q-bio.QM q-bio.GN physics.comp-ph""".split())
added tests/test_structural_change.py +29 −0
@@ -0,0 +1,29 @@
1 +import numpy as np, pandas as pd, sys, os
2 +sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
3 +from econometrics.structural_change import its_fit, placebo_breaks, bai_perron, pelt, bocpd
4 +
5 +rng = np.random.default_rng(1)
6 +dates = pd.date_range("2018-01-01", "2026-06-01", freq="MS")
7 +t = np.arange(len(dates)); ev = np.searchsorted(dates, pd.Timestamp("2022-12-01"))
8 +y = np.exp(5 + 0.01 * t + 0.4 * (t >= ev) + 0.02 * np.maximum(t - ev, 0) + 0.05 * rng.standard_normal(len(t)))
9 +
10 +
11 +def test_its_recovers_break():
12 + o, _ = its_fit(y, dates, "2022-12-01")
13 + assert abs(o["level_change"] - 0.4) < 0.1 and abs(o["slope_change"] - 0.02) < 0.01 and o["level_p"] < 0.01
14 +
15 +
16 +def test_placebo_rank_true_event_is_extreme():
17 + o, df = placebo_breaks(y, dates, "2022-12-01")
18 + assert o["level_t_placebo_p"] <= 0.1
19 +
20 +
21 +def test_bai_perron_finds_break_near_event():
22 + r = bai_perron(np.log(y), dates, max_breaks=2)
23 + assert r["m_bic"] >= 1 and min(abs(b - ev) for b in r["breaks_bic"]) <= 3
24 +
25 +
26 +def test_pelt_and_bocpd_flag_event():
27 + cps = pelt(np.log(y)); assert any(abs(c - ev) <= 3 for c in cps)
28 + _, cp, map_rl = bocpd(np.log(y))
29 + assert map_rl[ev - 1] > 20 and map_rl[ev + 3] <= 6 and cp[ev:ev + 4].max() > 0.3
30
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere