SPB Git
spboucher.ai

spb/wp44_uqo

Public

WP44: figure style (validated palette), figure batch 1, numbers macros, specialist brief template

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 3, 2026) parent 5473524 Browse files

23 changed files +717 −8

23 changed files
Unified Split
added agents/briefs/SPECIALIST_REVIEW_TEMPLATE.md +25 −0
@@ -0,0 +1,25 @@
1 +# Brief template — specialist reviewers (Agents E AI, F Systems, G Theory, H Security), Phase 5
2 +
3 +Inputs you receive (generated by Phase 4, never edited by hand):
4 +- `literature_synthesis/landmarks/<area>_landmarks.csv` — the candidate landmark works of your area: OpenAlex id, DOI/arXiv,
5 + title, venue, year, categories, c12/c36 citation windows, NCS36 (field-normalised), FWCI, abstract availability, OA full-text URL.
6 +- `literature_synthesis/landmarks/<area>_abstracts.jsonl` — title + abstract for each candidate (metadata only).
7 +- `results/graph/category_flows_yearly.csv`, `results/graph/category_counts_monthly.csv`, `results/econometrics/*` for your categories.
8 +- `manuscript/references.bib` (verified entries) — you may cite only keys that exist there or add entries you verify
9 + with Crossref/OpenAlex/arXiv single lookups (record them in `references_verification.csv`).
10 +
11 +Deliverables (structured, one directory per agent: `literature_synthesis/<area>/`):
12 +1. `claims.csv` — one row per scientific claim you make in the review: claim_id, claim (one sentence), claim_type
13 + ∈ {established, contested, speculative}, evidence_type ∈ {metadata_only, abstract, full_text, corpus_statistic},
14 + supporting_keys (bib keys or OpenAlex ids), contradicting_keys, corpus_artifact (file + column/row), confidence (0–1),
15 + verification_status ∈ {unverified, machine-verified, human-verified}, notes.
16 +2. `contributions.csv` — for each landmark you discuss: openalex_id, contribution_type ∈ {theory, method, empirical,
17 + benchmark, engineering, survey}, one-line description, whether the type was judged from abstract or full text.
18 +3. `review.md` — the disciplinary synthesis (3,000–6,000 words), sectioned by category, written in the WP44 academic
19 + style (CLAUDE.md §10): critical comparison, established vs speculative, unresolved problems (feeds Part X),
20 + conflicting findings reconciled explicitly, no causal language about ChatGPT, no "largest ever" claims.
21 +4. `open_problems.csv` — problem_id, statement, categories, why unresolved, testable direction, supporting_keys.
22 +
23 +Integrity rules (CLAUDE.md §1): never invent content of a paper you have only metadata for — say "the abstract reports";
24 +never call a preprint peer-reviewed; an abstract is not a full-text review (evidence_type must say which); every number
25 +comes from a corpus artefact you cite by file. Your review is checked by Agent K (critic) before any sentence enters the manuscript.
modified bibliometrics/02_collaboration.py +1 −1
@@ -18,7 +18,7 @@ def main():
18 18 # country output: full counting (work counted once per country) and fractional (1/n_countries)
19 19 con.execute("""CREATE OR REPLACE TEMP TABLE wc AS SELECT w.id, w.publication_year AS year, w.period, unnest(w.countries) AS country, 1.0/len(w.countries) AS frac FROM E w WHERE len(w.countries) > 0""")
20 20 con.execute(f"""COPY (SELECT year, country, count(*) n_full, round(sum(frac),1) n_frac FROM wc GROUP BY 1,2 ORDER BY 1, 3 DESC) TO '{out}/countries_yearly.csv' (HEADER)""")
21 − con.execute(f"""COPY (SELECT year, count(*) n, sum((n_countries>=2)::int) n_intl, round(sum((n_countries>=2)::int)*1.0/count(*),4) share_intl FROM E WHERE n_countries>0 GROUP BY 1 ORDER BY 1) TO '{out}/international_collab_yearly.csv' (HEADER)""")
21 + con.execute(f"""COPY (SELECT publication_year AS year, count(*) n, sum((n_countries>=2)::int) n_intl, round(sum((n_countries>=2)::int)*1.0/count(*),4) share_intl FROM E WHERE n_countries>0 GROUP BY 1 ORDER BY 1) TO '{out}/international_collab_yearly.csv' (HEADER)""")
22 22 con.execute("""CREATE OR REPLACE TEMP TABLE wi AS SELECT w.id, w.publication_year AS year, w.period, unnest(w.institution_types) AS itype FROM E w WHERE len(w.institution_types) > 0""")
23 23 con.execute(f"""COPY (SELECT year, itype, count(*) n FROM wi GROUP BY 1,2 ORDER BY 1,3 DESC) TO '{out}/institution_types_yearly.csv' (HEADER)""")
24 24 con.execute(f"""COPY (SELECT year, count(DISTINCT id) n_with_inst, count(DISTINCT CASE WHEN itype='company' THEN id END) n_company,
added figures/FIGURES.json +38 −0
@@ -0,0 +1,38 @@
1 +[
2 + {
3 + "file": "fig_prisma_flow",
4 + "caption": "PRISMA-style flow of record identification, exclusion and the final mapping corpus.",
5 + "sources": [
6 + "data/metadata/prisma_counts.json"
7 + ]
8 + },
9 + {
10 + "file": "fig_volume_yearly",
11 + "caption": "Eligible works per publication year by source membership (2026 incomplete).",
12 + "sources": [
13 + "results/bibliometrics/volume_yearly.csv"
14 + ]
15 + },
16 + {
17 + "file": "fig_volume_monthly_raw",
18 + "caption": "Monthly first appearances; January spikes reflect year-only publication dates and motivate the month-precision filter.",
19 + "sources": [
20 + "results/bibliometrics/volume_monthly.csv"
21 + ]
22 + },
23 + {
24 + "file": "fig_types_openness",
25 + "caption": "Document-type composition and metadata completeness by year.",
26 + "sources": [
27 + "results/bibliometrics/volume_yearly.csv"
28 + ]
29 + },
30 + {
31 + "file": "fig_collaboration",
32 + "caption": "Team size, international co-authorship and industry participation by year.",
33 + "sources": [
34 + "results/bibliometrics/team_size_yearly.csv",
35 + "results/bibliometrics/industry_share_yearly.csv"
36 + ]
37 + }
38 +]
\ No newline at end of file
added figures/fig_collaboration.pdf +0 −0

Binary file not shown.

added figures/fig_collaboration.png +0 −0

Binary file not shown.

added figures/fig_prisma_flow.pdf +0 −0

Binary file not shown.

added figures/fig_prisma_flow.png +0 −0

Binary file not shown.

added figures/fig_types_openness.pdf +0 −0

Binary file not shown.

added figures/fig_types_openness.png +0 −0

Binary file not shown.

added figures/fig_volume_monthly_raw.pdf +0 −0

Binary file not shown.

added figures/fig_volume_monthly_raw.png +0 −0

Binary file not shown.

added figures/fig_volume_yearly.pdf +0 −0

Binary file not shown.

added figures/fig_volume_yearly.png +0 −0

Binary file not shown.

added figures/make_figures.py +101 −0
@@ -0,0 +1,101 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — publication figures, batch 1 (corpus and volume). Every figure is generated from result files; none is drawn by hand.
3 +Run on the compute node: python figures/make_figures.py --root ~/wp44_uqo → figures/*.pdf + .png, and figures/FIGURES.json (inventory)."""
4 +import argparse, json, os, sys
5 +import numpy as np, pandas as pd, matplotlib
6 +matplotlib.use("Agg")
7 +import matplotlib.pyplot as plt
8 +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
9 +import style
10 +style.apply()
11 +INV = []
12 +
13 +
14 +def reg(name, caption, sources):
15 + INV.append({"file": name, "caption": caption, "sources": sources})
16 +
17 +
18 +def fig_prisma(R, out):
19 + P = json.load(open(f"{R}/data/metadata/prisma_counts.json")); fig, ax = plt.subplots(figsize=(7.2, 6.2)); ax.axis("off"); ax.grid(False)
20 + boxes = [("Identified (OpenAlex works with a CS topic, 2018–2026, snapshot 2026-10-14)", P["identified"]["openalex_total_records_loaded"]),
21 + ("Outside window (E1)", -P["excluded_sequential"]["e1_outside_window"]), ("Non-research types (E2)", -P["excluded_sequential"]["e2_type"]),
22 + ("XPAC expansion records, not DBLP-linked (E3)", -P["excluded_sequential"]["e3_xpac"]), ("No title (E4)", -P["excluded_sequential"]["e4_no_title"]),
23 + ("Preprint duplicates merged into the published version (E5)", -P["excluded_sequential"]["e5_duplicate"]),
24 + ("Generic deposit platforms without DBLP/arXiv support (E7)", -P["excluded_sequential"].get("e7_generic_deposit", 0)),
25 + ("After exclusions", P["after_exclusions"]), ("CS core — mapping corpus (DBLP ∨ primary CS ∨ CS topic ∧ arXiv cs.*)", P["cs_core_eligible"]),
26 + ("CS penumbra — secondary CS topic only (sensitivity set)", P["cs_penumbra_eligible_otherwise"])]
27 + y = 0.95
28 + for label, n in boxes:
29 + excl = n < 0; txt = f"{abs(n):,}"
30 + ax.text(0.02 if not excl else 0.40, y, (label if not excl else "− " + label) + f"\n{txt}", fontsize=8.2, va="top", ha="left",
31 + bbox=dict(boxstyle="round,pad=0.4", fc="white", ec=style.AXIS if not excl else style.GRID, lw=0.8), transform=ax.transAxes, color=style.INK)
32 + y -= 0.095
33 + ax.set_title("PRISMA-style flow of the WP44 mapping corpus", loc="left")
34 + style.save(fig, f"{out}/fig_prisma_flow.pdf"); reg("fig_prisma_flow", "PRISMA-style flow of record identification, exclusion and the final mapping corpus.", ["data/metadata/prisma_counts.json"])
35 +
36 +
37 +def fig_yearly_volume(R, out):
38 + d = pd.read_csv(f"{R}/results/bibliometrics/volume_yearly.csv"); d = d[d.year <= 2026]
39 + fig, ax = plt.subplots(figsize=(6.8, 3.4))
40 + cols = [("n", "All eligible works"), ("n_dblp", "DBLP-indexed"), ("n_cs_primary", "Primary field = CS"), ("n_arxiv", "arXiv version linked")]
41 + for i, (c, lab) in enumerate(cols):
42 + ax.plot(d.year, d[c] / 1e3, marker="o", label=lab, color=style.CATEGORICAL[i])
43 + ax.annotate(lab, (d.year.iloc[-1], d[c].iloc[-1] / 1e3), xytext=(4, 0), textcoords="offset points", fontsize=7, color=style.INK2, va="center")
44 + ax.axvspan(2026, 2026.6, color=style.GRID, alpha=0.5, lw=0); ax.text(2026.3, ax.get_ylim()[0] + 0.02 * ax.get_ylim()[1], "2026\n(partial)", fontsize=7, color=style.INK2, va="bottom", ha="center")
45 + ax.set_ylabel("Works (thousands)"); ax.set_xlabel(""); ax.set_xlim(2017.7, 2027.4); ax.legend(loc="upper left")
46 + ax.set_title("Eligible computer-science works per publication year, by membership", loc="left")
47 + style.save(fig, f"{out}/fig_volume_yearly.pdf"); reg("fig_volume_yearly", "Eligible works per publication year by source membership (2026 incomplete).", ["results/bibliometrics/volume_yearly.csv"])
48 +
49 +
50 +def fig_monthly(R, out):
51 + d = pd.read_csv(f"{R}/results/bibliometrics/volume_monthly.csv")
52 + m = d.groupby("first_month")["n"].sum().reset_index(); m = m[(m.first_month >= "2018-01") & (m.first_month <= "2026-10")]
53 + # month-precision only: records dated YYYY-01-01 are year-precision and are excluded from monthly series (A-002); here approximated by dropping January spikes is NOT done:
54 + # volume_monthly is built from first_month of all eligible works; the precise month-only series comes from category_counts_monthly (n_month_precision). We plot both when available.
55 + x = pd.to_datetime(m.first_month + "-01"); fig, ax = plt.subplots(figsize=(6.8, 3.2))
56 + ax.plot(x, m.n / 1e3, color=style.CATEGORICAL[0], lw=1.2, label="All eligible works (first appearance month)")
57 + style.event_line(ax, pd.Timestamp("2022-11-30"))
58 + ax.set_ylabel("Works per month (thousands)"); ax.set_title("Monthly first appearances of eligible works (January spikes = year-precision dates)", loc="left")
59 + ax.legend(loc="upper left"); style.save(fig, f"{out}/fig_volume_monthly_raw.pdf")
60 + reg("fig_volume_monthly_raw", "Monthly first appearances; January spikes reflect year-only publication dates and motivate the month-precision filter.", ["results/bibliometrics/volume_monthly.csv"])
61 +
62 +
63 +def fig_types_sources(R, out):
64 + d = pd.read_csv(f"{R}/results/bibliometrics/volume_yearly.csv"); d = d[d.year <= 2026]
65 + fig, axes = plt.subplots(1, 2, figsize=(7.2, 3.0))
66 + ax = axes[0]
67 + for i, (c, lab) in enumerate([("n_article", "Journal article"), ("n_conf", "Conference paper"), ("n_preprint", "Preprint")]):
68 + ax.plot(d.year, 100 * d[c] / d.n, marker="o", label=lab, color=style.CATEGORICAL[i])
69 + ax.set_ylabel("Share of eligible works (%)"); ax.set_title("Document types", loc="left"); ax.legend()
70 + ax = axes[1]
71 + for i, (c, lab) in enumerate([("n_abstract", "Abstract available"), ("n_oa", "Open access"), ("n_arxiv", "arXiv linked")]):
72 + ax.plot(d.year, 100 * d[c] / d.n, marker="o", label=lab, color=style.CATEGORICAL[i + 3])
73 + ax.set_title("Metadata completeness and openness", loc="left"); ax.legend(); ax.set_ylabel("Share of eligible works (%)")
74 + style.save(fig, f"{out}/fig_types_openness.pdf"); reg("fig_types_openness", "Document-type composition and metadata completeness by year.", ["results/bibliometrics/volume_yearly.csv"])
75 +
76 +
77 +def fig_collab(R, out):
78 + p = f"{R}/results/bibliometrics/team_size_yearly.csv"
79 + if not os.path.exists(p):
80 + return
81 + d = pd.read_csv(p); d = d[d.year <= 2026]; ind = pd.read_csv(f"{R}/results/bibliometrics/industry_share_yearly.csv")
82 + fig, axes = plt.subplots(1, 2, figsize=(7.2, 3.0))
83 + ax = axes[0]; ax.plot(d.year, d.mean_authors, marker="o", color=style.CATEGORICAL[0], label="Mean authors per work"); ax.set_ylabel("Authors"); ax.set_title("Team size", loc="left"); ax.legend()
84 + ax = axes[1]; ax.plot(d.year, 100 * d.intl / d.n, marker="o", color=style.CATEGORICAL[1], label="≥ 2 countries (all works)")
85 + ax.plot(ind.year, 100 * ind.share_company, marker="o", color=style.CATEGORICAL[2], label="Company affiliation (works with affiliations)")
86 + ax.set_ylabel("Share (%)"); ax.set_title("International and industry participation", loc="left"); ax.legend()
87 + style.save(fig, f"{out}/fig_collaboration.pdf"); reg("fig_collaboration", "Team size, international co-authorship and industry participation by year.", ["results/bibliometrics/team_size_yearly.csv", "results/bibliometrics/industry_share_yearly.csv"])
88 +
89 +
90 +def main():
91 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); out = f"{a.root}/figures"; os.makedirs(out, exist_ok=True)
92 + for f in (fig_prisma, fig_yearly_volume, fig_monthly, fig_types_sources, fig_collab):
93 + try:
94 + f(a.root, out); print("ok", f.__name__)
95 + except Exception as e:
96 + print("FAIL", f.__name__, e)
97 + json.dump(INV, open(f"{out}/FIGURES.json", "w"), indent=1)
98 +
99 +
100 +if __name__ == "__main__":
101 + main()
added figures/style.py +44 −0
@@ -0,0 +1,44 @@
1 +"""WP44 figure style (matplotlib, vector PDF). Palette = validated reference palette of the dataviz skill
2 +(categorical slots in fixed order, validated 2026-11-03 with scripts/validate_palette.js, light surface #fcfcfb).
3 +Rules applied everywhere: one y-axis per panel (no dual axes), thin marks, recessive grid, direct labels for <= 4 series,
4 +legend when >= 2 series, sequential = single blue hue, diverging = blue/orange with neutral midpoint, text in ink tokens."""
5 +import matplotlib as mpl
6 +import matplotlib.pyplot as plt
7 +
8 +CATEGORICAL = ["#2a78d6", "#eb6834", "#1baf7a", "#eda100", "#e87ba4", "#008300", "#6250d6", "#e34948"]
9 +SEQUENTIAL = ["#cde2fb", "#b7d3f6", "#9ec5f4", "#86b6ef", "#6da7ec", "#5598e7", "#3987e5", "#2a78d6", "#256abf", "#1c5cab", "#184f95", "#104281", "#0d366b"]
10 +DIVERGING = ["#0d366b", "#2a78d6", "#86b6ef", "#f0efec", "#f7a97f", "#eb6834", "#8a3412"]
11 +INK, INK2, MUTED, GRID, AXIS, SURFACE = "#0b0b0b", "#52514e", "#898781", "#e1e0d9", "#c3c2b7", "#fcfcfb"
12 +EVENT = "#52514e" # reference-date marker (ChatGPT release), never a series colour
13 +
14 +
15 +def apply():
16 + mpl.rcParams.update({
17 + "figure.dpi": 120, "savefig.dpi": 300, "figure.facecolor": "white", "axes.facecolor": "white",
18 + "font.family": "serif", "font.size": 9, "axes.titlesize": 10, "axes.labelsize": 9, "legend.fontsize": 8, "xtick.labelsize": 8, "ytick.labelsize": 8,
19 + "axes.edgecolor": AXIS, "axes.linewidth": 0.6, "axes.spines.top": False, "axes.spines.right": False,
20 + "axes.grid": True, "grid.color": GRID, "grid.linewidth": 0.5, "axes.axisbelow": True,
21 + "xtick.color": MUTED, "ytick.color": MUTED, "axes.labelcolor": INK2, "text.color": INK,
22 + "lines.linewidth": 1.4, "lines.markersize": 4, "legend.frameon": False,
23 + "axes.prop_cycle": mpl.cycler(color=CATEGORICAL), "pdf.fonttype": 42, "ps.fonttype": 42,
24 + })
25 +
26 +
27 +def seq_cmap():
28 + return mpl.colors.LinearSegmentedColormap.from_list("wp44_seq", SEQUENTIAL)
29 +
30 +
31 +def div_cmap():
32 + return mpl.colors.LinearSegmentedColormap.from_list("wp44_div", DIVERGING)
33 +
34 +
35 +def event_line(ax, x, label="ChatGPT release (2022-11-30)"):
36 + ax.axvline(x, color=EVENT, linewidth=0.8, linestyle="--", zorder=1)
37 + ax.annotate(label, xy=(x, 1.0), xycoords=("data", "axes fraction"), xytext=(3, -2), textcoords="offset points", fontsize=7, color=INK2, va="top")
38 +
39 +
40 +def save(fig, path):
41 + fig.savefig(path, bbox_inches="tight")
42 + if path.endswith(".pdf"):
43 + fig.savefig(path[:-4] + ".png", bbox_inches="tight")
44 + plt.close(fig)
modified manuscript/tables/numbers.tex +34 −1
@@ -1 +1,34 @@
1 −% machine-generated numbers (tables/make_numbers.py) — placeholder until results exist
1 +% machine-generated by tables/make_numbers.py — do not edit
2 +\newcommand{\LinkCreated}{2026-11-03T20:50:56.182276+00:00}
3 +\newcommand{\LinkOaCsRecords}{15{,}490{,}159}
4 +\newcommand{\LinkOaCsDistinctIds}{15{,}490{,}159}
5 +\newcommand{\LinkDblpTwoZeroOneEightTwoZeroTwoSix}{4{,}347{,}365}
6 +\newcommand{\LinkDblpWithDoi}{4{,}052{,}326}
7 +\newcommand{\LinkDblpWithArxiv}{0}
8 +\newcommand{\LinkLinkDblpOaTitleYear}{179{,}990}
9 +\newcommand{\LinkLinkDblpOaDoi}{2{,}562{,}431}
10 +\newcommand{\LinkDblpLinkedDistinct}{2{,}736{,}903}
11 +\newcommand{\LinkDblpUnlinked}{1{,}610{,}462}
12 +\newcommand{\LinkDblpUnlinkedByTypePhdthesis}{26{,}841}
13 +\newcommand{\LinkDblpUnlinkedByTypeIncollection}{3{,}362}
14 +\newcommand{\LinkDblpUnlinkedByTypeInproceedings}{477{,}235}
15 +\newcommand{\LinkDblpUnlinkedByTypeBook}{1{,}398}
16 +\newcommand{\LinkDblpUnlinkedByTypeArticle}{1{,}084{,}640}
17 +\newcommand{\LinkDblpUnlinkedByTypeData}{16{,}986}
18 +\newcommand{\LinkDblpUnlinkedWithDoiInSlim}{1{,}077{,}407}
19 +\newcommand{\LinkArxivCsRecords}{915{,}115}
20 +\newcommand{\LinkArxivCreatedTwoZeroOneEightTwoZeroTwoSix}{890{,}852}
21 +\newcommand{\LinkLinkArxivOaArxivId}{792{,}672}
22 +\newcommand{\LinkLinkArxivOaDoi}{520}
23 +\newcommand{\LinkLinkArxivOaTitleYear}{2{,}143}
24 +\newcommand{\LinkArxivLinkedDistinct}{744{,}680}
25 +\newcommand{\LinkAclRecords}{135{,}558}
26 +\newcommand{\LinkAclTwoZeroOneEightTwoZeroTwoSix}{85{,}625}
27 +\newcommand{\LinkLinkAclOaDoi}{57{,}474}
28 +\newcommand{\LinkLinkAclOaTitleYear}{14{,}904}
29 +\newcommand{\LinkAclLinkedDistinctTwoZeroOneEightTwoZeroTwoSix}{72{,}223}
30 +\newcommand{\LinkNeededOaIdsSecondPassDblpDoi}{1{,}077{,}406}
31 +\newcommand{\LinkNeededOaIdsSecondPassArxivDoi}{39{,}200}
32 +\newcommand{\LinkSlimTotal}{464{,}765{,}485}
33 +\newcommand{\LinkSlimCsAnyTwoZeroOneEightTwoZeroTwoSix}{15{,}490{,}159}
34 +\newcommand{\LinkSeconds}{10}
added tables/make_numbers.py +47 −0
@@ -0,0 +1,47 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — generate manuscript/tables/numbers.tex: every number quoted in the prose is a LaTeX macro produced here from
3 +result files (never typed by hand). Macro names use letters only (no digits: `\\nCoreEligible`, not `\\n2026`).
4 +Sources: data/metadata/prisma_counts.json, linkage_report.json, results/bibliometrics/*.csv, results/econometrics/*.json,
5 +taxonomy/validation/metrics.json, results/emerging/metrics_h12.json (each optional: missing → macro undefined → compile error,
6 +which is intended: the prose cannot cite a number that does not exist)."""
7 +import json, os, sys, csv, re
8 +R = sys.argv[1] if len(sys.argv) > 1 else os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
9 +out = []
10 +WORDS = {"0": "Zero", "1": "One", "2": "Two", "3": "Three", "4": "Four", "5": "Five", "6": "Six", "7": "Seven", "8": "Eight", "9": "Nine"}
11 +def name(s):
12 + s = re.sub(r"[^A-Za-z0-9]+", " ", s).title().replace(" ", "")
13 + return "".join(WORDS[ch] if ch.isdigit() else ch for ch in s)
14 +def fmt(v):
15 + if isinstance(v, bool): return str(v)
16 + if isinstance(v, int): return f"{v:,}".replace(",", "{,}")
17 + if isinstance(v, float):
18 + return f"{v:,.3f}".replace(",", "{,}") if abs(v) < 100 else f"{v:,.1f}".replace(",", "{,}")
19 + return str(v).replace("%", "\\%").replace("_", "\\_")
20 +def emit(k, v):
21 + out.append(f"\\newcommand{{\\{name(k)}}}{{{fmt(v)}}}")
22 +def walk(prefix, obj):
23 + if isinstance(obj, dict):
24 + for k, v in obj.items(): walk(f"{prefix} {k}", v)
25 + elif isinstance(obj, (int, float, str, bool)) and obj is not None:
26 + emit(prefix, obj)
27 +p = os.path.join(R, "data/metadata/prisma_counts.json")
28 +if os.path.exists(p): walk("prisma", json.load(open(p)))
29 +p = os.path.join(R, "data/metadata/linkage_report.json")
30 +if os.path.exists(p): walk("link", json.load(open(p)))
31 +for f in ["volume_yearly", "citation_windows_yearly", "international_collab_yearly", "industry_share_yearly", "venue_hhi_yearly"]:
32 + p = os.path.join(R, "results/bibliometrics", f + ".csv")
33 + if os.path.exists(p):
34 + for row in csv.DictReader(open(p)):
35 + y = row.get("year") or row.get("publication_year")
36 + for k, v in row.items():
37 + if k in ("year", "publication_year") or v in ("", None): continue
38 + try: v = int(v) if re.fullmatch(r"-?\d+", v) else float(v)
39 + except ValueError: pass
40 + emit(f"{f} {y} {k}", v)
41 +for f, key in [("results/econometrics/its_results.json", "its"), ("taxonomy/validation/metrics.json", "val"), ("results/emerging/metrics_h12.json", "emerg"), ("results/emerging/metrics_h24.json", "emergTwoFour")]:
42 + p = os.path.join(R, f)
43 + if os.path.exists(p):
44 + d = json.load(open(p))
45 + walk(key, {k: v for k, v in d.items() if not isinstance(v, (list,))})
46 +open(os.path.join(R, "manuscript/tables/numbers.tex"), "w").write("% machine-generated by tables/make_numbers.py — do not edit\n" + "\n".join(out) + "\n")
47 +print(len(out), "macros")
added taxonomy/build_ontology.py +366 −0
@@ -0,0 +1,366 @@
1 +#!/usr/bin/env python3
2 +"""Build and validate the WP44 computer-science ontology.
3 +
4 +Reads the mapping data modules in taxonomy/src/ (area_*.py, areas.py, unmapped_cs.py), validates them against the
5 +OpenAlex topic export and the DBLP stream export, and writes:
6 + taxonomy/ontology.yaml (schema of agents/briefs/AGENT_D_TAXONOMY.md)
7 + taxonomy/topic_map.csv (one row per (topic_id, category_id, weight))
8 + taxonomy/ONTOLOGY.md (generated section between <!-- BEGIN GENERATED --> / <!-- END GENERATED --> markers;
9 + hand-written prose outside the markers is preserved)
10 +Run: /opt/homebrew/bin/python3 taxonomy/build_ontology.py [--no-write]
11 +Exit status is non-zero if any validation fails.
12 +"""
13 +from __future__ import annotations
14 +
15 +import csv
16 +import datetime as dt
17 +import glob
18 +import importlib.util
19 +import os
20 +import re
21 +import sys
22 +from collections import Counter, defaultdict
23 +
24 +HERE = os.path.dirname(os.path.abspath(__file__))
25 +SRC = os.path.join(HERE, "src")
26 +TOPICS_CSV = os.path.join(HERE, "openalex_topics_2026-10-14.csv")
27 +STREAMS_CSV = os.path.join(HERE, "dblp_streams_2018_2026.csv")
28 +OUT_YAML = os.path.join(HERE, "ontology.yaml")
29 +OUT_MAP = os.path.join(HERE, "topic_map.csv")
30 +OUT_MD = os.path.join(HERE, "ONTOLOGY.md")
31 +VERSION, CREATED = "1.0", "2026-11-03"
32 +CS_FIELD = "17"
33 +TOP_STREAMS = 600
34 +MARK_BEGIN, MARK_END = "<!-- BEGIN GENERATED -->", "<!-- END GENERATED -->"
35 +FIELDS = {"title", "abstract", "both"}
36 +LIST_KEYS = ("topics", "subfields", "arxiv", "streams", "keywords", "exclusions", "related", "parent_of", "child_of")
37 +
38 +errors: list[str] = []
39 +
40 +
41 +def err(msg: str) -> None:
42 + errors.append(msg)
43 +
44 +
45 +def load_module(path: str):
46 + name = os.path.splitext(os.path.basename(path))[0]
47 + spec = importlib.util.spec_from_file_location(name, path)
48 + mod = importlib.util.module_from_spec(spec)
49 + spec.loader.exec_module(mod) # type: ignore[union-attr]
50 + return mod
51 +
52 +
53 +def wt_ok(w) -> bool:
54 + return isinstance(w, (int, float)) and 0 < float(w) <= 1
55 +
56 +
57 +def main(write: bool = True) -> int:
58 + topics = {r["topic_id"]: r for r in csv.DictReader(open(TOPICS_CSV, encoding="utf-8"))}
59 + subfields = {int(r["subfield_id"]): r["subfield"] for r in topics.values()}
60 + streams = {r["stream"]: r for r in csv.DictReader(open(STREAMS_CSV, encoding="utf-8"))}
61 + areas_mod = load_module(os.path.join(SRC, "areas.py"))
62 + unmapped_mod = load_module(os.path.join(SRC, "unmapped_cs.py"))
63 + AREAS = {a[0]: {"id": a[0], "name": a[1], "description": a[2]} for a in areas_mod.AREAS}
64 + ARXIV = areas_mod.ARXIV_CATEGORIES
65 +
66 + cats: list[dict] = []
67 + for path in sorted(glob.glob(os.path.join(SRC, "area_*.py"))):
68 + mod = load_module(path)
69 + for c in mod.CATEGORIES:
70 + c["_module"] = os.path.basename(path)
71 + cats.append(c)
72 + cats.sort(key=lambda c: c["number"])
73 + ids = [c["id"] for c in cats]
74 + idset = set(ids)
75 +
76 + # ---- structural validation -------------------------------------------------------------------------------------
77 + if len(cats) != 40:
78 + err(f"expected 40 categories, found {len(cats)}")
79 + if len(idset) != len(ids):
80 + err(f"duplicate category ids: {[i for i, n in Counter(ids).items() if n > 1]}")
81 + if sorted(c["number"] for c in cats) != list(range(1, 41)):
82 + err("category numbers are not exactly 1..40")
83 + for c in cats:
84 + cid = c["id"]
85 + if not re.fullmatch(r"[a-z][a-z0-9_]*", cid):
86 + err(f"{cid}: id is not snake_case")
87 + if c["area"] not in AREAS:
88 + err(f"{cid}: unknown area {c['area']}")
89 + if not isinstance(c.get("definition"), str) or len(c["definition"]) < 60:
90 + err(f"{cid}: definition missing or too short")
91 + if not c.get("provenance"):
92 + err(f"{cid}: provenance missing")
93 + for k in LIST_KEYS:
94 + if not isinstance(c.get(k), list):
95 + err(f"{cid}: field {k} must be a list")
96 + # duplicates inside signal lists
97 + for k, keyfn in (("topics", lambda t: t[0]), ("subfields", lambda t: t[0]), ("arxiv", lambda t: t[0]),
98 + ("streams", lambda t: t[0]), ("keywords", lambda t: t[0])):
99 + keys = [keyfn(t) for t in c[k]]
100 + dup = [x for x, n in Counter(keys).items() if n > 1]
101 + if dup:
102 + err(f"{cid}: duplicate entries in {k}: {dup}")
103 + for tid, w, note in c["topics"]:
104 + if tid not in topics:
105 + err(f"{cid}: unknown OpenAlex topic id {tid}")
106 + if not wt_ok(w):
107 + err(f"{cid}: topic {tid} weight {w} not in (0,1]")
108 + if not note:
109 + err(f"{cid}: topic {tid} has no note")
110 + for sid, w in c["subfields"]:
111 + if sid not in subfields:
112 + err(f"{cid}: unknown subfield id {sid}")
113 + if not wt_ok(w):
114 + err(f"{cid}: subfield {sid} weight {w} not in (0,1]")
115 + for a, w in c["arxiv"]:
116 + if a not in ARXIV:
117 + err(f"{cid}: arXiv category {a} not in admitted list")
118 + if not wt_ok(w):
119 + err(f"{cid}: arXiv {a} weight {w} not in (0,1]")
120 + for s, w in c["streams"]:
121 + if s not in streams:
122 + err(f"{cid}: unknown DBLP stream {s}")
123 + if not wt_ok(w):
124 + err(f"{cid}: stream {s} weight {w} not in (0,1]")
125 + for pat, w, field in c["keywords"]:
126 + try:
127 + re.compile(pat)
128 + except re.error as e:
129 + err(f"{cid}: keyword regex does not compile: {pat[:60]}... ({e})")
130 + if not wt_ok(w):
131 + err(f"{cid}: keyword weight {w} not in (0,1]: {pat[:40]}")
132 + if field not in FIELDS:
133 + err(f"{cid}: keyword field {field!r} invalid")
134 + for pat in c["exclusions"]:
135 + try:
136 + re.compile(pat)
137 + except re.error as e:
138 + err(f"{cid}: exclusion regex does not compile: {pat[:60]}... ({e})")
139 + for k in ("related", "parent_of", "child_of"):
140 + for other in c[k]:
141 + if other not in idset:
142 + err(f"{cid}: {k} references unknown category {other}")
143 + if other == cid:
144 + err(f"{cid}: {k} references itself")
145 + if len(set(c["related"])) != len(c["related"]):
146 + err(f"{cid}: duplicate related entries")
147 + # parent/child symmetry
148 + by_id = {c["id"]: c for c in cats}
149 + for c in cats:
150 + for ch in c["parent_of"]:
151 + if ch in by_id and c["id"] not in by_id[ch]["child_of"]:
152 + err(f"{c['id']} parent_of {ch} but {ch} does not declare child_of {c['id']}")
153 + for p in c["child_of"]:
154 + if p in by_id and c["id"] not in by_id[p]["parent_of"]:
155 + err(f"{c['id']} child_of {p} but {p} does not declare parent_of {c['id']}")
156 + # keyword sufficiency for the categories the brief singles out
157 + must_keyword = {"llm", "foundation_models", "multimodal_ai", "generative_ai", "ai_agents", "ai_safety",
158 + "federated_learning", "sustainable_computing", "autonomous_scientific_discovery",
159 + "privacy_preserving_computation", "blockchain", "quantum_computing", "edge_computing"}
160 + kw_sufficient = {c["id"]: any(float(w) >= 1.0 for _, w, _ in c["keywords"]) for c in cats}
161 + for cid in must_keyword:
162 + if cid in by_id and not kw_sufficient[cid]:
163 + err(f"{cid}: must have at least one keyword pattern of weight 1.0 (threshold reachable alone)")
164 +
165 + # ---- coverage statistics ---------------------------------------------------------------------------------------
166 + topic_cats: dict[str, list[tuple[str, float]]] = defaultdict(list)
167 + for c in cats:
168 + for tid, w, _ in c["topics"]:
169 + topic_cats[tid].append((c["id"], float(w)))
170 + cs_topics = {t for t, r in topics.items() if r["field_id"] == CS_FIELD}
171 + cs_mapped = sorted(t for t in cs_topics if t in topic_cats)
172 + cs_unmapped = sorted(t for t in cs_topics if t not in topic_cats)
173 + reasons = unmapped_mod.UNMAPPED_REASONS
174 + for t in cs_unmapped:
175 + if t not in reasons:
176 + err(f"unmapped CS topic {t} ({topics[t]['display_name']}, subfield {topics[t]['subfield_id']}, "
177 + f"{topics[t]['works_count']} works) has no reason in src/unmapped_cs.py")
178 + for t in reasons:
179 + if t not in topics:
180 + err(f"unmapped_cs.py: unknown topic id {t}")
181 + elif t in topic_cats:
182 + err(f"unmapped_cs.py: topic {t} is listed as unmapped but is mapped to {[c for c, _ in topic_cats[t]]}")
183 + noncs_mapped = sorted(t for t in topic_cats if topics[t]["field_id"] != CS_FIELD)
184 + noncs_by_field = Counter(f"{topics[t]['field_id']} {topics[t]['field']}" for t in noncs_mapped)
185 +
186 + stream_cats: dict[str, list[tuple[str, float]]] = defaultdict(list)
187 + for c in cats:
188 + for s, w in c["streams"]:
189 + stream_cats[s].append((c["id"], float(w)))
190 + ranked = sorted(streams.values(), key=lambda r: -int(r["n"]))
191 + top = [r for r in ranked[:TOP_STREAMS] if r["stream"] != "journals/corr"]
192 + top_mapped = [r for r in top if r["stream"] in stream_cats]
193 + top_unmapped = [r for r in top if r["stream"] not in stream_cats]
194 + total_recs = sum(int(r["n"]) for r in ranked if r["stream"] != "journals/corr")
195 + mapped_recs = sum(int(streams[s]["n"]) for s in stream_cats if s in streams and s != "journals/corr")
196 +
197 + if errors:
198 + print("VALIDATION FAILED")
199 + for e in errors:
200 + print(" -", e)
201 + return 1
202 +
203 + # ---- outputs ---------------------------------------------------------------------------------------------------
204 + import yaml # imported late so validation errors are reported even without pyyaml
205 +
206 + def cat_yaml(c: dict) -> dict:
207 + return {
208 + "id": c["id"], "number": c["number"], "name": c["name"], "area": c["area"], "definition": c["definition"],
209 + "openalex_topics": [{"id": t, "weight": float(w), "note": n} for t, w, n in c["topics"]],
210 + "openalex_subfields": [{"id": int(s), "weight": float(w)} for s, w in c["subfields"]],
211 + "arxiv_categories": [{"id": a, "weight": float(w)} for a, w in c["arxiv"]],
212 + "dblp_streams": [{"key": s, "weight": float(w)} for s, w in c["streams"]],
213 + "keywords": [{"pattern": p, "weight": float(w), "field": f} for p, w, f in c["keywords"]],
214 + "exclusions": list(c["exclusions"]),
215 + "related": list(c["related"]), "parent_of": list(c["parent_of"]), "child_of": list(c["child_of"]),
216 + "provenance": c["provenance"],
217 + }
218 +
219 + area_leaves = defaultdict(list)
220 + for c in cats:
221 + area_leaves[c["area"]].append(c["id"])
222 + doc = {
223 + "version": VERSION, "created": CREATED,
224 + "sources": {"openalex_topics": os.path.basename(TOPICS_CSV), "dblp_streams": os.path.basename(STREAMS_CSV),
225 + "generator": "taxonomy/build_ontology.py", "data_modules": "taxonomy/src/*.py"},
226 + "areas": [{"id": a["id"], "name": a["name"], "description": a["description"], "leaves": area_leaves[a["id"]]}
227 + for a in AREAS.values()],
228 + "categories": [cat_yaml(c) for c in cats],
229 + "assignment_rule": {
230 + "score": "sum of weights of matched signals, each signal type capped at 1.0, then sum over signal types",
231 + "signal_types": list(areas_mod.SIGNAL_TYPES),
232 + "threshold": 1.0,
233 + "primary": "argmax of score; ties broken by the more specific category (child before parent), then by lower number",
234 + "exclusions": "if any exclusion regex matches the concatenated title+abstract, the category is vetoed for that work",
235 + "keyword_field": "title patterns are matched on the title only; abstract on the abstract only; both on title+abstract",
236 + "notes": "OpenAlex topic weights assume the work's primary topic (or any of its up-to-three topics; the classifier "
237 + "takes the max over matched topics before capping). Weights below 1.0 are deliberate: such signals must be "
238 + "corroborated by another signal type to reach the threshold.",
239 + },
240 + }
241 + if write:
242 + with open(OUT_YAML, "w", encoding="utf-8") as fh:
243 + yaml.safe_dump(doc, fh, allow_unicode=True, sort_keys=False, width=1000)
244 + # round-trip check: every regex survives YAML serialisation
245 + back = yaml.safe_load(open(OUT_YAML, encoding="utf-8"))
246 + for c0, c1 in zip(cats, back["categories"]):
247 + for (p, _, _), k in zip(c0["keywords"], c1["keywords"]):
248 + assert p == k["pattern"], f"YAML round-trip altered a regex in {c0['id']}"
249 + re.compile(k["pattern"])
250 + for p, q in zip(c0["exclusions"], c1["exclusions"]):
251 + assert p == q
252 + re.compile(q)
253 + with open(OUT_MAP, "w", encoding="utf-8", newline="") as fh:
254 + w = csv.writer(fh)
255 + w.writerow(["topic_id", "category_id", "weight", "topic_name", "field_id", "field", "subfield_id", "subfield",
256 + "works_count", "note"])
257 + for c in cats:
258 + for tid, wt, note in sorted(c["topics"], key=lambda t: (-float(t[1]), t[0])):
259 + r = topics[tid]
260 + w.writerow([tid, c["id"], f"{float(wt):.2f}", r["display_name"], r["field_id"], r["field"],
261 + r["subfield_id"], r["subfield"], r["works_count"], note])
262 +
263 + # ---- generated markdown ----------------------------------------------------------------------------------------
264 + L: list[str] = []
265 + L.append(f"_Generated by `taxonomy/build_ontology.py` on {dt.date.today().isoformat()} from "
266 + f"`{os.path.basename(TOPICS_CSV)}` ({len(topics):,} topics) and `{os.path.basename(STREAMS_CSV)}` "
267 + f"({len(streams):,} streams). Do not edit between the markers._\n")
268 + L.append("## Areas and leaves\n")
269 + L.append("| Area | Name | Leaves |")
270 + L.append("|---|---|---|")
271 + for a in AREAS.values():
272 + L.append(f"| `{a['id']}` | {a['name']} | {', '.join('`'+x+'`' for x in area_leaves[a['id']])} |")
273 + L.append("\n## The 40 categories\n")
274 + for c in cats:
275 + L.append(f"**{c['number']}. {c['name']}** (`{c['id']}`, area `{c['area']}`) ")
276 + L.append(f"{c['definition']} ")
277 + edges = []
278 + if c["child_of"]:
279 + edges.append("child of " + ", ".join(f"`{x}`" for x in c["child_of"]))
280 + if c["parent_of"]:
281 + edges.append("parent of " + ", ".join(f"`{x}`" for x in c["parent_of"]))
282 + if c["related"]:
283 + edges.append("related to " + ", ".join(f"`{x}`" for x in c["related"]))
284 + L.append("_" + "; ".join(edges) + "._\n")
285 + L.append("## Hierarchy (parent → child edges)\n")
286 + for c in cats:
287 + for ch in c["parent_of"]:
288 + L.append(f"- `{c['id']}` → `{ch}`")
289 + L.append("\n## Signal counts per category\n")
290 + L.append("| # | id | topics | subfields | arXiv | DBLP streams | keywords | exclusions | keyword alone reaches threshold |")
291 + L.append("|---|---|---|---|---|---|---|---|---|")
292 + for c in cats:
293 + L.append(f"| {c['number']} | `{c['id']}` | {len(c['topics'])} | {len(c['subfields'])} | {len(c['arxiv'])} | "
294 + f"{len(c['streams'])} | {len(c['keywords'])} | {len(c['exclusions'])} | {'yes' if kw_sufficient[c['id']] else 'no'} |")
295 + L.append(f"\nTotals: {sum(len(c['topics']) for c in cats)} topic→category weights over {len(topic_cats)} distinct topics; "
296 + f"{sum(len(c['streams']) for c in cats)} stream→category weights over {len(stream_cats)} distinct streams; "
297 + f"{sum(len(c['keywords']) for c in cats)} keyword patterns; {sum(len(c['exclusions']) for c in cats)} exclusion patterns.\n")
298 + L.append("## OpenAlex coverage\n")
299 + L.append(f"- Field 17 (Computer Science): **{len(cs_mapped)} of {len(cs_topics)} topics mapped** "
300 + f"({100*len(cs_mapped)/len(cs_topics):.1f} %); {len(cs_unmapped)} left unmapped with an explicit reason below.")
301 + L.append(f"- Topics mapped from other fields: {len(noncs_mapped)} — " +
302 + "; ".join(f"{k}: {v}" for k, v in sorted(noncs_by_field.items())) + ".")
303 + multi = sum(1 for t, l in topic_cats.items() if len(l) > 1)
304 + L.append(f"- Topics mapped to more than one category: {multi} of {len(topic_cats)}.")
305 + L.append("\n### Mapped CS topics per subfield\n")
306 + L.append("| Subfield | topics | mapped | unmapped |")
307 + L.append("|---|---|---|---|")
308 + bysf = defaultdict(lambda: [0, 0])
309 + for t in cs_topics:
310 + bysf[(topics[t]["subfield_id"], topics[t]["subfield"])][0 if t in topic_cats else 1] += 1
311 + for (sid, sname), (m, u) in sorted(bysf.items()):
312 + L.append(f"| {sid} {sname} | {m+u} | {m} | {u} |")
313 + L.append("\n### Unmapped CS topics (field 17) with reason\n")
314 + L.append("| topic | name | subfield | works | reason |")
315 + L.append("|---|---|---|---|---|")
316 + for t in sorted(cs_unmapped, key=lambda t: -int(topics[t]["works_count"])):
317 + r = topics[t]
318 + L.append(f"| {t} | {r['display_name']} | {r['subfield_id']} | {int(r['works_count']):,} | {reasons[t]} |")
319 + L.append("\n## DBLP coverage\n")
320 + L.append(f"- Streams mapped: **{len(stream_cats)}** (any rank); records covered (2018–2026, excluding CoRR): "
321 + f"{mapped_recs:,} of {total_recs:,} ({100*mapped_recs/total_recs:.1f} %).")
322 + L.append(f"- Of the {len(top)} largest streams (CoRR excluded), {len(top_mapped)} are mapped "
323 + f"({100*len(top_mapped)/len(top):.1f} %). The {len(top_unmapped)} unmapped large streams are listed below; they "
324 + f"are either out of scope (medicine, geoscience, education, management, electronics) or too generic to be a "
325 + f"category signal (IEEE Access, Sensors, Symmetry).")
326 + L.append("\n| stream | records | name |")
327 + L.append("|---|---|---|")
328 + for r in top_unmapped:
329 + L.append(f"| `{r['stream']}` | {int(r['n']):,} | {(r['journal'] or r['booktitle'])} |")
330 + L.append("\n## Topics mapped to several categories (audit list)\n")
331 + L.append("| topic | name | categories (weight) |")
332 + L.append("|---|---|---|")
333 + for t in sorted(topic_cats, key=lambda t: (-len(topic_cats[t]), t)):
334 + if len(topic_cats[t]) > 1:
335 + L.append(f"| {t} | {topics[t]['display_name']} | " +
336 + ", ".join(f"`{c}` ({w:.1f})" for c, w in sorted(topic_cats[t], key=lambda x: -x[1])) + " |")
337 + generated = "\n".join(L) + "\n"
338 +
339 + if write:
340 + if os.path.exists(OUT_MD):
341 + text = open(OUT_MD, encoding="utf-8").read()
342 + if MARK_BEGIN in text and MARK_END in text:
343 + pre = text.split(MARK_BEGIN)[0]
344 + post = text.split(MARK_END)[1]
345 + text = pre + MARK_BEGIN + "\n" + generated + MARK_END + post
346 + else:
347 + text = text.rstrip() + "\n\n" + MARK_BEGIN + "\n" + generated + MARK_END + "\n"
348 + else:
349 + text = ("# WP44 Ontology of Computer-Science Research\n\n(prose to be written)\n\n" + MARK_BEGIN + "\n" +
350 + generated + MARK_END + "\n")
351 + open(OUT_MD, "w", encoding="utf-8").write(text)
352 +
353 + # ---- console summary -------------------------------------------------------------------------------------------
354 + print(f"OK: 40 categories, {len(topic_cats)} topics, {len(stream_cats)} streams, "
355 + f"{sum(len(c['keywords']) for c in cats)} keyword patterns")
356 + print(f"CS topics mapped {len(cs_mapped)}/{len(cs_topics)}; non-CS topics mapped {len(noncs_mapped)}; "
357 + f"top-{TOP_STREAMS} streams mapped {len(top_mapped)}/{len(top)}; records covered {100*mapped_recs/total_recs:.1f}%")
358 + print("per category (topics/streams/keywords):")
359 + for c in cats:
360 + print(f" {c['number']:2d} {c['id']:32s} {len(c['topics']):3d} {len(c['streams']):3d} {len(c['keywords']):3d}"
361 + f"{'' if kw_sufficient[c['id']] else ' (no 1.0 keyword)'}")
362 + return 0
363 +
364 +
365 +if __name__ == "__main__":
366 + sys.exit(main(write="--no-write" not in sys.argv))
modified taxonomy/src/area_ai_ml_b.py +5 −2
@@ -85,14 +85,17 @@ dict(
85 85 ("T12761", 0.3, "data stream mining (concept drift)"),
86 86 ("T12205", 0.3, "time series forecasting"),
87 87 ("T13373", 0.2, "data mining and ML applications"),
88 − ("T12676", 0.2, "extreme learning machines")],
88 + ("T12676", 0.2, "extreme learning machines"),
89 + ("T11512", 0.6, "anomaly detection techniques and applications"),
90 + ("T10820", 0.3, "fuzzy logic and control systems (soft computing)"),
91 + ("T11063", 0.3, "rough sets and fuzzy logic (granular computing, feature selection)")],
89 92 subfields=[(1702, 0.3)],
90 93 arxiv=[("stat.ML", 0.8), ("cs.LG", 0.4), ("math.ST", 0.3), ("math.OC", 0.2)],
91 94 streams=[("conf/colt", 1.0), ("conf/alt", 1.0), ("journals/jmlr", 0.8), ("conf/aistats", 0.8), ("conf/uai", 0.8),
92 95 ("journals/ml", 0.6), ("journals/tmlr", 0.4), ("conf/nips", 0.3), ("conf/icml", 0.3), ("conf/iclr", 0.2),
93 96 ("journals/tnn", 0.3), ("journals/nn", 0.3), ("journals/pr", 0.2), ("conf/pkdd", 0.3), ("conf/kdd", 0.2),
94 97 ("conf/icdm", 0.2), ("journals/pami", 0.2), ("journals/ijar", 0.4), ("journals/sac", 0.4),
95 − ("journals/csda", 0.3), ("journals/ml", 0.6), ("journals/cstat", 0.2), ("journals/eswa", 0.1),
98 + ("journals/csda", 0.3), ("journals/cstat", 0.2), ("journals/eswa", 0.1),
96 99 ("journals/kbs", 0.1), ("journals/isci", 0.1), ("journals/ijon", 0.2), ("journals/nca", 0.1),
97 100 ("journals/mlc", 0.3), ("journals/apin", 0.2), ("journals/air", 0.2)],
98 101 keywords=[
modified taxonomy/src/area_compsci.py +3 −2
@@ -28,7 +28,8 @@ dict(
28 28 ("T10541", 0.2, "nonlinear differential equations analysis"), ("T10194", 0.2, "nonlinear PDEs"),
29 29 ("T12589", 0.2, "differential equations and boundary problems"), ("T10288", 0.2, "fractional differential equations"),
30 30 ("T11152", 0.1, "stochastic processes and statistical mechanics"), ("T12809", 0.2, "contact mechanics and variational inequalities"),
31 − ("T13734", 0.1, "advanced computational techniques (expert systems)"), ("T13608", 0.3, "advanced computational techniques in science and engineering")],
31 + ("T13734", 0.1, "advanced computational techniques (expert systems)"), ("T13608", 0.3, "advanced computational techniques in science and engineering"),
32 + ("T13398", 0.4, "data analysis with R (statistical and mathematical software packages)")],
32 33 subfields=[(1703, 0.1)],
33 34 arxiv=[("cs.NA", 1.0), ("cs.MS", 1.0), ("cs.CE", 1.0), ("physics.comp-ph", 0.8), ("math.OC", 0.2), ("cs.DC", 0.1)],
34 35 streams=[("journals/siamsc", 1.0), ("journals/jcphy", 1.0), ("journals/cphysics", 1.0), ("journals/jcam", 1.0), ("journals/na", 1.0),
@@ -42,7 +43,7 @@ dict(
42 43 ("journals/siamma", 0.3), ("journals/cacie", 0.3), ("journals/cmpb", 0.3), ("journals/csda", 0.3), ("journals/cssc", 0.3),
43 44 ("journals/cmc", 0.3), ("journals/axioms", 0.2), ("journals/mp", 0.2), ("journals/jota", 0.2), ("conf/ipps", 0.2),
44 45 ("journals/ecoi", 0.2), ("journals/jns", 0.2), ("journals/tpds", 0.1), ("journals/jpdc", 0.1), ("journals/siamjo", 0.1),
45 − ("journals/symmetry", 0.1), ("journals/entropy", 0.1), ("journals/mathematics", 0.0)],
46 + ("journals/symmetry", 0.1), ("journals/entropy", 0.1)],
46 47 keywords=[
47 48 (r"(?i)\b(scientific|numerical|high-?performance|computational|simulation) (computing|simulations?|methods?|analysis|modell?ing|software|linear algebra|workflows?|physics|science|fluid dynamics|mechanics|chemistry|materials science|engineering|electromagnetics|acoustics|astrophysics|geoscience|seismology|codes?|libraries|kernels?|applications?|frameworks?|solvers?|schemes?|algorithms?|experiments?|studies)\b|\bscientific (machine learning|ML|software|codes?|applications?|workflows?|visuali[sz]ation)\b|\bSciML\b", 1.0, "both"),
48 49 (r"(?i)\b(finite (element|difference|volume) (methods?|schemes?|discreti[sz]ations?|analysis|solvers?|models?|simulations?)|FEM\b|FEA\b|FDM\b|FVM\b|isogeometric|spectral (methods?|elements?)|discontinuous Galerkin|DG methods?|multigrid|algebraic multigrid|AMG\b|domain decomposition|preconditioners?|preconditioning|Krylov|GMRES|BiCGStab|conjugate gradient|CG solvers?|sparse (linear (systems?|solvers?)|matrix|matrices|direct solvers?|factori[sz]ation)|linear solvers?|iterative solvers?|eigenvalue (problems?|solvers?|computations?)|eigensolvers?|Lanczos|Arnoldi|SVD\b|singular value decomposition|QR (factori[sz]ation|decomposition|algorithm)|LU (factori[sz]ation|decomposition)|Cholesky|BLAS|LAPACK|ScaLAPACK|MAGMA|cuBLAS|PETSc|Trilinos|hypre|FEniCS|deal\.II|MFEM|Firedrake|MOOSE|OpenFOAM|LAMMPS|GROMACS|NAMD|AMBER|Quantum ESPRESSO|VASP|CP2K|Gaussian \d\d|ORCA|GAMESS|NWChem|molecular dynamics (simulations?|codes?|software|packages?|engines?)|MD simulations?|Monte Carlo (simulations?|methods?|integration|sampling)|lattice Boltzmann|particle-?in-?cell|PIC (simulations?|codes?)|smoothed particle|SPH\b|N-?body|fast multipole|FMM\b|Barnes-?Hut|adaptive mesh refinement|AMR\b|mesh (generation|refinement|adaptation|quality)|unstructured (meshes?|grids?)|structured grids?|stencil (computations?|codes?)|time[- ]stepping|time integration|Runge-?Kutta|implicit (schemes?|solvers?|time)|ODE solvers?|PDE solvers?|partial differential equations?|PDEs?\b|ordinary differential equations?|ODEs?\b|stiff (systems|equations|ODEs)|boundary value problems?|initial value problems?|uncertainty quantification|UQ\b|polynomial chaos|stochastic (collocation|Galerkin|PDEs?)|sensitivity analysis|Sobol|reduced[- ]order model(s|ing)?|ROM\b|model order reduction|MOR\b|proper orthogonal decomposition|POD\b|dynamic mode decomposition|DMD\b|surrogate model(s|ing)?|emulators?|Gaussian process emulat|Kriging for simulation|physics-?informed (neural|machine learning|deep)|PINNs?\b|neural operators?|Fourier neural operator|FNO\b|DeepONet|operator learning|neural (PDE|ODE) solvers?|differentiable (physics|simulation|programming|solvers?)|automatic differentiation|adjoint (methods?|sensitivity|solvers?)|inverse problems?|data assimilation|Kalman filter for (simulation|assimilation)|ensemble Kalman|4D-?Var|floating-?point (arithmetic|precision|errors?|formats?)|mixed-?precision (arithmetic|solvers?|algorithms?|iterative)|interval arithmetic|verified (numerics|computing)|rounding errors?|numerical (stability|accuracy|precision|errors?)|computer algebra|symbolic computation|symbolic-?numeric|quadrature|cubature|numerical integration|interpolation (schemes?|methods?)|splines?|radial basis functions?|RBF\b|tensor (decompositions?|trains?|networks? for (PDE|simulation)|methods? for)|low-?rank (approximation|tensor)|hierarchical matrices|H-?matrices|fast (solvers?|transforms?)|FFT\b|fast Fourier|HPC (applications?|codes?|software|workflows?|systems? for science)|exascale (computing|applications?|software|codes?)|ECP\b|petascale|supercomput\w+|performance portability|Kokkos|RAJA|SYCL|OpenACC|OpenMP offload|CUDA (for|implementation of) (simulation|solvers?|PDE)|GPU-?accelerated (simulation|solvers?|computing|codes?|linear algebra)|GPU (implementation|acceleration) of (numerical|simulation|solvers?|FEM|molecular)|co-?design of (scientific|HPC)|scientific (workflow|data) management|in[- ]situ (visuali[sz]ation|analysis)|checkpoint(ing)? (for|of) (simulation|HPC)|research software (engineering|sustainability)|RSE\b|reproducib(le|ility) (of|in) (computational|scientific|simulation))\b", 1.0, "both"),
modified taxonomy/src/area_hardware_b.py +1 −1
@@ -23,7 +23,7 @@ dict(
23 23 ("conf/date", 0.1), ("journals/tc", 0.1), ("conf/isca", 0.1), ("conf/micro", 0.1), ("conf/hpca", 0.1),
24 24 ("conf/asplos", 0.1), ("journals/jcphy", 0.1), ("journals/cphysics", 0.1), ("journals/symmetry", 0.1),
25 25 ("conf/stoc", 0.1), ("conf/focs", 0.1), ("journals/natmi", 0.05), ("journals/ncs", 0.1), ("journals/mlst", 0.1),
26 − ("conf/sc", 0.1), ("journals/tpds", 0.05), ("conf/ipps", 0.05), ("journals/tqc", 0.0)],
26 + ("conf/sc", 0.1), ("journals/tpds", 0.05), ("conf/ipps", 0.05)],
27 27 keywords=[
28 28 (r"(?i)\bquantum (comput(ing|ers?|ation|ational)|algorithms?|circuits?|gates?|processors?|hardware|supremacy|advantage|error[- ]correct(ion|ing)|error mitigation|codes?|annealing|annealers?|machine learning|neural networks?|kernels?|simulation|simulators?|emulat\w+|information|key distribution|cryptograph\w*|communication|internet|networks?|networking|teleportation|entanglement distribution|repeaters?|memor(y|ies)|sensing|metrology|software|programming|compil(ers?|ation|ing)|control|optimi[sz]ation|chemistry (simulation|algorithms?|on quantum)|walks?|Fourier|phase estimation|approximate optimi[sz]ation|variational|bits?|speed-?ups?|volume|random (circuits?|sampling)|dots? (for|as) qubits?|devices? for computing|technolog(y|ies)|instruction sets?|transpil\w+|benchmark\w*|cloud|service|workflows?|HPC|data (centers?|loading)|RAM|QRAM|generative|reinforcement|transformers?|Boltzmann|support vector|convolutional|federated|language models?|LLMs?|finance|chemistry|many-?body simulation on|advantage experiments?|readout|crosstalk|calibration|characteri[sz]ation|tomography|process tomography|state preparation|amplitude (amplification|estimation|encoding)|linear (systems?|algebra) algorithms?|singular value|eigensolvers?|dynamics simulation|Monte Carlo (acceleration|algorithms)|game theory|secret sharing|money|signatures?|commitment|oblivious|zero-?knowledge|interactive proofs?|complexity|Turing|cellular automata|dot cellular automata|LDPC|surface codes?|resource (estimation|theory for computing)|logic synthesis|layout synthesis|mapping|routing (for|of) qubits?|qubit (mapping|routing|allocation))\b", 1.0, "both"),
29 29 (r"(?i)\b(qubits?|qudits?|QAOA|VQE\b|variational quantum|VQAs?\b|NISQ|surface codes?|colou?r codes?|stabili[sz]er codes?|toric codes?|CSS codes?|qLDPC|quantum LDPC|magic state|T-?gates?|T-?count|Clifford\+T|fault-?tolerant quantum|logical qubits?|physical qubits?|transmons?|superconducting qubits?|flux(onium)? qubits?|trapped[- ]ions?|ion[- ]traps?|neutral atoms?|Rydberg (atoms?|arrays?|blockade)|photonic (qubits?|quantum)|boson sampling|Gaussian boson|spin qubits?|silicon qubits?|quantum dots? qubits?|NV cent(er|re)s? qubits?|Majorana (zero modes|qubits?|fermions for)|topological (qubits?|quantum comput)|anyons? (for|in) (computing|braiding)|braiding|Shor'?s algorithm|Grover'?s? (algorithm|search)|HHL\b|Harrow-?Hassidim|quantum Fourier transform|QFT\b|Hamiltonian simulation|Trotteri[sz]ation|Trotter (steps?|error)|qubitization|block encoding|LCU\b|linear combination of unitaries|QSVT|quantum signal processing|amplitude amplification|QKD\b|BB84|E91|B92|decoy state|measurement-?device-?independent|entanglement (swapping|purification|distillation|witnesses?|entropy)|Bell (states?|inequalit(y|ies)|tests?|pairs?)|GHZ states?|cluster states?|measurement-?based quantum|one-?way quantum|decoherence|dephasing|T1\b|T2\b|coherence times?|gate fidelit(y|ies)|two-?qubit gates?|CNOT|CZ gates?|iSWAP|Toffoli|randomized benchmarking|cross-?entropy benchmarking|XEB\b|gate set tomography|quantum (noise|channels?|Shannon|capacity|discord|coherence resource|error channels?)|depolarizing (noise|channel)|Pauli (noise|twirling|frames?)|post-?quantum|quantum-?safe|Qiskit|Cirq|PennyLane|QuTiP|OpenQASM|QASM|Q#\b|Quipper|Silq|Braket|tket|Quantinuum|IonQ|Rigetti|D-?Wave|IBM Quantum|Google Sycamore|Sycamore|Zuchongzhi|Jiuzhang|Quantum Volume|CLOPS|cross-?resonance|ZZ (coupling|crosstalk)|tunable couplers?|Josephson (junctions?|parametric)|dilution refrigerator (for|and) (qubits?|quantum)|cryo-?CMOS|cryogenic control|quantum-?classical (hybrid|interface|co-?design)|hybrid quantum|tensor networks? (simulation|contraction|methods?) (of|for) quantum (circuits?|computing)|matrix product states? (for|and) quantum (circuits?|computing)|Schrödinger-?Feynman|stabili[sz]er simulation|Clifford (circuits?|gates?|group|simulation)|Ising machines?|coherent Ising|quantum-?inspired|annealing (schedules?|hardware)|adiabatic quantum|QUBO\b|quadratic unconstrained binary)\b", 1.0, "both"),
modified taxonomy/src/area_security_a.py +2 −1
@@ -24,7 +24,8 @@ dict(
24 24 ("T11598", 0.4, "Internet traffic analysis and secure e-voting"), ("T10951", 0.4, "cryptographic implementations (side channels)"),
25 25 ("T11017", 0.3, "chaos-based image encryption"), ("T12943", 0.3, "COVID-19 contact tracing"),
26 26 ("T13270", 0.2, "QR code applications"), ("T10270", 0.2, "blockchain security"),
27 − ("T12131", 0.2, "wireless signal modulation classification (spectrum security)")],
27 + ("T12131", 0.2, "wireless signal modulation classification (spectrum security)"),
28 + ("T11512", 0.3, "anomaly detection techniques (intrusion/fraud detection)")],
28 29 subfields=[],
29 30 arxiv=[("cs.CR", 1.0)],
30 31 streams=[("conf/ccs", 1.0), ("conf/uss", 1.0), ("conf/sp", 1.0), ("conf/ndss", 1.0), ("conf/esorics", 1.0), ("conf/raid", 1.0),
added taxonomy/src/unmapped_cs.py +50 −0
@@ -0,0 +1,50 @@
1 +"""Reasons for the OpenAlex field-17 (Computer Science) topics deliberately left out of topic_map.csv.
2 +
3 +build_ontology.py fails if a field-17 topic is neither mapped by an area module nor listed here, and if a topic listed
4 +here is in fact mapped. Reasons are short and auditable; the grouping mirrors why OpenAlex places the cluster in
5 +field 17 although it is not computer-science research in the sense of CLAUDE.md section 4.
6 +"""
7 +
8 +_EDU = ("Education / e-learning research (pedagogy, learning outcomes, teacher development); not a CS research category "
9 + "of CLAUDE.md section 4. Educational technology as software is captured through software_engineering/hci venues "
10 + "and keywords when the contribution is technical.")
11 +_MEDIA = "Media, communication or cultural studies cluster placed in field 17 by OpenAlex; not computer-science research."
12 +_PHYS = "Physical-science or engineering cluster (no computing contribution) placed in field 17 by OpenAlex."
13 +_GENERIC = ("Generic, uninformative cluster label (OpenAlex residual topic) whose keyword list has no specific "
14 + "computer-science content; works land in categories via venue, arXiv and keyword signals instead.")
15 +_EMPTY = "Residual OpenAlex topic with 0 works in the export; nothing to map."
16 +_OTHER = "Out-of-scope discipline (psychiatry, economics, healthcare policy, literature, environmental engineering)."
17 +
18 +UNMAPPED_REASONS = {
19 + # education and e-learning (subfields 1702, 1705, 1706, 1710)
20 + **{t: _EDU for t in ["T11370", "T12171", "T12395", "T12408", "T13016", "T13081", "T13386", "T13482", "T13559",
21 + "T13565", "T13602", "T13676", "T13873", "T13932", "T14025", "T14103", "T14216", "T14246",
22 + "T14403", "T14409", "T14430", "T14455", "T14496", "T14498", "T14516", "T14519", "T13836",
23 + "T13964", "T13567"]},
24 + # media / communication / culture
25 + "T13745": _MEDIA + " (204,789 works: 'Media and Digital Communication', the largest unmapped CS topic.)",
26 + "T13194": "ICT for development / digital divide studies (development studies); " + _MEDIA.lower(),
27 + "T14029": "Literary studies ('Thoreau and American Literature'); " + _MEDIA.lower(),
28 + "T14345": "Philosophy of science and research methodology; " + _MEDIA.lower(),
29 + "T12797": "History of computing technologies (history of science); kept out of the research-category map; could feed Part II background.",
30 + "T12805": "Cognitive science and mental mapping (psychology/geography); " + _OTHER.lower(),
31 + # physical sciences / engineering clusters
32 + "T11276": _PHYS + " (solar radiation and photovoltaics)",
33 + "T12157": _PHYS + " (geochemistry and geologic mapping)",
34 + "T13018": _PHYS + " (seismology and earthquake studies)",
35 + "T11187": _PHYS + " (nonlinear dynamics and pattern formation; mathematical physics)",
36 + "T12564": _PHYS + " (sensor technology and measurement instrumentation)",
37 + "T13615": _PHYS + " (transportation systems and safety; traffic engineering)",
38 + "T13935": _PHYS + " (mathematical control systems; control theory without computing content)",
39 + "T14215": _PHYS + " (chemical and environmental engineering)",
40 + "T14387": _PHYS + " (environmental engineering and cultural studies, 2 works)",
41 + # other disciplines
42 + "T14381": _OTHER + " (psychiatry, mental health, neuroscience)",
43 + "T13867": _OTHER + " (economic growth and development)",
44 + "T14071": _OTHER + " (healthcare during the COVID-19 pandemic)",
45 + "T13062": ("Cognitive informatics / brain informatics (concept algebra, abstract intelligence): a small, idiosyncratic "
46 + "cluster with no counterpart in the 40 categories; left unmapped rather than forced into ai_agents."),
47 + # generic or residual clusters
48 + **{t: _GENERIC for t in ["T13674", "T13523", "T13898", "T14351", "T13345"]},
49 + **{t: _EMPTY for t in ["T13514", "T13693", "T13748", "T14064", "T14155", "T14174", "T14175", "T14413", "T14431"]},
50 +}
51
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere