SPB Git
spboucher.ai

spb/wp44_uqo

Public

WP44: figure batch 2, embeddings multi-node drivers, semantic atlas, writer briefs

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 3, 2026) parent a5b49ef Browse files

28 changed files +333 −6

28 changed files
Unified Split
added agents/briefs/AGENT_K_CRITIC.md +12 −0
@@ -0,0 +1,12 @@
1 +# Brief — Agent K (Scientific Critic)
2 +
3 +You receive a structured artefact (a specialist review directory, an econometric result set, or a manuscript part) and
4 +return `reports/critic/<artefact>_critique.md` + `<artefact>_issues.csv` (issue_id, location, severity ∈ {blocking, major,
5 +minor}, type ∈ {unsupported_claim, causal_language, missing_uncertainty, metadata_as_fulltext, preprint_as_peer_reviewed,
6 +number_without_artifact, citation_unverified, overclaim, inconsistency, method_weakness}, quote, required_fix).
7 +Checks (CLAUDE.md §1, §10): every number traces to a file; every claim has evidence_type and supporting keys; no causal
8 +language about ChatGPT ("after" not "because"); preprints never called peer-reviewed; abstracts never treated as full text;
9 +"largest ever" and similar superlatives absent unless independently established; uncertainty reported; competing
10 +explanations considered (indexing lag, OpenAlex classifier drift, venue growth, XPAC, duplicates). You may run the scripts to
11 +re-derive a number. Verdict: ACCEPT / ACCEPT WITH FIXES / REJECT, with the blocking issues listed first. You never edit the
12 +artefact yourself.
added agents/briefs/WRITER_COMMON.md +26 −0
@@ -0,0 +1,26 @@
1 +# Common rules for manuscript writers (Agent L instances), WP44
2 +
3 +Repository: /Users/simon-pierreboucher/Desktop/UQO/Recherche/Recherche-UQO/informatique/wp44 . Read CLAUDE.md §1, §9, §10 and
4 +RESEARCH_PLAN.md first. The manuscript is `manuscript/main.tex` (UQO template, natbib author-year, `\bibliographystyle{aer}`,
5 +`\part{}` already declared in main.tex — you write `\section`/`\subsection` content only, in the file assigned to you).
6 +
7 +Style (CLAUDE.md §10): formal academic English, critical, precise, no promotional language, no repetitive AI prose (vary
8 +sentence structure, no "In this section we…" openers in every subsection, no bullet lists in body prose except where a list is
9 +the natural structure), real mathematical notation where it carries meaning, assumptions and limitations explicit. Length:
10 +the target page count for your part is given in your brief; do not pad.
11 +
12 +Citations: `\citep{key}` / `\citet{key}`. You may cite only keys that exist in `manuscript/references.bib` (being built by
13 +another agent — check it) OR keys you add to YOUR OWN bib file `manuscript/references_<part>.bib`, every entry verified
14 +by a Crossref/OpenAlex/arXiv single lookup (`curl -s "https://api.crossref.org/works/<DOI>?mailto=spbou4@icloud.com"`,
15 +`https://api.openalex.org/works/https://doi.org/<DOI>`, `https://export.arxiv.org/api/query?id_list=<id>`), recorded in
16 +`manuscript/references_verification_<part>.csv` (key, doi_or_arxiv, verified_via, verified_title, match). Never invent a
17 +reference. If you cannot verify, do not cite. Keys: `surnameYYYYkeyword`. Main-bib keys you reuse: do not redefine them.
18 +
19 +Numbers: never type a corpus number. Use the macros of `manuscript/tables/numbers.tex` (generated; list them with
20 +`grep newcommand manuscript/tables/numbers.tex`). If a number you need has no macro, write `\pending{macro name}` and list
21 +it in your report. Results sections are written by others; do not describe empirical findings of WP44 beyond the macros.
22 +
23 +Integrity: no causal claim about ChatGPT; preprints are "preprints"; what you know only from an abstract is said so; no
24 +"largest review ever". Deliverable = your .tex file(s), your bib + verification CSV, and a report ≤ 20 lines (what you wrote,
25 +word count, citations added, pending macros, anything uncertain). Compile check: `cd manuscript && latexmk -pdf -interaction=nonstopmode main.tex`
26 +may fail on other parts — check only that YOUR file has no LaTeX syntax errors (e.g. with `lacheck` or by including it alone in a scratch document).
added agents/briefs/WRITER_PART02.md +18 −0
@@ -0,0 +1,18 @@
1 +# Brief — Part II "The Evolution of Computer Science" (file manuscript/sections/part02_evolution.tex; bib manuscript/references_part02.bib)
2 +
3 +Target 25–35 pages (≈ 12,000–16,000 words). Purpose (CLAUDE.md §9): the conceptual and historical background needed to
4 +interpret post-2022 developments — not a history of computing in general. Suggested sections (adapt):
5 +1. The disciplinary structure of computer science (ACM CCS, the 40-category ontology of this paper, how fields relate; how
6 + bibliometric field classifications — OpenAlex topics, DBLP venues, arXiv categories — encode that structure and where they disagree).
7 +2. Three prior transformations and how they propagated across subfields: the internet/web era, the deep-learning wave
8 + (2012–2017: AlexNet, ResNet, seq2seq, attention), the transformer/pretraining era (2017–2022: Transformer, BERT, GPT-2/3,
9 + scaling laws, CLIP, diffusion, AlphaFold). For each: what changed in publication practice (preprints, benchmarks, industry labs).
10 +3. The state of each non-AI area at the baseline (2018–2022): systems (cloud, serverless, disaggregation), hardware
11 + (end of Dennard scaling, accelerators, "golden age"), theory (fine-grained complexity, learning theory), PL/verification
12 + (Rust, proof assistants), security/crypto (post-quantum standardisation, ML security), HCI, graphics (NeRF), quantum (NISQ),
13 + bioinformatics (AlphaFold 2) — concise, citation-backed.
14 +4. November 2022 as a reference date: what ChatGPT was and was not (an interface + RLHF over GPT-3.5), the preceding
15 + releases (InstructGPT, Codex, Stable Diffusion), why the date is used descriptively and not causally (CLAUDE.md §6.5).
16 +5. What the literature already claims about the post-2022 transformation (growth of AI papers, LLM-assisted writing, arXiv
17 + submission surges, benchmark saturation, industry concentration) and why those claims need the corpus-level test of this paper.
18 +Landmark references: verify each (group G/H of the reference brief). Coordinate with `manuscript/references.bib` if present.
added agents/briefs/WRITER_PART03.md +15 −0
@@ -0,0 +1,15 @@
1 +# Brief — Part III "Data and Systematic Review Protocol" (file manuscript/sections/part03_data.tex; bib manuscript/references_part03.bib)
2 +
3 +Target 25–35 pages. Explain corpus construction, source coverage, deduplication, eligibility and research integrity
4 +(CLAUDE.md §9, PRISMA 2020 items where applicable). Sources of truth to read: RESEARCH_PLAN.md §2–§3, protocol/PROTOCOL.md,
5 +protocol/amendments.csv, reports/PHASE0_FEASIBILITY.md, data/raw/README.md, ingestion/*.py (what each does, measured sizes),
6 +systematic_review/define_corpus.py (exclusion codes E1–E7, the CS-core rule, date precision), data/metadata/linkage_report.json,
7 +data/metadata/prisma_counts.json, figures/FIGURES.json (fig_prisma_flow, fig_volume_yearly, fig_types_openness exist in
8 +manuscript/figures/ — reference them with \includegraphics{fig_prisma_flow.pdf} etc.; copy from figures/ to manuscript/figures/ if missing).
9 +Sections: 1 sources and access (licences, snapshot release, API budget facts, Semantic Scholar unavailable); 2 identification
10 +(what each source contributes, sizes via macros); 3 linkage (methods, counts via macros, the second snapshot pass and why);
11 +4 deduplication of preprint/published pairs; 5 eligibility (codes, counts, decisions A-001/A-002 with their evidence);
12 +6 the CS-core vs penumbra design and the strict core; 7 date precision, indexing lag, right truncation (created_date is
13 +unusable for lag — explain); 8 quality indicators and known biases (DBLP coverage, OpenAlex topic noise, XPAC, Zenodo flood,
14 +missing abstracts, forward-dated records); 9 research-integrity procedures (preregistration, amendments log, human checkpoints,
15 +agent register, no fabrication rules). Every number through macros (`\PrismaCsCoreEligible`, `\LinkDblpLinkedDistinct`, …).
added agents/briefs/WRITER_PART04.md +27 −0
@@ -0,0 +1,27 @@
1 +# Brief — Part IV "Computational Methodology" (file manuscript/sections/part04_methodology.tex; bib manuscript/references_part04.bib)
2 +
3 +Target 35–50 pages (≈ 15,000–22,000 words) with real mathematics. Document EXACTLY what the code does — read these files
4 +before writing: RESEARCH_PLAN.md §3–§7, protocol/PROTOCOL.md, ingestion/link_sources.py, systematic_review/define_corpus.py,
5 +taxonomy/classify.py + taxonomy/ontology.yaml (assignment_rule) + taxonomy/VALIDATION.md, bibliometrics/01–03, knowledge_graph/01–03,
6 +econometrics/structural_change.py + run_structural_change.py, embeddings/embed_corpus.py + semantic_atlas.py,
7 +experiments/emerging_detection.py. Where the code and the plan differ, describe the code and flag the difference in your report.
8 +Sections:
9 +1. Notation and the corpus as a measurable object (works w, dates, categories C(w) ⊆ 𝒦, periods B/P, month-precision filter).
10 +2. Record linkage and deduplication (deterministic DOI/arXiv keys, normalised-title blocking, precision estimation design).
11 +3. Multi-label classification: score s_k(w) = Σ_signal min(1, Σ weights), threshold, primary rule, validation metrics
12 + (precision/recall/F1, Cohen's κ, Jaccard), and the known failure modes.
13 +4. Bibliometric indicators with definitions and estimators: counts, shares, CAGR, fixed-window citations c12/c36,
14 + field-normalised NCS, FWCI (OpenAlex definition, cite the documentation), Gini, HHI, fractional counting, international
15 + collaboration share, Rao–Stirling and Leinster–Cobbold diversity (define with formulas), citation-age bias and why fixed windows.
16 +5. Networks: citation graph internal/external, category flow matrices F_t, modularity Q (Newman–Girvan formula as implemented
17 + in knowledge_graph/02), cross-area share; co-authorship/country networks.
18 +6. Semantic mapping: SPECTER2/SciNCL/MiniLM embeddings, cosine geometry, UMAP + HDBSCAN (state assumptions), stability by
19 + split-half ARI, category centroid drift, novelty definition.
20 +7. Structural change: ITS model with seasonality and HAC errors (write the regression), placebo distribution and its p-value,
21 + Bai–Perron DP with BIC/LWZ (state the SSR criterion and minimum segment h), PELT penalty, BOCPD with Normal–Inverse-Gamma
22 + conjugacy (give the predictive and the run-length recursion), alternative event dates, right-truncation handling.
23 +8. Emerging-topic detection: features, label definition at horizon h, train/test cutoffs, metrics, look-ahead safeguards,
24 + the preregistered known-emergence set.
25 +9. Knowledge-graph extraction: schema, provenance span, machine verification, confidence; what is and is not evidence.
26 +10. Uncertainty and reproducibility: bootstrap/HAC intervals, seeds, ledgers, hashes.
27 +Cite method sources (Waltman 2016, Bai–Perron, Killick, Adams–MacKay, McInnes UMAP/HDBSCAN, Cohan SPECTER, Stirling, Newman…) — verify each.
added embeddings/collect_multinode.sh +4 −0
@@ -0,0 +1,4 @@
1 +#!/bin/sh
2 +cd ~/wp44_uqo; MODEL=${1:-allenai/specter2_base}; TAG=$(echo $MODEL | sed 's#/#__#g')
3 +for node in M3U96a M4M64a M4M64c; do rsync -aq $node:~/wp44_uqo/data/processed/embeddings/$TAG/ data/processed/embeddings/$TAG/ && echo "$node collected"; done
4 +ls data/processed/embeddings/$TAG/shard_*.npy | wc -l
added embeddings/run_multinode.sh +17 −0
@@ -0,0 +1,17 @@
1 +#!/bin/sh
2 +# Distribute embedding shards over the nodes (run from M3U96b after `embed_corpus.py --prepare`).
3 +# Each node gets a contiguous shard range proportional to its measured throughput; results are rsynced back to M3U96b.
4 +# Usage: sh embeddings/run_multinode.sh allenai/specter2_base
5 +cd ~/wp44_uqo; MODEL=${1:-allenai/specter2_base}; TAG=$(echo $MODEL | sed 's#/#__#g')
6 +NS=$(ls -d data/processed/embeddings/input/shard=* | wc -l | tr -d ' ')
7 +# weights: M3U96b 3, M3U96a 3, M4M64a 2, M4M64c 2 (M3 Ultra ≈ 300 texts/s, M4 Max ≈ 200 texts/s measured/estimated)
8 +A=$((NS*3/10)); B=$((NS*3/10)); C=$((NS*2/10)); D=$((NS - A - B - C))
9 +echo "shards: $NS → M3U96b 0-$A, M3U96a $A-$((A+B)), M4M64a $((A+B))-$((A+B+C)), M4M64c $((A+B+C))-$NS"
10 +for spec in "M3U96a:$A:$((A+B))" "M4M64a:$((A+B)):$((A+B+C))" "M4M64c:$((A+B+C)):$NS"; do
11 + node=${spec%%:*}; rest=${spec#*:}; from=${rest%%:*}; to=${rest#*:}
12 + ssh $node "mkdir -p ~/wp44_uqo/data/processed/embeddings/input"
13 + k=$from; while [ $k -lt $to ]; do rsync -aq data/processed/embeddings/input/shard=$k $node:~/wp44_uqo/data/processed/embeddings/input/ ; k=$((k+1)); done
14 + ssh $node "cd ~/wp44_uqo; . .venv/bin/activate; nohup python embeddings/embed_corpus.py --root ~/wp44_uqo --run --model $MODEL --shard-from $from --shard-to $to > logs/embed_$TAG.log 2>&1 &" && echo "$node started $from-$to"
15 +done
16 +. .venv/bin/activate; nohup python embeddings/embed_corpus.py --root ~/wp44_uqo --run --model $MODEL --shard-from 0 --shard-to $A > logs/embed_$TAG.log 2>&1 &
17 +echo "M3U96b started 0-$A; collect with: sh embeddings/collect_multinode.sh $MODEL"
added embeddings/semantic_atlas.py +93 −0
@@ -0,0 +1,93 @@
1 +#!/usr/bin/env python3
2 +"""WP44 — semantic research mapping (RQ2/RQ5/RQ10 inputs): UMAP projection, HDBSCAN clusters, stability, temporal drift.
3 +Inputs: data/processed/embeddings/<model>/shard_*.npy (+ ids), corpus_works, classification (primary categories).
4 +Steps: (1) stratified sample of --n works (seed 44) for the 2-D atlas; (2) HDBSCAN on a 50-D UMAP of the sample, cluster labels
5 +and top TF-IDF terms per cluster; (3) stability: ARI between HDBSCAN partitions of two bootstrap halves projected into the same
6 +UMAP (repeated 5×); (4) category centroids per year in the native 768-D space, cosine drift year-over-year, inter-category
7 +cosine similarity matrix per period (baseline vs post) → tests H4/H5 at the semantic level; (5) per-work novelty = 1 − max cosine
8 +similarity to the previous 24 months' centroid of the same category (feeds experiments/emerging_detection.py).
9 +Outputs: results/semantic/{atlas_sample.parquet, clusters.csv, cluster_terms.csv, stability.json, category_drift.csv,
10 +category_similarity_<period>.csv, novelty.parquet}."""
11 +import argparse, glob, json, os, numpy as np, pandas as pd, duckdb
12 +
13 +
14 +def load_embeddings(root, tag):
15 + E, I = [], []
16 + for f in sorted(glob.glob(f"{root}/data/processed/embeddings/{tag}/shard_*.npy")):
17 + if f.endswith(".ids.npy"):
18 + continue
19 + E.append(np.load(f)); I.append(np.load(f.replace(".npy", ".ids.npy")))
20 + return np.concatenate(E).astype(np.float32), np.concatenate(I)
21 +
22 +
23 +def main():
24 + ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); ap.add_argument("--model", default="allenai/specter2_base"); ap.add_argument("--n", type=int, default=200000)
25 + ap.add_argument("--seed", type=int, default=44); a = ap.parse_args(); R = a.root; tag = a.model.replace("/", "__"); out = f"{R}/results/semantic"; os.makedirs(out, exist_ok=True)
26 + E, ids = load_embeddings(R, tag); print("embeddings", E.shape)
27 + con = duckdb.connect(f"{R}/data/processed/wp44.duckdb", read_only=True)
28 + con.execute(f"CREATE OR REPLACE VIEW work_primary AS SELECT * FROM read_parquet('{R}/data/processed/classification/primary/*.parquet')")
29 + meta = con.execute("SELECT c.id AS work_id, c.publication_year AS year, c.period, c.first_month, p.primary_category FROM corpus_works c LEFT JOIN work_primary p ON p.work_id = c.id WHERE c.eligible").df().set_index("work_id")
30 + pos = {w: i for i, w in enumerate(ids)}; meta = meta[meta.index.isin(pos)]; meta["row"] = [pos[w] for w in meta.index]
31 + rng = np.random.default_rng(a.seed)
32 + # (4) category centroids per year + drift + similarity
33 + rows = []; cents = {}
34 + for (cat, y), g in meta.dropna(subset=["primary_category"]).groupby(["primary_category", "year"]):
35 + if len(g) < 100:
36 + continue
37 + v = E[g.row.values].mean(0); v /= np.linalg.norm(v) + 1e-9; cents[(cat, y)] = v
38 + for (cat, y), v in cents.items():
39 + prev = cents.get((cat, y - 1)); rows.append({"category": cat, "year": y, "n": int(((meta.primary_category == cat) & (meta.year == y)).sum()), "drift_cos_dist": None if prev is None else float(1 - v @ prev)})
40 + pd.DataFrame(rows).to_csv(f"{out}/category_drift.csv", index=False)
41 + for per, yrs in [("baseline", range(2018, 2023)), ("post", range(2023, 2027))]:
42 + cats = sorted({c for (c, y) in cents if y in yrs}); M = np.zeros((len(cats), len(cats)))
43 + for i, c1 in enumerate(cats):
44 + v1 = np.mean([cents[(c1, y)] for y in yrs if (c1, y) in cents], 0)
45 + for j, c2 in enumerate(cats):
46 + v2 = np.mean([cents[(c2, y)] for y in yrs if (c2, y) in cents], 0); M[i, j] = float(v1 @ v2 / (np.linalg.norm(v1) * np.linalg.norm(v2) + 1e-9))
47 + pd.DataFrame(M, index=cats, columns=cats).to_csv(f"{out}/category_similarity_{per}.csv")
48 + # (5) novelty vs trailing 24-month centroid of the same category
49 + meta_m = meta.dropna(subset=["primary_category"]).copy(); months = sorted(meta_m.first_month.unique()); nov = np.full(len(meta_m), np.nan, dtype=np.float32)
50 + mcent = {}
51 + for (cat, m), g in meta_m.groupby(["primary_category", "first_month"]):
52 + mcent[(cat, m)] = (E[g.row.values].sum(0), len(g))
53 + midx = {m: i for i, m in enumerate(months)}; arr_pos = {w: i for i, w in enumerate(meta_m.index)}
54 + for (cat, m), g in meta_m.groupby(["primary_category", "first_month"]):
55 + i0 = midx[m]; prev = [mcent[(cat, mm)] for mm in months[max(0, i0 - 24):i0] if (cat, mm) in mcent]
56 + if not prev:
57 + continue
58 + v = np.sum([p[0] for p in prev], 0) / max(sum(p[1] for p in prev), 1); v /= np.linalg.norm(v) + 1e-9
59 + sims = E[g.row.values] @ v
60 + for w, s in zip(g.index, sims):
61 + nov[arr_pos[w]] = 1 - s
62 + meta_m["novelty"] = nov; meta_m[["novelty", "primary_category", "first_month"]].reset_index().to_parquet(f"{out}/novelty.parquet")
63 + # (1–3) atlas on a sample
64 + import umap, hdbscan
65 + from sklearn.metrics import adjusted_rand_score
66 + samp = meta.sample(min(a.n, len(meta)), random_state=a.seed); X = E[samp.row.values]
67 + red = umap.UMAP(n_components=50, n_neighbors=30, min_dist=0.0, metric="cosine", random_state=a.seed).fit_transform(X)
68 + lab = hdbscan.HDBSCAN(min_cluster_size=200, min_samples=20).fit_predict(red)
69 + xy = umap.UMAP(n_components=2, n_neighbors=30, min_dist=0.1, metric="cosine", random_state=a.seed).fit_transform(X)
70 + samp = samp.assign(cluster=lab, x=xy[:, 0], y=xy[:, 1]); samp.reset_index().to_parquet(f"{out}/atlas_sample.parquet")
71 + titles = con.execute(f"SELECT id AS work_id, title FROM corpus_works WHERE id IN (SELECT work_id FROM read_parquet('{out}/atlas_sample.parquet'))").df().set_index("work_id").title
72 + from sklearn.feature_extraction.text import TfidfVectorizer
73 + docs = samp.index.map(lambda w: titles.get(w, "") or ""); vec = TfidfVectorizer(max_features=50000, stop_words="english", ngram_range=(1, 2), min_df=5); T = vec.fit_transform(docs); terms = np.array(vec.get_feature_names_out())
74 + crow = []
75 + for c in sorted(set(lab)):
76 + m = lab == c; v = np.asarray(T[m].mean(0)).ravel(); top = terms[np.argsort(-v)[:12]]
77 + pc = samp[m].primary_category.value_counts(normalize=True)
78 + crow.append({"cluster": int(c), "n": int(m.sum()), "top_terms": "; ".join(top), "top_category": pc.index[0] if len(pc) else None, "top_category_share": float(pc.iloc[0]) if len(pc) else None,
79 + "share_post": float((samp[m].period == "post").mean())})
80 + pd.DataFrame(crow).to_csv(f"{out}/clusters.csv", index=False)
81 + aris = []
82 + for r in range(5):
83 + idx = rng.permutation(len(X)); h1, h2 = idx[: len(X) // 2], idx[len(X) // 2:]
84 + l1 = hdbscan.HDBSCAN(min_cluster_size=100, min_samples=20).fit_predict(red[h1]); l2 = hdbscan.HDBSCAN(min_cluster_size=100, min_samples=20).fit_predict(red[h2])
85 + # compare through nearest-neighbour label transfer from half 1 to half 2
86 + from sklearn.neighbors import KNeighborsClassifier
87 + knn = KNeighborsClassifier(n_neighbors=5).fit(red[h1], l1); aris.append(float(adjusted_rand_score(l2, knn.predict(red[h2]))))
88 + json.dump({"n_sample": int(len(X)), "n_clusters": int(len(set(lab)) - (1 if -1 in lab else 0)), "noise_share": float((lab == -1).mean()), "ari_bootstrap": aris, "ari_mean": float(np.mean(aris))}, open(f"{out}/stability.json", "w"), indent=1)
89 + print("atlas done", len(set(lab)), "clusters; ARI", np.mean(aris))
90 +
91 +
92 +if __name__ == "__main__":
93 + main()
modified figures/FIGURES.json +35 −0
@@ -34,5 +34,40 @@
34 34 "results/bibliometrics/team_size_yearly.csv",
35 35 "results/bibliometrics/industry_share_yearly.csv"
36 36 ]
37 + },
38 + {
39 + "file": "fig_citation_inequality",
40 + "caption": "Fixed-window citation levels and inequality by publication year (years with complete 36-month windows).",
41 + "sources": [
42 + "results/bibliometrics/citation_windows_yearly.csv"
43 + ]
44 + },
45 + {
46 + "file": "fig_field_inflows",
47 + "caption": "Share of references pointing outside the corpus, by cited OpenAlex field and citing year.",
48 + "sources": [
49 + "results/graph/field_flows_yearly.csv"
50 + ]
51 + },
52 + {
53 + "file": "fig_cited_age",
54 + "caption": "Distribution of the age of cited works for selected citing years.",
55 + "sources": [
56 + "results/graph/cited_age_yearly.csv"
57 + ]
58 + },
59 + {
60 + "file": "fig_venue_concentration",
61 + "caption": "Number of distinct venues and venue concentration (HHI, top-source share) by year.",
62 + "sources": [
63 + "results/bibliometrics/venue_hhi_yearly.csv"
64 + ]
65 + },
66 + {
67 + "file": "fig_country_shares",
68 + "caption": "Country shares of output with fractional counting, top eight countries.",
69 + "sources": [
70 + "results/bibliometrics/countries_yearly.csv"
71 + ]
37 72 }
38 73 ]
\ No newline at end of file
added figures/fig_citation_inequality.pdf +0 −0

Binary file not shown.

added figures/fig_citation_inequality.png +0 −0

Binary file not shown.

added figures/fig_cited_age.pdf +0 −0

Binary file not shown.

added figures/fig_cited_age.png +0 −0

Binary file not shown.

modified figures/fig_collaboration.pdf +0 −0

Binary file not shown.

added figures/fig_country_shares.pdf +0 −0

Binary file not shown.

added figures/fig_country_shares.png +0 −0

Binary file not shown.

added figures/fig_field_inflows.pdf +0 −0

Binary file not shown.

added figures/fig_field_inflows.png +0 −0

Binary file not shown.

modified figures/fig_prisma_flow.pdf +0 −0

Binary file not shown.

modified figures/fig_types_openness.pdf +0 −0

Binary file not shown.

added figures/fig_venue_concentration.pdf +0 −0

Binary file not shown.

added figures/fig_venue_concentration.png +0 −0

Binary file not shown.

modified figures/fig_volume_monthly_raw.pdf +0 −0

Binary file not shown.

modified figures/fig_volume_yearly.pdf +0 −0

Binary file not shown.

modified figures/fig_volume_yearly.png +0 −0

Binary file not shown.

modified figures/make_figures.py +74 −1
@@ -89,7 +89,7 @@ def fig_collab(R, out):
89 89
90 90 def main():
91 91 ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); out = f"{a.root}/figures"; os.makedirs(out, exist_ok=True)
92 − for f in (fig_prisma, fig_yearly_volume, fig_monthly, fig_types_sources, fig_collab):
92 + for f in (fig_prisma, fig_yearly_volume, fig_monthly, fig_types_sources, fig_collab) + _BATCH2:
93 93 try:
94 94 f(a.root, out); print("ok", f.__name__)
95 95 except Exception as e:
@@ -97,5 +97,78 @@ def main():
97 97 json.dump(INV, open(f"{out}/FIGURES.json", "w"), indent=1)
98 98
99 99
100 +
101 +
102 +# ---------------- batch 2: citations and knowledge flows (generated from results/bibliometrics + results/graph) ----------------
103 +def fig_citation_inequality(R, out):
104 + p = f"{R}/results/bibliometrics/citation_windows_yearly.csv"
105 + if not os.path.exists(p):
106 + return
107 + d = pd.read_csv(p); d = d[d.year <= 2023] # c36 requires three full years of citation data
108 + fig, axes = plt.subplots(1, 2, figsize=(7.2, 3.0))
109 + ax = axes[0]; ax.plot(d.year, d.mean_c36, marker="o", color=style.CATEGORICAL[0], label="Mean c36"); ax.plot(d.year, d.p99_c36 / 10, marker="o", color=style.CATEGORICAL[1], label="p99 c36 ÷ 10")
110 + ax.set_title("Fixed-window citations (36 months)", loc="left"); ax.set_ylabel("Citations"); ax.legend()
111 + ax = axes[1]; ax.plot(d.year, d.gini_c36, marker="o", color=style.CATEGORICAL[2], label="Gini of c36"); ax.plot(d.year, d.share_uncited_36, marker="o", color=style.CATEGORICAL[3], label="Share uncited at 36 months")
112 + ax.set_title("Citation inequality", loc="left"); ax.set_ylim(0, 1); ax.legend()
113 + style.save(fig, f"{out}/fig_citation_inequality.pdf"); reg("fig_citation_inequality", "Fixed-window citation levels and inequality by publication year (years with complete 36-month windows).", ["results/bibliometrics/citation_windows_yearly.csv"])
114 +
115 +
116 +def fig_field_inflows(R, out):
117 + p = f"{R}/results/graph/field_flows_yearly.csv"
118 + if not os.path.exists(p):
119 + return
120 + d = pd.read_csv(p); d = d[(d.citing_year >= 2018) & (d.citing_year <= 2026)]
121 + names = {17: "Computer science (outside corpus)", 26: "Mathematics", 22: "Engineering", 31: "Physics & astronomy", 27: "Medicine", 33: "Social sciences", 13: "Biochem./genetics", 18: "Decision sciences", 28: "Neuroscience", 20: "Economics & finance"}
122 + tot = d.groupby("citing_year").n.sum(); ext = d[d.kind == "external"]
123 + top = ext.groupby("cited_field_id").n.sum().nlargest(8).index
124 + fig, ax = plt.subplots(figsize=(6.8, 3.4))
125 + for i, f in enumerate(top):
126 + s = ext[ext.cited_field_id == f].groupby("citing_year").n.sum() / tot * 100
127 + ax.plot(s.index, s.values, marker="o", color=style.CATEGORICAL[i % 8], label=names.get(int(f), f"field {int(f)}"), lw=1.2)
128 + ax.annotate(names.get(int(f), str(int(f))), (s.index[-1], s.values[-1]), xytext=(3, 0), textcoords="offset points", fontsize=6.5, color=style.INK2, va="center")
129 + ax.set_ylabel("Share of all references (%)"); ax.set_title("References from the CS corpus to works outside it, by cited field", loc="left"); ax.legend(ncol=2, fontsize=6.5)
130 + style.save(fig, f"{out}/fig_field_inflows.pdf"); reg("fig_field_inflows", "Share of references pointing outside the corpus, by cited OpenAlex field and citing year.", ["results/graph/field_flows_yearly.csv"])
131 +
132 +
133 +def fig_cited_age(R, out):
134 + p = f"{R}/results/graph/cited_age_yearly.csv"
135 + if not os.path.exists(p):
136 + return
137 + d = pd.read_csv(p); d = d[(d.age >= 0) & (d.age <= 30)]
138 + fig, ax = plt.subplots(figsize=(6.8, 3.2))
139 + for i, y in enumerate([2018, 2020, 2022, 2024, 2026]):
140 + g = d[d.citing_year == y]; s = g.n / g.n.sum()
141 + ax.plot(g.age, s * 100, color=style.SEQUENTIAL[2 + 2 * i], label=str(y), lw=1.3)
142 + ax.set_xlabel("Age of cited work (years)"); ax.set_ylabel("Share of references (%)"); ax.set_title("Reference age distribution by citing year", loc="left"); ax.legend(title="Citing year")
143 + style.save(fig, f"{out}/fig_cited_age.pdf"); reg("fig_cited_age", "Distribution of the age of cited works for selected citing years.", ["results/graph/cited_age_yearly.csv"])
144 +
145 +
146 +def fig_venue_hhi(R, out):
147 + p = f"{R}/results/bibliometrics/venue_hhi_yearly.csv"
148 + if not os.path.exists(p):
149 + return
150 + d = pd.read_csv(p); d = d[d.year <= 2026]
151 + fig, axes = plt.subplots(1, 2, figsize=(7.2, 3.0))
152 + axes[0].plot(d.year, d.n_sources, marker="o", color=style.CATEGORICAL[0]); axes[0].set_title("Distinct sources (venues) per year", loc="left")
153 + axes[1].plot(d.year, d.hhi * 1e4, marker="o", color=style.CATEGORICAL[1], label="HHI × 10⁴"); axes[1].plot(d.year, d.top1_share * 100, marker="o", color=style.CATEGORICAL[2], label="Top source share (%)"); axes[1].set_title("Venue concentration", loc="left"); axes[1].legend()
154 + style.save(fig, f"{out}/fig_venue_concentration.pdf"); reg("fig_venue_concentration", "Number of distinct venues and venue concentration (HHI, top-source share) by year.", ["results/bibliometrics/venue_hhi_yearly.csv"])
155 +
156 +
157 +def fig_countries(R, out):
158 + p = f"{R}/results/bibliometrics/countries_yearly.csv"
159 + if not os.path.exists(p):
160 + return
161 + d = pd.read_csv(p); d = d[d.year <= 2026]; tot = d.groupby("year").n_frac.sum(); top = d.groupby("country").n_frac.sum().nlargest(8).index
162 + fig, ax = plt.subplots(figsize=(6.8, 3.4))
163 + for i, c in enumerate(top):
164 + s = d[d.country == c].set_index("year").n_frac / tot * 100
165 + ax.plot(s.index, s.values, marker="o", color=style.CATEGORICAL[i], label=c, lw=1.2); ax.annotate(c, (s.index[-1], s.values[-1]), xytext=(3, 0), textcoords="offset points", fontsize=7, color=style.INK2, va="center")
166 + ax.set_ylabel("Share of fractional output (%)"); ax.set_title("Country shares of eligible CS output (fractional counting, top 8)", loc="left"); ax.legend(ncol=4, fontsize=7)
167 + style.save(fig, f"{out}/fig_country_shares.pdf"); reg("fig_country_shares", "Country shares of output with fractional counting, top eight countries.", ["results/bibliometrics/countries_yearly.csv"])
168 +
169 +
170 +_BATCH2 = (fig_citation_inequality, fig_field_inflows, fig_cited_age, fig_venue_hhi, fig_countries)
171 +
172 +
100 173 if __name__ == "__main__":
101 174 main()
modified taxonomy/classify.py +1 −1
@@ -103,7 +103,7 @@ def prepare(root, db, out_dir, chunk_rows):
103 103 (SELECT list(subfield_id ORDER BY rank) FROM oa_work_topics t WHERE t.work_id = w.id) AS subfield_ids,
104 104 (SELECT flatten(list(x.categories)) FROM link_arxiv_oa l JOIN arxiv_keys x USING(arxiv_id) WHERE l.oa_id = w.id) AS arxiv_cats,
105 105 (SELECT list(DISTINCT d.stream) FROM link_dblp_oa l JOIN dblp_keys d USING(dblp_key) WHERE l.oa_id = w.id) AS dblp_streams
106 − FROM corpus_works w LEFT JOIN oa_abstracts a ON a.work_id = w.id""")
106 + FROM corpus_works w LEFT JOIN oa_abstracts a ON a.work_id = w.id WHERE w.eligible OR w.cs_penumbra""")
107 107 n = con.execute("SELECT count(*) FROM cls_input").fetchone()[0]
108 108 con.execute(f"""COPY (SELECT *, (row_number() OVER ()) // {chunk_rows} AS chunk FROM cls_input) TO '{out_dir}/input' (FORMAT PARQUET, PARTITION_BY (chunk), OVERWRITE_OR_IGNORE, COMPRESSION ZSTD)""")
109 109 print("prepared", n, "works")
modified taxonomy/run_classify_shards.sh +11 −4
@@ -1,14 +1,21 @@
1 1 #!/bin/sh
2 −# Classify the corpus with N independent processes (macOS multiprocessing pool hangs with pyarrow in spawned workers).
3 −# Usage: sh taxonomy/run_classify_shards.sh [N] — waits for the pass-2 chain (CHAIN_DONE) before preparing the input.
4 −cd ~/wp44_uqo; . .venv/bin/activate; ulimit -n 65536; N=${1:-20}
2 +# Classify the corpus with independent processes on two nodes (M3U96b local + M3U96a remote); macOS pools hang with pyarrow.
3 +# Usage: sh taxonomy/run_classify_shards.sh [N_total] — waits for the pass-2 chain (CHAIN_DONE) before preparing the input.
4 +cd ~/wp44_uqo; . .venv/bin/activate; ulimit -n 65536; N=${1:-56}; HALF=$((N/2)); R=M3U96a
5 5 while ! grep -q CHAIN_DONE logs/define_corpus_v2.log 2>/dev/null; do sleep 60; done
6 6 rm -rf data/processed/classification; mkdir -p logs/classify
7 7 python taxonomy/classify.py --root ~/wp44_uqo --ontology taxonomy/ontology.yaml --prepare > logs/classify/prepare.log 2>&1
8 −i=0; while [ $i -lt $N ]; do
8 +ssh $R "mkdir -p ~/wp44_uqo/data/processed/classification ~/wp44_uqo/taxonomy ~/wp44_uqo/logs/classify; rm -rf ~/wp44_uqo/data/processed/classification/*"
9 +rsync -aq data/processed/classification/input $R:~/wp44_uqo/data/processed/classification/
10 +rsync -aq taxonomy/classify.py taxonomy/ontology.yaml taxonomy/__init__.py $R:~/wp44_uqo/taxonomy/
11 +i=0; while [ $i -lt $HALF ]; do
9 12 nohup python taxonomy/classify.py --root ~/wp44_uqo --ontology taxonomy/ontology.yaml --run --shard $i/$N > logs/classify/shard_$i.log 2>&1 &
10 13 i=$((i+1)); done
14 +ssh $R "cd ~/wp44_uqo; . .venv/bin/activate; i=$HALF; while [ \$i -lt $N ]; do nohup python taxonomy/classify.py --root ~/wp44_uqo --ontology taxonomy/ontology.yaml --run --shard \$i/$N > logs/classify/shard_\$i.log 2>&1 & i=\$((i+1)); done; echo remote started"
11 15 wait
16 +while ssh $R "pgrep -f 'classify.py --root' >/dev/null"; do sleep 60; done
17 +rsync -aq $R:~/wp44_uqo/data/processed/classification/cats/ data/processed/classification/cats/
18 +rsync -aq $R:~/wp44_uqo/data/processed/classification/primary/ data/processed/classification/primary/
12 19 python - <<EOF2
13 20 import duckdb, json, datetime as dt, yaml
14 21 con = duckdb.connect(); o = yaml.safe_load(open("taxonomy/ontology.yaml"))
15 22
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere