SPB Git
spboucher.ai

spb/wp44_uqo

Public

WP44: figure batch 2, embeddings multi-node drivers, semantic atlas, writer briefs

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 3, 2026) parent a5b49ef Browse files

28 changed files +333 −6

28 changed files
Unified Split
added agents/briefs/AGENT_K_CRITIC.md +12 −0
@@ -0,0 +1,12 @@
1+# Brief — Agent K (Scientific Critic)
2+
3+You receive a structured artefact (a specialist review directory, an econometric result set, or a manuscript part) and
4+return `reports/critic/<artefact>_critique.md` + `<artefact>_issues.csv` (issue_id, location, severity ∈ {blocking, major,
5+minor}, type ∈ {unsupported_claim, causal_language, missing_uncertainty, metadata_as_fulltext, preprint_as_peer_reviewed,
6+number_without_artifact, citation_unverified, overclaim, inconsistency, method_weakness}, quote, required_fix).
7+Checks (CLAUDE.md §1, §10): every number traces to a file; every claim has evidence_type and supporting keys; no causal
8+language about ChatGPT ("after" not "because"); preprints never called peer-reviewed; abstracts never treated as full text;
9+"largest ever" and similar superlatives absent unless independently established; uncertainty reported; competing
10+explanations considered (indexing lag, OpenAlex classifier drift, venue growth, XPAC, duplicates). You may run the scripts to
11+re-derive a number. Verdict: ACCEPT / ACCEPT WITH FIXES / REJECT, with the blocking issues listed first. You never edit the
12+artefact yourself.
added agents/briefs/WRITER_COMMON.md +26 −0
@@ -0,0 +1,26 @@
1+# Common rules for manuscript writers (Agent L instances), WP44
2+
3+Repository: /Users/simon-pierreboucher/Desktop/UQO/Recherche/Recherche-UQO/informatique/wp44 . Read CLAUDE.md §1, §9, §10 and
4+RESEARCH_PLAN.md first. The manuscript is `manuscript/main.tex` (UQO template, natbib author-year, `\bibliographystyle{aer}`,
5+`\part{}` already declared in main.tex — you write `\section`/`\subsection` content only, in the file assigned to you).
6+
7+Style (CLAUDE.md §10): formal academic English, critical, precise, no promotional language, no repetitive AI prose (vary
8+sentence structure, no "In this section we…" openers in every subsection, no bullet lists in body prose except where a list is
9+the natural structure), real mathematical notation where it carries meaning, assumptions and limitations explicit. Length:
10+the target page count for your part is given in your brief; do not pad.
11+
12+Citations: `\citep{key}` / `\citet{key}`. You may cite only keys that exist in `manuscript/references.bib` (being built by
13+another agent — check it) OR keys you add to YOUR OWN bib file `manuscript/references_<part>.bib`, every entry verified
14+by a Crossref/OpenAlex/arXiv single lookup (`curl -s "https://api.crossref.org/works/<DOI>?mailto=spbou4@icloud.com"`,
15+`https://api.openalex.org/works/https://doi.org/<DOI>`, `https://export.arxiv.org/api/query?id_list=<id>`), recorded in
16+`manuscript/references_verification_<part>.csv` (key, doi_or_arxiv, verified_via, verified_title, match). Never invent a
17+reference. If you cannot verify, do not cite. Keys: `surnameYYYYkeyword`. Main-bib keys you reuse: do not redefine them.
18+
19+Numbers: never type a corpus number. Use the macros of `manuscript/tables/numbers.tex` (generated; list them with
20+`grep newcommand manuscript/tables/numbers.tex`). If a number you need has no macro, write `\pending{macro name}` and list
21+it in your report. Results sections are written by others; do not describe empirical findings of WP44 beyond the macros.
22+
23+Integrity: no causal claim about ChatGPT; preprints are "preprints"; what you know only from an abstract is said so; no
24+"largest review ever". Deliverable = your .tex file(s), your bib + verification CSV, and a report ≤ 20 lines (what you wrote,
25+word count, citations added, pending macros, anything uncertain). Compile check: `cd manuscript && latexmk -pdf -interaction=nonstopmode main.tex`
26+may fail on other parts — check only that YOUR file has no LaTeX syntax errors (e.g. with `lacheck` or by including it alone in a scratch document).
added agents/briefs/WRITER_PART02.md +18 −0
@@ -0,0 +1,18 @@
1+# Brief — Part II "The Evolution of Computer Science" (file manuscript/sections/part02_evolution.tex; bib manuscript/references_part02.bib)
2+
3+Target 25–35 pages (≈ 12,000–16,000 words). Purpose (CLAUDE.md §9): the conceptual and historical background needed to
4+interpret post-2022 developments — not a history of computing in general. Suggested sections (adapt):
5+1. The disciplinary structure of computer science (ACM CCS, the 40-category ontology of this paper, how fields relate; how
6+ bibliometric field classifications — OpenAlex topics, DBLP venues, arXiv categories — encode that structure and where they disagree).
7+2. Three prior transformations and how they propagated across subfields: the internet/web era, the deep-learning wave
8+ (2012–2017: AlexNet, ResNet, seq2seq, attention), the transformer/pretraining era (2017–2022: Transformer, BERT, GPT-2/3,
9+ scaling laws, CLIP, diffusion, AlphaFold). For each: what changed in publication practice (preprints, benchmarks, industry labs).
10+3. The state of each non-AI area at the baseline (2018–2022): systems (cloud, serverless, disaggregation), hardware
11+ (end of Dennard scaling, accelerators, "golden age"), theory (fine-grained complexity, learning theory), PL/verification
12+ (Rust, proof assistants), security/crypto (post-quantum standardisation, ML security), HCI, graphics (NeRF), quantum (NISQ),
13+ bioinformatics (AlphaFold 2) — concise, citation-backed.
14+4. November 2022 as a reference date: what ChatGPT was and was not (an interface + RLHF over GPT-3.5), the preceding
15+ releases (InstructGPT, Codex, Stable Diffusion), why the date is used descriptively and not causally (CLAUDE.md §6.5).
16+5. What the literature already claims about the post-2022 transformation (growth of AI papers, LLM-assisted writing, arXiv
17+ submission surges, benchmark saturation, industry concentration) and why those claims need the corpus-level test of this paper.
18+Landmark references: verify each (group G/H of the reference brief). Coordinate with `manuscript/references.bib` if present.
added agents/briefs/WRITER_PART03.md +15 −0
@@ -0,0 +1,15 @@
1+# Brief — Part III "Data and Systematic Review Protocol" (file manuscript/sections/part03_data.tex; bib manuscript/references_part03.bib)
2+
3+Target 25–35 pages. Explain corpus construction, source coverage, deduplication, eligibility and research integrity
4+(CLAUDE.md §9, PRISMA 2020 items where applicable). Sources of truth to read: RESEARCH_PLAN.md §2–§3, protocol/PROTOCOL.md,
5+protocol/amendments.csv, reports/PHASE0_FEASIBILITY.md, data/raw/README.md, ingestion/*.py (what each does, measured sizes),
6+systematic_review/define_corpus.py (exclusion codes E1–E7, the CS-core rule, date precision), data/metadata/linkage_report.json,
7+data/metadata/prisma_counts.json, figures/FIGURES.json (fig_prisma_flow, fig_volume_yearly, fig_types_openness exist in
8+manuscript/figures/ — reference them with \includegraphics{fig_prisma_flow.pdf} etc.; copy from figures/ to manuscript/figures/ if missing).
9+Sections: 1 sources and access (licences, snapshot release, API budget facts, Semantic Scholar unavailable); 2 identification
10+(what each source contributes, sizes via macros); 3 linkage (methods, counts via macros, the second snapshot pass and why);
11+4 deduplication of preprint/published pairs; 5 eligibility (codes, counts, decisions A-001/A-002 with their evidence);
12+6 the CS-core vs penumbra design and the strict core; 7 date precision, indexing lag, right truncation (created_date is
13+unusable for lag — explain); 8 quality indicators and known biases (DBLP coverage, OpenAlex topic noise, XPAC, Zenodo flood,
14+missing abstracts, forward-dated records); 9 research-integrity procedures (preregistration, amendments log, human checkpoints,
15+agent register, no fabrication rules). Every number through macros (`\PrismaCsCoreEligible`, `\LinkDblpLinkedDistinct`, …).
added agents/briefs/WRITER_PART04.md +27 −0
@@ -0,0 +1,27 @@
1+# Brief — Part IV "Computational Methodology" (file manuscript/sections/part04_methodology.tex; bib manuscript/references_part04.bib)
2+
3+Target 35–50 pages (≈ 15,000–22,000 words) with real mathematics. Document EXACTLY what the code does — read these files
4+before writing: RESEARCH_PLAN.md §3–§7, protocol/PROTOCOL.md, ingestion/link_sources.py, systematic_review/define_corpus.py,
5+taxonomy/classify.py + taxonomy/ontology.yaml (assignment_rule) + taxonomy/VALIDATION.md, bibliometrics/01–03, knowledge_graph/01–03,
6+econometrics/structural_change.py + run_structural_change.py, embeddings/embed_corpus.py + semantic_atlas.py,
7+experiments/emerging_detection.py. Where the code and the plan differ, describe the code and flag the difference in your report.
8+Sections:
9+1. Notation and the corpus as a measurable object (works w, dates, categories C(w) ⊆ 𝒦, periods B/P, month-precision filter).
10+2. Record linkage and deduplication (deterministic DOI/arXiv keys, normalised-title blocking, precision estimation design).
11+3. Multi-label classification: score s_k(w) = Σ_signal min(1, Σ weights), threshold, primary rule, validation metrics
12+ (precision/recall/F1, Cohen's κ, Jaccard), and the known failure modes.
13+4. Bibliometric indicators with definitions and estimators: counts, shares, CAGR, fixed-window citations c12/c36,
14+ field-normalised NCS, FWCI (OpenAlex definition, cite the documentation), Gini, HHI, fractional counting, international
15+ collaboration share, Rao–Stirling and Leinster–Cobbold diversity (define with formulas), citation-age bias and why fixed windows.
16+5. Networks: citation graph internal/external, category flow matrices F_t, modularity Q (Newman–Girvan formula as implemented
17+ in knowledge_graph/02), cross-area share; co-authorship/country networks.
18+6. Semantic mapping: SPECTER2/SciNCL/MiniLM embeddings, cosine geometry, UMAP + HDBSCAN (state assumptions), stability by
19+ split-half ARI, category centroid drift, novelty definition.
20+7. Structural change: ITS model with seasonality and HAC errors (write the regression), placebo distribution and its p-value,
21+ Bai–Perron DP with BIC/LWZ (state the SSR criterion and minimum segment h), PELT penalty, BOCPD with Normal–Inverse-Gamma
22+ conjugacy (give the predictive and the run-length recursion), alternative event dates, right-truncation handling.
23+8. Emerging-topic detection: features, label definition at horizon h, train/test cutoffs, metrics, look-ahead safeguards,
24+ the preregistered known-emergence set.
25+9. Knowledge-graph extraction: schema, provenance span, machine verification, confidence; what is and is not evidence.
26+10. Uncertainty and reproducibility: bootstrap/HAC intervals, seeds, ledgers, hashes.
27+Cite method sources (Waltman 2016, Bai–Perron, Killick, Adams–MacKay, McInnes UMAP/HDBSCAN, Cohan SPECTER, Stirling, Newman…) — verify each.
added embeddings/collect_multinode.sh +4 −0
@@ -0,0 +1,4 @@
1+#!/bin/sh
2+cd ~/wp44_uqo; MODEL=${1:-allenai/specter2_base}; TAG=$(echo $MODEL | sed 's#/#__#g')
3+for node in M3U96a M4M64a M4M64c; do rsync -aq $node:~/wp44_uqo/data/processed/embeddings/$TAG/ data/processed/embeddings/$TAG/ && echo "$node collected"; done
4+ls data/processed/embeddings/$TAG/shard_*.npy | wc -l
added embeddings/run_multinode.sh +17 −0
@@ -0,0 +1,17 @@
1+#!/bin/sh
2+# Distribute embedding shards over the nodes (run from M3U96b after `embed_corpus.py --prepare`).
3+# Each node gets a contiguous shard range proportional to its measured throughput; results are rsynced back to M3U96b.
4+# Usage: sh embeddings/run_multinode.sh allenai/specter2_base
5+cd ~/wp44_uqo; MODEL=${1:-allenai/specter2_base}; TAG=$(echo $MODEL | sed 's#/#__#g')
6+NS=$(ls -d data/processed/embeddings/input/shard=* | wc -l | tr -d ' ')
7+# weights: M3U96b 3, M3U96a 3, M4M64a 2, M4M64c 2 (M3 Ultra ≈ 300 texts/s, M4 Max ≈ 200 texts/s measured/estimated)
8+A=$((NS*3/10)); B=$((NS*3/10)); C=$((NS*2/10)); D=$((NS - A - B - C))
9+echo "shards: $NS → M3U96b 0-$A, M3U96a $A-$((A+B)), M4M64a $((A+B))-$((A+B+C)), M4M64c $((A+B+C))-$NS"
10+for spec in "M3U96a:$A:$((A+B))" "M4M64a:$((A+B)):$((A+B+C))" "M4M64c:$((A+B+C)):$NS"; do
11+ node=${spec%%:*}; rest=${spec#*:}; from=${rest%%:*}; to=${rest#*:}
12+ ssh $node "mkdir -p ~/wp44_uqo/data/processed/embeddings/input"
13+ k=$from; while [ $k -lt $to ]; do rsync -aq data/processed/embeddings/input/shard=$k $node:~/wp44_uqo/data/processed/embeddings/input/ ; k=$((k+1)); done
14+ ssh $node "cd ~/wp44_uqo; . .venv/bin/activate; nohup python embeddings/embed_corpus.py --root ~/wp44_uqo --run --model $MODEL --shard-from $from --shard-to $to > logs/embed_$TAG.log 2>&1 &" && echo "$node started $from-$to"
15+done
16+. .venv/bin/activate; nohup python embeddings/embed_corpus.py --root ~/wp44_uqo --run --model $MODEL --shard-from 0 --shard-to $A > logs/embed_$TAG.log 2>&1 &
17+echo "M3U96b started 0-$A; collect with: sh embeddings/collect_multinode.sh $MODEL"
added embeddings/semantic_atlas.py +93 −0
@@ -0,0 +1,93 @@
1+#!/usr/bin/env python3
2+"""WP44 — semantic research mapping (RQ2/RQ5/RQ10 inputs): UMAP projection, HDBSCAN clusters, stability, temporal drift.
3+Inputs: data/processed/embeddings/<model>/shard_*.npy (+ ids), corpus_works, classification (primary categories).
4+Steps: (1) stratified sample of --n works (seed 44) for the 2-D atlas; (2) HDBSCAN on a 50-D UMAP of the sample, cluster labels
5+and top TF-IDF terms per cluster; (3) stability: ARI between HDBSCAN partitions of two bootstrap halves projected into the same
6+UMAP (repeated 5×); (4) category centroids per year in the native 768-D space, cosine drift year-over-year, inter-category
7+cosine similarity matrix per period (baseline vs post) → tests H4/H5 at the semantic level; (5) per-work novelty = 1 − max cosine
8+similarity to the previous 24 months' centroid of the same category (feeds experiments/emerging_detection.py).
9+Outputs: results/semantic/{atlas_sample.parquet, clusters.csv, cluster_terms.csv, stability.json, category_drift.csv,
10+category_similarity_<period>.csv, novelty.parquet}."""
11+import argparse, glob, json, os, numpy as np, pandas as pd, duckdb
12+
13+
14+def load_embeddings(root, tag):
15+ E, I = [], []
16+ for f in sorted(glob.glob(f"{root}/data/processed/embeddings/{tag}/shard_*.npy")):
17+ if f.endswith(".ids.npy"):
18+ continue
19+ E.append(np.load(f)); I.append(np.load(f.replace(".npy", ".ids.npy")))
20+ return np.concatenate(E).astype(np.float32), np.concatenate(I)
21+
22+
23+def main():
24+ ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); ap.add_argument("--model", default="allenai/specter2_base"); ap.add_argument("--n", type=int, default=200000)
25+ ap.add_argument("--seed", type=int, default=44); a = ap.parse_args(); R = a.root; tag = a.model.replace("/", "__"); out = f"{R}/results/semantic"; os.makedirs(out, exist_ok=True)
26+ E, ids = load_embeddings(R, tag); print("embeddings", E.shape)
27+ con = duckdb.connect(f"{R}/data/processed/wp44.duckdb", read_only=True)
28+ con.execute(f"CREATE OR REPLACE VIEW work_primary AS SELECT * FROM read_parquet('{R}/data/processed/classification/primary/*.parquet')")
29+ meta = con.execute("SELECT c.id AS work_id, c.publication_year AS year, c.period, c.first_month, p.primary_category FROM corpus_works c LEFT JOIN work_primary p ON p.work_id = c.id WHERE c.eligible").df().set_index("work_id")
30+ pos = {w: i for i, w in enumerate(ids)}; meta = meta[meta.index.isin(pos)]; meta["row"] = [pos[w] for w in meta.index]
31+ rng = np.random.default_rng(a.seed)
32+ # (4) category centroids per year + drift + similarity
33+ rows = []; cents = {}
34+ for (cat, y), g in meta.dropna(subset=["primary_category"]).groupby(["primary_category", "year"]):
35+ if len(g) < 100:
36+ continue
37+ v = E[g.row.values].mean(0); v /= np.linalg.norm(v) + 1e-9; cents[(cat, y)] = v
38+ for (cat, y), v in cents.items():
39+ prev = cents.get((cat, y - 1)); rows.append({"category": cat, "year": y, "n": int(((meta.primary_category == cat) & (meta.year == y)).sum()), "drift_cos_dist": None if prev is None else float(1 - v @ prev)})
40+ pd.DataFrame(rows).to_csv(f"{out}/category_drift.csv", index=False)
41+ for per, yrs in [("baseline", range(2018, 2023)), ("post", range(2023, 2027))]:
42+ cats = sorted({c for (c, y) in cents if y in yrs}); M = np.zeros((len(cats), len(cats)))
43+ for i, c1 in enumerate(cats):
44+ v1 = np.mean([cents[(c1, y)] for y in yrs if (c1, y) in cents], 0)
45+ for j, c2 in enumerate(cats):
46+ v2 = np.mean([cents[(c2, y)] for y in yrs if (c2, y) in cents], 0); M[i, j] = float(v1 @ v2 / (np.linalg.norm(v1) * np.linalg.norm(v2) + 1e-9))
47+ pd.DataFrame(M, index=cats, columns=cats).to_csv(f"{out}/category_similarity_{per}.csv")
48+ # (5) novelty vs trailing 24-month centroid of the same category
49+ meta_m = meta.dropna(subset=["primary_category"]).copy(); months = sorted(meta_m.first_month.unique()); nov = np.full(len(meta_m), np.nan, dtype=np.float32)
50+ mcent = {}
51+ for (cat, m), g in meta_m.groupby(["primary_category", "first_month"]):
52+ mcent[(cat, m)] = (E[g.row.values].sum(0), len(g))
53+ midx = {m: i for i, m in enumerate(months)}; arr_pos = {w: i for i, w in enumerate(meta_m.index)}
54+ for (cat, m), g in meta_m.groupby(["primary_category", "first_month"]):
55+ i0 = midx[m]; prev = [mcent[(cat, mm)] for mm in months[max(0, i0 - 24):i0] if (cat, mm) in mcent]
56+ if not prev:
57+ continue
58+ v = np.sum([p[0] for p in prev], 0) / max(sum(p[1] for p in prev), 1); v /= np.linalg.norm(v) + 1e-9
59+ sims = E[g.row.values] @ v
60+ for w, s in zip(g.index, sims):
61+ nov[arr_pos[w]] = 1 - s
62+ meta_m["novelty"] = nov; meta_m[["novelty", "primary_category", "first_month"]].reset_index().to_parquet(f"{out}/novelty.parquet")
63+ # (1–3) atlas on a sample
64+ import umap, hdbscan
65+ from sklearn.metrics import adjusted_rand_score
66+ samp = meta.sample(min(a.n, len(meta)), random_state=a.seed); X = E[samp.row.values]
67+ red = umap.UMAP(n_components=50, n_neighbors=30, min_dist=0.0, metric="cosine", random_state=a.seed).fit_transform(X)
68+ lab = hdbscan.HDBSCAN(min_cluster_size=200, min_samples=20).fit_predict(red)
69+ xy = umap.UMAP(n_components=2, n_neighbors=30, min_dist=0.1, metric="cosine", random_state=a.seed).fit_transform(X)
70+ samp = samp.assign(cluster=lab, x=xy[:, 0], y=xy[:, 1]); samp.reset_index().to_parquet(f"{out}/atlas_sample.parquet")
71+ titles = con.execute(f"SELECT id AS work_id, title FROM corpus_works WHERE id IN (SELECT work_id FROM read_parquet('{out}/atlas_sample.parquet'))").df().set_index("work_id").title
72+ from sklearn.feature_extraction.text import TfidfVectorizer
73+ docs = samp.index.map(lambda w: titles.get(w, "") or ""); vec = TfidfVectorizer(max_features=50000, stop_words="english", ngram_range=(1, 2), min_df=5); T = vec.fit_transform(docs); terms = np.array(vec.get_feature_names_out())
74+ crow = []
75+ for c in sorted(set(lab)):
76+ m = lab == c; v = np.asarray(T[m].mean(0)).ravel(); top = terms[np.argsort(-v)[:12]]
77+ pc = samp[m].primary_category.value_counts(normalize=True)
78+ crow.append({"cluster": int(c), "n": int(m.sum()), "top_terms": "; ".join(top), "top_category": pc.index[0] if len(pc) else None, "top_category_share": float(pc.iloc[0]) if len(pc) else None,
79+ "share_post": float((samp[m].period == "post").mean())})
80+ pd.DataFrame(crow).to_csv(f"{out}/clusters.csv", index=False)
81+ aris = []
82+ for r in range(5):
83+ idx = rng.permutation(len(X)); h1, h2 = idx[: len(X) // 2], idx[len(X) // 2:]
84+ l1 = hdbscan.HDBSCAN(min_cluster_size=100, min_samples=20).fit_predict(red[h1]); l2 = hdbscan.HDBSCAN(min_cluster_size=100, min_samples=20).fit_predict(red[h2])
85+ # compare through nearest-neighbour label transfer from half 1 to half 2
86+ from sklearn.neighbors import KNeighborsClassifier
87+ knn = KNeighborsClassifier(n_neighbors=5).fit(red[h1], l1); aris.append(float(adjusted_rand_score(l2, knn.predict(red[h2]))))
88+ json.dump({"n_sample": int(len(X)), "n_clusters": int(len(set(lab)) - (1 if -1 in lab else 0)), "noise_share": float((lab == -1).mean()), "ari_bootstrap": aris, "ari_mean": float(np.mean(aris))}, open(f"{out}/stability.json", "w"), indent=1)
89+ print("atlas done", len(set(lab)), "clusters; ARI", np.mean(aris))
90+
91+
92+if __name__ == "__main__":
93+ main()
modified figures/FIGURES.json +35 −0
@@ -34,5 +34,40 @@
34 "results/bibliometrics/team_size_yearly.csv", 34 "results/bibliometrics/team_size_yearly.csv",
35 "results/bibliometrics/industry_share_yearly.csv" 35 "results/bibliometrics/industry_share_yearly.csv"
36 ] 36 ]
37+ },
38+ {
39+ "file": "fig_citation_inequality",
40+ "caption": "Fixed-window citation levels and inequality by publication year (years with complete 36-month windows).",
41+ "sources": [
42+ "results/bibliometrics/citation_windows_yearly.csv"
43+ ]
44+ },
45+ {
46+ "file": "fig_field_inflows",
47+ "caption": "Share of references pointing outside the corpus, by cited OpenAlex field and citing year.",
48+ "sources": [
49+ "results/graph/field_flows_yearly.csv"
50+ ]
51+ },
52+ {
53+ "file": "fig_cited_age",
54+ "caption": "Distribution of the age of cited works for selected citing years.",
55+ "sources": [
56+ "results/graph/cited_age_yearly.csv"
57+ ]
58+ },
59+ {
60+ "file": "fig_venue_concentration",
61+ "caption": "Number of distinct venues and venue concentration (HHI, top-source share) by year.",
62+ "sources": [
63+ "results/bibliometrics/venue_hhi_yearly.csv"
64+ ]
65+ },
66+ {
67+ "file": "fig_country_shares",
68+ "caption": "Country shares of output with fractional counting, top eight countries.",
69+ "sources": [
70+ "results/bibliometrics/countries_yearly.csv"
71+ ]
37 } 72 }
38 ] 73 ]
\ No newline at end of file
added figures/fig_citation_inequality.pdf +0 −0

Binary file not shown.

added figures/fig_citation_inequality.png +0 −0

Binary file not shown.

added figures/fig_cited_age.pdf +0 −0

Binary file not shown.

added figures/fig_cited_age.png +0 −0

Binary file not shown.

modified figures/fig_collaboration.pdf +0 −0

Binary file not shown.

added figures/fig_country_shares.pdf +0 −0

Binary file not shown.

added figures/fig_country_shares.png +0 −0

Binary file not shown.

added figures/fig_field_inflows.pdf +0 −0

Binary file not shown.

added figures/fig_field_inflows.png +0 −0

Binary file not shown.

modified figures/fig_prisma_flow.pdf +0 −0

Binary file not shown.

modified figures/fig_types_openness.pdf +0 −0

Binary file not shown.

added figures/fig_venue_concentration.pdf +0 −0

Binary file not shown.

added figures/fig_venue_concentration.png +0 −0

Binary file not shown.

modified figures/fig_volume_monthly_raw.pdf +0 −0

Binary file not shown.

modified figures/fig_volume_yearly.pdf +0 −0

Binary file not shown.

modified figures/fig_volume_yearly.png +0 −0

Binary file not shown.

modified figures/make_figures.py +74 −1
@@ -89,7 +89,7 @@ def fig_collab(R, out):
89 89
90 def main(): 90 def main():
91 ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); out = f"{a.root}/figures"; os.makedirs(out, exist_ok=True) 91 ap = argparse.ArgumentParser(); ap.add_argument("--root", required=True); a = ap.parse_args(); out = f"{a.root}/figures"; os.makedirs(out, exist_ok=True)
92− for f in (fig_prisma, fig_yearly_volume, fig_monthly, fig_types_sources, fig_collab): 92+ for f in (fig_prisma, fig_yearly_volume, fig_monthly, fig_types_sources, fig_collab) + _BATCH2:
93 try: 93 try:
94 f(a.root, out); print("ok", f.__name__) 94 f(a.root, out); print("ok", f.__name__)
95 except Exception as e: 95 except Exception as e:
@@ -97,5 +97,78 @@ def main():
97 json.dump(INV, open(f"{out}/FIGURES.json", "w"), indent=1) 97 json.dump(INV, open(f"{out}/FIGURES.json", "w"), indent=1)
98 98
99 99
100+
101+
102+# ---------------- batch 2: citations and knowledge flows (generated from results/bibliometrics + results/graph) ----------------
103+def fig_citation_inequality(R, out):
104+ p = f"{R}/results/bibliometrics/citation_windows_yearly.csv"
105+ if not os.path.exists(p):
106+ return
107+ d = pd.read_csv(p); d = d[d.year <= 2023] # c36 requires three full years of citation data
108+ fig, axes = plt.subplots(1, 2, figsize=(7.2, 3.0))
109+ ax = axes[0]; ax.plot(d.year, d.mean_c36, marker="o", color=style.CATEGORICAL[0], label="Mean c36"); ax.plot(d.year, d.p99_c36 / 10, marker="o", color=style.CATEGORICAL[1], label="p99 c36 ÷ 10")
110+ ax.set_title("Fixed-window citations (36 months)", loc="left"); ax.set_ylabel("Citations"); ax.legend()
111+ ax = axes[1]; ax.plot(d.year, d.gini_c36, marker="o", color=style.CATEGORICAL[2], label="Gini of c36"); ax.plot(d.year, d.share_uncited_36, marker="o", color=style.CATEGORICAL[3], label="Share uncited at 36 months")
112+ ax.set_title("Citation inequality", loc="left"); ax.set_ylim(0, 1); ax.legend()
113+ style.save(fig, f"{out}/fig_citation_inequality.pdf"); reg("fig_citation_inequality", "Fixed-window citation levels and inequality by publication year (years with complete 36-month windows).", ["results/bibliometrics/citation_windows_yearly.csv"])
114+
115+
116+def fig_field_inflows(R, out):
117+ p = f"{R}/results/graph/field_flows_yearly.csv"
118+ if not os.path.exists(p):
119+ return
120+ d = pd.read_csv(p); d = d[(d.citing_year >= 2018) & (d.citing_year <= 2026)]
121+ names = {17: "Computer science (outside corpus)", 26: "Mathematics", 22: "Engineering", 31: "Physics & astronomy", 27: "Medicine", 33: "Social sciences", 13: "Biochem./genetics", 18: "Decision sciences", 28: "Neuroscience", 20: "Economics & finance"}
122+ tot = d.groupby("citing_year").n.sum(); ext = d[d.kind == "external"]
123+ top = ext.groupby("cited_field_id").n.sum().nlargest(8).index
124+ fig, ax = plt.subplots(figsize=(6.8, 3.4))
125+ for i, f in enumerate(top):
126+ s = ext[ext.cited_field_id == f].groupby("citing_year").n.sum() / tot * 100
127+ ax.plot(s.index, s.values, marker="o", color=style.CATEGORICAL[i % 8], label=names.get(int(f), f"field {int(f)}"), lw=1.2)
128+ ax.annotate(names.get(int(f), str(int(f))), (s.index[-1], s.values[-1]), xytext=(3, 0), textcoords="offset points", fontsize=6.5, color=style.INK2, va="center")
129+ ax.set_ylabel("Share of all references (%)"); ax.set_title("References from the CS corpus to works outside it, by cited field", loc="left"); ax.legend(ncol=2, fontsize=6.5)
130+ style.save(fig, f"{out}/fig_field_inflows.pdf"); reg("fig_field_inflows", "Share of references pointing outside the corpus, by cited OpenAlex field and citing year.", ["results/graph/field_flows_yearly.csv"])
131+
132+
133+def fig_cited_age(R, out):
134+ p = f"{R}/results/graph/cited_age_yearly.csv"
135+ if not os.path.exists(p):
136+ return
137+ d = pd.read_csv(p); d = d[(d.age >= 0) & (d.age <= 30)]
138+ fig, ax = plt.subplots(figsize=(6.8, 3.2))
139+ for i, y in enumerate([2018, 2020, 2022, 2024, 2026]):
140+ g = d[d.citing_year == y]; s = g.n / g.n.sum()
141+ ax.plot(g.age, s * 100, color=style.SEQUENTIAL[2 + 2 * i], label=str(y), lw=1.3)
142+ ax.set_xlabel("Age of cited work (years)"); ax.set_ylabel("Share of references (%)"); ax.set_title("Reference age distribution by citing year", loc="left"); ax.legend(title="Citing year")
143+ style.save(fig, f"{out}/fig_cited_age.pdf"); reg("fig_cited_age", "Distribution of the age of cited works for selected citing years.", ["results/graph/cited_age_yearly.csv"])
144+
145+
146+def fig_venue_hhi(R, out):
147+ p = f"{R}/results/bibliometrics/venue_hhi_yearly.csv"
148+ if not os.path.exists(p):
149+ return
150+ d = pd.read_csv(p); d = d[d.year <= 2026]
151+ fig, axes = plt.subplots(1, 2, figsize=(7.2, 3.0))
152+ axes[0].plot(d.year, d.n_sources, marker="o", color=style.CATEGORICAL[0]); axes[0].set_title("Distinct sources (venues) per year", loc="left")
153+ axes[1].plot(d.year, d.hhi * 1e4, marker="o", color=style.CATEGORICAL[1], label="HHI × 10⁴"); axes[1].plot(d.year, d.top1_share * 100, marker="o", color=style.CATEGORICAL[2], label="Top source share (%)"); axes[1].set_title("Venue concentration", loc="left"); axes[1].legend()
154+ style.save(fig, f"{out}/fig_venue_concentration.pdf"); reg("fig_venue_concentration", "Number of distinct venues and venue concentration (HHI, top-source share) by year.", ["results/bibliometrics/venue_hhi_yearly.csv"])
155+
156+
157+def fig_countries(R, out):
158+ p = f"{R}/results/bibliometrics/countries_yearly.csv"
159+ if not os.path.exists(p):
160+ return
161+ d = pd.read_csv(p); d = d[d.year <= 2026]; tot = d.groupby("year").n_frac.sum(); top = d.groupby("country").n_frac.sum().nlargest(8).index
162+ fig, ax = plt.subplots(figsize=(6.8, 3.4))
163+ for i, c in enumerate(top):
164+ s = d[d.country == c].set_index("year").n_frac / tot * 100
165+ ax.plot(s.index, s.values, marker="o", color=style.CATEGORICAL[i], label=c, lw=1.2); ax.annotate(c, (s.index[-1], s.values[-1]), xytext=(3, 0), textcoords="offset points", fontsize=7, color=style.INK2, va="center")
166+ ax.set_ylabel("Share of fractional output (%)"); ax.set_title("Country shares of eligible CS output (fractional counting, top 8)", loc="left"); ax.legend(ncol=4, fontsize=7)
167+ style.save(fig, f"{out}/fig_country_shares.pdf"); reg("fig_country_shares", "Country shares of output with fractional counting, top eight countries.", ["results/bibliometrics/countries_yearly.csv"])
168+
169+
170+_BATCH2 = (fig_citation_inequality, fig_field_inflows, fig_cited_age, fig_venue_hhi, fig_countries)
171+
172+
100 if __name__ == "__main__": 173 if __name__ == "__main__":
101 main() 174 main()
modified taxonomy/classify.py +1 −1
@@ -103,7 +103,7 @@ def prepare(root, db, out_dir, chunk_rows):
103 (SELECT list(subfield_id ORDER BY rank) FROM oa_work_topics t WHERE t.work_id = w.id) AS subfield_ids, 103 (SELECT list(subfield_id ORDER BY rank) FROM oa_work_topics t WHERE t.work_id = w.id) AS subfield_ids,
104 (SELECT flatten(list(x.categories)) FROM link_arxiv_oa l JOIN arxiv_keys x USING(arxiv_id) WHERE l.oa_id = w.id) AS arxiv_cats, 104 (SELECT flatten(list(x.categories)) FROM link_arxiv_oa l JOIN arxiv_keys x USING(arxiv_id) WHERE l.oa_id = w.id) AS arxiv_cats,
105 (SELECT list(DISTINCT d.stream) FROM link_dblp_oa l JOIN dblp_keys d USING(dblp_key) WHERE l.oa_id = w.id) AS dblp_streams 105 (SELECT list(DISTINCT d.stream) FROM link_dblp_oa l JOIN dblp_keys d USING(dblp_key) WHERE l.oa_id = w.id) AS dblp_streams
106− FROM corpus_works w LEFT JOIN oa_abstracts a ON a.work_id = w.id""") 106+ FROM corpus_works w LEFT JOIN oa_abstracts a ON a.work_id = w.id WHERE w.eligible OR w.cs_penumbra""")
107 n = con.execute("SELECT count(*) FROM cls_input").fetchone()[0] 107 n = con.execute("SELECT count(*) FROM cls_input").fetchone()[0]
108 con.execute(f"""COPY (SELECT *, (row_number() OVER ()) // {chunk_rows} AS chunk FROM cls_input) TO '{out_dir}/input' (FORMAT PARQUET, PARTITION_BY (chunk), OVERWRITE_OR_IGNORE, COMPRESSION ZSTD)""") 108 con.execute(f"""COPY (SELECT *, (row_number() OVER ()) // {chunk_rows} AS chunk FROM cls_input) TO '{out_dir}/input' (FORMAT PARQUET, PARTITION_BY (chunk), OVERWRITE_OR_IGNORE, COMPRESSION ZSTD)""")
109 print("prepared", n, "works") 109 print("prepared", n, "works")
modified taxonomy/run_classify_shards.sh +11 −4
@@ -1,14 +1,21 @@
1 #!/bin/sh 1 #!/bin/sh
2−# Classify the corpus with N independent processes (macOS multiprocessing pool hangs with pyarrow in spawned workers). 2+# Classify the corpus with independent processes on two nodes (M3U96b local + M3U96a remote); macOS pools hang with pyarrow.
3−# Usage: sh taxonomy/run_classify_shards.sh [N] — waits for the pass-2 chain (CHAIN_DONE) before preparing the input. 3+# Usage: sh taxonomy/run_classify_shards.sh [N_total] — waits for the pass-2 chain (CHAIN_DONE) before preparing the input.
4−cd ~/wp44_uqo; . .venv/bin/activate; ulimit -n 65536; N=${1:-20} 4+cd ~/wp44_uqo; . .venv/bin/activate; ulimit -n 65536; N=${1:-56}; HALF=$((N/2)); R=M3U96a
5 while ! grep -q CHAIN_DONE logs/define_corpus_v2.log 2>/dev/null; do sleep 60; done 5 while ! grep -q CHAIN_DONE logs/define_corpus_v2.log 2>/dev/null; do sleep 60; done
6 rm -rf data/processed/classification; mkdir -p logs/classify 6 rm -rf data/processed/classification; mkdir -p logs/classify
7 python taxonomy/classify.py --root ~/wp44_uqo --ontology taxonomy/ontology.yaml --prepare > logs/classify/prepare.log 2>&1 7 python taxonomy/classify.py --root ~/wp44_uqo --ontology taxonomy/ontology.yaml --prepare > logs/classify/prepare.log 2>&1
8−i=0; while [ $i -lt $N ]; do 8+ssh $R "mkdir -p ~/wp44_uqo/data/processed/classification ~/wp44_uqo/taxonomy ~/wp44_uqo/logs/classify; rm -rf ~/wp44_uqo/data/processed/classification/*"
9+rsync -aq data/processed/classification/input $R:~/wp44_uqo/data/processed/classification/
10+rsync -aq taxonomy/classify.py taxonomy/ontology.yaml taxonomy/__init__.py $R:~/wp44_uqo/taxonomy/
11+i=0; while [ $i -lt $HALF ]; do
9 nohup python taxonomy/classify.py --root ~/wp44_uqo --ontology taxonomy/ontology.yaml --run --shard $i/$N > logs/classify/shard_$i.log 2>&1 & 12 nohup python taxonomy/classify.py --root ~/wp44_uqo --ontology taxonomy/ontology.yaml --run --shard $i/$N > logs/classify/shard_$i.log 2>&1 &
10 i=$((i+1)); done 13 i=$((i+1)); done
14+ssh $R "cd ~/wp44_uqo; . .venv/bin/activate; i=$HALF; while [ \$i -lt $N ]; do nohup python taxonomy/classify.py --root ~/wp44_uqo --ontology taxonomy/ontology.yaml --run --shard \$i/$N > logs/classify/shard_\$i.log 2>&1 & i=\$((i+1)); done; echo remote started"
11 wait 15 wait
16+while ssh $R "pgrep -f 'classify.py --root' >/dev/null"; do sleep 60; done
17+rsync -aq $R:~/wp44_uqo/data/processed/classification/cats/ data/processed/classification/cats/
18+rsync -aq $R:~/wp44_uqo/data/processed/classification/primary/ data/processed/classification/primary/
12 python - <<EOF2 19 python - <<EOF2
13 import duckdb, json, datetime as dt, yaml 20 import duckdb, json, datetime as dt, yaml
14 con = duckdb.connect(); o = yaml.safe_load(open("taxonomy/ontology.yaml")) 21 con = duckdb.connect(); o = yaml.safe_load(open("taxonomy/ontology.yaml"))
15 22
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere