SPB Git

spb/wp44_uqo

Public

WP44: semantic pipeline script, Part IX brief

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 3, 2026) parent 5aec0c2 Browse files

2 changed files +39 −0

added agents/briefs/WRITER_PART09.md +26 −0
@@ -0,0 +1,26 @@
1 +# Brief — Part IX "Empirical Results" (file manuscript/sections/part09_results.tex; bib manuscript/references_part09.bib)
2 +
3 +Target 40–60 pages: ALL verified quantitative findings, with figures and tables, organised by research question RQ1–RQ10 and
4 +hypothesis H1–H8 (protocol/PROTOCOL.md P1). Follow WRITER_COMMON.md. Sources (corpus v2, classification v1.0):
5 +- Corpus & taxonomy: data/metadata/prisma_counts.json, linkage_report.json, linkage_sample_summary.json (H-1 auto precision),
6 + data/processed/classification/classification_report.json (copy in reports/), taxonomy/validation/metrics.json (+ disagreements.csv counts).
7 +- Volume/composition (RQ1): results/bibliometrics/volume_yearly.csv, volume_monthly.csv, results/graph/category_counts_yearly.csv,
8 + category_counts_monthly.csv, results/bibliometrics/cagr_by_category.csv, diversity_yearly.csv, category_share_ci_yearly.csv.
9 +- Structural change (RQ2, H1, H3): results/econometrics/its_results.json (series total and cat_<id>), breaks.csv, placebo_*.csv,
10 + reports/PHASE4_STRUCTURAL_CHANGE_TOTAL.md (preliminary reading).
11 +- Influence (RQ3): results/bibliometrics/citation_windows_yearly.csv, top_cited_by_year.csv, literature_synthesis/landmarks/selection_report.json.
12 +- Networks (RQ4–RQ6, H4–H6): results/graph/{field_flows_yearly, category_flows_yearly, area_flows_yearly, cross_area_share_yearly,
13 + modularity_yearly, category_cooccurrence_yearly, cited_age_yearly, reference_coverage_yearly}.csv; results/bibliometrics/{team_size,
14 + international_collab, industry_share, countries, venue_hhi}_yearly.csv (Part VIII covers the actor networks in depth: summarise and cross-reference).
15 +- Semantic (RQ5, H4 semantic level): results/semantic/{category_drift.csv, category_similarity_baseline.csv, category_similarity_post.csv,
16 + clusters.csv, stability.json} if present (else \pending{semantic atlas}).
17 +- Openness/reproducibility (RQ9, H7): volume_yearly.csv columns n_arxiv, n_oa, n_preprint, n_abstract; top_sources_by_period.csv.
18 +- Emerging detection (RQ10, H8): results/emerging/metrics_h12.json, metrics_h24.json, predictions.csv, known_set_lead_times.csv.
19 +- Figures in manuscript/figures/ (see figures/FIGURES.json for captions/sources) and tables in manuscript/tables/ (TABLES.json):
20 + include every relevant one with \includegraphics / \input and a \label; you may add tables to tables/make_tables.py (append only) and re-run it.
21 +Rules: every number through macros (`python3 tables/make_numbers.py .` first; macro names derive from JSON keys and CSV
22 +columns — e.g. \PrismaCsCoreEligible, \ItsSeriesTotalItsChatgptSlopeChange is NOT generated; for ITS numbers use the generated
23 +`\ItsSeries...` macros if present, else add the table and cite it) or generated tables. State each hypothesis, what the test
24 +shows, and the verdict (supported / not supported / inconclusive) with uncertainty; null and opposite results are reported as
25 +such; no causal language; right-truncation and date-precision caveats on every time-series statement. Flag with \pending{H-4}
26 +the interpretive statements that await PI review.
added scripts/run_semantic.sh +13 −0
@@ -0,0 +1,13 @@
1 +#!/bin/sh
2 +# Semantic layer on M3U96b: waits for PHASE26_DONE and for the embeddings collection flag (logs/embeddings_collected.flag,
3 +# raised from the laptop by embeddings/collect_multinode.sh), then builds the atlas, the k-NN edges and re-runs diversity with
4 +# semantic similarities and the figures.
5 +cd ~/wp44_uqo; . .venv/bin/activate; export WP44_DUCKDB_MEM=${WP44_DUCKDB_MEM:-40GB}; ulimit -n 65536
6 +while ! grep -q PHASE26_DONE logs/p26_pipeline.log 2>/dev/null; do sleep 60; done
7 +while [ ! -f logs/embeddings_collected.flag ]; do sleep 60; done
8 +run() { echo "[$(date +%H:%M:%S)] $1"; shift; "$@" > logs/sem_$(echo $1 | tr '/' '_' | sed 's/\.py//').log 2>&1 || echo "FAILED: $*"; }
9 +run "semantic atlas" python embeddings/semantic_atlas.py --root ~/wp44_uqo --n 200000
10 +run "semantic edges" python knowledge_graph/04_semantic_edges.py --root ~/wp44_uqo --k 10 --min-sim 0.80
11 +run "diversity (semantic)" python bibliometrics/04_diversity_cagr.py --root ~/wp44_uqo
12 +run "figures" python figures/make_figures.py --root ~/wp44_uqo
13 +echo SEMANTIC_DONE >> logs/sem_pipeline.log
14
Menu

Theme