# Reproducibility runbook — WP44

Everything below was executed on M3U96b (Mac Studio M3 Ultra, 28 cores, 96 GB, macOS 27.0) in `~/wp44_uqo` with the
locked environment `reproducibility/environment_M3U96b.txt` (Python 3.12.15 via uv). Raw sources are public; the exact
inputs are pinned by manifests and hashes (`data/raw/README.md`, ledgers). Nothing requires credentials except the two LLM
raters of the taxonomy validation (Anthropic API key, never stored in the repository).

| Step | Command | Inputs | Outputs | Wall time (measured) |
|---|---|---|---|---|
| 1 OpenAlex works snapshot (pass 1, CS filter + slim table) | `python ingestion/openalex_snapshot.py --out data/raw/openalex/works --manifest data/raw/openalex/works_manifest_2026-10-14.json --workers 8` | S3 public bucket, release 2026-10-14 | `works/cs/*.jsonl.zst` (15,490,159), `works/slim/*.parquet` (464,765,485), `ledger.sqlite` | 1 h 46 min at ≈ 108 MB/s |
| 2 OpenAlex entities | `python ingestion/openalex_entities.py --out data/raw/openalex` | S3 | `topics/ … institutions/`, `entities_ledger.json` | 5 min |
| 3 DBLP | `python ingestion/dblp_parse.py --xml data/raw/dblp/dblp.xml.gz --out data/raw/dblp` | dump 2026-11-02 (MD5 verified) | `dblp_publications.parquet` (8,840,866), report | 4 min 45 s |
| 4 arXiv OAI-PMH | `python ingestion/arxiv_oai.py --out data/raw/arxiv --from 2018-01-01 --until 2026-11-02` | OAI feed, set `cs` | raw XML pages (758), `arxiv_cs.parquet` (915,115) | ≈ 1 h |
| 5 ACL Anthology | `python ingestion/acl_parse.py --bib data/raw/acl/anthology+abstracts.bib.gz --out data/raw/acl` | bib export | `acl_anthology.parquet` (138,796) | 10 s |
| 6 Normalise | `python ingestion/build_corpus.py --cs-dir data/raw/openalex/works/cs --out data/processed --workers 12` | step 1 | `tables/<table>/*.parquet` | 2 min 10 s |
| 7 Compact | `ulimit -n 65536; python ingestion/compact_tables.py ~/wp44_uqo` | step 6 | `compact/<table>.parquet` | 30 s |
| 8 Link | `python ingestion/link_sources.py --root ~/wp44_uqo` | steps 3–7 | DuckDB `wp44.duckdb`, `link_*`, `data/metadata/linkage_report.json`, `needed_oa_ids.csv` | 10 s |
| 9 Second pass (DBLP/arXiv-linked works outside the CS-topic filter) | `python ingestion/openalex_snapshot.py --out data/raw/openalex/works_pass2 --manifest … --workers 8 --id-list data/metadata/needed_oa_ids.csv` then `ingestion/run_pass2_chain.sh` (steps 6–8 with suffix `_p2`, then 10–11) | S3 | `works_pass2/cs/*.jsonl.zst`, `compact/<table>_p2.parquet` | ≈ 1 h 50 min |
| 10 Corpus definition | `python systematic_review/define_corpus.py --root ~/wp44_uqo` | step 8 | `corpus_works`, `data/metadata/prisma_counts.json` | 2 min |
| 11 Volume, collaboration, citations, graph | `python bibliometrics/01_volume_series.py …`, `02_collaboration.py`, `03_citation_impact.py`, `knowledge_graph/01_citation_graph.py` | step 10 | `results/bibliometrics/*.csv`, `results/graph/*`, `data/processed/graph/*` | minutes |
| 12 Classification | `python taxonomy/classify.py --root ~/wp44_uqo --ontology taxonomy/ontology.yaml --prepare --run` | steps 8, 10, `taxonomy/ontology.yaml` | `data/processed/classification/{cats,primary}/*.parquet`, report | (to be measured) |
| 13 Validation sample + raters | `python taxonomy/validate_sample.py --root … --draw`; `--rate --model claude-haiku-5-5`; `--rate --model claude-sonnet-5-5`; `--metrics` | step 12 | `taxonomy/validation/*` | (to be measured) |
| 14 Embeddings | `python embeddings/embed_corpus.py --root … --prepare`; `--run --model allenai/specter2_base` (sharded over nodes) | step 10 | `data/processed/embeddings/<model>/shard_*.npy` | ≈ 300 texts/s per M3 Ultra |
| 15 Structural change, emerging detection, figures, tables | `econometrics/run_structural_change.py`, `experiments/emerging_detection.py`, `figures/make_figures.py`, `tables/make_numbers.py` | 11–14 | `results/econometrics`, `results/emerging`, `figures/`, `manuscript/tables/numbers.tex` | minutes |
| 16 Manuscript | `cd manuscript && make` | all | `manuscript/main.pdf` | — |

Tests: `python -m pytest -q tests/` (structural-change toolkit, classification scoring, corpus rules).