WP35: leakage check (A-001), Holm table, replication guide, raw-data README, final audit draft, series README rows
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
10 changed files +220 −1
added
data/raw/README.md
+19 −0
@@ -0,0 +1,19 @@ | |||||
| 1 | +# Raw data (not committed) | ||||
| 2 | + | ||||
| 3 | +Raw files are downloaded by `python -m wp35.acquisition.fetch` into `data/raw/<key>/` and are never modified. Their | ||||
| 4 | +release, URL, retrieval time, licence, size and SHA-256 are recorded in `data/metadata/source_registry.json`; a re-run | ||||
| 5 | +verifies the hash of any existing file and refuses to overwrite it. Nothing in this directory is redistributed in the | ||||
| 6 | +repository because of the D-PLACE (CC BY-NC 4.0) and V-Dem (non-commercial, citation) terms; every file can be | ||||
| 7 | +re-obtained from its public URL and checked byte for byte against the registry. | ||||
| 8 | + | ||||
| 9 | +| key | file | release (retrieved 2026-11-01) | | ||||
| 10 | +|---|---|---| | ||||
| 11 | +| seshat_polaris2026 | Polaris2026.xlsx | Seshat Polaris-2026 snapshot, GitHub `Seshat-Global-History-Databank/build_polaris_dataset` @ 55ca5fbc2b (MIT) | | ||||
| 12 | +| seshat_equinox2020 | Equinox2020.05.2023.xlsx | Equinox-2020 spreadsheet, same repository | | ||||
| 13 | +| seshat_equinox_continuous_polities | Timescale.SM_Data.xlsx | Zenodo 8120128 (CC BY 4.0), MD5 verified | | ||||
| 14 | +| maddison_2023_xlsx / _dta | mpd2023_web.xlsx / .dta | Maddison Project Database 2023, dataverse.nl 421302 / 421303 | | ||||
| 15 | +| dplace_ea_v3_3 | dplace-dataset-ea-v3.3.zip | D-PLACE Ethnographic Atlas CLDF v3.3 (CC BY-NC 4.0) | | ||||
| 16 | +| vdem_v16_rdata / _codebook | vdem_v16.RData / vdem_v16_codebook.RData | V-Dem v16 from the official `vdemdata` R package @ f4dd26922e | | ||||
| 17 | +| glottolog_cldf_languages | glottolog_v5.3_languages.csv | Glottolog 5.3 CLDF languages table (CC BY 4.0) | | ||||
| 18 | + | ||||
| 19 | +Total size about 65 MB. | ||||
modified
paper/main.pdf
+0 −0
Binary file not shown.
modified
paper/sections/results_historical.tex
+1 −1
@@ -54,7 +54,7 @@ Specification & Term & Coef. & SE & $n$ \\ | |||||
| 54 | \end{tabular} | 54 | \end{tabular} | ||
| 55 | \end{table} | 55 | \end{table} | ||
| 56 | 56 | ||||
| 57 | −The pre-registration asked a different question, and the rest of Table~\ref{tab:h2-seshat} answers it. Scale alone is associated with severity at $0.26$ per standard deviation (SE 0.14); capacity alone at $-0.03$ (SE 0.20); the Turchin-style complexity component at $-0.01$ (SE 0.24); the persistence indicator at $0.35$ (SE 0.24). When scale and capacity are entered separately, scale carries $0.34$ (SE 0.16) and capacity $-0.21$ (SE 0.19): capacity has the sign the hypothesis predicts, but it is not distinguishable from zero. The placebo index, which replaces capacity by the share of production-infrastructure indicators that were excluded from $K$ precisely because they are not coordination capacity, is associated with severity at $0.26$ (SE 0.14)---the same as scale alone, and within sampling error of $P$. Out of sample (Table~\ref{tab:h2-seshat-cv}), the model with $P$ has a higher log-likelihood than the fixed-effects-only, capacity-only and complexity-only models in 79--85\% of the fifty cross-validation folds, but beats the scale-only model in 75\%, the persistence model in 69\% and the scale-plus-capacity model in 75\%, below the 80\% rule; with macro-regions held out it loses to scale alone in seven of ten folds and to scale plus capacity in eight. For the binary outcome, which was declared underpowered in advance, the coefficient is negative and imprecise ($-0.18$, SE 0.20 with region effects; $-0.07$, SE 0.18 without), and no model discriminates severe from non-severe crises out of sample (areas under the curve 0.62--0.64 for every model, including the baselines). Reform consequences are unrelated to $P$. The conclusion we are obliged to draw is that what the mismatch index predicts about crisis severity is what polity scale predicts, and that nothing in the data distinguishes a capacity shortfall from a large population. | 57 | +The pre-registration asked a different question, and the rest of Table~\ref{tab:h2-seshat} answers it. Scale alone is associated with severity at $0.26$ per standard deviation (SE 0.14); capacity alone at $-0.03$ (SE 0.20); the Turchin-style complexity component at $-0.01$ (SE 0.24); the persistence indicator at $0.35$ (SE 0.24). When scale and capacity are entered separately, scale carries $0.34$ (SE 0.16) and capacity $-0.21$ (SE 0.19): capacity has the sign the hypothesis predicts, but it is not distinguishable from zero. The placebo index, which replaces capacity by the share of production-infrastructure indicators that were excluded from $K$ precisely because they are not coordination capacity, is associated with severity at $0.26$ (SE 0.14)---the same as scale alone, and within sampling error of $P$. Out of sample (Table~\ref{tab:h2-seshat-cv}), the model with $P$ has a higher log-likelihood than the fixed-effects-only, capacity-only and complexity-only models in 79--85\% of the fifty cross-validation folds, but beats the scale-only model in 75\%, the persistence model in 69\% and the scale-plus-capacity model in 75\%, below the 80\% rule; with macro-regions held out it loses to scale alone in seven of ten folds and to scale plus capacity in eight. Because the principal-component loadings of $D$ and the standardization of $P$ were estimated on the full polity sample (without any outcome) before cross-validation, we repeated the leave-one-region-out comparison with loadings and standardization re-estimated inside each training fold: $P$ then beats scale alone in two of ten folds (\texttt{h2\_seshat\_cv\_noleakage.csv}), so the verdict does not depend on that pre-processing. For the binary outcome, which was declared underpowered in advance, the coefficient is negative and imprecise ($-0.18$, SE 0.20 with region effects; $-0.07$, SE 0.18 without), and no model discriminates severe from non-severe crises out of sample (areas under the curve 0.62--0.64 for every model, including the baselines). Reform consequences are unrelated to $P$. The conclusion we are obliged to draw is that what the mismatch index predicts about crisis severity is what polity scale predicts, and that nothing in the data distinguishes a capacity shortfall from a large population. | ||
| 58 | 58 | ||||
| 59 | \begin{table}[H] | 59 | \begin{table}[H] | ||
| 60 | \centering\footnotesize | 60 | \centering\footnotesize | ||
added
replication/README.md
+41 −0
@@ -0,0 +1,41 @@ | |||||
| 1 | +# Replication guide — UQO WP 35 | ||||
| 2 | + | ||||
| 3 | +## Environment | ||||
| 4 | +- macOS or Linux, Python 3.12 (managed with `uv`), TeX Live with `latexmk` and the `aer` bibliography style, `make`. | ||||
| 5 | +- Pinned dependencies in `pyproject.toml` (numpy 2.3.4, pandas 2.3.3, scipy 1.16.2, statsmodels 0.14.5, lifelines 0.30.0, | ||||
| 6 | + scikit-learn 1.7.2, networkx 3.5, matplotlib 3.10.7, pyreadr 0.5.3, openpyxl 3.1.5, pyarrow 21.0.0, pytest 8.4.2, …). | ||||
| 7 | + | ||||
| 8 | +```bash | ||||
| 9 | +uv venv -p 3.12 .venv && uv pip install -p .venv/bin/python -e . | ||||
| 10 | +``` | ||||
| 11 | + | ||||
| 12 | +## One-command reproduction | ||||
| 13 | +```bash | ||||
| 14 | +make all # fetch → data → measure → econ → sim → figures → paper | ||||
| 15 | +make test # 11 tests | ||||
| 16 | +``` | ||||
| 17 | +Individual targets, in dependency order: | ||||
| 18 | + | ||||
| 19 | +| target | command | outputs | time (M5 Max laptop) | | ||||
| 20 | +|---|---|---|---| | ||||
| 21 | +| fetch | `python -m wp35.acquisition.fetch` | `data/raw/*`, `data/metadata/source_registry.json` | ~1 min (65 MB) | | ||||
| 22 | +| data | `python -m wp35.harmonization.build_panels` | `data/processed/*.parquet`, `results/diagnostics/*`, coverage figures | ~1 min | | ||||
| 23 | +| measure | `python -m wp35.measurement.build_indices` | `data/processed/*_indices.parquet`, `results/empirical/measurement_*`, `power_h2a_*` | ~10 s | | ||||
| 24 | +| econ | `python -m wp35.econometrics.run_all` (= h1, h2_seshat, h2_modern) | `results/empirical/h1_*`, `h2_*`, `h4_*`, `h5_*` | ~15 min (modern panel dominates) | | ||||
| 25 | +| sim | `python -m wp35.simulations.run_experiments && python -m wp35.simulations.analyze` | `results/simulated/*` and `fig_sim_*` | ~1 min on 12 cores | | ||||
| 26 | +| figures | `python -m wp35.visualization.make_all` | `paper/tables/*.tex`, `paper/figures/*.pdf` | ~10 s | | ||||
| 27 | +| paper | `python -m wp35.validation.build_bib && cd paper && latexmk -pdf main.tex` | `paper/references.bib`, `paper/main.pdf` | ~1 min | | ||||
| 28 | + | ||||
| 29 | +## Determinism | ||||
| 30 | +- Estimation seed 20261101 (cross-validation folds, power simulations); simulation seeds 0–7; IRT is deterministic (EM). | ||||
| 31 | +- `python -m wp35.validation.verify_bib` re-queries Crossref and may differ if Crossref metadata changes; the verified list used for the paper is `research/prior_art.csv` + `research/bib_overrides.csv`. | ||||
| 32 | + | ||||
| 33 | +## Provenance of every number | ||||
| 34 | +Each table fragment in `paper/tables/` is written by `src/wp35/visualization/make_all.py` from a file in `results/`; | ||||
| 35 | +each figure likewise. The execution log `results/logs/execution_log.md` records every command and every correction | ||||
| 36 | +made during the study; `results/logs/final_research_audit.md` lists completed, failed and blocked tasks. | ||||
| 37 | + | ||||
| 38 | +## Pre-registration | ||||
| 39 | +`research/preregistration.md` was frozen before any predictor–outcome association was computed; SHA-256 | ||||
| 40 | +`ecda8804e17f931789be8e274ba1ff989000de356700a85b638b043928182e3a` (recorded in `research/decisions.md`, D-003). | ||||
| 41 | +Verify with `shasum -a 256 research/preregistration.md`. | ||||
modified
research/decisions.md
+1 −0
@@ -9,3 +9,4 @@ | |||||
| 9 | | D-005 | 2026-11-01 | 1 | Historical unit = polity, not polity-year; CrisisDB test = severity conditional on crisis; add polity hazard (Y2) and modern onset panel (Y4–Y6) | 93.7% of Polaris facts are undated; CrisisDB is crisis-selected (150/626 polities) | before | | 9 | | D-005 | 2026-11-01 | 1 | Historical unit = polity, not polity-year; CrisisDB test = severity conditional on crisis; add polity hazard (Y2) and modern onset panel (Y4–Y6) | 93.7% of Polaris facts are undated; CrisisDB is crisis-selected (150/626 polities) | before | | ||
| 10 | | D-006 | 2026-11-01 | 1 | Macro-regions and era blocks fixed in `configs/regions.yaml` (10 macro-regions, 6 eras) | Needed for FE and leave-region-out; 42 regions too many for N≈150 | before | | 10 | | D-006 | 2026-11-01 | 1 | Macro-regions and era blocks fixed in `configs/regions.yaml` (10 macro-regions, 6 eras) | Needed for FE and leave-region-out; 42 regions too many for N≈150 | before | | ||
| 11 | | D-007 | 2026-11-01 | 1 | No paid LLM API calls; literature verified via Crossref/public pages | No keys configured; mandate forbids unapproved spending | — | | 11 | | D-007 | 2026-11-01 | 1 | No paid LLM API calls; literature verified via Crossref/public pages | No keys configured; mandate forbids unapproved spending | — | | ||
| 12 | +| A-001 | 2026-11-01 | 4 | Added a fold-internal re-estimation of the $D$ loadings and of the standardization of $P$ for the Seshat leave-one-region-out comparison (`econometrics/leakage_check.py`) | Pre-registration §(mandate rule) requires preprocessing fitted on training folds for predictive tests; the main CV used full-sample (outcome-free) preprocessing | after seeing the main CV results, before the adversarial reviews; conclusion unchanged (P beats scale-only in 2/10 folds vs 3/10) | | ||||
added
results/empirical/h2_seshat_cv_noleakage.csv
+11 −0
@@ -0,0 +1,11 @@ | |||||
| 1 | +held_out,n_test,ll_A2_P,ll_B1_D,P_wins | ||||
| 2 | +Central_Asia,3,-0.490219768651215,-0.48427034002006586,False | ||||
| 3 | +East_Asia,23,-0.6190229903188945,-0.6154344820757043,False | ||||
| 4 | +Europe,15,-0.5416072201098796,-0.5550136443510748,True | ||||
| 5 | +North_Africa,9,-0.6361460685815892,-0.6322435440965184,False | ||||
| 6 | +North_America,2,-0.6866907544329729,-0.6829083738744576,False | ||||
| 7 | +South_America,3,-0.5428807701200635,-0.5219865203262556,False | ||||
| 8 | +South_Asia,5,-0.5565224201137469,-0.5571969007432479,True | ||||
| 9 | +Southeast_Asia_Oceania,6,-0.5486244832780911,-0.5379889522153865,False | ||||
| 10 | +Southwest_Asia,17,-0.6046176865728712,-0.6025778312028729,False | ||||
| 11 | +Sub_Saharan_Africa,4,-0.5882895353572101,-0.5757067212308518,False | ||||
added
results/empirical/holm_primary_tests.csv
+4 −0
@@ -0,0 +1,4 @@ | |||||
| 1 | +test,p_raw,p_holm | ||||
| 2 | +"H1 Seshat primary (S1, K_mean)",1.9434199250145317e-42,5.830259775043595e-42 | ||||
| 3 | +"H2-A Seshat primary (A1, P)",0.0004870020882165,0.0004870020882165 | ||||
| 4 | +"H2-C modern primary (C1, P)",8.131284893393963e-06,1.6262569786787926e-05 | ||||
modified
results/logs/execution_log.md
+1 −0
@@ -29,3 +29,4 @@ | |||||
| 29 | | 2026-11-01 06:50 | 5 | Simulation model revised after inspection (Shannon code lengths to match Theorem 3; symmetric link removal 50%; spatially correlated dialects; fixed pre/post window for difference-in-differences; high-gain break-even cells) ; `run_experiments` + `analyze` rerun | `results/simulated/runs_main.parquet` (6,048), `runs_ablation.parquet` (584), `simulation_summary.json`, `cells_main.csv`, `paired_vs_none.csv`, `institution_loss_shares.csv`, `shock_effects.csv`, `ablation_*.csv`; figures `fig_sim_*.pdf` | | 29 | | 2026-11-01 06:50 | 5 | Simulation model revised after inspection (Shannon code lengths to match Theorem 3; symmetric link removal 50%; spatially correlated dialects; fixed pre/post window for difference-in-differences; high-gain break-even cells) ; `run_experiments` + `analyze` rerun | `results/simulated/runs_main.parquet` (6,048), `runs_ablation.parquet` (584), `simulation_summary.json`, `cells_main.csv`, `paired_vs_none.csv`, `institution_loss_shares.csv`, `shock_effects.csv`, `ablation_*.csv`; figures `fig_sim_*.pdf` | | ||
| 30 | | 2026-11-01 07:30 | 7 | `python -m wp35.visualization.make_all` (26 table fragments, 15 figures) ; `python -m wp35.validation.build_bib` (182 entries: Crossref by DOI + verified manual) | `paper/tables/*.tex`, `paper/figures/*.pdf`, `paper/references.bib` | | 30 | | 2026-11-01 07:30 | 7 | `python -m wp35.visualization.make_all` (26 table fragments, 15 figures) ; `python -m wp35.validation.build_bib` (182 entries: Crossref by DOI + verified manual) | `paper/tables/*.tex`, `paper/figures/*.pdf`, `paper/references.bib` | | ||
| 31 | | 2026-11-01 08:00 | 8 | Paper written (sections: introduction, literature [fork], theory [fork], data, methods, results_historical, results_modern, simulation, synthesis, limitations, conclusion, appendix) ; `latexmk -pdf main.tex` — fixes: `\input{}` → primitive `\input` in tabulars (misplaced \noalign), aer.bst header/month/underscore/encoding issues in references.bib, novelty table rotated | `paper/main.pdf` 73 pages, 0 undefined citations/references | | 31 | | 2026-11-01 08:00 | 8 | Paper written (sections: introduction, literature [fork], theory [fork], data, methods, results_historical, results_modern, simulation, synthesis, limitations, conclusion, appendix) ; `latexmk -pdf main.tex` — fixes: `\input{}` → primitive `\input` in tabulars (misplaced \noalign), aer.bst header/month/underscore/encoding issues in references.bib, novelty table rotated | `paper/main.pdf` 73 pages, 0 undefined citations/references | | ||
| 32 | +| 2026-11-01 08:20 | 4 | `python -m wp35.econometrics.leakage_check` (fold-internal PCA loadings and standardization, LORO) | `results/empirical/h2_seshat_cv_noleakage.csv`: P beats D-only in 2/10 folds (main run: 3/10) ; amendment A-001 | | ||||
added
results/logs/final_research_audit.md
+93 −0
@@ -0,0 +1,93 @@ | |||||
| 1 | +# WP35 — Final research audit (checklist) | ||||
| 2 | + | ||||
| 3 | +Date: 2026-11-01. Executor: Claude Code (claude-fable-5-1), autonomous, under the PI's mandate (`docs/MANDATE_VERBATIM.txt`). | ||||
| 4 | +Status codes: **DONE** (code ran, output exists, source traceable, limitations documented), **PARTIAL**, **FAILED**, | ||||
| 5 | +**BLOCKED** (needs PI action or data that does not exist), **NOT EXECUTED** (planned, not run). | ||||
| 6 | + | ||||
| 7 | +## Phase 0 — Environment and feasibility | ||||
| 8 | +| Task | Status | Evidence | | ||||
| 9 | +|---|---|---| | ||||
| 10 | +| Hardware/tools/LaTeX/connectivity audit | DONE | `results/logs/feasibility_report.md` | | ||||
| 11 | +| Real sample downloads with hashes | DONE | `data/metadata/source_registry.json` (9 sources, SHA-256, retrieval times) | | ||||
| 12 | +| LLM API access check | DONE (none used) | no keys configured; no paid calls; `anthropic` MCP failed to connect this session | | ||||
| 13 | + | ||||
| 14 | +## Phase 1 — Literature and pre-registration | ||||
| 15 | +| Task | Status | Evidence | | ||||
| 16 | +|---|---|---| | ||||
| 17 | +| Reference verification against Crossref | DONE | `research/prior_art.csv` (145/173 verified by query; corrections and books in `bib_overrides.csv`; 182 entries in `paper/references.bib`) | | ||||
| 18 | +| Novelty matrix | DONE | `research/literature_review.md` §2; `docs/novelty_search_report.md`; paper Table 1 | | ||||
| 19 | +| Pre-registration frozen before outcome associations | DONE | `research/preregistration.md`, SHA-256 `ecda8804…` in `research/decisions.md` D-003 | | ||||
| 20 | + | ||||
| 21 | +## Phase 2 — Data engineering | ||||
| 22 | +| Task | Status | Evidence | | ||||
| 23 | +|---|---|---| | ||||
| 24 | +| Seshat Polaris-2026 parser (codes, ranges, dates, flags, pre-crisis aggregates, CrisisDB, succession) | DONE | `src/wp35/harmonization/seshat.py`; `data/processed/seshat_*.parquet`; 132 unparsed values logged | | ||||
| 25 | +| D-PLACE EA v3.3 + Glottolog families | DONE | `dplace.py`; `dplace_ea_societies.parquet`; code midpoints logged | | ||||
| 26 | +| V-Dem v16 + Maddison 2023 crosswalk and events | DONE | `modern.py`; `modern_panel.parquet`; `crosswalk_vdem_maddison.csv` (39 unmatched) | | ||||
| 27 | +| Coverage tables, temporal figures, missingness diagnostics, data dictionary | DONE | `results/diagnostics/*`, `paper/figures/fig_seshat_*`, `data/metadata/data_dictionary.md` | | ||||
| 28 | +| Equinox-2020 alternative release parsed | NOT EXECUTED | listed in pre-registration §6 item 8; raw file downloaded; parser not written (time) | | ||||
| 29 | + | ||||
| 30 | +## Phase 3 — Theory and measurement | ||||
| 31 | +| Task | Status | Evidence | | ||||
| 32 | +|---|---|---| | ||||
| 33 | +| Formal framework: 5 theorems with proofs, 3 conjectures, mapping to tests | DONE | `research/framework.md`, paper §3 | | ||||
| 34 | +| D, K, P indices; IRT alternative; PC alternative; bounds; invariance; leave-one-block-out | DONE | `src/wp35/measurement/build_indices.py`; `results/empirical/measurement_*` | | ||||
| 35 | +| A-priori power (MDE) | DONE | `power_h2a_binary.csv`, `power_h2a_fractional.csv`; Y1b declared underpowered a priori | | ||||
| 36 | +| Measurement report | DONE | `research/measurement.md` | | ||||
| 37 | + | ||||
| 38 | +## Phase 4 — Historical econometrics | ||||
| 39 | +| Task | Status | Evidence | | ||||
| 40 | +|---|---|---| | ||||
| 41 | +| H1 (Seshat OLS/mixed; EA Conley + ordered logit; modern within/differences) | DONE | `h1_seshat.csv`, `h1_ea.csv`, `h1_modern.csv` | | ||||
| 42 | +| H2-A primary (fractional logit), baselines, placebo, robustness set, LORO + 10×5 CV, 80% rule | DONE | `h2_seshat_primary.csv`, `_robustness.csv`, `_cv_*.csv`; fold-internal leakage check `h2_seshat_cv_noleakage.csv` (A-001) | | ||||
| 43 | +| H2-A binary (Y1b) | DONE (underpowered; quasi-separation → region FE/HC1) | `h2_seshat_primary.csv` rows A1b–A3b | | ||||
| 44 | +| Y2 Cox, Y3 succession | DONE | `h2_seshat_y2_y3.csv` | | ||||
| 45 | +| Dated-only "strict" pre-crisis sensitivity | FAILED (not estimable) | too few dated facts; see `research/limitations.md` A.2 | | ||||
| 46 | +| H2-C modern: pooled/conditional/LPM, baselines, placebo, negative control, forward chaining, LORO, calibration, Oster/Cinelli–Hazlett | DONE | `h2_modern_primary.csv`, `_robustness.csv`, `_cv_*`, `h2_modern_cv_predictions.parquet` | | ||||
| 47 | +| H4 (hinge/spline OOS) both panels | DONE | `h4_seshat_summary.json`, `h4_modern_summary.json`, `h4_modern_curve.csv` | | ||||
| 48 | +| H5 (held-out blocks) both panels | DONE | `h5_seshat_blocks.csv`, `h5_modern_blocks.csv` | | ||||
| 49 | +| Holm correction across the three primary tests | DONE | `results/empirical/holm_primary_tests.csv` (all adjusted p < 0.01; the H2 tests nevertheless fail the pre-registered out-of-sample rule) | | ||||
| 50 | +| Identification note | DONE | `research/identification.md`, paper §5.5 | | ||||
| 51 | + | ||||
| 52 | +## Phase 5 — Simulations | ||||
| 53 | +| Task | Status | Evidence | | ||||
| 54 | +|---|---|---| | ||||
| 55 | +| ABM with heterogeneous agents, networks, bandwidth, shocks, five imposed regimes | DONE | `src/wp35/simulations/model.py` | | ||||
| 56 | +| Factorial design (6,048 runs, 8 seeds) + ablations (584) | DONE | `results/simulated/runs_main.parquet`, `runs_ablation.parquet` | | ||||
| 57 | +| Predefined metrics incl. bits per success, overhead, Gini, latency; Shannon quantities where defined | DONE | `simulation_summary.json`, `cells_main.csv` | | ||||
| 58 | +| Fragility search (cells where institutions lose) | DONE | `institution_loss_shares.csv`, `cells_institution_vs_none.csv` | | ||||
| 59 | +| Theorem checks in silico | DONE | `simulation_summary.json` → `theorem_checks` | | ||||
| 60 | + | ||||
| 61 | +## Phase 6 — Integration | ||||
| 62 | +| Task | Status | Evidence | | ||||
| 63 | +|---|---|---| | ||||
| 64 | +| Observable vs non-observable mechanisms; rivals; consistency; scope conditions | DONE | `research/integration.md`, paper §9 | | ||||
| 65 | + | ||||
| 66 | +## Phase 7 — Figures, tables, reproducibility | ||||
| 67 | +| Task | Status | Evidence | | ||||
| 68 | +|---|---|---| | ||||
| 69 | +| All tables/figures generated by script | DONE | `src/wp35/visualization/make_all.py` (26 fragments, 15 figures) | | ||||
| 70 | +| Tests | DONE | `tests/` — 11 passed | | ||||
| 71 | +| Clean-environment full reproduction | PARTIAL | every target was run in sequence from the venv in this session; a from-scratch `make all` on a second machine was not run | | ||||
| 72 | + | ||||
| 73 | +## Phase 8 — Paper | ||||
| 74 | +| Task | Status | Evidence | | ||||
| 75 | +|---|---|---| | ||||
| 76 | +| LaTeX manuscript compiled, references resolved, figures present | DONE | `paper/main.pdf` 73 pp.; 0 undefined citations/references; 6 small overfull boxes | | ||||
| 77 | +| Abstract written after analyses | DONE | — | | ||||
| 78 | + | ||||
| 79 | +## Phase 9 — Adversarial review | ||||
| 80 | +| Task | Status | Evidence | | ||||
| 81 | +|---|---|---| | ||||
| 82 | +| Three independent reviews | see `docs/reviews/` | filled in below after the reviews | | ||||
| 83 | +| Response to reviewers and implemented corrections | see `docs/reviews/response_to_reviewers.md` | filled in below | | ||||
| 84 | + | ||||
| 85 | +## Blocked / requires PI | ||||
| 86 | +- Creation of the spbgit remote `wp35_uqo.git` and `git push` (authorization). | ||||
| 87 | +- Any external publication (SSRN) and the website entry (`spboucher.ai`). | ||||
| 88 | +- Equinox-2020 replication (not blocked, simply not executed). | ||||
| 89 | + | ||||
| 90 | +## Headline verdict (for the record) | ||||
| 91 | +H1 supported as association (all datasets). H2 **not supported** beyond baselines in either panel (pre-registered rule). | ||||
| 92 | +H3 supported in operating-cost sense (SIMULATED), refuted in total-cost sense at short horizons. H4 rejected in Seshat; | ||||
| 93 | +marginal in modern panel. H5: sign stable in every held-out block. Universal framing abandoned per pre-registration §7. | ||||
added
src/wp35/econometrics/leakage_check.py
+49 −0
@@ -0,0 +1,49 @@ | |||||
| 1 | +"""Leakage check for the Seshat out-of-sample comparison: the D principal-component loadings and the standardization of | ||||
| 2 | +P were estimated on the full polity sample (without outcomes) before cross-validation. Here both are re-estimated INSIDE | ||||
| 3 | +each leave-one-macro-region-out training fold (loadings from polities outside the held-out region; z-scores from | ||||
| 4 | +training crisis records) and the A2-vs-B1 comparison is repeated. Usage: python -m wp35.econometrics.leakage_check | ||||
| 5 | +""" | ||||
| 6 | +from __future__ import annotations | ||||
| 7 | +import numpy as np, pandas as pd, statsmodels.api as sm, statsmodels.formula.api as smf | ||||
| 8 | +from pathlib import Path | ||||
| 9 | +from wp35.econometrics.common import oos_loglik_binomial | ||||
| 10 | +from wp35.measurement.build_indices import pc1_pairwise | ||||
| 11 | + | ||||
| 12 | +ROOT = Path(__file__).resolve().parents[3]; PROC = ROOT / "data/processed"; RES = ROOT / "results/empirical" | ||||
| 13 | +BIN = sm.families.Binomial(); XS = "X_warfare + X_prod_infra + epidemic_v" | ||||
| 14 | + | ||||
| 15 | + | ||||
| 16 | +def main(): | ||||
| 17 | + ind = pd.read_parquet(PROC / "seshat_indices.parquet"); cr = pd.read_parquet(PROC / "seshat_h2_sample.parquet") | ||||
| 18 | + agg = pd.read_parquet(PROC / "seshat_precrisis_agg.parquet") | ||||
| 19 | + pw = agg.pivot(index="crisis_id", columns="variable", values="value_mean"); plog = agg.pivot(index="crisis_id", columns="variable", values="logv_mean") | ||||
| 20 | + Dc = pd.DataFrame({"log_pop": plog.get("polity_population"), "log_terr": plog.get("polity_territory"), "log_cap": plog.get("population_of_the_largest_settlement"), "settl_hier": pw.get("settlement_hierarchy")}, index=pw.index) | ||||
| 21 | + Dp = ind.set_index("polity_id")[["log_pop", "log_terr", "log_cap", "settl_hier"]] | ||||
| 22 | + rows = [] | ||||
| 23 | + for reg in sorted(cr.macro_region.dropna().unique()): | ||||
| 24 | + tr_pol = ind[ind.macro_region != reg] | ||||
| 25 | + _, load, _ = pc1_pairwise(Dp.loc[tr_pol.polity_id]) # loadings from training polities only | ||||
| 26 | + mu, sd = Dp.loc[tr_pol.polity_id].mean(), Dp.loc[tr_pol.polity_id].std(ddof=0) | ||||
| 27 | + Z = (Dc - mu) / sd; D_new = (Z.fillna(0) * load).sum(axis=1) / (Z.notna() * load.abs()).sum(axis=1).replace(0, np.nan) | ||||
| 28 | + D_new[Z.notna().sum(axis=1) < 2] = np.nan | ||||
| 29 | + d = cr.set_index("crisis_id").copy(); d["D_new"] = D_new.reindex(d.index) | ||||
| 30 | + tr, te = d[d.macro_region != reg], d[d.macro_region == reg] | ||||
| 31 | + zd = lambda s: (s - tr.D_new.mean()) / tr.D_new.std(); zk = lambda s: (s - tr.K_mean.mean()) / tr.K_mean.std() | ||||
| 32 | + for part in (tr, te): | ||||
| 33 | + part["P_new"] = zd(part.D_new) - zk(part.K_mean); part["D_new_z"] = zd(part.D_new) | ||||
| 34 | + need = ["y1_severity", "P_new", "D_new_z", "X_warfare", "X_prod_infra", "epidemic_v", "era"] | ||||
| 35 | + tr2, te2 = tr.dropna(subset=need), te.dropna(subset=need) | ||||
| 36 | + if len(te2) == 0: | ||||
| 37 | + continue | ||||
| 38 | + try: | ||||
| 39 | + mA = smf.glm(f"y1_severity ~ P_new + {XS} + C(era)", tr2, family=BIN).fit(); mB = smf.glm(f"y1_severity ~ D_new_z + {XS} + C(era)", tr2, family=BIN).fit() | ||||
| 40 | + pA, pB = mA.predict(te2), mB.predict(te2); ok = np.isfinite(pA) & np.isfinite(pB) | ||||
| 41 | + rows.append({"held_out": reg, "n_test": int(ok.sum()), "ll_A2_P": oos_loglik_binomial(te2.y1_severity[ok], pA[ok]), "ll_B1_D": oos_loglik_binomial(te2.y1_severity[ok], pB[ok])}) | ||||
| 42 | + except Exception as e: | ||||
| 43 | + rows.append({"held_out": reg, "error": str(e)[:60]}) | ||||
| 44 | + out = pd.DataFrame(rows); out["P_wins"] = out.ll_A2_P > out.ll_B1_D; out.to_csv(RES / "h2_seshat_cv_noleakage.csv", index=False) | ||||
| 45 | + print(out.round(4).to_string(index=False)); print("win share of P over D-only with fold-internal loadings/standardization:", round(float(out.P_wins.mean()), 3)) | ||||
| 46 | + | ||||
| 47 | + | ||||
| 48 | +if __name__ == "__main__": | ||||
| 49 | + main() | ||||
| 50 | |||||