SPB Git
spboucher.ai

spb/wp35_uqo

Public

WP35 Phase 0: environment audit, source registry, raw acquisition (8 sources, SHA-256), feasibility report

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Nov 1, 2026) Browse files

28 changed files +991 −0

28 changed files
Unified Split
added .env.example +3 −0
@@ -0,0 +1,3 @@
1+# WP35 does not require any secret. Optional variables, left empty by default:
2+# ANTHROPIC_API_KEY= # not used; any LLM-assisted step requires PI approval and would be logged
3+# WP35_N_JOBS=8 # parallelism for simulation sweeps
added .gitignore +26 −0
@@ -0,0 +1,26 @@
1+# Raw downloads are preserved locally but NOT committed (hashes in data/metadata/source_registry.json)
2+data/raw/**
3+!data/raw/README.md
4+data/interim/**
5+!data/interim/.gitkeep
6+# Python
7+.venv/
8+__pycache__/
9+*.pyc
10+.pytest_cache/
11+*.egg-info/
12+# Large simulation traces
13+results/simulated/raw/**
14+# LaTeX build artifacts (paper/main.pdf IS committed)
15+paper/*.aux
16+paper/*.log
17+paper/*.out
18+paper/*.toc
19+paper/*.lof
20+paper/*.lot
21+paper/*.fls
22+paper/*.fdb_latexmk
23+paper/*.bbl
24+paper/*.blg
25+paper/*.synctex.gz
26+.DS_Store
added CLAUDE.md +14 −0
@@ -0,0 +1,14 @@
1+# WP35 — The Social Compression Hypothesis (project instructions)
2+
3+Mandate: `docs/MANDATE_VERBATIM.txt` (pasted by the PI 2026-11-01). Series conventions: `../../HARMONISATION.md`.
4+Working paper number **35**, folder `economie/social-compression-hypothesis`, future remote `gitsrv:srv/git/wp35_uqo.git`
5+(push only with PI authorisation).
6+
7+Rules that bind every edit in this repository:
8+1. No fabricated data, runs, citations or numbers. Every number in the paper traces to a file under `results/` produced by a
9+ script under `src/wp35/` or `scripts/`, with the command recorded in `results/logs/execution_log.md`.
10+2. Labels: EMPIRICAL (observed data), SIMULATED (agent-based), CONJECTURE, PROPOSITION (proved). Never mix.
11+3. `research/preregistration.md` is frozen (hash in `research/decisions.md`) before any D/K × outcome association is computed.
12+ Deviations are logged as numbered amendments in `research/decisions.md`.
13+4. Raw files in `data/raw/` are immutable (hashes in `data/metadata/source_registry.json`). Missing ≠ zero. Dates keep their ranges.
14+5. Python: `.venv/bin/python` (3.12, pinned in `pyproject.toml`). Tests: `.venv/bin/pytest`. Paper: `cd paper && make`.
added Makefile +24 −0
@@ -0,0 +1,24 @@
1+# One-command reproduction. Each target is idempotent; raw downloads are verified by hash, never re-fetched.
2+PY=.venv/bin/python
3+.PHONY: all env fetch data measure econ sim figures paper test clean
4+all: fetch data measure econ sim figures paper
5+env:
6+ uv venv -p 3.12 .venv && uv pip install -p .venv/bin/python -e .
7+fetch:
8+ $(PY) -m wp35.acquisition.fetch
9+data:
10+ $(PY) -m wp35.harmonization.build_panels
11+measure:
12+ $(PY) -m wp35.measurement.build_indices
13+econ:
14+ $(PY) -m wp35.econometrics.run_all
15+sim:
16+ $(PY) -m wp35.simulations.run_experiments
17+figures:
18+ $(PY) -m wp35.visualization.make_all
19+paper:
20+ cd paper && latexmk -pdf main.tex
21+test:
22+ .venv/bin/pytest -q
23+clean:
24+ cd paper && latexmk -C
added configs/sources.yaml +92 −0
@@ -0,0 +1,92 @@
1+# WP35 source registry (inputs). Retrieval dates, sizes and SHA-256 are written by
2+# src/wp35/acquisition/fetch.py into data/metadata/source_registry.json. Nothing here is a
3+# coverage count: coverage is computed from the parsed files (Phase 2).
4+sources:
5+ seshat_polaris2026:
6+ dataset: "Seshat Global History Databank — Polaris-2026 snapshot"
7+ publisher: "Seshat: Global History Databank (University of Oxford / Complexity Science Hub)"
8+ release: "Polaris-2026 (repository commit 55ca5fbc2b, 2026-02-25, 'final version')"
9+ url: "https://raw.githubusercontent.com/Seshat-Global-History-Databank/build_polaris_dataset/55ca5fbc2b/Polaris2026.xlsx"
10+ landing_page: "https://seshatdatabank.info/data/"
11+ license: "MIT (repository LICENSE, Copyright (c) 2025 Seshat Global History Databank); data citation requested"
12+ citation: "Seshat: Global History Databank, Polaris-2026 data release, https://seshatdatabank.info/data/ (see also Turchin et al. 2018 PNAS; Turchin et al. 2015 Cliodynamics)"
13+ filename: "Polaris2026.xlsx"
14+ units: "polity-time rows; variables coded 0/1 (absent/present) or numeric with uncertainty ranges; see sheets"
15+ role: "historical polity panel: coordination demand (population, territory, hierarchy), capacity (administration, writing, records, infrastructure), and outcomes"
16+ seshat_equinox2020:
17+ dataset: "Seshat Global History Databank — Equinox-2020 snapshot (spreadsheet copy stored in the Polaris repository)"
18+ publisher: "Seshat: Global History Databank"
19+ release: "Equinox-2020, spreadsheet dated 05.2023 (repository commit 55ca5fbc2b)"
20+ url: "https://raw.githubusercontent.com/Seshat-Global-History-Databank/build_polaris_dataset/55ca5fbc2b/Equinox2020.05.2023.xlsx"
21+ landing_page: "https://seshatdatabank.info/data/"
22+ license: "MIT (repository LICENSE); original Equinox release on Zenodo (record 6629022) is labelled AGPL-3.0"
23+ citation: "Seshat: Global History Databank, Equinox-2020 data release"
24+ filename: "Equinox2020.05.2023.xlsx"
25+ units: "polity-time rows"
26+ role: "robustness: alternative Seshat release"
27+ seshat_equinox_continuous_polities:
28+ dataset: "Seshat Databank: Equinox Release with Culturally and Institutionally Continuous Polities (Timescale SM data)"
29+ publisher: "Seshat / Zenodo record 8120128"
30+ release: "Zenodo 8120128 (2023-07-06)"
31+ url: "https://zenodo.org/api/records/8120128/files/Timescale.SM_Data.xlsx/content"
32+ landing_page: "https://zenodo.org/records/8120128"
33+ license: "CC-BY-4.0"
34+ citation: "Zenodo record 8120128 (Seshat Equinox release with continuous polities), DOI 10.5281/zenodo.8120128"
35+ filename: "Timescale.SM_Data.xlsx"
36+ units: "polity sequences with start/end dates"
37+ role: "polity continuity / sequence boundaries (candidate for event definitions)"
38+ maddison_2023_xlsx:
39+ dataset: "Maddison Project Database 2023"
40+ publisher: "Groningen Growth and Development Centre, University of Groningen"
41+ release: "MPD 2023 (Bolt & van Zanden 2024)"
42+ url: "https://dataverse.nl/api/access/datafile/421302"
43+ landing_page: "https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2023"
44+ license: "Free for academic use with citation (GGDC terms); check landing page"
45+ citation: "Bolt, J. and J. L. van Zanden (2024), 'Maddison-style estimates of the evolution of the world economy: A new 2023 update', Journal of Economic Surveys, 1-41."
46+ filename: "mpd2023_web.xlsx"
47+ units: "country-year; GDP per capita in 2011 international USD; population in thousands"
48+ role: "modern/long-run economic outcomes and population (country-year panel; NOT mapped onto ancient polities)"
49+ maddison_2023_dta:
50+ dataset: "Maddison Project Database 2023 (Stata file)"
51+ publisher: "GGDC"
52+ release: "MPD 2023"
53+ url: "https://dataverse.nl/api/access/datafile/421303"
54+ landing_page: "https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2023"
55+ license: "Free for academic use with citation"
56+ citation: "Bolt & van Zanden (2024)"
57+ filename: "mpd2023_web.dta"
58+ units: "country-year"
59+ role: "cross-check of the xlsx parse"
60+ dplace_ea_v3_3:
61+ dataset: "D-PLACE dataset derived from Murdock et al. 1999 'Ethnographic Atlas' (CLDF)"
62+ publisher: "D-PLACE (Kirby et al. 2016) / GitHub D-PLACE/dplace-dataset-ea"
63+ release: "v3.3 (2025-12-19)"
64+ url: "https://api.github.com/repos/D-PLACE/dplace-dataset-ea/zipball/v3.3"
65+ landing_page: "https://d-place.org/"
66+ license: "CC BY-NC 4.0"
67+ citation: "Kirby, K. R. et al. (2016). D-PLACE: A Global Database of Cultural, Linguistic and Environmental Diversity. PLoS ONE 11(7): e0158391. Murdock, G. P. et al. (1999) Ethnographic Atlas (World Cultures)."
68+ filename: "dplace-dataset-ea-v3.3.zip"
69+ units: "one row per society (~1,290 societies expected per documentation; actual count computed in Phase 2); coded categorical variables EA001-EA114 + environmental"
70+ role: "ethnographic cross-section: political integration (EA033), jurisdictional hierarchy, settlement size (EA031), class stratification; Galton's problem handled with language family / region"
71+ vdem_v16_rdata:
72+ dataset: "V-Dem Country-Year Dataset v16 (vdemdata R package copy)"
73+ publisher: "Varieties of Democracy Institute, University of Gothenburg (official GitHub package vdeminstitute/vdemdata)"
74+ release: "v16 (package commit f4dd26922e, 2026-03-17 'v16 release')"
75+ url: "https://raw.githubusercontent.com/vdeminstitute/vdemdata/f4dd26922e/data/vdem.RData"
76+ landing_page: "https://www.v-dem.net/data/the-v-dem-dataset/"
77+ license: "V-Dem data: free for non-commercial use with citation (V-Dem terms of use); package GPL-3"
78+ citation: "Coppedge, M. et al. (2026). V-Dem [Country-Year/Country-Date] Dataset v16. Varieties of Democracy (V-Dem) Project. https://doi.org/10.23696/vdemds26 (DOI to be verified in Phase 2 against the codebook)"
79+ filename: "vdem_v16.RData"
80+ units: "country-year, 1789-2025 (to be verified)"
81+ role: "modern institutional panel: state capacity proxies (rigorous and impartial public administration v2clrspct, legislature, civil service, census, etc.) and regime instability outcomes; democracy indices NOT used as capacity"
82+ vdem_v16_codebook:
83+ dataset: "V-Dem codebook (R object)"
84+ publisher: "V-Dem Institute"
85+ release: "v16"
86+ url: "https://raw.githubusercontent.com/vdeminstitute/vdemdata/f4dd26922e/data/codebook.RData"
87+ landing_page: "https://www.v-dem.net/data/the-v-dem-dataset/"
88+ license: "GPL-3 (package)"
89+ citation: "Coppedge et al. (2026) V-Dem Codebook v16"
90+ filename: "vdem_v16_codebook.RData"
91+ units: "variable metadata"
92+ role: "data dictionary for V-Dem"
added data/interim/.gitkeep +0 −0
added data/metadata/source_registry.json +178 −0
@@ -0,0 +1,178 @@
1+{
2+ "seshat_polaris2026": {
3+ "dataset": "Seshat Global History Databank — Polaris-2026 snapshot",
4+ "publisher": "Seshat: Global History Databank (University of Oxford / Complexity Science Hub)",
5+ "release": "Polaris-2026 (repository commit 55ca5fbc2b, 2026-02-25, 'final version')",
6+ "url": "https://raw.githubusercontent.com/Seshat-Global-History-Databank/build_polaris_dataset/55ca5fbc2b/Polaris2026.xlsx",
7+ "landing_page": "https://seshatdatabank.info/data/",
8+ "license": "MIT (repository LICENSE, Copyright (c) 2025 Seshat Global History Databank); data citation requested",
9+ "citation": "Seshat: Global History Databank, Polaris-2026 data release, https://seshatdatabank.info/data/ (see also Turchin et al. 2018 PNAS; Turchin et al. 2015 Cliodynamics)",
10+ "filename": "Polaris2026.xlsx",
11+ "units": "polity-time rows; variables coded 0/1 (absent/present) or numeric with uncertainty ranges; see sheets",
12+ "role": "historical polity panel: coordination demand (population, territory, hierarchy), capacity (administration, writing, records, infrastructure), and outcomes",
13+ "key": "seshat_polaris2026",
14+ "local_path": "data/raw/seshat_polaris2026/Polaris2026.xlsx",
15+ "http_status": 200,
16+ "final_url": "https://raw.githubusercontent.com/Seshat-Global-History-Databank/build_polaris_dataset/55ca5fbc2b/Polaris2026.xlsx",
17+ "content_type": "application/octet-stream",
18+ "retrieved_at_utc": "2026-11-01T07:48:51+00:00",
19+ "elapsed_s": 0.42,
20+ "size_bytes": 2899093,
21+ "sha256": "cf60c9f76eeda6db545521831a9201e65c64d2b74d4aeb55445cd2b564456c41",
22+ "status": "downloaded"
23+ },
24+ "seshat_equinox2020": {
25+ "dataset": "Seshat Global History Databank — Equinox-2020 snapshot (spreadsheet copy stored in the Polaris repository)",
26+ "publisher": "Seshat: Global History Databank",
27+ "release": "Equinox-2020, spreadsheet dated 05.2023 (repository commit 55ca5fbc2b)",
28+ "url": "https://raw.githubusercontent.com/Seshat-Global-History-Databank/build_polaris_dataset/55ca5fbc2b/Equinox2020.05.2023.xlsx",
29+ "landing_page": "https://seshatdatabank.info/data/",
30+ "license": "MIT (repository LICENSE); original Equinox release on Zenodo (record 6629022) is labelled AGPL-3.0",
31+ "citation": "Seshat: Global History Databank, Equinox-2020 data release",
32+ "filename": "Equinox2020.05.2023.xlsx",
33+ "units": "polity-time rows",
34+ "role": "robustness: alternative Seshat release",
35+ "key": "seshat_equinox2020",
36+ "local_path": "data/raw/seshat_equinox2020/Equinox2020.05.2023.xlsx",
37+ "http_status": 200,
38+ "final_url": "https://raw.githubusercontent.com/Seshat-Global-History-Databank/build_polaris_dataset/55ca5fbc2b/Equinox2020.05.2023.xlsx",
39+ "content_type": "application/octet-stream",
40+ "retrieved_at_utc": "2026-11-01T07:48:51+00:00",
41+ "elapsed_s": 0.32,
42+ "size_bytes": 1720038,
43+ "sha256": "7964d74c3ae3a24775323ef61a101ed3ca14ed0d3c47d606b3ce5e28e64968be",
44+ "status": "downloaded"
45+ },
46+ "seshat_equinox_continuous_polities": {
47+ "dataset": "Seshat Databank: Equinox Release with Culturally and Institutionally Continuous Polities (Timescale SM data)",
48+ "publisher": "Seshat / Zenodo record 8120128",
49+ "release": "Zenodo 8120128 (2023-07-06)",
50+ "url": "https://zenodo.org/api/records/8120128/files/Timescale.SM_Data.xlsx/content",
51+ "landing_page": "https://zenodo.org/records/8120128",
52+ "license": "CC-BY-4.0",
53+ "citation": "Zenodo record 8120128 (Seshat Equinox release with continuous polities), DOI 10.5281/zenodo.8120128",
54+ "filename": "Timescale.SM_Data.xlsx",
55+ "units": "polity sequences with start/end dates",
56+ "role": "polity continuity / sequence boundaries (candidate for event definitions)",
57+ "key": "seshat_equinox_continuous_polities",
58+ "local_path": "data/raw/seshat_equinox_continuous_polities/Timescale.SM_Data.xlsx",
59+ "http_status": 200,
60+ "final_url": "https://zenodo.org/api/records/8120128/files/Timescale.SM_Data.xlsx/content",
61+ "content_type": "application/octet-stream, application/octet-stream",
62+ "retrieved_at_utc": "2026-11-01T07:49:02+00:00",
63+ "elapsed_s": 0.77,
64+ "size_bytes": 70197,
65+ "sha256": "4eb6c1a72de9a96da6c8ffd877f421f41b268ab93b08db1f27b617d1d038b8bc",
66+ "status": "downloaded"
67+ },
68+ "maddison_2023_xlsx": {
69+ "dataset": "Maddison Project Database 2023",
70+ "publisher": "Groningen Growth and Development Centre, University of Groningen",
71+ "release": "MPD 2023 (Bolt & van Zanden 2024)",
72+ "url": "https://dataverse.nl/api/access/datafile/421302",
73+ "landing_page": "https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2023",
74+ "license": "Free for academic use with citation (GGDC terms); check landing page",
75+ "citation": "Bolt, J. and J. L. van Zanden (2024), 'Maddison-style estimates of the evolution of the world economy: A new 2023 update', Journal of Economic Surveys, 1-41.",
76+ "filename": "mpd2023_web.xlsx",
77+ "units": "country-year; GDP per capita in 2011 international USD; population in thousands",
78+ "role": "modern/long-run economic outcomes and population (country-year panel; NOT mapped onto ancient polities)",
79+ "key": "maddison_2023_xlsx",
80+ "local_path": "data/raw/maddison_2023_xlsx/mpd2023_web.xlsx",
81+ "http_status": 200,
82+ "final_url": "https://objectstore.surf.nl/store/10.34894/INZBF2/18f1947b695-c2971390baa0?response-content-disposition=attachment%3B%20filename%2A%3DUTF-8%27%27mpd2023_web.xlsx&response-content-type=application%2Fvnd.openxmlformats-officedocument.spreadsheetml.sheet&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20261101T074852Z&X-Amz-SignedHeaders=host&X-Amz-Credential=bf79a78ef3bb4505a9f98459f8a454ad%2F20261101%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Expires=3600&X-Amz-Signature=48c8642edc981174906fd2a3374b61b9e781e7abd672d73a5a4b1956a2785419",
83+ "content_type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
84+ "retrieved_at_utc": "2026-11-01T07:48:53+00:00",
85+ "elapsed_s": 1.64,
86+ "size_bytes": 4903804,
87+ "sha256": "ecc5916ca12789b983fc4be437f8a354bbf4291323605324ac3e0aea4c57cbb6",
88+ "status": "downloaded"
89+ },
90+ "maddison_2023_dta": {
91+ "dataset": "Maddison Project Database 2023 (Stata file)",
92+ "publisher": "GGDC",
93+ "release": "MPD 2023",
94+ "url": "https://dataverse.nl/api/access/datafile/421303",
95+ "landing_page": "https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2023",
96+ "license": "Free for academic use with citation",
97+ "citation": "Bolt & van Zanden (2024)",
98+ "filename": "mpd2023_web.dta",
99+ "units": "country-year",
100+ "role": "cross-check of the xlsx parse",
101+ "key": "maddison_2023_dta",
102+ "local_path": "data/raw/maddison_2023_dta/mpd2023_web.dta",
103+ "http_status": 200,
104+ "final_url": "https://objectstore.surf.nl/store/10.34894/INZBF2/18f1947b459-7fbdbbff0a80?response-content-disposition=attachment%3B%20filename%2A%3DUTF-8%27%27maddison2023_web.dta&response-content-type=application%2Fx-stata-14&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20261101T074854Z&X-Amz-SignedHeaders=host&X-Amz-Credential=bf79a78ef3bb4505a9f98459f8a454ad%2F20261101%2Fdataverse%2Fs3%2Faws4_request&X-Amz-Expires=3600&X-Amz-Signature=343a3db7b4ed4a49e9b82cd1801b2267f8975b938f281e12e511fe155367d84c",
105+ "content_type": "application/x-stata-14",
106+ "retrieved_at_utc": "2026-11-01T07:48:55+00:00",
107+ "elapsed_s": 1.79,
108+ "size_bytes": 10892389,
109+ "sha256": "2bc62ccbd795d4d330ef9ead6dc29c39e956922b43264640eba52d2c244841cb",
110+ "status": "downloaded"
111+ },
112+ "dplace_ea_v3_3": {
113+ "dataset": "D-PLACE dataset derived from Murdock et al. 1999 'Ethnographic Atlas' (CLDF)",
114+ "publisher": "D-PLACE (Kirby et al. 2016) / GitHub D-PLACE/dplace-dataset-ea",
115+ "release": "v3.3 (2025-12-19)",
116+ "url": "https://api.github.com/repos/D-PLACE/dplace-dataset-ea/zipball/v3.3",
117+ "landing_page": "https://d-place.org/",
118+ "license": "CC BY-NC 4.0",
119+ "citation": "Kirby, K. R. et al. (2016). D-PLACE: A Global Database of Cultural, Linguistic and Environmental Diversity. PLoS ONE 11(7): e0158391. Murdock, G. P. et al. (1999) Ethnographic Atlas (World Cultures).",
120+ "filename": "dplace-dataset-ea-v3.3.zip",
121+ "units": "one row per society (~1,290 societies expected per documentation; actual count computed in Phase 2); coded categorical variables EA001-EA114 + environmental",
122+ "role": "ethnographic cross-section: political integration (EA033), jurisdictional hierarchy, settlement size (EA031), class stratification; Galton's problem handled with language family / region",
123+ "key": "dplace_ea_v3_3",
124+ "local_path": "data/raw/dplace_ea_v3_3/dplace-dataset-ea-v3.3.zip",
125+ "http_status": 200,
126+ "final_url": "https://codeload.github.com/D-PLACE/dplace-dataset-ea/legacy.zip/refs/tags/v3.3",
127+ "content_type": "application/zip",
128+ "retrieved_at_utc": "2026-11-01T07:49:03+00:00",
129+ "elapsed_s": 1.36,
130+ "size_bytes": 7039773,
131+ "sha256": "9ec031e011019952f53c633e8421792e7ab88a2ba1fea93027b74272c3ee2dbb",
132+ "status": "downloaded"
133+ },
134+ "vdem_v16_rdata": {
135+ "dataset": "V-Dem Country-Year Dataset v16 (vdemdata R package copy)",
136+ "publisher": "Varieties of Democracy Institute, University of Gothenburg (official GitHub package vdeminstitute/vdemdata)",
137+ "release": "v16 (package commit f4dd26922e, 2026-03-17 'v16 release')",
138+ "url": "https://raw.githubusercontent.com/vdeminstitute/vdemdata/f4dd26922e/data/vdem.RData",
139+ "landing_page": "https://www.v-dem.net/data/the-v-dem-dataset/",
140+ "license": "V-Dem data: free for non-commercial use with citation (V-Dem terms of use); package GPL-3",
141+ "citation": "Coppedge, M. et al. (2026). V-Dem [Country-Year/Country-Date] Dataset v16. Varieties of Democracy (V-Dem) Project. https://doi.org/10.23696/vdemds26 (DOI to be verified in Phase 2 against the codebook)",
142+ "filename": "vdem_v16.RData",
143+ "units": "country-year, 1789-2025 (to be verified)",
144+ "role": "modern institutional panel: state capacity proxies (rigorous and impartial public administration v2clrspct, legislature, civil service, census, etc.) and regime instability outcomes; democracy indices NOT used as capacity",
145+ "key": "vdem_v16_rdata",
146+ "local_path": "data/raw/vdem_v16_rdata/vdem_v16.RData",
147+ "http_status": 200,
148+ "final_url": "https://raw.githubusercontent.com/vdeminstitute/vdemdata/f4dd26922e/data/vdem.RData",
149+ "content_type": "application/octet-stream",
150+ "retrieved_at_utc": "2026-11-01T07:48:56+00:00",
151+ "elapsed_s": 0.72,
152+ "size_bytes": 33878255,
153+ "sha256": "39b412d39a061c18f20c98e4ad4d6355b05a0441df31be4ee9aec420dc3d95ea",
154+ "status": "downloaded"
155+ },
156+ "vdem_v16_codebook": {
157+ "dataset": "V-Dem codebook (R object)",
158+ "publisher": "V-Dem Institute",
159+ "release": "v16",
160+ "url": "https://raw.githubusercontent.com/vdeminstitute/vdemdata/f4dd26922e/data/codebook.RData",
161+ "landing_page": "https://www.v-dem.net/data/the-v-dem-dataset/",
162+ "license": "GPL-3 (package)",
163+ "citation": "Coppedge et al. (2026) V-Dem Codebook v16",
164+ "filename": "vdem_v16_codebook.RData",
165+ "units": "variable metadata",
166+ "role": "data dictionary for V-Dem",
167+ "key": "vdem_v16_codebook",
168+ "local_path": "data/raw/vdem_v16_codebook/vdem_v16_codebook.RData",
169+ "http_status": 200,
170+ "final_url": "https://raw.githubusercontent.com/vdeminstitute/vdemdata/f4dd26922e/data/codebook.RData",
171+ "content_type": "application/octet-stream",
172+ "retrieved_at_utc": "2026-11-01T07:48:56+00:00",
173+ "elapsed_s": 0.12,
174+ "size_bytes": 148633,
175+ "sha256": "2840d879cb9071b1f742f5f18c5373518a586ac90454b58dbf5f4309077f0700",
176+ "status": "downloaded"
177+ }
178+}
\ No newline at end of file
added docs/MANDATE.md +48 −0
@@ -0,0 +1,48 @@
1+# WP35 — Mandate (verbatim, pasted by the PI on 2026-11-01)
2+
3+CLAUDE.md — WP35: The Social Compression Hypothesis
4+Mission
5+Conduct an autonomous, rigorous interdisciplinary research project entitled "The Social Compression Hypothesis: Toward a Universal Theory of Human Coordination, Institutions, and Civilizational Complexity." The project is a UQO working paper (WP35). Develop and critically test a mathematical theory of how societies coordinate at scale using institutions, shared standards, information infrastructure, and administrative capacity. Produce a complete English-language LaTeX research paper, reproducible datasets and code, figures, tables, and technical appendices.
6+
7+Do not assume the hypothesis is true. Seek disconfirming evidence. The objective is a publishable contribution, not a predetermined positive finding. The human PI approves authorship, external publication, spending, and irreversible actions.
8+Absolute scientific rules
9+- Never fabricate downloads, data, observations, citations, experimental runs, estimates, significance, proofs, figures, or discoveries.
10+- Separate empirical observations, simulated outcomes, conjectures, and proven propositions in all outputs.
11+- Do not call historical institutional indices literal Shannon compression: direct information-theoretic measurement requires defined messages, distributions, and coding schemes.
12+- Verify citations, source versions, licensing, codebooks, coverage, and dates against primary sources. Record SHA-256 hashes, access dates, and transformations.
13+- Never treat missing data as zero. Propagate historical uncertainty and respect approximate dates.
14+- Avoid data leakage: fit preprocessing and measurement models on training folds only for predictive tests.
15+- Pre-register primary hypotheses, outcomes, and model specifications before examining outcome associations.
16+- Report null results, negative findings, alternative explanations, robustness failures, and limits of causal inference.
17+- Never submit papers, disclose credentials, incur expenses, or redistribute restricted data without authorization.
18+- Proceed phase by phase autonomously; maintain an execution log and verify each deliverable before proceeding.
19+Core questions and hypotheses
20+Question: Can the mismatch between societal coordination demand and institutional coordination capacity predict subsequent institutional instability and help explain the historical scaling of human societies?
21+
22+H1: Greater coordination capacity is associated with larger sustainable social scale, conditional on plausible confounders.
23+H2: Higher lagged coordination pressure predicts subsequent instability beyond conventional baselines.
24+H3: In controlled multiagent simulations, shared protocols and institutions can reduce communication cost per successfully coordinated task.
25+H4: There may be nonlinear thresholds, but these must outperform smooth alternatives out of sample.
26+H5: Results should generalize across held-out regions and periods, or the theory's domain must be narrowed.
27+
28+Competing explanations include military competition, environmental shocks, geography, resource scarcity, inequality, coercion, elite conflict, cultural diffusion, and selection effects.
29+Measurement framework
30+Define D as coordination demand (population, territory, dispersion, division of labor, organizational complexity), K as coordination capacity (administrative institutions, standard rules, communication infrastructure, recordkeeping), P as mismatch, and Y as an independently coded instability outcome.
31+
32+A candidate measure is: P_it = z(log D_it) - z(log K_it)
33+Only use this if D and K are strictly positive and the scales and standardization are defensible. Compare transparent indices with latent-variable models and report measurement uncertainty and invariance across regions/epochs. Avoid overlapping indicators between predictor and outcome. Examine reverse causality and endogeneity. For simulations, directly measure messages, bytes, task completion, error rates, and overhead; where justified compute entropy or rate-distortion metrics with explicit probability models.
34+Data sources: obtain and audit real data
35+1. Seshat Global History Databank — https://seshatdatabank.info/data/ .
36+2. Maddison Project Database — https://www.rug.nl/ggdc/historicaldevelopment/maddison/ .
37+3. D-PLACE / Ethnographic Atlas — https://d-place.org/ and https://github.com/D-PLACE/dplace-dataset-ea .
38+4. V-Dem — https://www.v-dem.net/data/the-v-dem-dataset/ .
39+5. Optional, if needed: World Bank WDI, Quality of Government, and UCDP conflict data, with separate provenance records.
40+Maintain a machine-readable source registry with dataset, release, URL, retrieval date, license, coverage, units, original variable names, checksums, and citation. Keep Seshat, D-PLACE, and modern country-year panels separate unless defensible, documented crosswalks exist. Audit missingness and uncertainty. Never invent coverage counts.
41+Project structure
42+wp35_social_compression/ CLAUDE.md README.md pyproject.toml Makefile .env.example research/{preregistration,literature_review,framework,measurement,identification,decisions,limitations}.md data/{raw,interim,processed,metadata}/ src/wp35/{acquisition,harmonization,measurement,econometrics,simulations,visualization,validation}/ configs/ tests/ results/{empirical,simulated,diagnostics,logs}/ paper/{main.tex,references.bib,figures/,tables/,appendix/} replication/
43+Use pinned Python dependencies, deterministic seeds, modular scripts, unit and integration tests, and one-command reproduction. Prefer pandas/polars, numpy, scipy, statsmodels, linearmodels, scikit-learn, and appropriate Bayesian tools. For simulation use a transparent agent-based framework with network modeling. Use LaTeX and latexmk when available.
44+Sequential autonomous phases
45+Phase 0 — Audit environment and feasibility. Phase 1 — Literature review and preregistration. Phase 2 — Real data engineering. Phase 3 — Theory and measurement. Phase 4 — Historical econometrics (pre-specified event-risk model Pr(Y_{i,t+h}=1)=Λ(α+βP_it+γ'X_it+μ_i+τ_t); survival approaches; baselines; forward-chaining and leave-region-out validation). Phase 5 — Multiagent simulation experiments (five institutional regimes a–e; predefine metrics; factorial, seeds, ablations; SIMULATED label). Phase 6 — Theory-to-data integration. Phase 7 — Figures, tables, reproducibility. Phase 8 — Write and compile paper. Phase 9 — Adversarial peer review (institutional economist, historical anthropologist, computational social scientist) + response.
46+Required outputs: paper/main.tex + main.pdf; references.bib + novelty matrix; research/preregistration.md, measurement.md, identification.md, limitations.md; acquisition scripts, source registry, dictionary, hashes; executable analyses + simulation code with configs and seeds; reproducible figures/tables with provenance; tests, logs, replication instructions, adversarial review; results/logs/final_research_audit.md with checklist.
47+Completion criteria: a task is complete only when its code ran, its output exists, its source is traceable, and its limitations are documented. If the data do not support the proposed universal theory, write the strongest defensible narrower paper and explain why.
48+(full verbatim text of the paste is also archived at docs/MANDATE_VERBATIM.txt)
added docs/MANDATE_VERBATIM.txt +101 −0
@@ -0,0 +1,101 @@
1+CLAUDE.md — WP35: The Social Compression Hypothesis
2+Mission
3+Conduct an autonomous, rigorous interdisciplinary research project entitled "The Social Compression Hypothesis: Toward a Universal Theory of Human Coordination, Institutions, and Civilizational Complexity." The project is a UQO working paper (WP35). Develop and critically test a mathematical theory of how societies coordinate at scale using institutions, shared standards, information infrastructure, and administrative capacity. Produce a complete English-language LaTeX research paper, reproducible datasets and code, figures, tables, and technical appendices.
4+
5+Do not assume the hypothesis is true. Seek disconfirming evidence. The objective is a publishable contribution, not a predetermined positive finding. The human PI approves authorship, external publication, spending, and irreversible actions.
6+Absolute scientific rules
7+- Never fabricate downloads, data, observations, citations, experimental runs, estimates, significance, proofs, figures, or discoveries.
8+- Separate empirical observations, simulated outcomes, conjectures, and proven propositions in all outputs.
9+- Do not call historical institutional indices literal Shannon compression: direct information-theoretic measurement requires defined messages, distributions, and coding schemes.
10+- Verify citations, source versions, licensing, codebooks, coverage, and dates against primary sources. Record SHA-256 hashes, access dates, and transformations.
11+- Never treat missing data as zero. Propagate historical uncertainty and respect approximate dates.
12+- Avoid data leakage: fit preprocessing and measurement models on training folds only for predictive tests.
13+- Pre-register primary hypotheses, outcomes, and model specifications before examining outcome associations.
14+- Report null results, negative findings, alternative explanations, robustness failures, and limits of causal inference.
15+- Never submit papers, disclose credentials, incur expenses, or redistribute restricted data without authorization.
16+- Proceed phase by phase autonomously; maintain an execution log and verify each deliverable before proceeding.
17+Core questions and hypotheses
18+Question: Can the mismatch between societal coordination demand and institutional coordination capacity predict subsequent institutional instability and help explain the historical scaling of human societies?
19+
20+H1: Greater coordination capacity is associated with larger sustainable social scale, conditional on plausible confounders.
21+
22+H2: Higher lagged coordination pressure predicts subsequent instability beyond conventional baselines.
23+
24+H3: In controlled multiagent simulations, shared protocols and institutions can reduce communication cost per successfully coordinated task.
25+
26+H4: There may be nonlinear thresholds, but these must outperform smooth alternatives out of sample.
27+
28+H5: Results should generalize across held-out regions and periods, or the theory's domain must be narrowed.
29+
30+Competing explanations include military competition, environmental shocks, geography, resource scarcity, inequality, coercion, elite conflict, cultural diffusion, and selection effects.
31+Measurement framework
32+Define D as coordination demand (population, territory, dispersion, division of labor, organizational complexity), K as coordination capacity (administrative institutions, standard rules, communication infrastructure, recordkeeping), P as mismatch, and Y as an independently coded instability outcome.
33+
34+A candidate measure is:
35+\[
36+P_{it}=z(\log D_{it})-z(\log K_{it})
37+\]
38+Only use this if D and K are strictly positive and the scales and standardization are defensible. Compare transparent indices with latent-variable models and report measurement uncertainty and invariance across regions/epochs. Avoid overlapping indicators between predictor and outcome. Examine reverse causality and endogeneity. For simulations, directly measure messages, bytes, task completion, error rates, and overhead; where justified compute entropy or rate-distortion metrics with explicit probability models.
39+Data sources: obtain and audit real data
40+1. Seshat Global History Databank — https://seshatdatabank.info/data/ . Historical polities, population, territory, settlement hierarchy, administration, bureaucracy, legal codes, writing, currency, roads, communication infrastructure. Inspect actual releases, codebooks, uncertain values, and temporal granularity.
41+2. Maddison Project Database — https://www.rug.nl/ggdc/historicaldevelopment/maddison/ . Long-run economic outcomes and population. Do not naively map modern country boundaries onto ancient polities.
42+3. D-PLACE / Ethnographic Atlas — https://d-place.org/ and https://github.com/D-PLACE/dplace-dataset-ea . Comparative social and political organization; account for cultural and spatial non-independence (Galton's problem).
43+4. V-Dem — https://www.v-dem.net/data/the-v-dem-dataset/ . Modern institutions and governance; do not equate democracy with coordination capacity.
44+5. Optional, if needed: World Bank WDI, Quality of Government, and UCDP conflict data, with separate provenance records.
45+Maintain a machine-readable source registry with dataset, release, URL, retrieval date, license, coverage, units, original variable names, checksums, and citation. Keep Seshat, D-PLACE, and modern country-year panels separate unless defensible, documented crosswalks exist. Audit missingness and uncertainty. Never invent coverage counts.
46+Project structure
47+wp35_social_compression/
48+ CLAUDE.md
49+ README.md
50+ pyproject.toml
51+ Makefile
52+ .env.example
53+ research/{preregistration,literature_review,framework,measurement,identification,decisions,limitations}.md
54+ data/{raw,interim,processed,metadata}/
55+ src/wp35/{acquisition,harmonization,measurement,econometrics,simulations,visualization,validation}/
56+ configs/
57+ tests/
58+ results/{empirical,simulated,diagnostics,logs}/
59+ paper/{main.tex,references.bib,figures/,tables/,appendix/}
60+ replication/
61+Use pinned Python dependencies, deterministic seeds, modular scripts, unit and integration tests, and one-command reproduction. Prefer pandas/polars, numpy, scipy, statsmodels, linearmodels, scikit-learn, and appropriate Bayesian tools. For simulation use a transparent agent-based framework with network modeling. Use LaTeX and latexmk when available.
62+Sequential autonomous phases
63+Phase 0 — Audit environment and feasibility
64+Inspect hardware, available tools, Python, LaTeX, connectivity, storage, and existing repository. Verify source downloads with small real samples. If OpenAI or Anthropic APIs are useful, check configured access with a minimal authorized test and record actual model IDs and limits; never assume access or spend money without approval. Produce results/logs/feasibility_report.md.
65+Phase 1 — Literature review and preregistration
66+Search and verify scholarship on transaction costs, state capacity, institutional economics, cultural evolution, collective action, information processing, and historical complexity. Evaluate relevant work by Coase, North, Ostrom, Hayek, Simon, Turchin, and others without presuming endorsement. Make a novelty matrix identifying precise contributions and existing similar hypotheses. Specify falsifiers, primary outcomes, horizons, models, exclusions, and competing explanations in research/preregistration.md before inspecting relationships with outcomes.
67+Phase 2 — Real data engineering
68+Download permitted source data; preserve raw files and hashes; implement validated parsers; build separate historical polity, ethnographic, and modern country panels. Explicitly handle uncertain historical intervals, duplicate entities, changing boundaries, missing values, and source provenance. Produce coverage tables, temporal distribution figures, missingness diagnostics, and a data dictionary. Use actual observed temporal resolution; never interpolate events and call them observations.
69+Phase 3 — Theory and measurement
70+Derive micro-to-macro propositions about coordination costs, standardization, institutions, scale, and resilience. Distinguish mathematical theorems (with proofs) from conjectures. Build D, K, and P using predeclared choices; compare alternative latent-variable approaches and weights. Quantify measurement error, cross-region comparability, and sensitivity to excluded indicators. Do not use the outcome to construct the predictor.
71+Phase 4 — Historical econometrics
72+Consider a pre-specified event-risk model:
73+\[
74+\Pr(Y_{i,t+h}=1)=\Lambda(\alpha+\beta P_{it}+\gamma'X_{it}+\mu_i+\tau_t).
75+\]
76+Use survival/event-history approaches if the historical event structure warrants them; handle censoring and uncertain event dates. Compare with population-only, development-only, institutional-only, and persistence baselines. Account for temporal and spatial dependence, clustering, small samples, multiple tests, reverse causality, and omitted variables. Use forward-chaining temporal splits and leave-region-out validation. Report effect sizes, intervals, calibration, discrimination, uncertainty, negative controls, placebo tests, and sensitivity to outcomes, periods, and missingness. Never claim causal effects without a credible identification strategy.
77+Phase 5 — Multiagent simulation experiments
78+Implement agents with heterogeneous information, resources, trust, limited attention, and interaction networks. Vary population, network structure, geography, communication bandwidth, and shocks. Compare: (a) no common institutions, (b) shared norms, (c) standardized records/protocols, (d) centralized administration, and (e) federated/modular coordination. Do not assume institutions emerge spontaneously if they are coded into the experiment.
79+
80+Predefine task success, welfare, inequality, coordination failures, latency, message count, bytes, and administrative overhead. Run factorial experiments with multiple seeds, uncertainty intervals, ablations, and adverse parameter regimes. Quantify tasks coordinated per bit or per unit cost, and compute Shannon quantities only where measurable. Search explicitly for cases where institutions reduce efficiency or create fragility. Keep all simulation findings labeled SIMULATED.
81+Phase 6 — Theory-to-data integration
82+Identify which theoretical mechanisms are observable in historical records and which are not. Compare the model against simpler rivals. Evaluate whether historical patterns are consistent with simulation-derived predictions, without treating agreement as proof of causal mechanism. State scope conditions and failed predictions prominently.
83+Phase 7 — Figures, tables, and reproducibility
84+Generate all tables and figures from executed scripts, including source coverage, missingness, variable definitions, uncertainty bands, historical trajectories, main estimates, robustness coefficients, out-of-sample calibration, simulation phase diagrams, scaling curves, and ablations. No invented numerical values. Every displayed number must trace to a machine-readable output and command. Run tests and reproduce the full pipeline from a clean environment where feasible.
85+Phase 8 — Write and compile paper
86+Write a substantial, professional English LaTeX manuscript containing: abstract; introduction; precise literature contribution; formal theory; data and historical measurement; econometric methods; results; agent-based design and results; synthesis; rival explanations; robustness; limitations; conclusion; verified references; technical appendices; and replication guide. Target academic clarity and rigor, not an arbitrary page count. Cite all nontrivial sourced facts. Do not write the abstract's findings until analyses are complete. Compile the PDF and inspect the output for broken references, missing figures, layout errors, and bibliography problems.
87+Phase 9 — Adversarial peer review
88+Create three independent critical reviews: (1) institutional economist, (2) historical anthropologist, (3) computational social scientist. Ask each to identify circular measurement, historical comparability problems, alternative mechanisms, simulation artifacts, data leakage, overclaiming, and statistical weaknesses. Implement justified corrections, rerun analyses, and provide a response-to-reviewers document. Do not manufacture positive reviewer endorsements.
89+Required outputs
90+- paper/main.tex and a compiled paper/main.pdf if the build environment permits.
91+- Verified paper/references.bib and literature novelty matrix.
92+- research/preregistration.md, research/measurement.md, research/identification.md, and research/limitations.md.
93+- Data acquisition scripts, source registry, dictionary, transformations, and hashes.
94+- Executable empirical analyses and simulation code with configurations and seeds.
95+- Real, reproducible figures and tables with provenance.
96+- Tests, logs, clean replication instructions, and adversarial review.
97+- results/logs/final_research_audit.md with an explicit checklist of completed, failed, and blocked tasks.
98+Completion criteria
99+A task is complete only when its code ran, its output exists, its source is traceable, and its limitations are documented. A research finding is supported only to the extent the evidence and design permit. If the data do not support the proposed universal theory, write the strongest defensible narrower paper and explain why. Scientific credibility takes priority over impressive claims.
100+
101+Start immediately with Phase 0, then Phase 1. Continue sequentially, test continuously, and maintain an auditable research trail.
added paper/.latexmkrc +5 −0
@@ -0,0 +1,5 @@
1+$pdf_mode = 1;
2+$pdflatex = 'pdflatex -interaction=nonstopmode -halt-on-error -synctex=1 %O %S';
3+$bibtex_use = 2;
4+$clean_ext = 'synctex.gz run.xml bbl bcf fdb_latexmk fls log aux out toc lof lot blg';
5+@default_files = ('main.tex');
added paper/Makefile +11 −0
@@ -0,0 +1,11 @@
1+.PHONY: all clean distclean
2+
3+all:
4+ latexmk
5+
6+clean:
7+ latexmk -c
8+
9+distclean:
10+ latexmk -C
11+ rm -f *.synctex.gz *.run.xml *.bbl *.bcf
added paper/main.tex +143 −0
@@ -0,0 +1,143 @@
1+% ============================================================================
2+% UQO Working Paper Template — Economics / Finance
3+% ============================================================================
4+\documentclass[12pt,letterpaper]{article}
5+
6+% --- Encoding & Language ---
7+\usepackage[utf8]{inputenc}
8+\usepackage[T1]{fontenc}
9+\usepackage[english]{babel}
10+
11+% --- Page Layout ---
12+\usepackage[
13+ letterpaper,
14+ top=1in,
15+ bottom=1in,
16+ left=1in,
17+ right=1in
18+]{geometry}
19+
20+% --- Typography ---
21+\usepackage{mathptmx} % Times-like font (text + math)
22+\usepackage{microtype} % Improved typographic refinements
23+\usepackage{setspace}
24+\onehalfspacing % 1.5 line spacing
25+
26+% --- Math ---
27+\usepackage{amsmath,amssymb,amsthm}
28+
29+% --- Tables ---
30+\usepackage{booktabs}
31+\usepackage{tabularx}
32+\usepackage{multirow}
33+\usepackage{threeparttable}
34+\usepackage{siunitx}
35+\sisetup{
36+ input-symbols = {()},
37+ table-format = -1.3,
38+ parse-numbers = false
39+}
40+
41+% --- Figures ---
42+\usepackage{graphicx}
43+\usepackage{subcaption}
44+\usepackage{float}
45+
46+% --- Colors & Links ---
47+\usepackage[dvipsnames]{xcolor}
48+\usepackage[
49+ colorlinks = true,
50+ linkcolor = NavyBlue,
51+ citecolor = NavyBlue,
52+ urlcolor = NavyBlue,
53+ breaklinks = true
54+]{hyperref}
55+
56+% --- Bibliography ---
57+\usepackage[authoryear,round,semicolon]{natbib}
58+\bibliographystyle{aer}
59+
60+% --- Captions ---
61+\usepackage[
62+ font = small,
63+ labelfont = bf,
64+ labelsep = period,
65+ justification = justified,
66+ singlelinecheck = false
67+]{caption}
68+
69+% --- Headers & Footers ---
70+\usepackage{fancyhdr}
71+\pagestyle{fancy}
72+\fancyhf{}
73+\renewcommand{\headrulewidth}{0pt}
74+\fancyfoot[C]{\thepage}
75+
76+% --- Section Formatting ---
77+\usepackage{titlesec}
78+\titleformat{\section}{\large\bfseries}{\thesection.}{0.5em}{}
79+\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection.}{0.5em}{}
80+\titleformat{\subsubsection}{\normalsize\itshape}{\thesubsubsection.}{0.5em}{}
81+
82+% --- Appendix Support ---
83+\usepackage[toc,page]{appendix}
84+
85+% --- Misc ---
86+\usepackage{enumitem}
87+\setlist{nosep,leftmargin=*}
88+
89+% ============================================================================
90+% METADATA — Edit these fields for each working paper
91+% ============================================================================
92+\newcommand{\WPnumber}{XX}
93+\newcommand{\WPtitle}{Your Working Paper Title Here}
94+\newcommand{\WPsubtitle}{} % Leave empty if no subtitle
95+\newcommand{\WPdate}{\today}
96+\newcommand{\WPversion}{1.0}
97+\newcommand{\WPabstract}{%
98+ This paper examines \ldots{}
99+ We find that \ldots{}
100+ Our results suggest \ldots{}
101+ These findings contribute to the literature on \ldots{}
102+}
103+\newcommand{\WPkeywords}{Keyword 1, Keyword 2, Keyword 3, Keyword 4}
104+\newcommand{\WPjel}{G12, G14, C58}
105+
106+% --- Author ---
107+\newcommand{\WPauthor}{Simon-Pierre Boucher}
108+\newcommand{\WPaffiliation}{%
109+ Département des sciences administratives\\
110+ Université du Québec en Outaouais%
111+}
112+\newcommand{\WPemail}{simon-pierre.boucher@uqo.ca}
113+\newcommand{\WPaddress}{%
114+ Gatineau -- Pavillon Alexandre-Taché\\
115+ 283, boulevard Alexandre-Taché\\
116+ Gatineau, Québec, Canada J9A 1L8%
117+}
118+
119+% ============================================================================
120+% DOCUMENT
121+% ============================================================================
122+\begin{document}
123+
124+% --- Title Page ---
125+\input{sections/titlepage}
126+
127+% --- Main Body ---
128+\input{sections/introduction}
129+\input{sections/literature}
130+\input{sections/data}
131+\input{sections/methodology}
132+\input{sections/results}
133+\input{sections/robustness}
134+\input{sections/conclusion}
135+
136+% --- References ---
137+\newpage
138+\bibliography{references}
139+
140+% --- Appendix ---
141+\input{appendix/appendix}
142+
143+\end{document}
added paper/sections/titlepage.tex +77 −0
@@ -0,0 +1,77 @@
1+% ============================================================================
2+% Title Page
3+% ============================================================================
4+\thispagestyle{empty}
5+
6+\begin{center}
7+
8+% --- Logo ---
9+\includegraphics[width=3.5cm]{uq_logo.jpg}
10+
11+\vspace{0.6cm}
12+
13+{\footnotesize\textsc{Université du Québec en Outaouais}}\\[0.15cm]
14+{\footnotesize\textsc{Département des sciences administratives}}
15+
16+\vspace{0.8cm}
17+
18+{\footnotesize\textsc{Working Paper No.~\WPnumber}}
19+
20+\vspace{1.2cm}
21+
22+% --- Title ---
23+{\LARGE\bfseries \WPtitle\par}
24+
25+\ifx\WPsubtitle\empty\else
26+ \vspace{0.3cm}
27+ {\large\itshape \WPsubtitle\par}
28+\fi
29+
30+\vspace{1.2cm}
31+
32+% --- Author ---
33+{\large \WPauthor}\\[0.3cm]
34+{\normalsize \WPaffiliation}\\[0.15cm]
35+{\normalsize \href{mailto:\WPemail}{\WPemail}}\\[0.15cm]
36+{\small \WPaddress}
37+
38+\vspace{0.8cm}
39+
40+% --- Date & Version ---
41+{\normalsize \WPdate}\\[0.1cm]
42+{\small Version~\WPversion}
43+
44+\end{center}
45+
46+\vfill
47+
48+\newpage
49+
50+% ============================================================================
51+% Abstract Page
52+% ============================================================================
53+\thispagestyle{empty}
54+
55+\vspace*{1cm}
56+
57+\noindent\rule{\textwidth}{0.4pt}
58+\vspace{0.3cm}
59+
60+\noindent\textbf{Abstract}
61+
62+\vspace{0.15cm}
63+
64+\noindent\WPabstract
65+
66+\vspace{0.4cm}
67+
68+\noindent\textbf{Keywords:} \WPkeywords
69+
70+\vspace{0.15cm}
71+
72+\noindent\textbf{JEL Classification:} \WPjel
73+
74+\vspace{0.3cm}
75+\noindent\rule{\textwidth}{0.4pt}
76+
77+\newpage
added paper/sections/titlepage_anon.tex +53 −0
@@ -0,0 +1,53 @@
1+% ============================================================================
2+% ANONYMOUS TITLE PAGE — for double-anonymized journal submission
3+% Used by paper/main_anon.tex (same macros as main.tex, no author block,
4+% no institution, no logo). Abstract/keywords/JEL page identical to titlepage.tex.
5+% ============================================================================
6+\thispagestyle{empty}
7+\begin{center}
8+
9+\vspace*{3cm}
10+
11+{\LARGE\bfseries \WPtitle\par}
12+
13+\ifx\WPsubtitle\empty\else
14+ \vspace{0.3cm}
15+ {\large\itshape \WPsubtitle\par}
16+\fi
17+
18+\vspace{1.5cm}
19+
20+{\normalsize [Author identity withheld for review]}
21+
22+\vspace{0.8cm}
23+
24+{\normalsize \WPdate}
25+
26+\end{center}
27+
28+\vfill
29+
30+\newpage
31+\thispagestyle{empty}
32+\vspace*{1cm}
33+
34+\noindent\rule{\textwidth}{0.4pt}
35+\vspace{0.3cm}
36+
37+\noindent\textbf{Abstract}
38+\vspace{0.15cm}
39+
40+\noindent\WPabstract
41+
42+\vspace{0.4cm}
43+
44+\noindent\textbf{Keywords:} \WPkeywords
45+
46+\vspace{0.15cm}
47+
48+\noindent\textbf{JEL Classification:} \WPjel
49+
50+\vspace{0.3cm}
51+\noindent\rule{\textwidth}{0.4pt}
52+
53+\newpage
added paper/uq_logo.jpg +0 −0

Binary file not shown.

added pyproject.toml +30 −0
@@ -0,0 +1,30 @@
1+[project]
2+name = "wp35"
3+version = "0.1.0"
4+description = "WP35 — The Social Compression Hypothesis: coordination demand, coordination capacity and institutional instability"
5+requires-python = ">=3.12,<3.13"
6+dependencies = [
7+ "numpy==2.3.4",
8+ "pandas==2.3.3",
9+ "polars==1.36.1",
10+ "scipy==1.16.2",
11+ "statsmodels==0.14.5",
12+ "linearmodels==6.1",
13+ "scikit-learn==1.7.2",
14+ "lifelines==0.30.0",
15+ "networkx==3.5",
16+ "matplotlib==3.10.7",
17+ "openpyxl==3.1.5",
18+ "pyreadr==0.5.3",
19+ "pyyaml==6.0.3",
20+ "requests==2.32.5",
21+ "pyarrow==21.0.0",
22+ "pytest==8.4.2",
23+ "tqdm==4.67.1",
24+ "pymc==5.26.1",
25+ "arviz==0.22.0",
26+]
27+[tool.setuptools.packages.find]
28+where = ["src"]
29+[tool.pytest.ini_options]
30+testpaths = ["tests"]
added results/logs/execution_log.md +12 −0
@@ -0,0 +1,12 @@
1+# WP35 execution log (append-only; times local, America/Toronto)
2+
3+| When | Phase | Action | Result / evidence |
4+|---|---|---|---|
5+| 2026-11-01 02:40 | 0 | Read series conventions (`HARMONISATION.md`, template, README of WP33/WP29) | WP35 = next free number; category `economie/` created |
6+| 2026-11-01 02:44 | 0 | Hardware/tool audit (`sysctl`, `df`, `which`, `latexmk --version`) | M5 Max 18 c / 48 GiB / 1.0 TiB free; TeX Live latexmk 4.88; uv 0.11.7; R 4.6.1 |
7+| 2026-11-01 02:45 | 0 | Connectivity probes (curl, 8 URLs) | all HTTP 200 |
8+| 2026-11-01 02:46 | 0 | Release discovery: Seshat data page → Polaris-2026 GitHub; Zenodo API; GitHub API (D-PLACE tags, vdemdata) ; Maddison page → dataverse 421302/421303 | releases recorded in `configs/sources.yaml` |
9+| 2026-11-01 02:47 | 0 | Project skeleton + venv (`uv venv -p 3.12`, `uv pip install -e .`) | import check OK |
10+| 2026-11-01 02:48 | 0 | `python -m wp35.acquisition.fetch` | 6/8 downloaded; Zenodo 406 and GitHub 415 caused by `Accept: application/octet-stream` → fixed to `*/*`, re-run: 8/8 ; registry `data/metadata/source_registry.json` |
11+| 2026-11-01 02:50 | 0 | Zenodo MD5 check (57a4f436…) ; sheet/column inspection of all files | see `results/logs/feasibility_report.md` §2 |
12+| 2026-11-01 02:55 | 0 | Feasibility report written | `results/logs/feasibility_report.md` |
added results/logs/feasibility_report.md +67 −0
@@ -0,0 +1,67 @@
1+# WP35 — Phase 0 feasibility report
2+
3+Date: 2026-11-01 (UTC-4). Executor: Claude Code (claude-fable-5-1) acting autonomously under the PI's mandate
4+(`docs/MANDATE_VERBATIM.txt`). Nothing in this report is inferred: every value below was read from a command output
5+recorded in `results/logs/execution_log.md`.
6+
7+## 1. Environment
8+
9+| Item | Observed |
10+|---|---|
11+| Machine | Apple M5 Max laptop, 18 cores, 48 GiB RAM, 1.0 TiB free on the data volume (df) |
12+| Cluster | MacLustr (29 Macs on LAN) reachable through `mld`/SSH; not required for Phases 0-4, may be used for simulation sweeps (Phase 5) |
13+| Python | `uv` 0.11.7 manages a project venv at `.venv` (CPython 3.12.13). System python3 is 3.14.4 (not used) |
14+| Pinned deps (pyproject.toml) | numpy 2.3.4, pandas 2.3.3, polars 1.36.1, scipy 1.16.2, statsmodels 0.14.5, linearmodels 6.1, scikit-learn 1.7.2, lifelines 0.30.0, networkx 3.5, matplotlib 3.10.7, pymc 5.26.1, arviz 0.22.0, pyreadr 0.5.3, openpyxl 3.1.5, pyarrow 21.0.0, pytest 8.4.2 — import check passed |
15+| LaTeX | TeX Live: latexmk 4.88 (9 March 2026), pdflatex, lualatex, bibtex at /Library/TeX/texbin; UQO template (`_modele-working-paper`) copied into `paper/` |
16+| Other tools | git, make, R 4.6.1 (no readxl), pandoc, jq, curl, wget, shasum |
17+| Connectivity | HTTP 200 on seshatdatabank.info, rug.nl (Maddison), d-place.org, github.com, v-dem.net, zenodo.org, api.github.com |
18+| LLM APIs | No `ANTHROPIC_*` / `OPENAI_*` variables in the shell environment; the `anthropic` MCP server failed to connect this session. **No paid API calls are planned or needed**: the research is executed by the agent itself; literature search uses public web search / Crossref. Any future API use requires PI approval. |
19+| Existing repository | None for WP35. Series conventions read from `Recherche-UQO/HARMONISATION.md` (numbering: WP34 exists, so this is WP35; new category folder `economie/`). |
20+
21+## 2. Data sources: actual releases retrieved (raw files preserved, SHA-256 in `data/metadata/source_registry.json`)
22+
23+| Key | Release actually obtained | Bytes | Licence (as read from source) | Notes from first inspection |
24+|---|---|---|---|---|
25+| seshat_polaris2026 | Polaris-2026 snapshot, GitHub `Seshat-Global-History-Databank/build_polaris_dataset` commit 55ca5fbc2b (2026-02-25) | 2,899,093 | MIT (repo LICENSE) | 8 sheets: Variables (173), Polities (626 rows; 42 `seshat_region`; start_year -13600…1921), General (7,239 rows, 516 polities, 21 vars), Social complexity (26,033 rows, 619 polities, 77 vars; `value_from/value_to`, `year_from/year_to`, `is_disputed`, `is_uncertain`), Warfare (17,536 rows, 376 polities, 49 vars), Luxury goods (1,191 rows), Religion (5,466 rows), **CrisisDB – Crisis consequences (166 rows, 150 polities; 23 consequence columns coded P/A/IP/IA/SU)** |
26+| seshat_equinox2020 | Equinox-2020 spreadsheet (05.2023) from the same repo | 1,720,038 | MIT (repo) / AGPL-3 on Zenodo 6629022 | single long-format sheet (NGA, Polity, Section, Subsection, Variable, Value.From/To, Date.From/To, Fact.Type, notes) |
27+| seshat_equinox_continuous_polities | Zenodo 8120128 `Timescale.SM_Data.xlsx`; MD5 verified = 57a4f436f8c7e62c80c4a49a67c6fe86 | 70,197 | CC-BY-4.0 | 1,396 rows: NGA, PolID, AbsTime, RelTime, SPC1, Culture.Sequence, Institutions.Sequence |
28+| maddison_2023_xlsx / _dta | MPD 2023 (dataverse.nl datafiles 421302/421303) | 4,903,804 / 10,892,389 | academic use with citation | `Full data` sheet: countrycode, country, region, year, gdppc, pop |
29+| dplace_ea_v3_3 | D-PLACE Ethnographic Atlas CLDF v3.3 (2025-12-19), zip of commit 5aa46ee | 7,039,773 | CC BY-NC 4.0 | cldf/{societies,variables,codes,data}.csv + sources.bib; raw/ folder with original tables |
30+| vdem_v16_rdata / _codebook | V-Dem v16 from the V-Dem Institute's official `vdemdata` R package (commit f4dd26922e, 2026-03-17) | 33,878,255 / 148,633 | V-Dem terms (non-commercial, cite); package GPL-3 | `vdem` data frame 28,092 country-years × 4,618 columns, years 1789–2025, 202 countries; codebook 781 rows |
31+
32+Sampling check: every file opened and parsed with pandas/openpyxl/pyreadr/zipfile (see execution log). No coverage
33+number above is a guess; each is a row count printed by the parser on the raw file.
34+
35+## 3. Feasibility assessment per phase
36+
37+- **Historical panel (Seshat)** — feasible. D-candidates: `polity_population`, `polity_territory`,
38+ `population_of_the_largest_settlement`, `settlement_hierarchy`, `administrative_level`, `military_level`,
39+ `religious_level`. K-candidates: `full_time_bureaucrat`, `examination_system`, `merit_promotion`,
40+ `specialized_government_building`, `formal_legal_code`, `judge`, `court`, `professional_lawyer`, writing/records
41+ (`written_record`, `script`, `lists_tables_and_classification`, `calendar`, `practical_literature`…), communication
42+ (`courier`, `postal_station`, `general_postal_service`, `road`, `bridge`, `canal`, `port`), money (`token`, `precious_metal`,
43+ `foreign_coin`, `indigenous_coin`, `paper_currency`), standards (`*_measurement_system`). Temporal resolution is
44+ *per fact* (year_from/year_to), so the panel must be built on observed dates, not interpolated.
45+ Outcome Y: CrisisDB crisis consequences (independent coding: decline, collapse, civil_war, uprising, fragmentation,
46+ conquest, …) covers 150 polities only → **the H2 event-risk test will have a small-N outcome**; polity end/transition
47+ (`polity_succeeding_entity`, end_year, relationship to preceding entity) is a second, larger but less independent outcome.
48+- **Ethnographic cross-section (D-PLACE EA)** — feasible for H1-type scale/organization associations with
49+ Galton controls (language family, region). No time dimension.
50+- **Modern panel (V-Dem + Maddison)** — feasible as a separate, modern replication panel (country-year); the
51+ mandate forbids mapping it onto ancient polities and forbids using democracy indices as capacity.
52+- **Simulations** — feasible locally (networkx + numpy); sweeps can be parallelised on the cluster if needed.
53+- **Paper** — latexmk + UQO template available.
54+
55+## 4. Risks identified before pre-registration
56+
57+1. Small outcome N in CrisisDB (150 polities, with 42 regions) → low power; pre-register a minimum-detectable-effect
58+ statement and treat the test as confirmatory only for the primary outcome.
59+2. Overlap between D and K indicators and Seshat's own aggregate complexity characteristics (Turchin et al. 2018 PC1):
60+ D and K will be built from disjoint indicator sets fixed in the pre-registration.
61+3. Reverse causality: crisis coding may inform the coding of institutions in the same period → use lagged predictors
62+ with observed dates and exclude facts dated inside the crisis window.
63+4. Uncertainty: `value_from/value_to`, `year_from/year_to`, `is_uncertain`, `is_disputed` must be carried through
64+ (interval midpoints + bounds as sensitivity).
65+5. Licence: D-PLACE is BY-NC (academic use OK, no redistribution of raw data); V-Dem/Seshat raw files not redistributed in the repo.
66+
67+Decision: proceed to Phase 1 (literature + pre-registration) **without** computing any D/K-outcome association.
added src/wp35/__init__.py +0 −0
added src/wp35/acquisition/__init__.py +0 −0
added src/wp35/acquisition/fetch.py +94 −0
@@ -0,0 +1,94 @@
1+"""Download every source listed in configs/sources.yaml into data/raw/<key>/ and record
2+SHA-256, size, HTTP status, retrieval timestamp in data/metadata/source_registry.json.
3+
4+Usage: python -m wp35.acquisition.fetch [--only KEY ...] [--force]
5+Raw files are immutable: a re-run verifies the hash of an existing file and never overwrites
6+it unless --force is given.
7+"""
8+from __future__ import annotations
9+import argparse, hashlib, json, sys, time
10+from datetime import datetime, timezone
11+from pathlib import Path
12+import requests, yaml
13+
14+ROOT = Path(__file__).resolve().parents[3]
15+RAW = ROOT / "data" / "raw"
16+META = ROOT / "data" / "metadata"
17+REGISTRY = META / "source_registry.json"
18+UA = "WP35-UQO-research-acquisition/0.1 (simon-pierre.boucher@uqo.ca)"
19+
20+
21+def sha256(path: Path) -> str:
22+ h = hashlib.sha256()
23+ with open(path, "rb") as f:
24+ for chunk in iter(lambda: f.read(1 << 20), b""):
25+ h.update(chunk)
26+ return h.hexdigest()
27+
28+
29+def load_registry() -> dict:
30+ return json.loads(REGISTRY.read_text()) if REGISTRY.exists() else {}
31+
32+
33+def fetch(key: str, spec: dict, force: bool, registry: dict) -> dict:
34+ dest_dir = RAW / key
35+ dest_dir.mkdir(parents=True, exist_ok=True)
36+ dest = dest_dir / spec["filename"]
37+ rec = dict(spec)
38+ rec["key"] = key
39+ rec["local_path"] = str(dest.relative_to(ROOT))
40+ if dest.exists() and not force:
41+ rec["status"] = "already_present"
42+ rec["sha256"] = sha256(dest)
43+ rec["size_bytes"] = dest.stat().st_size
44+ prev = registry.get(key, {})
45+ rec["retrieved_at_utc"] = prev.get("retrieved_at_utc")
46+ rec["http_status"] = prev.get("http_status")
47+ if prev.get("sha256") and prev["sha256"] != rec["sha256"]:
48+ rec["status"] = "HASH_MISMATCH_WITH_REGISTRY"
49+ return rec
50+ t0 = time.time()
51+ r = requests.get(spec["url"], headers={"User-Agent": UA, "Accept": "*/*"},
52+ timeout=300, stream=True, allow_redirects=True)
53+ rec["http_status"] = r.status_code
54+ rec["final_url"] = r.url
55+ rec["content_type"] = r.headers.get("Content-Type")
56+ if r.status_code != 200:
57+ rec["status"] = f"download_failed_http_{r.status_code}"
58+ return rec
59+ tmp = dest.with_suffix(dest.suffix + ".part")
60+ with open(tmp, "wb") as f:
61+ for chunk in r.iter_content(1 << 20):
62+ f.write(chunk)
63+ tmp.rename(dest)
64+ rec["retrieved_at_utc"] = datetime.now(timezone.utc).isoformat(timespec="seconds")
65+ rec["elapsed_s"] = round(time.time() - t0, 2)
66+ rec["size_bytes"] = dest.stat().st_size
67+ rec["sha256"] = sha256(dest)
68+ rec["status"] = "downloaded"
69+ return rec
70+
71+
72+def main(argv=None):
73+ ap = argparse.ArgumentParser()
74+ ap.add_argument("--only", nargs="*")
75+ ap.add_argument("--force", action="store_true")
76+ a = ap.parse_args(argv)
77+ specs = yaml.safe_load((ROOT / "configs" / "sources.yaml").read_text())["sources"]
78+ META.mkdir(parents=True, exist_ok=True)
79+ registry = load_registry()
80+ keys = a.only or list(specs)
81+ for k in keys:
82+ rec = fetch(k, specs[k], a.force, registry)
83+ registry[k] = rec
84+ print(f"{k:40s} {rec['status']:22s} {rec.get('size_bytes','?'):>10} B sha256={str(rec.get('sha256',''))[:16]}…")
85+ REGISTRY.write_text(json.dumps(registry, indent=2, ensure_ascii=False))
86+ bad = [k for k in keys if not registry[k]["status"] in ("downloaded", "already_present")]
87+ if bad:
88+ print("FAILED:", bad, file=sys.stderr)
89+ return 1
90+ return 0
91+
92+
93+if __name__ == "__main__":
94+ sys.exit(main())
added src/wp35/econometrics/__init__.py +0 −0
added src/wp35/harmonization/__init__.py +0 −0
added src/wp35/measurement/__init__.py +0 −0
added src/wp35/simulations/__init__.py +0 −0
added src/wp35/validation/__init__.py +0 −0
added src/wp35/visualization/__init__.py +0 −0
added tests/test_registry.py +13 −0
@@ -0,0 +1,13 @@
1+import hashlib, json
2+from pathlib import Path
3+ROOT = Path(__file__).resolve().parents[1]
4+
5+def test_registry_hashes_match_raw_files():
6+ reg = json.loads((ROOT / "data/metadata/source_registry.json").read_text())
7+ assert len(reg) >= 8
8+ for k, rec in reg.items():
9+ p = ROOT / rec["local_path"]
10+ if not p.exists(): # raw files are not committed; skip on a fresh clone
11+ continue
12+ h = hashlib.sha256(p.read_bytes()).hexdigest()
13+ assert h == rec["sha256"], k
14
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere