1CLAUDE.md — WP35: The Social Compression Hypothesis2Mission3Conduct an autonomous, rigorous interdisciplinary research project entitled "The Social Compression Hypothesis: Toward a Universal Theory of Human Coordination, Institutions, and Civilizational Complexity." The project is a UQO working paper (WP35). Develop and critically test a mathematical theory of how societies coordinate at scale using institutions, shared standards, information infrastructure, and administrative capacity. Produce a complete English-language LaTeX research paper, reproducible datasets and code, figures, tables, and technical appendices.45Do not assume the hypothesis is true. Seek disconfirming evidence. The objective is a publishable contribution, not a predetermined positive finding. The human PI approves authorship, external publication, spending, and irreversible actions.6Absolute scientific rules7- Never fabricate downloads, data, observations, citations, experimental runs, estimates, significance, proofs, figures, or discoveries.8- Separate empirical observations, simulated outcomes, conjectures, and proven propositions in all outputs.9- Do not call historical institutional indices literal Shannon compression: direct information-theoretic measurement requires defined messages, distributions, and coding schemes.10- Verify citations, source versions, licensing, codebooks, coverage, and dates against primary sources. Record SHA-256 hashes, access dates, and transformations.11- Never treat missing data as zero. Propagate historical uncertainty and respect approximate dates.12- Avoid data leakage: fit preprocessing and measurement models on training folds only for predictive tests.13- Pre-register primary hypotheses, outcomes, and model specifications before examining outcome associations.14- Report null results, negative findings, alternative explanations, robustness failures, and limits of causal inference.15- Never submit papers, disclose credentials, incur expenses, or redistribute restricted data without authorization.16- Proceed phase by phase autonomously; maintain an execution log and verify each deliverable before proceeding.17Core questions and hypotheses18Question: Can the mismatch between societal coordination demand and institutional coordination capacity predict subsequent institutional instability and help explain the historical scaling of human societies?1920H1: Greater coordination capacity is associated with larger sustainable social scale, conditional on plausible confounders.2122H2: Higher lagged coordination pressure predicts subsequent instability beyond conventional baselines.2324H3: In controlled multiagent simulations, shared protocols and institutions can reduce communication cost per successfully coordinated task.2526H4: There may be nonlinear thresholds, but these must outperform smooth alternatives out of sample.2728H5: Results should generalize across held-out regions and periods, or the theory's domain must be narrowed.2930Competing explanations include military competition, environmental shocks, geography, resource scarcity, inequality, coercion, elite conflict, cultural diffusion, and selection effects.31Measurement framework32Define D as coordination demand (population, territory, dispersion, division of labor, organizational complexity), K as coordination capacity (administrative institutions, standard rules, communication infrastructure, recordkeeping), P as mismatch, and Y as an independently coded instability outcome.3334A candidate measure is:35\[36P_{it}=z(\log D_{it})-z(\log K_{it})37\]38Only use this if D and K are strictly positive and the scales and standardization are defensible. Compare transparent indices with latent-variable models and report measurement uncertainty and invariance across regions/epochs. Avoid overlapping indicators between predictor and outcome. Examine reverse causality and endogeneity. For simulations, directly measure messages, bytes, task completion, error rates, and overhead; where justified compute entropy or rate-distortion metrics with explicit probability models.39Data sources: obtain and audit real data401. Seshat Global History Databank — https://seshatdatabank.info/data/ . Historical polities, population, territory, settlement hierarchy, administration, bureaucracy, legal codes, writing, currency, roads, communication infrastructure. Inspect actual releases, codebooks, uncertain values, and temporal granularity.412. Maddison Project Database — https://www.rug.nl/ggdc/historicaldevelopment/maddison/ . Long-run economic outcomes and population. Do not naively map modern country boundaries onto ancient polities.423. D-PLACE / Ethnographic Atlas — https://d-place.org/ and https://github.com/D-PLACE/dplace-dataset-ea . Comparative social and political organization; account for cultural and spatial non-independence (Galton's problem).434. V-Dem — https://www.v-dem.net/data/the-v-dem-dataset/ . Modern institutions and governance; do not equate democracy with coordination capacity.445. Optional, if needed: World Bank WDI, Quality of Government, and UCDP conflict data, with separate provenance records.45Maintain a machine-readable source registry with dataset, release, URL, retrieval date, license, coverage, units, original variable names, checksums, and citation. Keep Seshat, D-PLACE, and modern country-year panels separate unless defensible, documented crosswalks exist. Audit missingness and uncertainty. Never invent coverage counts.46Project structure47wp35_social_compression/48 CLAUDE.md49 README.md50 pyproject.toml51 Makefile52 .env.example53 research/{preregistration,literature_review,framework,measurement,identification,decisions,limitations}.md54 data/{raw,interim,processed,metadata}/55 src/wp35/{acquisition,harmonization,measurement,econometrics,simulations,visualization,validation}/56 configs/57 tests/58 results/{empirical,simulated,diagnostics,logs}/59 paper/{main.tex,references.bib,figures/,tables/,appendix/}60 replication/61Use pinned Python dependencies, deterministic seeds, modular scripts, unit and integration tests, and one-command reproduction. Prefer pandas/polars, numpy, scipy, statsmodels, linearmodels, scikit-learn, and appropriate Bayesian tools. For simulation use a transparent agent-based framework with network modeling. Use LaTeX and latexmk when available.62Sequential autonomous phases63Phase 0 — Audit environment and feasibility64Inspect hardware, available tools, Python, LaTeX, connectivity, storage, and existing repository. Verify source downloads with small real samples. If OpenAI or Anthropic APIs are useful, check configured access with a minimal authorized test and record actual model IDs and limits; never assume access or spend money without approval. Produce results/logs/feasibility_report.md.65Phase 1 — Literature review and preregistration66Search and verify scholarship on transaction costs, state capacity, institutional economics, cultural evolution, collective action, information processing, and historical complexity. Evaluate relevant work by Coase, North, Ostrom, Hayek, Simon, Turchin, and others without presuming endorsement. Make a novelty matrix identifying precise contributions and existing similar hypotheses. Specify falsifiers, primary outcomes, horizons, models, exclusions, and competing explanations in research/preregistration.md before inspecting relationships with outcomes.67Phase 2 — Real data engineering68Download permitted source data; preserve raw files and hashes; implement validated parsers; build separate historical polity, ethnographic, and modern country panels. Explicitly handle uncertain historical intervals, duplicate entities, changing boundaries, missing values, and source provenance. Produce coverage tables, temporal distribution figures, missingness diagnostics, and a data dictionary. Use actual observed temporal resolution; never interpolate events and call them observations.69Phase 3 — Theory and measurement70Derive micro-to-macro propositions about coordination costs, standardization, institutions, scale, and resilience. Distinguish mathematical theorems (with proofs) from conjectures. Build D, K, and P using predeclared choices; compare alternative latent-variable approaches and weights. Quantify measurement error, cross-region comparability, and sensitivity to excluded indicators. Do not use the outcome to construct the predictor.71Phase 4 — Historical econometrics72Consider a pre-specified event-risk model:73\[74\Pr(Y_{i,t+h}=1)=\Lambda(\alpha+\beta P_{it}+\gamma'X_{it}+\mu_i+\tau_t).75\]76Use survival/event-history approaches if the historical event structure warrants them; handle censoring and uncertain event dates. Compare with population-only, development-only, institutional-only, and persistence baselines. Account for temporal and spatial dependence, clustering, small samples, multiple tests, reverse causality, and omitted variables. Use forward-chaining temporal splits and leave-region-out validation. Report effect sizes, intervals, calibration, discrimination, uncertainty, negative controls, placebo tests, and sensitivity to outcomes, periods, and missingness. Never claim causal effects without a credible identification strategy.77Phase 5 — Multiagent simulation experiments78Implement agents with heterogeneous information, resources, trust, limited attention, and interaction networks. Vary population, network structure, geography, communication bandwidth, and shocks. Compare: (a) no common institutions, (b) shared norms, (c) standardized records/protocols, (d) centralized administration, and (e) federated/modular coordination. Do not assume institutions emerge spontaneously if they are coded into the experiment.7980Predefine task success, welfare, inequality, coordination failures, latency, message count, bytes, and administrative overhead. Run factorial experiments with multiple seeds, uncertainty intervals, ablations, and adverse parameter regimes. Quantify tasks coordinated per bit or per unit cost, and compute Shannon quantities only where measurable. Search explicitly for cases where institutions reduce efficiency or create fragility. Keep all simulation findings labeled SIMULATED.81Phase 6 — Theory-to-data integration82Identify which theoretical mechanisms are observable in historical records and which are not. Compare the model against simpler rivals. Evaluate whether historical patterns are consistent with simulation-derived predictions, without treating agreement as proof of causal mechanism. State scope conditions and failed predictions prominently.83Phase 7 — Figures, tables, and reproducibility84Generate all tables and figures from executed scripts, including source coverage, missingness, variable definitions, uncertainty bands, historical trajectories, main estimates, robustness coefficients, out-of-sample calibration, simulation phase diagrams, scaling curves, and ablations. No invented numerical values. Every displayed number must trace to a machine-readable output and command. Run tests and reproduce the full pipeline from a clean environment where feasible.85Phase 8 — Write and compile paper86Write a substantial, professional English LaTeX manuscript containing: abstract; introduction; precise literature contribution; formal theory; data and historical measurement; econometric methods; results; agent-based design and results; synthesis; rival explanations; robustness; limitations; conclusion; verified references; technical appendices; and replication guide. Target academic clarity and rigor, not an arbitrary page count. Cite all nontrivial sourced facts. Do not write the abstract's findings until analyses are complete. Compile the PDF and inspect the output for broken references, missing figures, layout errors, and bibliography problems.87Phase 9 — Adversarial peer review88Create three independent critical reviews: (1) institutional economist, (2) historical anthropologist, (3) computational social scientist. Ask each to identify circular measurement, historical comparability problems, alternative mechanisms, simulation artifacts, data leakage, overclaiming, and statistical weaknesses. Implement justified corrections, rerun analyses, and provide a response-to-reviewers document. Do not manufacture positive reviewer endorsements.89Required outputs90- paper/main.tex and a compiled paper/main.pdf if the build environment permits.91- Verified paper/references.bib and literature novelty matrix.92- research/preregistration.md, research/measurement.md, research/identification.md, and research/limitations.md.93- Data acquisition scripts, source registry, dictionary, transformations, and hashes.94- Executable empirical analyses and simulation code with configurations and seeds.95- Real, reproducible figures and tables with provenance.96- Tests, logs, clean replication instructions, and adversarial review.97- results/logs/final_research_audit.md with an explicit checklist of completed, failed, and blocked tasks.98Completion criteria99A task is complete only when its code ran, its output exists, its source is traceable, and its limitations are documented. A research finding is supported only to the extent the evidence and design permit. If the data do not support the proposed universal theory, write the strongest defensible narrower paper and explain why. Scientific credibility takes priority over impressive claims.100101Start immediately with Phase 0, then Phase 1. Continue sequentially, test continuously, and maintain an auditable research trail.102