WP35 Phase 3: formal framework (5 theorems, 3 conjectures), D/K/P indices with IRT alternative, measurement diagnostics, a-priori power

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Simon-Pierre Boucher committed 2 days ago (Nov 1, 2026) parent 8621449 Browse files

16 changed files +1,139 −0

added data/processed/modern_indices.parquet +0 −0

Binary file not shown.

added data/processed/seshat_crisis_indices.parquet +0 −0

Binary file not shown.

added data/processed/seshat_indices.parquet +0 −0

Binary file not shown.

added paper/figures/fig_irt_invariance.pdf +0 −0

Binary file not shown.

added paper/figures/fig_measurement_DK.pdf +0 −0

Binary file not shown.

modified research/bib_overrides.csv +2 −0

				`@@ -47,3 +47,5 @@ marzen2017\|10.1098/rsif.2017.0166\|Marzen & DeDeo 2017 J R Soc Interface — adde`
47	flack2012\|10.1098/rstb.2011.0214\|Flack 2012 Phil Trans B — added after novelty search	47	flack2012\|10.1098/rstb.2011.0214\|Flack 2012 Phil Trans B — added after novelty search
48	hoyer2024\|10.31235/osf.io/rk4gd\|Hoyer et al. 2024 SocArXiv preprint 'All crises are unhappy in their own way' — no journal version found 2026-11-01	48	hoyer2024\|10.31235/osf.io/rk4gd\|Hoyer et al. 2024 SocArXiv preprint 'All crises are unhappy in their own way' — no journal version found 2026-11-01
49	wolpert2017\|10.1017/9781316584200.010\|Wolpert 2017 'The many faces of state space compression' (same volume)	49	wolpert2017\|10.1017/9781316584200.010\|Wolpert 2017 'The many faces of state space compression' (same volume)
		50	+lindley1952\|10.1017/s0305004100027638\|Lindley 1952 Math. Proc. Cambridge Phil. Soc. 48 — verified by Crossref (used in framework.md Theorem 5)
		51	+loynes1962\|10.1017/s0305004100036781\|Loynes 1962 Math. Proc. Cambridge Phil. Soc. 58 — verified by Crossref

added research/framework.md +264 −0

++# WP35 — Formal framework: coordination demand, coordination capacity, and institutions as codes
++
++Status labels used throughout: **DEFINITION** (a convention), **THEOREM / PROPOSITION** (a statement proved below
++or a standard result with the proof reproduced), **CONJECTURE** (a statement we believe is worth testing, with no
++proof; what would falsify it is stated). Nothing in this file is an empirical finding. The results concern an
++idealized communication system; §8 states what is *not* claimed and which pre-registered test (if any) can speak to
++each statement. Citations use the keys of `research/reference_candidates.csv`.
++
++---
++
++## 1. Primitives and definitions
++
++**DEFINITION 1.1 (agents, tasks, plans).** A society has $N$ agents. In each period, coordination tasks arrive at rate
++$\lambda$ per agent. A task requires that $k \ge 2$ participants settle on a common element of a finite *plan set*
++$S$, $|S| = s$, the required plan being drawn independently across tasks from a distribution $p$ on $S$ with Shannon
++entropy $H(p) = -\sum_{x \in S} p(x)\log_2 p(x) \le \log_2 s$ (shannon1948; cover2006). Communication is counted in
++*messages* (one transmission between two agents) and in *bits* (the length of what is transmitted). Verification and
++record-keeping use a second channel whose cost is counted separately as *administrative overhead* (DEFINITION 2.1).
++
++**DEFINITION 1.2 (coordination demand).** The coordination demand of a period is the task volume that must be
++settled: $D = N \lambda k$ participant-slots per period. In the worst case in which every pair of agents must be
++reconciled, $D$ is bounded by the number of pairs, $N(N-1)/2$, i.e. $D = \Theta(N^2)$; under bounded $\lambda$ and
++$k$, $D = \Theta(N)$. Either way $D$ is increasing in $N$.
++
++**DEFINITION 1.3 (coordination capacity).** Fix a tolerated error rate $\varepsilon \in (0,1)$. The coordination
++capacity $K$ of an institutional system is the maximal number of tasks it can settle per period such that the
++fraction of tasks settled on a wrong or inconsistent plan is at most $\varepsilon$. $K$ depends on the available
++communication budget (bits per period), the code in use (§2), the routing structure (§5) and the overhead (§3).
++
++**DEFINITION 1.4 (utilization).** $\rho = D/K$. When $D$ and $K$ are measured in the same units (tasks per period),
++$\log D - \log K = \log \rho$ exactly.
++
++**Limitation (not a theorem).** The pre-registered empirical index $P = z(\log D) - z(K)$ uses standardized proxies
++(population, territory, settlement hierarchy for $D$; shares of administrative, legal/monetary, communication and
++record-keeping indicators for $K$). It is a monotone proxy for $\log\rho$ only under CONJECTURE C1 (§7); it is not a
++measurement of $\rho$, and the Seshat/V-Dem indicators are not Shannon quantities.
++
++---
++
++## 2. Theorem 1 — coding gain of a shared code
++
++**DEFINITION 2.1 (code).** A *shared code* is an injective map $c: S \to \{0,1\}^*$ known to all participants,
++with lengths $\ell(x) = |c(x)|$. It is *prefix-free* if no codeword is a prefix of another. Institutions in the sense of
++this paper (standardized names, units, legal forms, accounting categories, calendars) are shared codes in exactly this
++sense (arrow1974; cremer2007).
++
++**Assumption 2.2.** Settling a task means that the initiating participant identifies the required plan $x \in S$ to the
++other $k-1$ participants without ambiguity; messages to different participants are counted separately in messages
++but identically in bits per recipient, so we state the result per recipient.
++
++**THEOREM 1 (coding gain).**
++(i) Without a shared code, identifying an arbitrary element of $S$ without error requires at least
++$\lceil \log_2 s \rceil$ bits per recipient in the worst case.
++(ii) With a shared prefix-free code that is optimal for $p$, the expected length per recipient
++$L_p^\* = \min_{c\ \mathrm{prefix\text{-}free}} \sum_x p(x)\ell(x)$ satisfies
++$$H(p) \le L_p^\* < H(p) + 1 .$$
++(iii) Hence the expected saving per recipient, $G(p) = \lceil\log_2 s\rceil - L_p^\*$, obeys
++$G(p) > \lceil\log_2 s\rceil - H(p) - 1 \ge 0$ whenever $\lceil\log_2 s\rceil \ge H(p)+1$, and $G(p)$ is strictly
++positive as soon as $p$ is sufficiently concentrated ($H(p) < \lceil\log_2 s\rceil - 1$).
++
++*Proof.* (i) Without a shared code there is no prior agreement on which binary string denotes which plan, so the
++only error-free strategy is a fixed-length enumeration of $S$ agreed by the trivial convention of listing; any scheme
++that distinguishes $s$ alternatives with binary strings of a fixed length $m$ needs $2^m \ge s$, i.e.
++$m \ge \lceil\log_2 s\rceil$. (The point is not that $\log_2 s$ is a lower bound for *all* conceivable codes — it is the
++cost of the *codeless* baseline, in which the message must be self-describing.)
++
++(ii) *Lower bound.* For any prefix-free code the Kraft inequality holds: $\sum_x 2^{-\ell(x)} \le 1$ (cover2006,
++Thm 5.2.1). Put $q(x) = 2^{-\ell(x)}/Z$ with $Z = \sum_x 2^{-\ell(x)} \le 1$. Then
++$$\sum_x p(x)\ell(x) - H(p) = \sum_x p(x)\log_2\frac{p(x)}{2^{-\ell(x)}} = \sum_x p(x)\log_2\frac{p(x)}{q(x)} - \log_2 Z
++= D_{KL}(p\|q) - \log_2 Z \ge 0,$$
++because $D_{KL} \ge 0$ (Gibbs' inequality) and $-\log_2 Z \ge 0$. *Upper bound.* Choose
++$\ell(x) = \lceil -\log_2 p(x)\rceil$. Then $\sum_x 2^{-\ell(x)} \le \sum_x p(x) = 1$, so by the converse Kraft
++theorem (cover2006, Thm 5.2.1) a prefix-free code with these lengths exists, and
++$\sum_x p(x)\ell(x) < \sum_x p(x)(-\log_2 p(x) + 1) = H(p) + 1$. The optimal code does at least as well.
++
++(iii) Subtract (ii) from (i). $\square$
++
++**Remark 2.3.** Theorem 1 is the source-coding theorem of shannon1948 applied to the plan set. Its content for the
++paper is only this: *a shared code converts a worst-case identification cost into an average-case cost that is lower
++exactly to the extent that tasks are predictable.* The "compression" in the title of the hypothesis refers to this
++step and to nothing else.
++
++---
++
++## 3. Theorem 2 — break-even scale of an institution
++
++**DEFINITION 3.1 (institutional cost structure).** Adopting a code costs a fixed set-up $F \ge 0$ (designing and
++promulgating the standard), a learning cost $\ell \ge 0$ per agent (each of the $N$ agents must learn the code), and an
++administrative overhead $a \ge 0$ bits per task (verification, record-keeping, enforcement). It saves $G > 0$ bits per
++task (Theorem 1, per recipient; multiply by $k-1$ if desired — the argument is unchanged).
++
++**THEOREM 2 (break-even).** Over a horizon with $T$ tasks, the institution lowers total communication cost if and only
++if
++$$T\,(G - a) > F + N\ell .$$
++With $T = N\lambda\tau$ over $\tau$ periods, this is equivalent to
++$$\lambda\tau > \frac{F/N + \ell}{G - a}\quad\text{when } G > a,$$
++and the inequality can never hold when $a \ge G$.
++
++*Proof.* Total cost without the institution is $C_0 = T\lceil\log_2 s\rceil$. With it, $C_1 = F + N\ell + T(L_p^\* + a)$.
++$C_1 < C_0 \iff F + N\ell < T(\lceil\log_2 s\rceil - L_p^\* - a) = T(G - a)$. If $a \ge G$ the right-hand side is
++$\le 0 < F + N\ell$ unless $F = \ell = 0$ and $a = G$, in which case the costs are equal, so strict improvement is
++impossible. If $G > a$, substitute $T = N\lambda\tau$ and divide both sides by $N(G-a) > 0$. $\square$
++
++**Corollary 3.2 (scale advantage).** The threshold $(F/N + \ell)/(G-a)$ is decreasing in $N$: holding $\lambda,\tau$
++fixed, a larger society crosses the break-even point sooner, because the set-up cost is spread over more agents. It is
++*not* decreasing in $N$ through $\ell$: per-agent learning costs do not benefit from scale. This is the formal version
++of the observation that institutions "pay off at scale" (johnson1982; garicano2000; bolton1994), and it also makes
++precise when they do not: when overhead $a$ is close to the coding gain $G$, or when horizons $\tau$ are short.
++
++---
++
++## 4. Theorem 3 — mismatched code penalty (institutional rigidity)
++
++**THEOREM 3 (cost of a stale code).** Let the code be optimal for $p$ in the sense of using lengths
++$\ell_p(x) = \lceil -\log_2 p(x) \rceil$, and let the true task distribution be $q$ with $q \ll p$ (every plan with
++positive $q$-probability has positive $p$-probability). Then the expected length under $q$ satisfies
++$$H(q) + D_{KL}(q\|p) \le \mathbb{E}_q[\ell_p(X)] < H(q) + D_{KL}(q\|p) + 1 .$$
++The excess cost of using the $p$-code when the world is $q$, relative to the $q$-optimal code, lies in
++$(D_{KL}(q\|p) - 1,\ D_{KL}(q\|p) + 1)$ bits per task.
++
++*Proof.* Since $-\log_2 p(x) \le \ell_p(x) < -\log_2 p(x) + 1$,
++$$\mathbb{E}_q[\ell_p(X)] \ge \sum_x q(x)\left(-\log_2 p(x)\right) = \sum_x q(x)\log_2\frac{q(x)}{p(x)} + \sum_x q(x)\left(-\log_2 q(x)\right) = D_{KL}(q\|p) + H(q),$$
++and the same computation with the $+1$ gives the strict upper bound. The $q$-optimal code has expected length in
++$[H(q), H(q)+1)$ by Theorem 1(ii), which yields the stated interval for the excess. (This is the "wrong code" result of
++cover2006, Thm 5.4.3.) $\square$
++
++**Corollary 4.1 (fragility).** A shock that moves the task distribution from $p$ to $q$ raises the per-task
++communication cost of an existing institution by about $D_{KL}(q\|p)$ bits. Combined with Theorem 2, the institution
++remains worthwhile after the shock only if $G(p) - D_{KL}(q\|p) > a$ (up to the $\pm 1$ rounding). An institution can
++therefore *lower* efficiency after a shock, and the more specialized the code (the smaller $H(p)$ relative to
++$\log_2 s$), the larger the divergence a given shock can produce. This is the precise sense in which standardization
++trades efficiency for rigidity; it echoes tainter1988's declining marginal returns but gives the mechanism a unit (bits
++per task) and a magnitude (a divergence).
++
++---
++
++## 5. Theorem 4 — hierarchical routing bound and root load
++
++**DEFINITION 5.1 (hierarchy).** A hierarchy is a rooted tree whose $N$ leaves are the agents and whose internal
++nodes are administrative units; every internal node has at most $b \ge 2$ children; the depth (leaf-to-root distance)
++is $d = \lceil \log_b N \rceil$ for a complete $b$-ary tree. A task between two leaves is *settled by relay* when each
++participant sends a message up to their lowest common ancestor (LCA) and the LCA sends the settlement down.
++
++**THEOREM 4.**
++(i) *Message bound.* Any pairwise task can be settled with at most $2d$ messages (up-path plus down-path through the
++LCA), versus up to $N-1$ messages for a flat broadcast in which the initiator must reach every other agent. For $M$
++pairwise tasks the total is at most $2M\lceil\log_b N\rceil$.
++(ii) *Root load.* In a complete $b$-ary tree with $N = b^d$ leaves, if the two participants of a task are drawn
++uniformly at random among leaves (independently, with replacement), the probability that their LCA is the root is
++$1 - 1/b$. Hence the expected number of tasks that pass through the root is $M(1 - 1/b)$.
++(iii) *Federated / modular coordination.* If a fraction $\theta \in [0,1]$ of tasks is *local* — both participants
++belong to the same first-level module (child subtree of the root) by construction of the task process — and the
++remaining $1-\theta$ are drawn uniformly as in (ii), the expected root load is $M(1-\theta)(1 - 1/b)$.
++
++*Proof.* (i) In a tree there is a unique simple path between two leaves; it passes through their LCA and has length at
++most $2d$ since each leaf is at depth $\le d$ and the LCA is at depth $\ge 0$. Sending one message per edge along
++the two leaf-to-LCA paths settles the task in $\le 2d$ messages. A flat broadcast from one initiator to the other
++$N-1$ agents uses $N-1$ messages. Summing over $M$ tasks gives (i).
++(ii) The root has $b$ children, each the root of a subtree with $b^{d-1}$ leaves. The first participant lies in some
++child subtree $j$; the second participant, drawn uniformly and independently, lies in the same subtree $j$ with
++probability $b^{d-1}/b^d = 1/b$. The LCA is the root exactly when the two participants lie in different child subtrees,
++which has probability $1 - 1/b$. Expectation of a sum of $M$ indicators gives $M(1-1/b)$.
++(iii) Local tasks never reach the root (their LCA lies inside a module); the non-local fraction behaves as in (ii).
++By linearity of expectation the load is $M(1-\theta)(1-1/b)$. $\square$
++
++**Remark 5.2.** Theorem 4 formalizes the trade-off between centralized administration and federated coordination
++(radner1993; bolton1994): a hierarchy cuts the *message* cost of coordination from linear to logarithmic in $N$, but
++concentrates a constant fraction $1 - 1/b$ of all non-local tasks on the apex, whose processing rate then bounds $K$
++(Definition 1.3). Modularity reduces apex load by the locality fraction $\theta$ at the price of *not* reconciling
++across modules. These are the five regimes compared in the pre-registered simulations (no institution, shared norms,
++standardized records, centralized administration, federated), and the simulations measure messages and bytes
++directly rather than assuming the bounds.
++
++---
++
++## 6. Theorem 5 — backlog dynamics under mismatch
++
++**DEFINITION 6.1 (backlog).** Let $D_t$ be the number of tasks arriving in period $t$ and $K_t$ the number the system
++can settle in period $t$. The backlog of unsettled tasks follows the Lindley recursion
++$$B_{t+1} = \max\{0,\ B_t + D_t - K_t\},\qquad B_0 \ge 0 .$$
++
++**THEOREM 5.**
++(i) *Deterministic case.* If $D_t = D$ and $K_t = K$ for all $t$, then: if $D \le K$, $B_t \le \max\{B_0 - t(K-D), 0\}$,
++so the backlog reaches $0$ after at most $\lceil B_0/(K-D) \rceil$ periods when $D < K$ and stays constant at $B_0$
++when $D = K$; if $D > K$, $B_t = B_0 + t(D-K) \to \infty$ linearly.
++(ii) *Stochastic case.* If $(D_t - K_t)_{t\ge0}$ is i.i.d. with finite mean, then when $\mathbb{E}[D_t] < \mathbb{E}[K_t]$
++the backlog has a proper stationary distribution to which $B_t$ converges from any $B_0$, and when
++$\mathbb{E}[D_t] > \mathbb{E}[K_t]$, $B_t \to \infty$ almost surely. (Standard: the recursion is the single-server
++waiting-time recursion of Lindley (1952), and the stability dichotomy is Loynes' theorem (1962); the proof for (ii) is
++not reproduced here and the result is cited.)
++(iii) *Delay under a Markovian approximation.* If arrivals are Poisson with rate $\lambda_D$ and settlements are
++exponential with rate $\mu$ (an M/M/1 queue), with $\rho = \lambda_D/\mu < 1$, the expected sojourn time is
++$W = 1/(\mu - \lambda_D) = (1/\mu)/(1-\rho)$, which is increasing and convex in $\rho$ on $[0,1)$ and diverges as
++$\rho \uparrow 1$.
++
++*Proof of (i).* If $D \le K$, then $B_{t+1} = \max\{0, B_t - (K-D)\}$; by induction
++$B_t \le \max\{0, B_0 - t(K-D)\}$, with equality as long as the argument is nonnegative, which gives the hitting time
++when $D < K$ and constancy when $D = K$. If $D > K$, then $B_t + D - K > 0$ for all $t$ (since $B_t \ge 0$), so the
++maximum is never binding and $B_{t+1} = B_t + (D-K)$, hence $B_t = B_0 + t(D-K)$.
++*Proof of (iii).* The M/M/1 stationary number in system is geometric with parameter $\rho$, so the mean is
++$\rho/(1-\rho)$ and Little's law gives $W = \rho/((1-\rho)\lambda_D) = 1/(\mu(1-\rho)) = 1/(\mu - \lambda_D)$ (any
++standard queueing reference). The function $f(\rho) = (1-\rho)^{-1}$ has $f' = (1-\rho)^{-2} > 0$ and
++$f'' = 2(1-\rho)^{-3} > 0$ on $[0,1)$, and $f(\rho) \to \infty$ as $\rho \uparrow 1$. $\square$
++
++**Scope of Theorem 5.** This is a *mechanism*, not evidence. Polities are not single-server queues: capacity is
++endogenous (institutions are built in response to backlog), arrivals are correlated across periods and agents, and
++"unsettled tasks" are not observed in the historical record. The theorem is used in the paper only to motivate two
++qualitative predictions — that costs rise smoothly and convexly in utilization, and that sustained $\rho > 1$ is
++qualitatively different from $\rho < 1$ — and both are stated as conjectures below.
++
++---
++
++## 7. Conjectures (unproved; each with its falsifier)
++
++**CONJECTURE C1 (proxy monotonicity).** The pre-registered Seshat and V-Dem proxies are monotone in the model's
++primitives: the scale variables (population, territory, largest settlement, settlement hierarchy) increase with
++$D$ of Definition 1.2, and the four indicator blocks (administration, standardized rules and money, communication
++infrastructure, record-keeping) increase with $K$ of Definition 1.3, so that $P = z(\log D) - z(K)$ is monotone in
++$\log\rho$ up to measurement error. *Reason:* population and territory raise the number and the dispersion of
++participants; written records, legal codes, money and measurement systems are shared codes (Theorem 1); roads,
++couriers and postal systems raise bits per period; administrative levels implement the routing of Theorem 4.
++*Falsifier:* the proxies for $D$ and $K$ are so collinear (as Seshat's first principal component suggests; shin2020)
++that $P$ retains almost no variance, or alternative defensible $K$ weightings reverse the sign of $P$ for a
++substantial share of polities — both are reported in `research/measurement.md`.
++
++**CONJECTURE C2 (sustained overload raises breakdown probability).** Periods with $\rho > 1$ sustained longer than
++the time needed to build capacity raise the probability of institutional breakdown. *Mechanism family (not proved):*
++unsettled tasks accumulate (Theorem 5), so participants cannot form the common knowledge required for coordinated
++action (chwe2000) and local defection spreads by contagion through the interaction network (morris2000); institutions
++whose codes are stale after a shock lose their cost advantage (Theorem 3) and are abandoned. *Falsifier:* the
++pre-registered H2 tests find $\beta_P$ not positive, or $P$ does not beat the population-only and capacity-only
++baselines out of sample.
++
++**CONJECTURE C3 (smooth, not step).** The relation between $P$ and instability is convex and smooth away from
++$\rho \approx 1$ and steepens near it, rather than a discrete threshold. *Reason:* Theorem 5(iii) gives a smooth,
++convex delay in $\rho$; discontinuities would require an additional tipping mechanism (e.g. a critical mass in a
++contagion model) that the framework does not supply. *Falsifier:* the pre-registered H4 comparison finds that a hinge
++or step model beats the smooth model out of sample in $\ge 80\%$ of folds, which would favour a tipping mechanism;
++conversely, if neither nonlinear form beats the linear model, C3's "steepening" clause is unsupported.
++
++---
++
++## 8. What is not claimed, and which test speaks to each statement
++
++- **No Shannon measurement of history.** Theorems 1 and 3 are about codes for a plan set with a known distribution.
++  Seshat indicators (presence of writing, coins, courts, roads) are not message lengths, and no entropy or divergence
++  is computed from historical data anywhere in this project. Shannon quantities appear only in the simulations, where
++  the plan distribution $p$ and the messages are defined by the experimenter and can be measured.
++- **No causal claim.** The theorems describe an idealized communication system; the empirical tests are associational
++  and pre-registered as such (`research/preregistration.md`, §1 and §5); the identification discussion is in
++  `research/identification.md`.
++- **Endogeneity of capacity.** The framework takes $K$ as given within a period; historically, $K$ is built in response
++  to $D$ (johnson1982; shin2020). This is why H1 is pre-registered as an association without a direction.
++
++| Statement | What can speak to it | Pre-registered test |
++|---|---|---|
++| Theorem 1 (coding gain) | Simulated messages and bits with vs. without a shared code | H3 (regimes b, c vs a) |
++| Theorem 2 (break-even) | Simulated total cost across $N$, $\lambda$, horizon, overhead $a$ | H3 factorial cells; search for cells where institutions lose |
++| Theorem 3 (stale code) | Simulated shocks to the task distribution | H3 adverse regimes (shock treatment) |
++| Theorem 4 (routing, root load) | Simulated messages and apex load under centralized vs federated regimes | H3 (regimes d, e) |
++| Theorem 5 (backlog) | Not testable with the historical data (backlog unobserved); illustrated in simulation only | none (stated as mechanism) |
++| Conjecture C1 | Measurement diagnostics: collinearity of $D$ and $K$ proxies, sensitivity of $P$ to weights | `research/measurement.md` |
++| Conjecture C2 | Lagged $P$ and subsequent instability vs baselines, out of sample | H2 (Seshat Y1–Y3; V-Dem Y4–Y6), H5 |
++| Conjecture C3 | Linear vs hinge vs spline in $P$, out of sample | H4 |

added research/measurement.md +80 −0

++# WP35 — Measurement report (Phase 3)
++
++All numbers come from `results/empirical/measurement_diagnostics.json`, `measurement_diagnostics_modern.json`,
++`measurement_irt_item_parameters.csv`, `power_h2a_*.csv`, produced by `python -m wp35.measurement.build_indices`
++(2026-11-01). **No outcome variable was read by that module.** Definitions follow `research/preregistration.md` §3.
++
++## 1. Seshat (polity level)
++
++### 1.1 Coordination demand D
++Four logged scale variables (polity population, territory, largest-settlement population; settlement hierarchy levels
++unlogged). First principal component of their pairwise-complete correlation matrix explains **82.1%** of variance with
++near-equal loadings (log pop 0.26, log territory 0.24, log capital 0.26, settlement hierarchy 0.24). Score = loading-
++weighted mean of available standardized variables, requiring ≥ 2 of 4; **442 polities** have `D_pc1`; 299 have
++population alone (`log_pop`). Range bounds (`value_from`/`value_to`) propagate to `D_pc1_lo/hi`: the median half-width
++is 0.017 SD, the 90th percentile 0.093 SD, so range uncertainty in the scale variables is small relative to
++between-polity variation (the larger uncertainty is *which* polities have any estimate at all).
++
++### 1.2 Coordination capacity K
++Four blocks of presence indicators, each the share coded present among coded items (≥ 50% coded required):
++administration (5 items incl. rank-scaled administrative levels), standardized rules and money (10), communication
++infrastructure (7), record-keeping (9). `K_mean` = mean of blocks with ≥ 3 blocks: **344 polities**. Inter-block
++correlations 0.75–0.79; first PC of the blocks explains 82.4% (loadings 0.25 each), and `K_pc1` correlates 0.999 with
++`K_mean`. Cronbach α of the four blocks by macro-region ranges 0.83 (Europe) to 0.97 (South Asia); mean off-diagonal
++block correlation by era 0.67 (1001–1500) to 0.81 (pre-1000 BCE) — the four capacity blocks hang together
++in every region and era.
++Latent alternative: 2PL IRT (MAP, log-normal prior on discrimination, EM converged in 135 iterations) on the 30 binary
++items (disputed 0.5 → missing; administrative levels excluded) gives `K_irt` for **579 polities** (≥ 3 items). It
++correlates **0.96** with `K_mean`. Item difficulties order sensibly (tokens, non-written records, roads, couriers easy;
++paper currency, general postal service, examination system, merit promotion hard; `article` is nearly universal and
++uninformative). Five record-keeping items (written record, script, lists/tables, calendar, practical literature) hit the
++discrimination ceiling (8): they are coded almost jointly in Seshat, i.e. locally dependent; the latent score is
++therefore dominated by the "writing cluster", which is why we keep the transparent block mean as primary. Posterior SD of
++`K_irt`: median 0.34, 90th percentile 0.61 (on a unit-variance scale) — individual-polity capacity scores carry
++substantial uncertainty; this is carried into the econometrics through the IRT variant and the bound runs.
++Cross-region invariance: refitting the IRT separately on the Old World (n = 532) and on the Americas + Southeast
++Asia/Oceania (n = 94) gives item difficulties correlated **0.88** across the two fits (mean absolute difference 1.16
++logits, inflated by boundary estimates on the small New-World sample). Comparability is acceptable for the ordering
++of items but the New-World sample is too small for a strict invariance test; leave-region-out results (H5) are the
++operational safeguard.
++Leave-one-block-out: dropping any block leaves `K` correlated ≥ 0.98 with the full index and `P` correlated ≥ 0.95
++with `P_main`.
++
++### 1.3 Coordination pressure P and the collinearity problem
++`P_main = z(D_pc1) − z(K_mean)` is defined for **317 polities** and, in pre-crisis form, for **127 of the 166 CrisisDB
++records**. As anticipated from Turchin et al. (2018) and Shin et al. (2020), scale and capacity are strongly
++collinear: corr(`D_pc1`, `K_mean`) = **0.84** (0.80 with `K_irt`). Consequently `P_main` retains only **16.6%** of the
++variance of z(D)+z(K); it correlates 0.39 with D and −0.18 with K, and −0.04 with the Turchin-style single complexity
++axis (`complexity_pc1`, first PC of 57 social-complexity indicators, 49.7% of variance). The mismatch index is thus
++nearly orthogonal to overall complexity, which is exactly what makes the H2 test informative *if* P carries signal, and
++what makes it fragile: P is a residual, so measurement error in either D or K leaks into it. The pre-registered
++baselines (D-only, K-only, complexity-PC1-only) address this directly.
++
++### 1.4 A-priori power (pre-registration §5.8; random outcomes, actual P distribution, n = 127)
++| Outcome | β (per SD of P, logit) | power |
++|---|---|---|
++| Y1b binary (base rate 0.77) | 0.4 / 0.5 / 0.6 / 0.8 | 0.47 / 0.63 / 0.79 / 0.95 |
++| Y1 fractional (14 items, base 0.29) | 0.10 / 0.15 / 0.20 | 0.49 / 0.81 / 0.96 |
++**Decision (pre-registered rule):** the binary severe-outcome test Y1b is *underpowered a priori* (MDE ≈ 0.6 > 0.5);
++the fractional severity Y1 is adequately powered for effects of ≈ 0.15 log-odds per SD. Y1 remains the primary
++Seshat H2 outcome; Y1b is reported as secondary.
++
++## 2. Modern panel
++`K_modern` = mean of six standardized V-Dem items (≥ 4 coded): rigorous/impartial administration, meritocratic
++appointment, transparent laws, fiscal capacity source, territorial control, EMB capacity; PC1 explains 60.5% with
++loadings 0.13–0.19. `D_modern` = log Maddison population (1820+); `P_modern` defined for **15,697 country-years**.
++corr(D, K) = 0.06 (no collinearity problem in the modern panel). **Caveat fixed before analysis:** corr(`K_modern`,
++`v2x_polyarchy`) = **0.75** — the V-Dem capacity items are strongly associated with electoral democracy even though no
++democracy index enters K; all modern specifications therefore control for polyarchy as pre-registered, and the K-only
++baseline is reported.
++
++## 3. Ethnographic Atlas
++`log_community_size` from EA031 category midpoints (611 societies), `log_population` from continuous EA202 (953),
++`jurisdictional_hierarchy_beyond` = EA033 − 1 (0–4 levels; 1,155). No outcome; used for H1 only, with Glottolog
++family (173 families) and region controls.
++
++## 4. What the measurement model cannot do
++- Seshat K items are presence codes: they measure *existence*, not reach or quality, of institutions.
++- Most facts are undated, so "pre-crisis" K is mostly whole-polity K (see pre-registration §3.6).
++- No direct indicator of division of labour or dispersion in Polaris; D is scale only.
++- IRT local dependence among writing items; invariance only coarsely checked.

added results/empirical/measurement_diagnostics.json +321 −0

++{
++  "D_pc1_loadings": {
++    "log_pop": 0.2646,
++    "log_terr": 0.2366,
++    "log_cap": 0.2617,
++    "settl_hier": 0.2371
++  },
++  "D_pc1_var_share": 0.8209,
++  "K_pc1_loadings": {
++    "K_admin": 0.2481,
++    "K_rules_money": 0.2505,
++    "K_comm_infra": 0.2526,
++    "K_records": 0.2488
++  },
++  "K_pc1_var_share": 0.8239,
++  "complexity_pc1_var_share": 0.497,
++  "complexity_pc1_n_vars": 57,
++  "irt_loglik": -3345.5533228338218,
++  "irt_iterations": 135,
++  "n_D_pc1": 442,
++  "n_K_mean": 344,
++  "n_K_irt": 579,
++  "n_P_main": 317,
++  "n_P_main_crisis": 127,
++  "corr_zD_zK": {
++    "D_pc1~K_mean": 0.8384,
++    "D_pc1~K_irt": 0.7998,
++    "log_pop~K_mean": 0.7966,
++    "K_mean~K_irt": 0.9625,
++    "K_mean~K_pc1": 0.9994,
++    "P_main~complexity_pc1": -0.0434,
++    "P_main~D_pc1": 0.3855,
++    "P_main~K_mean": -0.1797
++  },
++  "P_variance_share_of_D_plus_K": 0.1655,
++  "block_correlations": {
++    "K_admin": {
++      "K_admin": 1.0,
++      "K_rules_money": 0.755,
++      "K_comm_infra": 0.766,
++      "K_records": 0.751
++    },
++    "K_rules_money": {
++      "K_admin": 0.755,
++      "K_rules_money": 1.0,
++      "K_comm_infra": 0.789,
++      "K_records": 0.757
++    },
++    "K_comm_infra": {
++      "K_admin": 0.766,
++      "K_rules_money": 0.789,
++      "K_comm_infra": 1.0,
++      "K_records": 0.773
++    },
++    "K_records": {
++      "K_admin": 0.751,
++      "K_rules_money": 0.757,
++      "K_comm_infra": 0.773,
++      "K_records": 1.0
++    }
++  },
++  "block_alpha_by_macro_region": {
++    "Central_Asia": {
++      "n": 17,
++      "alpha": 0.933
++    },
++    "East_Asia": {
++      "n": 35,
++      "alpha": 0.964
++    },
++    "Europe": {
++      "n": 57,
++      "alpha": 0.831
++    },
++    "North_Africa": {
++      "n": 27,
++      "alpha": 0.886
++    },
++    "North_America": {
++      "n": 19,
++      "alpha": 0.936
++    },
++    "South_Asia": {
++      "n": 11,
++      "alpha": 0.968
++    },
++    "Southeast_Asia_Oceania": {
++      "n": 19,
++      "alpha": 0.935
++    },
++    "Southwest_Asia": {
++      "n": 39,
++      "alpha": 0.88
++    }
++  },
++  "block_corr_mean_by_era": {
++    "1-1000": {
++      "n": 46,
++      "mean_offdiag_corr": 0.726
++    },
++    "1000BCE-0": {
++      "n": 35,
++      "mean_offdiag_corr": 0.756
++    },
++    "1001-1500": {
++      "n": 51,
++      "mean_offdiag_corr": 0.666
++    },
++    "1501-1800": {
++      "n": 38,
++      "mean_offdiag_corr": 0.785
++    },
++    "post-1800": {
++      "n": 20,
++      "mean_offdiag_corr": 0.765
++    },
++    "pre-1000BCE": {
++      "n": 43,
++      "mean_offdiag_corr": 0.805
++    }
++  },
++  "measurement_error": {
++    "D_pc1_halfwidth_median": 0.016672931644356626,
++    "D_pc1_halfwidth_q90": 0.09309851215013848,
++    "K_irt_psd_median": 0.34213753777518097,
++    "K_irt_psd_q90": 0.6062046010250374,
++    "sd_K_irt": 1.008347206077521,
++    "P_main_range_median": 0.036226778179016206
++  },
++  "irt_invariance": {
++    "items_compared": 30,
++    "corr_difficulty_old_vs_new": 0.88,
++    "mean_abs_diff_difficulty": 1.162,
++    "items": [
++      "full_time_bureaucrat",
++      "specialized_government_building",
++      "merit_promotion",
++      "examination_system",
++      "formal_legal_code",
++      "judge",
++      "court",
++      "professional_lawyer",
++      "indigenous_coin",
++      "foreign_coin",
++      "paper_currency",
++      "token",
++      "precious_metal",
++      "article",
++      "courier",
++      "postal_station",
++      "general_postal_service",
++      "road",
++      "bridge",
++      "canal",
++      "port",
++      "written_record",
++      "script",
++      "phonetic_alphabetic_writing",
++      "non_phonetic_writing",
++      "lists_tables_and_classification",
++      "calendar",
++      "practical_literature",
++      "nonwritten_record",
++      "mnemonic_device"
++    ],
++    "fits": {
++      "old_world": {
++        "n": 525,
++        "b": [
++          -0.567,
++          -0.598,
++          0.554,
++          1.04,
++          -0.659,
++          -0.089,
++          -0.159,
++          0.605,
++          -0.041,
++          -0.287,
++          2.037,
++          -1.102,
++          -1.05,
++          -3.99,
++          -0.984,
++          0.164,
++          1.134,
++          -1.121,
++          -0.949,
++          -0.328,
++          -0.901,
++          -0.736,
++          -0.845,
++          -0.409,
++          1.716,
++          -0.673,
++          -0.669,
++          -0.498,
++          -3.397,
++          -1.374
++        ],
++        "a": [
++          3.44,
++          3.543,
++          1.905,
++          3.328,
++          2.428,
++          2.984,
++          2.47,
++          2.412,
++          3.564,
++          1.758,
++          1.82,
++          0.646,
++          2.036,
++          0.626,
++          2.456,
++          2.305,
++          1.695,
++          3.24,
++          2.977,
++          2.077,
++          1.802,
++          8.0,
++          8.0,
++          1.644,
++          0.822,
++          8.0,
++          8.0,
++          6.448,
++          1.202,
++          1.277
++        ]
++      },
++      "new_world_oceania": {
++        "n": 94,
++        "b": [
++          0.128,
++          0.085,
++          2.663,
++          3.537,
++          0.354,
++          0.801,
++          0.728,
++          1.073,
++          1.881,
++          1.071,
++          2.593,
++          1.207,
++          0.757,
++          -3.221,
++          0.518,
++          1.58,
++          1.786,
++          0.072,
++          0.07,
++          0.565,
++          0.578,
++          0.159,
++          0.004,
++          0.736,
++          1.473,
++          0.642,
++          -0.049,
++          0.922,
++          -1.575,
++          -0.93
++        ],
++        "a": [
++          3.407,
++          1.466,
++          0.849,
++          1.541,
++          3.221,
++          2.923,
++          3.622,
++          3.145,
++          1.934,
++          1.517,
++          1.11,
++          0.467,
++          1.44,
++          0.454,
++          1.585,
++          1.687,
++          1.946,
++          1.387,
++          1.46,
++          1.677,
++          2.691,
++          2.461,
++          2.748,
++          3.894,
++          0.629,
++          2.858,
++          3.09,
++          3.732,
++          0.826,
++          0.696
++        ]
++      }
++    }
++  },
++  "leave_one_block_out": {
++    "admin": {
++      "corr_K_with_full": 0.9858,
++      "corr_P_with_main": 0.9575
++    },
++    "rules_money": {
++      "corr_K_with_full": 0.9877,
++      "corr_P_with_main": 0.9615
++    },
++    "comm_infra": {
++      "corr_K_with_full": 0.9874,
++      "corr_P_with_main": 0.9657
++    },
++    "records": {
++      "corr_K_with_full": 0.9816,
++      "corr_P_with_main": 0.9474
++    }
++  }
++}
+ \ No newline at end of file

added results/empirical/measurement_diagnostics_modern.json +72 −0

++{
++  "K_modern_pc1_loadings": {
++    "v2clrspct": 0.1859,
++    "v2stcritrecadm": 0.1678,
++    "v2cltrnslw": 0.1906,
++    "v2stfisccap": 0.15,
++    "v2svstterr": 0.1337,
++    "v2elembcap": 0.1721
++  },
++  "K_modern_pc1_var_share": 0.6052,
++  "n_P_modern": 15697,
++  "corr_D_K_modern": 0.0624,
++  "corr_K_modern_polyarchy": 0.7541,
++  "item_corr": {
++    "v2clrspct": {
++      "v2clrspct": 1.0,
++      "v2stcritrecadm": 0.644,
++      "v2cltrnslw": 0.813,
++      "v2stfisccap": 0.473,
++      "v2svstterr": 0.422,
++      "v2elembcap": 0.574
++    },
++    "v2stcritrecadm": {
++      "v2clrspct": 0.644,
++      "v2stcritrecadm": 1.0,
++      "v2cltrnslw": 0.621,
++      "v2stfisccap": 0.438,
++      "v2svstterr": 0.359,
++      "v2elembcap": 0.521
++    },
++    "v2cltrnslw": {
++      "v2clrspct": 0.813,
++      "v2stcritrecadm": 0.621,
++      "v2cltrnslw": 1.0,
++      "v2stfisccap": 0.513,
++      "v2svstterr": 0.424,
++      "v2elembcap": 0.654
++    },
++    "v2stfisccap": {
++      "v2clrspct": 0.473,
++      "v2stcritrecadm": 0.438,
++      "v2cltrnslw": 0.513,
++      "v2stfisccap": 1.0,
++      "v2svstterr": 0.374,
++      "v2elembcap": 0.497
++    },
++    "v2svstterr": {
++      "v2clrspct": 0.422,
++      "v2stcritrecadm": 0.359,
++      "v2cltrnslw": 0.424,
++      "v2stfisccap": 0.374,
++      "v2svstterr": 1.0,
++      "v2elembcap": 0.443
++    },
++    "v2elembcap": {
++      "v2clrspct": 0.574,
++      "v2stcritrecadm": 0.521,
++      "v2cltrnslw": 0.654,
++      "v2stfisccap": 0.497,
++      "v2svstterr": 0.443,
++      "v2elembcap": 1.0
++    }
++  },
++  "corr_K_by_region": {
++    "1": -0.085,
++    "2": 0.259,
++    "3": 0.081,
++    "4": 0.041,
++    "5": -0.045,
++    "6": -0.012
++  }
++}
+ \ No newline at end of file

added results/empirical/measurement_irt_item_parameters.csv +31 −0

++item,a,b,block
++full_time_bureaucrat,3.6527424079806603,-0.41859686853078504,admin
++specialized_government_building,3.066730654084431,-0.5085373165354822,admin
++merit_promotion,1.7802655820815738,0.7021586341368805,admin
++examination_system,3.3869433671617886,1.1375006596571404,admin
++formal_legal_code,2.9525895205555868,-0.4141059941613492,rules_money
++judge,3.552224718836929,0.060069448676890166,rules_money
++court,3.0760342165002372,0.008468451933063378,rules_money
++professional_lawyer,2.72592841935458,0.6535317471947827,rules_money
++indigenous_coin,3.7072105660727472,0.15112395390732436,rules_money
++foreign_coin,1.9763009083036096,-0.08042415203888174,rules_money
++paper_currency,1.4853538625942566,2.2492864209104257,rules_money
++token,0.579698340872784,-0.8294519474758425,rules_money
++precious_metal,2.1558974591242315,-0.6735737580810314,rules_money
++article,0.5286046257791284,-4.364025532653697,rules_money
++courier,2.670690519140207,-0.6724574117368375,comm_infra
++postal_station,2.560978399199805,0.3210643907483727,comm_infra
++general_postal_service,1.9077225274480532,1.1474143346185122,comm_infra
++road,2.8878850939209206,-0.8863284670864775,comm_infra
++bridge,2.7265013330964507,-0.7485140494607205,comm_infra
++canal,2.165331667573359,-0.1798629790826995,comm_infra
++port,2.2796348701360767,-0.5152583130598091,comm_infra
++written_record,8.0,-0.5885046260962389,records
++script,7.974647676175637,-0.6926954931493575,records
++phonetic_alphabetic_writing,2.1001761814441022,-0.1634518168751169,records
++non_phonetic_writing,0.6965894707957191,1.9288626298500053,records
++lists_tables_and_classification,7.643776344260694,-0.4587993518664177,records
++calendar,8.0,-0.5777488445589021,records
++practical_literature,8.0,-0.2566162216282787,records
++nonwritten_record,1.2026821075448688,-2.671075035463131,records
++mnemonic_device,1.038487055773755,-1.398153218842954,records

added results/empirical/power_h2a_binary.csv +8 −0

++beta,power,n
++0.2,0.1875,127
++0.3,0.31,127
++0.4,0.47,127
++0.5,0.625,127
++0.6,0.785,127
++0.8,0.945,127
++1.0,0.99,127

added results/empirical/power_h2a_fractional.csv +7 −0

++beta,power,n
++0.1,0.4925,127
++0.15,0.805,127
++0.2,0.9625,127
++0.3,1.0,127
++0.4,1.0,127
++0.5,1.0,127

modified results/logs/execution_log.md +3 −0

				`@@ -17,3 +17,6 @@`
17	\| 2026-11-01 03:55 \| 2 \| Glottolog 5.3 languages.csv registered + fetched (language families for Galton controls) \| registry 9 sources \|	17	\| 2026-11-01 03:55 \| 2 \| Glottolog 5.3 languages.csv registered + fetched (language families for Galton controls) \| registry 9 sources \|
18	\| 2026-11-01 04:10 \| 2 \| `python -m wp35.harmonization.build_panels` (3 runs: EA033 code parsing fixed via `ord`; EA202 continuous; Y6 redefined as last regime year with irregular end type after inspecting France 1789–1960) \| `data/processed/.parquet`, `results/diagnostics/`, `paper/figures/fig_seshat_temporal_coverage.pdf`, `fig_seshat_missingness.pdf`, `fig_dplace_focal_years.pdf`, `fig_modern_coverage.pdf`, `coverage_summary.json` \|	18	\| 2026-11-01 04:10 \| 2 \| `python -m wp35.harmonization.build_panels` (3 runs: EA033 code parsing fixed via `ord`; EA202 continuous; Y6 redefined as last regime year with irregular end type after inspecting France 1789–1960) \| `data/processed/.parquet`, `results/diagnostics/`, `paper/figures/fig_seshat_temporal_coverage.pdf`, `fig_seshat_missingness.pdf`, `fig_dplace_focal_years.pdf`, `fig_modern_coverage.pdf`, `coverage_summary.json` \|
19	\| 2026-11-01 04:15 \| 2 \| Data dictionary + parser tests (`pytest`: 6 passed) \| `data/metadata/data_dictionary.md`, `tests/test_harmonization.py` \|	19	\| 2026-11-01 04:15 \| 2 \| Data dictionary + parser tests (`pytest`: 6 passed) \| `data/metadata/data_dictionary.md`, `tests/test_harmonization.py` \|
		20	+\| 2026-11-01 04:40 \| 3 \| Formal framework drafted (fork agent) and reviewed: Theorems 1–5 with proofs, Conjectures C1–C3, mapping to tests; Lindley 1952 / Loynes 1962 verified by Crossref and added \| `research/framework.md` \|
		21	+\| 2026-11-01 04:50 \| 3 \| `python -m wp35.measurement.build_indices` (2 runs; IRT regularized with MAP priors after boundary estimates) \| `data/processed/seshat_indices.parquet`, `seshat_crisis_indices.parquet`, `modern_indices.parquet`; `results/empirical/measurement_`, `power_h2a_.csv`; figures `fig_measurement_DK.pdf`, `fig_irt_invariance.pdf` \|
		22	+\| 2026-11-01 05:00 \| 3 \| Measurement report: corr(D,K)=0.84, P keeps 16.6% of variance; MDE Y1b≈0.6 (underpowered), Y1≈0.15 \| `research/measurement.md` \|

added src/wp35/measurement/build_indices.py +288 −0

++"""Phase 3 measurement: build D, K, P (and rival covariates) exactly as pre-registered, with alternatives,
++measurement error, cross-region comparability diagnostics and an a-priori power computation.
++Usage: python -m wp35.measurement.build_indices
++No outcome (Y1, Y2, Y3, Y4-6) is read or used anywhere in this module.
++"""
++from __future__ import annotations
++import json
++from pathlib import Path
++import numpy as np
++import pandas as pd
++import matplotlib
++matplotlib.use("Agg")
++import matplotlib.pyplot as plt
++import statsmodels.api as sm
++from wp35.measurement.irt import fit_2pl
++from wp35.harmonization.build_panels import K_BLOCKS, STANDARDS, D_VARS, PROD_INFRA, MILITARY
++
++ROOT = Path(__file__).resolve().parents[3]
++PROC = ROOT / "data" / "processed"; RES = ROOT / "results" / "empirical"; FIG = ROOT / "paper" / "figures"
++RES.mkdir(parents=True, exist_ok=True); FIG.mkdir(parents=True, exist_ok=True)
++SEED = 20261101
++K_ITEMS_MODERN = ["v2clrspct", "v2stcritrecadm", "v2cltrnslw", "v2stfisccap", "v2svstterr", "v2elembcap"]
++BIN_K = [v for b in K_BLOCKS.values() for v in b if v != "administrative_level"]
++
++
++def z(x: pd.Series, ref: pd.Series | None = None) -> pd.Series:
++    ref = x if ref is None else ref
++    return (x - ref.mean()) / ref.std(ddof=0)
++
++
++def pc1_pairwise(df: pd.DataFrame, min_coded: int = 2):
++    """First principal component of the pairwise-complete correlation matrix; scores = loading-weighted mean of the
++    available standardized variables (requires >= min_coded coded). Loadings signed so that the sum is positive."""
++    Zs = (df - df.mean()) / df.std(ddof=0)
++    C = Zs.corr(min_periods=10).fillna(0).values
++    w, V = np.linalg.eigh(C); v = V[:, -1]; v = v if v.sum() > 0 else -v
++    load = pd.Series(v / np.abs(v).sum(), index=df.columns)
++    avail = Zs.notna()
++    num = (Zs.fillna(0) * load).sum(axis=1); den = (avail * load.abs()).sum(axis=1)
++    score = num / den.where(den > 0)
++    score[avail.sum(axis=1) < min_coded] = np.nan
++    return score, load, float(w[-1] / w.sum())
++
++
++def block_score(wide: pd.DataFrame, items: list[str], min_share: float = 0.5) -> pd.Series:
++    sub = wide.reindex(columns=items).copy()
++    if "administrative_level" in sub.columns:
++        r = sub["administrative_level"].rank(method="average")
++        sub["administrative_level"] = (r - 1) / (r.max() - 1)
++    coded = sub.notna().mean(axis=1)
++    return sub.mean(axis=1).where(coded >= min_share)
++
++
++def build_seshat():
++    wide = pd.read_parquet(PROC / "seshat_polity_wide.parquet")
++    agg = pd.read_parquet(PROC / "seshat_polity_agg.parquet")
++    meta = pd.read_parquet(PROC / "seshat_polity_meta.parquet").set_index("polity_id")
++    logw = agg.pivot(index="polity_id", columns="variable", values="logv_mean")
++    lo = agg.pivot(index="polity_id", columns="variable", values="lo_min"); hi = agg.pivot(index="polity_id", columns="variable", values="hi_max")
++    out = pd.DataFrame(index=wide.index)
++    # ---- D ----
++    Dmat = pd.DataFrame({"log_pop": logw["polity_population"], "log_terr": logw["polity_territory"],
++                         "log_cap": logw["population_of_the_largest_settlement"], "settl_hier": wide["settlement_hierarchy"]})
++    out["D_pc1"], D_load, D_var = pc1_pairwise(Dmat)
++    out["log_pop"] = Dmat.log_pop; out["log_terr"] = Dmat.log_terr; out["log_cap"] = Dmat.log_cap; out["settl_hier"] = Dmat.settl_hier
++    # bounds for measurement-error sensitivity (geometric variables only)
++    with np.errstate(divide="ignore"):
++        out["log_pop_lo"] = np.log(lo["polity_population"].where(lo["polity_population"] > 0)); out["log_pop_hi"] = np.log(hi["polity_population"].where(hi["polity_population"] > 0))
++        out["log_terr_lo"] = np.log(lo["polity_territory"].where(lo["polity_territory"] > 0)); out["log_terr_hi"] = np.log(hi["polity_territory"].where(hi["polity_territory"] > 0))
++    # ---- K blocks ----
++    for b, items in K_BLOCKS.items():
++        out[f"K_{b}"] = block_score(wide, items)
++    blocks = out[[f"K_{b}" for b in K_BLOCKS]]
++    out["K_mean"] = blocks.mean(axis=1).where(blocks.notna().sum(axis=1) >= 3)
++    out["K_n_blocks"] = blocks.notna().sum(axis=1)
++    out["K_pc1"], K_load, K_var = pc1_pairwise(blocks, min_coded=3)
++    # IRT on binary items (disputed 0.5 -> missing)
++    Xb = wide.reindex(columns=BIN_K).copy(); Xb = Xb.where(Xb != 0.5)
++    irt = fit_2pl(Xb.values)
++    out["K_irt"] = irt["eap"]; out["K_irt_psd"] = irt["psd"]; out["K_irt_nitems"] = irt["n_items"]
++    item_params = pd.DataFrame({"item": BIN_K, "a": irt["a"], "b": irt["b"], "block": [next(k for k, v in K_BLOCKS.items() if i in v) for i in BIN_K]})
++    item_params.to_csv(RES / "measurement_irt_item_parameters.csv", index=False)
++    # standards sub-index (sparse; latent only)
++    out["K_standards"] = block_score(wide, STANDARDS)
++    # ---- P ----
++    for dname in ["D_pc1", "log_pop"]:
++        for kname in ["K_mean", "K_irt", "K_pc1"]:
++            out[f"P_{dname}_{kname}"] = z(out[dname]) - z(out[kname])
++    out["P_main"] = out["P_D_pc1_K_mean"]
++    # standardization windows (sensitivity)
++    df = out.join(meta[["macro_region", "era"]])
++    out["P_within_era"] = df.groupby("era").D_pc1.transform(z) - df.groupby("era").K_mean.transform(z)
++    out["P_within_region"] = df.groupby("macro_region").D_pc1.transform(z) - df.groupby("macro_region").K_mean.transform(z)
++    out["P_rank"] = out.D_pc1.rank(pct=True) - out.K_mean.rank(pct=True)
++    # bounds
++    Dlo = pd.DataFrame({"log_pop": out.log_pop_lo, "log_terr": out.log_terr_lo, "log_cap": Dmat.log_cap, "settl_hier": Dmat.settl_hier})
++    Dhi = pd.DataFrame({"log_pop": out.log_pop_hi, "log_terr": out.log_terr_hi, "log_cap": Dmat.log_cap, "settl_hier": Dmat.settl_hier})
++    out["D_pc1_lo"] = ((Dlo - Dmat.mean()) / Dmat.std(ddof=0)).fillna(0).mul(D_load).sum(axis=1) / (Dlo.notna() * D_load.abs()).sum(axis=1).replace(0, np.nan)
++    out["D_pc1_hi"] = ((Dhi - Dmat.mean()) / Dmat.std(ddof=0)).fillna(0).mul(D_load).sum(axis=1) / (Dhi.notna() * D_load.abs()).sum(axis=1).replace(0, np.nan)
++    out.loc[out.D_pc1.isna(), ["D_pc1_lo", "D_pc1_hi"]] = np.nan
++    out["P_main_lo"] = z(out.D_pc1_lo, out.D_pc1) - z(out.K_mean); out["P_main_hi"] = z(out.D_pc1_hi, out.D_pc1) - z(out.K_mean)
++    # drop inferred codes sensitivity: recompute K_mean using only non-inferred facts
++    facts = pd.read_parquet(PROC / "seshat_facts.parquet")
++    ni = facts[(~facts.inferred) & facts.value.notna()].groupby(["polity_id", "variable"]).value.mean().unstack()
++    out["K_mean_noinf"] = pd.concat([block_score(ni, it) for it in K_BLOCKS.values()], axis=1).pipe(lambda b: b.mean(axis=1).where(b.notna().sum(axis=1) >= 3)).reindex(out.index)
++    out["P_main_noinf"] = z(out.D_pc1) - z(out.K_mean_noinf)
++    # ---- rival covariates (excluded from K by design) ----
++    wf = facts[facts.sheet == "Warfare"].groupby(["polity_id", "variable"]).value.mean().unstack()
++    out["X_warfare"] = wf.mean(axis=1).where(wf.notna().mean(axis=1) >= 0.5).reindex(out.index)
++    out["X_prod_infra"] = block_score(wide, PROD_INFRA)
++    mil = wide.reindex(columns=MILITARY).copy(); r = mil["military_level"].rank(); mil["military_level"] = (r - 1) / (r.max() - 1)
++    out["X_military"] = mil.mean(axis=1).where(mil.notna().mean(axis=1) >= 0.5)
++    out["X_moralizing"] = wide.get("moralizing_supernatural_punishment_and_reward")
++    rt = facts[facts.sheet == "Religion"].groupby(["polity_id", "variable"]).value.mean().unstack()
++    out["X_moralizing"] = rt.get("moralizing_supernatural_punishment_and_reward").reindex(out.index) if "moralizing_supernatural_punishment_and_reward" in rt else np.nan
++    # Turchin-style single complexity axis over ALL social-complexity indicators (baseline B3)
++    sc_vars = [v for v in facts[facts.sheet == "Social complexity"].variable.unique() if v in wide.columns]
++    scmat = wide[sc_vars].copy()
++    for v in ["polity_population", "polity_territory", "population_of_the_largest_settlement"]:
++        scmat[v] = logw[v]
++    scmat = scmat.loc[:, scmat.notna().sum() >= 100]
++    out["complexity_pc1"], C_load, C_var = pc1_pairwise(scmat, min_coded=5)
++    out = out.join(meta[["seshat_region", "homeNGA", "macro_region", "era", "start_year", "end_year", "midpoint", "duration", "censored", "centralization", "utm_zone"]])
++    out.index.name = "polity_id"
++    out.reset_index().to_parquet(PROC / "seshat_indices.parquet", index=False)
++    # ---- pre-crisis versions (same loadings/standardization as the full sample) ----
++    pre = pd.read_parquet(PROC / "seshat_precrisis_agg.parquet")
++    pw = pre.pivot(index="crisis_id", columns="variable", values="value_mean"); plog = pre.pivot(index="crisis_id", columns="variable", values="logv_mean")
++    cr = pd.DataFrame(index=pw.index)
++    Dp = pd.DataFrame({"log_pop": plog.get("polity_population"), "log_terr": plog.get("polity_territory"), "log_cap": plog.get("population_of_the_largest_settlement"), "settl_hier": pw.get("settlement_hierarchy")}, index=pw.index)
++    Zp = (Dp - Dmat.mean()) / Dmat.std(ddof=0)
++    cr["D_pc1"] = (Zp.fillna(0) * D_load).sum(axis=1) / (Zp.notna() * D_load.abs()).sum(axis=1).replace(0, np.nan)
++    cr.loc[Zp.notna().sum(axis=1) < 2, "D_pc1"] = np.nan
++    cr["log_pop"] = Dp.log_pop
++    for b, items in K_BLOCKS.items():
++        cr[f"K_{b}"] = block_score(pw, items)
++    bl = cr[[f"K_{b}" for b in K_BLOCKS]]; cr["K_mean"] = bl.mean(axis=1).where(bl.notna().sum(axis=1) >= 3)
++    Xp = pw.reindex(columns=BIN_K).where(lambda d: d != 0.5)
++    # EAP with the full-sample item parameters (no refit): posterior under fixed a, b
++    nodes, weights = np.polynomial.hermite_e.hermegauss(41); weights /= weights.sum()
++    from scipy.special import expit
++    Pq = expit(irt["a"][None, :] * (nodes[:, None] - irt["b"][None, :]))
++    Xv = Xp.values; obs = ~np.isnan(Xv); Xz_ = np.where(obs, Xv, 0)
++    L = Xz_ @ np.log(Pq + 1e-300).T + ((1 - Xz_) * obs) @ np.log(1 - Pq + 1e-300).T + np.log(weights)[None, :]
++    post = np.exp(L - L.max(axis=1, keepdims=True)); post /= post.sum(axis=1, keepdims=True)
++    cr["K_irt"] = np.where(obs.sum(axis=1) >= 3, post @ nodes, np.nan)
++    cr["K_pc1"] = (((bl - blocks.mean()) / blocks.std(ddof=0)).fillna(0) * K_load).sum(axis=1) / (bl.notna() * K_load.abs()).sum(axis=1).replace(0, np.nan)
++    cr.loc[bl.notna().sum(axis=1) < 3, "K_pc1"] = np.nan
++    for dname in ["D_pc1", "log_pop"]:
++        for kname in ["K_mean", "K_irt", "K_pc1"]:
++            cr[f"P_{dname}_{kname}"] = z(cr[dname], out[dname]) - z(cr[kname], out[kname])
++    cr["P_main"] = cr["P_D_pc1_K_mean"]
++    wfp = pre[pre.variable.isin(wf.columns)].pivot(index="crisis_id", columns="variable", values="value_mean")
++    cr["X_warfare"] = wfp.mean(axis=1).where(wfp.notna().mean(axis=1) >= 0.5).reindex(cr.index)
++    cr["X_prod_infra"] = block_score(pw, PROD_INFRA)
++    milp = pw.reindex(columns=MILITARY).copy(); milp["military_level"] = milp["military_level"].rank().pipe(lambda r: (r - 1) / (r.max() - 1)); cr["X_military"] = milp.mean(axis=1).where(milp.notna().mean(axis=1) >= 0.5)
++    scp = pw.reindex(columns=scmat.columns).copy()
++    for v in ["polity_population", "polity_territory", "population_of_the_largest_settlement"]:
++        scp[v] = plog.get(v)
++    Zc = (scp - scmat.mean()) / scmat.std(ddof=0)
++    cr["complexity_pc1"] = (Zc.fillna(0) * C_load).sum(axis=1) / (Zc.notna() * C_load.abs()).sum(axis=1).replace(0, np.nan)
++    cr.loc[Zc.notna().sum(axis=1) < 5, "complexity_pc1"] = np.nan
++    crisis = pd.read_parquet(PROC / "seshat_crisis.parquet").set_index("crisis_id")
++    cr["polity_id"] = crisis.polity_id; cr["crisis_start"] = crisis.crisis_start
++    cr = cr.join(meta[["macro_region", "era", "seshat_region", "homeNGA", "start_year"]], on="polity_id")
++    cr["years_since_start"] = cr.crisis_start - cr.start_year
++    cr.reset_index().to_parquet(PROC / "seshat_crisis_indices.parquet", index=False)
++    # ---- diagnostics: collinearity, comparability, measurement error ----
++    diag = {"D_pc1_loadings": D_load.round(4).to_dict(), "D_pc1_var_share": round(D_var, 4),
++            "K_pc1_loadings": K_load.round(4).to_dict(), "K_pc1_var_share": round(K_var, 4),
++            "complexity_pc1_var_share": round(C_var, 4), "complexity_pc1_n_vars": int(scmat.shape[1]),
++            "irt_loglik": irt["loglik"], "irt_iterations": irt["iterations"],
++            "n_D_pc1": int(out.D_pc1.notna().sum()), "n_K_mean": int(out.K_mean.notna().sum()), "n_K_irt": int(out.K_irt.notna().sum()),
++            "n_P_main": int(out.P_main.notna().sum()), "n_P_main_crisis": int(cr.P_main.notna().sum()),
++            "corr_zD_zK": {"D_pc1~K_mean": round(out.D_pc1.corr(out.K_mean), 4), "D_pc1~K_irt": round(out.D_pc1.corr(out.K_irt), 4),
++                            "log_pop~K_mean": round(out.log_pop.corr(out.K_mean), 4), "K_mean~K_irt": round(out.K_mean.corr(out.K_irt), 4),
++                            "K_mean~K_pc1": round(out.K_mean.corr(out.K_pc1), 4), "P_main~complexity_pc1": round(out.P_main.corr(out.complexity_pc1), 4),
++                            "P_main~D_pc1": round(out.P_main.corr(out.D_pc1), 4), "P_main~K_mean": round(out.P_main.corr(out.K_mean), 4)},
++            "P_variance_share_of_D_plus_K": round(float(out.P_main.var() / (z(out.D_pc1).var() + z(out.K_mean).var())), 4),
++            "block_correlations": blocks.corr().round(3).to_dict(),
++            "block_alpha_by_macro_region": {}, "block_corr_mean_by_era": {},
++            "measurement_error": {"D_pc1_halfwidth_median": float(((out.D_pc1_hi - out.D_pc1_lo) / 2).median()),
++                                  "D_pc1_halfwidth_q90": float(((out.D_pc1_hi - out.D_pc1_lo) / 2).quantile(0.9)),
++                                  "K_irt_psd_median": float(out.K_irt_psd.median()), "K_irt_psd_q90": float(out.K_irt_psd.quantile(0.9)),
++                                  "sd_K_irt": float(out.K_irt.std()), "P_main_range_median": float((out.P_main_hi - out.P_main_lo).abs().median())}}
++    for r, g in df.groupby("macro_region"):
++        b = g[[f"K_{k}" for k in K_BLOCKS]].dropna()
++        if len(b) >= 8:
++            k = b.shape[1]; alpha = k / (k - 1) * (1 - b.var(ddof=1).sum() / b.sum(axis=1).var(ddof=1))
++            diag["block_alpha_by_macro_region"][r] = {"n": int(len(b)), "alpha": round(float(alpha), 3)}
++    for e, g in df.groupby("era"):
++        b = g[[f"K_{k}" for k in K_BLOCKS]]
++        c = b.corr().values; diag["block_corr_mean_by_era"][e] = {"n": int(b.dropna().shape[0]), "mean_offdiag_corr": round(float(c[np.triu_indices(4, 1)].mean()), 3)}
++    # IRT invariance: refit on Old World vs Americas/Oceania and compare difficulties
++    new_world = df.macro_region.isin(["North_America", "South_America", "Southeast_Asia_Oceania"])
++    inv = {}
++    for name, mask in [("old_world", ~new_world), ("new_world_oceania", new_world)]:
++        f = fit_2pl(Xb[mask.values].values); inv[name] = {"n": int(mask.sum()), "b": f["b"].round(3).tolist(), "a": f["a"].round(3).tolist()}
++    bo, bn = np.array(inv["old_world"]["b"]), np.array(inv["new_world_oceania"]["b"])
++    ok = np.isfinite(bo) & np.isfinite(bn) & (np.abs(bo) < 5.9) & (np.abs(bn) < 5.9)
++    diag["irt_invariance"] = {"items_compared": int(ok.sum()), "corr_difficulty_old_vs_new": round(float(np.corrcoef(bo[ok], bn[ok])[0, 1]), 3),
++                              "mean_abs_diff_difficulty": round(float(np.abs(bo[ok] - bn[ok]).mean()), 3), "items": BIN_K, "fits": inv}
++    # leave-one-block-out sensitivity of K and P
++    loo = {}
++    for b in K_BLOCKS:
++        others = [f"K_{k}" for k in K_BLOCKS if k != b]; kk = out[others].mean(axis=1).where(out[others].notna().sum(axis=1) >= 2)
++        loo[b] = {"corr_K_with_full": round(float(kk.corr(out.K_mean)), 4), "corr_P_with_main": round(float((z(out.D_pc1) - z(kk)).corr(out.P_main)), 4)}
++    diag["leave_one_block_out"] = loo
++    (RES / "measurement_diagnostics.json").write_text(json.dumps(diag, indent=2))
++    # ---- figures ----
++    fig, axes = plt.subplots(1, 3, figsize=(12, 3.8))
++    axes[0].scatter(out.K_mean, out.D_pc1, s=8, alpha=0.5, c="#444"); axes[0].set_xlabel("K (mean of 4 blocks)"); axes[0].set_ylabel("D (PC1 of log scale variables)"); axes[0].set_title(f"r = {out.D_pc1.corr(out.K_mean):.2f}, n = {int((out.D_pc1.notna() & out.K_mean.notna()).sum())}", fontsize=9)
++    axes[1].scatter(out.K_mean, out.K_irt, s=8, alpha=0.5, c="#444"); axes[1].set_xlabel("K mean"); axes[1].set_ylabel("K IRT (2PL EAP)"); axes[1].set_title(f"r = {out.K_mean.corr(out.K_irt):.2f}", fontsize=9)
++    axes[2].errorbar(item_params.b, range(len(item_params)), xerr=None, fmt="o", ms=3, color="#444"); axes[2].set_yticks(range(len(item_params))); axes[2].set_yticklabels(item_params.item, fontsize=5); axes[2].set_xlabel("IRT difficulty b"); axes[2].set_title("Item difficulties", fontsize=9)
++    fig.tight_layout(); fig.savefig(FIG / "fig_measurement_DK.pdf"); plt.close(fig)
++    fig, ax = plt.subplots(figsize=(4.5, 4.2)); ax.scatter(bo[ok], bn[ok], s=12, c="#444"); lim = [min(bo[ok].min(), bn[ok].min()) - .3, max(bo[ok].max(), bn[ok].max()) + .3]; ax.plot(lim, lim, "k--", lw=.8)
++    ax.set_xlabel("difficulty, Old World fit"); ax.set_ylabel("difficulty, Americas/Oceania fit"); ax.set_title(f"IRT item invariance (r = {diag['irt_invariance']['corr_difficulty_old_vs_new']:.2f})", fontsize=9)
++    fig.tight_layout(); fig.savefig(FIG / "fig_irt_invariance.pdf"); plt.close(fig)
++    return out, cr, diag
++
++
++def build_modern():
++    p = pd.read_parquet(PROC / "modern_panel.parquet")
++    items = p[K_ITEMS_MODERN]
++    Zi = (items - items.mean()) / items.std(ddof=0)
++    p["K_modern"] = Zi.mean(axis=1).where(Zi.notna().sum(axis=1) >= 4)
++    p["K_modern_pc1"], Km_load, Km_var = pc1_pairwise(items, min_coded=4)
++    p["D_modern"] = p.log_pop
++    p["P_modern"] = z(p.D_modern) - z(p.K_modern)
++    p["P_modern_pc1"] = z(p.D_modern) - z(p.K_modern_pc1)
++    p["P_modern_within_year"] = p.groupby("year").D_modern.transform(z) - p.groupby("year").K_modern.transform(z)
++    p["P_modern_area"] = z(((z(p.log_pop) + z(p.log_area)) / 2)) - z(p.K_modern)
++    p.to_parquet(PROC / "modern_indices.parquet", index=False)
++    diag = {"K_modern_pc1_loadings": Km_load.round(4).to_dict(), "K_modern_pc1_var_share": round(Km_var, 4),
++            "n_P_modern": int(p.P_modern.notna().sum()), "corr_D_K_modern": round(float(p.D_modern.corr(p.K_modern)), 4),
++            "corr_K_modern_polyarchy": round(float(p.K_modern.corr(p.v2x_polyarchy)), 4),
++            "item_corr": items.corr().round(3).to_dict(),
++            "corr_K_by_region": {int(r): round(float(g.D_modern.corr(g.K_modern)), 3) for r, g in p.groupby("e_regionpol_6C") if g.P_modern.notna().sum() > 50}}
++    (RES / "measurement_diagnostics_modern.json").write_text(json.dumps(diag, indent=2))
++    return p, diag
++
++
++def power_h2a(cr: pd.DataFrame, n_rep: int = 400):
++    """A-priori MDE for H2-A: random outcomes assigned to the actual P distribution of crisis records (no real Y used)."""
++    rng = np.random.default_rng(SEED)
++    d = cr.dropna(subset=["P_main"]).copy()
++    P = z(d.P_main).values; reg = pd.factorize(d.seshat_region)[0]; n = len(P)
++    base = 0.77  # Y1b positive share read from coverage_summary (122/158); used only to set the intercept
++    res = []
++    for beta in [0.2, 0.3, 0.4, 0.5, 0.6, 0.8, 1.0]:
++        rej = 0
++        for _ in range(n_rep):
++            a0 = np.log(base / (1 - base))
++            y = rng.random(n) < 1 / (1 + np.exp(-(a0 + beta * P)))
++            try:
++                m = sm.Logit(y.astype(float), sm.add_constant(P)).fit(disp=0, cov_type="cluster", cov_kwds={"groups": reg})
++                rej += m.pvalues[1] < 0.05
++            except Exception:
++                pass
++        res.append({"beta": beta, "power": rej / n_rep, "n": n})
++    res = pd.DataFrame(res); res.to_csv(RES / "power_h2a_binary.csv", index=False)
++    # fractional outcome: Binomial(14, logistic(a0 + beta P)) with base severity 0.29
++    res2 = []
++    for beta in [0.1, 0.15, 0.2, 0.3, 0.4, 0.5]:
++        rej = 0
++        for _ in range(n_rep):
++            a0 = np.log(0.29 / 0.71); pr = 1 / (1 + np.exp(-(a0 + beta * P))); y = rng.binomial(14, pr) / 14
++            try:
++                m = sm.GLM(y, sm.add_constant(P), family=sm.families.Binomial()).fit(cov_type="cluster", cov_kwds={"groups": reg})
++                rej += m.pvalues[1] < 0.05
++            except Exception:
++                pass
++        res2.append({"beta": beta, "power": rej / n_rep, "n": n})
++    res2 = pd.DataFrame(res2); res2.to_csv(RES / "power_h2a_fractional.csv", index=False)
++    return res, res2
++
++
++def main():
++    out, cr, diag = build_seshat()
++    p, dm = build_modern()
++    pw1, pw2 = power_h2a(cr)
++    print(json.dumps({k: v for k, v in diag.items() if k in ("D_pc1_loadings", "D_pc1_var_share", "K_pc1_loadings", "n_D_pc1", "n_K_mean", "n_K_irt", "n_P_main", "n_P_main_crisis", "corr_zD_zK", "P_variance_share_of_D_plus_K", "measurement_error", "irt_invariance", "leave_one_block_out", "block_alpha_by_macro_region")}, indent=1, default=str)[:6000])
++    print(json.dumps({k: v for k, v in dm.items() if k != "item_corr"}, indent=1))
++    print("power binary:\n", pw1.to_string(index=False)); print("power fractional:\n", pw2.to_string(index=False))
++
++
++if __name__ == "__main__":
++    main()

added src/wp35/measurement/irt.py +63 −0

++"""Two-parameter logistic IRT (marginal maximum likelihood, EM with Gauss–Hermite quadrature).
++
++Items are binary (0/1); missing entries contribute nothing to the likelihood (missing-at-random by design, no
++imputation). Returns item discriminations a_j, difficulties b_j, EAP scores and posterior SDs for each unit.
++Deterministic (no randomness). Written for transparency rather than speed (N ≈ 600 × J ≈ 30 is instant).
++"""
++from __future__ import annotations
++import numpy as np
++from scipy.special import expit
++
++
++def fit_2pl(X: np.ndarray, n_quad: int = 41, max_iter: int = 1000, tol: float = 1e-6, min_items: int = 3,
++            prior_sd_log_a: float = 0.5, prior_sd_b: float = 2.0):
++    """MAP variant: log-normal prior on a (mean 0, sd prior_sd_log_a) and normal prior on b (sd prior_sd_b) to keep
++    items that are nearly constant in the sample from running to the boundary (Heywood-type cases)."""
++    X = np.asarray(X, dtype=float)
++    N, J = X.shape
++    obs = ~np.isnan(X)
++    Xz = np.where(obs, X, 0.0)
++    nodes, weights = np.polynomial.hermite_e.hermegauss(n_quad)  # standard normal prior
++    weights = weights / weights.sum()
++    a = np.ones(J); b = np.zeros(J)
++    ll_old = -np.inf
++    for it in range(max_iter):
++        # E-step: posterior over nodes for each unit
++        P = expit(a[None, :] * (nodes[:, None] - b[None, :]))          # Q x J
++        logP, log1P = np.log(P + 1e-300), np.log(1 - P + 1e-300)
++        # log-lik of unit i at node q = sum_j obs_ij [x_ij logP_qj + (1-x_ij) log(1-P_qj)]
++        L = Xz @ logP.T + ((1 - Xz) * obs) @ log1P.T                   # N x Q
++        L += np.log(weights)[None, :]
++        m = L.max(axis=1, keepdims=True)
++        post = np.exp(L - m); norm = post.sum(axis=1, keepdims=True); post /= norm
++        ll = float((m.squeeze() + np.log(norm.squeeze())).sum())
++        # M-step: Newton updates per item on expected counts
++        r = post.T @ Xz                                                 # Q x J expected correct
++        n = post.T @ obs.astype(float)                                  # Q x J expected tries
++        for j in range(J):
++            for _ in range(10):
++                p = expit(a[j] * (nodes - b[j]))
++                g_a = np.sum((r[:, j] - n[:, j] * p) * (nodes - b[j])) - (np.log(a[j]) / prior_sd_log_a ** 2) / a[j]
++                g_b = np.sum((r[:, j] - n[:, j] * p) * (-a[j])) - b[j] / prior_sd_b ** 2
++                w = n[:, j] * p * (1 - p)
++                H_aa = -np.sum(w * (nodes - b[j]) ** 2) - (1 - np.log(a[j])) / (prior_sd_log_a ** 2 * a[j] ** 2)
++                H_bb = -np.sum(w * a[j] ** 2) - 1 / prior_sd_b ** 2
++                H_ab = np.sum(w * a[j] * (nodes - b[j]))
++                H = np.array([[H_aa, H_ab], [H_ab, H_bb]]) - 1e-6 * np.eye(2)
++                step = np.linalg.solve(H, np.array([g_a, g_b]))
++                a[j] -= step[0]; b[j] -= step[1]
++                a[j] = float(np.clip(a[j], 0.05, 8.0)); b[j] = float(np.clip(b[j], -6, 6))
++                if np.abs(step).max() < 1e-6:
++                    break
++        if abs(ll - ll_old) < tol:
++            break
++        ll_old = ll
++    # final EAP
++    P = expit(a[None, :] * (nodes[:, None] - b[None, :]))
++    L = Xz @ np.log(P + 1e-300).T + ((1 - Xz) * obs) @ np.log(1 - P + 1e-300).T + np.log(weights)[None, :]
++    m = L.max(axis=1, keepdims=True); post = np.exp(L - m); post /= post.sum(axis=1, keepdims=True)
++    eap = post @ nodes
++    psd = np.sqrt(np.maximum(post @ nodes ** 2 - eap ** 2, 0))
++    n_items = obs.sum(axis=1)
++    eap = np.where(n_items >= min_items, eap, np.nan); psd = np.where(n_items >= min_items, psd, np.nan)
++    return {"a": a, "b": b, "eap": eap, "psd": psd, "loglik": ll, "iterations": it + 1, "n_items": n_items}

				`@@ -47,3 +47,5 @@ marzen2017\|10.1098/rsif.2017.0166\|Marzen & DeDeo 2017 J R Soc Interface — adde`
47	flack2012\|10.1098/rstb.2011.0214\|Flack 2012 Phil Trans B — added after novelty search	47	flack2012\|10.1098/rstb.2011.0214\|Flack 2012 Phil Trans B — added after novelty search
48	hoyer2024\|10.31235/osf.io/rk4gd\|Hoyer et al. 2024 SocArXiv preprint 'All crises are unhappy in their own way' — no journal version found 2026-11-01	48	hoyer2024\|10.31235/osf.io/rk4gd\|Hoyer et al. 2024 SocArXiv preprint 'All crises are unhappy in their own way' — no journal version found 2026-11-01
49	wolpert2017\|10.1017/9781316584200.010\|Wolpert 2017 'The many faces of state space compression' (same volume)	49	wolpert2017\|10.1017/9781316584200.010\|Wolpert 2017 'The many faces of state space compression' (same volume)
		50	+lindley1952\|10.1017/s0305004100027638\|Lindley 1952 Math. Proc. Cambridge Phil. Soc. 48 — verified by Crossref (used in framework.md Theorem 5)
		51	+loynes1962\|10.1017/s0305004100036781\|Loynes 1962 Math. Proc. Cambridge Phil. Soc. 58 — verified by Crossref

spb/wp35_uqo

WP35 Phase 3: formal framework (5 theorems, 3 conjectures), D/K/P indices with IRT alternative, measurement diagnostics, a-priori power

Theme