WP28: displayed equations throughout, GPT-6.1 author mention removed, FINAL_REVIEW draft
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
17 changed files +19,767 −56
added
FINAL_REVIEW.md
+67 −0
@@ -0,0 +1,67 @@ | |||||
| 1 | +# FINAL_REVIEW — adversarial referee audit (Phase 11) | ||||
| 2 | + | ||||
| 3 | +Three simulated review perspectives were applied to the manuscript and code. Each objection records (a) location, (b) correction made, | ||||
| 4 | +(c) new proof/test if any, (d) residual limitation. Unresolved items are listed candidly at the end with the verdict. | ||||
| 5 | + | ||||
| 6 | +## Referee A — theoretical econometrician | ||||
| 7 | + | ||||
| 8 | +A1. *"The centered random remainder of the jackknife is dropped in the prior draft's higher-order corollary."* (a) Prior draft Cor. 6.1; paper §4.2, Lemma 4. (b) Lemma 3–4 bound the remainder with the explicit constant A_r = Σ|w_j|s_j and carry it through Theorem 1; the proof shows it never binds for q ≥ 1 and binds only via K = o(N) for q = 0. (c) tests/test_estimators.py::test_pilot_bias_orders_exponential; symbolic S8. (d) none. | ||||
| 9 | + | ||||
| 10 | +A2. *"Uniformity along the triangular array: Assumption A1 must hold with a common constant for every sub-block size; is that realistic?"* (a) §3.3, ASSUMPTIONS.md. (b) The assumption is stated for the sub-sample family with n ≥ n₀ and a common C; it is verified exactly in the exponential model (C = 33θ0²/2, n₀ = 3, App. B) and discussed as implied by standard primitive conditions. (c) symbolic S2. (d) Primitive conditions for general models are cited (ZDW/HH-type moment conditions + validity of the formal expansion, Bhattacharya–Ghosh) but not re-derived; stated as such in §3.3. | ||||
| 11 | + | ||||
| 12 | +A3. *"Bias cancellation of expected coefficients does not give an L2 statement."* (a) §4.2. (b) Lemma 2 (mean) and Lemma 3 (L2) are separate; Lemma 5 combines them. (d) none. | ||||
| 13 | + | ||||
| 14 | +A4. *"Constants may diverge with p or with the order r."* (a) §3.3 remark, §11. (b) Stated explicitly: fixed p only; A_r and κ(V) tabulated (Table 2) and the conditioning discussion in §4.3. (d) No growing-p theorem; said so. | ||||
| 15 | + | ||||
| 16 | +A5. *"Is K = o(N^α) interpreted as a machine count?"* (b) §5.2 and the remark after Assumption 4: triangular-array statement, not a fixed-N machine limit; Theorem 2(vi) shows what happens at and beyond the boundary. (d) none. | ||||
| 17 | + | ||||
| 18 | +A6. *"Necessity: the sharpness is for one model and one estimator family."* (a) §6. (b) Stated in §6 and §11; Volgushev et al. cited as the template; exact pooling counterexample given. (d) No worst-case construction over a model class for general likelihoods (as Volgushev et al. do for QR); left open. | ||||
| 19 | + | ||||
| 20 | +A7. *"Coverage claims require a consistent variance estimator; the point-estimator result is not enough."* (b) Corollary 1 states the requirement and the extra p(p+1)/2 upload for the sandwich; Monte Carlo coverage uses the observed information at the estimate, and the misspecified design compares Fisher vs sandwich at the centralized estimate. (d) Sandwich-based coverage of the distributed estimators themselves is only computed at the centralized point in the current driver (se_sand_central); noted in the Monte Carlo section. | ||||
| 21 | + | ||||
| 22 | +A8. *"The boundary regime m ≍ N^{1/8} is unreachable; the −4θ0c⁴ limit is of theoretical interest only."* (b) Agreed and stated in §6 and in the boundary table; a finite-m prediction (exact bias + remainder variance) is reported and matches the Monte Carlo to three digits (Table boundary). (d) The (3,1) boundary (m ≍ N^{1/24}) is not approached at any feasible N; reported, not hidden. | ||||
| 23 | + | ||||
| 24 | +A9. *"Theorem 3(a) needs third derivatives; the stale-Hessian exponent is conditional on A4."* (b) A4 stated; the proof is explicit about where it is used. (d) A4's fourth-moment block condition is assumed, not derived from primitives. | ||||
| 25 | + | ||||
| 26 | +## Referee B — statistical optimization specialist | ||||
| 27 | + | ||||
| 28 | +B1. *"Huang–Huo already prove the one-step estimator; what is new?"* (a) §2, Table 1. (b) HH19's stated regime is k = O(√N) from the plain average; their k⁴N⁻⁴ term implies 3/4 (not stated by them); the jackknife start and the joint frontier (7/8, 11/12, general α(r,q)) are not in HH19; said explicitly in §1 ("What is not claimed") and §2. (d) none. | ||||
| 29 | + | ||||
| 30 | +B2. *"Zhu–Li–Wang's TWLSE reaches 3/4 with one local step; CEASE gives a rounds frontier; Gu–Chen reach 2/3 with analytic debiasing."* (b) All cited at theorem level (Table 1); Theorem 3(b) recovers CEASE's (t+1)/(t+2) and JLY's 2/3 exactly, which is presented as a consistency check rather than a new result. (d) none. | ||||
| 31 | + | ||||
| 32 | +B3. *"Why nested doubling splits? Dhaene–Jochmans show {2,3,…} is bias-optimal."* (b) §4.3 and Table 2 compare (1,2,3) and (1,2,4): smaller |β₃| (6 vs 8) but larger A₃ (12 vs 8) and a coarser finest block; the ablation evaluates both patterns numerically (Table abl). (d) No optimality theorem for patterns; the choice is empirical. | ||||
| 33 | + | ||||
| 34 | +B4. *"Split-sample corrections have twice the higher-order variance of delete-one (Hahn et al. 2024)."* (b) Cited in §2 and §11; exact Var J₂ in the exponential model shows the cost is O(m⁻³) there; Monte Carlo RMSE tables show the finite-sample ranking reversals. (d) No general higher-order variance expansion for the nested construction. | ||||
| 35 | + | ||||
| 36 | +B5. *"Newton steps are undamped; production code needs safeguards; the basin result shows divergence with positive probability."* (b) Theorem 2(iv) and the failure gallery; the library solves linear systems (no inverse), flags singular curvature, and the Monte Carlo reports failed replications rather than discarding them; a damped variant is not analysed (said in §6). (d) The theory is for the undamped map only. | ||||
| 37 | + | ||||
| 38 | +B6. *"CSL baseline: is the implementation faithful?"* (b) The full surrogate maximization (not only the quadratic step) is implemented with Newton on machine 1 and the exact global gradient; in the exponential model the surrogate maximizer equals the centralized MLE exactly (proved in the test), which is reported as a property of that model, not as a weakness of CSL. (d) CSL's proximal/regularized variants (CEASE) are not implemented; stated. | ||||
| 39 | + | ||||
| 40 | +B7. *"Quantization: relative rounding of Hessian entries is not a realistic compression scheme."* (b) Theorem 3(c) is stated for relative precision ε on every transmitted entry, which upper-bounds any scheme with that relative accuracy; the ablation implements exactly that. (d) Lattice/stochastic quantizers (Alimisis et al.) are not analysed. | ||||
| 41 | + | ||||
| 42 | +B8. *"The cost model is not a theorem."* (b) Proposition 5 is labelled conditional on the cost model; §9 says so. (d) none. | ||||
| 43 | + | ||||
| 44 | +## Referee C — distributed-systems / reproducibility specialist | ||||
| 45 | +(to be completed after the benchmark runs; see the "Benchmarks" entry in PHASE_STATUS.md) | ||||
| 46 | + | ||||
| 47 | +C1. *"In-memory partitions are not a cluster."* (b) §10 only reports runs of the TCP coordinator–worker protocol on physical hosts; Monte Carlo sections state that they are in-memory (no communication measured). | ||||
| 48 | +C2. *"Two communication phases are not one message."* (b) §3.2 counts phases, messages, numbers and bytes separately (Table 2); measured bytes include headers. | ||||
| 49 | +C3. *"Stragglers, start-up and scheduler delay."* (b) The protocol logs per-worker fit/derivative times and p50/p90/p99 reply latencies; benchmark tables report the slowest worker and latency percentiles. | ||||
| 50 | +C4. *"K partitions ≠ K hosts."* (b) Designs include 16 workers on 1, 2, 4, 8, 16 hosts and 64/128 workers on 16 hosts. | ||||
| 51 | +C5. *"Raw-data reproducibility."* (b) Every raw file has a manifest (git commit, versions, host, SHA-256); EXPERIMENT_MANIFEST.csv; seeds derived from a hash of the cell description; the prior draft's numbers are reproduced by a test. | ||||
| 52 | + | ||||
| 53 | +## Formal audit checklist | ||||
| 54 | +- [x] Every theorem's assumptions enumerated (ASSUMPTIONS.md, §3.3). | ||||
| 55 | +- [x] No proof relies on an unproved stronger bound (THEOREM_AUDIT.md ledger). | ||||
| 56 | +- [x] Expansions at m/2, m/4 valid for the stated m range (n₀ in A1; m/s_r ≥ n₀). | ||||
| 57 | +- [x] K = o(N^α) interpreted as a triangular-array rate. | ||||
| 58 | +- [x] Boundary constants independently checked (SymPy S5; tests; Monte Carlo + finite-m prediction). | ||||
| 59 | +- [x] Every table value has a traceable input (make_tables.py from results/). | ||||
| 60 | +- [x] Comparisons include ordinary averaging + two Newton rounds and exact sufficient-statistic pooling (exponential). | ||||
| 61 | +- [x] Communication savings distinguish bytes and rounds (Table 2; benchmarks). | ||||
| 62 | +- [x] Paper states openly which results are new, which are known, and that no lower bound is claimed. | ||||
| 63 | +- [x] AI contribution disclosure explicit (page 3). | ||||
| 64 | +- [ ] Paper/source sync and PDF render check — see FINAL_DELIVERY.md. | ||||
| 65 | + | ||||
| 66 | +## Verdict | ||||
| 67 | +(filled at delivery) | ||||
modified
LIMITATIONS_AND_CLAIMS.md
+1 −1
@@ -27,7 +27,7 @@ | |||||
| 27 | - Not a new estimator: the components (generalized jackknife; distributed one-step Newton) are known and cited. | 27 | - Not a new estimator: the components (generalized jackknife; distributed one-step Newton) are known and cited. | ||
| 28 | - Not a wall-clock or energy advantage in general; not a byte-count optimality. | 28 | - Not a wall-clock or energy advantage in general; not a byte-count optimality. | ||
| 29 | - Not valid for p growing with N, nor for a fixed data set with "the maximal number of machines". | 29 | - Not valid for p growing with N, nor for a fixed data set with "the maximal number of machines". | ||
| 30 | −- The "GPT-6.1" co-author designation is provenance of the prior draft at the researcher's request, not a legally accountable author; see the disclosure in the paper. | 30 | +- The prior note's "GPT-6.1" co-author designation is not carried into this paper (removed at the author's request); AI assistance is disclosed in the paper's provenance note. | ||
| 31 | 31 | ||||
| 32 | ## Replication-count statement | 32 | ## Replication-count statement | ||
| 33 | Prior draft: 500 replications × 3 designs (exact-Gamma, N = 2^20). This paper: exponential grid 2000 replications per cell (40 cells), logistic/Poisson 1000 (500 for p = 20 and the ill-conditioned/small-m designs), boundary study 2000/1000/400 by N, ablations 1000 (exponential) and 400 (logistic), skewed partitions 1000/300, benchmarks 3 repeats per design. Actual counts are recorded per cell in `experiments/EXPERIMENT_MANIFEST.csv` and in every table note. | 33 | Prior draft: 500 replications × 3 designs (exact-Gamma, N = 2^20). This paper: exponential grid 2000 replications per cell (40 cells), logistic/Poisson 1000 (500 for p = 20 and the ill-conditioned/small-m designs), boundary study 2000/1000/400 by N, ablations 1000 (exponential) and 400 (logistic), skewed partitions 1000/300, benchmarks 3 repeats per design. Actual counts are recorded per cell in `experiments/EXPERIMENT_MANIFEST.csv` and in every table note. | ||
modified
README.md
+2 −2
@@ -22,7 +22,7 @@ | |||||
| 22 | | SSRN | not yet submitted | | 22 | | SSRN | not yet submitted | | ||
| 23 | | Journal | — (candidates: Journal of Econometrics, Econometric Theory, JASA T&M, JCGS) | | 23 | | Journal | — (candidates: Journal of Econometrics, Econometric Theory, JASA T&M, JCGS) | | ||
| 24 | | Code & data | fully reproducible from `experiments/` (synthetic data only); raw Monte Carlo outputs committed in `results/raw/` with manifests | | 24 | | Code & data | fully reproducible from `experiments/` (synthetic data only); raw Monte Carlo outputs committed in `results/raw/` with manifests | | ||
| 25 | −| Provenance | develops the exploratory note "Local Bias Correction versus Global Synchronization in Distributed Maximum Likelihood" (Boucher & "GPT-6.1", 2026-10-28), frozen in `references/initial_pamle/`; see `SOURCE_PROVENANCE.md` and the disclosure on page 3 of the paper | | 25 | +| Provenance | develops the exploratory note "Local Bias Correction versus Global Synchronization in Distributed Maximum Likelihood" (Boucher & "GPT-6.1", 2026-10-28), frozen in `references/initial_pamle/`; see `SOURCE_PROVENANCE.md` and the disclosure following the abstract | | ||
| 26 | 26 | ||||
| 27 | ## 🎯 What this paper does | 27 | ## 🎯 What this paper does | ||
| 28 | 28 | ||||
@@ -131,5 +131,5 @@ See `LIMITATIONS_AND_CLAIMS.md`. Fixed $p$ and i.i.d. data only; sufficient expo | |||||
| 131 | ## 🗒 Notes | 131 | ## 🗒 Notes | ||
| 132 | 132 | ||||
| 133 | - Numbering: No. 28 is the next free number of the UQO series (`Recherche-UQO/README.md` stops at 27). The SPID database paper in `finance/russell3000-strategic-project-database` carries FIRC series No. 0001, not 28; see `SOURCE_PROVENANCE.md`. | 133 | - Numbering: No. 28 is the next free number of the UQO series (`Recherche-UQO/README.md` stops at 27). The SPID database paper in `finance/russell3000-strategic-project-database` carries FIRC series No. 0001, not 28; see `SOURCE_PROVENANCE.md`. | ||
| 134 | −- Authorship: the title page lists Simon-Pierre Boucher and preserves the "GPT-6.1" AI co-author designation of the prior draft, with a disclosure paragraph; for a journal submission the designation should move to the disclosure (flagged, not silently changed). | 134 | +- Authorship: the title page lists Simon-Pierre Boucher only; a provenance and AI-use disclosure follows the abstract (the prior note's AI co-author designation was removed at the author's request on 2026-10-28). | ||
| 135 | - Deviation from the template: `paper/main.tex` adds amsthm, algorithmic and TikZ packages; `sections/titlepage.tex` is the exact template copy. | 135 | - Deviation from the template: `paper/main.tex` adds amsthm, algorithmic and TikZ packages; `sections/titlepage.tex` is the exact template copy. | ||
added
experiments/fig_exponents.py
+15 −0
@@ -0,0 +1,15 @@ | |||||
| 1 | +#!/usr/bin/env python3 | ||||
| 2 | +"""Figure: sufficient exponents alpha(r,q) (exact curvature) and the stale / single-machine variants (Theorems 1 and 3).""" | ||||
| 3 | +import sys, pathlib | ||||
| 4 | +import matplotlib; matplotlib.use("Agg"); import matplotlib.pyplot as plt | ||||
| 5 | +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1] / "experiments")); from process_results import _setup, PAL, INK2, FIG | ||||
| 6 | +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1] / "src")); from pamle.communication import sufficient_exponent, sufficient_exponent_stale, sufficient_exponent_single | ||||
| 7 | +_setup(); fig, axes = plt.subplots(1, 3, figsize=(8.4, 2.8), sharey=True) | ||||
| 8 | +rs = [1, 2, 3, 4]; cols = ["blue", "orange", "aqua", "violet"]; mks = ["o", "s", "^", "D"] | ||||
| 9 | +for ax, (fn, title) in zip(axes, [(sufficient_exponent, "exact aggregated curvature"), (sufficient_exponent_stale, "stale averaged local curvature"), (sufficient_exponent_single, "single-machine curvature")]): | ||||
| 10 | + for q, c, mk in zip(range(4), cols, mks): | ||||
| 11 | + ax.plot(rs, [fn(r, q) for r in rs], marker=mk, color=PAL[c], label=f"q = {q}", markerfacecolor="white", markeredgewidth=1.2) | ||||
| 12 | + ax.set_xticks(rs); ax.set_xlabel("local order r"); ax.set_title(title, fontsize=8, loc="left"); ax.set_ylim(0.45, 1.0) | ||||
| 13 | + ax.axhline(0.5, color=INK2, lw=0.6, ls=":"); | ||||
| 14 | +axes[0].set_ylabel("sufficient exponent α in K = o(N^α)"); axes[0].legend(fontsize=7, loc="lower right") | ||||
| 15 | +FIG.mkdir(parents=True, exist_ok=True); fig.savefig(FIG / "fig_exponents.pdf"); print("ok") | ||||
modified
experiments/run_skewed.py
+11 −1
@@ -34,7 +34,13 @@ def run(model_name, p, N, K, ratio, reps, theta0, seed, out): | |||||
| 34 | g = np.zeros(p); H = np.zeros((p, p)) | 34 | g = np.zeros(p); H = np.zeros((p, p)) | ||
| 35 | for k, idx in enumerate(blocks): | 35 | for k, idx in enumerate(blocks): | ||
| 36 | gk, Hk = model.score_hessian(th[None], {kk: v[idx][None] for kk, v in data.items()}); g += sizes[k] / N_eff * gk[0]; H += sizes[k] / N_eff * Hk[0] | 36 | gk, Hk = model.score_hessian(th[None], {kk: v[idx][None] for kk, v in data.items()}); g += sizes[k] / N_eff * gk[0]; H += sizes[k] / N_eff * Hk[0] | ||
| 37 | − th = th + np.linalg.solve(-H, g) | 37 | + try: | ||
| 38 | + th = th + np.linalg.solve(-H, g) | ||||
| 39 | + except np.linalg.LinAlgError: # singular curvature at a wild pilot: record as failure | ||||
| 40 | + th = np.full(p, np.nan) | ||||
| 41 | + for qq in range(q + 1, 3): | ||||
| 42 | + res[f"gap_q{qq}"] = float("nan") | ||||
| 43 | + break | ||||
| 38 | rows.append(res) | 44 | rows.append(res) | ||
| 39 | if (b + 1) % max(1, reps // 5) == 0: | 45 | if (b + 1) % max(1, reps // 5) == 0: | ||
| 40 | print(f"[{model_name} K={K} ratio={ratio}] {b+1}/{reps} {time.time()-t0:.0f}s", flush=True); pd.DataFrame(rows).to_csv(out, index=False) | 46 | print(f"[{model_name} K={K} ratio={ratio}] {b+1}/{reps} {time.time()-t0:.0f}s", flush=True); pd.DataFrame(rows).to_csv(out, index=False) | ||
@@ -42,8 +48,12 @@ def run(model_name, p, N, K, ratio, reps, theta0, seed, out): | |||||
| 42 | 48 | ||||
| 43 | 49 | ||||
| 44 | if __name__ == "__main__": | 50 | if __name__ == "__main__": | ||
| 51 | + import sys | ||||
| 52 | + only = sys.argv[1] if len(sys.argv) > 1 else None | ||||
| 45 | (ROOT / "results" / "raw").mkdir(parents=True, exist_ok=True) | 53 | (ROOT / "results" / "raw").mkdir(parents=True, exist_ok=True) | ||
| 46 | for model_name, p, theta0 in (("exponential", 1, np.array([1.0])), ("logistic", 3, np.array([0.5, 1.0, -1.0]))): | 54 | for model_name, p, theta0 in (("exponential", 1, np.array([1.0])), ("logistic", 3, np.array([0.5, 1.0, -1.0]))): | ||
| 47 | for K, ratio in ((256, 1.0), (256, 4.0), (256, 16.0), (1024, 4.0), (1024, 16.0)): | 55 | for K, ratio in ((256, 1.0), (256, 4.0), (256, 16.0), (1024, 4.0), (1024, 16.0)): | ||
| 48 | seed = 20261028 + int(hashlib.sha256(f"skew {model_name} {K} {ratio}".encode()).hexdigest(), 16) % 10 ** 6 | 56 | seed = 20261028 + int(hashlib.sha256(f"skew {model_name} {K} {ratio}".encode()).hexdigest(), 16) % 10 ** 6 | ||
| 57 | + if only and only != f"{model_name}_K{K}_r{int(ratio)}": | ||||
| 58 | + continue | ||||
| 49 | run(model_name, p, 2 ** 18, K, ratio, 300 if model_name == "logistic" else 1000, theta0, seed, ROOT / "results" / "raw" / f"skew_{model_name}_K{K}_r{int(ratio)}.csv") | 59 | run(model_name, p, 2 ** 18, K, ratio, 300 if model_name == "logistic" else 1000, theta0, seed, ROOT / "results" / "raw" / f"skew_{model_name}_K{K}_r{int(ratio)}.csv") | ||
modified
paper/appendices/exponential.tex
+24 −3
@@ -3,7 +3,13 @@ | |||||
| 3 | Let $Y = \theta_0\bar X_n$, so that $nY \sim \mathrm{Gamma}(n,1)$, $\thetahat_n = \theta_0/Y$ and $R_n = \theta_0(1 - Y)^2/Y = \theta_0(Y^{-1} - 2 + Y)$. The Gamma moments $\E(nY)^{-k} = \Gamma(n-k)/\Gamma(n)$ ($k < n$) give $\E Y = 1$, $\E Y^2 = 1 + 1/n$, $\E Y^{-1} = n/(n-1)$ and $\E Y^{-2} = n^2/((n-1)(n-2))$. | 3 | Let $Y = \theta_0\bar X_n$, so that $nY \sim \mathrm{Gamma}(n,1)$, $\thetahat_n = \theta_0/Y$ and $R_n = \theta_0(1 - Y)^2/Y = \theta_0(Y^{-1} - 2 + Y)$. The Gamma moments $\E(nY)^{-k} = \Gamma(n-k)/\Gamma(n)$ ($k < n$) give $\E Y = 1$, $\E Y^2 = 1 + 1/n$, $\E Y^{-1} = n/(n-1)$ and $\E Y^{-2} = n^2/((n-1)(n-2))$. | ||
| 4 | 4 | ||||
| 5 | \begin{proof}[Proof of \cref{thm:exponential}(i)] | 5 | \begin{proof}[Proof of \cref{thm:exponential}(i)] | ||
| 6 | −$\E\thetahat_n = \theta_0\E Y^{-1} = \theta_0 n/(n-1)$ and $\Var\thetahat_n = \theta_0^2(\E Y^{-2} - (\E Y^{-1})^2) = \theta_0^2 n^2/((n-1)^2(n-2))$. Next $\E R_n = \theta_0(n/(n-1) - 2 + 1) = \theta_0/(n-1) = \theta_0\sum_{a\ge1}n^{-a}$, so $b_a = \theta_0$ for all $a$ and the remainder $\rho_n = \theta_0\sum_{a\ge r}n^{-a}$ satisfies $\norm{\rho_n} \le 2\theta_0 n^{-r}$ for $n \ge 2$. Expanding $(Y^{-1} - 2 + Y)^2 = Y^{-2} - 4Y^{-1} + 6 - 4Y + Y^2$, | 6 | +From the Gamma moments, | ||
| 7 | +\begin{align*} | ||||
| 8 | +\E\thetahat_n &= \theta_0\,\E Y^{-1} = \theta_0\,\frac{n}{n-1}, & | ||||
| 9 | +\Var\thetahat_n &= \theta_0^2\big(\E Y^{-2} - (\E Y^{-1})^2\big) = \theta_0^2\,\frac{n^2}{(n-1)^2(n-2)},\\ | ||||
| 10 | +\E R_n &= \theta_0\Big(\frac{n}{n-1} - 2 + 1\Big) = \frac{\theta_0}{n-1} = \theta_0\sum_{a\ge1}n^{-a}, | ||||
| 11 | +\end{align*} | ||||
| 12 | +so $b_a = \theta_0$ for all $a$ and the remainder $\rho_n = \theta_0\sum_{a\ge r}n^{-a}$ satisfies $\norm{\rho_n} \le 2\theta_0 n^{-r}$ for $n \ge 2$. Expanding $(Y^{-1} - 2 + Y)^2 = Y^{-2} - 4Y^{-1} + 6 - 4Y + Y^2$, | ||||
| 7 | \[ | 13 | \[ | ||
| 8 | \E R_n^2 = \theta_0^2\Big[\frac{n^2}{(n-1)(n-2)} - \frac{4n}{n-1} + 3 + \frac1n\Big] = \theta_0^2\,\frac{3n+2}{n(n-1)(n-2)}, | 14 | \E R_n^2 = \theta_0^2\Big[\frac{n^2}{(n-1)(n-2)} - \frac{4n}{n-1} + 3 + \frac1n\Big] = \theta_0^2\,\frac{3n+2}{n(n-1)(n-2)}, | ||
| 9 | \] | 15 | \] | ||
@@ -11,7 +17,16 @@ and $n^2\E R_n^2/\theta_0^2 = n(3n+2)/((n-1)(n-2))$ decreases from $33/2$ at $n | |||||
| 11 | \end{proof} | 17 | \end{proof} | ||
| 12 | 18 | ||||
| 13 | \begin{proof}[Proof of (ii)] | 19 | \begin{proof}[Proof of (ii)] | ||
| 14 | −A sub-block of level $j$ has $n = m/s_j$ observations, so its MLE has mean $\theta_0 n/(n-1) = \theta_0 m/(m - s_j)$. Hence $\E\Jr{r,k} = \theta_0\sum_j w_j m/(m-s_j)$ and $\E\Jr{r,k} - \theta_0 = \theta_0\sum_j w_j[m/(m-s_j) - 1] = \theta_0\sum_j w_j s_j/(m-s_j)$, using $\sum_j w_j = 1$. Expanding $s_j/(m - s_j) = \sum_{a\ge1}s_j^a m^{-a}$ and using \eqref{eq:vandermonde}, the terms $a < r$ vanish and the leading term is $\theta_0 m^{-r}\sum_j w_js_j^r = \betar\theta_0 m^{-r}$. For $s = (1,2)$: $\theta_0[2/(m-1) - 2/(m-2)] = -2\theta_0/((m-1)(m-2))$. For $(1,2,4)$: $\theta_0[(8/3)/(m-1) - 4/(m-2) + (4/3)/(m-4)] = 8\theta_0/((m-1)(m-2)(m-4))$. Both identities, and $\beta_2 = -2$, $\beta_3 = 8$, $\beta_4 = -64$, are verified in rational arithmetic in \texttt{theory/symbolic\_checks.py}. | 20 | +A sub-block of level $j$ has $n = m/s_j$ observations, so its MLE has mean $\theta_0 n/(n-1) = \theta_0 m/(m - s_j)$. Using $\sum_j w_j = 1$, | ||
| 21 | +\[ | ||||
| 22 | +\E\Jr{r,k} - \theta_0 = \theta_0\sum_{j=1}^r w_j\Big[\frac{m}{m-s_j} - 1\Big] = \theta_0\sum_{j=1}^r\frac{w_j\,s_j}{m-s_j} | ||||
| 23 | += \theta_0\sum_{a\ge1}\frac{1}{m^{a}}\sum_{j=1}^r w_j s_j^{\,a}, | ||||
| 24 | +\] | ||||
| 25 | +where the terms $a < r$ vanish by \eqref{eq:vandermonde} and the leading term is $\betar\theta_0 m^{-r}$. For the two default patterns, | ||||
| 26 | +\begin{align*} | ||||
| 27 | +s = (1,2):\qquad & \theta_0\Big[\frac{2}{m-1} - \frac{2}{m-2}\Big] = -\frac{2\theta_0}{(m-1)(m-2)},\\ | ||||
| 28 | +s = (1,2,4):\qquad & \theta_0\Big[\frac{8/3}{m-1} - \frac{4}{m-2} + \frac{4/3}{m-4}\Big] = \frac{8\theta_0}{(m-1)(m-2)(m-4)} . | ||||
| 29 | +\end{align*} Both identities, and $\beta_2 = -2$, $\beta_3 = 8$, $\beta_4 = -64$, are verified in rational arithmetic in \texttt{theory/symbolic\_checks.py}. | ||||
| 15 | \end{proof} | 30 | \end{proof} | ||
| 16 | 31 | ||||
| 17 | \begin{proof}[Proof of (iii)] | 32 | \begin{proof}[Proof of (iii)] | ||
@@ -19,7 +34,13 @@ Let $h = m/2$ and $S_A, S_B$ be the sums over the two halves, $S = S_A + S_B \si | |||||
| 19 | \end{proof} | 34 | \end{proof} | ||
| 20 | 35 | ||||
| 21 | \begin{proof}[Proof of (iv)] | 36 | \begin{proof}[Proof of (iv)] | ||
| 22 | −With $g_N(x) = 1/x - \bar X_N$ and $H_N(x) = -1/x^2$, $\mathcal N_N(x) = x + x^2(1/x - \bar X_N) = 2x - x^2\bar X_N$. Since $\thetaN = 1/\bar X_N$, $\mathcal N_N(x) - \thetaN = -\bar X_N x^2 + 2x - 1/\bar X_N = -\bar X_N(x - 1/\bar X_N)^2 = -(x-\thetaN)^2/\thetaN$. Writing $e_j = \iter{}{j} - \thetaN$ gives $e_{j+1} = -e_j^2/\thetaN$ and, by induction, $e_q = -e_0^{2^q}/\thetaN^{2^q-1}$. In units of $\thetaN$, $u_{j+1} = -u_j^2$ with $u_j = e_j/\thetaN$, so $\abs{u_j} \to 0$ iff $\abs{u_0} < 1$, i.e.\ $0 < x < 2\thetaN$; $\abs{u_0} > 1$ gives $\abs{u_j} \to \infty$ with $u_j < 0$ for $j \ge 1$; $u_0 = \pm1$ gives $u_1 = -1$ and then $u_j = -1$, i.e.\ the iterate $0$, where the log-likelihood is undefined. | 37 | +With $g_N(x) = 1/x - \bar X_N$ and $H_N(x) = -1/x^2$, | ||
| 38 | +\[ | ||||
| 39 | +\mathcal N_N(x) = x + x^2\Big(\frac1x - \bar X_N\Big) = 2x - x^2\bar X_N, | ||||
| 40 | +\qquad | ||||
| 41 | +\mathcal N_N(x) - \thetaN = -\bar X_N x^2 + 2x - \frac{1}{\bar X_N} = -\bar X_N\Big(x - \frac{1}{\bar X_N}\Big)^2 = -\frac{(x-\thetaN)^2}{\thetaN}, | ||||
| 42 | +\] | ||||
| 43 | +since $\thetaN = 1/\bar X_N$. Writing $e_j = \iter{}{j} - \thetaN$ gives $e_{j+1} = -e_j^2/\thetaN$ and, by induction, $e_q = -e_0^{2^q}/\thetaN^{2^q-1}$. In units of $\thetaN$, $u_{j+1} = -u_j^2$ with $u_j = e_j/\thetaN$, so $\abs{u_j} \to 0$ iff $\abs{u_0} < 1$, i.e.\ $0 < x < 2\thetaN$; $\abs{u_0} > 1$ gives $\abs{u_j} \to \infty$ with $u_j < 0$ for $j \ge 1$; $u_0 = \pm1$ gives $u_1 = -1$ and then $u_j = -1$, i.e.\ the iterate $0$, where the log-likelihood is undefined. | ||||
| 23 | \end{proof} | 44 | \end{proof} | ||
| 24 | 45 | ||||
| 25 | \begin{proof}[Proof of (v)--(vi)] | 46 | \begin{proof}[Proof of (v)--(vi)] | ||
modified
paper/appendices/proofs.tex
+56 −13
@@ -6,7 +6,11 @@ Throughout, $S_{j\ell,k}$ ($\ell = 1,\dots,s_j$) denotes the $\ell$-th sub-block | |||||
| 6 | For every $k$ and $j$, $s_j^{-1}\sum_{\ell=1}^{s_j} U_{S_{j\ell,k}} = U_k$, and hence $\sum_j w_j\, s_j^{-1}\sum_\ell U_{S_{j\ell,k}} = U_k$. | 6 | For every $k$ and $j$, $s_j^{-1}\sum_{\ell=1}^{s_j} U_{S_{j\ell,k}} = U_k$, and hence $\sum_j w_j\, s_j^{-1}\sum_\ell U_{S_{j\ell,k}} = U_k$. | ||
| 7 | \end{lemma} | 7 | \end{lemma} | ||
| 8 | \begin{proof} | 8 | \begin{proof} | ||
| 9 | −The sub-blocks of level $j$ partition $D_k$ into $s_j$ sets of equal size $m/s_j$, so $s_j^{-1}\sum_\ell U_{S_{j\ell,k}} = s_j^{-1}\sum_\ell (s_j/m)\sum_{i\in S_{j\ell,k}}\psi(X_i) = m^{-1}\sum_{i\in D_k}\psi(X_i) = U_k$. The second claim uses $\sum_j w_j = 1$. | 9 | +The sub-blocks of level $j$ partition $D_k$ into $s_j$ sets of equal size $m/s_j$, so | ||
| 10 | +\[ | ||||
| 11 | +\frac1{s_j}\sum_{\ell=1}^{s_j} U_{S_{j\ell,k}} = \frac1{s_j}\sum_{\ell=1}^{s_j}\frac{s_j}{m}\sum_{i\in S_{j\ell,k}}\psi(X_i) = \frac1m\sum_{i\in D_k}\psi(X_i) = U_k . | ||||
| 12 | +\] | ||||
| 13 | +The second claim uses $\sum_j w_j = 1$. | ||||
| 10 | \end{proof} | 14 | \end{proof} | ||
| 11 | 15 | ||||
| 12 | \begin{lemma}[Bias cancellation]\label{lem:bias} | 16 | \begin{lemma}[Bias cancellation]\label{lem:bias} | ||
@@ -17,10 +21,15 @@ Under \cref{ass:lin,ass:bias} with $m/s_r \ge n_0$, $\Jr{r,k} - \theta_0 = U_k + | |||||
| 17 | In particular $\E R^J_k$ is deterministic and does not depend on $k$. | 21 | In particular $\E R^J_k$ is deterministic and does not depend on $k$. | ||
| 18 | \end{lemma} | 22 | \end{lemma} | ||
| 19 | \begin{proof} | 23 | \begin{proof} | ||
| 20 | −Applying \eqref{eq:lin} to each sub-block and \cref{lem:influence}, $\Jr{r,k} - \theta_0 = \sum_j w_j s_j^{-1}\sum_\ell(U_{S_{j\ell,k}} + R_{S_{j\ell,k}}) = U_k + R^J_k$. Taking expectations with \eqref{eq:bias} at $n = m/s_j$, | 24 | +Applying \eqref{eq:lin} to each sub-block and \cref{lem:influence}, | ||
| 21 | \[ | 25 | \[ | ||
| 22 | −\E R^J_k = \sum_j w_j\Big[\sum_{a=1}^{r-1} b_a\,(s_j/m)^a + \rho_{m/s_j}\Big] = \sum_{a=1}^{r-1} b_a\, m^{-a}\sum_j w_j s_j^a + \sum_j w_j\rho_{m/s_j}, | 26 | +\Jr{r,k} - \theta_0 = \sum_{j=1}^r w_j\,\frac1{s_j}\sum_{\ell=1}^{s_j}\big(U_{S_{j\ell,k}} + R_{S_{j\ell,k}}\big) = U_k + R^J_k . | ||
| 23 | \] | 27 | \] | ||
| 28 | +Taking expectations with \eqref{eq:bias} at $n = m/s_j$, | ||||
| 29 | +\begin{align*} | ||||
| 30 | +\E R^J_k &= \sum_{j=1}^r w_j\Big[\sum_{a=1}^{r-1} b_a\Big(\frac{s_j}{m}\Big)^{a} + \rho_{m/s_j}\Big] | ||||
| 31 | + = \sum_{a=1}^{r-1}\frac{b_a}{m^{a}}\underbrace{\sum_{j=1}^r w_j s_j^{\,a}}_{=\,0} + \sum_{j=1}^r w_j\,\rho_{m/s_j}, | ||||
| 32 | +\end{align*} | ||||
| 24 | and the inner sums vanish by \eqref{eq:vandermonde}. The bound follows from $\norm{\rho_{m/s_j}} \le C_b(s_j/m)^r$. | 33 | and the inner sums vanish by \eqref{eq:vandermonde}. The bound follows from $\norm{\rho_{m/s_j}} \le C_b(s_j/m)^r$. | ||
| 25 | \end{proof} | 34 | \end{proof} | ||
| 26 | 35 | ||||
@@ -28,7 +37,12 @@ and the inner sums vanish by \eqref{eq:vandermonde}. The bound follows from $\no | |||||
| 28 | Under \cref{ass:lin}, $(\E\norm{R^J_k}^2)^{1/2} \le \sqrt C\,\Amp/m$ with $\Amp = \sum_j\abs{w_j}s_j$. | 37 | Under \cref{ass:lin}, $(\E\norm{R^J_k}^2)^{1/2} \le \sqrt C\,\Amp/m$ with $\Amp = \sum_j\abs{w_j}s_j$. | ||
| 29 | \end{lemma} | 38 | \end{lemma} | ||
| 30 | \begin{proof} | 39 | \begin{proof} | ||
| 31 | −By Minkowski's inequality in $L_2$, $(\E\norm{R^J_k}^2)^{1/2} \le \sum_j\abs{w_j}s_j^{-1}\sum_\ell(\E\norm{R_{S_{j\ell,k}}}^2)^{1/2} \le \sum_j\abs{w_j}s_j^{-1}\cdot s_j\cdot\sqrt C\,(s_j/m) = \sqrt C\,\Amp/m$. No independence between the nested sub-samples of a worker is used. | 40 | +By Minkowski's inequality in $L_2$, | ||
| 41 | +\[ | ||||
| 42 | +\big(\E\norm{R^J_k}^2\big)^{1/2} \le \sum_{j=1}^r\frac{\abs{w_j}}{s_j}\sum_{\ell=1}^{s_j}\big(\E\norm{R_{S_{j\ell,k}}}^2\big)^{1/2} | ||||
| 43 | +\le \sum_{j=1}^r\frac{\abs{w_j}}{s_j}\cdot s_j\cdot\sqrt C\,\frac{s_j}{m} = \frac{\sqrt C\,\Amp}{m} . | ||||
| 44 | +\] | ||||
| 45 | +No independence between the nested sub-samples of a worker is used. | ||||
| 32 | \end{proof} | 46 | \end{proof} | ||
| 33 | 47 | ||||
| 34 | \begin{lemma}[Averaged centered remainder]\label{lem:average} | 48 | \begin{lemma}[Averaged centered remainder]\label{lem:average} | ||
@@ -42,16 +56,24 @@ The blocks are disjoint, so the vectors $R^J_k - \E R^J_k$ are independent and c | |||||
| 42 | Under \cref{ass:lin,ass:bias}, $\pilot{r} - \thetaN = \E R^J_1 + K^{-1}\sum_k(R^J_k - \E R^J_k) - R_N = \op(m^{-r} + (m\sqrt K)^{-1} + N^{-1})$. | 56 | Under \cref{ass:lin,ass:bias}, $\pilot{r} - \thetaN = \E R^J_1 + K^{-1}\sum_k(R^J_k - \E R^J_k) - R_N = \op(m^{-r} + (m\sqrt K)^{-1} + N^{-1})$. | ||
| 43 | \end{lemma} | 57 | \end{lemma} | ||
| 44 | \begin{proof} | 58 | \begin{proof} | ||
| 45 | −Average \cref{lem:bias} over $k$: $\pilot{r} - \theta_0 = K^{-1}\sum_k U_k + K^{-1}\sum_k R^J_k = U_N + \E R^J_1 + K^{-1}\sum_k(R^J_k - \E R^J_k)$, since $K^{-1}\sum_k U_k = N^{-1}\sum_{i=1}^N\psi(X_i) = U_N$ (equal block sizes). Subtract $\thetaN - \theta_0 = U_N + R_N$ from \eqref{eq:lin} applied to the full sample; the influence terms cancel exactly. The orders follow from \cref{lem:bias,lem:average} and $\E\norm{R_N}^2 \le C N^{-2}$ via Markov's inequality. | 59 | +Averaging \cref{lem:bias} over $k$ and using $K^{-1}\sum_k U_k = N^{-1}\sum_{i=1}^N\psi(X_i) = U_N$ (equal block sizes), | ||
| 60 | +\[ | ||||
| 61 | +\pilot{r} - \theta_0 = U_N + \E R^J_1 + \frac1K\sum_{k=1}^K\big(R^J_k - \E R^J_k\big), \qquad \thetaN - \theta_0 = U_N + R_N, | ||||
| 62 | +\] | ||||
| 63 | +the second identity being \eqref{eq:lin} for the full sample. Subtracting, the influence terms cancel exactly. The orders follow from \cref{lem:bias,lem:average} and $\E\norm{R_N}^2 \le C N^{-2}$ via Markov's inequality. | ||||
| 46 | \end{proof} | 64 | \end{proof} | ||
| 47 | 65 | ||||
| 48 | \begin{lemma}[Quadratic contraction of one Newton step]\label{lem:newton} | 66 | \begin{lemma}[Quadratic contraction of one Newton step]\label{lem:newton} | ||
| 49 | Under \cref{ass:newton}, let $E_N$ be the event (of probability tending to one) on which $\thetaN \in U$ is an interior zero of $g_N$ and the bounds $M_N, \Lambda_N$ hold. On $E_N \cap \{\vartheta \in U\}$, $\norm{\mathcal N_N(\vartheta) - \thetaN} \le \tfrac12 M_N\Lambda_N\norm{\vartheta - \thetaN}^2$. | 67 | Under \cref{ass:newton}, let $E_N$ be the event (of probability tending to one) on which $\thetaN \in U$ is an interior zero of $g_N$ and the bounds $M_N, \Lambda_N$ hold. On $E_N \cap \{\vartheta \in U\}$, $\norm{\mathcal N_N(\vartheta) - \thetaN} \le \tfrac12 M_N\Lambda_N\norm{\vartheta - \thetaN}^2$. | ||
| 50 | \end{lemma} | 68 | \end{lemma} | ||
| 51 | \begin{proof} | 69 | \begin{proof} | ||
| 52 | −Let $e = \vartheta - \thetaN$. Since $U$ is convex, $\thetaN + te \in U$ for $t\in[0,1]$ and $g_N(\vartheta) = g_N(\thetaN) + \int_0^1 H_N(\thetaN + te)\,e\,\dd t = \int_0^1 H_N(\thetaN + te)\,e\,\dd t$. Hence | 70 | +Let $e = \vartheta - \thetaN$. Since $U$ is convex, $\thetaN + te \in U$ for $t\in[0,1]$, and because $g_N(\thetaN) = 0$, | ||
| 71 | +\[ | ||||
| 72 | +g_N(\vartheta) = \int_0^1 H_N(\thetaN + te)\,e\,\dd t . | ||||
| 73 | +\] | ||||
| 74 | +Hence | ||||
| 53 | \[ | 75 | \[ | ||
| 54 | −\mathcal N_N(\vartheta) - \thetaN = e - H_N(\vartheta)^{-1}g_N(\vartheta) = H_N(\vartheta)^{-1}\int_0^1\big[H_N(\vartheta) - H_N(\thetaN + te)\big]e\,\dd t . | 76 | +\mathcal N_N(\vartheta) - \thetaN = e - H_N(\vartheta)^{-1}g_N(\vartheta) = H_N(\vartheta)^{-1}\int_0^1\big[H_N(\vartheta) - H_N(\thetaN + te)\big]\,e\,\dd t . | ||
| 55 | \] | 77 | \] | ||
| 56 | The Lipschitz bound gives $\norm{H_N(\vartheta) - H_N(\thetaN + te)} \le \Lambda_N(1-t)\norm{e}$, and $\int_0^1(1-t)\,\dd t = 1/2$. | 78 | The Lipschitz bound gives $\norm{H_N(\vartheta) - H_N(\thetaN + te)} \le \Lambda_N(1-t)\norm{e}$, and $\int_0^1(1-t)\,\dd t = 1/2$. | ||
| 57 | \end{proof} | 79 | \end{proof} | ||
@@ -68,7 +90,13 @@ By \cref{lem:pilot} and $m \to \infty$ (which forces $K = o(N)$), $\pilot{r} - \ | |||||
| 68 | \[ | 90 | \[ | ||
| 69 | \sqrt N\norm{\iter{r}{q} - \thetaN} = \op\Big(\sqrt N\big[m^{-r} + (m\sqrt K)^{-1} + N^{-1}\big]^{2^q}\Big) \le \op\Big(3^{2^q}\sqrt N\big[m^{-r2^q} + (m\sqrt K)^{-2^q} + N^{-2^q}\big]\Big), | 91 | \sqrt N\norm{\iter{r}{q} - \thetaN} = \op\Big(\sqrt N\big[m^{-r} + (m\sqrt K)^{-1} + N^{-1}\big]^{2^q}\Big) \le \op\Big(3^{2^q}\sqrt N\big[m^{-r2^q} + (m\sqrt K)^{-2^q} + N^{-2^q}\big]\Big), | ||
| 70 | \] | 92 | \] | ||
| 71 | −using $(a+b+c)^{2^q} \le 3^{2^q}\max(a,b,c)^{2^q}$. With $m = N/K$: $\sqrt N m^{-r2^q} = N^{1/2}(K/N)^{r2^q} \to 0$ iff $K^{r2^q} = o(N^{r2^q - 1/2})$ iff $K = o(N^{1 - 1/(2^{q+1}r)})$. Next, $\sqrt N(m\sqrt K)^{-2^q} = N^{1/2}(\sqrt K/N)^{2^q} = N^{1/2}K^{2^{q-1}}N^{-2^q}$; for $q = 0$ this is $(K/N)^{1/2} \to 0$; for $q \ge 1$ and $K \le N$ it is at most $N^{1/2 + 2^{q-1} - 2^q} = N^{1/2 - 2^{q-1}} \le N^{-1/2} \to 0$. Finally $\sqrt N N^{-2^q} \to 0$. The limit law follows from $\sqrt N(\thetaN - \theta_0) \dto \mathcal N(0,\Sigma_\psi)$ and Slutsky's lemma. | 93 | +using $(a+b+c)^{2^q} \le 3^{2^q}\max(a,b,c)^{2^q}$. With $m = N/K$ the three terms are | ||
| 94 | +\begin{align*} | ||||
| 95 | +\sqrt N\, m^{-r2^q} &= N^{1/2}\Big(\frac KN\Big)^{r2^q} \to 0 \quad\Longleftrightarrow\quad K^{r2^q} = o\big(N^{r2^q - 1/2}\big) \quad\Longleftrightarrow\quad K = o\big(N^{1 - 1/(2^{q+1}r)}\big);\\ | ||||
| 96 | +\sqrt N\,(m\sqrt K)^{-2^q} &= N^{1/2}K^{2^{q-1}}N^{-2^q} = | ||||
| 97 | +\begin{cases} (K/N)^{1/2} \to 0, & q = 0 \ (\text{since } K = o(N)),\\ \le N^{1/2 - 2^{q-1}} \le N^{-1/2} \to 0, & q \ge 1,\ K \le N; \end{cases}\\ | ||||
| 98 | +\sqrt N\, N^{-2^q} &\to 0 . | ||||
| 99 | +\end{align*} The limit law follows from $\sqrt N(\thetaN - \theta_0) \dto \mathcal N(0,\Sigma_\psi)$ and Slutsky's lemma. | ||||
| 72 | \end{proof} | 100 | \end{proof} | ||
| 73 | 101 | ||||
| 74 | \begin{proof}[Proof of \cref{cor:inference}] | 102 | \begin{proof}[Proof of \cref{cor:inference}] | ||
@@ -78,21 +106,36 @@ $\sqrt N(\iter{r}{q} - \theta_0) \dto \mathcal N(0,\Sigma_\psi)$ and $\widehat\S | |||||
| 78 | \begin{proof}[Proof of \cref{thm:inexact}] | 106 | \begin{proof}[Proof of \cref{thm:inexact}] | ||
| 79 | Let $e = \vartheta - \thetaN$ and $\bar H := \int_0^1 H_N(\thetaN + te)\,\dd t$, so that $g_N(\vartheta) = \bar H e$ on $E_N\cap\{\vartheta\in U\}$. Then | 107 | Let $e = \vartheta - \thetaN$ and $\bar H := \int_0^1 H_N(\thetaN + te)\,\dd t$, so that $g_N(\vartheta) = \bar H e$ on $E_N\cap\{\vartheta\in U\}$. Then | ||
| 80 | \[ | 108 | \[ | ||
| 81 | −\tilde\vartheta - \thetaN = e - \widehat H^{-1}(\bar H e + \delta_g) = \widehat H^{-1}\big(\widehat H - \bar H\big)e - \widehat H^{-1}\delta_g, | 109 | +\tilde\vartheta - \thetaN = e - \widehat H^{-1}(\bar H e + \delta_g) = \widehat H^{-1}\big(\widehat H - \bar H\big)\,e - \widehat H^{-1}\delta_g, | ||
| 110 | +\] | ||||
| 111 | +and | ||||
| 112 | +\[ | ||||
| 113 | +\norm{\widehat H - \bar H} \le \norm{\widehat H - H_N(\thetaN)} + \int_0^1\norm{H_N(\thetaN) - H_N(\thetaN + te)}\,\dd t \le \norm{\widehat H - H_N(\thetaN)} + \tfrac12\Lambda_N\norm{e}, | ||||
| 82 | \] | 114 | \] | ||
| 83 | −and $\norm{\widehat H - \bar H} \le \norm{\widehat H - H_N(\thetaN)} + \int_0^1\norm{H_N(\thetaN) - H_N(\thetaN + te)}\dd t \le \norm{\widehat H - H_N(\thetaN)} + \tfrac12\Lambda_N\norm{e}$, which gives \eqref{eq:inexact}. Iterating with a fixed $\widehat H$ (or with a fresh $\widehat H$ of the same accuracy at each step) yields $e_{j+1} = \op(\delta_N e_j + e_j^2)$ and, by induction, $e_q = \op(e_0\delta_N^q + \sum_{\text{mixed}} \cdots + e_0^{2^q})$ in which every mixed term is dominated by $e_0\delta_N^{q}$ or $e_0^{2^q}$ once $e_0 \to 0$ and $\delta_N\to0$; the iterates remain in $U$ as in \cref{lem:qsteps} because $\delta_N + \kappa_N e_0 \pto 0$. With $e_0 = \op(m^{-r} + \sqrt K/N + N^{-1})$ the binding condition is $\sqrt N\,m^{-r}\delta_N^q \to 0$ (the terms $\sqrt N(\sqrt K/N)\delta_N^q$ and $\sqrt N N^{-1}\delta_N^q$ vanish for any $\delta_N \le 1$ and $K \le N$), together with the exact-curvature condition $\sqrt N e_0^{2^q} \to 0$ of \cref{thm:frontier}. | 115 | +which gives \eqref{eq:inexact}. Iterating with a fixed $\widehat H$ (or with a fresh $\widehat H$ of the same accuracy at each step) yields $e_{j+1} = \op(\delta_N e_j + e_j^2)$ and, by induction, $e_q = \op(e_0\delta_N^q + \sum_{\text{mixed}} \cdots + e_0^{2^q})$ in which every mixed term is dominated by $e_0\delta_N^{q}$ or $e_0^{2^q}$ once $e_0 \to 0$ and $\delta_N\to0$; the iterates remain in $U$ as in \cref{lem:qsteps} because $\delta_N + \kappa_N e_0 \pto 0$. With $e_0 = \op(m^{-r} + \sqrt K/N + N^{-1})$ the binding condition is $\sqrt N\,m^{-r}\delta_N^q \to 0$ (the terms $\sqrt N(\sqrt K/N)\delta_N^q$ and $\sqrt N N^{-1}\delta_N^q$ vanish for any $\delta_N \le 1$ and $K \le N$), together with the exact-curvature condition $\sqrt N e_0^{2^q} \to 0$ of \cref{thm:frontier}. | ||
| 84 | 116 | ||||
| 85 | −(a) Write $\widehat H - H_N(\thetaN) = K^{-1}\sum_k[H_{D_k}(\thetahat_{D_k}) - H_{D_k}(\theta_0)] + [H_N(\theta_0) - H_N(\thetaN)]$. The second bracket is $\op(\norm{\thetaN - \theta_0}) = \op(N^{-1/2})$ by \cref{ass:newton}. For the first, \cref{ass:third} gives, for each $k$ and $\Delta_k := \thetahat_{D_k} - \theta_0$, | 117 | +(a) Write | ||
| 118 | +\[ | ||||
| 119 | +\widehat H - H_N(\thetaN) = \frac1K\sum_{k=1}^K\big[H_{D_k}(\thetahat_{D_k}) - H_{D_k}(\theta_0)\big] + \big[H_N(\theta_0) - H_N(\thetaN)\big]. | ||||
| 120 | +\] The second bracket is $\op(\norm{\thetaN - \theta_0}) = \op(N^{-1/2})$ by \cref{ass:newton}. For the first, \cref{ass:third} gives, for each $k$ and $\Delta_k := \thetahat_{D_k} - \theta_0$, | ||||
| 86 | \[ | 121 | \[ | ||
| 87 | H_{D_k}(\thetahat_{D_k}) - H_{D_k}(\theta_0) = \nabla H_{D_k}(\theta_0)[\Delta_k] + r_k, \qquad \norm{r_k} \le \tfrac12\bar L_k\norm{\Delta_k}^2, | 122 | H_{D_k}(\thetahat_{D_k}) - H_{D_k}(\theta_0) = \nabla H_{D_k}(\theta_0)[\Delta_k] + r_k, \qquad \norm{r_k} \le \tfrac12\bar L_k\norm{\Delta_k}^2, | ||
| 88 | \] | 123 | \] | ||
| 89 | with $\bar L_k$ the block average of $L(X_i)$. By Cauchy--Schwarz and \cref{ass:third}, $\E\norm{r_k} \le \tfrac12(\E\bar L_k^2)^{1/2}(\E\norm{\Delta_k}^4)^{1/2} = O(m^{-1})$. Write $\nabla H_{D_k}(\theta_0) = T + V_k$ with $T = \E\nabla h(X;\theta_0)$ and $V_k$ a centered block average with $\E\norm{V_k}^2 = O(m^{-1})$, and $\Delta_k = U_k + R_{D_k}$. Then $K^{-1}\sum_k T[U_k] = T[U_N] = \op(N^{-1/2})$; $K^{-1}\sum_k T[R_{D_k}] = O_p(m^{-1})$ by \cref{ass:lin}; and $\E\norm{V_k[U_k]} \le (\E\norm{V_k}^2)^{1/2}(\E\norm{U_k}^2)^{1/2} = O(m^{-1})$. Collecting terms, $\delta_N = m^{-1} + N^{-1/2}$, and the condition $\sqrt N m^{-r}m^{-q} \to 0$ reads $K = o(N^{1 - 1/(2(r+q))})$; the $N^{-1/2}$ part of $\delta_N$ never binds. Since $r + q \le 2^q r$ for $q \ge 1$, this condition implies that of \cref{thm:frontier}. | 124 | with $\bar L_k$ the block average of $L(X_i)$. By Cauchy--Schwarz and \cref{ass:third}, $\E\norm{r_k} \le \tfrac12(\E\bar L_k^2)^{1/2}(\E\norm{\Delta_k}^4)^{1/2} = O(m^{-1})$. Write $\nabla H_{D_k}(\theta_0) = T + V_k$ with $T = \E\nabla h(X;\theta_0)$ and $V_k$ a centered block average with $\E\norm{V_k}^2 = O(m^{-1})$, and $\Delta_k = U_k + R_{D_k}$. Then $K^{-1}\sum_k T[U_k] = T[U_N] = \op(N^{-1/2})$; $K^{-1}\sum_k T[R_{D_k}] = O_p(m^{-1})$ by \cref{ass:lin}; and $\E\norm{V_k[U_k]} \le (\E\norm{V_k}^2)^{1/2}(\E\norm{U_k}^2)^{1/2} = O(m^{-1})$. Collecting terms, $\delta_N = m^{-1} + N^{-1/2}$, and the condition $\sqrt N m^{-r}m^{-q} \to 0$ reads $K = o(N^{1 - 1/(2(r+q))})$; the $N^{-1/2}$ part of $\delta_N$ never binds. Since $r + q \le 2^q r$ for $q \ge 1$, this condition implies that of \cref{thm:frontier}. | ||
| 90 | 125 | ||||
| 91 | −(b) $H_{D_1}(\vartheta) - H_N(\thetaN) = [H_{D_1}(\vartheta) - H_{D_1}(\theta_0)] + [H_{D_1}(\theta_0) - \bar H(\theta_0)] + [\bar H(\theta_0) - H_N(\theta_0)] + [H_N(\theta_0) - H_N(\thetaN)]$, where $\bar H(\theta_0) = \E h(X;\theta_0)$. The first and last brackets are $\op(\norm{\vartheta - \theta_0} + N^{-1/2})$ by the Lipschitz property (applied to $H_{D_1}$, for which the same bound is assumed as for $H_N$ on $U$, with probability tending to one); the second is $\op(m^{-1/2})$ and the third $\op(N^{-1/2})$ by the central limit theorem for block averages of $h(X_i;\theta_0)$. Hence $\delta_N = m^{-1/2} + \norm{\vartheta - \theta_0}$, and along the iteration $\norm{\iter{}{j} - \theta_0} = \op(N^{-1/2} + e_j)$, so the binding condition is $\sqrt N m^{-r}m^{-q/2} \to 0$, i.e.\ $K = o(N^{1 - 1/(2r+q)})$. | 126 | +(b) Decompose | ||
| 127 | +\[ | ||||
| 128 | +H_{D_1}(\vartheta) - H_N(\thetaN) = \big[H_{D_1}(\vartheta) - H_{D_1}(\theta_0)\big] + \big[H_{D_1}(\theta_0) - \bar H(\theta_0)\big] + \big[\bar H(\theta_0) - H_N(\theta_0)\big] + \big[H_N(\theta_0) - H_N(\thetaN)\big], | ||||
| 129 | +\] | ||||
| 130 | +where $\bar H(\theta_0) = \E h(X;\theta_0)$. The first and last brackets are $\op(\norm{\vartheta - \theta_0} + N^{-1/2})$ by the Lipschitz property (applied to $H_{D_1}$, for which the same bound is assumed as for $H_N$ on $U$, with probability tending to one); the second is $\op(m^{-1/2})$ and the third $\op(N^{-1/2})$ by the central limit theorem for block averages of $h(X_i;\theta_0)$. Hence $\delta_N = m^{-1/2} + \norm{\vartheta - \theta_0}$, and along the iteration $\norm{\iter{}{j} - \theta_0} = \op(N^{-1/2} + e_j)$, so the binding condition is $\sqrt N m^{-r}m^{-q/2} \to 0$, i.e.\ $K = o(N^{1 - 1/(2r+q)})$. | ||||
| 92 | 131 | ||||
| 93 | (c) Rounding each entry of $H_N(\vartheta)$ to $b$ significant bits yields $\widehat H$ with $\abs{\widehat H_{ab} - H_{N,ab}} \le 2^{-b}\abs{H_{N,ab}}$, hence $\norm{\widehat H - H_N(\vartheta)} \le p\,2^{-b}\norm{H_N(\vartheta)}_{\max}$, and $\norm{H_N(\vartheta) - H_N(\thetaN)} \le \Lambda_N\norm{e}$; so $\delta_N = O_p(\varepsilon) + \op(\norm{e})$, the second part being absorbed in the $\norm{e}^2$ term. Likewise $\norm{\delta_g} \le p\,\varepsilon\,\norm{g_N(\vartheta)}_{\max} \le p\,\varepsilon\,\norm{\bar H}\norm{e} = \op(\varepsilon\norm{e})$. Inserting into \eqref{eq:inexact} with $\norm{e_0} = \op(m^{-r} + \sqrt K/N + N^{-1})$, first-order equivalence requires $\sqrt N\varepsilon m^{-r} \to 0$ (the other two terms vanish for $\varepsilon \le 1$). With $K \sim N^\alpha$, $m^{-r} = N^{-r(1-\alpha)}$ up to constants, so $\sqrt N\varepsilon m^{-r} \to 0$ iff $\log_2(1/\varepsilon) = b \ge (\tfrac12 - r(1-\alpha))\log_2 N + \omega(1)$ when $\tfrac12 - r(1-\alpha) \ge 0$, i.e.\ $\alpha \ge 1 - 1/(2r)$; otherwise $\sqrt N m^{-r} \to 0$ already and any $b \ge 0$ works. | 132 | (c) Rounding each entry of $H_N(\vartheta)$ to $b$ significant bits yields $\widehat H$ with $\abs{\widehat H_{ab} - H_{N,ab}} \le 2^{-b}\abs{H_{N,ab}}$, hence $\norm{\widehat H - H_N(\vartheta)} \le p\,2^{-b}\norm{H_N(\vartheta)}_{\max}$, and $\norm{H_N(\vartheta) - H_N(\thetaN)} \le \Lambda_N\norm{e}$; so $\delta_N = O_p(\varepsilon) + \op(\norm{e})$, the second part being absorbed in the $\norm{e}^2$ term. Likewise $\norm{\delta_g} \le p\,\varepsilon\,\norm{g_N(\vartheta)}_{\max} \le p\,\varepsilon\,\norm{\bar H}\norm{e} = \op(\varepsilon\norm{e})$. Inserting into \eqref{eq:inexact} with $\norm{e_0} = \op(m^{-r} + \sqrt K/N + N^{-1})$, first-order equivalence requires $\sqrt N\varepsilon m^{-r} \to 0$ (the other two terms vanish for $\varepsilon \le 1$). With $K \sim N^\alpha$, $m^{-r} = N^{-r(1-\alpha)}$ up to constants, so $\sqrt N\varepsilon m^{-r} \to 0$ iff $\log_2(1/\varepsilon) = b \ge (\tfrac12 - r(1-\alpha))\log_2 N + \omega(1)$ when $\tfrac12 - r(1-\alpha) \ge 0$, i.e.\ $\alpha \ge 1 - 1/(2r)$; otherwise $\sqrt N m^{-r} \to 0$ already and any $b \ge 0$ works. | ||
| 94 | \end{proof} | 133 | \end{proof} | ||
| 95 | 134 | ||||
| 96 | \begin{proof}[Proof of \cref{prop:robust}] | 135 | \begin{proof}[Proof of \cref{prop:robust}] | ||
| 97 | −(i) With weights $m_k/N$, $\sum_k (m_k/N)U_k = N^{-1}\sum_i\psi(X_i) = U_N$, so \cref{lem:pilot} holds with $\E R^J$ replaced by $\sum_k(m_k/N)\E R^J_k$, of norm at most $C_b(\sum_j\abs{w_j}s_j^r)\,N^{-1}\sum_k m_k^{1-r} = C_b(\cdots)m_{\mathrm{eff}}^{-r}$, and with centered second moment $\sum_k(m_k/N)^2\,C\Amp^2/m_k^2 = C\Amp^2K/N^2$. The Newton steps use $g_N = \sum_k(m_k/N)g_{D_k}$ and $H_N = \sum_k(m_k/N)H_{D_k}$, which are the full-sample derivatives, so \cref{lem:newton,lem:qsteps} are unchanged. With equal weights, $K^{-1}\sum_k U_k - U_N = \sum_i c_i\psi(X_i)$ with $c_i = (Km_{k(i)})^{-1} - N^{-1}$; its covariance is $(\sum_i c_i^2)\Sigma_\psi$ and $\sum_i c_i^2 = \sum_k m_k[(Km_k)^{-1} - N^{-1}]^2 = K^{-2}\sum_k m_k^{-1} - N^{-1}$, which is $\ge 0$ with equality iff all $m_k$ are equal (Jensen); when the sizes are bounded away from equality in the sense $K^{-1}\sum_k (N/(Km_k)) \ge 1 + \eta$, this is $\ge \eta/N$, of exact order $N^{-1}$, so the discrepancy is of exact order $N^{-1/2}$ and $\sqrt N(\pilot{} - \thetaN)$ does not vanish. One Newton step squares it to $\op(N^{-1})$ by \cref{lem:newton}. (ii) The only properties of $\psi$ used in \cref{lem:influence,lem:bias,lem:remainder,lem:average,lem:pilot} are $\E\psi = 0$ and that the same $\psi$ appears in \eqref{eq:lin} for every sub-sample; \cref{ass:newton} is stated for the pseudo-MLE with the sandwich limit. The proofs go through verbatim. | 136 | +(i) With weights $m_k/N$, $\sum_k (m_k/N)U_k = N^{-1}\sum_i\psi(X_i) = U_N$, so \cref{lem:pilot} holds with $\E R^J$ replaced by $\sum_k(m_k/N)\E R^J_k$, whose norm and whose centered second moment satisfy | ||
| 137 | +\[ | ||||
| 138 | +\Big\|\sum_{k}\frac{m_k}{N}\E R^J_k\Big\| \le C_b\Big(\sum_j\abs{w_j}s_j^{\,r}\Big)\frac1N\sum_{k=1}^K m_k^{\,1-r} = C_b\Big(\sum_j\abs{w_j}s_j^{\,r}\Big)m_{\mathrm{eff}}^{-r}, \qquad | ||||
| 139 | +\sum_{k=1}^K\Big(\frac{m_k}{N}\Big)^2\frac{C\Amp^2}{m_k^2} = \frac{C\Amp^2 K}{N^2}. | ||||
| 140 | +\] The Newton steps use $g_N = \sum_k(m_k/N)g_{D_k}$ and $H_N = \sum_k(m_k/N)H_{D_k}$, which are the full-sample derivatives, so \cref{lem:newton,lem:qsteps} are unchanged. With equal weights, $K^{-1}\sum_k U_k - U_N = \sum_i c_i\psi(X_i)$ with $c_i = (Km_{k(i)})^{-1} - N^{-1}$; its covariance is $(\sum_i c_i^2)\Sigma_\psi$ and $\sum_i c_i^2 = \sum_k m_k[(Km_k)^{-1} - N^{-1}]^2 = K^{-2}\sum_k m_k^{-1} - N^{-1}$, which is $\ge 0$ with equality iff all $m_k$ are equal (Jensen); when the sizes are bounded away from equality in the sense $K^{-1}\sum_k (N/(Km_k)) \ge 1 + \eta$, this is $\ge \eta/N$, of exact order $N^{-1}$, so the discrepancy is of exact order $N^{-1/2}$ and $\sqrt N(\pilot{} - \thetaN)$ does not vanish. One Newton step squares it to $\op(N^{-1})$ by \cref{lem:newton}. (ii) The only properties of $\psi$ used in \cref{lem:influence,lem:bias,lem:remainder,lem:average,lem:pilot} are $\E\psi = 0$ and that the same $\psi$ appears in \eqref{eq:lin} for every sub-sample; \cref{ass:newton} is stated for the pseudo-MLE with the sandwich limit. The proofs go through verbatim. | ||||
| 98 | \end{proof} | 141 | \end{proof} | ||
modified
paper/main.pdf
+0 −0
Binary file not shown.
modified
paper/main.tex
+3 −1
@@ -13,6 +13,8 @@ | |||||
| 13 | \onehalfspacing | 13 | \onehalfspacing | ||
| 14 | \usepackage{amsmath,amssymb,amsthm} | 14 | \usepackage{amsmath,amssymb,amsthm} | ||
| 15 | \usepackage{mathtools} | 15 | \usepackage{mathtools} | ||
| 16 | +\usepackage{bm} | ||||
| 17 | +\allowdisplaybreaks | ||||
| 16 | \theoremstyle{plain} | 18 | \theoremstyle{plain} | ||
| 17 | \newtheorem{theorem}{Theorem} | 19 | \newtheorem{theorem}{Theorem} | ||
| 18 | \newtheorem{proposition}{Proposition} | 20 | \newtheorem{proposition}{Proposition} | ||
@@ -98,7 +100,7 @@ | |||||
| 98 | } | 100 | } | ||
| 99 | \newcommand{\WPkeywords}{distributed estimation, maximum likelihood, split-sample jackknife, one-step Newton estimator, communication complexity, asymptotic equivalence, Monte Carlo} | 101 | \newcommand{\WPkeywords}{distributed estimation, maximum likelihood, split-sample jackknife, one-step Newton estimator, communication complexity, asymptotic equivalence, Monte Carlo} | ||
| 100 | \newcommand{\WPjel}{C13, C15, C55, C63} | 102 | \newcommand{\WPjel}{C13, C15, C55, C63} | ||
| 101 | −\newcommand{\WPauthor}{Simon-Pierre Boucher\\[0.15cm]{\normalsize with the AI co-author designation GPT-6.1 (prior exploratory draft; see the disclosure on page~\pageref{sec:provenance})}} | 103 | +\newcommand{\WPauthor}{Simon-Pierre Boucher} | ||
| 102 | \newcommand{\WPaffiliation}{% | 104 | \newcommand{\WPaffiliation}{% | ||
| 103 | Département des sciences administratives\\ | 105 | Département des sciences administratives\\ | ||
| 104 | Université du Québec en Outaouais% | 106 | Université du Québec en Outaouais% | ||
modified
paper/sections/exactmodel.tex
+41 −9
@@ -1,19 +1,51 @@ | |||||
| 1 | \section{Exact analysis in the exponential model and sharpness}\label{sec:exact} | 1 | \section{Exact analysis in the exponential model and sharpness}\label{sec:exact} | ||
| 2 | 2 | ||||
| 3 | −Let $X_i \sim \mathrm{Exp}(\theta_0)$ in the rate parametrization. The average log-likelihood is $L_n(\theta) = \log\theta - \theta\bar X_n$, so $\thetahat_n = 1/\bar X_n$, $g_n(\theta) = 1/\theta - \bar X_n$ and $H_n(\theta) = -1/\theta^2$. The model is trivial for distributed inference, since $\sum_{i\in D_k}X_i$ is a sufficient statistic that any worker can upload, and the Hessian does not depend on the data, so that single-machine curvature is exact. It is used here because every quantity in \cref{sec:construction,sec:results} has a closed form, which makes it possible to check the assumptions, the constants and the boundary behaviour rather than assume them. The following statements are proved in \cref{app:exp} from the Gamma and Beta moments and verified symbolically in the replication package. | 3 | +Let $X_i \sim \mathrm{Exp}(\theta_0)$ in the rate parametrization. The average log-likelihood, score and Hessian on $n$ observations are | ||
| 4 | +\begin{equation}\label{eq:exp-model} | ||||
| 5 | +L_n(\theta) = \log\theta - \theta\,\bar X_n, \qquad g_n(\theta) = \frac1\theta - \bar X_n, \qquad H_n(\theta) = -\frac{1}{\theta^2}, \qquad \thetahat_n = \frac{1}{\bar X_n}. | ||||
| 6 | +\end{equation} | ||||
| 7 | +The model is trivial for distributed inference, since $\sum_{i\in D_k}X_i$ is a sufficient statistic that any worker can upload, and the Hessian does not depend on the data, so that single-machine curvature is exact. It is used here because every quantity in \cref{sec:construction,sec:results} has a closed form, which makes it possible to check the assumptions, the constants and the boundary behaviour rather than assume them. The statements below are proved in \cref{app:exp} from Gamma and Beta moments and verified symbolically in the replication package. Write $Y = \theta_0\bar X_n$, so that $nY \sim \mathrm{Gamma}(n,1)$, and | ||||
| 8 | +\begin{equation}\label{eq:exp-psi} | ||||
| 9 | +\psi(x) = \theta_0 - \theta_0^2\,x, \qquad R_n = \thetahat_n - \theta_0 - \frac1n\sum_{i=1}^n\psi(X_i) = \frac{(1 - \theta_0\bar X_n)^2}{\bar X_n} . | ||||
| 10 | +\end{equation} | ||||
| 4 | 11 | ||||
| 5 | \begin{theorem}[Exact exponential identities and boundary behaviour]\label{thm:exponential} | 12 | \begin{theorem}[Exact exponential identities and boundary behaviour]\label{thm:exponential} | ||
| 6 | −Let $n\bar X_n \sim \mathrm{Gamma}(n,\theta_0)$, $\psi(x) = \theta_0 - \theta_0^2 x$ and $R_n = \thetahat_n - \theta_0 - n^{-1}\sum_i\psi(X_i) = (1-\theta_0\bar X_n)^2/\bar X_n$. | ||||
| 7 | \begin{enumerate}[label=(\roman*)] | 13 | \begin{enumerate}[label=(\roman*)] | ||
| 8 | −\item For $n > 2$: $\E\thetahat_n = \theta_0\,n/(n-1)$, $\Var\thetahat_n = \theta_0^2 n^2/((n-1)^2(n-2))$, $\E R_n = \theta_0/(n-1) = \theta_0\sum_{a\ge1}n^{-a}$ and $\E R_n^2 = \theta_0^2(3n+2)/(n(n-1)(n-2))$, with $\sup_{n\ge3}n^2\E R_n^2/\theta_0^2 = 33/2$. Hence \cref{ass:lin,ass:bias} hold for every $r$ with $n_0 = 3$, $C = 33\theta_0^2/2$ and $b_a = \theta_0$. | 14 | +\item \emph{Moments.} For $n > 2$, | ||
| 9 | −\item For any pattern with $m/s_r > 1$: $\E\Jr{r,k} - \theta_0 = \theta_0\sum_j w_j s_j/(m - s_j) = \betar\theta_0 m^{-r}(1 + O(m^{-1}))$; in particular $\E\Jr{2,k} - \theta_0 = -2\theta_0/((m-1)(m-2))$ and $\E\Jr{3,k} - \theta_0 = 8\theta_0/((m-1)(m-2)(m-4))$. | 15 | +\begin{align} | ||
| 10 | −\item For $m > 4$: $\Var\Jr{2,k} = \theta_0^2\, m^2(m^2 - 6m + 17)/((m-4)(m-2)^2(m-1)^2) = \theta_0^2(m^{-1} + 4m^{-2} + O(m^{-3}))$, the same first two terms as $\Var\thetahat_m$; the variance cost of the half-block correction is $O(m^{-3})$ in this model. | 16 | +\E\thetahat_n &= \theta_0\,\frac{n}{n-1}, & | ||
| 11 | −\item Newton identities: $\mathcal N_N(x) = 2x - x^2\bar X_N$ and $\mathcal N_N(x) - \thetaN = -(x - \thetaN)^2/\thetaN$ exactly; after $q$ steps $\mathcal N_N^{q}(x) - \thetaN = -(x-\thetaN)^{2^q}/\thetaN^{2^q-1}$. The iteration converges iff $0 < x < 2\thetaN$ and diverges to $-\infty$ otherwise ($x = 2\thetaN$ maps to $0$ and stays there). | 17 | +\Var\thetahat_n &= \theta_0^2\,\frac{n^2}{(n-1)^2(n-2)}, \label{eq:exp-mom1}\\ | ||
| 12 | −\item Boundary: if $m \to \infty$ and $K/N^{\alphafr{r}{q}} \to c \in (0,\infty)$, then $\sqrt N(\iter{r}{q} - \thetaN) \pto -\theta_0(\betar c^{r})^{2^q}$ for $q \ge 1$ and $\pto \betar\theta_0 c^r$ for $q = 0$; for $(r,q) = (2,1)$ the limit is $-4\theta_0 c^4$, so that $\sqrt N(\iter{2}{1} - \theta_0) \dto \mathcal N(-4\theta_0c^4, \theta_0^2)$. | 18 | +\E R_n &= \frac{\theta_0}{n-1} = \theta_0\sum_{a\ge1} n^{-a}, & | ||
| 13 | −\item Necessity: if $m \to \infty$ and $K/N^{\alphafr{r}{q}} \to \infty$, then $\sqrt N\,\abs{\iter{r}{q} - \thetaN} \pto \infty$. | 19 | +\E R_n^2 &= \theta_0^2\,\frac{3n+2}{n(n-1)(n-2)}, \label{eq:exp-mom2} | ||
| 20 | +\end{align} | ||||
| 21 | +and $\sup_{n\ge3} n^2\,\E R_n^2/\theta_0^2 = 33/2$. Hence \cref{ass:lin,ass:bias} hold for every $r$ with $n_0 = 3$, $C = 33\theta_0^2/2$ and $b_a = \theta_0$ for all $a$. | ||||
| 22 | +\item \emph{Exact bias of the corrected estimators.} For any pattern with $m/s_r > 1$, | ||||
| 23 | +\begin{equation}\label{eq:exp-bias} | ||||
| 24 | +\E\Jr{r,k} - \theta_0 = \theta_0\sum_{j=1}^r \frac{w_j\, s_j}{m - s_j} = \betar\,\theta_0\, m^{-r}\big(1 + O(m^{-1})\big), | ||||
| 25 | +\end{equation} | ||||
| 26 | +in particular | ||||
| 27 | +\begin{equation}\label{eq:exp-bias23} | ||||
| 28 | +\E\Jr{2,k} - \theta_0 = -\frac{2\theta_0}{(m-1)(m-2)}, \qquad \E\Jr{3,k} - \theta_0 = \frac{8\theta_0}{(m-1)(m-2)(m-4)} . | ||||
| 29 | +\end{equation} | ||||
| 30 | +\item \emph{Exact variance of the half-block correction.} For $m > 4$, | ||||
| 31 | +\begin{equation}\label{eq:exp-varJ2} | ||||
| 32 | +\Var\Jr{2,k} = \theta_0^2\,\frac{m^2\,(m^2 - 6m + 17)}{(m-4)(m-2)^2(m-1)^2} = \theta_0^2\Big(\frac1m + \frac{4}{m^2} + O(m^{-3})\Big), | ||||
| 33 | +\end{equation} | ||||
| 34 | +the same first two terms as $\Var\thetahat_m$: the variance cost of the half-block correction is $O(m^{-3})$ in this model. | ||||
| 35 | +\item \emph{Newton identities and basin.} For every $x > 0$, | ||||
| 36 | +\begin{equation}\label{eq:exp-newton} | ||||
| 37 | +\mathcal N_N(x) = 2x - x^2\bar X_N, \qquad \mathcal N_N(x) - \thetaN = -\frac{(x - \thetaN)^2}{\thetaN}, \qquad \mathcal N_N^{\,q}(x) - \thetaN = -\frac{(x-\thetaN)^{2^q}}{\thetaN^{\,2^q-1}} . | ||||
| 38 | +\end{equation} | ||||
| 39 | +The iteration converges to $\thetaN$ if and only if $0 < x < 2\thetaN$; it diverges to $-\infty$ when $x < 0$ or $x > 2\thetaN$, and $x = 2\thetaN$ is mapped to $0$. | ||||
| 40 | +\item \emph{Boundary.} If $m \to \infty$ and $K/N^{\alphafr{r}{q}} \to c \in (0,\infty)$, | ||||
| 41 | +\begin{equation}\label{eq:exp-boundary} | ||||
| 42 | +\sqrt N\,\big(\iter{r}{q} - \thetaN\big) \;\pto\; \begin{cases} -\theta_0\,(\betar\, c^{\,r})^{2^q}, & q \ge 1, \\[2pt] \betar\,\theta_0\, c^{\,r}, & q = 0, \end{cases} | ||||
| 43 | +\end{equation} | ||||
| 44 | +so that for $(r,q) = (2,1)$ the limit is $-4\theta_0c^4$ and $\sqrt N(\iter{2}{1} - \theta_0) \dto \mathcal N(-4\theta_0c^4,\, \theta_0^2)$. | ||||
| 45 | +\item \emph{Necessity.} If $m \to \infty$ and $K/N^{\alphafr{r}{q}} \to \infty$, then $\sqrt N\,\abs{\iter{r}{q} - \thetaN} \pto \infty$. | ||||
| 14 | \end{enumerate} | 46 | \end{enumerate} | ||
| 15 | \end{theorem} | 47 | \end{theorem} | ||
| 16 | 48 | ||||
| 17 | −Items (v)--(vi) show that the exponent of \cref{thm:frontier} is sharp for the construction in this model: the small-$o$ cannot be weakened to $O$, and beyond the frontier the estimator is not $\sqrt N$-consistent for $\thetaN$. Three qualifications matter. The boundary regime $m \asymp N^{1/(2^{q+1}r)}$ is extremely slow to reach---for $(2,1)$, $m \asymp N^{1/8}$, so that $m \le 16$ for all $N \le 2^{24}$---and the finite-$m$ bias $-2\theta_0/((m-1)(m-2))$ exceeds its asymptotic form $-2\theta_0 m^{-2}$ by a factor $m^2/((m-1)(m-2))$ that is $1.5$ at $m = 6$; \cref{sec:mc} therefore reports both the limit and the exact finite-$m$ prediction. The sharpness is for this estimator family: it is not a lower bound over all distributed procedures, as pooling $\sum_i X_i$ shows. And the signs alternate: $\beta_2 = -2 < 0 < \beta_3 = 8$, so the higher-order pilot over-corrects where the half-block pilot under-corrects, a reason why finite-sample accuracy is not monotone in $r$. | 49 | +Items (v)--(vi) show that the exponent of \cref{thm:frontier} is sharp for the construction in this model: the small-$o$ cannot be weakened to $O$, and beyond the frontier the estimator is not $\sqrt N$-consistent for $\thetaN$. Three qualifications matter. The boundary regime $m \asymp N^{1/(2^{q+1}r)}$ is extremely slow to reach---for $(2,1)$, $m \asymp N^{1/8}$, so that $m \le 16$ for all $N \le 2^{24}$---and the finite-$m$ bias in \eqref{eq:exp-bias23} exceeds its leading term $-2\theta_0m^{-2}$ by the factor $m^2/((m-1)(m-2))$, which equals $1.5$ at $m = 6$; \cref{sec:mc} therefore reports both the limit and an exact finite-$m$ prediction. The sharpness is for this estimator family: it is not a lower bound over all distributed procedures, as pooling $\sum_i X_i$ shows. And the signs alternate, $\beta_2 = -2 < 0 < \beta_3 = 8$, so the higher-order pilot over-corrects where the half-block pilot under-corrects, one reason why finite-sample accuracy is not monotone in $r$. | ||
| 18 | 50 | ||||
| 19 | Item (iv) also gives an explicit failure gallery. Because the weights have negative entries, $\Jr{r,k}$ can be negative when a sub-block average is small; the pilot then lies outside the basin $(0, 2\thetaN)$ with positive probability for every finite $m$, and the Newton iteration diverges rather than converges. The probability is negligible for $m \ge 32$ and visible for $m \le 8$ in \cref{sec:mc}, where such replications are reported, not discarded. A production implementation would damp or project the step; the theory above describes the undamped map. | 51 | Item (iv) also gives an explicit failure gallery. Because the weights have negative entries, $\Jr{r,k}$ can be negative when a sub-block average is small; the pilot then lies outside the basin $(0, 2\thetaN)$ with positive probability for every finite $m$, and the Newton iteration diverges rather than converges. The probability is negligible for $m \ge 32$ and visible for $m \le 8$ in \cref{sec:mc}, where such replications are reported, not discarded. A production implementation would damp or project the step; the theory above describes the undamped map. | ||
modified
paper/sections/inexact.tex
+30 −13
@@ -1,27 +1,44 @@ | |||||
| 1 | \section{The price of inexact curvature}\label{sec:inexact} | 1 | \section{The price of inexact curvature}\label{sec:inexact} | ||
| 2 | 2 | ||||
| 3 | −The exact protocol uploads $p + p(p+1)/2$ numbers per worker in every derivative phase. Three cheaper variants are common in practice: a \emph{stale} Hessian, equal to the average of the local Hessians evaluated at the local MLEs and shipped once together with the pilot, after which every round costs $p$ numbers; a \emph{single-machine} Hessian evaluated at the current iterate on one worker, as in the surrogate-likelihood approach of \citet{jordan2019communication}; and a \emph{quantized} exact Hessian. The following perturbation theorem covers all three and inexact scores as well. Let $\vartheta$ be the current iterate, $e := \vartheta - \thetaN$, and let the coordinator compute $\tilde\vartheta := \vartheta - \widehat H^{-1}\widehat g$ with $\widehat g = g_N(\vartheta) + \delta_g$. | 3 | +The exact protocol uploads $p + p(p+1)/2$ numbers per worker in every derivative phase. Three cheaper variants are common in practice: a \emph{stale} Hessian, equal to the average of the local Hessians evaluated at the local MLEs and shipped once together with the pilot, after which every round costs $p$ numbers; a \emph{single-machine} Hessian evaluated at the current iterate on one worker, as in the surrogate-likelihood approach of \citet{jordan2019communication}; and a \emph{quantized} exact Hessian. The following perturbation theorem covers all three and inexact scores as well. Let $\vartheta$ be the current iterate, $e = \vartheta - \thetaN$, and let the coordinator compute | ||
| 4 | +\begin{equation}\label{eq:inexact-step} | ||||
| 5 | +\tilde\vartheta = \vartheta - \widehat H^{-1}\,\widehat g, \qquad \widehat g = g_N(\vartheta) + \delta_g . | ||||
| 6 | +\end{equation} | ||||
| 4 | 7 | ||||
| 5 | \begin{theorem}[Inexact Newton steps]\label{thm:inexact} | 8 | \begin{theorem}[Inexact Newton steps]\label{thm:inexact} | ||
| 6 | −Let \cref{ass:lin,ass:bias,ass:newton} hold, $\vartheta \pto \theta_0$, $\norm{\widehat H - H_N(\thetaN)} = \op(\delta_N)$ with $\delta_N \to 0$ and $\norm{\widehat H^{-1}} = \op(1)$. Then | 9 | +Let \cref{ass:lin,ass:bias,ass:newton} hold, $\vartheta \pto \theta_0$, $\norm{\widehat H - H_N(\thetaN)} = \op(\delta_N)$ with $\delta_N \to 0$, and $\norm{\widehat H^{-1}} = \op(1)$. Then | ||
| 7 | \begin{equation}\label{eq:inexact} | 10 | \begin{equation}\label{eq:inexact} | ||
| 8 | \norm{\tilde\vartheta - \thetaN} = \op\big(\norm{e}^2 + \delta_N\norm{e} + \norm{\delta_g}\big). | 11 | \norm{\tilde\vartheta - \thetaN} = \op\big(\norm{e}^2 + \delta_N\norm{e} + \norm{\delta_g}\big). | ||
| 9 | \end{equation} | 12 | \end{equation} | ||
| 10 | −With $q$ such steps from the order-$r$ pilot and a Hessian error $\delta_N$ at every step, $\norm{\iter{r}{q} - \thetaN} = \op(\norm{e_0}\delta_N^{\,q} + \norm{e_0}^{2^q} + \text{cross terms})$, and $\sqrt N(\iter{r}{q} - \thetaN) \pto 0$ provided $\sqrt N\,m^{-r}\delta_N^{\,q} \to 0$ together with the conditions of \cref{thm:frontier}. In particular: | 13 | +With $q$ such steps from the order-$r$ pilot, each with a Hessian error of order $\delta_N$, | ||
| 14 | +\begin{equation}\label{eq:inexact-q} | ||||
| 15 | +\norm{\iter{r}{q} - \thetaN} = \op\big(\norm{e_0}\,\delta_N^{\,q} + \norm{e_0}^{2^q}\big), \qquad e_0 = \pilot{r} - \thetaN, | ||||
| 16 | +\end{equation} | ||||
| 17 | +and $\sqrt N(\iter{r}{q} - \thetaN) \pto 0$ provided $\sqrt N\,m^{-r}\delta_N^{\,q} \to 0$ together with the conditions of \cref{thm:frontier}. In particular: | ||||
| 11 | \begin{enumerate}[label=(\alph*)] | 18 | \begin{enumerate}[label=(\alph*)] | ||
| 12 | −\item (\emph{stale averaged local curvature}) under \cref{ass:third}, $\widehat H = K^{-1}\sum_k H_{D_k}(\thetahat_{D_k})$ has $\delta_N = m^{-1} + N^{-1/2}$, and the sufficient exponent becomes $\alpha = 1 - 1/(2(r+q))$ for $q \ge 1$; | 19 | +\item \emph{Stale averaged local curvature.} Under \cref{ass:third}, $\widehat H = K^{-1}\sum_k H_{D_k}(\thetahat_{D_k})$ satisfies $\delta_N = m^{-1} + N^{-1/2}$, and for $q \ge 1$ the sufficient exponent becomes | ||
| 13 | −\item (\emph{single-machine curvature}) $\widehat H = H_{D_1}(\vartheta)$ has $\delta_N = m^{-1/2}$ (no third-derivative condition needed), and the sufficient exponent becomes $\alpha = 1 - 1/(2r + q)$; | 20 | +\begin{equation}\label{eq:alpha-stale} | ||
| 14 | −\item (\emph{relative quantization}) if the score and the Hessian are transmitted with relative precision $\varepsilon$ (every entry rounded to $b$ significant bits, $\varepsilon = 2^{-b}$), then \eqref{eq:inexact} holds with $\delta_N = \varepsilon$ and $\norm{\delta_g} = \op(\varepsilon\norm{e})$, so that one quantized step from the order-$r$ pilot is first-order equivalent to $\thetaN$ whenever $\sqrt N\,\varepsilon\, m^{-r} \to 0$ in addition to $K = o(N^{\alphafr{r}{1}})$. When $K \sim N^{\alpha}$ with $1 - 1/(2r) \le \alpha < \alphafr{r}{1}$, it suffices that | 21 | +\alpha_{\mathrm{stale}}(r,q) = 1 - \frac{1}{2\,(r+q)} . | ||
| 22 | +\end{equation} | ||||
| 23 | +\item \emph{Single-machine curvature.} $\widehat H = H_{D_1}(\vartheta)$ satisfies $\delta_N = m^{-1/2}$ (no third-derivative condition is needed), and the sufficient exponent becomes | ||||
| 24 | +\begin{equation}\label{eq:alpha-single} | ||||
| 25 | +\alpha_{\mathrm{single}}(r,q) = 1 - \frac{1}{2r+q} . | ||||
| 26 | +\end{equation} | ||||
| 27 | +\item \emph{Relative quantization.} If every entry of the transmitted score and Hessian is rounded to $b$ significant bits ($\varepsilon = 2^{-b}$), then \eqref{eq:inexact} holds with $\delta_N = \varepsilon$ and $\norm{\delta_g} = \op(\varepsilon\norm{e})$, so that one quantized step from the order-$r$ pilot is first-order equivalent to $\thetaN$ whenever $\sqrt N\,\varepsilon\, m^{-r} \to 0$ in addition to $K = o(N^{\alphafr{r}{1}})$. When $K \sim N^{\alpha}$ with $1 - 1/(2r) \le \alpha < \alphafr{r}{1}$, it suffices that | ||||
| 15 | \begin{equation}\label{eq:bits} | 28 | \begin{equation}\label{eq:bits} | ||
| 16 | −b \;\ge\; \Big(\tfrac12 - r(1-\alpha)\Big)\log_2 N + \omega(1), | 29 | +b \;\ge\; \Big(\tfrac12 - r\,(1-\alpha)\Big)\log_2 N + \omega(1), | ||
| 17 | \end{equation} | 30 | \end{equation} | ||
| 18 | −and for $\alpha < 1 - 1/(2r)$ any fixed number of bits suffices (the pilot alone is already first-order equivalent). | 31 | +and for $\alpha < 1 - 1/(2r)$ any fixed number of bits suffices, because the pilot alone is already first-order equivalent. | ||
| 19 | \end{enumerate} | 32 | \end{enumerate} | ||
| 20 | \end{theorem} | 33 | \end{theorem} | ||
| 21 | 34 | ||||
| 22 | −The proof (\cref{app:proofs}) is a matrix-perturbation argument around the integral form of the Newton remainder: $\tilde\vartheta - \thetaN = \widehat H^{-1}[\widehat H - \int_0^1 H_N(\thetaN + te)\,\dd t]\,e - \widehat H^{-1}\delta_g$, and the bracket is bounded by $\norm{\widehat H - H_N(\thetaN)} + \tfrac12\Lambda_N\norm{e}$. The theorem turns the qualitative statements in the literature---the $m^{-1/2}$ factor of \citet[Theorem~3.1]{jordan2019communication}, the $(\beta/\lambda)^2 \asymp m^{-1}$ contraction of averaged local Hessians in \citet[Theorem~2]{shamir2014communication} and \citet{fan2023communication}---into exponents. \Cref{tab:protocols} tabulates them with the communication they require. Case~(b) with $r = 1$ gives $\alpha = 1 - 1/(2+q) = (q+1)/(q+2)$, exactly the rounds frontier that \citet{fan2023communication} state for CEASE up to logarithms, and $q = 1$ gives the $2/3$ of \citet{jordan2019communication}; case~(a) with $(r,q) = (1,1)$ gives $3/4$, the regime of \citet{zhu2021least}. The stale variant has effective depth $r + q$ (linear in $q$) against $2^q r$ for exact curvature: with stale curvature a second round is worth one more local level, with exact curvature it is worth doubling the local order. | 35 | +The proof (\cref{app:proofs}) is a matrix-perturbation argument around the integral form of the Newton remainder, | ||
| 36 | +\begin{equation}\label{eq:inexact-identity} | ||||
| 37 | +\tilde\vartheta - \thetaN = \widehat H^{-1}\Big[\widehat H - \int_0^1 H_N(\thetaN + te)\,\dd t\Big]\,e \;-\; \widehat H^{-1}\delta_g, | ||||
| 38 | +\end{equation} | ||||
| 39 | +whose bracket is bounded by $\norm{\widehat H - H_N(\thetaN)} + \tfrac12\Lambda_N\norm{e}$. The theorem turns the qualitative statements in the literature---the $m^{-1/2}$ factor of \citet[Theorem~3.1]{jordan2019communication}, the $(\beta/\lambda)^2 \asymp m^{-1}$ contraction of averaged local Hessians in \citet[Theorem~2]{shamir2014communication} and \citet{fan2023communication}---into exponents. \Cref{tab:protocols} tabulates them with the communication they require. Case~(b) with $r = 1$ gives $\alpha = (q+1)/(q+2)$, exactly the rounds frontier that \citet{fan2023communication} state for CEASE up to logarithms, and $q = 1$ gives the $2/3$ of \citet{jordan2019communication}; case~(a) with $(r,q) = (1,1)$ gives $3/4$, the regime of \citet{zhu2021least}. The stale variant has effective depth $r + q$, linear in $q$, against $2^q r$ for exact curvature: with stale curvature a second round is worth one more local level; with exact curvature it is worth doubling the local order. | ||||
| 23 | 40 | ||||
| 24 | −Case~(c) is, to our knowledge, new in this form. It says that the precision needed for the derivative uplink is governed by how far the pilot already is from $\thetaN$: a pilot of order $r$ with $K \sim N^\alpha$ has error $m^{-r} = N^{-r(1-\alpha)}$, and a relative error $\varepsilon$ in the step costs $\varepsilon N^{-r(1-\alpha)}$, which is negligible at the $N^{-1/2}$ scale iff \eqref{eq:bits}. For the half-block jackknife at $\alpha = 0.8$ the requirement is $0.1\log_2 N$ bits, i.e.\ two bits of mantissa at $N = 2^{20}$, whereas the plain average at the same $\alpha$ needs $0.3\log_2N = 6$ bits and is outside its own frontier ($3/4$) anyway. The mantissa bits multiply the $p(p+1)/2$ Hessian entries per worker and round, so the saving in bytes is $(64 - b)/64$ of the Hessian payload. The statement is asymptotic in the same sense as \cref{thm:frontier}, and the Monte Carlo ablation of \cref{sec:mc} shows at which $b$ the quantization error becomes visible at finite $N$. | 41 | +Case~(c) is, to our knowledge, new in this form. The precision needed for the derivative uplink is governed by how far the pilot already is from $\thetaN$: a pilot of order $r$ with $K \sim N^\alpha$ has error $m^{-r} = N^{-r(1-\alpha)}$, and a relative error $\varepsilon$ in the step costs $\varepsilon N^{-r(1-\alpha)}$, which is negligible at the $N^{-1/2}$ scale if and only if \eqref{eq:bits} holds. For the half-block jackknife at $\alpha = 0.8$ the requirement is $0.1\log_2 N$ bits, two bits of mantissa at $N = 2^{20}$, whereas the plain average at the same $\alpha$ would need $0.3\log_2N = 6$ bits and is outside its own frontier ($3/4$) in any case. The mantissa bits multiply the $p(p+1)/2$ Hessian entries per worker and round, so the saving is $(64 - b)/64$ of the Hessian payload. The statement is asymptotic in the same sense as \cref{thm:frontier}; the ablation of \cref{sec:mc} shows at which $b$ the quantization error becomes visible at finite $N$. | ||
| 25 | 42 | ||||
| 26 | \begin{table}[t] | 43 | \begin{table}[t] | ||
| 27 | \centering\small | 44 | \centering\small | ||
@@ -34,15 +51,15 @@ Variant & Curvature used by the coordinator & Local fits & Uplink per worker (nu | |||||
| 34 | $(r,0)$ & none & $\sum_j s_j$ & $p$ & $1 - 1/(2r)$ \\ | 51 | $(r,0)$ & none & $\sum_j s_j$ & $p$ & $1 - 1/(2r)$ \\ | ||
| 35 | $(r,q)$ exact & aggregated $H_N(\theta)$ at each iterate & $\sum_j s_j$ & $p + q\,(p + p(p+1)/2)$ & $1 - 1/(2^{q+1}r)$ \\ | 52 | $(r,q)$ exact & aggregated $H_N(\theta)$ at each iterate & $\sum_j s_j$ & $p + q\,(p + p(p+1)/2)$ & $1 - 1/(2^{q+1}r)$ \\ | ||
| 36 | $(r,q)$ stale & $K^{-1}\sum_k H_{D_k}(\thetahat_{D_k})$, shipped once & $\sum_j s_j$ & $p + p(p+1)/2 + q\,p$ & $1 - 1/(2(r+q))$ \\ | 53 | $(r,q)$ stale & $K^{-1}\sum_k H_{D_k}(\thetahat_{D_k})$, shipped once & $\sum_j s_j$ & $p + p(p+1)/2 + q\,p$ & $1 - 1/(2(r+q))$ \\ | ||
| 37 | −$(r,q)$ single & $H_{D_1}(\theta)$ on worker 1 & $\sum_j s_j$ & $p + q\,p$ (+ none for curvature) & $1 - 1/(2r+q)$ \\ | 54 | +$(r,q)$ single & $H_{D_1}(\theta)$ on worker 1 & $\sum_j s_j$ & $p + q\,p$ & $1 - 1/(2r+q)$ \\ | ||
| 38 | −$(r,1)$ quantized & aggregated, $b$ mantissa bits & $\sum_j s_j$ & $p + (p + p(p+1)/2)\,b/64$ (in 64-bit units) & $1 - 1/(4r)$ if $b \ge (\tfrac12 - r(1-\alpha))\log_2 N + \omega(1)$ \\ | 55 | +$(r,1)$ quantized & aggregated, $b$ mantissa bits & $\sum_j s_j$ & $p + (p + p(p+1)/2)\,b/64$ & $1 - 1/(4r)$ if \eqref{eq:bits} \\ | ||
| 39 | AEE \citep{lin2011aggregated} & local Jacobians at local estimates & 1 & $p + p(p+1)/2$ & $< 1/4$ (their Thm 4.2) \\ | 56 | AEE \citep{lin2011aggregated} & local Jacobians at local estimates & 1 & $p + p(p+1)/2$ & $< 1/4$ (their Thm 4.2) \\ | ||
| 40 | CSL \citep{jordan2019communication}, $t$ rounds & machine 1, full surrogate & 1 & $t\,p$ & $1 - 1/(2 + t)$ from the average \\ | 57 | CSL \citep{jordan2019communication}, $t$ rounds & machine 1, full surrogate & 1 & $t\,p$ & $1 - 1/(2 + t)$ from the average \\ | ||
| 41 | exact pooling (exponential families) & sufficient statistic & 1 & $\dim(T)$ & any $K \le N$ \\ | 58 | exact pooling (exponential families) & sufficient statistic & 1 & $\dim(T)$ & any $K \le N$ \\ | ||
| 42 | \bottomrule | 59 | \bottomrule | ||
| 43 | \end{tabular} | 60 | \end{tabular} | ||
| 44 | \begin{tablenotes}\footnotesize | 61 | \begin{tablenotes}\footnotesize | ||
| 45 | −\item Downlink per worker: $qp$ numbers for every variant with $q \ge 1$. Headers, serialization and the pilot's validity flag are excluded; measured bytes appear in \cref{sec:bench}. Exponents are sufficient conditions from \cref{thm:frontier,thm:inexact}; the CSL and AEE rows are the cited authors' results translated to our notation. | 62 | +\item Downlink per worker: $qp$ numbers for every variant with $q \ge 1$. Headers, serialization and the pilot's validity flag are excluded; measured bytes appear in \cref{sec:bench}. Exponents are sufficient conditions from \cref{thm:frontier,thm:inexact}; the CSL and AEE rows are the cited authors' results translated to our notation; the quantized row is in units of 64-bit numbers. | ||
| 46 | \end{tablenotes} | 63 | \end{tablenotes} | ||
| 47 | \end{threeparttable} | 64 | \end{threeparttable} | ||
| 48 | \end{table} | 65 | \end{table} | ||
modified
paper/sections/introduction.tex
+19 −6
@@ -7,12 +7,25 @@ This paper studies the two ideas jointly. Each worker computes the MLE on its bl | |||||
| 7 | \paragraph{Contributions.} | 7 | \paragraph{Contributions.} | ||
| 8 | \begin{enumerate}[leftmargin=1.4em] | 8 | \begin{enumerate}[leftmargin=1.4em] | ||
| 9 | \item \emph{A joint order--round frontier} (\cref{thm:frontier}). Under a uniform stochastic expansion with a bias expansion of order $r$ and local Newton regularity, first-order equivalence to $\thetaN$ holds whenever | 9 | \item \emph{A joint order--round frontier} (\cref{thm:frontier}). Under a uniform stochastic expansion with a bias expansion of order $r$ and local Newton regularity, first-order equivalence to $\thetaN$ holds whenever | ||
| 10 | −\[ | 10 | +\begin{equation}\label{eq:frontier-intro} | ||
| 11 | −K = o\big(N^{\alphafr{r}{q}}\big), \qquad \alphafr{r}{q} := 1 - \frac{1}{2^{q+1} r}. | 11 | +K = o\big(N^{\alphafr{r}{q}}\big), \qquad \alphafr{r}{q} = 1 - \frac{1}{2^{\,q+1}\, r}. | ||
| 12 | −\] | 12 | +\end{equation} | ||
| 13 | −The exponent depends on $(r,q)$ only through the \emph{effective depth} $D = 2^{q} r$: doubling the local order and adding one synchronization round are exchangeable. The half-block jackknife with one Newton step gives $K = o(N^{7/8})$; the plain average with one step gives $3/4$; the quarter-block extrapolation with one step gives $11/12$. The proof keeps the centered random remainder of the local corrections, of order $(m\sqrt{K})^{-1}$, which cannot be dropped even though its mean vanishes. | 13 | +The exponent depends on $(r,q)$ only through the \emph{effective depth} $D = 2^{q} r$, so that doubling the local order and adding one synchronization round are exchangeable. The half-block jackknife with one Newton step gives $K = o(N^{7/8})$; the plain average with one step gives $3/4$; the quarter-block extrapolation with one step gives $11/12$. The proof keeps the centered random remainder of the local corrections, of order $(m\sqrt{K})^{-1}$, which cannot be dropped even though its mean vanishes. | ||
| 14 | −\item \emph{Exact analysis and sharpness} (\cref{thm:exponential}). In the exponential model we derive the exact bias and variance of the corrected local estimators from Gamma and Beta moments, an exact Newton error identity, the basin of attraction of the Newton map, and the boundary behaviour: if $K / N^{\alphafr{r}{q}} \to c \in (0,\infty)$ then $\sqrt{N}(\iter{r}{q} - \thetaN) \pto -\theta_0 (\betar c^r)^{2^q}$ for $q \ge 1$, where $\betar$ is an explicit rational constant ($\beta_2 = -2$, giving the shift $-4\theta_0 c^4$ for the half-block jackknife with one step), and the estimator is not even $\sqrt{N}$-close to $\thetaN$ when $K/N^{\alphafr{r}{q}} \to \infty$. The sufficient exponent is therefore necessary for this construction. | 14 | +\item \emph{Exact analysis and sharpness} (\cref{thm:exponential}). In the exponential model we derive the exact bias and variance of the corrected local estimators, an exact Newton error identity, the basin of attraction of the Newton map, and the boundary behaviour: if $K / N^{\alphafr{r}{q}} \to c \in (0,\infty)$ then | ||
| 15 | −\item \emph{The price of inexact curvature} (\cref{thm:inexact}). If the Newton step uses a Hessian within $\delta_N$ of $H_N(\thetaN)$ in operator norm, the admissible exponent changes in a computable way. Averaged local Hessians evaluated at the local optima (shipped once with the pilot) give $\alpha = 1 - 1/(2(r+q))$; a single machine's Hessian gives $\alpha = 1 - 1/(2r+q)$, which for $r=1$ reproduces the $(t+1)/(t+2)$ rounds frontier of \citet{fan2023communication} and the $2/3$ exponent of \citet{jordan2019communication}; relative quantization of the transmitted score and Hessian to $b$ bits costs nothing asymptotically once $b \ge (\tfrac12 - r(1-\alpha))\log_2 N + \omega(1)$ when $K \sim N^{\alpha}$, so that higher local order permits coarser derivative uplinks. | 15 | +\begin{equation}\label{eq:boundary-intro} | ||
| 16 | +\sqrt{N}\,\big(\iter{r}{q} - \thetaN\big) \;\pto\; -\theta_0\,\big(\betar\, c^{\,r}\big)^{2^q}, \qquad q \ge 1, | ||||
| 17 | +\end{equation} | ||||
| 18 | +where $\betar$ is an explicit rational constant ($\beta_2 = -2$, so that the half-block jackknife with one step has the limiting shift $-4\theta_0 c^4$), and the estimator is not even $\sqrt{N}$-close to $\thetaN$ when $K/N^{\alphafr{r}{q}} \to \infty$. The sufficient exponent is therefore necessary for this construction. | ||||
| 19 | +\item \emph{The price of inexact curvature} (\cref{thm:inexact}). If the Newton step uses a Hessian within $\delta_N$ of $H_N(\thetaN)$ in operator norm, the admissible exponent changes in a computable way: | ||||
| 20 | +\begin{equation}\label{eq:inexact-intro} | ||||
| 21 | +\alpha = 1 - \frac{1}{2(r+q)} \ \text{(stale averaged local Hessians)}, \qquad | ||||
| 22 | +\alpha = 1 - \frac{1}{2r+q} \ \text{(single-machine Hessian)}, | ||||
| 23 | +\end{equation} | ||||
| 24 | +the second of which reproduces, for $r = 1$, the $(t+1)/(t+2)$ rounds frontier of \citet{fan2023communication} and the $2/3$ exponent of \citet{jordan2019communication}. Relative quantization of the transmitted score and Hessian to $b$ bits costs nothing asymptotically once | ||||
| 25 | +\begin{equation}\label{eq:bits-intro} | ||||
| 26 | +b \;\ge\; \Big(\tfrac12 - r\,(1-\alpha)\Big)\log_2 N + \omega(1) \qquad \text{when } K \sim N^{\alpha}, | ||||
| 27 | +\end{equation} | ||||
| 28 | +so that higher local order permits coarser derivative uplinks. | ||||
| 16 | \item \emph{Evidence.} Paired-replication Monte Carlo experiments for exponential, logistic and Poisson likelihoods with $N$ up to $2^{22}$, an exact-model boundary study, ablations over split patterns, Newton rounds and derivative quantization, and timed benchmarks of the full coordinator--worker protocol on up to \benchHosts{} Apple-silicon hosts quantify the finite-sample variance cost of higher-order corrections (which can reverse the asymptotic ranking at moderate $m$), the fraction of local fits that fail, interval coverage, transmitted bytes per round and the measured exchange rate between local fits and synchronization rounds. | 29 | \item \emph{Evidence.} Paired-replication Monte Carlo experiments for exponential, logistic and Poisson likelihoods with $N$ up to $2^{22}$, an exact-model boundary study, ablations over split patterns, Newton rounds and derivative quantization, and timed benchmarks of the full coordinator--worker protocol on up to \benchHosts{} Apple-silicon hosts quantify the finite-sample variance cost of higher-order corrections (which can reverse the asymptotic ranking at moderate $m$), the fraction of local fits that fail, interval coverage, transmitted bytes per round and the measured exchange rate between local fits and synchronization rounds. | ||
| 17 | \end{enumerate} | 30 | \end{enumerate} | ||
| 18 | 31 | ||||
modified
paper/sections/provenance.tex
+3 −3
@@ -1,3 +1,3 @@ | |||||
| 1 | −\section*{Authorship, provenance and AI-use disclosure}\label{sec:provenance} | 1 | +\section*{Provenance and AI-use disclosure}\label{sec:provenance} | ||
| 2 | −\addcontentsline{toc}{section}{Authorship, provenance and AI-use disclosure} | 2 | +\addcontentsline{toc}{section}{Provenance and AI-use disclosure} | ||
| 3 | −This working paper develops an exploratory research note dated 28~October 2026 whose listed authors were Simon-Pierre Boucher and ``GPT-6.1'', an AI co-author designation that the researcher asked to preserve; the note is frozen with checksums in the replication package (\texttt{references/initial\_pamle/}). The present version was prepared by the researcher with Claude (Anthropic) acting as a research and programming assistant under his direction: the literature audit, proofs, code, experiments and text were generated in that collaboration and reviewed by the researcher, who bears sole scientific responsibility. The AI systems have no institutional affiliation, e-mail address or ORCID. Most journals do not accept AI systems as authors; for a journal submission the ``GPT-6.1'' designation should be moved from the author list to this disclosure, which is flagged here rather than silently changed. No funding line is claimed for this project. | 3 | +This working paper develops an exploratory research note of 28~October 2026 whose theoretical content was drafted with the help of a large language model; the note is frozen with checksums in the replication package (\texttt{references/initial\_pamle/}). The present version was prepared by the author with Claude (Anthropic) as a research and programming assistant: the literature audit, proofs, code, experiments and text were produced in that collaboration and reviewed by the author, who bears sole responsibility for the content. No funding is claimed for this project. | ||
modified
paper/sections/results.tex
+12 −2
@@ -20,7 +20,13 @@ Then $\sqrt N\,(\iter{r}{q} - \thetaN) \pto 0$, and consequently $\sqrt N(\iter{ | |||||
| 20 | \end{theorem} | 20 | \end{theorem} | ||
| 21 | 21 | ||||
| 22 | \begin{proof}[Proof sketch] | 22 | \begin{proof}[Proof sketch] | ||
| 23 | −By \eqref{eq:qbound}, $\sqrt N\norm{\iter{r}{q} - \thetaN}$ is bounded by a sum of three terms. The bias term is $\sqrt N\,m^{-r2^q} = N^{1/2}(K/N)^{r2^q}$, which vanishes iff $K^{r2^q} = o(N^{r2^q - 1/2})$, i.e.\ iff $K = o(N^{\alphafr{r}{q}})$. The random term is $\sqrt N(\sqrt K/N)^{2^q}$: for $q = 0$ it equals $\sqrt{K/N}$ and vanishes because $\alphafr{r}{0} < 1$; for $q \ge 1$ it is at most $N^{1/2 - 2^{q-1}} \le N^{-1/2}$ whenever $K \le N$. The full-sample term $\sqrt N N^{-2^q}$ vanishes. Consistency of the pilot, needed to apply \eqref{eq:contraction}, follows from \eqref{eq:pilot} and $m \to \infty$. The limit law follows from \cref{ass:newton} and Slutsky's lemma. \Cref{app:proofs} gives the details, including the event on which all iterates stay in $U$. | 23 | +By \eqref{eq:qbound}, $\sqrt N\norm{\iter{r}{q} - \thetaN}$ is bounded by a sum of three terms, which with $m = N/K$ read | ||
| 24 | +\begin{equation}\label{eq:three-terms} | ||||
| 25 | +\sqrt N\, m^{-r2^q} = N^{1/2}\Big(\frac KN\Big)^{r2^q}, \qquad | ||||
| 26 | +\sqrt N\Big(\frac{\sqrt K}{N}\Big)^{2^q} = N^{1/2}\,K^{2^{q-1}}N^{-2^q}, \qquad | ||||
| 27 | +\sqrt N\, N^{-2^q}. | ||||
| 28 | +\end{equation} | ||||
| 29 | +The first vanishes if and only if $K^{r2^q} = o(N^{r2^q - 1/2})$, that is $K = o(N^{\alphafr{r}{q}})$. The second equals $\sqrt{K/N}$ for $q = 0$ and vanishes because $\alphafr{r}{0} < 1$; for $q \ge 1$ it is at most $N^{1/2 - 2^{q-1}} \le N^{-1/2}$ whenever $K \le N$. The third always vanishes. Consistency of the pilot, needed to apply \eqref{eq:contraction}, follows from \eqref{eq:pilot} and $m \to \infty$. The limit law follows from \cref{ass:newton} and Slutsky's lemma. \Cref{app:proofs} gives the details, including the event on which all iterates stay in $U$. | ||||
| 24 | \end{proof} | 30 | \end{proof} | ||
| 25 | 31 | ||||
| 26 | \begin{table}[t] | 32 | \begin{table}[t] | ||
@@ -47,7 +53,11 @@ $r = 4$ (eighth-block, or $(1,2,3,4)$) & $7/8$ & $15/16$ & $31/32$ & $63/64$ \\ | |||||
| 47 | 53 | ||||
| 48 | \subsection{Inference} | 54 | \subsection{Inference} | ||
| 49 | \begin{corollary}[Inferential validity]\label{cor:inference} | 55 | \begin{corollary}[Inferential validity]\label{cor:inference} | ||
| 50 | −Under the conditions of \cref{thm:frontier}, let $\widehat\Sigma_N \pto \Sigma_\psi$. Then for every $j$ and level $1-\tau$ the interval $\iter{r}{q,j} \pm z_{1-\tau/2}\,(\widehat\Sigma_{N,jj}/N)^{1/2}$ has asymptotic coverage $1 - \tau$. Under correct specification $\widehat\Sigma_N = -H_N(\iter{r}{q})^{-1}$, which is available at no extra communication from the last derivative phase (for $q \ge 1$); under misspecification the sandwich $H_N^{-1}\,\widehat J_N\,H_N^{-1}$ with $\widehat J_N = K^{-1}\sum_k\abs{D_k}^{-1}\sum_{i\in D_k}\nabla\log f(X_i;\theta)\nabla\log f(X_i;\theta)^\top$ requires one additional $p(p+1)/2$ upload per worker in that phase. | 56 | +Under the conditions of \cref{thm:frontier}, let $\widehat\Sigma_N \pto \Sigma_\psi$. Then for every coordinate $j$ and level $1-\tau$ the interval | ||
| 57 | +\begin{equation}\label{eq:wald} | ||||
| 58 | +\iter{r}{q,j} \;\pm\; z_{1-\tau/2}\,\sqrt{\widehat\Sigma_{N,jj}/N} | ||||
| 59 | +\end{equation} | ||||
| 60 | +has asymptotic coverage $1 - \tau$. Under correct specification $\widehat\Sigma_N = -H_N(\iter{r}{q})^{-1}$, which is available at no extra communication from the last derivative phase (for $q \ge 1$); under misspecification the sandwich $H_N^{-1}\,\widehat J_N\,H_N^{-1}$ with $\widehat J_N = K^{-1}\sum_k\abs{D_k}^{-1}\sum_{i\in D_k}\nabla\log f(X_i;\theta)\nabla\log f(X_i;\theta)^\top$ requires one additional $p(p+1)/2$ upload per worker in that phase. | ||||
| 51 | \end{corollary} | 61 | \end{corollary} | ||
| 52 | The corollary says nothing about estimators outside the frontier: when $K/N^{\alphafr{r}{q}}$ does not vanish, the point estimator has a non-centered limit (\cref{thm:exponential}) and the interval under-covers, as the coverage tables of \cref{sec:mc} show for the plain average and the one-step estimator at small $m$. | 62 | The corollary says nothing about estimators outside the frontier: when $K/N^{\alphafr{r}{q}}$ does not vanish, the point estimator has a non-centered limit (\cref{thm:exponential}) and the interval under-covers, as the coverage tables of \cref{sec:mc} show for the plain average and the one-step estimator at small $m$. | ||
| 53 | 63 | ||||
modified
paper/sections/robustness.tex
+14 −2
@@ -4,8 +4,20 @@ Both extensions are known in the literature \citep{zhu2021least,gu2023heterogene | |||||
| 4 | 4 | ||||
| 5 | \begin{proposition}[Unequal block sizes and pseudo-likelihood]\label{prop:robust} | 5 | \begin{proposition}[Unequal block sizes and pseudo-likelihood]\label{prop:robust} | ||
| 6 | \begin{enumerate}[label=(\roman*)] | 6 | \begin{enumerate}[label=(\roman*)] | ||
| 7 | −\item Let the blocks have sizes $m_1,\dots,m_K$ with $\sum_k m_k = N$, $m_{\min} \to \infty$ and each $s_j \mid m_k$. With the size weights $m_k/N$ in the pilot, $\pilot{r} - \thetaN = \op(m_{\mathrm{eff}}^{-r} + \sqrt K/N + N^{-1})$ with $m_{\mathrm{eff}} := (N^{-1}\sum_k m_k^{1-r})^{-1/r} \ge m_{\min}$, and \cref{thm:frontier} holds with $m$ replaced by $m_{\mathrm{eff}}$ (the aggregated derivatives are size-weighted and equal the full-sample derivatives, so the Newton steps are unchanged). With equal weights $1/K$ the influence term is not preserved: $K^{-1}\sum_k U_k - U_N = \sum_i c_i\psi(X_i)$ with $c_i = 1/(Km_{k(i)}) - 1/N$, a centered term of order $(\sum_i c_i^2)^{1/2}$ that is of exact order $N^{-1/2}$ unless the sizes are asymptotically equal; the equal-weight pilot is therefore not first-order equivalent for $q = 0$, while for $q \ge 1$ the discrepancy is of order $N^{-1/2}$ and is absorbed by the first Newton step. | 7 | +\item Let the blocks have sizes $m_1,\dots,m_K$ with $\sum_k m_k = N$, $m_{\min} \to \infty$ and each $s_j \mid m_k$. With the size weights $m_k/N$ in the pilot, | ||
| 8 | −\item If the model is misspecified and $\theta_0$ is the pseudo-true value, all statements hold with $\psi = -\bar H(\theta_0)^{-1}\nabla\log f(X;\theta_0)$ and $\Sigma_\psi = \bar H(\theta_0)^{-1}\bar J(\theta_0)\bar H(\theta_0)^{-1}$, provided \cref{ass:lin,ass:bias,ass:newton} are read for the pseudo-MLE. The estimator is then first-order equivalent to the centralized pseudo-MLE and Wald intervals require the sandwich of \cref{cor:inference}; the observed-information intervals are not valid. | 8 | +\begin{equation}\label{eq:unequal-pilot} | ||
| 9 | +\pilot{r} - \thetaN = \op\Big(m_{\mathrm{eff}}^{-r} + \frac{\sqrt K}{N} + \frac1N\Big), \qquad m_{\mathrm{eff}} := \Big(\frac1N\sum_{k=1}^K m_k^{\,1-r}\Big)^{-1/r} \ \ge\ m_{\min}, | ||||
| 10 | +\end{equation} | ||||
| 11 | +and \cref{thm:frontier} holds with $m$ replaced by $m_{\mathrm{eff}}$; the aggregated derivatives are size-weighted and equal the full-sample derivatives, so the Newton steps are unchanged. With equal weights $1/K$ the influence term is not preserved: | ||||
| 12 | +\begin{equation}\label{eq:unequal-naive} | ||||
| 13 | +\frac1K\sum_{k=1}^K U_k - U_N = \sum_{i=1}^N c_i\,\psi(X_i), \qquad c_i = \frac{1}{K\,m_{k(i)}} - \frac1N, \qquad \sum_{i=1}^N c_i^2 = \frac{1}{K^2}\sum_{k=1}^K\frac{1}{m_k} - \frac1N \ \ge\ 0, | ||||
| 14 | +\end{equation} | ||||
| 15 | +with equality only for equal sizes; the discrepancy is of exact order $N^{-1/2}$ when the sizes are bounded away from equality, so the equal-weight pilot is not first-order equivalent for $q = 0$, while for $q \ge 1$ the first Newton step squares it to $\op(N^{-1})$. | ||||
| 16 | +\item If the model is misspecified and $\theta_0$ is the pseudo-true value, all statements hold with | ||||
| 17 | +\begin{equation}\label{eq:sandwich} | ||||
| 18 | +\psi(x) = -\bar H(\theta_0)^{-1}\,\nabla_\theta\log f(x;\theta_0), \qquad \Sigma_\psi = \bar H(\theta_0)^{-1}\,\bar J(\theta_0)\,\bar H(\theta_0)^{-1}, | ||||
| 19 | +\end{equation} | ||||
| 20 | +provided \cref{ass:lin,ass:bias,ass:newton} are read for the pseudo-MLE. The estimator is then first-order equivalent to the centralized pseudo-MLE and Wald intervals require the sandwich of \cref{cor:inference}; observed-information intervals are not valid. | ||||
| 9 | \end{enumerate} | 21 | \end{enumerate} | ||
| 10 | \end{proposition} | 22 | \end{proposition} | ||
| 11 | 23 | ||||
added
results/figures/fig_exponents.pdf
+0 −0
Binary file not shown.
added
results/raw/ablation_exponential_N3145728_m192.csv
+19469 −0
Large diff (19,469 lines) is not loaded by default to keep the page fast.Load diff
Diff truncated — file too large.