WP28: typographic pass (displayed equations, narrow tables, overfull boxes)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
12 changed files +19,849 −38
modified
paper/appendices/exponential.tex
+16 −4
@@ -5,7 +5,7 @@ Let $Y = \theta_0\bar X_n$, so that $nY \sim \mathrm{Gamma}(n,1)$, $\thetahat_n | ||
| 5 | 5 | \begin{proof}[Proof of \cref{thm:exponential}(i)] |
| 6 | 6 | From the Gamma moments, |
| 7 | 7 | \begin{align*} |
| 8 | −\E\thetahat_n &= \theta_0\,\E Y^{-1} = \theta_0\,\frac{n}{n-1}, & | |
| 8 | +\E\thetahat_n &= \theta_0\,\E Y^{-1} = \theta_0\,\frac{n}{n-1}, \\ | |
| 9 | 9 | \Var\thetahat_n &= \theta_0^2\big(\E Y^{-2} - (\E Y^{-1})^2\big) = \theta_0^2\,\frac{n^2}{(n-1)^2(n-2)},\\ |
| 10 | 10 | \E R_n &= \theta_0\Big(\frac{n}{n-1} - 2 + 1\Big) = \frac{\theta_0}{n-1} = \theta_0\sum_{a\ge1}n^{-a}, |
| 11 | 11 | \end{align*} |
@@ -26,11 +26,19 @@ where the terms $a < r$ vanish by \eqref{eq:vandermonde} and the leading term is | ||
| 26 | 26 | \begin{align*} |
| 27 | 27 | s = (1,2):\qquad & \theta_0\Big[\frac{2}{m-1} - \frac{2}{m-2}\Big] = -\frac{2\theta_0}{(m-1)(m-2)},\\ |
| 28 | 28 | s = (1,2,4):\qquad & \theta_0\Big[\frac{8/3}{m-1} - \frac{4}{m-2} + \frac{4/3}{m-4}\Big] = \frac{8\theta_0}{(m-1)(m-2)(m-4)} . |
| 29 | −\end{align*} Both identities, and $\beta_2 = -2$, $\beta_3 = 8$, $\beta_4 = -64$, are verified in rational arithmetic in \texttt{theory/symbolic\_checks.py}. | |
| 29 | +\end{align*} Both identities, and the constants $\beta_2 = -2$, $\beta_3 = 8$ and $\beta_4 = -64$, are verified in rational arithmetic in \texttt{theory/symbolic\_checks.py}. | |
| 30 | 30 | \end{proof} |
| 31 | 31 | |
| 32 | 32 | \begin{proof}[Proof of (iii)] |
| 33 | −Let $h = m/2$ and $S_A, S_B$ be the sums over the two halves, $S = S_A + S_B \sim \mathrm{Gamma}(m,\theta_0)$ and $W = S_A/S \sim \mathrm{Beta}(h,h)$, independent of $S$. Then $\thetahat_{D_k} = m/S$, $\thetahat_A = h/(SW)$, $\thetahat_B = h/(S(1-W))$ and $\Jr{2,k} = S^{-1}G(W)$ with $G(W) = 2m - \tfrac h2[W^{-1} + (1-W)^{-1}]$. Independence gives $\E\Jr{2,k} = \E S^{-1}\,\E G(W)$ and $\E\Jr{2,k}^2 = \E S^{-2}\,\E G(W)^2$, with $\E S^{-k} = \theta_0^k\Gamma(m-k)/\Gamma(m)$ and the Beta inverse moments $\E W^{-a}(1-W)^{-c} = \Gamma(h-a)\Gamma(h-c)\Gamma(2h)/(\Gamma(h)^2\Gamma(2h-a-c))$ for $a, c < h$. Simplifying the resulting rational functions (SymPy, check S6) gives the stated $\Var\Jr{2,k}$ and the expansion $\theta_0^2(m^{-1} + 4m^{-2} + O(m^{-3}))$; the same expansion holds for $\Var\thetahat_m = \theta_0^2m^2/((m-1)^2(m-2))$. The condition $m > 4$ ensures $h > 2$ so that all inverse moments exist. | |
| 33 | +Let $h = m/2$ and let $S_A, S_B$ be the sums over the two halves, $S = S_A + S_B \sim \mathrm{Gamma}(m,\theta_0)$ and $W = S_A/S \sim \mathrm{Beta}(h,h)$, independent of $S$. Then | |
| 34 | +\[ | |
| 35 | +\thetahat_{D_k} = \frac mS, \qquad \thetahat_A = \frac{h}{SW}, \qquad \thetahat_B = \frac{h}{S(1-W)}, \qquad | |
| 36 | +\Jr{2,k} = \frac{G(W)}{S}, \quad G(W) = 2m - \frac h2\Big[\frac1W + \frac{1}{1-W}\Big]. | |
| 37 | +\] Independence gives $\E\Jr{2,k} = \E S^{-1}\,\E G(W)$ and $\E\Jr{2,k}^2 = \E S^{-2}\,\E G(W)^2$, with | |
| 38 | +\[ | |
| 39 | +\E S^{-k} = \theta_0^k\,\frac{\Gamma(m-k)}{\Gamma(m)}, \qquad | |
| 40 | +\E\big[W^{-a}(1-W)^{-c}\big] = \frac{\Gamma(h-a)\,\Gamma(h-c)\,\Gamma(2h)}{\Gamma(h)^2\,\Gamma(2h-a-c)}, \qquad a, c < h . | |
| 41 | +\] Simplifying the resulting rational functions (SymPy, check S6) gives the stated $\Var\Jr{2,k}$ and the expansion $\theta_0^2(m^{-1} + 4m^{-2} + O(m^{-3}))$; the same expansion holds for $\Var\thetahat_m = \theta_0^2m^2/((m-1)^2(m-2))$. The condition $m > 4$ ensures $h > 2$ so that all inverse moments exist. | |
| 34 | 42 | \end{proof} |
| 35 | 43 | |
| 36 | 44 | \begin{proof}[Proof of (iv)] |
@@ -48,5 +56,9 @@ Let $\alpha = \alphafr{r}{q}$ and $K/N^\alpha \to c \in (0,\infty]$, with $m \to | ||
| 48 | 56 | \[ |
| 49 | 57 | N^{1/2^{q+1}}\,m^{-r} = N^{1/2^{q+1}}\,(K/N^{\alpha})^r\,N^{-r(1-\alpha)} = (K/N^\alpha)^r \to c^r, |
| 50 | 58 | \] |
| 51 | −while $N^{1/2^{q+1}}\sqrt K/N \le N^{1/2^{q+1} + \alpha/2 - 1} \to 0$ because $\alpha < 1$ and $1/2^{q+1} \le 1/2$ give $1/2^{q+1} + \alpha/2 - 1 < 0$; likewise $N^{1/2^{q+1}}N^{-1} \to 0$. Hence $N^{1/2^{q+1}}(\pilot{r} - \thetaN) \pto \betar\theta_0c^r$ when $c < \infty$ and $\pto \pm\infty$ when $c = \infty$. For $q = 0$, $N^{1/2^{q+1}} = \sqrt N$ and the claim follows. For $q \ge 1$, (iv) gives $\sqrt N(\iter{r}{q} - \thetaN) = -\sqrt N(\pilot{r} - \thetaN)^{2^q}/\thetaN^{2^q-1} = -[N^{1/2^{q+1}}(\pilot{r} - \thetaN)]^{2^q}/\thetaN^{2^q - 1}$, and $\thetaN \pto \theta_0$ with the continuous mapping theorem yields $-\theta_0(\betar c^r)^{2^q}$ when $c < \infty$ and divergence to $-\infty$ in probability when $c = \infty$. Combining with $\sqrt N(\thetaN - \theta_0)\dto\mathcal N(0,\theta_0^2)$ proves the displayed limit law for $(2,1)$. Strictly, (iv) requires the pilot to be positive for the iteration to be meaningful; $P(\pilot{r} \le 0) \to 0$ since $\pilot{r} \pto \theta_0 > 0$. | |
| 59 | +while $N^{1/2^{q+1}}\sqrt K/N \le N^{1/2^{q+1} + \alpha/2 - 1} \to 0$ because $\alpha < 1$ and $1/2^{q+1} \le 1/2$ give $1/2^{q+1} + \alpha/2 - 1 < 0$; likewise $N^{1/2^{q+1}}N^{-1} \to 0$. Hence $N^{1/2^{q+1}}(\pilot{r} - \thetaN) \pto \betar\theta_0c^r$ when $c < \infty$ and $\pto \pm\infty$ when $c = \infty$. For $q = 0$, $N^{1/2^{q+1}} = \sqrt N$ and the claim follows. For $q \ge 1$, (iv) gives | |
| 60 | +\[ | |
| 61 | +\sqrt N\big(\iter{r}{q} - \thetaN\big) = -\frac{\sqrt N\,(\pilot{r} - \thetaN)^{2^q}}{\thetaN^{\,2^q-1}} = -\frac{\big[N^{1/2^{q+1}}(\pilot{r} - \thetaN)\big]^{2^q}}{\thetaN^{\,2^q - 1}}, | |
| 62 | +\] | |
| 63 | +and $\thetaN \pto \theta_0$ with the continuous mapping theorem yields $-\theta_0(\betar c^r)^{2^q}$ when $c < \infty$ and divergence to $-\infty$ in probability when $c = \infty$. Combining with $\sqrt N(\thetaN - \theta_0)\dto\mathcal N(0,\theta_0^2)$ proves the displayed limit law for $(2,1)$. Strictly, (iv) requires the pilot to be positive for the iteration to be meaningful; $P(\pilot{r} \le 0) \to 0$ since $\pilot{r} \pto \theta_0 > 0$. | |
| 52 | 64 | \end{proof} |
modified
paper/appendices/proofs.tex
+16 −8
@@ -3,7 +3,10 @@ | ||
| 3 | 3 | Throughout, $S_{j\ell,k}$ ($\ell = 1,\dots,s_j$) denotes the $\ell$-th sub-block of level $j$ of worker $k$, of size $m/s_j$; $U_S = \abs{S}^{-1}\sum_{i\in S}\psi(X_i)$; $w$ solves \eqref{eq:vandermonde}. |
| 4 | 4 | |
| 5 | 5 | \begin{lemma}[Influence preservation]\label{lem:influence} |
| 6 | −For every $k$ and $j$, $s_j^{-1}\sum_{\ell=1}^{s_j} U_{S_{j\ell,k}} = U_k$, and hence $\sum_j w_j\, s_j^{-1}\sum_\ell U_{S_{j\ell,k}} = U_k$. | |
| 6 | +For every $k$ and $j$, | |
| 7 | +\[ | |
| 8 | +\frac1{s_j}\sum_{\ell=1}^{s_j} U_{S_{j\ell,k}} = U_k, \qquad\text{and hence}\qquad \sum_{j=1}^r w_j\,\frac1{s_j}\sum_{\ell=1}^{s_j} U_{S_{j\ell,k}} = U_k . | |
| 9 | +\] | |
| 7 | 10 | \end{lemma} |
| 8 | 11 | \begin{proof} |
| 9 | 12 | The sub-blocks of level $j$ partition $D_k$ into $s_j$ sets of equal size $m/s_j$, so |
@@ -72,9 +75,10 @@ Let $e = \vartheta - \thetaN$. Since $U$ is convex, $\thetaN + te \in U$ for $t\ | ||
| 72 | 75 | g_N(\vartheta) = \int_0^1 H_N(\thetaN + te)\,e\,\dd t . |
| 73 | 76 | \] |
| 74 | 77 | Hence |
| 75 | −\[ | |
| 76 | −\mathcal N_N(\vartheta) - \thetaN = e - H_N(\vartheta)^{-1}g_N(\vartheta) = H_N(\vartheta)^{-1}\int_0^1\big[H_N(\vartheta) - H_N(\thetaN + te)\big]\,e\,\dd t . | |
| 77 | −\] | |
| 78 | +\begin{align*} | |
| 79 | +\mathcal N_N(\vartheta) - \thetaN &= e - H_N(\vartheta)^{-1}g_N(\vartheta) \\ | |
| 80 | +&= H_N(\vartheta)^{-1}\int_0^1\big[H_N(\vartheta) - H_N(\thetaN + te)\big]\,e\,\dd t . | |
| 81 | +\end{align*} | |
| 78 | 82 | The Lipschitz bound gives $\norm{H_N(\vartheta) - H_N(\thetaN + te)} \le \Lambda_N(1-t)\norm{e}$, and $\int_0^1(1-t)\,\dd t = 1/2$. |
| 79 | 83 | \end{proof} |
| 80 | 84 | |
@@ -121,12 +125,16 @@ which gives \eqref{eq:inexact}. Iterating with a fixed $\widehat H$ (or with a f | ||
| 121 | 125 | \[ |
| 122 | 126 | H_{D_k}(\thetahat_{D_k}) - H_{D_k}(\theta_0) = \nabla H_{D_k}(\theta_0)[\Delta_k] + r_k, \qquad \norm{r_k} \le \tfrac12\bar L_k\norm{\Delta_k}^2, |
| 123 | 127 | \] |
| 124 | −with $\bar L_k$ the block average of $L(X_i)$. By Cauchy--Schwarz and \cref{ass:third}, $\E\norm{r_k} \le \tfrac12(\E\bar L_k^2)^{1/2}(\E\norm{\Delta_k}^4)^{1/2} = O(m^{-1})$. Write $\nabla H_{D_k}(\theta_0) = T + V_k$ with $T = \E\nabla h(X;\theta_0)$ and $V_k$ a centered block average with $\E\norm{V_k}^2 = O(m^{-1})$, and $\Delta_k = U_k + R_{D_k}$. Then $K^{-1}\sum_k T[U_k] = T[U_N] = \op(N^{-1/2})$; $K^{-1}\sum_k T[R_{D_k}] = O_p(m^{-1})$ by \cref{ass:lin}; and $\E\norm{V_k[U_k]} \le (\E\norm{V_k}^2)^{1/2}(\E\norm{U_k}^2)^{1/2} = O(m^{-1})$. Collecting terms, $\delta_N = m^{-1} + N^{-1/2}$, and the condition $\sqrt N m^{-r}m^{-q} \to 0$ reads $K = o(N^{1 - 1/(2(r+q))})$; the $N^{-1/2}$ part of $\delta_N$ never binds. Since $r + q \le 2^q r$ for $q \ge 1$, this condition implies that of \cref{thm:frontier}. | |
| 128 | +with $\bar L_k$ the block average of $L(X_i)$. By Cauchy--Schwarz and \cref{ass:third}, | |
| 129 | +\[ | |
| 130 | +\E\norm{r_k} \le \tfrac12\big(\E\bar L_k^2\big)^{1/2}\big(\E\norm{\Delta_k}^4\big)^{1/2} = O(m^{-1}). | |
| 131 | +\] Write $\nabla H_{D_k}(\theta_0) = T + V_k$ with $T = \E\nabla h(X;\theta_0)$ and $V_k$ a centered block average with $\E\norm{V_k}^2 = O(m^{-1})$, and $\Delta_k = U_k + R_{D_k}$. Then $K^{-1}\sum_k T[U_k] = T[U_N] = \op(N^{-1/2})$; $K^{-1}\sum_k T[R_{D_k}] = O_p(m^{-1})$ by \cref{ass:lin}; and $\E\norm{V_k[U_k]} \le (\E\norm{V_k}^2)^{1/2}(\E\norm{U_k}^2)^{1/2} = O(m^{-1})$. Collecting terms, $\delta_N = m^{-1} + N^{-1/2}$, and the condition $\sqrt N m^{-r}m^{-q} \to 0$ reads $K = o(N^{1 - 1/(2(r+q))})$; the $N^{-1/2}$ part of $\delta_N$ never binds. Since $r + q \le 2^q r$ for $q \ge 1$, this condition implies that of \cref{thm:frontier}. | |
| 125 | 132 | |
| 126 | 133 | (b) Decompose |
| 127 | −\[ | |
| 128 | −H_{D_1}(\vartheta) - H_N(\thetaN) = \big[H_{D_1}(\vartheta) - H_{D_1}(\theta_0)\big] + \big[H_{D_1}(\theta_0) - \bar H(\theta_0)\big] + \big[\bar H(\theta_0) - H_N(\theta_0)\big] + \big[H_N(\theta_0) - H_N(\thetaN)\big], | |
| 129 | −\] | |
| 134 | +\begin{align*} | |
| 135 | +H_{D_1}(\vartheta) - H_N(\thetaN) &= \big[H_{D_1}(\vartheta) - H_{D_1}(\theta_0)\big] + \big[H_{D_1}(\theta_0) - \bar H(\theta_0)\big] \\ | |
| 136 | +&\quad + \big[\bar H(\theta_0) - H_N(\theta_0)\big] + \big[H_N(\theta_0) - H_N(\thetaN)\big], | |
| 137 | +\end{align*} | |
| 130 | 138 | where $\bar H(\theta_0) = \E h(X;\theta_0)$. The first and last brackets are $\op(\norm{\vartheta - \theta_0} + N^{-1/2})$ by the Lipschitz property (applied to $H_{D_1}$, for which the same bound is assumed as for $H_N$ on $U$, with probability tending to one); the second is $\op(m^{-1/2})$ and the third $\op(N^{-1/2})$ by the central limit theorem for block averages of $h(X_i;\theta_0)$. Hence $\delta_N = m^{-1/2} + \norm{\vartheta - \theta_0}$, and along the iteration $\norm{\iter{}{j} - \theta_0} = \op(N^{-1/2} + e_j)$, so the binding condition is $\sqrt N m^{-r}m^{-q/2} \to 0$, i.e.\ $K = o(N^{1 - 1/(2r+q)})$. |
| 131 | 139 | |
| 132 | 140 | (c) Rounding each entry of $H_N(\vartheta)$ to $b$ significant bits yields $\widehat H$ with $\abs{\widehat H_{ab} - H_{N,ab}} \le 2^{-b}\abs{H_{N,ab}}$, hence $\norm{\widehat H - H_N(\vartheta)} \le p\,2^{-b}\norm{H_N(\vartheta)}_{\max}$, and $\norm{H_N(\vartheta) - H_N(\thetaN)} \le \Lambda_N\norm{e}$; so $\delta_N = O_p(\varepsilon) + \op(\norm{e})$, the second part being absorbed in the $\norm{e}^2$ term. Likewise $\norm{\delta_g} \le p\,\varepsilon\,\norm{g_N(\vartheta)}_{\max} \le p\,\varepsilon\,\norm{\bar H}\norm{e} = \op(\varepsilon\norm{e})$. Inserting into \eqref{eq:inexact} with $\norm{e_0} = \op(m^{-r} + \sqrt K/N + N^{-1})$, first-order equivalence requires $\sqrt N\varepsilon m^{-r} \to 0$ (the other two terms vanish for $\varepsilon \le 1$). With $K \sim N^\alpha$, $m^{-r} = N^{-r(1-\alpha)}$ up to constants, so $\sqrt N\varepsilon m^{-r} \to 0$ iff $\log_2(1/\varepsilon) = b \ge (\tfrac12 - r(1-\alpha))\log_2 N + \omega(1)$ when $\tfrac12 - r(1-\alpha) \ge 0$, i.e.\ $\alpha \ge 1 - 1/(2r)$; otherwise $\sqrt N m^{-r} \to 0$ already and any $b \ge 0$ works. |
modified
paper/appendices/replication.tex
+15 −9
@@ -3,18 +3,24 @@ | ||
| 3 | 3 | Everything reported is regenerated from committed raw outputs by the scripts in \texttt{experiments/}; no number in the tables or figures is typed by hand. The directory \texttt{references/initial\_pamle/} contains the prior draft, its archive and SHA-256 checksums; its Monte Carlo (seed 20261028, $N = 2^{20}$, $m \in \{256,64,16\}$, 500 replications) was re-run and matched to $5\times10^{-13}$ before any change was made, and the test \texttt{tests/test\_reproducibility.py} reproduces its summary table from the new code. |
| 4 | 4 | |
| 5 | 5 | \paragraph{Commands.} |
| 6 | −\begin{verbatim} | |
| 6 | +{\footnotesize\begin{verbatim} | |
| 7 | 7 | python3 -m pip install -r requirements.txt |
| 8 | 8 | pytest -q # unit tests, exact identities, protocol |
| 9 | 9 | python3 theory/symbolic_checks.py # SymPy verification of all identities |
| 10 | 10 | python3 experiments/run_monte_carlo.py --config experiments/configs/main_exponential.yaml |
| 11 | −python3 experiments/scatter_cluster.py --configs experiments/configs/main_logistic.yaml \ | |
| 12 | − experiments/configs/main_poisson.yaml experiments/configs/robustness.yaml --launch | |
| 13 | −python3 experiments/run_boundary.py; python3 experiments/run_ablations.py --model logistic | |
| 11 | +python3 experiments/scatter_cluster.py --launch \ | |
| 12 | + --configs experiments/configs/main_logistic.yaml \ | |
| 13 | + experiments/configs/main_poisson.yaml \ | |
| 14 | + experiments/configs/robustness.yaml # cluster | |
| 15 | +python3 experiments/run_boundary.py | |
| 16 | +python3 experiments/run_ablations.py --model logistic | |
| 14 | 17 | python3 experiments/run_skewed.py |
| 15 | −python3 experiments/run_benchmarks.py --config experiments/configs/benchmarks.yaml \ | |
| 16 | − --out results/raw/benchmarks.jsonl # requires the cluster | |
| 17 | −python3 experiments/process_results.py --figures; python3 experiments/make_tables.py | |
| 18 | −bash scripts/compile_paper.sh; bash scripts/validate_pdf.sh | |
| 19 | −\end{verbatim} | |
| 18 | +python3 experiments/run_benchmarks.py \ | |
| 19 | + --config experiments/configs/benchmarks.yaml \ | |
| 20 | + --out results/raw/benchmarks.jsonl # cluster | |
| 21 | +python3 experiments/process_results.py --figures | |
| 22 | +python3 experiments/make_tables.py | |
| 23 | +bash scripts/compile_paper.sh | |
| 24 | +bash scripts/validate_pdf.sh | |
| 25 | +\end{verbatim}} | |
| 20 | 26 | Seeds are derived from \texttt{seed\_base = 20261028} and a hash of the cell description; every raw file has a JSON manifest with git commit, Python and NumPy versions, host, wall time and the SHA-256 of the output (\texttt{experiments/EXPERIMENT\_MANIFEST.csv}). Monte Carlo cells were run on the laptop (Apple M5 Max, Python 3.14.4, NumPy 2.4.4) and on 18 cluster nodes (Python 3.9.6/3.14, NumPy 2.0.2/2.5.4); floating-point results differ across these environments at the level of $10^{-12}$ relative, far below the Monte Carlo standard errors reported. |
modified
paper/main.pdf
+0 −0
Binary file not shown.
modified
paper/sections/construction.tex
+10 −6
@@ -8,20 +8,20 @@ Let $V$ be the $r\times r$ Vandermonde matrix with entries $V_{aj} = s_j^{a}$, $ | ||
| 8 | 8 | i.e.\ $w = V^{-1}e_1$. For $s = (1,2)$, $w = (2,-1)$; for $(1,2,4)$, $w = (8/3,-2,1/3)$; for $(1,2,4,8)$, $w = (64/21,-8/3,2/3,-1/21)$; for $(1,2,3)$, $w = (3,-3,1)$. Two constants summarize the cost of a pattern: the \emph{leading bias coefficient} $\betar := \sum_j w_j s_j^{\,r}$, which multiplies the first uncancelled bias term, and the \emph{remainder amplification} $\Amp := \sum_j \abs{w_j}\, s_j$, which multiplies the $L_2$ bound of the stochastic remainder. \Cref{tab:weights} lists them; the weights are computed exactly in rational arithmetic in the replication code and the moment constraints are verified to machine precision. |
| 9 | 9 | |
| 10 | 10 | \begin{table}[t] |
| 11 | −\centering\small | |
| 11 | +\centering\scriptsize | |
| 12 | 12 | \begin{threeparttable} |
| 13 | 13 | \caption{Split patterns, weights and constants.}\label{tab:weights} |
| 14 | −\begin{tabular}{@{}lclrrrrr@{}} | |
| 14 | +\begin{tabular}{@{}lcp{3.6cm}rrrrr@{}} | |
| 15 | 15 | \toprule |
| 16 | 16 | Pattern $s$ & $r$ & Weights $w$ & $\betar$ & $\Amp$ & $\sum_j\abs{w_j}$ & $\kappa(V)$ & Local fits \\ |
| 17 | 17 | \midrule |
| 18 | 18 | $(1)$ & 1 & $1$ & $1$ & $1$ & $1$ & $1$ & 1 \\ |
| 19 | 19 | $(1,2)$ & 2 & $2,\,-1$ & $-2$ & $4$ & $3$ & $6.9$ & 3 \\ |
| 20 | 20 | $(1,3)$ & 2 & $3/2,\,-1/2$ & $-3$ & $3$ & $2$ & $5.8$ & 4 \\ |
| 21 | −$(1,2,4)$ & 3 & $8/3,\,-2,\,1/3$ & $8$ & $8$ & $5$ & $80$ & 7 \\ | |
| 21 | +$(1,2,4)$ & 3 & $8/3$, $-2$, $1/3$ & $8$ & $8$ & $5$ & $80$ & 7 \\ | |
| 22 | 22 | $(1,2,3)$ & 3 & $3,\,-3,\,1$ & $6$ & $12$ & $7$ & $71$ & 6 \\ |
| 23 | −$(1,2,4,8)$ & 4 & $64/21,\,-8/3,\,2/3,\,-1/21$ & $-64$ & $80/7$ & $6.4$ & $3284$ & 15 \\ | |
| 24 | −$(1,2,3,4)$ & 4 & $4,\,-6,\,4,\,-1$ & $-24$ & $32$ & $15$ & $1171$ & 10 \\ | |
| 23 | +$(1,2,4,8)$ & 4 & $64/21$, $-8/3$, $2/3$, $-1/21$ & $-64$ & $80/7$ & $6.4$ & $3284$ & 15 \\ | |
| 24 | +$(1,2,3,4)$ & 4 & $4$, $-6$, $4$, $-1$ & $-24$ & $32$ & $15$ & $1171$ & 10 \\ | |
| 25 | 25 | \bottomrule |
| 26 | 26 | \end{tabular} |
| 27 | 27 | \begin{tablenotes}\footnotesize |
@@ -35,7 +35,11 @@ Write $U_S := \abs{S}^{-1}\sum_{i\in S}\psi(X_i)$ and $U_k := U_{D_k}$. The sub- | ||
| 35 | 35 | \begin{equation}\label{eq:RJ} |
| 36 | 36 | R^{J}_{k} := \sum_{j=1}^r w_j\, s_j^{-1}\sum_{\ell=1}^{s_j} R_{S_{j\ell,k}}, |
| 37 | 37 | \end{equation} |
| 38 | −we have $\Jr{r,k} - \theta_0 = U_k + R^J_k$. The mean of $R^J_k$ is deterministic and, by \eqref{eq:bias} and \eqref{eq:vandermonde}, equals $\sum_{a<r} b_a m^{-a}\sum_j w_j s_j^a + \sum_j w_j\rho_{m/s_j} = O(m^{-r})$ (\cref{lem:bias}). Its $L_2$ norm is at most $\sqrt{C}\Amp/m$ by the triangle inequality (\cref{lem:remainder}); the nested fits of one worker are dependent, and no independence is used at this step. Across workers the blocks are independent, so the centered average has second moment at most $C\Amp^2/(Km^2)$ (\cref{lem:average}). Finally the full-sample expansion $\thetaN - \theta_0 = U_N + R_N$ with $U_N = K^{-1}\sum_k U_k$ cancels the common influence term exactly: | |
| 38 | +we have $\Jr{r,k} - \theta_0 = U_k + R^J_k$. The mean of $R^J_k$ is deterministic and, by \eqref{eq:bias} and \eqref{eq:vandermonde}, | |
| 39 | +\begin{equation}\label{eq:meanRJ} | |
| 40 | +\E R^J_k = \sum_{a=1}^{r-1}\frac{b_a}{m^{a}}\sum_{j=1}^r w_j s_j^{\,a} + \sum_{j=1}^r w_j\,\rho_{m/s_j} = O(m^{-r}) | |
| 41 | +\end{equation} | |
| 42 | +(\cref{lem:bias}), the first sum vanishing identically. Its $L_2$ norm is at most $\sqrt{C}\Amp/m$ by the triangle inequality (\cref{lem:remainder}); the nested fits of one worker are dependent, and no independence is used at this step. Across workers the blocks are independent, so the centered average has second moment at most $C\Amp^2/(Km^2)$ (\cref{lem:average}). Finally the full-sample expansion $\thetaN - \theta_0 = U_N + R_N$ with $U_N = K^{-1}\sum_k U_k$ cancels the common influence term exactly: | |
| 39 | 43 | \begin{equation}\label{eq:pilot} |
| 40 | 44 | \pilot{r} - \thetaN = \E R^J_k + K^{-1}\sum_k (R^J_k - \E R^J_k) - R_N = \op\!\Big(m^{-r} + \frac{1}{m\sqrt K} + \frac{1}{N}\Big), |
| 41 | 45 | \end{equation} |
modified
paper/sections/costmodel.tex
+5 −1
@@ -2,7 +2,11 @@ | ||
| 2 | 2 | |
| 3 | 3 | The frontier of \cref{tab:exponents} makes $(2,1)$, $(1,2)$ and $(4,0)$ statistically interchangeable at the level of exponents. Choosing among them is a question of cost, and the cost model below is proposed, calibrated and tested, not proved. Let $T_{\mathrm{fit}}(n,p)$ be the elapsed time of one local MLE fit on $n$ observations, $T_{\mathrm{der}}(m,p)$ that of one local score-and-Hessian evaluation, $\lambda$ the latency of one synchronized exchange (broadcast plus the slowest worker's reply, excluding payload), $B$ the effective uplink bandwidth at the coordinator, $b$ the bytes per transmitted number and $T_{\mathrm{solve}}(p)$ the coordinator's linear solve. For the doubling pattern the local phase of order $r$ performs $2^{r}-1$ fits touching every local observation $r$ times, so that with a fit cost $T_{\mathrm{fit}}(n,p) \approx c_0(p) + c_1(p)\,n$ (fixed overhead plus a linear pass, the empirical shape of Newton fits with a bounded number of iterations), |
| 4 | 4 | \begin{equation}\label{eq:cost} |
| 5 | −T(r,q) \;\approx\; \underbrace{(2^r - 1)c_0 + r\,c_1 m}_{\text{local, critical path of the slowest worker}} \;+\; q\Big[\lambda + T_{\mathrm{der}}(m,p) + \frac{K\,b\,(p + p(p+1)/2)}{B} + T_{\mathrm{solve}}(p)\Big] \;+\; T_{\mathrm{sched}}(K) + T_{\mathrm{straggle}}(K), | |
| 5 | +\begin{aligned} | |
| 6 | +T(r,q) \;\approx\;& \underbrace{(2^r - 1)\,c_0 + r\,c_1\, m}_{\text{local phase (slowest worker)}} | |
| 7 | +\;+\; q\,\Big[\underbrace{\lambda + T_{\mathrm{der}}(m,p) + \frac{K\,b\,(p + p(p+1)/2)}{B} + T_{\mathrm{solve}}(p)}_{\text{one synchronized derivative phase}}\Big] \\ | |
| 8 | +&+\; T_{\mathrm{sched}}(K) + T_{\mathrm{straggle}}(K), | |
| 9 | +\end{aligned} | |
| 6 | 10 | \end{equation} |
| 7 | 11 | where the last two terms collect worker start-up and the excess of the slowest worker over the mean. Sums are critical-path sums: the local fits of different workers overlap, the $q$ derivative phases do not. |
| 8 | 12 | |
modified
paper/sections/exactmodel.tex
+5 −1
@@ -34,7 +34,11 @@ in particular | ||
| 34 | 34 | the same first two terms as $\Var\thetahat_m$: the variance cost of the half-block correction is $O(m^{-3})$ in this model. |
| 35 | 35 | \item \emph{Newton identities and basin.} For every $x > 0$, |
| 36 | 36 | \begin{equation}\label{eq:exp-newton} |
| 37 | −\mathcal N_N(x) = 2x - x^2\bar X_N, \qquad \mathcal N_N(x) - \thetaN = -\frac{(x - \thetaN)^2}{\thetaN}, \qquad \mathcal N_N^{\,q}(x) - \thetaN = -\frac{(x-\thetaN)^{2^q}}{\thetaN^{\,2^q-1}} . | |
| 37 | +\begin{aligned} | |
| 38 | +\mathcal N_N(x) &= 2x - x^2\bar X_N, \qquad | |
| 39 | +\mathcal N_N(x) - \thetaN = -\frac{(x - \thetaN)^2}{\thetaN}, \\ | |
| 40 | +\mathcal N_N^{\,q}(x) - \thetaN &= -\frac{(x-\thetaN)^{2^q}}{\thetaN^{\,2^q-1}} . | |
| 41 | +\end{aligned} | |
| 38 | 42 | \end{equation} |
| 39 | 43 | The iteration converges to $\thetaN$ if and only if $0 < x < 2\thetaN$; it diverges to $-\infty$ when $x < 0$ or $x > 2\thetaN$, and $x = 2\thetaN$ is mapped to $0$. |
| 40 | 44 | \item \emph{Boundary.} If $m \to \infty$ and $K/N^{\alphafr{r}{q}} \to c \in (0,\infty)$, |
modified
paper/sections/inexact.tex
+2 −2
@@ -41,10 +41,10 @@ whose bracket is bounded by $\norm{\widehat H - H_N(\thetaN)} + \tfrac12\Lambda_ | ||
| 41 | 41 | Case~(c) is, to our knowledge, new in this form. The precision needed for the derivative uplink is governed by how far the pilot already is from $\thetaN$: a pilot of order $r$ with $K \sim N^\alpha$ has error $m^{-r} = N^{-r(1-\alpha)}$, and a relative error $\varepsilon$ in the step costs $\varepsilon N^{-r(1-\alpha)}$, which is negligible at the $N^{-1/2}$ scale if and only if \eqref{eq:bits} holds. For the half-block jackknife at $\alpha = 0.8$ the requirement is $0.1\log_2 N$ bits, two bits of mantissa at $N = 2^{20}$, whereas the plain average at the same $\alpha$ would need $0.3\log_2N = 6$ bits and is outside its own frontier ($3/4$) in any case. The mantissa bits multiply the $p(p+1)/2$ Hessian entries per worker and round, so the saving is $(64 - b)/64$ of the Hessian payload. The statement is asymptotic in the same sense as \cref{thm:frontier}; the ablation of \cref{sec:mc} shows at which $b$ the quantization error becomes visible at finite $N$. |
| 42 | 42 | |
| 43 | 43 | \begin{table}[t] |
| 44 | −\centering\small | |
| 44 | +\centering\scriptsize | |
| 45 | 45 | \begin{threeparttable} |
| 46 | 46 | \caption{Protocol variants: local fits, synchronized phases, uplink per worker, and sufficient exponents.}\label{tab:protocols} |
| 47 | −\begin{tabular}{@{}llccl@{}} | |
| 47 | +\begin{tabular}{@{}p{3.0cm}p{4.2cm}cp{3.3cm}p{3.6cm}@{}} | |
| 48 | 48 | \toprule |
| 49 | 49 | Variant & Curvature used by the coordinator & Local fits & Uplink per worker (numbers) & Sufficient exponent $\alpha$ \\ |
| 50 | 50 | \midrule |
modified
paper/sections/literature.tex
+3 −3
@@ -10,11 +10,11 @@ | ||
| 10 | 10 | |
| 11 | 11 | \paragraph{Gap.} None of these papers combines an extrapolated local start with aggregated-Hessian Newton rounds, states a frontier indexed by both the local order and the number of rounds, proves sharpness for a corrected construction, or expresses the effect of inexact or quantized Hessians as a change in the admissible exponent with a bit count. These are the four results of \cref{sec:results,sec:exact,sec:inexact}; the frontiers of \citet{huang2019distributed}, \citet{zhu2021least}, \citet{jordan2019communication} and \citet{fan2023communication} appear as the cells $(1,1)$, $(1,1)$ with local Hessians, and $(1,t)$ with a single-machine Hessian of our tables. |
| 12 | 12 | |
| 13 | −\begin{table}[t] | |
| 14 | −\centering\small | |
| 13 | +\begin{table}[p] | |
| 14 | +\centering\scriptsize | |
| 15 | 15 | \begin{threeparttable} |
| 16 | 16 | \caption{Closest results, compared at theorem level (our notation: $K$ blocks of size $m = N/K$).}\label{tab:lit} |
| 17 | −\begin{tabular}{@{}llllc@{}} | |
| 17 | +\begin{tabular}{@{}p{3.3cm}p{2.6cm}p{3.4cm}p{2.0cm}p{4.0cm}@{}} | |
| 18 | 18 | \toprule |
| 19 | 19 | Paper & Local correction & Global correction (curvature) & Uplink per worker & Regime for first-order equivalence \\ |
| 20 | 20 | \midrule |
modified
paper/sections/results.tex
+1 −2
@@ -23,8 +23,7 @@ Then $\sqrt N\,(\iter{r}{q} - \thetaN) \pto 0$, and consequently $\sqrt N(\iter{ | ||
| 23 | 23 | By \eqref{eq:qbound}, $\sqrt N\norm{\iter{r}{q} - \thetaN}$ is bounded by a sum of three terms, which with $m = N/K$ read |
| 24 | 24 | \begin{equation}\label{eq:three-terms} |
| 25 | 25 | \sqrt N\, m^{-r2^q} = N^{1/2}\Big(\frac KN\Big)^{r2^q}, \qquad |
| 26 | −\sqrt N\Big(\frac{\sqrt K}{N}\Big)^{2^q} = N^{1/2}\,K^{2^{q-1}}N^{-2^q}, \qquad | |
| 27 | −\sqrt N\, N^{-2^q}. | |
| 26 | +\sqrt N\Big(\frac{\sqrt K}{N}\Big)^{2^q} = N^{1/2}\,K^{2^{q-1}}N^{-2^q}, \qquad \sqrt N\, N^{-2^q}. | |
| 28 | 27 | \end{equation} |
| 29 | 28 | The first vanishes if and only if $K^{r2^q} = o(N^{r2^q - 1/2})$, that is $K = o(N^{\alphafr{r}{q}})$. The second equals $\sqrt{K/N}$ for $q = 0$ and vanishes because $\alphafr{r}{0} < 1$; for $q \ge 1$ it is at most $N^{1/2 - 2^{q-1}} \le N^{-1/2}$ whenever $K \le N$. The third always vanishes. Consistency of the pilot, needed to apply \eqref{eq:contraction}, follows from \eqref{eq:pilot} and $m \to \infty$. The limit law follows from \cref{ass:newton} and Slutsky's lemma. \Cref{app:proofs} gives the details, including the event on which all iterates stay in $U$. |
| 30 | 29 | \end{proof} |
modified
paper/sections/robustness.tex
+3 −2
@@ -4,13 +4,14 @@ Both extensions are known in the literature \citep{zhu2021least,gu2023heterogene | ||
| 4 | 4 | |
| 5 | 5 | \begin{proposition}[Unequal block sizes and pseudo-likelihood]\label{prop:robust} |
| 6 | 6 | \begin{enumerate}[label=(\roman*)] |
| 7 | −\item Let the blocks have sizes $m_1,\dots,m_K$ with $\sum_k m_k = N$, $m_{\min} \to \infty$ and each $s_j \mid m_k$. With the size weights $m_k/N$ in the pilot, | |
| 7 | +\item Let the blocks have sizes $m_1,\dots,m_K$, with $\sum_k m_k = N$, with $m_{\min} \to \infty$, and with each $s_j$ dividing every $m_k$. With the size weights $m_k/N$ in the pilot, | |
| 8 | 8 | \begin{equation}\label{eq:unequal-pilot} |
| 9 | 9 | \pilot{r} - \thetaN = \op\Big(m_{\mathrm{eff}}^{-r} + \frac{\sqrt K}{N} + \frac1N\Big), \qquad m_{\mathrm{eff}} := \Big(\frac1N\sum_{k=1}^K m_k^{\,1-r}\Big)^{-1/r} \ \ge\ m_{\min}, |
| 10 | 10 | \end{equation} |
| 11 | 11 | and \cref{thm:frontier} holds with $m$ replaced by $m_{\mathrm{eff}}$; the aggregated derivatives are size-weighted and equal the full-sample derivatives, so the Newton steps are unchanged. With equal weights $1/K$ the influence term is not preserved: |
| 12 | 12 | \begin{equation}\label{eq:unequal-naive} |
| 13 | −\frac1K\sum_{k=1}^K U_k - U_N = \sum_{i=1}^N c_i\,\psi(X_i), \qquad c_i = \frac{1}{K\,m_{k(i)}} - \frac1N, \qquad \sum_{i=1}^N c_i^2 = \frac{1}{K^2}\sum_{k=1}^K\frac{1}{m_k} - \frac1N \ \ge\ 0, | |
| 13 | +\frac1K\sum_{k=1}^K U_k - U_N = \sum_{i=1}^N c_i\,\psi(X_i), \qquad c_i = \frac{1}{K\,m_{k(i)}} - \frac1N, \qquad | |
| 14 | +\sum_{i=1}^N c_i^2 = \frac{1}{K^2}\sum_{k=1}^K\frac{1}{m_k} - \frac1N \ \ge\ 0, | |
| 14 | 15 | \end{equation} |
| 15 | 16 | with equality only for equal sizes; the discrepancy is of exact order $N^{-1/2}$ when the sizes are bounded away from equality, so the equal-weight pilot is not first-order equivalent for $q = 0$, while for $q \ge 1$ the first Newton step squares it to $\op(N^{-1})$. |
| 16 | 17 | \item If the model is misspecified and $\theta_0$ is the pseudo-true value, all statements hold with |
added
results/raw/ablation_exponential_N1572864_m24.csv
+19773 −0
Large diff (19,773 lines) is not loaded by default to keep the page fast.Load diff
Diff truncated — file too large.