SPB Git

spb/wp28_uqo

Public
7.8 KB · 41 lines latex
1\section{Related literature and positioning}\label{sec:lit}23\Cref{tab:lit} compares the closest results at the level of their theorems; the comparison was made on the full texts, with the caveats listed in the replication package when a published version could not be accessed.45\paragraph{One-shot averaging.} \citet{zhang2013communication} bound the mean squared error of the average by $O(1/N + 1/m^2)$ (their Theorem~1), which matches the $O(1/N)$ centralized rate only for $K \lesssim N^{1/2}$; \citet{rosenblatt2016optimality} show that the second-order bias of the average equals $m$ times that of $\thetaN$ (their Theorem~2); \citet{battey2018distributed} obtain the same $\sqrt{N}$ barrier in sparse high dimension; \citet{volgushev2019distributed} prove for quantile regression that $K = o(N^{1/2})$ is also \emph{necessary} for the average to be first-order equivalent, by exhibiting a model in which the remainder is exactly of order $1/m$ (their Theorem~3.2), and \citet{shang2017computational} establish a sharp ``maximal number of machines'' for smoothing splines. \citet{lin2011aggregated} weight the local estimates by local Jacobians evaluated at the local estimates; their equivalence theorem requires $K \ll N^{1/4}$.67\paragraph{Local debiasing.} The subsampled-average estimator of \citet{zhang2013communication} combines each local fit with a fit on a random subsample of size $\lceil \rho m\rceil$ as $(\bar\theta_1 - \rho\bar\theta_2)/(1-\rho)$, a one-level Richardson extrapolation that removes the $1/m^2$ term of the MSE at the price of a leading constant $(2+3\rho)/(1-\rho)^2$ and a tuning rate $\rho \to 0$ (their Theorem~4); they leave the use of several subsamples open. \citet{gu2023heterogeneity} estimate the leading bias analytically on each machine with sample splitting and weight by local information, obtaining the centralized $\chi^2$ limit for $K = o(m^2)$, i.e.\ $K = o(N^{2/3})$ (their Theorems~4--6), with a smaller constant than the subsampled average. \citet{wu2023subsampling} apply the delete-one jackknife within each subsample and average, with bias $O(m^{-2}) + O(N^{-1})$ for smooth functions of means (their Theorem~2). The split-sample weights themselves go back to \citet{quenouille1949approximate,quenouille1956notes} and \citet{schucany1971bias}; \citet{dhaene2015split} develop the general higher-order split-panel jackknife, with weights solving a Vandermonde-type system (their equations~5.1--5.3), prove that the first-order variance is unchanged and that partitions into $\{2,3,\dots,h+1\}$ parts minimize the inflation of the uncancelled bias terms, and leave the higher-order variance open. \citet{hahn2024efficient} show that in i.i.d.\ cross-sections the split-sample bias estimate is inconsistent and that split-sample corrections carry twice the higher-order variance of the delete-one jackknife (their Propositions~5--7), a cost that our exact variance formula and Monte Carlo results make explicit for nested splits.89\paragraph{Global corrections.} \citet{huang2019distributed} define the distributed one-step estimator: workers send scores and Hessians evaluated at the broadcast average and the coordinator takes one Newton step. Their Theorem~3.4 proves asymptotic equivalence to the full-sample M-estimator when $k = O(\sqrt{N})$; their Theorem~3.5 bounds the MSE by $2\tr\Sigma/N + O(N^{-2}) + O(k^4 N^{-4})$, a bound that is itself $o(N^{-1})$-exact whenever $k = o(N^{3/4})$, although that exponent is not stated. \citet{zhu2021least} prove that one \emph{local} Newton step per machine followed by information-weighted averaging is efficient iff $m / N^{1/4} \to \infty$, ``allowing $K = O(N^{3/4})$'' (their Theorem~2), and iterate the scheme. \citet{jordan2019communication} replace the aggregated Hessian by that of the first machine and send gradients only; their Theorem~3.1 bounds the one-round error by $C(\norm{\bar\theta - \thetaN}^2 + (N^{-1/2} + m^{-1/2})\norm{\bar\theta - \thetaN})$, a contraction by $m^{-1/2}$ per round. \citet{fan2023communication} add a proximal term and state the explicit rounds frontier (their remark after Theorem~4.1): from the plain average, $t$ single-machine-Hessian steps match the global minimizer when $m^{t+2} \gg N \cdot \mathrm{polylog}$, i.e.\ $K = o(N^{(t+1)/(t+2)})$ up to logarithms, and one averaged local-Hessian step in the linear model allows $K = o(N^{3/4})$. \citet{shamir2014communication} and \citet{wang2017efficient} are the optimization-side precursors; \citet{chen2022first} avoid the Hessian entirely by stochastic first-order iterations on one machine. \citet{yin2022newton} removes the quadratic terms of the one-step estimator analytically and obtains efficiency iff $K = o(N^{2/3})$. Reviews by \citet{gao2022review} and \citet{li2024selective} confirm that no joint order--round frontier was available at their writing.1011\paragraph{Gap.} None of these papers combines an extrapolated local start with aggregated-Hessian Newton rounds, states a frontier indexed by both the local order and the number of rounds, proves sharpness for a corrected construction, or expresses the effect of inexact or quantized Hessians as a change in the admissible exponent with a bit count. These are the four results of \cref{sec:results,sec:exact,sec:inexact}; the frontiers of \citet{huang2019distributed}, \citet{zhu2021least}, \citet{jordan2019communication} and \citet{fan2023communication} appear as the cells $(1,1)$, $(1,1)$ with local Hessians, and $(1,t)$ with a single-machine Hessian of our tables.1213\begin{table}[t]14\centering\small15\begin{threeparttable}16\caption{Closest results, compared at theorem level (our notation: $K$ blocks of size $m = N/K$).}\label{tab:lit}17\begin{tabular}{@{}llllc@{}}18\toprule19Paper & Local correction & Global correction (curvature) & Uplink per worker & Regime for first-order equivalence \\20\midrule21\citet{zhang2013communication} Thm 1 & none & none & $p$ & $K \lesssim N^{1/2}$ (MSE rate) \\22\citet{zhang2013communication} Thm 4 & one random subsample & none & $p$ & $K \lesssim N^{2/3}$ (MSE rate) \\23\citet{lin2011aggregated} Thm 4.2 & none & local-Jacobian weighting & $p + p^2$ & $K \ll N^{1/4}$ \\24\citet{huang2019distributed} Thm 3.4/3.5 & none & 1 step, aggregated Hessian & $p$; $p + p^2$ & $K = O(N^{1/2})$ stated; $N^{3/4}$ implicit \\25\citet{zhu2021least} Thm 2 & none & 1 local step + weighting & $p + p^2$ & $K = O(N^{3/4})$ \\26\citet{jordan2019communication} Cor 3.4 & none & $t$ rounds, machine-1 Hessian & $p$ per round & $K = o(N^{2/3})$ for $t = 1$ from the average \\27\citet{fan2023communication} rem.\ after Thm 4.1 & none & $t$ rounds, machine-1 Hessian & $p$ per round & $K = o(N^{(t+1)/(t+2)})$ (polylog) \\28\citet{gu2023heterogeneity} Thm 4--6 & analytic bias, split & information weighting & $p + p^2$ & $K = o(N^{2/3})$ \\29\citet{wu2023subsampling} Thm 2--4 & delete-one jackknife & none & $p$ & $K = o(N^{3/4})$ (functions of means) \\30\citet{yin2022newton} Thm 3.2 & none & 1 step + analytic adjustment & $p$ + summaries & $K = o(N^{2/3})$ \\31\citet{volgushev2019distributed} Cor 3.3 & none & none & $p$ & $K = o(N^{1/2})$ necessary (quantile regr.) \\32This paper, \cref{thm:frontier} & order-$r$ split extrapolation & $q$ steps, aggregated Hessian & $p$; $q(p + p(p+1)/2)$ & $K = o(N^{1 - 1/(2^{q+1} r)})$, sharp in \cref{thm:exponential} \\33This paper, \cref{thm:inexact} & order $r$ & $q$ steps, stale / machine-1 / $b$-bit Hessian & see \cref{tab:protocols} & $1 - \frac{1}{2(r+q)}$; $1 - \frac{1}{2r+q}$; bits $\ge (\tfrac12 - r(1-\alpha))\log_2 N$ \\34\bottomrule35\end{tabular}36\begin{tablenotes}\footnotesize37\item Regimes are sufficient conditions unless marked necessary; ``MSE rate'' means attainment of the $O(1/N)$ rate rather than $\sqrt{N}$-equivalence to $\thetaN$. Theorem numbers refer to the versions read (arXiv or accepted manuscripts where publisher pages were inaccessible; see \texttt{literature/PRIOR\_ART\_MATRIX.csv}).38\end{tablenotes}39\end{threeparttable}40\end{table}41
Files · dd2471a83ff836060f8585aef5236d77d6a11c15
Menu

Theme