SPB Git

spb/localvm-research Public License

Running LLMs larger than memory on a consumer Mac — falsification-driven research: margin-gated deferred refinement, out-of-core verification on Apple Silicon. TR-01 published.

Python 63.2% JavaScript 23.5% CSS 11.8% Shell 0.9% Makefile 0.5%
42.8 KB
View source Raw Blame History

# project: localvm-research document: research/notes/speculation_error_stability author: Simon-Pierre Boucher contact: contact@spboucher.ai created: 2026-08-11 status: draft

# Speculative execution, numerical error analysis, and output-decision stability (charter §4.7, §4.9, §4.10)

Reading notes for three intertwined questions that matter to localvm-research:

  1. §4.7 — Can we compute with a cheap approximation of the model and pay for the exact model only when needed (speculation + verification)?
  2. §4.9 — Can we bound the error introduced by approximation (quantization, truncated residuals, partial GEMM) tightly enough to know when the cheap answer is already correct?
  3. §4.10 — Empirically, how often does an approximate model make the same decision (same greedy token, same top-k, close distribution) as the exact model — because that frequency is exactly the fraction of tokens for which we would never need to touch the expensive weights.

The through-line: every speedup number in the speculative-decoding literature is simultaneously a measurement of decision stability. An acceptance rate of 0.8 for a draft model is the statement "the cheap model's greedy/sampled token matches the exact model's 80% of the time." This literature therefore already contains a large body of measured evidence for charter Experiments B, D, E and G.

# 1. Landscape

Three research communities touch our problem and barely cite each other:

  • Speculative decoding (ML systems, 2022–now). Draft-then-verify across future tokens. Mathematically mature (exact-distribution rejection sampling), enormous empirical corpus (2–6.5× wall-clock speedups), and, crucially for us, verified on Apple Silicon (Apple ReDrafter in MLX; QuantSpec explicitly targets edge). Verification granularity is always the whole target model over a block of tokens.
  • Certified robustness / formal verification of NNs (2018–now). Interval bound propagation, CROWN/LiRPA, zonotopes (DeepT), Lipschitz analysis. Gives sound guarantees of the form "no perturbation within ε can flip the argmax," which is exactly the primitive we would want for "certified early termination." Current reality: does not scale past small transformers; self-attention is not even globally Lipschitz.
  • Mixed-precision numerical linear algebra (Higham school, 2017–now). Iterative refinement with low-precision factorizations (GMRES-IR), probabilistic rounding-error analysis with √n instead of n error constants. This community routinely does what we want to do — compute cheap, refine to full accuracy with a guarantee — but for linear systems, not for transformer inference. It is the strongest source of transferable ideas that the LLM literature has not absorbed.

A fourth, more empirical strand — quantization-quality measurement (KL divergence, "flips," same-top-token rates from llama.cpp tooling and the Microsoft "Accuracy is Not All You Need" paper) — provides the measured token-stability numbers §4.10 asks for.

# 2. Technique-by-technique map

# 2.1 Classical speculative decoding (draft model + rejection sampling)

  • Mechanism. A small draft model proposes γ tokens autoregressively; the target model scores all γ+1 positions in one forward pass; a rejection-sampling rule accepts a prefix and resamples the first rejected position from a corrected residual distribution.
  • Exactness guarantee. The modified rejection sampling of Leviathan et al. / Chen et al. provably preserves the target model's output distribution exactly (token-level acceptance probability min(1, p(x)/q(x)), resample from norm(max(0, p−q))). This is the canonical proof that an approximate computation plus a cheap correction can equal the exact computation in distribution — the conceptual anchor for everything in this note.
  • Numbers. Leviathan et al.: 2–3× on T5-XXL (2.6× at temp=1, 3.4× at temp=0 on translation; 2.3×/3.1× summarization). Chen et al.: 2–2.5× on Chinchilla-70B. Note the consistent pattern: greedy (temp 0) speedups exceed sampling speedups, i.e., argmax decisions are easier to reproduce than full distributions — direct evidence for §4.10.
  • Hardware assumptions. Works precisely because decode is memory-bandwidth-bound: verifying k tokens costs ≈ the same wall-clock as generating 1, since the weights are read once either way. Apple Silicon: same regime (unified memory, bandwidth-bound decode); llama.cpp (--model-draft) and mlx_lm both ship draft-model speculation on Metal.
  • Limitation. Needs a well-aligned draft model; acceptance rate α drives everything. Real-world α measured at 0.6–0.8 (BentoML production benchmarks with EAGLE-3-class drafts), not the near-1.0 of idealized analyses.
  • Extension opportunity for us. Reinterpret "draft model" as "cheap representation of the same model" (low-bit base) and "target model" as "base + SSD-resident residual." The rejection-sampling correction then makes a progressive-precision runtime exact in distribution.

# 2.2 Medusa / EAGLE / ReDrafter / Speculative Streaming (auxiliary-head speculation)

  • Mechanism. Instead of a separate draft LLM, small heads (Medusa: parallel FFN heads; EAGLE: a 1-layer autoregressive head over target features; ReDrafter: an RNN head; Speculative Streaming: multi-stream attention inside the target) predict a tree of candidate continuations, verified with tree attention in one pass.
  • Numbers. Medusa 2.2–3.6× (Vicuna 7B/13B). EAGLE-3: 3.0–6.5×, average acceptance length up to 7.5 (HumanEval), 20–40% over EAGLE-2, 1.38× throughput even at batch 64 in SGLang. ReDrafter: up to 3.5 tokens/step, 2.8× on H100 — and on Apple Silicon via MLX: 1.37× on M1 Max, up to 2.3× on M2 Ultra (the single most directly relevant Apple-hardware datapoint in this literature). Apple's Speculative Streaming: 1.8–3.1× with ~10,000× fewer extra parameters than Medusa.
  • Exactness. All use strict speculative-sampling acceptance ⇒ lossless w.r.t. the target model.
  • Limitation. Heads must be trained (EAGLE-3 needs a training-time test procedure); acceptance is task-dependent (templated code ≫ open chat).
  • Opportunity. EAGLE's key insight — draft from the target's own hidden features, not from tokens — transfers to weight-paging: hidden states are a strong predictor of what comes next, hence of which weights will matter next (cf. §2.7).

# 2.3 Self-speculative decoding: the model drafts with a subset of itself

This family is the closest published relative of "speculate with cheap weights, verify with full weights."

  • Draft & Verify (Zhang et al.): draft by skipping layers of the target (selected by Bayesian optimization), verify with the full model. 1.2–1.56× (Llama-2-13B: 1.56× CNN/DM). No extra model, no training.
  • LayerSkip (Elhoushi et al., ACL 2024): train with layer dropout + early-exit loss so layer-E exits are usable drafts; verification reuses the draft's KV cache (the draft pass is a prefix of the verify pass, so drafting work is not thrown away). Speedups 1.34–2.16×. Measured acceptance: 68.9% (Llama-2-7B, exit at layer 8/32) and 74.5% (13B, exit 15/40) — i.e., roughly three quarters of token decisions made with ~25–40% of the depth survive exact verification. Strong quantitative support for charter hypothesis "decisions stabilize early."
  • Kangaroo (NeurIPS 2024): fixed shallow sub-network (2–3 layers!) + tiny adapter as the draft, with confidence-based early stopping of drafting (stop speculating when the draft's own confidence drops).
  • SWIFT (ICLR 2025): plug-and-play, optimizes the skipped-layer set on the fly per input stream; 1.3–1.6×, no training at all.
  • CLaSp: dynamic layer-skip schedule updated after every verification step using the last verified hidden states; 1.3–1.7× on Llama-3; for Llama-3-70B the optimum skipped 44 of 80 layers.
  • Hardware. All of these are single-model and memory-light — well suited to unified memory. Nothing architecturally CUDA-specific.
  • Limitation. The draft is a depth truncation. Nobody in this family drafts with a precision truncation of the weights while verifying with the full-precision weights streamed on demand (see §3).

# 2.4 Precision-level self-speculation (QSpec, QuantSpec, ML-SpecQD)

The newest and most relevant variant: the draft and target share the same weights at different precisions.

  • QSpec (EMNLP 2025). One W4-quantized weight set; draft in W4A4 (fast low-precision activations), verify in W4A16. Token generation between the two modes is "highly similar" (their Sec. 2.2 validation); switching cost near zero because weights and KV are shared. Up to 1.64× over the high-precision baseline, no quality loss, plug-and-play (no training).
  • QuantSpec (Apple + Berkeley, ICML 2025). Self-speculation for long context: draft uses 4-bit hierarchical KV cache + 4-bit weights, target uses full precision. Acceptance >90%, end-to-end up to ~2.5×, ~1.3× memory reduction. Explicitly motivated by edge deployment.
  • ML-SpecQD (2025). Multi-level pipeline: MXFP4 quantized draft (a direct 4-bit cast of the target — "quantized drafts as a turnkey solution, no custom draft pretraining") possibly itself accelerated by a smaller draft; >2× over 16-bit baselines.
  • Why this matters to us. These systems prove the core premise of a progressive-precision runtime: a 4-bit version of the same model agrees with the 16-bit version on the (large) majority of tokens, and the disagreements can be repaired exactly by a shared-weight verification pass. What none of them do: exploit the agreement to avoid storing/loading the high-precision weights in the first place (their full-precision operands stay resident; the win is arithmetic/bandwidth, not capacity).

# 2.5 BiLD, cascades, and routing (approximate-first with deferral — not exact)

  • BiLD (NeurIPS 2023). Small model decodes autoregressively; fallback policy hands control to the large model when small-model confidence (max prob) is below a threshold; rollback policy lets the large model revert the small model's recent tokens when their distance exceeds a threshold. Up to 2.12× on a T4 with "minimal" quality degradation — but the output is not guaranteed to match the large model (thresholded, not rejection-sampled). BiLD is the published prototype of "escalate to expensive weights only when uncertain."
  • FrugalGPT (2023) / RouteLLM (ICLR 2025) / cascades generally. Sequential escalation across models with a learned scorer + threshold: FrugalGPT matches GPT-4 quality at up to 98% lower cost, sending only ~1 query in 6 to the big model on some tasks; RouteLLM ~2× cost cut at 95% quality with 14% of queries routed strong. Cascade evidence at request granularity for what we want at token/weight granularity: most work is easy; a calibrated uncertainty signal can concentrate expensive compute on the hard minority.
  • Limitation. Confidence thresholds are heuristic; guarantees are empirical-statistical at best (see CALM for how to make them rigorous).

# 2.6 Speculation within the forward pass across time: SPEED, lookahead, SpecInfer, SpecExec

  • SPEED (Berkeley, NeurIPS-W 2023). Uses early-layer hidden states to predict the next token before the current token's forward pass finishes, launching future tokens' passes speculatively in a pipelined fashion, with invalidation logic to roll back mispredictions. This is genuine speculative execution inside the transformer computation (though for parameter-shared models). Closest published thing to "start computing before you are sure."
  • Lookahead decoding (ICML 2024). Jacobi fixed-point iteration on the token sequence + n-gram cache; no draft model; 1.5–2.3× (best on code). Shows the sequential dependency itself is partially artificial.
  • SpecInfer (ASPLOS 2024). Tree-based speculation + parallel tree verification; 2.6–3.5× for offloading-based inference on one GPU. First strong signal that speculation is more valuable when weights live off-device.
  • SpecExec (NeurIPS 2024). The extreme of that logic, aimed at consumer hardware with RAM offloading: huge draft trees (up to 2048 nodes) from a 7B draft are verified by an offloaded 70B target in one pass; Llama-2-70B at 4–6 tok/s (4-bit) or 2–3 tok/s (16-bit) on consumer GPUs, 10.6–18.7× over sequential offloaded decoding; generation rate ~20 accepted tokens per target-model pass. For localvm this is the key economics result: when reading the big model costs seconds, speculation amortizes one full-model weight sweep over ~20 tokens — bytes-read-per-token drops by the acceptance length.
  • LLM-42 (2026). Uses verified speculation to get deterministic inference cheaply: draft with fast batch-variant kernels, verify with batch-invariant kernels, roll back on mismatch. Notable as prior art for "same weights, cheaper numerics as draft, exact numerics as verifier."

# 2.7 Speculative weight loading (MoE expert prefetch; LLM-in-a-flash)

Speculation applied to which parameters to fetch, not which tokens to emit:

  • Deja Vu (ICML 2023). Contextual sparsity: small MLP predictors, fed by layer k's activations, predict which attention heads / FFN neurons layer k(+1) needs (asynchronous lookahead predictors). Up to ~85% sparsity with no quality drop measured, >2× latency vs FasterTransformer on OPT-175B. No verification: a mispredicted neuron is simply dropped (lossy, uncorrected).
  • LLM in a flash (Apple, ACL 2024). Keeps attention weights resident; predicts FFN sparsity (low-rank predictor) + sliding-window neuron reuse, loading only ~2% of FFN weights per token from flash; runs models ~2× DRAM size. Again predict-and-hope: no exactness correction; relies on ReLU-style sparsity that modern SwiGLU models lack natively.
  • MoE expert speculation (2025–26): MoE-SpeQ (a small on-device draft model predicts the sequence of experts future tokens will need and prefetches them, hiding PCIe I/O), Fate (cross-layer gate signals for prefetch, ~90%+ hit rates reported for hash-based SiDA on Switch), pre-gated MoE (retrained gate decouples selection from execution), Apple's SpecMD study of speculative expert prefetching + caching policies. Here misprediction is not lossy — the correct expert is fetched late (a stall, not an error). This is true speculative weight paging with rollback-to-exactness, but it exists only where the architecture already has discrete routable units (experts). No dense-model equivalent exists.

# 2.8 Early exit with statistical guarantees (CALM, CATs)

  • CATs (Schuster et al., 2021). Early-exit BERT with a meta consistency classifier + conformal-style calibration: guarantees the early-exit model's prediction equals the full model's with probability ≥ 1−ε on i.i.d. data. Classification only.
  • CALM (NeurIPS 2022). Extends this to autoregressive generation: per-token exit decisions (confidence = e.g. softmax top-1/top-2 margin) calibrated via distribution-free risk control so that sequence-level quality (ROUGE/BLEURT vs. the full model, or exact textual consistency) is provably maintained with user-chosen probability (e.g., 95%). Reported ~3× decode speedups at negligible quality change on summarization/translation/QA.
  • Significance for §4.9's key question. CALM is the strongest published answer to "can computation terminate when additional precision/depth can no longer change the output?" — answered statistically (calibrated risk control), not deterministically (no worst-case certificate). Nobody has done the same for precision (bit-width) instead of depth. That specific transfer — "CALM, but the axis is weight precision / residual count rather than layer count" — appears unclaimed (see §3).

# 2.9 Formal error bounds: Lipschitz, IBP/CROWN, zonotopes (§4.9)

  • Lipschitz status of attention. Kim, Papamakarios & Mnih (ICML 2021): standard dot-product self-attention is not Lipschitz on unbounded domains (they propose L2 attention that is). Castin et al., "How Smooth Is Attention?" (ICML 2024, Apple co-affiliation): on compact input sets the local Lipschitz constant of self-attention grows like √n in sequence length (tight for practical n), with mean-field bounds beyond. Verdict for our purposes: per-layer local Lipschitz constants exist but are input-radius-dependent and, composed over 30–80 layers, give astronomically loose (vacuous) worst-case logit bounds. Deterministic global certification of "ΔW cannot flip this token" via Lipschitz products is not practical for 7B+ models.
  • Bound propagation (IBP/CROWN/auto_LiRPA). auto_LiRPA (NeurIPS 2020) supports transformers and even model-weight perturbations (weights treated as graph inputs — conceptually exactly our ‖ΔW‖ → Δlogits question). But CROWN's complexity is O(m²n³) (m layers, n neurons/layer); published successes are BERT-small-scale classifiers and CIFAR/TinyImageNet CNNs. DeepT (PLDI 2021, multi-norm zonotopes) certifies "larger" transformers than prior work — meaning a few layers of BERT against synonym/ℓp attacks, minutes per instance. Four to five orders of magnitude of scale separate this literature from a 7B decoder. Usable insight, though: the margin certificate primitive — argmax is stable iff top-2 logit margin > 2·(bound on ‖Δlogits‖∞) — is trivial and cheap once you have any bound on Δlogits; the hard part is the bound.
  • Practical middle ground. Layer-wise empirical perturbation theory: the PTQ literature effectively works with ‖WX − ŴX‖_F as its per-layer error functional (GPTQ/OBC objective), and QEP (NeurIPS 2025) shows these per-layer errors accumulate near-exponentially with depth under independent layer-wise quantization (with a theoretical justification under mild conditions), which is why naive low-bit PTQ collapses. "Why Do Some Inputs Break Low-Bit Quantization?" (EMNLP 2025) finds input-dependent structure: full-precision residual-stream magnitudes predict which examples will have large quantization error (ρ = 0.82), with RMSNorm inverting magnitude relations and late-layer MLP gates amplifying. Both papers support a per-input, per-layer empirical error predictor (cheap features → predicted logit error) as the realistic substitute for formal bounds — i.e., the certificate becomes statistical, as in CALM.
  • Numerical-analysis imports. Carson & Higham's three-precision iterative refinement / GMRES-IR: factorize in低 precision (fp16), refine in higher precision, with proven convergence to working-precision accuracy for κ(A) up to 10⁸–10¹²; adopted on tensor cores (Haidar et al.). Higham & Mary's probabilistic rounding-error analysis and Connolly–Higham–Mary's stochastic-rounding analysis: replacing worst-case nu error constants by √n·u with high probability (rounding errors as mean-independent zero-mean RVs). Two transferable ideas: (a) refinement loops with guarantees — compute logits with cheap weights, estimate a residual, refine only if needed; (b) probabilistic rather than worst-case error budgets — realistic Δlogit estimates scale like √(accumulated variance), not the vacuous worst case. Nobody has written the "GMRES-IR of transformer inference" paper.

# 2.10 Dynamic per-token precision (no verification)

Recent, mostly 2025–26, and directly on our path: QuickSilver (per-token entropy → 8/4/2-bit Matryoshka bit-width mid-network), FlexQuant (perplexity-entropy-guided runtime bit-width switching, 1.3× with fine-grained precision management), DP-LLM (dynamic layer-wise precision on multi-scale quantized overlays, NeurIPS 2025), MoBiQuant (token-sensitivity mixture-of-bits, any-precision weights). All are feed-forward heuristic precision assignment — uncertainty gates precision, but nothing verifies the low-precision tokens afterward, so all are lossy with empirical-only quality claims. They confirm the mechanism (any-precision/Matryoshka weight layouts where low bits are a prefix of high bits) is implementable; none closes the loop with exactness.

# 3. Within-forward-pass speculation: prior art or gap?

The charter's key §4.7 question: can speculation happen inside a transformer forward pass — speculate with cheap weights, verify with full weights only when needed — rather than only across future tokens?

Verdict: the ingredients all exist separately; the specific mechanism does not appear to exist. This is a real gap, but a narrow one, and it must be positioned very carefully against five near-misses.

Documented near-misses, from farthest to closest:

  1. Depth-truncated self-speculation (Draft&Verify, LayerSkip, Kangaroo, SWIFT, CLaSp): draft = same weights, fewer layers; verify = all layers. Speculation is about future tokens; the full weights are read on every verification pass regardless. Verification granularity: whole model.
  2. Numerics-truncated self-speculation (QSpec: W4A4 draft vs W4A16 verify; LLM-42: fast kernels vs batch-invariant kernels): draft = same weights, cheaper arithmetic; verify = better arithmetic. This is literally "speculate cheap, verify with full computation" — but every token is verified, both operand sets stay resident, and the goal is arithmetic speed / determinism, not resident-set or bytes-per-token reduction.
  3. Precision-truncated drafts (ML-SpecQD, QuantSpec): draft = 4-bit cast of the target (weights and/or KV); verify = 16-bit target. Proves 4-bit-vs-16-bit token agreement is high enough (>90% acceptance) to power speculation — but the 16-bit model is fully resident and fully read.
  4. Speculative weight prefetch without exactness (Deja Vu, LLM in a flash): predict which weights the pass will need, load only those. Inside-the-forward-pass speculation about memory, but wrong predictions silently change the output (no verify/rollback path).
  5. Speculative weight prefetch with implicit exactness (MoE expert prefetching: MoE-SpeQ, Fate, SpecMD, pre-gated MoE): predicted experts are prefetched; a misprediction causes a stall while the right expert loads — output exactness preserved by waiting, not by approximating. Exists only for architectures with discrete routable units; there is no dense-model analog where the "unit" is a precision level or residual correction.
  6. Hidden-state speculation across the pipeline (SPEED): early-layer states used to start dependent computation early, with invalidation. Speculates on intermediate values inside the pass, but for pipelining parameter-shared layers, not for avoiding weight loads.

What does not exist (after searching "weight-level speculation," "approximate forward pass verification," "speculative dequantization," "progressive precision verification," "margin-gated precision escalation," cascade/deferral and self-speculation literatures): a runtime where the default forward pass uses a cheap resident representation (low-bit base), a per-token decision-uncertainty signal (top-2 logit margin, entropy, or a learned error predictor à la EMNLP-2025 residual-magnitude features) decides whether the cheap decision is trustworthy, and only on low-margin tokens are full-precision residuals streamed from storage to re-verify — with either (a) speculative-sampling-style correction giving exactness in distribution, or (b) CALM-style distribution-free risk calibration giving a statistical consistency guarantee, and with bytes-read-per-token as the optimization target. The closest conceptual statement in print is BiLD's fallback/rollback (2023) — but with two separate models, both resident, no storage tier, and no guarantee.

Honesty requirements for a novelty claim later (Phase 11): the claim cannot be "speculate cheap / verify expensive" (QSpec owns that), nor "load weights on demand by prediction" (Deja Vu / LLM-in-a-flash own that), nor "escalate on low confidence" (BiLD/cascades/CALM own that). The defensible claim is the composition: uncertainty-gated, storage-tiered precision escalation with an exactness or calibrated-risk correction, evaluated in bytes/token — plus, if we can make refinement incremental (reuse the low-bit matmul result and add only a residual term, GMRES-IR-style, instead of recomputing), a genuinely new kernel-level primitive.

# 4. How stable are token decisions really? (measured numbers)

Everything found that quantifies "same useful output despite different computation":

Same-greedy-token / top-1 agreement under quantization

  • llama.cpp's KL-divergence tooling reports "Same top p" (fraction of positions where quantized and fp16 models pick the same top token). Measured examples from a 4-bit-class (NVFP4/Q4) quantization run: Same top p = 90.87 ± 0.08% and 91.23 ± 0.07% (two variants of the same model), with Mean KLD ≈ 0.054, median KLD 0.018, 99th-percentile KLD 0.53, max KLD ~22–26. Community wisdom (blind test, llama.cpp discussion #5962): Q6_K/Q5_K statistically indistinguishable from fp16 in human preference; IQ2/IQ1 clearly distinguishable.
  • MLX-ecosystem measurements (smcleod, Qwen3.6-27B dense, top-K sparse KLD): mean KLD 0.014 (8-bit) → 0.029 (6-bit) → 0.059–0.113 (4-bit variants); on a 35B-A3B MoE, 4-bit KLD ranges 0.027 (DWQ) to 0.074 (RTN) — and MoE router protection changes rankings, i.e., which weights get precision matters more than average bpw.
  • Acceptance rates in precision-level self-speculation are direct agreement measurements: QuantSpec (4-bit weights + 4-bit hierarchical KV draft): >90% acceptance; QSpec validates W4A4 vs W4A16 generation as "highly similar"; production EAGLE-3 acceptance α ≈ 0.6–0.8 per token (BentoML).
  • Depth truncation: LayerSkip acceptance 68.9% (Llama-2-7B drafting from layer 8 of 32) and 74.5% (13B, layer 15 of 40); CLaSp finds Llama-3-70B tolerates skipping 44/80 layers in drafting at 1.64× peak speedup. Tuned Lens (Belrose et al.) formalizes "prediction depth" — the layer after which the top-1 prediction stops changing — and finds many tokens stabilize well before the final layer (easy tokens shallow, hard tokens deep), which is the mechanistic basis for all early-exit acceptance numbers.

Flips: same accuracy ≠ same answers ("Accuracy is Not All You Need", Microsoft 2024)

  • Quantized models within 1% aggregate accuracy of baseline nonetheless flip up to ~15% of individual answers (correct↔incorrect symmetrically); layer-dropping/pruning at matched accuracy reaches 25%+ flips. Flip rate correlates with KL divergence at Spearman 0.96–0.97, and with MT-Bench degradation. Consequence for us: aggregate benchmarks cannot certify a compressed runtime; per-token distance metrics (KL, flips, agreement) are the right quality currency — matching the charter's §8.2 quality metrics list.

Margins and their fragility

  • The whole early-exit line (CALM, CATs, BiLD fallback, Kangaroo's drafting stop) uses the softmax top-1/top-2 margin as the confidence signal, and it works — meaning margins are informative — but published distributions of logit gaps are surprisingly scarce. Unit 42's "logit-gap steering" measures refusal-vs-affirmation logit gaps in safety contexts and shows small logit shifts close them (margins there are small and exploitable). Thinking Machines' batch-invariance study is the sharpest evidence that a nontrivial share of tokens sit on a knife's edge: batch-size-dependent rounding differences alone (perturbations of order 10⁻⁵ relative) caused 1,000 identical greedy (temp-0) prompts to produce dozens of distinct completions on a production stack — any single flipped token then diverges the whole continuation. LLM-42 builds a serving system on precisely this fact.
  • Synthesis for Experiment G: expect a bimodal picture — most tokens have comfortable margins (hence 90%+ same-top-token at 4-bit, 70%+ at half-depth), but a persistent 5–15% of tokens are genuinely unstable, and those tokens are disproportionately the semantically load-bearing ones (flips paper; hard-token analyses in QuickSilver's entropy gating). Measuring the joint distribution (margin of cheap model, agreement with full model) on our own hardware/models is charter Experiment G and is cheap to run. No paper we found reports this joint distribution directly — a small but real measurement gap we can fill and publish.

Distribution-level repair

  • Speculative sampling's rejection rule is the only known mechanism that converts "approximately right most of the time" into "exactly the target distribution always" at bounded extra cost (one target pass per block). Tree variants (SpecInfer) generalize it to multi-candidate verification. Any localvm design wanting exactness should reuse this machinery unchanged.

# 5. Relevance to localvm-research: uncertainty-gated weight materialization

The proposed regime. Resident in unified memory: a low-bit base model (2–4 bit, MLX-native layout) + KV cache + a small error/uncertainty apparatus. On SSD: precision residuals (Matryoshka/any-precision layout so higher precision = base bits + extra bit-planes, or additive-quantization residual codebooks). Per token: run the base pass; compute the decision margin (plus, optionally, a learned per-layer error estimate from residual-stream features); if the decision is certifiably/confidently stable, emit; otherwise defer — keep drafting with the base and periodically verify the accumulated low-confidence block with residual-augmented weights streamed once per block, LayerSkip-economics style.

What exists to build on (per section above):

  • Agreement rates (90%+ at 4-bit; >90% QuantSpec acceptance) say the escalation rate can plausibly be 5–15% of tokens.
  • SpecExec proves the amortization math on consumer hardware: ~20 accepted tokens per full-weight sweep turns a 4.5 s/token offloaded model into 4–6 tok/s. Our version amortizes residual streaming instead of full-model streaming — strictly less data.
  • Rejection sampling (exactness) or CALM risk calibration (guaranteed consistency at chosen ε) supply the correctness story — nothing new needs to be proven mathematically for either mode.
  • MLX/Metal feasibility is de-risked by ReDrafter-on-MLX (Apple shipped speculative verification kernels on Metal) and by llama.cpp/mlx_lm speculative modes; batch-of-k verification is bandwidth-neutral on unified memory exactly as on CUDA.
  • Any-precision/Matryoshka weight layouts (QuickSilver, MoBiQuant, DP-LLM overlays) show prefix-decodable multi-precision storage is practical.

What's missing (the research):

  1. The joint stability measurement (margin of base vs. agreement with full model, per task domain) — nobody has published it; it decides whether the escalation rate is 5% or 40%. → Experiment G, run first.
  2. A cheap, calibrated per-token error signal that accounts for depth-compounding — QEP says errors compound across layers; EMNLP-2025 says compounding is predictable from residual magnitudes (ρ=0.82). A logit-margin threshold alone may be miscalibrated for the tokens where the base is confidently wrong (the dangerous quadrant). This is where formal §4.9 tools are vacuous and a small learned predictor + distribution-free calibration (CALM's recipe) is the realistic instrument.
  3. Incremental refinement kernels: recomputing the whole pass with residuals doubles compute; the GMRES-IR analogy suggests computing ΔY = (ΔW)X only (residual weights × cached activations) and adding it — requires caching per-layer activations for the deferred block (memory cost ~ activations × block length, cheap next to weights) and a Metal kernel for sparse/low-rank residual matmul. Nobody has published this primitive; it is also exactly charter Experiment D/E territory.
  4. Block-deferred verification policy: verifying uncertain tokens one-by-one would thrash the SSD; batching them inherits speculative decoding's rollback problem (a flipped early token invalidates later drafted tokens). The right policy (when to flush the uncertain block) is an open scheduling problem — SpecExec's budget analysis and Kangaroo's confidence-stop are the starting points.
  5. Failure mode to respect: if the base is 2-bit rather than 4-bit, agreement may collapse (llama.cpp IQ1/IQ2 blind-test results; max-KLD outliers of 20+ nats even at 4-bit) and escalation could approach 100% on hard domains — the charter's §17 "required working set is nearly the entire model" failure. The measurement in (1) resolves this cheaply before any engineering.

# Sources