spb/localvm-research Public License

1---2project: localvm-research3document: docs/publications/TR-014title: "Margin-Gated Deferred Refinement: Streaming Higher-Precision LLM Quality Than Fits in Memory on Consumer Apple Silicon"5author: Simon-Pierre Boucher6contact: contact@spboucher.ai7created: 2026-08-128status: final9type: technical-report10number: TR-0111---1213# Margin-Gated Deferred Refinement: Streaming Higher-Precision LLM Quality Than Fits in Memory on Consumer Apple Silicon1415**Technical Report TR-01 · August 12, 2026**16**Simon-Pierre Boucher** — contact@spboucher.ai17*All experiments in this report are reproducible from the repository: each result18carries a commit hash, configuration, seed, and hardware manifest. Raw results,19code, and the full research log are browsable on this platform.*2021## Abstract2223We ask whether an existing pretrained LLM whose preferred precision does not fit24in a consumer Mac's unified memory can still contribute its quality locally. On25an Apple M5 Max (48 GB), we first measure the substrate: the internal NVMe26sustains ~13.1 GB/s of sequential reads but only 67 MB/s at 4 KiB random QD1, and27saturated Metal GPU compute perturbs SSD streaming by less than 5%. We then show28that the top-1 logit margin of a low-bit quantized model strongly predicts its29disagreements with the full-precision model (AUROC 0.85–0.98), improving with30scale, while three plausible fine-grained escalation strategies — sensitive31layers, important weight blocks, temporal working sets — are each *refuted* by32measurement on a modern dense SwiGLU architecture. These results select a33specific runtime: a resident 4-bit base generates optimistically and defers34low-margin tokens; a same-checkpoint 8-bit variant, streamed from SSD (layer by35layer when it cannot co-reside), verifies windows and rolls back flips. At 1.7B,36verified output is statistically indistinguishable from resident-q8 while37streaming 237 MB/token — 9× under the checkpoint — with a runtime knob τ trading38bytes for quality. At 32B, where q8 (34.8 GB) cannot reside beside the base at39all, the system improves judge-scored quality by +0.28 nats over the only40configuration that fits, at 3.7 GB/token and 1.7 tok/s. We release all negative41results, and we attribute prior art precisely: the components are anticipated42(QSpec/QuantSpec; Lever; CATS; SpecExec; UGSD); the assembled composition, its43bytes-per-token objective, and its unified-memory instantiation were not found.4445## 1. Problem4647A pretrained checkpoint has one size; a machine has another. Standard practice48compresses the model until it fits (quantization) and accepts the quality loss,49or offloads weights and accepts catastrophic latency. The research question of50this project (charter §2) is whether these are really the only options:5152> Can total model size, resident size, and bytes-read-per-token be decoupled,53> so that RAM bounds the *working set* rather than the *model*?5455This report documents the first architecture selected and validated by our56falsification-driven campaign: eight preregistered micro-experiments (three57confirmed, three refuted, two substrate/prototype), each with explicit kill58criteria, followed by a two-scale prototype and an adversarial novelty check.5960## 2. The substrate: what an Apple SSD will actually give you6162Experiment H measured the internal NVMe (AP2048Z, 2 TB) under genuinely cold63conditions — after discovering that `F_NOCACHE` does **not** bypass pages already64resident in the unified page cache, so naïve "cold" benchmarks measure RAM65(42–115 GB/s readings that exceed the device's physics). With test files written66uncached and an `iostat` ground-truth log, the envelope is:6768![Figure 1 — Cold random-read throughput of the internal NVMe by block size and queue depth (log–log). The device ceiling ≈ 13.1 GB/s is reached at 1 MiB blocks, QD 8; 4 KiB QD1 delivers 67 MB/s — a 200× spread.](figures/f1_ssd_envelope.svg)6970**Figure 1.** Cold random-read throughput by block size and queue depth71(3 repeats/cell, iostat-validated). The fetch contract for any weight-streaming72design on this hardware: **≥ 256 KiB blocks at QD ≥ 4, or sequential**. Saturated73MLX matmul load concurrently costs < 5% — I/O and compute overlap freely on74unified memory.7576Two corollaries shaped everything downstream: storage bandwidth is *not* the77binding constraint (~650 MB/token is realistic at interactive rates), and the78constraint that *is* binding is the access pattern.7980## 3. Decisions are predictable; fine-grained escalation is not8182**Margins predict disagreement (expG, confirmed).** Teacher-forcing quantized83variants of Qwen3-1.7B over 48 bf16 greedy trajectories (6 domains, 6 14484positions/bit-width), the low-bit model's top-1 logit margin predicts whether its85greedy decision matches the reference: AUROC 0.852 (3-bit), 0.898 (4-bit), 0.97586(8-bit), uniform across domains. Disagreements concentrate sharply at small87margins (median disagree-margin 0.75 vs agree-margin 6.5 at 4-bit). Naïve affine882-bit quantization, by contrast, destroys the model outright (3% agreement,89KL ≈ 11.9) — the known 2–3-bit representational cliff, measured here with an90amplitude perplexity tables understate.9192![Figure 2 — Fraction of tokens that must be escalated to reach 99% greedy agreement, by base precision and model scale. Every point improves at 8B.](figures/f2_escalation_scale.svg)9394**Figure 2.** Escalation needed for 99% agreement falls with both precision and95model scale (36.6% → 22.6% at 4-bit going from 1.7B to 8B). The gating economics96improve exactly where the architecture is needed — larger models.9798**Progressive residual coding works (expD, confirmed).** Affine residual ladders99(3/3+3/3+3+3 and 4/4+4 bits, group 64) converge monotonically (hidden-state100error ÷5 per stage; KL ÷30–60), and a two-tier margin policy over them reaches10197.6% agreement while consulting the refinement for only 35% of tokens. At equal102*static* bytes, flat quantization is mildly better — the ladder's value is that103quality becomes a *runtime* variable, not a compression win.104105**Three escalation strategies die (expF, expA, expB — refuted).** To make106per-token refinement cheap we tested, with preregistered kill criteria:107layer-restricted escalation (sensitivity spread across depth groups is only1081.5×; repairing the top 29% of layers recovers just 24% of the loss); block-level109selection (95% of SwiGLU FFN energy needs 77% of 64-neuron blocks — real110neuron-level concentration exists at 20% but is *scattered*, and bundling to111SSD-viable granularity destroys it); and temporal working sets (the 95%-energy112block set's union covers 99.9% of the layer within 128 tokens; Jaccard between113consecutive tokens equals the random-set null). The DejaVu / LLM-in-a-flash114paradigm measurably does not transfer to dense modern SwiGLU models. What115remains is **batch amortization** — and Figure 1 says sequential amortized116streaming is nearly free.117118## 4. The architecture119120The evidence selects a specific runtime, assembled from measured parts:1211221. **Resident base**: the q4 quantization of the model lives in unified memory123   and generates greedily with a KV cache, recording each decision's margin.1242. **Deferral**: decisions with margin < τ are flagged; nothing blocks.1253. **Amortized verification sweeps**: every W tokens, the *same checkpoint's*126   q8 variant teacher-forces the window in one pass, feeding only the unseen127   suffix into a persistent incremental verify-KV-cache. When q8 fits beside128   the base (small models), it is simply resident; when it does not, it is129   **streamed layer-by-layer** from SSD (materialize → compute → re-lazify),130   one sequential pass over the checkpoint per sweep at the device ceiling.1314. **Rollback**: at the first checked flip (margin mode: deferred positions132   only; verify-all mode: every position), the runtime trims both KV caches,133   accepts the q8 token, and resumes.134135τ is a user-facing knob: it prices output quality in verification bytes.136137## 5. Results138139### 5.1 Quality–bytes continuum at 1.7B140141All outputs are scored by the bf16 reference model (mean per-token logprob of142the generated text). Token-exact fidelity is *not* a usable target on this143hardware: the q8 model itself flips 1.56% of its own greedy tokens between144prefill and incremental decode (Metal numerics), so even the reference only145"agrees with itself" at ~0.88 sequence similarity over 128 tokens.146147![Figure 3 — Output quality (bf16-judge mean logprob) versus verification bytes streamed per token at 1.7B. τ traces a continuum from pure-q4 to the resident-q8 ceiling.](figures/f3_quality_bytes_continuum.svg)148149**Figure 3.** The τ continuum at Qwen3-1.7B (24 prompts × 128 tokens, W=32):150150 MB/token buys 47% of the q4→q8 quality gap, 211 MB/token buys 83%, and151verify-all (237 MB/token — 9× under the 2.15 GB checkpoint) is statistically152indistinguishable from resident q8 (−0.1297 vs −0.1328). Throughput 69–110153tok/s against 287 (pure q4) and 200 (pure q8).154155### 5.2 The target regime: 32B on a 48 GB machine156157At Qwen3-32B, q4 (17.5 GB) is resident and q8 (34.8 GB) **cannot co-reside**158— total representation 52 GB on a 48 GB machine. Sweeps stream it sequentially159at ~11.6 GB/s (≈ 90% of the device ceiling), 3.0 s per sweep.160161![Figure 4 — Judge-scored quality at 32B: the streamed-verifier configurations versus the only configuration that actually fits.](figures/f4_32b_outofcore.svg)162163**Figure 4.** Qwen3-32B, 12 prompts × 96 tokens, independent 8B-bf16 judge.164Verify-all improves quality by +0.28 nats (38% relative) over pure q4 — the165only alternative that fits — at 3.72 GB/token logical (9.4× under the streamed166checkpoint) and 1.69 tok/s. A window-doubling test (W=64) improved bytes/token167by only 5%: the effective inter-sweep distance is bounded by the rollback rate168(~1/0.12 ≈ 8–15 tokens), not by W — the identified levers are τ calibration and169overlapping sweep I/O with generation.170171## 6. Relation to prior work (from the adversarial novelty check)172173Every *component* of this system is anticipated, and we attribute precisely:174precision-tiered draft/verify with shared weights (QSpec; Apple's QuantSpec;175ML-SpecQD); resident-draft with flash-streamed target on memory-limited devices176(Lever, 2026; CATS, 2026 — the closest systems, both published after our177literature sweep began); uncertainty-gated verification skipping (U-HLM, UGSD —178against cloud verifiers); window-amortized verification economics (SpecExec);179per-layer weight streaming in MLX (mlx-flash); mmap-based weight paging180(llama.cpp, proposed-but-unbuilt resident-speculator variant in discussion181#10466). Against this, four narrow claims survived adversarial search:182**(N1)** margin-gating where the gated resource is the *weight-streaming I/O*183of the same model's higher-precision variant on a single machine; **(N2)**184bytes-streamed-per-token vs checkpoint size as the explicit objective, with τ185exposed as a bytes↔quality knob; **(N3)** the training-free same-checkpoint186q4/q8 split demonstrated where the verifier physically cannot reside; **(N4)**187the unified-memory/Metal instantiation with persistent incremental verify-KV188cache and trim rollback. The margin is thin and shrinking; we recommend189re-running the novelty check before any external submission.190191## 7. Limitations192193Prototype speed (1.69 tok/s at 32B) is below interactive. The 32B quality194ceiling is unmeasurable on this machine by construction; the 8B judge is a195proxy, with no error bars at n=12; no task-level benchmarks (e.g., GSM8K) have196been run yet. We have not baselined against resident q5/q6 at 32B — which *do*197fit in 48 GB and may dominate this system at this scale; the architecture's198necessary regime is larger models (70B-class), not yet tested. All experiments199used one model family (Qwen3) and one machine.200201## 8. Reproducibility202203Hardware: Apple M5 Max, 6P+12E CPU, 40-core GPU, 48 GB unified memory, APPLE204SSD AP2048Z 2 TB, macOS 27.0. Stack: Python 3.14, MLX 0.32.0, mlx-lm 0.31.3.205Every result JSON embeds this manifest. Code paths:206`experiments/micro/exp{A,B,D,F,G,H}_*`, `experiments/candidate_01/`207(runtime.py, streaming_verifier.py, benchmark.py, benchmark_scale.py). Figures208in this report are regenerated from committed results by209`tools/make_pub_figures.py`. The append-only research log210(`research/LOG.md`) records every decision, including the three refuted211hypotheses and both measurement traps (F_NOCACHE semantics; prefill/decode212argmax instability).213214## References215216Key prior art (full 300+ entry bibliography with access dates on the217[Bibliography](/doc/research/bibliography.md) page; complete novelty analysis in218[novelty_check.md](/doc/research/novelty_check.md)):219QSpec (arXiv 2410.11305) · QuantSpec (Apple, arXiv 2502.10424) · Lever (arXiv2202605.16786) · CATS (arXiv 2605.11186) · UGSD (arXiv 2603.11397) · U-HLM (arXiv2212412.12687) · SpecExec (arXiv 2406.02532) · LLM in a flash (Apple, arXiv2222312.11514) · PowerInfer-2 (arXiv 2406.06282) · Any-Precision LLM (ICML 2024) ·223MatQuant (ICLR 2025) · BitStack (arXiv 2410.23918) · ParetoQ (arXiv 2502.02631) ·224SparseGPT/Wanda/DejaVu/TEAL (per state_of_the_art.md).225