---
project: localvm-research
document: docs/publications/TR-01
title: "Margin-Gated Deferred Refinement: Streaming Higher-Precision LLM Quality Than Fits in Memory on Consumer Apple Silicon"
author: Simon-Pierre Boucher
contact: contact@spboucher.ai
created: 2026-08-12
status: final
type: technical-report
number: TR-01
---

# Margin-Gated Deferred Refinement: Streaming Higher-Precision LLM Quality Than Fits in Memory on Consumer Apple Silicon

**Technical Report TR-01 · August 12, 2026**
**Simon-Pierre Boucher** — contact@spboucher.ai
*All experiments in this report are reproducible from the repository: each result
carries a commit hash, configuration, seed, and hardware manifest. Raw results,
code, and the full research log are browsable on this platform.*

## Abstract

We ask whether an existing pretrained LLM whose preferred precision does not fit
in a consumer Mac's unified memory can still contribute its quality locally. On
an Apple M5 Max (48 GB), we first measure the substrate: the internal NVMe
sustains ~13.1 GB/s of sequential reads but only 67 MB/s at 4 KiB random QD1, and
saturated Metal GPU compute perturbs SSD streaming by less than 5%. We then show
that the top-1 logit margin of a low-bit quantized model strongly predicts its
disagreements with the full-precision model (AUROC 0.85–0.98), improving with
scale, while three plausible fine-grained escalation strategies — sensitive
layers, important weight blocks, temporal working sets — are each *refuted* by
measurement on a modern dense SwiGLU architecture. These results select a
specific runtime: a resident 4-bit base generates optimistically and defers
low-margin tokens; a same-checkpoint 8-bit variant, streamed from SSD (layer by
layer when it cannot co-reside), verifies windows and rolls back flips. At 1.7B,
verified output is statistically indistinguishable from resident-q8 while
streaming 237 MB/token — 9× under the checkpoint — with a runtime knob τ trading
bytes for quality. At 32B, where q8 (34.8 GB) cannot reside beside the base at
all, the system improves judge-scored quality by +0.28 nats over the only
configuration that fits, at 3.7 GB/token and 1.7 tok/s. We release all negative
results, and we attribute prior art precisely: the components are anticipated
(QSpec/QuantSpec; Lever; CATS; SpecExec; UGSD); the assembled composition, its
bytes-per-token objective, and its unified-memory instantiation were not found.

## 1. Problem

A pretrained checkpoint has one size; a machine has another. Standard practice
compresses the model until it fits (quantization) and accepts the quality loss,
or offloads weights and accepts catastrophic latency. The research question of
this project (charter §2) is whether these are really the only options:

> Can total model size, resident size, and bytes-read-per-token be decoupled,
> so that RAM bounds the *working set* rather than the *model*?

This report documents the first architecture selected and validated by our
falsification-driven campaign: eight preregistered micro-experiments (three
confirmed, three refuted, two substrate/prototype), each with explicit kill
criteria, followed by a two-scale prototype and an adversarial novelty check.

## 2. The substrate: what an Apple SSD will actually give you

Experiment H measured the internal NVMe (AP2048Z, 2 TB) under genuinely cold
conditions — after discovering that `F_NOCACHE` does **not** bypass pages already
resident in the unified page cache, so naïve "cold" benchmarks measure RAM
(42–115 GB/s readings that exceed the device's physics). With test files written
uncached and an `iostat` ground-truth log, the envelope is:

![Figure 1 — Cold random-read throughput of the internal NVMe by block size and queue depth (log–log). The device ceiling ≈ 13.1 GB/s is reached at 1 MiB blocks, QD 8; 4 KiB QD1 delivers 67 MB/s — a 200× spread.](figures/f1_ssd_envelope.svg)

**Figure 1.** Cold random-read throughput by block size and queue depth
(3 repeats/cell, iostat-validated). The fetch contract for any weight-streaming
design on this hardware: **≥ 256 KiB blocks at QD ≥ 4, or sequential**. Saturated
MLX matmul load concurrently costs < 5% — I/O and compute overlap freely on
unified memory.

Two corollaries shaped everything downstream: storage bandwidth is *not* the
binding constraint (~650 MB/token is realistic at interactive rates), and the
constraint that *is* binding is the access pattern.

## 3. Decisions are predictable; fine-grained escalation is not

**Margins predict disagreement (expG, confirmed).** Teacher-forcing quantized
variants of Qwen3-1.7B over 48 bf16 greedy trajectories (6 domains, 6 144
positions/bit-width), the low-bit model's top-1 logit margin predicts whether its
greedy decision matches the reference: AUROC 0.852 (3-bit), 0.898 (4-bit), 0.975
(8-bit), uniform across domains. Disagreements concentrate sharply at small
margins (median disagree-margin 0.75 vs agree-margin 6.5 at 4-bit). Naïve affine
2-bit quantization, by contrast, destroys the model outright (3% agreement,
KL ≈ 11.9) — the known 2–3-bit representational cliff, measured here with an
amplitude perplexity tables understate.

![Figure 2 — Fraction of tokens that must be escalated to reach 99% greedy agreement, by base precision and model scale. Every point improves at 8B.](figures/f2_escalation_scale.svg)

**Figure 2.** Escalation needed for 99% agreement falls with both precision and
model scale (36.6% → 22.6% at 4-bit going from 1.7B to 8B). The gating economics
improve exactly where the architecture is needed — larger models.

**Progressive residual coding works (expD, confirmed).** Affine residual ladders
(3/3+3/3+3+3 and 4/4+4 bits, group 64) converge monotonically (hidden-state
error ÷5 per stage; KL ÷30–60), and a two-tier margin policy over them reaches
97.6% agreement while consulting the refinement for only 35% of tokens. At equal
*static* bytes, flat quantization is mildly better — the ladder's value is that
quality becomes a *runtime* variable, not a compression win.

**Three escalation strategies die (expF, expA, expB — refuted).** To make
per-token refinement cheap we tested, with preregistered kill criteria:
layer-restricted escalation (sensitivity spread across depth groups is only
1.5×; repairing the top 29% of layers recovers just 24% of the loss); block-level
selection (95% of SwiGLU FFN energy needs 77% of 64-neuron blocks — real
neuron-level concentration exists at 20% but is *scattered*, and bundling to
SSD-viable granularity destroys it); and temporal working sets (the 95%-energy
block set's union covers 99.9% of the layer within 128 tokens; Jaccard between
consecutive tokens equals the random-set null). The DejaVu / LLM-in-a-flash
paradigm measurably does not transfer to dense modern SwiGLU models. What
remains is **batch amortization** — and Figure 1 says sequential amortized
streaming is nearly free.

## 4. The architecture

The evidence selects a specific runtime, assembled from measured parts:

1. **Resident base**: the q4 quantization of the model lives in unified memory
   and generates greedily with a KV cache, recording each decision's margin.
2. **Deferral**: decisions with margin < τ are flagged; nothing blocks.
3. **Amortized verification sweeps**: every W tokens, the *same checkpoint's*
   q8 variant teacher-forces the window in one pass, feeding only the unseen
   suffix into a persistent incremental verify-KV-cache. When q8 fits beside
   the base (small models), it is simply resident; when it does not, it is
   **streamed layer-by-layer** from SSD (materialize → compute → re-lazify),
   one sequential pass over the checkpoint per sweep at the device ceiling.
4. **Rollback**: at the first checked flip (margin mode: deferred positions
   only; verify-all mode: every position), the runtime trims both KV caches,
   accepts the q8 token, and resumes.

τ is a user-facing knob: it prices output quality in verification bytes.

## 5. Results

### 5.1 Quality–bytes continuum at 1.7B

All outputs are scored by the bf16 reference model (mean per-token logprob of
the generated text). Token-exact fidelity is *not* a usable target on this
hardware: the q8 model itself flips 1.56% of its own greedy tokens between
prefill and incremental decode (Metal numerics), so even the reference only
"agrees with itself" at ~0.88 sequence similarity over 128 tokens.

![Figure 3 — Output quality (bf16-judge mean logprob) versus verification bytes streamed per token at 1.7B. τ traces a continuum from pure-q4 to the resident-q8 ceiling.](figures/f3_quality_bytes_continuum.svg)

**Figure 3.** The τ continuum at Qwen3-1.7B (24 prompts × 128 tokens, W=32):
150 MB/token buys 47% of the q4→q8 quality gap, 211 MB/token buys 83%, and
verify-all (237 MB/token — 9× under the 2.15 GB checkpoint) is statistically
indistinguishable from resident q8 (−0.1297 vs −0.1328). Throughput 69–110
tok/s against 287 (pure q4) and 200 (pure q8).

### 5.2 The target regime: 32B on a 48 GB machine

At Qwen3-32B, q4 (17.5 GB) is resident and q8 (34.8 GB) **cannot co-reside**
— total representation 52 GB on a 48 GB machine. Sweeps stream it sequentially
at ~11.6 GB/s (≈ 90% of the device ceiling), 3.0 s per sweep.

![Figure 4 — Judge-scored quality at 32B: the streamed-verifier configurations versus the only configuration that actually fits.](figures/f4_32b_outofcore.svg)

**Figure 4.** Qwen3-32B, 12 prompts × 96 tokens, independent 8B-bf16 judge.
Verify-all improves quality by +0.28 nats (38% relative) over pure q4 — the
only alternative that fits — at 3.72 GB/token logical (9.4× under the streamed
checkpoint) and 1.69 tok/s. A window-doubling test (W=64) improved bytes/token
by only 5%: the effective inter-sweep distance is bounded by the rollback rate
(~1/0.12 ≈ 8–15 tokens), not by W — the identified levers are τ calibration and
overlapping sweep I/O with generation.

## 6. Relation to prior work (from the adversarial novelty check)

Every *component* of this system is anticipated, and we attribute precisely:
precision-tiered draft/verify with shared weights (QSpec; Apple's QuantSpec;
ML-SpecQD); resident-draft with flash-streamed target on memory-limited devices
(Lever, 2026; CATS, 2026 — the closest systems, both published after our
literature sweep began); uncertainty-gated verification skipping (U-HLM, UGSD —
against cloud verifiers); window-amortized verification economics (SpecExec);
per-layer weight streaming in MLX (mlx-flash); mmap-based weight paging
(llama.cpp, proposed-but-unbuilt resident-speculator variant in discussion
#10466). Against this, four narrow claims survived adversarial search:
**(N1)** margin-gating where the gated resource is the *weight-streaming I/O*
of the same model's higher-precision variant on a single machine; **(N2)**
bytes-streamed-per-token vs checkpoint size as the explicit objective, with τ
exposed as a bytes↔quality knob; **(N3)** the training-free same-checkpoint
q4/q8 split demonstrated where the verifier physically cannot reside; **(N4)**
the unified-memory/Metal instantiation with persistent incremental verify-KV
cache and trim rollback. The margin is thin and shrinking; we recommend
re-running the novelty check before any external submission.

## 7. Limitations

Prototype speed (1.69 tok/s at 32B) is below interactive. The 32B quality
ceiling is unmeasurable on this machine by construction; the 8B judge is a
proxy, with no error bars at n=12; no task-level benchmarks (e.g., GSM8K) have
been run yet. We have not baselined against resident q5/q6 at 32B — which *do*
fit in 48 GB and may dominate this system at this scale; the architecture's
necessary regime is larger models (70B-class), not yet tested. All experiments
used one model family (Qwen3) and one machine.

## 8. Reproducibility

Hardware: Apple M5 Max, 6P+12E CPU, 40-core GPU, 48 GB unified memory, APPLE
SSD AP2048Z 2 TB, macOS 27.0. Stack: Python 3.14, MLX 0.32.0, mlx-lm 0.31.3.
Every result JSON embeds this manifest. Code paths:
`experiments/micro/exp{A,B,D,F,G,H}_*`, `experiments/candidate_01/`
(runtime.py, streaming_verifier.py, benchmark.py, benchmark_scale.py). Figures
in this report are regenerated from committed results by
`tools/make_pub_figures.py`. The append-only research log
(`research/LOG.md`) records every decision, including the three refuted
hypotheses and both measurement traps (F_NOCACHE semantics; prefill/decode
argmax instability).

## References

Key prior art (full 300+ entry bibliography with access dates on the
[Bibliography](/doc/research/bibliography.md) page; complete novelty analysis in
[novelty_check.md](/doc/research/novelty_check.md)):
QSpec (arXiv 2410.11305) · QuantSpec (Apple, arXiv 2502.10424) · Lever (arXiv
2605.16786) · CATS (arXiv 2605.11186) · UGSD (arXiv 2603.11397) · U-HLM (arXiv
2412.12687) · SpecExec (arXiv 2406.02532) · LLM in a flash (Apple, arXiv
2312.11514) · PowerInfer-2 (arXiv 2406.06282) · Any-Precision LLM (ICML 2024) ·
MatQuant (ICLR 2025) · BitStack (arXiv 2410.23918) · ParetoQ (arXiv 2502.02631) ·
SparseGPT/Wanda/DejaVu/TEAL (per state_of_the_art.md).