# project: localvm-research document: research/novelty_check author: Simon-Pierre Boucher contact: contact@spboucher.ai created: 2026-08-12 status: draft

# Phase 11 — Novelty verification of candidate_01 (margin-gated deferred refinement with a streamed verifier)

Method. Per charter §19, the working assumption was NOT NOVEL until evidence suggests otherwise. The search was adversarial: for each component of the architecture, and for the composition, I searched for conceptual equivalents (not just identical terminology) across arXiv (with emphasis on 2025–2026, i.e., after the Phase 1 sweep of 2026-08-11), ACL/EMNLP, MLSys/OSDI/ASPLOS-adjacent venues, OpenReview, and the llama.cpp / MLX / mlx-lm GitHub ecosystems. All sources are listed in §5 with access date 2026-08-12. The two closest prior systems (Lever, CATS — both May 2026, both absent from the Phase 1 notes) were read from their full PDFs, not abstracts.

# 1. The architecture under check (précis)

A single-machine runtime for unified-memory consumer hardware (Apple Silicon, MLX/Metal):

• A q4 quantization of the model is resident and decodes greedily at full speed, recording the top-1 logit margin of every token decision; low-margin decisions are deferred (margin mode) rather than escalated immediately.
• Every W tokens (W=32 measured), a q8 variant of the same model — which does not fit in memory beside the base — is streamed layer-by-layer from NVMe (materialize → compute → re-lazify; one strictly sequential pass over the checkpoint per sweep at the SSD device ceiling, ~11.6 GB/s) and teacher-forces the window in one pass against a persistent incremental q8 KV cache (O(window), not O(context)).
• On the first flip (checked at deferred positions only in margin mode; at all positions in verify-all mode) the runtime rolls back via KV-cache trim, accepts the q8 token, and resumes. Acceptance is deliberately approximate in margin mode (unchecked high-margin positions are trusted); verify-all is greedy-exact up to Metal numerics.
• Objective and headline metric: bytes streamed per generated token, decoupled from checkpoint size (9.4× under the q8 checkpoint at W=32 on Qwen3-32B); the margin threshold τ is a runtime knob trading bytes for quality (47%/83%/~100% of the q4→q8 quality gap at 150/211/237 MB/token on 1.7B; +0.28 nats over pure-q4 at 32B where q8 cannot run at all).

# 2. Per-component prior-art tables

Verdict scale: ANTICIPATED (the component exists in print, differences are cosmetic) · PARTIALLY ANTICIPATED (the idea exists but in a materially different setting or without a load-bearing property) · NOT FOUND (no conceptual equivalent located).

# 2.1 Precision-tiered self-speculation (q4 draft / q8 verify, shared weights)

# 2.2 Confidence/margin-gated selective verification or escalation

# 2.3 Layer-by-layer weight streaming from storage during a forward pass (unified memory)

# 2.4 Windowed/batched deferred verification with rollback, where verifier cost is I/O

# 2.5 The composition as a whole

Search phrasings tried (plus synonyms): "resident draft streamed verifier", "quantized draft full model verification SSD", "deferred verification language model rollback", "amortized weight streaming verification", "two-tier quantization runtime offload verify", "speculative decoding out-of-core verifier", "verifier offloading speculative decoding", "uncertainty triggered precision escalation", "self-speculative decoding offloading edge", "flash-backed target model speculative decoding", plus GitHub sweeps of llama.cpp / ik_llama.cpp / mlx / mlx-lm issues and community MLX streaming projects.

Composition verdict: NOT FOUND as assembled — but only by the conjunction, and the margin is thin. Every load-bearing part exists separately, three of them in systems published in 2025–2026 that the Phase 1 sweep missed (Lever, CATS, UGSD). Prior-art density around this exact point is increasing quarter over quarter.

# 3. Honest bottom line — what survives, what must be attributed

Must be attributed (do not claim):

1. Precision-tiered draft/verify on shared weights — QSpec, QuantSpec, ML-SpecQD own it.
2. Resident draft + storage-streamed verifier, amortized over a token window — Lever and CATS own the category (with SpecExec owning the amortization economics and llama.cpp #10466 as a public 2024 proposal). Our verify-all mode is, at the category level, "Lever/CATS with a precision-cast draft on Apple Silicon."
3. Uncertainty-gated skipping of verification / escalation to an expensive tier — U-HLM, UGSD, SPRINTER, with the older lineage BiLD / CALM / cascades.
4. Deliberately lossy acceptance — the relaxed/lossy-verification literature (top-R rank acceptance in UGSD; judge/relaxed acceptance lines; taxonomy in arXiv 2607.26627).
5. Rollback via KV trim; teacher-forcing a window in one pass — standard speculative decoding machinery.
6. Layer-by-layer materialize→compute→release streaming in MLX on Apple Silicon — mlx-flash (and AirLLM/ZeRO-Inference generically). We did not invent the scheduling pattern; we gave the streamed pass a verifier role.

Surviving as (narrowly) novel contributions:

• N1 — Margin-gated deferral of verification bytes. Using the resident draft's top-1 logit margin to decide whether higher-precision weights are streamed from local storage at all (which positions a sweep must check, and hence how much I/O the quality repair costs). Every located gating system gates compute or communication to a different, remote or resident model; none gates the weight-streaming I/O of the same model's higher-precision representation. NOT FOUND after targeted search.
• N2 — Bytes-streamed-per-token as the explicit objective, reported against checkpoint size (9.4× under at W=32), with the τ threshold exposed as a runtime bytes↔quality knob (47%→83%→~100% of the q4→q8 gap at 150→211→237 MB/token). Dynamic-precision systems (PMPD, HOBBIT, DP-LLM) have precision knobs without verification; lossy verification has quality knobs without a storage tier; no system prices quality in bytes-of-weights-streamed. NOT FOUND as a stated objective/metric.
• N3 — The training-free same-checkpoint precision pair as the draft/verify split of an out-of-core runtime (q4 resident / q8 streamed, no distillation, no auxiliary model), making "quality level that does not fit in RAM" usable on unified memory — demonstrated where the alternative (resident q8) is physically impossible (32B on 48 GB), judged +0.28 nats over the only configuration that fits. The conjunction is unoccupied; each half is not.
• N4 (engineering, weaker) — The unified-memory instantiation: strictly sequential sweep at SSD device ceiling, persistent incremental verify-KV-cache (O(window) sweeps) with trim-rollback, measured on Metal with the prefill/decode numerics ceiling identified. Claimable only as a measured system contribution on this platform, not as a concept.

Claims that are now weaker than when candidate_01 was designed: "first resident-draft/streamed-verifier runtime" (false — Lever, CATS), "first selective verification by draft uncertainty" (false — U-HLM, SPRINTER, UGSD), "first precision-pair self-speculation" (false — QSpec family), "first out-of-core amortized verification" (false — SpecExec, Lever, CATS).

# 4. Recommended claim language for the README (defensible phrasing)

What is new here is the conjunction, and two specific properties of it. Precision-pair self-speculation (QSpec, QuantSpec, ML-SpecQD), storage-streamed verifiers behind a resident draft (Lever, CATS), window-amortized verification of an offloaded target (SpecExec), and uncertainty-gated escalation (BiLD, CALM, U-HLM, UGSD) all exist separately. To our knowledge, no prior system (i) uses the draft's own top-1 logit margin to gate how many bytes of higher-precision weights are streamed from local NVMe to repair quality — making the margin threshold τ a runtime knob that prices output quality in bytes-per-token — and (ii) does so with a training-free precision pair of the same checkpoint (4-bit resident, 8-bit streamed) on unified-memory hardware, so that a quality level whose weights cannot reside in RAM remains usable, with bytes streamed per token decoupled from checkpoint size (measured 9.4× under the streamed checkpoint at W=32). We explicitly do not claim novelty for precision-tiered drafting, streamed verification, windowed rollback, or confidence-gated escalation taken individually; the attribution table in research/novelty_check.md maps each component to its prior art.

Avoid in all public text: "first", "novel runtime" unqualified, "no one has streamed a verifier", "quality becomes a runtime variable" without the qualifier via verification bytes (dynamic-precision papers already make quality a runtime variable without verification).

# 5. Sources

All accessed 2026-08-12.

• QSpec: Speculative Decoding with Complementary Quantization Schemes (EMNLP 2025) — https://arxiv.org/abs/2410.11305 (accessed 2026-08-12)
• QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache (Apple, ICML 2025) — https://machinelearning.apple.com/research/quantspec (accessed 2026-08-12)
• ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts — https://arxiv.org/pdf/2503.13565 (accessed 2026-08-12)
• Lever: Speculative LLM Inference on Smartphones (Wang et al., May 2026) — https://arxiv.org/pdf/2605.16786 (accessed 2026-08-12)
• CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration (Han et al., May 2026) — https://arxiv.org/abs/2605.11186 (accessed 2026-08-12)
• CATS code repository — https://github.com/ElizaFuLan/CATS.git (accessed 2026-08-12, via paper)
• LLMCad: Fast and Scalable On-device Large Language Model Inference — https://arxiv.org/pdf/2309.04255 (accessed 2026-08-12)
• EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding (IEEE TMC) — https://ieeexplore.ieee.org/document/10812936/ (accessed 2026-08-12)
• Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference (EMNLP 2025) — https://arxiv.org/abs/2412.18934 (accessed 2026-08-12)
• SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices — https://arxiv.org/html/2505.10259v2 (accessed 2026-08-12)
• Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding (SpecMoEOff) — https://arxiv.org/abs/2508.21706 (accessed 2026-08-12)
• SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices (NeurIPS 2024) — https://arxiv.org/html/2406.02532v1 (accessed 2026-08-12, re-verified)
• SpecInfer: Accelerating LLM Serving with Tree-based Speculative Inference and Verification (ASPLOS 2024) — https://arxiv.org/abs/2305.09781 (accessed 2026-08-12, re-verified)
• TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding (COLM 2024) — https://arxiv.org/abs/2404.11912 (accessed 2026-08-12)
• MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices (LCTES 2026) — https://doi.org/10.1145/3814943.3816174 (accessed 2026-08-12)
• HiSpec: Hierarchical Speculative Decoding for LLMs — https://arxiv.org/abs/2510.01336 (accessed 2026-08-12)
• SPRINTER: Speeding up Speculative Decoding via Sequential Approximate Verification — https://arxiv.org/abs/2502.04557 (accessed 2026-08-12)
• U-HLM: Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Models — https://arxiv.org/abs/2412.12687 (accessed 2026-08-12)
• Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmission — https://arxiv.org/pdf/2505.11788 (accessed 2026-08-12)
• Edge–Cloud Collaborative Speech Emotion Captioning via Token-Level Speculative Decoding (UGSD, Mar 2026) — https://arxiv.org/html/2603.11397v1 (accessed 2026-08-12)
• Quantize-Sample-and-Verify: LLM Acceleration via Adaptive Edge-Cloud Speculative Decoding — https://arxiv.org/pdf/2507.00605 (accessed 2026-08-12)
• Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits — https://arxiv.org/abs/2505.21594 (accessed 2026-08-12)
• DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving — https://arxiv.org/pdf/2511.21669 (accessed 2026-08-12)
• Speculative Decoding with Big Little Decoder (BiLD, NeurIPS 2023) — https://arxiv.org/abs/2302.07863 (accessed 2026-08-12, re-verified)
• Confident Adaptive Language Modeling (CALM, NeurIPS 2022) — https://arxiv.org/abs/2207.07061 (accessed 2026-08-12, re-verified)
• Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes (Jul 2026) — https://arxiv.org/abs/2607.26627 (accessed 2026-08-12)
• Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match (FLy) — https://arxiv.org/pdf/2511.22972 (accessed 2026-08-12)
• A Practical Investigation of Training-free Relaxed Speculative Decoding — https://arxiv.org/html/2607.08690 (accessed 2026-08-12)
• Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptance — https://arxiv.org/html/2507.22424v1 (accessed 2026-08-12)
• When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding — https://arxiv.org/html/2606.30265 (accessed 2026-08-12)
• Statistically-Lossless Quantization of Large Language Models (EAR metric; static, no runtime escalation) — https://arxiv.org/pdf/2605.02404 (accessed 2026-08-12)
• Mixed-Precision Quantization for Language Models: Techniques and Prospects (incl. HOBBIT precision-on-miss loading) — https://arxiv.org/html/2510.16805v1 (accessed 2026-08-12)
• Progressive Mixed-Precision Decoding for Efficient LLM Inference (PMPD) — https://arxiv.org/pdf/2410.13461 (accessed 2026-08-12, re-verified)
• Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware (Jul 2026) — https://arxiv.org/html/2607.17283v1 (accessed 2026-08-12)
• Speculative Speculative Decoding (ICLR 2026) — https://arxiv.org/pdf/2603.03251 (accessed 2026-08-12)
• llama.cpp discussion #10466: Speculative decoding potential for running big LLMs on consumer grade GPUs (resident speculator + layer ping-pong proposal, unimplemented) — https://github.com/ggml-org/llama.cpp/discussions/10466 (accessed 2026-08-12)
• llama.cpp issue #21453: Research: Speculative Decoding for Low-Latency CPU Inference — https://github.com/ggml-org/llama.cpp/issues/21453 (accessed 2026-08-12)
• ik_llama.cpp discussion #839: speculative decoding via network — https://github.com/ikawrakow/ik_llama.cpp/discussions/839 (accessed 2026-08-12)
• mlx-flash: Flash weight streaming for MLX (per-layer StreamingProxy, predictive I/O, every-token streaming, no draft/verify) — https://github.com/matt-k-wong/mlx-flash (accessed 2026-08-12)
• MLX issue #2878: On-disk (out-of-core) streaming of model weights for low-RAM inference (Dec 2025; no speculation proposed) — https://github.com/ml-explore/mlx/issues/2878 (accessed 2026-08-12)
• mlx-moe: router-selected expert streaming from SSD with MLX — https://github.com/mu-hashmi/mlx-moe (accessed 2026-08-12)
• vMLX: SSD expert streaming + disk cache for MoE on Apple Silicon — https://github.com/jjang-ai/vmlx (accessed 2026-08-12)
• dflash-mlx: Lossless DFlash speculative decoding for MLX (adaptive verify block, KV SSD spill; target resident) — https://github.com/bstnxbt/dflash-mlx (accessed 2026-08-12)
• MLX discussion #615: Loading models with mmap (0.025 tok/s overflow result) — https://github.com/ml-explore/mlx/discussions/615 (accessed 2026-08-12, re-verified)
• LLM in a flash: Efficient LLM Inference with Limited Memory (Apple) — https://arxiv.org/abs/2312.11514 (accessed 2026-08-12, re-verified)
• AirLLM (layer-by-layer streaming baseline) — https://github.com/lyogavin/airllm (accessed 2026-08-12)