spb/localvm-research Public License

1---2project: localvm-research3document: research/novelty_check4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6created: 2026-08-127status: draft8---910# Phase 11 — Novelty verification of candidate_01 (margin-gated deferred refinement with a streamed verifier)1112**Method.** Per charter §19, the working assumption was **NOT NOVEL until evidence suggests13otherwise**. The search was adversarial: for each component of the architecture, and for the14composition, I searched for conceptual equivalents (not just identical terminology) across15arXiv (with emphasis on 2025–2026, i.e., after the Phase 1 sweep of 2026-08-11), ACL/EMNLP,16MLSys/OSDI/ASPLOS-adjacent venues, OpenReview, and the llama.cpp / MLX / mlx-lm GitHub17ecosystems. All sources are listed in §5 with access date 2026-08-12. The two closest prior18systems (Lever, CATS — both May 2026, both absent from the Phase 1 notes) were read from19their full PDFs, not abstracts.2021---2223## 1. The architecture under check (précis)2425A single-machine runtime for unified-memory consumer hardware (Apple Silicon, MLX/Metal):2627- A **q4 quantization of the model is resident** and decodes greedily at full speed,28  recording the top-1 logit margin of every token decision; **low-margin decisions are29  deferred** (margin mode) rather than escalated immediately.30- Every W tokens (W=32 measured), a **q8 variant of the same model — which does not fit in31  memory beside the base** — is **streamed layer-by-layer from NVMe**32  (materialize → compute → re-lazify; one strictly sequential pass over the checkpoint per33  sweep at the SSD device ceiling, ~11.6 GB/s) and **teacher-forces the window in one pass**34  against a **persistent incremental q8 KV cache** (O(window), not O(context)).35- On the first flip (checked at deferred positions only in margin mode; at all positions in36  verify-all mode) the runtime **rolls back via KV-cache trim**, accepts the q8 token, and37  resumes. Acceptance is deliberately approximate in margin mode (unchecked38  high-margin positions are trusted); verify-all is greedy-exact up to Metal numerics.39- **Objective and headline metric: bytes streamed per generated token**, decoupled from40  checkpoint size (9.4× under the q8 checkpoint at W=32 on Qwen3-32B); the margin threshold41  τ is a runtime knob trading bytes for quality (47%/83%/~100% of the q4→q8 quality gap at42  150/211/237 MB/token on 1.7B; +0.28 nats over pure-q4 at 32B where q8 cannot run at all).4344---4546## 2. Per-component prior-art tables4748Verdict scale: **ANTICIPATED** (the component exists in print, differences are cosmetic) ·49**PARTIALLY ANTICIPATED** (the idea exists but in a materially different setting or without a50load-bearing property) · **NOT FOUND** (no conceptual equivalent located).5152### 2.1 Precision-tiered self-speculation (q4 draft / q8 verify, shared weights)5354| Closest prior art | What it does | What it does NOT do | Verdict |55|---|---|---|---|56| **QSpec** (EMNLP 2025, arXiv 2410.11305) | Same W4 weights; W4A4 draft, W4A16 verify; near-zero switching cost; lossless; 1.64× | Both operand sets resident; verifies every token; objective is arithmetic speed, not capacity or bytes/token | ANTICIPATED (core mechanism) |57| **QuantSpec** (Apple+Berkeley, ICML 2025) | Self-speculation with 4-bit weights + 4-bit hierarchical KV draft, full-precision verify; >90% acceptance; edge-motivated | Full-precision verifier fully resident and fully read; lossless only; latency objective | ANTICIPATED (core mechanism) |58| **ML-SpecQD** (arXiv 2503.13565) | MXFP4 direct cast of the target as turnkey draft (no training), 16-bit verify, >2× | Same: verifier resident; verify-all; latency framing | ANTICIPATED (core mechanism) |59| Sub-variant (a) verifier kept OUT of memory / streamed | **Lever** and **CATS** stream their verifier from flash (see §2.3/2.4) — but their drafts are a separate small model (Lever) or a distilled shallow sub-network (CATS), **never a precision cast of the same weights** | No published system pairs a low-bit resident cast with a streamed higher-precision variant of the *same* checkpoint | NOT FOUND (within the precision-tier family) |60| Sub-variant (b) verify only uncertainty-flagged positions | Exists outside this family (U-HLM, UGSD, SPRINTER — §2.2) | No precision-tiered system gates which positions are verified | NOT FOUND (within the precision-tier family) |61| Sub-variant (c) deliberately approximate acceptance | Rich lossy-verification literature (§2.2): rank/top-R acceptance, judge heads, relaxed acceptance in TensorRT-LLM/SGLang; BiLD thresholds; taxonomy in arXiv 2607.26627 | Not combined with precision tiers or a storage tier | ANTICIPATED (in other settings) |62| Sub-variant (d) bytes-per-token objective | SpecExec's economics *imply* bytes/token ÷ acceptance length; CATS names flash↔DRAM traffic as the bottleneck | Nobody states or reports **bytes-streamed-per-token vs checkpoint size** as the optimization target and headline result | NOT FOUND |6364### 2.2 Confidence/margin-gated selective verification or escalation6566| Closest prior art | What it does | What it does NOT do | Verdict |67|---|---|---|---|68| **U-HLM** (arXiv 2412.12687, IEEE 2025) — the sharpest hit | On-device SLM drafts; tokens whose on-device uncertainty ≤ threshold **skip uplink transmission and LLM verification entirely** (74.8% of uplinks skipped); rejection-behavior predicted from draft-side uncertainty | Uncertainty via temperature-perturbation consistency, not logit margin; expensive tier is a **remote different model**, cost is radio bytes, not weight-streaming I/O; no rollback of committed tokens; no storage tier | PARTIALLY ANTICIPATED |69| **UGSD** (arXiv 2603.11397, Mar 2026) | Edge draft; **entropy-gated escalation of only high-uncertainty token blocks** to a cloud verifier; top-R rank acceptance (lossy); suffix discarded on correction (rollback); adaptive block length; only 18.2% of tokens escalated | Cloud tier (communication cost), different model as verifier, domain-specific (speech emotion captioning); no weight streaming; no bytes/token framing | PARTIALLY ANTICIPATED |70| **SPRINTER** (arXiv 2502.04557) | Tiny learned verifier predicts whether the target would accept; **target LLM invoked only when a token is deemed unacceptable** (approximate, not exact) | Verifier is a learned classifier, not the draft's own margin; both models resident; no storage tier | PARTIALLY ANTICIPATED |71| **BiLD** (NeurIPS 2023) | Small model decodes; confidence-threshold fallback to large model + distance-threshold rollback of recent tokens | Two separate resident models; no storage tier; no exactness mode; latency objective | ANTICIPATED (gating + rollback pattern) |72| **CALM** (NeurIPS 2022) / cascades (FrugalGPT, RouteLLM) | Margin/confidence-calibrated early exit with distribution-free risk control; request-level routing to expensive models | Depth axis / request granularity; nothing about weight loading | ANTICIPATED (signal + calibration recipe) |73| Token-level margin gating of a **weight-loading (SSD-streaming) decision** | — | Searched: "uncertainty triggered precision escalation", "confidence-gated weight loading", "margin-gated verification bytes", dynamic-precision line (PMPD, HOBBIT, DP-LLM — all feed-forward, no verification), lossy-verification taxonomy (2607.26627 — no memory-constrained verifier) | **NOT FOUND** |7475### 2.3 Layer-by-layer weight streaming from storage during a forward pass (unified memory)7677| Closest prior art | What it does | What it does NOT do | Verdict |78|---|---|---|---|79| **AirLLM** (lyogavin/airllm) / **ZeRO-Inference** | Layer-by-layer load→compute→free; capacity decoupling only | Streams the whole model for **every token**; no draft; no selectivity | ANTICIPATED (streaming primitive) |80| **mlx-flash** (GitHub, matt-k-wong) | Exactly materialize→compute→release per layer **in MLX on Apple Silicon** (StreamingProxy wrapping, predictive I/O scheduler prefetching layer N+1 during layer N); "30B on 16 GB, 70B+ on 32 GB+" | Streams on every token; **no draft/verify role for the streamed model**; no periodicity, no quality tiering | ANTICIPATED (the MLX materialize→compute→re-lazify scheduling itself is not ours) |81| **mlx-moe**, **vMLX**, MLX issue #2878, MLX discussion #615 | SSD expert streaming for MoE in MLX; open feature request for out-of-core streaming (Dec 2025); maintainer position that weights must be materialized/wired for GPU use | Nothing combines a streamed model with a resident draft; #2878 does not mention speculation | supports NOT FOUND for the verifier-role variant |82| **Lever** (arXiv 2605.16786, May 2026) | **DRAM-resident draft, flash-resident target streamed at each verification invocation** on smartphones; "the SSM can remain resident in DRAM … the LLM can be stored in flash and invoked only intermittently"; I/O is 78–93% of verification latency (their Table 1) | Separate small draft model (not same weights); CPU/NPU + UFS, not unified-memory Metal; verification invoked every speculative cycle (no margin-gated skipping); lossless greedy acceptance; latency objective | ANTICIPATED (streamed-verifier-while-resident-draft as a category) |83| **CATS** (arXiv 2605.11186, May 2026) | Self-speculative on edge: shallow sub-network drafts under the DRAM budget; intermediate layers "**streamed from flash once per cycle**" for shallow verification; remaining layers "offloaded from the flash chunk by chunk" for target verify; peak memory = target model alone; 5.08× | Draft axis is **depth** (+ Reduced-KL distillation training), not a training-free precision cast; verification every cycle, not margin-deferred; no bytes/token metric; GPU/edge-Linux, not MLX/Metal | ANTICIPATED (periodic flash-streamed verifier, window-amortized) |84| **llama.cpp discussion #10466** (Nov 2024) | Public proposal: fully-resident speculator + target layers ping-ponged into VRAM from CPU RAM for verification | Never implemented; CPU-RAM/PCIe tier, not SSD; verify-all, exact | corroborates ANTICIPATED (idea circulating) |8586### 2.4 Windowed/batched deferred verification with rollback, where verifier cost is I/O8788| Closest prior art | What it does | What it does NOT do | Verdict |89|---|---|---|---|90| **SpecExec** (NeurIPS 2024) | Huge draft trees verified by a RAM-offloaded 70B in one pass; ~20 accepted tokens per full-model weight sweep; 10.6–18.7× over sequential offloaded decoding — the amortization economics of one weight sweep per window | Separate 7B draft; CUDA/PCIe; exact sampling; never states bytes/token as metric | ANTICIPATED (window-amortized I/O-bound verification) |91| **Lever / CATS** (above) | Explicitly amortize flash reads of the verifier over the accepted window ("the cost of loading and executing the LLM is amortized across several output tokens" — Lever §1) | See §2.3 | ANTICIPATED |92| **SpecOffload** (arXiv 2505.10259) / **SpecMoEOff** (arXiv 2508.21706) / **SpecInfer** (ASPLOS 2024) / **TriForce** (COLM 2024) | Speculation embedded into offloading pipelines (weights or KV) to hide transfer latency; TriForce: resident sparse-KV draft of the same weights, offloaded full KV verify | Throughput/batch regimes; PCIe tiers; verify-all; exact | ANTICIPATED |93| Rollback via KV-cache trim on first flip | Standard speculative-decoding machinery; BiLD rollback policy; UGSD suffix discard | — | ANTICIPATED (not claimable at all) |94| Window-amortized verification whose cost is I/O **and** whose frequency/coverage is margin-gated | — | Lever/CATS/SpecExec verify every window; U-HLM/UGSD gate but have no weight I/O | NOT FOUND |9596### 2.5 The composition as a whole9798Search phrasings tried (plus synonyms): "resident draft streamed verifier", "quantized draft99full model verification SSD", "deferred verification language model rollback", "amortized100weight streaming verification", "two-tier quantization runtime offload verify", "speculative101decoding out-of-core verifier", "verifier offloading speculative decoding", "uncertainty102triggered precision escalation", "self-speculative decoding offloading edge", "flash-backed103target model speculative decoding", plus GitHub sweeps of llama.cpp / ik_llama.cpp / mlx /104mlx-lm issues and community MLX streaming projects.105106| Candidate | Overlap with our composition | Decisive differences |107|---|---|---|108| **CATS** (May 2026) — nearest overall | Resident cheap variant of the model drafts; verifier layers streamed from flash once per window; rollback; single memory-limited device; peak memory = one model | Cheap variant is a **depth truncation requiring distillation training**, not a training-free precision cast; **every** cycle verified (no margin deferral; no τ knob; no approximate mode); metric is wall-clock speedup, not bytes/token; not unified-memory/Metal |109| **Lever** (May 2026) | DRAM-resident draft + flash-streamed target verifier, invocation cost amortized over the accepted window; I/O identified as the dominant verification cost | Separate draft model (no shared weights / no precision pair); verification never skipped by draft confidence; lossless; smartphone NPU/UFS |110| **U-HLM + UGSD** (Dec 2024 / Mar 2026) | Draft-side uncertainty gates whether the expensive tier is invoked at all; UGSD adds block granularity, lossy top-R acceptance and suffix rollback | Expensive tier is a **remote, different** model; the gated cost is communication, not weight-streaming bytes; nothing resides on or streams from storage |111| **QSpec / QuantSpec / ML-SpecQD** | The q4-drafts / q8-verifies-with-shared-weights pair, proven acceptance >90% | Verifier resident; verify-all; arithmetic-latency objective |112113**Composition verdict: NOT FOUND as assembled — but only by the conjunction, and the margin114is thin.** Every load-bearing part exists separately, three of them in systems published115in 2025–2026 that the Phase 1 sweep missed (Lever, CATS, UGSD). Prior-art density around116this exact point is increasing quarter over quarter.117118---119120## 3. Honest bottom line — what survives, what must be attributed121122**Must be attributed (do not claim):**1231241. *Precision-tiered draft/verify on shared weights* — **QSpec, QuantSpec, ML-SpecQD** own it.1252. *Resident draft + storage-streamed verifier, amortized over a token window* — **Lever**126   and **CATS** own the category (with SpecExec owning the amortization economics and127   llama.cpp #10466 as a public 2024 proposal). Our verify-all mode is, at the category128   level, "Lever/CATS with a precision-cast draft on Apple Silicon."1293. *Uncertainty-gated skipping of verification / escalation to an expensive tier* — **U-HLM,130   UGSD, SPRINTER**, with the older lineage **BiLD / CALM / cascades**.1314. *Deliberately lossy acceptance* — the relaxed/lossy-verification literature (top-R rank132   acceptance in UGSD; judge/relaxed acceptance lines; taxonomy in arXiv 2607.26627).1335. *Rollback via KV trim; teacher-forcing a window in one pass* — standard speculative134   decoding machinery.1356. *Layer-by-layer materialize→compute→release streaming in MLX on Apple Silicon* —136   **mlx-flash** (and AirLLM/ZeRO-Inference generically). We did not invent the scheduling137   pattern; we gave the streamed pass a verifier role.138139**Surviving as (narrowly) novel contributions:**140141- **N1 — Margin-gated deferral of verification *bytes*.** Using the resident draft's top-1142  logit margin to decide **whether higher-precision weights are streamed from local storage143  at all** (which positions a sweep must check, and hence how much I/O the quality repair144  costs). Every located gating system gates compute or communication to a *different,145  remote or resident* model; none gates the weight-streaming I/O of the *same* model's146  higher-precision representation. NOT FOUND after targeted search.147- **N2 — Bytes-streamed-per-token as the explicit objective**, reported against checkpoint148  size (9.4× under at W=32), with the τ threshold exposed as a **runtime bytes↔quality149  knob** (47%→83%→~100% of the q4→q8 gap at 150→211→237 MB/token). Dynamic-precision150  systems (PMPD, HOBBIT, DP-LLM) have precision knobs without verification; lossy151  verification has quality knobs without a storage tier; no system prices quality in152  bytes-of-weights-streamed. NOT FOUND as a stated objective/metric.153- **N3 — The training-free same-checkpoint precision pair as the draft/verify split of an154  out-of-core runtime** (q4 resident / q8 streamed, no distillation, no auxiliary model),155  making "quality level that does not fit in RAM" usable on unified memory — demonstrated156  where the alternative (resident q8) is physically impossible (32B on 48 GB), judged157  +0.28 nats over the only configuration that fits. The conjunction is unoccupied; each158  half is not.159- **N4 (engineering, weaker) — The unified-memory instantiation**: strictly sequential160  sweep at SSD device ceiling, persistent incremental verify-KV-cache (O(window) sweeps)161  with trim-rollback, measured on Metal with the prefill/decode numerics ceiling162  identified. Claimable only as a measured system contribution on this platform, not as a163  concept.164165**Claims that are now weaker than when candidate_01 was designed:** "first166resident-draft/streamed-verifier runtime" (false — Lever, CATS), "first selective167verification by draft uncertainty" (false — U-HLM, SPRINTER, UGSD), "first168precision-pair self-speculation" (false — QSpec family), "first out-of-core amortized169verification" (false — SpecExec, Lever, CATS).170171---172173## 4. Recommended claim language for the README (defensible phrasing)174175> **What is new here is the conjunction, and two specific properties of it.** Precision-pair176> self-speculation (QSpec, QuantSpec, ML-SpecQD), storage-streamed verifiers behind a177> resident draft (Lever, CATS), window-amortized verification of an offloaded target178> (SpecExec), and uncertainty-gated escalation (BiLD, CALM, U-HLM, UGSD) all exist179> separately. To our knowledge, no prior system (i) uses the draft's own top-1 logit margin180> to gate **how many bytes of higher-precision weights are streamed from local NVMe** to181> repair quality — making the margin threshold τ a runtime knob that prices output quality182> in bytes-per-token — and (ii) does so with a **training-free precision pair of the same183> checkpoint** (4-bit resident, 8-bit streamed) on unified-memory hardware, so that a184> quality level whose weights cannot reside in RAM remains usable, with bytes streamed per185> token decoupled from checkpoint size (measured 9.4× under the streamed checkpoint at186> W=32). We explicitly do not claim novelty for precision-tiered drafting, streamed187> verification, windowed rollback, or confidence-gated escalation taken individually; the188> attribution table in `research/novelty_check.md` maps each component to its prior art.189190Avoid in all public text: "first", "novel runtime" unqualified, "no one has streamed a191verifier", "quality becomes a runtime variable" without the qualifier *via verification192bytes* (dynamic-precision papers already make quality a runtime variable without193verification).194195---196197## 5. Sources198199All accessed 2026-08-12.200201- QSpec: Speculative Decoding with Complementary Quantization Schemes (EMNLP 2025) — https://arxiv.org/abs/2410.11305 (accessed 2026-08-12)202- QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache (Apple, ICML 2025) — https://machinelearning.apple.com/research/quantspec (accessed 2026-08-12)203- ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts — https://arxiv.org/pdf/2503.13565 (accessed 2026-08-12)204- Lever: Speculative LLM Inference on Smartphones (Wang et al., May 2026) — https://arxiv.org/pdf/2605.16786 (accessed 2026-08-12)205- CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration (Han et al., May 2026) — https://arxiv.org/abs/2605.11186 (accessed 2026-08-12)206- CATS code repository — https://github.com/ElizaFuLan/CATS.git (accessed 2026-08-12, via paper)207- LLMCad: Fast and Scalable On-device Large Language Model Inference — https://arxiv.org/pdf/2309.04255 (accessed 2026-08-12)208- EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding (IEEE TMC) — https://ieeexplore.ieee.org/document/10812936/ (accessed 2026-08-12)209- Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference (EMNLP 2025) — https://arxiv.org/abs/2412.18934 (accessed 2026-08-12)210- SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices — https://arxiv.org/html/2505.10259v2 (accessed 2026-08-12)211- Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding (SpecMoEOff) — https://arxiv.org/abs/2508.21706 (accessed 2026-08-12)212- SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices (NeurIPS 2024) — https://arxiv.org/html/2406.02532v1 (accessed 2026-08-12, re-verified)213- SpecInfer: Accelerating LLM Serving with Tree-based Speculative Inference and Verification (ASPLOS 2024) — https://arxiv.org/abs/2305.09781 (accessed 2026-08-12, re-verified)214- TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding (COLM 2024) — https://arxiv.org/abs/2404.11912 (accessed 2026-08-12)215- MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices (LCTES 2026) — https://doi.org/10.1145/3814943.3816174 (accessed 2026-08-12)216- HiSpec: Hierarchical Speculative Decoding for LLMs — https://arxiv.org/abs/2510.01336 (accessed 2026-08-12)217- SPRINTER: Speeding up Speculative Decoding via Sequential Approximate Verification — https://arxiv.org/abs/2502.04557 (accessed 2026-08-12)218- U-HLM: Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Models — https://arxiv.org/abs/2412.12687 (accessed 2026-08-12)219- Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmission — https://arxiv.org/pdf/2505.11788 (accessed 2026-08-12)220- Edge–Cloud Collaborative Speech Emotion Captioning via Token-Level Speculative Decoding (UGSD, Mar 2026) — https://arxiv.org/html/2603.11397v1 (accessed 2026-08-12)221- Quantize-Sample-and-Verify: LLM Acceleration via Adaptive Edge-Cloud Speculative Decoding — https://arxiv.org/pdf/2507.00605 (accessed 2026-08-12)222- Fast and Cost-effective Speculative Edge-Cloud Decoding with Early Exits — https://arxiv.org/abs/2505.21594 (accessed 2026-08-12)223- DSD: A Distributed Speculative Decoding Solution for Edge-Cloud Agile Large Model Serving — https://arxiv.org/pdf/2511.21669 (accessed 2026-08-12)224- Speculative Decoding with Big Little Decoder (BiLD, NeurIPS 2023) — https://arxiv.org/abs/2302.07863 (accessed 2026-08-12, re-verified)225- Confident Adaptive Language Modeling (CALM, NeurIPS 2022) — https://arxiv.org/abs/2207.07061 (accessed 2026-08-12, re-verified)226- Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes (Jul 2026) — https://arxiv.org/abs/2607.26627 (accessed 2026-08-12)227- Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match (FLy) — https://arxiv.org/pdf/2511.22972 (accessed 2026-08-12)228- A Practical Investigation of Training-free Relaxed Speculative Decoding — https://arxiv.org/html/2607.08690 (accessed 2026-08-12)229- Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptance — https://arxiv.org/html/2507.22424v1 (accessed 2026-08-12)230- When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding — https://arxiv.org/html/2606.30265 (accessed 2026-08-12)231- Statistically-Lossless Quantization of Large Language Models (EAR metric; static, no runtime escalation) — https://arxiv.org/pdf/2605.02404 (accessed 2026-08-12)232- Mixed-Precision Quantization for Language Models: Techniques and Prospects (incl. HOBBIT precision-on-miss loading) — https://arxiv.org/html/2510.16805v1 (accessed 2026-08-12)233- Progressive Mixed-Precision Decoding for Efficient LLM Inference (PMPD) — https://arxiv.org/pdf/2410.13461 (accessed 2026-08-12, re-verified)234- Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware (Jul 2026) — https://arxiv.org/html/2607.17283v1 (accessed 2026-08-12)235- Speculative Speculative Decoding (ICLR 2026) — https://arxiv.org/pdf/2603.03251 (accessed 2026-08-12)236- llama.cpp discussion #10466: Speculative decoding potential for running big LLMs on consumer grade GPUs (resident speculator + layer ping-pong proposal, unimplemented) — https://github.com/ggml-org/llama.cpp/discussions/10466 (accessed 2026-08-12)237- llama.cpp issue #21453: Research: Speculative Decoding for Low-Latency CPU Inference — https://github.com/ggml-org/llama.cpp/issues/21453 (accessed 2026-08-12)238- ik_llama.cpp discussion #839: speculative decoding via network — https://github.com/ikawrakow/ik_llama.cpp/discussions/839 (accessed 2026-08-12)239- mlx-flash: Flash weight streaming for MLX (per-layer StreamingProxy, predictive I/O, every-token streaming, no draft/verify) — https://github.com/matt-k-wong/mlx-flash (accessed 2026-08-12)240- MLX issue #2878: On-disk (out-of-core) streaming of model weights for low-RAM inference (Dec 2025; no speculation proposed) — https://github.com/ml-explore/mlx/issues/2878 (accessed 2026-08-12)241- mlx-moe: router-selected expert streaming from SSD with MLX — https://github.com/mu-hashmi/mlx-moe (accessed 2026-08-12)242- vMLX: SSD expert streaming + disk cache for MoE on Apple Silicon — https://github.com/jjang-ai/vmlx (accessed 2026-08-12)243- dflash-mlx: Lossless DFlash speculative decoding for MLX (adaptive verify block, KV SSD spill; target resident) — https://github.com/bstnxbt/dflash-mlx (accessed 2026-08-12)244- MLX discussion #615: Loading models with mmap (0.025 tok/s overflow result) — https://github.com/ml-explore/mlx/discussions/615 (accessed 2026-08-12, re-verified)245- LLM in a flash: Efficient LLM Inference with Limited Memory (Apple) — https://arxiv.org/abs/2312.11514 (accessed 2026-08-12, re-verified)246- AirLLM (layer-by-layer streaming baseline) — https://github.com/lyogavin/airllm (accessed 2026-08-12)247