SPB Git

spb/localvm-research Public License

Running LLMs larger than memory on a consumer Mac — falsification-driven research: margin-gated deferred refinement, out-of-core verification on Apple Silicon. TR-01 published.

Python 63.2% JavaScript 23.5% CSS 11.8% Shell 0.9% Makefile 0.5%

Phase 3: research gaps — 24 falsifiable approaches (G01–G24)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 6 h ago (Aug 12, 2026) parent 5efe3eb

Showing 2 changed files with +880 and −0

modified research/LOG.md +16 −0
@@ -106,3 +106,19 @@ Format per entry: date/time (local, with timezone) · question · experiment ·
106 106 - **Interpretation:** The gap space is real, narrow, and well-bounded — enough for ≥20
107 107 concrete gap hypotheses without inventing variants of one idea.
108 108 - **Decision:** Launch Phase 3 (research_gaps.md) now; Phase 4 ranking after review.
109 +
110 +---
111 +
112 +## 2026-08-12 09:05 EDT — Phase 3 complete: 24 research gaps
113 +
114 +- **Question:** Which genuinely different, falsifiable approaches exist in the open space?
115 +- **Experiment:** `research/research_gaps.md` — 24 gaps (G01–G24), six mandated items each,
116 + kill-numbers sized to the expH envelope; includes 3 contrarian gaps (G22 WarmEnough,
117 + G23 ConfidentlyWrong, G24 BasinCollapse) that can cheaply kill whole families.
118 +- **Result:** Highest-leverage falsification order derived from cluster analysis:
119 + (1) expG joint margin×agreement matrix → arbitrates G02/G17/G23/half-G24;
120 + (2) expA/B/C SwiGLU trace campaign → feeds G01/G06/G07/G12/G13 + cache simulator;
121 + (3) expD/expE ladder (recovery curves + Metal decode/gather microbench) → decides
122 + G01/G04/G05/G15/G16, rest of G24, gates G03.
123 +- **Interpretation:** The experiment plan now writes itself from gap dependencies.
124 +- **Decision:** Launch Phase 4 ranking; then implement expG first.
added research/research_gaps.md +864 −0
@@ -0,0 +1,864 @@
1 +---
2 +project: localvm-research
3 +document: research/research_gaps
4 +author: Simon-Pierre Boucher
5 +contact: contact@spboucher.ai
6 +created: 2026-08-12
7 +status: draft
8 +---
9 +
10 +# Research Gaps: Candidate Approaches for Over-Budget LLM Execution on Apple Silicon
11 +
12 +Phase 3 deliverable (charter §6). Each gap below is a substantially different candidate
13 +mechanism, generated by crossing the four decoupling axes identified in
14 +`research/state_of_the_art.md` §9 (precision, sparsity, sharing/delta, speculation) with
15 +the four systems levers (residency policy, layout, prefetch, verification), then pruning
16 +every cell already occupied per SOTA §7.1. Every gap cites the specific systems whose
17 +limitation it exploits, and every falsifying experiment is runnable on the target machine
18 +(Apple M5 Max, 48 GB unified memory, AP2048Z NVMe, MLX 0.32 / Metal, no CUDA).
19 +
20 +**Binding substrate numbers that every gap must respect** (expH, this repo, 2026-08-12):
21 +
22 +```text
23 +SSD random read: ~13 GB/s at 1 MiB QD8; 67 MB/s at 4 KiB QD1 (200× cliff)
24 +Access contract: 16 KB-aligned blocks, ≥256 KiB effective reads at QD≥4–8
25 +Per-token budget: ≤~650 MB SSD reads/token for ≥5 tok/s (50% I/O share);
26 + ≤~1.3 GB/token theoretical limit
27 +Overlap: SSD reads under full Metal matmul load lose <5% — prefetch is free
28 +RAM:SSD ratio: ~35:1 best case → non-overlapped residual traffic must stay
29 + ≲1–2% of weight bytes/token, or be amortized across tokens
30 +Decode rule: Metal kernels must be shift/mask-cheap (i-quant LUT penalty)
31 +Agreement priors: ~91% same-top-token at 4-bit; 69–75% acceptance at 25–40% depth;
32 + >90% QuantSpec precision-draft acceptance; 5–15% knife-edge tokens
33 +```
34 +
35 +Gaps are grouped by mechanism class (A–H). Groups do not imply combinations; each gap
36 +stands or falls on its own falsification criterion. Contrarian gaps (§H) are designed to
37 +kill the project's premises cheaply — per charter §17, they are as valuable as the rest.
38 +
39 +---
40 +
41 +## A. Progressive representations and refinement
42 +
43 +### G01 — ResidualPager: progressive precision as a demand-paged memory hierarchy
44 +
45 +1. **What existing systems do.** Any-Precision LLM (ICML 2024), MatQuant (ICLR 2025),
46 + BitStack (ICLR 2025), RRQ, and Drop-by-Drop store nested/progressive weight encodings
47 + whose quality scales with bits consumed; each slice matches dedicated models at that
48 + bitrate. M2Cache tiers neurons across HBM/DRAM/SSD at mixed precision.
49 +2. **What they do not do.** All §5 systems keep **every plane resident** and pick the
50 + operating point statically per deployment; none pages residuals from storage; none
51 + reports bytes-read-per-token (SOTA §5 common ledger). M2Cache never upgrades a fetched
52 + neuron's precision (no refinement) and assumes PCIe/CUDA.
53 +3. **Why it might matter.** A 2–3-bit base of a 70B model is ~17.5–26 GB — resident on a
54 + 48 GB Mac — while full quality lives on NVMe. RAM would bound *base* size, not model
55 + size: exactly the charter §2 decoupling (`resident size ≠ total size ≠ bytes/token`).
56 +4. **Plausible reason it could work.** Successive-refinement theory (Drop-by-Drop's
57 + grounding) guarantees monotone quality-in-bytes; BitDelta shows corrections quantize
58 + to ~1 bit near-losslessly; expH shows the SSD can deliver ~650 MB/token in ≥256 KiB
59 + blocks with free compute overlap; unified memory makes a fetched residual GPU-visible
60 + with zero copy — the mechanic discrete-GPU prior work lacked.
61 +5. **Biggest reason it might fail.** The 35:1 RAM:SSD ratio: if quality recovery needs
62 + residuals for a large, *unstable* fraction of blocks each token, traffic blows the
63 + 1–2% budget and cannot be hidden. BitStack already reports reconstruction overhead
64 + slowing inference even with everything resident.
65 +6. **Smallest falsifying experiment.** **expD + expB.** On a 7–8B model, measure
66 + KL/agreement at 2, 2+1, 2+2… planes with per-block refinement ranked by sensitivity;
67 + then Jaccard(t, t+1) of the refined-block set. **Kill:** reaching KL ≤ 0.05 to FP16
68 + requires touching >20% of residual bytes per token AND temporal reuse of the refined
69 + set is <50% — i.e., steady-state fetch traffic >650 MB/token for a 70B-class model.
70 +
71 +### G02 — MarginGate: decision-uncertainty-gated residual escalation with a correctness story
72 +
73 +1. **What existing systems do.** QSpec/QuantSpec/ML-SpecQD draft with a low-precision
74 + cast of the same weights and verify exactly — but both operand sets stay resident and
75 + every token is verified. BiLD escalates on low confidence (no guarantee, two separate
76 + resident models). CALM gives distribution-free calibrated early exit — for the
77 + **depth** axis only. PMPD/QuickSilver schedule precision per token position,
78 + feed-forward, with no verification.
79 +2. **What they do not do.** Nobody decides *how many residual planes to fetch from
80 + storage* based on whether the token decision is already stable (top-2 margin, entropy,
81 + or a learned error predictor), and nobody attaches either rejection-sampling exactness
82 + or CALM-style calibrated risk to that decision, and **no §5 or §6 paper optimizes or
83 + even reports bytes/token** (SOTA §7.3.2 — the verified open intersection).
84 +3. **Why it might matter.** If ~91% of tokens at 4-bit (and possibly 70–85% at 2–3-bit)
85 + are decided identically to FP16, then escalation — the only expensive event — happens
86 + on 5–15% of tokens, and residual streaming cost is paid only there: bytes/token
87 + collapses to base-read + (escalation rate × residual fetch).
88 +4. **Plausible reason it could work.** The correctness machinery already exists and is
89 + proven: speculative rejection sampling is exact; CALM calibration is distribution-free.
90 + Margins demonstrably carry signal (the entire early-exit line works). EMNLP-2025
91 + residual-magnitude features predict quantization breakage at ρ = 0.82, giving a cheap
92 + learned signal beyond raw margin.
93 +5. **Biggest reason it might fail.** The dangerous quadrant: tokens where the base is
94 + *confidently wrong* (high margin, wrong token). If disagreements are not concentrated
95 + at low margin, gating misses exactly the errors that matter — and the 4-bit KLD tail
96 + (max 22–26 nats) plus the "flips are load-bearing tokens" evidence says this quadrant
97 + is not empty. Also: escalation may exceed 40% on reasoning/code (Sirius's finding that
98 + sparsity specifically damages reasoning is a warning for any cheap-first scheme).
99 +6. **Smallest falsifying experiment.** **expG.** On 1B→8B models, 2-bit/3-bit/4-bit base
100 + vs FP16 reference, log per token: (base top-2 margin, base entropy, agreement with
101 + reference, reference token rank under base). Domains: code, math, chat, FR, EN,
102 + reasoning. **Kill:** AUROC of margin (or margin+learned features) for predicting
103 + disagreement <0.8, or required escalation rate at 99% effective agreement >40% on any
104 + major domain. (This one measurement also arbitrates G17, G23, G24.)
105 +
106 +### G03 — DeltaKernel: incremental refinement ΔY = (ΔW)·X instead of recompute
107 +
108 +1. **What existing systems do.** Every verification/refinement scheme in the literature
109 + (LayerSkip, QSpec, QuantSpec, Sirius dense correction) **recomputes the full layer**
110 + with the better operand. GMRES-IR (Carson–Higham) refines a cheap factorization with
111 + residual-driven corrections and convergence guarantees — in numerical linear algebra,
112 + never written down for transformer inference (SOTA §6.6, §7.3.3).
113 +2. **What they do not do.** No published primitive computes only the *correction term*
114 + `ΔY = (ΔW)X` against cached activations, where ΔW is the streamed residual plane
115 + (sparse, low-rank, or bitplane), and adds it to the cached base output.
116 +3. **Why it might matter.** Refinement cost becomes proportional to residual size, not
117 + layer size: a 10%-density residual costs ~10% of the base matmul in both bytes and
118 + FLOPs. Without this, every escalation (G02) or block verification (G17) doubles
119 + compute, halving the whole family's economics.
120 +4. **Plausible reason it could work.** Linearity: for linear layers Y = WX,
121 + Y_full = Y_base + (ΔW)X exactly. Nonlinearities (softmax, SwiGLU, norms) only need
122 + recomputation *downstream* of corrected layers, and expF maps which layers tolerate
123 + deferred correction. Activation caching for a deferred block costs
124 + ~hidden_dim × layers × block_len × 2 bytes ≈ tens of MB — trivial next to weights.
125 +5. **Biggest reason it might fail.** Attention breaks the linear shortcut (Q/K/V
126 + corrections propagate through softmax non-linearly), so exactness holds per-linear-op,
127 + not per-layer; and a sparse/irregular ΔW matvec on Metal may be gather-bound and
128 + slower than a dense recompute (the Endor/i-quant lesson transposed to residuals).
129 +6. **Smallest falsifying experiment.** **New micro (expE extension, expE_delta).**
130 + Implement in MLX/Metal: (a) dense base matvec, (b) fused ΔY with 5/10/25%-density
131 + block-sparse ΔW, (c) full recompute at base+residual precision. **Kill:** the ΔY
132 + kernel at 10% density is not ≥2× faster than full recompute, or numerical error of
133 + base+ΔY vs direct full compute exceeds quantization noise (>1e-3 relative on layer
134 + outputs).
135 +
136 +### G04 — RDLayout: rate-distortion-optimal embedded weight bitstreams, compiled to the SSD contract
137 +
138 +1. **What existing systems do.** EZW/SPIHT/EBCOT (JPEG2000) achieve per-block
139 + R-D-optimal truncation of significance-ordered bitstreams — 30-year-old, mature.
140 + BitStack unwittingly rediscovers embedded coding with SVD residual blocks but sorts
141 + by a global significance heuristic. GGUF i-quants use imatrix calibration to set
142 + *static per-tensor* precision. LLM-in-a-flash and Ripple optimize flash layout for
143 + co-activation, not for truncation.
144 +2. **What they do not do.** Nobody compiles weights into per-block embedded bitstreams
145 + with truncation points optimized against **calibration-set logit KL** (not MSE),
146 + aligned to the measured storage contract (16 KB pages, ≥256 KiB fetch bundles), so
147 + that "bytes loaded per matrix" is a continuous, precomputed quality knob
148 + (SOTA §5.2, notes/decomposition §4).
149 +3. **Why it might matter.** It converts layer sensitivity (expF) into *layout*, at
150 + compile time, with no runtime policy needed to get the first win: a static-but-
151 + nonuniform operating point strictly better than uniform bpw at the same bytes, and
152 + the substrate every dynamic gap (G01, G02, G17) would page from.
153 +4. **Plausible reason it could work.** Sensitivity-weighted allocation already beats
154 + uniform (SqueezeLLM, imatrix, LASER's "some matrices improve when truncated");
155 + EBCOT's Lagrangian truncation is exactly the right optimizer once distortion = KL;
156 + the charter's compile stage (§14) explicitly budgets unlimited offline time.
157 +5. **Biggest reason it might fail.** Diminishing returns: imatrix/k-quant mixing may
158 + already capture most of the nonuniform-allocation gain, leaving <0.01 KL at matched
159 + bytes — real but not research. Also per-block KL attribution is approximate (block
160 + interactions), so the "optimal" truncation may be optimal only for the calibration
161 + distribution.
162 +6. **Smallest falsifying experiment.** **expD + expF.** On an 8B model, compare at
163 + matched total bytes: uniform 3-bit; imatrix-style per-tensor mixing; per-block
164 + KL-optimized truncation of a 2-bit+residual embedded code. **Kill:** per-block R-D
165 + allocation improves KL-to-FP16 by <15% relative over per-tensor imatrix mixing at
166 + the same byte budget.
167 +
168 +### G05 — MetalTrellis: LUT-free sub-3-bit decode kernels for Apple GPUs
169 +
170 +1. **What existing systems do.** The 2–2.5 bpw frontier (QuIP#, AQLM, QTIP, EXL3,
171 + Any-Precision bitplane engines) is entirely CUDA; QTIP reaches >80% of peak GPU
172 + bandwidth using a bitshift trellis (~2 instructions/weight). llama.cpp's i-quants
173 + prove LUT-heavy decode *loses* on Apple GPUs (53.9 vs 63.1 tok/s despite half the
174 + bytes).
175 +2. **What they do not do.** Zero Metal implementations exist of bitplane matvec,
176 + MSB-sliced decode, or trellis decode (SOTA §7.3.5). Nobody knows whether the i-quant
177 + penalty applies to shift/mask-only progressive layouts on M-series GPUs.
178 +3. **Why it might matter.** Every progressive gap (G01–G04) presumes the base and its
179 + residual planes decode at near-dense speed on Metal. If only affine group-quant
180 + decodes fast, the design space collapses to MLX's existing formats and residuals must
181 + be affine-quantized deltas — a materially different (and weaker) encoding.
182 +4. **Plausible reason it could work.** Bitplane and bitshift-trellis decode are
183 + shift/mask-cheap by construction (satisfying the §1 decode rule); decode-bound
184 + kernels benefit from Apple's high ALU:BW ratio at batch 1 (dequant is free when
185 + memory-bound); MLX exposes custom-kernel plug points (`mode`).
186 +5. **Biggest reason it might fail.** Bitplane matvec needs bit-transpose gathers with
187 + poor memory coalescing on Apple's GPU cache hierarchy; trellis decode is sequential
188 + within a block, fighting SIMD-group width. The failure would be quantitative and
189 + final: the format tax eats the byte savings.
190 +6. **Smallest falsifying experiment.** **expE variant (expE_decode).** Implement three
191 + Metal matvecs at matched effective bpw (~2.5): MLX affine 2-bit+scales baseline;
192 + 2-of-8 bitplane; small bitshift trellis. Measure tok/s-equivalent throughput on
193 + 4096×4096. **Kill:** neither progressive variant reaches ≥70% of the affine
194 + baseline's throughput.
195 +
196 +---
197 +
198 +## B. Sparsity as a paging policy (SwiGLU, post-training)
199 +
200 +### G06 — TealPager: thresholded-SwiGLU sparsity as bytes/token reduction on Metal
201 +
202 +1. **What existing systems do.** TEAL achieves 40–50% model-wide training-free
203 + activation sparsity on stock SwiGLU models (1.53–1.8× decode, Triton/CUDA); CATS 50%
204 + FFN-only; GRIFFIN selects 50% of FF params once per sequence; DIP picks masks given
205 + cache contents. DejaVu/LLM-in-a-flash proved sparsity-driven paging — for ReLU
206 + models only (0.2 GB/token vs 13.4 GB).
207 +2. **What they do not do.** No Metal kernel exists for any of them (SOTA §3.3); nobody
208 + has published **bytes/token for thresholded-SwiGLU paging**; all working-set
209 + statistics (Jaccard, reuse distance) in print are ReLU-era.
210 +3. **Why it might matter.** 40–50% is only ~2× — but it is the *only* training-free
211 + selectivity mechanism for dense stock models (Llama-3, Qwen3-dense), and 2× on the
212 + FFN dominates once attention/KV are handled; combined with a 20–30% RAM deficit it
213 + is the difference between thrashing and fitting.
214 +4. **Plausible reason it could work.** TEAL's thresholds are calibration-only; GRIFFIN's
215 + per-sequence "flocking" granularity is inherently prefetch-friendly (choose once,
216 + fetch 256 KiB bundles, reuse all sequence); expH says the fetch side is comfortable
217 + if masks are stable.
218 +5. **Biggest reason it might fail.** Two independent cliffs: (a) Sirius showed
219 + contextual sparsity specifically damages reasoning/GSM8K — the 99%-retention claims
220 + are perplexity mirages; (b) a 50%-sparse gather matvec on Metal may run *slower*
221 + than dense (no sparse tensor cores; Endor's indexing-overhead point), saving bytes
222 + on paper and losing tokens/s in practice.
223 +6. **Smallest falsifying experiment.** **expA + expB (SwiGLU masks), then expE.**
224 + Measure on Llama-3-8B: model-wide safe sparsity at ≤0.05 KL and GSM8K delta ≤1 pt;
225 + mask Jaccard(t, t+1) and per-sequence union growth. **Kill:** safe sparsity <30%
226 + model-wide, or Jaccard(t, t+1) <0.6 (mask churn makes paging traffic exceed the
227 + dense-read baseline), or (expE) the Metal gather kernel at 50% sparsity is slower
228 + than the dense matvec.
229 +
230 +### G07 — WandaTiers: importance scores as residency policy, with dense correction
231 +
232 +1. **What existing systems do.** Wanda/SparseGPT compute cheap importance scores
233 + (|W|·‖x‖, Hessian) and **delete** weights — costing 27% PPL at 7B for 50%
234 + unstructured. Endor packs pruned weights into a dense-readable bitmap format
235 + (CUDA). Sirius restores reasoning quality lost to sparsity by dense-verifying ~11%
236 + of tokens at 78% of the efficiency gain. M2Cache ranks neurons statically for tier
237 + placement.
238 +2. **What they do not do.** Nobody uses one-shot importance scores as a **reversible
239 + tier assignment** — top-p% resident, bottom demoted to NVMe in a dense-readable
240 + format and re-materialized on demand — with a Sirius-style correction loop restoring
241 + the dense ceiling (SOTA §7.3.6). Pruning answers "total size"; this answers
242 + "residency".
243 +3. **Why it might matter.** It is the cheapest possible route to a working
244 + over-budget dense runtime: no re-encoding, no predictors, no new format — just
245 + placement. If cold-weight touch rates are low, it alone runs a 1.3–1.5× over-RAM
246 + model interactively.
247 +4. **Plausible reason it could work.** The demoted weights are, by construction, the
248 + ones whose absence changes outputs least (that's what the scores measure); misses
249 + can be served by zero (pruned behavior) immediately and corrected asynchronously;
250 + Sirius quantifies the correction cost at ~11% of tokens.
251 +5. **Biggest reason it might fail.** Wanda-style scores are *global averages*; the
252 + tokens that need demoted weights may be exactly the load-bearing ones ("Accuracy is
253 + Not All You Need": 25%+ flips for pruning-type interventions at matched accuracy).
254 + If per-token demand for cold weights is diffuse rather than rare, traffic is random,
255 + small, and constant — the 67 MB/s regime.
256 +6. **Smallest falsifying experiment.** **expA (blocked variant).** Demote the
257 + bottom-{20,35,50}% of FFN blocks by Wanda score; per token, measure (a)
258 + flip rate / KL vs dense with misses-as-zero, (b) how many demoted blocks would
259 + actually be needed to stay within KL ≤ 0.05. **Kill:** at 35% demotion, cold-block
260 + demand exceeds 5% of demoted bytes per token (≈ diffuse demand), or flip rate with
261 + Sirius-style 11% dense correction still exceeds 5%.
262 +
263 +---
264 +
265 +## C. Cache, replacement, and residency policy
266 +
267 +### G08 — MRURing: DBMIN-style per-tensor-class buffer pools with an MRU ring for cyclic scans
268 +
269 +1. **What existing systems do.** llama.cpp mmap and the MLX mmap prototype delegate
270 + replacement to the kernel's approximate LRU; all published expert caches
271 + (Eliseev–Mazur, PowerInfer-2) use LRU/LFU variants. Databases solved this in 1985:
272 + DBMIN assigns a per-file policy from the known access pattern, and for looping
273 + sequential scans **MRU is optimal while LRU is the worst case**.
274 +2. **What they do not do.** No LLM runtime applies MRU to the dense-decode weight scan
275 + (which is a perfect cyclic scan, same order every token), or partitions the budget
276 + into per-tensor-class pools (pin embeddings; MRU ring for dense; ARC for experts;
277 + sliding window for KV) — SOTA §4.7 OS imports, apparently unpublished anywhere.
278 +3. **Why it might matter.** For a dense model overflowing RAM by X GB, MRU keeps a
279 + stable (model−X) resident set and re-reads exactly X GB/token — the theoretical
280 + floor — where LRU re-reads *everything* (the measured 0.025 tok/s MLX failure).
281 + At X = 3 GB, the floor is ~0.23 s/token of I/O on our SSD: a 1.1× over-budget model
282 + becomes usable by policy alone.
283 +4. **Plausible reason it could work.** The access pattern is fully known (static layer
284 + order — the "query plan"); the math is 40 years old; expH shows the re-read stream
285 + (large sequential slices at QD≥4) hits ~13 GB/s and overlaps freely with compute.
286 +5. **Biggest reason it might fail.** The window of usefulness is narrow: the floor
287 + itself caps overflow at ~650 MB (5 tok/s) to ~6.5 GB (0.5 tok/s) — MRU rescues
288 + *slightly* over-budget models only, and macOS gives no direct control over UBC
289 + eviction, so the ring must be implemented with explicitly managed wired buffers
290 + under the iogpu wired limit (engineering risk: the panic cliff).
291 +6. **Smallest falsifying experiment.** **New micro (expI_cachesim) + expH numbers.**
292 + Trace-driven simulation (traces are free: layer order is static) of LRU vs MRU vs
293 + OPT at 5–30% overflow, then a minimal MLX prototype streaming the overflow slice.
294 + **Kill:** measured end-to-end gain of the MRU prototype over kernel-LRU mmap <1.5×
295 + at 10–20% overflow, or wired-arena management overhead eats the gain.
296 +
297 +### G09 — ArcExperts: scan-resistant expert cache + explicit slice reads for MoE on macOS
298 +
299 +1. **What existing systems do.** llama.cpp runs MoE overflow by mmap demand paging
300 + (kernel LRU); discussion #18758 measured that layout-aware explicit slice reads cut
301 + cold-decode reads/token 1418→370 (2.23×, +13–14% end-to-end); a two-tier expert
302 + cache is an open, unfilled feature request (#20757). ARC/2Q/LIRS ghost-list caches
303 + are standard in storage systems; nobody has published one for experts (SOTA §4.7).
304 +2. **What they do not do.** No shipping runtime combines: compiled 16 KB-aligned expert
305 + bundles, an ARC cache with ghost lists auto-partitioning recency vs frequency,
306 + QD≥8 explicit reads, and purgeable-warm demotion — on any OS, least of all macOS.
307 +3. **Why it might matter.** MoE is where selectivity is free (the router already picks
308 + ~2/8–8/128 experts); the entire gap between llama.cpp's current behavior and the
309 + achievable floor is cache/IO policy. Qwen3-30B-A3B/80B-class models on 16–48 GB Macs
310 + are the most immediately deployable win of the whole project.
311 +4. **Plausible reason it could work.** Every ingredient has isolated evidence
312 + (2.23× from layout alone; MoE temporal locality confirmed; ARC's superiority on
313 + mixed workloads is textbook); expert granularity (MBs) naturally satisfies the
314 + ≥256 KiB fetch contract.
315 +5. **Biggest reason it might fail.** Modern MoE routing may be so uniform (load-balanced
316 + by training) that no cache policy beats LRU by more than a few points — the gain
317 + would then be layout+QD only, which llama.cpp could (and may) merge upstream first,
318 + making this engineering, not research.
319 +6. **Smallest falsifying experiment.** **expB (MoE-trace variant).** Log real routing
320 + traces (llama.cpp/mlx-lm on Qwen3-30B-A3B) across domains; replay through LRU, MRU,
321 + ARC, LIRS, OPT simulators at cache = 25/50/75% of expert pool. **Kill:** ARC beats
322 + LRU by <5 points of hit rate at every cache size (policy headroom ≈ 0; only layout
323 + matters).
324 +
325 +### G10 — PurgeableWarm: kernel-cooperative weight caching via purgeable MTLHeaps
326 +
327 +1. **What existing systems do.** Inference runtimes either wire weights (mlx wired
328 + limit; panic at the cliff, mlx-lm #883) or leave them to the page cache (opaque
329 + eviction). macOS uniquely exposes purgeable/volatile memory with reclaim
330 + notification (`setPurgeableState`), used by graphics apps since iOS, never by an
331 + LLM runtime (SOTA §4.7 import #9).
332 +2. **What they do not do.** No runtime maintains a *warm* weight tier the OS may
333 + reclaim under pressure — converting "jetsam/kernel-panic risk" into "graceful
334 + quality/latency degradation," and no runtime distinguishes wired-hot /
335 + purgeable-warm / SSD-cold as three managed tiers.
336 +3. **Why it might matter.** Every other gap that caches anything (G01, G06–G09, G13)
337 + needs a warm tier that coexists with the user's other apps on a 48 GB machine;
338 + without cooperative purgeability, the practical resident budget is far below
339 + nominal RAM and failure is catastrophic rather than gradual.
340 +4. **Plausible reason it could work.** The API exists and ships; reclaim is
341 + block-granular (per-resource/heap); a purged block is exactly a cache miss, which
342 + the G08/G09/G11 machinery already handles; re-fetch at 13 GB/s makes wrong
343 + purges cheap.
344 +5. **Biggest reason it might fail.** Per-use volatile→nonvolatile state transitions on
345 + the render path may cost too much (an API round-trip per block per token), and the
346 + OS's purge order is not model-aware — it may purge the blocks we need most, making
347 + quality under pressure worse than a self-managed fixed budget.
348 +6. **Smallest falsifying experiment.** **New micro (expJ_purgeable).** Fill a purgeable
349 + MTLHeap with weight blocks; run a matvec loop touching them; apply memory pressure
350 + (`memory_pressure -l critical`); measure transition overhead per access, purge
351 + granularity, and recovery latency. **Kill:** state-transition overhead >10 µs/block
352 + per token, or purges are so coarse/unpredictable that post-pressure recovery costs
353 + more than a fixed self-managed budget with SSD misses.
354 +
355 +### G11 — NeverBlock: anti-caching semantics — compute never stalls on a weight miss
356 +
357 +1. **What existing systems do.** All paging runtimes (llama.cpp mmap, LLM-in-a-flash,
358 + PowerInfer-2) **block** on a miss (fault or explicit read stall); MoBiLE substitutes
359 + a smaller expert on miss (MoE-only, with training); anti-caching databases
360 + (H-Store, VLDB 2013) abort-and-restart transactions so no thread ever waits on disk.
361 +2. **What they do not do.** No LLM runtime has the invariant "a miss is served
362 + immediately by the resident approximation (base-precision block, zero, or substitute
363 + expert), the true bytes are fetched asynchronously, and the computation is optionally
364 + repaired later" — as an architectural rule rather than an ad-hoc trick (SOTA §4.7
365 + import #5).
366 +3. **Why it might matter.** Tail latency: expH's averages hide P99 fetch latency
367 + (PowerInfer-2 reported +40.9% P99); a never-block runtime converts I/O jitter into
368 + bounded, measurable quality jitter — which G02's verification machinery can then
369 + repair. It decouples token cadence from storage latency entirely.
370 +4. **Plausible reason it could work.** With a G01-style representation, a "miss" has a
371 + natural graceful value: the resident low-bit base — an approximation with *known,
372 + calibrated* error (expF), not garbage. The repair path is G03's ΔY kernel applied
373 + one-to-a-few tokens late.
374 +5. **Biggest reason it might fail.** Error compounding across layers (QEP:
375 + near-exponential with depth): a miss at layer 10 contaminates all downstream layers
376 + for that token, so "repair later" may require replaying the whole suffix — at which
377 + point never-block is just deferred blocking with extra bookkeeping.
378 +6. **Smallest falsifying experiment.** **expF.** Inject base-precision substitution at
379 + single layers and at realistic miss patterns (Poisson over layers at 5/10/20% miss
380 + rate) on an 8B model; measure flip rate and KL with and without one-token-deferred
381 + ΔY repair. **Kill:** at a 10% miss rate, deferred repair fails to keep flips <2%
382 + (vs the blocking baseline's 0%) — i.e., misses must be repaired synchronously
383 + anyway.
384 +
385 +---
386 +
387 +## D. Prefetch and prediction
388 +
389 +### G12 — HiddenPrefetch: learned hidden-state → future-weight-block prediction
390 +
391 +1. **What existing systems do.** Eliseev–Mazur apply layer k+1's router gate to layer
392 + k's hidden state (a static, 1-layer-lookahead heuristic) → 2–3 tok/s Mixtral on
393 + 11–16 GB; DejaVu/ShadowLLM exploit "slowly changing hidden states" to predict
394 + sparsity for FLOP-skipping (ShadowLLM: one early-layer predictor shadows all
395 + layers); Pythia (MICRO 2021) shows online-RL hardware prefetchers that optimize
396 + hit-rate-minus-wasted-bandwidth.
397 +2. **What they do not do.** Nobody trains a small predictor from hidden-state features
398 + to **SSD prefetch decisions N layers ahead** (N sized so lead time ≥ fetch latency),
399 + for dense-sparse blocks or residual planes rather than MoE experts, with wasted-bytes
400 + penalized (SOTA §3.1 extension; §4.7 import #6).
401 +3. **Why it might matter.** The paging gaps (G01, G06, G07) live or die on hiding
402 + fetch latency; expH says overlap is free *if* the runtime knows what to fetch
403 + ~1–5 ms early. Hidden states are richer context than any hardware prefetcher has
404 + ever had.
405 +4. **Plausible reason it could work.** ShadowLLM proved one early probe predicts
406 + all-layer needs with >15% accuracy gain over per-layer criteria; predictors cost
407 + <2.4% overhead in LLM-in-a-flash; a logistic probe per layer is microseconds on
408 + AMX.
409 +5. **Biggest reason it might fail.** Prediction quality may collapse exactly where it
410 + matters: rare blocks (the ones not already cached) are by definition the tail of the
411 + distribution, and PowerInfer-2's predictors cost 2.6 GB of DRAM at 47B — the
412 + apparatus can eat the budget it saves.
413 +6. **Smallest falsifying experiment.** **expB + a probe.** From expB traces, train a
414 + logistic probe: hidden state at layer k → block set needed at layer k+4. **Kill:**
415 + recall <70% at ≤2× overfetch for the *non-resident* block subset (measured against
416 + a realistic cache), or probe+metadata footprint >5% of the RAM budget.
417 +
418 +### G13 — DomainCache: temporary model specialization via semantic working sets
419 +
420 +1. **What existing systems do.** PowerInfer places hot neurons statically from global
421 + power-law statistics; GRIFFIN picks FF blocks per *sequence*; prompt/prefix caches
422 + specialize KV, never weights. No system precomputes or persists **per-domain weight
423 + working sets** ("coding session" vs "French chat") and swaps them at session
424 + granularity.
425 +2. **What they do not do.** Nobody has even measured whether same-domain prompts reuse
426 + the same weight regions substantially more than cross-domain prompts on modern
427 + SwiGLU models (all locality numbers are ReLU-era) — charter expC exists precisely
428 + because this is unmeasured.
429 +3. **Why it might matter.** Session granularity changes the economics completely:
430 + fetches amortize over thousands of tokens, the 35:1 ratio becomes irrelevant, and
431 + the mechanism composes with everything (warm the G08/G09 caches from a domain
432 + profile at session start — seconds of sequential 13 GB/s reads).
433 +4. **Plausible reason it could work.** MoE routers demonstrably develop
434 + domain-correlated expert usage; LoRA's success proves task-relevant weight subspaces
435 + are small; if activation masks have any cross-prompt structure, a domain profile
436 + captures it offline for free.
437 +5. **Biggest reason it might fail.** Transformer weight usage may be dominated by
438 + token-level (syntactic) rather than domain-level (semantic) factors — every prompt
439 + touching the same high-frequency circuitry plus a thin idiosyncratic tail — giving
440 + within-domain overlap barely above cross-domain overlap, i.e., nothing to cache at
441 + session granularity.
442 +6. **Smallest falsifying experiment.** **expC.** Important-block sets (expA criterion)
443 + for 50 prompts × 6 domains on an 8B model; compute within-domain vs cross-domain
444 + Jaccard of the union working sets. **Kill:** mean within-domain minus cross-domain
445 + Jaccard <0.10 absolute, or the domain-specific tail is <10% of the working set
446 + (everything important is universal → a single static hot set suffices, use G07).
447 +
448 +---
449 +
450 +## E. Decomposition and sharing as memory-hierarchy assignments
451 +
452 +### G14 — TrunkDelta: shared cross-layer trunk resident, per-layer deltas paged
453 +
454 +1. **What existing systems do.** Basis Sharing, DeltaLLM, and Relaxed Recursive
455 + Transformers show layers can be re-expressed as shared components + per-layer
456 + corrections (12–25% compression, light calibration); BitDelta shows finetune deltas
457 + quantize to ~1 bit near-losslessly, and iterating it yields a monotone stack of
458 + 1-bit masks. All are framed purely as *compression ratios*.
459 +2. **What they do not do.** Nobody assigns the decomposition to the memory hierarchy:
460 + shared trunk (touched by every layer, maximal reuse) wired in RAM; per-layer 1–2-bit
461 + delta masks streamed sequentially from SSD in layer order (SOTA §2.10 extension,
462 + §7.3 open intersection).
463 +3. **Why it might matter.** The trunk is read L times per token but stored once — its
464 + effective bytes/token *amortize across layers*, a reuse structure no flat encoding
465 + has. Delta streaming is perfectly sequential (layer order is static): the one access
466 + pattern our SSD serves at full 13 GB/s with trivial prefetch.
467 +4. **Plausible reason it could work.** BitDelta's ~1 bit/param deltas are real,
468 + measured structure; "Secretly Linear" reports ~0.99 Procrustes linearity between
469 + consecutive layers (adjacent layers nearly share a basis); sequential delta prefetch
470 + needs zero prediction machinery.
471 +5. **Biggest reason it might fail.** All published sharing needs light retraining
472 + (DeltaLLM: 30–40M tokens; RRT: distillation); zero-shot/calibration-only trunk
473 + extraction may degrade so much that deltas must carry most information — then delta
474 + bytes/layer ≈ layer bytes and nothing is saved. BitDelta's evidence is for
475 + *finetune-vs-base* pairs, not *layer-vs-layer* within one model — the transfer is
476 + unproven.
477 +6. **Smallest falsifying experiment.** **New micro (expK_trunk, expD-adjacent).** On an
478 + 8B model, fit a shared basis across layer groups by calibration-only alternating
479 + least squares (charter permits calibration passes); measure PPL/KL with trunk-only
480 + vs trunk+quantized-delta at delta budgets {0.5, 1, 2} bits/param. **Kill:**
481 + at trunk = 30% of total bytes, KL ≤ 0.1 requires deltas ≥3 bits/param
482 + (delta traffic ≈ flat-model traffic; no amortization win).
483 +
484 +### G15 — SpectralSplit: low-rank hot path resident + quantized sparse residual cold path
485 +
486 +1. **What existing systems do.** SVD-LLM/SliceGPT keep the low-rank part and **discard
487 + the residual** (quality collapses past ~25–40% compression because the residual
488 + carries most spectral energy); CALDERA encodes `W ≈ Q + LR` jointly as a *static*
489 + 2–2.5 bpw format (CUDA, both terms always read); LASER shows selected late-MLP
490 + matrices *improve* when their high-order components are dropped.
491 +2. **What they do not do.** Nobody stores the SVD residual at all (SVD family) or
492 + treats the two terms as different *tiers* (CALDERA): low-rank factors resident
493 + (dense small GEMMs — the one decomposition Metal already runs fast), residual on
494 + SSD, fetched per-block only where LASER-style analysis says the residual matters
495 + (SOTA §2.8 extension).
496 +3. **Why it might matter.** Unlike bitplanes (G01), both components use *existing,
497 + fast* MLX kernels today — no custom Metal work gates a prototype; and LASER implies
498 + the residual tier can be skipped entirely for some matrices (negative bytes).
499 +4. **Plausible reason it could work.** Rank-r hot path costs r(m+n)/mn of dense bytes
500 + (~12% at r=256 for 4096²); per-matrix residual need is measurable offline; the
501 + split is exact by construction (base + residual = W), so quality is a pure function
502 + of which residual blocks are fetched — a clean knob for G02's gating.
503 +5. **Biggest reason it might fail.** Weights are *not* globally low-rank (SOTA §2.8's
504 + headline): the low-rank base alone may be so poor (unlike a 2-bit quantized base,
505 + which is uniformly mediocre) that residual demand is near-universal per token —
506 + worse-behaved than G01's precision split, with double the kernel launches.
507 +6. **Smallest falsifying experiment.** **expD (SVD variant).** On an 8B model, base =
508 + rank-{128,256} per matrix + 3-bit quantized residual in importance-ranked blocks;
509 + measure KL vs fraction of residual blocks applied, per matrix class. **Kill:**
510 + base-alone KL >1.0 nats *and* reaching KL ≤ 0.05 needs >50% of residual bytes for
511 + most matrices — i.e., strictly dominated by the G01 bitplane split at equal bytes.
512 +
513 +### G16 — SeedSynth: regenerate weights from seeds — trade bandwidth for idle ALUs
514 +
515 +1. **What existing systems do.** SeedLM (Apple) replaces weight blocks with LFSR seeds
516 + + coefficients, regenerating pseudo-random bases at inference: 4-bit parity,
517 + data-free, ~4× speedup on an FPGA at 70B — explicitly trading memory bandwidth for
518 + free compute. No Metal port exists; DFloat11 shows the *wrong* version of this trade
519 + (entropy decode on the critical path → 2–3× slower).
520 +2. **What they do not do.** Nobody has tested generative reconstruction on Apple GPUs,
521 + where batch-1 decode is bandwidth-bound and 40 GPU cores sit partially idle —
522 + precisely the regime the trade targets; and nobody composes it with paging (seeds
523 + are ~100× smaller than blocks: a "prefetch" of seeds is nearly free).
524 +3. **Why it might matter.** If it works, it attacks bytes/token *below* the
525 + information-theoretic floor of stored-bits formats: resident bytes and streamed
526 + bytes both shrink to seeds+coefficients, and SSD traffic per block drops ~10×,
527 + trivially satisfying every budget in this document.
528 +4. **Plausible reason it could work.** LFSR generation is shift/XOR-only (passes the
529 + §1 decode rule, unlike DFloat11's Huffman); Apple authored SeedLM with edge
530 + inference as the stated motivation; the ALU:BW imbalance at batch 1 is real and
531 + measured.
532 +5. **Biggest reason it might fail.** Regeneration cost per weight might exceed the
533 + ~2 instructions/weight ceiling that QTIP shows is needed to stay bandwidth-limited;
534 + quality is 4-bit-parity at best (no path to 16-bit exactness — it cannot serve as
535 + a *verification* tier, only as a base), and per-block coefficient search at compile
536 + time is expensive for 70B-class models.
537 +6. **Smallest falsifying experiment.** **expE variant (expE_seed).** Metal kernel:
538 + LFSR-basis generation fused into matvec vs MLX affine 4-bit matvec at matched
539 + quality (SeedLM's published 4-bit parity settings on one 8B layer). **Kill:**
540 + fused generate+matvec throughput <80% of the 4-bit dense baseline (the compute
541 + trade loses on this GPU), or per-layer PPL parity claims fail to reproduce on a
542 + SwiGLU model.
543 +
544 +---
545 +
546 +## F. Scheduling, speculation, and amortization
547 +
548 +### G17 — BlockVerify: token-batch amortization of residual fetches (deferred verification)
549 +
550 +1. **What existing systems do.** LayerSkip/Draft&Verify verify a drafted token block
551 + with one full-model pass (depth axis, all weights resident and read every
552 + verification); SpecExec shows ~20 accepted tokens per full offloaded-weight sweep
553 + turns 4.5 s/token into 4–6 tok/s; Kangaroo stops drafting on low confidence;
554 + QuantSpec verifies with resident 16-bit weights every block.
555 +2. **What they do not do.** Nobody drafts N tokens with a *resident low-bit base* and
556 + then performs **one residual-augmented verification sweep** whose streamed residual
557 + bytes amortize over the accepted block — i.e., SpecExec's economics applied to
558 + residual planes instead of full weights, with rollback on rejection (SOTA §8 open:
559 + "block-deferred verification policy").
560 +3. **Why it might matter.** It divides residual bytes/token by the acceptance length:
561 + if base-vs-refined agreement resembles QuantSpec's (>90%), expected block length
562 + ≥7–10 and the 1–2% traffic ceiling relaxes to 10–20% of weight bytes per sweep —
563 + a qualitatively easier target than G02's per-token gating.
564 +4. **Plausible reason it could work.** The rejection-sampling correctness proof carries
565 + over unchanged (draft distribution = base model, target = refined model, same
566 + support); acceptance priors are measured (91% at 4-bit); verification sweeps read
567 + residuals *sequentially by layer* — the SSD-friendly pattern; MLX already ships
568 + speculative verification kernels on Metal (ReDrafter).
569 +5. **Biggest reason it might fail.** Acceptance at a 2–3-bit base (required for real
570 + oversubscription) is unmeasured and the ParetoQ transition suggests it may crater;
571 + and rollback economics: one early rejection wastes the whole drafted suffix *and*
572 + the residual sweep — at low acceptance the scheme is strictly worse than reading
573 + residuals per token.
574 +6. **Smallest falsifying experiment.** **expG (acceptance-length variant).** Simulate
575 + exactly: base = 2/3-bit MLX quant drafting blocks of 8–16, verify with 4/8-bit
576 + refined weights, standard rejection rule; measure expected accepted length per
577 + domain. **Kill:** mean accepted block length <4 at a 3-bit base on chat/code
578 + (amortization factor too small: sweep bytes ÷ 4 still exceeds the per-token budget
579 + for 1.5× over-RAM models).
580 +
581 +### G18 — SpecSweep: dense over-RAM models via tree drafting + amortized model sweeps (no residuals)
582 +
583 +1. **What existing systems do.** SpecExec achieves Llama-2-70B (4-bit) at 4–6 tok/s on
584 + consumer GPUs by verifying huge draft trees in one pass over RAM-offloaded weights
585 + (~20 accepted tokens/sweep, 10.6–18.7× over sequential offloaded decoding) — CUDA,
586 + PCIe staging. llama.cpp ships `--model-draft` on Metal but never composes it with
587 + an overflowed target; MLX mmap overflow decodes at 0.025 tok/s.
588 +2. **What they do not do.** Nobody has built the SpecExec economics on unified memory
589 + + NVMe (no PCIe staging, zero-copy Metal buffers, and a *13 GB/s* sweep of
590 + overflow bytes) — the pure port is unoccupied and needs no new representation at
591 + all (SOTA §6.4: "concepts portable").
592 +3. **Why it might matter.** It is the only gap that helps *dense stock models beyond
593 + RAM today* with zero model transformation: overflow of 20 GB sweeps in ~1.5 s; at
594 + 20 accepted tokens/sweep that is ~13 tok/s of I/O ceiling — interactive. It also
595 + provides the honest baseline every fancier gap (G01, G17) must beat.
596 +4. **Plausible reason it could work.** The arithmetic above uses only measured numbers
597 + (expH ceiling, SpecExec acceptance); MRU (G08) makes the swept overflow set stable;
598 + the draft model (1B, 4-bit, ~0.7 GB) is noise in the RAM budget.
599 +5. **Biggest reason it might fail.** SpecExec's ~20-token acceptance relied on very
600 + large draft trees and a strong draft model; with a small MLX draft and greedy-ish
601 + chat workloads acceptance may be 4–6, and tree verification batch width on Metal at
602 + batch≈64–256 may be compute-bound (unified-memory bandwidth shared with the
603 + concurrent sweep — the one contention expH did *not* measure: RAM-vs-RAM, not
604 + SSD-vs-RAM).
605 +6. **Smallest falsifying experiment.** **New micro (expL_specsweep).** Measure (a)
606 + acceptance length of Llama-3.2-1B drafting for Llama-3.1-70B-Q4 (both runnable;
607 + target via llama.cpp Metal, RAM-resident for the measurement) with tree width
608 + 16–64; (b) Metal batch-verification throughput at batch 64 concurrent with a
609 + 13 GB/s read stream. **Kill:** accepted length <5, or verification throughput under
610 + concurrent streaming drops the compound I/O+compute ceiling below 2 tok/s at 1.4×
611 + over-RAM.
612 +
613 +### G19 — RouterBias: cache-conditional expert routing on stock MoE, zero-shot
614 +
615 +1. **What existing systems do.** Cache-conditional experts (trained) bias routing
616 + toward resident experts; MoBiLE substitutes smaller experts on miss (trained);
617 + llama.cpp serves expert misses by blocking page faults. All model-side co-design in
618 + print requires training the router.
619 +2. **What they do not do.** Nobody has measured how much *zero-shot* router-logit
620 + biasing (a runtime knob: add λ·resident_mask to gate logits, renormalize) buys in
621 + hit rate on a stock MoE, at what quality cost — despite the router's output being
622 + a distribution over near-substitutes by construction.
623 +3. **Why it might matter.** For MoE on Macs (the nearest-term deployment target,
624 + cf. G09), even a 10–15 point hit-rate lift at negligible quality cost multiplies
625 + directly into tokens/s; it needs ~5 lines in llama.cpp/mlx-lm — the highest
626 + value-per-effort entry in this document if it works.
627 +4. **Plausible reason it could work.** Top-2 gate margins are often small (experts are
628 + partially redundant by load-balancing training pressure); swapping expert #2 for
629 + resident expert #3 when gates are near-tied is a bounded perturbation, measurable
630 + per token; the bias λ can be gated by the gate margin itself (only re-route
631 + near-ties).
632 +5. **Biggest reason it might fail.** The trained systems exist *because* zero-shot
633 + biasing failed to preserve quality: gate margins may be decisive exactly on the
634 + tokens that matter, and systematically starving rarely-resident experts could
635 + compound into distribution drift over long generations (a failure aggregate PPL
636 + won't show — flips will).
637 +6. **Smallest falsifying experiment.** **expF (MoE variant).** On Qwen3-30B-A3B in
638 + mlx-lm, apply margin-gated resident bias at λ sweeping 0→strong with simulated
639 + 50%-resident caches; measure hit-rate lift vs flips/KL/GSM8K. **Kill:** every λ
640 + that lifts hit rate ≥10 points costs >2% flips or >1 pt GSM8K.
641 +
642 +### G20 — AmxSidecar: CPU (AMX/Accelerate) as a free co-executor on unified memory
643 +
644 +1. **What existing systems do.** PowerInfer splits hot/cold across GPU/CPU — meaningful
645 + only with separate memories (its macOS port is CPU-only with "limited" gains);
646 + llama.cpp offloads whole layers to CPU as a capacity fallback; no runtime treats
647 + Apple's AMX/Accelerate as a *concurrent* engine for the irregular fraction of work
648 + (sparse gathers, residual GEMVs, predictor probes) while the GPU runs the dense
649 + base — on the *same* bytes, no copies (SOTA §4.3: "GPU/CPU split is meaningless on
650 + unified memory" — as a *capacity* split; as a *kernel-affinity* split it is
651 + unexplored).
652 +2. **What they do not do.** Nobody measures whether M-series CPU matvec bandwidth
653 + (~100+ GB/s via AMX) survives concurrent GPU saturation, or schedules
654 + irregular-access kernels (where GPUs are weakest: G03's ΔY, G06's gathers) on the
655 + CPU specifically to dodge the Metal gather penalty.
656 +3. **Why it might matter.** Several gaps (G03, G06, G15) may die on Metal gather-kernel
657 + cost alone (item 6 kills above); a CPU sidecar is the escape hatch — the P-cores
658 + are idle during decode, and unified memory means the split costs zero copies.
659 +4. **Plausible reason it could work.** expH already showed the memory controller
660 + tolerates concurrent SSD+GPU load (<5%); Accelerate's sgemv on P-cores is
661 + latency-optimized and cache-friendly for irregular rows; the sidecar's work share
662 + (5–15% of bytes) matches the CPU:GPU bandwidth ratio.
663 +5. **Biggest reason it might fail.** RAM-bandwidth contention (unmeasured in expH:
664 + that tested SSD-vs-GPU, not CPU-vs-GPU on RAM): if concurrent CPU streaming steals
665 + >15% of GPU bandwidth, batch-1 decode slows more than the sidecar contributes; and
666 + CPU↔GPU synchronization per layer (~µs via shared events) × 2 ops × 80 layers may
667 + dominate at 10 ms/token.
668 +6. **Smallest falsifying experiment.** **New micro (expM_contention).** Concurrent:
669 + GPU dense matvec loop (MLX) + CPU Accelerate GEMV loop on separate buffers; measure
670 + both throughputs vs isolation, plus shared-event sync latency. **Kill:** CPU
671 + sustains <50 GB/s effective while the GPU is saturated, or GPU loses >15%, or
672 + per-layer sync >50 µs.
673 +
674 +---
675 +
676 +## G. Attention-side working set
677 +
678 +### G21 — KVBudget: joint weight+KV working-set accounting, SparQ-class KV fetch on Metal
679 +
680 +1. **What existing systems do.** KV quantization on Metal is solved-enough (llama.cpp
681 + `--cache-type-k/v`, mlx-lm `--kv-bits`, KIVI/KVQuant algorithms); SparQ fetches only
682 + the KV rows whose keys matter (8× attention-transfer cut, CUDA); SolidAttention
683 + pages KV to SSD with medium-chosen units (FAST 2026, Linux). Every weight-side
684 + system in this document ignores KV growth.
685 +2. **What they do not do.** No system co-budgets weights and KV as one working set on
686 + unified memory (SOTA §2.7: "any weight working-set argument must co-model KV"), and
687 + SparQ-style selective KV fetch has no Metal implementation — at 128k context the KV
688 + cache of an 8B model is multiple GB and *grows*, silently eating the residency that
689 + G01/G07/G09 assume is theirs.
690 +3. **Why it might matter.** If KV bytes/token dominate past ~32k context, every
691 + weight-side gap's benefit saturates: the project would be optimizing the minority
692 + term. Conversely, SparQ + 2-bit KV on Metal could free gigabytes of residency for
693 + weight caches — a direct multiplier on every other gap.
694 +4. **Plausible reason it could work.** All the algorithms are training-free and
695 + published; the fetch pattern (top-r key rows) is a gather over a
696 + RAM-resident cache — no SSD in the loop, so the 35:1 ratio does not apply; mlx has
697 + quantized-KV plumbing to extend.
698 +5. **Biggest reason it might fail.** As standalone research it may be moot: at the
699 + target interactive contexts (4–32k), KV even at 4-bit may be <10% of per-token
700 + bytes for the models we care about — engineering worth doing, but not a gap that
701 + changes any conclusion; and the Metal gather cost question (same as G06 item 5b)
702 + applies to key-row fetching too.
703 +6. **Smallest falsifying experiment.** **New micro (expN_kvshare, one afternoon).**
704 + Instrument mlx-lm on 8B/30B models: measure per-token bytes read from weights vs KV
705 + at contexts {4k, 32k, 128k} with KV at {16, 4, 2} bits. **Kill (as a priority
706 + gap):** KV share of per-token bytes <10% at ≤32k context for 8B–30B models —
707 + then defer to Phase 10 engineering and strike it from the candidate list.
708 +
709 +---
710 +
711 +## H. Contrarian gaps — cheap ways the premise dies
712 +
713 +### G22 — WarmEnough: maybe the kernel page cache already solves residency once selectivity exists
714 +
715 +1. **What existing systems do.** llama.cpp mmap + kernel LRU *beat* its own direct-I/O
716 + experiment because the UBC retained the hot expert working set (#18758); the page
717 + cache is shared, persistent across runs, and warm-start economics on macOS are
718 + excellent (SOTA notes §5). All of Group C (G08–G11) presumes custom residency
719 + management beats the kernel.
720 +2. **What they do not do.** Nobody has published steady-state *warm* paging overhead
721 + for a selective workload (MoE/sparse) on macOS — every dramatic number (0.025
722 + tok/s, 2.23×) is a cold-start or dense-cyclic measurement. The custom-policy case
723 + rests on worst cases that warm selective workloads may never hit.
724 +3. **Why it might matter (i.e., why this must be tested).** If warm steady-state fault
725 + overhead is <10% of token time, Group C is ~40% of this document's engineering for
726 + ~nothing — the correct architecture would be "compile a good layout (G04),
727 + mmap it, let the kernel work, spend all effort on representation (Group A) and
728 + speculation (Group F)."
729 +4. **Plausible reason it could be true.** LRU's pathology is *cyclic dense* scans;
730 + selective workloads (experts, gated blocks) have genuine recency structure — the
731 + regime where LRU is fine; 16 KB faults with 64-page clustered readahead may
732 + amortize adequately; and the UBC uses all free RAM opportunistically, which no
733 + self-managed wired budget can (it must leave headroom for the OS).
734 +5. **Biggest reason it might be false.** Fault-path costs are per-16 KB-page Mach
735 + overhead (the #18758 result: explicit 1 MiB slice reads beat faulting by 2.23×
736 + *cold*) and the UBC cannot exploit model-aware prefetch (router lookahead) —
737 + warm hit rates may be fine while miss *latency* still gates P99 token time.
738 +6. **Smallest falsifying experiment (of the contrarian claim — and, if it survives,
739 + of G08–G11).** **New micro (expO_warmsteady, needs only llama.cpp + dtrace).** Run
740 + Qwen3-30B-A3B GGUF mmap'd with RAM constrained (wired ballast) to 50/75/100% of
741 + model size; after 2k warm-up tokens, measure faults/token, fault-time share, and
742 + tok/s vs the fully-resident baseline. **Decision:** warm fault-time share <10% of
743 + token time at 75% residency ⇒ G08–G11 are demoted to engineering polish;
744 + >25% ⇒ Group C stands.
745 +
746 +### G23 — ConfidentlyWrong: maybe uncertainty signals cannot find the tokens that matter
747 +
748 +1. **What existing systems do.** The entire escalation family (BiLD, CALM, cascades,
749 + Kangaroo, and this document's G02/G17) assumes cheap-model confidence correlates
750 + with cheap-vs-full disagreement. Published evidence is indirect: acceptance rates
751 + and margins work *on average*; the *joint distribution* (margin × agreement) has
752 + never been published (SOTA §6.7).
753 +2. **What they do not do.** No paper reports P(disagree | margin) for quantized bases,
754 + nor what fraction of disagreements occur at high margin — the confidently-wrong
755 + mass. Thinking-Machines' knife-edge result and the 22–26-nat KLD outliers hint the
756 + tail is adversarial, and "Accuracy is Not All You Need" shows flips concentrate on
757 + load-bearing answers.
758 +3. **Why it must be tested.** If ≥30% of disagreements sit above any usable margin
759 + threshold, gating (G02) silently ships exactly the errors users notice, and
760 + calibrated-risk variants must set thresholds so conservative that escalation
761 + approaches 100% — the charter §17 "prediction overhead exceeds savings" failure,
762 + in its sharpest form.
763 +4. **Plausible reason the contrarian claim could be true.** Quantization error is not
764 + random noise w.r.t. the model's confidence: it is *systematic* (per-block rounding
765 + biases), so it can shift logits coherently — producing high-confidence flips rather
766 + than low-margin dithering; MoE-router-protection evidence (rankings change with
767 + *which* weights get precision) shows exactly such systematic sensitivity.
768 +5. **Biggest reason it might be false.** The early-exit literature works in practice
769 + across many models and tasks; QuantSpec's >90% acceptance with naive verification
770 + implies gross miscalibration is not the norm at 4-bit; and learned error features
771 + (ρ = 0.82) can catch systematic error that raw margin misses.
772 +6. **Smallest falsifying experiment.** **expG (the same run as G02 — one experiment,
773 + two hypotheses).** From the per-token logs, compute P(disagree | margin decile) and
774 + the fraction of disagreements above the 50th-percentile margin. **Decision:**
775 + disagreement-detection AUROC <0.65 for margin+entropy+learned features at a 3-bit
776 + base ⇒ G02 and G17's gated forms die (block-level rejection sampling in G17/G18,
777 + which never trusts the base, survives); AUROC >0.85 ⇒ the escalation family is the
778 + project's center.
779 +
780 +### G24 — BasinCollapse: maybe the low-bit base is not a usable draft at all
781 +
782 +1. **What existing systems do.** Every progressive/escalation gap here (G01–G03, G11,
783 + G15, G17) presumes a 2–3-bit-effective resident base whose behavior is "the same
784 + model, slightly noisy." ParetoQ reports a sharp representational transition between
785 + 2 and 3 bits (≤2-bit leaves the pretrained basin); EfficientQAT needs QAT to make
786 + 2-bit behave; llama.cpp blind tests found IQ2/IQ1 plainly distinguishable; BiLLM's
787 + 1.08-bpw PPL of 8.41 is "clearly degraded" (SOTA §2.6).
788 +2. **What they do not do.** Published numbers are PPL/benchmarks per bitrate; nobody
789 + reports the quantities the paging premise needs: *agreement with the full model*
790 + and *smoothness of recovery per residual byte* at 2–3-bit PTQ bases — the actual
791 + gates for G01/G17.
792 +3. **Why it must be tested.** RAM arithmetic forces the question: a 70B model on
793 + 48 GB (minus KV, OS, apparatus) needs a ≤3-bit base; a 100–250 GB stretch-target
794 + checkpoint (charter §15) needs ~2-bit-effective. If agreement craters there,
795 + the honest conclusion is that this project's ceiling is ~1.5× oversubscription of
796 + 4-bit models — a finding worth publishing and pivoting on (charter §17).
797 +4. **Plausible reason the contrarian claim could be true.** The ParetoQ transition is
798 + corroborated independently (EfficientQAT's QAT requirement, the IQ blind tests);
799 + error compounding is near-exponential in depth (QEP), and 70–80-layer models give
800 + 2-bit noise dozens of chances to leave the basin; PTQ at 2-bit has no mechanism to
801 + protect routing/decision structure (the MoE-router-protection lesson).
802 +5. **Biggest reason it might be false.** The transition literature measures *standalone*
803 + quality, not *draft* quality: a base can be individually mediocre yet agree with its
804 + own refined version on easy tokens (which are the majority), and G04's R-D
805 + allocation + residual-aware calibration (DWQ loop) may buy back ~0.5 effective bits
806 + exactly where the basin risk concentrates.
807 +6. **Smallest falsifying experiment.** **expD + expG jointly (same models, one run
808 + matrix).** MLX-quantize an 8B model at {2, 2.5(mixed), 3, 4} bits; measure (a)
809 + same-top-token vs the 8-bit reference per domain, (b) KL-vs-residual-bytes recovery
810 + smoothness. **Kill (for the progressive family at stretch scale):** 3-bit agreement
811 + <75% on chat/code or recovery is non-smooth (KL plateaus until >50% of residual
812 + bytes applied). 2-bit failing alone only caps oversubscription at ~1.6× — record
813 + the measured ceiling either way.
814 +
815 +---
816 +
817 +## Clusters and dependencies
818 +
819 +**Shared infrastructure clusters** (build once, serve many):
820 +
821 +- **Trace/importance pipeline** (per-token block-importance and mask logging on MLX
822 + models): required by G01, G06, G07, G08, G09, G12, G13, G24 — this is the expA/expB/expC
823 + instrumentation and must be built first; it is the single largest shared dependency.
824 +- **Quantize/refine ladder** (MLX models at 2/2.5/3/4/8 bits + per-block residual
825 + application): required by G01, G02, G03, G04, G11, G15, G17, G24.
826 +- **Custom Metal microbench rig** (matvec kernel harness with bytes/throughput
827 + accounting): required by G03, G05, G06 (gather), G16, G20, G21 — all expE variants.
828 +- **Cache/paging simulator** (trace replay through LRU/MRU/ARC/OPT with expH cost
829 + model): required by G08, G09, G12, G22 — pure Python, cheap, decouples policy
830 + questions from runtime engineering.
831 +- **llama.cpp/mlx-lm instrumentation** (fault/read/tok-s logging on real runs):
832 + required by G09, G18, G19, G21, G22.
833 +
834 +**Falsification fan-in — experiments that arbitrate several gaps at once, in order:**
835 +
836 +1. **expG (joint margin × agreement, multi-bitrate, multi-domain)** — one run matrix
837 + decides G02 (gating viability), G17 (acceptance lengths), G23 (confidently-wrong
838 + mass), and half of G24 (agreement at 2–3 bits). Highest information per compute-hour
839 + in the project; no custom kernels needed (MLX quantized casts + logging). **Run first.**
840 +2. **expA/expB/expC trace campaign (SwiGLU importance + stability + domain locality)**
841 + — one logging campaign over an 8B model across six domains feeds G01 (residual-set
842 + stability), G06 (mask sparsity/stability), G07 (cold-block demand), G12 (prefetch
843 + predictability), G13 (domain structure), and supplies the traces for the G08/G09/G22
844 + cache simulator. **Run second, concurrently with 3 where possible.**
845 +3. **expD/expE ladder (recovery curves + Metal decode/gather costs)** — decides G01
846 + (bytes-to-recover), G04 (R-D gain), G05 (format tax), G15 (spectral vs precision
847 + split), G16 (bandwidth-for-compute trade), the other half of G24, and gates G03's
848 + kernel. This is where custom Metal work concentrates; do the pure-MLX parts (expD)
849 + before writing any kernel.
850 +
851 +**Standalone/cheap probes that can run anytime** (fit in idle time, each ≤1 day):
852 +expN_kvshare (G21 — may strike a gap from the list), expO_warmsteady (G22 — may strike
853 +four), expM_contention (G20), expJ_purgeable (G10), G19's router-bias sweep.
854 +
855 +**Dependency edges that order Phase 4 ranking:** G22 gates the *priority* (not the
856 +design) of G08–G11; G23/G24 gate the entire escalation/progressive families (G01–G03,
857 +G17) — which is why expG runs first; G05 gates whether G01/G04 use bitplanes or must
858 +fall back to affine-delta residuals; G20 is the contingency for G03/G06/G15 gather-kernel
859 +failures; G18 is the baseline that every representation-heavy gap (G01, G15, G17) must
860 +beat at equal oversubscription before being declared interesting. Contrarian outcomes
861 +are publishable results per charter §16–17 either way: if G22–G24 all "win," the honest
862 +deliverable is a measured map of why the regime does not exist on this hardware — with
863 +G18 + G09 + G04 as the salvage architecture (speculative sweeps over a well-laid-out,
864 +kernel-cached, R-D-compiled model), which is itself unoccupied territory.
865