spb/localvm-research Public License

1---2project: localvm-research3document: research/research_gaps4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6created: 2026-08-127status: draft8---910# Research Gaps: Candidate Approaches for Over-Budget LLM Execution on Apple Silicon1112Phase 3 deliverable (charter §6). Each gap below is a substantially different candidate13mechanism, generated by crossing the four decoupling axes identified in14`research/state_of_the_art.md` §9 (precision, sparsity, sharing/delta, speculation) with15the four systems levers (residency policy, layout, prefetch, verification), then pruning16every cell already occupied per SOTA §7.1. Every gap cites the specific systems whose17limitation it exploits, and every falsifying experiment is runnable on the target machine18(Apple M5 Max, 48 GB unified memory, AP2048Z NVMe, MLX 0.32 / Metal, no CUDA).1920**Binding substrate numbers that every gap must respect** (expH, this repo, 2026-08-12):2122```text23SSD random read:   ~13 GB/s at 1 MiB QD8;  67 MB/s at 4 KiB QD1  (200× cliff)24Access contract:   16 KB-aligned blocks, ≥256 KiB effective reads at QD≥4–825Per-token budget:  ≤~650 MB SSD reads/token for ≥5 tok/s (50% I/O share);26                   ≤~1.3 GB/token theoretical limit27Overlap:           SSD reads under full Metal matmul load lose <5% — prefetch is free28RAM:SSD ratio:     ~35:1 best case → non-overlapped residual traffic must stay29                   ≲1–2% of weight bytes/token, or be amortized across tokens30Decode rule:       Metal kernels must be shift/mask-cheap (i-quant LUT penalty)31Agreement priors:  ~91% same-top-token at 4-bit; 69–75% acceptance at 25–40% depth;32                   >90% QuantSpec precision-draft acceptance; 5–15% knife-edge tokens33```3435Gaps are grouped by mechanism class (A–H). Groups do not imply combinations; each gap36stands or falls on its own falsification criterion. Contrarian gaps (§H) are designed to37kill the project's premises cheaply — per charter §17, they are as valuable as the rest.3839---4041## A. Progressive representations and refinement4243### G01 — ResidualPager: progressive precision as a demand-paged memory hierarchy44451. **What existing systems do.** Any-Precision LLM (ICML 2024), MatQuant (ICLR 2025),46   BitStack (ICLR 2025), RRQ, and Drop-by-Drop store nested/progressive weight encodings47   whose quality scales with bits consumed; each slice matches dedicated models at that48   bitrate. M2Cache tiers neurons across HBM/DRAM/SSD at mixed precision.492. **What they do not do.** All §5 systems keep **every plane resident** and pick the50   operating point statically per deployment; none pages residuals from storage; none51   reports bytes-read-per-token (SOTA §5 common ledger). M2Cache never upgrades a fetched52   neuron's precision (no refinement) and assumes PCIe/CUDA.533. **Why it might matter.** A 2–3-bit base of a 70B model is ~17.5–26 GB — resident on a54   48 GB Mac — while full quality lives on NVMe. RAM would bound *base* size, not model55   size: exactly the charter §2 decoupling (`resident size ≠ total size ≠ bytes/token`).564. **Plausible reason it could work.** Successive-refinement theory (Drop-by-Drop's57   grounding) guarantees monotone quality-in-bytes; BitDelta shows corrections quantize58   to ~1 bit near-losslessly; expH shows the SSD can deliver ~650 MB/token in ≥256 KiB59   blocks with free compute overlap; unified memory makes a fetched residual GPU-visible60   with zero copy — the mechanic discrete-GPU prior work lacked.615. **Biggest reason it might fail.** The 35:1 RAM:SSD ratio: if quality recovery needs62   residuals for a large, *unstable* fraction of blocks each token, traffic blows the63   1–2% budget and cannot be hidden. BitStack already reports reconstruction overhead64   slowing inference even with everything resident.656. **Smallest falsifying experiment.** **expD + expB.** On a 7–8B model, measure66   KL/agreement at 2, 2+1, 2+2… planes with per-block refinement ranked by sensitivity;67   then Jaccard(t, t+1) of the refined-block set. **Kill:** reaching KL ≤ 0.05 to FP1668   requires touching >20% of residual bytes per token AND temporal reuse of the refined69   set is <50% — i.e., steady-state fetch traffic >650 MB/token for a 70B-class model.7071### G02 — MarginGate: decision-uncertainty-gated residual escalation with a correctness story72731. **What existing systems do.** QSpec/QuantSpec/ML-SpecQD draft with a low-precision74   cast of the same weights and verify exactly — but both operand sets stay resident and75   every token is verified. BiLD escalates on low confidence (no guarantee, two separate76   resident models). CALM gives distribution-free calibrated early exit — for the77   **depth** axis only. PMPD/QuickSilver schedule precision per token position,78   feed-forward, with no verification.792. **What they do not do.** Nobody decides *how many residual planes to fetch from80   storage* based on whether the token decision is already stable (top-2 margin, entropy,81   or a learned error predictor), and nobody attaches either rejection-sampling exactness82   or CALM-style calibrated risk to that decision, and **no §5 or §6 paper optimizes or83   even reports bytes/token** (SOTA §7.3.2 — the verified open intersection).843. **Why it might matter.** If ~91% of tokens at 4-bit (and possibly 70–85% at 2–3-bit)85   are decided identically to FP16, then escalation — the only expensive event — happens86   on 5–15% of tokens, and residual streaming cost is paid only there: bytes/token87   collapses to base-read + (escalation rate × residual fetch).884. **Plausible reason it could work.** The correctness machinery already exists and is89   proven: speculative rejection sampling is exact; CALM calibration is distribution-free.90   Margins demonstrably carry signal (the entire early-exit line works). EMNLP-202591   residual-magnitude features predict quantization breakage at ρ = 0.82, giving a cheap92   learned signal beyond raw margin.935. **Biggest reason it might fail.** The dangerous quadrant: tokens where the base is94   *confidently wrong* (high margin, wrong token). If disagreements are not concentrated95   at low margin, gating misses exactly the errors that matter — and the 4-bit KLD tail96   (max 22–26 nats) plus the "flips are load-bearing tokens" evidence says this quadrant97   is not empty. Also: escalation may exceed 40% on reasoning/code (Sirius's finding that98   sparsity specifically damages reasoning is a warning for any cheap-first scheme).996. **Smallest falsifying experiment.** **expG.** On 1B→8B models, 2-bit/3-bit/4-bit base100   vs FP16 reference, log per token: (base top-2 margin, base entropy, agreement with101   reference, reference token rank under base). Domains: code, math, chat, FR, EN,102   reasoning. **Kill:** AUROC of margin (or margin+learned features) for predicting103   disagreement <0.8, or required escalation rate at 99% effective agreement >40% on any104   major domain. (This one measurement also arbitrates G17, G23, G24.)105106### G03 — DeltaKernel: incremental refinement ΔY = (ΔW)·X instead of recompute1071081. **What existing systems do.** Every verification/refinement scheme in the literature109   (LayerSkip, QSpec, QuantSpec, Sirius dense correction) **recomputes the full layer**110   with the better operand. GMRES-IR (Carson–Higham) refines a cheap factorization with111   residual-driven corrections and convergence guarantees — in numerical linear algebra,112   never written down for transformer inference (SOTA §6.6, §7.3.3).1132. **What they do not do.** No published primitive computes only the *correction term*114   `ΔY = (ΔW)X` against cached activations, where ΔW is the streamed residual plane115   (sparse, low-rank, or bitplane), and adds it to the cached base output.1163. **Why it might matter.** Refinement cost becomes proportional to residual size, not117   layer size: a 10%-density residual costs ~10% of the base matmul in both bytes and118   FLOPs. Without this, every escalation (G02) or block verification (G17) doubles119   compute, halving the whole family's economics.1204. **Plausible reason it could work.** Linearity: for linear layers Y = WX,121   Y_full = Y_base + (ΔW)X exactly. Nonlinearities (softmax, SwiGLU, norms) only need122   recomputation *downstream* of corrected layers, and expF maps which layers tolerate123   deferred correction. Activation caching for a deferred block costs124   ~hidden_dim × layers × block_len × 2 bytes ≈ tens of MB — trivial next to weights.1255. **Biggest reason it might fail.** Attention breaks the linear shortcut (Q/K/V126   corrections propagate through softmax non-linearly), so exactness holds per-linear-op,127   not per-layer; and a sparse/irregular ΔW matvec on Metal may be gather-bound and128   slower than a dense recompute (the Endor/i-quant lesson transposed to residuals).1296. **Smallest falsifying experiment.** **New micro (expE extension, expE_delta).**130   Implement in MLX/Metal: (a) dense base matvec, (b) fused ΔY with 5/10/25%-density131   block-sparse ΔW, (c) full recompute at base+residual precision. **Kill:** the ΔY132   kernel at 10% density is not ≥2× faster than full recompute, or numerical error of133   base+ΔY vs direct full compute exceeds quantization noise (>1e-3 relative on layer134   outputs).135136### G04 — RDLayout: rate-distortion-optimal embedded weight bitstreams, compiled to the SSD contract1371381. **What existing systems do.** EZW/SPIHT/EBCOT (JPEG2000) achieve per-block139   R-D-optimal truncation of significance-ordered bitstreams — 30-year-old, mature.140   BitStack unwittingly rediscovers embedded coding with SVD residual blocks but sorts141   by a global significance heuristic. GGUF i-quants use imatrix calibration to set142   *static per-tensor* precision. LLM-in-a-flash and Ripple optimize flash layout for143   co-activation, not for truncation.1442. **What they do not do.** Nobody compiles weights into per-block embedded bitstreams145   with truncation points optimized against **calibration-set logit KL** (not MSE),146   aligned to the measured storage contract (16 KB pages, ≥256 KiB fetch bundles), so147   that "bytes loaded per matrix" is a continuous, precomputed quality knob148   (SOTA §5.2, notes/decomposition §4).1493. **Why it might matter.** It converts layer sensitivity (expF) into *layout*, at150   compile time, with no runtime policy needed to get the first win: a static-but-151   nonuniform operating point strictly better than uniform bpw at the same bytes, and152   the substrate every dynamic gap (G01, G02, G17) would page from.1534. **Plausible reason it could work.** Sensitivity-weighted allocation already beats154   uniform (SqueezeLLM, imatrix, LASER's "some matrices improve when truncated");155   EBCOT's Lagrangian truncation is exactly the right optimizer once distortion = KL;156   the charter's compile stage (§14) explicitly budgets unlimited offline time.1575. **Biggest reason it might fail.** Diminishing returns: imatrix/k-quant mixing may158   already capture most of the nonuniform-allocation gain, leaving <0.01 KL at matched159   bytes — real but not research. Also per-block KL attribution is approximate (block160   interactions), so the "optimal" truncation may be optimal only for the calibration161   distribution.1626. **Smallest falsifying experiment.** **expD + expF.** On an 8B model, compare at163   matched total bytes: uniform 3-bit; imatrix-style per-tensor mixing; per-block164   KL-optimized truncation of a 2-bit+residual embedded code. **Kill:** per-block R-D165   allocation improves KL-to-FP16 by <15% relative over per-tensor imatrix mixing at166   the same byte budget.167168### G05 — MetalTrellis: LUT-free sub-3-bit decode kernels for Apple GPUs1691701. **What existing systems do.** The 2–2.5 bpw frontier (QuIP#, AQLM, QTIP, EXL3,171   Any-Precision bitplane engines) is entirely CUDA; QTIP reaches >80% of peak GPU172   bandwidth using a bitshift trellis (~2 instructions/weight). llama.cpp's i-quants173   prove LUT-heavy decode *loses* on Apple GPUs (53.9 vs 63.1 tok/s despite half the174   bytes).1752. **What they do not do.** Zero Metal implementations exist of bitplane matvec,176   MSB-sliced decode, or trellis decode (SOTA §7.3.5). Nobody knows whether the i-quant177   penalty applies to shift/mask-only progressive layouts on M-series GPUs.1783. **Why it might matter.** Every progressive gap (G01–G04) presumes the base and its179   residual planes decode at near-dense speed on Metal. If only affine group-quant180   decodes fast, the design space collapses to MLX's existing formats and residuals must181   be affine-quantized deltas — a materially different (and weaker) encoding.1824. **Plausible reason it could work.** Bitplane and bitshift-trellis decode are183   shift/mask-cheap by construction (satisfying the §1 decode rule); decode-bound184   kernels benefit from Apple's high ALU:BW ratio at batch 1 (dequant is free when185   memory-bound); MLX exposes custom-kernel plug points (`mode`).1865. **Biggest reason it might fail.** Bitplane matvec needs bit-transpose gathers with187   poor memory coalescing on Apple's GPU cache hierarchy; trellis decode is sequential188   within a block, fighting SIMD-group width. The failure would be quantitative and189   final: the format tax eats the byte savings.1906. **Smallest falsifying experiment.** **expE variant (expE_decode).** Implement three191   Metal matvecs at matched effective bpw (~2.5): MLX affine 2-bit+scales baseline;192   2-of-8 bitplane; small bitshift trellis. Measure tok/s-equivalent throughput on193   4096×4096. **Kill:** neither progressive variant reaches ≥70% of the affine194   baseline's throughput.195196---197198## B. Sparsity as a paging policy (SwiGLU, post-training)199200### G06 — TealPager: thresholded-SwiGLU sparsity as bytes/token reduction on Metal2012021. **What existing systems do.** TEAL achieves 40–50% model-wide training-free203   activation sparsity on stock SwiGLU models (1.53–1.8× decode, Triton/CUDA); CATS 50%204   FFN-only; GRIFFIN selects 50% of FF params once per sequence; DIP picks masks given205   cache contents. DejaVu/LLM-in-a-flash proved sparsity-driven paging — for ReLU206   models only (0.2 GB/token vs 13.4 GB).2072. **What they do not do.** No Metal kernel exists for any of them (SOTA §3.3); nobody208   has published **bytes/token for thresholded-SwiGLU paging**; all working-set209   statistics (Jaccard, reuse distance) in print are ReLU-era.2103. **Why it might matter.** 40–50% is only ~2× — but it is the *only* training-free211   selectivity mechanism for dense stock models (Llama-3, Qwen3-dense), and 2× on the212   FFN dominates once attention/KV are handled; combined with a 20–30% RAM deficit it213   is the difference between thrashing and fitting.2144. **Plausible reason it could work.** TEAL's thresholds are calibration-only; GRIFFIN's215   per-sequence "flocking" granularity is inherently prefetch-friendly (choose once,216   fetch 256 KiB bundles, reuse all sequence); expH says the fetch side is comfortable217   if masks are stable.2185. **Biggest reason it might fail.** Two independent cliffs: (a) Sirius showed219   contextual sparsity specifically damages reasoning/GSM8K — the 99%-retention claims220   are perplexity mirages; (b) a 50%-sparse gather matvec on Metal may run *slower*221   than dense (no sparse tensor cores; Endor's indexing-overhead point), saving bytes222   on paper and losing tokens/s in practice.2236. **Smallest falsifying experiment.** **expA + expB (SwiGLU masks), then expE.**224   Measure on Llama-3-8B: model-wide safe sparsity at ≤0.05 KL and GSM8K delta ≤1 pt;225   mask Jaccard(t, t+1) and per-sequence union growth. **Kill:** safe sparsity <30%226   model-wide, or Jaccard(t, t+1) <0.6 (mask churn makes paging traffic exceed the227   dense-read baseline), or (expE) the Metal gather kernel at 50% sparsity is slower228   than the dense matvec.229230### G07 — WandaTiers: importance scores as residency policy, with dense correction2312321. **What existing systems do.** Wanda/SparseGPT compute cheap importance scores233   (|W|·‖x‖, Hessian) and **delete** weights — costing 27% PPL at 7B for 50%234   unstructured. Endor packs pruned weights into a dense-readable bitmap format235   (CUDA). Sirius restores reasoning quality lost to sparsity by dense-verifying ~11%236   of tokens at 78% of the efficiency gain. M2Cache ranks neurons statically for tier237   placement.2382. **What they do not do.** Nobody uses one-shot importance scores as a **reversible239   tier assignment** — top-p% resident, bottom demoted to NVMe in a dense-readable240   format and re-materialized on demand — with a Sirius-style correction loop restoring241   the dense ceiling (SOTA §7.3.6). Pruning answers "total size"; this answers242   "residency".2433. **Why it might matter.** It is the cheapest possible route to a working244   over-budget dense runtime: no re-encoding, no predictors, no new format — just245   placement. If cold-weight touch rates are low, it alone runs a 1.3–1.5× over-RAM246   model interactively.2474. **Plausible reason it could work.** The demoted weights are, by construction, the248   ones whose absence changes outputs least (that's what the scores measure); misses249   can be served by zero (pruned behavior) immediately and corrected asynchronously;250   Sirius quantifies the correction cost at ~11% of tokens.2515. **Biggest reason it might fail.** Wanda-style scores are *global averages*; the252   tokens that need demoted weights may be exactly the load-bearing ones ("Accuracy is253   Not All You Need": 25%+ flips for pruning-type interventions at matched accuracy).254   If per-token demand for cold weights is diffuse rather than rare, traffic is random,255   small, and constant — the 67 MB/s regime.2566. **Smallest falsifying experiment.** **expA (blocked variant).** Demote the257   bottom-{20,35,50}% of FFN blocks by Wanda score; per token, measure (a)258   flip rate / KL vs dense with misses-as-zero, (b) how many demoted blocks would259   actually be needed to stay within KL ≤ 0.05. **Kill:** at 35% demotion, cold-block260   demand exceeds 5% of demoted bytes per token (≈ diffuse demand), or flip rate with261   Sirius-style 11% dense correction still exceeds 5%.262263---264265## C. Cache, replacement, and residency policy266267### G08 — MRURing: DBMIN-style per-tensor-class buffer pools with an MRU ring for cyclic scans2682691. **What existing systems do.** llama.cpp mmap and the MLX mmap prototype delegate270   replacement to the kernel's approximate LRU; all published expert caches271   (Eliseev–Mazur, PowerInfer-2) use LRU/LFU variants. Databases solved this in 1985:272   DBMIN assigns a per-file policy from the known access pattern, and for looping273   sequential scans **MRU is optimal while LRU is the worst case**.2742. **What they do not do.** No LLM runtime applies MRU to the dense-decode weight scan275   (which is a perfect cyclic scan, same order every token), or partitions the budget276   into per-tensor-class pools (pin embeddings; MRU ring for dense; ARC for experts;277   sliding window for KV) — SOTA §4.7 OS imports, apparently unpublished anywhere.2783. **Why it might matter.** For a dense model overflowing RAM by X GB, MRU keeps a279   stable (model−X) resident set and re-reads exactly X GB/token — the theoretical280   floor — where LRU re-reads *everything* (the measured 0.025 tok/s MLX failure).281   At X = 3 GB, the floor is ~0.23 s/token of I/O on our SSD: a 1.1× over-budget model282   becomes usable by policy alone.2834. **Plausible reason it could work.** The access pattern is fully known (static layer284   order — the "query plan"); the math is 40 years old; expH shows the re-read stream285   (large sequential slices at QD≥4) hits ~13 GB/s and overlaps freely with compute.2865. **Biggest reason it might fail.** The window of usefulness is narrow: the floor287   itself caps overflow at ~650 MB (5 tok/s) to ~6.5 GB (0.5 tok/s) — MRU rescues288   *slightly* over-budget models only, and macOS gives no direct control over UBC289   eviction, so the ring must be implemented with explicitly managed wired buffers290   under the iogpu wired limit (engineering risk: the panic cliff).2916. **Smallest falsifying experiment.** **New micro (expI_cachesim) + expH numbers.**292   Trace-driven simulation (traces are free: layer order is static) of LRU vs MRU vs293   OPT at 5–30% overflow, then a minimal MLX prototype streaming the overflow slice.294   **Kill:** measured end-to-end gain of the MRU prototype over kernel-LRU mmap <1.5×295   at 10–20% overflow, or wired-arena management overhead eats the gain.296297### G09 — ArcExperts: scan-resistant expert cache + explicit slice reads for MoE on macOS2982991. **What existing systems do.** llama.cpp runs MoE overflow by mmap demand paging300   (kernel LRU); discussion #18758 measured that layout-aware explicit slice reads cut301   cold-decode reads/token 1418→370 (2.23×, +13–14% end-to-end); a two-tier expert302   cache is an open, unfilled feature request (#20757). ARC/2Q/LIRS ghost-list caches303   are standard in storage systems; nobody has published one for experts (SOTA §4.7).3042. **What they do not do.** No shipping runtime combines: compiled 16 KB-aligned expert305   bundles, an ARC cache with ghost lists auto-partitioning recency vs frequency,306   QD≥8 explicit reads, and purgeable-warm demotion — on any OS, least of all macOS.3073. **Why it might matter.** MoE is where selectivity is free (the router already picks308   ~2/8–8/128 experts); the entire gap between llama.cpp's current behavior and the309   achievable floor is cache/IO policy. Qwen3-30B-A3B/80B-class models on 16–48 GB Macs310   are the most immediately deployable win of the whole project.3114. **Plausible reason it could work.** Every ingredient has isolated evidence312   (2.23× from layout alone; MoE temporal locality confirmed; ARC's superiority on313   mixed workloads is textbook); expert granularity (MBs) naturally satisfies the314   ≥256 KiB fetch contract.3155. **Biggest reason it might fail.** Modern MoE routing may be so uniform (load-balanced316   by training) that no cache policy beats LRU by more than a few points — the gain317   would then be layout+QD only, which llama.cpp could (and may) merge upstream first,318   making this engineering, not research.3196. **Smallest falsifying experiment.** **expB (MoE-trace variant).** Log real routing320   traces (llama.cpp/mlx-lm on Qwen3-30B-A3B) across domains; replay through LRU, MRU,321   ARC, LIRS, OPT simulators at cache = 25/50/75% of expert pool. **Kill:** ARC beats322   LRU by <5 points of hit rate at every cache size (policy headroom ≈ 0; only layout323   matters).324325### G10 — PurgeableWarm: kernel-cooperative weight caching via purgeable MTLHeaps3263271. **What existing systems do.** Inference runtimes either wire weights (mlx wired328   limit; panic at the cliff, mlx-lm #883) or leave them to the page cache (opaque329   eviction). macOS uniquely exposes purgeable/volatile memory with reclaim330   notification (`setPurgeableState`), used by graphics apps since iOS, never by an331   LLM runtime (SOTA §4.7 import #9).3322. **What they do not do.** No runtime maintains a *warm* weight tier the OS may333   reclaim under pressure — converting "jetsam/kernel-panic risk" into "graceful334   quality/latency degradation," and no runtime distinguishes wired-hot /335   purgeable-warm / SSD-cold as three managed tiers.3363. **Why it might matter.** Every other gap that caches anything (G01, G06–G09, G13)337   needs a warm tier that coexists with the user's other apps on a 48 GB machine;338   without cooperative purgeability, the practical resident budget is far below339   nominal RAM and failure is catastrophic rather than gradual.3404. **Plausible reason it could work.** The API exists and ships; reclaim is341   block-granular (per-resource/heap); a purged block is exactly a cache miss, which342   the G08/G09/G11 machinery already handles; re-fetch at 13 GB/s makes wrong343   purges cheap.3445. **Biggest reason it might fail.** Per-use volatile→nonvolatile state transitions on345   the render path may cost too much (an API round-trip per block per token), and the346   OS's purge order is not model-aware — it may purge the blocks we need most, making347   quality under pressure worse than a self-managed fixed budget.3486. **Smallest falsifying experiment.** **New micro (expJ_purgeable).** Fill a purgeable349   MTLHeap with weight blocks; run a matvec loop touching them; apply memory pressure350   (`memory_pressure -l critical`); measure transition overhead per access, purge351   granularity, and recovery latency. **Kill:** state-transition overhead >10 µs/block352   per token, or purges are so coarse/unpredictable that post-pressure recovery costs353   more than a fixed self-managed budget with SSD misses.354355### G11 — NeverBlock: anti-caching semantics — compute never stalls on a weight miss3563571. **What existing systems do.** All paging runtimes (llama.cpp mmap, LLM-in-a-flash,358   PowerInfer-2) **block** on a miss (fault or explicit read stall); MoBiLE substitutes359   a smaller expert on miss (MoE-only, with training); anti-caching databases360   (H-Store, VLDB 2013) abort-and-restart transactions so no thread ever waits on disk.3612. **What they do not do.** No LLM runtime has the invariant "a miss is served362   immediately by the resident approximation (base-precision block, zero, or substitute363   expert), the true bytes are fetched asynchronously, and the computation is optionally364   repaired later" — as an architectural rule rather than an ad-hoc trick (SOTA §4.7365   import #5).3663. **Why it might matter.** Tail latency: expH's averages hide P99 fetch latency367   (PowerInfer-2 reported +40.9% P99); a never-block runtime converts I/O jitter into368   bounded, measurable quality jitter — which G02's verification machinery can then369   repair. It decouples token cadence from storage latency entirely.3704. **Plausible reason it could work.** With a G01-style representation, a "miss" has a371   natural graceful value: the resident low-bit base — an approximation with *known,372   calibrated* error (expF), not garbage. The repair path is G03's ΔY kernel applied373   one-to-a-few tokens late.3745. **Biggest reason it might fail.** Error compounding across layers (QEP:375   near-exponential with depth): a miss at layer 10 contaminates all downstream layers376   for that token, so "repair later" may require replaying the whole suffix — at which377   point never-block is just deferred blocking with extra bookkeeping.3786. **Smallest falsifying experiment.** **expF.** Inject base-precision substitution at379   single layers and at realistic miss patterns (Poisson over layers at 5/10/20% miss380   rate) on an 8B model; measure flip rate and KL with and without one-token-deferred381   ΔY repair. **Kill:** at a 10% miss rate, deferred repair fails to keep flips <2%382   (vs the blocking baseline's 0%) — i.e., misses must be repaired synchronously383   anyway.384385---386387## D. Prefetch and prediction388389### G12 — HiddenPrefetch: learned hidden-state → future-weight-block prediction3903911. **What existing systems do.** Eliseev–Mazur apply layer k+1's router gate to layer392   k's hidden state (a static, 1-layer-lookahead heuristic) → 2–3 tok/s Mixtral on393   11–16 GB; DejaVu/ShadowLLM exploit "slowly changing hidden states" to predict394   sparsity for FLOP-skipping (ShadowLLM: one early-layer predictor shadows all395   layers); Pythia (MICRO 2021) shows online-RL hardware prefetchers that optimize396   hit-rate-minus-wasted-bandwidth.3972. **What they do not do.** Nobody trains a small predictor from hidden-state features398   to **SSD prefetch decisions N layers ahead** (N sized so lead time ≥ fetch latency),399   for dense-sparse blocks or residual planes rather than MoE experts, with wasted-bytes400   penalized (SOTA §3.1 extension; §4.7 import #6).4013. **Why it might matter.** The paging gaps (G01, G06, G07) live or die on hiding402   fetch latency; expH says overlap is free *if* the runtime knows what to fetch403   ~1–5 ms early. Hidden states are richer context than any hardware prefetcher has404   ever had.4054. **Plausible reason it could work.** ShadowLLM proved one early probe predicts406   all-layer needs with >15% accuracy gain over per-layer criteria; predictors cost407   <2.4% overhead in LLM-in-a-flash; a logistic probe per layer is microseconds on408   AMX.4095. **Biggest reason it might fail.** Prediction quality may collapse exactly where it410   matters: rare blocks (the ones not already cached) are by definition the tail of the411   distribution, and PowerInfer-2's predictors cost 2.6 GB of DRAM at 47B — the412   apparatus can eat the budget it saves.4136. **Smallest falsifying experiment.** **expB + a probe.** From expB traces, train a414   logistic probe: hidden state at layer k → block set needed at layer k+4. **Kill:**415   recall <70% at ≤2× overfetch for the *non-resident* block subset (measured against416   a realistic cache), or probe+metadata footprint >5% of the RAM budget.417418### G13 — DomainCache: temporary model specialization via semantic working sets4194201. **What existing systems do.** PowerInfer places hot neurons statically from global421   power-law statistics; GRIFFIN picks FF blocks per *sequence*; prompt/prefix caches422   specialize KV, never weights. No system precomputes or persists **per-domain weight423   working sets** ("coding session" vs "French chat") and swaps them at session424   granularity.4252. **What they do not do.** Nobody has even measured whether same-domain prompts reuse426   the same weight regions substantially more than cross-domain prompts on modern427   SwiGLU models (all locality numbers are ReLU-era) — charter expC exists precisely428   because this is unmeasured.4293. **Why it might matter.** Session granularity changes the economics completely:430   fetches amortize over thousands of tokens, the 35:1 ratio becomes irrelevant, and431   the mechanism composes with everything (warm the G08/G09 caches from a domain432   profile at session start — seconds of sequential 13 GB/s reads).4334. **Plausible reason it could work.** MoE routers demonstrably develop434   domain-correlated expert usage; LoRA's success proves task-relevant weight subspaces435   are small; if activation masks have any cross-prompt structure, a domain profile436   captures it offline for free.4375. **Biggest reason it might fail.** Transformer weight usage may be dominated by438   token-level (syntactic) rather than domain-level (semantic) factors — every prompt439   touching the same high-frequency circuitry plus a thin idiosyncratic tail — giving440   within-domain overlap barely above cross-domain overlap, i.e., nothing to cache at441   session granularity.4426. **Smallest falsifying experiment.** **expC.** Important-block sets (expA criterion)443   for 50 prompts × 6 domains on an 8B model; compute within-domain vs cross-domain444   Jaccard of the union working sets. **Kill:** mean within-domain minus cross-domain445   Jaccard <0.10 absolute, or the domain-specific tail is <10% of the working set446   (everything important is universal → a single static hot set suffices, use G07).447448---449450## E. Decomposition and sharing as memory-hierarchy assignments451452### G14 — TrunkDelta: shared cross-layer trunk resident, per-layer deltas paged4534541. **What existing systems do.** Basis Sharing, DeltaLLM, and Relaxed Recursive455   Transformers show layers can be re-expressed as shared components + per-layer456   corrections (12–25% compression, light calibration); BitDelta shows finetune deltas457   quantize to ~1 bit near-losslessly, and iterating it yields a monotone stack of458   1-bit masks. All are framed purely as *compression ratios*.4592. **What they do not do.** Nobody assigns the decomposition to the memory hierarchy:460   shared trunk (touched by every layer, maximal reuse) wired in RAM; per-layer 1–2-bit461   delta masks streamed sequentially from SSD in layer order (SOTA §2.10 extension,462   §7.3 open intersection).4633. **Why it might matter.** The trunk is read L times per token but stored once — its464   effective bytes/token *amortize across layers*, a reuse structure no flat encoding465   has. Delta streaming is perfectly sequential (layer order is static): the one access466   pattern our SSD serves at full 13 GB/s with trivial prefetch.4674. **Plausible reason it could work.** BitDelta's ~1 bit/param deltas are real,468   measured structure; "Secretly Linear" reports ~0.99 Procrustes linearity between469   consecutive layers (adjacent layers nearly share a basis); sequential delta prefetch470   needs zero prediction machinery.4715. **Biggest reason it might fail.** All published sharing needs light retraining472   (DeltaLLM: 30–40M tokens; RRT: distillation); zero-shot/calibration-only trunk473   extraction may degrade so much that deltas must carry most information — then delta474   bytes/layer ≈ layer bytes and nothing is saved. BitDelta's evidence is for475   *finetune-vs-base* pairs, not *layer-vs-layer* within one model — the transfer is476   unproven.4776. **Smallest falsifying experiment.** **New micro (expK_trunk, expD-adjacent).** On an478   8B model, fit a shared basis across layer groups by calibration-only alternating479   least squares (charter permits calibration passes); measure PPL/KL with trunk-only480   vs trunk+quantized-delta at delta budgets {0.5, 1, 2} bits/param. **Kill:**481   at trunk = 30% of total bytes, KL ≤ 0.1 requires deltas ≥3 bits/param482   (delta traffic ≈ flat-model traffic; no amortization win).483484### G15 — SpectralSplit: low-rank hot path resident + quantized sparse residual cold path4854861. **What existing systems do.** SVD-LLM/SliceGPT keep the low-rank part and **discard487   the residual** (quality collapses past ~25–40% compression because the residual488   carries most spectral energy); CALDERA encodes `W ≈ Q + LR` jointly as a *static*489   2–2.5 bpw format (CUDA, both terms always read); LASER shows selected late-MLP490   matrices *improve* when their high-order components are dropped.4912. **What they do not do.** Nobody stores the SVD residual at all (SVD family) or492   treats the two terms as different *tiers* (CALDERA): low-rank factors resident493   (dense small GEMMs — the one decomposition Metal already runs fast), residual on494   SSD, fetched per-block only where LASER-style analysis says the residual matters495   (SOTA §2.8 extension).4963. **Why it might matter.** Unlike bitplanes (G01), both components use *existing,497   fast* MLX kernels today — no custom Metal work gates a prototype; and LASER implies498   the residual tier can be skipped entirely for some matrices (negative bytes).4994. **Plausible reason it could work.** Rank-r hot path costs r(m+n)/mn of dense bytes500   (~12% at r=256 for 4096²); per-matrix residual need is measurable offline; the501   split is exact by construction (base + residual = W), so quality is a pure function502   of which residual blocks are fetched — a clean knob for G02's gating.5035. **Biggest reason it might fail.** Weights are *not* globally low-rank (SOTA §2.8's504   headline): the low-rank base alone may be so poor (unlike a 2-bit quantized base,505   which is uniformly mediocre) that residual demand is near-universal per token —506   worse-behaved than G01's precision split, with double the kernel launches.5076. **Smallest falsifying experiment.** **expD (SVD variant).** On an 8B model, base =508   rank-{128,256} per matrix + 3-bit quantized residual in importance-ranked blocks;509   measure KL vs fraction of residual blocks applied, per matrix class. **Kill:**510   base-alone KL >1.0 nats *and* reaching KL ≤ 0.05 needs >50% of residual bytes for511   most matrices — i.e., strictly dominated by the G01 bitplane split at equal bytes.512513### G16 — SeedSynth: regenerate weights from seeds — trade bandwidth for idle ALUs5145151. **What existing systems do.** SeedLM (Apple) replaces weight blocks with LFSR seeds516   + coefficients, regenerating pseudo-random bases at inference: 4-bit parity,517   data-free, ~4× speedup on an FPGA at 70B — explicitly trading memory bandwidth for518   free compute. No Metal port exists; DFloat11 shows the *wrong* version of this trade519   (entropy decode on the critical path → 2–3× slower).5202. **What they do not do.** Nobody has tested generative reconstruction on Apple GPUs,521   where batch-1 decode is bandwidth-bound and 40 GPU cores sit partially idle —522   precisely the regime the trade targets; and nobody composes it with paging (seeds523   are ~100× smaller than blocks: a "prefetch" of seeds is nearly free).5243. **Why it might matter.** If it works, it attacks bytes/token *below* the525   information-theoretic floor of stored-bits formats: resident bytes and streamed526   bytes both shrink to seeds+coefficients, and SSD traffic per block drops ~10×,527   trivially satisfying every budget in this document.5284. **Plausible reason it could work.** LFSR generation is shift/XOR-only (passes the529   §1 decode rule, unlike DFloat11's Huffman); Apple authored SeedLM with edge530   inference as the stated motivation; the ALU:BW imbalance at batch 1 is real and531   measured.5325. **Biggest reason it might fail.** Regeneration cost per weight might exceed the533   ~2 instructions/weight ceiling that QTIP shows is needed to stay bandwidth-limited;534   quality is 4-bit-parity at best (no path to 16-bit exactness — it cannot serve as535   a *verification* tier, only as a base), and per-block coefficient search at compile536   time is expensive for 70B-class models.5376. **Smallest falsifying experiment.** **expE variant (expE_seed).** Metal kernel:538   LFSR-basis generation fused into matvec vs MLX affine 4-bit matvec at matched539   quality (SeedLM's published 4-bit parity settings on one 8B layer). **Kill:**540   fused generate+matvec throughput <80% of the 4-bit dense baseline (the compute541   trade loses on this GPU), or per-layer PPL parity claims fail to reproduce on a542   SwiGLU model.543544---545546## F. Scheduling, speculation, and amortization547548### G17 — BlockVerify: token-batch amortization of residual fetches (deferred verification)5495501. **What existing systems do.** LayerSkip/Draft&Verify verify a drafted token block551   with one full-model pass (depth axis, all weights resident and read every552   verification); SpecExec shows ~20 accepted tokens per full offloaded-weight sweep553   turns 4.5 s/token into 4–6 tok/s; Kangaroo stops drafting on low confidence;554   QuantSpec verifies with resident 16-bit weights every block.5552. **What they do not do.** Nobody drafts N tokens with a *resident low-bit base* and556   then performs **one residual-augmented verification sweep** whose streamed residual557   bytes amortize over the accepted block — i.e., SpecExec's economics applied to558   residual planes instead of full weights, with rollback on rejection (SOTA §8 open:559   "block-deferred verification policy").5603. **Why it might matter.** It divides residual bytes/token by the acceptance length:561   if base-vs-refined agreement resembles QuantSpec's (>90%), expected block length562   ≥7–10 and the 1–2% traffic ceiling relaxes to 10–20% of weight bytes per sweep —563   a qualitatively easier target than G02's per-token gating.5644. **Plausible reason it could work.** The rejection-sampling correctness proof carries565   over unchanged (draft distribution = base model, target = refined model, same566   support); acceptance priors are measured (91% at 4-bit); verification sweeps read567   residuals *sequentially by layer* — the SSD-friendly pattern; MLX already ships568   speculative verification kernels on Metal (ReDrafter).5695. **Biggest reason it might fail.** Acceptance at a 2–3-bit base (required for real570   oversubscription) is unmeasured and the ParetoQ transition suggests it may crater;571   and rollback economics: one early rejection wastes the whole drafted suffix *and*572   the residual sweep — at low acceptance the scheme is strictly worse than reading573   residuals per token.5746. **Smallest falsifying experiment.** **expG (acceptance-length variant).** Simulate575   exactly: base = 2/3-bit MLX quant drafting blocks of 8–16, verify with 4/8-bit576   refined weights, standard rejection rule; measure expected accepted length per577   domain. **Kill:** mean accepted block length <4 at a 3-bit base on chat/code578   (amortization factor too small: sweep bytes ÷ 4 still exceeds the per-token budget579   for 1.5× over-RAM models).580581### G18 — SpecSweep: dense over-RAM models via tree drafting + amortized model sweeps (no residuals)5825831. **What existing systems do.** SpecExec achieves Llama-2-70B (4-bit) at 4–6 tok/s on584   consumer GPUs by verifying huge draft trees in one pass over RAM-offloaded weights585   (~20 accepted tokens/sweep, 10.6–18.7× over sequential offloaded decoding) — CUDA,586   PCIe staging. llama.cpp ships `--model-draft` on Metal but never composes it with587   an overflowed target; MLX mmap overflow decodes at 0.025 tok/s.5882. **What they do not do.** Nobody has built the SpecExec economics on unified memory589   + NVMe (no PCIe staging, zero-copy Metal buffers, and a *13 GB/s* sweep of590   overflow bytes) — the pure port is unoccupied and needs no new representation at591   all (SOTA §6.4: "concepts portable").5923. **Why it might matter.** It is the only gap that helps *dense stock models beyond593   RAM today* with zero model transformation: overflow of 20 GB sweeps in ~1.5 s; at594   20 accepted tokens/sweep that is ~13 tok/s of I/O ceiling — interactive. It also595   provides the honest baseline every fancier gap (G01, G17) must beat.5964. **Plausible reason it could work.** The arithmetic above uses only measured numbers597   (expH ceiling, SpecExec acceptance); MRU (G08) makes the swept overflow set stable;598   the draft model (1B, 4-bit, ~0.7 GB) is noise in the RAM budget.5995. **Biggest reason it might fail.** SpecExec's ~20-token acceptance relied on very600   large draft trees and a strong draft model; with a small MLX draft and greedy-ish601   chat workloads acceptance may be 4–6, and tree verification batch width on Metal at602   batch≈64–256 may be compute-bound (unified-memory bandwidth shared with the603   concurrent sweep — the one contention expH did *not* measure: RAM-vs-RAM, not604   SSD-vs-RAM).6056. **Smallest falsifying experiment.** **New micro (expL_specsweep).** Measure (a)606   acceptance length of Llama-3.2-1B drafting for Llama-3.1-70B-Q4 (both runnable;607   target via llama.cpp Metal, RAM-resident for the measurement) with tree width608   16–64; (b) Metal batch-verification throughput at batch 64 concurrent with a609   13 GB/s read stream. **Kill:** accepted length <5, or verification throughput under610   concurrent streaming drops the compound I/O+compute ceiling below 2 tok/s at 1.4×611   over-RAM.612613### G19 — RouterBias: cache-conditional expert routing on stock MoE, zero-shot6146151. **What existing systems do.** Cache-conditional experts (trained) bias routing616   toward resident experts; MoBiLE substitutes smaller experts on miss (trained);617   llama.cpp serves expert misses by blocking page faults. All model-side co-design in618   print requires training the router.6192. **What they do not do.** Nobody has measured how much *zero-shot* router-logit620   biasing (a runtime knob: add λ·resident_mask to gate logits, renormalize) buys in621   hit rate on a stock MoE, at what quality cost — despite the router's output being622   a distribution over near-substitutes by construction.6233. **Why it might matter.** For MoE on Macs (the nearest-term deployment target,624   cf. G09), even a 10–15 point hit-rate lift at negligible quality cost multiplies625   directly into tokens/s; it needs ~5 lines in llama.cpp/mlx-lm — the highest626   value-per-effort entry in this document if it works.6274. **Plausible reason it could work.** Top-2 gate margins are often small (experts are628   partially redundant by load-balancing training pressure); swapping expert #2 for629   resident expert #3 when gates are near-tied is a bounded perturbation, measurable630   per token; the bias λ can be gated by the gate margin itself (only re-route631   near-ties).6325. **Biggest reason it might fail.** The trained systems exist *because* zero-shot633   biasing failed to preserve quality: gate margins may be decisive exactly on the634   tokens that matter, and systematically starving rarely-resident experts could635   compound into distribution drift over long generations (a failure aggregate PPL636   won't show — flips will).6376. **Smallest falsifying experiment.** **expF (MoE variant).** On Qwen3-30B-A3B in638   mlx-lm, apply margin-gated resident bias at λ sweeping 0→strong with simulated639   50%-resident caches; measure hit-rate lift vs flips/KL/GSM8K. **Kill:** every λ640   that lifts hit rate ≥10 points costs >2% flips or >1 pt GSM8K.641642### G20 — AmxSidecar: CPU (AMX/Accelerate) as a free co-executor on unified memory6436441. **What existing systems do.** PowerInfer splits hot/cold across GPU/CPU — meaningful645   only with separate memories (its macOS port is CPU-only with "limited" gains);646   llama.cpp offloads whole layers to CPU as a capacity fallback; no runtime treats647   Apple's AMX/Accelerate as a *concurrent* engine for the irregular fraction of work648   (sparse gathers, residual GEMVs, predictor probes) while the GPU runs the dense649   base — on the *same* bytes, no copies (SOTA §4.3: "GPU/CPU split is meaningless on650   unified memory" — as a *capacity* split; as a *kernel-affinity* split it is651   unexplored).6522. **What they do not do.** Nobody measures whether M-series CPU matvec bandwidth653   (~100+ GB/s via AMX) survives concurrent GPU saturation, or schedules654   irregular-access kernels (where GPUs are weakest: G03's ΔY, G06's gathers) on the655   CPU specifically to dodge the Metal gather penalty.6563. **Why it might matter.** Several gaps (G03, G06, G15) may die on Metal gather-kernel657   cost alone (item 6 kills above); a CPU sidecar is the escape hatch — the P-cores658   are idle during decode, and unified memory means the split costs zero copies.6594. **Plausible reason it could work.** expH already showed the memory controller660   tolerates concurrent SSD+GPU load (<5%); Accelerate's sgemv on P-cores is661   latency-optimized and cache-friendly for irregular rows; the sidecar's work share662   (5–15% of bytes) matches the CPU:GPU bandwidth ratio.6635. **Biggest reason it might fail.** RAM-bandwidth contention (unmeasured in expH:664   that tested SSD-vs-GPU, not CPU-vs-GPU on RAM): if concurrent CPU streaming steals665   >15% of GPU bandwidth, batch-1 decode slows more than the sidecar contributes; and666   CPU↔GPU synchronization per layer (~µs via shared events) × 2 ops × 80 layers may667   dominate at 10 ms/token.6686. **Smallest falsifying experiment.** **New micro (expM_contention).** Concurrent:669   GPU dense matvec loop (MLX) + CPU Accelerate GEMV loop on separate buffers; measure670   both throughputs vs isolation, plus shared-event sync latency. **Kill:** CPU671   sustains <50 GB/s effective while the GPU is saturated, or GPU loses >15%, or672   per-layer sync >50 µs.673674---675676## G. Attention-side working set677678### G21 — KVBudget: joint weight+KV working-set accounting, SparQ-class KV fetch on Metal6796801. **What existing systems do.** KV quantization on Metal is solved-enough (llama.cpp681   `--cache-type-k/v`, mlx-lm `--kv-bits`, KIVI/KVQuant algorithms); SparQ fetches only682   the KV rows whose keys matter (8× attention-transfer cut, CUDA); SolidAttention683   pages KV to SSD with medium-chosen units (FAST 2026, Linux). Every weight-side684   system in this document ignores KV growth.6852. **What they do not do.** No system co-budgets weights and KV as one working set on686   unified memory (SOTA §2.7: "any weight working-set argument must co-model KV"), and687   SparQ-style selective KV fetch has no Metal implementation — at 128k context the KV688   cache of an 8B model is multiple GB and *grows*, silently eating the residency that689   G01/G07/G09 assume is theirs.6903. **Why it might matter.** If KV bytes/token dominate past ~32k context, every691   weight-side gap's benefit saturates: the project would be optimizing the minority692   term. Conversely, SparQ + 2-bit KV on Metal could free gigabytes of residency for693   weight caches — a direct multiplier on every other gap.6944. **Plausible reason it could work.** All the algorithms are training-free and695   published; the fetch pattern (top-r key rows) is a gather over a696   RAM-resident cache — no SSD in the loop, so the 35:1 ratio does not apply; mlx has697   quantized-KV plumbing to extend.6985. **Biggest reason it might fail.** As standalone research it may be moot: at the699   target interactive contexts (4–32k), KV even at 4-bit may be <10% of per-token700   bytes for the models we care about — engineering worth doing, but not a gap that701   changes any conclusion; and the Metal gather cost question (same as G06 item 5b)702   applies to key-row fetching too.7036. **Smallest falsifying experiment.** **New micro (expN_kvshare, one afternoon).**704   Instrument mlx-lm on 8B/30B models: measure per-token bytes read from weights vs KV705   at contexts {4k, 32k, 128k} with KV at {16, 4, 2} bits. **Kill (as a priority706   gap):** KV share of per-token bytes <10% at ≤32k context for 8B–30B models —707   then defer to Phase 10 engineering and strike it from the candidate list.708709---710711## H. Contrarian gaps — cheap ways the premise dies712713### G22 — WarmEnough: maybe the kernel page cache already solves residency once selectivity exists7147151. **What existing systems do.** llama.cpp mmap + kernel LRU *beat* its own direct-I/O716   experiment because the UBC retained the hot expert working set (#18758); the page717   cache is shared, persistent across runs, and warm-start economics on macOS are718   excellent (SOTA notes §5). All of Group C (G08–G11) presumes custom residency719   management beats the kernel.7202. **What they do not do.** Nobody has published steady-state *warm* paging overhead721   for a selective workload (MoE/sparse) on macOS — every dramatic number (0.025722   tok/s, 2.23×) is a cold-start or dense-cyclic measurement. The custom-policy case723   rests on worst cases that warm selective workloads may never hit.7243. **Why it might matter (i.e., why this must be tested).** If warm steady-state fault725   overhead is <10% of token time, Group C is ~40% of this document's engineering for726   ~nothing — the correct architecture would be "compile a good layout (G04),727   mmap it, let the kernel work, spend all effort on representation (Group A) and728   speculation (Group F)."7294. **Plausible reason it could be true.** LRU's pathology is *cyclic dense* scans;730   selective workloads (experts, gated blocks) have genuine recency structure — the731   regime where LRU is fine; 16 KB faults with 64-page clustered readahead may732   amortize adequately; and the UBC uses all free RAM opportunistically, which no733   self-managed wired budget can (it must leave headroom for the OS).7345. **Biggest reason it might be false.** Fault-path costs are per-16 KB-page Mach735   overhead (the #18758 result: explicit 1 MiB slice reads beat faulting by 2.23×736   *cold*) and the UBC cannot exploit model-aware prefetch (router lookahead) —737   warm hit rates may be fine while miss *latency* still gates P99 token time.7386. **Smallest falsifying experiment (of the contrarian claim — and, if it survives,739   of G08–G11).** **New micro (expO_warmsteady, needs only llama.cpp + dtrace).** Run740   Qwen3-30B-A3B GGUF mmap'd with RAM constrained (wired ballast) to 50/75/100% of741   model size; after 2k warm-up tokens, measure faults/token, fault-time share, and742   tok/s vs the fully-resident baseline. **Decision:** warm fault-time share <10% of743   token time at 75% residency ⇒ G08–G11 are demoted to engineering polish;744   >25% ⇒ Group C stands.745746### G23 — ConfidentlyWrong: maybe uncertainty signals cannot find the tokens that matter7477481. **What existing systems do.** The entire escalation family (BiLD, CALM, cascades,749   Kangaroo, and this document's G02/G17) assumes cheap-model confidence correlates750   with cheap-vs-full disagreement. Published evidence is indirect: acceptance rates751   and margins work *on average*; the *joint distribution* (margin × agreement) has752   never been published (SOTA §6.7).7532. **What they do not do.** No paper reports P(disagree | margin) for quantized bases,754   nor what fraction of disagreements occur at high margin — the confidently-wrong755   mass. Thinking-Machines' knife-edge result and the 22–26-nat KLD outliers hint the756   tail is adversarial, and "Accuracy is Not All You Need" shows flips concentrate on757   load-bearing answers.7583. **Why it must be tested.** If ≥30% of disagreements sit above any usable margin759   threshold, gating (G02) silently ships exactly the errors users notice, and760   calibrated-risk variants must set thresholds so conservative that escalation761   approaches 100% — the charter §17 "prediction overhead exceeds savings" failure,762   in its sharpest form.7634. **Plausible reason the contrarian claim could be true.** Quantization error is not764   random noise w.r.t. the model's confidence: it is *systematic* (per-block rounding765   biases), so it can shift logits coherently — producing high-confidence flips rather766   than low-margin dithering; MoE-router-protection evidence (rankings change with767   *which* weights get precision) shows exactly such systematic sensitivity.7685. **Biggest reason it might be false.** The early-exit literature works in practice769   across many models and tasks; QuantSpec's >90% acceptance with naive verification770   implies gross miscalibration is not the norm at 4-bit; and learned error features771   (ρ = 0.82) can catch systematic error that raw margin misses.7726. **Smallest falsifying experiment.** **expG (the same run as G02 — one experiment,773   two hypotheses).** From the per-token logs, compute P(disagree | margin decile) and774   the fraction of disagreements above the 50th-percentile margin. **Decision:**775   disagreement-detection AUROC <0.65 for margin+entropy+learned features at a 3-bit776   base ⇒ G02 and G17's gated forms die (block-level rejection sampling in G17/G18,777   which never trusts the base, survives); AUROC >0.85 ⇒ the escalation family is the778   project's center.779780### G24 — BasinCollapse: maybe the low-bit base is not a usable draft at all7817821. **What existing systems do.** Every progressive/escalation gap here (G01–G03, G11,783   G15, G17) presumes a 2–3-bit-effective resident base whose behavior is "the same784   model, slightly noisy." ParetoQ reports a sharp representational transition between785   2 and 3 bits (≤2-bit leaves the pretrained basin); EfficientQAT needs QAT to make786   2-bit behave; llama.cpp blind tests found IQ2/IQ1 plainly distinguishable; BiLLM's787   1.08-bpw PPL of 8.41 is "clearly degraded" (SOTA §2.6).7882. **What they do not do.** Published numbers are PPL/benchmarks per bitrate; nobody789   reports the quantities the paging premise needs: *agreement with the full model*790   and *smoothness of recovery per residual byte* at 2–3-bit PTQ bases — the actual791   gates for G01/G17.7923. **Why it must be tested.** RAM arithmetic forces the question: a 70B model on793   48 GB (minus KV, OS, apparatus) needs a ≤3-bit base; a 100–250 GB stretch-target794   checkpoint (charter §15) needs ~2-bit-effective. If agreement craters there,795   the honest conclusion is that this project's ceiling is ~1.5× oversubscription of796   4-bit models — a finding worth publishing and pivoting on (charter §17).7974. **Plausible reason the contrarian claim could be true.** The ParetoQ transition is798   corroborated independently (EfficientQAT's QAT requirement, the IQ blind tests);799   error compounding is near-exponential in depth (QEP), and 70–80-layer models give800   2-bit noise dozens of chances to leave the basin; PTQ at 2-bit has no mechanism to801   protect routing/decision structure (the MoE-router-protection lesson).8025. **Biggest reason it might be false.** The transition literature measures *standalone*803   quality, not *draft* quality: a base can be individually mediocre yet agree with its804   own refined version on easy tokens (which are the majority), and G04's R-D805   allocation + residual-aware calibration (DWQ loop) may buy back ~0.5 effective bits806   exactly where the basin risk concentrates.8076. **Smallest falsifying experiment.** **expD + expG jointly (same models, one run808   matrix).** MLX-quantize an 8B model at {2, 2.5(mixed), 3, 4} bits; measure (a)809   same-top-token vs the 8-bit reference per domain, (b) KL-vs-residual-bytes recovery810   smoothness. **Kill (for the progressive family at stretch scale):** 3-bit agreement811   <75% on chat/code or recovery is non-smooth (KL plateaus until >50% of residual812   bytes applied). 2-bit failing alone only caps oversubscription at ~1.6× — record813   the measured ceiling either way.814815---816817## Clusters and dependencies818819**Shared infrastructure clusters** (build once, serve many):820821- **Trace/importance pipeline** (per-token block-importance and mask logging on MLX822  models): required by G01, G06, G07, G08, G09, G12, G13, G24 — this is the expA/expB/expC823  instrumentation and must be built first; it is the single largest shared dependency.824- **Quantize/refine ladder** (MLX models at 2/2.5/3/4/8 bits + per-block residual825  application): required by G01, G02, G03, G04, G11, G15, G17, G24.826- **Custom Metal microbench rig** (matvec kernel harness with bytes/throughput827  accounting): required by G03, G05, G06 (gather), G16, G20, G21 — all expE variants.828- **Cache/paging simulator** (trace replay through LRU/MRU/ARC/OPT with expH cost829  model): required by G08, G09, G12, G22 — pure Python, cheap, decouples policy830  questions from runtime engineering.831- **llama.cpp/mlx-lm instrumentation** (fault/read/tok-s logging on real runs):832  required by G09, G18, G19, G21, G22.833834**Falsification fan-in — experiments that arbitrate several gaps at once, in order:**8358361. **expG (joint margin × agreement, multi-bitrate, multi-domain)** — one run matrix837   decides G02 (gating viability), G17 (acceptance lengths), G23 (confidently-wrong838   mass), and half of G24 (agreement at 2–3 bits). Highest information per compute-hour839   in the project; no custom kernels needed (MLX quantized casts + logging). **Run first.**8402. **expA/expB/expC trace campaign (SwiGLU importance + stability + domain locality)**841   — one logging campaign over an 8B model across six domains feeds G01 (residual-set842   stability), G06 (mask sparsity/stability), G07 (cold-block demand), G12 (prefetch843   predictability), G13 (domain structure), and supplies the traces for the G08/G09/G22844   cache simulator. **Run second, concurrently with 3 where possible.**8453. **expD/expE ladder (recovery curves + Metal decode/gather costs)** — decides G01846   (bytes-to-recover), G04 (R-D gain), G05 (format tax), G15 (spectral vs precision847   split), G16 (bandwidth-for-compute trade), the other half of G24, and gates G03's848   kernel. This is where custom Metal work concentrates; do the pure-MLX parts (expD)849   before writing any kernel.850851**Standalone/cheap probes that can run anytime** (fit in idle time, each ≤1 day):852expN_kvshare (G21 — may strike a gap from the list), expO_warmsteady (G22 — may strike853four), expM_contention (G20), expJ_purgeable (G10), G19's router-bias sweep.854855**Dependency edges that order Phase 4 ranking:** G22 gates the *priority* (not the856design) of G08–G11; G23/G24 gate the entire escalation/progressive families (G01–G03,857G17) — which is why expG runs first; G05 gates whether G01/G04 use bitplanes or must858fall back to affine-delta residuals; G20 is the contingency for G03/G06/G15 gather-kernel859failures; G18 is the baseline that every representation-heavy gap (G01, G15, G17) must860beat at equal oversubscription before being declared interesting. Contrarian outcomes861are publishable results per charter §16–17 either way: if G22–G24 all "win," the honest862deliverable is a measured map of why the regime does not exist on this hardware — with863G18 + G09 + G04 as the salvage architecture (speculative sweeps over a well-laid-out,864kernel-cached, R-D-compiled model), which is itself unoccupied territory.865