spb/localvm-research Public License

1---2project: localvm-research3document: research/notes/decomposition_progressive4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6created: 2026-08-117status: draft8---910# Model Decomposition (§4.5) & Progressive / Approximate Computation (§4.6)1112Reading notes for charter sections 4.5 and 4.6. Scope: can pretrained weights be13re-represented as `base + residual`, `shared component + per-layer correction`, or14`low-rank + sparse/quantized residual` — and can computation itself be made15progressive, so that quality scales with bytes loaded rather than being fixed at16compile time. All quality numbers below are taken from the cited papers, not17reproduced locally. Access date for all sources: 2026-08-11.1819---2021## 1. Landscape2223The literature splits into four families that almost never talk to each other:24251. **Static decomposition** (SVD family, structural slicing, Kronecker/tensor-train,26   codebook VQ): transform the checkpoint once, into a *fixed-size* smaller27   representation. State of the art is good at 3–4 bits/param equivalent, usable at28   2 bits, and degrades sharply below that. None of these change the *runtime*29   contract: every byte of the compressed model is still read for every token.30312. **Sharing/delta decomposition** (cross-layer sharing, DeltaLLM, delta compression32   of finetunes): exploit redundancy *between* matrices — between layers of one33   model, or between a base model and its finetunes. The measured compressibility34   here (1-bit deltas that are near-lossless) is the strongest published evidence35   that large fractions of transformer weight information are redundant relative to36   a reference.37383. **Progressive / nested representations** (Any-Precision LLM, Matryoshka39   Quantization, BitStack, recurrent residual quantization, Matryoshka-style40   additive codebooks): one stored artifact yields *many* operating points; lower41   precision is a strict prefix/subset of higher precision. This is the family42   closest to the localvm-research thesis, and it is young (2024–2026).43444. **Adaptive computation** (early exit, dynamic depth, adaptive halting,45   approximate matmul, mixed-precision iterative refinement, error-bounded lossy46   compression): vary the *amount of computation or precision per input*, sometimes47   with provable error control. Almost all of it varies **depth** (skip layers) or48   **tokens** (route around blocks); essentially none of it varies **how much of49   each weight matrix is materialized** per token.5051The gap at the intersection: a representation where the runtime decides, per layer52and per token (or per short window), *how many bytes of a nested weight encoding to53read*, with the rest resident only on SSD. Family 3 provides the encoding; family 454provides the control policy; families 1–2 provide the redundancy evidence. No55published system combines them (Phase 11 must re-verify this before any claim).5657---5859## 2. Techniques6061### 2.1 SVD-family low-rank factorization (SVD, FWSVD, ASVD, SVD-LLM)6263- **Mechanism.** Replace `W (m×n)` with `U_k Σ_k V_kᵀ` (rank k), stored as two thin64  matrices. FWSVD weights the factorization by Fisher information of rows;65  ASVD scales by activation statistics; SVD-LLM adds truncation-aware data66  whitening (direct map from singular values to compression loss) plus a67  sequential parameter update after truncation.68- **Compression / bandwidth.** Parameter ratio = k(m+n)/(mn). Bandwidth savings are69  proportional and *dense* — low-rank GEMMs are two ordinary GEMMs, no exotic70  kernels needed.71- **Quality (from papers).** Vanilla SVD collapses quickly: at 40–60% compression,72  LLaMA-7B perplexities for SVD/FWSVD/ASVD reach the hundreds to tens of thousands.73  SVD-LLM keeps perplexity finite (e.g., ~13.1 at 40%, and 7.73 vs ASVD's 11.14 at74  20% compression on LLaMA-7B), a >99% perplexity reduction vs prior SVD baselines75  — but average downstream accuracy still drops substantially at ≥40%76  (0.33–0.41 averages vs uncompressed ~0.6+). Consistent picture: **weight matrices77  are not globally low-rank; only ~20–30% rank reduction is cheap.**78- **Calibration.** All modern variants need a small calibration set (whitening /79  activation stats); no retraining for SVD-LLM's base version.80- **Apple Silicon.** Excellent: factors are plain dense matmuls, trivially81  expressible in MLX; can be combined with MLX affine quantization.82- **Limitation.** Fixed rank chosen at compile time; quality cliff past moderate83  ratios; ignores that the residual `W − U_kΣ_kV_kᵀ` still contains most spectral84  energy in LLMs (singular values decay slowly).85- **Extension opportunity.** Use the low-rank part as a *resident hot path* and keep86  the residual on SSD, loaded on demand — none of these papers store the residual87  at all.8889### 2.2 Structural decomposition and rank-reduction as intervention (SliceGPT, LASER)9091- **SliceGPT** (ICLR 2024): applies orthogonal rotations (PCA of activations,92  exploiting computational invariance of pre-norm transformers) then deletes rows93  and columns, shrinking the embedding dimension. Removes up to 25% of parameters94  of Llama-2 70B / OPT-66B while keeping 99% zero-shot performance (90% for Phi-2);95  dense smaller matrices → real speedups (up to 1.55× throughput), no sparse96  kernels needed. Calibration only. Maps cleanly to MLX.97- **LASER** (ICLR 2024): replacing *selected* weight matrices (mostly later-layer98  MLPs) with low-rank approximations **improves** task accuracy, at times by up to99  30 percentage points, with no training. This is an intervention result, not a100  compression system, but it is first-order evidence that higher-order weight101  components of specific matrices are noise-like for some tasks.102- **Limitation / opportunity.** Both are static. LASER's finding suggests per-matrix103  rank sensitivity is highly non-uniform — exactly the profile a compile-time104  analyzer (charter Phase 10 "compilation stage") should measure per layer.105106### 2.3 Low-rank + quantized residual (CALDERA; randomized LPLR)107108- **Mechanism.** CALDERA (NeurIPS 2024): `W ≈ Q + LR` where `Q` is a full-rank but109  aggressively quantized backbone and `L,R` low-rank factors (also quantized,110  possibly at higher precision), obtained by calibration-aware alternating111  minimization with approximation-error bounds. Targets the 2–2.5 bit/param regime.112  Precursor: randomized low-rank + low-precision factorization (NeurIPS 2023) with113  explicit error bounds vs rank and bit-budget.114- **Relevance.** This is literally the `base + residual` shape asked for in §4.5,115  with the roles inverted (quantized full-rank base + low-rank correction). The116  decomposition is *additive*, so the terms can in principle be loaded and applied117  independently — CALDERA never exploits that at runtime.118- **Apple Silicon.** `LR` is dense GEMM (fine); `Q` uses QuIP#-style lattice119  codebooks in the reference implementation (CUDA kernels; Metal port nontrivial).120- **Limitation.** Single fixed operating point; joint optimization couples the121  terms, so dropping one term is not quality-graceful by construction.122- **Extension.** Re-derive the decomposition with a *progressivity constraint*123  (base alone must be usable) — cf. §2.8.124125### 2.4 Codebook / vector quantization of weights (AQLM, QuIP#, GPTVQ, VPTQ)126127- **Mechanism.** Represent weight groups as sums of codebook entries: AQLM =128  additive multi-codebook quantization (from the retrieval literature, i.e.129  product/additive quantization à la Babenko) optimized against calibration130  activations; QuIP# = randomized Hadamard incoherence + E8 lattice codebooks +131  finetuning; GPTVQ, VPTQ = vector PTQ variants ("blessing of dimensionality").132- **Quality.** Current SOTA at 2–3 bits/param. AQLM at ~4 bits: Llama-2 ppl 3.57 vs133  FP 3.46 (Wiki2, 70B-scale table in paper); AQLM repo reports a ~1-bit Llama-2-7B134  (1×8-bit codebook) at WikiText-2 ppl 7.85 (2025 update). QuIP# made 2-bit135  "viable" for the first time. Additive codebooks are *inherently residual*: each136  additional codebook refines the previous sum.137- **Calibration.** Heavy (hours of optimization; QuIP#/AQLM benefit from finetuning).138- **Apple Silicon.** Weakest point: efficient decode kernels are CUDA-only today;139  lattice decodes and LUT-heavy kernels need custom Metal work. MLX ships only140  affine (2/3/4/5/6/8-bit, group 32/64/128) and mxfp4/mxfp8/nvfp4 modes.141- **Limitation.** Fixed bit-width at compile time; decoding cost nontrivial.142- **Extension.** Drop-by-Drop / Matryoshka-supervised additive codebooks (§2.8)143  show the additive structure can be made *ordered* so codebooks can be dropped at144  runtime — the natural bridge to paging.145146### 2.5 Cross-layer parameter sharing (ALBERT, Subformer, Relaxed Recursive Transformers, Basis Sharing, DeltaLLM, ResidualTransformer)147148- **Mechanism.** Reuse one block of weights across layers. ALBERT (all layers149  share), Subformer (sandwich sharing), Universal Transformer (recurrence) —150  train-from-scratch results. The post-training versions matter more here:151  - **Relaxed Recursive Transformers** (DeepMind, ICLR 2025): convert an existing152    LLM into a model that loops a small block of layers, "relaxed" by per-layer153    LoRA modules initialized via truncated SVD of the layer-vs-shared-weight154    difference. Recursive models converted from 2×-larger models can outperform155    same-size pretrained models; with distillation they approach the original.156  - **Basis Sharing** (ICLR 2025): express weights of *different layers* as157    combinations of a shared set of SVD-derived basis vectors + per-layer158    coefficients; outperforms SVD-LLM at 20–50% compression, calibration-only.159  - **DeltaLLM** (2025): share weights between adjacent transformer blocks and add160    low-rank per-layer deltas; ~30–40M tokens of light training; 12% parameter161    reduction retaining ~90% performance on Llama/Phi; DeltaPhi 2.9B (24%162    reduction) matches a finetuned SlicedPhi 3.3B. ResidualTransformer (ICASSP163    2024) is the same idea for speech models.164- **Compression / bandwidth.** This is the interesting part for us: a shared block165  resident in RAM amortizes across layers; per-layer deltas are small. A "one base166  layer + N cheap diffs" model turns per-layer weight traffic into per-layer167  *delta* traffic.168- **Limitation.** Post-training conversion still needs some uptraining (RRT,169  DeltaLLM); pure zero-shot layer tying degrades badly. Compression ratios so far170  are modest (12–25%), far from the 10× regime.171- **Extension.** Nobody treats the shared block as a *cache-resident core* and the172  deltas as *SSD-resident pages*. Also untested: sharing + progressive delta173  precision (delta rank/bits as a knob per layer).174175### 2.6 Delta compression of finetunes (BitDelta, DeltaZip, Delta-CoMe)176177- **Mechanism.** Decompose a finetuned model as `base + Δ` and compress Δ:178  BitDelta quantizes Δ to **1 bit** (sign + per-matrix scale, scales distilled in179  minutes); DeltaZip (EuroSys 2025) uses GPTQ-style compression of Δ (~10×) inside180  a multi-tenant serving system; Delta-CoMe (NeurIPS 2024) allocates mixed181  precision to Δ's singular vectors by singular-value magnitude (near-lossless at182  ~1-bit average, and unlike BitDelta it also holds up on math/code finetunes).183- **Quality.** BitDelta: minimal degradation across Llama-2/Mistral/MPT up to 70B;184  >10× memory reduction for multi-model serving. BitDelta's own ablation is185  notable for §4.6: applying BitDelta *successively* (compress, treat result as new186  base, compress the new delta…) yields an increasingly granular stack of 1-bit187  masks whose quality *approaches the original monotonically* — an accidental188  progressive code.189- **Relevance as evidence.** Finetuning information ≈ 1 bit/param. This does not190  directly compress a base model, but it proves that "model = reference +191  extremely compressible correction" is a real structure in modern LLM weight192  space, and it motivates trying the same decomposition *within* one model193  (layer_i = layer_j + cheap delta; model = quantized self + cheap residual).194- **Apple Silicon.** Sign matrices + scale are trivially Metal-friendly (1-bit195  masks decode to ±scale; MLX has no built-in kernel but the op is simple).196197### 2.7 Kronecker and tensor-network decompositions (KnGPT2, TensorGPT, tensor trains)198199- **Mechanism.** `W ≈ A ⊗ B` (Kronecker; nearest-Kronecker via rank-1 SVD of200  reshaped W) or tensor-train factorization of reshaped weights/embeddings.201- **Quality.** Results are only convincing at GPT-2/BERT scale with mandatory202  retraining (KnGPT2, ACL 2022). TensorGPT compresses *embedding layers* 2×–65×203  training-free on GPT-2-class models, but embeddings are a small fraction of a204  modern LLM. No competitive 7B+ results without heavy retraining found.205- **Verdict for us.** Low priority: high implementation cost, weak post-training206  evidence at scale, and Kronecker-structured matmul kernels for Metal would be207  bespoke. Worth keeping only as a candidate basis for *shared dictionaries*.208209### 2.8 Progressive / nested weight representations (Any-Precision LLM, MatQuant, BitStack, RRQ, Drop-by-Drop)210211The family that matters most for localvm-research.212213- **Any-Precision LLM** (ICML 2024 oral): stores an n-bit (8-bit) "parent" model214  such that every k-bit child (3≤k<8) is obtained by taking the **most significant215  bits** — bit-plane overlay. Built post-training by "incremental upscaling" from a216  3-bit seed (< 1 minute for 7B after seed quantization); ships a specialized217  (GPU) engine with bit-plane-aware memory layout. Memory: supporting {3..8}-bit218  Llama-2-7B costs 8.4 GB vs 29.9 GB for separate models (3.56×). Each bit-width219  matches SOTA quality for that width.220- **Matryoshka Quantization** (MatQuant, DeepMind, ICLR 2025 oral): co-trains one221  int8 quantized model whose int4/int2 slices (MSBs) are all optimized jointly;222  int2 slices become up to ~10% more accurate than dedicated int2 QAT/OmniQuant —223  an int2-FFN Gemma-2 9B beats an int8-FFN Gemma-2 2B. Also allows layer-wise224  mix'n'match of precisions at inference. Requires QAT-style training.225- **BitStack** (ICLR 2025): training-free. Iterative significance-weighted226  decomposition produces ~1-bit-per-parameter **residual blocks**; blocks are227  sorted (universally, across the whole model, by importance) and stacked in228  storage as transmission units; the runtime loads as many blocks as current229  memory allows → **megabyte-level tradeoff between resident size and quality**,230  matching or beating GPTQ/AWQ at extreme ratios. This is the closest existing231  system to "quality scales with bytes loaded." Known weakness (noted in follow-up232  work, e.g. the AMQ paper): on-the-fly weight *reconstruction from residual233  blocks slows inference notably*.234- **Recurrent Residual Quantization** (RRQ, arXiv 2608.04048, 2026): calibration-235  free, additive stage-wise scheme — 2-bit RTN base + successive 2-bit RTN236  residual corrections gives 4/6/8-bit operating points from one package; the237  whole multi-precision package for Qwen3-8B builds in ~1,293 s (3.3× faster than238  MatGPTQ-style joint optimization).239- **Drop-by-Drop additive codebooks** (arXiv 2606.12876, 2026): AQLM-style multi-240  codebook quantization with Matryoshka supervision so codebooks are ordered241  coarse→fine and can be dropped at inference for progressive compression.242- **Apple Silicon feasibility.** Bit-plane and residual-stage layouts are exactly243  the kind of thing unified memory + mmap should be good at: each precision level244  is a separate contiguous region; upgrading precision = reading another region,245  not rewriting the resident one. No published Metal/MLX implementation of any of246  these exists (all engines are CUDA); MLX's affine quant kernels (2–8 bit) could247  serve stages if each stage is expressed as an affine-quantized tensor.248- **Common limitation.** All of them select the operating point **statically**249  (per deployment, per memory budget). None selects precision per token; none ties250  the residual stages to storage paging; none reports bytes-read-per-token.251252### 2.9 Early exit and dynamic depth (ACT, PonderNet, MSDNet, Depth-Adaptive Transformer, CALM, LayerSkip, Mixture-of-Depths)253254- **Mechanism lineage.** ACT (Graves 2016): learned halting for RNN steps.255  PonderNet (2021): stabilized probabilistic halting. MSDNet (ICLR 2018): anytime256  prediction with multi-scale features — the canonical "anytime NN". Depth-257  Adaptive Transformer (ICLR 2020): per-token decoder depth. CALM (NeurIPS 2022):258  confidence-gated early exit for LM generation with *sequence-level calibrated259  guarantees* (up to ~3× compute reduction, provably maintaining quality;260  addresses missing-KV problem of exited tokens). LayerSkip (Meta, 2024): layer261  dropout + shared early-exit head during training, then **self-speculative262  decoding** — early layers draft, remaining layers verify — 1.34–2.16× speedup263  with exact final quality. Mixture-of-Depths (2024): learned top-k token routing264  per block under a static compute budget.265- **Bandwidth reality check.** Early exit saves *depth* — and therefore also the266  weight bytes of skipped layers for that token — but batch dynamics and KV267  bookkeeping erode the savings; and for us the key limit is that exit decisions268  gate *whole layers*, the coarsest possible granularity.269- **Retraining.** CALM/LayerSkip/MoD all need training or finetuning with exit270  losses; nothing here is drop-in post-training on a frozen checkpoint (LayerSkip271  ships finetuned checkpoints; naive early exit on frozen models is poor).272- **Apple Silicon.** Conceptually trivial to port (it is control flow, not273  kernels); single-request local decoding on a Mac is actually the *friendly* case274  (no batch synchronization problem).275- **Extension opportunity.** LayerSkip's draft-then-verify structure is276  depth-based self-speculation. The unexplored dual: **precision-based277  self-speculation** — draft with a resident low-bit base (prefix of a nested278  representation, §2.8), verify/refine with residual planes only when the draft's279  top-1 margin is small. Verification reads extra bytes *only on demand*. (Charter280  Experiments D and G test exactly the preconditions.)281282### 2.10 Approximate matrix multiplication (Drineas–Kannan–Mahoney, Bolt, MADDNESS)283284- **Mechanism.** (a) Randomized sampling: sample columns/rows with length-squared285  probabilities → unbiased estimate of `AB` with Frobenius error `O(‖A‖‖B‖/√c)`286  (DKM, SIAM J. Comput. 2006; foundation of RandNLA). (b) Learned LUT methods:287  Bolt, MADDNESS (ICML 2021) — replace one operand's inner products with learned288  hash-bucket lookups; up to 10× better speed-quality than prior AMM on small289  matrices, ~100× vs exact in the best cases.290- **Reality for LLMs.** MADDNESS-class methods shine when one matrix is fixed and291  *tall-thin* regimes apply (classifier layers, kernels); accuracy at transformer292  scale is unproven, and LUT-gather-heavy inner loops are a poor match for GPU293  matmul pipelines (they beat CPUs, not tensor cores). Sampling-based AMM gives294  clean error bounds but errors are relative to matrix norms — too loose to295  certify token decisions directly.296- **Value to us.** Not as a drop-in kernel, but as the theory toolbox for297  **partial GEMM with error bars** (Experiment E): length-squared/leverage298  sampling tells us *which blocks matter most* and what error skipping the rest299  costs — i.e., a principled block-ordering for progressive evaluation.300301### 2.11 Adaptive-precision numerical computing (Wilkinson iterative refinement → GMRES-IR, five-precision IR)302303- **Mechanism.** Solve `Ax=b` with an LU factorization computed in *low* precision304  (cheap, fast), then iteratively refine: compute residual in high precision, solve305  a correction system (possibly by GMRES preconditioned with the low-precision306  factors), update. Carson & Higham (SIAM SISC 2018) formalized three-precision307  IR; Amestoy et al. extended to five precisions; NVIDIA/Dongarra demonstrated308  FP16-tensor-core factorizations refined to FP64 accuracy at ~4× speed.309- **Why it matters here.** This is the *canonical proof* in numerical computing310  that "cheap approximate operator + residual-driven correction loop" recovers311  full accuracy while doing most work at low precision. The transformer analogue —312  run layers with a low-bit base, monitor a residual/confidence signal, apply313  stored higher-precision corrections only when needed — is structurally identical314  and appears untried for *weights* (speculative decoding is the analogue for315  *tokens*).316- **Caveat.** IR has a convergence theory because `A` is the exact operator and317  the residual is exactly computable; in an LLM the "exact" layer output is not318  available without loading the full weights. The honest transferable idea is319  *correction-on-demand plus a cheap instability detector* (logit margins, §4.10320  of the charter), not certified refinement.321322### 2.12 Error-bounded lossy compression from HPC (ZFP, SZ)323324- **Mechanism.** ZFP (Lindstrom, TVCG 2014): fixed-rate or fixed-accuracy block325  transform coding of floating-point arrays, with *published round-off error326  analysis* (SIAM 2019) and random-access decode of 4^d blocks. SZ (Di & Cappello,327  IPDPS 2016): prediction + error-controlled quantization with strict pointwise328  error bounds; typically higher ratios than ZFP at equal bounds on many datasets.329- **Relevance.** These are mature, *error-budgeted*, block-random-access codecs330  for float arrays — exactly the engineering shape a weight-paging store needs331  (bounded per-block reconstruction error → feeds §4.9 perturbation analysis;332  block random access → mmap-friendly pages). ZFP's fixed-rate mode gives333  predictable page sizes. Neither has been evaluated as an LLM weight format334  (weights are not smooth fields, so their predictors may underperform; needs335  Experiment-H-style measurement).336- **Apple Silicon.** Both are C/C++ libraries that build on arm64; decode337  throughput vs Apple NVMe read speed is the number to measure.338339---340341## 3. Evidence of exploitable redundancy in pretrained transformers342343The strongest *measured* facts found, ordered by how directly they support a344base+residual execution model:3453461. **Finetune deltas carry ≈1 bit/param of information.** BitDelta quantizes the347   full delta of 7B–70B finetunes to 1 bit with minimal degradation; Delta-CoMe is348   near-lossless at ~1-bit average even for math/code finetunes. GPT-Zip/DeltaZip349   independently report ~10× delta compressibility. → Weight space has directions350   that are dramatically cheaper to encode *relative to a reference*.3513522. **Adjacent layers are highly similar / near-linear.** "Your Transformer is353   Secretly Linear" (ACL 2024) measures Procrustes linearity ≈0.99 between354   consecutive decoder layer embeddings across GPT/LLaMA/OPT/BLOOM, and shows some355   of the most-linear blocks can be removed or replaced by linear approximations356   with little loss. ShortGPT's Block Influence metric (cosine similarity between357   layer input and output) finds many layers barely transform the hidden state;358   removing them ("more redundant than you expect") costs little on benchmarks.359   "The Unreasonable Ineffectiveness of the Deeper Layers" (ICLR 2025) prunes360   large contiguous blocks of *deep* layers (selected by representational361   similarity) with minimal QA degradation after light QLoRA healing.3623633. **Layer weights are compressible against each other.** DeltaLLM: adjacent-block364   sharing + low-rank deltas retains ~90% performance at 12% reduction with only365   30–40M tokens of training. Basis Sharing: one shared SVD basis serves multiple366   layers' weights with per-layer coefficients and beats per-layer SVD-LLM at367   20–50% ratios. Relaxed Recursive Transformers: a looped shared block +368   SVD-initialized per-layer LoRA recovers most of the original model — i.e., much369   of a layer's identity is "shared trunk + small correction."3703714. **Selective rank reduction can even help.** LASER: replacing selected later-MLP372   matrices by low-rank approximations improves accuracy (up to +30 points on some373   tasks) — high-order components of specific matrices are noise-like.374   Complementary: SliceGPT removes 25% of parameters via activation-PCA rotation375   with 99% zero-shot retention on Llama-2 70B / OPT-66B.3763775. **Task adaptation is intrinsically low-dimensional.** Aghajanyan et al. (2020):378   RoBERTa-scale models can be finetuned to ~90% of full performance inside a379   random subspace of only ~hundreds of dimensions; pretraining *reduces* intrinsic380   dimension. → capability deltas, not just finetune deltas, are low-dimensional.3813826. **But global low-rankness of weights is a myth.** The SVD-family results (§2.1)383   consistently show steep quality loss past ~25–40% rank compression ("features384   are low-rank, weights are not"). Redundancy is *structured* (cross-layer,385   relative-to-reference, task-conditional) rather than uniform spectral decay.3863877. **Inference is bandwidth-bound, so redundancy = latency.** Every token reads388   every weight byte; on H100-class hardware compute outruns memory delivery by389   ~600× (Cloudflare "Unweight" engineering measurement). On Apple Silicon the390   ratio is smaller but the regime is the same — any byte not read per token is391   ~proportional latency, which is why decoupling "stored bytes" from "read bytes"392   (charter §2) is the right objective.393394---395396## 4. Progressive encodings from other fields397398Transferable design patterns, from oldest to newest:399400- **Embedded wavelet coding (EZW 1993, SPIHT 1996, JPEG2000/EBCOT).** Coefficients401  are transmitted in *significance order*, bit-plane by bit-plane; the bitstream402  can be truncated at any byte and decodes to the best possible image for that403  byte count ("each new bit conveys the maximum information"). JPEG2000's EBCOT404  adds independently coded blocks with optimized truncation points — i.e.,405  rate-distortion-optimal *per-block* truncation. **Transfer:** encode weight406  blocks as significance-ordered bit-planes/residual stages; "bytes loaded per407  matrix" becomes a continuous quality knob, and per-block truncation points can408  be optimized against layer sensitivity (Experiment F) instead of PSNR. BitStack409  (§2.8) is an unwitting rediscovery of this with residual SVD blocks; nobody has410  connected it to the mature R-D-optimal truncation machinery.411412- **Progressive meshes (Hoppe, SIGGRAPH 1996) and Nanite (UE5).** A mesh is stored413  as a coarse base + an ordered stream of refinements (vertex splits), giving414  lossless, continuous LoD, streaming, and *selective refinement* (refine only415  where the camera looks). Nanite industrializes this: fixed-size clusters in a416  hierarchical DAG, **streamed on demand so only visible detail resides in417  memory**, with LoD chosen per-cluster per-frame at ~pixel-error tolerance.418  **Transfer:** this is the exact architecture shape for weight paging — fixed-size419  weight "clusters" at multiple precisions, a residency set updated per token/420  window by a cheap importance signal (attention/activation statistics as the421  "camera"), error tolerance expressed in logit margin instead of pixels.422423- **Approximate query processing (Online Aggregation, SIGMOD 1997; BlinkDB,424  EuroSys 2013).** Answer first, refine continuously, with statistical error bars;425  BlinkDB answers queries over 17 TB in <2 s within 2–10% error by choosing among426  precomputed stratified samples given a per-query time or error budget.427  **Transfer:** the *interface* idea — inference under an explicit428  (latency | error) budget, where the runtime chooses how much of the model to429  consult and can report confidence; and the *offline* idea — precompute multiple430  "samples" (precision profiles) of the model optimized for expected workloads.431432- **Mixed-precision iterative refinement (Wilkinson 1963 → Carson–Higham 2018).**433  See §2.11: do the O(n³) work in cheap precision once, recover accuracy with434  cheap corrective iterations. The pattern "expensive operator approximated +435  residual-driven correction + convergence monitor" is the numerical-analysis436  ancestor of any progressive-weight-refinement runtime.437438- **Error-bounded scientific compression (ZFP/SZ).** See §2.12: block random439  access + guaranteed per-element error bounds is the storage-format discipline a440  weight pager should adopt (bounded weight perturbation → bounded logit441  perturbation via layer Lipschitz estimates, rather than hoping).442443---444445## 5. Relevance to localvm-research446447**Could a progressive base+residual representation let quality scale with bytes448loaded?** The evidence says the ingredients all exist and individually work:449450- Nested/progressive weight codes exist and are near-SOTA at each operating point451  (Any-Precision LLM, MatQuant, BitStack, RRQ). BitStack already demonstrates452  monotone quality-vs-resident-megabytes on 7B–70B models, training-free.453- Redundancy is real and structured (§3): a 2-bit-class base plausibly carries454  most behavior, and corrections are cheap *relative to* the base (BitDelta's455  iterated 1-bit masks converge to the original).456- Control policies with quality guarantees exist for the depth dimension (CALM's457  calibrated exits; LayerSkip's exact self-speculative verification).458- The systems patterns for demand-paged, error-budgeted, progressively refined459  data are mature in other fields (EBCOT truncation, Nanite residency, BlinkDB460  budgets, GMRES-IR refinement).461462**What has NOT been tried (candidate gaps for `research_gaps.md`):**4634641. **Token-/layer-conditional residual loading.** Every progressive system picks465   its operating point statically per deployment. No published system decides *per466   token* (or per small window) and *per layer* how many residual stages to apply,467   despite Experiment-B/D-style predictability being the obvious enabler. The468   marriage BitStack × CALM does not exist.4692. **Precision-based self-speculation on a nested code.** Draft with the resident470   low-bit prefix; verify/refine with SSD-resident residual planes only when the471   top-1 margin is small (LayerSkip's mechanism, transposed from depth to472   precision, with draft and verifier *sharing the same bytes*). Needs a Phase 11473   novelty sweep (search terms: quantized self-speculation, precision cascade474   decoding, progressive dequantization inference).4753. **Bytes-read-per-token as the optimized objective.** None of the §2.8 papers476   measures SSD/DRAM traffic per generated token; they measure resident size. On a477   48 GB M5 Max with a fast NVMe, the interesting regime is: base resident478   (~2 bits/param), residual planes mmap'd, and a policy that keeps *average*479   bytes/token far below checkpoint size. This is measurable with our480   instrumentation plan (fs_usage, vm_stat) and no one has published it.4814. **R-D-optimal truncation for weights.** Port EBCOT-style per-block optimized482   truncation to weight blocks, with distortion measured as calibration-set logit483   KL (not MSE), producing a *layer-sensitivity-aware* progressive layout at484   compile time (fits the charter's "compilation stage" exactly).4855. **Shared-basis trunk as the resident core.** Basis Sharing / RRT / DeltaLLM486   suggest "shared trunk resident + per-layer deltas paged". Unexplored as a487   memory-hierarchy assignment rather than a compression ratio.488489**Apple Silicon specifics.** Low-rank factors, sign-mask deltas, and affine-490quantized stages (2–8 bit, group 32/64/128) map directly onto today's MLX kernels;491bit-plane overlays and additive codebooks would need custom Metal kernels (all492published engines are CUDA). Unified memory removes the CPU↔GPU copy that makes493progressive loading painful on discrete GPUs: a residual plane read from NVMe into494a mapped buffer is immediately GPU-visible. The BitStack-reported reconstruction495slowdown is the main engineering risk — reconstruction must be fused into the496matmul (dequant-in-kernel, as MLX already does for affine quant) rather than497materialized. These claims about MLX/Metal feasibility are assessments to be498validated in Experiments D/E/H, not established facts.499500**Failure modes to respect** (charter §17): if per-token stage selection turns out501to need near-all stages for acceptable quality (working set ≈ whole model), or if502random 4–64 KB residual reads on Apple NVMe are too slow/thermally throttled503(Experiment H), the progressive-paging premise dies; the fallback value of this504literature is then "best static compressed format for MLX," which is already505well-served by existing work.506507---508509## Sources510511- SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression (ICLR 2025) — https://arxiv.org/html/2403.07378v3 (accessed 2026-08-11)512- SVD-LLM (ICLR 2025 proceedings abstract) — https://proceedings.iclr.cc/paper_files/paper/2025/hash/3104e1ab39875cf54fe1eb4473e7c5a1-Abstract-Conference.html (accessed 2026-08-11)513- SVD-LLM GitHub (AIoT-MLSys-Lab) — https://github.com/AIoT-MLSys-Lab/SVD-LLM (accessed 2026-08-11)514- ASVD: Activation-aware Singular Value Decomposition for Compressing LLMs — https://arxiv.org/abs/2312.05821 (accessed 2026-08-11)515- Language model compression with weighted low-rank factorization (FWSVD, ICLR 2022) — https://arxiv.org/abs/2207.00112 (accessed 2026-08-11)516- The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction (LASER, ICLR 2024) — https://arxiv.org/abs/2312.13558 (accessed 2026-08-11)517- LASER project page — https://pratyushasharma.github.io/laser (accessed 2026-08-11)518- SliceGPT: Compress Large Language Models by Deleting Rows and Columns (ICLR 2024) — https://arxiv.org/abs/2401.15024 (accessed 2026-08-11)519- Compressing Large Language Models using Low Rank and Low Precision Decomposition (CALDERA, NeurIPS 2024) — https://arxiv.org/abs/2405.18886 (accessed 2026-08-11)520- CALDERA GitHub (pilancilab) — https://github.com/pilancilab/caldera (accessed 2026-08-11)521- Matrix Compression via Randomized Low Rank and Low Precision Factorization (NeurIPS 2023) — https://neurips.cc/virtual/2023/poster/70291 (accessed 2026-08-11)522- Extreme Compression of Large Language Models via Additive Quantization (AQLM) — https://arxiv.org/html/2401.06118v2 (accessed 2026-08-11)523- AQLM GitHub (incl. ~1-bit Llama-2-7B result) — https://github.com/vahe1994/AQLM (accessed 2026-08-11)524- QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (ICML 2024) — https://proceedings.mlr.press/v235/tseng24a.html (accessed 2026-08-11)525- GPTVQ: The Blessing of Dimensionality for LLM Quantization — https://arxiv.org/abs/2402.15319 (accessed 2026-08-11)526- VPTQ: Extreme Low-bit Vector Post-Training Quantization for LLMs (Microsoft Research) — https://www.microsoft.com/en-us/research/publication/vptq-extreme-low-bit-vector-post-training-quantization-for-large-language-models (accessed 2026-08-11)527- Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (ICLR 2025) — https://arxiv.org/html/2410.20672v1 (accessed 2026-08-11)528- Subformer: Exploring Weight Sharing for Parameter Efficiency (Findings of EMNLP 2021) — https://aclanthology.org/2021.findings-emnlp.344.pdf (accessed 2026-08-11)529- Basis Sharing: Cross-Layer Parameter Sharing for LLM Compression (ICLR 2025) — https://arxiv.org/abs/2410.03765 (accessed 2026-08-11)530- Basis Sharing (ICLR 2025 proceedings PDF) — https://proceedings.iclr.cc/paper_files/paper/2025/file/238c98450b1d9e8055f94d22f303bb57-Paper-Conference.pdf (accessed 2026-08-11)531- DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weights — https://arxiv.org/abs/2501.18596 (accessed 2026-08-11)532- ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers (ICASSP 2024) — https://arxiv.org/abs/2310.02489 (accessed 2026-08-11)533- BitDelta: Your Fine-Tune May Only Be Worth One Bit (NeurIPS 2024) — https://arxiv.org/html/2402.10193v3 (accessed 2026-08-11)534- BitDelta NeurIPS poster page — https://neurips.cc/virtual/2024/poster/94736 (accessed 2026-08-11)535- DeltaZip: Compression for Foundation Models (EuroSys 2025; repo lists delta-compression literature) — https://github.com/eth-easl/deltazip (accessed 2026-08-11)536- Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for LLMs (NeurIPS 2024) — https://arxiv.org/abs/2406.08903 (accessed 2026-08-11)537- Kronecker Decomposition for GPT Compression (KnGPT2, ACL 2022) — https://aclanthology.org/2022.acl-short.24.pdf (accessed 2026-08-11)538- TensorGPT: Efficient Compression of LLMs based on Tensor-Train Decomposition — https://arxiv.org/html/2307.00526v2 (accessed 2026-08-11)539- BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments (ICLR 2025) — https://arxiv.org/abs/2410.23918 (accessed 2026-08-11)540- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 oral) — https://arxiv.org/pdf/2402.10517 (accessed 2026-08-11)541- Any-Precision LLM GitHub (SNU-ARC) — https://github.com/SNU-ARC/any-precision-llm (accessed 2026-08-11)542- Matryoshka Quantization (MatQuant, ICLR 2025 oral) — https://openreview.net/forum?id=phVWcUSGYP (accessed 2026-08-11)543- Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs — https://arxiv.org/abs/2608.04048 (accessed 2026-08-11)544- Multi-Bitwidth Quantization for LLMs Using Additive Codebooks (Drop-by-Drop) — https://arxiv.org/html/2606.12876v1 (accessed 2026-08-11)545- Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning — https://arxiv.org/abs/2012.13255 (accessed 2026-08-11)546- ShortGPT: Layers in Large Language Models are More Redundant Than You Expect — https://arxiv.org/html/2403.03853v1 (accessed 2026-08-11)547- The Unreasonable Ineffectiveness of the Deeper Layers (ICLR 2025) — https://arxiv.org/abs/2403.17887 (accessed 2026-08-11)548- Your Transformer is Secretly Linear (ACL 2024) — https://arxiv.org/abs/2405.12250 (accessed 2026-08-11)549- Confident Adaptive Language Modeling (CALM, NeurIPS 2022) — https://proceedings.neurips.cc/paper_files/paper/2022/hash/6fac9e316a4ae75ea244ddcef1982c71-Abstract-Conference.html (accessed 2026-08-11)550- Google Research blog: Accelerating text generation with CALM — https://research.google/blog/accelerating-text-generation-with-confident-adaptive-language-modeling-calm (accessed 2026-08-11)551- LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding — https://arxiv.org/html/2404.16710v1 (accessed 2026-08-11)552- Depth-Adaptive Transformer (ICLR 2020) — https://arxiv.org/abs/1910.10073 (accessed 2026-08-11)553- Adaptive Computation Time for Recurrent Neural Networks (Graves 2016) — https://arxiv.org/abs/1603.08983 (accessed 2026-08-11)554- PonderNet: Learning to Ponder — https://arxiv.org/abs/2107.05407 (accessed 2026-08-11)555- Multi-Scale Dense Networks for Resource Efficient Image Classification (MSDNet) — https://arxiv.org/abs/1703.09844 (accessed 2026-08-11)556- Mixture-of-Depths: Dynamically allocating compute in transformer-based language models — https://arxiv.org/abs/2404.02258 (accessed 2026-08-11)557- Multiplying Matrices Without Multiplying (MADDNESS, ICML 2021) — https://proceedings.mlr.press/v139/blalock21a/blalock21a.pdf (accessed 2026-08-11)558- Fast Monte Carlo Algorithms for Matrices I: Approximating Matrix Multiplication (Drineas, Kannan, Mahoney, SIAM J. Comput. 2006) — https://epubs.siam.org/doi/10.1137/S0097539704442684 (accessed 2026-08-11)559- Accelerating the Solution of Linear Systems by Iterative Refinement in Three Precisions (Carson & Higham, SIAM SISC 2018) — https://epubs.siam.org/doi/10.1137/17M1140819 (accessed 2026-08-11)560- Five-precision GMRES-based Iterative Refinement (Amestoy et al.) — https://eprints.maths.manchester.ac.uk/2852/1/paper.pdf (accessed 2026-08-11)561- What Is Iterative Refinement? (Nick Higham) — https://nhigham.com/2023/03/13/what-is-iterative-refinement (accessed 2026-08-11)562- zfp Compression Ratio and Quality (LLNL) — https://computing.llnl.gov/projects/zfp/zfp-compression-ratio-and-quality (accessed 2026-08-11)563- Error Analysis of ZFP Compression for Floating-Point Data (SIAM) — https://epubs.siam.org/doi/10.1137/18M1168832 (accessed 2026-08-11)564- Fast Error-bounded Lossy HPC Data Compression with SZ (Di & Cappello, IPDPS 2016) — https://www.mcs.anl.gov/papers/P5437-1115.pdf (accessed 2026-08-11)565- Embedded zerotrees of wavelet transforms (EZW) — https://en.wikipedia.org/wiki/Embedded_zerotrees_of_wavelet_transforms (accessed 2026-08-11)566- Wavelet and image compression: EZW / SPIHT / JPEG2000-EBCOT lecture notes (Cagnazzo, Télécom Paris) — https://perso.telecom-paristech.fr/tupin/ATHENS/COURSES/wavelet_athens_2012.pdf (accessed 2026-08-11)567- Progressive Meshes (Hoppe, SIGGRAPH 1996) — https://www.cs.jhu.edu/~misha/ReadingSeminar/Papers/Hoppe96.pdf (accessed 2026-08-11)568- Nanite Virtualized Geometry (Unreal Engine documentation) — https://dev.epicgames.com/documentation/unreal-engine/nanite-virtualized-geometry-in-unreal-engine?lang=en-US (accessed 2026-08-11)569- BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data (EuroSys 2013) — https://dl.acm.org/doi/10.1145/2465351.2465355 (accessed 2026-08-11)570- Readings in Database Systems (Red Book) ch. 8: Interactive Analytics — online aggregation & AQP context — http://www.redbook.io/ch8-interactive.html (accessed 2026-08-11)571- Unweight: how we compressed an LLM 22% without sacrificing quality (Cloudflare engineering, bandwidth-bound inference evidence) — https://blog.cloudflare.com/unweight-tensor-compression (accessed 2026-08-11)572- mlx.core.quantize documentation (supported modes, group sizes, bit widths) — https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.core.quantize.html (accessed 2026-08-11)573