spb/localvm-research Public License
Running LLMs larger than memory on a consumer Mac — falsification-driven research: margin-gated deferred refinement, out-of-core verification on Apple Silicon. TR-01 published.
Python 63.2%
JavaScript 23.5%
CSS 11.8%
Shell 0.9%
Makefile 0.5%
1---2project: localvm-research3document: README4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6created: 2026-08-117modified: 2026-08-128status: reviewed9---1011<div align="center">1213<img src="web/public/logo.svg" width="96" alt="localvm-research logo">1415# localvm-research1617**Running LLMs larger than memory on a consumer Mac**1819[](https://www.localvm.dev)20[](https://www.localvm.dev/about)21[](https://github.com/ml-explore/mlx)22[](https://www.localvm.dev/experiments)23[](https://www.localvm.dev/doc/research/bibliography.md)24[](https://www.localvm.dev/publications)25[](LICENSE)2627**Live research platform: [www.localvm.dev](https://www.localvm.dev)** — every hypothesis, result, figure, and failure, browsable.2829*Author: **Simon-Pierre Boucher** · [contact@spboucher.ai](mailto:contact@spboucher.ai)*3031</div>3233---3435## The question3637> Can an existing pretrained LLM whose memory requirements exceed a consumer38> Apple Silicon Mac be transformed **post-training** into an execution39> representation whose *instantaneous working set* is dramatically smaller40> than the full checkpoint — while preserving most of its capabilities?4142The project's central distinction, which everything here tries to decouple:4344```text45total model size ≠ resident model size ≠ bytes read per token ≠ parameters required for THIS token46```4748This is a falsification-driven research project run under a strict charter49([CLAUDE.md](CLAUDE.md)): hypotheses are registered with explicit kill criteria50*before* experiments run, no result is reported from an uncommitted tree, every51result file embeds a hardware manifest, and **negative results are kept and52published**. The full audit trail lives in the append-only53[research log](research/LOG.md).5455## Headline results5657All measured on the primary research machine — **Apple M5 Max, 6P+12E CPU,5840-core GPU, 48 GB unified memory, 2 TB AP2048Z NVMe, macOS 27.0**.5960| # | Result | Key numbers |61|---|--------|-------------|62| 1 | **SSD substrate envelope** (expH) | device ceiling ≈ **13.1 GB/s** (iostat-validated); 4 KiB QD1 random = **67 MB/s** → 1 MiB QD8 = **13.8 GB/s** (200× spread); saturated Metal GPU costs **< 5%** of SSD throughput |63| 2 | **Margins predict disagreement** (expG) | AUROC **0.85–0.98** (3/4/8-bit); domain-uniform; escalation for 99% agreement falls with scale: **36.6% → 22.6%** (1.7B → 8B, 4-bit) |64| 3 | **Naive 2-bit is dead** (expG) | affine 2-bit: **3% greedy agreement**, KL ≈ 11.9 — the 2–3-bit cliff, measured |65| 4 | **Progressive residual coding works** (expD) | hidden-state error **÷5 per stage**; two-tier margin policy: **97.6% agreement @ 35% escalation** |66| 5 | **Three escalation routes refuted** (expF/A/B) | layer sensitivity spread only **1.5×**; 95% FFN energy needs **77%** of 64-neuron blocks; 128-token working-set union = **99.9%** — dynamic sparsity paging does *not* transfer to dense SwiGLU models |67| 6 | **Quality–bytes continuum** (candidate, 1.7B) | verify-all output **statistically = resident-q8** (bf16 judge −0.1297 vs −0.1328) at **237 MB/token** streamed — **9× under the checkpoint**; τ knob: 150/211/237 MB/token ↔ 47%/83%/~100% of the q4→q8 gap |68| 7 | **Out-of-core at 32B** (candidate, scale) | q8-32B (34.8 GB) **cannot co-reside** with the q4 base on 48 GB — streamed layer-by-layer at **~11.6 GB/s**; **+0.28 nats** over pure-q4 (the only fitting config) at **3.7 GB/token** (9.4× under checkpoint), 1.69 tok/s |69| 8 | **Two macOS measurement traps** | `F_NOCACHE` does **not** evict resident pages (naïve "cold" benchmarks read RAM at 42–115 GB/s); Metal prefill vs incremental decode flips **1.56%/token** of a model's own greedy argmax — token-exact reproduction is not a coherent target |7071## The architecture — margin-gated deferred refinement7273Selected by the evidence (not by preference — see the74[candidate ranking](research/candidate_ranking.md) and three refutations above):75761. **Resident base** — the q4 quantization of the model lives in unified77 memory and generates greedily, recording each decision's **top-1 logit78 margin** (a free byproduct of the forward pass).792. **Deferral** — decisions with margin < τ are flagged; generation never blocks.803. **Amortized verification sweeps** — every W tokens, the *same checkpoint's*81 q8 variant teacher-forces the window in one pass through a persistent82 incremental verify-KV-cache. When q8 cannot co-reside (the target regime),83 it is **streamed layer-by-layer from SSD** (materialize → compute →84 re-lazify) — one *sequential* pass over the checkpoint per sweep, exactly85 the access pattern the measured SSD envelope rewards.864. **Rollback** — on the first checked flip the runtime trims both KV caches,87 accepts the q8 token, and resumes.8889**τ is a runtime knob that prices output quality in verification bytes** — a90property no static quantization format offers.9192Implementation: [`experiments/candidate_01/implementation/`](experiments/candidate_01/implementation/)93(`runtime.py` — generation/deferral/rollback loop; `streaming_verifier.py` —94per-layer weight streaming on unified memory).9596## The experimental campaign9798Every experiment has a preregistered `hypothesis.md` (seven-field scientific99block with falsification criterion) and a post-run `analysis.md`.100101| Experiment | Question | Verdict |102|------------|----------|---------|103| [expH — SSD feasibility](experiments/micro/expH_ssd_feasibility/) | Can the internal NVMe sustain weight-streaming rates? | ✅ **Confirmed** — with a strict ≥256 KiB / QD≥4 / sequential contract |104| [expG — decision stability](experiments/micro/expG_decision_stability/) | Does the cheap pass's margin predict disagreement? | ✅ **Confirmed** (AUROC 0.85–0.98, improves with scale) |105| [expD — progressive reconstruction](experiments/micro/expD_progressive_reconstruction/) | Do residual ladders converge? Does gating compose? | ✅ **Confirmed** (with an honest static-parity caveat) |106| [expF — error accumulation](experiments/micro/expF_error_accumulation/) | Is quantization damage concentrated in layers? | ❌ **Refuted** — diffuse (1.5× spread), repair is cooperative |107| [expA — weight concentration](experiments/micro/expA_weight_concentration/) | Is per-token FFN importance pageable at block granularity? | ❌ **Refuted** — neuron-level concentration exists but is scattered |108| [expB — token stability](experiments/micro/expB_token_stability/) | Is the important-block set temporally cacheable? | ❌ **Refuted** — union ≈ 100% within 128 tokens; Jaccard ≈ random null |109| [candidate_01 — 1.7B](experiments/candidate_01/) | Does the assembled runtime deliver a quality–bytes continuum? | ✅ q8-equivalent quality at 9× fewer bytes |110| [candidate_01 — 32B scale](experiments/candidate_01/benchmark_scale.py) | Does it work when the verifier physically cannot fit? | ✅ +0.28 nats over the only fitting alternative |111112The refutations are load-bearing: they killed the fashionable design (dynamic113sparsity paging à la DejaVu / LLM-in-a-flash) *on modern dense architectures*114with numbers, and left batch-amortized verification as the only surviving115route — which the prototype then validated.116117## Novelty, honestly stated118119An adversarial prior-art search ([novelty_check.md](research/novelty_check.md),12045 sources) found **every component anticipated** — precision-tiered121draft/verify (QSpec, Apple QuantSpec), resident-draft + flash-streamed target122(Lever 2026, CATS 2026), uncertainty-gated verification (U-HLM/UGSD, vs cloud),123window-amortized verification (SpecExec), MLX layer streaming (mlx-flash).124Four narrow claims survived: **(N1)** margin-gating of the *same model's*125higher-precision **weight-streaming I/O** on one machine; **(N2)**126bytes-streamed-per-token vs checkpoint size as the explicit objective, with τ127as the knob; **(N3)** the training-free same-checkpoint q4/q8 split128demonstrated where the verifier cannot reside; **(N4)** the129unified-memory/Metal instantiation. Attribution obligations are recorded and130the check should be re-run before any external submission.131132## Repository structure133134```text135localvm-research/136├── CLAUDE.md # research charter (methodology contract)137├── research/ # the scientific paper trail138│ ├── LOG.md # append-only research log (every decision)139│ ├── state_of_the_art.md # Phase 2 — technique taxonomy, overlap analysis140│ ├── research_gaps.md # Phase 3 — 24 falsifiable gaps (G01–G24)141│ ├── candidate_ranking.md # Phase 4 — 10-axis scoring, 4 candidates142│ ├── novelty_check.md # Phase 11 — adversarial prior-art verdict143│ ├── bibliography.md # 304 sources with URLs and access dates144│ └── notes/ # 5 Phase-1 literature theme notes145├── experiments/146│ ├── micro/expA…expH/ # preregistered micro-experiments147│ └── candidate_01/ # the prototype runtime + 1.7B/32B benchmarks148├── benchmarks/149│ ├── hardware_manifest.py # macOS-native fingerprint (embedded in results)150│ └── datasets/eval_prompts.json # 6-domain eval prompts (code/math/chat/FR/EN/reasoning)151├── results/ # raw JSON results (commit + config + seed + manifest)152├── src/localvm/ # library (quality/decision_stats.py, backends, …)153├── tools/ # check_headers.py, new_experiment.py, make_pub_figures.py154├── docs/publications/ # official technical reports (TR-01) + SVG figures155└── web/ # the public research platform (Express, deployed)156```157158## Reproduce it159160```bash161# 1. Environment (Apple Silicon Mac, macOS 14+)162python3 -m venv .venv163.venv/bin/pip install mlx mlx-lm numpy164165# 2. Fingerprint your machine (embedded in every result)166python3 benchmarks/hardware_manifest.py167168# 3. Substrate envelope (~15 min, writes results/expH_ssd_feasibility/)169cd experiments/micro/expH_ssd_feasibility && ../../../.venv/bin/python benchmark.py170171# 4. Decision stability (~10 min, downloads Qwen3-1.7B-bf16)172cd ../expG_decision_stability && ../../../.venv/bin/python benchmark.py173174# 5. The prototype (build quantized artifacts once, then run)175cd ../../candidate_01176../../.venv/bin/python benchmark.py --build177../../.venv/bin/python benchmark.py # 1.7B continuum178../../.venv/bin/python benchmark_scale.py # 32B out-of-core (downloads ~52 GB)179180# 6. Regenerate all publication figures from committed results181python3 tools/make_pub_figures.py182183# Quality gates184python3 tools/check_headers.py # mandatory header check (CI-style)185```186187Every benchmark writes `results/<experiment>/<UTC timestamp>/results.json`188with the full config and hardware manifest embedded.189190## Publications191192- **[TR-01 — Margin-Gated Deferred Refinement: Streaming Higher-Precision LLM193 Quality Than Fits in Memory on Consumer Apple Silicon](docs/publications/TR-01-margin-gated-deferred-refinement.md)**194 (August 2026) — the complete write-up with figures, prior-art attribution,195 and limitations. Also on the platform: [localvm.dev/publications](https://www.localvm.dev/publications).196197## Known limitations & roadmap198199Recorded in TR-01 §7 and the research log — the honest list: prototype speed200(1.69 tok/s at 32B) is below interactive; no resident-q5/q6 baseline at 32B201yet (they fit in 48 GB and may dominate at that scale — the architecture's202necessary regime is 70B-class, untested); the 8B judge is a proxy with no203error bars at n=12; no task-level benchmarks yet; one model family, one204machine. **Next:** resident q5/q6 baselines → task benchmarks (GSM8K-class) →205τ calibration on real quantized-artifact margins → 70B point → TR-02.206207## Author & citation208209**Simon-Pierre Boucher** — [contact@spboucher.ai](mailto:contact@spboucher.ai)210211```bibtex212@techreport{boucher2026localvm,213 author = {Boucher, Simon-Pierre},214 title = {Margin-Gated Deferred Refinement: Streaming Higher-Precision215 LLM Quality Than Fits in Memory on Consumer Apple Silicon},216 institution = {localvm-research},217 number = {TR-01},218 year = {2026},219 month = {August},220 url = {https://www.localvm.dev/publications}221}222```223224Citation metadata: [CITATION.cff](CITATION.cff).225226## License227228All rights reserved (research code) — © 2026 Simon-Pierre Boucher.229See [LICENSE](LICENSE). The research trail is public for reading and230verification at [www.localvm.dev](https://www.localvm.dev); reuse requires231written authorization.232