# localvm-research
**Running LLMs larger than memory on a consumer Mac**
[](https://www.localvm.dev)
[](https://www.localvm.dev/about)
[](https://github.com/ml-explore/mlx)
[](https://www.localvm.dev/experiments)
[](https://www.localvm.dev/doc/research/bibliography.md)
[](https://www.localvm.dev/publications)
[](LICENSE)
**Live research platform: [www.localvm.dev](https://www.localvm.dev)** — every hypothesis, result, figure, and failure, browsable.
*Author: **Simon-Pierre Boucher** · [contact@spboucher.ai](mailto:contact@spboucher.ai)*
---
## The question
> Can an existing pretrained LLM whose memory requirements exceed a consumer
> Apple Silicon Mac be transformed **post-training** into an execution
> representation whose *instantaneous working set* is dramatically smaller
> than the full checkpoint — while preserving most of its capabilities?
The project's central distinction, which everything here tries to decouple:
```text
total model size ≠ resident model size ≠ bytes read per token ≠ parameters required for THIS token
```
This is a falsification-driven research project run under a strict charter
([CLAUDE.md](CLAUDE.md)): hypotheses are registered with explicit kill criteria
*before* experiments run, no result is reported from an uncommitted tree, every
result file embeds a hardware manifest, and **negative results are kept and
published**. The full audit trail lives in the append-only
[research log](research/LOG.md).
## Headline results
All measured on the primary research machine — **Apple M5 Max, 6P+12E CPU,
40-core GPU, 48 GB unified memory, 2 TB AP2048Z NVMe, macOS 27.0**.
| # | Result | Key numbers |
|---|--------|-------------|
| 1 | **SSD substrate envelope** (expH) | device ceiling ≈ **13.1 GB/s** (iostat-validated); 4 KiB QD1 random = **67 MB/s** → 1 MiB QD8 = **13.8 GB/s** (200× spread); saturated Metal GPU costs **< 5%** of SSD throughput |
| 2 | **Margins predict disagreement** (expG) | AUROC **0.85–0.98** (3/4/8-bit); domain-uniform; escalation for 99% agreement falls with scale: **36.6% → 22.6%** (1.7B → 8B, 4-bit) |
| 3 | **Naive 2-bit is dead** (expG) | affine 2-bit: **3% greedy agreement**, KL ≈ 11.9 — the 2–3-bit cliff, measured |
| 4 | **Progressive residual coding works** (expD) | hidden-state error **÷5 per stage**; two-tier margin policy: **97.6% agreement @ 35% escalation** |
| 5 | **Three escalation routes refuted** (expF/A/B) | layer sensitivity spread only **1.5×**; 95% FFN energy needs **77%** of 64-neuron blocks; 128-token working-set union = **99.9%** — dynamic sparsity paging does *not* transfer to dense SwiGLU models |
| 6 | **Quality–bytes continuum** (candidate, 1.7B) | verify-all output **statistically = resident-q8** (bf16 judge −0.1297 vs −0.1328) at **237 MB/token** streamed — **9× under the checkpoint**; τ knob: 150/211/237 MB/token ↔ 47%/83%/~100% of the q4→q8 gap |
| 7 | **Out-of-core at 32B** (candidate, scale) | q8-32B (34.8 GB) **cannot co-reside** with the q4 base on 48 GB — streamed layer-by-layer at **~11.6 GB/s**; **+0.28 nats** over pure-q4 (the only fitting config) at **3.7 GB/token** (9.4× under checkpoint), 1.69 tok/s |
| 8 | **Two macOS measurement traps** | `F_NOCACHE` does **not** evict resident pages (naïve "cold" benchmarks read RAM at 42–115 GB/s); Metal prefill vs incremental decode flips **1.56%/token** of a model's own greedy argmax — token-exact reproduction is not a coherent target |
## The architecture — margin-gated deferred refinement
Selected by the evidence (not by preference — see the
[candidate ranking](research/candidate_ranking.md) and three refutations above):
1. **Resident base** — the q4 quantization of the model lives in unified
memory and generates greedily, recording each decision's **top-1 logit
margin** (a free byproduct of the forward pass).
2. **Deferral** — decisions with margin < τ are flagged; generation never blocks.
3. **Amortized verification sweeps** — every W tokens, the *same checkpoint's*
q8 variant teacher-forces the window in one pass through a persistent
incremental verify-KV-cache. When q8 cannot co-reside (the target regime),
it is **streamed layer-by-layer from SSD** (materialize → compute →
re-lazify) — one *sequential* pass over the checkpoint per sweep, exactly
the access pattern the measured SSD envelope rewards.
4. **Rollback** — on the first checked flip the runtime trims both KV caches,
accepts the q8 token, and resumes.
**τ is a runtime knob that prices output quality in verification bytes** — a
property no static quantization format offers.
Implementation: [`experiments/candidate_01/implementation/`](experiments/candidate_01/implementation/)
(`runtime.py` — generation/deferral/rollback loop; `streaming_verifier.py` —
per-layer weight streaming on unified memory).
## The experimental campaign
Every experiment has a preregistered `hypothesis.md` (seven-field scientific
block with falsification criterion) and a post-run `analysis.md`.
| Experiment | Question | Verdict |
|------------|----------|---------|
| [expH — SSD feasibility](experiments/micro/expH_ssd_feasibility/) | Can the internal NVMe sustain weight-streaming rates? | ✅ **Confirmed** — with a strict ≥256 KiB / QD≥4 / sequential contract |
| [expG — decision stability](experiments/micro/expG_decision_stability/) | Does the cheap pass's margin predict disagreement? | ✅ **Confirmed** (AUROC 0.85–0.98, improves with scale) |
| [expD — progressive reconstruction](experiments/micro/expD_progressive_reconstruction/) | Do residual ladders converge? Does gating compose? | ✅ **Confirmed** (with an honest static-parity caveat) |
| [expF — error accumulation](experiments/micro/expF_error_accumulation/) | Is quantization damage concentrated in layers? | ❌ **Refuted** — diffuse (1.5× spread), repair is cooperative |
| [expA — weight concentration](experiments/micro/expA_weight_concentration/) | Is per-token FFN importance pageable at block granularity? | ❌ **Refuted** — neuron-level concentration exists but is scattered |
| [expB — token stability](experiments/micro/expB_token_stability/) | Is the important-block set temporally cacheable? | ❌ **Refuted** — union ≈ 100% within 128 tokens; Jaccard ≈ random null |
| [candidate_01 — 1.7B](experiments/candidate_01/) | Does the assembled runtime deliver a quality–bytes continuum? | ✅ q8-equivalent quality at 9× fewer bytes |
| [candidate_01 — 32B scale](experiments/candidate_01/benchmark_scale.py) | Does it work when the verifier physically cannot fit? | ✅ +0.28 nats over the only fitting alternative |
The refutations are load-bearing: they killed the fashionable design (dynamic
sparsity paging à la DejaVu / LLM-in-a-flash) *on modern dense architectures*
with numbers, and left batch-amortized verification as the only surviving
route — which the prototype then validated.
## Novelty, honestly stated
An adversarial prior-art search ([novelty_check.md](research/novelty_check.md),
45 sources) found **every component anticipated** — precision-tiered
draft/verify (QSpec, Apple QuantSpec), resident-draft + flash-streamed target
(Lever 2026, CATS 2026), uncertainty-gated verification (U-HLM/UGSD, vs cloud),
window-amortized verification (SpecExec), MLX layer streaming (mlx-flash).
Four narrow claims survived: **(N1)** margin-gating of the *same model's*
higher-precision **weight-streaming I/O** on one machine; **(N2)**
bytes-streamed-per-token vs checkpoint size as the explicit objective, with τ
as the knob; **(N3)** the training-free same-checkpoint q4/q8 split
demonstrated where the verifier cannot reside; **(N4)** the
unified-memory/Metal instantiation. Attribution obligations are recorded and
the check should be re-run before any external submission.
## Repository structure
```text
localvm-research/
├── CLAUDE.md # research charter (methodology contract)
├── research/ # the scientific paper trail
│ ├── LOG.md # append-only research log (every decision)
│ ├── state_of_the_art.md # Phase 2 — technique taxonomy, overlap analysis
│ ├── research_gaps.md # Phase 3 — 24 falsifiable gaps (G01–G24)
│ ├── candidate_ranking.md # Phase 4 — 10-axis scoring, 4 candidates
│ ├── novelty_check.md # Phase 11 — adversarial prior-art verdict
│ ├── bibliography.md # 304 sources with URLs and access dates
│ └── notes/ # 5 Phase-1 literature theme notes
├── experiments/
│ ├── micro/expA…expH/ # preregistered micro-experiments
│ └── candidate_01/ # the prototype runtime + 1.7B/32B benchmarks
├── benchmarks/
│ ├── hardware_manifest.py # macOS-native fingerprint (embedded in results)
│ └── datasets/eval_prompts.json # 6-domain eval prompts (code/math/chat/FR/EN/reasoning)
├── results/ # raw JSON results (commit + config + seed + manifest)
├── src/localvm/ # library (quality/decision_stats.py, backends, …)
├── tools/ # check_headers.py, new_experiment.py, make_pub_figures.py
├── docs/publications/ # official technical reports (TR-01) + SVG figures
└── web/ # the public research platform (Express, deployed)
```
## Reproduce it
```bash
# 1. Environment (Apple Silicon Mac, macOS 14+)
python3 -m venv .venv
.venv/bin/pip install mlx mlx-lm numpy
# 2. Fingerprint your machine (embedded in every result)
python3 benchmarks/hardware_manifest.py
# 3. Substrate envelope (~15 min, writes results/expH_ssd_feasibility/)
cd experiments/micro/expH_ssd_feasibility && ../../../.venv/bin/python benchmark.py
# 4. Decision stability (~10 min, downloads Qwen3-1.7B-bf16)
cd ../expG_decision_stability && ../../../.venv/bin/python benchmark.py
# 5. The prototype (build quantized artifacts once, then run)
cd ../../candidate_01
../../.venv/bin/python benchmark.py --build
../../.venv/bin/python benchmark.py # 1.7B continuum
../../.venv/bin/python benchmark_scale.py # 32B out-of-core (downloads ~52 GB)
# 6. Regenerate all publication figures from committed results
python3 tools/make_pub_figures.py
# Quality gates
python3 tools/check_headers.py # mandatory header check (CI-style)
```
Every benchmark writes `results/