# project: localvm-research document: README author: Simon-Pierre Boucher contact: contact@spboucher.ai created: 2026-08-11 modified: 2026-08-12 status: reviewed

# localvm-research

Running LLMs larger than memory on a consumer Mac

Live research platform: www.localvm.dev — every hypothesis, result, figure, and failure, browsable.

Author: Simon-Pierre Boucher · contact@spboucher.ai

# The question

Can an existing pretrained LLM whose memory requirements exceed a consumer Apple Silicon Mac be transformed post-training into an execution representation whose instantaneous working set is dramatically smaller than the full checkpoint — while preserving most of its capabilities?

The project's central distinction, which everything here tries to decouple:

total model size  ≠  resident model size  ≠  bytes read per token  ≠  parameters required for THIS token

This is a falsification-driven research project run under a strict charter (CLAUDE.md): hypotheses are registered with explicit kill criteria before experiments run, no result is reported from an uncommitted tree, every result file embeds a hardware manifest, and negative results are kept and published. The full audit trail lives in the append-only research log.

# Headline results

All measured on the primary research machine — Apple M5 Max, 6P+12E CPU, 40-core GPU, 48 GB unified memory, 2 TB AP2048Z NVMe, macOS 27.0.

# The architecture — margin-gated deferred refinement

Selected by the evidence (not by preference — see the candidate ranking and three refutations above):

1. Resident base — the q4 quantization of the model lives in unified memory and generates greedily, recording each decision's top-1 logit margin (a free byproduct of the forward pass).
2. Deferral — decisions with margin < τ are flagged; generation never blocks.
3. Amortized verification sweeps — every W tokens, the same checkpoint's q8 variant teacher-forces the window in one pass through a persistent incremental verify-KV-cache. When q8 cannot co-reside (the target regime), it is streamed layer-by-layer from SSD (materialize → compute → re-lazify) — one sequential pass over the checkpoint per sweep, exactly the access pattern the measured SSD envelope rewards.
4. Rollback — on the first checked flip the runtime trims both KV caches, accepts the q8 token, and resumes.

τ is a runtime knob that prices output quality in verification bytes — a property no static quantization format offers.

Implementation: experiments/candidate_01/implementation/ (runtime.py — generation/deferral/rollback loop; streaming_verifier.py — per-layer weight streaming on unified memory).

# The experimental campaign

Every experiment has a preregistered hypothesis.md (seven-field scientific block with falsification criterion) and a post-run analysis.md.

The refutations are load-bearing: they killed the fashionable design (dynamic sparsity paging à la DejaVu / LLM-in-a-flash) on modern dense architectures with numbers, and left batch-amortized verification as the only surviving route — which the prototype then validated.

# Novelty, honestly stated

An adversarial prior-art search (novelty_check.md, 45 sources) found every component anticipated — precision-tiered draft/verify (QSpec, Apple QuantSpec), resident-draft + flash-streamed target (Lever 2026, CATS 2026), uncertainty-gated verification (U-HLM/UGSD, vs cloud), window-amortized verification (SpecExec), MLX layer streaming (mlx-flash). Four narrow claims survived: (N1) margin-gating of the same model's higher-precision weight-streaming I/O on one machine; (N2) bytes-streamed-per-token vs checkpoint size as the explicit objective, with τ as the knob; (N3) the training-free same-checkpoint q4/q8 split demonstrated where the verifier cannot reside; (N4) the unified-memory/Metal instantiation. Attribution obligations are recorded and the check should be re-run before any external submission.

# Repository structure

localvm-research/
├── CLAUDE.md                    # research charter (methodology contract)
├── research/                    # the scientific paper trail
│   ├── LOG.md                   #   append-only research log (every decision)
│   ├── state_of_the_art.md      #   Phase 2 — technique taxonomy, overlap analysis
│   ├── research_gaps.md         #   Phase 3 — 24 falsifiable gaps (G01–G24)
│   ├── candidate_ranking.md     #   Phase 4 — 10-axis scoring, 4 candidates
│   ├── novelty_check.md         #   Phase 11 — adversarial prior-art verdict
│   ├── bibliography.md          #   304 sources with URLs and access dates
│   └── notes/                   #   5 Phase-1 literature theme notes
├── experiments/
│   ├── micro/expA…expH/         # preregistered micro-experiments
│   └── candidate_01/            # the prototype runtime + 1.7B/32B benchmarks
├── benchmarks/
│   ├── hardware_manifest.py     # macOS-native fingerprint (embedded in results)
│   └── datasets/eval_prompts.json  # 6-domain eval prompts (code/math/chat/FR/EN/reasoning)
├── results/                     # raw JSON results (commit + config + seed + manifest)
├── src/localvm/                 # library (quality/decision_stats.py, backends, …)
├── tools/                       # check_headers.py, new_experiment.py, make_pub_figures.py
├── docs/publications/           # official technical reports (TR-01) + SVG figures
└── web/                         # the public research platform (Express, deployed)

# Reproduce it

# 1. Environment (Apple Silicon Mac, macOS 14+)
python3 -m venv .venv
.venv/bin/pip install mlx mlx-lm numpy

# 2. Fingerprint your machine (embedded in every result)
python3 benchmarks/hardware_manifest.py

# 3. Substrate envelope (~15 min, writes results/expH_ssd_feasibility/)
cd experiments/micro/expH_ssd_feasibility && ../../../.venv/bin/python benchmark.py

# 4. Decision stability (~10 min, downloads Qwen3-1.7B-bf16)
cd ../expG_decision_stability && ../../../.venv/bin/python benchmark.py

# 5. The prototype (build quantized artifacts once, then run)
cd ../../candidate_01
../../.venv/bin/python benchmark.py --build
../../.venv/bin/python benchmark.py            # 1.7B continuum
../../.venv/bin/python benchmark_scale.py      # 32B out-of-core (downloads ~52 GB)

# 6. Regenerate all publication figures from committed results
python3 tools/make_pub_figures.py

# Quality gates
python3 tools/check_headers.py                 # mandatory header check (CI-style)

Every benchmark writes results/<experiment>/<UTC timestamp>/results.json with the full config and hardware manifest embedded.

# Publications

• TR-01 — Margin-Gated Deferred Refinement: Streaming Higher-Precision LLM Quality Than Fits in Memory on Consumer Apple Silicon (August 2026) — the complete write-up with figures, prior-art attribution, and limitations. Also on the platform: localvm.dev/publications.

# Known limitations & roadmap

Recorded in TR-01 §7 and the research log — the honest list: prototype speed (1.69 tok/s at 32B) is below interactive; no resident-q5/q6 baseline at 32B yet (they fit in 48 GB and may dominate at that scale — the architecture's necessary regime is 70B-class, untested); the 8B judge is a proxy with no error bars at n=12; no task-level benchmarks yet; one model family, one machine. Next: resident q5/q6 baselines → task benchmarks (GSM8K-class) → τ calibration on real quantized-artifact margins → 70B point → TR-02.

# Author & citation

Simon-Pierre Boucher — contact@spboucher.ai

@techreport{boucher2026localvm,
  author      = {Boucher, Simon-Pierre},
  title       = {Margin-Gated Deferred Refinement: Streaming Higher-Precision
                 LLM Quality Than Fits in Memory on Consumer Apple Silicon},
  institution = {localvm-research},
  number      = {TR-01},
  year        = {2026},
  month       = {August},
  url         = {https://www.localvm.dev/publications}
}

Citation metadata: CITATION.cff.

# License

All rights reserved (research code) — © 2026 Simon-Pierre Boucher. See LICENSE. The research trail is public for reading and verification at www.localvm.dev; reuse requires written authorization.