SPB Git

spb/localvm-research Public License

Running LLMs larger than memory on a consumer Mac — falsification-driven research: margin-gated deferred refinement, out-of-core verification on Apple Silicon. TR-01 published.

Python 63.2% JavaScript 23.5% CSS 11.8% Shell 0.9% Makefile 0.5%
24.0 KB
View source Raw Blame History

# project: localvm-research document: research/LOG (append-only research log) author: Simon-Pierre Boucher contact: contact@spboucher.ai created: 2026-08-11 status: draft

# Research log (append-only)

Format per entry: date/time (local, with timezone) · question · experiment · result · interpretation · decision.

# 2026-08-11 23:40 EDT — Project bootstrap

  • Question: none (administrative).
  • Experiment: none.
  • Result: Repository initialized (git, main). Skeleton per charter §3 created. Mandatory tooling written and functional: tools/check_headers.py (§0.1 enforcement), tools/new_experiment.py (experiment scaffolding with seven-field hypothesis block), benchmarks/hardware_manifest.py (macOS-native fingerprinting).
  • Host hardware discovered: Apple M5 Max, 48 GB unified memory, macOS 27.0 (build 26A5388g) — within the charter's target class (16–64 GB Apple Silicon).
  • Interpretation: n/a.
  • Decision: Begin Phase 1 (ultra-deep literature research, charter §4) immediately, fanning out across the ten mandated areas (§4.1–§4.10). Deliverables: per-topic notes in research/notes/, every source logged in research/bibliography.md with URL and access date.

# 2026-08-12 00:15 EDT — Phase 1 complete: literature sweep across §4.1–§4.10

  • Question: What is already known about post-training transforms + out-of-core execution that decouple checkpoint size from resident memory / bytes-per-token, and what is missing?
  • Experiment: Five parallel deep literature sweeps (web, arXiv, GitHub, proceedings), one per theme cluster. Deliverables in research/notes/ (5 files, ~300 sources, all with access dates, merged into research/bibliography.md).
  • Result (key facts):
    • Nested/progressive weight encodings exist (Any-Precision LLM, MatQuant, BitStack, RRQ, CALDERA Q+LR) but ALL choose the operating point statically, keep everything resident, never page residuals from storage, and are CUDA-only.
    • 2–3-bit SOTA (QuIP#, AQLM, QTIP, GPTVQ) has zero Metal implementations; LUT-heavy decode is compute-bound on Apple GPUs — Apple formats must keep decode shift/mask-cheap.
    • A representational cliff sits between 2 and 3 bits (ParetoQ/EfficientQAT): a 2-bit base is the floor for staying in the pretrained basin.
    • Closest prior art on our hardware: Apple "LLM in a flash" (0.2 GB/token OPT-6.7B on M1 Max via predictors + windowing; ReLU-only, FFN-only, fp16, no code) and PowerInfer-2 (47B on a 24 GB phone; requires dReLU retraining). Nobody has built predictor/threshold- driven sparse weight paging on macOS/Metal/unified memory.
    • Measured decision stability: 4-bit quants agree with fp16 on ~90–91% of greedy tokens; the joint (cheap-pass margin × agreement) distribution is UNPUBLISHED — cheap for us to measure (expG) and decisive for any escalation design.
    • Precision-level self-speculation exists across tokens (QSpec, Apple QuantSpec ~2.5×,

      90% acceptance) but nobody gates weight loading on per-token decision uncertainty.

    • OS/DB ideas unapplied to LLM weights: MRU/DBMIN for cyclic dense scans (LRU provably worst-case for our pattern), ARC ghost lists for expert caches, anti-caching's never-block-on-miss, purgeable MTLHeaps as an OS-cooperative cache tier.
  • Interpretation: Independent sweeps converged on the same gap: a progressive, residency-tiered weight representation (low-bit resident base + SSD-resident residuals) with decision-uncertainty-driven refinement is unbuilt, and Apple unified memory + fast NVMe is the substrate where it is most plausible.
  • Decision: Proceed to Phase 2 (state-of-the-art map synthesized from notes), then Phase 3 gap generation (≥20 ideas). expH (SSD envelope) running concurrently.

# 2026-08-12 00:30 EDT — expH complete: SSD substrate envelope established

  • Question: Can the internal Apple NVMe sustain weight-streaming rates, at which block sizes, and does Metal GPU load contend with it? (charter §9.H)
  • Experiment: expH full run, 3 repeats/cell, adaptive ~3 s budgets, iostat ground truth. Results: results/expH_ssd_feasibility/20260812T034359Z/, code at commit b4a652d.
  • Result: Device ceiling ~13.1 GB/s (iostat-validated). Cold random: 67 MB/s @4 KiB QD1; 11.6 GB/s @256 KiB QD8; 13.8 GB/s @1 MiB QD8. Concurrent saturated MLX matmul costs <5%. Warm (unified memory) peak 136 GB/s → RAM:SSD ≈ 10:1, far gentler than discrete-GPU stacks. 4 MiB & threaded-sequential cells cache-contaminated (> iostat ceiling) — flagged invalid.
  • Interpretation: Storage is NOT the binding constraint: ~650 MB/token of SSD reads is realistic at interactive rates IF blocks are ≥256 KiB at QD≥4. The binding question moves to whether ≤650 MB/token of useful bytes preserves model behavior (expD/expG). Methodological: F_NOCACHE does not bypass resident pages; benchmark files must be written uncached; distrust any number above the concurrent iostat ceiling; 16 KiB pages.
  • Decision: Prioritize expD (progressive reconstruction) and expG (decision stability) after Phases 2–4 paperwork. Sequential-read claims embargoed until a >RAM-size-file rerun.

# 2026-08-12 00:50 EDT — Phase 2 complete: state-of-the-art map

  • Question: How does the technique landscape organize, where do approaches overlap, and which seemingly-novel ideas are already known?
  • Experiment: Synthesis of the five Phase 1 notes into research/state_of_the_art.md (356 lines, ~35 technique groups, six families, all charter §5 fields, overlap analysis, settled-vs-open ledger).
  • Result (load-bearing overlap findings): (1) quantization+offload and precision×tier unification already exist (llama.cpp, FlexGen, M2Cache) — but always statically, never refining a fetched block; (2) sparsity-driven flash paging exists (LLM-in-a-flash, PowerInfer-2, Ripple, DIP) but ReLU-only, never macOS/Metal, always predict-and-hope with no correction; (3) QSpec/QuantSpec prove >90% low/full-bit agreement with exact repair but keep BOTH precisions resident — the unclaimed move is using agreement to avoid LOADING high-precision bytes; (4) the progressive/nested code family never reports bytes/token — the charter's central metric is literally unreported there; (5) several "obvious" ideas are known including failure modes (weights-as-mmap collapse at 0.025 tok/s; DFloat11 decode tax; i-quant LUT penalty on Metal). Seven verified open intersections listed in §7.3.
  • Interpretation: The gap space is real, narrow, and well-bounded — enough for ≥20 concrete gap hypotheses without inventing variants of one idea.
  • Decision: Launch Phase 3 (research_gaps.md) now; Phase 4 ranking after review.

# 2026-08-12 09:05 EDT — Phase 3 complete: 24 research gaps

  • Question: Which genuinely different, falsifiable approaches exist in the open space?
  • Experiment: research/research_gaps.md — 24 gaps (G01–G24), six mandated items each, kill-numbers sized to the expH envelope; includes 3 contrarian gaps (G22 WarmEnough, G23 ConfidentlyWrong, G24 BasinCollapse) that can cheaply kill whole families.
  • Result: Highest-leverage falsification order derived from cluster analysis: (1) expG joint margin×agreement matrix → arbitrates G02/G17/G23/half-G24; (2) expA/B/C SwiGLU trace campaign → feeds G01/G06/G07/G12/G13 + cache simulator; (3) expD/expE ladder (recovery curves + Metal decode/gather microbench) → decides G01/G04/G05/G15/G16, rest of G24, gates G03.
  • Interpretation: The experiment plan now writes itself from gap dependencies.
  • Decision: Launch Phase 4 ranking; then implement expG first.

# 2026-08-12 11:40 EDT — Phase 4 complete: candidate ranking

  • Question: Which of the 24 gaps deserve prototyping effort, and in what composition?
  • Experiment: research/candidate_ranking.md — all 24 gaps scored 1–10 on the ten charter §7 axes with per-axis written reasoning; summary table; totals span 51–78 (top: G18 78, G02 77, G17/G23 76, G09 75, G24 74; bottom: G06 51).
  • Result: Four candidates selected (three compositions + one falsifier battery): C1 Progressive-Residual Runtime (G01+G02+G04, G03 kernel, G05 gate) — criteria B/C/D/E; C2 Amortized Verification Sweeps (G17+G18 on G08 MRU floor) — criteria A/B, exact, also the honest baseline; C3 MoE Residency Engine (G09+G12+G19, G10 warm tier) — criteria A/F; C4 Gate Zero contrarian battery (G23+G24+G22+expN) — runs first, gates 17/24 gaps. Promote/kill thresholds fixed numerically per candidate (AUROC 0.8/0.65–0.85 bands; ≤20% residual bytes + ≥50% reuse; acceptance ≥4–5; ARC ≥5 pts; warm-fault 10%/25%).
  • Interpretation: Totals reward safety; selection weighted Sig + cluster synergy + the expH substrate contract. Preferred single candidate: C1 — the only mechanism that turns model size → hardware into working set → hardware (charter §20).
  • Decision: Proceed to Phase 5 infrastructure, then expG run matrix (arbitrates G02/G17/G23/G24) before any candidate-specific engineering.

# 2026-08-12 09:40 EDT — Phase 4 complete + public research platform deployed

  • Question: Which candidates should be prototyped, and how is the project made auditable publicly?
  • Experiment: n/a (ranking synthesis + engineering).
  • Result: research/candidate_ranking.md — all 24 gaps scored on 10 axes (totals 51–78, differentiated). Selected: C1 Progressive-Residual Runtime (G01+G02+G04, preferred), C2 Amortized Verification Sweeps (G17+G18, honest baseline), C3 MoE Residency Engine (G09+G12+G19), C4 Gate-Zero falsifier battery (G22–G24 first, gates 17/24 gaps). Public platform live at https://www.localvm.dev (M3U96b, PM2 + ngrok) serving the full paper trail: docs, notes, experiments, raw results, code. Content snapshot at commit 9b94283.
  • Interpretation: Gate-Zero battery (expG-centric) is the rational first experiment.
  • Decision: Next session: implement expG (joint margin×agreement matrix) on a small model.

# 2026-08-12 10:35 EDT — expG complete: margin-gated escalation PROMOTED, naive 2-bit base DEAD

  • Question: Does the cheap pass's top-1 margin predict disagreement with the full model (the unpublished joint distribution that gates G02/G23/G24 and candidate C1)?
  • Experiment: expG on Qwen3-1.7B — 48 greedy bf16 trajectories (6 domains), teacher-forced MLX affine 2/3/4/8-bit variants, 6 144 positions/bit-width. Results: results/expG_decision_stability/20260812T042744Z/.
  • Result: AUROC(margin→disagreement): 0.852 @3-bit, 0.898 @4-bit, 0.975 @8-bit — all above the 0.8 promote threshold; per-domain uniform (0.88–0.92 @4-bit). Escalation to reach 99% agreement: 60.2% / 36.6% / 5.1% of tokens. Median margin agree vs disagree: 6.5 vs 0.75 (4-bit). Naive affine 2-bit collapsed: 3.0% agreement, KL 11.9, AUROC 0.605.
  • Interpretation: (1) G02 MarginGate PROMOTED — decision uncertainty is a real, free, domain-stable gating signal. (2) G23 weakened, not dead: 36.6% escalation @4-bit exceeds the hoped 10–30%; viability now depends on bytes-per-escalation. (3) G24 partially resolved: the 2–3-bit cliff is real and brutal for affine quantization — C1's resident base must be ≥3-bit affine or codebook 2-bit (G05 kernel work). (4) 8-bit ceiling: 98% agreement, 5.1% escalation.
  • Decision: Implement expD next (residual bytes per escalated token); plan an expG rerun at 3B–8B to measure the scale trend.

# 2026-08-12 11:20 EDT — expD complete: progressive residual coding validated; C1 operating point exists

  • Question: How fast do decisions/hidden states converge with residual stages, and does a margin-gated two-tier policy work? (gates G01 ResidualPager + C1 arithmetic)
  • Experiment: expD — affine g64 residual ladders (3/3+3/3+3+3, 4/4+4 bits) on Qwen3-1.7B, 48 trajectories, hidden states at 4 depths, two-tier policy grid. Results: results/expD_progressive_reconstruction/20260812T043508Z/.
  • Result: Monotone, fast convergence (hidden err ÷5 per stage; KL ÷30–60). 3+3 bits: 95.7% agreement; 4+4: 98.4%. Two-tier policy (4-bit base, τ=3): 97.6% agreement escalating 35% of tokens. AUROC stays 0.85–0.98 at every stage. Sanity: stage0 rows reproduce expG flat rows. Caveat: at equal static bytes, flat quantization mildly beats ladders (98.0% @9.0 vs 95.7% @8.0 bits) — the ladder's value is runtime-variable quality, not compression.
  • Interpretation: G01+G02 compose; C1's operating point exists mechanically. The binding unknown is bytes-per-escalation: full-residual escalation (~20 GB at 32B) breaks the expH budget; last-layer error concentration (0.83 vs 0.34 at 4-bit) suggests layer-restricted escalation could cut it ~4×.
  • Decision: expF (layer sensitivity) next; then expB (temporal locality of escalated set).

# 2026-08-12 12:05 EDT — expF complete: NEGATIVE — layer-restricted escalation gives no leverage

  • Question: Is quantization damage concentrated in a subset of layers, so escalation can be layer-restricted and bytes-per-escalation cut proportionally?
  • Experiment: expF — 7 depth-groups × degrade-one/repair-one + repair-top-k, 4-bit affine vs bf16, Qwen3-1.7B, 48 trajectories, 17 configs. Results: results/expF_error_accumulation/20260812T043945Z/.
  • Result: REFUTED. Sensitivity spread 1.5× (kill: <2×). Top-29% of layers repair 24% of lost agreement (sub-proportional). Degrade-one drops sum to 33 pts vs 12.5 joint (masking); repair-one values sum to 7.4 of 12.5 (cooperation). Last layers: largest hidden-state error (expD) but LEAST decision-sensitive — norms ≠ decisions.
  • Interpretation: Damage is diffuse across depth. C1's escalation-byte problem must be solved by temporal locality (expB), block-level selection (expA/expE), or batch amortization — the last converging with C2 (amortized verification). Negative result recorded per §10/§17.
  • Decision: Next: expA (block-granularity concentration) and expB (temporal stability) — the trace campaign that also feeds G06/G07/G12/G13 per the Phase 3 cluster analysis.

# 2026-08-12 13:10 EDT — expA + expB complete: NEGATIVE — dynamic sparsity paging dead on dense SwiGLU; evidence selects the C1→C2 merge

  • Question: Is per-token FFN importance concentrated at pageable block granularity (expA), and is the important set temporally/domain reusable (expB)?
  • Experiment: One instrumented trace (SwiGLU intermediate energy, 16-neuron blocks × 28 layers × 6 144 positions) feeding both analyses. Results: results/expA_weight_concentration/20260812T044829Z/, results/expB_token_stability/20260812T045224Z/.
  • Result: expA: 95% energy needs 77% of 64-neuron blocks (kill >70%); neuron-level concentration real (20%) but scattered — bundling destroys it; strong depth gradient (L27: 11%, early layers: ~80%). expB: Jaccard(t,t+1)=0.69 vs matched random null 0.64 (stickiness = set-size artifact); union working set 98% after 8 tokens, 99.9% after 128 (kill ≥95%); domain locality nil.
  • Interpretation: All three fine-grained routes for per-token escalation bytes are now closed by measurement: layers (expF), blocks (expA), temporal/domain caching (expB). The DejaVu/LLM-in-a-flash paradigm measurably does not transfer to dense SwiGLU models. G06/G12/G13 dead on this family; G07 survives only as static tiering. Remaining route is batch amortization, which expH's sequential numbers make attractive (full 1 GB residual streams in ~80 ms at 13 GB/s, shared across a queue of deferred low-margin tokens). C1 converges into C2: margin-gated deferred refinement with periodic sequential sweeps.
  • Decision: (1) Rerun expG at 8B to confirm margin-signal scaling; (2) then prototype candidate_01: 4-bit resident base + margin-gated deferral queue + periodic residual sweep with rollback, measured in bytes/token, tok/s, agreement, on this Mac.

# 2026-08-12 14:30 EDT — expG@8B (favorable scaling) + candidate_01 first run (mixed; metric insight)

  • Question: Does the margin signal hold at 8B, and does the selected architecture (margin-gated deferred refinement) work end-to-end on this Mac?
  • Experiment: expG rerun on Qwen3-8B (results/expG_decision_stability/20260812T051129Z/); candidate_01 prototype: q4 resident + windowed q8 sweeps + rollback via KV trim (results/candidate_01/20260812T051501Z/), 24 prompts × 128 tokens, W=32.
  • Result: 8B scaling favorable on every metric (4-bit: agree 87.4→92.1%, AUROC 0.90→0.92, esc@99% 36.6→22.6%). Candidate: bytes/token 154–231 MB logical (9–14× under q8 checkpoint, within expH budget) — PASSED; verify-all fidelity 0.883 = measured hardware ceiling (q8 self-agreement 1.56%/token prefill/decode flips ⇒ ~0.88 seq-ratio at 128 tokens); margin-mode seq-fidelity 0.55–0.65 — FAILED as registered, caused by divergence compounding of ~2% unchecked per-token error; throughput: margin τ=1.0 passed 1/3-of-q4, verify-all failed (full-context re-prefill per sweep — fixable via incremental verify-cache).
  • Interpretation: (1) Token-exact sequence reproduction is not a coherent target on Metal — the reference model cannot reproduce itself; quality-level metrics are now mandatory (§4.10 empirically forced). (2) The architecture's byte economics hold and improve with scale. (3) Sweep cost is an implementation artifact with a known fix.
  • Decision: Next cycle: quality-metric evaluation (bf16-judged logprobs, task checks), incremental verify-cache, then scale to a q8>RAM model — the regime this architecture is for.

# 2026-08-12 15:20 EDT — candidate_01 v2: quality claim CONFIRMED — q8-equivalent output with q4 resident

  • Question: Does quality-level evaluation vindicate the architecture, and does the incremental verify-cache fix sweep cost?
  • Experiment: candidate_01 v2 (results/candidate_01/20260812T052415Z/): persistent q8 verify KV cache (O(window) sweeps); all outputs judged by bf16 mean logprob.
  • Result: Judge scores — q4 floor -0.387; margin τ=1.0 -0.267 (47% of gap, 150 MB/token); τ=2.0 -0.176 (83%, 211 MB/token); verify-all -0.130 ≈ q8's -0.133 (indistinguishable, 237 MB/token, 9× under checkpoint). Throughput 69–110 tok/s at 1.7B.
  • Interpretation: The charter §16.D/§16.E criteria are now met at 1.7B scale: a progressive/conditional mechanism preserving quality while avoiding most weight loading, with a runtime quality↔bytes knob (τ) no static quantization offers. v1's fidelity failure confirmed as metric artifact. Remaining lever: rollback regeneration (accept q8 token runs from sweep logits — free speculative continuation).
  • Decision: Scale run at 32B (q4 17 GB resident, q8 35 GB streamed — q8 does not fit wired on 48 GB): the regime the architecture exists for. Artifacts downloading.

# 2026-08-12 16:40 EDT — 32B scale run: charter §16.A/B demonstrated in prototype form

  • Question: Does the architecture deliver in its target regime — a model configuration that does not fit in unified memory?
  • Experiment: Qwen3-32B on the 48 GB M5 Max: q4 resident (17.5 GB) + q8 (34.8 GB) layer-streamed per sweep via StreamingVerifier (materialize→compute→re-lazify; sequential SSD reads at ~11.6 GB/s ≈ expH ceiling). 12 prompts × 96 tokens. Results: results/candidate_01_scale32b/20260812T055506Z/ (smoke: 20260812T053042Z).
  • Result: verify-all: judge (8B bf16) -0.4667 vs pure-q4 -0.7506 (+0.28 nats, 38% rel.); 1.69 tok/s; 3.72 GB/token logical (9.4× under streamed checkpoint); sweeps 3.0 s. Margin τ=2.0 intermediate (-0.546, 1.62 tok/s). q8 cannot run resident on this machine at all.
  • Interpretation: Total representation (52 GB) exceeds unified memory (48 GB) yet its quality is usable locally — bytes-per-token decoupled from checkpoint size with a strictly sequential access pattern. Below interactive (1.65 tok/s): measured headroom = larger windows, speculative acceptance of q8 runs from sweep logits, I/O-compute overlap (<5% contention per expH). Limits recorded: no q8 ceiling measurable here; 8B judge is a proxy; no window tuning.
  • Decision: Phase 11 novelty check on the assembled architecture next; then throughput levers or additional scale points.

# 2026-08-12 — Phase 11 novelty verification: composition survives narrowly; three components anticipated

  • Question: Charter §19 — is the candidate_01 architecture (resident q4 + margin-gated deferral + periodically streamed q8 verifier, bytes/token objective) already in the literature? Adversarial stance: assumed NOT novel.
  • Experiment: Targeted web/arXiv/GitHub search per component and for the composition (~45 sources verified, key papers read from full PDFs). Deliverable: research/novelty_check.md.
  • Result: Three 2025–2026 systems missed by the Phase 1 sweep materially shrink the claim: Lever (2605.16786 — DRAM-resident draft + flash-streamed target verifier, amortized per window, smartphones), CATS (2605.11186 — self-speculative shallow draft
    • verifier layers streamed from flash once per cycle, edge), U-HLM/UGSD (2412.12687 / 2603.11397 — draft-side uncertainty gates whether verification happens at all, cloud tier, incl. lossy top-R acceptance + rollback). QSpec/QuantSpec/ML-SpecQD own the shared-weights precision pair; mlx-flash owns MLX layer streaming (materialize→compute→release, every token).
  • Interpretation: "Resident draft + streamed verifier" and "uncertainty-gated verification skipping" are ANTICIPATED as categories. Surviving as not-found: (N1) margin-gating the weight-streaming bytes of the same model's higher-precision variant on one machine; (N2) bytes-streamed-per-token vs checkpoint size as explicit objective, with τ as a runtime bytes↔quality knob; (N3) the training-free same-checkpoint q4/q8 pair as the draft/verify split of an out-of-core runtime; (N4, engineering) the unified-memory Metal instantiation with incremental verify-KV-cache. All public claims must use the attribution table and README language in novelty_check.md §3–§4.
  • Decision: Proceed with narrowed claims only. Positioning for any writeup: extend Lever/CATS's regime with (a) precision-cast drafts (no training), (b) margin-priced verification I/O, (c) bytes/token accounting. Re-run this check before publication — prior-art density in this exact area is rising quarter over quarter.

# 2026-08-12 18:05 EDT — W=64 lever test: NEGATIVE — rollback rate, not window size, bounds amortization

  • Question: Does doubling the sweep window halve bytes/token?
  • Experiment: 32B verify-all, W=64 vs W=32 (results/candidate_01_scale32b/20260812T061115Z/).
  • Result: GB/token 3.72→3.53 (−5%, not −50%); tok/s 1.69→1.59; judge −0.467→−0.514 (n=12, noise).
  • Interpretation: Effective inter-sweep distance is bounded by ~1/rollback-rate (≈8–15 tokens at 12%/token), so W beyond that is dead weight. Real levers: τ-calibrated margin mode (fewer checked positions → fewer rollback triggers) and sweep-I/O/generation overlap.
  • Decision: Session milestone reached; next cycle: honest baselines (resident q5/q6 at 32B), τ calibration on the real q4-32B margin distribution, task-level benchmarks, 70B point.

# 2026-08-12 20:10 EDT — TR-01 published: first official technical report

  • Question: n/a (dissemination).
  • Experiment: n/a.
  • Result: docs/publications/TR-01-margin-gated-deferred-refinement.md — full write-up of results to date (substrate, decision stability, three refutations, architecture, 1.7B continuum, 32B out-of-core run), with 4 SVG figures regenerated from committed results by tools/make_pub_figures.py. Prior-art attribution follows novelty_check.md; limitations section names the missing baselines (resident q5/q6 at 32B, task benchmarks, 70B). Published on the platform under the new Publications tab.
  • Interpretation: n/a.
  • Decision: TR-02 planned once the honest-baseline campaign (q5/q6, tasks, 70B) completes.