--- project: localvm-research document: research/LOG (append-only research log) author: Simon-Pierre Boucher contact: contact@spboucher.ai created: 2026-08-11 status: draft --- # Research log (append-only) Format per entry: date/time (local, with timezone) · question · experiment · result · interpretation · decision. --- ## 2026-08-11 23:40 EDT — Project bootstrap - **Question:** none (administrative). - **Experiment:** none. - **Result:** Repository initialized (git, `main`). Skeleton per charter §3 created. Mandatory tooling written and functional: `tools/check_headers.py` (§0.1 enforcement), `tools/new_experiment.py` (experiment scaffolding with seven-field hypothesis block), `benchmarks/hardware_manifest.py` (macOS-native fingerprinting). - **Host hardware discovered:** Apple M5 Max, 48 GB unified memory, macOS 27.0 (build 26A5388g) — within the charter's target class (16–64 GB Apple Silicon). - **Interpretation:** n/a. - **Decision:** Begin Phase 1 (ultra-deep literature research, charter §4) immediately, fanning out across the ten mandated areas (§4.1–§4.10). Deliverables: per-topic notes in `research/notes/`, every source logged in `research/bibliography.md` with URL and access date. --- ## 2026-08-12 00:15 EDT — Phase 1 complete: literature sweep across §4.1–§4.10 - **Question:** What is already known about post-training transforms + out-of-core execution that decouple checkpoint size from resident memory / bytes-per-token, and what is missing? - **Experiment:** Five parallel deep literature sweeps (web, arXiv, GitHub, proceedings), one per theme cluster. Deliverables in `research/notes/` (5 files, ~300 sources, all with access dates, merged into `research/bibliography.md`). - **Result (key facts):** - Nested/progressive weight encodings exist (Any-Precision LLM, MatQuant, BitStack, RRQ, CALDERA Q+LR) but ALL choose the operating point statically, keep everything resident, never page residuals from storage, and are CUDA-only. - 2–3-bit SOTA (QuIP#, AQLM, QTIP, GPTVQ) has zero Metal implementations; LUT-heavy decode is compute-bound on Apple GPUs — Apple formats must keep decode shift/mask-cheap. - A representational cliff sits between 2 and 3 bits (ParetoQ/EfficientQAT): a 2-bit base is the floor for staying in the pretrained basin. - Closest prior art on our hardware: Apple "LLM in a flash" (0.2 GB/token OPT-6.7B on M1 Max via predictors + windowing; ReLU-only, FFN-only, fp16, no code) and PowerInfer-2 (47B on a 24 GB phone; requires dReLU retraining). Nobody has built predictor/threshold- driven sparse weight paging on macOS/Metal/unified memory. - Measured decision stability: 4-bit quants agree with fp16 on ~90–91% of greedy tokens; the joint (cheap-pass margin × agreement) distribution is UNPUBLISHED — cheap for us to measure (expG) and decisive for any escalation design. - Precision-level self-speculation exists across tokens (QSpec, Apple QuantSpec ~2.5×, >90% acceptance) but nobody gates *weight loading* on per-token decision uncertainty. - OS/DB ideas unapplied to LLM weights: MRU/DBMIN for cyclic dense scans (LRU provably worst-case for our pattern), ARC ghost lists for expert caches, anti-caching's never-block-on-miss, purgeable MTLHeaps as an OS-cooperative cache tier. - **Interpretation:** Independent sweeps converged on the same gap: a *progressive, residency-tiered weight representation* (low-bit resident base + SSD-resident residuals) with *decision-uncertainty-driven refinement* is unbuilt, and Apple unified memory + fast NVMe is the substrate where it is most plausible. - **Decision:** Proceed to Phase 2 (state-of-the-art map synthesized from notes), then Phase 3 gap generation (≥20 ideas). expH (SSD envelope) running concurrently. --- ## 2026-08-12 00:30 EDT — expH complete: SSD substrate envelope established - **Question:** Can the internal Apple NVMe sustain weight-streaming rates, at which block sizes, and does Metal GPU load contend with it? (charter §9.H) - **Experiment:** expH full run, 3 repeats/cell, adaptive ~3 s budgets, iostat ground truth. Results: `results/expH_ssd_feasibility/20260812T034359Z/`, code at commit `b4a652d`. - **Result:** Device ceiling ~13.1 GB/s (iostat-validated). Cold random: 67 MB/s @4 KiB QD1; 11.6 GB/s @256 KiB QD8; 13.8 GB/s @1 MiB QD8. Concurrent saturated MLX matmul costs <5%. Warm (unified memory) peak 136 GB/s → RAM:SSD ≈ 10:1, far gentler than discrete-GPU stacks. 4 MiB & threaded-sequential cells cache-contaminated (> iostat ceiling) — flagged invalid. - **Interpretation:** Storage is NOT the binding constraint: ~650 MB/token of SSD reads is realistic at interactive rates IF blocks are ≥256 KiB at QD≥4. The binding question moves to whether ≤650 MB/token of *useful* bytes preserves model behavior (expD/expG). Methodological: F_NOCACHE does not bypass resident pages; benchmark files must be written uncached; distrust any number above the concurrent iostat ceiling; 16 KiB pages. - **Decision:** Prioritize expD (progressive reconstruction) and expG (decision stability) after Phases 2–4 paperwork. Sequential-read claims embargoed until a >RAM-size-file rerun. --- ## 2026-08-12 00:50 EDT — Phase 2 complete: state-of-the-art map - **Question:** How does the technique landscape organize, where do approaches overlap, and which seemingly-novel ideas are already known? - **Experiment:** Synthesis of the five Phase 1 notes into `research/state_of_the_art.md` (356 lines, ~35 technique groups, six families, all charter §5 fields, overlap analysis, settled-vs-open ledger). - **Result (load-bearing overlap findings):** (1) quantization+offload and precision×tier unification already exist (llama.cpp, FlexGen, M2Cache) — but always statically, never refining a fetched block; (2) sparsity-driven flash paging exists (LLM-in-a-flash, PowerInfer-2, Ripple, DIP) but ReLU-only, never macOS/Metal, always predict-and-hope with no correction; (3) QSpec/QuantSpec prove >90% low/full-bit agreement with exact repair but keep BOTH precisions resident — the unclaimed move is using agreement to avoid LOADING high-precision bytes; (4) the progressive/nested code family never reports bytes/token — the charter's central metric is literally unreported there; (5) several "obvious" ideas are known including failure modes (weights-as-mmap collapse at 0.025 tok/s; DFloat11 decode tax; i-quant LUT penalty on Metal). Seven verified open intersections listed in §7.3. - **Interpretation:** The gap space is real, narrow, and well-bounded — enough for ≥20 concrete gap hypotheses without inventing variants of one idea. - **Decision:** Launch Phase 3 (research_gaps.md) now; Phase 4 ranking after review. --- ## 2026-08-12 09:05 EDT — Phase 3 complete: 24 research gaps - **Question:** Which genuinely different, falsifiable approaches exist in the open space? - **Experiment:** `research/research_gaps.md` — 24 gaps (G01–G24), six mandated items each, kill-numbers sized to the expH envelope; includes 3 contrarian gaps (G22 WarmEnough, G23 ConfidentlyWrong, G24 BasinCollapse) that can cheaply kill whole families. - **Result:** Highest-leverage falsification order derived from cluster analysis: (1) expG joint margin×agreement matrix → arbitrates G02/G17/G23/half-G24; (2) expA/B/C SwiGLU trace campaign → feeds G01/G06/G07/G12/G13 + cache simulator; (3) expD/expE ladder (recovery curves + Metal decode/gather microbench) → decides G01/G04/G05/G15/G16, rest of G24, gates G03. - **Interpretation:** The experiment plan now writes itself from gap dependencies. - **Decision:** Launch Phase 4 ranking; then implement expG first. --- ## 2026-08-12 11:40 EDT — Phase 4 complete: candidate ranking - **Question:** Which of the 24 gaps deserve prototyping effort, and in what composition? - **Experiment:** `research/candidate_ranking.md` — all 24 gaps scored 1–10 on the ten charter §7 axes with per-axis written reasoning; summary table; totals span 51–78 (top: G18 78, G02 77, G17/G23 76, G09 75, G24 74; bottom: G06 51). - **Result:** Four candidates selected (three compositions + one falsifier battery): C1 Progressive-Residual Runtime (G01+G02+G04, G03 kernel, G05 gate) — criteria B/C/D/E; C2 Amortized Verification Sweeps (G17+G18 on G08 MRU floor) — criteria A/B, exact, also the honest baseline; C3 MoE Residency Engine (G09+G12+G19, G10 warm tier) — criteria A/F; C4 Gate Zero contrarian battery (G23+G24+G22+expN) — runs first, gates 17/24 gaps. Promote/kill thresholds fixed numerically per candidate (AUROC 0.8/0.65–0.85 bands; ≤20% residual bytes + ≥50% reuse; acceptance ≥4–5; ARC ≥5 pts; warm-fault 10%/25%). - **Interpretation:** Totals reward safety; selection weighted Sig + cluster synergy + the expH substrate contract. Preferred single candidate: C1 — the only mechanism that turns `model size → hardware` into `working set → hardware` (charter §20). - **Decision:** Proceed to Phase 5 infrastructure, then expG run matrix (arbitrates G02/G17/G23/G24) before any candidate-specific engineering. --- ## 2026-08-12 09:40 EDT — Phase 4 complete + public research platform deployed - **Question:** Which candidates should be prototyped, and how is the project made auditable publicly? - **Experiment:** n/a (ranking synthesis + engineering). - **Result:** `research/candidate_ranking.md` — all 24 gaps scored on 10 axes (totals 51–78, differentiated). Selected: C1 Progressive-Residual Runtime (G01+G02+G04, preferred), C2 Amortized Verification Sweeps (G17+G18, honest baseline), C3 MoE Residency Engine (G09+G12+G19), C4 Gate-Zero falsifier battery (G22–G24 first, gates 17/24 gaps). Public platform live at https://www.localvm.dev (M3U96b, PM2 + ngrok) serving the full paper trail: docs, notes, experiments, raw results, code. Content snapshot at commit 9b94283. - **Interpretation:** Gate-Zero battery (expG-centric) is the rational first experiment. - **Decision:** Next session: implement expG (joint margin×agreement matrix) on a small model. --- ## 2026-08-12 10:35 EDT — expG complete: margin-gated escalation PROMOTED, naive 2-bit base DEAD - **Question:** Does the cheap pass's top-1 margin predict disagreement with the full model (the unpublished joint distribution that gates G02/G23/G24 and candidate C1)? - **Experiment:** expG on Qwen3-1.7B — 48 greedy bf16 trajectories (6 domains), teacher-forced MLX affine 2/3/4/8-bit variants, 6 144 positions/bit-width. Results: `results/expG_decision_stability/20260812T042744Z/`. - **Result:** AUROC(margin→disagreement): 0.852 @3-bit, 0.898 @4-bit, 0.975 @8-bit — all above the 0.8 promote threshold; per-domain uniform (0.88–0.92 @4-bit). Escalation to reach 99% agreement: 60.2% / 36.6% / 5.1% of tokens. Median margin agree vs disagree: 6.5 vs 0.75 (4-bit). Naive affine 2-bit collapsed: 3.0% agreement, KL 11.9, AUROC 0.605. - **Interpretation:** (1) G02 MarginGate PROMOTED — decision uncertainty is a real, free, domain-stable gating signal. (2) G23 weakened, not dead: 36.6% escalation @4-bit exceeds the hoped 10–30%; viability now depends on bytes-per-escalation. (3) G24 partially resolved: the 2–3-bit cliff is real and brutal for affine quantization — C1's resident base must be ≥3-bit affine or codebook 2-bit (G05 kernel work). (4) 8-bit ceiling: 98% agreement, 5.1% escalation. - **Decision:** Implement expD next (residual bytes per escalated token); plan an expG rerun at 3B–8B to measure the scale trend. --- ## 2026-08-12 11:20 EDT — expD complete: progressive residual coding validated; C1 operating point exists - **Question:** How fast do decisions/hidden states converge with residual stages, and does a margin-gated two-tier policy work? (gates G01 ResidualPager + C1 arithmetic) - **Experiment:** expD — affine g64 residual ladders (3/3+3/3+3+3, 4/4+4 bits) on Qwen3-1.7B, 48 trajectories, hidden states at 4 depths, two-tier policy grid. Results: `results/expD_progressive_reconstruction/20260812T043508Z/`. - **Result:** Monotone, fast convergence (hidden err ÷5 per stage; KL ÷30–60). 3+3 bits: 95.7% agreement; 4+4: 98.4%. Two-tier policy (4-bit base, τ=3): 97.6% agreement escalating 35% of tokens. AUROC stays 0.85–0.98 at every stage. Sanity: stage0 rows reproduce expG flat rows. Caveat: at equal static bytes, flat quantization mildly beats ladders (98.0% @9.0 vs 95.7% @8.0 bits) — the ladder's value is runtime-variable quality, not compression. - **Interpretation:** G01+G02 compose; C1's operating point exists mechanically. The binding unknown is bytes-per-escalation: full-residual escalation (~20 GB at 32B) breaks the expH budget; last-layer error concentration (0.83 vs 0.34 at 4-bit) suggests layer-restricted escalation could cut it ~4×. - **Decision:** expF (layer sensitivity) next; then expB (temporal locality of escalated set). --- ## 2026-08-12 12:05 EDT — expF complete: NEGATIVE — layer-restricted escalation gives no leverage - **Question:** Is quantization damage concentrated in a subset of layers, so escalation can be layer-restricted and bytes-per-escalation cut proportionally? - **Experiment:** expF — 7 depth-groups × degrade-one/repair-one + repair-top-k, 4-bit affine vs bf16, Qwen3-1.7B, 48 trajectories, 17 configs. Results: `results/expF_error_accumulation/20260812T043945Z/`. - **Result:** REFUTED. Sensitivity spread 1.5× (kill: <2×). Top-29% of layers repair 24% of lost agreement (sub-proportional). Degrade-one drops sum to 33 pts vs 12.5 joint (masking); repair-one values sum to 7.4 of 12.5 (cooperation). Last layers: largest hidden-state error (expD) but LEAST decision-sensitive — norms ≠ decisions. - **Interpretation:** Damage is diffuse across depth. C1's escalation-byte problem must be solved by temporal locality (expB), block-level selection (expA/expE), or batch amortization — the last converging with C2 (amortized verification). Negative result recorded per §10/§17. - **Decision:** Next: expA (block-granularity concentration) and expB (temporal stability) — the trace campaign that also feeds G06/G07/G12/G13 per the Phase 3 cluster analysis. --- ## 2026-08-12 13:10 EDT — expA + expB complete: NEGATIVE — dynamic sparsity paging dead on dense SwiGLU; evidence selects the C1→C2 merge - **Question:** Is per-token FFN importance concentrated at pageable block granularity (expA), and is the important set temporally/domain reusable (expB)? - **Experiment:** One instrumented trace (SwiGLU intermediate energy, 16-neuron blocks × 28 layers × 6 144 positions) feeding both analyses. Results: `results/expA_weight_concentration/20260812T044829Z/`, `results/expB_token_stability/20260812T045224Z/`. - **Result:** expA: 95% energy needs 77% of 64-neuron blocks (kill >70%); neuron-level concentration real (20%) but scattered — bundling destroys it; strong depth gradient (L27: 11%, early layers: ~80%). expB: Jaccard(t,t+1)=0.69 vs matched random null 0.64 (stickiness = set-size artifact); union working set 98% after 8 tokens, 99.9% after 128 (kill ≥95%); domain locality nil. - **Interpretation:** All three fine-grained routes for per-token escalation bytes are now closed by measurement: layers (expF), blocks (expA), temporal/domain caching (expB). The DejaVu/LLM-in-a-flash paradigm measurably does not transfer to dense SwiGLU models. G06/G12/G13 dead on this family; G07 survives only as static tiering. Remaining route is batch amortization, which expH's sequential numbers make attractive (full 1 GB residual streams in ~80 ms at 13 GB/s, shared across a queue of deferred low-margin tokens). **C1 converges into C2: margin-gated deferred refinement with periodic sequential sweeps.** - **Decision:** (1) Rerun expG at 8B to confirm margin-signal scaling; (2) then prototype candidate_01: 4-bit resident base + margin-gated deferral queue + periodic residual sweep with rollback, measured in bytes/token, tok/s, agreement, on this Mac. --- ## 2026-08-12 14:30 EDT — expG@8B (favorable scaling) + candidate_01 first run (mixed; metric insight) - **Question:** Does the margin signal hold at 8B, and does the selected architecture (margin-gated deferred refinement) work end-to-end on this Mac? - **Experiment:** expG rerun on Qwen3-8B (`results/expG_decision_stability/20260812T051129Z/`); candidate_01 prototype: q4 resident + windowed q8 sweeps + rollback via KV trim (`results/candidate_01/20260812T051501Z/`), 24 prompts × 128 tokens, W=32. - **Result:** 8B scaling favorable on every metric (4-bit: agree 87.4→92.1%, AUROC 0.90→0.92, esc@99% 36.6→22.6%). Candidate: bytes/token 154–231 MB logical (9–14× under q8 checkpoint, within expH budget) — PASSED; verify-all fidelity 0.883 = measured hardware ceiling (q8 self-agreement 1.56%/token prefill/decode flips ⇒ ~0.88 seq-ratio at 128 tokens); margin-mode seq-fidelity 0.55–0.65 — FAILED as registered, caused by divergence compounding of ~2% unchecked per-token error; throughput: margin τ=1.0 passed 1/3-of-q4, verify-all failed (full-context re-prefill per sweep — fixable via incremental verify-cache). - **Interpretation:** (1) Token-exact sequence reproduction is not a coherent target on Metal — the reference model cannot reproduce itself; quality-level metrics are now mandatory (§4.10 empirically forced). (2) The architecture's byte economics hold and improve with scale. (3) Sweep cost is an implementation artifact with a known fix. - **Decision:** Next cycle: quality-metric evaluation (bf16-judged logprobs, task checks), incremental verify-cache, then scale to a q8>RAM model — the regime this architecture is for. --- ## 2026-08-12 15:20 EDT — candidate_01 v2: quality claim CONFIRMED — q8-equivalent output with q4 resident - **Question:** Does quality-level evaluation vindicate the architecture, and does the incremental verify-cache fix sweep cost? - **Experiment:** candidate_01 v2 (`results/candidate_01/20260812T052415Z/`): persistent q8 verify KV cache (O(window) sweeps); all outputs judged by bf16 mean logprob. - **Result:** Judge scores — q4 floor -0.387; margin τ=1.0 -0.267 (47% of gap, 150 MB/token); τ=2.0 -0.176 (83%, 211 MB/token); verify-all -0.130 ≈ q8's -0.133 (indistinguishable, 237 MB/token, 9× under checkpoint). Throughput 69–110 tok/s at 1.7B. - **Interpretation:** The charter §16.D/§16.E criteria are now met at 1.7B scale: a progressive/conditional mechanism preserving quality while avoiding most weight loading, with a runtime quality↔bytes knob (τ) no static quantization offers. v1's fidelity failure confirmed as metric artifact. Remaining lever: rollback regeneration (accept q8 token runs from sweep logits — free speculative continuation). - **Decision:** Scale run at 32B (q4 17 GB resident, q8 35 GB streamed — q8 does not fit wired on 48 GB): the regime the architecture exists for. Artifacts downloading. --- ## 2026-08-12 16:40 EDT — 32B scale run: charter §16.A/B demonstrated in prototype form - **Question:** Does the architecture deliver in its target regime — a model configuration that does not fit in unified memory? - **Experiment:** Qwen3-32B on the 48 GB M5 Max: q4 resident (17.5 GB) + q8 (34.8 GB) layer-streamed per sweep via StreamingVerifier (materialize→compute→re-lazify; sequential SSD reads at ~11.6 GB/s ≈ expH ceiling). 12 prompts × 96 tokens. Results: `results/candidate_01_scale32b/20260812T055506Z/` (smoke: 20260812T053042Z). - **Result:** verify-all: judge (8B bf16) -0.4667 vs pure-q4 -0.7506 (+0.28 nats, 38% rel.); 1.69 tok/s; 3.72 GB/token logical (9.4× under streamed checkpoint); sweeps 3.0 s. Margin τ=2.0 intermediate (-0.546, 1.62 tok/s). q8 cannot run resident on this machine at all. - **Interpretation:** Total representation (52 GB) exceeds unified memory (48 GB) yet its quality is usable locally — bytes-per-token decoupled from checkpoint size with a strictly sequential access pattern. Below interactive (1.65 tok/s): measured headroom = larger windows, speculative acceptance of q8 runs from sweep logits, I/O-compute overlap (<5% contention per expH). Limits recorded: no q8 ceiling measurable here; 8B judge is a proxy; no window tuning. - **Decision:** Phase 11 novelty check on the assembled architecture next; then throughput levers or additional scale points. --- ## 2026-08-12 — Phase 11 novelty verification: composition survives narrowly; three components anticipated - **Question:** Charter §19 — is the candidate_01 architecture (resident q4 + margin-gated deferral + periodically streamed q8 verifier, bytes/token objective) already in the literature? Adversarial stance: assumed NOT novel. - **Experiment:** Targeted web/arXiv/GitHub search per component and for the composition (~45 sources verified, key papers read from full PDFs). Deliverable: `research/novelty_check.md`. - **Result:** Three 2025–2026 systems missed by the Phase 1 sweep materially shrink the claim: **Lever** (2605.16786 — DRAM-resident draft + flash-streamed target verifier, amortized per window, smartphones), **CATS** (2605.11186 — self-speculative shallow draft + verifier layers streamed from flash once per cycle, edge), **U-HLM/UGSD** (2412.12687 / 2603.11397 — draft-side uncertainty gates whether verification happens at all, cloud tier, incl. lossy top-R acceptance + rollback). QSpec/QuantSpec/ML-SpecQD own the shared-weights precision pair; mlx-flash owns MLX layer streaming (materialize→compute→release, every token). - **Interpretation:** "Resident draft + streamed verifier" and "uncertainty-gated verification skipping" are ANTICIPATED as categories. Surviving as not-found: (N1) margin-gating the **weight-streaming bytes** of the same model's higher-precision variant on one machine; (N2) bytes-streamed-per-token vs checkpoint size as explicit objective, with τ as a runtime bytes↔quality knob; (N3) the training-free same-checkpoint q4/q8 pair as the draft/verify split of an out-of-core runtime; (N4, engineering) the unified-memory Metal instantiation with incremental verify-KV-cache. All public claims must use the attribution table and README language in novelty_check.md §3–§4. - **Decision:** Proceed with narrowed claims only. Positioning for any writeup: extend Lever/CATS's regime with (a) precision-cast drafts (no training), (b) margin-priced verification I/O, (c) bytes/token accounting. Re-run this check before publication — prior-art density in this exact area is rising quarter over quarter. --- ## 2026-08-12 18:05 EDT — W=64 lever test: NEGATIVE — rollback rate, not window size, bounds amortization - **Question:** Does doubling the sweep window halve bytes/token? - **Experiment:** 32B verify-all, W=64 vs W=32 (`results/candidate_01_scale32b/20260812T061115Z/`). - **Result:** GB/token 3.72→3.53 (−5%, not −50%); tok/s 1.69→1.59; judge −0.467→−0.514 (n=12, noise). - **Interpretation:** Effective inter-sweep distance is bounded by ~1/rollback-rate (≈8–15 tokens at 12%/token), so W beyond that is dead weight. Real levers: τ-calibrated margin mode (fewer checked positions → fewer rollback triggers) and sweep-I/O/generation overlap. - **Decision:** Session milestone reached; next cycle: honest baselines (resident q5/q6 at 32B), τ calibration on the real q4-32B margin distribution, task-level benchmarks, 70B point. --- ## 2026-08-12 20:10 EDT — TR-01 published: first official technical report - **Question:** n/a (dissemination). - **Experiment:** n/a. - **Result:** `docs/publications/TR-01-margin-gated-deferred-refinement.md` — full write-up of results to date (substrate, decision stability, three refutations, architecture, 1.7B continuum, 32B out-of-core run), with 4 SVG figures regenerated from committed results by `tools/make_pub_figures.py`. Prior-art attribution follows novelty_check.md; limitations section names the missing baselines (resident q5/q6 at 32B, task benchmarks, 70B). Published on the platform under the new Publications tab. - **Interpretation:** n/a. - **Decision:** TR-02 planned once the honest-baseline campaign (q5/q6, tasks, 70B) completes.