spb/localvm-research Public License

1---2project: localvm-research3document: candidate_01/hypothesis4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6created: 2026-08-127modified: 2026-08-128status: reviewed9---1011# Hypothesis — candidate_01 (margin-gated deferred-refinement runtime)1213The architecture the micro-experiment campaign selected: expG (margins predict14disagreement, AUROC 0.9), expD (residual refinement converges; two-tier policy15works), expF/expA/expB (fine-grained escalation routes all dead → amortize),16expH (sequential residual sweeps are nearly free: ~13 GB/s).1718```text19Hypothesis20  A runtime holding only a 4-bit-class base model resident, generating21  optimistically while deferring low-margin tokens, and verifying windows22  with a periodically streamed 8-bit-class refinement (rollback on flips)23  achieves: (a) ≥95% token fidelity to the pure-q8 greedy output in24  margin-gated mode and 100% in verify-all mode; (b) residual/verify25  bytes-per-token an order of magnitude below the q8 checkpoint size per26  token (sweep bytes ÷ window); (c) throughput within 2× of pure-q427  generation. In verify-all mode the system IS precision-tiered28  speculative decoding (QSpec-family); margin-gated mode is the novel29  relaxation trading exactness for fewer rollbacks and sweeps.3031Falsification criterion32  If rollback overhead + sweep cost push throughput below 1/3 of pure-q4,33  or margin-gated fidelity falls below 90%, or measured bytes/token shows34  no order-of-magnitude advantage over resident-q8 execution, the35  architecture is not competitive on Apple Silicon and the project pivots36  to C3 (MoE residency) as primary candidate.3738Method39  Qwen3-1.7B. Build (offline): q4 and q8 MLX conversions (group 64).40  Run: resident q4 generates greedily with KV cache, recording margins.41  Tokens with margin < τ are deferred. Every W generated tokens (or at42  EOS), a sweep loads the q8 model weights, teacher-forces the window in43  one pass, and (mode=margin) checks deferred positions only /44  (mode=verify-all) checks all positions; on first flip: rollback (trim45  base KV), accept the q8 token, resume. Sweep bytes counted = q8 weight46  bytes actually (re)loaded; at 1.7B the page cache hides re-reads, so47  logical bytes are also reported for scale extrapolation (documented).48  Configs: τ ∈ {1.0, 2.0}, W ∈ {32}, both modes; 24 prompts × 128 tokens.49  Metrics: tok/s end-to-end, sweep latency, rollback rate, deferral rate,50  fidelity vs pure-q8 greedy, bytes/token (physical and logical).5152Baseline53  Pure-q4 greedy and pure-q8 greedy (both fully resident) on the same54  prompts — the honest brackets for speed and quality. No straw men:55  resident-q8 is what a 48 GB Mac would actually do at 1.7B; the value56  proposition targets models where q8 does NOT fit, so the reported57  advantage is bytes-per-token structure, not wall-clock at 1.7B.5859Result60  MIXED (see analysis.md). Bytes claim PASSED (154–231 MB/token logical,61  9–14× under the q8 checkpoint). Verify-all exact up to the measured62  hardware ceiling (q8 self-agreement is only ~0.88 seq-ratio at 12863  tokens: 1.56%/token prefill/decode flips). Margin-mode sequence64  fidelity FAILED as registered (0.55–0.65) — divergence compounding65  makes token-exact reproduction the wrong metric. Throughput: margin66  τ=1.0 passed the 1/3-of-q4 line; verify-all failed (full-context67  re-prefill per sweep — fixable).6869Interpretation70  Mechanism works and its costs are understood; evaluation must move to71  quality-level metrics (charter §4.10, now empirically forced). Scale72  trend (expG-8B) favors the architecture: deferral shrinks with size.7374Next experiment75  Quality-metric re-evaluation; incremental q8 verify-cache; then scale76  to a model whose q8 exceeds RAM.77```78