---
project: localvm-research
document: candidate_01/hypothesis
author: Simon-Pierre Boucher
contact: contact@spboucher.ai
created: 2026-08-12
modified: 2026-08-12
status: reviewed
---

# Hypothesis — candidate_01 (margin-gated deferred-refinement runtime)

The architecture the micro-experiment campaign selected: expG (margins predict
disagreement, AUROC 0.9), expD (residual refinement converges; two-tier policy
works), expF/expA/expB (fine-grained escalation routes all dead → amortize),
expH (sequential residual sweeps are nearly free: ~13 GB/s).

```text
Hypothesis
  A runtime holding only a 4-bit-class base model resident, generating
  optimistically while deferring low-margin tokens, and verifying windows
  with a periodically streamed 8-bit-class refinement (rollback on flips)
  achieves: (a) ≥95% token fidelity to the pure-q8 greedy output in
  margin-gated mode and 100% in verify-all mode; (b) residual/verify
  bytes-per-token an order of magnitude below the q8 checkpoint size per
  token (sweep bytes ÷ window); (c) throughput within 2× of pure-q4
  generation. In verify-all mode the system IS precision-tiered
  speculative decoding (QSpec-family); margin-gated mode is the novel
  relaxation trading exactness for fewer rollbacks and sweeps.

Falsification criterion
  If rollback overhead + sweep cost push throughput below 1/3 of pure-q4,
  or margin-gated fidelity falls below 90%, or measured bytes/token shows
  no order-of-magnitude advantage over resident-q8 execution, the
  architecture is not competitive on Apple Silicon and the project pivots
  to C3 (MoE residency) as primary candidate.

Method
  Qwen3-1.7B. Build (offline): q4 and q8 MLX conversions (group 64).
  Run: resident q4 generates greedily with KV cache, recording margins.
  Tokens with margin < τ are deferred. Every W generated tokens (or at
  EOS), a sweep loads the q8 model weights, teacher-forces the window in
  one pass, and (mode=margin) checks deferred positions only /
  (mode=verify-all) checks all positions; on first flip: rollback (trim
  base KV), accept the q8 token, resume. Sweep bytes counted = q8 weight
  bytes actually (re)loaded; at 1.7B the page cache hides re-reads, so
  logical bytes are also reported for scale extrapolation (documented).
  Configs: τ ∈ {1.0, 2.0}, W ∈ {32}, both modes; 24 prompts × 128 tokens.
  Metrics: tok/s end-to-end, sweep latency, rollback rate, deferral rate,
  fidelity vs pure-q8 greedy, bytes/token (physical and logical).

Baseline
  Pure-q4 greedy and pure-q8 greedy (both fully resident) on the same
  prompts — the honest brackets for speed and quality. No straw men:
  resident-q8 is what a 48 GB Mac would actually do at 1.7B; the value
  proposition targets models where q8 does NOT fit, so the reported
  advantage is bytes-per-token structure, not wall-clock at 1.7B.

Result
  MIXED (see analysis.md). Bytes claim PASSED (154–231 MB/token logical,
  9–14× under the q8 checkpoint). Verify-all exact up to the measured
  hardware ceiling (q8 self-agreement is only ~0.88 seq-ratio at 128
  tokens: 1.56%/token prefill/decode flips). Margin-mode sequence
  fidelity FAILED as registered (0.55–0.65) — divergence compounding
  makes token-exact reproduction the wrong metric. Throughput: margin
  τ=1.0 passed the 1/3-of-q4 line; verify-all failed (full-context
  re-prefill per sweep — fixable).

Interpretation
  Mechanism works and its costs are understood; evaluation must move to
  quality-level metrics (charter §4.10, now empirically forced). Scale
  trend (expG-8B) favors the architecture: deferral shrinks with size.

Next experiment
  Quality-metric re-evaluation; incremental q8 verify-cache; then scale
  to a model whose q8 exceeds RAM.
```