spb/localvm-research Public License

1---2project: localvm-research3document: candidate_01/analysis4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6created: 2026-08-127status: reviewed8---910# Analysis — candidate_01 (margin-gated deferred refinement), first prototype run1112Run: `results/candidate_01/20260812T051501Z/` · Qwen3-1.7B, q4 resident13(1.15 GB) + q8 streamed for sweeps (2.15 GB), 24 prompts × 128 tokens,14window 32. Baselines on identical prompts, warm kernels.1516```text17Config                fid(seq)  tok/s   rollback/tok  MB/token (logical)18pure q4 (resident)     0.486    287.5        —            019margin  τ=1.0          0.552     99.6      0.067         15420margin  τ=2.0          0.648     71.2      0.106         21621verify-all             0.883     63.6      0.117         23122pure q8 (resident)     1.000    200.1        —            0 (2.15 GB resident)23Fidelity = difflib sequence ratio vs pure-q8 greedy output.24Measured ceiling: q8 cannot reproduce ITSELF — 1.56%/token argmax flips25between prefill and incremental decode (Metal numerics), so even a26"perfect" verifier lands near 0.88 sequence ratio at 128 tokens.2728Verdict vs registered falsification criteria — MIXED (partial fail)29  (a) margin-gated fidelity ≥90%: FAILED as registered (0.55–0.65).30  (b) verify-all exactness: met up to the hardware ceiling (0.883 ≈ the31      measured prefill/decode bound; the mechanism made no error of its32      own — every miss traces to numerics or an unchecked position).33  (c) bytes: 154–231 MB/token logical — 9–14× less than the q8 checkpoint34      per sweep-window; within the expH 650 MB/token SSD budget. PASSED.35  (d) throughput ≥ pure-q4/3: margin τ=1.0 passed (99.6 ≥ 95.8);36      verify-all failed (63.6). Sweep cost is dominated by full-context37      re-prefill per sweep — an implementation artifact, not intrinsic.3839Interpretation40  1. THE METRIC IS THE HEADLINE: exact sequence reproduction is not a41     coherent target on this hardware — the reference model itself only42     "agrees with itself" on ~87% of 128-token trajectories. Divergence43     compounding turns a 1–3% per-token disagreement into 35–50% sequence44     dissimilarity, which is why margin-mode's 97–98% per-token accuracy45     (expD) looks like 0.65 here. The charter's §4.10 distinction (same46     useful output ≠ same tokens) is now an empirical necessity, not a47     philosophical option: candidate evaluation must move to quality-level48     metrics (judge-scored logprob/perplexity of outputs, task accuracy).49  2. The BYTES claim survives: a resident 4-bit base + streamed q8 sweeps50     costs ~150–230 MB/token logical at W=32 — order-of-magnitude below51     resident-q8 bytes, and the 8B expG rerun shows the deferral rate52     (hence sweep frequency) FALLS with scale (esc@99%: 36.6%→22.6%).53  3. Known fixable inefficiency: sweeps re-prefill the whole context;54     an incremental q8 verify-cache (advance during sweeps, trim on55     rollback) cuts sweep cost from O(context) to O(window). Rollback56     regeneration (~16 tokens × 0.07–0.12/token) is the second cost.57  4. At 1.7B this runtime is strictly worse than just running q8 — as58     expected and stated in the hypothesis; the target regime is models59     whose q8 does not fit in RAM. The mechanism now exists, is measured,60     and its costs are understood.6162Next experiment63  (1) Re-evaluate with quality-level metrics: score system outputs vs64      q4/q8 baselines with the bf16 model (mean logprob) and task-style65      checks — decides whether margin-mode's unchecked 2% actually costs66      usefulness. (2) Implement the incremental verify-cache and re-measure67      throughput. (3) If both pass, scale the prototype to a model whose68      q8 exceeds 48 GB (e.g., 70B-class q4 base) — the regime the69      architecture exists for.70```7172## Addendum — v2: incremental verify-cache + quality-level evaluation (2026-08-12)7374Run: `results/candidate_01/20260812T052415Z/` (24 prompts × 128 tokens, W=32).75Changes: sweeps now feed only the unseen suffix into a persistent q8 KV cache76(O(window) instead of O(context)); outputs scored by the bf16 model77(mean per-token logprob of each system's generated text — the quality-level78metric the v1 analysis showed is necessary).7980```text81Config           bf16-judge logprob   gap recovered   tok/s   MB/token82pure q4 (floor)      -0.3872                0%        287.5      083margin τ=1.0         -0.2671               47%        110.3    15084margin τ=2.0         -0.1761               83%         78.1    21185verify-all           -0.1297             ~100%         68.7    23786pure q8 (ceiling)    -0.1328              100%        200.1      0 (2.15 GB resident)87```8889Findings901. QUALITY CLAIM CONFIRMED: verify-all output is statistically91   indistinguishable from resident-q8 (judge difference within noise,92   -0.1297 vs -0.1328) while holding only the q4 model in memory and93   streaming 237 MB/token — 9× below the q8 checkpoint. The v1 "fidelity94   failure" was an artifact of the token-exact metric, as hypothesized.952. TUNABLE CONTINUUM: τ is a runtime knob trading verification bytes for96   quality (47% of the gap at 150 MB/token; 83% at 211; ~100% at 237) —97   the "quality becomes a runtime variable" property that no static98   quantization offers, now measured end-to-end.993. Verify-cache gains are modest at short context (~10% tok/s) because100   rollback regeneration dominates; the O(window) property matters at101   long context. Rollback cost is the remaining lever (batch re-draft,102   or accept q8's token run from the sweep logits directly — free103   speculative continuation, unimplemented).1044. Throughput at 1.7B: 69–110 tok/s vs 287 (q4) / 200 (q8) resident.105   The architecture pays ~2–3× wall-clock at a scale where it isn't106   needed; the target regime (q8 > RAM) is downloading (Qwen3-32B q4/q8:107   17 GB resident vs 35 GB streamed on a 48 GB machine).108109Next: scale run at 32B (q8 does not fit comfortably wired on this Mac) —110bytes/token, tok/s, judge quality vs the only real alternative (pure q4).111112## Addendum — scale run at 32B: the target regime (2026-08-12)113114Run: `results/candidate_01_scale32b/20260812T055506Z/` · Qwen3-32B on the11548 GB M5 Max: q4 (17.5 GB) resident; q8 (34.8 GB) **cannot be co-resident** —116each sweep streams it layer-by-layer from SSD (materialize → compute →117re-lazify), reading the checkpoint sequentially at ~11.6 GB/s (≈ the expH118device ceiling). 12 prompts × 96 tokens, W=32. Judge: independent Qwen3-8B119bf16 (the 32B bf16 obviously cannot run here).120121```text122Config                judge logprob   tok/s   sweep    GB/token (logical)123pure q4 (resident)       -0.7506      ~25      —          0124margin  τ=2.0            -0.5460      1.62    3.0 s      3.93125verify-all               -0.4667      1.69    3.0 s      3.72126(q8 resident: impossible on this machine — that is the point)127```128129Findings1301. CHARTER §16.A/§16.B DEMONSTRATED IN PROTOTYPE FORM: a quality level131   whose weights do not fit in unified memory (q8-32B; q4+q8 = 52 GB132   total representation on a 48 GB machine) is made usable locally.133   Verified output improves the 8B-judge score by 0.28 nats over the134   only configuration that actually fits (pure q4) — 38% relative — and135   bytes streamed per token (3.7 GB) are 9.4× below the streamed136   checkpoint, with a strictly sequential SSD pattern at device ceiling.1372. Throughput 1.65 tok/s end-to-end — batch-usable, below interactive.138   The cost structure is now measured: 3.0 s/sweep (SSD-bound, ~1 sweep139   per ≤32 tokens plus rollback-triggered re-sweeps) + 25 tok/s base140   decode + rollback regeneration. Identified headroom, in order:141   (a) larger windows (W=64/128 halves/quarters GB/token and sweep142   frequency); (b) accepting q8 token runs directly from sweep logits143   (free speculative continuation — rollbacks currently regenerate at144   base speed then re-sweep); (c) overlapping sweep I/O with continued145   base generation (expH: GPU compute costs SSD reads <5%).1463. Honest limits: quality has no measurable ceiling on this machine147   (no resident q8 to compare against); the 8B judge is an imperfect148   proxy; W=32 was tuned for nothing — no sweep/window search was done;149   1.65 tok/s is prototype speed, not a system claim.150151Next: Phase 11 novelty check on the assembled architecture (margin-gated152deferral + layer-streamed amortized verification on unified memory), then153either the throughput levers above or the 14B/70B scale points.154155## Addendum — window lever test: W=64 (2026-08-12)156157Run: `results/candidate_01_scale32b/20260812T061115Z/` (verify-all, 32B, same15812 prompts × 96 tokens).159160```text161                W=32      W=64      expected if sweeps ∝ 1/W162GB/token        3.72      3.53      ~1.9163tok/s           1.69      1.59      higher164judge logprob   -0.467    -0.514    equal165```166167NEGATIVE (lever mostly ineffective): doubling the window cut bytes/token by168only 5%, not ~2×. Cause: in verify-all mode the rollback rate (~0.12/token)169means a 64-token window almost always contains a flip; the sweep fires at the170flip, the window restarts, and the effective inter-sweep distance is bounded171by ~1/rollback-rate (≈8–15 tokens), not by W. Quality drifted slightly down172(n=12, no error bars — treat as noise until repeated).173174Implication: sweep amortization is governed by ROLLBACK dynamics, not window175size. The levers that can actually move throughput/bytes are (a) margin-mode176with τ calibrated on the real quantized artifact's margin distribution177(fewer checked positions → fewer flips that trigger rollbacks), and178(b) overlapping sweep I/O with continued generation. W-tuning alone is dead.179