--- project: localvm-research document: candidate_01/analysis author: Simon-Pierre Boucher contact: contact@spboucher.ai created: 2026-08-12 status: reviewed --- # Analysis — candidate_01 (margin-gated deferred refinement), first prototype run Run: `results/candidate_01/20260812T051501Z/` · Qwen3-1.7B, q4 resident (1.15 GB) + q8 streamed for sweeps (2.15 GB), 24 prompts × 128 tokens, window 32. Baselines on identical prompts, warm kernels. ```text Config fid(seq) tok/s rollback/tok MB/token (logical) pure q4 (resident) 0.486 287.5 — 0 margin τ=1.0 0.552 99.6 0.067 154 margin τ=2.0 0.648 71.2 0.106 216 verify-all 0.883 63.6 0.117 231 pure q8 (resident) 1.000 200.1 — 0 (2.15 GB resident) Fidelity = difflib sequence ratio vs pure-q8 greedy output. Measured ceiling: q8 cannot reproduce ITSELF — 1.56%/token argmax flips between prefill and incremental decode (Metal numerics), so even a "perfect" verifier lands near 0.88 sequence ratio at 128 tokens. Verdict vs registered falsification criteria — MIXED (partial fail) (a) margin-gated fidelity ≥90%: FAILED as registered (0.55–0.65). (b) verify-all exactness: met up to the hardware ceiling (0.883 ≈ the measured prefill/decode bound; the mechanism made no error of its own — every miss traces to numerics or an unchecked position). (c) bytes: 154–231 MB/token logical — 9–14× less than the q8 checkpoint per sweep-window; within the expH 650 MB/token SSD budget. PASSED. (d) throughput ≥ pure-q4/3: margin τ=1.0 passed (99.6 ≥ 95.8); verify-all failed (63.6). Sweep cost is dominated by full-context re-prefill per sweep — an implementation artifact, not intrinsic. Interpretation 1. THE METRIC IS THE HEADLINE: exact sequence reproduction is not a coherent target on this hardware — the reference model itself only "agrees with itself" on ~87% of 128-token trajectories. Divergence compounding turns a 1–3% per-token disagreement into 35–50% sequence dissimilarity, which is why margin-mode's 97–98% per-token accuracy (expD) looks like 0.65 here. The charter's §4.10 distinction (same useful output ≠ same tokens) is now an empirical necessity, not a philosophical option: candidate evaluation must move to quality-level metrics (judge-scored logprob/perplexity of outputs, task accuracy). 2. The BYTES claim survives: a resident 4-bit base + streamed q8 sweeps costs ~150–230 MB/token logical at W=32 — order-of-magnitude below resident-q8 bytes, and the 8B expG rerun shows the deferral rate (hence sweep frequency) FALLS with scale (esc@99%: 36.6%→22.6%). 3. Known fixable inefficiency: sweeps re-prefill the whole context; an incremental q8 verify-cache (advance during sweeps, trim on rollback) cuts sweep cost from O(context) to O(window). Rollback regeneration (~16 tokens × 0.07–0.12/token) is the second cost. 4. At 1.7B this runtime is strictly worse than just running q8 — as expected and stated in the hypothesis; the target regime is models whose q8 does not fit in RAM. The mechanism now exists, is measured, and its costs are understood. Next experiment (1) Re-evaluate with quality-level metrics: score system outputs vs q4/q8 baselines with the bf16 model (mean logprob) and task-style checks — decides whether margin-mode's unchecked 2% actually costs usefulness. (2) Implement the incremental verify-cache and re-measure throughput. (3) If both pass, scale the prototype to a model whose q8 exceeds 48 GB (e.g., 70B-class q4 base) — the regime the architecture exists for. ``` ## Addendum — v2: incremental verify-cache + quality-level evaluation (2026-08-12) Run: `results/candidate_01/20260812T052415Z/` (24 prompts × 128 tokens, W=32). Changes: sweeps now feed only the unseen suffix into a persistent q8 KV cache (O(window) instead of O(context)); outputs scored by the bf16 model (mean per-token logprob of each system's generated text — the quality-level metric the v1 analysis showed is necessary). ```text Config bf16-judge logprob gap recovered tok/s MB/token pure q4 (floor) -0.3872 0% 287.5 0 margin τ=1.0 -0.2671 47% 110.3 150 margin τ=2.0 -0.1761 83% 78.1 211 verify-all -0.1297 ~100% 68.7 237 pure q8 (ceiling) -0.1328 100% 200.1 0 (2.15 GB resident) ``` Findings 1. QUALITY CLAIM CONFIRMED: verify-all output is statistically indistinguishable from resident-q8 (judge difference within noise, -0.1297 vs -0.1328) while holding only the q4 model in memory and streaming 237 MB/token — 9× below the q8 checkpoint. The v1 "fidelity failure" was an artifact of the token-exact metric, as hypothesized. 2. TUNABLE CONTINUUM: τ is a runtime knob trading verification bytes for quality (47% of the gap at 150 MB/token; 83% at 211; ~100% at 237) — the "quality becomes a runtime variable" property that no static quantization offers, now measured end-to-end. 3. Verify-cache gains are modest at short context (~10% tok/s) because rollback regeneration dominates; the O(window) property matters at long context. Rollback cost is the remaining lever (batch re-draft, or accept q8's token run from the sweep logits directly — free speculative continuation, unimplemented). 4. Throughput at 1.7B: 69–110 tok/s vs 287 (q4) / 200 (q8) resident. The architecture pays ~2–3× wall-clock at a scale where it isn't needed; the target regime (q8 > RAM) is downloading (Qwen3-32B q4/q8: 17 GB resident vs 35 GB streamed on a 48 GB machine). Next: scale run at 32B (q8 does not fit comfortably wired on this Mac) — bytes/token, tok/s, judge quality vs the only real alternative (pure q4). ## Addendum — scale run at 32B: the target regime (2026-08-12) Run: `results/candidate_01_scale32b/20260812T055506Z/` · Qwen3-32B on the 48 GB M5 Max: q4 (17.5 GB) resident; q8 (34.8 GB) **cannot be co-resident** — each sweep streams it layer-by-layer from SSD (materialize → compute → re-lazify), reading the checkpoint sequentially at ~11.6 GB/s (≈ the expH device ceiling). 12 prompts × 96 tokens, W=32. Judge: independent Qwen3-8B bf16 (the 32B bf16 obviously cannot run here). ```text Config judge logprob tok/s sweep GB/token (logical) pure q4 (resident) -0.7506 ~25 — 0 margin τ=2.0 -0.5460 1.62 3.0 s 3.93 verify-all -0.4667 1.69 3.0 s 3.72 (q8 resident: impossible on this machine — that is the point) ``` Findings 1. CHARTER §16.A/§16.B DEMONSTRATED IN PROTOTYPE FORM: a quality level whose weights do not fit in unified memory (q8-32B; q4+q8 = 52 GB total representation on a 48 GB machine) is made usable locally. Verified output improves the 8B-judge score by 0.28 nats over the only configuration that actually fits (pure q4) — 38% relative — and bytes streamed per token (3.7 GB) are 9.4× below the streamed checkpoint, with a strictly sequential SSD pattern at device ceiling. 2. Throughput 1.65 tok/s end-to-end — batch-usable, below interactive. The cost structure is now measured: 3.0 s/sweep (SSD-bound, ~1 sweep per ≤32 tokens plus rollback-triggered re-sweeps) + 25 tok/s base decode + rollback regeneration. Identified headroom, in order: (a) larger windows (W=64/128 halves/quarters GB/token and sweep frequency); (b) accepting q8 token runs directly from sweep logits (free speculative continuation — rollbacks currently regenerate at base speed then re-sweep); (c) overlapping sweep I/O with continued base generation (expH: GPU compute costs SSD reads <5%). 3. Honest limits: quality has no measurable ceiling on this machine (no resident q8 to compare against); the 8B judge is an imperfect proxy; W=32 was tuned for nothing — no sweep/window search was done; 1.65 tok/s is prototype speed, not a system claim. Next: Phase 11 novelty check on the assembled architecture (margin-gated deferral + layer-streamed amortized verification on unified memory), then either the throughput levers above or the 14B/70B scale points. ## Addendum — window lever test: W=64 (2026-08-12) Run: `results/candidate_01_scale32b/20260812T061115Z/` (verify-all, 32B, same 12 prompts × 96 tokens). ```text W=32 W=64 expected if sweeps ∝ 1/W GB/token 3.72 3.53 ~1.9 tok/s 1.69 1.59 higher judge logprob -0.467 -0.514 equal ``` NEGATIVE (lever mostly ineffective): doubling the window cut bytes/token by only 5%, not ~2×. Cause: in verify-all mode the rollback rate (~0.12/token) means a 64-token window almost always contains a flip; the sweep fires at the flip, the window restarts, and the effective inter-sweep distance is bounded by ~1/rollback-rate (≈8–15 tokens), not by W. Quality drifted slightly down (n=12, no error bars — treat as noise until repeated). Implication: sweep amortization is governed by ROLLBACK dynamics, not window size. The levers that can actually move throughput/bytes are (a) margin-mode with τ calibrated on the real quantized artifact's margin distribution (fewer checked positions → fewer flips that trigger rollbacks), and (b) overlapping sweep I/O with continued generation. W-tuning alone is dead.