# project: localvm-research document: candidate_01/analysis author: Simon-Pierre Boucher contact: contact@spboucher.ai created: 2026-08-12 status: reviewed

# Analysis — candidate_01 (margin-gated deferred refinement), first prototype run

Run: results/candidate_01/20260812T051501Z/ · Qwen3-1.7B, q4 resident (1.15 GB) + q8 streamed for sweeps (2.15 GB), 24 prompts × 128 tokens, window 32. Baselines on identical prompts, warm kernels.

Config                fid(seq)  tok/s   rollback/tok  MB/token (logical)
pure q4 (resident)     0.486    287.5        —            0
margin  τ=1.0          0.552     99.6      0.067         154
margin  τ=2.0          0.648     71.2      0.106         216
verify-all             0.883     63.6      0.117         231
pure q8 (resident)     1.000    200.1        —            0 (2.15 GB resident)
Fidelity = difflib sequence ratio vs pure-q8 greedy output.
Measured ceiling: q8 cannot reproduce ITSELF — 1.56%/token argmax flips
between prefill and incremental decode (Metal numerics), so even a
"perfect" verifier lands near 0.88 sequence ratio at 128 tokens.

Verdict vs registered falsification criteria — MIXED (partial fail)
  (a) margin-gated fidelity ≥90%: FAILED as registered (0.55–0.65).
  (b) verify-all exactness: met up to the hardware ceiling (0.883 ≈ the
      measured prefill/decode bound; the mechanism made no error of its
      own — every miss traces to numerics or an unchecked position).
  (c) bytes: 154–231 MB/token logical — 9–14× less than the q8 checkpoint
      per sweep-window; within the expH 650 MB/token SSD budget. PASSED.
  (d) throughput ≥ pure-q4/3: margin τ=1.0 passed (99.6 ≥ 95.8);
      verify-all failed (63.6). Sweep cost is dominated by full-context
      re-prefill per sweep — an implementation artifact, not intrinsic.

Interpretation
  1. THE METRIC IS THE HEADLINE: exact sequence reproduction is not a
     coherent target on this hardware — the reference model itself only
     "agrees with itself" on ~87% of 128-token trajectories. Divergence
     compounding turns a 1–3% per-token disagreement into 35–50% sequence
     dissimilarity, which is why margin-mode's 97–98% per-token accuracy
     (expD) looks like 0.65 here. The charter's §4.10 distinction (same
     useful output ≠ same tokens) is now an empirical necessity, not a
     philosophical option: candidate evaluation must move to quality-level
     metrics (judge-scored logprob/perplexity of outputs, task accuracy).
  2. The BYTES claim survives: a resident 4-bit base + streamed q8 sweeps
     costs ~150–230 MB/token logical at W=32 — order-of-magnitude below
     resident-q8 bytes, and the 8B expG rerun shows the deferral rate
     (hence sweep frequency) FALLS with scale (esc@99%: 36.6%→22.6%).
  3. Known fixable inefficiency: sweeps re-prefill the whole context;
     an incremental q8 verify-cache (advance during sweeps, trim on
     rollback) cuts sweep cost from O(context) to O(window). Rollback
     regeneration (~16 tokens × 0.07–0.12/token) is the second cost.
  4. At 1.7B this runtime is strictly worse than just running q8 — as
     expected and stated in the hypothesis; the target regime is models
     whose q8 does not fit in RAM. The mechanism now exists, is measured,
     and its costs are understood.

Next experiment
  (1) Re-evaluate with quality-level metrics: score system outputs vs
      q4/q8 baselines with the bf16 model (mean logprob) and task-style
      checks — decides whether margin-mode's unchecked 2% actually costs
      usefulness. (2) Implement the incremental verify-cache and re-measure
      throughput. (3) If both pass, scale the prototype to a model whose
      q8 exceeds 48 GB (e.g., 70B-class q4 base) — the regime the
      architecture exists for.

# Addendum — v2: incremental verify-cache + quality-level evaluation (2026-08-12)

Run: results/candidate_01/20260812T052415Z/ (24 prompts × 128 tokens, W=32). Changes: sweeps now feed only the unseen suffix into a persistent q8 KV cache (O(window) instead of O(context)); outputs scored by the bf16 model (mean per-token logprob of each system's generated text — the quality-level metric the v1 analysis showed is necessary).

Config           bf16-judge logprob   gap recovered   tok/s   MB/token
pure q4 (floor)      -0.3872                0%        287.5      0
margin τ=1.0         -0.2671               47%        110.3    150
margin τ=2.0         -0.1761               83%         78.1    211
verify-all           -0.1297             ~100%         68.7    237
pure q8 (ceiling)    -0.1328              100%        200.1      0 (2.15 GB resident)

Findings

1. QUALITY CLAIM CONFIRMED: verify-all output is statistically indistinguishable from resident-q8 (judge difference within noise, -0.1297 vs -0.1328) while holding only the q4 model in memory and streaming 237 MB/token — 9× below the q8 checkpoint. The v1 "fidelity failure" was an artifact of the token-exact metric, as hypothesized.
2. TUNABLE CONTINUUM: τ is a runtime knob trading verification bytes for quality (47% of the gap at 150 MB/token; 83% at 211; ~100% at 237) — the "quality becomes a runtime variable" property that no static quantization offers, now measured end-to-end.
3. Verify-cache gains are modest at short context (~10% tok/s) because rollback regeneration dominates; the O(window) property matters at long context. Rollback cost is the remaining lever (batch re-draft, or accept q8's token run from the sweep logits directly — free speculative continuation, unimplemented).
4. Throughput at 1.7B: 69–110 tok/s vs 287 (q4) / 200 (q8) resident. The architecture pays ~2–3× wall-clock at a scale where it isn't needed; the target regime (q8 > RAM) is downloading (Qwen3-32B q4/q8: 17 GB resident vs 35 GB streamed on a 48 GB machine).

Next: scale run at 32B (q8 does not fit comfortably wired on this Mac) — bytes/token, tok/s, judge quality vs the only real alternative (pure q4).

# Addendum — scale run at 32B: the target regime (2026-08-12)

Run: results/candidate_01_scale32b/20260812T055506Z/ · Qwen3-32B on the 48 GB M5 Max: q4 (17.5 GB) resident; q8 (34.8 GB) cannot be co-resident — each sweep streams it layer-by-layer from SSD (materialize → compute → re-lazify), reading the checkpoint sequentially at ~11.6 GB/s (≈ the expH device ceiling). 12 prompts × 96 tokens, W=32. Judge: independent Qwen3-8B bf16 (the 32B bf16 obviously cannot run here).

Config                judge logprob   tok/s   sweep    GB/token (logical)
pure q4 (resident)       -0.7506      ~25      —          0
margin  τ=2.0            -0.5460      1.62    3.0 s      3.93
verify-all               -0.4667      1.69    3.0 s      3.72
(q8 resident: impossible on this machine — that is the point)

Findings

1. CHARTER §16.A/§16.B DEMONSTRATED IN PROTOTYPE FORM: a quality level whose weights do not fit in unified memory (q8-32B; q4+q8 = 52 GB total representation on a 48 GB machine) is made usable locally. Verified output improves the 8B-judge score by 0.28 nats over the only configuration that actually fits (pure q4) — 38% relative — and bytes streamed per token (3.7 GB) are 9.4× below the streamed checkpoint, with a strictly sequential SSD pattern at device ceiling.
2. Throughput 1.65 tok/s end-to-end — batch-usable, below interactive. The cost structure is now measured: 3.0 s/sweep (SSD-bound, ~1 sweep per ≤32 tokens plus rollback-triggered re-sweeps) + 25 tok/s base decode + rollback regeneration. Identified headroom, in order: (a) larger windows (W=64/128 halves/quarters GB/token and sweep frequency); (b) accepting q8 token runs directly from sweep logits (free speculative continuation — rollbacks currently regenerate at base speed then re-sweep); (c) overlapping sweep I/O with continued base generation (expH: GPU compute costs SSD reads <5%).
3. Honest limits: quality has no measurable ceiling on this machine (no resident q8 to compare against); the 8B judge is an imperfect proxy; W=32 was tuned for nothing — no sweep/window search was done; 1.65 tok/s is prototype speed, not a system claim.

Next: Phase 11 novelty check on the assembled architecture (margin-gated deferral + layer-streamed amortized verification on unified memory), then either the throughput levers above or the 14B/70B scale points.

# Addendum — window lever test: W=64 (2026-08-12)

Run: results/candidate_01_scale32b/20260812T061115Z/ (verify-all, 32B, same 12 prompts × 96 tokens).

                W=32      W=64      expected if sweeps ∝ 1/W
GB/token        3.72      3.53      ~1.9
tok/s           1.69      1.59      higher
judge logprob   -0.467    -0.514    equal

NEGATIVE (lever mostly ineffective): doubling the window cut bytes/token by only 5%, not ~2×. Cause: in verify-all mode the rollback rate (~0.12/token) means a 64-token window almost always contains a flip; the sweep fires at the flip, the window restarts, and the effective inter-sweep distance is bounded by ~1/rollback-rate (≈8–15 tokens), not by W. Quality drifted slightly down (n=12, no error bars — treat as noise until repeated).

Implication: sweep amortization is governed by ROLLBACK dynamics, not window size. The levers that can actually move throughput/bytes are (a) margin-mode with τ calibrated on the real quantized artifact's margin distribution (fewer checked positions → fewer flips that trigger rollbacks), and (b) overlapping sweep I/O with continued generation. W-tuning alone is dead.