---
project: localvm-research
document: expG_decision_stability/analysis
author: Simon-Pierre Boucher
contact: contact@spboucher.ai
created: 2026-08-12
modified: 2026-08-12
status: reviewed
---

# Analysis — expG_decision_stability

Run: `results/expG_decision_stability/20260812T042744Z/` · code committed before run.
Model: Qwen3-1.7B (mlx-community bf16 reference), MLX affine quantization
(group 64, all divisible Linear layers) at 2/3/4/8 bits. 48 greedy reference
trajectories (8 per domain × 6 domains), 128 tokens each → 6 144 scored
positions per bit-width. Hardware: M5 Max / 48 GB / macOS 27 (manifest embedded).

```text
Hypothesis
  Margin of the cheap pass predicts disagreement with the full model:
  AUROC ≥ 0.8 at 3–4 bits; escalating 10–30% of tokens recovers ≥98–99%.

Falsification criterion
  AUROC < 0.65 at every bit-width, or >50% escalation needed at 4-bit.
  → NOT triggered.

Result (n = 6 144 positions/bit-width)
  bits  agree    mean KL   AUROC   esc@99%   median margin agree/disagree
   2    0.0296   11.89     0.605   99.0%     1.50 / 1.00
   3    0.7310    0.78     0.852   60.2%     5.25 / 1.12
   4    0.8740    0.199    0.898   36.6%     6.50 / 0.75
   8    0.9803    0.0039   0.975    5.1%     6.25 / 0.25
  Per-domain AUROC is uniform (4-bit: 0.880–0.921; math best, chat worst);
  no domain inverts the signal. Sanity: 4-bit agreement (87.4%) sits near
  published ~90–91% 'same top token' rates for 4-bit k-quants on larger
  models — plausible for a 1.7B model with plain affine quantization.

Interpretation
  1. PROMOTE G02 (MarginGate): AUROC 0.852–0.975 clears the 0.8 promote
     threshold at 3, 4, and 8 bits. Disagreements concentrate sharply at
     small margins (median disagree-margin ≤ 1.1 vs agree-margin ≥ 5.25).
     The margin signal is nearly free (it falls out of the cheap forward
     pass), and it generalizes across all six domains including reasoning.
  2. G23 (ConfidentlyWrong, contrarian) is weakened but not fully dead:
     at 4-bit, reaching 99% agreement still requires escalating 36.6% of
     tokens — above the hypothesized 10–30% band, though far below the 50%
     kill line. The gate works; whether it is *cheap enough* now depends on
     the per-escalation byte cost (expD's question, not this experiment's).
  3. NAIVE 2-BIT AFFINE IS DEAD as a resident base: 3.0% agreement,
     KL 11.9, AUROC 0.605 — the model leaves the pretrained basin entirely,
     confirming the ParetoQ 2–3-bit cliff on modern SwiGLU models with
     amplitude that perplexity tables understate. Candidate C1's resident
     base must therefore be ≥3-bit affine, or a codebook/trellis 2-bit
     format (which requires the G05 Metal kernel work). This partially
     answers G24 (BasinCollapse) for the affine family.
  4. The 8-bit row shows the ceiling: 98% agreement, 5.1% escalation for
     99% — an 8-bit-base system would stream almost nothing but also saves
     only 2× memory. The interesting operating region is a 4-bit-class base
     (0.5 byte/param resident) escalating ~1/3 of tokens, or a 3-bit base
     escalating more; which wins depends on residual fetch granularity.
  5. Note the escalation curves in results.json give the full frontier
     (escalated fraction vs residual disagreement) for policy design.

Next experiment
  expD (progressive reconstruction): measure how many *bytes* of residual
  (4→8-bit planes, per-layer or per-block) an escalated token actually
  needs to flip its decision to the reference — the product
  (escalation rate × bytes/escalation) against the expH budget
  (~650 MB/token) decides candidate C1's viability. Secondary: rerun expG
  at 3B–8B scale to check the expected agreement improvement with size.
```

## Addendum — scale check at 8B (2026-08-12)

Run: `results/expG_decision_stability/20260812T051129Z/` — same protocol,
Qwen3-8B bf16 reference, bits {3,4,8}.

```text
bits   agree(1.7B → 8B)      AUROC(1.7B → 8B)    esc@99%(1.7B → 8B)
 3     0.731 → 0.853         0.852 → 0.883       60.2% → 41.0%
 4     0.874 → 0.921         0.898 → 0.920       36.6% → 22.6%
 8     0.980 → 0.987         0.975 → 0.981        5.1% →  1.6%
```

Every metric improves with scale: bigger models are more robust to
quantization (consistent with published trends), margins are MORE
informative, and the escalation budget SHRINKS — at 8B a 4-bit base needs
q8-level correction on only ~23% of tokens for 99% agreement. The
margin-gated architecture's economics improve exactly where the project
needs it (the 30B+ regime where checkpoints stop fitting in RAM).