spb/localvm-research Public License

1---2project: localvm-research3document: expG_decision_stability/analysis4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6created: 2026-08-127modified: 2026-08-128status: reviewed9---1011# Analysis — expG_decision_stability1213Run: `results/expG_decision_stability/20260812T042744Z/` · code committed before run.14Model: Qwen3-1.7B (mlx-community bf16 reference), MLX affine quantization15(group 64, all divisible Linear layers) at 2/3/4/8 bits. 48 greedy reference16trajectories (8 per domain × 6 domains), 128 tokens each → 6 144 scored17positions per bit-width. Hardware: M5 Max / 48 GB / macOS 27 (manifest embedded).1819```text20Hypothesis21  Margin of the cheap pass predicts disagreement with the full model:22  AUROC ≥ 0.8 at 3–4 bits; escalating 10–30% of tokens recovers ≥98–99%.2324Falsification criterion25  AUROC < 0.65 at every bit-width, or >50% escalation needed at 4-bit.26  → NOT triggered.2728Result (n = 6 144 positions/bit-width)29  bits  agree    mean KL   AUROC   esc@99%   median margin agree/disagree30   2    0.0296   11.89     0.605   99.0%     1.50 / 1.0031   3    0.7310    0.78     0.852   60.2%     5.25 / 1.1232   4    0.8740    0.199    0.898   36.6%     6.50 / 0.7533   8    0.9803    0.0039   0.975    5.1%     6.25 / 0.2534  Per-domain AUROC is uniform (4-bit: 0.880–0.921; math best, chat worst);35  no domain inverts the signal. Sanity: 4-bit agreement (87.4%) sits near36  published ~90–91% 'same top token' rates for 4-bit k-quants on larger37  models — plausible for a 1.7B model with plain affine quantization.3839Interpretation40  1. PROMOTE G02 (MarginGate): AUROC 0.852–0.975 clears the 0.8 promote41     threshold at 3, 4, and 8 bits. Disagreements concentrate sharply at42     small margins (median disagree-margin ≤ 1.1 vs agree-margin ≥ 5.25).43     The margin signal is nearly free (it falls out of the cheap forward44     pass), and it generalizes across all six domains including reasoning.45  2. G23 (ConfidentlyWrong, contrarian) is weakened but not fully dead:46     at 4-bit, reaching 99% agreement still requires escalating 36.6% of47     tokens — above the hypothesized 10–30% band, though far below the 50%48     kill line. The gate works; whether it is *cheap enough* now depends on49     the per-escalation byte cost (expD's question, not this experiment's).50  3. NAIVE 2-BIT AFFINE IS DEAD as a resident base: 3.0% agreement,51     KL 11.9, AUROC 0.605 — the model leaves the pretrained basin entirely,52     confirming the ParetoQ 2–3-bit cliff on modern SwiGLU models with53     amplitude that perplexity tables understate. Candidate C1's resident54     base must therefore be ≥3-bit affine, or a codebook/trellis 2-bit55     format (which requires the G05 Metal kernel work). This partially56     answers G24 (BasinCollapse) for the affine family.57  4. The 8-bit row shows the ceiling: 98% agreement, 5.1% escalation for58     99% — an 8-bit-base system would stream almost nothing but also saves59     only 2× memory. The interesting operating region is a 4-bit-class base60     (0.5 byte/param resident) escalating ~1/3 of tokens, or a 3-bit base61     escalating more; which wins depends on residual fetch granularity.62  5. Note the escalation curves in results.json give the full frontier63     (escalated fraction vs residual disagreement) for policy design.6465Next experiment66  expD (progressive reconstruction): measure how many *bytes* of residual67  (4→8-bit planes, per-layer or per-block) an escalated token actually68  needs to flip its decision to the reference — the product69  (escalation rate × bytes/escalation) against the expH budget70  (~650 MB/token) decides candidate C1's viability. Secondary: rerun expG71  at 3B–8B scale to check the expected agreement improvement with size.72```7374## Addendum — scale check at 8B (2026-08-12)7576Run: `results/expG_decision_stability/20260812T051129Z/` — same protocol,77Qwen3-8B bf16 reference, bits {3,4,8}.7879```text80bits   agree(1.7B → 8B)      AUROC(1.7B → 8B)    esc@99%(1.7B → 8B)81 3     0.731 → 0.853         0.852 → 0.883       60.2% → 41.0%82 4     0.874 → 0.921         0.898 → 0.920       36.6% → 22.6%83 8     0.980 → 0.987         0.975 → 0.981        5.1% →  1.6%84```8586Every metric improves with scale: bigger models are more robust to87quantization (consistent with published trends), margins are MORE88informative, and the escalation budget SHRINKS — at 8B a 4-bit base needs89q8-level correction on only ~23% of tokens for 99% agreement. The90margin-gated architecture's economics improve exactly where the project91needs it (the 30B+ regime where checkpoints stop fitting in RAM).92