spb/localvm-research Public License

1---2project: localvm-research3document: expG_decision_stability/hypothesis4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6created: 2026-08-127modified: 2026-08-128status: reviewed9---1011# Hypothesis — expG_decision_stability1213The Gate-Zero experiment (candidate_ranking.md C4): measures the joint14distribution of (cheap-pass top-1 margin × agreement with the full-precision15model) that the literature does not report, and which decides whether16margin-gated escalation (G02 MarginGate, and with it candidate C1) is viable.1718```text19Hypothesis20  For a small pretrained LLM, the top-1 logit margin of a low-bit quantized21  variant is a strong predictor of whether its greedy token decision matches22  the full-precision model: disagreements concentrate at small margins.23  Concretely, margin achieves AUROC ≥ 0.8 as a disagreement detector at24  3–4 bits, so that escalating only the low-margin fraction of tokens25  (expected 10–30%) recovers ≥98–99% greedy agreement.2627Falsification criterion28  If AUROC < 0.65 at every bit-width, or if reaching 99% agreement requires29  escalating > 50% of tokens at 4-bit (i.e., disagreements are NOT30  concentrated at low margins), then decision-uncertainty is not a usable31  gating signal: G02/G23 die and candidate C1 must be redesigned around32  amortized verification (C2) instead.3334Method35  Model: Qwen3-1.7B (bf16 reference; representative modern architecture,36  runs fully in memory). Variants: MLX affine quantization at 2, 3, 4, 837  bits (group size 64), quantizing all linear layers.38  For each of 48 prompts across 6 domains (code, math, chat, FR, EN,39  reasoning; benchmarks/datasets/eval_prompts.json): generate 128 tokens40  greedily with the bf16 reference, then teacher-force every variant over41  the same token sequence and record, per position: variant top-1 margin42  (logit gap between its top-2 candidates), variant argmax, reference43  argmax, agreement bit, and KL(ref||variant).44  Analysis: per bit-width and per domain — agreement rate; margin45  distributions conditioned on agree/disagree; AUROC of margin as a46  disagreement detector; escalation curve (fraction of tokens with margin47  below τ vs residual disagreement rate after escalating those tokens);48  the operating point (escalation fraction) that reaches 99% agreement.49  Seed fixed (greedy decode; deterministic). Hardware manifest embedded.5051Baseline52  The 8-bit variant (known near-lossless: expected ≥99% agreement without53  any gating) anchors the top of the scale; published aggregate agreement54  rates for 4-bit (~90–91%, llama.cpp 'same top p' data) sanity-check our55  pipeline. No straw men: the comparison is against unconditional56  escalation (loading full weights for all tokens).5758Result59  CONFIRMED (kill criteria not triggered). AUROC: 0.852 (3-bit), 0.89860  (4-bit), 0.975 (8-bit); esc@99%: 60.2% / 36.6% / 5.1%. Naive affine61  2-bit collapsed entirely (3% agreement, AUROC 0.605) — the ParetoQ62  cliff, measured. Full numbers: results/expG_decision_stability/63  20260812T042744Z/ and analysis.md.6465Interpretation66  Margin is a strong, free, domain-uniform gating signal → G02 promoted.67  4-bit escalation need (36.6%) exceeds the hoped 10–30% band, so C1's68  economics now hinge on bytes-per-escalation (expD). Resident base must69  be ≥3-bit affine or codebook 2-bit (G05).7071Next experiment72  expD — bytes of residual needed per escalated token; then expG rerun at73  3B–8B scale.74```75