project: localvm-research document: expG_decision_stability/hypothesis author: Simon-Pierre Boucher contact: contact@spboucher.ai created: 2026-08-12 modified: 2026-08-12 status: reviewed
Hypothesis — expG_decision_stability
The Gate-Zero experiment (candidate_ranking.md C4): measures the joint distribution of (cheap-pass top-1 margin × agreement with the full-precision model) that the literature does not report, and which decides whether margin-gated escalation (G02 MarginGate, and with it candidate C1) is viable.
text
Hypothesis
For a small pretrained LLM, the top-1 logit margin of a low-bit quantized
variant is a strong predictor of whether its greedy token decision matches
the full-precision model: disagreements concentrate at small margins.
Concretely, margin achieves AUROC ≥ 0.8 as a disagreement detector at
3–4 bits, so that escalating only the low-margin fraction of tokens
(expected 10–30%) recovers ≥98–99% greedy agreement.
Falsification criterion
If AUROC < 0.65 at every bit-width, or if reaching 99% agreement requires
escalating > 50% of tokens at 4-bit (i.e., disagreements are NOT
concentrated at low margins), then decision-uncertainty is not a usable
gating signal: G02/G23 die and candidate C1 must be redesigned around
amortized verification (C2) instead.
Method
Model: Qwen3-1.7B (bf16 reference; representative modern architecture,
runs fully in memory). Variants: MLX affine quantization at 2, 3, 4, 8
bits (group size 64), quantizing all linear layers.
For each of 48 prompts across 6 domains (code, math, chat, FR, EN,
reasoning; benchmarks/datasets/eval_prompts.json): generate 128 tokens
greedily with the bf16 reference, then teacher-force every variant over
the same token sequence and record, per position: variant top-1 margin
(logit gap between its top-2 candidates), variant argmax, reference
argmax, agreement bit, and KL(ref||variant).
Analysis: per bit-width and per domain — agreement rate; margin
distributions conditioned on agree/disagree; AUROC of margin as a
disagreement detector; escalation curve (fraction of tokens with margin
below τ vs residual disagreement rate after escalating those tokens);
the operating point (escalation fraction) that reaches 99% agreement.
Seed fixed (greedy decode; deterministic). Hardware manifest embedded.
Baseline
The 8-bit variant (known near-lossless: expected ≥99% agreement without
any gating) anchors the top of the scale; published aggregate agreement
rates for 4-bit (~90–91%, llama.cpp 'same top p' data) sanity-check our
pipeline. No straw men: the comparison is against unconditional
escalation (loading full weights for all tokens).
Result
CONFIRMED (kill criteria not triggered). AUROC: 0.852 (3-bit), 0.898
(4-bit), 0.975 (8-bit); esc@99%: 60.2% / 36.6% / 5.1%. Naive affine
2-bit collapsed entirely (3% agreement, AUROC 0.605) — the ParetoQ
cliff, measured. Full numbers: results/expG_decision_stability/
20260812T042744Z/ and analysis.md.
Interpretation
Margin is a strong, free, domain-uniform gating signal → G02 promoted.
4-bit escalation need (36.6%) exceeds the hoped 10–30% band, so C1's
economics now hinge on bytes-per-escalation (expD). Resident base must
be ≥3-bit affine or codebook 2-bit (G05).
Next experiment
expD — bytes of residual needed per escalated token; then expG rerun at
3B–8B scale.