spb/modelmap Public License

1---2project: modelmap3document: expA_probe_reliability — analysis (run #1)4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6website: https://modelmap.io7created: 2026-08-128modified: 2026-08-129status: reviewed10---1112# Analysis — expA run #1: the harness failed its own validity gate (and that is the result)1314Run: `results/expA_probe_reliability/20260812T062605Z/results.json` (48 s wall).15Model: mlx-community/Qwen3-0.6B-4bit (28 layers) + random-init architecture twin.16Grid: 3 properties × 2 disjoint promptsets × 28 layers × 5 seeds, shuffled-label17control inside every probe, BH-FDR q=0.05, top-5 replication. Hypothesis and18validity criterion registered before the run.1920```text21Hypothesis              : probe maps are seed-stable but dataset-sensitive22                          (dataset variance > seed variance on ≥1/3 of layers).23Falsification criterion : seeds dominate; AND harness validity gate: twin24                          (random-init) selectivity must stay < 0.05.25Result                  : PRIMARY HYPOTHESIS UNTESTABLE — ceiling effect:26                          task accuracy 1.000 at every layer, every property,27                          both sets; seed SD = 0.0000; dataset shift = 0.0000.28                          VALIDITY GATE FAILED, decisively: the twin reaches29                          accuracy 1.00 at all 28 layers for all properties30                          (twin max selectivity 0.56–0.88 vs the registered31                          0.05 bar); mean real-minus-twin selectivity is32                          within ±0.06 — i.e. ZERO measurable trained-model33                          signal in the map. 28/28 layers "FDR-significant"34                          in the twin too: the statistics are fine, the35                          measurand is wrong.36Interpretation          : Level 1 FOR THE NEGATIVE CLAIM (controlled,37                          replicated across 5 seeds × 2 sets × 3 properties):38                          on lexically separable classes, per-layer linear39                          probes on mean-pooled representations measure the40                          tokenizer + architecture prior, not learned41                          computation. This is the probing-pitfalls42                          literature (notes §4.1/§4.8 — Hewitt & Liang,43                          Bolukbasi, Dead Salmons) reproduced in our own44                          hands on our own harness, caught by a45                          pre-registered gate rather than by a reviewer.46                          Published as the atlas's first entry47                          (atlas/qwen3-0.6b-4bit/probes/v1,48                          negative_result=true) — maps that fail their nulls49                          are first-class citizens here.50Next experiment         : expA run #2 with (a) promptsets v2 that remove51                          lexical separability (shared vocabulary across52                          classes; structure-borne properties), (b) probe53                          maps published only as REAL−TWIN differentials,54                          (c) last-token representations at matched55                          positions alongside mean-pooling, (d) a56                          class-balanced token-overlap statistic reported57                          per promptset as a difficulty certificate.58```5960## Notes6162- The ceiling itself was foreseeable in hindsight: template-generated classes63  differ lexically (FR vs EN vocabulary; `def`/`const` tokens; digit tokens),64  and random features over distinct token distributions are linearly65  separable. v1 promptsets keep their declared role — they are now the66  *positive control* corpus (any harness that fails to reach ceiling on them67  is broken) rather than a measurement corpus.68- The twin null earns permanent-fixture status: it is cheap (one extra69  capture pass) and it is the only control in the battery that caught this.70  Shuffled-label controls passed everywhere (selectivity 0.3–0.7) and would71  have let the illusion through.72- Wall-clock: full grid in 48 s on M5 Max — noise-floor science at this scale73  is essentially free; the expensive part was thinking, not compute.7475---7677# Analysis — expA run #2: differential maps work; the twin keeps teaching7879Run: `results/expA_probe_reliability/20260812T063856Z/results.json` (88.6 s).80v2 promptsets (structure-borne, token-balanced; overlap certificates in the81manifest), mean-pooled AND last-token reps, twin null on A sets, v182positive control. Hypothesis registered before the run.8384```text85Hypothesis (a) VALIDITY : twin max selectivity < 0.05 on token-balanced sets.86Result (a)              : FALSIFIED AGAIN — but the gradient is the finding.87                          word_order: twin acc 0.958–0.963 (sel 0.47–0.59);88                          agreement: twin acc 0.683–0.729 (sel 0.15–0.23);89                          arith_valid: twin acc 0.558–0.579 (sel 0.08–0.12).90                          Even with matched word sets, TOKENIZATION statistics91                          differ between classes (mid-sentence capitals,92                          position-dependent subwords in scrambles; is/are93                          bigram contexts) — the twin mines them. The strict94                          0.05 gate is unreachable for surface-correlated95                          properties; the differential doctrine (v1) is the96                          correct instrument, and v2 applies it.97Hypothesis (b) SIGNAL   : real−twin sel > 0.10 on ≥5 FDR-significant layers98                          for word_order AND agreement.99Result (b)              : SPLIT. agreement PASSES decisively — 25/28 layers100                          (mean pooling; 23/28 last-token), max Δsel +0.379,101                          real acc up to 0.967 vs twin 0.729. word_order102                          FAILS — 2/28 layers (0 last-token), max Δsel103                          +0.129: grammatical-vs-scrambled is essentially104                          null-dominated; NOT valid evidence of learned105                          syntax at this granularity.106Hypothesis (c) arith    : exploratory.107Result (c)              : POSITIVE SIGNAL — real acc 0.858 (mean) / 0.904108                          (last-token) vs twin 0.558/0.579; 5–6 signal109                          layers; max Δsel +0.358. Equation-validity is110                          linearly decodable above the architecture prior in111                          a 0.6B-4bit model, strongest in last-token reps112                          (consistent with computation completing at "=").113Hypothesis (d) variance : dataset shift > seed SD on ≥1/3 of layers.114Result (d)              : NOT SUPPORTED — 3–8/28 layers across cells. Off115                          ceiling, seed SD is finally nonzero and top-5-layer116                          replication drops to ~0.54 (vs the trivial 1.00 at117                          ceiling): the FIRST real noise-floor numbers of the118                          project. Seed and dataset variance are comparable119                          in this regime; neither dominates.120Positive control        : v1 lang_id_A at ceiling (1.000) — harness intact.121Interpretation          : Level 1 per property, differential claims only.122                          agreement + arith_valid are the project's first123                          POSITIVE maps (published in atlas probes/v2 with124                          per-property verdicts); word_order is flagged125                          null-dominated in the same entry. Twin gate:126                          retired as a binary gate, kept as a mandatory127                          reported baseline — the differential IS the map.128Next experiment         : (1) promote agreement/arith_valid toward Level 2:129                          second method (LEACE erasure damage) on the same130                          sets; (2) expC causal check on the top agreement131                          layers (ablation); (3) noise-floor deepening: more132                          seeds at fixed set to tighten replication CIs133                          (expD); (4) quantization drift of the agreement134                          differential map (candidate_02 entry point).135```136