--- project: modelmap document: expA_probe_reliability — analysis (run #1) author: Simon-Pierre Boucher contact: contact@spboucher.ai website: https://modelmap.io created: 2026-08-12 modified: 2026-08-12 status: reviewed --- # Analysis — expA run #1: the harness failed its own validity gate (and that is the result) Run: `results/expA_probe_reliability/20260812T062605Z/results.json` (48 s wall). Model: mlx-community/Qwen3-0.6B-4bit (28 layers) + random-init architecture twin. Grid: 3 properties × 2 disjoint promptsets × 28 layers × 5 seeds, shuffled-label control inside every probe, BH-FDR q=0.05, top-5 replication. Hypothesis and validity criterion registered before the run. ```text Hypothesis : probe maps are seed-stable but dataset-sensitive (dataset variance > seed variance on ≥1/3 of layers). Falsification criterion : seeds dominate; AND harness validity gate: twin (random-init) selectivity must stay < 0.05. Result : PRIMARY HYPOTHESIS UNTESTABLE — ceiling effect: task accuracy 1.000 at every layer, every property, both sets; seed SD = 0.0000; dataset shift = 0.0000. VALIDITY GATE FAILED, decisively: the twin reaches accuracy 1.00 at all 28 layers for all properties (twin max selectivity 0.56–0.88 vs the registered 0.05 bar); mean real-minus-twin selectivity is within ±0.06 — i.e. ZERO measurable trained-model signal in the map. 28/28 layers "FDR-significant" in the twin too: the statistics are fine, the measurand is wrong. Interpretation : Level 1 FOR THE NEGATIVE CLAIM (controlled, replicated across 5 seeds × 2 sets × 3 properties): on lexically separable classes, per-layer linear probes on mean-pooled representations measure the tokenizer + architecture prior, not learned computation. This is the probing-pitfalls literature (notes §4.1/§4.8 — Hewitt & Liang, Bolukbasi, Dead Salmons) reproduced in our own hands on our own harness, caught by a pre-registered gate rather than by a reviewer. Published as the atlas's first entry (atlas/qwen3-0.6b-4bit/probes/v1, negative_result=true) — maps that fail their nulls are first-class citizens here. Next experiment : expA run #2 with (a) promptsets v2 that remove lexical separability (shared vocabulary across classes; structure-borne properties), (b) probe maps published only as REAL−TWIN differentials, (c) last-token representations at matched positions alongside mean-pooling, (d) a class-balanced token-overlap statistic reported per promptset as a difficulty certificate. ``` ## Notes - The ceiling itself was foreseeable in hindsight: template-generated classes differ lexically (FR vs EN vocabulary; `def`/`const` tokens; digit tokens), and random features over distinct token distributions are linearly separable. v1 promptsets keep their declared role — they are now the *positive control* corpus (any harness that fails to reach ceiling on them is broken) rather than a measurement corpus. - The twin null earns permanent-fixture status: it is cheap (one extra capture pass) and it is the only control in the battery that caught this. Shuffled-label controls passed everywhere (selectivity 0.3–0.7) and would have let the illusion through. - Wall-clock: full grid in 48 s on M5 Max — noise-floor science at this scale is essentially free; the expensive part was thinking, not compute. --- # Analysis — expA run #2: differential maps work; the twin keeps teaching Run: `results/expA_probe_reliability/20260812T063856Z/results.json` (88.6 s). v2 promptsets (structure-borne, token-balanced; overlap certificates in the manifest), mean-pooled AND last-token reps, twin null on A sets, v1 positive control. Hypothesis registered before the run. ```text Hypothesis (a) VALIDITY : twin max selectivity < 0.05 on token-balanced sets. Result (a) : FALSIFIED AGAIN — but the gradient is the finding. word_order: twin acc 0.958–0.963 (sel 0.47–0.59); agreement: twin acc 0.683–0.729 (sel 0.15–0.23); arith_valid: twin acc 0.558–0.579 (sel 0.08–0.12). Even with matched word sets, TOKENIZATION statistics differ between classes (mid-sentence capitals, position-dependent subwords in scrambles; is/are bigram contexts) — the twin mines them. The strict 0.05 gate is unreachable for surface-correlated properties; the differential doctrine (v1) is the correct instrument, and v2 applies it. Hypothesis (b) SIGNAL : real−twin sel > 0.10 on ≥5 FDR-significant layers for word_order AND agreement. Result (b) : SPLIT. agreement PASSES decisively — 25/28 layers (mean pooling; 23/28 last-token), max Δsel +0.379, real acc up to 0.967 vs twin 0.729. word_order FAILS — 2/28 layers (0 last-token), max Δsel +0.129: grammatical-vs-scrambled is essentially null-dominated; NOT valid evidence of learned syntax at this granularity. Hypothesis (c) arith : exploratory. Result (c) : POSITIVE SIGNAL — real acc 0.858 (mean) / 0.904 (last-token) vs twin 0.558/0.579; 5–6 signal layers; max Δsel +0.358. Equation-validity is linearly decodable above the architecture prior in a 0.6B-4bit model, strongest in last-token reps (consistent with computation completing at "="). Hypothesis (d) variance : dataset shift > seed SD on ≥1/3 of layers. Result (d) : NOT SUPPORTED — 3–8/28 layers across cells. Off ceiling, seed SD is finally nonzero and top-5-layer replication drops to ~0.54 (vs the trivial 1.00 at ceiling): the FIRST real noise-floor numbers of the project. Seed and dataset variance are comparable in this regime; neither dominates. Positive control : v1 lang_id_A at ceiling (1.000) — harness intact. Interpretation : Level 1 per property, differential claims only. agreement + arith_valid are the project's first POSITIVE maps (published in atlas probes/v2 with per-property verdicts); word_order is flagged null-dominated in the same entry. Twin gate: retired as a binary gate, kept as a mandatory reported baseline — the differential IS the map. Next experiment : (1) promote agreement/arith_valid toward Level 2: second method (LEACE erasure damage) on the same sets; (2) expC causal check on the top agreement layers (ablation); (3) noise-floor deepening: more seeds at fixed set to tighten replication CIs (expD); (4) quantization drift of the agreement differential map (candidate_02 entry point). ```