Experiment plan — WP-AI-0001
1. Tiers and model matrix
| Tier | Models | Purpose | Backend | Status |
|---|---|---|---|---|
| 0 | toy decoder transformers trained from scratch (0.8 M params default; 4 layers, d = 128) | full visibility, controlled gap, exhaustive interventions | PyTorch (CPU/MPS) | pilot running |
| 1 | Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct (bf16, cached locally) | external validity, same-family different-size, cross-family, cross-tokenizer | MLX (layer-wrapping interventions) with a PyTorch/Transformers equivalence check | planned (after Gate D on Tier 0) |
| 2 | 7B–14B open weights (e.g. Qwen2.5-7B/14B, Llama-3.1-8B) | scale | MLX on M3U96a/b (96 GB) | conditional |
| 3 | larger | only with explicit approval | — | not planned |
Pairs (Tier 1): Qwen2.5-1.5B→3B (same family, same tokenizer, 28→36 layers); Llama-3.2-3B↔Qwen2.5-3B (different family, different tokenizer, 28 vs 36 layers); Tier 0: same architecture / different seeds, same architecture / different widths, different depths.
2. Task families
- A — OPSEQ (implemented): invented operators on Z_11, scratchpad trace; strata
target_iid,control_iid,comp_heldout,len_ood; difficulty knobtrace_stride. - B — FSA (generator implemented): random DFA execution over an invented alphabet.
- C — MORPH (generator implemented): context-dependent suffix rule on pseudo-words; Tier 1 variant with natural-language pseudo-morphology to be specified at Sprint 4.
- Negative-control battery (Tier 1): unrelated arithmetic, factual recall, instruction following, held-out text perplexity, calibration, refusal behaviour.
3. Phases (per CLAUDE.md §6) and their scripts
| Phase | Script | Output |
|---|---|---|
| 1 Baseline & gap | scripts/run_pilot.py |
gap table, Ω-step accuracies, no-op checks |
| 2 Localization | scripts/run_pilot.py (within-donor patching, ablation, heads) |
heatmaps, reproducibility |
| 3 Alignment | scripts/run_alignment.py (to write) |
Procrustes / ridge / CCA / random / unrelated-data maps with leakage audit |
| 4 Transplant | scripts/run_transplant.py (to write) |
activation graft, subspace graft, weight graft (same-arch), adapter graft |
| 5 Controls | same | sham, norm-matched random, unrelated-skill donor, shuffled token/layer, few-shot, steering, matched adapter FT, distillation |
| 6 Robustness | scripts/run_sweeps.py (to write) |
strength × layer × position × dimension × pair |
| 7 Mechanism | scripts/run_mechanism.py (to write) |
ablation→loss→rescue, bidirectional, dose-response |
| 8 Confirmatory | scripts/run_confirmatory.py (to write) |
frozen protocol on untouched splits |
4. Minimal viable experiment (MVE) and go/no-go
MVE = Tier 0, OPSEQ, Ω = qua, donor (all operators) vs recipient (Ω excluded), 3 seeds each.
Go to Sprint 3 (first graft) iff: (a) donor Ω-step accuracy ≥ 0.95 and recipient ≤ chance + 0.1 on
target_iid; (b) all no-op checks true; (c) at least one localization cell with restored-answer rate
≥ 0.8 and across-seed heatmap correlation ≥ 0.8. Otherwise diagnose (see RISK_REGISTER R1–R3).
5. Pre-declared confirmatory contrasts (Tier 0)
- activation graft vs matched sham (primary); 2. subspace graft vs norm-matched random subspace;
- weight graft (same architecture) vs shuffled-layer weight graft; 4. adapter graft vs matched-size
adapter trained on the same alignment data without donor signal. Each: Δ_T^adj with cluster-bootstrap CI,
Holm-corrected across the four families; retention on
control_iid.
6. Figures that the plan can support (data-dependent)
Gap distributions (F2), localization heatmap (F3), Pareto target-vs-retention (F5), portability matrix (F6), graft size vs effect (F7), ablation→rescue chain (F8), robustness (F9), failure taxonomy (F10).