Confirmatory Tier-0 analysis, paper draft complete for Tier 0, Tier-1 freeze rule, regression test, claim-evidence table
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
18 changed files +169 −48
modified
README.md
+19 −13
@@ -14,10 +14,14 @@ | ||
| 14 | 14 | > donor language model be transplanted into a recipient that demonstrably lacks it, without full-model |
| 15 | 15 | > training? This repository holds the preregistered protocol, the experiment harness (toy transformers with |
| 16 | 16 | > exact hook semantics; MLX/Transformers backends planned), the registry of runs and the paper. |
| 17 | −> **State on 2026-11-09:** Sprints 0–2 done on Tier 0 — donor/recipient gap manufactured and measured | |
| 18 | −> (donor 1.000 vs recipient 0.092 on the target operator, 3 seeds), hooks verified bitwise no-op, and the | |
| 19 | −> target operator localized by activation patching (executed at the read position by layer-0 attention and | |
| 20 | −> layer-0–2 MLPs; reproducible across seeds, r ≥ 0.96). **No transplant has been run yet.** | |
| 17 | +> **Result (Tier 0, preregistered confirmatory run, 2026-11-09):** a donor-in-the-loop activation graft — | |
| 18 | +> the recipient's post-layer-0 residual row at every read position replaced by the donor's row through an | |
| 19 | +> orthogonal Procrustes map fitted on *task-free* rows — lifts the recipient from chance (0.06–0.10) to 0.966 | |
| 20 | +> on held-out Ω programs (control-adjusted effect +0.875, lower bound above 0.25 in 9/9 pairs; control stratum | |
| 21 | +> 0.992), carries the rule to unseen operands (0.77 vs 0.94), needs eight of 128 dimensions, survives width, | |
| 22 | +> depth and stride-2 mismatches, and is anchored by ablation → rescue (0.11 → 1.00; sham 0.11). Regression-fitted | |
| 23 | +> maps (ridge, CCA) annihilate the capability (Proposition 1); donor-free adapters cache; conventional learning | |
| 24 | +> on the same items is perfect. Tier 1 (Qwen2.5-1.5B/3B, Llama-3.2-3B) and Tier 2 (Qwen2.5-7B) runs: see status. | |
| 21 | 25 | |
| 22 | 26 | --- |
| 23 | 27 | |
@@ -29,7 +33,7 @@ | ||
| 29 | 33 | | SSRN | not yet submitted | |
| 30 | 34 | | Journal | to be assessed after results (ML / mechanistic interpretability venue) | |
| 31 | 35 | | Code & data | synthetic data regenerated from seeds; every run has a manifest in `experiments/manifests/` | |
| 32 | −| Gates | A ✔ · B ✔ (Tier 0) · C ✔ (Tier 0) · D–F pending | | |
| 36 | +| Gates | A ✔ · B ✔ · C ✔ · D ✔ · E ✔ (Tier 0 confirmatory) · F in progress | | |
| 33 | 37 | |
| 34 | 38 | ## 🎯 What this paper does |
| 35 | 39 | |
@@ -49,15 +53,17 @@ N_D = Q_T(D) − Q_T(D \ S_D) donor necessity | ||
| 49 | 53 | |
| 50 | 54 | ## 🏆 Headline results |
| 51 | 55 | |
| 52 | −No transfer results yet. Pilot (exploratory, run `pilot_tier0_opseq_451ef5087182`): | |
| 53 | − | |
| 54 | −| Object | Estimate | | |
| 56 | +| Object | Estimate (run) | | |
| 55 | 57 | |---|---| |
| 56 | −| Donor vs recipient, target stratum (final answer, 3 seeds) | 1.000 vs 0.092 (chance 0.105); paired gap seed 0 +0.847 [+0.825, +0.870] | | |
| 57 | −| Ω-step accuracy | donor 1.000, recipient 0.062 | | |
| 58 | −| Control stratum | 1.000 vs 1.000 | | |
| 59 | −| Hook no-op (6 models) | bitwise identity in all checks | | |
| 60 | −| Localization of Ω (restored-answer rate) | 1.00 at the read position for `resid_post` L0–L3 and `attn_out` L0; 0.00 elsewhere; sham 0.00 | | |
| 58 | +| Donor vs recipient, target stratum | 1.000 vs 0.092; Ω-step 1.000 vs 0.062; control 1.000 both (`pilot_tier0_opseq_451ef5087182`) | | |
| 59 | +| Localization of Ω in the donor | residual row at the read position restores 1.00 at every layer; L0 attention 1.00; L0–L2 MLPs 1.00/0.90/0.83; sham 0.00 (pilot) | | |
| 60 | +| **Primary graft (confirmatory)** | target 0.966 (min 0.889), control 0.992 (min 0.946); Δ_T^adj +0.875, lower bound > 0.25 in 9/9 pairs → H2 supported; H4 (≥ 0.95 retention everywhere) not supported (`graft_tier0_confirmatory_0c2257f6803e`) | | |
| 61 | +| Controls (confirmatory) | sham map 0.091 (control 0.100), norm-matched random 0.091, shuffled item 0.103, shuffled layer 0.186, unrelated-skill donor 0.076 | | |
| 62 | +| Subspace k = 8 vs random k = 8 | 0.927 vs 0.077 (+0.850 [+0.837, +0.862]) | | |
| 63 | +| Rule vs cache | graft 0.941 seen / 0.768 unseen (ratio 0.82, H5 supported); donor-free adapter 0.702 / 0.000 | | |
| 64 | +| Ablation → rescue | 0.113 → 1.000 (sham 0.106) | | |
| 65 | +| Conventional learning on the same 3 000 items | supervised adapter 1.000; distillation 1.000 | | |
| 66 | +| Tier 1 / Tier 2 | runs in progress (`tier1_taskvector_*`) | | |
| 61 | 67 | |
| 62 | 68 | ## 🗂 Repository layout |
| 63 | 69 | |
added
docs/CLAIM_EVIDENCE.md
+26 −0
@@ -0,0 +1,26 @@ | ||
| 1 | +# Claim-to-evidence table — WP-AI-0001 (Gate F) | |
| 2 | + | |
| 3 | +Every claim in the manuscript maps to a run id, a results key and a generator. "Status" = confirmatory | |
| 4 | +(preregistered, new seeds), exploratory (protocol selection), or pilot. | |
| 5 | + | |
| 6 | +| # | Claim (paper section) | Evidence (run → key) | Generator | Status | | |
| 7 | +|---|---|---|---|---| | |
| 8 | +| C1 | Donor 1.000 vs recipient 0.092 on the target stratum; Ω-step 1.000 vs 0.062; control 1.000 both (§5.1) | `pilot_tier0_opseq_451ef5087182` → `across_seeds`, `across_seeds_omega_step` | `scripts/run_pilot.py` → `tab_pilot_gap.tex`, `fig_gap_tier0.pdf` | pilot | | |
| 9 | +| C2 | Hooks bitwise no-op; MLX wrapper bitwise no-op (§5.2) | pilot → `noop`; `tests/integration/test_mlx_noop.py` | pytest | pilot / test | | |
| 10 | +| C3 | Ω executed at the read position: resid patch 1.00 at every layer, attn_out L0 1.00, MLP L0–L2 1.00/0.90/0.83, head 0 of L0 0.62; across-seed r ≥ 0.96 (§5.3) | pilot → `localization` | `scripts/run_pilot.py` → `fig_localization_*.pdf` | pilot | | |
| 11 | +| C4 | Primary graft: target 0.966 (sd 0.037, min 0.889), control 0.992 (min 0.946); Δ_T^adj +0.875 (sd 0.042, min +0.776), lower bound > 0.25 in 9/9 pairs → H2 supported; H4 not supported (§5.4) | `graft_tier0_confirmatory_0c2257f6803e` → `CONFIRMATORY.md`, `confirmatory.json` (`primary`) | `scripts/confirmatory_analysis.py`, `tab_confirmatory.tex` | confirmatory | | |
| 12 | +| C5 | Controls: sham 0.091 (control 0.100), norm-matched random 0.091, shuffled item 0.103, shuffled layer 0.186 (control 1.000), unrelated donor 0.076 (control 1.000) (§5.4) | confirmatory → `conditions.ctl_*` | `tab_confirmatory.tex` | confirmatory | | |
| 13 | +| C6 | Subspace k=8 vs random k=8: +0.850 [+0.837, +0.862] (0.927 vs 0.077) (§5.4) | confirmatory → `secondary.subspace_k8_vs_random`, `holm` | `confirmatory_analysis.py` | confirmatory | | |
| 14 | +| C7 | Adapter graft +0.472 [+0.453, +0.490] (0.544); fresh process 0.344 [0.317, 0.372] vs 0.115 (§5.4) | confirmatory → `secondary.adapter_vs_base`, `persistent/fresh_process_eval.json` | `eval_persistent.py` | confirmatory | | |
| 15 | +| C8 | Weight graft L0 block vs shuffled: +0.556 [+0.537, +0.577]; retention 0.720 (min 0.310) (§5.4) | confirmatory → `secondary.weight_L0_block_vs_shuffled` | idem | confirmatory | | |
| 16 | +| C9 | Rule vs cache: Procrustes seen-fit 0.941 [0.919, 0.960] seen / 0.768 [0.727, 0.808] unseen, ratio 0.816 → H5 supported; adapter 0.702 / 0.000 (§5.4, §6) | confirmatory → `rule_vs_cache` | `fig_mechanism_tier0.pdf` (confirmatory build) | confirmatory | | |
| 17 | +| C10 | Ablation 0.113 → rescue 1.000, sham rescue 0.106; reverse graft 0.972 (§5.4, §6) | confirmatory → `mechanism` | idem | confirmatory | | |
| 18 | +| C11 | Width 128→96: 0.901 (L0→0), 1.000 (L1→1); depth 4→6: 0.783, 0.999, 1.000, 1.000; stride-2: 0.386, 0.760, 1.000, 1.000; shams 0.08–0.13 (§5.4) | confirmatory → `portability` | `tab_mismatch.tex` (exploratory build), `confirmatory.json` | confirmatory | | |
| 19 | +| C12 | Supervised adapter and distillation 1.000 on both strata (§5.4) | confirmatory → `conditions.baseline_*` | `tab_confirmatory.tex` | confirmatory | | |
| 20 | +| C13 | Exploratory: identity graft 0.873 same-init / 0.101 diff-init; ridge-on-Ω-rows 0.17; CCA 0.08; ridge task-free 0.81; Procrustes Ω-rows 0.79/0.70; task-free Procrustes 0.993/0.956 (§7, Table main) | `graft_tier0_30e3644aa801` → `conditions`, `across_pairs` | `tab_main_effects.tex`, `fig_pareto_tier0.pdf` | exploratory | | |
| 21 | +| C14 | Layer grid 1.00 for donor L1–L3 → any layer, L0→0 0.98, L0→1–3 0.75–0.82; dose 0.44/0.90/0.98/0.92/0.75/0.39 (§7) | v2 → `layer_grid`, `dose` | `fig_layer_dose_tier0.pdf` | exploratory | | |
| 22 | +| C15 | Subspace k = 1…32: 0.23, 0.30, 0.65, 0.94, 0.97, 0.98; random 0.08–0.17 (§7) | v2 → `conditions.subspace_*` | `fig_subspace_tier0.pdf` | exploratory | | |
| 23 | +| C16 | Steering ≤ 0.10; weight grafts L0 MLP 0.26 (0.96), block 0.64 (0.73), L0–2+embed 0.77 (0.75); Ω embedding only 0.075 (§7) | v2 → `conditions` | `tab_main_effects.tex` | exploratory | | |
| 24 | +| C17 | Annihilation proposition (§3) | Appendix A proof | — | theorem | | |
| 25 | +| C18 | Tier-1 within-model sufficiency / necessity; cross-model transfer and controls (§5.5) | `tier1_taskvector_*` (dev) and confirmatory split run | `tab_tier1*.tex`, `fig_tier1_transfer.pdf` | pending | | |
| 26 | +| C19 | Tier-2 (7B ↔ 3B) | M3U96b run | idem | pending | | |
modified
docs/COMPUTE_BUDGET.md
+7 −0
@@ -27,4 +27,11 @@ | ||
| 27 | 27 | streamed to disk (do not hold in memory). |
| 28 | 28 | - Tier 2 (7B–14B bf16): 14–28 GB weights; M3U96a/b only; one node per model. |
| 29 | 29 | |
| 30 | +| Model matrix (17 toy models × 3 000 steps) | ≈ 13 min MPS (≈ 45 s each) | | |
| 31 | +| Transplant run (9 pairs, all families, bootstrap 2 000) | 140 s (v1), 176 s (v2), 167 s (confirmatory) on MPS | | |
| 32 | +| Fresh-process persistence check | ≈ 40 s | | |
| 33 | +| Tier-1 quick (1 pair, 12 queries/task, bf16 3B + 1.5B) | 150 s MLX (laptop) | | |
| 34 | +| Tier-1 dev (3 models localized, 4 pairs, 22–40 queries/task, replace grafts) | measured in the run manifest (`tier1_taskvector_*`); localization ≈ 10–20 min per model when the GPU is shared | | |
| 35 | +| Qwen2.5-7B-Instruct bf16 download for Tier 2 (M3U96b) | ≈ 15 GB | | |
| 36 | + | |
| 30 | 37 | Rule: no parallel heavy runs until the single-node profile is recorded in `EXPERIMENT_LOG.md`. |
modified
docs/DECISION_LOG.md
+4 −0
@@ -40,3 +40,7 @@ All entries are dated; "before freeze" refers to the confirmatory freeze (PREREG | ||
| 40 | 40 | method = activation graft with task-free Procrustes at L0→L0, α = 1; confirmatory seed base 90 000. SHA-256 of |
| 41 | 41 | `docs/PREREGISTRATION.md` at freeze recorded below. |
| 42 | 42 | PREREGISTRATION.md sha256 at freeze: fe06d326bbbd27916d573a617e94d4443546f179587209e2e04a6893aad25d3e |
| 43 | +- **D11 (2026-11-09, FREEZE Tier 1).** Selection rule for the Tier-1/2 confirmatory contrast recorded in | |
| 44 | + PREREGISTRATION before the development results were read (eligibility = own-vector sufficiency ≥ 0.5; | |
| 45 | + selected condition = best dev condition; contrast = selected minus best control on confirmatory queries). | |
| 46 | + PREREGISTRATION.md sha256 at Tier-1 freeze: 85c5db69657144e6d6afecfb4fc346c2345d09455fb6134bee615567e5c705ad | |
modified
docs/PREREGISTRATION.md
+19 −1
@@ -111,4 +111,22 @@ Width mismatch (d 128 → 96) and depth mismatch (4 → 6 layers) with the task- | ||
| 111 | 111 | Everything else produced by the script (layer grid, dose, other maps, steering, other k) is reported as |
| 112 | 112 | exploratory and labelled so in the paper. |
| 113 | 113 | |
| 114 | −## Tier 1 (to be frozen in a separate section after the development run completes) | |
| 114 | +## Tier 1 — selection rule frozen BEFORE the development results are read (2026-11-09) | |
| 115 | + | |
| 116 | +Development run: `tier1_taskvector_e105e3d91386` (dev split; the selection data). Confirmatory run: same script with | |
| 117 | +`--split confirmatory` (disjoint query halves), executed after this rule is recorded and without any change to the | |
| 118 | +script's conditions. | |
| 119 | + | |
| 120 | +- **Eligible tasks** per (pair, task): the recipient's *own* task vector reaches dev zero-shot accuracy ≥ 0.5 at its | |
| 121 | + best layer (within-model sufficiency); otherwise the task is reported as "not vectorizable in the recipient" and | |
| 122 | + excluded from the confirmatory contrast (reported, not tested). | |
| 123 | +- **Selected condition** per eligible (pair, task): the transfer condition (map family × recipient layer × α, add | |
| 124 | + variants, rescaled variants, or the donor-in-the-loop replace graft) with the highest dev accuracy; ties broken | |
| 125 | + toward Procrustes, lower layer, α = 1, add before replace. | |
| 126 | +- **Confirmatory contrast**: on the confirmatory queries, accuracy of the selected condition minus accuracy of the | |
| 127 | + *best control* on the same queries (max over norm-matched random vector, unrelated-task vector, sham-map vector, | |
| 128 | + mapped donor zero-shot state, shuffled-query donor state), paired on items, bootstrap over items (2 000 draws). | |
| 129 | +- **Decision rule (H6, per pair)**: supported if the lower 95 % bound exceeds 0.10 for at least two eligible tasks; | |
| 130 | + "partial" if for one; "not supported" otherwise. Also reported: gain vs recipient zero-shot, recipient few-shot | |
| 131 | + (prompting ceiling), recipient own vector (within-model ceiling), held-out NLL change and cross-task accuracy. | |
| 132 | +- Tier 2 (Qwen2.5-7B pairs, run on M3U96b) follows the same rule. | |
modified
docs/REPRODUCIBILITY.md
+8 −1
@@ -6,7 +6,14 @@ python3 -m pip install -e ".[dev]" # or: uv pip install -e ".[dev]" | ||
| 6 | 6 | python3 -m pytest tests -q # unit + causal-sanity tests (CPU, < 30 s) |
| 7 | 7 | python3 scripts/audit_environment.py # writes docs/ENVIRONMENT_AUDIT.json (add --cluster to probe nodes) |
| 8 | 8 | python3 scripts/run_pilot.py --quick --device cpu --seeds 0 1 --n-boot 200 # CPU smoke test, ≈ 25 s |
| 9 | −python3 scripts/run_pilot.py --seeds 0 1 2 --steps 3000 # Tier-0 pilot (MPS ≈ minutes) | |
| 9 | +python3 scripts/run_pilot.py --seeds 0 1 2 --steps 3000 # Tier-0 pilot (MPS ≈ 5 min) | |
| 10 | +python3 scripts/train_models.py # 17-model matrix (≈ 13 min MPS) | |
| 11 | +python3 scripts/run_transplant.py # exploratory transplant run (≈ 3 min) | |
| 12 | +python3 scripts/run_transplant.py --tag confirmatory --seed-base 90000 # confirmatory run (prereg v1.0) | |
| 13 | +python3 scripts/eval_persistent.py --run <graft run id> --seed-base 95000 # saved adapter in a fresh process | |
| 14 | +python3 scripts/confirmatory_analysis.py --run <confirmatory run id> # pre-declared contrasts + Holm | |
| 15 | +python3 scripts/run_tier1.py [--quick] [--split dev|confirmatory] [--pairs a:b ...] # open-weight models (MLX) | |
| 16 | +python3 scripts/build_figures.py --run <run id>; python3 scripts/build_tables.py --graft ... ; scripts/build_paper.sh | |
| 10 | 17 | ``` |
| 11 | 18 | |
| 12 | 19 | - Every run writes `experiments/manifests/<run_id>.json` (git commit, config hash, data hashes, seeds, |
modified
docs/TASKS.md
+16 −9
@@ -25,13 +25,20 @@ Legend: [x] done · [~] in progress · [ ] todo · (gate) quality gate per CLAUD | ||
| 25 | 25 | - [ ] Subspace localization (DAS-style / PCA of Ω-contrast) with norm-matched random subspaces |
| 26 | 26 | - [ ] Necessity test on donor (H3) with control-stratum retention |
| 27 | 27 | |
| 28 | −## Sprint 3 — First transplant | |
| 29 | −- [ ] Alignment module: Procrustes, ridge, CCA, random, unrelated-data maps; leakage audit | |
| 30 | −- [ ] Activation graft + sham + norm-matched random + shuffled-token controls | |
| 31 | −- [ ] Gate D: reproducibility, overhead disclosure | |
| 28 | +## Sprint 3 — First transplant (done 2026-11-09) | |
| 29 | +- [x] Alignment module: identity, Procrustes, ridge, CCA, random orthogonal; task-row vs task-free fits; leakage audit by item hash | |
| 30 | +- [x] Activation graft + sham + norm-matched random + shuffled-token/layer + unrelated-donor controls | |
| 31 | +- [x] Gate D: reproducibility across 9 pairs, overhead disclosed (donor in the loop), map parameter counts | |
| 32 | 32 | |
| 33 | −## Sprint 4–7 | |
| 34 | −- [ ] Tier 1 MLX interventions (layer wrapping) + Transformers equivalence test | |
| 35 | −- [ ] Portability matrix, persistence (saved adapter), collateral battery | |
| 36 | −- [ ] Confirmatory freeze (PREREGISTRATION v1.0) and run | |
| 37 | −- [ ] Paper, figures, claim-evidence table (Gate F) | |
| 33 | +## Sprint 4–6 (done 2026-11-09) | |
| 34 | +- [x] Subspace, steering, adapter, weight, persistent grafts; supervised-adapter and distillation references | |
| 35 | +- [x] Layer grid, dose, width/depth mismatch, stride-2; ablation → rescue; reverse graft; rule-vs-cache | |
| 36 | +- [x] Confirmatory freeze (PREREGISTRATION v1.0, D10) and confirmatory run + analysis (H2, H5 supported; H4 not) | |
| 37 | +- [x] Tier-1 MLX layer wrapping with bitwise no-op test; task-vector localization + transfer script | |
| 38 | +- [~] Tier-1 development run (4 pairs) → Tier-1 freeze → confirmatory split run | |
| 39 | +- [~] Tier-2: Qwen2.5-7B ↔ 3B pairs on M3U96b | |
| 40 | +- [ ] Transformers equivalence test for the MLX backend (not done; reported as a limitation) | |
| 41 | + | |
| 42 | +## Sprint 7 — Manuscript | |
| 43 | +- [~] Paper sections written for Tier 0; Tier 1 pending; claim-evidence table (Gate F) pending | |
| 44 | +- [ ] Internal multi-agent review (UQO-WP-REVIEW pipeline) | |
modified
paper/appendix/appendix.tex
+12 −4
@@ -32,8 +32,8 @@ tested; the experiments in \S\ref{sec:results} estimate how far from it the fitt | ||
| 32 | 32 | |
| 33 | 33 | \section{Configurations, splits and model cards}\label{app:configs} |
| 34 | 34 | Tables of split hashes, model specifications and run identifiers are generated from the run manifests |
| 35 | −(\texttt{experiments/manifests/}) by \texttt{scripts/build\_tables.py}; the registry | |
| 36 | −(\texttt{experiments/registry/runs.jsonl}) lists every planned, running, failed and completed run. | |
| 35 | +(\texttt{experiments/manifests/}) by \url{scripts/build_tables.py}; the registry | |
| 36 | +(\url{experiments/registry/runs.jsonl}) lists every planned, running, failed and completed run. | |
| 37 | 37 | |
| 38 | 38 | \begin{table}[h]\centering\scriptsize |
| 39 | 39 | \caption{Tier-0 model matrix (\texttt{artifacts/models/matrix\_v1/index.json}): architecture, excluded and |
@@ -57,8 +57,16 @@ same- and different-initialization pairs. High $R^2$ does not predict transfer ( | ||
| 57 | 57 | \end{table} |
| 58 | 58 | |
| 59 | 59 | \section{Tier-1 prompts and tasks}\label{app:tier1} |
| 60 | −Few-shot prompt: $k{=}8$ blocks ``\texttt{Input: <x>\textbackslash nOutput: <y>\textbackslash n\textbackslash n}'' | |
| 61 | −followed by ``\texttt{Input: <query>\textbackslash nOutput:}''; zero-shot prompt: the last block only. | |
| 60 | +Few-shot prompt: $k{=}8$ demonstration blocks followed by the query block; zero-shot prompt: the query block only. | |
| 61 | +\begin{verbatim} | |
| 62 | +Input: <x_1> | |
| 63 | +Output: <y_1> | |
| 64 | + | |
| 65 | +... | |
| 66 | + | |
| 67 | +Input: <query> | |
| 68 | +Output: | |
| 69 | +\end{verbatim} | |
| 62 | 70 | Alignment prompts: the same frame with 0--3 random demonstration pairs whose outputs are random and a random |
| 63 | 71 | query (sentences, words, numbers, names), none of which encodes a task. Task item lists are in |
| 64 | 72 | \texttt{src/wp\_ai\_0001/datasets/tier1\_tasks.py} with their hashes in the run manifests. |
modified
paper/main.pdf
+0 −0
Binary file not shown.
modified
paper/main.tex
+2 −0
@@ -42,6 +42,8 @@ | ||
| 42 | 42 | |
| 43 | 43 | % --- Figures --- |
| 44 | 44 | \usepackage{graphicx} |
| 45 | +\usepackage{url} | |
| 46 | +\urlstyle{tt} | |
| 45 | 47 | \usepackage{subcaption} |
| 46 | 48 | \usepackage{float} |
| 47 | 49 | |
modified
paper/sections/experimental_design.tex
+1 −1
@@ -13,7 +13,7 @@ recomputation, hashed, and checked for item-level disjointness against all train | ||
| 13 | 13 | evaluation splits involved in a run; split hashes are stored in run manifests. Design decision D2 (trace |
| 14 | 14 | instead of a single answer token) was taken after a single-answer variant memorized without generalizing; |
| 15 | 15 | its consequence---that the Tier-0 capability is a per-step map once the trace is visible---is controlled |
| 16 | −by the \texttt{trace\_stride} knob (stride 2 forces two operators per emitted result). | |
| 16 | +by the \url{trace_stride} knob (stride 2 forces two operators per emitted result). | |
| 17 | 17 | |
| 18 | 18 | \paragraph{Models and roles.} Decoder-only transformers (4 layers, $d{=}128$, 4 heads, $0.80$\,M parameters; |
| 19 | 19 | pre-LN, explicit attention with per-head outputs exposed) are trained from scratch with teacher forcing on |
modified
paper/sections/limitations.tex
+2 −2
@@ -36,6 +36,6 @@ detection tools for such transfers. | ||
| 36 | 36 | |
| 37 | 37 | \paragraph{Reproducibility.} Every run has a manifest with git commit, configuration hash, split hashes, seeds, |
| 38 | 38 | library versions and hardware; the registry lists every planned, running, failed and completed run; figures and |
| 39 | −tables are regenerated from results files by \texttt{scripts/build\_figures.py} and \texttt{scripts/build\_tables.py}; | |
| 40 | −\texttt{docs/REPRODUCIBILITY.md} gives the commands (CPU smoke test $\approx$ 25\,s; Tier-0 pilot $\approx$ 5\,min; | |
| 39 | +tables are regenerated from results files by \url{scripts/build_figures.py} and \url{scripts/build_tables.py}; | |
| 40 | +\url{docs/REPRODUCIBILITY.md} gives the commands (CPU smoke test $\approx$ 25\,s; Tier-0 pilot $\approx$ 5\,min; | |
| 41 | 41 | Tier-0 transplant run $\approx$ 3\,min on an M5 Max; Tier-1 run minutes to an hour depending on pairs). |
modified
paper/sections/operational_theory.tex
+2 −2
@@ -39,7 +39,7 @@ We implement six families (Table~\ref{tab:families}). | ||
| 39 | 39 | \caption{Transplant families, their inference-time dependency on the donor, what is fitted, and the matched |
| 40 | 40 | control used for the control-adjusted effect.} |
| 41 | 41 | \label{tab:families} |
| 42 | −\begin{tabular}{llll} | |
| 42 | +\resizebox{\textwidth}{!}{\begin{tabular}{llll} | |
| 43 | 43 | \toprule |
| 44 | 44 | Family & donor at inference & fitted object (data) & matched control \\ |
| 45 | 45 | \midrule |
@@ -50,7 +50,7 @@ Adapter graft & no & affine $f: x_R \to A(x_D)$ (ridge) & supervised adapter wit | ||
| 50 | 50 | Weight graft & no & none (copy of blocks / rows) & shuffled-layer / unrelated-donor weights \\ |
| 51 | 51 | Persistent graft & no & saved adapter, fresh process & unmodified recipient on new items \\ |
| 52 | 52 | \bottomrule |
| 53 | −\end{tabular} | |
| 53 | +\end{tabular}} | |
| 54 | 54 | \end{table} The \emph{activation graft} replaces the recipient's |
| 55 | 55 | residual row at a read position by a mapped donor row computed on the same token ids |
| 56 | 56 | ($x_R \leftarrow x_R + \alpha(A(x_D) - x_R)$); the \emph{subspace graft} restricts the replacement to a |
modified
paper/sections/results.tex
+12 −10
@@ -1,12 +1,14 @@ | ||
| 1 | 1 | \section{Results}\label{sec:results} |
| 2 | 2 | |
| 3 | −\paragraph{Provenance.} Every number below names its run: the pilot \texttt{pilot\_tier0\_opseq\_451ef5087182} | |
| 4 | −(Gates B--C and localization), the exploratory transplant runs \texttt{graft\_tier0\_f50a28ca1bf7} (protocol v1) | |
| 5 | −and \texttt{graft\_tier0\_30e3644aa801} (v2), the confirmatory run | |
| 6 | −\texttt{graft\_tier0\_confirmatory\_0c2257f6803e} (preregistration v1.0, new seeds), and the Tier-1 runs of | |
| 3 | +\begin{sloppypar} | |
| 4 | +\paragraph{Provenance.} Every number below names its run: the pilot \url{pilot_tier0_opseq_451ef5087182} | |
| 5 | +(Gates B--C and localization), the exploratory transplant runs \url{graft_tier0_f50a28ca1bf7} (protocol v1) | |
| 6 | +and \url{graft_tier0_30e3644aa801} (v2), the confirmatory run | |
| 7 | +\url{graft_tier0_confirmatory_0c2257f6803e} (preregistration v1.0, new seeds), and the Tier-1 runs of | |
| 7 | 8 | \S\ref{sec:tier1}. Tables and figures are regenerated from the results files by |
| 8 | −\texttt{scripts/build\_tables.py} and \texttt{scripts/build\_figures.py}; the confirmatory decision rules are | |
| 9 | −evaluated by \texttt{scripts/confirmatory\_analysis.py}. | |
| 9 | +\url{scripts/build_tables.py} and \url{scripts/build_figures.py}; the confirmatory decision rules are | |
| 10 | +evaluated by \url{scripts/confirmatory_analysis.py}. | |
| 11 | +\end{sloppypar} | |
| 10 | 12 | |
| 11 | 13 | \subsection{Gate B: a manufactured, reproducible capability gap} |
| 12 | 14 | Three donor seeds trained on all eight operators and three recipient seeds trained with $\Omega$ excluded |
@@ -19,7 +21,7 @@ roles fail the length-OOD stratum ($0.086$ vs $0.091$), a limitation of absolute | ||
| 19 | 21 | treat as a stratum, not a target. |
| 20 | 22 | |
| 21 | 23 | \begin{table}[t]\centering\small |
| 22 | −\caption{Pilot gap qualification (run \texttt{pilot\_tier0\_opseq\_451ef5087182}): final-answer exact match by | |
| 24 | +\caption{Pilot gap qualification (run \url{pilot_tier0_opseq_451ef5087182}): final-answer exact match by | |
| 23 | 25 | stratum, mean (sd) over three seeds per role; paired gap with cluster-bootstrap 95\% interval over programs.} |
| 24 | 26 | \label{tab:pilot} |
| 25 | 27 | \input{tables/tab_pilot_gap} |
@@ -36,7 +38,7 @@ majority-class chance level (grey dashes).} | ||
| 36 | 38 | For all six pilot models, logits computed with (i) no hooks, (ii) capture hooks on all 27 hook points, and |
| 37 | 39 | (iii) a modifying hook registered but disabled are bitwise identical on 256 items, and repeated runs are |
| 38 | 40 | bitwise identical on MPS; the same modifying hook, when enabled, changes the output. The MLX backend passes |
| 39 | −the same test on Qwen2.5-1.5B and Llama-3.2-3B (\texttt{tests/integration/test\_mlx\_noop.py}). | |
| 41 | +the same test on Qwen2.5-1.5B and Llama-3.2-3B (\url{tests/integration/test_mlx_noop.py}). | |
| 40 | 42 | |
| 41 | 43 | \subsection{RQ1: where the donor executes $\Omega$} |
| 42 | 44 | On 490 clean/corrupted pairs the clean run scores $1.000$ and the sham (corrupted run scored on the clean |
@@ -73,7 +75,7 @@ destroys the control stratum ($0.100$); norm-matched random rows and shuffled-it | ||
| 73 | 75 | shuffled-layer rows ($0.186$) and the unrelated-skill donor ($0.076$) leave the control stratum intact. |
| 74 | 76 | |
| 75 | 77 | \begin{table}[t]\centering\scriptsize |
| 76 | −\caption{Confirmatory run \texttt{graft\_tier0\_confirmatory\_0c2257f6803e}: target- and control-stratum | |
| 78 | +\caption{Confirmatory run \url{graft_tier0_confirmatory_0c2257f6803e}: target- and control-stratum | |
| 77 | 79 | accuracy by condition, mean (sd) over same-initialization and different-initialization pairs, and the |
| 78 | 80 | control-adjusted effect against the sham map (mean of per-pair cluster-bootstrap estimates and bounds). |
| 79 | 81 | Families other than the activation graft are evaluated on the three matched-seed pairs.} |
@@ -142,5 +144,5 @@ $\Omega$ rows with operands in $\{0,\dots,5\}$, evaluated on items whose operand | ||
| 142 | 144 | \label{fig:mechanism}\end{figure} |
| 143 | 145 | |
| 144 | 146 | \subsection{Tier 1: open-weight models}\label{sec:tier1} |
| 145 | −[PLACEHOLDER --- filled from runs \texttt{tier1\_taskvector\_e105e3d91386} (development) and the confirmatory | |
| 147 | +[PLACEHOLDER --- filled from runs \url{tier1_taskvector_e105e3d91386} (development) and the confirmatory | |
| 146 | 148 | Tier-1 run after the Tier-1 freeze; see \S\ref{sec:design}.] |
modified
paper/sections/robustness.tex
+3 −3
@@ -1,11 +1,11 @@ | ||
| 1 | 1 | \section{Robustness and Failure Modes}\label{sec:robustness} |
| 2 | 2 | |
| 3 | −All numbers in this section are \emph{exploratory} (protocol v2 run \texttt{graft\_tier0\_30e3644aa801}, | |
| 3 | +All numbers in this section are \emph{exploratory} (protocol v2 run \url{graft_tier0_30e3644aa801}, | |
| 4 | 4 | matched seed-0 pair unless stated); they motivated the frozen protocol and are not re-tested on the |
| 5 | 5 | confirmatory items except where \S\ref{sec:results} says so. |
| 6 | 6 | |
| 7 | 7 | \begin{table}[t]\centering\scriptsize |
| 8 | −\caption{Exploratory run \texttt{graft\_tier0\_30e3644aa801} (protocol v2): target- and control-stratum | |
| 8 | +\caption{Exploratory run \url{graft_tier0_30e3644aa801} (protocol v2): target- and control-stratum | |
| 9 | 9 | accuracy by condition, mean (sd) over same-initialization and different-initialization pairs, and the |
| 10 | 10 | control-adjusted effect against the sham map.} |
| 11 | 11 | \label{tab:main} |
@@ -14,7 +14,7 @@ control-adjusted effect against the sham map.} | ||
| 14 | 14 | |
| 15 | 15 | \begin{table}[t]\centering\scriptsize |
| 16 | 16 | \caption{Architectural mismatch and stride-2 transfer (exploratory run): target accuracy, $\Omega$-step |
| 17 | −accuracy, control accuracy and in-sample $R^2$ of the map; \texttt{unrelated\_procrustes} = Procrustes fitted | |
| 17 | +accuracy, control accuracy and in-sample $R^2$ of the map; \url{unrelated_procrustes} = Procrustes fitted | |
| 18 | 18 | on task-free rows; \texttt{sham} = random orthogonal map.} |
| 19 | 19 | \label{tab:mismatch} |
| 20 | 20 | \resizebox{\textwidth}{!}{\input{tables/tab_mismatch}} |
modified
scripts/run_tier1.py
+1 −0
@@ -40,6 +40,7 @@ MODELS = { | ||
| 40 | 40 | "qwen1.5b": ("mlx-community/Qwen2.5-1.5B-Instruct-bf16", "Qwen2.5", "Apache-2.0"), |
| 41 | 41 | "qwen3b": ("mlx-community/Qwen2.5-3B-Instruct-bf16", "Qwen2.5", "Qwen Research License"), |
| 42 | 42 | "llama3b": ("mlx-community/Llama-3.2-3B-Instruct-bf16", "Llama-3.2", "Llama 3.2 Community License"), |
| 43 | + "qwen7b": ("mlx-community/Qwen2.5-7B-Instruct-bf16", "Qwen2.5", "Apache-2.0"), # Tier 2 (run on a 96 GB node) | |
| 43 | 44 | } |
| 44 | 45 | |
| 45 | 46 | |
modified
src/wp_ai_0001/models/__init__.py
+7 −2
@@ -1,2 +1,7 @@ | ||
| 1 | −from wp_ai_0001.models.hooks import HookManager, Intervention # noqa: F401 | |
| 2 | −from wp_ai_0001.models.toy_transformer import ToyTransformer # noqa: F401 | |
| 1 | +"""Model adapters. PyTorch toy models are imported lazily so that the MLX backend can be used on hosts | |
| 2 | +without torch (e.g. MLX-only cluster nodes).""" | |
| 3 | +try: # pragma: no cover - optional dependency | |
| 4 | + from wp_ai_0001.models.hooks import HookManager, Intervention # noqa: F401 | |
| 5 | + from wp_ai_0001.models.toy_transformer import ToyTransformer # noqa: F401 | |
| 6 | +except ModuleNotFoundError: # torch absent | |
| 7 | + pass | |
added
tests/regression/test_confirmatory_numbers.py
+28 −0
@@ -0,0 +1,28 @@ | ||
| 1 | +"""Regression: the numbers quoted in the paper for the confirmatory run are those in the registered results.""" | |
| 2 | +import json | |
| 3 | +from pathlib import Path | |
| 4 | + | |
| 5 | +import pytest | |
| 6 | + | |
| 7 | +ROOT = Path(__file__).resolve().parents[2] | |
| 8 | +RUN = "graft_tier0_confirmatory_0c2257f6803e" | |
| 9 | +P = ROOT / "experiments" / "summaries" / RUN / "confirmatory.json" | |
| 10 | + | |
| 11 | + | |
| 12 | +@pytest.mark.skipif(not P.exists(), reason="confirmatory results not present") | |
| 13 | +def test_primary_contrast_as_reported(): | |
| 14 | + c = json.loads(P.read_text()) | |
| 15 | + prim = c["primary"] | |
| 16 | + assert prim["n_pairs_lower_bound_above_sesoi"] == 9 | |
| 17 | + assert abs(prim["mean_adj"] - 0.875) < 0.0015 | |
| 18 | + assert prim["H2_supported"] is True and prim["H4_retention_supported"] is False | |
| 19 | + assert abs(c["secondary"]["subspace_k8_vs_random"]["estimate"] - 0.850) < 0.0015 | |
| 20 | + assert abs(c["secondary"]["rule_vs_cache"]["ratio"] - 0.816) < 0.0015 | |
| 21 | + assert c["mechanism"]["mech_rescue_from_intact_donor"]["donor_s1->donor_s0(ablated)"] == 1.0 | |
| 22 | + | |
| 23 | + | |
| 24 | +@pytest.mark.skipif(not P.exists(), reason="confirmatory results not present") | |
| 25 | +def test_manifest_completed_and_hashes_recorded(): | |
| 26 | + m = json.loads((ROOT / "experiments" / "manifests" / f"{RUN}.json").read_text()) | |
| 27 | + assert m["status"] == "completed" | |
| 28 | + assert {"target_iid", "control_iid", "align_target", "align_unrelated"} <= set(m["data_hashes"]) | |
| 29 | ||