WP-AI-0001: protocol, harness, Tier-0 pilot (sprints 0-2)
- Research gap (40 verified arXiv records), preregistration v0.1, experiment plan, compute budget, risk register, logs - Harness: OPSEQ/FSA/MORPH generators with exact validators, toy transformer with exact hook points, training, patching, bootstrap, manifests/registry - Pilot pilot_tier0_opseq_451ef5087182: donor/recipient gap, bitwise hook no-op, within-donor localization of the target operator - Paper skeleton with explicit placeholders; figures regenerated by scripts/build_figures.py Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
93 changed files +9,007 −0
added
.gitignore
+39 −0
@@ -0,0 +1,39 @@ | ||
| 1 | +# Raw & processed data — never committed (see data/raw/README.md for provenance) | |
| 2 | +data/raw/* | |
| 3 | +!data/raw/README.md | |
| 4 | +data/processed/* | |
| 5 | +!data/processed/.gitkeep | |
| 6 | + | |
| 7 | +# Python | |
| 8 | +__pycache__/ | |
| 9 | +*.pyc | |
| 10 | +.pytest_cache/ | |
| 11 | +.venv/ | |
| 12 | +*.egg-info/ | |
| 13 | + | |
| 14 | +# LaTeX build artifacts (paper/main.pdf IS committed) | |
| 15 | +paper/*.aux | |
| 16 | +paper/*.log | |
| 17 | +paper/*.out | |
| 18 | +paper/*.toc | |
| 19 | +paper/*.lof | |
| 20 | +paper/*.lot | |
| 21 | +paper/*.fls | |
| 22 | +paper/*.fdb_latexmk | |
| 23 | +paper/*.bbl | |
| 24 | +paper/*.blg | |
| 25 | +paper/*.spl | |
| 26 | +paper/*.synctex.gz | |
| 27 | +paper/*.run.xml | |
| 28 | +paper/*.bcf | |
| 29 | + | |
| 30 | +# Logs | |
| 31 | +logs/ | |
| 32 | +*.log | |
| 33 | + | |
| 34 | +# macOS | |
| 35 | +.DS_Store | |
| 36 | + | |
| 37 | +# WP-AI-0001 artifacts (regenerable; models + per-item outputs + run logs) | |
| 38 | +artifacts/ | |
| 39 | +!artifacts/.gitkeep | |
added
CLAUDE.md
+32 −0
@@ -0,0 +1,32 @@ | ||
| 1 | +# CLAUDE.md — WP-AI-0001: The Neural Organ Transplant Experiment | |
| 2 | + | |
| 3 | +**Causal transfer of cognitive capabilities across local large language models.** Research home: UQO | |
| 4 | +Research (`Recherche-UQO/ia/wp-ai-0001`). Language: English for all code, docs, figures, paper, logs. | |
| 5 | +Status: protocol + implementation in progress; **no transfer result has been established**. | |
| 6 | + | |
| 7 | +The full research mandate (sections 0–18: mission, estimands, RQ1–RQ8, task design, model matrix, | |
| 8 | +pipeline phases 1–8, metrics, statistics, architecture, gates A–F, figures, paper, registry, licensing, | |
| 9 | +roadmap, autonomous work rules) was supplied by the author on 2026-11-09 and is reproduced verbatim in | |
| 10 | +`docs/MANDATE.md`. It binds every session. Operational rules that follow from it: | |
| 11 | + | |
| 12 | +1. **Never fabricate** runs, metrics, citations, hardware or references. Every number in `docs/`, | |
| 13 | + `experiments/summaries/` and `paper/` must trace to a run id in `experiments/registry/runs.jsonl` and a | |
| 14 | + manifest in `experiments/manifests/`. Pending measurements are explicit placeholders. | |
| 15 | +2. **Gates in order** (A repository readiness → B baseline integrity → C mechanistic sanity → D transplant | |
| 16 | + validity → E confirmatory evidence → F paper integrity). A failed gate is diagnosed, not reinterpreted. | |
| 17 | +3. **Exploratory ≠ confirmatory.** `docs/PREREGISTRATION.md` v0.1 is registered; the confirmatory freeze | |
| 18 | + (v1.0) must precede generation of confirmatory splits. Design changes go to `docs/DECISION_LOG.md` | |
| 19 | + with "before/after freeze"; failures go to `docs/EXPERIMENT_LOG.md`. Unfavourable runs are never deleted. | |
| 20 | +4. **Persistence vocabulary.** Distinguish single-token intervention, within-sequence effect, | |
| 21 | + cross-prompt reuse, and saved checkpoint/adapter. Transient steering is never called learning. | |
| 22 | +5. **Controls are mandatory** for any transfer claim: unmodified recipient/donor, sham, norm-matched random, | |
| 23 | + unrelated-skill donor, shuffled-token/layer grafts, prompting, steering, matched adapter fine-tuning. | |
| 24 | +6. **Resources.** Laptop (M5 Max 48 GB) and the existing MacLustr SSH access only. Ask before paid compute, | |
| 25 | + checkpoints > 20 GB, publishing/submitting, or touching shared infrastructure. No secrets in the repo. | |
| 26 | +7. **Backends.** Tier 0 = PyTorch toy models (hooks exact, CPU-testable). Tier ≥ 1 = MLX layer wrapping | |
| 27 | + with a no-op test and a Transformers equivalence test; never claim a hook MLX does not expose. | |
| 28 | +8. **Remote.** `origin = gitsrv:srv/git/wpai0001_uqo.git` (spbgit on M1M32). Commit after each sprint | |
| 29 | + deliverable; `paper/main.pdf` is committed when it exists. | |
| 30 | + | |
| 31 | +Quick start: `python3 -m pytest tests -q` · `python3 scripts/audit_environment.py` · | |
| 32 | +`python3 scripts/run_pilot.py --quick --device cpu` · see `docs/REPRODUCIBILITY.md` and `docs/TASKS.md`. | |
added
README.md
+146 −0
@@ -0,0 +1,146 @@ | ||
| 1 | +<!-- Author: Simon-Pierre Boucher — simon-pierre.boucher@uqo.ca --> | |
| 2 | +# 🧠 The Neural Organ Transplant Experiment | |
| 3 | + | |
| 4 | +**UQO Working Paper No. AI-0001** — *Causal Transfer of Cognitive Capabilities Across Local Large Language Models* | |
| 5 | + | |
| 6 | +[](paper/main.pdf) | |
| 7 | +[](pyproject.toml) | |
| 8 | +[](tests/) | |
| 9 | +[](docs/REPRODUCIBILITY.md) | |
| 10 | +[](https://uqo.ca) | |
| 11 | +[](mailto:simon-pierre.boucher@uqo.ca) | |
| 12 | + | |
| 13 | +> **TL;DR** — Can a bounded, *causally identified* subsystem that implements a measurable capability in a | |
| 14 | +> donor language model be transplanted into a recipient that demonstrably lacks it, without full-model | |
| 15 | +> training? This repository holds the preregistered protocol, the experiment harness (toy transformers with | |
| 16 | +> exact hook semantics; MLX/Transformers backends planned), the registry of runs and the paper. | |
| 17 | +> **State on 2026-11-09:** Sprints 0–2 done on Tier 0 — donor/recipient gap manufactured and measured | |
| 18 | +> (donor 1.000 vs recipient 0.092 on the target operator, 3 seeds), hooks verified bitwise no-op, and the | |
| 19 | +> target operator localized by activation patching (executed at the read position by layer-0 attention and | |
| 20 | +> layer-0–2 MLPs; reproducible across seeds, r ≥ 0.96). **No transplant has been run yet.** | |
| 21 | + | |
| 22 | +--- | |
| 23 | + | |
| 24 | +## 📌 Status | |
| 25 | + | |
| 26 | +| Item | Value | | |
| 27 | +|---|---| | |
| 28 | +| Version | 0.1 protocol draft (November 2026) | | |
| 29 | +| SSRN | not yet submitted | | |
| 30 | +| Journal | to be assessed after results (ML / mechanistic interpretability venue) | | |
| 31 | +| Code & data | synthetic data regenerated from seeds; every run has a manifest in `experiments/manifests/` | | |
| 32 | +| Gates | A ✔ · B ✔ (Tier 0) · C ✔ (Tier 0) · D–F pending | | |
| 33 | + | |
| 34 | +## 🎯 What this paper does | |
| 35 | + | |
| 36 | +Prior work maps whole residual streams or concept directions between models (model stitching, activation | |
| 37 | +buses, cross-model steering) and the one controlled attempt at activation-mediated *capability* transfer | |
| 38 | +reports a null. We (i) causally localize a capability in the donor by interchange interventions, | |
| 39 | +(ii) manufacture or verify the recipient's lack of it, and (iii) compare each transplant family | |
| 40 | +(activation, subspace, circuit, weight, adapter, persistent) against matched sham, random and | |
| 41 | +unrelated-donor controls on held-out, leakage-controlled tasks. Estimands: | |
| 42 | + | |
| 43 | +``` | |
| 44 | +Δ_T = Q_T(R') − Q_T(R) transfer gain | |
| 45 | +Δ_T^adj = Δ_T − [Q_T(R_sham) − Q_T(R)] control-adjusted transfer (primary) | |
| 46 | +ρ_c = Q_c(R') / Q_c(R) retention on control strata | |
| 47 | +N_D = Q_T(D) − Q_T(D \ S_D) donor necessity | |
| 48 | +``` | |
| 49 | + | |
| 50 | +## 🏆 Headline results | |
| 51 | + | |
| 52 | +No transfer results yet. Pilot (exploratory, run `pilot_tier0_opseq_451ef5087182`): | |
| 53 | + | |
| 54 | +| Object | Estimate | | |
| 55 | +|---|---| | |
| 56 | +| Donor vs recipient, target stratum (final answer, 3 seeds) | 1.000 vs 0.092 (chance 0.105); paired gap seed 0 +0.847 [+0.825, +0.870] | | |
| 57 | +| Ω-step accuracy | donor 1.000, recipient 0.062 | | |
| 58 | +| Control stratum | 1.000 vs 1.000 | | |
| 59 | +| Hook no-op (6 models) | bitwise identity in all checks | | |
| 60 | +| Localization of Ω (restored-answer rate) | 1.00 at the read position for `resid_post` L0–L3 and `attn_out` L0; 0.00 elsewhere; sham 0.00 | | |
| 61 | + | |
| 62 | +## 🗂 Repository layout | |
| 63 | + | |
| 64 | +``` | |
| 65 | +wp-ai-0001/ | |
| 66 | +├── CLAUDE.md · docs/MANDATE.md # binding rules and the verbatim research mandate | |
| 67 | +├── src/wp_ai_0001/ # datasets (OPSEQ/FSA/MORPH), models + hooks, training, localization, | |
| 68 | +│ # evaluation, statistics, instrumentation, orchestration, visualization | |
| 69 | +├── tests/{unit,causal_sanity,...} # generators, bootstrap, hook no-op and intervention algebra | |
| 70 | +├── scripts/ # audit_environment, run_pilot, build_figures, build_paper | |
| 71 | +├── experiments/{registry,manifests,summaries} | |
| 72 | +├── artifacts/ # models, per-item outputs, figures (git-ignored) | |
| 73 | +├── docs/ # RESEARCH_GAP, PREREGISTRATION, EXPERIMENT_PLAN, COMPUTE_BUDGET, | |
| 74 | +│ # RISK_REGISTER, REPRODUCIBILITY, EXPERIMENT_LOG, DECISION_LOG, TASKS | |
| 75 | +├── literature/ # verified arXiv records + BibTeX; unverified candidates listed apart | |
| 76 | +└── paper/ # main.tex + sections/ + references.bib → main.pdf | |
| 77 | +``` | |
| 78 | + | |
| 79 | +## 🚀 Quick start | |
| 80 | + | |
| 81 | +```bash | |
| 82 | +python3 -m pip install -e ".[dev]" | |
| 83 | +python3 -m pytest tests -q | |
| 84 | +python3 scripts/audit_environment.py --cluster | |
| 85 | +python3 scripts/run_pilot.py --quick --device cpu --seeds 0 1 --n-boot 200 # ≈ 25 s CPU | |
| 86 | +python3 scripts/run_pilot.py --seeds 0 1 2 --steps 3000 # ≈ 5 min on an M5 Max (MPS) | |
| 87 | +python3 scripts/build_figures.py --run <run_id> && scripts/build_paper.sh | |
| 88 | +``` | |
| 89 | + | |
| 90 | +## 🗃 Data | |
| 91 | + | |
| 92 | +Synthetic micro-worlds generated from seeds (`src/wp_ai_0001/datasets/`), validated by exact recomputation, | |
| 93 | +hashed and checked for item-level disjointness across splits; see `data/README.md`. Tier-1 checkpoints | |
| 94 | +(Qwen2.5-1.5B/3B-Instruct, Llama-3.2-3B-Instruct, bf16) are cached locally and not redistributed. | |
| 95 | + | |
| 96 | +## 🔬 Methodology | |
| 97 | + | |
| 98 | +1. Gap qualification (Phase 1) — `scripts/run_pilot.py`, `training/train_toy.py`, `evaluation/metrics.py`. | |
| 99 | +2. Causal localization (Phase 2) — `localization/patching.py` (interchange, ablation, heads). | |
| 100 | +3. Alignment (Phase 3) — `alignment/` (to implement: Procrustes, ridge, CCA, random, unrelated-data). | |
| 101 | +4. Transplant families and controls (Phases 4–5) — `transplantation/` (to implement). | |
| 102 | +5. Robustness, mechanism, confirmatory (Phases 6–8) — planned, see `docs/EXPERIMENT_PLAN.md`. | |
| 103 | +6. Statistics — `statistics/bootstrap.py` (cluster bootstrap over programs; seed-level summaries). | |
| 104 | + | |
| 105 | +## 🖼 Figures & tables inventory | |
| 106 | + | |
| 107 | +| Figure | Content | Source run | | |
| 108 | +|---|---|---| | |
| 109 | +| `fig_gap_tier0.pdf` | donor/recipient accuracy per stratum, seeds, chance | `pilot_tier0_opseq_451ef5087182` | | |
| 110 | +| `fig_localization_patching_tier0.pdf` | layer × position restored-answer rate, 3 families | idem | | |
| 111 | +| `fig_localization_ablation_tier0.pdf` | layer × position accuracy after zero-ablation | idem | | |
| 112 | +| `fig_localization_heads_tier0.pdf` | layer × head patching at Ω token and read position | idem | | |
| 113 | + | |
| 114 | +## ⚠️ Limitations | |
| 115 | + | |
| 116 | +- Tier-0 capability is a per-step map once the trace is visible (risk R1); multi-step claims need | |
| 117 | + `trace_stride > 1` and Tier ≥ 1 evidence. | |
| 118 | +- No transplant, alignment or control battery has been executed; the paper contains placeholders. | |
| 119 | +- Length generalization fails for both roles (absolute positions); it is a stratum, not a target. | |
| 120 | + | |
| 121 | +## 📝 Citation | |
| 122 | + | |
| 123 | +```bibtex | |
| 124 | +@techreport{boucher2026organtransplant, | |
| 125 | + author = {Boucher, Simon-Pierre}, | |
| 126 | + title = {The Neural Organ Transplant Experiment: Causal Transfer of Cognitive Capabilities Across Local Large Language Models}, | |
| 127 | + institution = {Universit\'e du Qu\'ebec en Outaouais, D\'epartement des sciences administratives}, | |
| 128 | + type = {Working Paper}, | |
| 129 | + number = {AI-0001}, | |
| 130 | + year = {2026}, | |
| 131 | + month = {November} | |
| 132 | +} | |
| 133 | +``` | |
| 134 | + | |
| 135 | +## 👤 Author & contact | |
| 136 | + | |
| 137 | +**Simon-Pierre Boucher** — Département des sciences administratives, | |
| 138 | +Université du Québec en Outaouais (UQO), Gatineau, QC. | |
| 139 | +📧 [simon-pierre.boucher@uqo.ca](mailto:simon-pierre.boucher@uqo.ca) | |
| 140 | + | |
| 141 | +## 🗒 Notes | |
| 142 | + | |
| 143 | +- Numbering: project id `wp-ai-0001` (series "AI", number 0001) as requested in the mandate; repository | |
| 144 | + `wpai0001_uqo.git` on spbgit, following the FIRC-0001 precedent rather than the `wpN` sequence. | |
| 145 | +- Deviation from the template layout: `src/wp_ai_0001/training/` added (Tier-0 models are trained here); | |
| 146 | + experiment artifacts live under `experiments/` and `artifacts/` per the mandate instead of `results/`. | |
added
configs/experiments/.gitkeep
+0 −0
added
configs/interventions/.gitkeep
+0 −0
added
configs/models/.gitkeep
+0 −0
added
configs/tasks/.gitkeep
+0 −0
added
data/README.md
+10 −0
@@ -0,0 +1,10 @@ | ||
| 1 | +# Data policy — WP-AI-0001 | |
| 2 | + | |
| 3 | +All Tier-0 data are **synthetic** and regenerated from seeds by `src/wp_ai_0001/datasets/` (generator | |
| 4 | +versions are recorded in every split). Concrete split files are written under `artifacts/runs/<run_id>/data/` | |
| 5 | +(git-ignored) with SHA-256-derived hashes recorded in `experiments/manifests/<run_id>.json`. | |
| 6 | + | |
| 7 | +Tier ≥ 1 model checkpoints are downloaded from the Hugging Face hub into the user's cache (not committed). | |
| 8 | +Licenses: Qwen2.5 (Apache-2.0), Llama 3.2 (Llama 3.2 Community License — redistribution of derived | |
| 9 | +weights to be checked before release). No restricted or personal data are used. Nothing under `data/` is | |
| 10 | +committed except this file. | |
added
docs/COMPUTE_BUDGET.md
+30 −0
@@ -0,0 +1,30 @@ | ||
| 1 | +# Compute budget — WP-AI-0001 (measured where stated) | |
| 2 | + | |
| 3 | +## Resources audited 2026-11-09 (`scripts/audit_environment.py --cluster`, `docs/ENVIRONMENT_AUDIT.json`) | |
| 4 | + | |
| 5 | +- Laptop: Apple M5 Max, 48 GB unified memory, 18 cores, 921 GB free; Python 3.14.4; torch 2.12.0 (MPS on), | |
| 6 | + MLX 0.32.2 (Metal on), mlx_lm 0.31.3, transformers 5.8.1. Cached checkpoints: Qwen2.5-1.5B-Instruct-bf16, | |
| 7 | + Qwen2.5-3B-Instruct-bf16, Llama-3.2-3B-Instruct-bf16 (mlx-community), OLMoE-1B-7B-0125-Instruct-4bit. | |
| 8 | +- MacLustr nodes reachable with Python ≥ 3.14: M3U96a (M3 Ultra 96 GB, mlx_lm 0.31.3), M3U96b (M3 Ultra 96 GB, | |
| 9 | + mlx_lm 0.31.3, 110 GB free), M4M64a (M4 Max 64 GB, torch 2.12.1), M4M64b (M4 Max 64 GB, 16 GB free — not | |
| 10 | + usable for checkpoints). Nodes with only system Python 3.9 (M2U64, M4M64c, M5M36, M4BP48, m4mm, m4mn) | |
| 11 | + need `mld prepare`-style setup before use. M4M36 was unreachable. | |
| 12 | +- No paid compute is required for Tiers 0–1. Tier 2 fits M3U96a/b. | |
| 13 | + | |
| 14 | +## Measured costs (Tier 0) | |
| 15 | + | |
| 16 | +| Item | Measurement | | |
| 17 | +|---|---| | |
| 18 | +| Toy model 0.80 M params, 200 optimizer steps, batch 256 | 5.3 s MPS, 10.0 s CPU | | |
| 19 | +| 3 000 steps (one model) | ≈ 44 s MPS | | |
| 20 | +| Quick CPU smoke test (2 seeds × 2 roles, 200 steps, localization) | 22 s | | |
| 21 | +| Full pilot (3 seeds × 2 roles, 3 000 steps ≈ 45 s each, localization 3 families × 4 layers × 7 positions × 2 repeats + heads, bootstrap 2 000) | **276 s** on the laptop (MPS), run `pilot_tier0_opseq_451ef5087182` | | |
| 22 | + | |
| 23 | +## Projected (not measured) — to be replaced by pilot measurements | |
| 24 | + | |
| 25 | +- Tier 1 activation capture on a 3B bf16 model: ≈ 6–7 GB weights + activations; one layer × 1 000 prompts × | |
| 26 | + 32 tokens × 3 072 dims in fp16 ≈ 200 MB; feasible on the laptop. Full-layer capture for 36 layers is to be | |
| 27 | + streamed to disk (do not hold in memory). | |
| 28 | +- Tier 2 (7B–14B bf16): 14–28 GB weights; M3U96a/b only; one node per model. | |
| 29 | + | |
| 30 | +Rule: no parallel heavy runs until the single-node profile is recorded in `EXPERIMENT_LOG.md`. | |
added
docs/DECISION_LOG.md
+23 −0
@@ -0,0 +1,23 @@ | ||
| 1 | +# Decision log — WP-AI-0001 | |
| 2 | + | |
| 3 | +All entries are dated; "before freeze" refers to the confirmatory freeze (PREREGISTRATION v1.0, not yet done). | |
| 4 | + | |
| 5 | +- **D1 (2026-11-09, before freeze).** Project located at `Recherche-UQO/ia/wp-ai-0001` with git remote | |
| 6 | + `gitsrv:srv/git/wpai0001_uqo.git`, following the FIRC-0001 precedent for non-numeric IDs; UQO paper | |
| 7 | + template (`_modele-working-paper`) adopted; English throughout. | |
| 8 | +- **D2 (2026-11-09, before freeze).** OPSEQ output format changed from single final answer to a | |
| 9 | + *scratchpad trace* after the single-token variant failed to generalize (train loss → 0, dev ≈ 0.24–0.34 | |
| 10 | + on length ≤ 3; ≈ 0.13 on length ≤ 5). Consequence: the Tier-0 target capability is "execute Ω in context", | |
| 11 | + a per-step map, and compositional difficulty is controlled by `trace_stride`. Risk R1 recorded. | |
| 12 | +- **D3 (2026-11-09, before freeze).** Micro-world enlarged from 6 operators / length ≤ 4 (≈ 14 k unique | |
| 13 | + items, too small for 40 k training items) to 8 operators / length ≤ 5 (≈ 400 k). | |
| 14 | +- **D4 (2026-11-09, before freeze).** Toy models are PyTorch (hooks trivially exact, CPU-testable); MLX is | |
| 15 | + reserved for Tier ≥ 1 with a layer-wrapping intervention mechanism and an equivalence test. | |
| 16 | +- **D5 (2026-11-09, before freeze).** The pilot is exploratory; localization selection uses the pilot's | |
| 17 | + development localization set; confirmatory evaluation splits will be generated from new seeds after | |
| 18 | + freeze. | |
| 19 | +- **D6 (2026-11-09, before freeze).** Localization sweep families in the pilot were `resid_post`, `attn_out`, | |
| 20 | + `mlp_out` (plus heads). The `embed`/`resid_pre.0` family was not swept, which is why patching at the Ω | |
| 21 | + token restores nothing; it is added to the next sweep. The pilot's localization set (length-3 programs, | |
| 22 | + Ω at index 1, corrupt = `zib`) is a *development* set; confirmatory localization will use new seeds and | |
| 23 | + other corrupt operators. | |
added
docs/ENVIRONMENT_AUDIT.json
+175 −0
@@ -0,0 +1,175 @@ | ||
| 1 | +{ | |
| 2 | + "utc": "2026-11-09T06:50:06+00:00", | |
| 3 | + "hardware": { | |
| 4 | + "platform": "macOS-27.0-arm64-arm-64bit-Mach-O", | |
| 5 | + "machine": "arm64", | |
| 6 | + "hostname_hash": "9ad7358c662e", | |
| 7 | + "cpu_count": 18, | |
| 8 | + "chip": "Apple M5 Max", | |
| 9 | + "memory_gb": 48.0, | |
| 10 | + "torch_mps": true, | |
| 11 | + "torch_cuda": false | |
| 12 | + }, | |
| 13 | + "libraries": { | |
| 14 | + "python": "3.14.4", | |
| 15 | + "numpy": "2.4.4", | |
| 16 | + "torch": "2.12.0", | |
| 17 | + "scipy": "1.17.1", | |
| 18 | + "pydantic": "2.13.4", | |
| 19 | + "mlx": "?", | |
| 20 | + "mlx_lm": "0.31.3", | |
| 21 | + "transformers": "5.8.1", | |
| 22 | + "safetensors": "0.7.0" | |
| 23 | + }, | |
| 24 | + "backends": { | |
| 25 | + "mlx": { | |
| 26 | + "version": "0.32.2", | |
| 27 | + "metal": true | |
| 28 | + }, | |
| 29 | + "torch": { | |
| 30 | + "version": "2.12.0", | |
| 31 | + "mps": true, | |
| 32 | + "cuda": false | |
| 33 | + } | |
| 34 | + }, | |
| 35 | + "hf_cache": [ | |
| 36 | + { | |
| 37 | + "repo": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", | |
| 38 | + "size_gb": 5.46 | |
| 39 | + }, | |
| 40 | + { | |
| 41 | + "repo": "mlx-community/Llama-3.2-3B-Instruct-bf16", | |
| 42 | + "size_gb": 11.99 | |
| 43 | + }, | |
| 44 | + { | |
| 45 | + "repo": "mlx-community/OLMoE-1B-7B-0125-Instruct-4bit", | |
| 46 | + "size_gb": 7.26 | |
| 47 | + }, | |
| 48 | + { | |
| 49 | + "repo": "mlx-community/Qwen2.5-1.5B-Instruct-bf16", | |
| 50 | + "size_gb": 5.77 | |
| 51 | + }, | |
| 52 | + { | |
| 53 | + "repo": "mlx-community/Qwen2.5-3B-Instruct-bf16", | |
| 54 | + "size_gb": 11.52 | |
| 55 | + }, | |
| 56 | + { | |
| 57 | + "repo": "mlx-community/whisper-large-v3-turbo", | |
| 58 | + "size_gb": 3.01 | |
| 59 | + }, | |
| 60 | + { | |
| 61 | + "repo": "sentence-transformers/all-MiniLM-L6-v2", | |
| 62 | + "size_gb": 0.17 | |
| 63 | + }, | |
| 64 | + { | |
| 65 | + "repo": "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2", | |
| 66 | + "size_gb": 0.89 | |
| 67 | + } | |
| 68 | + ], | |
| 69 | + "cluster": { | |
| 70 | + "M3U96a": { | |
| 71 | + "reachable": true, | |
| 72 | + "chip": "Apple M3 Ultra", | |
| 73 | + "memory_gb": 96, | |
| 74 | + "free_disk_gb": 231, | |
| 75 | + "python": "3.14.5", | |
| 76 | + "mlx_lm": "0.31.3", | |
| 77 | + "torch": "none", | |
| 78 | + "hf_models_cached": 9 | |
| 79 | + }, | |
| 80 | + "M3U96b": { | |
| 81 | + "reachable": true, | |
| 82 | + "chip": "Apple M3 Ultra", | |
| 83 | + "memory_gb": 96, | |
| 84 | + "free_disk_gb": 109, | |
| 85 | + "python": "3.14.4", | |
| 86 | + "mlx_lm": "0.31.3", | |
| 87 | + "torch": "none", | |
| 88 | + "hf_models_cached": 11 | |
| 89 | + }, | |
| 90 | + "M2U64": { | |
| 91 | + "reachable": true, | |
| 92 | + "chip": "Apple M2 Ultra", | |
| 93 | + "memory_gb": 64, | |
| 94 | + "free_disk_gb": 244, | |
| 95 | + "python": "3.9.6", | |
| 96 | + "mlx_lm": "none", | |
| 97 | + "torch": "none", | |
| 98 | + "hf_models_cached": 2 | |
| 99 | + }, | |
| 100 | + "M4M64a": { | |
| 101 | + "reachable": true, | |
| 102 | + "chip": "Apple M4 Max", | |
| 103 | + "memory_gb": 64, | |
| 104 | + "free_disk_gb": 285, | |
| 105 | + "python": "3.14.4", | |
| 106 | + "mlx_lm": "none", | |
| 107 | + "torch": "2.12.1", | |
| 108 | + "hf_models_cached": 9 | |
| 109 | + }, | |
| 110 | + "M4M64b": { | |
| 111 | + "reachable": true, | |
| 112 | + "chip": "Apple M4 Max", | |
| 113 | + "memory_gb": 64, | |
| 114 | + "free_disk_gb": 16, | |
| 115 | + "python": "3.14.4", | |
| 116 | + "mlx_lm": "none", | |
| 117 | + "torch": "none", | |
| 118 | + "hf_models_cached": 2 | |
| 119 | + }, | |
| 120 | + "M4M64c": { | |
| 121 | + "reachable": true, | |
| 122 | + "chip": "Apple M4 Max", | |
| 123 | + "memory_gb": 64, | |
| 124 | + "free_disk_gb": 277, | |
| 125 | + "python": "3.9.6", | |
| 126 | + "mlx_lm": "none", | |
| 127 | + "torch": "none", | |
| 128 | + "hf_models_cached": 2 | |
| 129 | + }, | |
| 130 | + "M5M36": { | |
| 131 | + "reachable": true, | |
| 132 | + "chip": "Apple M5 Max", | |
| 133 | + "memory_gb": 36, | |
| 134 | + "free_disk_gb": 302, | |
| 135 | + "python": "3.9.6", | |
| 136 | + "mlx_lm": "none", | |
| 137 | + "torch": "none", | |
| 138 | + "hf_models_cached": 5 | |
| 139 | + }, | |
| 140 | + "M4M36": { | |
| 141 | + "reachable": false, | |
| 142 | + "error": "ssh: connect to host m4m36.maclustr.io port 22: Operation timed out" | |
| 143 | + }, | |
| 144 | + "M4BP48": { | |
| 145 | + "reachable": true, | |
| 146 | + "chip": "Apple M4 Max", | |
| 147 | + "memory_gb": 48, | |
| 148 | + "free_disk_gb": 806, | |
| 149 | + "python": "3.9.6", | |
| 150 | + "mlx_lm": "none", | |
| 151 | + "torch": "none", | |
| 152 | + "hf_models_cached": 5 | |
| 153 | + }, | |
| 154 | + "m4mm": { | |
| 155 | + "reachable": true, | |
| 156 | + "chip": "Apple M4 Pro", | |
| 157 | + "memory_gb": 48, | |
| 158 | + "free_disk_gb": 346, | |
| 159 | + "python": "3.9.6", | |
| 160 | + "mlx_lm": "none", | |
| 161 | + "torch": "none", | |
| 162 | + "hf_models_cached": 1 | |
| 163 | + }, | |
| 164 | + "m4mn": { | |
| 165 | + "reachable": true, | |
| 166 | + "chip": "Apple M4 Pro", | |
| 167 | + "memory_gb": 48, | |
| 168 | + "free_disk_gb": 338, | |
| 169 | + "python": "3.9.6", | |
| 170 | + "mlx_lm": "none", | |
| 171 | + "torch": "none", | |
| 172 | + "hf_models_cached": 1 | |
| 173 | + } | |
| 174 | + } | |
| 175 | +} | |
| \ No newline at end of file | ||
added
docs/EXPERIMENT_LOG.md
+43 −0
@@ -0,0 +1,43 @@ | ||
| 1 | +# Experiment log — WP-AI-0001 | |
| 2 | + | |
| 3 | +| Date (UTC) | Run / action | Status | Evidence | Notes | | |
| 4 | +|---|---|---|---|---| | |
| 5 | +| 2026-11-09 | Environment audit (laptop + 11 cluster nodes) | completed | `docs/ENVIRONMENT_AUDIT.json` | M4M36 unreachable; mlx_lm only on M3U96a/b + laptop | | |
| 6 | +| 2026-11-09 | Literature verification (40 arXiv records) | completed | `literature/arxiv_verified_2026-11-09.txt`, `literature/references_verified.bib` | abstracts read for 9 closest works | | |
| 7 | +| 2026-11-09 | Single-token OPSEQ learnability probe (4 layers, d=128, 4 000 steps) | failed (dev ≈ 0.13; length ≤ 3 subset: train loss → 0, dev 0.24) | console only, not archived | led to D2 (scratchpad) | | |
| 8 | +| 2026-11-09 | Scratchpad OPSEQ learnability probe (3 000 steps) | completed | console only | dev 1.0 by step 500; held-out compositions 1.0; length OOD 0.086 | | |
| 9 | +| 2026-11-09 | `pilot_tier0_opseq_quick_d5339bcad6b3` (CPU smoke, 2 seeds, 200 steps) | completed | `experiments/summaries/pilot_tier0_opseq_quick_d5339bcad6b3/` | pipeline end-to-end; models untrained by design | | |
| 10 | +| 2026-11-09 | `pilot_tier0_opseq_451ef5087182` (MPS, 3 seeds × 2 roles, 3 000 steps, 0.80 M params) | **completed** (276 s) | `experiments/summaries/pilot_tier0_opseq_451ef5087182/{results.json,SUMMARY.md}`, manifest, 4 figures | see §Pilot results below | | |
| 11 | + | |
| 12 | +## Pilot results — `pilot_tier0_opseq_451ef5087182` (exploratory; 2026-11-09) | |
| 13 | + | |
| 14 | +**Gate B (gap).** Final-answer exact match, mean over 3 seeds (sd): donor 1.000 (0.000) vs recipient | |
| 15 | +0.092 (0.053) on `target_iid` (majority-class chance 0.105); both 1.000 on `control_iid`; `comp_heldout` | |
| 16 | +1.000 vs 0.450 (held-out pairs often contain Ω); `len_ood` 0.086 vs 0.091 (both fail: absolute positions | |
| 17 | +beyond training length). Ω-step accuracy on `target_iid`: donor 1.000, recipient 0.062. Paired gap on | |
| 18 | +identical items, seed 0: +0.847 [+0.825, +0.870] (cluster bootstrap over programs, 2 000 draws). | |
| 19 | +Recipient trace accuracy on `target_iid`: 0.110 / 0.024 / 0.013 (seeds 0/1/2). | |
| 20 | + | |
| 21 | +**Gate C (hooks).** For all 6 trained models: repeat-run bitwise equality, capture-all bitwise identity, | |
| 22 | +disabled-modifying-hook bitwise identity = True; enabled zero hook changes output = True (256 items). | |
| 23 | + | |
| 24 | +**RQ1 (localization, within donor, n = 490 clean/corrupt pairs, read position r1 → predicts r2).** | |
| 25 | +Clean 1.000, sham (corrupted run scored on clean answer) 0.000 in all seeds. Restored-answer rate when | |
| 26 | +patching clean into corrupt (mean of 3 seeds): `resid_post` = 1.00 at the read position r1 at every layer | |
| 27 | +L0–L3 and 0.00 at every other position (including the Ω token position); `attn_out` = 1.00 at (L0, r1) only; | |
| 28 | +`mlp_out` at r1: L0 1.00, L1 0.90, L2 0.83, L3 0.17. Zero-ablation accuracy at r1: `resid_post` 0.04–0.09, | |
| 29 | +`attn_out` L0 0.13 (L1–L3 1.00), `mlp_out` L0 0.21, L1 0.97. Head patching at r1: L0 h0 0.62, h1 0.16, | |
| 30 | +h2 0.04, others 0.00; head patching at the Ω token: 0.00 everywhere. Repeat max |Δ| = 0.0000 for all | |
| 31 | +sweeps; across-seed Pearson of flip heatmaps: resid_post 1.000, attn_out 1.000, mlp_out 0.964 (min pair). | |
| 32 | + | |
| 33 | +**Reading (observed → interpretation).** Observed: the Ω computation is executed entirely at the read | |
| 34 | +position: layer-0 attention (mostly head 0) moves the operator identity from the Ω token to r1, and | |
| 35 | +layer-0–2 MLPs at r1 compute the value map. Patching the Ω token position itself restores nothing because | |
| 36 | +layer-0 attention reads the *embedding* (hook `resid_pre.0`/`embed`, not swept) and no later attention | |
| 37 | +re-reads it. Interpretation (not yet tested): the candidate organ for an activation graft is the | |
| 38 | +residual stream at r1 after L0 (or the L0 attention output at r1); its *content* is the (operator, value) | |
| 39 | +pair, so a transplant must convey the Ω-specific part of that content. Go/no-go for Sprint 3: all three | |
| 40 | +criteria of EXPERIMENT_PLAN §4 are met. | |
| 41 | + | |
| 42 | +**Pilot variability for power (PREREGISTRATION §5).** Seed sd of recipient target accuracy 0.053; cluster | |
| 43 | +bootstrap half-width of the paired gap ≈ 0.022 with 1 000 items / ≈ 900 programs. | |
added
docs/EXPERIMENT_PLAN.md
+56 −0
@@ -0,0 +1,56 @@ | ||
| 1 | +# Experiment plan — WP-AI-0001 | |
| 2 | + | |
| 3 | +## 1. Tiers and model matrix | |
| 4 | + | |
| 5 | +| Tier | Models | Purpose | Backend | Status | | |
| 6 | +|---|---|---|---|---| | |
| 7 | +| 0 | toy decoder transformers trained from scratch (0.8 M params default; 4 layers, d = 128) | full visibility, controlled gap, exhaustive interventions | PyTorch (CPU/MPS) | **pilot running** | | |
| 8 | +| 1 | Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct (bf16, cached locally) | external validity, same-family different-size, cross-family, cross-tokenizer | MLX (layer-wrapping interventions) with a PyTorch/Transformers equivalence check | planned (after Gate D on Tier 0) | | |
| 9 | +| 2 | 7B–14B open weights (e.g. Qwen2.5-7B/14B, Llama-3.1-8B) | scale | MLX on M3U96a/b (96 GB) | conditional | | |
| 10 | +| 3 | larger | only with explicit approval | — | not planned | | |
| 11 | + | |
| 12 | +Pairs (Tier 1): Qwen2.5-1.5B→3B (same family, same tokenizer, 28→36 layers); Llama-3.2-3B↔Qwen2.5-3B | |
| 13 | +(different family, different tokenizer, 28 vs 36 layers); Tier 0: same architecture / different seeds, | |
| 14 | +same architecture / different widths, different depths. | |
| 15 | + | |
| 16 | +## 2. Task families | |
| 17 | + | |
| 18 | +- **A — OPSEQ** (implemented): invented operators on Z_11, scratchpad trace; strata `target_iid`, | |
| 19 | + `control_iid`, `comp_heldout`, `len_ood`; difficulty knob `trace_stride`. | |
| 20 | +- **B — FSA** (generator implemented): random DFA execution over an invented alphabet. | |
| 21 | +- **C — MORPH** (generator implemented): context-dependent suffix rule on pseudo-words; Tier 1 variant | |
| 22 | + with natural-language pseudo-morphology to be specified at Sprint 4. | |
| 23 | +- Negative-control battery (Tier 1): unrelated arithmetic, factual recall, instruction following, | |
| 24 | + held-out text perplexity, calibration, refusal behaviour. | |
| 25 | + | |
| 26 | +## 3. Phases (per CLAUDE.md §6) and their scripts | |
| 27 | + | |
| 28 | +| Phase | Script | Output | | |
| 29 | +|---|---|---| | |
| 30 | +| 1 Baseline & gap | `scripts/run_pilot.py` | gap table, Ω-step accuracies, no-op checks | | |
| 31 | +| 2 Localization | `scripts/run_pilot.py` (within-donor patching, ablation, heads) | heatmaps, reproducibility | | |
| 32 | +| 3 Alignment | `scripts/run_alignment.py` (to write) | Procrustes / ridge / CCA / random / unrelated-data maps with leakage audit | | |
| 33 | +| 4 Transplant | `scripts/run_transplant.py` (to write) | activation graft, subspace graft, weight graft (same-arch), adapter graft | | |
| 34 | +| 5 Controls | same | sham, norm-matched random, unrelated-skill donor, shuffled token/layer, few-shot, steering, matched adapter FT, distillation | | |
| 35 | +| 6 Robustness | `scripts/run_sweeps.py` (to write) | strength × layer × position × dimension × pair | | |
| 36 | +| 7 Mechanism | `scripts/run_mechanism.py` (to write) | ablation→loss→rescue, bidirectional, dose-response | | |
| 37 | +| 8 Confirmatory | `scripts/run_confirmatory.py` (to write) | frozen protocol on untouched splits | | |
| 38 | + | |
| 39 | +## 4. Minimal viable experiment (MVE) and go/no-go | |
| 40 | + | |
| 41 | +MVE = Tier 0, OPSEQ, Ω = `qua`, donor (all operators) vs recipient (Ω excluded), 3 seeds each. | |
| 42 | +Go to Sprint 3 (first graft) iff: (a) donor Ω-step accuracy ≥ 0.95 and recipient ≤ chance + 0.1 on | |
| 43 | +`target_iid`; (b) all no-op checks true; (c) at least one localization cell with restored-answer rate | |
| 44 | +≥ 0.8 and across-seed heatmap correlation ≥ 0.8. Otherwise diagnose (see RISK_REGISTER R1–R3). | |
| 45 | + | |
| 46 | +## 5. Pre-declared confirmatory contrasts (Tier 0) | |
| 47 | + | |
| 48 | +1. activation graft vs matched sham (primary); 2. subspace graft vs norm-matched random subspace; | |
| 49 | +3. weight graft (same architecture) vs shuffled-layer weight graft; 4. adapter graft vs matched-size | |
| 50 | +adapter trained on the same alignment data without donor signal. Each: Δ_T^adj with cluster-bootstrap CI, | |
| 51 | +Holm-corrected across the four families; retention on `control_iid`. | |
| 52 | + | |
| 53 | +## 6. Figures that the plan can support (data-dependent) | |
| 54 | + | |
| 55 | +Gap distributions (F2), localization heatmap (F3), Pareto target-vs-retention (F5), portability matrix | |
| 56 | +(F6), graft size vs effect (F7), ablation→rescue chain (F8), robustness (F9), failure taxonomy (F10). | |
added
docs/MANDATE.md
+367 −0
@@ -0,0 +1,367 @@ | ||
| 1 | +# Research mandate — WP-AI-0001 (supplied by the author on 2026-11-09, reproduced verbatim) | |
| 2 | + | |
| 3 | +CLAUDE.md — WP-AI-0001 | |
| 4 | + | |
| 5 | +The Neural Organ Transplant Experiment | |
| 6 | + | |
| 7 | +Causal Transfer of Cognitive Capabilities Across Local Large Language Models | |
| 8 | + | |
| 9 | +Project ID: wp-ai-0001 | |
| 10 | +Research home: UQO Research | |
| 11 | +Document language: English (all code documentation, figures, paper, logs, and reports) | |
| 12 | +Status: Research protocol and implementation mandate; no results may be claimed before execution. | |
| 13 | + | |
| 14 | +──────── | |
| 15 | + | |
| 16 | +0. MISSION AND NON-NEGOTIABLE STANDARD | |
| 17 | + | |
| 18 | +You are Claude Code acting as a principal research engineer, computational scientist, and rigorous coauthor. Build a complete, reproducible research program within the existing UQO Research environment under a new project named wp-ai-0001. | |
| 19 | + | |
| 20 | +Central question: Can a bounded, causally identified computational subsystem responsible for a measurable cognitive capability be transplanted from one open-weight language model (the donor) into another (the recipient), such that the recipient acquires or materially improves that capability without conventional full-model training? | |
| 21 | + | |
| 22 | +The word organ is a metaphor. Do not imply that LLMs have biological organs, consciousness, or localized faculties by default. The paper must operationalize a candidate functional subsystem and test whether it is transferable. | |
| 23 | + | |
| 24 | +A publication-quality result may be positive, negative, or conditional. Never fabricate empirical outcomes, references, citations, benchmarks, or performance numbers. Do not confuse a change in output distribution with genuine skill acquisition. Explicitly distinguish causal interventions, associations, and mechanistic interpretation. | |
| 25 | + | |
| 26 | +Primary deliverables: (1) executable open-weight experimentation framework, (2) documented experiment registry and artifacts, (3) validated empirical analysis, (4) rigorous English LaTeX paper, (5) publication-grade figures, (6) independent replication guide, and (7) honest limitations and failure analyses. | |
| 27 | + | |
| 28 | +1. FIRST ACTIONS: DISCOVER, AUDIT, PLAN | |
| 29 | + | |
| 30 | +Before writing implementation code: | |
| 31 | + | |
| 32 | +1. Locate the existing UQO Research repository/workspace. Inspect its README, project naming conventions, build systems, paper directories, data policies, Git remotes, and existing experimental frameworks. Never assume a path or remote URL. | |
| 33 | +2. Inspect the current machine and accessible MacLustr compute nodes, CPU/GPU/Metal support, RAM, storage, installed Python environments, MLX/PyTorch/Transformers versions, network accessibility, and permission boundaries. Never assume a cluster node is online. | |
| 34 | +3. Check whether wp-ai-0001 already exists. If so, preserve work and reconcile rather than overwrite. | |
| 35 | +4. Search current peer-reviewed and preprint literature with primary-source verification. At minimum investigate causal tracing, activation patching, interchange interventions, causal abstraction, sparse autoencoders, circuit discovery, model editing, task vectors, model merging, cross-model representation alignment, cross-architecture transfer, model stitching, knowledge transplantation, and inference-time activation steering. Record URLs, identifiers, publication dates, precise contributions, and how our approach differs. If web access is unavailable, mark literature review unverified and do not invent citations. | |
| 36 | +5. Write docs/RESEARCH_GAP.md, docs/PREREGISTRATION.md, docs/EXPERIMENT_PLAN.md, docs/COMPUTE_BUDGET.md, docs/RISK_REGISTER.md, and an implementation task tracker before major experiments. | |
| 37 | +6. Identify the minimal viable experiment and its go/no-go criteria. Begin with a small, affordable pilot before scaling. | |
| 38 | + | |
| 39 | +Do not claim novelty merely because the term "neural organ transplant" sounds new. The scientific novelty must survive comparison with prior model stitching, representation transfer, circuit transplantation, and activation interventions. | |
| 40 | + | |
| 41 | +2. CORE DEFINITIONS AND ESTIMANDS | |
| 42 | + | |
| 43 | +Let donor model be D, recipient model be R, task distribution T, candidate subsystem S_D, alignment/transplant operator A, and intervention operator I. Define a transplanted recipient R' = I(R, A(S_D)). | |
| 44 | + | |
| 45 | +Predefine a primary capability score Q_T(M) and control-task score Q_C(M), with uncertainty intervals. Let recipient baseline improvement be Δ_T = Q_T(R') - Q_T(R), and control retention Δ_C = Q_C(R') - Q_C(R). | |
| 46 | + | |
| 47 | +The primary causal estimand is the excess effect of a donor-specific intervention over matched sham, random, and nonfunctional donor interventions, estimated on held-out, leakage-controlled tasks. Report the difference in differences against a matched intervention control where appropriate, not only raw Δ_T. | |
| 48 | + | |
| 49 | +Operational definitions: | |
| 50 | + | |
| 51 | +• Candidate organ: a specified set of layers, heads, MLP units, activation subspaces, sparse features, or pathways whose perturbation produces repeatable, task-selective causal effects. | |
| 52 | +• Transplant: a predeclared intervention that injects, replaces, composes, or maps a donor-derived functional component into the recipient. | |
| 53 | +• Acquisition: held-out improvement on the target capability, beyond controls and chance, with retention of general capabilities and evidence against answer leakage. | |
| 54 | +• Selectivity: target-task improvement relative to prespecified negative-control tasks and matched compute/parameter interventions. | |
| 55 | +• Persistence: distinguish single-token intervention, within-sequence effect, across-prompts reuse, and saved model modification. Never call transient activation steering permanent learning. | |
| 56 | +• Portability: transfer across seeds, prompts, domains, model pairs, and architectures. | |
| 57 | + | |
| 58 | +A successful transplant does not require a literal physical copy of identical neurons. Separately evaluate exact-weight copying, mapped activation transfer, and learned adapter-based transplantation; label them precisely. | |
| 59 | + | |
| 60 | +3. RESEARCH QUESTIONS AND FALSIFIABLE HYPOTHESES | |
| 61 | + | |
| 62 | +RQ1 — Localization: Can a candidate capability-relevant subsystem be identified reproducibly through causal intervention rather than correlation alone? | |
| 63 | + | |
| 64 | +RQ2 — Sufficiency: Does transferring the subsystem increase recipient target-task performance relative to matched controls? | |
| 65 | + | |
| 66 | +RQ3 — Necessity: Does removing or perturbing the corresponding donor subsystem reduce donor performance selectively? | |
| 67 | + | |
| 68 | +RQ4 — Portability: How does transplant effectiveness vary with model family, architecture, size, layer depth, representation similarity, and tokenizer differences? | |
| 69 | + | |
| 70 | +RQ5 — Collateral damage: Which recipient capabilities degrade after transplantation, and how does damage scale with intervention strength and size? | |
| 71 | + | |
| 72 | +RQ6 — Persistence and mechanism: Are improvements sustained on unseen prompts, multi-step reasoning, compositional generalization, and adversarial paraphrases? Is the effect explained by reusable representations rather than answer copying? | |
| 73 | + | |
| 74 | +RQ7 — Transfer frontier: What is the minimum transplanted dimensionality/parameter budget needed for a specified improvement? | |
| 75 | + | |
| 76 | +RQ8 — Failure regimes: Under what conditions is transplantation impossible, unstable, or indistinguishable from simpler baselines? | |
| 77 | + | |
| 78 | +Preregister directional hypotheses, null hypotheses, primary outcomes, familywise comparisons, smallest effect size of interest, and failure thresholds. Do not retroactively relabel exploratory findings as confirmatory. | |
| 79 | + | |
| 80 | +4. TASK DESIGN: DEMONSTRATE A REAL CAPABILITY | |
| 81 | + | |
| 82 | +Use at least three distinct task families and separate development from confirmatory evaluation. | |
| 83 | + | |
| 84 | +A. Synthetic compositional reasoning (primary identification environment) | |
| 85 | + | |
| 86 | +Create a controlled, procedurally generated micro-world with invented operators, typed rules, multi-step transformations, and held-out compositions. Control training exposure, difficulty, and ground-truth answers. Build donor competence and recipient weakness through explicitly documented small-scale training or selection; do not assume an arbitrary donor has a capability the recipient lacks. | |
| 87 | + | |
| 88 | +Separate: memorization, operator recognition, length generalization, novel symbol substitutions, unseen compositions, distractors, and out-of-distribution combinations. Ensure disjoint generation seeds and avoid shared templates across splits. Test whether transplant conveys a rule rather than cached answers. | |
| 89 | + | |
| 90 | +B. Formal algorithmic reasoning | |
| 91 | + | |
| 92 | +Use verifiable tasks such as modular arithmetic, finite-state transformations, structured symbolic execution, constrained graph problems, and small proof checking. Prefer exact-match validators and independently generated solutions. Avoid evaluating solely with a judge LLM. | |
| 93 | + | |
| 94 | +C. Language/semantic capability | |
| 95 | + | |
| 96 | +Use narrowly defined multilingual morphological transformations, compositional syntax, or domain-specific semantic distinctions. Control lexical overlap and translation artifacts. Evaluate semantic transfer separately from superficial token copying. | |
| 97 | + | |
| 98 | +Negative-control battery | |
| 99 | + | |
| 100 | +Include unrelated arithmetic, general language modeling, factual recall, instruction following, calibration, refusal/safety behavior, and perplexity on clean held-out text where legally and technically appropriate. The battery must detect broad degradation, not just target improvements. | |
| 101 | + | |
| 102 | +Contamination safeguards | |
| 103 | + | |
| 104 | +Generate held-out samples after intervention design freeze where possible. Hash datasets, version generators, preserve seeds, record overlap tests, and audit leakage through adapters, mapping datasets, prompt exemplars, and evaluation harnesses. | |
| 105 | + | |
| 106 | +5. MODEL MATRIX AND HARDWARE STRATEGY | |
| 107 | + | |
| 108 | +Use openly licensed/open-weight models compatible with local inspection, selecting exact checkpoints only after license, architecture, and compute audits. | |
| 109 | + | |
| 110 | +Tier 0: tiny synthetic transformers trained from scratch (approximately 10M–100M parameters if practical). Full visibility, controlled training data, repeated seeds, and inexpensive exhaustive interventions. | |
| 111 | + | |
| 112 | +Tier 1: small open-weight instruction/base models (roughly 0.5B–3B) from at least two families where available. Prioritize reproducibility and manageable activations. | |
| 113 | + | |
| 114 | +Tier 2: medium open-weight models (roughly 7B–14B) for external validity, only after the Tier 0/1 methods work. | |
| 115 | + | |
| 116 | +Tier 3: larger models only if their scientific marginal value justifies measured time and memory costs. | |
| 117 | + | |
| 118 | +Construct donor/recipient pairs covering: same architecture/different seeds; same family/different sizes; different families/similar sizes; different tokenizers; matched and mismatched layer counts. Distinguish base from instruction-tuned checkpoints. | |
| 119 | + | |
| 120 | +On Apple Silicon, prefer MLX/MLX-LM where hooks and interventions are demonstrably supported. Use PyTorch/Transformers or an isolated reference implementation where MLX does not expose necessary internals. Never pretend a framework supports a hook that it does not. Maintain equivalence tests across backends. Account for unified memory, KV-cache footprint, activation capture, quantization, and memory pressure. Do not launch heavy parallel runs until resource profiling is complete. | |
| 121 | + | |
| 122 | +Record for every run: exact model ID and revision, architecture, license, tokenizer, precision, quantization, library versions, host hardware, memory, seed, generation settings, runtime, energy estimate if measured, and intervention provenance. | |
| 123 | + | |
| 124 | +6. EXPERIMENTAL PIPELINE | |
| 125 | + | |
| 126 | +Phase 1 — Baseline and gap qualification | |
| 127 | + | |
| 128 | +Measure donor and recipient capability across task strata. Require a meaningful, reproducible donor-recipient gap before attempting transplant. Run multiple seeds and report variance. Document any donor training separately from transplantation. | |
| 129 | + | |
| 130 | +Phase 2 — Localization and causal validation | |
| 131 | + | |
| 132 | +Collect activations with strict memory limits. Screen candidate layers, attention heads, MLP outputs, residual-stream subspaces, and sparse features. Use activation patching, ablation, causal tracing, path patching where valid, and independent validation. Compare against activation-norm-matched random subspaces and arbitrary heads/layers. Check whether localization findings persist under paraphrase and held-out data. Do not equate attention weights with causal explanations. | |
| 133 | + | |
| 134 | +Phase 3 — Representation alignment | |
| 135 | + | |
| 136 | +Test orthogonal Procrustes, linear mappings, canonical correlation or related similarity methods, low-rank adapters, and architecture-aware projection. Train mappings only on disjoint alignment data. Quantify alignment data size, parameter count, fitting compute, and information leaked through alignment. Include a random mapping and a mapping trained on unrelated data. | |
| 137 | + | |
| 138 | +Phase 4 — Transplant families | |
| 139 | + | |
| 140 | +Implement each as a separately named, auditable method: | |
| 141 | + | |
| 142 | +1. Activation graft: inject mapped donor activation at a specified recipient layer/token position. | |
| 143 | +2. Subspace graft: replace or add a selected capability-associated low-dimensional representation. | |
| 144 | +3. Circuit/path graft: replace or couple a bounded pathway where architectures and interfaces allow it. | |
| 145 | +4. Weight graft: transfer structurally compatible parameter subsets only with shape, scaling, and functional checks. | |
| 146 | +5. Adapter graft: install a small learned interface; account explicitly for any supervised learning in the adapter. | |
| 147 | +6. Persistent graft: optionally save a modified checkpoint or adapter and evaluate it across independent sessions. | |
| 148 | + | |
| 149 | +Not every transplant type will be feasible for every pair. Mark unsupported configurations explicitly rather than silently substituting methods. | |
| 150 | + | |
| 151 | +Phase 5 — Controls and alternatives | |
| 152 | + | |
| 153 | +At minimum compare with: unmodified recipient, unmodified donor, random graft, norm-matched random graft, unrelated-skill donor graft, shuffled-token graft, shuffled-layer graft, sham mapping, direct prompting/few-shot, retrieval of equivalent information, simple activation steering, matched-size adapter fine-tuning, and a conventional distillation baseline where budget permits. | |
| 154 | + | |
| 155 | +Match parameter counts, inference-time overhead, training examples, and compute where possible. If comparisons cannot be matched, state why and interpret narrowly. | |
| 156 | + | |
| 157 | +Phase 6 — Robustness and stress tests | |
| 158 | + | |
| 159 | +Sweep intervention strength, layer, token position, dimensionality, donor/recipient pair, prompt format, context length, temperature, task difficulty, and quantization. Evaluate repeated independent seeds and withheld families. Measure harmful interference and stability, including catastrophic failures and invalid outputs. | |
| 160 | + | |
| 161 | +Phase 7 — Causal mechanism tests | |
| 162 | + | |
| 163 | +Use bidirectional and reciprocal transfers when meaningful; donor ablation followed by rescue; transplant into incompatible recipients; dose-response; mediation-style analyses with assumptions stated; and specificity tests. Establish whether the transplanted component carries task-relevant information or merely biases answer tokens. | |
| 164 | + | |
| 165 | +Phase 8 — Final confirmatory evaluation | |
| 166 | + | |
| 167 | +Freeze primary methods, hyperparameters, model pairs, and evaluation scripts. Run on untouched confirmatory datasets. Publish both successes and failures, with all preregistered metrics and comparisons. | |
| 168 | + | |
| 169 | +7. QUANTITATIVE METRICS | |
| 170 | + | |
| 171 | +Primary: held-out exact accuracy or task-specific objectively validated score; treatment effect versus matched sham; target-control selectivity; retention of general capability. | |
| 172 | + | |
| 173 | +Secondary: donor ablation effect, rescue effect, transplant dimensionality, number of modified parameters, alignment data requirements, inference latency, peak memory, energy if measured, success across task strata, and stability across seeds. | |
| 174 | + | |
| 175 | +Useful derived quantities (define precisely before use): | |
| 176 | + | |
| 177 | +• Transfer gain: Q_target(R_graft) - Q_target(R_base). | |
| 178 | +• Control-adjusted transfer: [Q_target(R_graft)-Q_target(R_base)] - [Q_target(R_sham)-Q_target(R_base)]. | |
| 179 | +• Retention ratio for each control task, with careful treatment of near-zero baselines. | |
| 180 | +• Transfer efficiency: gain per graft dimension, parameter, training example, or compute unit (report units and avoid misleading cross-metric comparisons). | |
| 181 | +• Donor-necessity score: donor target performance before minus after selective ablation. | |
| 182 | +• Portability matrix: donor-by-recipient effect with uncertainty. | |
| 183 | + | |
| 184 | +Do not compress all outcomes into one opaque "intelligence transfer score" unless weights and sensitivity analyses are fully justified. | |
| 185 | + | |
| 186 | +8. STATISTICAL DESIGN | |
| 187 | + | |
| 188 | +Predefine the unit of inference and avoid pseudoreplication: questions nested within task templates, random seeds, model pairs, and intervention conditions are not independent. Use hierarchical or mixed-effects models where appropriate; cluster bootstrap or robust uncertainty estimates at the correct generation/template/model level; paired comparisons on identical questions; and multiple-comparison corrections for large intervention searches. | |
| 189 | + | |
| 190 | +Report confidence intervals, effect sizes, p-values where relevant, exact sample sizes, random seeds, and full null results. Separate exploratory localization selection from confirmatory testing. Perform prospective power/simulation analyses using pilot variability, not invented effect sizes. Test equivalence/non-inferiority for collateral damage only with justified margins. Include sensitivity to evaluation metric, task composition, and model pair. | |
| 191 | + | |
| 192 | +Avoid strong causal claims when intervention assignment, alignment training, or sample selection creates confounding. State precisely what is randomized and what is not. | |
| 193 | + | |
| 194 | +9. ENGINEERING ARCHITECTURE | |
| 195 | + | |
| 196 | +Create or adapt a maintainable project structure consistent with UQO Research: | |
| 197 | + | |
| 198 | +wp-ai-0001/ | |
| 199 | + CLAUDE.md | |
| 200 | + README.md | |
| 201 | + pyproject.toml | |
| 202 | + configs/ | |
| 203 | + models/ tasks/ experiments/ interventions/ | |
| 204 | + src/wp_ai_0001/ | |
| 205 | + models/ # model adapters, tokenizer compatibility, hooks | |
| 206 | + datasets/ # synthetic generators, validators, splits | |
| 207 | + baselines/ # model/task baselines and alternative methods | |
| 208 | + localization/ # ablation, patching, circuit discovery | |
| 209 | + alignment/ # donor-recipient representation mapping | |
| 210 | + transplantation/ # explicit intervention implementations | |
| 211 | + evaluation/ # blinded metrics and controls | |
| 212 | + statistics/ # estimation, uncertainty, multiplicity | |
| 213 | + instrumentation/ # memory, runtime, provenance, logs | |
| 214 | + visualization/ # publication figures | |
| 215 | + orchestration/ # local/cluster jobs, resumability | |
| 216 | + tests/ | |
| 217 | + unit/ integration/ causal_sanity/ regression/ | |
| 218 | + experiments/ | |
| 219 | + registry/ manifests/ summaries/ | |
| 220 | + data/ | |
| 221 | + README.md # data policy; do not commit restricted data | |
| 222 | + artifacts/ # generated artifacts, Git-ignored as appropriate | |
| 223 | + docs/ | |
| 224 | + RESEARCH_GAP.md | |
| 225 | + PREREGISTRATION.md | |
| 226 | + EXPERIMENT_PLAN.md | |
| 227 | + COMPUTE_BUDGET.md | |
| 228 | + RISK_REGISTER.md | |
| 229 | + REPRODUCIBILITY.md | |
| 230 | + EXPERIMENT_LOG.md | |
| 231 | + DECISION_LOG.md | |
| 232 | + paper/ | |
| 233 | + main.tex | |
| 234 | + references.bib | |
| 235 | + sections/ | |
| 236 | + figures/ | |
| 237 | + tables/ | |
| 238 | + appendix/ | |
| 239 | + submission/ | |
| 240 | + scripts/ | |
| 241 | + audit_environment.py | |
| 242 | + run_pilot.py | |
| 243 | + run_confirmatory.py | |
| 244 | + build_figures.py | |
| 245 | + build_paper.sh | |
| 246 | + | |
| 247 | +Use typed configuration and deterministic run IDs. Each experiment must have a manifest containing git commit, config hash, checkpoint revisions, data hashes, seeds, backend, hardware, timestamps, and outcome files. Support checkpoint/resume, interrupted jobs, structured logging, and a dry-run mode. Provide CPU-compatible smoke tests on toy models. Make paths configurable; never hardcode private tokens, endpoints, or personal file paths. Keep secrets in environment variables and redact them from logs. | |
| 248 | + | |
| 249 | +Do not create unnecessary distributed infrastructure before proving the scientific experiment works on one machine. | |
| 250 | + | |
| 251 | +10. VALIDATION AND QUALITY GATES | |
| 252 | + | |
| 253 | +Gate A — Repository readiness: existing workspace understood, new project isolated, environment audit passed, literature gap documented. | |
| 254 | + | |
| 255 | +Gate B — Baseline integrity: data validators correct, donor/recipient skill gap verified, contamination tests passed, repeated baseline runs consistent. | |
| 256 | + | |
| 257 | +Gate C — Mechanistic sanity: positive controls work on toy models; negative controls behave plausibly; hooks do not change outputs when disabled; activation shapes and token alignments are verified. | |
| 258 | + | |
| 259 | +Gate D — Transplant validity: graft is reproducible; sham and random controls implemented; information leakage audit passed; overhead and training cost disclosed. | |
| 260 | + | |
| 261 | +Gate E — Confirmatory evidence: untouched data, locked analysis, uncertainty estimates, and negative results all reported. | |
| 262 | + | |
| 263 | +Gate F — Paper integrity: every numeric claim traces to an artifact; every figure regenerates; citations verified; LaTeX compiles cleanly; independent reproduction instructions tested. | |
| 264 | + | |
| 265 | +A failed gate requires diagnosis, not rhetorical reinterpretation of failure as success. | |
| 266 | + | |
| 267 | +11. FIGURES AND TABLES — PUBLICATION STANDARD | |
| 268 | + | |
| 269 | +Produce vector PDF/SVG wherever feasible, legible at journal column widths, with accessible palettes, labeled units, uncertainty intervals, and no decorative pseudo-scientific graphics. | |
| 270 | + | |
| 271 | +Target figures (only when data support them): | |
| 272 | + | |
| 273 | +1. Conceptual donor → causal localization → alignment → recipient graft → evaluation diagram, clearly distinguishing hypotheses from observations. | |
| 274 | +2. Donor/recipient capability gap and held-out baseline distributions. | |
| 275 | +3. Layer × component causal localization heatmap with corrected significance/uncertainty. | |
| 276 | +4. Transplant mechanism architecture, token alignment, and dimensionality flow. | |
| 277 | +5. Target improvement versus control-task retention (Pareto frontier). | |
| 278 | +6. Donor-recipient portability matrix with confidence intervals. | |
| 279 | +7. Graft size versus effect size and latency/memory cost. | |
| 280 | +8. Ablation → loss → rescue causal chain. | |
| 281 | +9. Robustness across seeds, paraphrases, task difficulty, and unseen compositions. | |
| 282 | +10. Negative results and failure-mode taxonomy. | |
| 283 | + | |
| 284 | +Tables: model/checkpoint specifications; task splits; interventions and controls; primary effects with confidence intervals; collateral damage; compute cost; ablation study; reproducibility inventory. | |
| 285 | + | |
| 286 | +Never manufacture a figure to fill a visual quota. Every plotted point must derive from an identified data artifact. | |
| 287 | + | |
| 288 | +12. PAPER STRUCTURE AND WRITING STANDARD | |
| 289 | + | |
| 290 | +Target a strong machine-learning/mechanistic-interpretability contribution, not a marketing narrative. Assess fit for NeurIPS, ICLR, ICML, or an appropriate journal only after results and verified current submission rules; do not promise acceptance or assume formatting requirements. | |
| 291 | + | |
| 292 | +Suggested sections: | |
| 293 | + | |
| 294 | +1. Introduction: precise question, conceptual motivation, why transfer differs from conventional training and model merging, explicit contributions. | |
| 295 | +2. Related Work: critical positioning against model stitching, activation transfer, causal abstraction, representation alignment, editing, distillation, and mechanistic interpretability. | |
| 296 | +3. Operational Theory: candidate organs, transfer operators, causal estimands, and falsifiable hypotheses. | |
| 297 | +4. Experimental Design: controlled tasks, model pairs, interventions, controls, and identification. | |
| 298 | +5. Results: preregistered outcomes, uncertainty, generalization, costs, and negative findings. | |
| 299 | +6. Mechanistic Analysis: necessity, sufficiency, rescue, specificity, and limits. | |
| 300 | +7. Robustness and Failure Modes. | |
| 301 | +8. Discussion: what results establish, what they do not establish, and alternative explanations. | |
| 302 | +9. Limitations, Ethics, and Reproducibility. | |
| 303 | +10. Conclusion. | |
| 304 | +11. Appendices: full prompts, configs, model cards, supplementary tables, derivations, diagnostics, and compute details. | |
| 305 | + | |
| 306 | +The manuscript must have a substantial theoretical framework but not pretend that an intuitive metaphor is a theorem. Derive only justified mathematical statements and explicitly label conjectures. Use BibTeX records verified against primary sources. Do not invent author lists, publication metadata, DOIs, or conference versions. | |
| 307 | + | |
| 308 | +Paper title may be refined if findings warrant narrower claims. Do not use "transfer of intelligence" if evidence only shows task-specific activation bias. | |
| 309 | + | |
| 310 | +13. EXPERIMENT REGISTRY AND SCIENTIFIC AUDIT TRAIL | |
| 311 | + | |
| 312 | +For each run save: | |
| 313 | + | |
| 314 | +• research question, hypothesis, model pair, intervention family, control condition; | |
| 315 | +• code commit, environment lock, config and data hashes; | |
| 316 | +• training and alignment datasets and their split IDs; | |
| 317 | +• seed, intervention strength, layer, token positions, dimensionality; | |
| 318 | +• per-item outputs, ground truth, metrics, confidence intervals; | |
| 319 | +• resource usage, warnings, crashes, exclusions, and retry history; | |
| 320 | +• explicit status: planned / running / failed / completed / invalidated; | |
| 321 | +• a brief interpretation that separates observed evidence from speculation. | |
| 322 | + | |
| 323 | +Maintain docs/DECISION_LOG.md explaining methodological changes and whether they occurred before or after confirmatory lock. Never silently discard failed runs or change primary metrics after seeing outcomes. | |
| 324 | + | |
| 325 | +14. REPRODUCIBILITY, LICENSING, AND SECURITY | |
| 326 | + | |
| 327 | +Respect each model license and dataset license. Verify redistribution rights before releasing checkpoints or weights. Default to publishing code, configurations, synthetic generators, aggregate outputs, and scripts; release model-derived artifacts only when allowed. Do not exfiltrate proprietary model weights or secrets. Do not run arbitrary remote scripts without review. Avoid destructive changes to the wider UQO Research repository. | |
| 328 | + | |
| 329 | +Provide one-command setup where possible, a tiny CPU smoke test, a single-machine reference experiment, and optional multi-node execution. Record hardware-specific limitations, particularly MLX/Metal support. Include expected resource ranges measured from pilots, not guessed as facts. | |
| 330 | + | |
| 331 | +15. EXECUTION ROADMAP | |
| 332 | + | |
| 333 | +Sprint 0 — Research audit: inspect UQO Research, verify literature and licenses, write preregistration, resource budget, and gap statement. | |
| 334 | + | |
| 335 | +Sprint 1 — Controlled micro-world: build task generators and tiny-model baselines; validate donor competence and recipient weakness. | |
| 336 | + | |
| 337 | +Sprint 2 — Causal discovery: localize candidate subsystems, verify necessity and specificity, and benchmark null interventions. | |
| 338 | + | |
| 339 | +Sprint 3 — First transplant: implement one simple, falsifiable activation/subspace graft and a complete control battery. Do not prematurely scale. | |
| 340 | + | |
| 341 | +Sprint 4 — Generalization: extend to open-weight model families, mapping methods, and held-out compositional tasks. | |
| 342 | + | |
| 343 | +Sprint 5 — Mechanistic robustness: ablation/rescue, portability, persistence, collateral damage, and compute-efficiency frontiers. | |
| 344 | + | |
| 345 | +Sprint 6 — Confirmatory analysis: freeze protocol, run held-out tests, produce statistical tables and reproducible figures. | |
| 346 | + | |
| 347 | +Sprint 7 — Manuscript: write, audit, compile, and internally peer-review the paper. Maintain a claim-to-evidence traceability table. | |
| 348 | + | |
| 349 | +At each sprint: report completed artifacts, commands executed, actual results, unresolved risks, estimated remaining compute, and the next smallest scientifically meaningful experiment. | |
| 350 | + | |
| 351 | +16. AUTONOMOUS WORK RULES FOR CLAUDE CODE | |
| 352 | + | |
| 353 | +Work proactively within authorized resources. You may inspect code, design protocols, implement tests, and run bounded local experiments. Request explicit approval before substantial paid compute, destructive operations, publishing/submitting, modifying shared infrastructure, downloading exceptionally large checkpoints, or accessing restricted resources. | |
| 354 | + | |
| 355 | +Use independent critique passes for (a) causal inference, (b) representation alignment and leakage, (c) statistical validity, (d) code correctness, (e) literature novelty, and (f) paper claim-evidence consistency. If subagents are available, assign them scoped tasks and reconcile disagreements; do not treat agent agreement as empirical validation. | |
| 356 | + | |
| 357 | +Do not stop after writing a plan. Progress from the smallest working experiment through validated results, respecting approval and compute gates. Do not state that an experiment ran unless logs and artifacts prove it. Do not write fictional results sections. Use explicit placeholders for pending measurements. | |
| 358 | + | |
| 359 | +17. FIRST CONCRETE COMMAND TO EXECUTE | |
| 360 | + | |
| 361 | +Start by identifying the UQO Research root and current repository state. Create or safely reconcile wp-ai-0001. Perform the environment and literature audits. Write the preregistration and smallest pilot plan. Then implement a deterministic toy-model test that demonstrates the activation hook is a true no-op when disabled and that a controlled intervention produces a measurable, reproducible effect. Report those results before proceeding to transplantation claims. | |
| 362 | + | |
| 363 | +18. DEFINITION OF SUCCESS | |
| 364 | + | |
| 365 | +The strongest outcome is a causally validated, reproducible, selective, portable skill transfer that exceeds matched controls on genuinely unseen tasks without unacceptable collateral degradation. A rigorous demonstration that transplantation fails under well-defined architectural or representation mismatches is also a publishable scientific contribution if the boundary conditions are carefully established. | |
| 366 | + | |
| 367 | +Final principle: Build evidence strong enough that a skeptical independent researcher can reproduce the effect, rule out trivial explanations, and understand precisely what was—and was not—transferred. | |
added
docs/PREREGISTRATION.md
+67 −0
@@ -0,0 +1,67 @@ | ||
| 1 | +# Preregistration — WP-AI-0001 (v0.1, 2026-11-09) | |
| 2 | + | |
| 3 | +**Status:** draft registered *before* any transplant experiment. The Tier-0 pilot (`scripts/run_pilot.py`) | |
| 4 | +is explicitly *exploratory*: it serves to measure gap, verify hooks and localize the target subsystem. | |
| 5 | +Confirmatory analyses will be frozen in v1.0 of this document (hash recorded in `DECISION_LOG.md`) before | |
| 6 | +confirmatory data are generated. Changes after freeze are deviations and are logged. | |
| 7 | + | |
| 8 | +## 1. Estimands | |
| 9 | + | |
| 10 | +Notation as in CLAUDE.md §2. For a model pair (D, R), task family T with target stratum and control strata: | |
| 11 | + | |
| 12 | +- **Transfer gain** Δ_T = Q_T(R′) − Q_T(R), exact-match accuracy on held-out target items. | |
| 13 | +- **Control-adjusted transfer** Δ_T^adj = Δ_T − [Q_T(R_sham) − Q_T(R)] where R_sham is the *matched sham* | |
| 14 | + (same positions, layers, dimensionality and norm; donor content replaced by the control defined per method). | |
| 15 | +- **Retention** ρ_c = Q_c(R′) / Q_c(R) for each control stratum c with Q_c(R) ≥ 0.5; otherwise report Δ_c. | |
| 16 | +- **Donor necessity** N_D = Q_T(D) − Q_T(D_ablated) for the localized component. | |
| 17 | +- **Rescue** = Q_T(D_ablated + graft) − Q_T(D_ablated). | |
| 18 | + | |
| 19 | +Unit of inference: *items nested in programs (clusters)*; model seeds are a second level. CIs: cluster | |
| 20 | +bootstrap over programs within a seed; seed-level summaries report mean, sd and min over ≥ 3 seeds. | |
| 21 | + | |
| 22 | +## 2. Hypotheses (Tier 0, micro-world A, target operator Ω = `qua`) | |
| 23 | + | |
| 24 | +- **H1 (localization).** In donors trained on all operators, interchange interventions identify a set of | |
| 25 | + (layer, position) locations whose patching restores ≥ 0.8 of the clean answer rate at the Ω step, with | |
| 26 | + the sham (corrupt-into-corrupt) rate ≤ chance + 0.1. Null: no location exceeds sham + 0.2. | |
| 27 | +- **H2 (sufficiency / activation graft).** Mapped donor activations grafted at the identified location | |
| 28 | + raise the recipient's Ω-step accuracy by Δ_T^adj ≥ 0.25 (smallest effect of interest, SESOI) on | |
| 29 | + held-out target items. Null: Δ_T^adj ≤ 0.05. | |
| 30 | +- **H3 (necessity).** Ablating the identified donor component reduces donor Ω-step accuracy by | |
| 31 | + N_D ≥ 0.5 while control-stratum accuracy drops by ≤ 0.05. | |
| 32 | +- **H4 (selectivity).** Retention ρ_c ≥ 0.95 on `control_iid` for the graft that achieves H2. | |
| 33 | +- **H5 (rule vs cache).** The graft's effect on programs whose (Ω-neighbour, v0) combinations were absent | |
| 34 | + from the alignment data is at least 0.8 × its effect on seen combinations. Null: ≤ 0.5 ×. | |
| 35 | +- **H6 (portability, Tier 1).** Effects replicate (Δ_T^adj ≥ SESOI) for at least one cross-family pair; | |
| 36 | + we predict *failure* for mismatched-tokenizer pairs without a tokenizer bridge and record it as such. | |
| 37 | + | |
| 38 | +Directional predictions are stated for honesty; a null on any hypothesis is reported in full. | |
| 39 | + | |
| 40 | +## 3. Primary and secondary outcomes | |
| 41 | + | |
| 42 | +Primary (confirmatory): Δ_T^adj on `target_confirmatory` (generated after freeze from a new seed), | |
| 43 | +and ρ_c on `control_confirmatory`. Secondary: trace accuracy, Ω-step accuracy, N_D, rescue, graft | |
| 44 | +dimensionality, parameter count, latency, peak memory, seed stability. | |
| 45 | + | |
| 46 | +## 4. Multiplicity | |
| 47 | + | |
| 48 | +Localization sweeps (layers × positions × families × heads) are exploratory and selection is made on the | |
| 49 | +*development* localization set only. Confirmatory comparisons are limited to the pre-declared list in | |
| 50 | +`EXPERIMENT_PLAN.md` §5 (one primary contrast per method family; Holm correction across method families). | |
| 51 | + | |
| 52 | +## 5. Power | |
| 53 | + | |
| 54 | +Pilot variability will be recorded in `EXPERIMENT_LOG.md` after the pilot; the confirmatory sample size | |
| 55 | +(items and clusters) will be chosen by simulation from the pilot cluster bootstrap so that an effect of | |
| 56 | +SESOI = 0.25 has ≥ 0.9 power at α = 0.05 (two-sided) on the primary contrast. No effect sizes are invented. | |
| 57 | + | |
| 58 | +## 6. Exclusion rules | |
| 59 | + | |
| 60 | +Runs are excluded only for crashes, NaNs, or manifest/data-hash mismatches; all exclusions are listed in | |
| 61 | +the run manifest and in `EXPERIMENT_LOG.md`. No run is excluded for its outcome. | |
| 62 | + | |
| 63 | +## 7. Contamination safeguards | |
| 64 | + | |
| 65 | +Split hashes are stored in manifests; evaluation splits are generated with seeds disjoint from training | |
| 66 | +and alignment seeds; `assert_disjoint` is enforced at build time; alignment data for mappings never | |
| 67 | +include evaluation items (checked by item hash). | |
added
docs/REPRODUCIBILITY.md
+22 −0
@@ -0,0 +1,22 @@ | ||
| 1 | +# Reproducibility — WP-AI-0001 | |
| 2 | + | |
| 3 | +```bash | |
| 4 | +cd ia/wp-ai-0001 | |
| 5 | +python3 -m pip install -e ".[dev]" # or: uv pip install -e ".[dev]" | |
| 6 | +python3 -m pytest tests -q # unit + causal-sanity tests (CPU, < 30 s) | |
| 7 | +python3 scripts/audit_environment.py # writes docs/ENVIRONMENT_AUDIT.json (add --cluster to probe nodes) | |
| 8 | +python3 scripts/run_pilot.py --quick --device cpu --seeds 0 1 --n-boot 200 # CPU smoke test, ≈ 25 s | |
| 9 | +python3 scripts/run_pilot.py --seeds 0 1 2 --steps 3000 # Tier-0 pilot (MPS ≈ minutes) | |
| 10 | +``` | |
| 11 | + | |
| 12 | +- Every run writes `experiments/manifests/<run_id>.json` (git commit, config hash, data hashes, seeds, | |
| 13 | + libraries, hardware, status history), appends to `experiments/registry/runs.jsonl`, and stores results in | |
| 14 | + `experiments/summaries/<run_id>/` (`results.json`, `SUMMARY.md`). Per-item predictions are in | |
| 15 | + `artifacts/runs/<run_id>/` (git-ignored), models in `artifacts/models/<run_id>/`. | |
| 16 | +- Run IDs are deterministic functions of the experiment id and config hash; re-running with the same config | |
| 17 | + reuses trained models from `artifacts/models/<run_id>/` (delete to retrain). | |
| 18 | +- Determinism: PyTorch on MPS was verified bitwise-repeatable for this model (`noop.deterministic_repeat`); | |
| 19 | + localization sweeps repeated twice differ by 0.0. Training determinism across devices is *not* claimed | |
| 20 | + (CPU vs MPS kernels differ); seeds reproduce training on the same device/library versions. | |
| 21 | +- Data are synthetic and regenerated from seeds; the split hashes recorded in manifests must match. | |
| 22 | +- Secrets: none required. No tokens are read; paths are relative to the project root. | |
added
docs/RESEARCH_GAP.md
+84 −0
@@ -0,0 +1,84 @@ | ||
| 1 | +# Research gap — WP-AI-0001 (written 2026-11-09, before any transplant experiment) | |
| 2 | + | |
| 3 | +Literature status: **verified via arXiv API metadata and abstracts on 2026-11-09** for every record in | |
| 4 | +`literature/arxiv_verified_2026-11-09.txt` (identifiers, titles, first-submission dates, authors) and | |
| 5 | +`literature/references_verified.bib`. Full texts were not re-read for every item; claims below about | |
| 6 | +*what each paper does* come from the verified abstracts and the project's search notes. Anything not in | |
| 7 | +the verified list is marked (unverified) and must not be cited. | |
| 8 | + | |
| 9 | +## 1. What is already known | |
| 10 | + | |
| 11 | +**Representation similarity across models.** SVCCA (Raghu et al. 2017, 1706.05806), CKA (Kornblith et al. | |
| 12 | +2019, 1905.00414), the Platonic Representation Hypothesis (Huh et al. 2024, 2405.07987) and Multi-Way | |
| 13 | +Representation Alignment (Achara et al. 2026, 2602.06205) establish that independently trained networks | |
| 14 | +have *linearly related* representation spaces. Shared embedding geometry allows steering vectors to move | |
| 15 | +between models (Lee et al. 2025, 2503.21073). | |
| 16 | + | |
| 17 | +**Model stitching.** Bansal, Nakkiran & Barak (2021, 2106.07682) revisit stitching as a representation | |
| 18 | +comparison tool (vision). Chen, Merullo, Stolfo & Pavlick (2025, 2506.06609; NeurIPS 2025) show that affine | |
| 19 | +maps between residual streams of *same-family* language models of different sizes transfer SAEs, probes | |
| 20 | +and steering vectors. Kim, Mun & Han (2026, 2608.09521) learn a shared dense "activation bus" with one | |
| 21 | +encoder–decoder pair per model and show that an intermediate activation of one model can be consumed by | |
| 22 | +another model's frozen upper layers. | |
| 23 | + | |
| 24 | +**Cross-model feature dictionaries.** Lan et al. (2024, 2410.06981), Universal SAEs (Thasarathan et al. | |
| 25 | +2025, 2502.03714), SPARC (2025, 2507.06265) and SharedSAE (Ognev et al. 2026, 2609.04344) build concept | |
| 26 | +spaces shared across models, mostly for *interpretation*, not for conferring capabilities. | |
| 27 | + | |
| 28 | +**Cross-model steering and circuit transfer.** Agarwal (2026, 2608.05164) transfers SAE-derived steering | |
| 29 | +directions across five models (0.8B–8B) and reports functional transfer above ≈1.7B parameters for | |
| 30 | +*semantic concepts*. Cristofano (2026, 2601.16034) transfers *refusal* interventions across eight model | |
| 31 | +pairs by concept-basis reconstruction. Sinii et al. (2025, 2509.06608) find RL-induced reasoning steering | |
| 32 | +vectors transfer only within a family. | |
| 33 | + | |
| 34 | +**Capability transfer across scales.** Panigrahi et al. (2023, 2302.06600) graft ≈0.01 % of fine-tuned | |
| 35 | +parameters back onto a pre-trained model ("skill localization", same model). APM (Fan et al. 2026, | |
| 36 | +2608.13596) prunes a larger donor to a recipient's architecture and mixes it in with a micro coefficient; | |
| 37 | +SemAlign (Gu et al. 2025, 2510.24208) uses activations as the transfer medium but *trains* the recipient | |
| 38 | +layer by layer. Task arithmetic (Ilharco et al. 2022, 2212.04089), TIES (Yadav et al. 2023, 2306.01708) and | |
| 39 | +Git Re-Basin (Ainsworth et al. 2022, 2209.04836) merge *weights* of models with a common initialization. | |
| 40 | +Zero-shot tokenizer transfer (Minixhofer et al. 2024, 2405.07883) addresses the tokenizer boundary. | |
| 41 | + | |
| 42 | +**Causal methods.** Causal mediation (Vig et al. 2020, 2004.12265), causal tracing / ROME (Meng et al. 2022, | |
| 43 | +2202.05262), interchange interventions and causal abstraction (Geiger et al. 2023, 2301.04709 and DAS, | |
| 44 | +2303.02536), circuit discovery (Wang et al. 2022, 2211.00593; Conmy et al. 2023, 2304.14997), function / | |
| 45 | +task vectors (Todd et al. 2023, 2310.15213; Hendel et al. 2023, 2310.15916), activation steering (Turner | |
| 46 | +et al. 2023, 2308.10248; Li et al. 2023, 2306.03341; Zou et al. 2023, 2310.01405). | |
| 47 | + | |
| 48 | +**The closest negative result.** Zhang & Xin (2026, 2606.03280) learn a linear bridge from Pythia-160M to | |
| 49 | +Pythia-410M hidden states (cosine ≈ 0.97) and find that injected activations **do not** improve multi-hop | |
| 50 | +answering: additive injection is null, replacement is destructive. Their conclusion: *offline | |
| 51 | +representational alignment is not sufficient for useful causal communication inside the receiver.* | |
| 52 | + | |
| 53 | +## 2. What is missing (the gap this project targets) | |
| 54 | + | |
| 55 | +1. **No study starts from a causally identified subsystem.** Stitching, activation-bus and steering-transfer | |
| 56 | + work map *whole residual streams* or *concept directions*; APM prunes by activation salience. None first | |
| 57 | + establishes, by interchange interventions in the donor, that a bounded component is necessary and | |
| 58 | + sufficient for a *measurable capability*, and then asks whether *that* component transfers. | |
| 59 | +2. **No controlled donor/recipient capability gap.** Prior work takes two off-the-shelf models and measures | |
| 60 | + benchmark deltas. We *manufacture* the gap (recipient trained without the capability; Tier 0) or | |
| 61 | + verify it with repeated held-out measurement (Tier 1–2), so that "acquisition" is identifiable. | |
| 62 | +3. **No matched-control battery.** Sham mappings, norm-matched random grafts, unrelated-skill donors, | |
| 63 | + shuffled-token/layer grafts and matched-size adapter fine-tuning are not reported together in any of | |
| 64 | + the above; the negative result of 2606.03280 is itself uncontrolled with respect to these. | |
| 65 | +4. **Persistence is conflated.** Inference-time steering, saved adapters and weight edits are different | |
| 66 | + claims; we label single-token, within-sequence, cross-prompt and saved-checkpoint effects separately. | |
| 67 | +5. **Rule versus cache.** Whether a transferred component carries a *rule* (generalizing to unseen | |
| 68 | + compositions / symbols) rather than a bias toward answer tokens is not tested by existing transfer work. | |
| 69 | +6. **Failure boundaries.** Agarwal (2026) reports a scale discontinuity and Sinii et al. (2025) a family | |
| 70 | + boundary for *steering*; nothing comparable exists for *capability* transfer with causal controls. | |
| 71 | + | |
| 72 | +## 3. What would *not* be novel (and we will not claim) | |
| 73 | + | |
| 74 | +- Affine residual-stream mappings as such (Chen et al. 2025; Kim et al. 2026). | |
| 75 | +- Cross-model steering of concepts (Agarwal 2026; Lee et al. 2025). | |
| 76 | +- Grafting fine-tuned parameters within one model (Panigrahi et al. 2023). | |
| 77 | +- Any "organ" metaphor as a theoretical contribution; the contribution must be the causal design and the | |
| 78 | + measured transfer / non-transfer boundaries. | |
| 79 | + | |
| 80 | +## 4. Positioning sentence (to be refined after results) | |
| 81 | + | |
| 82 | +*We ask whether a component that is causally necessary and sufficient for a capability in a donor can be | |
| 83 | +installed in a recipient that demonstrably lacks it, under sham, random and matched-training controls, | |
| 84 | +and we map where this fails.* A rigorous negative result with boundary conditions is an acceptable outcome. | |
added
docs/RISK_REGISTER.md
+16 −0
@@ -0,0 +1,16 @@ | ||
| 1 | +# Risk register — WP-AI-0001 | |
| 2 | + | |
| 3 | +| ID | Risk | Likelihood | Impact | Mitigation / detection | | |
| 4 | +|---|---|---|---|---| | |
| 5 | +| R1 | Tier-0 task too easy (scratchpad makes Ω a lookup) → transfer of a table, not a rule | high | medium | `trace_stride > 1` variants; FSA family; H5 rule-vs-cache test on unseen (neighbour, v0) combinations | | |
| 6 | +| R2 | Tier-0 task too hard (single-token format failed to generalize in the first attempt, dev ≈ 0.3) | observed | high | scratchpad adopted (DECISION_LOG D2); keep single-token variant as a difficulty stratum only | | |
| 7 | +| R3 | Localization not reproducible across seeds | low (pilot smoke: r ≥ 0.97) | high | across-seed Pearson on heatmaps; selection on development set only | | |
| 8 | +| R4 | Graft effect is answer-token bias, not capability | medium | high | rule-vs-cache split, shuffled-token graft, logit-lens of grafted stream, control strata | | |
| 9 | +| R5 | Leakage through alignment data (mapping fitted on evaluation items) | medium | high | item-hash audit of alignment vs evaluation splits in every manifest | | |
| 10 | +| R6 | MLX does not expose hooks → silent substitution | medium | medium | layer-wrapping implementation with bitwise no-op test; equivalence test vs PyTorch on logits | | |
| 11 | +| R7 | Memory pressure on 3B activation capture | medium | medium | streamed capture; per-layer limits; profiling before sweeps | | |
| 12 | +| R8 | Multiple comparisons in sweeps inflate findings | high | high | exploratory/confirmatory separation; Holm across families; pre-declared contrasts | | |
| 13 | +| R9 | Literature overlap (2606.03280 negative; 2608.09521 activation bus) | certain | medium | position as controlled, causally identified extension; cite and compare explicitly | | |
| 14 | +| R10 | Cluster node unavailability | medium | low | laptop suffices for Tiers 0–1; nodes probed, never assumed | | |
| 15 | +| R11 | License restrictions on redistributing derived weights | medium | low | release code/configs/synthetic data; adapters only where licenses permit (Qwen: Apache-2.0; Llama 3.2: community license — check before release) | | |
| 16 | +| R12 | Over-claiming "intelligence transfer" | medium | high | title and claims limited to measured task-specific effects; claim-evidence table (Gate F) | | |
added
docs/TASKS.md
+37 −0
@@ -0,0 +1,37 @@ | ||
| 1 | +# Implementation task tracker — WP-AI-0001 | |
| 2 | + | |
| 3 | +Legend: [x] done · [~] in progress · [ ] todo · (gate) quality gate per CLAUDE.md §10 | |
| 4 | + | |
| 5 | +## Sprint 0 — Research audit | |
| 6 | +- [x] Locate UQO Research workspace, conventions, remotes | |
| 7 | +- [x] Environment audit (laptop + cluster) → `docs/ENVIRONMENT_AUDIT.json` | |
| 8 | +- [x] Literature verification → `literature/` | |
| 9 | +- [x] RESEARCH_GAP, PREREGISTRATION v0.1, EXPERIMENT_PLAN, COMPUTE_BUDGET, RISK_REGISTER, REPRODUCIBILITY, logs | |
| 10 | +- [x] Project skeleton, pyproject, tests (Gate A) | |
| 11 | +- [x] `scripts/build_figures.py`, `scripts/build_paper.sh`, paper skeleton compiles (placeholders) | |
| 12 | +- [x] git repo + `origin = gitsrv:srv/git/wpai0001_uqo.git` | |
| 13 | + | |
| 14 | +## Sprint 1 — Controlled micro-world | |
| 15 | +- [x] OPSEQ / FSA / MORPH generators + exact validators + disjointness + hashing | |
| 16 | +- [x] Toy transformer with hook points; no-op tests (Gate C part 1) | |
| 17 | +- [x] Training loop (teacher-forced scratchpad), greedy decoding, metrics, cluster bootstrap | |
| 18 | +- [x] Pilot run (3 seeds) — gap qualification (Gate B passed; run `pilot_tier0_opseq_451ef5087182`) | |
| 19 | +- [ ] FSA / MORPH toy baselines | |
| 20 | +- [ ] `trace_stride = 2` difficulty variant baseline | |
| 21 | + | |
| 22 | +## Sprint 2 — Causal discovery | |
| 23 | +- [x] Within-donor localization (patching, ablation, heads) with sham baseline and reproducibility (RQ1 pilot: Ω executed at the read position; L0 attention + L0–L2 MLPs) | |
| 24 | +- [ ] Add `embed`/`resid_pre` family and path patching Ω-token → r1 to the sweep | |
| 25 | +- [ ] Subspace localization (DAS-style / PCA of Ω-contrast) with norm-matched random subspaces | |
| 26 | +- [ ] Necessity test on donor (H3) with control-stratum retention | |
| 27 | + | |
| 28 | +## Sprint 3 — First transplant | |
| 29 | +- [ ] Alignment module: Procrustes, ridge, CCA, random, unrelated-data maps; leakage audit | |
| 30 | +- [ ] Activation graft + sham + norm-matched random + shuffled-token controls | |
| 31 | +- [ ] Gate D: reproducibility, overhead disclosure | |
| 32 | + | |
| 33 | +## Sprint 4–7 | |
| 34 | +- [ ] Tier 1 MLX interventions (layer wrapping) + Transformers equivalence test | |
| 35 | +- [ ] Portability matrix, persistence (saved adapter), collateral battery | |
| 36 | +- [ ] Confirmatory freeze (PREREGISTRATION v1.0) and run | |
| 37 | +- [ ] Paper, figures, claim-evidence table (Gate F) | |
added
experiments/manifests/pilot_tier0_opseq_451ef5087182.json
+223 −0
@@ -0,0 +1,223 @@ | ||
| 1 | +{ | |
| 2 | + "config": { | |
| 3 | + "description": "Tier-0 pilot: donor/recipient gap, hook no-op test, within-model causal localization", | |
| 4 | + "donor": { | |
| 5 | + "model": { | |
| 6 | + "d_ff": 512, | |
| 7 | + "d_model": 128, | |
| 8 | + "dropout": 0.0, | |
| 9 | + "max_seq_len": 24, | |
| 10 | + "n_heads": 4, | |
| 11 | + "n_layers": 4, | |
| 12 | + "tie_embeddings": false | |
| 13 | + }, | |
| 14 | + "name": "donor", | |
| 15 | + "role": "donor", | |
| 16 | + "task": { | |
| 17 | + "excluded_operators": [], | |
| 18 | + "family": "opseq", | |
| 19 | + "held_out_pairs": [ | |
| 20 | + [ | |
| 21 | + "zob", | |
| 22 | + "lar" | |
| 23 | + ], | |
| 24 | + [ | |
| 25 | + "qua", | |
| 26 | + "neg" | |
| 27 | + ] | |
| 28 | + ], | |
| 29 | + "max_len": 5, | |
| 30 | + "min_len": 1, | |
| 31 | + "modulus": 11, | |
| 32 | + "operators": [ | |
| 33 | + "zib", | |
| 34 | + "zob", | |
| 35 | + "qua", | |
| 36 | + "lar", | |
| 37 | + "neg", | |
| 38 | + "flip", | |
| 39 | + "dro", | |
| 40 | + "mik" | |
| 41 | + ], | |
| 42 | + "trace_stride": 1 | |
| 43 | + }, | |
| 44 | + "train": { | |
| 45 | + "batch_size": 256, | |
| 46 | + "device": "mps", | |
| 47 | + "eval_every": 500, | |
| 48 | + "lr": 0.001, | |
| 49 | + "n_dev": 1000, | |
| 50 | + "n_train": 40000, | |
| 51 | + "seed": 0, | |
| 52 | + "steps": 3000, | |
| 53 | + "warmup_steps": 100, | |
| 54 | + "weight_decay": 0.01 | |
| 55 | + } | |
| 56 | + }, | |
| 57 | + "experiment_id": "pilot_tier0_opseq", | |
| 58 | + "n_bootstrap": 2000, | |
| 59 | + "n_eval": 1000, | |
| 60 | + "recipient": { | |
| 61 | + "model": { | |
| 62 | + "d_ff": 512, | |
| 63 | + "d_model": 128, | |
| 64 | + "dropout": 0.0, | |
| 65 | + "max_seq_len": 24, | |
| 66 | + "n_heads": 4, | |
| 67 | + "n_layers": 4, | |
| 68 | + "tie_embeddings": false | |
| 69 | + }, | |
| 70 | + "name": "recipient", | |
| 71 | + "role": "recipient", | |
| 72 | + "task": { | |
| 73 | + "excluded_operators": [ | |
| 74 | + "qua" | |
| 75 | + ], | |
| 76 | + "family": "opseq", | |
| 77 | + "held_out_pairs": [ | |
| 78 | + [ | |
| 79 | + "zob", | |
| 80 | + "lar" | |
| 81 | + ], | |
| 82 | + [ | |
| 83 | + "qua", | |
| 84 | + "neg" | |
| 85 | + ] | |
| 86 | + ], | |
| 87 | + "max_len": 5, | |
| 88 | + "min_len": 1, | |
| 89 | + "modulus": 11, | |
| 90 | + "operators": [ | |
| 91 | + "zib", | |
| 92 | + "zob", | |
| 93 | + "qua", | |
| 94 | + "lar", | |
| 95 | + "neg", | |
| 96 | + "flip", | |
| 97 | + "dro", | |
| 98 | + "mik" | |
| 99 | + ], | |
| 100 | + "trace_stride": 1 | |
| 101 | + }, | |
| 102 | + "train": { | |
| 103 | + "batch_size": 256, | |
| 104 | + "device": "mps", | |
| 105 | + "eval_every": 500, | |
| 106 | + "lr": 0.001, | |
| 107 | + "n_dev": 1000, | |
| 108 | + "n_train": 40000, | |
| 109 | + "seed": 0, | |
| 110 | + "steps": 3000, | |
| 111 | + "warmup_steps": 100, | |
| 112 | + "weight_decay": 0.01 | |
| 113 | + } | |
| 114 | + }, | |
| 115 | + "seeds": [ | |
| 116 | + 0, | |
| 117 | + 1, | |
| 118 | + 2 | |
| 119 | + ], | |
| 120 | + "target_operator": "qua" | |
| 121 | + }, | |
| 122 | + "created_utc": "2026-11-09T06:49:35+00:00", | |
| 123 | + "data_hashes": { | |
| 124 | + "comp_heldout": { | |
| 125 | + "generator": "opseq@opseq-1.0", | |
| 126 | + "hash": "b1c6aebefbf3a5d5", | |
| 127 | + "n": 1000, | |
| 128 | + "seed": 20261121 | |
| 129 | + }, | |
| 130 | + "control_iid": { | |
| 131 | + "generator": "opseq@opseq-1.0", | |
| 132 | + "hash": "fa7c2327072b7d56", | |
| 133 | + "n": 1000, | |
| 134 | + "seed": 20261120 | |
| 135 | + }, | |
| 136 | + "donor_dev": { | |
| 137 | + "generator": "opseq@opseq-1.0", | |
| 138 | + "hash": "67055896a5ce52ea", | |
| 139 | + "n": 1000, | |
| 140 | + "seed": 20261112 | |
| 141 | + }, | |
| 142 | + "donor_train": { | |
| 143 | + "generator": "opseq@opseq-1.0", | |
| 144 | + "hash": "d213971f9e1feabf", | |
| 145 | + "n": 40000, | |
| 146 | + "seed": 20261110 | |
| 147 | + }, | |
| 148 | + "len_ood": { | |
| 149 | + "generator": "opseq@opseq-1.0", | |
| 150 | + "hash": "7ecb075ac45796ac", | |
| 151 | + "n": 500, | |
| 152 | + "seed": 20261122 | |
| 153 | + }, | |
| 154 | + "recipient_dev": { | |
| 155 | + "generator": "opseq@opseq-1.0", | |
| 156 | + "hash": "aa252ba85caa5d1f", | |
| 157 | + "n": 1000, | |
| 158 | + "seed": 20261113 | |
| 159 | + }, | |
| 160 | + "recipient_train": { | |
| 161 | + "generator": "opseq@opseq-1.0", | |
| 162 | + "hash": "c464e2683affd837", | |
| 163 | + "n": 40000, | |
| 164 | + "seed": 20261111 | |
| 165 | + }, | |
| 166 | + "target_iid": { | |
| 167 | + "generator": "opseq@opseq-1.0", | |
| 168 | + "hash": "f1a1ef205c8f35db", | |
| 169 | + "n": 1000, | |
| 170 | + "seed": 20261119 | |
| 171 | + } | |
| 172 | + }, | |
| 173 | + "experiment_id": "pilot_tier0_opseq", | |
| 174 | + "git": { | |
| 175 | + "branch": null, | |
| 176 | + "commit": null, | |
| 177 | + "dirty": null | |
| 178 | + }, | |
| 179 | + "hardware": { | |
| 180 | + "chip": "Apple M5 Max", | |
| 181 | + "cpu_count": 18, | |
| 182 | + "hostname_hash": "9ad7358c662e", | |
| 183 | + "machine": "arm64", | |
| 184 | + "memory_gb": 48.0, | |
| 185 | + "platform": "macOS-27.0-arm64-arm-64bit-Mach-O", | |
| 186 | + "torch_cuda": false, | |
| 187 | + "torch_mps": true | |
| 188 | + }, | |
| 189 | + "history": [ | |
| 190 | + { | |
| 191 | + "note": "", | |
| 192 | + "status": "running", | |
| 193 | + "utc": "2026-11-09T06:49:38+00:00" | |
| 194 | + }, | |
| 195 | + { | |
| 196 | + "note": "", | |
| 197 | + "status": "completed", | |
| 198 | + "utc": "2026-11-09T06:54:13+00:00" | |
| 199 | + } | |
| 200 | + ], | |
| 201 | + "interpretation": "", | |
| 202 | + "libraries": { | |
| 203 | + "mlx": "?", | |
| 204 | + "mlx_lm": "0.31.3", | |
| 205 | + "numpy": "2.4.4", | |
| 206 | + "pydantic": "2.13.4", | |
| 207 | + "python": "3.14.4", | |
| 208 | + "safetensors": "0.7.0", | |
| 209 | + "scipy": "1.17.1", | |
| 210 | + "torch": "2.12.0", | |
| 211 | + "transformers": "5.8.1" | |
| 212 | + }, | |
| 213 | + "outputs": { | |
| 214 | + "figures": "artifacts/figures/pilot_tier0_opseq_451ef5087182", | |
| 215 | + "items": "artifacts/runs/pilot_tier0_opseq_451ef5087182", | |
| 216 | + "models": "artifacts/models/pilot_tier0_opseq_451ef5087182", | |
| 217 | + "results": "experiments/summaries/pilot_tier0_opseq_451ef5087182/results.json" | |
| 218 | + }, | |
| 219 | + "run_id": "pilot_tier0_opseq_451ef5087182", | |
| 220 | + "seeds": {}, | |
| 221 | + "status": "completed", | |
| 222 | + "warnings": [] | |
| 223 | +} | |
| \ No newline at end of file | ||
added
experiments/manifests/pilot_tier0_opseq_quick_d5339bcad6b3.json
+222 −0
@@ -0,0 +1,222 @@ | ||
| 1 | +{ | |
| 2 | + "config": { | |
| 3 | + "description": "Tier-0 pilot: donor/recipient gap, hook no-op test, within-model causal localization", | |
| 4 | + "donor": { | |
| 5 | + "model": { | |
| 6 | + "d_ff": 128, | |
| 7 | + "d_model": 64, | |
| 8 | + "dropout": 0.0, | |
| 9 | + "max_seq_len": 24, | |
| 10 | + "n_heads": 2, | |
| 11 | + "n_layers": 3, | |
| 12 | + "tie_embeddings": false | |
| 13 | + }, | |
| 14 | + "name": "donor", | |
| 15 | + "role": "donor", | |
| 16 | + "task": { | |
| 17 | + "excluded_operators": [], | |
| 18 | + "family": "opseq", | |
| 19 | + "held_out_pairs": [ | |
| 20 | + [ | |
| 21 | + "zob", | |
| 22 | + "lar" | |
| 23 | + ], | |
| 24 | + [ | |
| 25 | + "qua", | |
| 26 | + "neg" | |
| 27 | + ] | |
| 28 | + ], | |
| 29 | + "max_len": 5, | |
| 30 | + "min_len": 1, | |
| 31 | + "modulus": 11, | |
| 32 | + "operators": [ | |
| 33 | + "zib", | |
| 34 | + "zob", | |
| 35 | + "qua", | |
| 36 | + "lar", | |
| 37 | + "neg", | |
| 38 | + "flip", | |
| 39 | + "dro", | |
| 40 | + "mik" | |
| 41 | + ], | |
| 42 | + "trace_stride": 1 | |
| 43 | + }, | |
| 44 | + "train": { | |
| 45 | + "batch_size": 256, | |
| 46 | + "device": "cpu", | |
| 47 | + "eval_every": 500, | |
| 48 | + "lr": 0.001, | |
| 49 | + "n_dev": 300, | |
| 50 | + "n_train": 3000, | |
| 51 | + "seed": 0, | |
| 52 | + "steps": 200, | |
| 53 | + "warmup_steps": 100, | |
| 54 | + "weight_decay": 0.01 | |
| 55 | + } | |
| 56 | + }, | |
| 57 | + "experiment_id": "pilot_tier0_opseq_quick", | |
| 58 | + "n_bootstrap": 200, | |
| 59 | + "n_eval": 300, | |
| 60 | + "recipient": { | |
| 61 | + "model": { | |
| 62 | + "d_ff": 128, | |
| 63 | + "d_model": 64, | |
| 64 | + "dropout": 0.0, | |
| 65 | + "max_seq_len": 24, | |
| 66 | + "n_heads": 2, | |
| 67 | + "n_layers": 3, | |
| 68 | + "tie_embeddings": false | |
| 69 | + }, | |
| 70 | + "name": "recipient", | |
| 71 | + "role": "recipient", | |
| 72 | + "task": { | |
| 73 | + "excluded_operators": [ | |
| 74 | + "qua" | |
| 75 | + ], | |
| 76 | + "family": "opseq", | |
| 77 | + "held_out_pairs": [ | |
| 78 | + [ | |
| 79 | + "zob", | |
| 80 | + "lar" | |
| 81 | + ], | |
| 82 | + [ | |
| 83 | + "qua", | |
| 84 | + "neg" | |
| 85 | + ] | |
| 86 | + ], | |
| 87 | + "max_len": 5, | |
| 88 | + "min_len": 1, | |
| 89 | + "modulus": 11, | |
| 90 | + "operators": [ | |
| 91 | + "zib", | |
| 92 | + "zob", | |
| 93 | + "qua", | |
| 94 | + "lar", | |
| 95 | + "neg", | |
| 96 | + "flip", | |
| 97 | + "dro", | |
| 98 | + "mik" | |
| 99 | + ], | |
| 100 | + "trace_stride": 1 | |
| 101 | + }, | |
| 102 | + "train": { | |
| 103 | + "batch_size": 256, | |
| 104 | + "device": "cpu", | |
| 105 | + "eval_every": 500, | |
| 106 | + "lr": 0.001, | |
| 107 | + "n_dev": 300, | |
| 108 | + "n_train": 3000, | |
| 109 | + "seed": 0, | |
| 110 | + "steps": 200, | |
| 111 | + "warmup_steps": 100, | |
| 112 | + "weight_decay": 0.01 | |
| 113 | + } | |
| 114 | + }, | |
| 115 | + "seeds": [ | |
| 116 | + 0, | |
| 117 | + 1 | |
| 118 | + ], | |
| 119 | + "target_operator": "qua" | |
| 120 | + }, | |
| 121 | + "created_utc": "2026-11-09T06:49:08+00:00", | |
| 122 | + "data_hashes": { | |
| 123 | + "comp_heldout": { | |
| 124 | + "generator": "opseq@opseq-1.0", | |
| 125 | + "hash": "68e83ab8472513d2", | |
| 126 | + "n": 300, | |
| 127 | + "seed": 20261121 | |
| 128 | + }, | |
| 129 | + "control_iid": { | |
| 130 | + "generator": "opseq@opseq-1.0", | |
| 131 | + "hash": "76bdfaf36a5f7519", | |
| 132 | + "n": 300, | |
| 133 | + "seed": 20261120 | |
| 134 | + }, | |
| 135 | + "donor_dev": { | |
| 136 | + "generator": "opseq@opseq-1.0", | |
| 137 | + "hash": "79b849f32dd6df25", | |
| 138 | + "n": 300, | |
| 139 | + "seed": 20261112 | |
| 140 | + }, | |
| 141 | + "donor_train": { | |
| 142 | + "generator": "opseq@opseq-1.0", | |
| 143 | + "hash": "da62d95aa07ed95a", | |
| 144 | + "n": 3000, | |
| 145 | + "seed": 20261110 | |
| 146 | + }, | |
| 147 | + "len_ood": { | |
| 148 | + "generator": "opseq@opseq-1.0", | |
| 149 | + "hash": "6d8c13dc8b5c47fa", | |
| 150 | + "n": 200, | |
| 151 | + "seed": 20261122 | |
| 152 | + }, | |
| 153 | + "recipient_dev": { | |
| 154 | + "generator": "opseq@opseq-1.0", | |
| 155 | + "hash": "c697914975bb93d3", | |
| 156 | + "n": 300, | |
| 157 | + "seed": 20261113 | |
| 158 | + }, | |
| 159 | + "recipient_train": { | |
| 160 | + "generator": "opseq@opseq-1.0", | |
| 161 | + "hash": "6d121a82cc83fe31", | |
| 162 | + "n": 3000, | |
| 163 | + "seed": 20261111 | |
| 164 | + }, | |
| 165 | + "target_iid": { | |
| 166 | + "generator": "opseq@opseq-1.0", | |
| 167 | + "hash": "88d9bb27db961222", | |
| 168 | + "n": 300, | |
| 169 | + "seed": 20261119 | |
| 170 | + } | |
| 171 | + }, | |
| 172 | + "experiment_id": "pilot_tier0_opseq_quick", | |
| 173 | + "git": { | |
| 174 | + "branch": null, | |
| 175 | + "commit": null, | |
| 176 | + "dirty": null | |
| 177 | + }, | |
| 178 | + "hardware": { | |
| 179 | + "chip": "Apple M5 Max", | |
| 180 | + "cpu_count": 18, | |
| 181 | + "hostname_hash": "9ad7358c662e", | |
| 182 | + "machine": "arm64", | |
| 183 | + "memory_gb": 48.0, | |
| 184 | + "platform": "macOS-27.0-arm64-arm-64bit-Mach-O", | |
| 185 | + "torch_cuda": false, | |
| 186 | + "torch_mps": true | |
| 187 | + }, | |
| 188 | + "history": [ | |
| 189 | + { | |
| 190 | + "note": "", | |
| 191 | + "status": "running", | |
| 192 | + "utc": "2026-11-09T06:49:09+00:00" | |
| 193 | + }, | |
| 194 | + { | |
| 195 | + "note": "", | |
| 196 | + "status": "completed", | |
| 197 | + "utc": "2026-11-09T06:49:31+00:00" | |
| 198 | + } | |
| 199 | + ], | |
| 200 | + "interpretation": "", | |
| 201 | + "libraries": { | |
| 202 | + "mlx": "?", | |
| 203 | + "mlx_lm": "0.31.3", | |
| 204 | + "numpy": "2.4.4", | |
| 205 | + "pydantic": "2.13.4", | |
| 206 | + "python": "3.14.4", | |
| 207 | + "safetensors": "0.7.0", | |
| 208 | + "scipy": "1.17.1", | |
| 209 | + "torch": "2.12.0", | |
| 210 | + "transformers": "5.8.1" | |
| 211 | + }, | |
| 212 | + "outputs": { | |
| 213 | + "figures": "artifacts/figures/pilot_tier0_opseq_quick_d5339bcad6b3", | |
| 214 | + "items": "artifacts/runs/pilot_tier0_opseq_quick_d5339bcad6b3", | |
| 215 | + "models": "artifacts/models/pilot_tier0_opseq_quick_d5339bcad6b3", | |
| 216 | + "results": "experiments/summaries/pilot_tier0_opseq_quick_d5339bcad6b3/results.json" | |
| 217 | + }, | |
| 218 | + "run_id": "pilot_tier0_opseq_quick_d5339bcad6b3", | |
| 219 | + "seeds": {}, | |
| 220 | + "status": "completed", | |
| 221 | + "warnings": [] | |
| 222 | +} | |
| \ No newline at end of file | ||
added
experiments/registry/runs.jsonl
+6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/summaries/pilot_tier0_opseq_451ef5087182/SUMMARY.md
+44 −0
Large diff (44 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/summaries/pilot_tier0_opseq_451ef5087182/results.json
+2303 −0
Large diff (2,304 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/summaries/pilot_tier0_opseq_quick_d5339bcad6b3/SUMMARY.md
+42 −0
Large diff (42 lines) is not loaded by default to keep the page fast.Load diff
added
experiments/summaries/pilot_tier0_opseq_quick_d5339bcad6b3/results.json
+1383 −0
Large diff (1,384 lines) is not loaded by default to keep the page fast.Load diff
added
literature/arxiv_verified_2026-11-09.txt
+45 −0
Large diff (46 lines) is not loaded by default to keep the page fast.Load diff
added
literature/references_unverified.md
+12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added
literature/references_verified.bib
+471 −0
Large diff (471 lines) is not loaded by default to keep the page fast.Load diff
added
paper/.latexmkrc
+5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added
paper/Makefile
+11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added
paper/appendix/appendix.tex
+6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added
paper/figures/FIGURE_PROVENANCE.md
+5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added
paper/figures/fig_gap_tier0.pdf
+0 −0
Binary file not shown.
added
paper/figures/fig_localization_ablation_tier0.pdf
+0 −0
Binary file not shown.
added
paper/figures/fig_localization_heads_tier0.pdf
+0 −0
Binary file not shown.
added
paper/figures/fig_localization_patching_tier0.pdf
+0 −0
Binary file not shown.
added
paper/main.pdf
+0 −0
Binary file not shown.
added
paper/main.tex
+147 −0
Large diff (147 lines) is not loaded by default to keep the page fast.Load diff
added
paper/references.bib
+471 −0
Large diff (471 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/conclusion.tex
+2 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/discussion.tex
+3 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/experimental_design.tex
+41 −0
Large diff (41 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/introduction.tex
+25 −0
Large diff (25 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/limitations.tex
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/mechanistic_analysis.tex
+2 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/operational_theory.tex
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/related_work.tex
+37 −0
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/results.tex
+61 −0
Large diff (61 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/robustness.tex
+3 −0
Large diff (3 lines) is not loaded by default to keep the page fast.Load diff
added
paper/sections/titlepage.tex
+77 −0
Large diff (77 lines) is not loaded by default to keep the page fast.Load diff
added
paper/submission/.gitkeep
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
paper/tables/.gitkeep
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
paper/uq_logo.jpg
+0 −0
Binary file not shown.
added
pyproject.toml
+29 −0
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/audit_environment.py
+91 −0
Large diff (91 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/build_figures.py
+52 −0
Large diff (52 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/build_paper.sh
+6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/run_pilot.py
+356 −0
Large diff (356 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/__init__.py
+6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/alignment/__init__.py
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/baselines/.gitkeep
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/baselines/__init__.py
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/config.py
+104 −0
Large diff (104 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/datasets/__init__.py
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/datasets/common.py
+166 −0
Large diff (166 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/datasets/fsa.py
+62 −0
Large diff (62 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/datasets/morph.py
+55 −0
Large diff (55 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/datasets/opseq.py
+131 −0
Large diff (131 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/evaluation/__init__.py
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/evaluation/metrics.py
+54 −0
Large diff (54 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/instrumentation/__init__.py
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/instrumentation/provenance.py
+123 −0
Large diff (123 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/localization/__init__.py
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/localization/patching.py
+120 −0
Large diff (120 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/models/__init__.py
+2 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/models/decoding.py
+28 −0
Large diff (28 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/models/hooks.py
+203 −0
Large diff (203 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/models/toy_transformer.py
+116 −0
Large diff (116 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/orchestration/__init__.py
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/orchestration/registry.py
+23 −0
Large diff (23 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/statistics/__init__.py
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/statistics/bootstrap.py
+54 −0
Large diff (54 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/training/__init__.py
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/training/train_toy.py
+102 −0
Large diff (102 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/transplantation/__init__.py
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/visualization/__init__.py
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp_ai_0001/visualization/style.py
+60 −0
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
added
tests/causal_sanity/test_hook_noop.py
+131 −0
Large diff (131 lines) is not loaded by default to keep the page fast.Load diff
added
tests/integration/.gitkeep
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
tests/regression/.gitkeep
+0 −0
Large diff (0 lines) is not loaded by default to keep the page fast.Load diff
added
tests/unit/test_bootstrap.py
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
tests/unit/test_generators.py
+77 −0
Large diff (78 lines) is not loaded by default to keep the page fast.Load diff