WP30: pilot complete, router fitted on dev, protocol frozen (404c52b34d96), core campaign launched
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
55 changed files +10,104 −750
modified
DECISIONS.md
+4 −0
@@ -17,3 +17,7 @@ All times America/Toronto unless noted. | ||
| 17 | 17 | - **D-011 Network path for E2/E6 (2026-10-30).** The laptop cannot open TCP connections to the nodes' LAN addresses (192.168.2.x; its SSH goes through Tailscale, whose ACL blocks non-SSH ports). Genuine two-node measurements therefore run **between two cluster nodes on the wired LAN**: sender/client on M3U96a (192.168.2.80), receiver worker on M3U96b (192.168.2.85). Measured link: iperf3 4 streams ≈ 940 Mbit/s (1 GbE, not 10 GbE as older notes suggested), RTT ≈ 1.2 ms. Both nodes concurrently host a CPU-light workload from another study (WP29 `claude --bare` executors); load average ≈ 1.8 on 32 cores at setup time; recorded as a limitation. |
| 18 | 18 | - **D-012 Learned mapper scope (Track C).** Level 2 (same family, different size): Qwen2.5-1.5B → Qwen2.5-3B share the tokenizer (token-aligned), both have 2 KV heads × 128 dims but 28 vs 36 layers → per-receiver-layer linear maps from an aligned sender layer, trained on train-split caches only (reconstruction loss; optional distillation). Level 3 (cross-family Llama-3.2-3B → Qwen2.5-3B) requires tokenizer alignment by character spans; attempted only if level 2 completes; labelled "adapted (SwaT-inspired), not a SwaT reproduction" (SwaT has no public checkpoint or licence). |
| 19 | 19 | - **D-013 Pilot scope reduced (2026-10-30).** After 120 trials the pilot ran at ≈5 trials/min (sender generations for question-aware A4/A5 dominate, GPU shared with mapper training), projecting > 12 h for 4 092 trials. Pilot reduced to 4 documents per family and 5 questions per document (1 984 trials) via `--resume` (trial keys unchanged). Pilot timings are not used for any claim; the core campaign runs alone on the GPU. |
| 20 | +- **D-014 Hosted sampling (2026-10-30).** The Anthropic Messages API (SDK 2.3.0, model claude-haiku-4-5) rejects `temperature` ("Extra inputs are not permitted"); hosted trials therefore use the provider's default sampling and are not greedy. OpenAI `gpt-5-nano` is called through the Responses API with `reasoning.effort=minimal` and no temperature. Hosted outputs are reproducible only up to provider nondeterminism; this is stated in the paper. | |
| 21 | +- **D-015 Core campaign sharding (2026-10-30).** To bound wall time, the core single-handoff campaign is sharded by model pair: Qwen2.5-3B pair on the laptop (M5 Max), Llama-3.2-3B pair on M3U96a (M3 Ultra). Trial keys are hardware-independent; records carry `hardware_id`. Latency comparisons are made within a pair (same hardware); cross-pair latency is not compared. Question-aware A4/A5 are limited to the first 3 questions of each synthetic document (`max_questions: 3`) because sender generation dominates run time. | |
| 22 | +- **D-016 Mapper variant for the core (2026-10-30, dev-only).** Two ridge mappers were fitted on the 60 train documents (248 569 tokens): v1 (aligned sender layer only, 9.46 M parameters, 289 s on M3U96b) and v2 (sender layers ℓs−1, ℓs, ℓs+1 as features, 28.3 M parameters, 483 s). Held-out relative reconstruction error (dev): K 0.120 → 0.098, V 0.396 → 0.322. In the dev transfer pilot both produced coherent but uninformative receiver outputs (mostly `unknown`; 0/20 correct for v1, 0/20 for v2). The core (E3_core) uses v2 as the registered learned mapper; v1 is kept as an artifact. No task or distillation loss was used (budget), which the paper states. | |
| 23 | +- **D-017 A5 prompt kept (2026-10-30, dev-only).** The pilot showed A5 (structured JSON) transmitting ≈ 8 facts / ≈ 800 bytes at the 512-token sender budget, because the exact quotes (provenance) consume most of the sender's output tokens while only spans are transmitted. An "exhaustive" prompt variant tested on 3 dev documents did not change fact coverage (8/7/7 facts, 726–836 bytes; 24-question accuracy 0.29 vs 0.21, within noise). The original prompt is kept. Consequence for the registered comparison T1: A4 and A5 are matched on sender budget and, incidentally, on transmitted bytes (≈ 0.8 kB each); the comparison is fair in bytes but A5 pays its provenance in sender tokens, which the paper discusses. | |
modified
configs/chains_core.yaml
+37 −9
@@ -1,15 +1,43 @@ | ||
| 1 | 1 | experiment_id: E4_core |
| 2 | −splits: [test] | |
| 3 | −lengths: [2000, 8000] | |
| 2 | +splits: | |
| 3 | +- test | |
| 4 | +lengths: | |
| 5 | +- 2000 | |
| 6 | +- 8000 | |
| 4 | 7 | limit_docs: 20 |
| 5 | −hops: [1, 2, 4, 8] | |
| 8 | +hops: | |
| 9 | +- 1 | |
| 10 | +- 2 | |
| 11 | +- 4 | |
| 12 | +- 8 | |
| 6 | 13 | budget_tokens: 512 |
| 7 | 14 | max_fetch_bytes: 1500 |
| 8 | 15 | max_lookups: 4 |
| 9 | 16 | chains: |
| 10 | − - {representation: A4, mode: closed, agents: [qwen3b]} | |
| 11 | − - {representation: A5, mode: closed, agents: [qwen3b]} | |
| 12 | − - {representation: A4, mode: repairable, agents: [qwen3b]} | |
| 13 | − - {representation: A5, mode: repairable, agents: [qwen3b]} | |
| 14 | − - {representation: A4, mode: closed, agents: [qwen3b, llama3b]} | |
| 15 | − - {representation: A5, mode: closed, agents: [qwen3b, llama3b]} | |
| 17 | +- representation: A4 | |
| 18 | + mode: closed | |
| 19 | + agents: | |
| 20 | + - qwen3b | |
| 21 | +- representation: A5 | |
| 22 | + mode: closed | |
| 23 | + agents: | |
| 24 | + - qwen3b | |
| 25 | +- representation: A4 | |
| 26 | + mode: repairable | |
| 27 | + agents: | |
| 28 | + - qwen3b | |
| 29 | +- representation: A5 | |
| 30 | + mode: repairable | |
| 31 | + agents: | |
| 32 | + - qwen3b | |
| 33 | +- representation: A4 | |
| 34 | + mode: closed | |
| 35 | + agents: | |
| 36 | + - qwen3b | |
| 37 | + - llama3b | |
| 38 | +- representation: A5 | |
| 39 | + mode: closed | |
| 40 | + agents: | |
| 41 | + - qwen3b | |
| 42 | + - llama3b | |
| 43 | +protocol_hash: 404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6 | |
modified
configs/core.yaml
+113 −32
@@ -1,35 +1,116 @@ | ||
| 1 | −# Confirmatory single-handoff campaign (E1/E2-process/E3-text). Locked test split. Fixed before any test outcome was examined. | |
| 2 | 1 | experiment_id: E1_core |
| 3 | −splits: [test] | |
| 4 | −families: [synthetic, exec, public_qasper, public_2wikimqa] | |
| 5 | −lengths: [2000, 8000, 16000] | |
| 6 | −limit_docs_per_family: 40 # synthetic/exec test = all 40; public: first 40 test docs in manifest (hash) order | |
| 7 | −max_questions_per_doc: 5 # synthetic: 5 of 15 question types per document (first five in generator order) | |
| 2 | +splits: | |
| 3 | +- test | |
| 4 | +families: | |
| 5 | +- synthetic | |
| 6 | +- exec | |
| 7 | +- public_qasper | |
| 8 | +- public_2wikimqa | |
| 9 | +lengths: | |
| 10 | +- 2000 | |
| 11 | +- 8000 | |
| 12 | +- 16000 | |
| 13 | +limit_docs_per_family: 40 | |
| 14 | +max_questions_per_doc: 5 | |
| 8 | 15 | model_pairs: |
| 9 | − - {sender: qwen3b, receiver: qwen3b} | |
| 10 | − - {sender: llama3b, receiver: llama3b} | |
| 11 | −generation: {max_tokens: 48, per_family: {exec: 320}} | |
| 12 | −seeds: [0] | |
| 16 | +- sender: qwen3b | |
| 17 | + receiver: qwen3b | |
| 18 | +- sender: llama3b | |
| 19 | + receiver: llama3b | |
| 20 | +generation: | |
| 21 | + max_tokens: 48 | |
| 22 | + per_family: | |
| 23 | + exec: 320 | |
| 24 | +seeds: | |
| 25 | +- 0 | |
| 13 | 26 | conditions: |
| 14 | − - {method: A0} | |
| 15 | − - {method: A1} | |
| 16 | − - {method: A2, budget_tokens: 512} | |
| 17 | − - {method: A3, budget_tokens: 512, access: blind} | |
| 18 | − - {method: A3, budget_tokens: 512, access: aware} | |
| 19 | − - {method: A4, budget_tokens: 512, access: blind} | |
| 20 | − - {method: A4, budget_tokens: 512, access: aware, families: [synthetic]} | |
| 21 | − - {method: A5, budget_tokens: 512, access: blind} | |
| 22 | − - {method: A5, budget_tokens: 512, access: aware, families: [synthetic]} | |
| 23 | − - {method: A6, budget_tokens: 512, access: blind, max_fetch_bytes: 2000, max_lookups: 6} | |
| 24 | − - {method: A7, codec: zstd} | |
| 25 | − - {method: A8, budget_tokens: 512, access: blind} | |
| 26 | − - {method: B0} | |
| 27 | − - {method: B1} | |
| 28 | − - {method: B2, bits: 8} | |
| 29 | − - {method: B2, bits: 4} | |
| 30 | − - {method: B3, keep_frac: 0.25} | |
| 31 | − - {method: B4, keep_frac: 0.25} | |
| 32 | − - {method: B5, keep_frac: 0.25, access: aware} | |
| 33 | − - {method: B5, keep_frac: 0.25, access: blind} | |
| 34 | − - {method: B6, keep_frac: 0.25, access: aware} | |
| 35 | − - {method: B9, keep_frac: 0.25, bits: 8, access: aware, policy: attention} | |
| 27 | +- method: A0 | |
| 28 | +- method: A1 | |
| 29 | +- method: A2 | |
| 30 | + budget_tokens: 512 | |
| 31 | +- method: A3 | |
| 32 | + budget_tokens: 512 | |
| 33 | + access: blind | |
| 34 | +- method: A3 | |
| 35 | + budget_tokens: 512 | |
| 36 | + access: aware | |
| 37 | +- method: A4 | |
| 38 | + budget_tokens: 512 | |
| 39 | + access: blind | |
| 40 | +- method: A4 | |
| 41 | + budget_tokens: 512 | |
| 42 | + access: aware | |
| 43 | + families: | |
| 44 | + - synthetic | |
| 45 | + max_questions: 3 | |
| 46 | +- method: A5 | |
| 47 | + budget_tokens: 512 | |
| 48 | + access: blind | |
| 49 | +- method: A5 | |
| 50 | + budget_tokens: 512 | |
| 51 | + access: aware | |
| 52 | + families: | |
| 53 | + - synthetic | |
| 54 | + max_questions: 3 | |
| 55 | +- method: A6 | |
| 56 | + budget_tokens: 512 | |
| 57 | + access: blind | |
| 58 | + max_fetch_bytes: 2000 | |
| 59 | + max_lookups: 6 | |
| 60 | +- method: A7 | |
| 61 | + codec: zstd | |
| 62 | +- method: A8 | |
| 63 | + budget_tokens: 512 | |
| 64 | + access: blind | |
| 65 | +- method: B0 | |
| 66 | +- method: B1 | |
| 67 | +- method: B2 | |
| 68 | + bits: 8 | |
| 69 | +- method: B2 | |
| 70 | + bits: 4 | |
| 71 | +- method: B3 | |
| 72 | + keep_frac: 0.25 | |
| 73 | +- method: B4 | |
| 74 | + keep_frac: 0.25 | |
| 75 | +- method: B5 | |
| 76 | + keep_frac: 0.25 | |
| 77 | + access: aware | |
| 78 | +- method: B5 | |
| 79 | + keep_frac: 0.25 | |
| 80 | + access: blind | |
| 81 | +- method: B6 | |
| 82 | + keep_frac: 0.25 | |
| 83 | + access: aware | |
| 84 | +- method: B9 | |
| 85 | + keep_frac: 0.25 | |
| 86 | + bits: 8 | |
| 87 | + access: aware | |
| 88 | + policy: attention | |
| 89 | +frozen_extra_files: | |
| 90 | +- configs/core.yaml | |
| 91 | +- configs/core_qwen.yaml | |
| 92 | +- configs/core_llama.yaml | |
| 93 | +- configs/core_transfer.yaml | |
| 94 | +- configs/core_sender_alone.yaml | |
| 95 | +- configs/chains_core.yaml | |
| 96 | +- configs/fanin_core.yaml | |
| 97 | +- configs/hosted_core.yaml | |
| 98 | +- configs/router.yaml | |
| 99 | +- runs/checkpoints/router_policy.json | |
| 100 | +- runs/checkpoints/mapper_v2_qwen1b5_to_qwen3b.npz.manifest.json | |
| 101 | +- src/wp30/mapper.py | |
| 102 | +- src/wp30/router.py | |
| 103 | +- src/wp30/router_eval.py | |
| 104 | +- src/wp30/hosted.py | |
| 105 | +- src/wp30/hosted_run.py | |
| 106 | +- scripts/e6_network.py | |
| 107 | +- src/wp30/transport.py | |
| 108 | +core_experiment_ids: | |
| 109 | +- E1_core | |
| 110 | +- E3_core | |
| 111 | +- E3_core_sender | |
| 112 | +- E4_core | |
| 113 | +- E5_core | |
| 114 | +- E6_core | |
| 115 | +- ED_hosted_core | |
| 116 | +- E8_router | |
added
configs/core_llama.yaml
+88 −0
@@ -0,0 +1,88 @@ | ||
| 1 | +experiment_id: E1_core | |
| 2 | +splits: | |
| 3 | +- test | |
| 4 | +families: | |
| 5 | +- synthetic | |
| 6 | +- exec | |
| 7 | +- public_qasper | |
| 8 | +- public_2wikimqa | |
| 9 | +lengths: | |
| 10 | +- 2000 | |
| 11 | +- 8000 | |
| 12 | +- 16000 | |
| 13 | +limit_docs_per_family: 40 | |
| 14 | +max_questions_per_doc: 5 | |
| 15 | +model_pairs: | |
| 16 | +- sender: llama3b | |
| 17 | + receiver: llama3b | |
| 18 | +generation: | |
| 19 | + max_tokens: 48 | |
| 20 | + per_family: | |
| 21 | + exec: 320 | |
| 22 | +seeds: | |
| 23 | +- 0 | |
| 24 | +conditions: | |
| 25 | +- method: A0 | |
| 26 | +- method: A1 | |
| 27 | +- method: A2 | |
| 28 | + budget_tokens: 512 | |
| 29 | +- method: A3 | |
| 30 | + budget_tokens: 512 | |
| 31 | + access: blind | |
| 32 | +- method: A3 | |
| 33 | + budget_tokens: 512 | |
| 34 | + access: aware | |
| 35 | +- method: A4 | |
| 36 | + budget_tokens: 512 | |
| 37 | + access: blind | |
| 38 | +- method: A4 | |
| 39 | + budget_tokens: 512 | |
| 40 | + access: aware | |
| 41 | + families: | |
| 42 | + - synthetic | |
| 43 | + max_questions: 3 | |
| 44 | +- method: A5 | |
| 45 | + budget_tokens: 512 | |
| 46 | + access: blind | |
| 47 | +- method: A5 | |
| 48 | + budget_tokens: 512 | |
| 49 | + access: aware | |
| 50 | + families: | |
| 51 | + - synthetic | |
| 52 | + max_questions: 3 | |
| 53 | +- method: A6 | |
| 54 | + budget_tokens: 512 | |
| 55 | + access: blind | |
| 56 | + max_fetch_bytes: 2000 | |
| 57 | + max_lookups: 6 | |
| 58 | +- method: A7 | |
| 59 | + codec: zstd | |
| 60 | +- method: A8 | |
| 61 | + budget_tokens: 512 | |
| 62 | + access: blind | |
| 63 | +- method: B0 | |
| 64 | +- method: B1 | |
| 65 | +- method: B2 | |
| 66 | + bits: 8 | |
| 67 | +- method: B2 | |
| 68 | + bits: 4 | |
| 69 | +- method: B3 | |
| 70 | + keep_frac: 0.25 | |
| 71 | +- method: B4 | |
| 72 | + keep_frac: 0.25 | |
| 73 | +- method: B5 | |
| 74 | + keep_frac: 0.25 | |
| 75 | + access: aware | |
| 76 | +- method: B5 | |
| 77 | + keep_frac: 0.25 | |
| 78 | + access: blind | |
| 79 | +- method: B6 | |
| 80 | + keep_frac: 0.25 | |
| 81 | + access: aware | |
| 82 | +- method: B9 | |
| 83 | + keep_frac: 0.25 | |
| 84 | + bits: 8 | |
| 85 | + access: aware | |
| 86 | + policy: attention | |
| 87 | +note: E1_core shard for pair llama3b->llama3b; records merged into runs/raw/E1_core.jsonl | |
| 88 | +protocol_hash: 404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6 | |
added
configs/core_qwen.yaml
+88 −0
@@ -0,0 +1,88 @@ | ||
| 1 | +experiment_id: E1_core | |
| 2 | +splits: | |
| 3 | +- test | |
| 4 | +families: | |
| 5 | +- synthetic | |
| 6 | +- exec | |
| 7 | +- public_qasper | |
| 8 | +- public_2wikimqa | |
| 9 | +lengths: | |
| 10 | +- 2000 | |
| 11 | +- 8000 | |
| 12 | +- 16000 | |
| 13 | +limit_docs_per_family: 40 | |
| 14 | +max_questions_per_doc: 5 | |
| 15 | +model_pairs: | |
| 16 | +- sender: qwen3b | |
| 17 | + receiver: qwen3b | |
| 18 | +generation: | |
| 19 | + max_tokens: 48 | |
| 20 | + per_family: | |
| 21 | + exec: 320 | |
| 22 | +seeds: | |
| 23 | +- 0 | |
| 24 | +conditions: | |
| 25 | +- method: A0 | |
| 26 | +- method: A1 | |
| 27 | +- method: A2 | |
| 28 | + budget_tokens: 512 | |
| 29 | +- method: A3 | |
| 30 | + budget_tokens: 512 | |
| 31 | + access: blind | |
| 32 | +- method: A3 | |
| 33 | + budget_tokens: 512 | |
| 34 | + access: aware | |
| 35 | +- method: A4 | |
| 36 | + budget_tokens: 512 | |
| 37 | + access: blind | |
| 38 | +- method: A4 | |
| 39 | + budget_tokens: 512 | |
| 40 | + access: aware | |
| 41 | + families: | |
| 42 | + - synthetic | |
| 43 | + max_questions: 3 | |
| 44 | +- method: A5 | |
| 45 | + budget_tokens: 512 | |
| 46 | + access: blind | |
| 47 | +- method: A5 | |
| 48 | + budget_tokens: 512 | |
| 49 | + access: aware | |
| 50 | + families: | |
| 51 | + - synthetic | |
| 52 | + max_questions: 3 | |
| 53 | +- method: A6 | |
| 54 | + budget_tokens: 512 | |
| 55 | + access: blind | |
| 56 | + max_fetch_bytes: 2000 | |
| 57 | + max_lookups: 6 | |
| 58 | +- method: A7 | |
| 59 | + codec: zstd | |
| 60 | +- method: A8 | |
| 61 | + budget_tokens: 512 | |
| 62 | + access: blind | |
| 63 | +- method: B0 | |
| 64 | +- method: B1 | |
| 65 | +- method: B2 | |
| 66 | + bits: 8 | |
| 67 | +- method: B2 | |
| 68 | + bits: 4 | |
| 69 | +- method: B3 | |
| 70 | + keep_frac: 0.25 | |
| 71 | +- method: B4 | |
| 72 | + keep_frac: 0.25 | |
| 73 | +- method: B5 | |
| 74 | + keep_frac: 0.25 | |
| 75 | + access: aware | |
| 76 | +- method: B5 | |
| 77 | + keep_frac: 0.25 | |
| 78 | + access: blind | |
| 79 | +- method: B6 | |
| 80 | + keep_frac: 0.25 | |
| 81 | + access: aware | |
| 82 | +- method: B9 | |
| 83 | + keep_frac: 0.25 | |
| 84 | + bits: 8 | |
| 85 | + access: aware | |
| 86 | + policy: attention | |
| 87 | +note: E1_core shard for pair qwen3b->qwen3b; records merged into runs/raw/E1_core.jsonl | |
| 88 | +protocol_hash: 404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6 | |
modified
configs/core_sender_alone.yaml
+19 −9
@@ -1,14 +1,24 @@ | ||
| 1 | −# Sender-alone reference for E3 (Qwen2.5-1.5B answering with full context) on the same items. | |
| 2 | 1 | experiment_id: E3_core_sender |
| 3 | −splits: [test] | |
| 4 | −families: [synthetic, exec] | |
| 5 | −lengths: [2000, 8000] | |
| 2 | +splits: | |
| 3 | +- test | |
| 4 | +families: | |
| 5 | +- synthetic | |
| 6 | +- exec | |
| 7 | +lengths: | |
| 8 | +- 2000 | |
| 9 | +- 8000 | |
| 6 | 10 | limit_docs_per_family: null |
| 7 | 11 | max_questions_per_doc: 8 |
| 8 | 12 | model_pairs: |
| 9 | − - {sender: qwen1b5, receiver: qwen1b5} | |
| 10 | −generation: {max_tokens: 48, per_family: {exec: 320}} | |
| 11 | −seeds: [0] | |
| 13 | +- sender: qwen1b5 | |
| 14 | + receiver: qwen1b5 | |
| 15 | +generation: | |
| 16 | + max_tokens: 48 | |
| 17 | + per_family: | |
| 18 | + exec: 320 | |
| 19 | +seeds: | |
| 20 | +- 0 | |
| 12 | 21 | conditions: |
| 13 | − - {method: A1} | |
| 14 | − - {method: B0} | |
| 22 | +- method: A1 | |
| 23 | +- method: B0 | |
| 24 | +protocol_hash: 404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6 | |
modified
configs/core_transfer.yaml
+31 −13
@@ -1,18 +1,36 @@ | ||
| 1 | −# Track C confirmatory: same-family size ladder Qwen2.5-1.5B -> Qwen2.5-3B (shared tokenizer). Test split, synthetic + exec. | |
| 2 | 1 | experiment_id: E3_core |
| 3 | −splits: [test] | |
| 4 | −families: [synthetic, exec] | |
| 5 | −lengths: [2000, 8000] | |
| 2 | +splits: | |
| 3 | +- test | |
| 4 | +families: | |
| 5 | +- synthetic | |
| 6 | +- exec | |
| 7 | +lengths: | |
| 8 | +- 2000 | |
| 9 | +- 8000 | |
| 6 | 10 | limit_docs_per_family: null |
| 7 | 11 | max_questions_per_doc: 8 |
| 8 | 12 | model_pairs: |
| 9 | − - {sender: qwen1b5, receiver: qwen3b} | |
| 10 | −generation: {max_tokens: 48, per_family: {exec: 320}} | |
| 11 | −seeds: [0] | |
| 13 | +- sender: qwen1b5 | |
| 14 | + receiver: qwen3b | |
| 15 | +generation: | |
| 16 | + max_tokens: 48 | |
| 17 | + per_family: | |
| 18 | + exec: 320 | |
| 19 | +seeds: | |
| 20 | +- 0 | |
| 12 | 21 | conditions: |
| 13 | − - {method: A1} # receiver alone, full context (reference) | |
| 14 | − - {method: A4, budget_tokens: 512, access: blind} # sender-generated summary -> receiver | |
| 15 | − - {method: A5, budget_tokens: 512, access: blind} | |
| 16 | − - {method: C0} # identity negative control | |
| 17 | − - {method: C2, mapper: runs/checkpoints/mapper_qwen1b5_to_qwen3b.npz} | |
| 18 | − - {method: C3, mapper: runs/checkpoints/mapper_qwen1b5_to_qwen3b.npz, keep_frac: 0.25, access: aware} | |
| 22 | +- method: A1 | |
| 23 | +- method: A4 | |
| 24 | + budget_tokens: 512 | |
| 25 | + access: blind | |
| 26 | +- method: A5 | |
| 27 | + budget_tokens: 512 | |
| 28 | + access: blind | |
| 29 | +- method: C0 | |
| 30 | +- method: C2 | |
| 31 | + mapper: runs/checkpoints/mapper_v2_qwen1b5_to_qwen3b.npz | |
| 32 | +- method: C3 | |
| 33 | + mapper: runs/checkpoints/mapper_v2_qwen1b5_to_qwen3b.npz | |
| 34 | + keep_frac: 0.25 | |
| 35 | + access: aware | |
| 36 | +protocol_hash: 404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6 | |
modified
configs/fanin_core.yaml
+15 −4
@@ -1,9 +1,20 @@ | ||
| 1 | 1 | experiment_id: E5_core |
| 2 | −splits: [test] | |
| 3 | −lengths: [2000, 8000] | |
| 2 | +splits: | |
| 3 | +- test | |
| 4 | +lengths: | |
| 5 | +- 2000 | |
| 6 | +- 8000 | |
| 4 | 7 | limit_docs: 20 |
| 5 | −fan_in: [2, 4] | |
| 8 | +fan_in: | |
| 9 | +- 2 | |
| 10 | +- 4 | |
| 6 | 11 | budget_tokens: 512 |
| 7 | 12 | sender: qwen3b |
| 8 | 13 | aggregator: qwen3b |
| 9 | −methods: [full_context, concat_A4, merge_A5, shared_memory_A6, cache_concat_negative_control] | |
| 14 | +methods: | |
| 15 | +- full_context | |
| 16 | +- concat_A4 | |
| 17 | +- merge_A5 | |
| 18 | +- shared_memory_A6 | |
| 19 | +- cache_concat_negative_control | |
| 20 | +protocol_hash: 404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6 | |
modified
configs/hosted_core.yaml
+28 −10
@@ -1,15 +1,33 @@ | ||
| 1 | 1 | experiment_id: ED_hosted_core |
| 2 | −splits: [test] | |
| 3 | −families: [synthetic, exec, public_qasper] | |
| 4 | −lengths: [2000, 8000] | |
| 2 | +splits: | |
| 3 | +- test | |
| 4 | +families: | |
| 5 | +- synthetic | |
| 6 | +- exec | |
| 7 | +- public_qasper | |
| 8 | +lengths: | |
| 9 | +- 2000 | |
| 10 | +- 8000 | |
| 5 | 11 | limit_docs_per_family: 8 |
| 6 | 12 | max_questions_per_doc: 5 |
| 7 | 13 | sender: qwen3b |
| 8 | −providers: [anthropic, openai] | |
| 14 | +providers: | |
| 15 | +- anthropic | |
| 16 | +- openai | |
| 9 | 17 | conditions: |
| 10 | − - {method: A0} | |
| 11 | − - {method: A1} | |
| 12 | − - {method: A1, provider_cache: true} | |
| 13 | − - {method: A4, budget_tokens: 512, access: blind} | |
| 14 | − - {method: A5, budget_tokens: 512, access: blind} | |
| 15 | − - {method: A6, budget_tokens: 512, access: blind, max_fetch_bytes: 2000, max_lookups: 6} | |
| 18 | +- method: A0 | |
| 19 | +- method: A1 | |
| 20 | +- method: A1 | |
| 21 | + provider_cache: true | |
| 22 | +- method: A4 | |
| 23 | + budget_tokens: 512 | |
| 24 | + access: blind | |
| 25 | +- method: A5 | |
| 26 | + budget_tokens: 512 | |
| 27 | + access: blind | |
| 28 | +- method: A6 | |
| 29 | + budget_tokens: 512 | |
| 30 | + access: blind | |
| 31 | + max_fetch_bytes: 2000 | |
| 32 | + max_lookups: 6 | |
| 33 | +protocol_hash: 404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6 | |
modified
configs/pilot.yaml
+1 −1
@@ -2,7 +2,7 @@ experiment_id: E1_pilot | ||
| 2 | 2 | splits: [dev] |
| 3 | 3 | families: [synthetic, exec, public_qasper, public_2wikimqa] |
| 4 | 4 | lengths: [2000, 8000] |
| 5 | −limit_docs_per_family: 4 | |
| 5 | +limit_docs_per_family: 2 | |
| 6 | 6 | max_questions_per_doc: 5 |
| 7 | 7 | model_pairs: |
| 8 | 8 | - {sender: qwen3b, receiver: qwen3b} |
added
configs/transfer_pilot_v2.yaml
+12 −0
@@ -0,0 +1,12 @@ | ||
| 1 | +experiment_id: E3_pilot_v2 | |
| 2 | +splits: [dev] | |
| 3 | +families: [synthetic] | |
| 4 | +lengths: [2000, 8000] | |
| 5 | +limit_docs_per_family: 4 | |
| 6 | +max_questions_per_doc: 5 | |
| 7 | +model_pairs: | |
| 8 | + - {sender: qwen1b5, receiver: qwen3b} | |
| 9 | +generation: {max_tokens: 48} | |
| 10 | +seeds: [0] | |
| 11 | +conditions: | |
| 12 | + - {method: C2, mapper: runs/checkpoints/mapper_v2_qwen1b5_to_qwen3b.npz} | |
modified
paper/sections/limitations.tex
+46 −2
@@ -1,3 +1,47 @@ | ||
| 1 | −\section{Limitations} | |
| 1 | +\section{Failure analysis and limitations} | |
| 2 | 2 | \label{sec:limitations} |
| 3 | −To be written from results. | |
| 3 | + | |
| 4 | +\paragraph{Contextual, not parametric.} Every effect reported here disappears when the message or cache is removed (A0). No receiver | |
| 5 | +acquired knowledge; frozen models were given better or worse \emph{access} to information. Claims about ``learning'' are out of scope. | |
| 6 | + | |
| 7 | +\paragraph{Local white-box versus hosted black-box.} Cache experiments require a runtime that exposes attention tensors (MLX here). | |
| 8 | +Hosted APIs expose a billing abstraction (cached input tokens) whose precision, layout and residency are undisclosed; Track~D therefore | |
| 9 | +contains text methods only and its cached-token fields are never translated into bytes. | |
| 10 | + | |
| 11 | +\paragraph{Same checkpoint versus cross-family.} All cache reuse and transfer results concern identical checkpoints and revisions | |
| 12 | +(Track~B, two-node B1) or a same-family size ladder with a linear mapper (Track~C). Cross-family cache transfer was not executed: the | |
| 13 | +only published method with code (SwaT) released no checkpoint and no licence, and training a mapper of its size exceeded our budget. | |
| 14 | +The paper's title names knowledge transfer in general; its \emph{cache}-transfer evidence is same-family only. | |
| 15 | + | |
| 16 | +\paragraph{Query-aware versus reusable compression.} Attention-based selection is query-dependent; its question-blind variant (generic | |
| 17 | +window) is a different method with much lower quality (Section~\ref{sec:results}). Reusable (blind) messages and query-specific | |
| 18 | +(aware) messages are reported in separate panels and should not be ranked against each other. | |
| 19 | + | |
| 20 | +\paragraph{Translated-cache distribution shift.} The mapper was fitted on native sender caches; chains of mapped caches (feeding a mapped | |
| 21 | +cache into a further mapping) were not run, so no closure property is claimed. | |
| 22 | + | |
| 23 | +\paragraph{Budgets are measured but not universal.} Byte counts include manifests and quantization metadata; latencies include | |
| 24 | +serialization, loading and fresh-query processing; memory peaks are MLX allocator peaks. They are specific to \texttt{mlx-lm}~0.32.2 | |
| 25 | +kernels on one M5~Max and two M3~Ultra machines; a fused eviction kernel or a different attention backend would change the compression | |
| 26 | +and TTFT components. We compared methods on a common backend; we did not compare against vendor-optimized kernels. | |
| 27 | + | |
| 28 | +\paragraph{Network and reuse.} The two-node link is a 1~GbE LAN with 1.2~ms RTT; break-even positions scale with bandwidth and with | |
| 29 | +the receiver's prefill speed, both of which we report so that the analytic curves can be re-drawn. Queue delays were not imposed; the | |
| 30 | +cluster nodes carried a light concurrent CPU workload from another study. | |
| 31 | + | |
| 32 | +\paragraph{Contamination and synthetic facts.} The public QA texts may appear in pre-training corpora; the communication-dependence | |
| 33 | +conclusions rest on the synthetic private-fact family, whose facts are new by construction but whose templates are narrower than real | |
| 34 | +documents. The generated ledger is not a filing; the executable tasks are small. | |
| 35 | + | |
| 36 | +\paragraph{Scoring.} All scoring is deterministic (exact/numeric/yes-no/abstention/F1/hidden tests); no language-model judge and no human | |
| 37 | +annotation were used. The fact-ledger retention metric is string containment of normalized values, which under-counts paraphrased | |
| 38 | +retention and over-counts accidental matches of short values. | |
| 39 | + | |
| 40 | +\paragraph{Coverage.} Two 3B model families, three task families, one sender budget (512 tokens) in the core, one selection ratio | |
| 41 | +(1/4) for the registered comparisons. Audited but unexecuted methods are listed with reasons in Table~\ref{tab:taxonomy} and | |
| 42 | +Appendix~\ref{app:audit}; their absence is a coverage gap, not evidence about them. | |
| 43 | + | |
| 44 | +\paragraph{Auditability and security of latent messages.} A cache is not human-readable; our manifests bind it to a checkpoint | |
| 45 | +revision, tokenizer, chat template, prefix-token hash and document id, and the loader rejects mismatches, which protects against the | |
| 46 | +wrong-document handle. Structured evidence with spans is auditable but depends on the integrity of the source it points to; retrieved | |
| 47 | +passages are treated as data, never as instructions, and the public benchmark text never enters the operator's command channel. | |
modified
paper/sections/reproducibility.tex
+34 −3
@@ -1,3 +1,34 @@ | ||
| 1 | −\section{Reproducibility} | |
| 2 | −\label{sec:reproducibility} | |
| 3 | −To be written from results. | |
| 1 | +\section{Reproducibility and execution disclosure} | |
| 2 | +\label{sec:repro} | |
| 3 | + | |
| 4 | +\paragraph{How the study was executed.} All code, experiments, analyses and this manuscript were produced by Claude Code (Anthropic; | |
| 5 | +model \texttt{claude-fable-5-1}) acting as research operator under a written brief from the author, through actual terminal | |
| 6 | +commands on the author's machines: environment inspection, dependency pinning, implementation, launching inference, reading exit | |
| 7 | +codes and logs, diagnosing failures, re-running affected trials, analysis and compilation. The author provided the brief, the | |
| 8 | +hardware, API credentials already present in sibling projects, and no mid-run intervention; dated engineering decisions are in | |
| 9 | +\texttt{DECISIONS.md}. Models under test are the experimental subjects; the operator's own inference state was never an | |
| 10 | +experimental interface. This disclosure describes the process; it is not a claim that autonomous research is superior to any | |
| 11 | +alternative. | |
| 12 | + | |
| 13 | +\paragraph{Defects found and handled.} (i) The \texttt{mlx-lm} cache API changed between the system-installed version (0.31.3) and the | |
| 14 | +pinned environment (0.32.2); adapters were rewritten for the pinned API before any experiment. (ii) The receiver prefix boundary was | |
| 15 | +initially computed from the prefix text alone and did not match the tokenization of the full prompt; it is now the common token | |
| 16 | +prefix with a canonical question, and a mismatch raises. (iii) Heterogeneous retained lengths (B6) broke the shared attention mask; | |
| 17 | +the fresh query is processed token by token for such caches. (iv) The first executable-task specification (seven rules) was unsolvable | |
| 18 | +for 3B receivers even with the full ticket; it was simplified on development data before any test item was run (D-010). (v) Running | |
| 19 | +three GPU jobs concurrently exhausted unified memory and slowed the pilot by an order of magnitude; GPU jobs were serialized. | |
| 20 | + | |
| 21 | +\paragraph{Artifacts.} The repository contains the harness, pinned lock file, data manifests and generators, the frozen protocol and its | |
| 22 | +hash, every per-attempt record (\texttt{runs/raw}), sanitized logs, the mapper checkpoint and manifest, generated tables and figures | |
| 23 | +with a provenance index linking each to its source records and script, a claim ledger, and the minimal reproduction profile. | |
| 24 | +Checkpoints are not redistributed; their Hugging Face identifiers, revisions and weight hashes are recorded. Hosted-API trials require | |
| 25 | +the provider credentials and reproduce only up to provider nondeterminism. | |
| 26 | + | |
| 27 | +\paragraph{Reproduction commands.} | |
| 28 | +\begin{footnotesize}\begin{verbatim} | |
| 29 | +make reproduce-analysis # tables + figures from runs/raw, provenance audit | |
| 30 | +python -m wp30 reproduce --profile reproduction/minimal.yaml # re-run a fixed subset, compare greedy outputs | |
| 31 | +make paper # compile this document | |
| 32 | +\end{verbatim}\end{footnotesize} | |
| 33 | +Greedy decoding on MLX is deterministic for an identical prefill chunking but not bitwise across chunkings (Section~\ref{sec:correctness}); | |
| 34 | +the reproduction report lists identical output hashes and identical scores separately. | |
modified
paper/tables/protocol_hash.tex
+1 −1
@@ -1 +1 @@ | ||
| 1 | −unfrozen | |
| \ No newline at end of file | ||
| 1 | +404c52b34d964693 | |
added
protocol/core_matrix.csv
+6561 −0
Large diff (6,561 lines) is not loaded by default to keep the page fast.Load diff
modified
protocol/core_matrix_draft.csv
+160 −160
Large diff (880 lines) is not loaded by default to keep the page fast.Load diff
added
protocol/frozen.json
+56 −0
Large diff (57 lines) is not loaded by default to keep the page fast.Load diff
modified
protocol/hypotheses.md
+8 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
modified
protocol/pilot_matrix.csv
+0 −496
Large diff (517 lines) is not loaded by default to keep the page fast.Load diff
added
reproduction/minimal.yaml
+4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added
reproduction/minimal_config.yaml
+19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added
results/figures/PALETTE.txt
+1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added
results/figures/fig2_quality_vs_bytes.pdf
+0 −0
Binary file not shown.
added
results/figures/fig2_quality_vs_bytes.png
+0 −0
Binary file not shown.
added
results/figures/fig3_quality_vs_latency_memory.pdf
+0 −0
Binary file not shown.
added
results/figures/fig3_quality_vs_latency_memory.png
+0 −0
Binary file not shown.
added
results/figures/fig4_latency_decomposition.pdf
+0 −0
Binary file not shown.
added
results/figures/fig4_latency_decomposition.png
+0 −0
Binary file not shown.
added
results/figures/figA_quality_vs_bytes_aware.pdf
+0 −0
Binary file not shown.
added
results/figures/figA_quality_vs_bytes_aware.png
+0 −0
Binary file not shown.
added
results/figures/figA_quality_vs_bytes_blind.pdf
+0 −0
Binary file not shown.
added
results/figures/figA_quality_vs_bytes_blind.png
+0 −0
Binary file not shown.
added
results/provenance.json
+122 −0
Large diff (123 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/E1_pilot_by_question_type.csv
+109 −0
Large diff (109 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/E1_pilot_failures.csv
+18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/E1_pilot_primary.csv
+63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/E1_pilot_primary_tests.csv
+6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added
results/tables/ED_hosted.csv
+9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added
runs/checkpoints/mapper_v2_qwen1b5_to_qwen3b.npz
+0 −0
Binary file not shown.
added
runs/checkpoints/mapper_v2_qwen1b5_to_qwen3b.npz.manifest.json
+202 −0
Large diff (203 lines) is not loaded by default to keep the page fast.Load diff
added
runs/checkpoints/router_policy.json
+1148 −0
Large diff (1,149 lines) is not loaded by default to keep the page fast.Load diff
modified
runs/raw/E1_pilot.jsonl
+496 −0
Large diff (499 lines) is not loaded by default to keep the page fast.Load diff
added
runs/raw/E3_pilot.jsonl
+107 −0
Large diff (107 lines) is not loaded by default to keep the page fast.Load diff
added
runs/raw/E3_pilot_v2.jsonl
+20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added
runs/raw/E4_pilot.jsonl
+330 −0
Large diff (330 lines) is not loaded by default to keep the page fast.Load diff
added
runs/raw/ED_hosted_pilot_m3u96b.jsonl
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
scripts/run_e6_core.sh
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
modified
src/wp30/analysis/analyze.py
+4 −2
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp30/analysis/chains_recompute.py
+31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp30/analysis/claims.py
+12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
modified
src/wp30/analysis/load.py
+6 −2
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
added
src/wp30/analysis/numbers.py
+39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
modified
src/wp30/chains.py
+2 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
modified
src/wp30/hosted.py
+1 −1
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
modified
src/wp30/runner.py
+11 −5
Large diff (52 lines) is not loaded by default to keep the page fast.Load diff