SPB Git
spboucher.ai

spb/wp30_uqo

Public

WP30: pilot complete, router fitted on dev, protocol frozen (404c52b34d96), core campaign launched

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Oct 30, 2026) parent 5c60942 Browse files

55 changed files +10,104 −750

55 changed files
Load all 55 files Unified Split
modified DECISIONS.md +4 −0
@@ -17,3 +17,7 @@ All times America/Toronto unless noted.
17 - **D-011 Network path for E2/E6 (2026-10-30).** The laptop cannot open TCP connections to the nodes' LAN addresses (192.168.2.x; its SSH goes through Tailscale, whose ACL blocks non-SSH ports). Genuine two-node measurements therefore run **between two cluster nodes on the wired LAN**: sender/client on M3U96a (192.168.2.80), receiver worker on M3U96b (192.168.2.85). Measured link: iperf3 4 streams ≈ 940 Mbit/s (1 GbE, not 10 GbE as older notes suggested), RTT ≈ 1.2 ms. Both nodes concurrently host a CPU-light workload from another study (WP29 `claude --bare` executors); load average ≈ 1.8 on 32 cores at setup time; recorded as a limitation. 17 - **D-011 Network path for E2/E6 (2026-10-30).** The laptop cannot open TCP connections to the nodes' LAN addresses (192.168.2.x; its SSH goes through Tailscale, whose ACL blocks non-SSH ports). Genuine two-node measurements therefore run **between two cluster nodes on the wired LAN**: sender/client on M3U96a (192.168.2.80), receiver worker on M3U96b (192.168.2.85). Measured link: iperf3 4 streams ≈ 940 Mbit/s (1 GbE, not 10 GbE as older notes suggested), RTT ≈ 1.2 ms. Both nodes concurrently host a CPU-light workload from another study (WP29 `claude --bare` executors); load average ≈ 1.8 on 32 cores at setup time; recorded as a limitation.
18 - **D-012 Learned mapper scope (Track C).** Level 2 (same family, different size): Qwen2.5-1.5B → Qwen2.5-3B share the tokenizer (token-aligned), both have 2 KV heads × 128 dims but 28 vs 36 layers → per-receiver-layer linear maps from an aligned sender layer, trained on train-split caches only (reconstruction loss; optional distillation). Level 3 (cross-family Llama-3.2-3B → Qwen2.5-3B) requires tokenizer alignment by character spans; attempted only if level 2 completes; labelled "adapted (SwaT-inspired), not a SwaT reproduction" (SwaT has no public checkpoint or licence). 18 - **D-012 Learned mapper scope (Track C).** Level 2 (same family, different size): Qwen2.5-1.5B → Qwen2.5-3B share the tokenizer (token-aligned), both have 2 KV heads × 128 dims but 28 vs 36 layers → per-receiver-layer linear maps from an aligned sender layer, trained on train-split caches only (reconstruction loss; optional distillation). Level 3 (cross-family Llama-3.2-3B → Qwen2.5-3B) requires tokenizer alignment by character spans; attempted only if level 2 completes; labelled "adapted (SwaT-inspired), not a SwaT reproduction" (SwaT has no public checkpoint or licence).
19 - **D-013 Pilot scope reduced (2026-10-30).** After 120 trials the pilot ran at ≈5 trials/min (sender generations for question-aware A4/A5 dominate, GPU shared with mapper training), projecting > 12 h for 4 092 trials. Pilot reduced to 4 documents per family and 5 questions per document (1 984 trials) via `--resume` (trial keys unchanged). Pilot timings are not used for any claim; the core campaign runs alone on the GPU. 19 - **D-013 Pilot scope reduced (2026-10-30).** After 120 trials the pilot ran at ≈5 trials/min (sender generations for question-aware A4/A5 dominate, GPU shared with mapper training), projecting > 12 h for 4 092 trials. Pilot reduced to 4 documents per family and 5 questions per document (1 984 trials) via `--resume` (trial keys unchanged). Pilot timings are not used for any claim; the core campaign runs alone on the GPU.
20+- **D-014 Hosted sampling (2026-10-30).** The Anthropic Messages API (SDK 2.3.0, model claude-haiku-4-5) rejects `temperature` ("Extra inputs are not permitted"); hosted trials therefore use the provider's default sampling and are not greedy. OpenAI `gpt-5-nano` is called through the Responses API with `reasoning.effort=minimal` and no temperature. Hosted outputs are reproducible only up to provider nondeterminism; this is stated in the paper.
21+- **D-015 Core campaign sharding (2026-10-30).** To bound wall time, the core single-handoff campaign is sharded by model pair: Qwen2.5-3B pair on the laptop (M5 Max), Llama-3.2-3B pair on M3U96a (M3 Ultra). Trial keys are hardware-independent; records carry `hardware_id`. Latency comparisons are made within a pair (same hardware); cross-pair latency is not compared. Question-aware A4/A5 are limited to the first 3 questions of each synthetic document (`max_questions: 3`) because sender generation dominates run time.
22+- **D-016 Mapper variant for the core (2026-10-30, dev-only).** Two ridge mappers were fitted on the 60 train documents (248 569 tokens): v1 (aligned sender layer only, 9.46 M parameters, 289 s on M3U96b) and v2 (sender layers ℓs−1, ℓs, ℓs+1 as features, 28.3 M parameters, 483 s). Held-out relative reconstruction error (dev): K 0.120 → 0.098, V 0.396 → 0.322. In the dev transfer pilot both produced coherent but uninformative receiver outputs (mostly `unknown`; 0/20 correct for v1, 0/20 for v2). The core (E3_core) uses v2 as the registered learned mapper; v1 is kept as an artifact. No task or distillation loss was used (budget), which the paper states.
23+- **D-017 A5 prompt kept (2026-10-30, dev-only).** The pilot showed A5 (structured JSON) transmitting ≈ 8 facts / ≈ 800 bytes at the 512-token sender budget, because the exact quotes (provenance) consume most of the sender's output tokens while only spans are transmitted. An "exhaustive" prompt variant tested on 3 dev documents did not change fact coverage (8/7/7 facts, 726–836 bytes; 24-question accuracy 0.29 vs 0.21, within noise). The original prompt is kept. Consequence for the registered comparison T1: A4 and A5 are matched on sender budget and, incidentally, on transmitted bytes (≈ 0.8 kB each); the comparison is fair in bytes but A5 pays its provenance in sender tokens, which the paper discusses.
modified configs/chains_core.yaml +37 −9
@@ -1,15 +1,43 @@
1 experiment_id: E4_core 1 experiment_id: E4_core
2−splits: [test] 2+splits:
3−lengths: [2000, 8000] 3+- test
4+lengths:
5+- 2000
6+- 8000
4 limit_docs: 20 7 limit_docs: 20
5−hops: [1, 2, 4, 8] 8+hops:
9+- 1
10+- 2
11+- 4
12+- 8
6 budget_tokens: 512 13 budget_tokens: 512
7 max_fetch_bytes: 1500 14 max_fetch_bytes: 1500
8 max_lookups: 4 15 max_lookups: 4
9 chains: 16 chains:
10− - {representation: A4, mode: closed, agents: [qwen3b]} 17+- representation: A4
11− - {representation: A5, mode: closed, agents: [qwen3b]} 18+ mode: closed
12− - {representation: A4, mode: repairable, agents: [qwen3b]} 19+ agents:
13− - {representation: A5, mode: repairable, agents: [qwen3b]} 20+ - qwen3b
14− - {representation: A4, mode: closed, agents: [qwen3b, llama3b]} 21+- representation: A5
15− - {representation: A5, mode: closed, agents: [qwen3b, llama3b]} 22+ mode: closed
23+ agents:
24+ - qwen3b
25+- representation: A4
26+ mode: repairable
27+ agents:
28+ - qwen3b
29+- representation: A5
30+ mode: repairable
31+ agents:
32+ - qwen3b
33+- representation: A4
34+ mode: closed
35+ agents:
36+ - qwen3b
37+ - llama3b
38+- representation: A5
39+ mode: closed
40+ agents:
41+ - qwen3b
42+ - llama3b
43+protocol_hash: 404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6
modified configs/core.yaml +113 −32
@@ -1,35 +1,116 @@
1−# Confirmatory single-handoff campaign (E1/E2-process/E3-text). Locked test split. Fixed before any test outcome was examined.
2 experiment_id: E1_core 1 experiment_id: E1_core
3−splits: [test] 2+splits:
4−families: [synthetic, exec, public_qasper, public_2wikimqa] 3+- test
5−lengths: [2000, 8000, 16000] 4+families:
6−limit_docs_per_family: 40 # synthetic/exec test = all 40; public: first 40 test docs in manifest (hash) order 5+- synthetic
7−max_questions_per_doc: 5 # synthetic: 5 of 15 question types per document (first five in generator order) 6+- exec
7+- public_qasper
8+- public_2wikimqa
9+lengths:
10+- 2000
11+- 8000
12+- 16000
13+limit_docs_per_family: 40
14+max_questions_per_doc: 5
8 model_pairs: 15 model_pairs:
9− - {sender: qwen3b, receiver: qwen3b} 16+- sender: qwen3b
10− - {sender: llama3b, receiver: llama3b} 17+ receiver: qwen3b
11−generation: {max_tokens: 48, per_family: {exec: 320}} 18+- sender: llama3b
12−seeds: [0] 19+ receiver: llama3b
20+generation:
21+ max_tokens: 48
22+ per_family:
23+ exec: 320
24+seeds:
25+- 0
13 conditions: 26 conditions:
14− - {method: A0} 27+- method: A0
15− - {method: A1} 28+- method: A1
16− - {method: A2, budget_tokens: 512} 29+- method: A2
17− - {method: A3, budget_tokens: 512, access: blind} 30+ budget_tokens: 512
18− - {method: A3, budget_tokens: 512, access: aware} 31+- method: A3
19− - {method: A4, budget_tokens: 512, access: blind} 32+ budget_tokens: 512
20− - {method: A4, budget_tokens: 512, access: aware, families: [synthetic]} 33+ access: blind
21− - {method: A5, budget_tokens: 512, access: blind} 34+- method: A3
22− - {method: A5, budget_tokens: 512, access: aware, families: [synthetic]} 35+ budget_tokens: 512
23− - {method: A6, budget_tokens: 512, access: blind, max_fetch_bytes: 2000, max_lookups: 6} 36+ access: aware
24− - {method: A7, codec: zstd} 37+- method: A4
25− - {method: A8, budget_tokens: 512, access: blind} 38+ budget_tokens: 512
26− - {method: B0} 39+ access: blind
27− - {method: B1} 40+- method: A4
28− - {method: B2, bits: 8} 41+ budget_tokens: 512
29− - {method: B2, bits: 4} 42+ access: aware
30− - {method: B3, keep_frac: 0.25} 43+ families:
31− - {method: B4, keep_frac: 0.25} 44+ - synthetic
32− - {method: B5, keep_frac: 0.25, access: aware} 45+ max_questions: 3
33− - {method: B5, keep_frac: 0.25, access: blind} 46+- method: A5
34− - {method: B6, keep_frac: 0.25, access: aware} 47+ budget_tokens: 512
35− - {method: B9, keep_frac: 0.25, bits: 8, access: aware, policy: attention} 48+ access: blind
49+- method: A5
50+ budget_tokens: 512
51+ access: aware
52+ families:
53+ - synthetic
54+ max_questions: 3
55+- method: A6
56+ budget_tokens: 512
57+ access: blind
58+ max_fetch_bytes: 2000
59+ max_lookups: 6
60+- method: A7
61+ codec: zstd
62+- method: A8
63+ budget_tokens: 512
64+ access: blind
65+- method: B0
66+- method: B1
67+- method: B2
68+ bits: 8
69+- method: B2
70+ bits: 4
71+- method: B3
72+ keep_frac: 0.25
73+- method: B4
74+ keep_frac: 0.25
75+- method: B5
76+ keep_frac: 0.25
77+ access: aware
78+- method: B5
79+ keep_frac: 0.25
80+ access: blind
81+- method: B6
82+ keep_frac: 0.25
83+ access: aware
84+- method: B9
85+ keep_frac: 0.25
86+ bits: 8
87+ access: aware
88+ policy: attention
89+frozen_extra_files:
90+- configs/core.yaml
91+- configs/core_qwen.yaml
92+- configs/core_llama.yaml
93+- configs/core_transfer.yaml
94+- configs/core_sender_alone.yaml
95+- configs/chains_core.yaml
96+- configs/fanin_core.yaml
97+- configs/hosted_core.yaml
98+- configs/router.yaml
99+- runs/checkpoints/router_policy.json
100+- runs/checkpoints/mapper_v2_qwen1b5_to_qwen3b.npz.manifest.json
101+- src/wp30/mapper.py
102+- src/wp30/router.py
103+- src/wp30/router_eval.py
104+- src/wp30/hosted.py
105+- src/wp30/hosted_run.py
106+- scripts/e6_network.py
107+- src/wp30/transport.py
108+core_experiment_ids:
109+- E1_core
110+- E3_core
111+- E3_core_sender
112+- E4_core
113+- E5_core
114+- E6_core
115+- ED_hosted_core
116+- E8_router
added configs/core_llama.yaml +88 −0
@@ -0,0 +1,88 @@
1+experiment_id: E1_core
2+splits:
3+- test
4+families:
5+- synthetic
6+- exec
7+- public_qasper
8+- public_2wikimqa
9+lengths:
10+- 2000
11+- 8000
12+- 16000
13+limit_docs_per_family: 40
14+max_questions_per_doc: 5
15+model_pairs:
16+- sender: llama3b
17+ receiver: llama3b
18+generation:
19+ max_tokens: 48
20+ per_family:
21+ exec: 320
22+seeds:
23+- 0
24+conditions:
25+- method: A0
26+- method: A1
27+- method: A2
28+ budget_tokens: 512
29+- method: A3
30+ budget_tokens: 512
31+ access: blind
32+- method: A3
33+ budget_tokens: 512
34+ access: aware
35+- method: A4
36+ budget_tokens: 512
37+ access: blind
38+- method: A4
39+ budget_tokens: 512
40+ access: aware
41+ families:
42+ - synthetic
43+ max_questions: 3
44+- method: A5
45+ budget_tokens: 512
46+ access: blind
47+- method: A5
48+ budget_tokens: 512
49+ access: aware
50+ families:
51+ - synthetic
52+ max_questions: 3
53+- method: A6
54+ budget_tokens: 512
55+ access: blind
56+ max_fetch_bytes: 2000
57+ max_lookups: 6
58+- method: A7
59+ codec: zstd
60+- method: A8
61+ budget_tokens: 512
62+ access: blind
63+- method: B0
64+- method: B1
65+- method: B2
66+ bits: 8
67+- method: B2
68+ bits: 4
69+- method: B3
70+ keep_frac: 0.25
71+- method: B4
72+ keep_frac: 0.25
73+- method: B5
74+ keep_frac: 0.25
75+ access: aware
76+- method: B5
77+ keep_frac: 0.25
78+ access: blind
79+- method: B6
80+ keep_frac: 0.25
81+ access: aware
82+- method: B9
83+ keep_frac: 0.25
84+ bits: 8
85+ access: aware
86+ policy: attention
87+note: E1_core shard for pair llama3b->llama3b; records merged into runs/raw/E1_core.jsonl
88+protocol_hash: 404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6
added configs/core_qwen.yaml +88 −0
@@ -0,0 +1,88 @@
1+experiment_id: E1_core
2+splits:
3+- test
4+families:
5+- synthetic
6+- exec
7+- public_qasper
8+- public_2wikimqa
9+lengths:
10+- 2000
11+- 8000
12+- 16000
13+limit_docs_per_family: 40
14+max_questions_per_doc: 5
15+model_pairs:
16+- sender: qwen3b
17+ receiver: qwen3b
18+generation:
19+ max_tokens: 48
20+ per_family:
21+ exec: 320
22+seeds:
23+- 0
24+conditions:
25+- method: A0
26+- method: A1
27+- method: A2
28+ budget_tokens: 512
29+- method: A3
30+ budget_tokens: 512
31+ access: blind
32+- method: A3
33+ budget_tokens: 512
34+ access: aware
35+- method: A4
36+ budget_tokens: 512
37+ access: blind
38+- method: A4
39+ budget_tokens: 512
40+ access: aware
41+ families:
42+ - synthetic
43+ max_questions: 3
44+- method: A5
45+ budget_tokens: 512
46+ access: blind
47+- method: A5
48+ budget_tokens: 512
49+ access: aware
50+ families:
51+ - synthetic
52+ max_questions: 3
53+- method: A6
54+ budget_tokens: 512
55+ access: blind
56+ max_fetch_bytes: 2000
57+ max_lookups: 6
58+- method: A7
59+ codec: zstd
60+- method: A8
61+ budget_tokens: 512
62+ access: blind
63+- method: B0
64+- method: B1
65+- method: B2
66+ bits: 8
67+- method: B2
68+ bits: 4
69+- method: B3
70+ keep_frac: 0.25
71+- method: B4
72+ keep_frac: 0.25
73+- method: B5
74+ keep_frac: 0.25
75+ access: aware
76+- method: B5
77+ keep_frac: 0.25
78+ access: blind
79+- method: B6
80+ keep_frac: 0.25
81+ access: aware
82+- method: B9
83+ keep_frac: 0.25
84+ bits: 8
85+ access: aware
86+ policy: attention
87+note: E1_core shard for pair qwen3b->qwen3b; records merged into runs/raw/E1_core.jsonl
88+protocol_hash: 404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6
modified configs/core_sender_alone.yaml +19 −9
@@ -1,14 +1,24 @@
1−# Sender-alone reference for E3 (Qwen2.5-1.5B answering with full context) on the same items.
2 experiment_id: E3_core_sender 1 experiment_id: E3_core_sender
3−splits: [test] 2+splits:
4−families: [synthetic, exec] 3+- test
5−lengths: [2000, 8000] 4+families:
5+- synthetic
6+- exec
7+lengths:
8+- 2000
9+- 8000
6 limit_docs_per_family: null 10 limit_docs_per_family: null
7 max_questions_per_doc: 8 11 max_questions_per_doc: 8
8 model_pairs: 12 model_pairs:
9− - {sender: qwen1b5, receiver: qwen1b5} 13+- sender: qwen1b5
10−generation: {max_tokens: 48, per_family: {exec: 320}} 14+ receiver: qwen1b5
11−seeds: [0] 15+generation:
16+ max_tokens: 48
17+ per_family:
18+ exec: 320
19+seeds:
20+- 0
12 conditions: 21 conditions:
13− - {method: A1} 22+- method: A1
14− - {method: B0} 23+- method: B0
24+protocol_hash: 404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6
modified configs/core_transfer.yaml +31 −13
@@ -1,18 +1,36 @@
1−# Track C confirmatory: same-family size ladder Qwen2.5-1.5B -> Qwen2.5-3B (shared tokenizer). Test split, synthetic + exec.
2 experiment_id: E3_core 1 experiment_id: E3_core
3−splits: [test] 2+splits:
4−families: [synthetic, exec] 3+- test
5−lengths: [2000, 8000] 4+families:
5+- synthetic
6+- exec
7+lengths:
8+- 2000
9+- 8000
6 limit_docs_per_family: null 10 limit_docs_per_family: null
7 max_questions_per_doc: 8 11 max_questions_per_doc: 8
8 model_pairs: 12 model_pairs:
9− - {sender: qwen1b5, receiver: qwen3b} 13+- sender: qwen1b5
10−generation: {max_tokens: 48, per_family: {exec: 320}} 14+ receiver: qwen3b
11−seeds: [0] 15+generation:
16+ max_tokens: 48
17+ per_family:
18+ exec: 320
19+seeds:
20+- 0
12 conditions: 21 conditions:
13− - {method: A1} # receiver alone, full context (reference) 22+- method: A1
14− - {method: A4, budget_tokens: 512, access: blind} # sender-generated summary -> receiver 23+- method: A4
15− - {method: A5, budget_tokens: 512, access: blind} 24+ budget_tokens: 512
16− - {method: C0} # identity negative control 25+ access: blind
17− - {method: C2, mapper: runs/checkpoints/mapper_qwen1b5_to_qwen3b.npz} 26+- method: A5
18− - {method: C3, mapper: runs/checkpoints/mapper_qwen1b5_to_qwen3b.npz, keep_frac: 0.25, access: aware} 27+ budget_tokens: 512
28+ access: blind
29+- method: C0
30+- method: C2
31+ mapper: runs/checkpoints/mapper_v2_qwen1b5_to_qwen3b.npz
32+- method: C3
33+ mapper: runs/checkpoints/mapper_v2_qwen1b5_to_qwen3b.npz
34+ keep_frac: 0.25
35+ access: aware
36+protocol_hash: 404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6
modified configs/fanin_core.yaml +15 −4
@@ -1,9 +1,20 @@
1 experiment_id: E5_core 1 experiment_id: E5_core
2−splits: [test] 2+splits:
3−lengths: [2000, 8000] 3+- test
4+lengths:
5+- 2000
6+- 8000
4 limit_docs: 20 7 limit_docs: 20
5−fan_in: [2, 4] 8+fan_in:
9+- 2
10+- 4
6 budget_tokens: 512 11 budget_tokens: 512
7 sender: qwen3b 12 sender: qwen3b
8 aggregator: qwen3b 13 aggregator: qwen3b
9−methods: [full_context, concat_A4, merge_A5, shared_memory_A6, cache_concat_negative_control] 14+methods:
15+- full_context
16+- concat_A4
17+- merge_A5
18+- shared_memory_A6
19+- cache_concat_negative_control
20+protocol_hash: 404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6
modified configs/hosted_core.yaml +28 −10
@@ -1,15 +1,33 @@
1 experiment_id: ED_hosted_core 1 experiment_id: ED_hosted_core
2−splits: [test] 2+splits:
3−families: [synthetic, exec, public_qasper] 3+- test
4−lengths: [2000, 8000] 4+families:
5+- synthetic
6+- exec
7+- public_qasper
8+lengths:
9+- 2000
10+- 8000
5 limit_docs_per_family: 8 11 limit_docs_per_family: 8
6 max_questions_per_doc: 5 12 max_questions_per_doc: 5
7 sender: qwen3b 13 sender: qwen3b
8−providers: [anthropic, openai] 14+providers:
15+- anthropic
16+- openai
9 conditions: 17 conditions:
10− - {method: A0} 18+- method: A0
11− - {method: A1} 19+- method: A1
12− - {method: A1, provider_cache: true} 20+- method: A1
13− - {method: A4, budget_tokens: 512, access: blind} 21+ provider_cache: true
14− - {method: A5, budget_tokens: 512, access: blind} 22+- method: A4
15− - {method: A6, budget_tokens: 512, access: blind, max_fetch_bytes: 2000, max_lookups: 6} 23+ budget_tokens: 512
24+ access: blind
25+- method: A5
26+ budget_tokens: 512
27+ access: blind
28+- method: A6
29+ budget_tokens: 512
30+ access: blind
31+ max_fetch_bytes: 2000
32+ max_lookups: 6
33+protocol_hash: 404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6
modified configs/pilot.yaml +1 −1
@@ -2,7 +2,7 @@ experiment_id: E1_pilot
2 splits: [dev] 2 splits: [dev]
3 families: [synthetic, exec, public_qasper, public_2wikimqa] 3 families: [synthetic, exec, public_qasper, public_2wikimqa]
4 lengths: [2000, 8000] 4 lengths: [2000, 8000]
5−limit_docs_per_family: 4 5+limit_docs_per_family: 2
6 max_questions_per_doc: 5 6 max_questions_per_doc: 5
7 model_pairs: 7 model_pairs:
8 - {sender: qwen3b, receiver: qwen3b} 8 - {sender: qwen3b, receiver: qwen3b}
added configs/transfer_pilot_v2.yaml +12 −0
@@ -0,0 +1,12 @@
1+experiment_id: E3_pilot_v2
2+splits: [dev]
3+families: [synthetic]
4+lengths: [2000, 8000]
5+limit_docs_per_family: 4
6+max_questions_per_doc: 5
7+model_pairs:
8+ - {sender: qwen1b5, receiver: qwen3b}
9+generation: {max_tokens: 48}
10+seeds: [0]
11+conditions:
12+ - {method: C2, mapper: runs/checkpoints/mapper_v2_qwen1b5_to_qwen3b.npz}
modified paper/sections/limitations.tex +46 −2
@@ -1,3 +1,47 @@
1−\section{Limitations} 1+\section{Failure analysis and limitations}
2 \label{sec:limitations} 2 \label{sec:limitations}
3−To be written from results. 3+
4+\paragraph{Contextual, not parametric.} Every effect reported here disappears when the message or cache is removed (A0). No receiver
5+acquired knowledge; frozen models were given better or worse \emph{access} to information. Claims about ``learning'' are out of scope.
6+
7+\paragraph{Local white-box versus hosted black-box.} Cache experiments require a runtime that exposes attention tensors (MLX here).
8+Hosted APIs expose a billing abstraction (cached input tokens) whose precision, layout and residency are undisclosed; Track~D therefore
9+contains text methods only and its cached-token fields are never translated into bytes.
10+
11+\paragraph{Same checkpoint versus cross-family.} All cache reuse and transfer results concern identical checkpoints and revisions
12+(Track~B, two-node B1) or a same-family size ladder with a linear mapper (Track~C). Cross-family cache transfer was not executed: the
13+only published method with code (SwaT) released no checkpoint and no licence, and training a mapper of its size exceeded our budget.
14+The paper's title names knowledge transfer in general; its \emph{cache}-transfer evidence is same-family only.
15+
16+\paragraph{Query-aware versus reusable compression.} Attention-based selection is query-dependent; its question-blind variant (generic
17+window) is a different method with much lower quality (Section~\ref{sec:results}). Reusable (blind) messages and query-specific
18+(aware) messages are reported in separate panels and should not be ranked against each other.
19+
20+\paragraph{Translated-cache distribution shift.} The mapper was fitted on native sender caches; chains of mapped caches (feeding a mapped
21+cache into a further mapping) were not run, so no closure property is claimed.
22+
23+\paragraph{Budgets are measured but not universal.} Byte counts include manifests and quantization metadata; latencies include
24+serialization, loading and fresh-query processing; memory peaks are MLX allocator peaks. They are specific to \texttt{mlx-lm}~0.32.2
25+kernels on one M5~Max and two M3~Ultra machines; a fused eviction kernel or a different attention backend would change the compression
26+and TTFT components. We compared methods on a common backend; we did not compare against vendor-optimized kernels.
27+
28+\paragraph{Network and reuse.} The two-node link is a 1~GbE LAN with 1.2~ms RTT; break-even positions scale with bandwidth and with
29+the receiver's prefill speed, both of which we report so that the analytic curves can be re-drawn. Queue delays were not imposed; the
30+cluster nodes carried a light concurrent CPU workload from another study.
31+
32+\paragraph{Contamination and synthetic facts.} The public QA texts may appear in pre-training corpora; the communication-dependence
33+conclusions rest on the synthetic private-fact family, whose facts are new by construction but whose templates are narrower than real
34+documents. The generated ledger is not a filing; the executable tasks are small.
35+
36+\paragraph{Scoring.} All scoring is deterministic (exact/numeric/yes-no/abstention/F1/hidden tests); no language-model judge and no human
37+annotation were used. The fact-ledger retention metric is string containment of normalized values, which under-counts paraphrased
38+retention and over-counts accidental matches of short values.
39+
40+\paragraph{Coverage.} Two 3B model families, three task families, one sender budget (512 tokens) in the core, one selection ratio
41+(1/4) for the registered comparisons. Audited but unexecuted methods are listed with reasons in Table~\ref{tab:taxonomy} and
42+Appendix~\ref{app:audit}; their absence is a coverage gap, not evidence about them.
43+
44+\paragraph{Auditability and security of latent messages.} A cache is not human-readable; our manifests bind it to a checkpoint
45+revision, tokenizer, chat template, prefix-token hash and document id, and the loader rejects mismatches, which protects against the
46+wrong-document handle. Structured evidence with spans is auditable but depends on the integrity of the source it points to; retrieved
47+passages are treated as data, never as instructions, and the public benchmark text never enters the operator's command channel.
modified paper/sections/reproducibility.tex +34 −3
@@ -1,3 +1,34 @@
1−\section{Reproducibility} 1+\section{Reproducibility and execution disclosure}
2−\label{sec:reproducibility} 2+\label{sec:repro}
3−To be written from results. 3+
4+\paragraph{How the study was executed.} All code, experiments, analyses and this manuscript were produced by Claude Code (Anthropic;
5+model \texttt{claude-fable-5-1}) acting as research operator under a written brief from the author, through actual terminal
6+commands on the author's machines: environment inspection, dependency pinning, implementation, launching inference, reading exit
7+codes and logs, diagnosing failures, re-running affected trials, analysis and compilation. The author provided the brief, the
8+hardware, API credentials already present in sibling projects, and no mid-run intervention; dated engineering decisions are in
9+\texttt{DECISIONS.md}. Models under test are the experimental subjects; the operator's own inference state was never an
10+experimental interface. This disclosure describes the process; it is not a claim that autonomous research is superior to any
11+alternative.
12+
13+\paragraph{Defects found and handled.} (i) The \texttt{mlx-lm} cache API changed between the system-installed version (0.31.3) and the
14+pinned environment (0.32.2); adapters were rewritten for the pinned API before any experiment. (ii) The receiver prefix boundary was
15+initially computed from the prefix text alone and did not match the tokenization of the full prompt; it is now the common token
16+prefix with a canonical question, and a mismatch raises. (iii) Heterogeneous retained lengths (B6) broke the shared attention mask;
17+the fresh query is processed token by token for such caches. (iv) The first executable-task specification (seven rules) was unsolvable
18+for 3B receivers even with the full ticket; it was simplified on development data before any test item was run (D-010). (v) Running
19+three GPU jobs concurrently exhausted unified memory and slowed the pilot by an order of magnitude; GPU jobs were serialized.
20+
21+\paragraph{Artifacts.} The repository contains the harness, pinned lock file, data manifests and generators, the frozen protocol and its
22+hash, every per-attempt record (\texttt{runs/raw}), sanitized logs, the mapper checkpoint and manifest, generated tables and figures
23+with a provenance index linking each to its source records and script, a claim ledger, and the minimal reproduction profile.
24+Checkpoints are not redistributed; their Hugging Face identifiers, revisions and weight hashes are recorded. Hosted-API trials require
25+the provider credentials and reproduce only up to provider nondeterminism.
26+
27+\paragraph{Reproduction commands.}
28+\begin{footnotesize}\begin{verbatim}
29+make reproduce-analysis # tables + figures from runs/raw, provenance audit
30+python -m wp30 reproduce --profile reproduction/minimal.yaml # re-run a fixed subset, compare greedy outputs
31+make paper # compile this document
32+\end{verbatim}\end{footnotesize}
33+Greedy decoding on MLX is deterministic for an identical prefill chunking but not bitwise across chunkings (Section~\ref{sec:correctness});
34+the reproduction report lists identical output hashes and identical scores separately.
modified paper/tables/protocol_hash.tex +1 −1
@@ -1 +1 @@
1−unfrozen
\ No newline at end of file
1+404c52b34d964693
added protocol/core_matrix.csv +6561 −0
Large diff (6,561 lines) is not loaded by default to keep the page fast.Load diff
modified protocol/core_matrix_draft.csv +160 −160
Large diff (880 lines) is not loaded by default to keep the page fast.Load diff
added protocol/frozen.json +56 −0
Large diff (57 lines) is not loaded by default to keep the page fast.Load diff
modified protocol/hypotheses.md +8 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
modified protocol/pilot_matrix.csv +0 −496
Large diff (517 lines) is not loaded by default to keep the page fast.Load diff
added reproduction/minimal.yaml +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added reproduction/minimal_config.yaml +19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added results/figures/PALETTE.txt +1 −0
Large diff (1 lines) is not loaded by default to keep the page fast.Load diff
added results/figures/fig2_quality_vs_bytes.pdf +0 −0

Binary file not shown.

added results/figures/fig2_quality_vs_bytes.png +0 −0

Binary file not shown.

added results/figures/fig3_quality_vs_latency_memory.pdf +0 −0

Binary file not shown.

added results/figures/fig3_quality_vs_latency_memory.png +0 −0

Binary file not shown.

added results/figures/fig4_latency_decomposition.pdf +0 −0

Binary file not shown.

added results/figures/fig4_latency_decomposition.png +0 −0

Binary file not shown.

added results/figures/figA_quality_vs_bytes_aware.pdf +0 −0

Binary file not shown.

added results/figures/figA_quality_vs_bytes_aware.png +0 −0

Binary file not shown.

added results/figures/figA_quality_vs_bytes_blind.pdf +0 −0

Binary file not shown.

added results/figures/figA_quality_vs_bytes_blind.png +0 −0

Binary file not shown.

added results/provenance.json +122 −0
Large diff (123 lines) is not loaded by default to keep the page fast.Load diff
added results/tables/E1_pilot_by_question_type.csv +109 −0
Large diff (109 lines) is not loaded by default to keep the page fast.Load diff
added results/tables/E1_pilot_failures.csv +18 −0
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added results/tables/E1_pilot_primary.csv +63 −0
Large diff (63 lines) is not loaded by default to keep the page fast.Load diff
added results/tables/E1_pilot_primary_tests.csv +6 −0
Large diff (6 lines) is not loaded by default to keep the page fast.Load diff
added results/tables/ED_hosted.csv +9 −0
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
added runs/checkpoints/mapper_v2_qwen1b5_to_qwen3b.npz +0 −0

Binary file not shown.

added runs/checkpoints/mapper_v2_qwen1b5_to_qwen3b.npz.manifest.json +202 −0
Large diff (203 lines) is not loaded by default to keep the page fast.Load diff
added runs/checkpoints/router_policy.json +1148 −0
Large diff (1,149 lines) is not loaded by default to keep the page fast.Load diff
modified runs/raw/E1_pilot.jsonl +496 −0
Large diff (499 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/E3_pilot.jsonl +107 −0
Large diff (107 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/E3_pilot_v2.jsonl +20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/E4_pilot.jsonl +330 −0
Large diff (330 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/ED_hosted_pilot_m3u96b.jsonl +35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added scripts/run_e6_core.sh +7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/analysis/analyze.py +4 −2
Large diff (18 lines) is not loaded by default to keep the page fast.Load diff
added src/wp30/analysis/chains_recompute.py +31 −0
Large diff (31 lines) is not loaded by default to keep the page fast.Load diff
added src/wp30/analysis/claims.py +12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/analysis/load.py +6 −2
Large diff (26 lines) is not loaded by default to keep the page fast.Load diff
added src/wp30/analysis/numbers.py +39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/chains.py +2 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/hosted.py +1 −1
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/runner.py +11 −5
Large diff (52 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere