WP30 — FINAL REPORT
Status: COMPLETE_WITHIN_DECLARED_SCOPE (with the explicit exclusion of cross-family cache transfer; see "Not executed").
Paper: paper/WP30_Boucher_Cache_Compression_Knowledge_Transfer.pdf (LaTeX in paper/). Protocol frozen 2026-10-30 04:32 UTC
(protocol/frozen.json, hash 404c52b3…); amendments A-001–A-004 in protocol/hypotheses.md; dated decisions D-001–D-019 in DECISIONS.md.
What was actually executed (all from Claude Code, real model inference)
| Campaign | What | Trials / runs | Where | Status |
|---|---|---|---|---|
| E0 | cache correctness suite (identity, prefix equivalence, positions, packed bytes, loader rejection, isolation) | 12 tests + 20 unit tests | laptop | 32/32 pass |
| E1_pilot | dev calibration, 31 conditions × 2 pairs | 1 037 | laptop | done (dev only) |
| E1_core | 22 conditions × 2 pairs × 160 test documents | 13 280 (+240 A8 retries) | laptop (Qwen pair, Llama public) + M3U96a (Llama synthetic/exec) | complete |
| E3_core (+sender) | Qwen2.5-1.5B→3B ridge mapper v2, identity control, text handoffs, sender-alone | 1 536 + 512 | M3U96a | complete |
| E4_core | relays 1/2/4/8 handoffs, closed/repairable, Qwen and Qwen/Llama, fact ledger | 7 200 | laptop + M3U96b (document parity shards) | complete |
| E5_core | fan-in k=2,4 with superseded revision + irrelevant source; 5 compositions incl. cache-concat negative control | 1 600 | laptop | complete |
| E6_core | genuine two-node transfer M3U96a→M3U96b (1 GbE, 940 Mbit/s): bf16/zstd/8-bit/4-bit caches vs remote text prefill vs local | 324 | cluster | complete |
| ED_hosted_core | A0–A6 messages → claude-haiku-4-5 and gpt-5-nano, provider prefix caching | 672 (0.53 USD; pilots 0.19 USD) | M3U96b + APIs | complete |
| E8 | router fitted on dev pilot, evaluated off-policy on the full-factorial test records | 640 instances × 2 regimes | laptop | complete |
| E9 | minimal reproduction from a clean process | 64 trials | laptop | 64/64 identical outputs and scores |
| E7_qtypes | EXPLORATORY: question types 6–15 (negation, revision, temporal, aggregation, units, unanswerable) | 5 600 planned | laptop | see STATUS |
Models: mlx-community/Qwen2.5-3B-Instruct-bf16, Llama-3.2-3B-Instruct-bf16, Qwen2.5-1.5B-Instruct-bf16 (revisions and weight hashes in
evidence/models.json). Hardware: MacBook Pro M5 Max 48 GB (laptop), two Mac Studio M3 Ultra 96 GB (M3U96a/b) on a wired 1 GbE LAN.
Runtimes: mlx 0.32.4, mlx-lm 0.32.2, torch 2.15 (MPS, LLMLingua-2 only), Python 3.12.13. Total wall time ≈ 10 h of campaigns (2026-10-29 23:00 →
2026-10-30 09:30 UTC) plus development; API spend 0.72 USD (cap 25 USD).
Principal findings (each with a claim-ledger row in results/claim_ledger.csv)
- Lossless representations (text, zstd text, native cache, reloaded cache, 8-bit packed cache) are indistinguishable in quality; their cost ranking inverts between cold (text wins) and reused (cache wins) regimes.
- 4-bit packed KV quantization: Llama-3.2-3B ≥ 97 %, Qwen2.5-3B 0 % (native mlx-lm path reproduces it) — H2 confirmed.
- SnapKV-style eviction at 1/4 with the question keeps 78–98 % of private-fact accuracy; with a generic window 12–19 %; sink+recent 18–20 %; random 0 %; every eviction scores 0 on executable tasks.
- At a 512-token sender budget, prose summaries (38–44 %) beat structured evidence with spans (12–15 %): the budget is spent on quotes — H1 rejected.
- Relays: loss at hop 1, then nearly lossless to hop 8 (H5 rejected); no invented identifiers; metered repair +6 pts retention for 5.9 kB.
- Fan-in: concatenated summaries ≈ full context on joins; merged structured records fail; cache concatenation (invalid) still answers 70 % at k=2 — a warning.
- Two-node: bf16 cache 215 MB at 8k tokens moves in 1.8 s; 8-bit cache answers in 1.7 s vs 3.7 s for remote text prefill on an M3 Ultra receiver.
- Same-family ridge mapper: 68 % of value variance reconstructed, 0–4 % content transfer, coherent "unknown" abstention — H10 confirmed.
- Hosted receivers recover everything from full text and the same ~30 % from the local summary; Anthropic caching needs ≥ 4 096 tokens.
- Router: equal quality at half the amortized latency (blind); no gain and higher cost (aware).
Not executed / limitations
- Cross-family cache transfer (SwaT: no checkpoint, no licence; training a 300M mapper exceeded the budget) → the transfer study is same-family only.
- Attention-Matching compaction, C2C, LatentMAS, H2O, KIVI kernels: audited, not run (CUDA/vLLM code; see Table 1 and
literature/method_notes.md). - One sender budget (512), one retention ratio (1/4) in the confirmatory family; 2-bit dropped from the core; public QA subsets of 40 documents.
- Llama-pair latencies for synthetic/exec come from M3U96a, public from the laptop (hardware recorded per record; within-pair, within-family comparisons only).
- Hosted sampling is provider default (temperature not accepted); A8 for the Llama pair was retried on the laptop after a missing-runtime failure on M3U96a.
- No human validation, no LLM judge; deterministic scorers only. Synthetic facts are new; public texts may be in pre-training corpora.
Reproduction
uv venv --python 3.12 .venv && source .venv/bin/activate && uv pip install -r reproduction/requirements-lock.txt && uv pip install -e .
make reproduce-analysis # tables, figures, provenance, audit (no inference)
python -m wp30 reproduce --profile reproduction/minimal.yaml # re-run 64 trials, compare outputs (needs the two checkpoints, ~15 GB)
make paperFull campaigns: configs/core*.yaml, chains_core*.yaml, fanin_core.yaml, hosted_core.yaml; two-node: scripts/run_e6_core.sh.
Execution disclosure
Operator: Claude Code (claude-fable-5-1). Human interventions: none during execution (brief provided at start). Defects found and fixed are listed in
the paper (Section 9) and in DECISIONS.md. No screenshots were captured; machine-readable logs (runs/logs) and rendered page images
(evidence/pages) are the execution evidence.