SPB Git
spboucher.ai

spb/wp30_uqo

Public

WP30: E6 two-node core measured (324 runs), analysis/break-even figure, network results section, collection scripts

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Oct 30, 2026) parent e6d9a3b Browse files

69 changed files +9,223 −149

69 changed files
Load all 69 files Unified Split
modified STATUS.md +8 −5
@@ -1,8 +1,11 @@
1 # STATUS.md — WP30 1 # STATUS.md — WP30
2 2
3−**Phase:** 3→4 complete (harness + live inference + cache correctness suite 12/12), 5 in progress (text methods implemented, smoke-tested), 6 pending 3+**Phase:** 8 (core single-handoff campaign running) — protocol frozen 2026-10-30 04:32 UTC, hash 404c52b34d964693…
4−**Completed gates:** Phase 0 (environment, scope), Phase 1 (literature audit: 34 sources, 3 audit files), Phase 2 (datasets: 800 items, keys isolated, 0 cross-split duplicates), Phase 3 (E0 smoke: 114 trials, real generation), Phase 4 (tests/test_cache_correctness.py 12 passed) 4+**Completed gates:** 0 environment · 1 literature (34 sources) · 2 datasets (800 items, keys isolated) · 3 live inference · 4 cache correctness (12/12) ·
5−**Current task:** pilot config, Llama receiver smoke, chain/fan-in smoke, remote worker (M3U96b) for E2/E6 5+5 text/evidence methods (A0–A8) · 6 advanced methods (B5/B6 faithful re-implementations; linear mapper v1/v2 trained on M3U96b; SwaT/C2C/LatentMAS/H2O/Compaction audited, not run) ·
6+7 pilot (E1_pilot 1 037 trials dev; E3_pilot/v2; E4_pilot; ED_hosted_pilot) + freeze
7+**Running:** E1_core Qwen shard (laptop, started 04:32 UTC); E4_pilot (M3U96a, finishing); ED_hosted_pilot (M3U96b)
8+**Queued:** E1_core Llama shard (M3U96a) · E6_core (M3U96a→M3U96b) · E4_core, E5_core, E3_core(+sender) (M3U96b) · ED_hosted_core · E8 router eval · analysis · paper
6 **Blockers:** none 9 **Blockers:** none
7−**Resume:** `source .venv/bin/activate && python -m wp30 run --config configs/pilot.yaml --resume` 10+**Resume commands:** laptop `python -m wp30 run --config configs/core_qwen.yaml --resume`; M3U96a `python -m wp30 run --config configs/core_llama.yaml --resume`;
8−**Run IDs so far:** E0_smoke (runs/raw/E0_smoke.jsonl) 11+M3U96b `python -m wp30.cli_extra chains configs/chains_core.yaml --resume`, `fanin configs/fanin_core.yaml --resume`, `python -m wp30 run --config configs/core_transfer.yaml --resume`
modified paper/sections/results.tex +12 −1
@@ -1,3 +1,14 @@
1 \section{Results} 1 \section{Results}
2 \label{sec:results} 2 \label{sec:results}
3−To be written from results. 3+\input{tables/numbers}
4+All numbers in this section are computed from the released per-attempt records by \texttt{src/wp30/analysis/analyze.py}; every table
5+cell and plotted point resolves to run identifiers through \texttt{results/provenance.json}. Quality is accuracy for the synthetic and
6+executable families and LongBench F1 for the public subsets; intervals are 95\,\% document-level cluster-bootstrap intervals; test
7+statistics follow the registered plan (Section~\ref{sec:protocol}). Scheduled core trials: \nCoreTrials.
8+
9+\input{sections/results_frontier}
10+\input{sections/results_transfer}
11+\input{sections/results_handoffs}
12+\input{sections/results_fanin}
13+\input{sections/results_network}
14+\input{sections/results_hosted}
added paper/sections/results_fanin.tex +1 −0
@@ -0,0 +1 @@
1+% pending
added paper/sections/results_frontier.tex +1 −0
@@ -0,0 +1 @@
1+% pending
added paper/sections/results_handoffs.tex +1 −0
@@ -0,0 +1 @@
1+% pending
added paper/sections/results_hosted.tex +1 −0
@@ -0,0 +1 @@
1+% pending
added paper/sections/results_network.tex +49 −0
@@ -0,0 +1,49 @@
1+\subsection{Two-node reuse and locality (E2/E6)}
2+\label{sec:network}
3+Figure~\ref{fig:breakeven} and Table~\ref{tab:network} report genuine two-node measurements: the sender node (M3U96a) prefills a
4+test document with Qwen2.5-3B, serializes the cache with its manifest, optionally quantizes or compresses it, and sends it over the
5+LAN to a receiver worker (M3U96b) that checks the manifest, installs the cache, prefills only the fresh question and decodes the
6+answer; the alternative sends the token ids and lets the worker prefill the whole document. Each point is the median of 18 runs
7+(6 test documents $\times$ 3 repeats, after one warm-up on both ends); the effective transport rate was
8+\EsixBONoneEZZZGbps{}~Gb/s on a link that iperf3 measured at 0.94~Gb/s.
9+
10+\begin{figure}[t]\centering
11+\includegraphics[width=\textwidth]{fig6_breakeven.pdf}
12+\caption{Reuse/network break-even. Left: measured round trip to a remote answer (medians, p25--p75 bars) for moving the native bf16 cache
13+(B1), the zstd-compressed cache, the 8-bit and 4-bit packed caches (B2), versus sending the text and prefilling remotely; the grey line is
14+the sender's own local prefill for scale. Right: analytic transfer time from the measured bandwidth (solid) and a hypothetical 10~Gb/s link
15+(dashed) against the measured remote prefill slope. Qwen2.5-3B; synthetic test documents; M3U96a $\to$ M3U96b.}\label{fig:breakeven}
16+\end{figure}
17+
18+\begin{table}[t]\centering\scriptsize
19+\caption{Two-node transfer (medians of 18 runs; ms unless stated). ``Send'' is the sender-side socket time; ``load'' is manifest check plus
20+deserialization at the worker; the round trip includes the worker's fresh-query prefill and 32 decoded tokens. 4-bit rows transfer a cache
21+that yields quality~0 for Qwen2.5 (Section~\ref{sec:results-frontier}) and are shown only for byte and time accounting.}\label{tab:network}
22+\begin{tabular}{@{}llrrrrr@{}}\toprule
23+Context class & Representation & payload (MB) & send & load & remote prefill & round trip \\\midrule
24+2k & text $\to$ remote prefill & 0.01 & -- & -- & \EsixTextRemotePrefillNoneTZZZPrefill & \EsixTextRemotePrefillNoneTZZZRt \\
25+2k & B1 bf16 cache & \EsixBONoneTZZZBytes & \EsixBONoneTZZZSend & \EsixBONoneTZZZLoad & 0 & \EsixBONoneTZZZRt \\
26+2k & B2 8-bit cache & \EsixBTENoneTZZZBytes & \EsixBTENoneTZZZSend & \EsixBTENoneTZZZLoad & 0 & \EsixBTENoneTZZZRt \\
27+8k & text $\to$ remote prefill & 0.03 & -- & -- & \EsixTextRemotePrefillNoneEZZZPrefill & \EsixTextRemotePrefillNoneEZZZRt \\
28+8k & B1 bf16 cache & \EsixBONoneEZZZBytes & \EsixBONoneEZZZSend & \EsixBONoneEZZZLoad & 0 & \EsixBONoneEZZZRt \\
29+8k & B2 8-bit cache & \EsixBTENoneEZZZBytes & \EsixBTENoneEZZZSend & \EsixBTENoneEZZZLoad & 0 & \EsixBTENoneEZZZRt \\
30+16k (${\approx}$11k tokens) & text $\to$ remote prefill & 0.06 & -- & -- & \EsixTextRemotePrefillNoneOSZZZPrefill & \EsixTextRemotePrefillNoneOSZZZRt \\
31+16k & B1 bf16 cache & \EsixBONoneOSZZZBytes & \EsixBONoneOSZZZSend & \EsixBONoneOSZZZLoad & 0 & \EsixBONoneOSZZZRt \\
32+16k & B2 8-bit cache & \EsixBTENoneOSZZZBytes & \EsixBTENoneOSZZZSend & \EsixBTENoneOSZZZLoad & 0 & \EsixBTENoneOSZZZRt \\\bottomrule
33+\end{tabular}
34+\end{table}
35+
36+Three observations follow. First, the serialized bf16 cache is three to four orders of magnitude larger than the text it replaces
37+(\EsixBONoneEZZZBytes{}~MB versus 29~kB at 8k tokens), and zstd recovers only ${\approx}20$\,\% of it (high-entropy payload; H8), while packed
38+8-bit quantization halves it. Second, on this link and with this receiver, moving the cache is nevertheless \emph{faster} than re-reading
39+the text at every context length we measured: at 8k tokens the bf16 transfer answers in \EsixBONoneEZZZRt{}~ms against
40+\EsixTextRemotePrefillNoneEZZZRt{}~ms for remote prefill, and the 8-bit cache in \EsixBTENoneEZZZRt{}~ms, because the M3~Ultra receiver prefills
41+Qwen2.5-3B at ${\approx}0.44$~ms per token (\EsixTextRemotePrefillNoneEZZZPrefill{}~ms for 8k) whereas 1~GbE moves the per-token cache
42+(${\approx}26$~kB bf16, ${\approx}14$~kB at 8 bits) in ${\approx}0.23$ and ${\approx}0.12$~ms. The break-even therefore depends on the ratio
43+of receiver prefill time to link time per token, which the right panel makes explicit: a faster receiver (the laptop prefills 8k tokens in
44+${\approx}1$~s) or a slower link would move the crossing to longer contexts, and a 10~Gb/s link would make transfer dominant everywhere. Third,
45+the cost of building the cache is paid by the sender and is counted once per document: when the cache is reused for $N$ queries at the
46+receiver, the per-query cost is the fresh-query prefill plus decoding (${\approx}$\EsixBONoneEZZZLoad{}~ms load and ${\approx}190$~ms to the
47+first token at 8k), while the text alternative pays the full prefill on each cold query unless the receiver keeps its own cache---at which
48+point the two strategies converge and the question becomes where to schedule the work. Equation~\eqref{eq:amortized} with the measured
49+components gives the break-even reuse count for any bandwidth; these are measured points on one link, not a universal bound.
added paper/sections/results_transfer.tex +1 −0
@@ -0,0 +1 @@
1+% pending
added paper/tables/numbers.tex +268 −0
@@ -0,0 +1,268 @@
1+% auto-generated by wp30.analysis.numbers — do not edit
2+\newcommand{\nCoreTrials}{3,560}
3+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAZ}{0}
4+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAZ}{0}
5+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAZ}{0}
6+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAZ}{0}
7+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAZ}{50}
8+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAZ}{50}
9+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAO}{99}
10+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAO}{97}
11+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAO}{100}
12+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAO}{29,157}
13+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAO}{973}
14+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAO}{973}
15+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOT}{21}
16+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOT}{17}
17+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOT}{24}
18+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOT}{2,244}
19+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOT}{112}
20+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOT}{108}
21+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTAware}{97}
22+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTAware}{95}
23+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTAware}{99}
24+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTAware}{1,368}
25+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTAware}{209}
26+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTAware}{207}
27+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTBlind}{38}
28+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTBlind}{35}
29+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTBlind}{39}
30+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTBlind}{1,247}
31+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTBlind}{158}
32+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTBlind}{152}
33+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{94}
34+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{88}
35+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{98}
36+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{382}
37+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{3,078}
38+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{190}
39+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{38}
40+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{28}
41+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{48}
42+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{1,424}
43+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{7,451}
44+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{132}
45+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{100}
46+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{100}
47+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{100}
48+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{118}
49+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{2,195}
50+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{179}
51+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{16}
52+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{13}
53+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{18}
54+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{828}
55+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{8,171}
56+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{85}
57+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASBudgetTokensFOTMaxFetchBytesTZZZMaxLookupsSBlind}{17}
58+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASBudgetTokensFOTMaxFetchBytesTZZZMaxLookupsSBlind}{14}
59+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASBudgetTokensFOTMaxFetchBytesTZZZMaxLookupsSBlind}{21}
60+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASBudgetTokensFOTMaxFetchBytesTZZZMaxLookupsSBlind}{636}
61+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASBudgetTokensFOTMaxFetchBytesTZZZMaxLookupsSBlind}{176}
62+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASBudgetTokensFOTMaxFetchBytesTZZZMaxLookupsSBlind}{176}
63+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASCodecZstd}{99}
64+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASCodecZstd}{97}
65+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASCodecZstd}{100}
66+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASCodecZstd}{2,725}
67+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASCodecZstd}{1,042}
68+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASCodecZstd}{1,042}
69+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAEBudgetTokensFOTBlind}{13}
70+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAEBudgetTokensFOTBlind}{8}
71+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAEBudgetTokensFOTBlind}{18}
72+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAEBudgetTokensFOTBlind}{2,571}
73+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAEBudgetTokensFOTBlind}{1,328}
74+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAEBudgetTokensFOTBlind}{162}
75+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBZ}{99}
76+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBZ}{97}
77+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBZ}{100}
78+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBZ}{215,449,381}
79+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBZ}{1,806}
80+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBZ}{189}
81+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBO}{99}
82+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBO}{97}
83+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBO}{100}
84+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBO}{215,486,273}
85+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBO}{1,661}
86+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBO}{214}
87+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsF}{0}
88+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsF}{0}
89+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsF}{0}
90+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsF}{60,635,200}
91+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsF}{1,794}
92+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsF}{747}
93+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsE}{99}
94+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsE}{97}
95+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsE}{100}
96+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsE}{114,502,809}
97+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsE}{1,281}
98+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsE}{180}
99+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTKeepFracZTF}{21}
100+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTKeepFracZTF}{18}
101+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTKeepFracZTF}{23}
102+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTKeepFracZTF}{53,874,493}
103+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTKeepFracZTF}{1,244}
104+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTKeepFracZTF}{126}
105+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTF}{0}
106+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTF}{0}
107+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTF}{0}
108+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTF}{53,874,488}
109+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTF}{1,735}
110+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTF}{678}
111+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFAware}{76}
112+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFAware}{70}
113+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFAware}{82}
114+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFAware}{53,874,532}
115+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFAware}{1,609}
116+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFAware}{140}
117+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFBlind}{12}
118+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFBlind}{7}
119+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFBlind}{18}
120+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFBlind}{53,874,533}
121+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFBlind}{1,387}
122+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFBlind}{106}
123+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBSKeepFracZTFAware}{65}
124+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBSKeepFracZTFAware}{59}
125+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBSKeepFracZTFAware}{71}
126+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBSKeepFracZTFAware}{53,874,422}
127+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBSKeepFracZTFAware}{1,842}
128+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBSKeepFracZTFAware}{544}
129+\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBNBitsEKeepFracZTFPolicyAttentionAware}{77}
130+\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBNBitsEKeepFracZTFPolicyAttentionAware}{71}
131+\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBNBitsEKeepFracZTFPolicyAttentionAware}{83}
132+\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBNBitsEKeepFracZTFPolicyAttentionAware}{28,646,537}
133+\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBNBitsEKeepFracZTFPolicyAttentionAware}{1,316}
134+\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBNBitsEKeepFracZTFPolicyAttentionAware}{148}
135+\newcommand{\DiffTO}{-21.8}
136+\newcommand{\LoTO}{-31.8}
137+\newcommand{\HiTO}{-12.4}
138+\newcommand{\PTO}{0.000}
139+\newcommand{\NTO}{170}
140+\newcommand{\DiffTT}{+0.0}
141+\newcommand{\LoTT}{+0.0}
142+\newcommand{\HiTT}{+0.0}
143+\newcommand{\PTT}{1.000}
144+\newcommand{\NTT}{165}
145+\newcommand{\DiffTT}{-98.8}
146+\newcommand{\LoTT}{-100.0}
147+\newcommand{\HiTT}{-97.0}
148+\newcommand{\PTT}{0.000}
149+\newcommand{\NTT}{165}
150+\newcommand{\DiffTF}{+55.8}
151+\newcommand{\LoTF}{+49.1}
152+\newcommand{\HiTF}{+62.4}
153+\newcommand{\PTF}{0.000}
154+\newcommand{\NTF}{165}
155+\newcommand{\DiffTS}{+0.0}
156+\newcommand{\LoTS}{+0.0}
157+\newcommand{\HiTS}{+0.0}
158+\newcommand{\PTS}{1.000}
159+\newcommand{\NTS}{170}
160+\newcommand{\EsixBONoneTZZZRt}{1,141}
161+\newcommand{\EsixBONoneTZZZBytes}{72}
162+\newcommand{\EsixBONoneTZZZSend}{583}
163+\newcommand{\EsixBONoneTZZZPrefill}{--}
164+\newcommand{\EsixBONoneTZZZLoad}{14}
165+\newcommand{\EsixBONoneTZZZGbps}{0.93}
166+\newcommand{\EsixBONoneEZZZRt}{2,667}
167+\newcommand{\EsixBONoneEZZZBytes}{215}
168+\newcommand{\EsixBONoneEZZZSend}{1,806}
169+\newcommand{\EsixBONoneEZZZPrefill}{--}
170+\newcommand{\EsixBONoneEZZZLoad}{40}
171+\newcommand{\EsixBONoneEZZZGbps}{0.92}
172+\newcommand{\EsixBONoneOSZZZRt}{4,453}
173+\newcommand{\EsixBONoneOSZZZBytes}{406}
174+\newcommand{\EsixBONoneOSZZZSend}{3,434}
175+\newcommand{\EsixBONoneOSZZZPrefill}{--}
176+\newcommand{\EsixBONoneOSZZZLoad}{73}
177+\newcommand{\EsixBONoneOSZZZGbps}{0.92}
178+\newcommand{\EsixBOZstdTZZZRt}{1,085}
179+\newcommand{\EsixBOZstdTZZZBytes}{72}
180+\newcommand{\EsixBOZstdTZZZSend}{444}
181+\newcommand{\EsixBOZstdTZZZPrefill}{--}
182+\newcommand{\EsixBOZstdTZZZLoad}{84}
183+\newcommand{\EsixBOZstdTZZZGbps}{0.92}
184+\newcommand{\EsixBOZstdEZZZRt}{2,424}
185+\newcommand{\EsixBOZstdEZZZBytes}{215}
186+\newcommand{\EsixBOZstdEZZZSend}{1,388}
187+\newcommand{\EsixBOZstdEZZZPrefill}{--}
188+\newcommand{\EsixBOZstdEZZZLoad}{241}
189+\newcommand{\EsixBOZstdEZZZGbps}{0.92}
190+\newcommand{\EsixBOZstdOSZZZRt}{3,968}
191+\newcommand{\EsixBOZstdOSZZZBytes}{406}
192+\newcommand{\EsixBOZstdOSZZZSend}{2,647}
193+\newcommand{\EsixBOZstdOSZZZPrefill}{--}
194+\newcommand{\EsixBOZstdOSZZZLoad}{428}
195+\newcommand{\EsixBOZstdOSZZZGbps}{0.92}
196+\newcommand{\EsixBTFNoneTZZZRt}{1,747}
197+\newcommand{\EsixBTFNoneTZZZBytes}{20}
198+\newcommand{\EsixBTFNoneTZZZSend}{139}
199+\newcommand{\EsixBTFNoneTZZZPrefill}{--}
200+\newcommand{\EsixBTFNoneTZZZLoad}{6}
201+\newcommand{\EsixBTFNoneTZZZGbps}{0.91}
202+\newcommand{\EsixBTFNoneEZZZRt}{2,231}
203+\newcommand{\EsixBTFNoneEZZZBytes}{61}
204+\newcommand{\EsixBTFNoneEZZZSend}{483}
205+\newcommand{\EsixBTFNoneEZZZPrefill}{--}
206+\newcommand{\EsixBTFNoneEZZZLoad}{15}
207+\newcommand{\EsixBTFNoneEZZZGbps}{0.91}
208+\newcommand{\EsixBTFNoneOSZZZRt}{2,873}
209+\newcommand{\EsixBTFNoneOSZZZBytes}{114}
210+\newcommand{\EsixBTFNoneOSZZZSend}{939}
211+\newcommand{\EsixBTFNoneOSZZZPrefill}{--}
212+\newcommand{\EsixBTFNoneOSZZZLoad}{24}
213+\newcommand{\EsixBTFNoneOSZZZGbps}{0.92}
214+\newcommand{\EsixBTENoneTZZZRt}{1,052}
215+\newcommand{\EsixBTENoneTZZZBytes}{38}
216+\newcommand{\EsixBTENoneTZZZSend}{293}
217+\newcommand{\EsixBTENoneTZZZPrefill}{--}
218+\newcommand{\EsixBTENoneTZZZLoad}{10}
219+\newcommand{\EsixBTENoneTZZZGbps}{0.91}
220+\newcommand{\EsixBTENoneEZZZRt}{1,676}
221+\newcommand{\EsixBTENoneEZZZBytes}{115}
222+\newcommand{\EsixBTENoneEZZZSend}{942}
223+\newcommand{\EsixBTENoneEZZZPrefill}{--}
224+\newcommand{\EsixBTENoneEZZZLoad}{25}
225+\newcommand{\EsixBTENoneEZZZGbps}{0.91}
226+\newcommand{\EsixBTENoneOSZZZRt}{2,682}
227+\newcommand{\EsixBTENoneOSZZZBytes}{216}
228+\newcommand{\EsixBTENoneOSZZZSend}{1,804}
229+\newcommand{\EsixBTENoneOSZZZPrefill}{--}
230+\newcommand{\EsixBTENoneOSZZZLoad}{43}
231+\newcommand{\EsixBTENoneOSZZZGbps}{0.91}
232+\newcommand{\EsixTextRemotePrefillNoneTZZZRt}{1,567}
233+\newcommand{\EsixTextRemotePrefillNoneTZZZBytes}{0}
234+\newcommand{\EsixTextRemotePrefillNoneTZZZSend}{0}
235+\newcommand{\EsixTextRemotePrefillNoneTZZZPrefill}{1,160}
236+\newcommand{\EsixTextRemotePrefillNoneTZZZLoad}{--}
237+\newcommand{\EsixTextRemotePrefillNoneTZZZGbps}{--}
238+\newcommand{\EsixTextRemotePrefillNoneEZZZRt}{3,667}
239+\newcommand{\EsixTextRemotePrefillNoneEZZZBytes}{0}
240+\newcommand{\EsixTextRemotePrefillNoneEZZZSend}{0}
241+\newcommand{\EsixTextRemotePrefillNoneEZZZPrefill}{3,241}
242+\newcommand{\EsixTextRemotePrefillNoneEZZZLoad}{--}
243+\newcommand{\EsixTextRemotePrefillNoneEZZZGbps}{--}
244+\newcommand{\EsixTextRemotePrefillNoneOSZZZRt}{7,553}
245+\newcommand{\EsixTextRemotePrefillNoneOSZZZBytes}{0}
246+\newcommand{\EsixTextRemotePrefillNoneOSZZZSend}{0}
247+\newcommand{\EsixTextRemotePrefillNoneOSZZZPrefill}{7,043}
248+\newcommand{\EsixTextRemotePrefillNoneOSZZZLoad}{--}
249+\newcommand{\EsixTextRemotePrefillNoneOSZZZGbps}{--}
250+\newcommand{\EsixLocalPrefillTZZZRt}{--}
251+\newcommand{\EsixLocalPrefillTZZZBytes}{--}
252+\newcommand{\EsixLocalPrefillTZZZSend}{--}
253+\newcommand{\EsixLocalPrefillTZZZPrefill}{944}
254+\newcommand{\EsixLocalPrefillTZZZLoad}{--}
255+\newcommand{\EsixLocalPrefillTZZZGbps}{--}
256+\newcommand{\EsixLocalPrefillEZZZRt}{--}
257+\newcommand{\EsixLocalPrefillEZZZBytes}{--}
258+\newcommand{\EsixLocalPrefillEZZZSend}{--}
259+\newcommand{\EsixLocalPrefillEZZZPrefill}{3,262}
260+\newcommand{\EsixLocalPrefillEZZZLoad}{--}
261+\newcommand{\EsixLocalPrefillEZZZGbps}{--}
262+\newcommand{\EsixLocalPrefillOSZZZRt}{--}
263+\newcommand{\EsixLocalPrefillOSZZZBytes}{--}
264+\newcommand{\EsixLocalPrefillOSZZZSend}{--}
265+\newcommand{\EsixLocalPrefillOSZZZPrefill}{6,912}
266+\newcommand{\EsixLocalPrefillOSZZZLoad}{--}
267+\newcommand{\EsixLocalPrefillOSZZZGbps}{--}
268+\newcommand{\protocolHash}{404c52b34d964693}
modified results/figures/fig2_quality_vs_bytes.pdf +0 −0

Binary file not shown.

modified results/figures/fig2_quality_vs_bytes.png +0 −0

Binary file not shown.

modified results/figures/fig3_quality_vs_latency_memory.pdf +0 −0

Binary file not shown.

modified results/figures/fig3_quality_vs_latency_memory.png +0 −0

Binary file not shown.

modified results/figures/fig4_latency_decomposition.pdf +0 −0

Binary file not shown.

modified results/figures/fig4_latency_decomposition.png +0 −0

Binary file not shown.

added results/figures/fig5_handoffs.pdf +0 −0

Binary file not shown.

added results/figures/fig5_handoffs.png +0 −0

Binary file not shown.

added results/figures/fig6_breakeven.pdf +0 −0

Binary file not shown.

added results/figures/fig6_breakeven.png +0 −0

Binary file not shown.

modified results/figures/figA_quality_vs_bytes_aware.pdf +0 −0

Binary file not shown.

modified results/figures/figA_quality_vs_bytes_aware.png +0 −0

Binary file not shown.

modified results/figures/figA_quality_vs_bytes_blind.pdf +0 −0

Binary file not shown.

modified results/figures/figA_quality_vs_bytes_blind.png +0 −0

Binary file not shown.

modified results/provenance.json +89 −33
@@ -1,122 +1,178 @@
1 { 1 {
2− "results/tables/E1_pilot_primary.csv": { 2+ "results/tables/E1_core_primary.csv": {
3 "sources": [ 3 "sources": [
4− "runs/raw/E1_pilot.jsonl" 4+ "runs/raw/E1_core.jsonl"
5 ], 5 ],
6 "script": "src/wp30/analysis/analyze.py", 6 "script": "src/wp30/analysis/analyze.py",
7− "generated_utc": "2026-10-30T04:16:26.754738Z", 7+ "generated_utc": "2026-10-30T05:17:32.110513Z",
8 "note": "per-condition quality (cluster bootstrap CI) and resource medians" 8 "note": "per-condition quality (cluster bootstrap CI) and resource medians"
9 }, 9 },
10− "results/tables/E1_pilot_failures.csv": { 10+ "results/tables/E1_core_failures.csv": {
11 "sources": [ 11 "sources": [
12− "runs/raw/E1_pilot.jsonl" 12+ "runs/raw/E1_core.jsonl"
13 ], 13 ],
14 "script": "src/wp30/analysis/analyze.py", 14 "script": "src/wp30/analysis/analyze.py",
15− "generated_utc": "2026-10-30T04:16:26.760771Z", 15+ "generated_utc": "2026-10-30T05:17:32.112093Z",
16 "note": "attempt status counts by method" 16 "note": "attempt status counts by method"
17 }, 17 },
18− "results/tables/E1_pilot_by_question_type.csv": { 18+ "results/tables/E1_core_by_question_type.csv": {
19 "sources": [ 19 "sources": [
20− "runs/raw/E1_pilot.jsonl" 20+ "runs/raw/E1_core.jsonl"
21 ], 21 ],
22 "script": "src/wp30/analysis/analyze.py", 22 "script": "src/wp30/analysis/analyze.py",
23− "generated_utc": "2026-10-30T04:16:27.015267Z", 23+ "generated_utc": "2026-10-30T05:17:32.219775Z",
24 "note": "synthetic accuracy by question type" 24 "note": "synthetic accuracy by question type"
25 }, 25 },
26− "results/tables/E1_pilot_primary_tests.csv": { 26+ "results/tables/E1_core_primary_tests.csv": {
27 "sources": [ 27 "sources": [
28− "runs/raw/E1_pilot.jsonl" 28+ "runs/raw/E1_core.jsonl"
29 ], 29 ],
30 "script": "src/wp30/analysis/analyze.py", 30 "script": "src/wp30/analysis/analyze.py",
31− "generated_utc": "2026-10-30T04:16:27.081235Z", 31+ "generated_utc": "2026-10-30T05:17:32.256966Z",
32 "note": "registered primary comparisons, paired, cluster bootstrap, Holm" 32 "note": "registered primary comparisons, paired, cluster bootstrap, Holm"
33 }, 33 },
34 "results/figures/fig2_quality_vs_bytes.pdf": { 34 "results/figures/fig2_quality_vs_bytes.pdf": {
35 "sources": [ 35 "sources": [
36− "runs/raw/E1_pilot.jsonl" 36+ "runs/raw/E1_core.jsonl"
37 ], 37 ],
38 "script": "src/wp30/analysis/analyze.py", 38 "script": "src/wp30/analysis/analyze.py",
39− "generated_utc": "2026-10-30T04:16:27.840452Z", 39+ "generated_utc": "2026-10-30T05:17:32.489153Z",
40 "note": "points = conditions; x = median payload bytes; y = mean quality" 40 "note": "points = conditions; x = median payload bytes; y = mean quality"
41 }, 41 },
42 "results/figures/fig2_quality_vs_bytes.png": { 42 "results/figures/fig2_quality_vs_bytes.png": {
43 "sources": [ 43 "sources": [
44− "runs/raw/E1_pilot.jsonl" 44+ "runs/raw/E1_core.jsonl"
45 ], 45 ],
46 "script": "src/wp30/analysis/analyze.py", 46 "script": "src/wp30/analysis/analyze.py",
47− "generated_utc": "2026-10-30T04:16:28.019470Z", 47+ "generated_utc": "2026-10-30T05:17:32.558579Z",
48 "note": "points = conditions; x = median payload bytes; y = mean quality" 48 "note": "points = conditions; x = median payload bytes; y = mean quality"
49 }, 49 },
50 "results/figures/figA_quality_vs_bytes_blind.pdf": { 50 "results/figures/figA_quality_vs_bytes_blind.pdf": {
51 "sources": [ 51 "sources": [
52− "runs/raw/E1_pilot.jsonl" 52+ "runs/raw/E1_core.jsonl"
53 ], 53 ],
54 "script": "src/wp30/analysis/analyze.py", 54 "script": "src/wp30/analysis/analyze.py",
55− "generated_utc": "2026-10-30T04:16:28.241874Z", 55+ "generated_utc": "2026-10-30T05:17:32.683315Z",
56 "note": "points = conditions; x = median payload bytes; y = mean quality" 56 "note": "points = conditions; x = median payload bytes; y = mean quality"
57 }, 57 },
58 "results/figures/figA_quality_vs_bytes_blind.png": { 58 "results/figures/figA_quality_vs_bytes_blind.png": {
59 "sources": [ 59 "sources": [
60− "runs/raw/E1_pilot.jsonl" 60+ "runs/raw/E1_core.jsonl"
61 ], 61 ],
62 "script": "src/wp30/analysis/analyze.py", 62 "script": "src/wp30/analysis/analyze.py",
63− "generated_utc": "2026-10-30T04:16:28.368816Z", 63+ "generated_utc": "2026-10-30T05:17:32.785959Z",
64 "note": "points = conditions; x = median payload bytes; y = mean quality" 64 "note": "points = conditions; x = median payload bytes; y = mean quality"
65 }, 65 },
66 "results/figures/figA_quality_vs_bytes_aware.pdf": { 66 "results/figures/figA_quality_vs_bytes_aware.pdf": {
67 "sources": [ 67 "sources": [
68− "runs/raw/E1_pilot.jsonl" 68+ "runs/raw/E1_core.jsonl"
69 ], 69 ],
70 "script": "src/wp30/analysis/analyze.py", 70 "script": "src/wp30/analysis/analyze.py",
71− "generated_utc": "2026-10-30T04:16:28.531968Z", 71+ "generated_utc": "2026-10-30T05:17:32.850894Z",
72 "note": "points = conditions; x = median payload bytes; y = mean quality" 72 "note": "points = conditions; x = median payload bytes; y = mean quality"
73 }, 73 },
74 "results/figures/figA_quality_vs_bytes_aware.png": { 74 "results/figures/figA_quality_vs_bytes_aware.png": {
75 "sources": [ 75 "sources": [
76− "runs/raw/E1_pilot.jsonl" 76+ "runs/raw/E1_core.jsonl"
77 ], 77 ],
78 "script": "src/wp30/analysis/analyze.py", 78 "script": "src/wp30/analysis/analyze.py",
79− "generated_utc": "2026-10-30T04:16:28.682029Z", 79+ "generated_utc": "2026-10-30T05:17:32.901784Z",
80 "note": "points = conditions; x = median payload bytes; y = mean quality" 80 "note": "points = conditions; x = median payload bytes; y = mean quality"
81 }, 81 },
82 "results/figures/fig3_quality_vs_latency_memory.pdf": { 82 "results/figures/fig3_quality_vs_latency_memory.pdf": {
83 "sources": [ 83 "sources": [
84− "runs/raw/E1_pilot.jsonl" 84+ "runs/raw/E1_core.jsonl"
85 ], 85 ],
86 "script": "src/wp30/analysis/analyze.py", 86 "script": "src/wp30/analysis/analyze.py",
87− "generated_utc": "2026-10-30T04:16:28.790718Z", 87+ "generated_utc": "2026-10-30T05:17:32.984034Z",
88 "note": "" 88 "note": ""
89 }, 89 },
90 "results/figures/fig3_quality_vs_latency_memory.png": { 90 "results/figures/fig3_quality_vs_latency_memory.png": {
91 "sources": [ 91 "sources": [
92− "runs/raw/E1_pilot.jsonl" 92+ "runs/raw/E1_core.jsonl"
93 ], 93 ],
94 "script": "src/wp30/analysis/analyze.py", 94 "script": "src/wp30/analysis/analyze.py",
95− "generated_utc": "2026-10-30T04:16:28.864644Z", 95+ "generated_utc": "2026-10-30T05:17:33.041107Z",
96 "note": "" 96 "note": ""
97 }, 97 },
98 "results/figures/fig4_latency_decomposition.pdf": { 98 "results/figures/fig4_latency_decomposition.pdf": {
99 "sources": [ 99 "sources": [
100− "runs/raw/E1_pilot.jsonl" 100+ "runs/raw/E1_core.jsonl"
101 ], 101 ],
102 "script": "src/wp30/analysis/analyze.py", 102 "script": "src/wp30/analysis/analyze.py",
103− "generated_utc": "2026-10-30T04:16:28.935399Z", 103+ "generated_utc": "2026-10-30T05:17:33.111092Z",
104 "note": "" 104 "note": ""
105 }, 105 },
106 "results/figures/fig4_latency_decomposition.png": { 106 "results/figures/fig4_latency_decomposition.png": {
107 "sources": [ 107 "sources": [
108− "runs/raw/E1_pilot.jsonl" 108+ "runs/raw/E1_core.jsonl"
109 ], 109 ],
110 "script": "src/wp30/analysis/analyze.py", 110 "script": "src/wp30/analysis/analyze.py",
111− "generated_utc": "2026-10-30T04:16:28.986377Z", 111+ "generated_utc": "2026-10-30T05:17:33.154858Z",
112 "note": "" 112 "note": ""
113 }, 113 },
114+ "results/figures/fig5_handoffs.pdf": {
115+ "sources": [
116+ "runs/raw/E4_pilot.jsonl"
117+ ],
118+ "script": "src/wp30/analysis/analyze.py",
119+ "generated_utc": "2026-10-30T05:17:34.567038Z",
120+ "note": "means over documents; chains defined in configs/chains*.yaml"
121+ },
122+ "results/figures/fig5_handoffs.png": {
123+ "sources": [
124+ "runs/raw/E4_pilot.jsonl"
125+ ],
126+ "script": "src/wp30/analysis/analyze.py",
127+ "generated_utc": "2026-10-30T05:17:34.616672Z",
128+ "note": "means over documents; chains defined in configs/chains*.yaml"
129+ },
130+ "results/tables/E4_pilot_handoffs.csv": {
131+ "sources": [
132+ "runs/raw/E4_pilot.jsonl"
133+ ],
134+ "script": "src/wp30/analysis/analyze.py",
135+ "generated_utc": "2026-10-30T05:17:34.663073Z",
136+ "note": "repeated handoffs"
137+ },
138+ "results/tables/E4_pilot_h4_primary_tests.csv": {
139+ "sources": [
140+ "runs/raw/E4_pilot.jsonl"
141+ ],
142+ "script": "src/wp30/analysis/analyze.py",
143+ "generated_utc": "2026-10-30T05:17:34.672312Z",
144+ "note": "registered primary comparisons, paired, cluster bootstrap, Holm"
145+ },
146+ "results/figures/fig6_breakeven.pdf": {
147+ "sources": [
148+ "runs/raw/E6_core.jsonl"
149+ ],
150+ "script": "src/wp30/analysis/analyze.py",
151+ "generated_utc": "2026-10-30T05:17:34.765577Z",
152+ "note": "left = measured (medians, p25-p75); right = analytic model from measured bandwidth and prefill slope; dashed 10 Gb/s is hypothetical"
153+ },
154+ "results/figures/fig6_breakeven.png": {
155+ "sources": [
156+ "runs/raw/E6_core.jsonl"
157+ ],
158+ "script": "src/wp30/analysis/analyze.py",
159+ "generated_utc": "2026-10-30T05:17:34.841447Z",
160+ "note": "left = measured (medians, p25-p75); right = analytic model from measured bandwidth and prefill slope; dashed 10 Gb/s is hypothetical"
161+ },
162+ "results/tables/E6_network.csv": {
163+ "sources": [
164+ "runs/raw/E6_core.jsonl"
165+ ],
166+ "script": "src/wp30/analysis/analyze.py",
167+ "generated_utc": "2026-10-30T05:17:34.846667Z",
168+ "note": "two-node transfer measurements"
169+ },
114 "results/tables/ED_hosted.csv": { 170 "results/tables/ED_hosted.csv": {
115 "sources": [ 171 "sources": [
116 "runs/raw/ED_hosted_pilot.jsonl" 172 "runs/raw/ED_hosted_pilot.jsonl"
117 ], 173 ],
118 "script": "src/wp30/analysis/analyze.py", 174 "script": "src/wp30/analysis/analyze.py",
119− "generated_utc": "2026-10-30T04:16:29.006067Z", 175+ "generated_utc": "2026-10-30T05:17:34.897707Z",
120 "note": "hosted receivers; cost estimated from provider price tables" 176 "note": "hosted receivers; cost estimated from provider price tables"
121 } 177 }
122 } 178 }
\ No newline at end of file
added results/tables/E1_core_by_question_type.csv +65 −0
@@ -0,0 +1,65 @@
1+condition,qtype,quality,lo,hi,n
2+A0,exact_copy,0.0,0.0,0.0,70
3+A0,multi_hop,0.0,0.0,0.0,35
4+A0,single_fact,0.0,0.0,0.0,70
5+A1,exact_copy,1.0,1.0,1.0,70
6+A1,multi_hop,0.9428571428571428,0.8571428571428571,1.0,35
7+A1,single_fact,1.0,1.0,1.0,70
8+A2|budget_tokens=512,exact_copy,0.14285714285714285,0.07142857142857142,0.22857142857142856,70
9+A2|budget_tokens=512,multi_hop,0.2,0.08571428571428572,0.34285714285714286,35
10+A2|budget_tokens=512,single_fact,0.2571428571428571,0.17142857142857143,0.34285714285714286,70
11+A3|budget_tokens=512|aware,exact_copy,1.0,1.0,1.0,70
12+A3|budget_tokens=512|aware,multi_hop,0.8571428571428571,0.7428571428571429,0.9714285714285714,35
13+A3|budget_tokens=512|aware,single_fact,1.0,1.0,1.0,70
14+A3|budget_tokens=512|blind,exact_copy,0.5,0.5,0.5,70
15+A3|budget_tokens=512|blind,multi_hop,0.8857142857142857,0.7850000000000004,0.9714285714285714,35
16+A3|budget_tokens=512|blind,single_fact,0.0,0.0,0.0,70
17+A4|budget_tokens=512|aware,exact_copy,0.8571428571428571,0.7142857142857143,0.9714285714285714,35
18+A4|budget_tokens=512|aware,single_fact,0.9857142857142858,0.9571428571428572,1.0,70
19+A4|budget_tokens=512|blind,exact_copy,0.4714285714285714,0.35714285714285715,0.6,70
20+A4|budget_tokens=512|blind,multi_hop,0.2857142857142857,0.14285714285714285,0.42857142857142855,35
21+A4|budget_tokens=512|blind,single_fact,0.3142857142857143,0.18571428571428572,0.44285714285714284,70
22+A5|budget_tokens=512|aware,exact_copy,1.0,1.0,1.0,34
23+A5|budget_tokens=512|aware,single_fact,1.0,1.0,1.0,68
24+A5|budget_tokens=512|blind,exact_copy,0.14705882352941177,0.07352941176470588,0.22058823529411764,68
25+A5|budget_tokens=512|blind,multi_hop,0.14705882352941177,0.029411764705882353,0.2647058823529412,34
26+A5|budget_tokens=512|blind,single_fact,0.17647058823529413,0.10294117647058823,0.25,68
27+A6|budget_tokens=512|max_fetch_bytes=2000|max_lookups=6|blind,exact_copy,0.1323529411764706,0.07352941176470588,0.22058823529411764,68
28+A6|budget_tokens=512|max_fetch_bytes=2000|max_lookups=6|blind,multi_hop,0.14705882352941177,0.029411764705882353,0.2647058823529412,34
29+A6|budget_tokens=512|max_fetch_bytes=2000|max_lookups=6|blind,single_fact,0.22058823529411764,0.1323529411764706,0.3235294117647059,68
30+A7|codec=zstd,exact_copy,1.0,1.0,1.0,68
31+A7|codec=zstd,multi_hop,0.9411764705882353,0.8529411764705882,1.0,34
32+A7|codec=zstd,single_fact,1.0,1.0,1.0,68
33+A8|budget_tokens=512|blind,exact_copy,0.10294117647058823,0.04411764705882353,0.17647058823529413,68
34+A8|budget_tokens=512|blind,multi_hop,0.0,0.0,0.0,34
35+A8|budget_tokens=512|blind,single_fact,0.22058823529411764,0.1323529411764706,0.3312499999999997,68
36+B0,exact_copy,1.0,1.0,1.0,68
37+B0,multi_hop,0.9411764705882353,0.8529411764705882,1.0,34
38+B0,single_fact,1.0,1.0,1.0,68
39+B1,exact_copy,1.0,1.0,1.0,68
40+B1,multi_hop,0.9411764705882353,0.8529411764705882,1.0,34
41+B1,single_fact,1.0,1.0,1.0,68
42+B2|bits=4,exact_copy,0.0,0.0,0.0,68
43+B2|bits=4,multi_hop,0.0,0.0,0.0,34
44+B2|bits=4,single_fact,0.0,0.0,0.0,68
45+B2|bits=8,exact_copy,1.0,1.0,1.0,68
46+B2|bits=8,multi_hop,0.9411764705882353,0.8529411764705882,1.0,34
47+B2|bits=8,single_fact,1.0,1.0,1.0,68
48+B3|keep_frac=0.25,exact_copy,0.3235294117647059,0.23529411764705882,0.40477941176470555,68
49+B3|keep_frac=0.25,multi_hop,0.058823529411764705,0.0,0.14705882352941177,34
50+B3|keep_frac=0.25,single_fact,0.14705882352941177,0.07352941176470588,0.23529411764705882,68
51+B4|keep_frac=0.25,exact_copy,0.0,0.0,0.0,68
52+B4|keep_frac=0.25,multi_hop,0.0,0.0,0.0,34
53+B4|keep_frac=0.25,single_fact,0.0,0.0,0.0,68
54+B5|keep_frac=0.25|aware,exact_copy,0.9411764705882353,0.8893382352941178,0.9852941176470589,68
55+B5|keep_frac=0.25|aware,multi_hop,0.5882352941176471,0.4411764705882353,0.7352941176470589,34
56+B5|keep_frac=0.25|aware,single_fact,0.6617647058823529,0.5882352941176471,0.7352941176470589,68
57+B5|keep_frac=0.25|blind,exact_copy,0.20588235294117646,0.11764705882352941,0.29411764705882354,68
58+B5|keep_frac=0.25|blind,multi_hop,0.058823529411764705,0.0,0.14705882352941177,34
59+B5|keep_frac=0.25|blind,single_fact,0.058823529411764705,0.014705882352941176,0.11764705882352941,68
60+B6|keep_frac=0.25|aware,exact_copy,0.8529411764705882,0.7794117647058824,0.9264705882352942,68
61+B6|keep_frac=0.25|aware,multi_hop,0.5294117647058824,0.38235294117647056,0.6764705882352942,34
62+B6|keep_frac=0.25|aware,single_fact,0.5147058823529411,0.5,0.5441176470588235,68
63+B9|bits=8|keep_frac=0.25|policy=attention|aware,exact_copy,0.9411764705882353,0.8893382352941178,0.9852941176470589,68
64+B9|bits=8|keep_frac=0.25|policy=attention|aware,multi_hop,0.6176470588235294,0.4411764705882353,0.7801470588235286,34
65+B9|bits=8|keep_frac=0.25|policy=attention|aware,single_fact,0.6617647058823529,0.5882352941176471,0.7352941176470589,68
added results/tables/E1_core_failures.csv +18 −0
@@ -0,0 +1,18 @@
1+method_id,invalid_output,success
2+A0,0,175
3+A1,0,175
4+A2,0,175
5+A3,0,350
6+A4,0,280
7+A5,0,272
8+A6,0,170
9+A7,0,170
10+A8,0,170
11+B0,0,170
12+B1,0,170
13+B2,13,327
14+B3,0,170
15+B4,12,158
16+B5,0,340
17+B6,0,170
18+B9,0,170
added results/tables/E1_core_primary.csv +23 −0
@@ -0,0 +1,23 @@
1+family,pair,condition,method,access,n,n_docs,quality,q_lo,q_hi,success_rate,payload_bytes_median,cache_bytes_median,cold_ms_p50,warm_ms_p50,amortized_ms_p50,ttft_ms_p50,peak_decode_mem_median,fetched_bytes_mean
2+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A0,A0,none,175,35,0.0,0.0,0.0,1.0,0.0,,50.39150104857981,50.38283404428512,50.38456744514406,13.24266695883125,6243951240.0,0.0
3+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A1,A1,none,175,35,0.9885714285714285,0.9714285714285714,1.0,1.0,29162.0,,970.1906249392778,970.1825840165839,970.1841922011226,14.666083035990596,7155474496.0,0.0
4+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A2|budget_tokens=512,A2,none,175,35,0.2,0.16,0.24,1.0,2246.0,,111.56629212200642,107.08154016174376,107.87064016330987,13.610875001177192,6567653464.0,0.0
5+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A3|budget_tokens=512|aware,A3,aware,175,35,0.9714285714285714,0.9485714285714286,0.9942857142857143,1.0,1365.0,,208.41120788827538,207.17641792725772,208.41120788827538,13.562750071287155,6562246744.0,0.0
6+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A3|budget_tokens=512|blind,A3,blind,175,35,0.37714285714285717,0.35428571428571426,0.3942857142857143,1.0,1246.0,,157.03170804772526,152.2996670100838,153.21449297480285,13.590875081717968,6562246744.0,0.0
7+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A4|budget_tokens=512|aware,A4,aware,105,35,0.9428571428571428,0.8857142857142857,0.9904761904761905,1.0,380.0,,3052.2540010279045,189.48554201051593,3052.2540010279045,13.359250035136938,6346302976.0,0.0
8+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A4|budget_tokens=512|blind,A4,blind,175,35,0.37142857142857144,0.28,0.4685714285714286,1.0,1440.0,,7431.057667010464,129.77629096712917,1631.4786000177264,13.581125065684319,6567358552.0,0.0
9+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A5|budget_tokens=512|aware,A5,aware,102,34,1.0,1.0,1.0,1.0,118.0,,2194.973748992197,178.98147902451456,2194.973748992197,13.401583069935441,6300656316.0,0.0
10+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A5|budget_tokens=512|blind,A5,blind,170,34,0.1588235294117647,0.12941176470588237,0.18235294117647058,1.0,828.0,,8171.243209391832,84.63735459372401,1730.138829536736,13.505270995665342,6529378068.0,0.0
11+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A6|budget_tokens=512|max_fetch_bytes=2000|max_lookups=6|blind,A6,blind,170,34,0.17058823529411765,0.13529411764705881,0.20588235294117646,1.0,636.5,,176.3136875233613,176.2877079891041,176.29290389595553,13.919333054218441,6808580172.0,1472.0529411764705
12+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A7|codec=zstd,A7,none,170,34,0.9882352941176471,0.9705882352941176,1.0,1.0,2725.0,,1041.768479917664,1041.6947924531996,1041.7095299460925,14.970291522331536,7155474496.0,0.0
13+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A8|budget_tokens=512|blind,A8,blind,170,34,0.12941176470588237,0.08235294117647059,0.18235294117647058,1.0,2571.0,,1328.1424794113263,162.30137506499887,366.3930131937377,13.771728496067226,6604304472.0,0.0
14+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B0,B0,blind,170,34,0.9882352941176471,0.9705882352941176,1.0,1.0,215449381.0,215433216.0,1806.011937558651,189.0929365181364,505.7045995024964,47.34012502012774,6901113148.0,0.0
15+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B1,B1,blind,170,34,0.9882352941176471,0.9705882352941176,1.0,1.0,215449409.0,215433216.0,1506.5432714764029,211.0927085741423,486.3688698387705,47.08545847097412,6901113148.0,0.0
16+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B2|bits=4,B2,blind,170,34,0.0,0.0,0.0,0.9235294117647059,60624832.0,60590592.0,1775.7291264715604,737.2901674243622,982.5587245519273,27.91222953237593,6614715196.0,0.0
17+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B2|bits=8,B2,blind,170,34,0.9882352941176471,0.9705882352941176,1.0,1.0,114483225.0,114448896.0,1257.6271875877865,180.21425051847473,405.6984704802744,30.57241643546149,6734196444.0,0.0
18+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B3|keep_frac=0.25,B3,blind,170,34,0.2,0.17058823529411765,0.22941176470588234,1.0,53874493.0,53858304.0,1208.6009380291216,124.03164606075734,359.70184119651094,24.07675050199032,6546047206.0,0.0
19+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B4|keep_frac=0.25,B4,blind,170,34,0.0,0.0,0.0,0.9294117647058824,53874487.5,53858304.0,1713.8423139113002,677.3689165711403,910.9542755642906,24.169479496777058,6546047206.0,0.0
20+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B5|keep_frac=0.25|aware,B5,aware,170,34,0.7588235294117647,0.6941176470588235,0.8176470588235294,1.0,53874532.0,53858304.0,1311.0786880133674,138.59477097867057,1311.0786880133674,23.73120852280408,6545719524.0,0.0
21+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B5|keep_frac=0.25|blind,B5,blind,170,34,0.11764705882352941,0.06470588235294118,0.17647058823529413,1.0,53874532.5,53858304.0,1312.3200226109475,102.40249993512407,356.17916370974854,23.995895986445248,6545719524.0,0.0
22+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B6|keep_frac=0.25|aware,B6,aware,170,34,0.6529411764705882,0.5941176470588235,0.711764705882353,1.0,53874422.0,53858304.0,1684.0188961941749,542.1115205390379,1684.0188961941749,419.2666039452888,6584735832.0,0.0
23+synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B9|bits=8|keep_frac=0.25|policy=attention|aware,B9,aware,170,34,0.7647058823529411,0.7058823529411765,0.8235294117647058,1.0,28646537.0,28612224.0,1303.7285626051016,144.63060395792127,1303.7285626051016,21.587333467323333,6506365736.0,0.0
added results/tables/E1_core_primary_tests.csv +6 −0
@@ -0,0 +1,6 @@
1+a,b,diff,lo,hi,p,n,n_clusters,mean_a,mean_b,rel_change,name,hypothesis,p_holm
2+A5|budget_tokens=512|blind,A4|budget_tokens=512|blind,-0.21764705882352942,-0.3176470588235294,-0.12352941176470589,0.0,170,34,0.1588235294117647,0.3764705882352941,-0.578125,"T1 A5 vs A4 (512, blind, synthetic)",RQ5/H1,0.0
3+B2|bits=8,B0,0.0,0.0,0.0,1.0,170,34,0.9882352941176471,0.9882352941176471,0.0,T2 B2-8bit vs B0,H2,1.0
4+B2|bits=4,B0,-0.9882352941176471,-1.0,-0.9705882352941176,0.0,170,34,0.0,0.9882352941176471,-1.0,T3 B2-4bit vs B0,H2,0.0
5+B5|keep_frac=0.25|aware,B3|keep_frac=0.25,0.5588235294117647,0.48823529411764705,0.6235294117647059,0.0,170,34,0.7588235294117647,0.2,2.7941176470588234,T4 B5-aware vs B3 (1/4),H3,0.0
6+A1,B0,0.0,0.0,0.0,1.0,170,34,0.9882352941176471,0.9882352941176471,0.0,"T6 A1 vs B0 (text vs native cache, same content)",consistency,1.0
added results/tables/E4_pilot_h4_primary_tests.csv +2 −0
@@ -0,0 +1,2 @@
1+a,b,diff,lo,hi,p,n,n_clusters,mean_a,mean_b,rel_change,name,hypothesis,p_holm
2+A5|closed|qwen3b,A4|closed|qwen3b,-0.15555555555555556,-0.26666666666666666,-0.044166666666668186,0.016,90,6,0.23333333333333334,0.3888888888888889,-0.4,T5 A5 chain vs A4 chain at 4 handoffs (closed),H5,0.016
added results/tables/E4_pilot_handoffs.csv +13 −0
@@ -0,0 +1,13 @@
1+chain,handoffs,accuracy,acc_lo,acc_hi,retention,ret_lo,ret_hi,unsupported_mean,fetched_bytes_mean,msg_bytes_mean,n,n_docs
2+A4|closed|qwen3b,1,0.35555555555555557,0.23861111111111122,0.45555555555555555,0.3411427188592271,0.24802696029262292,0.41230561174591024,0.0,0.0,1496.1666666666667,90,6
3+A4|closed|qwen3b,2,0.3888888888888889,0.26666666666666666,0.5,0.3411427188592271,0.24802696029262292,0.41230561174591024,0.0,0.0,1319.6666666666667,90,6
4+A4|closed|qwen3b,4,0.3888888888888889,0.26666666666666666,0.5,0.3411427188592271,0.24802696029262292,0.41230561174591024,0.0,0.0,1308.0,90,6
5+A4|repairable|qwen3b,1,0.35555555555555557,0.23861111111111122,0.45555555555555555,0.3411427188592271,0.24802696029262292,0.41230561174591024,0.0,0.0,1496.1666666666667,90,6
6+A4|repairable|qwen3b,2,0.37777777777777777,0.25555555555555554,0.4777777777777778,0.35717367518869075,0.25722446236559143,0.4332625448763508,0.0,841.0,1330.5,90,6
7+A4|repairable|qwen3b,4,0.4,0.28888888888888886,0.4777777777777778,0.3623820085220241,0.2618648870450786,0.4332625448763508,0.0,2523.0,1221.1666666666667,90,6
8+A5|closed|qwen3b,1,0.24444444444444444,0.2111111111111111,0.28888888888888886,0.18358831023356179,0.13623383339319786,0.22507474646000766,0.0,0.0,824.3333333333334,90,6
9+A5|closed|qwen3b,2,0.23333333333333334,0.2,0.2777777777777778,0.18867645125920282,0.14238196167631653,0.2268202257941064,0.0,0.0,799.5,90,6
10+A5|closed|qwen3b,4,0.23333333333333334,0.2,0.2777777777777778,0.18867645125920282,0.14238196167631653,0.2268202257941064,0.0,0.0,799.5,90,6
11+A5|repairable|qwen3b,1,0.24444444444444444,0.2111111111111111,0.28888888888888886,0.18358831023356179,0.13623383339319786,0.22507474646000766,0.0,0.0,824.3333333333334,90,6
12+A5|repairable|qwen3b,2,0.24444444444444444,0.2111111111111111,0.28888888888888886,0.20996848061574833,0.1863903767453057,0.23009174427959406,0.0,720.5,819.5,90,6
13+A5|repairable|qwen3b,4,0.24444444444444444,0.2111111111111111,0.28888888888888886,0.20996848061574833,0.1863903767453057,0.23009174427959406,0.0,1787.5,819.5,90,6
added results/tables/E6_network.csv +19 −0
@@ -0,0 +1,19 @@
1+label,method,bits,codec,target_tokens,n,bytes,roundtrip_ms_p50,send_ms_p50,remote_load_ms_p50,remote_ttft_p50,remote_prefill_p50,eff_gbps,correct,local_prefill_p50,local_ttft_p50,remote_decode_p50
2+lan,B1,,none,2000,18,71974604.0,1141.0682704881765,582.8466879902408,14.371166529599577,137.31991651002318,,0.9254324526074698,1.0,,,355.0059790140949
3+lan,B1,,none,8000,18,215486247.0,2667.018833453767,1805.5645630229264,39.85322843072936,196.9017084920779,,0.9235836467576799,1.0,,,461.4051670068875
4+lan,B1,,none,16000,18,405815405.0,4453.494437446352,3433.713915932458,72.80125049874187,238.5944375419058,,0.9209816171104153,1.0,,,602.7200620155782
5+lan,B1,,zstd,2000,18,71974604.0,1084.9161460064352,444.3944585509598,83.68525002151728,139.11475002532825,,0.9150994733434414,1.0,,,365.122249990236
6+lan,B1,,zstd,8000,18,215486247.0,2424.4880829937756,1388.2532505085692,240.60777097474784,193.3201244683005,,0.915824857333317,1.0,,,446.31345791276544
7+lan,B1,,zstd,16000,18,405815405.0,3967.637834022753,2646.7615000437945,427.9795209877193,240.90118752792478,,0.9184908360866041,1.0,,,521.4174375287257
8+lan,B2,4.0,none,2000,18,20272367.0,1747.0715835224837,138.67695850785822,6.4079585135914385,121.12035450991243,,0.9136673786435414,0.0,,,1438.9247084618546
9+lan,B2,4.0,none,8000,18,60635179.0,2231.101604003925,482.8348124283366,14.84154158970341,136.53143751434982,,0.91348579263575,0.0,,,1519.0357704996131
10+lan,B2,4.0,none,16000,18,114165828.0,2872.7289164671674,939.1097919433378,24.307999468874186,167.80024947365746,,0.9167853410304264,0.0,,,1636.0245210235007
11+lan,B2,8.0,none,2000,18,38262137.0,1052.2061869851314,293.04660455090925,10.100167011842132,119.31358353467658,,0.913827491903402,1.0,,,575.5362290074117
12+lan,B2,8.0,none,8000,18,114502788.0,1676.0850210557692,941.5643539978191,25.120604026596993,141.16060349624604,,0.9142826777406476,1.0,,,492.0222085202113
13+lan,B2,8.0,none,16000,18,215615724.0,2681.69347953517,1804.4933750061318,42.92881244327873,164.5570209948346,,0.9145833581075375,1.0,,,511.545687506441
14+lan,text_remote_prefill,,none,2000,18,7259.5,1567.0107079204172,0.029500515665858984,,46.87483300222084,1160.1554374792613,,1.0,,,361.99316702550277
15+lan,text_remote_prefill,,none,8000,18,29286.0,3666.9509790372103,0.032729003578424454,,43.280771002173424,3241.15325009916,,1.0,,,389.8708130000159
16+lan,text_remote_prefill,,none,16000,18,58842.5,7552.801541460212,0.03768753958866,,42.491624946706,7043.001604033634,,1.0,,,441.7973959352821
17+local,local_prefill,,,2000,18,,,,,,,,1.0,943.6670205323026,127.3181255091913,
18+local,local_prefill,,,8000,18,,,,,,,,1.0,3261.6252915468067,192.03377101803198,
19+local,local_prefill,,,16000,18,,,,,,,,1.0,6912.2674585087225,258.25952051673084,
modified results/tables/ED_hosted.csv +30 −8
@@ -1,9 +1,31 @@
1 family,provider_model,condition,quality,lo,hi,n,cost_usd_mean,ttft_ms_p50,total_ms_p50,success_rate 1 family,provider_model,condition,quality,lo,hi,n,cost_usd_mean,ttft_ms_p50,total_ms_p50,success_rate
2−synthetic,anthropic,A0|anthropic|,0.0,0.0,0.0,5,,,,0.0 2+exec,claude-haiku-4-5-20251001,A0|anthropic|,0.0,0.0,0.0,2,0.00045200000000000004,460.55416646413505,1190.856728469953,1.0
3−synthetic,anthropic,A1|anthropic|,0.0,0.0,0.0,5,,,,0.0 3+exec,claude-haiku-4-5-20251001,A1|anthropic|,1.0,1.0,1.0,2,0.0027229999999999997,437.0991045143455,1361.941104521975,1.0
4−synthetic,anthropic,A1|anthropic|provider_cache=True,0.0,0.0,0.0,5,,,,0.0 4+exec,claude-haiku-4-5-20251001,A1|anthropic|provider_cache=True,1.0,1.0,1.0,2,0.002728,408.70312548941,1359.8948334692977,1.0
5−synthetic,anthropic,A4|anthropic|budget_tokens=512,0.0,0.0,0.0,5,,,,0.0 5+exec,claude-haiku-4-5-20251001,A4|anthropic|budget_tokens=512,0.5,0.0,1.0,2,0.0009205,416.973645449616,13922.582250030246,1.0
6−synthetic,anthropic,A5|anthropic|budget_tokens=512,0.0,0.0,0.0,5,,,,0.0 6+exec,claude-haiku-4-5-20251001,A5|anthropic|budget_tokens=512,0.5,0.0,1.0,2,0.0011785,406.2307709828019,11177.729208953679,1.0
7−synthetic,gpt-5-nano-2025-08-07,A0|openai|,0.0,0.0,0.0,5,1.0850000000000001e-05,,1124.6261249762028,1.0 7+exec,gpt-5-nano-2025-08-07,A0|openai|,0.0,0.0,0.0,2,8.25e-05,,2476.5294790267944,1.0
8−synthetic,gpt-5-nano-2025-08-07,A1|openai|,1.0,1.0,1.0,5,8.5524e-05,,982.3185830609873,1.0 8+exec,gpt-5-nano-2025-08-07,A1|openai|,1.0,1.0,1.0,2,0.000168625,,3379.9536040751263,1.0
9−synthetic,gpt-5-nano-2025-08-07,A1|openai|provider_cache=True,1.0,1.0,1.0,5,2.562e-05,,814.3011669162661,1.0 9+exec,gpt-5-nano-2025-08-07,A1|openai|provider_cache=True,1.0,1.0,1.0,2,0.000165625,,2359.246875043027,1.0
10+exec,gpt-5-nano-2025-08-07,A4|openai|budget_tokens=512,0.5,0.0,1.0,2,0.000109025,,14741.153624490835,1.0
11+exec,gpt-5-nano-2025-08-07,A5|openai|budget_tokens=512,0.5,0.0,1.0,2,9.980000000000001e-05,,12000.74629142182,1.0
12+public_qasper,claude-haiku-4-5-20251001,A0|anthropic|,0.0,0.0,0.0,2,8.099999999999999e-05,429.7979999682866,462.2769169509411,1.0
13+public_qasper,claude-haiku-4-5-20251001,A1|anthropic|,0.2670454545454546,0.125,0.4090909090909091,2,0.0043159999999999995,506.47239544196054,829.3312710011378,1.0
14+public_qasper,claude-haiku-4-5-20251001,A1|anthropic|provider_cache=True,0.2988235294117647,0.11764705882352941,0.48,2,0.004982375,508.6571045103483,902.2489790222608,1.0
15+public_qasper,claude-haiku-4-5-20251001,A4|anthropic|budget_tokens=512,0.0,0.0,0.0,2,0.0005555,403.8986465311609,14453.188042039983,1.0
16+public_qasper,claude-haiku-4-5-20251001,A5|anthropic|budget_tokens=512,0.0,0.0,0.0,2,0.000359,651.0722080129199,21799.3393955403,1.0
17+public_qasper,gpt-5-nano-2025-08-07,A0|openai|,0.13636363636363635,0.0,0.2727272727272727,2,1.34e-05,,1065.4002914670855,1.0
18+public_qasper,gpt-5-nano-2025-08-07,A1|openai|,0.2608695652173913,0.0,0.5217391304347826,2,0.00020015,,1249.2346040089615,1.0
19+public_qasper,gpt-5-nano-2025-08-07,A1|openai|provider_cache=True,0.0,0.0,0.0,2,3.371e-05,,1359.2887704726309,1.0
20+public_qasper,gpt-5-nano-2025-08-07,A4|openai|budget_tokens=512,0.0,0.0,0.0,2,2.595e-05,,15146.043936954811,1.0
21+public_qasper,gpt-5-nano-2025-08-07,A5|openai|budget_tokens=512,0.0,0.0,0.0,2,2.4375000000000003e-05,,22089.14725051727,1.0
22+synthetic,claude-haiku-4-5-20251001,A0|anthropic|,0.0,0.0,0.0,10,8.450000000000001e-05,731.1436460004188,764.0473959618248,1.0
23+synthetic,claude-haiku-4-5-20251001,A1|anthropic|,1.0,1.0,1.0,10,0.0041795,909.5747084938921,1024.321937467903,1.0
24+synthetic,claude-haiku-4-5-20251001,A1|anthropic|provider_cache=True,1.0,1.0,1.0,10,0.0021281299999999998,633.088978996966,688.6951045016758,1.0
25+synthetic,claude-haiku-4-5-20251001,A4|anthropic|budget_tokens=512,0.3,0.2,0.4,10,0.000641,665.2437084703706,41555.6893760222,1.0
26+synthetic,claude-haiku-4-5-20251001,A5|anthropic|budget_tokens=512,0.1,0.0,0.2,10,0.00043750000000000006,1539.5217915065587,38661.886979069095,1.0
27+synthetic,gpt-5-nano-2025-08-07,A0|openai|,0.0,0.0,0.0,10,1.0835000000000002e-05,,1841.3237500353716,1.0
28+synthetic,gpt-5-nano-2025-08-07,A1|openai|,1.0,1.0,1.0,10,8.901400000000001e-05,,1116.5167084545828,1.0
29+synthetic,gpt-5-nano-2025-08-07,A1|openai|provider_cache=True,1.0,1.0,1.0,10,3.487e-05,,1043.0729584768414,1.0
30+synthetic,gpt-5-nano-2025-08-07,A4|openai|budget_tokens=512,0.3,0.2,0.4,10,3.4145000000000006e-05,,37726.64497885853,1.0
31+synthetic,gpt-5-nano-2025-08-07,A5|openai|budget_tokens=512,0.1,0.0,0.2,10,2.7650000000000005e-05,,26799.245104019064,1.0
added runs/raw/E1_core.jsonl +3841 −0
Large diff (3,841 lines) is not loaded by default to keep the page fast.Load diff
modified runs/raw/E4_pilot.jsonl +750 −0
Large diff (753 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/E6_core.jsonl +324 −0
Large diff (324 lines) is not loaded by default to keep the page fast.Load diff
modified runs/raw/ED_hosted_pilot.jsonl +160 −40
Large diff (200 lines) is not loaded by default to keep the page fast.Load diff
deleted runs/raw/ED_hosted_pilot_m3u96b.jsonl +0 −35
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02000_2000_A4_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02000_2000_A4_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02000_2000_A5_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02000_2000_A5_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02001_8000_A4_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02001_8000_A4_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02001_8000_A5_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02001_8000_A5_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02003_2000_A4_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02003_2000_A4_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02003_2000_A5_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02003_2000_A5_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02004_8000_A4_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02004_8000_A4_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02004_8000_A5_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02004_8000_A5_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02006_2000_A4_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02006_2000_A4_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02006_2000_A5_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02006_2000_A5_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02007_8000_A4_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02007_8000_A4_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02007_8000_A5_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02007_8000_A5_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added scripts/collect_remote_results.sh +38 −0
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added scripts/queue_m3u96a.sh +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added scripts/queue_m3u96b.sh +11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/analysis/analyze.py +30 −24
Large diff (75 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/analysis/numbers.py +8 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/chains.py +2 −1
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/fanin.py +2 −1
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/runner.py +2 −1
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere