SPB Git

spb/wp30_uqo

Public

WP30: E6 two-node core measured (324 runs), analysis/break-even figure, network results section, collection scripts

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Oct 30, 2026) parent e6d9a3b Browse files

69 changed files +9,223 −149

69 changed files
modified STATUS.md +8 −5
@@ -1,8 +1,11 @@
1 1 # STATUS.md — WP30
2 2
3 −**Phase:** 3→4 complete (harness + live inference + cache correctness suite 12/12), 5 in progress (text methods implemented, smoke-tested), 6 pending
4 −**Completed gates:** Phase 0 (environment, scope), Phase 1 (literature audit: 34 sources, 3 audit files), Phase 2 (datasets: 800 items, keys isolated, 0 cross-split duplicates), Phase 3 (E0 smoke: 114 trials, real generation), Phase 4 (tests/test_cache_correctness.py 12 passed)
5 −**Current task:** pilot config, Llama receiver smoke, chain/fan-in smoke, remote worker (M3U96b) for E2/E6
3 +**Phase:** 8 (core single-handoff campaign running) — protocol frozen 2026-10-30 04:32 UTC, hash 404c52b34d964693…
4 +**Completed gates:** 0 environment · 1 literature (34 sources) · 2 datasets (800 items, keys isolated) · 3 live inference · 4 cache correctness (12/12) ·
5 +5 text/evidence methods (A0–A8) · 6 advanced methods (B5/B6 faithful re-implementations; linear mapper v1/v2 trained on M3U96b; SwaT/C2C/LatentMAS/H2O/Compaction audited, not run) ·
6 +7 pilot (E1_pilot 1 037 trials dev; E3_pilot/v2; E4_pilot; ED_hosted_pilot) + freeze
7 +**Running:** E1_core Qwen shard (laptop, started 04:32 UTC); E4_pilot (M3U96a, finishing); ED_hosted_pilot (M3U96b)
8 +**Queued:** E1_core Llama shard (M3U96a) · E6_core (M3U96a→M3U96b) · E4_core, E5_core, E3_core(+sender) (M3U96b) · ED_hosted_core · E8 router eval · analysis · paper
6 9 **Blockers:** none
7 −**Resume:** `source .venv/bin/activate && python -m wp30 run --config configs/pilot.yaml --resume`
8 −**Run IDs so far:** E0_smoke (runs/raw/E0_smoke.jsonl)
10 +**Resume commands:** laptop `python -m wp30 run --config configs/core_qwen.yaml --resume`; M3U96a `python -m wp30 run --config configs/core_llama.yaml --resume`;
11 +M3U96b `python -m wp30.cli_extra chains configs/chains_core.yaml --resume`, `fanin configs/fanin_core.yaml --resume`, `python -m wp30 run --config configs/core_transfer.yaml --resume`
modified paper/sections/results.tex +12 −1
@@ -1,3 +1,14 @@
1 1 \section{Results}
2 2 \label{sec:results}
3 −To be written from results.
3 +\input{tables/numbers}
4 +All numbers in this section are computed from the released per-attempt records by \texttt{src/wp30/analysis/analyze.py}; every table
5 +cell and plotted point resolves to run identifiers through \texttt{results/provenance.json}. Quality is accuracy for the synthetic and
6 +executable families and LongBench F1 for the public subsets; intervals are 95\,\% document-level cluster-bootstrap intervals; test
7 +statistics follow the registered plan (Section~\ref{sec:protocol}). Scheduled core trials: \nCoreTrials.
8 +
9 +\input{sections/results_frontier}
10 +\input{sections/results_transfer}
11 +\input{sections/results_handoffs}
12 +\input{sections/results_fanin}
13 +\input{sections/results_network}
14 +\input{sections/results_hosted}
added paper/sections/results_fanin.tex +1 −0
@@ -0,0 +1 @@
1 +% pending
added paper/sections/results_frontier.tex +1 −0
@@ -0,0 +1 @@
1 +% pending
added paper/sections/results_handoffs.tex +1 −0
@@ -0,0 +1 @@
1 +% pending
added paper/sections/results_hosted.tex +1 −0
@@ -0,0 +1 @@
1 +% pending
added paper/sections/results_network.tex +49 −0
@@ -0,0 +1,49 @@
1 +\subsection{Two-node reuse and locality (E2/E6)}
2 +\label{sec:network}
3 +Figure~\ref{fig:breakeven} and Table~\ref{tab:network} report genuine two-node measurements: the sender node (M3U96a) prefills a
4 +test document with Qwen2.5-3B, serializes the cache with its manifest, optionally quantizes or compresses it, and sends it over the
5 +LAN to a receiver worker (M3U96b) that checks the manifest, installs the cache, prefills only the fresh question and decodes the
6 +answer; the alternative sends the token ids and lets the worker prefill the whole document. Each point is the median of 18 runs
7 +(6 test documents $\times$ 3 repeats, after one warm-up on both ends); the effective transport rate was
8 +\EsixBONoneEZZZGbps{}~Gb/s on a link that iperf3 measured at 0.94~Gb/s.
9 +
10 +\begin{figure}[t]\centering
11 +\includegraphics[width=\textwidth]{fig6_breakeven.pdf}
12 +\caption{Reuse/network break-even. Left: measured round trip to a remote answer (medians, p25--p75 bars) for moving the native bf16 cache
13 +(B1), the zstd-compressed cache, the 8-bit and 4-bit packed caches (B2), versus sending the text and prefilling remotely; the grey line is
14 +the sender's own local prefill for scale. Right: analytic transfer time from the measured bandwidth (solid) and a hypothetical 10~Gb/s link
15 +(dashed) against the measured remote prefill slope. Qwen2.5-3B; synthetic test documents; M3U96a $\to$ M3U96b.}\label{fig:breakeven}
16 +\end{figure}
17 +
18 +\begin{table}[t]\centering\scriptsize
19 +\caption{Two-node transfer (medians of 18 runs; ms unless stated). ``Send'' is the sender-side socket time; ``load'' is manifest check plus
20 +deserialization at the worker; the round trip includes the worker's fresh-query prefill and 32 decoded tokens. 4-bit rows transfer a cache
21 +that yields quality~0 for Qwen2.5 (Section~\ref{sec:results-frontier}) and are shown only for byte and time accounting.}\label{tab:network}
22 +\begin{tabular}{@{}llrrrrr@{}}\toprule
23 +Context class & Representation & payload (MB) & send & load & remote prefill & round trip \\\midrule
24 +2k & text $\to$ remote prefill & 0.01 & -- & -- & \EsixTextRemotePrefillNoneTZZZPrefill & \EsixTextRemotePrefillNoneTZZZRt \\
25 +2k & B1 bf16 cache & \EsixBONoneTZZZBytes & \EsixBONoneTZZZSend & \EsixBONoneTZZZLoad & 0 & \EsixBONoneTZZZRt \\
26 +2k & B2 8-bit cache & \EsixBTENoneTZZZBytes & \EsixBTENoneTZZZSend & \EsixBTENoneTZZZLoad & 0 & \EsixBTENoneTZZZRt \\
27 +8k & text $\to$ remote prefill & 0.03 & -- & -- & \EsixTextRemotePrefillNoneEZZZPrefill & \EsixTextRemotePrefillNoneEZZZRt \\
28 +8k & B1 bf16 cache & \EsixBONoneEZZZBytes & \EsixBONoneEZZZSend & \EsixBONoneEZZZLoad & 0 & \EsixBONoneEZZZRt \\
29 +8k & B2 8-bit cache & \EsixBTENoneEZZZBytes & \EsixBTENoneEZZZSend & \EsixBTENoneEZZZLoad & 0 & \EsixBTENoneEZZZRt \\
30 +16k (${\approx}$11k tokens) & text $\to$ remote prefill & 0.06 & -- & -- & \EsixTextRemotePrefillNoneOSZZZPrefill & \EsixTextRemotePrefillNoneOSZZZRt \\
31 +16k & B1 bf16 cache & \EsixBONoneOSZZZBytes & \EsixBONoneOSZZZSend & \EsixBONoneOSZZZLoad & 0 & \EsixBONoneOSZZZRt \\
32 +16k & B2 8-bit cache & \EsixBTENoneOSZZZBytes & \EsixBTENoneOSZZZSend & \EsixBTENoneOSZZZLoad & 0 & \EsixBTENoneOSZZZRt \\\bottomrule
33 +\end{tabular}
34 +\end{table}
35 +
36 +Three observations follow. First, the serialized bf16 cache is three to four orders of magnitude larger than the text it replaces
37 +(\EsixBONoneEZZZBytes{}~MB versus 29~kB at 8k tokens), and zstd recovers only ${\approx}20$\,\% of it (high-entropy payload; H8), while packed
38 +8-bit quantization halves it. Second, on this link and with this receiver, moving the cache is nevertheless \emph{faster} than re-reading
39 +the text at every context length we measured: at 8k tokens the bf16 transfer answers in \EsixBONoneEZZZRt{}~ms against
40 +\EsixTextRemotePrefillNoneEZZZRt{}~ms for remote prefill, and the 8-bit cache in \EsixBTENoneEZZZRt{}~ms, because the M3~Ultra receiver prefills
41 +Qwen2.5-3B at ${\approx}0.44$~ms per token (\EsixTextRemotePrefillNoneEZZZPrefill{}~ms for 8k) whereas 1~GbE moves the per-token cache
42 +(${\approx}26$~kB bf16, ${\approx}14$~kB at 8 bits) in ${\approx}0.23$ and ${\approx}0.12$~ms. The break-even therefore depends on the ratio
43 +of receiver prefill time to link time per token, which the right panel makes explicit: a faster receiver (the laptop prefills 8k tokens in
44 +${\approx}1$~s) or a slower link would move the crossing to longer contexts, and a 10~Gb/s link would make transfer dominant everywhere. Third,
45 +the cost of building the cache is paid by the sender and is counted once per document: when the cache is reused for $N$ queries at the
46 +receiver, the per-query cost is the fresh-query prefill plus decoding (${\approx}$\EsixBONoneEZZZLoad{}~ms load and ${\approx}190$~ms to the
47 +first token at 8k), while the text alternative pays the full prefill on each cold query unless the receiver keeps its own cache---at which
48 +point the two strategies converge and the question becomes where to schedule the work. Equation~\eqref{eq:amortized} with the measured
49 +components gives the break-even reuse count for any bandwidth; these are measured points on one link, not a universal bound.
added paper/sections/results_transfer.tex +1 −0
@@ -0,0 +1 @@
1 +% pending
added paper/tables/numbers.tex +268 −0
@@ -0,0 +1,268 @@
1 +% auto-generated by wp30.analysis.numbers — do not edit
2 +\newcommand{\nCoreTrials}{3,560}
3 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAZ}{0}
4 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAZ}{0}
5 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAZ}{0}
6 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAZ}{0}
7 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAZ}{50}
8 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAZ}{50}
9 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAO}{99}
10 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAO}{97}
11 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAO}{100}
12 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAO}{29,157}
13 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAO}{973}
14 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAO}{973}
15 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOT}{21}
16 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOT}{17}
17 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOT}{24}
18 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOT}{2,244}
19 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOT}{112}
20 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOT}{108}
21 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTAware}{97}
22 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTAware}{95}
23 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTAware}{99}
24 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTAware}{1,368}
25 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTAware}{209}
26 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTAware}{207}
27 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTBlind}{38}
28 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTBlind}{35}
29 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTBlind}{39}
30 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTBlind}{1,247}
31 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTBlind}{158}
32 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSATBudgetTokensFOTBlind}{152}
33 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{94}
34 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{88}
35 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{98}
36 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{382}
37 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{3,078}
38 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{190}
39 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{38}
40 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{28}
41 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{48}
42 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{1,424}
43 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{7,451}
44 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{132}
45 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{100}
46 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{100}
47 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{100}
48 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{118}
49 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{2,195}
50 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTAware}{179}
51 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{16}
52 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{13}
53 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{18}
54 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{828}
55 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{8,171}
56 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAFBudgetTokensFOTBlind}{85}
57 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASBudgetTokensFOTMaxFetchBytesTZZZMaxLookupsSBlind}{17}
58 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASBudgetTokensFOTMaxFetchBytesTZZZMaxLookupsSBlind}{14}
59 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASBudgetTokensFOTMaxFetchBytesTZZZMaxLookupsSBlind}{21}
60 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASBudgetTokensFOTMaxFetchBytesTZZZMaxLookupsSBlind}{636}
61 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASBudgetTokensFOTMaxFetchBytesTZZZMaxLookupsSBlind}{176}
62 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASBudgetTokensFOTMaxFetchBytesTZZZMaxLookupsSBlind}{176}
63 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASCodecZstd}{99}
64 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASCodecZstd}{97}
65 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASCodecZstd}{100}
66 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASCodecZstd}{2,725}
67 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASCodecZstd}{1,042}
68 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSASCodecZstd}{1,042}
69 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAEBudgetTokensFOTBlind}{13}
70 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAEBudgetTokensFOTBlind}{8}
71 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAEBudgetTokensFOTBlind}{18}
72 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAEBudgetTokensFOTBlind}{2,571}
73 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAEBudgetTokensFOTBlind}{1,328}
74 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSAEBudgetTokensFOTBlind}{162}
75 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBZ}{99}
76 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBZ}{97}
77 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBZ}{100}
78 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBZ}{215,449,381}
79 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBZ}{1,806}
80 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBZ}{189}
81 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBO}{99}
82 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBO}{97}
83 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBO}{100}
84 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBO}{215,486,273}
85 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBO}{1,661}
86 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBO}{214}
87 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsF}{0}
88 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsF}{0}
89 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsF}{0}
90 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsF}{60,635,200}
91 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsF}{1,794}
92 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsF}{747}
93 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsE}{99}
94 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsE}{97}
95 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsE}{100}
96 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsE}{114,502,809}
97 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsE}{1,281}
98 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTBitsE}{180}
99 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTKeepFracZTF}{21}
100 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTKeepFracZTF}{18}
101 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTKeepFracZTF}{23}
102 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTKeepFracZTF}{53,874,493}
103 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTKeepFracZTF}{1,244}
104 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBTKeepFracZTF}{126}
105 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTF}{0}
106 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTF}{0}
107 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTF}{0}
108 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTF}{53,874,488}
109 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTF}{1,735}
110 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTF}{678}
111 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFAware}{76}
112 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFAware}{70}
113 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFAware}{82}
114 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFAware}{53,874,532}
115 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFAware}{1,609}
116 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFAware}{140}
117 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFBlind}{12}
118 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFBlind}{7}
119 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFBlind}{18}
120 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFBlind}{53,874,533}
121 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFBlind}{1,387}
122 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBFKeepFracZTFBlind}{106}
123 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBSKeepFracZTFAware}{65}
124 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBSKeepFracZTFAware}{59}
125 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBSKeepFracZTFAware}{71}
126 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBSKeepFracZTFAware}{53,874,422}
127 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBSKeepFracZTFAware}{1,842}
128 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBSKeepFracZTFAware}{544}
129 +\newcommand{\QSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBNBitsEKeepFracZTFPolicyAttentionAware}{77}
130 +\newcommand{\QloSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBNBitsEKeepFracZTFPolicyAttentionAware}{71}
131 +\newcommand{\QhiSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBNBitsEKeepFracZTFPolicyAttentionAware}{83}
132 +\newcommand{\BSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBNBitsEKeepFracZTFPolicyAttentionAware}{28,646,537}
133 +\newcommand{\TcoldSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBNBitsEKeepFracZTFPolicyAttentionAware}{1,316}
134 +\newcommand{\TwarmSyntheticQwenTFTBInstructBfOSQwenTFTBInstructBfOSBNBitsEKeepFracZTFPolicyAttentionAware}{148}
135 +\newcommand{\DiffTO}{-21.8}
136 +\newcommand{\LoTO}{-31.8}
137 +\newcommand{\HiTO}{-12.4}
138 +\newcommand{\PTO}{0.000}
139 +\newcommand{\NTO}{170}
140 +\newcommand{\DiffTT}{+0.0}
141 +\newcommand{\LoTT}{+0.0}
142 +\newcommand{\HiTT}{+0.0}
143 +\newcommand{\PTT}{1.000}
144 +\newcommand{\NTT}{165}
145 +\newcommand{\DiffTT}{-98.8}
146 +\newcommand{\LoTT}{-100.0}
147 +\newcommand{\HiTT}{-97.0}
148 +\newcommand{\PTT}{0.000}
149 +\newcommand{\NTT}{165}
150 +\newcommand{\DiffTF}{+55.8}
151 +\newcommand{\LoTF}{+49.1}
152 +\newcommand{\HiTF}{+62.4}
153 +\newcommand{\PTF}{0.000}
154 +\newcommand{\NTF}{165}
155 +\newcommand{\DiffTS}{+0.0}
156 +\newcommand{\LoTS}{+0.0}
157 +\newcommand{\HiTS}{+0.0}
158 +\newcommand{\PTS}{1.000}
159 +\newcommand{\NTS}{170}
160 +\newcommand{\EsixBONoneTZZZRt}{1,141}
161 +\newcommand{\EsixBONoneTZZZBytes}{72}
162 +\newcommand{\EsixBONoneTZZZSend}{583}
163 +\newcommand{\EsixBONoneTZZZPrefill}{--}
164 +\newcommand{\EsixBONoneTZZZLoad}{14}
165 +\newcommand{\EsixBONoneTZZZGbps}{0.93}
166 +\newcommand{\EsixBONoneEZZZRt}{2,667}
167 +\newcommand{\EsixBONoneEZZZBytes}{215}
168 +\newcommand{\EsixBONoneEZZZSend}{1,806}
169 +\newcommand{\EsixBONoneEZZZPrefill}{--}
170 +\newcommand{\EsixBONoneEZZZLoad}{40}
171 +\newcommand{\EsixBONoneEZZZGbps}{0.92}
172 +\newcommand{\EsixBONoneOSZZZRt}{4,453}
173 +\newcommand{\EsixBONoneOSZZZBytes}{406}
174 +\newcommand{\EsixBONoneOSZZZSend}{3,434}
175 +\newcommand{\EsixBONoneOSZZZPrefill}{--}
176 +\newcommand{\EsixBONoneOSZZZLoad}{73}
177 +\newcommand{\EsixBONoneOSZZZGbps}{0.92}
178 +\newcommand{\EsixBOZstdTZZZRt}{1,085}
179 +\newcommand{\EsixBOZstdTZZZBytes}{72}
180 +\newcommand{\EsixBOZstdTZZZSend}{444}
181 +\newcommand{\EsixBOZstdTZZZPrefill}{--}
182 +\newcommand{\EsixBOZstdTZZZLoad}{84}
183 +\newcommand{\EsixBOZstdTZZZGbps}{0.92}
184 +\newcommand{\EsixBOZstdEZZZRt}{2,424}
185 +\newcommand{\EsixBOZstdEZZZBytes}{215}
186 +\newcommand{\EsixBOZstdEZZZSend}{1,388}
187 +\newcommand{\EsixBOZstdEZZZPrefill}{--}
188 +\newcommand{\EsixBOZstdEZZZLoad}{241}
189 +\newcommand{\EsixBOZstdEZZZGbps}{0.92}
190 +\newcommand{\EsixBOZstdOSZZZRt}{3,968}
191 +\newcommand{\EsixBOZstdOSZZZBytes}{406}
192 +\newcommand{\EsixBOZstdOSZZZSend}{2,647}
193 +\newcommand{\EsixBOZstdOSZZZPrefill}{--}
194 +\newcommand{\EsixBOZstdOSZZZLoad}{428}
195 +\newcommand{\EsixBOZstdOSZZZGbps}{0.92}
196 +\newcommand{\EsixBTFNoneTZZZRt}{1,747}
197 +\newcommand{\EsixBTFNoneTZZZBytes}{20}
198 +\newcommand{\EsixBTFNoneTZZZSend}{139}
199 +\newcommand{\EsixBTFNoneTZZZPrefill}{--}
200 +\newcommand{\EsixBTFNoneTZZZLoad}{6}
201 +\newcommand{\EsixBTFNoneTZZZGbps}{0.91}
202 +\newcommand{\EsixBTFNoneEZZZRt}{2,231}
203 +\newcommand{\EsixBTFNoneEZZZBytes}{61}
204 +\newcommand{\EsixBTFNoneEZZZSend}{483}
205 +\newcommand{\EsixBTFNoneEZZZPrefill}{--}
206 +\newcommand{\EsixBTFNoneEZZZLoad}{15}
207 +\newcommand{\EsixBTFNoneEZZZGbps}{0.91}
208 +\newcommand{\EsixBTFNoneOSZZZRt}{2,873}
209 +\newcommand{\EsixBTFNoneOSZZZBytes}{114}
210 +\newcommand{\EsixBTFNoneOSZZZSend}{939}
211 +\newcommand{\EsixBTFNoneOSZZZPrefill}{--}
212 +\newcommand{\EsixBTFNoneOSZZZLoad}{24}
213 +\newcommand{\EsixBTFNoneOSZZZGbps}{0.92}
214 +\newcommand{\EsixBTENoneTZZZRt}{1,052}
215 +\newcommand{\EsixBTENoneTZZZBytes}{38}
216 +\newcommand{\EsixBTENoneTZZZSend}{293}
217 +\newcommand{\EsixBTENoneTZZZPrefill}{--}
218 +\newcommand{\EsixBTENoneTZZZLoad}{10}
219 +\newcommand{\EsixBTENoneTZZZGbps}{0.91}
220 +\newcommand{\EsixBTENoneEZZZRt}{1,676}
221 +\newcommand{\EsixBTENoneEZZZBytes}{115}
222 +\newcommand{\EsixBTENoneEZZZSend}{942}
223 +\newcommand{\EsixBTENoneEZZZPrefill}{--}
224 +\newcommand{\EsixBTENoneEZZZLoad}{25}
225 +\newcommand{\EsixBTENoneEZZZGbps}{0.91}
226 +\newcommand{\EsixBTENoneOSZZZRt}{2,682}
227 +\newcommand{\EsixBTENoneOSZZZBytes}{216}
228 +\newcommand{\EsixBTENoneOSZZZSend}{1,804}
229 +\newcommand{\EsixBTENoneOSZZZPrefill}{--}
230 +\newcommand{\EsixBTENoneOSZZZLoad}{43}
231 +\newcommand{\EsixBTENoneOSZZZGbps}{0.91}
232 +\newcommand{\EsixTextRemotePrefillNoneTZZZRt}{1,567}
233 +\newcommand{\EsixTextRemotePrefillNoneTZZZBytes}{0}
234 +\newcommand{\EsixTextRemotePrefillNoneTZZZSend}{0}
235 +\newcommand{\EsixTextRemotePrefillNoneTZZZPrefill}{1,160}
236 +\newcommand{\EsixTextRemotePrefillNoneTZZZLoad}{--}
237 +\newcommand{\EsixTextRemotePrefillNoneTZZZGbps}{--}
238 +\newcommand{\EsixTextRemotePrefillNoneEZZZRt}{3,667}
239 +\newcommand{\EsixTextRemotePrefillNoneEZZZBytes}{0}
240 +\newcommand{\EsixTextRemotePrefillNoneEZZZSend}{0}
241 +\newcommand{\EsixTextRemotePrefillNoneEZZZPrefill}{3,241}
242 +\newcommand{\EsixTextRemotePrefillNoneEZZZLoad}{--}
243 +\newcommand{\EsixTextRemotePrefillNoneEZZZGbps}{--}
244 +\newcommand{\EsixTextRemotePrefillNoneOSZZZRt}{7,553}
245 +\newcommand{\EsixTextRemotePrefillNoneOSZZZBytes}{0}
246 +\newcommand{\EsixTextRemotePrefillNoneOSZZZSend}{0}
247 +\newcommand{\EsixTextRemotePrefillNoneOSZZZPrefill}{7,043}
248 +\newcommand{\EsixTextRemotePrefillNoneOSZZZLoad}{--}
249 +\newcommand{\EsixTextRemotePrefillNoneOSZZZGbps}{--}
250 +\newcommand{\EsixLocalPrefillTZZZRt}{--}
251 +\newcommand{\EsixLocalPrefillTZZZBytes}{--}
252 +\newcommand{\EsixLocalPrefillTZZZSend}{--}
253 +\newcommand{\EsixLocalPrefillTZZZPrefill}{944}
254 +\newcommand{\EsixLocalPrefillTZZZLoad}{--}
255 +\newcommand{\EsixLocalPrefillTZZZGbps}{--}
256 +\newcommand{\EsixLocalPrefillEZZZRt}{--}
257 +\newcommand{\EsixLocalPrefillEZZZBytes}{--}
258 +\newcommand{\EsixLocalPrefillEZZZSend}{--}
259 +\newcommand{\EsixLocalPrefillEZZZPrefill}{3,262}
260 +\newcommand{\EsixLocalPrefillEZZZLoad}{--}
261 +\newcommand{\EsixLocalPrefillEZZZGbps}{--}
262 +\newcommand{\EsixLocalPrefillOSZZZRt}{--}
263 +\newcommand{\EsixLocalPrefillOSZZZBytes}{--}
264 +\newcommand{\EsixLocalPrefillOSZZZSend}{--}
265 +\newcommand{\EsixLocalPrefillOSZZZPrefill}{6,912}
266 +\newcommand{\EsixLocalPrefillOSZZZLoad}{--}
267 +\newcommand{\EsixLocalPrefillOSZZZGbps}{--}
268 +\newcommand{\protocolHash}{404c52b34d964693}
modified results/figures/fig2_quality_vs_bytes.pdf +0 −0

Binary file not shown.

modified results/figures/fig2_quality_vs_bytes.png +0 −0

Binary file not shown.

modified results/figures/fig3_quality_vs_latency_memory.pdf +0 −0

Binary file not shown.

modified results/figures/fig3_quality_vs_latency_memory.png +0 −0

Binary file not shown.

modified results/figures/fig4_latency_decomposition.pdf +0 −0

Binary file not shown.

modified results/figures/fig4_latency_decomposition.png +0 −0

Binary file not shown.

added results/figures/fig5_handoffs.pdf +0 −0

Binary file not shown.

added results/figures/fig5_handoffs.png +0 −0

Binary file not shown.

added results/figures/fig6_breakeven.pdf +0 −0

Binary file not shown.

added results/figures/fig6_breakeven.png +0 −0

Binary file not shown.

modified results/figures/figA_quality_vs_bytes_aware.pdf +0 −0

Binary file not shown.

modified results/figures/figA_quality_vs_bytes_aware.png +0 −0

Binary file not shown.

modified results/figures/figA_quality_vs_bytes_blind.pdf +0 −0

Binary file not shown.

modified results/figures/figA_quality_vs_bytes_blind.png +0 −0

Binary file not shown.

modified results/provenance.json +89 −33
@@ -1,122 +1,178 @@
1 1 {
2 − "results/tables/E1_pilot_primary.csv": {
2 + "results/tables/E1_core_primary.csv": {
3 3 "sources": [
4 − "runs/raw/E1_pilot.jsonl"
4 + "runs/raw/E1_core.jsonl"
5 5 ],
6 6 "script": "src/wp30/analysis/analyze.py",
7 − "generated_utc": "2026-10-30T04:16:26.754738Z",
7 + "generated_utc": "2026-10-30T05:17:32.110513Z",
8 8 "note": "per-condition quality (cluster bootstrap CI) and resource medians"
9 9 },
10 − "results/tables/E1_pilot_failures.csv": {
10 + "results/tables/E1_core_failures.csv": {
11 11 "sources": [
12 − "runs/raw/E1_pilot.jsonl"
12 + "runs/raw/E1_core.jsonl"
13 13 ],
14 14 "script": "src/wp30/analysis/analyze.py",
15 − "generated_utc": "2026-10-30T04:16:26.760771Z",
15 + "generated_utc": "2026-10-30T05:17:32.112093Z",
16 16 "note": "attempt status counts by method"
17 17 },
18 − "results/tables/E1_pilot_by_question_type.csv": {
18 + "results/tables/E1_core_by_question_type.csv": {
19 19 "sources": [
20 − "runs/raw/E1_pilot.jsonl"
20 + "runs/raw/E1_core.jsonl"
21 21 ],
22 22 "script": "src/wp30/analysis/analyze.py",
23 − "generated_utc": "2026-10-30T04:16:27.015267Z",
23 + "generated_utc": "2026-10-30T05:17:32.219775Z",
24 24 "note": "synthetic accuracy by question type"
25 25 },
26 − "results/tables/E1_pilot_primary_tests.csv": {
26 + "results/tables/E1_core_primary_tests.csv": {
27 27 "sources": [
28 − "runs/raw/E1_pilot.jsonl"
28 + "runs/raw/E1_core.jsonl"
29 29 ],
30 30 "script": "src/wp30/analysis/analyze.py",
31 − "generated_utc": "2026-10-30T04:16:27.081235Z",
31 + "generated_utc": "2026-10-30T05:17:32.256966Z",
32 32 "note": "registered primary comparisons, paired, cluster bootstrap, Holm"
33 33 },
34 34 "results/figures/fig2_quality_vs_bytes.pdf": {
35 35 "sources": [
36 − "runs/raw/E1_pilot.jsonl"
36 + "runs/raw/E1_core.jsonl"
37 37 ],
38 38 "script": "src/wp30/analysis/analyze.py",
39 − "generated_utc": "2026-10-30T04:16:27.840452Z",
39 + "generated_utc": "2026-10-30T05:17:32.489153Z",
40 40 "note": "points = conditions; x = median payload bytes; y = mean quality"
41 41 },
42 42 "results/figures/fig2_quality_vs_bytes.png": {
43 43 "sources": [
44 − "runs/raw/E1_pilot.jsonl"
44 + "runs/raw/E1_core.jsonl"
45 45 ],
46 46 "script": "src/wp30/analysis/analyze.py",
47 − "generated_utc": "2026-10-30T04:16:28.019470Z",
47 + "generated_utc": "2026-10-30T05:17:32.558579Z",
48 48 "note": "points = conditions; x = median payload bytes; y = mean quality"
49 49 },
50 50 "results/figures/figA_quality_vs_bytes_blind.pdf": {
51 51 "sources": [
52 − "runs/raw/E1_pilot.jsonl"
52 + "runs/raw/E1_core.jsonl"
53 53 ],
54 54 "script": "src/wp30/analysis/analyze.py",
55 − "generated_utc": "2026-10-30T04:16:28.241874Z",
55 + "generated_utc": "2026-10-30T05:17:32.683315Z",
56 56 "note": "points = conditions; x = median payload bytes; y = mean quality"
57 57 },
58 58 "results/figures/figA_quality_vs_bytes_blind.png": {
59 59 "sources": [
60 − "runs/raw/E1_pilot.jsonl"
60 + "runs/raw/E1_core.jsonl"
61 61 ],
62 62 "script": "src/wp30/analysis/analyze.py",
63 − "generated_utc": "2026-10-30T04:16:28.368816Z",
63 + "generated_utc": "2026-10-30T05:17:32.785959Z",
64 64 "note": "points = conditions; x = median payload bytes; y = mean quality"
65 65 },
66 66 "results/figures/figA_quality_vs_bytes_aware.pdf": {
67 67 "sources": [
68 − "runs/raw/E1_pilot.jsonl"
68 + "runs/raw/E1_core.jsonl"
69 69 ],
70 70 "script": "src/wp30/analysis/analyze.py",
71 − "generated_utc": "2026-10-30T04:16:28.531968Z",
71 + "generated_utc": "2026-10-30T05:17:32.850894Z",
72 72 "note": "points = conditions; x = median payload bytes; y = mean quality"
73 73 },
74 74 "results/figures/figA_quality_vs_bytes_aware.png": {
75 75 "sources": [
76 − "runs/raw/E1_pilot.jsonl"
76 + "runs/raw/E1_core.jsonl"
77 77 ],
78 78 "script": "src/wp30/analysis/analyze.py",
79 − "generated_utc": "2026-10-30T04:16:28.682029Z",
79 + "generated_utc": "2026-10-30T05:17:32.901784Z",
80 80 "note": "points = conditions; x = median payload bytes; y = mean quality"
81 81 },
82 82 "results/figures/fig3_quality_vs_latency_memory.pdf": {
83 83 "sources": [
84 − "runs/raw/E1_pilot.jsonl"
84 + "runs/raw/E1_core.jsonl"
85 85 ],
86 86 "script": "src/wp30/analysis/analyze.py",
87 − "generated_utc": "2026-10-30T04:16:28.790718Z",
87 + "generated_utc": "2026-10-30T05:17:32.984034Z",
88 88 "note": ""
89 89 },
90 90 "results/figures/fig3_quality_vs_latency_memory.png": {
91 91 "sources": [
92 − "runs/raw/E1_pilot.jsonl"
92 + "runs/raw/E1_core.jsonl"
93 93 ],
94 94 "script": "src/wp30/analysis/analyze.py",
95 − "generated_utc": "2026-10-30T04:16:28.864644Z",
95 + "generated_utc": "2026-10-30T05:17:33.041107Z",
96 96 "note": ""
97 97 },
98 98 "results/figures/fig4_latency_decomposition.pdf": {
99 99 "sources": [
100 − "runs/raw/E1_pilot.jsonl"
100 + "runs/raw/E1_core.jsonl"
101 101 ],
102 102 "script": "src/wp30/analysis/analyze.py",
103 − "generated_utc": "2026-10-30T04:16:28.935399Z",
103 + "generated_utc": "2026-10-30T05:17:33.111092Z",
104 104 "note": ""
105 105 },
106 106 "results/figures/fig4_latency_decomposition.png": {
107 107 "sources": [
108 − "runs/raw/E1_pilot.jsonl"
108 + "runs/raw/E1_core.jsonl"
109 109 ],
110 110 "script": "src/wp30/analysis/analyze.py",
111 − "generated_utc": "2026-10-30T04:16:28.986377Z",
111 + "generated_utc": "2026-10-30T05:17:33.154858Z",
112 112 "note": ""
113 113 },
114 + "results/figures/fig5_handoffs.pdf": {
115 + "sources": [
116 + "runs/raw/E4_pilot.jsonl"
117 + ],
118 + "script": "src/wp30/analysis/analyze.py",
119 + "generated_utc": "2026-10-30T05:17:34.567038Z",
120 + "note": "means over documents; chains defined in configs/chains*.yaml"
121 + },
122 + "results/figures/fig5_handoffs.png": {
123 + "sources": [
124 + "runs/raw/E4_pilot.jsonl"
125 + ],
126 + "script": "src/wp30/analysis/analyze.py",
127 + "generated_utc": "2026-10-30T05:17:34.616672Z",
128 + "note": "means over documents; chains defined in configs/chains*.yaml"
129 + },
130 + "results/tables/E4_pilot_handoffs.csv": {
131 + "sources": [
132 + "runs/raw/E4_pilot.jsonl"
133 + ],
134 + "script": "src/wp30/analysis/analyze.py",
135 + "generated_utc": "2026-10-30T05:17:34.663073Z",
136 + "note": "repeated handoffs"
137 + },
138 + "results/tables/E4_pilot_h4_primary_tests.csv": {
139 + "sources": [
140 + "runs/raw/E4_pilot.jsonl"
141 + ],
142 + "script": "src/wp30/analysis/analyze.py",
143 + "generated_utc": "2026-10-30T05:17:34.672312Z",
144 + "note": "registered primary comparisons, paired, cluster bootstrap, Holm"
145 + },
146 + "results/figures/fig6_breakeven.pdf": {
147 + "sources": [
148 + "runs/raw/E6_core.jsonl"
149 + ],
150 + "script": "src/wp30/analysis/analyze.py",
151 + "generated_utc": "2026-10-30T05:17:34.765577Z",
152 + "note": "left = measured (medians, p25-p75); right = analytic model from measured bandwidth and prefill slope; dashed 10 Gb/s is hypothetical"
153 + },
154 + "results/figures/fig6_breakeven.png": {
155 + "sources": [
156 + "runs/raw/E6_core.jsonl"
157 + ],
158 + "script": "src/wp30/analysis/analyze.py",
159 + "generated_utc": "2026-10-30T05:17:34.841447Z",
160 + "note": "left = measured (medians, p25-p75); right = analytic model from measured bandwidth and prefill slope; dashed 10 Gb/s is hypothetical"
161 + },
162 + "results/tables/E6_network.csv": {
163 + "sources": [
164 + "runs/raw/E6_core.jsonl"
165 + ],
166 + "script": "src/wp30/analysis/analyze.py",
167 + "generated_utc": "2026-10-30T05:17:34.846667Z",
168 + "note": "two-node transfer measurements"
169 + },
114 170 "results/tables/ED_hosted.csv": {
115 171 "sources": [
116 172 "runs/raw/ED_hosted_pilot.jsonl"
117 173 ],
118 174 "script": "src/wp30/analysis/analyze.py",
119 − "generated_utc": "2026-10-30T04:16:29.006067Z",
175 + "generated_utc": "2026-10-30T05:17:34.897707Z",
120 176 "note": "hosted receivers; cost estimated from provider price tables"
121 177 }
122 178 }
\ No newline at end of file
added results/tables/E1_core_by_question_type.csv +65 −0
@@ -0,0 +1,65 @@
1 +condition,qtype,quality,lo,hi,n
2 +A0,exact_copy,0.0,0.0,0.0,70
3 +A0,multi_hop,0.0,0.0,0.0,35
4 +A0,single_fact,0.0,0.0,0.0,70
5 +A1,exact_copy,1.0,1.0,1.0,70
6 +A1,multi_hop,0.9428571428571428,0.8571428571428571,1.0,35
7 +A1,single_fact,1.0,1.0,1.0,70
8 +A2|budget_tokens=512,exact_copy,0.14285714285714285,0.07142857142857142,0.22857142857142856,70
9 +A2|budget_tokens=512,multi_hop,0.2,0.08571428571428572,0.34285714285714286,35
10 +A2|budget_tokens=512,single_fact,0.2571428571428571,0.17142857142857143,0.34285714285714286,70
11 +A3|budget_tokens=512|aware,exact_copy,1.0,1.0,1.0,70
12 +A3|budget_tokens=512|aware,multi_hop,0.8571428571428571,0.7428571428571429,0.9714285714285714,35
13 +A3|budget_tokens=512|aware,single_fact,1.0,1.0,1.0,70
14 +A3|budget_tokens=512|blind,exact_copy,0.5,0.5,0.5,70
15 +A3|budget_tokens=512|blind,multi_hop,0.8857142857142857,0.7850000000000004,0.9714285714285714,35
16 +A3|budget_tokens=512|blind,single_fact,0.0,0.0,0.0,70
17 +A4|budget_tokens=512|aware,exact_copy,0.8571428571428571,0.7142857142857143,0.9714285714285714,35
18 +A4|budget_tokens=512|aware,single_fact,0.9857142857142858,0.9571428571428572,1.0,70
19 +A4|budget_tokens=512|blind,exact_copy,0.4714285714285714,0.35714285714285715,0.6,70
20 +A4|budget_tokens=512|blind,multi_hop,0.2857142857142857,0.14285714285714285,0.42857142857142855,35
21 +A4|budget_tokens=512|blind,single_fact,0.3142857142857143,0.18571428571428572,0.44285714285714284,70
22 +A5|budget_tokens=512|aware,exact_copy,1.0,1.0,1.0,34
23 +A5|budget_tokens=512|aware,single_fact,1.0,1.0,1.0,68
24 +A5|budget_tokens=512|blind,exact_copy,0.14705882352941177,0.07352941176470588,0.22058823529411764,68
25 +A5|budget_tokens=512|blind,multi_hop,0.14705882352941177,0.029411764705882353,0.2647058823529412,34
26 +A5|budget_tokens=512|blind,single_fact,0.17647058823529413,0.10294117647058823,0.25,68
27 +A6|budget_tokens=512|max_fetch_bytes=2000|max_lookups=6|blind,exact_copy,0.1323529411764706,0.07352941176470588,0.22058823529411764,68
28 +A6|budget_tokens=512|max_fetch_bytes=2000|max_lookups=6|blind,multi_hop,0.14705882352941177,0.029411764705882353,0.2647058823529412,34
29 +A6|budget_tokens=512|max_fetch_bytes=2000|max_lookups=6|blind,single_fact,0.22058823529411764,0.1323529411764706,0.3235294117647059,68
30 +A7|codec=zstd,exact_copy,1.0,1.0,1.0,68
31 +A7|codec=zstd,multi_hop,0.9411764705882353,0.8529411764705882,1.0,34
32 +A7|codec=zstd,single_fact,1.0,1.0,1.0,68
33 +A8|budget_tokens=512|blind,exact_copy,0.10294117647058823,0.04411764705882353,0.17647058823529413,68
34 +A8|budget_tokens=512|blind,multi_hop,0.0,0.0,0.0,34
35 +A8|budget_tokens=512|blind,single_fact,0.22058823529411764,0.1323529411764706,0.3312499999999997,68
36 +B0,exact_copy,1.0,1.0,1.0,68
37 +B0,multi_hop,0.9411764705882353,0.8529411764705882,1.0,34
38 +B0,single_fact,1.0,1.0,1.0,68
39 +B1,exact_copy,1.0,1.0,1.0,68
40 +B1,multi_hop,0.9411764705882353,0.8529411764705882,1.0,34
41 +B1,single_fact,1.0,1.0,1.0,68
42 +B2|bits=4,exact_copy,0.0,0.0,0.0,68
43 +B2|bits=4,multi_hop,0.0,0.0,0.0,34
44 +B2|bits=4,single_fact,0.0,0.0,0.0,68
45 +B2|bits=8,exact_copy,1.0,1.0,1.0,68
46 +B2|bits=8,multi_hop,0.9411764705882353,0.8529411764705882,1.0,34
47 +B2|bits=8,single_fact,1.0,1.0,1.0,68
48 +B3|keep_frac=0.25,exact_copy,0.3235294117647059,0.23529411764705882,0.40477941176470555,68
49 +B3|keep_frac=0.25,multi_hop,0.058823529411764705,0.0,0.14705882352941177,34
50 +B3|keep_frac=0.25,single_fact,0.14705882352941177,0.07352941176470588,0.23529411764705882,68
51 +B4|keep_frac=0.25,exact_copy,0.0,0.0,0.0,68
52 +B4|keep_frac=0.25,multi_hop,0.0,0.0,0.0,34
53 +B4|keep_frac=0.25,single_fact,0.0,0.0,0.0,68
54 +B5|keep_frac=0.25|aware,exact_copy,0.9411764705882353,0.8893382352941178,0.9852941176470589,68
55 +B5|keep_frac=0.25|aware,multi_hop,0.5882352941176471,0.4411764705882353,0.7352941176470589,34
56 +B5|keep_frac=0.25|aware,single_fact,0.6617647058823529,0.5882352941176471,0.7352941176470589,68
57 +B5|keep_frac=0.25|blind,exact_copy,0.20588235294117646,0.11764705882352941,0.29411764705882354,68
58 +B5|keep_frac=0.25|blind,multi_hop,0.058823529411764705,0.0,0.14705882352941177,34
59 +B5|keep_frac=0.25|blind,single_fact,0.058823529411764705,0.014705882352941176,0.11764705882352941,68
60 +B6|keep_frac=0.25|aware,exact_copy,0.8529411764705882,0.7794117647058824,0.9264705882352942,68
61 +B6|keep_frac=0.25|aware,multi_hop,0.5294117647058824,0.38235294117647056,0.6764705882352942,34
62 +B6|keep_frac=0.25|aware,single_fact,0.5147058823529411,0.5,0.5441176470588235,68
63 +B9|bits=8|keep_frac=0.25|policy=attention|aware,exact_copy,0.9411764705882353,0.8893382352941178,0.9852941176470589,68
64 +B9|bits=8|keep_frac=0.25|policy=attention|aware,multi_hop,0.6176470588235294,0.4411764705882353,0.7801470588235286,34
65 +B9|bits=8|keep_frac=0.25|policy=attention|aware,single_fact,0.6617647058823529,0.5882352941176471,0.7352941176470589,68
added results/tables/E1_core_failures.csv +18 −0
@@ -0,0 +1,18 @@
1 +method_id,invalid_output,success
2 +A0,0,175
3 +A1,0,175
4 +A2,0,175
5 +A3,0,350
6 +A4,0,280
7 +A5,0,272
8 +A6,0,170
9 +A7,0,170
10 +A8,0,170
11 +B0,0,170
12 +B1,0,170
13 +B2,13,327
14 +B3,0,170
15 +B4,12,158
16 +B5,0,340
17 +B6,0,170
18 +B9,0,170
added results/tables/E1_core_primary.csv +23 −0
@@ -0,0 +1,23 @@
1 +family,pair,condition,method,access,n,n_docs,quality,q_lo,q_hi,success_rate,payload_bytes_median,cache_bytes_median,cold_ms_p50,warm_ms_p50,amortized_ms_p50,ttft_ms_p50,peak_decode_mem_median,fetched_bytes_mean
2 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A0,A0,none,175,35,0.0,0.0,0.0,1.0,0.0,,50.39150104857981,50.38283404428512,50.38456744514406,13.24266695883125,6243951240.0,0.0
3 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A1,A1,none,175,35,0.9885714285714285,0.9714285714285714,1.0,1.0,29162.0,,970.1906249392778,970.1825840165839,970.1841922011226,14.666083035990596,7155474496.0,0.0
4 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A2|budget_tokens=512,A2,none,175,35,0.2,0.16,0.24,1.0,2246.0,,111.56629212200642,107.08154016174376,107.87064016330987,13.610875001177192,6567653464.0,0.0
5 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A3|budget_tokens=512|aware,A3,aware,175,35,0.9714285714285714,0.9485714285714286,0.9942857142857143,1.0,1365.0,,208.41120788827538,207.17641792725772,208.41120788827538,13.562750071287155,6562246744.0,0.0
6 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A3|budget_tokens=512|blind,A3,blind,175,35,0.37714285714285717,0.35428571428571426,0.3942857142857143,1.0,1246.0,,157.03170804772526,152.2996670100838,153.21449297480285,13.590875081717968,6562246744.0,0.0
7 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A4|budget_tokens=512|aware,A4,aware,105,35,0.9428571428571428,0.8857142857142857,0.9904761904761905,1.0,380.0,,3052.2540010279045,189.48554201051593,3052.2540010279045,13.359250035136938,6346302976.0,0.0
8 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A4|budget_tokens=512|blind,A4,blind,175,35,0.37142857142857144,0.28,0.4685714285714286,1.0,1440.0,,7431.057667010464,129.77629096712917,1631.4786000177264,13.581125065684319,6567358552.0,0.0
9 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A5|budget_tokens=512|aware,A5,aware,102,34,1.0,1.0,1.0,1.0,118.0,,2194.973748992197,178.98147902451456,2194.973748992197,13.401583069935441,6300656316.0,0.0
10 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A5|budget_tokens=512|blind,A5,blind,170,34,0.1588235294117647,0.12941176470588237,0.18235294117647058,1.0,828.0,,8171.243209391832,84.63735459372401,1730.138829536736,13.505270995665342,6529378068.0,0.0
11 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A6|budget_tokens=512|max_fetch_bytes=2000|max_lookups=6|blind,A6,blind,170,34,0.17058823529411765,0.13529411764705881,0.20588235294117646,1.0,636.5,,176.3136875233613,176.2877079891041,176.29290389595553,13.919333054218441,6808580172.0,1472.0529411764705
12 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A7|codec=zstd,A7,none,170,34,0.9882352941176471,0.9705882352941176,1.0,1.0,2725.0,,1041.768479917664,1041.6947924531996,1041.7095299460925,14.970291522331536,7155474496.0,0.0
13 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,A8|budget_tokens=512|blind,A8,blind,170,34,0.12941176470588237,0.08235294117647059,0.18235294117647058,1.0,2571.0,,1328.1424794113263,162.30137506499887,366.3930131937377,13.771728496067226,6604304472.0,0.0
14 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B0,B0,blind,170,34,0.9882352941176471,0.9705882352941176,1.0,1.0,215449381.0,215433216.0,1806.011937558651,189.0929365181364,505.7045995024964,47.34012502012774,6901113148.0,0.0
15 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B1,B1,blind,170,34,0.9882352941176471,0.9705882352941176,1.0,1.0,215449409.0,215433216.0,1506.5432714764029,211.0927085741423,486.3688698387705,47.08545847097412,6901113148.0,0.0
16 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B2|bits=4,B2,blind,170,34,0.0,0.0,0.0,0.9235294117647059,60624832.0,60590592.0,1775.7291264715604,737.2901674243622,982.5587245519273,27.91222953237593,6614715196.0,0.0
17 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B2|bits=8,B2,blind,170,34,0.9882352941176471,0.9705882352941176,1.0,1.0,114483225.0,114448896.0,1257.6271875877865,180.21425051847473,405.6984704802744,30.57241643546149,6734196444.0,0.0
18 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B3|keep_frac=0.25,B3,blind,170,34,0.2,0.17058823529411765,0.22941176470588234,1.0,53874493.0,53858304.0,1208.6009380291216,124.03164606075734,359.70184119651094,24.07675050199032,6546047206.0,0.0
19 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B4|keep_frac=0.25,B4,blind,170,34,0.0,0.0,0.0,0.9294117647058824,53874487.5,53858304.0,1713.8423139113002,677.3689165711403,910.9542755642906,24.169479496777058,6546047206.0,0.0
20 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B5|keep_frac=0.25|aware,B5,aware,170,34,0.7588235294117647,0.6941176470588235,0.8176470588235294,1.0,53874532.0,53858304.0,1311.0786880133674,138.59477097867057,1311.0786880133674,23.73120852280408,6545719524.0,0.0
21 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B5|keep_frac=0.25|blind,B5,blind,170,34,0.11764705882352941,0.06470588235294118,0.17647058823529413,1.0,53874532.5,53858304.0,1312.3200226109475,102.40249993512407,356.17916370974854,23.995895986445248,6545719524.0,0.0
22 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B6|keep_frac=0.25|aware,B6,aware,170,34,0.6529411764705882,0.5941176470588235,0.711764705882353,1.0,53874422.0,53858304.0,1684.0188961941749,542.1115205390379,1684.0188961941749,419.2666039452888,6584735832.0,0.0
23 +synthetic,Qwen2.5-3B-Instruct-bf16->Qwen2.5-3B-Instruct-bf16,B9|bits=8|keep_frac=0.25|policy=attention|aware,B9,aware,170,34,0.7647058823529411,0.7058823529411765,0.8235294117647058,1.0,28646537.0,28612224.0,1303.7285626051016,144.63060395792127,1303.7285626051016,21.587333467323333,6506365736.0,0.0
added results/tables/E1_core_primary_tests.csv +6 −0
@@ -0,0 +1,6 @@
1 +a,b,diff,lo,hi,p,n,n_clusters,mean_a,mean_b,rel_change,name,hypothesis,p_holm
2 +A5|budget_tokens=512|blind,A4|budget_tokens=512|blind,-0.21764705882352942,-0.3176470588235294,-0.12352941176470589,0.0,170,34,0.1588235294117647,0.3764705882352941,-0.578125,"T1 A5 vs A4 (512, blind, synthetic)",RQ5/H1,0.0
3 +B2|bits=8,B0,0.0,0.0,0.0,1.0,170,34,0.9882352941176471,0.9882352941176471,0.0,T2 B2-8bit vs B0,H2,1.0
4 +B2|bits=4,B0,-0.9882352941176471,-1.0,-0.9705882352941176,0.0,170,34,0.0,0.9882352941176471,-1.0,T3 B2-4bit vs B0,H2,0.0
5 +B5|keep_frac=0.25|aware,B3|keep_frac=0.25,0.5588235294117647,0.48823529411764705,0.6235294117647059,0.0,170,34,0.7588235294117647,0.2,2.7941176470588234,T4 B5-aware vs B3 (1/4),H3,0.0
6 +A1,B0,0.0,0.0,0.0,1.0,170,34,0.9882352941176471,0.9882352941176471,0.0,"T6 A1 vs B0 (text vs native cache, same content)",consistency,1.0
added results/tables/E4_pilot_h4_primary_tests.csv +2 −0
@@ -0,0 +1,2 @@
1 +a,b,diff,lo,hi,p,n,n_clusters,mean_a,mean_b,rel_change,name,hypothesis,p_holm
2 +A5|closed|qwen3b,A4|closed|qwen3b,-0.15555555555555556,-0.26666666666666666,-0.044166666666668186,0.016,90,6,0.23333333333333334,0.3888888888888889,-0.4,T5 A5 chain vs A4 chain at 4 handoffs (closed),H5,0.016
added results/tables/E4_pilot_handoffs.csv +13 −0
@@ -0,0 +1,13 @@
1 +chain,handoffs,accuracy,acc_lo,acc_hi,retention,ret_lo,ret_hi,unsupported_mean,fetched_bytes_mean,msg_bytes_mean,n,n_docs
2 +A4|closed|qwen3b,1,0.35555555555555557,0.23861111111111122,0.45555555555555555,0.3411427188592271,0.24802696029262292,0.41230561174591024,0.0,0.0,1496.1666666666667,90,6
3 +A4|closed|qwen3b,2,0.3888888888888889,0.26666666666666666,0.5,0.3411427188592271,0.24802696029262292,0.41230561174591024,0.0,0.0,1319.6666666666667,90,6
4 +A4|closed|qwen3b,4,0.3888888888888889,0.26666666666666666,0.5,0.3411427188592271,0.24802696029262292,0.41230561174591024,0.0,0.0,1308.0,90,6
5 +A4|repairable|qwen3b,1,0.35555555555555557,0.23861111111111122,0.45555555555555555,0.3411427188592271,0.24802696029262292,0.41230561174591024,0.0,0.0,1496.1666666666667,90,6
6 +A4|repairable|qwen3b,2,0.37777777777777777,0.25555555555555554,0.4777777777777778,0.35717367518869075,0.25722446236559143,0.4332625448763508,0.0,841.0,1330.5,90,6
7 +A4|repairable|qwen3b,4,0.4,0.28888888888888886,0.4777777777777778,0.3623820085220241,0.2618648870450786,0.4332625448763508,0.0,2523.0,1221.1666666666667,90,6
8 +A5|closed|qwen3b,1,0.24444444444444444,0.2111111111111111,0.28888888888888886,0.18358831023356179,0.13623383339319786,0.22507474646000766,0.0,0.0,824.3333333333334,90,6
9 +A5|closed|qwen3b,2,0.23333333333333334,0.2,0.2777777777777778,0.18867645125920282,0.14238196167631653,0.2268202257941064,0.0,0.0,799.5,90,6
10 +A5|closed|qwen3b,4,0.23333333333333334,0.2,0.2777777777777778,0.18867645125920282,0.14238196167631653,0.2268202257941064,0.0,0.0,799.5,90,6
11 +A5|repairable|qwen3b,1,0.24444444444444444,0.2111111111111111,0.28888888888888886,0.18358831023356179,0.13623383339319786,0.22507474646000766,0.0,0.0,824.3333333333334,90,6
12 +A5|repairable|qwen3b,2,0.24444444444444444,0.2111111111111111,0.28888888888888886,0.20996848061574833,0.1863903767453057,0.23009174427959406,0.0,720.5,819.5,90,6
13 +A5|repairable|qwen3b,4,0.24444444444444444,0.2111111111111111,0.28888888888888886,0.20996848061574833,0.1863903767453057,0.23009174427959406,0.0,1787.5,819.5,90,6
added results/tables/E6_network.csv +19 −0
@@ -0,0 +1,19 @@
1 +label,method,bits,codec,target_tokens,n,bytes,roundtrip_ms_p50,send_ms_p50,remote_load_ms_p50,remote_ttft_p50,remote_prefill_p50,eff_gbps,correct,local_prefill_p50,local_ttft_p50,remote_decode_p50
2 +lan,B1,,none,2000,18,71974604.0,1141.0682704881765,582.8466879902408,14.371166529599577,137.31991651002318,,0.9254324526074698,1.0,,,355.0059790140949
3 +lan,B1,,none,8000,18,215486247.0,2667.018833453767,1805.5645630229264,39.85322843072936,196.9017084920779,,0.9235836467576799,1.0,,,461.4051670068875
4 +lan,B1,,none,16000,18,405815405.0,4453.494437446352,3433.713915932458,72.80125049874187,238.5944375419058,,0.9209816171104153,1.0,,,602.7200620155782
5 +lan,B1,,zstd,2000,18,71974604.0,1084.9161460064352,444.3944585509598,83.68525002151728,139.11475002532825,,0.9150994733434414,1.0,,,365.122249990236
6 +lan,B1,,zstd,8000,18,215486247.0,2424.4880829937756,1388.2532505085692,240.60777097474784,193.3201244683005,,0.915824857333317,1.0,,,446.31345791276544
7 +lan,B1,,zstd,16000,18,405815405.0,3967.637834022753,2646.7615000437945,427.9795209877193,240.90118752792478,,0.9184908360866041,1.0,,,521.4174375287257
8 +lan,B2,4.0,none,2000,18,20272367.0,1747.0715835224837,138.67695850785822,6.4079585135914385,121.12035450991243,,0.9136673786435414,0.0,,,1438.9247084618546
9 +lan,B2,4.0,none,8000,18,60635179.0,2231.101604003925,482.8348124283366,14.84154158970341,136.53143751434982,,0.91348579263575,0.0,,,1519.0357704996131
10 +lan,B2,4.0,none,16000,18,114165828.0,2872.7289164671674,939.1097919433378,24.307999468874186,167.80024947365746,,0.9167853410304264,0.0,,,1636.0245210235007
11 +lan,B2,8.0,none,2000,18,38262137.0,1052.2061869851314,293.04660455090925,10.100167011842132,119.31358353467658,,0.913827491903402,1.0,,,575.5362290074117
12 +lan,B2,8.0,none,8000,18,114502788.0,1676.0850210557692,941.5643539978191,25.120604026596993,141.16060349624604,,0.9142826777406476,1.0,,,492.0222085202113
13 +lan,B2,8.0,none,16000,18,215615724.0,2681.69347953517,1804.4933750061318,42.92881244327873,164.5570209948346,,0.9145833581075375,1.0,,,511.545687506441
14 +lan,text_remote_prefill,,none,2000,18,7259.5,1567.0107079204172,0.029500515665858984,,46.87483300222084,1160.1554374792613,,1.0,,,361.99316702550277
15 +lan,text_remote_prefill,,none,8000,18,29286.0,3666.9509790372103,0.032729003578424454,,43.280771002173424,3241.15325009916,,1.0,,,389.8708130000159
16 +lan,text_remote_prefill,,none,16000,18,58842.5,7552.801541460212,0.03768753958866,,42.491624946706,7043.001604033634,,1.0,,,441.7973959352821
17 +local,local_prefill,,,2000,18,,,,,,,,1.0,943.6670205323026,127.3181255091913,
18 +local,local_prefill,,,8000,18,,,,,,,,1.0,3261.6252915468067,192.03377101803198,
19 +local,local_prefill,,,16000,18,,,,,,,,1.0,6912.2674585087225,258.25952051673084,
modified results/tables/ED_hosted.csv +30 −8
@@ -1,9 +1,31 @@
1 1 family,provider_model,condition,quality,lo,hi,n,cost_usd_mean,ttft_ms_p50,total_ms_p50,success_rate
2 −synthetic,anthropic,A0|anthropic|,0.0,0.0,0.0,5,,,,0.0
3 −synthetic,anthropic,A1|anthropic|,0.0,0.0,0.0,5,,,,0.0
4 −synthetic,anthropic,A1|anthropic|provider_cache=True,0.0,0.0,0.0,5,,,,0.0
5 −synthetic,anthropic,A4|anthropic|budget_tokens=512,0.0,0.0,0.0,5,,,,0.0
6 −synthetic,anthropic,A5|anthropic|budget_tokens=512,0.0,0.0,0.0,5,,,,0.0
7 −synthetic,gpt-5-nano-2025-08-07,A0|openai|,0.0,0.0,0.0,5,1.0850000000000001e-05,,1124.6261249762028,1.0
8 −synthetic,gpt-5-nano-2025-08-07,A1|openai|,1.0,1.0,1.0,5,8.5524e-05,,982.3185830609873,1.0
9 −synthetic,gpt-5-nano-2025-08-07,A1|openai|provider_cache=True,1.0,1.0,1.0,5,2.562e-05,,814.3011669162661,1.0
2 +exec,claude-haiku-4-5-20251001,A0|anthropic|,0.0,0.0,0.0,2,0.00045200000000000004,460.55416646413505,1190.856728469953,1.0
3 +exec,claude-haiku-4-5-20251001,A1|anthropic|,1.0,1.0,1.0,2,0.0027229999999999997,437.0991045143455,1361.941104521975,1.0
4 +exec,claude-haiku-4-5-20251001,A1|anthropic|provider_cache=True,1.0,1.0,1.0,2,0.002728,408.70312548941,1359.8948334692977,1.0
5 +exec,claude-haiku-4-5-20251001,A4|anthropic|budget_tokens=512,0.5,0.0,1.0,2,0.0009205,416.973645449616,13922.582250030246,1.0
6 +exec,claude-haiku-4-5-20251001,A5|anthropic|budget_tokens=512,0.5,0.0,1.0,2,0.0011785,406.2307709828019,11177.729208953679,1.0
7 +exec,gpt-5-nano-2025-08-07,A0|openai|,0.0,0.0,0.0,2,8.25e-05,,2476.5294790267944,1.0
8 +exec,gpt-5-nano-2025-08-07,A1|openai|,1.0,1.0,1.0,2,0.000168625,,3379.9536040751263,1.0
9 +exec,gpt-5-nano-2025-08-07,A1|openai|provider_cache=True,1.0,1.0,1.0,2,0.000165625,,2359.246875043027,1.0
10 +exec,gpt-5-nano-2025-08-07,A4|openai|budget_tokens=512,0.5,0.0,1.0,2,0.000109025,,14741.153624490835,1.0
11 +exec,gpt-5-nano-2025-08-07,A5|openai|budget_tokens=512,0.5,0.0,1.0,2,9.980000000000001e-05,,12000.74629142182,1.0
12 +public_qasper,claude-haiku-4-5-20251001,A0|anthropic|,0.0,0.0,0.0,2,8.099999999999999e-05,429.7979999682866,462.2769169509411,1.0
13 +public_qasper,claude-haiku-4-5-20251001,A1|anthropic|,0.2670454545454546,0.125,0.4090909090909091,2,0.0043159999999999995,506.47239544196054,829.3312710011378,1.0
14 +public_qasper,claude-haiku-4-5-20251001,A1|anthropic|provider_cache=True,0.2988235294117647,0.11764705882352941,0.48,2,0.004982375,508.6571045103483,902.2489790222608,1.0
15 +public_qasper,claude-haiku-4-5-20251001,A4|anthropic|budget_tokens=512,0.0,0.0,0.0,2,0.0005555,403.8986465311609,14453.188042039983,1.0
16 +public_qasper,claude-haiku-4-5-20251001,A5|anthropic|budget_tokens=512,0.0,0.0,0.0,2,0.000359,651.0722080129199,21799.3393955403,1.0
17 +public_qasper,gpt-5-nano-2025-08-07,A0|openai|,0.13636363636363635,0.0,0.2727272727272727,2,1.34e-05,,1065.4002914670855,1.0
18 +public_qasper,gpt-5-nano-2025-08-07,A1|openai|,0.2608695652173913,0.0,0.5217391304347826,2,0.00020015,,1249.2346040089615,1.0
19 +public_qasper,gpt-5-nano-2025-08-07,A1|openai|provider_cache=True,0.0,0.0,0.0,2,3.371e-05,,1359.2887704726309,1.0
20 +public_qasper,gpt-5-nano-2025-08-07,A4|openai|budget_tokens=512,0.0,0.0,0.0,2,2.595e-05,,15146.043936954811,1.0
21 +public_qasper,gpt-5-nano-2025-08-07,A5|openai|budget_tokens=512,0.0,0.0,0.0,2,2.4375000000000003e-05,,22089.14725051727,1.0
22 +synthetic,claude-haiku-4-5-20251001,A0|anthropic|,0.0,0.0,0.0,10,8.450000000000001e-05,731.1436460004188,764.0473959618248,1.0
23 +synthetic,claude-haiku-4-5-20251001,A1|anthropic|,1.0,1.0,1.0,10,0.0041795,909.5747084938921,1024.321937467903,1.0
24 +synthetic,claude-haiku-4-5-20251001,A1|anthropic|provider_cache=True,1.0,1.0,1.0,10,0.0021281299999999998,633.088978996966,688.6951045016758,1.0
25 +synthetic,claude-haiku-4-5-20251001,A4|anthropic|budget_tokens=512,0.3,0.2,0.4,10,0.000641,665.2437084703706,41555.6893760222,1.0
26 +synthetic,claude-haiku-4-5-20251001,A5|anthropic|budget_tokens=512,0.1,0.0,0.2,10,0.00043750000000000006,1539.5217915065587,38661.886979069095,1.0
27 +synthetic,gpt-5-nano-2025-08-07,A0|openai|,0.0,0.0,0.0,10,1.0835000000000002e-05,,1841.3237500353716,1.0
28 +synthetic,gpt-5-nano-2025-08-07,A1|openai|,1.0,1.0,1.0,10,8.901400000000001e-05,,1116.5167084545828,1.0
29 +synthetic,gpt-5-nano-2025-08-07,A1|openai|provider_cache=True,1.0,1.0,1.0,10,3.487e-05,,1043.0729584768414,1.0
30 +synthetic,gpt-5-nano-2025-08-07,A4|openai|budget_tokens=512,0.3,0.2,0.4,10,3.4145000000000006e-05,,37726.64497885853,1.0
31 +synthetic,gpt-5-nano-2025-08-07,A5|openai|budget_tokens=512,0.1,0.0,0.2,10,2.7650000000000005e-05,,26799.245104019064,1.0
added runs/raw/E1_core.jsonl +3841 −0
Large diff (3,841 lines) is not loaded by default to keep the page fast.Load diff
modified runs/raw/E4_pilot.jsonl +750 −0
Large diff (753 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/E6_core.jsonl +324 −0
Large diff (324 lines) is not loaded by default to keep the page fast.Load diff
modified runs/raw/ED_hosted_pilot.jsonl +160 −40
Large diff (200 lines) is not loaded by default to keep the page fast.Load diff
deleted runs/raw/ED_hosted_pilot_m3u96b.jsonl +0 −35
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02000_2000_A4_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02000_2000_A4_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02000_2000_A5_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02000_2000_A5_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02001_8000_A4_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02001_8000_A4_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02001_8000_A5_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02001_8000_A5_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02003_2000_A4_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02003_2000_A4_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02003_2000_A5_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02003_2000_A5_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02004_8000_A4_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02004_8000_A4_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02004_8000_A5_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02004_8000_A5_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02006_2000_A4_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02006_2000_A4_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02006_2000_A5_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02006_2000_A5_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02007_8000_A4_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02007_8000_A4_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02007_8000_A5_closed_qwen3b.json +142 −0
Large diff (143 lines) is not loaded by default to keep the page fast.Load diff
added runs/raw/chains/E4_pilot_syn_02007_8000_A5_repairable_qwen3b.json +145 −0
Large diff (146 lines) is not loaded by default to keep the page fast.Load diff
added scripts/collect_remote_results.sh +38 −0
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added scripts/queue_m3u96a.sh +4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added scripts/queue_m3u96b.sh +11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/analysis/analyze.py +30 −24
Large diff (75 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/analysis/numbers.py +8 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/chains.py +2 −1
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/fanin.py +2 −1
Large diff (9 lines) is not loaded by default to keep the page fast.Load diff
modified src/wp30/runner.py +2 −1
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme