SPB Git
spboucher.ai

spb/wp30_uqo

Public

WP30: final analysis (E7 exploratory), paper layout pass, FINAL_REPORT, index updates

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Oct 30, 2026) parent 2ccc6b7 Browse files

91 changed files +3,272 −181

91 changed files
Load all 91 files Unified Split
modified FINAL_REPORT.md +1 −1
@@ -19,7 +19,7 @@ Paper: `paper/WP30_Boucher_Cache_Compression_Knowledge_Transfer.pdf` (LaTeX in `
19 | ED_hosted_core | A0–A6 messages → claude-haiku-4-5 and gpt-5-nano, provider prefix caching | 672 (0.53 USD; pilots 0.19 USD) | M3U96b + APIs | complete | 19 | ED_hosted_core | A0–A6 messages → claude-haiku-4-5 and gpt-5-nano, provider prefix caching | 672 (0.53 USD; pilots 0.19 USD) | M3U96b + APIs | complete |
20 | E8 | router fitted on dev pilot, evaluated off-policy on the full-factorial test records | 640 instances × 2 regimes | laptop | complete | 20 | E8 | router fitted on dev pilot, evaluated off-policy on the full-factorial test records | 640 instances × 2 regimes | laptop | complete |
21 | E9 | minimal reproduction from a clean process | 64 trials | laptop | 64/64 identical outputs and scores | 21 | E9 | minimal reproduction from a clean process | 64 trials | laptop | 64/64 identical outputs and scores |
22−| E7_qtypes | EXPLORATORY: question types 6–15 (negation, revision, temporal, aggregation, units, unanswerable) | 5 600 planned | laptop | see STATUS | 22+| E7_qtypes | EXPLORATORY: question types 6–15 (negation, revision, temporal, aggregation, units, unanswerable) | 5 600 | laptop | complete (appendix, exploratory) |
23 23
24 Models: `mlx-community/Qwen2.5-3B-Instruct-bf16`, `Llama-3.2-3B-Instruct-bf16`, `Qwen2.5-1.5B-Instruct-bf16` (revisions and weight hashes in 24 Models: `mlx-community/Qwen2.5-3B-Instruct-bf16`, `Llama-3.2-3B-Instruct-bf16`, `Qwen2.5-1.5B-Instruct-bf16` (revisions and weight hashes in
25 `evidence/models.json`). Hardware: MacBook Pro M5 Max 48 GB (laptop), two Mac Studio M3 Ultra 96 GB (M3U96a/b) on a wired 1 GbE LAN. 25 `evidence/models.json`). Hardware: MacBook Pro M5 Max 48 GB (laptop), two Mac Studio M3 Ultra 96 GB (M3U96a/b) on a wired 1 GbE LAN.
modified evidence/pages/p-02.png +0 −0

Binary file not shown.

modified evidence/pages/p-03.png +0 −0

Binary file not shown.

modified evidence/pages/p-04.png +0 −0

Binary file not shown.

modified evidence/pages/p-05.png +0 −0

Binary file not shown.

modified evidence/pages/p-06.png +0 −0

Binary file not shown.

modified evidence/pages/p-07.png +0 −0

Binary file not shown.

modified evidence/pages/p-08.png +0 −0

Binary file not shown.

modified evidence/pages/p-09.png +0 −0

Binary file not shown.

modified evidence/pages/p-10.png +0 −0

Binary file not shown.

modified evidence/pages/p-11.png +0 −0

Binary file not shown.

modified evidence/pages/p-12.png +0 −0

Binary file not shown.

modified evidence/pages/p-13.png +0 −0

Binary file not shown.

modified evidence/pages/p-14.png +0 −0

Binary file not shown.

modified evidence/pages/p-15.png +0 −0

Binary file not shown.

modified evidence/pages/p-16.png +0 −0

Binary file not shown.

modified evidence/pages/p-17.png +0 −0

Binary file not shown.

modified evidence/pages/p-18.png +0 −0

Binary file not shown.

modified evidence/pages/p-19.png +0 −0

Binary file not shown.

modified evidence/pages/p-20.png +0 −0

Binary file not shown.

modified evidence/pages/p-21.png +0 −0

Binary file not shown.

modified evidence/pages/p-22.png +0 −0

Binary file not shown.

modified evidence/pages/p-23.png +0 −0

Binary file not shown.

modified evidence/pages/p-24.png +0 −0

Binary file not shown.

modified evidence/pages/p-25.png +0 −0

Binary file not shown.

modified evidence/pages/p-26.png +0 −0

Binary file not shown.

modified evidence/pages/p-27.png +0 −0

Binary file not shown.

modified evidence/pages/p-28.png +0 −0

Binary file not shown.

modified evidence/pages/p-29.png +0 −0

Binary file not shown.

modified evidence/pages/p-30.png +0 −0

Binary file not shown.

modified evidence/pages/p-31.png +0 −0

Binary file not shown.

modified evidence/pages/p-32.png +0 −0

Binary file not shown.

modified evidence/pages/p-33.png +0 −0

Binary file not shown.

modified evidence/pages/p-34.png +0 −0

Binary file not shown.

modified evidence/pages/p-35.png +0 −0

Binary file not shown.

modified evidence/pages/p-36.png +0 −0

Binary file not shown.

modified evidence/pages/p-37.png +0 −0

Binary file not shown.

modified evidence/pages/p-38.png +0 −0

Binary file not shown.

modified evidence/pages/p-39.png +0 −0

Binary file not shown.

modified evidence/pages/p-40.png +0 −0

Binary file not shown.

modified evidence/pages/p-41.png +0 −0

Binary file not shown.

modified evidence/pages/p-42.png +0 −0

Binary file not shown.

modified evidence/pages/p-43.png +0 −0

Binary file not shown.

modified evidence/pages/p-44.png +0 −0

Binary file not shown.

added evidence/pages/p-45.png +0 −0

Binary file not shown.

added evidence/pages/p-46.png +0 −0

Binary file not shown.

added evidence/pages/p-47.png +0 −0

Binary file not shown.

added evidence/pages/p-48.png +0 −0

Binary file not shown.

added evidence/pages/p-49.png +0 −0

Binary file not shown.

modified evidence/pages/sheet_01.png +0 −0

Binary file not shown.

modified evidence/pages/sheet_02.png +0 −0

Binary file not shown.

modified evidence/pages/sheet_03.png +0 −0

Binary file not shown.

modified evidence/pages/sheet_04.png +0 −0

Binary file not shown.

modified evidence/pages/sheet_05.png +0 −0

Binary file not shown.

modified evidence/pages/sheet_06.png +0 −0

Binary file not shown.

added evidence/pages/sheet_07.png +0 −0

Binary file not shown.

added paper/WP30_Boucher_Cache_Compression_Knowledge_Transfer.pdf +0 −0

Binary file not shown.

modified paper/appendix/extra_tables.tex +10 −0
@@ -11,6 +11,16 @@
11 \end{table} 11 \end{table}
12 12
13 \subsection{Exploratory: question types not covered by the core (A-004)} 13 \subsection{Exploratory: question types not covered by the core (A-004)}
14+\label{app:qtypes}
15+The five-question core limit kept only single-fact, exact-copy and multi-hop questions. This exploratory run (5\,600 trials; amendment A-004;
16+no confirmatory test) asks the remaining ten question types of the 40 synthetic test documents for seven conditions. Three observations, all
17+exploratory: (i) \emph{revision/precedence} questions (``current budget under the later-date rule'') are the ones that lossy text loses most---full
18+text 75--88\,\%, blind summaries 12--20\,\%, structured evidence 4--8\,\%, sink+recent 10--15\,\%---while the question-aware attention cache keeps
19+76\,\% on Llama but only 28\,\% on Qwen; (ii) \emph{negation} (cancelled vs postponed, yes/no) is answered at 72--96\,\% by every condition that
20+retains anything, close to the base rate of the majority answer, and must not be read as evidence of fact transfer; (iii) \emph{aggregation}
21+(summing five to six amounts) is at 0\,\% for every condition including full text, and \emph{unit conversion} is solved only by Qwen (57--60\,\%
22+with full text, 8-bit cache or aware eviction) and never from a summary---these are receiver-capability limits, not communication effects.
23+Unanswerable questions were abstained on correctly 94--100\,\% of the time in every condition.
14 \begin{figure}[H]\centering\includegraphics[width=\textwidth]{figA_qtypes_exploratory.pdf} 24 \begin{figure}[H]\centering\includegraphics[width=\textwidth]{figA_qtypes_exploratory.pdf}
15 \caption{EXPLORATORY. Accuracy on negation, revision/precedence, temporal order, aggregation, unit conversion and unanswerable questions 25 \caption{EXPLORATORY. Accuracy on negation, revision/precedence, temporal order, aggregation, unit conversion and unanswerable questions
16 (questions 6--15 of the 40 synthetic test documents) for a subset of methods; no confirmatory test attached.}\end{figure} 26 (questions 6--15 of the 40 synthetic test documents) for a subset of methods; no confirmatory test attached.}\end{figure}
modified paper/figures/fig2_quality_vs_bytes.pdf +0 −0

Binary file not shown.

modified paper/figures/fig3_quality_vs_latency_memory.pdf +0 −0

Binary file not shown.

modified paper/figures/fig4_latency_decomposition.pdf +0 −0

Binary file not shown.

modified paper/figures/fig5_handoffs.pdf +0 −0

Binary file not shown.

modified paper/figures/fig6_breakeven.pdf +0 −0

Binary file not shown.

modified paper/figures/fig7_router.pdf +0 −0

Binary file not shown.

modified paper/figures/figA_qtypes_exploratory.pdf +0 −0

Binary file not shown.

modified paper/figures/figA_quality_vs_bytes_aware.pdf +0 −0

Binary file not shown.

modified paper/figures/figA_quality_vs_bytes_blind.pdf +0 −0

Binary file not shown.

modified paper/main.pdf +0 −0

Binary file not shown.

modified paper/main.tex +1 −0
@@ -95,6 +95,7 @@
95 \usepackage{longtable} 95 \usepackage{longtable}
96 \usepackage{array} 96 \usepackage{array}
97 \usepackage{pdflscape} 97 \usepackage{pdflscape}
98+\usepackage{rotating}
98 \usepackage{adjustbox} 99 \usepackage{adjustbox}
99 \newcommand{\WPnumber}{30} 100 \newcommand{\WPnumber}{30}
100 \newcommand{\WPtitle}{Beyond Text: Cache Compression and Knowledge Transfer in Multi-Agent Systems} 101 \newcommand{\WPtitle}{Beyond Text: Cache Compression and Knowledge Transfer in Multi-Agent Systems}
modified paper/sections/abstract_text.tex +10 −14
@@ -2,20 +2,16 @@
2 Agents built on language models hand information to one another as text, summaries, structured evidence, shared-memory references or---when 2 Agents built on language models hand information to one another as text, summaries, structured evidence, shared-memory references or---when
3 they share a checkpoint---attention state (KV cache). We ask under which task, model, cache, network and reuse conditions each representation 3 they share a checkpoint---attention state (KV cache). We ask under which task, model, cache, network and reuse conditions each representation
4 preserves task-relevant information at the lowest end-to-end cost, and answer with a reproducible study executed by an autonomous research 4 preserves task-relevant information at the lowest end-to-end cost, and answer with a reproducible study executed by an autonomous research
5−operator (Claude Code) on Apple-Silicon hardware with the MLX runtime, whose caches are inspectable tensors. Twenty-two conditions (text 5+operator (Claude Code) on Apple-Silicon hardware with the MLX runtime, whose caches are inspectable tensors. Twenty-two conditions (truncation, extractive selection, summaries,
6−truncation, extractive selection, summaries, structured evidence with spans, references with metered retrieval, a lossless codec, LLMLingua-2, 6+structured evidence, metered retrieval, a lossless codec, LLMLingua-2, native/serialized caches, packed 8/4-bit quantization, four eviction
7−native and serialized caches, packed 8/4-bit quantization, sink-plus-recent, random, SnapKV-style and PyramidKV-style eviction) are measured on 7+rules) are measured on Qwen2.5-3B and Llama-3.2-3B across private-fact ledgers, LongBench QA and executable tasks, in question-blind and question-aware regimes, with
8−Qwen2.5-3B and Llama-3.2-3B across private-fact ledgers, LongBench QA and executable tasks, in question-blind and question-aware regimes, with
9 measured bytes, synchronized latency spans and peak memory (13\,280 trials; frozen protocol, cluster bootstrap, Holm-corrected tests). Further 8 measured bytes, synchronized latency spans and peak memory (13\,280 trials; frozen protocol, cluster bootstrap, Holm-corrected tests). Further
10−experiments measure two-node cache transfer over a 1~GbE LAN (324 runs), relays of 1--8 handoffs with an evaluator-only fact ledger (7\,200), 9+experiments measure two-node cache transfer over a 1~GbE LAN, relays of 1--8 handoffs with an evaluator-only fact ledger, fan-in with
11−fan-in with conflicting evidence (1\,600), a same-family learned KV mapper, hosted receivers (672) and a development-fitted router. Lossless 10+conflicting evidence, a same-family learned KV mapper, hosted receivers and a development-fitted router. Lossless
12 representations are interchangeable in quality and differ only in cost, whose ranking inverts between cold and reused regimes; on the measured 11 representations are interchangeable in quality and differ only in cost, whose ranking inverts between cold and reused regimes; on the measured
13−link, moving an 8-bit cache beats re-reading the text at every length. Lossy choices are model- and regime-dependent: 4-bit quantization is free 12+link, moving an 8-bit cache beats re-reading the text at every length. Lossy choices are model- and regime-dependent: 4-bit quantization is free for Llama and
14−for Llama and fatal for Qwen; attention-based eviction keeps 78--98\,\% of private-fact accuracy with the question and 12--19\,\% without it, and 13+fatal for Qwen; attention-based eviction keeps 78--98\,\% of private-fact accuracy with the question and 12--19\,\% without it; no eviction
15−no eviction survives a code specification. At a 512-token budget prose summaries beat structured evidence (38--44 vs 12--15\,\%), relays are 14+survives a code specification. At a 512-token budget prose summaries beat structured evidence (38--44 vs 12--15\,\%); relays are
16−nearly lossless after the first hop and invent almost nothing, and metered repair buys a few points for kilobytes of re-reading. A ridge KV mapper 15+nearly lossless after the first hop and invent almost nothing. A ridge KV mapper transfers no
17−transfers no content while producing coherent abstention, and an invalid cache concatenation yields plausible answers---both argue for 16+content while producing coherent abstention---an argument for compatibility manifests and negative controls. The router halves amortized latency at equal quality when questions are unknown. Cross-family cache transfer was not executed. Code, records, protocol, checkpoint and a minimal reproduction are released.%
18−compatibility manifests and negative controls. The router halves amortized latency at equal quality when questions are unknown and does not help
19−otherwise. Cross-family cache transfer was not executed; the study is partial in that respect. Code, records, protocol, checkpoint and a
20−minimal reproduction (64/64 identical outputs) are released.%
21 } 17 }
modified paper/sections/adaptive.tex +2 −2
@@ -10,11 +10,11 @@ action was run on every instance), and the router's own lookup time (${\approx}1
10 Table~\ref{tab:router} compare it with the development-best static action, full context, the native cache, the question-aware attention 10 Table~\ref{tab:router} compare it with the development-best static action, full context, the native cache, the question-aware attention
11 cache and the infeasible oracle that picks the best action per instance. 11 cache and the infeasible oracle that picks the best action per instance.
12 12
13−\begin{figure}[t]\centering\includegraphics[width=\textwidth]{fig7_router.pdf} 13+\begin{figure}[H]\centering\includegraphics[width=\textwidth]{fig7_router.pdf}
14 \caption{Dev-fitted router against static policies on held-out test instances (both pairs, all families; $n=640$ per regime). The oracle is 14 \caption{Dev-fitted router against static policies on held-out test instances (both pairs, all families; $n=640$ per regime). The oracle is
15 an infeasible upper reference.}\label{fig:router}\end{figure} 15 an infeasible upper reference.}\label{fig:router}\end{figure}
16 16
17−\begin{table}[t]\centering\footnotesize 17+\begin{table}[H]\centering\footnotesize
18 \caption{Router evaluation on held-out test instances. Quality in percent with 95\,\% cluster-bootstrap CI; cost = mean amortized latency per 18 \caption{Router evaluation on held-out test instances. Quality in percent with 95\,\% cluster-bootstrap CI; cost = mean amortized latency per
19 query (ms, document build divided by the number of reusing questions); fallback = share of instances routed by the nearest-cell fallback.}\label{tab:router} 19 query (ms, document build divided by the number of reusing questions); fallback = share of instances routed by the nearest-cell fallback.}\label{tab:router}
20 \begin{tabular}{@{}llrrrr@{}}\toprule 20 \begin{tabular}{@{}llrrrr@{}}\toprule
modified paper/sections/formulation.tex +4 −4
@@ -49,10 +49,10 @@ with batch $B$, layers $L$, stored positions $T$, KV heads $H_{KV}$, head dimens
49 \eqref{eq:mkv} only as a consistency check: the harness measures stored tensor bytes from the arrays themselves (which, for the 49 \eqref{eq:mkv} only as a consistency check: the harness measures stored tensor bytes from the arrays themselves (which, for the
50 packed quantized caches, include the uint32 words, the per-group scales and biases) and measures the serialized payload from the file 50 packed quantized caches, include the uint32 words, the per-group scales and biases) and measures the serialized payload from the file
51 written to disk, including the compatibility manifest. Two ratios are reported with explicit denominators, 51 written to disk, including the compatibility manifest. Two ratios are reported with explicit denominators,
52−\begin{equation} 52+\begin{align}
53− CR_{\text{payload}} = \frac{\text{reference serialized payload bytes}}{\text{method serialized payload bytes}},\qquad 53+ CR_{\text{payload}} &= \frac{\text{reference serialized payload bytes}}{\text{method serialized payload bytes}},\\
54− CR_{KV} = \frac{\text{native cache tensor bytes}}{\text{stored cache tensor and metadata bytes}}, 54+ CR_{KV} &= \frac{\text{native cache tensor bytes}}{\text{stored cache tensor and metadata bytes}},
55−\end{equation} 55+\end{align}
56 and a text-token ratio is never treated as a cache-position ratio. 56 and a text-token ratio is never treated as a cache-position ratio.
57 57
58 \paragraph{Timing boundaries.} Each trial records separate wall-clock spans with device synchronization before and after 58 \paragraph{Timing boundaries.} Each trial records separate wall-clock spans with device synchronization before and after
modified paper/sections/methods.tex +1 −1
@@ -25,7 +25,7 @@ and \emph{custom} baselines; the label of every method is in Table~\ref{tab:taxo
25 \item[A7 lossless transport codec.] The raw text is compressed with Zstandard (level~3) for transport and decompressed at the receiver; 25 \item[A7 lossless transport codec.] The raw text is compressed with Zstandard (level~3) for transport and decompressed at the receiver;
26 the receiver prefill is unchanged. Codec times are measured. 26 the receiver prefill is unchanged. Codec times are measured.
27 \item[A8 learned prompt compression.] LLMLingua-2 \citep{pan2024llmlingua2}, official package (\texttt{llmlingua}~0.2.2) with the 27 \item[A8 learned prompt compression.] LLMLingua-2 \citep{pan2024llmlingua2}, official package (\texttt{llmlingua}~0.2.2) with the
28− released \texttt{xlm-roberta-large-meetingbank} compressor, rate $=$ budget/source tokens, run on the Metal backend of PyTorch 28+ released XLM-RoBERTa-large MeetingBank compressor, rate $=$ budget/source tokens, run on the Metal backend of PyTorch
29 (fp32). It is question-agnostic by construction. 29 (fp32). It is question-agnostic by construction.
30 \end{description} 30 \end{description}
31 A4 and A5 use the same sender checkpoint, the same budget and prompts of comparable length; both prompts were tuned on development 31 A4 and A5 use the same sender checkpoint, the same budget and prompts of comparable length; both prompts were tuned on development
modified paper/sections/reproducibility.tex +4 −3
@@ -26,9 +26,10 @@ the provider credentials and reproduce only up to provider nondeterminism.
26 26
27 \paragraph{Reproduction commands.} 27 \paragraph{Reproduction commands.}
28 \begin{footnotesize}\begin{verbatim} 28 \begin{footnotesize}\begin{verbatim}
29−make reproduce-analysis # tables + figures from runs/raw, provenance audit 29+make reproduce-analysis # tables + figures from runs/raw, provenance audit
30−python -m wp30 reproduce --profile reproduction/minimal.yaml # re-run a fixed subset, compare greedy outputs 30+python -m wp30 reproduce --profile reproduction/minimal.yaml
31−make paper # compile this document 31+ # re-run a fixed subset, compare greedy outputs
32+make paper # compile this document
32 \end{verbatim}\end{footnotesize} 33 \end{verbatim}\end{footnotesize}
33 Greedy decoding on MLX is deterministic for an identical prefill chunking but not bitwise across chunkings (Section~\ref{sec:correctness}); 34 Greedy decoding on MLX is deterministic for an identical prefill chunking but not bitwise across chunkings (Section~\ref{sec:correctness});
34 the reproduction report lists identical output hashes and identical scores separately. 35 the reproduction report lists identical output hashes and identical scores separately.
modified paper/sections/results_frontier.tex +6 −3
@@ -11,12 +11,12 @@ condition (median payload, mean quality, 95\,\% cluster-bootstrap CI). Text meth
11 $10^7$--$10^9$ bytes: a 3B-parameter cache of a document is three to four orders of magnitude larger than the document. Question-aware and 11 $10^7$--$10^9$ bytes: a 3B-parameter cache of a document is three to four orders of magnitude larger than the document. Question-aware and
12 question-blind conditions are both shown here; Appendix figures separate them.}\label{fig:frontier}\end{figure} 12 question-blind conditions are both shown here; Appendix figures separate them.}\label{fig:frontier}\end{figure}
13 13
14−\begin{table}[p]\centering\scriptsize 14+\begin{sidewaystable}[p]\centering\scriptsize
15 \caption{Primary comparison: quality (\%, with 95\,\% cluster-bootstrap CI) per condition, family and pair. Synthetic = private-fact accuracy 15 \caption{Primary comparison: quality (\%, with 95\,\% cluster-bootstrap CI) per condition, family and pair. Synthetic = private-fact accuracy
16 (5 questions per document: single fact, exact copy, multi-hop), exec = hidden-test pass (all 5 tests), QASPER/2Wiki = LongBench F1. 16 (5 questions per document: single fact, exact copy, multi-hop), exec = hidden-test pass (all 5 tests), QASPER/2Wiki = LongBench F1.
17 ``blind''/``aware'' = question access of the compressor. Latency and bytes for the synthetic family in Table~\ref{tab:cost}.}\label{tab:primary} 17 ``blind''/``aware'' = question access of the compressor. Latency and bytes for the synthetic family in Table~\ref{tab:cost}.}\label{tab:primary}
18 \adjustbox{max width=\textwidth}{\input{tables/table_primary}} 18 \adjustbox{max width=\textwidth}{\input{tables/table_primary}}
19−\end{table} 19+\end{sidewaystable}
20 20
21 \paragraph{Lossless representations are interchangeable; nothing else is.} Full text (A1), zstd-compressed text (A7), the native cache 21 \paragraph{Lossless representations are interchangeable; nothing else is.} Full text (A1), zstd-compressed text (A7), the native cache
22 (B0), the serialized and reloaded cache (B1) and the 8-bit packed cache (B2-8) give the same answers within noise on every family and pair 22 (B0), the serialized and reloaded cache (B1) and the 8-bit packed cache (B2-8) give the same answers within noise on every family and pair
@@ -53,7 +53,10 @@ simply absent from the message; the receiver answers ``unknown'' (correctly abst
53 retrieval (A6) repair little (\QSyntheticQwenAXBRBlind{}\,\%/\QSyntheticLlamaAXBRBlind{}\,\%) because the reference index inherits the same 53 retrieval (A6) repair little (\QSyntheticQwenAXBRBlind{}\,\%/\QSyntheticLlamaAXBRBlind{}\,\%) because the reference index inherits the same
54 coverage. Deterministic extraction (A3) is the strongest cheap method: blind centrality selection keeps \QSyntheticQwenAHBBlind{}\,\%, and with 54 coverage. Deterministic extraction (A3) is the strongest cheap method: blind centrality selection keeps \QSyntheticQwenAHBBlind{}\,\%, and with
55 the question (BM25) \QSyntheticQwenAHBAware{}\,\%/\QSyntheticLlamaAHBAware{}\,\%---indistinguishable from full text at 1/15 of the bytes. 55 the question (BM25) \QSyntheticQwenAHBAware{}\,\%/\QSyntheticLlamaAHBAware{}\,\%---indistinguishable from full text at 1/15 of the bytes.
56−Head/tail truncation (A2) keeps exactly the facts that fall in the kept windows (\QSyntheticQwenAWB{}\,\%). On the executable family the 56+Head/tail truncation (A2) keeps exactly the facts that fall in the kept windows (\QSyntheticQwenAWB{}\,\%). The question types that the
57+five-question core did not cover (negation, revision/precedence, temporal order, aggregation, units, unanswerable) were run as an exploratory
58+extension after the core and are reported, labelled as such, in Appendix~\ref{app:qtypes}; the headline there is that timestamped revisions are
59+the facts that summaries lose most. On the executable family the
57 ordering changes: blind extractive selection keeps the whole specification (\QExecQwenAHBBlind{}\,\%/\QExecLlamaAHBBlind{}\,\%) because the 60 ordering changes: blind extractive selection keeps the whole specification (\QExecQwenAHBBlind{}\,\%/\QExecLlamaAHBBlind{}\,\%) because the
58 rules are dense in digits and identifiers, and the Llama summary preserves the rules better than the Qwen summary 61 rules are dense in digits and identifiers, and the Llama summary preserves the rules better than the Qwen summary
59 (\QExecLlamaAFBBlind{} vs \QExecQwenAFBBlind{}\,\%). LLMLingua-2 (A8) at the same rate removes the very tokens the questions ask for 62 (\QExecLlamaAFBBlind{} vs \QExecQwenAFBBlind{}\,\%). LLMLingua-2 (A8) at the same rate removes the very tokens the questions ask for
modified paper/sections/results_hosted.tex +1 −1
@@ -10,7 +10,7 @@ receiver cannot recover a number that the message does not contain, and the refe
10 the 2\,000-byte retrieval budget. On QASPER the hosted models score higher F1 than the local receivers from the same material (0.48 versus 10 the 2\,000-byte retrieval budget. On QASPER the hosted models score higher F1 than the local receivers from the same material (0.48 versus
11 0.35 for A1), i.e., receiver capacity matters when the material is complete. 11 0.35 for A1), i.e., receiver capacity matters when the material is complete.
12 12
13−\begin{table}[t]\centering\scriptsize 13+\begin{table}[H]\centering\scriptsize
14 \caption{Hosted receivers on messages built by the local sender (test split, first 8 documents per family). Quality = accuracy (synthetic, 14 \caption{Hosted receivers on messages built by the local sender (test split, first 8 documents per family). Quality = accuracy (synthetic,
15 exec) or F1 (QASPER) with 95\,\% cluster-bootstrap CI; cost = estimated per-query cost from provider price tables (m\$ = 0.001~USD); TTFT 15 exec) or F1 (QASPER) with 95\,\% cluster-bootstrap CI; cost = estimated per-query cost from provider price tables (m\$ = 0.001~USD); TTFT
16 = time to first streamed content token (Anthropic only; the OpenAI Responses call was not streamed). ``+cache'' = documented prompt caching 16 = time to first streamed content token (Anthropic only; the OpenAI Responses call was not streamed). ``+cache'' = documented prompt caching
modified paper/tables/numbers.tex +10 −10
@@ -1,5 +1,5 @@
1 % auto-generated by wp30.analysis.numbers — do not edit 1 % auto-generated by wp30.analysis.numbers — do not edit
2−\newcommand{\nCoreTrials}{13,280} 2+\newcommand{\nCoreTrials}{13,520}
3 \newcommand{\QExecLlamaAZ}{0} 3 \newcommand{\QExecLlamaAZ}{0}
4 \newcommand{\QloExecLlamaAZ}{0} 4 \newcommand{\QloExecLlamaAZ}{0}
5 \newcommand{\QhiExecLlamaAZ}{0} 5 \newcommand{\QhiExecLlamaAZ}{0}
@@ -57,9 +57,9 @@
57 \newcommand{\QExecLlamaAEBBlind}{0} 57 \newcommand{\QExecLlamaAEBBlind}{0}
58 \newcommand{\QloExecLlamaAEBBlind}{0} 58 \newcommand{\QloExecLlamaAEBBlind}{0}
59 \newcommand{\QhiExecLlamaAEBBlind}{0} 59 \newcommand{\QhiExecLlamaAEBBlind}{0}
60−\newcommand{\BExecLlamaAEBBlind}{--} 60+\newcommand{\BExecLlamaAEBBlind}{2,398}
61−\newcommand{\TcoldExecLlamaAEBBlind}{0} 61+\newcommand{\TcoldExecLlamaAEBBlind}{1,808}
62−\newcommand{\TwarmExecLlamaAEBBlind}{0} 62+\newcommand{\TwarmExecLlamaAEBBlind}{1,636}
63 \newcommand{\QExecLlamaBZ}{95} 63 \newcommand{\QExecLlamaBZ}{95}
64 \newcommand{\QloExecLlamaBZ}{88} 64 \newcommand{\QloExecLlamaBZ}{88}
65 \newcommand{\QhiExecLlamaBZ}{100} 65 \newcommand{\QhiExecLlamaBZ}{100}
@@ -786,12 +786,12 @@
786 \newcommand{\BSyntheticLlamaASZstd}{2,726} 786 \newcommand{\BSyntheticLlamaASZstd}{2,726}
787 \newcommand{\TcoldSyntheticLlamaASZstd}{2,857} 787 \newcommand{\TcoldSyntheticLlamaASZstd}{2,857}
788 \newcommand{\TwarmSyntheticLlamaASZstd}{2,857} 788 \newcommand{\TwarmSyntheticLlamaASZstd}{2,857}
789−\newcommand{\QSyntheticLlamaAEBBlind}{0} 789+\newcommand{\QSyntheticLlamaAEBBlind}{16}
790−\newcommand{\QloSyntheticLlamaAEBBlind}{0} 790+\newcommand{\QloSyntheticLlamaAEBBlind}{10}
791−\newcommand{\QhiSyntheticLlamaAEBBlind}{0} 791+\newcommand{\QhiSyntheticLlamaAEBBlind}{22}
792−\newcommand{\BSyntheticLlamaAEBBlind}{--} 792+\newcommand{\BSyntheticLlamaAEBBlind}{2,583}
793−\newcommand{\TcoldSyntheticLlamaAEBBlind}{0} 793+\newcommand{\TcoldSyntheticLlamaAEBBlind}{523}
794−\newcommand{\TwarmSyntheticLlamaAEBBlind}{0} 794+\newcommand{\TwarmSyntheticLlamaAEBBlind}{108}
795 \newcommand{\QSyntheticLlamaBZ}{99} 795 \newcommand{\QSyntheticLlamaBZ}{99}
796 \newcommand{\QloSyntheticLlamaBZ}{98} 796 \newcommand{\QloSyntheticLlamaBZ}{98}
797 \newcommand{\QhiSyntheticLlamaBZ}{100} 797 \newcommand{\QhiSyntheticLlamaBZ}{100}
modified paper/tables/table_qtypes_exploratory.tex +14 −14
@@ -2,19 +2,19 @@
2 \toprule 2 \toprule
3 condition / pair & aggregation & multi\_hop & negation & revision & temporal & unanswerable & unit\_conversion \\ 3 condition / pair & aggregation & multi\_hop & negation & revision & temporal & unanswerable & unit\_conversion \\
4 \midrule 4 \midrule
5−A1 Llama-3.2-3B & 0 [0,0] & 100 [100,100] & 90 [80,100] & 73 [58,88] & 47 [20,73] & 100 [100,100] & 0 [0,0] \\ 5+A1 Llama-3.2-3B & 0 [0,0] & 100 [100,100] & 90 [82,96] & 75 [66,84] & 62 [48,78] & 95 [90,99] & 0 [0,0] \\
6−A1 Qwen2.5-3B & 0 [0,0] & 100 [100,100] & 100 [100,100] & 94 [84,100] & 44 [22,69] & 97 [91,100] & 50 [25,75] \\ 6+A1 Qwen2.5-3B & 0 [0,0] & 100 [100,100] & 96 [91,100] & 88 [81,94] & 55 [40,70] & 98 [94,100] & 57 [41,72] \\
7−A3 512 blind Llama-3.2-3B & 0 [0,0] & 0 [0,0] & 83 [70,93] & 73 [57,87] & 53 [33,73] & 100 [100,100] & 0 [0,0] \\ 7+A3 512 blind Llama-3.2-3B & 0 [0,0] & 0 [0,0] & 86 [79,92] & 84 [76,91] & 65 [50,80] & 100 [100,100] & 0 [0,0] \\
8−A3 512 blind Qwen2.5-3B & 0 [0,0] & 0 [0,0] & 93 [83,100] & 73 [53,90] & 33 [13,53] & 100 [100,100] & 0 [0,0] \\ 8+A3 512 blind Qwen2.5-3B & 0 [0,0] & 0 [0,0] & 90 [83,95] & 70 [61,79] & 60 [45,75] & 100 [100,100] & 0 [0,0] \\
9−A4 512 blind Llama-3.2-3B & 0 [0,0] & 33 [7,60] & 100 [100,100] & 17 [3,33] & 20 [0,40] & 100 [100,100] & 0 [0,0] \\ 9+A4 512 blind Llama-3.2-3B & 2 [0,8] & 30 [15,45] & 95 [89,100] & 12 [6,21] & 32 [18,48] & 100 [100,100] & 2 [0,8] \\
10−A4 512 blind Qwen2.5-3B & 0 [0,0] & 44 [19,69] & 72 [53,88] & 26 [13,43] & 33 [13,53] & 100 [100,100] & 40 [13,67] \\ 10+A4 512 blind Qwen2.5-3B & 0 [0,0] & 48 [32,62] & 72 [62,81] & 20 [11,30] & 48 [32,62] & 100 [100,100] & 32 [18,48] \\
11−A5 512 blind Llama-3.2-3B & 0 [0,0] & 13 [0,33] & 50 [37,63] & 13 [0,30] & 13 [0,33] & 100 [100,100] & 0 [0,0] \\ 11+A5 512 blind Llama-3.2-3B & 0 [0,0] & 10 [2,20] & 50 [38,60] & 8 [2,15] & 10 [2,20] & 100 [100,100] & 0 [0,0] \\
12−A5 512 blind Qwen2.5-3B & 0 [0,0] & 20 [7,40] & 53 [37,70] & 10 [0,27] & 0 [0,0] & 100 [100,100] & 20 [0,40] \\ 12+A5 512 blind Qwen2.5-3B & 0 [0,0] & 15 [5,28] & 49 [37,60] & 4 [0,10] & 8 [0,18] & 100 [100,100] & 15 [5,28] \\
13−B2 bits=8 Llama-3.2-3B & 0 [0,0] & 100 [100,100] & 90 [80,100] & 73 [58,88] & 47 [20,73] & 100 [100,100] & 0 [0,0] \\ 13+B2 bits=8 Llama-3.2-3B & 0 [0,0] & 100 [100,100] & 90 [82,96] & 76 [68,85] & 62 [48,78] & 95 [90,99] & 0 [0,0] \\
14−B2 bits=8 Qwen2.5-3B & 0 [0,0] & 100 [100,100] & 100 [100,100] & 93 [83,100] & 40 [13,67] & 97 [90,100] & 60 [33,87] \\ 14+B2 bits=8 Qwen2.5-3B & 0 [0,0] & 100 [100,100] & 94 [88,99] & 88 [81,94] & 55 [40,70] & 98 [94,100] & 60 [45,75] \\
15−B3 1/4 Llama-3.2-3B & 0 [0,0] & 20 [0,40] & 73 [60,83] & 7 [0,17] & 53 [27,73] & 93 [80,100] & 0 [0,0] \\ 15+B3 1/4 Llama-3.2-3B & 0 [0,0] & 15 [5,28] & 72 [65,80] & 15 [7,24] & 57 [42,72] & 94 [88,99] & 2 [0,8] \\
16−B3 1/4 Qwen2.5-3B & 0 [0,0] & 20 [0,40] & 87 [73,97] & 0 [0,0] & 53 [27,73] & 97 [90,100] & 7 [0,20] \\ 16+B3 1/4 Qwen2.5-3B & 0 [0,0] & 18 [8,30] & 91 [85,98] & 10 [2,19] & 55 [41,70] & 99 [96,100] & 8 [0,15] \\
17−B5 1/4 aware Llama-3.2-3B & 0 [0,0] & 100 [100,100] & 87 [77,97] & 73 [62,87] & 60 [40,80] & 100 [100,100] & 0 [0,0] \\ 17+B5 1/4 aware Llama-3.2-3B & 0 [0,0] & 95 [88,100] & 85 [76,92] & 76 [69,84] & 70 [55,85] & 95 [90,99] & 0 [0,0] \\
18−B5 1/4 aware Qwen2.5-3B & 0 [0,0] & 53 [27,80] & 100 [100,100] & 30 [17,43] & 53 [27,80] & 97 [90,100] & 47 [20,73] \\ 18+B5 1/4 aware Qwen2.5-3B & 0 [0,0] & 57 [42,72] & 96 [92,100] & 28 [18,38] & 55 [40,71] & 98 [94,100] & 60 [42,72] \\
19 \bottomrule 19 \bottomrule
20 \end{tabular} 20 \end{tabular}
modified results/figures/fig2_quality_vs_bytes.pdf +0 −0

Binary file not shown.

modified results/figures/fig3_quality_vs_latency_memory.pdf +0 −0

Binary file not shown.

modified results/figures/fig4_latency_decomposition.pdf +0 −0

Binary file not shown.

modified results/figures/fig5_handoffs.pdf +0 −0

Binary file not shown.

modified results/figures/fig6_breakeven.pdf +0 −0

Binary file not shown.

modified results/figures/fig7_router.pdf +0 −0

Binary file not shown.

modified results/figures/figA_qtypes_exploratory.pdf +0 −0

Binary file not shown.

modified results/figures/figA_qtypes_exploratory.png +0 −0

Binary file not shown.

modified results/figures/figA_quality_vs_bytes_aware.pdf +0 −0

Binary file not shown.

modified results/figures/figA_quality_vs_bytes_blind.pdf +0 −0

Binary file not shown.

modified results/provenance.json +30 −30
@@ -4,7 +4,7 @@
4 "runs/raw/E1_core.jsonl" 4 "runs/raw/E1_core.jsonl"
5 ], 5 ],
6 "script": "src/wp30/analysis/analyze.py", 6 "script": "src/wp30/analysis/analyze.py",
7− "generated_utc": "2026-10-30T09:02:37.609345Z", 7+ "generated_utc": "2026-10-30T09:36:25.855895Z",
8 "note": "per-condition quality (cluster bootstrap CI) and resource medians" 8 "note": "per-condition quality (cluster bootstrap CI) and resource medians"
9 }, 9 },
10 "results/tables/E1_core_failures.csv": { 10 "results/tables/E1_core_failures.csv": {
@@ -12,7 +12,7 @@
12 "runs/raw/E1_core.jsonl" 12 "runs/raw/E1_core.jsonl"
13 ], 13 ],
14 "script": "src/wp30/analysis/analyze.py", 14 "script": "src/wp30/analysis/analyze.py",
15− "generated_utc": "2026-10-30T09:02:37.611126Z", 15+ "generated_utc": "2026-10-30T09:36:25.857450Z",
16 "note": "attempt status counts by method" 16 "note": "attempt status counts by method"
17 }, 17 },
18 "results/tables/E1_core_by_question_type.csv": { 18 "results/tables/E1_core_by_question_type.csv": {
@@ -20,7 +20,7 @@
20 "runs/raw/E1_core.jsonl" 20 "runs/raw/E1_core.jsonl"
21 ], 21 ],
22 "script": "src/wp30/analysis/analyze.py", 22 "script": "src/wp30/analysis/analyze.py",
23− "generated_utc": "2026-10-30T09:02:37.730657Z", 23+ "generated_utc": "2026-10-30T09:36:25.963048Z",
24 "note": "synthetic accuracy by question type" 24 "note": "synthetic accuracy by question type"
25 }, 25 },
26 "results/tables/E1_core_primary_tests.csv": { 26 "results/tables/E1_core_primary_tests.csv": {
@@ -28,7 +28,7 @@
28 "runs/raw/E1_core.jsonl" 28 "runs/raw/E1_core.jsonl"
29 ], 29 ],
30 "script": "src/wp30/analysis/analyze.py", 30 "script": "src/wp30/analysis/analyze.py",
31− "generated_utc": "2026-10-30T09:02:37.786568Z", 31+ "generated_utc": "2026-10-30T09:36:26.010368Z",
32 "note": "registered primary comparisons, paired, cluster bootstrap, Holm" 32 "note": "registered primary comparisons, paired, cluster bootstrap, Holm"
33 }, 33 },
34 "results/figures/fig2_quality_vs_bytes.pdf": { 34 "results/figures/fig2_quality_vs_bytes.pdf": {
@@ -36,7 +36,7 @@
36 "runs/raw/E1_core.jsonl" 36 "runs/raw/E1_core.jsonl"
37 ], 37 ],
38 "script": "src/wp30/analysis/analyze.py", 38 "script": "src/wp30/analysis/analyze.py",
39− "generated_utc": "2026-10-30T09:02:38.459020Z", 39+ "generated_utc": "2026-10-30T09:36:26.618975Z",
40 "note": "points = conditions; x = median payload bytes; y = mean quality" 40 "note": "points = conditions; x = median payload bytes; y = mean quality"
41 }, 41 },
42 "results/figures/fig2_quality_vs_bytes.png": { 42 "results/figures/fig2_quality_vs_bytes.png": {
@@ -44,7 +44,7 @@
44 "runs/raw/E1_core.jsonl" 44 "runs/raw/E1_core.jsonl"
45 ], 45 ],
46 "script": "src/wp30/analysis/analyze.py", 46 "script": "src/wp30/analysis/analyze.py",
47− "generated_utc": "2026-10-30T09:02:38.666585Z", 47+ "generated_utc": "2026-10-30T09:36:26.800030Z",
48 "note": "points = conditions; x = median payload bytes; y = mean quality" 48 "note": "points = conditions; x = median payload bytes; y = mean quality"
49 }, 49 },
50 "results/figures/figA_quality_vs_bytes_blind.pdf": { 50 "results/figures/figA_quality_vs_bytes_blind.pdf": {
@@ -52,7 +52,7 @@
52 "runs/raw/E1_core.jsonl" 52 "runs/raw/E1_core.jsonl"
53 ], 53 ],
54 "script": "src/wp30/analysis/analyze.py", 54 "script": "src/wp30/analysis/analyze.py",
55− "generated_utc": "2026-10-30T09:02:39.176959Z", 55+ "generated_utc": "2026-10-30T09:36:27.219230Z",
56 "note": "points = conditions; x = median payload bytes; y = mean quality" 56 "note": "points = conditions; x = median payload bytes; y = mean quality"
57 }, 57 },
58 "results/figures/figA_quality_vs_bytes_blind.png": { 58 "results/figures/figA_quality_vs_bytes_blind.png": {
@@ -60,7 +60,7 @@
60 "runs/raw/E1_core.jsonl" 60 "runs/raw/E1_core.jsonl"
61 ], 61 ],
62 "script": "src/wp30/analysis/analyze.py", 62 "script": "src/wp30/analysis/analyze.py",
63− "generated_utc": "2026-10-30T09:02:39.359044Z", 63+ "generated_utc": "2026-10-30T09:36:27.370309Z",
64 "note": "points = conditions; x = median payload bytes; y = mean quality" 64 "note": "points = conditions; x = median payload bytes; y = mean quality"
65 }, 65 },
66 "results/figures/figA_quality_vs_bytes_aware.pdf": { 66 "results/figures/figA_quality_vs_bytes_aware.pdf": {
@@ -68,7 +68,7 @@
68 "runs/raw/E1_core.jsonl" 68 "runs/raw/E1_core.jsonl"
69 ], 69 ],
70 "script": "src/wp30/analysis/analyze.py", 70 "script": "src/wp30/analysis/analyze.py",
71− "generated_utc": "2026-10-30T09:02:39.746284Z", 71+ "generated_utc": "2026-10-30T09:36:27.681216Z",
72 "note": "points = conditions; x = median payload bytes; y = mean quality" 72 "note": "points = conditions; x = median payload bytes; y = mean quality"
73 }, 73 },
74 "results/figures/figA_quality_vs_bytes_aware.png": { 74 "results/figures/figA_quality_vs_bytes_aware.png": {
@@ -76,7 +76,7 @@
76 "runs/raw/E1_core.jsonl" 76 "runs/raw/E1_core.jsonl"
77 ], 77 ],
78 "script": "src/wp30/analysis/analyze.py", 78 "script": "src/wp30/analysis/analyze.py",
79− "generated_utc": "2026-10-30T09:02:39.940182Z", 79+ "generated_utc": "2026-10-30T09:36:27.846184Z",
80 "note": "points = conditions; x = median payload bytes; y = mean quality" 80 "note": "points = conditions; x = median payload bytes; y = mean quality"
81 }, 81 },
82 "results/figures/fig3_quality_vs_latency_memory.pdf": { 82 "results/figures/fig3_quality_vs_latency_memory.pdf": {
@@ -84,7 +84,7 @@
84 "runs/raw/E1_core.jsonl" 84 "runs/raw/E1_core.jsonl"
85 ], 85 ],
86 "script": "src/wp30/analysis/analyze.py", 86 "script": "src/wp30/analysis/analyze.py",
87− "generated_utc": "2026-10-30T09:02:40.285980Z", 87+ "generated_utc": "2026-10-30T09:36:28.116446Z",
88 "note": "" 88 "note": ""
89 }, 89 },
90 "results/figures/fig3_quality_vs_latency_memory.png": { 90 "results/figures/fig3_quality_vs_latency_memory.png": {
@@ -92,7 +92,7 @@
92 "runs/raw/E1_core.jsonl" 92 "runs/raw/E1_core.jsonl"
93 ], 93 ],
94 "script": "src/wp30/analysis/analyze.py", 94 "script": "src/wp30/analysis/analyze.py",
95− "generated_utc": "2026-10-30T09:02:40.475124Z", 95+ "generated_utc": "2026-10-30T09:36:28.280741Z",
96 "note": "" 96 "note": ""
97 }, 97 },
98 "results/figures/fig4_latency_decomposition.pdf": { 98 "results/figures/fig4_latency_decomposition.pdf": {
@@ -100,7 +100,7 @@
100 "runs/raw/E1_core.jsonl" 100 "runs/raw/E1_core.jsonl"
101 ], 101 ],
102 "script": "src/wp30/analysis/analyze.py", 102 "script": "src/wp30/analysis/analyze.py",
103− "generated_utc": "2026-10-30T09:02:40.560198Z", 103+ "generated_utc": "2026-10-30T09:36:28.355146Z",
104 "note": "" 104 "note": ""
105 }, 105 },
106 "results/figures/fig4_latency_decomposition.png": { 106 "results/figures/fig4_latency_decomposition.png": {
@@ -108,7 +108,7 @@
108 "runs/raw/E1_core.jsonl" 108 "runs/raw/E1_core.jsonl"
109 ], 109 ],
110 "script": "src/wp30/analysis/analyze.py", 110 "script": "src/wp30/analysis/analyze.py",
111− "generated_utc": "2026-10-30T09:02:40.620070Z", 111+ "generated_utc": "2026-10-30T09:36:28.408655Z",
112 "note": "" 112 "note": ""
113 }, 113 },
114 "results/figures/fig5_handoffs.pdf": { 114 "results/figures/fig5_handoffs.pdf": {
@@ -116,7 +116,7 @@
116 "runs/raw/E4_core.jsonl" 116 "runs/raw/E4_core.jsonl"
117 ], 117 ],
118 "script": "src/wp30/analysis/analyze.py", 118 "script": "src/wp30/analysis/analyze.py",
119− "generated_utc": "2026-10-30T09:02:45.615826Z", 119+ "generated_utc": "2026-10-30T09:36:33.036763Z",
120 "note": "means over documents; chains defined in configs/chains*.yaml" 120 "note": "means over documents; chains defined in configs/chains*.yaml"
121 }, 121 },
122 "results/figures/fig5_handoffs.png": { 122 "results/figures/fig5_handoffs.png": {
@@ -124,7 +124,7 @@
124 "runs/raw/E4_core.jsonl" 124 "runs/raw/E4_core.jsonl"
125 ], 125 ],
126 "script": "src/wp30/analysis/analyze.py", 126 "script": "src/wp30/analysis/analyze.py",
127− "generated_utc": "2026-10-30T09:02:45.675774Z", 127+ "generated_utc": "2026-10-30T09:36:33.089912Z",
128 "note": "means over documents; chains defined in configs/chains*.yaml" 128 "note": "means over documents; chains defined in configs/chains*.yaml"
129 }, 129 },
130 "results/tables/E4_core_handoffs.csv": { 130 "results/tables/E4_core_handoffs.csv": {
@@ -132,7 +132,7 @@
132 "runs/raw/E4_core.jsonl" 132 "runs/raw/E4_core.jsonl"
133 ], 133 ],
134 "script": "src/wp30/analysis/analyze.py", 134 "script": "src/wp30/analysis/analyze.py",
135− "generated_utc": "2026-10-30T09:02:45.778336Z", 135+ "generated_utc": "2026-10-30T09:36:33.182901Z",
136 "note": "repeated handoffs" 136 "note": "repeated handoffs"
137 }, 137 },
138 "results/tables/E4_core_h4_primary_tests.csv": { 138 "results/tables/E4_core_h4_primary_tests.csv": {
@@ -140,7 +140,7 @@
140 "runs/raw/E4_core.jsonl" 140 "runs/raw/E4_core.jsonl"
141 ], 141 ],
142 "script": "src/wp30/analysis/analyze.py", 142 "script": "src/wp30/analysis/analyze.py",
143− "generated_utc": "2026-10-30T09:02:45.791552Z", 143+ "generated_utc": "2026-10-30T09:36:33.192836Z",
144 "note": "registered primary comparisons, paired, cluster bootstrap, Holm" 144 "note": "registered primary comparisons, paired, cluster bootstrap, Holm"
145 }, 145 },
146 "results/tables/E5_core_fanin.csv": { 146 "results/tables/E5_core_fanin.csv": {
@@ -148,7 +148,7 @@
148 "runs/raw/E5_core.jsonl" 148 "runs/raw/E5_core.jsonl"
149 ], 149 ],
150 "script": "src/wp30/analysis/analyze.py", 150 "script": "src/wp30/analysis/analyze.py",
151− "generated_utc": "2026-10-30T09:02:45.898241Z", 151+ "generated_utc": "2026-10-30T09:36:33.287122Z",
152 "note": "fan-in by method and question type" 152 "note": "fan-in by method and question type"
153 }, 153 },
154 "results/figures/fig6_breakeven.pdf": { 154 "results/figures/fig6_breakeven.pdf": {
@@ -156,7 +156,7 @@
156 "runs/raw/E6_core.jsonl" 156 "runs/raw/E6_core.jsonl"
157 ], 157 ],
158 "script": "src/wp30/analysis/analyze.py", 158 "script": "src/wp30/analysis/analyze.py",
159− "generated_utc": "2026-10-30T09:02:45.992041Z", 159+ "generated_utc": "2026-10-30T09:36:33.373261Z",
160 "note": "left = measured (medians, p25-p75); right = analytic model from measured bandwidth and prefill slope; dashed 10 Gb/s is hypothetical" 160 "note": "left = measured (medians, p25-p75); right = analytic model from measured bandwidth and prefill slope; dashed 10 Gb/s is hypothetical"
161 }, 161 },
162 "results/figures/fig6_breakeven.png": { 162 "results/figures/fig6_breakeven.png": {
@@ -164,7 +164,7 @@
164 "runs/raw/E6_core.jsonl" 164 "runs/raw/E6_core.jsonl"
165 ], 165 ],
166 "script": "src/wp30/analysis/analyze.py", 166 "script": "src/wp30/analysis/analyze.py",
167− "generated_utc": "2026-10-30T09:02:46.065845Z", 167+ "generated_utc": "2026-10-30T09:36:33.443566Z",
168 "note": "left = measured (medians, p25-p75); right = analytic model from measured bandwidth and prefill slope; dashed 10 Gb/s is hypothetical" 168 "note": "left = measured (medians, p25-p75); right = analytic model from measured bandwidth and prefill slope; dashed 10 Gb/s is hypothetical"
169 }, 169 },
170 "results/tables/E6_network.csv": { 170 "results/tables/E6_network.csv": {
@@ -172,7 +172,7 @@
172 "runs/raw/E6_core.jsonl" 172 "runs/raw/E6_core.jsonl"
173 ], 173 ],
174 "script": "src/wp30/analysis/analyze.py", 174 "script": "src/wp30/analysis/analyze.py",
175− "generated_utc": "2026-10-30T09:02:46.070991Z", 175+ "generated_utc": "2026-10-30T09:36:33.447916Z",
176 "note": "two-node transfer measurements" 176 "note": "two-node transfer measurements"
177 }, 177 },
178 "results/tables/ED_hosted.csv": { 178 "results/tables/ED_hosted.csv": {
@@ -180,7 +180,7 @@
180 "runs/raw/ED_hosted_core.jsonl" 180 "runs/raw/ED_hosted_core.jsonl"
181 ], 181 ],
182 "script": "src/wp30/analysis/analyze.py", 182 "script": "src/wp30/analysis/analyze.py",
183− "generated_utc": "2026-10-30T09:02:46.140839Z", 183+ "generated_utc": "2026-10-30T09:36:33.511504Z",
184 "note": "hosted receivers; cost estimated from provider price tables" 184 "note": "hosted receivers; cost estimated from provider price tables"
185 }, 185 },
186 "results/tables/E3_core_transfer.csv": { 186 "results/tables/E3_core_transfer.csv": {
@@ -189,7 +189,7 @@
189 "runs/raw/E3_core_sender.jsonl" 189 "runs/raw/E3_core_sender.jsonl"
190 ], 190 ],
191 "script": "src/wp30/analysis/analyze.py", 191 "script": "src/wp30/analysis/analyze.py",
192− "generated_utc": "2026-10-30T09:02:46.269308Z", 192+ "generated_utc": "2026-10-30T09:36:33.631124Z",
193 "note": "same-family mapper transfer vs text handoffs and references" 193 "note": "same-family mapper transfer vs text handoffs and references"
194 }, 194 },
195 "results/tables/E3_core_primary_tests.csv": { 195 "results/tables/E3_core_primary_tests.csv": {
@@ -198,7 +198,7 @@
198 "runs/raw/E3_core_sender.jsonl" 198 "runs/raw/E3_core_sender.jsonl"
199 ], 199 ],
200 "script": "src/wp30/analysis/analyze.py", 200 "script": "src/wp30/analysis/analyze.py",
201− "generated_utc": "2026-10-30T09:02:46.347449Z", 201+ "generated_utc": "2026-10-30T09:36:33.680469Z",
202 "note": "registered primary comparisons, paired, cluster bootstrap, Holm" 202 "note": "registered primary comparisons, paired, cluster bootstrap, Holm"
203 }, 203 },
204 "results/tables/E7_qtypes.csv": { 204 "results/tables/E7_qtypes.csv": {
@@ -206,7 +206,7 @@
206 "runs/raw/E7_qtypes.jsonl" 206 "runs/raw/E7_qtypes.jsonl"
207 ], 207 ],
208 "script": "src/wp30/analysis/analyze.py", 208 "script": "src/wp30/analysis/analyze.py",
209− "generated_utc": "2026-10-30T09:02:46.532910Z", 209+ "generated_utc": "2026-10-30T09:36:33.924117Z",
210 "note": "EXPLORATORY: question types 6-15 on synthetic test documents" 210 "note": "EXPLORATORY: question types 6-15 on synthetic test documents"
211 }, 211 },
212 "results/figures/figA_qtypes_exploratory.pdf": { 212 "results/figures/figA_qtypes_exploratory.pdf": {
@@ -214,7 +214,7 @@
214 "runs/raw/E7_qtypes.jsonl" 214 "runs/raw/E7_qtypes.jsonl"
215 ], 215 ],
216 "script": "src/wp30/analysis/analyze.py", 216 "script": "src/wp30/analysis/analyze.py",
217− "generated_utc": "2026-10-30T09:02:46.612438Z", 217+ "generated_utc": "2026-10-30T09:36:33.996541Z",
218 "note": "exploratory extension A-004" 218 "note": "exploratory extension A-004"
219 }, 219 },
220 "results/figures/figA_qtypes_exploratory.png": { 220 "results/figures/figA_qtypes_exploratory.png": {
@@ -222,7 +222,7 @@
222 "runs/raw/E7_qtypes.jsonl" 222 "runs/raw/E7_qtypes.jsonl"
223 ], 223 ],
224 "script": "src/wp30/analysis/analyze.py", 224 "script": "src/wp30/analysis/analyze.py",
225− "generated_utc": "2026-10-30T09:02:46.668848Z", 225+ "generated_utc": "2026-10-30T09:36:34.046670Z",
226 "note": "exploratory extension A-004" 226 "note": "exploratory extension A-004"
227 }, 227 },
228 "results/figures/fig7_router.pdf": { 228 "results/figures/fig7_router.pdf": {
@@ -232,7 +232,7 @@
232 "runs/checkpoints/router_policy.json" 232 "runs/checkpoints/router_policy.json"
233 ], 233 ],
234 "script": "src/wp30/analysis/analyze.py", 234 "script": "src/wp30/analysis/analyze.py",
235− "generated_utc": "2026-10-30T09:02:46.768937Z", 235+ "generated_utc": "2026-10-30T09:36:34.131525Z",
236 "note": "oracle = infeasible upper reference" 236 "note": "oracle = infeasible upper reference"
237 }, 237 },
238 "results/figures/fig7_router.png": { 238 "results/figures/fig7_router.png": {
@@ -242,7 +242,7 @@
242 "runs/checkpoints/router_policy.json" 242 "runs/checkpoints/router_policy.json"
243 ], 243 ],
244 "script": "src/wp30/analysis/analyze.py", 244 "script": "src/wp30/analysis/analyze.py",
245− "generated_utc": "2026-10-30T09:02:46.853258Z", 245+ "generated_utc": "2026-10-30T09:36:34.206816Z",
246 "note": "oracle = infeasible upper reference" 246 "note": "oracle = infeasible upper reference"
247 } 247 }
248 } 248 }
\ No newline at end of file
modified results/tables/E7_qtypes.csv +98 −98
@@ -1,99 +1,99 @@
1 condition,qtype,pair,quality,lo,hi,n,abstained 1 condition,qtype,pair,quality,lo,hi,n,abstained
2−A1,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 2+A1,aggregation,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
3−A1,aggregation,Qwen2.5-3B,0.0,0.0,0.0,16,0.0 3+A1,aggregation,Qwen2.5-3B,0.0,0.0,0.0,40,0.0
4−A1,multi_hop,Llama-3.2-3B,1.0,1.0,1.0,15,0.0 4+A1,multi_hop,Llama-3.2-3B,1.0,1.0,1.0,40,0.0
5−A1,multi_hop,Qwen2.5-3B,1.0,1.0,1.0,16,0.0 5+A1,multi_hop,Qwen2.5-3B,1.0,1.0,1.0,40,0.0
6−A1,negation,Llama-3.2-3B,0.9,0.8,1.0,30,0.0 6+A1,negation,Llama-3.2-3B,0.9,0.825,0.9625,80,0.0
7−A1,negation,Qwen2.5-3B,1.0,1.0,1.0,32,0.0 7+A1,negation,Qwen2.5-3B,0.9625,0.9125,1.0,80,0.0
8−A1,revision,Llama-3.2-3B,0.7333333333333333,0.5825000000000004,0.8841666666666659,30,0.0 8+A1,revision,Llama-3.2-3B,0.75,0.6625,0.8375,80,0.0
9−A1,revision,Qwen2.5-3B,0.9375,0.84375,1.0,32,0.0 9+A1,revision,Qwen2.5-3B,0.875,0.8125,0.9375,80,0.0
10−A1,temporal,Llama-3.2-3B,0.4666666666666667,0.2,0.7333333333333333,15,0.0 10+A1,temporal,Llama-3.2-3B,0.625,0.475,0.775,40,0.0
11−A1,temporal,Qwen2.5-3B,0.4375,0.21718750000000064,0.6875,16,0.0 11+A1,temporal,Qwen2.5-3B,0.55,0.4,0.7,40,0.0
12−A1,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0 12+A1,unanswerable,Llama-3.2-3B,0.95,0.9,0.9875,80,0.95
13−A1,unanswerable,Qwen2.5-3B,0.96875,0.90625,1.0,32,0.96875 13+A1,unanswerable,Qwen2.5-3B,0.975,0.9375,1.0,80,0.975
14−A1,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 14+A1,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
15−A1,unit_conversion,Qwen2.5-3B,0.5,0.25,0.75,16,0.0 15+A1,unit_conversion,Qwen2.5-3B,0.575,0.41187500000000027,0.725,40,0.0
16−A3|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 16+A3|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
17−A3|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.0 17+A3|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,40,0.0
18−A3|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.0,0.0,0.0,15,1.0 18+A3|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.0,0.0,0.0,40,1.0
19−A3|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.0,0.0,0.0,15,1.0 19+A3|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.0,0.0,0.0,40,1.0
20−A3|budget_tokens=512|blind,negation,Llama-3.2-3B,0.8333333333333334,0.7,0.9333333333333333,30,0.0 20+A3|budget_tokens=512|blind,negation,Llama-3.2-3B,0.8625,0.7875,0.925,80,0.0
21−A3|budget_tokens=512|blind,negation,Qwen2.5-3B,0.9333333333333333,0.8333333333333334,1.0,30,0.0 21+A3|budget_tokens=512|blind,negation,Qwen2.5-3B,0.9,0.8309375000000001,0.95,80,0.0
22−A3|budget_tokens=512|blind,revision,Llama-3.2-3B,0.7333333333333333,0.5666666666666667,0.8666666666666667,30,0.0 22+A3|budget_tokens=512|blind,revision,Llama-3.2-3B,0.8375,0.7625,0.9125,80,0.0
23−A3|budget_tokens=512|blind,revision,Qwen2.5-3B,0.7333333333333333,0.5333333333333333,0.9,30,0.0 23+A3|budget_tokens=512|blind,revision,Qwen2.5-3B,0.7,0.6059375000000001,0.7875,80,0.0125
24−A3|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.5333333333333333,0.3333333333333333,0.7333333333333333,15,0.0 24+A3|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.65,0.5,0.8,40,0.0
25−A3|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.3333333333333333,0.13333333333333333,0.5333333333333333,15,0.0 25+A3|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.6,0.45,0.75,40,0.0
26−A3|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0 26+A3|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,80,1.0
27−A3|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,30,1.0 27+A3|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,80,1.0
28−A3|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.9333333333333333 28+A3|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,40,0.925
29−A3|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.0,0.0,0.0,15,0.6666666666666666 29+A3|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.0,0.0,0.0,40,0.675
30−A4|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 30+A4|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.025,0.0,0.075,40,0.0
31−A4|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.4 31+A4|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,40,0.35
32−A4|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.3333333333333333,0.06666666666666667,0.6,15,0.5333333333333333 32+A4|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.3,0.15,0.45,40,0.65
33−A4|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.4375,0.1875,0.6875,16,0.5 33+A4|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.475,0.325,0.625,40,0.475
34−A4|budget_tokens=512|blind,negation,Llama-3.2-3B,1.0,1.0,1.0,30,0.0 34+A4|budget_tokens=512|blind,negation,Llama-3.2-3B,0.95,0.8875,1.0,80,0.0125
35−A4|budget_tokens=512|blind,negation,Qwen2.5-3B,0.71875,0.53125,0.875,32,0.1875 35+A4|budget_tokens=512|blind,negation,Qwen2.5-3B,0.725,0.625,0.8125,80,0.175
36−A4|budget_tokens=512|blind,revision,Llama-3.2-3B,0.16666666666666666,0.03333333333333333,0.3333333333333333,30,0.3333333333333333 36+A4|budget_tokens=512|blind,revision,Llama-3.2-3B,0.125,0.0625,0.2125,80,0.375
37−A4|budget_tokens=512|blind,revision,Qwen2.5-3B,0.25806451612903225,0.12903225806451613,0.4266935483870965,31,0.5161290322580645 37+A4|budget_tokens=512|blind,revision,Qwen2.5-3B,0.2,0.1125,0.3,80,0.6125
38−A4|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.2,0.0,0.4,15,0.3333333333333333 38+A4|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.325,0.175,0.475,40,0.325
39−A4|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.3333333333333333,0.13333333333333333,0.5333333333333333,15,0.26666666666666666 39+A4|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.475,0.325,0.625,40,0.15
40−A4|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0 40+A4|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,80,1.0
41−A4|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,30,1.0 41+A4|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,80,1.0
42−A4|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.6666666666666666 42+A4|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.025,0.0,0.075,40,0.575
43−A4|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.4,0.13333333333333333,0.6666666666666666,15,0.6 43+A4|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.325,0.175,0.475,40,0.6
44−A5|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.3333333333333333 44+A5|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.0,0.0,0.0,40,0.475
45−A5|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.4666666666666667 45+A5|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,40,0.325
46−A5|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.13333333333333333,0.0,0.3333333333333333,15,0.8666666666666667 46+A5|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.1,0.025,0.2,40,0.9
47−A5|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.2,0.06666666666666667,0.4,15,0.7333333333333333 47+A5|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.15,0.05,0.275,40,0.75
48−A5|budget_tokens=512|blind,negation,Llama-3.2-3B,0.5,0.36666666666666664,0.6333333333333333,30,0.3 48+A5|budget_tokens=512|blind,negation,Llama-3.2-3B,0.5,0.375,0.6,80,0.375
49−A5|budget_tokens=512|blind,negation,Qwen2.5-3B,0.5333333333333333,0.36666666666666664,0.7,30,0.43333333333333335 49+A5|budget_tokens=512|blind,negation,Qwen2.5-3B,0.4875,0.36843750000000014,0.6,80,0.475
50−A5|budget_tokens=512|blind,revision,Llama-3.2-3B,0.13333333333333333,0.0,0.3,30,0.7666666666666667 50+A5|budget_tokens=512|blind,revision,Llama-3.2-3B,0.075,0.025,0.15,80,0.8375
51−A5|budget_tokens=512|blind,revision,Qwen2.5-3B,0.1,0.0,0.26666666666666666,30,0.8333333333333334 51+A5|budget_tokens=512|blind,revision,Qwen2.5-3B,0.0375,0.0,0.1,80,0.85
52−A5|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.13333333333333333,0.0,0.3333333333333333,15,0.8666666666666667 52+A5|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.1,0.025,0.2,40,0.85
53−A5|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.0,0.0,0.0,15,0.8666666666666667 53+A5|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.075,0.0,0.175,40,0.8
54−A5|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0 54+A5|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,80,1.0
55−A5|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,30,1.0 55+A5|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,80,1.0
56−A5|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.7333333333333333 56+A5|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,40,0.75
57−A5|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.2,0.0,0.4,15,0.3333333333333333 57+A5|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.15,0.05,0.275,40,0.45
58−B2|bits=8,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 58+B2|bits=8,aggregation,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
59−B2|bits=8,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.0 59+B2|bits=8,aggregation,Qwen2.5-3B,0.0,0.0,0.0,40,0.0
60−B2|bits=8,multi_hop,Llama-3.2-3B,1.0,1.0,1.0,15,0.0 60+B2|bits=8,multi_hop,Llama-3.2-3B,1.0,1.0,1.0,40,0.0
61−B2|bits=8,multi_hop,Qwen2.5-3B,1.0,1.0,1.0,15,0.0 61+B2|bits=8,multi_hop,Qwen2.5-3B,1.0,1.0,1.0,40,0.0
62−B2|bits=8,negation,Llama-3.2-3B,0.9,0.8,1.0,30,0.0 62+B2|bits=8,negation,Llama-3.2-3B,0.9,0.825,0.9625,80,0.0
63−B2|bits=8,negation,Qwen2.5-3B,1.0,1.0,1.0,30,0.0 63+B2|bits=8,negation,Qwen2.5-3B,0.9375,0.8809375000000002,0.9875,80,0.0
64−B2|bits=8,revision,Llama-3.2-3B,0.7333333333333333,0.5825000000000004,0.8841666666666659,30,0.0 64+B2|bits=8,revision,Llama-3.2-3B,0.7625,0.675,0.85,80,0.0
65−B2|bits=8,revision,Qwen2.5-3B,0.9333333333333333,0.8333333333333334,1.0,30,0.0 65+B2|bits=8,revision,Qwen2.5-3B,0.875,0.8125,0.9375,80,0.0
66−B2|bits=8,temporal,Llama-3.2-3B,0.4666666666666667,0.2,0.7333333333333333,15,0.0 66+B2|bits=8,temporal,Llama-3.2-3B,0.625,0.475,0.775,40,0.0
67−B2|bits=8,temporal,Qwen2.5-3B,0.4,0.13333333333333333,0.6666666666666666,15,0.0 67+B2|bits=8,temporal,Qwen2.5-3B,0.55,0.4,0.7,40,0.0
68−B2|bits=8,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0 68+B2|bits=8,unanswerable,Llama-3.2-3B,0.95,0.9,0.9875,80,0.95
69−B2|bits=8,unanswerable,Qwen2.5-3B,0.9666666666666667,0.9,1.0,30,0.9666666666666667 69+B2|bits=8,unanswerable,Qwen2.5-3B,0.975,0.9375,1.0,80,0.975
70−B2|bits=8,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 70+B2|bits=8,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
71−B2|bits=8,unit_conversion,Qwen2.5-3B,0.6,0.3333333333333333,0.8666666666666667,15,0.0 71+B2|bits=8,unit_conversion,Qwen2.5-3B,0.6,0.45,0.75,40,0.0
72−B3|keep_frac=0.25,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 72+B3|keep_frac=0.25,aggregation,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
73−B3|keep_frac=0.25,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.0 73+B3|keep_frac=0.25,aggregation,Qwen2.5-3B,0.0,0.0,0.0,40,0.0
74−B3|keep_frac=0.25,multi_hop,Llama-3.2-3B,0.2,0.0,0.4,15,0.3333333333333333 74+B3|keep_frac=0.25,multi_hop,Llama-3.2-3B,0.15,0.05,0.275,40,0.275
75−B3|keep_frac=0.25,multi_hop,Qwen2.5-3B,0.2,0.0,0.4,15,0.13333333333333333 75+B3|keep_frac=0.25,multi_hop,Qwen2.5-3B,0.175,0.075,0.3,40,0.1
76−B3|keep_frac=0.25,negation,Llama-3.2-3B,0.7333333333333333,0.6,0.8333333333333334,30,0.0 76+B3|keep_frac=0.25,negation,Llama-3.2-3B,0.725,0.65,0.8,80,0.0
77−B3|keep_frac=0.25,negation,Qwen2.5-3B,0.8666666666666667,0.7333333333333333,0.9666666666666667,30,0.0 77+B3|keep_frac=0.25,negation,Qwen2.5-3B,0.9125,0.85,0.975,80,0.0
78−B3|keep_frac=0.25,revision,Llama-3.2-3B,0.06666666666666667,0.0,0.16666666666666666,30,0.6666666666666666 78+B3|keep_frac=0.25,revision,Llama-3.2-3B,0.15,0.06843750000000012,0.2375,80,0.4875
79−B3|keep_frac=0.25,revision,Qwen2.5-3B,0.0,0.0,0.0,30,0.0 79+B3|keep_frac=0.25,revision,Qwen2.5-3B,0.1,0.025,0.1875,80,0.0
80−B3|keep_frac=0.25,temporal,Llama-3.2-3B,0.5333333333333333,0.26666666666666666,0.7333333333333333,15,0.0 80+B3|keep_frac=0.25,temporal,Llama-3.2-3B,0.575,0.425,0.725,40,0.0
81−B3|keep_frac=0.25,temporal,Qwen2.5-3B,0.5333333333333333,0.26666666666666666,0.7333333333333333,15,0.0 81+B3|keep_frac=0.25,temporal,Qwen2.5-3B,0.55,0.41187500000000027,0.7,40,0.0
82−B3|keep_frac=0.25,unanswerable,Llama-3.2-3B,0.9333333333333333,0.8,1.0,30,0.9333333333333333 82+B3|keep_frac=0.25,unanswerable,Llama-3.2-3B,0.9375,0.875,0.9875,80,0.9375
83−B3|keep_frac=0.25,unanswerable,Qwen2.5-3B,0.9666666666666667,0.9,1.0,30,0.9666666666666667 83+B3|keep_frac=0.25,unanswerable,Qwen2.5-3B,0.9875,0.9625,1.0,80,0.9875
84−B3|keep_frac=0.25,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.7333333333333333 84+B3|keep_frac=0.25,unit_conversion,Llama-3.2-3B,0.025,0.0,0.075,40,0.65
85−B3|keep_frac=0.25,unit_conversion,Qwen2.5-3B,0.06666666666666667,0.0,0.2,15,0.0 85+B3|keep_frac=0.25,unit_conversion,Qwen2.5-3B,0.075,0.0,0.15,40,0.0
86−B5|keep_frac=0.25|aware,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 86+B5|keep_frac=0.25|aware,aggregation,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
87−B5|keep_frac=0.25|aware,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.0 87+B5|keep_frac=0.25|aware,aggregation,Qwen2.5-3B,0.0,0.0,0.0,40,0.0
88−B5|keep_frac=0.25|aware,multi_hop,Llama-3.2-3B,1.0,1.0,1.0,15,0.0 88+B5|keep_frac=0.25|aware,multi_hop,Llama-3.2-3B,0.95,0.875,1.0,40,0.0
89−B5|keep_frac=0.25|aware,multi_hop,Qwen2.5-3B,0.5333333333333333,0.26666666666666666,0.8,15,0.0 89+B5|keep_frac=0.25|aware,multi_hop,Qwen2.5-3B,0.575,0.425,0.725,40,0.0
90−B5|keep_frac=0.25|aware,negation,Llama-3.2-3B,0.8666666666666667,0.7666666666666667,0.9666666666666667,30,0.0 90+B5|keep_frac=0.25|aware,negation,Llama-3.2-3B,0.85,0.7625,0.925,80,0.0
91−B5|keep_frac=0.25|aware,negation,Qwen2.5-3B,1.0,1.0,1.0,30,0.0 91+B5|keep_frac=0.25|aware,negation,Qwen2.5-3B,0.9625,0.925,1.0,80,0.0
92−B5|keep_frac=0.25|aware,revision,Llama-3.2-3B,0.7333333333333333,0.6158333333333337,0.8666666666666667,30,0.0 92+B5|keep_frac=0.25|aware,revision,Llama-3.2-3B,0.7625,0.6875,0.8375,80,0.0
93−B5|keep_frac=0.25|aware,revision,Qwen2.5-3B,0.3,0.16666666666666666,0.43333333333333335,30,0.0 93+B5|keep_frac=0.25|aware,revision,Qwen2.5-3B,0.275,0.175,0.375,80,0.0
94−B5|keep_frac=0.25|aware,temporal,Llama-3.2-3B,0.6,0.4,0.8,15,0.0 94+B5|keep_frac=0.25|aware,temporal,Llama-3.2-3B,0.7,0.55,0.85,40,0.0
95−B5|keep_frac=0.25|aware,temporal,Qwen2.5-3B,0.5333333333333333,0.26666666666666666,0.8,15,0.0 95+B5|keep_frac=0.25|aware,temporal,Qwen2.5-3B,0.55,0.4,0.7131249999999993,40,0.0
96−B5|keep_frac=0.25|aware,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0 96+B5|keep_frac=0.25|aware,unanswerable,Llama-3.2-3B,0.95,0.9,0.9875,80,0.95
97−B5|keep_frac=0.25|aware,unanswerable,Qwen2.5-3B,0.9666666666666667,0.9,1.0,30,0.9666666666666667 97+B5|keep_frac=0.25|aware,unanswerable,Qwen2.5-3B,0.975,0.9375,1.0,80,0.975
98−B5|keep_frac=0.25|aware,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 98+B5|keep_frac=0.25|aware,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
99−B5|keep_frac=0.25|aware,unit_conversion,Qwen2.5-3B,0.4666666666666667,0.2,0.7333333333333333,15,0.0 99+B5|keep_frac=0.25|aware,unit_conversion,Qwen2.5-3B,0.6,0.425,0.725,40,0.0
modified runs/raw/E7_qtypes.jsonl +3080 −0
Large diff (3,084 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere