SPB Git
spboucher.ai

spb/wp30_uqo

Public

WP30: final analysis (E7 exploratory), paper layout pass, FINAL_REPORT, index updates

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Simon-Pierre Boucher committed (Oct 30, 2026) parent 2ccc6b7 Browse files

91 changed files +3,272 −181

91 changed files
Load all 91 files Unified Split
modified FINAL_REPORT.md +1 −1
@@ -19,7 +19,7 @@ Paper: `paper/WP30_Boucher_Cache_Compression_Knowledge_Transfer.pdf` (LaTeX in `
19 19 | ED_hosted_core | A0–A6 messages → claude-haiku-4-5 and gpt-5-nano, provider prefix caching | 672 (0.53 USD; pilots 0.19 USD) | M3U96b + APIs | complete |
20 20 | E8 | router fitted on dev pilot, evaluated off-policy on the full-factorial test records | 640 instances × 2 regimes | laptop | complete |
21 21 | E9 | minimal reproduction from a clean process | 64 trials | laptop | 64/64 identical outputs and scores |
22 −| E7_qtypes | EXPLORATORY: question types 6–15 (negation, revision, temporal, aggregation, units, unanswerable) | 5 600 planned | laptop | see STATUS |
22 +| E7_qtypes | EXPLORATORY: question types 6–15 (negation, revision, temporal, aggregation, units, unanswerable) | 5 600 | laptop | complete (appendix, exploratory) |
23 23
24 24 Models: `mlx-community/Qwen2.5-3B-Instruct-bf16`, `Llama-3.2-3B-Instruct-bf16`, `Qwen2.5-1.5B-Instruct-bf16` (revisions and weight hashes in
25 25 `evidence/models.json`). Hardware: MacBook Pro M5 Max 48 GB (laptop), two Mac Studio M3 Ultra 96 GB (M3U96a/b) on a wired 1 GbE LAN.
modified evidence/pages/p-02.png +0 −0

Binary file not shown.

modified evidence/pages/p-03.png +0 −0

Binary file not shown.

modified evidence/pages/p-04.png +0 −0

Binary file not shown.

modified evidence/pages/p-05.png +0 −0

Binary file not shown.

modified evidence/pages/p-06.png +0 −0

Binary file not shown.

modified evidence/pages/p-07.png +0 −0

Binary file not shown.

modified evidence/pages/p-08.png +0 −0

Binary file not shown.

modified evidence/pages/p-09.png +0 −0

Binary file not shown.

modified evidence/pages/p-10.png +0 −0

Binary file not shown.

modified evidence/pages/p-11.png +0 −0

Binary file not shown.

modified evidence/pages/p-12.png +0 −0

Binary file not shown.

modified evidence/pages/p-13.png +0 −0

Binary file not shown.

modified evidence/pages/p-14.png +0 −0

Binary file not shown.

modified evidence/pages/p-15.png +0 −0

Binary file not shown.

modified evidence/pages/p-16.png +0 −0

Binary file not shown.

modified evidence/pages/p-17.png +0 −0

Binary file not shown.

modified evidence/pages/p-18.png +0 −0

Binary file not shown.

modified evidence/pages/p-19.png +0 −0

Binary file not shown.

modified evidence/pages/p-20.png +0 −0

Binary file not shown.

modified evidence/pages/p-21.png +0 −0

Binary file not shown.

modified evidence/pages/p-22.png +0 −0

Binary file not shown.

modified evidence/pages/p-23.png +0 −0

Binary file not shown.

modified evidence/pages/p-24.png +0 −0

Binary file not shown.

modified evidence/pages/p-25.png +0 −0

Binary file not shown.

modified evidence/pages/p-26.png +0 −0

Binary file not shown.

modified evidence/pages/p-27.png +0 −0

Binary file not shown.

modified evidence/pages/p-28.png +0 −0

Binary file not shown.

modified evidence/pages/p-29.png +0 −0

Binary file not shown.

modified evidence/pages/p-30.png +0 −0

Binary file not shown.

modified evidence/pages/p-31.png +0 −0

Binary file not shown.

modified evidence/pages/p-32.png +0 −0

Binary file not shown.

modified evidence/pages/p-33.png +0 −0

Binary file not shown.

modified evidence/pages/p-34.png +0 −0

Binary file not shown.

modified evidence/pages/p-35.png +0 −0

Binary file not shown.

modified evidence/pages/p-36.png +0 −0

Binary file not shown.

modified evidence/pages/p-37.png +0 −0

Binary file not shown.

modified evidence/pages/p-38.png +0 −0

Binary file not shown.

modified evidence/pages/p-39.png +0 −0

Binary file not shown.

modified evidence/pages/p-40.png +0 −0

Binary file not shown.

modified evidence/pages/p-41.png +0 −0

Binary file not shown.

modified evidence/pages/p-42.png +0 −0

Binary file not shown.

modified evidence/pages/p-43.png +0 −0

Binary file not shown.

modified evidence/pages/p-44.png +0 −0

Binary file not shown.

added evidence/pages/p-45.png +0 −0

Binary file not shown.

added evidence/pages/p-46.png +0 −0

Binary file not shown.

added evidence/pages/p-47.png +0 −0

Binary file not shown.

added evidence/pages/p-48.png +0 −0

Binary file not shown.

added evidence/pages/p-49.png +0 −0

Binary file not shown.

modified evidence/pages/sheet_01.png +0 −0

Binary file not shown.

modified evidence/pages/sheet_02.png +0 −0

Binary file not shown.

modified evidence/pages/sheet_03.png +0 −0

Binary file not shown.

modified evidence/pages/sheet_04.png +0 −0

Binary file not shown.

modified evidence/pages/sheet_05.png +0 −0

Binary file not shown.

modified evidence/pages/sheet_06.png +0 −0

Binary file not shown.

added evidence/pages/sheet_07.png +0 −0

Binary file not shown.

added paper/WP30_Boucher_Cache_Compression_Knowledge_Transfer.pdf +0 −0

Binary file not shown.

modified paper/appendix/extra_tables.tex +10 −0
@@ -11,6 +11,16 @@
11 11 \end{table}
12 12
13 13 \subsection{Exploratory: question types not covered by the core (A-004)}
14 +\label{app:qtypes}
15 +The five-question core limit kept only single-fact, exact-copy and multi-hop questions. This exploratory run (5\,600 trials; amendment A-004;
16 +no confirmatory test) asks the remaining ten question types of the 40 synthetic test documents for seven conditions. Three observations, all
17 +exploratory: (i) \emph{revision/precedence} questions (``current budget under the later-date rule'') are the ones that lossy text loses most---full
18 +text 75--88\,\%, blind summaries 12--20\,\%, structured evidence 4--8\,\%, sink+recent 10--15\,\%---while the question-aware attention cache keeps
19 +76\,\% on Llama but only 28\,\% on Qwen; (ii) \emph{negation} (cancelled vs postponed, yes/no) is answered at 72--96\,\% by every condition that
20 +retains anything, close to the base rate of the majority answer, and must not be read as evidence of fact transfer; (iii) \emph{aggregation}
21 +(summing five to six amounts) is at 0\,\% for every condition including full text, and \emph{unit conversion} is solved only by Qwen (57--60\,\%
22 +with full text, 8-bit cache or aware eviction) and never from a summary---these are receiver-capability limits, not communication effects.
23 +Unanswerable questions were abstained on correctly 94--100\,\% of the time in every condition.
14 24 \begin{figure}[H]\centering\includegraphics[width=\textwidth]{figA_qtypes_exploratory.pdf}
15 25 \caption{EXPLORATORY. Accuracy on negation, revision/precedence, temporal order, aggregation, unit conversion and unanswerable questions
16 26 (questions 6--15 of the 40 synthetic test documents) for a subset of methods; no confirmatory test attached.}\end{figure}
modified paper/figures/fig2_quality_vs_bytes.pdf +0 −0

Binary file not shown.

modified paper/figures/fig3_quality_vs_latency_memory.pdf +0 −0

Binary file not shown.

modified paper/figures/fig4_latency_decomposition.pdf +0 −0

Binary file not shown.

modified paper/figures/fig5_handoffs.pdf +0 −0

Binary file not shown.

modified paper/figures/fig6_breakeven.pdf +0 −0

Binary file not shown.

modified paper/figures/fig7_router.pdf +0 −0

Binary file not shown.

modified paper/figures/figA_qtypes_exploratory.pdf +0 −0

Binary file not shown.

modified paper/figures/figA_quality_vs_bytes_aware.pdf +0 −0

Binary file not shown.

modified paper/figures/figA_quality_vs_bytes_blind.pdf +0 −0

Binary file not shown.

modified paper/main.pdf +0 −0

Binary file not shown.

modified paper/main.tex +1 −0
@@ -95,6 +95,7 @@
95 95 \usepackage{longtable}
96 96 \usepackage{array}
97 97 \usepackage{pdflscape}
98 +\usepackage{rotating}
98 99 \usepackage{adjustbox}
99 100 \newcommand{\WPnumber}{30}
100 101 \newcommand{\WPtitle}{Beyond Text: Cache Compression and Knowledge Transfer in Multi-Agent Systems}
modified paper/sections/abstract_text.tex +10 −14
@@ -2,20 +2,16 @@
2 2 Agents built on language models hand information to one another as text, summaries, structured evidence, shared-memory references or---when
3 3 they share a checkpoint---attention state (KV cache). We ask under which task, model, cache, network and reuse conditions each representation
4 4 preserves task-relevant information at the lowest end-to-end cost, and answer with a reproducible study executed by an autonomous research
5 −operator (Claude Code) on Apple-Silicon hardware with the MLX runtime, whose caches are inspectable tensors. Twenty-two conditions (text
6 −truncation, extractive selection, summaries, structured evidence with spans, references with metered retrieval, a lossless codec, LLMLingua-2,
7 −native and serialized caches, packed 8/4-bit quantization, sink-plus-recent, random, SnapKV-style and PyramidKV-style eviction) are measured on
8 −Qwen2.5-3B and Llama-3.2-3B across private-fact ledgers, LongBench QA and executable tasks, in question-blind and question-aware regimes, with
5 +operator (Claude Code) on Apple-Silicon hardware with the MLX runtime, whose caches are inspectable tensors. Twenty-two conditions (truncation, extractive selection, summaries,
6 +structured evidence, metered retrieval, a lossless codec, LLMLingua-2, native/serialized caches, packed 8/4-bit quantization, four eviction
7 +rules) are measured on Qwen2.5-3B and Llama-3.2-3B across private-fact ledgers, LongBench QA and executable tasks, in question-blind and question-aware regimes, with
9 8 measured bytes, synchronized latency spans and peak memory (13\,280 trials; frozen protocol, cluster bootstrap, Holm-corrected tests). Further
10 −experiments measure two-node cache transfer over a 1~GbE LAN (324 runs), relays of 1--8 handoffs with an evaluator-only fact ledger (7\,200),
11 −fan-in with conflicting evidence (1\,600), a same-family learned KV mapper, hosted receivers (672) and a development-fitted router. Lossless
9 +experiments measure two-node cache transfer over a 1~GbE LAN, relays of 1--8 handoffs with an evaluator-only fact ledger, fan-in with
10 +conflicting evidence, a same-family learned KV mapper, hosted receivers and a development-fitted router. Lossless
12 11 representations are interchangeable in quality and differ only in cost, whose ranking inverts between cold and reused regimes; on the measured
13 −link, moving an 8-bit cache beats re-reading the text at every length. Lossy choices are model- and regime-dependent: 4-bit quantization is free
14 −for Llama and fatal for Qwen; attention-based eviction keeps 78--98\,\% of private-fact accuracy with the question and 12--19\,\% without it, and
15 −no eviction survives a code specification. At a 512-token budget prose summaries beat structured evidence (38--44 vs 12--15\,\%), relays are
16 −nearly lossless after the first hop and invent almost nothing, and metered repair buys a few points for kilobytes of re-reading. A ridge KV mapper
17 −transfers no content while producing coherent abstention, and an invalid cache concatenation yields plausible answers---both argue for
18 −compatibility manifests and negative controls. The router halves amortized latency at equal quality when questions are unknown and does not help
19 −otherwise. Cross-family cache transfer was not executed; the study is partial in that respect. Code, records, protocol, checkpoint and a
20 −minimal reproduction (64/64 identical outputs) are released.%
12 +link, moving an 8-bit cache beats re-reading the text at every length. Lossy choices are model- and regime-dependent: 4-bit quantization is free for Llama and
13 +fatal for Qwen; attention-based eviction keeps 78--98\,\% of private-fact accuracy with the question and 12--19\,\% without it; no eviction
14 +survives a code specification. At a 512-token budget prose summaries beat structured evidence (38--44 vs 12--15\,\%); relays are
15 +nearly lossless after the first hop and invent almost nothing. A ridge KV mapper transfers no
16 +content while producing coherent abstention---an argument for compatibility manifests and negative controls. The router halves amortized latency at equal quality when questions are unknown. Cross-family cache transfer was not executed. Code, records, protocol, checkpoint and a minimal reproduction are released.%
21 17 }
modified paper/sections/adaptive.tex +2 −2
@@ -10,11 +10,11 @@ action was run on every instance), and the router's own lookup time (${\approx}1
10 10 Table~\ref{tab:router} compare it with the development-best static action, full context, the native cache, the question-aware attention
11 11 cache and the infeasible oracle that picks the best action per instance.
12 12
13 −\begin{figure}[t]\centering\includegraphics[width=\textwidth]{fig7_router.pdf}
13 +\begin{figure}[H]\centering\includegraphics[width=\textwidth]{fig7_router.pdf}
14 14 \caption{Dev-fitted router against static policies on held-out test instances (both pairs, all families; $n=640$ per regime). The oracle is
15 15 an infeasible upper reference.}\label{fig:router}\end{figure}
16 16
17 −\begin{table}[t]\centering\footnotesize
17 +\begin{table}[H]\centering\footnotesize
18 18 \caption{Router evaluation on held-out test instances. Quality in percent with 95\,\% cluster-bootstrap CI; cost = mean amortized latency per
19 19 query (ms, document build divided by the number of reusing questions); fallback = share of instances routed by the nearest-cell fallback.}\label{tab:router}
20 20 \begin{tabular}{@{}llrrrr@{}}\toprule
modified paper/sections/formulation.tex +4 −4
@@ -49,10 +49,10 @@ with batch $B$, layers $L$, stored positions $T$, KV heads $H_{KV}$, head dimens
49 49 \eqref{eq:mkv} only as a consistency check: the harness measures stored tensor bytes from the arrays themselves (which, for the
50 50 packed quantized caches, include the uint32 words, the per-group scales and biases) and measures the serialized payload from the file
51 51 written to disk, including the compatibility manifest. Two ratios are reported with explicit denominators,
52 −\begin{equation}
53 − CR_{\text{payload}} = \frac{\text{reference serialized payload bytes}}{\text{method serialized payload bytes}},\qquad
54 − CR_{KV} = \frac{\text{native cache tensor bytes}}{\text{stored cache tensor and metadata bytes}},
55 −\end{equation}
52 +\begin{align}
53 + CR_{\text{payload}} &= \frac{\text{reference serialized payload bytes}}{\text{method serialized payload bytes}},\\
54 + CR_{KV} &= \frac{\text{native cache tensor bytes}}{\text{stored cache tensor and metadata bytes}},
55 +\end{align}
56 56 and a text-token ratio is never treated as a cache-position ratio.
57 57
58 58 \paragraph{Timing boundaries.} Each trial records separate wall-clock spans with device synchronization before and after
modified paper/sections/methods.tex +1 −1
@@ -25,7 +25,7 @@ and \emph{custom} baselines; the label of every method is in Table~\ref{tab:taxo
25 25 \item[A7 lossless transport codec.] The raw text is compressed with Zstandard (level~3) for transport and decompressed at the receiver;
26 26 the receiver prefill is unchanged. Codec times are measured.
27 27 \item[A8 learned prompt compression.] LLMLingua-2 \citep{pan2024llmlingua2}, official package (\texttt{llmlingua}~0.2.2) with the
28 − released \texttt{xlm-roberta-large-meetingbank} compressor, rate $=$ budget/source tokens, run on the Metal backend of PyTorch
28 + released XLM-RoBERTa-large MeetingBank compressor, rate $=$ budget/source tokens, run on the Metal backend of PyTorch
29 29 (fp32). It is question-agnostic by construction.
30 30 \end{description}
31 31 A4 and A5 use the same sender checkpoint, the same budget and prompts of comparable length; both prompts were tuned on development
modified paper/sections/reproducibility.tex +4 −3
@@ -26,9 +26,10 @@ the provider credentials and reproduce only up to provider nondeterminism.
26 26
27 27 \paragraph{Reproduction commands.}
28 28 \begin{footnotesize}\begin{verbatim}
29 −make reproduce-analysis # tables + figures from runs/raw, provenance audit
30 −python -m wp30 reproduce --profile reproduction/minimal.yaml # re-run a fixed subset, compare greedy outputs
31 −make paper # compile this document
29 +make reproduce-analysis # tables + figures from runs/raw, provenance audit
30 +python -m wp30 reproduce --profile reproduction/minimal.yaml
31 + # re-run a fixed subset, compare greedy outputs
32 +make paper # compile this document
32 33 \end{verbatim}\end{footnotesize}
33 34 Greedy decoding on MLX is deterministic for an identical prefill chunking but not bitwise across chunkings (Section~\ref{sec:correctness});
34 35 the reproduction report lists identical output hashes and identical scores separately.
modified paper/sections/results_frontier.tex +6 −3
@@ -11,12 +11,12 @@ condition (median payload, mean quality, 95\,\% cluster-bootstrap CI). Text meth
11 11 $10^7$--$10^9$ bytes: a 3B-parameter cache of a document is three to four orders of magnitude larger than the document. Question-aware and
12 12 question-blind conditions are both shown here; Appendix figures separate them.}\label{fig:frontier}\end{figure}
13 13
14 −\begin{table}[p]\centering\scriptsize
14 +\begin{sidewaystable}[p]\centering\scriptsize
15 15 \caption{Primary comparison: quality (\%, with 95\,\% cluster-bootstrap CI) per condition, family and pair. Synthetic = private-fact accuracy
16 16 (5 questions per document: single fact, exact copy, multi-hop), exec = hidden-test pass (all 5 tests), QASPER/2Wiki = LongBench F1.
17 17 ``blind''/``aware'' = question access of the compressor. Latency and bytes for the synthetic family in Table~\ref{tab:cost}.}\label{tab:primary}
18 18 \adjustbox{max width=\textwidth}{\input{tables/table_primary}}
19 −\end{table}
19 +\end{sidewaystable}
20 20
21 21 \paragraph{Lossless representations are interchangeable; nothing else is.} Full text (A1), zstd-compressed text (A7), the native cache
22 22 (B0), the serialized and reloaded cache (B1) and the 8-bit packed cache (B2-8) give the same answers within noise on every family and pair
@@ -53,7 +53,10 @@ simply absent from the message; the receiver answers ``unknown'' (correctly abst
53 53 retrieval (A6) repair little (\QSyntheticQwenAXBRBlind{}\,\%/\QSyntheticLlamaAXBRBlind{}\,\%) because the reference index inherits the same
54 54 coverage. Deterministic extraction (A3) is the strongest cheap method: blind centrality selection keeps \QSyntheticQwenAHBBlind{}\,\%, and with
55 55 the question (BM25) \QSyntheticQwenAHBAware{}\,\%/\QSyntheticLlamaAHBAware{}\,\%---indistinguishable from full text at 1/15 of the bytes.
56 −Head/tail truncation (A2) keeps exactly the facts that fall in the kept windows (\QSyntheticQwenAWB{}\,\%). On the executable family the
56 +Head/tail truncation (A2) keeps exactly the facts that fall in the kept windows (\QSyntheticQwenAWB{}\,\%). The question types that the
57 +five-question core did not cover (negation, revision/precedence, temporal order, aggregation, units, unanswerable) were run as an exploratory
58 +extension after the core and are reported, labelled as such, in Appendix~\ref{app:qtypes}; the headline there is that timestamped revisions are
59 +the facts that summaries lose most. On the executable family the
57 60 ordering changes: blind extractive selection keeps the whole specification (\QExecQwenAHBBlind{}\,\%/\QExecLlamaAHBBlind{}\,\%) because the
58 61 rules are dense in digits and identifiers, and the Llama summary preserves the rules better than the Qwen summary
59 62 (\QExecLlamaAFBBlind{} vs \QExecQwenAFBBlind{}\,\%). LLMLingua-2 (A8) at the same rate removes the very tokens the questions ask for
modified paper/sections/results_hosted.tex +1 −1
@@ -10,7 +10,7 @@ receiver cannot recover a number that the message does not contain, and the refe
10 10 the 2\,000-byte retrieval budget. On QASPER the hosted models score higher F1 than the local receivers from the same material (0.48 versus
11 11 0.35 for A1), i.e., receiver capacity matters when the material is complete.
12 12
13 −\begin{table}[t]\centering\scriptsize
13 +\begin{table}[H]\centering\scriptsize
14 14 \caption{Hosted receivers on messages built by the local sender (test split, first 8 documents per family). Quality = accuracy (synthetic,
15 15 exec) or F1 (QASPER) with 95\,\% cluster-bootstrap CI; cost = estimated per-query cost from provider price tables (m\$ = 0.001~USD); TTFT
16 16 = time to first streamed content token (Anthropic only; the OpenAI Responses call was not streamed). ``+cache'' = documented prompt caching
modified paper/tables/numbers.tex +10 −10
@@ -1,5 +1,5 @@
1 1 % auto-generated by wp30.analysis.numbers — do not edit
2 −\newcommand{\nCoreTrials}{13,280}
2 +\newcommand{\nCoreTrials}{13,520}
3 3 \newcommand{\QExecLlamaAZ}{0}
4 4 \newcommand{\QloExecLlamaAZ}{0}
5 5 \newcommand{\QhiExecLlamaAZ}{0}
@@ -57,9 +57,9 @@
57 57 \newcommand{\QExecLlamaAEBBlind}{0}
58 58 \newcommand{\QloExecLlamaAEBBlind}{0}
59 59 \newcommand{\QhiExecLlamaAEBBlind}{0}
60 −\newcommand{\BExecLlamaAEBBlind}{--}
61 −\newcommand{\TcoldExecLlamaAEBBlind}{0}
62 −\newcommand{\TwarmExecLlamaAEBBlind}{0}
60 +\newcommand{\BExecLlamaAEBBlind}{2,398}
61 +\newcommand{\TcoldExecLlamaAEBBlind}{1,808}
62 +\newcommand{\TwarmExecLlamaAEBBlind}{1,636}
63 63 \newcommand{\QExecLlamaBZ}{95}
64 64 \newcommand{\QloExecLlamaBZ}{88}
65 65 \newcommand{\QhiExecLlamaBZ}{100}
@@ -786,12 +786,12 @@
786 786 \newcommand{\BSyntheticLlamaASZstd}{2,726}
787 787 \newcommand{\TcoldSyntheticLlamaASZstd}{2,857}
788 788 \newcommand{\TwarmSyntheticLlamaASZstd}{2,857}
789 −\newcommand{\QSyntheticLlamaAEBBlind}{0}
790 −\newcommand{\QloSyntheticLlamaAEBBlind}{0}
791 −\newcommand{\QhiSyntheticLlamaAEBBlind}{0}
792 −\newcommand{\BSyntheticLlamaAEBBlind}{--}
793 −\newcommand{\TcoldSyntheticLlamaAEBBlind}{0}
794 −\newcommand{\TwarmSyntheticLlamaAEBBlind}{0}
789 +\newcommand{\QSyntheticLlamaAEBBlind}{16}
790 +\newcommand{\QloSyntheticLlamaAEBBlind}{10}
791 +\newcommand{\QhiSyntheticLlamaAEBBlind}{22}
792 +\newcommand{\BSyntheticLlamaAEBBlind}{2,583}
793 +\newcommand{\TcoldSyntheticLlamaAEBBlind}{523}
794 +\newcommand{\TwarmSyntheticLlamaAEBBlind}{108}
795 795 \newcommand{\QSyntheticLlamaBZ}{99}
796 796 \newcommand{\QloSyntheticLlamaBZ}{98}
797 797 \newcommand{\QhiSyntheticLlamaBZ}{100}
modified paper/tables/table_qtypes_exploratory.tex +14 −14
@@ -2,19 +2,19 @@
2 2 \toprule
3 3 condition / pair & aggregation & multi\_hop & negation & revision & temporal & unanswerable & unit\_conversion \\
4 4 \midrule
5 −A1 Llama-3.2-3B & 0 [0,0] & 100 [100,100] & 90 [80,100] & 73 [58,88] & 47 [20,73] & 100 [100,100] & 0 [0,0] \\
6 −A1 Qwen2.5-3B & 0 [0,0] & 100 [100,100] & 100 [100,100] & 94 [84,100] & 44 [22,69] & 97 [91,100] & 50 [25,75] \\
7 −A3 512 blind Llama-3.2-3B & 0 [0,0] & 0 [0,0] & 83 [70,93] & 73 [57,87] & 53 [33,73] & 100 [100,100] & 0 [0,0] \\
8 −A3 512 blind Qwen2.5-3B & 0 [0,0] & 0 [0,0] & 93 [83,100] & 73 [53,90] & 33 [13,53] & 100 [100,100] & 0 [0,0] \\
9 −A4 512 blind Llama-3.2-3B & 0 [0,0] & 33 [7,60] & 100 [100,100] & 17 [3,33] & 20 [0,40] & 100 [100,100] & 0 [0,0] \\
10 −A4 512 blind Qwen2.5-3B & 0 [0,0] & 44 [19,69] & 72 [53,88] & 26 [13,43] & 33 [13,53] & 100 [100,100] & 40 [13,67] \\
11 −A5 512 blind Llama-3.2-3B & 0 [0,0] & 13 [0,33] & 50 [37,63] & 13 [0,30] & 13 [0,33] & 100 [100,100] & 0 [0,0] \\
12 −A5 512 blind Qwen2.5-3B & 0 [0,0] & 20 [7,40] & 53 [37,70] & 10 [0,27] & 0 [0,0] & 100 [100,100] & 20 [0,40] \\
13 −B2 bits=8 Llama-3.2-3B & 0 [0,0] & 100 [100,100] & 90 [80,100] & 73 [58,88] & 47 [20,73] & 100 [100,100] & 0 [0,0] \\
14 −B2 bits=8 Qwen2.5-3B & 0 [0,0] & 100 [100,100] & 100 [100,100] & 93 [83,100] & 40 [13,67] & 97 [90,100] & 60 [33,87] \\
15 −B3 1/4 Llama-3.2-3B & 0 [0,0] & 20 [0,40] & 73 [60,83] & 7 [0,17] & 53 [27,73] & 93 [80,100] & 0 [0,0] \\
16 −B3 1/4 Qwen2.5-3B & 0 [0,0] & 20 [0,40] & 87 [73,97] & 0 [0,0] & 53 [27,73] & 97 [90,100] & 7 [0,20] \\
17 −B5 1/4 aware Llama-3.2-3B & 0 [0,0] & 100 [100,100] & 87 [77,97] & 73 [62,87] & 60 [40,80] & 100 [100,100] & 0 [0,0] \\
18 −B5 1/4 aware Qwen2.5-3B & 0 [0,0] & 53 [27,80] & 100 [100,100] & 30 [17,43] & 53 [27,80] & 97 [90,100] & 47 [20,73] \\
5 +A1 Llama-3.2-3B & 0 [0,0] & 100 [100,100] & 90 [82,96] & 75 [66,84] & 62 [48,78] & 95 [90,99] & 0 [0,0] \\
6 +A1 Qwen2.5-3B & 0 [0,0] & 100 [100,100] & 96 [91,100] & 88 [81,94] & 55 [40,70] & 98 [94,100] & 57 [41,72] \\
7 +A3 512 blind Llama-3.2-3B & 0 [0,0] & 0 [0,0] & 86 [79,92] & 84 [76,91] & 65 [50,80] & 100 [100,100] & 0 [0,0] \\
8 +A3 512 blind Qwen2.5-3B & 0 [0,0] & 0 [0,0] & 90 [83,95] & 70 [61,79] & 60 [45,75] & 100 [100,100] & 0 [0,0] \\
9 +A4 512 blind Llama-3.2-3B & 2 [0,8] & 30 [15,45] & 95 [89,100] & 12 [6,21] & 32 [18,48] & 100 [100,100] & 2 [0,8] \\
10 +A4 512 blind Qwen2.5-3B & 0 [0,0] & 48 [32,62] & 72 [62,81] & 20 [11,30] & 48 [32,62] & 100 [100,100] & 32 [18,48] \\
11 +A5 512 blind Llama-3.2-3B & 0 [0,0] & 10 [2,20] & 50 [38,60] & 8 [2,15] & 10 [2,20] & 100 [100,100] & 0 [0,0] \\
12 +A5 512 blind Qwen2.5-3B & 0 [0,0] & 15 [5,28] & 49 [37,60] & 4 [0,10] & 8 [0,18] & 100 [100,100] & 15 [5,28] \\
13 +B2 bits=8 Llama-3.2-3B & 0 [0,0] & 100 [100,100] & 90 [82,96] & 76 [68,85] & 62 [48,78] & 95 [90,99] & 0 [0,0] \\
14 +B2 bits=8 Qwen2.5-3B & 0 [0,0] & 100 [100,100] & 94 [88,99] & 88 [81,94] & 55 [40,70] & 98 [94,100] & 60 [45,75] \\
15 +B3 1/4 Llama-3.2-3B & 0 [0,0] & 15 [5,28] & 72 [65,80] & 15 [7,24] & 57 [42,72] & 94 [88,99] & 2 [0,8] \\
16 +B3 1/4 Qwen2.5-3B & 0 [0,0] & 18 [8,30] & 91 [85,98] & 10 [2,19] & 55 [41,70] & 99 [96,100] & 8 [0,15] \\
17 +B5 1/4 aware Llama-3.2-3B & 0 [0,0] & 95 [88,100] & 85 [76,92] & 76 [69,84] & 70 [55,85] & 95 [90,99] & 0 [0,0] \\
18 +B5 1/4 aware Qwen2.5-3B & 0 [0,0] & 57 [42,72] & 96 [92,100] & 28 [18,38] & 55 [40,71] & 98 [94,100] & 60 [42,72] \\
19 19 \bottomrule
20 20 \end{tabular}
modified results/figures/fig2_quality_vs_bytes.pdf +0 −0

Binary file not shown.

modified results/figures/fig3_quality_vs_latency_memory.pdf +0 −0

Binary file not shown.

modified results/figures/fig4_latency_decomposition.pdf +0 −0

Binary file not shown.

modified results/figures/fig5_handoffs.pdf +0 −0

Binary file not shown.

modified results/figures/fig6_breakeven.pdf +0 −0

Binary file not shown.

modified results/figures/fig7_router.pdf +0 −0

Binary file not shown.

modified results/figures/figA_qtypes_exploratory.pdf +0 −0

Binary file not shown.

modified results/figures/figA_qtypes_exploratory.png +0 −0

Binary file not shown.

modified results/figures/figA_quality_vs_bytes_aware.pdf +0 −0

Binary file not shown.

modified results/figures/figA_quality_vs_bytes_blind.pdf +0 −0

Binary file not shown.

modified results/provenance.json +30 −30
@@ -4,7 +4,7 @@
4 4 "runs/raw/E1_core.jsonl"
5 5 ],
6 6 "script": "src/wp30/analysis/analyze.py",
7 − "generated_utc": "2026-10-30T09:02:37.609345Z",
7 + "generated_utc": "2026-10-30T09:36:25.855895Z",
8 8 "note": "per-condition quality (cluster bootstrap CI) and resource medians"
9 9 },
10 10 "results/tables/E1_core_failures.csv": {
@@ -12,7 +12,7 @@
12 12 "runs/raw/E1_core.jsonl"
13 13 ],
14 14 "script": "src/wp30/analysis/analyze.py",
15 − "generated_utc": "2026-10-30T09:02:37.611126Z",
15 + "generated_utc": "2026-10-30T09:36:25.857450Z",
16 16 "note": "attempt status counts by method"
17 17 },
18 18 "results/tables/E1_core_by_question_type.csv": {
@@ -20,7 +20,7 @@
20 20 "runs/raw/E1_core.jsonl"
21 21 ],
22 22 "script": "src/wp30/analysis/analyze.py",
23 − "generated_utc": "2026-10-30T09:02:37.730657Z",
23 + "generated_utc": "2026-10-30T09:36:25.963048Z",
24 24 "note": "synthetic accuracy by question type"
25 25 },
26 26 "results/tables/E1_core_primary_tests.csv": {
@@ -28,7 +28,7 @@
28 28 "runs/raw/E1_core.jsonl"
29 29 ],
30 30 "script": "src/wp30/analysis/analyze.py",
31 − "generated_utc": "2026-10-30T09:02:37.786568Z",
31 + "generated_utc": "2026-10-30T09:36:26.010368Z",
32 32 "note": "registered primary comparisons, paired, cluster bootstrap, Holm"
33 33 },
34 34 "results/figures/fig2_quality_vs_bytes.pdf": {
@@ -36,7 +36,7 @@
36 36 "runs/raw/E1_core.jsonl"
37 37 ],
38 38 "script": "src/wp30/analysis/analyze.py",
39 − "generated_utc": "2026-10-30T09:02:38.459020Z",
39 + "generated_utc": "2026-10-30T09:36:26.618975Z",
40 40 "note": "points = conditions; x = median payload bytes; y = mean quality"
41 41 },
42 42 "results/figures/fig2_quality_vs_bytes.png": {
@@ -44,7 +44,7 @@
44 44 "runs/raw/E1_core.jsonl"
45 45 ],
46 46 "script": "src/wp30/analysis/analyze.py",
47 − "generated_utc": "2026-10-30T09:02:38.666585Z",
47 + "generated_utc": "2026-10-30T09:36:26.800030Z",
48 48 "note": "points = conditions; x = median payload bytes; y = mean quality"
49 49 },
50 50 "results/figures/figA_quality_vs_bytes_blind.pdf": {
@@ -52,7 +52,7 @@
52 52 "runs/raw/E1_core.jsonl"
53 53 ],
54 54 "script": "src/wp30/analysis/analyze.py",
55 − "generated_utc": "2026-10-30T09:02:39.176959Z",
55 + "generated_utc": "2026-10-30T09:36:27.219230Z",
56 56 "note": "points = conditions; x = median payload bytes; y = mean quality"
57 57 },
58 58 "results/figures/figA_quality_vs_bytes_blind.png": {
@@ -60,7 +60,7 @@
60 60 "runs/raw/E1_core.jsonl"
61 61 ],
62 62 "script": "src/wp30/analysis/analyze.py",
63 − "generated_utc": "2026-10-30T09:02:39.359044Z",
63 + "generated_utc": "2026-10-30T09:36:27.370309Z",
64 64 "note": "points = conditions; x = median payload bytes; y = mean quality"
65 65 },
66 66 "results/figures/figA_quality_vs_bytes_aware.pdf": {
@@ -68,7 +68,7 @@
68 68 "runs/raw/E1_core.jsonl"
69 69 ],
70 70 "script": "src/wp30/analysis/analyze.py",
71 − "generated_utc": "2026-10-30T09:02:39.746284Z",
71 + "generated_utc": "2026-10-30T09:36:27.681216Z",
72 72 "note": "points = conditions; x = median payload bytes; y = mean quality"
73 73 },
74 74 "results/figures/figA_quality_vs_bytes_aware.png": {
@@ -76,7 +76,7 @@
76 76 "runs/raw/E1_core.jsonl"
77 77 ],
78 78 "script": "src/wp30/analysis/analyze.py",
79 − "generated_utc": "2026-10-30T09:02:39.940182Z",
79 + "generated_utc": "2026-10-30T09:36:27.846184Z",
80 80 "note": "points = conditions; x = median payload bytes; y = mean quality"
81 81 },
82 82 "results/figures/fig3_quality_vs_latency_memory.pdf": {
@@ -84,7 +84,7 @@
84 84 "runs/raw/E1_core.jsonl"
85 85 ],
86 86 "script": "src/wp30/analysis/analyze.py",
87 − "generated_utc": "2026-10-30T09:02:40.285980Z",
87 + "generated_utc": "2026-10-30T09:36:28.116446Z",
88 88 "note": ""
89 89 },
90 90 "results/figures/fig3_quality_vs_latency_memory.png": {
@@ -92,7 +92,7 @@
92 92 "runs/raw/E1_core.jsonl"
93 93 ],
94 94 "script": "src/wp30/analysis/analyze.py",
95 − "generated_utc": "2026-10-30T09:02:40.475124Z",
95 + "generated_utc": "2026-10-30T09:36:28.280741Z",
96 96 "note": ""
97 97 },
98 98 "results/figures/fig4_latency_decomposition.pdf": {
@@ -100,7 +100,7 @@
100 100 "runs/raw/E1_core.jsonl"
101 101 ],
102 102 "script": "src/wp30/analysis/analyze.py",
103 − "generated_utc": "2026-10-30T09:02:40.560198Z",
103 + "generated_utc": "2026-10-30T09:36:28.355146Z",
104 104 "note": ""
105 105 },
106 106 "results/figures/fig4_latency_decomposition.png": {
@@ -108,7 +108,7 @@
108 108 "runs/raw/E1_core.jsonl"
109 109 ],
110 110 "script": "src/wp30/analysis/analyze.py",
111 − "generated_utc": "2026-10-30T09:02:40.620070Z",
111 + "generated_utc": "2026-10-30T09:36:28.408655Z",
112 112 "note": ""
113 113 },
114 114 "results/figures/fig5_handoffs.pdf": {
@@ -116,7 +116,7 @@
116 116 "runs/raw/E4_core.jsonl"
117 117 ],
118 118 "script": "src/wp30/analysis/analyze.py",
119 − "generated_utc": "2026-10-30T09:02:45.615826Z",
119 + "generated_utc": "2026-10-30T09:36:33.036763Z",
120 120 "note": "means over documents; chains defined in configs/chains*.yaml"
121 121 },
122 122 "results/figures/fig5_handoffs.png": {
@@ -124,7 +124,7 @@
124 124 "runs/raw/E4_core.jsonl"
125 125 ],
126 126 "script": "src/wp30/analysis/analyze.py",
127 − "generated_utc": "2026-10-30T09:02:45.675774Z",
127 + "generated_utc": "2026-10-30T09:36:33.089912Z",
128 128 "note": "means over documents; chains defined in configs/chains*.yaml"
129 129 },
130 130 "results/tables/E4_core_handoffs.csv": {
@@ -132,7 +132,7 @@
132 132 "runs/raw/E4_core.jsonl"
133 133 ],
134 134 "script": "src/wp30/analysis/analyze.py",
135 − "generated_utc": "2026-10-30T09:02:45.778336Z",
135 + "generated_utc": "2026-10-30T09:36:33.182901Z",
136 136 "note": "repeated handoffs"
137 137 },
138 138 "results/tables/E4_core_h4_primary_tests.csv": {
@@ -140,7 +140,7 @@
140 140 "runs/raw/E4_core.jsonl"
141 141 ],
142 142 "script": "src/wp30/analysis/analyze.py",
143 − "generated_utc": "2026-10-30T09:02:45.791552Z",
143 + "generated_utc": "2026-10-30T09:36:33.192836Z",
144 144 "note": "registered primary comparisons, paired, cluster bootstrap, Holm"
145 145 },
146 146 "results/tables/E5_core_fanin.csv": {
@@ -148,7 +148,7 @@
148 148 "runs/raw/E5_core.jsonl"
149 149 ],
150 150 "script": "src/wp30/analysis/analyze.py",
151 − "generated_utc": "2026-10-30T09:02:45.898241Z",
151 + "generated_utc": "2026-10-30T09:36:33.287122Z",
152 152 "note": "fan-in by method and question type"
153 153 },
154 154 "results/figures/fig6_breakeven.pdf": {
@@ -156,7 +156,7 @@
156 156 "runs/raw/E6_core.jsonl"
157 157 ],
158 158 "script": "src/wp30/analysis/analyze.py",
159 − "generated_utc": "2026-10-30T09:02:45.992041Z",
159 + "generated_utc": "2026-10-30T09:36:33.373261Z",
160 160 "note": "left = measured (medians, p25-p75); right = analytic model from measured bandwidth and prefill slope; dashed 10 Gb/s is hypothetical"
161 161 },
162 162 "results/figures/fig6_breakeven.png": {
@@ -164,7 +164,7 @@
164 164 "runs/raw/E6_core.jsonl"
165 165 ],
166 166 "script": "src/wp30/analysis/analyze.py",
167 − "generated_utc": "2026-10-30T09:02:46.065845Z",
167 + "generated_utc": "2026-10-30T09:36:33.443566Z",
168 168 "note": "left = measured (medians, p25-p75); right = analytic model from measured bandwidth and prefill slope; dashed 10 Gb/s is hypothetical"
169 169 },
170 170 "results/tables/E6_network.csv": {
@@ -172,7 +172,7 @@
172 172 "runs/raw/E6_core.jsonl"
173 173 ],
174 174 "script": "src/wp30/analysis/analyze.py",
175 − "generated_utc": "2026-10-30T09:02:46.070991Z",
175 + "generated_utc": "2026-10-30T09:36:33.447916Z",
176 176 "note": "two-node transfer measurements"
177 177 },
178 178 "results/tables/ED_hosted.csv": {
@@ -180,7 +180,7 @@
180 180 "runs/raw/ED_hosted_core.jsonl"
181 181 ],
182 182 "script": "src/wp30/analysis/analyze.py",
183 − "generated_utc": "2026-10-30T09:02:46.140839Z",
183 + "generated_utc": "2026-10-30T09:36:33.511504Z",
184 184 "note": "hosted receivers; cost estimated from provider price tables"
185 185 },
186 186 "results/tables/E3_core_transfer.csv": {
@@ -189,7 +189,7 @@
189 189 "runs/raw/E3_core_sender.jsonl"
190 190 ],
191 191 "script": "src/wp30/analysis/analyze.py",
192 − "generated_utc": "2026-10-30T09:02:46.269308Z",
192 + "generated_utc": "2026-10-30T09:36:33.631124Z",
193 193 "note": "same-family mapper transfer vs text handoffs and references"
194 194 },
195 195 "results/tables/E3_core_primary_tests.csv": {
@@ -198,7 +198,7 @@
198 198 "runs/raw/E3_core_sender.jsonl"
199 199 ],
200 200 "script": "src/wp30/analysis/analyze.py",
201 − "generated_utc": "2026-10-30T09:02:46.347449Z",
201 + "generated_utc": "2026-10-30T09:36:33.680469Z",
202 202 "note": "registered primary comparisons, paired, cluster bootstrap, Holm"
203 203 },
204 204 "results/tables/E7_qtypes.csv": {
@@ -206,7 +206,7 @@
206 206 "runs/raw/E7_qtypes.jsonl"
207 207 ],
208 208 "script": "src/wp30/analysis/analyze.py",
209 − "generated_utc": "2026-10-30T09:02:46.532910Z",
209 + "generated_utc": "2026-10-30T09:36:33.924117Z",
210 210 "note": "EXPLORATORY: question types 6-15 on synthetic test documents"
211 211 },
212 212 "results/figures/figA_qtypes_exploratory.pdf": {
@@ -214,7 +214,7 @@
214 214 "runs/raw/E7_qtypes.jsonl"
215 215 ],
216 216 "script": "src/wp30/analysis/analyze.py",
217 − "generated_utc": "2026-10-30T09:02:46.612438Z",
217 + "generated_utc": "2026-10-30T09:36:33.996541Z",
218 218 "note": "exploratory extension A-004"
219 219 },
220 220 "results/figures/figA_qtypes_exploratory.png": {
@@ -222,7 +222,7 @@
222 222 "runs/raw/E7_qtypes.jsonl"
223 223 ],
224 224 "script": "src/wp30/analysis/analyze.py",
225 − "generated_utc": "2026-10-30T09:02:46.668848Z",
225 + "generated_utc": "2026-10-30T09:36:34.046670Z",
226 226 "note": "exploratory extension A-004"
227 227 },
228 228 "results/figures/fig7_router.pdf": {
@@ -232,7 +232,7 @@
232 232 "runs/checkpoints/router_policy.json"
233 233 ],
234 234 "script": "src/wp30/analysis/analyze.py",
235 − "generated_utc": "2026-10-30T09:02:46.768937Z",
235 + "generated_utc": "2026-10-30T09:36:34.131525Z",
236 236 "note": "oracle = infeasible upper reference"
237 237 },
238 238 "results/figures/fig7_router.png": {
@@ -242,7 +242,7 @@
242 242 "runs/checkpoints/router_policy.json"
243 243 ],
244 244 "script": "src/wp30/analysis/analyze.py",
245 − "generated_utc": "2026-10-30T09:02:46.853258Z",
245 + "generated_utc": "2026-10-30T09:36:34.206816Z",
246 246 "note": "oracle = infeasible upper reference"
247 247 }
248 248 }
\ No newline at end of file
modified results/tables/E7_qtypes.csv +98 −98
@@ -1,99 +1,99 @@
1 1 condition,qtype,pair,quality,lo,hi,n,abstained
2 −A1,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0
3 −A1,aggregation,Qwen2.5-3B,0.0,0.0,0.0,16,0.0
4 −A1,multi_hop,Llama-3.2-3B,1.0,1.0,1.0,15,0.0
5 −A1,multi_hop,Qwen2.5-3B,1.0,1.0,1.0,16,0.0
6 −A1,negation,Llama-3.2-3B,0.9,0.8,1.0,30,0.0
7 −A1,negation,Qwen2.5-3B,1.0,1.0,1.0,32,0.0
8 −A1,revision,Llama-3.2-3B,0.7333333333333333,0.5825000000000004,0.8841666666666659,30,0.0
9 −A1,revision,Qwen2.5-3B,0.9375,0.84375,1.0,32,0.0
10 −A1,temporal,Llama-3.2-3B,0.4666666666666667,0.2,0.7333333333333333,15,0.0
11 −A1,temporal,Qwen2.5-3B,0.4375,0.21718750000000064,0.6875,16,0.0
12 −A1,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0
13 −A1,unanswerable,Qwen2.5-3B,0.96875,0.90625,1.0,32,0.96875
14 −A1,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.0
15 −A1,unit_conversion,Qwen2.5-3B,0.5,0.25,0.75,16,0.0
16 −A3|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0
17 −A3|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.0
18 −A3|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.0,0.0,0.0,15,1.0
19 −A3|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.0,0.0,0.0,15,1.0
20 −A3|budget_tokens=512|blind,negation,Llama-3.2-3B,0.8333333333333334,0.7,0.9333333333333333,30,0.0
21 −A3|budget_tokens=512|blind,negation,Qwen2.5-3B,0.9333333333333333,0.8333333333333334,1.0,30,0.0
22 −A3|budget_tokens=512|blind,revision,Llama-3.2-3B,0.7333333333333333,0.5666666666666667,0.8666666666666667,30,0.0
23 −A3|budget_tokens=512|blind,revision,Qwen2.5-3B,0.7333333333333333,0.5333333333333333,0.9,30,0.0
24 −A3|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.5333333333333333,0.3333333333333333,0.7333333333333333,15,0.0
25 −A3|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.3333333333333333,0.13333333333333333,0.5333333333333333,15,0.0
26 −A3|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0
27 −A3|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,30,1.0
28 −A3|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.9333333333333333
29 −A3|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.0,0.0,0.0,15,0.6666666666666666
30 −A4|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0
31 −A4|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.4
32 −A4|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.3333333333333333,0.06666666666666667,0.6,15,0.5333333333333333
33 −A4|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.4375,0.1875,0.6875,16,0.5
34 −A4|budget_tokens=512|blind,negation,Llama-3.2-3B,1.0,1.0,1.0,30,0.0
35 −A4|budget_tokens=512|blind,negation,Qwen2.5-3B,0.71875,0.53125,0.875,32,0.1875
36 −A4|budget_tokens=512|blind,revision,Llama-3.2-3B,0.16666666666666666,0.03333333333333333,0.3333333333333333,30,0.3333333333333333
37 −A4|budget_tokens=512|blind,revision,Qwen2.5-3B,0.25806451612903225,0.12903225806451613,0.4266935483870965,31,0.5161290322580645
38 −A4|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.2,0.0,0.4,15,0.3333333333333333
39 −A4|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.3333333333333333,0.13333333333333333,0.5333333333333333,15,0.26666666666666666
40 −A4|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0
41 −A4|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,30,1.0
42 −A4|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.6666666666666666
43 −A4|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.4,0.13333333333333333,0.6666666666666666,15,0.6
44 −A5|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.3333333333333333
45 −A5|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.4666666666666667
46 −A5|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.13333333333333333,0.0,0.3333333333333333,15,0.8666666666666667
47 −A5|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.2,0.06666666666666667,0.4,15,0.7333333333333333
48 −A5|budget_tokens=512|blind,negation,Llama-3.2-3B,0.5,0.36666666666666664,0.6333333333333333,30,0.3
49 −A5|budget_tokens=512|blind,negation,Qwen2.5-3B,0.5333333333333333,0.36666666666666664,0.7,30,0.43333333333333335
50 −A5|budget_tokens=512|blind,revision,Llama-3.2-3B,0.13333333333333333,0.0,0.3,30,0.7666666666666667
51 −A5|budget_tokens=512|blind,revision,Qwen2.5-3B,0.1,0.0,0.26666666666666666,30,0.8333333333333334
52 −A5|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.13333333333333333,0.0,0.3333333333333333,15,0.8666666666666667
53 −A5|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.0,0.0,0.0,15,0.8666666666666667
54 −A5|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0
55 −A5|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,30,1.0
56 −A5|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.7333333333333333
57 −A5|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.2,0.0,0.4,15,0.3333333333333333
58 −B2|bits=8,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0
59 −B2|bits=8,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.0
60 −B2|bits=8,multi_hop,Llama-3.2-3B,1.0,1.0,1.0,15,0.0
61 −B2|bits=8,multi_hop,Qwen2.5-3B,1.0,1.0,1.0,15,0.0
62 −B2|bits=8,negation,Llama-3.2-3B,0.9,0.8,1.0,30,0.0
63 −B2|bits=8,negation,Qwen2.5-3B,1.0,1.0,1.0,30,0.0
64 −B2|bits=8,revision,Llama-3.2-3B,0.7333333333333333,0.5825000000000004,0.8841666666666659,30,0.0
65 −B2|bits=8,revision,Qwen2.5-3B,0.9333333333333333,0.8333333333333334,1.0,30,0.0
66 −B2|bits=8,temporal,Llama-3.2-3B,0.4666666666666667,0.2,0.7333333333333333,15,0.0
67 −B2|bits=8,temporal,Qwen2.5-3B,0.4,0.13333333333333333,0.6666666666666666,15,0.0
68 −B2|bits=8,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0
69 −B2|bits=8,unanswerable,Qwen2.5-3B,0.9666666666666667,0.9,1.0,30,0.9666666666666667
70 −B2|bits=8,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.0
71 −B2|bits=8,unit_conversion,Qwen2.5-3B,0.6,0.3333333333333333,0.8666666666666667,15,0.0
72 −B3|keep_frac=0.25,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0
73 −B3|keep_frac=0.25,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.0
74 −B3|keep_frac=0.25,multi_hop,Llama-3.2-3B,0.2,0.0,0.4,15,0.3333333333333333
75 −B3|keep_frac=0.25,multi_hop,Qwen2.5-3B,0.2,0.0,0.4,15,0.13333333333333333
76 −B3|keep_frac=0.25,negation,Llama-3.2-3B,0.7333333333333333,0.6,0.8333333333333334,30,0.0
77 −B3|keep_frac=0.25,negation,Qwen2.5-3B,0.8666666666666667,0.7333333333333333,0.9666666666666667,30,0.0
78 −B3|keep_frac=0.25,revision,Llama-3.2-3B,0.06666666666666667,0.0,0.16666666666666666,30,0.6666666666666666
79 −B3|keep_frac=0.25,revision,Qwen2.5-3B,0.0,0.0,0.0,30,0.0
80 −B3|keep_frac=0.25,temporal,Llama-3.2-3B,0.5333333333333333,0.26666666666666666,0.7333333333333333,15,0.0
81 −B3|keep_frac=0.25,temporal,Qwen2.5-3B,0.5333333333333333,0.26666666666666666,0.7333333333333333,15,0.0
82 −B3|keep_frac=0.25,unanswerable,Llama-3.2-3B,0.9333333333333333,0.8,1.0,30,0.9333333333333333
83 −B3|keep_frac=0.25,unanswerable,Qwen2.5-3B,0.9666666666666667,0.9,1.0,30,0.9666666666666667
84 −B3|keep_frac=0.25,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.7333333333333333
85 −B3|keep_frac=0.25,unit_conversion,Qwen2.5-3B,0.06666666666666667,0.0,0.2,15,0.0
86 −B5|keep_frac=0.25|aware,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0
87 −B5|keep_frac=0.25|aware,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.0
88 −B5|keep_frac=0.25|aware,multi_hop,Llama-3.2-3B,1.0,1.0,1.0,15,0.0
89 −B5|keep_frac=0.25|aware,multi_hop,Qwen2.5-3B,0.5333333333333333,0.26666666666666666,0.8,15,0.0
90 −B5|keep_frac=0.25|aware,negation,Llama-3.2-3B,0.8666666666666667,0.7666666666666667,0.9666666666666667,30,0.0
91 −B5|keep_frac=0.25|aware,negation,Qwen2.5-3B,1.0,1.0,1.0,30,0.0
92 −B5|keep_frac=0.25|aware,revision,Llama-3.2-3B,0.7333333333333333,0.6158333333333337,0.8666666666666667,30,0.0
93 −B5|keep_frac=0.25|aware,revision,Qwen2.5-3B,0.3,0.16666666666666666,0.43333333333333335,30,0.0
94 −B5|keep_frac=0.25|aware,temporal,Llama-3.2-3B,0.6,0.4,0.8,15,0.0
95 −B5|keep_frac=0.25|aware,temporal,Qwen2.5-3B,0.5333333333333333,0.26666666666666666,0.8,15,0.0
96 −B5|keep_frac=0.25|aware,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0
97 −B5|keep_frac=0.25|aware,unanswerable,Qwen2.5-3B,0.9666666666666667,0.9,1.0,30,0.9666666666666667
98 −B5|keep_frac=0.25|aware,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.0
99 −B5|keep_frac=0.25|aware,unit_conversion,Qwen2.5-3B,0.4666666666666667,0.2,0.7333333333333333,15,0.0
2 +A1,aggregation,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
3 +A1,aggregation,Qwen2.5-3B,0.0,0.0,0.0,40,0.0
4 +A1,multi_hop,Llama-3.2-3B,1.0,1.0,1.0,40,0.0
5 +A1,multi_hop,Qwen2.5-3B,1.0,1.0,1.0,40,0.0
6 +A1,negation,Llama-3.2-3B,0.9,0.825,0.9625,80,0.0
7 +A1,negation,Qwen2.5-3B,0.9625,0.9125,1.0,80,0.0
8 +A1,revision,Llama-3.2-3B,0.75,0.6625,0.8375,80,0.0
9 +A1,revision,Qwen2.5-3B,0.875,0.8125,0.9375,80,0.0
10 +A1,temporal,Llama-3.2-3B,0.625,0.475,0.775,40,0.0
11 +A1,temporal,Qwen2.5-3B,0.55,0.4,0.7,40,0.0
12 +A1,unanswerable,Llama-3.2-3B,0.95,0.9,0.9875,80,0.95
13 +A1,unanswerable,Qwen2.5-3B,0.975,0.9375,1.0,80,0.975
14 +A1,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
15 +A1,unit_conversion,Qwen2.5-3B,0.575,0.41187500000000027,0.725,40,0.0
16 +A3|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
17 +A3|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,40,0.0
18 +A3|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.0,0.0,0.0,40,1.0
19 +A3|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.0,0.0,0.0,40,1.0
20 +A3|budget_tokens=512|blind,negation,Llama-3.2-3B,0.8625,0.7875,0.925,80,0.0
21 +A3|budget_tokens=512|blind,negation,Qwen2.5-3B,0.9,0.8309375000000001,0.95,80,0.0
22 +A3|budget_tokens=512|blind,revision,Llama-3.2-3B,0.8375,0.7625,0.9125,80,0.0
23 +A3|budget_tokens=512|blind,revision,Qwen2.5-3B,0.7,0.6059375000000001,0.7875,80,0.0125
24 +A3|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.65,0.5,0.8,40,0.0
25 +A3|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.6,0.45,0.75,40,0.0
26 +A3|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,80,1.0
27 +A3|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,80,1.0
28 +A3|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,40,0.925
29 +A3|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.0,0.0,0.0,40,0.675
30 +A4|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.025,0.0,0.075,40,0.0
31 +A4|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,40,0.35
32 +A4|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.3,0.15,0.45,40,0.65
33 +A4|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.475,0.325,0.625,40,0.475
34 +A4|budget_tokens=512|blind,negation,Llama-3.2-3B,0.95,0.8875,1.0,80,0.0125
35 +A4|budget_tokens=512|blind,negation,Qwen2.5-3B,0.725,0.625,0.8125,80,0.175
36 +A4|budget_tokens=512|blind,revision,Llama-3.2-3B,0.125,0.0625,0.2125,80,0.375
37 +A4|budget_tokens=512|blind,revision,Qwen2.5-3B,0.2,0.1125,0.3,80,0.6125
38 +A4|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.325,0.175,0.475,40,0.325
39 +A4|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.475,0.325,0.625,40,0.15
40 +A4|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,80,1.0
41 +A4|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,80,1.0
42 +A4|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.025,0.0,0.075,40,0.575
43 +A4|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.325,0.175,0.475,40,0.6
44 +A5|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.0,0.0,0.0,40,0.475
45 +A5|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,40,0.325
46 +A5|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.1,0.025,0.2,40,0.9
47 +A5|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.15,0.05,0.275,40,0.75
48 +A5|budget_tokens=512|blind,negation,Llama-3.2-3B,0.5,0.375,0.6,80,0.375
49 +A5|budget_tokens=512|blind,negation,Qwen2.5-3B,0.4875,0.36843750000000014,0.6,80,0.475
50 +A5|budget_tokens=512|blind,revision,Llama-3.2-3B,0.075,0.025,0.15,80,0.8375
51 +A5|budget_tokens=512|blind,revision,Qwen2.5-3B,0.0375,0.0,0.1,80,0.85
52 +A5|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.1,0.025,0.2,40,0.85
53 +A5|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.075,0.0,0.175,40,0.8
54 +A5|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,80,1.0
55 +A5|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,80,1.0
56 +A5|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,40,0.75
57 +A5|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.15,0.05,0.275,40,0.45
58 +B2|bits=8,aggregation,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
59 +B2|bits=8,aggregation,Qwen2.5-3B,0.0,0.0,0.0,40,0.0
60 +B2|bits=8,multi_hop,Llama-3.2-3B,1.0,1.0,1.0,40,0.0
61 +B2|bits=8,multi_hop,Qwen2.5-3B,1.0,1.0,1.0,40,0.0
62 +B2|bits=8,negation,Llama-3.2-3B,0.9,0.825,0.9625,80,0.0
63 +B2|bits=8,negation,Qwen2.5-3B,0.9375,0.8809375000000002,0.9875,80,0.0
64 +B2|bits=8,revision,Llama-3.2-3B,0.7625,0.675,0.85,80,0.0
65 +B2|bits=8,revision,Qwen2.5-3B,0.875,0.8125,0.9375,80,0.0
66 +B2|bits=8,temporal,Llama-3.2-3B,0.625,0.475,0.775,40,0.0
67 +B2|bits=8,temporal,Qwen2.5-3B,0.55,0.4,0.7,40,0.0
68 +B2|bits=8,unanswerable,Llama-3.2-3B,0.95,0.9,0.9875,80,0.95
69 +B2|bits=8,unanswerable,Qwen2.5-3B,0.975,0.9375,1.0,80,0.975
70 +B2|bits=8,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
71 +B2|bits=8,unit_conversion,Qwen2.5-3B,0.6,0.45,0.75,40,0.0
72 +B3|keep_frac=0.25,aggregation,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
73 +B3|keep_frac=0.25,aggregation,Qwen2.5-3B,0.0,0.0,0.0,40,0.0
74 +B3|keep_frac=0.25,multi_hop,Llama-3.2-3B,0.15,0.05,0.275,40,0.275
75 +B3|keep_frac=0.25,multi_hop,Qwen2.5-3B,0.175,0.075,0.3,40,0.1
76 +B3|keep_frac=0.25,negation,Llama-3.2-3B,0.725,0.65,0.8,80,0.0
77 +B3|keep_frac=0.25,negation,Qwen2.5-3B,0.9125,0.85,0.975,80,0.0
78 +B3|keep_frac=0.25,revision,Llama-3.2-3B,0.15,0.06843750000000012,0.2375,80,0.4875
79 +B3|keep_frac=0.25,revision,Qwen2.5-3B,0.1,0.025,0.1875,80,0.0
80 +B3|keep_frac=0.25,temporal,Llama-3.2-3B,0.575,0.425,0.725,40,0.0
81 +B3|keep_frac=0.25,temporal,Qwen2.5-3B,0.55,0.41187500000000027,0.7,40,0.0
82 +B3|keep_frac=0.25,unanswerable,Llama-3.2-3B,0.9375,0.875,0.9875,80,0.9375
83 +B3|keep_frac=0.25,unanswerable,Qwen2.5-3B,0.9875,0.9625,1.0,80,0.9875
84 +B3|keep_frac=0.25,unit_conversion,Llama-3.2-3B,0.025,0.0,0.075,40,0.65
85 +B3|keep_frac=0.25,unit_conversion,Qwen2.5-3B,0.075,0.0,0.15,40,0.0
86 +B5|keep_frac=0.25|aware,aggregation,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
87 +B5|keep_frac=0.25|aware,aggregation,Qwen2.5-3B,0.0,0.0,0.0,40,0.0
88 +B5|keep_frac=0.25|aware,multi_hop,Llama-3.2-3B,0.95,0.875,1.0,40,0.0
89 +B5|keep_frac=0.25|aware,multi_hop,Qwen2.5-3B,0.575,0.425,0.725,40,0.0
90 +B5|keep_frac=0.25|aware,negation,Llama-3.2-3B,0.85,0.7625,0.925,80,0.0
91 +B5|keep_frac=0.25|aware,negation,Qwen2.5-3B,0.9625,0.925,1.0,80,0.0
92 +B5|keep_frac=0.25|aware,revision,Llama-3.2-3B,0.7625,0.6875,0.8375,80,0.0
93 +B5|keep_frac=0.25|aware,revision,Qwen2.5-3B,0.275,0.175,0.375,80,0.0
94 +B5|keep_frac=0.25|aware,temporal,Llama-3.2-3B,0.7,0.55,0.85,40,0.0
95 +B5|keep_frac=0.25|aware,temporal,Qwen2.5-3B,0.55,0.4,0.7131249999999993,40,0.0
96 +B5|keep_frac=0.25|aware,unanswerable,Llama-3.2-3B,0.95,0.9,0.9875,80,0.95
97 +B5|keep_frac=0.25|aware,unanswerable,Qwen2.5-3B,0.975,0.9375,1.0,80,0.975
98 +B5|keep_frac=0.25|aware,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,40,0.0
99 +B5|keep_frac=0.25|aware,unit_conversion,Qwen2.5-3B,0.6,0.425,0.725,40,0.0
modified runs/raw/E7_qtypes.jsonl +3080 −0
Large diff (3,084 lines) is not loaded by default to keep the page fast.Load diff
Menu

Theme
↑↓ navigate ↵ open esc close / or ⌘K anywhere