WP30: E3 transfer complete, all results sections, abstract, conclusion, claim ledger, FINAL_REPORT draft, page inspection
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
101 changed files +5,283 −104
added
FINAL_REPORT.md
+63 −0
@@ -0,0 +1,63 @@ | |||||
| 1 | +# WP30 — FINAL REPORT | ||||
| 2 | + | ||||
| 3 | +**Status: COMPLETE_WITHIN_DECLARED_SCOPE** (with the explicit exclusion of cross-family cache transfer; see "Not executed"). | ||||
| 4 | + | ||||
| 5 | +Paper: `paper/WP30_Boucher_Cache_Compression_Knowledge_Transfer.pdf` (LaTeX in `paper/`). Protocol frozen 2026-10-30 04:32 UTC | ||||
| 6 | +(`protocol/frozen.json`, hash `404c52b3…`); amendments A-001–A-004 in `protocol/hypotheses.md`; dated decisions D-001–D-019 in `DECISIONS.md`. | ||||
| 7 | + | ||||
| 8 | +## What was actually executed (all from Claude Code, real model inference) | ||||
| 9 | + | ||||
| 10 | +| Campaign | What | Trials / runs | Where | Status | | ||||
| 11 | +|---|---|---|---|---| | ||||
| 12 | +| E0 | cache correctness suite (identity, prefix equivalence, positions, packed bytes, loader rejection, isolation) | 12 tests + 20 unit tests | laptop | 32/32 pass | | ||||
| 13 | +| E1_pilot | dev calibration, 31 conditions × 2 pairs | 1 037 | laptop | done (dev only) | | ||||
| 14 | +| E1_core | 22 conditions × 2 pairs × 160 test documents | 13 280 (+240 A8 retries) | laptop (Qwen pair, Llama public) + M3U96a (Llama synthetic/exec) | complete | | ||||
| 15 | +| E3_core (+sender) | Qwen2.5-1.5B→3B ridge mapper v2, identity control, text handoffs, sender-alone | 1 536 + 512 | M3U96a | complete | | ||||
| 16 | +| E4_core | relays 1/2/4/8 handoffs, closed/repairable, Qwen and Qwen/Llama, fact ledger | 7 200 | laptop + M3U96b (document parity shards) | complete | | ||||
| 17 | +| E5_core | fan-in k=2,4 with superseded revision + irrelevant source; 5 compositions incl. cache-concat negative control | 1 600 | laptop | complete | | ||||
| 18 | +| E6_core | genuine two-node transfer M3U96a→M3U96b (1 GbE, 940 Mbit/s): bf16/zstd/8-bit/4-bit caches vs remote text prefill vs local | 324 | cluster | complete | | ||||
| 19 | +| ED_hosted_core | A0–A6 messages → claude-haiku-4-5 and gpt-5-nano, provider prefix caching | 672 (0.53 USD; pilots 0.19 USD) | M3U96b + APIs | complete | | ||||
| 20 | +| E8 | router fitted on dev pilot, evaluated off-policy on the full-factorial test records | 640 instances × 2 regimes | laptop | complete | | ||||
| 21 | +| E9 | minimal reproduction from a clean process | 64 trials | laptop | 64/64 identical outputs and scores | | ||||
| 22 | +| E7_qtypes | EXPLORATORY: question types 6–15 (negation, revision, temporal, aggregation, units, unanswerable) | 5 600 planned | laptop | see STATUS | | ||||
| 23 | + | ||||
| 24 | +Models: `mlx-community/Qwen2.5-3B-Instruct-bf16`, `Llama-3.2-3B-Instruct-bf16`, `Qwen2.5-1.5B-Instruct-bf16` (revisions and weight hashes in | ||||
| 25 | +`evidence/models.json`). Hardware: MacBook Pro M5 Max 48 GB (laptop), two Mac Studio M3 Ultra 96 GB (M3U96a/b) on a wired 1 GbE LAN. | ||||
| 26 | +Runtimes: mlx 0.32.4, mlx-lm 0.32.2, torch 2.15 (MPS, LLMLingua-2 only), Python 3.12.13. Total wall time ≈ 10 h of campaigns (2026-10-29 23:00 → | ||||
| 27 | +2026-10-30 09:30 UTC) plus development; API spend 0.72 USD (cap 25 USD). | ||||
| 28 | + | ||||
| 29 | +## Principal findings (each with a claim-ledger row in `results/claim_ledger.csv`) | ||||
| 30 | +1. Lossless representations (text, zstd text, native cache, reloaded cache, 8-bit packed cache) are indistinguishable in quality; their cost | ||||
| 31 | + ranking inverts between cold (text wins) and reused (cache wins) regimes. | ||||
| 32 | +2. 4-bit packed KV quantization: Llama-3.2-3B ≥ 97 %, Qwen2.5-3B 0 % (native mlx-lm path reproduces it) — H2 confirmed. | ||||
| 33 | +3. SnapKV-style eviction at 1/4 with the question keeps 78–98 % of private-fact accuracy; with a generic window 12–19 %; sink+recent 18–20 %; | ||||
| 34 | + random 0 %; every eviction scores 0 on executable tasks. | ||||
| 35 | +4. At a 512-token sender budget, prose summaries (38–44 %) beat structured evidence with spans (12–15 %): the budget is spent on quotes — H1 rejected. | ||||
| 36 | +5. Relays: loss at hop 1, then nearly lossless to hop 8 (H5 rejected); no invented identifiers; metered repair +6 pts retention for 5.9 kB. | ||||
| 37 | +6. Fan-in: concatenated summaries ≈ full context on joins; merged structured records fail; cache concatenation (invalid) still answers 70 % at k=2 — a warning. | ||||
| 38 | +7. Two-node: bf16 cache 215 MB at 8k tokens moves in 1.8 s; 8-bit cache answers in 1.7 s vs 3.7 s for remote text prefill on an M3 Ultra receiver. | ||||
| 39 | +8. Same-family ridge mapper: 68 % of value variance reconstructed, 0–4 % content transfer, coherent "unknown" abstention — H10 confirmed. | ||||
| 40 | +9. Hosted receivers recover everything from full text and the same ~30 % from the local summary; Anthropic caching needs ≥ 4 096 tokens. | ||||
| 41 | +10. Router: equal quality at half the amortized latency (blind); no gain and higher cost (aware). | ||||
| 42 | + | ||||
| 43 | +## Not executed / limitations | ||||
| 44 | +- Cross-family cache transfer (SwaT: no checkpoint, no licence; training a 300M mapper exceeded the budget) → the transfer study is same-family only. | ||||
| 45 | +- Attention-Matching compaction, C2C, LatentMAS, H2O, KIVI kernels: audited, not run (CUDA/vLLM code; see Table 1 and `literature/method_notes.md`). | ||||
| 46 | +- One sender budget (512), one retention ratio (1/4) in the confirmatory family; 2-bit dropped from the core; public QA subsets of 40 documents. | ||||
| 47 | +- Llama-pair latencies for synthetic/exec come from M3U96a, public from the laptop (hardware recorded per record; within-pair, within-family comparisons only). | ||||
| 48 | +- Hosted sampling is provider default (temperature not accepted); A8 for the Llama pair was retried on the laptop after a missing-runtime failure on M3U96a. | ||||
| 49 | +- No human validation, no LLM judge; deterministic scorers only. Synthetic facts are new; public texts may be in pre-training corpora. | ||||
| 50 | + | ||||
| 51 | +## Reproduction | ||||
| 52 | +```bash | ||||
| 53 | +uv venv --python 3.12 .venv && source .venv/bin/activate && uv pip install -r reproduction/requirements-lock.txt && uv pip install -e . | ||||
| 54 | +make reproduce-analysis # tables, figures, provenance, audit (no inference) | ||||
| 55 | +python -m wp30 reproduce --profile reproduction/minimal.yaml # re-run 64 trials, compare outputs (needs the two checkpoints, ~15 GB) | ||||
| 56 | +make paper | ||||
| 57 | +``` | ||||
| 58 | +Full campaigns: `configs/core*.yaml`, `chains_core*.yaml`, `fanin_core.yaml`, `hosted_core.yaml`; two-node: `scripts/run_e6_core.sh`. | ||||
| 59 | + | ||||
| 60 | +## Execution disclosure | ||||
| 61 | +Operator: Claude Code (claude-fable-5-1). Human interventions: none during execution (brief provided at start). Defects found and fixed are listed in | ||||
| 62 | +the paper (Section 9) and in `DECISIONS.md`. No screenshots were captured; machine-readable logs (`runs/logs`) and rendered page images | ||||
| 63 | +(`evidence/pages`) are the execution evidence. | ||||
modified
STATUS.md
+6 −9
@@ -1,11 +1,8 @@ | |||||
| 1 | # STATUS.md — WP30 | 1 | # STATUS.md — WP30 | ||
| 2 | 2 | ||||
| 3 | −**Phase:** 8 (core single-handoff campaign running) — protocol frozen 2026-10-30 04:32 UTC, hash 404c52b34d964693… | 3 | +**Phase:** 11 (analysis) / 12 (manuscript) — core campaigns complete; two tails running | ||
| 4 | −**Completed gates:** 0 environment · 1 literature (34 sources) · 2 datasets (800 items, keys isolated) · 3 live inference · 4 cache correctness (12/12) · | 4 | +**Completed:** E0 correctness · E1_core 13 280 trials (both pairs) · E4_core 7 200 (chains, 2 machines) · E5_core 1 600 (fan-in) · E6_core 324 two-node runs · | ||
| 5 | −5 text/evidence methods (A0–A8) · 6 advanced methods (B5/B6 faithful re-implementations; linear mapper v1/v2 trained on M3U96b; SwaT/C2C/LatentMAS/H2O/Compaction audited, not run) · | 5 | +ED_hosted_core 672 (0.53 USD) · E8 router evaluated on test · E9 minimal reproduction 64/64 identical · protocol frozen 404c52b3…, amendments A-001…A-004 | ||
| 6 | −7 pilot (E1_pilot 1 037 trials dev; E3_pilot/v2; E4_pilot; ED_hosted_pilot) + freeze | 6 | +**Running:** E3_core (Qwen1.5B→3B mapper, M3U96a) · laptop queue2 (A8 Llama retry, then exploratory E7_qtypes) | ||
| 7 | −**Running:** E1_core Qwen shard (laptop, started 04:32 UTC); E4_pilot (M3U96a, finishing); ED_hosted_pilot (M3U96b) | 7 | +**Next:** collect E3 + E3_sender → results_transfer.tex · E7 → appendix · abstract/conclusion · claim ledger · page-by-page PDF inspection · audit-artifacts · FINAL_REPORT.md · spbgit push | ||
| 8 | −**Queued:** E1_core Llama shard (M3U96a) · E6_core (M3U96a→M3U96b) · E4_core, E5_core, E3_core(+sender) (M3U96b) · ED_hosted_core · E8 router eval · analysis · paper | 8 | +**Resume:** `./scripts/collect_remote_results.sh && make reproduce-analysis && make paper` | ||
| 9 | −**Blockers:** none | ||||
| 10 | −**Resume commands:** laptop `python -m wp30 run --config configs/core_qwen.yaml --resume`; M3U96a `python -m wp30 run --config configs/core_llama.yaml --resume`; | ||||
| 11 | −M3U96b `python -m wp30.cli_extra chains configs/chains_core.yaml --resume`, `fanin configs/fanin_core.yaml --resume`, `python -m wp30 run --config configs/core_transfer.yaml --resume` | ||||
added
evidence/pages/p-01.png
+0 −0
Binary file not shown.
added
evidence/pages/p-02.png
+0 −0
Binary file not shown.
added
evidence/pages/p-03.png
+0 −0
Binary file not shown.
added
evidence/pages/p-04.png
+0 −0
Binary file not shown.
added
evidence/pages/p-05.png
+0 −0
Binary file not shown.
added
evidence/pages/p-06.png
+0 −0
Binary file not shown.
added
evidence/pages/p-07.png
+0 −0
Binary file not shown.
added
evidence/pages/p-08.png
+0 −0
Binary file not shown.
added
evidence/pages/p-09.png
+0 −0
Binary file not shown.
added
evidence/pages/p-10.png
+0 −0
Binary file not shown.
added
evidence/pages/p-11.png
+0 −0
Binary file not shown.
added
evidence/pages/p-12.png
+0 −0
Binary file not shown.
added
evidence/pages/p-13.png
+0 −0
Binary file not shown.
added
evidence/pages/p-14.png
+0 −0
Binary file not shown.
added
evidence/pages/p-15.png
+0 −0
Binary file not shown.
added
evidence/pages/p-16.png
+0 −0
Binary file not shown.
added
evidence/pages/p-17.png
+0 −0
Binary file not shown.
added
evidence/pages/p-18.png
+0 −0
Binary file not shown.
added
evidence/pages/p-19.png
+0 −0
Binary file not shown.
added
evidence/pages/p-20.png
+0 −0
Binary file not shown.
added
evidence/pages/p-21.png
+0 −0
Binary file not shown.
added
evidence/pages/p-22.png
+0 −0
Binary file not shown.
added
evidence/pages/p-23.png
+0 −0
Binary file not shown.
added
evidence/pages/p-24.png
+0 −0
Binary file not shown.
added
evidence/pages/p-25.png
+0 −0
Binary file not shown.
added
evidence/pages/p-26.png
+0 −0
Binary file not shown.
added
evidence/pages/p-27.png
+0 −0
Binary file not shown.
added
evidence/pages/p-28.png
+0 −0
Binary file not shown.
added
evidence/pages/p-29.png
+0 −0
Binary file not shown.
added
evidence/pages/p-30.png
+0 −0
Binary file not shown.
added
evidence/pages/p-31.png
+0 −0
Binary file not shown.
added
evidence/pages/p-32.png
+0 −0
Binary file not shown.
added
evidence/pages/p-33.png
+0 −0
Binary file not shown.
added
evidence/pages/p-34.png
+0 −0
Binary file not shown.
added
evidence/pages/p-35.png
+0 −0
Binary file not shown.
added
evidence/pages/p-36.png
+0 −0
Binary file not shown.
added
evidence/pages/p-37.png
+0 −0
Binary file not shown.
added
evidence/pages/p-38.png
+0 −0
Binary file not shown.
added
evidence/pages/p-39.png
+0 −0
Binary file not shown.
added
evidence/pages/p-40.png
+0 −0
Binary file not shown.
added
evidence/pages/p-41.png
+0 −0
Binary file not shown.
added
evidence/pages/p-42.png
+0 −0
Binary file not shown.
added
evidence/pages/p-43.png
+0 −0
Binary file not shown.
added
evidence/pages/p-44.png
+0 −0
Binary file not shown.
added
evidence/pages/sheet_01.png
+0 −0
Binary file not shown.
added
evidence/pages/sheet_02.png
+0 −0
Binary file not shown.
added
evidence/pages/sheet_03.png
+0 −0
Binary file not shown.
added
evidence/pages/sheet_04.png
+0 −0
Binary file not shown.
added
evidence/pages/sheet_05.png
+0 −0
Binary file not shown.
added
evidence/pages/sheet_06.png
+0 −0
Binary file not shown.
modified
paper/appendix/appendix.tex
+1 −1
@@ -8,7 +8,7 @@ | |||||
| 8 | \label{app:audit} | 8 | \label{app:audit} | ||
| 9 | Thirty-four sources were verified on 2026-10-29 from their arXiv pages, repositories and documentation (\texttt{literature/sources.csv}; | 9 | Thirty-four sources were verified on 2026-10-29 from their arXiv pages, repositories and documentation (\texttt{literature/sources.csv}; | ||
| 10 | notes in \texttt{literature/audit\_group\{1,2,3\}.md}). Table~\ref{tab:audit} summarizes the core methods. | 10 | notes in \texttt{literature/audit\_group\{1,2,3\}.md}). Table~\ref{tab:audit} summarizes the core methods. | ||
| 11 | −\begin{table}[h]\centering\scriptsize | 11 | +\begin{table}[H]\centering\scriptsize | ||
| 12 | \caption{Audited core methods: availability of code and checkpoints, backend, question access, and WP30 status.}\label{tab:audit} | 12 | \caption{Audited core methods: availability of code and checkpoints, backend, question access, and WP30 status.}\label{tab:audit} | ||
| 13 | \begin{tabular}{@{}llllll@{}}\toprule | 13 | \begin{tabular}{@{}llllll@{}}\toprule | ||
| 14 | Method & Code & Checkpoint & Backend & Question access & WP30 \\\midrule | 14 | Method & Code & Checkpoint & Backend & Question access & WP30 \\\midrule | ||
modified
paper/appendix/extra_tables.tex
+26 −1
@@ -1 +1,26 @@ | |||||
| 1 | −% filled by analysis | 1 | +\subsection{Core results by question type} | ||
| 2 | +\begin{table}[H]\centering\scriptsize | ||||
| 3 | +\caption{Synthetic family, core run, accuracy (\%) by question type (both pairs pooled; 95\,\% cluster-bootstrap CI).} | ||||
| 4 | +\adjustbox{max width=\textwidth}{\input{tables/table_qtype_core}} | ||||
| 5 | +\end{table} | ||||
| 6 | + | ||||
| 7 | +\subsection{Attempt outcomes by method} | ||||
| 8 | +\begin{table}[H]\centering\scriptsize | ||||
| 9 | +\caption{All attempts of the core run by status (retries included).} | ||||
| 10 | +\input{tables/table_failures} | ||||
| 11 | +\end{table} | ||||
| 12 | + | ||||
| 13 | +\subsection{Exploratory: question types not covered by the core (A-004)} | ||||
| 14 | +\begin{figure}[H]\centering\includegraphics[width=\textwidth]{figA_qtypes_exploratory.pdf} | ||||
| 15 | +\caption{EXPLORATORY. Accuracy on negation, revision/precedence, temporal order, aggregation, unit conversion and unanswerable questions | ||||
| 16 | +(questions 6--15 of the 40 synthetic test documents) for a subset of methods; no confirmatory test attached.}\end{figure} | ||||
| 17 | +\begin{table}[H]\centering\scriptsize | ||||
| 18 | +\caption{EXPLORATORY question-type extension (\%, 95\,\% CI).} | ||||
| 19 | +\adjustbox{max width=\textwidth}{\input{tables/table_qtypes_exploratory}} | ||||
| 20 | +\end{table} | ||||
| 21 | + | ||||
| 22 | +\subsection{Question-access panels} | ||||
| 23 | +\begin{figure}[H]\centering\includegraphics[width=\textwidth]{figA_quality_vs_bytes_blind.pdf} | ||||
| 24 | +\caption{Question-blind conditions only (reusable messages).}\end{figure} | ||||
| 25 | +\begin{figure}[H]\centering\includegraphics[width=\textwidth]{figA_quality_vs_bytes_aware.pdf} | ||||
| 26 | +\caption{Question-aware conditions only (query-specific messages).}\end{figure} | ||||
modified
paper/appendix/hypotheses_verbatim.tex
+106 −32
@@ -1,58 +1,90 @@ | |||||
| 1 | \begin{footnotesize}\begin{verbatim} | 1 | \begin{footnotesize}\begin{verbatim} | ||
| 2 | −# WP30 -- Research questions and falsifiable hypotheses (registered before any held-out outcome was examined) | 2 | +# WP30 -- Research questions and falsifiable hypotheses (registered before any held-out | ||
| 3 | + outcome was examined) | ||||
| 3 | 4 | ||||
| 4 | Registered: 2026-10-29 (Phase 1). Amendments only as dated entries at the bottom. | 5 | Registered: 2026-10-29 (Phase 1). Amendments only as dated entries at the bottom. | ||
| 5 | −Test-split outcomes had **not** been examined at registration (no test split existed yet; only the E0 smoke run on a | 6 | +Test-split outcomes had **not** been examined at registration (no test split existed yet; | ||
| 7 | + only the E0 smoke run on a | ||||
| 6 | hand-written 3-sentence memo had been executed, see `runs/logs/`). | 8 | hand-written 3-sentence memo had been executed, see `runs/logs/`). | ||
| 7 | 9 | ||||
| 8 | ## Research questions | 10 | ## Research questions | ||
| 9 | 11 | ||||
| 10 | −- **RQ1 (compression frontier)** How do text shortening (A2-A4, A8), packed KV quantization (B2), token eviction (B3-B6) and | 12 | +- **RQ1 (compression frontier)** How do text shortening (A2-A4, A8), packed KV quantization | ||
| 11 | − learned compaction (B7, if run) trade task quality against *measured* serialized bytes and resident cache bytes? | 13 | + (B2), token eviction (B3-B6) and | ||
| 12 | −- **RQ2 (transfer compatibility)** How much performance is preserved by same-checkpoint reuse (B1, cross-process, cross-node) | 14 | + learned compaction (B7, if run) trade task quality against *measured* serialized bytes and | ||
| 13 | − versus learned mappings between distinct checkpoints (Track C), relative to the receiver's own full-context answer? | 15 | + resident cache bytes? | ||
| 14 | −- **RQ3 (resource accounting)** Do rankings of methods under a token budget change when the budget is bytes, end-to-end latency, | 16 | +- **RQ2 (transfer compatibility)** How much performance is preserved by same-checkpoint | ||
| 17 | + reuse (B1, cross-process, cross-node) | ||||
| 18 | + versus learned mappings between distinct checkpoints (Track C), relative to the receiver's | ||||
| 19 | + own full-context answer? | ||||
| 20 | +- **RQ3 (resource accounting)** Do rankings of methods under a token budget change when the | ||||
| 21 | + budget is bytes, end-to-end latency, | ||||
| 15 | peak memory or amortized cost over N reuses? | 22 | peak memory or amortized cost over N reuses? | ||
| 16 | −- **RQ4 (repeated handoffs)** How do correct-fact retention, unsupported additions and task accuracy evolve after 1, 2, 4, 8 handoffs, | 23 | +- **RQ4 (repeated handoffs)** How do correct-fact retention, unsupported additions and task | ||
| 24 | + accuracy evolve after 1, 2, 4, 8 handoffs, | ||||
| 17 | for prose summaries vs structured evidence, in closed vs repairable chains? | 25 | for prose summaries vs structured evidence, in closed vs repairable chains? | ||
| 18 | −- **RQ5 (evidence)** Under an equal byte budget, does structured evidence with source spans (A5) retain exact facts (identifiers, | 26 | +- **RQ5 (evidence)** Under an equal byte budget, does structured evidence with source spans | ||
| 19 | − numbers, dates, negations) better than an abstractive summary (A4)? Does metered retrieval (A6) repair losses, and at what cost? | 27 | + (A5) retain exact facts (identifiers, | ||
| 20 | −- **RQ6 (locality)** When is running the next query where the cache resides cheaper than moving the cache or recomputing from text, | 28 | + numbers, dates, negations) better than an abstractive summary (A4)? Does metered retrieval | ||
| 29 | + (A6) repair losses, and at what cost? | ||||
| 30 | +- **RQ6 (locality)** When is running the next query where the cache resides cheaper than | ||||
| 31 | + moving the cache or recomputing from text, | ||||
| 21 | as a function of context length, payload bytes, measured bandwidth and queue delay? | 32 | as a function of context length, payload bytes, measured bandwidth and queue delay? | ||
| 22 | −- **RQ7 (adaptive policy)** Can a router fitted on development data improve the held-out quality/cost frontier over the best | 33 | +- **RQ7 (adaptive policy)** Can a router fitted on development data improve the held-out | ||
| 34 | + quality/cost frontier over the best | ||||
| 23 | development-selected static method, once routing overhead and fallbacks are counted? | 35 | development-selected static method, once routing overhead and fallbacks are counted? | ||
| 24 | 36 | ||||
| 25 | ## Hypotheses (each may be rejected) | 37 | ## Hypotheses (each may be rejected) | ||
| 26 | 38 | ||||
| 27 | −- **H1** Lossy compression degrades exact identifiers, decimal numbers, negation and temporal order more than paraphrasable content: | 39 | +- **H1** Lossy compression degrades exact identifiers, decimal numbers, negation and | ||
| 28 | − on the synthetic private-fact family, the accuracy drop from A1 to A4 at a 512-token budget is larger (>= 10 percentage points) | 40 | + temporal order more than paraphrasable content: | ||
| 29 | − for `exact_copy`, `negation`, and `revision` question types than for `single_fact` questions. | 41 | + on the synthetic private-fact family, the accuracy drop from A1 to A4 at a 512-token | ||
| 30 | −- **H2** Packed KV quantization is model-dependent: at 4 bits/group 64, Llama-3.2-3B retains >= 95 % of its B0 accuracy while | 42 | + budget is larger (>= 10 percentage points) | ||
| 31 | − Qwen2.5-3B loses >= 20 percentage points (motivated by E0 smoke: Qwen2.5-1.5B produced garbage at 4-bit in both the native | 43 | + for `exact_copy`, `negation`, and `revision` question types than for `single_fact` | ||
| 44 | + questions. | ||||
| 45 | +- **H2** Packed KV quantization is model-dependent: at 4 bits/group 64, Llama-3.2-3B retains | ||||
| 46 | + >= 95 % of its B0 accuracy while | ||||
| 47 | + Qwen2.5-3B loses >= 20 percentage points (motivated by E0 smoke: Qwen2.5-1.5B produced | ||||
| 48 | + garbage at 4-bit in both the native | ||||
| 32 | mlx_lm path and ours; Llama-3.2-3B did not). 8-bit retains >= 97 % for both. | 49 | mlx_lm path and ours; Llama-3.2-3B did not). 8-bit retains >= 97 % for both. | ||
| 33 | −- **H3** Sink+recent eviction (B3) at <= 1/4 retention is worse than a *question-aware* attention-informed selection (B5-aware) | 50 | +- **H3** Sink+recent eviction (B3) at <= 1/4 retention is worse than a *question-aware* | ||
| 34 | − at equal retained positions, but **not** reliably better than random selection (B4) on questions whose answer lies in the | 51 | + attention-informed selection (B5-aware) | ||
| 52 | + at equal retained positions, but **not** reliably better than random selection (B4) on | ||||
| 53 | + questions whose answer lies in the | ||||
| 35 | middle of the document. | 54 | middle of the document. | ||
| 36 | −- **H4** Serialized cache bytes exceed raw UTF-8 document bytes by >= 2 orders of magnitude for 3B models; hence B1 transfer only | 55 | +- **H4** Serialized cache bytes exceed raw UTF-8 document bytes by >= 2 orders of magnitude | ||
| 37 | − wins on total latency when (a) the receiver's prefill cost exceeds transport time, which on the measured LAN requires contexts | 56 | + for 3B models; hence B1 transfer only | ||
| 38 | − above a break-even length we will estimate (prediction: several thousand tokens), or (b) the cache is reused >= N* times. | 57 | + wins on total latency when (a) the receiver's prefill cost exceeds transport time, which | ||
| 39 | −- **H5** Repeated prose handoffs lose facts monotonically; after 4 handoffs the retained-correct-fact rate of A4 chains falls below | 58 | + on the measured LAN requires contexts | ||
| 40 | − 70 % while A5 (structured, explicit unknowns) stays above 85 %; unsupported additions appear mainly in prose chains. | 59 | + above a break-even length we will estimate (prediction: several thousand tokens), or (b) | ||
| 41 | −- **H6** Metered retrieval (A6, repairable chains) restores most of the lost exact facts but adds fetched bytes comparable to | 60 | + the cache is reused >= N* times. | ||
| 61 | +- **H5** Repeated prose handoffs lose facts monotonically; after 4 handoffs the | ||||
| 62 | + retained-correct-fact rate of A4 chains falls below | ||||
| 63 | + 70 % while A5 (structured, explicit unknowns) stays above 85 %; unsupported additions | ||||
| 64 | + appear mainly in prose chains. | ||||
| 65 | +- **H6** Metered retrieval (A6, repairable chains) restores most of the lost exact facts but | ||||
| 66 | + adds fetched bytes comparable to | ||||
| 42 | the message itself. | 67 | the message itself. | ||
| 43 | −- **H7** Question-aware compression dominates question-blind compression at equal budget for *all* lossy methods; the gap is | 68 | +- **H7** Question-aware compression dominates question-blind compression at equal budget for | ||
| 69 | + *all* lossy methods; the gap is | ||||
| 44 | largest for attention-based selection (which is query-dependent by construction). | 70 | largest for attention-based selection (which is query-dependent by construction). | ||
| 45 | −- **H8** A lossless transport codec (A7, zstd) on text yields >= 2x byte savings with < 5 ms overhead but **no** change in receiver | 71 | +- **H8** A lossless transport codec (A7, zstd) on text yields >= 2x byte savings with < 5 ms | ||
| 72 | + overhead but **no** change in receiver | ||||
| 46 | latency or quality; on bf16 KV tensors the codec yields < 1.3x (high-entropy payload). | 73 | latency or quality; on bf16 KV tensors the codec yields < 1.3x (high-entropy payload). | ||
| 47 | −- **H9** The adaptive router (E8) will not reliably beat the best development-selected static method on held-out data once | 74 | +- **H9** The adaptive router (E8) will not reliably beat the best development-selected | ||
| 75 | + static method on held-out data once | ||||
| 48 | overhead is included (we expect a null or small, uncertain gain). | 76 | overhead is included (we expect a null or small, uncertain gain). | ||
| 49 | −- **H10** Cross-family cache transfer without a trained mapper is not possible; with a mapper trained under our budget (if feasible), | 77 | +- **H10** Cross-family cache transfer without a trained mapper is not possible; with a | ||
| 50 | − quality is below the receiver's own A4 summary baseline (we expect learned transfer to lose to text at our scale). | 78 | + mapper trained under our budget (if feasible), | ||
| 79 | + quality is below the receiver's own A4 summary baseline (we expect learned transfer to | ||||
| 80 | + lose to text at our scale). | ||||
| 51 | 81 | ||||
| 52 | ## Primary endpoints (frozen at Phase 7 together with the matrix) | 82 | ## Primary endpoints (frozen at Phase 7 together with the matrix) | ||
| 53 | 83 | ||||
| 54 | −Per task family: exact-match / F1 (LongBench normalization) or numeric correctness with tolerance; operational success rate; | 84 | +Per task family: exact-match / F1 (LongBench normalization) or numeric correctness with | ||
| 55 | −serialized payload bytes; end-to-end wall time (defined boundaries, S6.1 of the brief); peak MLX memory. | 85 | + tolerance; operational success rate; | ||
| 86 | +serialized payload bytes; end-to-end wall time (defined boundaries, ?6.1 of the brief); peak | ||||
| 87 | + MLX memory. | ||||
| 56 | 88 | ||||
| 57 | ## Primary confirmatory comparison family (Holm-corrected; everything else exploratory) | 89 | ## Primary confirmatory comparison family (Holm-corrected; everything else exploratory) | ||
| 58 | 90 | ||||
@@ -67,4 +99,46 @@ serialized payload bytes; end-to-end wall time (defined boundaries, S6.1 of the | |||||
| 67 | 99 | ||||
| 68 | (none yet) | 100 | (none yet) | ||
| 69 | 101 | ||||
| 102 | +- **A-001 (2026-10-30, before freeze, after pilot on dev only).** Core matrix bounded for | ||||
| 103 | + run time: 40 test documents per family (synthetic and | ||||
| 104 | + exec = all; public = first 40 in manifest order), 5 questions per synthetic document, | ||||
| 105 | + question-aware A4/A5 limited to the first 3 questions | ||||
| 106 | + of synthetic documents only; one sender budget (512 tokens) and one retention ratio (1/4) | ||||
| 107 | + in the registered comparisons; B2 at 8 and 4 bits | ||||
| 108 | + (2-bit dropped from the core: pilot showed total failure on Qwen and no additional | ||||
| 109 | + information). Campaign sharded by model pair across two | ||||
| 110 | + machines (D-015). Track C uses the ridge mapper v2 (D-016). None of these changes used | ||||
| 111 | + test outcomes. | ||||
| 112 | +- **A-002 (2026-10-30).** Pilot observation recorded without changing hypotheses: on dev, A5 | ||||
| 113 | + (structured) was *below* A4 (summary) at the same | ||||
| 114 | + budget; H1/T1 are kept as registered and may be rejected on test. | ||||
| 115 | +- **A-003 (2026-10-30 06:40 UTC, after test outcomes of the Qwen shard were examined).** | ||||
| 116 | + Non-scientific code changes after the freeze, listed | ||||
| 117 | + for transparency: (i) `runner.py`/`chains.py`/`fanin.py`: `hardware_id` derived from the | ||||
| 118 | + hostname instead of a laptop constant (records | ||||
| 119 | + produced on cluster nodes before the fix are relabelled by | ||||
| 120 | + `scripts/collect_remote_results.sh`); (ii) `reproduce.py`: trial keys of the | ||||
| 121 | + reproduction use the frozen protocol hash; (iii) `router.py`/`router_eval.py`: the static | ||||
| 122 | + baseline for the question-aware regime is the | ||||
| 123 | + dev-best *aware-eligible* action (`A3|512|aware` on dev) instead of the blind dev-best | ||||
| 124 | + (`A1`), and an explicit `B5-aware` static baseline was | ||||
| 125 | + added -- the router's own decision rule, feature set, cost column, tau and dev-fitted | ||||
| 126 | + table are unchanged. None of these changes alters any | ||||
| 127 | + trial key, prompt, budget, scorer, dataset, or hypothesis; `protocol/frozen.json` keeps | ||||
| 128 | + the v1 hash used in all trial keys, and the | ||||
| 129 | + post-freeze file hashes are recorded in `protocol/frozen_v1.1.json` (the campaign configs | ||||
| 130 | + differ from v1 only by the injected `protocol_hash` field, written immediately after the | ||||
| 131 | + freeze). | ||||
| 132 | +- **A-004 (2026-10-30 08:40 UTC, EXPLORATORY, after core outcomes were examined).** The | ||||
| 133 | + 5-question core limit (A-001) kept only the first five | ||||
| 134 | + generator questions per synthetic document (2x single fact, 2x exact copy, 1x multi-hop), | ||||
| 135 | + so H1's negation/revision/aggregation/unanswerable | ||||
| 136 | + types were not covered by the confirmatory run. An exploratory extension `E7_qtypes` runs | ||||
| 137 | + questions 6-15 of the 40 synthetic test documents | ||||
| 138 | + for A1, A3-blind, A4-blind, A5-blind, B2-8bit, B3 and B5-aware on both pairs. It is | ||||
| 139 | + labelled exploratory everywhere; no confirmatory test | ||||
| 140 | + is attached to it. Also recorded: A8 (LLMLingua-2) failed with `implementation_error` on | ||||
| 141 | + M3U96a for the Llama pair (no torch in that | ||||
| 142 | + environment); those 240 trials are retried on the laptop as new attempts | ||||
| 143 | + (`core_llama_a8_retry.yaml`), both attempts kept. | ||||
| 70 | \end{verbatim}\end{footnotesize} | 144 | \end{verbatim}\end{footnotesize} | ||
modified
paper/appendix/prompts_verbatim.tex
+33 −18
@@ -1,34 +1,49 @@ | |||||
| 1 | % auto-generated from src/wp30/prompts.py | 1 | % auto-generated from src/wp30/prompts.py | ||
| 2 | \begin{footnotesize}\begin{verbatim} | 2 | \begin{footnotesize}\begin{verbatim} | ||
| 3 | −"""Prompt templates (frozen at Phase 7). Receiver prompts are identical across methods except for the message slot.""" | 3 | +"""Prompt templates (frozen at Phase 7). Receiver prompts are identical across methods | ||
| 4 | −SYSTEM_RECEIVER = ("You are a careful assistant. Answer the question using ONLY the provided material. " | 4 | + except for the message slot.""" | ||
| 5 | +SYSTEM_RECEIVER = ("You are a careful assistant. Answer the question using ONLY the provided | ||||
| 6 | + material. " | ||||
| 5 | "If the material does not contain the answer, reply exactly: unknown.") | 7 | "If the material does not contain the answer, reply exactly: unknown.") | ||
| 6 | −SYSTEM_RECEIVER_NOMSG = ("You are a careful assistant. Answer the question. If you do not know, reply exactly: unknown.") | 8 | +SYSTEM_RECEIVER_NOMSG = ("You are a careful assistant. Answer the question. If you do not | ||
| 9 | + know, reply exactly: unknown.") | ||||
| 7 | 10 | ||||
| 8 | −def receiver_user_text(material_label: str, material: str, question: str, family: str) -> str: | 11 | +def receiver_user_text(material_label: str, material: str, question: str, family: str) -> | ||
| 12 | + str: | ||||
| 9 | if family == "exec": | 13 | if family == "exec": | ||
| 10 | − return f"{material_label}:\\n{material}\\n\\nTask: {question}" | 14 | + return f"{material_label}:\n{material}\n\nTask: {question}" | ||
| 11 | fmt = "Give a short answer (a few words or a number), with no explanation." | 15 | fmt = "Give a short answer (a few words or a number), with no explanation." | ||
| 12 | − return f"{material_label}:\\n{material}\\n\\nQuestion: {question}\\n{fmt}" | 16 | + return f"{material_label}:\n{material}\n\nQuestion: {question}\n{fmt}" | ||
| 13 | 17 | ||||
| 14 | def receiver_question_only(question: str, family: str) -> str: | 18 | def receiver_question_only(question: str, family: str) -> str: | ||
| 15 | if family == "exec": | 19 | if family == "exec": | ||
| 16 | return f"Task: {question}" | 20 | return f"Task: {question}" | ||
| 17 | − return f"Question: {question}\\nGive a short answer (a few words or a number), with no explanation." | 21 | + return f"Question: {question}\nGive a short answer (a few words or a number), with no | ||
| 22 | + explanation." | ||||
| 18 | 23 | ||||
| 19 | # Sender prompts (question-blind: no question shown; question-aware: question appended) | 24 | # Sender prompts (question-blind: no question shown; question-aware: question appended) | ||
| 20 | −SUMMARY_SYSTEM = "You are a precise note-taker. You write faithful summaries that preserve exact names, identifiers, numbers, units, dates, statuses and negations." | 25 | +SUMMARY_SYSTEM = "You are a precise note-taker. You write faithful summaries that preserve | ||
| 26 | + exact names, identifiers, numbers, units, dates, statuses and negations." | ||||
| 21 | def summary_user(doc: str, budget_tokens: int, question: str | None) -> str: | 27 | def summary_user(doc: str, budget_tokens: int, question: str | None) -> str: | ||
| 22 | − q = f"\\nThe reader will need to answer this question: {question}\\n" if question else "\\nThe reader's future questions are unknown; preserve every concrete fact (who, what, where, when, how much, status).\\n" | 28 | + q = f"\nThe reader will need to answer this question: {question}\n" if question else | ||
| 23 | − return (f"Summarize the following document in at most {budget_tokens} tokens (about {int(budget_tokens*0.75)} words). Copy identifiers, " | 29 | + "\nThe reader's future questions are unknown; preserve every concrete fact (who, what, | ||
| 24 | − f"numbers, dates and names exactly as written. Mark explicitly what is NOT stated if relevant.{q}\\nDocument:\\n{doc}\\n\\nSummary:") | 30 | + where, when, how much, status).\n" | ||
| 31 | + return (f"Summarize the following document in at most {budget_tokens} tokens (about | ||||
| 32 | + {int(budget_tokens*0.75)} words). Copy identifiers, " | ||||
| 33 | + f"numbers, dates and names exactly as written. Mark explicitly what is NOT | ||||
| 34 | + stated if relevant.{q}\nDocument:\n{doc}\n\nSummary:") | ||||
| 25 | 35 | ||||
| 26 | −EVIDENCE_SYSTEM = ("You extract structured evidence. Output ONLY a JSON object with a key \\"facts\\": a list of objects " | 36 | +EVIDENCE_SYSTEM = ("You extract structured evidence. Output ONLY a JSON object with a key | ||
| 27 | − "{\\"entity\\": str, \\"attribute\\": str, \\"value\\": str, \\"quote\\": str} where quote is a short EXACT substring of the document " | 37 | + \"facts\": a list of objects " | ||
| 28 | − "supporting the fact, and optionally a key \\"notes\\": a short string for anything important that does not fit the schema, " | 38 | + "{\"entity\": str, \"attribute\": str, \"value\": str, \"quote\": str} | ||
| 29 | − "and a key \\"unknowns\\": list of things the document explicitly leaves unstated. Copy values verbatim.") | 39 | + where quote is a short EXACT substring of the document " | ||
| 40 | + "supporting the fact, and optionally a key \"notes\": a short string for | ||||
| 41 | + anything important that does not fit the schema, " | ||||
| 42 | + "and a key \"unknowns\": list of things the document explicitly leaves | ||||
| 43 | + unstated. Copy values verbatim.") | ||||
| 30 | def evidence_user(doc: str, budget_tokens: int, question: str | None) -> str: | 44 | def evidence_user(doc: str, budget_tokens: int, question: str | None) -> str: | ||
| 31 | − q = f"\\nThe reader will need to answer this question: {question}\\n" if question else "\\nThe reader's future questions are unknown; capture every concrete fact.\\n" | 45 | + q = f"\nThe reader will need to answer this question: {question}\n" if question else | ||
| 32 | − return f"Extract structured evidence from the document. Keep the JSON under {budget_tokens} tokens.{q}\\nDocument:\\n{doc}\\n\\nJSON:" | 46 | + "\nThe reader's future questions are unknown; capture every concrete fact.\n" | ||
| 33 | − | 47 | + return f"Extract structured evidence from the document. Keep the JSON under | ||
| 48 | + {budget_tokens} tokens.{q}\nDocument:\n{doc}\n\nJSON:" | ||||
| 34 | \end{verbatim}\end{footnotesize} | 49 | \end{verbatim}\end{footnotesize} | ||
modified
paper/figures/fig2_quality_vs_bytes.pdf
+0 −0
Binary file not shown.
modified
paper/figures/fig3_quality_vs_latency_memory.pdf
+0 −0
Binary file not shown.
modified
paper/figures/fig4_latency_decomposition.pdf
+0 −0
Binary file not shown.
modified
paper/figures/fig5_handoffs.pdf
+0 −0
Binary file not shown.
modified
paper/figures/fig6_breakeven.pdf
+0 −0
Binary file not shown.
modified
paper/figures/fig7_router.pdf
+0 −0
Binary file not shown.
added
paper/figures/figA_qtypes_exploratory.pdf
+0 −0
Binary file not shown.
modified
paper/figures/figA_quality_vs_bytes_aware.pdf
+0 −0
Binary file not shown.
modified
paper/figures/figA_quality_vs_bytes_blind.pdf
+0 −0
Binary file not shown.
modified
paper/main.pdf
+0 −0
Binary file not shown.
modified
paper/sections/abstract_text.tex
+21 −1
@@ -1 +1,21 @@ | |||||
| 1 | −\newcommand{\WPabstract}{Abstract to be written after the results.} | 1 | +\newcommand{\WPabstract}{% | ||
| 2 | +Agents built on language models hand information to one another as text, summaries, structured evidence, shared-memory references or---when | ||||
| 3 | +they share a checkpoint---attention state (KV cache). We ask under which task, model, cache, network and reuse conditions each representation | ||||
| 4 | +preserves task-relevant information at the lowest end-to-end cost, and answer with a reproducible study executed by an autonomous research | ||||
| 5 | +operator (Claude Code) on Apple-Silicon hardware with the MLX runtime, whose caches are inspectable tensors. Twenty-two conditions (text | ||||
| 6 | +truncation, extractive selection, summaries, structured evidence with spans, references with metered retrieval, a lossless codec, LLMLingua-2, | ||||
| 7 | +native and serialized caches, packed 8/4-bit quantization, sink-plus-recent, random, SnapKV-style and PyramidKV-style eviction) are measured on | ||||
| 8 | +Qwen2.5-3B and Llama-3.2-3B across private-fact ledgers, LongBench QA and executable tasks, in question-blind and question-aware regimes, with | ||||
| 9 | +measured bytes, synchronized latency spans and peak memory (13\,280 trials; frozen protocol, cluster bootstrap, Holm-corrected tests). Further | ||||
| 10 | +experiments measure two-node cache transfer over a 1~GbE LAN (324 runs), relays of 1--8 handoffs with an evaluator-only fact ledger (7\,200), | ||||
| 11 | +fan-in with conflicting evidence (1\,600), a same-family learned KV mapper, hosted receivers (672) and a development-fitted router. Lossless | ||||
| 12 | +representations are interchangeable in quality and differ only in cost, whose ranking inverts between cold and reused regimes; on the measured | ||||
| 13 | +link, moving an 8-bit cache beats re-reading the text at every length. Lossy choices are model- and regime-dependent: 4-bit quantization is free | ||||
| 14 | +for Llama and fatal for Qwen; attention-based eviction keeps 78--98\,\% of private-fact accuracy with the question and 12--19\,\% without it, and | ||||
| 15 | +no eviction survives a code specification. At a 512-token budget prose summaries beat structured evidence (38--44 vs 12--15\,\%), relays are | ||||
| 16 | +nearly lossless after the first hop and invent almost nothing, and metered repair buys a few points for kilobytes of re-reading. A ridge KV mapper | ||||
| 17 | +transfers no content while producing coherent abstention, and an invalid cache concatenation yields plausible answers---both argue for | ||||
| 18 | +compatibility manifests and negative controls. The router halves amortized latency at equal quality when questions are unknown and does not help | ||||
| 19 | +otherwise. Cross-family cache transfer was not executed; the study is partial in that respect. Code, records, protocol, checkpoint and a | ||||
| 20 | +minimal reproduction (64/64 identical outputs) are released.% | ||||
| 21 | +} | ||||
modified
paper/sections/conclusion.tex
+21 −1
@@ -1,3 +1,23 @@ | |||||
| 1 | \section{Conclusion} | 1 | \section{Conclusion} | ||
| 2 | \label{sec:conclusion} | 2 | \label{sec:conclusion} | ||
| 3 | −To be written from results. | 3 | +We asked under which conditions each way of passing information between model executions preserves what matters at the lowest measured cost, | ||
| 4 | +and we answered with a single harness on commodity hardware, real cache tensors, measured bytes and synchronized clocks, released records for | ||||
| 5 | +every attempt, and a protocol frozen before the test split was touched. The bounded findings are these. (1) Lossless representations are | ||||
| 6 | +interchangeable: text, its zstd encoding, the native cache, the reloaded cache and the 8-bit packed cache give identical answers; the choice | ||||
| 7 | +among them is a pure cost question, and the cost ranking flips between the cold case (text wins: no build) and the reused case (cache wins: | ||||
| 8 | +no prefill), with a 1~GbE link and a 3B model already favouring moving an 8-bit cache over re-reading at two thousand tokens. | ||||
| 9 | +(2) Every lossy choice we measured is model- or regime-dependent: 4-bit per-token quantization is free for Llama-3.2-3B and fatal for | ||||
| 10 | +Qwen2.5-3B; attention-based eviction is excellent with the question and useless without it, and no eviction survives a code specification; | ||||
| 11 | +LLMLingua-2 drops exactly the identifiers private-fact questions ask for. (3) At a 512-token sender budget, prose summaries carried more than | ||||
| 12 | +structured evidence with provenance, in single handoffs, in eight-hop relays and in fan-in, because provenance is paid in the sender's tokens; | ||||
| 13 | +relays themselves were nearly lossless after the first hop and invented almost nothing, and metered repair bought a few points for kilobytes of | ||||
| 14 | +re-reading. (4) A same-family linear KV mapper that reconstructs most of the receiver's cache variance transfers no content, and produces the | ||||
| 15 | +most dangerous kind of failure---coherent abstention---while an invalid cache concatenation produced plausible answers; compatibility manifests | ||||
| 16 | +and negative controls are therefore part of any cache-transfer protocol, not an afterthought. (5) A router fitted on development data matched | ||||
| 17 | +the best static policy at half the amortized latency when questions were unknown and offered nothing when they were known. | ||||
| 18 | + | ||||
| 19 | +Open questions follow directly. The structured-evidence result should be re-run with budgets that separate provenance from values and with | ||||
| 20 | +senders large enough to fill them; the break-even map needs a faster receiver and a slower link to locate its crossing; the transfer ladder's | ||||
| 21 | +cross-family rung needs a trained mapper with generation and distillation terms, and its evaluation must keep the sender-alone and | ||||
| 22 | +receiver-alone references and an identity control beside it; and the fixed-point behaviour of structured relays suggests that the first | ||||
| 23 | +message, not the chain, is where compression research for multi-agent systems should concentrate. | ||||
modified
paper/sections/results_transfer.tex
+28 −1
@@ -1 +1,28 @@ | |||||
| 1 | −% pending | 1 | +\subsection{Cross-checkpoint transfer: the same-family linear mapper (E3)} | ||
| 2 | +\label{sec:transfer} | ||||
| 3 | +Table~\ref{tab:transfer} reports the compatibility ladder's second rung on the synthetic (27 documents of 2k and 8k tokens, 8 questions each) and | ||||
| 4 | +executable (40) test items: the receiver Qwen2.5-3B reading the document itself (A1), the sender Qwen2.5-1.5B's own answers with full context | ||||
| 5 | +(sender alone), two text handoffs from the 1.5B sender (A4, A5), the identity-transfer negative control (C0: sender cache fed to the receiver | ||||
| 6 | +without mapping) and the ridge mapper (C2) alone and followed by question-aware selection (C3). The mapper was fitted on 60 training documents | ||||
| 7 | +(248\,569 token positions, 28.3~M parameters, 483~s on M3U96b, held-out reconstruction error $0.10$ for keys and $0.32$ for values, relative | ||||
| 8 | +Frobenius; \texttt{runs/checkpoints}); it needs the sender's cache and nothing else at the receiver (no source text, no prefill). | ||||
| 9 | + | ||||
| 10 | +\begin{table}[t]\centering\scriptsize | ||||
| 11 | +\caption{Same-family transfer Qwen2.5-1.5B $\to$ Qwen2.5-3B (test split; M3U96a). Accuracy (\%, 95\,\% cluster-bootstrap CI), share of answers | ||||
| 12 | +containing ``unknown'', accuracy by question type, median payload and warm per-query latency. Negation questions are yes/no.}\label{tab:transfer} | ||||
| 13 | +\adjustbox{max width=\textwidth}{\input{tables/table_transfer}} | ||||
| 14 | +\end{table} | ||||
| 15 | + | ||||
| 16 | +\paragraph{The mapped cache is well-formed but empty (H10 confirmed).} The receiver generates fluent, on-task text from the mapped cache---it | ||||
| 17 | +answers ``unknown'' 69\,\% of the time and never produces the garbled tokens of the identity control---but it recovers essentially none of the | ||||
| 18 | +content: 0\,\% on multi-hop, 2--4\,\% on single facts and exact copies, 0\,\% on code. Its aggregate score of 24.5\,\% comes from the yes/no negation | ||||
| 19 | +questions (93\,\%), which a receiver that answers ``no'' by default gets right at the base rate (70\,\% of projects are not cancelled); the sender's | ||||
| 20 | +own full-context answers show that the facts were available on the sender side (96\,\%). The 1.5B sender's prose summary transfers more | ||||
| 21 | +(28\,\%, 72\,\% on negation, 9--20\,\% on content) and its structured extraction less (12\,\%); the paired difference between the mapper and the | ||||
| 22 | +summary is $-3.7$ points (95\,\% CI $[-11.6, +4.2]$, $p=0.37$), and between the mapper and the receiver's own reading $-75$ points. Selection on | ||||
| 23 | +the mapped cache (C3) changes nothing. A linear map that explains 68\,\% of the receiver's value variance on held-out documents is therefore | ||||
| 24 | +not enough to carry a single identifier across checkpoints: cross-checkpoint transfer needs the generation and distillation terms that | ||||
| 25 | +published mappers train with, at a cost we did not have; the measured by-product is that a \emph{wrong} cache can be indistinguishable from an | ||||
| 26 | +empty one in the receiver's behaviour (coherent abstention), which is why Section~\ref{sec:fanin}'s manifests matter. Rungs 1 (same checkpoint, | ||||
| 27 | +two nodes) and 3 (cross-family) are covered by Section~\ref{sec:network} and the coverage table respectively; the cross-family rung was not | ||||
| 28 | +executed. | ||||
added
paper/tables/table_failures.tex
+23 −0
@@ -0,0 +1,23 @@ | |||||
| 1 | +\begin{tabular}{@{}lrrr@{}} | ||||
| 2 | +\toprule | ||||
| 3 | +method & implementation\_error & invalid\_output & success \\ | ||||
| 4 | +\midrule | ||||
| 5 | +A0 & 0 & 0 & 640 \\ | ||||
| 6 | +A1 & 0 & 0 & 640 \\ | ||||
| 7 | +A2 & 0 & 0 & 640 \\ | ||||
| 8 | +A3 & 0 & 0 & 1280 \\ | ||||
| 9 | +A4 & 0 & 0 & 880 \\ | ||||
| 10 | +A5 & 0 & 0 & 880 \\ | ||||
| 11 | +A6 & 0 & 0 & 640 \\ | ||||
| 12 | +A7 & 0 & 0 & 640 \\ | ||||
| 13 | +A8 & 240 & 0 & 640 \\ | ||||
| 14 | +B0 & 0 & 0 & 640 \\ | ||||
| 15 | +B1 & 0 & 0 & 640 \\ | ||||
| 16 | +B2 & 0 & 29 & 1251 \\ | ||||
| 17 | +B3 & 0 & 0 & 640 \\ | ||||
| 18 | +B4 & 0 & 45 & 595 \\ | ||||
| 19 | +B5 & 0 & 0 & 1280 \\ | ||||
| 20 | +B6 & 0 & 0 & 640 \\ | ||||
| 21 | +B9 & 0 & 0 & 640 \\ | ||||
| 22 | +\bottomrule | ||||
| 23 | +\end{tabular} | ||||
modified
paper/tables/table_primary.tex
+1 −1
@@ -13,7 +13,7 @@ A5 512 aware & 100 [100,100] & 95 [91,98] & -- & -- & -- & -- & -- & -- \\ | |||||
| 13 | A5 512 blind & 15 [12,18] & 12 [9,16] & 12 [2,22] & 0 [0,0] & 9 [2,18] & 5 [1,11] & 12 [4,23] & 6 [0,13] \\ | 13 | A5 512 blind & 15 [12,18] & 12 [9,16] & 12 [2,22] & 0 [0,0] & 9 [2,18] & 5 [1,11] & 12 [4,23] & 6 [0,13] \\ | ||
| 14 | A6 512 blind & 18 [14,22] & 18 [14,22] & 22 [10,35] & 15 [5,28] & 9 [2,17] & 7 [2,13] & 11 [2,22] & 10 [2,19] \\ | 14 | A6 512 blind & 18 [14,22] & 18 [14,22] & 22 [10,35] & 15 [5,28] & 9 [2,17] & 7 [2,13] & 11 [2,22] & 10 [2,19] \\ | ||
| 15 | A7 zstd & 99 [98,100] & 99 [98,100] & 100 [100,100] & 95 [88,100] & 35 [24,47] & 25 [15,36] & 39 [25,54] & 33 [20,47] \\ | 15 | A7 zstd & 99 [98,100] & 99 [98,100] & 100 [100,100] & 95 [88,100] & 35 [24,47] & 25 [15,36] & 39 [25,54] & 33 [20,47] \\ | ||
| 16 | −A8 512 blind & 14 [9,18] & 0 [0,0] & 0 [0,0] & 0 [0,0] & 17 [10,25] & 23 [13,33] & 6 [0,14] & 8 [0,18] \\ | 16 | +A8 512 blind & 14 [9,18] & 16 [10,22] & 0 [0,0] & 0 [0,0] & 17 [10,25] & 23 [13,33] & 6 [0,14] & 8 [0,18] \\ | ||
| 17 | B0 & 99 [98,100] & 99 [98,100] & 100 [100,100] & 95 [88,100] & 35 [24,47] & 25 [15,36] & 36 [22,50] & 33 [20,47] \\ | 17 | B0 & 99 [98,100] & 99 [98,100] & 100 [100,100] & 95 [88,100] & 35 [24,47] & 25 [15,36] & 36 [22,50] & 33 [20,47] \\ | ||
| 18 | B1 & 99 [98,100] & 99 [98,100] & 100 [100,100] & 95 [88,100] & 35 [24,47] & 25 [15,36] & 36 [22,50] & 33 [20,47] \\ | 18 | B1 & 99 [98,100] & 99 [98,100] & 100 [100,100] & 95 [88,100] & 35 [24,47] & 25 [15,36] & 36 [22,50] & 33 [20,47] \\ | ||
| 19 | B2 bits=4 & 0 [0,0] & 98 [94,100] & 0 [0,0] & 100 [100,100] & 1 [0,2] & 25 [15,36] & 0 [0,1] & 22 [10,34] \\ | 19 | B2 bits=4 & 0 [0,0] & 98 [94,100] & 0 [0,0] & 100 [100,100] & 1 [0,2] & 25 [15,36] & 0 [0,1] & 22 [10,34] \\ | ||
added
paper/tables/table_qtype_core.tex
+28 −0
@@ -0,0 +1,28 @@ | |||||
| 1 | +\begin{tabular}{@{}lrrr@{}} | ||||
| 2 | +\toprule | ||||
| 3 | +condition & exact\_copy & multi\_hop & single\_fact \\ | ||||
| 4 | +\midrule | ||||
| 5 | +A0 & 0 [0,0] & 0 [0,0] & 0 [0,0] \\ | ||||
| 6 | +A1 & 99 [98,100] & 98 [94,100] & 99 [98,100] \\ | ||||
| 7 | +A2 512 & 14 [8,20] & 20 [8,32] & 26 [18,35] \\ | ||||
| 8 | +A3 512 aware & 99 [98,100] & 92 [86,98] & 100 [100,100] \\ | ||||
| 9 | +A3 512 blind & 51 [50,52] & 94 [88,99] & 0 [0,0] \\ | ||||
| 10 | +A4 512 aware & 79 [69,88] & -- & 94 [90,98] \\ | ||||
| 11 | +A4 512 blind & 45 [38,53] & 46 [35,57] & 34 [24,45] \\ | ||||
| 12 | +A5 512 aware & 98 [94,100] & -- & 98 [96,99] \\ | ||||
| 13 | +A5 512 blind & 13 [8,18] & 11 [4,21] & 16 [10,22] \\ | ||||
| 14 | +A6 512 blind & 15 [9,21] & 16 [6,26] & 21 [14,29] \\ | ||||
| 15 | +A7 zstd & 99 [98,100] & 98 [94,100] & 99 [98,100] \\ | ||||
| 16 | +A8 512 blind & 10 [5,16] & 5 [1,10] & 24 [16,34] \\ | ||||
| 17 | +B0 & 99 [98,100] & 98 [94,100] & 99 [98,100] \\ | ||||
| 18 | +B1 & 99 [98,100] & 98 [94,100] & 99 [98,100] \\ | ||||
| 19 | +B2 bits=4 & 49 [47,50] & 49 [46,50] & 49 [47,50] \\ | ||||
| 20 | +B2 bits=8 & 99 [98,100] & 98 [94,100] & 99 [98,100] \\ | ||||
| 21 | +B3 1/4 & 31 [23,38] & 12 [2,22] & 11 [5,18] \\ | ||||
| 22 | +B4 1/4 & 0 [0,0] & 0 [0,0] & 0 [0,0] \\ | ||||
| 23 | +B5 1/4 aware & 97 [94,99] & 82 [76,90] & 82 [79,86] \\ | ||||
| 24 | +B5 1/4 blind & 20 [13,27] & 15 [8,22] & 11 [6,17] \\ | ||||
| 25 | +B6 1/4 aware & 92 [88,96] & 78 [69,85] & 74 [71,76] \\ | ||||
| 26 | +B9 bits=8 1/4 att aware & 97 [94,99] & 84 [78,90] & 82 [79,86] \\ | ||||
| 27 | +\bottomrule | ||||
| 28 | +\end{tabular} | ||||
added
paper/tables/table_qtypes_exploratory.tex
+20 −0
@@ -0,0 +1,20 @@ | |||||
| 1 | +\begin{tabular}{@{}lrrrrrrr@{}} | ||||
| 2 | +\toprule | ||||
| 3 | +condition / pair & aggregation & multi\_hop & negation & revision & temporal & unanswerable & unit\_conversion \\ | ||||
| 4 | +\midrule | ||||
| 5 | +A1 Llama-3.2-3B & 0 [0,0] & 100 [100,100] & 90 [80,100] & 73 [58,88] & 47 [20,73] & 100 [100,100] & 0 [0,0] \\ | ||||
| 6 | +A1 Qwen2.5-3B & 0 [0,0] & 100 [100,100] & 100 [100,100] & 94 [84,100] & 44 [22,69] & 97 [91,100] & 50 [25,75] \\ | ||||
| 7 | +A3 512 blind Llama-3.2-3B & 0 [0,0] & 0 [0,0] & 83 [70,93] & 73 [57,87] & 53 [33,73] & 100 [100,100] & 0 [0,0] \\ | ||||
| 8 | +A3 512 blind Qwen2.5-3B & 0 [0,0] & 0 [0,0] & 93 [83,100] & 73 [53,90] & 33 [13,53] & 100 [100,100] & 0 [0,0] \\ | ||||
| 9 | +A4 512 blind Llama-3.2-3B & 0 [0,0] & 33 [7,60] & 100 [100,100] & 17 [3,33] & 20 [0,40] & 100 [100,100] & 0 [0,0] \\ | ||||
| 10 | +A4 512 blind Qwen2.5-3B & 0 [0,0] & 44 [19,69] & 72 [53,88] & 26 [13,43] & 33 [13,53] & 100 [100,100] & 40 [13,67] \\ | ||||
| 11 | +A5 512 blind Llama-3.2-3B & 0 [0,0] & 13 [0,33] & 50 [37,63] & 13 [0,30] & 13 [0,33] & 100 [100,100] & 0 [0,0] \\ | ||||
| 12 | +A5 512 blind Qwen2.5-3B & 0 [0,0] & 20 [7,40] & 53 [37,70] & 10 [0,27] & 0 [0,0] & 100 [100,100] & 20 [0,40] \\ | ||||
| 13 | +B2 bits=8 Llama-3.2-3B & 0 [0,0] & 100 [100,100] & 90 [80,100] & 73 [58,88] & 47 [20,73] & 100 [100,100] & 0 [0,0] \\ | ||||
| 14 | +B2 bits=8 Qwen2.5-3B & 0 [0,0] & 100 [100,100] & 100 [100,100] & 93 [83,100] & 40 [13,67] & 97 [90,100] & 60 [33,87] \\ | ||||
| 15 | +B3 1/4 Llama-3.2-3B & 0 [0,0] & 20 [0,40] & 73 [60,83] & 7 [0,17] & 53 [27,73] & 93 [80,100] & 0 [0,0] \\ | ||||
| 16 | +B3 1/4 Qwen2.5-3B & 0 [0,0] & 20 [0,40] & 87 [73,97] & 0 [0,0] & 53 [27,73] & 97 [90,100] & 7 [0,20] \\ | ||||
| 17 | +B5 1/4 aware Llama-3.2-3B & 0 [0,0] & 100 [100,100] & 87 [77,97] & 73 [62,87] & 60 [40,80] & 100 [100,100] & 0 [0,0] \\ | ||||
| 18 | +B5 1/4 aware Qwen2.5-3B & 0 [0,0] & 53 [27,80] & 100 [100,100] & 30 [17,43] & 53 [27,80] & 97 [90,100] & 47 [20,73] \\ | ||||
| 19 | +\bottomrule | ||||
| 20 | +\end{tabular} | ||||
added
paper/tables/table_transfer.tex
+22 −0
@@ -0,0 +1,22 @@ | |||||
| 1 | +\begin{tabular}{@{}llrrrrrrrrrr@{}} | ||||
| 2 | +\toprule | ||||
| 3 | +family & condition & n & accuracy (\\%) & `unknown' share & single\_fact & exact\_copy & multi\_hop & negation & exec & payload B & warm ms \\ | ||||
| 4 | +\midrule | ||||
| 5 | +exec & A1 & 40 & 100 [100,100] & 0.00 & -- & -- & -- & -- & 1.00 & 7,817.00 & 1,912.17 \\ | ||||
| 6 | +exec & A4|512|blind & 40 & 0 [0,0] & 0.28 & -- & -- & -- & -- & 0.00 & 938.00 & 1,025.84 \\ | ||||
| 7 | +exec & A5|512|blind & 40 & 0 [0,0] & 0.85 & -- & -- & -- & -- & 0.00 & 936.00 & 215.30 \\ | ||||
| 8 | +exec & C0 & 40 & 0 [0,0] & 0.00 & -- & -- & -- & -- & 0.00 & 69,762,785 & 3,979.97 \\ | ||||
| 9 | +exec & C2 & 40 & 0 [0,0] & 0.00 & -- & -- & -- & -- & 0.00 & 69,762,886 & 1,098.63 \\ | ||||
| 10 | +exec & C3|1/4|aware & 40 & 0 [0,0] & 0.00 & -- & -- & -- & -- & 0.00 & 17,452,748 & 939.02 \\ | ||||
| 11 | +exec & sender alone A1 & 40 & 0 [0,0] & 0.00 & -- & -- & -- & -- & 0.00 & 7,817.00 & 909.11 \\ | ||||
| 12 | +exec & sender alone B0 & 40 & 0 [0,0] & 0.00 & -- & -- & -- & -- & 0.00 & 54,260,105 & 621.42 \\ | ||||
| 13 | +synthetic & A1 & 216 & 100 [99,100] & 0.00 & 1.00 & 1.00 & 0.98 & 1.00 & -- & 7,555.00 & 890.79 \\ | ||||
| 14 | +synthetic & A4|512|blind & 216 & 28 [22,35] & 0.64 & 0.09 & 0.20 & 0.11 & 0.72 & -- & 1,263.00 & 250.39 \\ | ||||
| 15 | +synthetic & A5|512|blind & 216 & 12 [7,16] & 0.73 & 0.07 & 0.09 & 0.07 & 0.22 & -- & 596.00 & 166.89 \\ | ||||
| 16 | +synthetic & C0 & 216 & 0 [0,0] & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & -- & 77,688,549 & 632.53 \\ | ||||
| 17 | +synthetic & C2 & 216 & 25 [22,27] & 0.69 & 0.02 & 0.04 & 0.00 & 0.93 & -- & 77,688,650 & 85.57 \\ | ||||
| 18 | +synthetic & C3|1/4|aware & 216 & 25 [23,27] & 0.69 & 0.04 & 0.02 & 0.00 & 0.94 & -- & 19,443,415 & 56.23 \\ | ||||
| 19 | +synthetic & sender alone A1 & 216 & 96 [94,98] & 0.00 & 0.98 & 0.98 & 0.94 & 0.93 & -- & 7,555.00 & 448.40 \\ | ||||
| 20 | +synthetic & sender alone B0 & 216 & 95 [93,98] & 0.00 & 0.98 & 0.98 & 0.94 & 0.91 & -- & 60,424,589 & 104.16 \\ | ||||
| 21 | +\bottomrule | ||||
| 22 | +\end{tabular} | ||||
added
results/claim_ledger.csv
+21 −0
@@ -0,0 +1,21 @@ | |||||
| 1 | +claim_id,claim,experiment_ids,evidence_table_or_figure,rows_or_filter,script,status,limitation | ||||
| 2 | +C01,"Lossless representations (A1, A7, B0, B1, B2-8bit) give the same quality within noise on every family and pair",E1_core,"results/tables/E1_core_primary_tests.csv (T2, T6); table_primary","rows T2,T6",analyze.py,confirmed,"two 3B pairs, 160 test documents" | ||||
| 3 | +C02,4-bit packed KV quantization destroys Qwen2.5-3B (0%) but not Llama-3.2-3B (>=97%),E1_core,E1_core_primary.csv B2|bits=4; primary_tests T3,condition B2|bits=4,analyze.py,confirmed (H2),"mlx per-token affine quantization, group 64" | ||||
| 4 | +C03,Attention-informed eviction with the question (B5-aware) beats sink+recent at 1/4 by +44 points; blind window variant collapses,E1_core,primary_tests T4; E1_core_primary.csv,B5 vs B3,analyze.py,"confirmed (H3 first half, H7)",question-aware method | ||||
| 5 | +C04,All eviction methods score 0 on executable tasks at 1/4 retention,E1_core,E1_core_primary.csv family=exec,"B3,B4,B5,B6,B9",analyze.py,measured,"3B receivers, 6-rule spec" | ||||
| 6 | +C05,"Structured evidence (A5) is below prose summary (A4) at the same 512-token budget (-27 points, synthetic)",E1_core,primary_tests T1,T1,analyze.py,H1 rejected,sender spends budget on quotes; ~8 facts of 62 | ||||
| 7 | +C06,zstd shrinks text ~3.5x with no quality/latency effect; ~20% on bf16 caches,"E1_core, E6_core",E1_core_primary.csv A7; E6_network.csv codec=zstd,A7; B1 zstd,analyze.py,confirmed (H8),zstd level 3 | ||||
| 8 | +C07,Serialized 3B cache is 3-4 orders of magnitude larger than the text,"E1_core, E6_core",E1_core_primary.csv payload_bytes_median; E6_network.csv bytes,B0 vs A1,analyze.py,measured,"bf16, Qwen 215 MB vs 29 kB at ~8k tokens" | ||||
| 9 | +C08,"On 1 GbE with an M3 Ultra receiver, moving the cache (bf16 or 8-bit) answers faster than remote text prefill at 2k-16k tokens",E6_core,E6_network.csv; fig6,lan rows,analyze.py,measured,"one link, one receiver; break-even depends on prefill/link ratio" | ||||
| 10 | +C09,Repeated prose/structured relays lose facts at hop 1 and are nearly lossless afterwards (no monotone decay),E4_core,E4_core_handoffs.csv; fig5,hops 1-8,analyze.py,H5 rejected,ledger metric = normalized value containment | ||||
| 11 | +C10,"Unsupported additions are near zero in 7,200 relayed messages",E4_core,E4_core_handoffs.csv unsupported_mean,all chains,analyze.py,measured,regex-based detection | ||||
| 12 | +C11,Metered repair raises prose-chain retention 32->38% and accuracy 40->44% for ~5.9 kB cumulative fetch,E4_core,E4_core_handoffs.csv repairable,A4 repairable,analyze.py,H6 partial,repair limited to entities still named | ||||
| 13 | +C12,Concatenated summaries compose as well as full context on multi-hop joins; merged structured records do not,E5_core,E5_core_fanin.csv; table_fanin,concat_A4 vs merge_A5,analyze.py,measured,"k=2,4; 3B aggregator" | ||||
| 14 | +C13,Cache concatenation of independently prefilled sources yields plausible answers (70% multi-hop at k=2) despite invalid positions,E5_core,E5_core_fanin.csv cache_concat_negative_control,"k=2,4",analyze.py,negative control,diagnostic only | ||||
| 15 | +C14,"Hosted receivers recover 100% from full text, ~30% from the local 512-token summary, 12.5% from structured JSON",ED_hosted_core,ED_hosted.csv; table_hosted,synthetic rows,analyze.py,measured,provider default sampling; 8 documents/family | ||||
| 16 | +C15,"Anthropic prompt caching did not engage below the documented 4,096-token minimum (2k docs) and halved estimated cost on 8k docs",ED_hosted_core,runs/raw/ED_hosted_core.jsonl provider_cached_usage,A1 provider_cache rows,hosted_run.py,measured,billing fields only | ||||
| 17 | +C16,Dev-fitted router matches best static quality at half the amortized latency in the blind regime; no gain and higher cost in the aware regime,E8_router (E1_core records),E8_router_summary.csv; E8_router_tests.csv; fig7,router vs best_static_dev,router_eval.py,H9 mixed,off-policy on full factorial; dev table from pilot | ||||
| 18 | +C17,Linear ridge KV mapper (Qwen 1.5B->3B) yields coherent but uninformative receiver outputs; below sender-summary transfer,E3_core,E3_core tables,C2 vs A4,analyze.py,confirmed (H10): C2 24.5% overall but 0-4% on content questions; negation yes/no at base rate,reconstruction-only mapper; no SwaT reproduction | ||||
| 19 | +C18,Identity round trip and 100% retention reproduce native logits exactly; cached-prefix equivalence holds within registered tolerance,E0 tests,tests/test_cache_correctness.py,12 tests,pytest,confirmed,MLX nondeterminism across chunking documented | ||||
| 20 | +C19,Minimal reproduction reran 64 trials from a clean process with identical outputs and scores,E9,reproduction/report.json,all,reproduce.py,confirmed,"same machine, same chunking" | ||||
| 21 | +C20,Cross-family cache transfer was not executed (no SwaT checkpoint/licence; mapper budget),coverage,literature/audit_group1.md; table1_taxonomy,SwaT row,n/a,coverage gap,partial transfer study | ||||
modified
results/figures/fig2_quality_vs_bytes.pdf
+0 −0
Binary file not shown.
modified
results/figures/fig2_quality_vs_bytes.png
+0 −0
Binary file not shown.
modified
results/figures/fig3_quality_vs_latency_memory.pdf
+0 −0
Binary file not shown.
modified
results/figures/fig4_latency_decomposition.pdf
+0 −0
Binary file not shown.
modified
results/figures/fig5_handoffs.pdf
+0 −0
Binary file not shown.
modified
results/figures/fig6_breakeven.pdf
+0 −0
Binary file not shown.
modified
results/figures/fig7_router.pdf
+0 −0
Binary file not shown.
added
results/figures/figA_qtypes_exploratory.pdf
+0 −0
Binary file not shown.
added
results/figures/figA_qtypes_exploratory.png
+0 −0
Binary file not shown.
modified
results/figures/figA_quality_vs_bytes_aware.pdf
+0 −0
Binary file not shown.
modified
results/figures/figA_quality_vs_bytes_blind.pdf
+0 −0
Binary file not shown.
modified
results/figures/figA_quality_vs_bytes_blind.png
+0 −0
Binary file not shown.
modified
results/provenance.json
+67 −25
@@ -4,7 +4,7 @@ | |||||
| 4 | "runs/raw/E1_core.jsonl" | 4 | "runs/raw/E1_core.jsonl" | ||
| 5 | ], | 5 | ], | ||
| 6 | "script": "src/wp30/analysis/analyze.py", | 6 | "script": "src/wp30/analysis/analyze.py", | ||
| 7 | − "generated_utc": "2026-10-30T08:41:10.888608Z", | 7 | + "generated_utc": "2026-10-30T09:02:37.609345Z", | ||
| 8 | "note": "per-condition quality (cluster bootstrap CI) and resource medians" | 8 | "note": "per-condition quality (cluster bootstrap CI) and resource medians" | ||
| 9 | }, | 9 | }, | ||
| 10 | "results/tables/E1_core_failures.csv": { | 10 | "results/tables/E1_core_failures.csv": { | ||
@@ -12,7 +12,7 @@ | |||||
| 12 | "runs/raw/E1_core.jsonl" | 12 | "runs/raw/E1_core.jsonl" | ||
| 13 | ], | 13 | ], | ||
| 14 | "script": "src/wp30/analysis/analyze.py", | 14 | "script": "src/wp30/analysis/analyze.py", | ||
| 15 | − "generated_utc": "2026-10-30T08:41:10.897925Z", | 15 | + "generated_utc": "2026-10-30T09:02:37.611126Z", | ||
| 16 | "note": "attempt status counts by method" | 16 | "note": "attempt status counts by method" | ||
| 17 | }, | 17 | }, | ||
| 18 | "results/tables/E1_core_by_question_type.csv": { | 18 | "results/tables/E1_core_by_question_type.csv": { | ||
@@ -20,7 +20,7 @@ | |||||
| 20 | "runs/raw/E1_core.jsonl" | 20 | "runs/raw/E1_core.jsonl" | ||
| 21 | ], | 21 | ], | ||
| 22 | "script": "src/wp30/analysis/analyze.py", | 22 | "script": "src/wp30/analysis/analyze.py", | ||
| 23 | − "generated_utc": "2026-10-30T08:41:11.023256Z", | 23 | + "generated_utc": "2026-10-30T09:02:37.730657Z", | ||
| 24 | "note": "synthetic accuracy by question type" | 24 | "note": "synthetic accuracy by question type" | ||
| 25 | }, | 25 | }, | ||
| 26 | "results/tables/E1_core_primary_tests.csv": { | 26 | "results/tables/E1_core_primary_tests.csv": { | ||
@@ -28,7 +28,7 @@ | |||||
| 28 | "runs/raw/E1_core.jsonl" | 28 | "runs/raw/E1_core.jsonl" | ||
| 29 | ], | 29 | ], | ||
| 30 | "script": "src/wp30/analysis/analyze.py", | 30 | "script": "src/wp30/analysis/analyze.py", | ||
| 31 | − "generated_utc": "2026-10-30T08:41:11.097268Z", | 31 | + "generated_utc": "2026-10-30T09:02:37.786568Z", | ||
| 32 | "note": "registered primary comparisons, paired, cluster bootstrap, Holm" | 32 | "note": "registered primary comparisons, paired, cluster bootstrap, Holm" | ||
| 33 | }, | 33 | }, | ||
| 34 | "results/figures/fig2_quality_vs_bytes.pdf": { | 34 | "results/figures/fig2_quality_vs_bytes.pdf": { | ||
@@ -36,7 +36,7 @@ | |||||
| 36 | "runs/raw/E1_core.jsonl" | 36 | "runs/raw/E1_core.jsonl" | ||
| 37 | ], | 37 | ], | ||
| 38 | "script": "src/wp30/analysis/analyze.py", | 38 | "script": "src/wp30/analysis/analyze.py", | ||
| 39 | − "generated_utc": "2026-10-30T08:41:11.754908Z", | 39 | + "generated_utc": "2026-10-30T09:02:38.459020Z", | ||
| 40 | "note": "points = conditions; x = median payload bytes; y = mean quality" | 40 | "note": "points = conditions; x = median payload bytes; y = mean quality" | ||
| 41 | }, | 41 | }, | ||
| 42 | "results/figures/fig2_quality_vs_bytes.png": { | 42 | "results/figures/fig2_quality_vs_bytes.png": { | ||
@@ -44,7 +44,7 @@ | |||||
| 44 | "runs/raw/E1_core.jsonl" | 44 | "runs/raw/E1_core.jsonl" | ||
| 45 | ], | 45 | ], | ||
| 46 | "script": "src/wp30/analysis/analyze.py", | 46 | "script": "src/wp30/analysis/analyze.py", | ||
| 47 | − "generated_utc": "2026-10-30T08:41:11.951700Z", | 47 | + "generated_utc": "2026-10-30T09:02:38.666585Z", | ||
| 48 | "note": "points = conditions; x = median payload bytes; y = mean quality" | 48 | "note": "points = conditions; x = median payload bytes; y = mean quality" | ||
| 49 | }, | 49 | }, | ||
| 50 | "results/figures/figA_quality_vs_bytes_blind.pdf": { | 50 | "results/figures/figA_quality_vs_bytes_blind.pdf": { | ||
@@ -52,7 +52,7 @@ | |||||
| 52 | "runs/raw/E1_core.jsonl" | 52 | "runs/raw/E1_core.jsonl" | ||
| 53 | ], | 53 | ], | ||
| 54 | "script": "src/wp30/analysis/analyze.py", | 54 | "script": "src/wp30/analysis/analyze.py", | ||
| 55 | − "generated_utc": "2026-10-30T08:41:12.396339Z", | 55 | + "generated_utc": "2026-10-30T09:02:39.176959Z", | ||
| 56 | "note": "points = conditions; x = median payload bytes; y = mean quality" | 56 | "note": "points = conditions; x = median payload bytes; y = mean quality" | ||
| 57 | }, | 57 | }, | ||
| 58 | "results/figures/figA_quality_vs_bytes_blind.png": { | 58 | "results/figures/figA_quality_vs_bytes_blind.png": { | ||
@@ -60,7 +60,7 @@ | |||||
| 60 | "runs/raw/E1_core.jsonl" | 60 | "runs/raw/E1_core.jsonl" | ||
| 61 | ], | 61 | ], | ||
| 62 | "script": "src/wp30/analysis/analyze.py", | 62 | "script": "src/wp30/analysis/analyze.py", | ||
| 63 | − "generated_utc": "2026-10-30T08:41:12.558258Z", | 63 | + "generated_utc": "2026-10-30T09:02:39.359044Z", | ||
| 64 | "note": "points = conditions; x = median payload bytes; y = mean quality" | 64 | "note": "points = conditions; x = median payload bytes; y = mean quality" | ||
| 65 | }, | 65 | }, | ||
| 66 | "results/figures/figA_quality_vs_bytes_aware.pdf": { | 66 | "results/figures/figA_quality_vs_bytes_aware.pdf": { | ||
@@ -68,7 +68,7 @@ | |||||
| 68 | "runs/raw/E1_core.jsonl" | 68 | "runs/raw/E1_core.jsonl" | ||
| 69 | ], | 69 | ], | ||
| 70 | "script": "src/wp30/analysis/analyze.py", | 70 | "script": "src/wp30/analysis/analyze.py", | ||
| 71 | − "generated_utc": "2026-10-30T08:41:12.888916Z", | 71 | + "generated_utc": "2026-10-30T09:02:39.746284Z", | ||
| 72 | "note": "points = conditions; x = median payload bytes; y = mean quality" | 72 | "note": "points = conditions; x = median payload bytes; y = mean quality" | ||
| 73 | }, | 73 | }, | ||
| 74 | "results/figures/figA_quality_vs_bytes_aware.png": { | 74 | "results/figures/figA_quality_vs_bytes_aware.png": { | ||
@@ -76,7 +76,7 @@ | |||||
| 76 | "runs/raw/E1_core.jsonl" | 76 | "runs/raw/E1_core.jsonl" | ||
| 77 | ], | 77 | ], | ||
| 78 | "script": "src/wp30/analysis/analyze.py", | 78 | "script": "src/wp30/analysis/analyze.py", | ||
| 79 | − "generated_utc": "2026-10-30T08:41:13.061312Z", | 79 | + "generated_utc": "2026-10-30T09:02:39.940182Z", | ||
| 80 | "note": "points = conditions; x = median payload bytes; y = mean quality" | 80 | "note": "points = conditions; x = median payload bytes; y = mean quality" | ||
| 81 | }, | 81 | }, | ||
| 82 | "results/figures/fig3_quality_vs_latency_memory.pdf": { | 82 | "results/figures/fig3_quality_vs_latency_memory.pdf": { | ||
@@ -84,7 +84,7 @@ | |||||
| 84 | "runs/raw/E1_core.jsonl" | 84 | "runs/raw/E1_core.jsonl" | ||
| 85 | ], | 85 | ], | ||
| 86 | "script": "src/wp30/analysis/analyze.py", | 86 | "script": "src/wp30/analysis/analyze.py", | ||
| 87 | − "generated_utc": "2026-10-30T08:41:13.355100Z", | 87 | + "generated_utc": "2026-10-30T09:02:40.285980Z", | ||
| 88 | "note": "" | 88 | "note": "" | ||
| 89 | }, | 89 | }, | ||
| 90 | "results/figures/fig3_quality_vs_latency_memory.png": { | 90 | "results/figures/fig3_quality_vs_latency_memory.png": { | ||
@@ -92,7 +92,7 @@ | |||||
| 92 | "runs/raw/E1_core.jsonl" | 92 | "runs/raw/E1_core.jsonl" | ||
| 93 | ], | 93 | ], | ||
| 94 | "script": "src/wp30/analysis/analyze.py", | 94 | "script": "src/wp30/analysis/analyze.py", | ||
| 95 | − "generated_utc": "2026-10-30T08:41:13.527311Z", | 95 | + "generated_utc": "2026-10-30T09:02:40.475124Z", | ||
| 96 | "note": "" | 96 | "note": "" | ||
| 97 | }, | 97 | }, | ||
| 98 | "results/figures/fig4_latency_decomposition.pdf": { | 98 | "results/figures/fig4_latency_decomposition.pdf": { | ||
@@ -100,7 +100,7 @@ | |||||
| 100 | "runs/raw/E1_core.jsonl" | 100 | "runs/raw/E1_core.jsonl" | ||
| 101 | ], | 101 | ], | ||
| 102 | "script": "src/wp30/analysis/analyze.py", | 102 | "script": "src/wp30/analysis/analyze.py", | ||
| 103 | − "generated_utc": "2026-10-30T08:41:13.606317Z", | 103 | + "generated_utc": "2026-10-30T09:02:40.560198Z", | ||
| 104 | "note": "" | 104 | "note": "" | ||
| 105 | }, | 105 | }, | ||
| 106 | "results/figures/fig4_latency_decomposition.png": { | 106 | "results/figures/fig4_latency_decomposition.png": { | ||
@@ -108,7 +108,7 @@ | |||||
| 108 | "runs/raw/E1_core.jsonl" | 108 | "runs/raw/E1_core.jsonl" | ||
| 109 | ], | 109 | ], | ||
| 110 | "script": "src/wp30/analysis/analyze.py", | 110 | "script": "src/wp30/analysis/analyze.py", | ||
| 111 | − "generated_utc": "2026-10-30T08:41:13.663220Z", | 111 | + "generated_utc": "2026-10-30T09:02:40.620070Z", | ||
| 112 | "note": "" | 112 | "note": "" | ||
| 113 | }, | 113 | }, | ||
| 114 | "results/figures/fig5_handoffs.pdf": { | 114 | "results/figures/fig5_handoffs.pdf": { | ||
@@ -116,7 +116,7 @@ | |||||
| 116 | "runs/raw/E4_core.jsonl" | 116 | "runs/raw/E4_core.jsonl" | ||
| 117 | ], | 117 | ], | ||
| 118 | "script": "src/wp30/analysis/analyze.py", | 118 | "script": "src/wp30/analysis/analyze.py", | ||
| 119 | − "generated_utc": "2026-10-30T08:41:18.022870Z", | 119 | + "generated_utc": "2026-10-30T09:02:45.615826Z", | ||
| 120 | "note": "means over documents; chains defined in configs/chains*.yaml" | 120 | "note": "means over documents; chains defined in configs/chains*.yaml" | ||
| 121 | }, | 121 | }, | ||
| 122 | "results/figures/fig5_handoffs.png": { | 122 | "results/figures/fig5_handoffs.png": { | ||
@@ -124,7 +124,7 @@ | |||||
| 124 | "runs/raw/E4_core.jsonl" | 124 | "runs/raw/E4_core.jsonl" | ||
| 125 | ], | 125 | ], | ||
| 126 | "script": "src/wp30/analysis/analyze.py", | 126 | "script": "src/wp30/analysis/analyze.py", | ||
| 127 | − "generated_utc": "2026-10-30T08:41:18.082563Z", | 127 | + "generated_utc": "2026-10-30T09:02:45.675774Z", | ||
| 128 | "note": "means over documents; chains defined in configs/chains*.yaml" | 128 | "note": "means over documents; chains defined in configs/chains*.yaml" | ||
| 129 | }, | 129 | }, | ||
| 130 | "results/tables/E4_core_handoffs.csv": { | 130 | "results/tables/E4_core_handoffs.csv": { | ||
@@ -132,7 +132,7 @@ | |||||
| 132 | "runs/raw/E4_core.jsonl" | 132 | "runs/raw/E4_core.jsonl" | ||
| 133 | ], | 133 | ], | ||
| 134 | "script": "src/wp30/analysis/analyze.py", | 134 | "script": "src/wp30/analysis/analyze.py", | ||
| 135 | − "generated_utc": "2026-10-30T08:41:18.179059Z", | 135 | + "generated_utc": "2026-10-30T09:02:45.778336Z", | ||
| 136 | "note": "repeated handoffs" | 136 | "note": "repeated handoffs" | ||
| 137 | }, | 137 | }, | ||
| 138 | "results/tables/E4_core_h4_primary_tests.csv": { | 138 | "results/tables/E4_core_h4_primary_tests.csv": { | ||
@@ -140,7 +140,7 @@ | |||||
| 140 | "runs/raw/E4_core.jsonl" | 140 | "runs/raw/E4_core.jsonl" | ||
| 141 | ], | 141 | ], | ||
| 142 | "script": "src/wp30/analysis/analyze.py", | 142 | "script": "src/wp30/analysis/analyze.py", | ||
| 143 | − "generated_utc": "2026-10-30T08:41:18.190264Z", | 143 | + "generated_utc": "2026-10-30T09:02:45.791552Z", | ||
| 144 | "note": "registered primary comparisons, paired, cluster bootstrap, Holm" | 144 | "note": "registered primary comparisons, paired, cluster bootstrap, Holm" | ||
| 145 | }, | 145 | }, | ||
| 146 | "results/tables/E5_core_fanin.csv": { | 146 | "results/tables/E5_core_fanin.csv": { | ||
@@ -148,7 +148,7 @@ | |||||
| 148 | "runs/raw/E5_core.jsonl" | 148 | "runs/raw/E5_core.jsonl" | ||
| 149 | ], | 149 | ], | ||
| 150 | "script": "src/wp30/analysis/analyze.py", | 150 | "script": "src/wp30/analysis/analyze.py", | ||
| 151 | − "generated_utc": "2026-10-30T08:41:18.294190Z", | 151 | + "generated_utc": "2026-10-30T09:02:45.898241Z", | ||
| 152 | "note": "fan-in by method and question type" | 152 | "note": "fan-in by method and question type" | ||
| 153 | }, | 153 | }, | ||
| 154 | "results/figures/fig6_breakeven.pdf": { | 154 | "results/figures/fig6_breakeven.pdf": { | ||
@@ -156,7 +156,7 @@ | |||||
| 156 | "runs/raw/E6_core.jsonl" | 156 | "runs/raw/E6_core.jsonl" | ||
| 157 | ], | 157 | ], | ||
| 158 | "script": "src/wp30/analysis/analyze.py", | 158 | "script": "src/wp30/analysis/analyze.py", | ||
| 159 | − "generated_utc": "2026-10-30T08:41:18.383673Z", | 159 | + "generated_utc": "2026-10-30T09:02:45.992041Z", | ||
| 160 | "note": "left = measured (medians, p25-p75); right = analytic model from measured bandwidth and prefill slope; dashed 10 Gb/s is hypothetical" | 160 | "note": "left = measured (medians, p25-p75); right = analytic model from measured bandwidth and prefill slope; dashed 10 Gb/s is hypothetical" | ||
| 161 | }, | 161 | }, | ||
| 162 | "results/figures/fig6_breakeven.png": { | 162 | "results/figures/fig6_breakeven.png": { | ||
@@ -164,7 +164,7 @@ | |||||
| 164 | "runs/raw/E6_core.jsonl" | 164 | "runs/raw/E6_core.jsonl" | ||
| 165 | ], | 165 | ], | ||
| 166 | "script": "src/wp30/analysis/analyze.py", | 166 | "script": "src/wp30/analysis/analyze.py", | ||
| 167 | − "generated_utc": "2026-10-30T08:41:18.456745Z", | 167 | + "generated_utc": "2026-10-30T09:02:46.065845Z", | ||
| 168 | "note": "left = measured (medians, p25-p75); right = analytic model from measured bandwidth and prefill slope; dashed 10 Gb/s is hypothetical" | 168 | "note": "left = measured (medians, p25-p75); right = analytic model from measured bandwidth and prefill slope; dashed 10 Gb/s is hypothetical" | ||
| 169 | }, | 169 | }, | ||
| 170 | "results/tables/E6_network.csv": { | 170 | "results/tables/E6_network.csv": { | ||
@@ -172,7 +172,7 @@ | |||||
| 172 | "runs/raw/E6_core.jsonl" | 172 | "runs/raw/E6_core.jsonl" | ||
| 173 | ], | 173 | ], | ||
| 174 | "script": "src/wp30/analysis/analyze.py", | 174 | "script": "src/wp30/analysis/analyze.py", | ||
| 175 | − "generated_utc": "2026-10-30T08:41:18.462390Z", | 175 | + "generated_utc": "2026-10-30T09:02:46.070991Z", | ||
| 176 | "note": "two-node transfer measurements" | 176 | "note": "two-node transfer measurements" | ||
| 177 | }, | 177 | }, | ||
| 178 | "results/tables/ED_hosted.csv": { | 178 | "results/tables/ED_hosted.csv": { | ||
@@ -180,9 +180,51 @@ | |||||
| 180 | "runs/raw/ED_hosted_core.jsonl" | 180 | "runs/raw/ED_hosted_core.jsonl" | ||
| 181 | ], | 181 | ], | ||
| 182 | "script": "src/wp30/analysis/analyze.py", | 182 | "script": "src/wp30/analysis/analyze.py", | ||
| 183 | − "generated_utc": "2026-10-30T08:41:18.529003Z", | 183 | + "generated_utc": "2026-10-30T09:02:46.140839Z", | ||
| 184 | "note": "hosted receivers; cost estimated from provider price tables" | 184 | "note": "hosted receivers; cost estimated from provider price tables" | ||
| 185 | }, | 185 | }, | ||
| 186 | + "results/tables/E3_core_transfer.csv": { | ||||
| 187 | + "sources": [ | ||||
| 188 | + "runs/raw/E3_core.jsonl", | ||||
| 189 | + "runs/raw/E3_core_sender.jsonl" | ||||
| 190 | + ], | ||||
| 191 | + "script": "src/wp30/analysis/analyze.py", | ||||
| 192 | + "generated_utc": "2026-10-30T09:02:46.269308Z", | ||||
| 193 | + "note": "same-family mapper transfer vs text handoffs and references" | ||||
| 194 | + }, | ||||
| 195 | + "results/tables/E3_core_primary_tests.csv": { | ||||
| 196 | + "sources": [ | ||||
| 197 | + "runs/raw/E3_core.jsonl", | ||||
| 198 | + "runs/raw/E3_core_sender.jsonl" | ||||
| 199 | + ], | ||||
| 200 | + "script": "src/wp30/analysis/analyze.py", | ||||
| 201 | + "generated_utc": "2026-10-30T09:02:46.347449Z", | ||||
| 202 | + "note": "registered primary comparisons, paired, cluster bootstrap, Holm" | ||||
| 203 | + }, | ||||
| 204 | + "results/tables/E7_qtypes.csv": { | ||||
| 205 | + "sources": [ | ||||
| 206 | + "runs/raw/E7_qtypes.jsonl" | ||||
| 207 | + ], | ||||
| 208 | + "script": "src/wp30/analysis/analyze.py", | ||||
| 209 | + "generated_utc": "2026-10-30T09:02:46.532910Z", | ||||
| 210 | + "note": "EXPLORATORY: question types 6-15 on synthetic test documents" | ||||
| 211 | + }, | ||||
| 212 | + "results/figures/figA_qtypes_exploratory.pdf": { | ||||
| 213 | + "sources": [ | ||||
| 214 | + "runs/raw/E7_qtypes.jsonl" | ||||
| 215 | + ], | ||||
| 216 | + "script": "src/wp30/analysis/analyze.py", | ||||
| 217 | + "generated_utc": "2026-10-30T09:02:46.612438Z", | ||||
| 218 | + "note": "exploratory extension A-004" | ||||
| 219 | + }, | ||||
| 220 | + "results/figures/figA_qtypes_exploratory.png": { | ||||
| 221 | + "sources": [ | ||||
| 222 | + "runs/raw/E7_qtypes.jsonl" | ||||
| 223 | + ], | ||||
| 224 | + "script": "src/wp30/analysis/analyze.py", | ||||
| 225 | + "generated_utc": "2026-10-30T09:02:46.668848Z", | ||||
| 226 | + "note": "exploratory extension A-004" | ||||
| 227 | + }, | ||||
| 186 | "results/figures/fig7_router.pdf": { | 228 | "results/figures/fig7_router.pdf": { | ||
| 187 | "sources": [ | 229 | "sources": [ | ||
| 188 | "results/tables/E8_router_summary.csv", | 230 | "results/tables/E8_router_summary.csv", | ||
@@ -190,7 +232,7 @@ | |||||
| 190 | "runs/checkpoints/router_policy.json" | 232 | "runs/checkpoints/router_policy.json" | ||
| 191 | ], | 233 | ], | ||
| 192 | "script": "src/wp30/analysis/analyze.py", | 234 | "script": "src/wp30/analysis/analyze.py", | ||
| 193 | − "generated_utc": "2026-10-30T08:41:18.669288Z", | 235 | + "generated_utc": "2026-10-30T09:02:46.768937Z", | ||
| 194 | "note": "oracle = infeasible upper reference" | 236 | "note": "oracle = infeasible upper reference" | ||
| 195 | }, | 237 | }, | ||
| 196 | "results/figures/fig7_router.png": { | 238 | "results/figures/fig7_router.png": { | ||
@@ -200,7 +242,7 @@ | |||||
| 200 | "runs/checkpoints/router_policy.json" | 242 | "runs/checkpoints/router_policy.json" | ||
| 201 | ], | 243 | ], | ||
| 202 | "script": "src/wp30/analysis/analyze.py", | 244 | "script": "src/wp30/analysis/analyze.py", | ||
| 203 | − "generated_utc": "2026-10-30T08:41:18.747352Z", | 245 | + "generated_utc": "2026-10-30T09:02:46.853258Z", | ||
| 204 | "note": "oracle = infeasible upper reference" | 246 | "note": "oracle = infeasible upper reference" | ||
| 205 | } | 247 | } | ||
| 206 | } | 248 | } | ||
| \ No newline at end of file | |||||
modified
results/tables/E1_core_by_question_type.csv
+3 −3
@@ -30,9 +30,9 @@ A6|budget_tokens=512|max_fetch_bytes=2000|max_lookups=6|blind,single_fact,0.2125 | |||||
| 30 | A7|codec=zstd,exact_copy,0.99375,0.97796875,1.0,160 | 30 | A7|codec=zstd,exact_copy,0.99375,0.97796875,1.0,160 | ||
| 31 | A7|codec=zstd,multi_hop,0.975,0.9375,1.0,80 | 31 | A7|codec=zstd,multi_hop,0.975,0.9375,1.0,80 | ||
| 32 | A7|codec=zstd,single_fact,0.99375,0.98125,1.0,160 | 32 | A7|codec=zstd,single_fact,0.99375,0.98125,1.0,160 | ||
| 33 | −A8|budget_tokens=512|blind,exact_copy,0.05,0.025,0.0875,160 | 33 | +A8|budget_tokens=512|blind,exact_copy,0.1,0.05,0.1625,160 | ||
| 34 | −A8|budget_tokens=512|blind,multi_hop,0.0125,0.0,0.0375,80 | 34 | +A8|budget_tokens=512|blind,multi_hop,0.05,0.0125,0.1,80 | ||
| 35 | −A8|budget_tokens=512|blind,single_fact,0.1125,0.06875,0.15953124999999985,160 | 35 | +A8|budget_tokens=512|blind,single_fact,0.24375,0.1625,0.3375,160 | ||
| 36 | B0,exact_copy,0.99375,0.97796875,1.0,160 | 36 | B0,exact_copy,0.99375,0.97796875,1.0,160 | ||
| 37 | B0,multi_hop,0.975,0.9375,1.0,80 | 37 | B0,multi_hop,0.975,0.9375,1.0,80 | ||
| 38 | B0,single_fact,0.99375,0.98125,1.0,160 | 38 | B0,single_fact,0.99375,0.98125,1.0,160 | ||
modified
results/tables/E1_core_failures.csv
+1 −1
@@ -7,7 +7,7 @@ A4,0,0,880 | |||||
| 7 | A5,0,0,880 | 7 | A5,0,0,880 | ||
| 8 | A6,0,0,640 | 8 | A6,0,0,640 | ||
| 9 | A7,0,0,640 | 9 | A7,0,0,640 | ||
| 10 | −A8,240,0,400 | 10 | +A8,240,0,640 | ||
| 11 | B0,0,0,640 | 11 | B0,0,0,640 | ||
| 12 | B1,0,0,640 | 12 | B1,0,0,640 | ||
| 13 | B2,0,29,1251 | 13 | B2,0,29,1251 | ||
modified
results/tables/E1_core_primary.csv
+2 −2
@@ -8,7 +8,7 @@ exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A4|budget_tokens=512 | |||||
| 8 | exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A5|budget_tokens=512|blind,A5,blind,40,40,0.0,0.0,0.0,1.0,1245.5,,8283.329896512441,1541.5113129420206,8283.329896512441,11.789770913310349,6766525528.0,6766525528.0,0.0 | 8 | exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A5|budget_tokens=512|blind,A5,blind,40,40,0.0,0.0,0.0,1.0,1245.5,,8283.329896512441,1541.5113129420206,8283.329896512441,11.789770913310349,6766525528.0,6766525528.0,0.0 | ||
| 9 | exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A6|budget_tokens=512|max_fetch_bytes=2000|max_lookups=6|blind,A6,blind,40,40,0.15,0.05,0.275,1.0,280.0,,1566.0093335900456,1565.9761460265145,1566.0093335900456,11.801395448856056,6751091800.0,6751091800.0,938.575 | 9 | exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A6|budget_tokens=512|max_fetch_bytes=2000|max_lookups=6|blind,A6,blind,40,40,0.15,0.05,0.275,1.0,280.0,,1566.0093335900456,1565.9761460265145,1566.0093335900456,11.801395448856056,6751091800.0,6751091800.0,938.575 | ||
| 10 | exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A7|codec=zstd,A7,none,40,40,0.95,0.875,1.0,1.0,1304.5,,1995.8721044822596,1995.802208490204,1995.8721044822596,12.078374973498285,7123374444.0,7123374444.0,0.0 | 10 | exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A7|codec=zstd,A7,none,40,40,0.95,0.875,1.0,1.0,1304.5,,1995.8721044822596,1995.802208490204,1995.8721044822596,12.078374973498285,7123374444.0,7123374444.0,0.0 | ||
| 11 | −exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A8|budget_tokens=512|blind,A8,blind,40,40,0.0,0.0,0.0,0.0,,,0.0,0.0,0.0,,,,0.0 | 11 | +exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A8|budget_tokens=512|blind,A8,blind,40,40,0.0,0.0,0.0,1.0,2397.5,,1807.6947699883021,1636.3695199252106,1807.6947699883021,13.637624972034246,6901300956.0,6901300956.0,0.0 | ||
| 12 | exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,B0,B0,blind,40,40,0.95,0.875,1.0,1.0,209433294.0,209420288.0,2161.7709995480254,1476.9027290167287,2161.7709995480254,51.358687051106244,7181169892.0,7181169892.0,0.0 | 12 | exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,B0,B0,blind,40,40,0.95,0.875,1.0,1.0,209433294.0,209420288.0,2161.7709995480254,1476.9027290167287,2161.7709995480254,51.358687051106244,7181169892.0,7181169892.0,0.0 | ||
| 13 | exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,B1,B1,blind,40,40,0.95,0.875,1.0,1.0,209433322.0,209420288.0,2183.0639789113775,1496.8550420016982,2183.0639789113775,49.82733295764774,7180973284.0,7180973284.0,0.0 | 13 | exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,B1,B1,blind,40,40,0.95,0.875,1.0,1.0,209433322.0,209420288.0,2183.0639789113775,1496.8550420016982,2183.0639789113775,49.82733295764774,7180973284.0,7180973284.0,0.0 | ||
| 14 | exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,B2|bits=4,B2,blind,40,40,1.0,1.0,1.0,1.0,58926513.0,58899456.0,2119.401790900156,1451.94422948407,2119.401790900156,42.334249475970864,6889044124.0,7112184172.0,0.0 | 14 | exec,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,B2|bits=4,B2,blind,40,40,1.0,1.0,1.0,1.0,58926513.0,58899456.0,2119.401790900156,1451.94422948407,2119.401790900156,42.334249475970864,6889044124.0,7112184172.0,0.0 | ||
@@ -130,7 +130,7 @@ synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A5|budget_token | |||||
| 130 | synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A5|budget_tokens=512|blind,A5,blind,200,40,0.125,0.09,0.155,1.0,842.0,,18998.95481200656,344.1750835045241,4091.6990083642304,23.827395518310368,6734748760.0,6734748760.0,0.0 | 130 | synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A5|budget_tokens=512|blind,A5,blind,200,40,0.125,0.09,0.155,1.0,842.0,,18998.95481200656,344.1750835045241,4091.6990083642304,23.827395518310368,6734748760.0,6734748760.0,0.0 | ||
| 131 | synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A6|budget_tokens=512|max_fetch_bytes=2000|max_lookups=6|blind,A6,blind,200,40,0.175,0.135,0.215,1.0,707.0,,492.9024174925871,492.8722715121694,492.87830070825294,25.794854445848614,6933195852.0,6933195852.0,1367.27 | 131 | synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A6|budget_tokens=512|max_fetch_bytes=2000|max_lookups=6|blind,A6,blind,200,40,0.175,0.135,0.215,1.0,707.0,,492.9024174925871,492.8722715121694,492.87830070825294,25.794854445848614,6933195852.0,6933195852.0,1367.27 | ||
| 132 | synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A7|codec=zstd,A7,none,200,40,0.99,0.975,1.0,1.0,2726.0,,2857.3816449497826,2857.2952494141646,2857.312528521288,27.074062498286366,7576794164.0,7576794164.0,0.0 | 132 | synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A7|codec=zstd,A7,none,200,40,0.99,0.975,1.0,1.0,2726.0,,2857.3816449497826,2857.2952494141646,2857.312528521288,27.074062498286366,7576794164.0,7576794164.0,0.0 | ||
| 133 | −synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A8|budget_tokens=512|blind,A8,blind,200,40,0.0,0.0,0.0,0.0,,,0.0,0.0,0.0,,,,0.0 | 133 | +synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,A8|budget_tokens=512|blind,A8,blind,200,40,0.16,0.105,0.225,1.0,2583.0,,522.9626869549975,108.42154250713065,216.73387542832643,13.577207922935486,6915931970.0,6915931970.0,0.0 | ||
| 134 | synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,B0,B0,blind,200,40,0.99,0.975,1.0,1.0,651268854.0,651255808.0,3253.5364581272006,393.2519589434378,973.5832870006561,136.40537502942607,8593077474.0,8593077474.0,0.0 | 134 | synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,B0,B0,blind,200,40,0.99,0.975,1.0,1.0,651268854.0,651255808.0,3253.5364581272006,393.2519589434378,973.5832870006561,136.40537502942607,8593077474.0,8593077474.0,0.0 | ||
| 135 | synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,B1,B1,blind,200,40,0.99,0.975,1.0,1.0,651268882.0,651255808.0,3351.503874640912,505.0313540850766,1057.6043623499572,144.29387502605096,8593069282.0,8593069282.0,0.0 | 135 | synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,B1,B1,blind,200,40,0.99,0.975,1.0,1.0,651268882.0,651255808.0,3351.503874640912,505.0313540850766,1057.6043623499572,144.29387502605096,8593069282.0,8593069282.0,0.0 | ||
| 136 | synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,B2|bits=4,B2,blind,200,40,0.975,0.945,0.995,1.0,183192937.0,183165696.0,3111.2300624372438,335.1737505290657,882.7701129950583,106.28564545186237,7628087269.0,7628087269.0,0.0 | 136 | synthetic,Llama-3.2-3B-Instruct-bf16->Llama-3.2-3B-Instruct-bf16,B2|bits=4,B2,blind,200,40,0.975,0.945,0.995,1.0,183192937.0,183165696.0,3111.2300624372438,335.1737505290657,882.7701129950583,106.28564545186237,7628087269.0,7628087269.0,0.0 | ||
added
results/tables/E3_core_primary_tests.csv
+3 −0
@@ -0,0 +1,3 @@ | |||||
| 1 | +a,b,diff,lo,hi,p,n,n_clusters,mean_a,mean_b,rel_change,name,hypothesis,p_holm | ||||
| 2 | +C2,A4|512|blind,-0.037037037037037035,-0.11574074074074074,0.041666666666666664,0.368,216,27,0.24537037037037038,0.2824074074074074,-0.13114754098360656,C2 mapper vs A4 summary (synthetic),H10,0.368 | ||||
| 3 | +C2,A1,-0.75,-0.7777777777777778,-0.7222222222222222,0.0,216,27,0.24537037037037038,0.9953703703703703,-0.7534883720930232,C2 mapper vs A1 receiver full context (synthetic),H10,0.0 | ||||
added
results/tables/E3_core_transfer.csv
+17 −0
@@ -0,0 +1,17 @@ | |||||
| 1 | +family,condition,quality,lo,hi,n,n_docs,payload_bytes_median,cold_ms_p50,warm_ms_p50,unknown_share,q_exec,q_exact_copy,q_multi_hop,q_negation,q_single_fact | ||||
| 2 | +exec,A1,1.0,1.0,1.0,40,40,7817.0,1912.1752704959363,1912.1676459326409,0.0,1.0,,,, | ||||
| 3 | +exec,A4|512|blind,0.0,0.0,0.0,40,40,938.0,2819.653937360272,1025.8437710581347,0.275,0.0,,,, | ||||
| 4 | +exec,A5|512|blind,0.0,0.0,0.0,40,40,936.0,4190.043583512306,215.29910451499745,0.85,0.0,,,, | ||||
| 5 | +exec,C0,0.0,0.0,0.0,40,40,69762785.0,4331.171416444704,3979.971270891838,0.0,0.0,,,, | ||||
| 6 | +exec,C2,0.0,0.0,0.0,40,40,69762886.0,2318.2933544158004,1098.6273545422591,0.0,0.0,,,, | ||||
| 7 | +exec,C3|1/4|aware,0.0,0.0,0.0,40,40,17452748.5,2254.7777493600734,939.0185408992693,0.0,0.0,,,, | ||||
| 8 | +exec,sender alone A1,0.0,0.0,0.0,40,40,7817.0,909.1166876023635,909.1070625581779,0.0,0.0,,,, | ||||
| 9 | +exec,sender alone B0,0.0,0.0,0.0,40,40,54260105.0,957.9157516709529,621.4223124552518,0.0,0.0,,,, | ||||
| 10 | +synthetic,A1,0.9953703703703703,0.9861111111111112,1.0,216,27,7555.0,890.8028129953891,890.7906879903749,0.0,,1.0,0.9814814814814815,1.0,1.0 | ||||
| 11 | +synthetic,A4|512|blind,0.2824074074074074,0.2175925925925926,0.35185185185185186,216,27,1263.0,4260.842582501937,250.3902085009031,0.6435185185185185,,0.2037037037037037,0.1111111111111111,0.7222222222222222,0.09259259259259259 | ||||
| 12 | +synthetic,A5|512|blind,0.11574074074074074,0.06944444444444445,0.16203703703703703,216,27,596.0,4229.795791499782,166.88918694853783,0.7314814814814815,,0.09259259259259259,0.07407407407407407,0.2222222222222222,0.07407407407407407 | ||||
| 13 | +synthetic,C0,0.0,0.0,0.0,216,27,77688549.0,1025.9398333728313,632.5281668687239,0.0,,0.0,0.0,0.0,0.0 | ||||
| 14 | +synthetic,C2,0.24537037037037038,0.2175925925925926,0.27314814814814814,216,27,77688650.0,1424.9500405276194,85.57341661071405,0.6898148148148148,,0.037037037037037035,0.0,0.9259259259259259,0.018518518518518517 | ||||
| 15 | +synthetic,C3|1/4|aware,0.25,0.22685185185185186,0.27314814814814814,216,27,19443415.0,1492.1443954808637,56.23004218796268,0.6851851851851852,,0.018518518518518517,0.0,0.9444444444444444,0.037037037037037035 | ||||
| 16 | +synthetic,sender alone A1,0.9583333333333334,0.9350694444444447,0.9814814814814815,216,27,7555.0,448.41252046171576,448.40406253933907,0.0,,0.9814814814814815,0.9444444444444444,0.9259259259259259,0.9814814814814815 | ||||
| 17 | +synthetic,sender alone B0,0.9537037037037037,0.9259259259259259,0.9814814814814815,216,27,60424589.0,478.51893870392814,104.15885498514399,0.0,,0.9814814814814815,0.9444444444444444,0.9074074074074074,0.9814814814814815 | ||||
added
results/tables/E7_qtypes.csv
+99 −0
@@ -0,0 +1,99 @@ | |||||
| 1 | +condition,qtype,pair,quality,lo,hi,n,abstained | ||||
| 2 | +A1,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 | ||||
| 3 | +A1,aggregation,Qwen2.5-3B,0.0,0.0,0.0,16,0.0 | ||||
| 4 | +A1,multi_hop,Llama-3.2-3B,1.0,1.0,1.0,15,0.0 | ||||
| 5 | +A1,multi_hop,Qwen2.5-3B,1.0,1.0,1.0,16,0.0 | ||||
| 6 | +A1,negation,Llama-3.2-3B,0.9,0.8,1.0,30,0.0 | ||||
| 7 | +A1,negation,Qwen2.5-3B,1.0,1.0,1.0,32,0.0 | ||||
| 8 | +A1,revision,Llama-3.2-3B,0.7333333333333333,0.5825000000000004,0.8841666666666659,30,0.0 | ||||
| 9 | +A1,revision,Qwen2.5-3B,0.9375,0.84375,1.0,32,0.0 | ||||
| 10 | +A1,temporal,Llama-3.2-3B,0.4666666666666667,0.2,0.7333333333333333,15,0.0 | ||||
| 11 | +A1,temporal,Qwen2.5-3B,0.4375,0.21718750000000064,0.6875,16,0.0 | ||||
| 12 | +A1,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0 | ||||
| 13 | +A1,unanswerable,Qwen2.5-3B,0.96875,0.90625,1.0,32,0.96875 | ||||
| 14 | +A1,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 | ||||
| 15 | +A1,unit_conversion,Qwen2.5-3B,0.5,0.25,0.75,16,0.0 | ||||
| 16 | +A3|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 | ||||
| 17 | +A3|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.0 | ||||
| 18 | +A3|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.0,0.0,0.0,15,1.0 | ||||
| 19 | +A3|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.0,0.0,0.0,15,1.0 | ||||
| 20 | +A3|budget_tokens=512|blind,negation,Llama-3.2-3B,0.8333333333333334,0.7,0.9333333333333333,30,0.0 | ||||
| 21 | +A3|budget_tokens=512|blind,negation,Qwen2.5-3B,0.9333333333333333,0.8333333333333334,1.0,30,0.0 | ||||
| 22 | +A3|budget_tokens=512|blind,revision,Llama-3.2-3B,0.7333333333333333,0.5666666666666667,0.8666666666666667,30,0.0 | ||||
| 23 | +A3|budget_tokens=512|blind,revision,Qwen2.5-3B,0.7333333333333333,0.5333333333333333,0.9,30,0.0 | ||||
| 24 | +A3|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.5333333333333333,0.3333333333333333,0.7333333333333333,15,0.0 | ||||
| 25 | +A3|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.3333333333333333,0.13333333333333333,0.5333333333333333,15,0.0 | ||||
| 26 | +A3|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0 | ||||
| 27 | +A3|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,30,1.0 | ||||
| 28 | +A3|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.9333333333333333 | ||||
| 29 | +A3|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.0,0.0,0.0,15,0.6666666666666666 | ||||
| 30 | +A4|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 | ||||
| 31 | +A4|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.4 | ||||
| 32 | +A4|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.3333333333333333,0.06666666666666667,0.6,15,0.5333333333333333 | ||||
| 33 | +A4|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.4375,0.1875,0.6875,16,0.5 | ||||
| 34 | +A4|budget_tokens=512|blind,negation,Llama-3.2-3B,1.0,1.0,1.0,30,0.0 | ||||
| 35 | +A4|budget_tokens=512|blind,negation,Qwen2.5-3B,0.71875,0.53125,0.875,32,0.1875 | ||||
| 36 | +A4|budget_tokens=512|blind,revision,Llama-3.2-3B,0.16666666666666666,0.03333333333333333,0.3333333333333333,30,0.3333333333333333 | ||||
| 37 | +A4|budget_tokens=512|blind,revision,Qwen2.5-3B,0.25806451612903225,0.12903225806451613,0.4266935483870965,31,0.5161290322580645 | ||||
| 38 | +A4|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.2,0.0,0.4,15,0.3333333333333333 | ||||
| 39 | +A4|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.3333333333333333,0.13333333333333333,0.5333333333333333,15,0.26666666666666666 | ||||
| 40 | +A4|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0 | ||||
| 41 | +A4|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,30,1.0 | ||||
| 42 | +A4|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.6666666666666666 | ||||
| 43 | +A4|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.4,0.13333333333333333,0.6666666666666666,15,0.6 | ||||
| 44 | +A5|budget_tokens=512|blind,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.3333333333333333 | ||||
| 45 | +A5|budget_tokens=512|blind,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.4666666666666667 | ||||
| 46 | +A5|budget_tokens=512|blind,multi_hop,Llama-3.2-3B,0.13333333333333333,0.0,0.3333333333333333,15,0.8666666666666667 | ||||
| 47 | +A5|budget_tokens=512|blind,multi_hop,Qwen2.5-3B,0.2,0.06666666666666667,0.4,15,0.7333333333333333 | ||||
| 48 | +A5|budget_tokens=512|blind,negation,Llama-3.2-3B,0.5,0.36666666666666664,0.6333333333333333,30,0.3 | ||||
| 49 | +A5|budget_tokens=512|blind,negation,Qwen2.5-3B,0.5333333333333333,0.36666666666666664,0.7,30,0.43333333333333335 | ||||
| 50 | +A5|budget_tokens=512|blind,revision,Llama-3.2-3B,0.13333333333333333,0.0,0.3,30,0.7666666666666667 | ||||
| 51 | +A5|budget_tokens=512|blind,revision,Qwen2.5-3B,0.1,0.0,0.26666666666666666,30,0.8333333333333334 | ||||
| 52 | +A5|budget_tokens=512|blind,temporal,Llama-3.2-3B,0.13333333333333333,0.0,0.3333333333333333,15,0.8666666666666667 | ||||
| 53 | +A5|budget_tokens=512|blind,temporal,Qwen2.5-3B,0.0,0.0,0.0,15,0.8666666666666667 | ||||
| 54 | +A5|budget_tokens=512|blind,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0 | ||||
| 55 | +A5|budget_tokens=512|blind,unanswerable,Qwen2.5-3B,1.0,1.0,1.0,30,1.0 | ||||
| 56 | +A5|budget_tokens=512|blind,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.7333333333333333 | ||||
| 57 | +A5|budget_tokens=512|blind,unit_conversion,Qwen2.5-3B,0.2,0.0,0.4,15,0.3333333333333333 | ||||
| 58 | +B2|bits=8,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 | ||||
| 59 | +B2|bits=8,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.0 | ||||
| 60 | +B2|bits=8,multi_hop,Llama-3.2-3B,1.0,1.0,1.0,15,0.0 | ||||
| 61 | +B2|bits=8,multi_hop,Qwen2.5-3B,1.0,1.0,1.0,15,0.0 | ||||
| 62 | +B2|bits=8,negation,Llama-3.2-3B,0.9,0.8,1.0,30,0.0 | ||||
| 63 | +B2|bits=8,negation,Qwen2.5-3B,1.0,1.0,1.0,30,0.0 | ||||
| 64 | +B2|bits=8,revision,Llama-3.2-3B,0.7333333333333333,0.5825000000000004,0.8841666666666659,30,0.0 | ||||
| 65 | +B2|bits=8,revision,Qwen2.5-3B,0.9333333333333333,0.8333333333333334,1.0,30,0.0 | ||||
| 66 | +B2|bits=8,temporal,Llama-3.2-3B,0.4666666666666667,0.2,0.7333333333333333,15,0.0 | ||||
| 67 | +B2|bits=8,temporal,Qwen2.5-3B,0.4,0.13333333333333333,0.6666666666666666,15,0.0 | ||||
| 68 | +B2|bits=8,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0 | ||||
| 69 | +B2|bits=8,unanswerable,Qwen2.5-3B,0.9666666666666667,0.9,1.0,30,0.9666666666666667 | ||||
| 70 | +B2|bits=8,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 | ||||
| 71 | +B2|bits=8,unit_conversion,Qwen2.5-3B,0.6,0.3333333333333333,0.8666666666666667,15,0.0 | ||||
| 72 | +B3|keep_frac=0.25,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 | ||||
| 73 | +B3|keep_frac=0.25,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.0 | ||||
| 74 | +B3|keep_frac=0.25,multi_hop,Llama-3.2-3B,0.2,0.0,0.4,15,0.3333333333333333 | ||||
| 75 | +B3|keep_frac=0.25,multi_hop,Qwen2.5-3B,0.2,0.0,0.4,15,0.13333333333333333 | ||||
| 76 | +B3|keep_frac=0.25,negation,Llama-3.2-3B,0.7333333333333333,0.6,0.8333333333333334,30,0.0 | ||||
| 77 | +B3|keep_frac=0.25,negation,Qwen2.5-3B,0.8666666666666667,0.7333333333333333,0.9666666666666667,30,0.0 | ||||
| 78 | +B3|keep_frac=0.25,revision,Llama-3.2-3B,0.06666666666666667,0.0,0.16666666666666666,30,0.6666666666666666 | ||||
| 79 | +B3|keep_frac=0.25,revision,Qwen2.5-3B,0.0,0.0,0.0,30,0.0 | ||||
| 80 | +B3|keep_frac=0.25,temporal,Llama-3.2-3B,0.5333333333333333,0.26666666666666666,0.7333333333333333,15,0.0 | ||||
| 81 | +B3|keep_frac=0.25,temporal,Qwen2.5-3B,0.5333333333333333,0.26666666666666666,0.7333333333333333,15,0.0 | ||||
| 82 | +B3|keep_frac=0.25,unanswerable,Llama-3.2-3B,0.9333333333333333,0.8,1.0,30,0.9333333333333333 | ||||
| 83 | +B3|keep_frac=0.25,unanswerable,Qwen2.5-3B,0.9666666666666667,0.9,1.0,30,0.9666666666666667 | ||||
| 84 | +B3|keep_frac=0.25,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.7333333333333333 | ||||
| 85 | +B3|keep_frac=0.25,unit_conversion,Qwen2.5-3B,0.06666666666666667,0.0,0.2,15,0.0 | ||||
| 86 | +B5|keep_frac=0.25|aware,aggregation,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 | ||||
| 87 | +B5|keep_frac=0.25|aware,aggregation,Qwen2.5-3B,0.0,0.0,0.0,15,0.0 | ||||
| 88 | +B5|keep_frac=0.25|aware,multi_hop,Llama-3.2-3B,1.0,1.0,1.0,15,0.0 | ||||
| 89 | +B5|keep_frac=0.25|aware,multi_hop,Qwen2.5-3B,0.5333333333333333,0.26666666666666666,0.8,15,0.0 | ||||
| 90 | +B5|keep_frac=0.25|aware,negation,Llama-3.2-3B,0.8666666666666667,0.7666666666666667,0.9666666666666667,30,0.0 | ||||
| 91 | +B5|keep_frac=0.25|aware,negation,Qwen2.5-3B,1.0,1.0,1.0,30,0.0 | ||||
| 92 | +B5|keep_frac=0.25|aware,revision,Llama-3.2-3B,0.7333333333333333,0.6158333333333337,0.8666666666666667,30,0.0 | ||||
| 93 | +B5|keep_frac=0.25|aware,revision,Qwen2.5-3B,0.3,0.16666666666666666,0.43333333333333335,30,0.0 | ||||
| 94 | +B5|keep_frac=0.25|aware,temporal,Llama-3.2-3B,0.6,0.4,0.8,15,0.0 | ||||
| 95 | +B5|keep_frac=0.25|aware,temporal,Qwen2.5-3B,0.5333333333333333,0.26666666666666666,0.8,15,0.0 | ||||
| 96 | +B5|keep_frac=0.25|aware,unanswerable,Llama-3.2-3B,1.0,1.0,1.0,30,1.0 | ||||
| 97 | +B5|keep_frac=0.25|aware,unanswerable,Qwen2.5-3B,0.9666666666666667,0.9,1.0,30,0.9666666666666667 | ||||
| 98 | +B5|keep_frac=0.25|aware,unit_conversion,Llama-3.2-3B,0.0,0.0,0.0,15,0.0 | ||||
| 99 | +B5|keep_frac=0.25|aware,unit_conversion,Qwen2.5-3B,0.4666666666666667,0.2,0.7333333333333333,15,0.0 | ||||
modified
runs/raw/E1_core.jsonl
+34 −0
@@ -13484,3 +13484,37 @@ | |||||
| 13484 | {"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "249692f5ce8f1d73a6740de8", "attempt_id": "fb51489c21df", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3c3d27f14e6841b389fd503bcab914bda201aa79", "timestamp_utc": "2026-10-30T08:42:10.966933Z", "task_family": "exec", "document_id": "exec_00204_2000", "document_hash": "0aaf1a095160aa21c4485db3b1b171e86d9d41b904ac2a2ea102cc13df19cd22", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00204_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2882882882882883, "origin_tokens": 1776, "compressed_tokens": 571, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1776, "source_tokens_receiver": 1776, "query_tokens": 677, "message_tokens": 571, "message_utf8_bytes": 2437, "cache_tensor_bytes": null, "serialized_payload_bytes": 2437, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 172.06537502352148, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.74979193042964, "ttft_ms": 13.673665933310986, "decode_ms": 1547.018332988955}, "total_wall_ms": 1642.768583027646, "ttft_ms": 13.673665933310986, "input_usage": {"receiver_prompt_tokens": 677}, "output_usage": {"output_tokens": 115}, "provider_cached_usage": null, "process_memory": 9840558080, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6902742760, "mlx_cache_bytes": 617350120, "process_rss_bytes": 9840558080}, "peak_prefill_memory": null, "peak_decode_memory": 6902742760, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n total_price = row[1] * row[2]\n discount = total_price * row[3] / 100\n total_price -= discount\n tax = total_price * row[4] / 100\n total_price -= tax\n if row[5]:\n total_price -= total_price * row[6] / 100\n net_revenue.append([row[0], total_price])\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00204_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "b7473c9b2ca34ae2ad15a2f4942179c42282d616b583ea59fc5a9c9279c50de4", "output_hash": "3a25256cab462a5e1cb32c2c1077112ba4d284b7186c251430927e593e89675f"} | 13484 | {"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "249692f5ce8f1d73a6740de8", "attempt_id": "fb51489c21df", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3c3d27f14e6841b389fd503bcab914bda201aa79", "timestamp_utc": "2026-10-30T08:42:10.966933Z", "task_family": "exec", "document_id": "exec_00204_2000", "document_hash": "0aaf1a095160aa21c4485db3b1b171e86d9d41b904ac2a2ea102cc13df19cd22", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00204_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2882882882882883, "origin_tokens": 1776, "compressed_tokens": 571, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1776, "source_tokens_receiver": 1776, "query_tokens": 677, "message_tokens": 571, "message_utf8_bytes": 2437, "cache_tensor_bytes": null, "serialized_payload_bytes": 2437, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 172.06537502352148, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.74979193042964, "ttft_ms": 13.673665933310986, "decode_ms": 1547.018332988955}, "total_wall_ms": 1642.768583027646, "ttft_ms": 13.673665933310986, "input_usage": {"receiver_prompt_tokens": 677}, "output_usage": {"output_tokens": 115}, "provider_cached_usage": null, "process_memory": 9840558080, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6902742760, "mlx_cache_bytes": 617350120, "process_rss_bytes": 9840558080}, "peak_prefill_memory": null, "peak_decode_memory": 6902742760, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n total_price = row[1] * row[2]\n discount = total_price * row[3] / 100\n total_price -= discount\n tax = total_price * row[4] / 100\n total_price -= tax\n if row[5]:\n total_price -= total_price * row[6] / 100\n net_revenue.append([row[0], total_price])\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00204_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "b7473c9b2ca34ae2ad15a2f4942179c42282d616b583ea59fc5a9c9279c50de4", "output_hash": "3a25256cab462a5e1cb32c2c1077112ba4d284b7186c251430927e593e89675f"} | ||
| 13485 | {"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "fcffc2920a4be15dd5396dac", "attempt_id": "09276543f2ae", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3c3d27f14e6841b389fd503bcab914bda201aa79", "timestamp_utc": "2026-10-30T08:42:12.835175Z", "task_family": "exec", "document_id": "exec_00205_2000", "document_hash": "c82f149ddbd77be80773d103825621fb187f06bb60d4fd570c73c2b3cd898ecb", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00205_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2879640044994376, "origin_tokens": 1778, "compressed_tokens": 578, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1778, "source_tokens_receiver": 1778, "query_tokens": 684, "message_tokens": 578, "message_utf8_bytes": 2327, "cache_tensor_bytes": null, "serialized_payload_bytes": 2327, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.99858301319182, "decode_payload_ms": 0.0, "receiver_prefill_ms": 81.34491695091128, "ttft_ms": 13.695165980607271, "decode_ms": 2792.174707981758}, "total_wall_ms": 2888.588665984571, "ttft_ms": 13.695165980607271, "input_usage": {"receiver_prompt_tokens": 684}, "output_usage": {"output_tokens": 203}, "provider_cached_usage": null, "process_memory": 9840607232, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6907199236, "mlx_cache_bytes": 621921284, "process_rss_bytes": 9840607232}, "peak_prefill_memory": null, "peak_decode_memory": 6907199236, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[3]:\n if row[2] < 2:\n net_revenue.append({\n 'revenue': row[0],\n 'unit_price': row[1],\n 'days': row[2],\n 'return': row[3],\n 'result': row[4]\n })\n else:\n discount = row[1] * row[4] / 100\n unit_price = row[1] - discount\n tax = unit_price * row[2] * row[4] / 100\n net_revenue.append({\n 'revenue': row[0] * row[4],\n 'unit_price': unit_price,\n 'days': row[2],\n 'return': row[3],\n 'result': row[4],\n 'tax': tax\n })\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00205_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "423a49a4accbb29d70f7498f3ebf493e0c0481ec1422a6875554275ac1745e8a", "output_hash": "a307b446a77dfaebc3d534f4333abf69203f14ae8a6eabb7ccb4cb5570ca3b0e"} | 13485 | {"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "fcffc2920a4be15dd5396dac", "attempt_id": "09276543f2ae", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3c3d27f14e6841b389fd503bcab914bda201aa79", "timestamp_utc": "2026-10-30T08:42:12.835175Z", "task_family": "exec", "document_id": "exec_00205_2000", "document_hash": "c82f149ddbd77be80773d103825621fb187f06bb60d4fd570c73c2b3cd898ecb", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00205_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2879640044994376, "origin_tokens": 1778, "compressed_tokens": 578, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1778, "source_tokens_receiver": 1778, "query_tokens": 684, "message_tokens": 578, "message_utf8_bytes": 2327, "cache_tensor_bytes": null, "serialized_payload_bytes": 2327, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.99858301319182, "decode_payload_ms": 0.0, "receiver_prefill_ms": 81.34491695091128, "ttft_ms": 13.695165980607271, "decode_ms": 2792.174707981758}, "total_wall_ms": 2888.588665984571, "ttft_ms": 13.695165980607271, "input_usage": {"receiver_prompt_tokens": 684}, "output_usage": {"output_tokens": 203}, "provider_cached_usage": null, "process_memory": 9840607232, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6907199236, "mlx_cache_bytes": 621921284, "process_rss_bytes": 9840607232}, "peak_prefill_memory": null, "peak_decode_memory": 6907199236, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[3]:\n if row[2] < 2:\n net_revenue.append({\n 'revenue': row[0],\n 'unit_price': row[1],\n 'days': row[2],\n 'return': row[3],\n 'result': row[4]\n })\n else:\n discount = row[1] * row[4] / 100\n unit_price = row[1] - discount\n tax = unit_price * row[2] * row[4] / 100\n net_revenue.append({\n 'revenue': row[0] * row[4],\n 'unit_price': unit_price,\n 'days': row[2],\n 'return': row[3],\n 'result': row[4],\n 'tax': tax\n })\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00205_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "423a49a4accbb29d70f7498f3ebf493e0c0481ec1422a6875554275ac1745e8a", "output_hash": "a307b446a77dfaebc3d534f4333abf69203f14ae8a6eabb7ccb4cb5570ca3b0e"} | ||
| 13486 | {"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "8ba5551920d488af0ada35b4", "attempt_id": "a9fa9ad4ba9f", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3c3d27f14e6841b389fd503bcab914bda201aa79", "timestamp_utc": "2026-10-30T08:42:15.948869Z", "task_family": "exec", "document_id": "exec_00206_2000", "document_hash": "e391c9d757bc03677c664222c3ff3f1bbab881d7f566c0072b184e0b90503339", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00206_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2925714285714286, "origin_tokens": 1750, "compressed_tokens": 584, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1750, "source_tokens_receiver": 1750, "query_tokens": 690, "message_tokens": 584, "message_utf8_bytes": 2394, "cache_tensor_bytes": null, "serialized_payload_bytes": 2394, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.46862507797778, "decode_payload_ms": 0.0, "receiver_prefill_ms": 81.24412503093481, "ttft_ms": 13.564125052653253, "decode_ms": 1656.4636249095201}, "total_wall_ms": 1752.6230829535052, "ttft_ms": 13.564125052653253, "input_usage": {"receiver_prompt_tokens": 690}, "output_usage": {"output_tokens": 123}, "provider_cached_usage": null, "process_memory": 9840607232, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6910574364, "mlx_cache_bytes": 625394716, "process_rss_bytes": 9840607232}, "peak_prefill_memory": null, "peak_decode_memory": 6910574364, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = 0\n for row in rows:\n if row[3] < 4:\n continue\n if row[3] < 4 and row[4] < 0:\n continue\n if row[4] < 0:\n net_revenue += row[2] * (1 - row[4])\n else:\n net_revenue += row[2] * (1 - row[4]) * (1 + row[4] / 100)\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00206_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "91535a844a7c8262e6f7eb2869f5f0ce078ee999e468a110e4336dff99f138d3", "output_hash": "1cd5a1a531494e01135bfff7b53b5e0ed8bcf8a7082638991c6dc75c3514efd9"} | 13486 | {"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "8ba5551920d488af0ada35b4", "attempt_id": "a9fa9ad4ba9f", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3c3d27f14e6841b389fd503bcab914bda201aa79", "timestamp_utc": "2026-10-30T08:42:15.948869Z", "task_family": "exec", "document_id": "exec_00206_2000", "document_hash": "e391c9d757bc03677c664222c3ff3f1bbab881d7f566c0072b184e0b90503339", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00206_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2925714285714286, "origin_tokens": 1750, "compressed_tokens": 584, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1750, "source_tokens_receiver": 1750, "query_tokens": 690, "message_tokens": 584, "message_utf8_bytes": 2394, "cache_tensor_bytes": null, "serialized_payload_bytes": 2394, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.46862507797778, "decode_payload_ms": 0.0, "receiver_prefill_ms": 81.24412503093481, "ttft_ms": 13.564125052653253, "decode_ms": 1656.4636249095201}, "total_wall_ms": 1752.6230829535052, "ttft_ms": 13.564125052653253, "input_usage": {"receiver_prompt_tokens": 690}, "output_usage": {"output_tokens": 123}, "provider_cached_usage": null, "process_memory": 9840607232, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6910574364, "mlx_cache_bytes": 625394716, "process_rss_bytes": 9840607232}, "peak_prefill_memory": null, "peak_decode_memory": 6910574364, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = 0\n for row in rows:\n if row[3] < 4:\n continue\n if row[3] < 4 and row[4] < 0:\n continue\n if row[4] < 0:\n net_revenue += row[2] * (1 - row[4])\n else:\n net_revenue += row[2] * (1 - row[4]) * (1 + row[4] / 100)\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00206_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "91535a844a7c8262e6f7eb2869f5f0ce078ee999e468a110e4336dff99f138d3", "output_hash": "1cd5a1a531494e01135bfff7b53b5e0ed8bcf8a7082638991c6dc75c3514efd9"} | ||
| 13487 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "497ed6b8d988e4442ffb66df", "attempt_id": "533b7a059565", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3c3d27f14e6841b389fd503bcab914bda201aa79", "timestamp_utc": "2026-10-30T08:42:17.925526Z", "task_family": "exec", "document_id": "exec_00207_2000", "document_hash": "7d024469b5d38776a899aa870f5af7e146953a19f1adc1d91f634cda62aa4c14", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00207_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2910744741330301, "origin_tokens": 1759, "compressed_tokens": 561, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1759, "source_tokens_receiver": 1759, "query_tokens": 667, "message_tokens": 561, "message_utf8_bytes": 2495, "cache_tensor_bytes": null, "serialized_payload_bytes": 2495, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.45999998319894, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.29549999628216, "ttft_ms": 13.490624958649278, "decode_ms": 2664.541167090647}, "total_wall_ms": 2759.726958000101, "ttft_ms": 13.490624958649278, "input_usage": {"receiver_prompt_tokens": 667}, "output_usage": {"output_tokens": 195}, "provider_cached_usage": null, "process_memory": 9840623616, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6896844480, "mlx_cache_bytes": 611288000, "process_rss_bytes": 9840623616}, "peak_prefill_memory": null, "peak_decode_memory": 6896844480, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[5]:\n total_price = row[1] * row[2]\n discount = total_price * row[3] / 100\n total_price -= discount\n tax = total_price * row[4] / 100\n total_price += tax\n net_revenue.append({\n 'total_price': total_price,\n 'return': round(row[5] * row[1] * row[2], 2),\n 'result': round(total_price - row[5] * row[1] * row[2], 2)\n })\n else:\n net_revenue.append({\n 'total_price': row[1] * row[2],\n 'return': 0,\n 'result': row[1] * row[2]\n })\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00207_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "84f1ebed1a2b476e559b7b3f4cefd24b66211ebb600bf98a61fe13802fda636d", "output_hash": "2012fa6a70529683271c63eaa3a666e852f485fee479f79842debd9754e4217b"} | ||||
| 13488 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "99cc94326e4146b1b7165a7f", "attempt_id": "0128ef8e8046", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:20.921162Z", "task_family": "exec", "document_id": "exec_00208_2000", "document_hash": "8528b3fafcf58b9daa88ffa9d7b9b25fb6e225f9bb767849744c05510a5ace37", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00208_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2882882882882883, "origin_tokens": 1776, "compressed_tokens": 565, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1776, "source_tokens_receiver": 1776, "query_tokens": 672, "message_tokens": 565, "message_utf8_bytes": 2310, "cache_tensor_bytes": null, "serialized_payload_bytes": 2310, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 178.83129208348691, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.4949589073658, "ttft_ms": 14.29595798254013, "decode_ms": 2122.73499998264}, "total_wall_ms": 2219.3354580085725, "ttft_ms": 14.29595798254013, "input_usage": {"receiver_prompt_tokens": 672}, "output_usage": {"output_tokens": 155}, "provider_cached_usage": null, "process_memory": 9841491968, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6899744468, "mlx_cache_bytes": 614269908, "process_rss_bytes": 9841491968}, "peak_prefill_memory": null, "peak_decode_memory": 6899744468, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[4] and row[5] < 2:\n net_revenue.append({\n 'revenue': row[0],\n 'price_pct': row[6],\n 'days': row[7],\n 'unit_price': row[8],\n 'discount': row[9],\n 'tax': row[10]\n })\n net_revenue = [item for item in net_revenue if item['tax']]\n net_revenue = [round(item['revenue'] * (1 - item['price_pct'] / 100), 2) for item in net_revenue]\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00208_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "2d7a71bcd3ed95b9325ffe04e15279b6712cc9af7b11b359e09dce855a593b75", "output_hash": "b729cb4436c688b07cc5ec42ef27ac402004366a29dced7d5e60a665f5d75288"} | ||||
| 13489 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "47ac740aefc4ffb314dc60b5", "attempt_id": "122b4245320a", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:23.371658Z", "task_family": "exec", "document_id": "exec_00209_2000", "document_hash": "27a0f4409b3611173adf72865b52c6beef014bce0282fd0498be50deeed192c9", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00209_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2887760857304005, "origin_tokens": 1773, "compressed_tokens": 567, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1773, "source_tokens_receiver": 1773, "query_tokens": 673, "message_tokens": 567, "message_utf8_bytes": 2354, "cache_tensor_bytes": null, "serialized_payload_bytes": 2354, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 172.0899579813704, "decode_payload_ms": 0.0, "receiver_prefill_ms": 78.64887500181794, "ttft_ms": 13.771625002846122, "decode_ms": 1872.2131249960512}, "total_wall_ms": 1965.9930830821395, "ttft_ms": 13.771625002846122, "input_usage": {"receiver_prompt_tokens": 673}, "output_usage": {"output_tokens": 138}, "provider_cached_usage": null, "process_memory": 9841573888, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6899924696, "mlx_cache_bytes": 614466520, "process_rss_bytes": 9841573888}, "peak_prefill_memory": null, "peak_decode_memory": 6899924696, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[6]:\n total_price = row[1] * (1 - row[2] / 100)\n return_value = round(total_price - row[3], 1)\n net_revenue.append([row[0], row[1], row[2], row[3], return_value])\n else:\n total_price = row[1] * (1 - row[2] / 100)\n net_revenue.append([row[0], row[1], row[2], total_price])\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00209_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "8f90480bac1467f3586aff0573ad1b393158f21bf18e94a44644a0c0fc8ced87", "output_hash": "a77d36f6c9a6ab5ec42f96d81c94c4dd992271ae35660b7d5be83758b7f06359"} | ||||
| 13490 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "d82d6a231e7151db60b574f1", "attempt_id": "c68ca0fcdf4b", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:25.562074Z", "task_family": "exec", "document_id": "exec_00210_2000", "document_hash": "ffa82307f9b4986f3fc473b8b7df1e4f8f0e1c25d3b6e36cdcf4114a763427c5", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00210_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.28780213603147836, "origin_tokens": 1779, "compressed_tokens": 568, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1779, "source_tokens_receiver": 1779, "query_tokens": 674, "message_tokens": 568, "message_utf8_bytes": 2387, "cache_tensor_bytes": null, "serialized_payload_bytes": 2387, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.6894160490483, "decode_payload_ms": 0.0, "receiver_prefill_ms": 81.06112503446639, "ttft_ms": 13.499583001248538, "decode_ms": 1498.353625065647}, "total_wall_ms": 1594.2203749436885, "ttft_ms": 13.499583001248538, "input_usage": {"receiver_prompt_tokens": 674}, "output_usage": {"output_tokens": 111}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6901300956, "mlx_cache_bytes": 615859164, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6901300956, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n revenue, _price, _pct, days, return_value, _ = row\n if return_value < 3:\n continue\n discount = _price * _pct\n total_price = revenue - discount\n tax = total_price * 0.1\n total_price -= tax\n net_revenue.append((total_price, round(total_price, 2), days))\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00210_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "ValueError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "821e4a6998224f423dcccc86ab4ab3d5e99127736fd3c81de48b3bc8d4fadcfb", "output_hash": "fec0ade85339e7234c83e2d60ed984e03cea8b1b1e7b9315d99fc2cee40c0438"} | ||||
| 13491 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "dcb3968f5db34c4e2aa3c111", "attempt_id": "07a5d536c955", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:27.381046Z", "task_family": "exec", "document_id": "exec_00211_2000", "document_hash": "95faa51b0302db54273e9b4469b6eae92c251463238fdad494d437435c293548", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00211_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2910744741330301, "origin_tokens": 1759, "compressed_tokens": 568, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1759, "source_tokens_receiver": 1759, "query_tokens": 674, "message_tokens": 568, "message_utf8_bytes": 2477, "cache_tensor_bytes": null, "serialized_payload_bytes": 2477, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 172.00216697528958, "decode_payload_ms": 0.0, "receiver_prefill_ms": 81.26708306372166, "ttft_ms": 13.799750013276935, "decode_ms": 1116.2925839889795}, "total_wall_ms": 1212.6838329713792, "ttft_ms": 13.799750013276935, "input_usage": {"receiver_prompt_tokens": 674}, "output_usage": {"output_tokens": 83}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6901300956, "mlx_cache_bytes": 491078620, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6901300956, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = 0\n for row in rows:\n if row[3] == 'true':\n total_price = row[1] * (1 - row[2])\n return_value = total_price - (total_price * row[2] / 100)\n net_revenue += return_value\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00211_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "9280a89505b62652effd544d5c0c14ee514c76f6bacf88c57ce1f1d0dafffb71", "output_hash": "ba5fd7c67f0a4047766abb3cfc1671e5cafb478059f113c0da65846fe5adadc7"} | ||||
| 13492 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "8ba6edb7f2072035e1c2c36e", "attempt_id": "b005aa22a6df", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:28.817740Z", "task_family": "exec", "document_id": "exec_00212_2000", "document_hash": "2be5dc8496484a99fc11a00be474f4f7f139a8f491a2ebf3fff0db8e7804dd82", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00212_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2871564778463264, "origin_tokens": 1783, "compressed_tokens": 556, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1783, "source_tokens_receiver": 1783, "query_tokens": 662, "message_tokens": 556, "message_utf8_bytes": 2343, "cache_tensor_bytes": null, "serialized_payload_bytes": 2343, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.5197080047801, "decode_payload_ms": 0.0, "receiver_prefill_ms": 79.55270796082914, "ttft_ms": 13.581416918896139, "decode_ms": 1009.1769170248881}, "total_wall_ms": 1103.6108750849962, "ttft_ms": 13.581416918896139, "input_usage": {"receiver_prompt_tokens": 662}, "output_usage": {"output_tokens": 75}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6893846188, "mlx_cache_bytes": 483427244, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6893846188, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n total = 0\n for row in rows:\n if row[3] < 3:\n continue\n if row[2] < 0:\n continue\n total += (row[1] - row[2] * row[3]) * row[4]\n return total\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00212_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "146b533021f85a62b0b442f1c7858256c7643942d82bfc14a289b82840eb5aeb", "output_hash": "ac4ca858b5d72cf4f6aa0b7074365d712cf9f23a155262f7635c1c70db54f274"} | ||||
| 13493 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "a286376f5d8f3683b8910219", "attempt_id": "82987a3e4a9f", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:30.128352Z", "task_family": "exec", "document_id": "exec_00213_2000", "document_hash": "3c65c42333c25f1f3f90331747f08fafce263c8faa11a56d3a68bc58b2217327", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00213_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2964678633468442, "origin_tokens": 1727, "compressed_tokens": 569, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1727, "source_tokens_receiver": 1727, "query_tokens": 675, "message_tokens": 569, "message_utf8_bytes": 2403, "cache_tensor_bytes": null, "serialized_payload_bytes": 2403, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 157.29450003709644, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.9257080545649, "ttft_ms": 13.848165981471539, "decode_ms": 1144.9057089630514}, "total_wall_ms": 1240.9864579094574, "ttft_ms": 13.848165981471539, "input_usage": {"receiver_prompt_tokens": 675}, "output_usage": {"output_tokens": 85}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6901481184, "mlx_cache_bytes": 491275232, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6901481184, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n total = 0\n for row in rows:\n if row[6]:\n discount = row[2] * row[1]\n unit_price = row[1] - discount\n total += (row[0] * unit_price) / 100\n else:\n total += row[0]\n return round(total, 2)\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00213_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "f361625f3a060bb6cdacf62f72406152cc5b5504fc7bc0f3a2644719509fb2b7", "output_hash": "23840152737e815af651da4c9d5ec3607ebd257a12507429e459aea11c69253d"} | ||||
| 13494 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "85977675d46a496ecac43d97", "attempt_id": "0e3eaaa37da5", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:31.593322Z", "task_family": "exec", "document_id": "exec_00214_2000", "document_hash": "ac0a3ee69ec7da626616f6ffb5e443a4f26a3d7857caaa20f1d4ebfa5355d28b", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00214_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.28619340413638905, "origin_tokens": 1789, "compressed_tokens": 568, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1789, "source_tokens_receiver": 1789, "query_tokens": 674, "message_tokens": 568, "message_utf8_bytes": 2306, "cache_tensor_bytes": null, "serialized_payload_bytes": 2306, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.5933750383556, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.9099159669131, "ttft_ms": 13.659916934557259, "decode_ms": 1532.3253750102594}, "total_wall_ms": 1628.2140410039574, "ttft_ms": 13.659916934557259, "input_usage": {"receiver_prompt_tokens": 674}, "output_usage": {"output_tokens": 114}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6901300956, "mlx_cache_bytes": 615859164, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6901300956, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = 0\n for row in rows:\n if row[4] and row[5] < 3:\n continue\n price = row[2] * row[6]\n discount = row[2] * row[5] * row[7]\n tax = row[2] * row[5] * row[8]\n net_revenue += (price - discount) * row[5] * row[9]\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00214_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "9c6c34da6100909bea9860b9179341485319e5085f2831b74824af13dd019a52", "output_hash": "300e8ce2def91ca8951552171089751d878ffdd0e00dc6239a68ea55075d2bc9"} | ||||
| 13495 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "2d911474369be5b41b33aef5", "attempt_id": "6a2ce52eb411", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:33.446214Z", "task_family": "exec", "document_id": "exec_00215_2000", "document_hash": "a4094c70181359b8442ead384be9497e3fb0c46a4d9266cce07ca77bd9e26e2b", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00215_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.287478944413251, "origin_tokens": 1781, "compressed_tokens": 561, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1781, "source_tokens_receiver": 1781, "query_tokens": 668, "message_tokens": 561, "message_utf8_bytes": 2403, "cache_tensor_bytes": null, "serialized_payload_bytes": 2403, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.91362497396767, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.15595900360495, "ttft_ms": 13.627458014525473, "decode_ms": 2354.5712920604274}, "total_wall_ms": 2449.697708012536, "ttft_ms": 13.627458014525473, "input_usage": {"receiver_prompt_tokens": 668}, "output_usage": {"output_tokens": 175}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6897925828, "mlx_cache_bytes": 612385732, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6897925828, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row['return'] and row['units'] < 2:\n net_revenue.append({\n 'total_price': row['unit_price'] * row['units'] * (1 - row['pct']),\n 'return': row['return'],\n 'result': 0\n })\n else:\n net_revenue.append({\n 'total_price': row['unit_price'] * row['units'] * (1 - row['pct']),\n 'return': row['return'],\n 'result': row['total_price'] - (row['total_price'] * row['pct'] / 100) - (row['total_price'] * row['return'] / 100)\n })\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00215_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "550eb2ac15e1eb50b2b2b9937efd6fa3d2b485341b94e9e77bf64822c2ed7d46", "output_hash": "51badb0e56b2cb19c11b35f0067ab4371dd143d59c7ae31fdb406452b80ac71a"} | ||||
| 13496 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "aa88aea946fad20df71b7aac", "attempt_id": "b3d16af58634", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:36.121956Z", "task_family": "exec", "document_id": "exec_00216_2000", "document_hash": "1a218de32e6fe4bbe97927ea16330bda3b598ed66c1517824d94ff90f944e802", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00216_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2828729281767956, "origin_tokens": 1810, "compressed_tokens": 579, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1810, "source_tokens_receiver": 1810, "query_tokens": 685, "message_tokens": 579, "message_utf8_bytes": 2312, "cache_tensor_bytes": null, "serialized_payload_bytes": 2312, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.94533301517367, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.97695803735405, "ttft_ms": 13.529750052839518, "decode_ms": 1600.7696250453591}, "total_wall_ms": 1696.6219580499455, "ttft_ms": 13.529750052839518, "input_usage": {"receiver_prompt_tokens": 685}, "output_usage": {"output_tokens": 118}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6907379464, "mlx_cache_bytes": 622117896, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6907379464, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[6] < 6:\n continue\n revenue = row[1] * (1 - row[2] / 100)\n discount = row[1] * (row[2] / 100)\n tax = discount * (row[3] / 100)\n total_price = revenue - discount + tax\n net_revenue.append([row[0], round(total_price, 2)])\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00216_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "c0975c8f7571f34d1cd9d9a5e9b39e146019671e5a0ceea765fadf22de7b7e52", "output_hash": "6b592111c07261f4244bb1861f24f16a3ddd48d58940fd5eae1ef046189c9814"} | ||||
| 13497 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "025ee727b8d8620ea5b2d4b9", "attempt_id": "3df770973171", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:38.044691Z", "task_family": "exec", "document_id": "exec_00217_2000", "document_hash": "1223c7391b1da1aeae2d242c0e41de4639218cfe0c90ba9065f1cce42f0eedbd", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00217_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2924043403769275, "origin_tokens": 1751, "compressed_tokens": 584, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1751, "source_tokens_receiver": 1751, "query_tokens": 690, "message_tokens": 584, "message_utf8_bytes": 2401, "cache_tensor_bytes": null, "serialized_payload_bytes": 2401, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 172.5227499846369, "decode_payload_ms": 0.0, "receiver_prefill_ms": 81.4392080064863, "ttft_ms": 13.853417010977864, "decode_ms": 1307.8046248992905}, "total_wall_ms": 1404.468125081621, "ttft_ms": 13.853417010977864, "input_usage": {"receiver_prompt_tokens": 690}, "output_usage": {"output_tokens": 95}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6910574364, "mlx_cache_bytes": 625394716, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6910574364, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = 0\n for row in rows:\n if row[3] < 3:\n continue\n unit_price = row[1] * row[2]\n discount = row[1] * row[2] * row[3] / 100\n unit_price -= discount\n net_revenue += unit_price * row[0]\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00217_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "e5a6392a10268f82d9d3a60785b26607d413381787370acac61e71a77477cc95", "output_hash": "83275e808bae2f86c4674652db83a7d9f389b61b74f1b782aee075b4d93ab137"} | ||||
| 13498 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "641cc033dab82953821bbb76", "attempt_id": "8f5ed33ad031", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:39.675998Z", "task_family": "exec", "document_id": "exec_00218_2000", "document_hash": "971d94c4e7b4ae02142ffb6eb92af9d2cc0f6e4d52e94a64c5132b510197ce8a", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00218_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.29207073588134624, "origin_tokens": 1753, "compressed_tokens": 583, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1753, "source_tokens_receiver": 1753, "query_tokens": 689, "message_tokens": 583, "message_utf8_bytes": 2420, "cache_tensor_bytes": null, "serialized_payload_bytes": 2420, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 172.8406249312684, "decode_payload_ms": 0.0, "receiver_prefill_ms": 81.28320798277855, "ttft_ms": 13.886500033549964, "decode_ms": 1415.9948750166222}, "total_wall_ms": 1512.5300409272313, "ttft_ms": 13.886500033549964, "input_usage": {"receiver_prompt_tokens": 689}, "output_usage": {"output_tokens": 104}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6909198104, "mlx_cache_bytes": 624002072, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6909198104, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[3] < 5:\n continue\n unit_price = row[1]\n discount = row[2] / 100\n unit_price -= unit_price * discount\n tax = row[3] * unit_price * 0.1\n unit_price += tax\n net_revenue.append([row[0], unit_price])\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00218_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "40fd90e0803fcd3dc7e6e9f16768505edd868ec77860e7401b7b40a5c636dfeb", "output_hash": "dc5f865bffdcf95525727957a79355a89943889bcff8c6105beca968ccc6f898"} | ||||
| 13499 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "61b672f1008820066df718a5", "attempt_id": "41a11868919e", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:41.414048Z", "task_family": "exec", "document_id": "exec_00219_2000", "document_hash": "390edd848add1a4cf011660e8ac9a6a6aa381c4508a8f68635f3217ce20979cb", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00219_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.28845070422535213, "origin_tokens": 1775, "compressed_tokens": 560, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1775, "source_tokens_receiver": 1775, "query_tokens": 666, "message_tokens": 560, "message_utf8_bytes": 2454, "cache_tensor_bytes": null, "serialized_payload_bytes": 2454, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.64866696111858, "decode_payload_ms": 0.0, "receiver_prefill_ms": 79.80712503194809, "ttft_ms": 13.666957966051996, "decode_ms": 1776.1701670242473}, "total_wall_ms": 1870.9364999085665, "ttft_ms": 13.666957966051996, "input_usage": {"receiver_prompt_tokens": 666}, "output_usage": {"output_tokens": 132}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6896664252, "mlx_cache_bytes": 611091388, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6896664252, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n price = row[1]\n pct = row[2]\n days = row[6]\n return_value = row[0]\n if row[7]:\n return_value -= (row[0] * (price * pct / 100))\n if days > 3:\n return_value += (row[0] * (price * pct / 100))\n return_value = round(return_value, 2)\n net_revenue.append([return_value, price, pct, days])\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00219_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "448502755d7d30f09cc1db22fa65c54b8d68b3d1ca696958f2784d140711b43a", "output_hash": "3704ba8ff1c241b6f339b6da39a9133512684c0ed7a6392969bff0c3567288ae"} | ||||
| 13500 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "4f68a6cf073fe103dd9f84f5", "attempt_id": "6e6a1b30f8fe", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:43.508640Z", "task_family": "exec", "document_id": "exec_00220_2000", "document_hash": "31eca9e7afc70ac5a795edc1d1529c531e0d028342802a61f4496c013e13bf45", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00220_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.28942905596382135, "origin_tokens": 1769, "compressed_tokens": 567, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1769, "source_tokens_receiver": 1769, "query_tokens": 673, "message_tokens": 567, "message_utf8_bytes": 2254, "cache_tensor_bytes": null, "serialized_payload_bytes": 2254, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.0438330192119, "decode_payload_ms": 0.0, "receiver_prefill_ms": 78.24433292262256, "ttft_ms": 13.686000020243227, "decode_ms": 1927.0280000055209}, "total_wall_ms": 2020.3162910183892, "ttft_ms": 13.686000020243227, "input_usage": {"receiver_prompt_tokens": 673}, "output_usage": {"output_tokens": 143}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6899924696, "mlx_cache_bytes": 614466520, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6899924696, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[4] < 5:\n continue\n if row[2] == False:\n net_revenue.append({\n 'revenue': row[0],\n 'unit_price': row[1],\n 'discount': row[3],\n 'tax': row[4] * row[1] * row[3] / 100,\n 'result': row[0] - row[1] * row[3] + row[4] * row[1] * row[3] / 100\n })\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00220_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "dd2978260f5277083d44090d80dfcde060d1d384818216b10a649b61e16517b2", "output_hash": "48282ee7e4c979b8bbe577ac34c6b8952a468155aa6239e60b2613fb39a67b91"} | ||||
| 13501 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "fb6c04376c11e55db2650bc2", "attempt_id": "a80bccf747b2", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:45.752660Z", "task_family": "exec", "document_id": "exec_00221_2000", "document_hash": "e228dfaa0d04173ee28b43513e5fc06914edf5d0c02b09bf0fc534531a583669", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00221_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2882882882882883, "origin_tokens": 1776, "compressed_tokens": 578, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1776, "source_tokens_receiver": 1776, "query_tokens": 684, "message_tokens": 578, "message_utf8_bytes": 2193, "cache_tensor_bytes": null, "serialized_payload_bytes": 2193, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.16729100234807, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.82966599613428, "ttft_ms": 13.605042011477053, "decode_ms": 1829.5709999511018}, "total_wall_ms": 1925.2815840300173, "ttft_ms": 13.605042011477053, "input_usage": {"receiver_prompt_tokens": 684}, "output_usage": {"output_tokens": 136}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6907199236, "mlx_cache_bytes": 621921284, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6907199236, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[6]:\n unit_price = row[1]\n discount = row[2] * unit_price\n revenue = row[0] - discount\n tax = (revenue * row[3]) / 100\n net_revenue.append([row[0], row[1], row[2], row[3], revenue, tax])\n else:\n net_revenue.append([row[0], row[1], row[2], row[3], row[0], 0])\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00221_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "a2fc1c748d6e2e0d5a56ebddd9feb88ac4ac2392df78a72169af1a5b27895e5f", "output_hash": "5bb18c586d96a78edb7fa0f5e34844ecab050a8c059ed1451dfe72b71df7b924"} | ||||
| 13502 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "978c018455a5a6a1ac63d92e", "attempt_id": "dd655a675016", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:47.902508Z", "task_family": "exec", "document_id": "exec_00222_2000", "document_hash": "040260f25a42680fcc038386368506adaa004bc2592e75e3fbb5903b54cc19ab", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00222_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2925714285714286, "origin_tokens": 1750, "compressed_tokens": 576, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1750, "source_tokens_receiver": 1750, "query_tokens": 682, "message_tokens": 576, "message_utf8_bytes": 2463, "cache_tensor_bytes": null, "serialized_payload_bytes": 2463, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.50287504773587, "decode_payload_ms": 0.0, "receiver_prefill_ms": 81.104209064506, "ttft_ms": 13.847957947291434, "decode_ms": 1279.2211669730023}, "total_wall_ms": 1375.4802499897778, "ttft_ms": 13.847957947291434, "input_usage": {"receiver_prompt_tokens": 682}, "output_usage": {"output_tokens": 95}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6905937660, "mlx_cache_bytes": 620626940, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6905937660, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = 0\n for row in rows:\n if row[4]:\n unit_price = row[2] * row[3]\n discount = row[2] * row[3] * row[5]\n net_price = unit_price - discount\n tax = net_price * row[6]\n net_revenue += net_price - tax\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00222_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "8630e417e7ecd69b053de649fe0d893eaf61fe5755895d35bc941e95fbb938a4", "output_hash": "c5f1d843bd0119adbb498f3fd117bcf9128b0fd48df692923193bb16fe0b71d1"} | ||||
| 13503 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "23c5ec0cdcbfc99eda4dc7e3", "attempt_id": "a747ca243bae", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:49.501771Z", "task_family": "exec", "document_id": "exec_00223_2000", "document_hash": "7eef13486249d89cf05bda367ea9dae2ce2479b2236c1917b2f067c49a981619", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00223_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2924043403769275, "origin_tokens": 1751, "compressed_tokens": 573, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1751, "source_tokens_receiver": 1751, "query_tokens": 679, "message_tokens": 573, "message_utf8_bytes": 2449, "cache_tensor_bytes": null, "serialized_payload_bytes": 2449, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 170.88595894165337, "decode_payload_ms": 0.0, "receiver_prefill_ms": 81.08937507495284, "ttft_ms": 13.747042044997215, "decode_ms": 1596.2386670289561}, "total_wall_ms": 1692.375999991782, "ttft_ms": 13.747042044997215, "input_usage": {"receiver_prompt_tokens": 679}, "output_usage": {"output_tokens": 118}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6904004336, "mlx_cache_bytes": 618644464, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6904004336, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[6] < 6:\n continue\n if row[1] == False:\n unit_price = row[2]\n discount = row[3]\n tax = row[4]\n total_price = unit_price - (unit_price * discount / 100)\n if tax:\n total_price += (total_price * tax / 100)\n net_revenue.append([row[0], total_price])\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00223_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "d9108e0320ff70f5c37a52c76dea886912e6b01db5cec54004934b350df6a14d", "output_hash": "d90b3d25ac4f1b03d55d5654d503a1d1dfedd7292281136fe593373206ffa9ae"} | ||||
| 13504 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "89b2a97529b835b026ce7bbf", "attempt_id": "72f91793eef7", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:51.419124Z", "task_family": "exec", "document_id": "exec_00224_2000", "document_hash": "3e642abb57841365b827053f2437a250dcb419539389820bac778d4596c064fd", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00224_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.29307384087006294, "origin_tokens": 1747, "compressed_tokens": 577, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1747, "source_tokens_receiver": 1747, "query_tokens": 683, "message_tokens": 577, "message_utf8_bytes": 2374, "cache_tensor_bytes": null, "serialized_payload_bytes": 2374, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.2289999704808, "decode_payload_ms": 0.0, "receiver_prefill_ms": 81.49324997793883, "ttft_ms": 13.581749983131886, "decode_ms": 1471.4275830192491}, "total_wall_ms": 1567.844209028408, "ttft_ms": 13.581749983131886, "input_usage": {"receiver_prompt_tokens": 683}, "output_usage": {"output_tokens": 109}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6906117888, "mlx_cache_bytes": 620823552, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6906117888, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[4]:\n discount = row[2] * row[3]\n unit_price = row[1] - discount\n tax = unit_price * 0.975\n net = unit_price - tax\n net_revenue.append([row[0], round(net, 2)])\n else:\n net_revenue.append([row[0], row[1]])\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00224_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "db05d12476288df3b0020d8a6596118cf7292828db88dc6a1882fd6b8645ffe7", "output_hash": "131920fb22087c603c8c03b2a7751d283f14e60fcd9a79713282bd09d1a31774"} | ||||
| 13505 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "dc39ee11a8978e148425de6c", "attempt_id": "d1916819032e", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:53.212700Z", "task_family": "exec", "document_id": "exec_00225_2000", "document_hash": "54918c16a4ae3706a87d182112fca2a39c82c77e3a60fd25a8b47b958f2a6b21", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00225_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.289920724801812, "origin_tokens": 1766, "compressed_tokens": 537, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1766, "source_tokens_receiver": 1766, "query_tokens": 643, "message_tokens": 537, "message_utf8_bytes": 2448, "cache_tensor_bytes": null, "serialized_payload_bytes": 2448, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 172.50633309595287, "decode_payload_ms": 0.0, "receiver_prefill_ms": 79.88912495784461, "ttft_ms": 13.595500029623508, "decode_ms": 1212.232832913287}, "total_wall_ms": 1307.0872919633985, "ttft_ms": 13.595500029623508, "input_usage": {"receiver_prompt_tokens": 643}, "output_usage": {"output_tokens": 90}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6882934368, "mlx_cache_bytes": 472204128, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6882934368, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = 0\n for row in rows:\n if row[4]:\n unit_price = row[2] * row[3]\n discount = row[2] * row[5]\n net_price = unit_price - discount\n tax = net_price * row[6]\n net_revenue += net_price - tax\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00225_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "9b093efb24dab82665d1b9d9382a2742c0b65505d5f65df1ad6d248f9feae2c5", "output_hash": "6923babd25cb138c1058e523556898398dd65be219eff452a62e2d031f57ac35"} | ||||
| 13506 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "139fca0831cb1fc7c2d67d90", "attempt_id": "a7ef4b9ab385", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:54.744710Z", "task_family": "exec", "document_id": "exec_00226_2000", "document_hash": "7a73067df93b17aba1a608bce48f7a6ff16d480af8d23713ec889405f279fc7d", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00226_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.29157175398633256, "origin_tokens": 1756, "compressed_tokens": 571, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1756, "source_tokens_receiver": 1756, "query_tokens": 677, "message_tokens": 571, "message_utf8_bytes": 2519, "cache_tensor_bytes": null, "serialized_payload_bytes": 2519, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.5362500399351, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.89508407283574, "ttft_ms": 13.631999958306551, "decode_ms": 955.9395830146968}, "total_wall_ms": 1051.8002919852734, "ttft_ms": 13.631999958306551, "input_usage": {"receiver_prompt_tokens": 677}, "output_usage": {"output_tokens": 71}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6902742760, "mlx_cache_bytes": 492569576, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6902742760, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n total_price = 0\n for row in rows:\n if row[3] < 3:\n continue\n unit_price = row[1] - (row[1] * row[2] / 100)\n total_price += unit_price\n return total_price\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00226_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "33e9e194cec43017dddec4d3c723228289ddf300af66722f35a8ec7d5601325f", "output_hash": "b2bf737e68b5ac320de4942978295fdbbc1379a4999808840bbfa66f9f634bd4"} | ||||
| 13507 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "7db6b526486c17d5a19e82d3", "attempt_id": "9b98f90bd0fb", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:56.004152Z", "task_family": "exec", "document_id": "exec_00227_2000", "document_hash": "6afe19e10d57be0082db797187045b004680fa22a13a56a7b8f7e2a51b40597d", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00227_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.287478944413251, "origin_tokens": 1781, "compressed_tokens": 565, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1781, "source_tokens_receiver": 1781, "query_tokens": 671, "message_tokens": 565, "message_utf8_bytes": 2310, "cache_tensor_bytes": null, "serialized_payload_bytes": 2310, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 156.63258300628513, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.28445905074477, "ttft_ms": 13.715166016481817, "decode_ms": 847.5787499919534}, "total_wall_ms": 942.8761670133099, "ttft_ms": 13.715166016481817, "input_usage": {"receiver_prompt_tokens": 671}, "output_usage": {"output_tokens": 63}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6899367632, "mlx_cache_bytes": 489096144, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6899367632, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n total = 0\n for r in rows:\n if r[3] < 5:\n continue\n total += round((r[1] * (1 - r[2] / 100)), 1)\n return total\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00227_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "bd4e07f1a616236f364ee7cefdf01757ae715ef75b4a935e47120e40a77f3e44", "output_hash": "2d220646ae2a26932c763cfcbe679fbd40555124568628628ac4940a1291719a"} | ||||
| 13508 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "417b799672db8ff9fdcbf02d", "attempt_id": "58df1c483870", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:57.153796Z", "task_family": "exec", "document_id": "exec_00228_2000", "document_hash": "08d5c40fa24cccbcccaac14d408c241ab602604a712b727f12f570eb56887a74", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00228_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.289920724801812, "origin_tokens": 1766, "compressed_tokens": 562, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1766, "source_tokens_receiver": 1766, "query_tokens": 668, "message_tokens": 562, "message_utf8_bytes": 2336, "cache_tensor_bytes": null, "serialized_payload_bytes": 2336, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 156.7782920319587, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.49608301371336, "ttft_ms": 13.607500004582107, "decode_ms": 1779.5917079783976}, "total_wall_ms": 1874.9484160216525, "ttft_ms": 13.607500004582107, "input_usage": {"receiver_prompt_tokens": 668}, "output_usage": {"output_tokens": 132}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6897925828, "mlx_cache_bytes": 612385732, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6897925828, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = 0\n for row in rows:\n if row[5]: # if item is returned\n units_returned = row[6]\n unit_price = row[2]\n discount = row[3]\n tax = row[4]\n net_revenue += (units_returned * unit_price) - (units_returned * unit_price * discount) + (units_returned * unit_price * discount * tax)\n else:\n net_revenue += row[2] - row[3]\n return round(net_revenue, 2)\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00228_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "18a815a8f3f527b23d509bf0e540b732df6b778d849e7c4e010741e191f73b6e", "output_hash": "0bbd37fea4e618dc818ff883dacee83793a671bd4ed31bf4a589b44f1cf15886"} | ||||
| 13509 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "9e32511fb4ecaf4d7e549ca5", "attempt_id": "ba4dff172423", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:42:59.252806Z", "task_family": "exec", "document_id": "exec_00229_2000", "document_hash": "b6a34443a51bb95703036fc2467d0c50ad248f99f3c11c0a35d6bc7fcf0ad481", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00229_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2895927601809955, "origin_tokens": 1768, "compressed_tokens": 563, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1768, "source_tokens_receiver": 1768, "query_tokens": 669, "message_tokens": 563, "message_utf8_bytes": 2273, "cache_tensor_bytes": null, "serialized_payload_bytes": 2273, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.56270798295736, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.14262502547354, "ttft_ms": 13.650209060870111, "decode_ms": 1420.1994170434773}, "total_wall_ms": 1515.2930830372497, "ttft_ms": 13.650209060870111, "input_usage": {"receiver_prompt_tokens": 669}, "output_usage": {"output_tokens": 105}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6898106056, "mlx_cache_bytes": 612582344, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6898106056, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n total = 0\n for row in rows:\n if row[3]:\n if row[2] < 6:\n total += row[1] * (1 - row[2] / 100)\n else:\n total += row[1] * (1 - row[2] / 100) * (1 - row[2] / 100)\n else:\n total += row[1]\n return total\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00229_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "cf570e5ce845b463ac0e0b7e0c1573c8e1c07c4fdc6cf595fdc96f2e58a1f6f7", "output_hash": "547c1d5080b69951b1883a0d35e019141280a12cb9eb042a5700e3e0b198dad3"} | ||||
| 13510 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "f3438227c2d36b4280e97600", "attempt_id": "447df2132d8e", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:43:00.992893Z", "task_family": "exec", "document_id": "exec_00230_2000", "document_hash": "cbb9dba4a75d2570390205536d928042eae0341c49e7dcc11a2cea988c175ba9", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00230_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2912400455062571, "origin_tokens": 1758, "compressed_tokens": 575, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1758, "source_tokens_receiver": 1758, "query_tokens": 681, "message_tokens": 575, "message_utf8_bytes": 2478, "cache_tensor_bytes": null, "serialized_payload_bytes": 2478, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 170.58512510266155, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.99929103627801, "ttft_ms": 13.587624998763204, "decode_ms": 1536.7103329626843}, "total_wall_ms": 1632.6072909869254, "ttft_ms": 13.587624998763204, "input_usage": {"receiver_prompt_tokens": 681}, "output_usage": {"output_tokens": 114}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6904561400, "mlx_cache_bytes": 619234296, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6904561400, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[3] < 5:\n continue\n revenue = row[0] * row[2] * (1 - row[2] * row[4])\n tax = revenue * row[4]\n net = revenue - tax\n net_revenue.append([row[0], row[1], row[2], row[3], row[4], row[5], net])\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00230_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "15bc4685c90e187e047fd226f3ae070a44f807d190800d87432b1247b39d777f", "output_hash": "67738af5a692c6bd432d014dec44dbcdb183eea0e169ea3999b80b02708e7a8f"} | ||||
| 13511 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "417f3037ddb170c5eba5514a", "attempt_id": "278967976363", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:43:02.850007Z", "task_family": "exec", "document_id": "exec_00231_2000", "document_hash": "b599ec4f6ea744baef17b36d05ab828f105675eeebf81a780befd49f389ec9d3", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00231_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.28942905596382135, "origin_tokens": 1769, "compressed_tokens": 560, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1769, "source_tokens_receiver": 1769, "query_tokens": 666, "message_tokens": 560, "message_utf8_bytes": 2367, "cache_tensor_bytes": null, "serialized_payload_bytes": 2367, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.30504094529897, "decode_payload_ms": 0.0, "receiver_prefill_ms": 79.69599997159094, "ttft_ms": 13.524957932531834, "decode_ms": 1954.5689160004258}, "total_wall_ms": 2049.1593750193715, "ttft_ms": 13.524957932531834, "input_usage": {"receiver_prompt_tokens": 666}, "output_usage": {"output_tokens": 145}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6896664252, "mlx_cache_bytes": 611091388, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6896664252, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[3] < 5:\n continue\n if row[1] == 0:\n net_revenue.append([row[0], row[2], row[4], row[5], row[6]])\n else:\n discount = row[1] * row[2]\n unit_price = row[2] - discount\n tax = unit_price * row[5] / 100\n net_revenue.append([row[0], unit_price, row[4], row[5], row[6] - tax])\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00231_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "0a7cf7d36d55b2746d25950ba1eea3b37ca170eb5aef512f124c5f349a495175", "output_hash": "fa72dc69f0801d0b7185b5a96cd62b4b7edf0991e9e07bf04d0edca0b6ab474c"} | ||||
| 13512 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "9c31ec74c6c7239a5b8d2425", "attempt_id": "765439cc5491", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:43:05.123121Z", "task_family": "exec", "document_id": "exec_00232_2000", "document_hash": "94e54bf28dac5a404fdf94451d3e632c18f63dff001cc758f89f5239d27f659b", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00232_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.29024943310657597, "origin_tokens": 1764, "compressed_tokens": 583, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1764, "source_tokens_receiver": 1764, "query_tokens": 689, "message_tokens": 583, "message_utf8_bytes": 2392, "cache_tensor_bytes": null, "serialized_payload_bytes": 2392, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 170.66475003957748, "decode_payload_ms": 0.0, "receiver_prefill_ms": 81.18833298794925, "ttft_ms": 13.64324998576194, "decode_ms": 1334.701249958016}, "total_wall_ms": 1430.8841250604019, "ttft_ms": 13.64324998576194, "input_usage": {"receiver_prompt_tokens": 689}, "output_usage": {"output_tokens": 99}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6909198104, "mlx_cache_bytes": 624002072, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6909198104, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n total = 0\n for row in rows:\n if row[4]:\n if row[2] < 5:\n total += (row[1] - row[2] * row[3]) * row[5]\n else:\n total += row[1] * row[5]\n else:\n total += row[1] * row[5]\n return round(total, 2)\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00232_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "2779df1d50fd05a169f3780709dd8ff462bb7fac0793db2f3d291bdf6886efb0", "output_hash": "7a9e39d5f13655cb75a9bf3c806678bbd1303f68368d3510a3d0b989e7e2991b"} | ||||
| 13513 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "15d1a22545249e7df14cf7b9", "attempt_id": "36b13cca650c", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:43:06.778263Z", "task_family": "exec", "document_id": "exec_00233_2000", "document_hash": "c0f56e7b9b73df623955c8b6822884e4b53df031c58736213f498ed2b4d693dc", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00233_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2905788876276958, "origin_tokens": 1762, "compressed_tokens": 569, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1762, "source_tokens_receiver": 1762, "query_tokens": 675, "message_tokens": 569, "message_utf8_bytes": 2434, "cache_tensor_bytes": null, "serialized_payload_bytes": 2434, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.22029105667025, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.74895909521729, "ttft_ms": 13.602708000689745, "decode_ms": 1277.2740840446204}, "total_wall_ms": 1372.917874949053, "ttft_ms": 13.602708000689745, "input_usage": {"receiver_prompt_tokens": 675}, "output_usage": {"output_tokens": 95}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6901481184, "mlx_cache_bytes": 616055776, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6901481184, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = 0\n for row in rows:\n if row[3] < 3:\n continue\n unit_price = row[1] * row[2]\n discount = row[1] * row[2] * row[3] / 100\n unit_price -= discount\n net_revenue += unit_price * row[4]\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00233_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "7a7bd7ac5b59a6505d1891d602d084031b9cedc8e40c070443b850ce8d73e56f", "output_hash": "6b0b627e7aa4da77376f2bf48b04aea4178778e91af267a73a5d2c25c272e043"} | ||||
| 13514 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "e8ecba377aaff21f86df692c", "attempt_id": "290c00c6d011", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:43:08.376996Z", "task_family": "exec", "document_id": "exec_00234_2000", "document_hash": "3522724b66617b60e55fa67efc23998ff528130352e7a960ee1e052d697670a5", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00234_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.29357798165137616, "origin_tokens": 1744, "compressed_tokens": 569, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1744, "source_tokens_receiver": 1744, "query_tokens": 675, "message_tokens": 569, "message_utf8_bytes": 2504, "cache_tensor_bytes": null, "serialized_payload_bytes": 2504, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 172.45045898016542, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.94512496609241, "ttft_ms": 13.613666989840567, "decode_ms": 3052.1428750362247}, "total_wall_ms": 3148.0328330071643, "ttft_ms": 13.613666989840567, "input_usage": {"receiver_prompt_tokens": 675}, "output_usage": {"output_tokens": 226}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6901481184, "mlx_cache_bytes": 616055776, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6901481184, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n total_price = row['total_price']\n return_pct = row['return_pct']\n discount = row['discount']\n tax = row['tax']\n units = row['units']\n unit_price = row['unit_price']\n days = row['days']\n is_returned = row['is_returned']\n shipped_days = row['shipped_days']\n\n if is_returned:\n total_price -= units * unit_price * (discount / 100)\n total_price -= units * unit_price * (tax / 100)\n total_price = round(total_price, 1)\n\n net_revenue.append({\n 'total_price': total_price,\n 'return_pct': return_pct,\n 'discount': discount,\n 'tax': tax,\n 'units': units,\n 'unit_price': unit_price,\n 'days': days,\n 'is_returned': is_returned,\n 'shipped_days': shipped_days\n })\n\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00234_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "af22e608089fbe91e1860ef1c984588749c4f27891dfd740000af7070a5e6baf", "output_hash": "084cc93db66cac640271f6584db58241563a0fe1b936c6249015b23ed8a94f42"} | ||||
| 13515 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "445f29a9d6b7ea696aad9995", "attempt_id": "ec3faa2102bd", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:43:11.750826Z", "task_family": "exec", "document_id": "exec_00235_2000", "document_hash": "20bfca2f1ed7277720116b37988eb2bf9331505c61f7491ff7f131ed0b2d70e1", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00235_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.28893905191873587, "origin_tokens": 1772, "compressed_tokens": 555, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1772, "source_tokens_receiver": 1772, "query_tokens": 661, "message_tokens": 555, "message_utf8_bytes": 2405, "cache_tensor_bytes": null, "serialized_payload_bytes": 2405, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.58554203342646, "decode_payload_ms": 0.0, "receiver_prefill_ms": 79.1646660072729, "ttft_ms": 13.56295810546726, "decode_ms": 3667.922291904688}, "total_wall_ms": 3761.9722080416977, "ttft_ms": 13.56295810546726, "input_usage": {"receiver_prompt_tokens": 661}, "output_usage": {"output_tokens": 272}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6893469352, "mlx_cache_bytes": 607814568, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6893469352, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[3]:\n if row[2] < 6:\n net_revenue.append({\n 'total_price': row[0] * row[1],\n 'discount': row[2] * row[1],\n 'tax': 0,\n 'return': row[2] * row[1],\n 'result': row[2] * row[1] - row[2] * row[1]\n })\n else:\n net_revenue.append({\n 'total_price': row[0] * row[1],\n 'discount': 0,\n 'tax': row[0] * row[1] * row[2],\n 'return': 0,\n 'result': row[0] * row[1] * row[2]\n })\n else:\n net_revenue.append({\n 'total_price': row[0] * row[1],\n 'discount': 0,\n 'tax': row[0] * row[1] * row[2],\n 'return': 0,\n 'result': row[0] * row[1] * row[2]\n })\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00235_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "075331a617210b26cdde49035dcae1522276146d58964e86c6918af5f89f2c76", "output_hash": "4730108a74f722814b200acc44323cfd87cc630ba021d49ce0346f9b6a22345d"} | ||||
| 13516 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "0afab555476f2096be36b122", "attempt_id": "ac59ee619915", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:43:15.738341Z", "task_family": "exec", "document_id": "exec_00236_2000", "document_hash": "23053422d45849043d9eb3706daf105a581cd66b69e96220ab1285c7c92b2058", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00236_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2879640044994376, "origin_tokens": 1778, "compressed_tokens": 558, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1778, "source_tokens_receiver": 1778, "query_tokens": 664, "message_tokens": 558, "message_utf8_bytes": 2335, "cache_tensor_bytes": null, "serialized_payload_bytes": 2335, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 172.06883302424103, "decode_payload_ms": 0.0, "receiver_prefill_ms": 79.74229205865413, "ttft_ms": 13.753874925896525, "decode_ms": 1454.0045419707894}, "total_wall_ms": 1548.7868339987472, "ttft_ms": 13.753874925896525, "input_usage": {"receiver_prompt_tokens": 664}, "output_usage": {"output_tokens": 108}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6895107764, "mlx_cache_bytes": 609502132, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6895107764, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = 0\n for row in rows:\n if row[5]: # return\n if row[6] < 3:\n continue\n unit_price = row[7]\n discount = row[8]\n price_pct_days = row[9]\n total_price = unit_price - discount\n net_revenue += total_price * (1 - price_pct_days / 100)\n return round(net_revenue, 2)\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00236_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "db5a60832cf405c7d1bf66d6a49725a3e3704d0322eebaac23190ce58530c979", "output_hash": "2a878c0d44d41ce75fef86c29083194b25a4e8da2369dfce1a04a6f3fd423468"} | ||||
| 13517 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "9046a31f6157f55c3c551801", "attempt_id": "b5b923f3c5e9", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:43:17.510068Z", "task_family": "exec", "document_id": "exec_00237_2000", "document_hash": "a8320d4e9d153e4398af56ca92c73f39bb1c1b9e97609e9ec096c1157cebe872", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00237_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.28476084538375973, "origin_tokens": 1798, "compressed_tokens": 573, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1798, "source_tokens_receiver": 1798, "query_tokens": 679, "message_tokens": 573, "message_utf8_bytes": 2350, "cache_tensor_bytes": null, "serialized_payload_bytes": 2350, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 170.47075007576495, "decode_payload_ms": 0.0, "receiver_prefill_ms": 81.11183298751712, "ttft_ms": 13.52249993942678, "decode_ms": 1654.3575840769336}, "total_wall_ms": 1750.311000039801, "ttft_ms": 13.52249993942678, "input_usage": {"receiver_prompt_tokens": 679}, "output_usage": {"output_tokens": 123}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6904004336, "mlx_cache_bytes": 618644464, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6904004336, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[5] and row[6] < 5:\n continue\n if row[2] < 0.975:\n continue\n total_price = row[1] * (1 - row[2])\n if row[3]:\n total_price *= 1 + row[4]\n net_revenue.append([row[0], total_price, row[5], row[6], row[7], row[8]])\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00237_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "c366a2643874e918056d40df146f31f6ec7e70198fa23f9423595bc08528aaae", "output_hash": "a05062f8bd8bcaacee322fd2b1aa5dd3b72e7bf435f31e23fce3305886e2eb97"} | ||||
| 13518 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "9d962bca79b9a33461333901", "attempt_id": "1b9fd34d02c9", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:43:19.485440Z", "task_family": "exec", "document_id": "exec_00238_2000", "document_hash": "068a11a09e40cb07626e21d7abb095fb9c4779acc1a0db602ca10678ae79e3e0", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00238_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2939150401836969, "origin_tokens": 1742, "compressed_tokens": 564, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1742, "source_tokens_receiver": 1742, "query_tokens": 670, "message_tokens": 564, "message_utf8_bytes": 2465, "cache_tensor_bytes": null, "serialized_payload_bytes": 2465, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.4025839464739, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.21716703660786, "ttft_ms": 13.643624959513545, "decode_ms": 1595.2362089883536}, "total_wall_ms": 1690.3827090281993, "ttft_ms": 13.643624959513545, "input_usage": {"receiver_prompt_tokens": 670}, "output_usage": {"output_tokens": 118}, "provider_cached_usage": null, "process_memory": 9841590272, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6898482892, "mlx_cache_bytes": 612975564, "process_rss_bytes": 9841590272}, "peak_prefill_memory": null, "peak_decode_memory": 6898482892, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n total_price = row[2] * row[3]\n discount = total_price * row[4] / 100\n total_price -= discount\n tax = total_price * row[5] / 100\n total_price -= tax\n return_value = total_price * row[6] if not row[7] else 0\n net_revenue.append([row[0], total_price, return_value])\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00238_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "b07a5a326581dbb41464437f52ded46bbe4731191cb74947cf09aeb2b8828fd6", "output_hash": "1bbac4a49245c624c3c2bd889b3ca5bb98fa6bc6503d49536693893c0d6ed886"} | ||||
| 13519 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "18dd976d493f0cc2aca7d0d5", "attempt_id": "e8df83d39b05", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:43:21.399526Z", "task_family": "exec", "document_id": "exec_00239_2000", "document_hash": "015c78e1e480fb805976a26912f0308a79a0c4b2122313463b5ff6b5ee996a9b", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00239_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2922374429223744, "origin_tokens": 1752, "compressed_tokens": 557, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1752, "source_tokens_receiver": 1752, "query_tokens": 663, "message_tokens": 557, "message_utf8_bytes": 2394, "cache_tensor_bytes": null, "serialized_payload_bytes": 2394, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 170.9343750262633, "decode_payload_ms": 0.0, "receiver_prefill_ms": 80.18466690555215, "ttft_ms": 13.709082966670394, "decode_ms": 1455.5911660427228}, "total_wall_ms": 1550.8362920954823, "ttft_ms": 13.709082966670394, "input_usage": {"receiver_prompt_tokens": 663}, "output_usage": {"output_tokens": 108}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6894730928, "mlx_cache_bytes": 609108912, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6894730928, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = 0\n for row in rows:\n price = float(row[1])\n pct = float(row[2])\n days = int(row[3])\n discount = 1 - (pct / 100)\n unit_price = price * discount\n if days < 5:\n return 0\n return_value = unit_price * days\n net_revenue += return_value\n return round(net_revenue, 2)\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00239_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "2ec13143b18b0a4408855920d30d16679b1a082132485e80ff8efb8e60896c44", "output_hash": "d4c43b000897eba9cd21aca9b01f0eb968ad1ae5f82df480b826b3ae5be72cf6"} | ||||
| 13520 | +{"schema_version": "wp30-attempt-1", "study_id": "WP30", "experiment_id": "E1_core", "trial_id": "fb6346ff10694f9fe950cae9", "attempt_id": "8f48401b3cd4", "run_group_id": "E1_core:A8|budget_tokens=512|blind", "protocol_hash": "404c52b34d964693da72ef459a036ffc42fbf5f6ba8748cc9de0c554ebebddd6", "config_hash": "5c54f742b14d64de50176f3e49c89e86fed0974a2d46d770553a5f18af6194e0", "git_commit": "3a436c8586dd6931359f6444e44174dea069c358", "timestamp_utc": "2026-10-30T08:43:23.175346Z", "task_family": "exec", "document_id": "exec_00240_2000", "document_hash": "4c3927b8378a2b949e85371101d959a0adebcbe41a2d39f239aae883de382e19", "question_id": "q00_exec", "split": "test", "cluster_id": "exec_00240_2000", "sender_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "receiver_model_revision": "mlx-community/Llama-3.2-3B-Instruct-bf16@6d88ba43024fef71b10e52e101c7cd4598322601", "tokenizer_hashes": {"sender": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763", "receiver": "b9d8872cf3a99da99e1fc4b0f71232a451632dc49ca4c6771549b4efa5c1c763"}, "backend_versions": {"mlx": "0.32.4", "mlx_lm": "0.32.2"}, "hardware_id": "laptop-m5max-48g", "device_state": {"mlx_active_before": 6425506824}, "weight_precision": "bf16", "method_id": "A8", "implementation_fidelity": "official", "compression_config": {"budget_tokens": 512, "access": "blind", "realized": {"budget_tokens": 512, "rate": 0.2887760857304005, "origin_tokens": 1773, "compressed_tokens": 568, "compressor": "microsoft/llmlingua-2-xlm-roberta-large-meetingbank", "device": "mps"}}, "question_access": "blind", "topology": "single_handoff", "handoff_index": 1, "reuse_count": 1, "cache_hit": false, "cache_compatibility_hash": null, "random_seed": 0, "source_tokens_sender": 1773, "source_tokens_receiver": 1773, "query_tokens": 674, "message_tokens": 568, "message_utf8_bytes": 2303, "cache_tensor_bytes": null, "serialized_payload_bytes": 2303, "side_information_bytes": 0, "fetched_bytes": 0, "transport_bytes_if_measured": null, "timing_spans": {"encode_ms": 171.60775000229478, "decode_payload_ms": 0.0, "receiver_prefill_ms": 81.0097079956904, "ttft_ms": 13.409790932200849, "decode_ms": 2412.74058399722}, "total_wall_ms": 2508.4400409832597, "ttft_ms": 13.409790932200849, "input_usage": {"receiver_prompt_tokens": 674}, "output_usage": {"output_tokens": 179}, "provider_cached_usage": null, "process_memory": 9841606656, "backend_memory": {"mlx_active_bytes": 6425506824, "mlx_peak_bytes": 6901300956, "mlx_cache_bytes": 615859164, "process_rss_bytes": 9841606656}, "peak_prefill_memory": null, "peak_decode_memory": 6901300956, "mapper_memory": null, "cost_with_source": null, "prediction_artifact": "```python\ndef net_revenue(rows):\n net_revenue = []\n for row in rows:\n if row[5]:\n return 0\n if row[6] < 3:\n return 0\n if row[7] < 0:\n return 0\n discount = row[7] * row[8]\n unit_price = row[9] - discount\n if unit_price <= 0:\n return 0\n tax = unit_price * row[10]\n if tax < 0:\n return 0\n net_revenue.append((row[0], row[1], row[2], row[3], row[4], row[5], row[6], row[7], row[8], row[9], row[10], tax))\n return net_revenue\n```", "evidence_artifact": null, "reference_key_id": "keys/exec/exec_00240_2000.json", "task_scores": {"correct": 0.0, "pass_rate": 0.0, "error": "KeyError"}, "status": "success", "error_code": null, "retry_of": null, "log_artifact": "runs/logs/E1_core.log", "payload_hash": "eeaeec04c695faed51a1f865bb9cc4b14a5d4162d5865c77be298854127484b5", "output_hash": "a426ad6acca423027bc0e4a0807926a2448ad0a9411b54d4fb5eb09f17d01b70"} | ||||
added
runs/raw/E3_core.jsonl
+1536 −0
Large diff (1,536 lines) is not loaded by default to keep the page fast.Load diff
added
runs/raw/E3_core_sender.jsonl
+512 −0
Large diff (512 lines) is not loaded by default to keep the page fast.Load diff
added
runs/raw/E7_qtypes.jsonl
+2520 −0
Large diff (2,520 lines) is not loaded by default to keep the page fast.Load diff
modified
src/wp30/analysis/analyze.py
+14 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
modified
src/wp30/analysis/claims.py
+32 −8
Large diff (44 lines) is not loaded by default to keep the page fast.Load diff
modified
src/wp30/analysis/latex_tables.py
+23 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff