SPB Git

spb/wp30_uqo

Public
6.4 KB · 73 lines latex
1\section{Experimental protocol}2\label{sec:protocol}34\subsection{Models, hardware and runtimes}5Table~\ref{tab:models} lists the checkpoints (all ungated \texttt{mlx-community} conversions with bf16 weights, so weight quantization is6not a confound), their attention geometry and measured prefill times. The local node is a MacBook Pro with an Apple M5 Max (18 cores,748~GiB unified memory, macOS~27.0), Python~3.12.13, \texttt{mlx}~0.32.4, \texttt{mlx-lm}~0.32.2. Two-node experiments use two Mac Studio8M3~Ultra nodes of the author's cluster (M3U96a as sender, M3U96b as receiver worker; 96~GB each; identical \texttt{mlx}/\texttt{mlx-lm}9pins and checkpoint revisions) connected by a wired LAN measured at 940~Mbit/s (iperf3, four streams) with 1.2~ms RTT. Both cluster nodes10concurrently hosted a CPU-light workload from another study (load average $\approx 1.8$ on 32 cores); timing on the laptop was taken11with no other GPU process.1213\begin{table}[t]\centering\scriptsize14\caption{Checkpoints, geometry and measured prefill (ms, laptop). Cache bytes per token are the dense bf16 formula \eqref{eq:mkv}15with $B=1$.}\label{tab:models}16\input{tables/table2_models}17\end{table}1819\subsection{Task families and splits}20\begin{description}[leftmargin=1.4em,style=nextline]21\item[Synthetic private-fact ledgers.] A deterministic generator (seeded) writes a fictional capital-projects register: six projects per22  document with invented names, random identifiers (\texttt{XX-00X-0000}), amounts and currencies, dates, sites, suppliers, approvals or23  rejections, cancelled-versus-postponed statuses, timestamped revisions with an explicit precedence rule, a list of ``proposals under24  review'' with no attributes (distractors with similar names), and boilerplate filler to reach 2\,000, 8\,000 or 16\,000 target tokens.25  Fifteen questions per document cover single-fact retrieval at varying positions, exact copying, multi-hop joins, negation,26  revision/precedence, temporal order, aggregation, unit conversion and two unanswerable questions (correct action: \texttt{unknown}).27  An evaluator-only fact ledger (62 atomic facts per document) and the answer key are stored apart from the agent-visible text. The28  facts are not inferable from pre-training. A procedurally generated ledger is not a real filing.29\item[Public long-document QA.] LongBench~v1 \citep{bai2024longbench} subsets QASPER \citep{dasigi2021qasper} and 2WikiMultihopQA30  \citep{ho2020twowiki} (raw \texttt{data.zip}; the dataset script is no longer supported by the \texttt{datasets} library), official31  F1 with the LongBench normalization. Documents are used in full. We run \emph{subsets} (the first 40 test documents of each subset in32  manifest order) and publish the identifiers. These texts may appear in pre-training corpora; they serve external validity, not the33  contamination-free test of communication dependence.34\item[Executable tasks.] A generated engineering ticket specifies six business rules for a function \texttt{net\_revenue(rows)}35  together with a buggy implementation missing two rules; the agent must output the corrected function, which is executed in an36  isolated \texttt{python -I} subprocess against five hidden test cases (exact float comparison). Agents never see the tests.37\end{description}38Splits are by document (train/dev/test seeds for the generated families; a hash rule for the public subsets), with exact-duplicate39checks across splits (none found). Train is used for mapper fitting and prompt tuning, dev for method selection and router fitting,40test for the locked campaign (Table~\ref{tab:datasets}). Answer keys live in \texttt{data/generated/keys} and are read only by the41scorer; a leakage test asserts that agent-visible files contain no answers, facts or tests.4243\begin{table}[t]\centering\footnotesize44\caption{Datasets, independent units (documents), questions and median length by split.}\label{tab:datasets}45\input{tables/table3_datasets}46\end{table}4748\subsection{Cache correctness before performance}49\label{sec:correctness}50Twelve tests run a real 1.5B checkpoint (Appendix~\ref{app:tests}). Identity round trip (serialize, load) and 100\,\% retention through51the sparse adapter reproduce the native next-token logits \emph{exactly}. Cached-prefix-plus-query and full prefill agree within a52registered distributional tolerance: MLX kernels are not bitwise deterministic across prefill chunkings even in float32 (we measured a53maximum logit difference of 0.10 between a one-chunk and a token-by-token prefill, with $\max|\Delta p|\approx7\times10^{-7}$ and54KL $\approx3\times10^{-8}$), so equivalence is defined as KL $<10^{-5}$, $\max|\Delta p|<10^{-4}$ and identical greedy top-1 in float3255(KL $<10^{-3}$ in bf16). The sparse-cache mask path agrees with token-by-token appends; preserving original positions differs measurably56from relabelling them; packed storage has the intended bit width (uint32 words, ratios 1.9/3.6/6.4 for 8/4/2 bits on a 412-token57prefix including scales and biases); the loader rejects a wrong prefix hash or a different checkpoint; branches do not share mutable58state; and the no-message control does not know the private answer.5960\subsection{Pilot, freeze and campaign}61A development pilot (\texttt{E1\_pilot}: 16 documents, 31 conditions, two model pairs) calibrated budgets, revealed implementation62failures and estimated variance and run time. The confirmatory protocol was then frozen (\texttt{protocol/frozen.json}, hash63\texttt{\input{tables/protocol_hash}}) with the final hypotheses, prompts, budgets, scorers and analysis plan, before any test-split64item was evaluated. The core single-handoff campaign (\texttt{E1\_core}) covers 160 test documents (40 per family), 22 conditions and65two model pairs (13\,600 scheduled trials); the handoff, fan-in, transfer, network and hosted campaigns are described with their results.6667\subsection{Statistics}68The unit is a (document, question, condition, pair) trial; documents are the independent clusters. Means and paired differences on69identical instances are reported with document-level cluster-bootstrap 95\,\% intervals (2\,000 replicates). Six registered primary70comparisons are Holm-corrected; everything else is exploratory. Quality is accuracy (synthetic, executable) or F1 (public); failed or71invalid attempts count as 0 in the operational quality and are also tabulated by category. Latency is reported as p50/p95 with sample72counts, from the spans defined in Section~\ref{sec:formulation}.73
Files · f4474e1b03aebffdfcd19ab989b35165571329e6
Menu

Theme