SPB Git

spb/wp29_uqo

Public
2.6 KB · 21 lines latex
1\subsection*{Human interventions}2\begin{longtable}{p{0.5in}p{1.1in}p{0.9in}p{0.8in}p{2.1in}p{0.9in}}3\toprule4ID & UTC & Actor & Category & Description & Duration / basis \\5\midrule6\endhead7I-001 & 2026-10-29T02:38 & researcher (S.-P. Boucher) & preparation & Prepared the WP29 mandate (CLAUDE.md text) outside the session and pasted it into a Claude Code session with the instruction 'fait ce projet de recherche'. & unknown / not measured (historical preparation time unknown) \\8\bottomrule9\end{longtable}10\subsection*{Amendments to frozen designs}11\begin{longtable}{p{0.5in}p{0.9in}p{0.6in}p{0.9in}p{3.4in}}12\toprule13ID & UTC & Phase & Timing & Description and rationale \\14\midrule15\endhead16A-001 & 2026-10-30T03:00 & 02 & before\ results & Provisional live-inference envelope set by the executor: per benchmark run 4.00 USD / 1500 s / 80 turns per session; per transfer run 12.00 USD / 3600 s; total study envelope 800 USD (pilot + main + transfer + recovery + ablations) with the confirmatory scale to be chosen after the pilot. \textit{Rationale:} The mandate requires a precise budget decision but none was stated; the researcher is not available mid-run; prior UQO studies spent $\approx$1,079 USD of API inference (WP26), so 800 USD is a conservative envelope. The researcher may raise or lower it; any ch \\17A-002 & 2026-10-30T04:40 & 06 & before\ confirmatory\ results & Final evaluators are now executed on a composed workspace (public bundle + the run's collected out/) instead of the run directory alone, which lacked data/ and made source-grounded checks (F4 'numbers\_in\_sources') fail for correct answers. All pilot runs evaluated before the fix are re-evaluated with the same oracles; original receipts are kept in runs/pilot/\textless{}run\textgreater{}/evaluation\_receipt\_pre-A00 \textit{Rationale:} Pilot diagnosis: F4-dev-01 failed in all four arms with a spurious fabrication violation although every required criterion passed. Harness defect, not an oracle change. \\18A-003 & 2026-10-30T04:02 & 06 & before\ confirmatory\ results & Preregistration frozen as v1.0 (adds §10 pilot-informed decisions: confirmatory design 24 tasks $\times$ 4 arms $\times$ 3 repeats = 288 runs, seed 301); freeze manifest protocol/freeze\_manifest\_conf.json records SHA-256 of all 24 confirmatory bundles and oracles, arm texts, gate, launcher, scheduler, evaluator, configs and analysis scripts; tasks/registry.csv stamped frozen\_utc. \textit{Rationale:} Scale decision per §6 of the preregistration using measured pilot costs; no hypothesis, endpoint or analysis rule changed. \\19\bottomrule20\end{longtable}21
Files · 4f59ed568641ceda6f39552d5cc3a441aa812088
Menu

Theme