SPB Git

spb/wp29_uqo

Public
3.9 KB · 23 lines latex
1\subsection*{Human interventions}2\begin{longtable}{p{0.5in}p{1.1in}p{0.9in}p{0.8in}p{2.1in}p{0.9in}}3\toprule4ID & UTC & Actor & Category & Description & Duration / basis \\5\midrule6\endhead7I-001 & 2026-10-29T02:38 & researcher (S.-P. Boucher) & preparation & Prepared the WP29 mandate (CLAUDE.md text) outside the session and pasted it into a Claude Code session with the instruction 'fait ce projet de recherche'. & unknown / not measured (historical preparation time unknown) \\8\bottomrule9\end{longtable}10\subsection*{Amendments to frozen designs}11\begin{longtable}{p{0.5in}p{0.9in}p{0.6in}p{0.9in}p{3.4in}}12\toprule13ID & UTC & Phase & Timing & Description and rationale \\14\midrule15\endhead16A-001 & 2026-10-30T03:00 & 02 & before\ results & Provisional live-inference envelope set by the executor: per benchmark run 4.00 USD / 1500 s / 80 turns per session; per transfer run 12.00 USD / 3600 s; total study envelope 800 USD (pilot + main + transfer + recovery + ablations) with the confirmatory scale to be chosen after the pilot. \textit{Rationale:} The mandate requires a precise budget decision but none was stated; the researcher is not available mid-run; prior UQO studies spent $\approx$1,079 USD of API inference (WP26), so 800 USD is a conservative envelope. The researcher may raise or lower it; any ch \\17A-002 & 2026-10-30T04:40 & 06 & before\ confirmatory\ results & Final evaluators are now executed on a composed workspace (public bundle + the run's collected out/) instead of the run directory alone, which lacked data/ and made source-grounded checks (F4 'numbers\_in\_sources') fail for correct answers. All pilot runs evaluated before the fix are re-evaluated with the same oracles; original receipts are kept in runs/pilot/\textless{}run\textgreater{}/evaluation\_receipt\_pre-A00 \textit{Rationale:} Pilot diagnosis: F4-dev-01 failed in all four arms with a spurious fabrication violation although every required criterion passed. Harness defect, not an oracle change. \\18A-003 & 2026-10-30T04:02 & 06 & before\ confirmatory\ results & Preregistration frozen as v1.0 (adds §10 pilot-informed decisions: confirmatory design 24 tasks $\times$ 4 arms $\times$ 3 repeats = 288 runs, seed 301); freeze manifest protocol/freeze\_manifest\_conf.json records SHA-256 of all 24 confirmatory bundles and oracles, arm texts, gate, launcher, scheduler, evaluator, configs and analysis scripts; tasks/registry.csv stamped frozen\_utc. \textit{Rationale:} Scale decision per §6 of the preregistration using measured pilot costs; no hypothesis, endpoint or analysis rule changed. \\19A-004 & 2026-10-30T04:40 & 07 & during\ confirmatory\ runs\ before\ any\ analysis & Harness defect: continuation sessions (gate-feedback or review-repair resumptions) were inserted into the sessions ledger with the same client session id as the initial session, violating the primary key and aborting the run as an infrastructure error after the reviewer had returned 'revise' (first occurrence RC0094, F4-conf-02 arm D, 04:33 UTC). Fix: ledger rows keyed per launch (session\_id\#seq) with the client se \textit{Rationale:} Infrastructure outage policy of the preregistration (§7): balanced reruns of infrastructure errors; the error affects only runs needing a continuation (arm C/D), so without the rerun the comparison would be biased against C/D. \\20A-005 & 2026-10-30T05:40 & 09 & after\ results\ sensitivity\ only & Two confirmatory oracles were found to reject correct behaviour in every arm: F1-conf-01 required feasibility='complete' although one window month is genuinely undefined in the source (all 12 runs reported 'partial' with the right reason and passed every criterion); F6-conf-04 expected n\_analysed=1200 although the generator had produced 12 firms with impossible negative training hours that 11 of 12 runs detected and \textit{Rationale:} Evaluator validity audit required by the mandate (§13.1, §14.4); the defects affect all arms symmetrically. \\21\bottomrule22\end{longtable}23
Files · 6df43c338e49477c769c36e7f7d249af2563db7f
Menu

Theme