SPB Git

spb/wp29_uqo

Public
12.1 KB · 85 lines latex
1\section{Main results}2\label{sec:results}34\subsection{Execution accounting}56Table~\ref{tab:accounting} reconciles the ledger with the frozen assignment. All \nMainRuns{} confirmatory runs were launched; \nMainTimeouts{} hit the wall-clock limit and \nMainBudget{} exhausted the budget cap. One run (RC0094, F4-conf-02, arm D) ended in an infrastructure error after its reviewer returned a \emph{revise} verdict, because the harness inserted the resumed session into the ledger under the same client session identifier as the initial session (amendment A-004); the defect was fixed and the run re-executed as a fresh attempt after the batch under the same run identifier, with the first attempt's transitions and sessions retained in the ledger and its \costAbortedAttempt{} USD reported in the accounting (Appendix~\ref{app:stats}) but not in the run's envelope. No other continuation was needed in the batch: the controller gate passed at its first cycle in every arm C and D run (\nGateControllerFail{} controller gate failures in \nRunsC{}+\nRunsD{} runs), and \nReviewsRevise{} of \nReviews{} reviews returned \emph{revise} (one review wrote no verdict file). Every executor session finished on its own (\texttt{result.subtype = success}); no human intervened in any session.78\begin{table}[htbp]\centering\small9\caption{Assignment and execution accounting by split. ``Completed'' = executor session finished on its own; ``Evaluated'' = final receipt stored; ``Successes'' = scientific success under the frozen oracles.}10\label{tab:accounting}11\input{../results/tables/tab_accounting}12\end{table}1314\subsection{Primary endpoint}1516Table~\ref{tab:arms} and Figure~\ref{fig:primary} give the task-weighted rate of intervention-free verified completion by arm and the prespecified contrasts. The four arms are indistinguishable: $\hat p_A = \pA$, $\hat p_B = \pB$, $\hat p_C = \pC$, $\hat p_D = \pD$. The primary contrast is $\hat\Delta_{DA} = \dDA$ (task-cluster bootstrap 95\% interval $[\dDAlo, \dDAhi]$; sign-flip permutation $p = \dDAp$); across the \nConfTasks{} tasks, D beat A on \dDAfav{} tasks (favouring D / tied / favouring A). The mechanism contrasts are equally null: B$-$A $= \dBA$ $[\dBAlo, \dBAhi]$, C$-$B $= \dCB$ $[\dCBlo, \dCBhi]$, D$-$C $= \dDC$ $[\dDClo, \dDChi]$. The interval for the primary contrast runs from about three percentage points against D to about six in its favour; at this task difficulty and with this model, the protocol, the gates and the review changed neither the completion rate nor, as shown below, the violation rate, while they raised the cost per run by a factor of \costRatioB{} (B), \costRatioC{} (C) and \costRatioD{} (D) relative to the open instruction.1718\begin{table}[htbp]\centering\small19\caption{Primary and secondary endpoints by arm (confirmatory split, frozen oracles). $\hat p_a$ is the task-weighted completion rate of Eq.~\eqref{eq:phat}; ``Correct (any)'' ignores the envelope; ``Viol.\ rate'' is the share of runs with at least one critical violation; ``Repro.'' is the share whose replication script regenerated the results in a fresh copy; ``Quality'' is the weighted component score.}20\label{tab:arms}21\input{../results/tables/tab_arms_confirmatory}22\end{table}2324\begin{figure}[htbp]\centering25\includegraphics[width=\textwidth]{../results/figures/fig_primary_confirmatory.pdf}26\caption{(a) Task-weighted completion by arm with task-cluster bootstrap intervals; (b) paired task-level contrasts with 95\% bootstrap intervals, permutation $p$-values and the count of tasks favouring each side. Confirmatory split, \nMainRuns{} runs, \nConfTasks{} task clusters.}27\label{fig:primary}28\end{figure}2930\begin{table}[htbp]\centering\small31\caption{Prespecified contrasts for the primary endpoint and secondary endpoints (task-cluster bootstrap 95\% intervals, sign-flip permutation $p$, Holm adjustment over the three secondary contrasts).}32\label{tab:contrasts}33\input{../results/tables/tab_contrasts_confirmatory}34\end{table}3536\subsection{Where the failures are}3738Figure~\ref{fig:errors} and Table~\ref{tab:family} locate the \nMainNonSuccess{} non-successes of the frozen scoring. On inspection of every one of them, together with the adversarial evaluator audit of Appendix~\ref{app:evaluators}, all of them are defects of our oracles rather than of the executors, and they fall into four groups. First, F1-conf-01: the source publishes no consumer-price index for October 2025, so the twelve-month growth rate of one window month is undefined; all twelve executor sessions computed every required value correctly, reported \texttt{feasibility = partial} with exactly that reason, and were scored as a false completion because the frozen oracle required \texttt{complete} although neither the task statement nor the common-key specification defines the label. Second, F6-conf-04: the data generator had produced twelve firms with negative training hours; eleven of twelve sessions detected the impossible values, excluded them with a documented exclusion ledger and a data-issue entry, and reported sensitivity estimates on the full sample, which the frozen oracle scored as a wrong count and a wrong baseline because its reference used all 1,200 rows (the one session that kept all rows passed). Third, F3-conf-02: one arm-A session reported its error metrics rounded to four decimals against a tolerance of $10^{-6}$ relative that no precision rule in the task justified; its forecast ledger is internally consistent. Fourth, F6-conf-01: the study's only arm-specific critical violation, an invalid-inference flag on one arm-D run of the genuine-null weekday task, is a regular-expression false positive: the claim text ``conclusion code: no\_detectable\_effect'' matched the oracle's pattern for an effect claim while its negation guard failed on the underscore.3940We did not change the frozen scoring. Under the rubric-corrected re-scoring (amendments A-005 and A-007: \texttt{partial} with the correct reason accepted; the 1,188-row sample with an independently recomputed baseline accepted at the frozen tolerances; a relative tolerance of $10^{-3}$ for metrics rounded to four decimals; the conclusion code excluded from the effect-claim pattern), every arm completes every task: $\sensA$, $\sensB$, $\sensC$, $\sensD$, and all \nSensChanged{} non-successes flip. This re-scoring is a post hoc sensitivity analysis of the kind the analysis plan calls for under ``uncertain rubric labels''; it is not a prespecified contrast, and it is reported to make the reading of the frozen result unambiguous: the benchmark, at this difficulty and with this model, is at the ceiling in every arm, and the frozen scoring's \nMainViolRuns{} runs with a ``violation'' measure one feasibility label and one regular expression, not scientific misconduct. No leakage, fabrication, unreported-substitution or provenance violation occurred in any of the \nMainRuns{} runs under either scoring. The length-matched control (arm L; Section~\ref{sec:recovery}) and the development pilot (23 of 24 successes) point in the same direction.4142\begin{figure}[htbp]\centering43\includegraphics[width=\textwidth]{../results/figures/fig_quality_confirmatory.pdf}44\caption{(a) Quality components by arm (task-weighted means); (b) share of runs with at least one critical violation, by type and arm. Confirmatory split, frozen oracles.}45\label{fig:quality}46\end{figure}4748\begin{figure}[htbp]\centering49\includegraphics[width=\textwidth]{../results/figures/fig_errors_confirmatory.pdf}50\caption{Failed scientific criteria by task and criterion, stacked by arm (number of runs; denominator \nRepeats{} runs per task and arm).}51\label{fig:errors}52\end{figure}5354\begin{table}[htbp]\centering\small55\caption{Task-family heterogeneity: completion and violation rates by family and arm (four tasks per family, \nRepeats{} repeats).}56\label{tab:family}57\input{../results/tables/tab_family_confirmatory}58\end{table}5960\subsection{What the gates and the reviewer actually did}6162The absence of an effect on outcomes does not mean the mechanisms were inert; it means they found nothing to fix at the level the oracles measure. The post-hoc contract check that all arms receive failed for \nContractFailA{} arm-A and \nContractFailB{} arm-B runs (\contractFailKinds{} failed checks: claims referencing absent result keys, an exclusion ledger that does not sum, a missing report section; \nReproFailAB{} reproducibility failures), and for no arm C or D run: the gate did enforce the deliverable contract, and every replication script in every arm regenerated its results. The scientific criteria were met with or without that contract, because the evaluators score values, not formatting. Reviewers in arm D returned on average \reviewIssuesMean{} issues per review, almost all minor (unreproduced context percentages, over-assured wording); \nReviewsRevise{} of \nReviews{} reviews asked for a revision and one review (RC0159) ended without a verdict file, which the harness silently treated as acceptance and which we count as a missing measurement. The single \emph{revise} verdict was returned in the aborted first attempt of RC0094 (amendment A-004), and its second attempt was accepted at the first review; consequently \nRepairSessions{} review-repair sessions and \nGateCycleSessions{} controller gate-feedback sessions executed in the benchmark, and hypothesis H3 (separate review improves claim validity) is not evaluable from the benchmark: arm D's repair loop was available but never triggered. The behavioural audit found no outside-workspace path references in any run and no voluntary invocation of the gate program by arms A or B. One qualification applies to every continuation in this study: because of the launcher defect of amendment A-006, the text of the gate feedback and of the review feedback never reached the executor as prompt text; the resumed sessions received a one-word prompt and found the controller's gate receipts and the reviewer's \texttt{review.json} in their workspace, which is how the gate cycles of the transfer study were resolved. A post-fix verification set (the smoke task under all four arms and the real-time forecasting project under arm C, run with the corrected launcher after the analysis) is reported in Table~\ref{tab:postfix}; it is not pooled with the frozen results.6364\subsection{Cost, time and the quality--cost frontier}6566Table~\ref{tab:cost} and Figure~\ref{fig:cost} report the resource side with full denominators. Mean inference cost per run (client list-price estimate) rose from \costA{} USD in A to \costB{} (B), \costC{} (C) and \costD{} USD (D, of which the reviewer session is about a third; \costDallAttempts{} USD when the \costAbortedAttempt{} USD of RC0094's aborted first attempt is charged to the arm); median wall time rose from \wallA{} s to \wallD{} s. Because completion did not change, cost per verified completion rose in the same proportion (\costPerSuccessA{} to \costPerSuccessD{} USD). The frontier in Figure~\ref{fig:cost}(b) is flat: the additional spending bought contract compliance and reviewer commentary, not scientific validity. The total inference cost of the confirmatory stage was \costMainTotal{} USD; all splits together (smoke, pilot, main, rerun, transfer, recovery, ablation) cost \costAllSessions{} USD of inference, which is small next to the builder effort reported in Section~\ref{sec:reflexive}.6768\begin{table}[htbp]\centering\small69\caption{Runtime, usage and cost by arm (USD, client list-price estimates; wall time in seconds; ``Cost/success'' = total cost divided by the number of verified completions).}70\label{tab:cost}71\input{../results/tables/tab_cost_confirmatory}72\end{table}7374\begin{figure}[htbp]\centering75\includegraphics[width=\textwidth]{../results/figures/fig_cost_confirmatory.pdf}76\caption{(a) Inference cost and wall time of every confirmatory run; (b) quality--cost frontier: mean cost per run against task-weighted completion, by arm.}77\label{fig:cost}78\end{figure}7980\begin{table}[htbp]\centering\small81\caption{Post-fix verification runs (corrected launcher, full user prompt; amendment A-006). Development smoke task under all four arms and the real-time forecasting project under arm C. Not pooled with the frozen results.}82\label{tab:postfix}83\input{../results/tables/tab_postfix}84\end{table}85
Files · main
Menu

Theme