spb/wp42_uqo

1% ---------------------------------------------------------------------------2% Section 7 — Data and measurement (Agent H; every number from analysis/tables,3% analysis/derived, data/validated or docs/PHASE_5_AUDIT.md; the source file is4% named in a comment next to each exhibit)5% ---------------------------------------------------------------------------6\section{Data and measurement}\label{sec:data}78This section describes the data that the laboratory of Section~\ref{sec:infrastructure} produced when the design of Section~\ref{sec:design} was executed: the run-level datasets and their variables, the descriptive statistics of the runs by arm, the three evaluation instruments and the degree to which they agree, the measurement of cost, the treatment of missing values and of the amendments that affected the data, and the layout of the immutable raw records and of the validated datasets from which every exhibit of Sections~\ref{sec:results} and~\ref{sec:reliability} is computed. No number in this section or in the two that follow is typed by hand: each comes from a table or derived file written by the analysis scripts, whose name is given in a comment in the source of the manuscript next to the exhibit, and the extra descriptives that the writing required were computed by a dedicated script whose output is itself a table of the reproducibility package.%9\footnote{Script \texttt{extra\_descriptives.py} in \texttt{analysis/robustness/}, output \texttt{table14\_extra\_descriptives.csv} in \texttt{analysis/tables/}. The other sources are \texttt{estimate.py} in \texttt{analysis/econometrics/} (tables 1--8 and \texttt{estimates\_pilot.json} in \texttt{analysis/derived/}), and in \texttt{analysis/robustness/}: \texttt{audit\_pilot.py} (the \texttt{audit\_*} tables), \texttt{sensitivity\_models.py} (tables 9, 9b, 9c), \texttt{recovery\_experiment.py} (table 10 and \texttt{recovery\_experiment.json}) and \texttt{economic\_sensitivity.py} (tables 11--13).}1011\subsection{The run-level datasets}\label{sec:data-datasets}1213The unit of observation is the run. Three stages were executed and are analysed separately, as the preregistration requires when a stage is not pooled with the pilot. The \emph{pilot} comprises 150 runs: the ten validated scenarios of Table~\ref{tab:pilot}, each run under the five arms A0--A4 with three repeats. The first two repeats (100 runs, seed 4242) are the preregistered pilot; the third repeat (50 runs, seed 4243) was added under the scale decision rule of Section~\ref{sec:design-scale} after the pilot's mean total cost per run, 1.07~dollars including judges, satisfied the rule (amendment A-009), and was pooled with the first two because the freeze verification found no change to any scenario, oracle, rubric or arm text, the only modified file being the judge sanitisation of amendment A-008. The \emph{sensitivity} stage comprises 40 runs: arm A3 on the same ten scenarios with two repeats each under \texttt{claude-sonnet-5-5} and \texttt{claude-fable-5-1}. The \emph{recovery} stage comprises 40 runs: the exploratory addendum of amendment A-011, in which arms A3 and A4 were run on the ten scenarios with two repeats each and the first session was killed by the controller 120 seconds after launch. Two smoke runs on a dummy scenario and one development run on a pre-validation bundle of HR-01 exist in the database and are excluded, as amendment A-007 records. In total, 230 experimental runs were executed and evaluated.14% source: data/validated/runs_{pilot,sensitivity,recovery}.csv; protocol/amendments.csv A-007, A-009, A-0111516All 230 runs were executed between the evening of 2~November and the early hours of 3~November 2026, Eastern time: the pilot runs started between 00:32 and 02:26~UTC on 3~November, the sensitivity runs between 01:19 and 01:46~UTC, and the recovery runs between 02:00 and 02:26~UTC, interleaved over the four executor nodes M3U96a, M3U96b, M4M64a and M4M64b. The pilot's 150 runs are distributed as 37, 40, 38 and 35 runs over the four nodes; arm A2, which requires the operating-system automation layer, ran only on M3U96a (12 runs) and M4M64a (18 runs), as the design prescribes. Every run was executed by Claude Code version 2.1.314, which is recorded in the initialisation event of each of the 258 raw event streams in the repository; the validated export's \texttt{claude\_code\_version} column was not populated by the export step and the version is therefore read from the raw streams, which are the authoritative record. Every pilot and sensitivity run ended with a result event of subtype \texttt{success}: no run timed out, exhausted its budget or reached the turn cap, and every run received three judge outputs. All 40 recovery-stage first sessions ended by the induced interruption, recorded as a timeout.17% source: data/raw/wp42.sqlite (runs.started_utc, ended_utc); table14 block H_spend (nodes, result_subtypes, timed_out, n_judges); raw_stream.jsonl init events (claude_code_version 2.1.314 in 258 streams)1819Each run carries 54 variables in the validated run table. They derive from the four sources listed in Table~\ref{tab:variables}: the event stream of the session (number of events, tool calls by tool and by family, the sequence of tool families from which application transitions are counted, tool errors, retries, time of the first tool call, tool execution time), the client's result event (result subtype, number of turns, token usage by model, cost), the collection step (number of artefacts, the input-integrity flag from the hash check, the completion record's status, questions, issues found, assumptions), and the evaluation pipeline (deterministic completion and score, the number of critical violations, the quality score with the three per-judge scores and acceptability votes, the abstention status and whether it was justified). Recovery variables (whether a recovery was attempted and succeeded, the first and final public self-check status, the recovery cost and wall time) are populated for arm A4 and for the recovery stage. Alongside the run table, the validated data comprise a receipts table with one row per run (evaluator version, deterministic completion and score, the hash of the oracle that produced the receipt, the per-criterion results and the list of critical violations), a judgments table with one row per judge and run (450 rows for the pilot; judge index, judge model, score, per-criterion scores and acceptability vote, cost and time of judgment), and an events table with one row per normalised event (sequence, elapsed time, event type, tool, status, error class, summary). Each receipt records a single oracle hash per scenario across the 150 runs, which confirms that no oracle or evaluator changed during the pilot; the evaluators are at version 1.1 for eight scenarios and 1.0 for HR-01, and the number of required criteria per scenario ranges from 9 (ACC-01) to 14 (BI-01).20% source: data/validated/runs_pilot.csv columns; receipts_pilot.csv; judgments_pilot.csv; events_pilot.parquet; table14 block O_criteria2122\subsection{Descriptive statistics by arm}\label{sec:data-descriptives}2324Table~\ref{tab:descriptives} reports, for the 150 pilot runs, the process and cost measurements by arm. Three regularities organise the table. First, the restricted arm A0 is the busiest and the least effective: it makes the most tool calls (29.4 per run on average, with a 90th percentile of 58.8 and a maximum of 80), records by far the most tool errors (11.9 per run, against 0.7 to 2.0 in the other arms), uses the most turns, and yet is the fastest (median wall time 162 seconds) and the cheapest (0.71 dollars). Its 356 error events are all failed calls of the file tools, since no other tool exists in that arm: 270 are attempts to read a path that does not exist, 55 are refusals of the file reader to open a binary input, 30 are attempts to read a directory as a file, and one is a failed edit. The pattern is that of an agent searching for inputs it cannot open and producing text substitutes for the binary deliverables it cannot write, a behaviour that Section~\ref{sec:results} shows to be the source of its zero acceptance. Second, among the four arms with a shell, the number of tool calls rises with the size of the tool set, from 12.4--12.9 in A1 and A2 to 17.2 and 18.8 in A3 and A4, as do application transitions (3.3 and 4.4 in A1 and A2 against 9.3 in A3 and A4), the number of artefacts delivered (5.2 to 7.7), wall time (median 214--230 seconds) and cost (0.78 to 0.93 dollars). The additional calls of A3 and A4 are browser and office calls: Section~\ref{sec:reliability-policy} shows that 23 per cent of A3's calls and 19 per cent of A4's are browser calls, and 13--15 per cent office calls, families that do not exist in A1 and barely in A2. Third, the time that tools themselves consume is small in every arm---between 0.6 and 6.8 seconds per run on average---so that wall time is almost entirely model inference and token streaming; the first tool call arrives after about three seconds in every arm.25% source: analysis/tables/table1_outcomes_by_arm_pilot.csv; table14 blocks A_process, O_tokens, O_a0_errors (summary_prefix counts 270/55/30/1), M_families2627\begin{table}[htbp]28\centering29\caption{Process and cost measurements of the 150 pilot runs, by arm}30\label{tab:descriptives}31\footnotesize32\setlength{\tabcolsep}{4pt}33\begin{adjustbox}{max width=\textwidth}34\begin{tabular}{@{}lrrrrr@{}}35\toprule36 & A0 & A1 & A2 & A3 & A4 \\37\midrule38Runs (scenarios) & 30 (10) & 30 (10) & 30 (10) & 30 (10) & 30 (10) \\39\addlinespace40Tool calls, mean & 29.4 & 12.9 & 12.4 & 17.2 & 18.8 \\41\quad median (p10--p90) & 19 (14--58.8) & 12.5 (8--18) & 11.5 (8--18) & 16 (12--24.1) & 18 (14--24.3) \\42Tool errors, mean & 11.9 & 0.9 & 0.7 & 0.9 & 2.0 \\43Retries, mean & 11.0 & 0.9 & 0.7 & 0.9 & 2.0 \\44Application transitions, mean & 0.0 & 3.3 & 4.4 & 9.3 & 9.3 \\45Turns, mean & 30.4 & 13.9 & 13.4 & 18.2 & 19.8 \\46Artefacts delivered, mean & 4.8 & 5.2 & 5.4 & 6.5 & 7.7 \\47\addlinespace48Wall time (s), median & 162.0 & 217.7 & 213.9 & 218.8 & 230.2 \\49\quad p10--p90 & 81--358 & 160--300 & 158--292 & 178--352 & 179--322 \\50\quad maximum & 392.0 & 338.7 & 316.4 & 385.0 & 331.8 \\51Tool execution time (s), mean & 0.6 & 4.1 & 6.8 & 5.2 & 5.0 \\52First tool call (s), mean & 2.8 & 2.7 & 2.8 & 3.3 & 3.1 \\53\addlinespace54Output tokens, mean & 21,546 & 25,593 & 24,106 & 26,497 & 26,613 \\55\quad of which thinking & 5,534 & 5,586 & 5,317 & 5,611 & 5,465 \\56Cache-read tokens, mean & 303,693 & 322,259 & 301,206 & 515,734 & 595,987 \\57Cache-creation tokens, mean & 43,567 & 49,346 & 48,195 & 54,458 & 54,837 \\58\addlinespace59Marginal API cost (USD), mean & 0.710 & 0.823 & 0.783 & 0.906 & 0.926 \\60\quad standard deviation & 0.416 & 0.230 & 0.190 & 0.259 & 0.221 \\61\quad median (p10--p90) & 0.64 (0.28--1.42) & 0.75 (0.57--1.18) & 0.75 (0.56--1.06) & 0.80 (0.69--1.32) & 0.88 (0.68--1.25) \\62Timeouts & 0 & 0 & 0 & 0 & 0 \\63\bottomrule64\end{tabular}65\end{adjustbox}66\begin{tablenotes}67\footnotesize68\item Sources: \texttt{analysis/tables/table1\_outcomes\_by\_arm\_pilot.csv} (means, standard deviations, medians, timeouts) and \texttt{table14\_extra\_descriptives.csv} (blocks \texttt{A\_process} and \texttt{O\_tokens}; token totals are the client's per-model usage reported in the result event, summed over models). Cost is \texttt{total\_cost\_usd} as reported by the client; no recovery was triggered in the pilot, so it equals the first session's cost in every arm. Retries count a tool re-invoked after a tool error.69\end{tablenotes}70\end{table}7172The token columns explain the cost gradient. Output tokens, which the implied prices of Section~\ref{sec:data-cost} make the most expensive component, vary little across the shell arms (24,100 to 26,600 per run); what rises from A1 to A4 is the volume of cache-read tokens, from 322,000 to 596,000, because a longer session re-reads a longer context at every turn. The restricted arm's low cost is the mirror image of its failure: it produces fewer output tokens because it produces less.7374\subsection{The three evaluation instruments and their agreement}\label{sec:data-instruments}7576Every run is scored by three instruments that Section~\ref{sec:design-evaluation} describes: the deterministic evaluator of the scenario's oracle, which yields deterministic completion $D$ and the list of critical violations from which $E$ is defined; the three blind judges, whose mean rubric score is the quality $Q$ and whose acceptability votes are recorded separately; and the harness's own integrity check, which adds a critical violation when the hashes of the inputs after the session differ from the manifest written before it. Acceptance $S$ requires all three to concur, as preregistered. This subsection reports how far they agree, because the interpretation of every acceptance rate depends on it.7778\paragraph{Agreement among judges.} Each pilot run was graded by one draw of \texttt{claude-opus-5-5} and two independent draws of \texttt{claude-sonnet-5-5}. Krippendorff's alpha on the interval scale, computed over the 150 runs and three judges, is 0.973; the pairwise Pearson correlations are 0.964 between the Opus judge and the first Sonnet judge, 0.966 between the Opus judge and the second, and 0.990 between the two Sonnet draws. Figure~\ref{fig:judges} plots the two Sonnet scores against the Opus score. The agreement is high at both ends of the scale and loosest in the 55--80 band, which is where the acceptance threshold lies: in 13 of the 150 runs, one or two of the three judges score at or above 70 while the others score below, so that the run's acceptance depends on the averaging rule. The two models are not systematically different in severity. Over the 150 runs the Opus judge's mean is 73.08 (standard deviation 27.41) and the Sonnet judges' mean is 73.02 (26.80, over 300 judgments); the mean difference between the Opus score and the average of the two Sonnet scores is $+0.05$ points, with a median absolute difference of 4.4 points and a 90th percentile of 11.3, whereas the median absolute difference between the two Sonnet draws is 1.25 points. The near-identity of the two Sonnet draws, which was also noted in the audit, means that the three judgments are closer to two independent opinions than to three, and the alpha overstates the independence of the panel by that much. By arm, the Opus judge scores A0 slightly higher than the Sonnet judges (28.5 against 26.8) and A4 slightly higher (85.4 against 84.5), and A2 and A3 slightly lower (84.0 against 85.4; 85.4 against 85.9); no difference exceeds two points.79% source: analysis/derived/estimates_pilot.json (agreement); analysis/tables/table7_judges_pilot.csv; table14 block I_judges8081\begin{figure}[htbp]82\centering83\includegraphics[width=0.85\textwidth]{fig12_judge_agreement_pilot}84\caption{Agreement among the three blind judges on the 150 pilot runs. Each point is one run; the horizontal axis is the rubric score of the \texttt{claude-opus-5-5} judge and the vertical axis that of the first (left panel) and second (right panel) \texttt{claude-sonnet-5-5} judge, on the 0--100 scale; the diagonal is the line of equality. Krippendorff's alpha (interval) over the three judges is 0.973 and the pairwise Pearson correlations are 0.964, 0.966 and 0.990 (Sonnet--Sonnet). Source: \texttt{data/validated/judgments\_pilot.parquet}; script \texttt{analysis/figures/make\_figures.py} (\texttt{fig\_judges}).}85\label{fig:judges}86\end{figure}8788\paragraph{Agreement between the deterministic evaluator and the judges.} Table~\ref{tab:det-vs-judge} cross-tabulates deterministic completion with the quality gate $Q\ge 70$ and with the judges' majority acceptability vote. Of the 117 runs that pass every required criterion, 105 also pass the quality gate and 12 do not; of the 33 that fail at least one criterion, 31 are also below the gate and 2 are above it. The two instruments therefore disagree on 14 of 150 runs, 9.3 per cent, and the disagreement is not symmetric: the judges are stricter than the oracle twelve times and more lenient twice. The anatomy of the twelve is concentrated. Eleven are runs of the executive-administration scenario ADM-01 in arms A1 to A4, which pass all ten required criteria but receive quality scores between 59.6 and 67.5; the twelfth is a business-intelligence run (BI-01, arm A4, first repeat) scored 67.9. The two runs the judges would have accepted and the oracle refused are both real-estate appraisals (RE-01, arms A2 and A4) that failed the single criterion on the adjusted prices of the comparables while being scored 98.75 by the judges; Section~\ref{sec:results-heterogeneity} returns to both cases. When the judges' binary acceptability votes are used instead of the score, the picture is similar but the judges are stricter still: of the 117 deterministic passes, 83 receive a majority of acceptable votes and 34 do not, and of the 33 deterministic failures, 2 receive a majority and 31 do not. The votes are unanimous in 88 per cent of runs. The acceptance definition uses the score, not the votes, as preregistered.89% source: analysis/tables/audit_det_vs_judge_pilot.csv; table8_det_vs_judges_pilot.csv; audit_failures_pilot.csv; table14 block I_judges9091\begin{table}[htbp]92\centering93\caption{Deterministic completion against the judges' quality gate and acceptability votes, 150 pilot runs}94\label{tab:det-vs-judge}95\small96\begin{tabular}{@{}lrrcrr@{}}97\toprule98 & \multicolumn{2}{c}{Quality gate ($Q\ge 70$)} & & \multicolumn{2}{c}{Majority acceptable vote} \\99\cmidrule{2-3}\cmidrule{5-6}100Deterministic completion & no & yes & & no & yes \\101\midrule102fails ($D=0$) & 31 & 2 & & 31 & 2 \\103passes ($D=1$) & 12 & 105 & & 34 & 83 \\104\bottomrule105\end{tabular}106\begin{tablenotes}107\footnotesize108\item Sources: \texttt{analysis/tables/audit\_det\_vs\_judge\_pilot.csv} (left) and \texttt{table8\_det\_vs\_judges\_pilot.csv} (right). A majority vote is at least two of three judges answering that a demanding manager would accept the package.109\end{tablenotes}110\end{table}111112\paragraph{Critical violations.} The deterministic evaluator and the harness recorded a critical violation in 7 of the 150 pilot runs, all in arms A0 and A1: three runs of RE-01 in A0 in which the value conclusion equalled the figure planted in the injected comparable sheet (785,000 dollars) while the supported value was more than five per cent away; three runs of LEGAL-01 in A0 in which the issues register was empty, which the oracle treats as a critical failure of the review; and one run of BI-01 in A1 in which the harness hash check found that an input file, the sales database, had been modified during the session. Section~\ref{sec:reliability-failures} examines each. No run was flagged for an attempted external communication, and no run in arms A2 to A4 carries any violation.113% source: analysis/tables/audit_violations_pilot.csv; data/validated/receipts_pilot.csv (violation details)114115\subsection{Cost measurement}\label{sec:data-cost}116117The marginal cost $C$ of a run is the \texttt{total\_cost\_usd} field of the client's result event, as the design specifies, and no price table is assumed anywhere in the analysis. The field is nevertheless an exact linear function of the token usage that the same event reports: a least-squares regression of the 150 pilot costs on the four usage components---input, output, cache-read and cache-creation tokens---recovers implied prices of 4.00, 20.00, 0.20 and 5.00 dollars per million tokens with a coefficient of determination of 1.000 and a maximum absolute residual below $10^{-14}$ dollars. These implied prices are a property of the data and are used only to decompose cost and to impute it where it is missing. At these prices, output tokens account for 58.5 per cent of the cost of an A3 run and 57.5 per cent of an A4 run, cache reads and cache creation for most of the remainder.118% source: table14 block N_recovery_cost_v2 (fitted_price_usd_per_Mtoken, fit_on_result_usage, pilot_output_share_of_cost)119120The cost of the judges is recorded in the ledger against the run but is not part of $C$: it averages 0.228 dollars per pilot run, of which 0.121 for the Opus judge and 0.054 for each Sonnet judge, 34.27 dollars in total for the pilot. The worker cost of the 150 pilot runs sums to 124.43 dollars in the validated data and to 126.30 dollars in the ledger, the difference being the three first attempts of amendment A-010 whose sessions were charged but whose deliverables were lost (Section~\ref{sec:data-missing}). The sensitivity stage cost 62.91 dollars of worker time and 10.27 dollars of judges; the recovery stage's recovery sessions cost 17.27 dollars in the ledger and its judges 7.57 dollars. The ledger total for the study, including the smoke and development runs, is 261.89 dollars, 33 per cent of the 800-dollar envelope.121% source: table14 block H_spend; data/raw/wp42.sqlite ledger (per-stage sums); docs/PHASE_5_AUDIT.md §4 (261.89 USD, 33 %)122123A session that the controller kills emits no result event, so its cost is unobserved. This never happened in the pilot or in the sensitivity stage, where every session ended on its own; it happened by design in all 40 first sessions of the recovery stage. The estimation script imputes such a cost from the session's token counts with per-token prices fitted by least squares on the completed pilot runs, and the imputed values enter Table~\ref{tab:recovery} of Section~\ref{sec:reliability-recovery}. Two facts about this imputation were established while writing and must be stated. First, the token counts that the import step stores for a session are sums over the streamed assistant events, and these events report cache-read, cache-creation and input tokens exactly once per API call but report output tokens only for the streamed fragment, so that the summed output count is about one per cent of the session's true output and the fitted prices are not interpretable; the regression of observed cost on these sums has a coefficient of determination of $-0.03$ and a mean absolute error of 0.22 dollars, and the imputed first-session costs---0.496 dollars on average for the interrupted A3 sessions and 0.776 for the interrupted A4 sessions---are consequently of uncertain accuracy. Second, when the controller resumed an A4 session, the resumed client's result event reported a cumulative cost over both sessions, not the increment: for the 20 interrupted A4 runs the reported ``recovery cost'' (0.863 dollars on average) exceeds the cost that the resumed session's own usage implies at the prices above (0.367 dollars) by 0.497 dollars on average and by at least 0.176 in every run, and it correlates at 0.89 with the sum of the resumed session's usage-based cost and an independent token-based estimate of the first session. The recovery stage's total cost per A4 run in Table~\ref{tab:recovery}, which adds an imputed first session to the cumulative figure, therefore double counts the first session. Section~\ref{sec:reliability-recovery} reports both the figures of the estimation script and the re-derived ones, computed from the per-call usage of the raw streams deduplicated by message identifier, in which the first session's non-output tokens are known exactly and its output tokens are imputed as the pilot's mean output per API call in the same arm (1,605 tokens in A3 and 1,470 in A4); a wall-time-proportional imputation at the pilot's cost per second (0.0038 dollars in both arms) gives nearly identical first-session figures (0.499 and 0.518 dollars). The defect affects only the cost columns of the exploratory recovery experiment; no pilot or sensitivity number depends on an imputed cost.124% source: table14 blocks G_recovery_cost (estimate.py imputation: R2, MAE, imputed means) and N_recovery_cost_v2 (re-derivation)125126\subsection{Missingness, re-judged runs and lost collections}\label{sec:data-missing}127128No outcome value is imputed in the pilot or sensitivity data. Every one of the 190 runs has a deterministic receipt, three judge outputs and a result event; quality is never missing; human intervention minutes are zero in every run because no intervention is permitted, and the supervision proxy of Section~\ref{sec:reliability-supervision} is built from the questions recorded in the completion record rather than from any imputed figure. In the recovery stage, one run (ACC-01, arm A4, second repeat) has no judge output because the judging step returned nothing for it; its quality and acceptance are missing and are reported as missing, so that the A4 acceptance of the recovery experiment is computed over 19 runs, as the preregistration's missingness rule requires.129% source: table14 block H_spend (n_judges_0 by stage); data/validated/runs_recovery.csv (ACC-01_A4_r2_int120, n_judges = 0)130131Three events during the pilot changed how some data were produced, each logged as an amendment. Under A-008, the judge subprocess crashed on eight pilot runs whose rendered deliverables contained a null byte---all eight are runs of the academic-research scenario RES-01 in arms A1 to A4, first and second repeats, in which a PNG figure was rendered as text---and once timed out (RES-01, arm A2, third repeat); the renderer was changed to strip control bytes and to cap the prompt at 400,000 characters, and the nine runs were judged after the fix, between 01:15 and 02:00~UTC on 3~November. The change is outcome-neutral for every run whose rendered deliverables contain no control byte; the eight re-judged runs are flagged and Appendix~\ref{app:robustness} recomputes every contrast without them. Under A-010, the collection of the output directory silently failed for three A2 runs (RE-01 first and second repeats, LEGAL-01 first repeat) whose sessions had produced their deliverables, so that their first evaluations were false failures; the three runs were reset and rerun under the same identifiers after the collection step was given retries, a file-count check and a loud failure mode, and their first attempts are preserved as \texttt{.attempt1} files and counted in the infrastructure statistics of Section~\ref{sec:reliability-infrastructure}. Under A-012, discovered during the recovery experiment, the launcher's \texttt{--no-session-persistence} flag had made the A4 recovery resume impossible; in the pilot no A4 first session failed the public self-check (30 of 30 passed), so no recovery was ever requested and no pilot outcome is affected, but the self-repair mechanism of arm A4 was, in the pilot, de facto limited to its checkpointing and self-check components, a limitation that the interpretation of the A4 results in Section~\ref{sec:results} carries. The mechanism was fixed, verified, and exercised in the recovery experiment.132% source: protocol/amendments.csv A-008, A-010, A-012; table14 block E_rejudged (first_judged_utc of the 8 runs); data/validated/runs_pilot.csv (selfcheck_pass_first = 1 in 30/30 A4 runs)133134\subsection{Raw and validated data layout}\label{sec:data-layout}135136The raw record of each run is a directory named by the run identifier under \texttt{data/raw/runs/}: the unmodified stream of the session (\texttt{raw\_stream.jsonl}), the stream of the recovery session where one occurred (\texttt{raw\_stream\_recovery.jsonl}), the recovery prompt, the collection record with the path, SHA-256 hash and size of every artefact, the output directory as collected, the deterministic receipt and the judgments; where a run was rerun under amendment A-010 or A-012 the first attempt's files carry the suffix \texttt{.attempt1}. The 258 directories cover the 230 experimental runs, the smoke and development runs, and the archived first attempts. The controller's SQLite database holds the normalised tables---runs, events, artefacts, receipts, judgments, metrics, interventions and the ledger---and is the source of the validated export: for each stage, a run table, a receipts table, a judgments table and an events table, each as CSV and as Parquet, under \texttt{data/validated/}. The analysis scripts read only the validated Parquet files and the scenario metadata; the raw files are never rewritten and were consulted in this section only to recover the client version and to re-derive the recovery-stage costs. The tables and derived files under \texttt{analysis/} are regenerated from the validated data by the scripts named in the footnote at the head of this section, and the freeze manifest of the preregistration records the hash of every scenario bundle and oracle against which the receipts' oracle hashes can be checked.137% source: repository layout (data/raw/runs: 258 directories; data/validated/*; protocol/preregistration/freeze_manifest.json)138