SPB Git

spb/wp29_uqo

Public
3.0 KB · 6 lines csv
1amendment_id,timestamp_utc,phase,before_or_after_results,document,description,rationale,previous_version_ref2A-001,2026-10-30T03:00:00Z,02,before_results,configs/budgets.yaml,"Provisional live-inference envelope set by the executor: per benchmark run 4.00 USD / 1500 s / 80 turns per session; per transfer run 12.00 USD / 3600 s; total study envelope 800 USD (pilot + main + transfer + recovery + ablations) with the confirmatory scale to be chosen after the pilot.","The mandate requires a precise budget decision but none was stated; the researcher is not available mid-run; prior UQO studies spent ≈1,079 USD of API inference (WP26), so 800 USD is a conservative envelope. The researcher may raise or lower it; any change will be a new amendment.",n/a3A-002,2026-10-30T04:40:00Z,06,before_confirmatory_results,src/pgar/evaluation.py,"Final evaluators are now executed on a composed workspace (public bundle + the run's collected out/) instead of the run directory alone, which lacked data/ and made source-grounded checks (F4 'numbers_in_sources') fail for correct answers. All pilot runs evaluated before the fix are re-evaluated with the same oracles; original receipts are kept in runs/pilot/<run>/evaluation_receipt_pre-A002.json.","Pilot diagnosis: F4-dev-01 failed in all four arms with a spurious fabrication violation although every required criterion passed. Harness defect, not an oracle change.",runs/pilot/*/evaluation_receipt_pre-A002.json4A-003,2026-10-30T04:02:39Z,06,before_confirmatory_results,protocol/preregistration.md,"Preregistration frozen as v1.0 (adds §10 pilot-informed decisions: confirmatory design 24 tasks × 4 arms × 3 repeats = 288 runs, seed 301); freeze manifest protocol/freeze_manifest_conf.json records SHA-256 of all 24 confirmatory bundles and oracles, arm texts, gate, launcher, scheduler, evaluator, configs and analysis scripts; tasks/registry.csv stamped frozen_utc.","Scale decision per §6 of the preregistration using measured pilot costs; no hypothesis, endpoint or analysis rule changed.",git 2c826565A-004,2026-10-30T04:40:00Z,07,during_confirmatory_runs_before_any_analysis,src/pgar/scheduler.py,"Harness defect: continuation sessions (gate-feedback or review-repair resumptions) were inserted into the sessions ledger with the same client session id as the initial session, violating the primary key and aborting the run as an infrastructure error after the reviewer had returned 'revise' (first occurrence RC0094, F4-conf-02 arm D, 04:33 UTC). Fix: ledger rows keyed per launch (session_id#seq) with the client session id kept in the command record. Affected runs keep status 'error' (attempt 1, cost charged to the accounting) and are re-executed under the attempt-numbered rerun policy with the fixed harness after the batch; the fix does not alter any treatment, task, oracle or analysis rule.","Infrastructure outage policy of the preregistration (§7): balanced reruns of infrastructure errors; the error affects only runs needing a continuation (arm C/D), so without the rerun the comparison would be biased against C/D.",git 142b6886
Files · 502400bef152b80194d2a72e83992531749d18e1
Menu

Theme