SPB Git

spb/wp42_uqo

Public
4.7 KB · 14 lines csv
1id,utc,phase,decision,reason,status2A-001,2026-11-02T23:40:00Z,0,Study envelope 800 USD; 4.00 USD per session; 1800 s wall; 150 turns,The mandate gives no budget; same envelope as WP29; PI may change it,pending PI confirmation3A-002,2026-11-02T23:40:00Z,0,Human arms H0/H1 designed but not executed,No participants nor ethics approval exist; the mandate forbids LLM imitations of humans,pending PI confirmation4A-003,2026-11-02T23:40:00Z,0,A2 = headless macOS automation layer (osascript/System Events/macOS utilities) without GUI app control,GUI Apple Events time out over SSH (TCC consent cannot be answered on headless nodes),recorded5A-004,2026-11-02T23:40:00Z,0,Primary worker model claude-opus-5-5; Sonnet 5.5 and Fable 5.1 as sensitivities on A3,Cost: Fable 2.5x Opus per token in the probe; Opus was the WP29 subject,recorded6A-005,2026-11-02T23:40:00Z,0,Blind professional evaluators are LLM judges (3 per run); human expert review left to the PI,No human evaluators available in-session,recorded7A-006,2026-11-03T01:05:00Z,2,Per-scenario caps = max(global 4 USD / 1800 s; task.json budget_usd / time_limit_s) — ACC-01 (workflow 150 invoices) 6 USD / 3600 s; RES-01 6 USD / 2700 s; BANK-01 5 USD / 2400 s; LEGAL-01 2400 s; A4 recovery gets the remainder,Scenario authors sized the caps to the designed workload; set before any pilot run and before the freeze,recorded8A-007,2026-11-03T02:40:00Z,3,Freeze v1.0 of the 10 validated scenarios + oracles + harness (protocol/preregistration/freeze_manifest.json) after validator fixes (evaluators 1.1; RE-01/HR-01/RES-01/LEGAL-01/ADM-01 bundles regenerated at the same seed),All 10 validator solutions pass their evaluator and empty outputs fail (scripts check at freeze); no pilot run executed before this freeze except the dev run HR-01_A3_r1 on the pre-validation HR-01 bundle (stage dev; excluded),recorded9A-008,2026-11-03T04:10:00Z,4,Judge prompt sanitisation (NUL/control bytes removed; 400k-char cap) applied after 8 pilot runs crashed the judge subprocess (ValueError embedded null byte); those 8 runs (all RES-01) were judged after the fix,Harness defect found during the pilot; the change is outcome-neutral for every run whose rendered deliverables contain no NUL byte; the 8 re-judged runs are flagged for a sensitivity analysis,recorded10A-009,2026-11-03T04:15:00Z,4,Scale decision per preregistration §7: pilot mean total cost per run 1.07 USD (worker + judges) → add R=3 for all 50 cells (stage pilot; seed 4243) and run the model sensitivities Sonnet 5.5 and Fable 5.1 on arm A3 × 10 scenarios × 2 repeats (stage sensitivity),Rule satisfied: 1.15×1.07×50 = 62 USD ≤ 800−107−100; sensitivities 1.15×1.07×40 = 49 USD ≤ remaining−50; no scenario/oracle/arm change since the freeze (verify_freeze: only judge.py per A-008),recorded11A-010,2026-11-03T04:50:00Z,5,Collection defect: rsync of out/ silently failed for 3 pilot runs (RE-01_A2_r1 M4M64a; RE-01_A2_r2 M3U96a; LEGAL-01_A2_r1 M4M64a) whose sessions had produced the deliverables (events show out/ listed) — infrastructure error under §8; the 3 runs are reset and rerun with the same ids after the fix (rsync/ssh retries with backoff; file-count check; loud failure → infra_error),Likely sshd MaxStartups throttling under 12 concurrent sessions + collection traffic; the lost runs' original attempts are kept in the infrastructure-stability statistics (events preserved in data/raw/runs/<id>/raw_stream.jsonl.attempt1),recorded12A-011,2026-11-03T04:55:00Z,5,Exploratory addendum (not in the preregistration): Autonomous Recovery experiment — 10 scenarios × {A3 interrupted at 120 s without recovery; A4 interrupted at 120 s with its bounded recovery resume} × 2 repeats (stage recovery; seed 4246; Opus). ARI = P(final public self-check pass | interrupted) in A4; acceptance after recovery compared with the A3 interrupted control and with the uninterrupted A3/A4 cells,The pilot's A4 arm never triggered a recovery (20/20 first sessions passed the public self-check) so the ARI is unidentified in the main design; an induced interruption identifies it. Labelled exploratory in the manuscript,recorded13A-012,2026-11-03T06:05:00Z,5,Harness defect: the launcher passed --no-session-persistence so the A4 bounded recovery (--resume) could never start (result error_during_execution in 0 ms). Fixed; verified with an interrupted Haiku run that resumed and passed the self-check. All A4 runs of the recovery stage (first attempt) are reset and rerun; A3 interrupted runs that completed are kept,In the pilot (R=3) no A4 first session failed the public self-check so no recovery was ever requested: pilot A4 outcomes are unaffected but the A4 "self-repair" mechanism was de facto limited to checkpointing + self-check in the pilot. Stated in the manuscript,recorded14
Files · c1985c356f0ac4e60242a79d67a2c60f24936c2e
Menu

Theme