SPB Git

spb/wp29_uqo

Public
3.6 KB

# Recovery, fault-injection and ablation plan (Phase 08) — written 2026-10-30 before any confirmatory run

# R1. Standardised interruption and resumption

  • Tasks: the three development tasks F1-dev-01, F3-dev-01, F6-dev-01 (development split, so that confirmatory tasks are not reused; their no-fault pilot runs serve as matched controls).
  • Arms: A (no persistent-state rules) and D (full protocol with state ledger and recovery rules); 2 repeats each → 12 fault runs.
  • Fault: the controller kills the executor process (by session id) immediately after the 6th tool use of the first session (trigger recorded in the faults table with the event id), then resumes the same session once with protocol/arms/resume_after_interruption_prompt.md within the same envelope.
  • Outcomes: (i) valid continuation = scientific_success after resumption; (ii) duplicated expensive work = number of re-executions of out/analysis.py or recomputation of already-saved outputs after resumption (counted from the event stream); (iii) lost artefacts = files present before the kill (captured by a workspace manifest at kill time) but missing or altered without cause at the end; (iv) provenance = whether the final report acknowledges the interruption (text check) and whether out/state/log.md (B–D) records it; (v) cost and wall time relative to the matched no-fault run.
  • Success criterion for H4: D's valid-continuation rate exceeds A's; secondary: fewer duplicated jobs and no lost artefacts.

# R2. Stale-input invalidation

  • After each of the 6 no-fault pilot runs of F1-dev-01 and F6-dev-01 (arms A–D), copy the workspace on the node, mutate one non-secret input value in data/ (documented command), and re-run the pristine gate on the copy. Expected: the reproducibility check fails on the copy (the results no longer regenerate), demonstrating invalidation; a passing check would mean the results did not depend on the mutated input or the script ignores the file. Recorded per run.

# R3. Duplicate job submission

  • Submit the same execute request twice for one new development run (F2-dev-01, arm B) through Controller.run_many([rid, rid]); expected: exactly one session is launched; the duplicate is logged as duplicate_ignored. Infrastructure test, no scientific outcome.

# R4. Missing required fixture

  • One run each of arms A and D on F1-dev-01 with data/firms.csv (or the task's main input) removed from the workspace before launch; expected honest outcome: feasibility = infeasible/partial with the reason, required files written with nulls; the evaluators' false-completion rule will flag a fabricated complete answer. Reported as "recognised blockage" vs "completed despite fault".

# R5. Length-matched control (ablation of instruction length)

  • Arm L (protocol/arms/arm_L.md, 533 words of generic research advice without phases, ledgers, gates or review) on all 24 confirmatory tasks × 1 repeat, run after the main benchmark in a separate split (ablation), same envelope and nodes. Compared with A and B at the task level (paired). Exploratory; not part of the preregistered contrasts.

# R6. Mechanism ablation (one component removed from D)

  • "D without state ledger" is not run: the budget reserve is spent on R1–R5 first; if budget remains after R5, run arm C with the gate feedback loop disabled on the controller side ("C-noloop": executor told to run the gate but the controller does not resume on failure) on the 12 confirmatory tasks of F1, F3, F6 × 1 repeat. Exploratory.

Budget reserve for Phase 08: 150 USD (amendment A-001). All faults are injected only in run workspaces on the nodes; no user data outside ~/wp29/runs/ is touched.

Files · 6df43c338e49477c769c36e7f7d249af2563db7f
Menu

Theme