Scientific delegation contract — WP29 (PGAR)
A PGAR protocol is a tuple P = (Q, Θ, D, A, G, V, B, R, L). This document states how each component is serialised and used in the implemented harness (src/pgar/). Nothing here is a theorem; the invariants below are workflow guarantees enforced by code, not guarantees of scientific truth.
| Component | Meaning | Serialisation in WP29 | Used by |
|---|---|---|---|
| Q — questions and scope | the scientific question and estimand(s) of a task | TASK.md §Scientific question (public bundle); for WP29 itself: protocol/preregistration.md §1 |
executor (all arms); evaluator |
| Θ — estimands, hypotheses, assumptions | what is estimated, under which assumptions, which interpretations are admissible | TASK.md §Scientific question + §Information-availability rules; oracle REFERENCE.md (hidden) |
executor; evaluator |
| D — data, versions, availability | inputs, provenance, hashes, information-availability rule | data/README.md + bundle.json (SHA-256 of every input), TASK.md §Inputs/§Information-availability rules |
executor; harness (workspace verification); evaluator |
| A — authorised actions and tools | the tool inventory and permission policy | launcher flags (src/pgar/launcher.py::build_command): --bare --tools Bash,Read,Write,Edit,Glob,Grep --allowedTools … --permission-prompts none, offline nodes, pinned venv |
harness (identical across arms) |
| G — phase graph | ordered phases with dependencies | protocol text (arms B/C/D: P0→P1→P2→P3→P4→P5) + the executor-maintained ledger out/state/phases.json; harness-level run state machine contracts.RUN_TRANSITIONS |
executor (B/C/D); gate (C/D: phase_ledger_consistent) |
| V — verification criteria and evaluators | runtime (oracle-free) and final (hidden-oracle) checks | gate_spec.json + pgar_gate.py (runtime, C/D + post hoc for all); ~/wp29-oracles/<task>/evaluate.py + oracle.json (final); receipt schema evaluators/public_evaluation_interface/receipt_schema.json |
harness; evaluator host only |
| B — budgets and stopping | USD cap, wall clock, turns, gate/review cycles | configs/budgets.yaml → Envelope (src/pgar/budgets.py); enforced by --max-budget-usd, --max-turns, controller timeout + remote kill |
harness (identical envelope across arms; reviewer draws from the same envelope) |
| R — recovery, amendment, escalation | what happens after interruption, failure or a changed input | protocol text (B/C/D recovery rules) + harness resume (--resume <session>), rerun-attempt ledger, protocol/amendments.csv |
executor; harness; researcher |
| L — evidence and provenance records | run manifest, event stream, receipts, transcripts, screenshots, interventions | SQLite runs/state.sqlite (tables runs, run_transitions, sessions, events, gate_receipts, reviews, evaluations, screenshots, faults, jobs, interventions) + per-run directory runs/<split>/<run_id>/ (raw_stream.jsonl, sessions/*.jsonl, out/, ctl/, prepare.json, artifact_manifest.json, contract_receipt.json, evaluation_receipt.json, behaviour_audit.json) |
everything downstream |
States and transitions
Run states (harness, contracts.RUN_TRANSITIONS): not_started → prepared → running → {completed, timeout, budget_exhausted, error, interrupted}; interrupted → running (resume after a standardised fault); {completed, timeout, budget_exhausted, error} → evaluated; error → prepared (balanced rerun policy for infrastructure errors, attempt-numbered); void for assignments withdrawn before any inference. Phase states (protocol, executor-maintained): pending, running, passed, failed, blocked, stale, skipped_optional.
Invariants actually enforced (and their limits)
- No
evaluatedwithout a receipt —StateStore.transitionrefuses the transition unlessevaluation_receiptis set (testtests/unit/test_state.py). Limit: it checks existence, not the scientific validity of the receipt. - Changed upstream input invalidates downstream results — the gate's reproducibility check re-runs
out/analysis.pyon the currentdata/and compares without/results.json; a mutated input makes the regeneration differ (recovery.stale_input_check). Limit: only detects dependence through the replication script; hand-typed numbers are detected as non-reproducible, not as "stale". - Envelope — each session is launched with
--max-budget-usd = remainingand the controller kills the remote process at the wall limit; continuation requiresremaining ≥ 0.25 USDand≥ 120 s. Limit: the client-side cost estimate can overshoot the cap slightly (documented by Anthropic); the overshoot is recorded, not hidden. - No "verified" claim without evidence — executors must list claims in
out/claims.jsonwithresults_keysthat exist (gate checkclaims_json); final evaluators score claims against hidden references. Limit: the gate checks referential integrity, not truth. - Skipped optional phases cannot support claims —
phase_ledger_consistentrequires every non-terminal phase to bepassed,blockedorskipped_optional, and apassedphase to carry ≥1 recorded check; the final evaluator ignores phase ledgers entirely (it scores artefacts), so a skipped phase cannot "pass" a criterion. Limit: an executor can write a check string without running it — the reproducibility check and the final oracle are the real safeguards. - Idempotent jobs — every execute/session job is logged in the
jobstable before submission; a duplicate key is rejected (test_job_ledger_idempotent; stress test in Phase 08). - Nested-invocation guard — executors have no
Agenttool; the launcher refuses to start ifPGAR_DEPTHis set; executors run withPGAR_DEPTH=1on another host.
Observable autonomy (preserved in the database)
For each run: status/terminal_reason (workflow autonomy: did the executor finish without operational help?), evaluations.scientific_success (scientific completion), human_intervention_ids in the manifest (empty ⇒ intervention-free). The primary endpoint Y = scientific_success ∧ no critical violation ∧ no intervention ∧ within envelope (status ∈ {completed}; a timeout/budget_exhausted run keeps its artefacts and is evaluated, but Y = 0 by construction because the envelope was exceeded). "Autonomous but wrong" = completed ∧ ¬success; "assisted correct" = success ∧ intervention present (expected to be empty in the benchmark: no human intervened in any executor session).