SPB Git

spb/wp29_uqo

Public
9.1 KB

# Phase status — WP29

Append-only log. Times are UTC unless stated. "Executor" = the Claude Code session (claude-fable-5-1, client 2.1.314) operated by S.-P. Boucher.

# Phase 00 — Inspect identity, sources and environment

  • Started 2026-10-29 ≈ 02:40 UTC (2026-10-29 22:40 local). Closed 2026-10-29 ≈ 03:30 UTC.
  • Work performed: read series index (Recherche-UQO/README.md, HARMONISATION.md) → no WP29 collision; created repository skeleton; preserved the full mandate verbatim in protocol/WP29_FULL_MANDATE.md; retrieved 13 corpus files with SHA-256 (research/corpus_manifest.csv); downloaded 5 official Claude Code doc pages; probed claude --version/--help/auth status; ran 4 live sentinel/capability probes (laptop, bare vs normal mode, append-system-prompt, node M3U96a); aligned 4 executor nodes to client 2.1.314 + pinned Python venv; wrote reports/environment_audit.md.
  • Inputs: mandate pasted by the researcher (2026-10-29); client 2.1.314; docs retrieved 2026-10-29.
  • Outputs: reports/environment_audit.md, research/corpus_manifest.csv, research/corpus/*, docs/claude-code-docs/*, protocol/WP29_FULL_MANDATE.md.
  • Checks executed: HTTP status + hash per corpus file; bare-mode sentinel probes (pass: bare loads 144 tokens, normal mode 49k tokens incl. user CLAUDE.md); node→laptop SSH unreachable (boundary holds).
  • Costs: probes ≈ 0.0016 USD of API inference (haiku/opus probes); no human minutes measured beyond the pasting of the mandate.
  • Interventions: see reports/interventions.csv (I-001: mandate preparation and pasting by the researcher, duration unknown).
  • Unresolved: packaged artefacts absent (replaced by live retrieval); budget envelope set provisionally by the executor (Phase 02).
  • Gate: identity protected ✔; capabilities vs missing inputs distinguished ✔; no credentials exposed ✔. PASSED.
  • Next eligible: Phase 01 (literature + corpus coding, launched as two background subagents at ≈ 03:25 UTC) and Phase 02 (protocol, tasks, evaluators) in parallel.

# Phase 01 — Literature and scientific blueprint (in progress → closed with Phase 02 artefacts)

  • Started 2026-10-29 ≈ 03:25 UTC. Corpus coding completed ≈ 03:45 UTC by a Claude Code subagent (research/corpus_coding_protocol.md, code_corpus.py, corpus_automated.csv, corpus_coding.csv, corpus_summary.md; single coder, no intercoder reliability). Literature review delegated to a second subagent (still running at the time of this entry; outputs research/literature_matrix.csv, novelty_matrix.csv, source_notes.md, references_verified.bib).
  • Blueprint: protocol/scientific_contract.md, protocol/treatment_definitions.md, protocol/evaluation_plan.md (this phase and Phase 02 overlap by design: the smallest executable study was built and smoke-tested before the literature files landed).
  • Gate: closest prior work identified — pending the literature subagent's report; novelty qualified in research/source_notes.md when it lands. PARTIALLY PASSED (blueprint ✔, literature pending).

# Phase 02 — Protocol, task suite and evaluator specification

  • Started 2026-10-30 ≈ 03:05 UTC. Wrote tasks/TASK_AUTHORING_SPEC.md (frozen interface v1), evaluators/public_evaluation_interface/receipt_schema.json, the four arm texts + reviewer + feedback prompts (protocol/arms/), configs/*.yaml, protocol/screenshot_protocol.md, the smoke task F2-smoke-01 (bundle, card, hidden oracle with independent numpy reference), and the family brief tasks/FAMILY_AGENT_BRIEF.md; six task-family subagents launched ≈ 03:20 UTC (1 dev + 4 conf tasks each, with hidden oracles and validation sets).
  • Checks: content-equivalence audit of arm texts (no task facts in protocols); endpoint/estimand definitions written before any pilot run.
  • Gate: information matched across arms ✔; success/failure definitions frozen for the pilot ✔ (protocol/evaluation_plan.md); hidden-answer protection plan ✔ (reports/isolation_audit.md); negative/infeasible cases required in F5/F6 briefs ✔ (verification when the agents report). PASSED for the pilot design.

# Phase 03 — Minimal harness

  • Started 2026-10-30 ≈ 03:10 UTC. Implemented src/pgar/{contracts,state,events,launcher,isolation,provenance,budgets,runtime_gates,recovery,scheduler,evaluation,exports,cli}.py (≈1,300 lines, standard library + numpy/pandas/jsonschema/yaml on the controller; the gate is standard-library only). Deployed the gate (sha256 1760d13e…) and a mode-600 key file on the four nodes; pinned venvs; make test: 7 infrastructure tests pass (state machine, idempotent job ledger, invariant "no evaluated without receipt", event normalisation, gate accepts a correct artefact and rejects three planted defects, smoke evaluator accepts correct_out and rejects a wrong-SE/fabricated-count artefact). The bundle-hygiene test is pending until all task agents finish.
  • Failures fixed: (i) sqlite3 connection shared across threads returned a spurious KeyError → all reads serialised behind the store lock; (ii) shlex.quote("~/…") prevented tilde expansion on nodes → absolute remote root; (iii) behaviour audit false positives from truncated summaries → audit now parses the verbatim stream with a path regex.
  • Gate: harness rejects known defects ✔ (tests); worker access to hidden material prevented by tested controls ✔ (reports/isolation_audit.md, sentinel probes on all nodes). PASSED.

# Phase 04 — Capture system and evidence viewer

  • Implemented src/pgar/evidence_viewer.py (read-only HTTP dashboard over runs/state.sqlite, port 8765) and src/pgar/capture.py (Quartz window lookup + screencapture -l, manifest in the screenshots table). Captures so far: S002 native_live Terminal window tailing the live stream of RD0002 (arm D) during the smoke batch; S003 native_live Terminal window of the builder session itself (incidental, kept for the reflexive trace); S001 dashboard capture failed during the batch (Safari window off-screen; screencapture could not create the image) — a post-run viewer capture will be taken and labelled viewer_postrun. No synthetic imagery exists.
  • Gate: ≥1 genuine capture linked to an actual event ✔ (S002 ↔ RD0002 session b425e9f5…); no synthetic UI ✔; public evidence not yet published (originals in evidence/restricted/, to be inspected before copying). PASSED (with the recorded capture failure).

# Phase 05 — Real smoke experiment

  • Executed 2026-10-30 03:30–03:36 UTC: assignment seed 101 → RD0001 (A, M3U96a), RD0002 (D, M3U96b), RD0003 (C, M4M64a), RD0004 (B, M4M64b), task F2-smoke-01, model claude-opus-5-5 (reported by the client in modelUsage), client 2.1.314. First attempt: RD0003/RD0004 failed at workspace verification (tilde bug, no inference) and RD0001/RD0002 hit the thread-safety bug before launch — all four recorded in the ledger (run_transitions) and rerun under the attempt-numbered rerun policy after the fixes.
  • Outcomes (all live): 4 of 4 executor sessions finished (result.subtype = success); scientific_success = 1 in all four arms (5/5 criteria, 0 violations, reproducible). Costs (client estimate, USD): A 0.120 (6 turns, 53 s), B 0.311 (11 turns, 103 s), C 0.237 (7 turns, 85 s; gate run 3× by the executor, controller gate passed at cycle 1), D 0.418 = executor 0.303 + reviewer 0.114 (11 + 5 turns; reviewer verdict accept with 2 minor issues; no repair cycle). First-turn input tokens A 2,238 / B 3,717 / C 4,111 / D 4,319.
  • Checks: workspace manifests verified identical before each launch; behaviour audit: 0 outside-workspace references in all arms; gate invocations by executors: A 0, B 0, C 3, D 4; transcripts collected (runs/development/RD000x/sessions/*.jsonl).
  • Gate: four genuine task executions accounted for incl. the four infrastructure failures ✔; isolation and treatment delivery verified ✔ (effective-context hashes differ by arm only through the protocol text; token counts consistent); results are development observations ✔. PASSED. Next: pilot on the six development tasks when the family agents deliver them.

# Phase 01 — closure (2026-10-30 ≈ 04:10 UTC)

  • Literature subagent delivered research/literature_matrix.csv (84 verified rows, 8 streams), novelty_matrix.csv (18 systems/benchmarks), source_notes.md, references_verified.bib (84 entries, compiled with aer.bst, 0 bibtex errors), build_literature_matrix.py + verify_cache/ (raw Crossref/arXiv/Semantic Scholar/OpenReview responses). Two unverifiable items dropped and recorded; two DOIs corrected during verification.
  • Prior-use check: the phrase "protocol-guided autonomous research" and the acronym PGAR have no prior use in this field; closest concurrent work is XScientist (arXiv:2607.12301, a protocol artefact without controlled comparison) and data-to-paper (NEJM AI 2025).
  • Gate: closest prior work identified ✔; novelty qualified (the contribution is the controlled comparison of instruction regimes with a fixed, isolated executor and protected oracles, not the idea of protocols) ✔; the study tests the method rather than assuming it works ✔. PASSED.
Files · 32c4ccff60dd76c81da4f5b63d24bd0b55824e43
Menu

Theme