1# Evaluation plan (frozen for the pilot)23## Final evaluation (common to all arms, evaluator host only)41. Harness collects `out/` from the node, runs the **common contract check** (the same gate program, post hoc, never fed back to arms A/B) which also provides the **fresh re-run reproducibility** result (analysis.py re-executed in a clean copy on the node with the pinned venv).52. `~/wp29-oracles/<task_id>/evaluate.py` (deterministic, no LLM) produces the receipt: required criteria (task-specific, tolerances justified in REFERENCE.md), critical violations (leakage, fabrication, invalid_inference, unreported_substitution, false_completion, provenance), quality components, `scientific_success` = all criteria ∧ no violation.63. Quality score = 0.40·correctness + 0.20·integrity + 0.15·reproducibility + 0.15·reporting + 0.10·baseline (weights frozen 2026-10-30 before the pilot). Components are always reported separately.74. Receipts are validated against `evaluators/public_evaluation_interface/receipt_schema.json`, hashed and stored (`evaluations` table + `runs/<split>/<run>/evaluation_receipt.json`).89## Primary endpoint10Y_iar = 1{scientific_success ∧ status = completed (within the envelope) ∧ no human intervention}. Timeout/budget-exhausted runs are evaluated and reported but Y = 0. Infrastructure errors (`error` status with no inference) follow the balanced rerun policy (attempt-numbered rerun of the same assignment; the original attempt stays in the ledger) and are reported separately.1112## Secondary endpoints13As listed in the mandate §7.2, each computed from the receipt/manifest tables: correct completion ignoring the envelope; quality components; violation counts by type; unsupported claims (claims with status `supported` whose results key fails its criterion — computed where the oracle maps claims to keys); reproducibility; correct detection of infeasible/null-truth tasks (criterion in those tasks' oracles) and false blocking on feasible tasks (`feasibility ≠ complete` on a feasible task = critical `false_completion`-class violation "false_blocking", recorded by the evaluators as `false_completion` with a description — reported separately in the analysis); recovery outcomes (Phase 08); latency, usage, cost (client estimate), human minutes.1415## Evaluator validation16Every oracle ships `validation/validate.py` with ≥1 correct_out and ≥2 wrong_out artefacts; results are aggregated into `reports/evaluator_validation.md` (false acceptance / false rejection per task on the seeded artefacts). The smoke evaluator is additionally unit-tested (`tests/unit/test_gate_and_evaluator.py`).1718## Model-based grading (supplementary, not primary)19Not used for the primary endpoint. If used in Phase 09 for report quality, the grader model, prompt, rubric and its own validation on seeded artefacts will be reported; same-family dependence will be stated.2021## Blinding22Evaluators are deterministic programs and do not see arm labels; run ids are neutral (`RD0001`…). Artefacts can reveal the arm (e.g. `out/state/phases.json` exists only in B/C/D) — irrelevant for deterministic evaluators, relevant for any human/LLM review (documented as imperfect blinding).2324## Human involvement25No human reviewer is available for the benchmark; those cells are "unavailable". Human interventions are logged in `reports/interventions.csv` (and the `interventions` table). Historical preparation time of the mandate is unknown and is reported as unknown.26