# Evaluation plan (frozen for the pilot)

# Final evaluation (common to all arms, evaluator host only)

1. Harness collects out/ from the node, runs the common contract check (the same gate program, post hoc, never fed back to arms A/B) which also provides the fresh re-run reproducibility result (analysis.py re-executed in a clean copy on the node with the pinned venv).
2. ~/wp29-oracles/<task_id>/evaluate.py (deterministic, no LLM) produces the receipt: required criteria (task-specific, tolerances justified in REFERENCE.md), critical violations (leakage, fabrication, invalid_inference, unreported_substitution, false_completion, provenance), quality components, scientific_success = all criteria ∧ no violation.
3. Quality score = 0.40·correctness + 0.20·integrity + 0.15·reproducibility + 0.15·reporting + 0.10·baseline (weights frozen 2026-10-30 before the pilot). Components are always reported separately.
4. Receipts are validated against evaluators/public_evaluation_interface/receipt_schema.json, hashed and stored (evaluations table + runs/<split>/<run>/evaluation_receipt.json).

# Primary endpoint

Y_iar = 1{scientific_success ∧ status = completed (within the envelope) ∧ no human intervention}. Timeout/budget-exhausted runs are evaluated and reported but Y = 0. Infrastructure errors (error status with no inference) follow the balanced rerun policy (attempt-numbered rerun of the same assignment; the original attempt stays in the ledger) and are reported separately.

# Secondary endpoints

As listed in the mandate §7.2, each computed from the receipt/manifest tables: correct completion ignoring the envelope; quality components; violation counts by type; unsupported claims (claims with status supported whose results key fails its criterion — computed where the oracle maps claims to keys); reproducibility; correct detection of infeasible/null-truth tasks (criterion in those tasks' oracles) and false blocking on feasible tasks (feasibility ≠ complete on a feasible task = critical false_completion-class violation "false_blocking", recorded by the evaluators as false_completion with a description — reported separately in the analysis); recovery outcomes (Phase 08); latency, usage, cost (client estimate), human minutes.

# Evaluator validation

Every oracle ships validation/validate.py with ≥1 correct_out and ≥2 wrong_out artefacts; results are aggregated into reports/evaluator_validation.md (false acceptance / false rejection per task on the seeded artefacts). The smoke evaluator is additionally unit-tested (tests/unit/test_gate_and_evaluator.py).

# Model-based grading (supplementary, not primary)

Not used for the primary endpoint. If used in Phase 09 for report quality, the grader model, prompt, rubric and its own validation on seeded artefacts will be reported; same-family dependence will be stated.

# Blinding

Evaluators are deterministic programs and do not see arm labels; run ids are neutral (RD0001…). Artefacts can reveal the arm (e.g. out/state/phases.json exists only in B/C/D) — irrelevant for deterministic evaluators, relevant for any human/LLM review (documented as imperfect blinding).

# Human involvement

No human reviewer is available for the benchmark; those cells are "unavailable". Human interventions are logged in reports/interventions.csv (and the interventions table). Historical preparation time of the mandate is unknown and is reported as unknown.