# Task authoring specification (WP29 benchmark) — frozen interface v1

Every benchmark task is a bounded scientific task that a fresh, offline Claude Code session must solve inside a workspace that contains only the public bundle. Hidden reference material lives on the evaluator host under ~/wp29-oracles/<task_id>/ and is never copied to executor nodes.

# 1. Identifiers and splits

• task_id = <family>-<split>-<nn> with family ∈ {F1,…,F6}, split ∈ {dev, conf}, nn two digits (e.g. F3-dev-01, F3-conf-02). Smoke task: F2-smoke-01. Transfer projects: T1-forecast, T2-simulation, T3-synthesis (split transfer).
• Development/pilot tasks (dev) are used for the smoke test and the pilot and may be inspected freely. Confirmatory tasks (conf) are frozen (hash recorded) before any confirmatory run and are never used for tuning.

# 2. Public bundle — tasks/public_bundles/<task_id>/

TASK.md            # the complete task statement (see §3)
data/…             # inputs (CSV/JSON/TXT/PDF-extracted text); real data carry a data/README.md with source URL, retrieval UTC, license, SHA-256
gate_spec.json     # oracle-free output contract used by the gate program (see §5)
bundle.json        # {"task_id","family","split","version","created_utc","files":{path:sha256}}

Nothing in the bundle may reveal planted defects, reference values, or the evaluation logic.

# 3. TASK.md template (all sections mandatory, in this order)

1. # <Task title>
2. ## Scientific question — one paragraph; name the estimand(s) precisely.
3. ## Inputs — one bullet per file under data/: columns/fields, units, period, provenance (for real data: source, URL, retrieval date, license; for synthetic data: "simulated data" — say so, never pretend it is observed).
4. ## Information-availability rules — e.g. "Forecasts for origin t may use only observations dated ≤ t", "The column x is not usable because …" — only rules a careful scientist needs; never describe planted traps.
5. ## Required outputs — exact list:
   • out/results.json with a table of keys: key, type (number / integer / string / boolean / array / object), unit, meaning; the common keys (§4) must be listed too;
   • out/report.md with the six sections Question; Data and exclusions; Method; Results; Uncertainty; Limitations and open issues;
   • out/analysis.py — replication entry point: ~/wp29/venv/bin/python out/analysis.py run from the workspace root must regenerate out/results.json from data/ only, deterministically;
   • out/claims.json — list of {"id","text","results_keys","status","caveat"} (status ∈ supported / unsupported / not_evaluated) — required in every arm (it is an output obligation, not a treatment);
   • any task-specific figure/table (e.g. out/figure1.png).
6. ## Constraints — offline; Python path; "do not modify data/"; time guidance ("this task is designed to be completed in well under 20 minutes of work").

Tasks may be deliberately infeasible or contain a null-result truth; then the correct answer is the honest statement through the common keys (§4) and null values where appropriate.

# 4. Common keys in out/results.json (every task)

# 5. gate_spec.json (public, oracle-free)

{
  "task_id": "F2-smoke-01",
  "required_files": ["out/results.json","out/report.md","out/analysis.py","out/claims.json"],
  "results_keys": {"beta_hat": {"type": "number", "nullable": false}, "...": {}},
  "report_sections": ["Question","Data and exclusions","Method","Results","Uncertainty","Limitations and open issues"],
  "family_checks": ["exclusion_ledger_sums", "ci_contains_estimate", "chronology"],
  "ci_pairs": [["beta_hat","beta_ci_low","beta_ci_high"]],
  "chronology_pairs": [["train_end_date","forecast_origin"]],
  "analysis_timeout_s": 300,
  "reproducibility_tolerance_rel": 1e-6
}

family_checks available in the gate: exclusion_ledger_sums (if out/exclusions.csv exists, n_before − n_removed = n_after per row and chain consistency), ci_contains_estimate (for each listed triple), chronology (for each pair, first ≤ second as ISO dates), positive_se (keys listed in positive_keys), claims_reference_existing_keys, phase_ledger_consistent (if out/state/phases.json exists: every "passed" phase has ≥1 check). The gate never embeds reference values.

# 6. Hidden oracle — ~/wp29-oracles/<task_id>/

oracle.json      # reference values, tolerances, criteria, trap description, provenance (see below)
evaluate.py      # def evaluate(out_dir: str, oracle: dict, workspace_dir: str) -> dict  (receipt, §7)
generator.py     # for synthetic data: regenerates data/ and oracle.json from the seed (deterministic)
REFERENCE.md     # how the reference values were obtained (analytical derivation / independent code / source page+table) and who could access them

oracle.json fields: task_id, version, generator_seed (or null), oracle_provenance ∈ {analytical, independent_code, external_source, designer_extraction}, reference (key → value), tolerances (key → {"abs":…} and/or {"rel":…}, justified in REFERENCE.md from statistical/numerical reasoning), required_criteria (list of {"id","description","kind":"value|flag|null_expected|text_regex|custom","key"?,…}), critical_violation_checks (list of {"id","type": leakage|fabrication|invalid_inference|unreported_substitution|false_completion|provenance, "description", "detection": …}), trap_notes (what was planted and why), contamination_risk (low/medium/high + why), difficulty_rationale.

# 7. Evaluation receipt (returned by evaluate.py; schema in evaluators/public_evaluation_interface/receipt_schema.json)

{"task_id":"…","run_id":"…","evaluator_version":"…","evaluated_utc":"…",
 "required_criteria":[{"id":"…","description":"…","passed":true,"observed":…,"expected":…,"tolerance":…}],
 "critical_violations":[{"id":"…","type":"…","description":"…","evidence":"…"}],
 "scientific_success": true,
 "quality_components":{"correctness":0.0,"integrity":0.0,"reproducibility":0.0,"reporting":0.0,"baseline":0.0},
 "notes":"…"}

scientific_success = every required criterion passed and no critical violation. evaluate.py must be deterministic, standard-library + numpy/pandas only, must never call a language model, and must handle missing/invalid files gracefully (criterion failed, not an exception). It is executed by the harness, not by the executor.

Quality components (frozen weights 0.40 / 0.20 / 0.15 / 0.15 / 0.10): correctness = fraction of required criteria passed; integrity = 1 if no critical violation else 0; reproducibility = 1 if the harness's fresh re-run of out/analysis.py reproduces out/results.json (computed by the harness, not by evaluate.py); reporting = fraction of {six sections present, claims.json valid and each claim's keys exist, limitations non-empty, every numeric claim traceable}; baseline = 1 if the task-defined baseline key is present and correctly computed (criterion in oracle), else 0 (0.5 if present but wrong).

# 8. Family requirements (4 confirmatory + 1 development task per family)

• F1 Data provenance and audit — real or synthetic tabular data with planted, documented defects (duplicates, unit mixes, timestamp shifts, variables unavailable at origin, changing universe, corrupted file). Required: documented analytic dataset (out/analytic.csv), out/exclusions.csv, counts. Reference: independently computed counts/aggregates from the generator or from an independent script. At least 2 of the 5 tasks use real public data.
• F2 Estimation and inference — regressions with specified fixed effects, clustered/robust SEs under a known dependence design, Monte Carlo coverage, a deliberately inappropriate SE to detect. Reference: analytical identities or an independently coded estimator; tolerances from the sampling variability of the reference quantity.
• F3 Forecasting and temporal information — rolling-origin evaluation on real public series (e.g. FRED fredgraph.csv?id=…, ALFRED vintages, Bank of Canada Valet, Statistics Canada, World Bank), features with release lags, revised targets, a future-informed normalisation trap, multi-horizon overlapping targets. Must require a naïve/seasonal-naïve baseline. Reference checks verify chronology (e.g. the forecast error cannot beat an information-feasible bound; a leakage-detector compares the executor's errors with the oracle's own leak-free and leaky implementations).
• F4 Source-grounded evidence synthesis — accessible full-text sources (open-access papers, official statistical releases) provided as text extracts under data/ (respect rights: short excerpts/tables, with citation). Tasks: extract coefficients and uncertainties with table/page locators, check whether a quoted claim is supported, separate incompatible effect-size definitions. Reference: values extracted by the task designer with locators (designer_extraction) — report this dependence.
• F5 Theory, mechanisms and numerical validation — a tractable equilibrium with a budget identity, an estimator with a known bias, a proposition admitting a counterexample, a solver with a known limiting solution. Reference: analytical. Include one task whose stated proposition is false (counterexample required) and one whose identity is deliberately broken in the data.
• F6 Integrated mini-projects — complete bounded chain (question → audit → baseline → method → results → uncertainty → failure discussion → figure/table → short report) on real data where possible; at least one genuine negative-result / invalid-premise task.

Each task gets a card tasks/task_cards/<task_id>.md with: question, data type (observed/synthetic), scope, prerequisites, allowed information, expected artifacts, rubric summary (criteria + critical checks, without reference values), oracle provenance, difficulty rationale, budget class (benchmark: 4.00 USD cap, 25 min, 80 turns per session), anticipated failure modes, contamination risk.

# 9. Registry

tasks/registry.csv: task_id, family, split, data_type (observed/synthetic/mixed), real_data (yes/no), source_documents (yes/no), oracle_provenance, bundle_sha256 (hash of bundle.json), oracle_sha256, infeasible_or_null_truth (yes/no), created_utc, frozen_utc (empty until freeze).