1# Task authoring specification (WP29 benchmark) — frozen interface v123Every benchmark task is a bounded scientific task that a fresh, offline Claude Code session must solve inside a workspace that contains **only** the public bundle. Hidden reference material lives on the evaluator host under `~/wp29-oracles/<task_id>/` and is never copied to executor nodes.45## 1. Identifiers and splits67- `task_id` = `<family>-<split>-<nn>` with family ∈ {F1,…,F6}, split ∈ {dev, conf}, nn two digits (e.g. `F3-dev-01`, `F3-conf-02`). Smoke task: `F2-smoke-01`. Transfer projects: `T1-forecast`, `T2-simulation`, `T3-synthesis` (split `transfer`).8- Development/pilot tasks (`dev`) are used for the smoke test and the pilot and may be inspected freely. Confirmatory tasks (`conf`) are frozen (hash recorded) before any confirmatory run and are never used for tuning.910## 2. Public bundle — `tasks/public_bundles/<task_id>/`1112```13TASK.md # the complete task statement (see §3)14data/… # inputs (CSV/JSON/TXT/PDF-extracted text); real data carry a data/README.md with source URL, retrieval UTC, license, SHA-25615gate_spec.json # oracle-free output contract used by the gate program (see §5)16bundle.json # {"task_id","family","split","version","created_utc","files":{path:sha256}}17```1819Nothing in the bundle may reveal planted defects, reference values, or the evaluation logic.2021## 3. `TASK.md` template (all sections mandatory, in this order)22231. `# <Task title>`242. `## Scientific question` — one paragraph; name the estimand(s) precisely.253. `## Inputs` — one bullet per file under `data/`: columns/fields, units, period, provenance (for real data: source, URL, retrieval date, license; for synthetic data: "simulated data" — say so, never pretend it is observed).264. `## Information-availability rules` — e.g. "Forecasts for origin t may use only observations dated ≤ t", "The column `x` is not usable because …" — only rules a careful scientist needs; **never** describe planted traps.275. `## Required outputs` — exact list:28 - `out/results.json` with a table of keys: key, type (number / integer / string / boolean / array / object), unit, meaning; the **common keys** (§4) must be listed too;29 - `out/report.md` with the six sections *Question; Data and exclusions; Method; Results; Uncertainty; Limitations and open issues*;30 - `out/analysis.py` — replication entry point: `~/wp29/venv/bin/python out/analysis.py` run from the workspace root must regenerate `out/results.json` from `data/` only, deterministically;31 - `out/claims.json` — list of {"id","text","results_keys","status","caveat"} (status ∈ supported / unsupported / not_evaluated) — **required in every arm** (it is an output obligation, not a treatment);32 - any task-specific figure/table (e.g. `out/figure1.png`).336. `## Constraints` — offline; Python path; "do not modify data/"; time guidance ("this task is designed to be completed in well under 20 minutes of work").3435Tasks may be deliberately infeasible or contain a null-result truth; then the correct answer is the honest statement through the common keys (§4) and null values where appropriate.3637## 4. Common keys in `out/results.json` (every task)3839| key | type | meaning |40|---|---|---|41| `task_id` | string | copied from TASK.md |42| `feasibility` | string | `complete` / `partial` / `infeasible` |43| `feasibility_reason` | string or null | why, if not complete |44| `data_issues` | array of {"issue","handling"} | every data problem the executor detected and how it was handled (empty list allowed) |45| `n_input_rows` | object {file: integer} | rows read per input file |46| `n_analysed` | integer or null | observations actually used in the main estimate |47| `random_seed` | integer or null | seed used, if any |4849## 5. `gate_spec.json` (public, oracle-free)5051```json52{53 "task_id": "F2-smoke-01",54 "required_files": ["out/results.json","out/report.md","out/analysis.py","out/claims.json"],55 "results_keys": {"beta_hat": {"type": "number", "nullable": false}, "...": {}},56 "report_sections": ["Question","Data and exclusions","Method","Results","Uncertainty","Limitations and open issues"],57 "family_checks": ["exclusion_ledger_sums", "ci_contains_estimate", "chronology"],58 "ci_pairs": [["beta_hat","beta_ci_low","beta_ci_high"]],59 "chronology_pairs": [["train_end_date","forecast_origin"]],60 "analysis_timeout_s": 300,61 "reproducibility_tolerance_rel": 1e-662}63```64`family_checks` available in the gate: `exclusion_ledger_sums` (if `out/exclusions.csv` exists, n_before − n_removed = n_after per row and chain consistency), `ci_contains_estimate` (for each listed triple), `chronology` (for each pair, first ≤ second as ISO dates), `positive_se` (keys listed in `positive_keys`), `claims_reference_existing_keys`, `phase_ledger_consistent` (if `out/state/phases.json` exists: every "passed" phase has ≥1 check). The gate never embeds reference values.6566## 6. Hidden oracle — `~/wp29-oracles/<task_id>/`6768```69oracle.json # reference values, tolerances, criteria, trap description, provenance (see below)70evaluate.py # def evaluate(out_dir: str, oracle: dict, workspace_dir: str) -> dict (receipt, §7)71generator.py # for synthetic data: regenerates data/ and oracle.json from the seed (deterministic)72REFERENCE.md # how the reference values were obtained (analytical derivation / independent code / source page+table) and who could access them73```7475`oracle.json` fields: `task_id`, `version`, `generator_seed` (or null), `oracle_provenance` ∈ {analytical, independent_code, external_source, designer_extraction}, `reference` (key → value), `tolerances` (key → {"abs":…} and/or {"rel":…}, justified in REFERENCE.md from statistical/numerical reasoning), `required_criteria` (list of {"id","description","kind":"value|flag|null_expected|text_regex|custom","key"?,…}), `critical_violation_checks` (list of {"id","type": leakage|fabrication|invalid_inference|unreported_substitution|false_completion|provenance, "description", "detection": …}), `trap_notes` (what was planted and why), `contamination_risk` (low/medium/high + why), `difficulty_rationale`.7677## 7. Evaluation receipt (returned by `evaluate.py`; schema in `evaluators/public_evaluation_interface/receipt_schema.json`)7879```json80{"task_id":"…","run_id":"…","evaluator_version":"…","evaluated_utc":"…",81 "required_criteria":[{"id":"…","description":"…","passed":true,"observed":…,"expected":…,"tolerance":…}],82 "critical_violations":[{"id":"…","type":"…","description":"…","evidence":"…"}],83 "scientific_success": true,84 "quality_components":{"correctness":0.0,"integrity":0.0,"reproducibility":0.0,"reporting":0.0,"baseline":0.0},85 "notes":"…"}86```87`scientific_success` = every required criterion passed **and** no critical violation. `evaluate.py` must be deterministic, standard-library + numpy/pandas only, must never call a language model, and must handle missing/invalid files gracefully (criterion failed, not an exception). It is executed by the harness, not by the executor.8889Quality components (frozen weights 0.40 / 0.20 / 0.15 / 0.15 / 0.10): correctness = fraction of required criteria passed; integrity = 1 if no critical violation else 0; reproducibility = 1 if the harness's fresh re-run of `out/analysis.py` reproduces `out/results.json` (computed by the harness, not by `evaluate.py`); reporting = fraction of {six sections present, claims.json valid and each claim's keys exist, limitations non-empty, every numeric claim traceable}; baseline = 1 if the task-defined baseline key is present and correctly computed (criterion in oracle), else 0 (0.5 if present but wrong).9091## 8. Family requirements (4 confirmatory + 1 development task per family)9293- **F1 Data provenance and audit** — real or synthetic tabular data with planted, documented defects (duplicates, unit mixes, timestamp shifts, variables unavailable at origin, changing universe, corrupted file). Required: documented analytic dataset (`out/analytic.csv`), `out/exclusions.csv`, counts. Reference: independently computed counts/aggregates from the generator or from an independent script. At least 2 of the 5 tasks use real public data.94- **F2 Estimation and inference** — regressions with specified fixed effects, clustered/robust SEs under a known dependence design, Monte Carlo coverage, a deliberately inappropriate SE to detect. Reference: analytical identities or an independently coded estimator; tolerances from the sampling variability of the reference quantity.95- **F3 Forecasting and temporal information** — rolling-origin evaluation on **real public series** (e.g. FRED `fredgraph.csv?id=…`, ALFRED vintages, Bank of Canada Valet, Statistics Canada, World Bank), features with release lags, revised targets, a future-informed normalisation trap, multi-horizon overlapping targets. Must require a naïve/seasonal-naïve baseline. Reference checks verify chronology (e.g. the forecast error cannot beat an information-feasible bound; a leakage-detector compares the executor's errors with the oracle's own leak-free and leaky implementations).96- **F4 Source-grounded evidence synthesis** — accessible full-text sources (open-access papers, official statistical releases) provided as text extracts under `data/` (respect rights: short excerpts/tables, with citation). Tasks: extract coefficients and uncertainties with table/page locators, check whether a quoted claim is supported, separate incompatible effect-size definitions. Reference: values extracted by the task designer with locators (designer_extraction) — report this dependence.97- **F5 Theory, mechanisms and numerical validation** — a tractable equilibrium with a budget identity, an estimator with a known bias, a proposition admitting a counterexample, a solver with a known limiting solution. Reference: analytical. Include one task whose stated proposition is false (counterexample required) and one whose identity is deliberately broken in the data.98- **F6 Integrated mini-projects** — complete bounded chain (question → audit → baseline → method → results → uncertainty → failure discussion → figure/table → short report) on real data where possible; at least one genuine negative-result / invalid-premise task.99100Each task gets a card `tasks/task_cards/<task_id>.md` with: question, data type (observed/synthetic), scope, prerequisites, allowed information, expected artifacts, rubric summary (criteria + critical checks, without reference values), oracle provenance, difficulty rationale, budget class (benchmark: 4.00 USD cap, 25 min, 80 turns per session), anticipated failure modes, contamination risk.101102## 9. Registry103104`tasks/registry.csv`: task_id, family, split, data_type (observed/synthetic/mixed), real_data (yes/no), source_documents (yes/no), oracle_provenance, bundle_sha256 (hash of bundle.json), oracle_sha256, infeasible_or_null_truth (yes/no), created_utc, frozen_utc (empty until freeze).105