Evaluator validation (seeded artefacts)
Generated 2026-10-30T04:34:23+00:00 by evaluators/run_validations.py. Each hidden oracle ships a validation/ set with ≥1 correct_out (an independently written solution whose analysis.py is re-run) and ≥2 wrong_out artefacts (planted scientific errors, fabrication, false completion or leakage). The evaluator must accept the former and reject the latter; a PASS means every seeded case received the expected verdict (false acceptance = 0 and false rejection = 0 on the seeded set). The seeded set is small and designer-authored; it bounds gross evaluator errors, not the full space of valid solutions.
| task_id | result | last line of validation output |
|---|---|---|
| F1-conf-01 | PASS | ALL PASS |
| F1-conf-02 | PASS | ALL PASS |
| F1-conf-03 | PASS | ALL PASS |
| F1-conf-04 | PASS | ALL PASS |
| F1-dev-01 | PASS | ALL PASS |
| F2-conf-01 | PASS | ALL OK |
| F2-conf-02 | PASS | ALL OK |
| F2-conf-03 | PASS | ALL OK |
| F2-conf-04 | PASS | ALL OK |
| F2-dev-01 | PASS | ALL OK |
| F2-smoke-01 | no validate.py | |
| F3-conf-01 | PASS | ALL PASS |
| F3-conf-02 | PASS | ALL PASS |
| F3-conf-03 | PASS | ALL PASS |
| F3-conf-04 | PASS | ALL PASS |
| F3-dev-01 | PASS | ALL PASS |
| F4-conf-01 | PASS | VALIDATION PASSED |
| F4-conf-02 | PASS | VALIDATION PASSED |
| F4-conf-03 | PASS | VALIDATION PASSED |
| F4-conf-04 | PASS | VALIDATION PASSED |
| F4-dev-01 | PASS | VALIDATION PASSED |
| F4-sources | no validate.py | |
| F5-conf-01 | PASS | ALL PASS |
| F5-conf-02 | PASS | ALL PASS |
| F5-conf-03 | PASS | ALL PASS |
| F5-conf-04 | PASS | ALL PASS |
| F5-dev-01 | PASS | ALL PASS |
| F6-conf-01 | PASS | VALIDATION PASSED |
| F6-conf-02 | PASS | VALIDATION PASSED |
| F6-conf-03 | PASS | VALIDATION PASSED |
| F6-conf-04 | PASS | VALIDATION PASSED |
| F6-dev-01 | PASS | VALIDATION PASSED |
| T1-forecast | PASS | ALL PASS |
| T2-simulation | PASS | ALL PASS |
| T3-synthesis | PASS | ALL PASS |
Summary: 33 PASS, 0 FAIL, 2 without validation set, of 35 oracles.