Evaluator validation (seeded artefacts)
Generated 2026-10-30T03:58:32+00:00 by evaluators/run_validations.py. Each hidden oracle ships a validation/ set with ≥1 correct_out (an independently written solution whose analysis.py is re-run) and ≥2 wrong_out artefacts (planted scientific errors, fabrication, false completion or leakage). The evaluator must accept the former and reject the latter; a PASS means every seeded case received the expected verdict (false acceptance = 0 and false rejection = 0 on the seeded set). The seeded set is small and designer-authored; it bounds gross evaluator errors, not the full space of valid solutions.
| task_id | result | last line of validation output |
|---|---|---|
| F1-conf-01 | PASS | ALL PASS |
| F1-conf-02 | PASS | ALL PASS |
| F1-conf-03 | PASS | ALL PASS |
| F1-conf-04 | PASS | ALL PASS |
| F1-dev-01 | PASS | ALL PASS |
| F2-conf-01 | PASS | ALL OK |
| F2-conf-02 | PASS | ALL OK |
| F2-conf-03 | PASS | ALL OK |
| F2-conf-04 | PASS | ALL OK |
| F2-dev-01 | PASS | ALL OK |
| F2-smoke-01 | no validate.py | |
| F3-conf-01 | PASS | ALL PASS |
| F3-conf-02 | PASS | ALL PASS |
| F3-conf-03 | PASS | ALL PASS |
| F3-conf-04 | PASS | ALL PASS |
| F3-dev-01 | PASS | ALL PASS |
| F4-conf-01 | PASS | VALIDATION PASSED |
| F4-conf-02 | PASS | VALIDATION PASSED |
| F4-conf-03 | PASS | VALIDATION PASSED |
| F4-conf-04 | PASS | VALIDATION PASSED |
| F4-dev-01 | PASS | VALIDATION PASSED |
| F4-sources | no validate.py | |
| F5-conf-01 | PASS | ALL PASS |
| F5-conf-02 | PASS | ALL PASS |
| F5-conf-03 | PASS | ALL PASS |
| F5-conf-04 | PASS | ALL PASS |
| F5-dev-01 | PASS | ALL PASS |
| F6-conf-01 | PASS | VALIDATION PASSED |
| F6-conf-02 | PASS | VALIDATION PASSED |
| F6-conf-03 | PASS | VALIDATION PASSED |
| F6-conf-04 | PASS | VALIDATION PASSED |
| F6-dev-01 | PASS | VALIDATION PASSED |
Summary: 30 PASS, 0 FAIL, 2 without validation set, of 32 oracles.