1Every hidden oracle ships a seeded validation set (one or more independently written correct solutions whose replication script is re-run, and two to six wrong solutions). The table lists the outcome of \texttt{evaluators/run\_validations.py} executed before the freeze: PASS means every seeded case received the expected verdict (no false acceptance and no false rejection on the seeded set). The seeded sets bound gross evaluator errors; they do not cover the space of valid solutions.23\begin{longtable}{p{1.0in}p{0.6in}p{4.6in}}4\toprule5Task & Result & Last line of the validation output \\6\midrule7\endhead8F1-conf-01 & PASS & ALL PASS \\9F1-conf-02 & PASS & ALL PASS \\10F1-conf-03 & PASS & ALL PASS \\11F1-conf-04 & PASS & ALL PASS \\12F1-dev-01 & PASS & ALL PASS \\13F2-conf-01 & PASS & ALL OK \\14F2-conf-02 & PASS & ALL OK \\15F2-conf-03 & PASS & ALL OK \\16F2-conf-04 & PASS & ALL OK \\17F2-dev-01 & PASS & ALL OK \\18F2-smoke-01 & no validate.py & \\19F3-conf-01 & PASS & ALL PASS \\20F3-conf-02 & PASS & ALL PASS \\21F3-conf-03 & PASS & ALL PASS \\22F3-conf-04 & PASS & ALL PASS \\23F3-dev-01 & PASS & ALL PASS \\24F4-conf-01 & PASS & VALIDATION PASSED \\25F4-conf-02 & PASS & VALIDATION PASSED \\26F4-conf-03 & PASS & VALIDATION PASSED \\27F4-conf-04 & PASS & VALIDATION PASSED \\28F4-dev-01 & PASS & VALIDATION PASSED \\29F4-sources & no validate.py & \\30F5-conf-01 & PASS & ALL PASS \\31F5-conf-02 & PASS & ALL PASS \\32F5-conf-03 & PASS & ALL PASS \\33F5-conf-04 & PASS & ALL PASS \\34F5-dev-01 & PASS & ALL PASS \\35F6-conf-01 & PASS & VALIDATION PASSED \\36F6-conf-02 & PASS & VALIDATION PASSED \\37F6-conf-03 & PASS & VALIDATION PASSED \\38F6-conf-04 & PASS & VALIDATION PASSED \\39F6-dev-01 & PASS & VALIDATION PASSED \\40\bottomrule41\end{longtable}42