Every hidden oracle ships a seeded validation set (one or more independently written correct solutions whose replication script is re-run, and two to six wrong solutions). The table lists the outcome of \texttt{evaluators/run\_validations.py} executed before the freeze: PASS means every seeded case received the expected verdict (no false acceptance and no false rejection on the seeded set). The seeded sets bound gross evaluator errors; they do not cover the space of valid solutions.

\begin{longtable}{p{1.0in}p{0.6in}p{4.6in}}
\toprule
Task & Result & Last line of the validation output \\
\midrule
\endhead
F1-conf-01 & PASS & ALL PASS \\
F1-conf-02 & PASS & ALL PASS \\
F1-conf-03 & PASS & ALL PASS \\
F1-conf-04 & PASS & ALL PASS \\
F1-dev-01 & PASS & ALL PASS \\
F2-conf-01 & PASS & ALL OK \\
F2-conf-02 & PASS & ALL OK \\
F2-conf-03 & PASS & ALL OK \\
F2-conf-04 & PASS & ALL OK \\
F2-dev-01 & PASS & ALL OK \\
F2-smoke-01 & no validate.py &  \\
F3-conf-01 & PASS & ALL PASS \\
F3-conf-02 & PASS & ALL PASS \\
F3-conf-03 & PASS & ALL PASS \\
F3-conf-04 & PASS & ALL PASS \\
F3-dev-01 & PASS & ALL PASS \\
F4-conf-01 & PASS & VALIDATION PASSED \\
F4-conf-02 & PASS & VALIDATION PASSED \\
F4-conf-03 & PASS & VALIDATION PASSED \\
F4-conf-04 & PASS & VALIDATION PASSED \\
F4-dev-01 & PASS & VALIDATION PASSED \\
F4-sources & no validate.py &  \\
F5-conf-01 & PASS & ALL PASS \\
F5-conf-02 & PASS & ALL PASS \\
F5-conf-03 & PASS & ALL PASS \\
F5-conf-04 & PASS & ALL PASS \\
F5-dev-01 & PASS & ALL PASS \\
F6-conf-01 & PASS & VALIDATION PASSED \\
F6-conf-02 & PASS & VALIDATION PASSED \\
F6-conf-03 & PASS & VALIDATION PASSED \\
F6-conf-04 & PASS & VALIDATION PASSED \\
F6-dev-01 & PASS & VALIDATION PASSED \\
\bottomrule
\end{longtable}