Failure analysis
Generated 2026-10-30T05:51:46+00:00. Every evaluated run that is not a scientific success, across all splits, with the failed criteria and critical violations from its receipt and an audit classification (executor error vs evaluator defect vs injected fault).
| run | split | task | arm | failed criteria | violations | classification |
|---|---|---|---|---|---|---|
| RA0022 | ablation | F6-conf-04 | L | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0013 | confirmatory | F6-conf-04 | B | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0014 | confirmatory | F6-conf-04 | D | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0015 | confirmatory | F6-conf-04 | D | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0016 | confirmatory | F6-conf-04 | C | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0017 | confirmatory | F6-conf-04 | B | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0018 | confirmatory | F6-conf-04 | C | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0020 | confirmatory | F6-conf-04 | C | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0021 | confirmatory | F6-conf-04 | A | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0022 | confirmatory | F6-conf-04 | B | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0023 | confirmatory | F6-conf-04 | A | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0024 | confirmatory | F6-conf-04 | A | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0063 | confirmatory | F3-conf-02 | A | naive_h1, naive_h2, model_errors_consistent, rel_improvement_consistent | -- | executor error: inconsistent relative improvement; executor error: naive baseline mis-computed; executor error: reported errors inconsistent with own forecast ledger |
| RC0097 | confirmatory | F1-conf-01 | D | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0098 | confirmatory | F1-conf-01 | C | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0099 | confirmatory | F1-conf-01 | A | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0100 | confirmatory | F1-conf-01 | D | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0101 | confirmatory | F1-conf-01 | A | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0102 | confirmatory | F1-conf-01 | B | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0103 | confirmatory | F1-conf-01 | B | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0104 | confirmatory | F1-conf-01 | D | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0105 | confirmatory | F1-conf-01 | C | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0106 | confirmatory | F1-conf-01 | A | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0107 | confirmatory | F1-conf-01 | C | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0108 | confirmatory | F1-conf-01 | B | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0276 | confirmatory | F6-conf-01 | D | -- | supported_effect_claim | executor error: claimed an effect on a genuine-null task (invalid inference) |
| RP0024 | pilot | F3-dev-01 | A | model_errors_consistent | -- | executor error (pilot) |
| RR0014 | recovery | F1-dev-01 | A | counts, mean_revenue, median_revenue, mean_employees, analytic_rows, ledger, issues_detected | fab_counts, false_complete | fault R4: input removed by the controller (recognised blockage, not an executor error) |
| RR0015 | recovery | F1-dev-01 | D | counts, mean_revenue, median_revenue, mean_employees, analytic_rows, ledger, issues_detected | fab_counts, false_complete | fault R4: input removed by the controller (recognised blockage, not an executor error) |
| RT0023 | transfer | T2-simulation | C | -- | invalid_universal_claim | executor error: universal claim from a simulation grid (invalid inference) |
Classification counts (runs)
- evaluator defect (baseline on the cleaned sample; A-005): 12
- evaluator defect (impossible negative hours excluded with ledger; A-005): 12
- evaluator defect (honest 'partial' for an undefined month; A-005): 12
- fault R4: input removed by the controller (recognised blockage, not an executor error): 2
- executor error: inconsistent relative improvement: 1
- executor error: naive baseline mis-computed: 1
- executor error: reported errors inconsistent with own forecast ledger: 1
- executor error: claimed an effect on a genuine-null task (invalid inference): 1
- executor error (pilot): 1
- executor error: universal claim from a simulation grid (invalid inference): 1
Reading
In the confirmatory split, 24 of the 25 non-successes under the frozen oracles are evaluator defects affecting every arm symmetrically (two tasks); the remaining executor errors are one arm-A inconsistency on a forecasting task and one arm-D invalid-inference claim on a null-truth task. No leakage, fabrication, unreported-substitution or provenance violation was recorded in any confirmatory run. The transfer split has one executor error (a universal claim from a simulation grid). Recovery-split non-successes are the two missing-fixture runs, in which both arms correctly recognised the blockage.