Failure analysis
Generated 2026-10-30T06:07:09+00:00. Every evaluated run that is not a scientific success, across all splits, with the failed criteria and critical violations from its receipt and an audit classification (executor error vs evaluator defect vs injected fault).
| run | split | task | arm | failed criteria | violations | classification |
|---|---|---|---|---|---|---|
| RA0022 | ablation | F6-conf-04 | L | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0013 | confirmatory | F6-conf-04 | B | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0014 | confirmatory | F6-conf-04 | D | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0015 | confirmatory | F6-conf-04 | D | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0016 | confirmatory | F6-conf-04 | C | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0017 | confirmatory | F6-conf-04 | B | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0018 | confirmatory | F6-conf-04 | C | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0020 | confirmatory | F6-conf-04 | C | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0021 | confirmatory | F6-conf-04 | A | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0022 | confirmatory | F6-conf-04 | B | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0023 | confirmatory | F6-conf-04 | A | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0024 | confirmatory | F6-conf-04 | A | counts, naive | -- | evaluator defect (baseline on the cleaned sample; A-005); evaluator defect (impossible negative hours excluded with ledger; A-005) |
| RC0063 | confirmatory | F3-conf-02 | A | naive_h1, naive_h2, model_errors_consistent, rel_improvement_consistent | -- | evaluator defect (metrics rounded to 4 decimals vs 1e-6 tolerance; A-007); evaluator defect (rounding vs 1e-6 tolerance; A-007); evaluator defect (rounding; A-007) |
| RC0097 | confirmatory | F1-conf-01 | D | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0098 | confirmatory | F1-conf-01 | C | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0099 | confirmatory | F1-conf-01 | A | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0100 | confirmatory | F1-conf-01 | D | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0101 | confirmatory | F1-conf-01 | A | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0102 | confirmatory | F1-conf-01 | B | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0103 | confirmatory | F1-conf-01 | B | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0104 | confirmatory | F1-conf-01 | D | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0105 | confirmatory | F1-conf-01 | C | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0106 | confirmatory | F1-conf-01 | A | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0107 | confirmatory | F1-conf-01 | C | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0108 | confirmatory | F1-conf-01 | B | -- | false_complete | evaluator defect (honest 'partial' for an undefined month; A-005) |
| RC0276 | confirmatory | F6-conf-01 | D | -- | supported_effect_claim | evaluator defect (regex false positive on the conclusion code no_detectable_effect; A-007) |
| RP0024 | pilot | F3-dev-01 | A | model_errors_consistent | -- | executor error (pilot) |
| RR0014 | recovery | F1-dev-01 | A | counts, mean_revenue, median_revenue, mean_employees, analytic_rows, ledger, issues_detected | fab_counts, false_complete | fault R4: input removed by the controller (recognised blockage, not an executor error) |
| RR0015 | recovery | F1-dev-01 | D | counts, mean_revenue, median_revenue, mean_employees, analytic_rows, ledger, issues_detected | fab_counts, false_complete | fault R4: input removed by the controller (recognised blockage, not an executor error) |
| RT0023 | transfer | T2-simulation | C | -- | invalid_universal_claim | executor error: universal claim from a simulation grid (invalid inference) |
Classification counts (runs)
- evaluator defect (baseline on the cleaned sample; A-005): 12
- evaluator defect (impossible negative hours excluded with ledger; A-005): 12
- evaluator defect (honest 'partial' for an undefined month; A-005): 12
- fault R4: input removed by the controller (recognised blockage, not an executor error): 2
- evaluator defect (metrics rounded to 4 decimals vs 1e-6 tolerance; A-007): 1
- evaluator defect (rounding vs 1e-6 tolerance; A-007): 1
- evaluator defect (rounding; A-007): 1
- evaluator defect (regex false positive on the conclusion code no_detectable_effect; A-007): 1
- executor error (pilot): 1
- executor error: universal claim from a simulation grid (invalid inference): 1
Reading
In the confirmatory split, all 25 non-successes under the frozen oracles are evaluator defects (four tasks: an undefined feasibility label, a generator artefact, a rounding tolerance, a regex false positive; amendments A-005/A-007); under the corrected rubric every arm completes every task. No leakage, fabrication, unreported-substitution or provenance violation was recorded in any confirmatory run. The transfer split has one executor error (a universal claim from a simulation grid). Recovery-split non-successes are the two missing-fixture runs, in which both arms correctly recognised the blockage.