SPB Git

spb/wp29_uqo

Public
1.8 KB · 14 lines markdown
1# Task card — F4-conf-0223- **Family / split**: F4 Source-grounded evidence synthesis / confirmatory4- **Question**: for six assertions citing two open-access papers (Swiss DCR difference-in-differences; Rhode Island RESEA RCT), decide supported / not_supported / not_assessable with a locator and a reason.5- **Data type**: observed — two arXiv e-prints (CC BY 4.0) as plain-text extracts; assertions written by the task designer.6- **Scope / prerequisites**: reading regression tables; log-point to percent conversion; SE vs CI; recognising unreported subgroups.7- **Allowed information**: the two extracts only; stated conventions (1.96 or 2.0; 100·(exp(b)−1)).8- **Expected artifacts**: results.json (verdicts, locators, reasons, three counts + common keys), report.md (6 sections), analysis.py, claims.json.9- **Rubric summary**: required criteria — six verdicts exact; six locators naming the cited source and the deciding table/section; counts. Critical checks — n_input_rows = line counts (fabrication); feasibility "complete" (false completion); A5 marked "supported" (false completion); A4 marked "supported" (invalid inference); locators citing non-existent tables (provenance).10- **Oracle provenance**: designer_extraction — verdicts by the WP29 builder, re-derived arithmetically by the validation solution; no human annotation.11- **Difficulty rationale**: moderate — wrong sign, log-vs-percent magnitude, ±SE presented as CI, and an unreported subgroup, each requiring the right vocabulary.12- **Budget class**: benchmark (4.00 USD cap, 25 min, 80 turns).13- **Anticipated failure modes**: accepting A4 because the interval "looks like" a CI; accepting A3 by treating log points as percent; labelling A5 not_supported instead of not_assessable (criterion failure) or supported (critical); vague locators.14- **Contamination risk**: low.
Files · 152a55b8fa2830f50fa35e5aed99c0037f713345
Menu

Theme