SPB Git

spb/wp29_uqo

Public
1.6 KB · 14 lines markdown
1# Task card — F4-conf-0423- **Family / split**: F4 Source-grounded evidence synthesis / confirmatory4- **Question**: verify five bibliographic entries with attributed findings against the provided source extracts (title, year, finding); list the unverifiable ones.5- **Data type**: observed — five arXiv e-prints (CC BY 4.0) as plain-text extracts with provenance headers; reference list written by the designer.6- **Scope / prerequisites**: bibliographic care; reading tables and abstracts; stated year convention.7- **Allowed information**: the extracts only.8- **Expected artifacts**: results.json (`reference_checks`, `unverifiable_references`, `n_verified` + common keys), report.md, analysis.py, claims.json.9- **Rubric summary**: required criteria — unverifiable set exact; three boolean flags per reference; n_verified; five locators. Critical checks — n_input_rows = line counts (fabrication); feasibility "complete"; R3 finding marked supported (false completion); locators citing non-existent tables (provenance).10- **Oracle provenance**: designer_extraction; validation solution re-derives all checks; no human annotation.11- **Difficulty rationale**: moderate — one entry disagrees on all three grounds, one legitimately dated entry (first-version year) must not be over-flagged, finding checks need specific cells.12- **Budget class**: benchmark (4.00 USD cap, 25 min, 80 turns).13- **Anticipated failure modes**: over-flagging R4 for its year; accepting R3 because the arXiv number and authors are right; missing the "event year vs years four–five" contradiction; vague locators.14- **Contamination risk**: low.
Files · 4f59ed568641ceda6f39552d5cc3a441aa812088
Menu

Theme