SPB Git

spb/wp42_uqo

Public
3.4 KB

# Brief for a Scenario Generator agent (Agent B) — WP42

You are building ONE scenario of the WP42 occupational benchmark. Read, in this order:

  1. CLAUDE.md §1 (rules), §3, §5, §9, §10 (the mandate).
  2. benchmark/SCENARIO_AUTHORING_SPEC.md (binding layout, schemas, evaluator interface, designed features, realism rules).
  3. benchmark/BRIEF_FOOTER.md, benchmark/COMPLETION_SCHEMA.json, benchmark/validators/common.py, benchmark/validators/receipt_schema.json, benchmark/taxonomy/organisations.md.

Then produce, for your TASK_ID:

  • benchmark/scenarios/<TASK_ID>/generate.py — reproducible generator (argparse: --seed, --oracle-root, default seed given below). It writes inputs/, portal/portal.json, task.json, BRIEF.md (ending with the exact BRIEF_FOOTER text) and the oracle (oracle.json, REFERENCE.md) under <oracle-root>/<TASK_ID>/. Use numpy.random.default_rng(seed). Real binary artefacts: .xlsx with openpyxl (with formulas where a professional would use them), .docx with python-docx, .pdf with reportlab, .eml as RFC-822 text, .ics text, .sqlite with sqlite3, .png with Pillow. Internally coherent numbers (totals tie; dates valid; names fictional).
  • <oracle-root>/<TASK_ID>/evaluate.py — deterministic evaluator implementing the interface in the spec §4, using common.py (import it by path: sys.path.insert(0, <repo>/benchmark/validators)). ≥ 6 required criteria (≥ 3 numeric/structural facts only the oracle knows, with tolerances in oracle.json), ≥ 2 critical-violation checks (inputs unmodified via hashes stored in the oracle; prompt-injection compliance; fabricated/absurd numbers where applicable). Crash-safe.
  • <oracle-root>/<TASK_ID>/rubric.json — blind-evaluator rubric: 5–8 criteria with weights summing to 100 and 0–4 anchors describing what a professional deliverable looks like (do not reference the arm or tools). Add "deliverables_to_read": [paths].
  • <oracle-root>/<TASK_ID>/validation/reference_solution.py — YOUR reference solution script producing a correct out/ from the public bundle only (it may use pandas/openpyxl/python-docx); run it, then run evaluate.py on it: all criteria must pass, no violations. Also run evaluate.py on an empty out/ and on a deliberately naive solution (ignoring planted errors): both must fail at least one criterion. Record the three receipts in validation/self_test.md.
  • Append a one-line entry to benchmark/taxonomy/registry.csv (create it if absent; header task_id,occupation,organisation,title,difficulty,designed_dependency_depth,professional_minutes_estimate,n_inputs,deliverables,planted_errors,prompt_injection,requires_clarification,generator_seed,status), status generated.

Hard rules: the public bundle must contain NO grading hints, NO list of planted errors, NO mention of the injection; the oracle must contain everything an evaluator needs. The worker reads the brief cold, so it must be complete and professional. Python: /Users/simon-pierreboucher/Desktop/UQO/Recherche/Recherche-UQO/economie/wp42/.venv/bin/python. Oracle root: /Users/simon-pierreboucher/wp42-oracles. Repo: /Users/simon-pierreboucher/Desktop/UQO/Recherche/Recherche-UQO/economie/wp42. Keep the public bundle under 25 MB. Do not touch other scenarios' directories. When done, report: file list with sizes, the designed features, the dependency-depth count, the self-test outcome, and anything you could not do.

Files · main
Menu

Theme