# Agents, deliberation protocols and aggregation (WP 26)

Code: src/agentic_valuation/agents/ (roles, packets, runner), src/agentic_valuation/debate/ (committees, protocols, stopping, aggregation), src/agentic_valuation/experiments/ (run_arm, freeze, providers_check). CLI: av run-arm, av freeze, av providers-check. Tests: tests/test_agents_*.py, tests/test_debate_*.py (mock provider, synthetic data, < 1 s).

Every role is a simulated analytic role, not a licensed Quebec appraiser; the prompts say so. Nothing here is a professional opinion of value.

# 1. Roles

The system prompt of a role is SHARED_PREAMBLE + ROLE_ID: <id> + role text + OUTPUT_RULES (agents/roles.py). The preamble states the task (market value in CAD as of the valuation date, a transaction-price prediction, not a fundamental value), the information cutoff, the evidence discipline (cite ids, no invented inspection, keep missing data missing, traceable adjustments) and the JSON-only answer format. Prompts are stable constants; their SHA-256 is part of results/freeze.json.

roles_for("default12" | "all20" | "shuffled12" | [ids]), placebo_roles(n), shuffled_roles(seed) (default prompts permuted across the default views — alternative placebo), role_prompt_hashes().

# 2. Evidence packets and views (agents/packets.py)

build_packet(analytic_row, comps, market, geo, cfg) → EvidencePacket with subject (P), comparables (C01..C12, only sale_date ≤ cutoff), market (M1), geo (G1 or null) and notes. Invariants: no sale price, no street/address, lat/lng rounded to 3 decimals, missing_fields filled, prior sale dropped unless strictly before the cutoff. A leak assertion runs on every packet.

role_view(packet, role) keeps only the role's sections (evidence ids preserved, evidence_ids_visible listed). render_user_prompt(view, extra, kind=…, verification_flags=…) produces:

Evidence packet <id> — valuation date …, information cutoff ….
Sections visible to your role: ….
<evidence_packet>{view json}</evidence_packet>
<committee_state>{"round":r,"committee_median":…,"estimates":[{role,point,q05,q95,reasoning_summary}…],
                  "evidence_ids_cited_by_others_outside_your_view":[…],"critiques_addressed_to_you":[…]}</committee_state>   (rounds ≥ 1)
<verification_flags>[…]</verification_flags>                                                                              (arm H, flagged agents)
TASK: … (estimate | critique | revision | skeptic_verdict | self_critique | self_refine)

The committee table is anonymised (role ids, never model/provider). Evidence exchange is by ids only: a role learns that others cited G1, never its content and never the price.

verify_evidence(estimate, packet, allowed_ids=None) → flags unknown_evidence_id:<id>, unknown_comparable:<id>, adjustment_out_of_band:<id> (adjusted price outside ±40 % of the raw comparable price), point_outside_comparable_hull (outside the adjusted-comparable range ±25 %). faithfulness_summary gives the counts used by the evaluation layer.

# 3. One turn (agents/runner.py)

run_turn(provider, role, view_prompt, schema, model, effort, round, kind) → AgentTurn with tokens, cached tokens, estimated cost, latency, prompt hash, request id, validation retries and error. A failed call or a schema failure after ≤ 2 corrective retries yields estimate=None + error; nothing is fabricated. SpendCapExceeded propagates to the runner. ProviderPool keeps one adapter per model id (own concurrency semaphore) on a shared ledger; force_mock=True unless the run is --live with keys.

# 4. Protocols (debate/protocols.py)

independent(packet, roles, assignment, pool)                       # F (E with roles=[generalist])
  turns ← gather(estimate(r) for r in roles)                          round 0, concurrent
  finalize

structured_debate(packet, roles, assignment, pool, rounds, blinded_skeptic, evidence_verification, placebo)   # G / H / I
  debaters ← roles minus skeptic if blinded_skeptic
  turns ← gather(estimate(r) for r in debaters)                       round 0
  flags ← verify(current) if evidence_verification
  prev_iqr ← IQR(log points)
  for r_no in 1..rounds:
      state_r ← committee_state(current, for_role=r, visible_ids(r)) # anonymised table + committee_median + ids cited outside r's view
      critiques ← gather(critique(r, state_r, flags[r]) for r in debaters)       CritiqueSet
      addressed[target] ← critiques whose target_role = target (critic anonymised)
      revisions ← gather(revise(r, state_r + addressed[r], flags[r]) for r in debaters)   AgentEstimate
      current ← latest estimate per role; flags ← verify(current)
      if |IQR − prev_iqr| < 1 % · prev_iqr and r_no < rounds: note "stopped_early_at=r_no"; break
  if blinded_skeptic: verdict ← estimate(skeptic, packet + final table only)     appended (round rounds_run+1)
  transcript.rounds ← rounds actually run; notes ← stopping, flags, verdict
  finalize

cost_matched_single(packet, k, model, pool)                          # J
  samples ← gather(estimate(generalist) × k)                          same prompt; vendor sampling provides the variation
  chain: self_critique(sample_1) → self_refine                        2 extra turns
  final = sample_01..sample_k + self_refine; aggregated += samples_mean, samples_median, self_refine

finalize: final point/quantiles per role, aggregated (rules of §6), aggregated_interval ([lo90, hi90] for median-of-quantiles and mixture), totals (tokens, estimated cost, wall-clock latency of the property). Failed turns are excluded from the committee state and from aggregation; the transcript keeps them.

Committees (debate/committees.py): assign(roles, committee, seed, tx_id, tier) → role → model id from configs/models.yaml cost_matched_pairs[tier]; mixed_random = balanced permutation seeded by (seed, tx_id), recorded in transcript.assignment.

Stopping (debate/stopping.py): relative change of the IQR of log points < 1 %; recorded, never used to choose the best round.

# 5. Prompt-caching layout

Vendors cache a stable prefix. Per role the system prompt is constant across properties (≈ 1,100–1,180 tokens by the ¼-character heuristic; 837 of them are the shared preamble) and carries the Anthropic cache_control breakpoint (providers/anthropic_provider.py); OpenAI caches the prefix automatically. Everything that varies — packet view, committee state, flags, task line — is in the user message. Consequences: (i) the 12 role prompts are cached after the first property of a run; (ii) critique and revision turns of the same role within a property share the system prefix but not the user content; (iii) the mock counts tokens as if nothing were cached, so its costs are upper bounds for input tokens.

# 6. Aggregation (debate/aggregation.py)

All in log space, converted back. Point rules: mean, median, trimmed_mean_20 (20 % trimmed from each tail, scipy convention), inverse_variance (weights ∝ 1/Var of each role's validation log errors), stacking (NNLS of log price on the roles' log points, weights normalised to sum 1), robust_bayes (Student-t, 4 df, location by IRLS, scale from validation). Intervals: median_of_quantiles (default) and mixture (equal-weight mixture of the roles' distributions, quantile functions interpolated in log space). fit_weights(valid_transcripts, truth) → results/aggregation_weights.json, applied unchanged to test/holdout. Without weights the runner uses equal weights (inverse_variance, stacking) and a MAD scale (robust_bayes) and writes weights_source in the manifest — never silent.

predictions.parquet has one row per rule plus a mixture row (point = mixture median, quantiles = mixture quantiles); the other rows carry the median-of-quantiles interval. Arm J adds samples_mean, samples_median, self_refine rows.

# 7. Runner, artefacts, freeze

av run-arm --arm G_debate --rounds 2 --committee mixed_random --split validation --n 24 [--live] [--tier mid] [--run-id …] [--synthetic] → results/runs/<run_id>/{predictions.parquet, transcripts/<tx_id>.json, packets/<tx_id>.json, manifest.json}. The arm label in predictions spells out the rounds grid of G (G_debate_r1/r2/r4), as the evaluation layer and the preregistration expect; H and I keep their fixed 2 rounds. Properties are valued 4 at a time (--concurrency), agents of a round concurrently, provider semaphores (6 per model) bound the calls. Resumable (existing transcripts skipped, predictions rebuilt from all transcripts). Mock unless --live and keys are present; mock/synthetic run ids start with _ and are skipped by scripts/20_evaluate.py. AV_MAX_SPEND_USD → the ledger raises, the runner stops, writes what exists and sets spend_cap_hit in the manifest. Subsample: the first n (sorted by tx_id) of data/<base>/llm_subsamples.parquet.

av freeze → results/freeze.json (SHA-256 of the 33 prompts, the 4 configs, schemas.py, agents/, debate/, providers/, model ids, treatments); av freeze --check lists deviations. av providers-check lists both vendors' models through the SDKs and verifies every id of configs/models.yaml (2026-10-27: all 9 found).

# 8. Cost per property — estimates, not invoices

Token counts below are mock counts (characters ÷ 4, system prompt counted on every call, no caching) observed on the synthetic smoke runs of 2026-10-27; real tokenisers differ (JSON with many numbers usually tokenises less favourably), vendors bill actual usage, and prompt caching lowers the input side. Prices are the list prices of configs/models.yaml (mid tier $2 / $10 per M tokens; small tier $0.10 / $0.50).

Reading: with repeats: 12, J is token-matched to F, not to G2 (as configs/treatments.yaml describes); matching G2 would need k ≈ 70 samples or a longer self-refine chain — a decision for the pilot. Scaling: validation tier (240 properties × 3 committees) ≈ $55 for F, ≈ $350 for G2 at mid tier before caching; configs/budgets.yaml assumed 3.5 k input / 0.7 k output per turn, i.e. more output than the mock produces, so real costs may be higher on the output side (reasoning models) and lower on the input side (caching). The pilot tier (24 properties) is the place to calibrate these numbers; the ledger (results/ledger/<run_id>.jsonl) records every call.