# Agents, deliberation protocols and aggregation (WP 26) Code: `src/agentic_valuation/agents/` (roles, packets, runner), `src/agentic_valuation/debate/` (committees, protocols, stopping, aggregation), `src/agentic_valuation/experiments/` (run_arm, freeze, providers_check). CLI: `av run-arm`, `av freeze`, `av providers-check`. Tests: `tests/test_agents_*.py`, `tests/test_debate_*.py` (mock provider, synthetic data, < 1 s). Every role is a **simulated analytic role**, not a licensed Quebec appraiser; the prompts say so. Nothing here is a professional opinion of value. ## 1. Roles The system prompt of a role is `SHARED_PREAMBLE` + `ROLE_ID: ` + role text + `OUTPUT_RULES` (`agents/roles.py`). The preamble states the task (market value in CAD as of the valuation date, a transaction-price prediction, not a fundamental value), the information cutoff, the evidence discipline (cite ids, no invented inspection, keep missing data missing, traceable adjustments) and the JSON-only answer format. Prompts are stable constants; their SHA-256 is part of `results/freeze.json`. | id | view (least privilege) | description | |---|---|---| | `appraisal_method` | subject, comparables, market | Certified-appraisal-method specialist (sales comparison, cost, income when relevant) | | `comparables` | subject, comparables | Comparable-sales selection and adjustment specialist | | `hedonic` | subject, comparables, market | Hedonic / econometric specialist (implicit prices from the comparable sample) | | `market_economist` | subject, market | Local housing-market economist (price levels, index path, liquidity, assessment ratio) | | `building_condition` | subject | Building-condition specialist; **no inspection available**, condition stays unobserved | | `geospatial_risk` | subject, geo, market | Geospatial, accessibility and environmental-risk specialist | | `underwriting` | all | Mortgage-underwriting and collateral-risk specialist (explicit downside stress) | | `buyer_wtp` | all | Buyer / household willingness-to-pay analyst | | `seller_strategy` | subject, comparables, market | Seller / listing-strategy analyst (expected closing price) | | `rental_usercost` | subject, comparables, market | Rental-income / user-cost analyst (rents unobserved, yields stated as assumptions) | | `data_auditor` | all | Data-quality, leakage and provenance auditor (reports inconsistencies) | | `skeptic` | all | Independent skeptical reviewer / red team | | *optional* `construction_cost` | subject, market | Replacement-cost-less-depreciation cross-check | | *optional* `zoning` | subject, geo | Urban-planning / zoning analyst (conditional reasoning, no by-law assertions) | | *optional* `macro_rates` | subject, market | Macro / mortgage-rate analyst (rate field usually unavailable) | | *optional* `energy` | subject | Energy-efficiency analyst (vintage-based) | | *optional* `climate_risk` | subject, geo | Climate-risk analyst (flood exposure only) | | *optional* `regional_expert` | subject, comparables, market | Regional market expert | | *optional* `uncertainty_auditor` | all | Calibrated-interval auditor | | *optional* `fairness_auditor` | all | Attribute-driven, not area-proxy-driven, valuation | | `generalist` | all | Single analyst (arms E and J) | | `placebo_1..12` | view of default role *k* | Generic prompt, same information partition as F/G (arm I) | `roles_for("default12" | "all20" | "shuffled12" | [ids])`, `placebo_roles(n)`, `shuffled_roles(seed)` (default prompts permuted across the default views — alternative placebo), `role_prompt_hashes()`. ## 2. Evidence packets and views (`agents/packets.py`) `build_packet(analytic_row, comps, market, geo, cfg)` → `EvidencePacket` with `subject` (P), `comparables` (C01..C12, only `sale_date ≤ cutoff`), `market` (M1), `geo` (G1 or null) and notes. Invariants: no sale price, no street/address, lat/lng rounded to 3 decimals, `missing_fields` filled, prior sale dropped unless strictly before the cutoff. A leak assertion runs on every packet. `role_view(packet, role)` keeps only the role's sections (evidence ids preserved, `evidence_ids_visible` listed). `render_user_prompt(view, extra, kind=…, verification_flags=…)` produces: ``` Evidence packet — valuation date …, information cutoff …. Sections visible to your role: …. {view json} {"round":r,"committee_median":…,"estimates":[{role,point,q05,q95,reasoning_summary}…], "evidence_ids_cited_by_others_outside_your_view":[…],"critiques_addressed_to_you":[…]} (rounds ≥ 1) […] (arm H, flagged agents) TASK: … (estimate | critique | revision | skeptic_verdict | self_critique | self_refine) ``` The committee table is anonymised (role ids, never model/provider). Evidence exchange is by **ids only**: a role learns that others cited `G1`, never its content and never the price. `verify_evidence(estimate, packet, allowed_ids=None)` → flags `unknown_evidence_id:`, `unknown_comparable:`, `adjustment_out_of_band:` (adjusted price outside ±40 % of the raw comparable price), `point_outside_comparable_hull` (outside the adjusted-comparable range ±25 %). `faithfulness_summary` gives the counts used by the evaluation layer. ## 3. One turn (`agents/runner.py`) `run_turn(provider, role, view_prompt, schema, model, effort, round, kind)` → `AgentTurn` with tokens, cached tokens, estimated cost, latency, prompt hash, request id, validation retries and `error`. A failed call or a schema failure after ≤ 2 corrective retries yields `estimate=None` + `error`; nothing is fabricated. `SpendCapExceeded` propagates to the runner. `ProviderPool` keeps one adapter per model id (own concurrency semaphore) on a shared ledger; `force_mock=True` unless the run is `--live` with keys. ## 4. Protocols (`debate/protocols.py`) ``` independent(packet, roles, assignment, pool) # F (E with roles=[generalist]) turns ← gather(estimate(r) for r in roles) round 0, concurrent finalize structured_debate(packet, roles, assignment, pool, rounds, blinded_skeptic, evidence_verification, placebo) # G / H / I debaters ← roles minus skeptic if blinded_skeptic turns ← gather(estimate(r) for r in debaters) round 0 flags ← verify(current) if evidence_verification prev_iqr ← IQR(log points) for r_no in 1..rounds: state_r ← committee_state(current, for_role=r, visible_ids(r)) # anonymised table + committee_median + ids cited outside r's view critiques ← gather(critique(r, state_r, flags[r]) for r in debaters) CritiqueSet addressed[target] ← critiques whose target_role = target (critic anonymised) revisions ← gather(revise(r, state_r + addressed[r], flags[r]) for r in debaters) AgentEstimate current ← latest estimate per role; flags ← verify(current) if |IQR − prev_iqr| < 1 % · prev_iqr and r_no < rounds: note "stopped_early_at=r_no"; break if blinded_skeptic: verdict ← estimate(skeptic, packet + final table only) appended (round rounds_run+1) transcript.rounds ← rounds actually run; notes ← stopping, flags, verdict finalize cost_matched_single(packet, k, model, pool) # J samples ← gather(estimate(generalist) × k) same prompt; vendor sampling provides the variation chain: self_critique(sample_1) → self_refine 2 extra turns final = sample_01..sample_k + self_refine; aggregated += samples_mean, samples_median, self_refine ``` `finalize`: final point/quantiles per role, `aggregated` (rules of §6), `aggregated_interval` (`[lo90, hi90]` for median-of-quantiles and mixture), totals (tokens, estimated cost, wall-clock latency of the property). Failed turns are excluded from the committee state and from aggregation; the transcript keeps them. Committees (`debate/committees.py`): `assign(roles, committee, seed, tx_id, tier)` → role → model id from `configs/models.yaml cost_matched_pairs[tier]`; `mixed_random` = balanced permutation seeded by `(seed, tx_id)`, recorded in `transcript.assignment`. Stopping (`debate/stopping.py`): relative change of the IQR of log points < 1 %; recorded, never used to choose the best round. ## 5. Prompt-caching layout Vendors cache a stable prefix. Per role the **system prompt is constant across properties** (≈ 1,100–1,180 tokens by the ¼-character heuristic; 837 of them are the shared preamble) and carries the Anthropic `cache_control` breakpoint (`providers/anthropic_provider.py`); OpenAI caches the prefix automatically. Everything that varies — packet view, committee state, flags, task line — is in the user message. Consequences: (i) the 12 role prompts are cached after the first property of a run; (ii) critique and revision turns of the same role within a property share the system prefix but not the user content; (iii) the mock counts tokens as if nothing were cached, so its costs are upper bounds for input tokens. ## 6. Aggregation (`debate/aggregation.py`) All in log space, converted back. Point rules: `mean`, `median`, `trimmed_mean_20` (20 % trimmed from each tail, scipy convention), `inverse_variance` (weights ∝ 1/Var of each role's validation log errors), `stacking` (NNLS of log price on the roles' log points, weights normalised to sum 1), `robust_bayes` (Student-t, 4 df, location by IRLS, scale from validation). Intervals: `median_of_quantiles` (default) and `mixture` (equal-weight mixture of the roles' distributions, quantile functions interpolated in log space). `fit_weights(valid_transcripts, truth)` → `results/aggregation_weights.json`, applied unchanged to test/holdout. Without weights the runner uses equal weights (inverse_variance, stacking) and a MAD scale (robust_bayes) and writes `weights_source` in the manifest — never silent. `predictions.parquet` has one row per rule plus a `mixture` row (point = mixture median, quantiles = mixture quantiles); the other rows carry the median-of-quantiles interval. Arm J adds `samples_mean`, `samples_median`, `self_refine` rows. ## 7. Runner, artefacts, freeze `av run-arm --arm G_debate --rounds 2 --committee mixed_random --split validation --n 24 [--live] [--tier mid] [--run-id …] [--synthetic]` → `results/runs//{predictions.parquet, transcripts/.json, packets/.json, manifest.json}`. The arm label in predictions spells out the rounds grid of G (`G_debate_r1/r2/r4`), as the evaluation layer and the preregistration expect; H and I keep their fixed 2 rounds. Properties are valued 4 at a time (`--concurrency`), agents of a round concurrently, provider semaphores (6 per model) bound the calls. Resumable (existing transcripts skipped, predictions rebuilt from all transcripts). Mock unless `--live` and keys are present; mock/synthetic run ids start with `_` and are skipped by `scripts/20_evaluate.py`. `AV_MAX_SPEND_USD` → the ledger raises, the runner stops, writes what exists and sets `spend_cap_hit` in the manifest. Subsample: the first *n* (sorted by tx_id) of `data//llm_subsamples.parquet`. `av freeze` → `results/freeze.json` (SHA-256 of the 33 prompts, the 4 configs, `schemas.py`, `agents/`, `debate/`, `providers/`, model ids, treatments); `av freeze --check` lists deviations. `av providers-check` lists both vendors' models through the SDKs and verifies every id of `configs/models.yaml` (2026-10-27: all 9 found). ## 8. Cost per property — estimates, not invoices Token counts below are **mock counts** (characters ÷ 4, system prompt counted on every call, no caching) observed on the synthetic smoke runs of 2026-10-27; real tokenisers differ (JSON with many numbers usually tokenises less favourably), vendors bill actual usage, and prompt caching lowers the input side. Prices are the list prices of `configs/models.yaml` (mid tier $2 / $10 per M tokens; small tier $0.10 / $0.50). | turn kind (12-role committee) | input tok | output tok | mid-tier USD | |---|---|---|---| | estimate (round 0) | ≈ 2,034 | ≈ 182 | ≈ 0.0059 | | critique | ≈ 2,550 | ≈ 441 | ≈ 0.0095 | | revision | ≈ 2,949 | ≈ 182 | ≈ 0.0077 | | arm | turns / property | tokens / property (mock) | mid tier USD | small tier USD | |---|---|---|---|---| | E single generalist | 1 | ≈ 2.4 k | ≈ 0.006 | ≈ 0.0003 | | F independent | 12 | 27–36 k (obs. 26.6–36.4 k) | 0.07–0.09 (obs. 0.071–0.090) | ≈ 0.004 | | G1 | 36 | ≈ 105 k | ≈ 0.28 | ≈ 0.014 | | G2 | 60 | 174–223 k (obs.; one property stopped at round 1: 106 k) | 0.48–0.58 (obs. 0.484–0.583; 0.289 when stopped early) | ≈ 0.025 | | G4 | 108 | ≈ 320 k | ≈ 0.90 | ≈ 0.045 | | H2 (11 debaters + blinded skeptic) | 34 (r1) / 56 (r2) | ≈ 165 k | ≈ 0.45 | ≈ 0.023 | | I2 placebo | 60 | ≈ G2 | ≈ 0.48 | ≈ 0.025 | | J (k = 12 + self-refine) | 14 | ≈ 32 k | ≈ 0.09 | ≈ 0.005 | Reading: with `repeats: 12`, J is token-matched to **F**, not to G2 (as `configs/treatments.yaml` describes); matching G2 would need k ≈ 70 samples or a longer self-refine chain — a decision for the pilot. Scaling: validation tier (240 properties × 3 committees) ≈ $55 for F, ≈ $350 for G2 at mid tier before caching; `configs/budgets.yaml` assumed 3.5 k input / 0.7 k output per turn, i.e. more output than the mock produces, so real costs may be higher on the output side (reasoning models) and lower on the input side (caching). The pilot tier (24 properties) is the place to calibrate these numbers; the ledger (`results/ledger/.jsonl`) records every call.