# Judge Protocol (pairwise duels)

Author: Simon-Pierre Boucher — contact@spboucher.ai

Open-ended domains (writing, safety_refusal_quality) are ranked by pairwise duels judged by LLMs, feeding a Bradley-Terry model.

# Rules

1. ≥2 judge models from different providers per duel (configured via the JUDGE_MODELS env var — validated at runtime; never hardcoded).
2. No self-judging: a model never judges a duel it participates in.
3. Position swap: judges see the two responses in opposite orders; the swap is recorded on every verdict row.
4. Style-length guard: the judge prompt explicitly forbids rewarding verbosity or style over substance. Length-bias correlation is computed on published runs.
5. Transparency: judge agreement rate and position-bias rate are stored in the run's fit_diagnostics and published on the methodology dashboard.
6. Verdicts are 1, 2, or TIE (unparseable verdicts count as ties, conservatively).

The exact comparison prompt lives in apps/worker/src/judges/config.ts (judgePrompt) and is public. Grading rubrics for scored items are never public — but duel judging is rubric-free by design, so nothing secret ships to judges.

# Aggregation

Ties are encoded as half-wins. Bradley-Terry strengths are fitted with an MM algorithm with light damping (apps/psychometrics/llmindex_psycho/bt.py), log-strengths standardized to a θ-like scale (btStrengthToTheta in packages/scoring), then rescaled like any other domain score.