# Judge Protocol (pairwise duels)

**Author:** Simon-Pierre Boucher — contact@spboucher.ai

Open-ended domains (`writing`, `safety_refusal_quality`) are ranked by pairwise duels judged by
LLMs, feeding a Bradley-Terry model.

## Rules

1. **≥2 judge models from different providers** per duel (configured via the `JUDGE_MODELS` env
   var — validated at runtime; never hardcoded).
2. **No self-judging:** a model never judges a duel it participates in.
3. **Position swap:** judges see the two responses in opposite orders; the swap is recorded on
   every verdict row.
4. **Style-length guard:** the judge prompt explicitly forbids rewarding verbosity or style over
   substance. Length-bias correlation is computed on published runs.
5. **Transparency:** judge agreement rate and position-bias rate are stored in the run's
   `fit_diagnostics` and published on the methodology dashboard.
6. Verdicts are `1`, `2`, or `TIE` (unparseable verdicts count as ties, conservatively).

The exact comparison prompt lives in `apps/worker/src/judges/config.ts` (`judgePrompt`) and is
public. Grading rubrics for *scored items* are never public — but duel judging is rubric-free by
design, so nothing secret ships to judges.

## Aggregation

Ties are encoded as half-wins. Bradley-Terry strengths are fitted with an MM algorithm with light
damping (`apps/psychometrics/llmindex_psycho/bt.py`), log-strengths standardized to a θ-like scale
(`btStrengthToTheta` in `packages/scoring`), then rescaled like any other domain score.