spb/llmindex Public

1# Judge Protocol (pairwise duels)23**Author:** Simon-Pierre Boucher — contact@spboucher.ai45Open-ended domains (`writing`, `safety_refusal_quality`) are ranked by pairwise duels judged by6LLMs, feeding a Bradley-Terry model.78## Rules9101. **≥2 judge models from different providers** per duel (configured via the `JUDGE_MODELS` env11   var — validated at runtime; never hardcoded).122. **No self-judging:** a model never judges a duel it participates in.133. **Position swap:** judges see the two responses in opposite orders; the swap is recorded on14   every verdict row.154. **Style-length guard:** the judge prompt explicitly forbids rewarding verbosity or style over16   substance. Length-bias correlation is computed on published runs.175. **Transparency:** judge agreement rate and position-bias rate are stored in the run's18   `fit_diagnostics` and published on the methodology dashboard.196. Verdicts are `1`, `2`, or `TIE` (unparseable verdicts count as ties, conservatively).2021The exact comparison prompt lives in `apps/worker/src/judges/config.ts` (`judgePrompt`) and is22public. Grading rubrics for *scored items* are never public — but duel judging is rubric-free by23design, so nothing secret ships to judges.2425## Aggregation2627Ties are encoded as half-wins. Bradley-Terry strengths are fitted with an MM algorithm with light28damping (`apps/psychometrics/llmindex_psycho/bt.py`), log-strengths standardized to a θ-like scale29(`btStrengthToTheta` in `packages/scoring`), then rescaled like any other domain score.30