SPB Git

spb/llmindex Public

The discriminative, contamination-resistant, fully transparent LLM ranking — updated live.

webappaillmbenchmark
TypeScript 77.9% TeX 15.2% Python 3.7% SQL 1.4% JavaScript 1.1% Shell 0.5%
12.4 KB
View source Raw Blame History

# CLAUDE.md — LLM Index Platform (www.llmindex.io)

This file provides guidance to Claude Code when working in this repository.

Author: Simon-Pierre Boucher — Contact: contact@spboucher.ai

# 1. Project Overview

LLM Index (llmindex.io) is a public platform that ranks large language models globally and per task/subject domain, with a methodology built around novel, highly discriminative metrics — not yet another static leaderboard on saturated benchmarks.

Core product promise: "The most discriminative, contamination-resistant, fully transparent LLM ranking."

# What makes the methodology "legendary"

Classic leaderboards fail because top models cluster at 95%+ on saturated benchmarks (no discrimination) and test sets leak into training data (contamination). This index is designed to fix both:

  1. IRT-based scoring (Item Response Theory) — every question has an estimated difficulty and a discrimination parameter; questions that fail to separate strong from weak models are automatically down-weighted or retired. Model ability (θ) is estimated via a 2PL model, not raw accuracy.
  2. Dynamic item generation — evaluation items are generated/perturbed programmatically (templated variants, paraphrases, value substitutions) so no fixed test set can be memorized. Contamination resistance is measured explicitly (fixed-vs-perturbed accuracy gap = contamination_delta).
  3. Pairwise Bradley-Terry / Elo layer — for open-ended tasks (writing, reasoning explanations), LLM-judged pairwise duels feed a Bradley-Terry model with judge-bias correction (position swap, style-length normalization).
  4. Consistency score — same item asked k times / in k paraphrases; variance of answers is a first-class metric (consistency), because a model that flips answers is less trustworthy at equal accuracy.
  5. Calibration — models must express confidence; Brier score / ECE per domain (calibration).
  6. Efficiency frontier — score-per-dollar and score-per-second (via OpenRouter pricing + measured latency), rendered as a Pareto frontier, never as a single mashed-up number.

# Score structure

  • Global Index (0–1000) — IRT ability rescaled, with confidence interval.
  • Per-domain scores — domains in packages/scoring/src/domains.ts (v1: code, math, reasoning, writing, knowledge, multilingual, instruction_following, safety_refusal_quality).
  • Sub-metrics per domain — accuracy_irt, consistency, calibration, contamination_delta, latency_p50, cost_per_1k_items.
  • All scores stored with CI bounds (score_low, score, score_high) and an INDEX_VERSION.
  • Weights and IRT hyperparameters live in packages/scoring/src/weights.ts — never hardcode weights elsewhere.
  • Every methodology change bumps INDEX_VERSION (semver) + changelog entry in docs/methodology/CHANGELOG.md.

# 2. Tech Stack

  • Monorepo: pnpm workspaces + Turborepo
  • Language: TypeScript everywhere (strict). Python 3.12 in apps/psychometrics for IRT fitting (py-irt / custom 2PL with PyTorch).
  • Frontend: Next.js 14 (App Router), React 18, Tailwind CSS, shadcn/ui, Recharts (rankings, Pareto frontiers, radar charts per domain)
  • API: Next.js route handlers for public API (/api/v1/*) + tRPC internal
  • Database: PostgreSQL 16 (Prisma). Heavy tables: eval_items, model_responses, pairwise_duels, score_runs.
  • Cache: Redis (leaderboard reads, rate limiting)
  • Jobs: BullMQ workers in apps/worker (eval batches, duel scheduling, IRT refit triggers)
  • Model access: OpenRouter API only (see §6)
  • Testing: Vitest, Playwright (e2e), pytest (psychometrics)
  • Lint/format: ESLint + Prettier, ruff. CI fails on warnings.

# 3. Repository Layout

text
llmindex/
├── apps/
│   ├── web/             # Next.js site + public API
│   ├── worker/          # BullMQ workers: eval runner, duel runner, judges
│   └── psychometrics/   # Python: IRT fitting, Bradley-Terry, calibration calc
├── packages/
│   ├── scoring/         # Pure TS aggregation of fitted parameters → scores (NO I/O)
│   ├── items/           # Item bank: templates, generators, perturbation engine
│   ├── openrouter/      # Typed OpenRouter client (retry, cost tracking, streaming)
│   ├── db/              # Prisma schema + client + seeds
│   ├── ui/              # Shared components
│   └── config/          # Shared eslint/ts/tailwind configs
├── data/
│   ├── item-bank/       # Versioned item templates (committed) — never raw answers in web app
│   └── runs/            # Run manifests (hashes, counts) — payloads live in DB
├── docs/
│   └── methodology/     # Public methodology, changelog, IRT spec, judge protocol
├── infra/               # deploy.sh, ngrok.yml, systemd units, docker-compose
└── CLAUDE.md

# 4. Mandatory File Header Convention

Every source file created or substantially modified (TS, TSX, Python, SQL migrations, shell scripts) must begin with this header (adapted to the language's comment syntax):

ts
/**
 * llmindex.io — <short file purpose>
 * Author:  Simon-Pierre Boucher
 * Contact: contact@spboucher.ai
 * License: Proprietary — © Simon-Pierre Boucher, all rights reserved
 */

Python / shell:

python
# llmindex.io — <short file purpose>
# Author:  Simon-Pierre Boucher
# Contact: contact@spboucher.ai
# License: Proprietary — © Simon-Pierre Boucher, all rights reserved
  • The header is enforced by pnpm lint:headers (script in infra/check-headers.mjs) and runs in CI. New files without the header fail CI.
  • Do not add the header to generated files (.next/, Prisma client, lockfiles) or JSON/Markdown data files.
  • The site footer and public API responses also credit: "maintainer": "Simon-Pierre Boucher <contact@spboucher.ai>".

# 5. Commands

bash
pnpm install                    # install all workspaces
pnpm dev                        # web on :3000
pnpm build                      # turbo build all
pnpm test                       # vitest
pnpm test:e2e                   # playwright
pnpm lint && pnpm typecheck     # must pass pre-commit
pnpm lint:headers               # verify author headers on all source files
pnpm db:migrate && pnpm db:seed
pnpm eval:run --model <slug> --domain <domain> --n 200   # launch eval batch
pnpm duel:run --domain writing --pairs 500               # pairwise duels
pnpm index:refit                # trigger IRT + BT refit, writes new score run
cd apps/psychometrics && make fit   # standalone psychometric fitting

# 6. OpenRouter Integration Rules

All model calls go through packages/openrouter — never call model provider APIs directly.

  • Base URL: https://openrouter.ai/api/v1 (OpenAI-compatible chat completions). Key from OPENROUTER_API_KEY env var; never hardcode, log, or commit it.
  • Required headers on every request: Authorization: Bearer ..., HTTP-Referer: https://www.llmindex.io, X-Title: LLM Index.
  • Model slugs (e.g. anthropic/claude-..., openai/..., google/...) come from the models DB table, synced daily from OpenRouter's /models endpoint (which also gives pricing → feeds cost_per_1k_items). Never hardcode model slugs in source.
  • Every response stores: model slug, exact request params (temperature, max_tokens, seed if supported), raw response, token usage, measured latency, cost. Full audit trail — a ranking without stored raw responses is invalid.
  • Eval defaults: temperature: 0 for scored items; consistency runs use the model's default temperature, k=5 samples.
  • Retries: exponential backoff on 429/5xx, max 5; a model failing >2% of a batch flags the run degraded (shown in UI, excluded from ranking until re-run).
  • Cost guardrails: MAX_RUN_COST_USD env cap per batch; the worker refuses to start a batch whose estimated cost exceeds it.
  • Judge models for pairwise duels are configured in apps/worker/src/judges/config.ts: always ≥2 judge models from different providers, position-swapped, with agreement rate logged. A model never judges duels involving itself.

# 7. Evaluation Integrity Rules (non-negotiable)

  1. Item bank secrecy: answer keys and grading rubrics never ship to the client bundle or public API. Templates are public (methodology transparency); instantiated items + keys stay server-side.
  2. Perturbation before every run: scored batches use freshly perturbed items; the fixed "anchor" subset (for longitudinal comparability) is ≤20% of any run.
  3. No score without a run: every displayed number traces to a score_runs row (model set, item set hash, INDEX_VERSION, fit diagnostics). Historical runs are immutable.
  4. Discrimination hygiene: after each refit, items with discrimination a < 0.3 or |difficulty| beyond ±3 logits are auto-flagged for retirement review.
  5. Judge bias reporting: every duel-based score publishes judge agreement, position-bias rate, and length-bias correlation in the methodology dashboard.
  6. If code and docs/methodology/METHODOLOGY.md disagree — stop and flag, don't silently pick one.

# 8. Public API & UI Rules

  • /api/v1/leaderboard — global ranking (paginated, cached 1h), includes index_version, CI bounds
  • /api/v1/leaderboard/:domain — per-domain ranking with sub-metrics
  • /api/v1/models/:slug — full profile: radar chart data, Pareto position, run history
  • /api/v1/methodology — machine-readable weights, IRT hyperparams, version
  • Rate limit: 60 req/min anonymous, 600 with API key; 429 + Retry-After.
  • Breaking changes ⇒ /api/v2, never mutate v1 shapes.
  • UI must always show uncertainty (CI whiskers) and never present the efficiency frontier as a single blended score.
  • Rankings pages are SSG/ISR (revalidate 1h) for SEO; model comparison pages target "model A vs model B" queries.

# 9. Environments & Deployment

  • local — Postgres + Redis via infra/docker-compose.dev.yml (no SQLite shortcut).
  • staging/prod — node m3u96b, exposed publicly via ngrok mapped to www.llmindex.io (same infra pattern as the airiskindex deployment on this node; separate ports and services).

Process layout on m3u96b:

  • llmindex-web.service (systemd) → Next.js standalone on 127.0.0.1:3100
  • llmindex-worker.service → BullMQ workers (eval/duel/refit)
  • Postgres + Redis via infra/docker-compose.prod.yml (dedicated DB llmindex)
  • ngrok agent: one config with multiple endpoints if the node also serves other sites

ngrok endpoint (infra/ngrok.yml fragment, merged into /etc/ngrok/ngrok.yml):

yaml
endpoints:
  - name: llmindex
    url: https://www.llmindex.io   # custom domain added in ngrok dashboard + DNS CNAME
    upstream:
      url: 3100

Deploy (infra/deploy.sh):

bash
ssh m3u96b
cd /srv/llmindex
git pull --ff-only origin main
pnpm install --frozen-lockfile
pnpm build
pnpm db:migrate:deploy
sudo systemctl restart llmindex-web llmindex-worker
curl -fsS http://127.0.0.1:3100/api/v1/health     # local check
curl -fsS https://www.llmindex.io/api/v1/health   # tunnel check — mandatory

Env vars (/srv/llmindex/.env, template infra/.env.example):

text
DATABASE_URL=postgresql://...
REDIS_URL=redis://...
OPENROUTER_API_KEY=...
MAX_RUN_COST_USD=50
NGROK_AUTHTOKEN=...
PUBLIC_BASE_URL=https://www.llmindex.io
MAINTAINER_NAME="Simon-Pierre Boucher"
MAINTAINER_EMAIL=contact@spboucher.ai

Never print, log, or commit secrets.

Rollback: git checkout <previous-tag> → rebuild → restart services. DB: forward-fix migrations only; nightly pg_dump to /srv/backups (14-day retention).

# 10. When Working in This Repo, Claude Should

  • Run pnpm typecheck && pnpm lint && pnpm lint:headers && pnpm test before declaring any task done.
  • Add the Simon-Pierre Boucher header (§4) to every new source file, matched to the language's comment syntax.
  • Touch packages/scoring, IRT hyperparameters, or judge config only with tests + a methodology changelog entry.
  • Ask before: destructive migrations, changing weights/domains, editing ngrok/systemd config, changing public API shapes, or launching eval batches that could exceed the cost cap.
  • Never expose answer keys, judge prompts with rubrics, or raw API keys through the web app or public API.
  • Prefer conventional commits (feat(items): ..., fix(openrouter): ..., infra(deploy): ...).
  • When uncertain about OpenRouter specifics (endpoints, model slugs, pricing fields), fetch their live docs//models endpoint rather than guessing.