spb/llmindex Public

1# CLAUDE.md — LLM Index Platform (www.llmindex.io)23This file provides guidance to Claude Code when working in this repository.45> **Author:** Simon-Pierre Boucher — **Contact:** contact@spboucher.ai67---89## 1. Project Overview1011**LLM Index** (llmindex.io) is a public platform that ranks large language models **globally and per task/subject domain**, with a methodology built around **novel, highly discriminative metrics** — not yet another static leaderboard on saturated benchmarks.1213**Core product promise:** "The most discriminative, contamination-resistant, fully transparent LLM ranking."1415### What makes the methodology "legendary"16Classic leaderboards fail because top models cluster at 95%+ on saturated benchmarks (no discrimination) and test sets leak into training data (contamination). This index is designed to fix both:17181. **IRT-based scoring (Item Response Theory)** — every question has an estimated difficulty and a **discrimination parameter**; questions that fail to separate strong from weak models are automatically down-weighted or retired. Model ability (θ) is estimated via a 2PL model, not raw accuracy.192. **Dynamic item generation** — evaluation items are generated/perturbed programmatically (templated variants, paraphrases, value substitutions) so no fixed test set can be memorized. Contamination resistance is measured explicitly (fixed-vs-perturbed accuracy gap = `contamination_delta`).203. **Pairwise Bradley-Terry / Elo layer** — for open-ended tasks (writing, reasoning explanations), LLM-judged pairwise duels feed a Bradley-Terry model with judge-bias correction (position swap, style-length normalization).214. **Consistency score** — same item asked k times / in k paraphrases; variance of answers is a first-class metric (`consistency`), because a model that flips answers is less trustworthy at equal accuracy.225. **Calibration** — models must express confidence; Brier score / ECE per domain (`calibration`).236. **Efficiency frontier** — score-per-dollar and score-per-second (via OpenRouter pricing + measured latency), rendered as a Pareto frontier, never as a single mashed-up number.2425### Score structure26- **Global Index (0–1000)** — IRT ability rescaled, with confidence interval.27- **Per-domain scores** — domains in `packages/scoring/src/domains.ts` (v1: `code`, `math`, `reasoning`, `writing`, `knowledge`, `multilingual`, `instruction_following`, `safety_refusal_quality`).28- **Sub-metrics per domain** — `accuracy_irt`, `consistency`, `calibration`, `contamination_delta`, `latency_p50`, `cost_per_1k_items`.29- All scores stored with CI bounds (`score_low`, `score`, `score_high`) and an `INDEX_VERSION`.30- Weights and IRT hyperparameters live in `packages/scoring/src/weights.ts` — **never hardcode weights elsewhere**.31- Every methodology change bumps `INDEX_VERSION` (semver) + changelog entry in `docs/methodology/CHANGELOG.md`.3233---3435## 2. Tech Stack3637- **Monorepo:** pnpm workspaces + Turborepo38- **Language:** TypeScript everywhere (strict). Python 3.12 in `apps/psychometrics` for IRT fitting (`py-irt` / custom 2PL with PyTorch).39- **Frontend:** Next.js 14 (App Router), React 18, Tailwind CSS, shadcn/ui, Recharts (rankings, Pareto frontiers, radar charts per domain)40- **API:** Next.js route handlers for public API (`/api/v1/*`) + tRPC internal41- **Database:** PostgreSQL 16 (Prisma). Heavy tables: `eval_items`, `model_responses`, `pairwise_duels`, `score_runs`.42- **Cache:** Redis (leaderboard reads, rate limiting)43- **Jobs:** BullMQ workers in `apps/worker` (eval batches, duel scheduling, IRT refit triggers)44- **Model access:** **OpenRouter API only** (see §6)45- **Testing:** Vitest, Playwright (e2e), pytest (psychometrics)46- **Lint/format:** ESLint + Prettier, ruff. CI fails on warnings.4748## 3. Repository Layout4950```51llmindex/52├── apps/53│   ├── web/             # Next.js site + public API54│   ├── worker/          # BullMQ workers: eval runner, duel runner, judges55│   └── psychometrics/   # Python: IRT fitting, Bradley-Terry, calibration calc56├── packages/57│   ├── scoring/         # Pure TS aggregation of fitted parameters → scores (NO I/O)58│   ├── items/           # Item bank: templates, generators, perturbation engine59│   ├── openrouter/      # Typed OpenRouter client (retry, cost tracking, streaming)60│   ├── db/              # Prisma schema + client + seeds61│   ├── ui/              # Shared components62│   └── config/          # Shared eslint/ts/tailwind configs63├── data/64│   ├── item-bank/       # Versioned item templates (committed) — never raw answers in web app65│   └── runs/            # Run manifests (hashes, counts) — payloads live in DB66├── docs/67│   └── methodology/     # Public methodology, changelog, IRT spec, judge protocol68├── infra/               # deploy.sh, ngrok.yml, systemd units, docker-compose69└── CLAUDE.md70```7172## 4. Mandatory File Header Convention7374**Every source file created or substantially modified** (TS, TSX, Python, SQL migrations, shell scripts) must begin with this header (adapted to the language's comment syntax):7576```ts77/**78 * llmindex.io — <short file purpose>79 * Author:  Simon-Pierre Boucher80 * Contact: contact@spboucher.ai81 * License: Proprietary — © Simon-Pierre Boucher, all rights reserved82 */83```8485Python / shell:86```python87# llmindex.io — <short file purpose>88# Author:  Simon-Pierre Boucher89# Contact: contact@spboucher.ai90# License: Proprietary — © Simon-Pierre Boucher, all rights reserved91```9293- The header is enforced by `pnpm lint:headers` (script in `infra/check-headers.mjs`) and runs in CI. New files without the header fail CI.94- Do not add the header to generated files (`.next/`, Prisma client, lockfiles) or JSON/Markdown data files.95- The site footer and public API responses also credit: `"maintainer": "Simon-Pierre Boucher <contact@spboucher.ai>"`.9697## 5. Commands9899```bash100pnpm install                    # install all workspaces101pnpm dev                        # web on :3000102pnpm build                      # turbo build all103pnpm test                       # vitest104pnpm test:e2e                   # playwright105pnpm lint && pnpm typecheck     # must pass pre-commit106pnpm lint:headers               # verify author headers on all source files107pnpm db:migrate && pnpm db:seed108pnpm eval:run --model <slug> --domain <domain> --n 200   # launch eval batch109pnpm duel:run --domain writing --pairs 500               # pairwise duels110pnpm index:refit                # trigger IRT + BT refit, writes new score run111cd apps/psychometrics && make fit   # standalone psychometric fitting112```113114## 6. OpenRouter Integration Rules115116All model calls go through `packages/openrouter` — never call model provider APIs directly.117118- Base URL: `https://openrouter.ai/api/v1` (OpenAI-compatible chat completions). Key from `OPENROUTER_API_KEY` env var; never hardcode, log, or commit it.119- Required headers on every request: `Authorization: Bearer ...`, `HTTP-Referer: https://www.llmindex.io`, `X-Title: LLM Index`.120- Model slugs (e.g. `anthropic/claude-...`, `openai/...`, `google/...`) come from the `models` DB table, synced daily from OpenRouter's `/models` endpoint (which also gives pricing → feeds `cost_per_1k_items`). **Never hardcode model slugs in source.**121- Every response stores: model slug, exact request params (temperature, max_tokens, seed if supported), raw response, token usage, measured latency, cost. Full audit trail — a ranking without stored raw responses is invalid.122- Eval defaults: `temperature: 0` for scored items; consistency runs use the model's default temperature, k=5 samples.123- Retries: exponential backoff on 429/5xx, max 5; a model failing >2% of a batch flags the run `degraded` (shown in UI, excluded from ranking until re-run).124- Cost guardrails: `MAX_RUN_COST_USD` env cap per batch; the worker refuses to start a batch whose estimated cost exceeds it.125- Judge models for pairwise duels are configured in `apps/worker/src/judges/config.ts`: always ≥2 judge models from **different providers**, position-swapped, with agreement rate logged. A model never judges duels involving itself.126127## 7. Evaluation Integrity Rules (non-negotiable)1281291. **Item bank secrecy:** answer keys and grading rubrics never ship to the client bundle or public API. Templates are public (methodology transparency); instantiated items + keys stay server-side.1302. **Perturbation before every run:** scored batches use freshly perturbed items; the fixed "anchor" subset (for longitudinal comparability) is ≤20% of any run.1313. **No score without a run:** every displayed number traces to a `score_runs` row (model set, item set hash, `INDEX_VERSION`, fit diagnostics). Historical runs are immutable.1324. **Discrimination hygiene:** after each refit, items with discrimination `a < 0.3` or |difficulty| beyond ±3 logits are auto-flagged for retirement review.1335. **Judge bias reporting:** every duel-based score publishes judge agreement, position-bias rate, and length-bias correlation in the methodology dashboard.1346. If code and `docs/methodology/METHODOLOGY.md` disagree — stop and flag, don't silently pick one.135136## 8. Public API & UI Rules137138- `/api/v1/leaderboard` — global ranking (paginated, cached 1h), includes `index_version`, CI bounds139- `/api/v1/leaderboard/:domain` — per-domain ranking with sub-metrics140- `/api/v1/models/:slug` — full profile: radar chart data, Pareto position, run history141- `/api/v1/methodology` — machine-readable weights, IRT hyperparams, version142- Rate limit: 60 req/min anonymous, 600 with API key; `429` + `Retry-After`.143- Breaking changes ⇒ `/api/v2`, never mutate v1 shapes.144- UI must always show uncertainty (CI whiskers) and never present the efficiency frontier as a single blended score.145- Rankings pages are SSG/ISR (revalidate 1h) for SEO; model comparison pages target "model A vs model B" queries.146147## 9. Environments & Deployment148149- `local` — Postgres + Redis via `infra/docker-compose.dev.yml` (no SQLite shortcut).150- `staging/prod` — node **m3u96b**, exposed publicly via **ngrok** mapped to `www.llmindex.io` (same infra pattern as the airiskindex deployment on this node; separate ports and services).151152**Process layout on m3u96b:**153- `llmindex-web.service` (systemd) → Next.js standalone on `127.0.0.1:3100`154- `llmindex-worker.service` → BullMQ workers (eval/duel/refit)155- Postgres + Redis via `infra/docker-compose.prod.yml` (dedicated DB `llmindex`)156- `ngrok` agent: one config with **multiple endpoints** if the node also serves other sites157158**ngrok endpoint (`infra/ngrok.yml` fragment, merged into `/etc/ngrok/ngrok.yml`):**159```yaml160endpoints:161  - name: llmindex162    url: https://www.llmindex.io   # custom domain added in ngrok dashboard + DNS CNAME163    upstream:164      url: 3100165```166167**Deploy (`infra/deploy.sh`):**168```bash169ssh m3u96b170cd /srv/llmindex171git pull --ff-only origin main172pnpm install --frozen-lockfile173pnpm build174pnpm db:migrate:deploy175sudo systemctl restart llmindex-web llmindex-worker176curl -fsS http://127.0.0.1:3100/api/v1/health     # local check177curl -fsS https://www.llmindex.io/api/v1/health   # tunnel check — mandatory178```179180**Env vars** (`/srv/llmindex/.env`, template `infra/.env.example`):181```182DATABASE_URL=postgresql://...183REDIS_URL=redis://...184OPENROUTER_API_KEY=...185MAX_RUN_COST_USD=50186NGROK_AUTHTOKEN=...187PUBLIC_BASE_URL=https://www.llmindex.io188MAINTAINER_NAME="Simon-Pierre Boucher"189MAINTAINER_EMAIL=contact@spboucher.ai190```191Never print, log, or commit secrets.192193**Rollback:** `git checkout <previous-tag>` → rebuild → restart services. DB: forward-fix migrations only; nightly `pg_dump` to `/srv/backups` (14-day retention).194195## 10. When Working in This Repo, Claude Should196197- Run `pnpm typecheck && pnpm lint && pnpm lint:headers && pnpm test` before declaring any task done.198- Add the Simon-Pierre Boucher header (§4) to every new source file, matched to the language's comment syntax.199- Touch `packages/scoring`, IRT hyperparameters, or judge config only with tests + a methodology changelog entry.200- Ask before: destructive migrations, changing weights/domains, editing ngrok/systemd config, changing public API shapes, or launching eval batches that could exceed the cost cap.201- Never expose answer keys, judge prompts with rubrics, or raw API keys through the web app or public API.202- Prefer conventional commits (`feat(items): ...`, `fix(openrouter): ...`, `infra(deploy): ...`).203- When uncertain about OpenRouter specifics (endpoints, model slugs, pricing fields), fetch their live docs/`/models` endpoint rather than guessing.204