# CLAUDE.md — LLM Index Platform (www.llmindex.io)

This file provides guidance to Claude Code when working in this repository.

> **Author:** Simon-Pierre Boucher — **Contact:** contact@spboucher.ai

---

## 1. Project Overview

**LLM Index** (llmindex.io) is a public platform that ranks large language models **globally and per task/subject domain**, with a methodology built around **novel, highly discriminative metrics** — not yet another static leaderboard on saturated benchmarks.

**Core product promise:** "The most discriminative, contamination-resistant, fully transparent LLM ranking."

### What makes the methodology "legendary"
Classic leaderboards fail because top models cluster at 95%+ on saturated benchmarks (no discrimination) and test sets leak into training data (contamination). This index is designed to fix both:

1. **IRT-based scoring (Item Response Theory)** — every question has an estimated difficulty and a **discrimination parameter**; questions that fail to separate strong from weak models are automatically down-weighted or retired. Model ability (θ) is estimated via a 2PL model, not raw accuracy.
2. **Dynamic item generation** — evaluation items are generated/perturbed programmatically (templated variants, paraphrases, value substitutions) so no fixed test set can be memorized. Contamination resistance is measured explicitly (fixed-vs-perturbed accuracy gap = `contamination_delta`).
3. **Pairwise Bradley-Terry / Elo layer** — for open-ended tasks (writing, reasoning explanations), LLM-judged pairwise duels feed a Bradley-Terry model with judge-bias correction (position swap, style-length normalization).
4. **Consistency score** — same item asked k times / in k paraphrases; variance of answers is a first-class metric (`consistency`), because a model that flips answers is less trustworthy at equal accuracy.
5. **Calibration** — models must express confidence; Brier score / ECE per domain (`calibration`).
6. **Efficiency frontier** — score-per-dollar and score-per-second (via OpenRouter pricing + measured latency), rendered as a Pareto frontier, never as a single mashed-up number.

### Score structure
- **Global Index (0–1000)** — IRT ability rescaled, with confidence interval.
- **Per-domain scores** — domains in `packages/scoring/src/domains.ts` (v1: `code`, `math`, `reasoning`, `writing`, `knowledge`, `multilingual`, `instruction_following`, `safety_refusal_quality`).
- **Sub-metrics per domain** — `accuracy_irt`, `consistency`, `calibration`, `contamination_delta`, `latency_p50`, `cost_per_1k_items`.
- All scores stored with CI bounds (`score_low`, `score`, `score_high`) and an `INDEX_VERSION`.
- Weights and IRT hyperparameters live in `packages/scoring/src/weights.ts` — **never hardcode weights elsewhere**.
- Every methodology change bumps `INDEX_VERSION` (semver) + changelog entry in `docs/methodology/CHANGELOG.md`.

---

## 2. Tech Stack

- **Monorepo:** pnpm workspaces + Turborepo
- **Language:** TypeScript everywhere (strict). Python 3.12 in `apps/psychometrics` for IRT fitting (`py-irt` / custom 2PL with PyTorch).
- **Frontend:** Next.js 14 (App Router), React 18, Tailwind CSS, shadcn/ui, Recharts (rankings, Pareto frontiers, radar charts per domain)
- **API:** Next.js route handlers for public API (`/api/v1/*`) + tRPC internal
- **Database:** PostgreSQL 16 (Prisma). Heavy tables: `eval_items`, `model_responses`, `pairwise_duels`, `score_runs`.
- **Cache:** Redis (leaderboard reads, rate limiting)
- **Jobs:** BullMQ workers in `apps/worker` (eval batches, duel scheduling, IRT refit triggers)
- **Model access:** **OpenRouter API only** (see §6)
- **Testing:** Vitest, Playwright (e2e), pytest (psychometrics)
- **Lint/format:** ESLint + Prettier, ruff. CI fails on warnings.

## 3. Repository Layout

```
llmindex/
├── apps/
│   ├── web/             # Next.js site + public API
│   ├── worker/          # BullMQ workers: eval runner, duel runner, judges
│   └── psychometrics/   # Python: IRT fitting, Bradley-Terry, calibration calc
├── packages/
│   ├── scoring/         # Pure TS aggregation of fitted parameters → scores (NO I/O)
│   ├── items/           # Item bank: templates, generators, perturbation engine
│   ├── openrouter/      # Typed OpenRouter client (retry, cost tracking, streaming)
│   ├── db/              # Prisma schema + client + seeds
│   ├── ui/              # Shared components
│   └── config/          # Shared eslint/ts/tailwind configs
├── data/
│   ├── item-bank/       # Versioned item templates (committed) — never raw answers in web app
│   └── runs/            # Run manifests (hashes, counts) — payloads live in DB
├── docs/
│   └── methodology/     # Public methodology, changelog, IRT spec, judge protocol
├── infra/               # deploy.sh, ngrok.yml, systemd units, docker-compose
└── CLAUDE.md
```

## 4. Mandatory File Header Convention

**Every source file created or substantially modified** (TS, TSX, Python, SQL migrations, shell scripts) must begin with this header (adapted to the language's comment syntax):

```ts
/**
 * llmindex.io — <short file purpose>
 * Author:  Simon-Pierre Boucher
 * Contact: contact@spboucher.ai
 * License: Proprietary — © Simon-Pierre Boucher, all rights reserved
 */
```

Python / shell:
```python
# llmindex.io — <short file purpose>
# Author:  Simon-Pierre Boucher
# Contact: contact@spboucher.ai
# License: Proprietary — © Simon-Pierre Boucher, all rights reserved
```

- The header is enforced by `pnpm lint:headers` (script in `infra/check-headers.mjs`) and runs in CI. New files without the header fail CI.
- Do not add the header to generated files (`.next/`, Prisma client, lockfiles) or JSON/Markdown data files.
- The site footer and public API responses also credit: `"maintainer": "Simon-Pierre Boucher <contact@spboucher.ai>"`.

## 5. Commands

```bash
pnpm install                    # install all workspaces
pnpm dev                        # web on :3000
pnpm build                      # turbo build all
pnpm test                       # vitest
pnpm test:e2e                   # playwright
pnpm lint && pnpm typecheck     # must pass pre-commit
pnpm lint:headers               # verify author headers on all source files
pnpm db:migrate && pnpm db:seed
pnpm eval:run --model <slug> --domain <domain> --n 200   # launch eval batch
pnpm duel:run --domain writing --pairs 500               # pairwise duels
pnpm index:refit                # trigger IRT + BT refit, writes new score run
cd apps/psychometrics && make fit   # standalone psychometric fitting
```

## 6. OpenRouter Integration Rules

All model calls go through `packages/openrouter` — never call model provider APIs directly.

- Base URL: `https://openrouter.ai/api/v1` (OpenAI-compatible chat completions). Key from `OPENROUTER_API_KEY` env var; never hardcode, log, or commit it.
- Required headers on every request: `Authorization: Bearer ...`, `HTTP-Referer: https://www.llmindex.io`, `X-Title: LLM Index`.
- Model slugs (e.g. `anthropic/claude-...`, `openai/...`, `google/...`) come from the `models` DB table, synced daily from OpenRouter's `/models` endpoint (which also gives pricing → feeds `cost_per_1k_items`). **Never hardcode model slugs in source.**
- Every response stores: model slug, exact request params (temperature, max_tokens, seed if supported), raw response, token usage, measured latency, cost. Full audit trail — a ranking without stored raw responses is invalid.
- Eval defaults: `temperature: 0` for scored items; consistency runs use the model's default temperature, k=5 samples.
- Retries: exponential backoff on 429/5xx, max 5; a model failing >2% of a batch flags the run `degraded` (shown in UI, excluded from ranking until re-run).
- Cost guardrails: `MAX_RUN_COST_USD` env cap per batch; the worker refuses to start a batch whose estimated cost exceeds it.
- Judge models for pairwise duels are configured in `apps/worker/src/judges/config.ts`: always ≥2 judge models from **different providers**, position-swapped, with agreement rate logged. A model never judges duels involving itself.

## 7. Evaluation Integrity Rules (non-negotiable)

1. **Item bank secrecy:** answer keys and grading rubrics never ship to the client bundle or public API. Templates are public (methodology transparency); instantiated items + keys stay server-side.
2. **Perturbation before every run:** scored batches use freshly perturbed items; the fixed "anchor" subset (for longitudinal comparability) is ≤20% of any run.
3. **No score without a run:** every displayed number traces to a `score_runs` row (model set, item set hash, `INDEX_VERSION`, fit diagnostics). Historical runs are immutable.
4. **Discrimination hygiene:** after each refit, items with discrimination `a < 0.3` or |difficulty| beyond ±3 logits are auto-flagged for retirement review.
5. **Judge bias reporting:** every duel-based score publishes judge agreement, position-bias rate, and length-bias correlation in the methodology dashboard.
6. If code and `docs/methodology/METHODOLOGY.md` disagree — stop and flag, don't silently pick one.

## 8. Public API & UI Rules

- `/api/v1/leaderboard` — global ranking (paginated, cached 1h), includes `index_version`, CI bounds
- `/api/v1/leaderboard/:domain` — per-domain ranking with sub-metrics
- `/api/v1/models/:slug` — full profile: radar chart data, Pareto position, run history
- `/api/v1/methodology` — machine-readable weights, IRT hyperparams, version
- Rate limit: 60 req/min anonymous, 600 with API key; `429` + `Retry-After`.
- Breaking changes ⇒ `/api/v2`, never mutate v1 shapes.
- UI must always show uncertainty (CI whiskers) and never present the efficiency frontier as a single blended score.
- Rankings pages are SSG/ISR (revalidate 1h) for SEO; model comparison pages target "model A vs model B" queries.

## 9. Environments & Deployment

- `local` — Postgres + Redis via `infra/docker-compose.dev.yml` (no SQLite shortcut).
- `staging/prod` — node **m3u96b**, exposed publicly via **ngrok** mapped to `www.llmindex.io` (same infra pattern as the airiskindex deployment on this node; separate ports and services).

**Process layout on m3u96b:**
- `llmindex-web.service` (systemd) → Next.js standalone on `127.0.0.1:3100`
- `llmindex-worker.service` → BullMQ workers (eval/duel/refit)
- Postgres + Redis via `infra/docker-compose.prod.yml` (dedicated DB `llmindex`)
- `ngrok` agent: one config with **multiple endpoints** if the node also serves other sites

**ngrok endpoint (`infra/ngrok.yml` fragment, merged into `/etc/ngrok/ngrok.yml`):**
```yaml
endpoints:
  - name: llmindex
    url: https://www.llmindex.io   # custom domain added in ngrok dashboard + DNS CNAME
    upstream:
      url: 3100
```

**Deploy (`infra/deploy.sh`):**
```bash
ssh m3u96b
cd /srv/llmindex
git pull --ff-only origin main
pnpm install --frozen-lockfile
pnpm build
pnpm db:migrate:deploy
sudo systemctl restart llmindex-web llmindex-worker
curl -fsS http://127.0.0.1:3100/api/v1/health     # local check
curl -fsS https://www.llmindex.io/api/v1/health   # tunnel check — mandatory
```

**Env vars** (`/srv/llmindex/.env`, template `infra/.env.example`):
```
DATABASE_URL=postgresql://...
REDIS_URL=redis://...
OPENROUTER_API_KEY=...
MAX_RUN_COST_USD=50
NGROK_AUTHTOKEN=...
PUBLIC_BASE_URL=https://www.llmindex.io
MAINTAINER_NAME="Simon-Pierre Boucher"
MAINTAINER_EMAIL=contact@spboucher.ai
```
Never print, log, or commit secrets.

**Rollback:** `git checkout <previous-tag>` → rebuild → restart services. DB: forward-fix migrations only; nightly `pg_dump` to `/srv/backups` (14-day retention).

## 10. When Working in This Repo, Claude Should

- Run `pnpm typecheck && pnpm lint && pnpm lint:headers && pnpm test` before declaring any task done.
- Add the Simon-Pierre Boucher header (§4) to every new source file, matched to the language's comment syntax.
- Touch `packages/scoring`, IRT hyperparameters, or judge config only with tests + a methodology changelog entry.
- Ask before: destructive migrations, changing weights/domains, editing ngrok/systemd config, changing public API shapes, or launching eval batches that could exceed the cost cap.
- Never expose answer keys, judge prompts with rubrics, or raw API keys through the web app or public API.
- Prefer conventional commits (`feat(items): ...`, `fix(openrouter): ...`, `infra(deploy): ...`).
- When uncertain about OpenRouter specifics (endpoints, model slugs, pricing fields), fetch their live docs/`/models` endpoint rather than guessing.