feat: LLM Index platform v0.2.0 — discriminative, contamination-resistant live LLM ranking
IRT 2PL + Bradley-Terry scoring over 12 domains with dynamically generated items: hardened math/reasoning/code/knowledge/multilingual/instruction, home-made simulated agentic (+context-load) and terminal benches, SVG logo duels (3-judge cross-provider panel), vision OCR. Robust extraction cascade, parallel evaluation lanes with live-updating leaderboard, per-model full transparency pages, public API v1, detailed methodology with references. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
130 changed files +14,699 −0
added
.claude/scheduled_tasks.lock
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +{"sessionId":"7d353114-c357-4fc8-ac79-41f0f2b4c0c6","pid":60585,"procStart":"Wed Aug 5 07:37:10 2026","acquiredAt":1785917940273} | |
| \ No newline at end of file | ||
added
.gitignore
+18 −0
@@ -0,0 +1,18 @@ | ||
| 1 | +node_modules/ | |
| 2 | +.next/ | |
| 3 | +dist/ | |
| 4 | +.turbo/ | |
| 5 | +*.tsbuildinfo | |
| 6 | +.env | |
| 7 | +.env.* | |
| 8 | +!infra/.env.example | |
| 9 | +.DS_Store | |
| 10 | +__pycache__/ | |
| 11 | +*.pyc | |
| 12 | +.venv/ | |
| 13 | +.pytest_cache/ | |
| 14 | +coverage/ | |
| 15 | +playwright-report/ | |
| 16 | +test-results/ | |
| 17 | +data/runs/*.json | |
| 18 | +!data/runs/README.md | |
added
.prettierrc
+6 −0
@@ -0,0 +1,6 @@ | ||
| 1 | +{ | |
| 2 | + "semi": true, | |
| 3 | + "singleQuote": true, | |
| 4 | + "trailingComma": "all", | |
| 5 | + "printWidth": 100 | |
| 6 | +} | |
added
CLAUDE.md
+203 −0
@@ -0,0 +1,203 @@ | ||
| 1 | +# CLAUDE.md — LLM Index Platform (www.llmindex.io) | |
| 2 | + | |
| 3 | +This file provides guidance to Claude Code when working in this repository. | |
| 4 | + | |
| 5 | +> **Author:** Simon-Pierre Boucher — **Contact:** contact@spboucher.ai | |
| 6 | + | |
| 7 | +--- | |
| 8 | + | |
| 9 | +## 1. Project Overview | |
| 10 | + | |
| 11 | +**LLM Index** (llmindex.io) is a public platform that ranks large language models **globally and per task/subject domain**, with a methodology built around **novel, highly discriminative metrics** — not yet another static leaderboard on saturated benchmarks. | |
| 12 | + | |
| 13 | +**Core product promise:** "The most discriminative, contamination-resistant, fully transparent LLM ranking." | |
| 14 | + | |
| 15 | +### What makes the methodology "legendary" | |
| 16 | +Classic leaderboards fail because top models cluster at 95%+ on saturated benchmarks (no discrimination) and test sets leak into training data (contamination). This index is designed to fix both: | |
| 17 | + | |
| 18 | +1. **IRT-based scoring (Item Response Theory)** — every question has an estimated difficulty and a **discrimination parameter**; questions that fail to separate strong from weak models are automatically down-weighted or retired. Model ability (θ) is estimated via a 2PL model, not raw accuracy. | |
| 19 | +2. **Dynamic item generation** — evaluation items are generated/perturbed programmatically (templated variants, paraphrases, value substitutions) so no fixed test set can be memorized. Contamination resistance is measured explicitly (fixed-vs-perturbed accuracy gap = `contamination_delta`). | |
| 20 | +3. **Pairwise Bradley-Terry / Elo layer** — for open-ended tasks (writing, reasoning explanations), LLM-judged pairwise duels feed a Bradley-Terry model with judge-bias correction (position swap, style-length normalization). | |
| 21 | +4. **Consistency score** — same item asked k times / in k paraphrases; variance of answers is a first-class metric (`consistency`), because a model that flips answers is less trustworthy at equal accuracy. | |
| 22 | +5. **Calibration** — models must express confidence; Brier score / ECE per domain (`calibration`). | |
| 23 | +6. **Efficiency frontier** — score-per-dollar and score-per-second (via OpenRouter pricing + measured latency), rendered as a Pareto frontier, never as a single mashed-up number. | |
| 24 | + | |
| 25 | +### Score structure | |
| 26 | +- **Global Index (0–1000)** — IRT ability rescaled, with confidence interval. | |
| 27 | +- **Per-domain scores** — domains in `packages/scoring/src/domains.ts` (v1: `code`, `math`, `reasoning`, `writing`, `knowledge`, `multilingual`, `instruction_following`, `safety_refusal_quality`). | |
| 28 | +- **Sub-metrics per domain** — `accuracy_irt`, `consistency`, `calibration`, `contamination_delta`, `latency_p50`, `cost_per_1k_items`. | |
| 29 | +- All scores stored with CI bounds (`score_low`, `score`, `score_high`) and an `INDEX_VERSION`. | |
| 30 | +- Weights and IRT hyperparameters live in `packages/scoring/src/weights.ts` — **never hardcode weights elsewhere**. | |
| 31 | +- Every methodology change bumps `INDEX_VERSION` (semver) + changelog entry in `docs/methodology/CHANGELOG.md`. | |
| 32 | + | |
| 33 | +--- | |
| 34 | + | |
| 35 | +## 2. Tech Stack | |
| 36 | + | |
| 37 | +- **Monorepo:** pnpm workspaces + Turborepo | |
| 38 | +- **Language:** TypeScript everywhere (strict). Python 3.12 in `apps/psychometrics` for IRT fitting (`py-irt` / custom 2PL with PyTorch). | |
| 39 | +- **Frontend:** Next.js 14 (App Router), React 18, Tailwind CSS, shadcn/ui, Recharts (rankings, Pareto frontiers, radar charts per domain) | |
| 40 | +- **API:** Next.js route handlers for public API (`/api/v1/*`) + tRPC internal | |
| 41 | +- **Database:** PostgreSQL 16 (Prisma). Heavy tables: `eval_items`, `model_responses`, `pairwise_duels`, `score_runs`. | |
| 42 | +- **Cache:** Redis (leaderboard reads, rate limiting) | |
| 43 | +- **Jobs:** BullMQ workers in `apps/worker` (eval batches, duel scheduling, IRT refit triggers) | |
| 44 | +- **Model access:** **OpenRouter API only** (see §6) | |
| 45 | +- **Testing:** Vitest, Playwright (e2e), pytest (psychometrics) | |
| 46 | +- **Lint/format:** ESLint + Prettier, ruff. CI fails on warnings. | |
| 47 | + | |
| 48 | +## 3. Repository Layout | |
| 49 | + | |
| 50 | +``` | |
| 51 | +llmindex/ | |
| 52 | +├── apps/ | |
| 53 | +│ ├── web/ # Next.js site + public API | |
| 54 | +│ ├── worker/ # BullMQ workers: eval runner, duel runner, judges | |
| 55 | +│ └── psychometrics/ # Python: IRT fitting, Bradley-Terry, calibration calc | |
| 56 | +├── packages/ | |
| 57 | +│ ├── scoring/ # Pure TS aggregation of fitted parameters → scores (NO I/O) | |
| 58 | +│ ├── items/ # Item bank: templates, generators, perturbation engine | |
| 59 | +│ ├── openrouter/ # Typed OpenRouter client (retry, cost tracking, streaming) | |
| 60 | +│ ├── db/ # Prisma schema + client + seeds | |
| 61 | +│ ├── ui/ # Shared components | |
| 62 | +│ └── config/ # Shared eslint/ts/tailwind configs | |
| 63 | +├── data/ | |
| 64 | +│ ├── item-bank/ # Versioned item templates (committed) — never raw answers in web app | |
| 65 | +│ └── runs/ # Run manifests (hashes, counts) — payloads live in DB | |
| 66 | +├── docs/ | |
| 67 | +│ └── methodology/ # Public methodology, changelog, IRT spec, judge protocol | |
| 68 | +├── infra/ # deploy.sh, ngrok.yml, systemd units, docker-compose | |
| 69 | +└── CLAUDE.md | |
| 70 | +``` | |
| 71 | + | |
| 72 | +## 4. Mandatory File Header Convention | |
| 73 | + | |
| 74 | +**Every source file created or substantially modified** (TS, TSX, Python, SQL migrations, shell scripts) must begin with this header (adapted to the language's comment syntax): | |
| 75 | + | |
| 76 | +```ts | |
| 77 | +/** | |
| 78 | + * llmindex.io — <short file purpose> | |
| 79 | + * Author: Simon-Pierre Boucher | |
| 80 | + * Contact: contact@spboucher.ai | |
| 81 | + * License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 82 | + */ | |
| 83 | +``` | |
| 84 | + | |
| 85 | +Python / shell: | |
| 86 | +```python | |
| 87 | +# llmindex.io — <short file purpose> | |
| 88 | +# Author: Simon-Pierre Boucher | |
| 89 | +# Contact: contact@spboucher.ai | |
| 90 | +# License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 91 | +``` | |
| 92 | + | |
| 93 | +- The header is enforced by `pnpm lint:headers` (script in `infra/check-headers.mjs`) and runs in CI. New files without the header fail CI. | |
| 94 | +- Do not add the header to generated files (`.next/`, Prisma client, lockfiles) or JSON/Markdown data files. | |
| 95 | +- The site footer and public API responses also credit: `"maintainer": "Simon-Pierre Boucher <contact@spboucher.ai>"`. | |
| 96 | + | |
| 97 | +## 5. Commands | |
| 98 | + | |
| 99 | +```bash | |
| 100 | +pnpm install # install all workspaces | |
| 101 | +pnpm dev # web on :3000 | |
| 102 | +pnpm build # turbo build all | |
| 103 | +pnpm test # vitest | |
| 104 | +pnpm test:e2e # playwright | |
| 105 | +pnpm lint && pnpm typecheck # must pass pre-commit | |
| 106 | +pnpm lint:headers # verify author headers on all source files | |
| 107 | +pnpm db:migrate && pnpm db:seed | |
| 108 | +pnpm eval:run --model <slug> --domain <domain> --n 200 # launch eval batch | |
| 109 | +pnpm duel:run --domain writing --pairs 500 # pairwise duels | |
| 110 | +pnpm index:refit # trigger IRT + BT refit, writes new score run | |
| 111 | +cd apps/psychometrics && make fit # standalone psychometric fitting | |
| 112 | +``` | |
| 113 | + | |
| 114 | +## 6. OpenRouter Integration Rules | |
| 115 | + | |
| 116 | +All model calls go through `packages/openrouter` — never call model provider APIs directly. | |
| 117 | + | |
| 118 | +- Base URL: `https://openrouter.ai/api/v1` (OpenAI-compatible chat completions). Key from `OPENROUTER_API_KEY` env var; never hardcode, log, or commit it. | |
| 119 | +- Required headers on every request: `Authorization: Bearer ...`, `HTTP-Referer: https://www.llmindex.io`, `X-Title: LLM Index`. | |
| 120 | +- Model slugs (e.g. `anthropic/claude-...`, `openai/...`, `google/...`) come from the `models` DB table, synced daily from OpenRouter's `/models` endpoint (which also gives pricing → feeds `cost_per_1k_items`). **Never hardcode model slugs in source.** | |
| 121 | +- Every response stores: model slug, exact request params (temperature, max_tokens, seed if supported), raw response, token usage, measured latency, cost. Full audit trail — a ranking without stored raw responses is invalid. | |
| 122 | +- Eval defaults: `temperature: 0` for scored items; consistency runs use the model's default temperature, k=5 samples. | |
| 123 | +- Retries: exponential backoff on 429/5xx, max 5; a model failing >2% of a batch flags the run `degraded` (shown in UI, excluded from ranking until re-run). | |
| 124 | +- Cost guardrails: `MAX_RUN_COST_USD` env cap per batch; the worker refuses to start a batch whose estimated cost exceeds it. | |
| 125 | +- Judge models for pairwise duels are configured in `apps/worker/src/judges/config.ts`: always ≥2 judge models from **different providers**, position-swapped, with agreement rate logged. A model never judges duels involving itself. | |
| 126 | + | |
| 127 | +## 7. Evaluation Integrity Rules (non-negotiable) | |
| 128 | + | |
| 129 | +1. **Item bank secrecy:** answer keys and grading rubrics never ship to the client bundle or public API. Templates are public (methodology transparency); instantiated items + keys stay server-side. | |
| 130 | +2. **Perturbation before every run:** scored batches use freshly perturbed items; the fixed "anchor" subset (for longitudinal comparability) is ≤20% of any run. | |
| 131 | +3. **No score without a run:** every displayed number traces to a `score_runs` row (model set, item set hash, `INDEX_VERSION`, fit diagnostics). Historical runs are immutable. | |
| 132 | +4. **Discrimination hygiene:** after each refit, items with discrimination `a < 0.3` or |difficulty| beyond ±3 logits are auto-flagged for retirement review. | |
| 133 | +5. **Judge bias reporting:** every duel-based score publishes judge agreement, position-bias rate, and length-bias correlation in the methodology dashboard. | |
| 134 | +6. If code and `docs/methodology/METHODOLOGY.md` disagree — stop and flag, don't silently pick one. | |
| 135 | + | |
| 136 | +## 8. Public API & UI Rules | |
| 137 | + | |
| 138 | +- `/api/v1/leaderboard` — global ranking (paginated, cached 1h), includes `index_version`, CI bounds | |
| 139 | +- `/api/v1/leaderboard/:domain` — per-domain ranking with sub-metrics | |
| 140 | +- `/api/v1/models/:slug` — full profile: radar chart data, Pareto position, run history | |
| 141 | +- `/api/v1/methodology` — machine-readable weights, IRT hyperparams, version | |
| 142 | +- Rate limit: 60 req/min anonymous, 600 with API key; `429` + `Retry-After`. | |
| 143 | +- Breaking changes ⇒ `/api/v2`, never mutate v1 shapes. | |
| 144 | +- UI must always show uncertainty (CI whiskers) and never present the efficiency frontier as a single blended score. | |
| 145 | +- Rankings pages are SSG/ISR (revalidate 1h) for SEO; model comparison pages target "model A vs model B" queries. | |
| 146 | + | |
| 147 | +## 9. Environments & Deployment | |
| 148 | + | |
| 149 | +- `local` — Postgres + Redis via `infra/docker-compose.dev.yml` (no SQLite shortcut). | |
| 150 | +- `staging/prod` — node **m3u96b**, exposed publicly via **ngrok** mapped to `www.llmindex.io` (same infra pattern as the airiskindex deployment on this node; separate ports and services). | |
| 151 | + | |
| 152 | +**Process layout on m3u96b:** | |
| 153 | +- `llmindex-web.service` (systemd) → Next.js standalone on `127.0.0.1:3100` | |
| 154 | +- `llmindex-worker.service` → BullMQ workers (eval/duel/refit) | |
| 155 | +- Postgres + Redis via `infra/docker-compose.prod.yml` (dedicated DB `llmindex`) | |
| 156 | +- `ngrok` agent: one config with **multiple endpoints** if the node also serves other sites | |
| 157 | + | |
| 158 | +**ngrok endpoint (`infra/ngrok.yml` fragment, merged into `/etc/ngrok/ngrok.yml`):** | |
| 159 | +```yaml | |
| 160 | +endpoints: | |
| 161 | + - name: llmindex | |
| 162 | + url: https://www.llmindex.io # custom domain added in ngrok dashboard + DNS CNAME | |
| 163 | + upstream: | |
| 164 | + url: 3100 | |
| 165 | +``` | |
| 166 | + | |
| 167 | +**Deploy (`infra/deploy.sh`):** | |
| 168 | +```bash | |
| 169 | +ssh m3u96b | |
| 170 | +cd /srv/llmindex | |
| 171 | +git pull --ff-only origin main | |
| 172 | +pnpm install --frozen-lockfile | |
| 173 | +pnpm build | |
| 174 | +pnpm db:migrate:deploy | |
| 175 | +sudo systemctl restart llmindex-web llmindex-worker | |
| 176 | +curl -fsS http://127.0.0.1:3100/api/v1/health # local check | |
| 177 | +curl -fsS https://www.llmindex.io/api/v1/health # tunnel check — mandatory | |
| 178 | +``` | |
| 179 | + | |
| 180 | +**Env vars** (`/srv/llmindex/.env`, template `infra/.env.example`): | |
| 181 | +``` | |
| 182 | +DATABASE_URL=postgresql://... | |
| 183 | +REDIS_URL=redis://... | |
| 184 | +OPENROUTER_API_KEY=... | |
| 185 | +MAX_RUN_COST_USD=50 | |
| 186 | +NGROK_AUTHTOKEN=... | |
| 187 | +PUBLIC_BASE_URL=https://www.llmindex.io | |
| 188 | +MAINTAINER_NAME="Simon-Pierre Boucher" | |
| 189 | +MAINTAINER_EMAIL=contact@spboucher.ai | |
| 190 | +``` | |
| 191 | +Never print, log, or commit secrets. | |
| 192 | + | |
| 193 | +**Rollback:** `git checkout <previous-tag>` → rebuild → restart services. DB: forward-fix migrations only; nightly `pg_dump` to `/srv/backups` (14-day retention). | |
| 194 | + | |
| 195 | +## 10. When Working in This Repo, Claude Should | |
| 196 | + | |
| 197 | +- Run `pnpm typecheck && pnpm lint && pnpm lint:headers && pnpm test` before declaring any task done. | |
| 198 | +- Add the Simon-Pierre Boucher header (§4) to every new source file, matched to the language's comment syntax. | |
| 199 | +- Touch `packages/scoring`, IRT hyperparameters, or judge config only with tests + a methodology changelog entry. | |
| 200 | +- Ask before: destructive migrations, changing weights/domains, editing ngrok/systemd config, changing public API shapes, or launching eval batches that could exceed the cost cap. | |
| 201 | +- Never expose answer keys, judge prompts with rubrics, or raw API keys through the web app or public API. | |
| 202 | +- Prefer conventional commits (`feat(items): ...`, `fix(openrouter): ...`, `infra(deploy): ...`). | |
| 203 | +- When uncertain about OpenRouter specifics (endpoints, model slugs, pricing fields), fetch their live docs/`/models` endpoint rather than guessing. | |
added
README.md
+190 −0
@@ -0,0 +1,190 @@ | ||
| 1 | +<div align="center"> | |
| 2 | + | |
| 3 | +<img src="docs/assets/logo.svg" width="110" alt="LLM Index logo" /> | |
| 4 | + | |
| 5 | +# LLM Index | |
| 6 | + | |
| 7 | +**The discriminative, contamination-resistant, fully transparent LLM ranking — updated live.** | |
| 8 | + | |
| 9 | +[**www.llmindex.io**](https://www.llmindex.io) · [Methodology](https://www.llmindex.io/methodology) · [Public API](https://www.llmindex.io/api/v1/leaderboard) | |
| 10 | + | |
| 11 | +<!-- status pastilles --> | |
| 12 | +[](https://www.llmindex.io) | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | +<!-- metric pastilles --> | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | +<!-- stack pastilles --> | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | +</div> | |
| 41 | + | |
| 42 | +--- | |
| 43 | + | |
| 44 | +## Why another index? Because leaderboards are broken. | |
| 45 | + | |
| 46 | +Classic leaderboards fail twice: **top models cluster at 95%+ on saturated benchmarks** (zero | |
| 47 | +discrimination), and **fixed test sets leak into training data** (contamination). LLM Index is | |
| 48 | +engineered against both, from the psychometrics up: | |
| 49 | + | |
| 50 | +| | | | |
| 51 | +|---|---| | |
| 52 | +| 🎯 **IRT 2PL scoring** | Every item has a fitted difficulty *b* and discrimination *a*; ability θ is a MAP estimate with Fisher-information standard errors. Items that don't separate models are auto-retired. Raw accuracy is never the score. | | |
| 53 | +| 🎲 **Dynamic item generation** | Every scored batch is freshly generated from seeded template generators (values, paraphrases, structures). There is no fixed test set to memorize — the fixed-vs-fresh gap is *published* per model as `contamination_delta`. | | |
| 54 | +| 🤖 **Home-made agentic bench** | Simulated tool-calling environments (triage-under-policy, treasury ledger, deployment DAGs) with distractor tools; a deterministic simulator computes the unique correct call sequence — graded by canonical-JSON equality, no judges. | | |
| 55 | +| 🧠 **Agentic under context load** | 120–300-row generated ledgers packed with near-miss decoys: the model must find, order, and act on the few matching records. Prompt length is the difficulty knob. | | |
| 56 | +| 💻 **Home-made terminal bench** | No shell executes: a closed, unambiguous POSIX subset is simulated in TypeScript. Models predict exact pipeline stdout, file trees after `mv/cp/rm/cd`, and `&&`/`\|\|` exit-code traces. | | |
| 57 | +| 🎨 **SVG logo duels** | Models reproduce real-world logos in raw SVG from memory; a 3-judge cross-provider panel (position-swapped, never self-judging) feeds a Bradley-Terry fit. | | |
| 58 | +| 👁 **Vision OCR under clutter** | Generated scenes (rotated codes, noise, low-contrast, decoys) rasterized to PNG; ambiguous glyphs excluded by design. Text-only models skip; weights renormalize. | | |
| 59 | +| 🛡 **Extraction that never cheats models** | Lenient cascade (`ANSWER:` in any markdown, `FINAL ANSWER`, `\boxed{}`, fenced blocks), number normalization, 16k completion budgets, truncated ≠ wrong. Formatting is measured in its own domain — never silently everywhere. | | |
| 60 | +| 📡 **Live, one model at a time** | Parallel evaluation lanes stream results; after every completed model the IRT refit re-runs and the public leaderboard re-ranks in real time. | | |
| 61 | +| 🔍 **Total transparency** | Every model has a page showing **every answer on every test**, judge verdicts, confidence, latency and cost. Every number traces to an immutable score run. Answer keys never leave the server. | | |
| 62 | + | |
| 63 | +## Screenshots | |
| 64 | + | |
| 65 | +<div align="center"> | |
| 66 | + | |
| 67 | +**Live leaderboard — desktop** | |
| 68 | + | |
| 69 | +<img src="docs/assets/home-desktop.png" width="860" alt="LLM Index home, live leaderboard" /> | |
| 70 | + | |
| 71 | +<table> | |
| 72 | +<tr> | |
| 73 | +<td align="center" width="300"> | |
| 74 | + | |
| 75 | +**Mobile** | |
| 76 | + | |
| 77 | +<img src="docs/assets/home-mobile.png" width="240" alt="Mobile view" /> | |
| 78 | + | |
| 79 | +</td> | |
| 80 | +<td align="center"> | |
| 81 | + | |
| 82 | +**Model transparency page** | |
| 83 | + | |
| 84 | +<img src="docs/assets/model-page.png" width="560" alt="Model profile with every answer" /> | |
| 85 | + | |
| 86 | +</td> | |
| 87 | +</tr> | |
| 88 | +</table> | |
| 89 | + | |
| 90 | +**Methodology — public, detailed, with live sample items and full references** | |
| 91 | + | |
| 92 | +<img src="docs/assets/methodology.png" width="860" alt="Methodology page" /> | |
| 93 | + | |
| 94 | +</div> | |
| 95 | + | |
| 96 | +## The 12 domains | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | +Domain weights are **equal by design** — the maximum-entropy prior; any other weighting is an | |
| 112 | +editorial value judgment. Per-domain scores are always published so you can re-weight. Cost and | |
| 113 | +latency are **never** blended into quality: they live on a separate Pareto frontier. | |
| 114 | + | |
| 115 | +## Architecture | |
| 116 | + | |
| 117 | +``` | |
| 118 | +llmindex/ | |
| 119 | +├── apps/ | |
| 120 | +│ ├── web/ # Next.js 14 — live leaderboard, transparency pages, /api/v1/* | |
| 121 | +│ ├── worker/ # eval runner · duel runner · parallel benchmark orchestrator · refit | |
| 122 | +│ └── psychometrics/ # Python: 2PL IRT (MAP + Fisher SE), Bradley-Terry (MM + SE), calibration | |
| 123 | +├── packages/ | |
| 124 | +│ ├── scoring/ # domains, weights, INDEX_VERSION, pure score aggregation (no I/O) | |
| 125 | +│ ├── items/ # 25 template generators + simulators + robust grading cascade | |
| 126 | +│ ├── openrouter/ # typed client: retries, timeouts, cost tracking, multimodal | |
| 127 | +│ └── db/ # Prisma: models, eval_items, model_responses, pairwise_duels, score_runs | |
| 128 | +├── docs/methodology/ # METHODOLOGY, CHANGELOG (semver), IRT_SPEC, JUDGE_PROTOCOL | |
| 129 | +└── data/item-bank/ # public template manifest (instantiated items stay server-side) | |
| 130 | +``` | |
| 131 | + | |
| 132 | +**Pipeline:** OpenRouter catalog sync → seeded batch generation → parallel evaluation lanes | |
| 133 | +(temperature 0 scored + k consistency samples, full audit trail: raw responses, tokens, latency, | |
| 134 | +cost) → lenient extraction → 2PL fit per domain + Bradley-Terry for duels → scores with 95% CIs → | |
| 135 | +leaderboard updates live → discrimination hygiene auto-flags dead items. | |
| 136 | + | |
| 137 | +## Public API | |
| 138 | + | |
| 139 | +```bash | |
| 140 | +curl https://www.llmindex.io/api/v1/leaderboard # global ranking + CIs | |
| 141 | +curl https://www.llmindex.io/api/v1/leaderboard/agentic # per-domain + sub-metrics | |
| 142 | +curl https://www.llmindex.io/api/v1/models/anthropic/claude-sonnet-5 | |
| 143 | +curl https://www.llmindex.io/api/v1/methodology # machine-readable weights & hyperparams | |
| 144 | +curl https://www.llmindex.io/api/v1/benchmark/progress # live run status | |
| 145 | +``` | |
| 146 | + | |
| 147 | +Rate limits: 60 req/min anonymous, 600 with an API key. Breaking changes ship as `/api/v2` — v1 | |
| 148 | +shapes are frozen. | |
| 149 | + | |
| 150 | +## Run it | |
| 151 | + | |
| 152 | +```bash | |
| 153 | +pnpm install | |
| 154 | +pnpm db:migrate && pnpm db:seed # sync models + pricing from OpenRouter | |
| 155 | +pnpm dev # web on :3000 | |
| 156 | + | |
| 157 | +pnpm eval:run --model <slug> --domain math --n 30 --k 2 | |
| 158 | +pnpm duel:run --domain svg_design --pairs 100 | |
| 159 | +pnpm benchmark:run --parallel 6 # full live benchmark, refit after every model | |
| 160 | +pnpm index:refit # IRT + BT refit → new immutable score run | |
| 161 | + | |
| 162 | +pnpm typecheck && pnpm lint && pnpm lint:headers && pnpm test # all gates | |
| 163 | +``` | |
| 164 | + | |
| 165 | +## Integrity rules (non-negotiable) | |
| 166 | + | |
| 167 | +1. Answer keys and rubrics never ship to the client or public API. | |
| 168 | +2. Scored batches use freshly perturbed items; the frozen anchor subset is ≤20% of any run. | |
| 169 | +3. Every displayed number traces to an immutable `score_runs` row (item-set hash, model set, | |
| 170 | + index version, fit diagnostics). Batches with >2% failed calls are excluded until re-run. | |
| 171 | +4. Weights and IRT hyperparameters live in exactly one file and are served machine-readable. | |
| 172 | +5. Every methodology change bumps the semver `INDEX_VERSION` with a public changelog entry. | |
| 173 | + | |
| 174 | +The design draws on 60+ published sources (metabench, GSM-Symbolic, ZebraLogic, R-Horizon, | |
| 175 | +τ-bench, BFCL, Terminal-Bench audits, Math-Verify, ReasonIF, …) — full list with links on the | |
| 176 | +[methodology page](https://www.llmindex.io/methodology). Everything here is original: our own | |
| 177 | +environments, items, simulators and graders. | |
| 178 | + | |
| 179 | +--- | |
| 180 | + | |
| 181 | +<div align="center"> | |
| 182 | + | |
| 183 | +**Author & maintainer:** Simon-Pierre Boucher · [contact@spboucher.ai](mailto:contact@spboucher.ai) | |
| 184 | + | |
| 185 | +License: Proprietary — © Simon-Pierre Boucher, all rights reserved. Source visible for | |
| 186 | +transparency and auditability. | |
| 187 | + | |
| 188 | +<img src="docs/assets/logo.svg" width="28" alt="" /> | |
| 189 | + | |
| 190 | +</div> | |
added
apps/psychometrics/Makefile
+19 −0
@@ -0,0 +1,19 @@ | ||
| 1 | +# llmindex.io — psychometrics make targets | |
| 2 | +# Author: Simon-Pierre Boucher | |
| 3 | +# Contact: contact@spboucher.ai | |
| 4 | +# License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 5 | + | |
| 6 | +PY ?= python3 | |
| 7 | +INPUT ?= ../../data/runs/latest-input.json | |
| 8 | +OUTPUT ?= ../../data/runs/latest-output.json | |
| 9 | + | |
| 10 | +.PHONY: fit test install | |
| 11 | + | |
| 12 | +install: | |
| 13 | + $(PY) -m pip install -r requirements.txt | |
| 14 | + | |
| 15 | +fit: | |
| 16 | + $(PY) fit.py --input $(INPUT) --output $(OUTPUT) | |
| 17 | + | |
| 18 | +test: | |
| 19 | + $(PY) -m pytest -q | |
added
apps/psychometrics/fit.py
+95 −0
@@ -0,0 +1,95 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +# llmindex.io — fit CLI: response matrices JSON → fitted 2PL parameters JSON | |
| 3 | +# Author: Simon-Pierre Boucher | |
| 4 | +# Contact: contact@spboucher.ai | |
| 5 | +# License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 6 | +# | |
| 7 | +# Pure compute: reads the matrix file written by the worker (no DB access), | |
| 8 | +# fits a 2PL per domain, writes thetas/SEs and item parameters back to JSON. | |
| 9 | + | |
| 10 | +from __future__ import annotations | |
| 11 | + | |
| 12 | +import argparse | |
| 13 | +import json | |
| 14 | +import sys | |
| 15 | + | |
| 16 | +import numpy as np | |
| 17 | + | |
| 18 | +from llmindex_psycho.bt import fit_bradley_terry | |
| 19 | +from llmindex_psycho.irt import fit_2pl | |
| 20 | + | |
| 21 | + | |
| 22 | +def main() -> int: | |
| 23 | + parser = argparse.ArgumentParser(description="llmindex 2PL fit") | |
| 24 | + parser.add_argument("--input", required=True) | |
| 25 | + parser.add_argument("--output", required=True) | |
| 26 | + args = parser.parse_args() | |
| 27 | + | |
| 28 | + with open(args.input) as f: | |
| 29 | + payload = json.load(f) | |
| 30 | + | |
| 31 | + hyper = payload.get("hyperparams", {}) | |
| 32 | + max_iter = int(hyper.get("maxIterations", 500)) | |
| 33 | + tol = float(hyper.get("tolerance", 1e-6)) | |
| 34 | + | |
| 35 | + out: dict = {"domains": {}} | |
| 36 | + for domain, data in payload.get("domains", {}).items(): | |
| 37 | + models = data["models"] | |
| 38 | + items = data["items"] | |
| 39 | + matrix = np.array( | |
| 40 | + [[np.nan if v is None else float(v) for v in row] for row in data["responses"]], | |
| 41 | + dtype=float, | |
| 42 | + ) | |
| 43 | + result = fit_2pl(matrix, max_iterations=max_iter, tolerance=tol) | |
| 44 | + out["domains"][domain] = { | |
| 45 | + "models": [ | |
| 46 | + {"slug": slug, "theta": float(t), "se": float(se)} | |
| 47 | + for slug, t, se in zip(models, result.theta, result.theta_se) | |
| 48 | + ], | |
| 49 | + "items": [ | |
| 50 | + {"id": item_id, "a": float(a), "b": float(b)} | |
| 51 | + for item_id, a, b in zip(items, result.a, result.b) | |
| 52 | + ], | |
| 53 | + "diagnostics": { | |
| 54 | + "iterations": result.iterations, | |
| 55 | + "converged": result.converged, | |
| 56 | + "final_loglik": result.final_loglik, | |
| 57 | + **result.diagnostics, | |
| 58 | + }, | |
| 59 | + } | |
| 60 | + print( | |
| 61 | + f"[fit] {domain}: {len(models)} models × {len(items)} items — " | |
| 62 | + f"{'converged' if result.converged else 'max iterations'} @ {result.iterations}", | |
| 63 | + file=sys.stderr, | |
| 64 | + ) | |
| 65 | + | |
| 66 | + # Judged duel domains: Bradley-Terry over wins matrices; log-strengths are | |
| 67 | + # standardized to a theta-like scale so downstream scoring is uniform. | |
| 68 | + out["duel_domains"] = {} | |
| 69 | + for domain, data in payload.get("duel_domains", {}).items(): | |
| 70 | + models = data["models"] | |
| 71 | + wins = np.array(data["wins"], dtype=float) | |
| 72 | + result = fit_bradley_terry(wins) | |
| 73 | + sd = result.log_strength.std() or 1.0 | |
| 74 | + theta = result.log_strength / sd | |
| 75 | + theta_se = result.log_strength_se / sd | |
| 76 | + out["duel_domains"][domain] = { | |
| 77 | + "models": [ | |
| 78 | + {"slug": slug, "theta": float(t), "se": float(min(se, 3.0))} | |
| 79 | + for slug, t, se in zip(models, theta, theta_se) | |
| 80 | + ], | |
| 81 | + "diagnostics": { | |
| 82 | + "iterations": result.iterations, | |
| 83 | + "converged": result.converged, | |
| 84 | + "comparisons": float(wins.sum()), | |
| 85 | + }, | |
| 86 | + } | |
| 87 | + print(f"[fit] duel {domain}: {len(models)} models, {wins.sum():.0f} comparisons", file=sys.stderr) | |
| 88 | + | |
| 89 | + with open(args.output, "w") as f: | |
| 90 | + json.dump(out, f, indent=2) | |
| 91 | + return 0 | |
| 92 | + | |
| 93 | + | |
| 94 | +if __name__ == "__main__": | |
| 95 | + raise SystemExit(main()) | |
added
apps/psychometrics/llmindex_psycho/__init__.py
+8 −0
@@ -0,0 +1,8 @@ | ||
| 1 | +# llmindex.io — psychometrics package (IRT, Bradley-Terry, calibration) | |
| 2 | +# Author: Simon-Pierre Boucher | |
| 3 | +# Contact: contact@spboucher.ai | |
| 4 | +# License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 5 | + | |
| 6 | +from .irt import fit_2pl # noqa: F401 | |
| 7 | +from .bt import fit_bradley_terry # noqa: F401 | |
| 8 | +from .calibration import brier_score, expected_calibration_error # noqa: F401 | |
added
apps/psychometrics/llmindex_psycho/bt.py
+61 −0
@@ -0,0 +1,61 @@ | ||
| 1 | +# llmindex.io — Bradley-Terry fit (MM algorithm, ties as half-wins) | |
| 2 | +# Author: Simon-Pierre Boucher | |
| 3 | +# Contact: contact@spboucher.ai | |
| 4 | +# License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 5 | + | |
| 6 | +from __future__ import annotations | |
| 7 | + | |
| 8 | +from dataclasses import dataclass | |
| 9 | + | |
| 10 | +import numpy as np | |
| 11 | + | |
| 12 | + | |
| 13 | +@dataclass | |
| 14 | +class BTResult: | |
| 15 | + log_strength: np.ndarray # (M,) log-strengths, mean 0 | |
| 16 | + log_strength_se: np.ndarray # (M,) SEs from the Fisher information | |
| 17 | + iterations: int | |
| 18 | + converged: bool | |
| 19 | + | |
| 20 | + | |
| 21 | +def fit_bradley_terry( | |
| 22 | + wins: np.ndarray, | |
| 23 | + max_iterations: int = 1000, | |
| 24 | + tolerance: float = 1e-10, | |
| 25 | + damping: float = 0.1, | |
| 26 | +) -> BTResult: | |
| 27 | + """Fit Bradley-Terry strengths from a wins matrix. | |
| 28 | + | |
| 29 | + wins[i, j] = (possibly fractional) number of wins of i over j. | |
| 30 | + Ties should be pre-encoded as 0.5 win to each side. | |
| 31 | + `damping` adds a tiny uniform prior so isolated models stay finite. | |
| 32 | + """ | |
| 33 | + W = np.asarray(wins, dtype=float) | |
| 34 | + if W.ndim != 2 or W.shape[0] != W.shape[1]: | |
| 35 | + raise ValueError("wins must be a square matrix") | |
| 36 | + M = W.shape[0] | |
| 37 | + # Regularization: everyone gets `damping` phantom wins vs everyone else. | |
| 38 | + W = W + damping * (np.ones((M, M)) - np.eye(M)) | |
| 39 | + N = W + W.T # total comparisons between each pair | |
| 40 | + p = np.ones(M) | |
| 41 | + converged = False | |
| 42 | + it = 0 | |
| 43 | + for it in range(1, max_iterations + 1): | |
| 44 | + denom = (N / (p[:, None] + p[None, :] + 1e-300)).sum(1) - np.diag( | |
| 45 | + N / (p[:, None] + p[None, :] + 1e-300) | |
| 46 | + ) | |
| 47 | + new_p = W.sum(1) / np.maximum(denom, 1e-300) | |
| 48 | + new_p = new_p / np.exp(np.log(new_p + 1e-300).mean()) # geometric-mean normalize | |
| 49 | + if np.max(np.abs(new_p - p)) < tolerance: | |
| 50 | + p = new_p | |
| 51 | + converged = True | |
| 52 | + break | |
| 53 | + p = new_p | |
| 54 | + log_strength = np.log(p + 1e-300) | |
| 55 | + log_strength -= log_strength.mean() | |
| 56 | + # Fisher information of log-strengths: I_ii = sum_j N_ij * p_ij * (1 - p_ij) | |
| 57 | + # where p_ij = p_i / (p_i + p_j); SE_i = 1 / sqrt(I_ii). | |
| 58 | + P = p[:, None] / (p[:, None] + p[None, :] + 1e-300) | |
| 59 | + info = (N * P * (1 - P)).sum(1) - np.diag(N * P * (1 - P)) | |
| 60 | + se = 1.0 / np.sqrt(np.maximum(info, 1e-9)) | |
| 61 | + return BTResult(log_strength=log_strength, log_strength_se=se, iterations=it, converged=converged) | |
added
apps/psychometrics/llmindex_psycho/calibration.py
+37 −0
@@ -0,0 +1,37 @@ | ||
| 1 | +# llmindex.io — calibration metrics: Brier score, expected calibration error | |
| 2 | +# Author: Simon-Pierre Boucher | |
| 3 | +# Contact: contact@spboucher.ai | |
| 4 | +# License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 5 | + | |
| 6 | +from __future__ import annotations | |
| 7 | + | |
| 8 | +import numpy as np | |
| 9 | + | |
| 10 | + | |
| 11 | +def brier_score(confidence: np.ndarray, correct: np.ndarray) -> float: | |
| 12 | + """Mean squared error between reported confidence and outcome.""" | |
| 13 | + c = np.asarray(confidence, dtype=float) | |
| 14 | + y = np.asarray(correct, dtype=float) | |
| 15 | + if c.shape != y.shape or c.size == 0: | |
| 16 | + raise ValueError("confidence and correct must be same-shape, non-empty") | |
| 17 | + return float(np.mean((c - y) ** 2)) | |
| 18 | + | |
| 19 | + | |
| 20 | +def expected_calibration_error( | |
| 21 | + confidence: np.ndarray, correct: np.ndarray, bins: int = 10 | |
| 22 | +) -> float: | |
| 23 | + """Standard ECE with equal-width confidence bins.""" | |
| 24 | + c = np.asarray(confidence, dtype=float) | |
| 25 | + y = np.asarray(correct, dtype=float) | |
| 26 | + if c.shape != y.shape or c.size == 0: | |
| 27 | + raise ValueError("confidence and correct must be same-shape, non-empty") | |
| 28 | + edges = np.linspace(0, 1, bins + 1) | |
| 29 | + idx = np.clip(np.digitize(c, edges[1:-1]), 0, bins - 1) | |
| 30 | + ece = 0.0 | |
| 31 | + for b in range(bins): | |
| 32 | + sel = idx == b | |
| 33 | + n = sel.sum() | |
| 34 | + if n == 0: | |
| 35 | + continue | |
| 36 | + ece += (n / c.size) * abs(y[sel].mean() - c[sel].mean()) | |
| 37 | + return float(ece) | |
added
apps/psychometrics/llmindex_psycho/irt.py
+132 −0
@@ -0,0 +1,132 @@ | ||
| 1 | +# llmindex.io — custom 2PL IRT fit (numpy MAP via Adam, missing-aware) | |
| 2 | +# Author: Simon-Pierre Boucher | |
| 3 | +# Contact: contact@spboucher.ai | |
| 4 | +# License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 5 | +# | |
| 6 | +# 2PL: P(correct) = sigmoid(a_i * (theta_m - b_i)) | |
| 7 | +# MAP estimation with priors: | |
| 8 | +# theta ~ N(0, 1) b ~ N(0, 1.5) log a ~ N(0, 0.5) | |
| 9 | +# Missing responses are masked. Theta SE comes from the Fisher information | |
| 10 | +# (plus prior precision). Numpy-based (no torch) so the fit runs anywhere. | |
| 11 | + | |
| 12 | +from __future__ import annotations | |
| 13 | + | |
| 14 | +from dataclasses import dataclass, field | |
| 15 | + | |
| 16 | +import numpy as np | |
| 17 | + | |
| 18 | + | |
| 19 | +def _sigmoid(x: np.ndarray) -> np.ndarray: | |
| 20 | + return 1.0 / (1.0 + np.exp(-np.clip(x, -30, 30))) | |
| 21 | + | |
| 22 | + | |
| 23 | +@dataclass | |
| 24 | +class Fit2PLResult: | |
| 25 | + theta: np.ndarray # (M,) | |
| 26 | + theta_se: np.ndarray # (M,) | |
| 27 | + a: np.ndarray # (I,) discrimination | |
| 28 | + b: np.ndarray # (I,) difficulty (logits) | |
| 29 | + iterations: int | |
| 30 | + converged: bool | |
| 31 | + final_loglik: float | |
| 32 | + diagnostics: dict = field(default_factory=dict) | |
| 33 | + | |
| 34 | + | |
| 35 | +def fit_2pl( | |
| 36 | + responses: np.ndarray, | |
| 37 | + max_iterations: int = 500, | |
| 38 | + tolerance: float = 1e-6, | |
| 39 | + lr: float = 0.05, | |
| 40 | + prior_theta_sd: float = 1.0, | |
| 41 | + prior_b_sd: float = 1.5, | |
| 42 | + prior_log_a_sd: float = 0.5, | |
| 43 | + seed: int = 0, | |
| 44 | +) -> Fit2PLResult: | |
| 45 | + """Fit a 2PL model on an (M models × I items) matrix of {0,1,nan}.""" | |
| 46 | + X = np.asarray(responses, dtype=float) | |
| 47 | + if X.ndim != 2: | |
| 48 | + raise ValueError("responses must be a 2D matrix (models × items)") | |
| 49 | + M, I = X.shape | |
| 50 | + mask = ~np.isnan(X) | |
| 51 | + if mask.sum() == 0: | |
| 52 | + raise ValueError("no observed responses") | |
| 53 | + Xf = np.nan_to_num(X, nan=0.0) | |
| 54 | + | |
| 55 | + rng = np.random.default_rng(seed) | |
| 56 | + # Warm start: theta from row accuracy, b from item difficulty (logit of failure rate). | |
| 57 | + row_acc = np.where(mask.sum(1) > 0, Xf.sum(1) / np.maximum(mask.sum(1), 1), 0.5) | |
| 58 | + col_acc = np.where(mask.sum(0) > 0, Xf.sum(0) / np.maximum(mask.sum(0), 1), 0.5) | |
| 59 | + theta = np.clip(np.log(row_acc + 1e-3) - np.log(1 - row_acc + 1e-3), -2, 2) | |
| 60 | + theta = theta - theta.mean() | |
| 61 | + b = np.clip(-(np.log(col_acc + 1e-3) - np.log(1 - col_acc + 1e-3)), -2.5, 2.5) | |
| 62 | + log_a = rng.normal(0.0, 0.01, size=I) | |
| 63 | + | |
| 64 | + # Adam state | |
| 65 | + params = [theta, b, log_a] | |
| 66 | + m_state = [np.zeros_like(p) for p in params] | |
| 67 | + v_state = [np.zeros_like(p) for p in params] | |
| 68 | + beta1, beta2, eps = 0.9, 0.999, 1e-8 | |
| 69 | + | |
| 70 | + prev_obj = -np.inf | |
| 71 | + converged = False | |
| 72 | + it = 0 | |
| 73 | + for it in range(1, max_iterations + 1): | |
| 74 | + a = np.exp(log_a) | |
| 75 | + Z = a[None, :] * (theta[:, None] - b[None, :]) | |
| 76 | + P = _sigmoid(Z) | |
| 77 | + R = np.where(mask, Xf - P, 0.0) # residuals on observed cells | |
| 78 | + | |
| 79 | + g_theta = (R * a[None, :]).sum(1) - theta / prior_theta_sd**2 | |
| 80 | + g_b = (-(R * a[None, :])).sum(0) - b / prior_b_sd**2 | |
| 81 | + g_a = (R * (theta[:, None] - b[None, :])).sum(0) | |
| 82 | + g_log_a = g_a * a - log_a / prior_log_a_sd**2 | |
| 83 | + | |
| 84 | + grads = [g_theta, g_b, g_log_a] | |
| 85 | + for j, (p, g) in enumerate(zip(params, grads)): | |
| 86 | + m_state[j] = beta1 * m_state[j] + (1 - beta1) * g | |
| 87 | + v_state[j] = beta2 * v_state[j] + (1 - beta2) * g * g | |
| 88 | + mhat = m_state[j] / (1 - beta1**it) | |
| 89 | + vhat = v_state[j] / (1 - beta2**it) | |
| 90 | + p += lr * mhat / (np.sqrt(vhat) + eps) | |
| 91 | + | |
| 92 | + # Identification: center abilities each step (scale is pinned by priors). | |
| 93 | + shift = theta.mean() | |
| 94 | + theta -= shift | |
| 95 | + b -= shift | |
| 96 | + | |
| 97 | + with np.errstate(divide="ignore", invalid="ignore"): | |
| 98 | + ll = np.where(mask, Xf * np.log(P + 1e-12) + (1 - Xf) * np.log(1 - P + 1e-12), 0.0).sum() | |
| 99 | + obj = ( | |
| 100 | + ll | |
| 101 | + - 0.5 * (theta**2).sum() / prior_theta_sd**2 | |
| 102 | + - 0.5 * (b**2).sum() / prior_b_sd**2 | |
| 103 | + - 0.5 * (log_a**2).sum() / prior_log_a_sd**2 | |
| 104 | + ) | |
| 105 | + if abs(obj - prev_obj) < tolerance * (1 + abs(prev_obj)): | |
| 106 | + converged = True | |
| 107 | + break | |
| 108 | + prev_obj = obj | |
| 109 | + | |
| 110 | + a = np.exp(log_a) | |
| 111 | + Z = a[None, :] * (theta[:, None] - b[None, :]) | |
| 112 | + P = _sigmoid(Z) | |
| 113 | + info = (mask * (a[None, :] ** 2) * P * (1 - P)).sum(1) + 1.0 / prior_theta_sd**2 | |
| 114 | + theta_se = 1.0 / np.sqrt(info) | |
| 115 | + final_ll = float( | |
| 116 | + np.where(mask, Xf * np.log(P + 1e-12) + (1 - Xf) * np.log(1 - P + 1e-12), 0.0).sum() | |
| 117 | + ) | |
| 118 | + | |
| 119 | + return Fit2PLResult( | |
| 120 | + theta=theta, | |
| 121 | + theta_se=theta_se, | |
| 122 | + a=a, | |
| 123 | + b=b, | |
| 124 | + iterations=it, | |
| 125 | + converged=converged, | |
| 126 | + final_loglik=final_ll, | |
| 127 | + diagnostics={ | |
| 128 | + "observed_cells": int(mask.sum()), | |
| 129 | + "models": int(M), | |
| 130 | + "items": int(I), | |
| 131 | + }, | |
| 132 | + ) | |
added
apps/psychometrics/requirements.txt
+2 −0
@@ -0,0 +1,2 @@ | ||
| 1 | +numpy>=1.26 | |
| 2 | +pytest>=8.0 | |
added
apps/psychometrics/tests/test_psycho.py
+78 −0
@@ -0,0 +1,78 @@ | ||
| 1 | +# llmindex.io — psychometrics tests: parameter recovery on synthetic data | |
| 2 | +# Author: Simon-Pierre Boucher | |
| 3 | +# Contact: contact@spboucher.ai | |
| 4 | +# License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 5 | + | |
| 6 | +import numpy as np | |
| 7 | +import pytest | |
| 8 | + | |
| 9 | +from llmindex_psycho.bt import fit_bradley_terry | |
| 10 | +from llmindex_psycho.calibration import brier_score, expected_calibration_error | |
| 11 | +from llmindex_psycho.irt import fit_2pl | |
| 12 | + | |
| 13 | + | |
| 14 | +def _synthetic_2pl(M=12, I=80, seed=7): | |
| 15 | + rng = np.random.default_rng(seed) | |
| 16 | + theta = rng.normal(0, 1, M) | |
| 17 | + a = np.exp(rng.normal(0, 0.4, I)) | |
| 18 | + b = rng.normal(0, 1.2, I) | |
| 19 | + P = 1 / (1 + np.exp(-a[None, :] * (theta[:, None] - b[None, :]))) | |
| 20 | + X = (rng.random((M, I)) < P).astype(float) | |
| 21 | + return theta, a, b, X | |
| 22 | + | |
| 23 | + | |
| 24 | +def test_2pl_recovers_ability_ordering(): | |
| 25 | + theta_true, _, _, X = _synthetic_2pl() | |
| 26 | + result = fit_2pl(X, max_iterations=800) | |
| 27 | + corr = np.corrcoef(theta_true, result.theta)[0, 1] | |
| 28 | + assert corr > 0.85, f"theta recovery correlation too low: {corr:.3f}" | |
| 29 | + assert np.all(result.theta_se > 0) | |
| 30 | + assert np.all(result.a > 0) | |
| 31 | + | |
| 32 | + | |
| 33 | +def test_2pl_handles_missing_cells(): | |
| 34 | + _, _, _, X = _synthetic_2pl() | |
| 35 | + Xm = X.copy() | |
| 36 | + rng = np.random.default_rng(1) | |
| 37 | + Xm[rng.random(X.shape) < 0.3] = np.nan | |
| 38 | + result = fit_2pl(Xm, max_iterations=600) | |
| 39 | + assert np.isfinite(result.theta).all() | |
| 40 | + assert np.isfinite(result.theta_se).all() | |
| 41 | + | |
| 42 | + | |
| 43 | +def test_2pl_se_shrinks_with_more_items(): | |
| 44 | + _, _, _, X = _synthetic_2pl(M=8, I=120, seed=3) | |
| 45 | + few = fit_2pl(X[:, :15], max_iterations=600) | |
| 46 | + many = fit_2pl(X, max_iterations=600) | |
| 47 | + assert many.theta_se.mean() < few.theta_se.mean() | |
| 48 | + | |
| 49 | + | |
| 50 | +def test_2pl_rejects_empty(): | |
| 51 | + with pytest.raises(ValueError): | |
| 52 | + fit_2pl(np.full((3, 3), np.nan)) | |
| 53 | + | |
| 54 | + | |
| 55 | +def test_bradley_terry_recovers_ordering(): | |
| 56 | + rng = np.random.default_rng(11) | |
| 57 | + strength = np.array([2.0, 1.0, 0.0, -1.0, -2.0]) | |
| 58 | + M = len(strength) | |
| 59 | + wins = np.zeros((M, M)) | |
| 60 | + for i in range(M): | |
| 61 | + for j in range(M): | |
| 62 | + if i == j: | |
| 63 | + continue | |
| 64 | + p = 1 / (1 + np.exp(-(strength[i] - strength[j]))) | |
| 65 | + wins[i, j] = rng.binomial(40, p) | |
| 66 | + result = fit_bradley_terry(wins) | |
| 67 | + assert result.converged | |
| 68 | + assert list(np.argsort(-result.log_strength)) == [0, 1, 2, 3, 4] | |
| 69 | + | |
| 70 | + | |
| 71 | +def test_calibration_metrics(): | |
| 72 | + conf = np.array([0.9, 0.9, 0.1, 0.1]) | |
| 73 | + correct = np.array([1.0, 1.0, 0.0, 0.0]) | |
| 74 | + assert brier_score(conf, correct) == pytest.approx(0.01) | |
| 75 | + assert expected_calibration_error(conf, correct) == pytest.approx(0.1) | |
| 76 | + overconfident = np.full(100, 0.99) | |
| 77 | + outcomes = np.zeros(100) | |
| 78 | + assert expected_calibration_error(overconfident, outcomes) > 0.9 | |
added
apps/web/.eslintrc.json
+4 −0
@@ -0,0 +1,4 @@ | ||
| 1 | +{ | |
| 2 | + "root": true, | |
| 3 | + "extends": ["next/core-web-vitals"] | |
| 4 | +} | |
added
apps/web/app/api/v1/benchmark/progress/route.ts
+19 −0
@@ -0,0 +1,19 @@ | ||
| 1 | +/** | |
| 2 | + * llmindex.io — GET /api/v1/benchmark/progress (live run status, uncached) | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 6 | + */ | |
| 7 | +import { type NextRequest } from 'next/server'; | |
| 8 | +import { readBenchmarkProgress } from '@/lib/progress'; | |
| 9 | +import { apiJson } from '@/lib/api'; | |
| 10 | +import { rateLimit } from '@/lib/rate-limit'; | |
| 11 | + | |
| 12 | +export const dynamic = 'force-dynamic'; | |
| 13 | + | |
| 14 | +export async function GET(req: NextRequest) { | |
| 15 | + const limited = await rateLimit(req); | |
| 16 | + if (limited) return limited; | |
| 17 | + const progress = await readBenchmarkProgress(); | |
| 18 | + return apiJson({ progress }); | |
| 19 | +} | |
added
apps/web/app/api/v1/health/route.ts
+25 −0
@@ -0,0 +1,25 @@ | ||
| 1 | +/** | |
| 2 | + * llmindex.io — GET /api/v1/health | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 6 | + */ | |
| 7 | +import { prisma } from '@llmindex/db'; | |
| 8 | +import { INDEX_VERSION } from '@llmindex/scoring'; | |
| 9 | +import { apiJson } from '@/lib/api'; | |
| 10 | + | |
| 11 | +export const dynamic = 'force-dynamic'; | |
| 12 | + | |
| 13 | +export async function GET() { | |
| 14 | + let db = false; | |
| 15 | + try { | |
| 16 | + await prisma.$queryRaw`SELECT 1`; | |
| 17 | + db = true; | |
| 18 | + } catch { | |
| 19 | + db = false; | |
| 20 | + } | |
| 21 | + return apiJson( | |
| 22 | + { status: db ? 'ok' : 'degraded', db, index_version: INDEX_VERSION }, | |
| 23 | + { status: db ? 200 : 503 }, | |
| 24 | + ); | |
| 25 | +} | |
added
apps/web/app/api/v1/leaderboard/[domain]/route.ts
+49 −0
@@ -0,0 +1,49 @@ | ||
| 1 | +/** | |
| 2 | + * llmindex.io — GET /api/v1/leaderboard/:domain (per-domain ranking + sub-metrics) | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 6 | + */ | |
| 7 | +import { type NextRequest } from 'next/server'; | |
| 8 | +import { isDomain } from '@llmindex/scoring'; | |
| 9 | +import { getLeaderboard } from '@/lib/data'; | |
| 10 | +import { apiError, apiJson } from '@/lib/api'; | |
| 11 | +import { rateLimit } from '@/lib/rate-limit'; | |
| 12 | + | |
| 13 | +export const dynamic = 'force-dynamic'; | |
| 14 | + | |
| 15 | +export async function GET(req: NextRequest, { params }: { params: { domain: string } }) { | |
| 16 | + const limited = await rateLimit(req); | |
| 17 | + if (limited) return limited; | |
| 18 | + if (!isDomain(params.domain)) return apiError(404, 'unknown_domain'); | |
| 19 | + const search = req.nextUrl.searchParams; | |
| 20 | + const limit = Math.min(200, Math.max(1, Number(search.get('limit') ?? 50))); | |
| 21 | + const offset = Math.max(0, Number(search.get('offset') ?? 0)); | |
| 22 | + const data = await getLeaderboard(params.domain, limit, offset); | |
| 23 | + if (!data) return apiError(503, 'no_score_run_available'); | |
| 24 | + return apiJson( | |
| 25 | + { | |
| 26 | + index_version: data.run.indexVersion, | |
| 27 | + domain: params.domain, | |
| 28 | + run: { | |
| 29 | + id: data.run.id, | |
| 30 | + kind: data.run.kind, | |
| 31 | + status: data.run.status, | |
| 32 | + created_at: data.run.createdAt, | |
| 33 | + notes: data.run.notes, | |
| 34 | + }, | |
| 35 | + pagination: { limit, offset, count: data.entries.length }, | |
| 36 | + entries: data.entries.map((e) => ({ | |
| 37 | + rank: e.rank, | |
| 38 | + model: e.slug, | |
| 39 | + name: e.name, | |
| 40 | + provider: e.provider, | |
| 41 | + score: e.score, | |
| 42 | + score_low: e.scoreLow, | |
| 43 | + score_high: e.scoreHigh, | |
| 44 | + sub_metrics: e.subMetrics, | |
| 45 | + })), | |
| 46 | + }, | |
| 47 | + { cacheSeconds: 3600 }, | |
| 48 | + ); | |
| 49 | +} | |
added
apps/web/app/api/v1/leaderboard/route.ts
+45 −0
@@ -0,0 +1,45 @@ | ||
| 1 | +/** | |
| 2 | + * llmindex.io — GET /api/v1/leaderboard (global ranking, paginated, cached 1h) | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 6 | + */ | |
| 7 | +import { type NextRequest } from 'next/server'; | |
| 8 | +import { getLeaderboard } from '@/lib/data'; | |
| 9 | +import { apiError, apiJson } from '@/lib/api'; | |
| 10 | +import { rateLimit } from '@/lib/rate-limit'; | |
| 11 | + | |
| 12 | +export const dynamic = 'force-dynamic'; | |
| 13 | + | |
| 14 | +export async function GET(req: NextRequest) { | |
| 15 | + const limited = await rateLimit(req); | |
| 16 | + if (limited) return limited; | |
| 17 | + const params = req.nextUrl.searchParams; | |
| 18 | + const limit = Math.min(200, Math.max(1, Number(params.get('limit') ?? 50))); | |
| 19 | + const offset = Math.max(0, Number(params.get('offset') ?? 0)); | |
| 20 | + const data = await getLeaderboard('global', limit, offset); | |
| 21 | + if (!data) return apiError(503, 'no_score_run_available'); | |
| 22 | + return apiJson( | |
| 23 | + { | |
| 24 | + index_version: data.run.indexVersion, | |
| 25 | + run: { | |
| 26 | + id: data.run.id, | |
| 27 | + kind: data.run.kind, | |
| 28 | + status: data.run.status, | |
| 29 | + created_at: data.run.createdAt, | |
| 30 | + notes: data.run.notes, | |
| 31 | + }, | |
| 32 | + pagination: { limit, offset, count: data.entries.length }, | |
| 33 | + entries: data.entries.map((e) => ({ | |
| 34 | + rank: e.rank, | |
| 35 | + model: e.slug, | |
| 36 | + name: e.name, | |
| 37 | + provider: e.provider, | |
| 38 | + score: e.score, | |
| 39 | + score_low: e.scoreLow, | |
| 40 | + score_high: e.scoreHigh, | |
| 41 | + })), | |
| 42 | + }, | |
| 43 | + { cacheSeconds: 3600 }, | |
| 44 | + ); | |
| 45 | +} | |
added
apps/web/app/api/v1/methodology/route.ts
+39 −0
@@ -0,0 +1,39 @@ | ||
| 1 | +/** | |
| 2 | + * llmindex.io — GET /api/v1/methodology (machine-readable weights + IRT hyperparams) | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 6 | + */ | |
| 7 | +import { type NextRequest } from 'next/server'; | |
| 8 | +import { | |
| 9 | + CONTAMINATION_DELTA_FLOOR, | |
| 10 | + DOMAINS, | |
| 11 | + DOMAIN_WEIGHTS, | |
| 12 | + INDEX_VERSION, | |
| 13 | + IRT_HYPERPARAMS, | |
| 14 | + SUBMETRIC_WEIGHTS, | |
| 15 | + THETA_SCALE, | |
| 16 | +} from '@llmindex/scoring'; | |
| 17 | +import { apiJson } from '@/lib/api'; | |
| 18 | +import { rateLimit } from '@/lib/rate-limit'; | |
| 19 | + | |
| 20 | +export const dynamic = 'force-dynamic'; | |
| 21 | + | |
| 22 | +export async function GET(req: NextRequest) { | |
| 23 | + const limited = await rateLimit(req); | |
| 24 | + if (limited) return limited; | |
| 25 | + return apiJson( | |
| 26 | + { | |
| 27 | + index_version: INDEX_VERSION, | |
| 28 | + domains: DOMAINS, | |
| 29 | + domain_weights: DOMAIN_WEIGHTS, | |
| 30 | + submetric_weights: SUBMETRIC_WEIGHTS, | |
| 31 | + irt_hyperparams: IRT_HYPERPARAMS, | |
| 32 | + theta_scale: THETA_SCALE, | |
| 33 | + contamination_delta_floor: CONTAMINATION_DELTA_FLOOR, | |
| 34 | + notes: | |
| 35 | + 'Latency and cost are never blended into quality scores; they are published as an efficiency (Pareto) frontier.', | |
| 36 | + }, | |
| 37 | + { cacheSeconds: 3600 }, | |
| 38 | + ); | |
| 39 | +} | |
added
apps/web/app/api/v1/models/[...slug]/route.ts
+62 −0
@@ -0,0 +1,62 @@ | ||
| 1 | +/** | |
| 2 | + * llmindex.io — GET /api/v1/models/:slug (full model profile; slug contains "/") | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 6 | + */ | |
| 7 | +import { type NextRequest } from 'next/server'; | |
| 8 | +import { getModelProfile } from '@/lib/data'; | |
| 9 | +import { apiError, apiJson } from '@/lib/api'; | |
| 10 | +import { rateLimit } from '@/lib/rate-limit'; | |
| 11 | + | |
| 12 | +export const dynamic = 'force-dynamic'; | |
| 13 | + | |
| 14 | +export async function GET(req: NextRequest, { params }: { params: { slug: string[] } }) { | |
| 15 | + const limited = await rateLimit(req); | |
| 16 | + if (limited) return limited; | |
| 17 | + const slug = params.slug.join('/'); | |
| 18 | + const profile = await getModelProfile(slug); | |
| 19 | + if (!profile) return apiError(404, 'model_not_found_or_not_scored'); | |
| 20 | + return apiJson( | |
| 21 | + { | |
| 22 | + index_version: profile.run.indexVersion, | |
| 23 | + model: { | |
| 24 | + slug: profile.slug, | |
| 25 | + name: profile.name, | |
| 26 | + provider: profile.provider, | |
| 27 | + context_length: profile.contextLength, | |
| 28 | + prompt_price_per_1m_usd: profile.promptPricePerM, | |
| 29 | + completion_price_per_1m_usd: profile.completionPricePerM, | |
| 30 | + }, | |
| 31 | + run: { | |
| 32 | + id: profile.run.id, | |
| 33 | + kind: profile.run.kind, | |
| 34 | + status: profile.run.status, | |
| 35 | + created_at: profile.run.createdAt, | |
| 36 | + notes: profile.run.notes, | |
| 37 | + }, | |
| 38 | + global: profile.global | |
| 39 | + ? { | |
| 40 | + score: profile.global.score, | |
| 41 | + score_low: profile.global.scoreLow, | |
| 42 | + score_high: profile.global.scoreHigh, | |
| 43 | + } | |
| 44 | + : null, | |
| 45 | + domains: profile.domains.map((d) => ({ | |
| 46 | + domain: d.domain, | |
| 47 | + score: d.score, | |
| 48 | + score_low: d.scoreLow, | |
| 49 | + score_high: d.scoreHigh, | |
| 50 | + sub_metrics: d.subMetrics, | |
| 51 | + })), | |
| 52 | + run_history: profile.runHistory.map((h) => ({ | |
| 53 | + run_id: h.runId, | |
| 54 | + index_version: h.indexVersion, | |
| 55 | + kind: h.kind, | |
| 56 | + created_at: h.createdAt, | |
| 57 | + score: h.score, | |
| 58 | + })), | |
| 59 | + }, | |
| 60 | + { cacheSeconds: 3600 }, | |
| 61 | + ); | |
| 62 | +} | |
added
apps/web/app/globals.css
+32 −0
@@ -0,0 +1,32 @@ | ||
| 1 | +/* | |
| 2 | + * llmindex.io — global styles (clean light theme) | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 6 | + */ | |
| 7 | +@tailwind base; | |
| 8 | +@tailwind components; | |
| 9 | +@tailwind utilities; | |
| 10 | + | |
| 11 | +:root { | |
| 12 | + color-scheme: light; | |
| 13 | +} | |
| 14 | + | |
| 15 | +body { | |
| 16 | + @apply bg-zinc-50 text-zinc-700 antialiased; | |
| 17 | +} | |
| 18 | + | |
| 19 | +@keyframes row-in { | |
| 20 | + from { | |
| 21 | + opacity: 0; | |
| 22 | + transform: translateY(6px); | |
| 23 | + } | |
| 24 | + to { | |
| 25 | + opacity: 1; | |
| 26 | + transform: translateY(0); | |
| 27 | + } | |
| 28 | +} | |
| 29 | + | |
| 30 | +.animate-row-in { | |
| 31 | + animation: row-in 0.6s ease-out; | |
| 32 | +} | |
added
apps/web/app/icon.svg
+14 −0
@@ -0,0 +1,14 @@ | ||
| 1 | +<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64"> | |
| 2 | + <defs> | |
| 3 | + <linearGradient id="g" x1="0" y1="1" x2="1" y2="0"> | |
| 4 | + <stop offset="0" stop-color="#10b981"/> | |
| 5 | + <stop offset="1" stop-color="#22d3ee"/> | |
| 6 | + </linearGradient> | |
| 7 | + </defs> | |
| 8 | + <rect width="64" height="64" rx="14" fill="#09090b"/> | |
| 9 | + <rect x="11" y="37" width="9" height="16" rx="2.5" fill="#3f3f46"/> | |
| 10 | + <rect x="24" y="29" width="9" height="24" rx="2.5" fill="#71717a"/> | |
| 11 | + <rect x="37" y="19" width="9" height="34" rx="2.5" fill="url(#g)"/> | |
| 12 | + <path d="M13 29 L47 12" stroke="url(#g)" stroke-width="3.5" stroke-linecap="round"/> | |
| 13 | + <circle cx="49" cy="11" r="4.5" fill="#22d3ee"/> | |
| 14 | +</svg> | |
added
apps/web/app/layout.tsx
+66 −0
@@ -0,0 +1,66 @@ | ||
| 1 | +/** | |
| 2 | + * llmindex.io — root layout (responsive header + maintainer-credited footer) | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 6 | + */ | |
| 7 | +import type { Metadata, Viewport } from 'next'; | |
| 8 | +import Link from 'next/link'; | |
| 9 | +import { Logo } from '@/components/Logo'; | |
| 10 | +import './globals.css'; | |
| 11 | + | |
| 12 | +export const metadata: Metadata = { | |
| 13 | + metadataBase: new URL(process.env.PUBLIC_BASE_URL ?? 'https://www.llmindex.io'), | |
| 14 | + title: { | |
| 15 | + default: 'LLM Index — discriminative, contamination-resistant LLM rankings', | |
| 16 | + template: '%s · LLM Index', | |
| 17 | + }, | |
| 18 | + description: | |
| 19 | + 'The most discriminative, contamination-resistant, fully transparent LLM ranking. IRT-based scoring, dynamic item generation, pairwise duels, calibration and an efficiency frontier — updated live.', | |
| 20 | +}; | |
| 21 | + | |
| 22 | +export const viewport: Viewport = { | |
| 23 | + themeColor: '#fafafa', | |
| 24 | + width: 'device-width', | |
| 25 | + initialScale: 1, | |
| 26 | +}; | |
| 27 | + | |
| 28 | +export default function RootLayout({ children }: { children: React.ReactNode }) { | |
| 29 | + return ( | |
| 30 | + <html lang="en"> | |
| 31 | + <body className="min-h-screen"> | |
| 32 | + <header className="sticky top-0 z-40 border-b border-zinc-200 bg-white/85 shadow-sm shadow-zinc-200/50 backdrop-blur"> | |
| 33 | + <nav className="mx-auto flex max-w-6xl items-center justify-between gap-3 px-4 py-3"> | |
| 34 | + <Link href="/" aria-label="LLM Index home"> | |
| 35 | + <Logo /> | |
| 36 | + </Link> | |
| 37 | + <div className="flex items-center gap-4 text-sm sm:gap-6"> | |
| 38 | + <Link href="/" className="text-zinc-600 transition-colors hover:text-zinc-900"> | |
| 39 | + Leaderboard | |
| 40 | + </Link> | |
| 41 | + <Link href="/methodology" className="text-zinc-600 transition-colors hover:text-zinc-900"> | |
| 42 | + Methodology | |
| 43 | + </Link> | |
| 44 | + <a | |
| 45 | + href="/api/v1/leaderboard" | |
| 46 | + className="hidden rounded-full border border-zinc-300 px-3 py-1 text-zinc-700 transition-colors hover:border-emerald-400 hover:text-emerald-600 sm:inline-block" | |
| 47 | + > | |
| 48 | + API | |
| 49 | + </a> | |
| 50 | + </div> | |
| 51 | + </nav> | |
| 52 | + </header> | |
| 53 | + <main className="mx-auto max-w-6xl px-4 py-6 sm:py-10">{children}</main> | |
| 54 | + <footer className="border-t border-zinc-200 px-4 py-6 text-center text-xs leading-relaxed text-zinc-500"> | |
| 55 | + © {new Date().getFullYear()} LLM Index — maintained by Simon-Pierre Boucher ·{' '} | |
| 56 | + <a className="underline hover:text-zinc-700" href="mailto:contact@spboucher.ai"> | |
| 57 | + contact@spboucher.ai | |
| 58 | + </a> | |
| 59 | + <br className="sm:hidden" /> | |
| 60 | + <span className="hidden sm:inline"> · </span> | |
| 61 | + All scores carry 95% confidence intervals · Methodology is fully public | |
| 62 | + </footer> | |
| 63 | + </body> | |
| 64 | + </html> | |
| 65 | + ); | |
| 66 | +} | |
added
apps/web/app/leaderboard/[domain]/page.tsx
+60 −0
@@ -0,0 +1,60 @@ | ||
| 1 | +/** | |
| 2 | + * llmindex.io — per-domain leaderboard with sub-metrics (live-updating) | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 6 | + */ | |
| 7 | +import Link from 'next/link'; | |
| 8 | +import { notFound } from 'next/navigation'; | |
| 9 | +import { DOMAINS, INDEX_VERSION, isDomain } from '@llmindex/scoring'; | |
| 10 | +import { LiveLeaderboard, type ApiEntry, type ApiRun } from '@/components/LiveLeaderboard'; | |
| 11 | +import { getLeaderboard } from '@/lib/data'; | |
| 12 | + | |
| 13 | +export const revalidate = 300; | |
| 14 | + | |
| 15 | +export function generateStaticParams() { | |
| 16 | + return DOMAINS.map((domain) => ({ domain })); | |
| 17 | +} | |
| 18 | + | |
| 19 | +export default async function DomainLeaderboardPage({ params }: { params: { domain: string } }) { | |
| 20 | + if (!isDomain(params.domain)) notFound(); | |
| 21 | + const data = await getLeaderboard(params.domain); | |
| 22 | + const title = params.domain.replaceAll('_', ' '); | |
| 23 | + | |
| 24 | + const initialRun: ApiRun | null = data | |
| 25 | + ? { | |
| 26 | + id: data.run.id, | |
| 27 | + kind: data.run.kind, | |
| 28 | + status: data.run.status, | |
| 29 | + created_at: new Date(data.run.createdAt).toISOString(), | |
| 30 | + notes: data.run.notes, | |
| 31 | + } | |
| 32 | + : null; | |
| 33 | + const initialEntries: ApiEntry[] = (data?.entries ?? []).map((e) => ({ | |
| 34 | + rank: e.rank, | |
| 35 | + model: e.slug, | |
| 36 | + name: e.name, | |
| 37 | + provider: e.provider, | |
| 38 | + score: e.score, | |
| 39 | + score_low: e.scoreLow, | |
| 40 | + score_high: e.scoreHigh, | |
| 41 | + sub_metrics: e.subMetrics, | |
| 42 | + })); | |
| 43 | + | |
| 44 | + return ( | |
| 45 | + <div className="space-y-6"> | |
| 46 | + <div className="flex flex-wrap items-center gap-3"> | |
| 47 | + <Link href="/" className="text-sm text-zinc-500 hover:text-zinc-900"> | |
| 48 | + ← Global | |
| 49 | + </Link> | |
| 50 | + <h1 className="text-xl font-bold capitalize text-zinc-900 sm:text-2xl">{title}</h1> | |
| 51 | + </div> | |
| 52 | + <LiveLeaderboard | |
| 53 | + domain={params.domain} | |
| 54 | + initialRun={initialRun} | |
| 55 | + initialEntries={initialEntries} | |
| 56 | + indexVersion={data?.run.indexVersion ?? INDEX_VERSION} | |
| 57 | + /> | |
| 58 | + </div> | |
| 59 | + ); | |
| 60 | +} | |
added
apps/web/app/methodology/page.tsx
+280 −0
@@ -0,0 +1,280 @@ | ||
| 1 | +/** | |
| 2 | + * llmindex.io — methodology page: full pipeline detail, live sample items, references | |
| 3 | + * Author: Simon-Pierre Boucher | |
| 4 | + * Contact: contact@spboucher.ai | |
| 5 | + * License: Proprietary — © Simon-Pierre Boucher, all rights reserved | |
| 6 | + * | |
| 7 | + * Sample items are rendered from documentation-only seeds ("docs:<template>") | |
| 8 | + * that are never used in scored runs — templates are public by design, scored | |
| 9 | + * instantiations are not. | |
| 10 | + */ | |
| 11 | +import { | |
| 12 | + CONTAMINATION_DELTA_FLOOR, | |
| 13 | + DOMAINS, | |
| 14 | + DOMAIN_WEIGHTS, | |
| 15 | + DUEL_DOMAINS, | |
| 16 | + INDEX_VERSION, | |
| 17 | + IRT_HYPERPARAMS, | |
| 18 | + SUBMETRIC_WEIGHTS, | |
| 19 | + THETA_SCALE, | |
| 20 | +} from '@llmindex/scoring'; | |
| 21 | +import { TEMPLATES, createRng } from '@llmindex/items'; | |
| 22 | + | |
| 23 | +export const revalidate = 3600; | |
| 24 | + | |
| 25 | +const PILLARS: Array<{ title: string; body: string }> = [ | |
| 26 | + { | |
| 27 | + title: '1 · IRT-based scoring (2PL)', | |
| 28 | + body: 'Every item carries a fitted difficulty (b) and discrimination (a): P(correct) = σ(a·(θ−b)). Ability θ is a MAP estimate with priors θ~N(0,1), b~N(0,1.5), log a~N(0,0.5), fitted by a missing-aware numpy optimizer; θ standard errors come from the Fisher information. Items solved by everyone (or no one) carry zero ranking information — items with a<0.3 or |b|>3 are auto-flagged for retirement after every refit. Raw accuracy is never the score.', | |
| 29 | + }, | |
| 30 | + { | |
| 31 | + title: '2 · Dynamic item generation', | |
| 32 | + body: 'Items are instantiated from versioned template generators with seeded value substitution, paraphrase rotation and structural perturbation — fresh for every scored batch, so no fixed test set exists to memorize. A frozen anchor stream (≤20% of any run) is kept identical across runs; the accuracy gap anchor−fresh is published per model as contamination_delta and maps to a resistance sub-metric via clamp01(1 − Δ/0.2).', | |
| 33 | + }, | |
| 34 | + { | |
| 35 | + title: '3 · Difficulty engineered for discrimination', | |
| 36 | + body: 'Item information peaks where a model has ~50% success probability. Generators therefore stack proven difficulty knobs: 6-8-step dependent arithmetic chains (errors compound multiplicatively), chained sub-problems whose answers feed forward, provably-inert distractor clauses, counterfactual rules (base-7/8/9/11/13 arithmetic), interior-rank deduction with decoy entities, nested control-flow traces, and constraint stacking — each knob re-randomized per run.', | |
| 37 | + }, | |
| 38 | + { | |
| 39 | + title: '4 · Agentic: simulated tool-calling', | |
| 40 | + body: 'Home-made mock environments (support-desk triage under policy, treasury ledger with overdraft pre-funding, dependency-ordered deployments) present a tool catalog salted with distractor tools. A deterministic simulator computes the unique correct call sequence; grading is canonical-JSON equality of the emitted sequence — binary, no judges, no partial credit. A dedicated context-load family buries the relevant records among hundreds of near-miss decoys (same customer/wrong region, right region/wrong status), making prompt length itself the difficulty knob.', | |
| 41 | + }, | |
| 42 | + { | |
| 43 | + title: '5 · Terminal: simulated shell, exact prediction', | |
| 44 | + body: 'No shell ever executes. A closed, unambiguous POSIX subset (fixed-string grep, cut, byte-order C-locale sort, integer-only awk aggregation, head/tail) is simulated in TypeScript over generated CSV data; models predict exact stdout, final file trees after mv/cp/rm/cd sequences with relative paths, and && / || short-circuit execution traces with exit codes. Locale-dependent, GNU/BSD-divergent and float-formatting constructs are excluded by design, so every answer is unambiguous.', | |
| 45 | + }, | |
| 46 | + { | |
| 47 | + title: '6 · Vision OCR under clutter', | |
| 48 | + body: 'Generated SVG scenes — rasterized server-side — embed target codes among rotated decoy codes, noise strokes and low-contrast patches, plus rendered mini-tables requiring grounded arithmetic. The glyph set excludes visually ambiguous characters (0/O, 1/l/I, 5/S, 8/B), so difficulty comes from clutter and selection, never unfair ambiguity. Text-only models skip the domain; their Global Index renormalizes.', | |
| 49 | + }, | |
| 50 | + { | |
| 51 | + title: '7 · Judged duels + Bradley-Terry (writing, safety, SVG design)', | |
| 52 | + body: 'Open-ended domains use pairwise duels: both models answer the same generated task (constrained creative writing; delicate gray-zone situations; reproducing a real-world logo in raw SVG from memory). A 3-judge cross-provider panel rates each duel with recorded position swaps; a model never judges its own duel; empty responses never count as wins. Verdicts feed a Bradley-Terry fit (MM algorithm, ties as half-wins, Fisher-information SEs), standardized to the same θ scale as IRT domains. Judge panel agreement is published with every run.', | |
| 53 | + }, | |
| 54 | + { | |
| 55 | + title: '8 · Robust answer extraction (measured, not guessed)', | |
| 56 | + body: 'Extraction never confounds formatting with ability: an ordered cascade accepts "ANSWER: x" in any markdown wrapping, FINAL ANSWER variants, LaTeX \\boxed{}, and fenced code blocks for JSON/multi-line answers; numbers are normalized across thousands separators, currency signs and units; hyphen/space orthography variants are unified. Truncated completions (finish_reason=length) are unscored — never counted as wrong — and every model gets a 16k-token completion budget so reasoning models can finish thinking.', | |
| 57 | + }, | |
| 58 | + { | |
| 59 | + title: '9 · Consistency, calibration, cost', | |
| 60 | + body: 'Each scored item is also sampled k times at the model\'s default temperature; the share of samples agreeing with the modal answer is the consistency sub-metric. Every item demands a 0-100 confidence line; calibration = 1 − ECE over 10 bins. Latency p50 and measured cost per 1k items are published but NEVER blended into quality — they live on a separate Pareto frontier.', | |
| 61 | + }, | |
| 62 | +]; | |
| 63 | + | |
| 64 | +const REFERENCES: Array<{ label: string; url: string }> = [ | |
| 65 | + { label: 'metabench — sparse IRT benchmark distillation (ICLR 2025)', url: 'https://openreview.net/forum?id=4T33izzFpK' }, | |
| 66 | + { label: 'tinyBenchmarks — 100-item IRT evaluation (ICML 2024)', url: 'https://arxiv.org/abs/2402.14992' }, | |
| 67 | + { label: 'ATLAS — Fisher-information adaptive testing for LLMs', url: 'https://arxiv.org/abs/2511.04689' }, | |
| 68 | + { label: 'GSM-Symbolic — template perturbation & memorization gaps (Apple)', url: 'https://arxiv.org/abs/2410.05229' }, | |
| 69 | + { label: 'GSM-IC — irrelevant-context distractors (ICML 2023)', url: 'https://arxiv.org/abs/2302.00093' }, | |
| 70 | + { label: 'Reasoning or Reciting? — counterfactual rule perturbation (NAACL 2024)', url: 'https://arxiv.org/abs/2307.02477' }, | |
| 71 | + { label: 'Faith and Fate — compositional depth limits (NeurIPS 2023)', url: 'https://arxiv.org/abs/2305.18654' }, | |
| 72 | + { label: 'ZebraLogic — constraint-satisfaction scaling curse (ICML 2025)', url: 'https://arxiv.org/abs/2502.01100' }, | |
| 73 | + { label: 'GSM-Infinite — computational-graph difficulty scaling (ICML 2025)', url: 'https://arxiv.org/abs/2502.05252' }, | |
| 74 | + { label: 'R-Horizon — chained-problem discrimination amplification', url: 'https://arxiv.org/abs/2510.08189' }, | |
| 75 | + { label: 'IFEval — verifiable instruction constraints', url: 'https://arxiv.org/abs/2311.07911' }, | |
| 76 | + { label: 'MMLU-Pro — distractor design & discrimination (NeurIPS 2024)', url: 'https://arxiv.org/abs/2406.01574' }, | |
| 77 | + { label: 'Humanity\'s Last Exam — frontier-failure item filtering (Nature 2025)', url: 'https://www.nature.com/articles/s41586-025-09962-4' }, | |
| 78 | + { label: 'NPHardEval — monthly regenerated complexity-class items', url: 'https://arxiv.org/abs/2312.14890' }, | |
| 79 | + { label: 'Can We Trust IRT for AI Evaluation? — small-N caveats', url: 'https://arxiv.org/html/2607.15190v1' }, | |
| 80 | + { label: 'τ-bench / τ²-bench — policy-constrained agents, pass^k (Sierra)', url: 'https://arxiv.org/pdf/2406.12045' }, | |
| 81 | + { label: 'BFCL v3 — AST/state-based tool-call grading (Berkeley)', url: 'https://gorilla.cs.berkeley.edu/blogs/13_bfcl_v3_multi_turn.html' }, | |
| 82 | + { label: 'GAIA — normalized exact-match agent grading', url: 'https://huggingface.co/papers/2311.12983' }, | |
| 83 | + { label: 'Establishing Best Practices for Rigorous Agentic Benchmarks (ABC)', url: 'https://arxiv.org/abs/2507.02825' }, | |
| 84 | + { label: 'SWE-Bench+ — leakage & weak-test inflation audit', url: 'https://arxiv.org/pdf/2410.06992' }, | |
| 85 | + { label: 'Terminal-Bench 2.0 — containerized terminal tasks', url: 'https://www.tbench.ai/' }, | |
| 86 | + { label: 'The Pilot/ForgeCode cheating audit (UPenn) — why we grade by simulation', url: 'https://debugml.github.io/cheating-agents/' }, | |
| 87 | + { label: 'CRUXEval — output-prediction evaluation paradigm', url: 'https://arxiv.org/abs/2401.03065' }, | |
| 88 | + { label: 'InterCode — execution-graded interactive shell tasks', url: 'https://arxiv.org/abs/2306.14898' }, | |
| 89 | + { label: 'Smoosh — mechanized POSIX shell semantics (why we whitelist)', url: 'https://arxiv.org/abs/1907.05308' }, | |
| 90 | + { label: 'LiveBench — procedural regeneration against contamination', url: 'https://arxiv.org/abs/2406.19314' }, | |
| 91 | + { label: 'Math-Verify — extraction leniency reshuffles leaderboards (HF)', url: 'https://huggingface.co/blog/math_verify_leaderboard' }, | |
| 92 | + { label: 'xFinder — regex extraction is only ~74% accurate', url: 'https://arxiv.org/abs/2405.11874' }, | |
| 93 | + { label: 'ReasonIF — reasoning models ignore format instructions', url: 'https://arxiv.org/abs/2510.15211' }, | |
| 94 | + { label: 'FormatSpread — 76-point spreads from formatting alone', url: 'https://arxiv.org/abs/2310.11324' }, | |
| 95 | + { label: 'Just Ask for Calibration — verbalized confidence (EMNLP 2023)', url: 'https://aclanthology.org/2023.emnlp-main.330/' }, | |
| 96 | + { label: 'WebArena-Verified — checker misalignment fixes', url: 'https://openreview.net/forum?id=94tlGxmqkN' }, | |
| 97 | +]; | |
| 98 | + | |
| 99 | +export default function MethodologyPage() { | |
| 100 | + const samples = TEMPLATES.map((t) => { | |
| 101 | + const seed = `docs:${t.id}`; | |
| 102 | + const item = t.render(createRng(seed), seed); | |
| 103 | + return { template: t, item }; | |
| 104 | + }); | |
| 105 | + const byDomain = new Map<string, typeof samples>(); | |
| 106 | + for (const s of samples) { | |
| 107 | + byDomain.set(s.item.domain, [...(byDomain.get(s.item.domain) ?? []), s]); | |
| 108 | + } | |
| 109 | + | |
| 110 | + return ( | |
| 111 | + <div className="space-y-10"> | |
| 112 | + <div className="space-y-2"> | |
| 113 | + <h1 className="text-2xl font-bold text-zinc-900 sm:text-3xl">Methodology</h1> | |
| 114 | + <p className="max-w-3xl text-sm leading-relaxed text-zinc-600"> | |
| 115 | + Index version {INDEX_VERSION}. Everything below is reproducible: template generators are | |
| 116 | + public, every displayed number traces to an immutable score run with stored raw | |
| 117 | + responses, and the machine-readable configuration is served at{' '} | |
| 118 | + <a href="/api/v1/methodology" className="underline hover:text-zinc-900"> | |
| 119 | + /api/v1/methodology | |
| 120 | + </a> | |
| 121 | + . Scored item instantiations and answer keys never leave the server. | |
| 122 | + </p> | |
| 123 | + </div> | |
| 124 | + | |
| 125 | + <section className="grid gap-4 md:grid-cols-2"> | |
| 126 | + {PILLARS.map((p) => ( | |
| 127 | + <div key={p.title} className="rounded-xl border border-zinc-200 p-5"> | |
| 128 | + <h2 className="mb-2 font-semibold text-zinc-900">{p.title}</h2> | |
| 129 | + <p className="text-sm leading-relaxed text-zinc-600">{p.body}</p> | |
| 130 | + </div> | |
| 131 | + ))} | |
| 132 | + </section> | |
| 133 | + | |
| 134 | + <section className="space-y-3"> | |
| 135 | + <h2 className="text-xl font-semibold text-zinc-900">The pipeline, end to end</h2> | |
| 136 | + <ol className="list-decimal space-y-2 pl-5 text-sm leading-relaxed text-zinc-600"> | |
| 137 | + <li>Model catalog and per-token pricing sync daily from the OpenRouter /models endpoint; the ranked roster is explicit configuration.</li> | |
| 138 | + <li>Per model, per graded domain: a fresh seeded batch (30 items, ≤20% anchors) is generated; every item is asked at temperature 0 (scored) plus k consistency samples at default temperature; every call stores exact request params, raw response, tokens, measured latency and cost.</li> | |
| 139 | + <li>Answers are extracted by the lenient cascade; truncated completions are unscored; batches with >2% failed calls are flagged degraded and excluded from fits until re-run.</li> | |
| 140 | + <li>After every model completes, the 2PL fit re-runs over all complete same-version batches; Bradley-Terry re-fits the duel domains; scores publish with 95% CIs and the leaderboard updates live.</li> | |
| 141 | + <li>Items with a<{IRT_HYPERPARAMS.minDiscrimination} or |b|>{IRT_HYPERPARAMS.maxAbsDifficultyLogits} logits are auto-flagged for retirement; every methodology change bumps the semver index version with a public changelog.</li> | |
| 142 | + </ol> | |
| 143 | + </section> | |
| 144 | + | |
| 145 | + <section className="space-y-4"> | |
| 146 | + <h2 className="text-xl font-semibold text-zinc-900">Real sample questions, per domain</h2> | |
| 147 | + <p className="text-sm text-zinc-500"> | |
| 148 | + Rendered live from the public template generators with documentation-only seeds (never | |
| 149 | + used in scoring — scored runs draw fresh seeds every time). Reference answers shown where | |
| 150 | + the item is mechanically graded. | |
| 151 | + </p> | |
| 152 | + {[...byDomain.entries()].map(([domain, list]) => ( | |
| 153 | + <div key={domain} className="space-y-2"> | |
| 154 | + <h3 className="text-sm font-semibold uppercase tracking-wide text-emerald-600"> | |
| 155 | + {domain.replaceAll('_', ' ')} | |
| 156 | + </h3> | |
| 157 | + {list.map(({ template, item }) => ( | |
| 158 | + <details key={template.id} className="rounded-xl border border-zinc-200"> | |
| 159 | + <summary className="cursor-pointer select-none px-4 py-3 text-sm text-zinc-800 hover:bg-zinc-50"> | |
| 160 | + <code className="text-emerald-700">{template.id}</code> | |
| 161 | + <span className="ml-2 text-xs text-zinc-500">{template.description}</span> | |
| 162 | + </summary> | |
| 163 | + <div className="space-y-3 border-t border-zinc-200 p-4"> | |
| 164 | + {item.svg && ( | |
| 165 | + <div | |
| 166 | + className="max-w-md overflow-hidden rounded-lg border border-zinc-300" | |
| 167 | + dangerouslySetInnerHTML={{ __html: item.svg }} | |
| 168 | + /> | |
| 169 | + )} | |
| 170 | + <pre className="max-h-96 overflow-auto whitespace-pre-wrap rounded-lg bg-zinc-100 p-3 text-xs leading-relaxed text-zinc-700"> | |
| 171 | + {item.prompt} | |
| 172 | + </pre> | |
| 173 | + <p className="text-xs text-zinc-500"> | |
| 174 | + grading: <code>{item.grading}</code> | |
| 175 | + {item.answerKey && item.grading !== 'constraints' && ( | |
| 176 | + <> | |
| 177 | + {' · '}reference answer (docs seed only):{' '} | |
| 178 | + <code className="break-all text-zinc-700"> | |
| 179 | + {item.answerKey.slice(0, 300)} | |
| 180 | + {item.answerKey.length > 300 ? '…' : ''} | |
| 181 | + </code> | |
| 182 | + </> | |
| 183 | + )} | |
| 184 | + {item.grading === 'constraints' && ( | |
| 185 | + <> | |
| 186 | + {' · '}graded by mechanical constraint checkers:{' '} | |
| 187 | + <code className="break-all text-zinc-700">{item.answerKey}</code> | |
| 188 | + </> | |
| 189 | + )} | |
| 190 | + </p> | |
| 191 | + </div> | |
| 192 | + </details> | |
| 193 | + ))} | |
| 194 | + </div> | |
| 195 | + ))} | |
| 196 | + <div className="rounded-xl border border-zinc-200 p-4 text-sm text-zinc-600"> | |
| 197 | + <span className="font-semibold text-zinc-800">Duel domains</span> —{' '} | |
| 198 | + {DUEL_DOMAINS.map((d) => d.replaceAll('_', ' ')).join(', ')}: open-ended generated tasks | |
| 199 | + (constrained writing briefs; gray-zone assistance scenarios; “reproduce the | |
| 200 | + <brand> logo in raw SVG from memory”) judged pairwise by a 3-judge | |
| 201 | + cross-provider panel and aggregated with Bradley-Terry. Judge prompts are public in the | |
| 202 | + repository; they contain no secret rubrics. | |
| 203 | + </div> | |
| 204 | + </section> | |
| 205 | + | |
| 206 | + <section className="space-y-3"> | |
| 207 | + <h2 className="text-xl font-semibold text-zinc-900">Weights (v{INDEX_VERSION}) — and why</h2> | |
| 208 | + <p className="max-w-3xl text-sm leading-relaxed text-zinc-600"> | |
| 209 | + Domain weights are <span className="text-zinc-800">equal by design</span>: absent a | |
| 210 | + task-utility function, any unequal weighting is an editorial value judgment; the | |
| 211 | + maximum-entropy prior is the only non-arbitrary default, and per-domain scores are always | |
| 212 | + published so you can re-weight for your own use case. Sub-metric weights favor the latent | |
| 213 | + ability estimate (accuracy_irt {SUBMETRIC_WEIGHTS.accuracy_irt}) with robustness | |
| 214 | + corrections for answer stability ({SUBMETRIC_WEIGHTS.consistency}) and template | |
| 215 | + memorization ({SUBMETRIC_WEIGHTS.contamination_resistance}), and a smaller calibration | |
| 216 | + term ({SUBMETRIC_WEIGHTS.calibration}) reflecting its higher measurement noise at current | |
| 217 | + sample sizes. Latency and cost are excluded from quality entirely — Pareto frontier only. | |
| 218 | + </p> | |
| 219 | + <div className="grid gap-6 md:grid-cols-2"> | |
| 220 | + <div> | |
| 221 | + <h3 className="mb-2 text-sm uppercase tracking-wide text-zinc-500">Domain weights (Global Index)</h3> | |
| 222 | + <ul className="space-y-1 text-sm text-zinc-700"> | |
| 223 | + {DOMAINS.map((d) => ( | |
| 224 | + <li key={d} className="flex justify-between border-b border-zinc-100 py-1"> | |
| 225 | + <span className="capitalize">{d.replaceAll('_', ' ')}</span> | |
| 226 | + <span className="tabular-nums">{DOMAIN_WEIGHTS[d].toFixed(4)}</span> | |
| 227 | + </li> | |
| 228 | + ))} | |
| 229 | + </ul> | |
| 230 | + </div> | |
| 231 | + <div> | |
| 232 | + <h3 className="mb-2 text-sm uppercase tracking-wide text-zinc-500">Sub-metric weights (domain composite)</h3> | |
| 233 | + <ul className="space-y-1 text-sm text-zinc-700"> | |
| 234 | + {Object.entries(SUBMETRIC_WEIGHTS).map(([k, w]) => ( | |
| 235 | + <li key={k} className="flex justify-between border-b border-zinc-100 py-1"> | |
| 236 | + <span>{k}</span> | |
| 237 | + <span className="tabular-nums">{w.toFixed(2)}</span> | |
| 238 | + </li> | |
| 239 | + ))} | |
| 240 | + </ul> | |
| 241 | + <p className="mt-3 text-xs text-zinc-500"> | |
| 242 | + Missing sub-metrics renormalize (a domain measured on accuracy alone neither gains | |
| 243 | + nor loses). θ rescale: {THETA_SCALE.center} + {THETA_SCALE.slope}·θ; contamination | |
| 244 | + floor Δ={CONTAMINATION_DELTA_FLOOR}. | |
| 245 | + </p> | |
| 246 | + </div> | |
| 247 | + </div> | |
| 248 | + </section> | |
| 249 | + | |
| 250 | + <section className="space-y-2"> | |
| 251 | + <h2 className="text-xl font-semibold text-zinc-900">IRT hyperparameters</h2> | |
| 252 | + <pre className="overflow-x-auto rounded-xl border border-zinc-200 bg-zinc-50 p-4 text-xs text-zinc-700"> | |
| 253 | + {JSON.stringify(IRT_HYPERPARAMS, null, 2)} | |
| 254 | + </pre> | |
| 255 | + </section> | |
| 256 | + | |
| 257 | + <section className="space-y-3"> | |
| 258 | + <h2 className="text-xl font-semibold text-zinc-900">References & influences</h2> | |
| 259 | + <p className="max-w-3xl text-sm text-zinc-600"> | |
| 260 | + The design draws on the following research — as inspiration and evidence, not as source | |
| 261 | + material: every environment, template, item and grader here is original and home-made. | |
| 262 | + </p> | |
| 263 | + <ul className="grid gap-1.5 text-sm sm:grid-cols-2"> | |
| 264 | + {REFERENCES.map((r) => ( | |
| 265 | + <li key={r.url}> | |
| 266 | + <a | |
| 267 | + href={r.url} | |
| 268 | + target="_blank" | |
| 269 | + rel="noreferrer" | |
| 270 | + className="text-zinc-600 underline decoration-zinc-300 hover:text-emerald-600" | |
| 271 | + > | |
| 272 | + {r.label} | |
| 273 | + </a> | |
| 274 | + </li> | |
| 275 | + ))} | |
| 276 | + </ul> | |
| 277 | + </section> | |
| 278 | + </div> | |
| 279 | + ); | |
| 280 | +} | |
added
apps/web/app/models/[...slug]/page.tsx
+249 −0
Large diff (249 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/app/page.tsx
+144 −0
Large diff (144 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/components/LiveLeaderboard.tsx
+368 −0
Large diff (368 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/components/Logo.tsx
+37 −0
Large diff (37 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/components/ProviderLogo.tsx
+84 −0
Large diff (84 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/e2e/smoke.spec.ts
+32 −0
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/lib/api.ts
+29 −0
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/lib/data.ts
+308 −0
Large diff (308 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/lib/progress.ts
+65 −0
Large diff (65 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/lib/rate-limit.ts
+65 −0
Large diff (65 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/next-env.d.ts
+5 −0
Large diff (5 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/next.config.mjs
+19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/package.json
+39 −0
Large diff (39 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/playwright.config.ts
+23 −0
Large diff (23 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/postcss.config.mjs
+12 −0
Large diff (12 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/tailwind.config.ts
+21 −0
Large diff (21 lines) is not loaded by default to keep the page fast.Load diff
added
apps/web/tsconfig.json
+17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/.eslintrc.json
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/package.json
+38 −0
Large diff (38 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/src/benchmark.ts
+200 −0
Large diff (200 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/src/cli/benchmark-run.ts
+35 −0
Large diff (35 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/src/cli/duel-run.ts
+42 −0
Large diff (42 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/src/cli/eval-run.ts
+47 −0
Large diff (47 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/src/cli/refit.ts
+20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/src/duel-runner.ts
+169 −0
Large diff (169 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/src/env.ts
+30 −0
Large diff (30 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/src/eval-runner.ts
+229 −0
Large diff (229 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/src/index.ts
+72 −0
Large diff (72 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/src/judges/config.ts
+62 −0
Large diff (62 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/src/progress.ts
+61 −0
Large diff (61 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/src/refit.ts
+419 −0
Large diff (419 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/src/worker.test.ts
+52 −0
Large diff (52 lines) is not loaded by default to keep the page fast.Load diff
added
apps/worker/tsconfig.json
+4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added
data/item-bank/templates.v1.json
+19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added
data/runs/README.md
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
docs/assets/home-desktop.png
+0 −0
Binary file not shown.
added
docs/assets/home-mobile.png
+0 −0
Binary file not shown.
added
docs/assets/logo.svg
+14 −0
Large diff (14 lines) is not loaded by default to keep the page fast.Load diff
added
docs/assets/methodology.png
+0 −0
Binary file not shown.
added
docs/assets/model-page.png
+0 −0
Binary file not shown.
added
docs/methodology/CHANGELOG.md
+60 −0
Large diff (60 lines) is not loaded by default to keep the page fast.Load diff
added
docs/methodology/IRT_SPEC.md
+34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added
docs/methodology/JUDGE_PROTOCOL.md
+29 −0
Large diff (29 lines) is not loaded by default to keep the page fast.Load diff
added
docs/methodology/METHODOLOGY.md
+78 −0
Large diff (78 lines) is not loaded by default to keep the page fast.Load diff
added
infra/.env.example
+17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added
infra/check-headers.mjs
+47 −0
Large diff (47 lines) is not loaded by default to keep the page fast.Load diff
added
infra/deploy.sh
+46 −0
Large diff (46 lines) is not loaded by default to keep the page fast.Load diff
added
infra/docker-compose.dev.yml
+20 −0
Large diff (20 lines) is not loaded by default to keep the page fast.Load diff
added
infra/docker-compose.prod.yml
+25 −0
Large diff (25 lines) is not loaded by default to keep the page fast.Load diff
added
infra/ecosystem.config.cjs
+62 −0
Large diff (62 lines) is not loaded by default to keep the page fast.Load diff
added
infra/ngrok.yml
+8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added
infra/systemd/llmindex-web.service
+17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added
infra/systemd/llmindex-worker.service
+17 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff
added
package.json
+32 −0
Large diff (32 lines) is not loaded by default to keep the page fast.Load diff
added
packages/config/package.json
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
packages/config/tsconfig.base.json
+19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added
packages/db/.eslintrc.json
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
packages/db/package.json
+34 −0
Large diff (34 lines) is not loaded by default to keep the page fast.Load diff
added
packages/db/prisma/migrations/0001_init/migration.sql
+155 −0
Large diff (155 lines) is not loaded by default to keep the page fast.Load diff
added
packages/db/prisma/migrations/0002_model_vision/migration.sql
+2 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
packages/db/prisma/migrations/migration_lock.toml
+2 −0
Large diff (2 lines) is not loaded by default to keep the page fast.Load diff
added
packages/db/prisma/schema.prisma
+154 −0
Large diff (154 lines) is not loaded by default to keep the page fast.Load diff
added
packages/db/src/index.ts
+19 −0
Large diff (19 lines) is not loaded by default to keep the page fast.Load diff
added
packages/db/src/seed.ts
+189 −0
Large diff (189 lines) is not loaded by default to keep the page fast.Load diff
added
packages/db/tsconfig.json
+4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/.eslintrc.json
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/package.json
+27 −0
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/answer.ts
+248 −0
Large diff (248 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/index.ts
+111 −0
Large diff (111 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/items.test.ts
+151 −0
Large diff (151 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/rng.ts
+56 −0
Large diff (56 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/templates/agentic-load.ts
+144 −0
Large diff (144 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/templates/agentic.ts
+323 −0
Large diff (323 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/templates/code.ts
+131 −0
Large diff (131 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/templates/duels.ts
+98 −0
Large diff (98 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/templates/instruction.ts
+120 −0
Large diff (120 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/templates/knowledge.ts
+78 −0
Large diff (78 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/templates/math.ts
+187 −0
Large diff (187 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/templates/multilingual.ts
+132 −0
Large diff (132 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/templates/reasoning.ts
+90 −0
Large diff (90 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/templates/terminal.ts
+366 −0
Large diff (366 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/templates/vision.ts
+160 −0
Large diff (160 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/src/types.ts
+53 −0
Large diff (53 lines) is not loaded by default to keep the page fast.Load diff
added
packages/items/tsconfig.json
+4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added
packages/openrouter/.eslintrc.json
+10 −0
Large diff (10 lines) is not loaded by default to keep the page fast.Load diff
added
packages/openrouter/package.json
+24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added
packages/openrouter/src/client.test.ts
+97 −0
Large diff (97 lines) is not loaded by default to keep the page fast.Load diff
added
packages/openrouter/src/client.ts
+139 −0
Large diff (139 lines) is not loaded by default to keep the page fast.Load diff
added
packages/openrouter/src/index.ts
+8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added
packages/openrouter/src/types.ts
+67 −0
Large diff (67 lines) is not loaded by default to keep the page fast.Load diff
added
packages/openrouter/tsconfig.json
+4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added
packages/scoring/.eslintrc.json
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
packages/scoring/package.json
+24 −0
Large diff (24 lines) is not loaded by default to keep the page fast.Load diff
added
packages/scoring/src/domains.ts
+48 −0
Large diff (48 lines) is not loaded by default to keep the page fast.Load diff
added
packages/scoring/src/index.ts
+162 −0
Large diff (162 lines) is not loaded by default to keep the page fast.Load diff
added
packages/scoring/src/scoring.test.ts
+98 −0
Large diff (98 lines) is not loaded by default to keep the page fast.Load diff
added
packages/scoring/src/weights.ts
+71 −0
Large diff (71 lines) is not loaded by default to keep the page fast.Load diff
added
packages/scoring/tsconfig.json
+4 −0
Large diff (4 lines) is not loaded by default to keep the page fast.Load diff
added
packages/ui/.eslintrc.json
+7 −0
Large diff (7 lines) is not loaded by default to keep the page fast.Load diff
added
packages/ui/package.json
+27 −0
Large diff (27 lines) is not loaded by default to keep the page fast.Load diff
added
packages/ui/src/index.tsx
+61 −0
Large diff (61 lines) is not loaded by default to keep the page fast.Load diff
added
packages/ui/tsconfig.json
+8 −0
Large diff (8 lines) is not loaded by default to keep the page fast.Load diff
added
pnpm-lock.yaml
+5638 −0
Large diff (5,638 lines) is not loaded by default to keep the page fast.Load diff
added
pnpm-workspace.yaml
+11 −0
Large diff (11 lines) is not loaded by default to keep the page fast.Load diff
added
turbo.json
+16 −0
Large diff (17 lines) is not loaded by default to keep the page fast.Load diff