spb/airiskindex Public

1# LLM Rater — Anthropic API Reference (August 2026)23Practical, current reference for the task-rating pipeline in `apps/worker/src/raters/` (CLAUDE.md §6).4Scope: model choice, Message Batches API, structured outputs, prompt caching, rate limits, TypeScript SDK5patterns, and a concrete cost estimate for rating ~18,000 O*NET task statements × 3 samples.67All facts verified against the official docs on **2026-08-05**. Note: `https://docs.claude.com/en/api/overview`8(the URL in CLAUDE.md §6) now 301-redirects to `https://platform.claude.com/docs/en/api/overview` — the docs9moved to `platform.claude.com`.1011Sources:1213- Models overview — https://platform.claude.com/docs/en/about-claude/models/overview14- Pricing — https://platform.claude.com/docs/en/about-claude/pricing15- Batch processing — https://platform.claude.com/docs/en/build-with-claude/batch-processing16- Rate limits — https://platform.claude.com/docs/en/api/rate-limits17- Structured outputs — https://platform.claude.com/docs/en/build-with-claude/structured-outputs18- Prompt caching — https://platform.claude.com/docs/en/build-with-claude/prompt-caching1920---2122## 1. Current model lineup and pricing (per MTok, standard API)2324| Model | ID (set via `RATER_MODEL`) | Context / max output | Input | Output | Batch input | Batch output | Cache read |25|---|---|---|---|---|---|---|---|26| Claude Fable 5 | `claude-fable-5` | 1M / 128K | $10 | $50 | $5 | $25 | $1.00 |27| Claude Opus 5 | `claude-opus-5` | 1M / 128K | $5 | $25 | $2.50 | $12.50 | $0.50 |28| Claude Opus 4.8 | `claude-opus-4-8` | 1M / 128K | $5 | $25 | $2.50 | $12.50 | $0.50 |29| Claude Sonnet 5 (intro, **through 2026-08-31**) | `claude-sonnet-5` | 1M / 128K | $2 | $10 | $1 | $5 | $0.20 |30| Claude Sonnet 5 (from 2026-09-01) | `claude-sonnet-5` | 1M / 128K | $3 | $15 | $1.50 | $7.50 | $0.30 |31| Claude Sonnet 4.6 | `claude-sonnet-4-6` | 1M / 128K | $3 | $15 | $1.50 | $7.50 | $0.30 |32| Claude Haiku 4.5 | `claude-haiku-4-5` | 200K / 64K | $1 | $5 | $0.50 | $2.50 | $0.10 |3334Cache writes: 1.25× base input (5-min TTL) or 2× base input (1-hour TTL). Cache multipliers **stack with the35batch discount** (confirmed in the pricing doc: "These multipliers stack with other pricing modifiers,36including the Batch API discount"). All model IDs from the 4.6 generation onward are dateless pinned37snapshots — no date suffix to append.3839Tokenizer note: models from Opus 4.7 onward (incl. Fable 5, Opus 5, Sonnet 5) use a tokenizer that yields40~30% more tokens for the same text than Sonnet 4.6/Haiku 4.5. Budget token estimates per model, not globally.4142### Which tier for large-scale 5-point task rating?4344For a structured, rubric-guided 5-point rating with a short rationale (a classification-plus-justification45task, not open-ended reasoning):4647- **Haiku 4.5** — cheapest ($0.50/$2.50 batch), fast, supports structured outputs. Likely adequate for the48  bulk of clear-cut tasks, but weakest calibration on ambiguous tasks — expect more >1-point disagreements49  flowing into the expert-panel queue (which costs human time).50- **Claude Sonnet 5** — **recommended default.** Near-Opus quality on judgment tasks; at the introductory51  price ($1/$5 batch through Aug 31, 2026) the full 54k-rating job costs ~$160–250 (see §7) — the marginal52  cost over Haiku is trivial relative to the human-review pipeline it feeds. Caveat: adaptive thinking is on53  by default and thinking tokens bill as output; for this task set `thinking: {type: "disabled"}` (accepted54  on Sonnet 5) or keep it on with `output_config: {effort: "low"}` and budget extra output tokens.55- **Opus 5 / Fable 5** — overkill for the volume run. Total quality of the index is bounded by the56  methodology and human validation, not by Opus-vs-Sonnet deltas on a 5-point scale. Best use: (a) rate the57  expert-panel calibration subset with Opus 5 as a second opinion, or (b) adjudicate flagged disagreements.5859Practical plan: pilot ~500 tasks on both Haiku 4.5 and Sonnet 5, compare agreement with the human 5% sample,60and pick per the disagreement rate. Sampling-variance note for the 3-samples design: **Sonnet 5, Opus 5, and61Opus 4.7+ reject non-default `temperature`/`top_p`/`top_k` (400 error)** — you cannot set temperature for62the 3 samples on those models; between-sample variance is whatever the model naturally produces. Haiku 4.563and Sonnet 4.6 still accept temperature.6465## 2. Message Batches API6667Docs: https://platform.claude.com/docs/en/build-with-claude/batch-processing6869- **Endpoints:** `POST /v1/messages/batches` (create), `GET /v1/messages/batches/{id}` (poll),70  `GET <results_url>` (stream `.jsonl` results), `POST /v1/messages/batches/{id}/cancel`, `GET /v1/messages/batches` (list).71- **Discount:** flat **50% off both input and output tokens**, all models, all features (vision, tools,72  structured outputs, prompt caching all supported inside batches).73- **Limits:** max **100,000 requests or 256 MB** per batch, whichever comes first. Each request needs a74  `custom_id` matching `^[a-zA-Z0-9_-]{1,64}$`. `max_tokens` must be ≥ 1 (`max_tokens: 0` cache pre-warming75  is rejected inside batches).76- **Timing:** most batches finish < 1 hour; hard **24-hour expiration** — requests not processed by then77  come back as `expired` (not billed) and must be resubmitted.78- **Results:** available at `results_url` once `processing_status === "ended"`; delivered as JSONL, **in79  arbitrary order — always key by `custom_id`, never by position**. Results are downloadable for **29 days**80  after batch creation; persist them to `data/derived/ratings/` + DB immediately.81- **Per-request result types:** `succeeded` (has `.result.message`), `errored` (invalid request → fix and82  resubmit; server error → safe to retry; not billed), `canceled` (not billed), `expired` (not billed —83  resubmit).84- **Extended output:** Opus 5/4.8/4.7/4.6, Sonnet 5/4.6 support up to 300k output tokens in batches via the85  `output-300k-2026-03-24` beta header (not needed for 500-token ratings).86- **No server-side idempotency key on batch create** — dedupe is your job (below).8788### Marrying batches with BullMQ8990Our invariant (CLAUDE.md §6): BullMQ job ID = deterministic hash of `task_id + prompt_version`. Extend to91`hash(task_id + prompt_version + sample_index)` since each task is rated 3×. Recommended architecture:92931. **`rating-request` rows, not per-rating jobs.** Persist one DB row per (task, prompt_version, sample)94   with status `pending`. A hex SHA-256 (truncated to 32–48 chars) of `taskId:promptVersion:sampleIdx`95   satisfies both the BullMQ job-ID and the batch `custom_id` charset — **use the same string for both**, so96   a batch result maps 1:1 to a job/row.972. **`batch-submitter` job** (BullMQ, repeatable or triggered): collects up to 100k `pending` rows, calls98   `batches.create()`, stores `batch_id` on the rows *before* flipping them to `submitted` — if the process99   dies after create but before persist, on restart list recent batches and reconcile by `custom_id` rather100   than re-creating (this is the idempotency seam; the API will happily accept duplicate custom_ids across101   batches and bill you twice).1023. **`batch-poller` job** with BullMQ job ID = `poll:${batchId}` (deterministic → re-enqueue is a no-op),103   repeat/delay ~60s until `processing_status === "ended"`.1044. **`batch-ingester`** streams results, and per result: store raw JSON response, parsed score, model,105   prompt version, timestamp (full audit trail); `errored`(server)/`expired` → flip row back to `pending`106   so the next submitter run resubmits; `errored`(invalid_request) → dead-letter for inspection.1075. A prompt-version bump changes every hash → new custom_ids → old cached rows are naturally invalidated,108   exactly matching the CLAUDE.md rule.109110The whole 54,000-rating run fits in **one batch** (well under 100k requests and 256 MB), even at the Start111tier queue limit (200k requests in processing queue).112113## 3. Structured outputs for the 5-point rating114115**Structured outputs is GA** (no beta header; the old `structured-outputs-2025-11-13` header and top-level116`output_format` request param are deprecated transition shims). Two mechanisms:1171181. **JSON outputs** — `output_config: {format: {type: "json_schema", schema: {...}}}` constrains the119   response text to schema-valid JSON.1202. **Strict tool use** — `strict: true` on a tool definition; guarantees `tool_use.input` validates.121122**Recommendation: use JSON outputs (`output_config.format`), not tool-forced JSON.** Rationale:123124- It's the purpose-built mechanism for "the response *is* the structured object" — no fake tool, no125  `tool_choice` forcing, one fewer moving part in the audit trail.126- Guaranteed-valid JSON with `required` fields → the "parsed score" column can be extracted without retry127  loops.128- Works with the **Batches API**, streaming, and thinking. (Incompatible with citations and prefilling —129  neither is used here.)130- Schema limits that matter to us: `enum` is supported (use `"score": {"enum": [1,2,3,4,5]}` — do **not**131  use `minimum`/`maximum`, numeric range constraints are unsupported); no `minLength`/`maxLength` on the132  rationale string (enforce length in the prompt); `additionalProperties: false` is mandatory on every133  object.134- First use of a schema pays a one-time grammar-compilation latency; compiled grammars are cached 24h —135  irrelevant inside a batch run that reuses one schema 54,000×. Note that **changing `output_config.format`136  invalidates the prompt cache**, so treat the schema like the rubric: versioned with `RATER_PROMPT_VERSION`.137138Suggested schema:139140```json141{142  "type": "object",143  "properties": {144    "score": { "type": "integer", "enum": [1, 2, 3, 4, 5] },145    "rationale": { "type": "string", "description": "2-3 sentence justification citing the rubric" },146    "confidence": { "type": "string", "enum": ["low", "medium", "high"] }147  },148  "required": ["score", "rationale", "confidence"],149  "additionalProperties": false150}151```152153Structured outputs injects a system-prompt preamble explaining the format (small, fixed token overhead per154request).155156## 4. Prompt caching for the shared rubric157158Layout: `tools` → `system` → `messages` renders in that order and caching is a **byte-exact prefix match**.159Put the ~2k-token rubric in `system` with `cache_control` on its last block; the per-task variable content160(task statement, occupation context) goes in the user message, after the breakpoint:161162```ts163system: [164  { type: "text", text: RUBRIC_V3,            // frozen per RATER_PROMPT_VERSION — no timestamps, no task data165    cache_control: { type: "ephemeral", ttl: "1h" } },166],167messages: [{ role: "user", content: `Occupation: ${occ}\nTask: ${taskStatement}\nRate this task.` }]168```169170Key facts:171172- **Pricing:** cache read = 0.1× base input; write = 1.25× (5-min TTL) or 2× (1-hour TTL). **Stacks with the173  batch discount** → a cached rubric token inside a batch costs 0.05× base input.174- **Inside batches use the 1-hour TTL** (official recommendation): batch requests process concurrently over175  up to an hour, so 5-min entries can lapse between hits. Caveat: cache hits inside a batch are176  best-effort — parallel workers may each miss; treat the §7 "with caching" numbers as the optimistic bound.177- **Minimum cacheable prefix is model-dependent and non-monotonic:** 512 tokens (Opus 5/Fable 5), 1,024178  (Opus 4.8, Sonnet 5, Sonnet 4.6), 2,048 (Opus 4.7), **4,096 (Haiku 4.5, Opus 4.6)**. **A 2k-token rubric179  silently will not cache on Haiku 4.5** — no error, just `cache_creation_input_tokens: 0`. If Haiku is180  chosen, either accept uncached input (still cheap) or grow the cached prefix ≥4,096 tokens (e.g. include181  the scoring examples/anchors in the system block).182- Cost math for the rubric alone (Sonnet 5 intro, batch, 54k requests, 2,000 tokens):183  uncached = 54,000 × 2,000 × $1/MTok = **$108**; cached (1 write + 54k reads at 0.05×) ≈ 108M × $0.10/MTok184  ≈ **$10.80**. ~10× saving on the shared-prefix portion.185- Verify via `usage.cache_read_input_tokens` in each batch result; zero across the run means a silent186  invalidator (non-deterministic serialization, per-request content above the breakpoint).187188## 5. Rate limits relevant to batch rating throughput189190Docs: https://platform.claude.com/docs/en/api/rate-limits — organizations sit on Start / Build / Scale /191Custom tiers (auto-assigned by usage history; monthly spend caps of $500 / $1,000 / $200,000).192193**Message Batches API has its own limits, shared across all models** (separate from Messages ITPM/OTPM):194195| Tier | API requests/min | Max batch requests in processing queue | Max requests per batch |196|---|---|---|---|197| Start | 1,000 | 200,000 | 100,000 |198| Build | 2,000 | 300,000 | 100,000 |199| Scale | 4,000 | 500,000 | 100,000 |200201Implications for us: 54,000 ratings fit in a single batch at any tier; even a full-index recompute with202several prompt versions in flight stays under the Start-tier queue (200k). Batch throughput inside the203queue is demand-based, not tier-based — under load, more requests may hit the 24h expiry; the ingester's204resubmit path (§2) handles that.205206For any **synchronous** rating path (e.g. on-demand re-rate of a single task): limits are per-model RPM +207ITPM/OTPM (e.g. Start tier, Sonnet 5: 1,000 RPM / 2M ITPM / 400k OTPM). **Cache reads do not count toward208ITPM** on current models, so the cached rubric also multiplies effective sync throughput. Opus 5 and209Sonnet 5 each have rate-limit buckets separate from the combined Opus 4.x / Sonnet 4.x pools. On 429, honor210`retry-after`; the SDK does this automatically (default 2 retries).211212## 6. TypeScript SDK (`@anthropic-ai/sdk`) patterns213214The SDK auto-retries 408/409/429/5xx with exponential backoff (`maxRetries` default 2; timeout default21510 min, in **milliseconds** on TS). Sketch of the worker pieces (model from `RATER_MODEL`, never hardcoded):216217```ts218import Anthropic from "@anthropic-ai/sdk";219import { createHash } from "node:crypto";220221const client = new Anthropic(); // ANTHROPIC_API_KEY from env222const MODEL = process.env.RATER_MODEL!;223const PROMPT_VERSION = process.env.RATER_PROMPT_VERSION!;224225export const ratingId = (taskId: string, sample: number) =>226  createHash("sha256").update(`${taskId}:${PROMPT_VERSION}:${sample}`).digest("hex").slice(0, 48);227// valid as BullMQ job ID *and* batch custom_id (^[a-zA-Z0-9_-]{1,64}$)228229// --- batch-submitter job ---230export async function submitBatch(rows: PendingRating[]) {231  const batch = await client.messages.batches.create({232    requests: rows.map((r) => ({233      custom_id: r.id, // = ratingId(...)234      params: {235        model: MODEL,236        max_tokens: 1024, // headroom over the ~500-token rating237        system: [{ type: "text" as const, text: RUBRIC,238                   cache_control: { type: "ephemeral" as const, ttl: "1h" as const } }],239        output_config: { format: { type: "json_schema", schema: RATING_SCHEMA } },240        messages: [{ role: "user" as const, content: r.taskPrompt }],241      },242    })),243  });244  await db.markSubmitted(rows.map((r) => r.id), batch.id); // persist batch_id BEFORE returning245  return batch.id;246}247248// --- batch-poller job (BullMQ delayed/repeatable, jobId: `poll:${batchId}`) ---249export async function pollBatch(batchId: string): Promise<boolean> {250  const batch = await client.messages.batches.retrieve(batchId);251  return batch.processing_status === "ended"; // else re-schedule in ~60s252}253254// --- batch-ingester job ---255export async function ingestResults(batchId: string) {256  for await (const result of await client.messages.batches.results(batchId)) {257    switch (result.result.type) {258      case "succeeded": {259        const msg = result.result.message;260        const text = msg.content.find((b) => b.type === "text")?.text ?? "";261        await db.storeRating({262          customId: result.custom_id,          // → task_id + prompt_version + sample263          model: msg.model,264          promptVersion: PROMPT_VERSION,265          rawResponse: JSON.stringify(msg),    // full audit trail (CLAUDE.md §6)266          parsed: JSON.parse(text),            // schema-guaranteed {score, rationale, confidence}267          usage: msg.usage,                    // incl. cache_read_input_tokens for cost telemetry268          ratedAt: new Date().toISOString(),269        });270        break;271      }272      case "errored":273        if (result.result.error.type === "invalid_request") await db.deadLetter(result.custom_id, result.result);274        else await db.markPending(result.custom_id); // server error — next submitter run retries275        break;276      case "expired":277      case "canceled":278        await db.markPending(result.custom_id);279        break;280    }281  }282}283```284285Notes: results arrive in arbitrary order — key everything by `custom_id`. For sync one-off calls, prefer286`client.messages.parse()` with `zodOutputFormat(...)` from `@anthropic-ai/sdk/helpers/zod` (typed287`parsed_output`); for batches, validate the JSON against the same Zod schema at ingest time. Handle errors288with typed classes (`Anthropic.RateLimitError`, `Anthropic.APIError`), never string-matching.289290## 7. Cost estimate: 18,000 tasks × 3 samples = 54,000 ratings291292Assumptions: rubric 2,000 tokens (shared, cacheable), per-task suffix ~300 tokens (statement + occupation293context + instruction), completion ~500 tokens. Totals: **input 124.2M tokens** (108M rubric + 16.2M294variable), **output 27M tokens**. "With caching" = 1h-TTL cache, optimistic ~100% hit rate (real batch runs295will land between the last two columns), cache read = 0.1× input, stacked with the 50% batch discount296(0.05× net). Prices per §1.297298| Model | Sync, no cache | Batch only (50%) | Batch + rubric caching |299|---|---:|---:|---:|300| Haiku 4.5 ($1/$5) | $259 | **$130** | ≈$130 (2k rubric **below Haiku's 4,096-token cache minimum** — won't cache; pad rubric ≥4k to reach ≈$92) |301| Sonnet 5 — intro thru 2026-08-31 ($2/$10) | $518 | $259 | **≈$162** ($10.80 cached rubric + $16.20 variable input + $135 output) |302| Sonnet 5 / Sonnet 4.6 — standard ($3/$15) | $778 | $389 | **≈$243** ($16.20 + $24.30 + $202.50) |303| Opus 5 ($5/$25) | $1,296 | $648 | **≈$405** ($27 + $40.50 + $337.50) |304| Fable 5 ($10/$50) | $2,592 | $1,296 | ≈$810 (not recommended for this workload) |305306Takeaways:307308- **Output tokens dominate** once caching is on (83% of the Sonnet cost). Keep rationales tight in the309  prompt, and control thinking (Sonnet 5/Opus 5 think by default; thinking bills as output — disable it or310  set `effort: "low"` or the 500-token completion assumption breaks).311- The entire volume run costs **$130–$405** depending on model — negligible against the human-expert312  pipeline. This argues for Sonnet 5 (or even an Opus 5 second-pass on flagged tasks) over Haiku313  penny-pinching; if the run happens before **2026-09-01**, Sonnet 5's intro pricing makes it ~$162.314- A full re-run per `INDEX_VERSION`/`RATER_PROMPT_VERSION` bump is affordable, which supports the315  methodology-integrity rule that recomputations create new immutable runs.316317## Where the docs contradict / update CLAUDE.md assumptions3183191. **Docs URL moved:** CLAUDE.md §6 points to `https://docs.claude.com/en/api/overview`; that 301-redirects320   to `https://platform.claude.com/docs/en/api/overview`. Update the reference.3212. **Temperature-based sampling is gone on current models:** if the "3 samples per task" design assumed322   `temperature > 0` resampling, note that Sonnet 5 / Opus 5 / Opus 4.7+ **reject** non-default323   `temperature`/`top_p`/`top_k` with a 400. Variance across samples is natural model stochasticity only324   (or use Haiku 4.5 / Sonnet 4.6, which still accept temperature).3253. **`output_format` param is deprecated** — any prototype code using it should move to326   `output_config: {format: ...}` (GA, no beta header).3274. No contradiction on the audit-trail/idempotency requirements — the Batch API's `custom_id` +328   result-streaming model fits the deterministic-hash design directly; the only gap is that batch **create**329   has no server-side idempotency key, so the submitter must persist `batch_id` transactionally (§2.2).330