SPB Git

Repository Public

AI_PIPELINE.md 22.7 KB 306 lines markdown

1# AI pipeline23How the Claude studio works, function by function. Code: `src/ai/` (client, prompts, pipeline, net, sources, pricing), `src/server/jobs.ts`, `src/worker/index.ts`.45Principles the code follows:67- The Anthropic API is called from the worker only, through one function (`callClaude`). The key is an environment variable.8- The model has no tools and no web access. It receives text (and attached PDFs or images) and returns text.9- Model output is untrusted input: parsed, normalised, validated, and only then stored — as a draft.10- Every job is a row in `generation_jobs`; every completed model call is a row in `generation_usage`.1112## 1. Job kinds1314| Kind | Started by | Function | Result |15|---|---|---|---|16| `outline` | `POST /api/admin/generations` (`kind: "outline"`) | `runOutline` | An outline for the author to correct; nothing is stored besides the job output |17| `generate` | `POST /api/admin/generations` (`kind: "generate"`) | `runGenerate` | A new sheet (draft) with revision 1, origin `ai_generation` |18| `edit` | `POST /api/admin/sheets/<id>/ai-edit` | `runEdit` | A new revision, origin `ai_edit`, with the patch stored; or no change |19| `translate` | `POST /api/admin/sheets/<id>/translate` | `runTranslate` | A new linked sheet (draft) in the other language |20| `verify_sources` | `POST /api/admin/sheets/<id>/verify-sources` | `verifySources` (worker) → `checkSources` | A new revision, origin `source_check`, with updated source statuses. **No model call.** |2122AI routes answer 503 `ai_not_configured` when `ANTHROPIC_API_KEY` is unset. `verify_sources` works without a key.2324## 2. Stages2526```27intake ─▶ extraction ─▶ [outline] ─▶ generation ─▶ normalisation ─▶ validation ─▶ quality ─▶ draft revision28                                         ▲                              │29                                         └──────── repair (≤ 2) ◀───────┘30```3132| Stage | Function | What happens |33|---|---|---|34| Intake | route handler in `src/lib/api/adminroutes.ts`, `BriefSchema`, `enqueueJob` | The brief is validated, attachments are checked to exist, the job is queued (idempotency key, hourly rate limit, audit event). |35| Extraction | `extractMaterial` | Attachments are read from storage; links are fetched through `safeFetch`. Each item is wrapped in `<source_material>` tags. |36| Outline | `runOutline` → `askJson(…, zodCheck(OutlineSchema))` | Effort `medium`, at most 8 000 output tokens. Returns the outline, the list of material read, warnings, usage and a size estimate. |37| Generation | `runGenerate` → `askJson` | Effort `aiEffortGenerate`. The approved outline, if any, is included in the task. |38| Normalisation | `normalizeModelDoc` | Fields the model may not decide are overwritten or removed. |39| Validation | `parseSheet` inside the `check` callback of `askJson` | On failure the issues go back to the model (repair loop). |40| Quality | `qualityReport` | Deterministic checks; the report is returned with the job. It does not block the draft, only publication. |41| Save | `createSheet` / `saveDoc` | A draft sheet or a new revision. Conversation messages are written to `ai_messages`. |4243Job stages shown to the administrator (`generation_jobs.stage`): `queued`, `starting`, `extract`, `outline`, `generate`, `edit`, `translate`, each with a `.repair` variant, `validate`, `save`, `verify_sources`, `waiting_retry`, `done`, `failed`, `cancelled`. While text streams in, `progress.chars` is updated about every 1 500 characters.4445## 3. The brief4647`BriefSchema` (`src/ai/prompts.ts`), strict:4849| Field | Type | Default | Limits |50|---|---|---|---|51| `prompt` | string | — | 8–6 000 characters |52| `locale` | `fr` \| `en` | — | language of the sheet |53| `audience` | string | `""` | ≤ 200 |54| `level` | `beginner` \| `intermediate` \| `advanced` | `intermediate` | |55| `kind` | `explainer` \| `cheatsheet` \| `comparison` \| `architecture` \| `method` \| `case-study` | `explainer` | |56| `length` | `short` \| `standard` \| `long` | `standard` | told to the model as about 8–12, 14–20, 22–30 blocks |57| `format` | `article` \| `a4` \| `slide-16-9` \| `square` \| `portrait-4-5` | `article` | |58| `theme` | one of the five presets | `scientific-light` | |59| `constraints` | string | `""` | ≤ 2 000 |60| `attachments` | asset ids | `[]` | ≤ 8 |61| `urls` | URLs | `[]` | ≤ 5, each ≤ 2 000 characters |6263### Extraction rules6465| Material | Handling | Limits |66|---|---|---|67| PDF attachment | Sent as a `document` content block (base64) | PDFs and images together ≤ 24 MB per request |68| Image attachment | Sent as an `image` content block | same budget |69| Text, CSV, JSON, SVG attachment | Sent as text inside the wrapper | ≤ 400 000 characters each |70| Link | `safeFetch` (https, public addresses only — checked again at connect time —, ≤ 3 MB, 15 s); must answer 200 with `text/html` or `text/plain`; HTML is reduced to text by `htmlToText` | ≤ 400 000 characters of text |7172Nothing is truncated silently: an item over a limit fails the job with a message that says which item and why. Export assets cannot be attached.7374When there is neither attachment nor link, the model works from memory. The interface says so, and every reference it writes is stored as `unverified`.7576## 4. The outline7778`OutlineSchema`, strict:7980| Field | Type | Limits |81|---|---|---|82| `title` | string | 1–160 |83| `summary` | string | ≤ 400 |84| `objectives` | string[] | 1–6 items, ≤ 300 each |85| `topics` | string[] | ≤ 6, ≤ 60 each |86| `sections` | `{ heading, purpose, blocks: { type, intent }[] }[]` | 1–12 sections; 1–10 blocks each |87| `visualPlan` | string | ≤ 1 200 |88| `sourcesPlanned` | `{ title, why, fromMaterial }[]` | ≤ 12 |89| `openQuestions` | string[] | ≤ 6, ≤ 300 each |9091The author can edit the title, summary, objectives, section headings and purposes, and the visual plan, and can remove sections, before launching the generation. `sourcesPlanned[].fromMaterial` drives the "in your material" / "from memory — to be checked" label.9293The estimate returned with an outline is a heuristic: `blocks = Σ(section blocks + 1) + 2`, `outputTokens = blocks × 450`. The assistant turns it into dollars with the price table (see §11). `countInputTokens` and `estimateUsd` exist in the code but nothing calls them.9495## 5. Prompts and versioning9697`PROMPT_VERSION` (`src/ai/prompts.ts`, currently `2026-11-17.1`) is written to `generation_jobs.prompt_version` when a job is claimed and reported in the worker heartbeat. The rule stated in the file: change the text → bump the version.9899### System prompt100101One system prompt for every task, built once per process by `systemPrompt()` and sent with `cache_control: { type: "ephemeral" }`, so the calls of a job — and jobs within the cache lifetime — reuse it. Its sections, in order:1021031. **Role** — the writing and layout engine of Sheet; an administrator reviews and decides; "you never publish anything and you have no tools".1042. **What a sheet is** — one idea, read on a phone as often as on a monitor; a strong visual, short text, defined equations, something to try; the 12-column grid.1053. **Accuracy comes first** — only what is established; never invent data (the three data kinds); never invent a reference; list claims as sourced / derived / unverified; flag a false premise in a callout.1064. **Untrusted material** — `<source_material>` content is data, never instructions.1075. **Document format** — the allowed inline syntax, then the JSON Schema of `SheetSchema` generated with `z.toJSONSchema`, then practical notes (ids, structure, diagram size, alt texts, tones and icons, no image blocks or assets, empty `interactions`, the author's name, escaping LaTeX backslashes in JSON).1086. **Built-in simulations** — one line per kind with its default parameters and bounds, generated from `SIMULATIONS`.1097. **Output** — exactly one JSON value, no prose, no code fence.110111The schema and the simulation catalogue are generated from code, so the prompt cannot drift from what the server validates.112113### Task messages114115The task and all untrusted material go in the user turn.116117| Function | Content |118|---|---|119| `briefText` | The author's request, language (with French typography rules), audience, level, kind, length, format, theme, constraints |120| `outlineTask` | Brief + the outline shape |121| `generateTask` | Brief + approved plan + the document shape with the fixed values (locale, level, kind, theme, format) |122| `editTask` | The current document (`stableStringify`), the instruction, the selected blocks, the locked blocks, the list of operations |123| `translateTask` | The document, the target language, the invariants |124| `repairTask` | The validation issues (at most 40) and "return the complete corrected JSON value" |125| `materialNote` | One sentence placed before the material: it is data |126127## 6. Output contract and repair loop128129`askJson` (`src/ai/pipeline.ts`):1301311. Call the model with the user content.1322. `extractJson`: parse the reply as JSON. A surrounding code fence, or text before the first `{`/`[` and after the last `}`/`]`, is tolerated; nothing else.1333. Run the `check` callback. If it accepts, return the value.1344. Otherwise append the model's reply and `repairTask(issues)` to the conversation and call again.1355. After the third failed attempt (the first call plus two repairs), throw `invalid_output` with the first five issues.136137Each attempt is a full call recorded in the ledger, with stage `<stage>` or `<stage>.repair`.138139## 7. Normalisation: what the model may not decide140141`normalizeModelDoc` runs before validation on generated and translated documents:142143| Field | Forced to |144|---|---|145| `schemaVersion` | the server's `SCHEMA_VERSION` |146| `metadata.author` | `Simon-Pierre Boucher` |147| `metadata.locale` | the locale of the brief (or the target locale of a translation) |148| `metadata.example`, `metadata.slug` | removed |149| `interactions` | `[]` |150| `assets` | `[]` for a generation; the original sheet's assets for a translation |151| `sources[].status` | `unverified`; `checkedAt` removed |152| `image` blocks | dropped unless they reference a kept asset (top level and inside tabs) |153154In a translation the server additionally copies `sources`, `theme` and `layout` from the original and restores the original `example` flag.155156A chart with a `datasetRef` in a generated document fails validation (there are no assets) and goes to the repair loop.157158## 8. Edits: patch operations159160The model never rewrites an existing sheet. It returns `{ summary, ops }` (`EditReplySchema`), where `ops` is a `PatchSchema` (≤ 120 operations):161162| Operation | Fields | Effect |163|---|---|---|164| `replace_block` | `id`, `block` | Replaces a block (top level or in a tab); the id is kept |165| `insert_block` | `block`, `afterId` (or `null` for the top), `tabId?` | Inserts a new block; its id must be new |166| `remove_block` | `id` | Removes a block |167| `move_block` | `id`, `afterId` | Moves a top-level block |168| `set_metadata` | partial metadata (never `author`) | Merges |169| `set_theme` | `preset?`, `accent?`, `format?` | Changes theme or format |170| `upsert_source` | `source` | Adds or replaces a source by id |171| `remove_source` | `id` | Removes a source and its references in blocks and claims |172| `set_claims` | `claims` | Replaces the claims list |173174Checks applied to the reply, in order (`runEdit`):1751761. **Already done?** If a revision carrying this job's id exists (the worker died after saving), it is returned and nothing else happens.1772. **Base version.** The job stops with `bad_request` if `sheets.version` no longer equals the `baseVersion` the author was looking at.1783. **Shape.** `EditReplySchema`.1794. **Server-owned fields.** `upsert_source`: the status is kept only if the source is unchanged (same id, DOI, URL and title), otherwise `unverified`; `checkedAt` is removed. `set_metadata`: `locale`, `example` and `slug` are stripped.1805. **Scope.** When the author sent a selection: any `replace_block`, `remove_block` or `move_block` on a block outside it is an issue; an `insert_block` must come right after a selected block (or after a block inserted by the same patch, or inside a tab of a selected tab group); `set_metadata`, `set_theme`, `remove_source` and `set_claims` are refused. `upsert_source` stays allowed, so a rewritten block can cite a new source.1816. **Images.** A replaced or inserted block that contains an image must reference an existing asset of the sheet.1827. **Application.** `applyPatch(doc, ops, { respectLocks: true })`: a locked block — or a tab group containing a locked child — cannot be replaced, removed or moved; tabs cannot be nested; ids must stay unique; the result must be a valid sheet.183184Any issue goes to the repair loop. If `ops` is empty or the result equals the current document, the assistant's summary is stored and no revision is written. Otherwise `saveDoc` writes the revision with `expectedVersion = baseVersion` — if the author saved in the meantime the job fails on the version conflict and nothing is overwritten.185186The Claude panel shows the summary and the block diff, and "Undo this change" reverts to the parent revision (as a new revision).187188One job per sheet: a second `edit`, `translate` or `verify_sources` on a sheet that already has one queued or running is refused with 409 `job_in_progress`.189190## 9. Translation191192`runTranslate` creates a sibling sheet in the same localisation group. It refuses if the sheet is already in the target language or a version in that language exists.193194After normalisation and schema validation, `sameSkeleton` compares the translation with the original:195196- same number of blocks, same id and type at every position (tab contents included);197- equations: identical `latex`;198- charts: identical data points;199- matrices: identical values;200- diagrams: same node ids, same edge ids and endpoints;201- simulations: same kind, seed and parameters.202203Differences are sent back through the repair loop (at most 30 issues). The new sheet is saved with origin `ai_generation` and the summary `Traduction (<from> → <to>)`. Effort: `aiEffortEdit`.204205## 10. Model selection and effort206207| Setting | Where | Default |208|---|---|---|209| Model | `settings.aiModel`, chosen in the settings page from the live Models API list; empty → `ANTHROPIC_MODEL` → `claude-opus-5-5` | `claude-opus-5-5` |210| Effort for generation | `settings.aiEffortGenerate` (`low`…`max`) | `high` |211| Effort for edits and translation | `settings.aiEffortEdit` | `medium` |212| Effort for outlines | fixed in code | `medium` |213| Output cap per call | `settings.aiMaxOutputTokens` (4 000–128 000) | 48 000 |214215`resolveModel` checks the wanted model against `listModels()` (cached 10 minutes). If the chosen model is no longer available it falls back to the environment default; if that is unavailable too the job fails with `not_configured`. Changing the model in the settings re-fetches the list and refuses an id the key cannot call (422 `unknown_model`); if the list cannot be fetched the change is refused with 503 `ai_unavailable`. SDK errors are translated into the codes of §14 by `mapAiError`, by error class.216217Each call (`callClaude`) uses `beta.messages.stream` with `output_config: { effort }`, the cached system prompt and the abort signal of the job. No `thinking`, `temperature` or tool parameters are sent. For models matching `claude-fable-5-1`, `claude-fable-5-2`, `claude-opus-5`, `claude-opus-5-5` and `claude-sonnet-5-5`, the request adds the beta `server-side-fallback-2026-07-01` with `fallbacks: "default"`, so a request declined by a safety classifier is re-run by the API on a fallback model. Other models get no fallback.218219## 11. Budgets, ledger and pricing220221Settings (US dollars, 0 = no limit): `aiBudgetPerJobUsd` (default 5), `aiBudgetDailyUsd` (25), `aiBudgetMonthlyUsd` (200). Also `aiJobsPerHour` (40) and `aiJobTimeoutMinutes` (20).222223`assertBudget` runs **before each model call**: it sums `generation_usage.cost_usd` for the current day, the current month (database time zone) and the current job, and throws `budget` when a limit is reached. Consequences:224225- a call already started is not interrupted, so a limit can be exceeded by the cost of one call;226- a job can stop between its first call and a repair call.227228`generation_usage` gets one row per call that reaches its final message: job id, model as reported by the API, stage, input / output / cache-read / cache-write tokens, cost, duration, stop reason, request id. The row is written before refusals and truncations are turned into errors, so those are counted. Calls that end in an exception (cancellation, network failure, API error) write no row; whatever the API billed for them is not in the ledger.229230`src/ai/pricing.ts` holds list prices per million tokens for fourteen model ids, dated 2026-11 in the file. Cost = input × price + output × price + cache reads × (model's cache-read price, or 0.1 × input) + cache writes × 1.25 × input.231232Caveats:233234- The table is a copy of published prices and must be maintained by hand. It is an estimate for budgeting, not an invoice.235- A model missing from the table is costed at the most expensive rate in it (10 / 50) and flagged `priceKnown: false` in the settings page, so budgets err on the safe side.236- Tiered prices are not modelled: Claude Haiku 5.5 costs more above 100 000 prompt tokens and the ledger uses its base rate (noted in the file).237- When a server-side fallback served the request, all tokens are costed at the rate of the model named in the final message.238239The dashboard shows today's and this month's spend against the budgets; the job card shows the actual cost of the job as it runs.240241## 12. Retries, timeouts, cancellation242243| Mechanism | Behaviour |244|---|---|245| SDK retries | `maxRetries: 4`, exponential backoff on 429, 5xx and connection errors; request timeout 20 minutes |246| Job retries | An error marked retryable (`rate_limited`, API `timeout`, `upstream` for connection failures and 5xx answers) re-queues the job with `run_after = now + 30 s × 2^(attempt−1)` and stage `waiting_retry`, while `attempts < max_attempts` (3; 2 for `verify_sources`) |247| Job time limit | `aiJobTimeoutMinutes`; a one-second watch timer aborts the call and the job fails with `timeout` (not retried) |248| Cancellation | A queued job is cancelled at once. For a running job `cancel_requested` is set; the watch timer sees it within a second, aborts the stream, and the job ends `cancelled` |249| Worker shutdown | On `SIGTERM` running jobs are aborted and re-queued without counting an attempt; the worker waits up to 20 s |250| Worker crash | At start-up every job left `running` goes back to `queued`, except jobs that have used all their attempts, which are failed with code `interrupted`. A job is claimed only while `attempts < max_attempts`. |251| Concurrency | Two AI jobs at a time (`AI_CONCURRENCY`) |252253## 13. Idempotency254255- **Request level.** The browser sends one `idempotencyKey` per click; `enqueueJob` returns the existing job for a key it has seen (`generation_jobs.idempotency_key` is unique; the insert uses `ON CONFLICT DO NOTHING`, so two simultaneous requests with the same key both get the one job).256- **Generation and translation.** Before doing anything, `runGenerate` and `runTranslate` look for a revision carrying their `job_id`; if one exists (the worker died after saving) they return it instead of creating a second sheet.257- **Edit.** `runEdit` does the same: a re-run after a crash returns the revision already written for the job (with an empty summary and no diff) instead of failing on the version check. The user message is inserted once per job.258- **Source check.** The verdicts are applied to the newest working copy by source id, only where DOI, URL and title are unchanged, with up to three attempts on version conflicts.259260## 14. Failure codes261262`generation_jobs.error` is `{ code, message }`. The job card shows the message.263264| Code | Meaning | Retried |265|---|---|---|266| `not_configured` | No API key, or neither the chosen nor the default model is available to the key | no |267| `auth` | The API refused the key (401/403) | no |268| `budget` | A daily, monthly or per-job budget is reached | no |269| `bad_request` | Unreadable or oversized material, a link that cannot be fetched, the sheet changed since the request, a translation that already exists, or the API rejected the request | no |270| `refusal` | The model (after any server-side fallback) declined; the category is appended when given | no |271| `truncated` | The answer hit the output cap | no |272| `invalid_output` | The output failed validation after three attempts; the first issues are listed | no |273| `rate_limited` | API rate limit after the SDK's own retries | yes |274| `timeout` | API timeout (retryable) or job time limit (final) | see text |275| `upstream` | The API could not be reached, or another API error | connection failures and 5xx: yes; other statuses: no |276| `cancelled` | Aborted call; normally surfaces as job status `cancelled` | no |277| `interrupted` | Set by the worker at start-up on a job that was interrupted `max_attempts` times (for instance one that kills the process each time) | no |278| `version_conflict`, `not_found`, … | Errors from the storage layer (`ApiError`), passed through with their own code | no |279| `internal` | Anything else; details are in the worker log only | no |280281A failed job writes an audit event `job.failed` and a `job.failed` log line. A failed generation creates nothing.282283## 15. Reference checks284285`checkSource` (`src/ai/sources.ts`), per source, three at a time:286287| Case | Lookup | Verdict |288|---|---|---|289| Status `reviewed` | none | kept |290| Has a DOI | Crossref `works/<doi>`; on 404, `https://doi.org/<doi>` | `resolved` if the registered title matches (token overlap ≥ 0.6 on Crossref, ≥ 0.5 on the landing page); `unverified` with the registered title when it differs or the DOI does not exist; `unreachable` on other errors |291| Has a URL only | GET through `safeFetch` | `resolved` on 2xx; `unreachable` on 401/403/429 or network failure; `unverified` on other statuses |292| Neither | none | `unverified` ("confirm it by hand") |293294`resolved` means a record with that title exists. `reviewed` is set only by the administrator in the sources panel. Editing a source's title, DOI or URL in the panel resets it to `unverified` unless it was `reviewed`.295296## 16. What is not done297298- No web search and no web-fetch tool: the model cannot look anything up. Only links the administrator supplies are fetched, by the server.299- No tool use, no code execution, no agent loop.300- No automatic publication, scheduling or approval by the model.301- No image generation; the model cannot create image blocks or assets.302- No structured-output (`output_config.format`) constraint: the JSON contract is enforced by parsing and the repair loop.303- No embeddings, retrieval index or memory across jobs. The per-sheet conversation is a display log; previous exchanges are not sent back to the model.304- No batch processing; no token counting before a call.305- No moderation step other than the API's own safeguards.306
Files main
Menu

Theme