SPB Git

Repository Public

OPERATIONS.md 18.7 KB 284 lines markdown

1# Operations23Running Sheet day to day: processes, logs, health, the job queue, incidents, housekeeping, upgrades.45## 1. Processes67| Process | Entry point | State it holds | Restart cost |8|---|---|---|---|9| Web | `server.ts` | none (sessions, limits and jobs are in PostgreSQL); a 10-minute in-memory cache of the model list | open event streams drop; browsers fall back to polling |10| Worker | `src/worker/index.ts` | the running jobs and one headless browser (closed after 2 minutes idle) | running jobs are re-queued |11| Backup | `scripts/backup.sh` | — | scheduled, runs and exits |1213Run **one** worker. At start-up a worker puts every job marked `running` back in the queue, which would steal the jobs of another live worker (the single-worker assumption is stated in a comment in `src/worker/index.ts`). Inside the single worker, two AI jobs and one export run at a time.1415Worker start-up sequence: check that `STORAGE_ROOT` is writable (fatal if not) → start Chromium once to prove it works (warning if it does not) → fail jobs that used all their attempts (`interrupted`) → re-queue jobs left running → write the heartbeat → run housekeeping → `LISTEN sheet_jobs` → loop (wake on notification or every 2 s).1617Signals: both processes handle `SIGTERM` and `SIGINT`. Web: stops accepting connections, exits within 8 s. Worker: aborts model calls, re-queues its jobs without counting the attempt, waits up to 20 s, closes the browser and the database pool.1819## 2. Logs2021Both processes write to stdout/stderr; where the lines end up is decided by the process manager (PM2 log files, journald, `docker logs`). `LOG_LEVEL` (`debug`, `info`, `warn`, `error`; default `info`) applies to lines written through `src/lib/log.ts`.2223### Worker: JSON lines2425One JSON object per line: `t` (ISO time), `level`, `msg`, then fields. `warn` and `error` go to stderr. Anything that looks like an Anthropic key (`sk-ant-…`) is truncated before the line is written. Document bodies and request bodies are never logged.2627| `msg` | Level | Fields |28|---|---|---|29| `worker.started` | info | `worker` (host:pid), `requeued`, `chromium`, `promptVersion` |30| `worker.chromium_missing` | warn | `detail` |31| `worker.stopping` | info | `running` |32| `worker.loop`, `worker.fatal`, `exports.tick` | error | `error`, `detail` |33| `job.started` | info | `jobId`, `kind`, `attempt` |34| `job.succeeded` | info | `jobId`, `kind`, `ms` |35| `job.retry` | warn | `jobId`, `kind`, `code`, `inSeconds` |36| `job.cancelled` | info | `jobId`, `kind` |37| `job.failed` | error | `jobId`, `kind`, `code`, `message`; for non-AI errors also `error`, `detail` |38| `ai.call` | info | `stage`, `jobId`, `model`, `inputTokens`, `outputTokens`, `cacheRead`, `costUsd`, `ms`, `stop` |39| `export.succeeded` | info | `exportId`, `format`, `ms` |40| `export.failed` | error | `exportId`, `format`, `attempt`, `detail` |41| `export.fonts_missing` | warn | `detail` |42| `housekeeping.failed` | warn | `error`, `detail` |4344### Web: little4546The web process writes three kinds of JSON lines of its own — `web.listening` (port, host, mode), `web.unhandled`, and `api.unhandled` (`error`, `detail`) for an unexpected error inside an API handler (`src/lib/http.ts`; the client receives `500 {"error":{"code":"internal"}}`) — plus whatever Next.js prints.4748There is no access log and no per-request line; use the reverse proxy's access log. Sign-ins, failures, publications, uploads and job events are not in the logs: they are in the `audit_events` table (studio → Revisions → audit, or SQL).4950```sql51SELECT at, actor, action, entity, entity_id, meta, ip FROM audit_events ORDER BY at DESC LIMIT 50;52SELECT at, meta->>'why' AS why, ip FROM audit_events WHERE action = 'login.failed' AND at > now() - interval '1 day';53```5455No error-reporting service (Sentry or similar) is integrated.5657## 3. Health5859| Check | How | Meaning |60|---|---|---|61| Liveness | `GET /api/health` | `200 {"ok":true}`: the web process answers and `SELECT 1` works. `503 {"ok":false}` otherwise. No detail is exposed. |62| Dashboard tiles | `/admin` (`GET /api/admin/dashboard`) | Database (version and size), Claude API (key present, model list reachable, model available — spends no tokens), job queue and worker, storage (write probe in `STORAGE_ROOT`), antivirus (`clamd` probe, or "not configured") |63| Worker heartbeat | `worker_heartbeat` row, updated every 15 s | The dashboard calls the worker alive when the last beat is under 45 s old; otherwise "stopped" or "never started" |64| Chromium | in the heartbeat `info` (`chromium`, `chromiumDetail`) | evaluated once, when the worker starts |65| Spend | dashboard, from `generation_usage` | today and this month against the budgets |6667```sql68SELECT name, now() - beat_at AS age, info FROM worker_heartbeat;69```7071The web process does not monitor the worker beyond reading that row; alert on it yourself if exports and AI jobs matter (for example: page when `age > interval '2 minutes'`).7273## 4. The job queue7475Two tables, same mechanics: `generation_jobs` (AI jobs and source checks) and `export_jobs`.7677### States7879```80generation_jobs:   queued ──claim──▶ running ──▶ succeeded81                      ▲                 │──────▶ failed82                      │                 │──────▶ cancelled83                      └── retry / worker restart / shutdown84                   queued ──cancel──▶ cancelled8586export_jobs:       queued ──▶ running ──▶ succeeded | failed      (2 attempts, 20 s apart)87```8889A job is claimed with `UPDATE … WHERE id = (SELECT … FOR UPDATE SKIP LOCKED LIMIT 1)`, oldest first, when `run_after <= now()` and `attempts < max_attempts`. `attempts` is incremented at each claim. `stage` and `progress` describe what a running job is doing; `locked_by` is the worker (`host:pid`).9091Retries: retryable AI errors re-queue with a delay of 30 s × 2^(attempt − 1) while `attempts < max_attempts` (3; 2 for source checks). Exports are retried once after 20 s.9293### Inspect9495```sql96-- what is waiting or running97SELECT id, kind, status, stage, attempts, max_attempts, run_after, locked_by, created_at, started_at98FROM generation_jobs WHERE status IN ('queued', 'running') ORDER BY created_at;99100-- recent failures and why101SELECT id, kind, error->>'code' AS code, error->>'message' AS message, attempts, finished_at102FROM generation_jobs WHERE status = 'failed' ORDER BY finished_at DESC LIMIT 20;103104-- what a job cost, call by call105SELECT at, stage, model, input_tokens, output_tokens, cache_read_tokens, cost_usd, stop_reason, request_id106FROM generation_usage WHERE job_id = '<uuid>' ORDER BY at;107108-- spend per day109SELECT date_trunc('day', at) AS day, sum(cost_usd) AS usd, count(*) AS calls FROM generation_usage GROUP BY 1 ORDER BY 1 DESC LIMIT 14;110111-- exports112SELECT id, format, status, attempts, error, created_at, finished_at FROM export_jobs ORDER BY created_at DESC LIMIT 20;113```114115In the studio: `GET /api/admin/generations` lists recent jobs; the job card shows stage, cost and error.116117### Cancel118119From the studio: the **Cancel** button on a job card (`POST /api/admin/generations/<id>/cancel`). In SQL:120121```sql122-- queued: cancelled at once123UPDATE generation_jobs SET status = 'cancelled', stage = 'cancelled', finished_at = now() WHERE id = '<uuid>' AND status = 'queued';124-- running: the worker sees the flag within a second and aborts the model call125UPDATE generation_jobs SET cancel_requested = true WHERE id = '<uuid>' AND status = 'running';126```127128Exports have no cancel; delete the queued row, or let it finish.129130### Re-queue131132```sql133-- run a failed job again (same input, attempts reset)134UPDATE generation_jobs135SET status = 'queued', stage = 'queued', attempts = 0, run_after = now(), error = NULL, finished_at = NULL, cancel_requested = false, locked_by = NULL136WHERE id = '<uuid>' AND status = 'failed';137SELECT pg_notify('sheet_jobs', '<uuid>');   -- optional: the worker also polls every 2 s138139-- a failed export140UPDATE export_jobs SET status = 'queued', attempts = 0, run_after = now(), error = NULL, finished_at = NULL, locked_by = NULL WHERE id = '<uuid>';141```142143Before re-queuing, know what a re-run does:144145- `generate` and `translate` first look for a revision already carrying the job's id and return it if found: a re-run never creates a second sheet.146- `edit` returns the revision it already wrote, if any. Otherwise it checks that the sheet is still at the version the request was based on; if not, it fails again with "The sheet changed after this request was sent" — re-send the request from the editor instead.147- `outline` and `verify_sources` can be re-run freely.148- A re-run spends tokens again.149150### A job that is stuck in `running`151152With the worker alive, a job cannot outlive `aiJobTimeoutMinutes` (default 20): the watch timer aborts it. If the worker died, the job stays `running` until a worker starts, which re-queues it. `attempts` grows by one per claim; a job that has used all its attempts is not claimed again, and at the next worker start it is failed with code `interrupted` ("The job was interrupted too many times and was stopped"). A queued job with `attempts >= max_attempts` stays queued until that start-up pass; re-queue it with `attempts = 0` if it should run.153154## 5. Common incidents155156### The Claude API key is refused157158Signs: jobs fail with code `auth` ("The Anthropic API key was refused"); the dashboard's Claude tile is failing.159160Do: check `ANTHROPIC_API_KEY` in the environment of **both** processes (the worker makes the calls; the web process lists models), restart them after changing it. If the tile says the model is not available, pick another model in Settings; a chosen model that disappeared falls back to `ANTHROPIC_MODEL` automatically when that one exists.161162### No key at all163164The studio works as a manual editor. AI routes answer `503 ai_not_configured`; the assistant and the Claude panel say so. Source checks still work.165166### Budget reached167168Signs: jobs fail with code `budget` and a message naming the limit (daily, monthly or per job); the dashboard shows "Budget reached".169170Do: wait for the next day or month, or raise the budget in Settings → Claude (0 = no limit). The check runs before each model call, so a job can stop between its first call and a repair call. To see what consumed it, use the spend queries above.171172### Exports fail because Chromium is missing173174Signs: `worker.chromium_missing` at start-up ("Chromium could not start …"); export jobs fail with a message about the executable; published sheets have no PDF button.175176Do: as the user that runs the worker, `node_modules/.bin/playwright-core install chromium-headless-shell` (add `--with-deps` on Debian/Ubuntu, as root, for system libraries), or set `CHROMIUM_PATH`. Restart the worker: the check — a real launch of the browser — runs only at start-up.177178A PDF request for a single block is refused up front (`422 whole_sheet_only`); export one block as PNG or SVG. Other export failures: `Render page answered HTTP …` means the worker's browser could not load `INTERNAL_URL/render/<token>` — check that `INTERNAL_URL` reaches the web process from the worker and that `APP_SECRET` is the same for both. "The image would be … px" means the PNG is too large; use a lower scale or PDF. "The sheet did not lay out at a readable width; export aborted" means the render page collapsed in the headless browser; the worker refuses to produce an unreadable file — check the render page by hand before retrying.179180To produce the missing reader PDF of a published sheet, publish it again (a new publication queues a new PDF), or insert the job:181182```sql183INSERT INTO export_jobs (sheet_id, publication_id, format, options, is_public)184SELECT id, publication_id, 'pdf', '{}', true FROM sheets WHERE slug = '<slug>' AND locale = '<fr|en>' AND status = 'published';185```186187### The worker is down188189Signs: dashboard "Worker stopped" or "never started"; jobs stay `queued`; exports stay pending.190191Do: read the worker's last log lines (`worker.fatal` gives the reason — most often `STORAGE_ROOT is not usable` or a database connection failure), fix, restart. Nothing is lost: queued jobs wait, running ones are re-queued at start-up.192193### Disk full194195Signs: uploads and exports fail; the storage tile fails its write probe; PostgreSQL errors if the database volume is the one that filled; backups fail.196197Where space goes: `STORAGE_ROOT` (uploads are capped by `STORAGE_QUOTA_GB`; exports are outside the quota and are pruned after seven days), `BACKUP_DIR` (a full copy of storage per day, `BACKUP_KEEP_DAYS` of them), process logs, PostgreSQL.198199Do: lower `BACKUP_KEEP_DAYS` or move `BACKUP_DIR`; rotate logs; delete unused files in Media & data (a file used by a sheet cannot be deleted). Studio exports older than seven days are removed by the worker's hourly housekeeping. Do not delete files under `STORAGE_ROOT` by hand: the `assets` rows would point at nothing.200201```sql202SELECT kind, count(*), pg_size_pretty(sum(bytes)) FROM assets WHERE deleted_at IS NULL GROUP BY kind ORDER BY sum(bytes) DESC;203```204205### Version conflicts206207Signs: the editor shows "The sheet was changed elsewhere … Reload"; an AI edit fails with "The sheet changed after this request was sent"; API answers `409 version_conflict`.208209Cause: two writers on one sheet — two tabs, or a job that finished after a manual save. This is the optimistic lock doing its work; nothing was overwritten.210211Do: reload the tab that lost. Its unsaved changes are gone from the server's point of view; copy anything important out of the tab before reloading. Then re-send the AI request if needed.212213### "Invalid document — not saved"214215The working copy in the browser does not pass the schema; autosave is suspended for the whole sheet until it does. The banner lists the first issues. Fix the issue named in the banner, or undo.216217### Lost passkeys2182191. Another enrolled device still works: sign in there and remove the lost passkey in Settings.2202. A recovery code: sign-in page → "Lost your device? Use a recovery code". The 30-minute session can only enrol a new passkey; doing so revokes all other passkeys and sessions and issues new codes.2213. Nothing left: on the server, interactively, `node_modules/.bin/tsx scripts/setup-token.ts --recovery`, open the printed link within 30 minutes on the new device. Without `--recovery` the new passkey is added and the old ones are kept.222223Each path is in the audit trail (`login.recovery_code`, `recovery.cli`, `passkey.enrolled.*`).224225### Locked out by failed attempts226227`429 locked_out` with `retryAfterSeconds`. The lock-out is per client address and computed from `login.failed` events of the last hour (5 → 1 min, 10 → 5 min, 20 → 1 h). It clears by itself. If every client appears under the same address (`local`, or a proxy's address), the proxy is not sending `X-Forwarded-For`, or a second proxy sits between the client and the one that appends the last value.228229### Passkeys stop working after a domain change230231Passkeys are bound to the RP ID (`WEBAUTHN_RP_ID`, default the host of `APP_URL`). Changing the host invalidates all of them. Either set `WEBAUTHN_RP_ID` to the old value where the new host is a subdomain of it, or re-enrol with `scripts/setup-token.ts --recovery`.232233### A published sheet has broken images234235Images are served from `/media/<id>` only when flagged public, which happens at publication for the images listed in the published document; unpublishing a sheet withdraws the flag from images no other published sheet uses. Publishing again re-flags them. Private previews serve their images through `/preview/<token>/asset/<id>` and do not depend on the flag.236237## 6. Housekeeping done by the worker238239Once at start-up and then every hour:240241| Table / files | Rule |242|---|---|243| `webauthn_challenges` | deleted 1 hour after expiry |244| `rate_limits` | windows older than 2 days deleted |245| `sessions` | deleted 30 days after expiry or revocation (a session is refused after 14 days, or 7 days without use, well before its row is deleted) |246| `setup_tokens` | deleted 30 days after expiry |247| `preview_tokens` | deleted 7 days after expiry |248| export files | assets of kind `export` older than 7 days are deleted (file and row marked), except the reader PDF of a publication that is currently live |249| `export_jobs` | non-public rows older than 30 days deleted |250251Not pruned: `audit_events`, `generation_jobs`, `generation_usage`, `sheet_revisions`, `sheet_views`, `ai_messages`. They grow slowly; prune by hand if needed, keeping in mind that budgets read `generation_usage` for the current day and month only.252253If the worker is down, none of this runs.254255## 7. Upgrading dependencies256257```bash258pnpm outdated259pnpm update <package>            # or edit package.json, then pnpm install260pnpm typecheck && pnpm lint && pnpm test:unit && pnpm test261```262263Points of attention, from how the code uses each dependency:264265| Dependency | What to check |266|---|---|267| `next`, `react` | `AGENTS.md` warns that this Next.js version differs from older ones; read the guides under `node_modules/next/dist/docs/`. Re-check the CSP in `next.config.ts` against what the new version emits inline. |268| `@simplewebauthn/server` (pinned to 14.1.0) and `/browser` | Upgrade both together; option and response shapes change between majors. `tests/api.test.ts` exercises the ceremonies with a software authenticator. |269| `@anthropic-ai/sdk` | `src/ai/client.ts` uses `beta.messages.stream`, `output_config.effort`, the `server-side-fallback-2026-07-01` beta and `stop_details`. `tests/fake-anthropic.ts` must keep speaking the streaming protocol the SDK expects. |270| `playwright-core` | Reinstall the browser after every upgrade (`playwright-core install chromium-headless-shell`); each version pins a browser revision. |271| `katex` | Affects rendering and the LaTeX syntax check of the quality report; published equations that used to pass may fail on the next save. |272| `zod` | The system prompt embeds `z.toJSONSchema(SheetSchema)`; a change in its output changes the prompt — bump `PROMPT_VERSION`. |273| `postgres` | The queue relies on `sql.listen`, transactions and `sql.json`. |274| `htmlparser2` | The SVG sanitiser; run `tests/unit/renderer.sanitize.test.ts`. |275| PostgreSQL | Major upgrades by dump and restore (the backup format is a `pg_dump` custom archive; use the matching `pg_restore`). |276277Model changes need no deployment: the settings page lists the models the key can call. Update `src/ai/pricing.ts` when prices change or a new model appears; an unknown model is costed at the highest rate in the table.278279## 8. Rollback280281See [DEPLOYMENT.md](DEPLOYMENT.md) §16. In short: code-only rollbacks are a checkout, install, build and restart; a rollback across a migration or a schema-version bump needs the database and storage backup taken before the upgrade.282283Content rollbacks need no operator: every save is a revision, and restoring one (studio → Revisions) writes a new revision. A bad publication is undone by restoring the previous revision and publishing again, or by unpublishing.284
Files main
Menu

Theme