Operations
Running Sheet day to day: processes, logs, health, the job queue, incidents, housekeeping, upgrades.
1. Processes
| Process | Entry point | State it holds | Restart cost |
|---|---|---|---|
| Web | server.ts |
none (sessions, limits and jobs are in PostgreSQL); a 10-minute in-memory cache of the model list | open event streams drop; browsers fall back to polling |
| Worker | src/worker/index.ts |
the running jobs and one headless browser (closed after 2 minutes idle) | running jobs are re-queued |
| Backup | scripts/backup.sh |
— | scheduled, runs and exits |
Run one worker. At start-up a worker puts every job marked running back in the queue, which would steal the jobs of another live worker (the single-worker assumption is stated in a comment in src/worker/index.ts). Inside the single worker, two AI jobs and one export run at a time.
Worker start-up sequence: check that STORAGE_ROOT is writable (fatal if not) → start Chromium once to prove it works (warning if it does not) → fail jobs that used all their attempts (interrupted) → re-queue jobs left running → write the heartbeat → run housekeeping → LISTEN sheet_jobs → loop (wake on notification or every 2 s).
Signals: both processes handle SIGTERM and SIGINT. Web: stops accepting connections, exits within 8 s. Worker: aborts model calls, re-queues its jobs without counting the attempt, waits up to 20 s, closes the browser and the database pool.
2. Logs
Both processes write to stdout/stderr; where the lines end up is decided by the process manager (PM2 log files, journald, docker logs). LOG_LEVEL (debug, info, warn, error; default info) applies to lines written through src/lib/log.ts.
Worker: JSON lines
One JSON object per line: t (ISO time), level, msg, then fields. warn and error go to stderr. Anything that looks like an Anthropic key (sk-ant-…) is truncated before the line is written. Document bodies and request bodies are never logged.
msg |
Level | Fields |
|---|---|---|
worker.started |
info | worker (host:pid), requeued, chromium, promptVersion |
worker.chromium_missing |
warn | detail |
worker.stopping |
info | running |
worker.loop, worker.fatal, exports.tick |
error | error, detail |
job.started |
info | jobId, kind, attempt |
job.succeeded |
info | jobId, kind, ms |
job.retry |
warn | jobId, kind, code, inSeconds |
job.cancelled |
info | jobId, kind |
job.failed |
error | jobId, kind, code, message; for non-AI errors also error, detail |
ai.call |
info | stage, jobId, model, inputTokens, outputTokens, cacheRead, costUsd, ms, stop |
export.succeeded |
info | exportId, format, ms |
export.failed |
error | exportId, format, attempt, detail |
export.fonts_missing |
warn | detail |
housekeeping.failed |
warn | error, detail |
Web: little
The web process writes three kinds of JSON lines of its own — web.listening (port, host, mode), web.unhandled, and api.unhandled (error, detail) for an unexpected error inside an API handler (src/lib/http.ts; the client receives 500 {"error":{"code":"internal"}}) — plus whatever Next.js prints.
There is no access log and no per-request line; use the reverse proxy's access log. Sign-ins, failures, publications, uploads and job events are not in the logs: they are in the audit_events table (studio → Revisions → audit, or SQL).
SELECT at, actor, action, entity, entity_id, meta, ip FROM audit_events ORDER BY at DESC LIMIT 50;
SELECT at, meta->>'why' AS why, ip FROM audit_events WHERE action = 'login.failed' AND at > now() - interval '1 day';No error-reporting service (Sentry or similar) is integrated.
3. Health
| Check | How | Meaning |
|---|---|---|
| Liveness | GET /api/health |
200 {"ok":true}: the web process answers and SELECT 1 works. 503 {"ok":false} otherwise. No detail is exposed. |
| Dashboard tiles | /admin (GET /api/admin/dashboard) |
Database (version and size), Claude API (key present, model list reachable, model available — spends no tokens), job queue and worker, storage (write probe in STORAGE_ROOT), antivirus (clamd probe, or "not configured") |
| Worker heartbeat | worker_heartbeat row, updated every 15 s |
The dashboard calls the worker alive when the last beat is under 45 s old; otherwise "stopped" or "never started" |
| Chromium | in the heartbeat info (chromium, chromiumDetail) |
evaluated once, when the worker starts |
| Spend | dashboard, from generation_usage |
today and this month against the budgets |
SELECT name, now() - beat_at AS age, info FROM worker_heartbeat;The web process does not monitor the worker beyond reading that row; alert on it yourself if exports and AI jobs matter (for example: page when age > interval '2 minutes').
4. The job queue
Two tables, same mechanics: generation_jobs (AI jobs and source checks) and export_jobs.
States
generation_jobs: queued ──claim──▶ running ──▶ succeeded
▲ │──────▶ failed
│ │──────▶ cancelled
└── retry / worker restart / shutdown
queued ──cancel──▶ cancelled
export_jobs: queued ──▶ running ──▶ succeeded | failed (2 attempts, 20 s apart)A job is claimed with UPDATE … WHERE id = (SELECT … FOR UPDATE SKIP LOCKED LIMIT 1), oldest first, when run_after <= now() and attempts < max_attempts. attempts is incremented at each claim. stage and progress describe what a running job is doing; locked_by is the worker (host:pid).
Retries: retryable AI errors re-queue with a delay of 30 s × 2^(attempt − 1) while attempts < max_attempts (3; 2 for source checks). Exports are retried once after 20 s.
Inspect
-- what is waiting or running
SELECT id, kind, status, stage, attempts, max_attempts, run_after, locked_by, created_at, started_at
FROM generation_jobs WHERE status IN ('queued', 'running') ORDER BY created_at;
-- recent failures and why
SELECT id, kind, error->>'code' AS code, error->>'message' AS message, attempts, finished_at
FROM generation_jobs WHERE status = 'failed' ORDER BY finished_at DESC LIMIT 20;
-- what a job cost, call by call
SELECT at, stage, model, input_tokens, output_tokens, cache_read_tokens, cost_usd, stop_reason, request_id
FROM generation_usage WHERE job_id = '<uuid>' ORDER BY at;
-- spend per day
SELECT date_trunc('day', at) AS day, sum(cost_usd) AS usd, count(*) AS calls FROM generation_usage GROUP BY 1 ORDER BY 1 DESC LIMIT 14;
-- exports
SELECT id, format, status, attempts, error, created_at, finished_at FROM export_jobs ORDER BY created_at DESC LIMIT 20;In the studio: GET /api/admin/generations lists recent jobs; the job card shows stage, cost and error.
Cancel
From the studio: the Cancel button on a job card (POST /api/admin/generations/<id>/cancel). In SQL:
-- queued: cancelled at once
UPDATE generation_jobs SET status = 'cancelled', stage = 'cancelled', finished_at = now() WHERE id = '<uuid>' AND status = 'queued';
-- running: the worker sees the flag within a second and aborts the model call
UPDATE generation_jobs SET cancel_requested = true WHERE id = '<uuid>' AND status = 'running';Exports have no cancel; delete the queued row, or let it finish.
Re-queue
-- run a failed job again (same input, attempts reset)
UPDATE generation_jobs
SET status = 'queued', stage = 'queued', attempts = 0, run_after = now(), error = NULL, finished_at = NULL, cancel_requested = false, locked_by = NULL
WHERE id = '<uuid>' AND status = 'failed';
SELECT pg_notify('sheet_jobs', '<uuid>'); -- optional: the worker also polls every 2 s
-- a failed export
UPDATE export_jobs SET status = 'queued', attempts = 0, run_after = now(), error = NULL, finished_at = NULL, locked_by = NULL WHERE id = '<uuid>';Before re-queuing, know what a re-run does:
generateandtranslatefirst look for a revision already carrying the job's id and return it if found: a re-run never creates a second sheet.editreturns the revision it already wrote, if any. Otherwise it checks that the sheet is still at the version the request was based on; if not, it fails again with "The sheet changed after this request was sent" — re-send the request from the editor instead.outlineandverify_sourcescan be re-run freely.- A re-run spends tokens again.
A job that is stuck in running
With the worker alive, a job cannot outlive aiJobTimeoutMinutes (default 20): the watch timer aborts it. If the worker died, the job stays running until a worker starts, which re-queues it. attempts grows by one per claim; a job that has used all its attempts is not claimed again, and at the next worker start it is failed with code interrupted ("The job was interrupted too many times and was stopped"). A queued job with attempts >= max_attempts stays queued until that start-up pass; re-queue it with attempts = 0 if it should run.
5. Common incidents
The Claude API key is refused
Signs: jobs fail with code auth ("The Anthropic API key was refused"); the dashboard's Claude tile is failing.
Do: check ANTHROPIC_API_KEY in the environment of both processes (the worker makes the calls; the web process lists models), restart them after changing it. If the tile says the model is not available, pick another model in Settings; a chosen model that disappeared falls back to ANTHROPIC_MODEL automatically when that one exists.
No key at all
The studio works as a manual editor. AI routes answer 503 ai_not_configured; the assistant and the Claude panel say so. Source checks still work.
Budget reached
Signs: jobs fail with code budget and a message naming the limit (daily, monthly or per job); the dashboard shows "Budget reached".
Do: wait for the next day or month, or raise the budget in Settings → Claude (0 = no limit). The check runs before each model call, so a job can stop between its first call and a repair call. To see what consumed it, use the spend queries above.
Exports fail because Chromium is missing
Signs: worker.chromium_missing at start-up ("Chromium could not start …"); export jobs fail with a message about the executable; published sheets have no PDF button.
Do: as the user that runs the worker, node_modules/.bin/playwright-core install chromium-headless-shell (add --with-deps on Debian/Ubuntu, as root, for system libraries), or set CHROMIUM_PATH. Restart the worker: the check — a real launch of the browser — runs only at start-up.
A PDF request for a single block is refused up front (422 whole_sheet_only); export one block as PNG or SVG. Other export failures: Render page answered HTTP … means the worker's browser could not load INTERNAL_URL/render/<token> — check that INTERNAL_URL reaches the web process from the worker and that APP_SECRET is the same for both. "The image would be … px" means the PNG is too large; use a lower scale or PDF. "The sheet did not lay out at a readable width; export aborted" means the render page collapsed in the headless browser; the worker refuses to produce an unreadable file — check the render page by hand before retrying.
To produce the missing reader PDF of a published sheet, publish it again (a new publication queues a new PDF), or insert the job:
INSERT INTO export_jobs (sheet_id, publication_id, format, options, is_public)
SELECT id, publication_id, 'pdf', '{}', true FROM sheets WHERE slug = '<slug>' AND locale = '<fr|en>' AND status = 'published';The worker is down
Signs: dashboard "Worker stopped" or "never started"; jobs stay queued; exports stay pending.
Do: read the worker's last log lines (worker.fatal gives the reason — most often STORAGE_ROOT is not usable or a database connection failure), fix, restart. Nothing is lost: queued jobs wait, running ones are re-queued at start-up.
Disk full
Signs: uploads and exports fail; the storage tile fails its write probe; PostgreSQL errors if the database volume is the one that filled; backups fail.
Where space goes: STORAGE_ROOT (uploads are capped by STORAGE_QUOTA_GB; exports are outside the quota and are pruned after seven days), BACKUP_DIR (a full copy of storage per day, BACKUP_KEEP_DAYS of them), process logs, PostgreSQL.
Do: lower BACKUP_KEEP_DAYS or move BACKUP_DIR; rotate logs; delete unused files in Media & data (a file used by a sheet cannot be deleted). Studio exports older than seven days are removed by the worker's hourly housekeeping. Do not delete files under STORAGE_ROOT by hand: the assets rows would point at nothing.
SELECT kind, count(*), pg_size_pretty(sum(bytes)) FROM assets WHERE deleted_at IS NULL GROUP BY kind ORDER BY sum(bytes) DESC;Version conflicts
Signs: the editor shows "The sheet was changed elsewhere … Reload"; an AI edit fails with "The sheet changed after this request was sent"; API answers 409 version_conflict.
Cause: two writers on one sheet — two tabs, or a job that finished after a manual save. This is the optimistic lock doing its work; nothing was overwritten.
Do: reload the tab that lost. Its unsaved changes are gone from the server's point of view; copy anything important out of the tab before reloading. Then re-send the AI request if needed.
"Invalid document — not saved"
The working copy in the browser does not pass the schema; autosave is suspended for the whole sheet until it does. The banner lists the first issues. Fix the issue named in the banner, or undo.
Lost passkeys
- Another enrolled device still works: sign in there and remove the lost passkey in Settings.
- A recovery code: sign-in page → "Lost your device? Use a recovery code". The 30-minute session can only enrol a new passkey; doing so revokes all other passkeys and sessions and issues new codes.
- Nothing left: on the server, interactively,
node_modules/.bin/tsx scripts/setup-token.ts --recovery, open the printed link within 30 minutes on the new device. Without--recoverythe new passkey is added and the old ones are kept.
Each path is in the audit trail (login.recovery_code, recovery.cli, passkey.enrolled.*).
Locked out by failed attempts
429 locked_out with retryAfterSeconds. The lock-out is per client address and computed from login.failed events of the last hour (5 → 1 min, 10 → 5 min, 20 → 1 h). It clears by itself. If every client appears under the same address (local, or a proxy's address), the proxy is not sending X-Forwarded-For, or a second proxy sits between the client and the one that appends the last value.
Passkeys stop working after a domain change
Passkeys are bound to the RP ID (WEBAUTHN_RP_ID, default the host of APP_URL). Changing the host invalidates all of them. Either set WEBAUTHN_RP_ID to the old value where the new host is a subdomain of it, or re-enrol with scripts/setup-token.ts --recovery.
A published sheet has broken images
Images are served from /media/<id> only when flagged public, which happens at publication for the images listed in the published document; unpublishing a sheet withdraws the flag from images no other published sheet uses. Publishing again re-flags them. Private previews serve their images through /preview/<token>/asset/<id> and do not depend on the flag.
6. Housekeeping done by the worker
Once at start-up and then every hour:
| Table / files | Rule |
|---|---|
webauthn_challenges |
deleted 1 hour after expiry |
rate_limits |
windows older than 2 days deleted |
sessions |
deleted 30 days after expiry or revocation (a session is refused after 14 days, or 7 days without use, well before its row is deleted) |
setup_tokens |
deleted 30 days after expiry |
preview_tokens |
deleted 7 days after expiry |
| export files | assets of kind export older than 7 days are deleted (file and row marked), except the reader PDF of a publication that is currently live |
export_jobs |
non-public rows older than 30 days deleted |
Not pruned: audit_events, generation_jobs, generation_usage, sheet_revisions, sheet_views, ai_messages. They grow slowly; prune by hand if needed, keeping in mind that budgets read generation_usage for the current day and month only.
If the worker is down, none of this runs.
7. Upgrading dependencies
pnpm outdated
pnpm update <package> # or edit package.json, then pnpm install
pnpm typecheck && pnpm lint && pnpm test:unit && pnpm testPoints of attention, from how the code uses each dependency:
| Dependency | What to check |
|---|---|
next, react |
AGENTS.md warns that this Next.js version differs from older ones; read the guides under node_modules/next/dist/docs/. Re-check the CSP in next.config.ts against what the new version emits inline. |
@simplewebauthn/server (pinned to 14.1.0) and /browser |
Upgrade both together; option and response shapes change between majors. tests/api.test.ts exercises the ceremonies with a software authenticator. |
@anthropic-ai/sdk |
src/ai/client.ts uses beta.messages.stream, output_config.effort, the server-side-fallback-2026-07-01 beta and stop_details. tests/fake-anthropic.ts must keep speaking the streaming protocol the SDK expects. |
playwright-core |
Reinstall the browser after every upgrade (playwright-core install chromium-headless-shell); each version pins a browser revision. |
katex |
Affects rendering and the LaTeX syntax check of the quality report; published equations that used to pass may fail on the next save. |
zod |
The system prompt embeds z.toJSONSchema(SheetSchema); a change in its output changes the prompt — bump PROMPT_VERSION. |
postgres |
The queue relies on sql.listen, transactions and sql.json. |
htmlparser2 |
The SVG sanitiser; run tests/unit/renderer.sanitize.test.ts. |
| PostgreSQL | Major upgrades by dump and restore (the backup format is a pg_dump custom archive; use the matching pg_restore). |
Model changes need no deployment: the settings page lists the models the key can call. Update src/ai/pricing.ts when prices change or a new model appears; an unknown model is costed at the highest rate in the table.
8. Rollback
See DEPLOYMENT.md §16. In short: code-only rollbacks are a checkout, install, build and restart; a rollback across a migration or a schema-version bump needs the database and storage backup taken before the upgrade.
Content rollbacks need no operator: every save is a revision, and restoring one (studio → Revisions) writes a new revision. A bad publication is undone by restoring the previous revision and publishing again, or by unpublishing.