Architecture
This document describes the code as it is. File references are relative to the project root.
1. Components
┌──────────────────────────────────────────────┐
Reader / administrator │ web process — server.ts │
(browser) │ Node HTTP server (timeouts, graceful stop) │
│ HTTPS │ └─ Next.js 16 App Router │
▼ │ (site)/… public pages │
reverse proxy ──── HTTP ───────▶ │ admin/… studio pages │
(TLS, X-Forwarded-For, │ api/public/* anonymous JSON API │
body limit, compression) │ api/auth/* WebAuthn, sessions │
│ api/admin/* studio API │
│ api/health liveness │
│ media/<id> public files │
│ preview/<token> private draft preview │
│ (+ /asset/<id> for its images) │
│ render/<token> bare page for exports │
└───────────────┬───────────────┬──────────────┘
│ │
┌─────────▼────┐ ┌──────▼────────┐
│ PostgreSQL 17│ │ STORAGE_ROOT │
│ (one schema) │ │ ab/cd/<uuid> │
└─────────▲────┘ └──────▲────────┘
│ │
┌───────────────┴───────────────┴──────────────┐
│ worker process — src/worker/index.ts │
│ AI jobs (2 at a time) → api.anthropic.com │
│ verify_sources → Crossref, doi.org, │
│ source URLs │
│ exports (1 at a time) → headless Chromium │
│ → web /render/… │
│ heartbeat (15 s), housekeeping (hourly) │
└──────────────────────────────────────────────┘
scripts/backup.sh (scheduled by the operator) → BACKUP_DIRThere is no Redis, no message broker and no object store. PostgreSQL holds documents, the job queues, sessions, rate-limit counters, the audit trail and the AI usage ledger. Files live in one directory tree (STORAGE_ROOT) that both processes must be able to read and write.
The web process is server.ts: a plain node:http server that hands every request to Next.js, sets requestTimeout to 5 minutes, headersTimeout to 60 s, keepAliveTimeout to 65 s, and closes on SIGTERM/SIGINT (forced exit after 8 s). It listens on HOST (default 0.0.0.0) and PORT (default 8700).
2. Module map
A single Node package. Boundaries are kept by directory and by import direction.
src/schema ◀── src/renderer ◀──┐
▲ src/sim ◀───────┤
│ │
└──────── src/server ◀── src/ai ◀── src/worker
▲ ▲
└── src/lib/api (route tables) ◀── src/app (routes)
src/components (UI)| Module | Responsibility | May import | Notes |
|---|---|---|---|
src/schema |
Sheet Document Schema v1 (Zod), parseSheet, migrateSheet, stableStringify, traversal, applyPatch, diffSheets, slugify |
zod only |
No React, no Next, no database. Runs in the browser too. |
src/renderer |
Block components, SheetView, diagram layout, chart drawing, inline parser, KaTeX wrapper, SVG sanitiser, theme palettes, standalone SVG export |
schema, sim |
export-svg.ts imports react-dom/server and is used by the worker and unit tests only. |
src/sim |
The 13 simulations: pure compute, a View, shared numeric core, the expression parser |
schema, renderer/chart |
registry.ts is server-safe (used by the quality report and the prompts). |
src/templates |
12 built-in templates as code | schema |
Each build(locale) result is passed through mustParseSheet. |
src/lib |
env, db (one postgres client, pool of 10), auth, session-store, ratelimit, audit, http, log, crypto, client fetch helper |
— | lib/api/*routes.ts are the three API route tables; they import server and ai. |
src/server |
All writes to sheets (sheets.ts), assets and scanning, storage driver, job queue API, export requests, quality report, settings, render tokens, dashboard/audit/preview helpers |
schema, renderer (sanitiser, LaTeX check), sim/registry, lib |
misc.ts also imports ai/client for the dashboard health tile. |
src/ai |
client.ts (the only caller of the Anthropic API), prompts.ts, pipeline.ts, net.ts, sources.ts, pricing.ts |
schema, server, lib, sim/registry |
Server and worker only; the SDK is in serverExternalPackages. |
src/worker |
Queue loop, job state transitions, export rendering | everything server-side | Not part of the Next.js build. |
src/components |
Site chrome, studio pages, editor | schema, renderer, sim, lib/client-api, i18n; types only from server and ai |
Client components must not import lib/env or lib/db. |
src/app |
Routes; thin wrappers over server functions and the route tables |
all |
The import direction is a convention; there is no lint rule that enforces it.
API surfaces
Each surface is one catch-all route whose guard runs before a handler is looked up (src/lib/api/router.ts), so a handler cannot be reached without its guard.
| Surface | Route file | Table | Guard |
|---|---|---|---|
/api/public/* |
src/app/api/public/[...path]/route.ts |
src/lib/api/publicroutes.ts |
none (rate limit per client address inside each handler) |
/api/auth/* |
src/app/api/auth/[...path]/route.ts |
src/lib/api/authroutes.ts |
per route: same-origin check, rate limit, session + CSRF where relevant |
/api/admin/* |
src/app/api/admin/[...path]/route.ts |
src/lib/api/adminroutes.ts |
requireAdmin: passkey session, CSRF header on non-GET |
3. Request lifecycles
(a) An anonymous reader opens /s/<slug>
GET /s/<slug>
└─ src/app/(site)/s/[slug]/page.tsx (force-dynamic)
├─ getLocale() cookie sheet_locale, else Accept-Language
├─ getPublished(slug, locale) src/server/sheets.ts
│ SELECT … FROM sheets s JOIN sheet_publications p ON p.id = s.publication_id
│ WHERE s.slug = $1 AND s.deleted_at IS NULL AND s.status = 'published'
│ ├─ no row → slug_redirects → permanentRedirect(/s/<new slug>) or 404
│ └─ row → mustParseSheet(p.doc) (migrated on read if the schema moved on)
├─ relatedPublished(…) shared topics first
└─ render: JSON-LD + SheetReader → SheetView(mode "public", assetUrl = /media/<id>; a private preview uses /preview/<token>/asset/<id>)
└─ ViewBeacon (client) → POST /api/public/views after 3 s when the tab is visibleThe page reads the publication snapshot, never sheets.doc. Images are fetched from /media/<id>, which serves only assets flagged is_public. Metadata (generateMetadata) reuses the same query through React.cache. Nothing is cached server-side; the JSON API under /api/public sends Cache-Control: public, max-age=….
(b) A save in the editor
Editor (client) — src/components/editor/Editor.tsx
change → useDocState.commit → autosave timer (2.5 s) or ⌘S
parseSheet(doc) in the browser ── invalid → "Invalid document — not saved", nothing sent
PATCH /api/admin/sheets/<id> { doc, expectedVersion } + X-CSRF-Token
requireAdmin → body read under a 2 MB cap (readBody) → Zod → validDoc(doc) (422 with issues if invalid)
saveDoc() — src/server/sheets.ts, one transaction
SELECT … FROM sheets WHERE id = $1 FOR UPDATE
version ≠ expectedVersion → 409 version_conflict
locale changed → 422 locale_locked
same doc_hash as current → { changed: false } (no revision)
INSERT sheet_revisions (number = max + 1, parent_id = current, origin 'manual')
UPDATE sheets SET doc, title, summary, current_revision_id, version = version + 1
syncTaxonomy: sheet_topics, sheet_tags, sources / sheet_sources
← { version, revisionId, revisionNumber, diff, quality }A 409 puts the editor in a "conflict" state that stops autosave and offers a reload. Between saves, the editor asks POST /api/admin/quality (debounced 900 ms) for the report of the unsaved document; it is the same function that gates publication.
(c) A Claude generation job
POST /api/admin/generations { kind: "generate", brief, outline, idempotencyKey }
requireAdmin → BriefSchema / OutlineSchema → ANTHROPIC_API_KEY present? → attachments exist?
enqueueJob() — src/server/jobs.ts
same idempotency_key → return the existing job
rate limit jobs:<user> (aiJobsPerHour)
INSERT generation_jobs (status 'queued') → pg_notify('sheet_jobs') → audit job.queued.generate
← job view
Browser: EventSource /api/admin/generations/<id>/events
the web process re-reads the job row every 500 ms and emits an event when it changes
Worker — src/worker/index.ts
LISTEN sheet_jobs (wake-up) + 2 s poll
claim(): UPDATE … WHERE id = (SELECT … WHERE status = 'queued' AND run_after <= now()
ORDER BY created_at FOR UPDATE SKIP LOCKED LIMIT 1)
runJob(): watch timer (cancel flag, time limit) → resolveModel → runGenerate()
extractMaterial attachments from storage, URLs through safeFetch
askJson callClaude (budget check, stream, ledger row) → extractJson
→ normalizeModelDoc → parseSheet; up to 2 repair rounds
qualityReport
createSheet sheets row + revision 1 (origin 'ai_generation', job_id) — draft
ai_conversations / ai_messages
UPDATE generation_jobs SET status = 'succeeded', output = …
Browser: on "succeeded" → /admin/sheets/<sheetId>/editSee AI_PIPELINE.md for each stage.
(d) An export
POST /api/admin/exports { sheetId, format: pdf|png|svg, scale, blockId?, revisionId? }
requireAdmin → rate limit export:<user> (120/h) → requestExport() — src/server/exports.ts
blockId must be a valid id; svg needs a blockId; pdf + blockId → 422 whole_sheet_only
INSERT export_jobs (revision_id = current revision unless given) → pg_notify('sheet_jobs')
Browser polls GET /api/admin/exports/<id> every 1.2 s
Worker — exportsTick() (one export at a time), src/worker/export.ts
svg: renderBlockSvg(block) react-dom/server → theme tokens resolved to colours, fonts embedded
png / pdf:
renderToken({ kind, id }, 600 s) HMAC-SHA256 with APP_SECRET
Chromium (playwright-core) → INTERNAL_URL/render/<token>
page.route: every request to another origin than INTERNAL_URL is aborted
/render/<token>/asset/<id> serves only images referenced by that document
a layout narrower than 280 px aborts the export (collapsed page)
png: element screenshot at deviceScaleFactor = scale (refused above 16 000 px a side)
pdf: fixed formats → one page of the format's size; article → A4 with margins and a footer
createExportAsset() → assets row (kind 'export') + file → export_jobs.asset_id
Browser: download /api/admin/assets/<assetId>/file?download=1On publication, when allow_download is on, publish() inserts a public PDF export job for the new publication and notifies the worker; readers get it through /media/<assetId>. JSON export is not a job: GET /api/admin/sheets/<id>/export.json returns stableStringify(doc, 2).
4. Publication model
Four things, three of them immutable.
| Thing | Table / column | Mutable? | Meaning |
|---|---|---|---|
| Working copy | sheets.doc |
yes | What the studio shows. Always equal to the document of sheets.current_revision_id. |
| Revision | sheet_revisions |
no | Every saved state, numbered per sheet, with origin (manual, ai_generation, ai_edit, revert, import, template, source_check), parent, optional patch and job id. Reverting writes a new revision. |
| Publication | sheet_publications |
no | A copy of one revision taken at publication time, with title, summary, reading time and search text. Publishing again adds a row. |
| Pointer | sheets.publication_id |
yes | The one publication the public sees. NULL when not published. |
publish() (src/server/sheets.ts) runs in one transaction: lock the sheet, check expectedVersion, run the quality report and refuse on errors, mark the previous publication unpublished_at, insert the new publication from the current revision, move the pointer and set status = 'published', flag the document's image assets is_public, queue the reader PDF, write the audit event. Readers see the old publication or the new one, never a mixture. A table constraint ties the two columns together: (status = 'published') = (publication_id IS NOT NULL).
Statuses are draft, in_review, published, archived. in_review and archived are labels set from the sheet panel; they carry no workflow. There is no scheduled publication.
unpublish() clears the pointer, sets the status back to draft, clears featured, and withdraws the public flag from the sheet's reader PDFs and from the images of that publication that no other published sheet uses. deleteSheet() is a soft delete (deleted_at), refused while published; the slug becomes free again because the unique index is partial.
Slug changes: updateSettings() records the old slug in slug_redirects if the sheet has ever been public, and public lookups answer 301 from the old address.
A slug change does not touch publication rows: sheet_publications.slug keeps the slug the sheet had when that publication was made; the live address is sheets.slug.
Languages: a French and an English sheet are two sheets rows sharing a localization_id. A sheet's locale cannot change (locale_locked); translation creates the sibling.
5. Data model
All tables are created by db/migrations/001_init.sql. Keys are UUIDs unless noted; timestamps are timestamptz. scripts/migrate.ts adds schema_migrations (name, checksum, applied_at) and refuses to run if an applied file was modified.
Administrator and authentication
| Table | Purpose | Notable columns |
|---|---|---|
users |
The administrator. Plural only to allow a second editor later; role is constrained to admin. |
email (citext, unique), display_name, locale, disabled_at |
passkey_credentials |
WebAuthn public material. | credential_id (unique), public_key, counter, transports, device_type, backed_up, aaguid, name, last_used_at, revoked_at |
setup_tokens |
One-time enrolment tokens issued from the shell. | token_hash (SHA-256), purpose (setup/recovery), expires_at, used_at |
recovery_codes |
Ten one-use codes. | code_hash, used_at |
webauthn_challenges |
Short-lived challenges, deleted when read. | challenge, kind (registration/authentication/reauthentication), binding, expires_at |
sessions |
Server-side sessions. | token_hash, csrf_token, method (passkey/setup/recovery_code), reauth_at, expires_at, last_seen_at (a session unused for 7 days is refused), revoked_at, ip, user_agent |
rate_limits |
Fixed-window counters. | primary key (key, window_start), count |
audit_events |
Append-only trail. | actor (admin/system/worker/anonymous), action, entity, entity_id, meta (jsonb), ip |
settings |
Non-secret settings as key → jsonb. | validated by SettingsSchema (src/server/settings.ts) |
worker_heartbeat |
One row, name = 'worker'. |
beat_at, info (worker id, running jobs, Chromium state, prompt version) |
Content
| Table | Purpose | Notable columns |
|---|---|---|
topics |
Categories with French and English names. | slug, name_fr, name_en, tone, position |
tags |
Free keywords. | slug, name |
sheet_localizations |
Groups the language versions of one sheet. | — |
sheets |
One sheet in one language. | slug (unique per locale among live rows), locale, localization_id, status, doc, current_revision_id, publication_id, version (optimistic lock), featured, featured_position, allow_download, seo_title, seo_description, canonical_url, og_asset_id, deleted_at |
sheet_revisions |
Immutable saved states. | number (unique per sheet), doc, doc_hash, origin, summary, patch, parent_id, job_id |
sheet_publications |
Immutable public snapshots. | revision_id, number, doc, slug, title, summary, reading_minutes, search_text, generated search_fr / search_en tsvectors (GIN), trigram index on the title, unpublished_at |
slug_redirects |
Old public addresses. | primary key (locale, old_slug) |
sheet_topics, sheet_tags |
Link tables, rebuilt from the document's metadata on every save. | |
preview_tokens |
Private preview links. | token_hash, expires_at, revoked_at |
templates |
The author's own templates (built-in ones live in code). | key, doc, locale, deleted_at |
sheet_views |
View counter. | primary key (sheet_id, day), views |
Media, data, sources
| Table | Purpose | Notable columns |
|---|---|---|
assets |
Uploads and generated exports. | kind (image/svg/pdf/text/csv/json/export), mime, bytes, sha256, storage_key, width, height, alt, credit, license, scan_status (clean/unscanned), is_public, deleted_at |
datasets |
Parsed shape of a CSV/JSON upload. | columns, row_count, sample (first 20 rows) |
sources |
Cross-sheet reference library keyed by fingerprint (doi:…, url:…, title:…). The copy inside each document stays authoritative for rendering. |
status, checked_at |
sheet_sources |
Which sheet cites which library source. | local_id |
Claude studio and exports
| Table | Purpose | Notable columns |
|---|---|---|
ai_conversations, ai_messages |
One conversation per sheet: the author's requests and the assistant's summaries. | job_id, revision_id, diff |
generation_jobs |
Queue for outline, generate, edit, translate, verify_sources. |
status, stage, progress, input, output, error, attempts, max_attempts, run_after, cancel_requested, idempotency_key (unique), model, prompt_version, locked_by, locked_at |
generation_usage |
One row per completed model call: the ledger budgets are checked against. | model, stage, token counts, cost_usd, duration_ms, stop_reason, request_id |
export_jobs |
Queue for exports. | revision_id or publication_id, format, options (scale, blockId), status, asset_id, is_public, attempts |
6. Design system
Two independent token sets.
Site and studio chrome — src/app/globals.css. Tailwind 4 with @theme mapping colour utilities (bg-paper, text-muted, border-line, text-accent, text-bad…) to --ui-* custom properties. Light values on :root, dark values under prefers-color-scheme: dark unless the visitor chose a mode (html[data-mode], stored in localStorage as sheet-mode and applied before first paint by an inline script in src/app/layout.tsx). Shared utilities: btn, btn-primary, btn-danger, btn-ghost, btn-sm, field, card, eyebrow, safe-b, safe-x. Controls have a 44 px minimum height. Fonts are self-hosted through @fontsource-variable: Inter (text), Fraunces (display), JetBrains Mono.
Sheets — src/renderer/sheet.css. A sheet reads only --sh-* tokens set by .sheet[data-theme] and .sheet[data-accent], so a dark sheet stays dark on a light page.
| Token | Role |
|---|---|
--sh-bg, --sh-surface, --sh-raised |
backgrounds |
--sh-ink, --sh-muted, --sh-faint |
text |
--sh-line, --sh-grid |
rules, chart grid |
--sh-t-<tone>, --sh-t-<tone>-soft |
strong and soft colour of each of the 8 tones |
--sh-accent, --sh-accent-soft |
resolved from theme.accent |
--sh-contra |
opposite side of diverging scales (red, or blue when the accent is red, pink or amber) |
Theme presets: scientific-light, scientific-dark, editorial, blueprint, finance. Documents never carry colours, only a preset, an accent tone and per-element tones. src/renderer/themes.ts is a TypeScript copy of the palettes for outputs that cannot read CSS variables (standalone SVG); tests/unit/renderer.themes.test.ts compares the two.
Layout: .sh-frame is a container (container: sheet / inline-size), so the 12-column grid responds to the width of the sheet itself — a narrow preview in the editor gets the phone layout. Fixed formats (a4, slide-16-9, square, portrait-4-5) keep their pixel size and scroll horizontally on small screens. Print rules avoid breaking inside blocks and hide interactive controls.
7. Technical trade-offs
| Decision | Reason | Cost |
|---|---|---|
| One package with modular folders, not a monorepo | One deployable; a workspace would add tooling for no isolation gain | Module boundaries rely on discipline, not on package boundaries |
| SQL migrations with checksums, no ORM | The queries that matter (queue claim, full-text ranking, generated tsvector columns) are plain SQL; postgres tagged templates keep them parameterised |
Row types are written by hand next to each query |
| No Redis: queue, rate limits and sessions in PostgreSQL | One fewer service; jobs and sessions survive restarts; FOR UPDATE SKIP LOCKED gives exactly-one claim |
Rate limits cost a write per request; the progress stream polls the job row (every 500 ms per open stream); LISTEN/NOTIFY is used only to wake the worker |
| A separate worker process | Model calls last minutes and exports need a browser; neither belongs in a request handler | Two processes to supervise; they must share STORAGE_ROOT |
| Documents are data, not HTML | The model, the editor and imports can only produce what the schema allows; the renderer never interprets markup | New visual ideas need a schema and renderer change |
| Exports through headless Chromium on the real page | PDF and PNG use the same fonts, KaTeX output and CSS as the site | A browser binary must be installed next to the worker |
| A custom Node server in front of Next.js | Explicit timeouts and graceful stop; .env loaded the same way for scripts and worker |
tsx is a runtime dependency |
Static CSP with 'unsafe-inline' |
Next's inline bootstrap scripts would otherwise need a per-request nonce on every page (comment in next.config.ts) |
Script injection is not blocked by CSP; see SECURITY.md |
Gateway compresses, app does not (compress: false) |
Avoids double work behind a proxy | A proxy without compression serves uncompressed responses |
| Built-in templates live in code | Versioned and tested against the schema | Changing one needs a deployment |
Only the local storage driver |
A directory or NAS mount is enough for one node | STORAGE_DRIVER refuses any other value; no S3 |
| Undo history as whole documents (100 steps) | Documents are small JSON values | Memory grows with very large sheets |
| Published documents are migrated when read, never rewritten | Publications stay immutable across schema versions | Every read pays the migration once SCHEMA_VERSION moves |