SPB Git

Repository Public

ARCHITECTURE.md 24.5 KB 296 lines markdown

1# Architecture23This document describes the code as it is. File references are relative to the project root.45## 1. Components67```8                                   ┌──────────────────────────────────────────────┐9  Reader / administrator           │ web process — server.ts                      │10  (browser)                        │  Node HTTP server (timeouts, graceful stop)  │11      │  HTTPS                     │  └─ Next.js 16 App Router                    │12      ▼                            │      (site)/…         public pages           │13  reverse proxy ──── HTTP ───────▶ │      admin/…          studio pages           │14  (TLS, X-Forwarded-For,           │      api/public/*     anonymous JSON API     │15   body limit, compression)        │      api/auth/*       WebAuthn, sessions     │16                                   │      api/admin/*      studio API             │17                                   │      api/health       liveness               │18                                   │      media/<id>       public files           │19                                   │      preview/<token>  private draft preview  │20                                   │        (+ /asset/<id> for its images)        │21                                   │      render/<token>   bare page for exports   │22                                   └───────────────┬───────────────┬──────────────┘23                                                   │               │24                                         ┌─────────▼────┐   ┌──────▼────────┐25                                         │ PostgreSQL 17│   │ STORAGE_ROOT  │26                                         │ (one schema) │   │ ab/cd/<uuid>  │27                                         └─────────▲────┘   └──────▲────────┘28                                                   │               │29                                   ┌───────────────┴───────────────┴──────────────┐30                                   │ worker process — src/worker/index.ts         │31                                   │  AI jobs (2 at a time)  → api.anthropic.com  │32                                   │  verify_sources         → Crossref, doi.org, │33                                   │                           source URLs        │34                                   │  exports (1 at a time)  → headless Chromium  │35                                   │                           → web /render/…    │36                                   │  heartbeat (15 s), housekeeping (hourly)     │37                                   └──────────────────────────────────────────────┘3839  scripts/backup.sh  (scheduled by the operator)  → BACKUP_DIR40```4142There is no Redis, no message broker and no object store. PostgreSQL holds documents, the job queues, sessions, rate-limit counters, the audit trail and the AI usage ledger. Files live in one directory tree (`STORAGE_ROOT`) that both processes must be able to read and write.4344The web process is `server.ts`: a plain `node:http` server that hands every request to Next.js, sets `requestTimeout` to 5 minutes, `headersTimeout` to 60 s, `keepAliveTimeout` to 65 s, and closes on `SIGTERM`/`SIGINT` (forced exit after 8 s). It listens on `HOST` (default `0.0.0.0`) and `PORT` (default `8700`).4546## 2. Module map4748A single Node package. Boundaries are kept by directory and by import direction.4950```51  src/schema  ◀──  src/renderer  ◀──┐52       ▲           src/sim  ◀───────┤53       │                            │54       └────────  src/server ◀── src/ai ◀── src/worker55                       ▲            ▲56                       └── src/lib/api (route tables) ◀── src/app (routes)57                                                           src/components (UI)58```5960| Module | Responsibility | May import | Notes |61|---|---|---|---|62| `src/schema` | Sheet Document Schema v1 (Zod), `parseSheet`, `migrateSheet`, `stableStringify`, traversal, `applyPatch`, `diffSheets`, `slugify` | `zod` only | No React, no Next, no database. Runs in the browser too. |63| `src/renderer` | Block components, `SheetView`, diagram layout, chart drawing, inline parser, KaTeX wrapper, SVG sanitiser, theme palettes, standalone SVG export | `schema`, `sim` | `export-svg.ts` imports `react-dom/server` and is used by the worker and unit tests only. |64| `src/sim` | The 13 simulations: pure `compute`, a `View`, shared numeric core, the expression parser | `schema`, `renderer/chart` | `registry.ts` is server-safe (used by the quality report and the prompts). |65| `src/templates` | 12 built-in templates as code | `schema` | Each `build(locale)` result is passed through `mustParseSheet`. |66| `src/lib` | `env`, `db` (one `postgres` client, pool of 10), `auth`, `session-store`, `ratelimit`, `audit`, `http`, `log`, `crypto`, client fetch helper | — | `lib/api/*routes.ts` are the three API route tables; they import `server` and `ai`. |67| `src/server` | All writes to sheets (`sheets.ts`), assets and scanning, storage driver, job queue API, export requests, quality report, settings, render tokens, dashboard/audit/preview helpers | `schema`, `renderer` (sanitiser, LaTeX check), `sim/registry`, `lib` | `misc.ts` also imports `ai/client` for the dashboard health tile. |68| `src/ai` | `client.ts` (the only caller of the Anthropic API), `prompts.ts`, `pipeline.ts`, `net.ts`, `sources.ts`, `pricing.ts` | `schema`, `server`, `lib`, `sim/registry` | Server and worker only; the SDK is in `serverExternalPackages`. |69| `src/worker` | Queue loop, job state transitions, export rendering | everything server-side | Not part of the Next.js build. |70| `src/components` | Site chrome, studio pages, editor | `schema`, `renderer`, `sim`, `lib/client-api`, `i18n`; **types only** from `server` and `ai` | Client components must not import `lib/env` or `lib/db`. |71| `src/app` | Routes; thin wrappers over `server` functions and the route tables | all | |7273The import direction is a convention; there is no lint rule that enforces it.7475### API surfaces7677Each surface is one catch-all route whose guard runs before a handler is looked up (`src/lib/api/router.ts`), so a handler cannot be reached without its guard.7879| Surface | Route file | Table | Guard |80|---|---|---|---|81| `/api/public/*` | `src/app/api/public/[...path]/route.ts` | `src/lib/api/publicroutes.ts` | none (rate limit per client address inside each handler) |82| `/api/auth/*` | `src/app/api/auth/[...path]/route.ts` | `src/lib/api/authroutes.ts` | per route: same-origin check, rate limit, session + CSRF where relevant |83| `/api/admin/*` | `src/app/api/admin/[...path]/route.ts` | `src/lib/api/adminroutes.ts` | `requireAdmin`: passkey session, CSRF header on non-GET |8485## 3. Request lifecycles8687### (a) An anonymous reader opens `/s/<slug>`8889```90GET /s/<slug>91 └─ src/app/(site)/s/[slug]/page.tsx  (force-dynamic)92     ├─ getLocale()                       cookie sheet_locale, else Accept-Language93     ├─ getPublished(slug, locale)        src/server/sheets.ts94     │    SELECT … FROM sheets s JOIN sheet_publications p ON p.id = s.publication_id95     │    WHERE s.slug = $1 AND s.deleted_at IS NULL AND s.status = 'published'96     │    ├─ no row → slug_redirects → permanentRedirect(/s/<new slug>) or 40497     │    └─ row → mustParseSheet(p.doc)  (migrated on read if the schema moved on)98     ├─ relatedPublished(…)               shared topics first99     └─ render: JSON-LD + SheetReader → SheetView(mode "public", assetUrl = /media/<id>; a private preview uses /preview/<token>/asset/<id>)100         └─ ViewBeacon (client) → POST /api/public/views after 3 s when the tab is visible101```102103The page reads the publication snapshot, never `sheets.doc`. Images are fetched from `/media/<id>`, which serves only assets flagged `is_public`. Metadata (`generateMetadata`) reuses the same query through `React.cache`. Nothing is cached server-side; the JSON API under `/api/public` sends `Cache-Control: public, max-age=…`.104105### (b) A save in the editor106107```108Editor (client) — src/components/editor/Editor.tsx109 change → useDocState.commit → autosave timer (2.5 s) or ⌘S110   parseSheet(doc) in the browser  ── invalid → "Invalid document — not saved", nothing sent111   PATCH /api/admin/sheets/<id>  { doc, expectedVersion }   + X-CSRF-Token112     requireAdmin → body read under a 2 MB cap (readBody) → Zod → validDoc(doc)  (422 with issues if invalid)113     saveDoc()  — src/server/sheets.ts, one transaction114       SELECT … FROM sheets WHERE id = $1 FOR UPDATE115       version ≠ expectedVersion → 409 version_conflict116       locale changed            → 422 locale_locked117       same doc_hash as current  → { changed: false }  (no revision)118       INSERT sheet_revisions (number = max + 1, parent_id = current, origin 'manual')119       UPDATE sheets SET doc, title, summary, current_revision_id, version = version + 1120       syncTaxonomy: sheet_topics, sheet_tags, sources / sheet_sources121     ← { version, revisionId, revisionNumber, diff, quality }122```123124A 409 puts the editor in a "conflict" state that stops autosave and offers a reload. Between saves, the editor asks `POST /api/admin/quality` (debounced 900 ms) for the report of the unsaved document; it is the same function that gates publication.125126### (c) A Claude generation job127128```129POST /api/admin/generations { kind: "generate", brief, outline, idempotencyKey }130  requireAdmin → BriefSchema / OutlineSchema → ANTHROPIC_API_KEY present? → attachments exist?131  enqueueJob()  — src/server/jobs.ts132    same idempotency_key → return the existing job133    rate limit jobs:<user> (aiJobsPerHour)134    INSERT generation_jobs (status 'queued')  → pg_notify('sheet_jobs') → audit job.queued.generate135  ← job view136137Browser: EventSource /api/admin/generations/<id>/events138  the web process re-reads the job row every 500 ms and emits an event when it changes139140Worker — src/worker/index.ts141  LISTEN sheet_jobs (wake-up) + 2 s poll142  claim(): UPDATE … WHERE id = (SELECT … WHERE status = 'queued' AND run_after <= now()143                                 ORDER BY created_at FOR UPDATE SKIP LOCKED LIMIT 1)144  runJob(): watch timer (cancel flag, time limit) → resolveModel → runGenerate()145    extractMaterial   attachments from storage, URLs through safeFetch146    askJson           callClaude (budget check, stream, ledger row) → extractJson147                      → normalizeModelDoc → parseSheet; up to 2 repair rounds148    qualityReport149    createSheet       sheets row + revision 1 (origin 'ai_generation', job_id) — draft150    ai_conversations / ai_messages151  UPDATE generation_jobs SET status = 'succeeded', output = …152Browser: on "succeeded" → /admin/sheets/<sheetId>/edit153```154155See [AI_PIPELINE.md](AI_PIPELINE.md) for each stage.156157### (d) An export158159```160POST /api/admin/exports { sheetId, format: pdf|png|svg, scale, blockId?, revisionId? }161  requireAdmin → rate limit export:<user> (120/h) → requestExport()  — src/server/exports.ts162    blockId must be a valid id; svg needs a blockId; pdf + blockId → 422 whole_sheet_only163    INSERT export_jobs (revision_id = current revision unless given) → pg_notify('sheet_jobs')164Browser polls GET /api/admin/exports/<id> every 1.2 s165166Worker — exportsTick() (one export at a time), src/worker/export.ts167  svg: renderBlockSvg(block)  react-dom/server → theme tokens resolved to colours, fonts embedded168  png / pdf:169    renderToken({ kind, id }, 600 s)            HMAC-SHA256 with APP_SECRET170    Chromium (playwright-core) → INTERNAL_URL/render/<token>171       page.route: every request to another origin than INTERNAL_URL is aborted172       /render/<token>/asset/<id> serves only images referenced by that document173    a layout narrower than 280 px aborts the export (collapsed page)174    png: element screenshot at deviceScaleFactor = scale (refused above 16 000 px a side)175    pdf: fixed formats → one page of the format's size; article → A4 with margins and a footer176  createExportAsset() → assets row (kind 'export') + file → export_jobs.asset_id177Browser: download /api/admin/assets/<assetId>/file?download=1178```179180On publication, when `allow_download` is on, `publish()` inserts a public PDF export job for the new publication and notifies the worker; readers get it through `/media/<assetId>`. JSON export is not a job: `GET /api/admin/sheets/<id>/export.json` returns `stableStringify(doc, 2)`.181182## 4. Publication model183184Four things, three of them immutable.185186| Thing | Table / column | Mutable? | Meaning |187|---|---|---|---|188| Working copy | `sheets.doc` | yes | What the studio shows. Always equal to the document of `sheets.current_revision_id`. |189| Revision | `sheet_revisions` | no | Every saved state, numbered per sheet, with origin (`manual`, `ai_generation`, `ai_edit`, `revert`, `import`, `template`, `source_check`), parent, optional patch and job id. Reverting writes a new revision. |190| Publication | `sheet_publications` | no | A copy of one revision taken at publication time, with title, summary, reading time and search text. Publishing again adds a row. |191| Pointer | `sheets.publication_id` | yes | The one publication the public sees. `NULL` when not published. |192193`publish()` (`src/server/sheets.ts`) runs in one transaction: lock the sheet, check `expectedVersion`, run the quality report and refuse on errors, mark the previous publication `unpublished_at`, insert the new publication from the current revision, move the pointer and set `status = 'published'`, flag the document's image assets `is_public`, queue the reader PDF, write the audit event. Readers see the old publication or the new one, never a mixture. A table constraint ties the two columns together: `(status = 'published') = (publication_id IS NOT NULL)`.194195Statuses are `draft`, `in_review`, `published`, `archived`. `in_review` and `archived` are labels set from the sheet panel; they carry no workflow. There is no scheduled publication.196197`unpublish()` clears the pointer, sets the status back to `draft`, clears `featured`, and withdraws the public flag from the sheet's reader PDFs and from the images of that publication that no other published sheet uses. `deleteSheet()` is a soft delete (`deleted_at`), refused while published; the slug becomes free again because the unique index is partial.198199Slug changes: `updateSettings()` records the old slug in `slug_redirects` if the sheet has ever been public, and public lookups answer 301 from the old address.200201A slug change does not touch publication rows: `sheet_publications.slug` keeps the slug the sheet had when that publication was made; the live address is `sheets.slug`.202203Languages: a French and an English sheet are two `sheets` rows sharing a `localization_id`. A sheet's locale cannot change (`locale_locked`); translation creates the sibling.204205## 5. Data model206207All tables are created by `db/migrations/001_init.sql`. Keys are UUIDs unless noted; timestamps are `timestamptz`. `scripts/migrate.ts` adds `schema_migrations (name, checksum, applied_at)` and refuses to run if an applied file was modified.208209### Administrator and authentication210211| Table | Purpose | Notable columns |212|---|---|---|213| `users` | The administrator. Plural only to allow a second editor later; `role` is constrained to `admin`. | `email` (citext, unique), `display_name`, `locale`, `disabled_at` |214| `passkey_credentials` | WebAuthn public material. | `credential_id` (unique), `public_key`, `counter`, `transports`, `device_type`, `backed_up`, `aaguid`, `name`, `last_used_at`, `revoked_at` |215| `setup_tokens` | One-time enrolment tokens issued from the shell. | `token_hash` (SHA-256), `purpose` (`setup`/`recovery`), `expires_at`, `used_at` |216| `recovery_codes` | Ten one-use codes. | `code_hash`, `used_at` |217| `webauthn_challenges` | Short-lived challenges, deleted when read. | `challenge`, `kind` (`registration`/`authentication`/`reauthentication`), `binding`, `expires_at` |218| `sessions` | Server-side sessions. | `token_hash`, `csrf_token`, `method` (`passkey`/`setup`/`recovery_code`), `reauth_at`, `expires_at`, `last_seen_at` (a session unused for 7 days is refused), `revoked_at`, `ip`, `user_agent` |219| `rate_limits` | Fixed-window counters. | primary key `(key, window_start)`, `count` |220| `audit_events` | Append-only trail. | `actor` (`admin`/`system`/`worker`/`anonymous`), `action`, `entity`, `entity_id`, `meta` (jsonb), `ip` |221| `settings` | Non-secret settings as key → jsonb. | validated by `SettingsSchema` (`src/server/settings.ts`) |222| `worker_heartbeat` | One row, `name = 'worker'`. | `beat_at`, `info` (worker id, running jobs, Chromium state, prompt version) |223224### Content225226| Table | Purpose | Notable columns |227|---|---|---|228| `topics` | Categories with French and English names. | `slug`, `name_fr`, `name_en`, `tone`, `position` |229| `tags` | Free keywords. | `slug`, `name` |230| `sheet_localizations` | Groups the language versions of one sheet. | — |231| `sheets` | One sheet in one language. | `slug` (unique per locale among live rows), `locale`, `localization_id`, `status`, `doc`, `current_revision_id`, `publication_id`, `version` (optimistic lock), `featured`, `featured_position`, `allow_download`, `seo_title`, `seo_description`, `canonical_url`, `og_asset_id`, `deleted_at` |232| `sheet_revisions` | Immutable saved states. | `number` (unique per sheet), `doc`, `doc_hash`, `origin`, `summary`, `patch`, `parent_id`, `job_id` |233| `sheet_publications` | Immutable public snapshots. | `revision_id`, `number`, `doc`, `slug`, `title`, `summary`, `reading_minutes`, `search_text`, generated `search_fr` / `search_en` tsvectors (GIN), trigram index on the title, `unpublished_at` |234| `slug_redirects` | Old public addresses. | primary key `(locale, old_slug)` |235| `sheet_topics`, `sheet_tags` | Link tables, rebuilt from the document's metadata on every save. | |236| `preview_tokens` | Private preview links. | `token_hash`, `expires_at`, `revoked_at` |237| `templates` | The author's own templates (built-in ones live in code). | `key`, `doc`, `locale`, `deleted_at` |238| `sheet_views` | View counter. | primary key `(sheet_id, day)`, `views` |239240### Media, data, sources241242| Table | Purpose | Notable columns |243|---|---|---|244| `assets` | Uploads and generated exports. | `kind` (`image`/`svg`/`pdf`/`text`/`csv`/`json`/`export`), `mime`, `bytes`, `sha256`, `storage_key`, `width`, `height`, `alt`, `credit`, `license`, `scan_status` (`clean`/`unscanned`), `is_public`, `deleted_at` |245| `datasets` | Parsed shape of a CSV/JSON upload. | `columns`, `row_count`, `sample` (first 20 rows) |246| `sources` | Cross-sheet reference library keyed by `fingerprint` (`doi:…`, `url:…`, `title:…`). The copy inside each document stays authoritative for rendering. | `status`, `checked_at` |247| `sheet_sources` | Which sheet cites which library source. | `local_id` |248249### Claude studio and exports250251| Table | Purpose | Notable columns |252|---|---|---|253| `ai_conversations`, `ai_messages` | One conversation per sheet: the author's requests and the assistant's summaries. | `job_id`, `revision_id`, `diff` |254| `generation_jobs` | Queue for `outline`, `generate`, `edit`, `translate`, `verify_sources`. | `status`, `stage`, `progress`, `input`, `output`, `error`, `attempts`, `max_attempts`, `run_after`, `cancel_requested`, `idempotency_key` (unique), `model`, `prompt_version`, `locked_by`, `locked_at` |255| `generation_usage` | One row per completed model call: the ledger budgets are checked against. | `model`, `stage`, token counts, `cost_usd`, `duration_ms`, `stop_reason`, `request_id` |256| `export_jobs` | Queue for exports. | `revision_id` or `publication_id`, `format`, `options` (`scale`, `blockId`), `status`, `asset_id`, `is_public`, `attempts` |257258## 6. Design system259260Two independent token sets.261262**Site and studio chrome** — `src/app/globals.css`. Tailwind 4 with `@theme` mapping colour utilities (`bg-paper`, `text-muted`, `border-line`, `text-accent`, `text-bad`…) to `--ui-*` custom properties. Light values on `:root`, dark values under `prefers-color-scheme: dark` unless the visitor chose a mode (`html[data-mode]`, stored in `localStorage` as `sheet-mode` and applied before first paint by an inline script in `src/app/layout.tsx`). Shared utilities: `btn`, `btn-primary`, `btn-danger`, `btn-ghost`, `btn-sm`, `field`, `card`, `eyebrow`, `safe-b`, `safe-x`. Controls have a 44 px minimum height. Fonts are self-hosted through `@fontsource-variable`: Inter (text), Fraunces (display), JetBrains Mono.263264**Sheets** — `src/renderer/sheet.css`. A sheet reads only `--sh-*` tokens set by `.sheet[data-theme]` and `.sheet[data-accent]`, so a dark sheet stays dark on a light page.265266| Token | Role |267|---|---|268| `--sh-bg`, `--sh-surface`, `--sh-raised` | backgrounds |269| `--sh-ink`, `--sh-muted`, `--sh-faint` | text |270| `--sh-line`, `--sh-grid` | rules, chart grid |271| `--sh-t-<tone>`, `--sh-t-<tone>-soft` | strong and soft colour of each of the 8 tones |272| `--sh-accent`, `--sh-accent-soft` | resolved from `theme.accent` |273| `--sh-contra` | opposite side of diverging scales (red, or blue when the accent is red, pink or amber) |274275Theme presets: `scientific-light`, `scientific-dark`, `editorial`, `blueprint`, `finance`. Documents never carry colours, only a preset, an accent tone and per-element tones. `src/renderer/themes.ts` is a TypeScript copy of the palettes for outputs that cannot read CSS variables (standalone SVG); `tests/unit/renderer.themes.test.ts` compares the two.276277Layout: `.sh-frame` is a container (`container: sheet / inline-size`), so the 12-column grid responds to the width of the sheet itself — a narrow preview in the editor gets the phone layout. Fixed formats (`a4`, `slide-16-9`, `square`, `portrait-4-5`) keep their pixel size and scroll horizontally on small screens. Print rules avoid breaking inside blocks and hide interactive controls.278279## 7. Technical trade-offs280281| Decision | Reason | Cost |282|---|---|---|283| One package with modular folders, not a monorepo | One deployable; a workspace would add tooling for no isolation gain | Module boundaries rely on discipline, not on package boundaries |284| SQL migrations with checksums, no ORM | The queries that matter (queue claim, full-text ranking, generated tsvector columns) are plain SQL; `postgres` tagged templates keep them parameterised | Row types are written by hand next to each query |285| No Redis: queue, rate limits and sessions in PostgreSQL | One fewer service; jobs and sessions survive restarts; `FOR UPDATE SKIP LOCKED` gives exactly-one claim | Rate limits cost a write per request; the progress stream polls the job row (every 500 ms per open stream); `LISTEN/NOTIFY` is used only to wake the worker |286| A separate worker process | Model calls last minutes and exports need a browser; neither belongs in a request handler | Two processes to supervise; they must share `STORAGE_ROOT` |287| Documents are data, not HTML | The model, the editor and imports can only produce what the schema allows; the renderer never interprets markup | New visual ideas need a schema and renderer change |288| Exports through headless Chromium on the real page | PDF and PNG use the same fonts, KaTeX output and CSS as the site | A browser binary must be installed next to the worker |289| A custom Node server in front of Next.js | Explicit timeouts and graceful stop; `.env` loaded the same way for scripts and worker | `tsx` is a runtime dependency |290| Static CSP with `'unsafe-inline'` | Next's inline bootstrap scripts would otherwise need a per-request nonce on every page (comment in `next.config.ts`) | Script injection is not blocked by CSP; see [SECURITY.md](SECURITY.md) |291| Gateway compresses, app does not (`compress: false`) | Avoids double work behind a proxy | A proxy without compression serves uncompressed responses |292| Built-in templates live in code | Versioned and tested against the schema | Changing one needs a deployment |293| Only the `local` storage driver | A directory or NAS mount is enough for one node | `STORAGE_DRIVER` refuses any other value; no S3 |294| Undo history as whole documents (100 steps) | Documents are small JSON values | Memory grows with very large sheets |295| Published documents are migrated when read, never rewritten | Publications stay immutable across schema versions | Every read pays the migration once `SCHEMA_VERSION` moves |296
Files main
Menu

Theme