SPB Git

Repository Public

ARCHITECTURE.md 24.5 KB

# Architecture

This document describes the code as it is. File references are relative to the project root.

# 1. Components

text
                                   ┌──────────────────────────────────────────────┐
  Reader / administrator           │ web process — server.ts                      │
  (browser)                        │  Node HTTP server (timeouts, graceful stop)  │
      │  HTTPS                     │  └─ Next.js 16 App Router                    │
      ▼                            │      (site)/…         public pages           │
  reverse proxy ──── HTTP ───────▶ │      admin/…          studio pages           │
  (TLS, X-Forwarded-For,           │      api/public/*     anonymous JSON API     │
   body limit, compression)        │      api/auth/*       WebAuthn, sessions     │
                                   │      api/admin/*      studio API             │
                                   │      api/health       liveness               │
                                   │      media/<id>       public files           │
                                   │      preview/<token>  private draft preview  │
                                   │        (+ /asset/<id> for its images)        │
                                   │      render/<token>   bare page for exports   │
                                   └───────────────┬───────────────┬──────────────┘
                                                   │               │
                                         ┌─────────▼────┐   ┌──────▼────────┐
                                         │ PostgreSQL 17│   │ STORAGE_ROOT  │
                                         │ (one schema) │   │ ab/cd/<uuid>  │
                                         └─────────▲────┘   └──────▲────────┘
                                                   │               │
                                   ┌───────────────┴───────────────┴──────────────┐
                                   │ worker process — src/worker/index.ts         │
                                   │  AI jobs (2 at a time)  → api.anthropic.com  │
                                   │  verify_sources         → Crossref, doi.org, │
                                   │                           source URLs        │
                                   │  exports (1 at a time)  → headless Chromium  │
                                   │                           → web /render/…    │
                                   │  heartbeat (15 s), housekeeping (hourly)     │
                                   └──────────────────────────────────────────────┘

  scripts/backup.sh  (scheduled by the operator)  → BACKUP_DIR

There is no Redis, no message broker and no object store. PostgreSQL holds documents, the job queues, sessions, rate-limit counters, the audit trail and the AI usage ledger. Files live in one directory tree (STORAGE_ROOT) that both processes must be able to read and write.

The web process is server.ts: a plain node:http server that hands every request to Next.js, sets requestTimeout to 5 minutes, headersTimeout to 60 s, keepAliveTimeout to 65 s, and closes on SIGTERM/SIGINT (forced exit after 8 s). It listens on HOST (default 0.0.0.0) and PORT (default 8700).

# 2. Module map

A single Node package. Boundaries are kept by directory and by import direction.

text
  src/schema  ◀──  src/renderer  ◀──┐
       ▲           src/sim  ◀───────┤
       │                            │
       └────────  src/server ◀── src/ai ◀── src/worker
                       ▲            ▲
                       └── src/lib/api (route tables) ◀── src/app (routes)
                                                           src/components (UI)
Module Responsibility May import Notes
src/schema Sheet Document Schema v1 (Zod), parseSheet, migrateSheet, stableStringify, traversal, applyPatch, diffSheets, slugify zod only No React, no Next, no database. Runs in the browser too.
src/renderer Block components, SheetView, diagram layout, chart drawing, inline parser, KaTeX wrapper, SVG sanitiser, theme palettes, standalone SVG export schema, sim export-svg.ts imports react-dom/server and is used by the worker and unit tests only.
src/sim The 13 simulations: pure compute, a View, shared numeric core, the expression parser schema, renderer/chart registry.ts is server-safe (used by the quality report and the prompts).
src/templates 12 built-in templates as code schema Each build(locale) result is passed through mustParseSheet.
src/lib env, db (one postgres client, pool of 10), auth, session-store, ratelimit, audit, http, log, crypto, client fetch helper — lib/api/*routes.ts are the three API route tables; they import server and ai.
src/server All writes to sheets (sheets.ts), assets and scanning, storage driver, job queue API, export requests, quality report, settings, render tokens, dashboard/audit/preview helpers schema, renderer (sanitiser, LaTeX check), sim/registry, lib misc.ts also imports ai/client for the dashboard health tile.
src/ai client.ts (the only caller of the Anthropic API), prompts.ts, pipeline.ts, net.ts, sources.ts, pricing.ts schema, server, lib, sim/registry Server and worker only; the SDK is in serverExternalPackages.
src/worker Queue loop, job state transitions, export rendering everything server-side Not part of the Next.js build.
src/components Site chrome, studio pages, editor schema, renderer, sim, lib/client-api, i18n; types only from server and ai Client components must not import lib/env or lib/db.
src/app Routes; thin wrappers over server functions and the route tables all

The import direction is a convention; there is no lint rule that enforces it.

# API surfaces

Each surface is one catch-all route whose guard runs before a handler is looked up (src/lib/api/router.ts), so a handler cannot be reached without its guard.

Surface Route file Table Guard
/api/public/* src/app/api/public/[...path]/route.ts src/lib/api/publicroutes.ts none (rate limit per client address inside each handler)
/api/auth/* src/app/api/auth/[...path]/route.ts src/lib/api/authroutes.ts per route: same-origin check, rate limit, session + CSRF where relevant
/api/admin/* src/app/api/admin/[...path]/route.ts src/lib/api/adminroutes.ts requireAdmin: passkey session, CSRF header on non-GET

# 3. Request lifecycles

# (a) An anonymous reader opens /s/<slug>

text
GET /s/<slug>
 └─ src/app/(site)/s/[slug]/page.tsx  (force-dynamic)
     ├─ getLocale()                       cookie sheet_locale, else Accept-Language
     ├─ getPublished(slug, locale)        src/server/sheets.ts
     │    SELECT … FROM sheets s JOIN sheet_publications p ON p.id = s.publication_id
     │    WHERE s.slug = $1 AND s.deleted_at IS NULL AND s.status = 'published'
     │    ├─ no row → slug_redirects → permanentRedirect(/s/<new slug>) or 404
     │    └─ row → mustParseSheet(p.doc)  (migrated on read if the schema moved on)
     ├─ relatedPublished(…)               shared topics first
     └─ render: JSON-LD + SheetReader → SheetView(mode "public", assetUrl = /media/<id>; a private preview uses /preview/<token>/asset/<id>)
         └─ ViewBeacon (client) → POST /api/public/views after 3 s when the tab is visible

The page reads the publication snapshot, never sheets.doc. Images are fetched from /media/<id>, which serves only assets flagged is_public. Metadata (generateMetadata) reuses the same query through React.cache. Nothing is cached server-side; the JSON API under /api/public sends Cache-Control: public, max-age=….

# (b) A save in the editor

text
Editor (client) — src/components/editor/Editor.tsx
 change → useDocState.commit → autosave timer (2.5 s) or ⌘S
   parseSheet(doc) in the browser  ── invalid → "Invalid document — not saved", nothing sent
   PATCH /api/admin/sheets/<id>  { doc, expectedVersion }   + X-CSRF-Token
     requireAdmin → body read under a 2 MB cap (readBody) → Zod → validDoc(doc)  (422 with issues if invalid)
     saveDoc()  — src/server/sheets.ts, one transaction
       SELECT … FROM sheets WHERE id = $1 FOR UPDATE
       version ≠ expectedVersion → 409 version_conflict
       locale changed            → 422 locale_locked
       same doc_hash as current  → { changed: false }  (no revision)
       INSERT sheet_revisions (number = max + 1, parent_id = current, origin 'manual')
       UPDATE sheets SET doc, title, summary, current_revision_id, version = version + 1
       syncTaxonomy: sheet_topics, sheet_tags, sources / sheet_sources
     ← { version, revisionId, revisionNumber, diff, quality }

A 409 puts the editor in a "conflict" state that stops autosave and offers a reload. Between saves, the editor asks POST /api/admin/quality (debounced 900 ms) for the report of the unsaved document; it is the same function that gates publication.

# (c) A Claude generation job

text
POST /api/admin/generations { kind: "generate", brief, outline, idempotencyKey }
  requireAdmin → BriefSchema / OutlineSchema → ANTHROPIC_API_KEY present? → attachments exist?
  enqueueJob()  — src/server/jobs.ts
    same idempotency_key → return the existing job
    rate limit jobs:<user> (aiJobsPerHour)
    INSERT generation_jobs (status 'queued')  → pg_notify('sheet_jobs') → audit job.queued.generate
  ← job view

Browser: EventSource /api/admin/generations/<id>/events
  the web process re-reads the job row every 500 ms and emits an event when it changes

Worker — src/worker/index.ts
  LISTEN sheet_jobs (wake-up) + 2 s poll
  claim(): UPDATE … WHERE id = (SELECT … WHERE status = 'queued' AND run_after <= now()
                                 ORDER BY created_at FOR UPDATE SKIP LOCKED LIMIT 1)
  runJob(): watch timer (cancel flag, time limit) → resolveModel → runGenerate()
    extractMaterial   attachments from storage, URLs through safeFetch
    askJson           callClaude (budget check, stream, ledger row) → extractJson
                      → normalizeModelDoc → parseSheet; up to 2 repair rounds
    qualityReport
    createSheet       sheets row + revision 1 (origin 'ai_generation', job_id) — draft
    ai_conversations / ai_messages
  UPDATE generation_jobs SET status = 'succeeded', output = …
Browser: on "succeeded" → /admin/sheets/<sheetId>/edit

See AI_PIPELINE.md for each stage.

# (d) An export

text
POST /api/admin/exports { sheetId, format: pdf|png|svg, scale, blockId?, revisionId? }
  requireAdmin → rate limit export:<user> (120/h) → requestExport()  — src/server/exports.ts
    blockId must be a valid id; svg needs a blockId; pdf + blockId → 422 whole_sheet_only
    INSERT export_jobs (revision_id = current revision unless given) → pg_notify('sheet_jobs')
Browser polls GET /api/admin/exports/<id> every 1.2 s

Worker — exportsTick() (one export at a time), src/worker/export.ts
  svg: renderBlockSvg(block)  react-dom/server → theme tokens resolved to colours, fonts embedded
  png / pdf:
    renderToken({ kind, id }, 600 s)            HMAC-SHA256 with APP_SECRET
    Chromium (playwright-core) → INTERNAL_URL/render/<token>
       page.route: every request to another origin than INTERNAL_URL is aborted
       /render/<token>/asset/<id> serves only images referenced by that document
    a layout narrower than 280 px aborts the export (collapsed page)
    png: element screenshot at deviceScaleFactor = scale (refused above 16 000 px a side)
    pdf: fixed formats → one page of the format's size; article → A4 with margins and a footer
  createExportAsset() → assets row (kind 'export') + file → export_jobs.asset_id
Browser: download /api/admin/assets/<assetId>/file?download=1

On publication, when allow_download is on, publish() inserts a public PDF export job for the new publication and notifies the worker; readers get it through /media/<assetId>. JSON export is not a job: GET /api/admin/sheets/<id>/export.json returns stableStringify(doc, 2).

# 4. Publication model

Four things, three of them immutable.

Thing Table / column Mutable? Meaning
Working copy sheets.doc yes What the studio shows. Always equal to the document of sheets.current_revision_id.
Revision sheet_revisions no Every saved state, numbered per sheet, with origin (manual, ai_generation, ai_edit, revert, import, template, source_check), parent, optional patch and job id. Reverting writes a new revision.
Publication sheet_publications no A copy of one revision taken at publication time, with title, summary, reading time and search text. Publishing again adds a row.
Pointer sheets.publication_id yes The one publication the public sees. NULL when not published.

publish() (src/server/sheets.ts) runs in one transaction: lock the sheet, check expectedVersion, run the quality report and refuse on errors, mark the previous publication unpublished_at, insert the new publication from the current revision, move the pointer and set status = 'published', flag the document's image assets is_public, queue the reader PDF, write the audit event. Readers see the old publication or the new one, never a mixture. A table constraint ties the two columns together: (status = 'published') = (publication_id IS NOT NULL).

Statuses are draft, in_review, published, archived. in_review and archived are labels set from the sheet panel; they carry no workflow. There is no scheduled publication.

unpublish() clears the pointer, sets the status back to draft, clears featured, and withdraws the public flag from the sheet's reader PDFs and from the images of that publication that no other published sheet uses. deleteSheet() is a soft delete (deleted_at), refused while published; the slug becomes free again because the unique index is partial.

Slug changes: updateSettings() records the old slug in slug_redirects if the sheet has ever been public, and public lookups answer 301 from the old address.

A slug change does not touch publication rows: sheet_publications.slug keeps the slug the sheet had when that publication was made; the live address is sheets.slug.

Languages: a French and an English sheet are two sheets rows sharing a localization_id. A sheet's locale cannot change (locale_locked); translation creates the sibling.

# 5. Data model

All tables are created by db/migrations/001_init.sql. Keys are UUIDs unless noted; timestamps are timestamptz. scripts/migrate.ts adds schema_migrations (name, checksum, applied_at) and refuses to run if an applied file was modified.

# Administrator and authentication

Table Purpose Notable columns
users The administrator. Plural only to allow a second editor later; role is constrained to admin. email (citext, unique), display_name, locale, disabled_at
passkey_credentials WebAuthn public material. credential_id (unique), public_key, counter, transports, device_type, backed_up, aaguid, name, last_used_at, revoked_at
setup_tokens One-time enrolment tokens issued from the shell. token_hash (SHA-256), purpose (setup/recovery), expires_at, used_at
recovery_codes Ten one-use codes. code_hash, used_at
webauthn_challenges Short-lived challenges, deleted when read. challenge, kind (registration/authentication/reauthentication), binding, expires_at
sessions Server-side sessions. token_hash, csrf_token, method (passkey/setup/recovery_code), reauth_at, expires_at, last_seen_at (a session unused for 7 days is refused), revoked_at, ip, user_agent
rate_limits Fixed-window counters. primary key (key, window_start), count
audit_events Append-only trail. actor (admin/system/worker/anonymous), action, entity, entity_id, meta (jsonb), ip
settings Non-secret settings as key → jsonb. validated by SettingsSchema (src/server/settings.ts)
worker_heartbeat One row, name = 'worker'. beat_at, info (worker id, running jobs, Chromium state, prompt version)

# Content

Table Purpose Notable columns
topics Categories with French and English names. slug, name_fr, name_en, tone, position
tags Free keywords. slug, name
sheet_localizations Groups the language versions of one sheet. —
sheets One sheet in one language. slug (unique per locale among live rows), locale, localization_id, status, doc, current_revision_id, publication_id, version (optimistic lock), featured, featured_position, allow_download, seo_title, seo_description, canonical_url, og_asset_id, deleted_at
sheet_revisions Immutable saved states. number (unique per sheet), doc, doc_hash, origin, summary, patch, parent_id, job_id
sheet_publications Immutable public snapshots. revision_id, number, doc, slug, title, summary, reading_minutes, search_text, generated search_fr / search_en tsvectors (GIN), trigram index on the title, unpublished_at
slug_redirects Old public addresses. primary key (locale, old_slug)
sheet_topics, sheet_tags Link tables, rebuilt from the document's metadata on every save.
preview_tokens Private preview links. token_hash, expires_at, revoked_at
templates The author's own templates (built-in ones live in code). key, doc, locale, deleted_at
sheet_views View counter. primary key (sheet_id, day), views

# Media, data, sources

Table Purpose Notable columns
assets Uploads and generated exports. kind (image/svg/pdf/text/csv/json/export), mime, bytes, sha256, storage_key, width, height, alt, credit, license, scan_status (clean/unscanned), is_public, deleted_at
datasets Parsed shape of a CSV/JSON upload. columns, row_count, sample (first 20 rows)
sources Cross-sheet reference library keyed by fingerprint (doi:…, url:…, title:…). The copy inside each document stays authoritative for rendering. status, checked_at
sheet_sources Which sheet cites which library source. local_id

# Claude studio and exports

Table Purpose Notable columns
ai_conversations, ai_messages One conversation per sheet: the author's requests and the assistant's summaries. job_id, revision_id, diff
generation_jobs Queue for outline, generate, edit, translate, verify_sources. status, stage, progress, input, output, error, attempts, max_attempts, run_after, cancel_requested, idempotency_key (unique), model, prompt_version, locked_by, locked_at
generation_usage One row per completed model call: the ledger budgets are checked against. model, stage, token counts, cost_usd, duration_ms, stop_reason, request_id
export_jobs Queue for exports. revision_id or publication_id, format, options (scale, blockId), status, asset_id, is_public, attempts

# 6. Design system

Two independent token sets.

Site and studio chrome — src/app/globals.css. Tailwind 4 with @theme mapping colour utilities (bg-paper, text-muted, border-line, text-accent, text-bad…) to --ui-* custom properties. Light values on :root, dark values under prefers-color-scheme: dark unless the visitor chose a mode (html[data-mode], stored in localStorage as sheet-mode and applied before first paint by an inline script in src/app/layout.tsx). Shared utilities: btn, btn-primary, btn-danger, btn-ghost, btn-sm, field, card, eyebrow, safe-b, safe-x. Controls have a 44 px minimum height. Fonts are self-hosted through @fontsource-variable: Inter (text), Fraunces (display), JetBrains Mono.

Sheets — src/renderer/sheet.css. A sheet reads only --sh-* tokens set by .sheet[data-theme] and .sheet[data-accent], so a dark sheet stays dark on a light page.

Token Role
--sh-bg, --sh-surface, --sh-raised backgrounds
--sh-ink, --sh-muted, --sh-faint text
--sh-line, --sh-grid rules, chart grid
--sh-t-<tone>, --sh-t-<tone>-soft strong and soft colour of each of the 8 tones
--sh-accent, --sh-accent-soft resolved from theme.accent
--sh-contra opposite side of diverging scales (red, or blue when the accent is red, pink or amber)

Theme presets: scientific-light, scientific-dark, editorial, blueprint, finance. Documents never carry colours, only a preset, an accent tone and per-element tones. src/renderer/themes.ts is a TypeScript copy of the palettes for outputs that cannot read CSS variables (standalone SVG); tests/unit/renderer.themes.test.ts compares the two.

Layout: .sh-frame is a container (container: sheet / inline-size), so the 12-column grid responds to the width of the sheet itself — a narrow preview in the editor gets the phone layout. Fixed formats (a4, slide-16-9, square, portrait-4-5) keep their pixel size and scroll horizontally on small screens. Print rules avoid breaking inside blocks and hide interactive controls.

# 7. Technical trade-offs

Decision Reason Cost
One package with modular folders, not a monorepo One deployable; a workspace would add tooling for no isolation gain Module boundaries rely on discipline, not on package boundaries
SQL migrations with checksums, no ORM The queries that matter (queue claim, full-text ranking, generated tsvector columns) are plain SQL; postgres tagged templates keep them parameterised Row types are written by hand next to each query
No Redis: queue, rate limits and sessions in PostgreSQL One fewer service; jobs and sessions survive restarts; FOR UPDATE SKIP LOCKED gives exactly-one claim Rate limits cost a write per request; the progress stream polls the job row (every 500 ms per open stream); LISTEN/NOTIFY is used only to wake the worker
A separate worker process Model calls last minutes and exports need a browser; neither belongs in a request handler Two processes to supervise; they must share STORAGE_ROOT
Documents are data, not HTML The model, the editor and imports can only produce what the schema allows; the renderer never interprets markup New visual ideas need a schema and renderer change
Exports through headless Chromium on the real page PDF and PNG use the same fonts, KaTeX output and CSS as the site A browser binary must be installed next to the worker
A custom Node server in front of Next.js Explicit timeouts and graceful stop; .env loaded the same way for scripts and worker tsx is a runtime dependency
Static CSP with 'unsafe-inline' Next's inline bootstrap scripts would otherwise need a per-request nonce on every page (comment in next.config.ts) Script injection is not blocked by CSP; see SECURITY.md
Gateway compresses, app does not (compress: false) Avoids double work behind a proxy A proxy without compression serves uncompressed responses
Built-in templates live in code Versioned and tested against the schema Changing one needs a deployment
Only the local storage driver A directory or NAS mount is enough for one node STORAGE_DRIVER refuses any other value; no S3
Undo history as whole documents (100 steps) Documents are small JSON values Memory grows with very large sheets
Published documents are migrated when read, never rewritten Publications stay immutable across schema versions Every read pays the migration once SCHEMA_VERSION moves
Files main
Menu

Theme