SPB Git

Repository Public

TESTING.md 8.6 KB

# Testing

Four suites. All of them run on a developer machine with PostgreSQL 17 and the Playwright browsers installed.

Suite Command What it exercises Needs
Unit pnpm test:unit Schema, patches, migrations, renderer (every block type, three modes), diagram layout, chart scales, SVG sanitiser, simulations (maths and rendering), templates (FR and EN), editor forms and diagram operations, quality report, outbound-address rules, pricing nothing
Integration pnpm test (scripts/test.sh) A production build, its worker and a real database, over HTTP: passkeys, sessions, CSRF, lock-out, recovery, CMS, draft isolation, uploads, Claude pipeline, exports, audit, restart, secret scan, backup → restore PostgreSQL, Chromium
Browser pnpm test:e2e (scripts/e2e.sh) The studio journey in Chromium with a virtual passkey authenticator; the studio at 390 px; the public site at seven widths in Chromium and in WebKit with an iPhone profile PostgreSQL, Chromium, WebKit
Real API REAL_AI=1 scripts/test.sh The same build against the real Anthropic API: one generation, one edit ANTHROPIC_API_KEY in .env; costs money

In the integration and browser suites the Anthropic API is replaced by tests/fake-anthropic.ts, a local server that speaks the streaming protocol and returns deterministic JSON (including deliberately wrong answers: invalid JSON, out-of-scope patches, a refusal, a document that claims its sources are verified). Production never loads it. Nothing else is mocked: the database, the worker, the queue, the headless browser and the file storage are the real ones.

# Results of the run of 2026-11-17 (laptop, macOS, Node 26.13, PostgreSQL 17.9)

Suite Result
Unit 711 passed, 0 failed (17 files)
Integration — tests/api.test.ts 53 passed, 0 failed
Integration — restart (tests/restart.test.ts) 4 passed: publications served, sessions kept, a job left "running" by a dead worker re-run to completion, worker heartbeat
Secret scan client bundle and both process logs contain neither the API key nor APP_SECRET; scripts/secrets-scan.ts clean on 255 files
Backup → restore encrypted dump and storage archive written, read back, restored into an empty database and directory; row counts equal on six tables; every restored file matches its recorded SHA-256; an archive with one byte appended is refused
Browser 27 passed, 0 failed (studio 13, studio on a phone 2, public pages 6 × 2 browsers)
Real API 2 passed — see below
Lint, types eslint and tsc --noEmit clean

# Real Claude run (model claude-opus-5-5, generation effort high)

  • Prompt: a short sheet on the exponential moving average with an equation, a diagram and a quiz.
  • Generation: accepted on the first answer (no repair round), 12 blocks (heading, callout, paragraph, equation, diagram, cards, quiz, references), 83 s, 9 885 output tokens, 0.34 USD (of which 0.15 USD is the first write of the system prompt to the prompt cache).
  • Edit ("add a one-sentence definition box right after the title, touch nothing else"): one block added, none changed or removed, 4.7 s, 0.04 USD (29 164 tokens read from the cache).
  • Every source in the generated sheet was stored as unverified, as designed.

# Production checks (https://sheet.spboucher.ai, 2026-11-17)

  • Public browser suite against the production address: 12 passed (6 in Chromium at seven widths, 6 in WebKit with an iPhone profile).
  • Lighthouse 13, mobile profile, lab data, one run per page:
Page Performance Accessibility Best practices SEO LCP CLS
/ 100 100 100 100 1.0 s 0.004
/s/melange-experts-routage-top-k 99 100 96 100 1.1 s 0
/search?q=garch 100 100 100 63 0.8 s 0

The search page is noindex on purpose, which is what lowers its SEO score. On the sheet page, "best practices" loses points for missing source maps and for a Content-Security-Policy note (the policy keeps 'unsafe-inline', see docs/SECURITY.md). The first measurement found two accessibility defects (contrast of secondary and accent text, a button label that did not start with its visible text) and a heading-order gap on list pages; all three were fixed and re-measured.

# What the integration suite asserts (summary)

  • Anonymous access: 401 on every /api/admin/* and credential route (18 GET, 14 write routes); studio pages redirect to sign-in and are noindex; security headers present.
  • Setup: registration refused from another origin, another RP ID or without user verification; a challenge cannot be used by another browser; replay refused; ten recovery codes, stored hashed; the setup token is single-use; bootstrap closed afterwards.
  • Sign-in: bad signature, unknown credential, wrong origin, missing user verification and replay all refused; session rotated at each sign-in and revoked at sign-out; CSRF header and same-origin enforced; progressive lock-out per address.
  • Passkey management: fresh verification required to add or remove; revoked passkey cannot sign in; the last passkey cannot be removed.
  • Recovery: a code opens a session that can only enrol a passkey; doing so revokes earlier passkeys, sessions and codes.
  • CMS: optimistic locking (409), schema validation (422) including unknown block types, unknown fields and another author name; revisions and revert; publication refused on quality errors; atomic publish; published copy unchanged while the draft moves on; slug change answers 308 from the old address; unpublish removes the sheet from page, API, sitemap and feeds.
  • Draft isolation: no draft slug appears in the public API, search, sitemap, RSS, JSON feed or home page; preview links work only with their token, are noindex, and die when revoked.
  • Injected markup: <script>, onerror= and javascript: links typed into a block are shown as text in the public HTML.
  • Uploads: type decided from bytes; HTML, executables, broken JSON and oversized files refused; a hostile SVG is stored without script, handlers, foreign content or external references and is served as a sandboxed attachment; CSV becomes a typed dataset; images stay private until a published sheet uses them.
  • Claude pipeline: brief validation; attachment handed over as data with its embedded instructions reported; idempotent job creation; author, locale and source status forced by the server; image blocks and assets from the model dropped; repair round on invalid output; clear failure when output never validates or the model declines; private addresses refused as source links; cancellation; patch-based edit with byte-identical untouched blocks, undo by revert; selection scope and locked blocks enforced; no-op and stale-version handling; translation with identical skeleton and sources; budget stop; usage ledger amounts.
  • Exports: PDF with embedded fonts; PNG at 1× and 2× (width doubles); standalone SVG with embedded fonts and no unresolved CSS variable; JSON export re-imported without loss; reader PDF offered on the public page.
  • Audit: sign-ins, failures, recovery, revocations, publications, uploads, settings and jobs are recorded; no secret appears in the trail.

# Not covered by automated tests

  • Face ID / Touch ID on real devices. The browser suite uses Chromium's virtual platform authenticator (user verification on, and once deliberately off). Enrolment and sign-in on a real iPhone (Safari) and a real Mac (Safari, Touch ID) must be done by hand once after deployment; Firefox was not run.
  • WebKit for the studio. WebKit has no virtual authenticator, so only the public pages run in WebKit (iPhone profile). The studio on iPhone Safari is therefore untested beyond layout at 390 px in Chromium.
  • VoiceOver. Accessibility is checked structurally (roles, labels, headings, focus, touch-target sizes), not with a screen reader.
  • Field Core Web Vitals. Only lab measurements exist (below); there is no real-user data yet.
  • ClamAV. The scanner protocol code is not exercised (no clamd in the test environment).
  • Reference checking against the network is not part of the suites (it needs the internet); it was run by hand on 2026-11-17: a real DOI with its real title resolved, the same DOI with an invented title was reported as registered under a different title, an invented DOI as non-existent, an arXiv URL as existing.
  • Docker Compose files in infra/ were not built or run.
  • Pinch-zoom and drag gestures in the diagram viewer and the visual diagram editor: the underlying operations are unit-tested and the editor opens in the browser suite, but multi-touch gestures were not driven.
Files main
Menu

Theme