SPB Git

Repository Public

TEST-REPORT-NEXT.md 16.0 KB

# SP Mail Next — test report

Statuses: PASS (executed, result observed), FAIL, BLOCKED (access, credential or hardware missing), NOT TESTED. A line is PASS only if the command was run on the stated commit and its output read; counts are copied from the runners' summaries. Secrets and message contents are absent.

Commits under test — backend + web: ead8d18 (branch next, repository sp-mail); Apple: b414110 (branch next, repository sp-mail-apple). Dates: 2026-11-20.

Environments: dev = backend process started from the checkout on the development laptop, real PostgreSQL 17, the real mail server (Stalwart 1.1.1 on BHS64) reached through an SSH tunnel and used only through the staging domain stg.spboucher.ai; browsers = Chromium (Playwright 1.65.1) with a virtual WebAuthn authenticator; Apple = macOS 27 on the laptop, iOS 26.5 simulators (iPhone 17 Pro, iPad Pro 13-inch). Production was not touched and nothing was deployed. No test sent mail outside stg.spboucher.ai.

Important context: the staging domain lives on the production mail server. Test mailboxes are created and purged through the backend's administration API; the registry added in this release refuses any address that is not a test prefix on stg.spboucher.ai.

# 1. Backend

# Suite Command Result Status
B1 Complete integration (existing 0.3.0 tests + Next) SPMAIL_TEST_PORT=8981 SPMAIL_TEST_DB=…/sp_mail_int_test node --import tsx --test --test-concurrency=1 tests/integration/api.test.ts tests/integration/templates.test.ts "tests/integration/v1-*.test.ts" "tests/integration/next-*.test.ts" 145 tests: 143 pass, 0 fail, 2 skipped (opt-in real-provider AI) PASS
B2 Pure unit tests node --import tsx --test "server/src/**/*.test.ts" 44 / 44 PASS
B3 SDK (TypeScript, Python, Swift example) and MCP server, generated types up to date pnpm test:sdk 31 / 31 PASS
B4 Type check pnpm typecheck clean PASS
B5 Real provider — Anthropic (claude-haiku-5-5), 5 calls, structured output validated, injected message not obeyed SPMAIL_TEST_AI_REAL=1 … next-ai.test.ts (run once on the AI branch) 9 / 9 PASS (on an earlier commit)
B6 Real provider — local MacLustr gateway — no key was created NOT TESTED
B7 Real embedding model (multilingual-e5-small, in process): three queries without keyword overlap rank the right message first one-off script, not in the suite observed once PASS (manual, earlier commit)
B8 Real APNs — no APNs credential configured; the pipeline was exercised against a local HTTP/2 fake only BLOCKED

What B1 covers for Next, with real mail between staging mailboxes: push token lifecycle and cross-user isolation, one alert per device per message (also across a server restart), JWT verified against the test key, preview levels, quiet hours and priority senders, per-mailbox mute, retry after 503, token deletion on 410, badge value and silent badge push, device removal on session revocation, nothing enqueued when unconfigured; snooze (folder, keyword, wake by the worker, cancel, missing message) and reminders (fire, resolved by a real reply); labels, saved searches, snippets, signatures with isolation between users; operator search across two mailboxes with stable pagination; attachment index and ZIP (bytes compared); contact harvest from real mail, vCard round trip, merge; undo send (held, cancelled with the draft and attachments kept, delivered once when not cancelled), sensitive-recipient confirmation; dashboard including the partial list; AI consent gates, rate limits, JSON validation and retry, prompt-injection containment (the model request carries the untrusted block and no tools; nothing is sent, moved or deleted whatever the model answers); rules (label, star, mark read, snooze, remind, event, draft reply; idempotent across a restart; never an outbound message; never applied to mail received before the rule was enabled); health, alert raised then resolved, notification dedup, administrator-only and re-authentication enforcement; watcher properties accepted by the real server; mailbox order; upstream 401/403 mapped to 502.

Not provoked by any test: alert conditions queue, disk, cert:*, delivery, webhooks, mail_server (threshold functions are unit-tested); APNs give-up after 6 attempts, 24 h expiry and 7-day prune; shared mailboxes (mailbox_grants) with push, contacts and search; two concurrent server instances; rule actions pin, notify_app, extract and the ai_category condition on real mail.

# 2. Web client

# Suite Command Result Status
W1 Unit (pure logic, editor model, offline queue, list model, sanitiser) pnpm test:web 376 / 376 PASS
W2 Main browser suite: desktop, mobile, shell, composer, productivity + intelligence, performance playwright test -c tests/e2e/playwright.config.ts (port 8982) 124 passed, 0 failed PASS
W3 Administration playwright test -c tests/e2e/admin/playwright.config.ts (port 8983) 39 passed PASS
W4 Templates studio playwright test -c tests/e2e/studio/playwright.config.ts (port 8984) 24 passed, 1 skipped (real AI, opt-in) PASS
W5 Offline and service worker playwright test -c tests/e2e/offline/playwright.config.ts (port 8985) 5 passed (the update-flow test last ran one CSS-only commit before the final one) PASS
W6 Test-mailbox registry node --import tsx --test tests/e2e/mailboxes.test.ts 3 / 3 PASS
W7 Accessibility (axe-core inside W2–W4) — 0 serious or critical violation on every screen checked, in light, dark and midnight; 172 checks on the productivity screens alone PASS (automated rules only)
W8 Browsers other than Chromium; installed web app (incl. iOS Safari); real touch devices; English interface in a browser — — NOT TESTED

Performance (W2, project perf; 10 000-message mailbox, median of 5, browser and backend on the same machine — no network latency): cold load to interactive list 164 ms, mailbox switch 106 ms, thread open 116 ms, search 790 ms (remarkably constant, 787–793 ms, which suggests a fixed delay rather than compute; not investigated), worst scroll frame 18.6 ms, JS heap after scrolling the whole list 7.8 MB. Budgets = measured + 25 % (tests/e2e/perf-budgets.json, PERFORMANCE.md). Bundle: entry path 628 kB raw / 201 kB gzip (was 498 / 155 at 0.3.0), post-login Next modules 201 / 61, everything else lazy.

Cross-device scenarios executed: read state changed in a second browser context appears in the first without a list refetch; a draft replaced from another device keeps both versions; actions taken offline reach the server exactly once after reconnect and the server's concurrent change wins; mailbox removed on the server disappears without a reload.

# 3. Apple apps

# Suite Command Result Status
A1 Core unit tests (sync hardening, Next client and store, rich text, composer support, helpers) + system package Scripts/test-core.sh 248 in 49 suites (39 integration tests skipped in this mode) + 26 in 7 suites PASS
A2 Core integration against the dev backend and real staging mail Scripts/test-core.sh --integration 248 passed, then 26 passed (run on commit d9b68e4 + the serial-run change, not re-run on the last UI commits) PASS (earlier commit)
A3 « Fails before » evidence for the audited core defects nine expectations run against an unmodified 1.1.0 copy 9 fail on 1.1.0; 8 pass on the new core, the ninth passes its redefined bound PASS
A4 Migration of a real 1.1.0 cache file to schema 2 opensACacheFileWrittenByTheReal110Core messages, sealed bodies, pending operations and outbox items readable PASS
A5 Builds, Debug, both targets + six extension targets Scripts/build.sh mac Debug, Scripts/build.sh ios Debug succeeded, 0 errors PASS
A6 In-app self-test, macOS Scripts/run-dev.sh --selftest 146 / 146 PASS
A7 In-app self-test, iPhone simulator Scripts/run-sim.sh 145 / 145 PASS
A8 In-app self-test, iPad simulator Scripts/run-sim.sh --device … 145 / 145 PASS
A9 Signed macOS Release build 2.0.0 (1), universal, three extensions Scripts/release.sh, Scripts/verify-release.sh built; verifier exit 0 (27 checks) PASS
A10 macOS notarisation of the build made from b414110 Scripts/release.sh, verify-release.sh, notarize.sh; then stapler validate, spctl -a -vv submission fae01265-6381-4b3e-abef-4bfd6d7a65a4 Accepted; app and DMG stapled; spctl: accepted, Notarized Developer ID PASS
A11 iOS archive and export Scripts/ios-archive.sh archive, export succeeded; IPA 18 MB, Apple Distribution, aps-environment: production, no App Group entitlement PASS
A12 Shortcut clashes Scripts/check-shortcuts.py 58 menu shortcuts, 0 clash PASS
A13 Localisation Scripts/make-strings.py --sync 1 348 + 4 + 68 keys, all with English PASS
A14 Push pipeline end to end in development: the app's registration reaches the backend, the backend's payloads reach a fake APNs server; same payloads delivered with simctl push: banner, actions, « mark as read » changes server state (iPhone), tap opens the conversation (iPad) manual + self-test observed PASS (simulator)
A15 Anything on a physical device: passkey sign-in with Face ID / Touch ID, biometric lock, real APNs, notification service extension running, widgets on a Home or Lock Screen, share extension from a real app, Handoff, background refresh timing, haptics, VoiceOver and Dynamic Type passes, Low Data / Low Power behaviour — — NOT TESTED
A16 The signed Release app launched; macOS widgets, share-menu extension, Services, Dock menu, menu bar extra interaction, window restoration after a real relaunch — compiled and signed only NOT TESTED
A17 2.0.0 apps against a real 0.3.0 server (capabilities absent) — unit-tested gating only NOT TESTED

The self-test runs against the real dev backend with a scripted AI stub and a fake APNs server: sign-in by development token, sync, compose, real sends between two staging mailboxes, labels, pin ordering, snooze and wake, reminders, operator search and saved searches, unified bulk actions, contacts, attachment gallery and ZIP, dashboard, AI summary and reply suggestion opening the composer without sending, writing assistant, rich text with inline image received with its cid part, lossless drafts, send later and cancel, undo send inside and outside the window, sensitive-recipient confirmation, delivery tracking in the outbox, notification routing and actions, device removal at sign-out, widget snapshot by privacy level, share hand-off, mailto:, lock gating of every window (simulated leave), mailbox removal keeping unsent drafts.

Text editing was exercised through the same delegate paths as typing, not by a person typing; drag and drop, Quick Look in the composer, snippet and template pickers, vCard import/export and « send and remind » are compiled only.

# 4. Failure injection and recovery

Scenario Where Status
Server restart with scheduled/held sends, pending webhooks, queued pushes, due snoozes, rule jobs: nothing lost, nothing done twice BI PASS
APNs 503 then success; 410 → token removed; unconfigured → nothing enqueued BI PASS
Lost HTTP response after the server accepted a submission → not sent twice (fake server, 6 capability sets; and once on the real server) CU, CI PASS
5xx bursts, cannotCalculateChanges, rejected pending operation followed by valid ones, interruption between delta pages CU PASS
Browser offline: shell boots, queued actions replay once, rejected operation reported, sign-out empties caches and queue WE PASS
Mail server unreachable (health, alert after three failures) BU only NOT TESTED end to end
Disk full, certificate expiry, stuck SMTP queue, antivirus down — NOT TESTED
Real network cut on a device (not emulation) — NOT TESTED

# 5. Release artefacts

Artefact State
Backend + web 1.0.0 in production since 2026-11-20 (§7); main = next, tag v1.0.0 on spbgit
macOS app dist/SP Mail.app 2.0.0 (1) and dist/SPMail-2.0.0.dmg (19.4 MB, SHA-256 d56625536a2c8a9ee8129dc120cee907aa5a4954e609d1a775e80b8503800f39) notarised and stapled; installed in /Applications (§7)
iOS app 2.0.0 (1) on App Store Connect, VALID (TestFlight)
Production 1.0.0, https://mail.spboucher.ai/api/health/ready → ok, HTTP/2

# 6. Housekeeping done

40 web-* and 3 api-* leftover test mailboxes purged from stg.spboucher.ai; recette@stg.spboucher.ai kept; the nine real spboucher.ai accounts counted before and after: unchanged.

# 7. Production deployment — 2026-11-20 (owner's instruction: « continue until everything is done and deployed »)

Deployed commit 3b4d6ec (version 1.0.0) on M4M64c, following RELEASE-NEXT.md.

# Step / check Observed Status
D1 Database backup with restore verification db-20261120T192451Z.dump.enc (96 K), « restauration vérifiée » PASS
D2 Mail-store backup on BHS64 edge-20261120T192452Z.tar.zst.enc (52 M), mail server stopped 1 s, active afterwards PASS
D3 git pull --ff-only (46817b4 → 3b4d6ec), pnpm install --frozen-lockfile, pnpm build build ok, service worker with 110 precached files PASS
D4 Restart with the new environment; migrations migration applied for 004, 005, 006, 007; workers started; no level-50 line in the log PASS
D5 Health /api/health/ready → {"ok":true,"version":"1.0.0","mail_server":"reachable"} locally and through https://mail.spboucher.ai; /api/v1/health database ok, workers on PASS
D6 API compatibility with an existing key (read-only MCP key) /api/v1/me 8 addresses; /api/v1/messages returns messages with the new pinned / snoozed / followup fields; openapi.json 200 PASS
D7 Web login page renders in headless Chromium over HTTP/2, no console error other than the expected 401 of /api/me; sw.js and manifest.webmanifest 200 PASS (signed-in screens not checked: needs the owner's passkey)
D8 Real send and receive in production: notifications@ → info@ through the spboucher-site key (one clearly labelled control message in the owner's inbox) sent, then delivered with 250 2.1.5 OK; watcher recorded the reception (watched_messages +1) PASS
D9 Existing webhook consumer (www.spboucher.ai) message.sent and message.delivered delivered, HTTP 200 PASS
D10 Alerts raised by the deploy none (alerts empty) PASS
D11 Push configuration: existing team APNs key Z9M555KPUY probed against api.push.apple.com and the sandbox with a dummy token for both SP Mail topics 400 BadDeviceToken three times — the key and topics are accepted by Apple PASS (authentication only; no notification delivered to a device)
D12 Local AI provider: key created on the MacLustr gateway; real-provider integration test with Anthropic and the local model 9 / 9, including summary, replies, writing and the ignored injection PASS
D13 Semantic search deliberately off (SPMAIL_AI_EMBED_BACKEND=off): the in-process model has not been run on the node and the process is capped at 800 MB NOT ENABLED
D14 Signed-in web session, passkey, real device push, external delivery to several providers — NOT TESTED (owner)

Apple: macOS 2.0.0 (1) notarised (submission fae01265-6381-4b3e-abef-4bfd6d7a65a4), installed in /Applications in place of 1.1.0, spctl accepted, launched for 14 s without crash report and quit cleanly (window content not inspected; 1.1.0 DMG kept in dist/). iOS: 2.0.0 (1) rebuilt from the final commit, uploaded with Scripts/ios-archive.sh upload (« UPLOAD SUCCEEDED », delivery 80816845-e303-43ba-b576-6693d30af30f), state on App Store Connect: VALID (app 6836619195); never installed on a device.

Files main
Menu

Theme