SP Mail Next — architecture changes and decisions
Status: living document, started 2026-11-20 on branch next (both repositories). SP Mail Next is an evolution of version 0.3.0 (backend/web) and 1.1.0 (4) (Apple apps). Nothing in docs/architecture.md is replaced: Stalwart on BHS64, the Fastify gateway on M4M64c, PostgreSQL, standard JMAP through the gateway, passkeys, /api/v1, SDKs, MCP and webhooks all stay as they are.
1. Principles kept
- Clients speak standard JMAP to the gateway for mail state (read/unread, flags, moves, deletion, drafts). This remains the single source of truth for message state: there is no parallel copy of mail in PostgreSQL.
- Everything that JMAP does not model lives once, on the server, in PostgreSQL, behind
/api/next/*: push devices and preferences, snoozes, reminders, labels, saved searches, snippets, signatures, contacts, rules, AI settings, alerts. The three clients read and write the same rows, so business rules are not reimplemented per platform. - Additive only. Migrations
004–006add tables and nullable columns; no existing column, route, scope, webhook event or SDK call changes meaning. A 0.3.0 client keeps working against a Next server, andGET /api/next/capabilitieslets a newer client detect an older server. - Sending still goes through the existing authorisation model. Sender identity = an address row the user may send from (
addresses.can_send, grants). No new path can set an arbitraryFrom. - No custom cryptography. Existing primitives only: WebAuthn, AES-256-GCM sealing with the master key (server) or a Keychain key (Apple), SHA-256 token hashes, ES256 JWT for APNs.
2. New server modules (server/src/next/)
| Module | Owns | Tables (migration) |
|---|---|---|
hooks.ts |
In-transaction hooks of the inbound watcher (workers/inbound.ts): modules enqueue work in the same transaction that advances the JMAP state, so nothing is lost or duplicated across a crash |
— |
push.ts, apns.ts |
APNs token lifecycle, per-device and per-user preferences, durable event-to-push queue with dedup, retry and metrics | push_devices (extended), push_prefs, push_outbox (004) |
snooze.ts |
Snooze and follow-up reminders with durable server state and a due-date worker | snoozes, reminders (004) |
organize.ts |
Labels, saved searches / smart folders, snippets, signatures | labels, saved_searches, snippets, signatures (005) |
contacts.ts |
Unified contacts: harvested correspondents + user-managed profiles, groups, vCard import/export, duplicates | contacts, contact_emails, contact_groups (005) |
search.ts |
One query language for the three clients, search across all mailboxes, attachment index | — (JMAP queries) |
dashboard.ts |
Command Center aggregates | — |
ai.ts, ai-providers.ts, semantic.ts |
Optional intelligence layer (off by default), provider abstraction (local MacLustr / Anthropic), consent, semantic index | ai_settings, ai_usage, mail_embeddings (006) |
rules.ts |
Post-delivery automation engine with idempotent runs and audit trail | rules, rule_runs (006) |
ops.ts |
Monitoring checks, alerts, storage trends, device management for administrators | alerts, usage_snapshots (006) |
All /api/next/* routes take a device session (web cookie or native bearer) — never an API key. Programmatic access stays on /api/v1 with scopes. Errors use the historical envelope {"error":"code","message":"…"}.
3. Decisions
D-N1 — Message state stays in JMAP; labels, pin, snooze are keywords
- Pin = keyword
$pinned(already used by the web client). Follow-up flag =$followup. Snoozed =$snoozed+ the message is moved to a per-mailbox folder namedSnoozed(no role). Labels = keywordslbl_<8 hex>; the label's name and colour live inlabels(user level, valid in every mailbox of the user). - Why: keywords synchronise through
Email/changeslike read/flag state, work offline through the existing pending-operation queues, and need no new sync channel.
D-N2 — Snooze and reminders are server-side and durable
A snooze is a row with a due date; a worker moves the message back to the Inbox, clears $snoozed, marks it unread and enqueues a push. Closing every client does not lose a snooze. Reminders (remind me if nobody replied by…) are evaluated on the server against the thread at the due date.
D-N3 — Undo send is a real server-side hold
POST /api/v1/messages/send gains undo_seconds (0–120). The message is stored as a scheduled outbound row with send_at = now + undo_seconds; nothing reaches Stalwart's queue before that. Undo = the existing DELETE /api/v1/messages/scheduled/{id}, which destroys the stored draft; before cancelling, the client reads the held content back (GET /api/next/compose/undo/:id) so it can reopen the composer with nothing lost. After the window the message is submitted by the existing dispatcher. No client-side timer is trusted.
D-N4 — Push: privacy by default, one pipeline
The inbound watcher is the only producer. For each received message that lands in an Inbox, a hook inserts one push_outbox row per eligible device, keyed by (device, mailbox, email) (dedup). A worker signs an ES256 provider token (cached 50 min), posts to APNs over HTTP/2, retries with backoff on 429/5xx/network errors, and deletes tokens on 410 Unregistered / BadDeviceToken. Payload content is governed by the user's preview preference: none (generic text), sender (default: sender name only) or full (sender + subject + preview). The payload always carries opaque ids (mb, email, thread) so the app can fetch over its authenticated session. Badge = unread Inbox total across the user's mailboxes, computed at send time; a silent push updates the badge when mail is read elsewhere. Quiet hours and priority senders are evaluated on the server.
APNs credentials are configuration (SPMAIL_APNS_KEY_FILE, SPMAIL_APNS_KEY_ID, SPMAIL_APNS_TEAM_ID); without them the pipeline is disabled and says so in /api/next/push/status — it never pretends to deliver.
D-N5 — Contacts live in PostgreSQL, not in Stalwart's per-account address books
A user owns several mail accounts (8 today); JMAP Contacts are per account. A user-level store gives one address book for all identities and all three clients. Correspondents are harvested by the watcher hook (received) and by sends; a harvested correspondent is only a suggestion until the user saves it.
D-N6 — One search language, parsed on the server
GET /api/next/search accepts the operator syntax (from: to: cc: subject: body: has:attachment is:unread is:starred is:pinned label: in: before: after: larger: smaller: filename: plus -negation, quoted phrases and OR) and translates it to JMAP filters across the selected mailboxes. Clients keep JMAP for the folder lists; search, smart folders and the attachment gallery use this route, so the three clients cannot diverge.
D-N7 — Intelligence is optional, consented, and treats mail as untrusted data
ai_settings.provider is off by default. local = the MacLustr LLM gateway (OpenAI-compatible, on the LAN side of the cluster): mail content does not leave the owner's infrastructure. anthropic requires a recorded explicit consent (consent_external_at) and every response states which provider processed the content. Mail content is passed as delimited data with a system instruction that it must never be followed; the model has no tools; outputs are text or schema-validated JSON; nothing is ever sent, moved or deleted by a model — replies are inserted into the composer for review.
D-N8 — Semantic search without new infrastructure
Embeddings are computed by a small multilingual model running inside the backend process's host (local, no third party) and stored as real[] in mail_embeddings; ranking is brute-force cosine in the backend, which is adequate for a personal installation (tens of thousands of messages) and avoids a PostgreSQL extension on the node. Hybrid search = JMAP text results ∪ semantic neighbours, re-ranked. Disabled unless the user enables indexing.
D-N9 — Rules complement Sieve, they do not duplicate it
Sieve (existing filter UI) runs at delivery inside Stalwart: file, forward, vacation. The rules engine runs after delivery from the watcher hook and does what Sieve cannot: labels, snooze/reminder, internal webhook event, push to another SP application, AI categorisation, draft reply. A rule can never send mail: draft_reply creates a draft. Runs are idempotent (rule_runs unique on rule + mailbox + email) and audited.
D-N10 — Apple extensions share state through an App Group
Widgets, the share extension and notification actions need the session token and a small snapshot. The apps gain the App Group group.ai.spboucher.spmail and a shared keychain access group. The mail cache itself stays in the app container: extensions read a sealed snapshot and call the API; they never open mail.sqlite.
4. /api/next contract
Conventions: JSON; timestamps ISO 8601 UTC; ids are UUIDs unless noted; mailbox_id is the SP Mail mailbox UUID (/api/me), email_id and thread_id are the JMAP ids of that mailbox. Lists return { "data": [...] }. Unknown mailbox or a mailbox of another user → 404.
4.1 Capabilities
GET /api/next/capabilities → { version, features: string[] }.
4.2 Push
| Route | Body → response |
|---|---|
POST /api/push/register (existing, extended) |
{ apns_token, environment, bundle_id, device_name?, locale?, app_version? } → { ok, device_id }. Re-registration with the same token updates the row and re-enables it |
GET /api/next/push/devices |
→ { data: [{ id, session_id, device_name, bundle_id, environment, enabled, current, created_at, last_registered_at, last_push_at, last_error }] } |
PATCH /api/next/push/devices/:id |
{ enabled?: boolean, prefs?: { mailboxes?: { [mailbox_id]: boolean }, sound?: boolean } } |
DELETE /api/next/push/devices/:id |
removes the token |
GET /api/next/push/prefs · PUT |
{ enabled, preview: "none"|"sender"|"full", quiet_hours: { start: "22:00", end: "07:00", tz: "America/Toronto" } | null, priority_senders: string[] (addresses or @domain, bypass quiet hours), priority_only: boolean, mailboxes: { [mailbox_id]: boolean }, badge: "inbox"|"none" } |
GET /api/next/push/status |
→ { configured, environment_hosts, queue: { pending, failed_24h, sent_24h }, last_error } |
POST /api/next/push/test |
{ device_id? } → enqueues a test notification to the caller's devices → { queued } |
APNs payload (alert): { aps: { alert: { title, subtitle?, body }, badge, sound?, "thread-id": "<mailbox_id>:<thread_id>", category: "MAIL_NEW", "mutable-content": 1, "interruption-level": "active"|"time-sensitive" }, mb, email, thread, kind: "new" }. Silent badge update: { aps: { "content-available": 1, badge }, kind: "badge" }. Snooze/reminder due: category: "MAIL_REMINDER", kind: "snooze"|"reminder".
4.3 Snooze and reminders
| Route | Body → response |
|---|---|
POST /api/next/snoozes |
{ mailbox_id, email_ids: string[] (≤ 200), until } → { data: Snooze[] }. Moves each message to Snoozed, sets $snoozed |
GET /api/next/snoozes?status=active |
→ { data: [{ id, mailbox_id, email_id, thread_id, subject, from, until, status: "active"|"done"|"cancelled"|"missing", created_at }] } |
PATCH /api/next/snoozes/:id |
{ until } |
DELETE /api/next/snoozes/:id |
wakes the message now (back to its origin folder, default Inbox) |
POST /api/next/reminders |
{ mailbox_id, email_id, due_at, only_if_no_reply?: boolean, note?: string } → Reminder |
GET /api/next/reminders?status=pending|due|done |
→ { data: [{ id, mailbox_id, email_id, thread_id, subject, counterpart, due_at, only_if_no_reply, note, status: "pending"|"due"|"done"|"resolved"|"cancelled", fired_at }] } |
PATCH /api/next/reminders/:id |
{ due_at?, note?, status?: "done" } |
DELETE /api/next/reminders/:id |
cancel |
When a reminder fires: status due, keyword $followup set on the message, push kind: "reminder". only_if_no_reply: if the thread received a message from someone else after the reminder was created, it becomes resolved silently.
4.4 Organisation
| Resource | Routes | Shape |
|---|---|---|
| Labels | GET/POST /api/next/labels, PATCH/DELETE /api/next/labels/:id |
{ id, name (≤ 40), color ("#rrggbb"), keyword ("lbl_…", server-assigned, immutable), position }. Deleting a label does not touch messages; clients ignore unknown lbl_ keywords |
| Saved searches (smart folders) | GET/POST /api/next/searches, PATCH/DELETE …/:id |
{ id, name, query (operator syntax), mailbox_ids: string[] | null (null = all), icon?, color?, pinned: boolean, position, notify: boolean } |
| Snippets | GET/POST /api/next/snippets, PATCH/DELETE …/:id |
{ id, name, shortcut? (≤ 24, unique per user, typed as ;shortcut), subject?, html, text, updated_at } — HTML sanitised with the compose allow-list |
| Signatures | GET/POST /api/next/signatures, PATCH/DELETE …/:id |
{ id, name, html, text, default_for: string[] (address ids), default_for_replies: string[] }. The per-address signature of 0.3.0 is exposed as the read-only entry source: "address" and stays the default until a signature claims the address |
4.5 Contacts
| Route | Notes |
|---|---|
GET /api/next/contacts?q=&group=&favorite=1&kind=saved|suggested|all&limit=&cursor= |
→ { data: Contact[], next_cursor } |
GET /api/next/contacts/suggest?q=&limit=8 |
autocomplete: ranked by saved > favourite > frequency × recency → { data: [{ name, email, contact_id?, favorite, score }] } |
POST /api/next/contacts · PATCH /api/next/contacts/:id · DELETE |
Contact = { id, kind: "saved"|"suggested", name, emails: [{ email, label? }], organization?, title?, phones?: [{ number, label? }], notes?, favorite, vip, groups: string[] (group ids), avatar_color, stats: { sent, received, last_at } } |
GET /api/next/contacts/:id/history?limit=20 |
recent messages exchanged with the contact across the user's mailboxes + pending reminders |
GET/POST /api/next/contact-groups, PATCH/DELETE …/:id |
{ id, name, color } |
POST /api/next/contacts/import (text/vcard, ≤ 2 MB) · GET /api/next/contacts/export.vcf |
vCard 3.0/4.0 |
GET /api/next/contacts/duplicates · POST /api/next/contacts/merge { keep, merge: string[] } |
duplicate = same normalised e-mail or same normalised name |
vip contacts are implicitly priority senders for push.
4.6 Search and attachments
| Route | Notes |
|---|---|
GET /api/next/search?q=&mailboxes=all|id,id&limit=50&cursor=&threads=1 |
→ { data: MessageSummary[], next_cursor, parsed: { terms, filters, errors } }. MessageSummary = { mailbox_id, email_id, thread_id, from, to, subject, preview, received_at, keywords, mailbox_ids, has_attachment, size, folder_roles } |
POST /api/next/search/parse { q } |
→ the parsed form (for chips and validation) |
GET /api/next/attachments?mailboxes=&kind=image|pdf|document|spreadsheet|archive|audio|video|other&min_size=&max_size=&q=&from=&after=&before=&limit=&cursor= |
→ { data: [{ mailbox_id, email_id, thread_id, blob_id, part_id, name, type, size, kind, from, subject, received_at, download_url, inline_ok }] } |
POST /api/next/attachments/zip { items: [{ mailbox_id, email_id, blob_id, name }] (≤ 50, ≤ 200 MB) } |
streams a ZIP |
4.7 Sending
POST /api/v1/messages/send gains undo_seconds, remind_at + remind_if_no_reply (creates a reminder once the message is submitted) and confirm_sensitive. PUT /api/next/compose/prefs holds { undo_seconds (default 10), sensitive: string[] (addresses or @domain), warn_missing_attachment: boolean, warn_external: boolean }. If a recipient matches sensitive and confirm_sensitive is not true → 409 confirmation_required with the matching recipients; nothing is queued. GET /api/next/compose/undo/:id returns the held message content for reopening the composer. POST /api/next/compose/undo/:id is the undo itself: it cancels the held message while keeping the draft and its attachment blobs in Drafts and returns the content (a plain DELETE …/scheduled/{id} destroys the draft, and its attachments with it); it answers 409 if the dispatcher has already taken the message.
4.8 Command Center
GET /api/next/dashboard → { inbox: [{ mailbox_id, address, unread, total }], attention: { reminders_due, flagged, snoozed_today, unanswered }, scheduled: Outbound[], pending_replies: MessageSummary[], recent_attachments: Attachment[], top_contacts: Contact[], storage: [{ mailbox_id, used_bytes, quota_bytes }], delivery_issues: Outbound[], sync: { watcher_lag_ms, mail_server: "reachable"|"unreachable", push: "configured"|"disabled" }, generated_at }. Widget layout is a client preference stored in users.prefs.dashboard.
partial: string[] names the blocks that could not be computed (response keys; attention.<counter> pinpoints one counter): their values are fallbacks, not measurements — clients show “unavailable”, never a zero. A partial result is cached 2 s instead of 15 s. scheduled[] and delivery_issues[] carry subject (read from the stored draft / message; null when it is gone). pending_replies holds at most 10 messages (30 threads scanned per mailbox; attention.unanswered counts the scanned ones).
PUT /api/next/mailboxes/order { ids } — ids is the complete list of the mailboxes the user OWNS, in the wanted order (each exactly once, else 400 invalid_request; a shared, foreign or unknown mailbox → 404 not_found). Writes mailboxes.position; /api/me and every mailbox list follow it. → { ok: true }.
4.9 Intelligence
| Route | Notes |
|---|---|
GET/PUT /api/next/ai/settings |
{ provider: "off"|"local"|"anthropic", consent_external_at, features: { summary, reply, write, triage, semantic }, language: "auto"|"fr"|"en", available: { local: boolean, anthropic: boolean } }. Switching to anthropic requires { consent_external: true } in the same request |
POST /api/next/ai/summarize { mailbox_id, thread_id, language?, refresh? } (refresh: true bypasses the 24 h cache; rate-limited like any call) |
→ { summary, decisions[], action_items[{ text, owner?, due? }], deadlines[{ text, date? }], timeline[{ at, who, what }], provider, model } |
POST /api/next/ai/replies { mailbox_id, email_id, tone: "concise"|"professional"|"detailed", language? } |
→ { suggestions: [{ subject?, text }], provider, model } |
POST /api/next/ai/write { action: "improve"|"grammar"|"shorten"|"expand"|"tone"|"translate"|"subject", text, tone?, target_language?, context? } |
→ { text | subjects[], provider, model } |
POST /api/next/ai/triage { mailbox_id, email_ids (≤ 25) } |
→ { data: [{ email_id, category: "personal"|"work"|"newsletter"|"notification"|"transaction"|"social"|"other", needs_action, important, reason }] }; cached in keywords-free server cache, never written to the message |
POST /api/next/ai/search { q, mailboxes?, limit? } |
hybrid semantic + keyword → same shape as /api/next/search (including folder_roles) plus score, match |
GET /api/next/ai/index/status · POST /api/next/ai/index/rebuild |
semantic index coverage; when unavailable, reason (English diagnostic) and a stable reason_code: disabled_on_server, endpoint_not_configured, endpoint_failed, model_not_installed, backend_failed, no_backend |
Every AI route answers 409 ai_disabled when the provider is off or the feature is not enabled, and 412 consent_required when an external provider lacks consent.
4.10 Rules
GET/POST /api/next/rules, PATCH/DELETE …/:id, GET …/:id/runs, POST /api/next/rules/test { rule, mailbox_id, email_id } (dry run). Rule = { id, name, enabled, position, mailbox_ids: string[] | null, match: "all"|"any", conditions: [{ field: "from"|"to"|"subject"|"list"|"has_attachment"|"is_newsletter"|"ai_category"|"size", op: "contains"|"equals"|"matches_domain"|"is"|"gt", value }], actions: [{ type: "label", label_id } | { type: "star" } | { type: "mark_read" } | { type: "pin" } | { type: "snooze", until_rule } | { type: "remind", after_hours } | { type: "event", name } | { type: "notify_app", app } | { type: "draft_reply", snippet_id | ai_tone } | { type: "extract", schema }], stop: boolean }.
A rule only applies to mail received after it became active: enabled_at (migration 007) is set at creation and moved by every disabled → enabled transition; the worker skips a message whose receivedAt is older (2 s clock-skew tolerance), so a queued job evaluated later never runs a rule created or re-enabled in the meantime. Run rows are { id, mailbox_id, email_id, thread_id, status, actions, error, at, finished_at } (thread_id: JMAP thread id within mailbox_id, null on rows written before migration 007).
4.11 Operations (administrators)
GET /api/admin/health (mail server, queue, disk, certificates, backups, DNS, push, workers, webhooks — each { status: "ok"|"warn"|"fail", detail, checked_at }), GET /api/admin/alerts?state=open|all, POST /api/admin/alerts/:id/ack, GET /api/admin/devices (sessions and push devices of all users; revocation requires re-authentication), GET /api/admin/trends?metric=storage|deliveries|api&days=30, GET /api/admin/security-events. Alerts are raised by a monitor worker (queue failures, low disk, certificate expiry < 14 days, backup older than 36 h or failed, authentication anomalies, delivery degradation) and delivered as an internal message to the admin mailbox and a push to administrators' devices; one alert per condition until it clears.
5. Compatibility and rollback
- Every migration is additive and wrapped in a transaction by the existing runner. Rolling the code back to 0.3.0 leaves unused tables behind and works unchanged (the 0.3.0 server ignores them). The only modified shared code paths are
workers/inbound.ts(extra properties requested, hooks called inside the existing transaction) and optional fields on two existing routes. - If a hook throws, the watcher transaction rolls back and the batch is retried on the next tick: a broken module delays webhooks instead of losing mail events, so hooks must be small and are covered by tests.
- Apple apps: the App Group migration moves nothing destructive — the mail cache stays in place; only the token is copied to the shared keychain group, the original entry being kept for one version.