SPB Git

Repository Public

ARCHITECTURE-NEXT.md 22.4 KB 164 lines markdown

1# SP Mail Next — architecture changes and decisions23Status: living document, started 2026-11-20 on branch `next` (both repositories). SP Mail Next is an evolution of version 0.3.0 (backend/web) and 1.1.0 (4) (Apple apps). Nothing in `docs/architecture.md` is replaced: Stalwart on BHS64, the Fastify gateway on M4M64c, PostgreSQL, standard JMAP through the gateway, passkeys, `/api/v1`, SDKs, MCP and webhooks all stay as they are.45## 1. Principles kept671. **Clients speak standard JMAP to the gateway** for mail state (read/unread, flags, moves, deletion, drafts). This remains the single source of truth for message state: there is no parallel copy of mail in PostgreSQL.82. **Everything that JMAP does not model lives once, on the server, in PostgreSQL**, behind `/api/next/*`: push devices and preferences, snoozes, reminders, labels, saved searches, snippets, signatures, contacts, rules, AI settings, alerts. The three clients read and write the same rows, so business rules are not reimplemented per platform.93. **Additive only.** Migrations `004`–`006` add tables and nullable columns; no existing column, route, scope, webhook event or SDK call changes meaning. A 0.3.0 client keeps working against a Next server, and `GET /api/next/capabilities` lets a newer client detect an older server.104. **Sending still goes through the existing authorisation model.** Sender identity = an address row the user may send from (`addresses.can_send`, grants). No new path can set an arbitrary `From`.115. **No custom cryptography.** Existing primitives only: WebAuthn, AES-256-GCM sealing with the master key (server) or a Keychain key (Apple), SHA-256 token hashes, ES256 JWT for APNs.1213## 2. New server modules (`server/src/next/`)1415| Module | Owns | Tables (migration) |16|---|---|---|17| `hooks.ts` | In-transaction hooks of the inbound watcher (`workers/inbound.ts`): modules enqueue work in the same transaction that advances the JMAP state, so nothing is lost or duplicated across a crash | — |18| `push.ts`, `apns.ts` | APNs token lifecycle, per-device and per-user preferences, durable event-to-push queue with dedup, retry and metrics | `push_devices` (extended), `push_prefs`, `push_outbox` (004) |19| `snooze.ts` | Snooze and follow-up reminders with durable server state and a due-date worker | `snoozes`, `reminders` (004) |20| `organize.ts` | Labels, saved searches / smart folders, snippets, signatures | `labels`, `saved_searches`, `snippets`, `signatures` (005) |21| `contacts.ts` | Unified contacts: harvested correspondents + user-managed profiles, groups, vCard import/export, duplicates | `contacts`, `contact_emails`, `contact_groups` (005) |22| `search.ts` | One query language for the three clients, search across all mailboxes, attachment index | — (JMAP queries) |23| `dashboard.ts` | Command Center aggregates | — |24| `ai.ts`, `ai-providers.ts`, `semantic.ts` | Optional intelligence layer (off by default), provider abstraction (local MacLustr / Anthropic), consent, semantic index | `ai_settings`, `ai_usage`, `mail_embeddings` (006) |25| `rules.ts` | Post-delivery automation engine with idempotent runs and audit trail | `rules`, `rule_runs` (006) |26| `ops.ts` | Monitoring checks, alerts, storage trends, device management for administrators | `alerts`, `usage_snapshots` (006) |2728All `/api/next/*` routes take a **device session** (web cookie or native bearer) — never an API key. Programmatic access stays on `/api/v1` with scopes. Errors use the historical envelope `{"error":"code","message":"…"}`.2930## 3. Decisions3132### D-N1 — Message state stays in JMAP; labels, pin, snooze are keywords33- Pin = keyword `$pinned` (already used by the web client). Follow-up flag = `$followup`. Snoozed = `$snoozed` + the message is moved to a per-mailbox folder named `Snoozed` (no role). Labels = keywords `lbl_<8 hex>`; the label's name and colour live in `labels` (user level, valid in every mailbox of the user).34- Why: keywords synchronise through `Email/changes` like read/flag state, work offline through the existing pending-operation queues, and need no new sync channel.3536### D-N2 — Snooze and reminders are server-side and durable37A snooze is a row with a due date; a worker moves the message back to the Inbox, clears `$snoozed`, marks it unread and enqueues a push. Closing every client does not lose a snooze. Reminders (`remind me if nobody replied by…`) are evaluated on the server against the thread at the due date.3839### D-N3 — Undo send is a real server-side hold40`POST /api/v1/messages/send` gains `undo_seconds` (0–120). The message is stored as a scheduled outbound row with `send_at = now + undo_seconds`; nothing reaches Stalwart's queue before that. Undo = the existing `DELETE /api/v1/messages/scheduled/{id}`, which destroys the stored draft; before cancelling, the client reads the held content back (`GET /api/next/compose/undo/:id`) so it can reopen the composer with nothing lost. After the window the message is submitted by the existing dispatcher. No client-side timer is trusted.4142### D-N4 — Push: privacy by default, one pipeline43The inbound watcher is the only producer. For each received message that lands in an Inbox, a hook inserts one `push_outbox` row per eligible device, keyed by `(device, mailbox, email)` (dedup). A worker signs an ES256 provider token (cached 50 min), posts to APNs over HTTP/2, retries with backoff on 429/5xx/network errors, and deletes tokens on `410 Unregistered` / `BadDeviceToken`. Payload content is governed by the user's `preview` preference: `none` (generic text), `sender` (default: sender name only) or `full` (sender + subject + preview). The payload always carries opaque ids (`mb`, `email`, `thread`) so the app can fetch over its authenticated session. Badge = unread Inbox total across the user's mailboxes, computed at send time; a silent push updates the badge when mail is read elsewhere. Quiet hours and priority senders are evaluated on the server.4445APNs credentials are configuration (`SPMAIL_APNS_KEY_FILE`, `SPMAIL_APNS_KEY_ID`, `SPMAIL_APNS_TEAM_ID`); without them the pipeline is disabled and says so in `/api/next/push/status` — it never pretends to deliver.4647### D-N5 — Contacts live in PostgreSQL, not in Stalwart's per-account address books48A user owns several mail accounts (8 today); JMAP Contacts are per account. A user-level store gives one address book for all identities and all three clients. Correspondents are harvested by the watcher hook (received) and by sends; a harvested correspondent is only a *suggestion* until the user saves it.4950### D-N6 — One search language, parsed on the server51`GET /api/next/search` accepts the operator syntax (`from: to: cc: subject: body: has:attachment is:unread is:starred is:pinned label: in: before: after: larger: smaller: filename:` plus `-negation`, quoted phrases and `OR`) and translates it to JMAP filters across the selected mailboxes. Clients keep JMAP for the folder lists; search, smart folders and the attachment gallery use this route, so the three clients cannot diverge.5253### D-N7 — Intelligence is optional, consented, and treats mail as untrusted data54`ai_settings.provider` is `off` by default. `local` = the MacLustr LLM gateway (OpenAI-compatible, on the LAN side of the cluster): mail content does not leave the owner's infrastructure. `anthropic` requires a recorded explicit consent (`consent_external_at`) and every response states which provider processed the content. Mail content is passed as delimited data with a system instruction that it must never be followed; the model has no tools; outputs are text or schema-validated JSON; nothing is ever sent, moved or deleted by a model — replies are inserted into the composer for review.5556### D-N8 — Semantic search without new infrastructure57Embeddings are computed by a small multilingual model running inside the backend process's host (local, no third party) and stored as `real[]` in `mail_embeddings`; ranking is brute-force cosine in the backend, which is adequate for a personal installation (tens of thousands of messages) and avoids a PostgreSQL extension on the node. Hybrid search = JMAP text results ∪ semantic neighbours, re-ranked. Disabled unless the user enables indexing.5859### D-N9 — Rules complement Sieve, they do not duplicate it60Sieve (existing filter UI) runs at delivery inside Stalwart: file, forward, vacation. The rules engine runs after delivery from the watcher hook and does what Sieve cannot: labels, snooze/reminder, internal webhook event, push to another SP application, AI categorisation, draft reply. A rule can never send mail: `draft_reply` creates a draft. Runs are idempotent (`rule_runs` unique on rule + mailbox + email) and audited.6162### D-N10 — Apple extensions share state through an App Group63Widgets, the share extension and notification actions need the session token and a small snapshot. The apps gain the App Group `group.ai.spboucher.spmail` and a shared keychain access group. The mail cache itself stays in the app container: extensions read a sealed snapshot and call the API; they never open `mail.sqlite`.6465## 4. `/api/next` contract6667Conventions: JSON; timestamps ISO 8601 UTC; ids are UUIDs unless noted; `mailbox_id` is the SP Mail mailbox UUID (`/api/me`), `email_id` and `thread_id` are the JMAP ids of that mailbox. Lists return `{ "data": [...] }`. Unknown mailbox or a mailbox of another user → 404.6869### 4.1 Capabilities70`GET /api/next/capabilities` → `{ version, features: string[] }`.7172### 4.2 Push73| Route | Body → response |74|---|---|75| `POST /api/push/register` (existing, extended) | `{ apns_token, environment, bundle_id, device_name?, locale?, app_version? }` → `{ ok, device_id }`. Re-registration with the same token updates the row and re-enables it |76| `GET /api/next/push/devices` | → `{ data: [{ id, session_id, device_name, bundle_id, environment, enabled, current, created_at, last_registered_at, last_push_at, last_error }] }` |77| `PATCH /api/next/push/devices/:id` | `{ enabled?: boolean, prefs?: { mailboxes?: { [mailbox_id]: boolean }, sound?: boolean } }` |78| `DELETE /api/next/push/devices/:id` | removes the token |79| `GET /api/next/push/prefs` · `PUT` | `{ enabled, preview: "none"\|"sender"\|"full", quiet_hours: { start: "22:00", end: "07:00", tz: "America/Toronto" } \| null, priority_senders: string[] (addresses or `@domain`, bypass quiet hours), priority_only: boolean, mailboxes: { [mailbox_id]: boolean }, badge: "inbox"\|"none" }` |80| `GET /api/next/push/status` | → `{ configured, environment_hosts, queue: { pending, failed_24h, sent_24h }, last_error }` |81| `POST /api/next/push/test` | `{ device_id? }` → enqueues a test notification to the caller's devices → `{ queued }` |8283APNs payload (alert): `{ aps: { alert: { title, subtitle?, body }, badge, sound?, "thread-id": "<mailbox_id>:<thread_id>", category: "MAIL_NEW", "mutable-content": 1, "interruption-level": "active"|"time-sensitive" }, mb, email, thread, kind: "new" }`. Silent badge update: `{ aps: { "content-available": 1, badge }, kind: "badge" }`. Snooze/reminder due: `category: "MAIL_REMINDER"`, `kind: "snooze"|"reminder"`.8485### 4.3 Snooze and reminders86| Route | Body → response |87|---|---|88| `POST /api/next/snoozes` | `{ mailbox_id, email_ids: string[] (≤ 200), until }` → `{ data: Snooze[] }`. Moves each message to `Snoozed`, sets `$snoozed` |89| `GET /api/next/snoozes?status=active` | → `{ data: [{ id, mailbox_id, email_id, thread_id, subject, from, until, status: "active"\|"done"\|"cancelled"\|"missing", created_at }] }` |90| `PATCH /api/next/snoozes/:id` | `{ until }` |91| `DELETE /api/next/snoozes/:id` | wakes the message now (back to its origin folder, default Inbox) |92| `POST /api/next/reminders` | `{ mailbox_id, email_id, due_at, only_if_no_reply?: boolean, note?: string }` → `Reminder` |93| `GET /api/next/reminders?status=pending\|due\|done` | → `{ data: [{ id, mailbox_id, email_id, thread_id, subject, counterpart, due_at, only_if_no_reply, note, status: "pending"\|"due"\|"done"\|"resolved"\|"cancelled", fired_at }] }` |94| `PATCH /api/next/reminders/:id` | `{ due_at?, note?, status?: "done" }` |95| `DELETE /api/next/reminders/:id` | cancel |9697When a reminder fires: status `due`, keyword `$followup` set on the message, push `kind: "reminder"`. `only_if_no_reply`: if the thread received a message from someone else after the reminder was created, it becomes `resolved` silently.9899### 4.4 Organisation100| Resource | Routes | Shape |101|---|---|---|102| Labels | `GET/POST /api/next/labels`, `PATCH/DELETE /api/next/labels/:id` | `{ id, name (≤ 40), color ("#rrggbb"), keyword ("lbl_…", server-assigned, immutable), position }`. Deleting a label does not touch messages; clients ignore unknown `lbl_` keywords |103| Saved searches (smart folders) | `GET/POST /api/next/searches`, `PATCH/DELETE …/:id` | `{ id, name, query (operator syntax), mailbox_ids: string[] \| null (null = all), icon?, color?, pinned: boolean, position, notify: boolean }` |104| Snippets | `GET/POST /api/next/snippets`, `PATCH/DELETE …/:id` | `{ id, name, shortcut? (≤ 24, unique per user, typed as `;shortcut`), subject?, html, text, updated_at }` — HTML sanitised with the compose allow-list |105| Signatures | `GET/POST /api/next/signatures`, `PATCH/DELETE …/:id` | `{ id, name, html, text, default_for: string[] (address ids), default_for_replies: string[] }`. The per-address signature of 0.3.0 is exposed as the read-only entry `source: "address"` and stays the default until a signature claims the address |106107### 4.5 Contacts108| Route | Notes |109|---|---|110| `GET /api/next/contacts?q=&group=&favorite=1&kind=saved\|suggested\|all&limit=&cursor=` | → `{ data: Contact[], next_cursor }` |111| `GET /api/next/contacts/suggest?q=&limit=8` | autocomplete: ranked by saved > favourite > frequency × recency → `{ data: [{ name, email, contact_id?, favorite, score }] }` |112| `POST /api/next/contacts` · `PATCH /api/next/contacts/:id` · `DELETE` | `Contact = { id, kind: "saved"\|"suggested", name, emails: [{ email, label? }], organization?, title?, phones?: [{ number, label? }], notes?, favorite, vip, groups: string[] (group ids), avatar_color, stats: { sent, received, last_at } }` |113| `GET /api/next/contacts/:id/history?limit=20` | recent messages exchanged with the contact across the user's mailboxes + pending reminders |114| `GET/POST /api/next/contact-groups`, `PATCH/DELETE …/:id` | `{ id, name, color }` |115| `POST /api/next/contacts/import` (text/vcard, ≤ 2 MB) · `GET /api/next/contacts/export.vcf` | vCard 3.0/4.0 |116| `GET /api/next/contacts/duplicates` · `POST /api/next/contacts/merge { keep, merge: string[] }` | duplicate = same normalised e-mail or same normalised name |117118`vip` contacts are implicitly priority senders for push.119120### 4.6 Search and attachments121| Route | Notes |122|---|---|123| `GET /api/next/search?q=&mailboxes=all\|id,id&limit=50&cursor=&threads=1` | → `{ data: MessageSummary[], next_cursor, parsed: { terms, filters, errors } }`. `MessageSummary = { mailbox_id, email_id, thread_id, from, to, subject, preview, received_at, keywords, mailbox_ids, has_attachment, size, folder_roles }` |124| `POST /api/next/search/parse { q }` | → the parsed form (for chips and validation) |125| `GET /api/next/attachments?mailboxes=&kind=image\|pdf\|document\|spreadsheet\|archive\|audio\|video\|other&min_size=&max_size=&q=&from=&after=&before=&limit=&cursor=` | → `{ data: [{ mailbox_id, email_id, thread_id, blob_id, part_id, name, type, size, kind, from, subject, received_at, download_url, inline_ok }] }` |126| `POST /api/next/attachments/zip { items: [{ mailbox_id, email_id, blob_id, name }] (≤ 50, ≤ 200 MB) }` | streams a ZIP |127128### 4.7 Sending129`POST /api/v1/messages/send` gains `undo_seconds`, `remind_at` + `remind_if_no_reply` (creates a reminder once the message is submitted) and `confirm_sensitive`. `PUT /api/next/compose/prefs` holds `{ undo_seconds (default 10), sensitive: string[] (addresses or `@domain`), warn_missing_attachment: boolean, warn_external: boolean }`. If a recipient matches `sensitive` and `confirm_sensitive` is not true → `409 confirmation_required` with the matching recipients; nothing is queued. `GET /api/next/compose/undo/:id` returns the held message content for reopening the composer. `POST /api/next/compose/undo/:id` is the undo itself: it cancels the held message while **keeping the draft and its attachment blobs** in Drafts and returns the content (a plain `DELETE …/scheduled/{id}` destroys the draft, and its attachments with it); it answers 409 if the dispatcher has already taken the message.130131### 4.8 Command Center132`GET /api/next/dashboard` → `{ inbox: [{ mailbox_id, address, unread, total }], attention: { reminders_due, flagged, snoozed_today, unanswered }, scheduled: Outbound[], pending_replies: MessageSummary[], recent_attachments: Attachment[], top_contacts: Contact[], storage: [{ mailbox_id, used_bytes, quota_bytes }], delivery_issues: Outbound[], sync: { watcher_lag_ms, mail_server: "reachable"|"unreachable", push: "configured"|"disabled" }, generated_at }`. Widget layout is a client preference stored in `users.prefs.dashboard`.133134`partial: string[]` names the blocks that could not be computed (response keys; `attention.<counter>` pinpoints one counter): their values are fallbacks, not measurements — clients show “unavailable”, never a zero. A partial result is cached 2 s instead of 15 s. `scheduled[]` and `delivery_issues[]` carry `subject` (read from the stored draft / message; `null` when it is gone). `pending_replies` holds at most 10 messages (30 threads scanned per mailbox; `attention.unanswered` counts the scanned ones).135136`PUT /api/next/mailboxes/order { ids }` — `ids` is the complete list of the mailboxes the user OWNS, in the wanted order (each exactly once, else `400 invalid_request`; a shared, foreign or unknown mailbox → `404 not_found`). Writes `mailboxes.position`; `/api/me` and every mailbox list follow it. → `{ ok: true }`.137138### 4.9 Intelligence139| Route | Notes |140|---|---|141| `GET/PUT /api/next/ai/settings` | `{ provider: "off"\|"local"\|"anthropic", consent_external_at, features: { summary, reply, write, triage, semantic }, language: "auto"\|"fr"\|"en", available: { local: boolean, anthropic: boolean } }`. Switching to `anthropic` requires `{ consent_external: true }` in the same request |142| `POST /api/next/ai/summarize { mailbox_id, thread_id, language?, refresh? }` (`refresh: true` bypasses the 24 h cache; rate-limited like any call) | → `{ summary, decisions[], action_items[{ text, owner?, due? }], deadlines[{ text, date? }], timeline[{ at, who, what }], provider, model }` |143| `POST /api/next/ai/replies { mailbox_id, email_id, tone: "concise"\|"professional"\|"detailed", language? }` | → `{ suggestions: [{ subject?, text }], provider, model }` |144| `POST /api/next/ai/write { action: "improve"\|"grammar"\|"shorten"\|"expand"\|"tone"\|"translate"\|"subject", text, tone?, target_language?, context? }` | → `{ text \| subjects[], provider, model }` |145| `POST /api/next/ai/triage { mailbox_id, email_ids (≤ 25) }` | → `{ data: [{ email_id, category: "personal"\|"work"\|"newsletter"\|"notification"\|"transaction"\|"social"\|"other", needs_action, important, reason }] }`; cached in keywords-free server cache, never written to the message |146| `POST /api/next/ai/search { q, mailboxes?, limit? }` | hybrid semantic + keyword → same shape as `/api/next/search` (including `folder_roles`) plus `score`, `match` |147| `GET /api/next/ai/index/status` · `POST /api/next/ai/index/rebuild` | semantic index coverage; when unavailable, `reason` (English diagnostic) and a stable `reason_code`: `disabled_on_server`, `endpoint_not_configured`, `endpoint_failed`, `model_not_installed`, `backend_failed`, `no_backend` |148149Every AI route answers `409 ai_disabled` when the provider is `off` or the feature is not enabled, and `412 consent_required` when an external provider lacks consent.150151### 4.10 Rules152`GET/POST /api/next/rules`, `PATCH/DELETE …/:id`, `GET …/:id/runs`, `POST /api/next/rules/test { rule, mailbox_id, email_id }` (dry run). `Rule = { id, name, enabled, position, mailbox_ids: string[] | null, match: "all"|"any", conditions: [{ field: "from"|"to"|"subject"|"list"|"has_attachment"|"is_newsletter"|"ai_category"|"size", op: "contains"|"equals"|"matches_domain"|"is"|"gt", value }], actions: [{ type: "label", label_id } | { type: "star" } | { type: "mark_read" } | { type: "pin" } | { type: "snooze", until_rule } | { type: "remind", after_hours } | { type: "event", name } | { type: "notify_app", app } | { type: "draft_reply", snippet_id | ai_tone } | { type: "extract", schema }], stop: boolean }`.153154A rule only applies to mail received after it became active: `enabled_at` (migration 007) is set at creation and moved by every disabled → enabled transition; the worker skips a message whose `receivedAt` is older (2 s clock-skew tolerance), so a queued job evaluated later never runs a rule created or re-enabled in the meantime. Run rows are `{ id, mailbox_id, email_id, thread_id, status, actions, error, at, finished_at }` (`thread_id`: JMAP thread id within `mailbox_id`, `null` on rows written before migration 007).155156### 4.11 Operations (administrators)157`GET /api/admin/health` (mail server, queue, disk, certificates, backups, DNS, push, workers, webhooks — each `{ status: "ok"|"warn"|"fail", detail, checked_at }`), `GET /api/admin/alerts?state=open|all`, `POST /api/admin/alerts/:id/ack`, `GET /api/admin/devices` (sessions and push devices of all users; revocation requires re-authentication), `GET /api/admin/trends?metric=storage|deliveries|api&days=30`, `GET /api/admin/security-events`. Alerts are raised by a monitor worker (queue failures, low disk, certificate expiry < 14 days, backup older than 36 h or failed, authentication anomalies, delivery degradation) and delivered as an internal message to the admin mailbox and a push to administrators' devices; one alert per condition until it clears.158159## 5. Compatibility and rollback160161- Every migration is additive and wrapped in a transaction by the existing runner. Rolling the code back to 0.3.0 leaves unused tables behind and works unchanged (the 0.3.0 server ignores them). The only modified shared code paths are `workers/inbound.ts` (extra properties requested, hooks called inside the existing transaction) and optional fields on two existing routes.162- If a hook throws, the watcher transaction rolls back and the batch is retried on the next tick: a broken module delays webhooks instead of losing mail events, so hooks must be small and are covered by tests.163- Apple apps: the App Group migration moves nothing destructive — the mail cache stays in place; only the token is copied to the shared keychain group, the original entry being kept for one version.164
Files main
Menu

Theme