SP Mail Next — operations: health, alerts, devices, trends
Status: implemented on branch next (2026-11-20). Contract: ARCHITECTURE-NEXT.md §4.11. Code: server/src/next/ops.ts, tables alerts and usage_snapshots (migration 006_intelligence_ops.sql). These routes complement the existing console (/api/admin/overview, /metrics, /audit, /dns); they read the same sources and do not replace them. The shell monitor on BHS64 (infra/monitoring/sp-mail-monitor.sh) stays: it watches the mail server from the outside, this watches the platform from the backend.
All routes require an administrator session. Revoking someone's session or push device also requires a passkey re-authentication (< 5 min) and is audited.
1. GET /api/admin/health[?fresh=1]
{ "status": "ok|warn|fail", "generated_at": "…",
"checks": { "<name>": { "status": "ok|warn|fail|unknown", "detail": "human sentence", "checked_at": "…", "data": { … } } } }unknown means "could not be determined from here" (it never worsens the overall status). Results are cached per check; fresh=1 re-runs everything (the certificate probe can take up to 5 s).
| Check | What it does | Status rules | Cache |
|---|---|---|---|
database |
select 1, latency |
ok | 30 s |
mail_server |
Stalwart management API (x:Domain/get), latency |
fail when unreachable; warn above 3 s | 30 s |
queue |
x:QueuedMessage: size, age of the oldest message, failing messages (envelopes only) |
fail when a message is failing or the oldest is > 60 min; warn above 50 queued; unknown when the server is unreachable | 30 s |
disk |
statfs of SPMAIL_OPS_DISK_PATH (default: the server's working directory) on the backend node |
warn < 10 % free, fail < 5 % | 60 s |
certificates |
TLS expiry of the web origin (443), STARTTLS on SPMAIL_MX_HOST:25, and Stalwart's certificate store |
fail when expired, warn < 14 days. The SMTP probe only works from the server side: a connection failure or timeout is unknown, never fail. A non-https origin (tests) is unknown |
30 min |
backups |
SPMAIL_BACKUP_STATUS file ({ at, ok, error }, written by infra/backup/pull-edge.sh) |
fail when the last run failed or the file is unreadable; warn when older than 36 h, fail beyond 72 h; unknown when not configured | 60 s |
dns |
the existing live DNS report (MX, SPF, DKIM, DMARC, MX host, PTR) for every hosted domain | warn when something is not aligned | 15 min |
workers |
heartbeats of the Next workers (rules engine, semantic indexer, ops monitor, usage snapshot, AI cache purge), rule and semantic queue depth, outbound messages stuck in queued > 10 min, watcher idle time |
warn when a worker missed 3 intervals, the rules queue is > 10 min behind or an outbound message is stuck; warn when SPMAIL_WORKERS=off |
15 s |
webhooks |
failed deliveries in 24 h, pending backlog | warn ≥ 1 failed, fail ≥ SPMAIL_OPS_WEBHOOK_FAIL_THRESHOLD (5) |
60 s |
push |
APNs configured? pending / sent / failed in 24 h from push_outbox |
ok with "not configured" when the three APNs variables are absent; warn when failures ≥ successes; unknown when the push tables are absent | 60 s |
The heartbeat registry is in memory (beat(name, intervalMs) in ops.ts), fed by the Next workers; the historical workers are observed through their effects (stuck outbound, watcher idle time, webhook backlog) rather than instrumented.
2. Alerts
A monitor worker runs every SPMAIL_OPS_MONITOR_INTERVAL_MS (60 s; SPMAIL_OPS_MONITOR=off disables it, the health route keeps working). Each round it evaluates the conditions below. One open alert per condition key: a partial unique index on alerts(key) where state <> 'resolved' enforces it. While the condition lasts the row's last_seen, count, severity and detail are updated. When the condition is evaluated and no longer holds, the alert becomes resolved. A check that could not run (unknown) neither raises nor resolves anything.
| Key | Condition | Severity |
|---|---|---|
mail_server |
management API unreachable on 3 consecutive rounds | fail |
queue |
a queued message is failing, or the oldest is older than 1 h | fail |
disk |
free space < 10 % / < 5 % | warn / fail |
cert:<name> |
certificate expires in < 14 days / has expired (web:<host>, smtp:<mx>, stalwart:<names>) |
warn / fail |
backup |
last backup failed, status file unreadable, or last success older than 36 h | warn (36–72 h) / fail |
auth_anomaly |
≥ 10 login.failed + reauth.failed in audit_log within 10 minutes |
warn |
delivery |
≥ 20 % bounced or failed among ≥ 10 outbound messages created in the last hour | fail |
webhooks |
≥ 5 webhook deliveries failed definitively in 24 h | warn |
Notification. On first raise, and at most once every SPMAIL_OPS_RENOTIFY_HOURS (12) while the alert stays open, every active administrator receives:
- a push
kind: "alert"on their devices (throughenqueueUserPush; time-sensitive forfail), when the push pipeline is configured; - an internal message in the Inbox of their primary mailbox: an RFC 5322 message built by the backend (
FromandTo= the mailbox itself,Auto-Submitted: auto-generated,X-SPMail-Alert: <key>), uploaded as a blob and filed with JMAPEmail/import. No SMTP, noEmailSubmission, no outbound row: an alert cannot leave the platform.
If neither channel could deliver to anyone (mail server down and no push), the notification is retried on the next round. Each notification writes alert.notified to audit_log (alert, severity, admins, messages, pushes); resolution writes alert.resolved.
Acknowledging (POST /api/admin/alerts/:id/ack) stops reminders; the alert stays listed (state acked) until the condition clears. If the condition comes back after a resolution, a new alert row is created and notified.
Routes: GET /api/admin/alerts?state=open|all&limit= (open = open + acked), POST /api/admin/alerts/:id/ack (409 not_open when already acked or resolved).
Side effect to know: the internal message is a real message in the mailbox, so the inbound watcher treats it like any received mail (webhook message.received, push, the administrator's own rules). draft_reply rules skip it (Auto-Submitted).
3. Devices
GET /api/admin/devices → { sessions: [{ id, user_id, handle, kind, device_name, platform, ip, user_agent, created_at, last_seen_at, expires_at, current }], push_devices: [{ …columns of push_devices except the token, handle, token_tail }] }. The APNs token is never returned (last 6 characters only).
DELETE /api/admin/devices/sessions/:id— revokes the session immediately and removes its push tokens. Requires re-authentication (403 reauth_required), audited asadmin.session.revoke.DELETE /api/admin/devices/push/:id— removes one push device. Same requirements, audited asadmin.push.remove.
A user's own devices remain managed by the existing /api/sessions and /api/next/push/devices routes.
4. Trends and security events
GET /api/admin/trends?metric=storage|deliveries|api&days=30[&refresh=1]storage: per day, total and per mailboxused_bytes(fromusage_snapshots).deliveries: per dayreceived, sent, delivered, bounced, failed(fromusage_snapshots).api: per dayrequests, errors, rate_limited, avg_ms(live fromapi_usage, kept 90 days). A worker writes today's snapshot (UTC day) every hour; the route refreshes it when it is older than 10 minutes or onrefresh=1. History starts on the day this version is deployed; there is no backfill.
GET /api/admin/security-events?limit=&before=&action=&hours=— the security-relevant part ofaudit_log(logins and failures, re-authentications, enrolments, passkeys, sessions, API keys, approvals, mailbox grants, webhook allow-list, AI consent and provider changes, alerts, retention), newest first, withnext_beforefor paging andfailed_24hcounters.actionis a prefix filter (login,alert,apikey…).
5. Configuration
| Variable | Default | Purpose |
|---|---|---|
SPMAIL_OPS_MONITOR |
on |
off disables the alert monitor |
SPMAIL_OPS_MONITOR_INTERVAL_MS |
60000 |
monitor period |
SPMAIL_OPS_DISK_PATH |
working directory | volume to watch (set it to the data volume of the backend node) |
SPMAIL_OPS_DISK_WARN_PCT / _FAIL_PCT |
10 / 5 |
free-space thresholds |
SPMAIL_OPS_CERT_WARN_DAYS |
14 |
certificate warning window |
SPMAIL_OPS_BACKUP_MAX_AGE_HOURS |
36 |
backup staleness (twice this = fail) |
SPMAIL_OPS_RENOTIFY_HOURS |
12 |
reminder interval for an open alert |
SPMAIL_OPS_WEBHOOK_FAIL_THRESHOLD |
5 |
failed webhook deliveries in 24 h |
SPMAIL_BACKUP_STATUS (existing) |
— | path of the backup status file; without it the backup check is unknown and never alerts |
Production notes: point SPMAIL_BACKUP_STATUS at the status file that pull-edge.sh writes on the node that pulls the backups (if that is not the backend node, copy or mount it — otherwise the backup alert stays silent). The administrator's primary mailbox is the first mailbox of that user by position; alerts land there.
6. Tests
tests/integration/next-ops.test.ts (real server, PostgreSQL, staging mail server): health shape and real values, a backup alert raised by making the status file stale, deduplication across monitor rounds, the internal message read back from the administrator's Inbox through JMAP, escalation, 12 h reminder, acknowledgement, resolution, re-raise, authentication anomaly, devices and re-authentication, trends, security events. Unit tests (server/src/next/ops.test.ts): queue statistics, backup, disk and certificate thresholds, alert message format.
Not covered by an executed test: the queue, disk, cert:*, delivery, webhooks and mail_server alert conditions (their threshold functions are unit-tested; the conditions themselves were not provoked), the STARTTLS probe against a reachable port 25, and push delivery of an alert to a real device.