SPB Git

Repository Public

OPERATIONS-NEXT.md 10.0 KB

# SP Mail Next — operations: health, alerts, devices, trends

Status: implemented on branch next (2026-11-20). Contract: ARCHITECTURE-NEXT.md §4.11. Code: server/src/next/ops.ts, tables alerts and usage_snapshots (migration 006_intelligence_ops.sql). These routes complement the existing console (/api/admin/overview, /metrics, /audit, /dns); they read the same sources and do not replace them. The shell monitor on BHS64 (infra/monitoring/sp-mail-monitor.sh) stays: it watches the mail server from the outside, this watches the platform from the backend.

All routes require an administrator session. Revoking someone's session or push device also requires a passkey re-authentication (< 5 min) and is audited.

# 1. GET /api/admin/health[?fresh=1]

json
{ "status": "ok|warn|fail", "generated_at": "…",
  "checks": { "<name>": { "status": "ok|warn|fail|unknown", "detail": "human sentence", "checked_at": "…", "data": { … } } } }

unknown means "could not be determined from here" (it never worsens the overall status). Results are cached per check; fresh=1 re-runs everything (the certificate probe can take up to 5 s).

Check What it does Status rules Cache
database select 1, latency ok 30 s
mail_server Stalwart management API (x:Domain/get), latency fail when unreachable; warn above 3 s 30 s
queue x:QueuedMessage: size, age of the oldest message, failing messages (envelopes only) fail when a message is failing or the oldest is > 60 min; warn above 50 queued; unknown when the server is unreachable 30 s
disk statfs of SPMAIL_OPS_DISK_PATH (default: the server's working directory) on the backend node warn < 10 % free, fail < 5 % 60 s
certificates TLS expiry of the web origin (443), STARTTLS on SPMAIL_MX_HOST:25, and Stalwart's certificate store fail when expired, warn < 14 days. The SMTP probe only works from the server side: a connection failure or timeout is unknown, never fail. A non-https origin (tests) is unknown 30 min
backups SPMAIL_BACKUP_STATUS file ({ at, ok, error }, written by infra/backup/pull-edge.sh) fail when the last run failed or the file is unreadable; warn when older than 36 h, fail beyond 72 h; unknown when not configured 60 s
dns the existing live DNS report (MX, SPF, DKIM, DMARC, MX host, PTR) for every hosted domain warn when something is not aligned 15 min
workers heartbeats of the Next workers (rules engine, semantic indexer, ops monitor, usage snapshot, AI cache purge), rule and semantic queue depth, outbound messages stuck in queued > 10 min, watcher idle time warn when a worker missed 3 intervals, the rules queue is > 10 min behind or an outbound message is stuck; warn when SPMAIL_WORKERS=off 15 s
webhooks failed deliveries in 24 h, pending backlog warn ≥ 1 failed, fail ≥ SPMAIL_OPS_WEBHOOK_FAIL_THRESHOLD (5) 60 s
push APNs configured? pending / sent / failed in 24 h from push_outbox ok with "not configured" when the three APNs variables are absent; warn when failures ≥ successes; unknown when the push tables are absent 60 s

The heartbeat registry is in memory (beat(name, intervalMs) in ops.ts), fed by the Next workers; the historical workers are observed through their effects (stuck outbound, watcher idle time, webhook backlog) rather than instrumented.

# 2. Alerts

A monitor worker runs every SPMAIL_OPS_MONITOR_INTERVAL_MS (60 s; SPMAIL_OPS_MONITOR=off disables it, the health route keeps working). Each round it evaluates the conditions below. One open alert per condition key: a partial unique index on alerts(key) where state <> 'resolved' enforces it. While the condition lasts the row's last_seen, count, severity and detail are updated. When the condition is evaluated and no longer holds, the alert becomes resolved. A check that could not run (unknown) neither raises nor resolves anything.

Key Condition Severity
mail_server management API unreachable on 3 consecutive rounds fail
queue a queued message is failing, or the oldest is older than 1 h fail
disk free space < 10 % / < 5 % warn / fail
cert:<name> certificate expires in < 14 days / has expired (web:<host>, smtp:<mx>, stalwart:<names>) warn / fail
backup last backup failed, status file unreadable, or last success older than 36 h warn (36–72 h) / fail
auth_anomaly ≥ 10 login.failed + reauth.failed in audit_log within 10 minutes warn
delivery ≥ 20 % bounced or failed among ≥ 10 outbound messages created in the last hour fail
webhooks ≥ 5 webhook deliveries failed definitively in 24 h warn

Notification. On first raise, and at most once every SPMAIL_OPS_RENOTIFY_HOURS (12) while the alert stays open, every active administrator receives:

  1. a push kind: "alert" on their devices (through enqueueUserPush; time-sensitive for fail), when the push pipeline is configured;
  2. an internal message in the Inbox of their primary mailbox: an RFC 5322 message built by the backend (From and To = the mailbox itself, Auto-Submitted: auto-generated, X-SPMail-Alert: <key>), uploaded as a blob and filed with JMAP Email/import. No SMTP, no EmailSubmission, no outbound row: an alert cannot leave the platform.

If neither channel could deliver to anyone (mail server down and no push), the notification is retried on the next round. Each notification writes alert.notified to audit_log (alert, severity, admins, messages, pushes); resolution writes alert.resolved.

Acknowledging (POST /api/admin/alerts/:id/ack) stops reminders; the alert stays listed (state acked) until the condition clears. If the condition comes back after a resolution, a new alert row is created and notified.

Routes: GET /api/admin/alerts?state=open|all&limit= (open = open + acked), POST /api/admin/alerts/:id/ack (409 not_open when already acked or resolved).

Side effect to know: the internal message is a real message in the mailbox, so the inbound watcher treats it like any received mail (webhook message.received, push, the administrator's own rules). draft_reply rules skip it (Auto-Submitted).

# 3. Devices

GET /api/admin/devices → { sessions: [{ id, user_id, handle, kind, device_name, platform, ip, user_agent, created_at, last_seen_at, expires_at, current }], push_devices: [{ …columns of push_devices except the token, handle, token_tail }] }. The APNs token is never returned (last 6 characters only).

  • DELETE /api/admin/devices/sessions/:id — revokes the session immediately and removes its push tokens. Requires re-authentication (403 reauth_required), audited as admin.session.revoke.
  • DELETE /api/admin/devices/push/:id — removes one push device. Same requirements, audited as admin.push.remove.

A user's own devices remain managed by the existing /api/sessions and /api/next/push/devices routes.

  • GET /api/admin/trends?metric=storage|deliveries|api&days=30[&refresh=1]
    • storage: per day, total and per mailbox used_bytes (from usage_snapshots).
    • deliveries: per day received, sent, delivered, bounced, failed (from usage_snapshots).
    • api: per day requests, errors, rate_limited, avg_ms (live from api_usage, kept 90 days). A worker writes today's snapshot (UTC day) every hour; the route refreshes it when it is older than 10 minutes or on refresh=1. History starts on the day this version is deployed; there is no backfill.
  • GET /api/admin/security-events?limit=&before=&action=&hours= — the security-relevant part of audit_log (logins and failures, re-authentications, enrolments, passkeys, sessions, API keys, approvals, mailbox grants, webhook allow-list, AI consent and provider changes, alerts, retention), newest first, with next_before for paging and failed_24h counters. action is a prefix filter (login, alert, apikey…).

# 5. Configuration

Variable Default Purpose
SPMAIL_OPS_MONITOR on off disables the alert monitor
SPMAIL_OPS_MONITOR_INTERVAL_MS 60000 monitor period
SPMAIL_OPS_DISK_PATH working directory volume to watch (set it to the data volume of the backend node)
SPMAIL_OPS_DISK_WARN_PCT / _FAIL_PCT 10 / 5 free-space thresholds
SPMAIL_OPS_CERT_WARN_DAYS 14 certificate warning window
SPMAIL_OPS_BACKUP_MAX_AGE_HOURS 36 backup staleness (twice this = fail)
SPMAIL_OPS_RENOTIFY_HOURS 12 reminder interval for an open alert
SPMAIL_OPS_WEBHOOK_FAIL_THRESHOLD 5 failed webhook deliveries in 24 h
SPMAIL_BACKUP_STATUS (existing) — path of the backup status file; without it the backup check is unknown and never alerts

Production notes: point SPMAIL_BACKUP_STATUS at the status file that pull-edge.sh writes on the node that pulls the backups (if that is not the backend node, copy or mount it — otherwise the backup alert stays silent). The administrator's primary mailbox is the first mailbox of that user by position; alerts land there.

# 6. Tests

tests/integration/next-ops.test.ts (real server, PostgreSQL, staging mail server): health shape and real values, a backup alert raised by making the status file stale, deduplication across monitor rounds, the internal message read back from the administrator's Inbox through JMAP, escalation, 12 h reminder, acknowledgement, resolution, re-raise, authentication anomaly, devices and re-authentication, trends, security events. Unit tests (server/src/next/ops.test.ts): queue statistics, backup, disk and certificate thresholds, alert message format.

Not covered by an executed test: the queue, disk, cert:*, delivery, webhooks and mail_server alert conditions (their threshold functions are unit-tested; the conditions themselves were not provoked), the STARTTLS probe against a reachable port 25, and push delivery of an alert to a real device.

Files main
Menu

Theme