SPB Git

Repository Public

OPERATIONS-NEXT.md 10.0 KB 98 lines markdown

1# SP Mail Next — operations: health, alerts, devices, trends23Status: implemented on branch `next` (2026-11-20). Contract: `ARCHITECTURE-NEXT.md` §4.11. Code: `server/src/next/ops.ts`, tables `alerts` and `usage_snapshots` (migration `006_intelligence_ops.sql`). These routes complement the existing console (`/api/admin/overview`, `/metrics`, `/audit`, `/dns`); they read the same sources and do not replace them. The shell monitor on BHS64 (`infra/monitoring/sp-mail-monitor.sh`) stays: it watches the mail server from the outside, this watches the platform from the backend.45All routes require an administrator session. Revoking someone's session or push device also requires a passkey re-authentication (< 5 min) and is audited.67## 1. `GET /api/admin/health[?fresh=1]`89```json10{ "status": "ok|warn|fail", "generated_at": "…",11  "checks": { "<name>": { "status": "ok|warn|fail|unknown", "detail": "human sentence", "checked_at": "…", "data": { … } } } }12```1314`unknown` means "could not be determined from here" (it never worsens the overall status). Results are cached per check; `fresh=1` re-runs everything (the certificate probe can take up to 5 s).1516| Check | What it does | Status rules | Cache |17|---|---|---|---|18| `database` | `select 1`, latency | ok | 30 s |19| `mail_server` | Stalwart management API (`x:Domain/get`), latency | fail when unreachable; warn above 3 s | 30 s |20| `queue` | `x:QueuedMessage`: size, age of the oldest message, failing messages (envelopes only) | fail when a message is failing or the oldest is > 60 min; warn above 50 queued; unknown when the server is unreachable | 30 s |21| `disk` | `statfs` of `SPMAIL_OPS_DISK_PATH` (default: the server's working directory) on the backend node | warn < 10 % free, fail < 5 % | 60 s |22| `certificates` | TLS expiry of the web origin (443), STARTTLS on `SPMAIL_MX_HOST:25`, and Stalwart's certificate store | fail when expired, warn < 14 days. The SMTP probe only works from the server side: a connection failure or timeout is `unknown`, never `fail`. A non-https origin (tests) is `unknown` | 30 min |23| `backups` | `SPMAIL_BACKUP_STATUS` file (`{ at, ok, error }`, written by `infra/backup/pull-edge.sh`) | fail when the last run failed or the file is unreadable; warn when older than 36 h, fail beyond 72 h; unknown when not configured | 60 s |24| `dns` | the existing live DNS report (MX, SPF, DKIM, DMARC, MX host, PTR) for every hosted domain | warn when something is not aligned | 15 min |25| `workers` | heartbeats of the Next workers (rules engine, semantic indexer, ops monitor, usage snapshot, AI cache purge), rule and semantic queue depth, outbound messages stuck in `queued` > 10 min, watcher idle time | warn when a worker missed 3 intervals, the rules queue is > 10 min behind or an outbound message is stuck; warn when `SPMAIL_WORKERS=off` | 15 s |26| `webhooks` | failed deliveries in 24 h, pending backlog | warn ≥ 1 failed, fail ≥ `SPMAIL_OPS_WEBHOOK_FAIL_THRESHOLD` (5) | 60 s |27| `push` | APNs configured? pending / sent / failed in 24 h from `push_outbox` | ok with "not configured" when the three APNs variables are absent; warn when failures ≥ successes; unknown when the push tables are absent | 60 s |2829The heartbeat registry is in memory (`beat(name, intervalMs)` in `ops.ts`), fed by the Next workers; the historical workers are observed through their effects (stuck outbound, watcher idle time, webhook backlog) rather than instrumented.3031## 2. Alerts3233A monitor worker runs every `SPMAIL_OPS_MONITOR_INTERVAL_MS` (60 s; `SPMAIL_OPS_MONITOR=off` disables it, the health route keeps working). Each round it evaluates the conditions below. **One open alert per condition key**: a partial unique index on `alerts(key) where state <> 'resolved'` enforces it. While the condition lasts the row's `last_seen`, `count`, `severity` and `detail` are updated. When the condition is evaluated and no longer holds, the alert becomes `resolved`. A check that could not run (`unknown`) neither raises nor resolves anything.3435| Key | Condition | Severity |36|---|---|---|37| `mail_server` | management API unreachable on 3 consecutive rounds | fail |38| `queue` | a queued message is failing, or the oldest is older than 1 h | fail |39| `disk` | free space < 10 % / < 5 % | warn / fail |40| `cert:<name>` | certificate expires in < 14 days / has expired (`web:<host>`, `smtp:<mx>`, `stalwart:<names>`) | warn / fail |41| `backup` | last backup failed, status file unreadable, or last success older than 36 h | warn (36–72 h) / fail |42| `auth_anomaly` | ≥ 10 `login.failed` + `reauth.failed` in `audit_log` within 10 minutes | warn |43| `delivery` | ≥ 20 % bounced or failed among ≥ 10 outbound messages created in the last hour | fail |44| `webhooks` | ≥ 5 webhook deliveries failed definitively in 24 h | warn |4546**Notification.** On first raise, and at most once every `SPMAIL_OPS_RENOTIFY_HOURS` (12) while the alert stays `open`, every active administrator receives:47481. a push `kind: "alert"` on their devices (through `enqueueUserPush`; time-sensitive for `fail`), when the push pipeline is configured;492. an **internal message** in the Inbox of their primary mailbox: an RFC 5322 message built by the backend (`From` and `To` = the mailbox itself, `Auto-Submitted: auto-generated`, `X-SPMail-Alert: <key>`), uploaded as a blob and filed with JMAP `Email/import`. No SMTP, no `EmailSubmission`, no outbound row: an alert cannot leave the platform.5051If neither channel could deliver to anyone (mail server down and no push), the notification is retried on the next round. Each notification writes `alert.notified` to `audit_log` (`alert`, `severity`, `admins`, `messages`, `pushes`); resolution writes `alert.resolved`.5253**Acknowledging** (`POST /api/admin/alerts/:id/ack`) stops reminders; the alert stays listed (state `acked`) until the condition clears. If the condition comes back after a resolution, a new alert row is created and notified.5455Routes: `GET /api/admin/alerts?state=open|all&limit=` (`open` = `open` + `acked`), `POST /api/admin/alerts/:id/ack` (`409 not_open` when already acked or resolved).5657Side effect to know: the internal message is a real message in the mailbox, so the inbound watcher treats it like any received mail (webhook `message.received`, push, the administrator's own rules). `draft_reply` rules skip it (`Auto-Submitted`).5859## 3. Devices6061`GET /api/admin/devices` → `{ sessions: [{ id, user_id, handle, kind, device_name, platform, ip, user_agent, created_at, last_seen_at, expires_at, current }], push_devices: [{ …columns of push_devices except the token, handle, token_tail }] }`. The APNs token is never returned (last 6 characters only).6263- `DELETE /api/admin/devices/sessions/:id` — revokes the session immediately and removes its push tokens. Requires re-authentication (`403 reauth_required`), audited as `admin.session.revoke`.64- `DELETE /api/admin/devices/push/:id` — removes one push device. Same requirements, audited as `admin.push.remove`.6566A user's own devices remain managed by the existing `/api/sessions` and `/api/next/push/devices` routes.6768## 4. Trends and security events6970- `GET /api/admin/trends?metric=storage|deliveries|api&days=30[&refresh=1]`71  - `storage`: per day, total and per mailbox `used_bytes` (from `usage_snapshots`).72  - `deliveries`: per day `received, sent, delivered, bounced, failed` (from `usage_snapshots`).73  - `api`: per day `requests, errors, rate_limited, avg_ms` (live from `api_usage`, kept 90 days).74  A worker writes today's snapshot (UTC day) every hour; the route refreshes it when it is older than 10 minutes or on `refresh=1`. History starts on the day this version is deployed; there is no backfill.75- `GET /api/admin/security-events?limit=&before=&action=&hours=` — the security-relevant part of `audit_log` (logins and failures, re-authentications, enrolments, passkeys, sessions, API keys, approvals, mailbox grants, webhook allow-list, AI consent and provider changes, alerts, retention), newest first, with `next_before` for paging and `failed_24h` counters. `action` is a prefix filter (`login`, `alert`, `apikey`…).7677## 5. Configuration7879| Variable | Default | Purpose |80|---|---|---|81| `SPMAIL_OPS_MONITOR` | `on` | `off` disables the alert monitor |82| `SPMAIL_OPS_MONITOR_INTERVAL_MS` | `60000` | monitor period |83| `SPMAIL_OPS_DISK_PATH` | working directory | volume to watch (set it to the data volume of the backend node) |84| `SPMAIL_OPS_DISK_WARN_PCT` / `_FAIL_PCT` | `10` / `5` | free-space thresholds |85| `SPMAIL_OPS_CERT_WARN_DAYS` | `14` | certificate warning window |86| `SPMAIL_OPS_BACKUP_MAX_AGE_HOURS` | `36` | backup staleness (twice this = fail) |87| `SPMAIL_OPS_RENOTIFY_HOURS` | `12` | reminder interval for an open alert |88| `SPMAIL_OPS_WEBHOOK_FAIL_THRESHOLD` | `5` | failed webhook deliveries in 24 h |89| `SPMAIL_BACKUP_STATUS` (existing) | — | path of the backup status file; without it the backup check is `unknown` and never alerts |9091Production notes: point `SPMAIL_BACKUP_STATUS` at the status file that `pull-edge.sh` writes on the node that pulls the backups (if that is not the backend node, copy or mount it — otherwise the backup alert stays silent). The administrator's *primary mailbox* is the first mailbox of that user by position; alerts land there.9293## 6. Tests9495`tests/integration/next-ops.test.ts` (real server, PostgreSQL, staging mail server): health shape and real values, a backup alert raised by making the status file stale, deduplication across monitor rounds, the internal message read back from the administrator's Inbox through JMAP, escalation, 12 h reminder, acknowledgement, resolution, re-raise, authentication anomaly, devices and re-authentication, trends, security events. Unit tests (`server/src/next/ops.test.ts`): queue statistics, backup, disk and certificate thresholds, alert message format.9697Not covered by an executed test: the `queue`, `disk`, `cert:*`, `delivery`, `webhooks` and `mail_server` alert conditions (their threshold functions are unit-tested; the conditions themselves were not provoked), the STARTTLS probe against a reachable port 25, and push delivery of an alert to a real device.98
Files main
Menu

Theme