Operations
Where things are
| Item | Location |
|---|---|
| Application (git clone, source of truth) | M4M64c:~/apps/sp-message (origin: spbgit sp-message.git) |
| Processes | PM2 sp-message (API, WebSocket, web), sp-message-backup (03:25 daily) |
| Database | PostgreSQL 17, sp_message, local socket/127.0.0.1 |
| Encrypted objects | M4M64c:~/sp-message-data/blobs |
| Backup key | M4M64c:~/.config/sp-message/backup.key — copy it off the node; without it backups cannot be read |
| Backups | M4M64c:~/backups/sp-message (14 database archives + mirror of objects) |
| Manifest | M1M32:~/dispatch/apps/sp-message.json (environment, TURN secret) |
| TURN relay | BHS64: coturn, /etc/turnserver.conf (original kept as .dist), ufw 3478 udp/tcp and 49160–49400 udp |
| Public route | Caddy on BHS64, managed by mld |
Do not mld move this app: the database and the objects do not follow.
Operator commands (on M4M64c)
cd ~/apps/sp-message
export PATH=/opt/homebrew/bin:$PATH DATABASE_URL=postgres://127.0.0.1/sp_message SPM_ORIGIN=https://message.spboucher.ai SPM_RP_ID=message.spboucher.ai
node dist/cli.mjs status
node dist/cli.mjs bootstrap --ttl 60 # first administrator, single use
node dist/cli.mjs bootstrap --recover # additional administrator when all admin passkeys are lost
node dist/cli.mjs invite --for "Name" --hours 72
node dist/cli.mjs revoke-sessions <handle>Change and release
ssh M4M64c
cd ~/apps/sp-message && git pull # or edit here
pnpm install --frozen-lockfile && node scripts/build.mjs && node_modules/.bin/vite build
pm2 restart sp-message # migrations run at start; clients reconnect by themselves
git add -A && git commit -m "…" && git push origin mainRun pnpm test and pnpm test:e2e on a machine with a sp_message_test database before a release.
Health and monitoring
GET /api/health/live(process),GET /api/health/ready(database and data directory writable).mld status --live;mld healrestarts the process and the route every 5 minutes if needed.- Logs:
pm2 logs sp-message(method, path, status; no bodies). Backup log:pm2 logs sp-message-backup. - Admin → Overview shows accounts, devices, connections, storage and whether TURN is configured.
- TURN:
ssh BHS64 systemctl status coturn,journalctl -u coturn.
Backup and recovery
scripts/backup.sh dumps the database, encrypts it (AES-256, PBKDF2, key file), mirrors the objects, then restores
the archive into a scratch database and compares row counts; it fails loudly otherwise.
Recovery on a fresh node:
mld deploy sp-message --node <node>(creates an empty database),pm2 stop sp-message.- Copy
backup.keyto~/.config/sp-message/, the archive andblobs/to~/backups/sp-message/. dropdb sp_message && createdb sp_message && zsh scripts/restore.sh ~/backups/sp-message/db-<stamp>.dump.encpm2 start sp-message.
What a restore brings back: accounts, passkeys, devices, conversations, undelivered ciphertext, files. What it cannot bring back: messages already delivered and purged, and anything that only lived on a lost device. Messages sent between the backup and the failure are lost for devices that had not fetched them.
Incidents
| Situation | Action |
|---|---|
| A person lost one device | They remove it in Settings → Devices from another device |
| A person lost every device, has the recovery key | They sign in with their passkey on a new device and choose “Use my recovery key” |
| A person lost every device and the recovery key | Admin → People → Manage → Reset encryption identity; the person enrols again; contacts see a key-change warning and must accept and add them back |
| A person lost every passkey | Their account cannot sign in; disable it and invite them again |
| Administrator lost every passkey | node dist/cli.mjs bootstrap --recover on the node |
| Suspected compromise of an account | Admin → People → Disable (sessions end, devices are removed from all groups) |
| TURN secret leaked | New secret in /etc/turnserver.conf and in the manifest (SPM_TURN_SECRET), restart coturn, mld deploy sp-message |