SPB Git

Repository Public

OPERATIONS.md 4.2 KB

# Operations

# Where things are

Item Location
Application (git clone, source of truth) M4M64c:~/apps/sp-message (origin: spbgit sp-message.git)
Processes PM2 sp-message (API, WebSocket, web), sp-message-backup (03:25 daily)
Database PostgreSQL 17, sp_message, local socket/127.0.0.1
Encrypted objects M4M64c:~/sp-message-data/blobs
Backup key M4M64c:~/.config/sp-message/backup.key — copy it off the node; without it backups cannot be read
Backups M4M64c:~/backups/sp-message (14 database archives + mirror of objects)
Manifest M1M32:~/dispatch/apps/sp-message.json (environment, TURN secret)
TURN relay BHS64: coturn, /etc/turnserver.conf (original kept as .dist), ufw 3478 udp/tcp and 49160–49400 udp
Public route Caddy on BHS64, managed by mld

Do not mld move this app: the database and the objects do not follow.

# Operator commands (on M4M64c)

bash
cd ~/apps/sp-message
export PATH=/opt/homebrew/bin:$PATH DATABASE_URL=postgres://127.0.0.1/sp_message SPM_ORIGIN=https://message.spboucher.ai SPM_RP_ID=message.spboucher.ai
node dist/cli.mjs status
node dist/cli.mjs bootstrap --ttl 60            # first administrator, single use
node dist/cli.mjs bootstrap --recover           # additional administrator when all admin passkeys are lost
node dist/cli.mjs invite --for "Name" --hours 72
node dist/cli.mjs revoke-sessions <handle>

# Change and release

bash
ssh M4M64c
cd ~/apps/sp-message && git pull            # or edit here
pnpm install --frozen-lockfile && node scripts/build.mjs && node_modules/.bin/vite build
pm2 restart sp-message                      # migrations run at start; clients reconnect by themselves
git add -A && git commit -m "…" && git push origin main

Run pnpm test and pnpm test:e2e on a machine with a sp_message_test database before a release.

# Health and monitoring

  • GET /api/health/live (process), GET /api/health/ready (database and data directory writable).
  • mld status --live; mld heal restarts the process and the route every 5 minutes if needed.
  • Logs: pm2 logs sp-message (method, path, status; no bodies). Backup log: pm2 logs sp-message-backup.
  • Admin → Overview shows accounts, devices, connections, storage and whether TURN is configured.
  • TURN: ssh BHS64 systemctl status coturn, journalctl -u coturn.

# Backup and recovery

scripts/backup.sh dumps the database, encrypts it (AES-256, PBKDF2, key file), mirrors the objects, then restores the archive into a scratch database and compares row counts; it fails loudly otherwise.

Recovery on a fresh node:

  1. mld deploy sp-message --node <node> (creates an empty database), pm2 stop sp-message.
  2. Copy backup.key to ~/.config/sp-message/, the archive and blobs/ to ~/backups/sp-message/.
  3. dropdb sp_message && createdb sp_message && zsh scripts/restore.sh ~/backups/sp-message/db-<stamp>.dump.enc
  4. pm2 start sp-message.

What a restore brings back: accounts, passkeys, devices, conversations, undelivered ciphertext, files. What it cannot bring back: messages already delivered and purged, and anything that only lived on a lost device. Messages sent between the backup and the failure are lost for devices that had not fetched them.

# Incidents

Situation Action
A person lost one device They remove it in Settings → Devices from another device
A person lost every device, has the recovery key They sign in with their passkey on a new device and choose “Use my recovery key”
A person lost every device and the recovery key Admin → People → Manage → Reset encryption identity; the person enrols again; contacts see a key-change warning and must accept and add them back
A person lost every passkey Their account cannot sign in; disable it and invite them again
Administrator lost every passkey node dist/cli.mjs bootstrap --recover on the node
Suspected compromise of an account Admin → People → Disable (sessions end, devices are removed from all groups)
TURN secret leaked New secret in /etc/turnserver.conf and in the manifest (SPM_TURN_SECRET), restart coturn, mld deploy sp-message
Files main
Menu

Theme