Repository Public
1# Operations23## Where things are45| Item | Location |6|---|---|7| Application (git clone, source of truth) | `M4M64c:~/apps/sp-message` (origin: spbgit `sp-message.git`) |8| Processes | PM2 `sp-message` (API, WebSocket, web), `sp-message-backup` (03:25 daily) |9| Database | PostgreSQL 17, `sp_message`, local socket/127.0.0.1 |10| Encrypted objects | `M4M64c:~/sp-message-data/blobs` |11| Backup key | `M4M64c:~/.config/sp-message/backup.key` — copy it off the node; without it backups cannot be read |12| Backups | `M4M64c:~/backups/sp-message` (14 database archives + mirror of objects) |13| Manifest | `M1M32:~/dispatch/apps/sp-message.json` (environment, TURN secret) |14| TURN relay | BHS64: `coturn`, `/etc/turnserver.conf` (original kept as `.dist`), ufw 3478 udp/tcp and 49160–49400 udp |15| Public route | Caddy on BHS64, managed by `mld` |1617Do not `mld move` this app: the database and the objects do not follow.1819## Operator commands (on M4M64c)2021```bash22cd ~/apps/sp-message23export PATH=/opt/homebrew/bin:$PATH DATABASE_URL=postgres://127.0.0.1/sp_message SPM_ORIGIN=https://message.spboucher.ai SPM_RP_ID=message.spboucher.ai24node dist/cli.mjs status25node dist/cli.mjs bootstrap --ttl 60 # first administrator, single use26node dist/cli.mjs bootstrap --recover # additional administrator when all admin passkeys are lost27node dist/cli.mjs invite --for "Name" --hours 7228node dist/cli.mjs revoke-sessions <handle>29```3031## Change and release3233```bash34ssh M4M64c35cd ~/apps/sp-message && git pull # or edit here36pnpm install --frozen-lockfile && node scripts/build.mjs && node_modules/.bin/vite build37pm2 restart sp-message # migrations run at start; clients reconnect by themselves38git add -A && git commit -m "…" && git push origin main39```4041Run `pnpm test` and `pnpm test:e2e` on a machine with a `sp_message_test` database before a release.4243## Health and monitoring4445- `GET /api/health/live` (process), `GET /api/health/ready` (database and data directory writable).46- `mld status --live`; `mld heal` restarts the process and the route every 5 minutes if needed.47- Logs: `pm2 logs sp-message` (method, path, status; no bodies). Backup log: `pm2 logs sp-message-backup`.48- Admin → Overview shows accounts, devices, connections, storage and whether TURN is configured.49- TURN: `ssh BHS64 systemctl status coturn`, `journalctl -u coturn`.5051## Backup and recovery5253`scripts/backup.sh` dumps the database, encrypts it (AES-256, PBKDF2, key file), mirrors the objects, then **restores54the archive into a scratch database and compares row counts**; it fails loudly otherwise.5556Recovery on a fresh node:57581. `mld deploy sp-message --node <node>` (creates an empty database), `pm2 stop sp-message`.592. Copy `backup.key` to `~/.config/sp-message/`, the archive and `blobs/` to `~/backups/sp-message/`.603. `dropdb sp_message && createdb sp_message && zsh scripts/restore.sh ~/backups/sp-message/db-<stamp>.dump.enc`614. `pm2 start sp-message`.6263What a restore brings back: accounts, passkeys, devices, conversations, undelivered ciphertext, files. What it cannot64bring back: messages already delivered and purged, and anything that only lived on a lost device. Messages sent65between the backup and the failure are lost for devices that had not fetched them.6667## Incidents6869| Situation | Action |70|---|---|71| A person lost one device | They remove it in Settings → Devices from another device |72| A person lost every device, has the recovery key | They sign in with their passkey on a new device and choose “Use my recovery key” |73| A person lost every device and the recovery key | Admin → People → Manage → *Reset encryption identity*; the person enrols again; contacts see a key-change warning and must accept and add them back |74| A person lost every passkey | Their account cannot sign in; disable it and invite them again |75| Administrator lost every passkey | `node dist/cli.mjs bootstrap --recover` on the node |76| Suspected compromise of an account | Admin → People → Disable (sessions end, devices are removed from all groups) |77| TURN secret leaked | New secret in `/etc/turnserver.conf` and in the manifest (`SPM_TURN_SECRET`), restart coturn, `mld deploy sp-message` |78