Operations (MacLustr)
Topology
Internet ── TLS (Caddy, Let's Encrypt, HSTS) ── BHS64 51.161.112.61 ── WireGuard wg1 ── node:8740 sp-vault (API + web, PM2)
127.0.0.1:8742 sp-vault-managed (PM2, loopback only)
PostgreSQL 17: sp_vault, sp_vault_managed (local socket/loopback)
~/sp-vault-data/attachments (client-side encrypted chunks)- Deployed and routed by
mldfrom the gateway M1M32; manifestinfrastructure/sp-vault.mld.json(copy toM1M32:~/dispatch/apps/sp-vault.json). The app is pinned after first placement: data does not followmld move. - Source of truth after deployment: the git clone on the node (
~/apps/sp-vault, origingitsrv:srv/git/sp-vault.git). Apple targets (apple/) are built on the laptop. - Environments: production = database
sp_vaulton the node. Development / tests = laptop databasessp_vault_test,sp_vault_e2e_test(wiped at every run), RP IDlocalhost, throwaway keys. No production data ever leaves the node except as encrypted backups. There is no shared staging; a staging instance is a second manifest with another port, database and domain.
Files that matter on the node (all 0600, directory 0700)
| Path | What | If lost |
|---|---|---|
~/.config/sp-vault/backup.key |
Encrypts database dumps | Old backups unreadable (live data unaffected) |
~/.config/sp-vault/managed.key |
Master key of managed secrets | Managed secrets unrecoverable → re-enter them. Personal vaults unaffected |
~/.config/sp-vault/managed.token |
API ↔ managed process token | Regenerate, restart both processes |
Copies of backup.key and managed.key belong off the node (laptop ~/Desktop/Cluster/secrets/sp-vault/), and
managed.key must not sit next to database backups. Neither file gives access to any personal vault.
Routine
mld status --live | grep sp-vault # where it runs, health
ssh <node> 'cd ~/apps/sp-vault && export PATH=/opt/homebrew/bin:$PATH DATABASE_URL=postgres://127.0.0.1/sp_vault SPV_ORIGIN=https://vault.spboucher.ai SPV_RP_ID=vault.spboucher.ai SPV_DATA_DIR=$HOME/sp-vault-data && node dist/cli.mjs status'
…node dist/cli.mjs invite --for "Name" --hours 24 # single-use invitation link (creates an account, never a device)
…node dist/cli.mjs revoke-sessions <handle> # ends server sessions; does not and cannot touch vault keys
ssh <node> 'pm2 logs sp-vault --lines 50 --nostream' # logs: route templates, status, duration — no contentThe operator CLI has no command that reads, resets or recovers a vault. That is deliberate.
Update (remote-first)
On the node: git pull (or edit + commit) → pnpm install --frozen-lockfile && pnpm build → pm2 restart sp-vault sp-vault-managed → curl -fsS localhost:8740/api/health/ready → git push origin main.
Rollout rules: migrations are additive and run inside a transaction with an advisory lock at start-up; a
migration that drops or rewrites data needs a dump first (scripts/backup.sh) and the owner's go-ahead. Envelope
version v/alg never change meaning: a new format is a new number, and clients must read all published ones
before any client writes the new one. Rollback = git checkout <previous tag> && pnpm build && pm2 restart;
additive migrations stay in place.
Backups, RPO / RTO
sp-vault-backup(PM2 cron, 03:40): encryptedpg_dumpof both databases, rsync snapshot of attachments, each dump restored into a scratch database and row-counted before the run is declared good; 30 dumps kept.- RPO 24 h for server-held state with the nightly job. Clients keep a full encrypted copy of what they synced, and users can export self-contained encrypted backups at any time, so a node loss within the RPO window loses at most the changes no surviving device had synced.
- RTO ≈ 1 h:
mld deploy sp-vault --node <other>, copy~/.config/sp-vault, restore dumps (openssl enc -d … | pg_restore -d sp_vault), rsync attachments back,mld tunnel route sp-vault. - Offsite: the nightly job writes to the same node. Copying
~/backups/sp-vaultto another site (already encrypted; BHS64 or the laptop) isscripts/offsite-pull.sh <node> <dest>; scheduling it is an open decision (destination, retention). Until then a disk failure of the node relies on client copies and user backups. - Point-in-time recovery (WAL archiving) is not configured; nightly dumps only.
- Restore drills: record them in
docs/operations/RESTORE-DRILLS.md.
Monitoring
/api/health/ready (checked by mld heal every 5 min). node dist/cli.mjs status gives counts only. No
metrics endpoint exposes per-account or per-item data. Disk: attachments are capped at 100 MiB per file and
5 GiB per account.
Hardening checklist
HSTS preload header, CSP + Trusted Types, no CORS, cookies __Host- + SameSite=Strict + HttpOnly + Secure,
rate limits on every authentication path, request bodies ≤ 2 MiB (chunks 1 MiB), PostgreSQL bound to loopback,
FileVault state of the node (cluster nodes boot unattended with FileVault off — the database volume is not
encrypted at rest; vault content is ciphertext regardless, metadata and managed-secret ciphertext rely on
file permissions and on managed.key being a separate file), macOS updates per mld harden.
Incident playbook
- Node or database stolen / leaked: personal vault content stays encrypted; treat metadata as disclosed;
rotate every managed secret at its issuer; rotate
managed.key; end sessions (delete from sessions). - A user device lost unlocked: user revokes it (Devices) and runs Account rekey, then changes key passwords.
- Suspected malicious web bundle: stop the route (
mld tunnel rm sp-vault), comparedist/webhashes with a clean build of the tagged commit, tell users to use native apps and rekey. - Agent misbehaving: kill switch (Connect & Agents), then revoke its identity and rotate the secrets it used.