# Operations

For the operator of skills.spboucher.ai. Commands that touch production need the operator's approval each time.

Deployed on 2026-11-09: node M3U96b, directory ~/apps/skills-spboucher-ai, port 8570, PM2 processes skills-web and skills-maint (daily 04:50: cleanup + backup), database skills_spboucher (role skills_app), data in ~/data/skills-spboucher-ai, backups in ~/backups/skills-spboucher-ai replicated to M3U96a. Manifest copy: infra/mld/skills-spboucher-ai.json. Confirm the current node with mld status.

# Topology

Internet ── TLS ──> Caddy on the gateway (BHS64) ── WireGuard ──> node:8570  Next.js (PM2)
                                                                     │
                                                                     ├── PostgreSQL 17 (local socket)
                                                                     └── DATA_DIR/{blobs,staging}  (outside the repo)

One process serves the UI, /api/v1 and /mcp. DNS: A skills.spboucher.ai -> 51.161.112.61 (gateway). The route to the node is set by mld deploy.

# Environment

.env in the application directory on the node, mode 600, never committed. A copy belongs in the operator's secrets folder, nowhere else.

SKILLS_DISABLE_RATE_LIMIT=1 exists for tests and is ignored when NODE_ENV=production.

# Deploying with mld

First deployment (creates the app; afterwards the node clone is the source of truth):

# 1. manifest on the gateway: M1M32:~/dispatch/apps/skills-spboucher-ai.json  (copy kept in infra/)
# 2. stage and deploy
mld stage ~/Desktop/Cluster/projets/skills-spboucher-ai skills-spboucher-ai
mld deploy skills-spboucher-ai
mld adopt skills-spboucher-ai
mld status --live

The manifest's post_sync hook checks that .env exists, runs pnpm install --frozen-lockfile, pnpm db:migrate, creates DATA_DIR, and builds Next.js. The health check is GET /api/v1/health/ready.

Later changes are made on the node:

ssh <node>
cd ~/apps/skills-spboucher-ai
# edit, then
pnpm typecheck && pnpm test          # tests create and drop skills_test_* databases; they never touch the production database
pnpm db:migrate && pnpm build && pm2 restart skills-web
git add -A && git commit -m "…" && git push origin main

Never run destructive experiments against the production database or DATA_DIR. Use a scratch database and directory.

# Database migrations

• pnpm db:migrate applies new files from packages/db/migrations in order, each in a transaction, and records a checksum.
• An applied file must not be edited; the migrator aborts on a checksum mismatch.
• Migrations are forward-only. Rollback strategy: take a pg_dump before migrating; to undo, either write a compensating migration or restore the dump.

# Bootstrap (once)

Run on the node, in an interactive terminal:

cd ~/apps/skills-spboucher-ai
pnpm -s admin:bootstrap --email <owner email> --name "<display name>" --namespace <owner namespace> --password-out ~/skills-owner-initial-password.txt

• Creates the single platform owner and their namespace (marked identity-verified), and writes an audit event without the password.
• Refuses to run if an owner or administrator already exists.
• Refuses to run without a TTY unless --non-interactive is passed by a trusted deploy script.
• --password-out writes the generated initial password to a new file with mode 600 (it fails if the file exists). Without it, the password is printed once on stdout.
• The password must be changed at first sign-in; until then the session cannot call the API. Move the file to the secrets folder or delete it after the first sign-in.

Then:

pnpm -s admin collections seed     # creates the eight editorial collections, empty
pnpm -s admin status

# The Claude Code publisher token

Issue it on the node and pipe it straight into the workstation's credential store, so it is never displayed, pasted or written to a shell history:

ssh <node> 'cd ~/apps/skills-spboucher-ai && pnpm -s admin token create \
  --user <owner email> --name claude-code-publisher \
  --scopes artifacts:read,artifacts:create,artifacts:update,uploads:create,releases:publish,releases:update \
  --namespace <owner namespace> --expires-in 90d' | skills login --token-stdin
skills whoami

• The token goes to stdout; the metadata (id, prefix, scopes, expiry) goes to stderr, so the pipe carries only the secret.
• Omitting --scopes uses exactly that default publisher set.
• Add --ip <CIDR> to pin the token to an address range.
• skills login stores it in the macOS Keychain (service skills-registry, account = registry host). Claude Code uses it by running the CLI; it never needs to read the value.
• For a one-off process-scoped use without storing: SKILLS_TOKEN="$(…)" skills publish … in a subshell. Do not export it in a profile or write it to an env file.

The publisher token cannot revoke releases, deprecate artifacts, manage tokens, read the audit trail or moderate. For those, use the dashboard, or a separate short-lived token:

# elevated: administrators only, at most 24 hours
pnpm -s admin token create --user <owner email> --name moderation-2026-11-09 --scopes admin:moderate --expires-in 2h --token-out ~/mod.token

# Rotation and revocation

skills auth token rotate                  # rotates the stored credential; the new token is stored, never printed
skills auth token revoke <id|prefix>      # needs tokens:write unless it is the current token
# from the server, for any token:
pnpm -s admin token list --user <owner email>
pnpm -s admin token rotate --id <token id> | skills login --token-stdin
pnpm -s admin token revoke --id <token id> --reason "laptop replaced"

Rotation revokes the old token in the same transaction. Rotate at least every 90 days (the default expiry forces it) and immediately after any suspicion of exposure. An expired token cannot be rotated; issue a new one.

# Account recovery

pnpm -s admin recover --email <email> --password-out ~/recovery.txt

Sets a temporary password that must be changed at next sign-in and revokes all of that user's sessions. API tokens are not touched; revoke them separately if the account was compromised.

Creating a contributor account:

pnpm -s admin user create --email <email> --name "<name>" --namespace <namespace> --password-out ~/contrib.txt
pnpm -s admin verify-namespace --name <namespace>            # after checking who they are
pnpm -s admin verify-namespace --name <namespace> --revoke

Send the initial password over a channel you trust. No email is sent by the system.

# Backups and restore drill

infra/backup.sh (run it as pnpm backup; restore drill: pnpm restore:drill): a compressed pg_dump of the database plus a copy of DATA_DIR/blobs, written to a backup directory on the node and replicated to a second node. Schedule it daily.

What must be backed up: the database, DATA_DIR/blobs, and .env (separately, in the secrets folder). DATA_DIR/staging is disposable.

Restore drill (do it once after deploy and after any change to the backup script):

createdb skills_restore_drill
gunzip -c <backup>/skills-<date>.sql.gz | psql skills_restore_drill
psql skills_restore_drill -c "select count(*) from releases"
# verify blobs against the database
psql -At skills_restore_drill -c "select sha256, archive_object_key from releases" | while IFS='|' read -r sha key; do
  echo "$sha  <backup>/blobs/$key" | shasum -a 256 -c - >/dev/null || echo "MISMATCH $key"
done
dropdb skills_restore_drill

Every blob's path contains its SHA-256, so integrity can be checked at any time without the database: shasum -a 256 <file> must equal the file name.

Blobs are immutable and deduplicated; there is no lifecycle expiry. Revoked releases keep their blob so the record stays verifiable; removing bytes for a legal reason is a manual, logged operator action.

# Monitoring

• Liveness: GET /api/v1/health/live -> {"status":"ok"}.
• Readiness: GET /api/v1/health/ready -> 200 with database and storage true, or 503.
• mld heal on the gateway restarts the PM2 process and re-points the route if the health check fails.
• Errors: unexpected exceptions are logged as one JSON line to stderr (pm2 logs skills-web) with requestId, route and stack. The client receives only the request id, which is also in the X-Request-Id response header. Expected errors (4xx) are not logged; security-relevant ones are in the audit table.
• pnpm -s admin status prints counts of users, artifacts, releases, active tokens, open reports, quarantined releases and audit events.
• Useful queries:

-- denied actions in the last day
select at, actor_label, action, target_label, ip, detail from audit_events where result = 'denied' and at > now() - interval '1 day' order by at desc;
-- publishing outcomes in the last week
select action, count(*) from audit_events where action in ('release.publish','release.quarantine','release.reject') and at > now() - interval '7 days' group by 1;

External uptime checks and alerting are Planned.

# Housekeeping

pnpm -s admin cleanup

Expires upload sessions that are more than an hour past their deadline and deletes their staged bytes; deletes idempotency keys older than two days, sessions expired for more than 30 days, and download-dedupe hashes older than two days. Schedule it daily. It never touches releases, blobs or audit events.

# Incident response

For every incident: write down times and actions as you go, quote request ids, and keep the audit rows (they cannot be deleted anyway).

# Leaked or stolen API token

1. Revoke it: pnpm -s admin token revoke --id <id> --reason "leaked" (find the id with token list, or by prefix in the audit log).
2. Review what it did: select * from audit_events where actor_token_id = '<id>' order by at;
3. For every release it published that the owner does not recognise: revoke the release with a public reason (dashboard, or POST /releases/{id}/revoke).
4. Issue a replacement token. Find out how it leaked before reusing the same storage.
5. If the leak could include .env: rotate TOKEN_PEPPER (this invalidates all tokens and sessions), restart, re-issue tokens.

# Malicious or compromised release

1. Revoke the version with a clear public reason. Downloads return 410 from that moment and latest moves to the previous good version.
2. If the whole artifact is hostile: take it down (POST /admin/artifacts/{id}/moderate with action: "takedown", confirm: true).
3. Check the publisher's other artifacts and tokens; revoke tokens, and set the user's status to suspended in the database if the account is hostile.
4. Resolve the related report with a note.
5. If users may have installed it, say so publicly on the artifact page (revocation reason) and by direct contact where possible. Security advisories as a product feature are Planned; until then the revocation reason is the public record.

# Compromised human account

1. pnpm -s admin recover --email <email> (revokes sessions, forces a new password).
2. Revoke all of the user's tokens.
3. Review their audit trail and the releases published in the suspect window.

# Takedown or licence request

1. Record the request as a report (category license or impersonation) if the requester did not.
2. Decide; if granted, take the artifact down or revoke the version with a reason. Reply to the requester.
3. The publisher may appeal by email to the security contact; restoring is action: "restore".

# Host or database compromise

Stop the process, preserve the disk state, rotate TOKEN_PEPPER and the database credentials, restore from the last trusted backup, verify every blob against its file name, and re-issue credentials. Compare releases.sha256 from the backup with production before reopening.

# Logs and retention

Security contact: contact@spboucher.ai.