Repository Public MIT
1# Operations23For the operator of skills.spboucher.ai. Commands that touch production need the operator's approval each time.45Deployed on 2026-11-09: node **M3U96b**, directory `~/apps/skills-spboucher-ai`, port 8570, PM2 processes `skills-web` and `skills-maint` (daily 04:50: cleanup + backup), database `skills_spboucher` (role `skills_app`), data in `~/data/skills-spboucher-ai`, backups in `~/backups/skills-spboucher-ai` replicated to M3U96a. Manifest copy: `infra/mld/skills-spboucher-ai.json`. Confirm the current node with `mld status`.67## Topology89```text10Internet ── TLS ──> Caddy on the gateway (BHS64) ── WireGuard ──> node:8570 Next.js (PM2)11 │12 ├── PostgreSQL 17 (local socket)13 └── DATA_DIR/{blobs,staging} (outside the repo)14```1516One process serves the UI, `/api/v1` and `/mcp`. DNS: `A skills.spboucher.ai -> 51.161.112.61` (gateway). The route to the node is set by `mld deploy`.1718## Environment1920`.env` in the application directory on the node, mode 600, never committed. A copy belongs in the operator's secrets folder, nowhere else.2122| Variable | Required | Meaning |23|---|---|---|24| `DATABASE_URL` | yes | PostgreSQL connection string. Unix socket form: `postgres://localhost/skills?host=/tmp` |25| `TOKEN_PEPPER` | yes | 32+ random characters (`openssl rand -hex 32`). HMAC key for token and session verifiers. Changing it invalidates every token and session |26| `DATA_DIR` | yes | Absolute directory for `blobs/` and `staging/`, outside the repository |27| `PUBLIC_URL` | yes in production | `https://skills.spboucher.ai`. Used for links and for the same-origin check |28| `PORT` | yes | 8570 |29| `NODE_ENV` | yes | `production` |3031`SKILLS_DISABLE_RATE_LIMIT=1` exists for tests and is ignored when `NODE_ENV=production`.3233## Deploying with mld3435First deployment (creates the app; afterwards the node clone is the source of truth):3637```bash38# 1. manifest on the gateway: M1M32:~/dispatch/apps/skills-spboucher-ai.json (copy kept in infra/)39# 2. stage and deploy40mld stage ~/Desktop/Cluster/projets/skills-spboucher-ai skills-spboucher-ai41mld deploy skills-spboucher-ai42mld adopt skills-spboucher-ai43mld status --live44```4546The manifest's `post_sync` hook checks that `.env` exists, runs `pnpm install --frozen-lockfile`, `pnpm db:migrate`, creates `DATA_DIR`, and builds Next.js. The health check is `GET /api/v1/health/ready`.4748Later changes are made on the node:4950```bash51ssh <node>52cd ~/apps/skills-spboucher-ai53# edit, then54pnpm typecheck && pnpm test # tests create and drop skills_test_* databases; they never touch the production database55pnpm db:migrate && pnpm build && pm2 restart skills-web56git add -A && git commit -m "…" && git push origin main57```5859Never run destructive experiments against the production database or `DATA_DIR`. Use a scratch database and directory.6061## Database migrations6263- `pnpm db:migrate` applies new files from `packages/db/migrations` in order, each in a transaction, and records a checksum.64- An applied file must not be edited; the migrator aborts on a checksum mismatch.65- Migrations are forward-only. Rollback strategy: take a `pg_dump` before migrating; to undo, either write a compensating migration or restore the dump.6667## Bootstrap (once)6869Run on the node, in an interactive terminal:7071```bash72cd ~/apps/skills-spboucher-ai73pnpm -s admin:bootstrap --email <owner email> --name "<display name>" --namespace <owner namespace> --password-out ~/skills-owner-initial-password.txt74```7576- Creates the single platform owner and their namespace (marked identity-verified), and writes an audit event without the password.77- Refuses to run if an owner or administrator already exists.78- Refuses to run without a TTY unless `--non-interactive` is passed by a trusted deploy script.79- `--password-out` writes the generated initial password to a new file with mode 600 (it fails if the file exists). Without it, the password is printed once on stdout.80- The password must be changed at first sign-in; until then the session cannot call the API. Move the file to the secrets folder or delete it after the first sign-in.8182Then:8384```bash85pnpm -s admin collections seed # creates the eight editorial collections, empty86pnpm -s admin status87```8889## The Claude Code publisher token9091Issue it on the node and pipe it straight into the workstation's credential store, so it is never displayed, pasted or written to a shell history:9293```bash94ssh <node> 'cd ~/apps/skills-spboucher-ai && pnpm -s admin token create \95 --user <owner email> --name claude-code-publisher \96 --scopes artifacts:read,artifacts:create,artifacts:update,uploads:create,releases:publish,releases:update \97 --namespace <owner namespace> --expires-in 90d' | skills login --token-stdin98skills whoami99```100101- The token goes to stdout; the metadata (id, prefix, scopes, expiry) goes to stderr, so the pipe carries only the secret.102- Omitting `--scopes` uses exactly that default publisher set.103- Add `--ip <CIDR>` to pin the token to an address range.104- `skills login` stores it in the macOS Keychain (service `skills-registry`, account = registry host). Claude Code uses it by running the CLI; it never needs to read the value.105- For a one-off process-scoped use without storing: `SKILLS_TOKEN="$(…)" skills publish …` in a subshell. Do not export it in a profile or write it to an env file.106107The publisher token cannot revoke releases, deprecate artifacts, manage tokens, read the audit trail or moderate. For those, use the dashboard, or a separate short-lived token:108109```bash110# elevated: administrators only, at most 24 hours111pnpm -s admin token create --user <owner email> --name moderation-2026-11-09 --scopes admin:moderate --expires-in 2h --token-out ~/mod.token112```113114## Rotation and revocation115116```bash117skills auth token rotate # rotates the stored credential; the new token is stored, never printed118skills auth token revoke <id|prefix> # needs tokens:write unless it is the current token119# from the server, for any token:120pnpm -s admin token list --user <owner email>121pnpm -s admin token rotate --id <token id> | skills login --token-stdin122pnpm -s admin token revoke --id <token id> --reason "laptop replaced"123```124125Rotation revokes the old token in the same transaction. Rotate at least every 90 days (the default expiry forces it) and immediately after any suspicion of exposure. An expired token cannot be rotated; issue a new one.126127## Account recovery128129```bash130pnpm -s admin recover --email <email> --password-out ~/recovery.txt131```132133Sets a temporary password that must be changed at next sign-in and revokes all of that user's sessions. API tokens are not touched; revoke them separately if the account was compromised.134135Creating a contributor account:136137```bash138pnpm -s admin user create --email <email> --name "<name>" --namespace <namespace> --password-out ~/contrib.txt139pnpm -s admin verify-namespace --name <namespace> # after checking who they are140pnpm -s admin verify-namespace --name <namespace> --revoke141```142143Send the initial password over a channel you trust. No email is sent by the system.144145## Backups and restore drill146147`infra/backup.sh` (run it as `pnpm backup`; restore drill: `pnpm restore:drill`): a compressed `pg_dump` of the database plus a copy of `DATA_DIR/blobs`, written to a backup directory on the node and replicated to a second node. Schedule it daily.148149What must be backed up: the database, `DATA_DIR/blobs`, and `.env` (separately, in the secrets folder). `DATA_DIR/staging` is disposable.150151Restore drill (do it once after deploy and after any change to the backup script):152153```bash154createdb skills_restore_drill155gunzip -c <backup>/skills-<date>.sql.gz | psql skills_restore_drill156psql skills_restore_drill -c "select count(*) from releases"157# verify blobs against the database158psql -At skills_restore_drill -c "select sha256, archive_object_key from releases" | while IFS='|' read -r sha key; do159 echo "$sha <backup>/blobs/$key" | shasum -a 256 -c - >/dev/null || echo "MISMATCH $key"160done161dropdb skills_restore_drill162```163164Every blob's path contains its SHA-256, so integrity can be checked at any time without the database: `shasum -a 256 <file>` must equal the file name.165166Blobs are immutable and deduplicated; there is no lifecycle expiry. Revoked releases keep their blob so the record stays verifiable; removing bytes for a legal reason is a manual, logged operator action.167168## Monitoring169170- Liveness: `GET /api/v1/health/live` -> `{"status":"ok"}`.171- Readiness: `GET /api/v1/health/ready` -> 200 with `database` and `storage` true, or 503.172- `mld heal` on the gateway restarts the PM2 process and re-points the route if the health check fails.173- Errors: unexpected exceptions are logged as one JSON line to stderr (`pm2 logs skills-web`) with `requestId`, route and stack. The client receives only the request id, which is also in the `X-Request-Id` response header. Expected errors (4xx) are not logged; security-relevant ones are in the audit table.174- `pnpm -s admin status` prints counts of users, artifacts, releases, active tokens, open reports, quarantined releases and audit events.175- Useful queries:176177```sql178-- denied actions in the last day179select at, actor_label, action, target_label, ip, detail from audit_events where result = 'denied' and at > now() - interval '1 day' order by at desc;180-- publishing outcomes in the last week181select action, count(*) from audit_events where action in ('release.publish','release.quarantine','release.reject') and at > now() - interval '7 days' group by 1;182```183184External uptime checks and alerting are Planned.185186## Housekeeping187188```bash189pnpm -s admin cleanup190```191192Expires upload sessions that are more than an hour past their deadline and deletes their staged bytes; deletes idempotency keys older than two days, sessions expired for more than 30 days, and download-dedupe hashes older than two days. Schedule it daily. It never touches releases, blobs or audit events.193194## Incident response195196For every incident: write down times and actions as you go, quote request ids, and keep the audit rows (they cannot be deleted anyway).197198### Leaked or stolen API token1992001. Revoke it: `pnpm -s admin token revoke --id <id> --reason "leaked"` (find the id with `token list`, or by prefix in the audit log).2012. Review what it did: `select * from audit_events where actor_token_id = '<id>' order by at;`2023. For every release it published that the owner does not recognise: revoke the release with a public reason (dashboard, or `POST /releases/{id}/revoke`).2034. Issue a replacement token. Find out how it leaked before reusing the same storage.2045. If the leak could include `.env`: rotate `TOKEN_PEPPER` (this invalidates all tokens and sessions), restart, re-issue tokens.205206### Malicious or compromised release2072081. Revoke the version with a clear public reason. Downloads return 410 from that moment and `latest` moves to the previous good version.2092. If the whole artifact is hostile: take it down (`POST /admin/artifacts/{id}/moderate` with `action: "takedown"`, `confirm: true`).2103. Check the publisher's other artifacts and tokens; revoke tokens, and set the user's status to `suspended` in the database if the account is hostile.2114. Resolve the related report with a note.2125. If users may have installed it, say so publicly on the artifact page (revocation reason) and by direct contact where possible. Security advisories as a product feature are Planned; until then the revocation reason is the public record.213214### Compromised human account2152161. `pnpm -s admin recover --email <email>` (revokes sessions, forces a new password).2172. Revoke all of the user's tokens.2183. Review their audit trail and the releases published in the suspect window.219220### Takedown or licence request2212221. Record the request as a report (category `license` or `impersonation`) if the requester did not.2232. Decide; if granted, take the artifact down or revoke the version with a reason. Reply to the requester.2243. The publisher may appeal by email to the security contact; restoring is `action: "restore"`.225226### Host or database compromise227228Stop the process, preserve the disk state, rotate `TOKEN_PEPPER` and the database credentials, restore from the last trusted backup, verify every blob against its file name, and re-issue credentials. Compare `releases.sha256` from the backup with production before reopening.229230## Logs and retention231232| Data | Where | Retention |233|---|---|---|234| Audit events (actor, action, target, result, IP, user agent, request id) | PostgreSQL, append-only | kept indefinitely in V1; a retention policy with archival is Planned |235| Download counts | daily aggregates per release | indefinitely (no personal data) |236| Download dedupe hashes | salted per day, truncated | purged after two days |237| Sessions | PostgreSQL | deleted 30 days after expiry by `cleanup` |238| Application error log | PM2 log files on the node | rotate with PM2; contains request ids and stack traces, no request bodies |239| Abuse reports, including optional reporter contact | PostgreSQL, moderators only | kept while relevant to moderation history |240241Security contact: contact@spboucher.ai.242