# AKOS operations runbook

Deployed node: M4M64c (~/apps/akos, PM2 akos-web, akos-worker, akos-proxy), public URL https://akos.spboucher.ai via the MacLustr Tunnel (BHS64 Caddy → wg1 10.67.0.33:8520). Orchestrated by mld from M1M32 (manifest M1M32:~/dispatch/apps/akos.json). Remote-first: the deployed directory is the git repository (origin spbgit akos.git).

# Daily commands

mld status --live | grep akos          # placement + health
ssh M4M64c 'export PATH=/opt/homebrew/bin:$PATH; pm2 ls | grep akos; pm2 logs akos-web --lines 50 --nostream'
curl -s https://akos.spboucher.ai/api/v1/health
akos status && akos doctor             # from any machine with the CLI + token

If the laptop cannot reach M4M64c directly (Tailscale relay hiccup seen 2026-11-01), use the jump alias ssh akosnode (ProxyJump M1M32 → 192.168.2.103).

# Processes (PM2, restart on boot via pm2 startup + mld heal every 5 min)

Restart: pm2 restart akos-web akos-worker akos-proxy. Logs: JSON lines in ~/.pm2/logs/akos-*-out.log (secrets redacted at the logger).

# Change procedure (remote-first)

1. ssh M4M64c → cd ~/apps/akos && git status --short && git log --oneline -3.
2. Edit on the node (or scp round-trip), then make test (offline, uses akos_test) and make test-node (sandbox + backup drill).
3. pm2 restart akos-web akos-worker akos-proxy; curl 127.0.0.1:8520/api/v1/health; curl https://akos.spboucher.ai/api/v1/health.
4. git add -A && git commit -m "…" && git push origin main on the node.
5. Web UI changes: cd web && pnpm build on the node (dist is served by akos-web; excluded from git/rsync). Migrations: add akos/migrations/000N_*.sql (append-only); applied automatically at akos-web start and by python -m akos migrate.

# Secrets

• ~/apps/akos/.env (0600) holds SESSION_SECRET, bootstrap admin password, DATABASE_URL; copy in ~/Desktop/Cluster/secrets/akos/.env (laptop, outside git). Rotate SESSION_SECRET = all sessions logged out (tokens unaffected).
• Provider credentials live in the macOS login keychain of M4M64c (service akos); AKOS stores vault://keychain/akos/<name> only. Onboard: UI → Credential Vault → Create (value is write-only) or POST /api/v1/credentials. Rotate/revoke from the same page; every use is audited.
• To list keychain items: security find-generic-password -s akos -a <name> (value with -w; avoid in shared terminals).

# Backups

• Nightly 03:40 UTC (worker scheduler) → scripts/backup.sh: pg_dump -Fc + artifacts tarball + sha256 → ~/backups/akos/ (keep 14) → rsync M3U96b:~/backups/akos/.
• Manual: POST /api/v1/ops/backup (admin) or bash scripts/backup.sh on the node. Status: Operations page or GET /api/v1/ops/backups.
• Restore drill (tested by make test-node): bash scripts/restore.sh ~/backups/akos/akos-<ts>.dump akos_restore_test [artifacts.tgz]. To restore the live DB: stop PM2 processes, scripts/restore.sh <dump> akos, extract artifacts into data/, start processes.

# Observability

• GET /api/v1/ops/health (DB latency, migrations, live workers, proxy, sandbox, disk), GET /api/v1/ops/metrics (p50/p95 execution latency, job stats, egress decisions, by-API), GET /api/v1/audit/verify (hash chain).
• Every request/execution carries X-Correlation-Id; logs are JSON with cid.
• mld heal restarts missing PM2 processes/routes; agentd sees the site through the registry.

# Recovery

# Rollback

mld tunnel rm akos → pm2 delete akos-web akos-worker akos-proxy → (optional) dropdb akos after a backup → remove ~/apps/akos, manifest, keychain items (security delete-generic-password -s akos -a <name>), claude mcp remove akos -s user on clients.