Repository Public
1# Operations runbook23Routine procedures for the production instance on M5M36. Diagnosis by symptom is in4[troubleshooting.md](troubleshooting.md); deployment mechanics in [deployment.md](deployment.md); recovery5in [backup-recovery.md](backup-recovery.md).67Conventions: commands run on the node as the cluster operator account (`ssh M5M36`), in8`~/apps/claude-remote` unless stated. `mld …` commands run from the gateway (or its laptop wrapper).9Nothing here needs sudo.1011## Reference1213| Item | Value |14|---|---|15| Public URL | `https://claude.spboucher.ai` |16| Local health | `http://127.0.0.1:8620/api/v1/health` |17| PM2 processes | `claude-remote-runtime`, `claude-remote-server`, `claude-remote-backup` (03:35 nightly) |18| Code | `~/apps/claude-remote` (git; source of truth) |19| Data | `~/claude-remote-data` (0700) |20| Workspace | `~/workspace` |21| Backups | `~/backups/claude-remote/*.tar.gz.enc` (14 kept) |22| Backup passphrase | `~/.config/claude-remote/backup.pass` |23| tmux socket | `tmux -L claude-remote …` |24| Database | `psql "$DATABASE_URL"` after `set -a && . ./.env && set +a` |25| Manifest and secrets | gateway: `~/dispatch/apps/claude-remote.json` |2627## Daily / weekly checks2829| Check | How | Expected |30|---|---|---|31| Service up | `curl -fsS https://claude.spboucher.ai/api/v1/health` | `{"ok":true,"version":"…"}` |32| Processes | `pm2 list` | runtime and server `online`, low restart counts; backup `stopped` between runs is normal |33| Components | System page in the application | database, runtime, tmux `ONLINE`; claude `ONLINE` once authenticated; agentd `ONLINE` if configured |34| Last backup | `ls -lt ~/backups/claude-remote \| head -3` | a file dated last night |35| Disk | System page (disk free) or `df -h ~` | comfortable margin; task logs can reach 256 MiB each |36| Audit trail | System page → audit, or `GET /api/v1/system/audit` | no unexpected `auth.login` failures or unknown client addresses |37| Active browser sessions | Settings → Security | only your devices |3839There is no external alerting: these checks are manual.4041## Start, stop, restart4243```bash44pm2 restart claude-remote-server # browsers reconnect; sessions and tasks untouched45pm2 restart claude-remote-runtime # attach clients dropped, then re-attached; tmux untouched46pm2 stop claude-remote-server # the public site stops answering (gateway error)47pm2 logs claude-remote-server --lines 100 --nostream48```4950- Both can be restarted in one command, as in the update procedure. The server reconnects to the51 runtime by itself with a 1–10 s backoff, in whatever order they come up.52- Restart the server after every rebuild of the web application (the CSP hashes are computed at start).53- **Never run `tmux -L claude-remote kill-server` on production**: it ends every session and task54 process. Sessions become `RECOVERABLE`, running tasks `LOST`.55- From the gateway, `mld heal` restarts missing processes and checks the route; it also runs every56 5 minutes on its own.5758## Planned node reboot59601. Tell yourself what will be lost: every running process inside sessions and tasks.612. In the application, let running tasks finish or cancel them; note which Claude sessions matter.623. Reboot. The node logs in automatically; PM2 and `mld heal` bring the processes back.634. Check health, then open the Sessions page: former live sessions are `RECOVERABLE`. Restart the ones64 you need (`claude --resume` for known conversations) and close the others.6566## Update the application6768Remote-first, on the node — full procedure in [deployment.md](deployment.md):6970```bash71cd ~/apps/claude-remote && git status72# edit73pnpm -r build74set -a && . ./.env && set +a && node apps/server/dist/migrate.js75pm2 restart claude-remote-runtime claude-remote-server76curl -fsS http://127.0.0.1:8620/api/v1/health77git add -A && git commit -m "…" && git push origin main78```7980## Sessions and tasks from the shell8182```bash83tmux -L claude-remote list-sessions # cr-<uuid> = terminal sessions, crt-<uuid> = tasks84tmux -L claude-remote capture-pane -p -t '=cr-<uuid>:' | tail -40 # read-only look at a pane85ls ~/claude-remote-data/tasks/<id>/ # command, log, exit, meta.json86```8788Prefer the application for terminating sessions and cancelling tasks: it records the state transition89and the audit event. A tmux session killed by hand shows up as `RECOVERABLE` (or the task as `LOST`).9091Do not `tmux attach` to a `cr-*` session from an SSH terminal while a browser is using it: the window92follows the most recently used client's size.9394## Backups9596```bash97set -a && . ./.env && set +a && deploy/backup.sh # manual backup; prints "backup ok: <file> (<bytes>)"98pm2 logs claude-remote-backup --lines 20 --nostream # result of the nightly run99```100101Restore drill and disaster recovery: [backup-recovery.md](backup-recovery.md). Recommended monthly:102one restore drill into a scratch database. Outstanding: copy archives off the node; store the passphrase103elsewhere.104105## Access management106107| Task | How |108|---|---|109| Revoke a browser session | Settings → Security → sessions list (its WebSockets close immediately) |110| Change the password | Settings → Security (revokes all other sessions) |111| Add / remove a passkey, enable / disable TOTP | Settings → Security (the last second factor cannot be removed) |112| Emergency lock-out of everyone | `psql "$DATABASE_URL" -c "UPDATE auth_sessions SET revoked_at = now() WHERE revoked_at IS NULL"` — open sockets are closed within 60 s; then change the password |113| Take the site off the Internet | from the gateway: `mld tunnel rm claude-remote` (route removed; the service keeps running locally), or `pm2 stop claude-remote-server` on the node |114| Lost second factors | break-glass in [backup-recovery.md](backup-recovery.md) |115116## Configuration changes117118| Change | Where | Then |119|---|---|---|120| Any `CR_*` variable, `CR_EXEC_HOSTS`, agentd URL | manifest on the gateway (regenerate with `deploy/make-manifest.py`, which keeps the secrets, or edit) | redeploy through `mld`; verify health |121| Allow a new remote task host | authorise the node's SSH key on that host (owner decision — it grants a shell there), add the alias to `CR_EXEC_HOSTS` | redeploy; test with a trivial task |122| Add a workspace root | `CR_WORKSPACE_ROOTS` (`label=/abs/path`, labels unique; server and runtime must agree) | redeploy |123| Rotate `CR_RUNTIME_TOKEN` | remove it from the manifest's environment and regenerate | redeploy (both processes restart; sessions survive) |124| Rotate `CR_SECRET_KEY` | **invalidates the enrolled TOTP secret** | only together with a TOTP re-enrolment (keep a passkey registered first) |125| Claude Code upgrade | `npm install -g @anthropic-ai/claude-code` on the node | new sessions use it; check the System page, the MCP page and a hook-driven `WAITING_FOR_INPUT`, because the application parses CLI output formats observed with 2.1.x |126127## Data housekeeping128129Automatic (daily janitor in the server): lifecycle events (90 days by default, user setting), node130metrics (14 days), login attempts (30 days), audit events (400 days), notifications (90 days), expired131auth sessions (7 days after expiry).132133Manual:134135- `~/claude-remote-data/tasks/<id>/` directories are never deleted by the application. Remove old ones by136 hand when disk space matters (the task row keeps its metadata; its log is then gone).137- Archived sessions stay in the database; there is no purge of `terminal_sessions` rows.138- Unregistering a project never deletes its files; remove the directory in a shell if wanted.139140## Incident notes141142When something unexpected happens, record: time, what the System page showed, `pm2 list`, the last 100143lines of both logs (they contain no secrets, cookies or terminal data by construction), and the audit144entries around the time. Add the outcome to [implementation-log.md](implementation-log.md).145146## Limitations / not verified147148- No monitoring or alerting beyond `mld heal` and the cluster's own node telemetry; a failed backup or a149 full disk is not signalled.150- The planned-reboot procedure has not been executed on M5M36.151- `mld tunnel rm` and the redeploy-after-manifest-change path are cluster-orchestrator operations taken152 from the cluster documentation; they were not re-run for this document.153- Task directory cleanup and session purge are manual; no tooling exists for them.154- Rotation procedures for `CR_RUNTIME_TOKEN` and `CR_SECRET_KEY` are derived from `make-manifest.py` and155 the code; they were not rehearsed.156