spb/claude-remote

1# Backup and recovery23Source: `deploy/backup.sh`, `deploy/restore.sh`, `deploy/make-manifest.py`.45Status on 2026-11-10: the first nightly-format backup was produced on M5M36 (about 20 KB, the database6being almost empty) and a restore drill into a scratch database succeeded (schema restored, row counts7printed). **Backups are not yet copied off the node.**89## What is backed up1011`deploy/backup.sh` runs as the PM2 process `claude-remote-backup` every night at 03:35 (and can be run by12hand). It writes `~/backups/claude-remote/claude-remote-YYYYmmdd-HHMMSS.tar.gz.enc` (mode 0600) and keeps13the 14 most recent archives (`KEEP`).1415| In the archive | Content |16|---|---|17| `db.dump` | `pg_dump --format=custom --no-owner --no-privileges` of the `claude_remote` database: administrator account (password hash, sealed TOTP secret, passkey public keys), auth sessions (token hashes), projects, terminal session metadata and lifecycle events, tasks and artifact metadata, cluster inventory and metrics, notifications, audit trail, settings, migration history |18| `task-meta.tar` | `tasks/<id>/meta.json` of every task (not the logs) |19| `claude-config.tar` | `~/.claude/settings.json`, `~/.claude/skills`, `~/.claude/commands`, `~/.claude/agents` (those that exist) |20| `created_at` | UTC timestamp |2122Encryption: `openssl enc -aes-256-cbc -pbkdf2 -iter 200000 -salt` with the passphrase file23`~/.config/claude-remote/backup.pass` (0600), created with 48 random bytes on the first run.24**Without that file a backup cannot be restored.**2526## What is NOT backed up2728| Item | Where it lives | How it is recovered |29|---|---|---|30| Project files (`~/workspace`) | node disk | from their git remotes; **uncommitted or unpushed work is lost with the node** |31| Application secrets (`CR_SECRET_KEY`, `CR_RUNTIME_TOKEN`, database password, agentd token) | the mld manifest on the gateway (`~/dispatch/apps/claude-remote.json`) and `.env` on the node | from the gateway manifest — `make-manifest.py` reuses them |32| Backup passphrase | `~/.config/claude-remote/backup.pass` on the node | only from the copy the owner made |33| Claude Code credentials | macOS Keychain / Claude Code's own files | `/login` again |34| Claude conversation transcripts (`~/.claude/projects`) | node disk | not recoverable |35| MCP server configuration and API keys (`~/.claude.json`) | node disk | re-enter through the MCP page |36| Global `~/.claude/CLAUDE.md`, plugins | node disk | rewrite / reinstall |37| Node SSH key | `~/.ssh` on the node | generate a new one, re-authorise |38| Task logs, command files | `~/claude-remote-data/tasks` | not recoverable |39| Running processes, terminal history | tmux memory | not recoverable (see [session-persistence.md](session-persistence.md)) |40| Setup token, hook token, sockets | data directory | regenerated |4142## Restore4344```bash45deploy/restore.sh <archive.tar.gz.enc> <target DATABASE_URL> [--with-claude-config]46```4748- Decrypts with `$PASS_FILE` (default `~/.config/claude-remote/backup.pass`), prints the archive date.49- `pg_restore --no-owner --no-privileges --exit-on-error` into the target, which **must exist and be empty**.50- Prints the row counts of `users`, `projects`, `terminal_sessions`, `audit_events`.51- `--with-claude-config` additionally extracts `claude-config.tar` into `$HOME` (overwrites the listed files).52- It does not restore `task-meta.tar`; extract it by hand into `$CR_DATA_DIR` if task metadata on disk matters.5354### Restore drill (do this periodically; it does not touch production)5556```bash57ssh M5M3658cd ~/apps/claude-remote59createdb cr_restore_drill60deploy/restore.sh "$(ls -1t ~/backups/claude-remote/*.enc | head -1)" postgres://localhost/cr_restore_drill61dropdb cr_restore_drill62```6364Expected output: `archive created at: …`, four `name=count` lines, `restore ok`.6566## Recommended and not yet done67681. **Copy the archives to another machine.** They currently stay on M5M36, next to the database they69   protect. For example a nightly `rsync -a ~/backups/claude-remote/ <other-node>:backups/claude-remote/`70   towards a node the M5M36 key can reach. Not configured.712. **Keep the passphrase somewhere else** (owner step 5 of the first-run list), and not next to the off-node72   copies.733. **Protect the gateway manifest**: it is the only other copy of `CR_SECRET_KEY`.7475## Recovery scenarios7677| Scenario | Procedure |78|---|---|79| A process crashed | PM2 restarts it; `mld heal` restarts missing processes within 5 minutes. Nothing to restore |80| Host rebooted | Processes come back; live sessions become `RECOVERABLE`, running tasks `LOST`. Restart sessions from the UI |81| Bad deployment | on the node: `git checkout <good commit>`, rebuild, restart ([deployment.md](deployment.md)). Migrations are forward-only: if a migration must be undone, restore the database from the previous night |82| Database corrupted or data deleted by mistake | stop `claude-remote-server`; create an empty database; `deploy/restore.sh <archive> <url>`; point `DATABASE_URL` at it (or drop and recreate `claude_remote` and restore into it); start the server. Changes since the backup are lost |83| All second factors lost | see "Break-glass" below |84| Node lost | disaster-recovery runbook below |8586## Disaster recovery runbook: M5M36 lost → new node8788Prerequisites that must exist **outside** M5M36: a backup archive, the backup passphrase, the gateway89manifest. If the archive or the passphrase is missing, the database cannot be recovered: deploy fresh,90run the first-run setup again, and re-create projects by cloning.91921. **Choose and prepare the node** (criteria in [node-selection.md](node-selection.md)): always-on Mac93   Studio, tunnel peer, Node, pnpm, PM2. Install `tmux`, `ripgrep`, `postgresql@17` (started as a Homebrew94   service) and the Claude Code CLI (`npm install -g @anthropic-ai/claude-code`).952. **Regenerate the manifest on the gateway** for the new node, keeping the secrets:96   `python3 make-manifest.py --node <NEW> > ~/dispatch/apps/claude-remote.json.new`, review, then replace97   the manifest. The generator reuses `CR_SECRET_KEY`, `CR_RUNTIME_TOKEN`, `DATABASE_URL` and98   `CR_EXEC_HOSTS` from the existing file.993. **Create the database by hand on the new node.** Because `DATABASE_URL` is reused, the manifest's100   `post_sync` hook does *not* contain the role/database creation commands (they are emitted only when a101   new password is generated). Create the role `claude_remote` with the password from the manifest's102   `DATABASE_URL`, the database `claude_remote` owned by it, and `create extension pgcrypto`.1034. **Deploy the code** from the git server (`claude-remote.git`) through `mld` with the new manifest104   (`mld deploy claude-remote --node <NEW>`), then adopt the directory again. The build and the migrations105   run on the node; the server starts with an empty schema.1065. **Restore the database.** Copy the archive and the passphrase file to the node, then:107   `pm2 stop claude-remote-server claude-remote-runtime`; `dropdb claude_remote && createdb -O claude_remote claude_remote`108   (plus `pgcrypto`); `PASS_FILE=<path> deploy/restore.sh <archive> "$DATABASE_URL" --with-claude-config`;109   `pm2 start claude-remote-runtime claude-remote-server`. The restored `schema_migrations` table makes110   the next migration run a no-op unless newer migrations exist. With an administrator in the database,111   the server deletes the setup token at start.1126. **Public route.** `mld deploy` repoints `https://claude.spboucher.ai` to the new node; DNS does not113   change. Check `https://claude.spboucher.ai/api/v1/health`.1147. **Log in.** Password and passkeys work as before (the WebAuthn RP ID is the unchanged hostname). TOTP115   works only if `CR_SECRET_KEY` is the original one.1168. **Re-establish what is not in the backup:** run `/login` in a Claude session; generate a new SSH key on117   the node, authorise it on the git server (git-shell only) and on the `CR_EXEC_HOSTS` nodes, and remove118   the lost node's key from those `authorized_keys`; re-enter MCP servers and their API keys; rewrite the119   sanitized global `CLAUDE.md`. The runtime re-installs its hook entries by itself.1209. **Projects.** Project rows are restored but their directories are not. Re-clone each repository into121   the same relative path under `~/workspace` (the project table stores root label + relative path).12210. **Sessions and tasks.** Sessions that were live at backup time become `RECOVERABLE`; because the123    conversation transcripts were not backed up, restarting them with `--resume` is not expected to find124    the conversation — close them and start new sessions. Tasks that were running become `LOST`.12511. **Re-enable backups** (the PM2 process is part of the manifest), restore the passphrase file at126    `~/.config/claude-remote/backup.pass` so that old and new archives share one passphrase, run a backup127    and a restore drill.12812. Revoke all browser sessions from Settings → Security if the lost node's disk may be read by someone else,129    and change the password.130131## Break-glass: all second factors lost132133There is no recovery code and no reset endpoint. The login logic sends an account with no second factor134to enrolment after a correct password, so the manual procedure on the node is:135136```sql137-- psql "$DATABASE_URL"   (password must still be known)138UPDATE users SET totp_secret_enc = NULL, totp_last_counter = NULL;139DELETE FROM devices;140UPDATE auth_sessions SET revoked_at = now() WHERE revoked_at IS NULL;141```142143then log in with the password and enrol a new factor immediately. Anyone with shell access to the node144account can do this, which is consistent with the trust model in [security.md](security.md). A lost145password has no supported procedure other than restoring a backup or re-running setup on an empty database.146147## Limitations / not verified148149- The disaster-recovery runbook has **not been rehearsed** on a second node. Steps 2–5 follow from reading150  `make-manifest.py` and `restore.sh`; step 4's exact `mld` invocation depends on the orchestrator.151- The break-glass SQL is derived from the login code path and was not exercised.152- Only the restore drill into a scratch database was actually performed (once, with an almost empty database).153- AES-256-CBC provides confidentiality but no integrity check; a corrupted archive is detected only when154  `tar` or `pg_restore` fails.155- There is no monitoring of backup success: a failing nightly run is visible only in156  `pm2 logs claude-remote-backup` and by the absence of a new file.157- Recovery point objective is up to 24 hours for metadata; project files have no backup beyond git.158159## Codex Remote instance160161The second instance (see [deployment.md](deployment.md), "Codex Remote instance") is backed up by the same162two scripts, selected by the environment variable `CR_APP_NAME` (default `claude-remote`). Every name163derives from it; `CR_AGENT` (`claude` | `codex`) selects which agent configuration is saved and, when164unset, is inferred (`codex-remote` ⇒ `codex`, anything else ⇒ `claude`). With neither variable set the165scripts behave exactly as described above.166167| | Claude Remote (default) | Codex Remote (`CR_APP_NAME=codex-remote`) |168|---|---|---|169| PM2 process, schedule | `claude-remote-backup`, 03:35 | `codex-remote-backup`, 03:50 |170| Archives | `~/backups/claude-remote/claude-remote-*.tar.gz.enc` | `~/backups/codex-remote/codex-remote-*.tar.gz.enc` |171| Passphrase | `~/.config/claude-remote/backup.pass` | `~/.config/codex-remote/backup.pass` (a different passphrase; copy it too) |172| Database dumped | `claude_remote` | `codex_remote` (whatever `DATABASE_URL` designates) |173| Task metadata | `~/claude-remote-data/tasks/*/meta.json` | `~/codex-remote-data/tasks/*/meta.json` |174| Agent configuration member | `claude-config.tar` | `codex-config.tar` |175| Secrets (not in the archive) | `~/dispatch/apps/claude-remote.json` on the gateway | `~/dispatch/apps/codex-remote.json` on the gateway |176177The manifest sets `CR_APP_NAME=codex-remote` in the environment of `codex-remote-backup`. Retention178(`KEEP`, 14) applies per instance: each run prunes only the archives carrying its own prefix.179180### What is and is not backed up (Codex)181182`codex-config.tar` holds, relative to the Codex home (`$CODEX_HOME` if set, otherwise `~/.codex`), an183explicit allow-list — each item only if it exists:184185- `config.toml`186- `AGENTS.md`187- `prompts/`188189Never in the archive:190191| Item | Where it lives | How it is recovered |192|---|---|---|193| Codex credentials (`auth.json`) | Codex home | `codex login` again |194| Codex session rollouts, history, sqlite state, logs | Codex home | not recoverable |195196`auth.json`, `*.sqlite*`, `*.db`, `*.jsonl` and `*.log` are additionally excluded by name, so such a file197placed inside `prompts/` is not archived either. `config.toml` is saved as it is: if API keys were written198into it (for example in an MCP server's `env` table), they are in the archive — encrypted, like the rest.199The other rows of "What is NOT backed up" above apply with the Codex names (`~/workspace` is shared by the200two instances and is in neither backup).201202### Restore (Codex)203204```bash205CR_APP_NAME=codex-remote deploy/restore.sh <archive.tar.gz.enc> <target DATABASE_URL> [--with-codex-config]206```207208- `PASS_FILE` defaults to `~/.config/codex-remote/backup.pass`.209- `--with-codex-config` extracts `codex-config.tar` into the Codex home (`$CODEX_HOME` or `~/.codex`,210  created if missing), overwriting `config.toml`, `AGENTS.md` and the files of `prompts/`. A restored211  `config.toml` carries the `notify` entry it had at backup time.212- `--with-agent-config` is accepted by both instances and means the configuration of that instance's213  agent. Passing the other agent's option (`--with-claude-config` to the Codex instance or the reverse) is214  refused before anything is restored.215216Restore drill, as above with the Codex names:217218```bash219ssh M5M36220cd ~/apps/codex-remote221createdb cxr_restore_drill222CR_APP_NAME=codex-remote deploy/restore.sh "$(ls -1t ~/backups/codex-remote/*.enc | head -1)" postgres://localhost/cxr_restore_drill223dropdb cxr_restore_drill224```225226The recovery scenarios and the disaster-recovery runbook apply to Codex Remote with the names substituted227(`make-manifest.py --agent codex`, role and database `codex_remote`, `codex-remote-*` processes,228`CR_APP_NAME=codex-remote … restore.sh … --with-codex-config`, `codex login` instead of `/login`).229230### Limitations / not verified (Codex)231232- Tested on a development machine only, with a temporary `HOME`, a stub `pg_dump` / `pg_restore` / `psql`233  and a fake `~/.codex`: the archive contained `config.toml`, `AGENTS.md`, `prompts/` and not `auth.json`;234  the restore put them back. No backup or restore of a real `codex_remote` database has been performed.235