Repository Public
1# Deployment23Production: node **M5M36** (Mac Studio, M5 Max, 18 CPU cores, 36 GB, macOS 27.0), deployed 2026-11-10 with4the cluster orchestrator `mld` from the gateway M1M32. The original plan is in5[deployment-plan.md](deployment-plan.md); node choice in [node-selection.md](node-selection.md); network6path in [network-architecture.md](network-architecture.md).78## What runs where910| Item | Value |11|---|---|12| Account | the existing cluster operator macOS account (not root, not a dedicated account — see [security.md](security.md)) |13| Code | `~/apps/claude-remote` on M5M36 — a git clone, `origin` = the cluster git server (`claude-remote.git`). **This directory is the source of truth.** |14| Data directory | `~/claude-remote-data` (0700): `run/` (socket, hook token, tmux config), `tasks/`, `setup-token` until the administrator exists |15| Workspace root | `~/workspace` (label `workspace`) |16| Backups | `~/backups/claude-remote`; passphrase `~/.config/claude-remote/backup.pass` |17| Database | PostgreSQL 17 (Homebrew service), loopback, role and database `claude_remote`, extension `pgcrypto` |18| Public route | `https://claude.spboucher.ai` → Caddy on the gateway BHS64 → WireGuard → M5M36:8620 |19| Manifest | `~/dispatch/apps/claude-remote.json` on M1M32 (holds the generated secrets; not in git) |20| Tools on the node | Node 26, pnpm 12, PM2 7, tmux 3.8, ripgrep, git, Claude Code CLI 2.1.322 (`/opt/homebrew/bin/claude`) |2122## PM2 processes2324| Name | Command (cwd `~/apps/claude-remote`) | Notes |25|---|---|---|26| `claude-remote-runtime` | `node apps/runtime/dist/main.js` | autorestart, `kill_timeout` 8 s, memory restart at 1500 MB. Stopping it never stops tmux sessions |27| `claude-remote-server` | `node apps/server/dist/main.js` | autorestart, memory restart at 1500 MB; listens on `0.0.0.0:8620`; applies pending migrations at start |28| `claude-remote-backup` | `/bin/zsh deploy/backup.sh` | `cron_restart` `35 3 * * *`, no autorestart; only `DATABASE_URL`, `CR_DATA_DIR`, `HOME`, `PATH` in its environment |2930`mld heal` (every 5 minutes, from the gateway) restarts missing PM2 processes and re-checks the tunnel31route. The node logs in automatically and restarts after a power loss (cluster `harden` policy).3233## The mld manifest (`deploy/make-manifest.py`)3435Run **on the gateway**; it prints the manifest JSON to stdout:3637```bash38python3 make-manifest.py [--node M5M36] [--port 8620] [--domain claude.spboucher.ai] \39 [--agentd-node M4M64a] [--agentd-url URL] > ~/dispatch/apps/claude-remote.json40```4142What it does:4344- **Secrets.** Generates `CR_RUNTIME_TOKEN` and `CR_SECRET_KEY` (32 random bytes, hex) and a database45 password, or **reuses** the values found in the existing manifest (`--existing`, default46 `~/dispatch/apps/claude-remote.json`) so that a redeploy keeps them. The agentd token is reused or read47 over SSH from the agentd node. Secrets are never printed anywhere else.48- **Environment.** One environment block shared by the runtime and server processes (table below).49- **Placement.** Pinned to the node (`placement.pin`), never the gateway.50- **`sync_excludes`.** `node_modules/`, `.next/`, `out/`, `dist/`, `data/`, `.env*`, test output.51- **`post_sync` hook** (runs on the node in the app directory): creates the data, workspace and backup52 directories (data 0700); on a first deployment creates the role, the database and `pgcrypto`; writes53 `.env` (0600); `pnpm install --frozen-lockfile`; `pnpm -r --workspace-concurrency=1 build`; checks that54 the three build outputs exist; runs `node apps/server/dist/migrate.js`.55- **Health.** `health_path` `/api/v1/health`; `tunnel.domain` for the public route.5657Standard sequence (as executed on 2026-11-10): generate the manifest → `mld stage <dir> claude-remote` →58`mld deploy claude-remote` (first without the public route, verified over the LAN) → restore drill →59`mld tunnel route claude-remote` → public verification → `mld adopt claude-remote`.6061After adoption, `mld deploy` refuses to overwrite a node directory whose git HEAD differs from the staging62copy or that has uncommitted changes. `mld stage` was needed only to create the application.6364## Environment variables6566`[S]` server, `[R]` runtime, `[S+R]` both (values must match). Full list with comments: `.env.example`.6768| Variable | Scope | Production value / default | Purpose |69|---|---|---|---|70| `NODE_ENV` | S | `production` | enforces https origin and the required secrets |71| `CR_PORT`, `CR_HOST` | S | `8620`, `0.0.0.0` (default `127.0.0.1`) | listener |72| `CR_PUBLIC_ORIGIN` | S | `https://claude.spboucher.ai` | exact origin: cookies, Origin checks, WebAuthn RP ID |73| `CR_TRUSTED_PROXIES` | S | loopback + the gateway's tunnel address | peers whose `X-Forwarded-For` is honoured |74| `DATABASE_URL` | S | generated | PostgreSQL connection |75| `CR_SECRET_KEY` | S | generated (64 hex) | AES-256-GCM key for TOTP secrets — **losing it invalidates enrolled TOTP** |76| `CR_WEB_DIR` | S | `~/apps/claude-remote/apps/web/out` | static export |77| `CR_NODE_ALIAS` | S | `M5M36` | this host's alias in agentd |78| `CR_AGENTD_URL`, `CR_AGENTD_TOKEN` | S | optional | cluster telemetry; unset ⇒ host-only view |79| `CR_LOCAL_MODELS` | S | unset | JSON array of local model endpoints (none exist) |80| `CR_SKILLS_REGISTRY_URL` | S | `https://skills.spboucher.ai` | read-only search adapter |81| `CR_DATA_DIR` | S+R | `~/claude-remote-data` | persistent state |82| `CR_WORKSPACE_ROOTS` | S+R | `workspace=~/workspace` (absolute path) | comma list `label=/abs/path` |83| `CR_RUNTIME_SOCKET` | S+R | `$CR_DATA_DIR/run/runtime.sock` | IPC socket (path ≤ 103 bytes) |84| `CR_RUNTIME_TOKEN` | S+R | generated (≥ 32 chars) | IPC bearer token |85| `CR_EXEC_HOSTS` | S+R | `M3U96a,M3U96b` | node aliases allowed as remote task hosts |86| `CR_TERM_BUFFER_BYTES` | S+R | 4 MiB | WebSocket backpressure high-water mark |87| `CR_LOG_LEVEL` | S+R | `info` | pino level |88| `CR_CLAUDE_BIN` | R | `/opt/homebrew/bin/claude` | Claude Code CLI |89| `CR_MAX_SESSIONS` | R | `12` | live terminal sessions |90| `CR_TMUX_SOCKET` | R | `claude-remote` | tmux `-L` name |91| `CR_SHELL` | R | unset ⇒ `$SHELL` (`/bin/zsh`) | login shell for sessions and tasks |92| `CR_REPLAY_LINES` | R | 5000 (max 20000) | history lines replayed on attach |93| `CR_INSTALL_CLAUDE_HOOKS` | R | `1` | merge the `cr-hook` entries into `~/.claude/settings.json`. Never set on a development machine |94| `CLAUDE_CONFIG_DIR` | R | unset ⇒ `~/.claude` | Claude Code configuration directory |9596`CR_EXEC_HOSTS` is kept from the previous manifest by `make-manifest.py` (default empty); it was set to97the two compute nodes after their owner-approved SSH authorisation.9899## First-run steps (owner)1001011. `ssh M5M36 'cat ~/claude-remote-data/setup-token'`1022. Open `https://claude.spboucher.ai/setup/` and create the administrator (password ≥ 12 characters).1033. Enrol a passkey and/or TOTP (mandatory).1044. Create a project, start a Claude session, run `/login` inside it.1055. Copy `~/.config/claude-remote/backup.pass` from the node to a safe place.106107## Updating — remote-first, on the node108109The node directory is the source of truth. Do not edit a laptop copy and redeploy.110111```bash112ssh M5M36113cd ~/apps/claude-remote114git status # must be clean before you start115# … edit …116pnpm install --frozen-lockfile # only if dependencies changed117pnpm -r build # the manifest uses: pnpm -r --workspace-concurrency=1 build118set -a && . ./.env && set +a # migrate.js needs DATABASE_URL119node apps/server/dist/migrate.js # idempotent; the server also migrates at start120pm2 restart claude-remote-runtime claude-remote-server121curl -fsS http://127.0.0.1:8620/api/v1/health122git add -A && git commit -m "…" && git push origin main # on the node123```124125Effects of the restart: browsers lose their WebSockets and re-attach by themselves; terminal sessions and126running tasks are untouched (see [session-persistence.md](session-persistence.md)). Restart only what127changed: a web- or server-only change needs `claude-remote-server` alone.128129Before a larger change, run the gate where the toolchain is available: `scripts/ci.sh`.130131Changing configuration: edit the manifest on the gateway (or regenerate it with `make-manifest.py`, which132keeps the secrets) and redeploy through `mld`, so that `.env` and the PM2 environment stay consistent.133Editing `.env` on the node alone does not change the environment PM2 gives to the two main processes.134135## Rollback and removal136137- Code: `git revert` / `git checkout <commit>` on the node, rebuild, restart. Migrations are forward-only138 (there are no down migrations).139- Remove the service: `mld retire claude-remote` (removes the route and the processes). Data stays in140 `~/claude-remote-data`, PostgreSQL and `~/backups/claude-remote`.141142## Limitations / not verified143144- Single node; no staging environment. Tests run on a development machine, not on the node.145- No hosted CI: the git server is a bare-repository host. `scripts/ci.sh` is a manual gate.146- A full redeploy from scratch through `mld` on a second node has not been rehearsed (see the disaster147 recovery runbook in [backup-recovery.md](backup-recovery.md)).148- Moving the application (`mld move`) loses live sessions and requires re-authenticating Claude Code on149 the new node; the manifest's placement note says so.150- Claude Code on M5M36 is not authenticated yet and no administrator exists (owner steps above).151- The exact `mld` sub-command names above are those of the cluster's orchestrator as documented for the152 cluster; this repository contains only the manifest generator, not `mld` itself.153154## Codex Remote instance (second instance on the same node)155156The same code base deployed a second time on M5M36 as **Codex Remote** (`https://codex.spboucher.ai`),157driving the OpenAI Codex CLI instead of Claude Code. The two instances are independent: stopping, updating158or restoring one does not touch the other. They share only the code repository, the node's PostgreSQL159server and the workspace root `~/workspace`.160161| Item | Claude Remote | Codex Remote |162|---|---|---|163| mld application / manifest | `claude-remote` / `~/dispatch/apps/claude-remote.json` | `codex-remote` / `~/dispatch/apps/codex-remote.json` |164| Code on the node | `~/apps/claude-remote` | `~/apps/codex-remote` |165| Data directory | `~/claude-remote-data` | `~/codex-remote-data` |166| Port, public route | 8620, `claude.spboucher.ai` | 8630, `codex.spboucher.ai` |167| Database role and database | `claude_remote` | `codex_remote` |168| tmux socket (`-L`) | `claude-remote` | `codex-remote` |169| PM2 processes | `claude-remote-runtime`, `-server`, `-backup` | `codex-remote-runtime`, `-server`, `-backup` |170| Backups, passphrase | `~/backups/claude-remote`, `~/.config/claude-remote/backup.pass`, 03:35 | `~/backups/codex-remote`, `~/.config/codex-remote/backup.pass`, 03:50 |171| Agent CLI | `/opt/homebrew/bin/claude` | `/opt/homebrew/bin/codex` |172173Environment differences, all produced by `make-manifest.py --agent codex` (everything else in the table174"Environment variables" above applies with the Codex names substituted):175176| Variable | Scope | Codex Remote value | Purpose |177|---|---|---|---|178| `CR_AGENT` | S+R | `codex` | which agent CLI this instance drives (default `claude`) |179| `CR_AGENT_BIN` | R | `/opt/homebrew/bin/codex` | the Codex CLI; replaces `CR_CLAUDE_BIN`, which is not emitted |180| `CR_INSTALL_AGENT_HOOKS` | R | `1` | install the runtime's `notify` entry in `~/.codex/config.toml`; replaces `CR_INSTALL_CLAUDE_HOOKS`, which is not emitted |181| `CR_TMUX_SOCKET` | R | `codex-remote` | separate tmux server, so the two instances never see each other's sessions |182| `CR_APP_NAME` | backup process only | `codex-remote` | makes `deploy/backup.sh` use the Codex paths and save the Codex configuration |183184The Claude manifest is unchanged by the `--agent` option: `make-manifest.py` without it (or with185`--agent claude`) produces exactly the previous output.186187### Steps1881891. **DNS first.** Create the record `A codex.spboucher.ai → 51.161.112.61` (the gateway BHS64).1902. **Codex CLI on the node.** `/opt/homebrew/bin/codex` must exist on M5M36 (the manifest points at that191 path). The other tools are those of the Claude instance.1923. **Generate the manifest on the gateway** (same script, same rules as above; a first run generates new193 secrets, later runs reuse those in `~/dispatch/apps/codex-remote.json`):194195 ```bash196 python3 make-manifest.py --agent codex [--node M5M36] [--port 8630] [--domain codex.spboucher.ai] \197 [--agentd-node M4M64a] [--agentd-url URL] > ~/dispatch/apps/codex-remote.json198 ```199200 `CR_AGENTD_URL` and `CR_EXEC_HOSTS` are not copied from the Claude manifest: pass `--agentd-url`, and201 set `CR_EXEC_HOSTS` in the manifest if remote task hosts are wanted, before deploying.2024. **Role and database.** Nothing to do by hand on a first deployment: because a new database password was203 generated, the manifest's `post_sync` hook contains the commands that create the role `codex_remote`,204 the database `codex_remote` and `pgcrypto`, and they run on the node during `mld deploy`. As for Claude205 Remote, a manifest regenerated from an existing one reuses `DATABASE_URL` and does **not** contain206 them — so do not regenerate between step 3 and the first successful deployment, or create the role and207 database by hand (see [backup-recovery.md](backup-recovery.md), runbook step 3, with the Codex names).2085. **Stage and deploy**, the same sequence as for Claude Remote: `mld stage <dir> codex-remote` (a checkout209 of the same repository) → `mld deploy codex-remote --node M5M36` (first without the public route,210 verified over the LAN: `curl -fsS http://127.0.0.1:8630/api/v1/health` on the node) →211 `mld tunnel route codex-remote` → public verification (`https://codex.spboucher.ai/api/v1/health`) →212 `mld adopt codex-remote`.2136. **First-run steps (owner).**214 1. `ssh M5M36 'cat ~/codex-remote-data/setup-token'`215 2. Open `https://codex.spboucher.ai/setup/` and create the administrator (password ≥ 12 characters).216 The administrator, second factors, projects and settings of Codex Remote are its own (separate217 database); passkeys enrolled on `claude.spboucher.ai` do not apply (different WebAuthn RP ID).218 3. Enrol a passkey and/or TOTP (mandatory).219 4. Create a project, open a shell session of the application and run `codex login` in it.220 5. Copy `~/.config/codex-remote/backup.pass` from the node to a safe place.221222### Codex hook (`notify`)223224With `CR_INSTALL_AGENT_HOOKS=1` the runtime installs its `notify` entry in `~/.codex/config.toml`225(`$CODEX_HOME` if set). If that file already contains a **different** `notify`, the runtime refuses to226overwrite it and reports the fact in the runtime `/health`; sessions still work, without the hook-driven227status updates. To resolve it, remove or merge the existing `notify` by hand and restart228`codex-remote-runtime`.229230### Updating231232Same remote-first procedure as above, in `~/apps/codex-remote`, restarting `codex-remote-runtime` /233`codex-remote-server` and checking port 8630. The two directories are two clones of one repository: a234change committed and pushed from one must be pulled, rebuilt and restarted in the other for both instances235to run the same code.236237### Limitations / not verified (Codex Remote)238239- As of 2026-11-10 the Codex instance has **not been deployed**: the steps above are the Claude sequence240 with the Codex names. The generated manifest was checked by reading it, not by running it through `mld`.241- How `mld adopt` treats a second application directory cloned from the same repository242 (`claude-remote.git`) has not been tried.243- `codex login` from a remote terminal has not been exercised on the node.244- Removing the instance: `mld retire codex-remote`; data stays in `~/codex-remote-data`, the245 `codex_remote` database and `~/backups/codex-remote`. Not rehearsed.246