Repository Public
1# Troubleshooting23Symptoms, the cause as implemented in the code, and what to do. Commands assume a shell on the node4(`ssh M5M36`, then `cd ~/apps/claude-remote`). Routine procedures are in5[operations-runbook.md](operations-runbook.md).67## First look89```bash10pm2 list # claude-remote-runtime / -server online?11curl -s http://127.0.0.1:8620/api/v1/health # {"ok":true,"version":"…"}12pm2 logs claude-remote-server --lines 50 --nostream13pm2 logs claude-remote-runtime --lines 50 --nostream14tmux -L claude-remote list-sessions # cr-<uuid> sessions, crt-<uuid> tasks15```1617In the application, the **System** page shows one line per component (database, runtime, tmux, claude,18agentd), each with its own check time, plus host metrics and the audit trail.1920## Site unreachable2122| Symptom | Likely cause | Action |23|---|---|---|24| Browser cannot connect, or the gateway answers 502 | server process down, node down, or tunnel down | `pm2 list` on the node; from the gateway `mld status --live`, `mld tunnel status`; `mld heal` restarts missing processes and re-poses the route |25| Local health fails, process restarting in a loop | invalid configuration | `pm2 logs claude-remote-server`: the message starts with `Invalid configuration:` and lists each problem (origin not https, missing `DATABASE_URL`, `CR_RUNTIME_TOKEN` < 32 chars, `CR_SECRET_KEY` not 64 hex, bad `CR_TRUSTED_PROXIES`) |26| Server exits at start (migrations cannot run), or a running server answers 500 | PostgreSQL down or wrong credentials | `brew services list`; `psql "$DATABASE_URL" -c 'select 1'` |27| Blank or broken pages after rebuilding the web app | the server computes the CSP script hashes of `apps/web/out` at start, so a rebuilt export served by a server that was not restarted has its inline scripts blocked; or an old service-worker cache | restart `claude-remote-server` after every web build; hard-reload the browser |2829## Authentication3031| Symptom | Cause | Action |32|---|---|---|33| Setup page: `invalid_setup_token` | the token is regenerated at every server start while no administrator exists; or pasted with whitespace | read the file again: `cat ~/claude-remote-data/setup-token` |34| `/setup/` says not available / `POST /auth/setup` 404 | an administrator already exists | use `/login/` |35| 429 `locked` with a countdown | failure throttling (5 per username, 10 per IP, 100 global per 15 min; lock doubles from 60 s to 15 min) | wait for `retryAfterSec`. A correct password does not clear a TOTP lock |36| 429 `rate_limited` | more than 20 credential requests per minute from one IP, or 600 API calls per minute per session | wait a minute |37| TOTP code always refused | device clock off by more than one 30 s step; code already used once; `CR_SECRET_KEY` changed since enrolment | fix the clock; wait for the next code; if the key changed, see break-glass in [backup-recovery.md](backup-recovery.md) |38| Passkey prompt fails or is refused | the site is opened under another origin than `CR_PUBLIC_ORIGIN` (LAN address, other hostname); challenge older than 5 minutes | use the public URL; retry |39| Every POST answers 403 `bad_origin` | the page is served from an origin different from `CR_PUBLIC_ORIGIN` (for example the LAN address) | use the public URL |40| 403 `bad_csrf` | the tab's CSRF token no longer matches the session (for example the session was replaced by a login in another tab) | reload the page |41| 403 `second_factor_required` | session has not completed the second factor (15-minute restricted session) | finish the TOTP / passkey step |42| Logged out unexpectedly | idle 12 h, absolute 30 d, password changed elsewhere, or session revoked from Settings → Security | log in again; running sessions are unaffected |4344## Terminal4546| Symptom | Cause | Action |47|---|---|---|48| "Reconnecting… Runtime unavailable" (close 4503) | runtime process down or restarting; socket missing | `pm2 list`, `pm2 logs claude-remote-runtime`. Sessions keep running in tmux and re-attach when the runtime is back |49| Runtime does not start: `another runtime is already listening` | a second instance | `pm2 list`; stop the duplicate |50| Runtime does not start: `CR_RUNTIME_TOKEN must be set`, `tmux not found on PATH`, `socket directory … must not be accessible to group/others`, `CR_RUNTIME_SOCKET path is too long` | configuration / environment | fix the manifest environment; `PATH` must include the Homebrew bin directory; the socket path must be ≤ 103 bytes |51| Runtime exits at start after a manual edit of `~/.claude/settings.json` | with `CR_INSTALL_CLAUDE_HOOKS=1` the hook merge refuses an unparsable settings file and the start fails | repair the JSON, restart the runtime |52| "Session ended (exit N)" | the process exited (close 4410) | Restart starts a new process; nothing is restarted automatically |53| Session shows `RECOVERABLE` | the tmux session no longer exists (host reboot, tmux server killed) | Restart: fresh process, `claude --resume <id>` or `--continue`. Process memory is gone |54| New session refused: 429 `too_many_sessions` | `CR_MAX_SESSIONS` (12) live sessions | terminate unused sessions |55| New session `FAILED` immediately | `claude` not found, or working directory missing / outside the workspace root | System page → claude component (`CLI not found`); check `CR_CLAUDE_BIN`; check the project directory exists |56| Session never shows `WAITING_FOR_INPUT` | hooks not installed (`CR_INSTALL_CLAUDE_HOOKS` unset — always the case with `scripts/dev-up.sh`), or `curl` missing | `grep cr-hook ~/.claude/settings.json`; restart the runtime with the variable set; the state is informational only |57| Claude asks to log in | Claude Code is not authenticated on the node | run `/login` in the session (owner step) |58| Scrollback short or reset after a resize | the mirror replays at most `CR_REPLAY_LINES` (5000) lines / 2 MiB and rebuilds the scrollback when the pane is resized | expected; raise `CR_REPLAY_LINES` (max 20000) if needed |59| Screen not updating in a second, differently sized browser | the tmux window follows the client used last; the other client is caught up when sizes agree or when it is used | type in that browser, or resize it |60| Connection closes with "Connection too slow for the output rate" (4429) | outbound buffer above `CR_TERM_BUFFER_BYTES` for 30 s | the session continues; the client reconnects. Reduce the output or use a better link |61| Every attach fails; runtime log mentions `posix_spawnp failed` | node-pty's `spawn-helper` lost its executable bit (pnpm store) | the runtime repairs it at start; restart the runtime, or `chmod +x` the helper under `node_modules/node-pty` |6263## Files and projects6465| Symptom | Cause | Action |66|---|---|---|67| Save answers 409 `conflict` | the file changed on disk since it was opened (sha mismatch) | the dialog offers reload, view diff, or overwrite |68| 413 `too_large` / 415 `binary` on open | editor limit 2 MiB, text only | download the file instead |69| 403 `path_outside_root` | the path, or a symlink on it, leaves the project root | by design |70| Upload 409 `exists` | destination exists | retry with overwrite |71| Upload fails near 200 MiB | upload cap | use `scp`/`rsync` or split |72| Project creation 409 `project_exists` | the directory is already registered | open the existing project |73| Clone fails: 400 `invalid_git_url` | only `https://`, `ssh://`, `git@host:path`, `gitsrv:` are accepted | fix the URL |74| Clone fails: 502 `clone_failed` / 504 `clone_timeout` | authentication or network: git never prompts (`GIT_TERMINAL_PROMPT=0`), 10-minute limit | test `git ls-remote <url>` in a shell session with the node key |75| Search returns partial results | hard limits: 200 hits, 10 s, 20 000 entries; `.git`, `node_modules`, `.next`, `dist`, `.venv`, `__pycache__` skipped | narrow the directory |7677## Tasks7879| Symptom | Cause | Action |80|---|---|---|81| 403 `host_not_allowed` | host not in `CR_EXEC_HOSTS` | use `local` or an allow-listed node |82| Remote task fails at once with exit 255 | `ssh` could not connect (`BatchMode`, 10 s connect timeout): key not authorised or alias unknown to the node's SSH config | `ssh -o BatchMode=yes <host> true` in a shell session |83| Task `LOST` | the runner disappeared without an exit file (reboot, tmux server killed) | re-create it; `LOST` is never retried automatically |84| Retry refused: 409 `not_idempotent` | retry is offered only for tasks created as idempotent | create a new task |85| Log stops growing but the task runs | log cap reached (256 MiB by default); the task is not blocked | redirect output to a file in the command |86| `TIMED_OUT` | `timeoutSec` reached; the runtime sent TERM then KILL | raise the timeout |8788## Skills, MCP, cluster8990| Symptom | Cause | Action |91|---|---|---|92| MCP page shows status `unknown` | the default listing is configuration only | request live status; it runs `claude mcp list` (up to 45 s cold, cached 60 s) |93| MCP add fails: 502 `mcp_add_failed`, 409 `exists` | the CLI refused; name already used in that scope | check the name and scope; CLI output is deliberately not forwarded because it can echo secrets |94| Skills registry "not connected" | registry unreachable or `CR_SKILLS_REGISTRY_URL` unset | informational; search only |95| Install from registry: 501 | not implemented | install from a directory inside a project |96| Cluster page: nodes `UNKNOWN` / stale | agentd unreachable or metrics older than 90 s | System page → agentd component; check `CR_AGENTD_URL` and the token |97| Cluster page shows only the host | `CR_AGENTD_URL` not set | set it in the manifest and redeploy |98| Local Models empty | no endpoint configured — none exists on the cluster | expected |99100## Build, tests, development101102| Symptom | Cause | Action |103|---|---|---|104| `scripts/dev-up.sh`: "already running" | pid files in `/tmp/cr-dev-data/run` point at live processes | `scripts/dev-down.sh` |105| `dev-up.sh` serves "API only" | `apps/web/out` not built | `pnpm --filter @cr/web build` |106| Server tests fail immediately | need a local PostgreSQL with `createdb`/`dropdb` on `PATH` | start PostgreSQL |107| Runtime / integration tests fail on a machine without `tmux` or `claude` | both are required (no prompt is ever sent) | install them |108| Typecheck errors in `apps/*` about `@cr/shared` | the shared package is not built | `pnpm --filter @cr/shared build` (first step of `scripts/ci.sh`) |109| `pnpm lint` fails | ESLint is referenced by the root script but not configured | not part of the gate |110111## Limitations / not verified112113- Entries about production behaviour after a real reboot, real saturation (4429) and real remote SSH114 failures are derived from the code and tests, not from incidents observed in production.115- `mld` commands are those of the cluster orchestrator; they are not part of this repository.116