# Troubleshooting

Symptoms, the cause as implemented in the code, and what to do. Commands assume a shell on the node (ssh M5M36, then cd ~/apps/claude-remote). Routine procedures are in operations-runbook.md.

# First look

bash

pm2 list                                            # claude-remote-runtime / -server online?
curl -s http://127.0.0.1:8620/api/v1/health         # {"ok":true,"version":"…"}
pm2 logs claude-remote-server --lines 50 --nostream
pm2 logs claude-remote-runtime --lines 50 --nostream
tmux -L claude-remote list-sessions                 # cr-<uuid> sessions, crt-<uuid> tasks

In the application, the System page shows one line per component (database, runtime, tmux, claude, agentd), each with its own check time, plus host metrics and the audit trail.

# Site unreachable

Symptom	Likely cause	Action
Browser cannot connect, or the gateway answers 502	server process down, node down, or tunnel down	`pm2 list` on the node; from the gateway `mld status --live`, `mld tunnel status`; `mld heal` restarts missing processes and re-poses the route
Local health fails, process restarting in a loop	invalid configuration	`pm2 logs claude-remote-server`: the message starts with `Invalid configuration:` and lists each problem (origin not https, missing `DATABASE_URL`, `CR_RUNTIME_TOKEN` < 32 chars, `CR_SECRET_KEY` not 64 hex, bad `CR_TRUSTED_PROXIES`)
Server exits at start (migrations cannot run), or a running server answers 500	PostgreSQL down or wrong credentials	`brew services list`; `psql "$DATABASE_URL" -c 'select 1'`
Blank or broken pages after rebuilding the web app	the server computes the CSP script hashes of `apps/web/out` at start, so a rebuilt export served by a server that was not restarted has its inline scripts blocked; or an old service-worker cache	restart `claude-remote-server` after every web build; hard-reload the browser

# Authentication

Symptom	Cause	Action
Setup page: `invalid_setup_token`	the token is regenerated at every server start while no administrator exists; or pasted with whitespace	read the file again: `cat ~/claude-remote-data/setup-token`
`/setup/` says not available / `POST /auth/setup` 404	an administrator already exists	use `/login/`
429 `locked` with a countdown	failure throttling (5 per username, 10 per IP, 100 global per 15 min; lock doubles from 60 s to 15 min)	wait for `retryAfterSec`. A correct password does not clear a TOTP lock
429 `rate_limited`	more than 20 credential requests per minute from one IP, or 600 API calls per minute per session	wait a minute
TOTP code always refused	device clock off by more than one 30 s step; code already used once; `CR_SECRET_KEY` changed since enrolment	fix the clock; wait for the next code; if the key changed, see break-glass in backup-recovery.md
Passkey prompt fails or is refused	the site is opened under another origin than `CR_PUBLIC_ORIGIN` (LAN address, other hostname); challenge older than 5 minutes	use the public URL; retry
Every POST answers 403 `bad_origin`	the page is served from an origin different from `CR_PUBLIC_ORIGIN` (for example the LAN address)	use the public URL
403 `bad_csrf`	the tab's CSRF token no longer matches the session (for example the session was replaced by a login in another tab)	reload the page
403 `second_factor_required`	session has not completed the second factor (15-minute restricted session)	finish the TOTP / passkey step
Logged out unexpectedly	idle 12 h, absolute 30 d, password changed elsewhere, or session revoked from Settings → Security	log in again; running sessions are unaffected

# Terminal

Symptom	Cause	Action
"Reconnecting… Runtime unavailable" (close 4503)	runtime process down or restarting; socket missing	`pm2 list`, `pm2 logs claude-remote-runtime`. Sessions keep running in tmux and re-attach when the runtime is back
Runtime does not start: `another runtime is already listening`	a second instance	`pm2 list`; stop the duplicate
Runtime does not start: `CR_RUNTIME_TOKEN must be set`, `tmux not found on PATH`, `socket directory … must not be accessible to group/others`, `CR_RUNTIME_SOCKET path is too long`	configuration / environment	fix the manifest environment; `PATH` must include the Homebrew bin directory; the socket path must be ≤ 103 bytes
Runtime exits at start after a manual edit of `~/.claude/settings.json`	with `CR_INSTALL_CLAUDE_HOOKS=1` the hook merge refuses an unparsable settings file and the start fails	repair the JSON, restart the runtime
"Session ended (exit N)"	the process exited (close 4410)	Restart starts a new process; nothing is restarted automatically
Session shows `RECOVERABLE`	the tmux session no longer exists (host reboot, tmux server killed)	Restart: fresh process, `claude --resume <id>` or `--continue`. Process memory is gone
New session refused: 429 `too_many_sessions`	`CR_MAX_SESSIONS` (12) live sessions	terminate unused sessions
New session `FAILED` immediately	`claude` not found, or working directory missing / outside the workspace root	System page → claude component (`CLI not found`); check `CR_CLAUDE_BIN`; check the project directory exists
Session never shows `WAITING_FOR_INPUT`	hooks not installed (`CR_INSTALL_CLAUDE_HOOKS` unset — always the case with `scripts/dev-up.sh`), or `curl` missing	`grep cr-hook ~/.claude/settings.json`; restart the runtime with the variable set; the state is informational only
Claude asks to log in	Claude Code is not authenticated on the node	run `/login` in the session (owner step)
Scrollback short or reset after a resize	the mirror replays at most `CR_REPLAY_LINES` (5000) lines / 2 MiB and rebuilds the scrollback when the pane is resized	expected; raise `CR_REPLAY_LINES` (max 20000) if needed
Screen not updating in a second, differently sized browser	the tmux window follows the client used last; the other client is caught up when sizes agree or when it is used	type in that browser, or resize it
Connection closes with "Connection too slow for the output rate" (4429)	outbound buffer above `CR_TERM_BUFFER_BYTES` for 30 s	the session continues; the client reconnects. Reduce the output or use a better link
Every attach fails; runtime log mentions `posix_spawnp failed`	node-pty's `spawn-helper` lost its executable bit (pnpm store)	the runtime repairs it at start; restart the runtime, or `chmod +x` the helper under `node_modules/node-pty`

# Files and projects

Symptom	Cause	Action
Save answers 409 `conflict`	the file changed on disk since it was opened (sha mismatch)	the dialog offers reload, view diff, or overwrite
413 `too_large` / 415 `binary` on open	editor limit 2 MiB, text only	download the file instead
403 `path_outside_root`	the path, or a symlink on it, leaves the project root	by design
Upload 409 `exists`	destination exists	retry with overwrite
Upload fails near 200 MiB	upload cap	use `scp`/`rsync` or split
Project creation 409 `project_exists`	the directory is already registered	open the existing project
Clone fails: 400 `invalid_git_url`	only `https://`, `ssh://`, `git@host:path`, `gitsrv:` are accepted	fix the URL
Clone fails: 502 `clone_failed` / 504 `clone_timeout`	authentication or network: git never prompts (`GIT_TERMINAL_PROMPT=0`), 10-minute limit	test `git ls-remote <url>` in a shell session with the node key
Search returns partial results	hard limits: 200 hits, 10 s, 20 000 entries; `.git`, `node_modules`, `.next`, `dist`, `.venv`, `__pycache__` skipped	narrow the directory

# Tasks

Symptom	Cause	Action
403 `host_not_allowed`	host not in `CR_EXEC_HOSTS`	use `local` or an allow-listed node
Remote task fails at once with exit 255	`ssh` could not connect (`BatchMode`, 10 s connect timeout): key not authorised or alias unknown to the node's SSH config	`ssh -o BatchMode=yes <host> true` in a shell session
Task `LOST`	the runner disappeared without an exit file (reboot, tmux server killed)	re-create it; `LOST` is never retried automatically
Retry refused: 409 `not_idempotent`	retry is offered only for tasks created as idempotent	create a new task
Log stops growing but the task runs	log cap reached (256 MiB by default); the task is not blocked	redirect output to a file in the command
`TIMED_OUT`	`timeoutSec` reached; the runtime sent TERM then KILL	raise the timeout

# Skills, MCP, cluster

Symptom	Cause	Action
MCP page shows status `unknown`	the default listing is configuration only	request live status; it runs `claude mcp list` (up to 45 s cold, cached 60 s)
MCP add fails: 502 `mcp_add_failed`, 409 `exists`	the CLI refused; name already used in that scope	check the name and scope; CLI output is deliberately not forwarded because it can echo secrets
Skills registry "not connected"	registry unreachable or `CR_SKILLS_REGISTRY_URL` unset	informational; search only
Install from registry: 501	not implemented	install from a directory inside a project
Cluster page: nodes `UNKNOWN` / stale	agentd unreachable or metrics older than 90 s	System page → agentd component; check `CR_AGENTD_URL` and the token
Cluster page shows only the host	`CR_AGENTD_URL` not set	set it in the manifest and redeploy
Local Models empty	no endpoint configured — none exists on the cluster	expected

# Build, tests, development

Symptom	Cause	Action
`scripts/dev-up.sh`: "already running"	pid files in `/tmp/cr-dev-data/run` point at live processes	`scripts/dev-down.sh`
`dev-up.sh` serves "API only"	`apps/web/out` not built	`pnpm --filter @cr/web build`
Server tests fail immediately	need a local PostgreSQL with `createdb`/`dropdb` on `PATH`	start PostgreSQL
Runtime / integration tests fail on a machine without `tmux` or `claude`	both are required (no prompt is ever sent)	install them
Typecheck errors in `apps/*` about `@cr/shared`	the shared package is not built	`pnpm --filter @cr/shared build` (first step of `scripts/ci.sh`)
`pnpm lint` fails	ESLint is referenced by the root script but not configured	not part of the gate

# Limitations / not verified

Entries about production behaviour after a real reboot, real saturation (4429) and real remote SSH failures are derived from the code and tests, not from incidents observed in production.
mld commands are those of the cluster orchestrator; they are not part of this repository.

spb claude-remote