# Operations — AutoScholar.ai on MacLustr

# Topology (as deployed)

Secrets live only in M1M32:~/dispatch/apps/autoscholar.json (pm2 env, mode 600) and /etc/autoscholar/worker.env on the worker (root:asworker 0640). Never in git.

# Routine commands

# status
mld status --live | grep autoscholar
ssh M3U96b 'pm2 ls; pm2 logs autoscholar-scheduler --lines 50 --nostream'
ssh M4M64c 'sudo launchctl print system/io.autoscholar.worker | head -20; tail -50 /var/autoscholar/logs/worker.out.log'

# deploy a change (remote-first): edit on the node, then
ssh M3U96b 'cd ~/apps/autoscholar && pnpm install --frozen-lockfile && pnpm build && pm2 restart autoscholar autoscholar-scheduler && git add -A && git commit -m "…" && git push origin main'

# worker update
bash services/worker/install/bundle.sh && rsync -a --delete dist-bundle/worker/app/ M4M64c:/tmp/autoscholar-worker-bundle/ \
  && ssh M4M64c 'sudo rsync -a --delete /tmp/autoscholar-worker-bundle/ /var/autoscholar/bundle/ && sudo chown -R asworker:staff /var/autoscholar/bundle && sudo launchctl kickstart -k system/io.autoscholar.worker'

# worker self-test (CLAUDE.md loading, denials, sandbox, LaTeX)
ssh M4M64c 'sudo -u asworker bash -c "set -a; . /etc/autoscholar/worker.env; set +a; cd /var/autoscholar/bundle && HOME=/var/autoscholar/home node dist/main.js selftest"'

# Founder controls (admin → Settings)

• dispatch.paused — emergency pause of all new dispatches (workers keep heartbeating; queued runs wait).
• publishing.halted — halts automatic publication globally; publishing.haltedModes per research mode. Runs then stop in awaiting_approval.
• Worker drain (finish current run, take no new work) and quarantine (refuse the worker entirely) from the MacLustr page.
• Cancel / retry per run; override of soft gate failures with recorded rationale; manual publish; retract; withdraw artifacts; versioned metadata edits. All audited (audit_events).
• Env switches (require restart): DISPATCH_ENABLED=0, PUBLISHING_ENABLED=0.

# Backups and restore

• Nightly on M3U96b (pm2 cron process autoscholar-backup, 03:20): pg_dump -Fc autoscholar + rsync of the artifact store to M3U96a:~/backups/autoscholar/ (7 daily copies, 4 weekly). Script: scripts/backup.sh.
• Restore test (documented in docs/TEST_REPORT.md): scripts/restore-test.sh <dump> creates autoscholar_restore, restores, counts papers/runs/artifacts, verifies 20 random artifact digests against artifacts.sha256, drops the DB.
• Search index: PostgreSQL generated tsvector column — rebuilt automatically on restore (no separate index).

# Monitoring and alerts

• Structured JSON logs (pino) with redaction; pm2 logrotate on M3U96b.
• Admin Overview: queue/running/failed counts, worker heartbeat age, cost; Security events (failed logins, lockouts).
• mld heal (every 5 min on M1M32) restarts missing pm2 processes and re-points the tunnel route; worker LaunchDaemon has KeepAlive.
• Degradation modes: control plane down → public site down but no data loss (workers retry, leases expire, runs resume from checkpoint); worker down → queue waits, runs in progress resume on the next worker after lease expiry; Anthropic API failure → stage fails, attempt retried up to maxAttempts, then failed with the error visible in the run timeline.

# Cost ceilings

• Per run: maxBudgetUsd (default 25, hard max 2000) enforced client-side by the SDK (maxBudgetUsd) per stage and by the runner across stages; provider_usage records per stage; the run page shows spend vs cap.
• Platform: DEFAULT_MAX_BUDGET_USD and dispatch.paused as the brake. Provider spend reported is the SDK's client-side estimate (not the invoice).

# Data retention

• Runs, events, artifacts: retained indefinitely for provenance (private). Workspaces on the worker (/var/autoscholar/runs/<id>) can be deleted after published/failed (scripts/worker-gc.sh, keeps 14 days).
• Published artifacts are immutable; retraction keeps the page and notice; withdrawal removes delivery (410) but keeps the private copy.

# Onboarding another MacLustr worker

1. bash services/worker/install/bundle.sh; rsync dist-bundle/worker/app/ to <node>:/tmp/autoscholar-worker-bundle/.
2. ssh <node> 'sudo bash /tmp/autoscholar-worker-bundle/install/install-macos.sh --control-plane http://<M3U96b LAN IP>:8470 --name <alias> --enrollment-secret <secret from the control-plane manifest> --anthropic-key-file /tmp/key --bundle /tmp/autoscholar-worker-bundle' (then rm -P /tmp/key).
3. Watch it appear in admin → MacLustr; run the self-test. Rotate the enrollment secret in the control-plane manifest afterwards if desired.