spb/autoscholar
Public
1# Operations — AutoScholar.ai on MacLustr23## Topology (as deployed)45| Component | Where | How it runs | Health |6| -------------------------------- | ------------------------------------------------------------------------------ | ---------------------------------------------- | ----------------------------------------------------------- |7| Web + API (`apps/web`) | M3U96b `~/apps/autoscholar`, port 8470 | pm2 `autoscholar` (`next start`) | `GET /api/public/topics` (200), `GET /` |8| Scheduler (`services/scheduler`) | M3U96b, same checkout | pm2 `autoscholar-scheduler` (`AUTO_MIGRATE=1`) | log lines `tick` / `scheduler started` |9| PostgreSQL 17 | M3U96b (Homebrew), db `autoscholar`, role `autoscholar` | brew service | `pg_isready -h 127.0.0.1` |10| Artifact store | M3U96b `~/apps/autoscholar-data/artifacts` (private `runs/`, public `public/`) | filesystem | `du -sh` |11| Worker | M4M64c, user `asworker`, `/var/autoscholar` | LaunchDaemon `io.autoscholar.worker` | `/var/autoscholar/logs/worker.*.log`; admin → MacLustr page |12| Public ingress | BHS64 Caddy `autoscholar.ai → 10.67.0.11:8470` | MacLustr Tunnel (set by `mld deploy`) | `curl -I https://www.autoscholar.ai` |1314Secrets live only in `M1M32:~/dispatch/apps/autoscholar.json` (pm2 env, mode 600) and `/etc/autoscholar/worker.env` on the worker (root:asworker 0640). Never in git.1516## Routine commands1718```bash19# status20mld status --live | grep autoscholar21ssh M3U96b 'pm2 ls; pm2 logs autoscholar-scheduler --lines 50 --nostream'22ssh M4M64c 'sudo launchctl print system/io.autoscholar.worker | head -20; tail -50 /var/autoscholar/logs/worker.out.log'2324# deploy a change (remote-first): edit on the node, then25ssh M3U96b 'cd ~/apps/autoscholar && pnpm install --frozen-lockfile && pnpm build && pm2 restart autoscholar autoscholar-scheduler && git add -A && git commit -m "…" && git push origin main'2627# worker update28bash services/worker/install/bundle.sh && rsync -a --delete dist-bundle/worker/app/ M4M64c:/tmp/autoscholar-worker-bundle/ \29 && ssh M4M64c 'sudo rsync -a --delete /tmp/autoscholar-worker-bundle/ /var/autoscholar/bundle/ && sudo chown -R asworker:staff /var/autoscholar/bundle && sudo launchctl kickstart -k system/io.autoscholar.worker'3031# worker self-test (CLAUDE.md loading, denials, sandbox, LaTeX)32ssh M4M64c 'sudo -u asworker bash -c "set -a; . /etc/autoscholar/worker.env; set +a; cd /var/autoscholar/bundle && HOME=/var/autoscholar/home node dist/main.js selftest"'33```3435## Founder controls (admin → Settings)3637- `dispatch.paused` — emergency pause of all new dispatches (workers keep heartbeating; queued runs wait).38- `publishing.halted` — halts automatic publication globally; `publishing.haltedModes` per research mode. Runs then stop in `awaiting_approval`.39- Worker **drain** (finish current run, take no new work) and **quarantine** (refuse the worker entirely) from the MacLustr page.40- Cancel / retry per run; override of soft gate failures with recorded rationale; manual publish; retract; withdraw artifacts; versioned metadata edits. All audited (`audit_events`).41- Env switches (require restart): `DISPATCH_ENABLED=0`, `PUBLISHING_ENABLED=0`.4243## Backups and restore4445- Nightly on M3U96b (pm2 cron process `autoscholar-backup`, 03:20): `pg_dump -Fc autoscholar` + `rsync` of the artifact store to `M3U96a:~/backups/autoscholar/` (7 daily copies, 4 weekly). Script: `scripts/backup.sh`.46- Restore test (documented in docs/TEST_REPORT.md): `scripts/restore-test.sh <dump>` creates `autoscholar_restore`, restores, counts papers/runs/artifacts, verifies 20 random artifact digests against `artifacts.sha256`, drops the DB.47- Search index: PostgreSQL generated `tsvector` column — rebuilt automatically on restore (no separate index).4849## Monitoring and alerts5051- Structured JSON logs (pino) with redaction; pm2 logrotate on M3U96b.52- Admin Overview: queue/running/failed counts, worker heartbeat age, cost; Security events (failed logins, lockouts).53- `mld heal` (every 5 min on M1M32) restarts missing pm2 processes and re-points the tunnel route; worker LaunchDaemon has `KeepAlive`.54- Degradation modes: control plane down → public site down but no data loss (workers retry, leases expire, runs resume from checkpoint); worker down → queue waits, runs in progress resume on the next worker after lease expiry; Anthropic API failure → stage fails, attempt retried up to `maxAttempts`, then `failed` with the error visible in the run timeline.5556## Cost ceilings5758- Per run: `maxBudgetUsd` (default 25, hard max 2000) enforced client-side by the SDK (`maxBudgetUsd`) per stage and by the runner across stages; `provider_usage` records per stage; the run page shows spend vs cap.59- Platform: `DEFAULT_MAX_BUDGET_USD` and `dispatch.paused` as the brake. Provider spend reported is the SDK's client-side estimate (not the invoice).6061## Data retention6263- Runs, events, artifacts: retained indefinitely for provenance (private). Workspaces on the worker (`/var/autoscholar/runs/<id>`) can be deleted after `published`/`failed` (`scripts/worker-gc.sh`, keeps 14 days).64- Published artifacts are immutable; retraction keeps the page and notice; withdrawal removes delivery (410) but keeps the private copy.6566## Onboarding another MacLustr worker67681. `bash services/worker/install/bundle.sh`; rsync `dist-bundle/worker/app/` to `<node>:/tmp/autoscholar-worker-bundle/`.692. `ssh <node> 'sudo bash /tmp/autoscholar-worker-bundle/install/install-macos.sh --control-plane http://<M3U96b LAN IP>:8470 --name <alias> --enrollment-secret <secret from the control-plane manifest> --anthropic-key-file /tmp/key --bundle /tmp/autoscholar-worker-bundle'` (then `rm -P /tmp/key`).703. Watch it appear in admin → MacLustr; run the self-test. Rotate the enrollment secret in the control-plane manifest afterwards if desired.71