# Operations — AutoScholar.ai on MacLustr

## Topology (as deployed)

| Component                        | Where                                                                          | How it runs                                    | Health                                                      |
| -------------------------------- | ------------------------------------------------------------------------------ | ---------------------------------------------- | ----------------------------------------------------------- |
| Web + API (`apps/web`)           | M3U96b `~/apps/autoscholar`, port 8470                                         | pm2 `autoscholar` (`next start`)               | `GET /api/public/topics` (200), `GET /`                     |
| Scheduler (`services/scheduler`) | M3U96b, same checkout                                                          | pm2 `autoscholar-scheduler` (`AUTO_MIGRATE=1`) | log lines `tick` / `scheduler started`                      |
| PostgreSQL 17                    | M3U96b (Homebrew), db `autoscholar`, role `autoscholar`                        | brew service                                   | `pg_isready -h 127.0.0.1`                                   |
| Artifact store                   | M3U96b `~/apps/autoscholar-data/artifacts` (private `runs/`, public `public/`) | filesystem                                     | `du -sh`                                                    |
| Worker                           | M4M64c, user `asworker`, `/var/autoscholar`                                    | LaunchDaemon `io.autoscholar.worker`           | `/var/autoscholar/logs/worker.*.log`; admin → MacLustr page |
| Public ingress                   | BHS64 Caddy `autoscholar.ai → 10.67.0.11:8470`                                 | MacLustr Tunnel (set by `mld deploy`)          | `curl -I https://autoscholar.ai`                            |

Secrets live only in `M1M32:~/dispatch/apps/autoscholar.json` (pm2 env, mode 600) and `/etc/autoscholar/worker.env` on the worker (root:asworker 0640). Never in git.

## Routine commands

```bash
# status
mld status --live | grep autoscholar
ssh M3U96b 'pm2 ls; pm2 logs autoscholar-scheduler --lines 50 --nostream'
ssh M4M64c 'sudo launchctl print system/io.autoscholar.worker | head -20; tail -50 /var/autoscholar/logs/worker.out.log'

# deploy a change (remote-first): edit on the node, then
ssh M3U96b 'cd ~/apps/autoscholar && pnpm install --frozen-lockfile && pnpm build && pm2 restart autoscholar autoscholar-scheduler && git add -A && git commit -m "…" && git push origin main'

# worker update
bash services/worker/install/bundle.sh && rsync -a --delete dist-bundle/worker/app/ M4M64c:/tmp/autoscholar-worker-bundle/ \
  && ssh M4M64c 'sudo rsync -a --delete /tmp/autoscholar-worker-bundle/ /var/autoscholar/bundle/ && sudo chown -R asworker:staff /var/autoscholar/bundle && sudo launchctl kickstart -k system/io.autoscholar.worker'

# worker self-test (CLAUDE.md loading, denials, sandbox, LaTeX)
ssh M4M64c 'sudo -u asworker bash -c "set -a; . /etc/autoscholar/worker.env; set +a; cd /var/autoscholar/bundle && HOME=/var/autoscholar/home node dist/main.js selftest"'
```

## Founder controls (admin → Settings)

- `dispatch.paused` — emergency pause of all new dispatches (workers keep heartbeating; queued runs wait).
- `publishing.halted` — halts automatic publication globally; `publishing.haltedModes` per research mode. Runs then stop in `awaiting_approval`.
- Worker **drain** (finish current run, take no new work) and **quarantine** (refuse the worker entirely) from the MacLustr page.
- Cancel / retry per run; override of soft gate failures with recorded rationale; manual publish; retract; withdraw artifacts; versioned metadata edits. All audited (`audit_events`).
- Env switches (require restart): `DISPATCH_ENABLED=0`, `PUBLISHING_ENABLED=0`.

## Backups and restore

- Nightly on M3U96b (pm2 cron process `autoscholar-backup`, 03:20): `pg_dump -Fc autoscholar` + `rsync` of the artifact store to `M3U96a:~/backups/autoscholar/` (7 daily copies, 4 weekly). Script: `scripts/backup.sh`.
- Restore test (documented in docs/TEST_REPORT.md): `scripts/restore-test.sh <dump>` creates `autoscholar_restore`, restores, counts papers/runs/artifacts, verifies 20 random artifact digests against `artifacts.sha256`, drops the DB.
- Search index: PostgreSQL generated `tsvector` column — rebuilt automatically on restore (no separate index).

## Monitoring and alerts

- Structured JSON logs (pino) with redaction; pm2 logrotate on M3U96b.
- Admin Overview: queue/running/failed counts, worker heartbeat age, cost; Security events (failed logins, lockouts).
- `mld heal` (every 5 min on M1M32) restarts missing pm2 processes and re-points the tunnel route; worker LaunchDaemon has `KeepAlive`.
- Degradation modes: control plane down → public site down but no data loss (workers retry, leases expire, runs resume from checkpoint); worker down → queue waits, runs in progress resume on the next worker after lease expiry; Anthropic API failure → stage fails, attempt retried up to `maxAttempts`, then `failed` with the error visible in the run timeline.

## Cost ceilings

- Per run: `maxBudgetUsd` (default 25, hard max 2000) enforced client-side by the SDK (`maxBudgetUsd`) per stage and by the runner across stages; `provider_usage` records per stage; the run page shows spend vs cap.
- Platform: `DEFAULT_MAX_BUDGET_USD` and `dispatch.paused` as the brake. Provider spend reported is the SDK's client-side estimate (not the invoice).

## Data retention

- Runs, events, artifacts: retained indefinitely for provenance (private). Workspaces on the worker (`/var/autoscholar/runs/<id>`) can be deleted after `published`/`failed` (`scripts/worker-gc.sh`, keeps 14 days).
- Published artifacts are immutable; retraction keeps the page and notice; withdrawal removes delivery (410) but keeps the private copy.

## Onboarding another MacLustr worker

1. `bash services/worker/install/bundle.sh`; rsync `dist-bundle/worker/app/` to `<node>:/tmp/autoscholar-worker-bundle/`.
2. `ssh <node> 'sudo bash /tmp/autoscholar-worker-bundle/install/install-macos.sh --control-plane http://<M3U96b LAN IP>:8470 --name <alias> --enrollment-secret <secret from the control-plane manifest> --anthropic-key-file /tmp/key --bundle /tmp/autoscholar-worker-bundle'` (then `rm -P /tmp/key`).
3. Watch it appear in admin → MacLustr; run the self-test. Rotate the enrollment secret in the control-plane manifest afterwards if desired.