spb/autoscholar

1# Architecture decisions — AutoScholar.ai23Each decision records what was verified vs assumed. Deviations from the master build instructions are explicit.45## D-01 Monorepo and language67pnpm workspace, TypeScript 5 (strict), Node ≥ 22 (nodes run 25/26), Biome for lint/format, Vitest for unit/integration, Playwright for e2e. Rationale: single toolchain on macOS Apple Silicon nodes that already have Node + pnpm (mld runtime detection).89## D-02 Web and API in one Next.js app (deviation from `apps/api`)1011`apps/web` (Next.js 16, React 19, Tailwind 4) serves the public library, the admin panel and **all HTTP APIs** as route handlers (`/api/public/*`, `/api/admin/*`, `/api/internal/workers/*`). The suggested separate `apps/api` service is folded into `apps/web` to keep one public port behind the tunnel. Business logic lives in packages (`db`, `auth`, `schemas`, `storage`, `research-contract`, `maclustr-adapter`) so the API surface stays thin and testable. Deviation documented here; subsystem responsibilities are unchanged.1213## D-03 Durable queue = PostgreSQL (`FOR UPDATE SKIP LOCKED`)1415No Redis dependency. Runs/attempts/leases live in Postgres with lease expiry, fencing tokens (monotonic per run), idempotency keys (unique constraints) and an append-only event ledger. Redis exists on M3U96b but adds nothing at this scale.1617## D-04 Control plane host = M3U96b:84701819Verified: 96 GB RAM, Postgres 17 + pm2 present, wg1 peer, free ports. Two pm2 processes: `autoscholar` (Next.js) and `autoscholar-scheduler` (orchestration loop). Deployed through `mld` with `tunnel.domain = autoscholar.ai`.2021## D-05 Worker host = M4M64c, dedicated user `asworker`2223Verified: empty node, 64 GB, macOS 26.4, Seatbelt + pf + launchd available. The worker daemon runs as a non-admin user via a LaunchDaemon (`io.autoscholar.worker`), not under the operator session; workspaces in `/var/autoscholar/runs/<run-id>`; env file `/etc/autoscholar/worker.env` (root:asworker 0640) holds the Anthropic key and the worker token. The operator installs it once with `services/worker/install/install-macos.sh` (sudo). The platform never uses SSH or sudo at runtime.2425## D-06 Worker transport = outbound LAN HTTP with scoped token2627Workers poll `http://<control-plane LAN IP>:8470/api/internal/workers/*` (lease, heartbeat, events, checkpoints, artifact upload). Public tunnel traffic to `/api/internal/*` is rejected. Enrollment: one-time enrollment secret → long-lived per-worker token (hashed in DB, revocable, rotated by `POST …/rotate`). mTLS not used (no PKI in cluster); the LAN is private and tokens are scoped. Documented limitation.2829## D-07 Artifact store = versioned filesystem store with SHA-256 (deviation from S3)3031No S3-compatible service exists in the cluster and no new paid product is introduced. `packages/storage` implements an immutable content-addressed store (`<root>/runs/<run-id>/<stage>/<path>` + `.sha256` sidecars, write-once, verified on read) with a `StorageBackend` interface so S3 can be added later. Public artifacts are promoted (copied, re-hashed) into `<root>/public/<paper-id>/v<n>/` and served by the web app with immutable cache headers. Backups: nightly `pg_dump` + rsync of the store to M3U96a (docs/OPERATIONS.md).3233## D-08 Research engine = Claude Agent SDK (TypeScript) with API key3435`query()` with `settingSources: ['project']` so the frozen `CLAUDE.md` in the run workspace is the project instruction file (verified by integration test in Phase 5), `permissionMode: 'dontAsk'`, explicit `allowedTools`, `disallowedTools` scoped rules, `PreToolUse` hook policy (platform-side), SDK `sandbox` (Seatbelt + hostname-allowlisted egress proxy, `failIfUnavailable: true`, `allowUnsandboxedCommands: false`), `maxBudgetUsd`/`maxTurns`/wall-clock abort from the manifest, `persistSession: false`, isolated `CLAUDE_CONFIG_DIR` and `HOME` per run. Auth: `ANTHROPIC_API_KEY` only (subscription login is not allowed for third-party products per official docs).3637## D-09 Stage execution split3839Stages 0–8 run on the worker as independently checkpointed SDK sessions (one session per stage, fresh context; the reviewer stage uses a separate session that cannot edit the manuscript directory — enforced by hook + filesystem `denyWrite`). Stage 9 (quality decision) runs **on the control plane** (`services/qa`) over the uploaded artifacts, never from text returned by the model. Stage 10 (publication) runs on the control plane (`services/publisher`) with credentials the worker does not have.4041## D-10 Password hashing4243Argon2id via `@node-rs/argon2` (prebuilt darwin-arm64). Bootstrap password `admin123` is read from `ADMIN_BOOTSTRAP_PASSWORD` **only when the `admin_users` table is empty**, hashed immediately, flagged `must_change_password`, and the env value is never logged. After rotation the old hash is kept in `password_changes` for audit and rejected on login.4445## D-11 Search = PostgreSQL full-text (`tsvector` + GIN) + trigram for titles4647No search engine. Re-evaluate only with measurements.4849## D-12 Paper identifiers5051`AS-WP-<year>-<4-digit>` from a Postgres sequence per year inside the publication transaction; slugs derived from the title + ID; DOIs not issued (no agency membership) — UI shows "No DOI".5253## D-13 LaTeX toolchain = tectonic5455Installed on the worker via Homebrew (no LaTeX exists on any node). Proprietary class `research/templates/latex/autoscholar.cls` + `autoscholar-plots.mplstyle`. PDF checks: `pdfinfo`-equivalent via `pdf-lib`/`pdfjs` in Node (page count, fonts embedded, text extractable) plus rendered page previews with `pdftoppm` if available (poppler installed with tectonic step) else `mutool`.5657## D-14 Local development without MacLustr5859`pnpm dev` runs web + scheduler against local Postgres (laptop has Postgres 17) with `WORKER_MODE=simulated` for tests only; simulated results are marked `synthetic: true` and can never be published (hard gate).6061## D-15 Deviation: no separate `apps/api`, `services/qa` and `services/publisher` are libraries invoked by the scheduler process6263They keep their own directories and tests but execute inside `autoscholar-scheduler` to avoid three extra pm2 processes with identical DB access. Responsibilities unchanged.6465## D-16 Greenfield development exception to remote-first66Until the first deployment the repository was authored on the operator laptop (fast iteration, local Postgres, Playwright). First deployment: `mld stage` → `mld deploy --node M3U96b` → `mld adopt` (node directory = git clone, origin spbgit `autoscholar.git`, commit e709d44). From then on changes are committed and pushed through spbgit and pulled on the node (`git pull && pnpm install && build && pm2 restart`); the worker is a separate bundle (`services/worker/install/bundle.sh`) rsynced to `/var/autoscholar/bundle` on M4M64c. After launch the standard remote-first rule applies (edit on the node).6768## D-17 Internal API source check69`next start` sets `X-Forwarded-For` to the socket address for direct connections; the gateway sets it to the public client address. The internal worker API therefore trusts the *first* XFF hop: it must be inside `INTERNAL_API_ALLOWED_CIDRS` (LAN/loopback), which excludes any request that came through the tunnel.70