|
1 |
+# Test report — AutoScholar.ai |
|
2 |
+ |
|
3 |
+Updated 2026-10-30. Every line below was actually executed; results are copied, not assumed. Live-run sections are appended as runs complete. |
|
4 |
+ |
|
5 |
+## Automated suites (laptop, 2026-10-30) |
|
6 |
+ |
|
7 |
+| Suite | Command | Result | |
|
8 |
+|---|---|---| |
|
9 |
+| Lint/format | `pnpm lint` (Biome 2.6) | clean (2 infos) | |
|
10 |
+| Typecheck | `pnpm -r typecheck`, `apps/web tsc --noEmit` | 0 errors | |
|
11 |
+| Unit (22 tests) | `pnpm vitest run --project unit` | 22 passed: storage write-once/digest/traversal; auth Argon2id/CSRF/limiter/lockout; instruction validation; artifact contract/promotion/human-review rules; BibTeX/cite parsing; reported-value tracing; **policy hook** (sudo/ssh/launchctl/pm2/brew/LAN curl/credential paths/dangerouslyDisableSandbox denied; research commands allowed; writes confined incl. symlink escape and frozen paths; reviewer write scope; SSRF targets for WebFetch; sensitive reads; unknown tools) | |
|
12 |
+| Integration (12 tests, PostgreSQL `autoscholar_test`) | `AUTOSCHOLAR_INTEGRATION=1 pnpm vitest run --project integration` | 12 passed: bootstrap admin123 → forced rotation → reuse rejected → old session revoked; lockout after 5 failures; upload hashing/versioning/dedup/rejection; 3 concurrent launches with one idempotency key → one run; two workers cannot lease the same run; stale fencing tokens rejected; cooperative cancel; worker crash → lease expiry → retry resumes from checkpoint with higher fencing token → attempt limit → failed → admin retry; heartbeat revokes foreign leases; no dispatch to workers without sandbox; disabled worker rejected; simulated run → `quality_failed` with `worker.real` hard failure, never publishable, events redacted; scheduler runtime cap idempotent; concurrent publication → single `AS-WP-2026-0001`, idempotent re-publish, private `input/CLAUDE.md` and `code/.env` absent from the public bundle | |
|
13 |
+| E2E (25 tests, Playwright + axe) | `pnpm test:e2e` | 25 passed (public pages a11y wcag2a/aa on 11 pages; responsive screenshots 390/820/1440; dark mode; keyboard skip link; search/filters/pagination/empty state; paper page + PDF route 200 with digest header; traversal/unlisted 404/400; citation routes; JSON-LD; no private fields in public API; feeds/sitemap/robots/security headers; 14 GET + 15 POST admin routes 401 signed-out; wrong password 401; bootstrap rotation flow; CSRF 403; XSS sanitized; oversize 413; platform-targeting and `.exe` uploads 400; idempotent launch 201→200; live timeline; sign-out) | |
|
14 |
+ |
|
15 |
+## Production checks (2026-10-30) |
|
16 |
+ |
|
17 |
+| Check | Result | |
|
18 |
+|---|---| |
|
19 |
+| Control plane deployed via `mld deploy autoscholar --node M3U96b` | pm2 `autoscholar`, `autoscholar-scheduler` online; local health 200; tunnel route `autoscholar.ai → M3U96b:8470` created on BHS64 | |
|
20 |
+| Public HTTPS | **blocked**: GoDaddy zone has no A record (Caddy: `no valid A records found for autoscholar.ai`) — founder action required | |
|
21 |
+| Admin bootstrap in production | login `admin123` → `mustChangePassword: true`; privileged call before rotation → 403; rotation accepted; `admin123` → 401 afterwards; new password login ok (`mustChangePassword: false`). Executed from M1M32 over the LAN with an explicit cookie header. | |
|
22 |
+| Worker install (M4M64c) | user `asworker` (uid 502, non-admin, hidden), LaunchDaemon running, `/etc/autoscholar/worker.env` root:asworker 0640, pf anchor loaded (LAN/Tailscale/CGNAT blocked for uid 502 except 192.168.2.85:8470 and DNS), tectonic bundle cached, enrolled (`compute_workers`: online, sandboxAvailable=true, Tectonic 0.17.0) | |
|
23 |
+| Worker self-test (`node dist/main.js selftest`, as asworker) | `ok: true` — CLAUDE.md passphrase recalled and written to `results/passphrase.txt` (instruction loading proven); denials recorded by enforcement: `Write: write outside the run workspace`, `Bash: LAN/loopback network access`, `ToolSearch` (then allowed); Seatbelt blocked `cp … /tmp/passphrase-backup.txt` (`Operation not permitted`); no write outside the workspace observed; LaTeX class compiled (2 pages); cost USD 0.13. With the production policy prompt the model declines the same steps on its own; enforcement was verified with the policy prompt removed. | |
|
24 |
+| Backup + restore | `scripts/backup.sh` → pg_dump + rsync to `M3U96a:~/backups/autoscholar` (dumps listed offsite); `scripts/restore-test.sh` restored into `autoscholar_restore`, counted rows, verified artifact digests, dropped the DB | |
|
25 |
+| Fault found and fixed during the first live run | second upload of a changed artifact within one attempt collided with the write-once store → HTTP 500 → runner reported a failed attempt → scheduler retried from the preflight checkpoint on attempt 2 (recovery path worked as designed). Fix: per-version storage keys; redeployed with `scripts/deploy-control-plane.sh` (≈5 s gap) | |
|
26 |
+ |
|
27 |
+## Live research runs |
|
28 |
+ |
|
29 |
+Recorded below as they complete (run IDs, stages, cost, QA outcome, publication). |