Test report — AutoScholar.ai
Updated 2026-10-30. Every line below was actually executed; results are copied, not assumed. Live-run sections are appended as runs complete.
Automated suites (laptop, 2026-10-30)
| Suite | Command | Result |
|---|---|---|
| Lint/format | pnpm lint (Biome 2.6) |
clean (2 infos) |
| Typecheck | pnpm -r typecheck, apps/web tsc --noEmit |
0 errors |
| Unit (22 tests) | pnpm vitest run --project unit |
22 passed: storage write-once/digest/traversal; auth Argon2id/CSRF/limiter/lockout; instruction validation; artifact contract/promotion/human-review rules; BibTeX/cite parsing; reported-value tracing; policy hook (sudo/ssh/launchctl/pm2/brew/LAN curl/credential paths/dangerouslyDisableSandbox denied; research commands allowed; writes confined incl. symlink escape and frozen paths; reviewer write scope; SSRF targets for WebFetch; sensitive reads; unknown tools) |
Integration (12 tests, PostgreSQL autoscholar_test) |
AUTOSCHOLAR_INTEGRATION=1 pnpm vitest run --project integration |
12 passed: bootstrap admin123 → forced rotation → reuse rejected → old session revoked; lockout after 5 failures; upload hashing/versioning/dedup/rejection; 3 concurrent launches with one idempotency key → one run; two workers cannot lease the same run; stale fencing tokens rejected; cooperative cancel; worker crash → lease expiry → retry resumes from checkpoint with higher fencing token → attempt limit → failed → admin retry; heartbeat revokes foreign leases; no dispatch to workers without sandbox; disabled worker rejected; simulated run → quality_failed with worker.real hard failure, never publishable, events redacted; scheduler runtime cap idempotent; concurrent publication → single AS-WP-2026-0001, idempotent re-publish, private input/CLAUDE.md and code/.env absent from the public bundle |
| E2E (25 tests, Playwright + axe) | pnpm test:e2e |
25 passed (public pages a11y wcag2a/aa on 11 pages; responsive screenshots 390/820/1440; dark mode; keyboard skip link; search/filters/pagination/empty state; paper page + PDF route 200 with digest header; traversal/unlisted 404/400; citation routes; JSON-LD; no private fields in public API; feeds/sitemap/robots/security headers; 14 GET + 15 POST admin routes 401 signed-out; wrong password 401; bootstrap rotation flow; CSRF 403; XSS sanitized; oversize 413; platform-targeting and .exe uploads 400; idempotent launch 201→200; live timeline; sign-out) |
Production checks (2026-10-30)
| Check | Result |
|---|---|
Control plane deployed via mld deploy autoscholar --node M3U96b |
pm2 autoscholar, autoscholar-scheduler online; local health 200; tunnel route autoscholar.ai → M3U96b:8470 created on BHS64 |
| Public HTTPS | blocked: GoDaddy zone has no A record (Caddy: no valid A records found for autoscholar.ai) — founder action required |
| Admin bootstrap in production | login admin123 → mustChangePassword: true; privileged call before rotation → 403; rotation accepted; admin123 → 401 afterwards; new password login ok (mustChangePassword: false). Executed from M1M32 over the LAN with an explicit cookie header. |
| Worker install (M4M64c) | user asworker (uid 502, non-admin, hidden), LaunchDaemon running, /etc/autoscholar/worker.env root:asworker 0640, pf anchor loaded (LAN/Tailscale/CGNAT blocked for uid 502 except 192.168.2.85:8470 and DNS), tectonic bundle cached, enrolled (compute_workers: online, sandboxAvailable=true, Tectonic 0.17.0) |
Worker self-test (node dist/main.js selftest, as asworker) |
ok: true — CLAUDE.md passphrase recalled and written to results/passphrase.txt (instruction loading proven); denials recorded by enforcement: Write: write outside the run workspace, Bash: LAN/loopback network access, ToolSearch (then allowed); Seatbelt blocked cp … /tmp/passphrase-backup.txt (Operation not permitted); no write outside the workspace observed; LaTeX class compiled (2 pages); cost USD 0.13. With the production policy prompt the model declines the same steps on its own; enforcement was verified with the policy prompt removed. |
| Backup + restore | scripts/backup.sh → pg_dump + rsync to M3U96a:~/backups/autoscholar (dumps listed offsite); scripts/restore-test.sh restored into autoscholar_restore, counted rows, verified artifact digests, dropped the DB |
| Fault found and fixed during the first live run | second upload of a changed artifact within one attempt collided with the write-once store → HTTP 500 → runner reported a failed attempt → scheduler retried from the preflight checkpoint on attempt 2 (recovery path worked as designed). Fix: per-version storage keys; redeployed with scripts/deploy-control-plane.sh (≈5 s gap) |
Production smoke, load and accessibility (2026-10-30, over the LAN)
- All 15 public routes return 200 (home, research, topics, series, about, methodology, infrastructure, transparency, 3 legal pages, feed.xml, sitemap.xml, robots.txt) and an unknown paper returns 404;
robots.txtdisallows/adminand/api. - All 12 admin pages return 200 with a valid session; signed-out admin pages redirect (307) to login; admin APIs return 401.
- Load (20 concurrent clients, 15 s, mixed public pages/APIs from M1M32): 14 683 requests, 979 req/s, p50 17 ms, p95 39 ms, p99 47 ms, max 236 ms; the only errors were HTTP 429 from the public rate limiter (600/min per address) — confirmed by a second 5 s burst returning 100 % 429.
- Accessibility: axe-core wcag2a/aa on 11 public pages in the e2e suite (0 violations); keyboard skip link and reduced-motion verified.
- Dependency audit:
pnpm audit --prod→ 0 known vulnerabilities after upgrading pdfjs-dist to 6.4 (GHSA-hq66-cqwq-w95j).
Live research runs
Recorded below as they complete (run IDs, stages, cost, QA outcome, publication).