Acceptance matrix
Each row maps a required scenario to the automated test(s) or scripted procedure that produces its evidence. Results are recorded in docs/acceptance-report.md with PASS / FAIL / BLOCKED, the exact command and output excerpts. Offline (fixture) tests run in CI (make test); live tests are marked @pytest.mark.live and run separately (make test-live).
| ID | Scenario | Evidence producer | Mode |
|---|---|---|---|
| A1 | Unknown public API → provisional AKP + real execution + inspectable in UI | scripts/demo_discovery.sh (Open-Meteo via OpenAPI + probe) + UI |
live |
| A2 | Generated Python client performs a real request; schema and errors tested | tests/live/test_generated_client.py, offline tests/unit/test_codegen.py |
both |
| A3 | Knowledge persists after service restart / node reboot / fresh Claude session | scripts/acceptance_persistence.sh (pm2 restart, count rows before/after), fresh claude -p session log |
live |
| A4 | Cross-project reuse without rediscovery | tests/integration/test_cross_project.py + examples/cross-project-reuse/ with fresh Claude Code project |
both |
| A5 | Opaque MCP labelled; export policy blocks verified claim | tests/integration/test_opaque_mcp.py (examples/opaque-mcp/) |
offline |
| A6 | Enforcement: sandbox + proxy block unregistered egress, allow registered; bypass routes disclosed | tests/security/test_enforcement.py (runs on macOS node) |
live-local |
| A7 | Credentials: scoped fake secret works, unauthorized fails, rotation/revocation, no leakage | tests/security/test_secret_leakage.py, tests/integration/test_credentials.py |
offline |
| A8 | Research export reruns offline from stored inputs; live revision limits reported | tests/reproduction/test_export_replay.py + akos research export on a clean checkout |
both |
| A9 | Cross-source join validated (ids, units, row counts, lineage); incompatible granularity refused | tests/integration/test_composition.py |
offline |
| A10 | Drift: schema change detected, dependents flagged, debt created, fix versioned and revalidated | tests/integration/test_drift.py |
offline |
| A11 | Worker crash recovery + DB backup/restore | tests/integration/test_recovery.py, scripts/backup.sh + scripts/restore.sh drill |
live-local |
| A12 | Auth, tenant separation, SSRF, prompt injection, rate limits, audit logging | tests/security/* |
offline |
| A13 | Public DNS, TLS, UI, REST, MCP from outside the host | scripts/acceptance_external.sh (from laptop) + claude mcp list |
live |
| A14 | Mobile UI at iPhone width, no placeholders | tests/e2e/ui.spec.ts (Playwright, 390×844) |
live |
| A15 | Benchmarks: API lookup, job throughput, page response | scripts/bench.py |
live |