ORIX Astra — Test plan
Results of running this plan are in TEST_RESULTS.md. A test that has not been run is listed there as not run; nothing is marked as passing by assumption.
1. Levels
| Level | Where | How to run | Needs |
|---|---|---|---|
| Unit and contract | Tests/AstraCoreTests, AstraProvidersTests, AstraPersistenceTests, AstraAgentsTests |
Scripts/build.sh test (ASTRA_NO_MLX=1 swift test) |
Nothing: no network, no key |
| Local runtime (pure) | Tests/AstraLocalTests |
swift test --disable-keychain --filter AstraLocalTests |
Nothing (fake download server, injected memory and thermal readings) |
| Live provider smoke | Tests/AstraProvidersTests/LiveSmokeTests.swift |
source Scripts/smoke-env.sh && ASTRA_NO_MLX=1 swift test --filter LiveSmoke |
Development keys, network; costs well under one cent |
| Live model matrix (opt-in) | same file | add ASTRA_LIVE_MATRIX=1 |
Calls every listed chat model once; costs several cents |
| Real MLX inference | Scripts/verify-models.sh |
downloads each candidate, loads it, generates, cancels, reloads offline | Apple-silicon Mac, about 12 GB of downloads |
| UI | UITests/AstraUITests.swift |
Scripts/build.sh uitest |
Simulator "ORIXAstra iPhone" |
| Visual | Scripts/shot.sh |
screenshots of every primary screen, light and dark | Simulator; live screens need the development keys in the environment |
| Device | manual, RELEASE_CHECKLIST.md §Device |
— | A physical iPhone |
2. What each suite proves
Core (AstraCoreTests)
Model reference round-trips; JSON value fidelity (large integers, no trap on out-of-range numbers); server-sent-event parsing (comments, multi-line data, unterminated tail, event-name scoping); HTTP status → typed error mapping; transient classification; URL error wrapping; no secret in any error message; redaction of seven key shapes in seven contexts; endpoint policy (HTTPS, plain HTTP refused by default and limited to local ranges when allowed, credentials in URL refused); custom header hygiene; token estimation; context budgeting for the three strategies, tool-result grouping, single oversized turn; cost from usage, cache subsets, provider-reported cost, unavailable vs zero, on-device, preflight upper bound, formatting; parameter negotiation; stream validation; orchestrator guarantees (exactly one terminal event in five failure shapes, retry only before any content, no retry of permanent errors, retry limit, long rate-limit delays not waited out, cancellation, resolver failure); outcome timing.
Providers (AstraProvidersTests)
For OpenAI, Anthropic, Gemini, xAI, OpenRouter and custom endpoints: request encoding (headers, body shape, no key in the URL), streamed text, tool calls with fragmented arguments, reasoning summaries, usage normalisation (Anthropic cache accounting, Gemini thought tokens, provider-reported cost), mid-stream errors, twelve HTTP error classes each, unsupported-parameter fallback, model list parsing, cancellation, and a key-leak sweep over errors, URLs, bodies and diagnostics. Fixtures are sanitised real captures; seventeen are the exact bytes received during the live run and are replayed.
Persistence (AstraPersistenceTests)
Migration and schema stability; conversation CRUD, duplication, search with hostile queries (quotes, operators, emoji, CJK); message tree: edit-and-resend, regenerate, branch selection, deterministic order with identical timestamps; partial-answer persistence and recovery of interrupted messages; attachments (hash naming, traversal attempts, dedupe, orphan cleanup); arena sessions and exports (CSV escaping, formula neutralisation); agent profiles, runs, redacted audit steps; prompt templates with version history and variable rendering; usage aggregates; diagnostics ring buffer; memories; documents, chunking, ranked search; exports without secrets; granular deletion; concurrent writers; a 5 000-message conversation performance check.
Agents (AstraAgentsTests)
Calculator (precedence, associativity, errors, limits, a table of expressions); date and time with a fixed clock and daylight-saving edges; data transforms and CSV quoting; JSON Schema validation; allowlist enforcement (a tool outside the profile never executes); the approval matrix; a denied call lets the run continue; injection scenarios (a document instructing a write or an exfiltrating fetch executes nothing without approval; envelope spoofing neutralised; scanner flags); SSRF refusals; both cycle patterns; every budget type; cancellation in each phase; checkpoint round-trip and resume; redaction in audit events; output truncation.
Local (AstraLocalTests)
Catalogue decoding and validation; hash-pinning acceptance of online cards; memory and thermal governors with injected readings; download coordinator against a fake server (range resume, hash mismatch, disk-full preflight, traversal attempt, oversize body, mirror down → fallback, duplicate prevention, cancel cleans up, recovery after relaunch); model store removal and usage.
UI (AstraUITests)
First launch through onboarding; skip; every tab opens; the empty chat state points to Models; the provider key sheet reports a failed connection for a bad key and never leaves the typed key readable on screen; a custom endpoint with plain HTTP to a public host is refused with an explanation.
3. Manual passes (simulator)
For each primary screen (onboarding ×3, chat list, chat with a streamed answer, model picker, conversation settings, models, provider key sheet, endpoint sheet, arena, agents with an approval, knowledge, lab sections) in dark and light:
- Layout at default size and at the largest accessibility text size; nothing clipped or overlapping.
- VoiceOver order and labels (Accessibility Inspector audit).
- Reduce Motion: no moving animation remains; Reduce Transparency and Increase Contrast render correctly.
- Every visible control does something; every failure shows a message with a way forward.
- Streaming: the transcript follows only while at the bottom; scrolling up is never overridden; Stop works at once and the partial answer is kept.
4. Device-only checks (cannot be run without an iPhone)
MLX model download, load and generation on the device, with measured time to first token, tokens per second and peak memory; offline generation in Airplane Mode; behaviour under memory pressure and at elevated thermal state; app lifecycle during a generation (lock, background, return); Foundation Models availability and generation on an eligible device; Face ID app lock; dictation and its on-device indication; haptics; frame pacing while streaming and scrolling a long chat (Instruments); battery impact.
5. Quality gates before a TestFlight build
Scripts/build.sh debugand a Release archive succeed with no warning in first-party code.- All unit and contract suites pass; live smoke passes for every provider that has a key.
- Every model in the bundled catalogue has a verification record from a real run.
Scripts/secret-scan.shis clean for the repository and for the built app bundle.- UI tests pass on the simulator.
TEST_RESULTS.mdlists what was and was not verified, with dates and machines.