Repository Public MIT
1# ORIX Astra — Test plan23Results of running this plan are in `TEST_RESULTS.md`. A test that has not been run is listed there as not run; nothing is marked as passing by assumption.45## 1. Levels67| Level | Where | How to run | Needs |8|---|---|---|---|9| Unit and contract | `Tests/AstraCoreTests`, `AstraProvidersTests`, `AstraPersistenceTests`, `AstraAgentsTests` | `Scripts/build.sh test` (`ASTRA_NO_MLX=1 swift test`) | Nothing: no network, no key |10| Local runtime (pure) | `Tests/AstraLocalTests` | `swift test --disable-keychain --filter AstraLocalTests` | Nothing (fake download server, injected memory and thermal readings) |11| Live provider smoke | `Tests/AstraProvidersTests/LiveSmokeTests.swift` | `source Scripts/smoke-env.sh && ASTRA_NO_MLX=1 swift test --filter LiveSmoke` | Development keys, network; costs well under one cent |12| Live model matrix (opt-in) | same file | add `ASTRA_LIVE_MATRIX=1` | Calls every listed chat model once; costs several cents |13| Real MLX inference | `Scripts/verify-models.sh` | downloads each candidate, loads it, generates, cancels, reloads offline | Apple-silicon Mac, about 12 GB of downloads |14| UI | `UITests/AstraUITests.swift` | `Scripts/build.sh uitest` | Simulator "ORIXAstra iPhone" |15| Visual | `Scripts/shot.sh` | screenshots of every primary screen, light and dark | Simulator; live screens need the development keys in the environment |16| Device | manual, `RELEASE_CHECKLIST.md` §Device | — | A physical iPhone |1718## 2. What each suite proves1920### Core (`AstraCoreTests`)21Model reference round-trips; JSON value fidelity (large integers, no trap on out-of-range numbers); server-sent-event parsing (comments, multi-line data, unterminated tail, event-name scoping); HTTP status → typed error mapping; transient classification; URL error wrapping; **no secret in any error message**; redaction of seven key shapes in seven contexts; endpoint policy (HTTPS, plain HTTP refused by default and limited to local ranges when allowed, credentials in URL refused); custom header hygiene; token estimation; context budgeting for the three strategies, tool-result grouping, single oversized turn; cost from usage, cache subsets, provider-reported cost, unavailable vs zero, on-device, preflight upper bound, formatting; parameter negotiation; stream validation; orchestrator guarantees (exactly one terminal event in five failure shapes, retry only before any content, no retry of permanent errors, retry limit, long rate-limit delays not waited out, cancellation, resolver failure); outcome timing.2223### Providers (`AstraProvidersTests`)24For OpenAI, Anthropic, Gemini, xAI, OpenRouter and custom endpoints: request encoding (headers, body shape, no key in the URL), streamed text, tool calls with fragmented arguments, reasoning summaries, usage normalisation (Anthropic cache accounting, Gemini thought tokens, provider-reported cost), mid-stream errors, twelve HTTP error classes each, unsupported-parameter fallback, model list parsing, cancellation, and a key-leak sweep over errors, URLs, bodies and diagnostics. Fixtures are sanitised real captures; seventeen are the exact bytes received during the live run and are replayed.2526### Persistence (`AstraPersistenceTests`)27Migration and schema stability; conversation CRUD, duplication, search with hostile queries (quotes, operators, emoji, CJK); message tree: edit-and-resend, regenerate, branch selection, deterministic order with identical timestamps; partial-answer persistence and recovery of interrupted messages; attachments (hash naming, traversal attempts, dedupe, orphan cleanup); arena sessions and exports (CSV escaping, formula neutralisation); agent profiles, runs, redacted audit steps; prompt templates with version history and variable rendering; usage aggregates; diagnostics ring buffer; memories; documents, chunking, ranked search; exports without secrets; granular deletion; concurrent writers; a 5 000-message conversation performance check.2829### Agents (`AstraAgentsTests`)30Calculator (precedence, associativity, errors, limits, a table of expressions); date and time with a fixed clock and daylight-saving edges; data transforms and CSV quoting; JSON Schema validation; allowlist enforcement (a tool outside the profile never executes); the approval matrix; a denied call lets the run continue; injection scenarios (a document instructing a write or an exfiltrating fetch executes nothing without approval; envelope spoofing neutralised; scanner flags); SSRF refusals; both cycle patterns; every budget type; cancellation in each phase; checkpoint round-trip and resume; redaction in audit events; output truncation.3132### Local (`AstraLocalTests`)33Catalogue decoding and validation; hash-pinning acceptance of online cards; memory and thermal governors with injected readings; download coordinator against a fake server (range resume, hash mismatch, disk-full preflight, traversal attempt, oversize body, mirror down → fallback, duplicate prevention, cancel cleans up, recovery after relaunch); model store removal and usage.3435### UI (`AstraUITests`)36First launch through onboarding; skip; every tab opens; the empty chat state points to Models; the provider key sheet reports a failed connection for a bad key and never leaves the typed key readable on screen; a custom endpoint with plain HTTP to a public host is refused with an explanation.3738## 3. Manual passes (simulator)3940For each primary screen (onboarding ×3, chat list, chat with a streamed answer, model picker, conversation settings, models, provider key sheet, endpoint sheet, arena, agents with an approval, knowledge, lab sections) in dark and light:4142- Layout at default size and at the largest accessibility text size; nothing clipped or overlapping.43- VoiceOver order and labels (Accessibility Inspector audit).44- Reduce Motion: no moving animation remains; Reduce Transparency and Increase Contrast render correctly.45- Every visible control does something; every failure shows a message with a way forward.46- Streaming: the transcript follows only while at the bottom; scrolling up is never overridden; Stop works at once and the partial answer is kept.4748## 4. Device-only checks (cannot be run without an iPhone)4950MLX model download, load and generation on the device, with measured time to first token, tokens per second and peak memory; offline generation in Airplane Mode; behaviour under memory pressure and at elevated thermal state; app lifecycle during a generation (lock, background, return); Foundation Models availability and generation on an eligible device; Face ID app lock; dictation and its on-device indication; haptics; frame pacing while streaming and scrolling a long chat (Instruments); battery impact.5152## 5. Quality gates before a TestFlight build53541. `Scripts/build.sh debug` and a Release archive succeed with no warning in first-party code.552. All unit and contract suites pass; live smoke passes for every provider that has a key.563. Every model in the bundled catalogue has a verification record from a real run.574. `Scripts/secret-scan.sh` is clean for the repository and for the built app bundle.585. UI tests pass on the simulator.596. `TEST_RESULTS.md` lists what was and was not verified, with dates and machines.60