ORIX Astra — Test results
Run on 2026-11-21 and 2026-11-22. Machine: Mac17,6 (Apple M5 Max, 48 GB), macOS 27.0, Xcode 27.2 (Swift 6.4.2). Simulator: iPhone 17 Pro, iOS 26.5 runtime ("ORIXAstra iPhone"). No physical iPhone was attached at any point: nothing in this document is a device measurement.
Every line below is something that was actually run. Section 7 lists what was not.
1. Automated suites
| Suite | Command | Result |
|---|---|---|
| Core contracts | ASTRA_NO_MLX=1 swift test --filter AstraCoreTests |
47 tests, 10 suites, pass |
| Cloud providers (fixtures) | … --filter AstraProvidersTests |
107 tests, 9 suites, pass (about 214 cases with parameterised ones; the 8 live tests are skipped without keys) |
| Persistence | … --filter AstraPersistenceTests |
78 tests, 15 suites, pass |
| Agent runtime | … --filter AstraAgentsTests |
163 tests, 22 suites, pass |
| Local runtime (pure) | ASTRA_NO_UI=1 swift test --filter AstraLocalTests |
67 tests, 7 suites, pass |
| UI (simulator) | Scripts/build.sh uitest |
7 tests, pass (onboarding, skip, every tab, empty-state routing, key sheet, endpoint policy, real model download) |
| Total | 462 unit and contract tests + 7 UI tests, all passing |
Build: Scripts/build.sh debug succeeds for the iOS Simulator with no warning in first-party code.
Secret scan: Scripts/secret-scan.sh clean. In addition, each of the six real development key values was searched for verbatim across the project: found in 0 files.
2. Cloud providers, live (real APIs)
Run by LiveSmokeTests with the owner's existing development keys, injected from the environment and never written to disk.
| Provider | API used | Model called | Models listed | Streamed text | Tool round trip | Image input | Reasoning summary | Bad key → unauthorized |
|---|---|---|---|---|---|---|---|---|
| OpenAI | Responses | gpt-6-luna |
51 | pass (usage 23 in / 8 out) | pass | pass | pass | pass |
| Anthropic | Messages (2026-10-21) |
claude-haiku-5-5 |
15 | pass (34 / 10) | pass | pass | pass | pass |
| Google Gemini | streamGenerateContent |
gemini-3.1-flash-lite |
22 | pass (14 / 4) | pass | pass | pass | pass |
| xAI | Responses | grok-4.3 |
7 | pass (200 / 4, 128 cached) | pass | pass | pass | pass |
| OpenRouter | Chat Completions | openai/gpt-6-luna |
441 | pass (23 / 8, served by "OpenAI", cost reported by the provider) | pass | pass | pass | pass |
| Custom endpoint (owner's OpenAI-compatible server) | Chat Completions | first listed model | 1 | pass (26 / 5) | pass | not run | not run | not run |
- The reported model id matched the requested one in every case.
- Model matrix (opt-in, one bounded request to each listed chat model with temperature and reasoning set): 90 of 90 completed — OpenAI 41 (the
-protier excluded), Anthropic 15, Gemini 22, xAI 7, custom 1, and 4 of OpenRouter's 441. - Tool round trips with reasoning enabled: 14 of 14 across the five providers.
- One transient failure was observed: an xAI HTTP 429 ("model at capacity") on a rerun, mapped to
rateLimited; the next run passed. - Seventeen of the test fixtures are the exact bytes received during these runs, replayed in
RecordedTranscriptTests.
3. The app, live in the simulator
Each item was run in the built app and looked at (screenshots in docs/screenshots/).
| Flow | Result |
|---|---|
| First launch and onboarding | Works |
Streamed chat with Anthropic (claude-haiku-5-5) |
Works: Markdown, bold, lists, a syntax-highlighted code block with copy, a reasoning summary, the provider logo |
Streamed chat with OpenAI (gpt-6-luna) |
Works: display and inline LaTeX typeset, Markdown table with inline math, pinned to the end while streaming |
| Missing key | The chat shows "No API key" with a button to Models; nothing crashes |
| Arena, three providers at once (OpenAI, Anthropic, Gemini; then OpenAI, Anthropic, xAI in blind mode) | Works: independent streams, measured first-token and total time, token counts, dated cost estimate, "first to finish" as a speed fact, blind labels with cost hidden until reveal |
| Agent run (Analyst profile, calculator tool, OpenAI) | Works: completed in 2 steps and 6 s; the result ($1,520.32 monthly payment for the test loan) is correct |
| Lab: usage ledger and charts | Works with the real requests above (tokens, cached input, reasoning tokens, known cost) |
| Models: catalogue, fit verdicts, provider key states | Works |
| Model download from the mirror, SHA-256 verification, removal | Works (UI test: 148 MB model installed and removed in 18.5 s) |
| Apple Intelligence | Fails in the iOS Simulator. The system reports the model as available, but generation returns GenerationError -1. It passes on macOS 27 (§4). Not verified on an iPhone |
4. On-device MLX models
Verified by Scripts/verify-models.sh on the Mac (same MLX framework and the same weights the app downloads; not an iPhone). A model passes only if it downloads through the app's own download coordinator with every SHA-256 matching, loads, answers a two-turn conversation reusing the KV cache, stops within 3 s of a mid-stream cancellation, reloads from disk and completes the benchmark.
| Model | Result | Download | Speed (Mac) | Peak memory (Mac) |
|---|---|---|---|---|
| Qwen3 0.6B, 4-bit | pass | 0.35 GB | 423 tok/s | 716 MB |
| Qwen3 1.7B, 4-bit | pass | 0.98 GB | 347 tok/s | 1,541 MB |
| Qwen3 4B, 4-bit | pass | 2.28 GB | 177 tok/s | 2,880 MB |
| Qwen2.5 0.5B Instruct, 4-bit | pass | 0.29 GB | 562 tok/s | 672 MB |
| Gemma 3 1B (QAT), 4-bit (QAT) | pass | 0.77 GB | 331 tok/s | 898 MB |
| Llama 3.2 1B Instruct, 4-bit | pass | 0.71 GB | 537 tok/s | 1,194 MB |
| Llama 3.2 3B Instruct, 4-bit | pass | 1.82 GB | 228 tok/s | 2,457 MB |
| Phi-4 mini Instruct, 4-bit | pass | 2.18 GB | 187 tok/s | 2,856 MB |
| SmolLM2 135M Instruct, 16-bit | pass | 0.27 GB | 399 tok/s | 417 MB |
| SmolLM2 135M Instruct, 8-bit | pass | 0.15 GB | 501 tok/s | 342 MB |
| SmolLM2 360M Instruct, 16-bit | pass | 0.73 GB | 313 tok/s | 1,059 MB |
| SmolLM2 360M Instruct, 6-bit | pass | 0.30 GB | 507 tok/s | 626 MB |
| SmolLM2 1.7B Instruct, 16-bit | pass | 3.43 GB | 116 tok/s | 3,778 MB |
Three further candidates are not in the catalogue: gemma-3-4b-it-qat-4bit loaded, generated and answered an image question, but its Hub card declares a Qwen licence and an InternVL base while its configuration says gemma3, so it is excluded for unclear provenance; SmolLM2-1.7B-Instruct-4bit and SmolLM2-360M-Instruct-4bit do not exist on Hugging Face under those names. TestFlight build 1.0.0 (1) bundles the first seven models verified and receives the other six through the online catalogue; build 1.0.0 (2) bundles all thirteen. Full details, revisions and licences: MODEL_COMPATIBILITY.md.
The local runtime's real-inference tests (streaming and usage, KV-cache reuse, cancellation, thinking, a tool-call round trip, a stop sequence) pass on Qwen3 0.6B on the Mac. An Apple Foundation Models multi-turn conversation also passed on macOS 27 (streamed, recalled a fact from the first turn, reported 86 input and 7 output tokens).
These speeds are from an M5 Max and say nothing about an iPhone. They show that each model loads and generates correctly with this runtime.
5. Model library mirror
https://www.orixlab.dev/astra/models/v1/ (node m4mm, internal SSD), checked on 2026-11-22:
| Check | Result |
|---|---|
catalog.json |
HTTP 200, 13 models (10 mirrored, 3 served from Hugging Face only) |
| Range request on a weights file | HTTP 206, exact byte count; Accept-Ranges, ETag and X-Checksum-Sha256 present |
| Path traversal attempt, unlisted file | HTTP 404 |
| Checksums of every mirrored file (10 models, 88 files, 10 GB), recomputed on the node | match the catalogue |
| A file of a model that is not mirrored (licence) | HTTP 404: the route refuses it |
| Download speed, same 335 MB file, from this Mac | mirror about 35 MB/s; Hugging Face about 8 MB/s |
| Same, from the Beauharnois server | mirror about 74 MB/s; Hugging Face about 13 MB/s |
Both measurement points are close to the hosting network, so these figures are an upper bound: a user elsewhere is limited by the hosting connection's upload rate. If the mirror fails, the app falls back to Hugging Face automatically and verifies the same hashes.
6. Defects found and fixed during testing
| Found by | Defect | Fix |
|---|---|---|
| Unit test | Custom header values containing \r\n passed validation (\r\n is one Character in Swift) |
Check Unicode scalars |
| Contrast computation | Three colour pairs below 4.5:1 (light cyan, light success, white on dark danger fill) | Tokens corrected; dangerFill added |
| Live chat | Display LaTeX fell back to raw source on iOS (the math label only implements intrinsicContentSize) |
iOS path rewritten to size from the intrinsic size and draw the typeset list |
| Live chat | Long answers left the transcript blank after repeated programmatic scrolls of a lazy stack | Transcript is now a regular stack over a 60-message window with paging of earlier messages |
| Live chat | "Jump to latest" showed while already at the end (container size excludes the area under the navigation bar) | Test against the visible rectangle |
| UI test + process sample | The Models screen hung on scroll (lazy-layout placement loop) | Page-level lazy stacks with bounded content replaced by regular stacks |
| Live Arena | A card's header and actions were clipped | Page-style container sized to the available area; tab bar hidden on the comparison screen |
| Live Arena | "2009.9 tok/s" for an answer delivered in one burst after hidden reasoning | Speed is shown only when visible text streamed for at least 0.5 s, counting visible tokens only |
| Lab | Cost unknown for a request sent before the provider's list had loaded | Fall back to the dated curated prices |
| Lab | The per-day chart repeated the same date on every tick | One tick per day, minimum one-week domain |
7. Not verified
- Anything on a physical iPhone: MLX download, load and generation on the device; memory behaviour and the increased-memory-limit entitlement; thermal behaviour; offline use in Airplane Mode; app lifecycle during a generation; Face ID app lock; dictation; haptics; frame pacing and battery impact.
- Apple Intelligence generation on iOS (fails in the simulator, passes on macOS; see §3 and §4).
- VoiceOver, largest Dynamic Type sizes, Reduce Motion as manual passes: the code implements them, they were not walked through by a person.
- Light mode for most screens: Settings and a chat error state were looked at in light; the others only in dark.
- Agents: an approval sheet, a denied call, a file read or write, a web fetch, resume after termination. These paths are covered by the runtime's unit tests but were not exercised in the running app.
- Knowledge: importing a document and searching it in the running app.
- Chat: attachments, edit-and-resend, regenerate, branch arrows, bookmarks, export, model switch mid-conversation in the running app (covered by persistence tests, not exercised on screen).
- Lab: prompt studio, benchmarks (MLX is unavailable in the simulator), diagnostics, model explorer, memory, data export and deletion in the running app.
- Provider paths covered by fixtures only: prompt-cache hits, quota, overload, content-filter and prompt-block responses. OpenAI
-promodels and 437 of OpenRouter's models were never called. - French localisation: not done. The app ships in English; strings are localisable.
- Instruments profiling (startup, frame pacing, memory).