SPB Git

Repository Public MIT

TEST_RESULTS.md 11.5 KB 129 lines markdown

1# ORIX Astra — Test results23Run on 2026-11-21 and 2026-11-22. Machine: Mac17,6 (Apple M5 Max, 48 GB), macOS 27.0, Xcode 27.2 (Swift 6.4.2). Simulator: iPhone 17 Pro, iOS 26.5 runtime ("ORIXAstra iPhone"). **No physical iPhone was attached at any point: nothing in this document is a device measurement.**45Every line below is something that was actually run. Section 7 lists what was not.67## 1. Automated suites89| Suite | Command | Result |10|---|---|---|11| Core contracts | `ASTRA_NO_MLX=1 swift test --filter AstraCoreTests` | **47 tests, 10 suites, pass** |12| Cloud providers (fixtures) | `… --filter AstraProvidersTests` | **107 tests, 9 suites, pass** (about 214 cases with parameterised ones; the 8 live tests are skipped without keys) |13| Persistence | `… --filter AstraPersistenceTests` | **78 tests, 15 suites, pass** |14| Agent runtime | `… --filter AstraAgentsTests` | **163 tests, 22 suites, pass** |15| Local runtime (pure) | `ASTRA_NO_UI=1 swift test --filter AstraLocalTests` | **67 tests, 7 suites, pass** |16| UI (simulator) | `Scripts/build.sh uitest` | **7 tests, pass** (onboarding, skip, every tab, empty-state routing, key sheet, endpoint policy, real model download) |17| **Total** | | **462 unit and contract tests + 7 UI tests, all passing** |1819Build: `Scripts/build.sh debug` succeeds for the iOS Simulator with no warning in first-party code.2021Secret scan: `Scripts/secret-scan.sh` clean. In addition, each of the six real development key values was searched for verbatim across the project: found in 0 files.2223## 2. Cloud providers, live (real APIs)2425Run by `LiveSmokeTests` with the owner's existing development keys, injected from the environment and never written to disk.2627| Provider | API used | Model called | Models listed | Streamed text | Tool round trip | Image input | Reasoning summary | Bad key → `unauthorized` |28|---|---|---|---|---|---|---|---|---|29| OpenAI | Responses | `gpt-6-luna` | 51 | pass (usage 23 in / 8 out) | pass | pass | pass | pass |30| Anthropic | Messages (`2026-10-21`) | `claude-haiku-5-5` | 15 | pass (34 / 10) | pass | pass | pass | pass |31| Google Gemini | `streamGenerateContent` | `gemini-3.1-flash-lite` | 22 | pass (14 / 4) | pass | pass | pass | pass |32| xAI | Responses | `grok-4.3` | 7 | pass (200 / 4, 128 cached) | pass | pass | pass | pass |33| OpenRouter | Chat Completions | `openai/gpt-6-luna` | 441 | pass (23 / 8, served by "OpenAI", cost reported by the provider) | pass | pass | pass | pass |34| Custom endpoint (owner's OpenAI-compatible server) | Chat Completions | first listed model | 1 | pass (26 / 5) | pass | not run | not run | not run |3536- The reported model id matched the requested one in every case.37- **Model matrix** (opt-in, one bounded request to each listed chat model with temperature and reasoning set): **90 of 90 completed** — OpenAI 41 (the `-pro` tier excluded), Anthropic 15, Gemini 22, xAI 7, custom 1, and 4 of OpenRouter's 441.38- Tool round trips with reasoning enabled: 14 of 14 across the five providers.39- One transient failure was observed: an xAI HTTP 429 ("model at capacity") on a rerun, mapped to `rateLimited`; the next run passed.40- Seventeen of the test fixtures are the exact bytes received during these runs, replayed in `RecordedTranscriptTests`.4142## 3. The app, live in the simulator4344Each item was run in the built app and looked at (screenshots in `docs/screenshots/`).4546| Flow | Result |47|---|---|48| First launch and onboarding | Works |49| Streamed chat with Anthropic (`claude-haiku-5-5`) | Works: Markdown, bold, lists, a syntax-highlighted code block with copy, a reasoning summary, the provider logo |50| Streamed chat with OpenAI (`gpt-6-luna`) | Works: display and inline LaTeX typeset, Markdown table with inline math, pinned to the end while streaming |51| Missing key | The chat shows "No API key" with a button to Models; nothing crashes |52| Arena, three providers at once (OpenAI, Anthropic, Gemini; then OpenAI, Anthropic, xAI in blind mode) | Works: independent streams, measured first-token and total time, token counts, dated cost estimate, "first to finish" as a speed fact, blind labels with cost hidden until reveal |53| Agent run (Analyst profile, calculator tool, OpenAI) | Works: completed in 2 steps and 6 s; the result ($1,520.32 monthly payment for the test loan) is correct |54| Lab: usage ledger and charts | Works with the real requests above (tokens, cached input, reasoning tokens, known cost) |55| Models: catalogue, fit verdicts, provider key states | Works |56| Model download from the mirror, SHA-256 verification, removal | Works (UI test: 148 MB model installed and removed in 18.5 s) |57| Apple Intelligence | **Fails in the iOS Simulator.** The system reports the model as available, but generation returns `GenerationError -1`. It passes on macOS 27 (§4). Not verified on an iPhone |5859## 4. On-device MLX models6061Verified by `Scripts/verify-models.sh` **on the Mac** (same MLX framework and the same weights the app downloads; not an iPhone). A model passes only if it downloads through the app's own download coordinator with every SHA-256 matching, loads, answers a two-turn conversation reusing the KV cache, stops within 3 s of a mid-stream cancellation, reloads from disk and completes the benchmark.6263| Model | Result | Download | Speed (Mac) | Peak memory (Mac) |64|---|---|---|---|---|65| Qwen3 0.6B, 4-bit | pass | 0.35 GB | 423 tok/s | 716 MB |66| Qwen3 1.7B, 4-bit | pass | 0.98 GB | 347 tok/s | 1,541 MB |67| Qwen3 4B, 4-bit | pass | 2.28 GB | 177 tok/s | 2,880 MB |68| Qwen2.5 0.5B Instruct, 4-bit | pass | 0.29 GB | 562 tok/s | 672 MB |69| Gemma 3 1B (QAT), 4-bit (QAT) | pass | 0.77 GB | 331 tok/s | 898 MB |70| Llama 3.2 1B Instruct, 4-bit | pass | 0.71 GB | 537 tok/s | 1,194 MB |71| Llama 3.2 3B Instruct, 4-bit | pass | 1.82 GB | 228 tok/s | 2,457 MB |72| Phi-4 mini Instruct, 4-bit | pass | 2.18 GB | 187 tok/s | 2,856 MB |73| SmolLM2 135M Instruct, 16-bit | pass | 0.27 GB | 399 tok/s | 417 MB |74| SmolLM2 135M Instruct, 8-bit | pass | 0.15 GB | 501 tok/s | 342 MB |75| SmolLM2 360M Instruct, 16-bit | pass | 0.73 GB | 313 tok/s | 1,059 MB |76| SmolLM2 360M Instruct, 6-bit | pass | 0.30 GB | 507 tok/s | 626 MB |77| SmolLM2 1.7B Instruct, 16-bit | pass | 3.43 GB | 116 tok/s | 3,778 MB |7879Three further candidates are not in the catalogue: `gemma-3-4b-it-qat-4bit` loaded, generated and answered an image question, but its Hub card declares a Qwen licence and an InternVL base while its configuration says `gemma3`, so it is excluded for unclear provenance; `SmolLM2-1.7B-Instruct-4bit` and `SmolLM2-360M-Instruct-4bit` do not exist on Hugging Face under those names. TestFlight build 1.0.0 (1) bundles the first seven models verified and receives the other six through the online catalogue; build 1.0.0 (2) bundles all thirteen. Full details, revisions and licences: `MODEL_COMPATIBILITY.md`.8081The local runtime's real-inference tests (streaming and usage, KV-cache reuse, cancellation, thinking, a tool-call round trip, a stop sequence) pass on Qwen3 0.6B on the Mac. An Apple Foundation Models multi-turn conversation also passed **on macOS 27** (streamed, recalled a fact from the first turn, reported 86 input and 7 output tokens).8283These speeds are from an M5 Max and say nothing about an iPhone. They show that each model loads and generates correctly with this runtime.8485## 5. Model library mirror8687`https://www.orixlab.dev/astra/models/v1/` (node m4mm, internal SSD), checked on 2026-11-22:8889| Check | Result |90|---|---|91| `catalog.json` | HTTP 200, 13 models (10 mirrored, 3 served from Hugging Face only) |92| Range request on a weights file | HTTP 206, exact byte count; `Accept-Ranges`, `ETag` and `X-Checksum-Sha256` present |93| Path traversal attempt, unlisted file | HTTP 404 |94| Checksums of every mirrored file (10 models, 88 files, 10 GB), recomputed on the node | match the catalogue |95| A file of a model that is not mirrored (licence) | HTTP 404: the route refuses it |96| Download speed, same 335 MB file, from this Mac | mirror about 35 MB/s; Hugging Face about 8 MB/s |97| Same, from the Beauharnois server | mirror about 74 MB/s; Hugging Face about 13 MB/s |9899Both measurement points are close to the hosting network, so these figures are an upper bound: a user elsewhere is limited by the hosting connection's upload rate. If the mirror fails, the app falls back to Hugging Face automatically and verifies the same hashes.100101## 6. Defects found and fixed during testing102103| Found by | Defect | Fix |104|---|---|---|105| Unit test | Custom header values containing `\r\n` passed validation (`\r\n` is one `Character` in Swift) | Check Unicode scalars |106| Contrast computation | Three colour pairs below 4.5:1 (light cyan, light success, white on dark danger fill) | Tokens corrected; `dangerFill` added |107| Live chat | Display LaTeX fell back to raw source on iOS (the math label only implements `intrinsicContentSize`) | iOS path rewritten to size from the intrinsic size and draw the typeset list |108| Live chat | Long answers left the transcript blank after repeated programmatic scrolls of a lazy stack | Transcript is now a regular stack over a 60-message window with paging of earlier messages |109| Live chat | "Jump to latest" showed while already at the end (container size excludes the area under the navigation bar) | Test against the visible rectangle |110| UI test + process sample | The Models screen hung on scroll (lazy-layout placement loop) | Page-level lazy stacks with bounded content replaced by regular stacks |111| Live Arena | A card's header and actions were clipped | Page-style container sized to the available area; tab bar hidden on the comparison screen |112| Live Arena | "2009.9 tok/s" for an answer delivered in one burst after hidden reasoning | Speed is shown only when visible text streamed for at least 0.5 s, counting visible tokens only |113| Lab | Cost unknown for a request sent before the provider's list had loaded | Fall back to the dated curated prices |114| Lab | The per-day chart repeated the same date on every tick | One tick per day, minimum one-week domain |115116## 7. Not verified117118- **Anything on a physical iPhone**: MLX download, load and generation on the device; memory behaviour and the increased-memory-limit entitlement; thermal behaviour; offline use in Airplane Mode; app lifecycle during a generation; Face ID app lock; dictation; haptics; frame pacing and battery impact.119- **Apple Intelligence generation on iOS** (fails in the simulator, passes on macOS; see §3 and §4).120- **VoiceOver, largest Dynamic Type sizes, Reduce Motion** as manual passes: the code implements them, they were not walked through by a person.121- **Light mode** for most screens: Settings and a chat error state were looked at in light; the others only in dark.122- Agents: an approval sheet, a denied call, a file read or write, a web fetch, resume after termination. These paths are covered by the runtime's unit tests but were not exercised in the running app.123- Knowledge: importing a document and searching it in the running app.124- Chat: attachments, edit-and-resend, regenerate, branch arrows, bookmarks, export, model switch mid-conversation in the running app (covered by persistence tests, not exercised on screen).125- Lab: prompt studio, benchmarks (MLX is unavailable in the simulator), diagnostics, model explorer, memory, data export and deletion in the running app.126- Provider paths covered by fixtures only: prompt-cache hits, quota, overload, content-filter and prompt-block responses. OpenAI `-pro` models and 437 of OpenRouter's models were never called.127- French localisation: not done. The app ships in English; strings are localisable.128- Instruments profiling (startup, frame pacing, memory).129
Files main
Menu

Theme