SPB Git

Repository Public MIT

MODEL_COMPATIBILITY.md 14.1 KB 126 lines markdown

1# Model compatibility23Generated by `Scripts/verify-models.sh` from the records in `build/verify/`. Every number below was measured by running the model;4nothing is copied from a model card.56**All measurements were taken on a Mac: Mac17,6 (Apple M5 Max, 48 GB), macOS 27.0.0, with mlx-swift-lm 3.33.0 (release build), on 2026-11-21 to 2026-11-22.7They are not iPhone measurements.** No iPhone was attached to this Mac; see "Not yet verified on a physical iPhone" below.89One-minute load average during the run: 8 to 16 on 18 cores.1011What "pass" means: the repository's current commit was resolved from the Hugging Face API, every file was downloaded through12AstraLocal's `DownloadCoordinator` and its SHA-256 recomputed from disk (and matched against the hash the Hub publishes for13LFS files), the model was loaded by AstraLocal's MLX runtime, answered a two-turn conversation with the second turn reusing14the KV cache, stopped within 3 s of a mid-stream cancellation with `AstraError.cancelled`, loaded again from disk and answered,15and completed the fixed four-prompt benchmark. Speed is the median decode rate of the three warm prompts; time to first token16is the warm median; peak memory is the highest MLX allocator peak during the benchmark (weights included, short prompts).1718## Results1920| Model (repository) | Result | Revision | Quantization | Download | Speed | TTFT warm / cold | Peak memory | In catalog |21|---|---|---|---|---|---|---|---|---|22| `mlx-community/Qwen3-0.6B-4bit` | **pass** | `73e3e38d981303bc594367cd910ea6eb48349da8` | 4-bit | 0.33 GB (9 files) | 423.3 tok/s | 0.013 s / 0.011 s | 683 MB | yes |23| `mlx-community/Qwen3-1.7B-4bit` | **pass** | `3b1b1768f8f8cf8351c712464f906e86c2b8269e` | 4-bit | 0.92 GB (9 files) | 347.3 tok/s | 0.018 s / 0.022 s | 1470 MB | yes |24| `mlx-community/Qwen3-4B-4bit` | **pass** | `4dcb3d101c2a062e5c1d4bb173588c54ea6c4d25` | 4-bit | 2.12 GB (9 files) | 176.9 tok/s | 0.036 s / 0.027 s | 2747 MB | yes |25| `mlx-community/Qwen2.5-0.5B-Instruct-4bit` | **pass** | `a5339a4131f135d0fdc6a5c8b5bbed2753bbe0f3` | 4-bit | 0.27 GB (9 files) | 562.3 tok/s | 0.007 s / 0.009 s | 641 MB | yes |26| `mlx-community/gemma-3-1b-it-qat-4bit` | **pass** | `15fed4eafb456c6fcb2a1165f19ac609670ed14b` | 4-bit (QAT) | 0.72 GB (8 files) | 330.9 tok/s | 0.015 s / 0.025 s | 856 MB | yes |27| `mlx-community/gemma-3-4b-it-qat-4bit` | **pass** | `3d9ef289111449933c22761961f16a5df237ce2a` | 4-bit (QAT) | 2.83 GB (12 files) | 172.5 tok/s | 0.038 s / 0.036 s | 3316 MB | **no** |28| `mlx-community/Llama-3.2-1B-Instruct-4bit` | **pass** | `08231374eeacb049a0eade7922910865b8fce912` | 4-bit | 0.66 GB (6 files) | 537.4 tok/s | 0.011 s / 0.016 s | 1139 MB | yes |29| `mlx-community/Llama-3.2-3B-Instruct-4bit` | **pass** | `7f0dc925e0d0afb0322d96f9255cfddf2ba5636e` | 4-bit | 1.70 GB (6 files) | 227.7 tok/s | 0.031 s / 0.031 s | 2344 MB | yes |30| `mlx-community/Phi-4-mini-instruct-4bit` | **pass** | `ac1c269cb4222a4e136a3d09edad301056c1f36a` | 4-bit | 2.03 GB (12 files) | 186.9 tok/s | 0.022 s / 0.025 s | 2723 MB | yes |31| `mlx-community/SmolLM2-135M-Instruct` | **pass** | `422de227b90002f443a21a58b1087f6ee7632731` | none (16-bit float) | 0.26 GB (8 files) | 399.2 tok/s | 0.008 s / 0.008 s | 397 MB | yes |32| `mlx-community/SmolLM2-135M-Instruct-8bit` | **pass** | `0f0d9b8218915bc34d401e1a340b8c049d300d5e` | 8-bit | 0.14 GB (8 files) | 501.3 tok/s | 0.008 s / 0.009 s | 326 MB | yes |33| `mlx-community/SmolLM2-360M-Instruct` | **pass** | `5d5e6fee5beb5d4d4d85e2203ef23e1230a9aac7` | none (16-bit float) | 0.68 GB (8 files) | 312.8 tok/s | 0.012 s / 0.015 s | 1010 MB | yes |34| `mlx-community/SmolLM2-360M-Instruct-6bit` | **pass** | `642affd1f9e387d1b56c745894afc83795aebe1d` | 6-bit | 0.28 GB (8 files) | 507.0 tok/s | 0.011 s / 0.014 s | 597 MB | yes |35| `mlx-community/SmolLM2-1.7B-Instruct` | **pass** | `464b438b25894353a54836ec5d5a09157e62aa71` | none (16-bit float) | 3.19 GB (8 files) | 115.7 tok/s | 0.020 s / 0.073 s | 3603 MB | yes |36| `mlx-community/SmolLM2-1.7B-Instruct-4bit` | **fail** (resolve) | – | – | – | – | – | – | no |37| `mlx-community/SmolLM2-360M-Instruct-4bit` | **fail** (resolve) | – | – | – | – | – | – | no |3839## Licenses and re-hosting4041`redistributable` in the catalog is `true` only for "Yes". The others are downloaded from Hugging Face only, pending the owner's decision.4243| Model | License | May the owner re-host the weights? | Conditions (summary, not legal advice) | License tag on the Hub card |44|---|---|---|---|---|45| `Qwen3-0.6B-4bit` | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) | Yes | Keep the license text and any NOTICE with the files; state changes if the weights are modified. | `apache-2.0` |46| `Qwen3-1.7B-4bit` | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) | Yes | Keep the license text and any NOTICE with the files; state changes if the weights are modified. | `apache-2.0` |47| `Qwen3-4B-4bit` | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) | Yes | Keep the license text and any NOTICE with the files; state changes if the weights are modified. | `apache-2.0` |48| `Qwen2.5-0.5B-Instruct-4bit` | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) | Yes | Keep the license text and any NOTICE with the files; state changes if the weights are modified. | `apache-2.0` |49| `gemma-3-1b-it-qat-4bit` | [Gemma Terms of Use](https://ai.google.dev/gemma/terms) | Conditional | Redistribution is allowed only if recipients are bound by the same use restrictions (Prohibited Use Policy), receive a copy of the Gemma Terms, and modified files carry a notice. Served from Hugging Face only until the owner decides. | `gemma` |50| `gemma-3-4b-it-qat-4bit` | [Gemma Terms of Use](https://ai.google.dev/gemma/terms) | Conditional | Redistribution is allowed only if recipients are bound by the same use restrictions (Prohibited Use Policy), receive a copy of the Gemma Terms, and modified files carry a notice. Served from Hugging Face only until the owner decides. | `other` |51| `Llama-3.2-1B-Instruct-4bit` | [Llama 3.2 Community License](https://www.llama.com/llama3_2/license/) | Conditional | Redistribution is allowed only with a copy of the license, the “Built with Llama” attribution shown prominently, the notice file kept, and the Acceptable Use Policy respected. Served from Hugging Face only until the owner decides. | `llama3.2` |52| `Llama-3.2-3B-Instruct-4bit` | [Llama 3.2 Community License](https://www.llama.com/llama3_2/license/) | Conditional | Redistribution is allowed only with a copy of the license, the “Built with Llama” attribution shown prominently, the notice file kept, and the Acceptable Use Policy respected. Served from Hugging Face only until the owner decides. | `llama3.2` |53| `Phi-4-mini-instruct-4bit` | [MIT](https://opensource.org/license/mit) | Yes | Keep the copyright and permission notice with the files. | `mit` |54| `SmolLM2-135M-Instruct` | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) | Yes | Keep the license text and any NOTICE with the files; state changes if the weights are modified. | `apache-2.0` |55| `SmolLM2-135M-Instruct-8bit` | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) | Yes | Keep the license text and any NOTICE with the files; state changes if the weights are modified. | `apache-2.0` |56| `SmolLM2-360M-Instruct` | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) | Yes | Keep the license text and any NOTICE with the files; state changes if the weights are modified. | `apache-2.0` |57| `SmolLM2-360M-Instruct-6bit` | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) | Yes | Keep the license text and any NOTICE with the files; state changes if the weights are modified. | `apache-2.0` |58| `SmolLM2-1.7B-Instruct` | [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0) | Yes | Keep the license text and any NOTICE with the files; state changes if the weights are modified. | `apache-2.0` |5960## What each repository actually is6162Read from the downloaded `config.json` and from the Hub API, not assumed from the name.6364| Model | `model_type` | Context (config) | Vision tower | Base model on the Hub card | Recalled the fact in turn 2 | Thinking block | Tool call | Image input | Load time | Weights in memory |65|---|---|---|---|---|---|---|---|---|---|---|66| `Qwen3-0.6B-4bit` | `qwen3` | 40960 | no | Qwen/Qwen3-0.6B | yes | yes | yes | not run | 0.14 s | 320 MB |67| `Qwen3-1.7B-4bit` | `qwen3` | 40960 | no | Qwen/Qwen3-1.7B | yes | yes | yes | not run | 0.10 s | 923 MB |68| `Qwen3-4B-4bit` | `qwen3` | 40960 | no | Qwen/Qwen3-4B | yes | yes | yes | not run | 0.12 s | 2160 MB |69| `Qwen2.5-0.5B-Instruct-4bit` | `qwen2` | 32768 | no | Qwen/Qwen2.5-0.5B | yes | not run | no | not run | 0.11 s | 268 MB |70| `gemma-3-1b-it-qat-4bit` | `gemma3_text` | 32768 | no | google/gemma-3-1b-it | yes | not run | not run | not run | 0.26 s | 702 MB |71| `gemma-3-4b-it-qat-4bit` | `gemma3` | 131072 | yes | OpenGVLab/InternVL3-1B-Instruct | yes | not run | not run | yes | 0.28 s | 2860 MB |72| `Llama-3.2-1B-Instruct-4bit` | `llama` | 131072 | no | mlx-community/Llama-3.2-1B-Instruct-bf16 | yes | not run | yes | not run | 0.13 s | 663 MB |73| `Llama-3.2-3B-Instruct-4bit` | `llama` | 131072 | no | mlx-community/Llama-3.2-3B-Instruct | yes | not run | yes | not run | 0.14 s | 1724 MB |74| `Phi-4-mini-instruct-4bit` | `phi3` | 131072 | no | microsoft/Phi-4-mini-instruct | yes | not run | no | not run | 0.18 s | 2058 MB |75| `SmolLM2-135M-Instruct` | `llama` | 8192 | no | HuggingFaceTB/SmolLM2-135M-Instruct | no | not run | not run | not run | 0.03 s | 258 MB |76| `SmolLM2-135M-Instruct-8bit` | `llama` | 8192 | no | mlx-community/SmolLM2-135M-Instruct | no | not run | not run | not run | 0.04 s | 138 MB |77| `SmolLM2-360M-Instruct` | `llama` | 8192 | no | HuggingFaceTB/SmolLM2-360M-Instruct | yes | not run | not run | not run | 0.04 s | 691 MB |78| `SmolLM2-360M-Instruct-6bit` | `llama` | 8192 | no | HuggingFaceTB/SmolLM2-360M-Instruct | yes | not run | not run | not run | 0.05 s | 284 MB |79| `SmolLM2-1.7B-Instruct` | `llama` | 8192 | no | HuggingFaceTB/SmolLM2-1.7B-Instruct | yes | not run | not run | not run | 0.15 s | 3264 MB |8081"not run" in the three probe columns means the chat template does not declare the feature (or there is no vision tower), so nothing was attempted. "no" means it was attempted and did not work.8283## Failures and exclusions8485- `mlx-community/gemma-3-4b-it-qat-4bit`: ran correctly but is **kept out of the catalog**. Provenance unclear: the Hub card declares license “other” and base model “OpenGVLab/InternVL3-1B-Instruct”, which does not match a Gemma 3 model (config model_type “gemma3”).86- `mlx-community/SmolLM2-1.7B-Instruct-4bit`: failed at *resolve*: Hugging Face has no public repository named “mlx-community/SmolLM2-1.7B-Instruct-4bit” (HTTP 401).87- `mlx-community/SmolLM2-360M-Instruct-4bit`: failed at *resolve*: Hugging Face has no public repository named “mlx-community/SmolLM2-360M-Instruct-4bit” (HTTP 401).8889## Sample output9091First turn of the verification chat ("My name is Sam and my favourite number is 42. Greet me by name in one short sentence."), temperature 0.9293- `Qwen3-0.6B-4bit`: Hello, I'm Sam, and my favorite number is 42! 😊94- `Qwen3-1.7B-4bit`: Hello, Sam! You love the number 42.95- `Qwen3-4B-4bit`: Hello, Sam! It's great to see you. Your favorite number is 42.96- `Qwen2.5-0.5B-Instruct-4bit`: Hello, my dear Sam! Welcome to my home, and I am glad to meet you.97- `gemma-3-1b-it-qat-4bit`: Sam, hello!98- `gemma-3-4b-it-qat-4bit`: Hello Sam, it’s great to meet you – and a big fan of 42 too!99- `Llama-3.2-1B-Instruct-4bit`: Hello Sam.100- `Llama-3.2-3B-Instruct-4bit`: Hi Sam, nice to meet you, and I see you're a fan of the iconic number 42!101- `Phi-4-mini-instruct-4bit`: Hello Sam!102- `SmolLM2-135M-Instruct`: Sam, welcome to my little corner of the world. I'm glad you're here.103- `SmolLM2-135M-Instruct-8bit`: Sam, welcome to my little corner of the world. I'm glad you're here.104- `SmolLM2-360M-Instruct`: Hello, my name is Sam.105- `SmolLM2-360M-Instruct-6bit`: Sam106- `SmolLM2-1.7B-Instruct`: "Hello Sam, nice to meet you!"107108## Not yet verified on a physical iPhone109110Nothing in this document was run on an iPhone or iPad. In particular, the following are unknown until a device run:111112- Whether each model loads and generates on iOS at all (MLX does not run in the iOS Simulator, so there is no simulator result either).113- Tokens per second, time to first token and peak memory on any iPhone. The Mac figures above come from a machine with far more memory bandwidth and must not be shown as iPhone figures.114- Behaviour under the iOS memory limit (jetsam): the `minimumDeviceMemoryGB` and `estimatedRuntimeMemoryBytes` fields of the catalog are estimates computed from the weight size, a KV-cache formula at 4 096 tokens and a fixed overhead. They are not measurements.115- Thermal behaviour during long generations, and the effect of the thermal policy.116- Downloads on a cellular connection, and resuming after iOS suspends the app (downloads use a foreground session).117- Whether the larger models (2 GB and more of weights) are usable on 8 GB devices in practice.118119## Apple Foundation Models (system model): verified on this Mac only120121- **macOS, verified.** On this Mac (macOS 27.0) the live test `appleSystemModelStreamsAMultiTurnAnswer` (`ASTRA_APPLE_FM_TESTS=1`) ran on 2026-11-22: a three-turn conversation replayed through a `Transcript` streamed an answer that recalled a fact from the first turn, ended with one `.messageCompleted`, and the framework reported token usage (86 input, 7 output), which the adapter passed on. Cancellation with the system model was not exercised.122- **iOS Simulator, failed.** A generation was attempted in the iOS Simulator (iOS 26.5 runtime, Mac host on macOS 27) on 2026-11-21. `SystemLanguageModel.default.availability` reported `.available`, yet the generation failed with `LanguageModelSession.GenerationError` code -1. Availability in the Simulator is therefore not a reliable signal; the app now says so in its error message when an unrecognised failure happens there.123- **iPhone / iPad, NOT verified.** Nothing was run on an Apple Intelligence-eligible device. It needs one.124125The `verified` record of each catalog entry says `"platform": "macOS"` for this reason. It should be replaced by an iOS record only after a real run on the device it names.126
Files main
Menu

Theme