spb/zyquo-local Public MIT

1<!--2  MLX-RESEARCH.md3  Zyquo Local45  Author: Simon-Pierre Boucher6  Mail: contact@spboucher.ai7-->89# MLX Swift Stack Research — Phase 0.A1011Researched 2026-07-30 against live official sources (raw file fetches at exact12tags, GitHub releases API) plus empirical build tests on this machine13(M5 Max, 48 GB, macOS 27.0 beta, Swift 6.4 CLT).1415## 1. Packages and SPM coordinates1617| Package | Repo | Version used | Products we link |18|---|---|---|---|19| mlx-swift | `https://github.com/ml-explore/mlx-swift` | **0.31.6** (2026-07-02) | (transitive) MLX, MLXNN, MLXFast… |20| mlx-swift-lm | `https://github.com/ml-explore/mlx-swift-lm` | **3.31.4** (2026-06-30) | `MLXLLM`, `MLXLMCommon`, `MLXHuggingFace` |21| swift-huggingface | `https://github.com/huggingface/swift-huggingface` | **0.9.0** | `HuggingFace` |22| swift-transformers | `https://github.com/huggingface/swift-transformers` | **1.3.x** (1.3.3) | `Tokenizers` |2324Key facts:2526- The LLM layer (MLXLLM/MLXLMCommon/MLXVLM) **moved out of mlx-swift-examples**27  into `mlx-swift-lm`. mlx-swift-examples is frozen for libraries (last 2.29.1).28- **mlx-swift-lm 3.x is a breaking major**: it no longer depends on29  swift-transformers/Hub. `Downloader`, `Tokenizer`, `TokenizerLoader` are now30  protocols in MLXLMCommon; concrete impls come from the `MLXHuggingFace`31  macro library (`#huggingFaceTokenizerLoader()`, `#hubDownloader()`) backed by32  swift-huggingface + swift-transformers, which the app must declare itself.33- mlx-swift 0.31.5+ tags require a Swift ≥6.3 toolchain34  (`swift-tools-version: 6.3;(experimentalCGen)`). We have Swift 6.4 → fine.35- Minimum platform: macOS 14. Our `LSMinimumSystemVersion` = 14.0, arm64 only.3637## 2. Loading, generating, streaming — verified API (tag 3.31.4)3839Load from a **local directory** (our only load path — downloads are ours):4041```swift42import MLXLLM, MLXLMCommon, MLXHuggingFace, Tokenizers4344let container: ModelContainer = try await loadModelContainer(45    from: modelDirectoryURL,46    using: #huggingFaceTokenizerLoader()47)48```4950- `LLMModelFactory._load` reads `config.json`, dispatches on `model_type` via51  `LLMTypeRegistry`, honors `generation_config.json` EOS overrides, and52  `loadWeights` enumerates **every `*.safetensors` recursively** (the shard53  index json is not consulted).54- There is **no progress callback for local-directory loads** — model load55  progress in UI is indeterminate (spinner + elapsed), not a percent bar.5657Multi-turn with KV-cache reuse — `ChatSession` (MLXLMCommon):5859```swift60let session = ChatSession(container,61    instructions: systemPrompt,62    generateParameters: params,63    history: restoredMessages)                       // rehydration supported64for try await g in session.streamDetails(to: prompt) {65    switch g {66    case .chunk(let text): ...67    case .info(let info):  ... // stats68    case .toolCall:        break69    }70}71session.clear()   // resets history + KV cache, keeps instructions72```7374KV cache is held inside the session across turns (no re-prefill). ChatSession75is NOT thread-safe; `ModelContainer` is Sendable. One session per conversation,76owned by the `InferenceEngine` actor. Changing system prompt/params → rebuild77the session from persisted history (KV cache is lost, acceptable).7879Lower level (available if needed): `container.prepare(input:)` +80`container.generate(input:parameters:) -> AsyncStream<Generation>`,81`generateTask(...)` for deterministic early-stop cleanup, `TokenIterator`.8283`Generation` events: `.chunk(String)`, `.info(GenerateCompletionInfo)`,84`.toolCall(ToolCall)`. Stop reasons: `.stop`, `.length`, `.cancelled`.8586### GenerateParameters (exact fields, verified)8788`maxTokens: Int?`, `maxKVSize: Int?`, `kvBits: Int?` (4/8), `kvGroupSize`,89`quantizedKVStart`, `kvScheme: String?`, `temperature: Float` (default 0.6,900 = greedy), `topP: Float` (1.0), `topK: Int` (0), `minP: Float` (0),91`seed: UInt64?`, `repetitionPenalty: Float?`, `repetitionContextSize` (20),92presence/frequency penalties + context sizes, `prefillStepSize` (512).9394### Stats — GenerateCompletionInfo (final `.info` stream element)9596`promptTokenCount`, `generationTokenCount`, `promptTime`, `generateTime`,97`stopReason`, computed `promptTokensPerSecond`, `tokensPerSecond`.98TTFT = time from send to first `.chunk` (we measure it ourselves).99100### Unload101102No explicit API: release all `ChatSession`/`ModelContainer` references, then103`MLX.GPU.clearCache()`. Verified guidance from the repo's own reference docs.104105## 3. Model directory format (required files)106107| File | Required | Notes |108|---|---|---|109| `config.json` | ✅ | `model_type`, quantization block, EOS ids |110| `*.safetensors` | ✅ | all shards, MLX-convention (scales/biases for quant) |111| `tokenizer.json` + `tokenizer_config.json` | ✅ | tokenizer + chat template |112| `model.safetensors.index.json` | ⬜ not read by Swift stack; keep anyway |113| `chat_template.jinja` / `*.jinja` | ⬜ newer HF layout |114| `generation_config.json` | ⬜ EOS override |115| `special_tokens_map.json` | ⬜ |116117Download filter (mirrors the package's own): `*.safetensors`, `*.json`,118`*.jinja`. `DownloadManager` downloads exactly this set.119120## 4. Supported `model_type` architectures (LLMTypeRegistry @ 3.31.4)121122`mistral, mixtral, llama, phi, phi3, phimoe, gemma, gemma2, gemma3,123gemma3_text, gemma3n, gemma4, gemma4_unified, gemma4_text, qwen2, qwen3,124qwen3_moe, qwen3_next, qwen3_5, qwen3_5_moe, qwen3_5_text, minicpm,125starcoder2, cohere, openelm, internlm2, deepseek_v3, granite,126granitemoehybrid, mimo, mimo_v2_flash, minimax, glm4, glm4_moe,127glm4_moe_lite, acereason, falcon_h1, bitnet, smollm3, ernie4_5, lfm2,128baichuan_m1, exaone4, gpt_oss, lille-130m, olmoe, olmo2, olmo3, bailing_moe,129lfm2_moe, nanochat, nemotron_h, afmoe, jamba, mamba2, mistral3, apertus,130nemotron_labs_diffusion`131132This list ships in the app (`SupportedArchitectures`) to warn before133downloading an unsupported model (checked against `config.model_type` from the134Hub API with `config=true`).135136## 5. The no-Xcode build question — RESOLVED EMPIRICALLY137138Upstream docs claim "SwiftPM (command line) cannot build the Metal shaders…139use xcodebuild". **This is outdated for Swift 6.4**: the new swift-build140backend runs `CompileMetalFile` on mlx-swift's `.metal` resources. Verified on141this machine:1421431. CLT-only `swift build` fails with `unable to spawn process 'metal'` —144   the compiler is looked up **via `PATH`** (proven with a logging stub).1452. With `~/Developer/Metal.xctoolchain/usr/bin` (Metal Toolchain component146   v17.5.188, copied from cluster node M4M36) prepended to `PATH`,147   `swift build` succeeds and the binary executes on GPU (`Device(gpu, 0)`).1483. Most kernels are JIT-compiled at runtime by the Metal framework (no149   toolchain needed at runtime); AOT kernels (gemv, SDPA, norms, rope…) are150   compiled at build time into the `mlx-swift_Cmlx` resource bundle. Phase 2's151   PoC (real attention workload) re-confirms the metallib is bundled/found.152153**Recipe** (full details + SwiftUI SDK pin in `docs/BUILD.md`): plain154`swift build` + Metal toolchain on PATH + `SDKROOT=MacOSX26.5.sdk`.155No Xcode IDE, no `.xcodeproj`, `xcodebuild` not needed.156157For the `.app` bundle, the `mlx-swift_Cmlx.bundle` resource bundle must be158copied into `Contents/Resources/` (Makefile handles it); for bare CLI runs159from `.build/`, the bundle sits next to the executable so it is found.160161## 6. Hub downloads: decision162163**Custom `URLSession`-based `DownloadManager`** (per-file pause/resume/cancel,164HTTP Range resume across restarts, app-managed model folder), NOT165swift-huggingface's snapshot API (no per-file pause handle, imposes HF cache166layout). swift-transformers is still used for **tokenization** (non-negotiable167— it is what the MLX stack expects, via `#huggingFaceTokenizerLoader()`).168Full HF HTTP API contract documented in `docs/MODELS.md` §1.169170## 7. Memory model171172APIs (verified @ mlx-swift 0.31.6): `MLX.GPU.activeMemory / cacheMemory /173peakMemory / snapshot() / resetPeakMemory() / set(cacheLimit:) /174set(memoryLimit:relaxed:) / clearCache() / deviceInfo()` (architecture,175memorySize, maxRecommendedWorkingSetSize).176177Sizing rules used by `MemoryAdvisor`:178179- Weights ≈ `params × bits/8 × 1.08` (scales/biases overhead).180- KV cache ≈ `2 × layers × kvHeads × headDim × contextTokens × 2 bytes` (fp16);181  `kvBits` 8/4 halves/quarters it; `maxKVSize` caps it (rotating cache).182- +10–20 % working set for activations; macOS GPU-wired ceiling ≈ 70–75 % of183  unified RAM → verdicts: **Fits** if weights+KV ≤ 60 % of physical RAM,184  **Tight** ≤ 75 %, **Too large** above.185- Physical RAM via `sysctl hw.memsize`; live footprint via `GPU.snapshot()`.186187## 8. Open flags188189- mlx-swift #430 (metallib via build plugin) and #416 (`setMetallibPath`) were190  open as of 2026-07-30 — not relied upon.191- API surface verified at tag 3.31.4; `main` has drifted (MLXFoundationModels,192  MLXGuidedGeneration) — re-verify before any version bump.193