# Zyquo MLX — MLX Framework Research (Ground Truth 2026-07-30) > Phase 0.A research document. Every version number and API name below was > verified against live sources (GitHub releases API, raw source files at > `main`, PyPI JSON, official docs) on 2026-07-30, plus an empirical build test > on this machine. Items not verifiable against a primary source are marked > *UNVERIFIED*. **Version snapshot (live-verified):** | Component | Version | Date | Source | |---|---|---|---| | `mlx` (core, PyPI/GitHub) | **0.32.0** | 2026-07-07 | github.com/ml-explore/mlx | | `mlx-swift` | **0.31.6** | 2026-07-02 | github.com/ml-explore/mlx-swift | | `mlx-swift-lm` | **3.31.4** | 2026-06-30 | github.com/ml-explore/mlx-swift-lm | | `mlx-lm` (PyPI) | **0.31.3** | 2026-04-22 | pypi.org/project/mlx-lm | | `swift-transformers` | **1.3.3** | 2026-05-16 | github.com/huggingface/swift-transformers | | `swift-huggingface` (HubClient) | **0.9.0** | — | github.com/huggingface/swift-huggingface | | `mlx-vlm` (PyPI) | **0.6.8** | 2026-07-27 | github.com/Blaizzy/mlx-vlm | | `mlx-whisper` (PyPI) | **0.4.3** | 2025-08-29 | ml-explore/mlx-examples/whisper | | `mlx-audio` (PyPI) | **0.4.6** | 2026-07-25 | github.com/Blaizzy/mlx-audio | --- ## 1. What MLX Is — Core Primitives - **An array framework for ML on Apple Silicon** from Apple ML research: NumPy-like `mlx.core`, NN modules `mlx.nn`, `mlx.optimizers`. C++ core with a Metal backend (a CUDA/Linux backend also exists now — visible in mlx-swift's `Package.swift`, which conditionally compiles `mlx/backend/cuda` on Linux). - **Lazy evaluation & `eval()`**: operations record a compute graph; nothing executes until `mx.eval(...)` or an implicit eval (printing, `.item()`, NumPy conversion, `mx.save`). Laziness enables transformations (`grad`, `vmap`, `compile`), dead-output elimination, and loading weights without materializing intermediates. Guidance: eval at outer-loop boundaries (after an optimizer step); graphs of "tens to thousands of ops" per eval are the sweet spot. (ml-explore.github.io/mlx — lazy evaluation guide, fetched.) - **Unified memory**: arrays live in memory shared by CPU and GPU. There is no data placement — only *operation* placement via `stream:`/`device:` (`mx.gpu` / `mx.cpu`). The scheduler manages cross-stream dependencies automatically. (Unified-memory guide, fetched.) - **Autodiff**: `mx.grad` / `mx.value_and_grad`, composable with `mx.vmap` and `mx.compile`. Module-aware: `nn.value_and_grad(model, loss_fn)`; Swift: `MLXNN.valueAndGrad(model:_:)` (3 public overloads, verified in source). - **Dispatch to Metal**: the core dispatches lazily-built graphs to precompiled + runtime-JIT Metal kernels (`backend/metal/kernels`, `jit_kernels.cpp`, `default.metallib`); quantized matmul and scaled-dot-product-attention are dedicated kernels. --- ## 2. Swift vs. Python — The Honest Capability Split (2026-07-30) **Major reorg (critical):** `MLXLLM`, `MLXVLM`, `MLXLMCommon`, `MLXEmbedders` moved out of `mlx-swift-examples` into **`ml-explore/mlx-swift-lm`** (3.x). 3.x decoupled downloads/tokenizers into protocols (`Downloader`, `Tokenizer`, `TokenizerLoader`) with concrete impls supplied by the **`MLXHuggingFace` macros** (`#hubDownloader()`, `#huggingfaceTokenizerLoader()`, `#huggingFaceLoadModelContainer(...)`) backed by `swift-huggingface` + `swift-transformers`. `mlx-swift-examples` (renamed package `mlx-libraries`) now only ships `StableDiffusion` and `MLXMNIST` + example apps. | Capability | Native Swift today | Python | Zyquo MLX strategy | |---|---|---|---| | LLM inference (streaming, KV cache, chat) | ✅ MLXLLM/MLXLMCommon (`ChatSession`, `ModelContainer`, speculative decoding, KV-cache quantization) | mlx-lm | **Swift** | | VLM inference | ✅ MLXVLM (smaller zoo; no omni/audio) | mlx-vlm (broader) | **Swift**; Py optional for exotic/omni | | Embeddings | ✅ MLXEmbedders | mlx-embeddings (thin, v0.1.0) | **Swift** | | Speech (STT/TTS) | ❌ no official Swift package | ✅ mlx-whisper / mlx-audio | **Python bridge** | | Image generation | ✅ `StableDiffusion` (SD 2.1 / SDXL-turbo) in mlx-swift-examples; FLUX ❌ | ✅ mlx-examples flux + SD scripts | Swift for SD; **Python bridge for FLUX** | | LoRA/DoRA training | ✅ `LoRATrain` (MLXLLM) — real but text-only datasets, no grad-checkpoint/accum, no numbered checkpoints | ✅ mlx_lm.lora (chat/completions datasets, mask_prompt, full/dora, grad checkpoint/accum, checkpoints, schedules) | **Python bridge primary** (see TRAINING-RESEARCH.md §8.3) | | QLoRA (quantized base) | ✅ `QLoRALinear` | ✅ implicit when base is quantized | Both | | Adapter interchange | ✅ `LoRAContainer.from(directory:)` — documented mlx-lm-compatible (`adapter_config.json` + `adapters.safetensors`) | native | Interoperable both ways | | Full fine-tuning | ❌ no packaged pipeline (primitives exist) | ✅ `--fine-tune-type full` | **Python bridge** | | Quantization + HF→MLX conversion | ✅ **new native Swift**: `LLMModelFactory.convert(from:to:options:)` (safetensors input only, LLMs; affine/mxfp4) | ✅ mlx_lm.convert (+ AWQ/DWQ/GPTQ/dynamic, mixed recipes, `.bin`, GGUF export, VLM convert) | Swift for standard LLM convert/quant; **Python for advanced** | | Fuse adapter → disk | ✅ in-memory fuse; packaged fuse-to-disk one-liner UNVERIFIED | ✅ mlx_lm.fuse (+ `--dequantize`, GGUF) | **Python safest**; Swift feasible | | Evaluation (ppl / lm-eval) | partial (`LoRATrain.evaluate` loss) | ✅ mlx_lm.evaluate (lm-eval), mlx_lm.perplexity, `--test` | **Python bridge** | | OpenAI-compatible server | ❌ | ✅ mlx_lm.server / mlx_vlm.server | out of scope v1 | --- ## 3. Model Types & the Current MLX Zoo ### 3.1 Swift-native architectures (verified from factory registries at main) - **LLM** (`LLMTypeRegistry`, keyed by `config.json` `model_type`): llama, mistral, mixtral, mistral3, phi, phi3, phimoe, gemma, gemma2, gemma3/gemma3_text, gemma3n, gemma4/gemma4_unified/gemma4_text, qwen2, qwen3, qwen3_moe, qwen3_next, qwen3_5, qwen3_5_moe, qwen3_5_text, smollm3, deepseek_v2, deepseek_v3, glm4, glm4_moe, glm4_moe_lite, gpt_oss, granite, granitemoehybrid, cohere, starcoder2, openelm, internlm2, minicpm, mimo, minimax, falcon_h1, bitnet, ernie4_5, lfm2, lfm2_moe, exaone4, olmoe, olmo2, olmo3, bailing_moe, nemotron_h, jamba, mamba2, apertus, and more (Qwen2.5 = `qwen2`; DeepSeek-R1 distills use their base arch). - **VLM** (`VLMModelFactory`): paligemma, qwen2_vl, qwen2_5_vl, qwen3_vl, qwen3_5, qwen3_5_moe, idefics3, gemma3, gemma4, gemma4_unified, smolvlm, fastvlm (llava_qwen2), pixtral, mistral3, lfm2_vl, glm_ocr. - **Embedders** (`EmbedderTypeRegistry`): bert, roberta, xlm-roberta, distilbert, nomic_bert, qwen3 (Qwen3-Embedding), lfm2, gemma3/gemma3n (EmbeddingGemma). Curated `EmbedderRegistry` includes bge family, MiniLM, nomic-embed-text v1/v1.5, e5, mixedbread, Qwen3-Embedding. - **Image gen**: `StableDiffusion` product (SD 2.1, SDXL-turbo). - **Bonus (new since 2025, all verified in mlx-swift-lm):** native conversion + quantization (`ModelConversion.swift`), LoRA/DoRA adapters + fusing, LoRA training, speculative decoding + MTP drafters, KV-cache quantization (incl. "TurboQuant"), guided generation (`MLXGuidedGeneration`, JSON Schema/EBNF via vendored xgrammar), tool calling with per-family parsers, prompt-cache persistence, `MLXFoundationModels` bridge (macOS 27 SDK). ### 3.2 Python-first packages | Package | What | Interface | |---|---|---| | mlx-lm 0.31.3 | LLM inference/serve/convert/quant/train | CLIs: `mlx_lm.generate/chat/convert/fuse/lora/server/evaluate/manage/benchmark/cache_prompt/perplexity/upload`; quant CLIs `mlx_lm.awq/dwq/gptq/dynamic_quant`. Python: `load()`, `generate()`, `stream_generate()` (yields `GenerationResponse{text, token, prompt_tps, generation_tps, peak_memory, finish_reason}`), `make_sampler(...)` | | mlx-vlm 0.6.8 | VLM + omni (audio/video) inference & LoRA | `mlx_vlm.generate/chat/convert/server`; huge zoo (Qwen-VL/3.5, gemma 3/4, DeepSeek-OCR, Moondream, MiniCPM-V/o, Pixtral, LLaVA, Idefics…) | | mlx-whisper 0.4.3 | STT | `mlx_whisper audio.mp3 [-f fmt --model repo]`; API `mlx_whisper.transcribe()`; needs ffmpeg | | mlx-audio 0.4.6 | TTS/STT/STS | `mlx_audio.tts.generate`, `mlx_audio.stt.generate`, `mlx_audio.convert`, `mlx_audio.server` | | mlx-examples (repo) | SD + FLUX image gen (script-based), FLUX LoRA (dreambooth.py) | `stable_diffusion/txt2image.py`, `flux/txt2image.py --model schnell\|dev` | --- ## 4. Inference API Specifics (Swift-first, exact names from source) ### 4.1 Loading (mlx-swift-lm 3.x) ```swift // Free functions in MLXLMCommon (ModelFactory.swift): loadModelContainer(from: any Downloader, using: any TokenizerLoader, configuration: ModelConfiguration, progressHandler: @Sendable (Progress) -> Void) async throws -> ModelContainer loadModelContainer(from: any Downloader, using: any TokenizerLoader, id: String, revision: String = "main", …) async throws -> ModelContainer loadModelContainer(from directory: URL, using: any TokenizerLoader) async throws -> ModelContainer // pure-local — ideal for our library // Factory-scoped: LLMModelFactory.shared.loadContainer(...) / VLMModelFactory.shared... // Macros (MLXHuggingFace): #huggingFaceLoadModelContainer(configuration:), #hubDownloader(), #huggingfaceTokenizerLoader() ``` `ModelContainer` is a `Sendable final class` wrapping `ModelContext` (model + tokenizer + processor) with `perform { }` / `update { }` access; exposes `configuration`, `tokenizer`, `processor`, `modelDirectory`, `prepare(input:) -> LMInput`, `generate(input:parameters:) -> AsyncStream`, `encode/decode`, `applyChatTemplate(messages:)`. High-level chat: `ChatSession(model)` → `respond(to:)` / `streamResponse(to:)` / `streamDetails(...)`, `clear()`, `saveCache(to:)`; supports instructions, tools, speculative decoding. ### 4.2 Streaming generation (core path) ```swift generate(input: LMInput, cache: [KVCache]? = nil, parameters: GenerateParameters, context: ModelContext, …) throws -> AsyncStream enum Generation { case chunk(String); case info(GenerateCompletionInfo); case toolCall(ToolCall) } struct GenerateCompletionInfo { promptTokenCount: Int; generationTokenCount: Int promptTime: TimeInterval // → TTFT generateTime: TimeInterval // → tok/s stopReason: GenerateStopReason // .stop / .length / .cancelled // + speculative-decoding stats } ``` Callback-based `generate` overloads are **deprecated** in favor of `AsyncStream`. Cancellation = cancel the Task / terminate the stream (`generateTask` returns `(AsyncStream, Task)`). Token-level: `generateTokens(...) -> AsyncStream`. Per-run stats for the Playground (tok/s, TTFT) come from `GenerateCompletionInfo`. ### 4.3 GenerateParameters (verified fields) `prefillStepSize`, `maxTokens`, `maxKVSize` (→ `RotatingKVCache`), `kvBits` + `kvGroupSize` (default 64) + `quantizedKVStart`, `kvScheme` ("affine4"/"affine8"/"turbo8v3"…), `temperature`, `topP`, `topK`, `minP`, `seed: UInt64?` (nil = entropy; global RNG via `MLXRandom.seed`), `repetitionPenalty` (+ context size). ### 4.4 Tokenizers & chat templates `MLXLMCommon.Tokenizer` protocol: `encode(text:addSpecialTokens:)`, `decode(tokenIds:skipSpecialTokens:)`, `bosToken/eosToken`, `applyChatTemplate(messages:tools:additionalContext:) throws -> [Int]`. Concrete impl from swift-transformers via the MLXHuggingFace macro adapter. Messages: `Chat.Message` (`.system/.user/.assistant/.tool`, with `images:/videos:/audios:`) → `UserInput` → `processor.prepare(input:)` → `LMInput`. VLM: `UserInput(chat: [.user("Describe", images: [.url(fileURL)])], processing: .init(resize: …))`. ### 4.5 KV cache `KVCache` protocol; `KVCacheSimple`, `RotatingKVCache`, `QuantizedKVCache`, `ChunkedKVCache`, `MambaCache`, `CacheList`; persistence via `savePromptCache(...)` / `loadPromptCache(...)`. ### 4.6 Memory management (RENAMED — important) `MLX.GPU.set(cacheLimit:)` & friends are **deprecated**; current API is the **`MLX.Memory`** enum (`Source/MLX/Memory.swift`): ```swift Memory.cacheLimit = 2 * 1024 * 1024 // deprecated alias: GPU.set(cacheLimit:) Memory.memoryLimit = … // default 1.5× device recommended working set Memory.activeMemory; Memory.cacheMemory; Memory.peakMemory let snap = Memory.snapshot() // Snapshot{activeMemory, cacheMemory, peakMemory}, .delta(other) Memory.clearCache() // free all cached buffers GPU.deviceInfo() // architecture, maxBufferSize, maxRecommendedWorkingSetSize, memorySize GPU.resetPeakMemory() ``` **Unload pattern (drives Playground/Engine):** release all `ModelContainer` references → `Memory.clearCache()` → verify with `Memory.snapshot()` deltas. Doc comment (verified): small cache limits (~2 MB) often perform as well as unconstrained — expose in Settings › Compute. ### 4.7 Embeddings (MLXEmbedders 3.x) ```swift let container = try await EmbedderModelFactory.shared.loadContainer( from: #hubDownloader(), using: #huggingfaceTokenizerLoader(), configuration: EmbedderRegistry.nomic_text_v1_5) let vectors: [[Float]] = await container.perform { model, tokenizer, pooling in let ids = MLXArray(tokenizer.encode(text: text, addSpecialTokens: true))[.newAxis, 0...] let out = model(ids, positionIds: nil, tokenTypeIds: nil, attentionMask: nil) return pooling(out, normalize: true).asArray(Float.self) // Pooling.Strategy: .mean/.cls/.first/.last/.max/.none } ``` ### 4.8 Training primitives in Swift `MLXNN.valueAndGrad`; `MLXOptimizers`: SGD, RMSprop, AdaGrad, AdaDelta, Adam, AdamW, Adamax, Lion, Adafactor, Muon, MultiOptimizer; schedulers: `exponentialDecay`, `stepDecay`, `cosineDecay`, `linearSchedule`, `joinSchedules`. `MLX.QuantizationMode`: `.affine`, `.mxfp4`. Details of `LoRATrain` in `docs/TRAINING-RESEARCH.md` §8. --- ## 5. Quantization & Conversion ### 5.1 MLX core quantization (verified docs) `mx.quantize(w, group_size=None, bits=None, mode='affine', *, global_scale=None)` — **affine**: group sizes 32/64/128, bits **2,3,4 (default),5,6,8**, scale+bias per group. **mxfp4/mxfp8**: E8M0 scales (group 32); **nvfp4**: E4M3 scale (group 16). **No NF4 mode exists.** ### 5.2 `mlx_lm.convert` (verified argparse at main) ``` mlx_lm.convert --hf-path [--mlx-path mlx_model] [-q] [--q-bits N] [--q-group-size G] [--q-mode affine|mxfp4|nvfp4|mxfp8] [--quant-predicate mixed_2_6|mixed_3_4|mixed_3_6|mixed_4_6] [--dtype float16|bfloat16|float32] [-d/--dequantize] [--upload-repo] [--trust-remote-code] ``` Mode defaults: affine → (group 64, 4 bits); mxfp4 → (32, 4); nvfp4 → (16, 4); mxfp8 → (32, 8). Mixed recipes are llama.cpp-Q4_K_M-style (affine only). Advanced (beyond round-to-nearest): **`mlx_lm.awq`, `mlx_lm.dwq`, `mlx_lm.gptq`, `mlx_lm.dynamic_quant`** — first-class CLIs. `convert()` also exists as a Python API. Swift-native equivalent: ```swift let result = try await LLMModelFactory.shared.convert( from: directory, to: outputDirectory, options: .init(bits: 4, groupSize: 64, mode: .affine), progressHandler: { p in /* ModelConversionProgress{stage, fractionCompleted, message} */ }) // safetensors input only; PyTorch .bin explicitly out of scope; LLMs only ``` ### 5.3 On-disk MLX model format (verified `save_model` in utils.py) - `config.json` with top-level `"quantization": {"group_size", "bits", "mode"}` (+ per-layer overrides for mixed/partial quantization). - Weights: `model.safetensors` or sharded `model-0000X-of-0000N.safetensors` (**max 5 GB/shard**), each with safetensors metadata `{"format": "mlx"}`. - `model.safetensors.index.json` with `metadata.total_size`, `metadata.total_parameters`, `weight_map`. - Tokenizer files (`tokenizer.save_pretrained`), copied `*.py` + `generation_config.json`, and an mlx-tagged `README.md` model card. → `ModelStore` validates a local model dir by checking: `config.json` (`model_type`, optional `quantization`), weight file(s) + index consistency, tokenizer files present. --- ## 6. Build & Execution Strategy (resolved — details in `docs/BUILD.md`) **Build (empirically tested on this Mac, 2026-07-30):** `swift build` with CLT only compiles all C++/Swift then **fails** at `CompileMetalFile steel_attention.metal` — `unable to spawn process 'metal'`. The CLT ships no Metal compiler; no prebuilt-metallib distribution of mlx-swift exists yet (PR #430 / mlx#3597 / mlx-swift#416 in flight, not landed). The Metal Toolchain is an Xcode 26+ downloadable component (`xcodebuild -downloadComponent metalToolchain`) and cannot be installed with CLT alone. **→ Adapted rule per charter 0.A.6: install full Xcode.app strictly as a toolchain, drive everything via command-line `xcodebuild`/`swift build` from the Makefile. No IDE. No hand-authored `.xcodeproj`.** The app bundle must ship the **`mlx-swift_Cmlx.bundle`** resource bundle (contains `default.metallib`); existence proof of a signed/notarized xcodebuild-built mlx-swift app: sfomuseum/Docent (mlx-swift issue #345). **Execution model (the 0.A.6 decision):** 1. **Native Swift (in-process)** — LLM/VLM/embeddings inference, Stable Diffusion image gen, standard HF→MLX conversion + affine/mxfp4 quantization of safetensors LLMs, adapter load/unload/fuse in memory. 2. **Embedded Python venv (out-of-process via `PythonRunner`)** — fine-tuning (LoRA/QLoRA/DoRA/full via `mlx_lm.lora` API with a custom `TrainingCallback` → JSON-lines protocol), adapter fuse-to-disk + GGUF export, advanced quantization (AWQ/DWQ/GPTQ/mixed), `.bin` conversions, Whisper STT, mlx-audio TTS, FLUX image gen, evaluation (`--test` ppl, `mlx_lm.evaluate`). - Venv: **Python 3.12 pinned** (mlx wheels are cp310–cp314, macOS ≥ 14 arm64-only — aligns with the Apple-Silicon gate), provisioned with **uv** into `~/Library/Application Support/ZyquoMLX/py/` on first run (never inside the signed bundle; never touching system Python). Pins: `mlx-lm==0.31.3`, `mlx-vlm==0.6.8`, `mlx-whisper==0.4.3`, `mlx-audio==0.4.6` (+ lockfile via `uv pip compile`); uv supports `--offline` + local wheel cache for repair-offline behavior. - Progress protocol: our own JSON lines on stdout emitted by pinned helper scripts (`PyBridge/scripts/`) — **never scrape mlx-lm's stdout** (format changed between 0.31.3 and main; see TRAINING-RESEARCH.md §5). --- ## 7. Memory & Performance - **Inference RAM ≈ weights + KV cache + overhead.** Weights: 4-bit affine gs64 ≈ 0.5625 B/param (verified against real repo sizes, see MODELS.md); 8-bit ≈ 1.07; bf16 ≈ 2.0. KV cache (fp16): `2 × layers × kvHeads × headDim × seqLen × 2 bytes` — e.g. Qwen3-8B (36L, 8 KV heads, 128 dim) at 8k ctx ≈ 1.2 GB; quantizable via `kvBits`/`kvScheme`. (Formula standard practice; per-model numbers *UNVERIFIED* against one official doc — `MemoryAdvisor` computes from config.json fields and calibrates in Phase 7.) - **`Memory.memoryLimit` defaults to 1.5× `maxRecommendedWorkingSetSize`** (verified doc comment); macOS caps GPU-visible memory ≈ 66–75% of unified RAM — query `GPU.deviceInfo().maxRecommendedWorkingSetSize` at runtime for exact gating rather than hardcoding. - **Training memory**: LoRA ≈ 2–4× the inference footprint of the same base (optimizer state + activations); QLoRA on a 4-bit base is the 16–32 GB path; full FT ≈ 8 B/param + activations. Full tables in TRAINING-RESEARCH.md §6 and MODELS.md §3. - **Throughput** (third-party 2026 benchmarks, *indicative, UNVERIFIED first-party*): M4 Max ≈ 62 tok/s on 8B-4bit, ≈ 38 tok/s on 14B-4bit; M4 Air 16 GB ≈ 25–35 tok/s on 9B-4bit; generation is memory-bandwidth-bound. First-party per-run stats come from `GenerateCompletionInfo` / `GenerationResponse`. - **This Mac (dev machine): M5 Max, 48 GB** → comfortable up to ~35B 4-bit inference, QLoRA ≤ 32B, LoRA ≤ 14B, full FT ≤ ~3–4B. --- ## 8. Dependency Decision (feeds Phase 1 `Package.swift`) ```swift .package(url: "https://github.com/ml-explore/mlx-swift", .upToNextMinor(from: "0.31.6")), .package(url: "https://github.com/ml-explore/mlx-swift-lm", .upToNextMajor(from: "3.31.4")), .package(url: "https://github.com/huggingface/swift-huggingface", from: "0.9.0"), .package(url: "https://github.com/huggingface/swift-transformers", from: "1.3.0"), // StableDiffusion (image gen) from mlx-swift-examples ("mlx-libraries") — add in Phase 6 if SD ships in v1 ``` Platform floor: macOS 14 (mlx-swift Package.swift) — Zyquo MLX targets **macOS 14+, arm64 only**. Note: mlx-swift `main` uses `swift-tools-version: 6.3 (experimentalCGen)` — depend on **released tags only**, never branch `main`. ### Primary sources github.com/ml-explore/{mlx, mlx-swift, mlx-swift-lm, mlx-swift-examples, mlx-lm, mlx-examples} (releases API + raw files at main) · ml-explore.github.io/mlx (lazy eval, unified memory, quantize docs) · pypi.org/pypi/{mlx, mlx-lm, mlx-vlm, mlx-whisper, mlx-audio}/json · github.com/huggingface/{swift-transformers, swift-huggingface} · github.com/Blaizzy/{mlx-vlm, mlx-audio} · mlx-swift PR #430, issues #345/#349 · local empirical build test (this Mac, 2026-07-30).