# Zyquo MLX — MLX Framework Research (Ground Truth 2026-07-30)

Phase 0.A research document. Every version number and API name below was verified against live sources (GitHub releases API, raw source files at main, PyPI JSON, official docs) on 2026-07-30, plus an empirical build test on this machine. Items not verifiable against a primary source are marked UNVERIFIED.

Version snapshot (live-verified):

# 1. What MLX Is — Core Primitives

• An array framework for ML on Apple Silicon from Apple ML research: NumPy-like mlx.core, NN modules mlx.nn, mlx.optimizers. C++ core with a Metal backend (a CUDA/Linux backend also exists now — visible in mlx-swift's Package.swift, which conditionally compiles mlx/backend/cuda on Linux).
• Lazy evaluation & eval(): operations record a compute graph; nothing executes until mx.eval(...) or an implicit eval (printing, .item(), NumPy conversion, mx.save). Laziness enables transformations (grad, vmap, compile), dead-output elimination, and loading weights without materializing intermediates. Guidance: eval at outer-loop boundaries (after an optimizer step); graphs of "tens to thousands of ops" per eval are the sweet spot. (ml-explore.github.io/mlx — lazy evaluation guide, fetched.)
• Unified memory: arrays live in memory shared by CPU and GPU. There is no data placement — only operation placement via stream:/device: (mx.gpu / mx.cpu). The scheduler manages cross-stream dependencies automatically. (Unified-memory guide, fetched.)
• Autodiff: mx.grad / mx.value_and_grad, composable with mx.vmap and mx.compile. Module-aware: nn.value_and_grad(model, loss_fn); Swift: MLXNN.valueAndGrad(model:_:) (3 public overloads, verified in source).
• Dispatch to Metal: the core dispatches lazily-built graphs to precompiled + runtime-JIT Metal kernels (backend/metal/kernels, jit_kernels.cpp, default.metallib); quantized matmul and scaled-dot-product-attention are dedicated kernels.

# 2. Swift vs. Python — The Honest Capability Split (2026-07-30)

Major reorg (critical): MLXLLM, MLXVLM, MLXLMCommon, MLXEmbedders moved out of mlx-swift-examples into ml-explore/mlx-swift-lm (3.x). 3.x decoupled downloads/tokenizers into protocols (Downloader, Tokenizer, TokenizerLoader) with concrete impls supplied by the MLXHuggingFace macros (#hubDownloader(), #huggingfaceTokenizerLoader(), #huggingFaceLoadModelContainer(...)) backed by swift-huggingface + swift-transformers. mlx-swift-examples (renamed package mlx-libraries) now only ships StableDiffusion and MLXMNIST + example apps.

# 3. Model Types & the Current MLX Zoo

# 3.1 Swift-native architectures (verified from factory registries at main)

• LLM (LLMTypeRegistry, keyed by config.json model_type): llama, mistral, mixtral, mistral3, phi, phi3, phimoe, gemma, gemma2, gemma3/gemma3_text, gemma3n, gemma4/gemma4_unified/gemma4_text, qwen2, qwen3, qwen3_moe, qwen3_next, qwen3_5, qwen3_5_moe, qwen3_5_text, smollm3, deepseek_v2, deepseek_v3, glm4, glm4_moe, glm4_moe_lite, gpt_oss, granite, granitemoehybrid, cohere, starcoder2, openelm, internlm2, minicpm, mimo, minimax, falcon_h1, bitnet, ernie4_5, lfm2, lfm2_moe, exaone4, olmoe, olmo2, olmo3, bailing_moe, nemotron_h, jamba, mamba2, apertus, and more (Qwen2.5 = qwen2; DeepSeek-R1 distills use their base arch).
• VLM (VLMModelFactory): paligemma, qwen2_vl, qwen2_5_vl, qwen3_vl, qwen3_5, qwen3_5_moe, idefics3, gemma3, gemma4, gemma4_unified, smolvlm, fastvlm (llava_qwen2), pixtral, mistral3, lfm2_vl, glm_ocr.
• Embedders (EmbedderTypeRegistry): bert, roberta, xlm-roberta, distilbert, nomic_bert, qwen3 (Qwen3-Embedding), lfm2, gemma3/gemma3n (EmbeddingGemma). Curated EmbedderRegistry includes bge family, MiniLM, nomic-embed-text v1/v1.5, e5, mixedbread, Qwen3-Embedding.
• Image gen: StableDiffusion product (SD 2.1, SDXL-turbo).
• Bonus (new since 2025, all verified in mlx-swift-lm): native conversion + quantization (ModelConversion.swift), LoRA/DoRA adapters + fusing, LoRA training, speculative decoding + MTP drafters, KV-cache quantization (incl. "TurboQuant"), guided generation (MLXGuidedGeneration, JSON Schema/EBNF via vendored xgrammar), tool calling with per-family parsers, prompt-cache persistence, MLXFoundationModels bridge (macOS 27 SDK).

# 3.2 Python-first packages

# 4. Inference API Specifics (Swift-first, exact names from source)

# 4.1 Loading (mlx-swift-lm 3.x)

// Free functions in MLXLMCommon (ModelFactory.swift):
loadModelContainer(from: any Downloader, using: any TokenizerLoader,
                   configuration: ModelConfiguration,
                   progressHandler: @Sendable (Progress) -> Void) async throws -> ModelContainer
loadModelContainer(from: any Downloader, using: any TokenizerLoader,
                   id: String, revision: String = "main", …) async throws -> ModelContainer
loadModelContainer(from directory: URL, using: any TokenizerLoader)
                   async throws -> ModelContainer   // pure-local — ideal for our library
// Factory-scoped: LLMModelFactory.shared.loadContainer(...) / VLMModelFactory.shared...
// Macros (MLXHuggingFace): #huggingFaceLoadModelContainer(configuration:), #hubDownloader(), #huggingfaceTokenizerLoader()

ModelContainer is a Sendable final class wrapping ModelContext (model + tokenizer + processor) with perform { } / update { } access; exposes configuration, tokenizer, processor, modelDirectory, prepare(input:) -> LMInput, generate(input:parameters:) -> AsyncStream<Generation>, encode/decode, applyChatTemplate(messages:).

High-level chat: ChatSession(model) → respond(to:) / streamResponse(to:) / streamDetails(...), clear(), saveCache(to:); supports instructions, tools, speculative decoding.

# 4.2 Streaming generation (core path)

generate(input: LMInput, cache: [KVCache]? = nil, parameters: GenerateParameters,
         context: ModelContext, …) throws -> AsyncStream<Generation>

enum Generation { case chunk(String); case info(GenerateCompletionInfo); case toolCall(ToolCall) }
struct GenerateCompletionInfo {
    promptTokenCount: Int; generationTokenCount: Int
    promptTime: TimeInterval      // → TTFT
    generateTime: TimeInterval    // → tok/s
    stopReason: GenerateStopReason  // .stop / .length / .cancelled
    // + speculative-decoding stats
}

Callback-based generate overloads are deprecated in favor of AsyncStream. Cancellation = cancel the Task / terminate the stream (generateTask returns (AsyncStream, Task)). Token-level: generateTokens(...) -> AsyncStream<TokenGeneration>. Per-run stats for the Playground (tok/s, TTFT) come from GenerateCompletionInfo.

# 4.3 GenerateParameters (verified fields)

prefillStepSize, maxTokens, maxKVSize (→ RotatingKVCache), kvBits + kvGroupSize (default 64) + quantizedKVStart, kvScheme ("affine4"/"affine8"/"turbo8v3"…), temperature, topP, topK, minP, seed: UInt64? (nil = entropy; global RNG via MLXRandom.seed), repetitionPenalty (+ context size).

# 4.4 Tokenizers & chat templates

MLXLMCommon.Tokenizer protocol: encode(text:addSpecialTokens:), decode(tokenIds:skipSpecialTokens:), bosToken/eosToken, applyChatTemplate(messages:tools:additionalContext:) throws -> [Int]. Concrete impl from swift-transformers via the MLXHuggingFace macro adapter. Messages: Chat.Message (.system/.user/.assistant/.tool, with images:/videos:/audios:) → UserInput → processor.prepare(input:) → LMInput. VLM: UserInput(chat: [.user("Describe", images: [.url(fileURL)])], processing: .init(resize: …)).

# 4.5 KV cache

KVCache protocol; KVCacheSimple, RotatingKVCache, QuantizedKVCache, ChunkedKVCache, MambaCache, CacheList; persistence via savePromptCache(...) / loadPromptCache(...).

# 4.6 Memory management (RENAMED — important)

MLX.GPU.set(cacheLimit:) & friends are deprecated; current API is the MLX.Memory enum (Source/MLX/Memory.swift):

Memory.cacheLimit = 2 * 1024 * 1024   // deprecated alias: GPU.set(cacheLimit:)
Memory.memoryLimit = …                 // default 1.5× device recommended working set
Memory.activeMemory; Memory.cacheMemory; Memory.peakMemory
let snap = Memory.snapshot()           // Snapshot{activeMemory, cacheMemory, peakMemory}, .delta(other)
Memory.clearCache()                    // free all cached buffers
GPU.deviceInfo()                       // architecture, maxBufferSize, maxRecommendedWorkingSetSize, memorySize
GPU.resetPeakMemory()

Unload pattern (drives Playground/Engine): release all ModelContainer references → Memory.clearCache() → verify with Memory.snapshot() deltas. Doc comment (verified): small cache limits (~2 MB) often perform as well as unconstrained — expose in Settings › Compute.

# 4.7 Embeddings (MLXEmbedders 3.x)

let container = try await EmbedderModelFactory.shared.loadContainer(
    from: #hubDownloader(), using: #huggingfaceTokenizerLoader(),
    configuration: EmbedderRegistry.nomic_text_v1_5)
let vectors: [[Float]] = await container.perform { model, tokenizer, pooling in
    let ids = MLXArray(tokenizer.encode(text: text, addSpecialTokens: true))[.newAxis, 0...]
    let out = model(ids, positionIds: nil, tokenTypeIds: nil, attentionMask: nil)
    return pooling(out, normalize: true).asArray(Float.self)
    // Pooling.Strategy: .mean/.cls/.first/.last/.max/.none
}

# 4.8 Training primitives in Swift

MLXNN.valueAndGrad; MLXOptimizers: SGD, RMSprop, AdaGrad, AdaDelta, Adam, AdamW, Adamax, Lion, Adafactor, Muon, MultiOptimizer; schedulers: exponentialDecay, stepDecay, cosineDecay, linearSchedule, joinSchedules. MLX.QuantizationMode: .affine, .mxfp4. Details of LoRATrain in docs/TRAINING-RESEARCH.md §8.

# 5. Quantization & Conversion

# 5.1 MLX core quantization (verified docs)

mx.quantize(w, group_size=None, bits=None, mode='affine', *, global_scale=None) — affine: group sizes 32/64/128, bits 2,3,4 (default),5,6,8, scale+bias per group. mxfp4/mxfp8: E8M0 scales (group 32); nvfp4: E4M3 scale (group 16). No NF4 mode exists.

# 5.2 mlx_lm.convert (verified argparse at main)

mlx_lm.convert --hf-path <repo-or-dir> [--mlx-path mlx_model]
  [-q] [--q-bits N] [--q-group-size G]
  [--q-mode affine|mxfp4|nvfp4|mxfp8]
  [--quant-predicate mixed_2_6|mixed_3_4|mixed_3_6|mixed_4_6]
  [--dtype float16|bfloat16|float32] [-d/--dequantize] [--upload-repo] [--trust-remote-code]

Mode defaults: affine → (group 64, 4 bits); mxfp4 → (32, 4); nvfp4 → (16, 4); mxfp8 → (32, 8). Mixed recipes are llama.cpp-Q4_K_M-style (affine only). Advanced (beyond round-to-nearest): mlx_lm.awq, mlx_lm.dwq, mlx_lm.gptq, mlx_lm.dynamic_quant — first-class CLIs. convert() also exists as a Python API. Swift-native equivalent:

let result = try await LLMModelFactory.shared.convert(
    from: directory, to: outputDirectory,
    options: .init(bits: 4, groupSize: 64, mode: .affine),
    progressHandler: { p in /* ModelConversionProgress{stage, fractionCompleted, message} */ })
// safetensors input only; PyTorch .bin explicitly out of scope; LLMs only

# 5.3 On-disk MLX model format (verified save_model in utils.py)

• config.json with top-level "quantization": {"group_size", "bits", "mode"} (+ per-layer overrides for mixed/partial quantization).
• Weights: model.safetensors or sharded model-0000X-of-0000N.safetensors (max 5 GB/shard), each with safetensors metadata {"format": "mlx"}.
• model.safetensors.index.json with metadata.total_size, metadata.total_parameters, weight_map.
• Tokenizer files (tokenizer.save_pretrained), copied *.py + generation_config.json, and an mlx-tagged README.md model card.

→ ModelStore validates a local model dir by checking: config.json (model_type, optional quantization), weight file(s) + index consistency, tokenizer files present.

# 6. Build & Execution Strategy (resolved — details in docs/BUILD.md)

Build (empirically tested on this Mac, 2026-07-30): swift build with CLT only compiles all C++/Swift then fails at CompileMetalFile steel_attention.metal — unable to spawn process 'metal'. The CLT ships no Metal compiler; no prebuilt-metallib distribution of mlx-swift exists yet (PR #430 / mlx#3597 / mlx-swift#416 in flight, not landed). The Metal Toolchain is an Xcode 26+ downloadable component (xcodebuild -downloadComponent metalToolchain) and cannot be installed with CLT alone. → Adapted rule per charter 0.A.6: install full Xcode.app strictly as a toolchain, drive everything via command-line xcodebuild/swift build from the Makefile. No IDE. No hand-authored .xcodeproj. The app bundle must ship the mlx-swift_Cmlx.bundle resource bundle (contains default.metallib); existence proof of a signed/notarized xcodebuild-built mlx-swift app: sfomuseum/Docent (mlx-swift issue #345).

Execution model (the 0.A.6 decision):

1. Native Swift (in-process) — LLM/VLM/embeddings inference, Stable Diffusion image gen, standard HF→MLX conversion + affine/mxfp4 quantization of safetensors LLMs, adapter load/unload/fuse in memory.
2. Embedded Python venv (out-of-process via PythonRunner) — fine-tuning (LoRA/QLoRA/DoRA/full via mlx_lm.lora API with a custom TrainingCallback → JSON-lines protocol), adapter fuse-to-disk + GGUF export, advanced quantization (AWQ/DWQ/GPTQ/mixed), .bin conversions, Whisper STT, mlx-audio TTS, FLUX image gen, evaluation (--test ppl, mlx_lm.evaluate).
   • Venv: Python 3.12 pinned (mlx wheels are cp310–cp314, macOS ≥ 14 arm64-only — aligns with the Apple-Silicon gate), provisioned with uv into ~/Library/Application Support/ZyquoMLX/py/ on first run (never inside the signed bundle; never touching system Python). Pins: mlx-lm==0.31.3, mlx-vlm==0.6.8, mlx-whisper==0.4.3, mlx-audio==0.4.6 (+ lockfile via uv pip compile); uv supports --offline + local wheel cache for repair-offline behavior.
   • Progress protocol: our own JSON lines on stdout emitted by pinned helper scripts (PyBridge/scripts/) — never scrape mlx-lm's stdout (format changed between 0.31.3 and main; see TRAINING-RESEARCH.md §5).

# 7. Memory & Performance

• Inference RAM ≈ weights + KV cache + overhead. Weights: 4-bit affine gs64 ≈ 0.5625 B/param (verified against real repo sizes, see MODELS.md); 8-bit ≈ 1.07; bf16 ≈ 2.0. KV cache (fp16): 2 × layers × kvHeads × headDim × seqLen × 2 bytes — e.g. Qwen3-8B (36L, 8 KV heads, 128 dim) at 8k ctx ≈ 1.2 GB; quantizable via kvBits/kvScheme. (Formula standard practice; per-model numbers UNVERIFIED against one official doc — MemoryAdvisor computes from config.json fields and calibrates in Phase 7.)
• Memory.memoryLimit defaults to 1.5× maxRecommendedWorkingSetSize (verified doc comment); macOS caps GPU-visible memory ≈ 66–75% of unified RAM — query GPU.deviceInfo().maxRecommendedWorkingSetSize at runtime for exact gating rather than hardcoding.
• Training memory: LoRA ≈ 2–4× the inference footprint of the same base (optimizer state + activations); QLoRA on a 4-bit base is the 16–32 GB path; full FT ≈ 8 B/param + activations. Full tables in TRAINING-RESEARCH.md §6 and MODELS.md §3.
• Throughput (third-party 2026 benchmarks, indicative, UNVERIFIED first-party): M4 Max ≈ 62 tok/s on 8B-4bit, ≈ 38 tok/s on 14B-4bit; M4 Air 16 GB ≈ 25–35 tok/s on 9B-4bit; generation is memory-bandwidth-bound. First-party per-run stats come from GenerateCompletionInfo / GenerationResponse.
• This Mac (dev machine): M5 Max, 48 GB → comfortable up to ~35B 4-bit inference, QLoRA ≤ 32B, LoRA ≤ 14B, full FT ≤ ~3–4B.

# 8. Dependency Decision (feeds Phase 1 Package.swift)

.package(url: "https://github.com/ml-explore/mlx-swift", .upToNextMinor(from: "0.31.6")),
.package(url: "https://github.com/ml-explore/mlx-swift-lm", .upToNextMajor(from: "3.31.4")),
.package(url: "https://github.com/huggingface/swift-huggingface", from: "0.9.0"),
.package(url: "https://github.com/huggingface/swift-transformers", from: "1.3.0"),
// StableDiffusion (image gen) from mlx-swift-examples ("mlx-libraries") — add in Phase 6 if SD ships in v1

Platform floor: macOS 14 (mlx-swift Package.swift) — Zyquo MLX targets macOS 14+, arm64 only. Note: mlx-swift main uses swift-tools-version: 6.3 (experimentalCGen) — depend on released tags only, never branch main.

# Primary sources

github.com/ml-explore/{mlx, mlx-swift, mlx-swift-lm, mlx-swift-examples, mlx-lm, mlx-examples} (releases API + raw files at main) · ml-explore.github.io/mlx (lazy eval, unified memory, quantize docs) · pypi.org/pypi/{mlx, mlx-lm, mlx-vlm, mlx-whisper, mlx-audio}/json · github.com/huggingface/{swift-transformers, swift-huggingface} · github.com/Blaizzy/{mlx-vlm, mlx-audio} · mlx-swift PR #430, issues #345/#349 · local empirical build test (this Mac, 2026-07-30).