SPB Git

spb/zyquo-local Public MIT

Native macOS AI chat that runs LLMs 100% locally on Apple Silicon with MLX — no cloud, no API keys.

macosswiftaimlxlocal-llm
Swift 97.2% Shell 1.8% Makefile 1%

phase0: MLX stack + Hub research docs, 30-model verified catalog

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
simon-pierre boucher committed 12 days ago (Jul 30, 2026) parent b516d57

Showing 3 changed files with +362 and −13

added docs/MLX-RESEARCH.md +192 −0
@@ -0,0 +1,192 @@
1 +<!--
2 + MLX-RESEARCH.md
3 + Zyquo Local
4 +
5 + Author: Simon-Pierre Boucher
6 + Mail: contact@spboucher.ai
7 +-->
8 +
9 +# MLX Swift Stack Research — Phase 0.A
10 +
11 +Researched 2026-07-30 against live official sources (raw file fetches at exact
12 +tags, GitHub releases API) plus empirical build tests on this machine
13 +(M5 Max, 48 GB, macOS 27.0 beta, Swift 6.4 CLT).
14 +
15 +## 1. Packages and SPM coordinates
16 +
17 +| Package | Repo | Version used | Products we link |
18 +|---|---|---|---|
19 +| mlx-swift | `https://github.com/ml-explore/mlx-swift` | **0.31.6** (2026-07-02) | (transitive) MLX, MLXNN, MLXFast… |
20 +| mlx-swift-lm | `https://github.com/ml-explore/mlx-swift-lm` | **3.31.4** (2026-06-30) | `MLXLLM`, `MLXLMCommon`, `MLXHuggingFace` |
21 +| swift-huggingface | `https://github.com/huggingface/swift-huggingface` | **0.9.0** | `HuggingFace` |
22 +| swift-transformers | `https://github.com/huggingface/swift-transformers` | **1.3.x** (1.3.3) | `Tokenizers` |
23 +
24 +Key facts:
25 +
26 +- The LLM layer (MLXLLM/MLXLMCommon/MLXVLM) **moved out of mlx-swift-examples**
27 + into `mlx-swift-lm`. mlx-swift-examples is frozen for libraries (last 2.29.1).
28 +- **mlx-swift-lm 3.x is a breaking major**: it no longer depends on
29 + swift-transformers/Hub. `Downloader`, `Tokenizer`, `TokenizerLoader` are now
30 + protocols in MLXLMCommon; concrete impls come from the `MLXHuggingFace`
31 + macro library (`#huggingFaceTokenizerLoader()`, `#hubDownloader()`) backed by
32 + swift-huggingface + swift-transformers, which the app must declare itself.
33 +- mlx-swift 0.31.5+ tags require a Swift ≥6.3 toolchain
34 + (`swift-tools-version: 6.3;(experimentalCGen)`). We have Swift 6.4 → fine.
35 +- Minimum platform: macOS 14. Our `LSMinimumSystemVersion` = 14.0, arm64 only.
36 +
37 +## 2. Loading, generating, streaming — verified API (tag 3.31.4)
38 +
39 +Load from a **local directory** (our only load path — downloads are ours):
40 +
41 +```swift
42 +import MLXLLM, MLXLMCommon, MLXHuggingFace, Tokenizers
43 +
44 +let container: ModelContainer = try await loadModelContainer(
45 + from: modelDirectoryURL,
46 + using: #huggingFaceTokenizerLoader()
47 +)
48 +```
49 +
50 +- `LLMModelFactory._load` reads `config.json`, dispatches on `model_type` via
51 + `LLMTypeRegistry`, honors `generation_config.json` EOS overrides, and
52 + `loadWeights` enumerates **every `*.safetensors` recursively** (the shard
53 + index json is not consulted).
54 +- There is **no progress callback for local-directory loads** — model load
55 + progress in UI is indeterminate (spinner + elapsed), not a percent bar.
56 +
57 +Multi-turn with KV-cache reuse — `ChatSession` (MLXLMCommon):
58 +
59 +```swift
60 +let session = ChatSession(container,
61 + instructions: systemPrompt,
62 + generateParameters: params,
63 + history: restoredMessages) // rehydration supported
64 +for try await g in session.streamDetails(to: prompt) {
65 + switch g {
66 + case .chunk(let text): ...
67 + case .info(let info): ... // stats
68 + case .toolCall: break
69 + }
70 +}
71 +session.clear() // resets history + KV cache, keeps instructions
72 +```
73 +
74 +KV cache is held inside the session across turns (no re-prefill). ChatSession
75 +is NOT thread-safe; `ModelContainer` is Sendable. One session per conversation,
76 +owned by the `InferenceEngine` actor. Changing system prompt/params → rebuild
77 +the session from persisted history (KV cache is lost, acceptable).
78 +
79 +Lower level (available if needed): `container.prepare(input:)` +
80 +`container.generate(input:parameters:) -> AsyncStream<Generation>`,
81 +`generateTask(...)` for deterministic early-stop cleanup, `TokenIterator`.
82 +
83 +`Generation` events: `.chunk(String)`, `.info(GenerateCompletionInfo)`,
84 +`.toolCall(ToolCall)`. Stop reasons: `.stop`, `.length`, `.cancelled`.
85 +
86 +### GenerateParameters (exact fields, verified)
87 +
88 +`maxTokens: Int?`, `maxKVSize: Int?`, `kvBits: Int?` (4/8), `kvGroupSize`,
89 +`quantizedKVStart`, `kvScheme: String?`, `temperature: Float` (default 0.6,
90 +0 = greedy), `topP: Float` (1.0), `topK: Int` (0), `minP: Float` (0),
91 +`seed: UInt64?`, `repetitionPenalty: Float?`, `repetitionContextSize` (20),
92 +presence/frequency penalties + context sizes, `prefillStepSize` (512).
93 +
94 +### Stats — GenerateCompletionInfo (final `.info` stream element)
95 +
96 +`promptTokenCount`, `generationTokenCount`, `promptTime`, `generateTime`,
97 +`stopReason`, computed `promptTokensPerSecond`, `tokensPerSecond`.
98 +TTFT = time from send to first `.chunk` (we measure it ourselves).
99 +
100 +### Unload
101 +
102 +No explicit API: release all `ChatSession`/`ModelContainer` references, then
103 +`MLX.GPU.clearCache()`. Verified guidance from the repo's own reference docs.
104 +
105 +## 3. Model directory format (required files)
106 +
107 +| File | Required | Notes |
108 +|---|---|---|
109 +| `config.json` | ✅ | `model_type`, quantization block, EOS ids |
110 +| `*.safetensors` | ✅ | all shards, MLX-convention (scales/biases for quant) |
111 +| `tokenizer.json` + `tokenizer_config.json` | ✅ | tokenizer + chat template |
112 +| `model.safetensors.index.json` | ⬜ not read by Swift stack; keep anyway |
113 +| `chat_template.jinja` / `*.jinja` | ⬜ newer HF layout |
114 +| `generation_config.json` | ⬜ EOS override |
115 +| `special_tokens_map.json` | ⬜ |
116 +
117 +Download filter (mirrors the package's own): `*.safetensors`, `*.json`,
118 +`*.jinja`. `DownloadManager` downloads exactly this set.
119 +
120 +## 4. Supported `model_type` architectures (LLMTypeRegistry @ 3.31.4)
121 +
122 +`mistral, mixtral, llama, phi, phi3, phimoe, gemma, gemma2, gemma3,
123 +gemma3_text, gemma3n, gemma4, gemma4_unified, gemma4_text, qwen2, qwen3,
124 +qwen3_moe, qwen3_next, qwen3_5, qwen3_5_moe, qwen3_5_text, minicpm,
125 +starcoder2, cohere, openelm, internlm2, deepseek_v3, granite,
126 +granitemoehybrid, mimo, mimo_v2_flash, minimax, glm4, glm4_moe,
127 +glm4_moe_lite, acereason, falcon_h1, bitnet, smollm3, ernie4_5, lfm2,
128 +baichuan_m1, exaone4, gpt_oss, lille-130m, olmoe, olmo2, olmo3, bailing_moe,
129 +lfm2_moe, nanochat, nemotron_h, afmoe, jamba, mamba2, mistral3, apertus,
130 +nemotron_labs_diffusion`
131 +
132 +This list ships in the app (`SupportedArchitectures`) to warn before
133 +downloading an unsupported model (checked against `config.model_type` from the
134 +Hub API with `config=true`).
135 +
136 +## 5. The no-Xcode build question — RESOLVED EMPIRICALLY
137 +
138 +Upstream docs claim "SwiftPM (command line) cannot build the Metal shaders…
139 +use xcodebuild". **This is outdated for Swift 6.4**: the new swift-build
140 +backend runs `CompileMetalFile` on mlx-swift's `.metal` resources. Verified on
141 +this machine:
142 +
143 +1. CLT-only `swift build` fails with `unable to spawn process 'metal'` —
144 + the compiler is looked up **via `PATH`** (proven with a logging stub).
145 +2. With `~/Developer/Metal.xctoolchain/usr/bin` (Metal Toolchain component
146 + v17.5.188, copied from cluster node M4M36) prepended to `PATH`,
147 + `swift build` succeeds and the binary executes on GPU (`Device(gpu, 0)`).
148 +3. Most kernels are JIT-compiled at runtime by the Metal framework (no
149 + toolchain needed at runtime); AOT kernels (gemv, SDPA, norms, rope…) are
150 + compiled at build time into the `mlx-swift_Cmlx` resource bundle. Phase 2's
151 + PoC (real attention workload) re-confirms the metallib is bundled/found.
152 +
153 +**Recipe** (full details + SwiftUI SDK pin in `docs/BUILD.md`): plain
154 +`swift build` + Metal toolchain on PATH + `SDKROOT=MacOSX26.5.sdk`.
155 +No Xcode IDE, no `.xcodeproj`, `xcodebuild` not needed.
156 +
157 +For the `.app` bundle, the `mlx-swift_Cmlx.bundle` resource bundle must be
158 +copied into `Contents/Resources/` (Makefile handles it); for bare CLI runs
159 +from `.build/`, the bundle sits next to the executable so it is found.
160 +
161 +## 6. Hub downloads: decision
162 +
163 +**Custom `URLSession`-based `DownloadManager`** (per-file pause/resume/cancel,
164 +HTTP Range resume across restarts, app-managed model folder), NOT
165 +swift-huggingface's snapshot API (no per-file pause handle, imposes HF cache
166 +layout). swift-transformers is still used for **tokenization** (non-negotiable
167 +— it is what the MLX stack expects, via `#huggingFaceTokenizerLoader()`).
168 +Full HF HTTP API contract documented in `docs/MODELS.md` §1.
169 +
170 +## 7. Memory model
171 +
172 +APIs (verified @ mlx-swift 0.31.6): `MLX.GPU.activeMemory / cacheMemory /
173 +peakMemory / snapshot() / resetPeakMemory() / set(cacheLimit:) /
174 +set(memoryLimit:relaxed:) / clearCache() / deviceInfo()` (architecture,
175 +memorySize, maxRecommendedWorkingSetSize).
176 +
177 +Sizing rules used by `MemoryAdvisor`:
178 +
179 +- Weights ≈ `params × bits/8 × 1.08` (scales/biases overhead).
180 +- KV cache ≈ `2 × layers × kvHeads × headDim × contextTokens × 2 bytes` (fp16);
181 + `kvBits` 8/4 halves/quarters it; `maxKVSize` caps it (rotating cache).
182 +- +10–20 % working set for activations; macOS GPU-wired ceiling ≈ 70–75 % of
183 + unified RAM → verdicts: **Fits** if weights+KV ≤ 60 % of physical RAM,
184 + **Tight** ≤ 75 %, **Too large** above.
185 +- Physical RAM via `sysctl hw.memsize`; live footprint via `GPU.snapshot()`.
186 +
187 +## 8. Open flags
188 +
189 +- mlx-swift #430 (metallib via build plugin) and #416 (`setMetallibPath`) were
190 + open as of 2026-07-30 — not relied upon.
191 +- API surface verified at tag 3.31.4; `main` has drifted (MLXFoundationModels,
192 + MLXGuidedGeneration) — re-verify before any version bump.
added docs/MODELS.md +148 −0
@@ -0,0 +1,148 @@
1 +<!--
2 + MODELS.md
3 + Zyquo Local
4 +
5 + Author: Simon-Pierre Boucher
6 + Mail: contact@spboucher.ai
7 +-->
8 +
9 +# Hugging Face Hub Integration + Curated Catalog — Phase 0.B
10 +
11 +Researched live against huggingface.co on 2026-07-30. Every catalog size was
12 +verified by summing `/tree/main?recursive=true` (decimal GB, matches Finder).
13 +This file is the single source of truth for `ModelCatalog.swift` — the two
14 +must never drift apart (updated together per methodology rule 7).
15 +
16 +## 1. HF Hub HTTP API contract (implemented by `HubService`)
17 +
18 +Base `https://huggingface.co`, plain `URLSession`, distinctive `User-Agent`
19 +(`ZyquoLocal/<version>`), optional `Authorization: Bearer hf_…`.
20 +
21 +### Search — `GET /api/models`
22 +Params: `author=mlx-community`, `search=<text>`, `pipeline_tag=text-generation`
23 +(filters out whisper/embeddings noise), `filter=mlx`, `sort=downloads|likes|
24 +createdAt|lastModified|trendingScore`, `direction=-1`, `limit≤100`,
25 +`full=true` (adds filename-only siblings), `config=true` (adds
26 +`config.model_type` → architecture-compat check).
27 +Response items: `id`, `likes`, `downloads`, `private`, `gated`
28 +(`false|"auto"|"manual"`), `tags[]`, `pipeline_tag`, `library_name`,
29 +`createdAt` (ISO8601 fractional). **Pagination via `Link` response header
30 +cursor** (`rel="next"`), no page numbers.
31 +
32 +### Model info + file sizes — `GET /api/models/{owner}/{repo}?blobs=true`
33 +Cheapest single call for sizes: each sibling becomes `{rfilename, size,
34 +lfs{sha256,size}?}`. Canonical alternative: `GET
35 +/api/models/{id}/tree/main?recursive=true` (`{type:"file", size, path,
36 +lfs?}`). **Never use `usedStorage`** (counts all revisions — verified 682 MB
37 +reported vs 351 MB actual). `safetensors.total` param count is unreliable for
38 +quantized repos (packed uint32).
39 +
40 +### Download — `GET /{repo}/resolve/main/{path}`
41 +- LFS/large → **302** to signed, time-limited CDN URL; small files → **307**.
42 + Always follow redirects; re-resolve if a paused download's URL expired.
43 +- Initial response headers (grab via HEAD): `x-linked-size` (real size),
44 + `x-linked-etag` (sha256 for LFS — used for integrity), `x-repo-commit`,
45 + `accept-ranges: bytes`.
46 +- **Resume: `Range: bytes=N-` → 206 verified.**
47 +- ⚠️ Swift gotcha: `URLSession` forwards `Authorization` across the cross-host
48 + redirect — strip it in `willPerformHTTPRedirection` when host changes.
49 +
50 +### Errors / limits
51 +- Anonymous + missing repo → **401** (not 404!). Gated repo: API info 200 with
52 + `gated` set; `resolve` 401 anonymous / 403 token-without-grant → in-app hint
53 + to add an HF token in Settings.
54 +- Rate limits (5-min windows): anonymous 500 API / 3 000 resolve per IP; free
55 + token 1 000 / 5 000. On **429** parse `RateLimit` header, back off exactly.
56 +
57 +## 2. mlx-community naming scheme
58 +
59 +`mlx-community/{BaseModel}-{quant}` — suffix semantics (≈ GB per B params):
60 +
61 +| Suffix | Meaning | ≈GB/B | Notes |
62 +|---|---|---|---|
63 +| `-4bit` | uniform 4-bit (group 64) | 0.57 | standard choice |
64 +| `-5bit`/`-6bit` | uniform 5/6-bit | 0.7/0.8 | near-lossless at 6 |
65 +| `-8bit` | uniform 8-bit | 1.1 | virtually lossless, 2× RAM |
66 +| `-bf16` | unquantized | 2.1 | reference quality |
67 +| `-4bit-DWQ` | distilled weight quant | 0.6 | 4-bit size ≈ 6-bit quality; prefer |
68 +| `-qat-4bit` | vendor QAT (Gemma) | 0.6+ | best 4-bit Gemma quality |
69 +| `OptiQ-4bit` | 2026 mixed 4/8-bit (KL-sensitivity) + often MTP head | 0.65 | ~1.4× decode speedup; prefer on Qwen3.5/3.6, gemma-4 |
70 +| `MXFP4-Q8` | native MXFP4 MoE + 8-bit rest | — | intended format for gpt-oss |
71 +
72 +Weights rule of thumb: **GB ≈ params(B) × bits ÷ 8 × 1.1**.
73 +
74 +## 3. Featured catalog — 30 models, live-verified 2026-07-30
75 +
76 +All `gated: false`, all architectures present in `LLMTypeRegistry`
77 +(see MLX-RESEARCH.md §4). Min RAM = smallest tier that runs it comfortably
78 +with useful context.
79 +
80 +### Tiny (≤4B)
81 +| Repo ID | Params | Quant | GB | Min RAM | Tags | One-liner |
82 +|---|---|---|---|---|---|---|
83 +| `mlx-community/Qwen3-0.6B-4bit` | 0.6B | 4bit | 0.35 | 8 | tiny·general | Smallest useful chat model; instant loads. |
84 +| `mlx-community/LFM2.5-1.2B-Instruct-4bit` | 1.2B | 4bit | 0.66 | 8 | tiny·general | Liquid AI's 2026 edge model; punchy and very fast. |
85 +| `mlx-community/Llama-3.2-1B-Instruct-4bit` | 1B | 4bit | 0.71 | 8 | tiny·general | The classic 1B; most-downloaded tiny LLM. |
86 +| `mlx-community/gemma-3-1b-it-qat-4bit` | 1B | QAT-4bit | 0.77 | 8 | tiny·general | Google QAT checkpoint — best quality-per-byte at 1B. |
87 +| `mlx-community/Qwen3-1.7B-4bit` | 1.7B | 4bit | 0.98 | 8 | tiny·general·reasoning | Hybrid thinking modes in under 1 GB. |
88 +| `mlx-community/SmolLM3-3B-4bit` | 3B | 4bit | 1.75 | 8 | tiny·general | HF's fully-open 3B; long context, optional reasoning. |
89 +| `mlx-community/Llama-3.2-3B-Instruct-4bit` | 3B | 4bit | 1.82 | 8 | tiny·general | The default "runs anywhere" pick. |
90 +| `mlx-community/Qwen3-4B-Instruct-2507-4bit` | 4B | 4bit | 2.28 | 8 | tiny·general | 2507 refresh — best ≤4B all-rounder. |
91 +| `mlx-community/gemma-3-4b-it-qat-4bit` | 4B | QAT-4bit | 3.03 | 8 | tiny·general | Vision-capable 4B with QAT quality. |
92 +
93 +### Mid (7–20B)
94 +| Repo ID | Params | Quant | GB | Min RAM | Tags | One-liner |
95 +|---|---|---|---|---|---|---|
96 +| `mlx-community/Llama-3.1-8B-Instruct-4bit` | 8B | 4bit | 4.53 | 16 | mid·general | The reference 8B; huge prompt/finetune ecosystem. |
97 +| `mlx-community/Qwen3-8B-4bit` | 8B | 4bit | 4.62 | 16 | mid·general·reasoning | Best-selling 8B; thinking mode on demand. |
98 +| `mlx-community/gemma-3-12b-it-qat-4bit` | 12B | QAT-4bit | 8.07 | 16 | mid·general | Sweet spot for 16 GB Macs; strong writing. |
99 +| `mlx-community/Qwen3.5-9B-OptiQ-4bit` | 9B | OptiQ-4bit | 8.22 | 16 | mid·general | 2026 Qwen3.5 gen; top mid-size quality. |
100 +| `mlx-community/phi-4-4bit` | 14.7B | 4bit | 8.26 | 16 | mid·general·reasoning | Microsoft dense 14B; excels at math/STEM. |
101 +| `mlx-community/Qwen3-14B-4bit` | 14B | 4bit | 8.32 | 16 | mid·general·reasoning | Stronger sibling of Qwen3-8B; 16 GB flagship. |
102 +| `mlx-community/gpt-oss-20b-MXFP4-Q8` | 20.9B MoE | MXFP4+Q8 | 12.10 | 16 (24 comfy) | mid·general·reasoning | OpenAI open-weights MoE; #1 download in the org. |
103 +
104 +### Large (24B+)
105 +| Repo ID | Params | Quant | GB | Min RAM | Tags | One-liner |
106 +|---|---|---|---|---|---|---|
107 +| `mlx-community/Mistral-Small-3.2-24B-Instruct-2506-4bit` | 24B | 4bit | 13.28 | 24 | large·general | Fast dense 24B, low hallucination, good tool use. |
108 +| `mlx-community/gemma-3-27b-it-qat-4bit` | 27B | QAT-4bit | 16.87 | 32 | large·general | Gemma 3 flagship with QAT; superb chat quality. |
109 +| `mlx-community/GLM-4.7-Flash-4bit` | 30B MoE | 4bit | 16.87 | 32 | large·general·coding | Zhipu's 2026 fast MoE; strong agentic/coding. |
110 +| `mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit` | 30B-A3B MoE | 4bit | 17.20 | 32 | large·general | 3B active params → big-model quality at small-model speed. |
111 +| `mlx-community/Qwen3-32B-4bit` | 32B | 4bit | 18.45 | 32 | large·general·reasoning | Dense 32B with thinking; slower but deeper than the MoE. |
112 +| `mlx-community/Qwen3.6-27B-OptiQ-4bit` | 27B | OptiQ-4bit | 20.00 | 32 | large·general | 2026 Qwen3.6 dense; MTP head ≈1.4× faster decode. |
113 +| `mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit` | 35B-A3B MoE | OptiQ-4bit | 24.69 | 48 (32 tight) | large·general | 2026 successor to Qwen3-30B-A3B. |
114 +| `mlx-community/Llama-3.3-70B-Instruct-4bit` | 70B | 4bit | 39.71 | 64+ | large·general | The 70B reference; 64 GB+ Macs only. |
115 +
116 +### Coding
117 +| Repo ID | Params | Quant | GB | Min RAM | Tags | One-liner |
118 +|---|---|---|---|---|---|---|
119 +| `mlx-community/Qwen2.5-Coder-7B-Instruct-4bit` | 7B | 4bit | 4.30 | 16 (8 tight) | coding | The default small local code model. |
120 +| `mlx-community/Qwen2.5-Coder-14B-Instruct-4bit` | 14B | 4bit | 8.32 | 16 | coding | Noticeably better completions on 16 GB Macs. |
121 +| `mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit` | 30B-A3B MoE | 4bit | 17.20 | 32 | coding·large | Best local coding model for 32 GB; fast agentic loops. |
122 +
123 +### Reasoning
124 +| Repo ID | Params | Quant | GB | Min RAM | Tags | One-liner |
125 +|---|---|---|---|---|---|---|
126 +| `mlx-community/DeepSeek-R1-0528-Qwen3-8B-4bit` | 8B | 4bit | 4.62 | 16 (8 tight) | reasoning | R1-0528 distill onto Qwen3-8B; `<think>` chain-of-thought. |
127 +| `mlx-community/DeepSeek-R1-Distill-Qwen-14B-4bit` | 14B | 4bit | 8.32 | 16 | reasoning | Most-downloaded R1 distill; great math on 16 GB. |
128 +| `mlx-community/Qwen3-30B-A3B-Thinking-2507-4bit` | 30B-A3B MoE | 4bit | 17.20 | 32 | reasoning·large | Current best local reasoner ≤32 GB. |
129 +
130 +Curation notes: `QwQ-32B-4bit` verified but superseded (near-zero recent
131 +downloads); `Ministral-8B` and `R1-Distill-Qwen-7B` dropped as superseded by
132 +newer entries; 100 GB+ 2026 giants (Kimi-K2.5 658 GB, GLM-5.2 418 GB,
133 +DeepSeek-V4-Flash 152 GB) excluded — no consumer Mac fits them.
134 +
135 +## 4. RAM recommendation table (feeds MemoryAdvisor + badges)
136 +
137 +Needed RAM ≈ weights + KV cache (0.5–4 GB by context) + 4–6 GB macOS headroom.
138 +GPU-wired ceiling ≈ 70–75 % of unified RAM → comfortable = weights ≤ ~60 %.
139 +
140 +| Mac RAM | Comfortable @4bit | Tight | Catalog examples |
141 +|---|---|---|---|
142 +| 8 GB | ≤4B (≤2.5 GB) | 7–8B (4.5 GB, short ctx) | Qwen3-4B-2507, Llama-3.2-3B |
143 +| 16 GB | 7–14B (4.3–9 GB) | gpt-oss-20b (12.1 GB) | Qwen3-14B, gemma-3-12b-qat |
144 +| 24 GB | 14B + long ctx; 24B (13.3 GB) | 27–30B (≈17 GB) | Mistral-Small-3.2 |
145 +| 32 GB | 24–32B (13–18.5 GB) | 35B-A3B (24.7 GB) | Qwen3-30B-A3B, gemma-3-27b |
146 +| 48 GB | 32–35B + long ctx | 70B (39.7 GB) | Qwen3.6-35B-A3B |
147 +| 64 GB | 70B (39.7 GB) | — | Llama-3.3-70B |
148 +| 128 GB | 70B @8bit / 100B MoE | ~150 GB repos: no | gpt-oss-120b |
modified docs/PLAN.md +22 −13
@@ -10,20 +10,29 @@
10 10
11 11 Machine: Apple M5 Max, 18 cores, 48 GB RAM, macOS 27.0 (beta), Swift 6.4 (Command Line Tools only, no Xcode IDE installed).
12 12
13 −## Phase 0 — Intensive research (in progress)
13 +## Phase 0 — Intensive research — ✅ DONE (2026-07-30)
14 14
15 −- [ ] Research current mlx-swift packages, versions, SPM coordinates, min macOS
16 −- [ ] Research current MLXLLM/MLXLMCommon API surface (load, tokenize, chat template, streaming, params, KV cache, unload)
17 −- [ ] Document model directory format (required files)
18 −- [ ] Document supported architectures list
19 −- [ ] Resolve the no-Xcode build question EMPIRICALLY (scratch SPM package + `swift build` with CLT only)
20 −- [ ] Decide swift-transformers vs direct HTTP HubService
21 −- [ ] Document memory model + GPU cache limits + RAM estimation
22 −- [ ] Write `docs/MLX-RESEARCH.md`
23 −- [ ] Research HF Hub HTTP API (search, model info, resolve URLs, LFS, tokens)
24 −- [ ] Curate 20–30 Featured models, live-verified on the Hub with sizes
25 −- [ ] RAM recommendation table
26 −- [ ] Write `docs/MODELS.md`
15 +- [x] Research current mlx-swift packages, versions, SPM coordinates, min macOS
16 +- [x] Research current MLXLLM/MLXLMCommon API surface (load, tokenize, chat template, streaming, params, KV cache, unload)
17 +- [x] Document model directory format (required files)
18 +- [x] Document supported architectures list
19 +- [x] Resolve the no-Xcode build question EMPIRICALLY (scratch SPM package + `swift build` with CLT only)
20 +- [x] Decide swift-transformers vs direct HTTP HubService
21 +- [x] Document memory model + GPU cache limits + RAM estimation
22 +- [x] Write `docs/MLX-RESEARCH.md`
23 +- [x] Research HF Hub HTTP API (search, model info, resolve URLs, LFS, tokens)
24 +- [x] Curate 20–30 Featured models, live-verified on the Hub with sizes
25 +- [x] RAM recommendation table
26 +- [x] Write `docs/MODELS.md`
27 +
28 +**Checkpoint:** Stack pinned: mlx-swift-lm 3.31.4 (MLXLLM/MLXLMCommon/MLXHuggingFace)
29 ++ swift-huggingface 0.9.0 + swift-transformers 1.3.x on mlx-swift 0.31.6.
30 +No-Xcode question resolved empirically: Swift 6.4 `swift build` compiles Metal
31 +kernels when a standalone `Metal.xctoolchain` is on PATH (copied from cluster
32 +node M4M36 → `~/Developer/Metal.xctoolchain`); GPU execution verified. SwiftUI
33 +needs `SDKROOT` pinned to MacOSX26.5.sdk (macro-plugin issue on 27.x SDKs).
34 +Downloads: custom URLSession manager (Range-resume verified 206); tokenization
35 +via swift-transformers. 30-model catalog live-verified with real sizes.
27 36
28 37 ## Phase 1 — Project setup — pending
29 38 ## Phase 2 — Architecture + inference PoC — pending
30 39