spb/zyquo-mlx Public MIT

1<!--2  PLAN.md3  Zyquo MLX45  Author: Simon-Pierre Boucher6  Mail: contact@spboucher.ai7-->89# Zyquo MLX — Execution Plan1011Strict phase order 0 → 8. One phase at a time. Each phase ends with a checkpoint12(build, run, zero warnings, 3–5 line summary here) before the next begins.1314---1516## Phase 0 — Mandatory Intensive Web Research ✅ (completed 2026-07-30)1718- [x] 0.A `docs/MLX-RESEARCH.md` — MLX framework deep dive19  - [x] Core primitives: arrays, lazy eval / `eval()`, unified memory, streams/devices, autodiff20  - [x] Swift vs. Python honest capability split (current, verified against live repos)21  - [x] Model types & current MLX model zoo (LLM / VLM / embeddings / speech / image-gen)22  - [x] Inference API specifics (Swift-first: loading, tokenizers, streaming, params, KV cache, memory release)23  - [x] Quantization & conversion (tools, exact commands, on-disk format)24  - [x] Build & execution strategy resolved (Metal toolchain question TESTED locally)25  - [x] Memory & performance (RAM estimation for inference vs. training)26- [x] 0.B `docs/TRAINING-RESEARCH.md` — fine-tuning on MLX27  - [x] LoRA / QLoRA exact workflow, hyperparameters, adapter save/fuse28  - [x] Full fine-tuning feasibility & memory cost29  - [x] Dataset formats (chat / prompt-completion / text), split, templating, validation30  - [x] Training observability: metrics, checkpoint cadence, resume31  - [x] Evaluation: held-out loss/perplexity, base vs. tuned comparison32- [x] 0.C `docs/MODELS.md` — Hub integration + curated catalog33  - [x] Hugging Face Hub HTTP API (search, info, tree, resolve, LFS, token)34  - [x] Featured catalog across types/sizes, live-verified `mlx-community` repo IDs (43 repos, 0 failures)35  - [x] RAM table (8–128 GB) for inference AND LoRA fine-tuning36- [x] `docs/BUILD.md` — no-Xcode-IDE build recipe incl. Metal (tested on this Mac)37- [x] Phase checkpoint: docs complete, traceable, committed3839**Phase 0 summary:** Verified the mid-2026 MLX ecosystem live (mlx 0.32.0,40mlx-swift 0.31.6, mlx-swift-lm 3.31.4, mlx-lm 0.31.3): LM libraries moved to41`ml-explore/mlx-swift-lm` 3.x, which now natively covers LLM/VLM/embeddings42inference, LoRA/QLoRA training, and safetensors conversion/quantization; the43Python bridge is required only for rich-dataset training, full FT, speech,44FLUX, advanced quant, and evaluation. Locally proved CLT-only builds fail at45Metal kernel compilation → strategy: full Xcode as toolchain, CLI-only builds,46`mlx-swift_Cmlx.bundle` shipped in the app. Catalog: 43 models live-verified.4748### Local ground truth (recorded 2026-07-30)4950- Host: Apple M5 Max, 48 GB RAM, macOS 27.0 (25A5388g toolchain era), arm6451- Swift 6.4 (swiftlang-6.4.0.25.4), **Command Line Tools only** at52  `/Library/Developer/CommandLineTools` — `xcodebuild` unavailable, **no `metal`53  compiler in CLT**54- **TESTED 2026-07-30 (clean run, `/tmp/zyquo-mlx-buildtest`, mlx-swift 0.31.x):**55  `swift build` with CLT only compiles all of Cmlx C++/Swift, then fails at56  `CompileMetalFile steel_attention.metal` with57  `error: unable to spawn process 'metal' (No such file or directory)`.58  → CLT-only build is IMPOSSIBLE; a real Metal toolchain (full Xcode, CLI-only59  usage) is required. Per charter 0.A.6 the rule adapts to: command-line60  `swift build`/`xcodebuild` only, no Xcode IDE, no hand-authored `.xcodeproj`.61- No Xcode.app present anywhere on this machine yet → Phase 1 must provision the62  Xcode toolchain (download strategy documented in `docs/BUILD.md`)63- Python 3.14.4 (Homebrew) + `uv` available; `rsvg-convert` + `iconutil` present64  for the Phase 5 icon pipeline6566---6768## Phase 1 — Project Setup ✅ (completed 2026-07-30)6970- [x] Xcode 26.6 toolchain provisioned (CLI-only; Metal toolchain 17F109 installed)71- [x] `Package.swift` — target `ZyquoMLX`, deps per MLX-RESEARCH.md §8 (resolve exact: mlx-swift 0.31.6, mlx-swift-lm 3.31.4, swift-transformers 1.3.3, swift-huggingface 0.9.0)72- [x] `Makefile` — build / app / dev / run / icon / clean (+ release stub for Phase 8)73- [x] `Info.plist` — Zyquo MLX, com.zyquo.mlx, macOS 14+, arm64, developer-tools74- [x] `@main` SwiftUI entry point + Apple-Silicon gate + terminal activation75- [x] Python bootstrap plan wired (first-run venv per BUILD.md §3.4 — code lands Phase 3)76- [x] `README.md` (build instructions)77- [x] Phase checkpoint: build green (0 warnings), `Zyquo MLX.app` assembled, launched, window verified by screenshot7879**Phase 1 summary:** Xcode 26.6 installed as toolchain-only; resolved the build80definitively — classic SwiftPM silently skips Metal (broken GPU path), but81`swift build --build-system swiftbuild -c release` compiles all kernels and82emits `mlx-swift_Cmlx.bundle/default.metallib` (~90 s clean, 0 warnings), so83no xcodebuild is needed. `make app` assembles an ad-hoc-signed `Zyquo MLX.app`84with all resource bundles; app launches with the Apple-Silicon gate in place.85BUILD.md §3.1 updated with the resolved recipe.86## Phase 2 — Architecture + Inference POC ✅ (completed 2026-07-30)8788- [x] Folder architecture per charter (App/ Models/ Engine/ Training/ Hub/ Services/ established with real code; Convert/ Data/ PyBridge/ arrive with Phase 3, DesignSystem/ ViewModels/ Views/ with Phase 4 — no dead placeholder files)89- [x] `Models/` domain types: `LocalModel`, `ModelType`, `Dataset`, `TrainingRun`, `Checkpoint`, `Job` (+ `HyperParams` mirroring mlx-lm defaults)90- [x] `Engine/`: `InferenceEngine` actor + `ModelTypeAdapters` (LLM/VLM/Embeddings via LLMModelFactory/VLMModelFactory/EmbedderModelFactory) + `GenerationParams` + `MemoryAdvisor` (device-derived working-set gating)91- [x] `Hub/ModelStore`: local library scan/validate (config.json, shard-index consistency, quantization metadata, param-count derivation)92- [x] CLI POC mode (`ZyquoMLX --infer <dir>` / `--embed <dir>`) before UI93- [x] PHASE GATE: LLM (Qwen3-0.6B-4bit) streamed at **603.7 tok/s, TTFT 0.83 s**, unload freed 319.8 MB (verified); embeddings (all-MiniLM-L6-v2-4bit) returned 384-dim L2-normed vectors with correct similarity ordering (0.86 related vs 0.72 unrelated)94- [x] Phase checkpoint: build green, 0 warnings, arm64-only9596**Phase 2 summary:** Domain models, `InferenceEngine` actor with typed97adapters, `ModelStore` validation, and `MemoryAdvisor` are in place; every98dependency API was verified against the checked-out mlx-swift-lm 3.31.499sources (no guessed names — `#huggingFaceTokenizerLoader` macro requires100`import Tokenizers` at the expansion site; non-Sendable `UserInput` goes101through `perform(nonSendable:)`). CLI POC proves the two-model-type gate with102real downloads in the app's Models library. Build is arm64-only via103`--arch arm64` (swiftbuild otherwise builds universal and x86_64 fails).104## Phase 3 — Training, Quantization & Conversion ✅ (completed 2026-07-30)105106- [x] `PyBridge/PythonRunner` — Process wrapper over the venv, JSON-lines progress protocol, cancellation107- [x] `PyBridge/PythonEnvironment` — uv-provisioned pinned venv in App Support (idempotent adopt/verify/repair; mlx-lm 0.31.3 now, vlm/whisper/audio on demand)108- [x] `PyBridge/scripts/` — zyquo_train.py (TrainingCallback → JSON; bypasses `lora.run()` which drops the callback in 0.31.3), zyquo_fuse.py, zyquo_convert.py109- [x] `Data/DatasetService` + `DatasetFormats` — import/validate JSONL (chat/completions/text), deterministic split, malformed-row report with fixes, token stats, preview110- [x] `Training/TrainingService` + `RunStore` + `MetricsStream` — cancellable runs, persisted state + metrics.jsonl, numbered checkpoints, warm-start resume, MemoryAdvisor gating111- [x] `Convert/ConversionService` + `QuantConfig` — Swift-native affine/mxfp4 quant (size preview accurate to 0.1%); Python bridge for fuse/convert112- [x] CLI: `--train`, `--fuse`, `--quantize`, `--validate-dataset` (UI in Phase 6)113- [x] PHASE GATE: QLoRA on Qwen3-0.6B-4bit, 48-row chat dataset — live loss 4.62→0.10 (val 6.31→0.14) at ~2,300 tok/s, checkpoints at 40/80/120, adapter fused (auto-dequantize), fused model answers in the trained "⚒ From the forge:" style114- [x] Phase checkpoint: build green, 0 warnings115116**Phase 3 summary:** Full foundry core working end-to-end from the app. Two117upstream landmines found and handled: (1) mlx-lm 0.31.3's `lora.run()`118silently discards the caller's TrainingCallback — our driver replicates119`run()`'s flow and calls `train_model` directly; (2) fusing into a quantized120base re-quantizes and rounds away small LoRA deltas — Zyquo defaults to121dequantize-on-fuse, recommends adapter-attached inference, and warns on122re-quantization (documented in TRAINING-RESEARCH.md §2.2). Swift-native123quantization verified: 1.19 GB fp16 → 335.5 MB 4-bit, predicted 335.3 MB.124## Phase 4 — Design System & UI ✅ (completed 2026-07-30)125126- [x] `DesignSystem/ZyquoTheme` — copper-on-slate tokens (light flagship + derived dark), typography (SF Mono for metrics), 4pt spacing grid, radii, 0.5pt hairlines, motion; zero raw hex in views (hex lives only inside the token table)127- [x] Workbench shell: 240pt translucent navigator (sections + counts, footer: settings gear, active-job spinner, live RAM readout), 52pt contextual header, 1360×880 / min 1080×700128- [x] Screens: Models (Installed rows w/ type badge + quant + size + RAM verdict + reveal-in-Finder; Discover state designed), Datasets (import JSONL w/ validation flow), Train (run history w/ status pills), Convert / Playground / Evaluate intentional empty states129- [x] Settings window (native tabs: General, Compute, Python, Storage, Appearance — HF token vault + Shortcuts + Advanced land with their Phase 6 features)130- [x] Quality gate: light + dark reviewed via screenshots — copper identity reads clearly in both, cards/hairlines/baselines aligned131- [x] Phase checkpoint: build green, 0 warnings132133**Phase 4 summary:** The token system (`ZyquoTheme`) is the single source of134truth for every color/font/spacing value; shared components (`ZyquoCard`,135`StatusPill`, `TypeBadge`, `VerdictBadge`, `EmptyStateView`) established. The136workbench shell renders the real Phase 3 artifacts (4 models incl. the fused137forge model, 1 dataset, 3 runs) with live MemoryAdvisor verdicts. Run138configurator, live loss charts, playground interactions, and Hub Discover are139Phase 6 features and will be built on this spec.140## Phase 5 — App Icon ✅ (completed 2026-07-30)141142- [x] Two SVG directions rendered and compared: **Z-anvil/spark chosen** (stronger forge story via glow + anvil); Z-core-node kept in assets for reference143- [x] Refined: crisp geometric anvil (horn/face/waist/base + hot copper line), Z lifted for breathing room, restrained base glow; reviewed at 512 + zoom144- [x] Simplified small variant (Z only, thicker rim) for 16/32 px145- [x] `make icon` (scripts/make-icon.sh): rsvg-convert → full iconset (16→1024 incl @2x, small variant ≤32) → `iconutil` → AppIcon.icns; embedded via `make app`146- [x] Phase checkpoint: size-ladder review 16→256 sharp; verified in the real Dock — unmistakably the copper Zyquo sibling147## Phase 6 — Features ✅ (completed 2026-07-30)148149- [x] `Hub/HubService` (live mlx search + tree sizes + resolve, HF token via Keychain `HFTokenStore`), `Hub/DownloadManager` (Range-resumable, verified end-to-end: 351 MB repo downloaded with progress → auto-installed), `Services/Catalog` (16 Featured entries from MODELS.md w/ RAM verdicts)150- [x] Models › Discover: Featured catalog + debounced live search, download/pause with progress, installed detection151- [x] Playground: streaming chat verified in-UI (578 tok/s · TTFT 1.47 s shown live), VLM image attach, embeddings inspector w/ vectors + ranked cosine similarities, load/unload with freed-bytes readout152- [x] Train: configurator sheet (RAM gating + MemoryAdvisor suggestions inline, method/hyperparams with help text) + run detail verified in-UI (copper/slate loss chart from persisted metrics, stat strip, SF Mono console w/ autoscroll, checkpoints, cancel/warm-resume)153- [x] Convert: Quantize (Swift-native, live size preview) / Fuse (de-quantize smart default + explanation) / Convert-from-HF job cards with progress154- [x] Evaluate: base-vs-candidate side-by-side compare with per-side tok/s + TTFT (sequential load/unload)155- [x] Polish: HF token in Keychain (Settings › Hugging Face), Foundry menu with ⌘L/⌘D/⌘R/⌘U/⌘I/⌘E section shortcuts156- [x] Coherence sweep: headers ✓, naming ✓, no TODO/dead code ✓, zero raw hex in views ✓157158**Phase 6 summary:** All foundry features are wired into the workbench UI on159top of the Phase 3 services. Live verification on this Mac: run detail renders160the real QLoRA run's curves; Playground streamed a real reply with stats; the161Hub pipeline searched, downloaded, and installed a real repo with resumable162progress. One heuristic fix surfaced by verification: decoder-style embedding163repos (Qwen3-Embedding ships a `Qwen3ForCausalLM` config) are detected by164name — the downloaded embedder then produced correct 1024-dim vectors.165## Phase 7 — Verification ✅ (completed 2026-07-30)166167- [x] Speech pipeline added (mlx-whisper via PyBridge + Playground panel + CLI)168- [x] Inference matrix green: LLM ×3 (578–630 tok/s), embeddings ×2 (incl. decoder-style), VLM + image (correctly read the icon's Z; 97 tok/s), speech (perfect 1.0 s transcript); memory release verified per type (0.3–2.9 GB freed)169- [x] Training: QLoRA + LoRA to completion (loss 4.62→0.10 / val 6.31→0.14), live metrics, checkpoints, **cancel → cancelled state** (fix), **warm resume provably loads adapters** (resume VAL = 0.136), fuse w/ auto-dequantize, behavior change verified; OOM config blocked with suggestions170- [x] Convert/quantize: HF→MLX Python path (331 MB, runs at 630 tok/s; snapshot-quirk bypass documented) + Swift-native quantize (size prediction accurate to 0.1 %)171- [x] Catalog: 16/16 Featured repos live-green172- [x] Python env: bootstrap, healthy-venv adopt, on-demand extras, JSON protocol173- [x] `docs/VERIFICATION.md` green table (image-gen: documented deferral — upstream pipelines are script-only)174175**Phase 7 summary:** Full lifecycle proven end-to-end on this Mac across four176model types plus training/convert/hub/python-env. Three real-world defects177were found *by* verification and fixed: cancelled runs mis-recorded as178completed, decoder-style embedding repos mis-typed as LLM, and stale sharded179indexes in upstream repos rejected by validation. Results in VERIFICATION.md.180## Phase 8 — Signing & Notarization ✅ (completed 2026-07-30)181182- [x] zyquo-term setup inspected: identity `Developer ID Application: Simon-Pierre Boucher (3YM54G49SN)` + profile `MacLustr-Notarize` reused; identity verified present in keychain (no secrets printed/committed)183- [x] `Support/entitlements.plist` — Hardened Runtime, minimal posture (no sandbox, no allow-jit — MLX's Metal runtime compile is GPU shader compilation, not process JIT)184- [x] `make release`: nested-first signing → `ditto` zip → `notarytool submit --wait` → staple → verify185- [x] Signed + notarized + stapled DMG (`make dmg`)186- [x] **PHASE GATE GREEN on first submission:** `spctl -a -vv` → `accepted, source=Notarized Developer ID`; `stapler validate` OK; notarized app launches187- [x] Final coherence sweep: headers ✓, naming ✓, no TODO/dead code ✓, zero warnings ✓188189**Phase 8 summary:** `make release` produces a Developer ID-signed, notarized,190stapled `Zyquo MLX.app` (plus DMG) entirely from the command line — no Xcode191IDE, no `.xcodeproj`. Notarization passed on the first attempt with the192minimal entitlements posture, confirming BUILD.md §4's analysis. The193bundled-Python notarization minefield never arises because the venv lives in194Application Support, outside the signed bundle.195196---197198## Definition of Done — final review (2026-07-30)199200- ✅ `make release` → Developer ID-signed, notarized, stapled app, no Xcode IDE201- ✅ Inference across MLX-supported types: LLM/VLM/embeddings/speech verified202  with streaming + memory release (image-gen: documented deferral — upstream203  pipelines are script-only; Swift StableDiffusion lib is the follow-up path)204- ✅ Real LoRA + QLoRA end-to-end: live metrics, checkpoints, cancel, warm205  resume, fuse, export — with two upstream landmines found and handled206- ✅ Conversion + quantization produce valid, running MLX models (Swift-native207  + Python paths); export lands in the shared Models library208- ✅ Datasets import/validate/split/preview; Hub browse + resumable downloads +209  curated catalog live-verified (16/16 + 43-repo sweep)210- ✅ MemoryAdvisor: device-derived gating, verdicts on every surface, size211  prediction measured at 0.1 % accuracy; blocked-config UX verified212- ✅ Copper-on-slate icon (Z dominant, anvil/spark story), SVG source of truth,213  Dock-verified; light theme flagship + derived dark, token-only styling214- ✅ Naming coherent (`Zyquo MLX` / `com.zyquo.mlx` / `ZyquoMLX`); headers on215  every code file (swept); zero warnings; phase-prefixed commit history216- ✅ All Phase 0 research docs complete and traceable into the implementation217