phase7: verification complete — full matrix green (LLM/VLM/embeddings/speech, training incl cancel+resume, convert/quantize, hub, py-env); stale-index + VLM fixes; docs/VERIFICATION.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
4 changed files +101 −10
modified
Sources/ZyquoMLX/Hub/ModelStore.swift
+4 −1
@@ -96,7 +96,10 @@ actor ModelStore { | |||||
| 96 | let referenced = Set(weightMap.values) | 96 | let referenced = Set(weightMap.values) | ||
| 97 | let present = Set(weightFiles.map { ($0 as NSString).lastPathComponent }) | 97 | let present = Set(weightFiles.map { ($0 as NSString).lastPathComponent }) | ||
| 98 | let missing = referenced.subtracting(present).sorted() | 98 | let missing = referenced.subtracting(present).sorted() | ||
| 99 | − if !missing.isEmpty { | 99 | + // Some mlx-community repos ship a consolidated | ||
| 100 | + // model.safetensors next to a stale sharded index (seen live: | ||||
| 101 | + // Qwen3-VL-4B-Instruct-4bit) — treat that as valid. | ||||
| 102 | + if !missing.isEmpty && !present.contains("model.safetensors") { | ||||
| 100 | throw ModelValidationError.inconsistentShards(missing: missing) | 103 | throw ModelValidationError.inconsistentShards(missing: missing) | ||
| 101 | } | 104 | } | ||
| 102 | } | 105 | } | ||
modified
Sources/ZyquoMLX/PyBridge/scripts/zyquo_convert.py
+10 −0
@@ -43,6 +43,16 @@ def main(): | |||||
| 43 | 43 | ||||
| 44 | from mlx_lm import convert | 44 | from mlx_lm import convert | ||
| 45 | 45 | ||||
| 46 | + # mlx-lm 0.31.3 first downloads weights with allow_patterns, then its | ||||
| 47 | + # copy step demands the FULL snapshot with local_files_only=True and | ||||
| 48 | + # trips IncompleteSnapshotError on aux files (.gitattributes, LICENSE…). | ||||
| 49 | + # Pre-fetch the complete snapshot for remote repos to sidestep it. | ||||
| 50 | + import os | ||||
| 51 | + if not os.path.isdir(args.hf_path): | ||||
| 52 | + from huggingface_hub import snapshot_download | ||||
| 53 | + emit({"event": "stage", "stage": "downloading"}) | ||||
| 54 | + snapshot_download(repo_id=args.hf_path) | ||||
| 55 | + | ||||
| 46 | kwargs = { | 56 | kwargs = { | ||
| 47 | "hf_path": args.hf_path, | 57 | "hf_path": args.hf_path, | ||
| 48 | "mlx_path": args.mlx_path, | 58 | "mlx_path": args.mlx_path, | ||
modified
docs/PLAN.md
+15 −9
@@ -162,13 +162,19 @@ Hub pipeline searched, downloaded, and installed a real repo with resumable | |||||
| 162 | progress. One heuristic fix surfaced by verification: decoder-style embedding | 162 | progress. One heuristic fix surfaced by verification: decoder-style embedding | ||
| 163 | repos (Qwen3-Embedding ships a `Qwen3ForCausalLM` config) are detected by | 163 | repos (Qwen3-Embedding ships a `Qwen3ForCausalLM` config) are detected by | ||
| 164 | name — the downloaded embedder then produced correct 1024-dim vectors. | 164 | name — the downloaded embedder then produced correct 1024-dim vectors. | ||
| 165 | −## Phase 7 — Verification (in progress) | 165 | +## Phase 7 — Verification ✅ (completed 2026-07-30) | ||
| 166 | − | 166 | + | ||
| 167 | −- [ ] Speech pipeline added (mlx-whisper via PyBridge) — closes the model-type matrix gap | 167 | +- [x] Speech pipeline added (mlx-whisper via PyBridge + Playground panel + CLI) | ||
| 168 | −- [ ] Inference matrix: LLM (streaming/cancel/memory), embeddings ×2, VLM + image, speech transcription; tok/s + TTFT recorded | 168 | +- [x] Inference matrix green: LLM ×3 (578–630 tok/s), embeddings ×2 (incl. decoder-style), VLM + image (correctly read the icon's Z; 97 tok/s), speech (perfect 1.0 s transcript); memory release verified per type (0.3–2.9 GB freed) | ||
| 169 | −- [ ] Training: LoRA + QLoRA to completion, live metrics, checkpoints, warm resume, fuse, behavior change; OOM config blocked by MemoryAdvisor | 169 | +- [x] Training: QLoRA + LoRA to completion (loss 4.62→0.10 / val 6.31→0.14), live metrics, checkpoints, **cancel → cancelled state** (fix), **warm resume provably loads adapters** (resume VAL = 0.136), fuse w/ auto-dequantize, behavior change verified; OOM config blocked with suggestions | ||
| 170 | −- [ ] Convert/quantize: HF→MLX (Python path) + Swift-native quantize; outputs load and run; sizes correct | 170 | +- [x] Convert/quantize: HF→MLX Python path (331 MB, runs at 630 tok/s; snapshot-quirk bypass documented) + Swift-native quantize (size prediction accurate to 0.1 %) | ||
| 171 | −- [ ] Catalog dry-verify: all Featured repo IDs live-checked against the Hub | 171 | +- [x] Catalog: 16/16 Featured repos live-green | ||
| 172 | −- [ ] Python env: bootstrap/adopt/repair verified | 172 | +- [x] Python env: bootstrap, healthy-venv adopt, on-demand extras, JSON protocol | ||
| 173 | −- [ ] Green results table in docs/VERIFICATION.md (image-gen: documented deferral — upstream FLUX is script-only, no packaged pipeline) | 173 | +- [x] `docs/VERIFICATION.md` green table (image-gen: documented deferral — upstream pipelines are script-only) | ||
| 174 | + | ||||
| 175 | +**Phase 7 summary:** Full lifecycle proven end-to-end on this Mac across four | ||||
| 176 | +model types plus training/convert/hub/python-env. Three real-world defects | ||||
| 177 | +were found *by* verification and fixed: cancelled runs mis-recorded as | ||||
| 178 | +completed, decoder-style embedding repos mis-typed as LLM, and stale sharded | ||||
| 179 | +indexes in upstream repos rejected by validation. Results in VERIFICATION.md. | ||||
| 174 | ## Phase 8 — Signing & Notarization (not started) | 180 | ## Phase 8 — Signing & Notarization (not started) | ||
added
docs/VERIFICATION.md
+72 −0
@@ -0,0 +1,72 @@ | |||||
| 1 | +<!-- | ||||
| 2 | + VERIFICATION.md | ||||
| 3 | + Zyquo MLX | ||||
| 4 | + | ||||
| 5 | + Author: Simon-Pierre Boucher | ||||
| 6 | + Mail: contact@spboucher.ai | ||||
| 7 | +--> | ||||
| 8 | + | ||||
| 9 | +# Zyquo MLX — Phase 7 Verification Results | ||||
| 10 | + | ||||
| 11 | +All tests executed on the dev machine (Apple M5 Max, 48 GB, macOS 27.0) | ||||
| 12 | +on 2026-07-30, driven through the app (CLI mode and/or UI — both use the same | ||||
| 13 | +services). ✅ = green. | ||||
| 14 | + | ||||
| 15 | +## 1. Inference matrix | ||||
| 16 | + | ||||
| 17 | +| Capability | Model | Result | Metrics / notes | | ||||
| 18 | +|---|---|---|---| | ||||
| 19 | +| Text LLM, streaming | mlx-community/Qwen3-0.6B-4bit | ✅ | 603.7 tok/s, TTFT 0.83 s; stop-reason + unload freed 319.8 MB (verified via `Memory.snapshot` delta) | | ||||
| 20 | +| Text LLM (converted in-app) | qwen3-0.6b-converted-4bit | ✅ | 630.2 tok/s, TTFT 0.03 s, coherent output | | ||||
| 21 | +| Text LLM in UI (streaming chat) | zyquo-forge-qwen3-0.6b-4bit | ✅ | 578.2 tok/s, TTFT 1.47 s shown live in Playground header | | ||||
| 22 | +| Embeddings | mlx-community/all-MiniLM-L6-v2-4bit | ✅ | 384-dim L2-normed; similarity ranks correctly (0.86 related > 0.72 unrelated); 3 texts in 1.70 s | | ||||
| 23 | +| Embeddings (decoder-style) | mlx-community/Qwen3-Embedding-0.6B-4bit-DWQ | ✅ | 1024-dim; 2 texts in 0.36 s; required name-based type detection (config says `Qwen3ForCausalLM`) | | ||||
| 24 | +| VLM + image | mlx-community/Qwen3-VL-4B-Instruct-4bit | ✅ | Correctly described the app icon ("the letter Z"); 97.2 tok/s, TTFT 1.95 s, unload freed 2.89 GB | | ||||
| 25 | +| Speech (STT) | mlx-community/whisper-large-v3-turbo | ✅ | Perfect transcript of a 7 s English clip in 1.0 s (incl. "Zyquo"); language auto-detect verified (fr voice → fr) | | ||||
| 26 | +| Image generation | — | ⚠️ deferred | Upstream FLUX/SD Python pipelines are script-only (mlx-examples, no PyPI package); Swift `StableDiffusion` lib available as a follow-up. Documented deferral, not a regression. | | ||||
| 27 | + | ||||
| 28 | +## 2. Training | ||||
| 29 | + | ||||
| 30 | +| Test | Result | Notes | | ||||
| 31 | +|---|---|---| | ||||
| 32 | +| QLoRA to completion (4-bit base) | ✅ | Qwen3-0.6B-4bit, 48-row chat dataset, 120 iters: train loss **4.62 → 0.10**, val **6.31 → 0.14**, ~2,300 tok/s, peak 0.8 GB | | ||||
| 33 | +| LoRA (fp16 base) | ✅ | zyquo-forge fp16 base, 400 iters at ~3,300 tok/s, peak 1.6 GB | | ||||
| 34 | +| Live metrics | ✅ | JSON-lines callback protocol → charted in UI (verified screenshot); stdout never parsed | | ||||
| 35 | +| Checkpoints | ✅ | `0000040/0000080/0000120_adapters.safetensors` on cadence | | ||||
| 36 | +| Cancel | ✅ | Mid-run cancel at iter ~435/5000 → state `cancelled`, checkpoints intact (fix landed: cancelled stream no longer reports `completed`) | | ||||
| 37 | +| Warm resume | ✅ | Resume of the cancelled run starts at VAL 0.136 (== trained level, not the fresh 6.31) — adapter weights provably loaded | | ||||
| 38 | +| Adapter fuse | ✅ | Fused model answers in the trained "⚒ From the forge:" style; **auto-dequantize on quantized bases** (without it, re-quantization rounds small adapters away — verified both ways) | | ||||
| 39 | +| OOM gating | ✅ | full FT @ batch 256 blocked pre-run by MemoryAdvisor with actionable suggestions; batch-64 borderline config surfaced a runtime failure with persisted `failed` state | | ||||
| 40 | + | ||||
| 41 | +## 3. Convert / quantize | ||||
| 42 | + | ||||
| 43 | +| Test | Result | Notes | | ||||
| 44 | +|---|---|---| | ||||
| 45 | +| HF → MLX + 4-bit quant (Python path) | ✅ | `Qwen/Qwen3-0.6B` → 331 MB, "4.501 bits per weight", loads + generates at 630 tok/s. Driver pre-fetches the full snapshot (mlx-lm 0.31.3 `IncompleteSnapshotError` quirk documented in the script) | | ||||
| 46 | +| Swift-native quantization | ✅ | fp16 fused model 1.19 GB → 335.5 MB actual vs 335.3 MB predicted (0.1 % accuracy) | | ||||
| 47 | +| Validation of outputs | ✅ | Both outputs pass `ModelStore` validation and run | | ||||
| 48 | + | ||||
| 49 | +## 4. Hub & catalog | ||||
| 50 | + | ||||
| 51 | +| Test | Result | Notes | | ||||
| 52 | +|---|---|---| | ||||
| 53 | +| Featured catalog live-verified | ✅ | 16/16 repos HTTP 200 on 2026-07-30 (plus the 43-repo sweep in MODELS.md) | | ||||
| 54 | +| Live search | ✅ | `HubService.search` (mlx filter, downloads sort) feeding Discover | | ||||
| 55 | +| Resumable downloads | ✅ | 3 repos downloaded via `DownloadManager` (0.35–3.1 GB) with byte-level progress; per-file Range resume logic; pause keeps partial bytes | | ||||
| 56 | +| Stale-index repos | ✅ | Qwen3-VL ships a consolidated `model.safetensors` + stale sharded index — validator accepts consolidated layout (fix landed) | | ||||
| 57 | + | ||||
| 58 | +## 5. Python environment | ||||
| 59 | + | ||||
| 60 | +| Test | Result | Notes | | ||||
| 61 | +|---|---|---| | ||||
| 62 | +| Bootstrap (uv, Python 3.12, pinned mlx-lm 0.31.3) | ✅ | Provisioned into `~/Library/Application Support/ZyquoMLX/py/venv` | | ||||
| 63 | +| Adopt existing healthy venv | ✅ | Marker deleted → next Python use verified imports and re-adopted without reinstall | | ||||
| 64 | +| On-demand extras | ✅ | `mlx-whisper==0.4.3` installed automatically at first transcription | | ||||
| 65 | +| JSON progress protocol | ✅ | All pipelines (train/fuse/convert/transcribe) speak JSON lines; two upstream landmines bypassed and documented (callback drop in `lora.run()`, incomplete-snapshot error) | | ||||
| 66 | + | ||||
| 67 | +## Artifacts kept for development | ||||
| 68 | + | ||||
| 69 | +`Models/`: Qwen3-0.6B-4bit (346 MB), all-MiniLM (14 MB), Qwen3-Embedding-DWQ | ||||
| 70 | +(351 MB), whisper-large-v3-turbo (1.6 GB), Qwen3-VL-4B (3.1 GB), forge fp16 + | ||||
| 71 | +4-bit variants, converted Qwen3. `Datasets/zyquo-forge-style`, `Runs/` with | ||||
| 72 | +metrics history. Clean up via Settings › Storage when desired. | ||||
| 73 | |||||