1# Models23## Storage45```6<LLM_API_ROOT>/models/7├── mlx/<family>/<Repo-Name>/ safetensors + config.json + tokenizer (+ llm-api.json manifest)8├── gguf/<family>/<Repo-Name-GGUF>/ one chosen quantization (+ mmproj for vision)9├── embeddings/ rerankers/ vision/ same layout, by role10└── manifests/<org--repo>.json copy of every download manifest11```1213One recommended quantization per model; the downloader picks it (`Q4_K_M` → `Q5_K_M` → `Q6_K` → `Q8_0` → `MXFP4`… for GGUF) unless a `quant` is given. Nothing is ever deleted by a scan.1415## Discovery (`POST /api/models/rescan`, startup)1617For each directory: `config.json` + `*.safetensors` → MLX model (architecture, layers, heads, quant bits from `quantization`, params from safetensors headers — packed `uint32` ×32/bits); `*.gguf` → llama.cpp model (GGUF header: architecture, `block_count`, `head_count_kv`, `key_length`, `context_length`, `file_type`, tensor shapes). Type detection: embedding (name / pipeline / pooling), reranker, vision (`vision_config`, mmproj, model type), thinking / tools (chat template). Registry ids are stable slugs (`qwen3-4b-instruct-2507-4bit`); a directory keeps its id across rescans. Manual overrides (task, context, KV bits, pooling, extra llama args) live in `overrides` and survive rescans.1819## Size classes (estimated RAM at the recommended context)2021TINY < 5 GB · SMALL 5–10 · MEDIUM 10–20 · LARGE 20–35 · XL 35–45 · TOO_LARGE > budget.2223## Compatibility2425| status | meaning |26|---|---|27| `compatible` | fits with ≥ 16K context in the budget |28| `compatible_with_restrictions` | fits, but context must stay below 16K (the recommended context is enforced at load) |29| `experimental` | runs but unverified: architecture not in the llama.cpp known list, VLM type falling back to text, very low quantization on a large model |30| `not_recommended` | fits only through heavy swap (≤ absolute limit but > budget) — not loadable without `force` |31| `incompatible` | runtime missing, architecture unsupported by the installed mlx-lm/mlx-vlm, or > absolute limit |3233## Quantization policy (Harvester scoring)3435≤ 8B prefer 8/6-bit · 10–20B Q6/Q5 · 20–40B Q5/Q4 · 40–80B Q4 · larger Q3 only when genuinely useful. Lower than that is penalized (`too_low`).3637## Model Harvester3839`POST /api/harvest/scan` lists trusted authors (MLX: `mlx-community`; GGUF: `unsloth`, `bartowski`, `ggml-org`, `lmstudio-community`), excludes ASR/TTS/image/video/NSFW/base/draft repos, estimates RAM from the listing config + safetensors metadata (MLX) or from the chosen GGUF file (one repo-tree call), evaluates compatibility, scores (popularity, recency, size, quantization policy), and **dedupes by base model + runtime** (only the best quantization per base model is proposed). Candidates land in the Harvester page with a suggested starter library (small/medium/large general, coding, reasoning, vision, embedding, reranker). Selected candidates are queued through the normal inspected download path.4041## Starter library on M1M64 (downloaded 2026-09-10, all mlx-community unless noted)4243| slot | model | RAM est. |44|---|---|---|45| tiny test | Qwen3-4B-Instruct-2507-4bit, Llama-3.2-3B-Instruct-4bit, gemma-3-1b-it Q4_K_M (GGUF) | 4–9 GB |46| small general / vision | Qwen3.5-9B-MLX-4bit | ~8 GB |47| medium general | gemma-4-12B-it-qat-4bit | ~10 GB |48| reasoning | gpt-oss-20b-MXFP4-Q8 | ~14 GB |49| coding | Devstral-Small-2-24B-Instruct-2512-4bit, Qwen3-Coder-30B-A3B-Instruct-4bit | 15–20 GB |50| large general | Qwen3.8-27B-4bit, Qwen3.6-35B-A3B-4bit (MoE), gemma-4-26b-a4b-it-4bit (MoE) | 17–24 GB |51| XL | Llama-3.3-70B-Instruct-4bit | ~42 GB (context ≤ 8K) |52| embedding | Qwen3-Embedding-0.6B-8bit (MLX), embeddinggemma-300M (GGUF) | < 1 GB, can stay resident |53| reranker | Qwen3-Reranker-0.6B-4bit | < 1 GB |5455Versions verified against the installed runtimes: mlx 0.32.2, mlx-lm 0.31.3, mlx-vlm 0.7.0, llama.cpp 0.4.0 (Homebrew), huggingface-hub 1.31.0.56