# Models

# Storage

text

<LLM_API_ROOT>/models/
├── mlx/<family>/<Repo-Name>/           safetensors + config.json + tokenizer (+ llm-api.json manifest)
├── gguf/<family>/<Repo-Name-GGUF>/     one chosen quantization (+ mmproj for vision)
├── embeddings/  rerankers/  vision/    same layout, by role
└── manifests/<org--repo>.json          copy of every download manifest

One recommended quantization per model; the downloader picks it (Q4_K_M → Q5_K_M → Q6_K → Q8_0 → MXFP4… for GGUF) unless a quant is given. Nothing is ever deleted by a scan.

# Discovery (`POST /api/models/rescan`, startup)

For each directory: config.json + *.safetensors → MLX model (architecture, layers, heads, quant bits from quantization, params from safetensors headers — packed uint32 ×32/bits); *.gguf → llama.cpp model (GGUF header: architecture, block_count, head_count_kv, key_length, context_length, file_type, tensor shapes). Type detection: embedding (name / pipeline / pooling), reranker, vision (vision_config, mmproj, model type), thinking / tools (chat template). Registry ids are stable slugs (qwen3-4b-instruct-2507-4bit); a directory keeps its id across rescans. Manual overrides (task, context, KV bits, pooling, extra llama args) live in overrides and survive rescans.

# Size classes (estimated RAM at the recommended context)

TINY < 5 GB · SMALL 5–10 · MEDIUM 10–20 · LARGE 20–35 · XL 35–45 · TOO_LARGE > budget.

# Compatibility

status	meaning
`compatible`	fits with ≥ 16K context in the budget
`compatible_with_restrictions`	fits, but context must stay below 16K (the recommended context is enforced at load)
`experimental`	runs but unverified: architecture not in the llama.cpp known list, VLM type falling back to text, very low quantization on a large model
`not_recommended`	fits only through heavy swap (≤ absolute limit but > budget) — not loadable without `force`
`incompatible`	runtime missing, architecture unsupported by the installed mlx-lm/mlx-vlm, or > absolute limit

# Quantization policy (Harvester scoring)

≤ 8B prefer 8/6-bit · 10–20B Q6/Q5 · 20–40B Q5/Q4 · 40–80B Q4 · larger Q3 only when genuinely useful. Lower than that is penalized (too_low).

# Model Harvester

POST /api/harvest/scan lists trusted authors (MLX: mlx-community; GGUF: unsloth, bartowski, ggml-org, lmstudio-community), excludes ASR/TTS/image/video/NSFW/base/draft repos, estimates RAM from the listing config + safetensors metadata (MLX) or from the chosen GGUF file (one repo-tree call), evaluates compatibility, scores (popularity, recency, size, quantization policy), and dedupes by base model + runtime (only the best quantization per base model is proposed). Candidates land in the Harvester page with a suggested starter library (small/medium/large general, coding, reasoning, vision, embedding, reranker). Selected candidates are queued through the normal inspected download path.

# Starter library on M1M64 (downloaded 2026-09-10, all mlx-community unless noted)

slot	model	RAM est.
tiny test	Qwen3-4B-Instruct-2507-4bit, Llama-3.2-3B-Instruct-4bit, gemma-3-1b-it Q4_K_M (GGUF)	4–9 GB
small general / vision	Qwen3.5-9B-MLX-4bit	~8 GB
medium general	gemma-4-12B-it-qat-4bit	~10 GB
reasoning	gpt-oss-20b-MXFP4-Q8	~14 GB
coding	Devstral-Small-2-24B-Instruct-2512-4bit, Qwen3-Coder-30B-A3B-Instruct-4bit	15–20 GB
large general	Qwen3.8-27B-4bit, Qwen3.6-35B-A3B-4bit (MoE), gemma-4-26b-a4b-it-4bit (MoE)	17–24 GB
XL	Llama-3.3-70B-Instruct-4bit	~42 GB (context ≤ 8K)
embedding	Qwen3-Embedding-0.6B-8bit (MLX), embeddinggemma-300M (GGUF)	< 1 GB, can stay resident
reranker	Qwen3-Reranker-0.6B-4bit	< 1 GB

Versions verified against the installed runtimes: mlx 0.32.2, mlx-lm 0.31.3, mlx-vlm 0.7.0, llama.cpp 0.4.0 (Homebrew), huggingface-hub 1.31.0.