SPB Git forge
spboucher.ai ↗

spb/llm-api

Public
0commits 0branches 0releases
0 Bsize
maindefault branch
—last push
5.3 KB
View source Raw Blame History

# API

Base URL: https://www.llm-api.io/v1 (also reachable at http://127.0.0.1:8300/v1 on the node). Interactive OpenAPI schema: /openapi.

# Authentication

Authorization: Bearer llm_live_… (or X-API-Key). Keys are created in the console (Keys page) or llm-api create-key <name> [--admin]. Scopes: inference (OpenAI endpoints), admin (management API). The console session cookie also works for /v1 (playground).

# OpenAI-compatible endpoints

Method Path Notes
GET /v1/models installed models + aliases; extra fields: runtime, quantization, estimated_ram_gb, context, compatibility, status, capabilities
GET /v1/models/{id}
POST /v1/chat/completions stream, temperature, top_p, max_tokens/max_completion_tokens, stop (≤ 8), seed, presence_penalty, frequency_penalty, logit_bias, tools, stream_options.include_usage (always on)
POST /v1/completions prompt, echo, same sampling params
POST /v1/embeddings input (str or ≤ 256 strings), dimensions, encoding_format
POST /v1/rerank query, documents (≤ 200), top_n, return_documents, instruction

Extensions (ignored by SDKs):

  • timings: {ttft_ms, prompt_ms, generation_ms, total_ms, prompt_tps, generation_tps, peak_memory_gb} on responses and on the final stream chunk.
  • usage.prompt_tokens_details.cached_tokens when the prompt cache was reused.
  • reasoning_content on message / delta for thinking models. Control with chat_template_kwargs: {"enable_thinking": false}, reasoning: {"effort": "none"} or reasoning_effort.
  • top_k, min_p, repetition_penalty (MLX).
  • model: "auto" — routed by prompt (code → coder, long/analytical → reasoning, images → vision, embeddings → embedding), else current model / default alias / default model setting.

# Model resolution

model may be a registry id (qwen3.8-27b-4bit), an alias (coder), the directory name, or the Hugging Face repository (mlx-community/Qwen3.8-27B-4bit).

# Errors

json
{"error": {"message": "Model requires approximately 52.0 GB at a 32768 context but the safe limit is 45 GB.",
           "type": "model_memory_error", "code": "MODEL_TOO_LARGE", "param": null, "estimate": {...}}}
HTTP code when
400 INVALID_REQUEST, CONTEXT_TOO_LARGE, TEMPLATE_ERROR, VISION_UNSUPPORTED bad body, prompt longer than the loaded context
401 INVALID_API_KEY, UNAUTHENTICATED
403 FORBIDDEN, CSRF, DOWNLOADS_DISABLED, PATH_NOT_ALLOWED
404 MODEL_NOT_FOUND
409 CONFLICT, ALREADY_INSTALLED
413 BODY_TOO_LARGE > 20 MB
422 MODEL_INCOMPATIBLE, RUNTIME_UNSUPPORTED, WRONG_MODEL_TYPE
429 RATE_LIMITED 600 req/min per IP; 8 logins/min
502 WORKER_UNREACHABLE, DOWNLOAD_FAILED
503 MODEL_LOAD_FAILED, MODEL_LOAD_TIMEOUT, WORKER_CRASHED, WORKER_BUSY
504 GENERATION_TIMEOUT
507 MODEL_TOO_LARGE, INSUFFICIENT_DISK

# Management API (/api, admin)

Session mutations need header X-LLM-CSRF: 1.

  • Auth: GET /api/auth/status, POST /api/auth/setup|login|logout|password, GET /api/auth/me
  • Models: GET /api/models[?include_missing], GET /api/models/{id} (benchmarks, events, memory curve), GET /api/models/{id}/files, POST /api/models/{id}/load {context?, force?}, POST /api/models/{id}/unload, PATCH /api/models/{id} (favorite, pinned, enabled, notes, tags, name, overrides), POST /api/models/{id}/pin?pinned=, POST /api/models/{id}/favorite?favorite=, DELETE /api/models/{id} {confirm: id, keep_benchmarks}, POST /api/models/rescan, POST /api/models/{id}/benchmark {max_tokens, runs, long_prompt}, GET /api/models/{id}/benchmarks, POST /api/models/{id}/tokenize
  • Aliases: GET /api/aliases, PUT /api/aliases {alias, model_id}, DELETE /api/aliases/{alias}
  • Downloads: POST /api/models/inspect {repository, quant?}, POST /api/models/download {repository, quant?, force?}, GET /api/downloads, POST /api/downloads/{job}/retry, GET /api/jobs[?kind], POST /api/jobs/{id}/cancel
  • Harvester: POST /api/harvest/scan {runtimes, authors, limit_per_author, min_downloads, max_ram_gb, families, tasks, search}, GET /api/harvest/candidates[?task&runtime&family&size_class&q&include_duplicates] (+ starter slots), POST /api/harvest/select|dismiss {repo_id, selected}, POST /api/harvest/queue
  • System: GET /api/system, /api/system/memory, /api/system/gpu, /api/system/storage, /api/system/processes, /api/system/health, /api/system/metrics?minutes&hours, GET /api/runtime/status, GET /api/runtime/current-model
  • Keys: GET/POST /api/keys, PATCH /api/keys/{id} {name}, DELETE /api/keys/{id}
  • Settings: GET/PATCH /api/settings
  • Logs: GET /api/logs/audit|events|requests, GET /api/logs/worker/{model_id}
  • Live: GET /api/events — SSE stream: snapshot, manager, model (load progress), metrics (every 3 s), job, request, alert, settings.

# Status

GET /health (public):

json
{"status": "ok", "hardware": {"chip": "Apple M1 Max", "memory_gb": 64.0, "gpu_cores": 32},
 "model": {"loaded": true, "id": "qwen3-4b-instruct-2507-4bit", "loaded_models": ["…"]},
 "memory": {"used_gb": 23.1, "available_gb": 40.9, "pressure": "normal"}, "uptime_seconds": 1234, "version": "0.1.0"}