spb/llm-api-toolkits

1# Caching, context and cost23## Prompt caching45The cache matches a **prefix** in the order `tools` → `system` → `messages`. Put stable content6first and mark the end of it:78```json9"system": [{"type": "text", "text": "<long stable instructions>", "cache_control": {"type": "ephemeral"}}]10```1112or set `"cache_control": {"type": "ephemeral"}` at the top level for automatic placement. At most134 breakpoints. Anything that changes before a breakpoint invalidates what follows it: a timestamp14in the system prompt, JSON serialised in a different key order, a tool list that varies.1516Verify, do not assume: `usage.cache_creation_input_tokens` on the first call,17`usage.cache_read_input_tokens` on the next. Zero reads on repeated calls means the prefix is18changing or is below the model's minimum cacheable size (a ~9K-token prefix cached on19`claude-haiku-5-5` in this skill's tests). Writes cost 1.25× input (5 min) or 2× (1 h); reads cost20a small fraction of input — see `references/models.md`.2122## Long conversations2324| Tool | Beta flag | What it does | Not on |25|---|---|---|---|26| Compaction | `compact-2026-01-12` | Summarises older turns (`context_management.edits: [{"type": "compact_20260112"}]`) | 4.5 models |27| Context editing | `context-management-2025-06-27` | Drops old tool results (`clear_tool_uses_20250919`) or thinking (`clear_thinking_20251015`) | — |28| Task budget | `task-budgets-2026-03-13` | `output_config.task_budget: {"type": "tokens", "total": ≥20000}` lets the model pace a run | 4.5 and 4.6 models |2930With compaction, keep appending the full `response.content`: compaction blocks must be preserved.3132Mid-conversation operator instructions: append `{"role": "system", "content": "…"}` to `messages`33instead of editing the top-level `system`, which would reset the cache. Placement is strict: the34entry goes **directly after the user message** it applies to — `[…, user, system]` — never right35after an assistant turn (`[…, assistant, system, user]` is a 400). Rejected entirely by the 4.5,364.6 and Opus 4.7 models.3738## Cost levers, cheapest first39401. Prompt caching. 2. Shorter prompts and tool results. 3. Batches (50%) for non-urgent work.414. Lower `effort`. 5. A cheaper model. Measure cost per completed task, not per request: a cheaper42request that needs more turns is not cheaper. Thinking tokens are billed as output tokens.4344```bash45node scripts/claude-api.mjs cost claude-sonnet-5-5 10000 2000 --requests 500 --batch46node scripts/claude-api.mjs compare 10000 2000 --requests 50047```48