spb/wp30_uqo

1# Literature audit — Group 3 (WP30 "Cache Compression and Knowledge Transfer in Multi-Agent Systems")23Access date for every URL below: **2026-10-29**. Nothing in this file comes from memory; every fact is tied to a page fetched that day (WebFetch / curl of the GitHub, Hugging Face, PyPI and arXiv APIs). Where a page did not state something, it is marked **unverified**.45Raw files downloaded for inspection (kept in `/tmp/audit_dl/`, not part of the repo): mlx-lm `README.md`, `mlx_lm/cache_prompt.py`, `mlx_lm/generate.py`, `mlx_lm/models/cache.py`; LLMLingua `llmlingua/prompt_compressor.py`, `setup.py`, `llmlingua/version.py`; LongBench `LongBench/README.md`, `LongBench/task.md`; RULER `README.md`.67---89## 1. LLMLingua-2 (`llmlingua2_2024`)1011**URLs fetched**12- https://arxiv.org/abs/2403.1296813- https://github.com/microsoft/LLMLingua (+ GitHub API `repos/microsoft/LLMLingua`, `commits/main`)14- https://raw.githubusercontent.com/microsoft/LLMLingua/main/llmlingua/prompt_compressor.py, `.../setup.py`, `.../llmlingua/version.py`15- https://huggingface.co/microsoft/llmlingua-2-xlm-roberta-large-meetingbank (+ `api/models/...?blobs=true`)16- https://huggingface.co/microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank (+ `api/models/...?blobs=true`)17- https://pypi.org/project/llmlingua/ (+ `pypi.org/pypi/llmlingua/json`)1819**Paper**20- Title: "LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression"21- Authors (13): Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, Dongmei Zhang22- arXiv v1: 19 Mar 2024; v2 (latest): 12 Aug 202423- arXiv Comments field: "Accepted at Findings of ACL 2024" → venue = **Findings of ACL 2024** (confirmed; GitHub README also lists "ACL 2024 Findings").24- Algorithm (from abstract, verbatim): "We formulate prompt compression as a token classification problem." A bidirectional Transformer encoder predicts keep/drop per token; training data is an extractive compression dataset distilled from GPT-4 on MeetingBank. Compression ratios 2x–5x reported. The HF card states each token receives a probability `p_preserve` used as the compression metric. The phrase "extractive token dropping with a target rate" is our paraphrase; the API exposes `rate` (fraction of tokens kept, default 0.5) and `target_token` (absolute budget, overrides `rate`, applies at context level) — both confirmed in `prompt_compressor.py` lines 221–222, 279–280, 325–328, 375–387.2526**Repository**27- https://github.com/microsoft/LLMLingua — license **MIT** (GitHub API `license.spdx_id = MIT`; `setup.py` line 52 `license="MIT License"`).28- Default branch `main`, HEAD on 2026-10-29: `5a4c78ae18ab17a98cf997e8259354e546081d64` (committed 2026-09-10). Stars 6 783. GitHub "Releases" section empty.29- README usage for LLMLingua-2 (verbatim):30  ```python31  llm_lingua = PromptCompressor(32      model_name="microsoft/llmlingua-2-xlm-roberta-large-meetingbank",33      use_llmlingua2=True, # Whether to use llmlingua-234  )35  compressed_prompt = llm_lingua.compress_prompt(prompt, rate=0.33, force_tokens = ['\n', '?'])36  ```37- Related papers listed in README: LLMLingua (EMNLP 2023), LongLLMLingua (ACL 2024), LLMLingua-2 (ACL 2024 Findings), SecurityLingua (CoLM 2025).3839**pip package**40- Name: `llmlingua`. Latest PyPI release **0.2.2, 2024-04-09** (`version.py` on `main` still reads 0.2.2 → no newer release since). Requires Python >= 3.8.0. License "MIT License".41- `install_requires`: `transformers>=4.26.0`, `accelerate`, `torch`, `tiktoken`, `nltk`, `numpy` (setup.py lines 25–31 and PyPI JSON). No CUDA-specific dependency.42- CPU support: `PromptCompressor.__init__(…, device_map: str = "cuda", …)` (line 74); `load_model` accepts any `device_map` containing `"cuda"`, `"cpu"` or `"mps"` (line 138) and forces `torch_dtype=torch.float32` when `device_map != "cuda"` (line 145). → **Runs on CPU / MPS with plain torch by passing `device_map="cpu"` (or `"mps"`)**; default is `"cuda"`. README/PyPI do not state this in prose (only "<8GB GPU memory" for the quantised Llama-2 variant of LLMLingua-1).43- Model class used for LLMLingua-2: `AutoModelForTokenClassification` (line 21, 132).4445**Checkpoints on Hugging Face (exact IDs, verified via API on 2026-10-29)**46| Model ID | License tag | Params (safetensors) | model.safetensors | Base | Downloads/last month | Revision |47|---|---|---|---|---|---|---|48| `microsoft/llmlingua-2-xlm-roberta-large-meetingbank` | mit | 558 945 282 (F32) ≈ 0.56 B | 2 235.8 MB | XLM-RoBERTa large | 90 749 | `ebaba9b0e874dadd3003ffcff828e4397e568089` (lastModified 2025-01-08) |49| `microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank` | apache-2.0 | 177 341 186 (F32) ≈ 0.18 B | 709.4 MB | google-bert/bert-base-multilingual-cased | 723 568 | `5f0c82792b7ea14c6484e015b6a072009496b7f2` (lastModified 2025-01-08) |50Both are `pipeline_tag: token-classification`. Note the licence mismatch between the two cards (MIT vs Apache-2.0); the repo is MIT.5152**Verification level:** page fetched (arXiv, GitHub, HF, PyPI) + source code read.5354---5556## 2. LongBench (`longbench2024`)5758**URLs fetched**59- https://arxiv.org/abs/2308.1450860- https://aclanthology.org/2024.acl-long.172/61- https://github.com/THUDM/LongBench (+ API), https://raw.githubusercontent.com/THUDM/LongBench/main/LongBench/README.md, `.../LongBench/task.md`62- https://huggingface.co/datasets/THUDM/LongBench (page + `/blob/main/README.md` + `api/datasets/THUDM/LongBench`)63- https://huggingface.co/api/datasets/THUDM/LongBench-v2 ; https://arxiv.org/abs/2412.1520464- Original-license checks: https://huggingface.co/datasets/allenai/qasper ; https://hotpotqa.github.io/ ; https://github.com/StonyBrookNLP/musique ; https://github.com/Alab-NII/2wikimultihop6566**Paper**67- Title: "LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding"68- Authors (13): Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li69- arXiv v1 28 Aug 2023; v2 19 Jun 2024; Comments: "ACL 2024".70- ACL Anthology: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)", 2024, pp. 3119–3137, DOI 10.18653/v1/2024.acl-long.172. Anthology states materials from 2016 on are CC BY 4.0 (paper PDF licence).71- Abstract: 21 datasets, 6 task categories, English + Chinese; avg length ≈ 6 711 words (EN) / 13 386 characters (ZH); 4 750 test samples in total.7273**Repository / dataset**74- GitHub https://github.com/THUDM/LongBench — license **MIT** (API). HEAD `2e00731f8d0bff23dc4325161044d0ed8af94c1e` (2025-01-15). Top-level README is now LongBench **v2**; v1 files were moved under `LongBench/`.75- HF dataset: `THUDM/LongBench` now **redirects to `zai-org/LongBench`** (API `id: zai-org/LongBench`, sha `5e628be450b7e67fb7ae6e201bd6d8f7056f7672`, lastModified 2024-12-18). Dataset card YAML has **no `license` field** (tags: task_categories question-answering/text-generation/summarization/text-classification; languages en, zh; size 1K<n<10K). → dataset-level licence **unverified / not stated**; code is MIT.76- Data fields (card + `LongBench/README.md` lines 117–119, verbatim): `input` "The input/command for the task, usually short, such as questions in QA, queries in Few-shot tasks, etc"; `context` "The long context required for the task, such as documents, cross-file code, few-shot examples in Few-shot tasks"; `answers` "A List of all true answers"; plus `length`, `dataset`, `language`, `all_classes`, `_id`. → **Yes, source documents are included in the `context` field.**77- Official metrics and counts (from `LongBench/task.md` and HF card, identical):7879| Subset (HF config) | Task type | Metric | Avg len (words) | Language | #Samples |80|---|---|---|---|---|---|81| `qasper` (Qasper) | Single-doc QA | **F1** | 3 619 | EN | **200** |82| `multifieldqa_en` (MultiFieldQA-en) | Single-doc QA | **F1** | 4 559 | EN | **150** |83| `hotpotqa` (HotpotQA) | Multi-doc QA | **F1** | 9 151 | EN | **200** |84| `2wikimqa` (2WikiMultihopQA) | Multi-doc QA | **F1** | 4 887 | EN | **200** |85| `musique` (MuSiQue) | Multi-doc QA | **F1** | 11 214 | EN | **200** |86(LongBench-E length-balanced variants: hotpotqa 100/100/100, 2wikimqa 100/100/100, multifieldqa_en 67/70/13, qasper 100/100/24 for 0–4k/4–8k/8k+; no musique in LongBench-E.) Total LongBench: 4 750 samples; most subsets 200, code subsets 500.8788- Construction (task.md lines 61–63, paraphrased): HotpotQA / 2WikiMultihopQA / MuSiQue / DuReader are "built based on the original datasets and processed to be suitable for long context evaluation" — for validation-set questions, the evidence passage containing the answer plus several distracting articles form the context. Qasper (and NarrativeQA, GovReport, QMSum, MultiNews) "directly use the data provided by the original papers" via ZeroSCROLLS templates. MultiFieldQA-en/zh are **newly annotated by the LongBench authors** (long articles from ~10 sources — LaTeX papers, judicial documents, government reports, Google-indexed PDFs — annotated by PhD/master students).89- Original licences of the five subsets (verified on the upstream pages):90  - Qasper: **CC BY 4.0** (HF `allenai/qasper` card, "Licensing Information: CC BY 4.0").91  - HotpotQA: **CC BY-SA 4.0** ("HotpotQA is distributed under a CC BY-SA 4.0 License", hotpotqa.github.io).92  - MuSiQue: **CC BY 4.0** ("MuSiQue is distributed under a CC BY 4.0 License", GitHub StonyBrookNLP/musique).93  - 2WikiMultihopQA: repo **Apache-2.0** (GitHub Alab-NII/2wikimultihop); no separate data licence stated → data licence **unverified**.94  - MultiFieldQA-en: created by LongBench authors; no licence stated anywhere → **unverified** (defaults to whatever the dataset card implies, which is nothing).95  - LongBench itself makes **no statement** about the licences of the underlying datasets (checked README, task.md, HF card).9697**LongBench v2**98- Exists: "LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks", arXiv 2412.15204 (v1 19 Dec 2024, v2 3 Jan 2025), 12 authors (Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li). arXiv Comments: "26 pages, 13 figures" — **venue not stated on arXiv (unverified)**.99- Differences from v1 (README verbatim): "Compared to the first generation of LongBench, LongBench v2 is much longer and much more challenging." 503 multiple-choice questions, contexts 8k–2M words, six task categories (single-doc QA, multi-doc QA, long in-context learning, long-dialogue history, code repository, long structured data); human experts 53.7 % accuracy (15-min limit), best model 50.1 % direct. Format: `_id, domain, sub_domain, difficulty, length, question, choice_A..D, answer, context`. Metric = accuracy (MCQ), not F1. HF `zai-org/LongBench-v2` card licence tag **apache-2.0** (sha `2b48e494…`).100- Implication for WP30: v1 subsets provide free-text F1 QA with explicit `context`; v2 is MCQ with much longer contexts — different metric, not a drop-in replacement.101102**Verification level:** page fetched (arXiv, ACL Anthology, GitHub raw, HF API, four upstream licence pages).103104---105106## 3. RULER (`ruler2024`)107108**URLs fetched**109- https://github.com/NVIDIA/RULER (+ API, raw README.md)110- https://arxiv.org/abs/2404.06654111112**Paper**113- arXiv ID confirmed: **2404.06654**. Title: "RULER: What's the Real Context Size of Your Long-Context Language Models?"114- Authors (8): Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg115- v1 9 Apr 2024; v2 11 Apr 2024; v3 (latest) 6 Aug 2024. arXiv Comments: "COLM 2024; Code is available at …" → venue **COLM 2024 confirmed** (the GitHub BibTeX still says `journal={arXiv preprint arXiv:2404.06654}`).116- Abstract: "We evaluate 17 long-context LMs with 13 representative tasks in RULER"; adds multi-hop tracing and aggregation categories beyond vanilla NIAH.117118**Repository**119- License **Apache-2.0** (API `license.spdx_id`). HEAD `c3f5e3b4f87f97e048793bb510a3a6b19a46bf3a` (2026-07-22). Stars 1 629.120- README line 3: "RULER generates synthetic examples to evaluate long-context language models with configurable sequence length and task complexity. We benchmark 17 open-source models across 4 task categories (in total 13 tasks)".121- Task table (README lines 140–146, verbatim configuration names):122  - **Retrieval** — `niah`: `type_haystack`: `repeat/essay/needle` ("repeat: repeated noise sentences", "essay: Paul Graham Essays", "needle: distracted needles"); `type_needle_k` / `type_needle_v`: `words/numbers/uuids` ("words: adjective-noun", "numbers: 7 digits", "uuids: 32 digits"); `num_needle_k` ("add multiple needles in haystack" = multi-key), `num_needle_v` ("retrieve multiple values from a single key" = multi-value), `num_needle_q` ("retrieve multiple values from multiple keys" = multi-query).123  - **Multi-hop Tracing** — `variable_tracking`: `num_chains`, `num_hops`.124  - **Aggregation** — `common_words_extraction` (`freq_cw`, `freq_ucw`, `num_cw`) and `freq_words_extraction` (`alpha` of the word-frequency distribution; 3 words returned in their evaluations).125  - **Question Answering** — `qa`: `dataset`: `squad` or `hotpotqa` ("the short-context qa dataset we use").126- Haystack generation: synthetic — either repeated noise sentences, concatenated Paul Graham essays (downloaded from the gkamradt NIAH repo and paulgraham.com via `download_paulgraham_essay.py`), or distracting needles; QA data downloaded from SQuAD and HotpotQA (`download_qa_dataset.sh`). Reported lengths: 4K, 8K, 16K, 32K, 64K, 128K. README caveat (line 168): only 13 task configurations are used in the paper; "it is by no means comprehensive enough and it cannot replace the more preferred realistic tasks."127- Licence of the generated data depends on upstream sources (Paul Graham essays, SQuAD CC BY-SA 4.0, HotpotQA CC BY-SA 4.0) — not stated in the repo → **unverified**.128129**Verification level:** page fetched (arXiv, GitHub API, raw README).130131---132133## 4. mlx-lm (`mlxlm2026`)134135**URLs fetched**136- https://github.com/ml-explore/mlx-lm (README), https://github.com/ml-explore/mlx-lm/releases, GitHub API `releases/latest`, `license`, `commits/main`137- https://pypi.org/pypi/mlx-lm/json138- raw `mlx_lm/cache_prompt.py`, `mlx_lm/generate.py`, `mlx_lm/models/cache.py`139140**Release / licence**141- Latest release **v0.32.2**, published **2026-10-16T02:49:02Z** (GitHub) / PyPI upload 2026-10-16T02:49:33. Previous: v0.32.1 (2026-10-15), v0.32.0 (2026-10-15), v0.31.3 (2026-04-22).142- `main` HEAD on 2026-10-29: `6b572ab1b531dd8fd2d13f6f36a89c2eccda18ea` (pushed 2026-10-29T18:35Z). Stars 7 507.143- License **MIT** (GitHub API license endpoint). PyPI `license` field is empty.144- PyPI requires Python >= 3.11; deps: `mlx>=0.32.2; platform_system == "Darwin"`, `numpy`, `transformers>=5.7.0`, `sentencepiece`, `protobuf`, `pyyaml`, `jinja2`; extras `cuda12`, `cuda13`, `cpu` (→ `mlx[cuda12|cuda13|cpu]>=0.32.2`), `vision`, `train`, `evaluate`, `test`.145146**Prompt-cache features — exact names found**147- README "Long Prompts and Generations" section (lines 197–240) lists "A rotating fixed-size key-value cache", "Prompt caching", "A configurable prefill step size". Verbatim:148  - "To use the rotating key-value cache pass the argument `--max-kv-size n` where `n` can be any integer."149  - "Caching prompts can substantially speedup reusing the same long context with different queries. To cache a prompt use `mlx_lm.cache_prompt`."150  - Example: `cat prompt.txt | mlx_lm.cache_prompt --model mistralai/Mistral-7B-Instruct-v0.3 --prompt - --prompt-cache-file mistral_prompt.safetensors` then `mlx_lm.generate --prompt-cache-file mistral_prompt.safetensors --prompt "\nSummarize the above text."`151  - "The cached prompt is treated as a prefix to the supplied prompt. Also notice when using a cached prompt, the model to use is read from the cache and need not be supplied explicitly."152  - "Prompt caching can also be used in the Python API in order to avoid recomputing the prompt … See the [example](examples/chat.py)."153  - `--prefill-step-size n` (default 2048).154- CLI flags in `mlx_lm/cache_prompt.py` (lines 44–79) and `mlx_lm/generate.py` (lines 186–224), verbatim help strings:155  - `--max-kv-size` — "Set the maximum key-value cache size"156  - `--prompt-cache-file` — "The file to save the prompt cache in" (required in cache_prompt)157  - `--kv-bits` — "Number of bits for KV cache quantization. Defaults to no quantization."158  - `--kv-group-size` — "Group size for KV cache quantization." (default 64)159  - `--quantized-kv-start` — "When --kv-bits is set, start quantizing the KV cache from this step onwards." (default `DEFAULT_QUANTIZED_KV_START`)160  - `generate.py` lines 2102/2106 raise if `--kv-bits` / `--kv-group-size` "does not match the kv cache loaded from --prompt-cache-file".161- Python API in `mlx_lm/models/cache.py`: `make_prompt_cache(model, max_kv_size=None)` (line 15; uses `model.make_cache()` if present, else `RotatingKVCache(max_size=max_kv_size, keep=4)` per layer when `max_kv_size` is set, else `KVCache()`), `save_prompt_cache(file_name, cache, metadata=None)` (line 43, `.safetensors`), `load_prompt_cache(file_name, return_metadata=False)` (line 82), `can_trim_prompt_cache` (115), `trim_prompt_cache(cache, num_tokens)` (122). Classes: `_BaseCache`, `ConcatenateKVCache`, `QuantizedKVCache` (line 251), `KVCache` (339), `RotatingKVCache` (423), `ArraysCache`, `ChunkedKVCache`, `CacheList`, `BatchKVCache`, `BatchRotatingKVCache`, plus `TokenBuffer`, `PromptTrieResult`, `PromptTrie`, `LRUPromptCache`. `KVCache.to_quantized(group_size=64, bits=4) -> QuantizedKVCache` (line 396); `RotatingKVCache.to_quantized` raises "RotatingKVCache Quantization NYI" (line 554).162- **Not in README** (only in source): `--kv-bits`, `--kv-group-size`, `--quantized-kv-start`, `make_prompt_cache`, `save_prompt_cache`, `load_prompt_cache`, `QuantizedKVCache`, `RotatingKVCache` (README says "rotating fixed-size key-value cache" without the class name).163164**Verification level:** page fetched + source read at HEAD `6b572ab`.165166---167168## 5. Hugging Face Transformers KV-cache docs (`hf_kvcache_docs`)169170**URLs fetched**171- https://huggingface.co/docs/transformers/en/kv_cache (page title "Cache strategies"; internal links point to **v5.21.1**)172- https://huggingface.co/docs/transformers/en/internal/generation_utils (section "Caches")173174**Cache classes documented on the `kv_cache` page (v5.21.1)**175- Comparison table: "Dynamic Cache", "Static Cache", "Quantized Cache" (columns: supports sliding layers / offloading / torch.compile / expected memory).176- `DynamicCache` — "the default cache class for all models"; `past_key_values = DynamicCache(config=model.config)`.177- `StaticCache` — `cache_implementation="static"`, enables compilation of the decode step.178- Cache offloading — `cache_implementation="offloaded"` (dynamic) / `"offloaded_static"` (static), or `DynamicCache(config=…, offloading=True)`; option `offload_only_non_sliding`. (No separate `OffloadedCache` class is named on this page.)179- `QuantizedCache` — `cache_implementation="quantized"` with `cache_config={"backend": "hqq"|"quanto"|"fp8", "nbits", "axis_key", "axis_value", "q_group_size", "residual_length"}`; "hqq supports int1, int2, int3, int4, and int8", "quanto supports int2 and int4 … default quantization backend", "fp8 stores the KV values in 8-bit floating point (float8_e4m3fn)".180- `EncoderDecoderCache` — wrapper for self- and cross-attention caches.181- Model-specific caches (Mamba) mentioned without class names.182- **Not named on the v5.21.1 page:** `QuantoQuantizedCache`, `HQQQuantizedCache`, `OffloadedCache`, `OffloadedStaticCache`, `SlidingWindowCache`, `HybridCache`, `SinkCache` (these were separate classes in older 4.x docs; in v5 quantisation/offloading/sliding are *layer* options). Sliding-window behaviour is described for Mistral/Gemma2 under DynamicCache/StaticCache.183- API reference `internal/generation_utils` "Caches" section lists exactly: `CacheLayerMixin`, `DynamicLayer`, `StaticLayer`, `StaticSlidingWindowLayer`, `QuantoQuantizedLayer`, `HQQQuantizedLayer`, `Fp8QuantizedLayer`, `Cache`, `DynamicCache`, `StaticCache`, `QuantizedCache`, `EncoderDecoderCache`. Methods on `Cache`: `update`, `early_initialization`, `get_seq_length`, `get_mask_sizes`, `get_max_length`, `reset`, `reorder_cache`, `crop`, `batch_repeat_interleave`, … (`to_legacy_cache` / `from_legacy_cache` / `batch_split` not found in the fetched text).184185**Serialisation / reuse across calls**186- Section "Iterative generation": one `DynamicCache` object is passed as `past_key_values` across successive `generate()` calls in a chat loop.187- Section "Prefill a cache (prefix caching)" (verbatim heading): "you may want to fill a Cache with kv pairs for a certain prefix prompt and reuse it to generate different sequences." Example pre-fills a `StaticCache(config=model.config, max_cache_len=1024)` with a forward pass, then for each query does `past_key_values = copy.deepcopy(prompt_cache)` and `model.generate(**new_inputs, past_key_values=past_key_values, …)`.188- Section "Crop a cache": `past_key_values.crop(-3)` rolls back tokens; `activate_past_recording()` needed for sliding-window/linear-attention layers.189- `cache_position` is **not mentioned** on the fetched `kv_cache` page (unverified for this version). No built-in save-to-disk API is documented on this page (reuse is in-process via object/deepcopy; serialisation would be via `torch.save` of the object — not stated → unverified).190191**Verification level:** page fetched (two pages, full text).192193---194195## 6. Anthropic prompt caching + models overview (`anthropic_prompt_caching`)196197**URLs fetched**198- https://platform.claude.com/docs/en/build-with-claude/prompt-caching (full page, 165 867 chars, read in two parts)199- https://platform.claude.com/docs/en/about-claude/models/overview (served as "Models overview", canonical `…/docs/en/models/overview`)200201**How `cache_control` works (verbatim where quoted)**202- Two modes: **automatic caching** — "Add a single `cache_control` field at the top level of your request. The system automatically applies the cache breakpoint to the last cacheable block and moves it forward as conversations grow." — and **explicit breakpoints** — "Place `cache_control` directly on individual content blocks". Block value: `{"cache_control": {"type": "ephemeral"}}`; only `ephemeral` is supported. Up to **4** explicit breakpoints.203- Prefix order: "Cache prefixes are created in the following order: `tools`, `system`, then `messages`." Cache hit requires 100 % identical prefix up to and including the marked block.204- Lookup: "Cache writes occur only at your breakpoint … a hash of the prefix ending at that block." "Cache reads look backward for entries that prior requests wrote." "The lookback window is 20 blocks."205- Multi-turn pattern: the final block of the final message is marked each turn; earlier blocks no longer marked are still hits (and refreshes) within the TTL.206- Invalidation hierarchy `tools → system → messages`; e.g. changing tool definitions invalidates everything; adding/removing images invalidates tools+system levels; "Thinking blocks cannot be cached directly with `cache_control`". JSON key-order instability (Swift, Go) breaks caches.207- Data retention: "KV (key-value) cache representations and cryptographic hashes of cached content are held in memory only and are not stored at rest." Caches isolated by organisation and, on the Claude API / Claude Platform on AWS / Microsoft Foundry, by workspace.208209**Minimum cacheable tokens (table on page)**210- 512: Claude Fable 5.1, Claude Mythos 5.1, Claude Opus 5.5, Claude Opus 5, Claude Sonnet 5.5, Claude Fable 5, Claude Mythos 5, Claude Haiku 5.5211- 1 024: Claude Opus 4.8, Claude Sonnet 5, Claude Sonnet 4.6, Claude Sonnet 4.5, Claude Opus 4.1, Claude Opus 4, Claude Sonnet 4212- 2 048: Claude Mythos Preview, Claude Opus 4.7, Claude Haiku 3.5213- 4 096: Claude Opus 4.6, Claude Opus 4.5, Claude Haiku 4.5214- "Shorter prompts cannot be cached, even if marked with `cache_control` … processed without caching, and no error is returned."215216**TTL**217- Default **5 minutes**, refreshed at no extra cost on every hit; optional **1 hour** via `"cache_control": {"type": "ephemeral", "ttl": "1h"}` at higher write cost. "The lifetime is measured from the start of the request that writes or reads the cache entry, not from the end of its response."218219**Usage fields (verbatim)**220- `cache_creation_input_tokens`: "Number of tokens written to the cache when creating a new entry"221- `cache_read_input_tokens`: "Number of tokens retrieved from the cache for this request"222- `input_tokens`: "Number of input tokens that were not read from or used to create a cache (that is, tokens after the last cache breakpoint)"223- `cache_creation`: object with `ephemeral_5m_input_tokens`, `ephemeral_1h_input_tokens`; `cache_creation_input_tokens` equals their sum.224- `total_input_tokens = cache_read_input_tokens + cache_creation_input_tokens + input_tokens`.225226**Pricing multipliers (as stated)**227- "Cache write tokens with a 5-minute TTL are 1.25 times the base input tokens price." "Cache write tokens with a 1-hour TTL are 2 times the base input tokens price." "Cache read tokens are 0.1 times the base input tokens price" — footnotes: Fable 5.1 / Mythos 5.1 reads at **0.025x**; Opus 5.5 / Sonnet 5.5 reads at **0.05x**; all others 0.1x. "Cache breakpoints themselves don't add any cost."228- Per-model table (USD / MTok: base input / 5m write / 1h write / read / output): Fable 5.1 10 / 12.50 / 20 / 0.25 / 50 · Mythos 5.1 10 / 12.50 / 20 / 0.25 / 50 · Fable 5 10 / 12.50 / 20 / 1 / 50 · Mythos 5 10 / 12.50 / 20 / 1 / 50 · Opus 5.5 4 / 5 / 8 / 0.20 / 20 · Opus 5 5 / 6.25 / 10 / 0.50 / 25 · Opus 4.8, 4.7, 4.6, 4.5 5 / 6.25 / 10 / 0.50 / 25 · Opus 4.1, Opus 4 15 / 18.75 / 30 / 1.50 / 75 · Sonnet 5.5 2 / 2.50 / 4 / 0.10 / 10 · Sonnet 5 2 / 2.50 / 4 / 0.20 / 10 · Sonnet 4.6, 4.5, 4 3 / 3.75 / 6 / 0.30 / 15 · Haiku 5.5 (≤100K) 0.10 / 0.125 / 0.20 / 0.01 / 0.50 · Haiku 5.5 (>100K) 0.50 / 0.625 / 1 / 0.05 / 2.50 · Haiku 4.5 1 / 1.25 / 2 / 0.10 / 5 · Haiku 3.5 0.80 / 1 / 1.60 / 0.08 / 4.229230**Supported models (as stated)**231- "Prompt caching (both automatic and explicit) is supported on all active Claude models." Page metadata lists: `claude-fable-5-1`, `claude-mythos-5-1`, `claude-fable-5`, `claude-mythos-5`, `claude-mythos-preview`, `claude-opus-5-5`, `claude-opus-5`, `claude-sonnet-5-5`, `claude-sonnet-5`, `claude-haiku-5-5`, `claude-haiku-4-5-20251001`, `claude-opus-4-8`, `claude-opus-4-7`, `claude-opus-4-6`, `claude-sonnet-4-6`, `claude-opus-4-5-20251101`, `claude-sonnet-4-5-20250929`. Automatic (top-level) caching unavailable on legacy Bedrock integration (Opus 4.6 and earlier).232- Batches API: cache hits "on a best-effort basis".233234**Models overview page (current lineup on 2026-10-29)**235| Model | Claude API ID (= alias; dateless IDs are pinned snapshots) | Input $/MTok | Output $/MTok | Context | Max output | Cutoff | Retirement not before |236|---|---|---|---|---|---|---|---|237| Claude Fable 5.1 | `claude-fable-5-1` | 10 | 50 | 1M | 128K | Jun 2026 | 2027-09-01 |238| Claude Opus 5.5 | `claude-opus-5-5` | 4 | 20 | 1M | 128K | Jun 2026 | 2027-09-22 |239| Claude Sonnet 5.5 | `claude-sonnet-5-5` | 2 | 10 | 1M | 128K | Jun 2026 | 2027-09-28 |240| Claude Haiku 5.5 | `claude-haiku-5-5` | from 0.10 | from 0.50 | 1M | 128K | Jun 2026 | 2027-10-07 |241Footnote on page: "prompt cache reads cost 10% of the base input price (2.5% on Claude Fable 5.1 and Claude Mythos 5.1, 5% on Claude Opus 5.5 and Claude Sonnet 5.5)". Legacy models still available (IDs per caching page): Fable 5 (`claude-fable-5`), Opus 5 (`claude-opus-5`), Opus 4.8 (`claude-opus-4-8`), Opus 4.7 (`claude-opus-4-7`), Opus 4.6 (`claude-opus-4-6`), Opus 4.5 (`claude-opus-4-5-20251101`), Sonnet 5 (`claude-sonnet-5`), Sonnet 4.6 (`claude-sonnet-4-6`), Haiku 4.5 (`claude-haiku-4-5-20251001`). Bedrock IDs: `anthropic.claude-fable-5-1`, `anthropic.claude-opus-5-5`, `anthropic.claude-sonnet-5-5`, `anthropic.claude-haiku-5-5`. The individual legacy-model prices on the overview page were not expanded; they appear in the caching pricing table above.242243**Verification level:** page fetched (both pages, full text). Docs are living pages — cite with access date.244245---246247## 7. Additional 2025–2026 work (web searches run 2026-10-29)248249Searches: (a) "prompt compression survey 2025 arXiv LLM"; (b) "KV cache compression survey 2025 arXiv"; (c) "multi-agent LLM communication cost token efficiency agent memory sharing KV cache 2025 2026 arXiv"; (d) "shared memory / memory sharing LLM agents cross-agent knowledge transfer compressed context". Each candidate's arXiv abstract page was then fetched.250251| # | arXiv | Title | Authors | Venue (as stated) | Relevance | Verification |252|---|---|---|---|---|---|---|253| 1 | 2410.12388 (v2 17 Oct 2024) | Prompt Compression for Large Language Models: A Survey | Zongqian Li, Yinhong Liu, Yixuan Su, Nigel Collier | arXiv Comments empty; search snippet says NAACL 2025 Long (pp. 7182–7195) — venue unverified | Taxonomy hard vs soft prompt compression; positions LLMLingua-2 | arXiv page fetched; venue snippet only |254| 2 | 2508.06297 (8 Aug 2025) | KV Cache Compression for Inference Efficiency in LLMs: A Review | Yanyu Liu, Jingying Fu, Sixiang Liu, Yitian Zou, You Fu, Jiehan Zhou, Shouhua Zhang | "12 pages" (no venue) | Review of selective/quantisation/attention-based KV compression | arXiv page fetched |255| 3 | 2503.11816 (v3 22 Apr 2025) | Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques | Neusha Javidnia, Bita Darvish Rouhani, Farinaz Koushanfar | IEEE CICC 2025 (invited) | Taxonomy + accuracy/latency evaluation of KV compression | arXiv page fetched |256| 4 | 2503.24000 (31 Mar 2025) | Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving | Wei Gao, Xinyu Zhou, Peng Sun, Tianwei Zhang, Yonggang Wen | MLSys 2025 | Shows KV compression can lengthen outputs and raise end-to-end latency; serving-side benchmark | arXiv page fetched |257| 5 | 2510.12872 (v2 1 Nov 2025) | KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems | Hancheng Ye, Zhengqi Gao, Mingyuan Ma, Qinsi Wang, Yuzhe Fu, Ming-Yu Chung, Yueqian Lin, Zhijian Liu, Jianyi Zhang, Danyang Zhuo, Yiran Chen | NeurIPS 2025 | Training-free cross-agent KV-cache reuse with context-aware offsets; ~6.7x prefill speedup (3 agents) — closest prior work to WP30 | arXiv page fetched |258| 6 | 2604.03143 (3 Apr 2026) | TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing | Zhuohang Bian, Feiyang Wu, Chengrui Zhang, Hangcheng Dong, Yun Liang, Youwei Zhuo | "preprint under review" | All-gather KV sharing, sparse diffs vs master copy; up to 17.5x lower per-agent KV storage | arXiv page fetched |259| 7 | 2602.01053 (v2 31 May 2026) | LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents | Hyesung Jeon, Hyeongju Ha, Jae-Joon Kim | ICML 2026 Poster (journal-ref field) | Shares base-model KV part across LoRA agents, low-rank adapter part | arXiv page fetched |260| 8 | 2603.23234 (v2 28 May 2026) | MemCollab: Cross-Model Memory Collaboration via Contrastive Trajectory Distillation | Yurui Chang, Yiran Wu, Qingyun Wu, Lu Lin | none stated | Shared memory across agents with different backbones; knowledge transfer at the memory (not KV) level | arXiv page fetched |261262Seen in search snippets only (not fetched, not in CSV): "Latent Briefing" (Ramp Labs blog, KV-cache compaction for agent hand-off, LongBench v2, −65 % worker tokens); "Context Engineering 2.0" (arXiv 2510.26493, survey); INMS "Memory Sharing for LLM based Agents" (arXiv 2404.09982 v3, 2026 revision); TOAP (Zenodo, June 2026).263264---265266## Open items / not verified267- LongBench dataset-level licence (no `license` field on HF card) and licences of `2wikimqa` data and `multifieldqa_en`.268- LongBench v2 publication venue (arXiv Comments silent).269- Licence of RULER's generated haystacks (depends on Paul Graham essays / SQuAD / HotpotQA).270- Whether Transformers v5.21.1 documents `cache_position` on the KV-cache page (not seen) and a to-disk serialisation API (not documented on that page).271- NAACL 2025 venue of the prompt-compression survey (search snippet only).272- `reproduction_status` for all sources = pending.273