--- project: localvm-research document: research/notes/out_of_core_memory_systems author: Simon-Pierre Boucher contact: contact@spboucher.ai created: 2026-08-11 status: draft --- # Out-of-core LLM inference and memory systems (charter §4.4 + §4.8) Research notes on out-of-core inference systems, macOS/Apple Silicon substrate facts, and memory-systems ideas from OS/architecture/databases that have not yet been translated to neural-weight execution. Target hardware framing throughout: **Apple M5 Max, 48 GB unified memory, macOS 27, internal Apple NVMe AP2048Z (2 TB), Metal/MLX**. All sources listed in §6 were located and, where marked, fetched on 2026-08-11. --- ## 1. Landscape Out-of-core inference means executing a model whose weights (and/or KV cache) exceed fast memory, by staging data across a hierarchy: GPU HBM → CPU DRAM → NVMe → (network). Every published system is a point in a small design space defined by four questions: 1. **What is the unit of paging?** Whole layers (AirLLM, ZeRO-Inference), tensors/blocks (FlexGen, llama.cpp pages), neurons/neuron clusters (PowerInfer-1/2, LLM-in-a-flash, M2Cache), experts (MoE offloading), or KV blocks (SolidAttention, InstInfer, Swarm). 2. **What decides residency?** Static placement (FlexGen's linear program, PowerInfer's offline hot/cold profiling), OS demand paging (llama.cpp mmap), explicit LRU/segmented caches (Eliseev–Mazur, PowerInfer-2, M2Cache), or predictors (LLM-in-a-flash sparsity predictors, speculative expert prefetch). 3. **What hides transfer latency?** Batch-level pipelining (FlexGen, ZeRO-Inference), compute/I-O overlap at neuron-cluster granularity (PowerInfer-2), speculative prefetch (Eliseev–Mazur, FlexInfer), or nothing (AirLLM). 4. **What is the true bottleneck?** Almost never capacity. For dense models streamed per token it is **storage bandwidth**; for sparse/selective access it is **small-random-read latency and read amplification**; for discrete-GPU systems it is **PCIe bandwidth and CPU↔GPU synchronization**; for batch-throughput systems the bottleneck is deliberately traded against latency. A key structural observation for this project: **most of the literature assumes Linux + discrete GPU**, where there are two memory tiers above storage (VRAM and DRAM) joined by PCIe (~16–64 GB/s). On Apple Silicon the VRAM/DRAM distinction vanishes (unified memory), so the hierarchy collapses to exactly two tiers: **unified RAM (~460–614 GB/s on M5 Max) and internal NVMe (~5–7 GB/s sequential, far less for small random reads)**. That is a ~100:1 bandwidth cliff with no intermediate tier — but also no PCIe copy, no pinned-memory staging, and a legitimate zero-copy path from mmap'd file pages into GPU-visible Metal buffers. Several published designs (FlexGen's DRAM tier, PowerInfer's GPU/CPU split) are meaningless on this substrate; others (LLM-in-a-flash, PowerInfer-2) map almost directly onto it. The single most important accounting identity (charter §2/§8.3): out-of-core systems decouple *capacity* (checkpoint can exceed RAM) but only the sparsity/selectivity systems decouple **bytes-read-per-token** from checkpoint size. AirLLM proves capacity decoupling alone is nearly worthless (full checkpoint read per token → minutes/token); LLM-in-a-flash and MoE-offloading prove bytes/token decoupling is where all the leverage is. --- ## 2. Systems ### 2.1 FlexGen (ICML 2023) - **Mechanism:** Offloading for *throughput-oriented* (latency-insensitive) generation on one commodity GPU. Formulates tensor placement/compute scheduling across GPU/CPU/disk as a linear program over a "zig-zag" block schedule; compresses weights and KV cache to 4 bit. OPT-175B on a single 16 GB T4-class GPU at ~1 token/s *effective* throughput with very large batches. - **Resident vs streamed:** weights partitioned across GPU/CPU/NVMe by the LP; entire active layer streamed per use; KV cache also tiered. - **Bottleneck:** PCIe bandwidth + disk bandwidth, amortized over huge batches; per-request latency is minutes. Bytes/token for the *batch* ≈ full model per layer-pass, divided by batch size per request. - **Hardware assumptions:** Linux, discrete GPU, separate CPU DRAM tier, pinned-memory DMA. None of this maps to Apple Silicon; the LP-placement idea does (placement between RAM and SSD), the batching trick does not help interactive local use. - **macOS arm64:** not supported (CUDA-only). - **Extension opportunity:** the *formalization* — solve residency as an optimization problem given measured tier bandwidths — is reusable for a RAM/SSD split on Mac. ### 2.2 DeepSpeed-Inference / ZeRO-Inference / DeepNVMe - **Mechanism:** ZeRO-Inference streams model weights layer-by-layer from CPU DRAM or NVMe into GPU, overlapping fetch of layer *k+1* with compute of layer *k*; recent versions add weight quantization and KV-cache offload ("20× throughput" claims). DeepNVMe adds io_uring/NVIDIA-GDS-based direct NVMe↔GPU transfer paths. Reported: OPT-175B full-offload ~43 tokens/s (CPU-offload) and ~30 tokens/s (NVMe) *aggregate* with large batches; prefetching improves NVMe offload 1.13–1.21×. - **Resident vs streamed:** essentially nothing resident; **entire model read per forward pass** — bytes/token = checkpoint size unless batched. - **Bottleneck:** PCIe/NVMe *bandwidth* (sequential; layer-granular reads are large, so latency and random I/O are non-issues). Throughput scales with batch size only. - **Hardware assumptions:** Linux (io_uring, O_DIRECT, GDS), discrete GPU. io_uring and O_DIRECT do not exist on macOS. - **Extension opportunity:** demonstrates the hard ceiling of dense layer-streaming: tokens/s ≤ storage_bandwidth / bytes_not_resident. On our SSD (~5–7 GB/s), a 70 GB non-resident model can never exceed ~0.1 tok/s dense, no matter the engineering. ### 2.3 "LLM in a flash" (Apple, arXiv 2312.11514) — the closest prior art to this project - **Mechanism:** keep dense attention weights + embeddings in DRAM (~50 % of model), keep FFN weights on flash; a low-rank predictor (r=128 early layers → r=1024 late layers, <2.4 % overhead) predicts which ReLU-sparse FFN neurons will activate; only those rows are read. **Windowing** (reuse neurons active in the last k=4 tokens; only load the delta) and **row–column bundling** (store up-proj column + down-proj row of the same neuron contiguously, doubling read chunk size) reduce and enlarge I/O respectively. - **What's resident vs streamed:** ~52 % of model in DRAM (OPT-6.7B: 52.1 %, Falcon-7B: 52.9 %); per token only **2.4 % (OPT) / 3.1 % (Falcon) of FFN neurons** are loaded. - **Measured (on macOS!):** Apple **M1 Max, 1 TB SSD** and M2 Ultra 2 TB: >6 GiB/s for a 1 GiB linear read, but random-read throughput *increases with chunk size and thread count*; effective ~2.25 GB/s after bundling. I/O latency per token dropped from 2196 ms (naive half-model reload) to **105 ms** on M1 Max. 4–5× CPU and 20–25× GPU speedup vs naive loading; runs models ~2× DRAM size. - **Bottleneck:** flash random-read latency / read amplification — attacked by making reads bigger (bundling) and fewer (windowing, predictor). - **Limitations:** depends on **ReLU-level activation sparsity** (OPT, ReLU-fied Falcon); modern SwiGLU models (Llama-3, Qwen) have far weaker natural sparsity. No public code. Only ~2× DRAM demonstrated. - **Extension opportunity:** it validates the whole premise of localvm-research *on our exact platform*, and its flash-throughput-vs-chunk-size measurements are directly reusable priors for expH. Open gap: doing this for SwiGLU/MoE models, and integrating with Metal GPU compute rather than CPU. ### 2.4 PowerInfer (SOSP 2024) - **Mechanism:** neuron activations follow a power law: a small set of "hot" neurons fire for most inputs. Hot neurons are preloaded into limited GPU VRAM; cold neurons are computed on CPU from DRAM (avoiding PCIe transfer); adaptive per-layer activation predictors + neuron-aware sparse kernels. Up to 11.69× over llama.cpp on a 4090; runs OPT-175B on one consumer GPU. - **Resident vs streamed:** hot neurons resident in VRAM, cold in DRAM; nothing streamed from disk in the base design (model ≤ DRAM assumed). - **Bottleneck addressed:** PCIe transfer + VRAM capacity; **assumes model fits in DRAM** — it is a VRAM/DRAM tiering system, not a true out-of-core system. - **macOS mapping:** the GPU/CPU split is meaningless under unified memory; the analog is **hot neurons wired in RAM / cold neurons on SSD**, i.e., exactly the LLM-in-a-flash regime. The transferable asset is the *power-law hotness statistics* and the offline profiling methodology (feeds our expA/expC). ### 2.5 PowerInfer-2 (arXiv 2406.06282) - **Mechanism:** smartphone (24 GB Snapdragon, UFS 4.0 flash) inference of models beyond DRAM. Decomposes matmuls into **neuron clusters**; NPU handles dense prefill with large clusters, CPU handles sparse decode with small clusters; **segmented neuron cache** with per-segment policies; fine-grained I/O/compute pipelining at cluster granularity; I/O reads sized/aligned to flash characteristics. - **Reported:** first to run a 47B model on a phone; up to 11.68 tokens/s for TurboSparse- Mixtral-47B; ~22× faster than llama.cpp-class baselines when the model doesn't fit. - **Bottleneck:** UFS random-read latency + bandwidth, hidden with cluster-level pipelining; synchronization overhead of fine-grained pipelining explicitly engineered around. - **Hardware assumptions:** Android/Linux, heterogeneous NPU/CPU, UFS (much slower than Apple NVMe). **This is the closest architectural template for a Mac**: single shared memory pool + flash, no discrete GPU. Its flash is ~4× slower than ours; unified-memory Metal compute is far stronger than a phone CPU — the design should transfer favorably. - **Limitation:** requires activation-sparse ("TurboSparse") fine-tuned model variants — violates our "original model is the source model" constraint unless sparsity is achieved post-hoc. ### 2.6 M2Cache (arXiv 2410.14740) - **Mechanism:** neuron-level **mixed-precision** + **three-tier cache**: neuron-level mixed-precision LRU cache in GPU HBM → layer-aware DRAM cache → full model on SSD. A predictor scores neuron activity; important neurons run FP16, less active ones are quantized more aggressively or demoted. - **Relevance:** first system I found that combines *precision* and *tier* as one axis — i.e., the residency decision and the precision decision are unified. Directly relevant to charter expD (progressive reconstruction): the "compressed tier" idea (§4.8 zswap analogy) already has one instantiation. - **Bottleneck:** SSD bandwidth; hidden with precision reduction (fewer bytes) rather than only prediction. - **macOS:** CUDA; concepts portable. ### 2.7 SolidAttention (FAST 2026) - **Mechanism:** KV-cache (not weights) on SSD for memory-constrained PCs; identifies the conflict between *dynamic sparse attention* (wants small random reads) and *SSD characteristics* (want large sequential reads); consolidates KV pairs into blocks as the transfer unit, transforming irregular access into coarse-grained sequential access. Up to 3.1× faster inference, 98 % KV memory reduction at 128k context, ≤11 % throughput degradation vs in-memory. - **Lesson:** identical shape to LLM-in-a-flash's bundling lesson, proven for KV instead of weights: **on flash, the paging unit must be chosen by the storage medium, not by the model's natural granularity**. Any localvm design must co-design block layout with the ~16–256 KB sweet spot of Apple NVMe. ### 2.8 InstInfer / computational storage (arXiv 2409.04992) - **Mechanism:** offloads attention computation *into* CSD (computational storage drives), so KV never crosses the host bus. Near-storage inference. - **macOS relevance:** none directly (Apple SSD controllers are closed), but conceptually: Apple's SSD controller already does inline encryption/compression; "compute where data lives" on a Mac translates to *decompression/dequantization on GPU at load time*, which MLX quantized kernels already do. ### 2.9 llama.cpp (mmap + Metal) — most important existing macOS evidence - **Mechanism:** since PR #613/issue #91 (2023), model files are `mmap`'d; loading is lazy via page faults and the OS page cache (100× faster warm loads, half the memory — weights live once in the unified buffer cache, shared across processes). On Apple Silicon, the **Metal backend accesses the mmap'd weights zero-copy via `MTLResourceStorageModeShared`** (ggerganov: Metal "looks directly at the memory mapped buffers"; CUDA by contrast copies into device buffers). GGUF tensor data is alignment- padded, which is what makes wrapping file pages in Metal buffers possible. - **Out-of-core behavior:** when a model (esp. MoE) exceeds RAM, execution *works* by OS demand paging: hot expert pages stay in the UBC, cold ones fault in from SSD. Discussion #18758 (Dec 2025–2026) measured this concretely, including **on an M5 Pro (Apple SSD AP1024Z) with Qwen3-Next-80B-A3B**: expert weights are 95.4 % of file bytes; replacing fault-based streaming with explicit layout-aware slice reads gave +13–14 % end-to-end and reduced cold-decode reads from 1418 to 370 per token (**2.23× faster cold-decode I/O**); mmap beat direct I/O because the page cache retains the hot expert working set across tokens/runs. - **Bottleneck:** SSD random-read amplification (16 KB page faults scattered across expert tensors) + page-cache eviction unpredictability. Layout (contiguous per-expert placement) is as important as caching policy. - **Limitations:** replacement policy is the kernel's (approximate LRU, opaque, scan-vulnerable); no model-aware prefetch (a router decision is known *before* the expert FFN runs, but nothing uses it); feature request #20757 (two-tier GPU+RAM expert cache with pluggable eviction) is open — i.e., **the gap is acknowledged and unfilled**. - **Runs on macOS arm64:** yes, first-class. ### 2.10 MLX (Apple) - **Mechanism:** arrays live in unified memory; device chosen *per operation* (`stream=mx.cpu/mx.gpu`) with automatic cross-stream dependencies; **lazy evaluation** builds a graph and materializes arrays only when needed; `mx.load` on safetensors/GGUF is lazy — weights are read from file when first evaluated, not all at load. - **Out-of-core status:** none. Maintainer (awni, discussion #615) states mmap wouldn't solve the problem since weights must still be materialized in (wired) memory for GPU use; lazy loading is the offered substitute. A community mmap prototype (antbob) found the practical blockers: safetensors tensor offsets are not page-aligned (Metal `bytesNoCopy` needs page alignment), and once the model exceeds RAM, uncontrolled page- cache eviction collapsed a 70 GB model on 64 GB hardware to **0.025 tokens/s** (vs 6 tok/s quantized-fits-in-RAM). This is the negative result that defines our problem: naive mmap + kernel LRU is catastrophically bad for cyclic dense weight access. - **Memory control:** `mx.set_wired_limit` / Metal **residency sets** pin working memory; macOS `iogpu.wired_limit_mb` caps total GPU-wired memory (default ~66–75 % of RAM). mlx-lm issue #883 documents the failure mode when wiring is unbounded: IOGPUMemory kernel panic — wired memory is invisible to compressor/jetsam. - **Extension opportunity:** MLX is the natural host for our runtime (custom Metal kernels, lazy graph, quantized matmuls), but every paging/caching mechanism must be built *by us* — MLX offers none. ### 2.11 MoE offloading line: Eliseev & Mazur; caching/prefetch analyses; MoBiLE; cache-conditional experts - **Eliseev & Mazur (arXiv 2312.17238):** Mixtral-8x7B on 11–16 GB consumer GPUs. Exploits (a) temporal locality of expert choice between adjacent tokens → **LRU expert cache**; (b) hidden state of layer *k* already predicts layer *k+1*'s router choice → **speculative expert prefetch** (apply next layer's gate to current hidden state). 2–3 tokens/s on T4/RTX 3060-class GPUs with mixed quantization. This is "branch prediction for weights" in embryonic form. - **In-depth caching/prefetching analysis (arXiv 2511.05814):** measures expert reuse and prefetch accuracy across MoE models — confirms LRU-friendly temporal locality and cross-layer predictability are general, not Mixtral quirks. - **Mixture of cache-conditional experts (arXiv 2412.00099):** *inverts* the problem — biases the router toward experts already in cache (cache-aware routing), trading a little quality for dramatically fewer misses. Notable: the model adapts to the memory system rather than vice versa. - **MoBiLE (arXiv 2510.12357):** "big/little" experts on consumer GPUs — fallback to smaller substitutes for missing experts rather than blocking on I/O. A quality-for-latency miss handler — architecturally interesting for us: a miss need not stall if a low-precision resident approximation exists (connects to expD). ### 2.12 Petals (BitTorrent-style distributed inference) - **Mechanism:** transformer blocks sharded across volunteer consumer GPUs; activations forwarded peer-to-peer (DHT discovery); Llama-3.1-405B / BLOOM-176B at ~1 step/s — claimed up to 10× faster than local disk offloading. - **Relevance:** replaces the SSD tier with a network tier (both ~GB/s, both high-latency) — confirms that *activations are the cheap thing to move; weights are the expensive thing*. For localvm the analogous observation: moving hidden states between compute contexts is ~MB/token; moving weights is ~GB/token. Any decomposition should ship activations, not weights. (Also relevant to the MacLustr cluster as a side path, though out of scope for the single-Mac charter.) ### 2.13 AirLLM (layer-by-layer streaming) - **Mechanism:** load layer → compute → free → next layer; peak memory ~4 GB for a 70B model. Bytes/token = **entire checkpoint** (every layer re-read per token unless cached); a single response takes 15–30 minutes. - **Value:** the perfect *straw-man baseline* for our harness — pure capacity decoupling with zero bytes/token decoupling. Its existence proves the charter's core inequality (total size ≠ resident size ≠ bytes/token) is the entire game. ### 2.14 FlexInfer (arXiv 2503.03777) and Glinthawk (arXiv 2501.11779) - **FlexInfer:** on-device offloading with asynchronous prefetching, **balanced memory locking** (explicitly budgeting pinned vs pageable memory), and flexible tensor preservation; up to 12.5× over existing offloading under tight memory. The "balanced memory locking" idea maps directly onto the macOS wired-limit / residency-set tension documented in §3. - **Glinthawk:** two-tier architecture for *offline* batch inference (throughput regime, like FlexGen) — noted for completeness; wrong latency regime for us. --- ## 3. macOS / Apple Silicon substrate (API-level facts) ### 3.1 Files, page cache, and the absence of direct I/O - **No `O_DIRECT`, no `posix_fadvise`.** The macOS substitute is `fcntl(fd, F_NOCACHE, 1)`: it *hints* that pages should not be cached going forward, but (a) it does **not purge already-cached pages** — subsequent reads still hit them; (b) other processes can keep re-populating the cache; (c) it is per-fd advisory, not a DMA path (Apple dev forums #25464; fio issue #48). Community direct-I/O libraries document that with F_NOCACHE, **unaligned reads are still buffered**; to actually bypass the cache, offset/length should be 4096-byte (better: 16 KB page) aligned (ronomon/direct-io). - **Unified Buffer Cache (UBC):** since Mac OS X, the buffer cache and VM page cache are one; `mmap`'d file pages *are* the file cache pages. Consequences: warm model loads are ~free (llama.cpp's 100× warm-load speedup); memory shows as "cached files," is reclaimable, and is shared across processes mapping the same model file. - **Eviction is opaque:** the kernel's replacement is approximate LRU over the whole system; there is no `fadvise(DONTNEED/WILLNEED)`; `madvise` exists but with weaker semantics. `purge(8)` clears the disk cache for cold-start experiments; `vm_stat` exposes pageins/pageouts/compressor counters for instrumentation (expH must use both). - **Writeback of `mmap(MAP_SHARED)` dirty pages** is at the OS's discretion until `msync` (Apple dev forums #763058) — matters if we ever write compiled-model caches through mmap. - **APFS:** copy-on-write, 4 KB blocks; clones and sparse files are free — useful for storing multiple weight layouts of the same checkpoint without duplicating cold data. (Performance note: APFS metadata ops are slow relative to data reads; large flat blob files with internal indexing beat many-small-files layouts.) ### 3.2 Memory: compression, wiring, jetsam - **Compressor:** since OS X 10.9, LRU-cold anonymous pages are compressed (WKdm-family algorithm, tiny 16-entry dictionary, ~2:1 on pointer/integer-rich data) before any swap. **Quantized/fp16 weights are near-incompressible entropy**, so the compressor gives ~0 benefit on weight pages while burning CPU — weight overflow should go to *file-backed* (evict-don't-compress) memory, never anonymous memory. This asymmetry (file-backed pages get dropped, anonymous pages get compressed/swapped) is a design lever. - **Wired memory:** GPU-active buffers must be wired (non-pageable, non-compressible). Cap is `iogpu.wired_limit_mb` — default ≈ 66–75 % of RAM (≈ 32–36 GB on our 48 GB M5 Max), adjustable via `sudo sysctl iogpu.wired_limit_mb=N` (resets on reboot; unsupported by Apple; leave 8–16 GB headroom). MLX exposes `set_wired_limit` / residency sets. Over-wiring does not trigger graceful jetsam — mlx-lm #883 shows it can end in an **IOGPUMemory kernel panic**; macOS prefers compression over jetsam-style killing, but wired memory is exempt from both, hence the panic path. - **Page size is 16 KB** on Apple Silicon — the natural minimum paging unit for any weight-block store (also the fault granularity that produced the 1418 reads/token in llama.cpp #18758). ### 3.3 Metal: zero-copy, heaps, purgeability - **`newBufferWithBytesNoCopy` / `makeBuffer(bytesNoCopy:)`** wraps existing memory as a `MTLBuffer` with **no copy**, but the pointer must be page-aligned and the length a multiple of page size, and the memory must come from `mmap`/`vm_allocate` (not `malloc`) (Apple docs; dev forums #8011). This is exactly how llama.cpp gets the GPU to read weights straight out of the page cache. **Design consequence: our compiled weight format must place every independently-pageable block on a 16 KB boundary** (GGUF does alignment padding; safetensors does not — the root cause of MLX's mmap dead-end). - **`MTLStorageModeShared`:** CPU and GPU share the allocation in system memory — the default and correct mode on Apple Silicon (no managed/private copies needed). - **`MTLHeap`:** suballocate many buffers from one allocation; resources can alias; **`setPurgeableState`** on a heap makes its whole backing memory *volatile* — the OS may reclaim it under pressure and tells you on reacquire whether contents survived. **A purgeable MTLHeap is a kernel-cooperative weight cache**: warm blocks live there, the OS reclaims them instead of paging/killing, and we re-fault from SSD on loss. No LLM runtime uses this today (see §4). - **Residency sets / wired limit:** the modern mechanism to guarantee the *hot* tier stays resident during command-buffer execution (MLX's wired-memory doc). ### 3.4 Apple NVMe (AP-class) measured behavior - **Sequential:** recent MacBook Pro internal SSDs (AP2048/AP4096-class) measure ~5.4–7.3 GB/s reads (Blackmagic/AmorphousDiskMark reports for M4 Max 2–4 TB; M5-generation press claims ~2× M4 SSD speed). "LLM in a flash" measured **>6 GiB/s for a 1 GiB linear read on an M1 Max 1 TB**. - **Small random reads are the cliff:** community AmorphousDiskMark results consistently show a **4K QD1 dip on Apple Silicon** — on the order of tens of MB/s (one documented M1 Pro result: ~32 MB/s = ~8 K IOPS = ~120 µs effective latency), i.e. **~200× below sequential**. Throughput recovers with (a) larger blocks and (b) concurrency: Apple's paper reports random-read throughput "increases with the size of sequential chunks and the number of threads," reaching ~2.25 GB/s effective with 32 KB-bundled multi-threaded reads. Rule of thumb for design: **≥256 KB blocks at QD≥8, or don't bother**; expH must measure our exact AP2048Z across 16 KB–4 MB, QD1–32, cold (`purge`) vs warm, with and without F_NOCACHE, and *while Metal compute runs* (shared memory-controller contention is unmeasured in the literature). - **Bandwidth ratio on target:** M5 Max unified memory = 460–614 GB/s (per Apple specs; config-dependent) vs ~6 GB/s SSD sequential → **~80–100:1**; vs realistic mixed random ~2 GB/s → **~250:1**. Every design decision follows from this ratio. --- ## 4. OS/architecture/database ideas not yet translated to neural-weight execution 1. **Working-set theory (Denning 1968).** No LLM runtime measures a τ-window weight working set W(τ), yet the concept transfers exactly: the set of weight blocks touched in the last τ tokens. Denning's thrashing criterion (if RAM < W(τ), throughput collapses) gives a *principled admission test*: measure W(τ) per model/workload offline and predict — before running — whether a config will thrash (this is what MLX's 0.025 tok/s mmap failure was: W(τ) = whole model for dense access). Working-set *analytics* (Denning's later surveys) also give the math for choosing cache sizes from reuse- distance histograms — directly applicable to expB. 2. **Scan-resistant / adaptive replacement (ARC, 2Q, LIRS).** Kernel page cache and all published expert caches use (approximate) LRU. But transformer weight access is a *mixed* workload: dense layers are pure cyclic scans (LRU's pathological worst case — it evicts every block just before reuse), while MoE experts have recency+frequency structure. ARC's ghost lists would auto-partition between the two. **Nobody has published an ARC/LIRS weight cache.** Even better than ARC: 3. **MRU for cyclic scans (DBMIN's insight).** Databases learned in 1985 that for a looping sequential scan over N pages with a buffer of B