spb/modelmap Public License

1---2project: modelmap3document: expH_capture_cost_frontier — analysis (run #1)4author: Simon-Pierre Boucher5contact: contact@spboucher.ai6website: https://modelmap.io7created: 2026-08-128modified: 2026-08-129status: reviewed10---1112# Analysis — expH run #1 (storage formats + hook overhead)1314Run: `results/expH_capture_cost_frontier/20260812T052441Z/results.json`15Hardware: Apple M5 Max, 48 GB unified memory, macOS 27.0 · torch 2.13.0 (MPS) ·16MLX 0.32.0 · Python 3.14.4. Hypothesis registered before the run in17`hypothesis.md`; config, seed, commit and full manifest embedded in the JSON.1819```text20Hypothesis              : (A) raw mmap ≥ 2× zarr on random-batch reads (warm);21                          (B) retain-capture < 2× plain inference on both22                          backends; copy+write dominates capture cost.23Falsification criterion : (A) dies at ratio < 1.25×; (B) dies at retain > 2×24                          or total > 3×.25Method / Baseline       : as registered (see hypothesis.md) — 3 repeats/cell,26                          plain-forward baselines, sequential-scan references.27Result                  : (A) CONFIRMED. Random-batch reads: raw-mmap 4.09 GB/s,28                          safetensors(mmap) 4.26, zarr-uncompressed 1.27,29                          zarr-zstd 0.38 → mmap/zarr ratio 3.2× (uncompressed)30                          and 10.8× (zstd), both past the 2× bar.31                          Writes: safetensors 8.10 GB/s (single large write),32                          raw-mmap chunked 2.82, zarr-uncompressed 1.49,33                          zarr-zstd 0.54. Seq scans: 3.9–7.3 GB/s all formats.34                          (B) CONFIRMED, asymmetrically. torch-MPS: plain35                          46.7 ms/fwd (±1.6) → retain 57.2 (1.22×) →36                          retain+copy+write 68.8 (1.47×). MLX: plain 36.237                          (±2.6) → retain 36.9 (1.02×) → +copy+write 40.238                          (1.11×). Retain never exceeds 1.25×; totals never39                          exceed 1.5× — far under the 2×/3× kill lines.40                          The "copy+write dominates" sub-claim holds on MLX41                          (+3.3 ms vs +0.7 ms retain) but on torch-MPS the42                          two costs are comparable (+10.5 ms retain,43                          +11.6 ms copy+write) — retain itself is expensive44                          on MPS, matching its eager hook materialization.45                          Bonus observation (not registered, Level 0): MLX46                          baseline is 1.29× faster than torch-MPS on the47                          identical architecture, and MLX capture is nearly48                          free (2%), consistent with lazy-graph retention.49Interpretation          : Level 0–1. Within-machine replication only50                          (3 repeats/cell, tight variance ≤ ±4 ms); single51                          hardware, single size class, warm cache, synthetic52                          model — all declared in advance. Design decisions53                          this licenses NOW: (1) activation stores are54                          mmap-backed flat files (raw or safetensors container55                          for self-description) — zarr is eliminated for the56                          SAE-shuffle pattern unless cold-cache reverses the57                          ordering; (2) MLX is the preferred capture backend,58                          with torch-MPS as the compatibility path; (3)59                          capture-cost is NOT the bottleneck at this scale —60                          the 4 TB/500M-token storage wall (notes §4.9)61                          remains the real constraint.62Next experiment         : run #2 = cold-cache pass (purge/F_NOCACHE) to test63                          whether the mmap>zarr ordering survives; run #3 =64                          same capture modes on a real 0.5B checkpoint via65                          mlx-lm vs transformers-MPS; then replicate the whole66                          grid on ≥2 cluster Macs (M3U96a, M2U64) to promote67                          the cost table toward Level 1 cross-hardware.68```6970## Notes7172- zarr numbers use chunk = one write-block (4096×4096); different chunk73  geometries could narrow the gap — a registered caveat, not tested here.74- The torch-MPS retain overhead (22%) is per-layer-output retention of75  ~100 MB/forward; capture pipelines that subsample layers will scale it down76  roughly linearly.77- All timing cells and raw seconds are in the results JSON with the hardware78  manifest; the figure-ready aggregation is deferred to the atlas pipeline.7980---8182# Analysis — expH run #2 (cold cache, second hardware) — **HYPOTHESIS FALSIFIED**8384Run: `results/expH_capture_cost_frontier/*_run2_M3U96a/results.json`85Hardware: Apple M3 Ultra (Mac Studio, MacLustr M3U96a), 96 GB, macOS.86Store enlarged to 200k×4096 fp16 (~1.6 GB); `sudo purge` before **every**87timed read repetition. Hypothesis registered before the run.8889```text90Hypothesis              : warm ordering survives cold — mmap ≥ 1.5× zarr on91                          random-batch reads.92Falsification criterion : dies if any zarr variant beats mmap (ratio <1.25×).93Result                  : FALSIFIED — the ordering INVERTS cold.94                          Random-batch (cold): zarr-uncompressed 0.62 GB/s,95                          safetensors(mmap) 0.32, zarr-zstd 0.19,96                          raw-mmap 0.14 → zarr-uncompressed beats mmap 4.4×.97                          Sequential (cold): zarr-uncompressed 3.47,98                          zarr-zstd 3.18, raw-mmap 0.64, safetensors 0.53.99Interpretation          : Level 0–1, and the mechanism is legible: cold mmap100                          reads are page-fault-driven — effectively ~8–16 KiB101                          random reads at queue depth 1, which localvm's102                          cold-cache SSD grid already showed run at103                          ~0.07–0.14 GB/s. zarr instead read()s whole 32 MiB104                          chunks, exactly the ≥256 KiB / high-QD regime where105                          Apple NVMe delivers. The warm-cache run #1 numbers106                          measured memory, not disk. Cross-project agreement107                          (modelmap run #2 ↔ localvm expH) is strong108                          convergent evidence for the real rule:109                          ***IO granularity decides, not the container.***110                          Design decision REVISED: for stores larger than111                          RAM, either use chunked containers or access mmap112                          in chunk-aligned batched reads; never row-level113                          random access. Caveat: run #1 was M5 Max, run #2114                          M3 Ultra — the warm-vs-cold contrast within run #2's115                          own machine is what carries the claim; hardware116                          confound noted for the cross-run comparison.117Next experiment         : run #4 (registered idea, not yet run): mmap with118                          chunk-sorted, ≥1 MiB batched reads + F_RDAHEAD vs119                          zarr-uncompressed, cold, same machine — decides the120                          final store design. Also replicate run #2 on the121                          M5 Max laptop for the hardware axis.122```123124---125126# Analysis — expH run #3 (real 4-bit checkpoint via mlx-lm) — CONFIRMED127128Run: `results/expH_capture_cost_frontier/20260812T061742Z/results.json`129Model: mlx-community/Qwen3-0.6B-4bit (28 layers, d_model 1024), prefill of a130901-token prompt, M5 Max 48 GB. Hypothesis registered before the run.131132```text133Hypothesis              : retain ≤ 1.15× plain prefill; retain+write ≤ 1.5×.134Result                  : CONFIRMED, stronger than predicted.135                          plain 28.5 ms/prefill (31,644 tok/s) →136                          retain 28.6 ms (1.004× — capture is FREE) →137                          retain+copy+write 36.4 ms (1.28×, 24,733 tok/s).138Interpretation          : Level 0–1 (3 repeats, single machine/model). The139                          synthetic run #1 MLX result (1.02×) transfers to a140                          real quantized checkpoint. This run is also the141                          project's first QUANTIZED-model activation capture142                          in Python — the capability the Phase 1 survey found143                          nowhere (notes §4.9). Engineering note: quantized144                          embeddings pack weight shapes, so d_model must be145                          inferred from a runtime activation, and bf16146                          activations must be cast in MLX before NumPy147                          conversion (numpy has no bfloat16).148Next experiment         : same three modes at 1.7B/4B to trace the overhead149                          curve vs model size; then wire this Tap pattern150                          into src/modelmap/capture as the standard MLX151                          capture layer (expA depends on it).152```153