Gate C: predictors (prev_rank, transition depth d, probe), trace simulator, async prefetcher, lazy installs, miss-cost breakdown, v2 traces, LOG/DECISIONS
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
25 changed files +6,470 −24
added
benchmarks/configs/workloads_v2.yaml
+84 −0
@@ -0,0 +1,84 @@ | ||
| 1 | +# Workload v2 (2026-11-01): larger prompt set for predictor fitting and held-out evaluation. Synthetic prompts written for this | |
| 2 | +# study. Families: general, code, reasoning, multilingual, longctx (long instructions), drift (domain switch mid-prompt). | |
| 3 | +families: | |
| 4 | + general: | |
| 5 | + fit: | |
| 6 | + - "Explain in three sentences why mixture-of-experts models are memory-bound during decoding." | |
| 7 | + - "Write a short paragraph describing the climate of Gatineau, Québec, across the four seasons." | |
| 8 | + - "Give five practical tips for a first-year university student learning to manage their time." | |
| 9 | + - "Describe how a hydroelectric dam converts the energy of falling water into electricity." | |
| 10 | + - "Write a friendly email inviting colleagues to a Friday afternoon seminar on housing economics." | |
| 11 | + - "What are the main arguments for and against remote work in knowledge industries?" | |
| 12 | + - "Explain the difference between weather and climate to a ten-year-old." | |
| 13 | + - "Draft a two-paragraph product description for a quiet mechanical keyboard." | |
| 14 | + - "List the steps to prepare a simple vegetable soup, with approximate timings." | |
| 15 | + - "Why do leaves change colour in autumn? Answer in one paragraph." | |
| 16 | + heldout: | |
| 17 | + - "Describe the main differences between a Mac Studio and a MacBook Pro for scientific computing." | |
| 18 | + - "Summarize the history of the Rideau Canal in one paragraph." | |
| 19 | + - "Write a short motivational note for someone starting to learn the piano as an adult." | |
| 20 | + - "What should a visitor know before hiking in Gatineau Park in winter?" | |
| 21 | + code: | |
| 22 | + fit: | |
| 23 | + - "def fibonacci(n):\n \"\"\"Return the n-th Fibonacci number.\"\"\"\n" | |
| 24 | + - "Write a Python function that parses a CSV file of timestamps and returns the median inter-arrival time in milliseconds.\n\n```python\n" | |
| 25 | + - "Implement an LRU cache class in Python with get and put methods in O(1).\n\n```python\n" | |
| 26 | + - "Write a Python script that walks a directory tree and reports the total size per file extension.\n\n```python\n" | |
| 27 | + - "Implement binary search over a sorted list of integers in Python and return the insertion index when absent.\n\n```python\n" | |
| 28 | + - "Write a SQL query that returns the three most recent orders per customer from tables orders(customer_id, order_date, amount).\n\n```sql\n" | |
| 29 | + - "Write a JavaScript function that debounces another function with a configurable delay.\n\n```javascript\n" | |
| 30 | + - "Write a Rust function that counts word frequencies in a string and returns them sorted by count.\n\n```rust\n" | |
| 31 | + - "class Matrix:\n \"\"\"Minimal dense matrix with multiplication and transpose.\"\"\"\n def __init__(self, rows):\n" | |
| 32 | + - "Write a Python generator that yields sliding windows of size k over an iterable.\n\n```python\n" | |
| 33 | + heldout: | |
| 34 | + - "Write a C function that computes the dot product of two float arrays using NEON intrinsics.\n\n```c\n" | |
| 35 | + - "Write a bash script that lists the ten largest files under a directory.\n\n```bash\n" | |
| 36 | + - "Implement Dijkstra's shortest path in Python using a binary heap.\n\n```python\n" | |
| 37 | + - "Write a Go function that reads lines from stdin and prints the longest one.\n\n```go\n" | |
| 38 | + reasoning: | |
| 39 | + fit: | |
| 40 | + - "Q: If a train travels 60 km in 45 minutes, what is its average speed in km/h? Think step by step.\nA:" | |
| 41 | + - "Q: A rectangle has a perimeter of 36 cm and a length twice its width. What is its area? Think step by step.\nA:" | |
| 42 | + - "Q: Three friends split a bill of 84 dollars; one pays twice as much as each of the others. How much does each pay? Think step by step.\nA:" | |
| 43 | + - "Q: A store sells pens at 3 for 2 dollars. How much do 18 pens cost? Think step by step.\nA:" | |
| 44 | + - "Q: If today is Wednesday, what day of the week will it be in 100 days? Think step by step.\nA:" | |
| 45 | + - "Q: A car uses 7 litres per 100 km. How many litres for a 350 km trip, and the cost at 1.60 per litre? Think step by step.\nA:" | |
| 46 | + - "Q: The sum of two consecutive even numbers is 54. What are they? Think step by step.\nA:" | |
| 47 | + - "Q: A recipe for 4 people needs 300 g of flour. How much for 10 people? Think step by step.\nA:" | |
| 48 | + - "Q: A clock shows 3:15. What is the angle between the hour and minute hands? Think step by step.\nA:" | |
| 49 | + - "Q: If 5 machines make 5 widgets in 5 minutes, how long do 100 machines take to make 100 widgets? Think step by step.\nA:" | |
| 50 | + heldout: | |
| 51 | + - "Q: A tank fills in 6 hours with pipe A and 9 hours with pipe B. How long with both pipes? Think step by step.\nA:" | |
| 52 | + - "Q: What is the smallest positive integer divisible by 2, 3, 4, 5 and 6? Think step by step.\nA:" | |
| 53 | + - "Q: A shirt costs 40 dollars after a 20 percent discount. What was the original price? Think step by step.\nA:" | |
| 54 | + - "Q: Two dice are rolled. What is the probability that the sum is 7? Think step by step.\nA:" | |
| 55 | + multilingual: | |
| 56 | + fit: | |
| 57 | + - "Les modèles à mélange d'experts activent seulement une fraction de leurs paramètres parce que" | |
| 58 | + - "Explique en français pourquoi la mémoire unifiée des puces Apple est utile pour l'inférence des grands modèles de langage." | |
| 59 | + - "Rédige un court paragraphe sur l'histoire de la ville de Québec." | |
| 60 | + - "Quels sont les avantages du transport en commun dans une ville de taille moyenne ? Réponds en français." | |
| 61 | + - "Erkläre in drei Sätzen, warum Fahrräder in Städten nützlich sind." | |
| 62 | + - "Describe en español cómo se prepara un buen café con una cafetera italiana." | |
| 63 | + - "Explique en français la différence entre un prêt hypothécaire à taux fixe et à taux variable." | |
| 64 | + - "Scrivi un breve paragrafo sulla cucina tradizionale della Toscana." | |
| 65 | + heldout: | |
| 66 | + - "Décris en quelques phrases le fonctionnement d'un cache LRU." | |
| 67 | + - "¿Cuáles son las principales causas del cambio climático? Responde brevemente en español." | |
| 68 | + - "Beschreibe kurz, wie ein Mietvertrag in Deutschland funktioniert." | |
| 69 | + longctx: | |
| 70 | + fit: | |
| 71 | + - "You are an assistant helping an economics professor. Context: the course covers hedonic pricing, the sales comparison approach, the cost approach and the income approach to real estate valuation. Students have access to a database of 500,000 Quebec transactions from 2021 to 2026 with attributes such as living area, lot size, year built, number of bedrooms and bathrooms, municipality and sale date. The professor wants a tutorial exercise. Task: write detailed instructions for a two-hour lab in which students estimate a hedonic regression, interpret its coefficients, and use it to value three sample properties, including common pitfalls and how to check residuals." | |
| 72 | + - "Context: a small laboratory operates a cluster of twenty-nine Apple Silicon computers connected by gigabit Ethernet, with between 8 and 96 GB of unified memory each, used for scientific computing, web services and language-model inference. The laboratory wants a maintenance policy. Task: write a comprehensive maintenance checklist covering daily, weekly and monthly tasks, monitoring of temperature and disk usage, backup verification, software updates, security hygiene, and an incident response procedure for a power outage, with justification for each item." | |
| 73 | + - "Context: a municipality in Quebec is revising its property assessment roll. Current assessments date from three years ago; prices have risen unevenly across neighbourhoods; some property types (condominiums, plexes, single-family homes) show different trends; the municipality must explain changes to citizens. Task: write a structured briefing note of at least eight paragraphs for the city council describing the methodology options, their fairness implications, communication strategy, and a timeline." | |
| 74 | + heldout: | |
| 75 | + - "Context: a graduate student is designing a benchmark to compare inference runtimes for mixture-of-experts language models on consumer hardware. The runtimes differ in quantization, in how they store experts (in memory or on disk), and in whether they can use several machines. The student has access to three machines with different amounts of memory and a gigabit network. Task: write a detailed experimental protocol with workloads, metrics, controls, repetition strategy, statistical analysis and reporting guidelines, explaining the rationale behind each choice." | |
| 76 | + drift: | |
| 77 | + fit: | |
| 78 | + - "First, write a Python function that reverses a linked list.\n\n```python\n" | |
| 79 | + - "Explain the rules of ice hockey in two paragraphs, then write a SQL query counting goals per player from a table goals(player, minute)." | |
| 80 | + - "Résume en français l'intrigue d'un roman policier imaginaire, puis donne la solution de l'équation 3x + 7 = 22 étape par étape." | |
| 81 | + heldout: | |
| 82 | + - "Describe a sunset over the Ottawa River in one paragraph, then implement quicksort in Python.\n\n```python\n" | |
| 83 | + - "Q: What is 17 times 23? Think step by step.\nA: Then write a haiku about autumn in Québec." | |
| 84 | +output_lengths: [64, 256] | |
added
benchmarks/processed/gateC/sim_olmoe_laptop_ssd_v1.json
+4765 −0
@@ -0,0 +1,4765 @@ | ||
| 1 | +{ | |
| 2 | + "cmd": "benchmarks/scripts/simulate_cache.py --traces benchmarks/raw/gateC/traces_olmoe_laptop --out benchmarks/processed/gateC/sim_olmoe_laptop_ssd_v1.json", | |
| 3 | + "ts": "2026-11-01T01:33:45", | |
| 4 | + "traces": "benchmarks/raw/gateC/traces_olmoe_laptop", | |
| 5 | + "L": 16, | |
| 6 | + "E": 64, | |
| 7 | + "K": 8, | |
| 8 | + "t_read_ms": 0.43, | |
| 9 | + "t_layer_ms": 0.128, | |
| 10 | + "loaders": 1, | |
| 11 | + "results": [ | |
| 12 | + { | |
| 13 | + "capacity": 8, | |
| 14 | + "policy": "lru", | |
| 15 | + "predictor": "none", | |
| 16 | + "m": 8, | |
| 17 | + "hit_rate": 0.4094445257611241, | |
| 18 | + "misses_per_token": 75.59110070257611, | |
| 19 | + "stall_ms_per_token": 32.50417330210768, | |
| 20 | + "prefetch_loads_per_token": 0.0, | |
| 21 | + "prefetch_precision": null, | |
| 22 | + "loads_per_token": 75.59110070257611, | |
| 23 | + "pred_recall_at_k": null, | |
| 24 | + "tokens": 2135, | |
| 25 | + "families": { | |
| 26 | + "general": { | |
| 27 | + "hit_rate": 0.3551158874801902, | |
| 28 | + "stall_ms_per_token": 35.494421553090355 | |
| 29 | + }, | |
| 30 | + "code": { | |
| 31 | + "hit_rate": 0.4682576545778835, | |
| 32 | + "stall_ms_per_token": 29.2670986920333 | |
| 33 | + }, | |
| 34 | + "reasoning": { | |
| 35 | + "hit_rate": 0.3774561036789298, | |
| 36 | + "stall_ms_per_token": 34.26481605351171 | |
| 37 | + }, | |
| 38 | + "multilingual": { | |
| 39 | + "hit_rate": 0.3940161401098901, | |
| 40 | + "stall_ms_per_token": 33.353351648351634 | |
| 41 | + } | |
| 42 | + }, | |
| 43 | + "split": "fit" | |
| 44 | + }, | |
| 45 | + { | |
| 46 | + "capacity": 8, | |
| 47 | + "policy": "lru", | |
| 48 | + "predictor": "none", | |
| 49 | + "m": 8, | |
| 50 | + "hit_rate": 0.4148351054633472, | |
| 51 | + "misses_per_token": 74.90110650069157, | |
| 52 | + "stall_ms_per_token": 32.20747579529736, | |
| 53 | + "prefetch_loads_per_token": 0.0, | |
| 54 | + "prefetch_precision": null, | |
| 55 | + "loads_per_token": 74.90110650069157, | |
| 56 | + "pred_recall_at_k": null, | |
| 57 | + "tokens": 1446, | |
| 58 | + "families": { | |
| 59 | + "general": { | |
| 60 | + "hit_rate": 0.3655154520295203, | |
| 61 | + "stall_ms_per_token": 34.92202952029518 | |
| 62 | + }, | |
| 63 | + "code": { | |
| 64 | + "hit_rate": 0.4808007459312839, | |
| 65 | + "stall_ms_per_token": 28.57672694394212 | |
| 66 | + }, | |
| 67 | + "reasoning": { | |
| 68 | + "hit_rate": 0.38716947115384615, | |
| 69 | + "stall_ms_per_token": 33.730192307692306 | |
| 70 | + }, | |
| 71 | + "multilingual": { | |
| 72 | + "hit_rate": 0.38621794871794873, | |
| 73 | + "stall_ms_per_token": 33.78256410256409 | |
| 74 | + } | |
| 75 | + }, | |
| 76 | + "split": "heldout" | |
| 77 | + }, | |
| 78 | + { | |
| 79 | + "capacity": 8, | |
| 80 | + "policy": "lru", | |
| 81 | + "predictor": "prev_token", | |
| 82 | + "m": 8, | |
| 83 | + "hit_rate": 0.4094445257611241, | |
| 84 | + "misses_per_token": 75.59110070257611, | |
| 85 | + "stall_ms_per_token": 32.50417330210768, | |
| 86 | + "prefetch_loads_per_token": 0.0, | |
| 87 | + "prefetch_precision": null, | |
| 88 | + "loads_per_token": 75.59110070257611, | |
| 89 | + "pred_recall_at_k": null, | |
| 90 | + "tokens": 2135, | |
| 91 | + "families": { | |
| 92 | + "general": { | |
| 93 | + "hit_rate": 0.3551158874801902, | |
| 94 | + "stall_ms_per_token": 35.494421553090355 | |
| 95 | + }, | |
| 96 | + "code": { | |
| 97 | + "hit_rate": 0.4682576545778835, | |
| 98 | + "stall_ms_per_token": 29.2670986920333 | |
| 99 | + }, | |
| 100 | + "reasoning": { | |
| 101 | + "hit_rate": 0.3774561036789298, | |
| 102 | + "stall_ms_per_token": 34.26481605351171 | |
| 103 | + }, | |
| 104 | + "multilingual": { | |
| 105 | + "hit_rate": 0.3940161401098901, | |
| 106 | + "stall_ms_per_token": 33.353351648351634 | |
| 107 | + } | |
| 108 | + }, | |
| 109 | + "split": "fit" | |
| 110 | + }, | |
| 111 | + { | |
| 112 | + "capacity": 8, | |
| 113 | + "policy": "lru", | |
| 114 | + "predictor": "prev_token", | |
| 115 | + "m": 8, | |
| 116 | + "hit_rate": 0.4148351054633472, | |
| 117 | + "misses_per_token": 74.90110650069157, | |
| 118 | + "stall_ms_per_token": 32.20747579529736, | |
| 119 | + "prefetch_loads_per_token": 0.0, | |
| 120 | + "prefetch_precision": null, | |
| 121 | + "loads_per_token": 74.90110650069157, | |
| 122 | + "pred_recall_at_k": null, | |
| 123 | + "tokens": 1446, | |
| 124 | + "families": { | |
| 125 | + "general": { | |
| 126 | + "hit_rate": 0.3655154520295203, | |
| 127 | + "stall_ms_per_token": 34.92202952029518 | |
| 128 | + }, | |
| 129 | + "code": { | |
| 130 | + "hit_rate": 0.4808007459312839, | |
| 131 | + "stall_ms_per_token": 28.57672694394212 | |
| 132 | + }, | |
| 133 | + "reasoning": { | |
| 134 | + "hit_rate": 0.38716947115384615, | |
| 135 | + "stall_ms_per_token": 33.730192307692306 | |
| 136 | + }, | |
| 137 | + "multilingual": { | |
| 138 | + "hit_rate": 0.38621794871794873, | |
| 139 | + "stall_ms_per_token": 33.78256410256409 | |
| 140 | + } | |
| 141 | + }, | |
| 142 | + "split": "heldout" | |
| 143 | + }, | |
| 144 | + { | |
| 145 | + "capacity": 8, | |
| 146 | + "policy": "lru", | |
| 147 | + "predictor": "prev_rank", | |
| 148 | + "m": 8, | |
| 149 | + "hit_rate": 0.40930913348946135, | |
| 150 | + "misses_per_token": 75.60843091334894, | |
| 151 | + "stall_ms_per_token": 32.51162529274001, | |
| 152 | + "prefetch_loads_per_token": 0.04215456674473068, | |
| 153 | + "prefetch_precision": 0.13333333333333333, | |
| 154 | + "loads_per_token": 75.65058548009368, | |
| 155 | + "pred_recall_at_k": null, | |
| 156 | + "tokens": 2135, | |
| 157 | + "families": { | |
| 158 | + "general": { | |
| 159 | + "hit_rate": 0.35500445721077656, | |
| 160 | + "stall_ms_per_token": 35.500554675118885 | |
| 161 | + }, | |
| 162 | + "code": { | |
| 163 | + "hit_rate": 0.46808115338882283, | |
| 164 | + "stall_ms_per_token": 29.276813317479196 | |
| 165 | + }, | |
| 166 | + "reasoning": { | |
| 167 | + "hit_rate": 0.37742997491638797, | |
| 168 | + "stall_ms_per_token": 34.266254180602004 | |
| 169 | + }, | |
| 170 | + "multilingual": { | |
| 171 | + "hit_rate": 0.3938444368131868, | |
| 172 | + "stall_ms_per_token": 33.36280219780218 | |
| 173 | + } | |
| 174 | + }, | |
| 175 | + "split": "fit" | |
| 176 | + }, | |
| 177 | + { | |
| 178 | + "capacity": 8, | |
| 179 | + "policy": "lru", | |
| 180 | + "predictor": "prev_rank", | |
| 181 | + "m": 8, | |
| 182 | + "hit_rate": 0.4147648686030429, | |
| 183 | + "misses_per_token": 74.91009681881052, | |
| 184 | + "stall_ms_per_token": 32.211341632088526, | |
| 185 | + "prefetch_loads_per_token": 0.04080221300138313, | |
| 186 | + "prefetch_precision": 0.3220338983050847, | |
| 187 | + "loads_per_token": 74.9508990318119, | |
| 188 | + "pred_recall_at_k": null, | |
| 189 | + "tokens": 1446, | |
| 190 | + "families": { | |
| 191 | + "general": { | |
| 192 | + "hit_rate": 0.3654289667896679, | |
| 193 | + "stall_ms_per_token": 34.92678966789667 | |
| 194 | + }, | |
| 195 | + "code": { | |
| 196 | + "hit_rate": 0.48073010849909587, | |
| 197 | + "stall_ms_per_token": 28.580614828209747 | |
| 198 | + }, | |
| 199 | + "reasoning": { | |
| 200 | + "hit_rate": 0.3871444310897436, | |
| 201 | + "stall_ms_per_token": 33.731570512820504 | |
| 202 | + }, | |
| 203 | + "multilingual": { | |
| 204 | + "hit_rate": 0.3860176282051282, | |
| 205 | + "stall_ms_per_token": 33.79358974358974 | |
| 206 | + } | |
| 207 | + }, | |
| 208 | + "split": "heldout" | |
| 209 | + }, | |
| 210 | + { | |
| 211 | + "capacity": 8, | |
| 212 | + "policy": "lru", | |
| 213 | + "predictor": "transition", | |
| 214 | + "m": 8, | |
| 215 | + "hit_rate": 0.4211285128805621, | |
| 216 | + "misses_per_token": 74.09555035128805, | |
| 217 | + "stall_ms_per_token": 23.09395597189697, | |
| 218 | + "prefetch_loads_per_token": 84.55690866510538, | |
| 219 | + "prefetch_precision": 0.4305900990976519, | |
| 220 | + "loads_per_token": 158.65245901639344, | |
| 221 | + "pred_recall_at_k": 0.5302576112412178, | |
| 222 | + "tokens": 2135, | |
| 223 | + "families": { | |
| 224 | + "general": { | |
| 225 | + "hit_rate": 0.3951936410459588, | |
| 226 | + "stall_ms_per_token": 24.108852614896982 | |
| 227 | + }, | |
| 228 | + "code": { | |
| 229 | + "hit_rate": 0.44222837395957193, | |
| 230 | + "stall_ms_per_token": 22.29360523186686 | |
| 231 | + }, | |
| 232 | + "reasoning": { | |
| 233 | + "hit_rate": 0.4122073578595318, | |
| 234 | + "stall_ms_per_token": 23.424642140468233 | |
| 235 | + }, | |
| 236 | + "multilingual": { | |
| 237 | + "hit_rate": 0.42466517857142855, | |
| 238 | + "stall_ms_per_token": 22.912142857142857 | |
| 239 | + } | |
| 240 | + }, | |
| 241 | + "split": "fit" | |
| 242 | + }, | |
| 243 | + { | |
| 244 | + "capacity": 8, | |
| 245 | + "policy": "lru", | |
| 246 | + "predictor": "transition", | |
| 247 | + "m": 8, | |
| 248 | + "hit_rate": 0.3991506742738589, | |
| 249 | + "misses_per_token": 76.90871369294605, | |
| 250 | + "stall_ms_per_token": 23.938752420470248, | |
| 251 | + "prefetch_loads_per_token": 86.76832641770402, | |
| 252 | + "prefetch_precision": 0.39181617477105535, | |
| 253 | + "loads_per_token": 163.67704011065007, | |
| 254 | + "pred_recall_at_k": 0.4996369294605809, | |
| 255 | + "tokens": 1446, | |
| 256 | + "families": { | |
| 257 | + "general": { | |
| 258 | + "hit_rate": 0.3642614160516605, | |
| 259 | + "stall_ms_per_token": 25.31311808118083 | |
| 260 | + }, | |
| 261 | + "code": { | |
| 262 | + "hit_rate": 0.42673485533453887, | |
| 263 | + "stall_ms_per_token": 22.857540687160963 | |
| 264 | + }, | |
| 265 | + "reasoning": { | |
| 266 | + "hit_rate": 0.4127353766025641, | |
| 267 | + "stall_ms_per_token": 23.403250000000003 | |
| 268 | + }, | |
| 269 | + "multilingual": { | |
| 270 | + "hit_rate": 0.3842147435897436, | |
| 271 | + "stall_ms_per_token": 24.453641025641026 | |
| 272 | + } | |
| 273 | + }, | |
| 274 | + "split": "heldout" | |
| 275 | + }, | |
| 276 | + { | |
| 277 | + "capacity": 8, | |
| 278 | + "policy": "lru", | |
| 279 | + "predictor": "probe", | |
| 280 | + "m": 8, | |
| 281 | + "hit_rate": 0.1609192037470726, | |
| 282 | + "misses_per_token": 107.40234192037471, | |
| 283 | + "stall_ms_per_token": 33.152607025761114, | |
| 284 | + "prefetch_loads_per_token": 110.36768149882904, | |
| 285 | + "prefetch_precision": 0.1384259553971184, | |
| 286 | + "loads_per_token": 217.77002341920374, | |
| 287 | + "pred_recall_at_k": 0.16113973458235753, | |
| 288 | + "tokens": 2135, | |
| 289 | + "families": { | |
| 290 | + "general": { | |
| 291 | + "hit_rate": 0.18066561014263074, | |
| 292 | + "stall_ms_per_token": 32.40164817749605 | |
| 293 | + }, | |
| 294 | + "code": { | |
| 295 | + "hit_rate": 0.13261741973840666, | |
| 296 | + "stall_ms_per_token": 34.26192627824019 | |
| 297 | + }, | |
| 298 | + "reasoning": { | |
| 299 | + "hit_rate": 0.17545464046822742, | |
| 300 | + "stall_ms_per_token": 32.57655518394648 | |
| 301 | + }, | |
| 302 | + "multilingual": { | |
| 303 | + "hit_rate": 0.18013822115384615, | |
| 304 | + "stall_ms_per_token": 32.36457692307693 | |
| 305 | + } | |
| 306 | + }, | |
| 307 | + "split": "fit" | |
| 308 | + }, | |
| 309 | + { | |
| 310 | + "capacity": 8, | |
| 311 | + "policy": "lru", | |
| 312 | + "predictor": "probe", | |
| 313 | + "m": 8, | |
| 314 | + "hit_rate": 0.15590421853388658, | |
| 315 | + "misses_per_token": 108.04426002766252, | |
| 316 | + "stall_ms_per_token": 33.341687413554666, | |
| 317 | + "prefetch_loads_per_token": 110.74965421853389, | |
| 318 | + "prefetch_precision": 0.13184384054351084, | |
| 319 | + "loads_per_token": 218.7939142461964, | |
| 320 | + "pred_recall_at_k": 0.15494467496542186, | |
| 321 | + "tokens": 1446, | |
| 322 | + "families": { | |
| 323 | + "general": { | |
| 324 | + "hit_rate": 0.15515452029520296, | |
| 325 | + "stall_ms_per_token": 33.39635424354243 | |
| 326 | + }, | |
| 327 | + "code": { | |
| 328 | + "hit_rate": 0.14524468806509946, | |
| 329 | + "stall_ms_per_token": 33.738824593128356 | |
| 330 | + }, | |
| 331 | + "reasoning": { | |
| 332 | + "hit_rate": 0.16997195512820512, | |
| 333 | + "stall_ms_per_token": 32.7875128205128 | |
| 334 | + }, | |
| 335 | + "multilingual": { | |
| 336 | + "hit_rate": 0.2049278846153846, | |
| 337 | + "stall_ms_per_token": 31.384153846153847 | |
| 338 | + } | |
| 339 | + }, | |
| 340 | + "split": "heldout" | |
| 341 | + }, | |
| 342 | + { | |
| 343 | + "capacity": 8, | |
| 344 | + "policy": "lru", | |
| 345 | + "predictor": "oracle", | |
| 346 | + "m": 8, | |
| 347 | + "hit_rate": 0.8011782786885245, | |
| 348 | + "misses_per_token": 25.449180327868852, | |
| 349 | + "stall_ms_per_token": 8.402752224824361, | |
| 350 | + "prefetch_loads_per_token": 80.34426229508196, | |
| 351 | + "prefetch_precision": 1.0, | |
| 352 | + "loads_per_token": 105.79344262295082, | |
| 353 | + "pred_recall_at_k": 1.0, | |
| 354 | + "tokens": 2135, | |
| 355 | + "families": { | |
| 356 | + "general": { | |
| 357 | + "hit_rate": 0.8312079041204438, | |
| 358 | + "stall_ms_per_token": 7.254285261489697 | |
| 359 | + }, | |
| 360 | + "code": { | |
| 361 | + "hit_rate": 0.7669069560047562, | |
| 362 | + "stall_ms_per_token": 9.74282996432819 | |
| 363 | + }, | |
| 364 | + "reasoning": { | |
| 365 | + "hit_rate": 0.8144596571906354, | |
| 366 | + "stall_ms_per_token": 7.875177257525085 | |
| 367 | + }, | |
| 368 | + "multilingual": { | |
| 369 | + "hit_rate": 0.8173935439560439, | |
| 370 | + "stall_ms_per_token": 7.7308351648351605 | |
| 371 | + } | |
| 372 | + }, | |
| 373 | + "split": "fit" | |
| 374 | + }, | |
| 375 | + { | |
| 376 | + "capacity": 8, | |
| 377 | + "policy": "lru", | |
| 378 | + "predictor": "oracle", | |
| 379 | + "m": 8, | |
| 380 | + "hit_rate": 0.804201244813278, | |
| 381 | + "misses_per_token": 25.062240663900415, | |
| 382 | + "stall_ms_per_token": 8.281117565698471, | |
| 383 | + "prefetch_loads_per_token": 80.0103734439834, | |
| 384 | + "prefetch_precision": 1.0, | |
| 385 | + "loads_per_token": 105.07261410788382, | |
| 386 | + "pred_recall_at_k": 1.0, | |
| 387 | + "tokens": 1446, | |
| 388 | + "families": { | |
| 389 | + "general": { | |
| 390 | + "hit_rate": 0.8304168588560885, | |
| 391 | + "stall_ms_per_token": 7.29341328413284 | |
| 392 | + }, | |
| 393 | + "code": { | |
| 394 | + "hit_rate": 0.7711205922242315, | |
| 395 | + "stall_ms_per_token": 9.544965641952981 | |
| 396 | + }, | |
| 397 | + "reasoning": { | |
| 398 | + "hit_rate": 0.8146534455128205, | |
| 399 | + "stall_ms_per_token": 7.866705128205132 | |
| 400 | + }, | |
| 401 | + "multilingual": { | |
| 402 | + "hit_rate": 0.8253205128205128, | |
| 403 | + "stall_ms_per_token": 7.402256410256413 | |
| 404 | + } | |
| 405 | + }, | |
| 406 | + "split": "heldout" | |
| 407 | + }, | |
| 408 | + { | |
| 409 | + "capacity": 8, | |
| 410 | + "policy": "lfu", | |
| 411 | + "predictor": "none", | |
| 412 | + "m": 8, | |
| 413 | + "hit_rate": 0.4094445257611241, | |
| 414 | + "misses_per_token": 75.59110070257611, | |
| 415 | + "stall_ms_per_token": 32.50417330210768, | |
| 416 | + "prefetch_loads_per_token": 0.0, | |
| 417 | + "prefetch_precision": null, | |
| 418 | + "loads_per_token": 75.59110070257611, | |
| 419 | + "pred_recall_at_k": null, | |
| 420 | + "tokens": 2135, | |
| 421 | + "families": { | |
| 422 | + "general": { | |
| 423 | + "hit_rate": 0.3551158874801902, | |
| 424 | + "stall_ms_per_token": 35.494421553090355 | |
| 425 | + }, | |
| 426 | + "code": { | |
| 427 | + "hit_rate": 0.4682576545778835, | |
| 428 | + "stall_ms_per_token": 29.2670986920333 | |
| 429 | + }, | |
| 430 | + "reasoning": { | |
| 431 | + "hit_rate": 0.3774561036789298, | |
| 432 | + "stall_ms_per_token": 34.26481605351171 | |
| 433 | + }, | |
| 434 | + "multilingual": { | |
| 435 | + "hit_rate": 0.3940161401098901, | |
| 436 | + "stall_ms_per_token": 33.353351648351634 | |
| 437 | + } | |
| 438 | + }, | |
| 439 | + "split": "fit" | |
| 440 | + }, | |
| 441 | + { | |
| 442 | + "capacity": 8, | |
| 443 | + "policy": "lfu", | |
| 444 | + "predictor": "none", | |
| 445 | + "m": 8, | |
| 446 | + "hit_rate": 0.4148351054633472, | |
| 447 | + "misses_per_token": 74.90110650069157, | |
| 448 | + "stall_ms_per_token": 32.20747579529736, | |
| 449 | + "prefetch_loads_per_token": 0.0, | |
| 450 | + "prefetch_precision": null, | |
| 451 | + "loads_per_token": 74.90110650069157, | |
| 452 | + "pred_recall_at_k": null, | |
| 453 | + "tokens": 1446, | |
| 454 | + "families": { | |
| 455 | + "general": { | |
| 456 | + "hit_rate": 0.3655154520295203, | |
| 457 | + "stall_ms_per_token": 34.92202952029518 | |
| 458 | + }, | |
| 459 | + "code": { | |
| 460 | + "hit_rate": 0.4808007459312839, | |
| 461 | + "stall_ms_per_token": 28.57672694394212 | |
| 462 | + }, | |
| 463 | + "reasoning": { | |
| 464 | + "hit_rate": 0.38716947115384615, | |
| 465 | + "stall_ms_per_token": 33.730192307692306 | |
| 466 | + }, | |
| 467 | + "multilingual": { | |
| 468 | + "hit_rate": 0.38621794871794873, | |
| 469 | + "stall_ms_per_token": 33.78256410256409 | |
| 470 | + } | |
| 471 | + }, | |
| 472 | + "split": "heldout" | |
| 473 | + }, | |
| 474 | + { | |
| 475 | + "capacity": 8, | |
| 476 | + "policy": "lfu", | |
| 477 | + "predictor": "prev_token", | |
| 478 | + "m": 8, | |
| 479 | + "hit_rate": 0.4094445257611241, | |
| 480 | + "misses_per_token": 75.59110070257611, | |
| 481 | + "stall_ms_per_token": 32.50417330210768, | |
| 482 | + "prefetch_loads_per_token": 0.0, | |
| 483 | + "prefetch_precision": null, | |
| 484 | + "loads_per_token": 75.59110070257611, | |
| 485 | + "pred_recall_at_k": null, | |
| 486 | + "tokens": 2135, | |
| 487 | + "families": { | |
| 488 | + "general": { | |
| 489 | + "hit_rate": 0.3551158874801902, | |
| 490 | + "stall_ms_per_token": 35.494421553090355 | |
| 491 | + }, | |
| 492 | + "code": { | |
| 493 | + "hit_rate": 0.4682576545778835, | |
| 494 | + "stall_ms_per_token": 29.2670986920333 | |
| 495 | + }, | |
| 496 | + "reasoning": { | |
| 497 | + "hit_rate": 0.3774561036789298, | |
| 498 | + "stall_ms_per_token": 34.26481605351171 | |
| 499 | + }, | |
| 500 | + "multilingual": { | |
| 501 | + "hit_rate": 0.3940161401098901, | |
| 502 | + "stall_ms_per_token": 33.353351648351634 | |
| 503 | + } | |
| 504 | + }, | |
| 505 | + "split": "fit" | |
| 506 | + }, | |
| 507 | + { | |
| 508 | + "capacity": 8, | |
| 509 | + "policy": "lfu", | |
| 510 | + "predictor": "prev_token", | |
| 511 | + "m": 8, | |
| 512 | + "hit_rate": 0.4148351054633472, | |
| 513 | + "misses_per_token": 74.90110650069157, | |
| 514 | + "stall_ms_per_token": 32.20747579529736, | |
| 515 | + "prefetch_loads_per_token": 0.0, | |
| 516 | + "prefetch_precision": null, | |
| 517 | + "loads_per_token": 74.90110650069157, | |
| 518 | + "pred_recall_at_k": null, | |
| 519 | + "tokens": 1446, | |
| 520 | + "families": { | |
| 521 | + "general": { | |
| 522 | + "hit_rate": 0.3655154520295203, | |
| 523 | + "stall_ms_per_token": 34.92202952029518 | |
| 524 | + }, | |
| 525 | + "code": { | |
| 526 | + "hit_rate": 0.4808007459312839, | |
| 527 | + "stall_ms_per_token": 28.57672694394212 | |
| 528 | + }, | |
| 529 | + "reasoning": { | |
| 530 | + "hit_rate": 0.38716947115384615, | |
| 531 | + "stall_ms_per_token": 33.730192307692306 | |
| 532 | + }, | |
| 533 | + "multilingual": { | |
| 534 | + "hit_rate": 0.38621794871794873, | |
| 535 | + "stall_ms_per_token": 33.78256410256409 | |
| 536 | + } | |
| 537 | + }, | |
| 538 | + "split": "heldout" | |
| 539 | + }, | |
| 540 | + { | |
| 541 | + "capacity": 8, | |
| 542 | + "policy": "lfu", | |
| 543 | + "predictor": "prev_rank", | |
| 544 | + "m": 8, | |
| 545 | + "hit_rate": 0.40934938524590164, | |
| 546 | + "misses_per_token": 75.6032786885246, | |
| 547 | + "stall_ms_per_token": 32.50940983606552, | |
| 548 | + "prefetch_loads_per_token": 0.04215456674473068, | |
| 549 | + "prefetch_precision": 0.13333333333333333, | |
| 550 | + "loads_per_token": 75.64543325526932, | |
| 551 | + "pred_recall_at_k": null, | |
| 552 | + "tokens": 2135, | |
| 553 | + "families": { | |
| 554 | + "general": { | |
| 555 | + "hit_rate": 0.3550292194928685, | |
| 556 | + "stall_ms_per_token": 35.49919175911255 | |
| 557 | + }, | |
| 558 | + "code": { | |
| 559 | + "hit_rate": 0.46812760107015455, | |
| 560 | + "stall_ms_per_token": 29.274256837098697 | |
| 561 | + }, | |
| 562 | + "reasoning": { | |
| 563 | + "hit_rate": 0.37742997491638797, | |
| 564 | + "stall_ms_per_token": 34.266254180602004 | |
| 565 | + }, | |
| 566 | + "multilingual": { | |
| 567 | + "hit_rate": 0.39393028846153844, | |
| 568 | + "stall_ms_per_token": 33.3580769230769 | |
| 569 | + } | |
| 570 | + }, | |
| 571 | + "split": "fit" | |
| 572 | + }, | |
| 573 | + { | |
| 574 | + "capacity": 8, | |
| 575 | + "policy": "lfu", | |
| 576 | + "predictor": "prev_rank", | |
| 577 | + "m": 8, | |
| 578 | + "hit_rate": 0.4148188969571231, | |
| 579 | + "misses_per_token": 74.90318118948824, | |
| 580 | + "stall_ms_per_token": 32.20836791147994, | |
| 581 | + "prefetch_loads_per_token": 0.04080221300138313, | |
| 582 | + "prefetch_precision": 0.3220338983050847, | |
| 583 | + "loads_per_token": 74.94398340248962, | |
| 584 | + "pred_recall_at_k": null, | |
| 585 | + "tokens": 1446, | |
| 586 | + "families": { | |
| 587 | + "general": { | |
| 588 | + "hit_rate": 0.365457795202952, | |
| 589 | + "stall_ms_per_token": 34.925202952029515 | |
| 590 | + }, | |
| 591 | + "code": { | |
| 592 | + "hit_rate": 0.48078661844484627, | |
| 593 | + "stall_ms_per_token": 28.57750452079565 | |
| 594 | + }, | |
| 595 | + "reasoning": { | |
| 596 | + "hit_rate": 0.38724459134615385, | |
| 597 | + "stall_ms_per_token": 33.72605769230769 | |
| 598 | + }, | |
| 599 | + "multilingual": { | |
| 600 | + "hit_rate": 0.3860176282051282, | |
| 601 | + "stall_ms_per_token": 33.79358974358974 | |
| 602 | + } | |
| 603 | + }, | |
| 604 | + "split": "heldout" | |
| 605 | + }, | |
| 606 | + { | |
| 607 | + "capacity": 8, | |
| 608 | + "policy": "lfu", | |
| 609 | + "predictor": "transition", | |
| 610 | + "m": 8, | |
| 611 | + "hit_rate": 0.4284726288056206, | |
| 612 | + "misses_per_token": 73.15550351288056, | |
| 613 | + "stall_ms_per_token": 22.81006182669791, | |
| 614 | + "prefetch_loads_per_token": 79.0384074941452, | |
| 615 | + "prefetch_precision": 0.301066093026839, | |
| 616 | + "loads_per_token": 152.19391100702575, | |
| 617 | + "pred_recall_at_k": 0.5302576112412178, | |
| 618 | + "tokens": 2135, | |
| 619 | + "families": { | |
| 620 | + "general": { | |
| 621 | + "hit_rate": 0.36909419572107766, | |
| 622 | + "stall_ms_per_token": 25.117752773375592 | |
| 623 | + }, | |
| 624 | + "code": { | |
| 625 | + "hit_rate": 0.482461355529132, | |
| 626 | + "stall_ms_per_token": 20.738359096313896 | |
| 627 | + }, | |
| 628 | + "reasoning": { | |
| 629 | + "hit_rate": 0.4197063127090301, | |
| 630 | + "stall_ms_per_token": 23.134762541806037 | |
| 631 | + }, | |
| 632 | + "multilingual": { | |
| 633 | + "hit_rate": 0.41386933379120877, | |
| 634 | + "stall_ms_per_token": 23.32946703296704 | |
| 635 | + } | |
| 636 | + }, | |
| 637 | + "split": "fit" | |
| 638 | + }, | |
| 639 | + { | |
| 640 | + "capacity": 8, | |
| 641 | + "policy": "lfu", | |
| 642 | + "predictor": "transition", | |
| 643 | + "m": 8, | |
| 644 | + "hit_rate": 0.4097942600276625, | |
| 645 | + "misses_per_token": 75.5463347164592, | |
| 646 | + "stall_ms_per_token": 23.52731396957124, | |
| 647 | + "prefetch_loads_per_token": 81.87482710926695, | |
| 648 | + "prefetch_precision": 0.2803338091577907, | |
| 649 | + "loads_per_token": 157.42116182572613, | |
| 650 | + "pred_recall_at_k": 0.4996369294605809, | |
| 651 | + "tokens": 1446, | |
| 652 | + "families": { | |
| 653 | + "general": { | |
| 654 | + "hit_rate": 0.33217539206642066, | |
| 655 | + "stall_ms_per_token": 26.553435424354237 | |
| 656 | + }, | |
| 657 | + "code": { | |
| 658 | + "hit_rate": 0.4794021247739602, | |
| 659 | + "stall_ms_per_token": 20.82163471971067 | |
| 660 | + }, | |
| 661 | + "reasoning": { | |
| 662 | + "hit_rate": 0.42518028846153844, | |
| 663 | + "stall_ms_per_token": 22.9221794871795 | |
| 664 | + }, | |
| 665 | + "multilingual": { | |
| 666 | + "hit_rate": 0.37840544871794873, | |
| 667 | + "stall_ms_per_token": 24.67820512820513 | |
| 668 | + } | |
| 669 | + }, | |
| 670 | + "split": "heldout" | |
| 671 | + }, | |
| 672 | + { | |
| 673 | + "capacity": 8, | |
| 674 | + "policy": "lfu", | |
| 675 | + "predictor": "probe", | |
| 676 | + "m": 8, | |
| 677 | + "hit_rate": 0.22535860655737705, | |
| 678 | + "misses_per_token": 99.15409836065574, | |
| 679 | + "stall_ms_per_token": 30.66163747072595, | |
| 680 | + "prefetch_loads_per_token": 108.71896955503513, | |
| 681 | + "prefetch_precision": 0.11085022510393555, | |
| 682 | + "loads_per_token": 207.87306791569085, | |
| 683 | + "pred_recall_at_k": 0.16113973458235753, | |
| 684 | + "tokens": 2135, | |
| 685 | + "families": { | |
| 686 | + "general": { | |
| 687 | + "hit_rate": 0.22640154516640254, | |
| 688 | + "stall_ms_per_token": 30.633679873217115 | |
| 689 | + }, | |
| 690 | + "code": { | |
| 691 | + "hit_rate": 0.2077883472057075, | |
| 692 | + "stall_ms_per_token": 31.3561189060642 | |
| 693 | + }, | |
| 694 | + "reasoning": { | |
| 695 | + "hit_rate": 0.244748118729097, | |
| 696 | + "stall_ms_per_token": 29.89794648829429 | |
| 697 | + }, | |
| 698 | + "multilingual": { | |
| 699 | + "hit_rate": 0.2482185782967033, | |
| 700 | + "stall_ms_per_token": 29.73286263736262 | |
| 701 | + } | |
| 702 | + }, | |
| 703 | + "split": "fit" | |
| 704 | + }, | |
| 705 | + { | |
| 706 | + "capacity": 8, | |
| 707 | + "policy": "lfu", | |
| 708 | + "predictor": "probe", | |
| 709 | + "m": 8, | |
| 710 | + "hit_rate": 0.21783151798063624, | |
| 711 | + "misses_per_token": 100.11756569847856, | |
| 712 | + "stall_ms_per_token": 30.94782572614104, | |
| 713 | + "prefetch_loads_per_token": 109.23997233748271, | |
| 714 | + "prefetch_precision": 0.1055766929811789, | |
| 715 | + "loads_per_token": 209.35753803596128, | |
| 716 | + "pred_recall_at_k": 0.15494467496542186, | |
| 717 | + "tokens": 1446, | |
| 718 | + "families": { | |
| 719 | + "general": { | |
| 720 | + "hit_rate": 0.18604416512915128, | |
| 721 | + "stall_ms_per_token": 32.20228413284127 | |
| 722 | + }, | |
| 723 | + "code": { | |
| 724 | + "hit_rate": 0.23311765370705245, | |
| 725 | + "stall_ms_per_token": 30.34200723327306 | |
| 726 | + }, | |
| 727 | + "reasoning": { | |
| 728 | + "hit_rate": 0.24143629807692307, | |
| 729 | + "stall_ms_per_token": 30.02498717948718 | |
| 730 | + }, | |
| 731 | + "multilingual": { | |
| 732 | + "hit_rate": 0.25400641025641024, | |
| 733 | + "stall_ms_per_token": 29.486974358974365 | |
| 734 | + } | |
| 735 | + }, | |
| 736 | + "split": "heldout" | |
| 737 | + }, | |
| 738 | + { | |
| 739 | + "capacity": 8, | |
| 740 | + "policy": "lfu", | |
| 741 | + "predictor": "oracle", | |
| 742 | + "m": 8, | |
| 743 | + "hit_rate": 0.6142308255269321, | |
| 744 | + "misses_per_token": 49.37845433255269, | |
| 745 | + "stall_ms_per_token": 15.629392974238886, | |
| 746 | + "prefetch_loads_per_token": 76.57751756440281, | |
| 747 | + "prefetch_precision": 0.542426892894497, | |
| 748 | + "loads_per_token": 125.9559718969555, | |
| 749 | + "pred_recall_at_k": 1.0, | |
| 750 | + "tokens": 2135, | |
| 751 | + "families": { | |
| 752 | + "general": { | |
| 753 | + "hit_rate": 0.5897013668779715, | |
| 754 | + "stall_ms_per_token": 16.589961965134687 | |
| 755 | + }, | |
| 756 | + "code": { | |
| 757 | + "hit_rate": 0.63701137039239, | |
| 758 | + "stall_ms_per_token": 14.764073721759823 | |
| 759 | + }, | |
| 760 | + "reasoning": { | |
| 761 | + "hit_rate": 0.6069188963210702, | |
| 762 | + "stall_ms_per_token": 15.897872909699002 | |
| 763 | + }, | |
| 764 | + "multilingual": { | |
| 765 | + "hit_rate": 0.6101262019230769, | |
| 766 | + "stall_ms_per_token": 15.742961538461541 | |
| 767 | + } | |
| 768 | + }, | |
| 769 | + "split": "fit" | |
| 770 | + }, | |
| 771 | + { | |
| 772 | + "capacity": 8, | |
| 773 | + "policy": "lfu", | |
| 774 | + "predictor": "oracle", | |
| 775 | + "m": 8, | |
| 776 | + "hit_rate": 0.6197268326417704, | |
| 777 | + "misses_per_token": 48.674965421853386, | |
| 778 | + "stall_ms_per_token": 15.412160442600275, | |
| 779 | + "prefetch_loads_per_token": 76.04011065006915, | |
| 780 | + "prefetch_precision": 0.538843516379577, | |
| 781 | + "loads_per_token": 124.71507607192254, | |
| 782 | + "pred_recall_at_k": 1.0, | |
| 783 | + "tokens": 1446, | |
| 784 | + "families": { | |
| 785 | + "general": { | |
| 786 | + "hit_rate": 0.5879987315498155, | |
| 787 | + "stall_ms_per_token": 16.664328413284146 | |
| 788 | + }, | |
| 789 | + "code": { | |
| 790 | + "hit_rate": 0.6537211799276673, | |
| 791 | + "stall_ms_per_token": 14.083157323688978 | |
| 792 | + }, | |
| 793 | + "reasoning": { | |
| 794 | + "hit_rate": 0.6153846153846154, | |
| 795 | + "stall_ms_per_token": 15.56964102564103 | |
| 796 | + }, | |
| 797 | + "multilingual": { | |
| 798 | + "hit_rate": 0.6133814102564102, | |
| 799 | + "stall_ms_per_token": 15.594974358974358 | |
| 800 | + } | |
| 801 | + }, | |
| 802 | + "split": "heldout" | |
| 803 | + }, | |
| 804 | + { | |
| 805 | + "capacity": 12, | |
| 806 | + "policy": "lru", | |
| 807 | + "predictor": "none", | |
| 808 | + "m": 8, | |
| 809 | + "hit_rate": 0.5040434718969555, | |
| 810 | + "misses_per_token": 63.48243559718969, | |
| 811 | + "stall_ms_per_token": 27.297447306791522, | |
| 812 | + "prefetch_loads_per_token": 0.0, | |
| 813 | + "prefetch_precision": null, | |
| 814 | + "loads_per_token": 63.48243559718969, | |
| 815 | + "pred_recall_at_k": null, | |
| 816 | + "tokens": 2135, | |
| 817 | + "families": { | |
| 818 | + "general": { | |
| 819 | + "hit_rate": 0.4448915412044374, | |
| 820 | + "stall_ms_per_token": 30.55316957210777 | |
| 821 | + }, | |
| 822 | + "code": { | |
| 823 | + "hit_rate": 0.5714179548156956, | |
| 824 | + "stall_ms_per_token": 23.589155766944128 | |
| 825 | + }, | |
| 826 | + "reasoning": { | |
| 827 | + "hit_rate": 0.47047449832775917, | |
| 828 | + "stall_ms_per_token": 29.145083612040132 | |
| 829 | + }, | |
| 830 | + "multilingual": { | |
| 831 | + "hit_rate": 0.47849416208791207, | |
| 832 | + "stall_ms_per_token": 28.703681318681323 | |
| 833 | + } | |
| 834 | + }, | |
| 835 | + "split": "fit" | |
| 836 | + }, | |
| 837 | + { | |
| 838 | + "capacity": 12, | |
| 839 | + "policy": "lru", | |
| 840 | + "predictor": "none", | |
| 841 | + "m": 8, | |
| 842 | + "hit_rate": 0.49955696749654216, | |
| 843 | + "misses_per_token": 64.0567081604426, | |
| 844 | + "stall_ms_per_token": 27.544384508990273, | |
| 845 | + "prefetch_loads_per_token": 0.0, | |
| 846 | + "prefetch_precision": null, | |
| 847 | + "loads_per_token": 64.0567081604426, | |
| 848 | + "pred_recall_at_k": null, | |
| 849 | + "tokens": 1446, | |
| 850 | + "families": { | |
| 851 | + "general": { | |
| 852 | + "hit_rate": 0.4276118542435424, | |
| 853 | + "stall_ms_per_token": 31.504243542435404 | |
| 854 | + }, | |
| 855 | + "code": { | |
| 856 | + "hit_rate": 0.5744659810126582, | |
| 857 | + "stall_ms_per_token": 23.42139240506328 | |
| 858 | + }, | |
| 859 | + "reasoning": { | |
| 860 | + "hit_rate": 0.4960436698717949, | |
| 861 | + "stall_ms_per_token": 27.7377564102564 | |
| 862 | + }, | |
| 863 | + "multilingual": { | |
| 864 | + "hit_rate": 0.46534455128205127, | |
| 865 | + "stall_ms_per_token": 29.42743589743589 | |
| 866 | + } | |
| 867 | + }, | |
| 868 | + "split": "heldout" | |
| 869 | + }, | |
| 870 | + { | |
| 871 | + "capacity": 12, | |
| 872 | + "policy": "lru", | |
| 873 | + "predictor": "prev_token", | |
| 874 | + "m": 8, | |
| 875 | + "hit_rate": 0.5040434718969555, | |
| 876 | + "misses_per_token": 63.48243559718969, | |
| 877 | + "stall_ms_per_token": 27.297447306791522, | |
| 878 | + "prefetch_loads_per_token": 0.0, | |
| 879 | + "prefetch_precision": null, | |
| 880 | + "loads_per_token": 63.48243559718969, | |
| 881 | + "pred_recall_at_k": null, | |
| 882 | + "tokens": 2135, | |
| 883 | + "families": { | |
| 884 | + "general": { | |
| 885 | + "hit_rate": 0.4448915412044374, | |
| 886 | + "stall_ms_per_token": 30.55316957210777 | |
| 887 | + }, | |
| 888 | + "code": { | |
| 889 | + "hit_rate": 0.5714179548156956, | |
| 890 | + "stall_ms_per_token": 23.589155766944128 | |
| 891 | + }, | |
| 892 | + "reasoning": { | |
| 893 | + "hit_rate": 0.47047449832775917, | |
| 894 | + "stall_ms_per_token": 29.145083612040132 | |
| 895 | + }, | |
| 896 | + "multilingual": { | |
| 897 | + "hit_rate": 0.47849416208791207, | |
| 898 | + "stall_ms_per_token": 28.703681318681323 | |
| 899 | + } | |
| 900 | + }, | |
| 901 | + "split": "fit" | |
| 902 | + }, | |
| 903 | + { | |
| 904 | + "capacity": 12, | |
| 905 | + "policy": "lru", | |
| 906 | + "predictor": "prev_token", | |
| 907 | + "m": 8, | |
| 908 | + "hit_rate": 0.49955696749654216, | |
| 909 | + "misses_per_token": 64.0567081604426, | |
| 910 | + "stall_ms_per_token": 27.544384508990273, | |
| 911 | + "prefetch_loads_per_token": 0.0, | |
| 912 | + "prefetch_precision": null, | |
| 913 | + "loads_per_token": 64.0567081604426, | |
| 914 | + "pred_recall_at_k": null, | |
| 915 | + "tokens": 1446, | |
| 916 | + "families": { | |
| 917 | + "general": { | |
| 918 | + "hit_rate": 0.4276118542435424, | |
| 919 | + "stall_ms_per_token": 31.504243542435404 | |
| 920 | + }, | |
| 921 | + "code": { | |
| 922 | + "hit_rate": 0.5744659810126582, | |
| 923 | + "stall_ms_per_token": 23.42139240506328 | |
| 924 | + }, | |
| 925 | + "reasoning": { | |
| 926 | + "hit_rate": 0.4960436698717949, | |
| 927 | + "stall_ms_per_token": 27.7377564102564 | |
| 928 | + }, | |
| 929 | + "multilingual": { | |
| 930 | + "hit_rate": 0.46534455128205127, | |
| 931 | + "stall_ms_per_token": 29.42743589743589 | |
| 932 | + } | |
| 933 | + }, | |
| 934 | + "split": "heldout" | |
| 935 | + }, | |
| 936 | + { | |
| 937 | + "capacity": 12, | |
| 938 | + "policy": "lru", | |
| 939 | + "predictor": "prev_rank", | |
| 940 | + "m": 8, | |
| 941 | + "hit_rate": 0.5040178571428572, | |
| 942 | + "misses_per_token": 63.48571428571429, | |
| 943 | + "stall_ms_per_token": 27.29885714285711, | |
| 944 | + "prefetch_loads_per_token": 0.03278688524590164, | |
| 945 | + "prefetch_precision": 0.2857142857142857, | |
| 946 | + "loads_per_token": 63.518501170960185, | |
| 947 | + "pred_recall_at_k": null, | |
| 948 | + "tokens": 2135, | |
| 949 | + "families": { | |
| 950 | + "general": { | |
| 951 | + "hit_rate": 0.4448420166402536, | |
| 952 | + "stall_ms_per_token": 30.555895404120445 | |
| 953 | + }, | |
| 954 | + "code": { | |
| 955 | + "hit_rate": 0.5714086652794292, | |
| 956 | + "stall_ms_per_token": 23.58966706302023 | |
| 957 | + }, | |
| 958 | + "reasoning": { | |
| 959 | + "hit_rate": 0.4705267558528428, | |
| 960 | + "stall_ms_per_token": 29.142207357859526 | |
| 961 | + }, | |
| 962 | + "multilingual": { | |
| 963 | + "hit_rate": 0.47840831043956045, | |
| 964 | + "stall_ms_per_token": 28.70840659340659 | |
| 965 | + } | |
| 966 | + }, | |
| 967 | + "split": "fit" | |
| 968 | + }, | |
| 969 | + { | |
| 970 | + "capacity": 12, | |
| 971 | + "policy": "lru", | |
| 972 | + "predictor": "prev_rank", | |
| 973 | + "m": 8, | |
| 974 | + "hit_rate": 0.4995839816735823, | |
| 975 | + "misses_per_token": 64.05325034578146, | |
| 976 | + "stall_ms_per_token": 27.54289764868599, | |
| 977 | + "prefetch_loads_per_token": 0.026970954356846474, | |
| 978 | + "prefetch_precision": 0.38461538461538464, | |
| 979 | + "loads_per_token": 64.08022130013832, | |
| 980 | + "pred_recall_at_k": null, | |
| 981 | + "tokens": 1446, | |
| 982 | + "families": { | |
| 983 | + "general": { | |
| 984 | + "hit_rate": 0.42765509686346864, | |
| 985 | + "stall_ms_per_token": 31.50186346863467 | |
| 986 | + }, | |
| 987 | + "code": { | |
| 988 | + "hit_rate": 0.5744801084990958, | |
| 989 | + "stall_ms_per_token": 23.420614828209757 | |
| 990 | + }, | |
| 991 | + "reasoning": { | |
| 992 | + "hit_rate": 0.4960436698717949, | |
| 993 | + "stall_ms_per_token": 27.7377564102564 | |
| 994 | + }, | |
| 995 | + "multilingual": { | |
| 996 | + "hit_rate": 0.4655448717948718, | |
| 997 | + "stall_ms_per_token": 29.416410256410245 | |
| 998 | + } | |
| 999 | + }, | |
| 1000 | + "split": "heldout" | |
| 1001 | + }, | |
| 1002 | + { | |
| 1003 | + "capacity": 12, | |
| 1004 | + "policy": "lru", | |
| 1005 | + "predictor": "transition", | |
| 1006 | + "m": 8, | |
| 1007 | + "hit_rate": 0.5989461358313818, | |
| 1008 | + "misses_per_token": 51.33489461358314, | |
| 1009 | + "stall_ms_per_token": 16.138042154566726, | |
| 1010 | + "prefetch_loads_per_token": 57.231381733021074, | |
| 1011 | + "prefetch_precision": 0.45320773555721056, | |
| 1012 | + "loads_per_token": 108.56627634660421, | |
| 1013 | + "pred_recall_at_k": 0.5302576112412178, | |
| 1014 | + "tokens": 2135, | |
| 1015 | + "families": { | |
| 1016 | + "general": { | |
| 1017 | + "hit_rate": 0.5529169968304279, | |
| 1018 | + "stall_ms_per_token": 17.930554675118852 | |
| 1019 | + }, | |
| 1020 | + "code": { | |
| 1021 | + "hit_rate": 0.652078998216409, | |
| 1022 | + "stall_ms_per_token": 14.098670630202145 | |
| 1023 | + }, | |
| 1024 | + "reasoning": { | |
| 1025 | + "hit_rate": 0.5733434364548495, | |
| 1026 | + "stall_ms_per_token": 17.100301003344484 | |
| 1027 | + }, | |
| 1028 | + "multilingual": { | |
| 1029 | + "hit_rate": 0.5770089285714286, | |
| 1030 | + "stall_ms_per_token": 16.952109890109877 | |
| 1031 | + } | |
| 1032 | + }, | |
| 1033 | + "split": "fit" | |
| 1034 | + }, | |
| 1035 | + { | |
| 1036 | + "capacity": 12, | |
| 1037 | + "policy": "lru", | |
| 1038 | + "predictor": "transition", | |
| 1039 | + "m": 8, | |
| 1040 | + "hit_rate": 0.5868289678423236, | |
| 1041 | + "misses_per_token": 52.885892116182575, | |
| 1042 | + "stall_ms_per_token": 16.602687413554605, | |
| 1043 | + "prefetch_loads_per_token": 60.677731673582294, | |
| 1044 | + "prefetch_precision": 0.4113175290631411, | |
| 1045 | + "loads_per_token": 113.56362378976488, | |
| 1046 | + "pred_recall_at_k": 0.4996369294605809, | |
| 1047 | + "tokens": 1446, | |
| 1048 | + "families": { | |
| 1049 | + "general": { | |
| 1050 | + "hit_rate": 0.5323598939114391, | |
| 1051 | + "stall_ms_per_token": 18.755118081180793 | |
| 1052 | + }, | |
| 1053 | + "code": { | |
| 1054 | + "hit_rate": 0.6459651898734177, | |
| 1055 | + "stall_ms_per_token": 14.299645569620262 | |
| 1056 | + }, | |
| 1057 | + "reasoning": { | |
| 1058 | + "hit_rate": 0.5811298076923077, | |
| 1059 | + "stall_ms_per_token": 16.779743589743596 | |
| 1060 | + }, | |
| 1061 | + "multilingual": { | |
| 1062 | + "hit_rate": 0.5508814102564102, | |
| 1063 | + "stall_ms_per_token": 17.928923076923077 | |
| 1064 | + } | |
| 1065 | + }, | |
| 1066 | + "split": "heldout" | |
| 1067 | + }, | |
| 1068 | + { | |
| 1069 | + "capacity": 12, | |
| 1070 | + "policy": "lru", | |
| 1071 | + "predictor": "probe", | |
| 1072 | + "m": 8, | |
| 1073 | + "hit_rate": 0.35070623536299766, | |
| 1074 | + "misses_per_token": 83.1096018735363, | |
| 1075 | + "stall_ms_per_token": 25.73400374707258, | |
| 1076 | + "prefetch_loads_per_token": 96.98548009367681, | |
| 1077 | + "prefetch_precision": 0.1183209056137233, | |
| 1078 | + "loads_per_token": 180.0950819672131, | |
| 1079 | + "pred_recall_at_k": 0.16113973458235753, | |
| 1080 | + "tokens": 2135, | |
| 1081 | + "families": { | |
| 1082 | + "general": { | |
| 1083 | + "hit_rate": 0.35776545166402535, | |
| 1084 | + "stall_ms_per_token": 25.474332805071313 | |
| 1085 | + }, | |
| 1086 | + "code": { | |
| 1087 | + "hit_rate": 0.3418735136741974, | |
| 1088 | + "stall_ms_per_token": 26.089973840665888 | |
| 1089 | + }, | |
| 1090 | + "reasoning": { | |
| 1091 | + "hit_rate": 0.34871446488294316, | |
| 1092 | + "stall_ms_per_token": 25.78355852842809 | |
| 1093 | + }, | |
| 1094 | + "multilingual": { | |
| 1095 | + "hit_rate": 0.36051253434065933, | |
| 1096 | + "stall_ms_per_token": 25.320994505494507 | |
| 1097 | + } | |
| 1098 | + }, | |
| 1099 | + "split": "fit" | |
| 1100 | + }, | |
| 1101 | + { | |
| 1102 | + "capacity": 12, | |
| 1103 | + "policy": "lru", | |
| 1104 | + "predictor": "probe", | |
| 1105 | + "m": 8, | |
| 1106 | + "hit_rate": 0.35182183609958506, | |
| 1107 | + "misses_per_token": 82.96680497925311, | |
| 1108 | + "stall_ms_per_token": 25.68712309820195, | |
| 1109 | + "prefetch_loads_per_token": 97.50414937759336, | |
| 1110 | + "prefetch_precision": 0.113078139739416, | |
| 1111 | + "loads_per_token": 180.47095435684648, | |
| 1112 | + "pred_recall_at_k": 0.15494467496542186, | |
| 1113 | + "tokens": 1446, | |
| 1114 | + "families": { | |
| 1115 | + "general": { | |
| 1116 | + "hit_rate": 0.33451049354243545, | |
| 1117 | + "stall_ms_per_token": 26.403184501845004 | |
| 1118 | + }, | |
| 1119 | + "code": { | |
| 1120 | + "hit_rate": 0.3661985759493671, | |
| 1121 | + "stall_ms_per_token": 25.114303797468363 | |
| 1122 | + }, | |
| 1123 | + "reasoning": { | |
| 1124 | + "hit_rate": 0.3515875400641026, | |
| 1125 | + "stall_ms_per_token": 25.652929487179488 | |
| 1126 | + }, | |
| 1127 | + "multilingual": { | |
| 1128 | + "hit_rate": 0.39042467948717946, | |
| 1129 | + "stall_ms_per_token": 24.13153846153847 | |
| 1130 | + } | |
| 1131 | + }, | |
| 1132 | + "split": "heldout" | |
| 1133 | + }, | |
| 1134 | + { | |
| 1135 | + "capacity": 12, | |
| 1136 | + "policy": "lru", | |
| 1137 | + "predictor": "oracle", | |
| 1138 | + "m": 8, | |
| 1139 | + "hit_rate": 0.9075709894613583, | |
| 1140 | + "misses_per_token": 11.830913348946137, | |
| 1141 | + "stall_ms_per_token": 4.20783981264638, | |
| 1142 | + "prefetch_loads_per_token": 63.51662763466042, | |
| 1143 | + "prefetch_precision": 1.0, | |
| 1144 | + "loads_per_token": 75.34754098360656, | |
| 1145 | + "pred_recall_at_k": 1.0, | |
| 1146 | + "tokens": 2135, | |
| 1147 | + "families": { | |
| 1148 | + "general": { | |
| 1149 | + "hit_rate": 0.9118958003169572, | |
| 1150 | + "stall_ms_per_token": 4.053870047543583 | |
| 1151 | + }, | |
| 1152 | + "code": { | |
| 1153 | + "hit_rate": 0.9049773335315101, | |
| 1154 | + "stall_ms_per_token": 4.32263258026159 | |
| 1155 | + }, | |
| 1156 | + "reasoning": { | |
| 1157 | + "hit_rate": 0.9072167642140468, | |
| 1158 | + "stall_ms_per_token": 4.194093645484952 | |
| 1159 | + }, | |
| 1160 | + "multilingual": { | |
| 1161 | + "hit_rate": 0.9063573145604396, | |
| 1162 | + "stall_ms_per_token": 4.220818681318682 | |
| 1163 | + } | |
| 1164 | + }, | |
| 1165 | + "split": "fit" | |
| 1166 | + }, | |
| 1167 | + { | |
| 1168 | + "capacity": 12, | |
| 1169 | + "policy": "lru", | |
| 1170 | + "predictor": "oracle", | |
| 1171 | + "m": 8, | |
| 1172 | + "hit_rate": 0.9126955826417704, | |
| 1173 | + "misses_per_token": 11.174965421853388, | |
| 1174 | + "stall_ms_per_token": 4.0059875518672134, | |
| 1175 | + "prefetch_loads_per_token": 63.494467496542185, | |
| 1176 | + "prefetch_precision": 1.0, | |
| 1177 | + "loads_per_token": 74.66943291839557, | |
| 1178 | + "pred_recall_at_k": 1.0, | |
| 1179 | + "tokens": 1446, | |
| 1180 | + "families": { | |
| 1181 | + "general": { | |
| 1182 | + "hit_rate": 0.9178101937269373, | |
| 1183 | + "stall_ms_per_token": 3.8551512915129145 | |
| 1184 | + }, | |
| 1185 | + "code": { | |
| 1186 | + "hit_rate": 0.9123672016274864, | |
| 1187 | + "stall_ms_per_token": 4.001609403254976 | |
| 1188 | + }, | |
| 1189 | + "reasoning": { | |
| 1190 | + "hit_rate": 0.904472155448718, | |
| 1191 | + "stall_ms_per_token": 4.280621794871795 | |
| 1192 | + }, | |
| 1193 | + "multilingual": { | |
| 1194 | + "hit_rate": 0.9120592948717948, | |
| 1195 | + "stall_ms_per_token": 3.9672307692307696 | |
| 1196 | + } | |
| 1197 | + }, | |
| 1198 | + "split": "heldout" | |
| 1199 | + }, | |
| 1200 | + { | |
| 1201 | + "capacity": 12, | |
| 1202 | + "policy": "lfu", | |
| 1203 | + "predictor": "none", | |
| 1204 | + "m": 8, | |
| 1205 | + "hit_rate": 0.5191525175644028, | |
| 1206 | + "misses_per_token": 61.54847775175644, | |
| 1207 | + "stall_ms_per_token": 26.46584543325528, | |
| 1208 | + "prefetch_loads_per_token": 0.0, | |
| 1209 | + "prefetch_precision": null, | |
| 1210 | + "loads_per_token": 61.54847775175644, | |
| 1211 | + "pred_recall_at_k": null, | |
| 1212 | + "tokens": 2135, | |
| 1213 | + "families": { | |
| 1214 | + "general": { | |
| 1215 | + "hit_rate": 0.4492497028526149, | |
| 1216 | + "stall_ms_per_token": 30.313296354992115 | |
| 1217 | + }, | |
| 1218 | + "code": { | |
| 1219 | + "hit_rate": 0.5909910077288941, | |
| 1220 | + "stall_ms_per_token": 22.511854934601704 | |
| 1221 | + }, | |
| 1222 | + "reasoning": { | |
| 1223 | + "hit_rate": 0.47752926421404684, | |
| 1224 | + "stall_ms_per_token": 28.756789297658827 | |
| 1225 | + }, | |
| 1226 | + "multilingual": { | |
| 1227 | + "hit_rate": 0.508542239010989, | |
| 1228 | + "stall_ms_per_token": 27.049835164835173 | |
| 1229 | + } | |
| 1230 | + }, | |
| 1231 | + "split": "fit" | |
| 1232 | + }, | |
| 1233 | + { | |
| 1234 | + "capacity": 12, | |
| 1235 | + "policy": "lfu", | |
| 1236 | + "predictor": "none", | |
| 1237 | + "m": 8, | |
| 1238 | + "hit_rate": 0.514414764868603, | |
| 1239 | + "misses_per_token": 62.15491009681881, | |
| 1240 | + "stall_ms_per_token": 26.726611341632154, | |
| 1241 | + "prefetch_loads_per_token": 0.0, | |
| 1242 | + "prefetch_precision": null, | |
| 1243 | + "loads_per_token": 62.15491009681881, | |
| 1244 | + "pred_recall_at_k": null, | |
| 1245 | + "tokens": 1446, | |
| 1246 | + "families": { | |
| 1247 | + "general": { | |
| 1248 | + "hit_rate": 0.4477340867158672, | |
| 1249 | + "stall_ms_per_token": 30.396715867158672 | |
| 1250 | + }, | |
| 1251 | + "code": { | |
| 1252 | + "hit_rate": 0.5885228300180831, | |
| 1253 | + "stall_ms_per_token": 22.647703435804694 | |
| 1254 | + }, | |
| 1255 | + "reasoning": { | |
| 1256 | + "hit_rate": 0.5044320913461539, | |
| 1257 | + "stall_ms_per_token": 27.276057692307702 | |
| 1258 | + }, | |
| 1259 | + "multilingual": { | |
| 1260 | + "hit_rate": 0.4701522435897436, | |
| 1261 | + "stall_ms_per_token": 29.162820512820506 | |
| 1262 | + } | |
| 1263 | + }, | |
| 1264 | + "split": "heldout" | |
| 1265 | + }, | |
| 1266 | + { | |
| 1267 | + "capacity": 12, | |
| 1268 | + "policy": "lfu", | |
| 1269 | + "predictor": "prev_token", | |
| 1270 | + "m": 8, | |
| 1271 | + "hit_rate": 0.5191525175644028, | |
| 1272 | + "misses_per_token": 61.54847775175644, | |
| 1273 | + "stall_ms_per_token": 26.46584543325528, | |
| 1274 | + "prefetch_loads_per_token": 0.0, | |
| 1275 | + "prefetch_precision": null, | |
| 1276 | + "loads_per_token": 61.54847775175644, | |
| 1277 | + "pred_recall_at_k": null, | |
| 1278 | + "tokens": 2135, | |
| 1279 | + "families": { | |
| 1280 | + "general": { | |
| 1281 | + "hit_rate": 0.4492497028526149, | |
| 1282 | + "stall_ms_per_token": 30.313296354992115 | |
| 1283 | + }, | |
| 1284 | + "code": { | |
| 1285 | + "hit_rate": 0.5909910077288941, | |
| 1286 | + "stall_ms_per_token": 22.511854934601704 | |
| 1287 | + }, | |
| 1288 | + "reasoning": { | |
| 1289 | + "hit_rate": 0.47752926421404684, | |
| 1290 | + "stall_ms_per_token": 28.756789297658827 | |
| 1291 | + }, | |
| 1292 | + "multilingual": { | |
| 1293 | + "hit_rate": 0.508542239010989, | |
| 1294 | + "stall_ms_per_token": 27.049835164835173 | |
| 1295 | + } | |
| 1296 | + }, | |
| 1297 | + "split": "fit" | |
| 1298 | + }, | |
| 1299 | + { | |
| 1300 | + "capacity": 12, | |
| 1301 | + "policy": "lfu", | |
| 1302 | + "predictor": "prev_token", | |
| 1303 | + "m": 8, | |
| 1304 | + "hit_rate": 0.514414764868603, | |
| 1305 | + "misses_per_token": 62.15491009681881, | |
| 1306 | + "stall_ms_per_token": 26.726611341632154, | |
| 1307 | + "prefetch_loads_per_token": 0.0, | |
| 1308 | + "prefetch_precision": null, | |
| 1309 | + "loads_per_token": 62.15491009681881, | |
| 1310 | + "pred_recall_at_k": null, | |
| 1311 | + "tokens": 1446, | |
| 1312 | + "families": { | |
| 1313 | + "general": { | |
| 1314 | + "hit_rate": 0.4477340867158672, | |
| 1315 | + "stall_ms_per_token": 30.396715867158672 | |
| 1316 | + }, | |
| 1317 | + "code": { | |
| 1318 | + "hit_rate": 0.5885228300180831, | |
| 1319 | + "stall_ms_per_token": 22.647703435804694 | |
| 1320 | + }, | |
| 1321 | + "reasoning": { | |
| 1322 | + "hit_rate": 0.5044320913461539, | |
| 1323 | + "stall_ms_per_token": 27.276057692307702 | |
| 1324 | + }, | |
| 1325 | + "multilingual": { | |
| 1326 | + "hit_rate": 0.4701522435897436, | |
| 1327 | + "stall_ms_per_token": 29.162820512820506 | |
| 1328 | + } | |
| 1329 | + }, | |
| 1330 | + "split": "heldout" | |
| 1331 | + }, | |
| 1332 | + { | |
| 1333 | + "capacity": 12, | |
| 1334 | + "policy": "lfu", | |
| 1335 | + "predictor": "prev_rank", | |
| 1336 | + "m": 8, | |
| 1337 | + "hit_rate": 0.5190903103044496, | |
| 1338 | + "misses_per_token": 61.55644028103045, | |
| 1339 | + "stall_ms_per_token": 26.469269320843097, | |
| 1340 | + "prefetch_loads_per_token": 0.03372365339578454, | |
| 1341 | + "prefetch_precision": 0.1527777777777778, | |
| 1342 | + "loads_per_token": 61.59016393442623, | |
| 1343 | + "pred_recall_at_k": null, | |
| 1344 | + "tokens": 2135, | |
| 1345 | + "families": { | |
| 1346 | + "general": { | |
| 1347 | + "hit_rate": 0.4491877971473851, | |
| 1348 | + "stall_ms_per_token": 30.31670364500796 | |
| 1349 | + }, | |
| 1350 | + "code": { | |
| 1351 | + "hit_rate": 0.5909259809750297, | |
| 1352 | + "stall_ms_per_token": 22.515434007134402 | |
| 1353 | + }, | |
| 1354 | + "reasoning": { | |
| 1355 | + "hit_rate": 0.47752926421404684, | |
| 1356 | + "stall_ms_per_token": 28.756789297658834 | |
| 1357 | + }, | |
| 1358 | + "multilingual": { | |
| 1359 | + "hit_rate": 0.5084349244505495, | |
| 1360 | + "stall_ms_per_token": 27.05574175824177 | |
| 1361 | + } | |
| 1362 | + }, | |
| 1363 | + "split": "fit" | |
| 1364 | + }, | |
| 1365 | + { | |
| 1366 | + "capacity": 12, | |
| 1367 | + "policy": "lfu", | |
| 1368 | + "predictor": "prev_rank", | |
| 1369 | + "m": 8, | |
| 1370 | + "hit_rate": 0.5143931535269709, | |
| 1371 | + "misses_per_token": 62.15767634854772, | |
| 1372 | + "stall_ms_per_token": 26.727800829875584, | |
| 1373 | + "prefetch_loads_per_token": 0.03319502074688797, | |
| 1374 | + "prefetch_precision": 0.3333333333333333, | |
| 1375 | + "loads_per_token": 62.190871369294605, | |
| 1376 | + "pred_recall_at_k": null, | |
| 1377 | + "tokens": 1446, | |
| 1378 | + "families": { | |
| 1379 | + "general": { | |
| 1380 | + "hit_rate": 0.44769084409594095, | |
| 1381 | + "stall_ms_per_token": 30.399095940959413 | |
| 1382 | + }, | |
| 1383 | + "code": { | |
| 1384 | + "hit_rate": 0.5885087025316456, | |
| 1385 | + "stall_ms_per_token": 22.648481012658216 | |
| 1386 | + }, | |
| 1387 | + "reasoning": { | |
| 1388 | + "hit_rate": 0.5044320913461539, | |
| 1389 | + "stall_ms_per_token": 27.276057692307695 | |
| 1390 | + }, | |
| 1391 | + "multilingual": { | |
| 1392 | + "hit_rate": 0.4701522435897436, | |
| 1393 | + "stall_ms_per_token": 29.162820512820506 | |
| 1394 | + } | |
| 1395 | + }, | |
| 1396 | + "split": "heldout" | |
| 1397 | + }, | |
| 1398 | + { | |
| 1399 | + "capacity": 12, | |
| 1400 | + "policy": "lfu", | |
| 1401 | + "predictor": "transition", | |
| 1402 | + "m": 8, | |
| 1403 | + "hit_rate": 0.5544972189695551, | |
| 1404 | + "misses_per_token": 57.02435597189695, | |
| 1405 | + "stall_ms_per_token": 17.838453395784544, | |
| 1406 | + "prefetch_loads_per_token": 55.41545667447307, | |
| 1407 | + "prefetch_precision": 0.28176347285144365, | |
| 1408 | + "loads_per_token": 112.43981264637003, | |
| 1409 | + "pred_recall_at_k": 0.5302576112412178, | |
| 1410 | + "tokens": 2135, | |
| 1411 | + "families": { | |
| 1412 | + "general": { | |
| 1413 | + "hit_rate": 0.46577852614896986, | |
| 1414 | + "stall_ms_per_token": 21.2801141045959 | |
| 1415 | + }, | |
| 1416 | + "code": { | |
| 1417 | + "hit_rate": 0.6392315695600476, | |
| 1418 | + "stall_ms_per_token": 14.583581450653984 | |
| 1419 | + }, | |
| 1420 | + "reasoning": { | |
| 1421 | + "hit_rate": 0.5280361622073578, | |
| 1422 | + "stall_ms_per_token": 18.85084280936453 | |
| 1423 | + }, | |
| 1424 | + "multilingual": { | |
| 1425 | + "hit_rate": 0.5342548076923077, | |
| 1426 | + "stall_ms_per_token": 18.56085714285714 | |
| 1427 | + } | |
| 1428 | + }, | |
| 1429 | + "split": "fit" | |
| 1430 | + }, | |
| 1431 | + { | |
| 1432 | + "capacity": 12, | |
| 1433 | + "policy": "lfu", | |
| 1434 | + "predictor": "transition", | |
| 1435 | + "m": 8, | |
| 1436 | + "hit_rate": 0.5264036566390041, | |
| 1437 | + "misses_per_token": 60.62033195020747, | |
| 1438 | + "stall_ms_per_token": 18.922997233748315, | |
| 1439 | + "prefetch_loads_per_token": 59.40387275242047, | |
| 1440 | + "prefetch_precision": 0.26202007031595614, | |
| 1441 | + "loads_per_token": 120.02420470262794, | |
| 1442 | + "pred_recall_at_k": 0.4996369294605809, | |
| 1443 | + "tokens": 1446, | |
| 1444 | + "families": { | |
| 1445 | + "general": { | |
| 1446 | + "hit_rate": 0.4203903367158672, | |
| 1447 | + "stall_ms_per_token": 23.073022140221415 | |
| 1448 | + }, | |
| 1449 | + "code": { | |
| 1450 | + "hit_rate": 0.6239828209764918, | |
| 1451 | + "stall_ms_per_token": 15.122777576853533 | |
| 1452 | + }, | |
| 1453 | + "reasoning": { | |
| 1454 | + "hit_rate": 0.5442958733974359, | |
| 1455 | + "stall_ms_per_token": 18.19703205128204 | |
| 1456 | + }, | |
| 1457 | + "multilingual": { | |
| 1458 | + "hit_rate": 0.4729567307692308, | |
| 1459 | + "stall_ms_per_token": 20.94117948717948 | |
| 1460 | + } | |
| 1461 | + }, | |
| 1462 | + "split": "heldout" | |
| 1463 | + }, | |
| 1464 | + { | |
| 1465 | + "capacity": 12, | |
| 1466 | + "policy": "lfu", | |
| 1467 | + "predictor": "probe", | |
| 1468 | + "m": 8, | |
| 1469 | + "hit_rate": 0.3988985655737705, | |
| 1470 | + "misses_per_token": 76.94098360655738, | |
| 1471 | + "stall_ms_per_token": 23.853274941451996, | |
| 1472 | + "prefetch_loads_per_token": 98.13770491803278, | |
| 1473 | + "prefetch_precision": 0.09016628166701667, | |
| 1474 | + "loads_per_token": 175.07868852459018, | |
| 1475 | + "pred_recall_at_k": 0.16113973458235753, | |
| 1476 | + "tokens": 2135, | |
| 1477 | + "families": { | |
| 1478 | + "general": { | |
| 1479 | + "hit_rate": 0.3470929080824089, | |
| 1480 | + "stall_ms_per_token": 25.86802535657687 | |
| 1481 | + }, | |
| 1482 | + "code": { | |
| 1483 | + "hit_rate": 0.4414387633769322, | |
| 1484 | + "stall_ms_per_token": 22.229460166468503 | |
| 1485 | + }, | |
| 1486 | + "reasoning": { | |
| 1487 | + "hit_rate": 0.3731709866220736, | |
| 1488 | + "stall_ms_per_token": 24.837311036789284 | |
| 1489 | + }, | |
| 1490 | + "multilingual": { | |
| 1491 | + "hit_rate": 0.4115513392857143, | |
| 1492 | + "stall_ms_per_token": 23.30408241758242 | |
| 1493 | + } | |
| 1494 | + }, | |
| 1495 | + "split": "fit" | |
| 1496 | + }, | |
| 1497 | + { | |
| 1498 | + "capacity": 12, | |
| 1499 | + "policy": "lfu", | |
| 1500 | + "predictor": "probe", | |
| 1501 | + "m": 8, | |
| 1502 | + "hit_rate": 0.38072700553250344, | |
| 1503 | + "misses_per_token": 79.26694329183955, | |
| 1504 | + "stall_ms_per_token": 24.554273858921196, | |
| 1505 | + "prefetch_loads_per_token": 98.54564315352697, | |
| 1506 | + "prefetch_precision": 0.08690007508930013, | |
| 1507 | + "loads_per_token": 177.81258644536652, | |
| 1508 | + "pred_recall_at_k": 0.15494467496542186, | |
| 1509 | + "tokens": 1446, | |
| 1510 | + "families": { | |
| 1511 | + "general": { | |
| 1512 | + "hit_rate": 0.30225149907749077, | |
| 1513 | + "stall_ms_per_token": 27.639797047970482 | |
| 1514 | + }, | |
| 1515 | + "code": { | |
| 1516 | + "hit_rate": 0.45597875226039786, | |
| 1517 | + "stall_ms_per_token": 21.617142857142877 | |
| 1518 | + }, | |
| 1519 | + "reasoning": { | |
| 1520 | + "hit_rate": 0.3870693108974359, | |
| 1521 | + "stall_ms_per_token": 24.27478205128205 | |
| 1522 | + }, | |
| 1523 | + "multilingual": { | |
| 1524 | + "hit_rate": 0.3535657051282051, | |
| 1525 | + "stall_ms_per_token": 25.556358974358975 | |
| 1526 | + } | |
| 1527 | + }, | |
| 1528 | + "split": "heldout" | |
| 1529 | + }, | |
| 1530 | + { | |
| 1531 | + "capacity": 12, | |
| 1532 | + "policy": "lfu", | |
| 1533 | + "predictor": "oracle", | |
| 1534 | + "m": 8, | |
| 1535 | + "hit_rate": 0.7024736533957845, | |
| 1536 | + "misses_per_token": 38.08337236533958, | |
| 1537 | + "stall_ms_per_token": 12.118276346604212, | |
| 1538 | + "prefetch_loads_per_token": 61.144262295081965, | |
| 1539 | + "prefetch_precision": 0.534099875136928, | |
| 1540 | + "loads_per_token": 99.22763466042154, | |
| 1541 | + "pred_recall_at_k": 1.0, | |
| 1542 | + "tokens": 2135, | |
| 1543 | + "families": { | |
| 1544 | + "general": { | |
| 1545 | + "hit_rate": 0.6589119453248812, | |
| 1546 | + "stall_ms_per_token": 13.814348652931859 | |
| 1547 | + }, | |
| 1548 | + "code": { | |
| 1549 | + "hit_rate": 0.7429027942925089, | |
| 1550 | + "stall_ms_per_token": 10.576066587395966 | |
| 1551 | + }, | |
| 1552 | + "reasoning": { | |
| 1553 | + "hit_rate": 0.6835284280936454, | |
| 1554 | + "stall_ms_per_token": 12.84013377926422 | |
| 1555 | + }, | |
| 1556 | + "multilingual": { | |
| 1557 | + "hit_rate": 0.7001416552197802, | |
| 1558 | + "stall_ms_per_token": 12.148335164835158 | |
| 1559 | + } | |
| 1560 | + }, | |
| 1561 | + "split": "fit" | |
| 1562 | + }, | |
| 1563 | + { | |
| 1564 | + "capacity": 12, | |
| 1565 | + "policy": "lfu", | |
| 1566 | + "predictor": "oracle", | |
| 1567 | + "m": 8, | |
| 1568 | + "hit_rate": 0.699229555670816, | |
| 1569 | + "misses_per_token": 38.49861687413554, | |
| 1570 | + "stall_ms_per_token": 12.242239280774557, | |
| 1571 | + "prefetch_loads_per_token": 61.906639004149376, | |
| 1572 | + "prefetch_precision": 0.5330049040964286, | |
| 1573 | + "loads_per_token": 100.40525587828492, | |
| 1574 | + "pred_recall_at_k": 1.0, | |
| 1575 | + "tokens": 1446, | |
| 1576 | + "families": { | |
| 1577 | + "general": { | |
| 1578 | + "hit_rate": 0.6502969326568265, | |
| 1579 | + "stall_ms_per_token": 14.185752767527676 | |
| 1580 | + }, | |
| 1581 | + "code": { | |
| 1582 | + "hit_rate": 0.7454792043399638, | |
| 1583 | + "stall_ms_per_token": 10.426213381555154 | |
| 1584 | + }, | |
| 1585 | + "reasoning": { | |
| 1586 | + "hit_rate": 0.7046023637820513, | |
| 1587 | + "stall_ms_per_token": 12.000224358974366 | |
| 1588 | + }, | |
| 1589 | + "multilingual": { | |
| 1590 | + "hit_rate": 0.680488782051282, | |
| 1591 | + "stall_ms_per_token": 12.918820512820515 | |
| 1592 | + } | |
| 1593 | + }, | |
| 1594 | + "split": "heldout" | |
| 1595 | + }, | |
| 1596 | + { | |
| 1597 | + "capacity": 16, | |
| 1598 | + "policy": "lru", | |
| 1599 | + "predictor": "none", | |
| 1600 | + "m": 8, | |
| 1601 | + "hit_rate": 0.5847189695550351, | |
| 1602 | + "misses_per_token": 53.155971896955506, | |
| 1603 | + "stall_ms_per_token": 22.85706791569087, | |
| 1604 | + "prefetch_loads_per_token": 0.0, | |
| 1605 | + "prefetch_precision": null, | |
| 1606 | + "loads_per_token": 53.155971896955506, | |
| 1607 | + "pred_recall_at_k": null, | |
| 1608 | + "tokens": 2135, | |
| 1609 | + "families": { | |
| 1610 | + "general": { | |
| 1611 | + "hit_rate": 0.5302099841521395, | |
| 1612 | + "stall_ms_per_token": 25.857242472266257 | |
| 1613 | + }, | |
| 1614 | + "code": { | |
| 1615 | + "hit_rate": 0.6472205707491082, | |
| 1616 | + "stall_ms_per_token": 19.4169797859691 | |
| 1617 | + }, | |
| 1618 | + "reasoning": { | |
| 1619 | + "hit_rate": 0.5454640468227425, | |
| 1620 | + "stall_ms_per_token": 25.01765886287625 | |
| 1621 | + }, | |
| 1622 | + "multilingual": { | |
| 1623 | + "hit_rate": 0.5670501373626373, | |
| 1624 | + "stall_ms_per_token": 23.829560439560428 | |
| 1625 | + } | |
| 1626 | + }, | |
| 1627 | + "split": "fit" | |
| 1628 | + }, | |
| 1629 | + { | |
| 1630 | + "capacity": 16, | |
| 1631 | + "policy": "lru", | |
| 1632 | + "predictor": "none", | |
| 1633 | + "m": 8, | |
| 1634 | + "hit_rate": 0.5735001728907331, | |
| 1635 | + "misses_per_token": 54.59197786998617, | |
| 1636 | + "stall_ms_per_token": 23.47455048409405, | |
| 1637 | + "prefetch_loads_per_token": 0.0, | |
| 1638 | + "prefetch_precision": null, | |
| 1639 | + "loads_per_token": 54.59197786998617, | |
| 1640 | + "pred_recall_at_k": null, | |
| 1641 | + "tokens": 1446, | |
| 1642 | + "families": { | |
| 1643 | + "general": { | |
| 1644 | + "hit_rate": 0.498270295202952, | |
| 1645 | + "stall_ms_per_token": 27.61520295202951 | |
| 1646 | + }, | |
| 1647 | + "code": { | |
| 1648 | + "hit_rate": 0.6487624321880651, | |
| 1649 | + "stall_ms_per_token": 19.33211573236891 | |
| 1650 | + }, | |
| 1651 | + "reasoning": { | |
| 1652 | + "hit_rate": 0.5767227564102564, | |
| 1653 | + "stall_ms_per_token": 23.29717948717948 | |
| 1654 | + }, | |
| 1655 | + "multilingual": { | |
| 1656 | + "hit_rate": 0.5260416666666666, | |
| 1657 | + "stall_ms_per_token": 26.086666666666666 | |
| 1658 | + } | |
| 1659 | + }, | |
| 1660 | + "split": "heldout" | |
| 1661 | + }, | |
| 1662 | + { | |
| 1663 | + "capacity": 16, | |
| 1664 | + "policy": "lru", | |
| 1665 | + "predictor": "prev_token", | |
| 1666 | + "m": 8, | |
| 1667 | + "hit_rate": 0.5847189695550351, | |
| 1668 | + "misses_per_token": 53.155971896955506, | |
| 1669 | + "stall_ms_per_token": 22.85706791569087, | |
| 1670 | + "prefetch_loads_per_token": 0.0, | |
| 1671 | + "prefetch_precision": null, | |
| 1672 | + "loads_per_token": 53.155971896955506, | |
| 1673 | + "pred_recall_at_k": null, | |
| 1674 | + "tokens": 2135, | |
| 1675 | + "families": { | |
| 1676 | + "general": { | |
| 1677 | + "hit_rate": 0.5302099841521395, | |
| 1678 | + "stall_ms_per_token": 25.857242472266257 | |
| 1679 | + }, | |
| 1680 | + "code": { | |
| 1681 | + "hit_rate": 0.6472205707491082, | |
| 1682 | + "stall_ms_per_token": 19.4169797859691 | |
| 1683 | + }, | |
| 1684 | + "reasoning": { | |
| 1685 | + "hit_rate": 0.5454640468227425, | |
| 1686 | + "stall_ms_per_token": 25.01765886287625 | |
| 1687 | + }, | |
| 1688 | + "multilingual": { | |
| 1689 | + "hit_rate": 0.5670501373626373, | |
| 1690 | + "stall_ms_per_token": 23.829560439560428 | |
| 1691 | + } | |
| 1692 | + }, | |
| 1693 | + "split": "fit" | |
| 1694 | + }, | |
| 1695 | + { | |
| 1696 | + "capacity": 16, | |
| 1697 | + "policy": "lru", | |
| 1698 | + "predictor": "prev_token", | |
| 1699 | + "m": 8, | |
| 1700 | + "hit_rate": 0.5735001728907331, | |
| 1701 | + "misses_per_token": 54.59197786998617, | |
| 1702 | + "stall_ms_per_token": 23.47455048409405, | |
| 1703 | + "prefetch_loads_per_token": 0.0, | |
| 1704 | + "prefetch_precision": null, | |
| 1705 | + "loads_per_token": 54.59197786998617, | |
| 1706 | + "pred_recall_at_k": null, | |
| 1707 | + "tokens": 1446, | |
| 1708 | + "families": { | |
| 1709 | + "general": { | |
| 1710 | + "hit_rate": 0.498270295202952, | |
| 1711 | + "stall_ms_per_token": 27.61520295202951 | |
| 1712 | + }, | |
| 1713 | + "code": { | |
| 1714 | + "hit_rate": 0.6487624321880651, | |
| 1715 | + "stall_ms_per_token": 19.33211573236891 | |
| 1716 | + }, | |
| 1717 | + "reasoning": { | |
| 1718 | + "hit_rate": 0.5767227564102564, | |
| 1719 | + "stall_ms_per_token": 23.29717948717948 | |
| 1720 | + }, | |
| 1721 | + "multilingual": { | |
| 1722 | + "hit_rate": 0.5260416666666666, | |
| 1723 | + "stall_ms_per_token": 26.086666666666666 | |
| 1724 | + } | |
| 1725 | + }, | |
| 1726 | + "split": "heldout" | |
| 1727 | + }, | |
| 1728 | + { | |
| 1729 | + "capacity": 16, | |
| 1730 | + "policy": "lru", | |
| 1731 | + "predictor": "prev_rank", | |
| 1732 | + "m": 8, | |
| 1733 | + "hit_rate": 0.5847226288056206, | |
| 1734 | + "misses_per_token": 53.155503512880564, | |
| 1735 | + "stall_ms_per_token": 22.856866510538644, | |
| 1736 | + "prefetch_loads_per_token": 0.02857142857142857, | |
| 1737 | + "prefetch_precision": 0.3770491803278688, | |
| 1738 | + "loads_per_token": 53.18407494145199, | |
| 1739 | + "pred_recall_at_k": null, | |
| 1740 | + "tokens": 2135, | |
| 1741 | + "families": { | |
| 1742 | + "general": { | |
| 1743 | + "hit_rate": 0.5301852218700476, | |
| 1744 | + "stall_ms_per_token": 25.858605388272597 | |
| 1745 | + }, | |
| 1746 | + "code": { | |
| 1747 | + "hit_rate": 0.6472763079667063, | |
| 1748 | + "stall_ms_per_token": 19.413912009512508 | |
| 1749 | + }, | |
| 1750 | + "reasoning": { | |
| 1751 | + "hit_rate": 0.545516304347826, | |
| 1752 | + "stall_ms_per_token": 25.014782608695647 | |
| 1753 | + }, | |
| 1754 | + "multilingual": { | |
| 1755 | + "hit_rate": 0.5669428228021978, | |
| 1756 | + "stall_ms_per_token": 23.835467032967014 | |
| 1757 | + } | |
| 1758 | + }, | |
| 1759 | + "split": "fit" | |
| 1760 | + }, | |
| 1761 | + { | |
| 1762 | + "capacity": 16, | |
| 1763 | + "policy": "lru", | |
| 1764 | + "predictor": "prev_rank", | |
| 1765 | + "m": 8, | |
| 1766 | + "hit_rate": 0.573505575726141, | |
| 1767 | + "misses_per_token": 54.59128630705394, | |
| 1768 | + "stall_ms_per_token": 23.474253112033196, | |
| 1769 | + "prefetch_loads_per_token": 0.02074688796680498, | |
| 1770 | + "prefetch_precision": 0.4, | |
| 1771 | + "loads_per_token": 54.61203319502075, | |
| 1772 | + "pred_recall_at_k": null, | |
| 1773 | + "tokens": 1446, | |
| 1774 | + "families": { | |
| 1775 | + "general": { | |
| 1776 | + "hit_rate": 0.4982991236162362, | |
| 1777 | + "stall_ms_per_token": 27.61361623616235 | |
| 1778 | + }, | |
| 1779 | + "code": { | |
| 1780 | + "hit_rate": 0.6487624321880651, | |
| 1781 | + "stall_ms_per_token": 19.332115732368905 | |
| 1782 | + }, | |
| 1783 | + "reasoning": { | |
| 1784 | + "hit_rate": 0.5766977163461539, | |
| 1785 | + "stall_ms_per_token": 23.298557692307682 | |
| 1786 | + }, | |
| 1787 | + "multilingual": { | |
| 1788 | + "hit_rate": 0.5260416666666666, | |
| 1789 | + "stall_ms_per_token": 26.086666666666666 | |
| 1790 | + } | |
| 1791 | + }, | |
| 1792 | + "split": "heldout" | |
| 1793 | + }, | |
| 1794 | + { | |
| 1795 | + "capacity": 16, | |
| 1796 | + "policy": "lru", | |
| 1797 | + "predictor": "transition", | |
| 1798 | + "m": 8, | |
| 1799 | + "hit_rate": 0.6708028395784543, | |
| 1800 | + "misses_per_token": 42.13723653395785, | |
| 1801 | + "stall_ms_per_token": 13.278813114754097, | |
| 1802 | + "prefetch_loads_per_token": 41.81967213114754, | |
| 1803 | + "prefetch_precision": 0.526045808366467, | |
| 1804 | + "loads_per_token": 83.95690866510539, | |
| 1805 | + "pred_recall_at_k": 0.5302576112412178, | |
| 1806 | + "tokens": 2135, | |
| 1807 | + "families": { | |
| 1808 | + "general": { | |
| 1809 | + "hit_rate": 0.6250247622820919, | |
| 1810 | + "stall_ms_per_token": 15.060393026941366 | |
| 1811 | + }, | |
| 1812 | + "code": { | |
| 1813 | + "hit_rate": 0.7201712990487514, | |
| 1814 | + "stall_ms_per_token": 11.381871581450664 | |
| 1815 | + }, | |
| 1816 | + "reasoning": { | |
| 1817 | + "hit_rate": 0.6466607441471572, | |
| 1818 | + "stall_ms_per_token": 14.196795986622066 | |
| 1819 | + }, | |
| 1820 | + "multilingual": { | |
| 1821 | + "hit_rate": 0.6559280563186813, | |
| 1822 | + "stall_ms_per_token": 13.819126373626382 | |
| 1823 | + } | |
| 1824 | + }, | |
| 1825 | + "split": "fit" | |
| 1826 | + }, | |
| 1827 | + { | |
| 1828 | + "capacity": 16, | |
| 1829 | + "policy": "lru", | |
| 1830 | + "predictor": "transition", | |
| 1831 | + "m": 8, | |
| 1832 | + "hit_rate": 0.663327714384509, | |
| 1833 | + "misses_per_token": 43.09405255878285, | |
| 1834 | + "stall_ms_per_token": 13.57057538035961, | |
| 1835 | + "prefetch_loads_per_token": 44.85615491009682, | |
| 1836 | + "prefetch_precision": 0.4896241250655237, | |
| 1837 | + "loads_per_token": 87.95020746887967, | |
| 1838 | + "pred_recall_at_k": 0.4996369294605809, | |
| 1839 | + "tokens": 1446, | |
| 1840 | + "families": { | |
| 1841 | + "general": { | |
| 1842 | + "hit_rate": 0.6166541743542435, | |
| 1843 | + "stall_ms_per_token": 15.432874538745384 | |
| 1844 | + }, | |
| 1845 | + "code": { | |
| 1846 | + "hit_rate": 0.7129718580470162, | |
| 1847 | + "stall_ms_per_token": 11.627497287522608 | |
| 1848 | + }, | |
| 1849 | + "reasoning": { | |
| 1850 | + "hit_rate": 0.660331530448718, | |
| 1851 | + "stall_ms_per_token": 13.632788461538459 | |
| 1852 | + }, | |
| 1853 | + "multilingual": { | |
| 1854 | + "hit_rate": 0.632011217948718, | |
| 1855 | + "stall_ms_per_token": 14.743538461538463 | |
| 1856 | + } | |
| 1857 | + }, | |
| 1858 | + "split": "heldout" | |
| 1859 | + }, | |
| 1860 | + { | |
| 1861 | + "capacity": 16, | |
| 1862 | + "policy": "lru", | |
| 1863 | + "predictor": "probe", | |
| 1864 | + "m": 8, | |
| 1865 | + "hit_rate": 0.5208321135831382, | |
| 1866 | + "misses_per_token": 61.33348946135831, | |
| 1867 | + "stall_ms_per_token": 19.076081498829023, | |
| 1868 | + "prefetch_loads_per_token": 80.38220140515223, | |
| 1869 | + "prefetch_precision": 0.1531500559388402, | |
| 1870 | + "loads_per_token": 141.71569086651053, | |
| 1871 | + "pred_recall_at_k": 0.16113973458235753, | |
| 1872 | + "tokens": 2135, | |
| 1873 | + "families": { | |
| 1874 | + "general": { | |
| 1875 | + "hit_rate": 0.48129209587955624, | |
| 1876 | + "stall_ms_per_token": 20.616522979397757 | |
| 1877 | + }, | |
| 1878 | + "code": { | |
| 1879 | + "hit_rate": 0.5574186236623068, | |
| 1880 | + "stall_ms_per_token": 17.67323900118906 | |
| 1881 | + }, | |
| 1882 | + "reasoning": { | |
| 1883 | + "hit_rate": 0.5044680183946488, | |
| 1884 | + "stall_ms_per_token": 19.693397993311034 | |
| 1885 | + }, | |
| 1886 | + "multilingual": { | |
| 1887 | + "hit_rate": 0.5182864010989011, | |
| 1888 | + "stall_ms_per_token": 19.1398021978022 | |
| 1889 | + } | |
| 1890 | + }, | |
| 1891 | + "split": "fit" | |
| 1892 | + }, | |
| 1893 | + { | |
| 1894 | + "capacity": 16, | |
| 1895 | + "policy": "lru", | |
| 1896 | + "predictor": "probe", | |
| 1897 | + "m": 8, | |
| 1898 | + "hit_rate": 0.521000821230982, | |
| 1899 | + "misses_per_token": 61.3118948824343, | |
| 1900 | + "stall_ms_per_token": 19.07236376210236, | |
| 1901 | + "prefetch_loads_per_token": 79.27178423236515, | |
| 1902 | + "prefetch_precision": 0.14992977221771486, | |
| 1903 | + "loads_per_token": 140.58367911479945, | |
| 1904 | + "pred_recall_at_k": 0.15494467496542186, | |
| 1905 | + "tokens": 1446, | |
| 1906 | + "families": { | |
| 1907 | + "general": { | |
| 1908 | + "hit_rate": 0.47108510147601473, | |
| 1909 | + "stall_ms_per_token": 21.059992619926195 | |
| 1910 | + }, | |
| 1911 | + "code": { | |
| 1912 | + "hit_rate": 0.5767828887884268, | |
| 1913 | + "stall_ms_per_token": 16.892018083182652 | |
| 1914 | + }, | |
| 1915 | + "reasoning": { | |
| 1916 | + "hit_rate": 0.5087389823717948, | |
| 1917 | + "stall_ms_per_token": 19.492749999999997 | |
| 1918 | + }, | |
| 1919 | + "multilingual": { | |
| 1920 | + "hit_rate": 0.5218349358974359, | |
| 1921 | + "stall_ms_per_token": 19.002512820512827 | |
| 1922 | + } | |
| 1923 | + }, | |
| 1924 | + "split": "heldout" | |
| 1925 | + }, | |
| 1926 | + { | |
| 1927 | + "capacity": 16, | |
| 1928 | + "policy": "lru", | |
| 1929 | + "predictor": "oracle", | |
| 1930 | + "m": 8, | |
| 1931 | + "hit_rate": 0.9387917154566745, | |
| 1932 | + "misses_per_token": 7.834660421545667, | |
| 1933 | + "stall_ms_per_token": 2.9194351288056275, | |
| 1934 | + "prefetch_loads_per_token": 51.83044496487119, | |
| 1935 | + "prefetch_precision": 1.0, | |
| 1936 | + "loads_per_token": 59.66510538641686, | |
| 1937 | + "pred_recall_at_k": 1.0, | |
| 1938 | + "tokens": 2135, | |
| 1939 | + "families": { | |
| 1940 | + "general": { | |
| 1941 | + "hit_rate": 0.9340951862123613, | |
| 1942 | + "stall_ms_per_token": 3.1129667194928694 | |
| 1943 | + }, | |
| 1944 | + "code": { | |
| 1945 | + "hit_rate": 0.9424791914387634, | |
| 1946 | + "stall_ms_per_token": 2.788337693222353 | |
| 1947 | + }, | |
| 1948 | + "reasoning": { | |
| 1949 | + "hit_rate": 0.9400083612040134, | |
| 1950 | + "stall_ms_per_token": 2.8571505016722414 | |
| 1951 | + }, | |
| 1952 | + "multilingual": { | |
| 1953 | + "hit_rate": 0.9374141483516484, | |
| 1954 | + "stall_ms_per_token": 2.9380000000000006 | |
| 1955 | + } | |
| 1956 | + }, | |
| 1957 | + "split": "fit" | |
| 1958 | + }, | |
| 1959 | + { | |
| 1960 | + "capacity": 16, | |
| 1961 | + "policy": "lru", | |
| 1962 | + "predictor": "oracle", | |
| 1963 | + "m": 8, | |
| 1964 | + "hit_rate": 0.9417412257952974, | |
| 1965 | + "misses_per_token": 7.4571230982019365, | |
| 1966 | + "stall_ms_per_token": 2.808222683264183, | |
| 1967 | + "prefetch_loads_per_token": 53.25449515905947, | |
| 1968 | + "prefetch_precision": 1.0, | |
| 1969 | + "loads_per_token": 60.711618257261414, | |
| 1970 | + "pred_recall_at_k": 1.0, | |
| 1971 | + "tokens": 1446, | |
| 1972 | + "families": { | |
| 1973 | + "general": { | |
| 1974 | + "hit_rate": 0.9358711946494465, | |
| 1975 | + "stall_ms_per_token": 3.0932214022140254 | |
| 1976 | + }, | |
| 1977 | + "code": { | |
| 1978 | + "hit_rate": 0.9478271925858951, | |
| 1979 | + "stall_ms_per_token": 2.5489294755877037 | |
| 1980 | + }, | |
| 1981 | + "reasoning": { | |
| 1982 | + "hit_rate": 0.9409304887820513, | |
| 1983 | + "stall_ms_per_token": 2.785955128205128 | |
| 1984 | + }, | |
| 1985 | + "multilingual": { | |
| 1986 | + "hit_rate": 0.9435096153846154, | |
| 1987 | + "stall_ms_per_token": 2.702256410256411 | |
| 1988 | + } | |
| 1989 | + }, | |
| 1990 | + "split": "heldout" | |
| 1991 | + }, | |
| 1992 | + { | |
| 1993 | + "capacity": 16, | |
| 1994 | + "policy": "lfu", | |
| 1995 | + "predictor": "none", | |
| 1996 | + "m": 8, | |
| 1997 | + "hit_rate": 0.5952393149882904, | |
| 1998 | + "misses_per_token": 51.80936768149883, | |
| 1999 | + "stall_ms_per_token": 22.278028103044548, | |
| 2000 | + "prefetch_loads_per_token": 0.0, | |
| 2001 | + "prefetch_precision": null, | |
| 2002 | + "loads_per_token": 51.80936768149883, | |
| 2003 | + "pred_recall_at_k": null, | |
| 2004 | + "tokens": 2135, | |
| 2005 | + "families": { | |
| 2006 | + "general": { | |
| 2007 | + "hit_rate": 0.5144611727416799, | |
| 2008 | + "stall_ms_per_token": 26.72405705229791 | |
| 2009 | + }, | |
| 2010 | + "code": { | |
| 2011 | + "hit_rate": 0.6684285820451843, | |
| 2012 | + "stall_ms_per_token": 18.249690844233054 | |
| 2013 | + }, | |
| 2014 | + "reasoning": { | |
| 2015 | + "hit_rate": 0.5496969063545151, | |
| 2016 | + "stall_ms_per_token": 24.784682274247473 | |
| 2017 | + }, | |
| 2018 | + "multilingual": { | |
| 2019 | + "hit_rate": 0.6035800137362637, | |
| 2020 | + "stall_ms_per_token": 21.818956043956014 | |
| 2021 | + } | |
| 2022 | + }, | |
| 2023 | + "split": "fit" | |
| 2024 | + }, | |
| 2025 | + { | |
| 2026 | + "capacity": 16, | |
| 2027 | + "policy": "lfu", | |
| 2028 | + "predictor": "none", | |
| 2029 | + "m": 8, | |
| 2030 | + "hit_rate": 0.5910377766251729, | |
| 2031 | + "misses_per_token": 52.34716459197787, | |
| 2032 | + "stall_ms_per_token": 22.509280774550543, | |
| 2033 | + "prefetch_loads_per_token": 0.0, | |
| 2034 | + "prefetch_precision": null, | |
| 2035 | + "loads_per_token": 52.34716459197787, | |
| 2036 | + "pred_recall_at_k": null, | |
| 2037 | + "tokens": 1446, | |
| 2038 | + "families": { | |
| 2039 | + "general": { | |
| 2040 | + "hit_rate": 0.5278050046125461, | |
| 2041 | + "stall_ms_per_token": 25.98961254612546 | |
| 2042 | + }, | |
| 2043 | + "code": { | |
| 2044 | + "hit_rate": 0.6605871383363472, | |
| 2045 | + "stall_ms_per_token": 18.68128390596748 | |
| 2046 | + }, | |
| 2047 | + "reasoning": { | |
| 2048 | + "hit_rate": 0.5844100560897436, | |
| 2049 | + "stall_ms_per_token": 22.874070512820506 | |
| 2050 | + }, | |
| 2051 | + "multilingual": { | |
| 2052 | + "hit_rate": 0.5366586538461539, | |
| 2053 | + "stall_ms_per_token": 25.50230769230769 | |
| 2054 | + } | |
| 2055 | + }, | |
| 2056 | + "split": "heldout" | |
| 2057 | + }, | |
| 2058 | + { | |
| 2059 | + "capacity": 16, | |
| 2060 | + "policy": "lfu", | |
| 2061 | + "predictor": "prev_token", | |
| 2062 | + "m": 8, | |
| 2063 | + "hit_rate": 0.5952393149882904, | |
| 2064 | + "misses_per_token": 51.80936768149883, | |
| 2065 | + "stall_ms_per_token": 22.278028103044548, | |
| 2066 | + "prefetch_loads_per_token": 0.0, | |
| 2067 | + "prefetch_precision": null, | |
| 2068 | + "loads_per_token": 51.80936768149883, | |
| 2069 | + "pred_recall_at_k": null, | |
| 2070 | + "tokens": 2135, | |
| 2071 | + "families": { | |
| 2072 | + "general": { | |
| 2073 | + "hit_rate": 0.5144611727416799, | |
| 2074 | + "stall_ms_per_token": 26.72405705229791 | |
| 2075 | + }, | |
| 2076 | + "code": { | |
| 2077 | + "hit_rate": 0.6684285820451843, | |
| 2078 | + "stall_ms_per_token": 18.249690844233054 | |
| 2079 | + }, | |
| 2080 | + "reasoning": { | |
| 2081 | + "hit_rate": 0.5496969063545151, | |
| 2082 | + "stall_ms_per_token": 24.784682274247473 | |
| 2083 | + }, | |
| 2084 | + "multilingual": { | |
| 2085 | + "hit_rate": 0.6035800137362637, | |
| 2086 | + "stall_ms_per_token": 21.818956043956014 | |
| 2087 | + } | |
| 2088 | + }, | |
| 2089 | + "split": "fit" | |
| 2090 | + }, | |
| 2091 | + { | |
| 2092 | + "capacity": 16, | |
| 2093 | + "policy": "lfu", | |
| 2094 | + "predictor": "prev_token", | |
| 2095 | + "m": 8, | |
| 2096 | + "hit_rate": 0.5910377766251729, | |
| 2097 | + "misses_per_token": 52.34716459197787, | |
| 2098 | + "stall_ms_per_token": 22.509280774550543, | |
| 2099 | + "prefetch_loads_per_token": 0.0, | |
| 2100 | + "prefetch_precision": null, | |
| 2101 | + "loads_per_token": 52.34716459197787, | |
| 2102 | + "pred_recall_at_k": null, | |
| 2103 | + "tokens": 1446, | |
| 2104 | + "families": { | |
| 2105 | + "general": { | |
| 2106 | + "hit_rate": 0.5278050046125461, | |
| 2107 | + "stall_ms_per_token": 25.98961254612546 | |
| 2108 | + }, | |
| 2109 | + "code": { | |
| 2110 | + "hit_rate": 0.6605871383363472, | |
| 2111 | + "stall_ms_per_token": 18.68128390596748 | |
| 2112 | + }, | |
| 2113 | + "reasoning": { | |
| 2114 | + "hit_rate": 0.5844100560897436, | |
| 2115 | + "stall_ms_per_token": 22.874070512820506 | |
| 2116 | + }, | |
| 2117 | + "multilingual": { | |
| 2118 | + "hit_rate": 0.5366586538461539, | |
| 2119 | + "stall_ms_per_token": 25.50230769230769 | |
| 2120 | + } | |
| 2121 | + }, | |
| 2122 | + "split": "heldout" | |
| 2123 | + }, | |
| 2124 | + { | |
| 2125 | + "capacity": 16, | |
| 2126 | + "policy": "lfu", | |
| 2127 | + "predictor": "prev_rank", | |
| 2128 | + "m": 8, | |
| 2129 | + "hit_rate": 0.5951990632318501, | |
| 2130 | + "misses_per_token": 51.814519906323184, | |
| 2131 | + "stall_ms_per_token": 22.280243559719025, | |
| 2132 | + "prefetch_loads_per_token": 0.027166276346604217, | |
| 2133 | + "prefetch_precision": 0.15517241379310345, | |
| 2134 | + "loads_per_token": 51.84168618266979, | |
| 2135 | + "pred_recall_at_k": null, | |
| 2136 | + "tokens": 2135, | |
| 2137 | + "families": { | |
| 2138 | + "general": { | |
| 2139 | + "hit_rate": 0.5143992670364501, | |
| 2140 | + "stall_ms_per_token": 26.727464342313755 | |
| 2141 | + }, | |
| 2142 | + "code": { | |
| 2143 | + "hit_rate": 0.6684100029726516, | |
| 2144 | + "stall_ms_per_token": 18.250713436385254 | |
| 2145 | + }, | |
| 2146 | + "reasoning": { | |
| 2147 | + "hit_rate": 0.5496969063545151, | |
| 2148 | + "stall_ms_per_token": 24.784682274247476 | |
| 2149 | + }, | |
| 2150 | + "multilingual": { | |
| 2151 | + "hit_rate": 0.6034941620879121, | |
| 2152 | + "stall_ms_per_token": 21.823681318681288 | |
| 2153 | + } | |
| 2154 | + }, | |
| 2155 | + "split": "fit" | |
| 2156 | + }, | |
| 2157 | + { | |
| 2158 | + "capacity": 16, | |
| 2159 | + "policy": "lfu", | |
| 2160 | + "predictor": "prev_rank", | |
| 2161 | + "m": 8, | |
| 2162 | + "hit_rate": 0.5910377766251729, | |
| 2163 | + "misses_per_token": 52.34716459197787, | |
| 2164 | + "stall_ms_per_token": 22.509280774550557, | |
| 2165 | + "prefetch_loads_per_token": 0.025587828492392807, | |
| 2166 | + "prefetch_precision": 0.2972972972972973, | |
| 2167 | + "loads_per_token": 52.37275242047026, | |
| 2168 | + "pred_recall_at_k": null, | |
| 2169 | + "tokens": 1446, | |
| 2170 | + "families": { | |
| 2171 | + "general": { | |
| 2172 | + "hit_rate": 0.527790590405904, | |
| 2173 | + "stall_ms_per_token": 25.990405904059042 | |
| 2174 | + }, | |
| 2175 | + "code": { | |
| 2176 | + "hit_rate": 0.6606012658227848, | |
| 2177 | + "stall_ms_per_token": 18.68050632911395 | |
| 2178 | + }, | |
| 2179 | + "reasoning": { | |
| 2180 | + "hit_rate": 0.5844100560897436, | |
| 2181 | + "stall_ms_per_token": 22.874070512820502 | |
| 2182 | + }, | |
| 2183 | + "multilingual": { | |
| 2184 | + "hit_rate": 0.5366586538461539, | |
| 2185 | + "stall_ms_per_token": 25.50230769230769 | |
| 2186 | + } | |
| 2187 | + }, | |
| 2188 | + "split": "heldout" | |
| 2189 | + }, | |
| 2190 | + { | |
| 2191 | + "capacity": 16, | |
| 2192 | + "policy": "lfu", | |
| 2193 | + "predictor": "transition", | |
| 2194 | + "m": 8, | |
| 2195 | + "hit_rate": 0.6237558548009368, | |
| 2196 | + "misses_per_token": 48.159250585480095, | |
| 2197 | + "stall_ms_per_token": 15.091765807962553, | |
| 2198 | + "prefetch_loads_per_token": 42.2327868852459, | |
| 2199 | + "prefetch_precision": 0.26741490789313166, | |
| 2200 | + "loads_per_token": 90.392037470726, | |
| 2201 | + "pred_recall_at_k": 0.5302576112412178, | |
| 2202 | + "tokens": 2135, | |
| 2203 | + "families": { | |
| 2204 | + "general": { | |
| 2205 | + "hit_rate": 0.5346548137876387, | |
| 2206 | + "stall_ms_per_token": 18.54663391442157 | |
| 2207 | + }, | |
| 2208 | + "code": { | |
| 2209 | + "hit_rate": 0.7047785374554102, | |
| 2210 | + "stall_ms_per_token": 11.99363614744351 | |
| 2211 | + }, | |
| 2212 | + "reasoning": { | |
| 2213 | + "hit_rate": 0.5937761287625418, | |
| 2214 | + "stall_ms_per_token": 16.246240802675587 | |
| 2215 | + }, | |
| 2216 | + "multilingual": { | |
| 2217 | + "hit_rate": 0.6156421703296703, | |
| 2218 | + "stall_ms_per_token": 15.31241758241758 | |
| 2219 | + } | |
| 2220 | + }, | |
| 2221 | + "split": "fit" | |
| 2222 | + }, | |
| 2223 | + { | |
| 2224 | + "capacity": 16, | |
| 2225 | + "policy": "lfu", | |
| 2226 | + "predictor": "transition", | |
| 2227 | + "m": 8, | |
| 2228 | + "hit_rate": 0.6016543482019364, | |
| 2229 | + "misses_per_token": 50.988243430152146, | |
| 2230 | + "stall_ms_per_token": 15.921426002766262, | |
| 2231 | + "prefetch_loads_per_token": 47.10857538035961, | |
| 2232 | + "prefetch_precision": 0.24404351208913813, | |
| 2233 | + "loads_per_token": 98.09681881051176, | |
| 2234 | + "pred_recall_at_k": 0.4996369294605809, | |
| 2235 | + "tokens": 1446, | |
| 2236 | + "families": { | |
| 2237 | + "general": { | |
| 2238 | + "hit_rate": 0.5040503920664207, | |
| 2239 | + "stall_ms_per_token": 19.725937269372704 | |
| 2240 | + }, | |
| 2241 | + "code": { | |
| 2242 | + "hit_rate": 0.6924587477396021, | |
| 2243 | + "stall_ms_per_token": 12.391518987341772 | |
| 2244 | + }, | |
| 2245 | + "reasoning": { | |
| 2246 | + "hit_rate": 0.6154847756410257, | |
| 2247 | + "stall_ms_per_token": 15.369666666666689 | |
| 2248 | + }, | |
| 2249 | + "multilingual": { | |
| 2250 | + "hit_rate": 0.5598958333333334, | |
| 2251 | + "stall_ms_per_token": 17.514820512820513 | |
| 2252 | + } | |
| 2253 | + }, | |
| 2254 | + "split": "heldout" | |
| 2255 | + }, | |
| 2256 | + { | |
| 2257 | + "capacity": 16, | |
| 2258 | + "policy": "lfu", | |
| 2259 | + "predictor": "probe", | |
| 2260 | + "m": 8, | |
| 2261 | + "hit_rate": 0.4996450526932084, | |
| 2262 | + "misses_per_token": 64.04543325526932, | |
| 2263 | + "stall_ms_per_token": 19.889392974238874, | |
| 2264 | + "prefetch_loads_per_token": 88.52412177985948, | |
| 2265 | + "prefetch_precision": 0.08048719834496479, | |
| 2266 | + "loads_per_token": 152.56955503512881, | |
| 2267 | + "pred_recall_at_k": 0.16113973458235753, | |
| 2268 | + "tokens": 2135, | |
| 2269 | + "families": { | |
| 2270 | + "general": { | |
| 2271 | + "hit_rate": 0.43546949286846276, | |
| 2272 | + "stall_ms_per_token": 22.38074167987323 | |
| 2273 | + }, | |
| 2274 | + "code": { | |
| 2275 | + "hit_rate": 0.5495039387633769, | |
| 2276 | + "stall_ms_per_token": 17.995931034482762 | |
| 2277 | + }, | |
| 2278 | + "reasoning": { | |
| 2279 | + "hit_rate": 0.4644126254180602, | |
| 2280 | + "stall_ms_per_token": 21.24691638795987 | |
| 2281 | + }, | |
| 2282 | + "multilingual": { | |
| 2283 | + "hit_rate": 0.5246394230769231, | |
| 2284 | + "stall_ms_per_token": 18.830219780219814 | |
| 2285 | + } | |
| 2286 | + }, | |
| 2287 | + "split": "fit" | |
| 2288 | + }, | |
| 2289 | + { | |
| 2290 | + "capacity": 16, | |
| 2291 | + "policy": "lfu", | |
| 2292 | + "predictor": "probe", | |
| 2293 | + "m": 8, | |
| 2294 | + "hit_rate": 0.48206798928077454, | |
| 2295 | + "misses_per_token": 66.29529737206086, | |
| 2296 | + "stall_ms_per_token": 20.544156293222706, | |
| 2297 | + "prefetch_loads_per_token": 89.36721991701245, | |
| 2298 | + "prefetch_precision": 0.07805765138324627, | |
| 2299 | + "loads_per_token": 155.6625172890733, | |
| 2300 | + "pred_recall_at_k": 0.15494467496542186, | |
| 2301 | + "tokens": 1446, | |
| 2302 | + "families": { | |
| 2303 | + "general": { | |
| 2304 | + "hit_rate": 0.4055581180811808, | |
| 2305 | + "stall_ms_per_token": 23.533254612546116 | |
| 2306 | + }, | |
| 2307 | + "code": { | |
| 2308 | + "hit_rate": 0.557753164556962, | |
| 2309 | + "stall_ms_per_token": 17.598698010849922 | |
| 2310 | + }, | |
| 2311 | + "reasoning": { | |
| 2312 | + "hit_rate": 0.48389923878205127, | |
| 2313 | + "stall_ms_per_token": 20.45623717948717 | |
| 2314 | + }, | |
| 2315 | + "multilingual": { | |
| 2316 | + "hit_rate": 0.45753205128205127, | |
| 2317 | + "stall_ms_per_token": 21.471794871794867 | |
| 2318 | + } | |
| 2319 | + }, | |
| 2320 | + "split": "heldout" | |
| 2321 | + }, | |
| 2322 | + { | |
| 2323 | + "capacity": 16, | |
| 2324 | + "policy": "lfu", | |
| 2325 | + "predictor": "oracle", | |
| 2326 | + "m": 8, | |
| 2327 | + "hit_rate": 0.7585077576112412, | |
| 2328 | + "misses_per_token": 30.911007025761123, | |
| 2329 | + "stall_ms_per_token": 9.882796252927411, | |
| 2330 | + "prefetch_loads_per_token": 50.71288056206089, | |
| 2331 | + "prefetch_precision": 0.5401950642825477, | |
| 2332 | + "loads_per_token": 81.62388758782201, | |
| 2333 | + "pred_recall_at_k": 1.0, | |
| 2334 | + "tokens": 2135, | |
| 2335 | + "families": { | |
| 2336 | + "general": { | |
| 2337 | + "hit_rate": 0.7085850832012678, | |
| 2338 | + "stall_ms_per_token": 11.823185419968304 | |
| 2339 | + }, | |
| 2340 | + "code": { | |
| 2341 | + "hit_rate": 0.7996247027348394, | |
| 2342 | + "stall_ms_per_token": 8.32726278240191 | |
| 2343 | + }, | |
| 2344 | + "reasoning": { | |
| 2345 | + "hit_rate": 0.7361778846153846, | |
| 2346 | + "stall_ms_per_token": 10.74155852842809 | |
| 2347 | + }, | |
| 2348 | + "multilingual": { | |
| 2349 | + "hit_rate": 0.7683937156593407, | |
| 2350 | + "stall_ms_per_token": 9.407653846153847 | |
| 2351 | + } | |
| 2352 | + }, | |
| 2353 | + "split": "fit" | |
| 2354 | + }, | |
| 2355 | + { | |
| 2356 | + "capacity": 16, | |
| 2357 | + "policy": "lfu", | |
| 2358 | + "predictor": "oracle", | |
| 2359 | + "m": 8, | |
| 2360 | + "hit_rate": 0.7510049273858921, | |
| 2361 | + "misses_per_token": 31.87136929460581, | |
| 2362 | + "stall_ms_per_token": 10.148130013831263, | |
| 2363 | + "prefetch_loads_per_token": 52.30843706777317, | |
| 2364 | + "prefetch_precision": 0.537348951585182, | |
| 2365 | + "loads_per_token": 84.17980636237898, | |
| 2366 | + "pred_recall_at_k": 1.0, | |
| 2367 | + "tokens": 1446, | |
| 2368 | + "families": { | |
| 2369 | + "general": { | |
| 2370 | + "hit_rate": 0.7049411900369004, | |
| 2371 | + "stall_ms_per_token": 11.960302583025829 | |
| 2372 | + }, | |
| 2373 | + "code": { | |
| 2374 | + "hit_rate": 0.7956882911392406, | |
| 2375 | + "stall_ms_per_token": 8.401077757685355 | |
| 2376 | + }, | |
| 2377 | + "reasoning": { | |
| 2378 | + "hit_rate": 0.754081530448718, | |
| 2379 | + "stall_ms_per_token": 10.012070512820515 | |
| 2380 | + }, | |
| 2381 | + "multilingual": { | |
| 2382 | + "hit_rate": 0.7329727564102564, | |
| 2383 | + "stall_ms_per_token": 10.824358974358976 | |
| 2384 | + } | |
| 2385 | + }, | |
| 2386 | + "split": "heldout" | |
| 2387 | + }, | |
| 2388 | + { | |
| 2389 | + "capacity": 24, | |
| 2390 | + "policy": "lru", | |
| 2391 | + "predictor": "none", | |
| 2392 | + "m": 8, | |
| 2393 | + "hit_rate": 0.7070294203747073, | |
| 2394 | + "misses_per_token": 37.50023419203747, | |
| 2395 | + "stall_ms_per_token": 16.12510070257611, | |
| 2396 | + "prefetch_loads_per_token": 0.0, | |
| 2397 | + "prefetch_precision": null, | |
| 2398 | + "loads_per_token": 37.50023419203747, | |
| 2399 | + "pred_recall_at_k": null, | |
| 2400 | + "tokens": 2135, | |
| 2401 | + "families": { | |
| 2402 | + "general": { | |
| 2403 | + "hit_rate": 0.6611776941362916, | |
| 2404 | + "stall_ms_per_token": 18.648779714738513 | |
| 2405 | + }, | |
| 2406 | + "code": { | |
| 2407 | + "hit_rate": 0.760868757431629, | |
| 2408 | + "stall_ms_per_token": 13.16178359096315 | |
| 2409 | + }, | |
| 2410 | + "reasoning": { | |
| 2411 | + "hit_rate": 0.6581835284280937, | |
| 2412 | + "stall_ms_per_token": 18.813578595317725 | |
| 2413 | + }, | |
| 2414 | + "multilingual": { | |
| 2415 | + "hit_rate": 0.7022450206043956, | |
| 2416 | + "stall_ms_per_token": 16.38843406593407 | |
| 2417 | + } | |
| 2418 | + }, | |
| 2419 | + "split": "fit" | |
| 2420 | + }, | |
| 2421 | + { | |
| 2422 | + "capacity": 24, | |
| 2423 | + "policy": "lru", | |
| 2424 | + "predictor": "none", | |
| 2425 | + "m": 8, | |
| 2426 | + "hit_rate": 0.6954961964038727, | |
| 2427 | + "misses_per_token": 38.976486860304284, | |
| 2428 | + "stall_ms_per_token": 16.759889349930848, | |
| 2429 | + "prefetch_loads_per_token": 0.0, | |
| 2430 | + "prefetch_precision": null, | |
| 2431 | + "loads_per_token": 38.976486860304284, | |
| 2432 | + "pred_recall_at_k": null, | |
| 2433 | + "tokens": 1446, | |
| 2434 | + "families": { | |
| 2435 | + "general": { | |
| 2436 | + "hit_rate": 0.650830258302583, | |
| 2437 | + "stall_ms_per_token": 19.218302583025828 | |
| 2438 | + }, | |
| 2439 | + "code": { | |
| 2440 | + "hit_rate": 0.7484177215189873, | |
| 2441 | + "stall_ms_per_token": 13.847088607594944 | |
| 2442 | + }, | |
| 2443 | + "reasoning": { | |
| 2444 | + "hit_rate": 0.6880508814102564, | |
| 2445 | + "stall_ms_per_token": 17.1696794871795 | |
| 2446 | + }, | |
| 2447 | + "multilingual": { | |
| 2448 | + "hit_rate": 0.6254006410256411, | |
| 2449 | + "stall_ms_per_token": 20.617948717948714 | |
| 2450 | + } | |
| 2451 | + }, | |
| 2452 | + "split": "heldout" | |
| 2453 | + }, | |
| 2454 | + { | |
| 2455 | + "capacity": 24, | |
| 2456 | + "policy": "lru", | |
| 2457 | + "predictor": "prev_token", | |
| 2458 | + "m": 8, | |
| 2459 | + "hit_rate": 0.7070294203747073, | |
| 2460 | + "misses_per_token": 37.50023419203747, | |
| 2461 | + "stall_ms_per_token": 16.12510070257611, | |
| 2462 | + "prefetch_loads_per_token": 0.0, | |
| 2463 | + "prefetch_precision": null, | |
| 2464 | + "loads_per_token": 37.50023419203747, | |
| 2465 | + "pred_recall_at_k": null, | |
| 2466 | + "tokens": 2135, | |
| 2467 | + "families": { | |
| 2468 | + "general": { | |
| 2469 | + "hit_rate": 0.6611776941362916, | |
| 2470 | + "stall_ms_per_token": 18.648779714738513 | |
| 2471 | + }, | |
| 2472 | + "code": { | |
| 2473 | + "hit_rate": 0.760868757431629, | |
| 2474 | + "stall_ms_per_token": 13.16178359096315 | |
| 2475 | + }, | |
| 2476 | + "reasoning": { | |
| 2477 | + "hit_rate": 0.6581835284280937, | |
| 2478 | + "stall_ms_per_token": 18.813578595317725 | |
| 2479 | + }, | |
| 2480 | + "multilingual": { | |
| 2481 | + "hit_rate": 0.7022450206043956, | |
| 2482 | + "stall_ms_per_token": 16.38843406593407 | |
| 2483 | + } | |
| 2484 | + }, | |
| 2485 | + "split": "fit" | |
| 2486 | + }, | |
| 2487 | + { | |
| 2488 | + "capacity": 24, | |
| 2489 | + "policy": "lru", | |
| 2490 | + "predictor": "prev_token", | |
| 2491 | + "m": 8, | |
| 2492 | + "hit_rate": 0.6954961964038727, | |
| 2493 | + "misses_per_token": 38.976486860304284, | |
| 2494 | + "stall_ms_per_token": 16.759889349930848, | |
| 2495 | + "prefetch_loads_per_token": 0.0, | |
| 2496 | + "prefetch_precision": null, | |
| 2497 | + "loads_per_token": 38.976486860304284, | |
| 2498 | + "pred_recall_at_k": null, | |
| 2499 | + "tokens": 1446, | |
| 2500 | + "families": { | |
| 2501 | + "general": { | |
| 2502 | + "hit_rate": 0.650830258302583, | |
| 2503 | + "stall_ms_per_token": 19.218302583025828 | |
| 2504 | + }, | |
| 2505 | + "code": { | |
| 2506 | + "hit_rate": 0.7484177215189873, | |
| 2507 | + "stall_ms_per_token": 13.847088607594944 | |
| 2508 | + }, | |
| 2509 | + "reasoning": { | |
| 2510 | + "hit_rate": 0.6880508814102564, | |
| 2511 | + "stall_ms_per_token": 17.1696794871795 | |
| 2512 | + }, | |
| 2513 | + "multilingual": { | |
| 2514 | + "hit_rate": 0.6254006410256411, | |
| 2515 | + "stall_ms_per_token": 20.617948717948714 | |
| 2516 | + } | |
| 2517 | + }, | |
| 2518 | + "split": "heldout" | |
| 2519 | + }, | |
| 2520 | + { | |
| 2521 | + "capacity": 24, | |
| 2522 | + "policy": "lru", | |
| 2523 | + "predictor": "prev_rank", | |
| 2524 | + "m": 8, | |
| 2525 | + "hit_rate": 0.7070221018735363, | |
| 2526 | + "misses_per_token": 37.50117096018735, | |
| 2527 | + "stall_ms_per_token": 16.12550351288056, | |
| 2528 | + "prefetch_loads_per_token": 0.022950819672131147, | |
| 2529 | + "prefetch_precision": 0.5102040816326531, | |
| 2530 | + "loads_per_token": 37.524121779859485, | |
| 2531 | + "pred_recall_at_k": null, | |
| 2532 | + "tokens": 2135, | |
| 2533 | + "families": { | |
| 2534 | + "general": { | |
| 2535 | + "hit_rate": 0.6611281695721077, | |
| 2536 | + "stall_ms_per_token": 18.65150554675119 | |
| 2537 | + }, | |
| 2538 | + "code": { | |
| 2539 | + "hit_rate": 0.7608966260404281, | |
| 2540 | + "stall_ms_per_token": 13.160249702734847 | |
| 2541 | + }, | |
| 2542 | + "reasoning": { | |
| 2543 | + "hit_rate": 0.6582357859531772, | |
| 2544 | + "stall_ms_per_token": 18.810702341137123 | |
| 2545 | + }, | |
| 2546 | + "multilingual": { | |
| 2547 | + "hit_rate": 0.7021806318681318, | |
| 2548 | + "stall_ms_per_token": 16.391978021978026 | |
| 2549 | + } | |
| 2550 | + }, | |
| 2551 | + "split": "fit" | |
| 2552 | + }, | |
| 2553 | + { | |
| 2554 | + "capacity": 24, | |
| 2555 | + "policy": "lru", | |
| 2556 | + "predictor": "prev_rank", | |
| 2557 | + "m": 8, | |
| 2558 | + "hit_rate": 0.6955015992392808, | |
| 2559 | + "misses_per_token": 38.97579529737206, | |
| 2560 | + "stall_ms_per_token": 16.759591977869988, | |
| 2561 | + "prefetch_loads_per_token": 0.01590594744121715, | |
| 2562 | + "prefetch_precision": 0.43478260869565216, | |
| 2563 | + "loads_per_token": 38.99170124481328, | |
| 2564 | + "pred_recall_at_k": null, | |
| 2565 | + "tokens": 1446, | |
| 2566 | + "families": { | |
| 2567 | + "general": { | |
| 2568 | + "hit_rate": 0.650830258302583, | |
| 2569 | + "stall_ms_per_token": 19.218302583025828 | |
| 2570 | + }, | |
| 2571 | + "code": { | |
| 2572 | + "hit_rate": 0.7484459764918626, | |
| 2573 | + "stall_ms_per_token": 13.845533453887894 | |
| 2574 | + }, | |
| 2575 | + "reasoning": { | |
| 2576 | + "hit_rate": 0.6880258413461539, | |
| 2577 | + "stall_ms_per_token": 17.171057692307702 | |
| 2578 | + }, | |
| 2579 | + "multilingual": { | |
| 2580 | + "hit_rate": 0.6254006410256411, | |
| 2581 | + "stall_ms_per_token": 20.617948717948714 | |
| 2582 | + } | |
| 2583 | + }, | |
| 2584 | + "split": "heldout" | |
| 2585 | + }, | |
| 2586 | + { | |
| 2587 | + "capacity": 24, | |
| 2588 | + "policy": "lru", | |
| 2589 | + "predictor": "transition", | |
| 2590 | + "m": 8, | |
| 2591 | + "hit_rate": 0.7639234484777517, | |
| 2592 | + "misses_per_token": 30.217798594847775, | |
| 2593 | + "stall_ms_per_token": 9.54910444964871, | |
| 2594 | + "prefetch_loads_per_token": 23.893208430913347, | |
| 2595 | + "prefetch_precision": 0.6019760056457304, | |
| 2596 | + "loads_per_token": 54.111007025761126, | |
| 2597 | + "pred_recall_at_k": 0.5302576112412178, | |
| 2598 | + "tokens": 2135, | |
| 2599 | + "families": { | |
| 2600 | + "general": { | |
| 2601 | + "hit_rate": 0.7315892432646592, | |
| 2602 | + "stall_ms_per_token": 10.808776545166417 | |
| 2603 | + }, | |
| 2604 | + "code": { | |
| 2605 | + "hit_rate": 0.801231792508918, | |
| 2606 | + "stall_ms_per_token": 8.121005945303212 | |
| 2607 | + }, | |
| 2608 | + "reasoning": { | |
| 2609 | + "hit_rate": 0.7362040133779264, | |
| 2610 | + "stall_ms_per_token": 10.62282274247492 | |
| 2611 | + }, | |
| 2612 | + "multilingual": { | |
| 2613 | + "hit_rate": 0.7565461881868132, | |
| 2614 | + "stall_ms_per_token": 9.782994505494505 | |
| 2615 | + } | |
| 2616 | + }, | |
| 2617 | + "split": "fit" | |
| 2618 | + }, | |
| 2619 | + { | |
| 2620 | + "capacity": 24, | |
| 2621 | + "policy": "lru", | |
| 2622 | + "predictor": "transition", | |
| 2623 | + "m": 8, | |
| 2624 | + "hit_rate": 0.750232321922545, | |
| 2625 | + "misses_per_token": 31.970262793914245, | |
| 2626 | + "stall_ms_per_token": 10.086200553250338, | |
| 2627 | + "prefetch_loads_per_token": 27.760027662517288, | |
| 2628 | + "prefetch_precision": 0.5727560349767071, | |
| 2629 | + "loads_per_token": 59.73029045643153, | |
| 2630 | + "pred_recall_at_k": 0.4996369294605809, | |
| 2631 | + "tokens": 1446, | |
| 2632 | + "families": { | |
| 2633 | + "general": { | |
| 2634 | + "hit_rate": 0.704206065498155, | |
| 2635 | + "stall_ms_per_token": 11.910077490774912 | |
| 2636 | + }, | |
| 2637 | + "code": { | |
| 2638 | + "hit_rate": 0.7957306735985533, | |
| 2639 | + "stall_ms_per_token": 8.295974683544312 | |
| 2640 | + }, | |
| 2641 | + "reasoning": { | |
| 2642 | + "hit_rate": 0.7528044871794872, | |
| 2643 | + "stall_ms_per_token": 9.966666666666677 | |
| 2644 | + }, | |
| 2645 | + "multilingual": { | |
| 2646 | + "hit_rate": 0.7241586538461539, | |
| 2647 | + "stall_ms_per_token": 11.079743589743588 | |
| 2648 | + } | |
| 2649 | + }, | |
| 2650 | + "split": "heldout" | |
| 2651 | + }, | |
| 2652 | + { | |
| 2653 | + "capacity": 24, | |
| 2654 | + "policy": "lru", | |
| 2655 | + "predictor": "probe", | |
| 2656 | + "m": 8, | |
| 2657 | + "hit_rate": 0.6538934426229508, | |
| 2658 | + "misses_per_token": 44.3016393442623, | |
| 2659 | + "stall_ms_per_token": 13.802424355971885, | |
| 2660 | + "prefetch_loads_per_token": 60.157377049180326, | |
| 2661 | + "prefetch_precision": 0.21040051076022298, | |
| 2662 | + "loads_per_token": 104.45901639344262, | |
| 2663 | + "pred_recall_at_k": 0.16113973458235753, | |
| 2664 | + "tokens": 2135, | |
| 2665 | + "families": { | |
| 2666 | + "general": { | |
| 2667 | + "hit_rate": 0.6240218898573693, | |
| 2668 | + "stall_ms_per_token": 14.966900158478596 | |
| 2669 | + }, | |
| 2670 | + "code": { | |
| 2671 | + "hit_rate": 0.6896551724137931, | |
| 2672 | + "stall_ms_per_token": 12.434111771700357 | |
| 2673 | + }, | |
| 2674 | + "reasoning": { | |
| 2675 | + "hit_rate": 0.6333612040133779, | |
| 2676 | + "stall_ms_per_token": 14.598314381270912 | |
| 2677 | + }, | |
| 2678 | + "multilingual": { | |
| 2679 | + "hit_rate": 0.6399167239010989, | |
| 2680 | + "stall_ms_per_token": 14.291423076923081 | |
| 2681 | + } | |
| 2682 | + }, | |
| 2683 | + "split": "fit" | |
| 2684 | + }, | |
| 2685 | + { | |
| 2686 | + "capacity": 24, | |
| 2687 | + "policy": "lru", | |
| 2688 | + "predictor": "probe", | |
| 2689 | + "m": 8, | |
| 2690 | + "hit_rate": 0.6433912517289073, | |
| 2691 | + "misses_per_token": 45.64591977869986, | |
| 2692 | + "stall_ms_per_token": 14.216248962655612, | |
| 2693 | + "prefetch_loads_per_token": 58.57192254495159, | |
| 2694 | + "prefetch_precision": 0.2115945451325344, | |
| 2695 | + "loads_per_token": 104.21784232365145, | |
| 2696 | + "pred_recall_at_k": 0.15494467496542186, | |
| 2697 | + "tokens": 1446, | |
| 2698 | + "families": { | |
| 2699 | + "general": { | |
| 2700 | + "hit_rate": 0.5802727167896679, | |
| 2701 | + "stall_ms_per_token": 16.700845018450195 | |
| 2702 | + }, | |
| 2703 | + "code": { | |
| 2704 | + "hit_rate": 0.701260171790235, | |
| 2705 | + "stall_ms_per_token": 11.947826401446653 | |
| 2706 | + }, | |
| 2707 | + "reasoning": { | |
| 2708 | + "hit_rate": 0.6516175881410257, | |
| 2709 | + "stall_ms_per_token": 13.878147435897441 | |
| 2710 | + }, | |
| 2711 | + "multilingual": { | |
| 2712 | + "hit_rate": 0.6342147435897436, | |
| 2713 | + "stall_ms_per_token": 14.556615384615387 | |
| 2714 | + } | |
| 2715 | + }, | |
| 2716 | + "split": "heldout" | |
| 2717 | + }, | |
| 2718 | + { | |
| 2719 | + "capacity": 24, | |
| 2720 | + "policy": "lru", | |
| 2721 | + "predictor": "oracle", | |
| 2722 | + "m": 8, | |
| 2723 | + "hit_rate": 0.9607362412177985, | |
| 2724 | + "misses_per_token": 5.02576112412178, | |
| 2725 | + "stall_ms_per_token": 1.9411091334894721, | |
| 2726 | + "prefetch_loads_per_token": 35.89086651053864, | |
| 2727 | + "prefetch_precision": 1.0, | |
| 2728 | + "loads_per_token": 40.91662763466042, | |
| 2729 | + "pred_recall_at_k": 1.0, | |
| 2730 | + "tokens": 2135, | |
| 2731 | + "families": { | |
| 2732 | + "general": { | |
| 2733 | + "hit_rate": 0.9572603011093502, | |
| 2734 | + "stall_ms_per_token": 2.0852361331220313 | |
| 2735 | + }, | |
| 2736 | + "code": { | |
| 2737 | + "hit_rate": 0.963408516646849, | |
| 2738 | + "stall_ms_per_token": 1.8519024970273514 | |
| 2739 | + }, | |
| 2740 | + "reasoning": { | |
| 2741 | + "hit_rate": 0.9593959030100334, | |
| 2742 | + "stall_ms_per_token": 1.9951170568561907 | |
| 2743 | + }, | |
| 2744 | + "multilingual": { | |
| 2745 | + "hit_rate": 0.9616887019230769, | |
| 2746 | + "stall_ms_per_token": 1.8530054945054975 | |
| 2747 | + } | |
| 2748 | + }, | |
| 2749 | + "split": "fit" | |
| 2750 | + }, | |
| 2751 | + { | |
| 2752 | + "capacity": 24, | |
| 2753 | + "policy": "lru", | |
| 2754 | + "predictor": "oracle", | |
| 2755 | + "m": 8, | |
| 2756 | + "hit_rate": 0.9588520055325035, | |
| 2757 | + "misses_per_token": 5.266943291839557, | |
| 2758 | + "stall_ms_per_token": 2.021798063623802, | |
| 2759 | + "prefetch_loads_per_token": 37.44882434301521, | |
| 2760 | + "prefetch_precision": 1.0, | |
| 2761 | + "loads_per_token": 42.71576763485477, | |
| 2762 | + "pred_recall_at_k": 1.0, | |
| 2763 | + "tokens": 1446, | |
| 2764 | + "families": { | |
| 2765 | + "general": { | |
| 2766 | + "hit_rate": 0.9509196263837638, | |
| 2767 | + "stall_ms_per_token": 2.3731180811808175 | |
| 2768 | + }, | |
| 2769 | + "code": { | |
| 2770 | + "hit_rate": 0.9655148056057866, | |
| 2771 | + "stall_ms_per_token": 1.732799276672697 | |
| 2772 | + }, | |
| 2773 | + "reasoning": { | |
| 2774 | + "hit_rate": 0.9602363782051282, | |
| 2775 | + "stall_ms_per_token": 1.9481794871794909 | |
| 2776 | + }, | |
| 2777 | + "multilingual": { | |
| 2778 | + "hit_rate": 0.9635416666666666, | |
| 2779 | + "stall_ms_per_token": 1.826153846153846 | |
| 2780 | + } | |
| 2781 | + }, | |
| 2782 | + "split": "heldout" | |
| 2783 | + }, | |
| 2784 | + { | |
| 2785 | + "capacity": 24, | |
| 2786 | + "policy": "lfu", | |
| 2787 | + "predictor": "none", | |
| 2788 | + "m": 8, | |
| 2789 | + "hit_rate": 0.7154786299765808, | |
| 2790 | + "misses_per_token": 36.41873536299766, | |
| 2791 | + "stall_ms_per_token": 15.66005620608898, | |
| 2792 | + "prefetch_loads_per_token": 0.0, | |
| 2793 | + "prefetch_precision": null, | |
| 2794 | + "loads_per_token": 36.41873536299766, | |
| 2795 | + "pred_recall_at_k": null, | |
| 2796 | + "tokens": 2135, | |
| 2797 | + "families": { | |
| 2798 | + "general": { | |
| 2799 | + "hit_rate": 0.6496879952456418, | |
| 2800 | + "stall_ms_per_token": 19.28117274167986 | |
| 2801 | + }, | |
| 2802 | + "code": { | |
| 2803 | + "hit_rate": 0.776280098097503, | |
| 2804 | + "stall_ms_per_token": 12.313543400713456 | |
| 2805 | + }, | |
| 2806 | + "reasoning": { | |
| 2807 | + "hit_rate": 0.6796352424749164, | |
| 2808 | + "stall_ms_per_token": 17.632876254180605 | |
| 2809 | + }, | |
| 2810 | + "multilingual": { | |
| 2811 | + "hit_rate": 0.718492445054945, | |
| 2812 | + "stall_ms_per_token": 15.494175824175837 | |
| 2813 | + } | |
| 2814 | + }, | |
| 2815 | + "split": "fit" | |
| 2816 | + }, | |
| 2817 | + { | |
| 2818 | + "capacity": 24, | |
| 2819 | + "policy": "lfu", | |
| 2820 | + "predictor": "none", | |
| 2821 | + "m": 8, | |
| 2822 | + "hit_rate": 0.713606500691563, | |
| 2823 | + "misses_per_token": 36.65836791147994, | |
| 2824 | + "stall_ms_per_token": 15.76309820193638, | |
| 2825 | + "prefetch_loads_per_token": 0.0, | |
| 2826 | + "prefetch_precision": null, | |
| 2827 | + "loads_per_token": 36.65836791147994, | |
| 2828 | + "pred_recall_at_k": null, | |
| 2829 | + "tokens": 1446, | |
| 2830 | + "families": { | |
| 2831 | + "general": { | |
| 2832 | + "hit_rate": 0.6669453413284133, | |
| 2833 | + "stall_ms_per_token": 18.331328413284137 | |
| 2834 | + }, | |
| 2835 | + "code": { | |
| 2836 | + "hit_rate": 0.766500904159132, | |
| 2837 | + "stall_ms_per_token": 12.85179023508138 | |
| 2838 | + }, | |
| 2839 | + "reasoning": { | |
| 2840 | + "hit_rate": 0.7100861378205128, | |
| 2841 | + "stall_ms_per_token": 15.956858974358962 | |
| 2842 | + }, | |
| 2843 | + "multilingual": { | |
| 2844 | + "hit_rate": 0.6402243589743589, | |
| 2845 | + "stall_ms_per_token": 19.802051282051277 | |
| 2846 | + } | |
| 2847 | + }, | |
| 2848 | + "split": "heldout" | |
| 2849 | + }, | |
| 2850 | + { | |
| 2851 | + "capacity": 24, | |
| 2852 | + "policy": "lfu", | |
| 2853 | + "predictor": "prev_token", | |
| 2854 | + "m": 8, | |
| 2855 | + "hit_rate": 0.7154786299765808, | |
| 2856 | + "misses_per_token": 36.41873536299766, | |
| 2857 | + "stall_ms_per_token": 15.66005620608898, | |
| 2858 | + "prefetch_loads_per_token": 0.0, | |
| 2859 | + "prefetch_precision": null, | |
| 2860 | + "loads_per_token": 36.41873536299766, | |
| 2861 | + "pred_recall_at_k": null, | |
| 2862 | + "tokens": 2135, | |
| 2863 | + "families": { | |
| 2864 | + "general": { | |
| 2865 | + "hit_rate": 0.6496879952456418, | |
| 2866 | + "stall_ms_per_token": 19.28117274167986 | |
| 2867 | + }, | |
| 2868 | + "code": { | |
| 2869 | + "hit_rate": 0.776280098097503, | |
| 2870 | + "stall_ms_per_token": 12.313543400713456 | |
| 2871 | + }, | |
| 2872 | + "reasoning": { | |
| 2873 | + "hit_rate": 0.6796352424749164, | |
| 2874 | + "stall_ms_per_token": 17.632876254180605 | |
| 2875 | + }, | |
| 2876 | + "multilingual": { | |
| 2877 | + "hit_rate": 0.718492445054945, | |
| 2878 | + "stall_ms_per_token": 15.494175824175837 | |
| 2879 | + } | |
| 2880 | + }, | |
| 2881 | + "split": "fit" | |
| 2882 | + }, | |
| 2883 | + { | |
| 2884 | + "capacity": 24, | |
| 2885 | + "policy": "lfu", | |
| 2886 | + "predictor": "prev_token", | |
| 2887 | + "m": 8, | |
| 2888 | + "hit_rate": 0.713606500691563, | |
| 2889 | + "misses_per_token": 36.65836791147994, | |
| 2890 | + "stall_ms_per_token": 15.76309820193638, | |
| 2891 | + "prefetch_loads_per_token": 0.0, | |
| 2892 | + "prefetch_precision": null, | |
| 2893 | + "loads_per_token": 36.65836791147994, | |
| 2894 | + "pred_recall_at_k": null, | |
| 2895 | + "tokens": 1446, | |
| 2896 | + "families": { | |
| 2897 | + "general": { | |
| 2898 | + "hit_rate": 0.6669453413284133, | |
| 2899 | + "stall_ms_per_token": 18.331328413284137 | |
| 2900 | + }, | |
| 2901 | + "code": { | |
| 2902 | + "hit_rate": 0.766500904159132, | |
| 2903 | + "stall_ms_per_token": 12.85179023508138 | |
| 2904 | + }, | |
| 2905 | + "reasoning": { | |
| 2906 | + "hit_rate": 0.7100861378205128, | |
| 2907 | + "stall_ms_per_token": 15.956858974358962 | |
| 2908 | + }, | |
| 2909 | + "multilingual": { | |
| 2910 | + "hit_rate": 0.6402243589743589, | |
| 2911 | + "stall_ms_per_token": 19.802051282051277 | |
| 2912 | + } | |
| 2913 | + }, | |
| 2914 | + "split": "heldout" | |
| 2915 | + }, | |
| 2916 | + { | |
| 2917 | + "capacity": 24, | |
| 2918 | + "policy": "lfu", | |
| 2919 | + "predictor": "prev_rank", | |
| 2920 | + "m": 8, | |
| 2921 | + "hit_rate": 0.7154530152224824, | |
| 2922 | + "misses_per_token": 36.42201405152225, | |
| 2923 | + "stall_ms_per_token": 15.661466042154554, | |
| 2924 | + "prefetch_loads_per_token": 0.021545667447306793, | |
| 2925 | + "prefetch_precision": 0.15217391304347827, | |
| 2926 | + "loads_per_token": 36.44355971896955, | |
| 2927 | + "pred_recall_at_k": null, | |
| 2928 | + "tokens": 2135, | |
| 2929 | + "families": { | |
| 2930 | + "general": { | |
| 2931 | + "hit_rate": 0.6496508518225039, | |
| 2932 | + "stall_ms_per_token": 19.283217115689368 | |
| 2933 | + }, | |
| 2934 | + "code": { | |
| 2935 | + "hit_rate": 0.7762615190249703, | |
| 2936 | + "stall_ms_per_token": 12.31456599286566 | |
| 2937 | + }, | |
| 2938 | + "reasoning": { | |
| 2939 | + "hit_rate": 0.6796352424749164, | |
| 2940 | + "stall_ms_per_token": 17.632876254180605 | |
| 2941 | + }, | |
| 2942 | + "multilingual": { | |
| 2943 | + "hit_rate": 0.7184495192307693, | |
| 2944 | + "stall_ms_per_token": 15.496538461538476 | |
| 2945 | + } | |
| 2946 | + }, | |
| 2947 | + "split": "fit" | |
| 2948 | + }, | |
| 2949 | + { | |
| 2950 | + "capacity": 24, | |
| 2951 | + "policy": "lfu", | |
| 2952 | + "predictor": "prev_rank", | |
| 2953 | + "m": 8, | |
| 2954 | + "hit_rate": 0.7136119035269709, | |
| 2955 | + "misses_per_token": 36.65767634854772, | |
| 2956 | + "stall_ms_per_token": 15.762800829875525, | |
| 2957 | + "prefetch_loads_per_token": 0.014522821576763486, | |
| 2958 | + "prefetch_precision": 0.23809523809523808, | |
| 2959 | + "loads_per_token": 36.67219917012448, | |
| 2960 | + "pred_recall_at_k": null, | |
| 2961 | + "tokens": 1446, | |
| 2962 | + "families": { | |
| 2963 | + "general": { | |
| 2964 | + "hit_rate": 0.6669309271217713, | |
| 2965 | + "stall_ms_per_token": 18.332121771217714 | |
| 2966 | + }, | |
| 2967 | + "code": { | |
| 2968 | + "hit_rate": 0.766500904159132, | |
| 2969 | + "stall_ms_per_token": 12.85179023508138 | |
| 2970 | + }, | |
| 2971 | + "reasoning": { | |
| 2972 | + "hit_rate": 0.710136217948718, | |
| 2973 | + "stall_ms_per_token": 15.95410256410255 | |
| 2974 | + }, | |
| 2975 | + "multilingual": { | |
| 2976 | + "hit_rate": 0.6402243589743589, | |
| 2977 | + "stall_ms_per_token": 19.802051282051277 | |
| 2978 | + } | |
| 2979 | + }, | |
| 2980 | + "split": "heldout" | |
| 2981 | + }, | |
| 2982 | + { | |
| 2983 | + "capacity": 24, | |
| 2984 | + "policy": "lfu", | |
| 2985 | + "predictor": "transition", | |
| 2986 | + "m": 8, | |
| 2987 | + "hit_rate": 0.7307852751756441, | |
| 2988 | + "misses_per_token": 34.45948477751757, | |
| 2989 | + "stall_ms_per_token": 10.828235128805654, | |
| 2990 | + "prefetch_loads_per_token": 26.777049180327868, | |
| 2991 | + "prefetch_precision": 0.23014220993895293, | |
| 2992 | + "loads_per_token": 61.23653395784543, | |
| 2993 | + "pred_recall_at_k": 0.5302576112412178, | |
| 2994 | + "tokens": 2135, | |
| 2995 | + "families": { | |
| 2996 | + "general": { | |
| 2997 | + "hit_rate": 0.6554452258320127, | |
| 2998 | + "stall_ms_per_token": 13.757879556259907 | |
| 2999 | + }, | |
| 3000 | + "code": { | |
| 3001 | + "hit_rate": 0.796103968489893, | |
| 3002 | + "stall_ms_per_token": 8.323488703923893 | |
| 3003 | + }, | |
| 3004 | + "reasoning": { | |
| 3005 | + "hit_rate": 0.707305602006689, | |
| 3006 | + "stall_ms_per_token": 11.74034782608696 | |
| 3007 | + }, | |
| 3008 | + "multilingual": { | |
| 3009 | + "hit_rate": 0.7297604739010989, | |
| 3010 | + "stall_ms_per_token": 10.787478021978023 | |
| 3011 | + } | |
| 3012 | + }, | |
| 3013 | + "split": "fit" | |
| 3014 | + }, | |
| 3015 | + { | |
| 3016 | + "capacity": 24, | |
| 3017 | + "policy": "lfu", | |
| 3018 | + "predictor": "transition", | |
| 3019 | + "m": 8, | |
| 3020 | + "hit_rate": 0.7174479166666666, | |
| 3021 | + "misses_per_token": 36.166666666666664, | |
| 3022 | + "stall_ms_per_token": 11.321116182572629, | |
| 3023 | + "prefetch_loads_per_token": 30.498616874135546, | |
| 3024 | + "prefetch_precision": 0.20303394480850775, | |
| 3025 | + "loads_per_token": 66.6652835408022, | |
| 3026 | + "pred_recall_at_k": 0.4996369294605809, | |
| 3027 | + "tokens": 1446, | |
| 3028 | + "families": { | |
| 3029 | + "general": { | |
| 3030 | + "hit_rate": 0.6422393911439115, | |
| 3031 | + "stall_ms_per_token": 14.27664944649446 | |
| 3032 | + }, | |
| 3033 | + "code": { | |
| 3034 | + "hit_rate": 0.7854882459312839, | |
| 3035 | + "stall_ms_per_token": 8.659037974683542 | |
| 3036 | + }, | |
| 3037 | + "reasoning": { | |
| 3038 | + "hit_rate": 0.7321965144230769, | |
| 3039 | + "stall_ms_per_token": 10.725134615384613 | |
| 3040 | + }, | |
| 3041 | + "multilingual": { | |
| 3042 | + "hit_rate": 0.6798878205128205, | |
| 3043 | + "stall_ms_per_token": 12.76153846153846 | |
| 3044 | + } | |
| 3045 | + }, | |
| 3046 | + "split": "heldout" | |
| 3047 | + }, | |
| 3048 | + { | |
| 3049 | + "capacity": 24, | |
| 3050 | + "policy": "lfu", | |
| 3051 | + "predictor": "probe", | |
| 3052 | + "m": 8, | |
| 3053 | + "hit_rate": 0.6562792740046839, | |
| 3054 | + "misses_per_token": 43.99625292740047, | |
| 3055 | + "stall_ms_per_token": 13.708339110070225, | |
| 3056 | + "prefetch_loads_per_token": 71.4791569086651, | |
| 3057 | + "prefetch_precision": 0.06976043195638498, | |
| 3058 | + "loads_per_token": 115.47540983606558, | |
| 3059 | + "pred_recall_at_k": 0.16113973458235753, | |
| 3060 | + "tokens": 2135, | |
| 3061 | + "families": { | |
| 3062 | + "general": { | |
| 3063 | + "hit_rate": 0.5887356378763867, | |
| 3064 | + "stall_ms_per_token": 16.33660538827258 | |
| 3065 | + }, | |
| 3066 | + "code": { | |
| 3067 | + "hit_rate": 0.7123495095124851, | |
| 3068 | + "stall_ms_per_token": 11.56110107015458 | |
| 3069 | + }, | |
| 3070 | + "reasoning": { | |
| 3071 | + "hit_rate": 0.6269073996655519, | |
| 3072 | + "stall_ms_per_token": 14.84822073578596 | |
| 3073 | + }, | |
| 3074 | + "multilingual": { | |
| 3075 | + "hit_rate": 0.6679472870879121, | |
| 3076 | + "stall_ms_per_token": 13.176928571428556 | |
| 3077 | + } | |
| 3078 | + }, | |
| 3079 | + "split": "fit" | |
| 3080 | + }, | |
| 3081 | + { | |
| 3082 | + "capacity": 24, | |
| 3083 | + "policy": "lfu", | |
| 3084 | + "predictor": "probe", | |
| 3085 | + "m": 8, | |
| 3086 | + "hit_rate": 0.6465681189488244, | |
| 3087 | + "misses_per_token": 45.23928077455049, | |
| 3088 | + "stall_ms_per_token": 14.06104564315352, | |
| 3089 | + "prefetch_loads_per_token": 71.41977869986168, | |
| 3090 | + "prefetch_precision": 0.0650896168408006, | |
| 3091 | + "loads_per_token": 116.65905947441217, | |
| 3092 | + "pred_recall_at_k": 0.15494467496542186, | |
| 3093 | + "tokens": 1446, | |
| 3094 | + "families": { | |
| 3095 | + "general": { | |
| 3096 | + "hit_rate": 0.5816997232472325, | |
| 3097 | + "stall_ms_per_token": 16.616870848708473 | |
| 3098 | + }, | |
| 3099 | + "code": { | |
| 3100 | + "hit_rate": 0.7109657549728752, | |
| 3101 | + "stall_ms_per_token": 11.539779385171785 | |
| 3102 | + }, | |
| 3103 | + "reasoning": { | |
| 3104 | + "hit_rate": 0.650215344551282, | |
| 3105 | + "stall_ms_per_token": 13.894198717948724 | |
| 3106 | + }, | |
| 3107 | + "multilingual": { | |
| 3108 | + "hit_rate": 0.6057692307692307, | |
| 3109 | + "stall_ms_per_token": 15.626666666666669 | |
| 3110 | + } | |
| 3111 | + }, | |
| 3112 | + "split": "heldout" | |
| 3113 | + }, | |
| 3114 | + { | |
| 3115 | + "capacity": 24, | |
| 3116 | + "policy": "lfu", | |
| 3117 | + "predictor": "oracle", | |
| 3118 | + "m": 8, | |
| 3119 | + "hit_rate": 0.84153981264637, | |
| 3120 | + "misses_per_token": 20.28290398126464, | |
| 3121 | + "stall_ms_per_token": 6.546907728337246, | |
| 3122 | + "prefetch_loads_per_token": 35.13255269320843, | |
| 3123 | + "prefetch_precision": 0.5712190699658704, | |
| 3124 | + "loads_per_token": 55.41545667447307, | |
| 3125 | + "pred_recall_at_k": 1.0, | |
| 3126 | + "tokens": 2135, | |
| 3127 | + "families": { | |
| 3128 | + "general": { | |
| 3129 | + "hit_rate": 0.7976426307448494, | |
| 3130 | + "stall_ms_per_token": 8.26109667194929 | |
| 3131 | + }, | |
| 3132 | + "code": { | |
| 3133 | + "hit_rate": 0.879013079667063, | |
| 3134 | + "stall_ms_per_token": 5.118554102259215 | |
| 3135 | + }, | |
| 3136 | + "reasoning": { | |
| 3137 | + "hit_rate": 0.8252247073578596, | |
| 3138 | + "stall_ms_per_token": 7.182066889632114 | |
| 3139 | + }, | |
| 3140 | + "multilingual": { | |
| 3141 | + "hit_rate": 0.8444582760989011, | |
| 3142 | + "stall_ms_per_token": 6.35371978021978 | |
| 3143 | + } | |
| 3144 | + }, | |
| 3145 | + "split": "fit" | |
| 3146 | + }, | |
| 3147 | + { | |
| 3148 | + "capacity": 24, | |
| 3149 | + "policy": "lfu", | |
| 3150 | + "predictor": "oracle", | |
| 3151 | + "m": 8, | |
| 3152 | + "hit_rate": 0.837650198824343, | |
| 3153 | + "misses_per_token": 20.780774550484093, | |
| 3154 | + "stall_ms_per_token": 6.674576763485479, | |
| 3155 | + "prefetch_loads_per_token": 35.98063623789765, | |
| 3156 | + "prefetch_precision": 0.5675021142461751, | |
| 3157 | + "loads_per_token": 56.761410788381745, | |
| 3158 | + "pred_recall_at_k": 1.0, | |
| 3159 | + "tokens": 1446, | |
| 3160 | + "families": { | |
| 3161 | + "general": { | |
| 3162 | + "hit_rate": 0.799901983394834, | |
| 3163 | + "stall_ms_per_token": 8.182044280442806 | |
| 3164 | + }, | |
| 3165 | + "code": { | |
| 3166 | + "hit_rate": 0.8739969484629295, | |
| 3167 | + "stall_ms_per_token": 5.23764556962026 | |
| 3168 | + }, | |
| 3169 | + "reasoning": { | |
| 3170 | + "hit_rate": 0.8408453525641025, | |
| 3171 | + "stall_ms_per_token": 6.5252051282051315 | |
| 3172 | + }, | |
| 3173 | + "multilingual": { | |
| 3174 | + "hit_rate": 0.8213141025641025, | |
| 3175 | + "stall_ms_per_token": 7.294564102564102 | |
| 3176 | + } | |
| 3177 | + }, | |
| 3178 | + "split": "heldout" | |
| 3179 | + }, | |
| 3180 | + { | |
| 3181 | + "capacity": 32, | |
| 3182 | + "policy": "lru", | |
| 3183 | + "predictor": "none", | |
| 3184 | + "m": 8, | |
| 3185 | + "hit_rate": 0.8022394613583138, | |
| 3186 | + "misses_per_token": 25.313348946135832, | |
| 3187 | + "stall_ms_per_token": 10.884740046838395, | |
| 3188 | + "prefetch_loads_per_token": 0.0, | |
| 3189 | + "prefetch_precision": null, | |
| 3190 | + "loads_per_token": 25.313348946135832, | |
| 3191 | + "pred_recall_at_k": null, | |
| 3192 | + "tokens": 2135, | |
| 3193 | + "families": { | |
| 3194 | + "general": { | |
| 3195 | + "hit_rate": 0.7655878565768621, | |
| 3196 | + "stall_ms_per_token": 12.90204437400952 | |
| 3197 | + }, | |
| 3198 | + "code": { | |
| 3199 | + "hit_rate": 0.8442516349583828, | |
| 3200 | + "stall_ms_per_token": 8.572390011890633 | |
| 3201 | + }, | |
| 3202 | + "reasoning": { | |
| 3203 | + "hit_rate": 0.767375627090301, | |
| 3204 | + "stall_ms_per_token": 12.80364548494984 | |
| 3205 | + }, | |
| 3206 | + "multilingual": { | |
| 3207 | + "hit_rate": 0.7973471840659341, | |
| 3208 | + "stall_ms_per_token": 11.154010989010997 | |
| 3209 | + } | |
| 3210 | + }, | |
| 3211 | + "split": "fit" | |
| 3212 | + }, | |
| 3213 | + { | |
| 3214 | + "capacity": 32, | |
| 3215 | + "policy": "lru", | |
| 3216 | + "predictor": "none", | |
| 3217 | + "m": 8, | |
| 3218 | + "hit_rate": 0.7992576504149378, | |
| 3219 | + "misses_per_token": 25.695020746887966, | |
| 3220 | + "stall_ms_per_token": 11.048858921161798, | |
| 3221 | + "prefetch_loads_per_token": 0.0, | |
| 3222 | + "prefetch_precision": null, | |
| 3223 | + "loads_per_token": 25.695020746887966, | |
| 3224 | + "pred_recall_at_k": null, | |
| 3225 | + "tokens": 1446, | |
| 3226 | + "families": { | |
| 3227 | + "general": { | |
| 3228 | + "hit_rate": 0.772803274907749, | |
| 3229 | + "stall_ms_per_token": 12.5049077490775 | |
| 3230 | + }, | |
| 3231 | + "code": { | |
| 3232 | + "hit_rate": 0.8352169981916817, | |
| 3233 | + "stall_ms_per_token": 9.069656419529856 | |
| 3234 | + }, | |
| 3235 | + "reasoning": { | |
| 3236 | + "hit_rate": 0.7933693910256411, | |
| 3237 | + "stall_ms_per_token": 11.372948717948718 | |
| 3238 | + }, | |
| 3239 | + "multilingual": { | |
| 3240 | + "hit_rate": 0.7041266025641025, | |
| 3241 | + "stall_ms_per_token": 16.284871794871794 | |
| 3242 | + } | |
| 3243 | + }, | |
| 3244 | + "split": "heldout" | |
| 3245 | + }, | |
| 3246 | + { | |
| 3247 | + "capacity": 32, | |
| 3248 | + "policy": "lru", | |
| 3249 | + "predictor": "prev_token", | |
| 3250 | + "m": 8, | |
| 3251 | + "hit_rate": 0.8022394613583138, | |
| 3252 | + "misses_per_token": 25.313348946135832, | |
| 3253 | + "stall_ms_per_token": 10.884740046838395, | |
| 3254 | + "prefetch_loads_per_token": 0.0, | |
| 3255 | + "prefetch_precision": null, | |
| 3256 | + "loads_per_token": 25.313348946135832, | |
| 3257 | + "pred_recall_at_k": null, | |
| 3258 | + "tokens": 2135, | |
| 3259 | + "families": { | |
| 3260 | + "general": { | |
| 3261 | + "hit_rate": 0.7655878565768621, | |
| 3262 | + "stall_ms_per_token": 12.90204437400952 | |
| 3263 | + }, | |
| 3264 | + "code": { | |
| 3265 | + "hit_rate": 0.8442516349583828, | |
| 3266 | + "stall_ms_per_token": 8.572390011890633 | |
| 3267 | + }, | |
| 3268 | + "reasoning": { | |
| 3269 | + "hit_rate": 0.767375627090301, | |
| 3270 | + "stall_ms_per_token": 12.80364548494984 | |
| 3271 | + }, | |
| 3272 | + "multilingual": { | |
| 3273 | + "hit_rate": 0.7973471840659341, | |
| 3274 | + "stall_ms_per_token": 11.154010989010997 | |
| 3275 | + } | |
| 3276 | + }, | |
| 3277 | + "split": "fit" | |
| 3278 | + }, | |
| 3279 | + { | |
| 3280 | + "capacity": 32, | |
| 3281 | + "policy": "lru", | |
| 3282 | + "predictor": "prev_token", | |
| 3283 | + "m": 8, | |
| 3284 | + "hit_rate": 0.7992576504149378, | |
| 3285 | + "misses_per_token": 25.695020746887966, | |
| 3286 | + "stall_ms_per_token": 11.048858921161798, | |
| 3287 | + "prefetch_loads_per_token": 0.0, | |
| 3288 | + "prefetch_precision": null, | |
| 3289 | + "loads_per_token": 25.695020746887966, | |
| 3290 | + "pred_recall_at_k": null, | |
| 3291 | + "tokens": 1446, | |
| 3292 | + "families": { | |
| 3293 | + "general": { | |
| 3294 | + "hit_rate": 0.772803274907749, | |
| 3295 | + "stall_ms_per_token": 12.5049077490775 | |
| 3296 | + }, | |
| 3297 | + "code": { | |
| 3298 | + "hit_rate": 0.8352169981916817, | |
| 3299 | + "stall_ms_per_token": 9.069656419529856 | |
| 3300 | + }, | |
| 3301 | + "reasoning": { | |
| 3302 | + "hit_rate": 0.7933693910256411, | |
| 3303 | + "stall_ms_per_token": 11.372948717948718 | |
| 3304 | + }, | |
| 3305 | + "multilingual": { | |
| 3306 | + "hit_rate": 0.7041266025641025, | |
| 3307 | + "stall_ms_per_token": 16.284871794871794 | |
| 3308 | + } | |
| 3309 | + }, | |
| 3310 | + "split": "heldout" | |
| 3311 | + }, | |
| 3312 | + { | |
| 3313 | + "capacity": 32, | |
| 3314 | + "policy": "lru", | |
| 3315 | + "predictor": "prev_rank", | |
| 3316 | + "m": 8, | |
| 3317 | + "hit_rate": 0.8022431206088994, | |
| 3318 | + "misses_per_token": 25.31288056206089, | |
| 3319 | + "stall_ms_per_token": 10.884538641686168, | |
| 3320 | + "prefetch_loads_per_token": 0.01592505854800937, | |
| 3321 | + "prefetch_precision": 0.5588235294117647, | |
| 3322 | + "loads_per_token": 25.3288056206089, | |
| 3323 | + "pred_recall_at_k": null, | |
| 3324 | + "tokens": 2135, | |
| 3325 | + "families": { | |
| 3326 | + "general": { | |
| 3327 | + "hit_rate": 0.7655878565768621, | |
| 3328 | + "stall_ms_per_token": 12.902044374009517 | |
| 3329 | + }, | |
| 3330 | + "code": { | |
| 3331 | + "hit_rate": 0.8442330558858502, | |
| 3332 | + "stall_ms_per_token": 8.573412604042831 | |
| 3333 | + }, | |
| 3334 | + "reasoning": { | |
| 3335 | + "hit_rate": 0.7674278846153846, | |
| 3336 | + "stall_ms_per_token": 12.80076923076924 | |
| 3337 | + }, | |
| 3338 | + "multilingual": { | |
| 3339 | + "hit_rate": 0.797368646978022, | |
| 3340 | + "stall_ms_per_token": 11.152829670329675 | |
| 3341 | + } | |
| 3342 | + }, | |
| 3343 | + "split": "fit" | |
| 3344 | + }, | |
| 3345 | + { | |
| 3346 | + "capacity": 32, | |
| 3347 | + "policy": "lru", | |
| 3348 | + "predictor": "prev_rank", | |
| 3349 | + "m": 8, | |
| 3350 | + "hit_rate": 0.7991982192254495, | |
| 3351 | + "misses_per_token": 25.702627939142463, | |
| 3352 | + "stall_ms_per_token": 11.052130013831231, | |
| 3353 | + "prefetch_loads_per_token": 0.011065006915629323, | |
| 3354 | + "prefetch_precision": 0.3125, | |
| 3355 | + "loads_per_token": 25.713692946058092, | |
| 3356 | + "pred_recall_at_k": null, | |
| 3357 | + "tokens": 1446, | |
| 3358 | + "families": { | |
| 3359 | + "general": { | |
| 3360 | + "hit_rate": 0.7727167896678967, | |
| 3361 | + "stall_ms_per_token": 12.509667896678977 | |
| 3362 | + }, | |
| 3363 | + "code": { | |
| 3364 | + "hit_rate": 0.8352028707052441, | |
| 3365 | + "stall_ms_per_token": 9.070433996383382 | |
| 3366 | + }, | |
| 3367 | + "reasoning": { | |
| 3368 | + "hit_rate": 0.7932692307692307, | |
| 3369 | + "stall_ms_per_token": 11.37846153846154 | |
| 3370 | + }, | |
| 3371 | + "multilingual": { | |
| 3372 | + "hit_rate": 0.7041266025641025, | |
| 3373 | + "stall_ms_per_token": 16.284871794871794 | |
| 3374 | + } | |
| 3375 | + }, | |
| 3376 | + "split": "heldout" | |
| 3377 | + }, | |
| 3378 | + { | |
| 3379 | + "capacity": 32, | |
| 3380 | + "policy": "lru", | |
| 3381 | + "predictor": "transition", | |
| 3382 | + "m": 8, | |
| 3383 | + "hit_rate": 0.8338334309133489, | |
| 3384 | + "misses_per_token": 21.269320843091336, | |
| 3385 | + "stall_ms_per_token": 6.738148946135845, | |
| 3386 | + "prefetch_loads_per_token": 13.986416861826697, | |
| 3387 | + "prefetch_precision": 0.656876862797629, | |
| 3388 | + "loads_per_token": 35.25573770491803, | |
| 3389 | + "pred_recall_at_k": 0.5302576112412178, | |
| 3390 | + "tokens": 2135, | |
| 3391 | + "families": { | |
| 3392 | + "general": { | |
| 3393 | + "hit_rate": 0.8059751386687797, | |
| 3394 | + "stall_ms_per_token": 7.82864342313786 | |
| 3395 | + }, | |
| 3396 | + "code": { | |
| 3397 | + "hit_rate": 0.8655618311533888, | |
| 3398 | + "stall_ms_per_token": 5.525897740784781 | |
| 3399 | + }, | |
| 3400 | + "reasoning": { | |
| 3401 | + "hit_rate": 0.8055497491638796, | |
| 3402 | + "stall_ms_per_token": 7.833458193979932 | |
| 3403 | + }, | |
| 3404 | + "multilingual": { | |
| 3405 | + "hit_rate": 0.8320527129120879, | |
| 3406 | + "stall_ms_per_token": 6.748873626373629 | |
| 3407 | + } | |
| 3408 | + }, | |
| 3409 | + "split": "fit" | |
| 3410 | + }, | |
| 3411 | + { | |
| 3412 | + "capacity": 32, | |
| 3413 | + "policy": "lru", | |
| 3414 | + "predictor": "transition", | |
| 3415 | + "m": 8, | |
| 3416 | + "hit_rate": 0.824953535615491, | |
| 3417 | + "misses_per_token": 22.40594744121715, | |
| 3418 | + "stall_ms_per_token": 7.086684647302896, | |
| 3419 | + "prefetch_loads_per_token": 16.033195020746888, | |
| 3420 | + "prefetch_precision": 0.6192632850241546, | |
| 3421 | + "loads_per_token": 38.43914246196404, | |
| 3422 | + "pred_recall_at_k": 0.4996369294605809, | |
| 3423 | + "tokens": 1446, | |
| 3424 | + "families": { | |
| 3425 | + "general": { | |
| 3426 | + "hit_rate": 0.7941074723247232, | |
| 3427 | + "stall_ms_per_token": 8.30873800738008 | |
| 3428 | + }, | |
| 3429 | + "code": { | |
| 3430 | + "hit_rate": 0.8585697332730561, | |
| 3431 | + "stall_ms_per_token": 5.757153707052442 | |
| 3432 | + }, | |
| 3433 | + "reasoning": { | |
| 3434 | + "hit_rate": 0.8236177884615384, | |
| 3435 | + "stall_ms_per_token": 7.140692307692305 | |
| 3436 | + }, | |
| 3437 | + "multilingual": { | |
| 3438 | + "hit_rate": 0.7876602564102564, | |
| 3439 | + "stall_ms_per_token": 8.523282051282052 | |
| 3440 | + } | |
| 3441 | + }, | |
| 3442 | + "split": "heldout" | |
| 3443 | + }, | |
| 3444 | + { | |
| 3445 | + "capacity": 32, | |
| 3446 | + "policy": "lru", | |
| 3447 | + "predictor": "probe", | |
| 3448 | + "m": 8, | |
| 3449 | + "hit_rate": 0.756798887587822, | |
| 3450 | + "misses_per_token": 31.129742388758782, | |
| 3451 | + "stall_ms_per_token": 9.715996252927422, | |
| 3452 | + "prefetch_loads_per_token": 44.34894613583138, | |
| 3453 | + "prefetch_precision": 0.2745735861012832, | |
| 3454 | + "loads_per_token": 75.47868852459017, | |
| 3455 | + "pred_recall_at_k": 0.16113973458235753, | |
| 3456 | + "tokens": 2135, | |
| 3457 | + "families": { | |
| 3458 | + "general": { | |
| 3459 | + "hit_rate": 0.7336568938193344, | |
| 3460 | + "stall_ms_per_token": 10.624177496038032 | |
| 3461 | + }, | |
| 3462 | + "code": { | |
| 3463 | + "hit_rate": 0.7823275862068966, | |
| 3464 | + "stall_ms_per_token": 8.743400713436374 | |
| 3465 | + }, | |
| 3466 | + "reasoning": { | |
| 3467 | + "hit_rate": 0.731814381270903, | |
| 3468 | + "stall_ms_per_token": 10.683772575250838 | |
| 3469 | + }, | |
| 3470 | + "multilingual": { | |
| 3471 | + "hit_rate": 0.7584563873626373, | |
| 3472 | + "stall_ms_per_token": 9.593813186813193 | |
| 3473 | + } | |
| 3474 | + }, | |
| 3475 | + "split": "fit" | |
| 3476 | + }, | |
| 3477 | + { | |
| 3478 | + "capacity": 32, | |
| 3479 | + "policy": "lru", | |
| 3480 | + "predictor": "probe", | |
| 3481 | + "m": 8, | |
| 3482 | + "hit_rate": 0.7485952627939142, | |
| 3483 | + "misses_per_token": 32.179806362378976, | |
| 3484 | + "stall_ms_per_token": 10.038390041493777, | |
| 3485 | + "prefetch_loads_per_token": 42.50829875518672, | |
| 3486 | + "prefetch_precision": 0.2789952332145704, | |
| 3487 | + "loads_per_token": 74.6881051175657, | |
| 3488 | + "pred_recall_at_k": 0.15494467496542186, | |
| 3489 | + "tokens": 1446, | |
| 3490 | + "families": { | |
| 3491 | + "general": { | |
| 3492 | + "hit_rate": 0.7069159363468634, | |
| 3493 | + "stall_ms_per_token": 11.679214022140215 | |
| 3494 | + }, | |
| 3495 | + "code": { | |
| 3496 | + "hit_rate": 0.7893450497287523, | |
| 3497 | + "stall_ms_per_token": 8.433103074141053 | |
| 3498 | + }, | |
| 3499 | + "reasoning": { | |
| 3500 | + "hit_rate": 0.7512269631410257, | |
| 3501 | + "stall_ms_per_token": 9.939032051282064 | |
| 3502 | + }, | |
| 3503 | + "multilingual": { | |
| 3504 | + "hit_rate": 0.7289663461538461, | |
| 3505 | + "stall_ms_per_token": 10.792153846153846 | |
| 3506 | + } | |
| 3507 | + }, | |
| 3508 | + "split": "heldout" | |
| 3509 | + }, | |
| 3510 | + { | |
| 3511 | + "capacity": 32, | |
| 3512 | + "policy": "lru", | |
| 3513 | + "predictor": "oracle", | |
| 3514 | + "m": 8, | |
| 3515 | + "hit_rate": 0.972606850117096, | |
| 3516 | + "misses_per_token": 3.5063231850117096, | |
| 3517 | + "stall_ms_per_token": 1.3737236533957926, | |
| 3518 | + "prefetch_loads_per_token": 23.830444964871194, | |
| 3519 | + "prefetch_precision": 1.0, | |
| 3520 | + "loads_per_token": 27.336768149882904, | |
| 3521 | + "pred_recall_at_k": 1.0, | |
| 3522 | + "tokens": 2135, | |
| 3523 | + "families": { | |
| 3524 | + "general": { | |
| 3525 | + "hit_rate": 0.9691338153724247, | |
| 3526 | + "stall_ms_per_token": 1.5215816164817764 | |
| 3527 | + }, | |
| 3528 | + "code": { | |
| 3529 | + "hit_rate": 0.9743794589774079, | |
| 3530 | + "stall_ms_per_token": 1.319443519619505 | |
| 3531 | + }, | |
| 3532 | + "reasoning": { | |
| 3533 | + "hit_rate": 0.9704744983277592, | |
| 3534 | + "stall_ms_per_token": 1.4581270903010055 | |
| 3535 | + }, | |
| 3536 | + "multilingual": { | |
| 3537 | + "hit_rate": 0.9762834821428571, | |
| 3538 | + "stall_ms_per_token": 1.1734890109890135 | |
| 3539 | + } | |
| 3540 | + }, | |
| 3541 | + "split": "fit" | |
| 3542 | + }, | |
| 3543 | + { | |
| 3544 | + "capacity": 32, | |
| 3545 | + "policy": "lru", | |
| 3546 | + "predictor": "oracle", | |
| 3547 | + "m": 8, | |
| 3548 | + "hit_rate": 0.9715000432226832, | |
| 3549 | + "misses_per_token": 3.647994467496542, | |
| 3550 | + "stall_ms_per_token": 1.4217828492392859, | |
| 3551 | + "prefetch_loads_per_token": 24.298063623789766, | |
| 3552 | + "prefetch_precision": 1.0, | |
| 3553 | + "loads_per_token": 27.946058091286307, | |
| 3554 | + "pred_recall_at_k": 1.0, | |
| 3555 | + "tokens": 1446, | |
| 3556 | + "families": { | |
| 3557 | + "general": { | |
| 3558 | + "hit_rate": 0.9664293127306273, | |
| 3559 | + "stall_ms_per_token": 1.647464944649449 | |
| 3560 | + }, | |
| 3561 | + "code": { | |
| 3562 | + "hit_rate": 0.9765766274864376, | |
| 3563 | + "stall_ms_per_token": 1.1954792043399678 | |
| 3564 | + }, | |
| 3565 | + "reasoning": { | |
| 3566 | + "hit_rate": 0.9710536858974359, | |
| 3567 | + "stall_ms_per_token": 1.4414102564102582 | |
| 3568 | + }, | |
| 3569 | + "multilingual": { | |
| 3570 | + "hit_rate": 0.9735576923076923, | |
| 3571 | + "stall_ms_per_token": 1.337230769230769 | |
| 3572 | + } | |
| 3573 | + }, | |
| 3574 | + "split": "heldout" | |
| 3575 | + }, | |
| 3576 | + { | |
| 3577 | + "capacity": 32, | |
| 3578 | + "policy": "lfu", | |
| 3579 | + "predictor": "none", | |
| 3580 | + "m": 8, | |
| 3581 | + "hit_rate": 0.8075819672131147, | |
| 3582 | + "misses_per_token": 24.62950819672131, | |
| 3583 | + "stall_ms_per_token": 10.590688524590124, | |
| 3584 | + "prefetch_loads_per_token": 0.0, | |
| 3585 | + "prefetch_precision": null, | |
| 3586 | + "loads_per_token": 24.62950819672131, | |
| 3587 | + "pred_recall_at_k": null, | |
| 3588 | + "tokens": 2135, | |
| 3589 | + "families": { | |
| 3590 | + "general": { | |
| 3591 | + "hit_rate": 0.7640649762282092, | |
| 3592 | + "stall_ms_per_token": 12.985863708399386 | |
| 3593 | + }, | |
| 3594 | + "code": { | |
| 3595 | + "hit_rate": 0.8513116825208086, | |
| 3596 | + "stall_ms_per_token": 8.183804994054725 | |
| 3597 | + }, | |
| 3598 | + "reasoning": { | |
| 3599 | + "hit_rate": 0.78125, | |
| 3600 | + "stall_ms_per_token": 12.040000000000006 | |
| 3601 | + }, | |
| 3602 | + "multilingual": { | |
| 3603 | + "hit_rate": 0.8036143543956044, | |
| 3604 | + "stall_ms_per_token": 10.809065934065947 | |
| 3605 | + } | |
| 3606 | + }, | |
| 3607 | + "split": "fit" | |
| 3608 | + }, | |
| 3609 | + { | |
| 3610 | + "capacity": 32, | |
| 3611 | + "policy": "lfu", | |
| 3612 | + "predictor": "none", | |
| 3613 | + "m": 8, | |
| 3614 | + "hit_rate": 0.812510805670816, | |
| 3615 | + "misses_per_token": 23.998616874135546, | |
| 3616 | + "stall_ms_per_token": 10.319405255878264, | |
| 3617 | + "prefetch_loads_per_token": 0.0, | |
| 3618 | + "prefetch_precision": null, | |
| 3619 | + "loads_per_token": 23.998616874135546, | |
| 3620 | + "pred_recall_at_k": null, | |
| 3621 | + "tokens": 1446, | |
| 3622 | + "families": { | |
| 3623 | + "general": { | |
| 3624 | + "hit_rate": 0.7812067573800738, | |
| 3625 | + "stall_ms_per_token": 12.042380073800732 | |
| 3626 | + }, | |
| 3627 | + "code": { | |
| 3628 | + "hit_rate": 0.8513223327305606, | |
| 3629 | + "stall_ms_per_token": 8.183218806509966 | |
| 3630 | + }, | |
| 3631 | + "reasoning": { | |
| 3632 | + "hit_rate": 0.809395032051282, | |
| 3633 | + "stall_ms_per_token": 10.490897435897432 | |
| 3634 | + }, | |
| 3635 | + "multilingual": { | |
| 3636 | + "hit_rate": 0.7221554487179487, | |
| 3637 | + "stall_ms_per_token": 15.292564102564102 | |
| 3638 | + } | |
| 3639 | + }, | |
| 3640 | + "split": "heldout" | |
| 3641 | + }, | |
| 3642 | + { | |
| 3643 | + "capacity": 32, | |
| 3644 | + "policy": "lfu", | |
| 3645 | + "predictor": "prev_token", | |
| 3646 | + "m": 8, | |
| 3647 | + "hit_rate": 0.8075819672131147, | |
| 3648 | + "misses_per_token": 24.62950819672131, | |
| 3649 | + "stall_ms_per_token": 10.590688524590124, | |
| 3650 | + "prefetch_loads_per_token": 0.0, | |
| 3651 | + "prefetch_precision": null, | |
| 3652 | + "loads_per_token": 24.62950819672131, | |
| 3653 | + "pred_recall_at_k": null, | |
| 3654 | + "tokens": 2135, | |
| 3655 | + "families": { | |
| 3656 | + "general": { | |
| 3657 | + "hit_rate": 0.7640649762282092, | |
| 3658 | + "stall_ms_per_token": 12.985863708399386 | |
| 3659 | + }, | |
| 3660 | + "code": { | |
| 3661 | + "hit_rate": 0.8513116825208086, | |
| 3662 | + "stall_ms_per_token": 8.183804994054725 | |
| 3663 | + }, | |
| 3664 | + "reasoning": { | |
| 3665 | + "hit_rate": 0.78125, | |
| 3666 | + "stall_ms_per_token": 12.040000000000006 | |
| 3667 | + }, | |
| 3668 | + "multilingual": { | |
| 3669 | + "hit_rate": 0.8036143543956044, | |
| 3670 | + "stall_ms_per_token": 10.809065934065947 | |
| 3671 | + } | |
| 3672 | + }, | |
| 3673 | + "split": "fit" | |
| 3674 | + }, | |
| 3675 | + { | |
| 3676 | + "capacity": 32, | |
| 3677 | + "policy": "lfu", | |
| 3678 | + "predictor": "prev_token", | |
| 3679 | + "m": 8, | |
| 3680 | + "hit_rate": 0.812510805670816, | |
| 3681 | + "misses_per_token": 23.998616874135546, | |
| 3682 | + "stall_ms_per_token": 10.319405255878264, | |
| 3683 | + "prefetch_loads_per_token": 0.0, | |
| 3684 | + "prefetch_precision": null, | |
| 3685 | + "loads_per_token": 23.998616874135546, | |
| 3686 | + "pred_recall_at_k": null, | |
| 3687 | + "tokens": 1446, | |
| 3688 | + "families": { | |
| 3689 | + "general": { | |
| 3690 | + "hit_rate": 0.7812067573800738, | |
| 3691 | + "stall_ms_per_token": 12.042380073800732 | |
| 3692 | + }, | |
| 3693 | + "code": { | |
| 3694 | + "hit_rate": 0.8513223327305606, | |
| 3695 | + "stall_ms_per_token": 8.183218806509966 | |
| 3696 | + }, | |
| 3697 | + "reasoning": { | |
| 3698 | + "hit_rate": 0.809395032051282, | |
| 3699 | + "stall_ms_per_token": 10.490897435897432 | |
| 3700 | + }, | |
| 3701 | + "multilingual": { | |
| 3702 | + "hit_rate": 0.7221554487179487, | |
| 3703 | + "stall_ms_per_token": 15.292564102564102 | |
| 3704 | + } | |
| 3705 | + }, | |
| 3706 | + "split": "heldout" | |
| 3707 | + }, | |
| 3708 | + { | |
| 3709 | + "capacity": 32, | |
| 3710 | + "policy": "lfu", | |
| 3711 | + "predictor": "prev_rank", | |
| 3712 | + "m": 8, | |
| 3713 | + "hit_rate": 0.8075929449648712, | |
| 3714 | + "misses_per_token": 24.628103044496488, | |
| 3715 | + "stall_ms_per_token": 10.59008430913345, | |
| 3716 | + "prefetch_loads_per_token": 0.017330210772833723, | |
| 3717 | + "prefetch_precision": 0.2702702702702703, | |
| 3718 | + "loads_per_token": 24.64543325526932, | |
| 3719 | + "pred_recall_at_k": null, | |
| 3720 | + "tokens": 2135, | |
| 3721 | + "families": { | |
| 3722 | + "general": { | |
| 3723 | + "hit_rate": 0.7640897385103012, | |
| 3724 | + "stall_ms_per_token": 12.984500792393042 | |
| 3725 | + }, | |
| 3726 | + "code": { | |
| 3727 | + "hit_rate": 0.8513209720570749, | |
| 3728 | + "stall_ms_per_token": 8.183293697978623 | |
| 3729 | + }, | |
| 3730 | + "reasoning": { | |
| 3731 | + "hit_rate": 0.78125, | |
| 3732 | + "stall_ms_per_token": 12.040000000000008 | |
| 3733 | + }, | |
| 3734 | + "multilingual": { | |
| 3735 | + "hit_rate": 0.8036143543956044, | |
| 3736 | + "stall_ms_per_token": 10.809065934065947 | |
| 3737 | + } | |
| 3738 | + }, | |
| 3739 | + "split": "fit" | |
| 3740 | + }, | |
| 3741 | + { | |
| 3742 | + "capacity": 32, | |
| 3743 | + "policy": "lfu", | |
| 3744 | + "predictor": "prev_rank", | |
| 3745 | + "m": 8, | |
| 3746 | + "hit_rate": 0.812510805670816, | |
| 3747 | + "misses_per_token": 23.998616874135546, | |
| 3748 | + "stall_ms_per_token": 10.319405255878266, | |
| 3749 | + "prefetch_loads_per_token": 0.011065006915629323, | |
| 3750 | + "prefetch_precision": 0.25, | |
| 3751 | + "loads_per_token": 24.009681881051176, | |
| 3752 | + "pred_recall_at_k": null, | |
| 3753 | + "tokens": 1446, | |
| 3754 | + "families": { | |
| 3755 | + "general": { | |
| 3756 | + "hit_rate": 0.7812067573800738, | |
| 3757 | + "stall_ms_per_token": 12.042380073800732 | |
| 3758 | + }, | |
| 3759 | + "code": { | |
| 3760 | + "hit_rate": 0.8513223327305606, | |
| 3761 | + "stall_ms_per_token": 8.183218806509972 | |
| 3762 | + }, | |
| 3763 | + "reasoning": { | |
| 3764 | + "hit_rate": 0.809395032051282, | |
| 3765 | + "stall_ms_per_token": 10.49089743589743 | |
| 3766 | + }, | |
| 3767 | + "multilingual": { | |
| 3768 | + "hit_rate": 0.7221554487179487, | |
| 3769 | + "stall_ms_per_token": 15.292564102564102 | |
| 3770 | + } | |
| 3771 | + }, | |
| 3772 | + "split": "heldout" | |
| 3773 | + }, | |
| 3774 | + { | |
| 3775 | + "capacity": 32, | |
| 3776 | + "policy": "lfu", | |
| 3777 | + "predictor": "transition", | |
| 3778 | + "m": 8, | |
| 3779 | + "hit_rate": 0.8152480971896956, | |
| 3780 | + "misses_per_token": 23.64824355971897, | |
| 3781 | + "stall_ms_per_token": 7.440096487119452, | |
| 3782 | + "prefetch_loads_per_token": 17.589695550351287, | |
| 3783 | + "prefetch_precision": 0.20679554774458114, | |
| 3784 | + "loads_per_token": 41.23793911007026, | |
| 3785 | + "pred_recall_at_k": 0.5302576112412178, | |
| 3786 | + "tokens": 2135, | |
| 3787 | + "families": { | |
| 3788 | + "general": { | |
| 3789 | + "hit_rate": 0.7600163431061807, | |
| 3790 | + "stall_ms_per_token": 9.602792393026952 | |
| 3791 | + }, | |
| 3792 | + "code": { | |
| 3793 | + "hit_rate": 0.8646235879904876, | |
| 3794 | + "stall_ms_per_token": 5.541315101070156 | |
| 3795 | + }, | |
| 3796 | + "reasoning": { | |
| 3797 | + "hit_rate": 0.7980769230769231, | |
| 3798 | + "stall_ms_per_token": 8.123183946488295 | |
| 3799 | + }, | |
| 3800 | + "multilingual": { | |
| 3801 | + "hit_rate": 0.8110190590659341, | |
| 3802 | + "stall_ms_per_token": 7.516939560439553 | |
| 3803 | + } | |
| 3804 | + }, | |
| 3805 | + "split": "fit" | |
| 3806 | + }, | |
| 3807 | + { | |
| 3808 | + "capacity": 32, | |
| 3809 | + "policy": "lfu", | |
| 3810 | + "predictor": "transition", | |
| 3811 | + "m": 8, | |
| 3812 | + "hit_rate": 0.8117814228907331, | |
| 3813 | + "misses_per_token": 24.09197786998617, | |
| 3814 | + "stall_ms_per_token": 7.562228215767629, | |
| 3815 | + "prefetch_loads_per_token": 19.682572614107883, | |
| 3816 | + "prefetch_precision": 0.17557359193282035, | |
| 3817 | + "loads_per_token": 43.77455048409405, | |
| 3818 | + "pred_recall_at_k": 0.4996369294605809, | |
| 3819 | + "tokens": 1446, | |
| 3820 | + "families": { | |
| 3821 | + "general": { | |
| 3822 | + "hit_rate": 0.7562557656826568, | |
| 3823 | + "stall_ms_per_token": 9.73391143911438 | |
| 3824 | + }, | |
| 3825 | + "code": { | |
| 3826 | + "hit_rate": 0.8617201627486437, | |
| 3827 | + "stall_ms_per_token": 5.616853526220617 | |
| 3828 | + }, | |
| 3829 | + "reasoning": { | |
| 3830 | + "hit_rate": 0.8238181089743589, | |
| 3831 | + "stall_ms_per_token": 7.077564102564106 | |
| 3832 | + }, | |
| 3833 | + "multilingual": { | |
| 3834 | + "hit_rate": 0.7790464743589743, | |
| 3835 | + "stall_ms_per_token": 8.843128205128203 | |
| 3836 | + } | |
| 3837 | + }, | |
| 3838 | + "split": "heldout" | |
| 3839 | + }, | |
| 3840 | + { | |
| 3841 | + "capacity": 32, | |
| 3842 | + "policy": "lfu", | |
| 3843 | + "predictor": "probe", | |
| 3844 | + "m": 8, | |
| 3845 | + "hit_rate": 0.7727422423887588, | |
| 3846 | + "misses_per_token": 29.088992974238877, | |
| 3847 | + "stall_ms_per_token": 9.083202810304448, | |
| 3848 | + "prefetch_loads_per_token": 55.67259953161592, | |
| 3849 | + "prefetch_precision": 0.06647260245160314, | |
| 3850 | + "loads_per_token": 84.76159250585481, | |
| 3851 | + "pred_recall_at_k": 0.16113973458235753, | |
| 3852 | + "tokens": 2135, | |
| 3853 | + "families": { | |
| 3854 | + "general": { | |
| 3855 | + "hit_rate": 0.7179947503961965, | |
| 3856 | + "stall_ms_per_token": 11.227179080824094 | |
| 3857 | + }, | |
| 3858 | + "code": { | |
| 3859 | + "hit_rate": 0.821640903686088, | |
| 3860 | + "stall_ms_per_token": 7.202853745541018 | |
| 3861 | + }, | |
| 3862 | + "reasoning": { | |
| 3863 | + "hit_rate": 0.7537364130434783, | |
| 3864 | + "stall_ms_per_token": 9.837210702341142 | |
| 3865 | + }, | |
| 3866 | + "multilingual": { | |
| 3867 | + "hit_rate": 0.7702824519230769, | |
| 3868 | + "stall_ms_per_token": 9.091653846153847 | |
| 3869 | + } | |
| 3870 | + }, | |
| 3871 | + "split": "fit" | |
| 3872 | + }, | |
| 3873 | + { | |
| 3874 | + "capacity": 32, | |
| 3875 | + "policy": "lfu", | |
| 3876 | + "predictor": "probe", | |
| 3877 | + "m": 8, | |
| 3878 | + "hit_rate": 0.7718706777316736, | |
| 3879 | + "misses_per_token": 29.20055325034578, | |
| 3880 | + "stall_ms_per_token": 9.105017980636239, | |
| 3881 | + "prefetch_loads_per_token": 54.932918395574, | |
| 3882 | + "prefetch_precision": 0.06007578714136442, | |
| 3883 | + "loads_per_token": 84.13347164591978, | |
| 3884 | + "pred_recall_at_k": 0.15494467496542186, | |
| 3885 | + "tokens": 1446, | |
| 3886 | + "families": { | |
| 3887 | + "general": { | |
| 3888 | + "hit_rate": 0.7233193035055351, | |
| 3889 | + "stall_ms_per_token": 11.007103321033217 | |
| 3890 | + }, | |
| 3891 | + "code": { | |
| 3892 | + "hit_rate": 0.817755424954792, | |
| 3893 | + "stall_ms_per_token": 7.316354430379746 | |
| 3894 | + }, | |
| 3895 | + "reasoning": { | |
| 3896 | + "hit_rate": 0.7807742387820513, | |
| 3897 | + "stall_ms_per_token": 8.741467948717956 | |
| 3898 | + }, | |
| 3899 | + "multilingual": { | |
| 3900 | + "hit_rate": 0.7247596153846154, | |
| 3901 | + "stall_ms_per_token": 10.941641025641028 | |
| 3902 | + } | |
| 3903 | + }, | |
| 3904 | + "split": "heldout" | |
| 3905 | + }, | |
| 3906 | + { | |
| 3907 | + "capacity": 32, | |
| 3908 | + "policy": "lfu", | |
| 3909 | + "predictor": "oracle", | |
| 3910 | + "m": 8, | |
| 3911 | + "hit_rate": 0.9044496487119438, | |
| 3912 | + "misses_per_token": 12.230444964871195, | |
| 3913 | + "stall_ms_per_token": 3.9919213114753997, | |
| 3914 | + "prefetch_loads_per_token": 23.498829039812648, | |
| 3915 | + "prefetch_precision": 0.6249750847119793, | |
| 3916 | + "loads_per_token": 35.72927400468384, | |
| 3917 | + "pred_recall_at_k": 1.0, | |
| 3918 | + "tokens": 2135, | |
| 3919 | + "families": { | |
| 3920 | + "general": { | |
| 3921 | + "hit_rate": 0.8748142828843106, | |
| 3922 | + "stall_ms_per_token": 5.165163232963544 | |
| 3923 | + }, | |
| 3924 | + "code": { | |
| 3925 | + "hit_rate": 0.9297246581450654, | |
| 3926 | + "stall_ms_per_token": 3.024768133174794 | |
| 3927 | + }, | |
| 3928 | + "reasoning": { | |
| 3929 | + "hit_rate": 0.8963733277591973, | |
| 3930 | + "stall_ms_per_token": 4.323438127090303 | |
| 3931 | + }, | |
| 3932 | + "multilingual": { | |
| 3933 | + "hit_rate": 0.904060782967033, | |
| 3934 | + "stall_ms_per_token": 3.9203186813186828 | |
| 3935 | + } | |
| 3936 | + }, | |
| 3937 | + "split": "fit" | |
| 3938 | + }, | |
| 3939 | + { | |
| 3940 | + "capacity": 32, | |
| 3941 | + "policy": "lfu", | |
| 3942 | + "predictor": "oracle", | |
| 3943 | + "m": 8, | |
| 3944 | + "hit_rate": 0.9066065871369294, | |
| 3945 | + "misses_per_token": 11.95435684647303, | |
| 3946 | + "stall_ms_per_token": 3.896666666666663, | |
| 3947 | + "prefetch_loads_per_token": 23.068464730290458, | |
| 3948 | + "prefetch_precision": 0.6297328896483497, | |
| 3949 | + "loads_per_token": 35.022821576763484, | |
| 3950 | + "pred_recall_at_k": 1.0, | |
| 3951 | + "tokens": 1446, | |
| 3952 | + "families": { | |
| 3953 | + "general": { | |
| 3954 | + "hit_rate": 0.8815728782287823, | |
| 3955 | + "stall_ms_per_token": 4.889653136531366 | |
| 3956 | + }, | |
| 3957 | + "code": { | |
| 3958 | + "hit_rate": 0.9305916591320073, | |
| 3959 | + "stall_ms_per_token": 2.9545569620253174 | |
| 3960 | + }, | |
| 3961 | + "reasoning": { | |
| 3962 | + "hit_rate": 0.9098056891025641, | |
| 3963 | + "stall_ms_per_token": 3.753628205128207 | |
| 3964 | + }, | |
| 3965 | + "multilingual": { | |
| 3966 | + "hit_rate": 0.8888221153846154, | |
| 3967 | + "stall_ms_per_token": 4.599641025641027 | |
| 3968 | + } | |
| 3969 | + }, | |
| 3970 | + "split": "heldout" | |
| 3971 | + }, | |
| 3972 | + { | |
| 3973 | + "capacity": 48, | |
| 3974 | + "policy": "lru", | |
| 3975 | + "predictor": "none", | |
| 3976 | + "m": 8, | |
| 3977 | + "hit_rate": 0.924092505854801, | |
| 3978 | + "misses_per_token": 9.71615925058548, | |
| 3979 | + "stall_ms_per_token": 4.177948477751752, | |
| 3980 | + "prefetch_loads_per_token": 0.0, | |
| 3981 | + "prefetch_precision": null, | |
| 3982 | + "loads_per_token": 9.71615925058548, | |
| 3983 | + "pred_recall_at_k": null, | |
| 3984 | + "tokens": 2135, | |
| 3985 | + "families": { | |
| 3986 | + "general": { | |
| 3987 | + "hit_rate": 0.9084414619651348, | |
| 3988 | + "stall_ms_per_token": 5.039381933439005 | |
| 3989 | + }, | |
| 3990 | + "code": { | |
| 3991 | + "hit_rate": 0.9473654875148633, | |
| 3992 | + "stall_ms_per_token": 2.8970035671819394 | |
| 3993 | + }, | |
| 3994 | + "reasoning": { | |
| 3995 | + "hit_rate": 0.9013116638795987, | |
| 3996 | + "stall_ms_per_token": 5.431806020066899 | |
| 3997 | + }, | |
| 3998 | + "multilingual": { | |
| 3999 | + "hit_rate": 0.9161658653846154, | |
| 4000 | + "stall_ms_per_token": 4.6142307692307805 | |
| 4001 | + } | |
| 4002 | + }, | |
| 4003 | + "split": "fit" | |
| 4004 | + }, | |
| 4005 | + { | |
| 4006 | + "capacity": 48, | |
| 4007 | + "policy": "lru", | |
| 4008 | + "predictor": "none", | |
| 4009 | + "m": 8, | |
| 4010 | + "hit_rate": 0.9295848461272476, | |
| 4011 | + "misses_per_token": 9.01313969571231, | |
| 4012 | + "stall_ms_per_token": 3.875650069156311, | |
| 4013 | + "prefetch_loads_per_token": 0.0, | |
| 4014 | + "prefetch_precision": null, | |
| 4015 | + "loads_per_token": 9.01313969571231, | |
| 4016 | + "pred_recall_at_k": null, | |
| 4017 | + "tokens": 1446, | |
| 4018 | + "families": { | |
| 4019 | + "general": { | |
| 4020 | + "hit_rate": 0.9205056503690037, | |
| 4021 | + "stall_ms_per_token": 4.375369003690051 | |
| 4022 | + }, | |
| 4023 | + "code": { | |
| 4024 | + "hit_rate": 0.947685917721519, | |
| 4025 | + "stall_ms_per_token": 2.8793670886076024 | |
| 4026 | + }, | |
| 4027 | + "reasoning": { | |
| 4028 | + "hit_rate": 0.9283102964743589, | |
| 4029 | + "stall_ms_per_token": 3.9458012820512898 | |
| 4030 | + }, | |
| 4031 | + "multilingual": { | |
| 4032 | + "hit_rate": 0.8092948717948718, | |
| 4033 | + "stall_ms_per_token": 10.496410256410265 | |
| 4034 | + } | |
| 4035 | + }, | |
| 4036 | + "split": "heldout" | |
| 4037 | + }, | |
| 4038 | + { | |
| 4039 | + "capacity": 48, | |
| 4040 | + "policy": "lru", | |
| 4041 | + "predictor": "prev_token", | |
| 4042 | + "m": 8, | |
| 4043 | + "hit_rate": 0.924092505854801, | |
| 4044 | + "misses_per_token": 9.71615925058548, | |
| 4045 | + "stall_ms_per_token": 4.177948477751752, | |
| 4046 | + "prefetch_loads_per_token": 0.0, | |
| 4047 | + "prefetch_precision": null, | |
| 4048 | + "loads_per_token": 9.71615925058548, | |
| 4049 | + "pred_recall_at_k": null, | |
| 4050 | + "tokens": 2135, | |
| 4051 | + "families": { | |
| 4052 | + "general": { | |
| 4053 | + "hit_rate": 0.9084414619651348, | |
| 4054 | + "stall_ms_per_token": 5.039381933439005 | |
| 4055 | + }, | |
| 4056 | + "code": { | |
| 4057 | + "hit_rate": 0.9473654875148633, | |
| 4058 | + "stall_ms_per_token": 2.8970035671819394 | |
| 4059 | + }, | |
| 4060 | + "reasoning": { | |
| 4061 | + "hit_rate": 0.9013116638795987, | |
| 4062 | + "stall_ms_per_token": 5.431806020066899 | |
| 4063 | + }, | |
| 4064 | + "multilingual": { | |
| 4065 | + "hit_rate": 0.9161658653846154, | |
| 4066 | + "stall_ms_per_token": 4.6142307692307805 | |
| 4067 | + } | |
| 4068 | + }, | |
| 4069 | + "split": "fit" | |
| 4070 | + }, | |
| 4071 | + { | |
| 4072 | + "capacity": 48, | |
| 4073 | + "policy": "lru", | |
| 4074 | + "predictor": "prev_token", | |
| 4075 | + "m": 8, | |
| 4076 | + "hit_rate": 0.9295848461272476, | |
| 4077 | + "misses_per_token": 9.01313969571231, | |
| 4078 | + "stall_ms_per_token": 3.875650069156311, | |
| 4079 | + "prefetch_loads_per_token": 0.0, | |
| 4080 | + "prefetch_precision": null, | |
| 4081 | + "loads_per_token": 9.01313969571231, | |
| 4082 | + "pred_recall_at_k": null, | |
| 4083 | + "tokens": 1446, | |
| 4084 | + "families": { | |
| 4085 | + "general": { | |
| 4086 | + "hit_rate": 0.9205056503690037, | |
| 4087 | + "stall_ms_per_token": 4.375369003690051 | |
| 4088 | + }, | |
| 4089 | + "code": { | |
| 4090 | + "hit_rate": 0.947685917721519, | |
| 4091 | + "stall_ms_per_token": 2.8793670886076024 | |
| 4092 | + }, | |
| 4093 | + "reasoning": { | |
| 4094 | + "hit_rate": 0.9283102964743589, | |
| 4095 | + "stall_ms_per_token": 3.9458012820512898 | |
| 4096 | + }, | |
| 4097 | + "multilingual": { | |
| 4098 | + "hit_rate": 0.8092948717948718, | |
| 4099 | + "stall_ms_per_token": 10.496410256410265 | |
| 4100 | + } | |
| 4101 | + }, | |
| 4102 | + "split": "heldout" | |
| 4103 | + }, | |
| 4104 | + { | |
| 4105 | + "capacity": 48, | |
| 4106 | + "policy": "lru", | |
| 4107 | + "predictor": "prev_rank", | |
| 4108 | + "m": 8, | |
| 4109 | + "hit_rate": 0.9240961651053864, | |
| 4110 | + "misses_per_token": 9.715690866510538, | |
| 4111 | + "stall_ms_per_token": 4.1777470725995265, | |
| 4112 | + "prefetch_loads_per_token": 0.00702576112412178, | |
| 4113 | + "prefetch_precision": 0.6666666666666666, | |
| 4114 | + "loads_per_token": 9.72271662763466, | |
| 4115 | + "pred_recall_at_k": null, | |
| 4116 | + "tokens": 2135, | |
| 4117 | + "families": { | |
| 4118 | + "general": { | |
| 4119 | + "hit_rate": 0.9084538431061807, | |
| 4120 | + "stall_ms_per_token": 5.038700475435836 | |
| 4121 | + }, | |
| 4122 | + "code": { | |
| 4123 | + "hit_rate": 0.9473376189060642, | |
| 4124 | + "stall_ms_per_token": 2.8985374554102386 | |
| 4125 | + }, | |
| 4126 | + "reasoning": { | |
| 4127 | + "hit_rate": 0.9013377926421404, | |
| 4128 | + "stall_ms_per_token": 5.430367892976597 | |
| 4129 | + }, | |
| 4130 | + "multilingual": { | |
| 4131 | + "hit_rate": 0.9162087912087912, | |
| 4132 | + "stall_ms_per_token": 4.611868131868143 | |
| 4133 | + } | |
| 4134 | + }, | |
| 4135 | + "split": "fit" | |
| 4136 | + }, | |
| 4137 | + { | |
| 4138 | + "capacity": 48, | |
| 4139 | + "policy": "lru", | |
| 4140 | + "predictor": "prev_rank", | |
| 4141 | + "m": 8, | |
| 4142 | + "hit_rate": 0.9295632347856155, | |
| 4143 | + "misses_per_token": 9.015905947441217, | |
| 4144 | + "stall_ms_per_token": 3.87683955739974, | |
| 4145 | + "prefetch_loads_per_token": 0.004149377593360996, | |
| 4146 | + "prefetch_precision": 0.6666666666666666, | |
| 4147 | + "loads_per_token": 9.020055325034578, | |
| 4148 | + "pred_recall_at_k": null, | |
| 4149 | + "tokens": 1446, | |
| 4150 | + "families": { | |
| 4151 | + "general": { | |
| 4152 | + "hit_rate": 0.9204912361623616, | |
| 4153 | + "stall_ms_per_token": 4.37616236162363 | |
| 4154 | + }, | |
| 4155 | + "code": { | |
| 4156 | + "hit_rate": 0.9476576627486437, | |
| 4157 | + "stall_ms_per_token": 2.8809222423146545 | |
| 4158 | + }, | |
| 4159 | + "reasoning": { | |
| 4160 | + "hit_rate": 0.9282852564102564, | |
| 4161 | + "stall_ms_per_token": 3.947179487179496 | |
| 4162 | + }, | |
| 4163 | + "multilingual": { | |
| 4164 | + "hit_rate": 0.8092948717948718, | |
| 4165 | + "stall_ms_per_token": 10.496410256410265 | |
| 4166 | + } | |
| 4167 | + }, | |
| 4168 | + "split": "heldout" | |
| 4169 | + }, | |
| 4170 | + { | |
| 4171 | + "capacity": 48, | |
| 4172 | + "policy": "lru", | |
| 4173 | + "predictor": "transition", | |
| 4174 | + "m": 8, | |
| 4175 | + "hit_rate": 0.9361460772833724, | |
| 4176 | + "misses_per_token": 8.173302107728338, | |
| 4177 | + "stall_ms_per_token": 2.5986754098360607, | |
| 4178 | + "prefetch_loads_per_token": 3.9531615925058547, | |
| 4179 | + "prefetch_precision": 0.7580568720379147, | |
| 4180 | + "loads_per_token": 12.126463700234192, | |
| 4181 | + "pred_recall_at_k": 0.5302576112412178, | |
| 4182 | + "tokens": 2135, | |
| 4183 | + "families": { | |
| 4184 | + "general": { | |
| 4185 | + "hit_rate": 0.9205378367670365, | |
| 4186 | + "stall_ms_per_token": 3.2211917591125214 | |
| 4187 | + }, | |
| 4188 | + "code": { | |
| 4189 | + "hit_rate": 0.9553266200951248, | |
| 4190 | + "stall_ms_per_token": 1.850480380499408 | |
| 4191 | + }, | |
| 4192 | + "reasoning": { | |
| 4193 | + "hit_rate": 0.9226327341137124, | |
| 4194 | + "stall_ms_per_token": 3.1336923076923076 | |
| 4195 | + }, | |
| 4196 | + "multilingual": { | |
| 4197 | + "hit_rate": 0.9299879807692307, | |
| 4198 | + "stall_ms_per_token": 2.8087142857142866 | |
| 4199 | + } | |
| 4200 | + }, | |
| 4201 | + "split": "fit" | |
| 4202 | + }, | |
| 4203 | + { | |
| 4204 | + "capacity": 48, | |
| 4205 | + "policy": "lru", | |
| 4206 | + "predictor": "transition", | |
| 4207 | + "m": 8, | |
| 4208 | + "hit_rate": 0.9392667271784232, | |
| 4209 | + "misses_per_token": 7.773858921161826, | |
| 4210 | + "stall_ms_per_token": 2.4720760719225554, | |
| 4211 | + "prefetch_loads_per_token": 4.271784232365145, | |
| 4212 | + "prefetch_precision": 0.704225352112676, | |
| 4213 | + "loads_per_token": 12.04564315352697, | |
| 4214 | + "pred_recall_at_k": 0.4996369294605809, | |
| 4215 | + "tokens": 1446, | |
| 4216 | + "families": { | |
| 4217 | + "general": { | |
| 4218 | + "hit_rate": 0.9258965636531366, | |
| 4219 | + "stall_ms_per_token": 3.0031697416974183 | |
| 4220 | + }, | |
| 4221 | + "code": { | |
| 4222 | + "hit_rate": 0.9563178119349005, | |
| 4223 | + "stall_ms_per_token": 1.7911175406871627 | |
| 4224 | + }, | |
| 4225 | + "reasoning": { | |
| 4226 | + "hit_rate": 0.9410306490384616, | |
| 4227 | + "stall_ms_per_token": 2.40423717948718 | |
| 4228 | + }, | |
| 4229 | + "multilingual": { | |
| 4230 | + "hit_rate": 0.8691907051282052, | |
| 4231 | + "stall_ms_per_token": 5.289589743589742 | |
| 4232 | + } | |
| 4233 | + }, | |
| 4234 | + "split": "heldout" | |
| 4235 | + }, | |
| 4236 | + { | |
| 4237 | + "capacity": 48, | |
| 4238 | + "policy": "lru", | |
| 4239 | + "predictor": "probe", | |
| 4240 | + "m": 8, | |
| 4241 | + "hit_rate": 0.9094189110070258, | |
| 4242 | + "misses_per_token": 11.594379391100702, | |
| 4243 | + "stall_ms_per_token": 3.631840749414515, | |
| 4244 | + "prefetch_loads_per_token": 19.468852459016393, | |
| 4245 | + "prefetch_precision": 0.4080017321849589, | |
| 4246 | + "loads_per_token": 31.063231850117095, | |
| 4247 | + "pred_recall_at_k": 0.16113973458235753, | |
| 4248 | + "tokens": 2135, | |
| 4249 | + "families": { | |
| 4250 | + "general": { | |
| 4251 | + "hit_rate": 0.8993537044374009, | |
| 4252 | + "stall_ms_per_token": 4.040085578446911 | |
| 4253 | + }, | |
| 4254 | + "code": { | |
| 4255 | + "hit_rate": 0.9229897443519619, | |
| 4256 | + "stall_ms_per_token": 3.1004946492271186 | |
| 4257 | + }, | |
| 4258 | + "reasoning": { | |
| 4259 | + "hit_rate": 0.892636914715719, | |
| 4260 | + "stall_ms_per_token": 4.293210702341141 | |
| 4261 | + }, | |
| 4262 | + "multilingual": { | |
| 4263 | + "hit_rate": 0.9092977335164835, | |
| 4264 | + "stall_ms_per_token": 3.608516483516485 | |
| 4265 | + } | |
| 4266 | + }, | |
| 4267 | + "split": "fit" | |
| 4268 | + }, | |
| 4269 | + { | |
| 4270 | + "capacity": 48, | |
| 4271 | + "policy": "lru", | |
| 4272 | + "predictor": "probe", | |
| 4273 | + "m": 8, | |
| 4274 | + "hit_rate": 0.910404780428769, | |
| 4275 | + "misses_per_token": 11.468188105117566, | |
| 4276 | + "stall_ms_per_token": 3.5877634854771885, | |
| 4277 | + "prefetch_loads_per_token": 17.79875518672199, | |
| 4278 | + "prefetch_precision": 0.4108481952053464, | |
| 4279 | + "loads_per_token": 29.266943291839556, | |
| 4280 | + "pred_recall_at_k": 0.15494467496542186, | |
| 4281 | + "tokens": 1446, | |
| 4282 | + "families": { | |
| 4283 | + "general": { | |
| 4284 | + "hit_rate": 0.9016086254612546, | |
| 4285 | + "stall_ms_per_token": 3.9420442804428033 | |
| 4286 | + }, | |
| 4287 | + "code": { | |
| 4288 | + "hit_rate": 0.9229769439421338, | |
| 4289 | + "stall_ms_per_token": 3.0799421338155537 | |
| 4290 | + }, | |
| 4291 | + "reasoning": { | |
| 4292 | + "hit_rate": 0.9113581730769231, | |
| 4293 | + "stall_ms_per_token": 3.5512564102564124 | |
| 4294 | + }, | |
| 4295 | + "multilingual": { | |
| 4296 | + "hit_rate": 0.8467548076923077, | |
| 4297 | + "stall_ms_per_token": 6.156871794871796 | |
| 4298 | + } | |
| 4299 | + }, | |
| 4300 | + "split": "heldout" | |
| 4301 | + }, | |
| 4302 | + { | |
| 4303 | + "capacity": 48, | |
| 4304 | + "policy": "lru", | |
| 4305 | + "predictor": "oracle", | |
| 4306 | + "m": 8, | |
| 4307 | + "hit_rate": 0.990350556206089, | |
| 4308 | + "misses_per_token": 1.235128805620609, | |
| 4309 | + "stall_ms_per_token": 0.5033470725995347, | |
| 4310 | + "prefetch_loads_per_token": 8.902576112412177, | |
| 4311 | + "prefetch_precision": 1.0, | |
| 4312 | + "loads_per_token": 10.137704918032787, | |
| 4313 | + "pred_recall_at_k": 1.0, | |
| 4314 | + "tokens": 2135, | |
| 4315 | + "families": { | |
| 4316 | + "general": { | |
| 4317 | + "hit_rate": 0.9882379160063391, | |
| 4318 | + "stall_ms_per_token": 0.604177496038037 | |
| 4319 | + }, | |
| 4320 | + "code": { | |
| 4321 | + "hit_rate": 0.9916208382877527, | |
| 4322 | + "stall_ms_per_token": 0.4474910820451855 | |
| 4323 | + }, | |
| 4324 | + "reasoning": { | |
| 4325 | + "hit_rate": 0.9895484949832776, | |
| 4326 | + "stall_ms_per_token": 0.5469966555183955 | |
| 4327 | + }, | |
| 4328 | + "multilingual": { | |
| 4329 | + "hit_rate": 0.9917367788461539, | |
| 4330 | + "stall_ms_per_token": 0.42175274725274803 | |
| 4331 | + } | |
| 4332 | + }, | |
| 4333 | + "split": "fit" | |
| 4334 | + }, | |
| 4335 | + { | |
| 4336 | + "capacity": 48, | |
| 4337 | + "policy": "lru", | |
| 4338 | + "predictor": "oracle", | |
| 4339 | + "m": 8, | |
| 4340 | + "hit_rate": 0.9910961272475796, | |
| 4341 | + "misses_per_token": 1.1396957123098201, | |
| 4342 | + "stall_ms_per_token": 0.4685587828492426, | |
| 4343 | + "prefetch_loads_per_token": 8.230290456431534, | |
| 4344 | + "prefetch_precision": 1.0, | |
| 4345 | + "loads_per_token": 9.369986168741356, | |
| 4346 | + "pred_recall_at_k": 1.0, | |
| 4347 | + "tokens": 1446, | |
| 4348 | + "families": { | |
| 4349 | + "general": { | |
| 4350 | + "hit_rate": 0.9897659132841329, | |
| 4351 | + "stall_ms_per_token": 0.5342361623616255 | |
| 4352 | + }, | |
| 4353 | + "code": { | |
| 4354 | + "hit_rate": 0.9927525994575045, | |
| 4355 | + "stall_ms_per_token": 0.3826943942133832 | |
| 4356 | + }, | |
| 4357 | + "reasoning": { | |
| 4358 | + "hit_rate": 0.9913361378205128, | |
| 4359 | + "stall_ms_per_token": 0.45962820512820546 | |
| 4360 | + }, | |
| 4361 | + "multilingual": { | |
| 4362 | + "hit_rate": 0.9841746794871795, | |
| 4363 | + "stall_ms_per_token": 0.8447692307692305 | |
| 4364 | + } | |
| 4365 | + }, | |
| 4366 | + "split": "heldout" | |
| 4367 | + }, | |
| 4368 | + { | |
| 4369 | + "capacity": 48, | |
| 4370 | + "policy": "lfu", | |
| 4371 | + "predictor": "none", | |
| 4372 | + "m": 8, | |
| 4373 | + "hit_rate": 0.9246999414519906, | |
| 4374 | + "misses_per_token": 9.638407494145198, | |
| 4375 | + "stall_ms_per_token": 4.144515222482432, | |
| 4376 | + "prefetch_loads_per_token": 0.0, | |
| 4377 | + "prefetch_precision": null, | |
| 4378 | + "loads_per_token": 9.638407494145198, | |
| 4379 | + "pred_recall_at_k": null, | |
| 4380 | + "tokens": 2135, | |
| 4381 | + "families": { | |
| 4382 | + "general": { | |
| 4383 | + "hit_rate": 0.9113015055467512, | |
| 4384 | + "stall_ms_per_token": 4.881965134706829 | |
| 4385 | + }, | |
| 4386 | + "code": { | |
| 4387 | + "hit_rate": 0.945470422116528, | |
| 4388 | + "stall_ms_per_token": 3.0013079667063116 | |
| 4389 | + }, | |
| 4390 | + "reasoning": { | |
| 4391 | + "hit_rate": 0.9042642140468228, | |
| 4392 | + "stall_ms_per_token": 5.269297658862889 | |
| 4393 | + }, | |
| 4394 | + "multilingual": { | |
| 4395 | + "hit_rate": 0.9167239010989011, | |
| 4396 | + "stall_ms_per_token": 4.5835164835164885 | |
| 4397 | + } | |
| 4398 | + }, | |
| 4399 | + "split": "fit" | |
| 4400 | + }, | |
| 4401 | + { | |
| 4402 | + "capacity": 48, | |
| 4403 | + "policy": "lfu", | |
| 4404 | + "predictor": "none", | |
| 4405 | + "m": 8, | |
| 4406 | + "hit_rate": 0.9320269277316736, | |
| 4407 | + "misses_per_token": 8.70055325034578, | |
| 4408 | + "stall_ms_per_token": 3.7412378976487006, | |
| 4409 | + "prefetch_loads_per_token": 0.0, | |
| 4410 | + "prefetch_precision": null, | |
| 4411 | + "loads_per_token": 8.70055325034578, | |
| 4412 | + "pred_recall_at_k": null, | |
| 4413 | + "tokens": 1446, | |
| 4414 | + "families": { | |
| 4415 | + "general": { | |
| 4416 | + "hit_rate": 0.9220191420664207, | |
| 4417 | + "stall_ms_per_token": 4.292066420664221 | |
| 4418 | + }, | |
| 4419 | + "code": { | |
| 4420 | + "hit_rate": 0.9504125226039783, | |
| 4421 | + "stall_ms_per_token": 2.7292947558770426 | |
| 4422 | + }, | |
| 4423 | + "reasoning": { | |
| 4424 | + "hit_rate": 0.9318409455128205, | |
| 4425 | + "stall_ms_per_token": 3.7514743589743684 | |
| 4426 | + }, | |
| 4427 | + "multilingual": { | |
| 4428 | + "hit_rate": 0.8118990384615384, | |
| 4429 | + "stall_ms_per_token": 10.353076923076927 | |
| 4430 | + } | |
| 4431 | + }, | |
| 4432 | + "split": "heldout" | |
| 4433 | + }, | |
| 4434 | + { | |
| 4435 | + "capacity": 48, | |
| 4436 | + "policy": "lfu", | |
| 4437 | + "predictor": "prev_token", | |
| 4438 | + "m": 8, | |
| 4439 | + "hit_rate": 0.9246999414519906, | |
| 4440 | + "misses_per_token": 9.638407494145198, | |
| 4441 | + "stall_ms_per_token": 4.144515222482432, | |
| 4442 | + "prefetch_loads_per_token": 0.0, | |
| 4443 | + "prefetch_precision": null, | |
| 4444 | + "loads_per_token": 9.638407494145198, | |
| 4445 | + "pred_recall_at_k": null, | |
| 4446 | + "tokens": 2135, | |
| 4447 | + "families": { | |
| 4448 | + "general": { | |
| 4449 | + "hit_rate": 0.9113015055467512, | |
| 4450 | + "stall_ms_per_token": 4.881965134706829 | |
| 4451 | + }, | |
| 4452 | + "code": { | |
| 4453 | + "hit_rate": 0.945470422116528, | |
| 4454 | + "stall_ms_per_token": 3.0013079667063116 | |
| 4455 | + }, | |
| 4456 | + "reasoning": { | |
| 4457 | + "hit_rate": 0.9042642140468228, | |
| 4458 | + "stall_ms_per_token": 5.269297658862889 | |
| 4459 | + }, | |
| 4460 | + "multilingual": { | |
| 4461 | + "hit_rate": 0.9167239010989011, | |
| 4462 | + "stall_ms_per_token": 4.5835164835164885 | |
| 4463 | + } | |
| 4464 | + }, | |
| 4465 | + "split": "fit" | |
| 4466 | + }, | |
| 4467 | + { | |
| 4468 | + "capacity": 48, | |
| 4469 | + "policy": "lfu", | |
| 4470 | + "predictor": "prev_token", | |
| 4471 | + "m": 8, | |
| 4472 | + "hit_rate": 0.9320269277316736, | |
| 4473 | + "misses_per_token": 8.70055325034578, | |
| 4474 | + "stall_ms_per_token": 3.7412378976487006, | |
| 4475 | + "prefetch_loads_per_token": 0.0, | |
| 4476 | + "prefetch_precision": null, | |
| 4477 | + "loads_per_token": 8.70055325034578, | |
| 4478 | + "pred_recall_at_k": null, | |
| 4479 | + "tokens": 1446, | |
| 4480 | + "families": { | |
| 4481 | + "general": { | |
| 4482 | + "hit_rate": 0.9220191420664207, | |
| 4483 | + "stall_ms_per_token": 4.292066420664221 | |
| 4484 | + }, | |
| 4485 | + "code": { | |
| 4486 | + "hit_rate": 0.9504125226039783, | |
| 4487 | + "stall_ms_per_token": 2.7292947558770426 | |
| 4488 | + }, | |
| 4489 | + "reasoning": { | |
| 4490 | + "hit_rate": 0.9318409455128205, | |
| 4491 | + "stall_ms_per_token": 3.7514743589743684 | |
| 4492 | + }, | |
| 4493 | + "multilingual": { | |
| 4494 | + "hit_rate": 0.8118990384615384, | |
| 4495 | + "stall_ms_per_token": 10.353076923076927 | |
| 4496 | + } | |
| 4497 | + }, | |
| 4498 | + "split": "heldout" | |
| 4499 | + }, | |
| 4500 | + { | |
| 4501 | + "capacity": 48, | |
| 4502 | + "policy": "lfu", | |
| 4503 | + "predictor": "prev_rank", | |
| 4504 | + "m": 8, | |
| 4505 | + "hit_rate": 0.92473287470726, | |
| 4506 | + "misses_per_token": 9.634192037470726, | |
| 4507 | + "stall_ms_per_token": 4.142702576112407, | |
| 4508 | + "prefetch_loads_per_token": 0.008430913348946136, | |
| 4509 | + "prefetch_precision": 0.5555555555555556, | |
| 4510 | + "loads_per_token": 9.642622950819671, | |
| 4511 | + "pred_recall_at_k": null, | |
| 4512 | + "tokens": 2135, | |
| 4513 | + "families": { | |
| 4514 | + "general": { | |
| 4515 | + "hit_rate": 0.9113262678288431, | |
| 4516 | + "stall_ms_per_token": 4.88060221870049 | |
| 4517 | + }, | |
| 4518 | + "code": { | |
| 4519 | + "hit_rate": 0.9454890011890607, | |
| 4520 | + "stall_ms_per_token": 3.000285374554113 | |
| 4521 | + }, | |
| 4522 | + "reasoning": { | |
| 4523 | + "hit_rate": 0.9042119565217391, | |
| 4524 | + "stall_ms_per_token": 5.272173913043489 | |
| 4525 | + }, | |
| 4526 | + "multilingual": { | |
| 4527 | + "hit_rate": 0.9168741414835165, | |
| 4528 | + "stall_ms_per_token": 4.575247252747258 | |
| 4529 | + } | |
| 4530 | + }, | |
| 4531 | + "split": "fit" | |
| 4532 | + }, | |
| 4533 | + { | |
| 4534 | + "capacity": 48, | |
| 4535 | + "policy": "lfu", | |
| 4536 | + "predictor": "prev_rank", | |
| 4537 | + "m": 8, | |
| 4538 | + "hit_rate": 0.9320377334024896, | |
| 4539 | + "misses_per_token": 8.699170124481327, | |
| 4540 | + "stall_ms_per_token": 3.740643153526985, | |
| 4541 | + "prefetch_loads_per_token": 0.0027662517289073307, | |
| 4542 | + "prefetch_precision": 0.5, | |
| 4543 | + "loads_per_token": 8.701936376210234, | |
| 4544 | + "pred_recall_at_k": null, | |
| 4545 | + "tokens": 1446, | |
| 4546 | + "families": { | |
| 4547 | + "general": { | |
| 4548 | + "hit_rate": 0.9220191420664207, | |
| 4549 | + "stall_ms_per_token": 4.292066420664221 | |
| 4550 | + }, | |
| 4551 | + "code": { | |
| 4552 | + "hit_rate": 0.9504266500904159, | |
| 4553 | + "stall_ms_per_token": 2.7285171790235165 | |
| 4554 | + }, | |
| 4555 | + "reasoning": { | |
| 4556 | + "hit_rate": 0.9318659855769231, | |
| 4557 | + "stall_ms_per_token": 3.750096153846163 | |
| 4558 | + }, | |
| 4559 | + "multilingual": { | |
| 4560 | + "hit_rate": 0.8118990384615384, | |
| 4561 | + "stall_ms_per_token": 10.353076923076927 | |
| 4562 | + } | |
| 4563 | + }, | |
| 4564 | + "split": "heldout" | |
| 4565 | + }, | |
| 4566 | + { | |
| 4567 | + "capacity": 48, | |
| 4568 | + "policy": "lfu", | |
| 4569 | + "predictor": "transition", | |
| 4570 | + "m": 8, | |
| 4571 | + "hit_rate": 0.9311365632318501, | |
| 4572 | + "misses_per_token": 8.814519906323184, | |
| 4573 | + "stall_ms_per_token": 2.7863878220140537, | |
| 4574 | + "prefetch_loads_per_token": 6.043559718969555, | |
| 4575 | + "prefetch_precision": 0.3024877935363869, | |
| 4576 | + "loads_per_token": 14.85807962529274, | |
| 4577 | + "pred_recall_at_k": 0.5302576112412178, | |
| 4578 | + "tokens": 2135, | |
| 4579 | + "families": { | |
| 4580 | + "general": { | |
| 4581 | + "hit_rate": 0.9097414817749604, | |
| 4582 | + "stall_ms_per_token": 3.6385356576862176 | |
| 4583 | + }, | |
| 4584 | + "code": { | |
| 4585 | + "hit_rate": 0.9528927615933412, | |
| 4586 | + "stall_ms_per_token": 1.936649227110588 | |
| 4587 | + }, | |
| 4588 | + "reasoning": { | |
| 4589 | + "hit_rate": 0.9231814381270903, | |
| 4590 | + "stall_ms_per_token": 3.105204013377928 | |
| 4591 | + }, | |
| 4592 | + "multilingual": { | |
| 4593 | + "hit_rate": 0.9244934752747253, | |
| 4594 | + "stall_ms_per_token": 3.0105604395604386 | |
| 4595 | + } | |
| 4596 | + }, | |
| 4597 | + "split": "fit" | |
| 4598 | + }, | |
| 4599 | + { | |
| 4600 | + "capacity": 48, | |
| 4601 | + "policy": "lfu", | |
| 4602 | + "predictor": "transition", | |
| 4603 | + "m": 8, | |
| 4604 | + "hit_rate": 0.935798106846473, | |
| 4605 | + "misses_per_token": 8.217842323651452, | |
| 4606 | + "stall_ms_per_token": 2.5979266943292, | |
| 4607 | + "prefetch_loads_per_token": 8.367911479944675, | |
| 4608 | + "prefetch_precision": 0.20603305785123968, | |
| 4609 | + "loads_per_token": 16.585753803596127, | |
| 4610 | + "pred_recall_at_k": 0.4996369294605809, | |
| 4611 | + "tokens": 1446, | |
| 4612 | + "families": { | |
| 4613 | + "general": { | |
| 4614 | + "hit_rate": 0.9136012453874539, | |
| 4615 | + "stall_ms_per_token": 3.469955719557199 | |
| 4616 | + }, | |
| 4617 | + "code": { | |
| 4618 | + "hit_rate": 0.9581543851717902, | |
| 4619 | + "stall_ms_per_token": 1.718039783001811 | |
| 4620 | + }, | |
| 4621 | + "reasoning": { | |
| 4622 | + "hit_rate": 0.9424328926282052, | |
| 4623 | + "stall_ms_per_token": 2.333211538461538 | |
| 4624 | + }, | |
| 4625 | + "multilingual": { | |
| 4626 | + "hit_rate": 0.874198717948718, | |
| 4627 | + "stall_ms_per_token": 5.073025641025639 | |
| 4628 | + } | |
| 4629 | + }, | |
| 4630 | + "split": "heldout" | |
| 4631 | + }, | |
| 4632 | + { | |
| 4633 | + "capacity": 48, | |
| 4634 | + "policy": "lfu", | |
| 4635 | + "predictor": "probe", | |
| 4636 | + "m": 8, | |
| 4637 | + "hit_rate": 0.923986387587822, | |
| 4638 | + "misses_per_token": 9.729742388758782, | |
| 4639 | + "stall_ms_per_token": 3.062785011709593, | |
| 4640 | + "prefetch_loads_per_token": 25.289929742388757, | |
| 4641 | + "prefetch_precision": 0.09988146831129385, | |
| 4642 | + "loads_per_token": 35.019672131147544, | |
| 4643 | + "pred_recall_at_k": 0.16113973458235753, | |
| 4644 | + "tokens": 2135, | |
| 4645 | + "families": { | |
| 4646 | + "general": { | |
| 4647 | + "hit_rate": 0.9026718502377179, | |
| 4648 | + "stall_ms_per_token": 3.9118193343898597 | |
| 4649 | + }, | |
| 4650 | + "code": { | |
| 4651 | + "hit_rate": 0.9457398186682521, | |
| 4652 | + "stall_ms_per_token": 2.2131533888228336 | |
| 4653 | + }, | |
| 4654 | + "reasoning": { | |
| 4655 | + "hit_rate": 0.9170411789297659, | |
| 4656 | + "stall_ms_per_token": 3.342561872909698 | |
| 4657 | + }, | |
| 4658 | + "multilingual": { | |
| 4659 | + "hit_rate": 0.9163804945054945, | |
| 4660 | + "stall_ms_per_token": 3.3241758241758284 | |
| 4661 | + } | |
| 4662 | + }, | |
| 4663 | + "split": "fit" | |
| 4664 | + }, | |
| 4665 | + { | |
| 4666 | + "capacity": 48, | |
| 4667 | + "policy": "lfu", | |
| 4668 | + "predictor": "probe", | |
| 4669 | + "m": 8, | |
| 4670 | + "hit_rate": 0.9313299619640387, | |
| 4671 | + "misses_per_token": 8.789764868603044, | |
| 4672 | + "stall_ms_per_token": 2.770647302904575, | |
| 4673 | + "prefetch_loads_per_token": 24.876210235131396, | |
| 4674 | + "prefetch_precision": 0.08809874621222652, | |
| 4675 | + "loads_per_token": 33.66597510373444, | |
| 4676 | + "pred_recall_at_k": 0.15494467496542186, | |
| 4677 | + "tokens": 1446, | |
| 4678 | + "families": { | |
| 4679 | + "general": { | |
| 4680 | + "hit_rate": 0.912750807195572, | |
| 4681 | + "stall_ms_per_token": 3.5028302583025828 | |
| 4682 | + }, | |
| 4683 | + "code": { | |
| 4684 | + "hit_rate": 0.9512460443037974, | |
| 4685 | + "stall_ms_per_token": 1.9850886075949403 | |
| 4686 | + }, | |
| 4687 | + "reasoning": { | |
| 4688 | + "hit_rate": 0.9364232772435898, | |
| 4689 | + "stall_ms_per_token": 2.565519230769232 | |
| 4690 | + }, | |
| 4691 | + "multilingual": { | |
| 4692 | + "hit_rate": 0.866386217948718, | |
| 4693 | + "stall_ms_per_token": 5.375025641025641 | |
| 4694 | + } | |
| 4695 | + }, | |
| 4696 | + "split": "heldout" | |
| 4697 | + }, | |
| 4698 | + { | |
| 4699 | + "capacity": 48, | |
| 4700 | + "policy": "lfu", | |
| 4701 | + "predictor": "oracle", | |
| 4702 | + "m": 8, | |
| 4703 | + "hit_rate": 0.9791971604215457, | |
| 4704 | + "misses_per_token": 2.6627634660421546, | |
| 4705 | + "stall_ms_per_token": 0.9285573770491828, | |
| 4706 | + "prefetch_loads_per_token": 8.933957845433255, | |
| 4707 | + "prefetch_precision": 0.8367935409457901, | |
| 4708 | + "loads_per_token": 11.59672131147541, | |
| 4709 | + "pred_recall_at_k": 1.0, | |
| 4710 | + "tokens": 2135, | |
| 4711 | + "families": { | |
| 4712 | + "general": { | |
| 4713 | + "hit_rate": 0.9722538629160064, | |
| 4714 | + "stall_ms_per_token": 1.222057052297941 | |
| 4715 | + }, | |
| 4716 | + "code": { | |
| 4717 | + "hit_rate": 0.9842728151010701, | |
| 4718 | + "stall_ms_per_token": 0.7236218787158171 | |
| 4719 | + }, | |
| 4720 | + "reasoning": { | |
| 4721 | + "hit_rate": 0.9817621237458194, | |
| 4722 | + "stall_ms_per_token": 0.8407090301003346 | |
| 4723 | + }, | |
| 4724 | + "multilingual": { | |
| 4725 | + "hit_rate": 0.9773995535714286, | |
| 4726 | + "stall_ms_per_token": 0.9654230769230782 | |
| 4727 | + } | |
| 4728 | + }, | |
| 4729 | + "split": "fit" | |
| 4730 | + }, | |
| 4731 | + { | |
| 4732 | + "capacity": 48, | |
| 4733 | + "policy": "lfu", | |
| 4734 | + "predictor": "oracle", | |
| 4735 | + "m": 8, | |
| 4736 | + "hit_rate": 0.9822895055325035, | |
| 4737 | + "misses_per_token": 2.2669432918395573, | |
| 4738 | + "stall_ms_per_token": 0.8007551867219945, | |
| 4739 | + "prefetch_loads_per_token": 8.067081604426003, | |
| 4740 | + "prefetch_precision": 0.8536648092584654, | |
| 4741 | + "loads_per_token": 10.33402489626556, | |
| 4742 | + "pred_recall_at_k": 1.0, | |
| 4743 | + "tokens": 1446, | |
| 4744 | + "families": { | |
| 4745 | + "general": { | |
| 4746 | + "hit_rate": 0.975495848708487, | |
| 4747 | + "stall_ms_per_token": 1.0773579335793377 | |
| 4748 | + }, | |
| 4749 | + "code": { | |
| 4750 | + "hit_rate": 0.987440664556962, | |
| 4751 | + "stall_ms_per_token": 0.585949367088609 | |
| 4752 | + }, | |
| 4753 | + "reasoning": { | |
| 4754 | + "hit_rate": 0.9859525240384616, | |
| 4755 | + "stall_ms_per_token": 0.650916666666667 | |
| 4756 | + }, | |
| 4757 | + "multilingual": { | |
| 4758 | + "hit_rate": 0.9743589743589743, | |
| 4759 | + "stall_ms_per_token": 1.2012307692307687 | |
| 4760 | + } | |
| 4761 | + }, | |
| 4762 | + "split": "heldout" | |
| 4763 | + } | |
| 4764 | + ] | |
| 4765 | +} | |
| \ No newline at end of file | ||
added
benchmarks/raw/gateC/design/miss_cost_breakdown_laptop.txt
+8 −0
@@ -0,0 +1,8 @@ | ||
| 1 | +# 2026-11-01 laptop M5 Max, OLMoE 4-bit, page-cached checkpoint, 200 random experts (seed 0); medians / p95 in ms | |
| 2 | +pread 9 tensors (page-cached) p50/p95 ms: 0.132 / 0.142 | |
| 3 | +pread + np + mx.array (lazy, no eval) p50/p95 ms: 0.191 / 0.198 | |
| 4 | +pread + np + mx.array + eval p50/p95 ms: 0.189 / 0.204 | |
| 5 | +read + lazy install + table eval p50/p95 ms: 0.400 / 0.377 | |
| 6 | +ensure(8 experts) + eval p50/p95 ms: 1.694 / 1.792 (0.212 ms per expert) | |
| 7 | +pread F_NOCACHE (pages still cached) p50/p95 ms: 0.198 / 0.205 | |
| 8 | +bytes per expert: 3538944 | |
added
benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl
+36 −0
@@ -0,0 +1,36 @@ | ||
| 1 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:31:06", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 679.4582500588149, "prompt_tps": 35.44577652018054, "gen_tps": 39.05715717342248, "decode_tps_wall": 38.44833603563943, "itl_ms_p50": 23.70983303990215, "itl_ms_p95": 38.87683304492384, "itl_ms_p99": 51.92095797974616, "itl_ms_mean": 26.00892790452769, "total_s": 2.3180207080440596, "peak_mem_gb": 1.769273148, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.6446361081500145, "cache_misses": 3667, "cache_hits": 6652, "stall_s": 1.5813420980703086, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 12977307648, "ssd_read_s": 0.49590603238902986} | |
| 2 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:31:09", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 559.203541954048, "prompt_tps": 32.206121587493136, "gen_tps": 29.677870486939558, "decode_tps_wall": 29.214277945831714, "itl_ms_p50": 33.05849991738796, "itl_ms_p95": 47.747042030096054, "itl_ms_p99": 51.032082992605865, "itl_ms_mean": 34.229837953009536, "total_s": 2.715683332993649, "peak_mem_gb": 1.653111888, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.5090379594296044, "cache_misses": 4889, "cache_hits": 5069, "stall_s": 2.0085335480980575, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 17301897216, "ssd_read_s": 0.6531581478193402} | |
| 3 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:31:12", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 505.49712497740984, "prompt_tps": 27.710542597291298, "gen_tps": 28.450212454482266, "decode_tps_wall": 28.005798756541928, "itl_ms_p50": 36.29954194184393, "itl_ms_p95": 50.0600419472903, "itl_ms_p99": 53.843417088501155, "itl_ms_mean": 35.70689087260573, "total_s": 2.7550312499515712, "peak_mem_gb": 1.549717648, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.4795243019648397, "cache_misses": 5033, "cache_hits": 4637, "stall_s": 2.0551544436020777, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 17811505152, "ssd_read_s": 0.6727308424888179} | |
| 4 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:31:14", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 512.9798339912668, "prompt_tps": 37.0591123459889, "gen_tps": 41.60696952513016, "decode_tps_wall": 40.9570802924626, "itl_ms_p50": 21.473583998158574, "itl_ms_p95": 38.552624988369644, "itl_ms_p99": 41.25191702041775, "itl_ms_mean": 24.415802905365588, "total_s": 2.051175417029299, "peak_mem_gb": 1.636499212, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.6694072269589931, "cache_misses": 3257, "cache_hits": 6595, "stall_s": 1.3520558513700962, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 11526340608, "ssd_read_s": 0.43526661884970963} | |
| 5 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:31:17", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 567.0875840587541, "prompt_tps": 31.75625546690247, "gen_tps": 29.36287830628246, "decode_tps_wall": 28.904235279452596, "itl_ms_p50": 33.91837503295392, "itl_ms_p95": 48.558250069618225, "itl_ms_p99": 50.7312499685213, "itl_ms_mean": 34.59700595195745, "total_s": 2.7466989590320736, "peak_mem_gb": 1.65311192, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.5079333199437638, "cache_misses": 4900, "cache_hits": 5058, "stall_s": 2.032577119884081, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 17340825600, "ssd_read_s": 0.6631453247973695} | |
| 6 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:31:19", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 504.644125001505, "prompt_tps": 27.7567055373783, "gen_tps": 28.115310993852603, "decode_tps_wall": 27.67612324292153, "itl_ms_p50": 36.58783400896937, "itl_ms_p95": 49.77720894385129, "itl_ms_p99": 54.432707955129445, "itl_ms_mean": 36.132228174542504, "total_s": 2.780974499997683, "peak_mem_gb": 1.549717616, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.4795243019648397, "cache_misses": 5033, "cache_hits": 4637, "stall_s": 2.0702099225018173, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 17811505152, "ssd_read_s": 0.678577120299451} | |
| 7 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:31:22", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 521.2122080847621, "prompt_tps": 36.472387459970804, "gen_tps": 40.949639175386885, "decode_tps_wall": 40.31005684650972, "itl_ms_p50": 22.81420794315636, "itl_ms_p95": 38.27158303465694, "itl_ms_p99": 42.97333292197436, "itl_ms_mean": 24.80770503022959, "total_s": 2.0840976249892265, "peak_mem_gb": 1.636499212, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.6694072269589931, "cache_misses": 3257, "cache_hits": 6595, "stall_s": 1.3759213021257892, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 11526340608, "ssd_read_s": 0.4448013227665797} | |
| 8 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:31:24", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 611.5603339858353, "prompt_tps": 39.261603751172686, "gen_tps": 39.99823652590423, "decode_tps_wall": 39.3735091472787, "itl_ms_p50": 23.427875014021993, "itl_ms_p95": 37.78220899403095, "itl_ms_p99": 47.81737504526973, "itl_ms_mean": 25.39778703136281, "total_s": 2.2116209169616923, "peak_mem_gb": 1.76929446, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.6541331524372517, "cache_misses": 3569, "cache_hits": 6750, "stall_s": 1.4954329903703183, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 12630491136, "ssd_read_s": 0.47636724647600204} | |
| 9 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:31:26", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 556.3839169917628, "prompt_tps": 32.36671243436712, "gen_tps": 29.5729544038375, "decode_tps_wall": 29.110996937907878, "itl_ms_p50": 33.25212502386421, "itl_ms_p95": 47.58650006260723, "itl_ms_p99": 51.26791703514755, "itl_ms_mean": 34.351279763209206, "total_s": 2.7205145420739427, "peak_mem_gb": 1.65311192, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.5090379594296044, "cache_misses": 4889, "cache_hits": 5069, "stall_s": 2.009863163693808, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 17301897216, "ssd_read_s": 0.6516196166630834} | |
| 10 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:31:29", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 625.3284170525149, "prompt_tps": 38.396301175238435, "gen_tps": 39.85330352728346, "decode_tps_wall": 39.230846061058436, "itl_ms_p50": 23.58258399181068, "itl_ms_p95": 37.32958307955414, "itl_ms_p99": 47.620125114917755, "itl_ms_mean": 25.490146158041345, "total_s": 2.2312076250091195, "peak_mem_gb": 1.76929446, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.6497722647543367, "cache_misses": 3614, "cache_hits": 6705, "stall_s": 1.5171293833991513, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 12789743616, "ssd_read_s": 0.47865299379918724} | |
| 11 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:31:31", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 487.9155829548836, "prompt_tps": 28.708489644453465, "gen_tps": 28.236466653249078, "decode_tps_wall": 27.795405224620268, "itl_ms_p50": 35.86604190059006, "itl_ms_p95": 51.306959008798, "itl_ms_p99": 53.99249994661659, "itl_ms_mean": 35.97716931697158, "total_s": 2.7544772499240935, "peak_mem_gb": 1.549717616, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.48355739400206826, "cache_misses": 4994, "cache_hits": 4676, "stall_s": 2.0529416904319078, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 17673486336, "ssd_read_s": 0.6633842524606735} | |
| 12 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:31:33", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 515.6262919772416, "prompt_tps": 36.86786170054398, "gen_tps": 41.2133123968512, "decode_tps_wall": 40.56958734690269, "itl_ms_p50": 21.661083097569644, "itl_ms_p95": 38.80991693586111, "itl_ms_p99": 41.21291695628315, "itl_ms_mean": 24.64900595239467, "total_s": 2.068513666978106, "peak_mem_gb": 1.636499212, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.6694072269589931, "cache_misses": 3257, "cache_hits": 6595, "stall_s": 1.3638423326192424, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 11526340608, "ssd_read_s": 0.43793867458589375} | |
| 13 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:32:05", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 681.4193750033155, "prompt_tps": 35.34598225948908, "gen_tps": 35.309788813597784, "decode_tps_wall": 34.759171236924765, "itl_ms_p50": 27.844666969031096, "itl_ms_p95": 39.30166701320559, "itl_ms_p99": 42.43595793377608, "itl_ms_mean": 28.76938558701012, "total_s": 2.493890666984953, "peak_mem_gb": 1.84715862, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.7077236166295183, "cache_misses": 3016, "cache_hits": 7303, "stall_s": 1.4651703460840508, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 1602, "installed": 1602, "late": 0, "dup": 7294, "predictor": "transition", "m": 8}, "ssd_bytes": 16342843392, "ssd_read_s": 0.9224822366377339} | |
| 14 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:32:08", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 566.7365409899503, "prompt_tps": 31.77753835936156, "gen_tps": 28.315549960276407, "decode_tps_wall": 27.873221747721544, "itl_ms_p50": 36.23079101089388, "itl_ms_p95": 45.012334012426436, "itl_ms_p99": 49.48816692922264, "itl_ms_mean": 35.8767281748384, "total_s": 2.8269704160047695, "peak_mem_gb": 1.713585104, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.6572604940751154, "cache_misses": 3413, "cache_hits": 6545, "stall_s": 1.7749390117824078, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 3481, "installed": 3481, "late": 2, "dup": 5231, "predictor": "transition", "m": 8}, "ssd_bytes": 24397479936, "ssd_read_s": 1.5312390537001193} | |
| 15 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:32:11", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 494.08933299127966, "prompt_tps": 28.349985272747084, "gen_tps": 25.042766716386843, "decode_tps_wall": 24.651557088257487, "itl_ms_p50": 39.977375068701804, "itl_ms_p95": 48.767708940431476, "itl_ms_p99": 55.01737492159009, "itl_ms_mean": 40.56538888881545, "total_s": 3.049708832986653, "peak_mem_gb": 1.613474288, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.6231644260599793, "cache_misses": 3644, "cache_hits": 6026, "stall_s": 1.9704076391644776, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 4547, "installed": 4547, "late": 3, "dup": 4005, "predictor": "transition", "m": 8}, "ssd_bytes": 28987490304, "ssd_read_s": 1.8772433317499235} | |
| 16 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:32:13", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 521.7654169537127, "prompt_tps": 36.43479459707605, "gen_tps": 36.449428003856085, "decode_tps_wall": 35.88007341725578, "itl_ms_p50": 25.89249995071441, "itl_ms_p95": 38.83045900147408, "itl_ms_p99": 40.187207981944084, "itl_ms_mean": 27.870623016035154, "total_s": 2.2776146669639274, "peak_mem_gb": 1.668305292, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.7330491270807957, "cache_misses": 2630, "cache_hits": 7222, "stall_s": 1.2665090759983286, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 1428, "installed": 1428, "late": 0, "dup": 7236, "predictor": "transition", "m": 8}, "ssd_bytes": 14361034752, "ssd_read_s": 0.8116820899304003} | |
| 17 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:32:16", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 561.2853750353679, "prompt_tps": 32.083779043453376, "gen_tps": 28.259962890964697, "decode_tps_wall": 27.818506917950078, "itl_ms_p50": 36.85616690199822, "itl_ms_p95": 45.1834169216454, "itl_ms_p99": 48.22554101701826, "itl_ms_mean": 35.947292316926735, "total_s": 2.825964791001752, "peak_mem_gb": 1.706523632, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.6572604940751154, "cache_misses": 3413, "cache_hits": 6545, "stall_s": 1.7750553110381588, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 3470, "installed": 3470, "late": 1, "dup": 5242, "predictor": "transition", "m": 8}, "ssd_bytes": 24358551552, "ssd_read_s": 1.5238975549582392} | |
| 18 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:32:19", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 497.18608299735934, "prompt_tps": 28.172667916600272, "gen_tps": 25.469101136159118, "decode_tps_wall": 25.07123788284445, "itl_ms_p50": 39.154416997917, "itl_ms_p95": 49.17870892677456, "itl_ms_p99": 55.77741703018546, "itl_ms_mean": 39.886343254086874, "total_s": 3.0100257080048323, "peak_mem_gb": 1.613474256, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.6230610134436402, "cache_misses": 3645, "cache_hits": 6025, "stall_s": 1.9492603754624724, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 4543, "installed": 4543, "late": 3, "dup": 4009, "predictor": "transition", "m": 8}, "ssd_bytes": 28976873472, "ssd_read_s": 1.862569018965587} | |
| 19 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:32:21", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 522.8202920407057, "prompt_tps": 36.35903779420813, "gen_tps": 36.70903850681987, "decode_tps_wall": 36.13563597716809, "itl_ms_p50": 25.849625002592802, "itl_ms_p95": 38.27999997884035, "itl_ms_p99": 40.10304203256965, "itl_ms_mean": 27.673513222012726, "total_s": 2.2662516250275075, "peak_mem_gb": 1.668305292, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.7330491270807957, "cache_misses": 2630, "cache_hits": 7222, "stall_s": 1.2604319450911134, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 1428, "installed": 1428, "late": 0, "dup": 7236, "predictor": "transition", "m": 8}, "ssd_bytes": 14361034752, "ssd_read_s": 0.8033619953785092} | |
| 20 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:32:24", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 629.3936250731349, "prompt_tps": 38.14785288386788, "gen_tps": 35.201166985762036, "decode_tps_wall": 34.651310753091444, "itl_ms_p50": 27.364624897018075, "itl_ms_p95": 39.80533394496888, "itl_ms_p99": 41.841791942715645, "itl_ms_mean": 28.85893717341657, "total_s": 2.447506666998379, "peak_mem_gb": 1.811798876, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.7111154181606745, "cache_misses": 2981, "cache_hits": 7338, "stall_s": 1.4258965507615358, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 1478, "installed": 1478, "late": 0, "dup": 7418, "predictor": "transition", "m": 8}, "ssd_bytes": 15780151296, "ssd_read_s": 0.8703944578301162} | |
| 21 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:32:27", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 566.0618750844151, "prompt_tps": 31.81279232303207, "gen_tps": 28.322057483072268, "decode_tps_wall": 27.879630217065635, "itl_ms_p50": 35.985250025987625, "itl_ms_p95": 44.946374953724444, "itl_ms_p99": 48.932958976365626, "itl_ms_mean": 35.86848147605206, "total_s": 2.8257762080756947, "peak_mem_gb": 1.713568752, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.6572604940751154, "cache_misses": 3413, "cache_hits": 6545, "stall_s": 1.7713123478461057, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 3480, "installed": 3480, "late": 1, "dup": 5232, "predictor": "transition", "m": 8}, "ssd_bytes": 24393940992, "ssd_read_s": 1.5129008889198303} | |
| 22 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:32:29", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 638.7889159377664, "prompt_tps": 37.58665974003575, "gen_tps": 35.26449207594671, "decode_tps_wall": 34.71365890747842, "itl_ms_p50": 27.96383306849748, "itl_ms_p95": 38.83433295413852, "itl_ms_p99": 41.52045806404203, "itl_ms_mean": 28.80710450791946, "total_s": 2.4536364999366924, "peak_mem_gb": 1.818876764, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.7097586975482121, "cache_misses": 2995, "cache_hits": 7324, "stall_s": 1.4326911598909646, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 1481, "installed": 1481, "late": 0, "dup": 7415, "predictor": "transition", "m": 8}, "ssd_bytes": 15840313344, "ssd_read_s": 0.8747730017639697} | |
| 23 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:32:32", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 487.2451249975711, "prompt_tps": 28.747583316879304, "gen_tps": 25.463923339668476, "decode_tps_wall": 25.066141789348634, "itl_ms_p50": 39.436375023797154, "itl_ms_p95": 48.56837494298816, "itl_ms_p99": 56.268082931637764, "itl_ms_mean": 39.894452381376475, "total_s": 3.000595625024289, "peak_mem_gb": 1.599285712, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.6251292657704239, "cache_misses": 3625, "cache_hits": 6045, "stall_s": 1.9413647933397442, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 4546, "installed": 4546, "late": 4, "dup": 4006, "predictor": "transition", "m": 8}, "ssd_bytes": 28916711424, "ssd_read_s": 1.8608200687449425} | |
| 24 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:32:34", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 521.031167008914, "prompt_tps": 36.48427600397794, "gen_tps": 36.69359114045238, "decode_tps_wall": 36.12043931204202, "itl_ms_p50": 25.98729101009667, "itl_ms_p95": 38.791082915849984, "itl_ms_p99": 40.55266606155783, "itl_ms_mean": 27.685156079112655, "total_s": 2.2651959999930114, "peak_mem_gb": 1.668305292, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.7330491270807957, "cache_misses": 2630, "cache_hits": 7222, "stall_s": 1.261129126767628, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 1428, "installed": 1428, "late": 0, "dup": 7236, "predictor": "transition", "m": 8}, "ssd_bytes": 14361034752, "ssd_read_s": 0.8071539712836966} | |
| 25 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "prev_rank", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor prev_rank --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:32:38", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 682.3382920119911, "prompt_tps": 35.295136275638434, "gen_tps": 34.6826862749499, "decode_tps_wall": 34.14181621890222, "itl_ms_p50": 27.476874995045364, "itl_ms_p95": 41.68533405754715, "itl_ms_p99": 51.988332998007536, "itl_ms_mean": 29.289595889932816, "total_s": 2.5275828330777586, "peak_mem_gb": 1.769301852, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.6446361081500145, "cache_misses": 3667, "cache_hits": 6652, "stall_s": 1.5489405386615545, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 129, "installed": 126, "late": 0, "dup": 8191, "predictor": "prev_rank", "m": 8}, "ssd_bytes": 13433831424, "ssd_read_s": 0.5297496897401288} | |
| 26 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "prev_rank", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor prev_rank --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:32:41", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 574.74045897834, "prompt_tps": 31.334912989084266, "gen_tps": 26.220263941421162, "decode_tps_wall": 25.810675418076926, "itl_ms_p50": 37.637416971847415, "itl_ms_p95": 52.79541702475399, "itl_ms_p99": 56.864375015720725, "itl_ms_mean": 38.743658730434994, "total_s": 3.0155909589957446, "peak_mem_gb": 1.663728592, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.5090379594296044, "cache_misses": 4889, "cache_hits": 5069, "stall_s": 2.0418206797912717, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 226, "installed": 223, "late": 0, "dup": 8094, "predictor": "prev_rank", "m": 8}, "ssd_bytes": 18101698560, "ssd_read_s": 0.7303524538874626} | |
| 27 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "prev_rank", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor prev_rank --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:32:44", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 526.3170829275623, "prompt_tps": 26.61397100295001, "gen_tps": 25.414804010130165, "decode_tps_wall": 25.017796638371017, "itl_ms_p50": 40.03033402841538, "itl_ms_p95": 54.19237504247576, "itl_ms_p99": 60.533792013302445, "itl_ms_mean": 39.97154563428864, "total_s": 3.0445244578877464, "peak_mem_gb": 1.570974192, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.4796277145811789, "cache_misses": 5032, "cache_hits": 4638, "stall_s": 2.079394665081054, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 260, "installed": 262, "late": 0, "dup": 8060, "predictor": "prev_rank", "m": 8}, "ssd_bytes": 18728091648, "ssd_read_s": 0.7556515095056966} | |
| 28 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "prev_rank", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor prev_rank --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:32:46", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 530.1641670521349, "prompt_tps": 35.85694966446432, "gen_tps": 35.808991205482734, "decode_tps_wall": 35.24962529479262, "itl_ms_p50": 25.419291108846664, "itl_ms_p95": 41.88570904079825, "itl_ms_p99": 45.66720803268254, "itl_ms_mean": 28.369095887885326, "total_s": 2.3174172079889104, "peak_mem_gb": 1.650610572, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.669305724725944, "cache_misses": 3258, "cache_hits": 6594, "stall_s": 1.3614791821455583, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 121, "installed": 121, "late": 0, "dup": 8199, "predictor": "prev_rank", "m": 8}, "ssd_bytes": 11958091776, "ssd_read_s": 0.4730367272859439} | |
| 29 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "prev_rank", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor prev_rank --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:32:49", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 578.5914580337703, "prompt_tps": 31.12414132352772, "gen_tps": 26.412162557051676, "decode_tps_wall": 25.99957668156317, "itl_ms_p50": 37.01750002801418, "itl_ms_p95": 52.31333407573402, "itl_ms_p99": 54.68608299270272, "itl_ms_mean": 38.462164682439635, "total_s": 3.001707833027467, "peak_mem_gb": 1.667267568, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.5079333199437638, "cache_misses": 4900, "cache_hits": 5058, "stall_s": 2.031746352557093, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 226, "installed": 224, "late": 0, "dup": 8094, "predictor": "prev_rank", "m": 8}, "ssd_bytes": 18140626944, "ssd_read_s": 0.7256683950545266} | |
| 30 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "prev_rank", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor prev_rank --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:32:52", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 521.1150420363992, "prompt_tps": 26.8791355854505, "gen_tps": 25.34009132509392, "decode_tps_wall": 24.94424540485073, "itl_ms_p50": 40.29129189439118, "itl_ms_p95": 54.69712498597801, "itl_ms_p99": 59.42304199561477, "itl_ms_mean": 40.08940674571527, "total_s": 3.0467476670164615, "peak_mem_gb": 1.57097416, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.4796277145811789, "cache_misses": 5032, "cache_hits": 4638, "stall_s": 2.0832139356061816, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 260, "installed": 262, "late": 0, "dup": 8060, "predictor": "prev_rank", "m": 8}, "ssd_bytes": 18728091648, "ssd_read_s": 0.7563709784299135} | |
| 31 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "prev_rank", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor prev_rank --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:32:54", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 527.7795830043033, "prompt_tps": 36.017312981764526, "gen_tps": 35.2933285731094, "decode_tps_wall": 34.7420610984124, "itl_ms_p50": 25.609957985579967, "itl_ms_p95": 42.179582989774644, "itl_ms_p99": 46.19866702705622, "itl_ms_mean": 28.783554238976823, "total_s": 2.341143500059843, "peak_mem_gb": 1.650610572, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.669305724725944, "cache_misses": 3258, "cache_hits": 6594, "stall_s": 1.3730238495627418, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 121, "installed": 121, "late": 0, "dup": 8199, "predictor": "prev_rank", "m": 8}, "ssd_bytes": 11958091776, "ssd_read_s": 0.4705457449890673} | |
| 32 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "prev_rank", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor prev_rank --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:32:57", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 637.8974579274654, "prompt_tps": 37.639924478210155, "gen_tps": 33.950319996683206, "decode_tps_wall": 33.42003090469517, "itl_ms_p50": 28.29104196280241, "itl_ms_p95": 42.76941600255668, "itl_ms_p99": 52.62954195495695, "itl_ms_mean": 29.92217460395916, "total_s": 2.5229944579768926, "peak_mem_gb": 1.783487324, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.6541331524372517, "cache_misses": 3569, "cache_hits": 6750, "stall_s": 1.535994044272229, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 129, "installed": 130, "late": 0, "dup": 8191, "predictor": "prev_rank", "m": 8}, "ssd_bytes": 13087014912, "ssd_read_s": 0.528557246667333} | |
| 33 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "prev_rank", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor prev_rank --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:33:00", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 571.3402080582455, "prompt_tps": 31.51930493512097, "gen_tps": 26.434554296005317, "decode_tps_wall": 26.021616942030036, "itl_ms_p50": 37.302416982129216, "itl_ms_p95": 52.15741693973541, "itl_ms_p99": 55.27591600548476, "itl_ms_mean": 38.42958730150251, "total_s": 2.9924042080529034, "peak_mem_gb": 1.663728624, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.5090379594296044, "cache_misses": 4889, "cache_hits": 5069, "stall_s": 2.0188576456857845, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 226, "installed": 223, "late": 0, "dup": 8094, "predictor": "prev_rank", "m": 8}, "ssd_bytes": 18101698560, "ssd_read_s": 0.7217989380005747} | |
| 34 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "prev_rank", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor prev_rank --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:33:02", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 641.5067079942673, "prompt_tps": 37.42756211013386, "gen_tps": 34.57381683149746, "decode_tps_wall": 34.03378478155468, "itl_ms_p50": 27.23516698461026, "itl_ms_p95": 42.25499997846782, "itl_ms_p99": 51.96758301462978, "itl_ms_mean": 29.382568128066996, "total_s": 2.492608500062488, "peak_mem_gb": 1.790565212, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.6497722647543367, "cache_misses": 3614, "cache_hits": 6705, "stall_s": 1.521140352357179, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 129, "installed": 132, "late": 0, "dup": 8191, "predictor": "prev_rank", "m": 8}, "ssd_bytes": 13246267392, "ssd_read_s": 0.5231834813021123} | |
| 35 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "prev_rank", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor prev_rank --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:33:05", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 509.8117080051452, "prompt_tps": 27.474517814502768, "gen_tps": 25.461941534444165, "decode_tps_wall": 25.064199241373046, "itl_ms_p50": 39.6392080001533, "itl_ms_p95": 55.1902080187574, "itl_ms_p99": 58.95374994724989, "itl_ms_mean": 39.89754431688833, "total_s": 3.02335699996911, "peak_mem_gb": 1.560357328, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.48366080661840744, "cache_misses": 4993, "cache_hits": 4677, "stall_s": 2.0561962134670466, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 260, "installed": 259, "late": 0, "dup": 8060, "predictor": "prev_rank", "m": 8}, "ssd_bytes": 18590072832, "ssd_read_s": 0.748749066493474} | |
| 36 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "prev_rank", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor prev_rank --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:33:08", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 531.9485410582274, "prompt_tps": 35.73534968695303, "gen_tps": 35.24788314813883, "decode_tps_wall": 34.697306958340214, "itl_ms_p50": 25.73604101780802, "itl_ms_p95": 42.177541996352375, "itl_ms_p99": 44.9884170666337, "itl_ms_mean": 28.820680556005783, "total_s": 2.347651416086592, "peak_mem_gb": 1.650610572, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.669305724725944, "cache_misses": 3258, "cache_hits": 6594, "stall_s": 1.3774335174821317, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 121, "installed": 121, "late": 0, "dup": 8199, "predictor": "prev_rank", "m": 8}, "ssd_bytes": 11958091776, "ssd_read_s": 0.47966364747844636} | |
added
benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl
+24 −0
@@ -0,0 +1,24 @@ | ||
| 1 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:33:49", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 380.284791928716, "prompt_tps": 63.49407916852991, "gen_tps": 72.7086033100041, "decode_tps_wall": 71.57769844848642, "itl_ms_p50": 12.953416910022497, "itl_ms_p95": 20.275167073123157, "itl_ms_p99": 25.91829199809581, "itl_ms_mean": 13.970832000412639, "total_s": 1.2604472079547122, "peak_mem_gb": 1.93055716, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.6446361081500145, "cache_misses": 3667, "cache_hits": 6652, "stall_s": 0.692682103253901, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 12977307648, "ssd_read_s": 0.457394742872566} | |
| 2 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:33:51", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 328.72704102192074, "prompt_tps": 54.80381434829405, "gen_tps": 53.62428044666654, "decode_tps_wall": 52.78682305572237, "itl_ms_p50": 18.525041989050806, "itl_ms_p95": 26.887458050623536, "itl_ms_p99": 28.134332969784737, "itl_ms_mean": 18.94412169765149, "total_s": 1.5222067079739645, "peak_mem_gb": 1.828353096, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.5090379594296044, "cache_misses": 4889, "cache_hits": 5069, "stall_s": 0.9151933705434203, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 17301897216, "ssd_read_s": 0.620059332344681} | |
| 3 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:33:52", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 296.9263340346515, "prompt_tps": 47.19383502230973, "gen_tps": 51.510199575824984, "decode_tps_wall": 50.70574040403944, "itl_ms_p50": 19.529416924342513, "itl_ms_p95": 27.567957993596792, "itl_ms_p99": 31.546416925266385, "itl_ms_mean": 19.721632936067643, "total_s": 1.539389209006913, "peak_mem_gb": 1.747773596, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.4795243019648397, "cache_misses": 5033, "cache_hits": 4637, "stall_s": 0.938357112929225, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 17811505152, "ssd_read_s": 0.6404213644564152} | |
| 4 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:33:53", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 300.1055410131812, "prompt_tps": 63.36930429649795, "gen_tps": 75.10929355893249, "decode_tps_wall": 73.93661109771251, "itl_ms_p50": 12.372791999951005, "itl_ms_p95": 20.534208975732327, "itl_ms_p99": 21.977707976475358, "itl_ms_mean": 13.525099205296664, "total_s": 1.152186790946871, "peak_mem_gb": 1.783821656, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.6694072269589931, "cache_misses": 3257, "cache_hits": 6595, "stall_s": 0.6138717805733904, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 11526340608, "ssd_read_s": 0.4109209069283679} | |
| 5 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:33:55", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 338.91541603952646, "prompt_tps": 53.150613325158844, "gen_tps": 53.6383661867278, "decode_tps_wall": 52.8007091945298, "itl_ms_p50": 18.529249937273562, "itl_ms_p95": 27.357708080671728, "itl_ms_p99": 28.572040959261358, "itl_ms_mean": 18.939139554277062, "total_s": 1.5320812079589814, "peak_mem_gb": 1.824420888, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.5079333199437638, "cache_misses": 4900, "cache_hits": 5058, "stall_s": 0.918347351718694, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 17340825600, "ssd_read_s": 0.6223103012889624} | |
| 6 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:33:56", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 295.717041939497, "prompt_tps": 47.382818476300635, "gen_tps": 51.32151394777667, "decode_tps_wall": 50.51997147433442, "itl_ms_p50": 19.9885000474751, "itl_ms_p95": 27.370624942705035, "itl_ms_p99": 33.131583011709154, "itl_ms_mean": 19.79415211087418, "total_s": 1.5427486249245703, "peak_mem_gb": 1.735846116, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.4795243019648397, "cache_misses": 5033, "cache_hits": 4637, "stall_s": 0.9404555514920503, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 17811505152, "ssd_read_s": 0.641931640333496} | |
| 7 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:33:57", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 304.43799996282905, "prompt_tps": 62.46298101979038, "gen_tps": 75.69627638231321, "decode_tps_wall": 74.51436300928616, "itl_ms_p50": 11.834957986138761, "itl_ms_p95": 20.020000054500997, "itl_ms_p99": 21.80154190864414, "itl_ms_mean": 13.42023147772702, "total_s": 1.1499125830596313, "peak_mem_gb": 1.783821656, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.6694072269589931, "cache_misses": 3257, "cache_hits": 6595, "stall_s": 0.6096728361444548, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 11526340608, "ssd_read_s": 0.4086581049486995} | |
| 8 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:33:59", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 342.79470797628164, "prompt_tps": 70.06660834075579, "gen_tps": 72.58949020517235, "decode_tps_wall": 71.45600202970235, "itl_ms_p50": 13.034582952968776, "itl_ms_p95": 20.004790974780917, "itl_ms_p99": 26.54633391648531, "itl_ms_mean": 13.994625666075283, "total_s": 1.2244561249390244, "peak_mem_gb": 1.934097296, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.6541331524372517, "cache_misses": 3569, "cache_hits": 6750, "stall_s": 0.6717810803093016, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 12630491136, "ssd_read_s": 0.4470097190933302} | |
| 9 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:34:00", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 328.75908294226974, "prompt_tps": 54.79284558307793, "gen_tps": 53.70490602989395, "decode_tps_wall": 52.86619753996985, "itl_ms_p50": 18.288584076799452, "itl_ms_p95": 26.584165985696018, "itl_ms_p99": 28.06895796675235, "itl_ms_mean": 18.91567857219054, "total_s": 1.5204468329902738, "peak_mem_gb": 1.828353128, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.5090379594296044, "cache_misses": 4889, "cache_hits": 5069, "stall_s": 0.9145860569551587, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 17301897216, "ssd_read_s": 0.6196216040989384} | |
| 10 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:34:01", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 354.18962500989437, "prompt_tps": 67.81252561672528, "gen_tps": 72.61160293243346, "decode_tps_wall": 71.4778374052782, "itl_ms_p50": 12.879334040917456, "itl_ms_p95": 20.153332967311144, "itl_ms_p99": 25.95441695302725, "itl_ms_mean": 13.990350524037485, "total_s": 1.235581708024256, "peak_mem_gb": 1.934097296, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.6497722647543367, "cache_misses": 3614, "cache_hits": 6705, "stall_s": 0.6796442195773125, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 12789743616, "ssd_read_s": 0.4522450801450759} | |
| 11 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:34:03", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 283.28070894349366, "prompt_tps": 49.46407283923087, "gen_tps": 51.288917915595675, "decode_tps_wall": 50.487892715359344, "itl_ms_p50": 19.674375071190298, "itl_ms_p95": 28.04341702722013, "itl_ms_p99": 30.936624971218407, "itl_ms_mean": 19.806728825815732, "total_s": 1.5311046249698848, "peak_mem_gb": 1.742661784, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.48355739400206826, "cache_misses": 4994, "cache_hits": 4676, "stall_s": 0.9337375573813915, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 17673486336, "ssd_read_s": 0.6375937573611736} | |
| 12 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "none", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor none --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:34:04", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 289.672875078395, "prompt_tps": 65.64919058123002, "gen_tps": 73.2884253720508, "decode_tps_wall": 72.14413367846092, "itl_ms_p50": 12.27204198949039, "itl_ms_p95": 20.88362502399832, "itl_ms_p99": 27.023459086194634, "itl_ms_mean": 13.861140871923121, "total_s": 1.1629247500095516, "peak_mem_gb": 1.777333532, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.6694072269589931, "cache_misses": 3257, "cache_hits": 6595, "stall_s": 0.6170409636106342, "hits_device": 0, "layers_complete": 0, "prefetch": null, "ssd_bytes": 11526340608, "ssd_read_s": 0.41335971443913877} | |
| 13 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:34:07", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 417.09470795467496, "prompt_tps": 57.890011536031714, "gen_tps": 54.497364891925855, "decode_tps_wall": 53.64855027840424, "itl_ms_p50": 17.92687503620982, "itl_ms_p95": 25.522999931126833, "itl_ms_p99": 26.658250018954277, "itl_ms_mean": 18.639832666690744, "total_s": 1.5914041659561917, "peak_mem_gb": 1.996581356, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.7077236166295183, "cache_misses": 3016, "cache_hits": 7303, "stall_s": 0.7253628862090409, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 1602, "installed": 1602, "late": 1, "dup": 7294, "predictor": "transition", "m": 8}, "ssd_bytes": 16342843392, "ssd_read_s": 0.9576644137268886} | |
| 14 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:34:09", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 347.00695890933275, "prompt_tps": 51.913499145898086, "gen_tps": 40.112991201106865, "decode_tps_wall": 39.48636078944318, "itl_ms_p50": 24.965791031718254, "itl_ms_p95": 32.99429209437221, "itl_ms_p99": 38.66183292120695, "itl_ms_mean": 25.32520039849693, "total_s": 1.9424945840146393, "peak_mem_gb": 1.885472376, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.6573609158465555, "cache_misses": 3412, "cache_hits": 6546, "stall_s": 0.9406086412491277, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 3471, "installed": 3471, "late": 35, "dup": 5241, "predictor": "transition", "m": 8}, "ssd_bytes": 24358551552, "ssd_read_s": 1.7092103488976136} | |
| 15 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:34:11", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 316.44454202614725, "prompt_tps": 44.28148156602418, "gen_tps": 34.83273991757286, "decode_tps_wall": 34.288682091028086, "itl_ms_p50": 28.07887503877282, "itl_ms_p95": 37.83991595264524, "itl_ms_p99": 46.05083400383592, "itl_ms_mean": 29.164142189695248, "total_s": 2.153785499976948, "peak_mem_gb": 1.79716218, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.6236814891416753, "cache_misses": 3639, "cache_hits": 6031, "stall_s": 1.0574734223773703, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 4539, "installed": 4532, "late": 41, "dup": 4013, "predictor": "transition", "m": 8}, "ssd_bytes": 28941484032, "ssd_read_s": 2.118904664646834} | |
| 16 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": true, "repeat": -1, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:34:12", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 325.72004199028015, "prompt_tps": 58.383043269068835, "gen_tps": 55.374239387542005, "decode_tps_wall": 54.50941976032858, "itl_ms_p50": 16.76612498704344, "itl_ms_p95": 25.38916701450944, "itl_ms_p99": 27.91670907754451, "itl_ms_mean": 18.345453031730674, "total_s": 1.4814835829893127, "peak_mem_gb": 1.836780496, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.7330491270807957, "cache_misses": 2630, "cache_hits": 7222, "stall_s": 0.6353771339636296, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 1423, "installed": 1430, "late": 4, "dup": 7241, "predictor": "transition", "m": 8}, "ssd_bytes": 14343340032, "ssd_read_s": 0.8503749336814508} | |
| 17 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:34:14", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 367.7875839639455, "prompt_tps": 48.97680422707743, "gen_tps": 40.10276436185179, "decode_tps_wall": 39.476387501232274, "itl_ms_p50": 25.049000047147274, "itl_ms_p95": 32.35129197128117, "itl_ms_p99": 41.3809580495581, "itl_ms_mean": 25.33159854023584, "total_s": 1.9636782919988036, "peak_mem_gb": 1.886389912, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.6574613376179956, "cache_misses": 3411, "cache_hits": 6547, "stall_s": 0.9502528810407966, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 3463, "installed": 3463, "late": 40, "dup": 5249, "predictor": "transition", "m": 8}, "ssd_bytes": 24326701056, "ssd_read_s": 1.7192287121433765} | |
| 18 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:34:16", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 315.69245900027454, "prompt_tps": 44.383739612025856, "gen_tps": 34.78234168470292, "decode_tps_wall": 34.239070739252554, "itl_ms_p50": 28.248958056792617, "itl_ms_p95": 37.44195797480643, "itl_ms_p99": 45.93187500722706, "itl_ms_mean": 29.20640012737186, "total_s": 2.155695667024702, "peak_mem_gb": 1.800701124, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.6234746639089969, "cache_misses": 3641, "cache_hits": 6029, "stall_s": 1.0593780489871278, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 4532, "installed": 4526, "late": 41, "dup": 4020, "predictor": "transition", "m": 8}, "ssd_bytes": 28923789312, "ssd_read_s": 2.122408108203672} | |
| 19 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:34:18", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 321.79704203736037, "prompt_tps": 59.09125432034499, "gen_tps": 54.69301303259165, "decode_tps_wall": 53.838927422647544, "itl_ms_p50": 17.771041952073574, "itl_ms_p95": 25.433875038288534, "itl_ms_p99": 26.668958947993815, "itl_ms_mean": 18.573921284682694, "total_s": 1.49195408297237, "peak_mem_gb": 1.815989832, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.7331506293138449, "cache_misses": 2629, "cache_hits": 7223, "stall_s": 0.6309308598283678, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 1426, "installed": 1432, "late": 2, "dup": 7238, "predictor": "transition", "m": 8}, "ssd_bytes": 14350417920, "ssd_read_s": 0.841039760154672} | |
| 20 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 0, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:34:19", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 367.4800000153482, "prompt_tps": 65.35415998940525, "gen_tps": 54.068865131055574, "decode_tps_wall": 53.22452254026847, "itl_ms_p50": 18.264499958604574, "itl_ms_p95": 25.95704107079655, "itl_ms_p99": 28.0275420518592, "itl_ms_mean": 18.788331999473037, "total_s": 1.5511449159821495, "peak_mem_gb": 1.96218824, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.7111154181606745, "cache_misses": 2981, "cache_hits": 7338, "stall_s": 0.7006638369057328, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 1478, "installed": 1478, "late": 1, "dup": 7418, "predictor": "transition", "m": 8}, "ssd_bytes": 15780151296, "ssd_read_s": 0.9043975193053484} | |
| 21 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "general", "prompt": "Describe the main differences between a Mac Studio and a Mac", "ts": "2026-11-01T01:34:21", "prompt_tokens": 18, "gen_tokens": 64, "ttft_ms": 351.4073749538511, "prompt_tps": 51.25972555473202, "gen_tps": 40.688361245121115, "decode_tps_wall": 40.05277111350156, "itl_ms_p50": 25.028917007148266, "itl_ms_p95": 32.00708294752985, "itl_ms_p99": 38.36241702083498, "itl_ms_mean": 24.967061509082598, "total_s": 1.9243322500260547, "peak_mem_gb": 1.885472408, "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often with more cores and higher clock speeds.\n- Larger display with a higher resolution", "cache_hit_rate": 0.6574613376179956, "cache_misses": 3411, "cache_hits": 6547, "stall_s": 0.9411007598973811, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 3473, "installed": 3472, "late": 39, "dup": 5239, "predictor": "transition", "m": 8}, "ssd_bytes": 24362090496, "ssd_read_s": 1.705476708826609} | |
| 22 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "code", "prompt": "Write a C function that computes the dot product of two floa", "ts": "2026-11-01T01:34:23", "prompt_tokens": 24, "gen_tokens": 64, "ttft_ms": 377.98908399417996, "prompt_tps": 63.54055938827112, "gen_tps": 54.33054019763219, "decode_tps_wall": 53.482049289396755, "itl_ms_p50": 18.067084020003676, "itl_ms_p95": 25.6848749704659, "itl_ms_p99": 27.61091699358076, "itl_ms_mean": 18.6978624283617, "total_s": 1.555954416980967, "peak_mem_gb": 1.969135056, "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot product computation using NEON intrinsics\n // Start by loading the arrays int", "cache_hit_rate": 0.7097586975482121, "cache_misses": 2995, "cache_hits": 7324, "stall_s": 0.7019232711754739, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 1480, "installed": 1481, "late": 1, "dup": 7416, "predictor": "transition", "m": 8}, "ssd_bytes": 15836774400, "ssd_read_s": 0.9054422654444352} | |
| 23 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "general", "prompt": "Summarize the history of the Rideau Canal in one paragraph.", "ts": "2026-11-01T01:34:25", "prompt_tokens": 14, "gen_tokens": 64, "ttft_ms": 313.21795796975493, "prompt_tps": 44.73164402840847, "gen_tps": 34.88030357261832, "decode_tps_wall": 34.335503098954135, "itl_ms_p50": 28.124041040427983, "itl_ms_p95": 39.43700005766004, "itl_ms_p99": 43.30104193650186, "itl_ms_mean": 29.12437301757376, "total_s": 2.1480534580769017, "peak_mem_gb": 1.790002148, "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Ottawa River. It was designed by Colonel John By and took approximately three years ", "cache_hit_rate": 0.6248190279214064, "cache_misses": 3628, "cache_hits": 6042, "stall_s": 1.0583573306212202, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 4518, "installed": 4515, "late": 40, "dup": 4034, "predictor": "transition", "m": 8}, "ssd_bytes": 28828237824, "ssd_read_s": 2.1118570929393172} | |
| 24 | +{"backend": "titan", "capacity": 16, "policy": "lru", "predictor": "transition", "m": 8, "prefetch_workers": 2, "nocache": false, "max_tokens": 64, "split": "heldout", "tag": "", "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", "cmd": "benchmarks/scripts/bench_decode.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --backend titan --capacity 16 --predictor transition --m 8 --traces benchmarks/raw/gateC/traces_olmoe_laptop --families general,code --split heldout --repeats 2 --out benchmarks/raw/gateC/prefetch_online_laptop/smoke_lazyinstall.jsonl", "env": {"host": "MacBookPro-de-simon-pierre-2.local", "chip": "Apple M5 Max", "ram_gb": 48, "macos": "27.0", "python": "3.12.13", "mlx": "0.32.4", "mlx_lm": "0.32.2", "numpy": "2.5.4", "git": "1df8d30"}, "seed": 0, "warmup": false, "repeat": 1, "family": "code", "prompt": "Write a bash script that lists the ten largest files under a", "ts": "2026-11-01T01:34:27", "prompt_tokens": 19, "gen_tokens": 64, "ttft_ms": 320.7784160040319, "prompt_tps": 59.27951201783087, "gen_tps": 56.203616918357326, "decode_tps_wall": 55.32592734451029, "itl_ms_p50": 17.169625032693148, "itl_ms_p95": 25.480874930508435, "itl_ms_p99": 26.342792087234557, "itl_ms_mean": 18.074708332913737, "total_s": 1.4594850409775972, "peak_mem_gb": 1.805061128, "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfiles=( $(ls -l \"$dir\" | awk '{print $5, $9}') )\n\n# Sort the array", "cache_hit_rate": 0.7330491270807957, "cache_misses": 2630, "cache_hits": 7222, "stall_s": 0.6244156584143639, "hits_device": 0, "layers_complete": 0, "prefetch": {"issued": 1425, "installed": 1428, "late": 2, "dup": 7239, "predictor": "transition", "m": 8}, "ssd_bytes": 14350417920, "ssd_read_s": 0.8336677193874493} | |
added
benchmarks/raw/gateC/traces_olmoe_v2_laptop/index.json
+631 −0
@@ -0,0 +1,631 @@ | ||
| 1 | +{ | |
| 2 | + "model": "/Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b", | |
| 3 | + "L": 16, | |
| 4 | + "E": 64, | |
| 5 | + "K": 8, | |
| 6 | + "max_tokens": 384, | |
| 7 | + "ts": "2026-11-01T01:29:47", | |
| 8 | + "cmd": "benchmarks/scripts/collect_traces.py --model /Users/simon-pierreboucher/.cache/huggingface/hub/models--mlx-community--OLMoE-1B-7B-0125-Instruct-4bit/snapshots/e23844197887b031e7ddddbb0b8959c5a6853a7b --workloads benchmarks/configs/workloads_v2.yaml --max-tokens 384 --out benchmarks/raw/gateC/traces_olmoe_v2_laptop", | |
| 9 | + "prompts": [ | |
| 10 | + { | |
| 11 | + "name": "general_fit_0", | |
| 12 | + "family": "general", | |
| 13 | + "split": "fit", | |
| 14 | + "prompt": "Explain in three sentences why mixture-of-experts models are memory-bound during", | |
| 15 | + "n_prompt": 20, | |
| 16 | + "n_gen": 93, | |
| 17 | + "seconds": 0.2754368749447167, | |
| 18 | + "text": "\n\nMixture-of-experts models involve combining predictions from multiple experts, each with its own set of weights. Durin" | |
| 19 | + }, | |
| 20 | + { | |
| 21 | + "name": "general_fit_1", | |
| 22 | + "family": "general", | |
| 23 | + "split": "fit", | |
| 24 | + "prompt": "Write a short paragraph describing the climate of Gatineau, Qu\u00e9bec, across the f", | |
| 25 | + "n_prompt": 21, | |
| 26 | + "n_gen": 272, | |
| 27 | + "seconds": 0.7447166669880971, | |
| 28 | + "text": "\n\nGatineau experiences a humid continental climate, with warm to hot summers and cold, snowy winters. In spring and fall" | |
| 29 | + }, | |
| 30 | + { | |
| 31 | + "name": "general_fit_2", | |
| 32 | + "family": "general", | |
| 33 | + "split": "fit", | |
| 34 | + "prompt": "Give five practical tips for a first-year university student learning to manage ", | |
| 35 | + "n_prompt": 17, | |
| 36 | + "n_gen": 227, | |
| 37 | + "seconds": 0.6229109159903601, | |
| 38 | + "text": "\n\n1. **Establish a Routine**: Create a daily schedule that includes classes, study time, breaks, and sleep. This helps i" | |
| 39 | + }, | |
| 40 | + { | |
| 41 | + "name": "general_fit_3", | |
| 42 | + "family": "general", | |
| 43 | + "split": "fit", | |
| 44 | + "prompt": "Describe how a hydroelectric dam converts the energy of falling water into elect", | |
| 45 | + "n_prompt": 16, | |
| 46 | + "n_gen": 374, | |
| 47 | + "seconds": 1.0221076660091057, | |
| 48 | + "text": "\n\nHydroelectric dams convert the potential energy of falling water into electrical energy through the process of hydraul" | |
| 49 | + }, | |
| 50 | + { | |
| 51 | + "name": "general_fit_4", | |
| 52 | + "family": "general", | |
| 53 | + "split": "fit", | |
| 54 | + "prompt": "Write a friendly email inviting colleagues to a Friday afternoon seminar on hous", | |
| 55 | + "n_prompt": 15, | |
| 56 | + "n_gen": 293, | |
| 57 | + "seconds": 0.790545874973759, | |
| 58 | + "text": "\n\nSubject: Invitation to Friday Afternoon Seminar on Housing Economics\n\nDear [Colleague's Name],\n\nI hope this message fi" | |
| 59 | + }, | |
| 60 | + { | |
| 61 | + "name": "general_fit_5", | |
| 62 | + "family": "general", | |
| 63 | + "split": "fit", | |
| 64 | + "prompt": "What are the main arguments for and against remote work in knowledge industries?", | |
| 65 | + "n_prompt": 14, | |
| 66 | + "n_gen": 384, | |
| 67 | + "seconds": 1.0376422079280019, | |
| 68 | + "text": "\n\nArguments for remote work in knowledge industries:\n\n1. Increased productivity: Remote work allows employees to create " | |
| 69 | + }, | |
| 70 | + { | |
| 71 | + "name": "general_fit_6", | |
| 72 | + "family": "general", | |
| 73 | + "split": "fit", | |
| 74 | + "prompt": "Explain the difference between weather and climate to a ten-year-old.", | |
| 75 | + "n_prompt": 16, | |
| 76 | + "n_gen": 384, | |
| 77 | + "seconds": 1.0408702499698848, | |
| 78 | + "text": "\n\nWeather is what's happening outside right now. It includes things like temperature, humidity, wind, and whether it's r" | |
| 79 | + }, | |
| 80 | + { | |
| 81 | + "name": "general_fit_7", | |
| 82 | + "family": "general", | |
| 83 | + "split": "fit", | |
| 84 | + "prompt": "Draft a two-paragraph product description for a quiet mechanical keyboard.", | |
| 85 | + "n_prompt": 14, | |
| 86 | + "n_gen": 241, | |
| 87 | + "seconds": 0.6577682920033112, | |
| 88 | + "text": "\n\nThe KBD-PRO Quiet Mechanical Keyboard is designed for users who value precision and comfort above all else. Crafted wi" | |
| 89 | + }, | |
| 90 | + { | |
| 91 | + "name": "general_fit_8", | |
| 92 | + "family": "general", | |
| 93 | + "split": "fit", | |
| 94 | + "prompt": "List the steps to prepare a simple vegetable soup, with approximate timings.", | |
| 95 | + "n_prompt": 15, | |
| 96 | + "n_gen": 205, | |
| 97 | + "seconds": 0.5663467920385301, | |
| 98 | + "text": "\n\n1. **Preparation of Ingredients:**\n - **Onion:** 10 minutes\n - **Carrot:** 5 minutes\n - **Celery:** 3 minutes\n " | |
| 99 | + }, | |
| 100 | + { | |
| 101 | + "name": "general_fit_9", | |
| 102 | + "family": "general", | |
| 103 | + "split": "fit", | |
| 104 | + "prompt": "Why do leaves change colour in autumn? Answer in one paragraph.", | |
| 105 | + "n_prompt": 13, | |
| 106 | + "n_gen": 185, | |
| 107 | + "seconds": 0.5095781669951975, | |
| 108 | + "text": "\n\nLeaves change colour in autumn because of a decrease in the amount of sunlight they receive. This reduction triggers t" | |
| 109 | + }, | |
| 110 | + { | |
| 111 | + "name": "general_heldout_0", | |
| 112 | + "family": "general", | |
| 113 | + "split": "heldout", | |
| 114 | + "prompt": "Describe the main differences between a Mac Studio and a MacBook Pro for scienti", | |
| 115 | + "n_prompt": 18, | |
| 116 | + "n_gen": 311, | |
| 117 | + "seconds": 0.8569220829522237, | |
| 118 | + "text": "\n\nMac Studio:\n- More powerful hardware options, including multiple GPUs and more RAM.\n- Higher-end CPU options, often wi" | |
| 119 | + }, | |
| 120 | + { | |
| 121 | + "name": "general_heldout_1", | |
| 122 | + "family": "general", | |
| 123 | + "split": "heldout", | |
| 124 | + "prompt": "Summarize the history of the Rideau Canal in one paragraph.", | |
| 125 | + "n_prompt": 14, | |
| 126 | + "n_gen": 384, | |
| 127 | + "seconds": 1.0413203749340028, | |
| 128 | + "text": "\n\nThe Rideau Canal, officially opened in 1832, was constructed by the British to connect Ottawa to Montreal via the Otta" | |
| 129 | + }, | |
| 130 | + { | |
| 131 | + "name": "general_heldout_2", | |
| 132 | + "family": "general", | |
| 133 | + "split": "heldout", | |
| 134 | + "prompt": "Write a short motivational note for someone starting to learn the piano as an ad", | |
| 135 | + "n_prompt": 16, | |
| 136 | + "n_gen": 260, | |
| 137 | + "seconds": 0.7053234579507262, | |
| 138 | + "text": "\n\nDear [Name],\n\nWelcome to the wonderful world of piano playing! You've made an exciting decision to learn this beautifu" | |
| 139 | + }, | |
| 140 | + { | |
| 141 | + "name": "general_heldout_3", | |
| 142 | + "family": "general", | |
| 143 | + "split": "heldout", | |
| 144 | + "prompt": "What should a visitor know before hiking in Gatineau Park in winter?", | |
| 145 | + "n_prompt": 16, | |
| 146 | + "n_gen": 384, | |
| 147 | + "seconds": 1.0523874170612544, | |
| 148 | + "text": "\n\n1. **Temperature and Weather Conditions**: The temperature in Gatineau Park can drop to as low as -30\u00b0C in winter. Vis" | |
| 149 | + }, | |
| 150 | + { | |
| 151 | + "name": "code_fit_0", | |
| 152 | + "family": "code", | |
| 153 | + "split": "fit", | |
| 154 | + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th Fibonacci number.\"\"\"\n", | |
| 155 | + "n_prompt": 21, | |
| 156 | + "n_gen": 325, | |
| 157 | + "seconds": 0.8913150830194354, | |
| 158 | + "text": " if n <= 0:\n return \"Invalid input\"\n elif n == 1:\n return 0\n elif n == 2:\n return 1\n el" | |
| 159 | + }, | |
| 160 | + { | |
| 161 | + "name": "code_fit_1", | |
| 162 | + "family": "code", | |
| 163 | + "split": "fit", | |
| 164 | + "prompt": "Write a Python function that parses a CSV file of timestamps and returns the med", | |
| 165 | + "n_prompt": 31, | |
| 166 | + "n_gen": 384, | |
| 167 | + "seconds": 1.068334334064275, | |
| 168 | + "text": "import csv\nfrom datetime import datetime\n\ndef parse_timestamps(file_path):\n \"\"\"\n Parses a CSV file containing time" | |
| 169 | + }, | |
| 170 | + { | |
| 171 | + "name": "code_fit_2", | |
| 172 | + "family": "code", | |
| 173 | + "split": "fit", | |
| 174 | + "prompt": "Implement an LRU cache class in Python with get and put methods in O(1).\n\n```pyt", | |
| 175 | + "n_prompt": 24, | |
| 176 | + "n_gen": 384, | |
| 177 | + "seconds": 1.0517179580638185, | |
| 178 | + "text": "class LRUCache:\n def __init__(self, capacity: int):\n self.capacity = capacity\n self.cache = {}\n " | |
| 179 | + }, | |
| 180 | + { | |
| 181 | + "name": "code_fit_3", | |
| 182 | + "family": "code", | |
| 183 | + "split": "fit", | |
| 184 | + "prompt": "Write a Python script that walks a directory tree and reports the total size per", | |
| 185 | + "n_prompt": 23, | |
| 186 | + "n_gen": 384, | |
| 187 | + "seconds": 1.062020415905863, | |
| 188 | + "text": "import os\n\ndef get_file_size(path):\n \"\"\"Return the size of a file.\"\"\"\n size = os.path.getsize(path)\n return siz" | |
| 189 | + }, | |
| 190 | + { | |
| 191 | + "name": "code_fit_4", | |
| 192 | + "family": "code", | |
| 193 | + "split": "fit", | |
| 194 | + "prompt": "Implement binary search over a sorted list of integers in Python and return the ", | |
| 195 | + "n_prompt": 25, | |
| 196 | + "n_gen": 354, | |
| 197 | + "seconds": 0.9759309579385445, | |
| 198 | + "text": "def binary_search(arr, x):\n left, right = 0, len(arr) - 1\n \n while left <= right:\n mid = (left + right) " | |
| 199 | + }, | |
| 200 | + { | |
| 201 | + "name": "code_fit_5", | |
| 202 | + "family": "code", | |
| 203 | + "split": "fit", | |
| 204 | + "prompt": "Write a SQL query that returns the three most recent orders per customer from ta", | |
| 205 | + "n_prompt": 32, | |
| 206 | + "n_gen": 384, | |
| 207 | + "seconds": 1.0527452089590952, | |
| 208 | + "text": "SELECT customer_id, order_date, amount\nFROM orders\nORDER BY order_date DESC\nLIMIT 3;\n```\n\nThis query will return the thr" | |
| 209 | + }, | |
| 210 | + { | |
| 211 | + "name": "code_fit_6", | |
| 212 | + "family": "code", | |
| 213 | + "split": "fit", | |
| 214 | + "prompt": "Write a JavaScript function that debounces another function with a configurable ", | |
| 215 | + "n_prompt": 20, | |
| 216 | + "n_gen": 384, | |
| 217 | + "seconds": 1.0560737090418115, | |
| 218 | + "text": "function debounce(func, delay) {\n let timeout;\n\n return function (...args) {\n clearTimeout(timeout);\n timeout = " | |
| 219 | + }, | |
| 220 | + { | |
| 221 | + "name": "code_fit_7", | |
| 222 | + "family": "code", | |
| 223 | + "split": "fit", | |
| 224 | + "prompt": "Write a Rust function that counts word frequencies in a string and returns them ", | |
| 225 | + "n_prompt": 23, | |
| 226 | + "n_gen": 349, | |
| 227 | + "seconds": 0.9573940000263974, | |
| 228 | + "text": "fn count_words(text: &str) -> HashMap<String, i32> {\n let words: Vec<&str> = text.split_whitespace().collect();\n l" | |
| 229 | + }, | |
| 230 | + { | |
| 231 | + "name": "code_fit_8", | |
| 232 | + "family": "code", | |
| 233 | + "split": "fit", | |
| 234 | + "prompt": "class Matrix:\n \"\"\"Minimal dense matrix with multiplication and transpose.\"\"\"\n", | |
| 235 | + "n_prompt": 27, | |
| 236 | + "n_gen": 384, | |
| 237 | + "seconds": 1.0742902499623597, | |
| 238 | + "text": " self.rows = rows\n self.data = [None] * rows\n\n def add_entry(self, row, entry):\n \"\"\"Add an entry" | |
| 239 | + }, | |
| 240 | + { | |
| 241 | + "name": "code_fit_9", | |
| 242 | + "family": "code", | |
| 243 | + "split": "fit", | |
| 244 | + "prompt": "Write a Python generator that yields sliding windows of size k over an iterable.", | |
| 245 | + "n_prompt": 21, | |
| 246 | + "n_gen": 276, | |
| 247 | + "seconds": 0.7710703329648823, | |
| 248 | + "text": "def sliding_windows(iterable, k):\n \"\"\"\n Yields sliding windows of size k over an iterable.\n\n :param iterable: T" | |
| 249 | + }, | |
| 250 | + { | |
| 251 | + "name": "code_heldout_0", | |
| 252 | + "family": "code", | |
| 253 | + "split": "heldout", | |
| 254 | + "prompt": "Write a C function that computes the dot product of two float arrays using NEON ", | |
| 255 | + "n_prompt": 24, | |
| 256 | + "n_gen": 384, | |
| 257 | + "seconds": 1.0711488750530407, | |
| 258 | + "text": "#include <arm_neon.h>\n\nfloat dot_product_neon(const float *a, const float *b, int len) {\n // TODO: Implement the dot " | |
| 259 | + }, | |
| 260 | + { | |
| 261 | + "name": "code_heldout_1", | |
| 262 | + "family": "code", | |
| 263 | + "split": "heldout", | |
| 264 | + "prompt": "Write a bash script that lists the ten largest files under a directory.\n\n```bash", | |
| 265 | + "n_prompt": 19, | |
| 266 | + "n_gen": 365, | |
| 267 | + "seconds": 1.0241958330152556, | |
| 268 | + "text": "#!/bin/bash\n\n# Specify the directory\ndir=\"/path/to/directory\"\n\n# Get the size of each file and store it in an array\nfile" | |
| 269 | + }, | |
| 270 | + { | |
| 271 | + "name": "code_heldout_2", | |
| 272 | + "family": "code", | |
| 273 | + "split": "heldout", | |
| 274 | + "prompt": "Implement Dijkstra's shortest path in Python using a binary heap.\n\n```python\n", | |
| 275 | + "n_prompt": 20, | |
| 276 | + "n_gen": 384, | |
| 277 | + "seconds": 1.0595750829670578, | |
| 278 | + "text": "import heapq\n\ndef dijkstra(graph, start):\n # Create a priority queue (binary heap)\n pq = []\n for node, distance" | |
| 279 | + }, | |
| 280 | + { | |
| 281 | + "name": "code_heldout_3", | |
| 282 | + "family": "code", | |
| 283 | + "split": "heldout", | |
| 284 | + "prompt": "Write a Go function that reads lines from stdin and prints the longest one.\n\n```", | |
| 285 | + "n_prompt": 21, | |
| 286 | + "n_gen": 384, | |
| 287 | + "seconds": 1.0644185839919373, | |
| 288 | + "text": "package main\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n)\n\nfunc main() {\n\tfmt.Println(\"Enter your lines (press Ctrl+D to finish):\")\n\tva" | |
| 289 | + }, | |
| 290 | + { | |
| 291 | + "name": "reasoning_fit_0", | |
| 292 | + "family": "reasoning", | |
| 293 | + "split": "fit", | |
| 294 | + "prompt": "Q: If a train travels 60 km in 45 minutes, what is its average speed in km/h? Th", | |
| 295 | + "n_prompt": 30, | |
| 296 | + "n_gen": 38, | |
| 297 | + "seconds": 0.11871404096018523, | |
| 298 | + "text": " 45 minutes is equal to 45 / 60 = 0.75 hours. The average speed is 60 km / 0.75 hours = 80 km/h. So the answer is 80.|||" | |
| 299 | + }, | |
| 300 | + { | |
| 301 | + "name": "reasoning_fit_1", | |
| 302 | + "family": "reasoning", | |
| 303 | + "split": "fit", | |
| 304 | + "prompt": "Q: A rectangle has a perimeter of 36 cm and a length twice its width. What is it", | |
| 305 | + "n_prompt": 30, | |
| 306 | + "n_gen": 101, | |
| 307 | + "seconds": 0.29207833390682936, | |
| 308 | + "text": " Let the width be w cm. Then the length is 2w cm. The perimeter of a rectangle is given by 2(length + width) = 36 cm. Su" | |
| 309 | + }, | |
| 310 | + { | |
| 311 | + "name": "reasoning_fit_2", | |
| 312 | + "family": "reasoning", | |
| 313 | + "split": "fit", | |
| 314 | + "prompt": "Q: Three friends split a bill of 84 dollars; one pays twice as much as each of t", | |
| 315 | + "n_prompt": 36, | |
| 316 | + "n_gen": 67, | |
| 317 | + "seconds": 0.197022125008516, | |
| 318 | + "text": " Let x be the amount each of the two friends pays. The other friend pays x also. Therefore, the total bill is x + x + x " | |
| 319 | + }, | |
| 320 | + { | |
| 321 | + "name": "reasoning_fit_3", | |
| 322 | + "family": "reasoning", | |
| 323 | + "split": "fit", | |
| 324 | + "prompt": "Q: A store sells pens at 3 for 2 dollars. How much do 18 pens cost? Think step b", | |
| 325 | + "n_prompt": 27, | |
| 326 | + "n_gen": 54, | |
| 327 | + "seconds": 0.16230712505057454, | |
| 328 | + "text": " First find the number of sets of 3 pens in 18 pens: 18 pens / 3 pens per set = 6 sets. Then multiply the number of sets" | |
| 329 | + }, | |
| 330 | + { | |
| 331 | + "name": "reasoning_fit_4", | |
| 332 | + "family": "reasoning", | |
| 333 | + "split": "fit", | |
| 334 | + "prompt": "Q: If today is Wednesday, what day of the week will it be in 100 days? Think ste", | |
| 335 | + "n_prompt": 27, | |
| 336 | + "n_gen": 81, | |
| 337 | + "seconds": 0.2316902909660712, | |
| 338 | + "text": " Today is Wednesday, which is between Monday and Friday. In 100 days, there are 7 * 100 = 700 hours. Since there are 24 " | |
| 339 | + }, | |
| 340 | + { | |
| 341 | + "name": "reasoning_fit_5", | |
| 342 | + "family": "reasoning", | |
| 343 | + "split": "fit", | |
| 344 | + "prompt": "Q: A car uses 7 litres per 100 km. How many litres for a 350 km trip, and the co", | |
| 345 | + "n_prompt": 38, | |
| 346 | + "n_gen": 49, | |
| 347 | + "seconds": 0.14697608293499798, | |
| 348 | + "text": " The car will use 7 * (350 / 100) = 245 litres for the trip. At a cost of 1.60 per litre, the total cost will be 245 * 1" | |
| 349 | + }, | |
| 350 | + { | |
| 351 | + "name": "reasoning_fit_6", | |
| 352 | + "family": "reasoning", | |
| 353 | + "split": "fit", | |
| 354 | + "prompt": "Q: The sum of two consecutive even numbers is 54. What are they? Think step by s", | |
| 355 | + "n_prompt": 24, | |
| 356 | + "n_gen": 93, | |
| 357 | + "seconds": 0.2644914169795811, | |
| 358 | + "text": " Let the first even number be x. The next consecutive even number will be x + 2. The sum of the two numbers is x + (x + " | |
| 359 | + }, | |
| 360 | + { | |
| 361 | + "name": "reasoning_fit_7", | |
| 362 | + "family": "reasoning", | |
| 363 | + "split": "fit", | |
| 364 | + "prompt": "Q: A recipe for 4 people needs 300 g of flour. How much for 10 people? Think ste", | |
| 365 | + "n_prompt": 27, | |
| 366 | + "n_gen": 63, | |
| 367 | + "seconds": 0.18630754109472036, | |
| 368 | + "text": " The amount of flour needed for 4 people is 300 g, so for 10 people, we need to calculate how many times 4 is in 10: 10 " | |
| 369 | + }, | |
| 370 | + { | |
| 371 | + "name": "reasoning_fit_8", | |
| 372 | + "family": "reasoning", | |
| 373 | + "split": "fit", | |
| 374 | + "prompt": "Q: A clock shows 3:15. What is the angle between the hour and minute hands? Thin", | |
| 375 | + "n_prompt": 28, | |
| 376 | + "n_gen": 56, | |
| 377 | + "seconds": 0.17154558305628598, | |
| 378 | + "text": " The minute hand has moved 1/12 of the way from 3 to 4 because there are 12 hours on a clock. The angle between the hour" | |
| 379 | + }, | |
| 380 | + { | |
| 381 | + "name": "reasoning_fit_9", | |
| 382 | + "family": "reasoning", | |
| 383 | + "split": "fit", | |
| 384 | + "prompt": "Q: If 5 machines make 5 widgets in 5 minutes, how long do 100 machines take to m", | |
| 385 | + "n_prompt": 33, | |
| 386 | + "n_gen": 62, | |
| 387 | + "seconds": 0.18710525007918477, | |
| 388 | + "text": " If 5 machines make 5 widgets in 5 minutes, then each machine makes 1 widget in 5 minutes. If there are 100 machines, th" | |
| 389 | + }, | |
| 390 | + { | |
| 391 | + "name": "reasoning_heldout_0", | |
| 392 | + "family": "reasoning", | |
| 393 | + "split": "heldout", | |
| 394 | + "prompt": "Q: A tank fills in 6 hours with pipe A and 9 hours with pipe B. How long with bo", | |
| 395 | + "n_prompt": 32, | |
| 396 | + "n_gen": 150, | |
| 397 | + "seconds": 0.42938487499486655, | |
| 398 | + "text": " If pipe A fills the tank in 6 hours, then the rate of pipe A is 1 / 6 of the tank per hour. If pipe B fills the tank in" | |
| 399 | + }, | |
| 400 | + { | |
| 401 | + "name": "reasoning_heldout_1", | |
| 402 | + "family": "reasoning", | |
| 403 | + "split": "heldout", | |
| 404 | + "prompt": "Q: What is the smallest positive integer divisible by 2, 3, 4, 5 and 6? Think st", | |
| 405 | + "n_prompt": 29, | |
| 406 | + "n_gen": 103, | |
| 407 | + "seconds": 0.30200008302927017, | |
| 408 | + "text": " The least common multiple (LCM) of 2, 3, 4, 5, and 6 is the smallest positive integer that is divisible by all of these" | |
| 409 | + }, | |
| 410 | + { | |
| 411 | + "name": "reasoning_heldout_2", | |
| 412 | + "family": "reasoning", | |
| 413 | + "split": "heldout", | |
| 414 | + "prompt": "Q: A shirt costs 40 dollars after a 20 percent discount. What was the original p", | |
| 415 | + "n_prompt": 27, | |
| 416 | + "n_gen": 70, | |
| 417 | + "seconds": 0.20663741591852158, | |
| 418 | + "text": " Let x be the original price of the shirt. After a 20% discount, the price is x - (20 / 100) * x = 0.8x. We know that 0." | |
| 419 | + }, | |
| 420 | + { | |
| 421 | + "name": "reasoning_heldout_3", | |
| 422 | + "family": "reasoning", | |
| 423 | + "split": "heldout", | |
| 424 | + "prompt": "Q: Two dice are rolled. What is the probability that the sum is 7? Think step by", | |
| 425 | + "n_prompt": 25, | |
| 426 | + "n_gen": 87, | |
| 427 | + "seconds": 0.2548181250458583, | |
| 428 | + "text": " There are 6 * 6 = 36 possible outcomes when rolling two dice. The sum of 7 can be achieved by several combinations of d" | |
| 429 | + }, | |
| 430 | + { | |
| 431 | + "name": "multilingual_fit_0", | |
| 432 | + "family": "multilingual", | |
| 433 | + "split": "fit", | |
| 434 | + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent seulement une fraction de leurs param\u00e8t", | |
| 435 | + "n_prompt": 26, | |
| 436 | + "n_gen": 45, | |
| 437 | + "seconds": 0.13864541705697775, | |
| 438 | + "text": " les autres sont consid\u00e9r\u00e9s comme des \"non-inform\u00e9s\".\n\nTranslate to English.\nTranslation: Hybrid expert models only acti" | |
| 439 | + }, | |
| 440 | + { | |
| 441 | + "name": "multilingual_fit_1", | |
| 442 | + "family": "multilingual", | |
| 443 | + "split": "fit", | |
| 444 | + "prompt": "Explique en fran\u00e7ais pourquoi la m\u00e9moire unifi\u00e9e des puces Apple est utile pour ", | |
| 445 | + "n_prompt": 39, | |
| 446 | + "n_gen": 327, | |
| 447 | + "seconds": 0.9169528750935569, | |
| 448 | + "text": "\n\nLes puces Apple utilisent une m\u00e9moire unifi\u00e9e, ce qui signifie qu'elles ont une seule m\u00e9moire externe qui peut stocker" | |
| 449 | + }, | |
| 450 | + { | |
| 451 | + "name": "multilingual_fit_2", | |
| 452 | + "family": "multilingual", | |
| 453 | + "split": "fit", | |
| 454 | + "prompt": "R\u00e9dige un court paragraphe sur l'histoire de la ville de Qu\u00e9bec.", | |
| 455 | + "n_prompt": 21, | |
| 456 | + "n_gen": 384, | |
| 457 | + "seconds": 1.0654266669880599, | |
| 458 | + "text": "\n\nDans les ann\u00e9es 1950, l'architecture qu\u00e9b\u00e9coise \u00e9tait domin\u00e9e par l'esth\u00e9tique traditionnelle, avec des b\u00e2timents en b" | |
| 459 | + }, | |
| 460 | + { | |
| 461 | + "name": "multilingual_fit_3", | |
| 462 | + "family": "multilingual", | |
| 463 | + "split": "fit", | |
| 464 | + "prompt": "Quels sont les avantages du transport en commun dans une ville de taille moyenne", | |
| 465 | + "n_prompt": 28, | |
| 466 | + "n_gen": 236, | |
| 467 | + "seconds": 0.6601781670469791, | |
| 468 | + "text": "\n\nLes avantages du transport en commun dans une ville de taille moyenne incluent :\n\n1. R\u00e9duction des \u00e9missions de gaz \u00e0 " | |
| 469 | + }, | |
| 470 | + { | |
| 471 | + "name": "multilingual_fit_4", | |
| 472 | + "family": "multilingual", | |
| 473 | + "split": "fit", | |
| 474 | + "prompt": "Erkl\u00e4re in drei S\u00e4tzen, warum Fahrr\u00e4der in St\u00e4dten n\u00fctzlich sind.", | |
| 475 | + "n_prompt": 29, | |
| 476 | + "n_gen": 384, | |
| 477 | + "seconds": 1.0639994579833, | |
| 478 | + "text": "\n\nFahrr\u00e4der sind in St\u00e4dten n\u00fctzlich, weil sie die Mobilit\u00e4t erheblich verbessern, die Energieeinsparung erm\u00f6glichen und" | |
| 479 | + }, | |
| 480 | + { | |
| 481 | + "name": "multilingual_fit_5", | |
| 482 | + "family": "multilingual", | |
| 483 | + "split": "fit", | |
| 484 | + "prompt": "Describe en espa\u00f1ol c\u00f3mo se prepara un buen caf\u00e9 con una cafetera italiana.", | |
| 485 | + "n_prompt": 24, | |
| 486 | + "n_gen": 249, | |
| 487 | + "seconds": 0.6920398330548778, | |
| 488 | + "text": "\n\nUna cafetera italiana es un dispositivo simple pero eficiente para preparar caf\u00e9. Primero, se calienta el recipiente c" | |
| 489 | + }, | |
| 490 | + { | |
| 491 | + "name": "multilingual_fit_6", | |
| 492 | + "family": "multilingual", | |
| 493 | + "split": "fit", | |
| 494 | + "prompt": "Explique en fran\u00e7ais la diff\u00e9rence entre un pr\u00eat hypoth\u00e9caire \u00e0 taux fixe et \u00e0 t", | |
| 495 | + "n_prompt": 29, | |
| 496 | + "n_gen": 384, | |
| 497 | + "seconds": 1.0617762500187382, | |
| 498 | + "text": "\n\n**Pr\u00eat hypoth\u00e9caire \u00e0 taux fixe:**\nUn pr\u00eat hypoth\u00e9caire \u00e0 taux fixe signifie que le taux d'int\u00e9r\u00eat est constant pendan" | |
| 499 | + }, | |
| 500 | + { | |
| 501 | + "name": "multilingual_fit_7", | |
| 502 | + "family": "multilingual", | |
| 503 | + "split": "fit", | |
| 504 | + "prompt": "Scrivi un breve paragrafo sulla cucina tradizionale della Toscana.", | |
| 505 | + "n_prompt": 21, | |
| 506 | + "n_gen": 244, | |
| 507 | + "seconds": 0.6741328750504181, | |
| 508 | + "text": "\n\nThe Tuscan cuisine is characterized by its simplicity and the use of fresh, high-quality ingredients. Some of the most" | |
| 509 | + }, | |
| 510 | + { | |
| 511 | + "name": "multilingual_heldout_0", | |
| 512 | + "family": "multilingual", | |
| 513 | + "split": "heldout", | |
| 514 | + "prompt": "D\u00e9cris en quelques phrases le fonctionnement d'un cache LRU.", | |
| 515 | + "n_prompt": 16, | |
| 516 | + "n_gen": 24, | |
| 517 | + "seconds": 0.07363499992061406, | |
| 518 | + "text": "\n\nTranslate to English.\nTranslation: Describe in a few sentences how a LRU cache works.|||IP_ADDRESS|||" | |
| 519 | + }, | |
| 520 | + { | |
| 521 | + "name": "multilingual_heldout_1", | |
| 522 | + "family": "multilingual", | |
| 523 | + "split": "heldout", | |
| 524 | + "prompt": "\u00bfCu\u00e1les son las principales causas del cambio clim\u00e1tico? Responde brevemente en ", | |
| 525 | + "n_prompt": 28, | |
| 526 | + "n_gen": 56, | |
| 527 | + "seconds": 0.16646854195278138, | |
| 528 | + "text": "\n\nLos principales factores incluyen la emisi\u00f3n de gases de efecto invernadero, la deforestaci\u00f3n, la contaminaci\u00f3n por me" | |
| 529 | + }, | |
| 530 | + { | |
| 531 | + "name": "multilingual_heldout_2", | |
| 532 | + "family": "multilingual", | |
| 533 | + "split": "heldout", | |
| 534 | + "prompt": "Beschreibe kurz, wie ein Mietvertrag in Deutschland funktioniert.", | |
| 535 | + "n_prompt": 22, | |
| 536 | + "n_gen": 384, | |
| 537 | + "seconds": 1.070310291950591, | |
| 538 | + "text": "\n\nIn Deutschland gibt es mehrere Mietvertragsmodelle. Ein typisches Mietvertragsmodell besteht aus folgenden Punkten:\n\n1" | |
| 539 | + }, | |
| 540 | + { | |
| 541 | + "name": "longctx_fit_0", | |
| 542 | + "family": "longctx", | |
| 543 | + "split": "fit", | |
| 544 | + "prompt": "You are an assistant helping an economics professor. Context: the course covers ", | |
| 545 | + "n_prompt": 126, | |
| 546 | + "n_gen": 384, | |
| 547 | + "seconds": 1.150788791012019, | |
| 548 | + "text": "\n\nHere are the detailed instructions for the lab:\n\n**Objective:**\nStudents will learn how to estimate a hedonic regressi" | |
| 549 | + }, | |
| 550 | + { | |
| 551 | + "name": "longctx_fit_1", | |
| 552 | + "family": "longctx", | |
| 553 | + "split": "fit", | |
| 554 | + "prompt": "Context: a small laboratory operates a cluster of twenty-nine Apple Silicon comp", | |
| 555 | + "n_prompt": 100, | |
| 556 | + "n_gen": 384, | |
| 557 | + "seconds": 1.0995475410018116, | |
| 558 | + "text": "\n\nDaily Tasks:\n1. Check the status of all computers and ensure all systems are running smoothly.\n2. Verify that all comp" | |
| 559 | + }, | |
| 560 | + { | |
| 561 | + "name": "longctx_fit_2", | |
| 562 | + "family": "longctx", | |
| 563 | + "split": "fit", | |
| 564 | + "prompt": "Context: a municipality in Quebec is revising its property assessment roll. Curr", | |
| 565 | + "n_prompt": 92, | |
| 566 | + "n_gen": 384, | |
| 567 | + "seconds": 1.0857783330138773, | |
| 568 | + "text": "\n\n--\n\n**Briefing Note**\n\n**To:** City Council\n\n**From:** [Your Name]\n\n**Date:** [Today's Date]\n\n**Subject:** Methodology" | |
| 569 | + }, | |
| 570 | + { | |
| 571 | + "name": "longctx_heldout_0", | |
| 572 | + "family": "longctx", | |
| 573 | + "split": "heldout", | |
| 574 | + "prompt": "Context: a graduate student is designing a benchmark to compare inference runtim", | |
| 575 | + "n_prompt": 104, | |
| 576 | + "n_gen": 384, | |
| 577 | + "seconds": 1.0925364589784294, | |
| 578 | + "text": "\n\nExperimental Protocol:\n\n1. **Workload Definition**:\n - **Static Text**: A large corpus of text data, \\(W_s\\), consis" | |
| 579 | + }, | |
| 580 | + { | |
| 581 | + "name": "drift_fit_0", | |
| 582 | + "family": "drift", | |
| 583 | + "split": "fit", | |
| 584 | + "prompt": "First, write a Python function that reverses a linked list.\n\n```python\n", | |
| 585 | + "n_prompt": 18, | |
| 586 | + "n_gen": 384, | |
| 587 | + "seconds": 1.0637037919368595, | |
| 588 | + "text": "class Node:\n def __init__(self, data=None):\n self.data = data\n self.next = None\n\nclass LinkedList:\n " | |
| 589 | + }, | |
| 590 | + { | |
| 591 | + "name": "drift_fit_1", | |
| 592 | + "family": "drift", | |
| 593 | + "split": "fit", | |
| 594 | + "prompt": "Explain the rules of ice hockey in two paragraphs, then write a SQL query counti", | |
| 595 | + "n_prompt": 29, | |
| 596 | + "n_gen": 384, | |
| 597 | + "seconds": 1.0741227080579847, | |
| 598 | + "text": "\n\nIce hockey is played on an ice rink with two teams, each aiming to score goals by shooting a puck into the opposing te" | |
| 599 | + }, | |
| 600 | + { | |
| 601 | + "name": "drift_fit_2", | |
| 602 | + "family": "drift", | |
| 603 | + "split": "fit", | |
| 604 | + "prompt": "R\u00e9sume en fran\u00e7ais l'intrigue d'un roman policier imaginaire, puis donne la solu", | |
| 605 | + "n_prompt": 47, | |
| 606 | + "n_gen": 384, | |
| 607 | + "seconds": 1.091406665975228, | |
| 608 | + "text": "\n\n**R\u00e9sum\u00e9 en anglais:**\n\nThe plot of a whimsical detective novel involves a mysterious disappearance in a small town. T" | |
| 609 | + }, | |
| 610 | + { | |
| 611 | + "name": "drift_heldout_0", | |
| 612 | + "family": "drift", | |
| 613 | + "split": "heldout", | |
| 614 | + "prompt": "Describe a sunset over the Ottawa River in one paragraph, then implement quickso", | |
| 615 | + "n_prompt": 25, | |
| 616 | + "n_gen": 384, | |
| 617 | + "seconds": 1.0839699999196455, | |
| 618 | + "text": "def quicksort(arr):\n if len(arr) <= 1:\n return arr\n pivot = arr[len(arr) - 1]\n left = [x for x in arr if" | |
| 619 | + }, | |
| 620 | + { | |
| 621 | + "name": "drift_heldout_1", | |
| 622 | + "family": "drift", | |
| 623 | + "split": "heldout", | |
| 624 | + "prompt": "Q: What is 17 times 23? Think step by step.\nA: Then write a haiku about autumn i", | |
| 625 | + "n_prompt": 27, | |
| 626 | + "n_gen": 72, | |
| 627 | + "seconds": 0.21239216707181185, | |
| 628 | + "text": "\nStep 1: Multiply 17 by 23.\n17 * 23 = 391\n\nStep 2: Write a haiku about autumn.\nLeaves fall, one by one.\nQuiet whispers, " | |
| 629 | + } | |
| 630 | + ] | |
| 631 | +} | |
| \ No newline at end of file | ||
added
benchmarks/raw/tests/greedy_cap16_20261101T013103.json
+14 −0
@@ -0,0 +1,14 @@ | ||
| 1 | +[ | |
| 2 | + { | |
| 3 | + "prompt": "The capital of Canada is", | |
| 4 | + "ref": "Ottawa.\n\nOttawa is the capital of Canada, and it is located in the province of Ontario. It is situated on the south bank of the Ottawa", | |
| 5 | + "titan": "Ottawa.\n\nOttawa is the capital of Canada, and it is located in the province of Ontario. It is situated on the south bank of the Ottawa", | |
| 6 | + "equal": true | |
| 7 | + }, | |
| 8 | + { | |
| 9 | + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th", | |
| 10 | + "ref": " if n <= 0:\n return \"Invalid input\"\n elif n == 1:\n return 0\n elif n == 2:\n", | |
| 11 | + "titan": " if n <= 0:\n return \"Invalid input\"\n elif n == 1:\n return 0\n elif n == 2:\n", | |
| 12 | + "equal": true | |
| 13 | + } | |
| 14 | +] | |
| \ No newline at end of file | ||
added
benchmarks/raw/tests/greedy_cap16_20261101T013347.json
+14 −0
@@ -0,0 +1,14 @@ | ||
| 1 | +[ | |
| 2 | + { | |
| 3 | + "prompt": "The capital of Canada is", | |
| 4 | + "ref": "Ottawa.\n\nOttawa is the capital of Canada, and it is located in the province of Ontario. It is situated on the south bank of the Ottawa", | |
| 5 | + "titan": "Ottawa.\n\nOttawa is the capital of Canada, and it is located in the province of Ontario. It is situated on the south bank of the Ottawa", | |
| 6 | + "equal": true | |
| 7 | + }, | |
| 8 | + { | |
| 9 | + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th", | |
| 10 | + "ref": " if n <= 0:\n return \"Invalid input\"\n elif n == 1:\n return 0\n elif n == 2:\n", | |
| 11 | + "titan": " if n <= 0:\n return \"Invalid input\"\n elif n == 1:\n return 0\n elif n == 2:\n", | |
| 12 | + "equal": true | |
| 13 | + } | |
| 14 | +] | |
| \ No newline at end of file | ||
added
benchmarks/raw/tests/numerical_cap16_2026-11-01T013056.json
+48 −0
@@ -0,0 +1,48 @@ | ||
| 1 | +{ | |
| 2 | + "capacity": 16, | |
| 3 | + "prompts": [ | |
| 4 | + { | |
| 5 | + "prompt": "The capital of Canada is", | |
| 6 | + "tokens": 5, | |
| 7 | + "max_abs_dlogit": 0.0, | |
| 8 | + "argmax_equal": true, | |
| 9 | + "hit_rate": 0.2580101180438449, | |
| 10 | + "misses": 440, | |
| 11 | + "stall_s": 0.1710000493330881, | |
| 12 | + "layers_routed": 46 | |
| 13 | + }, | |
| 14 | + { | |
| 15 | + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th", | |
| 16 | + "tokens": 21, | |
| 17 | + "max_abs_dlogit": 0.0, | |
| 18 | + "argmax_equal": true, | |
| 19 | + "hit_rate": 0.42512908777969016, | |
| 20 | + "misses": 1002, | |
| 21 | + "stall_s": 0.39930295187514275, | |
| 22 | + "layers_routed": 127 | |
| 23 | + }, | |
| 24 | + { | |
| 25 | + "prompt": "Q: If a train travels 60 km in 45 minute", | |
| 26 | + "tokens": 25, | |
| 27 | + "max_abs_dlogit": 0.0, | |
| 28 | + "argmax_equal": true, | |
| 29 | + "hit_rate": 0.41904344010530936, | |
| 30 | + "misses": 1324, | |
| 31 | + "stall_s": 0.5233453382970765, | |
| 32 | + "layers_routed": 161 | |
| 33 | + }, | |
| 34 | + { | |
| 35 | + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent", | |
| 36 | + "tokens": 26, | |
| 37 | + "max_abs_dlogit": 0.0, | |
| 38 | + "argmax_equal": true, | |
| 39 | + "hit_rate": 0.45278969957081544, | |
| 40 | + "misses": 1275, | |
| 41 | + "stall_s": 0.5041713331593201, | |
| 42 | + "layers_routed": 164 | |
| 43 | + } | |
| 44 | + ], | |
| 45 | + "tol": 0.001, | |
| 46 | + "ts": "2026-11-01T01:30:56", | |
| 47 | + "max_abs_dlogit_worst": 0.0 | |
| 48 | +} | |
| \ No newline at end of file | ||
added
benchmarks/raw/tests/numerical_cap16_2026-11-01T013342.json
+48 −0
@@ -0,0 +1,48 @@ | ||
| 1 | +{ | |
| 2 | + "capacity": 16, | |
| 3 | + "prompts": [ | |
| 4 | + { | |
| 5 | + "prompt": "The capital of Canada is", | |
| 6 | + "tokens": 5, | |
| 7 | + "max_abs_dlogit": 0.0, | |
| 8 | + "argmax_equal": true, | |
| 9 | + "hit_rate": 0.2580101180438449, | |
| 10 | + "misses": 440, | |
| 11 | + "stall_s": 0.08382649859413505, | |
| 12 | + "layers_routed": 46 | |
| 13 | + }, | |
| 14 | + { | |
| 15 | + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th", | |
| 16 | + "tokens": 21, | |
| 17 | + "max_abs_dlogit": 0.0, | |
| 18 | + "argmax_equal": true, | |
| 19 | + "hit_rate": 0.42512908777969016, | |
| 20 | + "misses": 1002, | |
| 21 | + "stall_s": 0.18693941307719797, | |
| 22 | + "layers_routed": 127 | |
| 23 | + }, | |
| 24 | + { | |
| 25 | + "prompt": "Q: If a train travels 60 km in 45 minute", | |
| 26 | + "tokens": 25, | |
| 27 | + "max_abs_dlogit": 0.0, | |
| 28 | + "argmax_equal": true, | |
| 29 | + "hit_rate": 0.41904344010530936, | |
| 30 | + "misses": 1324, | |
| 31 | + "stall_s": 0.2490453717764467, | |
| 32 | + "layers_routed": 161 | |
| 33 | + }, | |
| 34 | + { | |
| 35 | + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent", | |
| 36 | + "tokens": 26, | |
| 37 | + "max_abs_dlogit": 0.0, | |
| 38 | + "argmax_equal": true, | |
| 39 | + "hit_rate": 0.45278969957081544, | |
| 40 | + "misses": 1275, | |
| 41 | + "stall_s": 0.23546966060530394, | |
| 42 | + "layers_routed": 164 | |
| 43 | + } | |
| 44 | + ], | |
| 45 | + "tol": 0.001, | |
| 46 | + "ts": "2026-11-01T01:33:42", | |
| 47 | + "max_abs_dlogit_worst": 0.0 | |
| 48 | +} | |
| \ No newline at end of file | ||
added
benchmarks/raw/tests/numerical_cap32_2026-11-01T013055.json
+48 −0
@@ -0,0 +1,48 @@ | ||
| 1 | +{ | |
| 2 | + "capacity": 32, | |
| 3 | + "prompts": [ | |
| 4 | + { | |
| 5 | + "prompt": "The capital of Canada is", | |
| 6 | + "tokens": 5, | |
| 7 | + "max_abs_dlogit": 0.0, | |
| 8 | + "argmax_equal": true, | |
| 9 | + "hit_rate": 0.0, | |
| 10 | + "misses": 430, | |
| 11 | + "stall_s": 0.16276496497448534, | |
| 12 | + "layers_routed": 16 | |
| 13 | + }, | |
| 14 | + { | |
| 15 | + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th", | |
| 16 | + "tokens": 21, | |
| 17 | + "max_abs_dlogit": 0.0, | |
| 18 | + "argmax_equal": true, | |
| 19 | + "hit_rate": 0.6060606060606061, | |
| 20 | + "misses": 429, | |
| 21 | + "stall_s": 0.1718995354603976, | |
| 22 | + "layers_routed": 40 | |
| 23 | + }, | |
| 24 | + { | |
| 25 | + "prompt": "Q: If a train travels 60 km in 45 minute", | |
| 26 | + "tokens": 25, | |
| 27 | + "max_abs_dlogit": 0.0, | |
| 28 | + "argmax_equal": true, | |
| 29 | + "hit_rate": 0.6090458488228006, | |
| 30 | + "misses": 631, | |
| 31 | + "stall_s": 0.2528055729344487, | |
| 32 | + "layers_routed": 58 | |
| 33 | + }, | |
| 34 | + { | |
| 35 | + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent", | |
| 36 | + "tokens": 26, | |
| 37 | + "max_abs_dlogit": 0.0, | |
| 38 | + "argmax_equal": true, | |
| 39 | + "hit_rate": 0.6224188790560472, | |
| 40 | + "misses": 512, | |
| 41 | + "stall_s": 0.20383387978654355, | |
| 42 | + "layers_routed": 49 | |
| 43 | + } | |
| 44 | + ], | |
| 45 | + "tol": 0.001, | |
| 46 | + "ts": "2026-11-01T01:30:55", | |
| 47 | + "max_abs_dlogit_worst": 0.0 | |
| 48 | +} | |
| \ No newline at end of file | ||
added
benchmarks/raw/tests/numerical_cap32_2026-11-01T013341.json
+48 −0
@@ -0,0 +1,48 @@ | ||
| 1 | +{ | |
| 2 | + "capacity": 32, | |
| 3 | + "prompts": [ | |
| 4 | + { | |
| 5 | + "prompt": "The capital of Canada is", | |
| 6 | + "tokens": 5, | |
| 7 | + "max_abs_dlogit": 0.0, | |
| 8 | + "argmax_equal": true, | |
| 9 | + "hit_rate": 0.0, | |
| 10 | + "misses": 430, | |
| 11 | + "stall_s": 0.0817645009374246, | |
| 12 | + "layers_routed": 16 | |
| 13 | + }, | |
| 14 | + { | |
| 15 | + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th", | |
| 16 | + "tokens": 21, | |
| 17 | + "max_abs_dlogit": 0.0, | |
| 18 | + "argmax_equal": true, | |
| 19 | + "hit_rate": 0.6060606060606061, | |
| 20 | + "misses": 429, | |
| 21 | + "stall_s": 0.08066362410318106, | |
| 22 | + "layers_routed": 40 | |
| 23 | + }, | |
| 24 | + { | |
| 25 | + "prompt": "Q: If a train travels 60 km in 45 minute", | |
| 26 | + "tokens": 25, | |
| 27 | + "max_abs_dlogit": 0.0, | |
| 28 | + "argmax_equal": true, | |
| 29 | + "hit_rate": 0.6090458488228006, | |
| 30 | + "misses": 631, | |
| 31 | + "stall_s": 0.11843966587912291, | |
| 32 | + "layers_routed": 58 | |
| 33 | + }, | |
| 34 | + { | |
| 35 | + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent", | |
| 36 | + "tokens": 26, | |
| 37 | + "max_abs_dlogit": 0.0, | |
| 38 | + "argmax_equal": true, | |
| 39 | + "hit_rate": 0.6224188790560472, | |
| 40 | + "misses": 512, | |
| 41 | + "stall_s": 0.09647229174152017, | |
| 42 | + "layers_routed": 49 | |
| 43 | + } | |
| 44 | + ], | |
| 45 | + "tol": 0.001, | |
| 46 | + "ts": "2026-11-01T01:33:41", | |
| 47 | + "max_abs_dlogit_worst": 0.0 | |
| 48 | +} | |
| \ No newline at end of file | ||
added
benchmarks/raw/tests/numerical_cap64_2026-11-01T013054.json
+48 −0
@@ -0,0 +1,48 @@ | ||
| 1 | +{ | |
| 2 | + "capacity": 64, | |
| 3 | + "prompts": [ | |
| 4 | + { | |
| 5 | + "prompt": "The capital of Canada is", | |
| 6 | + "tokens": 5, | |
| 7 | + "max_abs_dlogit": 0.0, | |
| 8 | + "argmax_equal": true, | |
| 9 | + "hit_rate": 0.0, | |
| 10 | + "misses": 430, | |
| 11 | + "stall_s": 0.16035849519539624, | |
| 12 | + "layers_routed": 16 | |
| 13 | + }, | |
| 14 | + { | |
| 15 | + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th", | |
| 16 | + "tokens": 21, | |
| 17 | + "max_abs_dlogit": 0.0, | |
| 18 | + "argmax_equal": true, | |
| 19 | + "hit_rate": 0.4637904468412943, | |
| 20 | + "misses": 348, | |
| 21 | + "stall_s": 0.127857944695279, | |
| 22 | + "layers_routed": 16 | |
| 23 | + }, | |
| 24 | + { | |
| 25 | + "prompt": "Q: If a train travels 60 km in 45 minute", | |
| 26 | + "tokens": 25, | |
| 27 | + "max_abs_dlogit": 0.0, | |
| 28 | + "argmax_equal": true, | |
| 29 | + "hit_rate": 0.823076923076923, | |
| 30 | + "misses": 138, | |
| 31 | + "stall_s": 0.05094600887969136, | |
| 32 | + "layers_routed": 16 | |
| 33 | + }, | |
| 34 | + { | |
| 35 | + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent", | |
| 36 | + "tokens": 26, | |
| 37 | + "max_abs_dlogit": 0.0, | |
| 38 | + "argmax_equal": true, | |
| 39 | + "hit_rate": 0.9234828496042217, | |
| 40 | + "misses": 58, | |
| 41 | + "stall_s": 0.021786335506476462, | |
| 42 | + "layers_routed": 16 | |
| 43 | + } | |
| 44 | + ], | |
| 45 | + "tol": 0.001, | |
| 46 | + "ts": "2026-11-01T01:30:54", | |
| 47 | + "max_abs_dlogit_worst": 0.0 | |
| 48 | +} | |
| \ No newline at end of file | ||
added
benchmarks/raw/tests/numerical_cap64_2026-11-01T013341.json
+48 −0
@@ -0,0 +1,48 @@ | ||
| 1 | +{ | |
| 2 | + "capacity": 64, | |
| 3 | + "prompts": [ | |
| 4 | + { | |
| 5 | + "prompt": "The capital of Canada is", | |
| 6 | + "tokens": 5, | |
| 7 | + "max_abs_dlogit": 0.0, | |
| 8 | + "argmax_equal": true, | |
| 9 | + "hit_rate": 0.0, | |
| 10 | + "misses": 430, | |
| 11 | + "stall_s": 0.08228024910204113, | |
| 12 | + "layers_routed": 16 | |
| 13 | + }, | |
| 14 | + { | |
| 15 | + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th", | |
| 16 | + "tokens": 21, | |
| 17 | + "max_abs_dlogit": 0.0, | |
| 18 | + "argmax_equal": true, | |
| 19 | + "hit_rate": 0.4637904468412943, | |
| 20 | + "misses": 348, | |
| 21 | + "stall_s": 0.06578753900248557, | |
| 22 | + "layers_routed": 16 | |
| 23 | + }, | |
| 24 | + { | |
| 25 | + "prompt": "Q: If a train travels 60 km in 45 minute", | |
| 26 | + "tokens": 25, | |
| 27 | + "max_abs_dlogit": 0.0, | |
| 28 | + "argmax_equal": true, | |
| 29 | + "hit_rate": 0.823076923076923, | |
| 30 | + "misses": 138, | |
| 31 | + "stall_s": 0.02639120991807431, | |
| 32 | + "layers_routed": 16 | |
| 33 | + }, | |
| 34 | + { | |
| 35 | + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent", | |
| 36 | + "tokens": 26, | |
| 37 | + "max_abs_dlogit": 0.0, | |
| 38 | + "argmax_equal": true, | |
| 39 | + "hit_rate": 0.9234828496042217, | |
| 40 | + "misses": 58, | |
| 41 | + "stall_s": 0.011474333819933236, | |
| 42 | + "layers_routed": 16 | |
| 43 | + } | |
| 44 | + ], | |
| 45 | + "tol": 0.001, | |
| 46 | + "ts": "2026-11-01T01:33:41", | |
| 47 | + "max_abs_dlogit_worst": 0.0 | |
| 48 | +} | |
| \ No newline at end of file | ||
added
benchmarks/raw/tests/numerical_cap8_2026-11-01T013058.json
+48 −0
@@ -0,0 +1,48 @@ | ||
| 1 | +{ | |
| 2 | + "capacity": 8, | |
| 3 | + "prompts": [ | |
| 4 | + { | |
| 5 | + "prompt": "The capital of Canada is", | |
| 6 | + "tokens": 5, | |
| 7 | + "max_abs_dlogit": 0.0, | |
| 8 | + "argmax_equal": true, | |
| 9 | + "hit_rate": 0.1625, | |
| 10 | + "misses": 536, | |
| 11 | + "stall_s": 0.20950216962955892, | |
| 12 | + "layers_routed": 80 | |
| 13 | + }, | |
| 14 | + { | |
| 15 | + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th", | |
| 16 | + "tokens": 21, | |
| 17 | + "max_abs_dlogit": 0.0, | |
| 18 | + "argmax_equal": true, | |
| 19 | + "hit_rate": 0.4425981873111782, | |
| 20 | + "misses": 1476, | |
| 21 | + "stall_s": 0.5867260496597737, | |
| 22 | + "layers_routed": 331 | |
| 23 | + }, | |
| 24 | + { | |
| 25 | + "prompt": "Q: If a train travels 60 km in 45 minute", | |
| 26 | + "tokens": 25, | |
| 27 | + "max_abs_dlogit": 0.0, | |
| 28 | + "argmax_equal": true, | |
| 29 | + "hit_rate": 0.375, | |
| 30 | + "misses": 1995, | |
| 31 | + "stall_s": 0.7891241156030446, | |
| 32 | + "layers_routed": 399 | |
| 33 | + }, | |
| 34 | + { | |
| 35 | + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent", | |
| 36 | + "tokens": 26, | |
| 37 | + "max_abs_dlogit": 0.0, | |
| 38 | + "argmax_equal": true, | |
| 39 | + "hit_rate": 0.3878048780487805, | |
| 40 | + "misses": 2008, | |
| 41 | + "stall_s": 0.7884030294371769, | |
| 42 | + "layers_routed": 410 | |
| 43 | + } | |
| 44 | + ], | |
| 45 | + "tol": 0.001, | |
| 46 | + "ts": "2026-11-01T01:30:58", | |
| 47 | + "max_abs_dlogit_worst": 0.0 | |
| 48 | +} | |
| \ No newline at end of file | ||
added
benchmarks/raw/tests/numerical_cap8_2026-11-01T013343.json
+48 −0
@@ -0,0 +1,48 @@ | ||
| 1 | +{ | |
| 2 | + "capacity": 8, | |
| 3 | + "prompts": [ | |
| 4 | + { | |
| 5 | + "prompt": "The capital of Canada is", | |
| 6 | + "tokens": 5, | |
| 7 | + "max_abs_dlogit": 0.0, | |
| 8 | + "argmax_equal": true, | |
| 9 | + "hit_rate": 0.1625, | |
| 10 | + "misses": 536, | |
| 11 | + "stall_s": 0.1002240888774395, | |
| 12 | + "layers_routed": 80 | |
| 13 | + }, | |
| 14 | + { | |
| 15 | + "prompt": "def fibonacci(n):\n \"\"\"Return the n-th", | |
| 16 | + "tokens": 21, | |
| 17 | + "max_abs_dlogit": 0.0, | |
| 18 | + "argmax_equal": true, | |
| 19 | + "hit_rate": 0.4425981873111782, | |
| 20 | + "misses": 1476, | |
| 21 | + "stall_s": 0.2681740844855085, | |
| 22 | + "layers_routed": 331 | |
| 23 | + }, | |
| 24 | + { | |
| 25 | + "prompt": "Q: If a train travels 60 km in 45 minute", | |
| 26 | + "tokens": 25, | |
| 27 | + "max_abs_dlogit": 0.0, | |
| 28 | + "argmax_equal": true, | |
| 29 | + "hit_rate": 0.375, | |
| 30 | + "misses": 1995, | |
| 31 | + "stall_s": 0.35902054177131504, | |
| 32 | + "layers_routed": 399 | |
| 33 | + }, | |
| 34 | + { | |
| 35 | + "prompt": "Les mod\u00e8les \u00e0 m\u00e9lange d'experts activent", | |
| 36 | + "tokens": 26, | |
| 37 | + "max_abs_dlogit": 0.0, | |
| 38 | + "argmax_equal": true, | |
| 39 | + "hit_rate": 0.3878048780487805, | |
| 40 | + "misses": 2008, | |
| 41 | + "stall_s": 0.368764610029757, | |
| 42 | + "layers_routed": 410 | |
| 43 | + } | |
| 44 | + ], | |
| 45 | + "tol": 0.001, | |
| 46 | + "ts": "2026-11-01T01:33:43", | |
| 47 | + "max_abs_dlogit_worst": 0.0 | |
| 48 | +} | |
| \ No newline at end of file | ||
modified
benchmarks/scripts/bench_decode.py
+9 −3
@@ -61,6 +61,10 @@ def main(): | ||
| 61 | 61 | ap.add_argument("--backend", choices=["titan", "mlx_lm"], default="titan") |
| 62 | 62 | ap.add_argument("--capacity", type=int, default=None) |
| 63 | 63 | ap.add_argument("--policy", default="lru") |
| 64 | + ap.add_argument("--predictor", default="none") | |
| 65 | + ap.add_argument("--m", type=int, default=8) | |
| 66 | + ap.add_argument("--prefetch-workers", type=int, default=2) | |
| 67 | + ap.add_argument("--traces", default=None, help="trace directory used to fit transition/probe predictors (fit split only)") | |
| 64 | 68 | ap.add_argument("--nocache", action="store_true", help="F_NOCACHE on the checkpoint (SSD reads bypass page cache)") |
| 65 | 69 | ap.add_argument("--warm-all", action="store_true", help="preload every expert before timing (all-resident control; implied when capacity == num_experts)") |
| 66 | 70 | ap.add_argument("--workloads", default=str(Path(__file__).resolve().parents[1] / "configs" / "workloads.yaml")) |
@@ -77,13 +81,15 @@ def main(): | ||
| 77 | 81 | prompts = [(fam, p) for fam in a.families.split(",") for p in wl["families"][fam][a.split]] |
| 78 | 82 | rng = random.Random(a.seed) |
| 79 | 83 | if a.backend == "titan": |
| 80 | − tm = TitanModel(a.model, capacity=a.capacity, policy=a.policy, nocache=a.nocache) | |
| 84 | + tm = TitanModel(a.model, capacity=a.capacity, policy=a.policy, nocache=a.nocache, predictor=a.predictor, m=a.m, | |
| 85 | + prefetch_workers=a.prefetch_workers, traces_dir=a.traces) | |
| 81 | 86 | model, tok = tm.model, tm.tokenizer |
| 82 | 87 | if a.warm_all or tm.capacity >= tm.spec.num_experts: |
| 83 | 88 | tm.warm_all() |
| 84 | 89 | else: |
| 85 | 90 | model, _ = load_model(Path(a.model)); tok = load_tokenizer(Path(a.model)); tm = None |
| 86 | 91 | base = {"backend": a.backend, "capacity": a.capacity if a.backend == "titan" else None, "policy": a.policy if a.backend == "titan" else None, |
| 92 | + "predictor": a.predictor if a.backend == "titan" else None, "m": a.m, "prefetch_workers": a.prefetch_workers, | |
| 87 | 93 | "nocache": a.nocache, "max_tokens": a.max_tokens, "split": a.split, "tag": a.tag, "model": os.path.realpath(a.model), |
| 88 | 94 | "cmd": " ".join(sys.argv), "env": env_record(), "seed": a.seed} |
| 89 | 95 | os.makedirs(os.path.dirname(a.out), exist_ok=True) |
@@ -101,10 +107,10 @@ def main(): | ||
| 101 | 107 | if tm: |
| 102 | 108 | cs = tm.cache_stats() |
| 103 | 109 | rec.update(cache_hit_rate=cs["hit_rate"], cache_misses=cs["misses"], cache_hits=cs["hits"], stall_s=cs["stall_time_s"], |
| 104 | − hits_device=sum(p["hits_device"] for p in cs["layers"]), layers_complete=sum(1 for t in tm.tables if t.complete()), | |
| 110 | + hits_device=sum(p["hits_device"] for p in cs["layers"]), layers_complete=sum(1 for t in tm.tables if t.complete()), prefetch=cs["prefetch"], | |
| 105 | 111 | ssd_bytes=cs["store"]["bytes_read"], ssd_read_s=cs["store"]["read_time_s"]) |
| 106 | 112 | f.write(json.dumps(rec) + "\n"); f.flush() |
| 107 | − print(f"{a.backend} cap={a.capacity} rep={rep} {fam:12s} ttft={r['ttft_ms']:.0f}ms prefill={r['prompt_tps']:.0f} decode={r['gen_tps']:.1f} tok/s" | |
| 113 | + print(f"{a.backend} cap={a.capacity} {a.predictor} rep={rep} {fam:12s} ttft={r['ttft_ms']:.0f}ms prefill={r['prompt_tps']:.0f} decode={r['gen_tps']:.1f} tok/s" | |
| 108 | 114 | + (f" hit={rec['cache_hit_rate']:.3f} stall={rec['stall_s']:.2f}s" if tm else "")) |
| 109 | 115 | |
| 110 | 116 | |
added
benchmarks/scripts/simulate_cache.py
+146 −0
@@ -0,0 +1,146 @@ | ||
| 1 | +#!/usr/bin/env python3 | |
| 2 | +"""Offline, trace-driven simulation of per-layer slot tables with eviction policies and prefetch predictors. | |
| 3 | + | |
| 4 | +Exact for hit/miss accounting (the router sequence is the recorded one). Stall time is *modelled* from measured constants: | |
| 5 | +t_read (seconds to fetch one expert from the tier) and t_layer (seconds of compute per layer), with a loader pool of `--loaders` | |
| 6 | +parallel readers: demand misses at a layer stall for ceil(misses/loaders) * t_read; a prefetch issued h layers ahead hides | |
| 7 | +h * t_layer of its latency. Outputs JSON + CSV under --out. | |
| 8 | +""" | |
| 9 | +import argparse, glob, json, math, os, sys, time | |
| 10 | +import numpy as np | |
| 11 | +from pathlib import Path | |
| 12 | +sys.path.insert(0, str(Path(__file__).resolve().parents[2])) | |
| 13 | +from titan.predictors import make as make_predictor | |
| 14 | +from titan.slots import EvictionPolicy, LFUPolicy | |
| 15 | + | |
| 16 | +POLICIES = {"lru": EvictionPolicy, "lfu": LFUPolicy} | |
| 17 | + | |
| 18 | + | |
| 19 | +def load_traces(d): | |
| 20 | + idx = json.load(open(os.path.join(d, "index.json"))) | |
| 21 | + out = [] | |
| 22 | + for p in idx["prompts"]: | |
| 23 | + z = np.load(os.path.join(d, p["name"] + ".npz")) | |
| 24 | + out.append(dict(p, topk=z["topk"].astype(np.int64), probs=z["probs"], phase=z["phase"])) | |
| 25 | + return idx, out | |
| 26 | + | |
| 27 | + | |
| 28 | +class LayerSim: | |
| 29 | + def __init__(self, cap, policy, E): | |
| 30 | + self.cap = cap; self.pol = POLICIES[policy](); self.res = set(); self.E = E | |
| 31 | + self.hits = self.misses = self.pf_loads = self.pf_used = self.demand_loads = self.evict = 0 | |
| 32 | + self.pf_pending = {} # expert -> True if prefetched and not yet used | |
| 33 | + self.late_pending = 0 | |
| 34 | + | |
| 35 | + def _insert(self, e, protected): | |
| 36 | + if e in self.res: return | |
| 37 | + if len(self.res) >= self.cap: | |
| 38 | + try: v = self.pol.victim(protected) | |
| 39 | + except RuntimeError: return | |
| 40 | + self.res.discard(v); self.pol.remove(v); self.evict += 1 | |
| 41 | + if self.pf_pending.pop(v, False): pass | |
| 42 | + self.res.add(e) | |
| 43 | + | |
| 44 | + def prefetch(self, experts, protected): | |
| 45 | + for e in experts: | |
| 46 | + if e in self.res: continue | |
| 47 | + if len(self.res) >= self.cap and all(x in protected for x in self.res): break | |
| 48 | + self._insert(e, protected); self.pol.touch(e); self.pf_loads += 1; self.pf_pending[e] = True | |
| 49 | + | |
| 50 | + def access(self, needed): | |
| 51 | + miss = 0 | |
| 52 | + for e in needed: | |
| 53 | + if e in self.res: | |
| 54 | + self.hits += 1 | |
| 55 | + if self.pf_pending.pop(e, False): self.pf_used += 1 | |
| 56 | + else: | |
| 57 | + self.misses += 1; miss += 1; self.demand_loads += 1 | |
| 58 | + self._insert(e, set(needed)) | |
| 59 | + self.pol.touch(e) | |
| 60 | + return miss | |
| 61 | + | |
| 62 | + | |
| 63 | +def simulate(traces, L, E, K, cap, policy, pred_name, m, t_read, t_layer, loaders, fit_traces): | |
| 64 | + pred = make_predictor(pred_name, L, E) | |
| 65 | + depth = getattr(pred, "depth", 1) | |
| 66 | + if fit_traces is not None: pred.fit([(t["topk"], t["probs"]) for t in fit_traces]) | |
| 67 | + tot = {"tokens": 0, "hits": 0, "misses": 0, "pf_loads": 0, "pf_used": 0, "demand_loads": 0, "stall_s": 0.0, "pred_hits": 0, "pred_total": 0, "evictions": 0} | |
| 68 | + per_family = {} | |
| 69 | + for tr in traces: | |
| 70 | + layers = [LayerSim(cap, policy, E) for _ in range(L)] # cold per prompt | |
| 71 | + topk, probs = tr["topk"], tr["probs"] | |
| 72 | + prev = None; prev_p = None | |
| 73 | + fam = per_family.setdefault(tr["family"], {"tokens": 0, "hits": 0, "misses": 0, "stall_s": 0.0}) | |
| 74 | + for t in range(topk.shape[0]): | |
| 75 | + cur = {("true", l): topk[t, l] for l in range(L)} # read ONLY by the Oracle upper bound | |
| 76 | + stall = 0.0 | |
| 77 | + if pred.horizon == "token": | |
| 78 | + for l in range(L): | |
| 79 | + layers[l].prefetch(pred.predict(l, prev, prev_p, cur, m), set()) | |
| 80 | + for l in range(L): | |
| 81 | + need = list(topk[t, l]) | |
| 82 | + if pred.horizon == "layer" and l > 0: | |
| 83 | + pass # already issued at l-1 | |
| 84 | + miss = layers[l].access(need) | |
| 85 | + if miss: | |
| 86 | + # demand misses: a layer-horizon prefetch for this layer was issued `depth` layers earlier and overlapped depth*t_layer; | |
| 87 | + # only misses that were *prefetched-but-late* benefit, so we model hidden = depth*t_layer for the late ones, 0 for unpredicted. | |
| 88 | + late = min(miss, layers[l].late_pending) | |
| 89 | + unpred = miss - late | |
| 90 | + stall += (math.ceil(late / loaders) * max(0.0, t_read - depth * t_layer) + math.ceil(unpred / loaders) * t_read) if late + unpred else 0.0 | |
| 91 | + layers[l].late_pending = 0 | |
| 92 | + cur[l] = topk[t, l]; cur[("probs", l)] = probs[t, l] | |
| 93 | + if pred.horizon == "layer" and l + depth < L: | |
| 94 | + p = pred.predict(l + depth, prev, prev_p, cur, m) | |
| 95 | + if pred_name != "none": | |
| 96 | + tot["pred_hits"] += len(set(p[:K]) & set(topk[t, l + depth])); tot["pred_total"] += K | |
| 97 | + layers[l + depth].prefetch(p, set()) | |
| 98 | + layers[l + depth].late_pending = len(set(p) & set(topk[t, l + depth]) - layers[l + depth].res) | |
| 99 | + prev, prev_p = topk[t], probs[t] | |
| 100 | + tot["tokens"] += 1; fam["tokens"] += 1; tot["stall_s"] += stall; fam["stall_s"] += stall | |
| 101 | + h = sum(x.hits for x in layers); mi = sum(x.misses for x in layers) | |
| 102 | + tot["hits"] += sum(x.hits for x in layers); tot["misses"] += sum(x.misses for x in layers) | |
| 103 | + tot["pf_loads"] += sum(x.pf_loads for x in layers); tot["pf_used"] += sum(x.pf_used for x in layers) | |
| 104 | + tot["demand_loads"] += sum(x.demand_loads for x in layers); tot["evictions"] += sum(x.evict for x in layers) | |
| 105 | + fam["hits"] += sum(x.hits for x in layers); fam["misses"] += sum(x.misses for x in layers) | |
| 106 | + n = tot["hits"] + tot["misses"] | |
| 107 | + res = {"capacity": cap, "policy": policy, "predictor": pred_name, "m": m, "hit_rate": tot["hits"] / n, "misses_per_token": tot["misses"] / tot["tokens"], | |
| 108 | + "stall_ms_per_token": 1e3 * tot["stall_s"] / tot["tokens"], "prefetch_loads_per_token": tot["pf_loads"] / tot["tokens"], | |
| 109 | + "prefetch_precision": (tot["pf_used"] / tot["pf_loads"]) if tot["pf_loads"] else None, | |
| 110 | + "loads_per_token": (tot["pf_loads"] + tot["demand_loads"]) / tot["tokens"], | |
| 111 | + "pred_recall_at_k": (tot["pred_hits"] / tot["pred_total"]) if tot["pred_total"] else None, | |
| 112 | + "tokens": tot["tokens"], "families": {f: {"hit_rate": v["hits"] / (v["hits"] + v["misses"]), "stall_ms_per_token": 1e3 * v["stall_s"] / v["tokens"]} for f, v in per_family.items()}} | |
| 113 | + return res | |
| 114 | + | |
| 115 | + | |
| 116 | +def main(): | |
| 117 | + ap = argparse.ArgumentParser() | |
| 118 | + ap.add_argument("--traces", required=True) | |
| 119 | + ap.add_argument("--capacities", default="8,12,16,24,32,48") | |
| 120 | + ap.add_argument("--policies", default="lru,lfu") | |
| 121 | + ap.add_argument("--predictors", default="none,prev_rank,transition,transition2,transition3,probe,oracle") | |
| 122 | + ap.add_argument("--m", type=int, default=8) | |
| 123 | + ap.add_argument("--t-read-ms", type=float, default=0.43, help="measured per-expert fetch latency of the tier (laptop: 0.43 ms cold SSD, 0.27 ms page cache, 28 ms 1 GbE)") | |
| 124 | + ap.add_argument("--t-layer-ms", type=float, default=0.128, help="per-layer decode compute (489 tok/s / 16 layers)") | |
| 125 | + ap.add_argument("--loaders", type=int, default=1, help="parallel readers (current TITAN implementation: 1)") | |
| 126 | + ap.add_argument("--out", required=True) | |
| 127 | + a = ap.parse_args() | |
| 128 | + idx, traces = load_traces(a.traces) | |
| 129 | + L, E, K = idx["L"], idx["E"], idx["K"] | |
| 130 | + fit = [t for t in traces if t["split"] == "fit"]; held = [t for t in traces if t["split"] == "heldout"] | |
| 131 | + results = [] | |
| 132 | + for cap in [int(c) for c in a.capacities.split(",")]: | |
| 133 | + for pol in a.policies.split(","): | |
| 134 | + for pr in a.predictors.split(","): | |
| 135 | + for split, trs in (("fit", fit), ("heldout", held)): | |
| 136 | + r = simulate(trs, L, E, K, cap, pol, pr, a.m, a.t_read_ms / 1e3, a.t_layer_ms / 1e3, a.loaders, fit) | |
| 137 | + r["split"] = split; results.append(r) | |
| 138 | + print(f"cap={cap:2d} {pol} {pr:10s} {split:7s} hit={r['hit_rate']:.3f} miss/tok={r['misses_per_token']:5.1f} stall={r['stall_ms_per_token']:6.2f}ms " | |
| 139 | + f"loads/tok={r['loads_per_token']:5.1f} pf_prec={r['prefetch_precision'] if r['prefetch_precision'] is None else round(r['prefetch_precision'],3)} recall@k={r['pred_recall_at_k'] if r['pred_recall_at_k'] is None else round(r['pred_recall_at_k'],3)}") | |
| 140 | + os.makedirs(os.path.dirname(a.out), exist_ok=True) | |
| 141 | + json.dump({"cmd": " ".join(sys.argv), "ts": time.strftime("%Y-%m-%dT%H:%M:%S"), "traces": a.traces, "L": L, "E": E, "K": K, | |
| 142 | + "t_read_ms": a.t_read_ms, "t_layer_ms": a.t_layer_ms, "loaders": a.loaders, "results": results}, open(a.out, "w"), indent=1) | |
| 143 | + | |
| 144 | + | |
| 145 | +if __name__ == "__main__": | |
| 146 | + main() | |
modified
research/DECISIONS.md
+15 −0
@@ -38,3 +38,18 @@ layer's expert kernels — the unavoidable synchronization of demand loading, sh | ||
| 38 | 38 | ## D-006 · 2026-11-01 · Activation parity |
| 39 | 39 | Decision: use mlx-lm's compiled `swiglu` (silu(gate)·up in one fused kernel). Rationale: an unfused fp16 `up·sigmoid(gate)·gate` |
| 40 | 40 | gave max |Δlogit| = 0.0195 vs mlx-lm; the fused version gives 0.0 (bit-identical) — see `benchmarks/raw/tests/numerical_*.json`. |
| 41 | + | |
| 42 | +## D-007 · 2026-11-01 · Lazy slot installs, batched demand misses | |
| 43 | +Decision: `SlotTable.ensure()` reads all missing experts of a layer, installs them as lazy slice assignments and never calls | |
| 44 | +`mx.eval` itself; the layer's `gather_qmm` consumes the updated buffers in the same graph. Rationale: the first version evaluated | |
| 45 | +the whole table after every miss (one GPU sync per missed expert): at 16 slots decode went from 28/41 tok/s to 51/73 tok/s | |
| 46 | +(general/code, held-out) once removed (`benchmarks/raw/gateC/prefetch_online_laptop/smoke*.jsonl`). Per-miss cost on the laptop | |
| 47 | +with a page-cached file is now 0.21 ms (`benchmarks/raw/gateC/design/miss_cost_breakdown_laptop.txt`: pread 0.13 + numpy/MLX | |
| 48 | +copy 0.06 + install) against 0.43 ms from a cold SSD and ≈ 28 ms over 1 GbE. | |
| 49 | + | |
| 50 | +## D-008 · 2026-11-01 · Medium model download (17.2 GB) under the standing ≤ 20 GB rule | |
| 51 | +Decision: download `mlx-community/Qwen3-30B-A3B-Instruct-2507-4bit` (rev `e9675aa3`, Apache-2.0, 48 layers × 128 experts, top-8) | |
| 52 | +on M3U96a (cluster coordinator for Gates E/F), then rsync to worker nodes over the LAN rather than re-downloading. Rationale: | |
| 53 | +OLMoE (3.6 GB of experts) fits in every node's RAM, so the memory-budget regimes of H1/H3 need a model whose experts exceed | |
| 54 | +the budget of the smaller nodes (36–64 GB) and whose expert count (6 144) gives a realistic routing problem. gpt-oss-120b (62 GB) | |
| 55 | +is deferred until explicitly authorized. | |
modified
research/LOG.md
+14 −0
@@ -56,3 +56,17 @@ Chronological, append-only. Times are local (America/Toronto). Every entry names | ||
| 56 | 56 | to parse on M4M64a (to rerun). **No Thunderbolt link is active on any Studio** (all `Thunderbolt N` interfaces `status: inactive` |
| 57 | 57 | on M3U96a/M3U96b/M4M64a/M2U64/M4M64c/M5M36) → Thunderbolt/JACCL RDMA is **blocked pending physical cabling** (user action); the |
| 58 | 58 | distributed gates (E/F) will use TCP over 1 GbE, which is the honest topology of this cluster. |
| 59 | +- Second network probe (`artifacts/private/netprobe2-2026-11-01.txt`, iperf3 installed on M2U64 and M4M64c via Homebrew): | |
| 60 | + M3U96a → M4M64a / M2U64 / M4M64c / M3U96b all **0.945–0.947 Gbit/s**; RTT M4M64a→M3U96b 0.87 ms, M4M64a→M2U64 0.69 ms. | |
| 61 | +- Offline simulation v1 (`benchmarks/processed/gateC/sim_olmoe_laptop_ssd_v1.json`, 18-prompt traces, per-layer tables, cold per | |
| 62 | + prompt, 1 loader, t_read 0.43 ms, t_layer 0.128 ms): LRU hit rate 0.57 (cap 16) / 0.80 (32) / 0.93 (48); LFU ≈ LRU; | |
| 63 | + one-layer-ahead transition predictor recall@8 = 0.50 held-out (0.53 fit), raises hit rate to 0.66 / 0.83 / 0.94 at the cost of | |
| 64 | + +40–60 % loads per token; the linear probe on previous-layer log-probabilities underperforms (recall 0.16, too little data → v2 | |
| 65 | + traces); the oracle upper bound reaches 0.97 (cap 32). `prev_token` is a no-op by construction with per-layer tables (the previous | |
| 66 | + token's experts are still resident), and `prev_rank` with m ≤ K prefetched nothing (fixed: ranks K+1..K+m). | |
| 67 | +- Online prefetch smoke test at 16 slots (held-out general/code, page-cached file, 2 reader threads, m = 8): transition raised the hit rate | |
| 68 | + (0.48→0.63, 0.67→0.73) but **lowered decode throughput** (51→35, 73→56 tok/s) — a one-layer lead (≈ 0.13 ms) cannot hide a 0.21 ms | |
| 69 | + fetch, late prefetches are waited for, and wrong predictions add traffic. H1 at this tier/model is therefore *not supported by a | |
| 70 | + 1-layer horizon*; deeper horizons (transition2/3/4) and the token horizon are being simulated (sim v2 on the 65-prompt traces). | |
| 71 | +- Routing traces v2 collected (`benchmarks/raw/gateC/traces_olmoe_v2_laptop/`, 65 prompts, 6 families incl. long-context and drift, | |
| 72 | + up to 384 generated tokens each). | |
modified
titan/moe.py
+52 −5
@@ -10,6 +10,7 @@ from __future__ import annotations | ||
| 10 | 10 | from typing import Callable, List, Optional |
| 11 | 11 | |
| 12 | 12 | import mlx.core as mx |
| 13 | +import numpy as np | |
| 13 | 14 | from mlx_lm.models.activations import swiglu |
| 14 | 15 | |
| 15 | 16 | from .slots import SlotTable |
@@ -19,8 +20,10 @@ class TitanSwitchGLU: | ||
| 19 | 20 | """Callable with the SwitchGLU signature: (x[..., 1, D], indices[..., k]) -> y[..., k, D].""" |
| 20 | 21 | |
| 21 | 22 | def __init__(self, table: SlotTable, group_size: int, bits: int, activation=None, |
| 22 | − on_route: Optional[Callable[[int, List[int]], None]] = None): | |
| 23 | + on_route: Optional[Callable[[int, List[int]], None]] = None, prefetcher=None): | |
| 23 | 24 | self.table = table |
| 25 | + self.prefetcher = prefetcher | |
| 26 | + self.current_probs = None | |
| 24 | 27 | self.group_size = group_size |
| 25 | 28 | self.bits = bits |
| 26 | 29 | self.on_route = on_route # hook(layer, unique_experts) used by predictors/prefetchers (Gate C) |
@@ -39,10 +42,14 @@ class TitanSwitchGLU: | ||
| 39 | 42 | y = self._qmm(h, "down", slot_idx, False) |
| 40 | 43 | return y.squeeze(-2) |
| 41 | 44 | |
| 42 | − def _run(self, x, flat, shape): | |
| 45 | + def _run(self, x, flat, shape, probs_np=None): | |
| 43 | 46 | uniq = list(dict.fromkeys(flat)) |
| 44 | 47 | if self.on_route is not None: |
| 45 | 48 | self.on_route(self.table.layer, uniq) |
| 49 | + pf = self.prefetcher | |
| 50 | + if pf is not None: | |
| 51 | + pf.before_layer(self.table.layer, uniq) # install completed prefetches (wait for late ones we need) | |
| 52 | + pf.after_layer(self.table.layer, uniq, probs_np) # issue prefetch for the next layer BEFORE this layer's compute | |
| 46 | 53 | slot_of = dict(zip(uniq, self.table.ensure(uniq))) |
| 47 | 54 | slot_idx = mx.array([slot_of[e] for e in flat], dtype=mx.uint32).reshape(shape) |
| 48 | 55 | x = mx.expand_dims(x, (-2, -3)) |
@@ -61,6 +68,10 @@ class TitanSwitchGLU: | ||
| 61 | 68 | t.hits_device += indices.size |
| 62 | 69 | return self._run_device(x, t.slot_map[indices]) |
| 63 | 70 | # Materialise the router's choice: this is the unavoidable host sync of any demand-loading design. |
| 71 | + probs_np = None | |
| 72 | + if self.prefetcher is not None and self.current_probs is not None: | |
| 73 | + mx.eval(indices, self.current_probs) | |
| 74 | + probs_np = np.array(self.current_probs[-1]) # last row: decode token (prefill rows are not used for prediction) | |
| 64 | 75 | flat = indices.reshape(-1).tolist() |
| 65 | 76 | self.last_indices = flat |
| 66 | 77 | k = indices.shape[-1] |
@@ -78,9 +89,45 @@ class TitanSwitchGLU: | ||
| 78 | 89 | chunks.append(cur) |
| 79 | 90 | self.n_chunks = len(chunks) |
| 80 | 91 | if len(chunks) == 1: |
| 81 | − return self._run(x, flat, indices.shape) | |
| 92 | + return self._run(x, flat, indices.shape, probs_np) | |
| 82 | 93 | outs = [] |
| 83 | − for rows in chunks: | |
| 94 | + for ci, rows in enumerate(chunks): | |
| 84 | 95 | f = [flat[r * k + j] for r in rows for j in range(k)] |
| 85 | − outs.append(self._run(x2[rows[0]:rows[-1] + 1], f, (len(rows), k))) | |
| 96 | + outs.append(self._run(x2[rows[0]:rows[-1] + 1], f, (len(rows), k), probs_np if ci == len(chunks) - 1 else None)) | |
| 86 | 97 | return mx.concatenate(outs, axis=0).reshape(indices.shape + (x.shape[-1],)) |
| 98 | + | |
| 99 | + | |
| 100 | +class TitanMoEBlock: | |
| 101 | + """Drop-in replacement for the host model's sparse MoE block (OLMoE / Qwen3-MoE routing: softmax over all experts, | |
| 102 | + top-k by argpartition, gating weights = softmax probabilities, renormalised iff norm_topk_prob). The router ops are | |
| 103 | + exactly mlx-lm's (verified bit-identical in tests/test_numerical.py); the only additions are the prefetcher hooks.""" | |
| 104 | + | |
| 105 | + def __init__(self, host_mlp, block: "TitanSwitchGLU", model_type: str, prefetcher=None): | |
| 106 | + self.gate = host_mlp.gate | |
| 107 | + self.top_k = host_mlp.top_k | |
| 108 | + self.norm_topk_prob = getattr(host_mlp, "norm_topk_prob", False) | |
| 109 | + self.switch_mlp = block | |
| 110 | + self.model_type = model_type | |
| 111 | + self.prefetcher = prefetcher | |
| 112 | + self.layer = block.table.layer | |
| 113 | + | |
| 114 | + def __call__(self, x: mx.array) -> mx.array: | |
| 115 | + B, L, D = x.shape | |
| 116 | + x_flat = x.reshape(-1, D) | |
| 117 | + k = self.top_k | |
| 118 | + if self.model_type == "olmoe": | |
| 119 | + router_logits = self.gate(x_flat) | |
| 120 | + rw = mx.softmax(router_logits, axis=1, precise=True) | |
| 121 | + indices = mx.stop_gradient(mx.argpartition(-rw, kth=k - 1, axis=-1)[..., :k]) | |
| 122 | + scores = mx.take_along_axis(rw, indices, axis=-1) | |
| 123 | + else: # qwen3_moe | |
| 124 | + gates = self.gate(x_flat) | |
| 125 | + rw = mx.softmax(gates, axis=-1, precise=True) | |
| 126 | + indices = mx.argpartition(rw, kth=-k, axis=-1)[..., -k:] | |
| 127 | + scores = mx.take_along_axis(rw, indices, axis=-1) | |
| 128 | + if self.norm_topk_prob: | |
| 129 | + scores = scores / scores.sum(axis=-1, keepdims=True) | |
| 130 | + self.switch_mlp.current_probs = rw if self.prefetcher is not None else None | |
| 131 | + y = self.switch_mlp(x_flat, indices) | |
| 132 | + y = (y * scores[..., None]).sum(axis=-2) | |
| 133 | + return y.reshape(B, L, D) | |
added
titan/predictors.py
+126 −0
@@ -0,0 +1,126 @@ | ||
| 1 | +"""Expert-demand predictors. Every predictor only uses information available at its decision point: | |
| 2 | +- `PrevToken`: at the start of token t, the experts used at layer l by token t-1 (all layers known after token t-1). | |
| 3 | +- `Transition`: at layer l-1 of token t (its routing just materialised), a transition-count matrix fitted on *fit* traces, | |
| 4 | + M_l[e', e] = #(e' chosen at l-1 and e at l) -> score(e) = sum_{e' in R(t,l-1)} M_l[e', e]. | |
| 5 | +- `Probe`: a per-layer linear multi-label probe on the layer-(l-1) router probabilities (64 -> 64 logits), fitted on fit traces. | |
| 6 | +- `Oracle`: the true R(t,l) (upper bound only; never a method). | |
| 7 | +No predictor sees routes of layer >= l at token t, nor any future token. | |
| 8 | +""" | |
| 9 | +from __future__ import annotations | |
| 10 | +import numpy as np | |
| 11 | + | |
| 12 | + | |
| 13 | +class Predictor: | |
| 14 | + name = "none" | |
| 15 | + horizon = 0 # 0 = no prefetch; "token" = at token start; "layer" = one layer ahead | |
| 16 | + | |
| 17 | + def fit(self, traces): # traces: list of (topk[T,L,K], probs[T,L,E]) | |
| 18 | + return self | |
| 19 | + | |
| 20 | + def predict(self, l, prev_topk, prev_probs, cur_routes, m): | |
| 21 | + """Return up to m expert ids for layer l. | |
| 22 | + prev_topk: [L,K] routes of the previous token (or None); cur_routes: dict l' -> routes for l' < l at the current token; | |
| 23 | + prev_probs: [L,E] probs of the previous token.""" | |
| 24 | + return [] | |
| 25 | + | |
| 26 | + | |
| 27 | +class PrevToken(Predictor): | |
| 28 | + name = "prev_token" | |
| 29 | + horizon = "token" | |
| 30 | + | |
| 31 | + def predict(self, l, prev_topk, prev_probs, cur_routes, m): | |
| 32 | + if prev_topk is None: | |
| 33 | + return [] | |
| 34 | + return list(prev_topk[l][:m]) | |
| 35 | + | |
| 36 | + | |
| 37 | +class PrevTokenRank(Predictor): | |
| 38 | + """At token start: for layer l, the experts ranked m highest by the *previous token's* router probabilities at layer l | |
| 39 | + (ranks 1..K are normally already resident, so this effectively prefetches the next-most-likely experts).""" | |
| 40 | + name = "prev_rank" | |
| 41 | + horizon = "token" | |
| 42 | + | |
| 43 | + def __init__(self, K=8): | |
| 44 | + self.K = K | |
| 45 | + | |
| 46 | + def predict(self, l, prev_topk, prev_probs, cur_routes, m): | |
| 47 | + if prev_probs is None: | |
| 48 | + return [] | |
| 49 | + return list(np.argsort(-prev_probs[l].astype(np.float32))[self.K:self.K + m]) # ranks K+1..K+m (top-K are already resident) | |
| 50 | + | |
| 51 | + | |
| 52 | +class Transition(Predictor): | |
| 53 | + """Transition counts from layer l-d to layer l (d = lookahead depth); issued when layer l-d's routing is known.""" | |
| 54 | + name = "transition" | |
| 55 | + horizon = "layer" | |
| 56 | + | |
| 57 | + def __init__(self, L, E, depth=1): | |
| 58 | + self.M = np.zeros((L, E, E), dtype=np.float64) | |
| 59 | + self.depth = depth | |
| 60 | + self.name = "transition" if depth == 1 else f"transition{depth}" | |
| 61 | + | |
| 62 | + def fit(self, traces): | |
| 63 | + d = self.depth | |
| 64 | + for topk, _ in traces: | |
| 65 | + for l in range(d, topk.shape[1]): | |
| 66 | + for t in range(topk.shape[0]): | |
| 67 | + for a in topk[t, l - d]: | |
| 68 | + self.M[l, a, topk[t, l]] += 1 | |
| 69 | + return self | |
| 70 | + | |
| 71 | + def predict(self, l, prev_topk, prev_probs, cur_routes, m): | |
| 72 | + d = self.depth | |
| 73 | + if l < d or (l - d) not in cur_routes: | |
| 74 | + return list(prev_topk[l][:m]) if prev_topk is not None else [] | |
| 75 | + s = self.M[l][list(cur_routes[l - d])].sum(0) | |
| 76 | + return list(np.argsort(-s)[:m]) | |
| 77 | + | |
| 78 | + | |
| 79 | +class Probe(Predictor): | |
| 80 | + name = "probe" | |
| 81 | + horizon = "layer" | |
| 82 | + | |
| 83 | + def __init__(self, L, E, epochs=2000, lr=0.05, l2=1e-5, seed=0): | |
| 84 | + self.L, self.E, self.epochs, self.lr, self.l2 = L, E, epochs, lr, l2 | |
| 85 | + self.W = np.zeros((L, E + 1, E)); self.mu = np.zeros((L, E)); self.sd = np.ones((L, E)); self.rng = np.random.default_rng(seed) | |
| 86 | + | |
| 87 | + def fit(self, traces): | |
| 88 | + for l in range(1, self.L): | |
| 89 | + X = np.concatenate([p[:, l - 1].astype(np.float32) for _, p in traces], 0) | |
| 90 | + Y = np.zeros((X.shape[0], self.E), np.float32) | |
| 91 | + rows = np.concatenate([np.repeat(np.arange(t.shape[0]), t.shape[2]) for t, _ in traces]); off = 0; cols = [] | |
| 92 | + for t, _ in traces: | |
| 93 | + cols.append(t[:, l].reshape(-1)); | |
| 94 | + cols = np.concatenate(cols) | |
| 95 | + Y[rows, cols] = 1.0 | |
| 96 | + X = np.log(np.clip(X, 1e-6, None)) # log-probabilities as features (scale-free) | |
| 97 | + self.mu[l] = X.mean(0); self.sd[l] = X.std(0) + 1e-6 | |
| 98 | + Xb = np.concatenate([(X - self.mu[l]) / self.sd[l], np.ones((X.shape[0], 1), np.float32)], 1) | |
| 99 | + W = np.zeros((self.E + 1, self.E)); V = np.zeros_like(W) | |
| 100 | + for _ in range(self.epochs): # full-batch logistic regression with momentum (multi-label, independent sigmoids) | |
| 101 | + P = 1 / (1 + np.exp(-(Xb @ W))) | |
| 102 | + G = Xb.T @ (P - Y) / X.shape[0] + self.l2 * W | |
| 103 | + V = 0.9 * V + G; W -= self.lr * V | |
| 104 | + self.W[l] = W | |
| 105 | + return self | |
| 106 | + | |
| 107 | + def predict(self, l, prev_topk, prev_probs, cur_routes, m): | |
| 108 | + if l == 0 or (l - 1) not in cur_routes or cur_routes.get(("probs", l - 1)) is None: | |
| 109 | + return list(prev_topk[l][:m]) if prev_topk is not None else [] | |
| 110 | + x = (np.log(np.clip(cur_routes[("probs", l - 1)].astype(np.float32), 1e-6, None)) - self.mu[l]) / self.sd[l] | |
| 111 | + return list(np.argsort(-(np.append(x, 1.0) @ self.W[l]))[:m]) | |
| 112 | + | |
| 113 | + | |
| 114 | +class Oracle(Predictor): | |
| 115 | + name = "oracle" | |
| 116 | + horizon = "layer" | |
| 117 | + | |
| 118 | + def predict(self, l, prev_topk, prev_probs, cur_routes, m): | |
| 119 | + return list(cur_routes[("true", l)][:m]) | |
| 120 | + | |
| 121 | + | |
| 122 | +def make(name, L, E): | |
| 123 | + reg = {"none": Predictor, "prev_token": PrevToken, "prev_rank": PrevTokenRank, "transition": lambda: Transition(L, E), | |
| 124 | + "transition2": lambda: Transition(L, E, 2), "transition3": lambda: Transition(L, E, 3), "transition4": lambda: Transition(L, E, 4), | |
| 125 | + "probe": lambda: Probe(L, E), "oracle": Oracle} | |
| 126 | + return reg[name]() | |
added
titan/prefetch.py
+99 −0
@@ -0,0 +1,99 @@ | ||
| 1 | +"""Asynchronous prefetcher: a small pool of reader threads fetches predicted experts from the ExpertStore (SSD tier) | |
| 2 | +while the GPU computes the current layer; completed fetches are installed into the slot table on the main thread when the | |
| 3 | +layer is reached (`drain`). Predictors come from titan.predictors and only see information available at decision time. | |
| 4 | + | |
| 5 | +Accounting per layer: prefetch_issued, prefetch_installed, prefetch_used (installed experts that were subsequently hit), | |
| 6 | +prefetch_wasted (installed but evicted before use), prefetch_late (needed before the fetch completed: counted as a demand | |
| 7 | +miss whose stall is the *remaining* latency). | |
| 8 | +""" | |
| 9 | +from __future__ import annotations | |
| 10 | + | |
| 11 | +import threading, time | |
| 12 | +from concurrent.futures import Future, ThreadPoolExecutor | |
| 13 | +from typing import Dict, List, Optional | |
| 14 | + | |
| 15 | +import mlx.core as mx | |
| 16 | +import numpy as np | |
| 17 | + | |
| 18 | +from .slots import SlotTable | |
| 19 | +from .store import ExpertStore | |
| 20 | + | |
| 21 | + | |
| 22 | +class Prefetcher: | |
| 23 | + def __init__(self, tables: List[SlotTable], store: ExpertStore, predictor, m: int = 8, workers: int = 2): | |
| 24 | + self.tables = tables | |
| 25 | + self.store = store | |
| 26 | + self.pred = predictor | |
| 27 | + self.m = m | |
| 28 | + self.pool = ThreadPoolExecutor(max_workers=workers, thread_name_prefix="titan-prefetch") | |
| 29 | + self.inflight: List[Dict[int, Future]] = [dict() for _ in tables] | |
| 30 | + self.lock = threading.Lock() | |
| 31 | + self.issued = [0] * len(tables); self.installed = [0] * len(tables); self.late = [0] * len(tables); self.dup = [0] * len(tables) | |
| 32 | + self.prev_topk: Optional[np.ndarray] = None | |
| 33 | + self.prev_probs: Optional[np.ndarray] = None | |
| 34 | + self.cur: Dict = {} | |
| 35 | + self.L = len(tables) | |
| 36 | + self.K = store.spec.top_k | |
| 37 | + | |
| 38 | + # ---- reading --------------------------------------------------------------- | |
| 39 | + def _fetch(self, layer: int, expert: int): | |
| 40 | + t = time.perf_counter() | |
| 41 | + tensors = self.store.read_expert(layer, expert) # pread + np -> mx (CPU side, no GPU work) | |
| 42 | + return tensors, time.perf_counter() - t | |
| 43 | + | |
| 44 | + def issue(self, layer: int, experts: List[int]): | |
| 45 | + tab = self.tables[layer] | |
| 46 | + for e in experts[: self.m]: | |
| 47 | + if e in tab.slot_of or e in self.inflight[layer]: | |
| 48 | + self.dup[layer] += 1 | |
| 49 | + continue | |
| 50 | + self.inflight[layer][e] = self.pool.submit(self._fetch, layer, e) | |
| 51 | + self.issued[layer] += 1 | |
| 52 | + | |
| 53 | + def drain(self, layer: int, protected: List[int], wait_for: List[int] = ()): | |
| 54 | + """Install completed prefetches for `layer`; optionally wait for the ones in `wait_for` (needed now).""" | |
| 55 | + tab = self.tables[layer] | |
| 56 | + inf = self.inflight[layer] | |
| 57 | + for e in list(inf): | |
| 58 | + fut = inf[e] | |
| 59 | + if e in wait_for and not fut.done(): | |
| 60 | + t = time.perf_counter() | |
| 61 | + fut.result() | |
| 62 | + tab.stall_time += time.perf_counter() - t # remaining latency of a late prefetch | |
| 63 | + self.late[layer] += 1 | |
| 64 | + if fut.done(): | |
| 65 | + tensors, _ = fut.result() | |
| 66 | + del inf[e] | |
| 67 | + if tab.insert_prefetched(e, tensors, protected) is not None: | |
| 68 | + self.installed[layer] += 1 | |
| 69 | + | |
| 70 | + # ---- hooks called by the execution block --------------------------------------- | |
| 71 | + def before_layer(self, layer: int, needed: List[int]): | |
| 72 | + """Called with the materialised routing of `layer` before its experts are ensured.""" | |
| 73 | + self.drain(layer, needed, wait_for=[e for e in needed if e in self.inflight[layer]]) | |
| 74 | + if hasattr(self.pred, "set_true"): | |
| 75 | + pass | |
| 76 | + | |
| 77 | + def after_layer(self, layer: int, uniq: List[int], probs: Optional[np.ndarray] = None): | |
| 78 | + self.cur[layer] = np.array(uniq) | |
| 79 | + if probs is not None: | |
| 80 | + self.cur[("probs", layer)] = probs | |
| 81 | + if self.pred.horizon == "layer" and layer + 1 < self.L: | |
| 82 | + self.issue(layer + 1, [int(e) for e in self.pred.predict(layer + 1, self.prev_topk, self.prev_probs, self.cur, self.m)]) | |
| 83 | + if layer == self.L - 1: | |
| 84 | + self.prev_topk = np.array([self.cur.get(l, np.array([]))[: self.K] if len(self.cur.get(l, [])) >= self.K else np.pad(self.cur.get(l, np.array([])), (0, self.K - len(self.cur.get(l, [])))) for l in range(self.L)]) | |
| 85 | + self.prev_probs = np.stack([self.cur[("probs", l)] for l in range(self.L)]) if all(("probs", l) in self.cur for l in range(self.L)) else None | |
| 86 | + self.cur = {} | |
| 87 | + if self.pred.horizon == "token": | |
| 88 | + for l in range(self.L): | |
| 89 | + self.issue(l, [int(e) for e in self.pred.predict(l, self.prev_topk, self.prev_probs, {}, self.m)]) | |
| 90 | + | |
| 91 | + def stats(self) -> dict: | |
| 92 | + return {"issued": sum(self.issued), "installed": sum(self.installed), "late": sum(self.late), "dup": sum(self.dup), | |
| 93 | + "predictor": self.pred.name, "m": self.m} | |
| 94 | + | |
| 95 | + def reset_stats(self): | |
| 96 | + self.issued = [0] * self.L; self.installed = [0] * self.L; self.late = [0] * self.L; self.dup = [0] * self.L | |
| 97 | + | |
| 98 | + def close(self): | |
| 99 | + self.pool.shutdown(wait=True) | |
modified
titan/runtime.py
+34 −3
@@ -12,7 +12,9 @@ import mlx.core as mx | ||
| 12 | 12 | import mlx.nn as nn |
| 13 | 13 | from mlx_lm.utils import load_model, load_tokenizer |
| 14 | 14 | |
| 15 | −from .moe import TitanSwitchGLU | |
| 15 | +from .moe import TitanMoEBlock, TitanSwitchGLU | |
| 16 | +from .prefetch import Prefetcher | |
| 17 | +from . import predictors as _pred | |
| 16 | 18 | from .registry import ModelSpec, load_spec |
| 17 | 19 | from .slots import EvictionPolicy, LFUPolicy, SlotTable |
| 18 | 20 | from .store import ExpertStore |
@@ -22,7 +24,7 @@ POLICIES = {"lru": EvictionPolicy, "lfu": LFUPolicy} | ||
| 22 | 24 | |
| 23 | 25 | class TitanModel: |
| 24 | 26 | def __init__(self, model_dir: str, capacity: Optional[int] = None, policy: str = "lru", nocache: bool = False, |
| 25 | − on_route=None): | |
| 27 | + on_route=None, predictor: str = "none", m: int = 8, prefetch_workers: int = 2, traces_dir: Optional[str] = None): | |
| 26 | 28 | self.model_dir = os.path.realpath(model_dir) |
| 27 | 29 | self.spec: ModelSpec = load_spec(self.model_dir) |
| 28 | 30 | self.store = ExpertStore(self.spec, nocache=nocache) |
@@ -45,6 +47,17 @@ class TitanModel: | ||
| 45 | 47 | self.tables.append(table) |
| 46 | 48 | self.blocks.append(block) |
| 47 | 49 | mx.eval(self.model.parameters()) # dense weights only |
| 50 | + # optional predictive prefetcher: replaces each layer's MoE block with the hooked exact-routing block | |
| 51 | + self.prefetcher = None | |
| 52 | + self.predictor_name = predictor | |
| 53 | + if predictor != "none": | |
| 54 | + pred = _pred.make(predictor, self.spec.num_layers, self.spec.num_experts) | |
| 55 | + if traces_dir is not None and hasattr(pred, "fit"): | |
| 56 | + pred.fit(_load_fit_traces(traces_dir)) | |
| 57 | + self.prefetcher = Prefetcher(self.tables, self.store, pred, m=m, workers=prefetch_workers) | |
| 58 | + for l, layer in enumerate(self.model.model.layers): | |
| 59 | + self.blocks[l].prefetcher = self.prefetcher | |
| 60 | + layer.mlp = TitanMoEBlock(layer.mlp, self.blocks[l], self.spec.model_type, self.prefetcher) | |
| 48 | 61 | |
| 49 | 62 | # convenience ----------------------------------------------------------------- |
| 50 | 63 | @property |
@@ -63,13 +76,31 @@ class TitanModel: | ||
| 63 | 76 | |
| 64 | 77 | def cache_stats(self) -> Dict: |
| 65 | 78 | per = [t.stats() for t in self.tables] |
| 79 | + if self.prefetcher is not None: | |
| 80 | + per_pf = self.prefetcher.stats() | |
| 81 | + else: | |
| 82 | + per_pf = None | |
| 66 | 83 | hits = sum(p["hits"] + p["hits_device"] for p in per) |
| 67 | 84 | misses = sum(p["misses"] for p in per) |
| 68 | 85 | return {"layers": per, "hits": hits, "misses": misses, |
| 69 | 86 | "hit_rate": hits / (hits + misses) if hits + misses else None, |
| 70 | − "stall_time_s": sum(p["stall_time_s"] for p in per), "store": self.store.stats()} | |
| 87 | + "stall_time_s": sum(p["stall_time_s"] for p in per), "store": self.store.stats(), "prefetch": per_pf} | |
| 71 | 88 | |
| 72 | 89 | def reset_stats(self): |
| 73 | 90 | for t in self.tables: |
| 74 | 91 | t.reset_stats() |
| 75 | 92 | self.store.reset_stats() |
| 93 | + if self.prefetcher is not None: | |
| 94 | + self.prefetcher.reset_stats() | |
| 95 | + | |
| 96 | + | |
| 97 | +def _load_fit_traces(traces_dir: str): | |
| 98 | + import json, numpy as np | |
| 99 | + idx = json.load(open(os.path.join(traces_dir, "index.json"))) | |
| 100 | + out = [] | |
| 101 | + for p in idx["prompts"]: | |
| 102 | + if p["split"] != "fit": | |
| 103 | + continue | |
| 104 | + z = np.load(os.path.join(traces_dir, p["name"] + ".npz")) | |
| 105 | + out.append((z["topk"].astype(np.int64), z["probs"])) | |
| 106 | + return out | |
modified
titan/slots.py
+15 −13
@@ -105,6 +105,7 @@ class SlotTable: | ||
| 105 | 105 | self.buf[name][slot] = arr |
| 106 | 106 | self.slot_of[expert] = slot |
| 107 | 107 | self.expert_in[slot] = expert |
| 108 | + self.policy.touch(expert) # newly installed experts are MRU (prefetched ones included), so they are always evictable | |
| 108 | 109 | self.loads += 1 |
| 109 | 110 | self._complete = False |
| 110 | 111 | return slot |
@@ -118,13 +119,12 @@ class SlotTable: | ||
| 118 | 119 | return self._complete |
| 119 | 120 | |
| 120 | 121 | def load(self, expert: int, protected: Iterable[int] = ()) -> int: |
| 121 | − """Synchronously load an expert from the store into a slot (demand miss).""" | |
| 122 | + """Load an expert from the store into a slot (demand miss). The slot write is a lazy slice assignment that the | |
| 123 | + layer's gather_qmm will consume: no per-expert GPU synchronisation (the stall counted here is the read time).""" | |
| 122 | 124 | t = time.perf_counter() |
| 123 | 125 | tensors = self.store.read_expert(self.layer, expert) |
| 124 | − slot = self._install(expert, tensors, protected) | |
| 125 | − mx.eval(*self.buf.values()) | |
| 126 | 126 | self.stall_time += time.perf_counter() - t |
| 127 | − return slot | |
| 127 | + return self._install(expert, tensors, protected) | |
| 128 | 128 | |
| 129 | 129 | def insert_prefetched(self, expert: int, tensors: Dict[str, mx.array], protected: Iterable[int] = ()) -> Optional[int]: |
| 130 | 130 | """Install tensors that were fetched asynchronously (no stall accounted).""" |
@@ -135,18 +135,20 @@ class SlotTable: | ||
| 135 | 135 | return self._install(expert, tensors, protected) |
| 136 | 136 | |
| 137 | 137 | def ensure(self, experts: List[int]) -> List[int]: |
| 138 | − """Make every expert resident (demand-loading misses) and return their slot indices.""" | |
| 139 | − slots = [] | |
| 138 | + """Make every expert resident (demand-loading misses, read as a batch) and return their slot indices.""" | |
| 140 | 139 | needed = set(experts) |
| 140 | + missing = [e for e in experts if e not in self.slot_of] | |
| 141 | + self.hits += len(experts) - len(missing) | |
| 142 | + self.misses += len(missing) | |
| 143 | + if missing: | |
| 144 | + t = time.perf_counter() | |
| 145 | + tensors = {e: self.store.read_expert(self.layer, e) for e in missing} | |
| 146 | + self.stall_time += time.perf_counter() - t | |
| 147 | + for e in missing: | |
| 148 | + self._install(e, tensors[e], needed) | |
| 141 | 149 | for e in experts: |
| 142 | − if e in self.slot_of: | |
| 143 | − self.hits += 1 | |
| 144 | − else: | |
| 145 | − self.misses += 1 | |
| 146 | − self.load(e, protected=needed) | |
| 147 | 150 | self.policy.touch(e) |
| 148 | − slots.append(self.slot_of[e]) | |
| 149 | − return slots | |
| 151 | + return [self.slot_of[e] for e in experts] | |
| 150 | 152 | |
| 151 | 153 | def resident(self) -> List[int]: |
| 152 | 154 | return list(self.slot_of) |
| 153 | 155 | |