SPB Git

spb/localvm-research Public License

Running LLMs larger than memory on a consumer Mac — falsification-driven research: margin-gated deferred refinement, out-of-core verification on Apple Silicon. TR-01 published.

Python 63.2% JavaScript 23.5% CSS 11.8% Shell 0.9% Makefile 0.5%

expF: NEGATIVE result — quantization damage diffuse across depth, no layer leverage

1.5x sensitivity spread (kill <2x); top-29% of layers repair only 24%
of lost agreement; masking + cooperative repair effects documented.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 5 h ago (Aug 12, 2026) parent 4398286

Showing 6 changed files with +1,066 and −7

modified experiments/micro/expF_error_accumulation/analysis.md +59 −3
@@ -3,10 +3,66 @@ project: localvm-research
3 3 document: expF_error_accumulation/analysis
4 4 author: Simon-Pierre Boucher
5 5 contact: contact@spboucher.ai
6 −created: 2026-08-11
7 −status: draft
6 +created: 2026-08-12
7 +status: reviewed
8 8 ---
9 9
10 10 # Analysis — expF_error_accumulation
11 11
12 −*To be written after results exist. Must include the seven-field block and the evidence standard of CLAUDE.md §10.*
12 +Run: `results/expF_error_accumulation/20260812T043945Z/` · code committed before
13 +run. Qwen3-1.7B, 28 layers in 7 depth-groups of 4, affine g64 4-bit, 48
14 +trajectories × 128 tokens teacher-forced, 17 configurations.
15 +
16 +```text
17 +Hypothesis / Falsification
18 + Hypothesized ≥3× sensitivity spread across depth groups and ≥40% of lost
19 + agreement recovered by repairing the top ~25% of layers.
20 + Kill criteria: <2× spread, or top-25% repair recovering <20%.
21 +
22 +Result — HYPOTHESIS REFUTED (spread criterion killed; repair borderline-negative)
23 + All-4-bit floor: 87.53% agreement (loss = 12.47 points vs bf16).
24 + DEGRADE-ONE (one group 4-bit, rest bf16): agreement drops span only
25 + 3.7–5.6 points across the 7 groups — a 1.5× spread (< the 2× kill
26 + line). Mid-depth groups (L12–19) are marginally most sensitive; the
27 + LAST group is among the LEAST sensitive (3.96) despite expD showing
28 + the largest hidden-state error there — late-layer error is large in
29 + norm but decision-benign.
30 + REPAIR-ONE (one group bf16, rest 4-bit): best single group (L12–15)
31 + recovers +1.5 points = 12% of the loss; worst 5%.
32 + REPAIR-TOP-K: top-2 groups (29% of layers) recover 24% of the loss;
33 + top-3 (43% of layers) recover 33% — consistently SUB-proportional.
34 + Cross-check of additivity: individual degrade-one drops sum to ~33
35 + points, yet degrading everything at once costs only 12.5 — errors
36 + partially mask each other; symmetrically, repair values sum to 7.4
37 + of the 12.5 lost — repair requires cooperation across depth.
38 +
39 +Interpretation
40 + 1. NEGATIVE RESULT (recorded per charter §10/§17): quantization damage
41 + to token decisions is DIFFUSE and cooperative across depth, not
42 + concentrated. Layer-restricted escalation cannot materially cut
43 + bytes-per-escalation: paying 29% of the residual bytes buys only
44 + 24% of the repair — worse than linear, no leverage.
45 + 2. expD's depth-concentrated hidden-state error was a red herring for
46 + decision repair: large late-layer representation drift coexists with
47 + benign decisions (norms ≠ decisions — a caution for any design that
48 + gates on hidden-state error instead of decision margin).
49 + 3. Consequently C1's bytes-per-escalation must come from the remaining
50 + mechanisms: (b) temporal locality — the residual working set of
51 + escalated tokens staying hot in RAM (expB); (c) sub-layer/block-level
52 + selection — repair only the weight blocks that matter for THIS token
53 + (expA/expE, finer grain than layers); or (d) batch-amortized
54 + escalation — G17/G18-style verification sweeps sharing one residual
55 + read across many queued low-margin tokens.
56 + 4. Design implication: if (b) also fails, C1 degrades into "resident
57 + 4-bit + rare whole-model refinement passes" — which is exactly
58 + candidate C2 (amortized verification). The two candidates are
59 + converging on the same mechanism from opposite ends; this is
60 + useful, not disappointing.
61 +
62 +Next experiment
63 + expA (weight contribution concentration at BLOCK granularity within
64 + layers) — the finer-grained version of the question expF just answered
65 + negatively at layer granularity; feeds expE (partial GEMM) directly.
66 + In parallel, expB (temporal stability of important blocks) decides the
67 + page-cache path (b).
68 +```
added experiments/micro/expF_error_accumulation/expF_full.log +27 −0
@@ -0,0 +1,27 @@
1 +loading mlx-community/Qwen3-1.7B-bf16 …
2 + Fetching 9 files: 0%| | 0/9 [00:00<?, ?it/s] Fetching 9 files: 100%|██████████| 9/9 [00:00<00:00, 5729.92it/s]
3 +48 trajectories in 49s
4 +precomputing 4-bit weights …
5 +all-4-bit floor:
6 + all_4bit: agree=0.8753 KL=0.1982
7 +degrade-one (rest bf16):
8 + degrade_g0_L0-3: agree=0.9473 KL=0.0344
9 + degrade_g1_L4-7: agree=0.9631 KL=0.0232
10 + degrade_g2_L8-11: agree=0.9505 KL=0.0328
11 + degrade_g3_L12-15: agree=0.9460 KL=0.0399
12 + degrade_g4_L16-19: agree=0.9440 KL=0.0422
13 + degrade_g5_L20-23: agree=0.9601 KL=0.0176
14 + degrade_g6_L24-27: agree=0.9604 KL=0.0160
15 +repair-one (rest 4-bit):
16 + repair_g0_L0-3: agree=0.8880 KL=0.1765
17 + repair_g1_L4-7: agree=0.8864 KL=0.1738
18 + repair_g2_L8-11: agree=0.8815 KL=0.1715
19 + repair_g3_L12-15: agree=0.8905 KL=0.1551
20 + repair_g4_L16-19: agree=0.8892 KL=0.1546
21 + repair_g5_L20-23: agree=0.8838 KL=0.1824
22 + repair_g6_L24-27: agree=0.8815 KL=0.1879
23 +repair-top-k (best groups bf16):
24 + repair_top2_groups_[3, 4]: agree=0.9053 KL=0.1126
25 + repair_top3_groups_[0, 3, 4]: agree=0.9160 KL=0.0865
26 +
27 +wrote /Users/simon-pierreboucher/Desktop/localvm-research/results/expF_error_accumulation/20260812T043945Z/results.json
modified experiments/micro/expF_error_accumulation/hypothesis.md +13 −4
@@ -4,7 +4,8 @@ document: expF_error_accumulation/hypothesis
4 4 author: Simon-Pierre Boucher
5 5 contact: contact@spboucher.ai
6 6 created: 2026-08-12
7 −status: draft
7 +modified: 2026-08-12
8 +status: reviewed
8 9 ---
9 10
10 11 # Hypothesis — expF_error_accumulation
@@ -48,11 +49,19 @@ Baseline
48 49 stage0 B) bracket every configuration. No straw men.
49 50
50 51 Result
51 − <filled after the run>
52 + REFUTED. Sensitivity spread across depth groups: 1.5× (kill line: <2×).
53 + Top-2 groups (29% of layers) repair only 24% of lost agreement —
54 + sub-proportional; repair values sum to 7.4 of 12.5 points lost (repair
55 + is cooperative, not concentrated). Full numbers:
56 + results/expF_error_accumulation/20260812T043945Z/ and analysis.md.
52 57
53 58 Interpretation
54 − <filled after the run>
59 + Quantization damage to decisions is diffuse across depth. Layer-
60 + restricted escalation gives no leverage; expD's late-layer hidden-state
61 + error is decision-benign (norms ≠ decisions). C1's bytes-per-escalation
62 + must come from temporal locality (expB), block-level selection
63 + (expA/expE), or batch amortization (→ convergence with C2).
55 64
56 65 Next experiment
57 − <filled after the run>
66 + expA (block-granularity concentration) + expB (temporal stability).
58 67 ```
modified research/LOG.md +19 −0
@@ -200,3 +200,22 @@ Format per entry: date/time (local, with timezone) · question · experiment ·
200 200 budget; last-layer error concentration (0.83 vs 0.34 at 4-bit) suggests layer-restricted
201 201 escalation could cut it ~4×.
202 202 - **Decision:** expF (layer sensitivity) next; then expB (temporal locality of escalated set).
203 +
204 +---
205 +
206 +## 2026-08-12 12:05 EDT — expF complete: NEGATIVE — layer-restricted escalation gives no leverage
207 +
208 +- **Question:** Is quantization damage concentrated in a subset of layers, so escalation can
209 + be layer-restricted and bytes-per-escalation cut proportionally?
210 +- **Experiment:** expF — 7 depth-groups × degrade-one/repair-one + repair-top-k, 4-bit affine
211 + vs bf16, Qwen3-1.7B, 48 trajectories, 17 configs.
212 + Results: `results/expF_error_accumulation/20260812T043945Z/`.
213 +- **Result:** REFUTED. Sensitivity spread 1.5× (kill: <2×). Top-29% of layers repair 24% of
214 + lost agreement (sub-proportional). Degrade-one drops sum to 33 pts vs 12.5 joint (masking);
215 + repair-one values sum to 7.4 of 12.5 (cooperation). Last layers: largest hidden-state error
216 + (expD) but LEAST decision-sensitive — norms ≠ decisions.
217 +- **Interpretation:** Damage is diffuse across depth. C1's escalation-byte problem must be
218 + solved by temporal locality (expB), block-level selection (expA/expE), or batch amortization
219 + — the last converging with C2 (amortized verification). Negative result recorded per §10/§17.
220 +- **Decision:** Next: expA (block-granularity concentration) and expB (temporal stability) —
221 + the trace campaign that also feeds G06/G07/G12/G13 per the Phase 3 cluster analysis.
added results/expF_error_accumulation/20260812T043757Z/results.json +474 −0
@@ -0,0 +1,474 @@
1 +{
2 + "experiment": "expF_error_accumulation",
3 + "author": "Simon-Pierre Boucher",
4 + "contact": "contact@spboucher.ai",
5 + "manifest": {
6 + "author": "Simon-Pierre Boucher",
7 + "contact": "contact@spboucher.ai",
8 + "project": "localvm-research",
9 + "collected_utc": "2026-08-12T04:37:57.954944+00:00",
10 + "chip": {
11 + "brand": "Apple M5 Max",
12 + "arch": "arm64",
13 + "cores_total": 18,
14 + "cores_performance": 6,
15 + "cores_efficiency": 12,
16 + "gpu_cores": 40
17 + },
18 + "memory": {
19 + "unified_bytes": 51539607552,
20 + "unified_gb": 48.0,
21 + "pagesize": 16384
22 + },
23 + "ssd": {
24 + "model": "APPLE SSD AP2048Z",
25 + "size": "2 TB",
26 + "smart_status": "Verified"
27 + },
28 + "os": {
29 + "product": "macOS",
30 + "version": "27.0",
31 + "build": "26A5388g",
32 + "kernel": "27.0.0"
33 + },
34 + "software": {
35 + "python": "3.14.4",
36 + "mlx": "0.32.0",
37 + "mlx_lm": "0.31.3",
38 + "torch": null,
39 + "numpy": "2.5.2"
40 + },
41 + "git": {
42 + "commit": "43982862c39de18d7d36f2b8e6888d2e42e0a420",
43 + "dirty_tree": false
44 + },
45 + "thermal_level_at_collect": null
46 + },
47 + "config": {
48 + "model": "mlx-community/Qwen3-1.7B-bf16",
49 + "gen_tokens": 24,
50 + "per_domain": 1,
51 + "groups": 7
52 + },
53 + "bits": 4,
54 + "group_size": 64,
55 + "n_layers": 28,
56 + "layer_groups": [
57 + [
58 + 0,
59 + 1,
60 + 2,
61 + 3
62 + ],
63 + [
64 + 4,
65 + 5,
66 + 6,
67 + 7
68 + ],
69 + [
70 + 8,
71 + 9,
72 + 10,
73 + 11
74 + ],
75 + [
76 + 12,
77 + 13,
78 + 14,
79 + 15
80 + ],
81 + [
82 + 16,
83 + 17,
84 + 18,
85 + 19
86 + ],
87 + [
88 + 20,
89 + 21,
90 + 22,
91 + 23
92 + ],
93 + [
94 + 24,
95 + 25,
96 + 26,
97 + 27
98 + ]
99 + ],
100 + "agreement_lost_all4bit": 0.04861111111111116,
101 + "repair_value_by_group": {
102 + "0": 0.006944444444444531,
103 + "1": 0.0,
104 + "2": -0.00694444444444442,
105 + "3": 0.01388888888888895,
106 + "4": 0.0,
107 + "5": 0.0,
108 + "6": 0.006944444444444531
109 + },
110 + "runs": {
111 + "all_4bit": {
112 + "agreement_rate": 0.9513888888888888,
113 + "mean_kl": 0.0858410894870758,
114 + "degraded_layers": [
115 + 0,
116 + 1,
117 + 2,
118 + 3,
119 + 4,
120 + 5,
121 + 6,
122 + 7,
123 + 8,
124 + 9,
125 + 10,
126 + 11,
127 + 12,
128 + 13,
129 + 14,
130 + 15,
131 + 16,
132 + 17,
133 + 18,
134 + 19,
135 + 20,
136 + 21,
137 + 22,
138 + 23,
139 + 24,
140 + 25,
141 + 26,
142 + 27
143 + ]
144 + },
145 + "degrade_g0_L0-3": {
146 + "agreement_rate": 0.9861111111111112,
147 + "mean_kl": 0.00840843003243208,
148 + "degraded_layers": [
149 + 0,
150 + 1,
151 + 2,
152 + 3
153 + ]
154 + },
155 + "degrade_g1_L4-7": {
156 + "agreement_rate": 0.9930555555555556,
157 + "mean_kl": 0.006097735371440649,
158 + "degraded_layers": [
159 + 4,
160 + 5,
161 + 6,
162 + 7
163 + ]
164 + },
165 + "degrade_g2_L8-11": {
166 + "agreement_rate": 0.9652777777777778,
167 + "mean_kl": 0.009526032023131847,
168 + "degraded_layers": [
169 + 8,
170 + 9,
171 + 10,
172 + 11
173 + ]
174 + },
175 + "degrade_g3_L12-15": {
176 + "agreement_rate": 0.9791666666666666,
177 + "mean_kl": 0.014411929994821548,
178 + "degraded_layers": [
179 + 12,
180 + 13,
181 + 14,
182 + 15
183 + ]
184 + },
185 + "degrade_g4_L16-19": {
186 + "agreement_rate": 0.9791666666666666,
187 + "mean_kl": 0.014998848550021648,
188 + "degraded_layers": [
189 + 16,
190 + 17,
191 + 18,
192 + 19
193 + ]
194 + },
195 + "degrade_g5_L20-23": {
196 + "agreement_rate": 0.9861111111111112,
197 + "mean_kl": 0.010291025973856449,
198 + "degraded_layers": [
199 + 20,
200 + 21,
201 + 22,
202 + 23
203 + ]
204 + },
205 + "degrade_g6_L24-27": {
206 + "agreement_rate": 0.9861111111111112,
207 + "mean_kl": 0.005828259978443384,
208 + "degraded_layers": [
209 + 24,
210 + 25,
211 + 26,
212 + 27
213 + ]
214 + },
215 + "repair_g0_L0-3": {
216 + "agreement_rate": 0.9583333333333334,
217 + "mean_kl": 0.07027784734964371,
218 + "degraded_layers": [
219 + 4,
220 + 5,
221 + 6,
222 + 7,
223 + 8,
224 + 9,
225 + 10,
226 + 11,
227 + 12,
228 + 13,
229 + 14,
230 + 15,
231 + 16,
232 + 17,
233 + 18,
234 + 19,
235 + 20,
236 + 21,
237 + 22,
238 + 23,
239 + 24,
240 + 25,
241 + 26,
242 + 27
243 + ]
244 + },
245 + "repair_g1_L4-7": {
246 + "agreement_rate": 0.9513888888888888,
247 + "mean_kl": 0.06150852143764496,
248 + "degraded_layers": [
249 + 0,
250 + 1,
251 + 2,
252 + 3,
253 + 8,
254 + 9,
255 + 10,
256 + 11,
257 + 12,
258 + 13,
259 + 14,
260 + 15,
261 + 16,
262 + 17,
263 + 18,
264 + 19,
265 + 20,
266 + 21,
267 + 22,
268 + 23,
269 + 24,
270 + 25,
271 + 26,
272 + 27
273 + ]
274 + },
275 + "repair_g2_L8-11": {
276 + "agreement_rate": 0.9444444444444444,
277 + "mean_kl": 0.088611900806427,
278 + "degraded_layers": [
279 + 0,
280 + 1,
281 + 2,
282 + 3,
283 + 4,
284 + 5,
285 + 6,
286 + 7,
287 + 12,
288 + 13,
289 + 14,
290 + 15,
291 + 16,
292 + 17,
293 + 18,
294 + 19,
295 + 20,
296 + 21,
297 + 22,
298 + 23,
299 + 24,
300 + 25,
301 + 26,
302 + 27
303 + ]
304 + },
305 + "repair_g3_L12-15": {
306 + "agreement_rate": 0.9652777777777778,
307 + "mean_kl": 0.04590831696987152,
308 + "degraded_layers": [
309 + 0,
310 + 1,
311 + 2,
312 + 3,
313 + 4,
314 + 5,
315 + 6,
316 + 7,
317 + 8,
318 + 9,
319 + 10,
320 + 11,
321 + 16,
322 + 17,
323 + 18,
324 + 19,
325 + 20,
326 + 21,
327 + 22,
328 + 23,
329 + 24,
330 + 25,
331 + 26,
332 + 27
333 + ]
334 + },
335 + "repair_g4_L16-19": {
336 + "agreement_rate": 0.9513888888888888,
337 + "mean_kl": 0.08659553527832031,
338 + "degraded_layers": [
339 + 0,
340 + 1,
341 + 2,
342 + 3,
343 + 4,
344 + 5,
345 + 6,
346 + 7,
347 + 8,
348 + 9,
349 + 10,
350 + 11,
351 + 12,
352 + 13,
353 + 14,
354 + 15,
355 + 20,
356 + 21,
357 + 22,
358 + 23,
359 + 24,
360 + 25,
361 + 26,
362 + 27
363 + ]
364 + },
365 + "repair_g5_L20-23": {
366 + "agreement_rate": 0.9513888888888888,
367 + "mean_kl": 0.06335537880659103,
368 + "degraded_layers": [
369 + 0,
370 + 1,
371 + 2,
372 + 3,
373 + 4,
374 + 5,
375 + 6,
376 + 7,
377 + 8,
378 + 9,
379 + 10,
380 + 11,
381 + 12,
382 + 13,
383 + 14,
384 + 15,
385 + 16,
386 + 17,
387 + 18,
388 + 19,
389 + 24,
390 + 25,
391 + 26,
392 + 27
393 + ]
394 + },
395 + "repair_g6_L24-27": {
396 + "agreement_rate": 0.9583333333333334,
397 + "mean_kl": 0.08016648888587952,
398 + "degraded_layers": [
399 + 0,
400 + 1,
401 + 2,
402 + 3,
403 + 4,
404 + 5,
405 + 6,
406 + 7,
407 + 8,
408 + 9,
409 + 10,
410 + 11,
411 + 12,
412 + 13,
413 + 14,
414 + 15,
415 + 16,
416 + 17,
417 + 18,
418 + 19,
419 + 20,
420 + 21,
421 + 22,
422 + 23
423 + ]
424 + },
425 + "repair_top2_groups_[0, 3]": {
426 + "agreement_rate": 0.9652777777777778,
427 + "mean_kl": 0.05832972005009651,
428 + "degraded_layers": [
429 + 4,
430 + 5,
431 + 6,
432 + 7,
433 + 8,
434 + 9,
435 + 10,
436 + 11,
437 + 16,
438 + 17,
439 + 18,
440 + 19,
441 + 20,
442 + 21,
443 + 22,
444 + 23,
445 + 24,
446 + 25,
447 + 26,
448 + 27
449 + ]
450 + },
451 + "repair_top3_groups_[0, 3, 6]": {
452 + "agreement_rate": 0.9513888888888888,
453 + "mean_kl": 0.04880715161561966,
454 + "degraded_layers": [
455 + 4,
456 + 5,
457 + 6,
458 + 7,
459 + 8,
460 + 9,
461 + 10,
462 + 11,
463 + 16,
464 + 17,
465 + 18,
466 + 19,
467 + 20,
468 + 21,
469 + 22,
470 + 23
471 + ]
472 + }
473 + }
474 +}
\ No newline at end of file
added results/expF_error_accumulation/20260812T043945Z/results.json +474 −0
@@ -0,0 +1,474 @@
1 +{
2 + "experiment": "expF_error_accumulation",
3 + "author": "Simon-Pierre Boucher",
4 + "contact": "contact@spboucher.ai",
5 + "manifest": {
6 + "author": "Simon-Pierre Boucher",
7 + "contact": "contact@spboucher.ai",
8 + "project": "localvm-research",
9 + "collected_utc": "2026-08-12T04:39:45.619501+00:00",
10 + "chip": {
11 + "brand": "Apple M5 Max",
12 + "arch": "arm64",
13 + "cores_total": 18,
14 + "cores_performance": 6,
15 + "cores_efficiency": 12,
16 + "gpu_cores": 40
17 + },
18 + "memory": {
19 + "unified_bytes": 51539607552,
20 + "unified_gb": 48.0,
21 + "pagesize": 16384
22 + },
23 + "ssd": {
24 + "model": "APPLE SSD AP2048Z",
25 + "size": "2 TB",
26 + "smart_status": "Verified"
27 + },
28 + "os": {
29 + "product": "macOS",
30 + "version": "27.0",
31 + "build": "26A5388g",
32 + "kernel": "27.0.0"
33 + },
34 + "software": {
35 + "python": "3.14.4",
36 + "mlx": "0.32.0",
37 + "mlx_lm": "0.31.3",
38 + "torch": null,
39 + "numpy": "2.5.2"
40 + },
41 + "git": {
42 + "commit": "43982862c39de18d7d36f2b8e6888d2e42e0a420",
43 + "dirty_tree": true
44 + },
45 + "thermal_level_at_collect": null
46 + },
47 + "config": {
48 + "model": "mlx-community/Qwen3-1.7B-bf16",
49 + "gen_tokens": 128,
50 + "per_domain": 8,
51 + "groups": 7
52 + },
53 + "bits": 4,
54 + "group_size": 64,
55 + "n_layers": 28,
56 + "layer_groups": [
57 + [
58 + 0,
59 + 1,
60 + 2,
61 + 3
62 + ],
63 + [
64 + 4,
65 + 5,
66 + 6,
67 + 7
68 + ],
69 + [
70 + 8,
71 + 9,
72 + 10,
73 + 11
74 + ],
75 + [
76 + 12,
77 + 13,
78 + 14,
79 + 15
80 + ],
81 + [
82 + 16,
83 + 17,
84 + 18,
85 + 19
86 + ],
87 + [
88 + 20,
89 + 21,
90 + 22,
91 + 23
92 + ],
93 + [
94 + 24,
95 + 25,
96 + 26,
97 + 27
98 + ]
99 + ],
100 + "agreement_lost_all4bit": 0.12467447916666663,
101 + "repair_value_by_group": {
102 + "0": 0.0126953125,
103 + "1": 0.01106770833333326,
104 + "2": 0.006184895833333259,
105 + "3": 0.01513671875,
106 + "4": 0.01383463541666663,
107 + "5": 0.00846354166666663,
108 + "6": 0.006184895833333259
109 + },
110 + "runs": {
111 + "all_4bit": {
112 + "agreement_rate": 0.8753255208333334,
113 + "mean_kl": 0.1982055902481079,
114 + "degraded_layers": [
115 + 0,
116 + 1,
117 + 2,
118 + 3,
119 + 4,
120 + 5,
121 + 6,
122 + 7,
123 + 8,
124 + 9,
125 + 10,
126 + 11,
127 + 12,
128 + 13,
129 + 14,
130 + 15,
131 + 16,
132 + 17,
133 + 18,
134 + 19,
135 + 20,
136 + 21,
137 + 22,
138 + 23,
139 + 24,
140 + 25,
141 + 26,
142 + 27
143 + ]
144 + },
145 + "degrade_g0_L0-3": {
146 + "agreement_rate": 0.947265625,
147 + "mean_kl": 0.03436492383480072,
148 + "degraded_layers": [
149 + 0,
150 + 1,
151 + 2,
152 + 3
153 + ]
154 + },
155 + "degrade_g1_L4-7": {
156 + "agreement_rate": 0.9630533854166666,
157 + "mean_kl": 0.023244991898536682,
158 + "degraded_layers": [
159 + 4,
160 + 5,
161 + 6,
162 + 7
163 + ]
164 + },
165 + "degrade_g2_L8-11": {
166 + "agreement_rate": 0.9505208333333334,
167 + "mean_kl": 0.03282814100384712,
168 + "degraded_layers": [
169 + 8,
170 + 9,
171 + 10,
172 + 11
173 + ]
174 + },
175 + "degrade_g3_L12-15": {
176 + "agreement_rate": 0.9459635416666666,
177 + "mean_kl": 0.03994213789701462,
178 + "degraded_layers": [
179 + 12,
180 + 13,
181 + 14,
182 + 15
183 + ]
184 + },
185 + "degrade_g4_L16-19": {
186 + "agreement_rate": 0.9440104166666666,
187 + "mean_kl": 0.042195770889520645,
188 + "degraded_layers": [
189 + 16,
190 + 17,
191 + 18,
192 + 19
193 + ]
194 + },
195 + "degrade_g5_L20-23": {
196 + "agreement_rate": 0.9601236979166666,
197 + "mean_kl": 0.017626306042075157,
198 + "degraded_layers": [
199 + 20,
200 + 21,
201 + 22,
202 + 23
203 + ]
204 + },
205 + "degrade_g6_L24-27": {
206 + "agreement_rate": 0.96044921875,
207 + "mean_kl": 0.016035445034503937,
208 + "degraded_layers": [
209 + 24,
210 + 25,
211 + 26,
212 + 27
213 + ]
214 + },
215 + "repair_g0_L0-3": {
216 + "agreement_rate": 0.8880208333333334,
217 + "mean_kl": 0.17652283608913422,
218 + "degraded_layers": [
219 + 4,
220 + 5,
221 + 6,
222 + 7,
223 + 8,
224 + 9,
225 + 10,
226 + 11,
227 + 12,
228 + 13,
229 + 14,
230 + 15,
231 + 16,
232 + 17,
233 + 18,
234 + 19,
235 + 20,
236 + 21,
237 + 22,
238 + 23,
239 + 24,
240 + 25,
241 + 26,
242 + 27
243 + ]
244 + },
245 + "repair_g1_L4-7": {
246 + "agreement_rate": 0.8863932291666666,
247 + "mean_kl": 0.17383402585983276,
248 + "degraded_layers": [
249 + 0,
250 + 1,
251 + 2,
252 + 3,
253 + 8,
254 + 9,
255 + 10,
256 + 11,
257 + 12,
258 + 13,
259 + 14,
260 + 15,
261 + 16,
262 + 17,
263 + 18,
264 + 19,
265 + 20,
266 + 21,
267 + 22,
268 + 23,
269 + 24,
270 + 25,
271 + 26,
272 + 27
273 + ]
274 + },
275 + "repair_g2_L8-11": {
276 + "agreement_rate": 0.8815104166666666,
277 + "mean_kl": 0.17145586013793945,
278 + "degraded_layers": [
279 + 0,
280 + 1,
281 + 2,
282 + 3,
283 + 4,
284 + 5,
285 + 6,
286 + 7,
287 + 12,
288 + 13,
289 + 14,
290 + 15,
291 + 16,
292 + 17,
293 + 18,
294 + 19,
295 + 20,
296 + 21,
297 + 22,
298 + 23,
299 + 24,
300 + 25,
301 + 26,
302 + 27
303 + ]
304 + },
305 + "repair_g3_L12-15": {
306 + "agreement_rate": 0.8904622395833334,
307 + "mean_kl": 0.1551409363746643,
308 + "degraded_layers": [
309 + 0,
310 + 1,
311 + 2,
312 + 3,
313 + 4,
314 + 5,
315 + 6,
316 + 7,
317 + 8,
318 + 9,
319 + 10,
320 + 11,
321 + 16,
322 + 17,
323 + 18,
324 + 19,
325 + 20,
326 + 21,
327 + 22,
328 + 23,
329 + 24,
330 + 25,
331 + 26,
332 + 27
333 + ]
334 + },
335 + "repair_g4_L16-19": {
336 + "agreement_rate": 0.88916015625,
337 + "mean_kl": 0.15464557707309723,
338 + "degraded_layers": [
339 + 0,
340 + 1,
341 + 2,
342 + 3,
343 + 4,
344 + 5,
345 + 6,
346 + 7,
347 + 8,
348 + 9,
349 + 10,
350 + 11,
351 + 12,
352 + 13,
353 + 14,
354 + 15,
355 + 20,
356 + 21,
357 + 22,
358 + 23,
359 + 24,
360 + 25,
361 + 26,
362 + 27
363 + ]
364 + },
365 + "repair_g5_L20-23": {
366 + "agreement_rate": 0.8837890625,
367 + "mean_kl": 0.18242128193378448,
368 + "degraded_layers": [
369 + 0,
370 + 1,
371 + 2,
372 + 3,
373 + 4,
374 + 5,
375 + 6,
376 + 7,
377 + 8,
378 + 9,
379 + 10,
380 + 11,
381 + 12,
382 + 13,
383 + 14,
384 + 15,
385 + 16,
386 + 17,
387 + 18,
388 + 19,
389 + 24,
390 + 25,
391 + 26,
392 + 27
393 + ]
394 + },
395 + "repair_g6_L24-27": {
396 + "agreement_rate": 0.8815104166666666,
397 + "mean_kl": 0.18794463574886322,
398 + "degraded_layers": [
399 + 0,
400 + 1,
401 + 2,
402 + 3,
403 + 4,
404 + 5,
405 + 6,
406 + 7,
407 + 8,
408 + 9,
409 + 10,
410 + 11,
411 + 12,
412 + 13,
413 + 14,
414 + 15,
415 + 16,
416 + 17,
417 + 18,
418 + 19,
419 + 20,
420 + 21,
421 + 22,
422 + 23
423 + ]
424 + },
425 + "repair_top2_groups_[3, 4]": {
426 + "agreement_rate": 0.9052734375,
427 + "mean_kl": 0.1125805675983429,
428 + "degraded_layers": [
429 + 0,
430 + 1,
431 + 2,
432 + 3,
433 + 4,
434 + 5,
435 + 6,
436 + 7,
437 + 8,
438 + 9,
439 + 10,
440 + 11,
441 + 20,
442 + 21,
443 + 22,
444 + 23,
445 + 24,
446 + 25,
447 + 26,
448 + 27
449 + ]
450 + },
451 + "repair_top3_groups_[0, 3, 4]": {
452 + "agreement_rate": 0.916015625,
453 + "mean_kl": 0.08650139719247818,
454 + "degraded_layers": [
455 + 4,
456 + 5,
457 + 6,
458 + 7,
459 + 8,
460 + 9,
461 + 10,
462 + 11,
463 + 20,
464 + 21,
465 + 22,
466 + 23,
467 + 24,
468 + 25,
469 + 26,
470 + 27
471 + ]
472 + }
473 + }
474 +}
\ No newline at end of file
475