WP31: phases 4-7 at tiny scale, baselines, evaluation modules, paper skeleton
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
141 changed files +7,118 −29
added
CITATION.cff
+11 −0
@@ -0,0 +1,11 @@ | ||
| 1 | +cff-version: 1.2.0 | |
| 2 | +message: "If you use this software or paper, please cite it as below." | |
| 3 | +title: "PRISME: Intervention-Supervised Sparse Workspace Editing for Small Language Models (UQO Working Paper 31)" | |
| 4 | +authors: | |
| 5 | + - family-names: Boucher | |
| 6 | + given-names: Simon-Pierre | |
| 7 | + affiliation: "Université du Québec en Outaouais" | |
| 8 | + email: simon-pierre.boucher@uqo.ca | |
| 9 | +date-released: 2026-10-30 | |
| 10 | +version: "1.0" | |
| 11 | +license: MIT | |
modified
FAILURES.md
+5 −0
@@ -5,3 +5,8 @@ | ||
| 5 | 5 | | 2026-10-29 | `test_incremental_equals_full_prefix_forced`: cached decoding ≠ full-prefix forward. | pytest, atol 1e-4 | Root cause: forced-action forward let edits resume after STOP; made STOP absorbing (D4). | |
| 6 | 6 | | 2026-10-29 | rsync exclusion `experiments` also dropped `configs/experiments/`; baseline launch on M3U96b failed twice (`FileNotFoundError`). | `experiments/logs/pretrain_enlarged102m_seed1.log` (first attempts) | anchored pattern `/experiments`. | |
| 7 | 7 | | 2026-10-29 | Stage B bootstrap degeneracy (brief §13.1 confirmed): with U = 0 and lr 1e-3, 700 steps of forced mixed-budget training left val loss identical at every budget (0.4144 / 0.4143 / 0.4143 / 0.4146 / 0.4164 for budgets 0/1/2/4/8); grad-norm 0.01–0.02. Diagnosis: scale mismatch — backbone residual rms 5.64 vs readout difference rms 0.06, so U must grow by ×100 before edits can affect logits. | `experiments/runs/p4_editor_v1_seed01_*/metrics.jsonl` (stopped at step ~750) | D9: fixed non-learned `readout_scale` = measured backbone residual rms at bootstrap (keeps U = 0 init and exact identity; equivalent to a scale-aware learning rate). Variants with nonzero U init (std 0.02) and lr 3e-3 kept running on M2U64 / M4M64c as evidence. | |
| 8 | +| 2026-10-30 | Editor collapse at lr 3e-3 (w = 128 mixed budgets): after 500 steps every budget gives exactly the budget-0 loss (0.4005) and the trainable gradient norm is 0.00 — the adapter learned to make all edits inert (gate/U → 0) instead of useful. | M4M64a `experiments/logs/p4_diag_mixed_w128_lr3e3.log`, run `p4_diag_mixed_w128_lr3e3_seed01_*` | "harmless collapse" is a real failure mode of the bootstrap; lr 1e-3 retained. | | |
| 9 | +| 2026-10-30 | Global-recurrence baseline run crashed at the final exact-match evaluation (`AttributeError: 'GlobalRecurrenceModel' object has no attribute 'W_K'`): the cached generator assumed PRISME caches. Training and the step-3000 checkpoint are intact. | M5M36 `experiments/logs/p4_gr_mixed_w64.log` | added `generate_uncached` fallback (full recomputation) for models without the bank; EM re-run from the checkpoint. | | |
| 10 | +| 2026-10-30 | Joint fine-tuning reference (backbone unfrozen, lr 3e-4, answer-span loss): validation loss rose from 0.4005 to 0.472 at every budget after 500 steps — the backbone degraded under the narrower objective and the large learning rate. | M4M64a `experiments/logs/p4_joint_mixed_w64.log` | Not a usable upper reference as run; a low-lr variant is deferred (not required by the pre-registered comparison). | | |
| 11 | +| 2026-10-30 | v2 tiny backbone (5.6M, 15k planned steps) overfit the 23k-item train split: validation loss (all tokens) 0.246 at step 6k → 0.282 at step 9k. Run stopped at step 9,300; the step-5,000 checkpoint (val 0.249) is used as the frozen v2 backbone. | `experiments/logs/p4_tiny_backbone_v2.log`, run `p4_tiny_backbone_v2_seed01_*` (state: stopped) | 42 epochs of a small synthetic set is too many; schedule not re-run (the checkpoint is adequate as a frozen comparator). | | |
| 12 | +| 2026-10-30 | Killed to free nodes for the 5-seed matrix: the degraded joint-finetune reference (M4M64a) and the v1 dense-writes run on the slow m4mn node (superseded by the v2 matrix `p4v2_dense`). | manifests remain in `experiments/runs/` on the nodes (state: running → killed) | — | | |
modified
PROGRESS.md
+7 −1
@@ -6,5 +6,11 @@ | ||
| 6 | 6 | | 2026-10-29 | 1 literature | done | `literature/papers.bib` (77 verified entries), `comparison_matrix.csv`, `novelty_audit.md` | all 12 mandated refs exist incl. RecurTrace 2609.03379 and ELR 2610.12582; C3 (differential zero-init bypass) judged not novel; C2 novel only in its multi-action+propensity form | PASS | preregistration frozen before any test split is read (see protocol/) | |
| 7 | 7 | | 2026-10-29 | 2 backbone + counts | done | `scripts/count_params.py --all-configs --device meta`; `scripts/estimate_flops.py` | backbone counts match brief exactly (99,502,848 / 446,744,064 / 1,149,339,648); workspace 2,631,173 vs brief 3,088,260 (reconciled, D3); ops table reproduces brief within 1.3 % | PASS | — | |
| 8 | 8 | | 2026-10-29 | 3 invariants | done | `python -m pytest tests -q` → 22 passed | 15 mandated invariants + controller target-freedom + dense ablation + 5 cache-equivalence tests | PASS | — | |
| 9 | −| 2026-10-29 | 4 learnability | running | `training.train_editor --config configs/experiments/p4_tiny_backbone.yaml --device mps` (laptop) | — | — | then editor bootstrap on frozen tiny backbone | | |
| 9 | +| 2026-10-29 | 4 learnability (backbone) | done | `p4_tiny_backbone.yaml` → run `p4_tiny_backbone_seed01_20261029232626` (2.1M params, 6k steps, char-level) | exact match (greedy): dev_iid arith 16.8 %, dev_template 13.7 %, dev_depth 3.0 %, compose 2.4 % (first-600-item subsets, arith-heavy) | PASS (weak backbone, usable comparator) | stronger v2 backbone (5.6M, 15k steps) running | | |
| 10 | +| 2026-10-29 | 4 learnability (overfit) | done | `p4_overfit64_b8.yaml` → run `p4_overfit64_b8_seed01_20261029235111` | 64 items, budget 8, frozen backbone: train loss 0.347 → 0.0024 in 200 steps; dev loss rises 0.73 → 1.36 (expected) | PASS (gradient path + capacity) | — | | |
| 11 | +| 2026-10-29 | 4 learnability (generalisation, v1 backbone) | done | zero-U scaled (laptop), U std 0.2 / 0.5 (M4M64c / M2U64), budget-8-only w64/w128 and mixed w128 lr 3e-3 (M2U64 / M4M64c / M4M64a) | after 2000 steps (U std 0.2): dev-iid loss −1.4 % at budgets 1–2, +1.7 % at budget 8; exact match dev-iid 43.8 → 45.8 % (b2), template/depth unchanged, compose 2.3 → 3.0 %; zero-U + scale alone: −0.6 %; lr 3e-3 w128: collapse to inert edits; global-recurrence adapter (same recipe): loss ≥ backbone at every budget, EM 43.8 → 45.8 % (b8) | PASS (weak, learnable; mechanism effect ≈ 1 %) | decisive 5-seed matrix on v2 backbone | | |
| 12 | +| 2026-10-30 | 4 decisive matrix (v2 backbone, 5 seeds × {PRISME, global recurrence, dense}) | running | `scripts/launch_seed_matrix.sh` → 15 runs on M2U64/M4M64c/M4M64a/M5M36/M4BP48/M4BP36 | — | — | aggregate with `scripts/make_tables.py` | | |
| 13 | +| 2026-10-30 | 5 counterfactual labels + oracle audit (tiny) | done | `labels_p4.yaml` → run `labels_p4_seed01_20261030000543` (80,632 train + 20,040 calib labels; 4,778 + 1,308 exhaustive-audit states); `evaluation.label_analysis` | gains: mean 0.004, 42 % positive, 37 % harmful; variance 86–89 % between states vs 11–14 % within state (action identity); 37 % of states have all 24 actions positive, 29 % all negative; one-step oracle 0.0166 nats/state = 0.0098 (perfect stop, random action) + 0.0067 (action selection); random action 0.0004 | PASS (oracle margin exists; action choice is ≈40 % of it) | controller fit | | |
| 14 | +| 2026-10-30 | 6 controller (tiny) | partial | `controller_p4.yaml` → `controller_p4_seed01_20261030000716` (z features): calib sign acc 49 %, captured fraction −3 %; sweep: best (λ=0.1, κ=0.5) loss −0.1 % with 0.01 edits/token; `controller_p4_hent.yaml` (h + backbone entropy): sign acc 59–60 % at 1–2k steps then overfits (early stopping added) | controller with z-only features = chance; h+entropy features predict the sign above chance but capture ≤ 5 % of the oracle opportunity | OPEN (H3 null so far at tiny scale) | rerun with early stopping; more labels | | |
| 15 | +| 2026-10-30 | 7 inference engine + latency (tiny) | done | `tests/integration/test_cache_equivalence.py` (5 tests pass); `evaluation.latency` on the laptop MPS → `results/raw/latency_p4_*_mps.json` | batch-1 decode: 2.8 ms/token (budget 0) → 5.3 (1 edit) → 19.3 ms (8 edits); dense 8 edits 17.3 ms (not slower than sparse at this scale); controller call ≈ +2.9 ms even with 0 edits | PASS (equivalence); dispatch overhead dominates at tiny scale (brief §13.6 confirmed) | re-measure at 103M | | |
| 10 | 16 | | 2026-10-29 | 8 pilot (Stage A) | running | M3U96a: `pretrain_103m.yaml` (run `pretrain103m_seed01_*`); M3U96b: `pretrain_enlarged_102m.yaml` | 14.8k tok/s, 196.6M tokens planned (~3.7 h each) | — | editor/controller stages after completion | |
modified
baselines/global_recurrence.py
+6 −2
@@ -50,7 +50,11 @@ class GlobalRecurrenceModel(nn.Module): | ||
| 50 | 50 | self.block = RecurrentBlock(w, heads or cfg.n_editor_heads, ffn_w or cfg.ffn_w, cfg.norm_eps) |
| 51 | 51 | self.ctx_proj = nn.Linear(2, w, bias=True) |
| 52 | 52 | self.up = nn.Linear(w, cfg.d_model, bias=False) |
| 53 | − nn.init.zeros_(self.up.weight); nn.init.zeros_(self.ctx_proj.weight); nn.init.zeros_(self.ctx_proj.bias) | |
| 53 | + nn.init.zeros_(self.ctx_proj.weight); nn.init.zeros_(self.ctx_proj.bias) | |
| 54 | + if cfg.readout_zero_init: | |
| 55 | + nn.init.zeros_(self.up.weight) | |
| 56 | + else: | |
| 57 | + nn.init.normal_(self.up.weight, std=cfg.readout_init_std) | |
| 54 | 58 | self.controller = nn.Identity() # placeholder so parameter-group helpers work |
| 55 | 59 | |
| 56 | 60 | def forward(self, tokens, actions=None, return_state=False): |
@@ -72,7 +76,7 @@ class GlobalRecurrenceModel(nn.Module): | ||
| 72 | 76 | xn = self.block(x + self.ctx_proj(ctx), mask) |
| 73 | 77 | x = torch.where(act.unsqueeze(-1), xn, x) # positions that stopped keep their state |
| 74 | 78 | n_edits += act.long() |
| 75 | − delta = self.up(x - x0) | |
| 79 | + delta = self.up(x - x0) * self.cfg.readout_scale | |
| 76 | 80 | edited = n_edits > 0 |
| 77 | 81 | delta = torch.where(edited.unsqueeze(-1), delta, torch.zeros_like(delta)) |
| 78 | 82 | return {"logits": self.backbone.head(H + delta), "H": H, "n_edits": n_edits, "delta_out": delta} |
modified
configs/experiments/controller_p4.yaml
+3 −3
@@ -2,10 +2,10 @@ | ||
| 2 | 2 | name: controller_p4 |
| 3 | 3 | seed: 1 |
| 4 | 4 | mode: prisme |
| 5 | −model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512, | |
| 5 | +model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512, readout_zero_init: false, readout_init_std: 0.2, readout_scale: 5.251, | |
| 6 | 6 | max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8} |
| 7 | −checkpoint: CHECKPOINT_DIR | |
| 8 | −labels: {train: LABELS_TRAIN, calib: LABELS_CALIB} | |
| 7 | +checkpoint: experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint | |
| 8 | +labels: {train: experiments/runs/labels_p4_seed01_20261030000543/labels_train.pt, calib: experiments/runs/labels_p4_seed01_20261030000543/labels_calib.pt} | |
| 9 | 9 | train: {steps: 3000, lr: 1.0e-3, weight_decay: 0.01, alpha: 1.0, beta: 0.5, gamma: 0.1, huber_delta: 0.1, log_every: 100, eval_every: 1000, seq_len: 320} |
| 10 | 10 | lambda_grid: [0.0, 0.01, 0.03, 0.1, 0.3] |
| 11 | 11 | kappa_grid: [0.0, 0.5] |
added
configs/experiments/controller_p4_hent.yaml
+50 −0
@@ -0,0 +1,50 @@ | ||
| 1 | +name: controller_p4_hent | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + readout_zero_init: false | |
| 13 | + readout_init_std: 0.2 | |
| 14 | + readout_scale: 5.251 | |
| 15 | + max_seq_len: 320 | |
| 16 | + w: 64 | |
| 17 | + ffn_w: 192 | |
| 18 | + n_slots: 16 | |
| 19 | + n_editor_heads: 2 | |
| 20 | + r_max: 8 | |
| 21 | +checkpoint: experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint | |
| 22 | +labels: | |
| 23 | + train: experiments/runs/labels_p4_seed01_20261030000543/labels_train.pt | |
| 24 | + calib: experiments/runs/labels_p4_seed01_20261030000543/labels_calib.pt | |
| 25 | +train: | |
| 26 | + steps: 3000 | |
| 27 | + lr: 0.001 | |
| 28 | + weight_decay: 0.01 | |
| 29 | + alpha: 1.0 | |
| 30 | + beta: 0.5 | |
| 31 | + gamma: 0.1 | |
| 32 | + huber_delta: 0.1 | |
| 33 | + log_every: 100 | |
| 34 | + eval_every: 1000 | |
| 35 | + seq_len: 320 | |
| 36 | +lambda_grid: | |
| 37 | +- 0.0 | |
| 38 | +- 0.01 | |
| 39 | +- 0.03 | |
| 40 | +- 0.1 | |
| 41 | +- 0.3 | |
| 42 | +kappa_grid: | |
| 43 | +- 0.0 | |
| 44 | +- 0.5 | |
| 45 | +sweep_batches: 10 | |
| 46 | +max_budget: 8 | |
| 47 | +selection_rule: max_loss_reduction_per_ops | |
| 48 | +model_overrides: | |
| 49 | + ctrl_input: h | |
| 50 | + ctrl_use_entropy: true | |
added
configs/experiments/dense_pilot_103m.yaml
+43 −0
@@ -0,0 +1,43 @@ | ||
| 1 | +seed: 1 | |
| 2 | +model: configs/prisme_103m.yaml | |
| 3 | +model_overrides: | |
| 4 | + readout_zero_init: false | |
| 5 | + readout_init_std: 0.2 | |
| 6 | + ctrl_input: h | |
| 7 | + ctrl_use_entropy: true | |
| 8 | +init: | |
| 9 | + backbone_checkpoint: experiments/runs/PRETRAIN103M_RUN/checkpoint | |
| 10 | + freeze_backbone: true | |
| 11 | + auto_readout_scale: true | |
| 12 | +data: | |
| 13 | + kind: tokens | |
| 14 | + train_bin: data/processed/fineweb_edu_300m/train.bin | |
| 15 | + val_bin: data/processed/fineweb_edu_300m/val.bin | |
| 16 | +budgets: | |
| 17 | +- 0 | |
| 18 | +- 1 | |
| 19 | +- 2 | |
| 20 | +- 4 | |
| 21 | +- 8 | |
| 22 | +budget_probs: | |
| 23 | +- 0.2 | |
| 24 | +- 0.2 | |
| 25 | +- 0.2 | |
| 26 | +- 0.2 | |
| 27 | +- 0.2 | |
| 28 | +budget_per: position | |
| 29 | +train: | |
| 30 | + batch_size: 16 | |
| 31 | + seq_len: 512 | |
| 32 | + steps: 3000 | |
| 33 | + lr: 0.001 | |
| 34 | + warmup: 100 | |
| 35 | + weight_decay: 0.1 | |
| 36 | + grad_clip: 1.0 | |
| 37 | + log_every: 20 | |
| 38 | + eval_every: 500 | |
| 39 | + eval_batches: 10 | |
| 40 | + ckpt_every: 1000 | |
| 41 | +final_exact_match: false | |
| 42 | +name: dense_pilot_103m | |
| 43 | +mode: dense | |
added
configs/experiments/editor_pilot_103m.yaml
+43 −0
@@ -0,0 +1,43 @@ | ||
| 1 | +seed: 1 | |
| 2 | +model: configs/prisme_103m.yaml | |
| 3 | +model_overrides: | |
| 4 | + readout_zero_init: false | |
| 5 | + readout_init_std: 0.2 | |
| 6 | + ctrl_input: h | |
| 7 | + ctrl_use_entropy: true | |
| 8 | +init: | |
| 9 | + backbone_checkpoint: experiments/runs/PRETRAIN103M_RUN/checkpoint | |
| 10 | + freeze_backbone: true | |
| 11 | + auto_readout_scale: true | |
| 12 | +data: | |
| 13 | + kind: tokens | |
| 14 | + train_bin: data/processed/fineweb_edu_300m/train.bin | |
| 15 | + val_bin: data/processed/fineweb_edu_300m/val.bin | |
| 16 | +budgets: | |
| 17 | +- 0 | |
| 18 | +- 1 | |
| 19 | +- 2 | |
| 20 | +- 4 | |
| 21 | +- 8 | |
| 22 | +budget_probs: | |
| 23 | +- 0.2 | |
| 24 | +- 0.2 | |
| 25 | +- 0.2 | |
| 26 | +- 0.2 | |
| 27 | +- 0.2 | |
| 28 | +budget_per: position | |
| 29 | +train: | |
| 30 | + batch_size: 16 | |
| 31 | + seq_len: 512 | |
| 32 | + steps: 3000 | |
| 33 | + lr: 0.001 | |
| 34 | + warmup: 100 | |
| 35 | + weight_decay: 0.1 | |
| 36 | + grad_clip: 1.0 | |
| 37 | + log_every: 20 | |
| 38 | + eval_every: 500 | |
| 39 | + eval_batches: 10 | |
| 40 | + ckpt_every: 1000 | |
| 41 | +final_exact_match: false | |
| 42 | +name: editor_pilot_103m | |
| 43 | +mode: prisme | |
added
configs/experiments/eval_p4_dev_gr.yaml
+15 −0
@@ -0,0 +1,15 @@ | ||
| 1 | +# Exact-match evaluation of the global-recurrence baseline (tiny scale) on the dev regimes. | |
| 2 | +out_dir: results/raw | |
| 3 | +data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1} | |
| 4 | +splits: [dev_iid, dev_template, dev_depth, dev_compose] | |
| 5 | +batch_size: 64 | |
| 6 | +seq_len: 320 | |
| 7 | +max_items: 400 | |
| 8 | +systems: | |
| 9 | + gr_mixed_w64_seed1: | |
| 10 | + mode: global_recurrence | |
| 11 | + baseline_kwargs: {ffn_w: 640} | |
| 12 | + model: from_checkpoint | |
| 13 | + checkpoint: experiments/runs/p4_gr_mixed_w64_seed01_20261030000500/checkpoint | |
| 14 | + controller: false | |
| 15 | + policies: [budget0, forced_b1, forced_b2, forced_b8] | |
added
configs/experiments/gr_pilot_103m.yaml
+45 −0
@@ -0,0 +1,45 @@ | ||
| 1 | +seed: 1 | |
| 2 | +model: configs/prisme_103m.yaml | |
| 3 | +model_overrides: | |
| 4 | + readout_zero_init: false | |
| 5 | + readout_init_std: 0.2 | |
| 6 | + ctrl_input: h | |
| 7 | + ctrl_use_entropy: true | |
| 8 | +init: | |
| 9 | + backbone_checkpoint: experiments/runs/PRETRAIN103M_RUN/checkpoint | |
| 10 | + freeze_backbone: true | |
| 11 | + auto_readout_scale: true | |
| 12 | +data: | |
| 13 | + kind: tokens | |
| 14 | + train_bin: data/processed/fineweb_edu_300m/train.bin | |
| 15 | + val_bin: data/processed/fineweb_edu_300m/val.bin | |
| 16 | +budgets: | |
| 17 | +- 0 | |
| 18 | +- 1 | |
| 19 | +- 2 | |
| 20 | +- 4 | |
| 21 | +- 8 | |
| 22 | +budget_probs: | |
| 23 | +- 0.2 | |
| 24 | +- 0.2 | |
| 25 | +- 0.2 | |
| 26 | +- 0.2 | |
| 27 | +- 0.2 | |
| 28 | +budget_per: position | |
| 29 | +train: | |
| 30 | + batch_size: 16 | |
| 31 | + seq_len: 512 | |
| 32 | + steps: 3000 | |
| 33 | + lr: 0.001 | |
| 34 | + warmup: 100 | |
| 35 | + weight_decay: 0.1 | |
| 36 | + grad_clip: 1.0 | |
| 37 | + log_every: 20 | |
| 38 | + eval_every: 500 | |
| 39 | + eval_batches: 10 | |
| 40 | + ckpt_every: 1000 | |
| 41 | +final_exact_match: false | |
| 42 | +name: gr_pilot_103m | |
| 43 | +mode: global_recurrence | |
| 44 | +baseline_kwargs: | |
| 45 | + ffn_w: 2560 | |
modified
configs/experiments/labels_p4.yaml
+2 −2
@@ -2,9 +2,9 @@ | ||
| 2 | 2 | name: labels_p4 |
| 3 | 3 | seed: 1 |
| 4 | 4 | mode: prisme |
| 5 | −model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512, | |
| 5 | +model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512, readout_zero_init: false, readout_init_std: 0.2, readout_scale: 5.251, | |
| 6 | 6 | max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8} |
| 7 | −checkpoint: CHECKPOINT_DIR | |
| 7 | +checkpoint: experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint | |
| 8 | 8 | use_policy: false |
| 9 | 9 | data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: answer} |
| 10 | 10 | collect: {batch_size: 32, seq_len: 320, max_prefix: 3, audit_every: 4} |
added
configs/experiments/latency_p4.yaml
+5 −0
@@ -0,0 +1,5 @@ | ||
| 1 | +# Tiny-scale latency microbenchmark (laptop or node): editor checkpoint v1 (U std 0.2), controller untrained -> forced budgets only matter. | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: from_checkpoint | |
| 5 | +checkpoint: experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint | |
added
configs/experiments/p4_dense_mixed_w64.yaml
+64 −0
@@ -0,0 +1,64 @@ | ||
| 1 | +name: p4_dense_mixed_w64 | |
| 2 | +seed: 1 | |
| 3 | +mode: dense | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 3000 | |
| 49 | + lr: 0.001 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 12 | |
| 56 | + ckpt_every: 1500 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 2 | |
| 60 | +- 8 | |
| 61 | +em_max_items: 400 | |
| 62 | +model_overrides: | |
| 63 | + readout_zero_init: false | |
| 64 | + readout_init_std: 0.2 | |
added
configs/experiments/p4_diag_b8only_w128.yaml
+59 −0
@@ -0,0 +1,59 @@ | ||
| 1 | +name: p4_diag_b8only_w128 | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 8 | |
| 34 | +budget_probs: | |
| 35 | +- 1.0 | |
| 36 | +budget_per: position | |
| 37 | +train: | |
| 38 | + batch_size: 32 | |
| 39 | + seq_len: 320 | |
| 40 | + steps: 3000 | |
| 41 | + lr: 0.001 | |
| 42 | + warmup: 100 | |
| 43 | + weight_decay: 0.1 | |
| 44 | + grad_clip: 1.0 | |
| 45 | + log_every: 50 | |
| 46 | + eval_every: 500 | |
| 47 | + eval_batches: 12 | |
| 48 | + ckpt_every: 1500 | |
| 49 | +em_budgets: | |
| 50 | +- 0 | |
| 51 | +- 2 | |
| 52 | +- 8 | |
| 53 | +em_max_items: 400 | |
| 54 | +model_overrides: | |
| 55 | + readout_zero_init: false | |
| 56 | + readout_init_std: 0.2 | |
| 57 | + w: 128 | |
| 58 | + ffn_w: 384 | |
| 59 | + n_editor_heads: 4 | |
added
configs/experiments/p4_diag_b8only_w64.yaml
+56 −0
@@ -0,0 +1,56 @@ | ||
| 1 | +name: p4_diag_b8only_w64 | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 8 | |
| 34 | +budget_probs: | |
| 35 | +- 1.0 | |
| 36 | +budget_per: position | |
| 37 | +train: | |
| 38 | + batch_size: 32 | |
| 39 | + seq_len: 320 | |
| 40 | + steps: 3000 | |
| 41 | + lr: 0.001 | |
| 42 | + warmup: 100 | |
| 43 | + weight_decay: 0.1 | |
| 44 | + grad_clip: 1.0 | |
| 45 | + log_every: 50 | |
| 46 | + eval_every: 500 | |
| 47 | + eval_batches: 12 | |
| 48 | + ckpt_every: 1500 | |
| 49 | +em_budgets: | |
| 50 | +- 0 | |
| 51 | +- 2 | |
| 52 | +- 8 | |
| 53 | +em_max_items: 400 | |
| 54 | +model_overrides: | |
| 55 | + readout_zero_init: false | |
| 56 | + readout_init_std: 0.2 | |
added
configs/experiments/p4_diag_mixed_w128_lr3e3.yaml
+67 −0
@@ -0,0 +1,67 @@ | ||
| 1 | +name: p4_diag_mixed_w128_lr3e3 | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 3000 | |
| 49 | + lr: 0.003 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 12 | |
| 56 | + ckpt_every: 1500 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 2 | |
| 60 | +- 8 | |
| 61 | +em_max_items: 400 | |
| 62 | +model_overrides: | |
| 63 | + readout_zero_init: false | |
| 64 | + readout_init_std: 0.2 | |
| 65 | + w: 128 | |
| 66 | + ffn_w: 384 | |
| 67 | + n_editor_heads: 4 | |
added
configs/experiments/p4_editor_v1_Ustd02.yaml
+64 −0
@@ -0,0 +1,64 @@ | ||
| 1 | +name: p4_editor_v1_Ustd02 | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 2000 | |
| 49 | + lr: 0.001 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 8 | |
| 56 | + ckpt_every: 1500 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 2 | |
| 60 | +- 8 | |
| 61 | +em_max_items: 400 | |
| 62 | +model_overrides: | |
| 63 | + readout_zero_init: false | |
| 64 | + readout_init_std: 0.2 | |
added
configs/experiments/p4_editor_v1_Ustd05.yaml
+64 −0
@@ -0,0 +1,64 @@ | ||
| 1 | +name: p4_editor_v1_Ustd05 | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 2000 | |
| 49 | + lr: 0.001 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 8 | |
| 56 | + ckpt_every: 1500 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 2 | |
| 60 | +- 8 | |
| 61 | +em_max_items: 400 | |
| 62 | +model_overrides: | |
| 63 | + readout_zero_init: false | |
| 64 | + readout_init_std: 0.5 | |
added
configs/experiments/p4_gr_mixed_w64.yaml
+66 −0
@@ -0,0 +1,66 @@ | ||
| 1 | +name: p4_gr_mixed_w64 | |
| 2 | +seed: 1 | |
| 3 | +mode: global_recurrence | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 3000 | |
| 49 | + lr: 0.001 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 12 | |
| 56 | + ckpt_every: 1500 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 2 | |
| 60 | +- 8 | |
| 61 | +em_max_items: 400 | |
| 62 | +model_overrides: | |
| 63 | + readout_zero_init: false | |
| 64 | + readout_init_std: 0.2 | |
| 65 | +baseline_kwargs: | |
| 66 | + ffn_w: 640 | |
added
configs/experiments/p4_joint_mixed_w64.yaml
+64 −0
@@ -0,0 +1,64 @@ | ||
| 1 | +name: p4_joint_mixed_w64 | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: false | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 3000 | |
| 49 | + lr: 0.0003 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 12 | |
| 56 | + ckpt_every: 1500 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 2 | |
| 60 | +- 8 | |
| 61 | +em_max_items: 400 | |
| 62 | +model_overrides: | |
| 63 | + readout_zero_init: false | |
| 64 | + readout_init_std: 0.2 | |
added
configs/experiments/p4_overfit64_b8.yaml
+55 −0
@@ -0,0 +1,55 @@ | ||
| 1 | +name: p4_overfit64_b8 | |
| 2 | +seed: 1 | |
| 3 | +mode: prisme | |
| 4 | +model: | |
| 5 | + name: tiny_p4 | |
| 6 | + vocab_size: 128 | |
| 7 | + d_model: 192 | |
| 8 | + n_layers: 4 | |
| 9 | + n_heads: 6 | |
| 10 | + n_kv_heads: 2 | |
| 11 | + ffn_width: 512 | |
| 12 | + max_seq_len: 320 | |
| 13 | + w: 64 | |
| 14 | + ffn_w: 192 | |
| 15 | + n_slots: 16 | |
| 16 | + n_editor_heads: 2 | |
| 17 | + r_max: 8 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + max_train_items: 64 | |
| 30 | +budgets: | |
| 31 | +- 8 | |
| 32 | +budget_probs: | |
| 33 | +- 1.0 | |
| 34 | +budget_per: position | |
| 35 | +train: | |
| 36 | + batch_size: 32 | |
| 37 | + seq_len: 320 | |
| 38 | + steps: 600 | |
| 39 | + lr: 0.001 | |
| 40 | + warmup: 100 | |
| 41 | + weight_decay: 0.1 | |
| 42 | + grad_clip: 1.0 | |
| 43 | + log_every: 50 | |
| 44 | + eval_every: 100 | |
| 45 | + eval_batches: 4 | |
| 46 | + ckpt_every: 1500 | |
| 47 | +em_budgets: | |
| 48 | +- 0 | |
| 49 | +- 2 | |
| 50 | +- 8 | |
| 51 | +em_max_items: 400 | |
| 52 | +model_overrides: | |
| 53 | + readout_zero_init: false | |
| 54 | + readout_init_std: 0.5 | |
| 55 | +final_exact_match: false | |
added
configs/experiments/p4v2_dense.yaml
+64 −0
@@ -0,0 +1,64 @@ | ||
| 1 | +seed: 1 | |
| 2 | +model: | |
| 3 | + name: tiny_p4v2 | |
| 4 | + vocab_size: 128 | |
| 5 | + d_model: 256 | |
| 6 | + n_layers: 6 | |
| 7 | + n_heads: 8 | |
| 8 | + n_kv_heads: 4 | |
| 9 | + ffn_width: 768 | |
| 10 | + max_seq_len: 320 | |
| 11 | + w: 64 | |
| 12 | + ffn_w: 192 | |
| 13 | + n_slots: 16 | |
| 14 | + n_editor_heads: 2 | |
| 15 | + r_max: 8 | |
| 16 | + readout_zero_init: false | |
| 17 | + readout_init_std: 0.2 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_v2_seed01_20261029233624/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 3000 | |
| 49 | + lr: 0.001 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 12 | |
| 56 | + ckpt_every: 3000 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 1 | |
| 60 | +- 2 | |
| 61 | +- 8 | |
| 62 | +em_max_items: 1000 | |
| 63 | +name: p4v2_dense | |
| 64 | +mode: dense | |
added
configs/experiments/p4v2_gr.yaml
+66 −0
@@ -0,0 +1,66 @@ | ||
| 1 | +seed: 1 | |
| 2 | +model: | |
| 3 | + name: tiny_p4v2 | |
| 4 | + vocab_size: 128 | |
| 5 | + d_model: 256 | |
| 6 | + n_layers: 6 | |
| 7 | + n_heads: 8 | |
| 8 | + n_kv_heads: 4 | |
| 9 | + ffn_width: 768 | |
| 10 | + max_seq_len: 320 | |
| 11 | + w: 64 | |
| 12 | + ffn_w: 192 | |
| 13 | + n_slots: 16 | |
| 14 | + n_editor_heads: 2 | |
| 15 | + r_max: 8 | |
| 16 | + readout_zero_init: false | |
| 17 | + readout_init_std: 0.2 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_v2_seed01_20261029233624/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 3000 | |
| 49 | + lr: 0.001 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 12 | |
| 56 | + ckpt_every: 3000 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 1 | |
| 60 | +- 2 | |
| 61 | +- 8 | |
| 62 | +em_max_items: 1000 | |
| 63 | +name: p4v2_gr | |
| 64 | +mode: global_recurrence | |
| 65 | +baseline_kwargs: | |
| 66 | + ffn_w: 640 | |
added
configs/experiments/p4v2_prisme.yaml
+64 −0
@@ -0,0 +1,64 @@ | ||
| 1 | +seed: 1 | |
| 2 | +model: | |
| 3 | + name: tiny_p4v2 | |
| 4 | + vocab_size: 128 | |
| 5 | + d_model: 256 | |
| 6 | + n_layers: 6 | |
| 7 | + n_heads: 8 | |
| 8 | + n_kv_heads: 4 | |
| 9 | + ffn_width: 768 | |
| 10 | + max_seq_len: 320 | |
| 11 | + w: 64 | |
| 12 | + ffn_w: 192 | |
| 13 | + n_slots: 16 | |
| 14 | + n_editor_heads: 2 | |
| 15 | + r_max: 8 | |
| 16 | + readout_zero_init: false | |
| 17 | + readout_init_std: 0.2 | |
| 18 | +init: | |
| 19 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_v2_seed01_20261029233624/checkpoint | |
| 20 | + freeze_backbone: true | |
| 21 | + auto_readout_scale: true | |
| 22 | +data: | |
| 23 | + kind: synthetic | |
| 24 | + tokenizer: char | |
| 25 | + splits_dir: data/splits/synthetic_v1 | |
| 26 | + loss_span: answer | |
| 27 | + eval_splits: | |
| 28 | + - dev_iid | |
| 29 | + - dev_template | |
| 30 | + - dev_depth | |
| 31 | + - dev_compose | |
| 32 | +budgets: | |
| 33 | +- 0 | |
| 34 | +- 1 | |
| 35 | +- 2 | |
| 36 | +- 4 | |
| 37 | +- 8 | |
| 38 | +budget_probs: | |
| 39 | +- 0.2 | |
| 40 | +- 0.2 | |
| 41 | +- 0.2 | |
| 42 | +- 0.2 | |
| 43 | +- 0.2 | |
| 44 | +budget_per: position | |
| 45 | +train: | |
| 46 | + batch_size: 32 | |
| 47 | + seq_len: 320 | |
| 48 | + steps: 3000 | |
| 49 | + lr: 0.001 | |
| 50 | + warmup: 100 | |
| 51 | + weight_decay: 0.1 | |
| 52 | + grad_clip: 1.0 | |
| 53 | + log_every: 50 | |
| 54 | + eval_every: 500 | |
| 55 | + eval_batches: 12 | |
| 56 | + ckpt_every: 3000 | |
| 57 | +em_budgets: | |
| 58 | +- 0 | |
| 59 | +- 1 | |
| 60 | +- 2 | |
| 61 | +- 8 | |
| 62 | +em_max_items: 1000 | |
| 63 | +name: p4v2_prisme | |
| 64 | +mode: prisme | |
modified
configs/experiments/routing_p4.yaml
+1 −1
@@ -1,7 +1,7 @@ | ||
| 1 | 1 | # Routing comparison (tiny scale) on dev splits with the LOCKED controller checkpoint. |
| 2 | 2 | seed: 1 |
| 3 | 3 | mode: prisme |
| 4 | −model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512, | |
| 4 | +model: {name: tiny_p4, vocab_size: 128, d_model: 192, n_layers: 4, n_heads: 6, n_kv_heads: 2, ffn_width: 512, readout_zero_init: false, readout_init_std: 0.2, readout_scale: 5.251, | |
| 5 | 5 | max_seq_len: 320, w: 64, ffn_w: 192, n_slots: 16, n_editor_heads: 2, r_max: 8} |
| 6 | 6 | checkpoint: CHECKPOINT_DIR |
| 7 | 7 | data: {kind: synthetic, tokenizer: char, splits_dir: data/splits/synthetic_v1, loss_span: answer, split: dev_iid} |
added
evaluation/label_analysis.py
+91 −0
@@ -0,0 +1,91 @@ | ||
| 1 | +#!/usr/bin/env python | |
| 2 | +"""Descriptive analysis of counterfactual labels: gain distribution, harm rate, variance decomposition of the | |
| 3 | +exhaustive 24-action audit into state and action components, per-action mean gain, oracle vs random vs STOP, | |
| 4 | +and the dependence on the prefix depth. Usage: python -m evaluation.label_analysis <labels_dir> [--out results/analyzed/labels_<name>.json] | |
| 5 | +""" | |
| 6 | +from __future__ import annotations | |
| 7 | + | |
| 8 | +import argparse, json, sys | |
| 9 | +from pathlib import Path | |
| 10 | + | |
| 11 | +import numpy as np | |
| 12 | +import torch | |
| 13 | + | |
| 14 | + | |
| 15 | +def analyse(path: Path) -> dict: | |
| 16 | + d = torch.load(path, weights_only=False) | |
| 17 | + recs = d["records"] | |
| 18 | + g_all, r_pre_all = [], [] | |
| 19 | + aud = [] | |
| 20 | + for r in recs: | |
| 21 | + v = r["valid"] | |
| 22 | + g_all.append(r["gain"][v].numpy().ravel()); r_pre_all.append(np.full(int(v.sum()) * 4, r["r_pre"])) | |
| 23 | + if "all_gains" in r: | |
| 24 | + aud.append((r["all_gains"][v].numpy(), r["base_loss"][v].numpy(), r["r_pre"])) | |
| 25 | + g = np.concatenate(g_all); rp = np.concatenate(r_pre_all) | |
| 26 | + out = {"n_labels": int(g.size), "mean": float(g.mean()), "std": float(g.std()), "median": float(np.median(g)), | |
| 27 | + "frac_pos": float((g > 0).mean()), "frac_neg": float((g < 0).mean()), "frac_abs_gt_0.01": float((np.abs(g) > 0.01).mean()), | |
| 28 | + "frac_abs_gt_0.1": float((np.abs(g) > 0.1).mean()), "q05": float(np.quantile(g, 0.05)), "q95": float(np.quantile(g, 0.95)), | |
| 29 | + "by_prefix_depth": {int(k): {"mean": float(g[rp == k].mean()), "frac_pos": float((g[rp == k] > 0).mean()), "n": int((rp == k).sum())} for k in np.unique(rp)}} | |
| 30 | + if aud: | |
| 31 | + G = np.concatenate([a for a, _, _ in aud]) # [n_states, A] | |
| 32 | + base = np.concatenate([b for _, b, _ in aud]) | |
| 33 | + A = G.shape[1] | |
| 34 | + state_mean = G.mean(1, keepdims=True) | |
| 35 | + total_var = G.var() | |
| 36 | + var_state = state_mean.var() # between-state variance of the mean gain | |
| 37 | + var_action_within = (G - state_mean).var() # within-state (action-specific) variance | |
| 38 | + best = G.max(1); best_clip = np.clip(best, 0, None) | |
| 39 | + rand = G.mean(1) | |
| 40 | + out["audit"] = { | |
| 41 | + "n_states": int(G.shape[0]), "n_actions": int(A), | |
| 42 | + "var_total": float(total_var), "var_between_states": float(var_state), "var_within_state_actions": float(var_action_within), | |
| 43 | + "within_share": float(var_action_within / max(total_var, 1e-12)), | |
| 44 | + "oracle_one_step_mean": float(best_clip.mean()), "random_action_mean": float(rand.mean()), | |
| 45 | + "stop_mean": 0.0, "frac_states_any_positive": float((best > 0).mean()), | |
| 46 | + "frac_states_all_positive": float((G > 0).all(1).mean()), "frac_states_all_negative": float((G < 0).all(1).mean()), | |
| 47 | + "mean_rank_corr_between_actions": None, | |
| 48 | + "per_action_mean": G.mean(0).tolist(), "per_action_frac_best": np.bincount(G.argmax(1), minlength=A).tolist(), | |
| 49 | + "base_loss_mean": float(base.mean()), | |
| 50 | + "oracle_gain_relative_to_base": float(best_clip.mean() / max(base.mean(), 1e-9)), | |
| 51 | + "single_vs_pair_mean": {"single": float(G[:, :16].mean()), "pair": float(G[:, 16:].mean())}, | |
| 52 | + "winners_curse_check_split_half": None, | |
| 53 | + } | |
| 54 | + # winner's-curse check: pick the best action on half the... not possible per-state (one label per action); | |
| 55 | + # instead measure how often the best action of state i is also the best of a *different* state with the | |
| 56 | + # same prefix depth (action consistency) vs chance 1/A. | |
| 57 | + am = G.argmax(1) | |
| 58 | + consist = (am[:-1] == am[1:]).mean() | |
| 59 | + out["audit"]["consecutive_state_best_action_agreement"] = float(consist) | |
| 60 | + # decomposition of the one-step oracle: continue/stop on the state mean vs choosing the best action | |
| 61 | + state_level = np.clip(G.mean(1), 0, None) | |
| 62 | + out["audit"]["oracle_state_level_mean"] = float(state_level.mean()) # STOP unless the average action helps | |
| 63 | + out["audit"]["oracle_action_selection_increment"] = float(best_clip.mean() - state_level.mean()) | |
| 64 | + out["audit"]["random_with_perfect_stop_mean"] = float(state_level.mean()) | |
| 65 | + out["audit"]["chance_agreement"] = 1.0 / A | |
| 66 | + # relation between base loss and oracle gain | |
| 67 | + hi = base > np.median(base) | |
| 68 | + out["audit"]["oracle_gain_high_vs_low_base_loss"] = {"high": float(best_clip[hi].mean()), "low": float(best_clip[~hi].mean())} | |
| 69 | + return out | |
| 70 | + | |
| 71 | + | |
| 72 | +def main(argv=None): | |
| 73 | + ap = argparse.ArgumentParser(description=__doc__) | |
| 74 | + ap.add_argument("labels_dir"); ap.add_argument("--out", default=None) | |
| 75 | + args = ap.parse_args(argv) | |
| 76 | + res = {} | |
| 77 | + for part in ["train", "calib"]: | |
| 78 | + p = Path(args.labels_dir) / f"labels_{part}.pt" | |
| 79 | + if p.exists(): | |
| 80 | + res[part] = analyse(p) | |
| 81 | + print(part, json.dumps({k: v for k, v in res[part].items() if k != "audit"}, indent=None)[:600]) | |
| 82 | + if "audit" in res[part]: | |
| 83 | + a = res[part]["audit"] | |
| 84 | + print(" audit:", {k: (round(v, 5) if isinstance(v, float) else v) for k, v in a.items() if k not in ("per_action_mean", "per_action_frac_best")}) | |
| 85 | + out = Path(args.out or f"results/analyzed/labels_{Path(args.labels_dir).name}.json") | |
| 86 | + out.parent.mkdir(parents=True, exist_ok=True); out.write_text(json.dumps(res, indent=1)) | |
| 87 | + print("wrote", out) | |
| 88 | + | |
| 89 | + | |
| 90 | +if __name__ == "__main__": | |
| 91 | + main() | |
added
evaluation/perplexity.py
+36 −0
@@ -0,0 +1,36 @@ | ||
| 1 | +"""Held-out token-level cross-entropy / perplexity at fixed forced budgets and under the learned controller.""" | |
| 2 | +from __future__ import annotations | |
| 3 | + | |
| 4 | +import math | |
| 5 | + | |
| 6 | +import numpy as np | |
| 7 | +import torch | |
| 8 | + | |
| 9 | +from prisme.losses import token_nll | |
| 10 | +from training.train_editor import sample_forced_actions | |
| 11 | + | |
| 12 | + | |
| 13 | +@torch.no_grad() | |
| 14 | +def perplexity_by_policy(model, cfg, batches_fn, device, cost, budgets=(0, 1, 2, 4, 8), controller=True, seed=0): | |
| 15 | + model.eval() | |
| 16 | + res = {} | |
| 17 | + for b in budgets: | |
| 18 | + rng = np.random.default_rng(seed) | |
| 19 | + tot, n = 0.0, 0 | |
| 20 | + for x, y in batches_fn(): | |
| 21 | + if b == 0 or not cfg.use_workspace: | |
| 22 | + logits = model(x, None)["logits"] | |
| 23 | + else: | |
| 24 | + acts, _ = sample_forced_actions(cfg, x.shape[0], x.shape[1], [b], [1.0], rng, device) | |
| 25 | + logits = model(x, acts)["logits"] | |
| 26 | + nll = token_nll(logits, y); v = y != -100 | |
| 27 | + tot += float(nll[v].sum()); n += int(v.sum()) | |
| 28 | + res[f"forced_b{b}"] = {"loss": tot / n, "ppl": math.exp(tot / n), "n_tokens": n} | |
| 29 | + if controller and cfg.use_workspace: | |
| 30 | + tot, n, ed = 0.0, 0, 0.0 | |
| 31 | + for x, y in batches_fn(): | |
| 32 | + out = model.forward_policy(x, cfg.r_max, cost) | |
| 33 | + nll = token_nll(out["logits"], y); v = y != -100 | |
| 34 | + tot += float(nll[v].sum()); n += int(v.sum()); ed += float(out["n_edits"][v].float().sum()) | |
| 35 | + res["controller"] = {"loss": tot / n, "ppl": math.exp(tot / n), "n_tokens": n, "edits_per_token": ed / n} | |
| 36 | + return res | |
added
evaluation/run.py
+92 −0
@@ -0,0 +1,92 @@ | ||
| 1 | +#!/usr/bin/env python | |
| 2 | +"""Locked evaluation driver. Reads an evaluation YAML listing systems (checkpoints + policies) and splits, refuses | |
| 3 | +to read any split whose name starts with `test` unless protocol/locked_test_manifest.json has "unsealed": true, | |
| 4 | +and writes one JSON per (system, split) under results/raw/ with full provenance. | |
| 5 | + | |
| 6 | +Usage: python -m evaluation.run --config configs/experiments/locked_evaluation.yaml [--device mps] [--only NAME] | |
| 7 | +""" | |
| 8 | +from __future__ import annotations | |
| 9 | + | |
| 10 | +import argparse, json, sys, time | |
| 11 | +from pathlib import Path | |
| 12 | + | |
| 13 | +import numpy as np | |
| 14 | +import torch | |
| 15 | + | |
| 16 | +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) | |
| 17 | +from prisme.flops import action_cost_proxy # noqa: E402 | |
| 18 | +from training.common import TokenBin, load_checkpoint, load_yaml, pick_device, seed_all, git_sha, sha256_file # noqa: E402 | |
| 19 | +from training.train_editor import build_model, sample_forced_actions # noqa: E402 | |
| 20 | +from data.synthetic_dataset import AnyTokenizer, SyntheticBatches, exact_match, load_items # noqa: E402 | |
| 21 | +from evaluation.perplexity import perplexity_by_policy # noqa: E402 | |
| 22 | + | |
| 23 | + | |
| 24 | +def check_lock(split: str): | |
| 25 | + if split.startswith("test"): | |
| 26 | + man = Path("protocol/locked_test_manifest.json") | |
| 27 | + ok = man.exists() and json.loads(man.read_text()).get("unsealed", False) | |
| 28 | + if not ok: | |
| 29 | + raise SystemExit(f"REFUSED: split '{split}' is locked (protocol/locked_test_manifest.json not unsealed)") | |
| 30 | + | |
| 31 | + | |
| 32 | +def main(argv=None) -> int: | |
| 33 | + ap = argparse.ArgumentParser(description=__doc__) | |
| 34 | + ap.add_argument("--config", required=True) | |
| 35 | + ap.add_argument("--device", default="auto") | |
| 36 | + ap.add_argument("--only", default=None) | |
| 37 | + args = ap.parse_args(argv) | |
| 38 | + exp = load_yaml(args.config); device = pick_device(args.device) | |
| 39 | + out_dir = Path(exp.get("out_dir", "results/raw")); out_dir.mkdir(parents=True, exist_ok=True) | |
| 40 | + for sysname, sysspec in exp["systems"].items(): | |
| 41 | + if args.only and sysname != args.only: | |
| 42 | + continue | |
| 43 | + seed_all(sysspec.get("seed", 1)) | |
| 44 | + cfg, model = build_model(sysspec); model = model.to(device).eval() | |
| 45 | + meta = load_checkpoint(sysspec["checkpoint"], model, strict=False) | |
| 46 | + for split in exp["splits"]: | |
| 47 | + check_lock(split) | |
| 48 | + t0 = time.time() | |
| 49 | + data = exp["data"] | |
| 50 | + rec = {"system": sysname, "split": split, "checkpoint": sysspec["checkpoint"], "checkpoint_sha256": meta.get("weights_sha256"), | |
| 51 | + "config_sha256": cfg.sha256(), "code_commit": git_sha(), "device": str(device), "precision": "fp32", "evaluated_at": time.strftime("%Y-%m-%dT%H:%M:%S%z")} | |
| 52 | + if data["kind"] == "tokens": | |
| 53 | + tb = TokenBin(Path(data["dir"]) / f"{split}.bin", exp["seq_len"]) | |
| 54 | + rec["data_sha256"] = sha256_file(Path(data["dir"]) / "manifest.json") | |
| 55 | + cost = torch.tensor(action_cost_proxy(cfg, exp["seq_len"]), device=device) | |
| 56 | + rec["perplexity"] = perplexity_by_policy(model, cfg, lambda: tb.sequential_batches(exp["batch_size"], device, exp.get("n_batches")), | |
| 57 | + device, cost, budgets=exp.get("budgets", [0, 1, 2, 4, 8]), controller=sysspec.get("controller", True)) | |
| 58 | + else: | |
| 59 | + tok = AnyTokenizer(data.get("tokenizer", "char")) | |
| 60 | + items = load_items(Path(data["splits_dir"]) / f"{split}.jsonl") | |
| 61 | + rec["data_sha256"] = sha256_file(Path(data["splits_dir"]) / "manifest.json") | |
| 62 | + sb = SyntheticBatches(items, tok, exp["seq_len"]) | |
| 63 | + cost = torch.tensor(action_cost_proxy(cfg, exp["seq_len"]), device=device) | |
| 64 | + rec["exact_match"] = {} | |
| 65 | + policies = sysspec.get("policies", ["budget0"]) | |
| 66 | + for pol in policies: | |
| 67 | + rng = np.random.default_rng(sysspec.get("seed", 1) + 7) | |
| 68 | + if pol == "budget0": | |
| 69 | + em = exact_match(model, sb, tok, device, 0, batch_size=exp["batch_size"], max_items=exp.get("max_items")) | |
| 70 | + elif pol.startswith("forced"): | |
| 71 | + b = int(pol.split("_b")[1]) | |
| 72 | + em = exact_match(model, sb, tok, device, b, batch_size=exp["batch_size"], max_items=exp.get("max_items"), | |
| 73 | + forced_actions_fn=lambda B_, n, b=b: sample_forced_actions(cfg, B_, n, [b], [1.0], rng, device)[0]) | |
| 74 | + elif pol == "controller": | |
| 75 | + em = exact_match(model, sb, tok, device, cfg.r_max, cost=cost, batch_size=exp["batch_size"], max_items=exp.get("max_items")) | |
| 76 | + elif pol.startswith("controller_lam"): | |
| 77 | + lam = float(pol.split("lam")[1]) | |
| 78 | + em = exact_match(model, sb, tok, device, cfg.r_max, cost=cost, batch_size=exp["batch_size"], max_items=exp.get("max_items"), lambda_cost=lam) | |
| 79 | + else: | |
| 80 | + raise ValueError(pol) | |
| 81 | + recs = em.pop("records") | |
| 82 | + rec["exact_match"][pol] = em | |
| 83 | + (out_dir / f"items_{sysname}_{split}_{pol}.json").write_text(json.dumps(recs)) | |
| 84 | + print(f"{sysname} {split} {pol}: acc {em['accuracy']:.4f} n={em['n']} edits/tok {em['edits_per_token']:.2f}", flush=True) | |
| 85 | + rec["seconds"] = round(time.time() - t0, 1) | |
| 86 | + (out_dir / f"eval_{sysname}_{split}.json").write_text(json.dumps(rec, indent=1)) | |
| 87 | + print("wrote", out_dir / f"eval_{sysname}_{split}.json", flush=True) | |
| 88 | + return 0 | |
| 89 | + | |
| 90 | + | |
| 91 | +if __name__ == "__main__": | |
| 92 | + raise SystemExit(main()) | |
added
experiments/logs/latency_p4_mps.log
+27 −0
@@ -0,0 +1,27 @@ | ||
| 1 | +{"policy": "budget0", "batch": 1, "budget": 0, "median_ms": 2.8017294825986028, "p95_ms": 3.1844482582528144, "mean_ms": 2.759531239765541, "n": 96, "cache_bytes": 245760} | |
| 2 | +{"policy": "forced_single", "batch": 1, "budget": 1, "forced_k": 1, "median_ms": 5.252917006146163, "p95_ms": 6.079749495256692, "mean_ms": 5.123651044414146, "n": 96, "cache_bytes": 245760} | |
| 3 | +{"policy": "forced_pair", "batch": 1, "budget": 1, "forced_k": 2, "median_ms": 5.21864602342248, "p95_ms": 6.357603764627129, "mean_ms": 4.836453521420481, "n": 96, "cache_bytes": 245760} | |
| 4 | +{"policy": "forced_single", "batch": 1, "budget": 2, "forced_k": 1, "median_ms": 7.243604050017893, "p95_ms": 9.018541488330811, "mean_ms": 7.380335543227072, "n": 96, "cache_bytes": 245760} | |
| 5 | +{"policy": "forced_pair", "batch": 1, "budget": 2, "forced_k": 2, "median_ms": 7.464978960342705, "p95_ms": 12.41436498821713, "mean_ms": 8.055094188118042, "n": 96, "cache_bytes": 245760} | |
| 6 | +{"policy": "forced_single", "batch": 1, "budget": 4, "forced_k": 1, "median_ms": 11.287728964816779, "p95_ms": 12.40296903415583, "mean_ms": 10.915309430856723, "n": 96, "cache_bytes": 245760} | |
| 7 | +{"policy": "forced_pair", "batch": 1, "budget": 4, "forced_k": 2, "median_ms": 11.351562512572855, "p95_ms": 12.37730251159519, "mean_ms": 11.322401035916604, "n": 96, "cache_bytes": 245760} | |
| 8 | +{"policy": "forced_single", "batch": 1, "budget": 8, "forced_k": 1, "median_ms": 19.331770483404398, "p95_ms": 21.22909395257011, "mean_ms": 19.320005204159923, "n": 96, "cache_bytes": 245760} | |
| 9 | +{"policy": "forced_pair", "batch": 1, "budget": 8, "forced_k": 2, "median_ms": 18.950854428112507, "p95_ms": 20.537573233013973, "mean_ms": 19.09837671701098, "n": 96, "cache_bytes": 245760} | |
| 10 | +{"policy": "controller", "batch": 1, "budget": 8, "median_ms": 5.7201660820283, "p95_ms": 23.911051946925, "mean_ms": 9.350690118784163, "n": 96, "cache_bytes": 245760} | |
| 11 | +{"policy": "dense_forced", "batch": 1, "budget": 1, "forced_k": 1, "median_ms": 4.747916478663683, "p95_ms": 8.084010769380257, "mean_ms": 4.76325996714877, "n": 96, "cache_bytes": 245760} | |
| 12 | +{"policy": "dense_forced", "batch": 1, "budget": 4, "forced_k": 1, "median_ms": 10.018958011642098, "p95_ms": 11.016781529178843, "mean_ms": 10.038378921308322, "n": 96, "cache_bytes": 245760} | |
| 13 | +{"policy": "dense_forced", "batch": 1, "budget": 8, "forced_k": 1, "median_ms": 17.32618745882064, "p95_ms": 27.340614731656387, "mean_ms": 17.98606334341457, "n": 96, "cache_bytes": 245760} | |
| 14 | +{"policy": "budget0", "batch": 8, "budget": 0, "median_ms": 3.36672953562811, "p95_ms": 3.923937038052827, "mean_ms": 3.3720702607145845, "n": 96, "cache_bytes": 1966080} | |
| 15 | +{"policy": "forced_single", "batch": 8, "budget": 1, "forced_k": 1, "median_ms": 6.341208005324006, "p95_ms": 10.401958308648318, "mean_ms": 6.62630293663824, "n": 96, "cache_bytes": 1966080} | |
| 16 | +{"policy": "forced_pair", "batch": 8, "budget": 1, "forced_k": 2, "median_ms": 6.1652500298805535, "p95_ms": 7.15568705345504, "mean_ms": 6.102194831328234, "n": 96, "cache_bytes": 1966080} | |
| 17 | +{"policy": "forced_single", "batch": 8, "budget": 2, "forced_k": 1, "median_ms": 8.590396028012037, "p95_ms": 14.236905990401283, "mean_ms": 8.708206226098506, "n": 96, "cache_bytes": 1966080} | |
| 18 | +{"policy": "forced_pair", "batch": 8, "budget": 2, "forced_k": 2, "median_ms": 8.381562540307641, "p95_ms": 9.846666041994467, "mean_ms": 8.429156674537808, "n": 96, "cache_bytes": 1966080} | |
| 19 | +{"policy": "forced_single", "batch": 8, "budget": 4, "forced_k": 1, "median_ms": 12.963792018126696, "p95_ms": 14.19885375071317, "mean_ms": 12.95540061983047, "n": 96, "cache_bytes": 1966080} | |
| 20 | +{"policy": "forced_pair", "batch": 8, "budget": 4, "forced_k": 2, "median_ms": 13.345645973458886, "p95_ms": 20.811073016375303, "mean_ms": 13.279389810728995, "n": 96, "cache_bytes": 1966080} | |
| 21 | +{"policy": "forced_single", "batch": 8, "budget": 8, "forced_k": 1, "median_ms": 22.172499971929938, "p95_ms": 26.029041764559224, "mean_ms": 19.927763494100265, "n": 96, "cache_bytes": 1966080} | |
| 22 | +{"policy": "forced_pair", "batch": 8, "budget": 8, "forced_k": 2, "median_ms": 22.192937438376248, "p95_ms": 24.253927753306925, "mean_ms": 21.646448845179595, "n": 96, "cache_bytes": 1966080} | |
| 23 | +{"policy": "controller", "batch": 8, "budget": 8, "median_ms": 24.369208491407335, "p95_ms": 26.040082942927256, "mean_ms": 21.67698220121868, "n": 96, "cache_bytes": 1966080} | |
| 24 | +{"policy": "dense_forced", "batch": 8, "budget": 1, "forced_k": 1, "median_ms": 5.198728991672397, "p95_ms": 6.063812266802415, "mean_ms": 5.117546953745962, "n": 96, "cache_bytes": 1966080} | |
| 25 | +{"policy": "dense_forced", "batch": 8, "budget": 4, "forced_k": 1, "median_ms": 10.355145495850593, "p95_ms": 11.753489437978715, "mean_ms": 10.392837262770627, "n": 96, "cache_bytes": 1966080} | |
| 26 | +{"policy": "dense_forced", "batch": 8, "budget": 8, "forced_k": 1, "median_ms": 17.042521038092673, "p95_ms": 19.3497292639222, "mean_ms": 16.37415804725606, "n": 96, "cache_bytes": 1966080} | |
| 27 | +wrote results/raw/latency_p4_editor_v1_Ustd02_seed01_20261029235050_mps.json | |
modified
experiments/logs/p4_editor_v1_scaled.log
+95 −0
@@ -3,3 +3,98 @@ readout_scale set to backbone residual rms = 5.251 | ||
| 3 | 3 | /Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_editor.py:55: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /Users/runner/work/pytorch/pytorch/torch/csrc/utils/tensor_numpy.cpp:214.) |
| 4 | 4 | return torch.from_numpy(acts).to(device), torch.from_numpy(np.ascontiguousarray(b)).to(device) |
| 5 | 5 | step 0 loss 0.2574 lr 1.00e-05 gn 0.02 (1s) |
| 6 | +step 50 loss 0.2963 lr 5.10e-04 gn 0.02 (35s) | |
| 7 | +step 100 loss 0.2928 lr 1.00e-03 gn 0.03 (68s) | |
| 8 | +step 150 loss 0.2752 lr 9.99e-04 gn 0.04 (101s) | |
| 9 | +step 200 loss 0.2912 lr 9.97e-04 gn 0.03 (134s) | |
| 10 | +step 250 loss 0.2685 lr 9.94e-04 gn 0.02 (168s) | |
| 11 | +step 300 loss 0.2521 lr 9.89e-04 gn 0.02 (200s) | |
| 12 | +step 350 loss 0.3760 lr 9.84e-04 gn 0.10 (234s) | |
| 13 | +step 400 loss 0.2934 lr 9.76e-04 gn 0.03 (266s) | |
| 14 | +step 450 loss 0.2019 lr 9.68e-04 gn 0.06 (299s) | |
| 15 | + val loss by budget: {0: 0.4144, 1: 0.4139, 2: 0.4133, 4: 0.4131, 8: 0.4149} | |
| 16 | +step 500 loss 0.3621 lr 9.58e-04 gn 0.03 (327s) | |
| 17 | +step 550 loss 0.2949 lr 9.48e-04 gn 0.04 (349s) | |
| 18 | +step 600 loss 0.3686 lr 9.36e-04 gn 0.07 (372s) | |
| 19 | +step 650 loss 0.3048 lr 9.22e-04 gn 0.03 (407s) | |
| 20 | +step 700 loss 0.3155 lr 9.08e-04 gn 0.04 (440s) | |
| 21 | +step 750 loss 0.2576 lr 8.93e-04 gn 0.02 (478s) | |
| 22 | +step 800 loss 0.3423 lr 8.77e-04 gn 0.04 (527s) | |
| 23 | +step 850 loss 0.3550 lr 8.59e-04 gn 0.07 (579s) | |
| 24 | +step 900 loss 0.3444 lr 8.41e-04 gn 0.06 (627s) | |
| 25 | +step 950 loss 0.3196 lr 8.22e-04 gn 0.05 (671s) | |
| 26 | + val loss by budget: {0: 0.4144, 1: 0.4129, 2: 0.4124, 4: 0.4123, 8: 0.4146} | |
| 27 | +step 1000 loss 0.3007 lr 8.03e-04 gn 0.06 (751s) | |
| 28 | +step 1050 loss 0.2752 lr 7.82e-04 gn 0.10 (820s) | |
| 29 | +step 1100 loss 0.2404 lr 7.61e-04 gn 0.08 (889s) | |
| 30 | +step 1150 loss 0.3453 lr 7.39e-04 gn 0.05 (948s) | |
| 31 | +step 1200 loss 0.3205 lr 7.17e-04 gn 0.08 (987s) | |
| 32 | +step 1250 loss 0.3766 lr 6.94e-04 gn 0.11 (1018s) | |
| 33 | +step 1300 loss 0.2507 lr 6.70e-04 gn 0.03 (1053s) | |
| 34 | +step 1350 loss 0.2750 lr 6.47e-04 gn 0.09 (1087s) | |
| 35 | +step 1400 loss 0.2685 lr 6.23e-04 gn 0.09 (1124s) | |
| 36 | +step 1450 loss 0.3713 lr 5.99e-04 gn 0.09 (1158s) | |
| 37 | + val loss by budget: {0: 0.4144, 1: 0.412, 2: 0.412, 4: 0.4141, 8: 0.4231} | |
| 38 | +step 1500 loss 0.2672 lr 5.74e-04 gn 0.11 (1197s) | |
| 39 | +step 1550 loss 0.3123 lr 5.50e-04 gn 0.10 (1231s) | |
| 40 | +step 1600 loss 0.2820 lr 5.26e-04 gn 0.09 (1259s) | |
| 41 | +step 1650 loss 0.3476 lr 5.01e-04 gn 0.06 (1289s) | |
| 42 | +step 1700 loss 0.3768 lr 4.77e-04 gn 0.09 (1318s) | |
| 43 | +step 1750 loss 0.2676 lr 4.53e-04 gn 0.05 (1347s) | |
| 44 | +step 1800 loss 0.2471 lr 4.30e-04 gn 0.05 (1382s) | |
| 45 | +step 1850 loss 0.2802 lr 4.06e-04 gn 0.10 (1418s) | |
| 46 | +step 1900 loss 0.2870 lr 3.83e-04 gn 0.07 (1452s) | |
| 47 | +step 1950 loss 0.2158 lr 3.61e-04 gn 0.09 (1488s) | |
| 48 | + val loss by budget: {0: 0.4144, 1: 0.4119, 2: 0.4129, 4: 0.4139, 8: 0.4241} | |
| 49 | +step 2000 loss 0.2418 lr 3.39e-04 gn 0.09 (1524s) | |
| 50 | +step 2050 loss 0.2704 lr 3.18e-04 gn 0.12 (1553s) | |
| 51 | +step 2100 loss 0.2859 lr 2.97e-04 gn 0.15 (1583s) | |
| 52 | +step 2150 loss 0.2684 lr 2.78e-04 gn 0.06 (1611s) | |
| 53 | +step 2200 loss 0.2357 lr 2.59e-04 gn 0.11 (1640s) | |
| 54 | +step 2250 loss 0.2331 lr 2.41e-04 gn 0.06 (1674s) | |
| 55 | +step 2300 loss 0.2750 lr 2.23e-04 gn 0.08 (1709s) | |
| 56 | +step 2350 loss 0.2091 lr 2.07e-04 gn 0.10 (1742s) | |
| 57 | +step 2400 loss 0.1803 lr 1.92e-04 gn 0.08 (1774s) | |
| 58 | +step 2450 loss 0.3071 lr 1.78e-04 gn 0.09 (1804s) | |
| 59 | + val loss by budget: {0: 0.4144, 1: 0.41, 2: 0.41, 4: 0.4111, 8: 0.4188} | |
| 60 | +step 2500 loss 0.3278 lr 1.64e-04 gn 0.13 (1842s) | |
| 61 | +step 2550 loss 0.2849 lr 1.52e-04 gn 0.09 (1877s) | |
| 62 | +step 2600 loss 0.2684 lr 1.42e-04 gn 0.12 (1911s) | |
| 63 | +step 2650 loss 0.2542 lr 1.32e-04 gn 0.16 (1946s) | |
| 64 | +step 2700 loss 0.1905 lr 1.24e-04 gn 0.08 (1978s) | |
| 65 | +step 2750 loss 0.2685 lr 1.16e-04 gn 0.13 (2007s) | |
| 66 | +step 2800 loss 0.2449 lr 1.11e-04 gn 0.10 (2034s) | |
| 67 | +step 2850 loss 0.2741 lr 1.06e-04 gn 0.17 (2059s) | |
| 68 | +step 2900 loss 0.2866 lr 1.03e-04 gn 0.13 (2084s) | |
| 69 | +step 2950 loss 0.2046 lr 1.01e-04 gn 0.07 (2104s) | |
| 70 | +step 2999 loss 0.3118 lr 1.00e-04 gn 0.08 (2120s) | |
| 71 | + val loss by budget: {0: 0.4144, 1: 0.4093, 2: 0.4092, 4: 0.411, 8: 0.4192} | |
| 72 | + exact match dev_iid budget 0: 0.4375 (n=400) by family {'fsm': 0.640625, 'bind': 0.38372093023255816, 'arith': 0.22784810126582278, 'excl': 0.27906976744186046, 'sched': 0.6941176470588235} | |
| 73 | +Traceback (most recent call last): | |
| 74 | + File "<frozen runpy>", line 198, in _run_module_as_main | |
| 75 | + File "<frozen runpy>", line 88, in _run_code | |
| 76 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_editor.py", line 186, in <module> | |
| 77 | + metrics_file=str(run.dir / "metrics.jsonl")) | |
| 78 | + ^^^^^^ | |
| 79 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_editor.py", line 173, in main | |
| 80 | + for name, vs in val_sets.items(): | |
| 81 | + ^^^^^^^^^^^^^^^^^^^^ | |
| 82 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 158, in no_grad | |
| 83 | + return func(*args, **kwargs) | |
| 84 | + ^^^^^^^^^^^^^^^^^^^^^ | |
| 85 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/data/synthetic_dataset.py", line 105, in exact_match | |
| 86 | + out = generate(model, prompt, max_new, max_budget=max_budget, cost=cost, temperature=0.0, eos_id=tok.eot_id, | |
| 87 | + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 88 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 158, in no_grad | |
| 89 | + return func(*args, **kwargs) | |
| 90 | + ^^^^^^^^^^^^^^^^^^^^^ | |
| 91 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/prisme/generation.py", line 115, in generate | |
| 92 | + logits, _, _ = step_logits(model, prompt, cache, max_budget, cost, fa, lambda_cost, kappa_risk) | |
| 93 | + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 94 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 158, in no_grad | |
| 95 | + return func(*args, **kwargs) | |
| 96 | + ^^^^^^^^^^^^^^^^^^^^^ | |
| 97 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/prisme/generation.py", line 43, in step_logits | |
| 98 | + if cfg.ctrl_use_entropy: | |
| 99 | + ^^^^^^^^^^^^^^^^^^^^ | |
| 100 | +AttributeError: 'PrismeConfig' object has no attribute 'ctrl_use_entropy' | |
added
experiments/logs/p4_overfit64_b8.log
+24 −0
@@ -0,0 +1,24 @@ | ||
| 1 | +loaded backbone from experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint missing: 35 | |
| 2 | +readout_scale set to backbone residual rms = 5.310 | |
| 3 | +/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_editor.py:55: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /Users/runner/work/pytorch/pytorch/torch/csrc/utils/tensor_numpy.cpp:214.) | |
| 4 | + return torch.from_numpy(acts).to(device), torch.from_numpy(np.ascontiguousarray(b)).to(device) | |
| 5 | +step 0 loss 0.3472 lr 1.00e-05 gn 1.32 (4s) | |
| 6 | +step 50 loss 0.1013 lr 5.10e-04 gn 0.40 (41s) | |
| 7 | + val loss by budget: {8: 0.7307} | |
| 8 | +step 100 loss 0.0165 lr 1.00e-03 gn 0.16 (90s) | |
| 9 | +step 150 loss 0.0053 lr 9.78e-04 gn 0.02 (137s) | |
| 10 | + val loss by budget: {8: 1.1005} | |
| 11 | +step 200 loss 0.0024 lr 9.14e-04 gn 0.01 (173s) | |
| 12 | +step 250 loss 0.0010 lr 8.15e-04 gn 0.00 (206s) | |
| 13 | + val loss by budget: {8: 1.3595} | |
| 14 | +step 300 loss 0.0008 lr 6.89e-04 gn 0.00 (254s) | |
| 15 | +step 350 loss 0.0004 lr 5.50e-04 gn 0.00 (300s) | |
| 16 | + val loss by budget: {8: 1.4991} | |
| 17 | +step 400 loss 0.0004 lr 4.11e-04 gn 0.00 (347s) | |
| 18 | +step 450 loss 0.0004 lr 2.85e-04 gn 0.00 (391s) | |
| 19 | + val loss by budget: {8: 1.6038} | |
| 20 | +step 500 loss 0.0006 lr 1.86e-04 gn 0.00 (437s) | |
| 21 | +step 550 loss 0.0003 lr 1.22e-04 gn 0.00 (480s) | |
| 22 | +step 599 loss 0.0003 lr 1.00e-04 gn 0.00 (517s) | |
| 23 | + val loss by budget: {8: 1.645} | |
| 24 | +run p4_overfit64_b8_seed01_20261029235111 done | |
modified
experiments/logs/p4_tiny_backbone_v2.log
+80 −0
@@ -25,3 +25,83 @@ step 1900 loss 0.2829 lr 9.74e-04 gn 0.10 (380s) | ||
| 25 | 25 | step 2000 loss 0.2506 lr 9.71e-04 gn 0.12 (400s) |
| 26 | 26 | step 2100 loss 0.2389 lr 9.67e-04 gn 0.09 (426s) |
| 27 | 27 | step 2200 loss 0.2490 lr 9.63e-04 gn 0.15 (455s) |
| 28 | +step 2300 loss 0.2360 lr 9.60e-04 gn 0.16 (484s) | |
| 29 | +step 2400 loss 0.2411 lr 9.55e-04 gn 0.09 (512s) | |
| 30 | +step 2500 loss 0.2048 lr 9.51e-04 gn 0.07 (541s) | |
| 31 | +step 2600 loss 0.2770 lr 9.47e-04 gn 0.11 (570s) | |
| 32 | +step 2700 loss 0.2219 lr 9.42e-04 gn 0.07 (600s) | |
| 33 | +step 2800 loss 0.2500 lr 9.37e-04 gn 0.11 (629s) | |
| 34 | +step 2900 loss 0.2512 lr 9.32e-04 gn 0.08 (658s) | |
| 35 | + val loss by budget: {0: 0.2491} | |
| 36 | +step 3000 loss 0.2259 lr 9.27e-04 gn 0.08 (688s) | |
| 37 | +step 3100 loss 0.2322 lr 9.22e-04 gn 0.09 (716s) | |
| 38 | +step 3200 loss 0.2616 lr 9.16e-04 gn 0.10 (743s) | |
| 39 | +step 3300 loss 0.2512 lr 9.11e-04 gn 0.09 (763s) | |
| 40 | +step 3400 loss 0.2397 lr 9.05e-04 gn 0.12 (782s) | |
| 41 | +step 3500 loss 0.2581 lr 8.99e-04 gn 0.10 (804s) | |
| 42 | +step 3600 loss 0.2327 lr 8.93e-04 gn 0.09 (833s) | |
| 43 | +step 3700 loss 0.2532 lr 8.86e-04 gn 0.09 (862s) | |
| 44 | +step 3800 loss 0.2413 lr 8.80e-04 gn 0.08 (893s) | |
| 45 | +step 3900 loss 0.2293 lr 8.73e-04 gn 0.09 (927s) | |
| 46 | + val loss by budget: {0: 0.2467} | |
| 47 | +step 4000 loss 0.2489 lr 8.66e-04 gn 0.08 (966s) | |
| 48 | +step 4100 loss 0.2392 lr 8.60e-04 gn 0.09 (1009s) | |
| 49 | +step 4200 loss 0.2620 lr 8.53e-04 gn 0.10 (1046s) | |
| 50 | +step 4300 loss 0.2340 lr 8.45e-04 gn 0.10 (1077s) | |
| 51 | +step 4400 loss 0.2159 lr 8.38e-04 gn 0.09 (1104s) | |
| 52 | +step 4500 loss 0.2391 lr 8.31e-04 gn 0.11 (1129s) | |
| 53 | +step 4600 loss 0.2459 lr 8.23e-04 gn 0.10 (1153s) | |
| 54 | +step 4700 loss 0.2321 lr 8.15e-04 gn 0.08 (1177s) | |
| 55 | +step 4800 loss 0.2316 lr 8.07e-04 gn 0.08 (1201s) | |
| 56 | +step 4900 loss 0.2222 lr 8.00e-04 gn 0.09 (1225s) | |
| 57 | + val loss by budget: {0: 0.2458} | |
| 58 | +step 5000 loss 0.2277 lr 7.91e-04 gn 0.08 (1252s) | |
| 59 | +step 5100 loss 0.2377 lr 7.83e-04 gn 0.11 (1277s) | |
| 60 | +step 5200 loss 0.2480 lr 7.75e-04 gn 0.10 (1302s) | |
| 61 | +step 5300 loss 0.2394 lr 7.67e-04 gn 0.09 (1327s) | |
| 62 | +step 5400 loss 0.2269 lr 7.58e-04 gn 0.09 (1351s) | |
| 63 | +step 5500 loss 0.2226 lr 7.50e-04 gn 0.07 (1379s) | |
| 64 | +step 5600 loss 0.2372 lr 7.41e-04 gn 0.10 (1411s) | |
| 65 | +step 5700 loss 0.2575 lr 7.32e-04 gn 0.15 (1438s) | |
| 66 | +step 5800 loss 0.2327 lr 7.23e-04 gn 0.09 (1467s) | |
| 67 | +step 5900 loss 0.2266 lr 7.14e-04 gn 0.08 (1495s) | |
| 68 | + val loss by budget: {0: 0.2471} | |
| 69 | +step 6000 loss 0.2355 lr 7.05e-04 gn 0.09 (1525s) | |
| 70 | +step 6100 loss 0.2458 lr 6.96e-04 gn 0.09 (1556s) | |
| 71 | +step 6200 loss 0.2058 lr 6.87e-04 gn 0.08 (1585s) | |
| 72 | +step 6300 loss 0.2245 lr 6.78e-04 gn 0.10 (1614s) | |
| 73 | +step 6400 loss 0.2180 lr 6.69e-04 gn 0.10 (1642s) | |
| 74 | +step 6500 loss 0.2223 lr 6.59e-04 gn 0.08 (1670s) | |
| 75 | +step 6600 loss 0.2298 lr 6.50e-04 gn 0.09 (1693s) | |
| 76 | +step 6700 loss 0.2283 lr 6.41e-04 gn 0.09 (1719s) | |
| 77 | +step 6800 loss 0.2490 lr 6.31e-04 gn 0.11 (1745s) | |
| 78 | +step 6900 loss 0.2456 lr 6.22e-04 gn 0.11 (1766s) | |
| 79 | + val loss by budget: {0: 0.2523} | |
| 80 | +step 7000 loss 0.2237 lr 6.12e-04 gn 0.10 (1798s) | |
| 81 | +step 7100 loss 0.2181 lr 6.03e-04 gn 0.10 (1827s) | |
| 82 | +step 7200 loss 0.1884 lr 5.93e-04 gn 0.10 (1857s) | |
| 83 | +step 7300 loss 0.2220 lr 5.84e-04 gn 0.09 (1884s) | |
| 84 | +step 7400 loss 0.2312 lr 5.74e-04 gn 0.10 (1914s) | |
| 85 | +step 7500 loss 0.2148 lr 5.64e-04 gn 0.10 (1942s) | |
| 86 | +step 7600 loss 0.2208 lr 5.55e-04 gn 0.09 (1966s) | |
| 87 | +step 7700 loss 0.2331 lr 5.45e-04 gn 0.11 (1992s) | |
| 88 | +step 7800 loss 0.2063 lr 5.36e-04 gn 0.11 (2017s) | |
| 89 | +step 7900 loss 0.2102 lr 5.26e-04 gn 0.10 (2040s) | |
| 90 | + val loss by budget: {0: 0.264} | |
| 91 | +step 8000 loss 0.2142 lr 5.16e-04 gn 0.11 (2066s) | |
| 92 | +step 8100 loss 0.1659 lr 5.07e-04 gn 0.08 (2095s) | |
| 93 | +step 8200 loss 0.2003 lr 4.97e-04 gn 0.13 (2124s) | |
| 94 | +step 8300 loss 0.2093 lr 4.88e-04 gn 0.12 (2152s) | |
| 95 | +step 8400 loss 0.1932 lr 4.78e-04 gn 0.11 (2181s) | |
| 96 | +step 8500 loss 0.2004 lr 4.69e-04 gn 0.12 (2207s) | |
| 97 | +step 8600 loss 0.2052 lr 4.59e-04 gn 0.13 (2233s) | |
| 98 | +step 8700 loss 0.2038 lr 4.50e-04 gn 0.12 (2261s) | |
| 99 | +step 8800 loss 0.1986 lr 4.41e-04 gn 0.13 (2290s) | |
| 100 | +step 8900 loss 0.2058 lr 4.31e-04 gn 0.12 (2321s) | |
| 101 | + val loss by budget: {0: 0.2822} | |
| 102 | +step 9000 loss 0.1914 lr 4.22e-04 gn 0.14 (2351s) | |
| 103 | +step 9100 loss 0.2039 lr 4.13e-04 gn 0.15 (2380s) | |
| 104 | +step 9200 loss 0.1904 lr 4.04e-04 gn 0.15 (2407s) | |
| 105 | +step 9300 loss 0.1848 lr 3.95e-04 gn 0.13 (2432s) | |
| 106 | +step 9400 loss 0.1662 lr 3.86e-04 gn 0.14 (2455s) | |
| 107 | +step 9500 loss 0.1821 lr 3.77e-04 gn 0.16 (2475s) | |
added
experiments/logs/routing_p4_dev_iid.log
+16 −0
@@ -0,0 +1,16 @@ | ||
| 1 | +Traceback (most recent call last): | |
| 2 | + File "<frozen runpy>", line 198, in _run_module_as_main | |
| 3 | + File "<frozen runpy>", line 88, in _run_code | |
| 4 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/evaluation/routing.py", line 123, in <module> | |
| 5 | + raise SystemExit(main()) | |
| 6 | + ^^^^^^ | |
| 7 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/evaluation/routing.py", line 73, in main | |
| 8 | + meta = load_checkpoint(exp["checkpoint"], model, strict=False) | |
| 9 | + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 10 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/common.py", line 174, in load_checkpoint | |
| 11 | + sd = load_file(str(path / "model.safetensors")) | |
| 12 | + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 13 | + File "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/.venv/lib/python3.12/site-packages/safetensors/torch.py", line 359, in load_file | |
| 14 | + with safe_open(filename, framework="pt", device=device, backend=backend) as f: | |
| 15 | + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
| 16 | +FileNotFoundError: No such file or directory: CHECKPOINT_DIR/model.safetensors | |
added
experiments/logs/setup3.log
+5 −0
@@ -0,0 +1,5 @@ | ||
| 1 | +ssh: connect to host m4mm.maclustr.io port 22: Operation timed out | |
| 2 | +rsync(31456): error: unexpected end of file | |
| 3 | +ssh: connect to host m4mm.maclustr.io port 22: Operation timed out | |
| 4 | +rsync(32600): error: unexpected end of file | |
| 5 | +SETUP3DONE | |
added
experiments/logs/setup4.log
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +DONE | |
added
experiments/logs/setup_M4BP36.log
+7 −0
@@ -0,0 +1,7 @@ | ||
| 1 | + "hostname": "simonpierresMBP", | |
| 2 | + "cpu_brand": "Apple M4 Max", | |
| 3 | + "torch": "2.15.0", | |
| 4 | + "mps": true, | |
| 5 | + "gpu_cores": 32, | |
| 6 | + "mps": { | |
| 7 | +setup done on M4BP36 | |
added
experiments/logs/setup_M4BP48.log
+7 −0
@@ -0,0 +1,7 @@ | ||
| 1 | + "hostname": "simonpirresMBP2", | |
| 2 | + "cpu_brand": "Apple M4 Max", | |
| 3 | + "torch": "2.15.0", | |
| 4 | + "mps": true, | |
| 5 | + "gpu_cores": null, | |
| 6 | + "mps": { | |
| 7 | +setup done on M4BP48 | |
added
experiments/logs/setup_M5M36.log
+7 −0
@@ -0,0 +1,7 @@ | ||
| 1 | + "hostname": "M5M36", | |
| 2 | + "cpu_brand": "Apple M5 Max", | |
| 3 | + "torch": "2.15.0", | |
| 4 | + "mps": true, | |
| 5 | + "gpu_cores": 32, | |
| 6 | + "mps": { | |
| 7 | +setup done on M5M36 | |
added
experiments/logs/setup_m4mm.log
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +ssh: connect to host m4mm.maclustr.io port 22: Operation timed out | |
added
experiments/logs/setup_m4mn.log
+7 −0
@@ -0,0 +1,7 @@ | ||
| 1 | + "hostname": "m4mn", | |
| 2 | + "cpu_brand": "Apple M4 Pro", | |
| 3 | + "torch": "2.15.0", | |
| 4 | + "mps": true, | |
| 5 | + "gpu_cores": 16, | |
| 6 | + "mps": { | |
| 7 | +setup done on m4mn | |
added
experiments/runs/controller_p4_hent_seed01_20261030001419/config.yaml
+83 −0
@@ -0,0 +1,83 @@ | ||
| 1 | +experiment: | |
| 2 | + name: controller_p4_hent | |
| 3 | + seed: 1 | |
| 4 | + mode: prisme | |
| 5 | + model: | |
| 6 | + name: tiny_p4 | |
| 7 | + vocab_size: 128 | |
| 8 | + d_model: 192 | |
| 9 | + n_layers: 4 | |
| 10 | + n_heads: 6 | |
| 11 | + n_kv_heads: 2 | |
| 12 | + ffn_width: 512 | |
| 13 | + readout_zero_init: false | |
| 14 | + readout_init_std: 0.2 | |
| 15 | + readout_scale: 5.251 | |
| 16 | + max_seq_len: 320 | |
| 17 | + w: 64 | |
| 18 | + ffn_w: 192 | |
| 19 | + n_slots: 16 | |
| 20 | + n_editor_heads: 2 | |
| 21 | + r_max: 8 | |
| 22 | + checkpoint: experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint | |
| 23 | + labels: | |
| 24 | + train: experiments/runs/labels_p4_seed01_20261030000543/labels_train.pt | |
| 25 | + calib: experiments/runs/labels_p4_seed01_20261030000543/labels_calib.pt | |
| 26 | + train: | |
| 27 | + steps: 3000 | |
| 28 | + lr: 0.001 | |
| 29 | + weight_decay: 0.01 | |
| 30 | + alpha: 1.0 | |
| 31 | + beta: 0.5 | |
| 32 | + gamma: 0.1 | |
| 33 | + huber_delta: 0.1 | |
| 34 | + log_every: 100 | |
| 35 | + eval_every: 1000 | |
| 36 | + seq_len: 320 | |
| 37 | + lambda_grid: | |
| 38 | + - 0.0 | |
| 39 | + - 0.01 | |
| 40 | + - 0.03 | |
| 41 | + - 0.1 | |
| 42 | + - 0.3 | |
| 43 | + kappa_grid: | |
| 44 | + - 0.0 | |
| 45 | + - 0.5 | |
| 46 | + sweep_batches: 10 | |
| 47 | + max_budget: 8 | |
| 48 | + selection_rule: max_loss_reduction_per_ops | |
| 49 | + model_overrides: | |
| 50 | + ctrl_input: h | |
| 51 | + ctrl_use_entropy: true | |
| 52 | +model: | |
| 53 | + name: tiny_p4 | |
| 54 | + vocab_size: 128 | |
| 55 | + d_model: 192 | |
| 56 | + n_layers: 4 | |
| 57 | + n_heads: 6 | |
| 58 | + n_kv_heads: 2 | |
| 59 | + ffn_width: 512 | |
| 60 | + max_seq_len: 320 | |
| 61 | + rope_theta: 10000.0 | |
| 62 | + norm_eps: 1.0e-05 | |
| 63 | + tie_embeddings: true | |
| 64 | + init_std: 0.02 | |
| 65 | + use_workspace: true | |
| 66 | + w: 64 | |
| 67 | + ffn_w: 192 | |
| 68 | + n_slots: 16 | |
| 69 | + n_editor_blocks: 2 | |
| 70 | + n_editor_heads: 2 | |
| 71 | + r_max: 8 | |
| 72 | + eta_max: 0.25 | |
| 73 | + clip_c: 1.0 | |
| 74 | + clip_eps: 1.0e-06 | |
| 75 | + readout_zero_init: false | |
| 76 | + readout_init_std: 0.2 | |
| 77 | + readout_scale: 5.251 | |
| 78 | + controller_hidden: 0 | |
| 79 | + ctrl_input: h | |
| 80 | + ctrl_use_entropy: true | |
| 81 | + lambda_cost: 0.0 | |
| 82 | + kappa_risk: 0.0 | |
| 83 | + dense_writes: false | |
added
experiments/runs/controller_p4_hent_seed01_20261030001419/manifest.json
+18 −0
@@ -0,0 +1,18 @@ | ||
| 1 | +{ | |
| 2 | + "run_id": "controller_p4_hent_seed01_20261030001419", | |
| 3 | + "name": "controller_p4_hent", | |
| 4 | + "seed": 1, | |
| 5 | + "code_commit": "unknown", | |
| 6 | + "config_sha256": "caa5877ed6cf8481cd27e5d1d40defbdca7d0883fcadccb275cfbc67e1831f63", | |
| 7 | + "started_at": "2026-10-30T00:14:19-0400", | |
| 8 | + "host": "M4M64c", | |
| 9 | + "python": "3.12.14", | |
| 10 | + "torch": "2.15.0", | |
| 11 | + "command": "/Users/simon-pierreboucher/wp31/prisme-slm/training/train_controller.py --config configs/experiments/controller_p4_hent.yaml --device mps", | |
| 12 | + "state": "running", | |
| 13 | + "wall_seconds": 0.02, | |
| 14 | + "device": "mps", | |
| 15 | + "checkpoint_parent": "experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint", | |
| 16 | + "labels_train": "experiments/runs/labels_p4_seed01_20261030000543/labels_train.pt", | |
| 17 | + "labels_calib": "experiments/runs/labels_p4_seed01_20261030000543/labels_calib.pt" | |
| 18 | +} | |
| \ No newline at end of file | ||
added
experiments/runs/controller_p4_hent_seed01_20261030001419/metrics.jsonl
+34 −0
@@ -0,0 +1,34 @@ | ||
| 1 | +{"step": 0, "loss": 0.5879008173942566, "gain": 0.048051055520772934, "harm": 1.007463812828064, "nll": 0.36117854714393616, "t": 0.7} | |
| 2 | +{"step": 100, "loss": 0.08753873407840729, "gain": 0.002216772409155965, "harm": 0.6285114288330078, "nll": -2.289337635040283, "t": 6.77} | |
| 3 | +{"step": 200, "loss": -0.01789146661758423, "gain": 0.0013027192326262593, "harm": 0.48355597257614136, "nll": -2.6097216606140137, "t": 12.08} | |
| 4 | +{"step": 300, "loss": 0.14531171321868896, "gain": 0.0017730493564158678, "harm": 0.6629282832145691, "nll": -1.8792548179626465, "t": 17.1} | |
| 5 | +{"step": 400, "loss": -0.006550252437591553, "gain": 0.0009536277502775192, "harm": 0.5190280675888062, "nll": -2.6701788902282715, "t": 22.35} | |
| 6 | +{"step": 500, "loss": 0.04105012118816376, "gain": 0.0015389672480523586, "harm": 0.5556439161300659, "nll": -2.383107900619507, "t": 27.66} | |
| 7 | +{"step": 600, "loss": 0.04611881077289581, "gain": 0.0011181652080267668, "harm": 0.5664311051368713, "nll": -2.3821489810943604, "t": 32.9} | |
| 8 | +{"step": 700, "loss": -0.059603556990623474, "gain": 0.0006694693001918495, "harm": 0.45795416831970215, "nll": -2.89250111579895, "t": 38.42} | |
| 9 | +{"step": 800, "loss": -0.05368393659591675, "gain": 0.000779554306063801, "harm": 0.485246866941452, "nll": -2.9708690643310547, "t": 43.52} | |
| 10 | +{"step": 900, "loss": -0.08059479296207428, "gain": 0.0011186806950718164, "harm": 0.3924700617790222, "nll": -2.779484987258911, "t": 48.62} | |
| 11 | +{"step": 999, "calib_huber": 0.0015227061516034626, "calib_sign_acc": 0.6036926147704591, "calib_harm_brier": 0.16776222031035584, "calib_harm_ece": 0.04239033907651901, "calib_n_labels": 20040, "calib_oracle_opportunity_per_state": 0.016578502246728365, "calib_regret_per_state": 0.015838351180429488, "calib_captured_fraction": 0.04464523123281183, "calib_top1_match": 0.047400611620795105, "calib_n_audit_states": 1308, "t": 55.25} | |
| 12 | +{"step": 1000, "loss": -0.09612229466438293, "gain": 0.000809765886515379, "harm": 0.385701060295105, "nll": -2.8978257179260254, "t": 55.3} | |
| 13 | +{"step": 1100, "loss": -0.0674409419298172, "gain": 0.0006381550920195878, "harm": 0.47498008608818054, "nll": -3.0556914806365967, "t": 60.43} | |
| 14 | +{"step": 1200, "loss": -0.06907917559146881, "gain": 0.00209057261236012, "harm": 0.31956344842910767, "nll": -2.3095147609710693, "t": 65.74} | |
| 15 | +{"step": 1300, "loss": -0.07463987171649933, "gain": 0.0005809835274703801, "harm": 0.4702664017677307, "nll": -3.1035404205322266, "t": 71.04} | |
| 16 | +{"step": 1400, "loss": -0.1494535505771637, "gain": 0.00042246372322551906, "harm": 0.3577497899532318, "nll": -3.287508964538574, "t": 76.3} | |
| 17 | +{"step": 1500, "loss": -0.15540194511413574, "gain": 0.0003082210896536708, "harm": 0.35388779640197754, "nll": -3.326540470123291, "t": 81.73} | |
| 18 | +{"step": 1600, "loss": -0.07748806476593018, "gain": 0.0004072380834259093, "harm": 0.42705395817756653, "nll": -2.9142227172851562, "t": 86.84} | |
| 19 | +{"step": 1700, "loss": -0.15845593810081482, "gain": 0.00047498493222519755, "harm": 0.40316909551620483, "nll": -3.6051547527313232, "t": 92.23} | |
| 20 | +{"step": 1800, "loss": -0.14415833353996277, "gain": 0.0004161176912020892, "harm": 0.43192508816719055, "nll": -3.605369806289673, "t": 97.57} | |
| 21 | +{"step": 1900, "loss": -0.0428461879491806, "gain": 0.0005006617866456509, "harm": 0.4234634041786194, "nll": -2.550785541534424, "t": 102.8} | |
| 22 | +{"step": 1999, "calib_huber": 0.0011455121228913824, "calib_sign_acc": 0.5922155688622754, "calib_harm_brier": 0.16248037144095598, "calib_harm_ece": 0.0631457045674324, "calib_n_labels": 20040, "calib_oracle_opportunity_per_state": 0.016578502246728365, "calib_regret_per_state": 0.01774148319474783, "calib_captured_fraction": -0.07014994061052593, "calib_top1_match": 0.06880733944954129, "calib_n_audit_states": 1308, "t": 109.57} | |
| 23 | +{"step": 2000, "loss": -0.13388130068778992, "gain": 0.00047501263907179236, "harm": 0.3959818482398987, "nll": -3.323472261428833, "t": 109.65} | |
| 24 | +{"step": 2100, "loss": -0.2064000368118286, "gain": 0.00024229599512182176, "harm": 0.36609935760498047, "nll": -3.8969199657440186, "t": 114.79} | |
| 25 | +{"step": 2200, "loss": -0.19953200221061707, "gain": 0.0003932354156859219, "harm": 0.3626585602760315, "nll": -3.8125452995300293, "t": 119.86} | |
| 26 | +{"step": 2300, "loss": -0.16303084790706635, "gain": 0.0020531106274574995, "harm": 0.3887399733066559, "nll": -3.5945394039154053, "t": 124.73} | |
| 27 | +{"step": 2400, "loss": -0.24259337782859802, "gain": 0.000600491592194885, "harm": 0.2846256494522095, "nll": -3.8550667762756348, "t": 129.77} | |
| 28 | +{"step": 2500, "loss": -0.20485253632068634, "gain": 0.00038846320239827037, "harm": 0.364496111869812, "nll": -3.8748903274536133, "t": 135.27} | |
| 29 | +{"step": 2600, "loss": -0.1810571551322937, "gain": 0.0006402016151696444, "harm": 0.263848215341568, "nll": -3.1362144947052, "t": 140.5} | |
| 30 | +{"step": 2700, "loss": -0.025970667600631714, "gain": 0.0004840951878577471, "harm": 0.40832704305648804, "nll": -2.306182861328125, "t": 145.93} | |
| 31 | +{"step": 2800, "loss": -0.21789175271987915, "gain": 0.0002601139713078737, "harm": 0.3145527243614197, "nll": -3.754282236099243, "t": 151.33} | |
| 32 | +{"step": 2900, "loss": -0.17875054478645325, "gain": 0.0003675541374832392, "harm": 0.3699408769607544, "nll": -3.640885353088379, "t": 156.61} | |
| 33 | +{"step": 2999, "loss": -0.15400457382202148, "gain": 0.0003117829910479486, "harm": 0.338265985250473, "nll": -3.2344934940338135, "t": 161.58} | |
| 34 | +{"step": 2999, "calib_huber": 0.0012971028417884233, "calib_sign_acc": 0.49820359281437127, "calib_harm_brier": 0.1648747727780523, "calib_harm_ece": 0.08678887784481049, "calib_n_labels": 20040, "calib_oracle_opportunity_per_state": 0.016578502246728365, "calib_regret_per_state": 0.01721291950354153, "calib_captured_fraction": -0.038267465140788826, "calib_top1_match": 0.08486238532110092, "calib_n_audit_states": 1308, "t": 162.92} | |
added
experiments/runs/labels_p4_seed01_20261030000543/config.yaml
+72 −0
@@ -0,0 +1,72 @@ | ||
| 1 | +experiment: | |
| 2 | + name: labels_p4 | |
| 3 | + seed: 1 | |
| 4 | + mode: prisme | |
| 5 | + model: | |
| 6 | + name: tiny_p4 | |
| 7 | + vocab_size: 128 | |
| 8 | + d_model: 192 | |
| 9 | + n_layers: 4 | |
| 10 | + n_heads: 6 | |
| 11 | + n_kv_heads: 2 | |
| 12 | + ffn_width: 512 | |
| 13 | + readout_zero_init: false | |
| 14 | + readout_init_std: 0.2 | |
| 15 | + readout_scale: 5.251 | |
| 16 | + max_seq_len: 320 | |
| 17 | + w: 64 | |
| 18 | + ffn_w: 192 | |
| 19 | + n_slots: 16 | |
| 20 | + n_editor_heads: 2 | |
| 21 | + r_max: 8 | |
| 22 | + checkpoint: experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint | |
| 23 | + use_policy: false | |
| 24 | + data: | |
| 25 | + kind: synthetic | |
| 26 | + tokenizer: char | |
| 27 | + splits_dir: data/splits/synthetic_v1 | |
| 28 | + loss_span: answer | |
| 29 | + collect: | |
| 30 | + batch_size: 32 | |
| 31 | + seq_len: 320 | |
| 32 | + max_prefix: 3 | |
| 33 | + audit_every: 4 | |
| 34 | + partitions: | |
| 35 | + train: | |
| 36 | + split: train | |
| 37 | + n_batches: 120 | |
| 38 | + stream_seed: 11 | |
| 39 | + calib: | |
| 40 | + split: calib | |
| 41 | + n_batches: 30 | |
| 42 | + stream_seed: 12 | |
| 43 | +model: | |
| 44 | + name: tiny_p4 | |
| 45 | + vocab_size: 128 | |
| 46 | + d_model: 192 | |
| 47 | + n_layers: 4 | |
| 48 | + n_heads: 6 | |
| 49 | + n_kv_heads: 2 | |
| 50 | + ffn_width: 512 | |
| 51 | + max_seq_len: 320 | |
| 52 | + rope_theta: 10000.0 | |
| 53 | + norm_eps: 1.0e-05 | |
| 54 | + tie_embeddings: true | |
| 55 | + init_std: 0.02 | |
| 56 | + use_workspace: true | |
| 57 | + w: 64 | |
| 58 | + ffn_w: 192 | |
| 59 | + n_slots: 16 | |
| 60 | + n_editor_blocks: 2 | |
| 61 | + n_editor_heads: 2 | |
| 62 | + r_max: 8 | |
| 63 | + eta_max: 0.25 | |
| 64 | + clip_c: 1.0 | |
| 65 | + clip_eps: 1.0e-06 | |
| 66 | + readout_zero_init: false | |
| 67 | + readout_init_std: 0.2 | |
| 68 | + readout_scale: 5.251 | |
| 69 | + controller_hidden: 0 | |
| 70 | + lambda_cost: 0.0 | |
| 71 | + kappa_risk: 0.0 | |
| 72 | + dense_writes: false | |
added
experiments/runs/labels_p4_seed01_20261030000543/manifest.json
+40 −0
@@ -0,0 +1,40 @@ | ||
| 1 | +{ | |
| 2 | + "run_id": "labels_p4_seed01_20261030000543", | |
| 3 | + "name": "labels_p4", | |
| 4 | + "seed": 1, | |
| 5 | + "code_commit": "unknown", | |
| 6 | + "config_sha256": "15edb45e43ccdc208c60ce418f7324a53f2877e36912c0f6ad7abf3ddfde3218", | |
| 7 | + "started_at": "2026-10-30T00:05:43-0400", | |
| 8 | + "host": "M4M64c", | |
| 9 | + "python": "3.12.14", | |
| 10 | + "torch": "2.15.0", | |
| 11 | + "command": "/Users/simon-pierreboucher/wp31/prisme-slm/training/collect_interventions.py --config configs/experiments/labels_p4.yaml --device mps", | |
| 12 | + "state": "completed", | |
| 13 | + "wall_seconds": 33.33, | |
| 14 | + "device": "mps", | |
| 15 | + "checkpoint_parent": "experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint", | |
| 16 | + "checkpoint_sha256": "aabf646812efd9f1fdedbcccbf73004a44516484ad036798fdf3a7b80e5cefb0", | |
| 17 | + "finished_at": "2026-10-30T00:06:16-0400", | |
| 18 | + "partitions": { | |
| 19 | + "train": { | |
| 20 | + "file": "experiments/runs/labels_p4_seed01_20261030000543/labels_train.pt", | |
| 21 | + "n_states": 20158, | |
| 22 | + "n_labels": 80632, | |
| 23 | + "mean_gain": 0.0041654990054667, | |
| 24 | + "frac_positive": 0.4210983216762543, | |
| 25 | + "frac_harm": 0.37486356496810913, | |
| 26 | + "seconds": 26.5, | |
| 27 | + "n_audit_batches": 30 | |
| 28 | + }, | |
| 29 | + "calib": { | |
| 30 | + "file": "experiments/runs/labels_p4_seed01_20261030000543/labels_calib.pt", | |
| 31 | + "n_states": 5010, | |
| 32 | + "n_labels": 20040, | |
| 33 | + "mean_gain": 0.0005741285858675838, | |
| 34 | + "frac_positive": 0.4181137681007385, | |
| 35 | + "frac_harm": 0.37619760632514954, | |
| 36 | + "seconds": 6.6, | |
| 37 | + "n_audit_batches": 8 | |
| 38 | + } | |
| 39 | + } | |
| 40 | +} | |
| \ No newline at end of file | ||
added
experiments/runs/labels_p4_seed01_20261030000543/metrics.jsonl
+0 −0
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/checkpoint/checkpoint.json
+37 −0
@@ -0,0 +1,37 @@ | ||
| 1 | +{ | |
| 2 | + "config": { | |
| 3 | + "name": "tiny_p4", | |
| 4 | + "vocab_size": 128, | |
| 5 | + "d_model": 192, | |
| 6 | + "n_layers": 4, | |
| 7 | + "n_heads": 6, | |
| 8 | + "n_kv_heads": 2, | |
| 9 | + "ffn_width": 512, | |
| 10 | + "max_seq_len": 320, | |
| 11 | + "rope_theta": 10000.0, | |
| 12 | + "norm_eps": 1e-05, | |
| 13 | + "tie_embeddings": true, | |
| 14 | + "init_std": 0.02, | |
| 15 | + "use_workspace": true, | |
| 16 | + "w": 64, | |
| 17 | + "ffn_w": 192, | |
| 18 | + "n_slots": 16, | |
| 19 | + "n_editor_blocks": 2, | |
| 20 | + "n_editor_heads": 2, | |
| 21 | + "r_max": 8, | |
| 22 | + "eta_max": 0.25, | |
| 23 | + "clip_c": 1.0, | |
| 24 | + "clip_eps": 1e-06, | |
| 25 | + "readout_zero_init": false, | |
| 26 | + "readout_init_std": 0.2, | |
| 27 | + "readout_scale": 5.251033782958984, | |
| 28 | + "controller_hidden": 0, | |
| 29 | + "lambda_cost": 0.0, | |
| 30 | + "kappa_risk": 0.0, | |
| 31 | + "dense_writes": false | |
| 32 | + }, | |
| 33 | + "saved_at": "2026-10-29T23:59:36-0400", | |
| 34 | + "step": 2000, | |
| 35 | + "run_id": "p4_editor_v1_Ustd02_seed01_20261029235050", | |
| 36 | + "weights_sha256": "aabf646812efd9f1fdedbcccbf73004a44516484ad036798fdf3a7b80e5cefb0" | |
| 37 | +} | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/config.yaml
+107 −0
@@ -0,0 +1,107 @@ | ||
| 1 | +experiment: | |
| 2 | + name: p4_editor_v1_Ustd02 | |
| 3 | + seed: 1 | |
| 4 | + mode: prisme | |
| 5 | + model: | |
| 6 | + name: tiny_p4 | |
| 7 | + vocab_size: 128 | |
| 8 | + d_model: 192 | |
| 9 | + n_layers: 4 | |
| 10 | + n_heads: 6 | |
| 11 | + n_kv_heads: 2 | |
| 12 | + ffn_width: 512 | |
| 13 | + max_seq_len: 320 | |
| 14 | + w: 64 | |
| 15 | + ffn_w: 192 | |
| 16 | + n_slots: 16 | |
| 17 | + n_editor_heads: 2 | |
| 18 | + r_max: 8 | |
| 19 | + init: | |
| 20 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 21 | + freeze_backbone: true | |
| 22 | + auto_readout_scale: true | |
| 23 | + data: | |
| 24 | + kind: synthetic | |
| 25 | + tokenizer: char | |
| 26 | + splits_dir: data/splits/synthetic_v1 | |
| 27 | + loss_span: answer | |
| 28 | + eval_splits: | |
| 29 | + - dev_iid | |
| 30 | + - dev_template | |
| 31 | + - dev_depth | |
| 32 | + - dev_compose | |
| 33 | + budgets: | |
| 34 | + - 0 | |
| 35 | + - 1 | |
| 36 | + - 2 | |
| 37 | + - 4 | |
| 38 | + - 8 | |
| 39 | + budget_probs: | |
| 40 | + - 0.2 | |
| 41 | + - 0.2 | |
| 42 | + - 0.2 | |
| 43 | + - 0.2 | |
| 44 | + - 0.2 | |
| 45 | + budget_per: position | |
| 46 | + train: | |
| 47 | + batch_size: 32 | |
| 48 | + seq_len: 320 | |
| 49 | + steps: 2000 | |
| 50 | + lr: 0.001 | |
| 51 | + warmup: 100 | |
| 52 | + weight_decay: 0.1 | |
| 53 | + grad_clip: 1.0 | |
| 54 | + log_every: 50 | |
| 55 | + eval_every: 500 | |
| 56 | + eval_batches: 8 | |
| 57 | + ckpt_every: 1500 | |
| 58 | + em_budgets: | |
| 59 | + - 0 | |
| 60 | + - 2 | |
| 61 | + - 8 | |
| 62 | + em_max_items: 400 | |
| 63 | + model_overrides: | |
| 64 | + readout_zero_init: false | |
| 65 | + readout_init_std: 0.2 | |
| 66 | +model: | |
| 67 | + name: tiny_p4 | |
| 68 | + vocab_size: 128 | |
| 69 | + d_model: 192 | |
| 70 | + n_layers: 4 | |
| 71 | + n_heads: 6 | |
| 72 | + n_kv_heads: 2 | |
| 73 | + ffn_width: 512 | |
| 74 | + max_seq_len: 320 | |
| 75 | + rope_theta: 10000.0 | |
| 76 | + norm_eps: 1.0e-05 | |
| 77 | + tie_embeddings: true | |
| 78 | + init_std: 0.02 | |
| 79 | + use_workspace: true | |
| 80 | + w: 64 | |
| 81 | + ffn_w: 192 | |
| 82 | + n_slots: 16 | |
| 83 | + n_editor_blocks: 2 | |
| 84 | + n_editor_heads: 2 | |
| 85 | + r_max: 8 | |
| 86 | + eta_max: 0.25 | |
| 87 | + clip_c: 1.0 | |
| 88 | + clip_eps: 1.0e-06 | |
| 89 | + readout_zero_init: false | |
| 90 | + readout_init_std: 0.2 | |
| 91 | + readout_scale: 1.0 | |
| 92 | + controller_hidden: 0 | |
| 93 | + lambda_cost: 0.0 | |
| 94 | + kappa_risk: 0.0 | |
| 95 | + dense_writes: false | |
| 96 | +resolved_train: | |
| 97 | + batch_size: 32 | |
| 98 | + seq_len: 320 | |
| 99 | + steps: 2000 | |
| 100 | + lr: 0.001 | |
| 101 | + warmup: 100 | |
| 102 | + weight_decay: 0.1 | |
| 103 | + grad_clip: 1.0 | |
| 104 | + log_every: 50 | |
| 105 | + eval_every: 500 | |
| 106 | + eval_batches: 8 | |
| 107 | + ckpt_every: 1500 | |
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_compose_b0.json
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +[{"uid": "adb4de54695a9bb1", "family": "compose", "template": 2, "depth": 3, "pred": "-6", "truth": "-4", "correct": false, "edits_per_token": 0.0}, {"uid": "17d806db0f8e44ba", "family": "compose", "template": 2, "depth": 3, "pred": "4", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "b5ffd2ed7506d620", "family": "compose", "template": 2, "depth": 3, "pred": "0", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "4d9b3b7c91729026", "family": "compose", "template": 2, "depth": 3, "pred": "20", "truth": "112", "correct": false, "edits_per_token": 0.0}, {"uid": "de8f6fac20b4e338", "family": "compose", "template": 2, "depth": 3, "pred": "16", "truth": "134", "correct": false, "edits_per_token": 0.0}, {"uid": "d6af255f2972e0c7", "family": "compose", "template": 2, "depth": 3, "pred": "-12", "truth": "-21", "correct": false, "edits_per_token": 0.0}, {"uid": "ce09aab8fc2de9a7", "family": "compose", "template": 2, "depth": 3, "pred": "9", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "1b8a09e522f8d1cd", "family": "compose", "template": 2, "depth": 3, "pred": "0", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "23643e00862cbcbc", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "0d759e78b7643382", "family": "compose", "template": 2, "depth": 3, "pred": "2", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "075a42b41e2eb43f", "family": "compose", "template": 2, "depth": 3, "pred": "12", "truth": "105", "correct": false, "edits_per_token": 0.0}, {"uid": "099a437b780737aa", "family": "compose", "template": 2, "depth": 3, "pred": "6", "truth": "15", "correct": false, "edits_per_token": 0.0}, {"uid": "9940c6a87a6eb6da", "family": "compose", "template": 2, "depth": 3, "pred": "-3", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "f4cb14c0f01c9975", "family": "compose", "template": 2, "depth": 3, "pred": "5", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "647cb5523cd7578b", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "26", "correct": false, "edits_per_token": 0.0}, {"uid": "4a1f85b09c4659f8", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "c525b9ac8039e7a3", "family": "compose", "template": 2, "depth": 3, "pred": "13", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "5035fb98e9aebf76", "family": "compose", "template": 2, "depth": 3, "pred": "0", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "72d9e15b7c0802d9", "family": "compose", "template": 2, "depth": 3, "pred": "6", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "28f5e3fea5a138fb", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "6b3e4232df264f36", "family": "compose", "template": 2, "depth": 3, "pred": "14", "truth": "146", "correct": false, "edits_per_token": 0.0}, {"uid": "8ab11bef7411cbe0", "family": "compose", "template": 2, "depth": 3, "pred": "0", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "96faa66b43a818a1", "family": "compose", "template": 2, "depth": 3, "pred": "6", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "e15f68866737a591", "family": "compose", "template": 2, "depth": 3, "pred": "-8", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "d1bb41bf830b565e", "family": "compose", "template": 2, "depth": 3, "pred": "-2", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "1d98f98efa362f80", "family": "compose", "template": 2, "depth": 3, "pred": "-4", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "665f9491c208ce8e", "family": "compose", "template": 2, "depth": 3, "pred": "14", "truth": "0", "correct": false, "edits_per_token": 0.0}, {"uid": "f54b0fe3fb897d9e", "family": "compose", "template": 2, "depth": 3, "pred": "16", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "44d407369e058248", "family": "compose", "template": 2, "depth": 3, "pred": "23", "truth": "158", "correct": false, "edits_per_token": 0.0}, {"uid": "bf7511876035c1fd", "family": "compose", "template": 2, "depth": 3, "pred": "-1", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "add68ae2a1e04601", "family": "compose", "template": 2, "depth": 3, "pred": "-4", "truth": "-3", "correct": false, "edits_per_token": 0.0}, {"uid": "ae11be33564637ac", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "a1fb5e3c4fe6050c", "family": "compose", "template": 2, "depth": 3, "pred": "10", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "b7a7f826d8f54edf", "family": "compose", "template": 2, "depth": 3, "pred": "0", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "41d7e4fb4c698ad5", "family": "compose", "template": 2, "depth": 3, "pred": "-3", "truth": "-4", "correct": false, "edits_per_token": 0.0}, {"uid": "1cb2706b7f6fbb2b", "family": "compose", "template": 2, "depth": 3, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "3e70547ce95699c3", "family": "compose", "template": 2, "depth": 3, "pred": "10", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "3d11c839f1453ce6", "family": "compose", "template": 2, "depth": 3, "pred": "16", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "d58b04457399d716", "family": "compose", "template": 2, "depth": 3, "pred": "-6", "truth": "-14", "correct": false, "edits_per_token": 0.0}, {"uid": "c894735781f67d70", "family": "compose", "template": 2, "depth": 3, "pred": "-12", "truth": "-17", "correct": false, "edits_per_token": 0.0}, {"uid": "6510140707968d83", "family": "compose", "template": 2, "depth": 3, "pred": "-4", "truth": "0", "correct": false, "edits_per_token": 0.0}, {"uid": "1b41aaffed0ebaad", "family": "compose", "template": 2, "depth": 3, "pred": "-2", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "672b5c8f32a69ff3", "family": "compose", "template": 2, "depth": 3, "pred": "13", "truth": "25", "correct": false, "edits_per_token": 0.0}, {"uid": "237b6b3f75aad340", "family": "compose", "template": 2, "depth": 3, "pred": "3", "truth": "38", "correct": false, "edits_per_token": 0.0}, {"uid": "8b320774cd5bb130", "family": "compose", "template": 2, "depth": 3, "pred": "6", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "a4f4ff84e82ca377", "family": "compose", "template": 0, "depth": 3, "pred": "-2", "truth": "-9", "correct": false, "edits_per_token": 0.0}, {"uid": "4e40f161e5396048", "family": "compose", "template": 0, "depth": 3, "pred": "2", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "8b82f59cbe3434e0", "family": "compose", "template": 0, "depth": 3, "pred": "-3", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "db71d89837843152", "family": "compose", "template": 0, "depth": 3, "pred": "2", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "0197263a81f5749b", "family": "compose", "template": 0, "depth": 3, "pred": "4", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "54ebfde1773c2e3f", "family": "compose", "template": 0, "depth": 3, "pred": "-2", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "0256ae6470dae63c", "family": "compose", "template": 0, "depth": 3, "pred": "12", "truth": "19", "correct": false, "edits_per_token": 0.0}, {"uid": "2df799eeaded8dfe", "family": "compose", "template": 0, "depth": 3, "pred": "-8", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "eba447db451d227b", "family": "compose", "template": 0, "depth": 3, "pred": "5", "truth": "12", "correct": false, "edits_per_token": 0.0}, {"uid": "39c693ae9553e888", "family": "compose", "template": 0, "depth": 3, "pred": "1", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "717c3337957264df", "family": "compose", "template": 0, "depth": 3, "pred": "-6", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "f728f34a28205d99", "family": "compose", "template": 0, "depth": 3, "pred": "6", "truth": "47", "correct": false, "edits_per_token": 0.0}, {"uid": "d2514f0dd399aff4", "family": "compose", "template": 0, "depth": 3, "pred": "-3", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "529a8d6290df2519", "family": "compose", "template": 0, "depth": 3, "pred": "-8", "truth": "-10", "correct": false, "edits_per_token": 0.0}, {"uid": "d9ca9c93444f8be3", "family": "compose", "template": 0, "depth": 3, "pred": "-4", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "e0995d7984dacaee", "family": "compose", "template": 0, "depth": 3, "pred": "10", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "46861a528973f71b", "family": "compose", "template": 0, "depth": 3, "pred": "13", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "86907a1d4cfdd2aa", "family": "compose", "template": 0, "depth": 3, "pred": "2", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "4f5302ec715318f4", "family": "compose", "template": 0, "depth": 3, "pred": "12", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "5f5957f1a7eb0d1a", "family": "compose", "template": 0, "depth": 3, "pred": "-2", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "162aae75dc526041", "family": "compose", "template": 0, "depth": 3, "pred": "-12", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "74f86bfe34852fc7", "family": "compose", "template": 0, "depth": 3, "pred": "2", "truth": "0", "correct": false, "edits_per_token": 0.0}, {"uid": "03bfd6467dbb0da6", "family": "compose", "template": 0, "depth": 3, "pred": "0", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "7d0b4917eaa90a1c", "family": "compose", "template": 0, "depth": 3, "pred": "10", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "1afcc311fcdf51a7", "family": "compose", "template": 0, "depth": 3, "pred": "-2", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "3e29569c45e931be", "family": "compose", "template": 0, "depth": 3, "pred": "16", "truth": "28", "correct": false, "edits_per_token": 0.0}, {"uid": "203b0a6421469c30", "family": "compose", "template": 0, "depth": 3, "pred": "-4", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "22f135fea0674737", "family": "compose", "template": 0, "depth": 3, "pred": "4", "truth": "40", "correct": false, "edits_per_token": 0.0}, {"uid": "f3e9d937a0536532", "family": "compose", "template": 0, "depth": 3, "pred": "12", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "aafa1e44ca944136", "family": "compose", "template": 0, "depth": 3, "pred": "-6", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "1b976dace0d9a61f", "family": "compose", "template": 0, "depth": 3, "pred": "0", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "108e674dc2236456", "family": "compose", "template": 0, "depth": 3, "pred": "6", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "47e15055e7489308", "family": "compose", "template": 0, "depth": 3, "pred": "-6", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "becf1a65023cb394", "family": "compose", "template": 0, "depth": 3, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 0.0}, {"uid": "bb21eaadb8237e64", "family": "compose", "template": 0, "depth": 3, "pred": "0", "truth": "23", "correct": false, "edits_per_token": 0.0}, {"uid": "e8e95e457e0ff61c", "family": "compose", "template": 0, "depth": 3, "pred": "-10", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "7545d7b397cf7f92", "family": "compose", "template": 0, "depth": 3, "pred": "-6", "truth": "-15", "correct": false, "edits_per_token": 0.0}, {"uid": "b1ea2f279468da9c", "family": "compose", "template": 0, "depth": 3, "pred": "3", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "94b0faad6cfc5da0", "family": "compose", "template": 0, "depth": 3, "pred": "16", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "cd59981fea8d6b31", "family": "compose", "template": 0, "depth": 3, "pred": "0", "truth": "-10", "correct": false, "edits_per_token": 0.0}, {"uid": "fab73cdef573e6ee", "family": "compose", "template": 0, "depth": 3, "pred": "8", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "e11d203f1ac8ab57", "family": "compose", "template": 0, "depth": 3, "pred": "18", "truth": "33", "correct": false, "edits_per_token": 0.0}, {"uid": "1b2bc47aab6fb042", "family": "compose", "template": 0, "depth": 3, "pred": "24", "truth": "179", "correct": false, "edits_per_token": 0.0}, {"uid": "06fa3df1bf999d20", "family": "compose", "template": 0, "depth": 3, "pred": "10", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "3d679575c07dffbd", "family": "compose", "template": 0, "depth": 3, "pred": "10", "truth": "22", "correct": false, "edits_per_token": 0.0}, {"uid": "ebab68f0c66d94b2", "family": "compose", "template": 1, "depth": 3, "pred": "14", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "4b655e468b467a1e", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "6333686e41458a19", "family": "compose", "template": 1, "depth": 3, "pred": "-5", "truth": "-5", "correct": true, "edits_per_token": 0.0}, {"uid": "4904a77f2f08c9bf", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "aea9f4148ec22f94", "family": "compose", "template": 1, "depth": 3, "pred": "0", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "ae8a8299a9c7e34e", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "6570c0e205b5d048", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "33", "correct": false, "edits_per_token": 0.0}, {"uid": "cdf5dbc0e8fb28ba", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "47", "correct": false, "edits_per_token": 0.0}, {"uid": "ff6c5e131420c7fa", "family": "compose", "template": 1, "depth": 3, "pred": "-3", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "43b668257f793894", "family": "compose", "template": 1, "depth": 3, "pred": "0", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "f134711ed22583b1", "family": "compose", "template": 1, "depth": 3, "pred": "-1", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "d609372c6cf824c2", "family": "compose", "template": 1, "depth": 3, "pred": "12", "truth": "23", "correct": false, "edits_per_token": 0.0}, {"uid": "58b39aade95d19cc", "family": "compose", "template": 1, "depth": 3, "pred": "4", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "3abaecdd95360608", "family": "compose", "template": 1, "depth": 3, "pred": "4", "truth": "25", "correct": false, "edits_per_token": 0.0}, {"uid": "da42f0492b37e85f", "family": "compose", "template": 1, "depth": 3, "pred": "13", "truth": "19", "correct": false, "edits_per_token": 0.0}, {"uid": "ef200ce61ab718cc", "family": "compose", "template": 1, "depth": 3, "pred": "20", "truth": "117", "correct": false, "edits_per_token": 0.0}, {"uid": "b76a09fc66e8f823", "family": "compose", "template": 1, "depth": 3, "pred": "-4", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "708025be67bbf44c", "family": "compose", "template": 1, "depth": 3, "pred": "13", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "75c6941e53662b69", "family": "compose", "template": 1, "depth": 3, "pred": "11", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "5742f3bbf7ae33ef", "family": "compose", "template": 1, "depth": 3, "pred": "0", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "fef642e44943ca2e", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "15", "correct": false, "edits_per_token": 0.0}, {"uid": "eab7cf6c522a4d74", "family": "compose", "template": 1, "depth": 3, "pred": "-2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "39dc58cd1ddad214", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "42", "correct": false, "edits_per_token": 0.0}, {"uid": "c982c4b4b62b46a2", "family": "compose", "template": 1, "depth": 3, "pred": "0", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "ec9d2b1e581198de", "family": "compose", "template": 1, "depth": 3, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 0.0}, {"uid": "ed5b7032ccb677f6", "family": "compose", "template": 1, "depth": 3, "pred": "20", "truth": "92", "correct": false, "edits_per_token": 0.0}, {"uid": "e85efd8a49131fbb", "family": "compose", "template": 1, "depth": 3, "pred": "1", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "9b1937873462dae3", "family": "compose", "template": 1, "depth": 3, "pred": "8", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "92f8c134b74ab497", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "3bdbc819e2658ce6", "family": "compose", "template": 1, "depth": 3, "pred": "10", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "09ab4572a068e609", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "-10", "correct": false, "edits_per_token": 0.0}, {"uid": "197d7789fd7d2320", "family": "compose", "template": 1, "depth": 3, "pred": "8", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "1b288c05c50ab28a", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "f6c0c00532c7b6ec", "family": "compose", "template": 1, "depth": 3, "pred": "14", "truth": "119", "correct": false, "edits_per_token": 0.0}, {"uid": "1ad9ac7133c048a6", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "4a203c954bc59a55", "family": "compose", "template": 1, "depth": 3, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "e063bbdd65e40fba", "family": "compose", "template": 1, "depth": 3, "pred": "10", "truth": "15", "correct": false, "edits_per_token": 0.0}, {"uid": "8047e5d21252c647", "family": "compose", "template": 1, "depth": 3, "pred": "8", "truth": "24", "correct": false, "edits_per_token": 0.0}, {"uid": "1a308cbea09e1820", "family": "compose", "template": 1, "depth": 3, "pred": "-2", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "ce045884547d667e", "family": "compose", "template": 1, "depth": 3, "pred": "20", "truth": "31", "correct": false, "edits_per_token": 0.0}, {"uid": "f10d7c733d8c484d", "family": "compose", "template": 1, "depth": 3, "pred": "10", "truth": "49", "correct": false, "edits_per_token": 0.0}, {"uid": "14744517e85781d6", "family": "compose", "template": 1, "depth": 3, "pred": "4", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "a9c7c3ba93568f28", "family": "compose", "template": 1, "depth": 3, "pred": "2", "truth": "53", "correct": false, "edits_per_token": 0.0}, {"uid": "3447bfb6e0f058ff", "family": "compose", "template": 1, "depth": 3, "pred": "9", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "c77194dd735c892a", "family": "compose", "template": 2, "depth": 4, "pred": "16", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "f69e0f2af4310da1", "family": "compose", "template": 2, "depth": 4, "pred": "-4", "truth": "-9", "correct": false, "edits_per_token": 0.0}, {"uid": "b4288699c21fb462", "family": "compose", "template": 2, "depth": 4, "pred": "-1", "truth": "-3", "correct": false, "edits_per_token": 0.0}, {"uid": "aa53af096c007782", "family": "compose", "template": 2, "depth": 4, "pred": "18", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "7ebb4ddcb7cd4922", "family": "compose", "template": 2, "depth": 4, "pred": "10", "truth": "-3", "correct": false, "edits_per_token": 0.0}, {"uid": "27eb9dbb08f6be89", "family": "compose", "template": 2, "depth": 4, "pred": "-2", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "60a194c1a80d55c2", "family": "compose", "template": 2, "depth": 4, "pred": "2", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "ac25a665c5b8ba5c", "family": "compose", "template": 2, "depth": 4, "pred": "-1", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "c83e49945f947ee7", "family": "compose", "template": 2, "depth": 4, "pred": "14", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "6fb906a72214b74a", "family": "compose", "template": 2, "depth": 4, "pred": "5", "truth": "81", "correct": false, "edits_per_token": 0.0}, {"uid": "45e407d76733740a", "family": "compose", "template": 2, "depth": 4, "pred": "-8", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "61de5971ee5e66d8", "family": "compose", "template": 2, "depth": 4, "pred": "-16", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "c83db0f0d1e51c97", "family": "compose", "template": 2, "depth": 4, "pred": "0", "truth": "-10", "correct": false, "edits_per_token": 0.0}, {"uid": "dc1cd6108451a386", "family": "compose", "template": 2, "depth": 4, "pred": "2", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "e258998e6e182ce5", "family": "compose", "template": 2, "depth": 4, "pred": "12", "truth": "-3", "correct": false, "edits_per_token": 0.0}, {"uid": "6133cf12f6a73d18", "family": "compose", "template": 2, "depth": 4, "pred": "-6", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "67454e042a0ed655", "family": "compose", "template": 2, "depth": 4, "pred": "-2", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "356b028001a44989", "family": "compose", "template": 2, "depth": 4, "pred": "-1", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "a52f72107bb33d1f", "family": "compose", "template": 2, "depth": 4, "pred": "2", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "04b5402245e39d49", "family": "compose", "template": 2, "depth": 4, "pred": "7", "truth": "19", "correct": false, "edits_per_token": 0.0}, {"uid": "4c0fb74ec5f153e8", "family": "compose", "template": 2, "depth": 4, "pred": "5", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "7f06cb4a08547083", "family": "compose", "template": 2, "depth": 4, "pred": "0", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "e83018fb177d9290", "family": "compose", "template": 2, "depth": 4, "pred": "14", "truth": "-18", "correct": false, "edits_per_token": 0.0}, {"uid": "980118a6d9a0fb70", "family": "compose", "template": 2, "depth": 4, "pred": "0", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "508adf2fa51fa4b7", "family": "compose", "template": 2, "depth": 4, "pred": "8", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "44f7ab9028e2f3a6", "family": "compose", "template": 2, "depth": 4, "pred": "6", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "024cf84c9fa808a8", "family": "compose", "template": 2, "depth": 4, "pred": "10", "truth": "29", "correct": false, "edits_per_token": 0.0}, {"uid": "31a85a27753002e3", "family": "compose", "template": 2, "depth": 4, "pred": "-8", "truth": "-4", "correct": false, "edits_per_token": 0.0}, {"uid": "bddfd858fa52f0fc", "family": "compose", "template": 2, "depth": 4, "pred": "4", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "53ea3a107aa4124d", "family": "compose", "template": 2, "depth": 4, "pred": "16", "truth": "40", "correct": false, "edits_per_token": 0.0}, {"uid": "fb41d24e9b524dbf", "family": "compose", "template": 2, "depth": 4, "pred": "-4", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "daacbc2f9cc02c6e", "family": "compose", "template": 2, "depth": 4, "pred": "5", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "ba48e47b18027d09", "family": "compose", "template": 2, "depth": 4, "pred": "1", "truth": "-9", "correct": false, "edits_per_token": 0.0}, {"uid": "f8dd559e23329aa6", "family": "compose", "template": 2, "depth": 4, "pred": "-2", "truth": "12", "correct": false, "edits_per_token": 0.0}, {"uid": "89e113217d4e8dde", "family": "compose", "template": 2, "depth": 4, "pred": "-6", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "f63237735ba6b2b2", "family": "compose", "template": 2, "depth": 4, "pred": "18", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "c5ed770de008b78f", "family": "compose", "template": 2, "depth": 4, "pred": "2", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "d7b169bfa088fc4b", "family": "compose", "template": 2, "depth": 4, "pred": "-16", "truth": "-15", "correct": false, "edits_per_token": 0.0}, {"uid": "5e099825a8eb3846", "family": "compose", "template": 2, "depth": 4, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "df5aa3e5e3ab531f", "family": "compose", "template": 2, "depth": 4, "pred": "3", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "a7cb8b36fbb34d80", "family": "compose", "template": 2, "depth": 4, "pred": "-2", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "ca89ac23e031b4d9", "family": "compose", "template": 2, "depth": 4, "pred": "18", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "57e8a444a52e9659", "family": "compose", "template": 2, "depth": 4, "pred": "6", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "019d77a6be125d5f", "family": "compose", "template": 2, "depth": 4, "pred": "2", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "4e31b04ed838d8a4", "family": "compose", "template": 0, "depth": 4, "pred": "8", "truth": "179", "correct": false, "edits_per_token": 0.0}, {"uid": "cd4ad97c104f4ccc", "family": "compose", "template": 0, "depth": 4, "pred": "-12", "truth": "-25", "correct": false, "edits_per_token": 0.0}, {"uid": "67a8b958f2783833", "family": "compose", "template": 0, "depth": 4, "pred": "0", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "c08c4460a5101f62", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "393fb7f6b6570266", "family": "compose", "template": 0, "depth": 4, "pred": "-6", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "c52da7a400507ead", "family": "compose", "template": 0, "depth": 4, "pred": "0", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "5cac6a518c45b812", "family": "compose", "template": 0, "depth": 4, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "0bf5c23a7131ed77", "family": "compose", "template": 0, "depth": 4, "pred": "4", "truth": "0", "correct": false, "edits_per_token": 0.0}, {"uid": "25efb341025a1c02", "family": "compose", "template": 0, "depth": 4, "pred": "-3", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "4a79def086294a3a", "family": "compose", "template": 0, "depth": 4, "pred": "6", "truth": "-3", "correct": false, "edits_per_token": 0.0}, {"uid": "faf455eb5168841e", "family": "compose", "template": 0, "depth": 4, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "b746ed0dc37016b2", "family": "compose", "template": 0, "depth": 4, "pred": "3", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "4a36f3514b13957d", "family": "compose", "template": 0, "depth": 4, "pred": "15", "truth": "113", "correct": false, "edits_per_token": 0.0}, {"uid": "17199e564f6e3262", "family": "compose", "template": 0, "depth": 4, "pred": "12", "truth": "102", "correct": false, "edits_per_token": 0.0}, {"uid": "e677a33994695496", "family": "compose", "template": 0, "depth": 4, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "53dac0369fd28ac0", "family": "compose", "template": 0, "depth": 4, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "c2cfa1bf34bc1d6d", "family": "compose", "template": 0, "depth": 4, "pred": "-8", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "8fedc1effa6611a3", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "51", "correct": false, "edits_per_token": 0.0}, {"uid": "0e005022cc77f8c6", "family": "compose", "template": 0, "depth": 4, "pred": "-15", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "26405d91a56f7855", "family": "compose", "template": 0, "depth": 4, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "4f7dea5defe99ded", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "49c5a431f2ef6c26", "family": "compose", "template": 0, "depth": 4, "pred": "12", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "aaf95ea7d25fa328", "family": "compose", "template": 0, "depth": 4, "pred": "7", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "a2efa1cc868910a2", "family": "compose", "template": 0, "depth": 4, "pred": "-6", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "d8797d338a3b92a8", "family": "compose", "template": 0, "depth": 4, "pred": "-16", "truth": "-10", "correct": false, "edits_per_token": 0.0}, {"uid": "819700d90a033274", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "6b79b778c94c3340", "family": "compose", "template": 0, "depth": 4, "pred": "-8", "truth": "-3", "correct": false, "edits_per_token": 0.0}, {"uid": "f46dd9ba49c5f9e5", "family": "compose", "template": 0, "depth": 4, "pred": "-3", "truth": "-10", "correct": false, "edits_per_token": 0.0}, {"uid": "6297fac714f70477", "family": "compose", "template": 0, "depth": 4, "pred": "4", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "5bc9e1d49fcaabba", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "30", "correct": false, "edits_per_token": 0.0}, {"uid": "095e7a1076d7cbe1", "family": "compose", "template": 0, "depth": 4, "pred": "10", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "02e62b8d24c38e10", "family": "compose", "template": 0, "depth": 4, "pred": "4", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "e92f9a90aaa752ab", "family": "compose", "template": 0, "depth": 4, "pred": "-4", "truth": "-12", "correct": false, "edits_per_token": 0.0}, {"uid": "b8f1efdf78952406", "family": "compose", "template": 0, "depth": 4, "pred": "-16", "truth": "-17", "correct": false, "edits_per_token": 0.0}, {"uid": "ac30d19ea27ec58e", "family": "compose", "template": 0, "depth": 4, "pred": "6", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "d754429b5b350068", "family": "compose", "template": 0, "depth": 4, "pred": "0", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "1ea10fa59288e773", "family": "compose", "template": 0, "depth": 4, "pred": "6", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "325032d22f88969d", "family": "compose", "template": 0, "depth": 4, "pred": "10", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "ce5754d69875787b", "family": "compose", "template": 0, "depth": 4, "pred": "-8", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "d221bad35e9a7253", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "3702fc94bbb38ee7", "family": "compose", "template": 0, "depth": 4, "pred": "10", "truth": "23", "correct": false, "edits_per_token": 0.0}, {"uid": "4acd5466e30298d3", "family": "compose", "template": 0, "depth": 4, "pred": "20", "truth": "35", "correct": false, "edits_per_token": 0.0}, {"uid": "c5c331dd1513732d", "family": "compose", "template": 0, "depth": 4, "pred": "10", "truth": "21", "correct": false, "edits_per_token": 0.0}, {"uid": "de6345b12fdb1b98", "family": "compose", "template": 0, "depth": 4, "pred": "20", "truth": "32", "correct": false, "edits_per_token": 0.0}, {"uid": "92ccd36e2de558cf", "family": "compose", "template": 1, "depth": 4, "pred": "10", "truth": "39", "correct": false, "edits_per_token": 0.0}, {"uid": "50faa50b3396ffae", "family": "compose", "template": 1, "depth": 4, "pred": "7", "truth": "12", "correct": false, "edits_per_token": 0.0}, {"uid": "29dee9386795c180", "family": "compose", "template": 1, "depth": 4, "pred": "-6", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "238c0bc8c05a35a2", "family": "compose", "template": 1, "depth": 4, "pred": "-2", "truth": "0", "correct": false, "edits_per_token": 0.0}, {"uid": "38c93e00c30f778c", "family": "compose", "template": 1, "depth": 4, "pred": "-24", "truth": "-17", "correct": false, "edits_per_token": 0.0}, {"uid": "78e560fe43169af8", "family": "compose", "template": 1, "depth": 4, "pred": "20", "truth": "-4", "correct": false, "edits_per_token": 0.0}, {"uid": "b842bbdfb385a2c8", "family": "compose", "template": 1, "depth": 4, "pred": "0", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "ca2e93da50248f99", "family": "compose", "template": 1, "depth": 4, "pred": "2", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "404b290af2697cc1", "family": "compose", "template": 1, "depth": 4, "pred": "14", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "80a7754cfb080d93", "family": "compose", "template": 1, "depth": 4, "pred": "-6", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "63602722310a02d6", "family": "compose", "template": 1, "depth": 4, "pred": "10", "truth": "42", "correct": false, "edits_per_token": 0.0}, {"uid": "34b4950e5d8a0b1e", "family": "compose", "template": 1, "depth": 4, "pred": "0", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "29708655a0f9fa6d", "family": "compose", "template": 1, "depth": 4, "pred": "-3", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "7f8cff76916c511e", "family": "compose", "template": 1, "depth": 4, "pred": "2", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "985ece2df7f3c619", "family": "compose", "template": 1, "depth": 4, "pred": "20", "truth": "146", "correct": false, "edits_per_token": 0.0}, {"uid": "1358a81db4f24e15", "family": "compose", "template": 1, "depth": 4, "pred": "20", "truth": "116", "correct": false, "edits_per_token": 0.0}, {"uid": "48031d8c67e39d8b", "family": "compose", "template": 1, "depth": 4, "pred": "-14", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "a6147eea1031f132", "family": "compose", "template": 1, "depth": 4, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "6582654868e1e4a0", "family": "compose", "template": 1, "depth": 4, "pred": "10", "truth": "115", "correct": false, "edits_per_token": 0.0}, {"uid": "3587b0aaef8cf3e1", "family": "compose", "template": 1, "depth": 4, "pred": "8", "truth": "160", "correct": false, "edits_per_token": 0.0}, {"uid": "2cce12ccba5151de", "family": "compose", "template": 1, "depth": 4, "pred": "6", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "5d28412b23afa91f", "family": "compose", "template": 1, "depth": 4, "pred": "12", "truth": "50", "correct": false, "edits_per_token": 0.0}, {"uid": "b021356c73afb769", "family": "compose", "template": 1, "depth": 4, "pred": "-6", "truth": "-12", "correct": false, "edits_per_token": 0.0}, {"uid": "98b7fa299d84ef2e", "family": "compose", "template": 1, "depth": 4, "pred": "-6", "truth": "-15", "correct": false, "edits_per_token": 0.0}, {"uid": "3d8b4948bd221577", "family": "compose", "template": 1, "depth": 4, "pred": "-4", "truth": "-12", "correct": false, "edits_per_token": 0.0}, {"uid": "923ce5295838ed05", "family": "compose", "template": 1, "depth": 4, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "06fef04b9fe5f342", "family": "compose", "template": 1, "depth": 4, "pred": "-8", "truth": "-16", "correct": false, "edits_per_token": 0.0}, {"uid": "4a350417dc3523cc", "family": "compose", "template": 1, "depth": 4, "pred": "-4", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "fc87f1aa022adcc2", "family": "compose", "template": 1, "depth": 4, "pred": "6", "truth": "-11", "correct": false, "edits_per_token": 0.0}, {"uid": "2e19bd8266852d8b", "family": "compose", "template": 1, "depth": 4, "pred": "6", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "abdac9d2fc00886e", "family": "compose", "template": 1, "depth": 4, "pred": "-4", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "83101cc76f2de188", "family": "compose", "template": 1, "depth": 4, "pred": "-15", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "2d00a93eb0c1b20a", "family": "compose", "template": 1, "depth": 4, "pred": "-3", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "545514d052c79cda", "family": "compose", "template": 1, "depth": 4, "pred": "4", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "8f05f418121b41a7", "family": "compose", "template": 1, "depth": 4, "pred": "13", "truth": "82", "correct": false, "edits_per_token": 0.0}, {"uid": "09dd937e1ef4b6c0", "family": "compose", "template": 1, "depth": 4, "pred": "6", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "c21f46d5ca49a0ce", "family": "compose", "template": 1, "depth": 4, "pred": "20", "truth": "105", "correct": false, "edits_per_token": 0.0}, {"uid": "4686b81018c38653", "family": "compose", "template": 1, "depth": 4, "pred": "10", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "ac04d2d2a5b89df8", "family": "compose", "template": 1, "depth": 4, "pred": "3", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "2916a34b7a3c5497", "family": "compose", "template": 1, "depth": 4, "pred": "-3", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "8c9bbb68ef0ff75b", "family": "compose", "template": 1, "depth": 4, "pred": "-4", "truth": "-16", "correct": false, "edits_per_token": 0.0}, {"uid": "948b8c326fa8c94b", "family": "compose", "template": 1, "depth": 4, "pred": "7", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "9608a0a9a3cf4853", "family": "compose", "template": 1, "depth": 4, "pred": "5", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "52c595795ab95e82", "family": "compose", "template": 1, "depth": 4, "pred": "14", "truth": "22", "correct": false, "edits_per_token": 0.0}, {"uid": "b72fcba0a1ab05d0", "family": "compose", "template": 1, "depth": 4, "pred": "12", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "bee3aabc6bd5223a", "family": "compose", "template": 2, "depth": 5, "pred": "20", "truth": "28", "correct": false, "edits_per_token": 0.0}, {"uid": "8991d7fdb159825d", "family": "compose", "template": 2, "depth": 5, "pred": "6", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "f81d9500784660dd", "family": "compose", "template": 2, "depth": 5, "pred": "4", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "df2ebcbe11e34787", "family": "compose", "template": 2, "depth": 5, "pred": "-2", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "c020e029edb14f57", "family": "compose", "template": 2, "depth": 5, "pred": "10", "truth": "-14", "correct": false, "edits_per_token": 0.0}, {"uid": "61ab1fe0f9bf8174", "family": "compose", "template": 2, "depth": 5, "pred": "-3", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "2c3b4386fc0cd5e2", "family": "compose", "template": 2, "depth": 5, "pred": "6", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "3e451812e17f0f49", "family": "compose", "template": 2, "depth": 5, "pred": "7", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "5dcf4e4893b12b48", "family": "compose", "template": 2, "depth": 5, "pred": "10", "truth": "27", "correct": false, "edits_per_token": 0.0}, {"uid": "ade1e9e1062031d5", "family": "compose", "template": 2, "depth": 5, "pred": "16", "truth": "26", "correct": false, "edits_per_token": 0.0}, {"uid": "d476cf4369ac7e22", "family": "compose", "template": 2, "depth": 5, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "b3304b4efd082e9a", "family": "compose", "template": 2, "depth": 5, "pred": "0", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "e5288726427262e5", "family": "compose", "template": 2, "depth": 5, "pred": "13", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "894fcb5670fbca62", "family": "compose", "template": 2, "depth": 5, "pred": "13", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "51129c69cded6af2", "family": "compose", "template": 2, "depth": 5, "pred": "3", "truth": "19", "correct": false, "edits_per_token": 0.0}, {"uid": "dbcebfe32b261b36", "family": "compose", "template": 2, "depth": 5, "pred": "26", "truth": "406", "correct": false, "edits_per_token": 0.0}, {"uid": "f6d12b0297c1c550", "family": "compose", "template": 2, "depth": 5, "pred": "-12", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "76c08480dce34c43", "family": "compose", "template": 2, "depth": 5, "pred": "5", "truth": "39", "correct": false, "edits_per_token": 0.0}, {"uid": "baea2ba74c196937", "family": "compose", "template": 2, "depth": 5, "pred": "-16", "truth": "-30", "correct": false, "edits_per_token": 0.0}, {"uid": "3f59a13b0dc3dc17", "family": "compose", "template": 2, "depth": 5, "pred": "4", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "4caa849dad301058", "family": "compose", "template": 2, "depth": 5, "pred": "18", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "b348283e0f65231e", "family": "compose", "template": 2, "depth": 5, "pred": "-2", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "560fee240f31263a", "family": "compose", "template": 2, "depth": 5, "pred": "6", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "08dc517782bc8fef", "family": "compose", "template": 2, "depth": 5, "pred": "-8", "truth": "-9", "correct": false, "edits_per_token": 0.0}, {"uid": "28600ff91708289c", "family": "compose", "template": 2, "depth": 5, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "3f5eac859ba6fa37", "family": "compose", "template": 2, "depth": 5, "pred": "-6", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "9264073cc6ce8083", "family": "compose", "template": 2, "depth": 5, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 0.0}, {"uid": "958d4755d009e098", "family": "compose", "template": 2, "depth": 5, "pred": "-6", "truth": "-10", "correct": false, "edits_per_token": 0.0}, {"uid": "c1256114e43d3770", "family": "compose", "template": 2, "depth": 5, "pred": "4", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "99b6163909b9e9fd", "family": "compose", "template": 2, "depth": 5, "pred": "20", "truth": "27", "correct": false, "edits_per_token": 0.0}, {"uid": "dafeef05c5ec8fc4", "family": "compose", "template": 2, "depth": 5, "pred": "13", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "77682fc13ea83713", "family": "compose", "template": 2, "depth": 5, "pred": "5", "truth": "45", "correct": false, "edits_per_token": 0.0}, {"uid": "04de1db4dc636bcc", "family": "compose", "template": 2, "depth": 5, "pred": "14", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "c712c5b393de2310", "family": "compose", "template": 2, "depth": 5, "pred": "6", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "29e13e67d2c97073", "family": "compose", "template": 2, "depth": 5, "pred": "0", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "86397ff87e919200", "family": "compose", "template": 2, "depth": 5, "pred": "-16", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "5837733b6c9e24b1", "family": "compose", "template": 2, "depth": 5, "pred": "-6", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "394bf97876be6993", "family": "compose", "template": 2, "depth": 5, "pred": "14", "truth": "93", "correct": false, "edits_per_token": 0.0}, {"uid": "76b2875e84ee2d00", "family": "compose", "template": 2, "depth": 5, "pred": "16", "truth": "148", "correct": false, "edits_per_token": 0.0}, {"uid": "6a14cc56e1d5eaac", "family": "compose", "template": 2, "depth": 5, "pred": "13", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "d1bcc6c12488902d", "family": "compose", "template": 2, "depth": 5, "pred": "20", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "2cc159a55fb29fd7", "family": "compose", "template": 2, "depth": 5, "pred": "9", "truth": "-3", "correct": false, "edits_per_token": 0.0}, {"uid": "8e8cb310090cdf9f", "family": "compose", "template": 2, "depth": 5, "pred": "5", "truth": "22", "correct": false, "edits_per_token": 0.0}, {"uid": "bd894beab682cc7b", "family": "compose", "template": 2, "depth": 5, "pred": "3", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "20abae29de56cfbc", "family": "compose", "template": 0, "depth": 5, "pred": "20", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "1bffd281778527b5", "family": "compose", "template": 0, "depth": 5, "pred": "-3", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "76a3e1969e0dd187", "family": "compose", "template": 0, "depth": 5, "pred": "-16", "truth": "112", "correct": false, "edits_per_token": 0.0}, {"uid": "135b4dd9880a7711", "family": "compose", "template": 0, "depth": 5, "pred": "10", "truth": "119", "correct": false, "edits_per_token": 0.0}, {"uid": "98fe8dd4b77530ae", "family": "compose", "template": 0, "depth": 5, "pred": "6", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "ab504be8f60f8773", "family": "compose", "template": 0, "depth": 5, "pred": "20", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "ae8a7e76edd9b1ac", "family": "compose", "template": 0, "depth": 5, "pred": "-21", "truth": "-45", "correct": false, "edits_per_token": 0.0}, {"uid": "57d846cf5b774601", "family": "compose", "template": 0, "depth": 5, "pred": "8", "truth": "153", "correct": false, "edits_per_token": 0.0}, {"uid": "9291a187e465d9c8", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "40", "correct": false, "edits_per_token": 0.0}, {"uid": "3027f279111dbcae", "family": "compose", "template": 0, "depth": 5, "pred": "-4", "truth": "-17", "correct": false, "edits_per_token": 0.0}, {"uid": "184ff66abbf56a5c", "family": "compose", "template": 0, "depth": 5, "pred": "0", "truth": "-3", "correct": false, "edits_per_token": 0.0}, {"uid": "eb1a5d81063ecbcb", "family": "compose", "template": 0, "depth": 5, "pred": "24", "truth": "35", "correct": false, "edits_per_token": 0.0}, {"uid": "4746ec994270639a", "family": "compose", "template": 0, "depth": 5, "pred": "16", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "727e5ef3770421a6", "family": "compose", "template": 0, "depth": 5, "pred": "8", "truth": "127", "correct": false, "edits_per_token": 0.0}, {"uid": "20e006d873dd1edb", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "8e32ed5b07934152", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "91fc4c488e9ced3d", "family": "compose", "template": 0, "depth": 5, "pred": "7", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "4984028c93289a35", "family": "compose", "template": 0, "depth": 5, "pred": "-16", "truth": "35", "correct": false, "edits_per_token": 0.0}, {"uid": "55a3205999715b20", "family": "compose", "template": 0, "depth": 5, "pred": "6", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "18779c7d7329f224", "family": "compose", "template": 0, "depth": 5, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "cec5a99eca1756c7", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "a524d85a18a67e49", "family": "compose", "template": 0, "depth": 5, "pred": "-3", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "ffd488cb6bea85ff", "family": "compose", "template": 0, "depth": 5, "pred": "22", "truth": "95", "correct": false, "edits_per_token": 0.0}, {"uid": "a6bd916c42aee6c4", "family": "compose", "template": 0, "depth": 5, "pred": "-3", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "3c867544528e8d53", "family": "compose", "template": 0, "depth": 5, "pred": "4", "truth": "79", "correct": false, "edits_per_token": 0.0}, {"uid": "356c3a7de1144ea9", "family": "compose", "template": 0, "depth": 5, "pred": "4", "truth": "-4", "correct": false, "edits_per_token": 0.0}, {"uid": "4abed743053a2033", "family": "compose", "template": 0, "depth": 5, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "4a2f3f7d91c213de", "family": "compose", "template": 0, "depth": 5, "pred": "8", "truth": "97", "correct": false, "edits_per_token": 0.0}, {"uid": "51aea898105d4c43", "family": "compose", "template": 0, "depth": 5, "pred": "-3", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "f374d4ad191ded59", "family": "compose", "template": 0, "depth": 5, "pred": "6", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "2c485719d8098bbe", "family": "compose", "template": 0, "depth": 5, "pred": "-16", "truth": "-15", "correct": false, "edits_per_token": 0.0}, {"uid": "95343d962b76844d", "family": "compose", "template": 0, "depth": 5, "pred": "8", "truth": "87", "correct": false, "edits_per_token": 0.0}, {"uid": "86e8356113bdbd04", "family": "compose", "template": 0, "depth": 5, "pred": "4", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "5704b25879d5b987", "family": "compose", "template": 0, "depth": 5, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "904694feea1c0838", "family": "compose", "template": 0, "depth": 5, "pred": "10", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "8d1eaafea5c8860c", "family": "compose", "template": 0, "depth": 5, "pred": "3", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "697800c219134d74", "family": "compose", "template": 0, "depth": 5, "pred": "10", "truth": "92", "correct": false, "edits_per_token": 0.0}, {"uid": "c92c4e3ef89e0f14", "family": "compose", "template": 0, "depth": 5, "pred": "-1", "truth": "-14", "correct": false, "edits_per_token": 0.0}, {"uid": "d07423f5eb5e3551", "family": "compose", "template": 0, "depth": 5, "pred": "-5", "truth": "15", "correct": false, "edits_per_token": 0.0}, {"uid": "8477f9ed7d2a0ab7", "family": "compose", "template": 0, "depth": 5, "pred": "-21", "truth": "-26", "correct": false, "edits_per_token": 0.0}, {"uid": "d0ca46e01def43f3", "family": "compose", "template": 0, "depth": 5, "pred": "13", "truth": "76", "correct": false, "edits_per_token": 0.0}, {"uid": "c861a4351ff59cc1", "family": "compose", "template": 0, "depth": 5, "pred": "12", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "15fc685f62ca87b2", "family": "compose", "template": 0, "depth": 5, "pred": "16", "truth": "117", "correct": false, "edits_per_token": 0.0}, {"uid": "6f28dc9656cc50a0", "family": "compose", "template": 0, "depth": 5, "pred": "10", "truth": "28", "correct": false, "edits_per_token": 0.0}, {"uid": "26ab4ae08da814b0", "family": "compose", "template": 0, "depth": 5, "pred": "-6", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "24d6f7c2be67dbc5", "family": "compose", "template": 1, "depth": 5, "pred": "30", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "5c6fa527441afc1a", "family": "compose", "template": 1, "depth": 5, "pred": "-4", "truth": "47", "correct": false, "edits_per_token": 0.0}, {"uid": "45106f9401657f9f", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "113e562bce7bec31", "family": "compose", "template": 1, "depth": 5, "pred": "0", "truth": "-3", "correct": false, "edits_per_token": 0.0}, {"uid": "161d35d2d3f02cd0", "family": "compose", "template": 1, "depth": 5, "pred": "12", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "ce87e2ccc882a8d6", "family": "compose", "template": 1, "depth": 5, "pred": "30", "truth": "305", "correct": false, "edits_per_token": 0.0}, {"uid": "d91c153e794746cd", "family": "compose", "template": 1, "depth": 5, "pred": "-15", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "a0bdd324d33242e8", "family": "compose", "template": 1, "depth": 5, "pred": "4", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "ad0c5f4a911ccd20", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "89", "correct": false, "edits_per_token": 0.0}, {"uid": "0c169fbde9f9f89a", "family": "compose", "template": 1, "depth": 5, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 0.0}, {"uid": "b7b297fa76c6e714", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "19", "correct": false, "edits_per_token": 0.0}, {"uid": "57c40eb561e093fb", "family": "compose", "template": 1, "depth": 5, "pred": "20", "truth": "255", "correct": false, "edits_per_token": 0.0}, {"uid": "2469c98de51b50f4", "family": "compose", "template": 1, "depth": 5, "pred": "2", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "ea142ba35317bc7a", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "485dad2c0d7c8781", "family": "compose", "template": 1, "depth": 5, "pred": "-12", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "c0712e561dd7d544", "family": "compose", "template": 1, "depth": 5, "pred": "-12", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "867fd8edc71c7499", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "d8646639d466e365", "family": "compose", "template": 1, "depth": 5, "pred": "0", "truth": "-9", "correct": false, "edits_per_token": 0.0}, {"uid": "e746a5fb9df5a601", "family": "compose", "template": 1, "depth": 5, "pred": "-16", "truth": "-9", "correct": false, "edits_per_token": 0.0}, {"uid": "e2244c08a7883216", "family": "compose", "template": 1, "depth": 5, "pred": "2", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "335f485c56b8cd07", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "-3", "correct": false, "edits_per_token": 0.0}, {"uid": "e32a910f12bd4154", "family": "compose", "template": 1, "depth": 5, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "67199c1b4257d671", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "-3", "correct": false, "edits_per_token": 0.0}, {"uid": "8243fd85f87830ab", "family": "compose", "template": 1, "depth": 5, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 0.0}, {"uid": "e5bd0d5c5b641231", "family": "compose", "template": 1, "depth": 5, "pred": "30", "truth": "36", "correct": false, "edits_per_token": 0.0}, {"uid": "1eb34c67fb68ad53", "family": "compose", "template": 1, "depth": 5, "pred": "12", "truth": "67", "correct": false, "edits_per_token": 0.0}, {"uid": "c8e8110a6059c20c", "family": "compose", "template": 1, "depth": 5, "pred": "20", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "9e1f059c83a1b997", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "3b7858f2c2c02138", "family": "compose", "template": 1, "depth": 5, "pred": "20", "truth": "35", "correct": false, "edits_per_token": 0.0}, {"uid": "47065d12ee5d095b", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "21", "correct": false, "edits_per_token": 0.0}, {"uid": "c59706dde4249b37", "family": "compose", "template": 1, "depth": 5, "pred": "7", "truth": "0", "correct": false, "edits_per_token": 0.0}, {"uid": "9eb657feba948fa4", "family": "compose", "template": 1, "depth": 5, "pred": "4", "truth": "51", "correct": false, "edits_per_token": 0.0}, {"uid": "1ef4acae76dae63b", "family": "compose", "template": 1, "depth": 5, "pred": "-4", "truth": "12", "correct": false, "edits_per_token": 0.0}, {"uid": "a6fd971352be017e", "family": "compose", "template": 1, "depth": 5, "pred": "-5", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "927f02b06def3150", "family": "compose", "template": 1, "depth": 5, "pred": "12", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "fe187db12adedda5", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "28", "correct": false, "edits_per_token": 0.0}, {"uid": "a645851a332b3ab5", "family": "compose", "template": 1, "depth": 5, "pred": "10", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "cc3539a43f177d50", "family": "compose", "template": 1, "depth": 5, "pred": "6", "truth": "44", "correct": false, "edits_per_token": 0.0}, {"uid": "9a52960b9a529819", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "626758803a7b07cb", "family": "compose", "template": 1, "depth": 5, "pred": "6", "truth": "-4", "correct": false, "edits_per_token": 0.0}, {"uid": "014311836491dc15", "family": "compose", "template": 1, "depth": 5, "pred": "10", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "512fc4cb803c1910", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "867753646795af94", "family": "compose", "template": 1, "depth": 5, "pred": "3", "truth": "15", "correct": false, "edits_per_token": 0.0}, {"uid": "30092adda4a81df9", "family": "compose", "template": 1, "depth": 5, "pred": "12", "truth": "-1", "correct": false, "edits_per_token": 0.0}] | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_compose_b2.json
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +[{"uid": "adb4de54695a9bb1", "family": "compose", "template": 2, "depth": 3, "pred": "-6", "truth": "-4", "correct": false, "edits_per_token": 2.0}, {"uid": "17d806db0f8e44ba", "family": "compose", "template": 2, "depth": 3, "pred": "4", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "b5ffd2ed7506d620", "family": "compose", "template": 2, "depth": 3, "pred": "0", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "4d9b3b7c91729026", "family": "compose", "template": 2, "depth": 3, "pred": "20", "truth": "112", "correct": false, "edits_per_token": 2.0}, {"uid": "de8f6fac20b4e338", "family": "compose", "template": 2, "depth": 3, "pred": "15", "truth": "134", "correct": false, "edits_per_token": 2.0}, {"uid": "d6af255f2972e0c7", "family": "compose", "template": 2, "depth": 3, "pred": "-12", "truth": "-21", "correct": false, "edits_per_token": 2.0}, {"uid": "ce09aab8fc2de9a7", "family": "compose", "template": 2, "depth": 3, "pred": "9", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "1b8a09e522f8d1cd", "family": "compose", "template": 2, "depth": 3, "pred": "0", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "23643e00862cbcbc", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "0d759e78b7643382", "family": "compose", "template": 2, "depth": 3, "pred": "2", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "075a42b41e2eb43f", "family": "compose", "template": 2, "depth": 3, "pred": "12", "truth": "105", "correct": false, "edits_per_token": 2.0}, {"uid": "099a437b780737aa", "family": "compose", "template": 2, "depth": 3, "pred": "5", "truth": "15", "correct": false, "edits_per_token": 2.0}, {"uid": "9940c6a87a6eb6da", "family": "compose", "template": 2, "depth": 3, "pred": "-3", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "f4cb14c0f01c9975", "family": "compose", "template": 2, "depth": 3, "pred": "5", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "647cb5523cd7578b", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "26", "correct": false, "edits_per_token": 2.0}, {"uid": "4a1f85b09c4659f8", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "c525b9ac8039e7a3", "family": "compose", "template": 2, "depth": 3, "pred": "13", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "5035fb98e9aebf76", "family": "compose", "template": 2, "depth": 3, "pred": "0", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "72d9e15b7c0802d9", "family": "compose", "template": 2, "depth": 3, "pred": "7", "truth": "11", "correct": false, "edits_per_token": 2.0}, {"uid": "28f5e3fea5a138fb", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "11", "correct": false, "edits_per_token": 2.0}, {"uid": "6b3e4232df264f36", "family": "compose", "template": 2, "depth": 3, "pred": "14", "truth": "146", "correct": false, "edits_per_token": 2.0}, {"uid": "8ab11bef7411cbe0", "family": "compose", "template": 2, "depth": 3, "pred": "0", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "96faa66b43a818a1", "family": "compose", "template": 2, "depth": 3, "pred": "6", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "e15f68866737a591", "family": "compose", "template": 2, "depth": 3, "pred": "-7", "truth": "-7", "correct": true, "edits_per_token": 2.0}, {"uid": "d1bb41bf830b565e", "family": "compose", "template": 2, "depth": 3, "pred": "-2", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "1d98f98efa362f80", "family": "compose", "template": 2, "depth": 3, "pred": "-4", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "665f9491c208ce8e", "family": "compose", "template": 2, "depth": 3, "pred": "14", "truth": "0", "correct": false, "edits_per_token": 2.0}, {"uid": "f54b0fe3fb897d9e", "family": "compose", "template": 2, "depth": 3, "pred": "16", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "44d407369e058248", "family": "compose", "template": 2, "depth": 3, "pred": "23", "truth": "158", "correct": false, "edits_per_token": 2.0}, {"uid": "bf7511876035c1fd", "family": "compose", "template": 2, "depth": 3, "pred": "-1", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "add68ae2a1e04601", "family": "compose", "template": 2, "depth": 3, "pred": "-4", "truth": "-3", "correct": false, "edits_per_token": 2.0}, {"uid": "ae11be33564637ac", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "a1fb5e3c4fe6050c", "family": "compose", "template": 2, "depth": 3, "pred": "10", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "b7a7f826d8f54edf", "family": "compose", "template": 2, "depth": 3, "pred": "0", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "41d7e4fb4c698ad5", "family": "compose", "template": 2, "depth": 3, "pred": "-3", "truth": "-4", "correct": false, "edits_per_token": 2.0}, {"uid": "1cb2706b7f6fbb2b", "family": "compose", "template": 2, "depth": 3, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "3e70547ce95699c3", "family": "compose", "template": 2, "depth": 3, "pred": "10", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "3d11c839f1453ce6", "family": "compose", "template": 2, "depth": 3, "pred": "15", "truth": "18", "correct": false, "edits_per_token": 2.0}, {"uid": "d58b04457399d716", "family": "compose", "template": 2, "depth": 3, "pred": "-6", "truth": "-14", "correct": false, "edits_per_token": 2.0}, {"uid": "c894735781f67d70", "family": "compose", "template": 2, "depth": 3, "pred": "-12", "truth": "-17", "correct": false, "edits_per_token": 2.0}, {"uid": "6510140707968d83", "family": "compose", "template": 2, "depth": 3, "pred": "-4", "truth": "0", "correct": false, "edits_per_token": 2.0}, {"uid": "1b41aaffed0ebaad", "family": "compose", "template": 2, "depth": 3, "pred": "-2", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "672b5c8f32a69ff3", "family": "compose", "template": 2, "depth": 3, "pred": "13", "truth": "25", "correct": false, "edits_per_token": 2.0}, {"uid": "237b6b3f75aad340", "family": "compose", "template": 2, "depth": 3, "pred": "3", "truth": "38", "correct": false, "edits_per_token": 2.0}, {"uid": "8b320774cd5bb130", "family": "compose", "template": 2, "depth": 3, "pred": "6", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "a4f4ff84e82ca377", "family": "compose", "template": 0, "depth": 3, "pred": "-2", "truth": "-9", "correct": false, "edits_per_token": 2.0}, {"uid": "4e40f161e5396048", "family": "compose", "template": 0, "depth": 3, "pred": "2", "truth": "17", "correct": false, "edits_per_token": 2.0}, {"uid": "8b82f59cbe3434e0", "family": "compose", "template": 0, "depth": 3, "pred": "-3", "truth": "-1", "correct": false, "edits_per_token": 2.0}, {"uid": "db71d89837843152", "family": "compose", "template": 0, "depth": 3, "pred": "3", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "0197263a81f5749b", "family": "compose", "template": 0, "depth": 3, "pred": "4", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "54ebfde1773c2e3f", "family": "compose", "template": 0, "depth": 3, "pred": "-2", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "0256ae6470dae63c", "family": "compose", "template": 0, "depth": 3, "pred": "12", "truth": "19", "correct": false, "edits_per_token": 2.0}, {"uid": "2df799eeaded8dfe", "family": "compose", "template": 0, "depth": 3, "pred": "-8", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "eba447db451d227b", "family": "compose", "template": 0, "depth": 3, "pred": "5", "truth": "12", "correct": false, "edits_per_token": 2.0}, {"uid": "39c693ae9553e888", "family": "compose", "template": 0, "depth": 3, "pred": "1", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "717c3337957264df", "family": "compose", "template": 0, "depth": 3, "pred": "-6", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "f728f34a28205d99", "family": "compose", "template": 0, "depth": 3, "pred": "5", "truth": "47", "correct": false, "edits_per_token": 2.0}, {"uid": "d2514f0dd399aff4", "family": "compose", "template": 0, "depth": 3, "pred": "-3", "truth": "17", "correct": false, "edits_per_token": 2.0}, {"uid": "529a8d6290df2519", "family": "compose", "template": 0, "depth": 3, "pred": "-7", "truth": "-10", "correct": false, "edits_per_token": 2.0}, {"uid": "d9ca9c93444f8be3", "family": "compose", "template": 0, "depth": 3, "pred": "-4", "truth": "-7", "correct": false, "edits_per_token": 2.0}, {"uid": "e0995d7984dacaee", "family": "compose", "template": 0, "depth": 3, "pred": "10", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "46861a528973f71b", "family": "compose", "template": 0, "depth": 3, "pred": "13", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "86907a1d4cfdd2aa", "family": "compose", "template": 0, "depth": 3, "pred": "2", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "4f5302ec715318f4", "family": "compose", "template": 0, "depth": 3, "pred": "12", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "5f5957f1a7eb0d1a", "family": "compose", "template": 0, "depth": 3, "pred": "-2", "truth": "-7", "correct": false, "edits_per_token": 2.0}, {"uid": "162aae75dc526041", "family": "compose", "template": 0, "depth": 3, "pred": "-12", "truth": "-1", "correct": false, "edits_per_token": 2.0}, {"uid": "74f86bfe34852fc7", "family": "compose", "template": 0, "depth": 3, "pred": "2", "truth": "0", "correct": false, "edits_per_token": 2.0}, {"uid": "03bfd6467dbb0da6", "family": "compose", "template": 0, "depth": 3, "pred": "0", "truth": "11", "correct": false, "edits_per_token": 2.0}, {"uid": "7d0b4917eaa90a1c", "family": "compose", "template": 0, "depth": 3, "pred": "9", "truth": "17", "correct": false, "edits_per_token": 2.0}, {"uid": "1afcc311fcdf51a7", "family": "compose", "template": 0, "depth": 3, "pred": "-2", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "3e29569c45e931be", "family": "compose", "template": 0, "depth": 3, "pred": "16", "truth": "28", "correct": false, "edits_per_token": 2.0}, {"uid": "203b0a6421469c30", "family": "compose", "template": 0, "depth": 3, "pred": "-4", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "22f135fea0674737", "family": "compose", "template": 0, "depth": 3, "pred": "4", "truth": "40", "correct": false, "edits_per_token": 2.0}, {"uid": "f3e9d937a0536532", "family": "compose", "template": 0, "depth": 3, "pred": "11", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "aafa1e44ca944136", "family": "compose", "template": 0, "depth": 3, "pred": "-6", "truth": "-7", "correct": false, "edits_per_token": 2.0}, {"uid": "1b976dace0d9a61f", "family": "compose", "template": 0, "depth": 3, "pred": "0", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "108e674dc2236456", "family": "compose", "template": 0, "depth": 3, "pred": "6", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "47e15055e7489308", "family": "compose", "template": 0, "depth": 3, "pred": "-6", "truth": "-2", "correct": false, "edits_per_token": 2.0}, {"uid": "becf1a65023cb394", "family": "compose", "template": 0, "depth": 3, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 2.0}, {"uid": "bb21eaadb8237e64", "family": "compose", "template": 0, "depth": 3, "pred": "0", "truth": "23", "correct": false, "edits_per_token": 2.0}, {"uid": "e8e95e457e0ff61c", "family": "compose", "template": 0, "depth": 3, "pred": "-10", "truth": "-8", "correct": false, "edits_per_token": 2.0}, {"uid": "7545d7b397cf7f92", "family": "compose", "template": 0, "depth": 3, "pred": "-6", "truth": "-15", "correct": false, "edits_per_token": 2.0}, {"uid": "b1ea2f279468da9c", "family": "compose", "template": 0, "depth": 3, "pred": "3", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "94b0faad6cfc5da0", "family": "compose", "template": 0, "depth": 3, "pred": "16", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "cd59981fea8d6b31", "family": "compose", "template": 0, "depth": 3, "pred": "0", "truth": "-10", "correct": false, "edits_per_token": 2.0}, {"uid": "fab73cdef573e6ee", "family": "compose", "template": 0, "depth": 3, "pred": "8", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "e11d203f1ac8ab57", "family": "compose", "template": 0, "depth": 3, "pred": "18", "truth": "33", "correct": false, "edits_per_token": 2.0}, {"uid": "1b2bc47aab6fb042", "family": "compose", "template": 0, "depth": 3, "pred": "24", "truth": "179", "correct": false, "edits_per_token": 2.0}, {"uid": "06fa3df1bf999d20", "family": "compose", "template": 0, "depth": 3, "pred": "10", "truth": "-13", "correct": false, "edits_per_token": 2.0}, {"uid": "3d679575c07dffbd", "family": "compose", "template": 0, "depth": 3, "pred": "10", "truth": "22", "correct": false, "edits_per_token": 2.0}, {"uid": "ebab68f0c66d94b2", "family": "compose", "template": 1, "depth": 3, "pred": "14", "truth": "18", "correct": false, "edits_per_token": 2.0}, {"uid": "4b655e468b467a1e", "family": "compose", "template": 1, "depth": 3, "pred": "7", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "6333686e41458a19", "family": "compose", "template": 1, "depth": 3, "pred": "-5", "truth": "-5", "correct": true, "edits_per_token": 2.0}, {"uid": "4904a77f2f08c9bf", "family": "compose", "template": 1, "depth": 3, "pred": "5", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "aea9f4148ec22f94", "family": "compose", "template": 1, "depth": 3, "pred": "0", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "ae8a8299a9c7e34e", "family": "compose", "template": 1, "depth": 3, "pred": "5", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "6570c0e205b5d048", "family": "compose", "template": 1, "depth": 3, "pred": "7", "truth": "33", "correct": false, "edits_per_token": 2.0}, {"uid": "cdf5dbc0e8fb28ba", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "47", "correct": false, "edits_per_token": 2.0}, {"uid": "ff6c5e131420c7fa", "family": "compose", "template": 1, "depth": 3, "pred": "-3", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "43b668257f793894", "family": "compose", "template": 1, "depth": 3, "pred": "0", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "f134711ed22583b1", "family": "compose", "template": 1, "depth": 3, "pred": "-1", "truth": "17", "correct": false, "edits_per_token": 2.0}, {"uid": "d609372c6cf824c2", "family": "compose", "template": 1, "depth": 3, "pred": "12", "truth": "23", "correct": false, "edits_per_token": 2.0}, {"uid": "58b39aade95d19cc", "family": "compose", "template": 1, "depth": 3, "pred": "5", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "3abaecdd95360608", "family": "compose", "template": 1, "depth": 3, "pred": "4", "truth": "25", "correct": false, "edits_per_token": 2.0}, {"uid": "da42f0492b37e85f", "family": "compose", "template": 1, "depth": 3, "pred": "13", "truth": "19", "correct": false, "edits_per_token": 2.0}, {"uid": "ef200ce61ab718cc", "family": "compose", "template": 1, "depth": 3, "pred": "20", "truth": "117", "correct": false, "edits_per_token": 2.0}, {"uid": "b76a09fc66e8f823", "family": "compose", "template": 1, "depth": 3, "pred": "-4", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "708025be67bbf44c", "family": "compose", "template": 1, "depth": 3, "pred": "13", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "75c6941e53662b69", "family": "compose", "template": 1, "depth": 3, "pred": "11", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "5742f3bbf7ae33ef", "family": "compose", "template": 1, "depth": 3, "pred": "0", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "fef642e44943ca2e", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "15", "correct": false, "edits_per_token": 2.0}, {"uid": "eab7cf6c522a4d74", "family": "compose", "template": 1, "depth": 3, "pred": "-2", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "39dc58cd1ddad214", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "42", "correct": false, "edits_per_token": 2.0}, {"uid": "c982c4b4b62b46a2", "family": "compose", "template": 1, "depth": 3, "pred": "0", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "ec9d2b1e581198de", "family": "compose", "template": 1, "depth": 3, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 2.0}, {"uid": "ed5b7032ccb677f6", "family": "compose", "template": 1, "depth": 3, "pred": "22", "truth": "92", "correct": false, "edits_per_token": 2.0}, {"uid": "e85efd8a49131fbb", "family": "compose", "template": 1, "depth": 3, "pred": "1", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "9b1937873462dae3", "family": "compose", "template": 1, "depth": 3, "pred": "8", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "92f8c134b74ab497", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "3bdbc819e2658ce6", "family": "compose", "template": 1, "depth": 3, "pred": "10", "truth": "18", "correct": false, "edits_per_token": 2.0}, {"uid": "09ab4572a068e609", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "-10", "correct": false, "edits_per_token": 2.0}, {"uid": "197d7789fd7d2320", "family": "compose", "template": 1, "depth": 3, "pred": "8", "truth": "11", "correct": false, "edits_per_token": 2.0}, {"uid": "1b288c05c50ab28a", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "f6c0c00532c7b6ec", "family": "compose", "template": 1, "depth": 3, "pred": "14", "truth": "119", "correct": false, "edits_per_token": 2.0}, {"uid": "1ad9ac7133c048a6", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "4a203c954bc59a55", "family": "compose", "template": 1, "depth": 3, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "e063bbdd65e40fba", "family": "compose", "template": 1, "depth": 3, "pred": "9", "truth": "15", "correct": false, "edits_per_token": 2.0}, {"uid": "8047e5d21252c647", "family": "compose", "template": 1, "depth": 3, "pred": "8", "truth": "24", "correct": false, "edits_per_token": 2.0}, {"uid": "1a308cbea09e1820", "family": "compose", "template": 1, "depth": 3, "pred": "-2", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "ce045884547d667e", "family": "compose", "template": 1, "depth": 3, "pred": "20", "truth": "31", "correct": false, "edits_per_token": 2.0}, {"uid": "f10d7c733d8c484d", "family": "compose", "template": 1, "depth": 3, "pred": "9", "truth": "49", "correct": false, "edits_per_token": 2.0}, {"uid": "14744517e85781d6", "family": "compose", "template": 1, "depth": 3, "pred": "4", "truth": "17", "correct": false, "edits_per_token": 2.0}, {"uid": "a9c7c3ba93568f28", "family": "compose", "template": 1, "depth": 3, "pred": "2", "truth": "53", "correct": false, "edits_per_token": 2.0}, {"uid": "3447bfb6e0f058ff", "family": "compose", "template": 1, "depth": 3, "pred": "9", "truth": "-1", "correct": false, "edits_per_token": 2.0}, {"uid": "c77194dd735c892a", "family": "compose", "template": 2, "depth": 4, "pred": "16", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "f69e0f2af4310da1", "family": "compose", "template": 2, "depth": 4, "pred": "-4", "truth": "-9", "correct": false, "edits_per_token": 2.0}, {"uid": "b4288699c21fb462", "family": "compose", "template": 2, "depth": 4, "pred": "-1", "truth": "-3", "correct": false, "edits_per_token": 2.0}, {"uid": "aa53af096c007782", "family": "compose", "template": 2, "depth": 4, "pred": "17", "truth": "-1", "correct": false, "edits_per_token": 2.0}, {"uid": "7ebb4ddcb7cd4922", "family": "compose", "template": 2, "depth": 4, "pred": "9", "truth": "-3", "correct": false, "edits_per_token": 2.0}, {"uid": "27eb9dbb08f6be89", "family": "compose", "template": 2, "depth": 4, "pred": "-2", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "60a194c1a80d55c2", "family": "compose", "template": 2, "depth": 4, "pred": "2", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "ac25a665c5b8ba5c", "family": "compose", "template": 2, "depth": 4, "pred": "-1", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "c83e49945f947ee7", "family": "compose", "template": 2, "depth": 4, "pred": "14", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "6fb906a72214b74a", "family": "compose", "template": 2, "depth": 4, "pred": "5", "truth": "81", "correct": false, "edits_per_token": 2.0}, {"uid": "45e407d76733740a", "family": "compose", "template": 2, "depth": 4, "pred": "-8", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "61de5971ee5e66d8", "family": "compose", "template": 2, "depth": 4, "pred": "-16", "truth": "-1", "correct": false, "edits_per_token": 2.0}, {"uid": "c83db0f0d1e51c97", "family": "compose", "template": 2, "depth": 4, "pred": "0", "truth": "-10", "correct": false, "edits_per_token": 2.0}, {"uid": "dc1cd6108451a386", "family": "compose", "template": 2, "depth": 4, "pred": "2", "truth": "11", "correct": false, "edits_per_token": 2.0}, {"uid": "e258998e6e182ce5", "family": "compose", "template": 2, "depth": 4, "pred": "12", "truth": "-3", "correct": false, "edits_per_token": 2.0}, {"uid": "6133cf12f6a73d18", "family": "compose", "template": 2, "depth": 4, "pred": "-6", "truth": "-7", "correct": false, "edits_per_token": 2.0}, {"uid": "67454e042a0ed655", "family": "compose", "template": 2, "depth": 4, "pred": "-2", "truth": "-13", "correct": false, "edits_per_token": 2.0}, {"uid": "356b028001a44989", "family": "compose", "template": 2, "depth": 4, "pred": "-1", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "a52f72107bb33d1f", "family": "compose", "template": 2, "depth": 4, "pred": "2", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "04b5402245e39d49", "family": "compose", "template": 2, "depth": 4, "pred": "7", "truth": "19", "correct": false, "edits_per_token": 2.0}, {"uid": "4c0fb74ec5f153e8", "family": "compose", "template": 2, "depth": 4, "pred": "5", "truth": "18", "correct": false, "edits_per_token": 2.0}, {"uid": "7f06cb4a08547083", "family": "compose", "template": 2, "depth": 4, "pred": "0", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "e83018fb177d9290", "family": "compose", "template": 2, "depth": 4, "pred": "15", "truth": "-18", "correct": false, "edits_per_token": 2.0}, {"uid": "980118a6d9a0fb70", "family": "compose", "template": 2, "depth": 4, "pred": "0", "truth": "-2", "correct": false, "edits_per_token": 2.0}, {"uid": "508adf2fa51fa4b7", "family": "compose", "template": 2, "depth": 4, "pred": "8", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "44f7ab9028e2f3a6", "family": "compose", "template": 2, "depth": 4, "pred": "5", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "024cf84c9fa808a8", "family": "compose", "template": 2, "depth": 4, "pred": "10", "truth": "29", "correct": false, "edits_per_token": 2.0}, {"uid": "31a85a27753002e3", "family": "compose", "template": 2, "depth": 4, "pred": "-8", "truth": "-4", "correct": false, "edits_per_token": 2.0}, {"uid": "bddfd858fa52f0fc", "family": "compose", "template": 2, "depth": 4, "pred": "4", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "53ea3a107aa4124d", "family": "compose", "template": 2, "depth": 4, "pred": "16", "truth": "40", "correct": false, "edits_per_token": 2.0}, {"uid": "fb41d24e9b524dbf", "family": "compose", "template": 2, "depth": 4, "pred": "-4", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "daacbc2f9cc02c6e", "family": "compose", "template": 2, "depth": 4, "pred": "5", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "ba48e47b18027d09", "family": "compose", "template": 2, "depth": 4, "pred": "1", "truth": "-9", "correct": false, "edits_per_token": 2.0}, {"uid": "f8dd559e23329aa6", "family": "compose", "template": 2, "depth": 4, "pred": "-2", "truth": "12", "correct": false, "edits_per_token": 2.0}, {"uid": "89e113217d4e8dde", "family": "compose", "template": 2, "depth": 4, "pred": "-6", "truth": "-1", "correct": false, "edits_per_token": 2.0}, {"uid": "f63237735ba6b2b2", "family": "compose", "template": 2, "depth": 4, "pred": "17", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "c5ed770de008b78f", "family": "compose", "template": 2, "depth": 4, "pred": "2", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "d7b169bfa088fc4b", "family": "compose", "template": 2, "depth": 4, "pred": "-18", "truth": "-15", "correct": false, "edits_per_token": 2.0}, {"uid": "5e099825a8eb3846", "family": "compose", "template": 2, "depth": 4, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 2.0}, {"uid": "df5aa3e5e3ab531f", "family": "compose", "template": 2, "depth": 4, "pred": "3", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "a7cb8b36fbb34d80", "family": "compose", "template": 2, "depth": 4, "pred": "-2", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "ca89ac23e031b4d9", "family": "compose", "template": 2, "depth": 4, "pred": "17", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "57e8a444a52e9659", "family": "compose", "template": 2, "depth": 4, "pred": "5", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "019d77a6be125d5f", "family": "compose", "template": 2, "depth": 4, "pred": "2", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "4e31b04ed838d8a4", "family": "compose", "template": 0, "depth": 4, "pred": "8", "truth": "179", "correct": false, "edits_per_token": 2.0}, {"uid": "cd4ad97c104f4ccc", "family": "compose", "template": 0, "depth": 4, "pred": "-12", "truth": "-25", "correct": false, "edits_per_token": 2.0}, {"uid": "67a8b958f2783833", "family": "compose", "template": 0, "depth": 4, "pred": "0", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "c08c4460a5101f62", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "-1", "correct": false, "edits_per_token": 2.0}, {"uid": "393fb7f6b6570266", "family": "compose", "template": 0, "depth": 4, "pred": "-5", "truth": "11", "correct": false, "edits_per_token": 2.0}, {"uid": "c52da7a400507ead", "family": "compose", "template": 0, "depth": 4, "pred": "0", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "5cac6a518c45b812", "family": "compose", "template": 0, "depth": 4, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 2.0}, {"uid": "0bf5c23a7131ed77", "family": "compose", "template": 0, "depth": 4, "pred": "5", "truth": "0", "correct": false, "edits_per_token": 2.0}, {"uid": "25efb341025a1c02", "family": "compose", "template": 0, "depth": 4, "pred": "-2", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "4a79def086294a3a", "family": "compose", "template": 0, "depth": 4, "pred": "6", "truth": "-3", "correct": false, "edits_per_token": 2.0}, {"uid": "faf455eb5168841e", "family": "compose", "template": 0, "depth": 4, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 2.0}, {"uid": "b746ed0dc37016b2", "family": "compose", "template": 0, "depth": 4, "pred": "3", "truth": "-2", "correct": false, "edits_per_token": 2.0}, {"uid": "4a36f3514b13957d", "family": "compose", "template": 0, "depth": 4, "pred": "15", "truth": "113", "correct": false, "edits_per_token": 2.0}, {"uid": "17199e564f6e3262", "family": "compose", "template": 0, "depth": 4, "pred": "12", "truth": "102", "correct": false, "edits_per_token": 2.0}, {"uid": "e677a33994695496", "family": "compose", "template": 0, "depth": 4, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "53dac0369fd28ac0", "family": "compose", "template": 0, "depth": 4, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "c2cfa1bf34bc1d6d", "family": "compose", "template": 0, "depth": 4, "pred": "-8", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "8fedc1effa6611a3", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "51", "correct": false, "edits_per_token": 2.0}, {"uid": "0e005022cc77f8c6", "family": "compose", "template": 0, "depth": 4, "pred": "-15", "truth": "-13", "correct": false, "edits_per_token": 2.0}, {"uid": "26405d91a56f7855", "family": "compose", "template": 0, "depth": 4, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "4f7dea5defe99ded", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "-1", "correct": false, "edits_per_token": 2.0}, {"uid": "49c5a431f2ef6c26", "family": "compose", "template": 0, "depth": 4, "pred": "13", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "aaf95ea7d25fa328", "family": "compose", "template": 0, "depth": 4, "pred": "7", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "a2efa1cc868910a2", "family": "compose", "template": 0, "depth": 4, "pred": "-6", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "d8797d338a3b92a8", "family": "compose", "template": 0, "depth": 4, "pred": "-16", "truth": "-10", "correct": false, "edits_per_token": 2.0}, {"uid": "819700d90a033274", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "6b79b778c94c3340", "family": "compose", "template": 0, "depth": 4, "pred": "-8", "truth": "-3", "correct": false, "edits_per_token": 2.0}, {"uid": "f46dd9ba49c5f9e5", "family": "compose", "template": 0, "depth": 4, "pred": "-3", "truth": "-10", "correct": false, "edits_per_token": 2.0}, {"uid": "6297fac714f70477", "family": "compose", "template": 0, "depth": 4, "pred": "4", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "5bc9e1d49fcaabba", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "30", "correct": false, "edits_per_token": 2.0}, {"uid": "095e7a1076d7cbe1", "family": "compose", "template": 0, "depth": 4, "pred": "10", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "02e62b8d24c38e10", "family": "compose", "template": 0, "depth": 4, "pred": "5", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "e92f9a90aaa752ab", "family": "compose", "template": 0, "depth": 4, "pred": "-5", "truth": "-12", "correct": false, "edits_per_token": 2.0}, {"uid": "b8f1efdf78952406", "family": "compose", "template": 0, "depth": 4, "pred": "-16", "truth": "-17", "correct": false, "edits_per_token": 2.0}, {"uid": "ac30d19ea27ec58e", "family": "compose", "template": 0, "depth": 4, "pred": "6", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "d754429b5b350068", "family": "compose", "template": 0, "depth": 4, "pred": "0", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "1ea10fa59288e773", "family": "compose", "template": 0, "depth": 4, "pred": "7", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "325032d22f88969d", "family": "compose", "template": 0, "depth": 4, "pred": "10", "truth": "-8", "correct": false, "edits_per_token": 2.0}, {"uid": "ce5754d69875787b", "family": "compose", "template": 0, "depth": 4, "pred": "-8", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "d221bad35e9a7253", "family": "compose", "template": 0, "depth": 4, "pred": "3", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "3702fc94bbb38ee7", "family": "compose", "template": 0, "depth": 4, "pred": "10", "truth": "23", "correct": false, "edits_per_token": 2.0}, {"uid": "4acd5466e30298d3", "family": "compose", "template": 0, "depth": 4, "pred": "20", "truth": "35", "correct": false, "edits_per_token": 2.0}, {"uid": "c5c331dd1513732d", "family": "compose", "template": 0, "depth": 4, "pred": "10", "truth": "21", "correct": false, "edits_per_token": 2.0}, {"uid": "de6345b12fdb1b98", "family": "compose", "template": 0, "depth": 4, "pred": "20", "truth": "32", "correct": false, "edits_per_token": 2.0}, {"uid": "92ccd36e2de558cf", "family": "compose", "template": 1, "depth": 4, "pred": "10", "truth": "39", "correct": false, "edits_per_token": 2.0}, {"uid": "50faa50b3396ffae", "family": "compose", "template": 1, "depth": 4, "pred": "7", "truth": "12", "correct": false, "edits_per_token": 2.0}, {"uid": "29dee9386795c180", "family": "compose", "template": 1, "depth": 4, "pred": "-6", "truth": "18", "correct": false, "edits_per_token": 2.0}, {"uid": "238c0bc8c05a35a2", "family": "compose", "template": 1, "depth": 4, "pred": "-2", "truth": "0", "correct": false, "edits_per_token": 2.0}, {"uid": "38c93e00c30f778c", "family": "compose", "template": 1, "depth": 4, "pred": "-24", "truth": "-17", "correct": false, "edits_per_token": 2.0}, {"uid": "78e560fe43169af8", "family": "compose", "template": 1, "depth": 4, "pred": "20", "truth": "-4", "correct": false, "edits_per_token": 2.0}, {"uid": "b842bbdfb385a2c8", "family": "compose", "template": 1, "depth": 4, "pred": "0", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "ca2e93da50248f99", "family": "compose", "template": 1, "depth": 4, "pred": "2", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "404b290af2697cc1", "family": "compose", "template": 1, "depth": 4, "pred": "15", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "80a7754cfb080d93", "family": "compose", "template": 1, "depth": 4, "pred": "-7", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "63602722310a02d6", "family": "compose", "template": 1, "depth": 4, "pred": "10", "truth": "42", "correct": false, "edits_per_token": 2.0}, {"uid": "34b4950e5d8a0b1e", "family": "compose", "template": 1, "depth": 4, "pred": "0", "truth": "-1", "correct": false, "edits_per_token": 2.0}, {"uid": "29708655a0f9fa6d", "family": "compose", "template": 1, "depth": 4, "pred": "-3", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "7f8cff76916c511e", "family": "compose", "template": 1, "depth": 4, "pred": "2", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "985ece2df7f3c619", "family": "compose", "template": 1, "depth": 4, "pred": "20", "truth": "146", "correct": false, "edits_per_token": 2.0}, {"uid": "1358a81db4f24e15", "family": "compose", "template": 1, "depth": 4, "pred": "20", "truth": "116", "correct": false, "edits_per_token": 2.0}, {"uid": "48031d8c67e39d8b", "family": "compose", "template": 1, "depth": 4, "pred": "-14", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "a6147eea1031f132", "family": "compose", "template": 1, "depth": 4, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "6582654868e1e4a0", "family": "compose", "template": 1, "depth": 4, "pred": "10", "truth": "115", "correct": false, "edits_per_token": 2.0}, {"uid": "3587b0aaef8cf3e1", "family": "compose", "template": 1, "depth": 4, "pred": "8", "truth": "160", "correct": false, "edits_per_token": 2.0}, {"uid": "2cce12ccba5151de", "family": "compose", "template": 1, "depth": 4, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "5d28412b23afa91f", "family": "compose", "template": 1, "depth": 4, "pred": "12", "truth": "50", "correct": false, "edits_per_token": 2.0}, {"uid": "b021356c73afb769", "family": "compose", "template": 1, "depth": 4, "pred": "-6", "truth": "-12", "correct": false, "edits_per_token": 2.0}, {"uid": "98b7fa299d84ef2e", "family": "compose", "template": 1, "depth": 4, "pred": "-6", "truth": "-15", "correct": false, "edits_per_token": 2.0}, {"uid": "3d8b4948bd221577", "family": "compose", "template": 1, "depth": 4, "pred": "-4", "truth": "-12", "correct": false, "edits_per_token": 2.0}, {"uid": "923ce5295838ed05", "family": "compose", "template": 1, "depth": 4, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 2.0}, {"uid": "06fef04b9fe5f342", "family": "compose", "template": 1, "depth": 4, "pred": "-8", "truth": "-16", "correct": false, "edits_per_token": 2.0}, {"uid": "4a350417dc3523cc", "family": "compose", "template": 1, "depth": 4, "pred": "-5", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "fc87f1aa022adcc2", "family": "compose", "template": 1, "depth": 4, "pred": "6", "truth": "-11", "correct": false, "edits_per_token": 2.0}, {"uid": "2e19bd8266852d8b", "family": "compose", "template": 1, "depth": 4, "pred": "6", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "abdac9d2fc00886e", "family": "compose", "template": 1, "depth": 4, "pred": "-4", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "83101cc76f2de188", "family": "compose", "template": 1, "depth": 4, "pred": "-15", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "2d00a93eb0c1b20a", "family": "compose", "template": 1, "depth": 4, "pred": "-3", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "545514d052c79cda", "family": "compose", "template": 1, "depth": 4, "pred": "4", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "8f05f418121b41a7", "family": "compose", "template": 1, "depth": 4, "pred": "13", "truth": "82", "correct": false, "edits_per_token": 2.0}, {"uid": "09dd937e1ef4b6c0", "family": "compose", "template": 1, "depth": 4, "pred": "5", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "c21f46d5ca49a0ce", "family": "compose", "template": 1, "depth": 4, "pred": "21", "truth": "105", "correct": false, "edits_per_token": 2.0}, {"uid": "4686b81018c38653", "family": "compose", "template": 1, "depth": 4, "pred": "9", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "ac04d2d2a5b89df8", "family": "compose", "template": 1, "depth": 4, "pred": "3", "truth": "-1", "correct": false, "edits_per_token": 2.0}, {"uid": "2916a34b7a3c5497", "family": "compose", "template": 1, "depth": 4, "pred": "-3", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "8c9bbb68ef0ff75b", "family": "compose", "template": 1, "depth": 4, "pred": "-4", "truth": "-16", "correct": false, "edits_per_token": 2.0}, {"uid": "948b8c326fa8c94b", "family": "compose", "template": 1, "depth": 4, "pred": "7", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "9608a0a9a3cf4853", "family": "compose", "template": 1, "depth": 4, "pred": "5", "truth": "17", "correct": false, "edits_per_token": 2.0}, {"uid": "52c595795ab95e82", "family": "compose", "template": 1, "depth": 4, "pred": "14", "truth": "22", "correct": false, "edits_per_token": 2.0}, {"uid": "b72fcba0a1ab05d0", "family": "compose", "template": 1, "depth": 4, "pred": "12", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "bee3aabc6bd5223a", "family": "compose", "template": 2, "depth": 5, "pred": "20", "truth": "28", "correct": false, "edits_per_token": 2.0}, {"uid": "8991d7fdb159825d", "family": "compose", "template": 2, "depth": 5, "pred": "6", "truth": "17", "correct": false, "edits_per_token": 2.0}, {"uid": "f81d9500784660dd", "family": "compose", "template": 2, "depth": 5, "pred": "4", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "df2ebcbe11e34787", "family": "compose", "template": 2, "depth": 5, "pred": "-2", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "c020e029edb14f57", "family": "compose", "template": 2, "depth": 5, "pred": "10", "truth": "-14", "correct": false, "edits_per_token": 2.0}, {"uid": "61ab1fe0f9bf8174", "family": "compose", "template": 2, "depth": 5, "pred": "-3", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "2c3b4386fc0cd5e2", "family": "compose", "template": 2, "depth": 5, "pred": "6", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "3e451812e17f0f49", "family": "compose", "template": 2, "depth": 5, "pred": "7", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "5dcf4e4893b12b48", "family": "compose", "template": 2, "depth": 5, "pred": "9", "truth": "27", "correct": false, "edits_per_token": 2.0}, {"uid": "ade1e9e1062031d5", "family": "compose", "template": 2, "depth": 5, "pred": "15", "truth": "26", "correct": false, "edits_per_token": 2.0}, {"uid": "d476cf4369ac7e22", "family": "compose", "template": 2, "depth": 5, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 2.0}, {"uid": "b3304b4efd082e9a", "family": "compose", "template": 2, "depth": 5, "pred": "0", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "e5288726427262e5", "family": "compose", "template": 2, "depth": 5, "pred": "14", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "894fcb5670fbca62", "family": "compose", "template": 2, "depth": 5, "pred": "13", "truth": "18", "correct": false, "edits_per_token": 2.0}, {"uid": "51129c69cded6af2", "family": "compose", "template": 2, "depth": 5, "pred": "3", "truth": "19", "correct": false, "edits_per_token": 2.0}, {"uid": "dbcebfe32b261b36", "family": "compose", "template": 2, "depth": 5, "pred": "26", "truth": "406", "correct": false, "edits_per_token": 2.0}, {"uid": "f6d12b0297c1c550", "family": "compose", "template": 2, "depth": 5, "pred": "-12", "truth": "-2", "correct": false, "edits_per_token": 2.0}, {"uid": "76c08480dce34c43", "family": "compose", "template": 2, "depth": 5, "pred": "5", "truth": "39", "correct": false, "edits_per_token": 2.0}, {"uid": "baea2ba74c196937", "family": "compose", "template": 2, "depth": 5, "pred": "-16", "truth": "-30", "correct": false, "edits_per_token": 2.0}, {"uid": "3f59a13b0dc3dc17", "family": "compose", "template": 2, "depth": 5, "pred": "4", "truth": "17", "correct": false, "edits_per_token": 2.0}, {"uid": "4caa849dad301058", "family": "compose", "template": 2, "depth": 5, "pred": "18", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "b348283e0f65231e", "family": "compose", "template": 2, "depth": 5, "pred": "-2", "truth": "-13", "correct": false, "edits_per_token": 2.0}, {"uid": "560fee240f31263a", "family": "compose", "template": 2, "depth": 5, "pred": "6", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "08dc517782bc8fef", "family": "compose", "template": 2, "depth": 5, "pred": "-8", "truth": "-9", "correct": false, "edits_per_token": 2.0}, {"uid": "28600ff91708289c", "family": "compose", "template": 2, "depth": 5, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "3f5eac859ba6fa37", "family": "compose", "template": 2, "depth": 5, "pred": "-6", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "9264073cc6ce8083", "family": "compose", "template": 2, "depth": 5, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 2.0}, {"uid": "958d4755d009e098", "family": "compose", "template": 2, "depth": 5, "pred": "-5", "truth": "-10", "correct": false, "edits_per_token": 2.0}, {"uid": "c1256114e43d3770", "family": "compose", "template": 2, "depth": 5, "pred": "4", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "99b6163909b9e9fd", "family": "compose", "template": 2, "depth": 5, "pred": "20", "truth": "27", "correct": false, "edits_per_token": 2.0}, {"uid": "dafeef05c5ec8fc4", "family": "compose", "template": 2, "depth": 5, "pred": "13", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "77682fc13ea83713", "family": "compose", "template": 2, "depth": 5, "pred": "5", "truth": "45", "correct": false, "edits_per_token": 2.0}, {"uid": "04de1db4dc636bcc", "family": "compose", "template": 2, "depth": 5, "pred": "14", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "c712c5b393de2310", "family": "compose", "template": 2, "depth": 5, "pred": "6", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "29e13e67d2c97073", "family": "compose", "template": 2, "depth": 5, "pred": "0", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "86397ff87e919200", "family": "compose", "template": 2, "depth": 5, "pred": "-16", "truth": "18", "correct": false, "edits_per_token": 2.0}, {"uid": "5837733b6c9e24b1", "family": "compose", "template": 2, "depth": 5, "pred": "-5", "truth": "-7", "correct": false, "edits_per_token": 2.0}, {"uid": "394bf97876be6993", "family": "compose", "template": 2, "depth": 5, "pred": "14", "truth": "93", "correct": false, "edits_per_token": 2.0}, {"uid": "76b2875e84ee2d00", "family": "compose", "template": 2, "depth": 5, "pred": "16", "truth": "148", "correct": false, "edits_per_token": 2.0}, {"uid": "6a14cc56e1d5eaac", "family": "compose", "template": 2, "depth": 5, "pred": "13", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "d1bcc6c12488902d", "family": "compose", "template": 2, "depth": 5, "pred": "20", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "2cc159a55fb29fd7", "family": "compose", "template": 2, "depth": 5, "pred": "9", "truth": "-3", "correct": false, "edits_per_token": 2.0}, {"uid": "8e8cb310090cdf9f", "family": "compose", "template": 2, "depth": 5, "pred": "5", "truth": "22", "correct": false, "edits_per_token": 2.0}, {"uid": "bd894beab682cc7b", "family": "compose", "template": 2, "depth": 5, "pred": "3", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "20abae29de56cfbc", "family": "compose", "template": 0, "depth": 5, "pred": "20", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "1bffd281778527b5", "family": "compose", "template": 0, "depth": 5, "pred": "-3", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "76a3e1969e0dd187", "family": "compose", "template": 0, "depth": 5, "pred": "-16", "truth": "112", "correct": false, "edits_per_token": 2.0}, {"uid": "135b4dd9880a7711", "family": "compose", "template": 0, "depth": 5, "pred": "10", "truth": "119", "correct": false, "edits_per_token": 2.0}, {"uid": "98fe8dd4b77530ae", "family": "compose", "template": 0, "depth": 5, "pred": "6", "truth": "-2", "correct": false, "edits_per_token": 2.0}, {"uid": "ab504be8f60f8773", "family": "compose", "template": 0, "depth": 5, "pred": "20", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "ae8a7e76edd9b1ac", "family": "compose", "template": 0, "depth": 5, "pred": "-21", "truth": "-45", "correct": false, "edits_per_token": 2.0}, {"uid": "57d846cf5b774601", "family": "compose", "template": 0, "depth": 5, "pred": "8", "truth": "153", "correct": false, "edits_per_token": 2.0}, {"uid": "9291a187e465d9c8", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "40", "correct": false, "edits_per_token": 2.0}, {"uid": "3027f279111dbcae", "family": "compose", "template": 0, "depth": 5, "pred": "-4", "truth": "-17", "correct": false, "edits_per_token": 2.0}, {"uid": "184ff66abbf56a5c", "family": "compose", "template": 0, "depth": 5, "pred": "0", "truth": "-3", "correct": false, "edits_per_token": 2.0}, {"uid": "eb1a5d81063ecbcb", "family": "compose", "template": 0, "depth": 5, "pred": "24", "truth": "35", "correct": false, "edits_per_token": 2.0}, {"uid": "4746ec994270639a", "family": "compose", "template": 0, "depth": 5, "pred": "15", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "727e5ef3770421a6", "family": "compose", "template": 0, "depth": 5, "pred": "8", "truth": "127", "correct": false, "edits_per_token": 2.0}, {"uid": "20e006d873dd1edb", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "17", "correct": false, "edits_per_token": 2.0}, {"uid": "8e32ed5b07934152", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "91fc4c488e9ced3d", "family": "compose", "template": 0, "depth": 5, "pred": "7", "truth": "-13", "correct": false, "edits_per_token": 2.0}, {"uid": "4984028c93289a35", "family": "compose", "template": 0, "depth": 5, "pred": "-16", "truth": "35", "correct": false, "edits_per_token": 2.0}, {"uid": "55a3205999715b20", "family": "compose", "template": 0, "depth": 5, "pred": "7", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "18779c7d7329f224", "family": "compose", "template": 0, "depth": 5, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "cec5a99eca1756c7", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "a524d85a18a67e49", "family": "compose", "template": 0, "depth": 5, "pred": "-3", "truth": "-7", "correct": false, "edits_per_token": 2.0}, {"uid": "ffd488cb6bea85ff", "family": "compose", "template": 0, "depth": 5, "pred": "24", "truth": "95", "correct": false, "edits_per_token": 2.0}, {"uid": "a6bd916c42aee6c4", "family": "compose", "template": 0, "depth": 5, "pred": "-3", "truth": "-13", "correct": false, "edits_per_token": 2.0}, {"uid": "3c867544528e8d53", "family": "compose", "template": 0, "depth": 5, "pred": "5", "truth": "79", "correct": false, "edits_per_token": 2.0}, {"uid": "356c3a7de1144ea9", "family": "compose", "template": 0, "depth": 5, "pred": "4", "truth": "-4", "correct": false, "edits_per_token": 2.0}, {"uid": "4abed743053a2033", "family": "compose", "template": 0, "depth": 5, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "4a2f3f7d91c213de", "family": "compose", "template": 0, "depth": 5, "pred": "8", "truth": "97", "correct": false, "edits_per_token": 2.0}, {"uid": "51aea898105d4c43", "family": "compose", "template": 0, "depth": 5, "pred": "-3", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "f374d4ad191ded59", "family": "compose", "template": 0, "depth": 5, "pred": "7", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "2c485719d8098bbe", "family": "compose", "template": 0, "depth": 5, "pred": "-16", "truth": "-15", "correct": false, "edits_per_token": 2.0}, {"uid": "95343d962b76844d", "family": "compose", "template": 0, "depth": 5, "pred": "7", "truth": "87", "correct": false, "edits_per_token": 2.0}, {"uid": "86e8356113bdbd04", "family": "compose", "template": 0, "depth": 5, "pred": "4", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "5704b25879d5b987", "family": "compose", "template": 0, "depth": 5, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "904694feea1c0838", "family": "compose", "template": 0, "depth": 5, "pred": "10", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "8d1eaafea5c8860c", "family": "compose", "template": 0, "depth": 5, "pred": "3", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "697800c219134d74", "family": "compose", "template": 0, "depth": 5, "pred": "9", "truth": "92", "correct": false, "edits_per_token": 2.0}, {"uid": "c92c4e3ef89e0f14", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "-14", "correct": false, "edits_per_token": 2.0}, {"uid": "d07423f5eb5e3551", "family": "compose", "template": 0, "depth": 5, "pred": "-5", "truth": "15", "correct": false, "edits_per_token": 2.0}, {"uid": "8477f9ed7d2a0ab7", "family": "compose", "template": 0, "depth": 5, "pred": "-21", "truth": "-26", "correct": false, "edits_per_token": 2.0}, {"uid": "d0ca46e01def43f3", "family": "compose", "template": 0, "depth": 5, "pred": "13", "truth": "76", "correct": false, "edits_per_token": 2.0}, {"uid": "c861a4351ff59cc1", "family": "compose", "template": 0, "depth": 5, "pred": "12", "truth": "11", "correct": false, "edits_per_token": 2.0}, {"uid": "15fc685f62ca87b2", "family": "compose", "template": 0, "depth": 5, "pred": "16", "truth": "117", "correct": false, "edits_per_token": 2.0}, {"uid": "6f28dc9656cc50a0", "family": "compose", "template": 0, "depth": 5, "pred": "10", "truth": "28", "correct": false, "edits_per_token": 2.0}, {"uid": "26ab4ae08da814b0", "family": "compose", "template": 0, "depth": 5, "pred": "-7", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "24d6f7c2be67dbc5", "family": "compose", "template": 1, "depth": 5, "pred": "30", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "5c6fa527441afc1a", "family": "compose", "template": 1, "depth": 5, "pred": "-4", "truth": "47", "correct": false, "edits_per_token": 2.0}, {"uid": "45106f9401657f9f", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "113e562bce7bec31", "family": "compose", "template": 1, "depth": 5, "pred": "0", "truth": "-3", "correct": false, "edits_per_token": 2.0}, {"uid": "161d35d2d3f02cd0", "family": "compose", "template": 1, "depth": 5, "pred": "12", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "ce87e2ccc882a8d6", "family": "compose", "template": 1, "depth": 5, "pred": "30", "truth": "305", "correct": false, "edits_per_token": 2.0}, {"uid": "d91c153e794746cd", "family": "compose", "template": 1, "depth": 5, "pred": "-15", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "a0bdd324d33242e8", "family": "compose", "template": 1, "depth": 5, "pred": "5", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "ad0c5f4a911ccd20", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "89", "correct": false, "edits_per_token": 2.0}, {"uid": "0c169fbde9f9f89a", "family": "compose", "template": 1, "depth": 5, "pred": "11", "truth": "12", "correct": false, "edits_per_token": 2.0}, {"uid": "b7b297fa76c6e714", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "19", "correct": false, "edits_per_token": 2.0}, {"uid": "57c40eb561e093fb", "family": "compose", "template": 1, "depth": 5, "pred": "20", "truth": "255", "correct": false, "edits_per_token": 2.0}, {"uid": "2469c98de51b50f4", "family": "compose", "template": 1, "depth": 5, "pred": "2", "truth": "17", "correct": false, "edits_per_token": 2.0}, {"uid": "ea142ba35317bc7a", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "18", "correct": false, "edits_per_token": 2.0}, {"uid": "485dad2c0d7c8781", "family": "compose", "template": 1, "depth": 5, "pred": "-12", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "c0712e561dd7d544", "family": "compose", "template": 1, "depth": 5, "pred": "-12", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "867fd8edc71c7499", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "d8646639d466e365", "family": "compose", "template": 1, "depth": 5, "pred": "0", "truth": "-9", "correct": false, "edits_per_token": 2.0}, {"uid": "e746a5fb9df5a601", "family": "compose", "template": 1, "depth": 5, "pred": "-16", "truth": "-9", "correct": false, "edits_per_token": 2.0}, {"uid": "e2244c08a7883216", "family": "compose", "template": 1, "depth": 5, "pred": "2", "truth": "-1", "correct": false, "edits_per_token": 2.0}, {"uid": "335f485c56b8cd07", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "-3", "correct": false, "edits_per_token": 2.0}, {"uid": "e32a910f12bd4154", "family": "compose", "template": 1, "depth": 5, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "67199c1b4257d671", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "-3", "correct": false, "edits_per_token": 2.0}, {"uid": "8243fd85f87830ab", "family": "compose", "template": 1, "depth": 5, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 2.0}, {"uid": "e5bd0d5c5b641231", "family": "compose", "template": 1, "depth": 5, "pred": "30", "truth": "36", "correct": false, "edits_per_token": 2.0}, {"uid": "1eb34c67fb68ad53", "family": "compose", "template": 1, "depth": 5, "pred": "12", "truth": "67", "correct": false, "edits_per_token": 2.0}, {"uid": "c8e8110a6059c20c", "family": "compose", "template": 1, "depth": 5, "pred": "20", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "9e1f059c83a1b997", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "-8", "correct": false, "edits_per_token": 2.0}, {"uid": "3b7858f2c2c02138", "family": "compose", "template": 1, "depth": 5, "pred": "20", "truth": "35", "correct": false, "edits_per_token": 2.0}, {"uid": "47065d12ee5d095b", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "21", "correct": false, "edits_per_token": 2.0}, {"uid": "c59706dde4249b37", "family": "compose", "template": 1, "depth": 5, "pred": "7", "truth": "0", "correct": false, "edits_per_token": 2.0}, {"uid": "9eb657feba948fa4", "family": "compose", "template": 1, "depth": 5, "pred": "4", "truth": "51", "correct": false, "edits_per_token": 2.0}, {"uid": "1ef4acae76dae63b", "family": "compose", "template": 1, "depth": 5, "pred": "-4", "truth": "12", "correct": false, "edits_per_token": 2.0}, {"uid": "a6fd971352be017e", "family": "compose", "template": 1, "depth": 5, "pred": "-5", "truth": "11", "correct": false, "edits_per_token": 2.0}, {"uid": "927f02b06def3150", "family": "compose", "template": 1, "depth": 5, "pred": "12", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "fe187db12adedda5", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "28", "correct": false, "edits_per_token": 2.0}, {"uid": "a645851a332b3ab5", "family": "compose", "template": 1, "depth": 5, "pred": "10", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "cc3539a43f177d50", "family": "compose", "template": 1, "depth": 5, "pred": "7", "truth": "44", "correct": false, "edits_per_token": 2.0}, {"uid": "9a52960b9a529819", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "626758803a7b07cb", "family": "compose", "template": 1, "depth": 5, "pred": "6", "truth": "-4", "correct": false, "edits_per_token": 2.0}, {"uid": "014311836491dc15", "family": "compose", "template": 1, "depth": 5, "pred": "10", "truth": "11", "correct": false, "edits_per_token": 2.0}, {"uid": "512fc4cb803c1910", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "867753646795af94", "family": "compose", "template": 1, "depth": 5, "pred": "3", "truth": "15", "correct": false, "edits_per_token": 2.0}, {"uid": "30092adda4a81df9", "family": "compose", "template": 1, "depth": 5, "pred": "11", "truth": "-1", "correct": false, "edits_per_token": 2.0}] | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_compose_b8.json
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +[{"uid": "adb4de54695a9bb1", "family": "compose", "template": 2, "depth": 3, "pred": "-7", "truth": "-4", "correct": false, "edits_per_token": 8.0}, {"uid": "17d806db0f8e44ba", "family": "compose", "template": 2, "depth": 3, "pred": "4", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "b5ffd2ed7506d620", "family": "compose", "template": 2, "depth": 3, "pred": "1", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "4d9b3b7c91729026", "family": "compose", "template": 2, "depth": 3, "pred": "22", "truth": "112", "correct": false, "edits_per_token": 8.0}, {"uid": "de8f6fac20b4e338", "family": "compose", "template": 2, "depth": 3, "pred": "15", "truth": "134", "correct": false, "edits_per_token": 8.0}, {"uid": "d6af255f2972e0c7", "family": "compose", "template": 2, "depth": 3, "pred": "-9", "truth": "-21", "correct": false, "edits_per_token": 8.0}, {"uid": "ce09aab8fc2de9a7", "family": "compose", "template": 2, "depth": 3, "pred": "9", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "1b8a09e522f8d1cd", "family": "compose", "template": 2, "depth": 3, "pred": "0", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "23643e00862cbcbc", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "0d759e78b7643382", "family": "compose", "template": 2, "depth": 3, "pred": "1", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "075a42b41e2eb43f", "family": "compose", "template": 2, "depth": 3, "pred": "12", "truth": "105", "correct": false, "edits_per_token": 8.0}, {"uid": "099a437b780737aa", "family": "compose", "template": 2, "depth": 3, "pred": "5", "truth": "15", "correct": false, "edits_per_token": 8.0}, {"uid": "9940c6a87a6eb6da", "family": "compose", "template": 2, "depth": 3, "pred": "-3", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "f4cb14c0f01c9975", "family": "compose", "template": 2, "depth": 3, "pred": "5", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "647cb5523cd7578b", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "26", "correct": false, "edits_per_token": 8.0}, {"uid": "4a1f85b09c4659f8", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "c525b9ac8039e7a3", "family": "compose", "template": 2, "depth": 3, "pred": "13", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "5035fb98e9aebf76", "family": "compose", "template": 2, "depth": 3, "pred": "0", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "72d9e15b7c0802d9", "family": "compose", "template": 2, "depth": 3, "pred": "7", "truth": "11", "correct": false, "edits_per_token": 8.0}, {"uid": "28f5e3fea5a138fb", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "11", "correct": false, "edits_per_token": 8.0}, {"uid": "6b3e4232df264f36", "family": "compose", "template": 2, "depth": 3, "pred": "14", "truth": "146", "correct": false, "edits_per_token": 8.0}, {"uid": "8ab11bef7411cbe0", "family": "compose", "template": 2, "depth": 3, "pred": "0", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "96faa66b43a818a1", "family": "compose", "template": 2, "depth": 3, "pred": "6", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "e15f68866737a591", "family": "compose", "template": 2, "depth": 3, "pred": "-7", "truth": "-7", "correct": true, "edits_per_token": 8.0}, {"uid": "d1bb41bf830b565e", "family": "compose", "template": 2, "depth": 3, "pred": "-2", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "1d98f98efa362f80", "family": "compose", "template": 2, "depth": 3, "pred": "-4", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "665f9491c208ce8e", "family": "compose", "template": 2, "depth": 3, "pred": "14", "truth": "0", "correct": false, "edits_per_token": 8.0}, {"uid": "f54b0fe3fb897d9e", "family": "compose", "template": 2, "depth": 3, "pred": "16", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "44d407369e058248", "family": "compose", "template": 2, "depth": 3, "pred": "23", "truth": "158", "correct": false, "edits_per_token": 8.0}, {"uid": "bf7511876035c1fd", "family": "compose", "template": 2, "depth": 3, "pred": "-1", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "add68ae2a1e04601", "family": "compose", "template": 2, "depth": 3, "pred": "-3", "truth": "-3", "correct": true, "edits_per_token": 8.0}, {"uid": "ae11be33564637ac", "family": "compose", "template": 2, "depth": 3, "pred": "8", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "a1fb5e3c4fe6050c", "family": "compose", "template": 2, "depth": 3, "pred": "10", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "b7a7f826d8f54edf", "family": "compose", "template": 2, "depth": 3, "pred": "0", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "41d7e4fb4c698ad5", "family": "compose", "template": 2, "depth": 3, "pred": "-3", "truth": "-4", "correct": false, "edits_per_token": 8.0}, {"uid": "1cb2706b7f6fbb2b", "family": "compose", "template": 2, "depth": 3, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 8.0}, {"uid": "3e70547ce95699c3", "family": "compose", "template": 2, "depth": 3, "pred": "11", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "3d11c839f1453ce6", "family": "compose", "template": 2, "depth": 3, "pred": "15", "truth": "18", "correct": false, "edits_per_token": 8.0}, {"uid": "d58b04457399d716", "family": "compose", "template": 2, "depth": 3, "pred": "-6", "truth": "-14", "correct": false, "edits_per_token": 8.0}, {"uid": "c894735781f67d70", "family": "compose", "template": 2, "depth": 3, "pred": "-12", "truth": "-17", "correct": false, "edits_per_token": 8.0}, {"uid": "6510140707968d83", "family": "compose", "template": 2, "depth": 3, "pred": "-4", "truth": "0", "correct": false, "edits_per_token": 8.0}, {"uid": "1b41aaffed0ebaad", "family": "compose", "template": 2, "depth": 3, "pred": "-2", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "672b5c8f32a69ff3", "family": "compose", "template": 2, "depth": 3, "pred": "13", "truth": "25", "correct": false, "edits_per_token": 8.0}, {"uid": "237b6b3f75aad340", "family": "compose", "template": 2, "depth": 3, "pred": "3", "truth": "38", "correct": false, "edits_per_token": 8.0}, {"uid": "8b320774cd5bb130", "family": "compose", "template": 2, "depth": 3, "pred": "6", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "a4f4ff84e82ca377", "family": "compose", "template": 0, "depth": 3, "pred": "-2", "truth": "-9", "correct": false, "edits_per_token": 8.0}, {"uid": "4e40f161e5396048", "family": "compose", "template": 0, "depth": 3, "pred": "2", "truth": "17", "correct": false, "edits_per_token": 8.0}, {"uid": "8b82f59cbe3434e0", "family": "compose", "template": 0, "depth": 3, "pred": "-3", "truth": "-1", "correct": false, "edits_per_token": 8.0}, {"uid": "db71d89837843152", "family": "compose", "template": 0, "depth": 3, "pred": "3", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "0197263a81f5749b", "family": "compose", "template": 0, "depth": 3, "pred": "4", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "54ebfde1773c2e3f", "family": "compose", "template": 0, "depth": 3, "pred": "-2", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "0256ae6470dae63c", "family": "compose", "template": 0, "depth": 3, "pred": "12", "truth": "19", "correct": false, "edits_per_token": 8.0}, {"uid": "2df799eeaded8dfe", "family": "compose", "template": 0, "depth": 3, "pred": "-8", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "eba447db451d227b", "family": "compose", "template": 0, "depth": 3, "pred": "5", "truth": "12", "correct": false, "edits_per_token": 8.0}, {"uid": "39c693ae9553e888", "family": "compose", "template": 0, "depth": 3, "pred": "1", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "717c3337957264df", "family": "compose", "template": 0, "depth": 3, "pred": "-6", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "f728f34a28205d99", "family": "compose", "template": 0, "depth": 3, "pred": "5", "truth": "47", "correct": false, "edits_per_token": 8.0}, {"uid": "d2514f0dd399aff4", "family": "compose", "template": 0, "depth": 3, "pred": "-3", "truth": "17", "correct": false, "edits_per_token": 8.0}, {"uid": "529a8d6290df2519", "family": "compose", "template": 0, "depth": 3, "pred": "-7", "truth": "-10", "correct": false, "edits_per_token": 8.0}, {"uid": "d9ca9c93444f8be3", "family": "compose", "template": 0, "depth": 3, "pred": "-4", "truth": "-7", "correct": false, "edits_per_token": 8.0}, {"uid": "e0995d7984dacaee", "family": "compose", "template": 0, "depth": 3, "pred": "10", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "46861a528973f71b", "family": "compose", "template": 0, "depth": 3, "pred": "13", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "86907a1d4cfdd2aa", "family": "compose", "template": 0, "depth": 3, "pred": "2", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "4f5302ec715318f4", "family": "compose", "template": 0, "depth": 3, "pred": "12", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "5f5957f1a7eb0d1a", "family": "compose", "template": 0, "depth": 3, "pred": "-2", "truth": "-7", "correct": false, "edits_per_token": 8.0}, {"uid": "162aae75dc526041", "family": "compose", "template": 0, "depth": 3, "pred": "-12", "truth": "-1", "correct": false, "edits_per_token": 8.0}, {"uid": "74f86bfe34852fc7", "family": "compose", "template": 0, "depth": 3, "pred": "2", "truth": "0", "correct": false, "edits_per_token": 8.0}, {"uid": "03bfd6467dbb0da6", "family": "compose", "template": 0, "depth": 3, "pred": "0", "truth": "11", "correct": false, "edits_per_token": 8.0}, {"uid": "7d0b4917eaa90a1c", "family": "compose", "template": 0, "depth": 3, "pred": "9", "truth": "17", "correct": false, "edits_per_token": 8.0}, {"uid": "1afcc311fcdf51a7", "family": "compose", "template": 0, "depth": 3, "pred": "-2", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "3e29569c45e931be", "family": "compose", "template": 0, "depth": 3, "pred": "16", "truth": "28", "correct": false, "edits_per_token": 8.0}, {"uid": "203b0a6421469c30", "family": "compose", "template": 0, "depth": 3, "pred": "-4", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "22f135fea0674737", "family": "compose", "template": 0, "depth": 3, "pred": "4", "truth": "40", "correct": false, "edits_per_token": 8.0}, {"uid": "f3e9d937a0536532", "family": "compose", "template": 0, "depth": 3, "pred": "11", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "aafa1e44ca944136", "family": "compose", "template": 0, "depth": 3, "pred": "-6", "truth": "-7", "correct": false, "edits_per_token": 8.0}, {"uid": "1b976dace0d9a61f", "family": "compose", "template": 0, "depth": 3, "pred": "0", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "108e674dc2236456", "family": "compose", "template": 0, "depth": 3, "pred": "6", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "47e15055e7489308", "family": "compose", "template": 0, "depth": 3, "pred": "-7", "truth": "-2", "correct": false, "edits_per_token": 8.0}, {"uid": "becf1a65023cb394", "family": "compose", "template": 0, "depth": 3, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 8.0}, {"uid": "bb21eaadb8237e64", "family": "compose", "template": 0, "depth": 3, "pred": "0", "truth": "23", "correct": false, "edits_per_token": 8.0}, {"uid": "e8e95e457e0ff61c", "family": "compose", "template": 0, "depth": 3, "pred": "-10", "truth": "-8", "correct": false, "edits_per_token": 8.0}, {"uid": "7545d7b397cf7f92", "family": "compose", "template": 0, "depth": 3, "pred": "-6", "truth": "-15", "correct": false, "edits_per_token": 8.0}, {"uid": "b1ea2f279468da9c", "family": "compose", "template": 0, "depth": 3, "pred": "3", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "94b0faad6cfc5da0", "family": "compose", "template": 0, "depth": 3, "pred": "16", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "cd59981fea8d6b31", "family": "compose", "template": 0, "depth": 3, "pred": "0", "truth": "-10", "correct": false, "edits_per_token": 8.0}, {"uid": "fab73cdef573e6ee", "family": "compose", "template": 0, "depth": 3, "pred": "8", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "e11d203f1ac8ab57", "family": "compose", "template": 0, "depth": 3, "pred": "20", "truth": "33", "correct": false, "edits_per_token": 8.0}, {"uid": "1b2bc47aab6fb042", "family": "compose", "template": 0, "depth": 3, "pred": "24", "truth": "179", "correct": false, "edits_per_token": 8.0}, {"uid": "06fa3df1bf999d20", "family": "compose", "template": 0, "depth": 3, "pred": "10", "truth": "-13", "correct": false, "edits_per_token": 8.0}, {"uid": "3d679575c07dffbd", "family": "compose", "template": 0, "depth": 3, "pred": "10", "truth": "22", "correct": false, "edits_per_token": 8.0}, {"uid": "ebab68f0c66d94b2", "family": "compose", "template": 1, "depth": 3, "pred": "14", "truth": "18", "correct": false, "edits_per_token": 8.0}, {"uid": "4b655e468b467a1e", "family": "compose", "template": 1, "depth": 3, "pred": "7", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "6333686e41458a19", "family": "compose", "template": 1, "depth": 3, "pred": "-5", "truth": "-5", "correct": true, "edits_per_token": 8.0}, {"uid": "4904a77f2f08c9bf", "family": "compose", "template": 1, "depth": 3, "pred": "5", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "aea9f4148ec22f94", "family": "compose", "template": 1, "depth": 3, "pred": "0", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "ae8a8299a9c7e34e", "family": "compose", "template": 1, "depth": 3, "pred": "5", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "6570c0e205b5d048", "family": "compose", "template": 1, "depth": 3, "pred": "7", "truth": "33", "correct": false, "edits_per_token": 8.0}, {"uid": "cdf5dbc0e8fb28ba", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "47", "correct": false, "edits_per_token": 8.0}, {"uid": "ff6c5e131420c7fa", "family": "compose", "template": 1, "depth": 3, "pred": "-3", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "43b668257f793894", "family": "compose", "template": 1, "depth": 3, "pred": "0", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "f134711ed22583b1", "family": "compose", "template": 1, "depth": 3, "pred": "0", "truth": "17", "correct": false, "edits_per_token": 8.0}, {"uid": "d609372c6cf824c2", "family": "compose", "template": 1, "depth": 3, "pred": "12", "truth": "23", "correct": false, "edits_per_token": 8.0}, {"uid": "58b39aade95d19cc", "family": "compose", "template": 1, "depth": 3, "pred": "5", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "3abaecdd95360608", "family": "compose", "template": 1, "depth": 3, "pred": "4", "truth": "25", "correct": false, "edits_per_token": 8.0}, {"uid": "da42f0492b37e85f", "family": "compose", "template": 1, "depth": 3, "pred": "13", "truth": "19", "correct": false, "edits_per_token": 8.0}, {"uid": "ef200ce61ab718cc", "family": "compose", "template": 1, "depth": 3, "pred": "20", "truth": "117", "correct": false, "edits_per_token": 8.0}, {"uid": "b76a09fc66e8f823", "family": "compose", "template": 1, "depth": 3, "pred": "-4", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "708025be67bbf44c", "family": "compose", "template": 1, "depth": 3, "pred": "14", "truth": "14", "correct": true, "edits_per_token": 8.0}, {"uid": "75c6941e53662b69", "family": "compose", "template": 1, "depth": 3, "pred": "11", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "5742f3bbf7ae33ef", "family": "compose", "template": 1, "depth": 3, "pred": "0", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "fef642e44943ca2e", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "15", "correct": false, "edits_per_token": 8.0}, {"uid": "eab7cf6c522a4d74", "family": "compose", "template": 1, "depth": 3, "pred": "-2", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "39dc58cd1ddad214", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "42", "correct": false, "edits_per_token": 8.0}, {"uid": "c982c4b4b62b46a2", "family": "compose", "template": 1, "depth": 3, "pred": "0", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "ec9d2b1e581198de", "family": "compose", "template": 1, "depth": 3, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 8.0}, {"uid": "ed5b7032ccb677f6", "family": "compose", "template": 1, "depth": 3, "pred": "22", "truth": "92", "correct": false, "edits_per_token": 8.0}, {"uid": "e85efd8a49131fbb", "family": "compose", "template": 1, "depth": 3, "pred": "1", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "9b1937873462dae3", "family": "compose", "template": 1, "depth": 3, "pred": "8", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "92f8c134b74ab497", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "3bdbc819e2658ce6", "family": "compose", "template": 1, "depth": 3, "pred": "10", "truth": "18", "correct": false, "edits_per_token": 8.0}, {"uid": "09ab4572a068e609", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "-10", "correct": false, "edits_per_token": 8.0}, {"uid": "197d7789fd7d2320", "family": "compose", "template": 1, "depth": 3, "pred": "8", "truth": "11", "correct": false, "edits_per_token": 8.0}, {"uid": "1b288c05c50ab28a", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "f6c0c00532c7b6ec", "family": "compose", "template": 1, "depth": 3, "pred": "14", "truth": "119", "correct": false, "edits_per_token": 8.0}, {"uid": "1ad9ac7133c048a6", "family": "compose", "template": 1, "depth": 3, "pred": "6", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "4a203c954bc59a55", "family": "compose", "template": 1, "depth": 3, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "e063bbdd65e40fba", "family": "compose", "template": 1, "depth": 3, "pred": "9", "truth": "15", "correct": false, "edits_per_token": 8.0}, {"uid": "8047e5d21252c647", "family": "compose", "template": 1, "depth": 3, "pred": "9", "truth": "24", "correct": false, "edits_per_token": 8.0}, {"uid": "1a308cbea09e1820", "family": "compose", "template": 1, "depth": 3, "pred": "-2", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "ce045884547d667e", "family": "compose", "template": 1, "depth": 3, "pred": "20", "truth": "31", "correct": false, "edits_per_token": 8.0}, {"uid": "f10d7c733d8c484d", "family": "compose", "template": 1, "depth": 3, "pred": "9", "truth": "49", "correct": false, "edits_per_token": 8.0}, {"uid": "14744517e85781d6", "family": "compose", "template": 1, "depth": 3, "pred": "4", "truth": "17", "correct": false, "edits_per_token": 8.0}, {"uid": "a9c7c3ba93568f28", "family": "compose", "template": 1, "depth": 3, "pred": "2", "truth": "53", "correct": false, "edits_per_token": 8.0}, {"uid": "3447bfb6e0f058ff", "family": "compose", "template": 1, "depth": 3, "pred": "9", "truth": "-1", "correct": false, "edits_per_token": 8.0}, {"uid": "c77194dd735c892a", "family": "compose", "template": 2, "depth": 4, "pred": "16", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "f69e0f2af4310da1", "family": "compose", "template": 2, "depth": 4, "pred": "-4", "truth": "-9", "correct": false, "edits_per_token": 8.0}, {"uid": "b4288699c21fb462", "family": "compose", "template": 2, "depth": 4, "pred": "-1", "truth": "-3", "correct": false, "edits_per_token": 8.0}, {"uid": "aa53af096c007782", "family": "compose", "template": 2, "depth": 4, "pred": "17", "truth": "-1", "correct": false, "edits_per_token": 8.0}, {"uid": "7ebb4ddcb7cd4922", "family": "compose", "template": 2, "depth": 4, "pred": "9", "truth": "-3", "correct": false, "edits_per_token": 8.0}, {"uid": "27eb9dbb08f6be89", "family": "compose", "template": 2, "depth": 4, "pred": "-2", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "60a194c1a80d55c2", "family": "compose", "template": 2, "depth": 4, "pred": "2", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "ac25a665c5b8ba5c", "family": "compose", "template": 2, "depth": 4, "pred": "-2", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "c83e49945f947ee7", "family": "compose", "template": 2, "depth": 4, "pred": "14", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "6fb906a72214b74a", "family": "compose", "template": 2, "depth": 4, "pred": "5", "truth": "81", "correct": false, "edits_per_token": 8.0}, {"uid": "45e407d76733740a", "family": "compose", "template": 2, "depth": 4, "pred": "-8", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "61de5971ee5e66d8", "family": "compose", "template": 2, "depth": 4, "pred": "-17", "truth": "-1", "correct": false, "edits_per_token": 8.0}, {"uid": "c83db0f0d1e51c97", "family": "compose", "template": 2, "depth": 4, "pred": "0", "truth": "-10", "correct": false, "edits_per_token": 8.0}, {"uid": "dc1cd6108451a386", "family": "compose", "template": 2, "depth": 4, "pred": "2", "truth": "11", "correct": false, "edits_per_token": 8.0}, {"uid": "e258998e6e182ce5", "family": "compose", "template": 2, "depth": 4, "pred": "12", "truth": "-3", "correct": false, "edits_per_token": 8.0}, {"uid": "6133cf12f6a73d18", "family": "compose", "template": 2, "depth": 4, "pred": "-6", "truth": "-7", "correct": false, "edits_per_token": 8.0}, {"uid": "67454e042a0ed655", "family": "compose", "template": 2, "depth": 4, "pred": "-2", "truth": "-13", "correct": false, "edits_per_token": 8.0}, {"uid": "356b028001a44989", "family": "compose", "template": 2, "depth": 4, "pred": "-1", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "a52f72107bb33d1f", "family": "compose", "template": 2, "depth": 4, "pred": "2", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "04b5402245e39d49", "family": "compose", "template": 2, "depth": 4, "pred": "7", "truth": "19", "correct": false, "edits_per_token": 8.0}, {"uid": "4c0fb74ec5f153e8", "family": "compose", "template": 2, "depth": 4, "pred": "5", "truth": "18", "correct": false, "edits_per_token": 8.0}, {"uid": "7f06cb4a08547083", "family": "compose", "template": 2, "depth": 4, "pred": "0", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "e83018fb177d9290", "family": "compose", "template": 2, "depth": 4, "pred": "15", "truth": "-18", "correct": false, "edits_per_token": 8.0}, {"uid": "980118a6d9a0fb70", "family": "compose", "template": 2, "depth": 4, "pred": "0", "truth": "-2", "correct": false, "edits_per_token": 8.0}, {"uid": "508adf2fa51fa4b7", "family": "compose", "template": 2, "depth": 4, "pred": "8", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "44f7ab9028e2f3a6", "family": "compose", "template": 2, "depth": 4, "pred": "5", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "024cf84c9fa808a8", "family": "compose", "template": 2, "depth": 4, "pred": "11", "truth": "29", "correct": false, "edits_per_token": 8.0}, {"uid": "31a85a27753002e3", "family": "compose", "template": 2, "depth": 4, "pred": "-8", "truth": "-4", "correct": false, "edits_per_token": 8.0}, {"uid": "bddfd858fa52f0fc", "family": "compose", "template": 2, "depth": 4, "pred": "5", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "53ea3a107aa4124d", "family": "compose", "template": 2, "depth": 4, "pred": "16", "truth": "40", "correct": false, "edits_per_token": 8.0}, {"uid": "fb41d24e9b524dbf", "family": "compose", "template": 2, "depth": 4, "pred": "-4", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "daacbc2f9cc02c6e", "family": "compose", "template": 2, "depth": 4, "pred": "5", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "ba48e47b18027d09", "family": "compose", "template": 2, "depth": 4, "pred": "1", "truth": "-9", "correct": false, "edits_per_token": 8.0}, {"uid": "f8dd559e23329aa6", "family": "compose", "template": 2, "depth": 4, "pred": "-2", "truth": "12", "correct": false, "edits_per_token": 8.0}, {"uid": "89e113217d4e8dde", "family": "compose", "template": 2, "depth": 4, "pred": "-6", "truth": "-1", "correct": false, "edits_per_token": 8.0}, {"uid": "f63237735ba6b2b2", "family": "compose", "template": 2, "depth": 4, "pred": "17", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "c5ed770de008b78f", "family": "compose", "template": 2, "depth": 4, "pred": "2", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "d7b169bfa088fc4b", "family": "compose", "template": 2, "depth": 4, "pred": "-18", "truth": "-15", "correct": false, "edits_per_token": 8.0}, {"uid": "5e099825a8eb3846", "family": "compose", "template": 2, "depth": 4, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 8.0}, {"uid": "df5aa3e5e3ab531f", "family": "compose", "template": 2, "depth": 4, "pred": "3", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "a7cb8b36fbb34d80", "family": "compose", "template": 2, "depth": 4, "pred": "-2", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "ca89ac23e031b4d9", "family": "compose", "template": 2, "depth": 4, "pred": "17", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "57e8a444a52e9659", "family": "compose", "template": 2, "depth": 4, "pred": "5", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "019d77a6be125d5f", "family": "compose", "template": 2, "depth": 4, "pred": "1", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "4e31b04ed838d8a4", "family": "compose", "template": 0, "depth": 4, "pred": "8", "truth": "179", "correct": false, "edits_per_token": 8.0}, {"uid": "cd4ad97c104f4ccc", "family": "compose", "template": 0, "depth": 4, "pred": "-12", "truth": "-25", "correct": false, "edits_per_token": 8.0}, {"uid": "67a8b958f2783833", "family": "compose", "template": 0, "depth": 4, "pred": "0", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "c08c4460a5101f62", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "-1", "correct": false, "edits_per_token": 8.0}, {"uid": "393fb7f6b6570266", "family": "compose", "template": 0, "depth": 4, "pred": "-5", "truth": "11", "correct": false, "edits_per_token": 8.0}, {"uid": "c52da7a400507ead", "family": "compose", "template": 0, "depth": 4, "pred": "0", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "5cac6a518c45b812", "family": "compose", "template": 0, "depth": 4, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 8.0}, {"uid": "0bf5c23a7131ed77", "family": "compose", "template": 0, "depth": 4, "pred": "5", "truth": "0", "correct": false, "edits_per_token": 8.0}, {"uid": "25efb341025a1c02", "family": "compose", "template": 0, "depth": 4, "pred": "-2", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "4a79def086294a3a", "family": "compose", "template": 0, "depth": 4, "pred": "6", "truth": "-3", "correct": false, "edits_per_token": 8.0}, {"uid": "faf455eb5168841e", "family": "compose", "template": 0, "depth": 4, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 8.0}, {"uid": "b746ed0dc37016b2", "family": "compose", "template": 0, "depth": 4, "pred": "3", "truth": "-2", "correct": false, "edits_per_token": 8.0}, {"uid": "4a36f3514b13957d", "family": "compose", "template": 0, "depth": 4, "pred": "15", "truth": "113", "correct": false, "edits_per_token": 8.0}, {"uid": "17199e564f6e3262", "family": "compose", "template": 0, "depth": 4, "pred": "11", "truth": "102", "correct": false, "edits_per_token": 8.0}, {"uid": "e677a33994695496", "family": "compose", "template": 0, "depth": 4, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "53dac0369fd28ac0", "family": "compose", "template": 0, "depth": 4, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "c2cfa1bf34bc1d6d", "family": "compose", "template": 0, "depth": 4, "pred": "-8", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "8fedc1effa6611a3", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "51", "correct": false, "edits_per_token": 8.0}, {"uid": "0e005022cc77f8c6", "family": "compose", "template": 0, "depth": 4, "pred": "-15", "truth": "-13", "correct": false, "edits_per_token": 8.0}, {"uid": "26405d91a56f7855", "family": "compose", "template": 0, "depth": 4, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "4f7dea5defe99ded", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "-1", "correct": false, "edits_per_token": 8.0}, {"uid": "49c5a431f2ef6c26", "family": "compose", "template": 0, "depth": 4, "pred": "13", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "aaf95ea7d25fa328", "family": "compose", "template": 0, "depth": 4, "pred": "7", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "a2efa1cc868910a2", "family": "compose", "template": 0, "depth": 4, "pred": "-6", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "d8797d338a3b92a8", "family": "compose", "template": 0, "depth": 4, "pred": "-18", "truth": "-10", "correct": false, "edits_per_token": 8.0}, {"uid": "819700d90a033274", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "6b79b778c94c3340", "family": "compose", "template": 0, "depth": 4, "pred": "-8", "truth": "-3", "correct": false, "edits_per_token": 8.0}, {"uid": "f46dd9ba49c5f9e5", "family": "compose", "template": 0, "depth": 4, "pred": "-3", "truth": "-10", "correct": false, "edits_per_token": 8.0}, {"uid": "6297fac714f70477", "family": "compose", "template": 0, "depth": 4, "pred": "4", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "5bc9e1d49fcaabba", "family": "compose", "template": 0, "depth": 4, "pred": "2", "truth": "30", "correct": false, "edits_per_token": 8.0}, {"uid": "095e7a1076d7cbe1", "family": "compose", "template": 0, "depth": 4, "pred": "10", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "02e62b8d24c38e10", "family": "compose", "template": 0, "depth": 4, "pred": "5", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "e92f9a90aaa752ab", "family": "compose", "template": 0, "depth": 4, "pred": "-5", "truth": "-12", "correct": false, "edits_per_token": 8.0}, {"uid": "b8f1efdf78952406", "family": "compose", "template": 0, "depth": 4, "pred": "-16", "truth": "-17", "correct": false, "edits_per_token": 8.0}, {"uid": "ac30d19ea27ec58e", "family": "compose", "template": 0, "depth": 4, "pred": "6", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "d754429b5b350068", "family": "compose", "template": 0, "depth": 4, "pred": "0", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "1ea10fa59288e773", "family": "compose", "template": 0, "depth": 4, "pred": "7", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "325032d22f88969d", "family": "compose", "template": 0, "depth": 4, "pred": "11", "truth": "-8", "correct": false, "edits_per_token": 8.0}, {"uid": "ce5754d69875787b", "family": "compose", "template": 0, "depth": 4, "pred": "-8", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "d221bad35e9a7253", "family": "compose", "template": 0, "depth": 4, "pred": "3", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "3702fc94bbb38ee7", "family": "compose", "template": 0, "depth": 4, "pred": "11", "truth": "23", "correct": false, "edits_per_token": 8.0}, {"uid": "4acd5466e30298d3", "family": "compose", "template": 0, "depth": 4, "pred": "20", "truth": "35", "correct": false, "edits_per_token": 8.0}, {"uid": "c5c331dd1513732d", "family": "compose", "template": 0, "depth": 4, "pred": "10", "truth": "21", "correct": false, "edits_per_token": 8.0}, {"uid": "de6345b12fdb1b98", "family": "compose", "template": 0, "depth": 4, "pred": "20", "truth": "32", "correct": false, "edits_per_token": 8.0}, {"uid": "92ccd36e2de558cf", "family": "compose", "template": 1, "depth": 4, "pred": "10", "truth": "39", "correct": false, "edits_per_token": 8.0}, {"uid": "50faa50b3396ffae", "family": "compose", "template": 1, "depth": 4, "pred": "7", "truth": "12", "correct": false, "edits_per_token": 8.0}, {"uid": "29dee9386795c180", "family": "compose", "template": 1, "depth": 4, "pred": "-6", "truth": "18", "correct": false, "edits_per_token": 8.0}, {"uid": "238c0bc8c05a35a2", "family": "compose", "template": 1, "depth": 4, "pred": "-2", "truth": "0", "correct": false, "edits_per_token": 8.0}, {"uid": "38c93e00c30f778c", "family": "compose", "template": 1, "depth": 4, "pred": "-25", "truth": "-17", "correct": false, "edits_per_token": 8.0}, {"uid": "78e560fe43169af8", "family": "compose", "template": 1, "depth": 4, "pred": "20", "truth": "-4", "correct": false, "edits_per_token": 8.0}, {"uid": "b842bbdfb385a2c8", "family": "compose", "template": 1, "depth": 4, "pred": "0", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "ca2e93da50248f99", "family": "compose", "template": 1, "depth": 4, "pred": "2", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "404b290af2697cc1", "family": "compose", "template": 1, "depth": 4, "pred": "15", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "80a7754cfb080d93", "family": "compose", "template": 1, "depth": 4, "pred": "-7", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "63602722310a02d6", "family": "compose", "template": 1, "depth": 4, "pred": "10", "truth": "42", "correct": false, "edits_per_token": 8.0}, {"uid": "34b4950e5d8a0b1e", "family": "compose", "template": 1, "depth": 4, "pred": "0", "truth": "-1", "correct": false, "edits_per_token": 8.0}, {"uid": "29708655a0f9fa6d", "family": "compose", "template": 1, "depth": 4, "pred": "-3", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "7f8cff76916c511e", "family": "compose", "template": 1, "depth": 4, "pred": "2", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "985ece2df7f3c619", "family": "compose", "template": 1, "depth": 4, "pred": "20", "truth": "146", "correct": false, "edits_per_token": 8.0}, {"uid": "1358a81db4f24e15", "family": "compose", "template": 1, "depth": 4, "pred": "20", "truth": "116", "correct": false, "edits_per_token": 8.0}, {"uid": "48031d8c67e39d8b", "family": "compose", "template": 1, "depth": 4, "pred": "-14", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "a6147eea1031f132", "family": "compose", "template": 1, "depth": 4, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "6582654868e1e4a0", "family": "compose", "template": 1, "depth": 4, "pred": "9", "truth": "115", "correct": false, "edits_per_token": 8.0}, {"uid": "3587b0aaef8cf3e1", "family": "compose", "template": 1, "depth": 4, "pred": "8", "truth": "160", "correct": false, "edits_per_token": 8.0}, {"uid": "2cce12ccba5151de", "family": "compose", "template": 1, "depth": 4, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "5d28412b23afa91f", "family": "compose", "template": 1, "depth": 4, "pred": "12", "truth": "50", "correct": false, "edits_per_token": 8.0}, {"uid": "b021356c73afb769", "family": "compose", "template": 1, "depth": 4, "pred": "-6", "truth": "-12", "correct": false, "edits_per_token": 8.0}, {"uid": "98b7fa299d84ef2e", "family": "compose", "template": 1, "depth": 4, "pred": "-6", "truth": "-15", "correct": false, "edits_per_token": 8.0}, {"uid": "3d8b4948bd221577", "family": "compose", "template": 1, "depth": 4, "pred": "-3", "truth": "-12", "correct": false, "edits_per_token": 8.0}, {"uid": "923ce5295838ed05", "family": "compose", "template": 1, "depth": 4, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 8.0}, {"uid": "06fef04b9fe5f342", "family": "compose", "template": 1, "depth": 4, "pred": "-8", "truth": "-16", "correct": false, "edits_per_token": 8.0}, {"uid": "4a350417dc3523cc", "family": "compose", "template": 1, "depth": 4, "pred": "-5", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "fc87f1aa022adcc2", "family": "compose", "template": 1, "depth": 4, "pred": "6", "truth": "-11", "correct": false, "edits_per_token": 8.0}, {"uid": "2e19bd8266852d8b", "family": "compose", "template": 1, "depth": 4, "pred": "6", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "abdac9d2fc00886e", "family": "compose", "template": 1, "depth": 4, "pred": "-4", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "83101cc76f2de188", "family": "compose", "template": 1, "depth": 4, "pred": "-15", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "2d00a93eb0c1b20a", "family": "compose", "template": 1, "depth": 4, "pred": "-3", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "545514d052c79cda", "family": "compose", "template": 1, "depth": 4, "pred": "4", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "8f05f418121b41a7", "family": "compose", "template": 1, "depth": 4, "pred": "13", "truth": "82", "correct": false, "edits_per_token": 8.0}, {"uid": "09dd937e1ef4b6c0", "family": "compose", "template": 1, "depth": 4, "pred": "5", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "c21f46d5ca49a0ce", "family": "compose", "template": 1, "depth": 4, "pred": "22", "truth": "105", "correct": false, "edits_per_token": 8.0}, {"uid": "4686b81018c38653", "family": "compose", "template": 1, "depth": 4, "pred": "9", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "ac04d2d2a5b89df8", "family": "compose", "template": 1, "depth": 4, "pred": "3", "truth": "-1", "correct": false, "edits_per_token": 8.0}, {"uid": "2916a34b7a3c5497", "family": "compose", "template": 1, "depth": 4, "pred": "-3", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "8c9bbb68ef0ff75b", "family": "compose", "template": 1, "depth": 4, "pred": "-3", "truth": "-16", "correct": false, "edits_per_token": 8.0}, {"uid": "948b8c326fa8c94b", "family": "compose", "template": 1, "depth": 4, "pred": "7", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "9608a0a9a3cf4853", "family": "compose", "template": 1, "depth": 4, "pred": "5", "truth": "17", "correct": false, "edits_per_token": 8.0}, {"uid": "52c595795ab95e82", "family": "compose", "template": 1, "depth": 4, "pred": "14", "truth": "22", "correct": false, "edits_per_token": 8.0}, {"uid": "b72fcba0a1ab05d0", "family": "compose", "template": 1, "depth": 4, "pred": "12", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "bee3aabc6bd5223a", "family": "compose", "template": 2, "depth": 5, "pred": "20", "truth": "28", "correct": false, "edits_per_token": 8.0}, {"uid": "8991d7fdb159825d", "family": "compose", "template": 2, "depth": 5, "pred": "6", "truth": "17", "correct": false, "edits_per_token": 8.0}, {"uid": "f81d9500784660dd", "family": "compose", "template": 2, "depth": 5, "pred": "4", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "df2ebcbe11e34787", "family": "compose", "template": 2, "depth": 5, "pred": "-2", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "c020e029edb14f57", "family": "compose", "template": 2, "depth": 5, "pred": "11", "truth": "-14", "correct": false, "edits_per_token": 8.0}, {"uid": "61ab1fe0f9bf8174", "family": "compose", "template": 2, "depth": 5, "pred": "-3", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "2c3b4386fc0cd5e2", "family": "compose", "template": 2, "depth": 5, "pred": "6", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "3e451812e17f0f49", "family": "compose", "template": 2, "depth": 5, "pred": "7", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "5dcf4e4893b12b48", "family": "compose", "template": 2, "depth": 5, "pred": "9", "truth": "27", "correct": false, "edits_per_token": 8.0}, {"uid": "ade1e9e1062031d5", "family": "compose", "template": 2, "depth": 5, "pred": "15", "truth": "26", "correct": false, "edits_per_token": 8.0}, {"uid": "d476cf4369ac7e22", "family": "compose", "template": 2, "depth": 5, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 8.0}, {"uid": "b3304b4efd082e9a", "family": "compose", "template": 2, "depth": 5, "pred": "0", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "e5288726427262e5", "family": "compose", "template": 2, "depth": 5, "pred": "14", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "894fcb5670fbca62", "family": "compose", "template": 2, "depth": 5, "pred": "13", "truth": "18", "correct": false, "edits_per_token": 8.0}, {"uid": "51129c69cded6af2", "family": "compose", "template": 2, "depth": 5, "pred": "3", "truth": "19", "correct": false, "edits_per_token": 8.0}, {"uid": "dbcebfe32b261b36", "family": "compose", "template": 2, "depth": 5, "pred": "26", "truth": "406", "correct": false, "edits_per_token": 8.0}, {"uid": "f6d12b0297c1c550", "family": "compose", "template": 2, "depth": 5, "pred": "-12", "truth": "-2", "correct": false, "edits_per_token": 8.0}, {"uid": "76c08480dce34c43", "family": "compose", "template": 2, "depth": 5, "pred": "5", "truth": "39", "correct": false, "edits_per_token": 8.0}, {"uid": "baea2ba74c196937", "family": "compose", "template": 2, "depth": 5, "pred": "-16", "truth": "-30", "correct": false, "edits_per_token": 8.0}, {"uid": "3f59a13b0dc3dc17", "family": "compose", "template": 2, "depth": 5, "pred": "4", "truth": "17", "correct": false, "edits_per_token": 8.0}, {"uid": "4caa849dad301058", "family": "compose", "template": 2, "depth": 5, "pred": "18", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "b348283e0f65231e", "family": "compose", "template": 2, "depth": 5, "pred": "-2", "truth": "-13", "correct": false, "edits_per_token": 8.0}, {"uid": "560fee240f31263a", "family": "compose", "template": 2, "depth": 5, "pred": "6", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "08dc517782bc8fef", "family": "compose", "template": 2, "depth": 5, "pred": "-8", "truth": "-9", "correct": false, "edits_per_token": 8.0}, {"uid": "28600ff91708289c", "family": "compose", "template": 2, "depth": 5, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "3f5eac859ba6fa37", "family": "compose", "template": 2, "depth": 5, "pred": "-6", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "9264073cc6ce8083", "family": "compose", "template": 2, "depth": 5, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 8.0}, {"uid": "958d4755d009e098", "family": "compose", "template": 2, "depth": 5, "pred": "-5", "truth": "-10", "correct": false, "edits_per_token": 8.0}, {"uid": "c1256114e43d3770", "family": "compose", "template": 2, "depth": 5, "pred": "4", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "99b6163909b9e9fd", "family": "compose", "template": 2, "depth": 5, "pred": "20", "truth": "27", "correct": false, "edits_per_token": 8.0}, {"uid": "dafeef05c5ec8fc4", "family": "compose", "template": 2, "depth": 5, "pred": "13", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "77682fc13ea83713", "family": "compose", "template": 2, "depth": 5, "pred": "5", "truth": "45", "correct": false, "edits_per_token": 8.0}, {"uid": "04de1db4dc636bcc", "family": "compose", "template": 2, "depth": 5, "pred": "14", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "c712c5b393de2310", "family": "compose", "template": 2, "depth": 5, "pred": "6", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "29e13e67d2c97073", "family": "compose", "template": 2, "depth": 5, "pred": "0", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "86397ff87e919200", "family": "compose", "template": 2, "depth": 5, "pred": "-16", "truth": "18", "correct": false, "edits_per_token": 8.0}, {"uid": "5837733b6c9e24b1", "family": "compose", "template": 2, "depth": 5, "pred": "-5", "truth": "-7", "correct": false, "edits_per_token": 8.0}, {"uid": "394bf97876be6993", "family": "compose", "template": 2, "depth": 5, "pred": "14", "truth": "93", "correct": false, "edits_per_token": 8.0}, {"uid": "76b2875e84ee2d00", "family": "compose", "template": 2, "depth": 5, "pred": "16", "truth": "148", "correct": false, "edits_per_token": 8.0}, {"uid": "6a14cc56e1d5eaac", "family": "compose", "template": 2, "depth": 5, "pred": "13", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "d1bcc6c12488902d", "family": "compose", "template": 2, "depth": 5, "pred": "20", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "2cc159a55fb29fd7", "family": "compose", "template": 2, "depth": 5, "pred": "9", "truth": "-3", "correct": false, "edits_per_token": 8.0}, {"uid": "8e8cb310090cdf9f", "family": "compose", "template": 2, "depth": 5, "pred": "5", "truth": "22", "correct": false, "edits_per_token": 8.0}, {"uid": "bd894beab682cc7b", "family": "compose", "template": 2, "depth": 5, "pred": "3", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "20abae29de56cfbc", "family": "compose", "template": 0, "depth": 5, "pred": "20", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "1bffd281778527b5", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "76a3e1969e0dd187", "family": "compose", "template": 0, "depth": 5, "pred": "-16", "truth": "112", "correct": false, "edits_per_token": 8.0}, {"uid": "135b4dd9880a7711", "family": "compose", "template": 0, "depth": 5, "pred": "11", "truth": "119", "correct": false, "edits_per_token": 8.0}, {"uid": "98fe8dd4b77530ae", "family": "compose", "template": 0, "depth": 5, "pred": "6", "truth": "-2", "correct": false, "edits_per_token": 8.0}, {"uid": "ab504be8f60f8773", "family": "compose", "template": 0, "depth": 5, "pred": "20", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "ae8a7e76edd9b1ac", "family": "compose", "template": 0, "depth": 5, "pred": "-21", "truth": "-45", "correct": false, "edits_per_token": 8.0}, {"uid": "57d846cf5b774601", "family": "compose", "template": 0, "depth": 5, "pred": "8", "truth": "153", "correct": false, "edits_per_token": 8.0}, {"uid": "9291a187e465d9c8", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "40", "correct": false, "edits_per_token": 8.0}, {"uid": "3027f279111dbcae", "family": "compose", "template": 0, "depth": 5, "pred": "-4", "truth": "-17", "correct": false, "edits_per_token": 8.0}, {"uid": "184ff66abbf56a5c", "family": "compose", "template": 0, "depth": 5, "pred": "0", "truth": "-3", "correct": false, "edits_per_token": 8.0}, {"uid": "eb1a5d81063ecbcb", "family": "compose", "template": 0, "depth": 5, "pred": "24", "truth": "35", "correct": false, "edits_per_token": 8.0}, {"uid": "4746ec994270639a", "family": "compose", "template": 0, "depth": 5, "pred": "15", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "727e5ef3770421a6", "family": "compose", "template": 0, "depth": 5, "pred": "9", "truth": "127", "correct": false, "edits_per_token": 8.0}, {"uid": "20e006d873dd1edb", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "17", "correct": false, "edits_per_token": 8.0}, {"uid": "8e32ed5b07934152", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "91fc4c488e9ced3d", "family": "compose", "template": 0, "depth": 5, "pred": "7", "truth": "-13", "correct": false, "edits_per_token": 8.0}, {"uid": "4984028c93289a35", "family": "compose", "template": 0, "depth": 5, "pred": "-18", "truth": "35", "correct": false, "edits_per_token": 8.0}, {"uid": "55a3205999715b20", "family": "compose", "template": 0, "depth": 5, "pred": "7", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "18779c7d7329f224", "family": "compose", "template": 0, "depth": 5, "pred": "11", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "cec5a99eca1756c7", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "a524d85a18a67e49", "family": "compose", "template": 0, "depth": 5, "pred": "-3", "truth": "-7", "correct": false, "edits_per_token": 8.0}, {"uid": "ffd488cb6bea85ff", "family": "compose", "template": 0, "depth": 5, "pred": "23", "truth": "95", "correct": false, "edits_per_token": 8.0}, {"uid": "a6bd916c42aee6c4", "family": "compose", "template": 0, "depth": 5, "pred": "-3", "truth": "-13", "correct": false, "edits_per_token": 8.0}, {"uid": "3c867544528e8d53", "family": "compose", "template": 0, "depth": 5, "pred": "5", "truth": "79", "correct": false, "edits_per_token": 8.0}, {"uid": "356c3a7de1144ea9", "family": "compose", "template": 0, "depth": 5, "pred": "4", "truth": "-4", "correct": false, "edits_per_token": 8.0}, {"uid": "4abed743053a2033", "family": "compose", "template": 0, "depth": 5, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "4a2f3f7d91c213de", "family": "compose", "template": 0, "depth": 5, "pred": "8", "truth": "97", "correct": false, "edits_per_token": 8.0}, {"uid": "51aea898105d4c43", "family": "compose", "template": 0, "depth": 5, "pred": "-3", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "f374d4ad191ded59", "family": "compose", "template": 0, "depth": 5, "pred": "7", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "2c485719d8098bbe", "family": "compose", "template": 0, "depth": 5, "pred": "-16", "truth": "-15", "correct": false, "edits_per_token": 8.0}, {"uid": "95343d962b76844d", "family": "compose", "template": 0, "depth": 5, "pred": "7", "truth": "87", "correct": false, "edits_per_token": 8.0}, {"uid": "86e8356113bdbd04", "family": "compose", "template": 0, "depth": 5, "pred": "4", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "5704b25879d5b987", "family": "compose", "template": 0, "depth": 5, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "904694feea1c0838", "family": "compose", "template": 0, "depth": 5, "pred": "10", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "8d1eaafea5c8860c", "family": "compose", "template": 0, "depth": 5, "pred": "3", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "697800c219134d74", "family": "compose", "template": 0, "depth": 5, "pred": "9", "truth": "92", "correct": false, "edits_per_token": 8.0}, {"uid": "c92c4e3ef89e0f14", "family": "compose", "template": 0, "depth": 5, "pred": "-2", "truth": "-14", "correct": false, "edits_per_token": 8.0}, {"uid": "d07423f5eb5e3551", "family": "compose", "template": 0, "depth": 5, "pred": "-5", "truth": "15", "correct": false, "edits_per_token": 8.0}, {"uid": "8477f9ed7d2a0ab7", "family": "compose", "template": 0, "depth": 5, "pred": "-21", "truth": "-26", "correct": false, "edits_per_token": 8.0}, {"uid": "d0ca46e01def43f3", "family": "compose", "template": 0, "depth": 5, "pred": "13", "truth": "76", "correct": false, "edits_per_token": 8.0}, {"uid": "c861a4351ff59cc1", "family": "compose", "template": 0, "depth": 5, "pred": "12", "truth": "11", "correct": false, "edits_per_token": 8.0}, {"uid": "15fc685f62ca87b2", "family": "compose", "template": 0, "depth": 5, "pred": "16", "truth": "117", "correct": false, "edits_per_token": 8.0}, {"uid": "6f28dc9656cc50a0", "family": "compose", "template": 0, "depth": 5, "pred": "10", "truth": "28", "correct": false, "edits_per_token": 8.0}, {"uid": "26ab4ae08da814b0", "family": "compose", "template": 0, "depth": 5, "pred": "-6", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "24d6f7c2be67dbc5", "family": "compose", "template": 1, "depth": 5, "pred": "30", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "5c6fa527441afc1a", "family": "compose", "template": 1, "depth": 5, "pred": "-4", "truth": "47", "correct": false, "edits_per_token": 8.0}, {"uid": "45106f9401657f9f", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "113e562bce7bec31", "family": "compose", "template": 1, "depth": 5, "pred": "0", "truth": "-3", "correct": false, "edits_per_token": 8.0}, {"uid": "161d35d2d3f02cd0", "family": "compose", "template": 1, "depth": 5, "pred": "12", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "ce87e2ccc882a8d6", "family": "compose", "template": 1, "depth": 5, "pred": "30", "truth": "305", "correct": false, "edits_per_token": 8.0}, {"uid": "d91c153e794746cd", "family": "compose", "template": 1, "depth": 5, "pred": "-12", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "a0bdd324d33242e8", "family": "compose", "template": 1, "depth": 5, "pred": "5", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "ad0c5f4a911ccd20", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "89", "correct": false, "edits_per_token": 8.0}, {"uid": "0c169fbde9f9f89a", "family": "compose", "template": 1, "depth": 5, "pred": "11", "truth": "12", "correct": false, "edits_per_token": 8.0}, {"uid": "b7b297fa76c6e714", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "19", "correct": false, "edits_per_token": 8.0}, {"uid": "57c40eb561e093fb", "family": "compose", "template": 1, "depth": 5, "pred": "20", "truth": "255", "correct": false, "edits_per_token": 8.0}, {"uid": "2469c98de51b50f4", "family": "compose", "template": 1, "depth": 5, "pred": "2", "truth": "17", "correct": false, "edits_per_token": 8.0}, {"uid": "ea142ba35317bc7a", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "18", "correct": false, "edits_per_token": 8.0}, {"uid": "485dad2c0d7c8781", "family": "compose", "template": 1, "depth": 5, "pred": "-12", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "c0712e561dd7d544", "family": "compose", "template": 1, "depth": 5, "pred": "-12", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "867fd8edc71c7499", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "d8646639d466e365", "family": "compose", "template": 1, "depth": 5, "pred": "0", "truth": "-9", "correct": false, "edits_per_token": 8.0}, {"uid": "e746a5fb9df5a601", "family": "compose", "template": 1, "depth": 5, "pred": "-17", "truth": "-9", "correct": false, "edits_per_token": 8.0}, {"uid": "e2244c08a7883216", "family": "compose", "template": 1, "depth": 5, "pred": "2", "truth": "-1", "correct": false, "edits_per_token": 8.0}, {"uid": "335f485c56b8cd07", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "-3", "correct": false, "edits_per_token": 8.0}, {"uid": "e32a910f12bd4154", "family": "compose", "template": 1, "depth": 5, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "67199c1b4257d671", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "-3", "correct": false, "edits_per_token": 8.0}, {"uid": "8243fd85f87830ab", "family": "compose", "template": 1, "depth": 5, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 8.0}, {"uid": "e5bd0d5c5b641231", "family": "compose", "template": 1, "depth": 5, "pred": "30", "truth": "36", "correct": false, "edits_per_token": 8.0}, {"uid": "1eb34c67fb68ad53", "family": "compose", "template": 1, "depth": 5, "pred": "11", "truth": "67", "correct": false, "edits_per_token": 8.0}, {"uid": "c8e8110a6059c20c", "family": "compose", "template": 1, "depth": 5, "pred": "20", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "9e1f059c83a1b997", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "-8", "correct": false, "edits_per_token": 8.0}, {"uid": "3b7858f2c2c02138", "family": "compose", "template": 1, "depth": 5, "pred": "20", "truth": "35", "correct": false, "edits_per_token": 8.0}, {"uid": "47065d12ee5d095b", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "21", "correct": false, "edits_per_token": 8.0}, {"uid": "c59706dde4249b37", "family": "compose", "template": 1, "depth": 5, "pred": "7", "truth": "0", "correct": false, "edits_per_token": 8.0}, {"uid": "9eb657feba948fa4", "family": "compose", "template": 1, "depth": 5, "pred": "4", "truth": "51", "correct": false, "edits_per_token": 8.0}, {"uid": "1ef4acae76dae63b", "family": "compose", "template": 1, "depth": 5, "pred": "-4", "truth": "12", "correct": false, "edits_per_token": 8.0}, {"uid": "a6fd971352be017e", "family": "compose", "template": 1, "depth": 5, "pred": "-5", "truth": "11", "correct": false, "edits_per_token": 8.0}, {"uid": "927f02b06def3150", "family": "compose", "template": 1, "depth": 5, "pred": "11", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "fe187db12adedda5", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "28", "correct": false, "edits_per_token": 8.0}, {"uid": "a645851a332b3ab5", "family": "compose", "template": 1, "depth": 5, "pred": "10", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "cc3539a43f177d50", "family": "compose", "template": 1, "depth": 5, "pred": "7", "truth": "44", "correct": false, "edits_per_token": 8.0}, {"uid": "9a52960b9a529819", "family": "compose", "template": 1, "depth": 5, "pred": "14", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "626758803a7b07cb", "family": "compose", "template": 1, "depth": 5, "pred": "6", "truth": "-4", "correct": false, "edits_per_token": 8.0}, {"uid": "014311836491dc15", "family": "compose", "template": 1, "depth": 5, "pred": "9", "truth": "11", "correct": false, "edits_per_token": 8.0}, {"uid": "512fc4cb803c1910", "family": "compose", "template": 1, "depth": 5, "pred": "-2", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "867753646795af94", "family": "compose", "template": 1, "depth": 5, "pred": "3", "truth": "15", "correct": false, "edits_per_token": 8.0}, {"uid": "30092adda4a81df9", "family": "compose", "template": 1, "depth": 5, "pred": "11", "truth": "-1", "correct": false, "edits_per_token": 8.0}] | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_depth_b0.json
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +[{"uid": "05a9ab128b8f795a", "family": "arith", "template": 0, "depth": 5, "pred": "304", "truth": "206", "correct": false, "edits_per_token": 0.0}, {"uid": "ce77dc7f6121c65c", "family": "arith", "template": 0, "depth": 5, "pred": "3175", "truth": "3150", "correct": false, "edits_per_token": 0.0}, {"uid": "6d86083e9a296d70", "family": "arith", "template": 0, "depth": 5, "pred": "611", "truth": "467", "correct": false, "edits_per_token": 0.0}, {"uid": "bbbc075e9ed2d83b", "family": "arith", "template": 0, "depth": 5, "pred": "16880", "truth": "13248", "correct": false, "edits_per_token": 0.0}, {"uid": "9cc4e3fd8e271e9e", "family": "arith", "template": 0, "depth": 5, "pred": "1650", "truth": "1800", "correct": false, "edits_per_token": 0.0}, {"uid": "d93003ac9ddbe92e", "family": "arith", "template": 1, "depth": 5, "pred": "-12", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "0429a41c21791b48", "family": "arith", "template": 1, "depth": 5, "pred": "516", "truth": "485", "correct": false, "edits_per_token": 0.0}, {"uid": "252188905e760471", "family": "arith", "template": 1, "depth": 5, "pred": "20", "truth": "43", "correct": false, "edits_per_token": 0.0}, {"uid": "9d3b6d51fa8f268d", "family": "arith", "template": 1, "depth": 5, "pred": "-300", "truth": "-320", "correct": false, "edits_per_token": 0.0}, {"uid": "c38dbe1ef3f315bb", "family": "arith", "template": 0, "depth": 5, "pred": "-58", "truth": "-63", "correct": false, "edits_per_token": 0.0}, {"uid": "4ee8682a5c7cde40", "family": "arith", "template": 0, "depth": 5, "pred": "-10", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "213e53daae3f0053", "family": "arith", "template": 0, "depth": 5, "pred": "1688", "truth": "1113", "correct": false, "edits_per_token": 0.0}, {"uid": "8a499c6f7286285c", "family": "arith", "template": 1, "depth": 5, "pred": "-41", "truth": "-47", "correct": false, "edits_per_token": 0.0}, {"uid": "c7cc1d12232437f5", "family": "arith", "template": 1, "depth": 5, "pred": "307", "truth": "411", "correct": false, "edits_per_token": 0.0}, {"uid": "72633b6ff3769cb5", "family": "arith", "template": 1, "depth": 5, "pred": "34", "truth": "-3", "correct": false, "edits_per_token": 0.0}, {"uid": "50c2617c879f93b6", "family": "arith", "template": 1, "depth": 5, "pred": "148", "truth": "814", "correct": false, "edits_per_token": 0.0}, {"uid": "b2bc5413c8fada82", "family": "arith", "template": 1, "depth": 5, "pred": "519", "truth": "686", "correct": false, "edits_per_token": 0.0}, {"uid": "a8e4058450eb65d2", "family": "arith", "template": 1, "depth": 5, "pred": "108", "truth": "140", "correct": false, "edits_per_token": 0.0}, {"uid": "c709c22eb9084b6b", "family": "arith", "template": 0, "depth": 5, "pred": "-51", "truth": "-39", "correct": false, "edits_per_token": 0.0}, {"uid": "58498d59ed0e6ec5", "family": "arith", "template": 0, "depth": 5, "pred": "380", "truth": "387", "correct": false, "edits_per_token": 0.0}, {"uid": "d79b98b965dbc325", "family": "arith", "template": 0, "depth": 5, "pred": "48", "truth": "55", "correct": false, "edits_per_token": 0.0}, {"uid": "ac935e06e7c21bd6", "family": "arith", "template": 0, "depth": 5, "pred": "80", "truth": "54", "correct": false, "edits_per_token": 0.0}, {"uid": "32d9a5b74df38a67", "family": "arith", "template": 0, "depth": 5, "pred": "288", "truth": "303", "correct": false, "edits_per_token": 0.0}, {"uid": "bea3727c59f821e2", "family": "arith", "template": 0, "depth": 5, "pred": "68", "truth": "71", "correct": false, "edits_per_token": 0.0}, {"uid": "3d7cf92565334823", "family": "arith", "template": 1, "depth": 5, "pred": "-14", "truth": "-15", "correct": false, "edits_per_token": 0.0}, {"uid": "34ad0dad384435be", "family": "arith", "template": 1, "depth": 5, "pred": "-34", "truth": "-40", "correct": false, "edits_per_token": 0.0}, {"uid": "55f03d61b988deab", "family": "arith", "template": 1, "depth": 5, "pred": "10", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "27df0cd1c098a929", "family": "arith", "template": 0, "depth": 6, "pred": "2144", "truth": "11513", "correct": false, "edits_per_token": 0.0}, {"uid": "17fdac85f59e039f", "family": "arith", "template": 0, "depth": 6, "pred": "-11080", "truth": "-23292", "correct": false, "edits_per_token": 0.0}, {"uid": "c446df844a6f1622", "family": "arith", "template": 0, "depth": 6, "pred": "3004", "truth": "2742", "correct": false, "edits_per_token": 0.0}, {"uid": "a9dcf9180b73a327", "family": "arith", "template": 0, "depth": 6, "pred": "151", "truth": "153", "correct": false, "edits_per_token": 0.0}, {"uid": "bc63707555d01f5f", "family": "arith", "template": 1, "depth": 6, "pred": "208", "truth": "92", "correct": false, "edits_per_token": 0.0}, {"uid": "a2b0c9415ed1725f", "family": "arith", "template": 1, "depth": 6, "pred": "1000", "truth": "1260", "correct": false, "edits_per_token": 0.0}, {"uid": "8dde8846d9d18b1b", "family": "arith", "template": 1, "depth": 6, "pred": "-303", "truth": "-315", "correct": false, "edits_per_token": 0.0}, {"uid": "494a839dbd0029ee", "family": "arith", "template": 1, "depth": 6, "pred": "1600", "truth": "11670", "correct": false, "edits_per_token": 0.0}, {"uid": "fb29f63f32314415", "family": "arith", "template": 1, "depth": 6, "pred": "688", "truth": "597", "correct": false, "edits_per_token": 0.0}, {"uid": "f552c2e836022442", "family": "arith", "template": 0, "depth": 6, "pred": "205", "truth": "301", "correct": false, "edits_per_token": 0.0}, {"uid": "fa5a1426f63c833a", "family": "arith", "template": 0, "depth": 6, "pred": "408", "truth": "322", "correct": false, "edits_per_token": 0.0}, {"uid": "302be3fc789128e5", "family": "arith", "template": 1, "depth": 6, "pred": "716", "truth": "824", "correct": false, "edits_per_token": 0.0}, {"uid": "385a2e8bc4104ed9", "family": "arith", "template": 1, "depth": 6, "pred": "-1004", "truth": "-910", "correct": false, "edits_per_token": 0.0}, {"uid": "077321c4dae60411", "family": "arith", "template": 1, "depth": 6, "pred": "114", "truth": "270", "correct": false, "edits_per_token": 0.0}, {"uid": "bf56aaa4649ea27f", "family": "arith", "template": 0, "depth": 6, "pred": "81", "truth": "82", "correct": false, "edits_per_token": 0.0}, {"uid": "11486f50919b4ea9", "family": "arith", "template": 0, "depth": 6, "pred": "1402", "truth": "8026", "correct": false, "edits_per_token": 0.0}, {"uid": "e35111f507bfc75e", "family": "arith", "template": 0, "depth": 6, "pred": "4108", "truth": "3771", "correct": false, "edits_per_token": 0.0}, {"uid": "276bee3fb394936b", "family": "arith", "template": 0, "depth": 6, "pred": "977", "truth": "788", "correct": false, "edits_per_token": 0.0}, {"uid": "ec56a5b3f62592e2", "family": "arith", "template": 0, "depth": 6, "pred": "124", "truth": "101", "correct": false, "edits_per_token": 0.0}, {"uid": "aa399c0d7a7c20a0", "family": "arith", "template": 1, "depth": 6, "pred": "-36", "truth": "-41", "correct": false, "edits_per_token": 0.0}, {"uid": "2ac019c5c4b6627f", "family": "arith", "template": 1, "depth": 6, "pred": "-1100", "truth": "-1260", "correct": false, "edits_per_token": 0.0}, {"uid": "af2e45eba90c6547", "family": "arith", "template": 1, "depth": 6, "pred": "169", "truth": "171", "correct": false, "edits_per_token": 0.0}, {"uid": "3e8a249523db61de", "family": "arith", "template": 0, "depth": 6, "pred": "61", "truth": "66", "correct": false, "edits_per_token": 0.0}, {"uid": "d88e4dbe5c40f125", "family": "arith", "template": 0, "depth": 6, "pred": "-153", "truth": "-183", "correct": false, "edits_per_token": 0.0}, {"uid": "6314a41e324e4ee8", "family": "arith", "template": 1, "depth": 6, "pred": "-453", "truth": "-534", "correct": false, "edits_per_token": 0.0}, {"uid": "cc03f7312928ea94", "family": "arith", "template": 1, "depth": 6, "pred": "-1000", "truth": "-1032", "correct": false, "edits_per_token": 0.0}, {"uid": "4730814296f3ea5b", "family": "arith", "template": 1, "depth": 6, "pred": "-48", "truth": "-45", "correct": false, "edits_per_token": 0.0}, {"uid": "3852c3f413554f1f", "family": "arith", "template": 2, "depth": 5, "pred": "719", "truth": "719", "correct": true, "edits_per_token": 0.0}, {"uid": "deb86cd80d3700af", "family": "arith", "template": 2, "depth": 5, "pred": "-83", "truth": "-73", "correct": false, "edits_per_token": 0.0}, {"uid": "c119fa976ea67a76", "family": "arith", "template": 2, "depth": 5, "pred": "7544", "truth": "6371", "correct": false, "edits_per_token": 0.0}, {"uid": "c6d9a48ab45e03f3", "family": "arith", "template": 2, "depth": 5, "pred": "-5", "truth": "-4", "correct": false, "edits_per_token": 0.0}, {"uid": "ad1095bee52d4847", "family": "arith", "template": 2, "depth": 5, "pred": "555", "truth": "4148", "correct": false, "edits_per_token": 0.0}, {"uid": "1d9ddc2737b667ba", "family": "arith", "template": 2, "depth": 5, "pred": "-48", "truth": "-63", "correct": false, "edits_per_token": 0.0}, {"uid": "b228f892f896a53b", "family": "arith", "template": 2, "depth": 5, "pred": "3020", "truth": "5132", "correct": false, "edits_per_token": 0.0}, {"uid": "4e943d03290831f4", "family": "arith", "template": 2, "depth": 5, "pred": "1444", "truth": "1022", "correct": false, "edits_per_token": 0.0}, {"uid": "01e14be1e788dc29", "family": "arith", "template": 2, "depth": 5, "pred": "-33", "truth": "-78", "correct": false, "edits_per_token": 0.0}, {"uid": "4c03286b1e57814d", "family": "arith", "template": 2, "depth": 5, "pred": "-1533", "truth": "-1281", "correct": false, "edits_per_token": 0.0}, {"uid": "bd1d4550d7e49f3f", "family": "arith", "template": 2, "depth": 5, "pred": "23", "truth": "25", "correct": false, "edits_per_token": 0.0}, {"uid": "9afbbb94fae24c93", "family": "arith", "template": 2, "depth": 5, "pred": "24", "truth": "25", "correct": false, "edits_per_token": 0.0}, {"uid": "fa5a4ae928911164", "family": "arith", "template": 2, "depth": 5, "pred": "-100", "truth": "-67", "correct": false, "edits_per_token": 0.0}, {"uid": "ea603b34915972f6", "family": "arith", "template": 2, "depth": 6, "pred": "1177", "truth": "10389", "correct": false, "edits_per_token": 0.0}, {"uid": "cb94dfad560085d5", "family": "arith", "template": 2, "depth": 6, "pred": "-36", "truth": "-42", "correct": false, "edits_per_token": 0.0}, {"uid": "9f9e926d4210dd46", "family": "arith", "template": 2, "depth": 6, "pred": "-116", "truth": "-120", "correct": false, "edits_per_token": 0.0}, {"uid": "4715b609ce7f76a7", "family": "arith", "template": 2, "depth": 6, "pred": "-38", "truth": "-33", "correct": false, "edits_per_token": 0.0}, {"uid": "d50bb90d051433a3", "family": "arith", "template": 2, "depth": 6, "pred": "-30", "truth": "-32", "correct": false, "edits_per_token": 0.0}, {"uid": "d1150c19a63e23cb", "family": "arith", "template": 2, "depth": 6, "pred": "-110800080.", "truth": "-25200", "correct": false, "edits_per_token": 0.0}, {"uid": "d8cc53270006b243", "family": "arith", "template": 2, "depth": 6, "pred": "-27", "truth": "819", "correct": false, "edits_per_token": 0.0}, {"uid": "a98fefd64fdb5cee", "family": "bind", "template": 2, "depth": 5, "pred": "-6", "truth": "-4", "correct": false, "edits_per_token": 0.0}, {"uid": "1766b8b21658487d", "family": "bind", "template": 2, "depth": 5, "pred": "53", "truth": "61", "correct": false, "edits_per_token": 0.0}, {"uid": "35dcd024227861bc", "family": "bind", "template": 2, "depth": 5, "pred": "-11", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "5a49700e594f6138", "family": "bind", "template": 2, "depth": 5, "pred": "680", "truth": "420", "correct": false, "edits_per_token": 0.0}, {"uid": "6ca4fd9394c1d2ee", "family": "bind", "template": 2, "depth": 5, "pred": "220", "truth": "402", "correct": false, "edits_per_token": 0.0}, {"uid": "6e39b1d110ecd6c0", "family": "arith", "template": 2, "depth": 6, "pred": "180", "truth": "273", "correct": false, "edits_per_token": 0.0}, {"uid": "b1193e6957a95449", "family": "arith", "template": 2, "depth": 6, "pred": "56", "truth": "54", "correct": false, "edits_per_token": 0.0}, {"uid": "a14793b622e3127b", "family": "arith", "template": 2, "depth": 6, "pred": "-21", "truth": "-23", "correct": false, "edits_per_token": 0.0}, {"uid": "5782d4f05faeb92f", "family": "bind", "template": 2, "depth": 5, "pred": "-16", "truth": "-24", "correct": false, "edits_per_token": 0.0}, {"uid": "1c6b4860d164b9f2", "family": "bind", "template": 2, "depth": 5, "pred": "10", "truth": "12", "correct": false, "edits_per_token": 0.0}, {"uid": "4af96fa0ad806077", "family": "bind", "template": 2, "depth": 5, "pred": "62", "truth": "69", "correct": false, "edits_per_token": 0.0}, {"uid": "8441e8c811ca2537", "family": "bind", "template": 2, "depth": 5, "pred": "15", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "2b9a84b2ec1c5d36", "family": "bind", "template": 2, "depth": 5, "pred": "680", "truth": "1116", "correct": false, "edits_per_token": 0.0}, {"uid": "e670f4c748204cc8", "family": "bind", "template": 2, "depth": 5, "pred": "134", "truth": "102", "correct": false, "edits_per_token": 0.0}, {"uid": "f2b0d3fe9038acd2", "family": "arith", "template": 2, "depth": 6, "pred": "45", "truth": "42", "correct": false, "edits_per_token": 0.0}, {"uid": "daacb1ff7411a2c2", "family": "arith", "template": 2, "depth": 6, "pred": "90", "truth": "101", "correct": false, "edits_per_token": 0.0}, {"uid": "f17398d768802f6b", "family": "bind", "template": 2, "depth": 5, "pred": "95", "truth": "95", "correct": true, "edits_per_token": 0.0}, {"uid": "a881a48bc9d6d2fa", "family": "arith", "template": 2, "depth": 6, "pred": "-3", "truth": "-53", "correct": false, "edits_per_token": 0.0}, {"uid": "5f00de1c8f6a727b", "family": "bind", "template": 2, "depth": 5, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 0.0}, {"uid": "56f549cdb3619b33", "family": "bind", "template": 0, "depth": 5, "pred": "21", "truth": "29", "correct": false, "edits_per_token": 0.0}, {"uid": "70e96a9cddee5517", "family": "bind", "template": 0, "depth": 5, "pred": "134", "truth": "179", "correct": false, "edits_per_token": 0.0}, {"uid": "ea1484f36c572696", "family": "bind", "template": 0, "depth": 5, "pred": "70", "truth": "80", "correct": false, "edits_per_token": 0.0}, {"uid": "54a6e84b79b6c7df", "family": "bind", "template": 0, "depth": 5, "pred": "-12", "truth": "-12", "correct": true, "edits_per_token": 0.0}, {"uid": "fec334e2c8b64794", "family": "bind", "template": 0, "depth": 5, "pred": "71", "truth": "81", "correct": false, "edits_per_token": 0.0}, {"uid": "c3f0503818c31297", "family": "bind", "template": 0, "depth": 5, "pred": "93", "truth": "89", "correct": false, "edits_per_token": 0.0}, {"uid": "a978e3f3b6818825", "family": "bind", "template": 0, "depth": 5, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 0.0}, {"uid": "33b72f3d87631e86", "family": "bind", "template": 0, "depth": 5, "pred": "317", "truth": "667", "correct": false, "edits_per_token": 0.0}, {"uid": "4410175b31fd9ff9", "family": "bind", "template": 0, "depth": 5, "pred": "140", "truth": "192", "correct": false, "edits_per_token": 0.0}, {"uid": "177037480b40585d", "family": "bind", "template": 0, "depth": 5, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "bf86bbb1610dd587", "family": "bind", "template": 1, "depth": 5, "pred": "648", "truth": "588", "correct": false, "edits_per_token": 0.0}, {"uid": "82375062bef2aad9", "family": "bind", "template": 0, "depth": 5, "pred": "75", "truth": "85", "correct": false, "edits_per_token": 0.0}, {"uid": "084f23e813b1c2f2", "family": "bind", "template": 0, "depth": 5, "pred": "33", "truth": "40", "correct": false, "edits_per_token": 0.0}, {"uid": "dca41c3ec45e2945", "family": "bind", "template": 1, "depth": 5, "pred": "300", "truth": "580", "correct": false, "edits_per_token": 0.0}, {"uid": "ee7f986fce1fe6fc", "family": "bind", "template": 0, "depth": 5, "pred": "34", "truth": "41", "correct": false, "edits_per_token": 0.0}, {"uid": "c731265cddc84c46", "family": "bind", "template": 0, "depth": 5, "pred": "100", "truth": "101", "correct": false, "edits_per_token": 0.0}, {"uid": "9a41a8cfde6b00c6", "family": "bind", "template": 1, "depth": 5, "pred": "-14", "truth": "-14", "correct": true, "edits_per_token": 0.0}, {"uid": "8d19dd38ce14dfc5", "family": "bind", "template": 1, "depth": 5, "pred": "367", "truth": "2076", "correct": false, "edits_per_token": 0.0}, {"uid": "677f597fd752b712", "family": "bind", "template": 1, "depth": 5, "pred": "-15", "truth": "-19", "correct": false, "edits_per_token": 0.0}, {"uid": "1cec2e6bbdfaf026", "family": "bind", "template": 1, "depth": 5, "pred": "58", "truth": "56", "correct": false, "edits_per_token": 0.0}, {"uid": "d7f0bbe7ddd4451f", "family": "bind", "template": 1, "depth": 5, "pred": "10", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "f6d1f2f6e260e0e3", "family": "bind", "template": 1, "depth": 5, "pred": "100", "truth": "90", "correct": false, "edits_per_token": 0.0}, {"uid": "c18e4b7f918c9f69", "family": "bind", "template": 1, "depth": 5, "pred": "20", "truth": "24", "correct": false, "edits_per_token": 0.0}, {"uid": "90c73ad098280857", "family": "bind", "template": 1, "depth": 5, "pred": "-41", "truth": "-69", "correct": false, "edits_per_token": 0.0}, {"uid": "a2750a77af5a3ed4", "family": "bind", "template": 1, "depth": 5, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "beccfdc7b9edf6a8", "family": "bind", "template": 2, "depth": 6, "pred": "145", "truth": "280", "correct": false, "edits_per_token": 0.0}, {"uid": "dfd3cbb03a444dec", "family": "bind", "template": 2, "depth": 6, "pred": "43", "truth": "60", "correct": false, "edits_per_token": 0.0}, {"uid": "5da4fa98b5246175", "family": "bind", "template": 2, "depth": 6, "pred": "36", "truth": "47", "correct": false, "edits_per_token": 0.0}, {"uid": "e8bdc504c2e559a6", "family": "bind", "template": 2, "depth": 6, "pred": "76", "truth": "116", "correct": false, "edits_per_token": 0.0}, {"uid": "576442ddb2643bf3", "family": "bind", "template": 2, "depth": 6, "pred": "70", "truth": "107", "correct": false, "edits_per_token": 0.0}, {"uid": "90935ad82127780b", "family": "fsm", "template": 2, "depth": 5, "pred": "28", "truth": "28", "correct": true, "edits_per_token": 0.0}, {"uid": "efd0c340700fd585", "family": "bind", "template": 1, "depth": 5, "pred": "107", "truth": "129", "correct": false, "edits_per_token": 0.0}, {"uid": "8a00b1ceb8cc21d0", "family": "bind", "template": 2, "depth": 6, "pred": "-11", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "1710e2024ce0188e", "family": "bind", "template": 2, "depth": 6, "pred": "114", "truth": "582", "correct": false, "edits_per_token": 0.0}, {"uid": "332068c4dc811926", "family": "bind", "template": 2, "depth": 6, "pred": "-14", "truth": "-15", "correct": false, "edits_per_token": 0.0}, {"uid": "47e5102caba8b95d", "family": "bind", "template": 2, "depth": 6, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 0.0}, {"uid": "9193a977b5e6d754", "family": "bind", "template": 2, "depth": 6, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "69b42fd072531922", "family": "bind", "template": 1, "depth": 5, "pred": "-7", "truth": "-4", "correct": false, "edits_per_token": 0.0}, {"uid": "ad4b55e83f33e0e9", "family": "bind", "template": 2, "depth": 6, "pred": "27", "truth": "38", "correct": false, "edits_per_token": 0.0}, {"uid": "6a42a33ed3b404ed", "family": "bind", "template": 2, "depth": 6, "pred": "-3", "truth": "-4", "correct": false, "edits_per_token": 0.0}, {"uid": "5885e5d090992cd2", "family": "bind", "template": 2, "depth": 6, "pred": "-12", "truth": "-14", "correct": false, "edits_per_token": 0.0}, {"uid": "db61f97b4eb08ce6", "family": "fsm", "template": 2, "depth": 5, "pred": "9", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "1a52ddcf13f295de", "family": "fsm", "template": 2, "depth": 5, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "4764ec83e7c8c9b6", "family": "fsm", "template": 2, "depth": 5, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "cf3e9f98a9edf153", "family": "fsm", "template": 1, "depth": 5, "pred": "27", "truth": "29", "correct": false, "edits_per_token": 0.0}, {"uid": "485f2555fb5eff21", "family": "bind", "template": 0, "depth": 6, "pred": "2", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "76f66b2bb4f4d200", "family": "bind", "template": 0, "depth": 6, "pred": "93", "truth": "108", "correct": false, "edits_per_token": 0.0}, {"uid": "8a0414e10232f5c1", "family": "bind", "template": 0, "depth": 6, "pred": "34", "truth": "52", "correct": false, "edits_per_token": 0.0}, {"uid": "6fa2bc31af37d28d", "family": "bind", "template": 0, "depth": 6, "pred": "48", "truth": "101", "correct": false, "edits_per_token": 0.0}, {"uid": "52178894c71a55fc", "family": "bind", "template": 0, "depth": 6, "pred": "140", "truth": "204", "correct": false, "edits_per_token": 0.0}, {"uid": "bf993b5b68923b56", "family": "bind", "template": 0, "depth": 6, "pred": "34", "truth": "27", "correct": false, "edits_per_token": 0.0}, {"uid": "c5ee1489b5a7552b", "family": "bind", "template": 0, "depth": 6, "pred": "108", "truth": "268", "correct": false, "edits_per_token": 0.0}, {"uid": "2dbbd53f53e30073", "family": "bind", "template": 0, "depth": 6, "pred": "61", "truth": "78", "correct": false, "edits_per_token": 0.0}, {"uid": "7437da9eb53eb3a0", "family": "bind", "template": 0, "depth": 6, "pred": "11", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "408a34c94f17f5d7", "family": "bind", "template": 0, "depth": 6, "pred": "61", "truth": "120", "correct": false, "edits_per_token": 0.0}, {"uid": "1bad57bf846df87e", "family": "bind", "template": 0, "depth": 6, "pred": "16", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "3d0794faec305528", "family": "bind", "template": 1, "depth": 6, "pred": "4", "truth": "15", "correct": false, "edits_per_token": 0.0}, {"uid": "10c6fe8e473b05f7", "family": "bind", "template": 1, "depth": 6, "pred": "-12", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "897766cab535c928", "family": "bind", "template": 1, "depth": 6, "pred": "50", "truth": "63", "correct": false, "edits_per_token": 0.0}, {"uid": "37fcdefdc142c5c8", "family": "bind", "template": 1, "depth": 6, "pred": "-100", "truth": "-520", "correct": false, "edits_per_token": 0.0}, {"uid": "0fa8fa84eab24d5b", "family": "bind", "template": 1, "depth": 6, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "e17850521f774ccb", "family": "bind", "template": 1, "depth": 6, "pred": "-36", "truth": "-57", "correct": false, "edits_per_token": 0.0}, {"uid": "87604ac5e2432117", "family": "bind", "template": 0, "depth": 6, "pred": "-2", "truth": "-2", "correct": true, "edits_per_token": 0.0}, {"uid": "1bf2c062e3d35770", "family": "bind", "template": 0, "depth": 6, "pred": "-14", "truth": "-21", "correct": false, "edits_per_token": 0.0}, {"uid": "3e71f658a8d01416", "family": "bind", "template": 1, "depth": 6, "pred": "202", "truth": "612", "correct": false, "edits_per_token": 0.0}, {"uid": "6d9120572acc5f8c", "family": "bind", "template": 1, "depth": 6, "pred": "141", "truth": "421", "correct": false, "edits_per_token": 0.0}, {"uid": "262d1886f373a193", "family": "bind", "template": 1, "depth": 6, "pred": "48", "truth": "192", "correct": false, "edits_per_token": 0.0}, {"uid": "2df700b079b1ea09", "family": "bind", "template": 1, "depth": 6, "pred": "514", "truth": "254", "correct": false, "edits_per_token": 0.0}, {"uid": "9f22ff248a27d85b", "family": "bind", "template": 1, "depth": 6, "pred": "18", "truth": "21", "correct": false, "edits_per_token": 0.0}, {"uid": "b86f3a062bdd7b14", "family": "bind", "template": 1, "depth": 6, "pred": "24", "truth": "28", "correct": false, "edits_per_token": 0.0}, {"uid": "0ec61c8d74b0b12e", "family": "bind", "template": 1, "depth": 6, "pred": "-5", "truth": "-4", "correct": false, "edits_per_token": 0.0}, {"uid": "7e69b09c478d7cf0", "family": "bind", "template": 1, "depth": 6, "pred": "-33", "truth": "-103", "correct": false, "edits_per_token": 0.0}, {"uid": "3644e4d7e72723f2", "family": "fsm", "template": 2, "depth": 5, "pred": "11", "truth": "11", "correct": true, "edits_per_token": 0.0}, {"uid": "ef5c74f822ed4d09", "family": "fsm", "template": 2, "depth": 5, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "1d8a092a5663c2ca", "family": "fsm", "template": 2, "depth": 6, "pred": "18", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "13b840f1309c3d23", "family": "fsm", "template": 2, "depth": 6, "pred": "16", "truth": "27", "correct": false, "edits_per_token": 0.0}, {"uid": "3bf1de404d07020e", "family": "fsm", "template": 1, "depth": 5, "pred": "30", "truth": "24", "correct": false, "edits_per_token": 0.0}, {"uid": "cca744ebe723c85b", "family": "fsm", "template": 2, "depth": 5, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "976535ab3eb31bd8", "family": "fsm", "template": 2, "depth": 5, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 0.0}, {"uid": "93ee3b76c1253b5f", "family": "fsm", "template": 0, "depth": 5, "pred": "8", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "fa364f44c6759d61", "family": "fsm", "template": 0, "depth": 5, "pred": "59", "truth": "63", "correct": false, "edits_per_token": 0.0}, {"uid": "06585bc8d35c4f52", "family": "fsm", "template": 1, "depth": 5, "pred": "16", "truth": "22", "correct": false, "edits_per_token": 0.0}, {"uid": "cb31b4e31c03606f", "family": "fsm", "template": 1, "depth": 5, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "0324b1035a1b5e3b", "family": "fsm", "template": 1, "depth": 5, "pred": "24", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "70965bfd44497a12", "family": "fsm", "template": 2, "depth": 6, "pred": "8", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "9e2ced0a36e30f08", "family": "fsm", "template": 2, "depth": 6, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 0.0}, {"uid": "945b988367f1262b", "family": "fsm", "template": 2, "depth": 5, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "49d61413398cc89b", "family": "fsm", "template": 2, "depth": 6, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "cac6ffd4670e5f50", "family": "fsm", "template": 2, "depth": 6, "pred": "12", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "ec003e386e949698", "family": "fsm", "template": 0, "depth": 5, "pred": "4", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "b4c4da03d549d2ad", "family": "fsm", "template": 0, "depth": 5, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 0.0}, {"uid": "d4906001b8fd5965", "family": "fsm", "template": 2, "depth": 5, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 0.0}, {"uid": "054e529ea5d13efe", "family": "fsm", "template": 2, "depth": 5, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "af25cc7ee3868a05", "family": "fsm", "template": 0, "depth": 5, "pred": "11", "truth": "11", "correct": true, "edits_per_token": 0.0}, {"uid": "e6eb0871661f8cb0", "family": "fsm", "template": 2, "depth": 6, "pred": "10", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "d981e261ba455e59", "family": "fsm", "template": 1, "depth": 5, "pred": "4", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "e19b1ade5034200f", "family": "fsm", "template": 0, "depth": 5, "pred": "10", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "9084157c9bdc265e", "family": "fsm", "template": 0, "depth": 5, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 0.0}, {"uid": "966d9784f516b0b4", "family": "fsm", "template": 2, "depth": 5, "pred": "-1", "truth": "0", "correct": false, "edits_per_token": 0.0}, {"uid": "7e2c6414320592ae", "family": "fsm", "template": 0, "depth": 6, "pred": "15", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "b06f1eb977fbe9e0", "family": "fsm", "template": 1, "depth": 6, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "fa3b7f75f9e43b2f", "family": "fsm", "template": 1, "depth": 6, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "c7498914bccd7bf3", "family": "fsm", "template": 0, "depth": 5, "pred": "7", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "195ef7d3ccac4b8b", "family": "fsm", "template": 0, "depth": 5, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "7cb457b70a7065ab", "family": "fsm", "template": 1, "depth": 5, "pred": "8", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "4688a1575da3feae", "family": "fsm", "template": 2, "depth": 6, "pred": "-2", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "13d7a0661383bf5c", "family": "fsm", "template": 0, "depth": 6, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "8520bf5fba316dc5", "family": "fsm", "template": 0, "depth": 6, "pred": "13", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "28a5212e37e34ede", "family": "fsm", "template": 2, "depth": 6, "pred": "-2", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "c1db7537b69a1588", "family": "fsm", "template": 1, "depth": 5, "pred": "7", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "475faa1c40dd1c57", "family": "fsm", "template": 2, "depth": 6, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 0.0}, {"uid": "82de054bd19fee8f", "family": "fsm", "template": 2, "depth": 6, "pred": "-20", "truth": "-18", "correct": false, "edits_per_token": 0.0}, {"uid": "1599bc918482f147", "family": "fsm", "template": 2, "depth": 6, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "f13662f0f0c30826", "family": "fsm", "template": 0, "depth": 5, "pred": "6", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "b7e69cea18577e61", "family": "fsm", "template": 0, "depth": 5, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 0.0}, {"uid": "f01f401147209180", "family": "fsm", "template": 1, "depth": 6, "pred": "12", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "e45b754845391f3f", "family": "fsm", "template": 1, "depth": 5, "pred": "8", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "babac6e1d2340f83", "family": "fsm", "template": 1, "depth": 6, "pred": "0", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "f3dfeb8093398fba", "family": "fsm", "template": 2, "depth": 5, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "83156af5c9f2102f", "family": "fsm", "template": 1, "depth": 6, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 0.0}, {"uid": "436a870501934e95", "family": "fsm", "template": 2, "depth": 6, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 0.0}, {"uid": "17c22d2c6da7016d", "family": "fsm", "template": 0, "depth": 5, "pred": "1", "truth": "0", "correct": false, "edits_per_token": 0.0}, {"uid": "958a99eb728e15c5", "family": "fsm", "template": 0, "depth": 5, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "e969fccd3aa5aa4d", "family": "fsm", "template": 1, "depth": 5, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 0.0}, {"uid": "9e5d597a794e81e2", "family": "fsm", "template": 0, "depth": 5, "pred": "1", "truth": "0", "correct": false, "edits_per_token": 0.0}, {"uid": "53a3aa34d09ae7f0", "family": "fsm", "template": 0, "depth": 6, "pred": "13", "truth": "15", "correct": false, "edits_per_token": 0.0}, {"uid": "7ffa08ddf97e22b8", "family": "fsm", "template": 0, "depth": 6, "pred": "6", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "df4e72237d80a81c", "family": "fsm", "template": 1, "depth": 5, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "95a0ef58216182a2", "family": "fsm", "template": 1, "depth": 6, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "6c413e46c2f2d918", "family": "fsm", "template": 1, "depth": 6, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "38b00c5c8332094f", "family": "fsm", "template": 0, "depth": 6, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "c571f1db246e3ac7", "family": "fsm", "template": 1, "depth": 5, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "c8c65c1d9220669b", "family": "fsm", "template": 1, "depth": 6, "pred": "18", "truth": "28", "correct": false, "edits_per_token": 0.0}, {"uid": "2bd960c30ea3d343", "family": "fsm", "template": 0, "depth": 6, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 0.0}, {"uid": "dbd3b4ea0faa4bd5", "family": "fsm", "template": 0, "depth": 6, "pred": "6", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "ad56dff0f6c52a6e", "family": "fsm", "template": 0, "depth": 6, "pred": "-14", "truth": "-16", "correct": false, "edits_per_token": 0.0}, {"uid": "5aa303b1e8a3a50a", "family": "fsm", "template": 1, "depth": 6, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "a7aa7192117d2b91", "family": "fsm", "template": 0, "depth": 6, "pred": "13", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "3663d9bfed0657ac", "family": "fsm", "template": 1, "depth": 6, "pred": "-2", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "d442b8b4e9c4463b", "family": "fsm", "template": 1, "depth": 6, "pred": "11", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "f5d3a7b825df11db", "family": "fsm", "template": 0, "depth": 6, "pred": "7", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "5fae584fb4b0806f", "family": "fsm", "template": 1, "depth": 5, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 0.0}, {"uid": "f5d6e652e3c1c9c6", "family": "fsm", "template": 1, "depth": 6, "pred": "12", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "fae83630cc69d997", "family": "fsm", "template": 1, "depth": 6, "pred": "-4", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "ac2e2426143b2f6a", "family": "fsm", "template": 1, "depth": 6, "pred": "-7", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "3e7729193c5d67ef", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "b040bca9ccdd27d8", "family": "fsm", "template": 0, "depth": 6, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "ae8d496071780205", "family": "fsm", "template": 0, "depth": 6, "pred": "4", "truth": "-4", "correct": false, "edits_per_token": 0.0}, {"uid": "48b2c33f5bfc7b39", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "ffb3a068ddee91d8", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "0b5d578a6414c59b", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "plum fig", "correct": false, "edits_per_token": 0.0}, {"uid": "5457f87f4a4d9928", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "29cc176fd2bd357a", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "9a37aa52f6f4f0ef", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "yam", "correct": false, "edits_per_token": 0.0}, {"uid": "f6e2b9995ac5a2af", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "04e6cd46e148e861", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "c53436531a9e4322", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "kiwi lime", "correct": false, "edits_per_token": 0.0}, {"uid": "8c39602b8320d36d", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "451ab7989b156691", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "plum", "correct": false, "edits_per_token": 0.0}, {"uid": "64e6b2f20007b04f", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "06695366b2ca7f94", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "fe21b5c807616138", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "b3aeac08c3bf5403", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "7b4a99cf741c62f4", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 0.0}, {"uid": "544e5524578ed1a2", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 0.0}, {"uid": "8813abcc63108e4e", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "6ef02a442804899d", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "9542b67b66fba534", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "bea23de3fd39e931", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "f862e4eb48abf854", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "1e956d986d80a2e8", "family": "sched", "template": 0, "depth": 5, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "8c26836a775dfeef", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 0.0}, {"uid": "dbc95bbfbdbb8c83", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "589bdb5b41da7c65", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "e61d3770c79cde47", "family": "sched", "template": 0, "depth": 5, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "2fbd02a75eab5edf", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "bc00b85f3a4929cb", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "83d9e53bcbe7935d", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "19147a5ccac0f776", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "b6995c56551edd0f", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "597331784c8adb06", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "7a4868e5e81669d0", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "6ab7960babdc2e4d", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "8c9f86f29e505eb5", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "7d3f18625f00d8ee", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "1e7febaac638433d", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "192270ee3b6c26d9", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "2b26c9c6928a8458", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "65550dd4f783d103", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "637f9d17aabeb31c", "family": "sched", "template": 1, "depth": 5, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "ed8754896958fc23", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "date", "correct": false, "edits_per_token": 0.0}, {"uid": "daf704f320ba7df5", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 0.0}, {"uid": "72ac559ac49d72ce", "family": "sched", "template": 1, "depth": 5, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "b7380cfd5017d3d5", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "7bfb28727e572f4f", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 0.0}, {"uid": "5a7f7b85164f7d43", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "bean", "correct": false, "edits_per_token": 0.0}, {"uid": "5a9f6a2d1fca7c30", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "5dfac4f4fcff919d", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "c877e5915a1d82dd", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "91d7e38041c84a54", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "d88ead16fc9cb4dd", "family": "sched", "template": 2, "depth": 5, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "0470c678ba5e33bd", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 0.0}, {"uid": "2004484848062f39", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "4aec4964885ab8cd", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "f62aa798063a3946", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "5821393361b27572", "family": "sched", "template": 2, "depth": 5, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "9835b0f9d397f634", "family": "sched", "template": 2, "depth": 5, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "efd0e2bbf109da63", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "66c464d4968a2fe2", "family": "sched", "template": 0, "depth": 5, "pred": "none", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "ae5e2670afd5d884", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "738b6d8b10bdecc1", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "31e0d3601d129675", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "8c334375cbdeecbb", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "3146c4b45bd4ba36", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "16bc96ba9e32042e", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "7dc88230188b5ae0", "family": "sched", "template": 0, "depth": 5, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "ffdf323f5b0e8398", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 0.0}, {"uid": "031f5a4ec368d5ed", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "249d68235fab3319", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "ac5f146d3cf4c744", "family": "sched", "template": 0, "depth": 5, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "85e6171c2bff77ec", "family": "sched", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "a5eb1ad33216a365", "family": "sched", "template": 0, "depth": 5, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "fbfb9d7ebbd9ab42", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 0.0}, {"uid": "93e46dc65f4a0d60", "family": "sched", "template": 2, "depth": 6, "pred": "Dxclude 8,", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "7d44befcfd949a85", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "dfe372c23262434c", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "c68ec9cd887646d5", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "a01c26ea9b3e73d4", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 0.0}, {"uid": "6c51902842f6a23b", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 0.0}, {"uid": "1d040a988ed2fa83", "family": "sched", "template": 2, "depth": 5, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "165a925a98ad694c", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "e97fa901d509ffd6", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 0.0}, {"uid": "3676d84af524d9ae", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 0.0}, {"uid": "cd0b4315898f7965", "family": "sched", "template": 2, "depth": 5, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "7429be63b9fe2121", "family": "sched", "template": 0, "depth": 6, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "c696f5f9f596f2f7", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 0.0}, {"uid": "59f92777461a61ec", "family": "sched", "template": 0, "depth": 6, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "fb0fcdfe7002426e", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "871d1deab44403c5", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "plum", "correct": false, "edits_per_token": 0.0}, {"uid": "e09fe1db1040c253", "family": "sched", "template": 0, "depth": 5, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "70d22e9191a63ed3", "family": "sched", "template": 0, "depth": 5, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "b5b6b8f1e1b8e526", "family": "sched", "template": 2, "depth": 5, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "eb52c5c1389933b9", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "ae5ef219347ca8d1", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "bc760caf57d4fbeb", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "342a44e350c2c803", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "plum date", "correct": false, "edits_per_token": 0.0}, {"uid": "c1f8bff397b18871", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "fig", "correct": false, "edits_per_token": 0.0}, {"uid": "2315692be3c9225f", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "yam", "correct": false, "edits_per_token": 0.0}, {"uid": "8eb603b3bcb2b694", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "588fd0833653c113", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "1e067fba49f1bde2", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "7216a64146ae6acd", "family": "sched", "template": 1, "depth": 5, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "7845079b2715c407", "family": "sched", "template": 1, "depth": 5, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "5733b7ed7ff6f08e", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 0.0}, {"uid": "9565161477fd02c2", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "f3390e99576c3f52", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "685bcd1984a9ef37", "family": "sched", "template": 1, "depth": 6, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 0.0}, {"uid": "127824e03ba4e18f", "family": "sched", "template": 1, "depth": 6, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "7af3f6bc2435a9c3", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "af0de551a61187f6", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "bdaff9107551bcad", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "d5369281fc9cfcd4", "family": "sched", "template": 1, "depth": 5, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "994412f1bf4510f1", "family": "sched", "template": 1, "depth": 6, "pred": "B", "truth": "F", "correct": false, "edits_per_token": 0.0}, {"uid": "b1337bc8478cf188", "family": "sched", "template": 1, "depth": 6, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 0.0}, {"uid": "1591f1da64128221", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "bean", "correct": false, "edits_per_token": 0.0}, {"uid": "4ace02760358e0aa", "family": "sched", "template": 1, "depth": 5, "pred": "C", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "c38781630980610c", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "5cdeab7b7d3d00a8", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "c0b33bc616c270db", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "ae3d111057f9a376", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "0dbf96f152ee57cd", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 0.0}, {"uid": "2ec492b6141d5723", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "a8895e2feee026d1", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "63b451d568c16ffa", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "7a75d953e8308610", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 0.0}, {"uid": "551b7796162a1a33", "family": "sched", "template": 1, "depth": 5, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "0409fe31b46650ef", "family": "sched", "template": 0, "depth": 6, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "6fcf7fc1691a79a5", "family": "sched", "template": 2, "depth": 6, "pred": "A", "truth": "F", "correct": false, "edits_per_token": 0.0}, {"uid": "52e4c2db61cbda34", "family": "sched", "template": 0, "depth": 6, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "6b2e6485c06a7570", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "e7b6cecd9fd15087", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "f1ca4b2353614cc4", "family": "sched", "template": 0, "depth": 6, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "f71e8f3bf0d91481", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "227f0ada257e9567", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "0847283e1b47f87f", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "5c926222dc33129d", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 0.0}, {"uid": "6a5adbd9ac262749", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "dbccdf3934cf6baf", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "d3979db4d8709e58", "family": "sched", "template": 0, "depth": 6, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "da9eddd4e681af1a", "family": "sched", "template": 0, "depth": 6, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "bb2c190eae3c2125", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "7257616f0bb8b273", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "b442b080ca1d58bf", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "1762aafe566bb2f0", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "E", "correct": false, "edits_per_token": 0.0}, {"uid": "3478623940a0babb", "family": "sched", "template": 0, "depth": 6, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "d49274b1b41b7b91", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "54953d3704253d0a", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "4b3ba50fe8aed021", "family": "sched", "template": 1, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 0.0}, {"uid": "1564dfd3f2b289c5", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "420f41111b0b1478", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "fb5eb61f7d51b656", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "b3cf93f47535eec2", "family": "sched", "template": 1, "depth": 6, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "a98861d878768980", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "8c0cce3a8cc7a8a8", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "139fdebe4be35cb8", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "bd0288f2ce913d48", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "39f4a4e458ce30d0", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}] | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_depth_b2.json
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +[{"uid": "05a9ab128b8f795a", "family": "arith", "template": 0, "depth": 5, "pred": "204", "truth": "206", "correct": false, "edits_per_token": 2.0}, {"uid": "ce77dc7f6121c65c", "family": "arith", "template": 0, "depth": 5, "pred": "3173", "truth": "3150", "correct": false, "edits_per_token": 2.0}, {"uid": "6d86083e9a296d70", "family": "arith", "template": 0, "depth": 5, "pred": "611", "truth": "467", "correct": false, "edits_per_token": 2.0}, {"uid": "bbbc075e9ed2d83b", "family": "arith", "template": 0, "depth": 5, "pred": "16480", "truth": "13248", "correct": false, "edits_per_token": 2.0}, {"uid": "9cc4e3fd8e271e9e", "family": "arith", "template": 0, "depth": 5, "pred": "1080", "truth": "1800", "correct": false, "edits_per_token": 2.0}, {"uid": "d93003ac9ddbe92e", "family": "arith", "template": 1, "depth": 5, "pred": "-12", "truth": "18", "correct": false, "edits_per_token": 2.0}, {"uid": "0429a41c21791b48", "family": "arith", "template": 1, "depth": 5, "pred": "408", "truth": "485", "correct": false, "edits_per_token": 2.0}, {"uid": "252188905e760471", "family": "arith", "template": 1, "depth": 5, "pred": "20", "truth": "43", "correct": false, "edits_per_token": 2.0}, {"uid": "9d3b6d51fa8f268d", "family": "arith", "template": 1, "depth": 5, "pred": "-300", "truth": "-320", "correct": false, "edits_per_token": 2.0}, {"uid": "c38dbe1ef3f315bb", "family": "arith", "template": 0, "depth": 5, "pred": "-58", "truth": "-63", "correct": false, "edits_per_token": 2.0}, {"uid": "4ee8682a5c7cde40", "family": "arith", "template": 0, "depth": 5, "pred": "-10", "truth": "-7", "correct": false, "edits_per_token": 2.0}, {"uid": "213e53daae3f0053", "family": "arith", "template": 0, "depth": 5, "pred": "1688", "truth": "1113", "correct": false, "edits_per_token": 2.0}, {"uid": "8a499c6f7286285c", "family": "arith", "template": 1, "depth": 5, "pred": "-41", "truth": "-47", "correct": false, "edits_per_token": 2.0}, {"uid": "c7cc1d12232437f5", "family": "arith", "template": 1, "depth": 5, "pred": "517", "truth": "411", "correct": false, "edits_per_token": 2.0}, {"uid": "72633b6ff3769cb5", "family": "arith", "template": 1, "depth": 5, "pred": "34", "truth": "-3", "correct": false, "edits_per_token": 2.0}, {"uid": "50c2617c879f93b6", "family": "arith", "template": 1, "depth": 5, "pred": "148", "truth": "814", "correct": false, "edits_per_token": 2.0}, {"uid": "b2bc5413c8fada82", "family": "arith", "template": 1, "depth": 5, "pred": "607", "truth": "686", "correct": false, "edits_per_token": 2.0}, {"uid": "a8e4058450eb65d2", "family": "arith", "template": 1, "depth": 5, "pred": "108", "truth": "140", "correct": false, "edits_per_token": 2.0}, {"uid": "c709c22eb9084b6b", "family": "arith", "template": 0, "depth": 5, "pred": "-51", "truth": "-39", "correct": false, "edits_per_token": 2.0}, {"uid": "58498d59ed0e6ec5", "family": "arith", "template": 0, "depth": 5, "pred": "380", "truth": "387", "correct": false, "edits_per_token": 2.0}, {"uid": "d79b98b965dbc325", "family": "arith", "template": 0, "depth": 5, "pred": "48", "truth": "55", "correct": false, "edits_per_token": 2.0}, {"uid": "ac935e06e7c21bd6", "family": "arith", "template": 0, "depth": 5, "pred": "80", "truth": "54", "correct": false, "edits_per_token": 2.0}, {"uid": "32d9a5b74df38a67", "family": "arith", "template": 0, "depth": 5, "pred": "288", "truth": "303", "correct": false, "edits_per_token": 2.0}, {"uid": "bea3727c59f821e2", "family": "arith", "template": 0, "depth": 5, "pred": "68", "truth": "71", "correct": false, "edits_per_token": 2.0}, {"uid": "3d7cf92565334823", "family": "arith", "template": 1, "depth": 5, "pred": "-14", "truth": "-15", "correct": false, "edits_per_token": 2.0}, {"uid": "34ad0dad384435be", "family": "arith", "template": 1, "depth": 5, "pred": "-34", "truth": "-40", "correct": false, "edits_per_token": 2.0}, {"uid": "55f03d61b988deab", "family": "arith", "template": 1, "depth": 5, "pred": "10", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "27df0cd1c098a929", "family": "arith", "template": 0, "depth": 6, "pred": "2144", "truth": "11513", "correct": false, "edits_per_token": 2.0}, {"uid": "17fdac85f59e039f", "family": "arith", "template": 0, "depth": 6, "pred": "-11080", "truth": "-23292", "correct": false, "edits_per_token": 2.0}, {"uid": "c446df844a6f1622", "family": "arith", "template": 0, "depth": 6, "pred": "3004", "truth": "2742", "correct": false, "edits_per_token": 2.0}, {"uid": "a9dcf9180b73a327", "family": "arith", "template": 0, "depth": 6, "pred": "151", "truth": "153", "correct": false, "edits_per_token": 2.0}, {"uid": "bc63707555d01f5f", "family": "arith", "template": 1, "depth": 6, "pred": "208", "truth": "92", "correct": false, "edits_per_token": 2.0}, {"uid": "a2b0c9415ed1725f", "family": "arith", "template": 1, "depth": 6, "pred": "1350", "truth": "1260", "correct": false, "edits_per_token": 2.0}, {"uid": "8dde8846d9d18b1b", "family": "arith", "template": 1, "depth": 6, "pred": "-303", "truth": "-315", "correct": false, "edits_per_token": 2.0}, {"uid": "494a839dbd0029ee", "family": "arith", "template": 1, "depth": 6, "pred": "2100", "truth": "11670", "correct": false, "edits_per_token": 2.0}, {"uid": "fb29f63f32314415", "family": "arith", "template": 1, "depth": 6, "pred": "688", "truth": "597", "correct": false, "edits_per_token": 2.0}, {"uid": "f552c2e836022442", "family": "arith", "template": 0, "depth": 6, "pred": "205", "truth": "301", "correct": false, "edits_per_token": 2.0}, {"uid": "fa5a1426f63c833a", "family": "arith", "template": 0, "depth": 6, "pred": "408", "truth": "322", "correct": false, "edits_per_token": 2.0}, {"uid": "302be3fc789128e5", "family": "arith", "template": 1, "depth": 6, "pred": "716", "truth": "824", "correct": false, "edits_per_token": 2.0}, {"uid": "385a2e8bc4104ed9", "family": "arith", "template": 1, "depth": 6, "pred": "-1004", "truth": "-910", "correct": false, "edits_per_token": 2.0}, {"uid": "077321c4dae60411", "family": "arith", "template": 1, "depth": 6, "pred": "114", "truth": "270", "correct": false, "edits_per_token": 2.0}, {"uid": "bf56aaa4649ea27f", "family": "arith", "template": 0, "depth": 6, "pred": "91", "truth": "82", "correct": false, "edits_per_token": 2.0}, {"uid": "11486f50919b4ea9", "family": "arith", "template": 0, "depth": 6, "pred": "955", "truth": "8026", "correct": false, "edits_per_token": 2.0}, {"uid": "e35111f507bfc75e", "family": "arith", "template": 0, "depth": 6, "pred": "4108", "truth": "3771", "correct": false, "edits_per_token": 2.0}, {"uid": "276bee3fb394936b", "family": "arith", "template": 0, "depth": 6, "pred": "977", "truth": "788", "correct": false, "edits_per_token": 2.0}, {"uid": "ec56a5b3f62592e2", "family": "arith", "template": 0, "depth": 6, "pred": "124", "truth": "101", "correct": false, "edits_per_token": 2.0}, {"uid": "aa399c0d7a7c20a0", "family": "arith", "template": 1, "depth": 6, "pred": "-36", "truth": "-41", "correct": false, "edits_per_token": 2.0}, {"uid": "2ac019c5c4b6627f", "family": "arith", "template": 1, "depth": 6, "pred": "-1100", "truth": "-1260", "correct": false, "edits_per_token": 2.0}, {"uid": "af2e45eba90c6547", "family": "arith", "template": 1, "depth": 6, "pred": "169", "truth": "171", "correct": false, "edits_per_token": 2.0}, {"uid": "3e8a249523db61de", "family": "arith", "template": 0, "depth": 6, "pred": "62", "truth": "66", "correct": false, "edits_per_token": 2.0}, {"uid": "d88e4dbe5c40f125", "family": "arith", "template": 0, "depth": 6, "pred": "-153", "truth": "-183", "correct": false, "edits_per_token": 2.0}, {"uid": "6314a41e324e4ee8", "family": "arith", "template": 1, "depth": 6, "pred": "-453", "truth": "-534", "correct": false, "edits_per_token": 2.0}, {"uid": "cc03f7312928ea94", "family": "arith", "template": 1, "depth": 6, "pred": "-1000", "truth": "-1032", "correct": false, "edits_per_token": 2.0}, {"uid": "4730814296f3ea5b", "family": "arith", "template": 1, "depth": 6, "pred": "-50", "truth": "-45", "correct": false, "edits_per_token": 2.0}, {"uid": "3852c3f413554f1f", "family": "arith", "template": 2, "depth": 5, "pred": "719", "truth": "719", "correct": true, "edits_per_token": 2.0}, {"uid": "deb86cd80d3700af", "family": "arith", "template": 2, "depth": 5, "pred": "-83", "truth": "-73", "correct": false, "edits_per_token": 2.0}, {"uid": "c119fa976ea67a76", "family": "arith", "template": 2, "depth": 5, "pred": "7544", "truth": "6371", "correct": false, "edits_per_token": 2.0}, {"uid": "c6d9a48ab45e03f3", "family": "arith", "template": 2, "depth": 5, "pred": "-5", "truth": "-4", "correct": false, "edits_per_token": 2.0}, {"uid": "ad1095bee52d4847", "family": "arith", "template": 2, "depth": 5, "pred": "555", "truth": "4148", "correct": false, "edits_per_token": 2.0}, {"uid": "1d9ddc2737b667ba", "family": "arith", "template": 2, "depth": 5, "pred": "-48", "truth": "-63", "correct": false, "edits_per_token": 2.0}, {"uid": "b228f892f896a53b", "family": "arith", "template": 2, "depth": 5, "pred": "3020", "truth": "5132", "correct": false, "edits_per_token": 2.0}, {"uid": "4e943d03290831f4", "family": "arith", "template": 2, "depth": 5, "pred": "1444", "truth": "1022", "correct": false, "edits_per_token": 2.0}, {"uid": "01e14be1e788dc29", "family": "arith", "template": 2, "depth": 5, "pred": "-33", "truth": "-78", "correct": false, "edits_per_token": 2.0}, {"uid": "4c03286b1e57814d", "family": "arith", "template": 2, "depth": 5, "pred": "-1533", "truth": "-1281", "correct": false, "edits_per_token": 2.0}, {"uid": "bd1d4550d7e49f3f", "family": "arith", "template": 2, "depth": 5, "pred": "23", "truth": "25", "correct": false, "edits_per_token": 2.0}, {"uid": "9afbbb94fae24c93", "family": "arith", "template": 2, "depth": 5, "pred": "24", "truth": "25", "correct": false, "edits_per_token": 2.0}, {"uid": "fa5a4ae928911164", "family": "arith", "template": 2, "depth": 5, "pred": "-100", "truth": "-67", "correct": false, "edits_per_token": 2.0}, {"uid": "ea603b34915972f6", "family": "arith", "template": 2, "depth": 6, "pred": "1177", "truth": "10389", "correct": false, "edits_per_token": 2.0}, {"uid": "cb94dfad560085d5", "family": "arith", "template": 2, "depth": 6, "pred": "-36", "truth": "-42", "correct": false, "edits_per_token": 2.0}, {"uid": "9f9e926d4210dd46", "family": "arith", "template": 2, "depth": 6, "pred": "-116", "truth": "-120", "correct": false, "edits_per_token": 2.0}, {"uid": "4715b609ce7f76a7", "family": "arith", "template": 2, "depth": 6, "pred": "-38", "truth": "-33", "correct": false, "edits_per_token": 2.0}, {"uid": "d50bb90d051433a3", "family": "arith", "template": 2, "depth": 6, "pred": "-30", "truth": "-32", "correct": false, "edits_per_token": 2.0}, {"uid": "d1150c19a63e23cb", "family": "arith", "template": 2, "depth": 6, "pred": "-110800080.", "truth": "-25200", "correct": false, "edits_per_token": 2.0}, {"uid": "d8cc53270006b243", "family": "arith", "template": 2, "depth": 6, "pred": "-27", "truth": "819", "correct": false, "edits_per_token": 2.0}, {"uid": "a98fefd64fdb5cee", "family": "bind", "template": 2, "depth": 5, "pred": "-6", "truth": "-4", "correct": false, "edits_per_token": 2.0}, {"uid": "1766b8b21658487d", "family": "bind", "template": 2, "depth": 5, "pred": "53", "truth": "61", "correct": false, "edits_per_token": 2.0}, {"uid": "35dcd024227861bc", "family": "bind", "template": 2, "depth": 5, "pred": "-11", "truth": "-7", "correct": false, "edits_per_token": 2.0}, {"uid": "5a49700e594f6138", "family": "bind", "template": 2, "depth": 5, "pred": "680", "truth": "420", "correct": false, "edits_per_token": 2.0}, {"uid": "6ca4fd9394c1d2ee", "family": "bind", "template": 2, "depth": 5, "pred": "220", "truth": "402", "correct": false, "edits_per_token": 2.0}, {"uid": "6e39b1d110ecd6c0", "family": "arith", "template": 2, "depth": 6, "pred": "180", "truth": "273", "correct": false, "edits_per_token": 2.0}, {"uid": "b1193e6957a95449", "family": "arith", "template": 2, "depth": 6, "pred": "56", "truth": "54", "correct": false, "edits_per_token": 2.0}, {"uid": "a14793b622e3127b", "family": "arith", "template": 2, "depth": 6, "pred": "-17", "truth": "-23", "correct": false, "edits_per_token": 2.0}, {"uid": "5782d4f05faeb92f", "family": "bind", "template": 2, "depth": 5, "pred": "-16", "truth": "-24", "correct": false, "edits_per_token": 2.0}, {"uid": "1c6b4860d164b9f2", "family": "bind", "template": 2, "depth": 5, "pred": "10", "truth": "12", "correct": false, "edits_per_token": 2.0}, {"uid": "4af96fa0ad806077", "family": "bind", "template": 2, "depth": 5, "pred": "62", "truth": "69", "correct": false, "edits_per_token": 2.0}, {"uid": "8441e8c811ca2537", "family": "bind", "template": 2, "depth": 5, "pred": "10", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "2b9a84b2ec1c5d36", "family": "bind", "template": 2, "depth": 5, "pred": "680", "truth": "1116", "correct": false, "edits_per_token": 2.0}, {"uid": "e670f4c748204cc8", "family": "bind", "template": 2, "depth": 5, "pred": "134", "truth": "102", "correct": false, "edits_per_token": 2.0}, {"uid": "f2b0d3fe9038acd2", "family": "arith", "template": 2, "depth": 6, "pred": "45", "truth": "42", "correct": false, "edits_per_token": 2.0}, {"uid": "daacb1ff7411a2c2", "family": "arith", "template": 2, "depth": 6, "pred": "90", "truth": "101", "correct": false, "edits_per_token": 2.0}, {"uid": "f17398d768802f6b", "family": "bind", "template": 2, "depth": 5, "pred": "95", "truth": "95", "correct": true, "edits_per_token": 2.0}, {"uid": "a881a48bc9d6d2fa", "family": "arith", "template": 2, "depth": 6, "pred": "-3", "truth": "-53", "correct": false, "edits_per_token": 2.0}, {"uid": "5f00de1c8f6a727b", "family": "bind", "template": 2, "depth": 5, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 2.0}, {"uid": "56f549cdb3619b33", "family": "bind", "template": 0, "depth": 5, "pred": "21", "truth": "29", "correct": false, "edits_per_token": 2.0}, {"uid": "70e96a9cddee5517", "family": "bind", "template": 0, "depth": 5, "pred": "134", "truth": "179", "correct": false, "edits_per_token": 2.0}, {"uid": "ea1484f36c572696", "family": "bind", "template": 0, "depth": 5, "pred": "70", "truth": "80", "correct": false, "edits_per_token": 2.0}, {"uid": "54a6e84b79b6c7df", "family": "bind", "template": 0, "depth": 5, "pred": "-12", "truth": "-12", "correct": true, "edits_per_token": 2.0}, {"uid": "fec334e2c8b64794", "family": "bind", "template": 0, "depth": 5, "pred": "71", "truth": "81", "correct": false, "edits_per_token": 2.0}, {"uid": "c3f0503818c31297", "family": "bind", "template": 0, "depth": 5, "pred": "93", "truth": "89", "correct": false, "edits_per_token": 2.0}, {"uid": "a978e3f3b6818825", "family": "bind", "template": 0, "depth": 5, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 2.0}, {"uid": "33b72f3d87631e86", "family": "bind", "template": 0, "depth": 5, "pred": "315", "truth": "667", "correct": false, "edits_per_token": 2.0}, {"uid": "4410175b31fd9ff9", "family": "bind", "template": 0, "depth": 5, "pred": "240", "truth": "192", "correct": false, "edits_per_token": 2.0}, {"uid": "177037480b40585d", "family": "bind", "template": 0, "depth": 5, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "bf86bbb1610dd587", "family": "bind", "template": 1, "depth": 5, "pred": "648", "truth": "588", "correct": false, "edits_per_token": 2.0}, {"uid": "82375062bef2aad9", "family": "bind", "template": 0, "depth": 5, "pred": "75", "truth": "85", "correct": false, "edits_per_token": 2.0}, {"uid": "084f23e813b1c2f2", "family": "bind", "template": 0, "depth": 5, "pred": "33", "truth": "40", "correct": false, "edits_per_token": 2.0}, {"uid": "dca41c3ec45e2945", "family": "bind", "template": 1, "depth": 5, "pred": "300", "truth": "580", "correct": false, "edits_per_token": 2.0}, {"uid": "ee7f986fce1fe6fc", "family": "bind", "template": 0, "depth": 5, "pred": "34", "truth": "41", "correct": false, "edits_per_token": 2.0}, {"uid": "c731265cddc84c46", "family": "bind", "template": 0, "depth": 5, "pred": "90", "truth": "101", "correct": false, "edits_per_token": 2.0}, {"uid": "9a41a8cfde6b00c6", "family": "bind", "template": 1, "depth": 5, "pred": "-14", "truth": "-14", "correct": true, "edits_per_token": 2.0}, {"uid": "8d19dd38ce14dfc5", "family": "bind", "template": 1, "depth": 5, "pred": "335", "truth": "2076", "correct": false, "edits_per_token": 2.0}, {"uid": "677f597fd752b712", "family": "bind", "template": 1, "depth": 5, "pred": "-15", "truth": "-19", "correct": false, "edits_per_token": 2.0}, {"uid": "1cec2e6bbdfaf026", "family": "bind", "template": 1, "depth": 5, "pred": "57", "truth": "56", "correct": false, "edits_per_token": 2.0}, {"uid": "d7f0bbe7ddd4451f", "family": "bind", "template": 1, "depth": 5, "pred": "10", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "f6d1f2f6e260e0e3", "family": "bind", "template": 1, "depth": 5, "pred": "100", "truth": "90", "correct": false, "edits_per_token": 2.0}, {"uid": "c18e4b7f918c9f69", "family": "bind", "template": 1, "depth": 5, "pred": "20", "truth": "24", "correct": false, "edits_per_token": 2.0}, {"uid": "90c73ad098280857", "family": "bind", "template": 1, "depth": 5, "pred": "-56", "truth": "-69", "correct": false, "edits_per_token": 2.0}, {"uid": "a2750a77af5a3ed4", "family": "bind", "template": 1, "depth": 5, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "beccfdc7b9edf6a8", "family": "bind", "template": 2, "depth": 6, "pred": "145", "truth": "280", "correct": false, "edits_per_token": 2.0}, {"uid": "dfd3cbb03a444dec", "family": "bind", "template": 2, "depth": 6, "pred": "43", "truth": "60", "correct": false, "edits_per_token": 2.0}, {"uid": "5da4fa98b5246175", "family": "bind", "template": 2, "depth": 6, "pred": "36", "truth": "47", "correct": false, "edits_per_token": 2.0}, {"uid": "e8bdc504c2e559a6", "family": "bind", "template": 2, "depth": 6, "pred": "76", "truth": "116", "correct": false, "edits_per_token": 2.0}, {"uid": "576442ddb2643bf3", "family": "bind", "template": 2, "depth": 6, "pred": "70", "truth": "107", "correct": false, "edits_per_token": 2.0}, {"uid": "90935ad82127780b", "family": "fsm", "template": 2, "depth": 5, "pred": "28", "truth": "28", "correct": true, "edits_per_token": 2.0}, {"uid": "efd0c340700fd585", "family": "bind", "template": 1, "depth": 5, "pred": "107", "truth": "129", "correct": false, "edits_per_token": 2.0}, {"uid": "8a00b1ceb8cc21d0", "family": "bind", "template": 2, "depth": 6, "pred": "-11", "truth": "-8", "correct": false, "edits_per_token": 2.0}, {"uid": "1710e2024ce0188e", "family": "bind", "template": 2, "depth": 6, "pred": "172", "truth": "582", "correct": false, "edits_per_token": 2.0}, {"uid": "332068c4dc811926", "family": "bind", "template": 2, "depth": 6, "pred": "-14", "truth": "-15", "correct": false, "edits_per_token": 2.0}, {"uid": "47e5102caba8b95d", "family": "bind", "template": 2, "depth": 6, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 2.0}, {"uid": "9193a977b5e6d754", "family": "bind", "template": 2, "depth": 6, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "69b42fd072531922", "family": "bind", "template": 1, "depth": 5, "pred": "-7", "truth": "-4", "correct": false, "edits_per_token": 2.0}, {"uid": "ad4b55e83f33e0e9", "family": "bind", "template": 2, "depth": 6, "pred": "27", "truth": "38", "correct": false, "edits_per_token": 2.0}, {"uid": "6a42a33ed3b404ed", "family": "bind", "template": 2, "depth": 6, "pred": "-2", "truth": "-4", "correct": false, "edits_per_token": 2.0}, {"uid": "5885e5d090992cd2", "family": "bind", "template": 2, "depth": 6, "pred": "-12", "truth": "-14", "correct": false, "edits_per_token": 2.0}, {"uid": "db61f97b4eb08ce6", "family": "fsm", "template": 2, "depth": 5, "pred": "9", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "1a52ddcf13f295de", "family": "fsm", "template": 2, "depth": 5, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 2.0}, {"uid": "4764ec83e7c8c9b6", "family": "fsm", "template": 2, "depth": 5, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "cf3e9f98a9edf153", "family": "fsm", "template": 1, "depth": 5, "pred": "27", "truth": "29", "correct": false, "edits_per_token": 2.0}, {"uid": "485f2555fb5eff21", "family": "bind", "template": 0, "depth": 6, "pred": "2", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "76f66b2bb4f4d200", "family": "bind", "template": 0, "depth": 6, "pred": "93", "truth": "108", "correct": false, "edits_per_token": 2.0}, {"uid": "8a0414e10232f5c1", "family": "bind", "template": 0, "depth": 6, "pred": "34", "truth": "52", "correct": false, "edits_per_token": 2.0}, {"uid": "6fa2bc31af37d28d", "family": "bind", "template": 0, "depth": 6, "pred": "46", "truth": "101", "correct": false, "edits_per_token": 2.0}, {"uid": "52178894c71a55fc", "family": "bind", "template": 0, "depth": 6, "pred": "140", "truth": "204", "correct": false, "edits_per_token": 2.0}, {"uid": "bf993b5b68923b56", "family": "bind", "template": 0, "depth": 6, "pred": "34", "truth": "27", "correct": false, "edits_per_token": 2.0}, {"uid": "c5ee1489b5a7552b", "family": "bind", "template": 0, "depth": 6, "pred": "108", "truth": "268", "correct": false, "edits_per_token": 2.0}, {"uid": "2dbbd53f53e30073", "family": "bind", "template": 0, "depth": 6, "pred": "62", "truth": "78", "correct": false, "edits_per_token": 2.0}, {"uid": "7437da9eb53eb3a0", "family": "bind", "template": 0, "depth": 6, "pred": "11", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "408a34c94f17f5d7", "family": "bind", "template": 0, "depth": 6, "pred": "54", "truth": "120", "correct": false, "edits_per_token": 2.0}, {"uid": "1bad57bf846df87e", "family": "bind", "template": 0, "depth": 6, "pred": "16", "truth": "-1", "correct": false, "edits_per_token": 2.0}, {"uid": "3d0794faec305528", "family": "bind", "template": 1, "depth": 6, "pred": "4", "truth": "15", "correct": false, "edits_per_token": 2.0}, {"uid": "10c6fe8e473b05f7", "family": "bind", "template": 1, "depth": 6, "pred": "-12", "truth": "-13", "correct": false, "edits_per_token": 2.0}, {"uid": "897766cab535c928", "family": "bind", "template": 1, "depth": 6, "pred": "50", "truth": "63", "correct": false, "edits_per_token": 2.0}, {"uid": "37fcdefdc142c5c8", "family": "bind", "template": 1, "depth": 6, "pred": "-100", "truth": "-520", "correct": false, "edits_per_token": 2.0}, {"uid": "0fa8fa84eab24d5b", "family": "bind", "template": 1, "depth": 6, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "e17850521f774ccb", "family": "bind", "template": 1, "depth": 6, "pred": "-36", "truth": "-57", "correct": false, "edits_per_token": 2.0}, {"uid": "87604ac5e2432117", "family": "bind", "template": 0, "depth": 6, "pred": "-2", "truth": "-2", "correct": true, "edits_per_token": 2.0}, {"uid": "1bf2c062e3d35770", "family": "bind", "template": 0, "depth": 6, "pred": "-14", "truth": "-21", "correct": false, "edits_per_token": 2.0}, {"uid": "3e71f658a8d01416", "family": "bind", "template": 1, "depth": 6, "pred": "202", "truth": "612", "correct": false, "edits_per_token": 2.0}, {"uid": "6d9120572acc5f8c", "family": "bind", "template": 1, "depth": 6, "pred": "141", "truth": "421", "correct": false, "edits_per_token": 2.0}, {"uid": "262d1886f373a193", "family": "bind", "template": 1, "depth": 6, "pred": "48", "truth": "192", "correct": false, "edits_per_token": 2.0}, {"uid": "2df700b079b1ea09", "family": "bind", "template": 1, "depth": 6, "pred": "514", "truth": "254", "correct": false, "edits_per_token": 2.0}, {"uid": "9f22ff248a27d85b", "family": "bind", "template": 1, "depth": 6, "pred": "18", "truth": "21", "correct": false, "edits_per_token": 2.0}, {"uid": "b86f3a062bdd7b14", "family": "bind", "template": 1, "depth": 6, "pred": "24", "truth": "28", "correct": false, "edits_per_token": 2.0}, {"uid": "0ec61c8d74b0b12e", "family": "bind", "template": 1, "depth": 6, "pred": "-5", "truth": "-4", "correct": false, "edits_per_token": 2.0}, {"uid": "7e69b09c478d7cf0", "family": "bind", "template": 1, "depth": 6, "pred": "-33", "truth": "-103", "correct": false, "edits_per_token": 2.0}, {"uid": "3644e4d7e72723f2", "family": "fsm", "template": 2, "depth": 5, "pred": "11", "truth": "11", "correct": true, "edits_per_token": 2.0}, {"uid": "ef5c74f822ed4d09", "family": "fsm", "template": 2, "depth": 5, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 2.0}, {"uid": "1d8a092a5663c2ca", "family": "fsm", "template": 2, "depth": 6, "pred": "16", "truth": "17", "correct": false, "edits_per_token": 2.0}, {"uid": "13b840f1309c3d23", "family": "fsm", "template": 2, "depth": 6, "pred": "16", "truth": "27", "correct": false, "edits_per_token": 2.0}, {"uid": "3bf1de404d07020e", "family": "fsm", "template": 1, "depth": 5, "pred": "30", "truth": "24", "correct": false, "edits_per_token": 2.0}, {"uid": "cca744ebe723c85b", "family": "fsm", "template": 2, "depth": 5, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "976535ab3eb31bd8", "family": "fsm", "template": 2, "depth": 5, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 2.0}, {"uid": "93ee3b76c1253b5f", "family": "fsm", "template": 0, "depth": 5, "pred": "8", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "fa364f44c6759d61", "family": "fsm", "template": 0, "depth": 5, "pred": "59", "truth": "63", "correct": false, "edits_per_token": 2.0}, {"uid": "06585bc8d35c4f52", "family": "fsm", "template": 1, "depth": 5, "pred": "16", "truth": "22", "correct": false, "edits_per_token": 2.0}, {"uid": "cb31b4e31c03606f", "family": "fsm", "template": 1, "depth": 5, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 2.0}, {"uid": "0324b1035a1b5e3b", "family": "fsm", "template": 1, "depth": 5, "pred": "24", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "70965bfd44497a12", "family": "fsm", "template": 2, "depth": 6, "pred": "8", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "9e2ced0a36e30f08", "family": "fsm", "template": 2, "depth": 6, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 2.0}, {"uid": "945b988367f1262b", "family": "fsm", "template": 2, "depth": 5, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "49d61413398cc89b", "family": "fsm", "template": 2, "depth": 6, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 2.0}, {"uid": "cac6ffd4670e5f50", "family": "fsm", "template": 2, "depth": 6, "pred": "12", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "ec003e386e949698", "family": "fsm", "template": 0, "depth": 5, "pred": "4", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "b4c4da03d549d2ad", "family": "fsm", "template": 0, "depth": 5, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 2.0}, {"uid": "d4906001b8fd5965", "family": "fsm", "template": 2, "depth": 5, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 2.0}, {"uid": "054e529ea5d13efe", "family": "fsm", "template": 2, "depth": 5, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "af25cc7ee3868a05", "family": "fsm", "template": 0, "depth": 5, "pred": "11", "truth": "11", "correct": true, "edits_per_token": 2.0}, {"uid": "e6eb0871661f8cb0", "family": "fsm", "template": 2, "depth": 6, "pred": "9", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "d981e261ba455e59", "family": "fsm", "template": 1, "depth": 5, "pred": "4", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "e19b1ade5034200f", "family": "fsm", "template": 0, "depth": 5, "pred": "10", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "9084157c9bdc265e", "family": "fsm", "template": 0, "depth": 5, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 2.0}, {"uid": "966d9784f516b0b4", "family": "fsm", "template": 2, "depth": 5, "pred": "-1", "truth": "0", "correct": false, "edits_per_token": 2.0}, {"uid": "7e2c6414320592ae", "family": "fsm", "template": 0, "depth": 6, "pred": "15", "truth": "17", "correct": false, "edits_per_token": 2.0}, {"uid": "b06f1eb977fbe9e0", "family": "fsm", "template": 1, "depth": 6, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 2.0}, {"uid": "fa3b7f75f9e43b2f", "family": "fsm", "template": 1, "depth": 6, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "c7498914bccd7bf3", "family": "fsm", "template": 0, "depth": 5, "pred": "7", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "195ef7d3ccac4b8b", "family": "fsm", "template": 0, "depth": 5, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "7cb457b70a7065ab", "family": "fsm", "template": 1, "depth": 5, "pred": "8", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "4688a1575da3feae", "family": "fsm", "template": 2, "depth": 6, "pred": "-1", "truth": "-7", "correct": false, "edits_per_token": 2.0}, {"uid": "13d7a0661383bf5c", "family": "fsm", "template": 0, "depth": 6, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "8520bf5fba316dc5", "family": "fsm", "template": 0, "depth": 6, "pred": "13", "truth": "18", "correct": false, "edits_per_token": 2.0}, {"uid": "28a5212e37e34ede", "family": "fsm", "template": 2, "depth": 6, "pred": "-2", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "c1db7537b69a1588", "family": "fsm", "template": 1, "depth": 5, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 2.0}, {"uid": "475faa1c40dd1c57", "family": "fsm", "template": 2, "depth": 6, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 2.0}, {"uid": "82de054bd19fee8f", "family": "fsm", "template": 2, "depth": 6, "pred": "-20", "truth": "-18", "correct": false, "edits_per_token": 2.0}, {"uid": "1599bc918482f147", "family": "fsm", "template": 2, "depth": 6, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "f13662f0f0c30826", "family": "fsm", "template": 0, "depth": 5, "pred": "6", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "b7e69cea18577e61", "family": "fsm", "template": 0, "depth": 5, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 2.0}, {"uid": "f01f401147209180", "family": "fsm", "template": 1, "depth": 6, "pred": "12", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "e45b754845391f3f", "family": "fsm", "template": 1, "depth": 5, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 2.0}, {"uid": "babac6e1d2340f83", "family": "fsm", "template": 1, "depth": 6, "pred": "0", "truth": "-2", "correct": false, "edits_per_token": 2.0}, {"uid": "f3dfeb8093398fba", "family": "fsm", "template": 2, "depth": 5, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "83156af5c9f2102f", "family": "fsm", "template": 1, "depth": 6, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 2.0}, {"uid": "436a870501934e95", "family": "fsm", "template": 2, "depth": 6, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 2.0}, {"uid": "17c22d2c6da7016d", "family": "fsm", "template": 0, "depth": 5, "pred": "1", "truth": "0", "correct": false, "edits_per_token": 2.0}, {"uid": "958a99eb728e15c5", "family": "fsm", "template": 0, "depth": 5, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 2.0}, {"uid": "e969fccd3aa5aa4d", "family": "fsm", "template": 1, "depth": 5, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 2.0}, {"uid": "9e5d597a794e81e2", "family": "fsm", "template": 0, "depth": 5, "pred": "1", "truth": "0", "correct": false, "edits_per_token": 2.0}, {"uid": "53a3aa34d09ae7f0", "family": "fsm", "template": 0, "depth": 6, "pred": "13", "truth": "15", "correct": false, "edits_per_token": 2.0}, {"uid": "7ffa08ddf97e22b8", "family": "fsm", "template": 0, "depth": 6, "pred": "6", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "df4e72237d80a81c", "family": "fsm", "template": 1, "depth": 5, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "95a0ef58216182a2", "family": "fsm", "template": 1, "depth": 6, "pred": "0", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "6c413e46c2f2d918", "family": "fsm", "template": 1, "depth": 6, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 2.0}, {"uid": "38b00c5c8332094f", "family": "fsm", "template": 0, "depth": 6, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 2.0}, {"uid": "c571f1db246e3ac7", "family": "fsm", "template": 1, "depth": 5, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 2.0}, {"uid": "c8c65c1d9220669b", "family": "fsm", "template": 1, "depth": 6, "pred": "18", "truth": "28", "correct": false, "edits_per_token": 2.0}, {"uid": "2bd960c30ea3d343", "family": "fsm", "template": 0, "depth": 6, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 2.0}, {"uid": "dbd3b4ea0faa4bd5", "family": "fsm", "template": 0, "depth": 6, "pred": "6", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "ad56dff0f6c52a6e", "family": "fsm", "template": 0, "depth": 6, "pred": "-14", "truth": "-16", "correct": false, "edits_per_token": 2.0}, {"uid": "5aa303b1e8a3a50a", "family": "fsm", "template": 1, "depth": 6, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "a7aa7192117d2b91", "family": "fsm", "template": 0, "depth": 6, "pred": "13", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "3663d9bfed0657ac", "family": "fsm", "template": 1, "depth": 6, "pred": "-2", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "d442b8b4e9c4463b", "family": "fsm", "template": 1, "depth": 6, "pred": "11", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "f5d3a7b825df11db", "family": "fsm", "template": 0, "depth": 6, "pred": "7", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "5fae584fb4b0806f", "family": "fsm", "template": 1, "depth": 5, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 2.0}, {"uid": "f5d6e652e3c1c9c6", "family": "fsm", "template": 1, "depth": 6, "pred": "12", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "fae83630cc69d997", "family": "fsm", "template": 1, "depth": 6, "pred": "-5", "truth": "-7", "correct": false, "edits_per_token": 2.0}, {"uid": "ac2e2426143b2f6a", "family": "fsm", "template": 1, "depth": 6, "pred": "-7", "truth": "-8", "correct": false, "edits_per_token": 2.0}, {"uid": "3e7729193c5d67ef", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 2.0}, {"uid": "b040bca9ccdd27d8", "family": "fsm", "template": 0, "depth": 6, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "ae8d496071780205", "family": "fsm", "template": 0, "depth": 6, "pred": "6", "truth": "-4", "correct": false, "edits_per_token": 2.0}, {"uid": "48b2c33f5bfc7b39", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "ffb3a068ddee91d8", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "0b5d578a6414c59b", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "plum fig", "correct": false, "edits_per_token": 2.0}, {"uid": "5457f87f4a4d9928", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "29cc176fd2bd357a", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "9a37aa52f6f4f0ef", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "yam", "correct": false, "edits_per_token": 2.0}, {"uid": "f6e2b9995ac5a2af", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "04e6cd46e148e861", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "c53436531a9e4322", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "kiwi lime", "correct": false, "edits_per_token": 2.0}, {"uid": "8c39602b8320d36d", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "451ab7989b156691", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "plum", "correct": false, "edits_per_token": 2.0}, {"uid": "64e6b2f20007b04f", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "06695366b2ca7f94", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "fe21b5c807616138", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "b3aeac08c3bf5403", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "7b4a99cf741c62f4", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 2.0}, {"uid": "544e5524578ed1a2", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 2.0}, {"uid": "8813abcc63108e4e", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "6ef02a442804899d", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "9542b67b66fba534", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "bea23de3fd39e931", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "f862e4eb48abf854", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "1e956d986d80a2e8", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "8c26836a775dfeef", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 2.0}, {"uid": "dbc95bbfbdbb8c83", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "589bdb5b41da7c65", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "e61d3770c79cde47", "family": "sched", "template": 0, "depth": 5, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 2.0}, {"uid": "2fbd02a75eab5edf", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "bc00b85f3a4929cb", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "83d9e53bcbe7935d", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "19147a5ccac0f776", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "b6995c56551edd0f", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "597331784c8adb06", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 2.0}, {"uid": "7a4868e5e81669d0", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "6ab7960babdc2e4d", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 2.0}, {"uid": "8c9f86f29e505eb5", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "7d3f18625f00d8ee", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "1e7febaac638433d", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "192270ee3b6c26d9", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "2b26c9c6928a8458", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "65550dd4f783d103", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "637f9d17aabeb31c", "family": "sched", "template": 1, "depth": 5, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "ed8754896958fc23", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "date", "correct": false, "edits_per_token": 2.0}, {"uid": "daf704f320ba7df5", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 2.0}, {"uid": "72ac559ac49d72ce", "family": "sched", "template": 1, "depth": 5, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "b7380cfd5017d3d5", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "7bfb28727e572f4f", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 2.0}, {"uid": "5a7f7b85164f7d43", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "bean", "correct": false, "edits_per_token": 2.0}, {"uid": "5a9f6a2d1fca7c30", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "5dfac4f4fcff919d", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "c877e5915a1d82dd", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "91d7e38041c84a54", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "d88ead16fc9cb4dd", "family": "sched", "template": 2, "depth": 5, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 2.0}, {"uid": "0470c678ba5e33bd", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 2.0}, {"uid": "2004484848062f39", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "4aec4964885ab8cd", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "f62aa798063a3946", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "5821393361b27572", "family": "sched", "template": 2, "depth": 5, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "9835b0f9d397f634", "family": "sched", "template": 2, "depth": 5, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "efd0e2bbf109da63", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "66c464d4968a2fe2", "family": "sched", "template": 0, "depth": 5, "pred": "none", "truth": "D", "correct": false, "edits_per_token": 2.0}, {"uid": "ae5e2670afd5d884", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "738b6d8b10bdecc1", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "31e0d3601d129675", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "8c334375cbdeecbb", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "3146c4b45bd4ba36", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "16bc96ba9e32042e", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "7dc88230188b5ae0", "family": "sched", "template": 0, "depth": 5, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 2.0}, {"uid": "ffdf323f5b0e8398", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 2.0}, {"uid": "031f5a4ec368d5ed", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "249d68235fab3319", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "ac5f146d3cf4c744", "family": "sched", "template": 0, "depth": 5, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "85e6171c2bff77ec", "family": "sched", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "a5eb1ad33216a365", "family": "sched", "template": 0, "depth": 5, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "fbfb9d7ebbd9ab42", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 2.0}, {"uid": "93e46dc65f4a0d60", "family": "sched", "template": 2, "depth": 6, "pred": "Dxclude tas", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "7d44befcfd949a85", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "dfe372c23262434c", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "c68ec9cd887646d5", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "a01c26ea9b3e73d4", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 2.0}, {"uid": "6c51902842f6a23b", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 2.0}, {"uid": "1d040a988ed2fa83", "family": "sched", "template": 2, "depth": 5, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 2.0}, {"uid": "165a925a98ad694c", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "e97fa901d509ffd6", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 2.0}, {"uid": "3676d84af524d9ae", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 2.0}, {"uid": "cd0b4315898f7965", "family": "sched", "template": 2, "depth": 5, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "7429be63b9fe2121", "family": "sched", "template": 0, "depth": 6, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "c696f5f9f596f2f7", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 2.0}, {"uid": "59f92777461a61ec", "family": "sched", "template": 0, "depth": 6, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "fb0fcdfe7002426e", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "871d1deab44403c5", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "plum", "correct": false, "edits_per_token": 2.0}, {"uid": "e09fe1db1040c253", "family": "sched", "template": 0, "depth": 5, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "70d22e9191a63ed3", "family": "sched", "template": 0, "depth": 5, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "b5b6b8f1e1b8e526", "family": "sched", "template": 2, "depth": 5, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "eb52c5c1389933b9", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "ae5ef219347ca8d1", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "bc760caf57d4fbeb", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "342a44e350c2c803", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "plum date", "correct": false, "edits_per_token": 2.0}, {"uid": "c1f8bff397b18871", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "fig", "correct": false, "edits_per_token": 2.0}, {"uid": "2315692be3c9225f", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "yam", "correct": false, "edits_per_token": 2.0}, {"uid": "8eb603b3bcb2b694", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "588fd0833653c113", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "1e067fba49f1bde2", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "7216a64146ae6acd", "family": "sched", "template": 1, "depth": 5, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "7845079b2715c407", "family": "sched", "template": 1, "depth": 5, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "5733b7ed7ff6f08e", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 2.0}, {"uid": "9565161477fd02c2", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "f3390e99576c3f52", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "685bcd1984a9ef37", "family": "sched", "template": 1, "depth": 6, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 2.0}, {"uid": "127824e03ba4e18f", "family": "sched", "template": 1, "depth": 6, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "7af3f6bc2435a9c3", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "af0de551a61187f6", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "bdaff9107551bcad", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "d5369281fc9cfcd4", "family": "sched", "template": 1, "depth": 5, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "994412f1bf4510f1", "family": "sched", "template": 1, "depth": 6, "pred": "B", "truth": "F", "correct": false, "edits_per_token": 2.0}, {"uid": "b1337bc8478cf188", "family": "sched", "template": 1, "depth": 6, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 2.0}, {"uid": "1591f1da64128221", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "bean", "correct": false, "edits_per_token": 2.0}, {"uid": "4ace02760358e0aa", "family": "sched", "template": 1, "depth": 5, "pred": "C", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "c38781630980610c", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "5cdeab7b7d3d00a8", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 2.0}, {"uid": "c0b33bc616c270db", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "ae3d111057f9a376", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "0dbf96f152ee57cd", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 2.0}, {"uid": "2ec492b6141d5723", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 2.0}, {"uid": "a8895e2feee026d1", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "63b451d568c16ffa", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "7a75d953e8308610", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 2.0}, {"uid": "551b7796162a1a33", "family": "sched", "template": 1, "depth": 5, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "0409fe31b46650ef", "family": "sched", "template": 0, "depth": 6, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "6fcf7fc1691a79a5", "family": "sched", "template": 2, "depth": 6, "pred": "A", "truth": "F", "correct": false, "edits_per_token": 2.0}, {"uid": "52e4c2db61cbda34", "family": "sched", "template": 0, "depth": 6, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 2.0}, {"uid": "6b2e6485c06a7570", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "e7b6cecd9fd15087", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "f1ca4b2353614cc4", "family": "sched", "template": 0, "depth": 6, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 2.0}, {"uid": "f71e8f3bf0d91481", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "227f0ada257e9567", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "0847283e1b47f87f", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "5c926222dc33129d", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 2.0}, {"uid": "6a5adbd9ac262749", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "dbccdf3934cf6baf", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "d3979db4d8709e58", "family": "sched", "template": 0, "depth": 6, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "da9eddd4e681af1a", "family": "sched", "template": 0, "depth": 6, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "bb2c190eae3c2125", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "7257616f0bb8b273", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "b442b080ca1d58bf", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "1762aafe566bb2f0", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "E", "correct": false, "edits_per_token": 2.0}, {"uid": "3478623940a0babb", "family": "sched", "template": 0, "depth": 6, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "d49274b1b41b7b91", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "54953d3704253d0a", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "4b3ba50fe8aed021", "family": "sched", "template": 1, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 2.0}, {"uid": "1564dfd3f2b289c5", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "420f41111b0b1478", "family": "sched", "template": 1, "depth": 6, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "fb5eb61f7d51b656", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "b3cf93f47535eec2", "family": "sched", "template": 1, "depth": 6, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "a98861d878768980", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "8c0cce3a8cc7a8a8", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "139fdebe4be35cb8", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "bd0288f2ce913d48", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "39f4a4e458ce30d0", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}] | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_depth_b8.json
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +[{"uid": "05a9ab128b8f795a", "family": "arith", "template": 0, "depth": 5, "pred": "204", "truth": "206", "correct": false, "edits_per_token": 8.0}, {"uid": "ce77dc7f6121c65c", "family": "arith", "template": 0, "depth": 5, "pred": "3173", "truth": "3150", "correct": false, "edits_per_token": 8.0}, {"uid": "6d86083e9a296d70", "family": "arith", "template": 0, "depth": 5, "pred": "614", "truth": "467", "correct": false, "edits_per_token": 8.0}, {"uid": "bbbc075e9ed2d83b", "family": "arith", "template": 0, "depth": 5, "pred": "16480", "truth": "13248", "correct": false, "edits_per_token": 8.0}, {"uid": "9cc4e3fd8e271e9e", "family": "arith", "template": 0, "depth": 5, "pred": "1080", "truth": "1800", "correct": false, "edits_per_token": 8.0}, {"uid": "d93003ac9ddbe92e", "family": "arith", "template": 1, "depth": 5, "pred": "-12", "truth": "18", "correct": false, "edits_per_token": 8.0}, {"uid": "0429a41c21791b48", "family": "arith", "template": 1, "depth": 5, "pred": "408", "truth": "485", "correct": false, "edits_per_token": 8.0}, {"uid": "252188905e760471", "family": "arith", "template": 1, "depth": 5, "pred": "20", "truth": "43", "correct": false, "edits_per_token": 8.0}, {"uid": "9d3b6d51fa8f268d", "family": "arith", "template": 1, "depth": 5, "pred": "-300", "truth": "-320", "correct": false, "edits_per_token": 8.0}, {"uid": "c38dbe1ef3f315bb", "family": "arith", "template": 0, "depth": 5, "pred": "-58", "truth": "-63", "correct": false, "edits_per_token": 8.0}, {"uid": "4ee8682a5c7cde40", "family": "arith", "template": 0, "depth": 5, "pred": "-10", "truth": "-7", "correct": false, "edits_per_token": 8.0}, {"uid": "213e53daae3f0053", "family": "arith", "template": 0, "depth": 5, "pred": "1688", "truth": "1113", "correct": false, "edits_per_token": 8.0}, {"uid": "8a499c6f7286285c", "family": "arith", "template": 1, "depth": 5, "pred": "-41", "truth": "-47", "correct": false, "edits_per_token": 8.0}, {"uid": "c7cc1d12232437f5", "family": "arith", "template": 1, "depth": 5, "pred": "553", "truth": "411", "correct": false, "edits_per_token": 8.0}, {"uid": "72633b6ff3769cb5", "family": "arith", "template": 1, "depth": 5, "pred": "34", "truth": "-3", "correct": false, "edits_per_token": 8.0}, {"uid": "50c2617c879f93b6", "family": "arith", "template": 1, "depth": 5, "pred": "148", "truth": "814", "correct": false, "edits_per_token": 8.0}, {"uid": "b2bc5413c8fada82", "family": "arith", "template": 1, "depth": 5, "pred": "607", "truth": "686", "correct": false, "edits_per_token": 8.0}, {"uid": "a8e4058450eb65d2", "family": "arith", "template": 1, "depth": 5, "pred": "134", "truth": "140", "correct": false, "edits_per_token": 8.0}, {"uid": "c709c22eb9084b6b", "family": "arith", "template": 0, "depth": 5, "pred": "-51", "truth": "-39", "correct": false, "edits_per_token": 8.0}, {"uid": "58498d59ed0e6ec5", "family": "arith", "template": 0, "depth": 5, "pred": "380", "truth": "387", "correct": false, "edits_per_token": 8.0}, {"uid": "d79b98b965dbc325", "family": "arith", "template": 0, "depth": 5, "pred": "48", "truth": "55", "correct": false, "edits_per_token": 8.0}, {"uid": "ac935e06e7c21bd6", "family": "arith", "template": 0, "depth": 5, "pred": "80", "truth": "54", "correct": false, "edits_per_token": 8.0}, {"uid": "32d9a5b74df38a67", "family": "arith", "template": 0, "depth": 5, "pred": "288", "truth": "303", "correct": false, "edits_per_token": 8.0}, {"uid": "bea3727c59f821e2", "family": "arith", "template": 0, "depth": 5, "pred": "68", "truth": "71", "correct": false, "edits_per_token": 8.0}, {"uid": "3d7cf92565334823", "family": "arith", "template": 1, "depth": 5, "pred": "-14", "truth": "-15", "correct": false, "edits_per_token": 8.0}, {"uid": "34ad0dad384435be", "family": "arith", "template": 1, "depth": 5, "pred": "-34", "truth": "-40", "correct": false, "edits_per_token": 8.0}, {"uid": "55f03d61b988deab", "family": "arith", "template": 1, "depth": 5, "pred": "10", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "27df0cd1c098a929", "family": "arith", "template": 0, "depth": 6, "pred": "2888", "truth": "11513", "correct": false, "edits_per_token": 8.0}, {"uid": "17fdac85f59e039f", "family": "arith", "template": 0, "depth": 6, "pred": "-11080", "truth": "-23292", "correct": false, "edits_per_token": 8.0}, {"uid": "c446df844a6f1622", "family": "arith", "template": 0, "depth": 6, "pred": "3004", "truth": "2742", "correct": false, "edits_per_token": 8.0}, {"uid": "a9dcf9180b73a327", "family": "arith", "template": 0, "depth": 6, "pred": "91", "truth": "153", "correct": false, "edits_per_token": 8.0}, {"uid": "bc63707555d01f5f", "family": "arith", "template": 1, "depth": 6, "pred": "208", "truth": "92", "correct": false, "edits_per_token": 8.0}, {"uid": "a2b0c9415ed1725f", "family": "arith", "template": 1, "depth": 6, "pred": "1350", "truth": "1260", "correct": false, "edits_per_token": 8.0}, {"uid": "8dde8846d9d18b1b", "family": "arith", "template": 1, "depth": 6, "pred": "-303", "truth": "-315", "correct": false, "edits_per_token": 8.0}, {"uid": "494a839dbd0029ee", "family": "arith", "template": 1, "depth": 6, "pred": "2100", "truth": "11670", "correct": false, "edits_per_token": 8.0}, {"uid": "fb29f63f32314415", "family": "arith", "template": 1, "depth": 6, "pred": "3172", "truth": "597", "correct": false, "edits_per_token": 8.0}, {"uid": "f552c2e836022442", "family": "arith", "template": 0, "depth": 6, "pred": "205", "truth": "301", "correct": false, "edits_per_token": 8.0}, {"uid": "fa5a1426f63c833a", "family": "arith", "template": 0, "depth": 6, "pred": "408", "truth": "322", "correct": false, "edits_per_token": 8.0}, {"uid": "302be3fc789128e5", "family": "arith", "template": 1, "depth": 6, "pred": "724", "truth": "824", "correct": false, "edits_per_token": 8.0}, {"uid": "385a2e8bc4104ed9", "family": "arith", "template": 1, "depth": 6, "pred": "-1004", "truth": "-910", "correct": false, "edits_per_token": 8.0}, {"uid": "077321c4dae60411", "family": "arith", "template": 1, "depth": 6, "pred": "124", "truth": "270", "correct": false, "edits_per_token": 8.0}, {"uid": "bf56aaa4649ea27f", "family": "arith", "template": 0, "depth": 6, "pred": "91", "truth": "82", "correct": false, "edits_per_token": 8.0}, {"uid": "11486f50919b4ea9", "family": "arith", "template": 0, "depth": 6, "pred": "953", "truth": "8026", "correct": false, "edits_per_token": 8.0}, {"uid": "e35111f507bfc75e", "family": "arith", "template": 0, "depth": 6, "pred": "4102", "truth": "3771", "correct": false, "edits_per_token": 8.0}, {"uid": "276bee3fb394936b", "family": "arith", "template": 0, "depth": 6, "pred": "993", "truth": "788", "correct": false, "edits_per_token": 8.0}, {"uid": "ec56a5b3f62592e2", "family": "arith", "template": 0, "depth": 6, "pred": "124", "truth": "101", "correct": false, "edits_per_token": 8.0}, {"uid": "aa399c0d7a7c20a0", "family": "arith", "template": 1, "depth": 6, "pred": "-38", "truth": "-41", "correct": false, "edits_per_token": 8.0}, {"uid": "2ac019c5c4b6627f", "family": "arith", "template": 1, "depth": 6, "pred": "-800", "truth": "-1260", "correct": false, "edits_per_token": 8.0}, {"uid": "af2e45eba90c6547", "family": "arith", "template": 1, "depth": 6, "pred": "169", "truth": "171", "correct": false, "edits_per_token": 8.0}, {"uid": "3e8a249523db61de", "family": "arith", "template": 0, "depth": 6, "pred": "62", "truth": "66", "correct": false, "edits_per_token": 8.0}, {"uid": "d88e4dbe5c40f125", "family": "arith", "template": 0, "depth": 6, "pred": "-153", "truth": "-183", "correct": false, "edits_per_token": 8.0}, {"uid": "6314a41e324e4ee8", "family": "arith", "template": 1, "depth": 6, "pred": "-553", "truth": "-534", "correct": false, "edits_per_token": 8.0}, {"uid": "cc03f7312928ea94", "family": "arith", "template": 1, "depth": 6, "pred": "-1000", "truth": "-1032", "correct": false, "edits_per_token": 8.0}, {"uid": "4730814296f3ea5b", "family": "arith", "template": 1, "depth": 6, "pred": "-50", "truth": "-45", "correct": false, "edits_per_token": 8.0}, {"uid": "3852c3f413554f1f", "family": "arith", "template": 2, "depth": 5, "pred": "719", "truth": "719", "correct": true, "edits_per_token": 8.0}, {"uid": "deb86cd80d3700af", "family": "arith", "template": 2, "depth": 5, "pred": "-83", "truth": "-73", "correct": false, "edits_per_token": 8.0}, {"uid": "c119fa976ea67a76", "family": "arith", "template": 2, "depth": 5, "pred": "7544", "truth": "6371", "correct": false, "edits_per_token": 8.0}, {"uid": "c6d9a48ab45e03f3", "family": "arith", "template": 2, "depth": 5, "pred": "-5", "truth": "-4", "correct": false, "edits_per_token": 8.0}, {"uid": "ad1095bee52d4847", "family": "arith", "template": 2, "depth": 5, "pred": "555", "truth": "4148", "correct": false, "edits_per_token": 8.0}, {"uid": "1d9ddc2737b667ba", "family": "arith", "template": 2, "depth": 5, "pred": "-48", "truth": "-63", "correct": false, "edits_per_token": 8.0}, {"uid": "b228f892f896a53b", "family": "arith", "template": 2, "depth": 5, "pred": "3020", "truth": "5132", "correct": false, "edits_per_token": 8.0}, {"uid": "4e943d03290831f4", "family": "arith", "template": 2, "depth": 5, "pred": "1444", "truth": "1022", "correct": false, "edits_per_token": 8.0}, {"uid": "01e14be1e788dc29", "family": "arith", "template": 2, "depth": 5, "pred": "-33", "truth": "-78", "correct": false, "edits_per_token": 8.0}, {"uid": "4c03286b1e57814d", "family": "arith", "template": 2, "depth": 5, "pred": "-1533", "truth": "-1281", "correct": false, "edits_per_token": 8.0}, {"uid": "bd1d4550d7e49f3f", "family": "arith", "template": 2, "depth": 5, "pred": "23", "truth": "25", "correct": false, "edits_per_token": 8.0}, {"uid": "9afbbb94fae24c93", "family": "arith", "template": 2, "depth": 5, "pred": "24", "truth": "25", "correct": false, "edits_per_token": 8.0}, {"uid": "fa5a4ae928911164", "family": "arith", "template": 2, "depth": 5, "pred": "-100", "truth": "-67", "correct": false, "edits_per_token": 8.0}, {"uid": "ea603b34915972f6", "family": "arith", "template": 2, "depth": 6, "pred": "1013", "truth": "10389", "correct": false, "edits_per_token": 8.0}, {"uid": "cb94dfad560085d5", "family": "arith", "template": 2, "depth": 6, "pred": "-28", "truth": "-42", "correct": false, "edits_per_token": 8.0}, {"uid": "9f9e926d4210dd46", "family": "arith", "template": 2, "depth": 6, "pred": "-116", "truth": "-120", "correct": false, "edits_per_token": 8.0}, {"uid": "4715b609ce7f76a7", "family": "arith", "template": 2, "depth": 6, "pred": "-38", "truth": "-33", "correct": false, "edits_per_token": 8.0}, {"uid": "d50bb90d051433a3", "family": "arith", "template": 2, "depth": 6, "pred": "-30", "truth": "-32", "correct": false, "edits_per_token": 8.0}, {"uid": "d1150c19a63e23cb", "family": "arith", "template": 2, "depth": 6, "pred": "-21080", "truth": "-25200", "correct": false, "edits_per_token": 8.0}, {"uid": "d8cc53270006b243", "family": "arith", "template": 2, "depth": 6, "pred": "-273", "truth": "819", "correct": false, "edits_per_token": 8.0}, {"uid": "a98fefd64fdb5cee", "family": "bind", "template": 2, "depth": 5, "pred": "-6", "truth": "-4", "correct": false, "edits_per_token": 8.0}, {"uid": "1766b8b21658487d", "family": "bind", "template": 2, "depth": 5, "pred": "53", "truth": "61", "correct": false, "edits_per_token": 8.0}, {"uid": "35dcd024227861bc", "family": "bind", "template": 2, "depth": 5, "pred": "-11", "truth": "-7", "correct": false, "edits_per_token": 8.0}, {"uid": "5a49700e594f6138", "family": "bind", "template": 2, "depth": 5, "pred": "280", "truth": "420", "correct": false, "edits_per_token": 8.0}, {"uid": "6ca4fd9394c1d2ee", "family": "bind", "template": 2, "depth": 5, "pred": "220", "truth": "402", "correct": false, "edits_per_token": 8.0}, {"uid": "6e39b1d110ecd6c0", "family": "arith", "template": 2, "depth": 6, "pred": "180", "truth": "273", "correct": false, "edits_per_token": 8.0}, {"uid": "b1193e6957a95449", "family": "arith", "template": 2, "depth": 6, "pred": "44", "truth": "54", "correct": false, "edits_per_token": 8.0}, {"uid": "a14793b622e3127b", "family": "arith", "template": 2, "depth": 6, "pred": "-17", "truth": "-23", "correct": false, "edits_per_token": 8.0}, {"uid": "5782d4f05faeb92f", "family": "bind", "template": 2, "depth": 5, "pred": "-16", "truth": "-24", "correct": false, "edits_per_token": 8.0}, {"uid": "1c6b4860d164b9f2", "family": "bind", "template": 2, "depth": 5, "pred": "10", "truth": "12", "correct": false, "edits_per_token": 8.0}, {"uid": "4af96fa0ad806077", "family": "bind", "template": 2, "depth": 5, "pred": "62", "truth": "69", "correct": false, "edits_per_token": 8.0}, {"uid": "8441e8c811ca2537", "family": "bind", "template": 2, "depth": 5, "pred": "10", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "2b9a84b2ec1c5d36", "family": "bind", "template": 2, "depth": 5, "pred": "680", "truth": "1116", "correct": false, "edits_per_token": 8.0}, {"uid": "e670f4c748204cc8", "family": "bind", "template": 2, "depth": 5, "pred": "134", "truth": "102", "correct": false, "edits_per_token": 8.0}, {"uid": "f2b0d3fe9038acd2", "family": "arith", "template": 2, "depth": 6, "pred": "45", "truth": "42", "correct": false, "edits_per_token": 8.0}, {"uid": "daacb1ff7411a2c2", "family": "arith", "template": 2, "depth": 6, "pred": "90", "truth": "101", "correct": false, "edits_per_token": 8.0}, {"uid": "f17398d768802f6b", "family": "bind", "template": 2, "depth": 5, "pred": "95", "truth": "95", "correct": true, "edits_per_token": 8.0}, {"uid": "a881a48bc9d6d2fa", "family": "arith", "template": 2, "depth": 6, "pred": "-30", "truth": "-53", "correct": false, "edits_per_token": 8.0}, {"uid": "5f00de1c8f6a727b", "family": "bind", "template": 2, "depth": 5, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 8.0}, {"uid": "56f549cdb3619b33", "family": "bind", "template": 0, "depth": 5, "pred": "21", "truth": "29", "correct": false, "edits_per_token": 8.0}, {"uid": "70e96a9cddee5517", "family": "bind", "template": 0, "depth": 5, "pred": "134", "truth": "179", "correct": false, "edits_per_token": 8.0}, {"uid": "ea1484f36c572696", "family": "bind", "template": 0, "depth": 5, "pred": "73", "truth": "80", "correct": false, "edits_per_token": 8.0}, {"uid": "54a6e84b79b6c7df", "family": "bind", "template": 0, "depth": 5, "pred": "-12", "truth": "-12", "correct": true, "edits_per_token": 8.0}, {"uid": "fec334e2c8b64794", "family": "bind", "template": 0, "depth": 5, "pred": "71", "truth": "81", "correct": false, "edits_per_token": 8.0}, {"uid": "c3f0503818c31297", "family": "bind", "template": 0, "depth": 5, "pred": "93", "truth": "89", "correct": false, "edits_per_token": 8.0}, {"uid": "a978e3f3b6818825", "family": "bind", "template": 0, "depth": 5, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 8.0}, {"uid": "33b72f3d87631e86", "family": "bind", "template": 0, "depth": 5, "pred": "427", "truth": "667", "correct": false, "edits_per_token": 8.0}, {"uid": "4410175b31fd9ff9", "family": "bind", "template": 0, "depth": 5, "pred": "240", "truth": "192", "correct": false, "edits_per_token": 8.0}, {"uid": "177037480b40585d", "family": "bind", "template": 0, "depth": 5, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "bf86bbb1610dd587", "family": "bind", "template": 1, "depth": 5, "pred": "648", "truth": "588", "correct": false, "edits_per_token": 8.0}, {"uid": "82375062bef2aad9", "family": "bind", "template": 0, "depth": 5, "pred": "75", "truth": "85", "correct": false, "edits_per_token": 8.0}, {"uid": "084f23e813b1c2f2", "family": "bind", "template": 0, "depth": 5, "pred": "33", "truth": "40", "correct": false, "edits_per_token": 8.0}, {"uid": "dca41c3ec45e2945", "family": "bind", "template": 1, "depth": 5, "pred": "300", "truth": "580", "correct": false, "edits_per_token": 8.0}, {"uid": "ee7f986fce1fe6fc", "family": "bind", "template": 0, "depth": 5, "pred": "34", "truth": "41", "correct": false, "edits_per_token": 8.0}, {"uid": "c731265cddc84c46", "family": "bind", "template": 0, "depth": 5, "pred": "90", "truth": "101", "correct": false, "edits_per_token": 8.0}, {"uid": "9a41a8cfde6b00c6", "family": "bind", "template": 1, "depth": 5, "pred": "-14", "truth": "-14", "correct": true, "edits_per_token": 8.0}, {"uid": "8d19dd38ce14dfc5", "family": "bind", "template": 1, "depth": 5, "pred": "275", "truth": "2076", "correct": false, "edits_per_token": 8.0}, {"uid": "677f597fd752b712", "family": "bind", "template": 1, "depth": 5, "pred": "-15", "truth": "-19", "correct": false, "edits_per_token": 8.0}, {"uid": "1cec2e6bbdfaf026", "family": "bind", "template": 1, "depth": 5, "pred": "48", "truth": "56", "correct": false, "edits_per_token": 8.0}, {"uid": "d7f0bbe7ddd4451f", "family": "bind", "template": 1, "depth": 5, "pred": "10", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "f6d1f2f6e260e0e3", "family": "bind", "template": 1, "depth": 5, "pred": "100", "truth": "90", "correct": false, "edits_per_token": 8.0}, {"uid": "c18e4b7f918c9f69", "family": "bind", "template": 1, "depth": 5, "pred": "20", "truth": "24", "correct": false, "edits_per_token": 8.0}, {"uid": "90c73ad098280857", "family": "bind", "template": 1, "depth": 5, "pred": "-56", "truth": "-69", "correct": false, "edits_per_token": 8.0}, {"uid": "a2750a77af5a3ed4", "family": "bind", "template": 1, "depth": 5, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "beccfdc7b9edf6a8", "family": "bind", "template": 2, "depth": 6, "pred": "145", "truth": "280", "correct": false, "edits_per_token": 8.0}, {"uid": "dfd3cbb03a444dec", "family": "bind", "template": 2, "depth": 6, "pred": "42", "truth": "60", "correct": false, "edits_per_token": 8.0}, {"uid": "5da4fa98b5246175", "family": "bind", "template": 2, "depth": 6, "pred": "34", "truth": "47", "correct": false, "edits_per_token": 8.0}, {"uid": "e8bdc504c2e559a6", "family": "bind", "template": 2, "depth": 6, "pred": "63", "truth": "116", "correct": false, "edits_per_token": 8.0}, {"uid": "576442ddb2643bf3", "family": "bind", "template": 2, "depth": 6, "pred": "70", "truth": "107", "correct": false, "edits_per_token": 8.0}, {"uid": "90935ad82127780b", "family": "fsm", "template": 2, "depth": 5, "pred": "28", "truth": "28", "correct": true, "edits_per_token": 8.0}, {"uid": "efd0c340700fd585", "family": "bind", "template": 1, "depth": 5, "pred": "107", "truth": "129", "correct": false, "edits_per_token": 8.0}, {"uid": "8a00b1ceb8cc21d0", "family": "bind", "template": 2, "depth": 6, "pred": "-11", "truth": "-8", "correct": false, "edits_per_token": 8.0}, {"uid": "1710e2024ce0188e", "family": "bind", "template": 2, "depth": 6, "pred": "283", "truth": "582", "correct": false, "edits_per_token": 8.0}, {"uid": "332068c4dc811926", "family": "bind", "template": 2, "depth": 6, "pred": "-14", "truth": "-15", "correct": false, "edits_per_token": 8.0}, {"uid": "47e5102caba8b95d", "family": "bind", "template": 2, "depth": 6, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 8.0}, {"uid": "9193a977b5e6d754", "family": "bind", "template": 2, "depth": 6, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "69b42fd072531922", "family": "bind", "template": 1, "depth": 5, "pred": "-7", "truth": "-4", "correct": false, "edits_per_token": 8.0}, {"uid": "ad4b55e83f33e0e9", "family": "bind", "template": 2, "depth": 6, "pred": "27", "truth": "38", "correct": false, "edits_per_token": 8.0}, {"uid": "6a42a33ed3b404ed", "family": "bind", "template": 2, "depth": 6, "pred": "-2", "truth": "-4", "correct": false, "edits_per_token": 8.0}, {"uid": "5885e5d090992cd2", "family": "bind", "template": 2, "depth": 6, "pred": "-12", "truth": "-14", "correct": false, "edits_per_token": 8.0}, {"uid": "db61f97b4eb08ce6", "family": "fsm", "template": 2, "depth": 5, "pred": "9", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "1a52ddcf13f295de", "family": "fsm", "template": 2, "depth": 5, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 8.0}, {"uid": "4764ec83e7c8c9b6", "family": "fsm", "template": 2, "depth": 5, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 8.0}, {"uid": "cf3e9f98a9edf153", "family": "fsm", "template": 1, "depth": 5, "pred": "27", "truth": "29", "correct": false, "edits_per_token": 8.0}, {"uid": "485f2555fb5eff21", "family": "bind", "template": 0, "depth": 6, "pred": "2", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "76f66b2bb4f4d200", "family": "bind", "template": 0, "depth": 6, "pred": "93", "truth": "108", "correct": false, "edits_per_token": 8.0}, {"uid": "8a0414e10232f5c1", "family": "bind", "template": 0, "depth": 6, "pred": "34", "truth": "52", "correct": false, "edits_per_token": 8.0}, {"uid": "6fa2bc31af37d28d", "family": "bind", "template": 0, "depth": 6, "pred": "46", "truth": "101", "correct": false, "edits_per_token": 8.0}, {"uid": "52178894c71a55fc", "family": "bind", "template": 0, "depth": 6, "pred": "120", "truth": "204", "correct": false, "edits_per_token": 8.0}, {"uid": "bf993b5b68923b56", "family": "bind", "template": 0, "depth": 6, "pred": "34", "truth": "27", "correct": false, "edits_per_token": 8.0}, {"uid": "c5ee1489b5a7552b", "family": "bind", "template": 0, "depth": 6, "pred": "28", "truth": "268", "correct": false, "edits_per_token": 8.0}, {"uid": "2dbbd53f53e30073", "family": "bind", "template": 0, "depth": 6, "pred": "55", "truth": "78", "correct": false, "edits_per_token": 8.0}, {"uid": "7437da9eb53eb3a0", "family": "bind", "template": 0, "depth": 6, "pred": "11", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "408a34c94f17f5d7", "family": "bind", "template": 0, "depth": 6, "pred": "54", "truth": "120", "correct": false, "edits_per_token": 8.0}, {"uid": "1bad57bf846df87e", "family": "bind", "template": 0, "depth": 6, "pred": "16", "truth": "-1", "correct": false, "edits_per_token": 8.0}, {"uid": "3d0794faec305528", "family": "bind", "template": 1, "depth": 6, "pred": "4", "truth": "15", "correct": false, "edits_per_token": 8.0}, {"uid": "10c6fe8e473b05f7", "family": "bind", "template": 1, "depth": 6, "pred": "-12", "truth": "-13", "correct": false, "edits_per_token": 8.0}, {"uid": "897766cab535c928", "family": "bind", "template": 1, "depth": 6, "pred": "50", "truth": "63", "correct": false, "edits_per_token": 8.0}, {"uid": "37fcdefdc142c5c8", "family": "bind", "template": 1, "depth": 6, "pred": "-100", "truth": "-520", "correct": false, "edits_per_token": 8.0}, {"uid": "0fa8fa84eab24d5b", "family": "bind", "template": 1, "depth": 6, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "e17850521f774ccb", "family": "bind", "template": 1, "depth": 6, "pred": "-36", "truth": "-57", "correct": false, "edits_per_token": 8.0}, {"uid": "87604ac5e2432117", "family": "bind", "template": 0, "depth": 6, "pred": "-2", "truth": "-2", "correct": true, "edits_per_token": 8.0}, {"uid": "1bf2c062e3d35770", "family": "bind", "template": 0, "depth": 6, "pred": "-14", "truth": "-21", "correct": false, "edits_per_token": 8.0}, {"uid": "3e71f658a8d01416", "family": "bind", "template": 1, "depth": 6, "pred": "334", "truth": "612", "correct": false, "edits_per_token": 8.0}, {"uid": "6d9120572acc5f8c", "family": "bind", "template": 1, "depth": 6, "pred": "142", "truth": "421", "correct": false, "edits_per_token": 8.0}, {"uid": "262d1886f373a193", "family": "bind", "template": 1, "depth": 6, "pred": "28", "truth": "192", "correct": false, "edits_per_token": 8.0}, {"uid": "2df700b079b1ea09", "family": "bind", "template": 1, "depth": 6, "pred": "512", "truth": "254", "correct": false, "edits_per_token": 8.0}, {"uid": "9f22ff248a27d85b", "family": "bind", "template": 1, "depth": 6, "pred": "18", "truth": "21", "correct": false, "edits_per_token": 8.0}, {"uid": "b86f3a062bdd7b14", "family": "bind", "template": 1, "depth": 6, "pred": "24", "truth": "28", "correct": false, "edits_per_token": 8.0}, {"uid": "0ec61c8d74b0b12e", "family": "bind", "template": 1, "depth": 6, "pred": "-5", "truth": "-4", "correct": false, "edits_per_token": 8.0}, {"uid": "7e69b09c478d7cf0", "family": "bind", "template": 1, "depth": 6, "pred": "-33", "truth": "-103", "correct": false, "edits_per_token": 8.0}, {"uid": "3644e4d7e72723f2", "family": "fsm", "template": 2, "depth": 5, "pred": "11", "truth": "11", "correct": true, "edits_per_token": 8.0}, {"uid": "ef5c74f822ed4d09", "family": "fsm", "template": 2, "depth": 5, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 8.0}, {"uid": "1d8a092a5663c2ca", "family": "fsm", "template": 2, "depth": 6, "pred": "16", "truth": "17", "correct": false, "edits_per_token": 8.0}, {"uid": "13b840f1309c3d23", "family": "fsm", "template": 2, "depth": 6, "pred": "14", "truth": "27", "correct": false, "edits_per_token": 8.0}, {"uid": "3bf1de404d07020e", "family": "fsm", "template": 1, "depth": 5, "pred": "30", "truth": "24", "correct": false, "edits_per_token": 8.0}, {"uid": "cca744ebe723c85b", "family": "fsm", "template": 2, "depth": 5, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "976535ab3eb31bd8", "family": "fsm", "template": 2, "depth": 5, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 8.0}, {"uid": "93ee3b76c1253b5f", "family": "fsm", "template": 0, "depth": 5, "pred": "8", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "fa364f44c6759d61", "family": "fsm", "template": 0, "depth": 5, "pred": "59", "truth": "63", "correct": false, "edits_per_token": 8.0}, {"uid": "06585bc8d35c4f52", "family": "fsm", "template": 1, "depth": 5, "pred": "14", "truth": "22", "correct": false, "edits_per_token": 8.0}, {"uid": "cb31b4e31c03606f", "family": "fsm", "template": 1, "depth": 5, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 8.0}, {"uid": "0324b1035a1b5e3b", "family": "fsm", "template": 1, "depth": 5, "pred": "24", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "70965bfd44497a12", "family": "fsm", "template": 2, "depth": 6, "pred": "8", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "9e2ced0a36e30f08", "family": "fsm", "template": 2, "depth": 6, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 8.0}, {"uid": "945b988367f1262b", "family": "fsm", "template": 2, "depth": 5, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "49d61413398cc89b", "family": "fsm", "template": 2, "depth": 6, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 8.0}, {"uid": "cac6ffd4670e5f50", "family": "fsm", "template": 2, "depth": 6, "pred": "12", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "ec003e386e949698", "family": "fsm", "template": 0, "depth": 5, "pred": "4", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "b4c4da03d549d2ad", "family": "fsm", "template": 0, "depth": 5, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 8.0}, {"uid": "d4906001b8fd5965", "family": "fsm", "template": 2, "depth": 5, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 8.0}, {"uid": "054e529ea5d13efe", "family": "fsm", "template": 2, "depth": 5, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 8.0}, {"uid": "af25cc7ee3868a05", "family": "fsm", "template": 0, "depth": 5, "pred": "11", "truth": "11", "correct": true, "edits_per_token": 8.0}, {"uid": "e6eb0871661f8cb0", "family": "fsm", "template": 2, "depth": 6, "pred": "9", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "d981e261ba455e59", "family": "fsm", "template": 1, "depth": 5, "pred": "4", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "e19b1ade5034200f", "family": "fsm", "template": 0, "depth": 5, "pred": "10", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "9084157c9bdc265e", "family": "fsm", "template": 0, "depth": 5, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 8.0}, {"uid": "966d9784f516b0b4", "family": "fsm", "template": 2, "depth": 5, "pred": "-1", "truth": "0", "correct": false, "edits_per_token": 8.0}, {"uid": "7e2c6414320592ae", "family": "fsm", "template": 0, "depth": 6, "pred": "15", "truth": "17", "correct": false, "edits_per_token": 8.0}, {"uid": "b06f1eb977fbe9e0", "family": "fsm", "template": 1, "depth": 6, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 8.0}, {"uid": "fa3b7f75f9e43b2f", "family": "fsm", "template": 1, "depth": 6, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "c7498914bccd7bf3", "family": "fsm", "template": 0, "depth": 5, "pred": "7", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "195ef7d3ccac4b8b", "family": "fsm", "template": 0, "depth": 5, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "7cb457b70a7065ab", "family": "fsm", "template": 1, "depth": 5, "pred": "8", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "4688a1575da3feae", "family": "fsm", "template": 2, "depth": 6, "pred": "-1", "truth": "-7", "correct": false, "edits_per_token": 8.0}, {"uid": "13d7a0661383bf5c", "family": "fsm", "template": 0, "depth": 6, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "8520bf5fba316dc5", "family": "fsm", "template": 0, "depth": 6, "pred": "13", "truth": "18", "correct": false, "edits_per_token": 8.0}, {"uid": "28a5212e37e34ede", "family": "fsm", "template": 2, "depth": 6, "pred": "-2", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "c1db7537b69a1588", "family": "fsm", "template": 1, "depth": 5, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 8.0}, {"uid": "475faa1c40dd1c57", "family": "fsm", "template": 2, "depth": 6, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 8.0}, {"uid": "82de054bd19fee8f", "family": "fsm", "template": 2, "depth": 6, "pred": "-20", "truth": "-18", "correct": false, "edits_per_token": 8.0}, {"uid": "1599bc918482f147", "family": "fsm", "template": 2, "depth": 6, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 8.0}, {"uid": "f13662f0f0c30826", "family": "fsm", "template": 0, "depth": 5, "pred": "6", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "b7e69cea18577e61", "family": "fsm", "template": 0, "depth": 5, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 8.0}, {"uid": "f01f401147209180", "family": "fsm", "template": 1, "depth": 6, "pred": "12", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "e45b754845391f3f", "family": "fsm", "template": 1, "depth": 5, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 8.0}, {"uid": "babac6e1d2340f83", "family": "fsm", "template": 1, "depth": 6, "pred": "0", "truth": "-2", "correct": false, "edits_per_token": 8.0}, {"uid": "f3dfeb8093398fba", "family": "fsm", "template": 2, "depth": 5, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "83156af5c9f2102f", "family": "fsm", "template": 1, "depth": 6, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 8.0}, {"uid": "436a870501934e95", "family": "fsm", "template": 2, "depth": 6, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 8.0}, {"uid": "17c22d2c6da7016d", "family": "fsm", "template": 0, "depth": 5, "pred": "1", "truth": "0", "correct": false, "edits_per_token": 8.0}, {"uid": "958a99eb728e15c5", "family": "fsm", "template": 0, "depth": 5, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 8.0}, {"uid": "e969fccd3aa5aa4d", "family": "fsm", "template": 1, "depth": 5, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 8.0}, {"uid": "9e5d597a794e81e2", "family": "fsm", "template": 0, "depth": 5, "pred": "1", "truth": "0", "correct": false, "edits_per_token": 8.0}, {"uid": "53a3aa34d09ae7f0", "family": "fsm", "template": 0, "depth": 6, "pred": "13", "truth": "15", "correct": false, "edits_per_token": 8.0}, {"uid": "7ffa08ddf97e22b8", "family": "fsm", "template": 0, "depth": 6, "pred": "6", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "df4e72237d80a81c", "family": "fsm", "template": 1, "depth": 5, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 8.0}, {"uid": "95a0ef58216182a2", "family": "fsm", "template": 1, "depth": 6, "pred": "0", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "6c413e46c2f2d918", "family": "fsm", "template": 1, "depth": 6, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 8.0}, {"uid": "38b00c5c8332094f", "family": "fsm", "template": 0, "depth": 6, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 8.0}, {"uid": "c571f1db246e3ac7", "family": "fsm", "template": 1, "depth": 5, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 8.0}, {"uid": "c8c65c1d9220669b", "family": "fsm", "template": 1, "depth": 6, "pred": "18", "truth": "28", "correct": false, "edits_per_token": 8.0}, {"uid": "2bd960c30ea3d343", "family": "fsm", "template": 0, "depth": 6, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 8.0}, {"uid": "dbd3b4ea0faa4bd5", "family": "fsm", "template": 0, "depth": 6, "pred": "6", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "ad56dff0f6c52a6e", "family": "fsm", "template": 0, "depth": 6, "pred": "-14", "truth": "-16", "correct": false, "edits_per_token": 8.0}, {"uid": "5aa303b1e8a3a50a", "family": "fsm", "template": 1, "depth": 6, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "a7aa7192117d2b91", "family": "fsm", "template": 0, "depth": 6, "pred": "11", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "3663d9bfed0657ac", "family": "fsm", "template": 1, "depth": 6, "pred": "-1", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "d442b8b4e9c4463b", "family": "fsm", "template": 1, "depth": 6, "pred": "11", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "f5d3a7b825df11db", "family": "fsm", "template": 0, "depth": 6, "pred": "7", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "5fae584fb4b0806f", "family": "fsm", "template": 1, "depth": 5, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 8.0}, {"uid": "f5d6e652e3c1c9c6", "family": "fsm", "template": 1, "depth": 6, "pred": "12", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "fae83630cc69d997", "family": "fsm", "template": 1, "depth": 6, "pred": "-5", "truth": "-7", "correct": false, "edits_per_token": 8.0}, {"uid": "ac2e2426143b2f6a", "family": "fsm", "template": 1, "depth": 6, "pred": "-7", "truth": "-8", "correct": false, "edits_per_token": 8.0}, {"uid": "3e7729193c5d67ef", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 8.0}, {"uid": "b040bca9ccdd27d8", "family": "fsm", "template": 0, "depth": 6, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "ae8d496071780205", "family": "fsm", "template": 0, "depth": 6, "pred": "6", "truth": "-4", "correct": false, "edits_per_token": 8.0}, {"uid": "48b2c33f5bfc7b39", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "ffb3a068ddee91d8", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "0b5d578a6414c59b", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "plum fig", "correct": false, "edits_per_token": 8.0}, {"uid": "5457f87f4a4d9928", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "29cc176fd2bd357a", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "9a37aa52f6f4f0ef", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "yam", "correct": false, "edits_per_token": 8.0}, {"uid": "f6e2b9995ac5a2af", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "04e6cd46e148e861", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "c53436531a9e4322", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "kiwi lime", "correct": false, "edits_per_token": 8.0}, {"uid": "8c39602b8320d36d", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "451ab7989b156691", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "plum", "correct": false, "edits_per_token": 8.0}, {"uid": "64e6b2f20007b04f", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "06695366b2ca7f94", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "fe21b5c807616138", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "b3aeac08c3bf5403", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "7b4a99cf741c62f4", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 8.0}, {"uid": "544e5524578ed1a2", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 8.0}, {"uid": "8813abcc63108e4e", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "6ef02a442804899d", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "9542b67b66fba534", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "bea23de3fd39e931", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "f862e4eb48abf854", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "B", "correct": false, "edits_per_token": 8.0}, {"uid": "1e956d986d80a2e8", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "B", "correct": false, "edits_per_token": 8.0}, {"uid": "8c26836a775dfeef", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 8.0}, {"uid": "dbc95bbfbdbb8c83", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "589bdb5b41da7c65", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "e61d3770c79cde47", "family": "sched", "template": 0, "depth": 5, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 8.0}, {"uid": "2fbd02a75eab5edf", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "bc00b85f3a4929cb", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "83d9e53bcbe7935d", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "19147a5ccac0f776", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "b6995c56551edd0f", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "597331784c8adb06", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 8.0}, {"uid": "7a4868e5e81669d0", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "6ab7960babdc2e4d", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 8.0}, {"uid": "8c9f86f29e505eb5", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "7d3f18625f00d8ee", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "1e7febaac638433d", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "192270ee3b6c26d9", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "2b26c9c6928a8458", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "65550dd4f783d103", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "637f9d17aabeb31c", "family": "sched", "template": 1, "depth": 5, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "ed8754896958fc23", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "date", "correct": false, "edits_per_token": 8.0}, {"uid": "daf704f320ba7df5", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 8.0}, {"uid": "72ac559ac49d72ce", "family": "sched", "template": 1, "depth": 5, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "b7380cfd5017d3d5", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "7bfb28727e572f4f", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 8.0}, {"uid": "5a7f7b85164f7d43", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "bean", "correct": false, "edits_per_token": 8.0}, {"uid": "5a9f6a2d1fca7c30", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "5dfac4f4fcff919d", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "c877e5915a1d82dd", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "91d7e38041c84a54", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "d88ead16fc9cb4dd", "family": "sched", "template": 2, "depth": 5, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "0470c678ba5e33bd", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 8.0}, {"uid": "2004484848062f39", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "4aec4964885ab8cd", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "f62aa798063a3946", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "5821393361b27572", "family": "sched", "template": 2, "depth": 5, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "9835b0f9d397f634", "family": "sched", "template": 2, "depth": 5, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "efd0e2bbf109da63", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "66c464d4968a2fe2", "family": "sched", "template": 0, "depth": 5, "pred": "none", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "ae5e2670afd5d884", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "738b6d8b10bdecc1", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "31e0d3601d129675", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "8c334375cbdeecbb", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "3146c4b45bd4ba36", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "16bc96ba9e32042e", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "7dc88230188b5ae0", "family": "sched", "template": 0, "depth": 5, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "ffdf323f5b0e8398", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 8.0}, {"uid": "031f5a4ec368d5ed", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "249d68235fab3319", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "ac5f146d3cf4c744", "family": "sched", "template": 0, "depth": 5, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 8.0}, {"uid": "85e6171c2bff77ec", "family": "sched", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "a5eb1ad33216a365", "family": "sched", "template": 0, "depth": 5, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "fbfb9d7ebbd9ab42", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 8.0}, {"uid": "93e46dc65f4a0d60", "family": "sched", "template": 2, "depth": 6, "pred": "Dxclude tas", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "7d44befcfd949a85", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "dfe372c23262434c", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "c68ec9cd887646d5", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "a01c26ea9b3e73d4", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 8.0}, {"uid": "6c51902842f6a23b", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 8.0}, {"uid": "1d040a988ed2fa83", "family": "sched", "template": 2, "depth": 5, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "165a925a98ad694c", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "e97fa901d509ffd6", "family": "sched", "template": 0, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 8.0}, {"uid": "3676d84af524d9ae", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 8.0}, {"uid": "cd0b4315898f7965", "family": "sched", "template": 2, "depth": 5, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "7429be63b9fe2121", "family": "sched", "template": 0, "depth": 6, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "c696f5f9f596f2f7", "family": "sched", "template": 2, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 8.0}, {"uid": "59f92777461a61ec", "family": "sched", "template": 0, "depth": 6, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "fb0fcdfe7002426e", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "871d1deab44403c5", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "plum", "correct": false, "edits_per_token": 8.0}, {"uid": "e09fe1db1040c253", "family": "sched", "template": 0, "depth": 5, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "70d22e9191a63ed3", "family": "sched", "template": 0, "depth": 5, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "b5b6b8f1e1b8e526", "family": "sched", "template": 2, "depth": 5, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "eb52c5c1389933b9", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "ae5ef219347ca8d1", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "bc760caf57d4fbeb", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "342a44e350c2c803", "family": "excl", "template": 1, "depth": 5, "pred": "none", "truth": "plum date", "correct": false, "edits_per_token": 8.0}, {"uid": "c1f8bff397b18871", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "fig", "correct": false, "edits_per_token": 8.0}, {"uid": "2315692be3c9225f", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "yam", "correct": false, "edits_per_token": 8.0}, {"uid": "8eb603b3bcb2b694", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "588fd0833653c113", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "1e067fba49f1bde2", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "7216a64146ae6acd", "family": "sched", "template": 1, "depth": 5, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "7845079b2715c407", "family": "sched", "template": 1, "depth": 5, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "5733b7ed7ff6f08e", "family": "sched", "template": 1, "depth": 5, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 8.0}, {"uid": "9565161477fd02c2", "family": "excl", "template": 0, "depth": 5, "pred": "nut bean", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "f3390e99576c3f52", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "685bcd1984a9ef37", "family": "sched", "template": 1, "depth": 6, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 8.0}, {"uid": "127824e03ba4e18f", "family": "sched", "template": 1, "depth": 6, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "7af3f6bc2435a9c3", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "af0de551a61187f6", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "bdaff9107551bcad", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "d5369281fc9cfcd4", "family": "sched", "template": 1, "depth": 5, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "994412f1bf4510f1", "family": "sched", "template": 1, "depth": 6, "pred": "B", "truth": "F", "correct": false, "edits_per_token": 8.0}, {"uid": "b1337bc8478cf188", "family": "sched", "template": 1, "depth": 6, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 8.0}, {"uid": "1591f1da64128221", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "bean", "correct": false, "edits_per_token": 8.0}, {"uid": "4ace02760358e0aa", "family": "sched", "template": 1, "depth": 5, "pred": "C", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "c38781630980610c", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "5cdeab7b7d3d00a8", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "c0b33bc616c270db", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "ae3d111057f9a376", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "0dbf96f152ee57cd", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "E", "correct": false, "edits_per_token": 8.0}, {"uid": "2ec492b6141d5723", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "a8895e2feee026d1", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "63b451d568c16ffa", "family": "excl", "template": 2, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "7a75d953e8308610", "family": "sched", "template": 0, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 8.0}, {"uid": "551b7796162a1a33", "family": "sched", "template": 1, "depth": 5, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "0409fe31b46650ef", "family": "sched", "template": 0, "depth": 6, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 8.0}, {"uid": "6fcf7fc1691a79a5", "family": "sched", "template": 2, "depth": 6, "pred": "C", "truth": "F", "correct": false, "edits_per_token": 8.0}, {"uid": "52e4c2db61cbda34", "family": "sched", "template": 0, "depth": 6, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "6b2e6485c06a7570", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "e7b6cecd9fd15087", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "f1ca4b2353614cc4", "family": "sched", "template": 0, "depth": 6, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "f71e8f3bf0d91481", "family": "excl", "template": 0, "depth": 5, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "227f0ada257e9567", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "0847283e1b47f87f", "family": "sched", "template": 2, "depth": 6, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "5c926222dc33129d", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 8.0}, {"uid": "6a5adbd9ac262749", "family": "sched", "template": 2, "depth": 6, "pred": "D", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "dbccdf3934cf6baf", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "d3979db4d8709e58", "family": "sched", "template": 0, "depth": 6, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 8.0}, {"uid": "da9eddd4e681af1a", "family": "sched", "template": 0, "depth": 6, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "bb2c190eae3c2125", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "7257616f0bb8b273", "family": "excl", "template": 1, "depth": 5, "pred": "bean", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "b442b080ca1d58bf", "family": "excl", "template": 2, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "1762aafe566bb2f0", "family": "sched", "template": 1, "depth": 6, "pred": "B", "truth": "E", "correct": false, "edits_per_token": 8.0}, {"uid": "3478623940a0babb", "family": "sched", "template": 0, "depth": 6, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 8.0}, {"uid": "d49274b1b41b7b91", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "54953d3704253d0a", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "4b3ba50fe8aed021", "family": "sched", "template": 1, "depth": 6, "pred": "D", "truth": "F", "correct": false, "edits_per_token": 8.0}, {"uid": "1564dfd3f2b289c5", "family": "excl", "template": 1, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "420f41111b0b1478", "family": "sched", "template": 1, "depth": 6, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "fb5eb61f7d51b656", "family": "sched", "template": 1, "depth": 6, "pred": "C", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "b3cf93f47535eec2", "family": "sched", "template": 1, "depth": 6, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "a98861d878768980", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "8c0cce3a8cc7a8a8", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 8.0}, {"uid": "139fdebe4be35cb8", "family": "sched", "template": 1, "depth": 6, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 8.0}, {"uid": "bd0288f2ce913d48", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "39f4a4e458ce30d0", "family": "excl", "template": 0, "depth": 6, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}] | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_iid_b0.json
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +[{"uid": "d57de33b3b3613ce", "family": "fsm", "template": 2, "depth": 1, "pred": "8", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "2fcbef2f6deb569d", "family": "bind", "template": 2, "depth": 1, "pred": "18", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "facfe4b45751726b", "family": "bind", "template": 2, "depth": 1, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 0.0}, {"uid": "17c2f737ab6e9a12", "family": "bind", "template": 2, "depth": 1, "pred": "21", "truth": "21", "correct": true, "edits_per_token": 0.0}, {"uid": "330a5096d00ea26d", "family": "bind", "template": 2, "depth": 1, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "67c60958afd432d0", "family": "bind", "template": 2, "depth": 1, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "0df874a819030335", "family": "bind", "template": 2, "depth": 1, "pred": "60", "truth": "60", "correct": true, "edits_per_token": 0.0}, {"uid": "f8b6c686efa42185", "family": "bind", "template": 2, "depth": 1, "pred": "27", "truth": "26", "correct": false, "edits_per_token": 0.0}, {"uid": "61ab4c048732d1a3", "family": "fsm", "template": 2, "depth": 1, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "a0a9effb7a5a1771", "family": "fsm", "template": 2, "depth": 1, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "e539dc5c812c9f8f", "family": "arith", "template": 0, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "b9080d7276f3e48d", "family": "arith", "template": 1, "depth": 1, "pred": "14", "truth": "14", "correct": true, "edits_per_token": 0.0}, {"uid": "f19b2d158a1b2321", "family": "arith", "template": 0, "depth": 1, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "b50334287c335a68", "family": "arith", "template": 0, "depth": 1, "pred": "13", "truth": "13", "correct": true, "edits_per_token": 0.0}, {"uid": "5203cff28a90efc8", "family": "arith", "template": 0, "depth": 1, "pred": "14", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "b0fc8aed8de80f82", "family": "arith", "template": 1, "depth": 1, "pred": "85", "truth": "85", "correct": true, "edits_per_token": 0.0}, {"uid": "d3f8213a4e9e4170", "family": "arith", "template": 1, "depth": 1, "pred": "64", "truth": "66", "correct": false, "edits_per_token": 0.0}, {"uid": "6dbb743f0c127dc2", "family": "arith", "template": 1, "depth": 1, "pred": "100", "truth": "120", "correct": false, "edits_per_token": 0.0}, {"uid": "94bfbb133f315b17", "family": "arith", "template": 0, "depth": 1, "pred": "37", "truth": "37", "correct": true, "edits_per_token": 0.0}, {"uid": "c5dd86b6955d4021", "family": "arith", "template": 1, "depth": 1, "pred": "-7", "truth": "-7", "correct": true, "edits_per_token": 0.0}, {"uid": "75eceacb14c3d0e6", "family": "bind", "template": 0, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "f3b82055a8272259", "family": "bind", "template": 0, "depth": 1, "pred": "18", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "6fa8f401c1e6cf13", "family": "bind", "template": 0, "depth": 1, "pred": "22", "truth": "22", "correct": true, "edits_per_token": 0.0}, {"uid": "5252b66c833e4804", "family": "bind", "template": 0, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "0dd2fab9c5861d6d", "family": "bind", "template": 0, "depth": 1, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 0.0}, {"uid": "d8aa01c2e9dbb03f", "family": "bind", "template": 1, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "1d20f4c03960f839", "family": "arith", "template": 0, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "a0d75f1874311fde", "family": "bind", "template": 0, "depth": 1, "pred": "26", "truth": "27", "correct": false, "edits_per_token": 0.0}, {"uid": "2f252bbc438ce228", "family": "bind", "template": 0, "depth": 1, "pred": "23", "truth": "24", "correct": false, "edits_per_token": 0.0}, {"uid": "910350946f41fbf1", "family": "bind", "template": 1, "depth": 1, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "322ec1e8c6acc1ff", "family": "bind", "template": 1, "depth": 1, "pred": "24", "truth": "24", "correct": true, "edits_per_token": 0.0}, {"uid": "8f08654666473a62", "family": "bind", "template": 1, "depth": 1, "pred": "8", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "e74c59b882dcf2b2", "family": "bind", "template": 1, "depth": 1, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 0.0}, {"uid": "f4ea26539e1674ec", "family": "arith", "template": 0, "depth": 2, "pred": "28", "truth": "28", "correct": true, "edits_per_token": 0.0}, {"uid": "8461e207ba7b8587", "family": "arith", "template": 1, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "d43d09af762bceee", "family": "arith", "template": 1, "depth": 2, "pred": "41", "truth": "44", "correct": false, "edits_per_token": 0.0}, {"uid": "7ed459fad79fdf0b", "family": "bind", "template": 1, "depth": 1, "pred": "13", "truth": "12", "correct": false, "edits_per_token": 0.0}, {"uid": "ce3cbdbb1894d8f4", "family": "bind", "template": 1, "depth": 1, "pred": "14", "truth": "15", "correct": false, "edits_per_token": 0.0}, {"uid": "14947545562906ad", "family": "bind", "template": 1, "depth": 1, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "697d570c40384635", "family": "bind", "template": 2, "depth": 2, "pred": "11", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "b78e61c19225f49c", "family": "fsm", "template": 2, "depth": 2, "pred": "19", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "69953fd72fce090d", "family": "arith", "template": 0, "depth": 2, "pred": "36", "truth": "38", "correct": false, "edits_per_token": 0.0}, {"uid": "66cdadf971fee2ab", "family": "arith", "template": 0, "depth": 2, "pred": "69", "truth": "71", "correct": false, "edits_per_token": 0.0}, {"uid": "99c7a494c455e41d", "family": "arith", "template": 0, "depth": 2, "pred": "-64", "truth": "-60", "correct": false, "edits_per_token": 0.0}, {"uid": "8d21bd8bbb5eecad", "family": "arith", "template": 0, "depth": 2, "pred": "42", "truth": "41", "correct": false, "edits_per_token": 0.0}, {"uid": "5f3cf3ff50e3e9c1", "family": "arith", "template": 1, "depth": 2, "pred": "40", "truth": "41", "correct": false, "edits_per_token": 0.0}, {"uid": "ce6f8c794a4d97fb", "family": "arith", "template": 1, "depth": 2, "pred": "51", "truth": "55", "correct": false, "edits_per_token": 0.0}, {"uid": "56607dee807b8fd6", "family": "bind", "template": 2, "depth": 2, "pred": "17", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "206a446298d63b02", "family": "bind", "template": 2, "depth": 2, "pred": "40", "truth": "40", "correct": true, "edits_per_token": 0.0}, {"uid": "4dd1c85d972bb798", "family": "bind", "template": 2, "depth": 2, "pred": "-5", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "60d954acf09287f1", "family": "arith", "template": 0, "depth": 2, "pred": "48", "truth": "46", "correct": false, "edits_per_token": 0.0}, {"uid": "5521b704b5ac25a7", "family": "arith", "template": 1, "depth": 2, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 0.0}, {"uid": "b66b330f45989ec2", "family": "arith", "template": 1, "depth": 2, "pred": "12", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "f06a723961f75f4d", "family": "arith", "template": 1, "depth": 2, "pred": "-34", "truth": "-36", "correct": false, "edits_per_token": 0.0}, {"uid": "1ffed152f8b72d39", "family": "bind", "template": 2, "depth": 2, "pred": "4", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "3b4d5aeb1e813243", "family": "arith", "template": 0, "depth": 3, "pred": "108", "truth": "140", "correct": false, "edits_per_token": 0.0}, {"uid": "e56fea6bf4230916", "family": "bind", "template": 2, "depth": 2, "pred": "0", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "9f5c3b31e7d60900", "family": "bind", "template": 2, "depth": 2, "pred": "28", "truth": "31", "correct": false, "edits_per_token": 0.0}, {"uid": "ba60a76e7ce73c9a", "family": "arith", "template": 0, "depth": 3, "pred": "165", "truth": "165", "correct": true, "edits_per_token": 0.0}, {"uid": "caa2ffb073e3b554", "family": "arith", "template": 0, "depth": 3, "pred": "-104", "truth": "-80", "correct": false, "edits_per_token": 0.0}, {"uid": "f3df8ca3c7f3fe73", "family": "arith", "template": 0, "depth": 3, "pred": "-350", "truth": "-300", "correct": false, "edits_per_token": 0.0}, {"uid": "6a521ae83b9560cb", "family": "arith", "template": 0, "depth": 3, "pred": "308", "truth": "395", "correct": false, "edits_per_token": 0.0}, {"uid": "b7e7635fb68761fa", "family": "arith", "template": 1, "depth": 3, "pred": "80", "truth": "78", "correct": false, "edits_per_token": 0.0}, {"uid": "69168d7303f42428", "family": "arith", "template": 1, "depth": 3, "pred": "368", "truth": "352", "correct": false, "edits_per_token": 0.0}, {"uid": "3876cda47832bc78", "family": "arith", "template": 1, "depth": 3, "pred": "60", "truth": "66", "correct": false, "edits_per_token": 0.0}, {"uid": "101a933392cdc486", "family": "arith", "template": 0, "depth": 3, "pred": "-11", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "5d6734a6c29bf98b", "family": "arith", "template": 1, "depth": 3, "pred": "-36", "truth": "-38", "correct": false, "edits_per_token": 0.0}, {"uid": "e6b4bb7a789c1bbf", "family": "arith", "template": 1, "depth": 3, "pred": "-36", "truth": "-36", "correct": true, "edits_per_token": 0.0}, {"uid": "529ef2fbb6693bf4", "family": "arith", "template": 0, "depth": 3, "pred": "23", "truth": "27", "correct": false, "edits_per_token": 0.0}, {"uid": "4264ff894995b84a", "family": "arith", "template": 0, "depth": 4, "pred": "111", "truth": "69", "correct": false, "edits_per_token": 0.0}, {"uid": "4fd0a06acba9c2bd", "family": "arith", "template": 1, "depth": 3, "pred": "77", "truth": "75", "correct": false, "edits_per_token": 0.0}, {"uid": "11ddfaa4634944bc", "family": "arith", "template": 1, "depth": 3, "pred": "-12", "truth": "-11", "correct": false, "edits_per_token": 0.0}, {"uid": "81903019667d210a", "family": "arith", "template": 0, "depth": 4, "pred": "-21", "truth": "-24", "correct": false, "edits_per_token": 0.0}, {"uid": "f025613adc1381e6", "family": "arith", "template": 0, "depth": 4, "pred": "67", "truth": "67", "correct": true, "edits_per_token": 0.0}, {"uid": "6917d247eea87098", "family": "arith", "template": 1, "depth": 4, "pred": "1720", "truth": "1400", "correct": false, "edits_per_token": 0.0}, {"uid": "5ac4cc428fcee7ff", "family": "bind", "template": 0, "depth": 2, "pred": "112", "truth": "180", "correct": false, "edits_per_token": 0.0}, {"uid": "a7fa5ddf82e3acb6", "family": "bind", "template": 0, "depth": 2, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 0.0}, {"uid": "85a6f3db0aee68b6", "family": "fsm", "template": 0, "depth": 1, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "306790bc18e1ff0f", "family": "fsm", "template": 1, "depth": 1, "pred": "-5", "truth": "-5", "correct": true, "edits_per_token": 0.0}, {"uid": "7b35ad2e31d3fe01", "family": "fsm", "template": 1, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "f692ca3a30b9615b", "family": "arith", "template": 0, "depth": 4, "pred": "27", "truth": "27", "correct": true, "edits_per_token": 0.0}, {"uid": "00456af1ebb8a6f6", "family": "arith", "template": 1, "depth": 4, "pred": "-16", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "8636f76e7a05df5c", "family": "bind", "template": 0, "depth": 2, "pred": "-15", "truth": "-10", "correct": false, "edits_per_token": 0.0}, {"uid": "6894af54caeca8f0", "family": "bind", "template": 0, "depth": 2, "pred": "56", "truth": "56", "correct": true, "edits_per_token": 0.0}, {"uid": "15a26f31dc193ec8", "family": "bind", "template": 0, "depth": 2, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 0.0}, {"uid": "fbf1f2cb25ea61f5", "family": "bind", "template": 0, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "e1c776eb2e759087", "family": "arith", "template": 0, "depth": 4, "pred": "77", "truth": "66", "correct": false, "edits_per_token": 0.0}, {"uid": "b57b7bb076a588da", "family": "arith", "template": 1, "depth": 4, "pred": "368", "truth": "480", "correct": false, "edits_per_token": 0.0}, {"uid": "c96adbcbd0c51bd0", "family": "arith", "template": 1, "depth": 4, "pred": "50", "truth": "52", "correct": false, "edits_per_token": 0.0}, {"uid": "058b50085974c009", "family": "arith", "template": 1, "depth": 4, "pred": "19", "truth": "19", "correct": true, "edits_per_token": 0.0}, {"uid": "cbc5614b2da64a6a", "family": "arith", "template": 1, "depth": 4, "pred": "-48", "truth": "-50", "correct": false, "edits_per_token": 0.0}, {"uid": "ffd3f5ed636cb166", "family": "bind", "template": 0, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "e48b5252059e24a7", "family": "bind", "template": 1, "depth": 2, "pred": "-8", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "58cc6ff1fb3e1166", "family": "bind", "template": 1, "depth": 2, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 0.0}, {"uid": "9dbdbf7552d0f726", "family": "bind", "template": 1, "depth": 2, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 0.0}, {"uid": "02f04b4ab6addf22", "family": "fsm", "template": 2, "depth": 2, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 0.0}, {"uid": "a7df7a7b48f0e07c", "family": "fsm", "template": 2, "depth": 2, "pred": "-3", "truth": "-3", "correct": true, "edits_per_token": 0.0}, {"uid": "4d05c02585d8db9c", "family": "fsm", "template": 2, "depth": 2, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "54b414556cb4b641", "family": "fsm", "template": 2, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "afe7716d466e8527", "family": "arith", "template": 0, "depth": 4, "pred": "-12", "truth": "-10", "correct": false, "edits_per_token": 0.0}, {"uid": "a3a41fe21c3b68c3", "family": "arith", "template": 0, "depth": 4, "pred": "-93", "truth": "-106", "correct": false, "edits_per_token": 0.0}, {"uid": "1e60abcad3f7cd49", "family": "arith", "template": 1, "depth": 4, "pred": "26", "truth": "29", "correct": false, "edits_per_token": 0.0}, {"uid": "5b477191fed1d1ab", "family": "bind", "template": 1, "depth": 2, "pred": "-8", "truth": "-8", "correct": true, "edits_per_token": 0.0}, {"uid": "b618508093791a17", "family": "bind", "template": 1, "depth": 2, "pred": "210", "truth": "240", "correct": false, "edits_per_token": 0.0}, {"uid": "758626244ff1e265", "family": "bind", "template": 1, "depth": 2, "pred": "45", "truth": "45", "correct": true, "edits_per_token": 0.0}, {"uid": "53f6d7daa1e917ac", "family": "bind", "template": 2, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "7a489d579355664c", "family": "arith", "template": 2, "depth": 1, "pred": "48", "truth": "48", "correct": true, "edits_per_token": 0.0}, {"uid": "866964b48f15dc73", "family": "arith", "template": 2, "depth": 1, "pred": "22", "truth": "21", "correct": false, "edits_per_token": 0.0}, {"uid": "326ed5326c7a15fa", "family": "arith", "template": 2, "depth": 1, "pred": "64", "truth": "60", "correct": false, "edits_per_token": 0.0}, {"uid": "78b68accb86904aa", "family": "bind", "template": 1, "depth": 2, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 0.0}, {"uid": "12ab692979396f5c", "family": "bind", "template": 2, "depth": 3, "pred": "45", "truth": "36", "correct": false, "edits_per_token": 0.0}, {"uid": "4b618bfde19bcd4c", "family": "bind", "template": 2, "depth": 3, "pred": "-8", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "99b184a684af7db3", "family": "bind", "template": 2, "depth": 3, "pred": "-12", "truth": "-15", "correct": false, "edits_per_token": 0.0}, {"uid": "09b2aab4939d318c", "family": "fsm", "template": 2, "depth": 3, "pred": "19", "truth": "19", "correct": true, "edits_per_token": 0.0}, {"uid": "b93e590222b68eba", "family": "arith", "template": 2, "depth": 1, "pred": "-18", "truth": "-18", "correct": true, "edits_per_token": 0.0}, {"uid": "e12b670d52d4bcac", "family": "arith", "template": 2, "depth": 1, "pred": "31", "truth": "34", "correct": false, "edits_per_token": 0.0}, {"uid": "4247b2f7e094279e", "family": "bind", "template": 2, "depth": 3, "pred": "19", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "67dfb1d3b0773f0e", "family": "bind", "template": 2, "depth": 3, "pred": "48", "truth": "43", "correct": false, "edits_per_token": 0.0}, {"uid": "8254a0f979194a20", "family": "bind", "template": 2, "depth": 3, "pred": "10", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "7e2f3ac39edf259e", "family": "bind", "template": 2, "depth": 3, "pred": "29", "truth": "29", "correct": true, "edits_per_token": 0.0}, {"uid": "f78cbce45d518768", "family": "fsm", "template": 0, "depth": 1, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 0.0}, {"uid": "67327201e0703535", "family": "fsm", "template": 1, "depth": 2, "pred": "16", "truth": "15", "correct": false, "edits_per_token": 0.0}, {"uid": "ad02a284f829c979", "family": "fsm", "template": 1, "depth": 2, "pred": "16", "truth": "15", "correct": false, "edits_per_token": 0.0}, {"uid": "d66d39facf8c09dc", "family": "arith", "template": 2, "depth": 2, "pred": "22", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "961f7a505cc9733d", "family": "arith", "template": 2, "depth": 2, "pred": "-108", "truth": "-120", "correct": false, "edits_per_token": 0.0}, {"uid": "2a66eac56efc842a", "family": "arith", "template": 2, "depth": 2, "pred": "180", "truth": "195", "correct": false, "edits_per_token": 0.0}, {"uid": "0b6aca1ee4bcc5e4", "family": "arith", "template": 2, "depth": 2, "pred": "-48", "truth": "-54", "correct": false, "edits_per_token": 0.0}, {"uid": "a7520c88e128325d", "family": "arith", "template": 2, "depth": 2, "pred": "7", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "a4a050b886f6d6f1", "family": "arith", "template": 2, "depth": 2, "pred": "64", "truth": "68", "correct": false, "edits_per_token": 0.0}, {"uid": "822a99424bbcf8ea", "family": "arith", "template": 2, "depth": 2, "pred": "40", "truth": "42", "correct": false, "edits_per_token": 0.0}, {"uid": "8fbd3d2ce9fd04d2", "family": "bind", "template": 0, "depth": 3, "pred": "57", "truth": "60", "correct": false, "edits_per_token": 0.0}, {"uid": "2ef3a2f2a0328157", "family": "bind", "template": 0, "depth": 3, "pred": "36", "truth": "32", "correct": false, "edits_per_token": 0.0}, {"uid": "588fe9ad929612f6", "family": "bind", "template": 0, "depth": 3, "pred": "10", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "3db5b222471f96fa", "family": "bind", "template": 0, "depth": 3, "pred": "20", "truth": "20", "correct": true, "edits_per_token": 0.0}, {"uid": "b96935b62f495f58", "family": "arith", "template": 2, "depth": 3, "pred": "119", "truth": "149", "correct": false, "edits_per_token": 0.0}, {"uid": "4ce4e40dc4a8caf4", "family": "bind", "template": 0, "depth": 3, "pred": "80", "truth": "64", "correct": false, "edits_per_token": 0.0}, {"uid": "0acc51832dea68c1", "family": "bind", "template": 0, "depth": 3, "pred": "21", "truth": "23", "correct": false, "edits_per_token": 0.0}, {"uid": "6f0f31272e9dccb8", "family": "fsm", "template": 1, "depth": 2, "pred": "20", "truth": "22", "correct": false, "edits_per_token": 0.0}, {"uid": "43de2e7618ca4cb3", "family": "fsm", "template": 2, "depth": 3, "pred": "20", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "ffb7dab9fc31fbd5", "family": "arith", "template": 2, "depth": 3, "pred": "44", "truth": "42", "correct": false, "edits_per_token": 0.0}, {"uid": "25ab3f6cf0620687", "family": "arith", "template": 2, "depth": 3, "pred": "40", "truth": "38", "correct": false, "edits_per_token": 0.0}, {"uid": "6e8c3b9ad7c915f8", "family": "arith", "template": 2, "depth": 3, "pred": "-14", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "595ac6801eedc4f3", "family": "bind", "template": 0, "depth": 3, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 0.0}, {"uid": "76fdb2e6417c6f93", "family": "bind", "template": 1, "depth": 3, "pred": "12", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "d8e6469a252c53ab", "family": "fsm", "template": 2, "depth": 3, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "aa252e805cf95b6a", "family": "arith", "template": 2, "depth": 3, "pred": "17", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "e2bce580fe2875d8", "family": "arith", "template": 2, "depth": 3, "pred": "62", "truth": "60", "correct": false, "edits_per_token": 0.0}, {"uid": "a3811535637f42cf", "family": "arith", "template": 2, "depth": 3, "pred": "-14", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "920633664a3100c7", "family": "arith", "template": 2, "depth": 3, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "6d1db4bf8883e127", "family": "bind", "template": 1, "depth": 3, "pred": "22", "truth": "22", "correct": true, "edits_per_token": 0.0}, {"uid": "42b7111968d0fd6e", "family": "bind", "template": 1, "depth": 3, "pred": "77", "truth": "74", "correct": false, "edits_per_token": 0.0}, {"uid": "0d7d21b45a123ecc", "family": "bind", "template": 1, "depth": 3, "pred": "27", "truth": "28", "correct": false, "edits_per_token": 0.0}, {"uid": "0070225255190df8", "family": "bind", "template": 1, "depth": 3, "pred": "-5", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "9efc725b639fa1e2", "family": "bind", "template": 1, "depth": 3, "pred": "41", "truth": "42", "correct": false, "edits_per_token": 0.0}, {"uid": "e6d2938cf240e7d7", "family": "bind", "template": 1, "depth": 3, "pred": "-1", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "6333807fc55e47c2", "family": "bind", "template": 2, "depth": 4, "pred": "27", "truth": "27", "correct": true, "edits_per_token": 0.0}, {"uid": "73bab13d0adb93c9", "family": "bind", "template": 2, "depth": 4, "pred": "33", "truth": "31", "correct": false, "edits_per_token": 0.0}, {"uid": "7e4a5eedc90f317e", "family": "arith", "template": 2, "depth": 4, "pred": "577", "truth": "707", "correct": false, "edits_per_token": 0.0}, {"uid": "72ab6c11b6b99a0c", "family": "bind", "template": 2, "depth": 4, "pred": "8", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "82d8e4541e035995", "family": "bind", "template": 2, "depth": 4, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 0.0}, {"uid": "11248162bd234998", "family": "bind", "template": 2, "depth": 4, "pred": "61", "truth": "61", "correct": true, "edits_per_token": 0.0}, {"uid": "30512b180423dd81", "family": "fsm", "template": 0, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "5fff7cb1816a8b83", "family": "arith", "template": 2, "depth": 4, "pred": "-160", "truth": "-90", "correct": false, "edits_per_token": 0.0}, {"uid": "916e8e5dc57e71b5", "family": "arith", "template": 2, "depth": 4, "pred": "37", "truth": "33", "correct": false, "edits_per_token": 0.0}, {"uid": "86a2ceda3911a38b", "family": "arith", "template": 2, "depth": 4, "pred": "141", "truth": "165", "correct": false, "edits_per_token": 0.0}, {"uid": "407e416b5a6578ae", "family": "arith", "template": 2, "depth": 4, "pred": "216", "truth": "222", "correct": false, "edits_per_token": 0.0}, {"uid": "f3df5ab6d440dc66", "family": "bind", "template": 2, "depth": 4, "pred": "38", "truth": "40", "correct": false, "edits_per_token": 0.0}, {"uid": "d8aa4893aec3878d", "family": "bind", "template": 2, "depth": 4, "pred": "288", "truth": "316", "correct": false, "edits_per_token": 0.0}, {"uid": "effc949016a8ea12", "family": "fsm", "template": 0, "depth": 2, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 0.0}, {"uid": "efbc892e6b4a8d10", "family": "fsm", "template": 1, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "9ed8a702224d9157", "family": "fsm", "template": 1, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "447dc7313518c9cc", "family": "arith", "template": 2, "depth": 4, "pred": "-40", "truth": "-41", "correct": false, "edits_per_token": 0.0}, {"uid": "c9f30e7c90083e3b", "family": "arith", "template": 2, "depth": 4, "pred": "58", "truth": "70", "correct": false, "edits_per_token": 0.0}, {"uid": "414ba9ab6e9d47f0", "family": "fsm", "template": 2, "depth": 3, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 0.0}, {"uid": "d6bbe76ab7ae5dea", "family": "bind", "template": 0, "depth": 4, "pred": "5", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "ad80d63be705294d", "family": "fsm", "template": 0, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "69d134b029d15133", "family": "bind", "template": 0, "depth": 4, "pred": "-10", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "1ed0d6ff18c8180d", "family": "bind", "template": 0, "depth": 4, "pred": "11", "truth": "11", "correct": true, "edits_per_token": 0.0}, {"uid": "9679ca491537d4ae", "family": "fsm", "template": 0, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "06e9560336c69fc0", "family": "bind", "template": 0, "depth": 4, "pred": "100", "truth": "99", "correct": false, "edits_per_token": 0.0}, {"uid": "83785d82a70218c2", "family": "bind", "template": 0, "depth": 4, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "365ba7f8ce47ab26", "family": "bind", "template": 0, "depth": 4, "pred": "54", "truth": "51", "correct": false, "edits_per_token": 0.0}, {"uid": "f0b461762a54a509", "family": "bind", "template": 1, "depth": 4, "pred": "680", "truth": "368", "correct": false, "edits_per_token": 0.0}, {"uid": "ef8628813a4db970", "family": "bind", "template": 0, "depth": 4, "pred": "60", "truth": "64", "correct": false, "edits_per_token": 0.0}, {"uid": "a07ed018d16711ef", "family": "bind", "template": 1, "depth": 4, "pred": "101", "truth": "81", "correct": false, "edits_per_token": 0.0}, {"uid": "8566721d2752b3a8", "family": "bind", "template": 1, "depth": 4, "pred": "-12", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "14e0f0613ab5fafb", "family": "bind", "template": 1, "depth": 4, "pred": "43", "truth": "41", "correct": false, "edits_per_token": 0.0}, {"uid": "e5b10de969f4f1de", "family": "bind", "template": 1, "depth": 4, "pred": "-20", "truth": "-22", "correct": false, "edits_per_token": 0.0}, {"uid": "9a6304b25504acd5", "family": "fsm", "template": 2, "depth": 4, "pred": "13", "truth": "12", "correct": false, "edits_per_token": 0.0}, {"uid": "f9e4c5acf05b6575", "family": "bind", "template": 1, "depth": 4, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 0.0}, {"uid": "2efa0c713c22db5a", "family": "bind", "template": 1, "depth": 4, "pred": "102", "truth": "101", "correct": false, "edits_per_token": 0.0}, {"uid": "8ce65aa06b428682", "family": "fsm", "template": 0, "depth": 3, "pred": "22", "truth": "24", "correct": false, "edits_per_token": 0.0}, {"uid": "4f7a4c9e9fd88e23", "family": "fsm", "template": 1, "depth": 3, "pred": "-4", "truth": "-4", "correct": true, "edits_per_token": 0.0}, {"uid": "2468d102e506d6d5", "family": "fsm", "template": 2, "depth": 3, "pred": "12", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "6f3c8de73142af82", "family": "fsm", "template": 2, "depth": 4, "pred": "35", "truth": "37", "correct": false, "edits_per_token": 0.0}, {"uid": "ba3af2217536a7c6", "family": "fsm", "template": 0, "depth": 2, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "e063912681967ddf", "family": "fsm", "template": 0, "depth": 3, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 0.0}, {"uid": "583473e83be2cb0e", "family": "fsm", "template": 2, "depth": 3, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 0.0}, {"uid": "5a873e995116c10d", "family": "fsm", "template": 1, "depth": 3, "pred": "80", "truth": "72", "correct": false, "edits_per_token": 0.0}, {"uid": "05764ff46e4d28b8", "family": "fsm", "template": 2, "depth": 4, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "46b570766ec22e2a", "family": "fsm", "template": 0, "depth": 3, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "59f43bee78d1b7d5", "family": "fsm", "template": 0, "depth": 3, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 0.0}, {"uid": "eaf14d7290a1413b", "family": "fsm", "template": 1, "depth": 3, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "b7248f02464a9d5e", "family": "fsm", "template": 1, "depth": 3, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "9aaa1b5b75c855bb", "family": "fsm", "template": 2, "depth": 3, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 0.0}, {"uid": "a7203ce2418d2ce9", "family": "fsm", "template": 0, "depth": 4, "pred": "18", "truth": "18", "correct": true, "edits_per_token": 0.0}, {"uid": "b86f2cf77ff75b95", "family": "fsm", "template": 2, "depth": 4, "pred": "9", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "5480488e0f55e2a2", "family": "fsm", "template": 0, "depth": 3, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "ca83fc2e201063f6", "family": "fsm", "template": 0, "depth": 3, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 0.0}, {"uid": "324c8cdae8f3169c", "family": "fsm", "template": 1, "depth": 4, "pred": "-4", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "2723eee2e2d0f964", "family": "fsm", "template": 1, "depth": 3, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "8749497eaab6176d", "family": "fsm", "template": 1, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "f006a9e07042e43f", "family": "fsm", "template": 2, "depth": 4, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 0.0}, {"uid": "def842732bc0714f", "family": "fsm", "template": 2, "depth": 4, "pred": "-4", "truth": "-4", "correct": true, "edits_per_token": 0.0}, {"uid": "e4ca09128be213ea", "family": "fsm", "template": 0, "depth": 3, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 0.0}, {"uid": "a1c297725f5ae175", "family": "fsm", "template": 1, "depth": 4, "pred": "0", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "7d9f104b41e0b7a5", "family": "fsm", "template": 0, "depth": 4, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "536ead7bf94947ee", "family": "fsm", "template": 1, "depth": 4, "pred": "18", "truth": "18", "correct": true, "edits_per_token": 0.0}, {"uid": "9808a14025bb62db", "family": "fsm", "template": 1, "depth": 4, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "666a3ad814af7485", "family": "fsm", "template": 0, "depth": 4, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 0.0}, {"uid": "29e5f1d39e9dd718", "family": "excl", "template": 2, "depth": 1, "pred": "apple plum", "truth": "date", "correct": false, "edits_per_token": 0.0}, {"uid": "4425b91f251fc921", "family": "excl", "template": 2, "depth": 1, "pred": "plum pear", "truth": "lime", "correct": false, "edits_per_token": 0.0}, {"uid": "899865b46b45638d", "family": "sched", "template": 2, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "bb4df57d9a04ccab", "family": "fsm", "template": 1, "depth": 3, "pred": "-5", "truth": "-5", "correct": true, "edits_per_token": 0.0}, {"uid": "5aee313fef7904e6", "family": "fsm", "template": 2, "depth": 4, "pred": "0", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "b6a823db9fe091cf", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "eae15d4721c93c28", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "ebdd7e877a44588b", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "a467884db4d82ee1", "family": "fsm", "template": 1, "depth": 4, "pred": "10", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "a0e13fa29eaee344", "family": "fsm", "template": 0, "depth": 4, "pred": "18", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "55bddbb405630c49", "family": "excl", "template": 2, "depth": 1, "pred": "plum lime a", "truth": "kiwi plum apple", "correct": false, "edits_per_token": 0.0}, {"uid": "9e10c6f2c824ea03", "family": "fsm", "template": 1, "depth": 4, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "e29749c80774340d", "family": "fsm", "template": 1, "depth": 4, "pred": "-7", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "305ec8d13dc601f2", "family": "excl", "template": 2, "depth": 1, "pred": "apple fig b", "truth": "kiwi fig yam bean", "correct": false, "edits_per_token": 0.0}, {"uid": "c56841284720069d", "family": "fsm", "template": 0, "depth": 4, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 0.0}, {"uid": "b757b7fb6b160032", "family": "fsm", "template": 0, "depth": 4, "pred": "-8", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "8a849a68ac0bd89d", "family": "sched", "template": 1, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "24c9898cc5168e60", "family": "sched", "template": 1, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "b45de26fb239a53c", "family": "fsm", "template": 0, "depth": 4, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "b92a8bb6a6b6c2fc", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "24f6f2d470ed5ab8", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "265c9cd1ac273c94", "family": "sched", "template": 2, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "17e81034d97a73e1", "family": "excl", "template": 1, "depth": 2, "pred": "pear date y", "truth": "yam", "correct": false, "edits_per_token": 0.0}, {"uid": "52b9e62b2a6ca3c2", "family": "excl", "template": 2, "depth": 3, "pred": "fig", "truth": "yam", "correct": false, "edits_per_token": 0.0}, {"uid": "3ea463a1eb42a642", "family": "excl", "template": 2, "depth": 1, "pred": "bean fig pl", "truth": "bean plum date nut", "correct": false, "edits_per_token": 0.0}, {"uid": "290428df35319b49", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "e8f3f5315f909938", "family": "excl", "template": 2, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "679e75695e94a8a8", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "b9763ee8284b60af", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "50436cd63f7d344d", "family": "excl", "template": 1, "depth": 1, "pred": "bean apple", "truth": "bean date apple", "correct": false, "edits_per_token": 0.0}, {"uid": "f91565e19106bd0c", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "f377b423248cc145", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "fd984d3e4a7605fd", "family": "excl", "template": 0, "depth": 2, "pred": "pear kiwi", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "78ec3d641e6859a8", "family": "excl", "template": 2, "depth": 2, "pred": "kiwi pear", "truth": "date apple", "correct": false, "edits_per_token": 0.0}, {"uid": "ec07ebafd22ecb2e", "family": "excl", "template": 1, "depth": 2, "pred": "plum", "truth": "pear", "correct": false, "edits_per_token": 0.0}, {"uid": "637c344de20f5625", "family": "excl", "template": 2, "depth": 2, "pred": "kiwi nut", "truth": "apple date fig bean", "correct": false, "edits_per_token": 0.0}, {"uid": "b1127806bf880411", "family": "sched", "template": 0, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "a65132179f29f33e", "family": "excl", "template": 0, "depth": 1, "pred": "apple lime", "truth": "apple lime", "correct": true, "edits_per_token": 0.0}, {"uid": "610ab75a850b6358", "family": "sched", "template": 0, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "09a0f4119071f489", "family": "excl", "template": 2, "depth": 1, "pred": "pear apple", "truth": "pear kiwi fig nut", "correct": false, "edits_per_token": 0.0}, {"uid": "9c0d7b46b835ff7f", "family": "excl", "template": 2, "depth": 1, "pred": "plum fig pe", "truth": "plum kiwi date nut", "correct": false, "edits_per_token": 0.0}, {"uid": "e9948bd8b29e69d3", "family": "excl", "template": 2, "depth": 2, "pred": "lime yam ki", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "f948792de5a76409", "family": "excl", "template": 0, "depth": 1, "pred": "kiwi fig li", "truth": "apple kiwi lime fig", "correct": false, "edits_per_token": 0.0}, {"uid": "43382d613173a703", "family": "sched", "template": 1, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "4378e430fe1485f9", "family": "sched", "template": 1, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "7af9657dd3fba74a", "family": "excl", "template": 2, "depth": 2, "pred": "plum", "truth": "date plum", "correct": false, "edits_per_token": 0.0}, {"uid": "c9dda90c7d0ad8ed", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "2f7d7388cf61d3fe", "family": "sched", "template": 1, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "114ddae29f526626", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "d95b1cb8c2923e7e", "family": "excl", "template": 0, "depth": 1, "pred": "bean lime a", "truth": "bean lime apple", "correct": false, "edits_per_token": 0.0}, {"uid": "1dcd25a6eb16038d", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "59114c794ee3a087", "family": "excl", "template": 0, "depth": 2, "pred": "bean plum p", "truth": "bean", "correct": false, "edits_per_token": 0.0}, {"uid": "a9742f748ad1ad5c", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "ec7190a280bfbdb7", "family": "excl", "template": 2, "depth": 4, "pred": "apple", "truth": "lime", "correct": false, "edits_per_token": 0.0}, {"uid": "81a4bc002847b518", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "fce4affe8df7979a", "family": "excl", "template": 1, "depth": 1, "pred": "nut date li", "truth": "date apple lime", "correct": false, "edits_per_token": 0.0}, {"uid": "50c50f83a59dabd8", "family": "excl", "template": 2, "depth": 2, "pred": "yam plum pe", "truth": "fig bean kiwi plum apple", "correct": false, "edits_per_token": 0.0}, {"uid": "9a0b5958ebfcc667", "family": "sched", "template": 1, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "2b9fc22b15089625", "family": "excl", "template": 0, "depth": 1, "pred": "pear apple", "truth": "pear kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "2d7af060d17ced33", "family": "sched", "template": 1, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "22973924b5d55519", "family": "sched", "template": 1, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "9eba62cb7a72f599", "family": "sched", "template": 2, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "60adda9ef6e12d3c", "family": "excl", "template": 0, "depth": 1, "pred": "kiwi pear a", "truth": "kiwi bean apple", "correct": false, "edits_per_token": 0.0}, {"uid": "222e860f500afa48", "family": "excl", "template": 1, "depth": 1, "pred": "pear fig nu", "truth": "pear nut fig lime", "correct": false, "edits_per_token": 0.0}, {"uid": "883e00248131789c", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "b7eae28e49657a0d", "family": "sched", "template": 2, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "36bc16e15622ea97", "family": "sched", "template": 2, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "9b1170c69bedb5d0", "family": "excl", "template": 1, "depth": 3, "pred": "nut date", "truth": "plum date", "correct": false, "edits_per_token": 0.0}, {"uid": "e7a942f9776e1481", "family": "excl", "template": 1, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "03e8f11a87973087", "family": "sched", "template": 0, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "0641e2dbfe875c84", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "b60c5ce59c8ad584", "family": "sched", "template": 2, "depth": 3, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "a07385bb7e4ea97b", "family": "sched", "template": 0, "depth": 2, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "a942d085ec964894", "family": "sched", "template": 2, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "800f61c9e4ad0bf4", "family": "excl", "template": 1, "depth": 2, "pred": "fig lime", "truth": "nut bean", "correct": false, "edits_per_token": 0.0}, {"uid": "19887d3ca18adb40", "family": "sched", "template": 0, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "01dd5269d9e0a510", "family": "sched", "template": 0, "depth": 3, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "d1578ac0adce4a3f", "family": "sched", "template": 2, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "4f7ac4cb4667edb6", "family": "excl", "template": 0, "depth": 2, "pred": "lime apple", "truth": "bean date", "correct": false, "edits_per_token": 0.0}, {"uid": "9ebb20923f7ff360", "family": "sched", "template": 0, "depth": 3, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "45e5e22be65e2392", "family": "sched", "template": 2, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "58590db8d34f4bb6", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "e0aa686b67447872", "family": "excl", "template": 1, "depth": 1, "pred": "none", "truth": "lime fig", "correct": false, "edits_per_token": 0.0}, {"uid": "b9758ae4de312e38", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "apple", "correct": false, "edits_per_token": 0.0}, {"uid": "49b538b6cebfad32", "family": "sched", "template": 0, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "b41c090edc1c2570", "family": "sched", "template": 0, "depth": 2, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "57f25f911fd47979", "family": "excl", "template": 0, "depth": 2, "pred": "fig lime", "truth": "bean", "correct": false, "edits_per_token": 0.0}, {"uid": "52a3076c67262025", "family": "excl", "template": 0, "depth": 2, "pred": "bean fig", "truth": "bean fig date", "correct": false, "edits_per_token": 0.0}, {"uid": "5313d51deae8c2b8", "family": "excl", "template": 0, "depth": 3, "pred": "kiwi", "truth": "yam", "correct": false, "edits_per_token": 0.0}, {"uid": "fbb63a5c5980a546", "family": "excl", "template": 1, "depth": 1, "pred": "plum nut", "truth": "date apple", "correct": false, "edits_per_token": 0.0}, {"uid": "123ddaa3e5efb215", "family": "excl", "template": 1, "depth": 1, "pred": "fig bean pe", "truth": "fig pear nut kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "bf6d2e6f97942225", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "3a419d90cdd84493", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "yam plum nut", "correct": false, "edits_per_token": 0.0}, {"uid": "dc7a2b700ef3eec3", "family": "sched", "template": 0, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "a05184e028d82ee0", "family": "sched", "template": 0, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "b9e25b6eb179c90b", "family": "excl", "template": 0, "depth": 3, "pred": "kiwi bean p", "truth": "kiwi nut fig pear", "correct": false, "edits_per_token": 0.0}, {"uid": "914dc5465d528e9b", "family": "sched", "template": 0, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "64f71380d66efae2", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "33fe241315cf5b9f", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "1b7f8f05a6c5a7a3", "family": "excl", "template": 2, "depth": 2, "pred": "plum apple", "truth": "plum apple", "correct": true, "edits_per_token": 0.0}, {"uid": "5c13ea4ff7d521b5", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "2a2cf42fd19abbe2", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "df171a88e16d0896", "family": "excl", "template": 0, "depth": 2, "pred": "yam", "truth": "apple yam", "correct": false, "edits_per_token": 0.0}, {"uid": "a6b723b79a990fea", "family": "excl", "template": 1, "depth": 1, "pred": "nut lime zi", "truth": "nut kiwi lime fig", "correct": false, "edits_per_token": 0.0}, {"uid": "95569dc6b1f0424e", "family": "excl", "template": 2, "depth": 4, "pred": "fig", "truth": "nut fig", "correct": false, "edits_per_token": 0.0}, {"uid": "d5cb804202c06b04", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "26a8bdf6da59e053", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "8461de28662a09d0", "family": "excl", "template": 0, "depth": 1, "pred": "yam bean", "truth": "lime kiwi apple", "correct": false, "edits_per_token": 0.0}, {"uid": "5266b24716f21c1b", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "1301b2b886fc0fed", "family": "sched", "template": 1, "depth": 3, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "0ae3e633b9c73a6d", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "a34d5263586bf792", "family": "sched", "template": 1, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "72b16296ef29803c", "family": "sched", "template": 2, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "1650d27be4762458", "family": "sched", "template": 2, "depth": 4, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "d17c2abfac01fb85", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "f7bb7ae8302c82e3", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "ce88ea5df4ac7c7e", "family": "sched", "template": 1, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "89232c53de86f0c7", "family": "sched", "template": 2, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "7867942486d7f380", "family": "sched", "template": 0, "depth": 4, "pred": "none", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "c9726e189052aad1", "family": "sched", "template": 2, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 0.0}, {"uid": "302f485c304f8e1c", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "58e2a723355b2957", "family": "sched", "template": 0, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "376a16c13fc8af71", "family": "sched", "template": 0, "depth": 4, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "7eddd2c6ff72770f", "family": "sched", "template": 0, "depth": 4, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "ebcfc88ef93a66df", "family": "sched", "template": 2, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "89a256c76069f616", "family": "excl", "template": 1, "depth": 3, "pred": "plum", "truth": "plum lime", "correct": false, "edits_per_token": 0.0}, {"uid": "34292f4e57b84e60", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "b4ef07715474f790", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "084073be6e34810e", "family": "excl", "template": 2, "depth": 2, "pred": "lime", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "fec42ca9bcf0a20b", "family": "excl", "template": 1, "depth": 2, "pred": "lime", "truth": "date plum bean", "correct": false, "edits_per_token": 0.0}, {"uid": "9600b1b302d34721", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "e68c491cb5fe0db5", "family": "excl", "template": 0, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "8fbf1093b19d178d", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "152fa59b6f060034", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "6886f89d5fecebc2", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "25810ef41631a5a0", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "date", "correct": false, "edits_per_token": 0.0}, {"uid": "c2c1af96873ca940", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "493eb05c569e6806", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "yam fig date", "correct": false, "edits_per_token": 0.0}, {"uid": "3ca58bdf2e93bec8", "family": "excl", "template": 1, "depth": 2, "pred": "apple yam b", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "68cafc987827b672", "family": "excl", "template": 0, "depth": 1, "pred": "apple yam f", "truth": "apple lime nut kiwi bean", "correct": false, "edits_per_token": 0.0}, {"uid": "e5f870c9ac9c2bdc", "family": "excl", "template": 1, "depth": 2, "pred": "apple fig", "truth": "apple pear plum", "correct": false, "edits_per_token": 0.0}, {"uid": "4aba72aed6f820bb", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "bean", "correct": false, "edits_per_token": 0.0}, {"uid": "7e3d734f2577f012", "family": "excl", "template": 2, "depth": 3, "pred": "pear apple", "truth": "plum bean", "correct": false, "edits_per_token": 0.0}, {"uid": "da06a708de7351a1", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "e13f444f5e921059", "family": "sched", "template": 1, "depth": 4, "pred": "A", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "806dfca57003d938", "family": "sched", "template": 1, "depth": 4, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "5a3700bf53c0daf3", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "fa8b059c364916e0", "family": "sched", "template": 1, "depth": 4, "pred": "D", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "9fd10a99f31cee6e", "family": "sched", "template": 1, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "b3fe0fa2ac161f59", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "2489ee7572296937", "family": "sched", "template": 2, "depth": 4, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "80d519c064361dae", "family": "sched", "template": 2, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "5a66b965f98dc2cf", "family": "excl", "template": 0, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "4cd96fbf4f41a908", "family": "sched", "template": 0, "depth": 4, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "7a4191b42dce2b06", "family": "sched", "template": 1, "depth": 3, "pred": "none", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "b87404e94d9a09f6", "family": "excl", "template": 0, "depth": 2, "pred": "plum apple", "truth": "fig kiwi pear", "correct": false, "edits_per_token": 0.0}, {"uid": "2a9ecd3661b30257", "family": "sched", "template": 0, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 0.0}, {"uid": "5249781ebc348d37", "family": "sched", "template": 2, "depth": 4, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "2e4acd7d13860ac2", "family": "sched", "template": 2, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "2adc39419c8b49a4", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "fig lime", "correct": false, "edits_per_token": 0.0}, {"uid": "dca6675d28527584", "family": "excl", "template": 0, "depth": 3, "pred": "fig yam", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "c666c9854336f784", "family": "sched", "template": 0, "depth": 4, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "8f480bee1ead1646", "family": "excl", "template": 0, "depth": 3, "pred": "nut yam", "truth": "apple", "correct": false, "edits_per_token": 0.0}, {"uid": "cf67501c8d912952", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "d61260228e19e8ba", "family": "sched", "template": 2, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "71fb11da7ef26fa8", "family": "sched", "template": 0, "depth": 4, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "f12148b1efc8abed", "family": "excl", "template": 2, "depth": 4, "pred": "date", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "9e76644064b0dc70", "family": "excl", "template": 0, "depth": 3, "pred": "date yam", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "12b5acb0d562716c", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "fd29037f3e23207c", "family": "sched", "template": 1, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 0.0}, {"uid": "dbd4f3b44d2455e6", "family": "sched", "template": 1, "depth": 4, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "2bbd29545a49585f", "family": "sched", "template": 1, "depth": 4, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "e7a65045762e87f2", "family": "sched", "template": 1, "depth": 4, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "9476a68c1c10bc9b", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "dfb1aad2e59286cb", "family": "excl", "template": 0, "depth": 4, "pred": "pear", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "fe2915e5def8bf83", "family": "excl", "template": 1, "depth": 4, "pred": "fig apple n", "truth": "yam", "correct": false, "edits_per_token": 0.0}, {"uid": "548b800800b9067a", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "19a75d08ddd9d502", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "6c5b7174f7312602", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "11c0e819fd3fe52e", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}] | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_iid_b2.json
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +[{"uid": "d57de33b3b3613ce", "family": "fsm", "template": 2, "depth": 1, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 2.0}, {"uid": "2fcbef2f6deb569d", "family": "bind", "template": 2, "depth": 1, "pred": "18", "truth": "17", "correct": false, "edits_per_token": 2.0}, {"uid": "facfe4b45751726b", "family": "bind", "template": 2, "depth": 1, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 2.0}, {"uid": "17c2f737ab6e9a12", "family": "bind", "template": 2, "depth": 1, "pred": "21", "truth": "21", "correct": true, "edits_per_token": 2.0}, {"uid": "330a5096d00ea26d", "family": "bind", "template": 2, "depth": 1, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "67c60958afd432d0", "family": "bind", "template": 2, "depth": 1, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 2.0}, {"uid": "0df874a819030335", "family": "bind", "template": 2, "depth": 1, "pred": "60", "truth": "60", "correct": true, "edits_per_token": 2.0}, {"uid": "f8b6c686efa42185", "family": "bind", "template": 2, "depth": 1, "pred": "27", "truth": "26", "correct": false, "edits_per_token": 2.0}, {"uid": "61ab4c048732d1a3", "family": "fsm", "template": 2, "depth": 1, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "a0a9effb7a5a1771", "family": "fsm", "template": 2, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "e539dc5c812c9f8f", "family": "arith", "template": 0, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "b9080d7276f3e48d", "family": "arith", "template": 1, "depth": 1, "pred": "14", "truth": "14", "correct": true, "edits_per_token": 2.0}, {"uid": "f19b2d158a1b2321", "family": "arith", "template": 0, "depth": 1, "pred": "-2", "truth": "-2", "correct": true, "edits_per_token": 2.0}, {"uid": "b50334287c335a68", "family": "arith", "template": 0, "depth": 1, "pred": "13", "truth": "13", "correct": true, "edits_per_token": 2.0}, {"uid": "5203cff28a90efc8", "family": "arith", "template": 0, "depth": 1, "pred": "14", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "b0fc8aed8de80f82", "family": "arith", "template": 1, "depth": 1, "pred": "85", "truth": "85", "correct": true, "edits_per_token": 2.0}, {"uid": "d3f8213a4e9e4170", "family": "arith", "template": 1, "depth": 1, "pred": "64", "truth": "66", "correct": false, "edits_per_token": 2.0}, {"uid": "6dbb743f0c127dc2", "family": "arith", "template": 1, "depth": 1, "pred": "100", "truth": "120", "correct": false, "edits_per_token": 2.0}, {"uid": "94bfbb133f315b17", "family": "arith", "template": 0, "depth": 1, "pred": "37", "truth": "37", "correct": true, "edits_per_token": 2.0}, {"uid": "c5dd86b6955d4021", "family": "arith", "template": 1, "depth": 1, "pred": "-7", "truth": "-7", "correct": true, "edits_per_token": 2.0}, {"uid": "75eceacb14c3d0e6", "family": "bind", "template": 0, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "f3b82055a8272259", "family": "bind", "template": 0, "depth": 1, "pred": "18", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "6fa8f401c1e6cf13", "family": "bind", "template": 0, "depth": 1, "pred": "24", "truth": "22", "correct": false, "edits_per_token": 2.0}, {"uid": "5252b66c833e4804", "family": "bind", "template": 0, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 2.0}, {"uid": "0dd2fab9c5861d6d", "family": "bind", "template": 0, "depth": 1, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 2.0}, {"uid": "d8aa01c2e9dbb03f", "family": "bind", "template": 1, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 2.0}, {"uid": "1d20f4c03960f839", "family": "arith", "template": 0, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 2.0}, {"uid": "a0d75f1874311fde", "family": "bind", "template": 0, "depth": 1, "pred": "26", "truth": "27", "correct": false, "edits_per_token": 2.0}, {"uid": "2f252bbc438ce228", "family": "bind", "template": 0, "depth": 1, "pred": "24", "truth": "24", "correct": true, "edits_per_token": 2.0}, {"uid": "910350946f41fbf1", "family": "bind", "template": 1, "depth": 1, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 2.0}, {"uid": "322ec1e8c6acc1ff", "family": "bind", "template": 1, "depth": 1, "pred": "24", "truth": "24", "correct": true, "edits_per_token": 2.0}, {"uid": "8f08654666473a62", "family": "bind", "template": 1, "depth": 1, "pred": "8", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "e74c59b882dcf2b2", "family": "bind", "template": 1, "depth": 1, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 2.0}, {"uid": "f4ea26539e1674ec", "family": "arith", "template": 0, "depth": 2, "pred": "28", "truth": "28", "correct": true, "edits_per_token": 2.0}, {"uid": "8461e207ba7b8587", "family": "arith", "template": 1, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 2.0}, {"uid": "d43d09af762bceee", "family": "arith", "template": 1, "depth": 2, "pred": "41", "truth": "44", "correct": false, "edits_per_token": 2.0}, {"uid": "7ed459fad79fdf0b", "family": "bind", "template": 1, "depth": 1, "pred": "13", "truth": "12", "correct": false, "edits_per_token": 2.0}, {"uid": "ce3cbdbb1894d8f4", "family": "bind", "template": 1, "depth": 1, "pred": "14", "truth": "15", "correct": false, "edits_per_token": 2.0}, {"uid": "14947545562906ad", "family": "bind", "template": 1, "depth": 1, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 2.0}, {"uid": "697d570c40384635", "family": "bind", "template": 2, "depth": 2, "pred": "11", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "b78e61c19225f49c", "family": "fsm", "template": 2, "depth": 2, "pred": "19", "truth": "18", "correct": false, "edits_per_token": 2.0}, {"uid": "69953fd72fce090d", "family": "arith", "template": 0, "depth": 2, "pred": "36", "truth": "38", "correct": false, "edits_per_token": 2.0}, {"uid": "66cdadf971fee2ab", "family": "arith", "template": 0, "depth": 2, "pred": "69", "truth": "71", "correct": false, "edits_per_token": 2.0}, {"uid": "99c7a494c455e41d", "family": "arith", "template": 0, "depth": 2, "pred": "-62", "truth": "-60", "correct": false, "edits_per_token": 2.0}, {"uid": "8d21bd8bbb5eecad", "family": "arith", "template": 0, "depth": 2, "pred": "42", "truth": "41", "correct": false, "edits_per_token": 2.0}, {"uid": "5f3cf3ff50e3e9c1", "family": "arith", "template": 1, "depth": 2, "pred": "40", "truth": "41", "correct": false, "edits_per_token": 2.0}, {"uid": "ce6f8c794a4d97fb", "family": "arith", "template": 1, "depth": 2, "pred": "51", "truth": "55", "correct": false, "edits_per_token": 2.0}, {"uid": "56607dee807b8fd6", "family": "bind", "template": 2, "depth": 2, "pred": "17", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "206a446298d63b02", "family": "bind", "template": 2, "depth": 2, "pred": "40", "truth": "40", "correct": true, "edits_per_token": 2.0}, {"uid": "4dd1c85d972bb798", "family": "bind", "template": 2, "depth": 2, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 2.0}, {"uid": "60d954acf09287f1", "family": "arith", "template": 0, "depth": 2, "pred": "48", "truth": "46", "correct": false, "edits_per_token": 2.0}, {"uid": "5521b704b5ac25a7", "family": "arith", "template": 1, "depth": 2, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 2.0}, {"uid": "b66b330f45989ec2", "family": "arith", "template": 1, "depth": 2, "pred": "12", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "f06a723961f75f4d", "family": "arith", "template": 1, "depth": 2, "pred": "-34", "truth": "-36", "correct": false, "edits_per_token": 2.0}, {"uid": "1ffed152f8b72d39", "family": "bind", "template": 2, "depth": 2, "pred": "4", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "3b4d5aeb1e813243", "family": "arith", "template": 0, "depth": 3, "pred": "108", "truth": "140", "correct": false, "edits_per_token": 2.0}, {"uid": "e56fea6bf4230916", "family": "bind", "template": 2, "depth": 2, "pred": "1", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "9f5c3b31e7d60900", "family": "bind", "template": 2, "depth": 2, "pred": "28", "truth": "31", "correct": false, "edits_per_token": 2.0}, {"uid": "ba60a76e7ce73c9a", "family": "arith", "template": 0, "depth": 3, "pred": "165", "truth": "165", "correct": true, "edits_per_token": 2.0}, {"uid": "caa2ffb073e3b554", "family": "arith", "template": 0, "depth": 3, "pred": "-80", "truth": "-80", "correct": true, "edits_per_token": 2.0}, {"uid": "f3df8ca3c7f3fe73", "family": "arith", "template": 0, "depth": 3, "pred": "-350", "truth": "-300", "correct": false, "edits_per_token": 2.0}, {"uid": "6a521ae83b9560cb", "family": "arith", "template": 0, "depth": 3, "pred": "308", "truth": "395", "correct": false, "edits_per_token": 2.0}, {"uid": "b7e7635fb68761fa", "family": "arith", "template": 1, "depth": 3, "pred": "80", "truth": "78", "correct": false, "edits_per_token": 2.0}, {"uid": "69168d7303f42428", "family": "arith", "template": 1, "depth": 3, "pred": "368", "truth": "352", "correct": false, "edits_per_token": 2.0}, {"uid": "3876cda47832bc78", "family": "arith", "template": 1, "depth": 3, "pred": "60", "truth": "66", "correct": false, "edits_per_token": 2.0}, {"uid": "101a933392cdc486", "family": "arith", "template": 0, "depth": 3, "pred": "-9", "truth": "-8", "correct": false, "edits_per_token": 2.0}, {"uid": "5d6734a6c29bf98b", "family": "arith", "template": 1, "depth": 3, "pred": "-38", "truth": "-38", "correct": true, "edits_per_token": 2.0}, {"uid": "e6b4bb7a789c1bbf", "family": "arith", "template": 1, "depth": 3, "pred": "-36", "truth": "-36", "correct": true, "edits_per_token": 2.0}, {"uid": "529ef2fbb6693bf4", "family": "arith", "template": 0, "depth": 3, "pred": "25", "truth": "27", "correct": false, "edits_per_token": 2.0}, {"uid": "4264ff894995b84a", "family": "arith", "template": 0, "depth": 4, "pred": "111", "truth": "69", "correct": false, "edits_per_token": 2.0}, {"uid": "4fd0a06acba9c2bd", "family": "arith", "template": 1, "depth": 3, "pred": "77", "truth": "75", "correct": false, "edits_per_token": 2.0}, {"uid": "11ddfaa4634944bc", "family": "arith", "template": 1, "depth": 3, "pred": "-12", "truth": "-11", "correct": false, "edits_per_token": 2.0}, {"uid": "81903019667d210a", "family": "arith", "template": 0, "depth": 4, "pred": "-21", "truth": "-24", "correct": false, "edits_per_token": 2.0}, {"uid": "f025613adc1381e6", "family": "arith", "template": 0, "depth": 4, "pred": "77", "truth": "67", "correct": false, "edits_per_token": 2.0}, {"uid": "6917d247eea87098", "family": "arith", "template": 1, "depth": 4, "pred": "1720", "truth": "1400", "correct": false, "edits_per_token": 2.0}, {"uid": "5ac4cc428fcee7ff", "family": "bind", "template": 0, "depth": 2, "pred": "120", "truth": "180", "correct": false, "edits_per_token": 2.0}, {"uid": "a7fa5ddf82e3acb6", "family": "bind", "template": 0, "depth": 2, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 2.0}, {"uid": "85a6f3db0aee68b6", "family": "fsm", "template": 0, "depth": 1, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 2.0}, {"uid": "306790bc18e1ff0f", "family": "fsm", "template": 1, "depth": 1, "pred": "-5", "truth": "-5", "correct": true, "edits_per_token": 2.0}, {"uid": "7b35ad2e31d3fe01", "family": "fsm", "template": 1, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 2.0}, {"uid": "f692ca3a30b9615b", "family": "arith", "template": 0, "depth": 4, "pred": "27", "truth": "27", "correct": true, "edits_per_token": 2.0}, {"uid": "00456af1ebb8a6f6", "family": "arith", "template": 1, "depth": 4, "pred": "-16", "truth": "-13", "correct": false, "edits_per_token": 2.0}, {"uid": "8636f76e7a05df5c", "family": "bind", "template": 0, "depth": 2, "pred": "-15", "truth": "-10", "correct": false, "edits_per_token": 2.0}, {"uid": "6894af54caeca8f0", "family": "bind", "template": 0, "depth": 2, "pred": "52", "truth": "56", "correct": false, "edits_per_token": 2.0}, {"uid": "15a26f31dc193ec8", "family": "bind", "template": 0, "depth": 2, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 2.0}, {"uid": "fbf1f2cb25ea61f5", "family": "bind", "template": 0, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 2.0}, {"uid": "e1c776eb2e759087", "family": "arith", "template": 0, "depth": 4, "pred": "77", "truth": "66", "correct": false, "edits_per_token": 2.0}, {"uid": "b57b7bb076a588da", "family": "arith", "template": 1, "depth": 4, "pred": "488", "truth": "480", "correct": false, "edits_per_token": 2.0}, {"uid": "c96adbcbd0c51bd0", "family": "arith", "template": 1, "depth": 4, "pred": "52", "truth": "52", "correct": true, "edits_per_token": 2.0}, {"uid": "058b50085974c009", "family": "arith", "template": 1, "depth": 4, "pred": "19", "truth": "19", "correct": true, "edits_per_token": 2.0}, {"uid": "cbc5614b2da64a6a", "family": "arith", "template": 1, "depth": 4, "pred": "-55", "truth": "-50", "correct": false, "edits_per_token": 2.0}, {"uid": "ffd3f5ed636cb166", "family": "bind", "template": 0, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 2.0}, {"uid": "e48b5252059e24a7", "family": "bind", "template": 1, "depth": 2, "pred": "-7", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "58cc6ff1fb3e1166", "family": "bind", "template": 1, "depth": 2, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 2.0}, {"uid": "9dbdbf7552d0f726", "family": "bind", "template": 1, "depth": 2, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 2.0}, {"uid": "02f04b4ab6addf22", "family": "fsm", "template": 2, "depth": 2, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 2.0}, {"uid": "a7df7a7b48f0e07c", "family": "fsm", "template": 2, "depth": 2, "pred": "-3", "truth": "-3", "correct": true, "edits_per_token": 2.0}, {"uid": "4d05c02585d8db9c", "family": "fsm", "template": 2, "depth": 2, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "54b414556cb4b641", "family": "fsm", "template": 2, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 2.0}, {"uid": "afe7716d466e8527", "family": "arith", "template": 0, "depth": 4, "pred": "-12", "truth": "-10", "correct": false, "edits_per_token": 2.0}, {"uid": "a3a41fe21c3b68c3", "family": "arith", "template": 0, "depth": 4, "pred": "-93", "truth": "-106", "correct": false, "edits_per_token": 2.0}, {"uid": "1e60abcad3f7cd49", "family": "arith", "template": 1, "depth": 4, "pred": "26", "truth": "29", "correct": false, "edits_per_token": 2.0}, {"uid": "5b477191fed1d1ab", "family": "bind", "template": 1, "depth": 2, "pred": "-8", "truth": "-8", "correct": true, "edits_per_token": 2.0}, {"uid": "b618508093791a17", "family": "bind", "template": 1, "depth": 2, "pred": "210", "truth": "240", "correct": false, "edits_per_token": 2.0}, {"uid": "758626244ff1e265", "family": "bind", "template": 1, "depth": 2, "pred": "45", "truth": "45", "correct": true, "edits_per_token": 2.0}, {"uid": "53f6d7daa1e917ac", "family": "bind", "template": 2, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "7a489d579355664c", "family": "arith", "template": 2, "depth": 1, "pred": "48", "truth": "48", "correct": true, "edits_per_token": 2.0}, {"uid": "866964b48f15dc73", "family": "arith", "template": 2, "depth": 1, "pred": "22", "truth": "21", "correct": false, "edits_per_token": 2.0}, {"uid": "326ed5326c7a15fa", "family": "arith", "template": 2, "depth": 1, "pred": "64", "truth": "60", "correct": false, "edits_per_token": 2.0}, {"uid": "78b68accb86904aa", "family": "bind", "template": 1, "depth": 2, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 2.0}, {"uid": "12ab692979396f5c", "family": "bind", "template": 2, "depth": 3, "pred": "37", "truth": "36", "correct": false, "edits_per_token": 2.0}, {"uid": "4b618bfde19bcd4c", "family": "bind", "template": 2, "depth": 3, "pred": "-8", "truth": "-7", "correct": false, "edits_per_token": 2.0}, {"uid": "99b184a684af7db3", "family": "bind", "template": 2, "depth": 3, "pred": "-12", "truth": "-15", "correct": false, "edits_per_token": 2.0}, {"uid": "09b2aab4939d318c", "family": "fsm", "template": 2, "depth": 3, "pred": "19", "truth": "19", "correct": true, "edits_per_token": 2.0}, {"uid": "b93e590222b68eba", "family": "arith", "template": 2, "depth": 1, "pred": "-18", "truth": "-18", "correct": true, "edits_per_token": 2.0}, {"uid": "e12b670d52d4bcac", "family": "arith", "template": 2, "depth": 1, "pred": "31", "truth": "34", "correct": false, "edits_per_token": 2.0}, {"uid": "4247b2f7e094279e", "family": "bind", "template": 2, "depth": 3, "pred": "19", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "67dfb1d3b0773f0e", "family": "bind", "template": 2, "depth": 3, "pred": "48", "truth": "43", "correct": false, "edits_per_token": 2.0}, {"uid": "8254a0f979194a20", "family": "bind", "template": 2, "depth": 3, "pred": "11", "truth": "11", "correct": true, "edits_per_token": 2.0}, {"uid": "7e2f3ac39edf259e", "family": "bind", "template": 2, "depth": 3, "pred": "29", "truth": "29", "correct": true, "edits_per_token": 2.0}, {"uid": "f78cbce45d518768", "family": "fsm", "template": 0, "depth": 1, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 2.0}, {"uid": "67327201e0703535", "family": "fsm", "template": 1, "depth": 2, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 2.0}, {"uid": "ad02a284f829c979", "family": "fsm", "template": 1, "depth": 2, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 2.0}, {"uid": "d66d39facf8c09dc", "family": "arith", "template": 2, "depth": 2, "pred": "24", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "961f7a505cc9733d", "family": "arith", "template": 2, "depth": 2, "pred": "-108", "truth": "-120", "correct": false, "edits_per_token": 2.0}, {"uid": "2a66eac56efc842a", "family": "arith", "template": 2, "depth": 2, "pred": "230", "truth": "195", "correct": false, "edits_per_token": 2.0}, {"uid": "0b6aca1ee4bcc5e4", "family": "arith", "template": 2, "depth": 2, "pred": "-56", "truth": "-54", "correct": false, "edits_per_token": 2.0}, {"uid": "a7520c88e128325d", "family": "arith", "template": 2, "depth": 2, "pred": "7", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "a4a050b886f6d6f1", "family": "arith", "template": 2, "depth": 2, "pred": "64", "truth": "68", "correct": false, "edits_per_token": 2.0}, {"uid": "822a99424bbcf8ea", "family": "arith", "template": 2, "depth": 2, "pred": "40", "truth": "42", "correct": false, "edits_per_token": 2.0}, {"uid": "8fbd3d2ce9fd04d2", "family": "bind", "template": 0, "depth": 3, "pred": "57", "truth": "60", "correct": false, "edits_per_token": 2.0}, {"uid": "2ef3a2f2a0328157", "family": "bind", "template": 0, "depth": 3, "pred": "36", "truth": "32", "correct": false, "edits_per_token": 2.0}, {"uid": "588fe9ad929612f6", "family": "bind", "template": 0, "depth": 3, "pred": "10", "truth": "11", "correct": false, "edits_per_token": 2.0}, {"uid": "3db5b222471f96fa", "family": "bind", "template": 0, "depth": 3, "pred": "20", "truth": "20", "correct": true, "edits_per_token": 2.0}, {"uid": "b96935b62f495f58", "family": "arith", "template": 2, "depth": 3, "pred": "119", "truth": "149", "correct": false, "edits_per_token": 2.0}, {"uid": "4ce4e40dc4a8caf4", "family": "bind", "template": 0, "depth": 3, "pred": "80", "truth": "64", "correct": false, "edits_per_token": 2.0}, {"uid": "0acc51832dea68c1", "family": "bind", "template": 0, "depth": 3, "pred": "21", "truth": "23", "correct": false, "edits_per_token": 2.0}, {"uid": "6f0f31272e9dccb8", "family": "fsm", "template": 1, "depth": 2, "pred": "20", "truth": "22", "correct": false, "edits_per_token": 2.0}, {"uid": "43de2e7618ca4cb3", "family": "fsm", "template": 2, "depth": 3, "pred": "20", "truth": "18", "correct": false, "edits_per_token": 2.0}, {"uid": "ffb7dab9fc31fbd5", "family": "arith", "template": 2, "depth": 3, "pred": "44", "truth": "42", "correct": false, "edits_per_token": 2.0}, {"uid": "25ab3f6cf0620687", "family": "arith", "template": 2, "depth": 3, "pred": "40", "truth": "38", "correct": false, "edits_per_token": 2.0}, {"uid": "6e8c3b9ad7c915f8", "family": "arith", "template": 2, "depth": 3, "pred": "-13", "truth": "-13", "correct": true, "edits_per_token": 2.0}, {"uid": "595ac6801eedc4f3", "family": "bind", "template": 0, "depth": 3, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 2.0}, {"uid": "76fdb2e6417c6f93", "family": "bind", "template": 1, "depth": 3, "pred": "12", "truth": "11", "correct": false, "edits_per_token": 2.0}, {"uid": "d8e6469a252c53ab", "family": "fsm", "template": 2, "depth": 3, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "aa252e805cf95b6a", "family": "arith", "template": 2, "depth": 3, "pred": "17", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "e2bce580fe2875d8", "family": "arith", "template": 2, "depth": 3, "pred": "62", "truth": "60", "correct": false, "edits_per_token": 2.0}, {"uid": "a3811535637f42cf", "family": "arith", "template": 2, "depth": 3, "pred": "-14", "truth": "-13", "correct": false, "edits_per_token": 2.0}, {"uid": "920633664a3100c7", "family": "arith", "template": 2, "depth": 3, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 2.0}, {"uid": "6d1db4bf8883e127", "family": "bind", "template": 1, "depth": 3, "pred": "22", "truth": "22", "correct": true, "edits_per_token": 2.0}, {"uid": "42b7111968d0fd6e", "family": "bind", "template": 1, "depth": 3, "pred": "77", "truth": "74", "correct": false, "edits_per_token": 2.0}, {"uid": "0d7d21b45a123ecc", "family": "bind", "template": 1, "depth": 3, "pred": "27", "truth": "28", "correct": false, "edits_per_token": 2.0}, {"uid": "0070225255190df8", "family": "bind", "template": 1, "depth": 3, "pred": "-5", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "9efc725b639fa1e2", "family": "bind", "template": 1, "depth": 3, "pred": "41", "truth": "42", "correct": false, "edits_per_token": 2.0}, {"uid": "e6d2938cf240e7d7", "family": "bind", "template": 1, "depth": 3, "pred": "-1", "truth": "-2", "correct": false, "edits_per_token": 2.0}, {"uid": "6333807fc55e47c2", "family": "bind", "template": 2, "depth": 4, "pred": "27", "truth": "27", "correct": true, "edits_per_token": 2.0}, {"uid": "73bab13d0adb93c9", "family": "bind", "template": 2, "depth": 4, "pred": "33", "truth": "31", "correct": false, "edits_per_token": 2.0}, {"uid": "7e4a5eedc90f317e", "family": "arith", "template": 2, "depth": 4, "pred": "577", "truth": "707", "correct": false, "edits_per_token": 2.0}, {"uid": "72ab6c11b6b99a0c", "family": "bind", "template": 2, "depth": 4, "pred": "8", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "82d8e4541e035995", "family": "bind", "template": 2, "depth": 4, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 2.0}, {"uid": "11248162bd234998", "family": "bind", "template": 2, "depth": 4, "pred": "61", "truth": "61", "correct": true, "edits_per_token": 2.0}, {"uid": "30512b180423dd81", "family": "fsm", "template": 0, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 2.0}, {"uid": "5fff7cb1816a8b83", "family": "arith", "template": 2, "depth": 4, "pred": "-120", "truth": "-90", "correct": false, "edits_per_token": 2.0}, {"uid": "916e8e5dc57e71b5", "family": "arith", "template": 2, "depth": 4, "pred": "37", "truth": "33", "correct": false, "edits_per_token": 2.0}, {"uid": "86a2ceda3911a38b", "family": "arith", "template": 2, "depth": 4, "pred": "141", "truth": "165", "correct": false, "edits_per_token": 2.0}, {"uid": "407e416b5a6578ae", "family": "arith", "template": 2, "depth": 4, "pred": "216", "truth": "222", "correct": false, "edits_per_token": 2.0}, {"uid": "f3df5ab6d440dc66", "family": "bind", "template": 2, "depth": 4, "pred": "38", "truth": "40", "correct": false, "edits_per_token": 2.0}, {"uid": "d8aa4893aec3878d", "family": "bind", "template": 2, "depth": 4, "pred": "284", "truth": "316", "correct": false, "edits_per_token": 2.0}, {"uid": "effc949016a8ea12", "family": "fsm", "template": 0, "depth": 2, "pred": "11", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "efbc892e6b4a8d10", "family": "fsm", "template": 1, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "9ed8a702224d9157", "family": "fsm", "template": 1, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "447dc7313518c9cc", "family": "arith", "template": 2, "depth": 4, "pred": "-40", "truth": "-41", "correct": false, "edits_per_token": 2.0}, {"uid": "c9f30e7c90083e3b", "family": "arith", "template": 2, "depth": 4, "pred": "58", "truth": "70", "correct": false, "edits_per_token": 2.0}, {"uid": "414ba9ab6e9d47f0", "family": "fsm", "template": 2, "depth": 3, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 2.0}, {"uid": "d6bbe76ab7ae5dea", "family": "bind", "template": 0, "depth": 4, "pred": "5", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "ad80d63be705294d", "family": "fsm", "template": 0, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 2.0}, {"uid": "69d134b029d15133", "family": "bind", "template": 0, "depth": 4, "pred": "-10", "truth": "-8", "correct": false, "edits_per_token": 2.0}, {"uid": "1ed0d6ff18c8180d", "family": "bind", "template": 0, "depth": 4, "pred": "11", "truth": "11", "correct": true, "edits_per_token": 2.0}, {"uid": "9679ca491537d4ae", "family": "fsm", "template": 0, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 2.0}, {"uid": "06e9560336c69fc0", "family": "bind", "template": 0, "depth": 4, "pred": "100", "truth": "99", "correct": false, "edits_per_token": 2.0}, {"uid": "83785d82a70218c2", "family": "bind", "template": 0, "depth": 4, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "365ba7f8ce47ab26", "family": "bind", "template": 0, "depth": 4, "pred": "54", "truth": "51", "correct": false, "edits_per_token": 2.0}, {"uid": "f0b461762a54a509", "family": "bind", "template": 1, "depth": 4, "pred": "680", "truth": "368", "correct": false, "edits_per_token": 2.0}, {"uid": "ef8628813a4db970", "family": "bind", "template": 0, "depth": 4, "pred": "60", "truth": "64", "correct": false, "edits_per_token": 2.0}, {"uid": "a07ed018d16711ef", "family": "bind", "template": 1, "depth": 4, "pred": "101", "truth": "81", "correct": false, "edits_per_token": 2.0}, {"uid": "8566721d2752b3a8", "family": "bind", "template": 1, "depth": 4, "pred": "-12", "truth": "-8", "correct": false, "edits_per_token": 2.0}, {"uid": "14e0f0613ab5fafb", "family": "bind", "template": 1, "depth": 4, "pred": "43", "truth": "41", "correct": false, "edits_per_token": 2.0}, {"uid": "e5b10de969f4f1de", "family": "bind", "template": 1, "depth": 4, "pred": "-20", "truth": "-22", "correct": false, "edits_per_token": 2.0}, {"uid": "9a6304b25504acd5", "family": "fsm", "template": 2, "depth": 4, "pred": "13", "truth": "12", "correct": false, "edits_per_token": 2.0}, {"uid": "f9e4c5acf05b6575", "family": "bind", "template": 1, "depth": 4, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 2.0}, {"uid": "2efa0c713c22db5a", "family": "bind", "template": 1, "depth": 4, "pred": "102", "truth": "101", "correct": false, "edits_per_token": 2.0}, {"uid": "8ce65aa06b428682", "family": "fsm", "template": 0, "depth": 3, "pred": "22", "truth": "24", "correct": false, "edits_per_token": 2.0}, {"uid": "4f7a4c9e9fd88e23", "family": "fsm", "template": 1, "depth": 3, "pred": "-4", "truth": "-4", "correct": true, "edits_per_token": 2.0}, {"uid": "2468d102e506d6d5", "family": "fsm", "template": 2, "depth": 3, "pred": "12", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "6f3c8de73142af82", "family": "fsm", "template": 2, "depth": 4, "pred": "35", "truth": "37", "correct": false, "edits_per_token": 2.0}, {"uid": "ba3af2217536a7c6", "family": "fsm", "template": 0, "depth": 2, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 2.0}, {"uid": "e063912681967ddf", "family": "fsm", "template": 0, "depth": 3, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 2.0}, {"uid": "583473e83be2cb0e", "family": "fsm", "template": 2, "depth": 3, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 2.0}, {"uid": "5a873e995116c10d", "family": "fsm", "template": 1, "depth": 3, "pred": "80", "truth": "72", "correct": false, "edits_per_token": 2.0}, {"uid": "05764ff46e4d28b8", "family": "fsm", "template": 2, "depth": 4, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 2.0}, {"uid": "46b570766ec22e2a", "family": "fsm", "template": 0, "depth": 3, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "59f43bee78d1b7d5", "family": "fsm", "template": 0, "depth": 3, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 2.0}, {"uid": "eaf14d7290a1413b", "family": "fsm", "template": 1, "depth": 3, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 2.0}, {"uid": "b7248f02464a9d5e", "family": "fsm", "template": 1, "depth": 3, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 2.0}, {"uid": "9aaa1b5b75c855bb", "family": "fsm", "template": 2, "depth": 3, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 2.0}, {"uid": "a7203ce2418d2ce9", "family": "fsm", "template": 0, "depth": 4, "pred": "16", "truth": "18", "correct": false, "edits_per_token": 2.0}, {"uid": "b86f2cf77ff75b95", "family": "fsm", "template": 2, "depth": 4, "pred": "9", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "5480488e0f55e2a2", "family": "fsm", "template": 0, "depth": 3, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 2.0}, {"uid": "ca83fc2e201063f6", "family": "fsm", "template": 0, "depth": 3, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 2.0}, {"uid": "324c8cdae8f3169c", "family": "fsm", "template": 1, "depth": 4, "pred": "-4", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "2723eee2e2d0f964", "family": "fsm", "template": 1, "depth": 3, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "8749497eaab6176d", "family": "fsm", "template": 1, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "f006a9e07042e43f", "family": "fsm", "template": 2, "depth": 4, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 2.0}, {"uid": "def842732bc0714f", "family": "fsm", "template": 2, "depth": 4, "pred": "-4", "truth": "-4", "correct": true, "edits_per_token": 2.0}, {"uid": "e4ca09128be213ea", "family": "fsm", "template": 0, "depth": 3, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 2.0}, {"uid": "a1c297725f5ae175", "family": "fsm", "template": 1, "depth": 4, "pred": "0", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "7d9f104b41e0b7a5", "family": "fsm", "template": 0, "depth": 4, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 2.0}, {"uid": "536ead7bf94947ee", "family": "fsm", "template": 1, "depth": 4, "pred": "18", "truth": "18", "correct": true, "edits_per_token": 2.0}, {"uid": "9808a14025bb62db", "family": "fsm", "template": 1, "depth": 4, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 2.0}, {"uid": "666a3ad814af7485", "family": "fsm", "template": 0, "depth": 4, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 2.0}, {"uid": "29e5f1d39e9dd718", "family": "excl", "template": 2, "depth": 1, "pred": "apple plum", "truth": "date", "correct": false, "edits_per_token": 2.0}, {"uid": "4425b91f251fc921", "family": "excl", "template": 2, "depth": 1, "pred": "plum pear", "truth": "lime", "correct": false, "edits_per_token": 2.0}, {"uid": "899865b46b45638d", "family": "sched", "template": 2, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "bb4df57d9a04ccab", "family": "fsm", "template": 1, "depth": 3, "pred": "-5", "truth": "-5", "correct": true, "edits_per_token": 2.0}, {"uid": "5aee313fef7904e6", "family": "fsm", "template": 2, "depth": 4, "pred": "0", "truth": "-1", "correct": false, "edits_per_token": 2.0}, {"uid": "b6a823db9fe091cf", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "eae15d4721c93c28", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "ebdd7e877a44588b", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "a467884db4d82ee1", "family": "fsm", "template": 1, "depth": 4, "pred": "10", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "a0e13fa29eaee344", "family": "fsm", "template": 0, "depth": 4, "pred": "18", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "55bddbb405630c49", "family": "excl", "template": 2, "depth": 1, "pred": "plum lime a", "truth": "kiwi plum apple", "correct": false, "edits_per_token": 2.0}, {"uid": "9e10c6f2c824ea03", "family": "fsm", "template": 1, "depth": 4, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "e29749c80774340d", "family": "fsm", "template": 1, "depth": 4, "pred": "-7", "truth": "-8", "correct": false, "edits_per_token": 2.0}, {"uid": "305ec8d13dc601f2", "family": "excl", "template": 2, "depth": 1, "pred": "apple fig b", "truth": "kiwi fig yam bean", "correct": false, "edits_per_token": 2.0}, {"uid": "c56841284720069d", "family": "fsm", "template": 0, "depth": 4, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 2.0}, {"uid": "b757b7fb6b160032", "family": "fsm", "template": 0, "depth": 4, "pred": "-8", "truth": "-7", "correct": false, "edits_per_token": 2.0}, {"uid": "8a849a68ac0bd89d", "family": "sched", "template": 1, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "24c9898cc5168e60", "family": "sched", "template": 1, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "b45de26fb239a53c", "family": "fsm", "template": 0, "depth": 4, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 2.0}, {"uid": "b92a8bb6a6b6c2fc", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "24f6f2d470ed5ab8", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "265c9cd1ac273c94", "family": "sched", "template": 2, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "17e81034d97a73e1", "family": "excl", "template": 1, "depth": 2, "pred": "pear date y", "truth": "yam", "correct": false, "edits_per_token": 2.0}, {"uid": "52b9e62b2a6ca3c2", "family": "excl", "template": 2, "depth": 3, "pred": "fig", "truth": "yam", "correct": false, "edits_per_token": 2.0}, {"uid": "3ea463a1eb42a642", "family": "excl", "template": 2, "depth": 1, "pred": "bean fig pl", "truth": "bean plum date nut", "correct": false, "edits_per_token": 2.0}, {"uid": "290428df35319b49", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "e8f3f5315f909938", "family": "excl", "template": 2, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "679e75695e94a8a8", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "b9763ee8284b60af", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "50436cd63f7d344d", "family": "excl", "template": 1, "depth": 1, "pred": "bean apple", "truth": "bean date apple", "correct": false, "edits_per_token": 2.0}, {"uid": "f91565e19106bd0c", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "f377b423248cc145", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "fd984d3e4a7605fd", "family": "excl", "template": 0, "depth": 2, "pred": "pear kiwi", "truth": "nut", "correct": false, "edits_per_token": 2.0}, {"uid": "78ec3d641e6859a8", "family": "excl", "template": 2, "depth": 2, "pred": "kiwi pear", "truth": "date apple", "correct": false, "edits_per_token": 2.0}, {"uid": "ec07ebafd22ecb2e", "family": "excl", "template": 1, "depth": 2, "pred": "plum pear", "truth": "pear", "correct": false, "edits_per_token": 2.0}, {"uid": "637c344de20f5625", "family": "excl", "template": 2, "depth": 2, "pred": "kiwi nut", "truth": "apple date fig bean", "correct": false, "edits_per_token": 2.0}, {"uid": "b1127806bf880411", "family": "sched", "template": 0, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "a65132179f29f33e", "family": "excl", "template": 0, "depth": 1, "pred": "apple lime", "truth": "apple lime", "correct": true, "edits_per_token": 2.0}, {"uid": "610ab75a850b6358", "family": "sched", "template": 0, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "09a0f4119071f489", "family": "excl", "template": 2, "depth": 1, "pred": "pear apple", "truth": "pear kiwi fig nut", "correct": false, "edits_per_token": 2.0}, {"uid": "9c0d7b46b835ff7f", "family": "excl", "template": 2, "depth": 1, "pred": "plum fig pe", "truth": "plum kiwi date nut", "correct": false, "edits_per_token": 2.0}, {"uid": "e9948bd8b29e69d3", "family": "excl", "template": 2, "depth": 2, "pred": "lime nut ki", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "f948792de5a76409", "family": "excl", "template": 0, "depth": 1, "pred": "kiwi fig li", "truth": "apple kiwi lime fig", "correct": false, "edits_per_token": 2.0}, {"uid": "43382d613173a703", "family": "sched", "template": 1, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "4378e430fe1485f9", "family": "sched", "template": 1, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "7af9657dd3fba74a", "family": "excl", "template": 2, "depth": 2, "pred": "plum", "truth": "date plum", "correct": false, "edits_per_token": 2.0}, {"uid": "c9dda90c7d0ad8ed", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "2f7d7388cf61d3fe", "family": "sched", "template": 1, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "114ddae29f526626", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "d95b1cb8c2923e7e", "family": "excl", "template": 0, "depth": 1, "pred": "bean lime a", "truth": "bean lime apple", "correct": false, "edits_per_token": 2.0}, {"uid": "1dcd25a6eb16038d", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "59114c794ee3a087", "family": "excl", "template": 0, "depth": 2, "pred": "bean plum p", "truth": "bean", "correct": false, "edits_per_token": 2.0}, {"uid": "a9742f748ad1ad5c", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "ec7190a280bfbdb7", "family": "excl", "template": 2, "depth": 4, "pred": "apple", "truth": "lime", "correct": false, "edits_per_token": 2.0}, {"uid": "81a4bc002847b518", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "fce4affe8df7979a", "family": "excl", "template": 1, "depth": 1, "pred": "nut date li", "truth": "date apple lime", "correct": false, "edits_per_token": 2.0}, {"uid": "50c50f83a59dabd8", "family": "excl", "template": 2, "depth": 2, "pred": "yam plum pe", "truth": "fig bean kiwi plum apple", "correct": false, "edits_per_token": 2.0}, {"uid": "9a0b5958ebfcc667", "family": "sched", "template": 1, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "2b9fc22b15089625", "family": "excl", "template": 0, "depth": 1, "pred": "pear apple", "truth": "pear kiwi", "correct": false, "edits_per_token": 2.0}, {"uid": "2d7af060d17ced33", "family": "sched", "template": 1, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "22973924b5d55519", "family": "sched", "template": 1, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "9eba62cb7a72f599", "family": "sched", "template": 2, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "60adda9ef6e12d3c", "family": "excl", "template": 0, "depth": 1, "pred": "kiwi pear a", "truth": "kiwi bean apple", "correct": false, "edits_per_token": 2.0}, {"uid": "222e860f500afa48", "family": "excl", "template": 1, "depth": 1, "pred": "pear fig nu", "truth": "pear nut fig lime", "correct": false, "edits_per_token": 2.0}, {"uid": "883e00248131789c", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "b7eae28e49657a0d", "family": "sched", "template": 2, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "36bc16e15622ea97", "family": "sched", "template": 2, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 2.0}, {"uid": "9b1170c69bedb5d0", "family": "excl", "template": 1, "depth": 3, "pred": "nut date", "truth": "plum date", "correct": false, "edits_per_token": 2.0}, {"uid": "e7a942f9776e1481", "family": "excl", "template": 1, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "03e8f11a87973087", "family": "sched", "template": 0, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "0641e2dbfe875c84", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "b60c5ce59c8ad584", "family": "sched", "template": 2, "depth": 3, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "a07385bb7e4ea97b", "family": "sched", "template": 0, "depth": 2, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "a942d085ec964894", "family": "sched", "template": 2, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "800f61c9e4ad0bf4", "family": "excl", "template": 1, "depth": 2, "pred": "fig lime", "truth": "nut bean", "correct": false, "edits_per_token": 2.0}, {"uid": "19887d3ca18adb40", "family": "sched", "template": 0, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 2.0}, {"uid": "01dd5269d9e0a510", "family": "sched", "template": 0, "depth": 3, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "d1578ac0adce4a3f", "family": "sched", "template": 2, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "4f7ac4cb4667edb6", "family": "excl", "template": 0, "depth": 2, "pred": "lime apple", "truth": "bean date", "correct": false, "edits_per_token": 2.0}, {"uid": "9ebb20923f7ff360", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "45e5e22be65e2392", "family": "sched", "template": 2, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "58590db8d34f4bb6", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "e0aa686b67447872", "family": "excl", "template": 1, "depth": 1, "pred": "none", "truth": "lime fig", "correct": false, "edits_per_token": 2.0}, {"uid": "b9758ae4de312e38", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "apple", "correct": false, "edits_per_token": 2.0}, {"uid": "49b538b6cebfad32", "family": "sched", "template": 0, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "b41c090edc1c2570", "family": "sched", "template": 0, "depth": 2, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "57f25f911fd47979", "family": "excl", "template": 0, "depth": 2, "pred": "fig lime", "truth": "bean", "correct": false, "edits_per_token": 2.0}, {"uid": "52a3076c67262025", "family": "excl", "template": 0, "depth": 2, "pred": "bean fig", "truth": "bean fig date", "correct": false, "edits_per_token": 2.0}, {"uid": "5313d51deae8c2b8", "family": "excl", "template": 0, "depth": 3, "pred": "kiwi", "truth": "yam", "correct": false, "edits_per_token": 2.0}, {"uid": "fbb63a5c5980a546", "family": "excl", "template": 1, "depth": 1, "pred": "plum nut", "truth": "date apple", "correct": false, "edits_per_token": 2.0}, {"uid": "123ddaa3e5efb215", "family": "excl", "template": 1, "depth": 1, "pred": "fig bean pe", "truth": "fig pear nut kiwi", "correct": false, "edits_per_token": 2.0}, {"uid": "bf6d2e6f97942225", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "3a419d90cdd84493", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "yam plum nut", "correct": false, "edits_per_token": 2.0}, {"uid": "dc7a2b700ef3eec3", "family": "sched", "template": 0, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "a05184e028d82ee0", "family": "sched", "template": 0, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "b9e25b6eb179c90b", "family": "excl", "template": 0, "depth": 3, "pred": "kiwi bean p", "truth": "kiwi nut fig pear", "correct": false, "edits_per_token": 2.0}, {"uid": "914dc5465d528e9b", "family": "sched", "template": 0, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "64f71380d66efae2", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "33fe241315cf5b9f", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "1b7f8f05a6c5a7a3", "family": "excl", "template": 2, "depth": 2, "pred": "plum apple", "truth": "plum apple", "correct": true, "edits_per_token": 2.0}, {"uid": "5c13ea4ff7d521b5", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "2a2cf42fd19abbe2", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "df171a88e16d0896", "family": "excl", "template": 0, "depth": 2, "pred": "yam", "truth": "apple yam", "correct": false, "edits_per_token": 2.0}, {"uid": "a6b723b79a990fea", "family": "excl", "template": 1, "depth": 1, "pred": "nut lime zi", "truth": "nut kiwi lime fig", "correct": false, "edits_per_token": 2.0}, {"uid": "95569dc6b1f0424e", "family": "excl", "template": 2, "depth": 4, "pred": "fig", "truth": "nut fig", "correct": false, "edits_per_token": 2.0}, {"uid": "d5cb804202c06b04", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 2.0}, {"uid": "26a8bdf6da59e053", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "8461de28662a09d0", "family": "excl", "template": 0, "depth": 1, "pred": "yam bean", "truth": "lime kiwi apple", "correct": false, "edits_per_token": 2.0}, {"uid": "5266b24716f21c1b", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "1301b2b886fc0fed", "family": "sched", "template": 1, "depth": 3, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "0ae3e633b9c73a6d", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 2.0}, {"uid": "a34d5263586bf792", "family": "sched", "template": 1, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "72b16296ef29803c", "family": "sched", "template": 2, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "1650d27be4762458", "family": "sched", "template": 2, "depth": 4, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "d17c2abfac01fb85", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "f7bb7ae8302c82e3", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "ce88ea5df4ac7c7e", "family": "sched", "template": 1, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "89232c53de86f0c7", "family": "sched", "template": 2, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "7867942486d7f380", "family": "sched", "template": 0, "depth": 4, "pred": "none", "truth": "D", "correct": false, "edits_per_token": 2.0}, {"uid": "c9726e189052aad1", "family": "sched", "template": 2, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 2.0}, {"uid": "302f485c304f8e1c", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "58e2a723355b2957", "family": "sched", "template": 0, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "376a16c13fc8af71", "family": "sched", "template": 0, "depth": 4, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 2.0}, {"uid": "7eddd2c6ff72770f", "family": "sched", "template": 0, "depth": 4, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "ebcfc88ef93a66df", "family": "sched", "template": 2, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "89a256c76069f616", "family": "excl", "template": 1, "depth": 3, "pred": "plum", "truth": "plum lime", "correct": false, "edits_per_token": 2.0}, {"uid": "34292f4e57b84e60", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "b4ef07715474f790", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "084073be6e34810e", "family": "excl", "template": 2, "depth": 2, "pred": "lime", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "fec42ca9bcf0a20b", "family": "excl", "template": 1, "depth": 2, "pred": "date bean", "truth": "date plum bean", "correct": false, "edits_per_token": 2.0}, {"uid": "9600b1b302d34721", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "e68c491cb5fe0db5", "family": "excl", "template": 0, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "8fbf1093b19d178d", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 2.0}, {"uid": "152fa59b6f060034", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "6886f89d5fecebc2", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "25810ef41631a5a0", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "date", "correct": false, "edits_per_token": 2.0}, {"uid": "c2c1af96873ca940", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "493eb05c569e6806", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "yam fig date", "correct": false, "edits_per_token": 2.0}, {"uid": "3ca58bdf2e93bec8", "family": "excl", "template": 1, "depth": 2, "pred": "apple yam b", "truth": "nut", "correct": false, "edits_per_token": 2.0}, {"uid": "68cafc987827b672", "family": "excl", "template": 0, "depth": 1, "pred": "apple yam f", "truth": "apple lime nut kiwi bean", "correct": false, "edits_per_token": 2.0}, {"uid": "e5f870c9ac9c2bdc", "family": "excl", "template": 1, "depth": 2, "pred": "apple bean", "truth": "apple pear plum", "correct": false, "edits_per_token": 2.0}, {"uid": "4aba72aed6f820bb", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "bean", "correct": false, "edits_per_token": 2.0}, {"uid": "7e3d734f2577f012", "family": "excl", "template": 2, "depth": 3, "pred": "pear apple", "truth": "plum bean", "correct": false, "edits_per_token": 2.0}, {"uid": "da06a708de7351a1", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "e13f444f5e921059", "family": "sched", "template": 1, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 2.0}, {"uid": "806dfca57003d938", "family": "sched", "template": 1, "depth": 4, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "5a3700bf53c0daf3", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 2.0}, {"uid": "fa8b059c364916e0", "family": "sched", "template": 1, "depth": 4, "pred": "D", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "9fd10a99f31cee6e", "family": "sched", "template": 1, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "b3fe0fa2ac161f59", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "2489ee7572296937", "family": "sched", "template": 2, "depth": 4, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "80d519c064361dae", "family": "sched", "template": 2, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "5a66b965f98dc2cf", "family": "excl", "template": 0, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "4cd96fbf4f41a908", "family": "sched", "template": 0, "depth": 4, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 2.0}, {"uid": "7a4191b42dce2b06", "family": "sched", "template": 1, "depth": 3, "pred": "none", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "b87404e94d9a09f6", "family": "excl", "template": 0, "depth": 2, "pred": "plum apple", "truth": "fig kiwi pear", "correct": false, "edits_per_token": 2.0}, {"uid": "2a9ecd3661b30257", "family": "sched", "template": 0, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 2.0}, {"uid": "5249781ebc348d37", "family": "sched", "template": 2, "depth": 4, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 2.0}, {"uid": "2e4acd7d13860ac2", "family": "sched", "template": 2, "depth": 4, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "2adc39419c8b49a4", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "fig lime", "correct": false, "edits_per_token": 2.0}, {"uid": "dca6675d28527584", "family": "excl", "template": 0, "depth": 3, "pred": "fig apple", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "c666c9854336f784", "family": "sched", "template": 0, "depth": 4, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "8f480bee1ead1646", "family": "excl", "template": 0, "depth": 3, "pred": "nut yam", "truth": "apple", "correct": false, "edits_per_token": 2.0}, {"uid": "cf67501c8d912952", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "d61260228e19e8ba", "family": "sched", "template": 2, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "71fb11da7ef26fa8", "family": "sched", "template": 0, "depth": 4, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "f12148b1efc8abed", "family": "excl", "template": 2, "depth": 4, "pred": "date", "truth": "nut", "correct": false, "edits_per_token": 2.0}, {"uid": "9e76644064b0dc70", "family": "excl", "template": 0, "depth": 3, "pred": "date yam", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "12b5acb0d562716c", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "fd29037f3e23207c", "family": "sched", "template": 1, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 2.0}, {"uid": "dbd4f3b44d2455e6", "family": "sched", "template": 1, "depth": 4, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "2bbd29545a49585f", "family": "sched", "template": 1, "depth": 4, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "e7a65045762e87f2", "family": "sched", "template": 1, "depth": 4, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "9476a68c1c10bc9b", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "dfb1aad2e59286cb", "family": "excl", "template": 0, "depth": 4, "pred": "pear", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "fe2915e5def8bf83", "family": "excl", "template": 1, "depth": 4, "pred": "fig apple n", "truth": "yam", "correct": false, "edits_per_token": 2.0}, {"uid": "548b800800b9067a", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "19a75d08ddd9d502", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 2.0}, {"uid": "6c5b7174f7312602", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "11c0e819fd3fe52e", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}] | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_iid_b8.json
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +[{"uid": "d57de33b3b3613ce", "family": "fsm", "template": 2, "depth": 1, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 8.0}, {"uid": "2fcbef2f6deb569d", "family": "bind", "template": 2, "depth": 1, "pred": "17", "truth": "17", "correct": true, "edits_per_token": 8.0}, {"uid": "facfe4b45751726b", "family": "bind", "template": 2, "depth": 1, "pred": "14", "truth": "15", "correct": false, "edits_per_token": 8.0}, {"uid": "17c2f737ab6e9a12", "family": "bind", "template": 2, "depth": 1, "pred": "21", "truth": "21", "correct": true, "edits_per_token": 8.0}, {"uid": "330a5096d00ea26d", "family": "bind", "template": 2, "depth": 1, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "67c60958afd432d0", "family": "bind", "template": 2, "depth": 1, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "0df874a819030335", "family": "bind", "template": 2, "depth": 1, "pred": "52", "truth": "60", "correct": false, "edits_per_token": 8.0}, {"uid": "f8b6c686efa42185", "family": "bind", "template": 2, "depth": 1, "pred": "27", "truth": "26", "correct": false, "edits_per_token": 8.0}, {"uid": "61ab4c048732d1a3", "family": "fsm", "template": 2, "depth": 1, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 8.0}, {"uid": "a0a9effb7a5a1771", "family": "fsm", "template": 2, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 8.0}, {"uid": "e539dc5c812c9f8f", "family": "arith", "template": 0, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 8.0}, {"uid": "b9080d7276f3e48d", "family": "arith", "template": 1, "depth": 1, "pred": "14", "truth": "14", "correct": true, "edits_per_token": 8.0}, {"uid": "f19b2d158a1b2321", "family": "arith", "template": 0, "depth": 1, "pred": "-2", "truth": "-2", "correct": true, "edits_per_token": 8.0}, {"uid": "b50334287c335a68", "family": "arith", "template": 0, "depth": 1, "pred": "13", "truth": "13", "correct": true, "edits_per_token": 8.0}, {"uid": "5203cff28a90efc8", "family": "arith", "template": 0, "depth": 1, "pred": "14", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "b0fc8aed8de80f82", "family": "arith", "template": 1, "depth": 1, "pred": "85", "truth": "85", "correct": true, "edits_per_token": 8.0}, {"uid": "d3f8213a4e9e4170", "family": "arith", "template": 1, "depth": 1, "pred": "64", "truth": "66", "correct": false, "edits_per_token": 8.0}, {"uid": "6dbb743f0c127dc2", "family": "arith", "template": 1, "depth": 1, "pred": "100", "truth": "120", "correct": false, "edits_per_token": 8.0}, {"uid": "94bfbb133f315b17", "family": "arith", "template": 0, "depth": 1, "pred": "37", "truth": "37", "correct": true, "edits_per_token": 8.0}, {"uid": "c5dd86b6955d4021", "family": "arith", "template": 1, "depth": 1, "pred": "-7", "truth": "-7", "correct": true, "edits_per_token": 8.0}, {"uid": "75eceacb14c3d0e6", "family": "bind", "template": 0, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 8.0}, {"uid": "f3b82055a8272259", "family": "bind", "template": 0, "depth": 1, "pred": "18", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "6fa8f401c1e6cf13", "family": "bind", "template": 0, "depth": 1, "pred": "23", "truth": "22", "correct": false, "edits_per_token": 8.0}, {"uid": "5252b66c833e4804", "family": "bind", "template": 0, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 8.0}, {"uid": "0dd2fab9c5861d6d", "family": "bind", "template": 0, "depth": 1, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 8.0}, {"uid": "d8aa01c2e9dbb03f", "family": "bind", "template": 1, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 8.0}, {"uid": "1d20f4c03960f839", "family": "arith", "template": 0, "depth": 2, "pred": "9", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "a0d75f1874311fde", "family": "bind", "template": 0, "depth": 1, "pred": "26", "truth": "27", "correct": false, "edits_per_token": 8.0}, {"uid": "2f252bbc438ce228", "family": "bind", "template": 0, "depth": 1, "pred": "24", "truth": "24", "correct": true, "edits_per_token": 8.0}, {"uid": "910350946f41fbf1", "family": "bind", "template": 1, "depth": 1, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 8.0}, {"uid": "322ec1e8c6acc1ff", "family": "bind", "template": 1, "depth": 1, "pred": "24", "truth": "24", "correct": true, "edits_per_token": 8.0}, {"uid": "8f08654666473a62", "family": "bind", "template": 1, "depth": 1, "pred": "8", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "e74c59b882dcf2b2", "family": "bind", "template": 1, "depth": 1, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 8.0}, {"uid": "f4ea26539e1674ec", "family": "arith", "template": 0, "depth": 2, "pred": "28", "truth": "28", "correct": true, "edits_per_token": 8.0}, {"uid": "8461e207ba7b8587", "family": "arith", "template": 1, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 8.0}, {"uid": "d43d09af762bceee", "family": "arith", "template": 1, "depth": 2, "pred": "41", "truth": "44", "correct": false, "edits_per_token": 8.0}, {"uid": "7ed459fad79fdf0b", "family": "bind", "template": 1, "depth": 1, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 8.0}, {"uid": "ce3cbdbb1894d8f4", "family": "bind", "template": 1, "depth": 1, "pred": "14", "truth": "15", "correct": false, "edits_per_token": 8.0}, {"uid": "14947545562906ad", "family": "bind", "template": 1, "depth": 1, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 8.0}, {"uid": "697d570c40384635", "family": "bind", "template": 2, "depth": 2, "pred": "11", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "b78e61c19225f49c", "family": "fsm", "template": 2, "depth": 2, "pred": "18", "truth": "18", "correct": true, "edits_per_token": 8.0}, {"uid": "69953fd72fce090d", "family": "arith", "template": 0, "depth": 2, "pred": "36", "truth": "38", "correct": false, "edits_per_token": 8.0}, {"uid": "66cdadf971fee2ab", "family": "arith", "template": 0, "depth": 2, "pred": "69", "truth": "71", "correct": false, "edits_per_token": 8.0}, {"uid": "99c7a494c455e41d", "family": "arith", "template": 0, "depth": 2, "pred": "-62", "truth": "-60", "correct": false, "edits_per_token": 8.0}, {"uid": "8d21bd8bbb5eecad", "family": "arith", "template": 0, "depth": 2, "pred": "42", "truth": "41", "correct": false, "edits_per_token": 8.0}, {"uid": "5f3cf3ff50e3e9c1", "family": "arith", "template": 1, "depth": 2, "pred": "40", "truth": "41", "correct": false, "edits_per_token": 8.0}, {"uid": "ce6f8c794a4d97fb", "family": "arith", "template": 1, "depth": 2, "pred": "51", "truth": "55", "correct": false, "edits_per_token": 8.0}, {"uid": "56607dee807b8fd6", "family": "bind", "template": 2, "depth": 2, "pred": "17", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "206a446298d63b02", "family": "bind", "template": 2, "depth": 2, "pred": "40", "truth": "40", "correct": true, "edits_per_token": 8.0}, {"uid": "4dd1c85d972bb798", "family": "bind", "template": 2, "depth": 2, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 8.0}, {"uid": "60d954acf09287f1", "family": "arith", "template": 0, "depth": 2, "pred": "48", "truth": "46", "correct": false, "edits_per_token": 8.0}, {"uid": "5521b704b5ac25a7", "family": "arith", "template": 1, "depth": 2, "pred": "11", "truth": "12", "correct": false, "edits_per_token": 8.0}, {"uid": "b66b330f45989ec2", "family": "arith", "template": 1, "depth": 2, "pred": "12", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "f06a723961f75f4d", "family": "arith", "template": 1, "depth": 2, "pred": "-34", "truth": "-36", "correct": false, "edits_per_token": 8.0}, {"uid": "1ffed152f8b72d39", "family": "bind", "template": 2, "depth": 2, "pred": "4", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "3b4d5aeb1e813243", "family": "arith", "template": 0, "depth": 3, "pred": "108", "truth": "140", "correct": false, "edits_per_token": 8.0}, {"uid": "e56fea6bf4230916", "family": "bind", "template": 2, "depth": 2, "pred": "1", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "9f5c3b31e7d60900", "family": "bind", "template": 2, "depth": 2, "pred": "29", "truth": "31", "correct": false, "edits_per_token": 8.0}, {"uid": "ba60a76e7ce73c9a", "family": "arith", "template": 0, "depth": 3, "pred": "175", "truth": "165", "correct": false, "edits_per_token": 8.0}, {"uid": "caa2ffb073e3b554", "family": "arith", "template": 0, "depth": 3, "pred": "-80", "truth": "-80", "correct": true, "edits_per_token": 8.0}, {"uid": "f3df8ca3c7f3fe73", "family": "arith", "template": 0, "depth": 3, "pred": "-350", "truth": "-300", "correct": false, "edits_per_token": 8.0}, {"uid": "6a521ae83b9560cb", "family": "arith", "template": 0, "depth": 3, "pred": "309", "truth": "395", "correct": false, "edits_per_token": 8.0}, {"uid": "b7e7635fb68761fa", "family": "arith", "template": 1, "depth": 3, "pred": "80", "truth": "78", "correct": false, "edits_per_token": 8.0}, {"uid": "69168d7303f42428", "family": "arith", "template": 1, "depth": 3, "pred": "360", "truth": "352", "correct": false, "edits_per_token": 8.0}, {"uid": "3876cda47832bc78", "family": "arith", "template": 1, "depth": 3, "pred": "60", "truth": "66", "correct": false, "edits_per_token": 8.0}, {"uid": "101a933392cdc486", "family": "arith", "template": 0, "depth": 3, "pred": "-9", "truth": "-8", "correct": false, "edits_per_token": 8.0}, {"uid": "5d6734a6c29bf98b", "family": "arith", "template": 1, "depth": 3, "pred": "-38", "truth": "-38", "correct": true, "edits_per_token": 8.0}, {"uid": "e6b4bb7a789c1bbf", "family": "arith", "template": 1, "depth": 3, "pred": "-36", "truth": "-36", "correct": true, "edits_per_token": 8.0}, {"uid": "529ef2fbb6693bf4", "family": "arith", "template": 0, "depth": 3, "pred": "25", "truth": "27", "correct": false, "edits_per_token": 8.0}, {"uid": "4264ff894995b84a", "family": "arith", "template": 0, "depth": 4, "pred": "107", "truth": "69", "correct": false, "edits_per_token": 8.0}, {"uid": "4fd0a06acba9c2bd", "family": "arith", "template": 1, "depth": 3, "pred": "77", "truth": "75", "correct": false, "edits_per_token": 8.0}, {"uid": "11ddfaa4634944bc", "family": "arith", "template": 1, "depth": 3, "pred": "-11", "truth": "-11", "correct": true, "edits_per_token": 8.0}, {"uid": "81903019667d210a", "family": "arith", "template": 0, "depth": 4, "pred": "-27", "truth": "-24", "correct": false, "edits_per_token": 8.0}, {"uid": "f025613adc1381e6", "family": "arith", "template": 0, "depth": 4, "pred": "77", "truth": "67", "correct": false, "edits_per_token": 8.0}, {"uid": "6917d247eea87098", "family": "arith", "template": 1, "depth": 4, "pred": "1920", "truth": "1400", "correct": false, "edits_per_token": 8.0}, {"uid": "5ac4cc428fcee7ff", "family": "bind", "template": 0, "depth": 2, "pred": "120", "truth": "180", "correct": false, "edits_per_token": 8.0}, {"uid": "a7fa5ddf82e3acb6", "family": "bind", "template": 0, "depth": 2, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 8.0}, {"uid": "85a6f3db0aee68b6", "family": "fsm", "template": 0, "depth": 1, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 8.0}, {"uid": "306790bc18e1ff0f", "family": "fsm", "template": 1, "depth": 1, "pred": "-5", "truth": "-5", "correct": true, "edits_per_token": 8.0}, {"uid": "7b35ad2e31d3fe01", "family": "fsm", "template": 1, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 8.0}, {"uid": "f692ca3a30b9615b", "family": "arith", "template": 0, "depth": 4, "pred": "27", "truth": "27", "correct": true, "edits_per_token": 8.0}, {"uid": "00456af1ebb8a6f6", "family": "arith", "template": 1, "depth": 4, "pred": "-16", "truth": "-13", "correct": false, "edits_per_token": 8.0}, {"uid": "8636f76e7a05df5c", "family": "bind", "template": 0, "depth": 2, "pred": "-15", "truth": "-10", "correct": false, "edits_per_token": 8.0}, {"uid": "6894af54caeca8f0", "family": "bind", "template": 0, "depth": 2, "pred": "52", "truth": "56", "correct": false, "edits_per_token": 8.0}, {"uid": "15a26f31dc193ec8", "family": "bind", "template": 0, "depth": 2, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 8.0}, {"uid": "fbf1f2cb25ea61f5", "family": "bind", "template": 0, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 8.0}, {"uid": "e1c776eb2e759087", "family": "arith", "template": 0, "depth": 4, "pred": "77", "truth": "66", "correct": false, "edits_per_token": 8.0}, {"uid": "b57b7bb076a588da", "family": "arith", "template": 1, "depth": 4, "pred": "488", "truth": "480", "correct": false, "edits_per_token": 8.0}, {"uid": "c96adbcbd0c51bd0", "family": "arith", "template": 1, "depth": 4, "pred": "52", "truth": "52", "correct": true, "edits_per_token": 8.0}, {"uid": "058b50085974c009", "family": "arith", "template": 1, "depth": 4, "pred": "19", "truth": "19", "correct": true, "edits_per_token": 8.0}, {"uid": "cbc5614b2da64a6a", "family": "arith", "template": 1, "depth": 4, "pred": "-55", "truth": "-50", "correct": false, "edits_per_token": 8.0}, {"uid": "ffd3f5ed636cb166", "family": "bind", "template": 0, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 8.0}, {"uid": "e48b5252059e24a7", "family": "bind", "template": 1, "depth": 2, "pred": "-7", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "58cc6ff1fb3e1166", "family": "bind", "template": 1, "depth": 2, "pred": "20", "truth": "15", "correct": false, "edits_per_token": 8.0}, {"uid": "9dbdbf7552d0f726", "family": "bind", "template": 1, "depth": 2, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 8.0}, {"uid": "02f04b4ab6addf22", "family": "fsm", "template": 2, "depth": 2, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 8.0}, {"uid": "a7df7a7b48f0e07c", "family": "fsm", "template": 2, "depth": 2, "pred": "-3", "truth": "-3", "correct": true, "edits_per_token": 8.0}, {"uid": "4d05c02585d8db9c", "family": "fsm", "template": 2, "depth": 2, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 8.0}, {"uid": "54b414556cb4b641", "family": "fsm", "template": 2, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 8.0}, {"uid": "afe7716d466e8527", "family": "arith", "template": 0, "depth": 4, "pred": "-12", "truth": "-10", "correct": false, "edits_per_token": 8.0}, {"uid": "a3a41fe21c3b68c3", "family": "arith", "template": 0, "depth": 4, "pred": "-93", "truth": "-106", "correct": false, "edits_per_token": 8.0}, {"uid": "1e60abcad3f7cd49", "family": "arith", "template": 1, "depth": 4, "pred": "27", "truth": "29", "correct": false, "edits_per_token": 8.0}, {"uid": "5b477191fed1d1ab", "family": "bind", "template": 1, "depth": 2, "pred": "-8", "truth": "-8", "correct": true, "edits_per_token": 8.0}, {"uid": "b618508093791a17", "family": "bind", "template": 1, "depth": 2, "pred": "210", "truth": "240", "correct": false, "edits_per_token": 8.0}, {"uid": "758626244ff1e265", "family": "bind", "template": 1, "depth": 2, "pred": "45", "truth": "45", "correct": true, "edits_per_token": 8.0}, {"uid": "53f6d7daa1e917ac", "family": "bind", "template": 2, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "7a489d579355664c", "family": "arith", "template": 2, "depth": 1, "pred": "44", "truth": "48", "correct": false, "edits_per_token": 8.0}, {"uid": "866964b48f15dc73", "family": "arith", "template": 2, "depth": 1, "pred": "22", "truth": "21", "correct": false, "edits_per_token": 8.0}, {"uid": "326ed5326c7a15fa", "family": "arith", "template": 2, "depth": 1, "pred": "64", "truth": "60", "correct": false, "edits_per_token": 8.0}, {"uid": "78b68accb86904aa", "family": "bind", "template": 1, "depth": 2, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 8.0}, {"uid": "12ab692979396f5c", "family": "bind", "template": 2, "depth": 3, "pred": "37", "truth": "36", "correct": false, "edits_per_token": 8.0}, {"uid": "4b618bfde19bcd4c", "family": "bind", "template": 2, "depth": 3, "pred": "-8", "truth": "-7", "correct": false, "edits_per_token": 8.0}, {"uid": "99b184a684af7db3", "family": "bind", "template": 2, "depth": 3, "pred": "-12", "truth": "-15", "correct": false, "edits_per_token": 8.0}, {"uid": "09b2aab4939d318c", "family": "fsm", "template": 2, "depth": 3, "pred": "19", "truth": "19", "correct": true, "edits_per_token": 8.0}, {"uid": "b93e590222b68eba", "family": "arith", "template": 2, "depth": 1, "pred": "-17", "truth": "-18", "correct": false, "edits_per_token": 8.0}, {"uid": "e12b670d52d4bcac", "family": "arith", "template": 2, "depth": 1, "pred": "31", "truth": "34", "correct": false, "edits_per_token": 8.0}, {"uid": "4247b2f7e094279e", "family": "bind", "template": 2, "depth": 3, "pred": "19", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "67dfb1d3b0773f0e", "family": "bind", "template": 2, "depth": 3, "pred": "48", "truth": "43", "correct": false, "edits_per_token": 8.0}, {"uid": "8254a0f979194a20", "family": "bind", "template": 2, "depth": 3, "pred": "11", "truth": "11", "correct": true, "edits_per_token": 8.0}, {"uid": "7e2f3ac39edf259e", "family": "bind", "template": 2, "depth": 3, "pred": "29", "truth": "29", "correct": true, "edits_per_token": 8.0}, {"uid": "f78cbce45d518768", "family": "fsm", "template": 0, "depth": 1, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 8.0}, {"uid": "67327201e0703535", "family": "fsm", "template": 1, "depth": 2, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 8.0}, {"uid": "ad02a284f829c979", "family": "fsm", "template": 1, "depth": 2, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 8.0}, {"uid": "d66d39facf8c09dc", "family": "arith", "template": 2, "depth": 2, "pred": "24", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "961f7a505cc9733d", "family": "arith", "template": 2, "depth": 2, "pred": "-104", "truth": "-120", "correct": false, "edits_per_token": 8.0}, {"uid": "2a66eac56efc842a", "family": "arith", "template": 2, "depth": 2, "pred": "230", "truth": "195", "correct": false, "edits_per_token": 8.0}, {"uid": "0b6aca1ee4bcc5e4", "family": "arith", "template": 2, "depth": 2, "pred": "-56", "truth": "-54", "correct": false, "edits_per_token": 8.0}, {"uid": "a7520c88e128325d", "family": "arith", "template": 2, "depth": 2, "pred": "7", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "a4a050b886f6d6f1", "family": "arith", "template": 2, "depth": 2, "pred": "64", "truth": "68", "correct": false, "edits_per_token": 8.0}, {"uid": "822a99424bbcf8ea", "family": "arith", "template": 2, "depth": 2, "pred": "40", "truth": "42", "correct": false, "edits_per_token": 8.0}, {"uid": "8fbd3d2ce9fd04d2", "family": "bind", "template": 0, "depth": 3, "pred": "55", "truth": "60", "correct": false, "edits_per_token": 8.0}, {"uid": "2ef3a2f2a0328157", "family": "bind", "template": 0, "depth": 3, "pred": "36", "truth": "32", "correct": false, "edits_per_token": 8.0}, {"uid": "588fe9ad929612f6", "family": "bind", "template": 0, "depth": 3, "pred": "11", "truth": "11", "correct": true, "edits_per_token": 8.0}, {"uid": "3db5b222471f96fa", "family": "bind", "template": 0, "depth": 3, "pred": "20", "truth": "20", "correct": true, "edits_per_token": 8.0}, {"uid": "b96935b62f495f58", "family": "arith", "template": 2, "depth": 3, "pred": "119", "truth": "149", "correct": false, "edits_per_token": 8.0}, {"uid": "4ce4e40dc4a8caf4", "family": "bind", "template": 0, "depth": 3, "pred": "28", "truth": "64", "correct": false, "edits_per_token": 8.0}, {"uid": "0acc51832dea68c1", "family": "bind", "template": 0, "depth": 3, "pred": "21", "truth": "23", "correct": false, "edits_per_token": 8.0}, {"uid": "6f0f31272e9dccb8", "family": "fsm", "template": 1, "depth": 2, "pred": "19", "truth": "22", "correct": false, "edits_per_token": 8.0}, {"uid": "43de2e7618ca4cb3", "family": "fsm", "template": 2, "depth": 3, "pred": "20", "truth": "18", "correct": false, "edits_per_token": 8.0}, {"uid": "ffb7dab9fc31fbd5", "family": "arith", "template": 2, "depth": 3, "pred": "44", "truth": "42", "correct": false, "edits_per_token": 8.0}, {"uid": "25ab3f6cf0620687", "family": "arith", "template": 2, "depth": 3, "pred": "40", "truth": "38", "correct": false, "edits_per_token": 8.0}, {"uid": "6e8c3b9ad7c915f8", "family": "arith", "template": 2, "depth": 3, "pred": "-13", "truth": "-13", "correct": true, "edits_per_token": 8.0}, {"uid": "595ac6801eedc4f3", "family": "bind", "template": 0, "depth": 3, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 8.0}, {"uid": "76fdb2e6417c6f93", "family": "bind", "template": 1, "depth": 3, "pred": "12", "truth": "11", "correct": false, "edits_per_token": 8.0}, {"uid": "d8e6469a252c53ab", "family": "fsm", "template": 2, "depth": 3, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 8.0}, {"uid": "aa252e805cf95b6a", "family": "arith", "template": 2, "depth": 3, "pred": "17", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "e2bce580fe2875d8", "family": "arith", "template": 2, "depth": 3, "pred": "62", "truth": "60", "correct": false, "edits_per_token": 8.0}, {"uid": "a3811535637f42cf", "family": "arith", "template": 2, "depth": 3, "pred": "-14", "truth": "-13", "correct": false, "edits_per_token": 8.0}, {"uid": "920633664a3100c7", "family": "arith", "template": 2, "depth": 3, "pred": "3", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "6d1db4bf8883e127", "family": "bind", "template": 1, "depth": 3, "pred": "22", "truth": "22", "correct": true, "edits_per_token": 8.0}, {"uid": "42b7111968d0fd6e", "family": "bind", "template": 1, "depth": 3, "pred": "77", "truth": "74", "correct": false, "edits_per_token": 8.0}, {"uid": "0d7d21b45a123ecc", "family": "bind", "template": 1, "depth": 3, "pred": "27", "truth": "28", "correct": false, "edits_per_token": 8.0}, {"uid": "0070225255190df8", "family": "bind", "template": 1, "depth": 3, "pred": "-5", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "9efc725b639fa1e2", "family": "bind", "template": 1, "depth": 3, "pred": "43", "truth": "42", "correct": false, "edits_per_token": 8.0}, {"uid": "e6d2938cf240e7d7", "family": "bind", "template": 1, "depth": 3, "pred": "-2", "truth": "-2", "correct": true, "edits_per_token": 8.0}, {"uid": "6333807fc55e47c2", "family": "bind", "template": 2, "depth": 4, "pred": "27", "truth": "27", "correct": true, "edits_per_token": 8.0}, {"uid": "73bab13d0adb93c9", "family": "bind", "template": 2, "depth": 4, "pred": "33", "truth": "31", "correct": false, "edits_per_token": 8.0}, {"uid": "7e4a5eedc90f317e", "family": "arith", "template": 2, "depth": 4, "pred": "577", "truth": "707", "correct": false, "edits_per_token": 8.0}, {"uid": "72ab6c11b6b99a0c", "family": "bind", "template": 2, "depth": 4, "pred": "8", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "82d8e4541e035995", "family": "bind", "template": 2, "depth": 4, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 8.0}, {"uid": "11248162bd234998", "family": "bind", "template": 2, "depth": 4, "pred": "63", "truth": "61", "correct": false, "edits_per_token": 8.0}, {"uid": "30512b180423dd81", "family": "fsm", "template": 0, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 8.0}, {"uid": "5fff7cb1816a8b83", "family": "arith", "template": 2, "depth": 4, "pred": "-120", "truth": "-90", "correct": false, "edits_per_token": 8.0}, {"uid": "916e8e5dc57e71b5", "family": "arith", "template": 2, "depth": 4, "pred": "37", "truth": "33", "correct": false, "edits_per_token": 8.0}, {"uid": "86a2ceda3911a38b", "family": "arith", "template": 2, "depth": 4, "pred": "141", "truth": "165", "correct": false, "edits_per_token": 8.0}, {"uid": "407e416b5a6578ae", "family": "arith", "template": 2, "depth": 4, "pred": "210", "truth": "222", "correct": false, "edits_per_token": 8.0}, {"uid": "f3df5ab6d440dc66", "family": "bind", "template": 2, "depth": 4, "pred": "39", "truth": "40", "correct": false, "edits_per_token": 8.0}, {"uid": "d8aa4893aec3878d", "family": "bind", "template": 2, "depth": 4, "pred": "284", "truth": "316", "correct": false, "edits_per_token": 8.0}, {"uid": "effc949016a8ea12", "family": "fsm", "template": 0, "depth": 2, "pred": "11", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "efbc892e6b4a8d10", "family": "fsm", "template": 1, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 8.0}, {"uid": "9ed8a702224d9157", "family": "fsm", "template": 1, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 8.0}, {"uid": "447dc7313518c9cc", "family": "arith", "template": 2, "depth": 4, "pred": "-40", "truth": "-41", "correct": false, "edits_per_token": 8.0}, {"uid": "c9f30e7c90083e3b", "family": "arith", "template": 2, "depth": 4, "pred": "58", "truth": "70", "correct": false, "edits_per_token": 8.0}, {"uid": "414ba9ab6e9d47f0", "family": "fsm", "template": 2, "depth": 3, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 8.0}, {"uid": "d6bbe76ab7ae5dea", "family": "bind", "template": 0, "depth": 4, "pred": "5", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "ad80d63be705294d", "family": "fsm", "template": 0, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 8.0}, {"uid": "69d134b029d15133", "family": "bind", "template": 0, "depth": 4, "pred": "-11", "truth": "-8", "correct": false, "edits_per_token": 8.0}, {"uid": "1ed0d6ff18c8180d", "family": "bind", "template": 0, "depth": 4, "pred": "11", "truth": "11", "correct": true, "edits_per_token": 8.0}, {"uid": "9679ca491537d4ae", "family": "fsm", "template": 0, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 8.0}, {"uid": "06e9560336c69fc0", "family": "bind", "template": 0, "depth": 4, "pred": "100", "truth": "99", "correct": false, "edits_per_token": 8.0}, {"uid": "83785d82a70218c2", "family": "bind", "template": 0, "depth": 4, "pred": "8", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "365ba7f8ce47ab26", "family": "bind", "template": 0, "depth": 4, "pred": "54", "truth": "51", "correct": false, "edits_per_token": 8.0}, {"uid": "f0b461762a54a509", "family": "bind", "template": 1, "depth": 4, "pred": "680", "truth": "368", "correct": false, "edits_per_token": 8.0}, {"uid": "ef8628813a4db970", "family": "bind", "template": 0, "depth": 4, "pred": "60", "truth": "64", "correct": false, "edits_per_token": 8.0}, {"uid": "a07ed018d16711ef", "family": "bind", "template": 1, "depth": 4, "pred": "101", "truth": "81", "correct": false, "edits_per_token": 8.0}, {"uid": "8566721d2752b3a8", "family": "bind", "template": 1, "depth": 4, "pred": "-12", "truth": "-8", "correct": false, "edits_per_token": 8.0}, {"uid": "14e0f0613ab5fafb", "family": "bind", "template": 1, "depth": 4, "pred": "43", "truth": "41", "correct": false, "edits_per_token": 8.0}, {"uid": "e5b10de969f4f1de", "family": "bind", "template": 1, "depth": 4, "pred": "-20", "truth": "-22", "correct": false, "edits_per_token": 8.0}, {"uid": "9a6304b25504acd5", "family": "fsm", "template": 2, "depth": 4, "pred": "13", "truth": "12", "correct": false, "edits_per_token": 8.0}, {"uid": "f9e4c5acf05b6575", "family": "bind", "template": 1, "depth": 4, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 8.0}, {"uid": "2efa0c713c22db5a", "family": "bind", "template": 1, "depth": 4, "pred": "94", "truth": "101", "correct": false, "edits_per_token": 8.0}, {"uid": "8ce65aa06b428682", "family": "fsm", "template": 0, "depth": 3, "pred": "24", "truth": "24", "correct": true, "edits_per_token": 8.0}, {"uid": "4f7a4c9e9fd88e23", "family": "fsm", "template": 1, "depth": 3, "pred": "-4", "truth": "-4", "correct": true, "edits_per_token": 8.0}, {"uid": "2468d102e506d6d5", "family": "fsm", "template": 2, "depth": 3, "pred": "12", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "6f3c8de73142af82", "family": "fsm", "template": 2, "depth": 4, "pred": "35", "truth": "37", "correct": false, "edits_per_token": 8.0}, {"uid": "ba3af2217536a7c6", "family": "fsm", "template": 0, "depth": 2, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 8.0}, {"uid": "e063912681967ddf", "family": "fsm", "template": 0, "depth": 3, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 8.0}, {"uid": "583473e83be2cb0e", "family": "fsm", "template": 2, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "5a873e995116c10d", "family": "fsm", "template": 1, "depth": 3, "pred": "80", "truth": "72", "correct": false, "edits_per_token": 8.0}, {"uid": "05764ff46e4d28b8", "family": "fsm", "template": 2, "depth": 4, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 8.0}, {"uid": "46b570766ec22e2a", "family": "fsm", "template": 0, "depth": 3, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "59f43bee78d1b7d5", "family": "fsm", "template": 0, "depth": 3, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 8.0}, {"uid": "eaf14d7290a1413b", "family": "fsm", "template": 1, "depth": 3, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 8.0}, {"uid": "b7248f02464a9d5e", "family": "fsm", "template": 1, "depth": 3, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 8.0}, {"uid": "9aaa1b5b75c855bb", "family": "fsm", "template": 2, "depth": 3, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 8.0}, {"uid": "a7203ce2418d2ce9", "family": "fsm", "template": 0, "depth": 4, "pred": "16", "truth": "18", "correct": false, "edits_per_token": 8.0}, {"uid": "b86f2cf77ff75b95", "family": "fsm", "template": 2, "depth": 4, "pred": "9", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "5480488e0f55e2a2", "family": "fsm", "template": 0, "depth": 3, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 8.0}, {"uid": "ca83fc2e201063f6", "family": "fsm", "template": 0, "depth": 3, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 8.0}, {"uid": "324c8cdae8f3169c", "family": "fsm", "template": 1, "depth": 4, "pred": "-4", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "2723eee2e2d0f964", "family": "fsm", "template": 1, "depth": 3, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 8.0}, {"uid": "8749497eaab6176d", "family": "fsm", "template": 1, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "f006a9e07042e43f", "family": "fsm", "template": 2, "depth": 4, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 8.0}, {"uid": "def842732bc0714f", "family": "fsm", "template": 2, "depth": 4, "pred": "-4", "truth": "-4", "correct": true, "edits_per_token": 8.0}, {"uid": "e4ca09128be213ea", "family": "fsm", "template": 0, "depth": 3, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 8.0}, {"uid": "a1c297725f5ae175", "family": "fsm", "template": 1, "depth": 4, "pred": "0", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "7d9f104b41e0b7a5", "family": "fsm", "template": 0, "depth": 4, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 8.0}, {"uid": "536ead7bf94947ee", "family": "fsm", "template": 1, "depth": 4, "pred": "18", "truth": "18", "correct": true, "edits_per_token": 8.0}, {"uid": "9808a14025bb62db", "family": "fsm", "template": 1, "depth": 4, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 8.0}, {"uid": "666a3ad814af7485", "family": "fsm", "template": 0, "depth": 4, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 8.0}, {"uid": "29e5f1d39e9dd718", "family": "excl", "template": 2, "depth": 1, "pred": "apple plum", "truth": "date", "correct": false, "edits_per_token": 8.0}, {"uid": "4425b91f251fc921", "family": "excl", "template": 2, "depth": 1, "pred": "plum pear", "truth": "lime", "correct": false, "edits_per_token": 8.0}, {"uid": "899865b46b45638d", "family": "sched", "template": 2, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "bb4df57d9a04ccab", "family": "fsm", "template": 1, "depth": 3, "pred": "-5", "truth": "-5", "correct": true, "edits_per_token": 8.0}, {"uid": "5aee313fef7904e6", "family": "fsm", "template": 2, "depth": 4, "pred": "0", "truth": "-1", "correct": false, "edits_per_token": 8.0}, {"uid": "b6a823db9fe091cf", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "eae15d4721c93c28", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "ebdd7e877a44588b", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "a467884db4d82ee1", "family": "fsm", "template": 1, "depth": 4, "pred": "10", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "a0e13fa29eaee344", "family": "fsm", "template": 0, "depth": 4, "pred": "18", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "55bddbb405630c49", "family": "excl", "template": 2, "depth": 1, "pred": "plum lime a", "truth": "kiwi plum apple", "correct": false, "edits_per_token": 8.0}, {"uid": "9e10c6f2c824ea03", "family": "fsm", "template": 1, "depth": 4, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "e29749c80774340d", "family": "fsm", "template": 1, "depth": 4, "pred": "-7", "truth": "-8", "correct": false, "edits_per_token": 8.0}, {"uid": "305ec8d13dc601f2", "family": "excl", "template": 2, "depth": 1, "pred": "kiwi apple", "truth": "kiwi fig yam bean", "correct": false, "edits_per_token": 8.0}, {"uid": "c56841284720069d", "family": "fsm", "template": 0, "depth": 4, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 8.0}, {"uid": "b757b7fb6b160032", "family": "fsm", "template": 0, "depth": 4, "pred": "-8", "truth": "-7", "correct": false, "edits_per_token": 8.0}, {"uid": "8a849a68ac0bd89d", "family": "sched", "template": 1, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "24c9898cc5168e60", "family": "sched", "template": 1, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "b45de26fb239a53c", "family": "fsm", "template": 0, "depth": 4, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 8.0}, {"uid": "b92a8bb6a6b6c2fc", "family": "sched", "template": 2, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "24f6f2d470ed5ab8", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "265c9cd1ac273c94", "family": "sched", "template": 2, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "17e81034d97a73e1", "family": "excl", "template": 1, "depth": 2, "pred": "date nut", "truth": "yam", "correct": false, "edits_per_token": 8.0}, {"uid": "52b9e62b2a6ca3c2", "family": "excl", "template": 2, "depth": 3, "pred": "fig", "truth": "yam", "correct": false, "edits_per_token": 8.0}, {"uid": "3ea463a1eb42a642", "family": "excl", "template": 2, "depth": 1, "pred": "bean fig pl", "truth": "bean plum date nut", "correct": false, "edits_per_token": 8.0}, {"uid": "290428df35319b49", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "e8f3f5315f909938", "family": "excl", "template": 2, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "679e75695e94a8a8", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "b9763ee8284b60af", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "50436cd63f7d344d", "family": "excl", "template": 1, "depth": 1, "pred": "bean apple", "truth": "bean date apple", "correct": false, "edits_per_token": 8.0}, {"uid": "f91565e19106bd0c", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "f377b423248cc145", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "fd984d3e4a7605fd", "family": "excl", "template": 0, "depth": 2, "pred": "pear kiwi", "truth": "nut", "correct": false, "edits_per_token": 8.0}, {"uid": "78ec3d641e6859a8", "family": "excl", "template": 2, "depth": 2, "pred": "kiwi pear", "truth": "date apple", "correct": false, "edits_per_token": 8.0}, {"uid": "ec07ebafd22ecb2e", "family": "excl", "template": 1, "depth": 2, "pred": "plum pear", "truth": "pear", "correct": false, "edits_per_token": 8.0}, {"uid": "637c344de20f5625", "family": "excl", "template": 2, "depth": 2, "pred": "fig date be", "truth": "apple date fig bean", "correct": false, "edits_per_token": 8.0}, {"uid": "b1127806bf880411", "family": "sched", "template": 0, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "a65132179f29f33e", "family": "excl", "template": 0, "depth": 1, "pred": "apple lime", "truth": "apple lime", "correct": true, "edits_per_token": 8.0}, {"uid": "610ab75a850b6358", "family": "sched", "template": 0, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "09a0f4119071f489", "family": "excl", "template": 2, "depth": 1, "pred": "pear fig li", "truth": "pear kiwi fig nut", "correct": false, "edits_per_token": 8.0}, {"uid": "9c0d7b46b835ff7f", "family": "excl", "template": 2, "depth": 1, "pred": "plum fig pe", "truth": "plum kiwi date nut", "correct": false, "edits_per_token": 8.0}, {"uid": "e9948bd8b29e69d3", "family": "excl", "template": 2, "depth": 2, "pred": "lime nut ki", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "f948792de5a76409", "family": "excl", "template": 0, "depth": 1, "pred": "kiwi fig li", "truth": "apple kiwi lime fig", "correct": false, "edits_per_token": 8.0}, {"uid": "43382d613173a703", "family": "sched", "template": 1, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "4378e430fe1485f9", "family": "sched", "template": 1, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "7af9657dd3fba74a", "family": "excl", "template": 2, "depth": 2, "pred": "date yam ap", "truth": "date plum", "correct": false, "edits_per_token": 8.0}, {"uid": "c9dda90c7d0ad8ed", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "2f7d7388cf61d3fe", "family": "sched", "template": 1, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "114ddae29f526626", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "d95b1cb8c2923e7e", "family": "excl", "template": 0, "depth": 1, "pred": "bean lime a", "truth": "bean lime apple", "correct": false, "edits_per_token": 8.0}, {"uid": "1dcd25a6eb16038d", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "59114c794ee3a087", "family": "excl", "template": 0, "depth": 2, "pred": "bean plum p", "truth": "bean", "correct": false, "edits_per_token": 8.0}, {"uid": "a9742f748ad1ad5c", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "ec7190a280bfbdb7", "family": "excl", "template": 2, "depth": 4, "pred": "apple", "truth": "lime", "correct": false, "edits_per_token": 8.0}, {"uid": "81a4bc002847b518", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "fce4affe8df7979a", "family": "excl", "template": 1, "depth": 1, "pred": "nut date li", "truth": "date apple lime", "correct": false, "edits_per_token": 8.0}, {"uid": "50c50f83a59dabd8", "family": "excl", "template": 2, "depth": 2, "pred": "yam kiwi pl", "truth": "fig bean kiwi plum apple", "correct": false, "edits_per_token": 8.0}, {"uid": "9a0b5958ebfcc667", "family": "sched", "template": 1, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "2b9fc22b15089625", "family": "excl", "template": 0, "depth": 1, "pred": "pear apple", "truth": "pear kiwi", "correct": false, "edits_per_token": 8.0}, {"uid": "2d7af060d17ced33", "family": "sched", "template": 1, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "22973924b5d55519", "family": "sched", "template": 1, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "9eba62cb7a72f599", "family": "sched", "template": 2, "depth": 2, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "60adda9ef6e12d3c", "family": "excl", "template": 0, "depth": 1, "pred": "kiwi pear a", "truth": "kiwi bean apple", "correct": false, "edits_per_token": 8.0}, {"uid": "222e860f500afa48", "family": "excl", "template": 1, "depth": 1, "pred": "pear fig nu", "truth": "pear nut fig lime", "correct": false, "edits_per_token": 8.0}, {"uid": "883e00248131789c", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 8.0}, {"uid": "b7eae28e49657a0d", "family": "sched", "template": 2, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "36bc16e15622ea97", "family": "sched", "template": 2, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 8.0}, {"uid": "9b1170c69bedb5d0", "family": "excl", "template": 1, "depth": 3, "pred": "nut date", "truth": "plum date", "correct": false, "edits_per_token": 8.0}, {"uid": "e7a942f9776e1481", "family": "excl", "template": 1, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "03e8f11a87973087", "family": "sched", "template": 0, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "0641e2dbfe875c84", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "b60c5ce59c8ad584", "family": "sched", "template": 2, "depth": 3, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "a07385bb7e4ea97b", "family": "sched", "template": 0, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "a942d085ec964894", "family": "sched", "template": 2, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "800f61c9e4ad0bf4", "family": "excl", "template": 1, "depth": 2, "pred": "fig lime", "truth": "nut bean", "correct": false, "edits_per_token": 8.0}, {"uid": "19887d3ca18adb40", "family": "sched", "template": 0, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 8.0}, {"uid": "01dd5269d9e0a510", "family": "sched", "template": 0, "depth": 3, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 8.0}, {"uid": "d1578ac0adce4a3f", "family": "sched", "template": 2, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "4f7ac4cb4667edb6", "family": "excl", "template": 0, "depth": 2, "pred": "lime apple", "truth": "bean date", "correct": false, "edits_per_token": 8.0}, {"uid": "9ebb20923f7ff360", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "45e5e22be65e2392", "family": "sched", "template": 2, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "58590db8d34f4bb6", "family": "sched", "template": 2, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "e0aa686b67447872", "family": "excl", "template": 1, "depth": 1, "pred": "none", "truth": "lime fig", "correct": false, "edits_per_token": 8.0}, {"uid": "b9758ae4de312e38", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "apple", "correct": false, "edits_per_token": 8.0}, {"uid": "49b538b6cebfad32", "family": "sched", "template": 0, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "b41c090edc1c2570", "family": "sched", "template": 0, "depth": 2, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "57f25f911fd47979", "family": "excl", "template": 0, "depth": 2, "pred": "fig lime", "truth": "bean", "correct": false, "edits_per_token": 8.0}, {"uid": "52a3076c67262025", "family": "excl", "template": 0, "depth": 2, "pred": "bean fig", "truth": "bean fig date", "correct": false, "edits_per_token": 8.0}, {"uid": "5313d51deae8c2b8", "family": "excl", "template": 0, "depth": 3, "pred": "none", "truth": "yam", "correct": false, "edits_per_token": 8.0}, {"uid": "fbb63a5c5980a546", "family": "excl", "template": 1, "depth": 1, "pred": "plum nut pe", "truth": "date apple", "correct": false, "edits_per_token": 8.0}, {"uid": "123ddaa3e5efb215", "family": "excl", "template": 1, "depth": 1, "pred": "fig bean li", "truth": "fig pear nut kiwi", "correct": false, "edits_per_token": 8.0}, {"uid": "bf6d2e6f97942225", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "3a419d90cdd84493", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "yam plum nut", "correct": false, "edits_per_token": 8.0}, {"uid": "dc7a2b700ef3eec3", "family": "sched", "template": 0, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "a05184e028d82ee0", "family": "sched", "template": 0, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "b9e25b6eb179c90b", "family": "excl", "template": 0, "depth": 3, "pred": "kiwi bean p", "truth": "kiwi nut fig pear", "correct": false, "edits_per_token": 8.0}, {"uid": "914dc5465d528e9b", "family": "sched", "template": 0, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "64f71380d66efae2", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "33fe241315cf5b9f", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "1b7f8f05a6c5a7a3", "family": "excl", "template": 2, "depth": 2, "pred": "nut", "truth": "plum apple", "correct": false, "edits_per_token": 8.0}, {"uid": "5c13ea4ff7d521b5", "family": "sched", "template": 1, "depth": 2, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "2a2cf42fd19abbe2", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "df171a88e16d0896", "family": "excl", "template": 0, "depth": 2, "pred": "lime apple", "truth": "apple yam", "correct": false, "edits_per_token": 8.0}, {"uid": "a6b723b79a990fea", "family": "excl", "template": 1, "depth": 1, "pred": "nut kiwi ap", "truth": "nut kiwi lime fig", "correct": false, "edits_per_token": 8.0}, {"uid": "95569dc6b1f0424e", "family": "excl", "template": 2, "depth": 4, "pred": "fig", "truth": "nut fig", "correct": false, "edits_per_token": 8.0}, {"uid": "d5cb804202c06b04", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 8.0}, {"uid": "26a8bdf6da59e053", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "8461de28662a09d0", "family": "excl", "template": 0, "depth": 1, "pred": "yam bean", "truth": "lime kiwi apple", "correct": false, "edits_per_token": 8.0}, {"uid": "5266b24716f21c1b", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 8.0}, {"uid": "1301b2b886fc0fed", "family": "sched", "template": 1, "depth": 3, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "0ae3e633b9c73a6d", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 8.0}, {"uid": "a34d5263586bf792", "family": "sched", "template": 1, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "72b16296ef29803c", "family": "sched", "template": 2, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "1650d27be4762458", "family": "sched", "template": 2, "depth": 4, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "d17c2abfac01fb85", "family": "excl", "template": 1, "depth": 3, "pred": "nut kiwi", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "f7bb7ae8302c82e3", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "ce88ea5df4ac7c7e", "family": "sched", "template": 1, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "89232c53de86f0c7", "family": "sched", "template": 2, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "7867942486d7f380", "family": "sched", "template": 0, "depth": 4, "pred": "none", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "c9726e189052aad1", "family": "sched", "template": 2, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 8.0}, {"uid": "302f485c304f8e1c", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "58e2a723355b2957", "family": "sched", "template": 0, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "376a16c13fc8af71", "family": "sched", "template": 0, "depth": 4, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "7eddd2c6ff72770f", "family": "sched", "template": 0, "depth": 4, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "ebcfc88ef93a66df", "family": "sched", "template": 2, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "89a256c76069f616", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "plum lime", "correct": false, "edits_per_token": 8.0}, {"uid": "34292f4e57b84e60", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "b4ef07715474f790", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "084073be6e34810e", "family": "excl", "template": 2, "depth": 2, "pred": "lime", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "fec42ca9bcf0a20b", "family": "excl", "template": 1, "depth": 2, "pred": "date bean", "truth": "date plum bean", "correct": false, "edits_per_token": 8.0}, {"uid": "9600b1b302d34721", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "e68c491cb5fe0db5", "family": "excl", "template": 0, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "8fbf1093b19d178d", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 8.0}, {"uid": "152fa59b6f060034", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "6886f89d5fecebc2", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "25810ef41631a5a0", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "date", "correct": false, "edits_per_token": 8.0}, {"uid": "c2c1af96873ca940", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "493eb05c569e6806", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "yam fig date", "correct": false, "edits_per_token": 8.0}, {"uid": "3ca58bdf2e93bec8", "family": "excl", "template": 1, "depth": 2, "pred": "apple yam b", "truth": "nut", "correct": false, "edits_per_token": 8.0}, {"uid": "68cafc987827b672", "family": "excl", "template": 0, "depth": 1, "pred": "apple yam k", "truth": "apple lime nut kiwi bean", "correct": false, "edits_per_token": 8.0}, {"uid": "e5f870c9ac9c2bdc", "family": "excl", "template": 1, "depth": 2, "pred": "apple bean", "truth": "apple pear plum", "correct": false, "edits_per_token": 8.0}, {"uid": "4aba72aed6f820bb", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "bean", "correct": false, "edits_per_token": 8.0}, {"uid": "7e3d734f2577f012", "family": "excl", "template": 2, "depth": 3, "pred": "pear apple", "truth": "plum bean", "correct": false, "edits_per_token": 8.0}, {"uid": "da06a708de7351a1", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "e13f444f5e921059", "family": "sched", "template": 1, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 8.0}, {"uid": "806dfca57003d938", "family": "sched", "template": 1, "depth": 4, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "5a3700bf53c0daf3", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 8.0}, {"uid": "fa8b059c364916e0", "family": "sched", "template": 1, "depth": 4, "pred": "D", "truth": "B", "correct": false, "edits_per_token": 8.0}, {"uid": "9fd10a99f31cee6e", "family": "sched", "template": 1, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "b3fe0fa2ac161f59", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "2489ee7572296937", "family": "sched", "template": 2, "depth": 4, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "80d519c064361dae", "family": "sched", "template": 2, "depth": 4, "pred": "C", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "5a66b965f98dc2cf", "family": "excl", "template": 0, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "4cd96fbf4f41a908", "family": "sched", "template": 0, "depth": 4, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 8.0}, {"uid": "7a4191b42dce2b06", "family": "sched", "template": 1, "depth": 3, "pred": "none", "truth": "B", "correct": false, "edits_per_token": 8.0}, {"uid": "b87404e94d9a09f6", "family": "excl", "template": 0, "depth": 2, "pred": "none", "truth": "fig kiwi pear", "correct": false, "edits_per_token": 8.0}, {"uid": "2a9ecd3661b30257", "family": "sched", "template": 0, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 8.0}, {"uid": "5249781ebc348d37", "family": "sched", "template": 2, "depth": 4, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 8.0}, {"uid": "2e4acd7d13860ac2", "family": "sched", "template": 2, "depth": 4, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "2adc39419c8b49a4", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "fig lime", "correct": false, "edits_per_token": 8.0}, {"uid": "dca6675d28527584", "family": "excl", "template": 0, "depth": 3, "pred": "fig date ki", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "c666c9854336f784", "family": "sched", "template": 0, "depth": 4, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "8f480bee1ead1646", "family": "excl", "template": 0, "depth": 3, "pred": "nut yam", "truth": "apple", "correct": false, "edits_per_token": 8.0}, {"uid": "cf67501c8d912952", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "d61260228e19e8ba", "family": "sched", "template": 2, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "71fb11da7ef26fa8", "family": "sched", "template": 0, "depth": 4, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "f12148b1efc8abed", "family": "excl", "template": 2, "depth": 4, "pred": "date", "truth": "nut", "correct": false, "edits_per_token": 8.0}, {"uid": "9e76644064b0dc70", "family": "excl", "template": 0, "depth": 3, "pred": "date yam", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "12b5acb0d562716c", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "fd29037f3e23207c", "family": "sched", "template": 1, "depth": 4, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "dbd4f3b44d2455e6", "family": "sched", "template": 1, "depth": 4, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 8.0}, {"uid": "2bbd29545a49585f", "family": "sched", "template": 1, "depth": 4, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "e7a65045762e87f2", "family": "sched", "template": 1, "depth": 4, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "9476a68c1c10bc9b", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "dfb1aad2e59286cb", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "fe2915e5def8bf83", "family": "excl", "template": 1, "depth": 4, "pred": "fig apple n", "truth": "yam", "correct": false, "edits_per_token": 8.0}, {"uid": "548b800800b9067a", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "19a75d08ddd9d502", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 8.0}, {"uid": "6c5b7174f7312602", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "11c0e819fd3fe52e", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}] | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_template_b0.json
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +[{"uid": "3b112da5fd01aae7", "family": "fsm", "template": 4, "depth": 1, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "d5c0c39e453dcca1", "family": "fsm", "template": 4, "depth": 1, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "edd44a1f63841ef0", "family": "fsm", "template": 4, "depth": 1, "pred": "15", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "4a60e827734a2aad", "family": "fsm", "template": 4, "depth": 1, "pred": "13", "truth": "13", "correct": true, "edits_per_token": 0.0}, {"uid": "d2d4172d0d7eda08", "family": "fsm", "template": 4, "depth": 1, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 0.0}, {"uid": "033ec88b5e210826", "family": "fsm", "template": 4, "depth": 1, "pred": "14", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "97fc165388a1bca6", "family": "fsm", "template": 4, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "ee578f5db025d5f9", "family": "fsm", "template": 4, "depth": 1, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "c396858313b1867d", "family": "fsm", "template": 4, "depth": 2, "pred": "20", "truth": "20", "correct": true, "edits_per_token": 0.0}, {"uid": "62686c3fb5450969", "family": "bind", "template": 4, "depth": 1, "pred": "9", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "160deb3944fa94a9", "family": "fsm", "template": 4, "depth": 2, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 0.0}, {"uid": "ff167bdae9dae304", "family": "fsm", "template": 4, "depth": 2, "pred": "1", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "6e2f978b50da11db", "family": "fsm", "template": 4, "depth": 2, "pred": "-1", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "b3c197791022cdda", "family": "fsm", "template": 4, "depth": 2, "pred": "-1", "truth": "0", "correct": false, "edits_per_token": 0.0}, {"uid": "c6bc5deb98b3f6a4", "family": "fsm", "template": 4, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "07c9ad219cedeecc", "family": "fsm", "template": 4, "depth": 2, "pred": "1", "truth": "0", "correct": false, "edits_per_token": 0.0}, {"uid": "16a3e4051536fc5a", "family": "arith", "template": 4, "depth": 1, "pred": "-14", "truth": "-16", "correct": false, "edits_per_token": 0.0}, {"uid": "4f1f75fe235ae841", "family": "arith", "template": 4, "depth": 1, "pred": "-21", "truth": "-21", "correct": true, "edits_per_token": 0.0}, {"uid": "cd785088c9149dfb", "family": "arith", "template": 4, "depth": 1, "pred": "144", "truth": "162", "correct": false, "edits_per_token": 0.0}, {"uid": "1638c5c5115e7382", "family": "arith", "template": 4, "depth": 1, "pred": "54", "truth": "48", "correct": false, "edits_per_token": 0.0}, {"uid": "bb572994a445bdb5", "family": "arith", "template": 4, "depth": 1, "pred": "22", "truth": "21", "correct": false, "edits_per_token": 0.0}, {"uid": "ec5611b48356ecc9", "family": "arith", "template": 4, "depth": 1, "pred": "48", "truth": "42", "correct": false, "edits_per_token": 0.0}, {"uid": "f4a95344e3c4417f", "family": "arith", "template": 4, "depth": 1, "pred": "40", "truth": "30", "correct": false, "edits_per_token": 0.0}, {"uid": "e4af4779733a0b3a", "family": "arith", "template": 4, "depth": 1, "pred": "-18", "truth": "-19", "correct": false, "edits_per_token": 0.0}, {"uid": "7e8ac8f95805f2f3", "family": "arith", "template": 4, "depth": 1, "pred": "27", "truth": "29", "correct": false, "edits_per_token": 0.0}, {"uid": "036f0619e07338fd", "family": "bind", "template": 4, "depth": 1, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "e811bf68f0bfdded", "family": "bind", "template": 4, "depth": 1, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 0.0}, {"uid": "82f1818aa73ed36d", "family": "bind", "template": 4, "depth": 1, "pred": "13", "truth": "12", "correct": false, "edits_per_token": 0.0}, {"uid": "f4d0e013d4462dd0", "family": "bind", "template": 4, "depth": 1, "pred": "8", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "4dd12c9283c1639d", "family": "arith", "template": 4, "depth": 1, "pred": "38", "truth": "37", "correct": false, "edits_per_token": 0.0}, {"uid": "889301436be7278d", "family": "bind", "template": 4, "depth": 1, "pred": "48", "truth": "45", "correct": false, "edits_per_token": 0.0}, {"uid": "34157746c6a9aa26", "family": "bind", "template": 4, "depth": 1, "pred": "36", "truth": "30", "correct": false, "edits_per_token": 0.0}, {"uid": "972d347d02c3e4c9", "family": "bind", "template": 4, "depth": 1, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 0.0}, {"uid": "73358f5d5d68a323", "family": "bind", "template": 4, "depth": 1, "pred": "13", "truth": "13", "correct": true, "edits_per_token": 0.0}, {"uid": "621a81bcb0f3a1da", "family": "bind", "template": 4, "depth": 1, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 0.0}, {"uid": "be55eff500fd9de4", "family": "arith", "template": 4, "depth": 2, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 0.0}, {"uid": "8086e5fd466e56e3", "family": "arith", "template": 4, "depth": 2, "pred": "21", "truth": "23", "correct": false, "edits_per_token": 0.0}, {"uid": "1d3e8e419ebffaea", "family": "bind", "template": 3, "depth": 1, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 0.0}, {"uid": "03afb70c0ecdf774", "family": "fsm", "template": 4, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "53768ffc251ff5ea", "family": "arith", "template": 4, "depth": 2, "pred": "56", "truth": "56", "correct": true, "edits_per_token": 0.0}, {"uid": "856912fe4bd92aab", "family": "arith", "template": 4, "depth": 2, "pred": "720", "truth": "576", "correct": false, "edits_per_token": 0.0}, {"uid": "bcfa81650916d43a", "family": "arith", "template": 4, "depth": 2, "pred": "28", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "636ae00985bb6e39", "family": "bind", "template": 3, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "866b93f2b1cebb8a", "family": "bind", "template": 3, "depth": 1, "pred": "36", "truth": "33", "correct": false, "edits_per_token": 0.0}, {"uid": "2449760460938f84", "family": "bind", "template": 3, "depth": 1, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 0.0}, {"uid": "664b188b12b5afc4", "family": "arith", "template": 4, "depth": 2, "pred": "23", "truth": "27", "correct": false, "edits_per_token": 0.0}, {"uid": "efba878fb6ac5fa8", "family": "arith", "template": 4, "depth": 2, "pred": "0", "truth": "-36", "correct": false, "edits_per_token": 0.0}, {"uid": "149112577b7a6b0b", "family": "arith", "template": 4, "depth": 2, "pred": "-12", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "4f926ee35fd6594e", "family": "bind", "template": 3, "depth": 1, "pred": "14", "truth": "12", "correct": false, "edits_per_token": 0.0}, {"uid": "1432cede89ff82db", "family": "bind", "template": 3, "depth": 1, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 0.0}, {"uid": "ab35cb1ccb4a9ebc", "family": "bind", "template": 3, "depth": 1, "pred": "13", "truth": "13", "correct": true, "edits_per_token": 0.0}, {"uid": "7c739b0506d8ac40", "family": "bind", "template": 3, "depth": 1, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "4e25a4ae3b67d976", "family": "bind", "template": 3, "depth": 1, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 0.0}, {"uid": "22004f16631c31fa", "family": "arith", "template": 4, "depth": 2, "pred": "7", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "aead8eeee264a6dd", "family": "arith", "template": 4, "depth": 2, "pred": "-21", "truth": "-25", "correct": false, "edits_per_token": 0.0}, {"uid": "49a53bd60db6df22", "family": "bind", "template": 3, "depth": 1, "pred": "26", "truth": "24", "correct": false, "edits_per_token": 0.0}, {"uid": "efec1e4e4e19b40b", "family": "arith", "template": 4, "depth": 3, "pred": "-4", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "3e17152a2a3b58df", "family": "arith", "template": 4, "depth": 3, "pred": "42", "truth": "54", "correct": false, "edits_per_token": 0.0}, {"uid": "1d78f810c43669a2", "family": "fsm", "template": 3, "depth": 1, "pred": "3", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "33930d2cc436abac", "family": "arith", "template": 4, "depth": 3, "pred": "31", "truth": "30", "correct": false, "edits_per_token": 0.0}, {"uid": "e4333fd80de54f3c", "family": "arith", "template": 4, "depth": 3, "pred": "1", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "481c38df29183f6c", "family": "arith", "template": 4, "depth": 3, "pred": "22", "truth": "25", "correct": false, "edits_per_token": 0.0}, {"uid": "ef528c3a1e15bb39", "family": "arith", "template": 4, "depth": 3, "pred": "1", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "74899f1ef74b0a4c", "family": "arith", "template": 4, "depth": 3, "pred": "101", "truth": "106", "correct": false, "edits_per_token": 0.0}, {"uid": "70ebe3705ff6ea5f", "family": "arith", "template": 4, "depth": 3, "pred": "483", "truth": "531", "correct": false, "edits_per_token": 0.0}, {"uid": "8a16bdc38d436270", "family": "arith", "template": 4, "depth": 3, "pred": "-14", "truth": "-17", "correct": false, "edits_per_token": 0.0}, {"uid": "0a2078c86d5ac9a7", "family": "arith", "template": 4, "depth": 3, "pred": "90", "truth": "96", "correct": false, "edits_per_token": 0.0}, {"uid": "7384d429e86eb01f", "family": "fsm", "template": 4, "depth": 3, "pred": "20", "truth": "19", "correct": false, "edits_per_token": 0.0}, {"uid": "ed6bb16aadcb518a", "family": "arith", "template": 4, "depth": 4, "pred": "10", "truth": "-14", "correct": false, "edits_per_token": 0.0}, {"uid": "28c1d3a0fbe9487f", "family": "arith", "template": 4, "depth": 4, "pred": "11200", "truth": "2520", "correct": false, "edits_per_token": 0.0}, {"uid": "c517d1504114ad7a", "family": "bind", "template": 4, "depth": 2, "pred": "18", "truth": "15", "correct": false, "edits_per_token": 0.0}, {"uid": "f78d7ec7e37e6779", "family": "bind", "template": 4, "depth": 2, "pred": "280", "truth": "280", "correct": true, "edits_per_token": 0.0}, {"uid": "e544d5e12103f02f", "family": "bind", "template": 4, "depth": 2, "pred": "-18", "truth": "-22", "correct": false, "edits_per_token": 0.0}, {"uid": "ecd2ffe740cc3efc", "family": "bind", "template": 4, "depth": 2, "pred": "75", "truth": "70", "correct": false, "edits_per_token": 0.0}, {"uid": "dcc19f4449b249b1", "family": "bind", "template": 4, "depth": 2, "pred": "16", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "e0606ee0117afdad", "family": "bind", "template": 4, "depth": 2, "pred": "0", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "9a8b7ec79dbf6a8b", "family": "bind", "template": 4, "depth": 2, "pred": "25", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "2c429b7dc30d5227", "family": "bind", "template": 4, "depth": 2, "pred": "22", "truth": "21", "correct": false, "edits_per_token": 0.0}, {"uid": "cd27ac5d6df9a071", "family": "bind", "template": 4, "depth": 2, "pred": "4", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "f49f985044043dbb", "family": "fsm", "template": 3, "depth": 1, "pred": "6", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "ae8404400f627756", "family": "fsm", "template": 3, "depth": 1, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 0.0}, {"uid": "6e13355d1ea33b50", "family": "fsm", "template": 3, "depth": 1, "pred": "10", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "4937d4afa6b30a62", "family": "arith", "template": 4, "depth": 4, "pred": "288", "truth": "287", "correct": false, "edits_per_token": 0.0}, {"uid": "fe3cc19b2abb5508", "family": "arith", "template": 4, "depth": 4, "pred": "36", "truth": "37", "correct": false, "edits_per_token": 0.0}, {"uid": "0568f7f367b4cbff", "family": "arith", "template": 4, "depth": 4, "pred": "280", "truth": "1104", "correct": false, "edits_per_token": 0.0}, {"uid": "3a0bcfe44b9fee3a", "family": "bind", "template": 4, "depth": 2, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "debc0c1111837aef", "family": "bind", "template": 4, "depth": 2, "pred": "28", "truth": "25", "correct": false, "edits_per_token": 0.0}, {"uid": "dc655e2dd9f2365c", "family": "arith", "template": 4, "depth": 4, "pred": "951", "truth": "863", "correct": false, "edits_per_token": 0.0}, {"uid": "983666e948bd6063", "family": "arith", "template": 4, "depth": 4, "pred": "23", "truth": "31", "correct": false, "edits_per_token": 0.0}, {"uid": "9eec9b28436d7847", "family": "arith", "template": 4, "depth": 4, "pred": "66", "truth": "74", "correct": false, "edits_per_token": 0.0}, {"uid": "a3a5173b2df7abf5", "family": "arith", "template": 4, "depth": 4, "pred": "71", "truth": "81", "correct": false, "edits_per_token": 0.0}, {"uid": "41d9923c4340c334", "family": "bind", "template": 3, "depth": 2, "pred": "17", "truth": "17", "correct": true, "edits_per_token": 0.0}, {"uid": "ee9b7f5c7b8f41d7", "family": "fsm", "template": 4, "depth": 2, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "5fe833587ead4d87", "family": "arith", "template": 4, "depth": 4, "pred": "169", "truth": "123", "correct": false, "edits_per_token": 0.0}, {"uid": "f05bbf493d7397b4", "family": "bind", "template": 3, "depth": 2, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "697ac2b97e9fb733", "family": "bind", "template": 3, "depth": 2, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "90c53787b6cd368f", "family": "bind", "template": 3, "depth": 2, "pred": "51", "truth": "57", "correct": false, "edits_per_token": 0.0}, {"uid": "d54be281751cf2d9", "family": "bind", "template": 3, "depth": 2, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "1400d18be0006c2e", "family": "bind", "template": 3, "depth": 2, "pred": "7", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "11aa996915f24160", "family": "bind", "template": 3, "depth": 2, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "1184f7b32425db4e", "family": "fsm", "template": 4, "depth": 2, "pred": "-2", "truth": "-4", "correct": false, "edits_per_token": 0.0}, {"uid": "a3fbb3b73037e7fc", "family": "fsm", "template": 4, "depth": 3, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 0.0}, {"uid": "90a4a1941f06af3e", "family": "fsm", "template": 4, "depth": 3, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "e37479d5b242311d", "family": "fsm", "template": 4, "depth": 3, "pred": "none", "truth": "0", "correct": false, "edits_per_token": 0.0}, {"uid": "0479386e08d5202a", "family": "bind", "template": 3, "depth": 2, "pred": "-10", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "7a9a8bb485ecb944", "family": "bind", "template": 3, "depth": 2, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "ef46f38e2bfbd61e", "family": "bind", "template": 3, "depth": 2, "pred": "28", "truth": "28", "correct": true, "edits_per_token": 0.0}, {"uid": "f15d8d070656a044", "family": "fsm", "template": 4, "depth": 3, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 0.0}, {"uid": "d903d2766fcb00eb", "family": "bind", "template": 4, "depth": 3, "pred": "0", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "92276be46fe4ab30", "family": "bind", "template": 4, "depth": 3, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "e2ccd53faa0a3547", "family": "bind", "template": 4, "depth": 3, "pred": "20", "truth": "26", "correct": false, "edits_per_token": 0.0}, {"uid": "df17dd0c718e4fa4", "family": "bind", "template": 4, "depth": 3, "pred": "-14", "truth": "-11", "correct": false, "edits_per_token": 0.0}, {"uid": "b9820bfcb75201d0", "family": "bind", "template": 4, "depth": 3, "pred": "-16", "truth": "-12", "correct": false, "edits_per_token": 0.0}, {"uid": "0f1e89ceabf587f2", "family": "fsm", "template": 4, "depth": 3, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "96bac0fe597844b9", "family": "bind", "template": 4, "depth": 3, "pred": "76", "truth": "74", "correct": false, "edits_per_token": 0.0}, {"uid": "ba32a49748c6835f", "family": "bind", "template": 4, "depth": 3, "pred": "31", "truth": "30", "correct": false, "edits_per_token": 0.0}, {"uid": "61de92edd07c9f73", "family": "bind", "template": 4, "depth": 3, "pred": "284", "truth": "225", "correct": false, "edits_per_token": 0.0}, {"uid": "ae548ce56bcc7884", "family": "bind", "template": 4, "depth": 3, "pred": "48", "truth": "50", "correct": false, "edits_per_token": 0.0}, {"uid": "a393a7120cee660b", "family": "fsm", "template": 3, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "5a74de8f55ccd4ab", "family": "fsm", "template": 3, "depth": 1, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "ee1421b1d9f21ce7", "family": "fsm", "template": 3, "depth": 1, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "a28f7f5b424eb377", "family": "fsm", "template": 3, "depth": 2, "pred": "6", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "e37c6e22ee7cdf83", "family": "fsm", "template": 3, "depth": 2, "pred": "22", "truth": "24", "correct": false, "edits_per_token": 0.0}, {"uid": "0bd5ea1e3dcb6681", "family": "fsm", "template": 3, "depth": 2, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "a6defeaa073f159e", "family": "fsm", "template": 4, "depth": 4, "pred": "21", "truth": "21", "correct": true, "edits_per_token": 0.0}, {"uid": "17f316431a230f95", "family": "arith", "template": 3, "depth": 1, "pred": "10", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "688087012cf48422", "family": "bind", "template": 4, "depth": 3, "pred": "28", "truth": "35", "correct": false, "edits_per_token": 0.0}, {"uid": "10ed69a16258ca4e", "family": "fsm", "template": 3, "depth": 2, "pred": "4", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "103b828455dbf33c", "family": "fsm", "template": 3, "depth": 2, "pred": "21", "truth": "29", "correct": false, "edits_per_token": 0.0}, {"uid": "0d9f6bf4de9dd849", "family": "fsm", "template": 3, "depth": 2, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "fe253127243bbe69", "family": "arith", "template": 3, "depth": 1, "pred": "30", "truth": "30", "correct": true, "edits_per_token": 0.0}, {"uid": "418c76061f2eaea0", "family": "arith", "template": 3, "depth": 1, "pred": "60", "truth": "60", "correct": true, "edits_per_token": 0.0}, {"uid": "f06a672b579589f4", "family": "arith", "template": 3, "depth": 1, "pred": "20", "truth": "26", "correct": false, "edits_per_token": 0.0}, {"uid": "602e11044971aeb3", "family": "bind", "template": 3, "depth": 3, "pred": "81", "truth": "71", "correct": false, "edits_per_token": 0.0}, {"uid": "27d3b392810a6e03", "family": "arith", "template": 3, "depth": 1, "pred": "22", "truth": "28", "correct": false, "edits_per_token": 0.0}, {"uid": "05dd5abb67e57121", "family": "arith", "template": 3, "depth": 1, "pred": "40", "truth": "41", "correct": false, "edits_per_token": 0.0}, {"uid": "bab6b2ba0bc7ee6d", "family": "arith", "template": 3, "depth": 1, "pred": "-6", "truth": "-10", "correct": false, "edits_per_token": 0.0}, {"uid": "0888a35d47dd9c33", "family": "arith", "template": 3, "depth": 1, "pred": "7", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "50d9b8b5e09e4336", "family": "arith", "template": 3, "depth": 1, "pred": "-14", "truth": "-12", "correct": false, "edits_per_token": 0.0}, {"uid": "df86ed788c82dad6", "family": "arith", "template": 3, "depth": 1, "pred": "-3", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "bc9e03a4d69d9d25", "family": "bind", "template": 3, "depth": 3, "pred": "-10", "truth": "-9", "correct": false, "edits_per_token": 0.0}, {"uid": "bdd4082503abf525", "family": "bind", "template": 3, "depth": 3, "pred": "10", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "1703616b8801e017", "family": "bind", "template": 3, "depth": 3, "pred": "40", "truth": "48", "correct": false, "edits_per_token": 0.0}, {"uid": "f89d93ce767e7d1a", "family": "bind", "template": 3, "depth": 3, "pred": "165", "truth": "123", "correct": false, "edits_per_token": 0.0}, {"uid": "758c9b791db04f9d", "family": "bind", "template": 3, "depth": 3, "pred": "15", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "1485a8c2ccd127a9", "family": "fsm", "template": 4, "depth": 3, "pred": "8", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "314cf8cd1e62d84f", "family": "bind", "template": 3, "depth": 3, "pred": "24", "truth": "23", "correct": false, "edits_per_token": 0.0}, {"uid": "5b4fb5a0bc255002", "family": "bind", "template": 3, "depth": 3, "pred": "16", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "d0f8e1d5fa0aee5d", "family": "arith", "template": 3, "depth": 2, "pred": "16", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "296f1fab9ec37663", "family": "bind", "template": 3, "depth": 3, "pred": "-15", "truth": "-14", "correct": false, "edits_per_token": 0.0}, {"uid": "779fef7e1a0c194c", "family": "arith", "template": 3, "depth": 2, "pred": "50", "truth": "60", "correct": false, "edits_per_token": 0.0}, {"uid": "afec0c621aae8261", "family": "arith", "template": 3, "depth": 2, "pred": "300", "truth": "342", "correct": false, "edits_per_token": 0.0}, {"uid": "1fd1d41e1e4bf549", "family": "arith", "template": 3, "depth": 2, "pred": "30", "truth": "41", "correct": false, "edits_per_token": 0.0}, {"uid": "68ec3f53b05b8a8f", "family": "arith", "template": 3, "depth": 2, "pred": "8", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "2e46a8c4c11a017f", "family": "bind", "template": 3, "depth": 3, "pred": "34", "truth": "36", "correct": false, "edits_per_token": 0.0}, {"uid": "3742c76cef7c48b5", "family": "fsm", "template": 3, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "8f3914e60d8577ca", "family": "fsm", "template": 3, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "4302d428e1f3df93", "family": "arith", "template": 3, "depth": 2, "pred": "-14", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "0b98d2417cc0b84e", "family": "arith", "template": 3, "depth": 2, "pred": "30", "truth": "34", "correct": false, "edits_per_token": 0.0}, {"uid": "f480d24e45533797", "family": "arith", "template": 3, "depth": 2, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "aadf9012cfc9e911", "family": "arith", "template": 3, "depth": 2, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 0.0}, {"uid": "0132fdf42d9298ad", "family": "arith", "template": 3, "depth": 2, "pred": "130", "truth": "170", "correct": false, "edits_per_token": 0.0}, {"uid": "b5fc0c28422e9e56", "family": "sched", "template": 4, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "40ea1bb01295386f", "family": "arith", "template": 3, "depth": 2, "pred": "-30", "truth": "-30", "correct": true, "edits_per_token": 0.0}, {"uid": "09c3fef4d7253107", "family": "fsm", "template": 4, "depth": 3, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "65c150d0ca66cb88", "family": "fsm", "template": 4, "depth": 3, "pred": "6", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "e00e7b260265d9ea", "family": "sched", "template": 4, "depth": 1, "pred": "20", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "b3115967a12393ab", "family": "sched", "template": 4, "depth": 1, "pred": "12", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "23f2db205d8a23de", "family": "arith", "template": 3, "depth": 3, "pred": "12", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "3ff1d1bd3d1311ba", "family": "arith", "template": 3, "depth": 3, "pred": "20", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "19beb79343841fd2", "family": "fsm", "template": 4, "depth": 3, "pred": "-5", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "cc873df2e7df8bed", "family": "arith", "template": 3, "depth": 3, "pred": "10", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "d0326312724fda50", "family": "arith", "template": 3, "depth": 3, "pred": "40", "truth": "51", "correct": false, "edits_per_token": 0.0}, {"uid": "224bd8479cfb9a0c", "family": "arith", "template": 3, "depth": 3, "pred": "45", "truth": "409", "correct": false, "edits_per_token": 0.0}, {"uid": "231a329a8cb3ec7b", "family": "arith", "template": 3, "depth": 3, "pred": "60", "truth": "100", "correct": false, "edits_per_token": 0.0}, {"uid": "6682c6a5479651b2", "family": "bind", "template": 4, "depth": 4, "pred": "18", "truth": "19", "correct": false, "edits_per_token": 0.0}, {"uid": "fd930d8d771290fe", "family": "bind", "template": 4, "depth": 4, "pred": "-18", "truth": "-17", "correct": false, "edits_per_token": 0.0}, {"uid": "9d9b907741fd780b", "family": "bind", "template": 4, "depth": 4, "pred": "38", "truth": "43", "correct": false, "edits_per_token": 0.0}, {"uid": "d19fb0c13b39a42b", "family": "bind", "template": 4, "depth": 4, "pred": "22", "truth": "19", "correct": false, "edits_per_token": 0.0}, {"uid": "618344548ca6da97", "family": "bind", "template": 4, "depth": 4, "pred": "48", "truth": "45", "correct": false, "edits_per_token": 0.0}, {"uid": "20f6f1fda984c6cb", "family": "bind", "template": 4, "depth": 4, "pred": "76", "truth": "76", "correct": true, "edits_per_token": 0.0}, {"uid": "aeebdafd2629497c", "family": "arith", "template": 3, "depth": 3, "pred": "-30", "truth": "-33", "correct": false, "edits_per_token": 0.0}, {"uid": "a44cb3d220440087", "family": "arith", "template": 3, "depth": 3, "pred": "7", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "c11fba9caa3f34db", "family": "arith", "template": 3, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "23a7c0751d154177", "family": "arith", "template": 3, "depth": 3, "pred": "90", "truth": "124", "correct": false, "edits_per_token": 0.0}, {"uid": "bbb473fb8f37e9cc", "family": "bind", "template": 4, "depth": 4, "pred": "3", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "6872e478b699995f", "family": "bind", "template": 3, "depth": 4, "pred": "750", "truth": "450", "correct": false, "edits_per_token": 0.0}, {"uid": "2275fd5dc5422998", "family": "bind", "template": 4, "depth": 4, "pred": "5", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "dc78fd9d13fcf07f", "family": "bind", "template": 4, "depth": 4, "pred": "14", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "f42af3a25020bcdd", "family": "bind", "template": 4, "depth": 4, "pred": "18", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "94126479f3933d69", "family": "fsm", "template": 4, "depth": 4, "pred": "-4", "truth": "0", "correct": false, "edits_per_token": 0.0}, {"uid": "3eb45bde45d94b62", "family": "arith", "template": 3, "depth": 4, "pred": "36", "truth": "192", "correct": false, "edits_per_token": 0.0}, {"uid": "711de0ffe9fc3e1f", "family": "bind", "template": 3, "depth": 4, "pred": "-8", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "ed303359a37cd1ef", "family": "bind", "template": 3, "depth": 4, "pred": "-3", "truth": "-3", "correct": true, "edits_per_token": 0.0}, {"uid": "d66c7948ec115397", "family": "bind", "template": 3, "depth": 4, "pred": "-24", "truth": "-18", "correct": false, "edits_per_token": 0.0}, {"uid": "284e1b08cda8cec0", "family": "arith", "template": 3, "depth": 4, "pred": "-37", "truth": "-89", "correct": false, "edits_per_token": 0.0}, {"uid": "7646e2462cfd7f42", "family": "arith", "template": 3, "depth": 4, "pred": "20", "truth": "30", "correct": false, "edits_per_token": 0.0}, {"uid": "5a3e1f340b7e6a24", "family": "arith", "template": 3, "depth": 4, "pred": "8", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "47f19ffbc80df9f9", "family": "bind", "template": 3, "depth": 4, "pred": "52", "truth": "68", "correct": false, "edits_per_token": 0.0}, {"uid": "a51fb4531b491ec9", "family": "bind", "template": 3, "depth": 4, "pred": "55", "truth": "45", "correct": false, "edits_per_token": 0.0}, {"uid": "15a893f2a205d84e", "family": "fsm", "template": 4, "depth": 4, "pred": "-3", "truth": "-4", "correct": false, "edits_per_token": 0.0}, {"uid": "927fa8211f320a82", "family": "arith", "template": 3, "depth": 4, "pred": "51", "truth": "648", "correct": false, "edits_per_token": 0.0}, {"uid": "0bac982efe5d9b1b", "family": "arith", "template": 3, "depth": 4, "pred": "100", "truth": "121", "correct": false, "edits_per_token": 0.0}, {"uid": "906108b68beb6e66", "family": "arith", "template": 3, "depth": 4, "pred": "-8", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "814724b4fcfab767", "family": "arith", "template": 3, "depth": 4, "pred": "-56", "truth": "-136", "correct": false, "edits_per_token": 0.0}, {"uid": "040cf6134459ea3d", "family": "bind", "template": 3, "depth": 4, "pred": "47", "truth": "53", "correct": false, "edits_per_token": 0.0}, {"uid": "93d2eb4b87dcff06", "family": "bind", "template": 3, "depth": 4, "pred": "-10", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "6e672c37599bb93c", "family": "bind", "template": 3, "depth": 4, "pred": "10", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "d27676130eb57694", "family": "bind", "template": 3, "depth": 4, "pred": "210", "truth": "270", "correct": false, "edits_per_token": 0.0}, {"uid": "a65dce0bbb807f9f", "family": "arith", "template": 3, "depth": 4, "pred": "10", "truth": "-15", "correct": false, "edits_per_token": 0.0}, {"uid": "7279c276a9a815e1", "family": "fsm", "template": 3, "depth": 2, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "32578e3c7c2777ec", "family": "fsm", "template": 3, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "27eadcd0b3e689d6", "family": "arith", "template": 3, "depth": 4, "pred": "20", "truth": "19", "correct": false, "edits_per_token": 0.0}, {"uid": "b1cb2f851393b229", "family": "fsm", "template": 3, "depth": 3, "pred": "1", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "f715224d49251573", "family": "fsm", "template": 3, "depth": 3, "pred": "3", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "462877820fe8108c", "family": "fsm", "template": 4, "depth": 3, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "07a5c651bc9329e9", "family": "fsm", "template": 3, "depth": 3, "pred": "18", "truth": "40", "correct": false, "edits_per_token": 0.0}, {"uid": "826ef35bb8ef7e37", "family": "fsm", "template": 3, "depth": 3, "pred": "16", "truth": "16", "correct": true, "edits_per_token": 0.0}, {"uid": "d4cffd1bb7412e20", "family": "fsm", "template": 4, "depth": 4, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "d0c9dad4288a1b00", "family": "excl", "template": 4, "depth": 1, "pred": "11", "truth": "bean apple", "correct": false, "edits_per_token": 0.0}, {"uid": "865a714fc700aa8e", "family": "fsm", "template": 4, "depth": 4, "pred": "1", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "18cdb11c7bccda13", "family": "fsm", "template": 4, "depth": 4, "pred": "2", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "5bf0970eb4ec32b9", "family": "fsm", "template": 4, "depth": 4, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "361329006ff8d9f5", "family": "fsm", "template": 4, "depth": 4, "pred": "-2", "truth": "-2", "correct": true, "edits_per_token": 0.0}, {"uid": "d9c421488e96f445", "family": "sched", "template": 4, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "c03cc91b48845543", "family": "sched", "template": 4, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "6813628735e51bac", "family": "sched", "template": 4, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "2ccd00a18ae37ca6", "family": "sched", "template": 4, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "c0f7e76d56e30d0d", "family": "fsm", "template": 3, "depth": 2, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "8fbaf8252747c131", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "302d090f441609f2", "family": "sched", "template": 4, "depth": 2, "pred": "4", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "7dd9b367f661c4e7", "family": "fsm", "template": 3, "depth": 3, "pred": "6", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "5e0d345c061fcf1c", "family": "fsm", "template": 3, "depth": 3, "pred": "3", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "9dce772d152e9809", "family": "sched", "template": 4, "depth": 2, "pred": "16", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "0f5849ec46dda95a", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "28be04c444bcbab0", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "47fd20c3a4c07b85", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "7799aebdd3e85a73", "family": "fsm", "template": 3, "depth": 3, "pred": "8", "truth": "14", "correct": false, "edits_per_token": 0.0}, {"uid": "454b6b92d5dcdd20", "family": "fsm", "template": 3, "depth": 3, "pred": "4", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "2c9ea8bc86929ebe", "family": "sched", "template": 4, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "b2e6a268b51a1208", "family": "fsm", "template": 3, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "bb6ca3372f7dff2a", "family": "excl", "template": 4, "depth": 1, "pred": "11", "truth": "yam lime nut", "correct": false, "edits_per_token": 0.0}, {"uid": "6a2a5075397d961d", "family": "excl", "template": 4, "depth": 3, "pred": "nut nut app", "truth": "fig nut", "correct": false, "edits_per_token": 0.0}, {"uid": "7e85eb90c2ac7fc3", "family": "sched", "template": 4, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "6ed6b349d0ef224b", "family": "sched", "template": 4, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "cac36aaa74d1e76a", "family": "fsm", "template": 3, "depth": 4, "pred": "4", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "db28fb2ac93f520d", "family": "fsm", "template": 4, "depth": 4, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 0.0}, {"uid": "8c00cc854149641d", "family": "fsm", "template": 4, "depth": 4, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "7b1cf55cc5cd3e5f", "family": "excl", "template": 4, "depth": 2, "pred": "plum", "truth": "plum lime", "correct": false, "edits_per_token": 0.0}, {"uid": "b7813c6672a55af5", "family": "fsm", "template": 3, "depth": 4, "pred": "-2", "truth": "-5", "correct": false, "edits_per_token": 0.0}, {"uid": "6375fbd1da202d74", "family": "fsm", "template": 3, "depth": 4, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 0.0}, {"uid": "a677f27243a0dbba", "family": "excl", "template": 4, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "714117d0c4d93f5a", "family": "fsm", "template": 3, "depth": 3, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "c9232e7db55d45af", "family": "excl", "template": 4, "depth": 3, "pred": "date", "truth": "yam bean", "correct": false, "edits_per_token": 0.0}, {"uid": "0c9a758e4a02999d", "family": "excl", "template": 4, "depth": 1, "pred": "pear fig", "truth": "lime fig", "correct": false, "edits_per_token": 0.0}, {"uid": "98220b4db365ce22", "family": "fsm", "template": 3, "depth": 4, "pred": "-2", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "e233b1fa3a724f09", "family": "excl", "template": 4, "depth": 2, "pred": "none", "truth": "fig bean", "correct": false, "edits_per_token": 0.0}, {"uid": "c374d2a0d211477c", "family": "excl", "template": 4, "depth": 3, "pred": "date", "truth": "bean", "correct": false, "edits_per_token": 0.0}, {"uid": "b25c8b5fc92452b8", "family": "fsm", "template": 3, "depth": 4, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 0.0}, {"uid": "35afcdba50ef0539", "family": "fsm", "template": 3, "depth": 4, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "74bdd2fe3dd621df", "family": "excl", "template": 4, "depth": 2, "pred": "apple lime", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "7e5ef24472530b54", "family": "sched", "template": 4, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "7b5aa6da844a933a", "family": "sched", "template": 4, "depth": 3, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "961449ff650c677b", "family": "sched", "template": 4, "depth": 3, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "af551a231c42bd0d", "family": "sched", "template": 4, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "933151caa3a0be34", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "231d2a4a23603143", "family": "excl", "template": 4, "depth": 1, "pred": "plum lime f", "truth": "lime bean fig plum", "correct": false, "edits_per_token": 0.0}, {"uid": "f4e87456cd45616d", "family": "excl", "template": 4, "depth": 2, "pred": "lime pear", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "6e23082e8f378986", "family": "excl", "template": 4, "depth": 3, "pred": "plum", "truth": "bean", "correct": false, "edits_per_token": 0.0}, {"uid": "ac21be7ac437d92c", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "a650f55eb430eeac", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "9863e0ba8a116a26", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "f85bed8c86544d8d", "family": "fsm", "template": 3, "depth": 4, "pred": "6", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "f253cae418a07d8f", "family": "excl", "template": 3, "depth": 1, "pred": "none", "truth": "date", "correct": false, "edits_per_token": 0.0}, {"uid": "5765c7bbc9aead9c", "family": "fsm", "template": 3, "depth": 4, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 0.0}, {"uid": "4d2c0dd2cfe3be1c", "family": "excl", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "636a49756779e230", "family": "fsm", "template": 3, "depth": 4, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 0.0}, {"uid": "09460c43e15f2f46", "family": "fsm", "template": 3, "depth": 4, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 0.0}, {"uid": "dd76daa3c9e425c3", "family": "excl", "template": 4, "depth": 1, "pred": "date lime f", "truth": "fig nut lime yam kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "0c9e513614abcdba", "family": "excl", "template": 4, "depth": 1, "pred": "plum apple", "truth": "pear apple yam", "correct": false, "edits_per_token": 0.0}, {"uid": "369d97e8cff94de0", "family": "excl", "template": 4, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "740a7fe786d6b8a9", "family": "sched", "template": 4, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "8fb18ed65388b1e8", "family": "excl", "template": 4, "depth": 4, "pred": "yam", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "29d92bff2c35ea42", "family": "excl", "template": 4, "depth": 4, "pred": "yam", "truth": "lime", "correct": false, "edits_per_token": 0.0}, {"uid": "f8d4d6edfeb5089f", "family": "sched", "template": 4, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "ee81ecf47e6443b2", "family": "sched", "template": 4, "depth": 3, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "0ff325369010cfc0", "family": "sched", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "1af73c5067b38c2c", "family": "excl", "template": 3, "depth": 1, "pred": "fig", "truth": "lime plum apple date", "correct": false, "edits_per_token": 0.0}, {"uid": "78bf6eea827bc579", "family": "excl", "template": 4, "depth": 1, "pred": "plum apple", "truth": "apple yam pear", "correct": false, "edits_per_token": 0.0}, {"uid": "f7efbbd3fc00581b", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "f82b882fabca1cb7", "family": "excl", "template": 4, "depth": 2, "pred": "plum", "truth": "plum apple", "correct": false, "edits_per_token": 0.0}, {"uid": "faf4c370f026c6b8", "family": "sched", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "bfa2044b00bb86b8", "family": "sched", "template": 4, "depth": 3, "pred": "none", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "4d3505f6a55ac312", "family": "excl", "template": 4, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "1a3d40c434f1d4e3", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "b1b1ea9a4212d1a9", "family": "sched", "template": 4, "depth": 3, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "8c69b9bd3183ebb6", "family": "excl", "template": 3, "depth": 1, "pred": "none", "truth": "pear bean kiwi plum", "correct": false, "edits_per_token": 0.0}, {"uid": "10dc1311d193c3fa", "family": "excl", "template": 4, "depth": 1, "pred": "lime apple", "truth": "lime apple plum bean", "correct": false, "edits_per_token": 0.0}, {"uid": "82d6d862fe063960", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "11b5887e2e23a957", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "ef07c6682c4275d9", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "3335c404ce9bfcd5", "family": "excl", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "a6c279c43201f057", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "ede73a38c44a3ccf", "family": "sched", "template": 4, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "3e0de6f98d396f2a", "family": "excl", "template": 4, "depth": 1, "pred": "pear fig li", "truth": "kiwi pear bean apple", "correct": false, "edits_per_token": 0.0}, {"uid": "d36a0f43f40b6236", "family": "excl", "template": 4, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "ebb9ef7d9d066d95", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "4020752baa5a79b3", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "70e7b96a192ab155", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "ae63f7a481d76063", "family": "excl", "template": 4, "depth": 4, "pred": "none", "truth": "bean lime", "correct": false, "edits_per_token": 0.0}, {"uid": "f3e43a7b75ad9689", "family": "excl", "template": 3, "depth": 1, "pred": "date", "truth": "pear fig bean lime", "correct": false, "edits_per_token": 0.0}, {"uid": "586e75ebfbf62207", "family": "excl", "template": 4, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "9b4fed98a3e64c92", "family": "sched", "template": 3, "depth": 2, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "694f4323afee9216", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "37c09dc4c77fb59b", "family": "excl", "template": 4, "depth": 1, "pred": "apple bean", "truth": "apple", "correct": false, "edits_per_token": 0.0}, {"uid": "27852e223cde9fdd", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "d9a3a976635a4912", "family": "excl", "template": 3, "depth": 1, "pred": "nut", "truth": "kiwi bean date", "correct": false, "edits_per_token": 0.0}, {"uid": "5f308fa3d9d64764", "family": "excl", "template": 3, "depth": 1, "pred": "none", "truth": "lime fig yam", "correct": false, "edits_per_token": 0.0}, {"uid": "796295c79426c62a", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "pear date", "correct": false, "edits_per_token": 0.0}, {"uid": "f59adc461a2a3a23", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "pear fig", "correct": false, "edits_per_token": 0.0}, {"uid": "cb1f068a9f416569", "family": "excl", "template": 4, "depth": 4, "pred": "plum", "truth": "yam", "correct": false, "edits_per_token": 0.0}, {"uid": "08d7338346473ac3", "family": "excl", "template": 3, "depth": 2, "pred": "fig", "truth": "pear bean", "correct": false, "edits_per_token": 0.0}, {"uid": "a74bbd4fb82ecde6", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "apple plum", "correct": false, "edits_per_token": 0.0}, {"uid": "023c777eb4956132", "family": "excl", "template": 4, "depth": 3, "pred": "date", "truth": "date pear", "correct": false, "edits_per_token": 0.0}, {"uid": "8aa42a0a282b8980", "family": "excl", "template": 3, "depth": 2, "pred": "apple", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "53637f76c85d8c1c", "family": "excl", "template": 3, "depth": 2, "pred": "apple", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "d0cf89610f223fed", "family": "sched", "template": 4, "depth": 4, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "9c6fbac6d438fea4", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "38b1c41b753402e8", "family": "excl", "template": 4, "depth": 2, "pred": "plum", "truth": "plum yam nut", "correct": false, "edits_per_token": 0.0}, {"uid": "3e375d575f2a30b3", "family": "sched", "template": 4, "depth": 4, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "40e7271c0285929b", "family": "sched", "template": 4, "depth": 4, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "f62ddd4631b1e632", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "9b8f8f5e9e9a3a18", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "f33bce8dab09faee", "family": "excl", "template": 3, "depth": 3, "pred": "nut", "truth": "yam date", "correct": false, "edits_per_token": 0.0}, {"uid": "f7d02c5b717f1b26", "family": "excl", "template": 3, "depth": 4, "pred": "apple", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "2d9756e2a9606211", "family": "excl", "template": 4, "depth": 2, "pred": "bean yam ki", "truth": "bean date yam", "correct": false, "edits_per_token": 0.0}, {"uid": "a64b960f3eeb5d39", "family": "excl", "template": 4, "depth": 3, "pred": "none", "truth": "lime", "correct": false, "edits_per_token": 0.0}, {"uid": "75be04c29ab1f001", "family": "sched", "template": 3, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "7d09a16f57550745", "family": "sched", "template": 3, "depth": 2, "pred": "C", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "5048ccc397f05377", "family": "sched", "template": 3, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "bce8b50e0845842d", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "66b605e4db85a2c0", "family": "sched", "template": 4, "depth": 4, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "47a6146933016e02", "family": "excl", "template": 4, "depth": 2, "pred": "kiwi nut", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "7603bb4fbb4710c8", "family": "excl", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "2f57bbfc2ad4b319", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "f93d17df47bca1c9", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "4545d7fc45e828c0", "family": "sched", "template": 3, "depth": 3, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "f782173e21a6c1f2", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "80a37124c4ed9a39", "family": "excl", "template": 3, "depth": 1, "pred": "nut", "truth": "bean apple pear", "correct": false, "edits_per_token": 0.0}, {"uid": "bcce29e090c586d2", "family": "sched", "template": 3, "depth": 3, "pred": "16", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "193a55393141bab5", "family": "sched", "template": 3, "depth": 3, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "0324c41dffaa1cb8", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "087a279916282ea3", "family": "excl", "template": 4, "depth": 4, "pred": "date", "truth": "nut kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "9b2c677e487c27cb", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "fd1c02003b3d4554", "family": "sched", "template": 3, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "7df915ca06235913", "family": "excl", "template": 3, "depth": 2, "pred": "fig", "truth": "yam", "correct": false, "edits_per_token": 0.0}, {"uid": "1e4798197a754ca9", "family": "excl", "template": 4, "depth": 4, "pred": "um", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "2ca156e74b515e97", "family": "sched", "template": 3, "depth": 2, "pred": "C", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "6a2203bca865b619", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "823e185482c10eea", "family": "excl", "template": 3, "depth": 1, "pred": "date", "truth": "pear fig kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "41e74f93ba171d40", "family": "excl", "template": 4, "depth": 4, "pred": "none", "truth": "bean lime plum", "correct": false, "edits_per_token": 0.0}, {"uid": "57edf49d2957d214", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "nut apple date", "correct": false, "edits_per_token": 0.0}, {"uid": "808bfdae57495704", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "nut pear", "correct": false, "edits_per_token": 0.0}, {"uid": "38d379de426c6ae0", "family": "sched", "template": 3, "depth": 2, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "0b61a24242db3698", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "date", "correct": false, "edits_per_token": 0.0}, {"uid": "9cad194fa2e865be", "family": "excl", "template": 3, "depth": 1, "pred": "fig nut lim", "truth": "fig apple yam lime bean plum", "correct": false, "edits_per_token": 0.0}, {"uid": "16a2b0a2dbc217f0", "family": "excl", "template": 3, "depth": 1, "pred": "plum", "truth": "yam lime apple kiwi nut", "correct": false, "edits_per_token": 0.0}, {"uid": "302f7dfdbcebf7cf", "family": "excl", "template": 4, "depth": 4, "pred": "kiwi", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "88ac831d2cd557d9", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "0ea0157cec29a240", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "yam fig plum", "correct": false, "edits_per_token": 0.0}, {"uid": "49165e12e4d4c8d3", "family": "excl", "template": 3, "depth": 3, "pred": "date", "truth": "date bean kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "575bd19bd31d0b07", "family": "excl", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "3c98968bad82b469", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "fa3313c7eee9031c", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "c5b0ad7be361e95a", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "6bfdc71318271303", "family": "sched", "template": 3, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "0598403107623618", "family": "sched", "template": 3, "depth": 4, "pred": "none", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "28e4acdd7b79568a", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "38724d20f4d85bad", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "7c8636ccbdad97eb", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "9bd2514eedc4b4e1", "family": "excl", "template": 3, "depth": 3, "pred": "plum", "truth": "nut plum yam date fig", "correct": false, "edits_per_token": 0.0}, {"uid": "5a5831a28a079558", "family": "excl", "template": 3, "depth": 3, "pred": "plum", "truth": "plum apple fig kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "8ef030fff471d972", "family": "sched", "template": 3, "depth": 3, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "484f0cbf9521eb48", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "fdbe9a26369f78cb", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "apple bean pear", "correct": false, "edits_per_token": 0.0}, {"uid": "f2667067c94bd9c2", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "fig", "correct": false, "edits_per_token": 0.0}, {"uid": "636be724170a7508", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "kiwi plum pear yam apple", "correct": false, "edits_per_token": 0.0}, {"uid": "ed83ed43ad98583d", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "0263314c0f8bd6e3", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "9ae2d71807ac60ce", "family": "sched", "template": 3, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "6a39bb66c31d0c07", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "445a2e8e308d084c", "family": "sched", "template": 3, "depth": 4, "pred": "A", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "b54ac2bc6f7b24df", "family": "sched", "template": 3, "depth": 4, "pred": "D", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "20c1afb381170175", "family": "sched", "template": 3, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 0.0}, {"uid": "30430c28d192efc6", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "e51b5857bbb2ca50", "family": "sched", "template": 3, "depth": 4, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "5b90d13667067fff", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "e98454b7446dc760", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "date lime kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "933b4b87ec87c644", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "98b379559d3421a8", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 0.0}] | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_template_b2.json
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +[{"uid": "3b112da5fd01aae7", "family": "fsm", "template": 4, "depth": 1, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "d5c0c39e453dcca1", "family": "fsm", "template": 4, "depth": 1, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "edd44a1f63841ef0", "family": "fsm", "template": 4, "depth": 1, "pred": "15", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "4a60e827734a2aad", "family": "fsm", "template": 4, "depth": 1, "pred": "13", "truth": "13", "correct": true, "edits_per_token": 2.0}, {"uid": "d2d4172d0d7eda08", "family": "fsm", "template": 4, "depth": 1, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 2.0}, {"uid": "033ec88b5e210826", "family": "fsm", "template": 4, "depth": 1, "pred": "14", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "97fc165388a1bca6", "family": "fsm", "template": 4, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "ee578f5db025d5f9", "family": "fsm", "template": 4, "depth": 1, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 2.0}, {"uid": "c396858313b1867d", "family": "fsm", "template": 4, "depth": 2, "pred": "20", "truth": "20", "correct": true, "edits_per_token": 2.0}, {"uid": "62686c3fb5450969", "family": "bind", "template": 4, "depth": 1, "pred": "9", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "160deb3944fa94a9", "family": "fsm", "template": 4, "depth": 2, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 2.0}, {"uid": "ff167bdae9dae304", "family": "fsm", "template": 4, "depth": 2, "pred": "1", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "6e2f978b50da11db", "family": "fsm", "template": 4, "depth": 2, "pred": "-1", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "b3c197791022cdda", "family": "fsm", "template": 4, "depth": 2, "pred": "-1", "truth": "0", "correct": false, "edits_per_token": 2.0}, {"uid": "c6bc5deb98b3f6a4", "family": "fsm", "template": 4, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "07c9ad219cedeecc", "family": "fsm", "template": 4, "depth": 2, "pred": "1", "truth": "0", "correct": false, "edits_per_token": 2.0}, {"uid": "16a3e4051536fc5a", "family": "arith", "template": 4, "depth": 1, "pred": "-15", "truth": "-16", "correct": false, "edits_per_token": 2.0}, {"uid": "4f1f75fe235ae841", "family": "arith", "template": 4, "depth": 1, "pred": "-21", "truth": "-21", "correct": true, "edits_per_token": 2.0}, {"uid": "cd785088c9149dfb", "family": "arith", "template": 4, "depth": 1, "pred": "164", "truth": "162", "correct": false, "edits_per_token": 2.0}, {"uid": "1638c5c5115e7382", "family": "arith", "template": 4, "depth": 1, "pred": "51", "truth": "48", "correct": false, "edits_per_token": 2.0}, {"uid": "bb572994a445bdb5", "family": "arith", "template": 4, "depth": 1, "pred": "22", "truth": "21", "correct": false, "edits_per_token": 2.0}, {"uid": "ec5611b48356ecc9", "family": "arith", "template": 4, "depth": 1, "pred": "48", "truth": "42", "correct": false, "edits_per_token": 2.0}, {"uid": "f4a95344e3c4417f", "family": "arith", "template": 4, "depth": 1, "pred": "40", "truth": "30", "correct": false, "edits_per_token": 2.0}, {"uid": "e4af4779733a0b3a", "family": "arith", "template": 4, "depth": 1, "pred": "-18", "truth": "-19", "correct": false, "edits_per_token": 2.0}, {"uid": "7e8ac8f95805f2f3", "family": "arith", "template": 4, "depth": 1, "pred": "27", "truth": "29", "correct": false, "edits_per_token": 2.0}, {"uid": "036f0619e07338fd", "family": "bind", "template": 4, "depth": 1, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "e811bf68f0bfdded", "family": "bind", "template": 4, "depth": 1, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 2.0}, {"uid": "82f1818aa73ed36d", "family": "bind", "template": 4, "depth": 1, "pred": "14", "truth": "12", "correct": false, "edits_per_token": 2.0}, {"uid": "f4d0e013d4462dd0", "family": "bind", "template": 4, "depth": 1, "pred": "8", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "4dd12c9283c1639d", "family": "arith", "template": 4, "depth": 1, "pred": "38", "truth": "37", "correct": false, "edits_per_token": 2.0}, {"uid": "889301436be7278d", "family": "bind", "template": 4, "depth": 1, "pred": "56", "truth": "45", "correct": false, "edits_per_token": 2.0}, {"uid": "34157746c6a9aa26", "family": "bind", "template": 4, "depth": 1, "pred": "36", "truth": "30", "correct": false, "edits_per_token": 2.0}, {"uid": "972d347d02c3e4c9", "family": "bind", "template": 4, "depth": 1, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 2.0}, {"uid": "73358f5d5d68a323", "family": "bind", "template": 4, "depth": 1, "pred": "14", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "621a81bcb0f3a1da", "family": "bind", "template": 4, "depth": 1, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 2.0}, {"uid": "be55eff500fd9de4", "family": "arith", "template": 4, "depth": 2, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 2.0}, {"uid": "8086e5fd466e56e3", "family": "arith", "template": 4, "depth": 2, "pred": "21", "truth": "23", "correct": false, "edits_per_token": 2.0}, {"uid": "1d3e8e419ebffaea", "family": "bind", "template": 3, "depth": 1, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 2.0}, {"uid": "03afb70c0ecdf774", "family": "fsm", "template": 4, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "53768ffc251ff5ea", "family": "arith", "template": 4, "depth": 2, "pred": "56", "truth": "56", "correct": true, "edits_per_token": 2.0}, {"uid": "856912fe4bd92aab", "family": "arith", "template": 4, "depth": 2, "pred": "720", "truth": "576", "correct": false, "edits_per_token": 2.0}, {"uid": "bcfa81650916d43a", "family": "arith", "template": 4, "depth": 2, "pred": "28", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "636ae00985bb6e39", "family": "bind", "template": 3, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "866b93f2b1cebb8a", "family": "bind", "template": 3, "depth": 1, "pred": "36", "truth": "33", "correct": false, "edits_per_token": 2.0}, {"uid": "2449760460938f84", "family": "bind", "template": 3, "depth": 1, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 2.0}, {"uid": "664b188b12b5afc4", "family": "arith", "template": 4, "depth": 2, "pred": "23", "truth": "27", "correct": false, "edits_per_token": 2.0}, {"uid": "efba878fb6ac5fa8", "family": "arith", "template": 4, "depth": 2, "pred": "0", "truth": "-36", "correct": false, "edits_per_token": 2.0}, {"uid": "149112577b7a6b0b", "family": "arith", "template": 4, "depth": 2, "pred": "-12", "truth": "-13", "correct": false, "edits_per_token": 2.0}, {"uid": "4f926ee35fd6594e", "family": "bind", "template": 3, "depth": 1, "pred": "14", "truth": "12", "correct": false, "edits_per_token": 2.0}, {"uid": "1432cede89ff82db", "family": "bind", "template": 3, "depth": 1, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 2.0}, {"uid": "ab35cb1ccb4a9ebc", "family": "bind", "template": 3, "depth": 1, "pred": "13", "truth": "13", "correct": true, "edits_per_token": 2.0}, {"uid": "7c739b0506d8ac40", "family": "bind", "template": 3, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "4e25a4ae3b67d976", "family": "bind", "template": 3, "depth": 1, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 2.0}, {"uid": "22004f16631c31fa", "family": "arith", "template": 4, "depth": 2, "pred": "7", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "aead8eeee264a6dd", "family": "arith", "template": 4, "depth": 2, "pred": "-21", "truth": "-25", "correct": false, "edits_per_token": 2.0}, {"uid": "49a53bd60db6df22", "family": "bind", "template": 3, "depth": 1, "pred": "26", "truth": "24", "correct": false, "edits_per_token": 2.0}, {"uid": "efec1e4e4e19b40b", "family": "arith", "template": 4, "depth": 3, "pred": "-4", "truth": "-8", "correct": false, "edits_per_token": 2.0}, {"uid": "3e17152a2a3b58df", "family": "arith", "template": 4, "depth": 3, "pred": "42", "truth": "54", "correct": false, "edits_per_token": 2.0}, {"uid": "1d78f810c43669a2", "family": "fsm", "template": 3, "depth": 1, "pred": "3", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "33930d2cc436abac", "family": "arith", "template": 4, "depth": 3, "pred": "31", "truth": "30", "correct": false, "edits_per_token": 2.0}, {"uid": "e4333fd80de54f3c", "family": "arith", "template": 4, "depth": 3, "pred": "1", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "481c38df29183f6c", "family": "arith", "template": 4, "depth": 3, "pred": "23", "truth": "25", "correct": false, "edits_per_token": 2.0}, {"uid": "ef528c3a1e15bb39", "family": "arith", "template": 4, "depth": 3, "pred": "1", "truth": "-1", "correct": false, "edits_per_token": 2.0}, {"uid": "74899f1ef74b0a4c", "family": "arith", "template": 4, "depth": 3, "pred": "101", "truth": "106", "correct": false, "edits_per_token": 2.0}, {"uid": "70ebe3705ff6ea5f", "family": "arith", "template": 4, "depth": 3, "pred": "483", "truth": "531", "correct": false, "edits_per_token": 2.0}, {"uid": "8a16bdc38d436270", "family": "arith", "template": 4, "depth": 3, "pred": "-14", "truth": "-17", "correct": false, "edits_per_token": 2.0}, {"uid": "0a2078c86d5ac9a7", "family": "arith", "template": 4, "depth": 3, "pred": "90", "truth": "96", "correct": false, "edits_per_token": 2.0}, {"uid": "7384d429e86eb01f", "family": "fsm", "template": 4, "depth": 3, "pred": "20", "truth": "19", "correct": false, "edits_per_token": 2.0}, {"uid": "ed6bb16aadcb518a", "family": "arith", "template": 4, "depth": 4, "pred": "10", "truth": "-14", "correct": false, "edits_per_token": 2.0}, {"uid": "28c1d3a0fbe9487f", "family": "arith", "template": 4, "depth": 4, "pred": "1480", "truth": "2520", "correct": false, "edits_per_token": 2.0}, {"uid": "c517d1504114ad7a", "family": "bind", "template": 4, "depth": 2, "pred": "18", "truth": "15", "correct": false, "edits_per_token": 2.0}, {"uid": "f78d7ec7e37e6779", "family": "bind", "template": 4, "depth": 2, "pred": "280", "truth": "280", "correct": true, "edits_per_token": 2.0}, {"uid": "e544d5e12103f02f", "family": "bind", "template": 4, "depth": 2, "pred": "-21", "truth": "-22", "correct": false, "edits_per_token": 2.0}, {"uid": "ecd2ffe740cc3efc", "family": "bind", "template": 4, "depth": 2, "pred": "75", "truth": "70", "correct": false, "edits_per_token": 2.0}, {"uid": "dcc19f4449b249b1", "family": "bind", "template": 4, "depth": 2, "pred": "15", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "e0606ee0117afdad", "family": "bind", "template": 4, "depth": 2, "pred": "0", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "9a8b7ec79dbf6a8b", "family": "bind", "template": 4, "depth": 2, "pred": "25", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "2c429b7dc30d5227", "family": "bind", "template": 4, "depth": 2, "pred": "22", "truth": "21", "correct": false, "edits_per_token": 2.0}, {"uid": "cd27ac5d6df9a071", "family": "bind", "template": 4, "depth": 2, "pred": "4", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "f49f985044043dbb", "family": "fsm", "template": 3, "depth": 1, "pred": "6", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "ae8404400f627756", "family": "fsm", "template": 3, "depth": 1, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 2.0}, {"uid": "6e13355d1ea33b50", "family": "fsm", "template": 3, "depth": 1, "pred": "10", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "4937d4afa6b30a62", "family": "arith", "template": 4, "depth": 4, "pred": "288", "truth": "287", "correct": false, "edits_per_token": 2.0}, {"uid": "fe3cc19b2abb5508", "family": "arith", "template": 4, "depth": 4, "pred": "36", "truth": "37", "correct": false, "edits_per_token": 2.0}, {"uid": "0568f7f367b4cbff", "family": "arith", "template": 4, "depth": 4, "pred": "280", "truth": "1104", "correct": false, "edits_per_token": 2.0}, {"uid": "3a0bcfe44b9fee3a", "family": "bind", "template": 4, "depth": 2, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "debc0c1111837aef", "family": "bind", "template": 4, "depth": 2, "pred": "28", "truth": "25", "correct": false, "edits_per_token": 2.0}, {"uid": "dc655e2dd9f2365c", "family": "arith", "template": 4, "depth": 4, "pred": "951", "truth": "863", "correct": false, "edits_per_token": 2.0}, {"uid": "983666e948bd6063", "family": "arith", "template": 4, "depth": 4, "pred": "23", "truth": "31", "correct": false, "edits_per_token": 2.0}, {"uid": "9eec9b28436d7847", "family": "arith", "template": 4, "depth": 4, "pred": "68", "truth": "74", "correct": false, "edits_per_token": 2.0}, {"uid": "a3a5173b2df7abf5", "family": "arith", "template": 4, "depth": 4, "pred": "71", "truth": "81", "correct": false, "edits_per_token": 2.0}, {"uid": "41d9923c4340c334", "family": "bind", "template": 3, "depth": 2, "pred": "17", "truth": "17", "correct": true, "edits_per_token": 2.0}, {"uid": "ee9b7f5c7b8f41d7", "family": "fsm", "template": 4, "depth": 2, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "5fe833587ead4d87", "family": "arith", "template": 4, "depth": 4, "pred": "169", "truth": "123", "correct": false, "edits_per_token": 2.0}, {"uid": "f05bbf493d7397b4", "family": "bind", "template": 3, "depth": 2, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "697ac2b97e9fb733", "family": "bind", "template": 3, "depth": 2, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "90c53787b6cd368f", "family": "bind", "template": 3, "depth": 2, "pred": "51", "truth": "57", "correct": false, "edits_per_token": 2.0}, {"uid": "d54be281751cf2d9", "family": "bind", "template": 3, "depth": 2, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 2.0}, {"uid": "1400d18be0006c2e", "family": "bind", "template": 3, "depth": 2, "pred": "7", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "11aa996915f24160", "family": "bind", "template": 3, "depth": 2, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 2.0}, {"uid": "1184f7b32425db4e", "family": "fsm", "template": 4, "depth": 2, "pred": "-2", "truth": "-4", "correct": false, "edits_per_token": 2.0}, {"uid": "a3fbb3b73037e7fc", "family": "fsm", "template": 4, "depth": 3, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "90a4a1941f06af3e", "family": "fsm", "template": 4, "depth": 3, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 2.0}, {"uid": "e37479d5b242311d", "family": "fsm", "template": 4, "depth": 3, "pred": "none", "truth": "0", "correct": false, "edits_per_token": 2.0}, {"uid": "0479386e08d5202a", "family": "bind", "template": 3, "depth": 2, "pred": "-10", "truth": "-8", "correct": false, "edits_per_token": 2.0}, {"uid": "7a9a8bb485ecb944", "family": "bind", "template": 3, "depth": 2, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "ef46f38e2bfbd61e", "family": "bind", "template": 3, "depth": 2, "pred": "28", "truth": "28", "correct": true, "edits_per_token": 2.0}, {"uid": "f15d8d070656a044", "family": "fsm", "template": 4, "depth": 3, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 2.0}, {"uid": "d903d2766fcb00eb", "family": "bind", "template": 4, "depth": 3, "pred": "0", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "92276be46fe4ab30", "family": "bind", "template": 4, "depth": 3, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 2.0}, {"uid": "e2ccd53faa0a3547", "family": "bind", "template": 4, "depth": 3, "pred": "20", "truth": "26", "correct": false, "edits_per_token": 2.0}, {"uid": "df17dd0c718e4fa4", "family": "bind", "template": 4, "depth": 3, "pred": "-14", "truth": "-11", "correct": false, "edits_per_token": 2.0}, {"uid": "b9820bfcb75201d0", "family": "bind", "template": 4, "depth": 3, "pred": "-16", "truth": "-12", "correct": false, "edits_per_token": 2.0}, {"uid": "0f1e89ceabf587f2", "family": "fsm", "template": 4, "depth": 3, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "96bac0fe597844b9", "family": "bind", "template": 4, "depth": 3, "pred": "76", "truth": "74", "correct": false, "edits_per_token": 2.0}, {"uid": "ba32a49748c6835f", "family": "bind", "template": 4, "depth": 3, "pred": "31", "truth": "30", "correct": false, "edits_per_token": 2.0}, {"uid": "61de92edd07c9f73", "family": "bind", "template": 4, "depth": 3, "pred": "284", "truth": "225", "correct": false, "edits_per_token": 2.0}, {"uid": "ae548ce56bcc7884", "family": "bind", "template": 4, "depth": 3, "pred": "48", "truth": "50", "correct": false, "edits_per_token": 2.0}, {"uid": "a393a7120cee660b", "family": "fsm", "template": 3, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 2.0}, {"uid": "5a74de8f55ccd4ab", "family": "fsm", "template": 3, "depth": 1, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "ee1421b1d9f21ce7", "family": "fsm", "template": 3, "depth": 1, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "a28f7f5b424eb377", "family": "fsm", "template": 3, "depth": 2, "pred": "6", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "e37c6e22ee7cdf83", "family": "fsm", "template": 3, "depth": 2, "pred": "22", "truth": "24", "correct": false, "edits_per_token": 2.0}, {"uid": "0bd5ea1e3dcb6681", "family": "fsm", "template": 3, "depth": 2, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "a6defeaa073f159e", "family": "fsm", "template": 4, "depth": 4, "pred": "21", "truth": "21", "correct": true, "edits_per_token": 2.0}, {"uid": "17f316431a230f95", "family": "arith", "template": 3, "depth": 1, "pred": "10", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "688087012cf48422", "family": "bind", "template": 4, "depth": 3, "pred": "28", "truth": "35", "correct": false, "edits_per_token": 2.0}, {"uid": "10ed69a16258ca4e", "family": "fsm", "template": 3, "depth": 2, "pred": "4", "truth": "7", "correct": false, "edits_per_token": 2.0}, {"uid": "103b828455dbf33c", "family": "fsm", "template": 3, "depth": 2, "pred": "21", "truth": "29", "correct": false, "edits_per_token": 2.0}, {"uid": "0d9f6bf4de9dd849", "family": "fsm", "template": 3, "depth": 2, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "fe253127243bbe69", "family": "arith", "template": 3, "depth": 1, "pred": "30", "truth": "30", "correct": true, "edits_per_token": 2.0}, {"uid": "418c76061f2eaea0", "family": "arith", "template": 3, "depth": 1, "pred": "54", "truth": "60", "correct": false, "edits_per_token": 2.0}, {"uid": "f06a672b579589f4", "family": "arith", "template": 3, "depth": 1, "pred": "20", "truth": "26", "correct": false, "edits_per_token": 2.0}, {"uid": "602e11044971aeb3", "family": "bind", "template": 3, "depth": 3, "pred": "81", "truth": "71", "correct": false, "edits_per_token": 2.0}, {"uid": "27d3b392810a6e03", "family": "arith", "template": 3, "depth": 1, "pred": "22", "truth": "28", "correct": false, "edits_per_token": 2.0}, {"uid": "05dd5abb67e57121", "family": "arith", "template": 3, "depth": 1, "pred": "40", "truth": "41", "correct": false, "edits_per_token": 2.0}, {"uid": "bab6b2ba0bc7ee6d", "family": "arith", "template": 3, "depth": 1, "pred": "-6", "truth": "-10", "correct": false, "edits_per_token": 2.0}, {"uid": "0888a35d47dd9c33", "family": "arith", "template": 3, "depth": 1, "pred": "7", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "50d9b8b5e09e4336", "family": "arith", "template": 3, "depth": 1, "pred": "-14", "truth": "-12", "correct": false, "edits_per_token": 2.0}, {"uid": "df86ed788c82dad6", "family": "arith", "template": 3, "depth": 1, "pred": "-2", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "bc9e03a4d69d9d25", "family": "bind", "template": 3, "depth": 3, "pred": "-10", "truth": "-9", "correct": false, "edits_per_token": 2.0}, {"uid": "bdd4082503abf525", "family": "bind", "template": 3, "depth": 3, "pred": "10", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "1703616b8801e017", "family": "bind", "template": 3, "depth": 3, "pred": "40", "truth": "48", "correct": false, "edits_per_token": 2.0}, {"uid": "f89d93ce767e7d1a", "family": "bind", "template": 3, "depth": 3, "pred": "180", "truth": "123", "correct": false, "edits_per_token": 2.0}, {"uid": "758c9b791db04f9d", "family": "bind", "template": 3, "depth": 3, "pred": "14", "truth": "14", "correct": true, "edits_per_token": 2.0}, {"uid": "1485a8c2ccd127a9", "family": "fsm", "template": 4, "depth": 3, "pred": "8", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "314cf8cd1e62d84f", "family": "bind", "template": 3, "depth": 3, "pred": "24", "truth": "23", "correct": false, "edits_per_token": 2.0}, {"uid": "5b4fb5a0bc255002", "family": "bind", "template": 3, "depth": 3, "pred": "17", "truth": "17", "correct": true, "edits_per_token": 2.0}, {"uid": "d0f8e1d5fa0aee5d", "family": "arith", "template": 3, "depth": 2, "pred": "16", "truth": "20", "correct": false, "edits_per_token": 2.0}, {"uid": "296f1fab9ec37663", "family": "bind", "template": 3, "depth": 3, "pred": "-18", "truth": "-14", "correct": false, "edits_per_token": 2.0}, {"uid": "779fef7e1a0c194c", "family": "arith", "template": 3, "depth": 2, "pred": "50", "truth": "60", "correct": false, "edits_per_token": 2.0}, {"uid": "afec0c621aae8261", "family": "arith", "template": 3, "depth": 2, "pred": "300", "truth": "342", "correct": false, "edits_per_token": 2.0}, {"uid": "1fd1d41e1e4bf549", "family": "arith", "template": 3, "depth": 2, "pred": "30", "truth": "41", "correct": false, "edits_per_token": 2.0}, {"uid": "68ec3f53b05b8a8f", "family": "arith", "template": 3, "depth": 2, "pred": "8", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "2e46a8c4c11a017f", "family": "bind", "template": 3, "depth": 3, "pred": "34", "truth": "36", "correct": false, "edits_per_token": 2.0}, {"uid": "3742c76cef7c48b5", "family": "fsm", "template": 3, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "8f3914e60d8577ca", "family": "fsm", "template": 3, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 2.0}, {"uid": "4302d428e1f3df93", "family": "arith", "template": 3, "depth": 2, "pred": "-14", "truth": "-13", "correct": false, "edits_per_token": 2.0}, {"uid": "0b98d2417cc0b84e", "family": "arith", "template": 3, "depth": 2, "pred": "30", "truth": "34", "correct": false, "edits_per_token": 2.0}, {"uid": "f480d24e45533797", "family": "arith", "template": 3, "depth": 2, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "aadf9012cfc9e911", "family": "arith", "template": 3, "depth": 2, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 2.0}, {"uid": "0132fdf42d9298ad", "family": "arith", "template": 3, "depth": 2, "pred": "130", "truth": "170", "correct": false, "edits_per_token": 2.0}, {"uid": "b5fc0c28422e9e56", "family": "sched", "template": 4, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "40ea1bb01295386f", "family": "arith", "template": 3, "depth": 2, "pred": "-30", "truth": "-30", "correct": true, "edits_per_token": 2.0}, {"uid": "09c3fef4d7253107", "family": "fsm", "template": 4, "depth": 3, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "65c150d0ca66cb88", "family": "fsm", "template": 4, "depth": 3, "pred": "6", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "e00e7b260265d9ea", "family": "sched", "template": 4, "depth": 1, "pred": "20", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "b3115967a12393ab", "family": "sched", "template": 4, "depth": 1, "pred": "12", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "23f2db205d8a23de", "family": "arith", "template": 3, "depth": 3, "pred": "12", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "3ff1d1bd3d1311ba", "family": "arith", "template": 3, "depth": 3, "pred": "20", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "19beb79343841fd2", "family": "fsm", "template": 4, "depth": 3, "pred": "-5", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "cc873df2e7df8bed", "family": "arith", "template": 3, "depth": 3, "pred": "10", "truth": "18", "correct": false, "edits_per_token": 2.0}, {"uid": "d0326312724fda50", "family": "arith", "template": 3, "depth": 3, "pred": "40", "truth": "51", "correct": false, "edits_per_token": 2.0}, {"uid": "224bd8479cfb9a0c", "family": "arith", "template": 3, "depth": 3, "pred": "45", "truth": "409", "correct": false, "edits_per_token": 2.0}, {"uid": "231a329a8cb3ec7b", "family": "arith", "template": 3, "depth": 3, "pred": "60", "truth": "100", "correct": false, "edits_per_token": 2.0}, {"uid": "6682c6a5479651b2", "family": "bind", "template": 4, "depth": 4, "pred": "18", "truth": "19", "correct": false, "edits_per_token": 2.0}, {"uid": "fd930d8d771290fe", "family": "bind", "template": 4, "depth": 4, "pred": "-18", "truth": "-17", "correct": false, "edits_per_token": 2.0}, {"uid": "9d9b907741fd780b", "family": "bind", "template": 4, "depth": 4, "pred": "38", "truth": "43", "correct": false, "edits_per_token": 2.0}, {"uid": "d19fb0c13b39a42b", "family": "bind", "template": 4, "depth": 4, "pred": "22", "truth": "19", "correct": false, "edits_per_token": 2.0}, {"uid": "618344548ca6da97", "family": "bind", "template": 4, "depth": 4, "pred": "48", "truth": "45", "correct": false, "edits_per_token": 2.0}, {"uid": "20f6f1fda984c6cb", "family": "bind", "template": 4, "depth": 4, "pred": "76", "truth": "76", "correct": true, "edits_per_token": 2.0}, {"uid": "aeebdafd2629497c", "family": "arith", "template": 3, "depth": 3, "pred": "-40", "truth": "-33", "correct": false, "edits_per_token": 2.0}, {"uid": "a44cb3d220440087", "family": "arith", "template": 3, "depth": 3, "pred": "7", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "c11fba9caa3f34db", "family": "arith", "template": 3, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "23a7c0751d154177", "family": "arith", "template": 3, "depth": 3, "pred": "90", "truth": "124", "correct": false, "edits_per_token": 2.0}, {"uid": "bbb473fb8f37e9cc", "family": "bind", "template": 4, "depth": 4, "pred": "3", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "6872e478b699995f", "family": "bind", "template": 3, "depth": 4, "pred": "750", "truth": "450", "correct": false, "edits_per_token": 2.0}, {"uid": "2275fd5dc5422998", "family": "bind", "template": 4, "depth": 4, "pred": "5", "truth": "10", "correct": false, "edits_per_token": 2.0}, {"uid": "dc78fd9d13fcf07f", "family": "bind", "template": 4, "depth": 4, "pred": "14", "truth": "13", "correct": false, "edits_per_token": 2.0}, {"uid": "f42af3a25020bcdd", "family": "bind", "template": 4, "depth": 4, "pred": "18", "truth": "8", "correct": false, "edits_per_token": 2.0}, {"uid": "94126479f3933d69", "family": "fsm", "template": 4, "depth": 4, "pred": "-4", "truth": "0", "correct": false, "edits_per_token": 2.0}, {"uid": "3eb45bde45d94b62", "family": "arith", "template": 3, "depth": 4, "pred": "36", "truth": "192", "correct": false, "edits_per_token": 2.0}, {"uid": "711de0ffe9fc3e1f", "family": "bind", "template": 3, "depth": 4, "pred": "-9", "truth": "-7", "correct": false, "edits_per_token": 2.0}, {"uid": "ed303359a37cd1ef", "family": "bind", "template": 3, "depth": 4, "pred": "-3", "truth": "-3", "correct": true, "edits_per_token": 2.0}, {"uid": "d66c7948ec115397", "family": "bind", "template": 3, "depth": 4, "pred": "-24", "truth": "-18", "correct": false, "edits_per_token": 2.0}, {"uid": "284e1b08cda8cec0", "family": "arith", "template": 3, "depth": 4, "pred": "-48", "truth": "-89", "correct": false, "edits_per_token": 2.0}, {"uid": "7646e2462cfd7f42", "family": "arith", "template": 3, "depth": 4, "pred": "20", "truth": "30", "correct": false, "edits_per_token": 2.0}, {"uid": "5a3e1f340b7e6a24", "family": "arith", "template": 3, "depth": 4, "pred": "8", "truth": "16", "correct": false, "edits_per_token": 2.0}, {"uid": "47f19ffbc80df9f9", "family": "bind", "template": 3, "depth": 4, "pred": "52", "truth": "68", "correct": false, "edits_per_token": 2.0}, {"uid": "a51fb4531b491ec9", "family": "bind", "template": 3, "depth": 4, "pred": "55", "truth": "45", "correct": false, "edits_per_token": 2.0}, {"uid": "15a893f2a205d84e", "family": "fsm", "template": 4, "depth": 4, "pred": "-3", "truth": "-4", "correct": false, "edits_per_token": 2.0}, {"uid": "927fa8211f320a82", "family": "arith", "template": 3, "depth": 4, "pred": "47", "truth": "648", "correct": false, "edits_per_token": 2.0}, {"uid": "0bac982efe5d9b1b", "family": "arith", "template": 3, "depth": 4, "pred": "110", "truth": "121", "correct": false, "edits_per_token": 2.0}, {"uid": "906108b68beb6e66", "family": "arith", "template": 3, "depth": 4, "pred": "-8", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "814724b4fcfab767", "family": "arith", "template": 3, "depth": 4, "pred": "-56", "truth": "-136", "correct": false, "edits_per_token": 2.0}, {"uid": "040cf6134459ea3d", "family": "bind", "template": 3, "depth": 4, "pred": "47", "truth": "53", "correct": false, "edits_per_token": 2.0}, {"uid": "93d2eb4b87dcff06", "family": "bind", "template": 3, "depth": 4, "pred": "-10", "truth": "-6", "correct": false, "edits_per_token": 2.0}, {"uid": "6e672c37599bb93c", "family": "bind", "template": 3, "depth": 4, "pred": "10", "truth": "11", "correct": false, "edits_per_token": 2.0}, {"uid": "d27676130eb57694", "family": "bind", "template": 3, "depth": 4, "pred": "210", "truth": "270", "correct": false, "edits_per_token": 2.0}, {"uid": "a65dce0bbb807f9f", "family": "arith", "template": 3, "depth": 4, "pred": "10", "truth": "-15", "correct": false, "edits_per_token": 2.0}, {"uid": "7279c276a9a815e1", "family": "fsm", "template": 3, "depth": 2, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 2.0}, {"uid": "32578e3c7c2777ec", "family": "fsm", "template": 3, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "27eadcd0b3e689d6", "family": "arith", "template": 3, "depth": 4, "pred": "20", "truth": "19", "correct": false, "edits_per_token": 2.0}, {"uid": "b1cb2f851393b229", "family": "fsm", "template": 3, "depth": 3, "pred": "1", "truth": "-2", "correct": false, "edits_per_token": 2.0}, {"uid": "f715224d49251573", "family": "fsm", "template": 3, "depth": 3, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 2.0}, {"uid": "462877820fe8108c", "family": "fsm", "template": 4, "depth": 3, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "07a5c651bc9329e9", "family": "fsm", "template": 3, "depth": 3, "pred": "18", "truth": "40", "correct": false, "edits_per_token": 2.0}, {"uid": "826ef35bb8ef7e37", "family": "fsm", "template": 3, "depth": 3, "pred": "16", "truth": "16", "correct": true, "edits_per_token": 2.0}, {"uid": "d4cffd1bb7412e20", "family": "fsm", "template": 4, "depth": 4, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "d0c9dad4288a1b00", "family": "excl", "template": 4, "depth": 1, "pred": "11", "truth": "bean apple", "correct": false, "edits_per_token": 2.0}, {"uid": "865a714fc700aa8e", "family": "fsm", "template": 4, "depth": 4, "pred": "1", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "18cdb11c7bccda13", "family": "fsm", "template": 4, "depth": 4, "pred": "2", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "5bf0970eb4ec32b9", "family": "fsm", "template": 4, "depth": 4, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "361329006ff8d9f5", "family": "fsm", "template": 4, "depth": 4, "pred": "-2", "truth": "-2", "correct": true, "edits_per_token": 2.0}, {"uid": "d9c421488e96f445", "family": "sched", "template": 4, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "c03cc91b48845543", "family": "sched", "template": 4, "depth": 1, "pred": "1", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "6813628735e51bac", "family": "sched", "template": 4, "depth": 1, "pred": "16", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "2ccd00a18ae37ca6", "family": "sched", "template": 4, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "c0f7e76d56e30d0d", "family": "fsm", "template": 3, "depth": 2, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "8fbaf8252747c131", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "302d090f441609f2", "family": "sched", "template": 4, "depth": 2, "pred": "4", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "7dd9b367f661c4e7", "family": "fsm", "template": 3, "depth": 3, "pred": "6", "truth": "5", "correct": false, "edits_per_token": 2.0}, {"uid": "5e0d345c061fcf1c", "family": "fsm", "template": 3, "depth": 3, "pred": "3", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "9dce772d152e9809", "family": "sched", "template": 4, "depth": 2, "pred": "16", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "0f5849ec46dda95a", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "28be04c444bcbab0", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "47fd20c3a4c07b85", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "7799aebdd3e85a73", "family": "fsm", "template": 3, "depth": 3, "pred": "8", "truth": "14", "correct": false, "edits_per_token": 2.0}, {"uid": "454b6b92d5dcdd20", "family": "fsm", "template": 3, "depth": 3, "pred": "4", "truth": "2", "correct": false, "edits_per_token": 2.0}, {"uid": "2c9ea8bc86929ebe", "family": "sched", "template": 4, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "b2e6a268b51a1208", "family": "fsm", "template": 3, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 2.0}, {"uid": "bb6ca3372f7dff2a", "family": "excl", "template": 4, "depth": 1, "pred": "11", "truth": "yam lime nut", "correct": false, "edits_per_token": 2.0}, {"uid": "6a2a5075397d961d", "family": "excl", "template": 4, "depth": 3, "pred": "nut date", "truth": "fig nut", "correct": false, "edits_per_token": 2.0}, {"uid": "7e85eb90c2ac7fc3", "family": "sched", "template": 4, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "6ed6b349d0ef224b", "family": "sched", "template": 4, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "cac36aaa74d1e76a", "family": "fsm", "template": 3, "depth": 4, "pred": "4", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "db28fb2ac93f520d", "family": "fsm", "template": 4, "depth": 4, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 2.0}, {"uid": "8c00cc854149641d", "family": "fsm", "template": 4, "depth": 4, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 2.0}, {"uid": "7b1cf55cc5cd3e5f", "family": "excl", "template": 4, "depth": 2, "pred": "lime nut", "truth": "plum lime", "correct": false, "edits_per_token": 2.0}, {"uid": "b7813c6672a55af5", "family": "fsm", "template": 3, "depth": 4, "pred": "-2", "truth": "-5", "correct": false, "edits_per_token": 2.0}, {"uid": "6375fbd1da202d74", "family": "fsm", "template": 3, "depth": 4, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 2.0}, {"uid": "a677f27243a0dbba", "family": "excl", "template": 4, "depth": 2, "pred": "kiwi nut", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "714117d0c4d93f5a", "family": "fsm", "template": 3, "depth": 3, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 2.0}, {"uid": "c9232e7db55d45af", "family": "excl", "template": 4, "depth": 3, "pred": "date", "truth": "yam bean", "correct": false, "edits_per_token": 2.0}, {"uid": "0c9a758e4a02999d", "family": "excl", "template": 4, "depth": 1, "pred": "pear fig", "truth": "lime fig", "correct": false, "edits_per_token": 2.0}, {"uid": "98220b4db365ce22", "family": "fsm", "template": 3, "depth": 4, "pred": "-1", "truth": "-8", "correct": false, "edits_per_token": 2.0}, {"uid": "e233b1fa3a724f09", "family": "excl", "template": 4, "depth": 2, "pred": "none", "truth": "fig bean", "correct": false, "edits_per_token": 2.0}, {"uid": "c374d2a0d211477c", "family": "excl", "template": 4, "depth": 3, "pred": "date", "truth": "bean", "correct": false, "edits_per_token": 2.0}, {"uid": "b25c8b5fc92452b8", "family": "fsm", "template": 3, "depth": 4, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 2.0}, {"uid": "35afcdba50ef0539", "family": "fsm", "template": 3, "depth": 4, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 2.0}, {"uid": "74bdd2fe3dd621df", "family": "excl", "template": 4, "depth": 2, "pred": "apple kiwi", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "7e5ef24472530b54", "family": "sched", "template": 4, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "7b5aa6da844a933a", "family": "sched", "template": 4, "depth": 3, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "961449ff650c677b", "family": "sched", "template": 4, "depth": 3, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "af551a231c42bd0d", "family": "sched", "template": 4, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "933151caa3a0be34", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "231d2a4a23603143", "family": "excl", "template": 4, "depth": 1, "pred": "lime bean f", "truth": "lime bean fig plum", "correct": false, "edits_per_token": 2.0}, {"uid": "f4e87456cd45616d", "family": "excl", "template": 4, "depth": 2, "pred": "lime pear", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "6e23082e8f378986", "family": "excl", "template": 4, "depth": 3, "pred": "bean plum", "truth": "bean", "correct": false, "edits_per_token": 2.0}, {"uid": "ac21be7ac437d92c", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "a650f55eb430eeac", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "9863e0ba8a116a26", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "f85bed8c86544d8d", "family": "fsm", "template": 3, "depth": 4, "pred": "6", "truth": "9", "correct": false, "edits_per_token": 2.0}, {"uid": "f253cae418a07d8f", "family": "excl", "template": 3, "depth": 1, "pred": "none", "truth": "date", "correct": false, "edits_per_token": 2.0}, {"uid": "5765c7bbc9aead9c", "family": "fsm", "template": 3, "depth": 4, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 2.0}, {"uid": "4d2c0dd2cfe3be1c", "family": "excl", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "636a49756779e230", "family": "fsm", "template": 3, "depth": 4, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 2.0}, {"uid": "09460c43e15f2f46", "family": "fsm", "template": 3, "depth": 4, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 2.0}, {"uid": "dd76daa3c9e425c3", "family": "excl", "template": 4, "depth": 1, "pred": "date lime f", "truth": "fig nut lime yam kiwi", "correct": false, "edits_per_token": 2.0}, {"uid": "0c9e513614abcdba", "family": "excl", "template": 4, "depth": 1, "pred": "date plum p", "truth": "pear apple yam", "correct": false, "edits_per_token": 2.0}, {"uid": "369d97e8cff94de0", "family": "excl", "template": 4, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "740a7fe786d6b8a9", "family": "sched", "template": 4, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "8fb18ed65388b1e8", "family": "excl", "template": 4, "depth": 4, "pred": "yam", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "29d92bff2c35ea42", "family": "excl", "template": 4, "depth": 4, "pred": "yam", "truth": "lime", "correct": false, "edits_per_token": 2.0}, {"uid": "f8d4d6edfeb5089f", "family": "sched", "template": 4, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "ee81ecf47e6443b2", "family": "sched", "template": 4, "depth": 3, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "0ff325369010cfc0", "family": "sched", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "1af73c5067b38c2c", "family": "excl", "template": 3, "depth": 1, "pred": "fig", "truth": "lime plum apple date", "correct": false, "edits_per_token": 2.0}, {"uid": "78bf6eea827bc579", "family": "excl", "template": 4, "depth": 1, "pred": "plum apple", "truth": "apple yam pear", "correct": false, "edits_per_token": 2.0}, {"uid": "f7efbbd3fc00581b", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "D", "correct": false, "edits_per_token": 2.0}, {"uid": "f82b882fabca1cb7", "family": "excl", "template": 4, "depth": 2, "pred": "plum", "truth": "plum apple", "correct": false, "edits_per_token": 2.0}, {"uid": "faf4c370f026c6b8", "family": "sched", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "bfa2044b00bb86b8", "family": "sched", "template": 4, "depth": 3, "pred": "none", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "4d3505f6a55ac312", "family": "excl", "template": 4, "depth": 4, "pred": "fig", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "1a3d40c434f1d4e3", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "b1b1ea9a4212d1a9", "family": "sched", "template": 4, "depth": 3, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "8c69b9bd3183ebb6", "family": "excl", "template": 3, "depth": 1, "pred": "none", "truth": "pear bean kiwi plum", "correct": false, "edits_per_token": 2.0}, {"uid": "10dc1311d193c3fa", "family": "excl", "template": 4, "depth": 1, "pred": "lime apple", "truth": "lime apple plum bean", "correct": false, "edits_per_token": 2.0}, {"uid": "82d6d862fe063960", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "11b5887e2e23a957", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "ef07c6682c4275d9", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "3335c404ce9bfcd5", "family": "excl", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "a6c279c43201f057", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "ede73a38c44a3ccf", "family": "sched", "template": 4, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 2.0}, {"uid": "3e0de6f98d396f2a", "family": "excl", "template": 4, "depth": 1, "pred": "pear fig li", "truth": "kiwi pear bean apple", "correct": false, "edits_per_token": 2.0}, {"uid": "d36a0f43f40b6236", "family": "excl", "template": 4, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 2.0}, {"uid": "ebb9ef7d9d066d95", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "4020752baa5a79b3", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "70e7b96a192ab155", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "ae63f7a481d76063", "family": "excl", "template": 4, "depth": 4, "pred": "none", "truth": "bean lime", "correct": false, "edits_per_token": 2.0}, {"uid": "f3e43a7b75ad9689", "family": "excl", "template": 3, "depth": 1, "pred": "date", "truth": "pear fig bean lime", "correct": false, "edits_per_token": 2.0}, {"uid": "586e75ebfbf62207", "family": "excl", "template": 4, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "9b4fed98a3e64c92", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "694f4323afee9216", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "37c09dc4c77fb59b", "family": "excl", "template": 4, "depth": 1, "pred": "apple bean", "truth": "apple", "correct": false, "edits_per_token": 2.0}, {"uid": "27852e223cde9fdd", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "d9a3a976635a4912", "family": "excl", "template": 3, "depth": 1, "pred": "nut", "truth": "kiwi bean date", "correct": false, "edits_per_token": 2.0}, {"uid": "5f308fa3d9d64764", "family": "excl", "template": 3, "depth": 1, "pred": "none", "truth": "lime fig yam", "correct": false, "edits_per_token": 2.0}, {"uid": "796295c79426c62a", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "pear date", "correct": false, "edits_per_token": 2.0}, {"uid": "f59adc461a2a3a23", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "pear fig", "correct": false, "edits_per_token": 2.0}, {"uid": "cb1f068a9f416569", "family": "excl", "template": 4, "depth": 4, "pred": "yam", "truth": "yam", "correct": true, "edits_per_token": 2.0}, {"uid": "08d7338346473ac3", "family": "excl", "template": 3, "depth": 2, "pred": "fig", "truth": "pear bean", "correct": false, "edits_per_token": 2.0}, {"uid": "a74bbd4fb82ecde6", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "apple plum", "correct": false, "edits_per_token": 2.0}, {"uid": "023c777eb4956132", "family": "excl", "template": 4, "depth": 3, "pred": "date", "truth": "date pear", "correct": false, "edits_per_token": 2.0}, {"uid": "8aa42a0a282b8980", "family": "excl", "template": 3, "depth": 2, "pred": "date", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "53637f76c85d8c1c", "family": "excl", "template": 3, "depth": 2, "pred": "apple", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "d0cf89610f223fed", "family": "sched", "template": 4, "depth": 4, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 2.0}, {"uid": "9c6fbac6d438fea4", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "38b1c41b753402e8", "family": "excl", "template": 4, "depth": 2, "pred": "plum", "truth": "plum yam nut", "correct": false, "edits_per_token": 2.0}, {"uid": "3e375d575f2a30b3", "family": "sched", "template": 4, "depth": 4, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 2.0}, {"uid": "40e7271c0285929b", "family": "sched", "template": 4, "depth": 4, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 2.0}, {"uid": "f62ddd4631b1e632", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "9b8f8f5e9e9a3a18", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "f33bce8dab09faee", "family": "excl", "template": 3, "depth": 3, "pred": "nut", "truth": "yam date", "correct": false, "edits_per_token": 2.0}, {"uid": "f7d02c5b717f1b26", "family": "excl", "template": 3, "depth": 4, "pred": "apple", "truth": "nut", "correct": false, "edits_per_token": 2.0}, {"uid": "2d9756e2a9606211", "family": "excl", "template": 4, "depth": 2, "pred": "bean fig li", "truth": "bean date yam", "correct": false, "edits_per_token": 2.0}, {"uid": "a64b960f3eeb5d39", "family": "excl", "template": 4, "depth": 3, "pred": "nut", "truth": "lime", "correct": false, "edits_per_token": 2.0}, {"uid": "75be04c29ab1f001", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "7d09a16f57550745", "family": "sched", "template": 3, "depth": 2, "pred": "C", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "5048ccc397f05377", "family": "sched", "template": 3, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "bce8b50e0845842d", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "66b605e4db85a2c0", "family": "sched", "template": 4, "depth": 4, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 2.0}, {"uid": "47a6146933016e02", "family": "excl", "template": 4, "depth": 2, "pred": "kiwi nut", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "7603bb4fbb4710c8", "family": "excl", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "2f57bbfc2ad4b319", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "f93d17df47bca1c9", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "4545d7fc45e828c0", "family": "sched", "template": 3, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 2.0}, {"uid": "f782173e21a6c1f2", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "80a37124c4ed9a39", "family": "excl", "template": 3, "depth": 1, "pred": "nut", "truth": "bean apple pear", "correct": false, "edits_per_token": 2.0}, {"uid": "bcce29e090c586d2", "family": "sched", "template": 3, "depth": 3, "pred": "15", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "193a55393141bab5", "family": "sched", "template": 3, "depth": 3, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "0324c41dffaa1cb8", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "087a279916282ea3", "family": "excl", "template": 4, "depth": 4, "pred": "date", "truth": "nut kiwi", "correct": false, "edits_per_token": 2.0}, {"uid": "9b2c677e487c27cb", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "fd1c02003b3d4554", "family": "sched", "template": 3, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "7df915ca06235913", "family": "excl", "template": 3, "depth": 2, "pred": "fig", "truth": "yam", "correct": false, "edits_per_token": 2.0}, {"uid": "1e4798197a754ca9", "family": "excl", "template": 4, "depth": 4, "pred": "um", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "2ca156e74b515e97", "family": "sched", "template": 3, "depth": 2, "pred": "C", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "6a2203bca865b619", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "823e185482c10eea", "family": "excl", "template": 3, "depth": 1, "pred": "date", "truth": "pear fig kiwi", "correct": false, "edits_per_token": 2.0}, {"uid": "41e74f93ba171d40", "family": "excl", "template": 4, "depth": 4, "pred": "none", "truth": "bean lime plum", "correct": false, "edits_per_token": 2.0}, {"uid": "57edf49d2957d214", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "nut apple date", "correct": false, "edits_per_token": 2.0}, {"uid": "808bfdae57495704", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "nut pear", "correct": false, "edits_per_token": 2.0}, {"uid": "38d379de426c6ae0", "family": "sched", "template": 3, "depth": 2, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "0b61a24242db3698", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "date", "correct": false, "edits_per_token": 2.0}, {"uid": "9cad194fa2e865be", "family": "excl", "template": 3, "depth": 1, "pred": "fig nut lim", "truth": "fig apple yam lime bean plum", "correct": false, "edits_per_token": 2.0}, {"uid": "16a2b0a2dbc217f0", "family": "excl", "template": 3, "depth": 1, "pred": "plum", "truth": "yam lime apple kiwi nut", "correct": false, "edits_per_token": 2.0}, {"uid": "302f7dfdbcebf7cf", "family": "excl", "template": 4, "depth": 4, "pred": "kiwi", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "88ac831d2cd557d9", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 2.0}, {"uid": "0ea0157cec29a240", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "yam fig plum", "correct": false, "edits_per_token": 2.0}, {"uid": "49165e12e4d4c8d3", "family": "excl", "template": 3, "depth": 3, "pred": "date", "truth": "date bean kiwi", "correct": false, "edits_per_token": 2.0}, {"uid": "575bd19bd31d0b07", "family": "excl", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "3c98968bad82b469", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 2.0}, {"uid": "fa3313c7eee9031c", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "c5b0ad7be361e95a", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 2.0}, {"uid": "6bfdc71318271303", "family": "sched", "template": 3, "depth": 3, "pred": "none", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "0598403107623618", "family": "sched", "template": 3, "depth": 4, "pred": "none", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "28e4acdd7b79568a", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "38724d20f4d85bad", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "7c8636ccbdad97eb", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "9bd2514eedc4b4e1", "family": "excl", "template": 3, "depth": 3, "pred": "plum", "truth": "nut plum yam date fig", "correct": false, "edits_per_token": 2.0}, {"uid": "5a5831a28a079558", "family": "excl", "template": 3, "depth": 3, "pred": "plum", "truth": "plum apple fig kiwi", "correct": false, "edits_per_token": 2.0}, {"uid": "8ef030fff471d972", "family": "sched", "template": 3, "depth": 3, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "484f0cbf9521eb48", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "fdbe9a26369f78cb", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "apple bean pear", "correct": false, "edits_per_token": 2.0}, {"uid": "f2667067c94bd9c2", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "fig", "correct": false, "edits_per_token": 2.0}, {"uid": "636be724170a7508", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "kiwi plum pear yam apple", "correct": false, "edits_per_token": 2.0}, {"uid": "ed83ed43ad98583d", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "0263314c0f8bd6e3", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 2.0}, {"uid": "9ae2d71807ac60ce", "family": "sched", "template": 3, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 2.0}, {"uid": "6a39bb66c31d0c07", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "445a2e8e308d084c", "family": "sched", "template": 3, "depth": 4, "pred": "A", "truth": "D", "correct": false, "edits_per_token": 2.0}, {"uid": "b54ac2bc6f7b24df", "family": "sched", "template": 3, "depth": 4, "pred": "D", "truth": "A", "correct": false, "edits_per_token": 2.0}, {"uid": "20c1afb381170175", "family": "sched", "template": 3, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 2.0}, {"uid": "30430c28d192efc6", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 2.0}, {"uid": "e51b5857bbb2ca50", "family": "sched", "template": 3, "depth": 4, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 2.0}, {"uid": "5b90d13667067fff", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 2.0}, {"uid": "e98454b7446dc760", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "date lime kiwi", "correct": false, "edits_per_token": 2.0}, {"uid": "933b4b87ec87c644", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 2.0}, {"uid": "98b379559d3421a8", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 2.0}] | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/em_dev_template_b8.json
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +[{"uid": "3b112da5fd01aae7", "family": "fsm", "template": 4, "depth": 1, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "d5c0c39e453dcca1", "family": "fsm", "template": 4, "depth": 1, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "edd44a1f63841ef0", "family": "fsm", "template": 4, "depth": 1, "pred": "15", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "4a60e827734a2aad", "family": "fsm", "template": 4, "depth": 1, "pred": "13", "truth": "13", "correct": true, "edits_per_token": 8.0}, {"uid": "d2d4172d0d7eda08", "family": "fsm", "template": 4, "depth": 1, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 8.0}, {"uid": "033ec88b5e210826", "family": "fsm", "template": 4, "depth": 1, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "97fc165388a1bca6", "family": "fsm", "template": 4, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 8.0}, {"uid": "ee578f5db025d5f9", "family": "fsm", "template": 4, "depth": 1, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 8.0}, {"uid": "c396858313b1867d", "family": "fsm", "template": 4, "depth": 2, "pred": "20", "truth": "20", "correct": true, "edits_per_token": 8.0}, {"uid": "62686c3fb5450969", "family": "bind", "template": 4, "depth": 1, "pred": "9", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "160deb3944fa94a9", "family": "fsm", "template": 4, "depth": 2, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 8.0}, {"uid": "ff167bdae9dae304", "family": "fsm", "template": 4, "depth": 2, "pred": "1", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "6e2f978b50da11db", "family": "fsm", "template": 4, "depth": 2, "pred": "-1", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "b3c197791022cdda", "family": "fsm", "template": 4, "depth": 2, "pred": "-1", "truth": "0", "correct": false, "edits_per_token": 8.0}, {"uid": "c6bc5deb98b3f6a4", "family": "fsm", "template": 4, "depth": 2, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "07c9ad219cedeecc", "family": "fsm", "template": 4, "depth": 2, "pred": "1", "truth": "0", "correct": false, "edits_per_token": 8.0}, {"uid": "16a3e4051536fc5a", "family": "arith", "template": 4, "depth": 1, "pred": "-15", "truth": "-16", "correct": false, "edits_per_token": 8.0}, {"uid": "4f1f75fe235ae841", "family": "arith", "template": 4, "depth": 1, "pred": "-21", "truth": "-21", "correct": true, "edits_per_token": 8.0}, {"uid": "cd785088c9149dfb", "family": "arith", "template": 4, "depth": 1, "pred": "164", "truth": "162", "correct": false, "edits_per_token": 8.0}, {"uid": "1638c5c5115e7382", "family": "arith", "template": 4, "depth": 1, "pred": "52", "truth": "48", "correct": false, "edits_per_token": 8.0}, {"uid": "bb572994a445bdb5", "family": "arith", "template": 4, "depth": 1, "pred": "22", "truth": "21", "correct": false, "edits_per_token": 8.0}, {"uid": "ec5611b48356ecc9", "family": "arith", "template": 4, "depth": 1, "pred": "48", "truth": "42", "correct": false, "edits_per_token": 8.0}, {"uid": "f4a95344e3c4417f", "family": "arith", "template": 4, "depth": 1, "pred": "40", "truth": "30", "correct": false, "edits_per_token": 8.0}, {"uid": "e4af4779733a0b3a", "family": "arith", "template": 4, "depth": 1, "pred": "-17", "truth": "-19", "correct": false, "edits_per_token": 8.0}, {"uid": "7e8ac8f95805f2f3", "family": "arith", "template": 4, "depth": 1, "pred": "27", "truth": "29", "correct": false, "edits_per_token": 8.0}, {"uid": "036f0619e07338fd", "family": "bind", "template": 4, "depth": 1, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "e811bf68f0bfdded", "family": "bind", "template": 4, "depth": 1, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "82f1818aa73ed36d", "family": "bind", "template": 4, "depth": 1, "pred": "14", "truth": "12", "correct": false, "edits_per_token": 8.0}, {"uid": "f4d0e013d4462dd0", "family": "bind", "template": 4, "depth": 1, "pred": "8", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "4dd12c9283c1639d", "family": "arith", "template": 4, "depth": 1, "pred": "37", "truth": "37", "correct": true, "edits_per_token": 8.0}, {"uid": "889301436be7278d", "family": "bind", "template": 4, "depth": 1, "pred": "56", "truth": "45", "correct": false, "edits_per_token": 8.0}, {"uid": "34157746c6a9aa26", "family": "bind", "template": 4, "depth": 1, "pred": "36", "truth": "30", "correct": false, "edits_per_token": 8.0}, {"uid": "972d347d02c3e4c9", "family": "bind", "template": 4, "depth": 1, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 8.0}, {"uid": "73358f5d5d68a323", "family": "bind", "template": 4, "depth": 1, "pred": "14", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "621a81bcb0f3a1da", "family": "bind", "template": 4, "depth": 1, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 8.0}, {"uid": "be55eff500fd9de4", "family": "arith", "template": 4, "depth": 2, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 8.0}, {"uid": "8086e5fd466e56e3", "family": "arith", "template": 4, "depth": 2, "pred": "22", "truth": "23", "correct": false, "edits_per_token": 8.0}, {"uid": "1d3e8e419ebffaea", "family": "bind", "template": 3, "depth": 1, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 8.0}, {"uid": "03afb70c0ecdf774", "family": "fsm", "template": 4, "depth": 2, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "53768ffc251ff5ea", "family": "arith", "template": 4, "depth": 2, "pred": "52", "truth": "56", "correct": false, "edits_per_token": 8.0}, {"uid": "856912fe4bd92aab", "family": "arith", "template": 4, "depth": 2, "pred": "720", "truth": "576", "correct": false, "edits_per_token": 8.0}, {"uid": "bcfa81650916d43a", "family": "arith", "template": 4, "depth": 2, "pred": "24", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "636ae00985bb6e39", "family": "bind", "template": 3, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 8.0}, {"uid": "866b93f2b1cebb8a", "family": "bind", "template": 3, "depth": 1, "pred": "35", "truth": "33", "correct": false, "edits_per_token": 8.0}, {"uid": "2449760460938f84", "family": "bind", "template": 3, "depth": 1, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 8.0}, {"uid": "664b188b12b5afc4", "family": "arith", "template": 4, "depth": 2, "pred": "24", "truth": "27", "correct": false, "edits_per_token": 8.0}, {"uid": "efba878fb6ac5fa8", "family": "arith", "template": 4, "depth": 2, "pred": "0", "truth": "-36", "correct": false, "edits_per_token": 8.0}, {"uid": "149112577b7a6b0b", "family": "arith", "template": 4, "depth": 2, "pred": "-12", "truth": "-13", "correct": false, "edits_per_token": 8.0}, {"uid": "4f926ee35fd6594e", "family": "bind", "template": 3, "depth": 1, "pred": "14", "truth": "12", "correct": false, "edits_per_token": 8.0}, {"uid": "1432cede89ff82db", "family": "bind", "template": 3, "depth": 1, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 8.0}, {"uid": "ab35cb1ccb4a9ebc", "family": "bind", "template": 3, "depth": 1, "pred": "13", "truth": "13", "correct": true, "edits_per_token": 8.0}, {"uid": "7c739b0506d8ac40", "family": "bind", "template": 3, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 8.0}, {"uid": "4e25a4ae3b67d976", "family": "bind", "template": 3, "depth": 1, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 8.0}, {"uid": "22004f16631c31fa", "family": "arith", "template": 4, "depth": 2, "pred": "7", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "aead8eeee264a6dd", "family": "arith", "template": 4, "depth": 2, "pred": "-22", "truth": "-25", "correct": false, "edits_per_token": 8.0}, {"uid": "49a53bd60db6df22", "family": "bind", "template": 3, "depth": 1, "pred": "26", "truth": "24", "correct": false, "edits_per_token": 8.0}, {"uid": "efec1e4e4e19b40b", "family": "arith", "template": 4, "depth": 3, "pred": "-4", "truth": "-8", "correct": false, "edits_per_token": 8.0}, {"uid": "3e17152a2a3b58df", "family": "arith", "template": 4, "depth": 3, "pred": "42", "truth": "54", "correct": false, "edits_per_token": 8.0}, {"uid": "1d78f810c43669a2", "family": "fsm", "template": 3, "depth": 1, "pred": "3", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "33930d2cc436abac", "family": "arith", "template": 4, "depth": 3, "pred": "31", "truth": "30", "correct": false, "edits_per_token": 8.0}, {"uid": "e4333fd80de54f3c", "family": "arith", "template": 4, "depth": 3, "pred": "7", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "481c38df29183f6c", "family": "arith", "template": 4, "depth": 3, "pred": "23", "truth": "25", "correct": false, "edits_per_token": 8.0}, {"uid": "ef528c3a1e15bb39", "family": "arith", "template": 4, "depth": 3, "pred": "1", "truth": "-1", "correct": false, "edits_per_token": 8.0}, {"uid": "74899f1ef74b0a4c", "family": "arith", "template": 4, "depth": 3, "pred": "101", "truth": "106", "correct": false, "edits_per_token": 8.0}, {"uid": "70ebe3705ff6ea5f", "family": "arith", "template": 4, "depth": 3, "pred": "483", "truth": "531", "correct": false, "edits_per_token": 8.0}, {"uid": "8a16bdc38d436270", "family": "arith", "template": 4, "depth": 3, "pred": "-14", "truth": "-17", "correct": false, "edits_per_token": 8.0}, {"uid": "0a2078c86d5ac9a7", "family": "arith", "template": 4, "depth": 3, "pred": "90", "truth": "96", "correct": false, "edits_per_token": 8.0}, {"uid": "7384d429e86eb01f", "family": "fsm", "template": 4, "depth": 3, "pred": "20", "truth": "19", "correct": false, "edits_per_token": 8.0}, {"uid": "ed6bb16aadcb518a", "family": "arith", "template": 4, "depth": 4, "pred": "10", "truth": "-14", "correct": false, "edits_per_token": 8.0}, {"uid": "28c1d3a0fbe9487f", "family": "arith", "template": 4, "depth": 4, "pred": "1480", "truth": "2520", "correct": false, "edits_per_token": 8.0}, {"uid": "c517d1504114ad7a", "family": "bind", "template": 4, "depth": 2, "pred": "17", "truth": "15", "correct": false, "edits_per_token": 8.0}, {"uid": "f78d7ec7e37e6779", "family": "bind", "template": 4, "depth": 2, "pred": "360", "truth": "280", "correct": false, "edits_per_token": 8.0}, {"uid": "e544d5e12103f02f", "family": "bind", "template": 4, "depth": 2, "pred": "-21", "truth": "-22", "correct": false, "edits_per_token": 8.0}, {"uid": "ecd2ffe740cc3efc", "family": "bind", "template": 4, "depth": 2, "pred": "75", "truth": "70", "correct": false, "edits_per_token": 8.0}, {"uid": "dcc19f4449b249b1", "family": "bind", "template": 4, "depth": 2, "pred": "15", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "e0606ee0117afdad", "family": "bind", "template": 4, "depth": 2, "pred": "0", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "9a8b7ec79dbf6a8b", "family": "bind", "template": 4, "depth": 2, "pred": "25", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "2c429b7dc30d5227", "family": "bind", "template": 4, "depth": 2, "pred": "23", "truth": "21", "correct": false, "edits_per_token": 8.0}, {"uid": "cd27ac5d6df9a071", "family": "bind", "template": 4, "depth": 2, "pred": "4", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "f49f985044043dbb", "family": "fsm", "template": 3, "depth": 1, "pred": "6", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "ae8404400f627756", "family": "fsm", "template": 3, "depth": 1, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 8.0}, {"uid": "6e13355d1ea33b50", "family": "fsm", "template": 3, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 8.0}, {"uid": "4937d4afa6b30a62", "family": "arith", "template": 4, "depth": 4, "pred": "288", "truth": "287", "correct": false, "edits_per_token": 8.0}, {"uid": "fe3cc19b2abb5508", "family": "arith", "template": 4, "depth": 4, "pred": "36", "truth": "37", "correct": false, "edits_per_token": 8.0}, {"uid": "0568f7f367b4cbff", "family": "arith", "template": 4, "depth": 4, "pred": "280", "truth": "1104", "correct": false, "edits_per_token": 8.0}, {"uid": "3a0bcfe44b9fee3a", "family": "bind", "template": 4, "depth": 2, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "debc0c1111837aef", "family": "bind", "template": 4, "depth": 2, "pred": "28", "truth": "25", "correct": false, "edits_per_token": 8.0}, {"uid": "dc655e2dd9f2365c", "family": "arith", "template": 4, "depth": 4, "pred": "952", "truth": "863", "correct": false, "edits_per_token": 8.0}, {"uid": "983666e948bd6063", "family": "arith", "template": 4, "depth": 4, "pred": "24", "truth": "31", "correct": false, "edits_per_token": 8.0}, {"uid": "9eec9b28436d7847", "family": "arith", "template": 4, "depth": 4, "pred": "68", "truth": "74", "correct": false, "edits_per_token": 8.0}, {"uid": "a3a5173b2df7abf5", "family": "arith", "template": 4, "depth": 4, "pred": "71", "truth": "81", "correct": false, "edits_per_token": 8.0}, {"uid": "41d9923c4340c334", "family": "bind", "template": 3, "depth": 2, "pred": "17", "truth": "17", "correct": true, "edits_per_token": 8.0}, {"uid": "ee9b7f5c7b8f41d7", "family": "fsm", "template": 4, "depth": 2, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 8.0}, {"uid": "5fe833587ead4d87", "family": "arith", "template": 4, "depth": 4, "pred": "179", "truth": "123", "correct": false, "edits_per_token": 8.0}, {"uid": "f05bbf493d7397b4", "family": "bind", "template": 3, "depth": 2, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "697ac2b97e9fb733", "family": "bind", "template": 3, "depth": 2, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "90c53787b6cd368f", "family": "bind", "template": 3, "depth": 2, "pred": "51", "truth": "57", "correct": false, "edits_per_token": 8.0}, {"uid": "d54be281751cf2d9", "family": "bind", "template": 3, "depth": 2, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 8.0}, {"uid": "1400d18be0006c2e", "family": "bind", "template": 3, "depth": 2, "pred": "7", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "11aa996915f24160", "family": "bind", "template": 3, "depth": 2, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 8.0}, {"uid": "1184f7b32425db4e", "family": "fsm", "template": 4, "depth": 2, "pred": "-2", "truth": "-4", "correct": false, "edits_per_token": 8.0}, {"uid": "a3fbb3b73037e7fc", "family": "fsm", "template": 4, "depth": 3, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "90a4a1941f06af3e", "family": "fsm", "template": 4, "depth": 3, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 8.0}, {"uid": "e37479d5b242311d", "family": "fsm", "template": 4, "depth": 3, "pred": "4", "truth": "0", "correct": false, "edits_per_token": 8.0}, {"uid": "0479386e08d5202a", "family": "bind", "template": 3, "depth": 2, "pred": "-10", "truth": "-8", "correct": false, "edits_per_token": 8.0}, {"uid": "7a9a8bb485ecb944", "family": "bind", "template": 3, "depth": 2, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "ef46f38e2bfbd61e", "family": "bind", "template": 3, "depth": 2, "pred": "28", "truth": "28", "correct": true, "edits_per_token": 8.0}, {"uid": "f15d8d070656a044", "family": "fsm", "template": 4, "depth": 3, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 8.0}, {"uid": "d903d2766fcb00eb", "family": "bind", "template": 4, "depth": 3, "pred": "0", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "92276be46fe4ab30", "family": "bind", "template": 4, "depth": 3, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 8.0}, {"uid": "e2ccd53faa0a3547", "family": "bind", "template": 4, "depth": 3, "pred": "20", "truth": "26", "correct": false, "edits_per_token": 8.0}, {"uid": "df17dd0c718e4fa4", "family": "bind", "template": 4, "depth": 3, "pred": "-14", "truth": "-11", "correct": false, "edits_per_token": 8.0}, {"uid": "b9820bfcb75201d0", "family": "bind", "template": 4, "depth": 3, "pred": "-12", "truth": "-12", "correct": true, "edits_per_token": 8.0}, {"uid": "0f1e89ceabf587f2", "family": "fsm", "template": 4, "depth": 3, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "96bac0fe597844b9", "family": "bind", "template": 4, "depth": 3, "pred": "76", "truth": "74", "correct": false, "edits_per_token": 8.0}, {"uid": "ba32a49748c6835f", "family": "bind", "template": 4, "depth": 3, "pred": "31", "truth": "30", "correct": false, "edits_per_token": 8.0}, {"uid": "61de92edd07c9f73", "family": "bind", "template": 4, "depth": 3, "pred": "284", "truth": "225", "correct": false, "edits_per_token": 8.0}, {"uid": "ae548ce56bcc7884", "family": "bind", "template": 4, "depth": 3, "pred": "48", "truth": "50", "correct": false, "edits_per_token": 8.0}, {"uid": "a393a7120cee660b", "family": "fsm", "template": 3, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 8.0}, {"uid": "5a74de8f55ccd4ab", "family": "fsm", "template": 3, "depth": 1, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 8.0}, {"uid": "ee1421b1d9f21ce7", "family": "fsm", "template": 3, "depth": 1, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "a28f7f5b424eb377", "family": "fsm", "template": 3, "depth": 2, "pred": "7", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "e37c6e22ee7cdf83", "family": "fsm", "template": 3, "depth": 2, "pred": "22", "truth": "24", "correct": false, "edits_per_token": 8.0}, {"uid": "0bd5ea1e3dcb6681", "family": "fsm", "template": 3, "depth": 2, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "a6defeaa073f159e", "family": "fsm", "template": 4, "depth": 4, "pred": "21", "truth": "21", "correct": true, "edits_per_token": 8.0}, {"uid": "17f316431a230f95", "family": "arith", "template": 3, "depth": 1, "pred": "10", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "688087012cf48422", "family": "bind", "template": 4, "depth": 3, "pred": "28", "truth": "35", "correct": false, "edits_per_token": 8.0}, {"uid": "10ed69a16258ca4e", "family": "fsm", "template": 3, "depth": 2, "pred": "4", "truth": "7", "correct": false, "edits_per_token": 8.0}, {"uid": "103b828455dbf33c", "family": "fsm", "template": 3, "depth": 2, "pred": "21", "truth": "29", "correct": false, "edits_per_token": 8.0}, {"uid": "0d9f6bf4de9dd849", "family": "fsm", "template": 3, "depth": 2, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "fe253127243bbe69", "family": "arith", "template": 3, "depth": 1, "pred": "30", "truth": "30", "correct": true, "edits_per_token": 8.0}, {"uid": "418c76061f2eaea0", "family": "arith", "template": 3, "depth": 1, "pred": "54", "truth": "60", "correct": false, "edits_per_token": 8.0}, {"uid": "f06a672b579589f4", "family": "arith", "template": 3, "depth": 1, "pred": "20", "truth": "26", "correct": false, "edits_per_token": 8.0}, {"uid": "602e11044971aeb3", "family": "bind", "template": 3, "depth": 3, "pred": "81", "truth": "71", "correct": false, "edits_per_token": 8.0}, {"uid": "27d3b392810a6e03", "family": "arith", "template": 3, "depth": 1, "pred": "23", "truth": "28", "correct": false, "edits_per_token": 8.0}, {"uid": "05dd5abb67e57121", "family": "arith", "template": 3, "depth": 1, "pred": "40", "truth": "41", "correct": false, "edits_per_token": 8.0}, {"uid": "bab6b2ba0bc7ee6d", "family": "arith", "template": 3, "depth": 1, "pred": "-6", "truth": "-10", "correct": false, "edits_per_token": 8.0}, {"uid": "0888a35d47dd9c33", "family": "arith", "template": 3, "depth": 1, "pred": "7", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "50d9b8b5e09e4336", "family": "arith", "template": 3, "depth": 1, "pred": "-14", "truth": "-12", "correct": false, "edits_per_token": 8.0}, {"uid": "df86ed788c82dad6", "family": "arith", "template": 3, "depth": 1, "pred": "-2", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "bc9e03a4d69d9d25", "family": "bind", "template": 3, "depth": 3, "pred": "-11", "truth": "-9", "correct": false, "edits_per_token": 8.0}, {"uid": "bdd4082503abf525", "family": "bind", "template": 3, "depth": 3, "pred": "10", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "1703616b8801e017", "family": "bind", "template": 3, "depth": 3, "pred": "36", "truth": "48", "correct": false, "edits_per_token": 8.0}, {"uid": "f89d93ce767e7d1a", "family": "bind", "template": 3, "depth": 3, "pred": "180", "truth": "123", "correct": false, "edits_per_token": 8.0}, {"uid": "758c9b791db04f9d", "family": "bind", "template": 3, "depth": 3, "pred": "14", "truth": "14", "correct": true, "edits_per_token": 8.0}, {"uid": "1485a8c2ccd127a9", "family": "fsm", "template": 4, "depth": 3, "pred": "9", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "314cf8cd1e62d84f", "family": "bind", "template": 3, "depth": 3, "pred": "24", "truth": "23", "correct": false, "edits_per_token": 8.0}, {"uid": "5b4fb5a0bc255002", "family": "bind", "template": 3, "depth": 3, "pred": "16", "truth": "17", "correct": false, "edits_per_token": 8.0}, {"uid": "d0f8e1d5fa0aee5d", "family": "arith", "template": 3, "depth": 2, "pred": "16", "truth": "20", "correct": false, "edits_per_token": 8.0}, {"uid": "296f1fab9ec37663", "family": "bind", "template": 3, "depth": 3, "pred": "-17", "truth": "-14", "correct": false, "edits_per_token": 8.0}, {"uid": "779fef7e1a0c194c", "family": "arith", "template": 3, "depth": 2, "pred": "50", "truth": "60", "correct": false, "edits_per_token": 8.0}, {"uid": "afec0c621aae8261", "family": "arith", "template": 3, "depth": 2, "pred": "300", "truth": "342", "correct": false, "edits_per_token": 8.0}, {"uid": "1fd1d41e1e4bf549", "family": "arith", "template": 3, "depth": 2, "pred": "30", "truth": "41", "correct": false, "edits_per_token": 8.0}, {"uid": "68ec3f53b05b8a8f", "family": "arith", "template": 3, "depth": 2, "pred": "8", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "2e46a8c4c11a017f", "family": "bind", "template": 3, "depth": 3, "pred": "33", "truth": "36", "correct": false, "edits_per_token": 8.0}, {"uid": "3742c76cef7c48b5", "family": "fsm", "template": 3, "depth": 2, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "8f3914e60d8577ca", "family": "fsm", "template": 3, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 8.0}, {"uid": "4302d428e1f3df93", "family": "arith", "template": 3, "depth": 2, "pred": "-12", "truth": "-13", "correct": false, "edits_per_token": 8.0}, {"uid": "0b98d2417cc0b84e", "family": "arith", "template": 3, "depth": 2, "pred": "30", "truth": "34", "correct": false, "edits_per_token": 8.0}, {"uid": "f480d24e45533797", "family": "arith", "template": 3, "depth": 2, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "aadf9012cfc9e911", "family": "arith", "template": 3, "depth": 2, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 8.0}, {"uid": "0132fdf42d9298ad", "family": "arith", "template": 3, "depth": 2, "pred": "130", "truth": "170", "correct": false, "edits_per_token": 8.0}, {"uid": "b5fc0c28422e9e56", "family": "sched", "template": 4, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "40ea1bb01295386f", "family": "arith", "template": 3, "depth": 2, "pred": "-30", "truth": "-30", "correct": true, "edits_per_token": 8.0}, {"uid": "09c3fef4d7253107", "family": "fsm", "template": 4, "depth": 3, "pred": "4", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "65c150d0ca66cb88", "family": "fsm", "template": 4, "depth": 3, "pred": "6", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "e00e7b260265d9ea", "family": "sched", "template": 4, "depth": 1, "pred": "20", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "b3115967a12393ab", "family": "sched", "template": 4, "depth": 1, "pred": "11", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "23f2db205d8a23de", "family": "arith", "template": 3, "depth": 3, "pred": "12", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "3ff1d1bd3d1311ba", "family": "arith", "template": 3, "depth": 3, "pred": "20", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "19beb79343841fd2", "family": "fsm", "template": 4, "depth": 3, "pred": "-5", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "cc873df2e7df8bed", "family": "arith", "template": 3, "depth": 3, "pred": "10", "truth": "18", "correct": false, "edits_per_token": 8.0}, {"uid": "d0326312724fda50", "family": "arith", "template": 3, "depth": 3, "pred": "40", "truth": "51", "correct": false, "edits_per_token": 8.0}, {"uid": "224bd8479cfb9a0c", "family": "arith", "template": 3, "depth": 3, "pred": "45", "truth": "409", "correct": false, "edits_per_token": 8.0}, {"uid": "231a329a8cb3ec7b", "family": "arith", "template": 3, "depth": 3, "pred": "60", "truth": "100", "correct": false, "edits_per_token": 8.0}, {"uid": "6682c6a5479651b2", "family": "bind", "template": 4, "depth": 4, "pred": "18", "truth": "19", "correct": false, "edits_per_token": 8.0}, {"uid": "fd930d8d771290fe", "family": "bind", "template": 4, "depth": 4, "pred": "-18", "truth": "-17", "correct": false, "edits_per_token": 8.0}, {"uid": "9d9b907741fd780b", "family": "bind", "template": 4, "depth": 4, "pred": "38", "truth": "43", "correct": false, "edits_per_token": 8.0}, {"uid": "d19fb0c13b39a42b", "family": "bind", "template": 4, "depth": 4, "pred": "22", "truth": "19", "correct": false, "edits_per_token": 8.0}, {"uid": "618344548ca6da97", "family": "bind", "template": 4, "depth": 4, "pred": "48", "truth": "45", "correct": false, "edits_per_token": 8.0}, {"uid": "20f6f1fda984c6cb", "family": "bind", "template": 4, "depth": 4, "pred": "72", "truth": "76", "correct": false, "edits_per_token": 8.0}, {"uid": "aeebdafd2629497c", "family": "arith", "template": 3, "depth": 3, "pred": "-40", "truth": "-33", "correct": false, "edits_per_token": 8.0}, {"uid": "a44cb3d220440087", "family": "arith", "template": 3, "depth": 3, "pred": "7", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "c11fba9caa3f34db", "family": "arith", "template": 3, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "23a7c0751d154177", "family": "arith", "template": 3, "depth": 3, "pred": "90", "truth": "124", "correct": false, "edits_per_token": 8.0}, {"uid": "bbb473fb8f37e9cc", "family": "bind", "template": 4, "depth": 4, "pred": "3", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "6872e478b699995f", "family": "bind", "template": 3, "depth": 4, "pred": "750", "truth": "450", "correct": false, "edits_per_token": 8.0}, {"uid": "2275fd5dc5422998", "family": "bind", "template": 4, "depth": 4, "pred": "5", "truth": "10", "correct": false, "edits_per_token": 8.0}, {"uid": "dc78fd9d13fcf07f", "family": "bind", "template": 4, "depth": 4, "pred": "14", "truth": "13", "correct": false, "edits_per_token": 8.0}, {"uid": "f42af3a25020bcdd", "family": "bind", "template": 4, "depth": 4, "pred": "22", "truth": "8", "correct": false, "edits_per_token": 8.0}, {"uid": "94126479f3933d69", "family": "fsm", "template": 4, "depth": 4, "pred": "-2", "truth": "0", "correct": false, "edits_per_token": 8.0}, {"uid": "3eb45bde45d94b62", "family": "arith", "template": 3, "depth": 4, "pred": "36", "truth": "192", "correct": false, "edits_per_token": 8.0}, {"uid": "711de0ffe9fc3e1f", "family": "bind", "template": 3, "depth": 4, "pred": "-9", "truth": "-7", "correct": false, "edits_per_token": 8.0}, {"uid": "ed303359a37cd1ef", "family": "bind", "template": 3, "depth": 4, "pred": "-3", "truth": "-3", "correct": true, "edits_per_token": 8.0}, {"uid": "d66c7948ec115397", "family": "bind", "template": 3, "depth": 4, "pred": "-24", "truth": "-18", "correct": false, "edits_per_token": 8.0}, {"uid": "284e1b08cda8cec0", "family": "arith", "template": 3, "depth": 4, "pred": "-49", "truth": "-89", "correct": false, "edits_per_token": 8.0}, {"uid": "7646e2462cfd7f42", "family": "arith", "template": 3, "depth": 4, "pred": "20", "truth": "30", "correct": false, "edits_per_token": 8.0}, {"uid": "5a3e1f340b7e6a24", "family": "arith", "template": 3, "depth": 4, "pred": "8", "truth": "16", "correct": false, "edits_per_token": 8.0}, {"uid": "47f19ffbc80df9f9", "family": "bind", "template": 3, "depth": 4, "pred": "52", "truth": "68", "correct": false, "edits_per_token": 8.0}, {"uid": "a51fb4531b491ec9", "family": "bind", "template": 3, "depth": 4, "pred": "55", "truth": "45", "correct": false, "edits_per_token": 8.0}, {"uid": "15a893f2a205d84e", "family": "fsm", "template": 4, "depth": 4, "pred": "-3", "truth": "-4", "correct": false, "edits_per_token": 8.0}, {"uid": "927fa8211f320a82", "family": "arith", "template": 3, "depth": 4, "pred": "47", "truth": "648", "correct": false, "edits_per_token": 8.0}, {"uid": "0bac982efe5d9b1b", "family": "arith", "template": 3, "depth": 4, "pred": "110", "truth": "121", "correct": false, "edits_per_token": 8.0}, {"uid": "906108b68beb6e66", "family": "arith", "template": 3, "depth": 4, "pred": "-7", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "814724b4fcfab767", "family": "arith", "template": 3, "depth": 4, "pred": "-53", "truth": "-136", "correct": false, "edits_per_token": 8.0}, {"uid": "040cf6134459ea3d", "family": "bind", "template": 3, "depth": 4, "pred": "45", "truth": "53", "correct": false, "edits_per_token": 8.0}, {"uid": "93d2eb4b87dcff06", "family": "bind", "template": 3, "depth": 4, "pred": "-10", "truth": "-6", "correct": false, "edits_per_token": 8.0}, {"uid": "6e672c37599bb93c", "family": "bind", "template": 3, "depth": 4, "pred": "10", "truth": "11", "correct": false, "edits_per_token": 8.0}, {"uid": "d27676130eb57694", "family": "bind", "template": 3, "depth": 4, "pred": "220", "truth": "270", "correct": false, "edits_per_token": 8.0}, {"uid": "a65dce0bbb807f9f", "family": "arith", "template": 3, "depth": 4, "pred": "10", "truth": "-15", "correct": false, "edits_per_token": 8.0}, {"uid": "7279c276a9a815e1", "family": "fsm", "template": 3, "depth": 2, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 8.0}, {"uid": "32578e3c7c2777ec", "family": "fsm", "template": 3, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 8.0}, {"uid": "27eadcd0b3e689d6", "family": "arith", "template": 3, "depth": 4, "pred": "20", "truth": "19", "correct": false, "edits_per_token": 8.0}, {"uid": "b1cb2f851393b229", "family": "fsm", "template": 3, "depth": 3, "pred": "1", "truth": "-2", "correct": false, "edits_per_token": 8.0}, {"uid": "f715224d49251573", "family": "fsm", "template": 3, "depth": 3, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 8.0}, {"uid": "462877820fe8108c", "family": "fsm", "template": 4, "depth": 3, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 8.0}, {"uid": "07a5c651bc9329e9", "family": "fsm", "template": 3, "depth": 3, "pred": "18", "truth": "40", "correct": false, "edits_per_token": 8.0}, {"uid": "826ef35bb8ef7e37", "family": "fsm", "template": 3, "depth": 3, "pred": "16", "truth": "16", "correct": true, "edits_per_token": 8.0}, {"uid": "d4cffd1bb7412e20", "family": "fsm", "template": 4, "depth": 4, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "d0c9dad4288a1b00", "family": "excl", "template": 4, "depth": 1, "pred": "11", "truth": "bean apple", "correct": false, "edits_per_token": 8.0}, {"uid": "865a714fc700aa8e", "family": "fsm", "template": 4, "depth": 4, "pred": "1", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "18cdb11c7bccda13", "family": "fsm", "template": 4, "depth": 4, "pred": "2", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "5bf0970eb4ec32b9", "family": "fsm", "template": 4, "depth": 4, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 8.0}, {"uid": "361329006ff8d9f5", "family": "fsm", "template": 4, "depth": 4, "pred": "-2", "truth": "-2", "correct": true, "edits_per_token": 8.0}, {"uid": "d9c421488e96f445", "family": "sched", "template": 4, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "c03cc91b48845543", "family": "sched", "template": 4, "depth": 1, "pred": "1", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "6813628735e51bac", "family": "sched", "template": 4, "depth": 1, "pred": "15", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "2ccd00a18ae37ca6", "family": "sched", "template": 4, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "c0f7e76d56e30d0d", "family": "fsm", "template": 3, "depth": 2, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "8fbaf8252747c131", "family": "sched", "template": 4, "depth": 2, "pred": "10", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "302d090f441609f2", "family": "sched", "template": 4, "depth": 2, "pred": "4", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "7dd9b367f661c4e7", "family": "fsm", "template": 3, "depth": 3, "pred": "6", "truth": "5", "correct": false, "edits_per_token": 8.0}, {"uid": "5e0d345c061fcf1c", "family": "fsm", "template": 3, "depth": 3, "pred": "3", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "9dce772d152e9809", "family": "sched", "template": 4, "depth": 2, "pred": "15", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "0f5849ec46dda95a", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "28be04c444bcbab0", "family": "sched", "template": 4, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "47fd20c3a4c07b85", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "7799aebdd3e85a73", "family": "fsm", "template": 3, "depth": 3, "pred": "8", "truth": "14", "correct": false, "edits_per_token": 8.0}, {"uid": "454b6b92d5dcdd20", "family": "fsm", "template": 3, "depth": 3, "pred": "4", "truth": "2", "correct": false, "edits_per_token": 8.0}, {"uid": "2c9ea8bc86929ebe", "family": "sched", "template": 4, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "b2e6a268b51a1208", "family": "fsm", "template": 3, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 8.0}, {"uid": "bb6ca3372f7dff2a", "family": "excl", "template": 4, "depth": 1, "pred": "20", "truth": "yam lime nut", "correct": false, "edits_per_token": 8.0}, {"uid": "6a2a5075397d961d", "family": "excl", "template": 4, "depth": 3, "pred": "nut date", "truth": "fig nut", "correct": false, "edits_per_token": 8.0}, {"uid": "7e85eb90c2ac7fc3", "family": "sched", "template": 4, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "6ed6b349d0ef224b", "family": "sched", "template": 4, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "cac36aaa74d1e76a", "family": "fsm", "template": 3, "depth": 4, "pred": "4", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "db28fb2ac93f520d", "family": "fsm", "template": 4, "depth": 4, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 8.0}, {"uid": "8c00cc854149641d", "family": "fsm", "template": 4, "depth": 4, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 8.0}, {"uid": "7b1cf55cc5cd3e5f", "family": "excl", "template": 4, "depth": 2, "pred": "lime nut", "truth": "plum lime", "correct": false, "edits_per_token": 8.0}, {"uid": "b7813c6672a55af5", "family": "fsm", "template": 3, "depth": 4, "pred": "-2", "truth": "-5", "correct": false, "edits_per_token": 8.0}, {"uid": "6375fbd1da202d74", "family": "fsm", "template": 3, "depth": 4, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 8.0}, {"uid": "a677f27243a0dbba", "family": "excl", "template": 4, "depth": 2, "pred": "kiwi nut", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "714117d0c4d93f5a", "family": "fsm", "template": 3, "depth": 3, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 8.0}, {"uid": "c9232e7db55d45af", "family": "excl", "template": 4, "depth": 3, "pred": "date", "truth": "yam bean", "correct": false, "edits_per_token": 8.0}, {"uid": "0c9a758e4a02999d", "family": "excl", "template": 4, "depth": 1, "pred": "fig apple", "truth": "lime fig", "correct": false, "edits_per_token": 8.0}, {"uid": "98220b4db365ce22", "family": "fsm", "template": 3, "depth": 4, "pred": "-1", "truth": "-8", "correct": false, "edits_per_token": 8.0}, {"uid": "e233b1fa3a724f09", "family": "excl", "template": 4, "depth": 2, "pred": "none", "truth": "fig bean", "correct": false, "edits_per_token": 8.0}, {"uid": "c374d2a0d211477c", "family": "excl", "template": 4, "depth": 3, "pred": "date", "truth": "bean", "correct": false, "edits_per_token": 8.0}, {"uid": "b25c8b5fc92452b8", "family": "fsm", "template": 3, "depth": 4, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 8.0}, {"uid": "35afcdba50ef0539", "family": "fsm", "template": 3, "depth": 4, "pred": "5", "truth": "6", "correct": false, "edits_per_token": 8.0}, {"uid": "74bdd2fe3dd621df", "family": "excl", "template": 4, "depth": 2, "pred": "apple kiwi", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "7e5ef24472530b54", "family": "sched", "template": 4, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "7b5aa6da844a933a", "family": "sched", "template": 4, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "961449ff650c677b", "family": "sched", "template": 4, "depth": 3, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "af551a231c42bd0d", "family": "sched", "template": 4, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "933151caa3a0be34", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "231d2a4a23603143", "family": "excl", "template": 4, "depth": 1, "pred": "lime fig pl", "truth": "lime bean fig plum", "correct": false, "edits_per_token": 8.0}, {"uid": "f4e87456cd45616d", "family": "excl", "template": 4, "depth": 2, "pred": "lime pear", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "6e23082e8f378986", "family": "excl", "template": 4, "depth": 3, "pred": "bean plum", "truth": "bean", "correct": false, "edits_per_token": 8.0}, {"uid": "ac21be7ac437d92c", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "a650f55eb430eeac", "family": "sched", "template": 4, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "9863e0ba8a116a26", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "f85bed8c86544d8d", "family": "fsm", "template": 3, "depth": 4, "pred": "6", "truth": "9", "correct": false, "edits_per_token": 8.0}, {"uid": "f253cae418a07d8f", "family": "excl", "template": 3, "depth": 1, "pred": "none", "truth": "date", "correct": false, "edits_per_token": 8.0}, {"uid": "5765c7bbc9aead9c", "family": "fsm", "template": 3, "depth": 4, "pred": "5", "truth": "4", "correct": false, "edits_per_token": 8.0}, {"uid": "4d2c0dd2cfe3be1c", "family": "excl", "template": 4, "depth": 3, "pred": "5", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "636a49756779e230", "family": "fsm", "template": 3, "depth": 4, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 8.0}, {"uid": "09460c43e15f2f46", "family": "fsm", "template": 3, "depth": 4, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 8.0}, {"uid": "dd76daa3c9e425c3", "family": "excl", "template": 4, "depth": 1, "pred": "date lime f", "truth": "fig nut lime yam kiwi", "correct": false, "edits_per_token": 8.0}, {"uid": "0c9e513614abcdba", "family": "excl", "template": 4, "depth": 1, "pred": "date plum p", "truth": "pear apple yam", "correct": false, "edits_per_token": 8.0}, {"uid": "369d97e8cff94de0", "family": "excl", "template": 4, "depth": 2, "pred": "41", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "740a7fe786d6b8a9", "family": "sched", "template": 4, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "8fb18ed65388b1e8", "family": "excl", "template": 4, "depth": 4, "pred": "yam", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "29d92bff2c35ea42", "family": "excl", "template": 4, "depth": 4, "pred": "yam", "truth": "lime", "correct": false, "edits_per_token": 8.0}, {"uid": "f8d4d6edfeb5089f", "family": "sched", "template": 4, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "ee81ecf47e6443b2", "family": "sched", "template": 4, "depth": 3, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "0ff325369010cfc0", "family": "sched", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "1af73c5067b38c2c", "family": "excl", "template": 3, "depth": 1, "pred": "fig", "truth": "lime plum apple date", "correct": false, "edits_per_token": 8.0}, {"uid": "78bf6eea827bc579", "family": "excl", "template": 4, "depth": 1, "pred": "apple bean", "truth": "apple yam pear", "correct": false, "edits_per_token": 8.0}, {"uid": "f7efbbd3fc00581b", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "f82b882fabca1cb7", "family": "excl", "template": 4, "depth": 2, "pred": "yam date ki", "truth": "plum apple", "correct": false, "edits_per_token": 8.0}, {"uid": "faf4c370f026c6b8", "family": "sched", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "bfa2044b00bb86b8", "family": "sched", "template": 4, "depth": 3, "pred": "none", "truth": "B", "correct": false, "edits_per_token": 8.0}, {"uid": "4d3505f6a55ac312", "family": "excl", "template": 4, "depth": 4, "pred": "lime", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "1a3d40c434f1d4e3", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "b1b1ea9a4212d1a9", "family": "sched", "template": 4, "depth": 3, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "8c69b9bd3183ebb6", "family": "excl", "template": 3, "depth": 1, "pred": "none", "truth": "pear bean kiwi plum", "correct": false, "edits_per_token": 8.0}, {"uid": "10dc1311d193c3fa", "family": "excl", "template": 4, "depth": 1, "pred": "lime apple", "truth": "lime apple plum bean", "correct": false, "edits_per_token": 8.0}, {"uid": "82d6d862fe063960", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "11b5887e2e23a957", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "ef07c6682c4275d9", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "3335c404ce9bfcd5", "family": "excl", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "a6c279c43201f057", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "ede73a38c44a3ccf", "family": "sched", "template": 4, "depth": 3, "pred": "4", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "3e0de6f98d396f2a", "family": "excl", "template": 4, "depth": 1, "pred": "date pear f", "truth": "kiwi pear bean apple", "correct": false, "edits_per_token": 8.0}, {"uid": "d36a0f43f40b6236", "family": "excl", "template": 4, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 8.0}, {"uid": "ebb9ef7d9d066d95", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "4020752baa5a79b3", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "70e7b96a192ab155", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "ae63f7a481d76063", "family": "excl", "template": 4, "depth": 4, "pred": "none", "truth": "bean lime", "correct": false, "edits_per_token": 8.0}, {"uid": "f3e43a7b75ad9689", "family": "excl", "template": 3, "depth": 1, "pred": "date", "truth": "pear fig bean lime", "correct": false, "edits_per_token": 8.0}, {"uid": "586e75ebfbf62207", "family": "excl", "template": 4, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "9b4fed98a3e64c92", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "694f4323afee9216", "family": "sched", "template": 3, "depth": 1, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "37c09dc4c77fb59b", "family": "excl", "template": 4, "depth": 1, "pred": ". CCemainin", "truth": "apple", "correct": false, "edits_per_token": 8.0}, {"uid": "27852e223cde9fdd", "family": "sched", "template": 3, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "d9a3a976635a4912", "family": "excl", "template": 3, "depth": 1, "pred": "nut", "truth": "kiwi bean date", "correct": false, "edits_per_token": 8.0}, {"uid": "5f308fa3d9d64764", "family": "excl", "template": 3, "depth": 1, "pred": "none", "truth": "lime fig yam", "correct": false, "edits_per_token": 8.0}, {"uid": "796295c79426c62a", "family": "excl", "template": 3, "depth": 3, "pred": "B", "truth": "pear date", "correct": false, "edits_per_token": 8.0}, {"uid": "f59adc461a2a3a23", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "pear fig", "correct": false, "edits_per_token": 8.0}, {"uid": "cb1f068a9f416569", "family": "excl", "template": 4, "depth": 4, "pred": "date", "truth": "yam", "correct": false, "edits_per_token": 8.0}, {"uid": "08d7338346473ac3", "family": "excl", "template": 3, "depth": 2, "pred": "fig", "truth": "pear bean", "correct": false, "edits_per_token": 8.0}, {"uid": "a74bbd4fb82ecde6", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "apple plum", "correct": false, "edits_per_token": 8.0}, {"uid": "023c777eb4956132", "family": "excl", "template": 4, "depth": 3, "pred": "date", "truth": "date pear", "correct": false, "edits_per_token": 8.0}, {"uid": "8aa42a0a282b8980", "family": "excl", "template": 3, "depth": 2, "pred": "date", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "53637f76c85d8c1c", "family": "excl", "template": 3, "depth": 2, "pred": "apple", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "d0cf89610f223fed", "family": "sched", "template": 4, "depth": 4, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 8.0}, {"uid": "9c6fbac6d438fea4", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "38b1c41b753402e8", "family": "excl", "template": 4, "depth": 2, "pred": "plum", "truth": "plum yam nut", "correct": false, "edits_per_token": 8.0}, {"uid": "3e375d575f2a30b3", "family": "sched", "template": 4, "depth": 4, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "40e7271c0285929b", "family": "sched", "template": 4, "depth": 4, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "f62ddd4631b1e632", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "9b8f8f5e9e9a3a18", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "f33bce8dab09faee", "family": "excl", "template": 3, "depth": 3, "pred": "nut", "truth": "yam date", "correct": false, "edits_per_token": 8.0}, {"uid": "f7d02c5b717f1b26", "family": "excl", "template": 3, "depth": 4, "pred": "apple", "truth": "nut", "correct": false, "edits_per_token": 8.0}, {"uid": "2d9756e2a9606211", "family": "excl", "template": 4, "depth": 2, "pred": "bean fig ki", "truth": "bean date yam", "correct": false, "edits_per_token": 8.0}, {"uid": "a64b960f3eeb5d39", "family": "excl", "template": 4, "depth": 3, "pred": "nut", "truth": "lime", "correct": false, "edits_per_token": 8.0}, {"uid": "75be04c29ab1f001", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "7d09a16f57550745", "family": "sched", "template": 3, "depth": 2, "pred": "C", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "5048ccc397f05377", "family": "sched", "template": 3, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "bce8b50e0845842d", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 8.0}, {"uid": "66b605e4db85a2c0", "family": "sched", "template": 4, "depth": 4, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "47a6146933016e02", "family": "excl", "template": 4, "depth": 2, "pred": "date apple", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "7603bb4fbb4710c8", "family": "excl", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "2f57bbfc2ad4b319", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "f93d17df47bca1c9", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "4545d7fc45e828c0", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "f782173e21a6c1f2", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "80a37124c4ed9a39", "family": "excl", "template": 3, "depth": 1, "pred": "nut", "truth": "bean apple pear", "correct": false, "edits_per_token": 8.0}, {"uid": "bcce29e090c586d2", "family": "sched", "template": 3, "depth": 3, "pred": "15", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "193a55393141bab5", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "0324c41dffaa1cb8", "family": "sched", "template": 4, "depth": 4, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "087a279916282ea3", "family": "excl", "template": 4, "depth": 4, "pred": "date", "truth": "nut kiwi", "correct": false, "edits_per_token": 8.0}, {"uid": "9b2c677e487c27cb", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "fd1c02003b3d4554", "family": "sched", "template": 3, "depth": 2, "pred": "C", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "7df915ca06235913", "family": "excl", "template": 3, "depth": 2, "pred": "fig", "truth": "yam", "correct": false, "edits_per_token": 8.0}, {"uid": "1e4798197a754ca9", "family": "excl", "template": 4, "depth": 4, "pred": "um", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "2ca156e74b515e97", "family": "sched", "template": 3, "depth": 2, "pred": "C", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "6a2203bca865b619", "family": "sched", "template": 3, "depth": 2, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "823e185482c10eea", "family": "excl", "template": 3, "depth": 1, "pred": "date", "truth": "pear fig kiwi", "correct": false, "edits_per_token": 8.0}, {"uid": "41e74f93ba171d40", "family": "excl", "template": 4, "depth": 4, "pred": "none", "truth": "bean lime plum", "correct": false, "edits_per_token": 8.0}, {"uid": "57edf49d2957d214", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "nut apple date", "correct": false, "edits_per_token": 8.0}, {"uid": "808bfdae57495704", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "nut pear", "correct": false, "edits_per_token": 8.0}, {"uid": "38d379de426c6ae0", "family": "sched", "template": 3, "depth": 2, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "0b61a24242db3698", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "date", "correct": false, "edits_per_token": 8.0}, {"uid": "9cad194fa2e865be", "family": "excl", "template": 3, "depth": 1, "pred": "fig nut lim", "truth": "fig apple yam lime bean plum", "correct": false, "edits_per_token": 8.0}, {"uid": "16a2b0a2dbc217f0", "family": "excl", "template": 3, "depth": 1, "pred": "plum", "truth": "yam lime apple kiwi nut", "correct": false, "edits_per_token": 8.0}, {"uid": "302f7dfdbcebf7cf", "family": "excl", "template": 4, "depth": 4, "pred": "kiwi", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "88ac831d2cd557d9", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 8.0}, {"uid": "0ea0157cec29a240", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "yam fig plum", "correct": false, "edits_per_token": 8.0}, {"uid": "49165e12e4d4c8d3", "family": "excl", "template": 3, "depth": 3, "pred": "date", "truth": "date bean kiwi", "correct": false, "edits_per_token": 8.0}, {"uid": "575bd19bd31d0b07", "family": "excl", "template": 4, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "3c98968bad82b469", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 8.0}, {"uid": "fa3313c7eee9031c", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "c5b0ad7be361e95a", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 8.0}, {"uid": "6bfdc71318271303", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "0598403107623618", "family": "sched", "template": 3, "depth": 4, "pred": "none", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "28e4acdd7b79568a", "family": "excl", "template": 3, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "38724d20f4d85bad", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "7c8636ccbdad97eb", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "9bd2514eedc4b4e1", "family": "excl", "template": 3, "depth": 3, "pred": "plum", "truth": "nut plum yam date fig", "correct": false, "edits_per_token": 8.0}, {"uid": "5a5831a28a079558", "family": "excl", "template": 3, "depth": 3, "pred": "plum", "truth": "plum apple fig kiwi", "correct": false, "edits_per_token": 8.0}, {"uid": "8ef030fff471d972", "family": "sched", "template": 3, "depth": 3, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "484f0cbf9521eb48", "family": "sched", "template": 3, "depth": 3, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "fdbe9a26369f78cb", "family": "excl", "template": 3, "depth": 2, "pred": "date", "truth": "apple bean pear", "correct": false, "edits_per_token": 8.0}, {"uid": "f2667067c94bd9c2", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "fig", "correct": false, "edits_per_token": 8.0}, {"uid": "636be724170a7508", "family": "excl", "template": 3, "depth": 3, "pred": "date", "truth": "kiwi plum pear yam apple", "correct": false, "edits_per_token": 8.0}, {"uid": "ed83ed43ad98583d", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "0263314c0f8bd6e3", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 8.0}, {"uid": "9ae2d71807ac60ce", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "6a39bb66c31d0c07", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "445a2e8e308d084c", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "D", "correct": false, "edits_per_token": 8.0}, {"uid": "b54ac2bc6f7b24df", "family": "sched", "template": 3, "depth": 4, "pred": "D", "truth": "A", "correct": false, "edits_per_token": 8.0}, {"uid": "20c1afb381170175", "family": "sched", "template": 3, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 8.0}, {"uid": "30430c28d192efc6", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 8.0}, {"uid": "e51b5857bbb2ca50", "family": "sched", "template": 3, "depth": 4, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 8.0}, {"uid": "5b90d13667067fff", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 8.0}, {"uid": "e98454b7446dc760", "family": "excl", "template": 3, "depth": 3, "pred": "none", "truth": "date lime kiwi", "correct": false, "edits_per_token": 8.0}, {"uid": "933b4b87ec87c644", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 8.0}, {"uid": "98b379559d3421a8", "family": "excl", "template": 3, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 8.0}] | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/manifest.json
+169 −0
@@ -0,0 +1,169 @@ | ||
| 1 | +{ | |
| 2 | + "run_id": "p4_editor_v1_Ustd02_seed01_20261029235050", | |
| 3 | + "name": "p4_editor_v1_Ustd02", | |
| 4 | + "seed": 1, | |
| 5 | + "code_commit": "unknown", | |
| 6 | + "config_sha256": "d677af0752c565d76a56b78c1fa0003de1fbd0347957f8df0a21dd0dae86f639", | |
| 7 | + "started_at": "2026-10-29T23:50:50-0400", | |
| 8 | + "host": "M4M64c", | |
| 9 | + "python": "3.12.14", | |
| 10 | + "torch": "2.15.0", | |
| 11 | + "command": "/Users/simon-pierreboucher/wp31/prisme-slm/training/train_editor.py --config configs/experiments/p4_editor_v1_Ustd02.yaml --device mps", | |
| 12 | + "state": "completed", | |
| 13 | + "wall_seconds": 650.11, | |
| 14 | + "device": "mps", | |
| 15 | + "backend": "mps", | |
| 16 | + "precision": "fp32", | |
| 17 | + "n_params": 1765445, | |
| 18 | + "n_trainable": 120577, | |
| 19 | + "checkpoint_parent": "experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint", | |
| 20 | + "mode": "prisme", | |
| 21 | + "freeze_backbone": true, | |
| 22 | + "readout_scale": 5.251033782958984, | |
| 23 | + "finished_at": "2026-10-30T00:01:40-0400", | |
| 24 | + "train_tokens": 331026, | |
| 25 | + "peak_memory_bytes": 8829190144, | |
| 26 | + "final_exact_match": { | |
| 27 | + "dev_iid": { | |
| 28 | + "0": { | |
| 29 | + "accuracy": 0.4375, | |
| 30 | + "n": 400, | |
| 31 | + "by_family": { | |
| 32 | + "fsm": 0.640625, | |
| 33 | + "bind": 0.38372093023255816, | |
| 34 | + "arith": 0.22784810126582278, | |
| 35 | + "excl": 0.27906976744186046, | |
| 36 | + "sched": 0.6941176470588235 | |
| 37 | + }, | |
| 38 | + "edits_per_token": 0.0 | |
| 39 | + }, | |
| 40 | + "2": { | |
| 41 | + "accuracy": 0.4575, | |
| 42 | + "n": 400, | |
| 43 | + "by_family": { | |
| 44 | + "fsm": 0.671875, | |
| 45 | + "bind": 0.3953488372093023, | |
| 46 | + "arith": 0.27848101265822783, | |
| 47 | + "excl": 0.27906976744186046, | |
| 48 | + "sched": 0.7058823529411765 | |
| 49 | + }, | |
| 50 | + "edits_per_token": 2.0 | |
| 51 | + }, | |
| 52 | + "8": { | |
| 53 | + "accuracy": 0.4425, | |
| 54 | + "n": 400, | |
| 55 | + "by_family": { | |
| 56 | + "fsm": 0.671875, | |
| 57 | + "bind": 0.38372093023255816, | |
| 58 | + "arith": 0.21518987341772153, | |
| 59 | + "excl": 0.26744186046511625, | |
| 60 | + "sched": 0.7176470588235294 | |
| 61 | + }, | |
| 62 | + "edits_per_token": 8.0 | |
| 63 | + } | |
| 64 | + }, | |
| 65 | + "dev_template": { | |
| 66 | + "0": { | |
| 67 | + "accuracy": 0.2675, | |
| 68 | + "n": 400, | |
| 69 | + "by_family": { | |
| 70 | + "fsm": 0.4025974025974026, | |
| 71 | + "bind": 0.2222222222222222, | |
| 72 | + "arith": 0.08641975308641975, | |
| 73 | + "sched": 0.475, | |
| 74 | + "excl": 0.16049382716049382 | |
| 75 | + }, | |
| 76 | + "edits_per_token": 0.0 | |
| 77 | + }, | |
| 78 | + "2": { | |
| 79 | + "accuracy": 0.26, | |
| 80 | + "n": 400, | |
| 81 | + "by_family": { | |
| 82 | + "fsm": 0.4025974025974026, | |
| 83 | + "bind": 0.2345679012345679, | |
| 84 | + "arith": 0.07407407407407407, | |
| 85 | + "sched": 0.45, | |
| 86 | + "excl": 0.14814814814814814 | |
| 87 | + }, | |
| 88 | + "edits_per_token": 2.0 | |
| 89 | + }, | |
| 90 | + "8": { | |
| 91 | + "accuracy": 0.23, | |
| 92 | + "n": 400, | |
| 93 | + "by_family": { | |
| 94 | + "fsm": 0.36363636363636365, | |
| 95 | + "bind": 0.19753086419753085, | |
| 96 | + "arith": 0.07407407407407407, | |
| 97 | + "sched": 0.4125, | |
| 98 | + "excl": 0.1111111111111111 | |
| 99 | + }, | |
| 100 | + "edits_per_token": 8.0 | |
| 101 | + } | |
| 102 | + }, | |
| 103 | + "dev_depth": { | |
| 104 | + "0": { | |
| 105 | + "accuracy": 0.3125, | |
| 106 | + "n": 400, | |
| 107 | + "by_family": { | |
| 108 | + "arith": 0.0125, | |
| 109 | + "bind": 0.0875, | |
| 110 | + "fsm": 0.375, | |
| 111 | + "excl": 0.825, | |
| 112 | + "sched": 0.2625 | |
| 113 | + }, | |
| 114 | + "edits_per_token": 0.0 | |
| 115 | + }, | |
| 116 | + "2": { | |
| 117 | + "accuracy": 0.3175, | |
| 118 | + "n": 400, | |
| 119 | + "by_family": { | |
| 120 | + "arith": 0.0125, | |
| 121 | + "bind": 0.0875, | |
| 122 | + "fsm": 0.3875, | |
| 123 | + "excl": 0.825, | |
| 124 | + "sched": 0.275 | |
| 125 | + }, | |
| 126 | + "edits_per_token": 2.0 | |
| 127 | + }, | |
| 128 | + "8": { | |
| 129 | + "accuracy": 0.3075, | |
| 130 | + "n": 400, | |
| 131 | + "by_family": { | |
| 132 | + "arith": 0.0125, | |
| 133 | + "bind": 0.0875, | |
| 134 | + "fsm": 0.375, | |
| 135 | + "excl": 0.8, | |
| 136 | + "sched": 0.2625 | |
| 137 | + }, | |
| 138 | + "edits_per_token": 8.0 | |
| 139 | + } | |
| 140 | + }, | |
| 141 | + "dev_compose": { | |
| 142 | + "0": { | |
| 143 | + "accuracy": 0.0225, | |
| 144 | + "n": 400, | |
| 145 | + "by_family": { | |
| 146 | + "compose": 0.0225 | |
| 147 | + }, | |
| 148 | + "edits_per_token": 0.0 | |
| 149 | + }, | |
| 150 | + "2": { | |
| 151 | + "accuracy": 0.025, | |
| 152 | + "n": 400, | |
| 153 | + "by_family": { | |
| 154 | + "compose": 0.025 | |
| 155 | + }, | |
| 156 | + "edits_per_token": 2.0 | |
| 157 | + }, | |
| 158 | + "8": { | |
| 159 | + "accuracy": 0.03, | |
| 160 | + "n": 400, | |
| 161 | + "by_family": { | |
| 162 | + "compose": 0.03 | |
| 163 | + }, | |
| 164 | + "edits_per_token": 8.0 | |
| 165 | + } | |
| 166 | + } | |
| 167 | + }, | |
| 168 | + "metrics_file": "experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/metrics.jsonl" | |
| 169 | +} | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_Ustd02_seed01_20261029235050/metrics.jsonl
+47 −0
@@ -0,0 +1,47 @@ | ||
| 1 | +{"step": 0, "loss": 0.2580532431602478, "lr": 1e-05, "grad_norm": 0.23523882031440735, "tokens": 146, "tok_per_s": 612.4290144724328, "t": 0.68} | |
| 2 | +{"step": 50, "loss": 0.2984015941619873, "lr": 0.00051, "grad_norm": 0.15599730610847473, "tokens": 8382, "tok_per_s": 728.1660891908106, "t": 11.95} | |
| 3 | +{"step": 100, "loss": 0.2955377399921417, "lr": 0.001, "grad_norm": 0.09142272919416428, "tokens": 16548, "tok_per_s": 728.537435719802, "t": 23.16} | |
| 4 | +{"step": 150, "loss": 0.2785259485244751, "lr": 0.0009984630218530014, "grad_norm": 0.04951321333646774, "tokens": 24812, "tok_per_s": 734.603057487999, "t": 34.22} | |
| 5 | +{"step": 200, "loss": 0.29046913981437683, "lr": 0.0009938625865312252, "grad_norm": 0.0388338677585125, "tokens": 33400, "tok_per_s": 741.6563361894432, "t": 45.48} | |
| 6 | +{"step": 250, "loss": 0.26837068796157837, "lr": 0.0009862301196726987, "grad_norm": 0.027337750419974327, "tokens": 41583, "tok_per_s": 739.9594056582931, "t": 56.64} | |
| 7 | +{"step": 300, "loss": 0.2509698271751404, "lr": 0.0009756177587652855, "grad_norm": 0.023025518283247948, "tokens": 49989, "tok_per_s": 740.2985167140023, "t": 67.97} | |
| 8 | +{"step": 350, "loss": 0.38487178087234497, "lr": 0.0009620979969947759, "grad_norm": 0.1557064801454544, "tokens": 58214, "tok_per_s": 740.3920662747522, "t": 79.07} | |
| 9 | +{"step": 400, "loss": 0.28965502977371216, "lr": 0.0009457631880429201, "grad_norm": 0.02757183648645878, "tokens": 66391, "tok_per_s": 738.8809295625225, "t": 90.3} | |
| 10 | +{"step": 450, "loss": 0.20280273258686066, "lr": 0.0009267249152181378, "grad_norm": 0.07334977388381958, "tokens": 74699, "tok_per_s": 739.2581441097116, "t": 101.49} | |
| 11 | +{"step": 499, "val_loss_b0": 0.41442169654245364, "val_loss_b1": 0.4129690521716363, "val_loss_b2": 0.4127616287163427, "val_loss_b4": 0.4151448494966459, "val_loss_b8": 0.4207350474410036, "t": 114.16} | |
| 12 | +{"step": 500, "loss": 0.3544820547103882, "lr": 0.0009051132292283771, "grad_norm": 0.07510331273078918, "tokens": 83038, "tok_per_s": 728.8281414661615, "t": 114.38} | |
| 13 | +{"step": 550, "loss": 0.30062878131866455, "lr": 0.0008810757598029093, "grad_norm": 0.10836857557296753, "tokens": 91441, "tok_per_s": 731.9181828850625, "t": 125.38} | |
| 14 | +{"step": 600, "loss": 0.36118224263191223, "lr": 0.0008547767072315835, "grad_norm": 0.07137104123830795, "tokens": 99553, "tok_per_s": 731.0711095658081, "t": 136.62} | |
| 15 | +{"step": 650, "loss": 0.294386625289917, "lr": 0.0008263957207103506, "grad_norm": 0.07298284769058228, "tokens": 107666, "tok_per_s": 730.5660116324807, "t": 147.82} | |
| 16 | +{"step": 700, "loss": 0.3116520047187805, "lr": 0.0007961266711550921, "grad_norm": 0.05143333971500397, "tokens": 115959, "tok_per_s": 731.3087261743568, "t": 159.01} | |
| 17 | +{"step": 750, "loss": 0.2591821849346161, "lr": 0.0007641763268666831, "grad_norm": 0.04235551506280899, "tokens": 124302, "tok_per_s": 731.4750818386264, "t": 170.38} | |
| 18 | +{"step": 800, "loss": 0.33459532260894775, "lr": 0.0007307629410938364, "grad_norm": 0.08239558339118958, "tokens": 132734, "tok_per_s": 733.0145098590112, "t": 181.52} | |
| 19 | +{"step": 850, "loss": 0.34336617588996887, "lr": 0.0006961147611421076, "grad_norm": 0.07290179282426834, "tokens": 141025, "tok_per_s": 733.8649183056879, "t": 192.61} | |
| 20 | +{"step": 900, "loss": 0.3378755450248718, "lr": 0.0006604684692133597, "grad_norm": 0.1295628398656845, "tokens": 149035, "tok_per_s": 733.4884209107172, "t": 203.63} | |
| 21 | +{"step": 950, "loss": 0.31969067454338074, "lr": 0.0006240675656263302, "grad_norm": 0.08445288985967636, "tokens": 157135, "tok_per_s": 733.2837633929447, "t": 214.73} | |
| 22 | +{"step": 999, "val_loss_b0": 0.41442169654245364, "val_loss_b1": 0.4114615127200712, "val_loss_b2": 0.4107440534707873, "val_loss_b4": 0.411278999578014, "val_loss_b8": 0.41611300464193673, "t": 227.2} | |
| 23 | +{"step": 1000, "loss": 0.2974494695663452, "lr": 0.0005871607054625496, "grad_norm": 0.06395639479160309, "tokens": 165602, "tok_per_s": 729.6745772577765, "t": 227.4} | |
| 24 | +{"step": 1050, "loss": 0.27726855874061584, "lr": 0.00055, "grad_norm": 0.13304054737091064, "tokens": 173674, "tok_per_s": 728.9295376907252, "t": 238.7} | |
| 25 | +{"step": 1100, "loss": 0.23081760108470917, "lr": 0.0005128392945374505, "grad_norm": 0.09567917138338089, "tokens": 181834, "tok_per_s": 729.0928487383845, "t": 249.84} | |
| 26 | +{"step": 1150, "loss": 0.34129706025123596, "lr": 0.0004759324343736697, "grad_norm": 0.08240954577922821, "tokens": 190158, "tok_per_s": 729.7390719001983, "t": 261.03} | |
| 27 | +{"step": 1200, "loss": 0.3132610321044922, "lr": 0.00043953153078664045, "grad_norm": 0.1229400485754013, "tokens": 198573, "tok_per_s": 730.2571147345977, "t": 272.36} | |
| 28 | +{"step": 1250, "loss": 0.3715328574180603, "lr": 0.00040388523885789254, "grad_norm": 0.15301461517810822, "tokens": 206880, "tok_per_s": 730.6283857069637, "t": 283.6} | |
| 29 | +{"step": 1300, "loss": 0.2495521605014801, "lr": 0.00036923705890616387, "grad_norm": 0.04287685081362724, "tokens": 215095, "tok_per_s": 730.7486552068776, "t": 294.79} | |
| 30 | +{"step": 1350, "loss": 0.2587812542915344, "lr": 0.0003358236731333169, "grad_norm": 0.11645929515361786, "tokens": 223349, "tok_per_s": 731.0613859104535, "t": 305.96} | |
| 31 | +{"step": 1400, "loss": 0.2501791715621948, "lr": 0.00030387332884490803, "grad_norm": 0.1007695347070694, "tokens": 231716, "tok_per_s": 731.3535606308608, "t": 317.27} | |
| 32 | +{"step": 1450, "loss": 0.35891661047935486, "lr": 0.0002736042792896495, "grad_norm": 0.17963343858718872, "tokens": 239960, "tok_per_s": 731.8156869909545, "t": 328.34} | |
| 33 | +{"step": 1499, "val_loss_b0": 0.41442169654245364, "val_loss_b1": 0.40851796080772096, "val_loss_b2": 0.40876816782164893, "val_loss_b4": 0.4131663564764196, "val_loss_b8": 0.42335924882591214, "t": 340.73} | |
| 34 | +{"step": 1499, "checkpoint_sha256": "d4325f11d7f68754631fd07924bd4768d19b957cf8d4daed88b17c746bad1f64", "t": 340.75} | |
| 35 | +{"step": 1500, "loss": 0.26295551657676697, "lr": 0.0002452232927684166, "grad_norm": 0.15522685647010803, "tokens": 248357, "tok_per_s": 729.2761631646307, "t": 340.99} | |
| 36 | +{"step": 1550, "loss": 0.3109534978866577, "lr": 0.0002189242401970908, "grad_norm": 0.16692478954792023, "tokens": 256793, "tok_per_s": 730.2609578959001, "t": 352.09} | |
| 37 | +{"step": 1600, "loss": 0.283221572637558, "lr": 0.00019488677077162295, "grad_norm": 0.1639794409275055, "tokens": 265023, "tok_per_s": 729.8172581207525, "t": 363.58} | |
| 38 | +{"step": 1650, "loss": 0.35360851883888245, "lr": 0.00017327508478186216, "grad_norm": 0.2212497889995575, "tokens": 273283, "tok_per_s": 714.4591261343934, "t": 382.95} | |
| 39 | +{"step": 1700, "loss": 0.35748106241226196, "lr": 0.00015423681195707997, "grad_norm": 0.12465238571166992, "tokens": 281671, "tok_per_s": 699.6272868796068, "t": 403.04} | |
| 40 | +{"step": 1750, "loss": 0.2656850516796112, "lr": 0.00013790200300522413, "grad_norm": 0.07715098559856415, "tokens": 289799, "tok_per_s": 685.71970277419, "t": 423.06} | |
| 41 | +{"step": 1800, "loss": 0.24114304780960083, "lr": 0.00012438224123471442, "grad_norm": 0.07585620880126953, "tokens": 298140, "tok_per_s": 673.4204134982532, "t": 443.17} | |
| 42 | +{"step": 1850, "loss": 0.2697941064834595, "lr": 0.00011376988032730135, "grad_norm": 0.07818277925252914, "tokens": 306518, "tok_per_s": 662.754582787904, "t": 462.93} | |
| 43 | +{"step": 1900, "loss": 0.27652421593666077, "lr": 0.00010613741346877497, "grad_norm": 0.10166487842798233, "tokens": 314885, "tok_per_s": 652.3479962923444, "t": 483.14} | |
| 44 | +{"step": 1950, "loss": 0.20917129516601562, "lr": 0.00010153697814699858, "grad_norm": 0.10033971816301346, "tokens": 322988, "tok_per_s": 642.2730058727624, "t": 503.32} | |
| 45 | +{"step": 1999, "loss": 0.2973034679889679, "lr": 0.00010000061514140841, "grad_norm": 0.09640076756477356, "tokens": 331026, "tok_per_s": 633.4933472510902, "t": 522.98} | |
| 46 | +{"step": 1999, "val_loss_b0": 0.41442169654245364, "val_loss_b1": 0.4078555936090659, "val_loss_b2": 0.408420847571091, "val_loss_b4": 0.41316756238554775, "val_loss_b8": 0.4216234737129835, "t": 526.21} | |
| 47 | +{"step": 1999, "checkpoint_sha256": "aabf646812efd9f1fdedbcccbf73004a44516484ad036798fdf3a7b80e5cefb0", "t": 526.23} | |
added
experiments/runs/p4_editor_v1_scaled_seed01_20261029234334/checkpoint/checkpoint.json
+36 −0
@@ -0,0 +1,36 @@ | ||
| 1 | +{ | |
| 2 | + "config": { | |
| 3 | + "name": "tiny_p4", | |
| 4 | + "vocab_size": 128, | |
| 5 | + "d_model": 192, | |
| 6 | + "n_layers": 4, | |
| 7 | + "n_heads": 6, | |
| 8 | + "n_kv_heads": 2, | |
| 9 | + "ffn_width": 512, | |
| 10 | + "max_seq_len": 320, | |
| 11 | + "rope_theta": 10000.0, | |
| 12 | + "norm_eps": 1e-05, | |
| 13 | + "tie_embeddings": true, | |
| 14 | + "init_std": 0.02, | |
| 15 | + "use_workspace": true, | |
| 16 | + "w": 64, | |
| 17 | + "ffn_w": 192, | |
| 18 | + "n_slots": 16, | |
| 19 | + "n_editor_blocks": 2, | |
| 20 | + "n_editor_heads": 2, | |
| 21 | + "r_max": 8, | |
| 22 | + "eta_max": 0.25, | |
| 23 | + "clip_c": 1.0, | |
| 24 | + "clip_eps": 1e-06, | |
| 25 | + "readout_zero_init": true, | |
| 26 | + "readout_scale": 5.251033782958984, | |
| 27 | + "controller_hidden": 0, | |
| 28 | + "lambda_cost": 0.0, | |
| 29 | + "kappa_risk": 0.0, | |
| 30 | + "dense_writes": false | |
| 31 | + }, | |
| 32 | + "saved_at": "2026-10-30T00:18:58-0400", | |
| 33 | + "step": 3000, | |
| 34 | + "run_id": "p4_editor_v1_scaled_seed01_20261029234334", | |
| 35 | + "weights_sha256": "a1f686a87bcbb5c3da1dbc90e3048b457a7dd541388accedd820e1be47fb21e1" | |
| 36 | +} | |
| \ No newline at end of file | ||
added
experiments/runs/p4_editor_v1_scaled_seed01_20261029234334/em_dev_iid_b0.json
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +[{"uid": "d57de33b3b3613ce", "family": "fsm", "template": 2, "depth": 1, "pred": "8", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "2fcbef2f6deb569d", "family": "bind", "template": 2, "depth": 1, "pred": "18", "truth": "17", "correct": false, "edits_per_token": 0.0}, {"uid": "facfe4b45751726b", "family": "bind", "template": 2, "depth": 1, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 0.0}, {"uid": "17c2f737ab6e9a12", "family": "bind", "template": 2, "depth": 1, "pred": "21", "truth": "21", "correct": true, "edits_per_token": 0.0}, {"uid": "330a5096d00ea26d", "family": "bind", "template": 2, "depth": 1, "pred": "6", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "67c60958afd432d0", "family": "bind", "template": 2, "depth": 1, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "0df874a819030335", "family": "bind", "template": 2, "depth": 1, "pred": "60", "truth": "60", "correct": true, "edits_per_token": 0.0}, {"uid": "f8b6c686efa42185", "family": "bind", "template": 2, "depth": 1, "pred": "27", "truth": "26", "correct": false, "edits_per_token": 0.0}, {"uid": "61ab4c048732d1a3", "family": "fsm", "template": 2, "depth": 1, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "a0a9effb7a5a1771", "family": "fsm", "template": 2, "depth": 1, "pred": "4", "truth": "5", "correct": false, "edits_per_token": 0.0}, {"uid": "e539dc5c812c9f8f", "family": "arith", "template": 0, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "b9080d7276f3e48d", "family": "arith", "template": 1, "depth": 1, "pred": "14", "truth": "14", "correct": true, "edits_per_token": 0.0}, {"uid": "f19b2d158a1b2321", "family": "arith", "template": 0, "depth": 1, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "b50334287c335a68", "family": "arith", "template": 0, "depth": 1, "pred": "13", "truth": "13", "correct": true, "edits_per_token": 0.0}, {"uid": "5203cff28a90efc8", "family": "arith", "template": 0, "depth": 1, "pred": "14", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "b0fc8aed8de80f82", "family": "arith", "template": 1, "depth": 1, "pred": "85", "truth": "85", "correct": true, "edits_per_token": 0.0}, {"uid": "d3f8213a4e9e4170", "family": "arith", "template": 1, "depth": 1, "pred": "64", "truth": "66", "correct": false, "edits_per_token": 0.0}, {"uid": "6dbb743f0c127dc2", "family": "arith", "template": 1, "depth": 1, "pred": "100", "truth": "120", "correct": false, "edits_per_token": 0.0}, {"uid": "94bfbb133f315b17", "family": "arith", "template": 0, "depth": 1, "pred": "37", "truth": "37", "correct": true, "edits_per_token": 0.0}, {"uid": "c5dd86b6955d4021", "family": "arith", "template": 1, "depth": 1, "pred": "-7", "truth": "-7", "correct": true, "edits_per_token": 0.0}, {"uid": "75eceacb14c3d0e6", "family": "bind", "template": 0, "depth": 1, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "f3b82055a8272259", "family": "bind", "template": 0, "depth": 1, "pred": "18", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "6fa8f401c1e6cf13", "family": "bind", "template": 0, "depth": 1, "pred": "22", "truth": "22", "correct": true, "edits_per_token": 0.0}, {"uid": "5252b66c833e4804", "family": "bind", "template": 0, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "0dd2fab9c5861d6d", "family": "bind", "template": 0, "depth": 1, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 0.0}, {"uid": "d8aa01c2e9dbb03f", "family": "bind", "template": 1, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "1d20f4c03960f839", "family": "arith", "template": 0, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "a0d75f1874311fde", "family": "bind", "template": 0, "depth": 1, "pred": "26", "truth": "27", "correct": false, "edits_per_token": 0.0}, {"uid": "2f252bbc438ce228", "family": "bind", "template": 0, "depth": 1, "pred": "23", "truth": "24", "correct": false, "edits_per_token": 0.0}, {"uid": "910350946f41fbf1", "family": "bind", "template": 1, "depth": 1, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "322ec1e8c6acc1ff", "family": "bind", "template": 1, "depth": 1, "pred": "24", "truth": "24", "correct": true, "edits_per_token": 0.0}, {"uid": "8f08654666473a62", "family": "bind", "template": 1, "depth": 1, "pred": "8", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "e74c59b882dcf2b2", "family": "bind", "template": 1, "depth": 1, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 0.0}, {"uid": "f4ea26539e1674ec", "family": "arith", "template": 0, "depth": 2, "pred": "28", "truth": "28", "correct": true, "edits_per_token": 0.0}, {"uid": "8461e207ba7b8587", "family": "arith", "template": 1, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "d43d09af762bceee", "family": "arith", "template": 1, "depth": 2, "pred": "41", "truth": "44", "correct": false, "edits_per_token": 0.0}, {"uid": "7ed459fad79fdf0b", "family": "bind", "template": 1, "depth": 1, "pred": "13", "truth": "12", "correct": false, "edits_per_token": 0.0}, {"uid": "ce3cbdbb1894d8f4", "family": "bind", "template": 1, "depth": 1, "pred": "14", "truth": "15", "correct": false, "edits_per_token": 0.0}, {"uid": "14947545562906ad", "family": "bind", "template": 1, "depth": 1, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "697d570c40384635", "family": "bind", "template": 2, "depth": 2, "pred": "11", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "b78e61c19225f49c", "family": "fsm", "template": 2, "depth": 2, "pred": "19", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "69953fd72fce090d", "family": "arith", "template": 0, "depth": 2, "pred": "36", "truth": "38", "correct": false, "edits_per_token": 0.0}, {"uid": "66cdadf971fee2ab", "family": "arith", "template": 0, "depth": 2, "pred": "69", "truth": "71", "correct": false, "edits_per_token": 0.0}, {"uid": "99c7a494c455e41d", "family": "arith", "template": 0, "depth": 2, "pred": "-64", "truth": "-60", "correct": false, "edits_per_token": 0.0}, {"uid": "8d21bd8bbb5eecad", "family": "arith", "template": 0, "depth": 2, "pred": "42", "truth": "41", "correct": false, "edits_per_token": 0.0}, {"uid": "5f3cf3ff50e3e9c1", "family": "arith", "template": 1, "depth": 2, "pred": "40", "truth": "41", "correct": false, "edits_per_token": 0.0}, {"uid": "ce6f8c794a4d97fb", "family": "arith", "template": 1, "depth": 2, "pred": "51", "truth": "55", "correct": false, "edits_per_token": 0.0}, {"uid": "56607dee807b8fd6", "family": "bind", "template": 2, "depth": 2, "pred": "17", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "206a446298d63b02", "family": "bind", "template": 2, "depth": 2, "pred": "40", "truth": "40", "correct": true, "edits_per_token": 0.0}, {"uid": "4dd1c85d972bb798", "family": "bind", "template": 2, "depth": 2, "pred": "-5", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "60d954acf09287f1", "family": "arith", "template": 0, "depth": 2, "pred": "48", "truth": "46", "correct": false, "edits_per_token": 0.0}, {"uid": "5521b704b5ac25a7", "family": "arith", "template": 1, "depth": 2, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 0.0}, {"uid": "b66b330f45989ec2", "family": "arith", "template": 1, "depth": 2, "pred": "12", "truth": "13", "correct": false, "edits_per_token": 0.0}, {"uid": "f06a723961f75f4d", "family": "arith", "template": 1, "depth": 2, "pred": "-34", "truth": "-36", "correct": false, "edits_per_token": 0.0}, {"uid": "1ffed152f8b72d39", "family": "bind", "template": 2, "depth": 2, "pred": "4", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "3b4d5aeb1e813243", "family": "arith", "template": 0, "depth": 3, "pred": "108", "truth": "140", "correct": false, "edits_per_token": 0.0}, {"uid": "e56fea6bf4230916", "family": "bind", "template": 2, "depth": 2, "pred": "0", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "9f5c3b31e7d60900", "family": "bind", "template": 2, "depth": 2, "pred": "28", "truth": "31", "correct": false, "edits_per_token": 0.0}, {"uid": "ba60a76e7ce73c9a", "family": "arith", "template": 0, "depth": 3, "pred": "165", "truth": "165", "correct": true, "edits_per_token": 0.0}, {"uid": "caa2ffb073e3b554", "family": "arith", "template": 0, "depth": 3, "pred": "-104", "truth": "-80", "correct": false, "edits_per_token": 0.0}, {"uid": "f3df8ca3c7f3fe73", "family": "arith", "template": 0, "depth": 3, "pred": "-350", "truth": "-300", "correct": false, "edits_per_token": 0.0}, {"uid": "6a521ae83b9560cb", "family": "arith", "template": 0, "depth": 3, "pred": "308", "truth": "395", "correct": false, "edits_per_token": 0.0}, {"uid": "b7e7635fb68761fa", "family": "arith", "template": 1, "depth": 3, "pred": "80", "truth": "78", "correct": false, "edits_per_token": 0.0}, {"uid": "69168d7303f42428", "family": "arith", "template": 1, "depth": 3, "pred": "368", "truth": "352", "correct": false, "edits_per_token": 0.0}, {"uid": "3876cda47832bc78", "family": "arith", "template": 1, "depth": 3, "pred": "60", "truth": "66", "correct": false, "edits_per_token": 0.0}, {"uid": "101a933392cdc486", "family": "arith", "template": 0, "depth": 3, "pred": "-11", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "5d6734a6c29bf98b", "family": "arith", "template": 1, "depth": 3, "pred": "-36", "truth": "-38", "correct": false, "edits_per_token": 0.0}, {"uid": "e6b4bb7a789c1bbf", "family": "arith", "template": 1, "depth": 3, "pred": "-36", "truth": "-36", "correct": true, "edits_per_token": 0.0}, {"uid": "529ef2fbb6693bf4", "family": "arith", "template": 0, "depth": 3, "pred": "23", "truth": "27", "correct": false, "edits_per_token": 0.0}, {"uid": "4264ff894995b84a", "family": "arith", "template": 0, "depth": 4, "pred": "111", "truth": "69", "correct": false, "edits_per_token": 0.0}, {"uid": "4fd0a06acba9c2bd", "family": "arith", "template": 1, "depth": 3, "pred": "77", "truth": "75", "correct": false, "edits_per_token": 0.0}, {"uid": "11ddfaa4634944bc", "family": "arith", "template": 1, "depth": 3, "pred": "-12", "truth": "-11", "correct": false, "edits_per_token": 0.0}, {"uid": "81903019667d210a", "family": "arith", "template": 0, "depth": 4, "pred": "-21", "truth": "-24", "correct": false, "edits_per_token": 0.0}, {"uid": "f025613adc1381e6", "family": "arith", "template": 0, "depth": 4, "pred": "67", "truth": "67", "correct": true, "edits_per_token": 0.0}, {"uid": "6917d247eea87098", "family": "arith", "template": 1, "depth": 4, "pred": "1720", "truth": "1400", "correct": false, "edits_per_token": 0.0}, {"uid": "5ac4cc428fcee7ff", "family": "bind", "template": 0, "depth": 2, "pred": "112", "truth": "180", "correct": false, "edits_per_token": 0.0}, {"uid": "a7fa5ddf82e3acb6", "family": "bind", "template": 0, "depth": 2, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 0.0}, {"uid": "85a6f3db0aee68b6", "family": "fsm", "template": 0, "depth": 1, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "306790bc18e1ff0f", "family": "fsm", "template": 1, "depth": 1, "pred": "-5", "truth": "-5", "correct": true, "edits_per_token": 0.0}, {"uid": "7b35ad2e31d3fe01", "family": "fsm", "template": 1, "depth": 1, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "f692ca3a30b9615b", "family": "arith", "template": 0, "depth": 4, "pred": "27", "truth": "27", "correct": true, "edits_per_token": 0.0}, {"uid": "00456af1ebb8a6f6", "family": "arith", "template": 1, "depth": 4, "pred": "-16", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "8636f76e7a05df5c", "family": "bind", "template": 0, "depth": 2, "pred": "-15", "truth": "-10", "correct": false, "edits_per_token": 0.0}, {"uid": "6894af54caeca8f0", "family": "bind", "template": 0, "depth": 2, "pred": "56", "truth": "56", "correct": true, "edits_per_token": 0.0}, {"uid": "15a26f31dc193ec8", "family": "bind", "template": 0, "depth": 2, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 0.0}, {"uid": "fbf1f2cb25ea61f5", "family": "bind", "template": 0, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "e1c776eb2e759087", "family": "arith", "template": 0, "depth": 4, "pred": "77", "truth": "66", "correct": false, "edits_per_token": 0.0}, {"uid": "b57b7bb076a588da", "family": "arith", "template": 1, "depth": 4, "pred": "368", "truth": "480", "correct": false, "edits_per_token": 0.0}, {"uid": "c96adbcbd0c51bd0", "family": "arith", "template": 1, "depth": 4, "pred": "50", "truth": "52", "correct": false, "edits_per_token": 0.0}, {"uid": "058b50085974c009", "family": "arith", "template": 1, "depth": 4, "pred": "19", "truth": "19", "correct": true, "edits_per_token": 0.0}, {"uid": "cbc5614b2da64a6a", "family": "arith", "template": 1, "depth": 4, "pred": "-48", "truth": "-50", "correct": false, "edits_per_token": 0.0}, {"uid": "ffd3f5ed636cb166", "family": "bind", "template": 0, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "e48b5252059e24a7", "family": "bind", "template": 1, "depth": 2, "pred": "-8", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "58cc6ff1fb3e1166", "family": "bind", "template": 1, "depth": 2, "pred": "15", "truth": "15", "correct": true, "edits_per_token": 0.0}, {"uid": "9dbdbf7552d0f726", "family": "bind", "template": 1, "depth": 2, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 0.0}, {"uid": "02f04b4ab6addf22", "family": "fsm", "template": 2, "depth": 2, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 0.0}, {"uid": "a7df7a7b48f0e07c", "family": "fsm", "template": 2, "depth": 2, "pred": "-3", "truth": "-3", "correct": true, "edits_per_token": 0.0}, {"uid": "4d05c02585d8db9c", "family": "fsm", "template": 2, "depth": 2, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "54b414556cb4b641", "family": "fsm", "template": 2, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "afe7716d466e8527", "family": "arith", "template": 0, "depth": 4, "pred": "-12", "truth": "-10", "correct": false, "edits_per_token": 0.0}, {"uid": "a3a41fe21c3b68c3", "family": "arith", "template": 0, "depth": 4, "pred": "-93", "truth": "-106", "correct": false, "edits_per_token": 0.0}, {"uid": "1e60abcad3f7cd49", "family": "arith", "template": 1, "depth": 4, "pred": "26", "truth": "29", "correct": false, "edits_per_token": 0.0}, {"uid": "5b477191fed1d1ab", "family": "bind", "template": 1, "depth": 2, "pred": "-8", "truth": "-8", "correct": true, "edits_per_token": 0.0}, {"uid": "b618508093791a17", "family": "bind", "template": 1, "depth": 2, "pred": "210", "truth": "240", "correct": false, "edits_per_token": 0.0}, {"uid": "758626244ff1e265", "family": "bind", "template": 1, "depth": 2, "pred": "45", "truth": "45", "correct": true, "edits_per_token": 0.0}, {"uid": "53f6d7daa1e917ac", "family": "bind", "template": 2, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "7a489d579355664c", "family": "arith", "template": 2, "depth": 1, "pred": "48", "truth": "48", "correct": true, "edits_per_token": 0.0}, {"uid": "866964b48f15dc73", "family": "arith", "template": 2, "depth": 1, "pred": "22", "truth": "21", "correct": false, "edits_per_token": 0.0}, {"uid": "326ed5326c7a15fa", "family": "arith", "template": 2, "depth": 1, "pred": "64", "truth": "60", "correct": false, "edits_per_token": 0.0}, {"uid": "78b68accb86904aa", "family": "bind", "template": 1, "depth": 2, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 0.0}, {"uid": "12ab692979396f5c", "family": "bind", "template": 2, "depth": 3, "pred": "45", "truth": "36", "correct": false, "edits_per_token": 0.0}, {"uid": "4b618bfde19bcd4c", "family": "bind", "template": 2, "depth": 3, "pred": "-8", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "99b184a684af7db3", "family": "bind", "template": 2, "depth": 3, "pred": "-12", "truth": "-15", "correct": false, "edits_per_token": 0.0}, {"uid": "09b2aab4939d318c", "family": "fsm", "template": 2, "depth": 3, "pred": "19", "truth": "19", "correct": true, "edits_per_token": 0.0}, {"uid": "b93e590222b68eba", "family": "arith", "template": 2, "depth": 1, "pred": "-18", "truth": "-18", "correct": true, "edits_per_token": 0.0}, {"uid": "e12b670d52d4bcac", "family": "arith", "template": 2, "depth": 1, "pred": "31", "truth": "34", "correct": false, "edits_per_token": 0.0}, {"uid": "4247b2f7e094279e", "family": "bind", "template": 2, "depth": 3, "pred": "19", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "67dfb1d3b0773f0e", "family": "bind", "template": 2, "depth": 3, "pred": "48", "truth": "43", "correct": false, "edits_per_token": 0.0}, {"uid": "8254a0f979194a20", "family": "bind", "template": 2, "depth": 3, "pred": "10", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "7e2f3ac39edf259e", "family": "bind", "template": 2, "depth": 3, "pred": "29", "truth": "29", "correct": true, "edits_per_token": 0.0}, {"uid": "f78cbce45d518768", "family": "fsm", "template": 0, "depth": 1, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 0.0}, {"uid": "67327201e0703535", "family": "fsm", "template": 1, "depth": 2, "pred": "16", "truth": "15", "correct": false, "edits_per_token": 0.0}, {"uid": "ad02a284f829c979", "family": "fsm", "template": 1, "depth": 2, "pred": "16", "truth": "15", "correct": false, "edits_per_token": 0.0}, {"uid": "d66d39facf8c09dc", "family": "arith", "template": 2, "depth": 2, "pred": "22", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "961f7a505cc9733d", "family": "arith", "template": 2, "depth": 2, "pred": "-108", "truth": "-120", "correct": false, "edits_per_token": 0.0}, {"uid": "2a66eac56efc842a", "family": "arith", "template": 2, "depth": 2, "pred": "180", "truth": "195", "correct": false, "edits_per_token": 0.0}, {"uid": "0b6aca1ee4bcc5e4", "family": "arith", "template": 2, "depth": 2, "pred": "-48", "truth": "-54", "correct": false, "edits_per_token": 0.0}, {"uid": "a7520c88e128325d", "family": "arith", "template": 2, "depth": 2, "pred": "7", "truth": "9", "correct": false, "edits_per_token": 0.0}, {"uid": "a4a050b886f6d6f1", "family": "arith", "template": 2, "depth": 2, "pred": "64", "truth": "68", "correct": false, "edits_per_token": 0.0}, {"uid": "822a99424bbcf8ea", "family": "arith", "template": 2, "depth": 2, "pred": "40", "truth": "42", "correct": false, "edits_per_token": 0.0}, {"uid": "8fbd3d2ce9fd04d2", "family": "bind", "template": 0, "depth": 3, "pred": "57", "truth": "60", "correct": false, "edits_per_token": 0.0}, {"uid": "2ef3a2f2a0328157", "family": "bind", "template": 0, "depth": 3, "pred": "36", "truth": "32", "correct": false, "edits_per_token": 0.0}, {"uid": "588fe9ad929612f6", "family": "bind", "template": 0, "depth": 3, "pred": "10", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "3db5b222471f96fa", "family": "bind", "template": 0, "depth": 3, "pred": "20", "truth": "20", "correct": true, "edits_per_token": 0.0}, {"uid": "b96935b62f495f58", "family": "arith", "template": 2, "depth": 3, "pred": "119", "truth": "149", "correct": false, "edits_per_token": 0.0}, {"uid": "4ce4e40dc4a8caf4", "family": "bind", "template": 0, "depth": 3, "pred": "80", "truth": "64", "correct": false, "edits_per_token": 0.0}, {"uid": "0acc51832dea68c1", "family": "bind", "template": 0, "depth": 3, "pred": "21", "truth": "23", "correct": false, "edits_per_token": 0.0}, {"uid": "6f0f31272e9dccb8", "family": "fsm", "template": 1, "depth": 2, "pred": "20", "truth": "22", "correct": false, "edits_per_token": 0.0}, {"uid": "43de2e7618ca4cb3", "family": "fsm", "template": 2, "depth": 3, "pred": "20", "truth": "18", "correct": false, "edits_per_token": 0.0}, {"uid": "ffb7dab9fc31fbd5", "family": "arith", "template": 2, "depth": 3, "pred": "44", "truth": "42", "correct": false, "edits_per_token": 0.0}, {"uid": "25ab3f6cf0620687", "family": "arith", "template": 2, "depth": 3, "pred": "40", "truth": "38", "correct": false, "edits_per_token": 0.0}, {"uid": "6e8c3b9ad7c915f8", "family": "arith", "template": 2, "depth": 3, "pred": "-14", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "595ac6801eedc4f3", "family": "bind", "template": 0, "depth": 3, "pred": "12", "truth": "12", "correct": true, "edits_per_token": 0.0}, {"uid": "76fdb2e6417c6f93", "family": "bind", "template": 1, "depth": 3, "pred": "12", "truth": "11", "correct": false, "edits_per_token": 0.0}, {"uid": "d8e6469a252c53ab", "family": "fsm", "template": 2, "depth": 3, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "aa252e805cf95b6a", "family": "arith", "template": 2, "depth": 3, "pred": "17", "truth": "20", "correct": false, "edits_per_token": 0.0}, {"uid": "e2bce580fe2875d8", "family": "arith", "template": 2, "depth": 3, "pred": "62", "truth": "60", "correct": false, "edits_per_token": 0.0}, {"uid": "a3811535637f42cf", "family": "arith", "template": 2, "depth": 3, "pred": "-14", "truth": "-13", "correct": false, "edits_per_token": 0.0}, {"uid": "920633664a3100c7", "family": "arith", "template": 2, "depth": 3, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "6d1db4bf8883e127", "family": "bind", "template": 1, "depth": 3, "pred": "22", "truth": "22", "correct": true, "edits_per_token": 0.0}, {"uid": "42b7111968d0fd6e", "family": "bind", "template": 1, "depth": 3, "pred": "77", "truth": "74", "correct": false, "edits_per_token": 0.0}, {"uid": "0d7d21b45a123ecc", "family": "bind", "template": 1, "depth": 3, "pred": "27", "truth": "28", "correct": false, "edits_per_token": 0.0}, {"uid": "0070225255190df8", "family": "bind", "template": 1, "depth": 3, "pred": "-5", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "9efc725b639fa1e2", "family": "bind", "template": 1, "depth": 3, "pred": "41", "truth": "42", "correct": false, "edits_per_token": 0.0}, {"uid": "e6d2938cf240e7d7", "family": "bind", "template": 1, "depth": 3, "pred": "-1", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "6333807fc55e47c2", "family": "bind", "template": 2, "depth": 4, "pred": "27", "truth": "27", "correct": true, "edits_per_token": 0.0}, {"uid": "73bab13d0adb93c9", "family": "bind", "template": 2, "depth": 4, "pred": "33", "truth": "31", "correct": false, "edits_per_token": 0.0}, {"uid": "7e4a5eedc90f317e", "family": "arith", "template": 2, "depth": 4, "pred": "577", "truth": "707", "correct": false, "edits_per_token": 0.0}, {"uid": "72ab6c11b6b99a0c", "family": "bind", "template": 2, "depth": 4, "pred": "8", "truth": "7", "correct": false, "edits_per_token": 0.0}, {"uid": "82d8e4541e035995", "family": "bind", "template": 2, "depth": 4, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 0.0}, {"uid": "11248162bd234998", "family": "bind", "template": 2, "depth": 4, "pred": "61", "truth": "61", "correct": true, "edits_per_token": 0.0}, {"uid": "30512b180423dd81", "family": "fsm", "template": 0, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "5fff7cb1816a8b83", "family": "arith", "template": 2, "depth": 4, "pred": "-160", "truth": "-90", "correct": false, "edits_per_token": 0.0}, {"uid": "916e8e5dc57e71b5", "family": "arith", "template": 2, "depth": 4, "pred": "37", "truth": "33", "correct": false, "edits_per_token": 0.0}, {"uid": "86a2ceda3911a38b", "family": "arith", "template": 2, "depth": 4, "pred": "141", "truth": "165", "correct": false, "edits_per_token": 0.0}, {"uid": "407e416b5a6578ae", "family": "arith", "template": 2, "depth": 4, "pred": "216", "truth": "222", "correct": false, "edits_per_token": 0.0}, {"uid": "f3df5ab6d440dc66", "family": "bind", "template": 2, "depth": 4, "pred": "38", "truth": "40", "correct": false, "edits_per_token": 0.0}, {"uid": "d8aa4893aec3878d", "family": "bind", "template": 2, "depth": 4, "pred": "288", "truth": "316", "correct": false, "edits_per_token": 0.0}, {"uid": "effc949016a8ea12", "family": "fsm", "template": 0, "depth": 2, "pred": "10", "truth": "10", "correct": true, "edits_per_token": 0.0}, {"uid": "efbc892e6b4a8d10", "family": "fsm", "template": 1, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "9ed8a702224d9157", "family": "fsm", "template": 1, "depth": 2, "pred": "3", "truth": "3", "correct": true, "edits_per_token": 0.0}, {"uid": "447dc7313518c9cc", "family": "arith", "template": 2, "depth": 4, "pred": "-40", "truth": "-41", "correct": false, "edits_per_token": 0.0}, {"uid": "c9f30e7c90083e3b", "family": "arith", "template": 2, "depth": 4, "pred": "58", "truth": "70", "correct": false, "edits_per_token": 0.0}, {"uid": "414ba9ab6e9d47f0", "family": "fsm", "template": 2, "depth": 3, "pred": "-6", "truth": "-6", "correct": true, "edits_per_token": 0.0}, {"uid": "d6bbe76ab7ae5dea", "family": "bind", "template": 0, "depth": 4, "pred": "5", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "ad80d63be705294d", "family": "fsm", "template": 0, "depth": 2, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "69d134b029d15133", "family": "bind", "template": 0, "depth": 4, "pred": "-10", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "1ed0d6ff18c8180d", "family": "bind", "template": 0, "depth": 4, "pred": "11", "truth": "11", "correct": true, "edits_per_token": 0.0}, {"uid": "9679ca491537d4ae", "family": "fsm", "template": 0, "depth": 2, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "06e9560336c69fc0", "family": "bind", "template": 0, "depth": 4, "pred": "100", "truth": "99", "correct": false, "edits_per_token": 0.0}, {"uid": "83785d82a70218c2", "family": "bind", "template": 0, "depth": 4, "pred": "12", "truth": "10", "correct": false, "edits_per_token": 0.0}, {"uid": "365ba7f8ce47ab26", "family": "bind", "template": 0, "depth": 4, "pred": "54", "truth": "51", "correct": false, "edits_per_token": 0.0}, {"uid": "f0b461762a54a509", "family": "bind", "template": 1, "depth": 4, "pred": "680", "truth": "368", "correct": false, "edits_per_token": 0.0}, {"uid": "ef8628813a4db970", "family": "bind", "template": 0, "depth": 4, "pred": "60", "truth": "64", "correct": false, "edits_per_token": 0.0}, {"uid": "a07ed018d16711ef", "family": "bind", "template": 1, "depth": 4, "pred": "101", "truth": "81", "correct": false, "edits_per_token": 0.0}, {"uid": "8566721d2752b3a8", "family": "bind", "template": 1, "depth": 4, "pred": "-12", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "14e0f0613ab5fafb", "family": "bind", "template": 1, "depth": 4, "pred": "43", "truth": "41", "correct": false, "edits_per_token": 0.0}, {"uid": "e5b10de969f4f1de", "family": "bind", "template": 1, "depth": 4, "pred": "-20", "truth": "-22", "correct": false, "edits_per_token": 0.0}, {"uid": "9a6304b25504acd5", "family": "fsm", "template": 2, "depth": 4, "pred": "13", "truth": "12", "correct": false, "edits_per_token": 0.0}, {"uid": "f9e4c5acf05b6575", "family": "bind", "template": 1, "depth": 4, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 0.0}, {"uid": "2efa0c713c22db5a", "family": "bind", "template": 1, "depth": 4, "pred": "102", "truth": "101", "correct": false, "edits_per_token": 0.0}, {"uid": "8ce65aa06b428682", "family": "fsm", "template": 0, "depth": 3, "pred": "22", "truth": "24", "correct": false, "edits_per_token": 0.0}, {"uid": "4f7a4c9e9fd88e23", "family": "fsm", "template": 1, "depth": 3, "pred": "-4", "truth": "-4", "correct": true, "edits_per_token": 0.0}, {"uid": "2468d102e506d6d5", "family": "fsm", "template": 2, "depth": 3, "pred": "12", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "6f3c8de73142af82", "family": "fsm", "template": 2, "depth": 4, "pred": "35", "truth": "37", "correct": false, "edits_per_token": 0.0}, {"uid": "ba3af2217536a7c6", "family": "fsm", "template": 0, "depth": 2, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "e063912681967ddf", "family": "fsm", "template": 0, "depth": 3, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 0.0}, {"uid": "583473e83be2cb0e", "family": "fsm", "template": 2, "depth": 3, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 0.0}, {"uid": "5a873e995116c10d", "family": "fsm", "template": 1, "depth": 3, "pred": "80", "truth": "72", "correct": false, "edits_per_token": 0.0}, {"uid": "05764ff46e4d28b8", "family": "fsm", "template": 2, "depth": 4, "pred": "8", "truth": "8", "correct": true, "edits_per_token": 0.0}, {"uid": "46b570766ec22e2a", "family": "fsm", "template": 0, "depth": 3, "pred": "2", "truth": "3", "correct": false, "edits_per_token": 0.0}, {"uid": "59f43bee78d1b7d5", "family": "fsm", "template": 0, "depth": 3, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 0.0}, {"uid": "eaf14d7290a1413b", "family": "fsm", "template": 1, "depth": 3, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "b7248f02464a9d5e", "family": "fsm", "template": 1, "depth": 3, "pred": "-3", "truth": "-2", "correct": false, "edits_per_token": 0.0}, {"uid": "9aaa1b5b75c855bb", "family": "fsm", "template": 2, "depth": 3, "pred": "0", "truth": "0", "correct": true, "edits_per_token": 0.0}, {"uid": "a7203ce2418d2ce9", "family": "fsm", "template": 0, "depth": 4, "pred": "18", "truth": "18", "correct": true, "edits_per_token": 0.0}, {"uid": "b86f2cf77ff75b95", "family": "fsm", "template": 2, "depth": 4, "pred": "9", "truth": "6", "correct": false, "edits_per_token": 0.0}, {"uid": "5480488e0f55e2a2", "family": "fsm", "template": 0, "depth": 3, "pred": "7", "truth": "7", "correct": true, "edits_per_token": 0.0}, {"uid": "ca83fc2e201063f6", "family": "fsm", "template": 0, "depth": 3, "pred": "1", "truth": "1", "correct": true, "edits_per_token": 0.0}, {"uid": "324c8cdae8f3169c", "family": "fsm", "template": 1, "depth": 4, "pred": "-4", "truth": "-6", "correct": false, "edits_per_token": 0.0}, {"uid": "2723eee2e2d0f964", "family": "fsm", "template": 1, "depth": 3, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "8749497eaab6176d", "family": "fsm", "template": 1, "depth": 3, "pred": "2", "truth": "1", "correct": false, "edits_per_token": 0.0}, {"uid": "f006a9e07042e43f", "family": "fsm", "template": 2, "depth": 4, "pred": "6", "truth": "6", "correct": true, "edits_per_token": 0.0}, {"uid": "def842732bc0714f", "family": "fsm", "template": 2, "depth": 4, "pred": "-4", "truth": "-4", "correct": true, "edits_per_token": 0.0}, {"uid": "e4ca09128be213ea", "family": "fsm", "template": 0, "depth": 3, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 0.0}, {"uid": "a1c297725f5ae175", "family": "fsm", "template": 1, "depth": 4, "pred": "0", "truth": "2", "correct": false, "edits_per_token": 0.0}, {"uid": "7d9f104b41e0b7a5", "family": "fsm", "template": 0, "depth": 4, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "536ead7bf94947ee", "family": "fsm", "template": 1, "depth": 4, "pred": "18", "truth": "18", "correct": true, "edits_per_token": 0.0}, {"uid": "9808a14025bb62db", "family": "fsm", "template": 1, "depth": 4, "pred": "2", "truth": "2", "correct": true, "edits_per_token": 0.0}, {"uid": "666a3ad814af7485", "family": "fsm", "template": 0, "depth": 4, "pred": "9", "truth": "9", "correct": true, "edits_per_token": 0.0}, {"uid": "29e5f1d39e9dd718", "family": "excl", "template": 2, "depth": 1, "pred": "apple plum", "truth": "date", "correct": false, "edits_per_token": 0.0}, {"uid": "4425b91f251fc921", "family": "excl", "template": 2, "depth": 1, "pred": "plum pear", "truth": "lime", "correct": false, "edits_per_token": 0.0}, {"uid": "899865b46b45638d", "family": "sched", "template": 2, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "bb4df57d9a04ccab", "family": "fsm", "template": 1, "depth": 3, "pred": "-5", "truth": "-5", "correct": true, "edits_per_token": 0.0}, {"uid": "5aee313fef7904e6", "family": "fsm", "template": 2, "depth": 4, "pred": "0", "truth": "-1", "correct": false, "edits_per_token": 0.0}, {"uid": "b6a823db9fe091cf", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "eae15d4721c93c28", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "ebdd7e877a44588b", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "a467884db4d82ee1", "family": "fsm", "template": 1, "depth": 4, "pred": "10", "truth": "8", "correct": false, "edits_per_token": 0.0}, {"uid": "a0e13fa29eaee344", "family": "fsm", "template": 0, "depth": 4, "pred": "18", "truth": "16", "correct": false, "edits_per_token": 0.0}, {"uid": "55bddbb405630c49", "family": "excl", "template": 2, "depth": 1, "pred": "plum lime a", "truth": "kiwi plum apple", "correct": false, "edits_per_token": 0.0}, {"uid": "9e10c6f2c824ea03", "family": "fsm", "template": 1, "depth": 4, "pred": "5", "truth": "5", "correct": true, "edits_per_token": 0.0}, {"uid": "e29749c80774340d", "family": "fsm", "template": 1, "depth": 4, "pred": "-7", "truth": "-8", "correct": false, "edits_per_token": 0.0}, {"uid": "305ec8d13dc601f2", "family": "excl", "template": 2, "depth": 1, "pred": "apple fig b", "truth": "kiwi fig yam bean", "correct": false, "edits_per_token": 0.0}, {"uid": "c56841284720069d", "family": "fsm", "template": 0, "depth": 4, "pred": "-1", "truth": "-1", "correct": true, "edits_per_token": 0.0}, {"uid": "b757b7fb6b160032", "family": "fsm", "template": 0, "depth": 4, "pred": "-8", "truth": "-7", "correct": false, "edits_per_token": 0.0}, {"uid": "8a849a68ac0bd89d", "family": "sched", "template": 1, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "24c9898cc5168e60", "family": "sched", "template": 1, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "b45de26fb239a53c", "family": "fsm", "template": 0, "depth": 4, "pred": "4", "truth": "4", "correct": true, "edits_per_token": 0.0}, {"uid": "b92a8bb6a6b6c2fc", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "24f6f2d470ed5ab8", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "265c9cd1ac273c94", "family": "sched", "template": 2, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "17e81034d97a73e1", "family": "excl", "template": 1, "depth": 2, "pred": "pear date y", "truth": "yam", "correct": false, "edits_per_token": 0.0}, {"uid": "52b9e62b2a6ca3c2", "family": "excl", "template": 2, "depth": 3, "pred": "fig", "truth": "yam", "correct": false, "edits_per_token": 0.0}, {"uid": "3ea463a1eb42a642", "family": "excl", "template": 2, "depth": 1, "pred": "bean fig pl", "truth": "bean plum date nut", "correct": false, "edits_per_token": 0.0}, {"uid": "290428df35319b49", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "e8f3f5315f909938", "family": "excl", "template": 2, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "679e75695e94a8a8", "family": "sched", "template": 0, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "b9763ee8284b60af", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "50436cd63f7d344d", "family": "excl", "template": 1, "depth": 1, "pred": "bean apple", "truth": "bean date apple", "correct": false, "edits_per_token": 0.0}, {"uid": "f91565e19106bd0c", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "f377b423248cc145", "family": "sched", "template": 2, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "fd984d3e4a7605fd", "family": "excl", "template": 0, "depth": 2, "pred": "pear kiwi", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "78ec3d641e6859a8", "family": "excl", "template": 2, "depth": 2, "pred": "kiwi pear", "truth": "date apple", "correct": false, "edits_per_token": 0.0}, {"uid": "ec07ebafd22ecb2e", "family": "excl", "template": 1, "depth": 2, "pred": "plum", "truth": "pear", "correct": false, "edits_per_token": 0.0}, {"uid": "637c344de20f5625", "family": "excl", "template": 2, "depth": 2, "pred": "kiwi nut", "truth": "apple date fig bean", "correct": false, "edits_per_token": 0.0}, {"uid": "b1127806bf880411", "family": "sched", "template": 0, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "a65132179f29f33e", "family": "excl", "template": 0, "depth": 1, "pred": "apple lime", "truth": "apple lime", "correct": true, "edits_per_token": 0.0}, {"uid": "610ab75a850b6358", "family": "sched", "template": 0, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "09a0f4119071f489", "family": "excl", "template": 2, "depth": 1, "pred": "pear apple", "truth": "pear kiwi fig nut", "correct": false, "edits_per_token": 0.0}, {"uid": "9c0d7b46b835ff7f", "family": "excl", "template": 2, "depth": 1, "pred": "plum fig pe", "truth": "plum kiwi date nut", "correct": false, "edits_per_token": 0.0}, {"uid": "e9948bd8b29e69d3", "family": "excl", "template": 2, "depth": 2, "pred": "lime yam ki", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "f948792de5a76409", "family": "excl", "template": 0, "depth": 1, "pred": "kiwi fig li", "truth": "apple kiwi lime fig", "correct": false, "edits_per_token": 0.0}, {"uid": "43382d613173a703", "family": "sched", "template": 1, "depth": 1, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "4378e430fe1485f9", "family": "sched", "template": 1, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "7af9657dd3fba74a", "family": "excl", "template": 2, "depth": 2, "pred": "plum", "truth": "date plum", "correct": false, "edits_per_token": 0.0}, {"uid": "c9dda90c7d0ad8ed", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "2f7d7388cf61d3fe", "family": "sched", "template": 1, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "114ddae29f526626", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "d95b1cb8c2923e7e", "family": "excl", "template": 0, "depth": 1, "pred": "bean lime a", "truth": "bean lime apple", "correct": false, "edits_per_token": 0.0}, {"uid": "1dcd25a6eb16038d", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "59114c794ee3a087", "family": "excl", "template": 0, "depth": 2, "pred": "bean plum p", "truth": "bean", "correct": false, "edits_per_token": 0.0}, {"uid": "a9742f748ad1ad5c", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "ec7190a280bfbdb7", "family": "excl", "template": 2, "depth": 4, "pred": "apple", "truth": "lime", "correct": false, "edits_per_token": 0.0}, {"uid": "81a4bc002847b518", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "fce4affe8df7979a", "family": "excl", "template": 1, "depth": 1, "pred": "nut date li", "truth": "date apple lime", "correct": false, "edits_per_token": 0.0}, {"uid": "50c50f83a59dabd8", "family": "excl", "template": 2, "depth": 2, "pred": "yam plum pe", "truth": "fig bean kiwi plum apple", "correct": false, "edits_per_token": 0.0}, {"uid": "9a0b5958ebfcc667", "family": "sched", "template": 1, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "2b9fc22b15089625", "family": "excl", "template": 0, "depth": 1, "pred": "pear apple", "truth": "pear kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "2d7af060d17ced33", "family": "sched", "template": 1, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "22973924b5d55519", "family": "sched", "template": 1, "depth": 1, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "9eba62cb7a72f599", "family": "sched", "template": 2, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "60adda9ef6e12d3c", "family": "excl", "template": 0, "depth": 1, "pred": "kiwi pear a", "truth": "kiwi bean apple", "correct": false, "edits_per_token": 0.0}, {"uid": "222e860f500afa48", "family": "excl", "template": 1, "depth": 1, "pred": "pear fig nu", "truth": "pear nut fig lime", "correct": false, "edits_per_token": 0.0}, {"uid": "883e00248131789c", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "b7eae28e49657a0d", "family": "sched", "template": 2, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "36bc16e15622ea97", "family": "sched", "template": 2, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "9b1170c69bedb5d0", "family": "excl", "template": 1, "depth": 3, "pred": "nut date", "truth": "plum date", "correct": false, "edits_per_token": 0.0}, {"uid": "e7a942f9776e1481", "family": "excl", "template": 1, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "03e8f11a87973087", "family": "sched", "template": 0, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "0641e2dbfe875c84", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "b60c5ce59c8ad584", "family": "sched", "template": 2, "depth": 3, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "a07385bb7e4ea97b", "family": "sched", "template": 0, "depth": 2, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "a942d085ec964894", "family": "sched", "template": 2, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "800f61c9e4ad0bf4", "family": "excl", "template": 1, "depth": 2, "pred": "fig lime", "truth": "nut bean", "correct": false, "edits_per_token": 0.0}, {"uid": "19887d3ca18adb40", "family": "sched", "template": 0, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "01dd5269d9e0a510", "family": "sched", "template": 0, "depth": 3, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "d1578ac0adce4a3f", "family": "sched", "template": 2, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "4f7ac4cb4667edb6", "family": "excl", "template": 0, "depth": 2, "pred": "lime apple", "truth": "bean date", "correct": false, "edits_per_token": 0.0}, {"uid": "9ebb20923f7ff360", "family": "sched", "template": 0, "depth": 3, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "45e5e22be65e2392", "family": "sched", "template": 2, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "58590db8d34f4bb6", "family": "sched", "template": 2, "depth": 2, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "e0aa686b67447872", "family": "excl", "template": 1, "depth": 1, "pred": "none", "truth": "lime fig", "correct": false, "edits_per_token": 0.0}, {"uid": "b9758ae4de312e38", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "apple", "correct": false, "edits_per_token": 0.0}, {"uid": "49b538b6cebfad32", "family": "sched", "template": 0, "depth": 2, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "b41c090edc1c2570", "family": "sched", "template": 0, "depth": 2, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "57f25f911fd47979", "family": "excl", "template": 0, "depth": 2, "pred": "fig lime", "truth": "bean", "correct": false, "edits_per_token": 0.0}, {"uid": "52a3076c67262025", "family": "excl", "template": 0, "depth": 2, "pred": "bean fig", "truth": "bean fig date", "correct": false, "edits_per_token": 0.0}, {"uid": "5313d51deae8c2b8", "family": "excl", "template": 0, "depth": 3, "pred": "kiwi", "truth": "yam", "correct": false, "edits_per_token": 0.0}, {"uid": "fbb63a5c5980a546", "family": "excl", "template": 1, "depth": 1, "pred": "plum nut", "truth": "date apple", "correct": false, "edits_per_token": 0.0}, {"uid": "123ddaa3e5efb215", "family": "excl", "template": 1, "depth": 1, "pred": "fig bean pe", "truth": "fig pear nut kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "bf6d2e6f97942225", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "3a419d90cdd84493", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "yam plum nut", "correct": false, "edits_per_token": 0.0}, {"uid": "dc7a2b700ef3eec3", "family": "sched", "template": 0, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "a05184e028d82ee0", "family": "sched", "template": 0, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "b9e25b6eb179c90b", "family": "excl", "template": 0, "depth": 3, "pred": "kiwi bean p", "truth": "kiwi nut fig pear", "correct": false, "edits_per_token": 0.0}, {"uid": "914dc5465d528e9b", "family": "sched", "template": 0, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "64f71380d66efae2", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "33fe241315cf5b9f", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "1b7f8f05a6c5a7a3", "family": "excl", "template": 2, "depth": 2, "pred": "plum apple", "truth": "plum apple", "correct": true, "edits_per_token": 0.0}, {"uid": "5c13ea4ff7d521b5", "family": "sched", "template": 1, "depth": 2, "pred": "A", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "2a2cf42fd19abbe2", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "df171a88e16d0896", "family": "excl", "template": 0, "depth": 2, "pred": "yam", "truth": "apple yam", "correct": false, "edits_per_token": 0.0}, {"uid": "a6b723b79a990fea", "family": "excl", "template": 1, "depth": 1, "pred": "nut lime zi", "truth": "nut kiwi lime fig", "correct": false, "edits_per_token": 0.0}, {"uid": "95569dc6b1f0424e", "family": "excl", "template": 2, "depth": 4, "pred": "fig", "truth": "nut fig", "correct": false, "edits_per_token": 0.0}, {"uid": "d5cb804202c06b04", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "26a8bdf6da59e053", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "8461de28662a09d0", "family": "excl", "template": 0, "depth": 1, "pred": "yam bean", "truth": "lime kiwi apple", "correct": false, "edits_per_token": 0.0}, {"uid": "5266b24716f21c1b", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "1301b2b886fc0fed", "family": "sched", "template": 1, "depth": 3, "pred": "A", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "0ae3e633b9c73a6d", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "kiwi", "correct": false, "edits_per_token": 0.0}, {"uid": "a34d5263586bf792", "family": "sched", "template": 1, "depth": 2, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "72b16296ef29803c", "family": "sched", "template": 2, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "1650d27be4762458", "family": "sched", "template": 2, "depth": 4, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "d17c2abfac01fb85", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "f7bb7ae8302c82e3", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "ce88ea5df4ac7c7e", "family": "sched", "template": 1, "depth": 2, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "89232c53de86f0c7", "family": "sched", "template": 2, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "7867942486d7f380", "family": "sched", "template": 0, "depth": 4, "pred": "none", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "c9726e189052aad1", "family": "sched", "template": 2, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 0.0}, {"uid": "302f485c304f8e1c", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "58e2a723355b2957", "family": "sched", "template": 0, "depth": 3, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "376a16c13fc8af71", "family": "sched", "template": 0, "depth": 4, "pred": "C", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "7eddd2c6ff72770f", "family": "sched", "template": 0, "depth": 4, "pred": "none", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "ebcfc88ef93a66df", "family": "sched", "template": 2, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "89a256c76069f616", "family": "excl", "template": 1, "depth": 3, "pred": "plum", "truth": "plum lime", "correct": false, "edits_per_token": 0.0}, {"uid": "34292f4e57b84e60", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "b4ef07715474f790", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "084073be6e34810e", "family": "excl", "template": 2, "depth": 2, "pred": "lime", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "fec42ca9bcf0a20b", "family": "excl", "template": 1, "depth": 2, "pred": "lime", "truth": "date plum bean", "correct": false, "edits_per_token": 0.0}, {"uid": "9600b1b302d34721", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "e68c491cb5fe0db5", "family": "excl", "template": 0, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "8fbf1093b19d178d", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "152fa59b6f060034", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "6886f89d5fecebc2", "family": "sched", "template": 0, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "25810ef41631a5a0", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "date", "correct": false, "edits_per_token": 0.0}, {"uid": "c2c1af96873ca940", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "493eb05c569e6806", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "yam fig date", "correct": false, "edits_per_token": 0.0}, {"uid": "3ca58bdf2e93bec8", "family": "excl", "template": 1, "depth": 2, "pred": "apple yam b", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "68cafc987827b672", "family": "excl", "template": 0, "depth": 1, "pred": "apple yam f", "truth": "apple lime nut kiwi bean", "correct": false, "edits_per_token": 0.0}, {"uid": "e5f870c9ac9c2bdc", "family": "excl", "template": 1, "depth": 2, "pred": "apple fig", "truth": "apple pear plum", "correct": false, "edits_per_token": 0.0}, {"uid": "4aba72aed6f820bb", "family": "excl", "template": 2, "depth": 3, "pred": "none", "truth": "bean", "correct": false, "edits_per_token": 0.0}, {"uid": "7e3d734f2577f012", "family": "excl", "template": 2, "depth": 3, "pred": "pear apple", "truth": "plum bean", "correct": false, "edits_per_token": 0.0}, {"uid": "da06a708de7351a1", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "e13f444f5e921059", "family": "sched", "template": 1, "depth": 4, "pred": "A", "truth": "D", "correct": false, "edits_per_token": 0.0}, {"uid": "806dfca57003d938", "family": "sched", "template": 1, "depth": 4, "pred": "B", "truth": "C", "correct": false, "edits_per_token": 0.0}, {"uid": "5a3700bf53c0daf3", "family": "sched", "template": 1, "depth": 3, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "fa8b059c364916e0", "family": "sched", "template": 1, "depth": 4, "pred": "D", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "9fd10a99f31cee6e", "family": "sched", "template": 1, "depth": 3, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "b3fe0fa2ac161f59", "family": "excl", "template": 1, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "2489ee7572296937", "family": "sched", "template": 2, "depth": 4, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "80d519c064361dae", "family": "sched", "template": 2, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "5a66b965f98dc2cf", "family": "excl", "template": 0, "depth": 3, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "4cd96fbf4f41a908", "family": "sched", "template": 0, "depth": 4, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "7a4191b42dce2b06", "family": "sched", "template": 1, "depth": 3, "pred": "none", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "b87404e94d9a09f6", "family": "excl", "template": 0, "depth": 2, "pred": "plum apple", "truth": "fig kiwi pear", "correct": false, "edits_per_token": 0.0}, {"uid": "2a9ecd3661b30257", "family": "sched", "template": 0, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 0.0}, {"uid": "5249781ebc348d37", "family": "sched", "template": 2, "depth": 4, "pred": "C", "truth": "C", "correct": true, "edits_per_token": 0.0}, {"uid": "2e4acd7d13860ac2", "family": "sched", "template": 2, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "2adc39419c8b49a4", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "fig lime", "correct": false, "edits_per_token": 0.0}, {"uid": "dca6675d28527584", "family": "excl", "template": 0, "depth": 3, "pred": "fig yam", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "c666c9854336f784", "family": "sched", "template": 0, "depth": 4, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "8f480bee1ead1646", "family": "excl", "template": 0, "depth": 3, "pred": "nut yam", "truth": "apple", "correct": false, "edits_per_token": 0.0}, {"uid": "cf67501c8d912952", "family": "excl", "template": 2, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "d61260228e19e8ba", "family": "sched", "template": 2, "depth": 4, "pred": "A", "truth": "A", "correct": true, "edits_per_token": 0.0}, {"uid": "71fb11da7ef26fa8", "family": "sched", "template": 0, "depth": 4, "pred": "B", "truth": "B", "correct": true, "edits_per_token": 0.0}, {"uid": "f12148b1efc8abed", "family": "excl", "template": 2, "depth": 4, "pred": "date", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "9e76644064b0dc70", "family": "excl", "template": 0, "depth": 3, "pred": "date yam", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "12b5acb0d562716c", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "fd29037f3e23207c", "family": "sched", "template": 1, "depth": 4, "pred": "D", "truth": "D", "correct": true, "edits_per_token": 0.0}, {"uid": "dbd4f3b44d2455e6", "family": "sched", "template": 1, "depth": 4, "pred": "C", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "2bbd29545a49585f", "family": "sched", "template": 1, "depth": 4, "pred": "B", "truth": "A", "correct": false, "edits_per_token": 0.0}, {"uid": "e7a65045762e87f2", "family": "sched", "template": 1, "depth": 4, "pred": "A", "truth": "B", "correct": false, "edits_per_token": 0.0}, {"uid": "9476a68c1c10bc9b", "family": "excl", "template": 1, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "dfb1aad2e59286cb", "family": "excl", "template": 0, "depth": 4, "pred": "pear", "truth": "none", "correct": false, "edits_per_token": 0.0}, {"uid": "fe2915e5def8bf83", "family": "excl", "template": 1, "depth": 4, "pred": "fig apple n", "truth": "yam", "correct": false, "edits_per_token": 0.0}, {"uid": "548b800800b9067a", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "19a75d08ddd9d502", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "nut", "correct": false, "edits_per_token": 0.0}, {"uid": "6c5b7174f7312602", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}, {"uid": "11c0e819fd3fe52e", "family": "excl", "template": 0, "depth": 4, "pred": "none", "truth": "none", "correct": true, "edits_per_token": 0.0}] | |
| \ No newline at end of file | ||
modified
experiments/runs/p4_editor_v1_scaled_seed01_20261029234334/metrics.jsonl
+68 −0
@@ -1 +1,69 @@ | ||
| 1 | 1 | {"step": 0, "loss": 0.2574068307876587, "lr": 1e-05, "grad_norm": 0.02281644381582737, "tokens": 146, "tok_per_s": 201.8701259313982, "t": 1.5} |
| 2 | +{"step": 50, "loss": 0.29632481932640076, "lr": 0.00051, "grad_norm": 0.019611647352576256, "tokens": 8382, "tok_per_s": 238.8615009625485, "t": 35.87} | |
| 3 | +{"step": 100, "loss": 0.2928289771080017, "lr": 0.001, "grad_norm": 0.03218043968081474, "tokens": 16548, "tok_per_s": 242.5809658060073, "t": 68.99} | |
| 4 | +{"step": 150, "loss": 0.27522939443588257, "lr": 0.0009993400362330058, "grad_norm": 0.035960081964731216, "tokens": 24812, "tok_per_s": 245.6446848604652, "t": 101.79} | |
| 5 | +{"step": 200, "loss": 0.2911725342273712, "lr": 0.0009973620807194618, "grad_norm": 0.03377067297697067, "tokens": 33400, "tok_per_s": 250.01054118680793, "t": 134.37} | |
| 6 | +{"step": 250, "loss": 0.26847678422927856, "lr": 0.000994071935143687, "grad_norm": 0.024587862193584442, "tokens": 41583, "tok_per_s": 248.14157141989233, "t": 168.36} | |
| 7 | +{"step": 300, "loss": 0.2520502805709839, "lr": 0.000989479250069539, "grad_norm": 0.02395401895046234, "tokens": 49989, "tok_per_s": 249.9883153529742, "t": 200.74} | |
| 8 | +{"step": 350, "loss": 0.3760474622249603, "lr": 0.0009835974966336503, "grad_norm": 0.0955885574221611, "tokens": 58214, "tok_per_s": 248.86359307777582, "t": 234.7} | |
| 9 | +{"step": 400, "loss": 0.2934175729751587, "lr": 0.0009764439270322612, "grad_norm": 0.03379363194108009, "tokens": 66391, "tok_per_s": 249.30502340683375, "t": 267.08} | |
| 10 | +{"step": 450, "loss": 0.20187416672706604, "lr": 0.0009680395239175562, "grad_norm": 0.05836236849427223, "tokens": 74699, "tok_per_s": 250.12985566998898, "t": 299.42} | |
| 11 | +{"step": 499, "val_loss_b0": 0.41442169654245364, "val_loss_b1": 0.41388571598908874, "val_loss_b2": 0.41333505196876014, "val_loss_b4": 0.4130766558257467, "val_loss_b8": 0.4149458404813089, "t": 327.54} | |
| 12 | +{"step": 500, "loss": 0.3621078431606293, "lr": 0.0009584089388519307, "grad_norm": 0.03487864136695862, "tokens": 83038, "tok_per_s": 253.78134438509935, "t": 327.98} | |
| 13 | +{"step": 550, "loss": 0.2949325144290924, "lr": 0.0009475804200007103, "grad_norm": 0.039413344115018845, "tokens": 91441, "tok_per_s": 262.07085905576315, "t": 349.69} | |
| 14 | +{"step": 600, "loss": 0.36860066652297974, "lr": 0.0009355857292754152, "grad_norm": 0.06942126899957657, "tokens": 99553, "tok_per_s": 267.5088516205941, "t": 372.93} | |
| 15 | +{"step": 650, "loss": 0.3048055171966553, "lr": 0.0009224600491706008, "grad_norm": 0.033525556325912476, "tokens": 107666, "tok_per_s": 264.82925568339476, "t": 407.33} | |
| 16 | +{"step": 700, "loss": 0.3155032694339752, "lr": 0.0009082418795675396, "grad_norm": 0.035131558775901794, "tokens": 115959, "tok_per_s": 263.3350495385592, "t": 441.13} | |
| 17 | +{"step": 750, "loss": 0.25763794779777527, "lr": 0.0008929729248074364, "grad_norm": 0.01945130154490471, "tokens": 124302, "tok_per_s": 259.77709879773954, "t": 479.27} | |
| 18 | +{"step": 800, "loss": 0.34229332208633423, "lr": 0.0008766979713654089, "grad_norm": 0.042905353009700775, "tokens": 132734, "tok_per_s": 252.09902527133045, "t": 527.3} | |
| 19 | +{"step": 850, "loss": 0.3549925982952118, "lr": 0.0008594647564840406, "grad_norm": 0.06940905004739761, "tokens": 141025, "tok_per_s": 243.66067602748035, "t": 579.56} | |
| 20 | +{"step": 900, "loss": 0.34444135427474976, "lr": 0.0008413238281518224, "grad_norm": 0.05708182603120804, "tokens": 149035, "tok_per_s": 237.70578563676642, "t": 627.75} | |
| 21 | +{"step": 950, "loss": 0.3196276128292084, "lr": 0.0008223283968371944, "grad_norm": 0.04837086796760559, "tokens": 157135, "tok_per_s": 234.27442894491801, "t": 671.52} | |
| 22 | +{"step": 999, "val_loss_b0": 0.41442169654245364, "val_loss_b1": 0.41292231079373637, "val_loss_b2": 0.4124131649068348, "val_loss_b4": 0.41228707867603953, "val_loss_b8": 0.4145959596222967, "t": 750.59} | |
| 23 | +{"step": 1000, "loss": 0.3006693422794342, "lr": 0.000802534179413072, "grad_norm": 0.05650242790579796, "tokens": 165602, "tok_per_s": 220.37104848179126, "t": 752.25} | |
| 24 | +{"step": 1050, "loss": 0.2752276062965393, "lr": 0.0007819992357296598, "grad_norm": 0.10306810587644577, "tokens": 173674, "tok_per_s": 211.8841533546052, "t": 820.45} | |
| 25 | +{"step": 1100, "loss": 0.24044103920459747, "lr": 0.0007607837983149056, "grad_norm": 0.07654126733541489, "tokens": 181834, "tok_per_s": 204.64523292063996, "t": 889.31} | |
| 26 | +{"step": 1150, "loss": 0.3453167676925659, "lr": 0.0007389500957021191, "grad_norm": 0.047272052615880966, "tokens": 190158, "tok_per_s": 200.69316957919511, "t": 948.28} | |
| 27 | +{"step": 1200, "loss": 0.3205467164516449, "lr": 0.0007165621699029615, "grad_norm": 0.07610572874546051, "tokens": 198573, "tok_per_s": 201.2703617479415, "t": 987.38} | |
| 28 | +{"step": 1250, "loss": 0.3766207695007324, "lr": 0.000693685688561191, "grad_norm": 0.11205436289310455, "tokens": 206880, "tok_per_s": 203.23875172387238, "t": 1018.69} | |
| 29 | +{"step": 1300, "loss": 0.25073426961898804, "lr": 0.0006703877523381495, "grad_norm": 0.02866116352379322, "tokens": 215095, "tok_per_s": 204.32399051367523, "t": 1053.49} | |
| 30 | +{"step": 1350, "loss": 0.2750009000301361, "lr": 0.0006467366980949609, "grad_norm": 0.08874290436506271, "tokens": 223349, "tok_per_s": 205.43970213727562, "t": 1087.95} | |
| 31 | +{"step": 1400, "loss": 0.26846012473106384, "lr": 0.0006228018984487442, "grad_norm": 0.08993160724639893, "tokens": 231716, "tok_per_s": 206.17248491969283, "t": 1124.67} | |
| 32 | +{"step": 1450, "loss": 0.37126070261001587, "lr": 0.0005986535582907738, "grad_norm": 0.09381195902824402, "tokens": 239960, "tok_per_s": 207.25213909026385, "t": 1158.59} | |
| 33 | +{"step": 1499, "val_loss_b0": 0.41442169654245364, "val_loss_b1": 0.4119904505373996, "val_loss_b2": 0.41204987450836317, "val_loss_b4": 0.4140512712997278, "val_loss_b8": 0.4230532341513556, "t": 1196.93} | |
| 34 | +{"step": 1499, "checkpoint_sha256": "55a327447292a3b40730ece7a3c0c43fd5ac993486438745112a7545e8732f10", "t": 1197.03} | |
| 35 | +{"step": 1500, "loss": 0.26724329590797424, "lr": 0.000574362508863438, "grad_norm": 0.1111975610256195, "tokens": 248357, "tok_per_s": 207.49581380741336, "t": 1197.7} | |
| 36 | +{"step": 1550, "loss": 0.3123103976249695, "lr": 0.00055, "grad_norm": 0.09849677979946136, "tokens": 256793, "tok_per_s": 208.68812756438493, "t": 1231.29} | |
| 37 | +{"step": 1600, "loss": 0.28196197748184204, "lr": 0.0005256374911365621, "grad_norm": 0.08638253062963486, "tokens": 265023, "tok_per_s": 210.5317661187001, "t": 1259.6} | |
| 38 | +{"step": 1650, "loss": 0.347612589597702, "lr": 0.0005013464417092262, "grad_norm": 0.06267423927783966, "tokens": 273283, "tok_per_s": 212.0269164756034, "t": 1289.68} | |
| 39 | +{"step": 1700, "loss": 0.3768414855003357, "lr": 0.00047719810155125587, "grad_norm": 0.09401395171880722, "tokens": 281671, "tok_per_s": 213.709111042203, "t": 1318.79} | |
| 40 | +{"step": 1750, "loss": 0.26761454343795776, "lr": 0.0004532633019050392, "grad_norm": 0.05395130068063736, "tokens": 289799, "tok_per_s": 215.14359839898987, "t": 1347.78} | |
| 41 | +{"step": 1800, "loss": 0.24711786210536957, "lr": 0.0004296122476618507, "grad_norm": 0.049351055175065994, "tokens": 298140, "tok_per_s": 215.7011592696487, "t": 1382.97} | |
| 42 | +{"step": 1850, "loss": 0.2801760137081146, "lr": 0.00040631431143880916, "grad_norm": 0.09576767683029175, "tokens": 306518, "tok_per_s": 216.1789281953634, "t": 1418.67} | |
| 43 | +{"step": 1900, "loss": 0.28702354431152344, "lr": 0.00038343783009703845, "grad_norm": 0.06675335019826889, "tokens": 314885, "tok_per_s": 216.87856589520936, "t": 1452.67} | |
| 44 | +{"step": 1950, "loss": 0.21578460931777954, "lr": 0.000361049904297881, "grad_norm": 0.0865975022315979, "tokens": 322988, "tok_per_s": 217.08951165753467, "t": 1488.59} | |
| 45 | +{"step": 1999, "val_loss_b0": 0.41442169654245364, "val_loss_b1": 0.41191016618068294, "val_loss_b2": 0.4128569975650328, "val_loss_b4": 0.4138939249568673, "val_loss_b8": 0.4240893042282399, "t": 1524.81} | |
| 46 | +{"step": 2000, "loss": 0.24176457524299622, "lr": 0.0003392162016850945, "grad_norm": 0.09138377755880356, "tokens": 331190, "tok_per_s": 217.25065171432246, "t": 1525.24} | |
| 47 | +{"step": 2050, "loss": 0.27041375637054443, "lr": 0.0003180007642703402, "grad_norm": 0.12077879160642624, "tokens": 339365, "tok_per_s": 218.53047166446817, "t": 1553.72} | |
| 48 | +{"step": 2100, "loss": 0.2858930826187134, "lr": 0.000297465820586928, "grad_norm": 0.15457485616207123, "tokens": 347821, "tok_per_s": 219.6558923585326, "t": 1584.26} | |
| 49 | +{"step": 2150, "loss": 0.2684478759765625, "lr": 0.0002776716031628058, "grad_norm": 0.05642038583755493, "tokens": 355802, "tok_per_s": 220.89252553601239, "t": 1611.52} | |
| 50 | +{"step": 2200, "loss": 0.2356978803873062, "lr": 0.00025867617184817756, "grad_norm": 0.11221145838499069, "tokens": 363932, "tok_per_s": 221.90883218965223, "t": 1640.78} | |
| 51 | +{"step": 2250, "loss": 0.23309443891048431, "lr": 0.00024053524351595949, "grad_norm": 0.06473631411790848, "tokens": 372235, "tok_per_s": 222.31644292615823, "t": 1675.12} | |
| 52 | +{"step": 2300, "loss": 0.27500903606414795, "lr": 0.0002233020286345913, "grad_norm": 0.0791851282119751, "tokens": 380617, "tok_per_s": 222.7510982833804, "t": 1709.49} | |
| 53 | +{"step": 2350, "loss": 0.20908822119235992, "lr": 0.00020702707519256364, "grad_norm": 0.10462427139282227, "tokens": 388820, "tok_per_s": 223.1480124449076, "t": 1743.21} | |
| 54 | +{"step": 2400, "loss": 0.18027986586093903, "lr": 0.00019175812043246033, "grad_norm": 0.08221527934074402, "tokens": 397013, "tok_per_s": 223.75909173491473, "t": 1775.07} | |
| 55 | +{"step": 2450, "loss": 0.3071126937866211, "lr": 0.00017753995082939931, "grad_norm": 0.09175784885883331, "tokens": 405444, "tok_per_s": 224.78321984200625, "t": 1804.49} | |
| 56 | +{"step": 2499, "val_loss_b0": 0.41442169654245364, "val_loss_b1": 0.40995508423531746, "val_loss_b2": 0.409993490528033, "val_loss_b4": 0.41109113990817703, "val_loss_b8": 0.4188066938833886, "t": 1841.88} | |
| 57 | +{"step": 2500, "loss": 0.3278486132621765, "lr": 0.00016441427072458494, "grad_norm": 0.12872454524040222, "tokens": 413920, "tok_per_s": 224.74138191863418, "t": 1842.54} | |
| 58 | +{"step": 2550, "loss": 0.28487342596054077, "lr": 0.0001524195799992897, "grad_norm": 0.09486062824726105, "tokens": 422230, "tok_per_s": 224.9697021818197, "t": 1877.61} | |
| 59 | +{"step": 2600, "loss": 0.2684286832809448, "lr": 0.0001415910611480694, "grad_norm": 0.12426797300577164, "tokens": 430404, "tok_per_s": 225.21814513164202, "t": 1911.83} | |
| 60 | +{"step": 2650, "loss": 0.254168301820755, "lr": 0.00013196047608244387, "grad_norm": 0.16166745126247406, "tokens": 438498, "tok_per_s": 225.3601838181466, "t": 1946.54} | |
| 61 | +{"step": 2700, "loss": 0.1905134916305542, "lr": 0.00012355607296773896, "grad_norm": 0.07620202749967575, "tokens": 446658, "tok_per_s": 225.86205716807774, "t": 1978.35} | |
| 62 | +{"step": 2750, "loss": 0.26853248476982117, "lr": 0.00011640250336634969, "grad_norm": 0.13326406478881836, "tokens": 455155, "tok_per_s": 226.83726534529018, "t": 2007.3} | |
| 63 | +{"step": 2800, "loss": 0.2448510378599167, "lr": 0.00011052074993046101, "grad_norm": 0.10193030536174774, "tokens": 463083, "tok_per_s": 227.6757016091016, "t": 2034.74} | |
| 64 | +{"step": 2850, "loss": 0.2740918695926666, "lr": 0.00010592806485631325, "grad_norm": 0.17447443306446075, "tokens": 471665, "tok_per_s": 229.09464735586297, "t": 2059.6} | |
| 65 | +{"step": 2900, "loss": 0.2866029441356659, "lr": 0.00010263791928053818, "grad_norm": 0.12663234770298004, "tokens": 480193, "tok_per_s": 230.46142883895683, "t": 2084.39} | |
| 66 | +{"step": 2950, "loss": 0.20462633669376373, "lr": 0.0001006599637669943, "grad_norm": 0.06513027101755142, "tokens": 488284, "tok_per_s": 232.0436854411812, "t": 2105.05} | |
| 67 | +{"step": 2999, "loss": 0.31182020902633667, "lr": 0.0001000002640500325, "grad_norm": 0.07867877185344696, "tokens": 496298, "tok_per_s": 234.06802248820583, "t": 2121.09} | |
| 68 | +{"step": 2999, "val_loss_b0": 0.41442169654245364, "val_loss_b1": 0.40932418262090586, "val_loss_b2": 0.4092105990178872, "val_loss_b4": 0.410994329920141, "val_loss_b8": 0.4191988424950156, "t": 2123.99} | |
| 69 | +{"step": 2999, "checkpoint_sha256": "a1f686a87bcbb5c3da1dbc90e3048b457a7dd541388accedd820e1be47fb21e1", "t": 2124.04} | |
added
experiments/runs/p4_overfit64_b8_seed01_20261029235111/checkpoint/checkpoint.json
+37 −0
@@ -0,0 +1,37 @@ | ||
| 1 | +{ | |
| 2 | + "config": { | |
| 3 | + "name": "tiny_p4", | |
| 4 | + "vocab_size": 128, | |
| 5 | + "d_model": 192, | |
| 6 | + "n_layers": 4, | |
| 7 | + "n_heads": 6, | |
| 8 | + "n_kv_heads": 2, | |
| 9 | + "ffn_width": 512, | |
| 10 | + "max_seq_len": 320, | |
| 11 | + "rope_theta": 10000.0, | |
| 12 | + "norm_eps": 1e-05, | |
| 13 | + "tie_embeddings": true, | |
| 14 | + "init_std": 0.02, | |
| 15 | + "use_workspace": true, | |
| 16 | + "w": 64, | |
| 17 | + "ffn_w": 192, | |
| 18 | + "n_slots": 16, | |
| 19 | + "n_editor_blocks": 2, | |
| 20 | + "n_editor_heads": 2, | |
| 21 | + "r_max": 8, | |
| 22 | + "eta_max": 0.25, | |
| 23 | + "clip_c": 1.0, | |
| 24 | + "clip_eps": 1e-06, | |
| 25 | + "readout_zero_init": false, | |
| 26 | + "readout_init_std": 0.5, | |
| 27 | + "readout_scale": 5.3098063468933105, | |
| 28 | + "controller_hidden": 0, | |
| 29 | + "lambda_cost": 0.0, | |
| 30 | + "kappa_risk": 0.0, | |
| 31 | + "dense_writes": false | |
| 32 | + }, | |
| 33 | + "saved_at": "2026-10-29T23:59:50-0400", | |
| 34 | + "step": 600, | |
| 35 | + "run_id": "p4_overfit64_b8_seed01_20261029235111", | |
| 36 | + "weights_sha256": "bb9dc4e9f2ffa760b97cb1d16b164ff6dc35c9f7c3d1c4d1ee8e375ec7a99914" | |
| 37 | +} | |
| \ No newline at end of file | ||
added
experiments/runs/p4_overfit64_b8_seed01_20261029235111/config.yaml
+98 −0
@@ -0,0 +1,98 @@ | ||
| 1 | +experiment: | |
| 2 | + name: p4_overfit64_b8 | |
| 3 | + seed: 1 | |
| 4 | + mode: prisme | |
| 5 | + model: | |
| 6 | + name: tiny_p4 | |
| 7 | + vocab_size: 128 | |
| 8 | + d_model: 192 | |
| 9 | + n_layers: 4 | |
| 10 | + n_heads: 6 | |
| 11 | + n_kv_heads: 2 | |
| 12 | + ffn_width: 512 | |
| 13 | + max_seq_len: 320 | |
| 14 | + w: 64 | |
| 15 | + ffn_w: 192 | |
| 16 | + n_slots: 16 | |
| 17 | + n_editor_heads: 2 | |
| 18 | + r_max: 8 | |
| 19 | + init: | |
| 20 | + backbone_checkpoint: experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint | |
| 21 | + freeze_backbone: true | |
| 22 | + auto_readout_scale: true | |
| 23 | + data: | |
| 24 | + kind: synthetic | |
| 25 | + tokenizer: char | |
| 26 | + splits_dir: data/splits/synthetic_v1 | |
| 27 | + loss_span: answer | |
| 28 | + eval_splits: | |
| 29 | + - dev_iid | |
| 30 | + max_train_items: 64 | |
| 31 | + budgets: | |
| 32 | + - 8 | |
| 33 | + budget_probs: | |
| 34 | + - 1.0 | |
| 35 | + budget_per: position | |
| 36 | + train: | |
| 37 | + batch_size: 32 | |
| 38 | + seq_len: 320 | |
| 39 | + steps: 600 | |
| 40 | + lr: 0.001 | |
| 41 | + warmup: 100 | |
| 42 | + weight_decay: 0.1 | |
| 43 | + grad_clip: 1.0 | |
| 44 | + log_every: 50 | |
| 45 | + eval_every: 100 | |
| 46 | + eval_batches: 4 | |
| 47 | + ckpt_every: 1500 | |
| 48 | + em_budgets: | |
| 49 | + - 0 | |
| 50 | + - 2 | |
| 51 | + - 8 | |
| 52 | + em_max_items: 400 | |
| 53 | + model_overrides: | |
| 54 | + readout_zero_init: false | |
| 55 | + readout_init_std: 0.5 | |
| 56 | + final_exact_match: false | |
| 57 | +model: | |
| 58 | + name: tiny_p4 | |
| 59 | + vocab_size: 128 | |
| 60 | + d_model: 192 | |
| 61 | + n_layers: 4 | |
| 62 | + n_heads: 6 | |
| 63 | + n_kv_heads: 2 | |
| 64 | + ffn_width: 512 | |
| 65 | + max_seq_len: 320 | |
| 66 | + rope_theta: 10000.0 | |
| 67 | + norm_eps: 1.0e-05 | |
| 68 | + tie_embeddings: true | |
| 69 | + init_std: 0.02 | |
| 70 | + use_workspace: true | |
| 71 | + w: 64 | |
| 72 | + ffn_w: 192 | |
| 73 | + n_slots: 16 | |
| 74 | + n_editor_blocks: 2 | |
| 75 | + n_editor_heads: 2 | |
| 76 | + r_max: 8 | |
| 77 | + eta_max: 0.25 | |
| 78 | + clip_c: 1.0 | |
| 79 | + clip_eps: 1.0e-06 | |
| 80 | + readout_zero_init: false | |
| 81 | + readout_init_std: 0.5 | |
| 82 | + readout_scale: 1.0 | |
| 83 | + controller_hidden: 0 | |
| 84 | + lambda_cost: 0.0 | |
| 85 | + kappa_risk: 0.0 | |
| 86 | + dense_writes: false | |
| 87 | +resolved_train: | |
| 88 | + batch_size: 32 | |
| 89 | + seq_len: 320 | |
| 90 | + steps: 600 | |
| 91 | + lr: 0.001 | |
| 92 | + warmup: 100 | |
| 93 | + weight_decay: 0.1 | |
| 94 | + grad_clip: 1.0 | |
| 95 | + log_every: 50 | |
| 96 | + eval_every: 100 | |
| 97 | + eval_batches: 4 | |
| 98 | + ckpt_every: 1500 | |
added
experiments/runs/p4_overfit64_b8_seed01_20261029235111/manifest.json
+28 −0
@@ -0,0 +1,28 @@ | ||
| 1 | +{ | |
| 2 | + "run_id": "p4_overfit64_b8_seed01_20261029235111", | |
| 3 | + "name": "p4_overfit64_b8", | |
| 4 | + "seed": 1, | |
| 5 | + "code_commit": "a9d94e8a2b85c98bf5259c5029878fd47479c325", | |
| 6 | + "config_sha256": "01a6831eecd093a2b930d173815671fd24fdff125f33bf7df874e5fc0ad4b788", | |
| 7 | + "started_at": "2026-10-29T23:51:11-0400", | |
| 8 | + "host": "MacBookPro-de-simon-pierre-2.local", | |
| 9 | + "python": "3.12.13", | |
| 10 | + "torch": "2.15.0", | |
| 11 | + "command": "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_editor.py --config configs/experiments/p4_overfit64_b8.yaml --device mps", | |
| 12 | + "state": "completed", | |
| 13 | + "wall_seconds": 519.6, | |
| 14 | + "device": "mps", | |
| 15 | + "backend": "mps", | |
| 16 | + "precision": "fp32", | |
| 17 | + "n_params": 1765445, | |
| 18 | + "n_trainable": 120577, | |
| 19 | + "checkpoint_parent": "experiments/runs/p4_tiny_backbone_seed01_20261029232626/checkpoint", | |
| 20 | + "mode": "prisme", | |
| 21 | + "freeze_backbone": true, | |
| 22 | + "readout_scale": 5.3098063468933105, | |
| 23 | + "finished_at": "2026-10-29T23:59:50-0400", | |
| 24 | + "train_tokens": 95773, | |
| 25 | + "peak_memory_bytes": 6731120640, | |
| 26 | + "final_exact_match": {}, | |
| 27 | + "metrics_file": "experiments/runs/p4_overfit64_b8_seed01_20261029235111/metrics.jsonl" | |
| 28 | +} | |
| \ No newline at end of file | ||
added
experiments/runs/p4_overfit64_b8_seed01_20261029235111/metrics.jsonl
+20 −0
@@ -0,0 +1,20 @@ | ||
| 1 | +{"step": 0, "loss": 0.347229927778244, "lr": 1e-05, "grad_norm": 1.322141408920288, "tokens": 148, "tok_per_s": 37.75254109902415, "t": 5.5} | |
| 2 | +{"step": 50, "loss": 0.10128622502088547, "lr": 0.00051, "grad_norm": 0.4043615758419037, "tokens": 7949, "tok_per_s": 192.99241679614582, "t": 42.77} | |
| 3 | +{"step": 99, "val_loss_b8": 0.7306965569442118, "t": 90.8} | |
| 4 | +{"step": 100, "loss": 0.016504138708114624, "lr": 0.001, "grad_norm": 0.1634058803319931, "tokens": 15744, "tok_per_s": 174.6667163690665, "t": 91.71} | |
| 5 | +{"step": 150, "loss": 0.005306841805577278, "lr": 0.0009779754323328191, "grad_norm": 0.016647804528474808, "tokens": 23618, "tok_per_s": 172.09258657480268, "t": 138.82} | |
| 6 | +{"step": 199, "val_loss_b8": 1.1004879354152708, "t": 173.76} | |
| 7 | +{"step": 200, "loss": 0.00241711363196373, "lr": 0.0009140576474687263, "grad_norm": 0.008768348954617977, "tokens": 31693, "tok_per_s": 183.42497445922888, "t": 174.36} | |
| 8 | +{"step": 250, "loss": 0.0009894538670778275, "lr": 0.000814503363531613, "grad_norm": 0.0031006645876914263, "tokens": 39738, "tok_per_s": 193.07667895946577, "t": 207.39} | |
| 9 | +{"step": 299, "val_loss_b8": 1.3594950490487114, "t": 254.77} | |
| 10 | +{"step": 300, "loss": 0.0008304077200591564, "lr": 0.0006890576474687264, "grad_norm": 0.001891698339022696, "tokens": 47878, "tok_per_s": 188.51353850802386, "t": 255.56} | |
| 11 | +{"step": 350, "loss": 0.00040095686563290656, "lr": 0.00055, "grad_norm": 0.0012410919880494475, "tokens": 55733, "tok_per_s": 185.6831282288836, "t": 301.73} | |
| 12 | +{"step": 399, "val_loss_b8": 1.499051700142222, "t": 347.61} | |
| 13 | +{"step": 400, "loss": 0.0004196925729047507, "lr": 0.0004109423525312738, "grad_norm": 0.0009135301224887371, "tokens": 63742, "tok_per_s": 183.67223822194268, "t": 348.62} | |
| 14 | +{"step": 450, "loss": 0.0003603784425649792, "lr": 0.0002854966364683872, "grad_norm": 0.0007731085061095655, "tokens": 71807, "tok_per_s": 183.6377874356163, "t": 392.61} | |
| 15 | +{"step": 499, "val_loss_b8": 1.6038084745772216, "t": 438.01} | |
| 16 | +{"step": 500, "loss": 0.0006298388470895588, "lr": 0.00018594235253127373, "grad_norm": 0.0006691015441901982, "tokens": 79967, "tok_per_s": 182.82307496171111, "t": 438.98} | |
| 17 | +{"step": 550, "loss": 0.000337488716468215, "lr": 0.00012202456766718093, "grad_norm": 0.0003788845206145197, "tokens": 87959, "tok_per_s": 183.06438353653638, "t": 482.06} | |
| 18 | +{"step": 599, "loss": 0.0002673463022802025, "lr": 0.00010000888261473832, "grad_norm": 0.00025494094006717205, "tokens": 95773, "tok_per_s": 185.19145987663353, "t": 518.73} | |
| 19 | +{"step": 599, "val_loss_b8": 1.6449561009545786, "t": 519.47} | |
| 20 | +{"step": 599, "checkpoint_sha256": "bb9dc4e9f2ffa760b97cb1d16b164ff6dc35c9f7c3d1c4d1ee8e375ec7a99914", "t": 519.6} | |
added
experiments/runs/p4_tiny_backbone_v2_seed01_20261029233624/checkpoint/checkpoint.json
+35 −0
@@ -0,0 +1,35 @@ | ||
| 1 | +{ | |
| 2 | + "config": { | |
| 3 | + "name": "tiny_p4v2", | |
| 4 | + "vocab_size": 128, | |
| 5 | + "d_model": 256, | |
| 6 | + "n_layers": 6, | |
| 7 | + "n_heads": 8, | |
| 8 | + "n_kv_heads": 4, | |
| 9 | + "ffn_width": 768, | |
| 10 | + "max_seq_len": 320, | |
| 11 | + "rope_theta": 10000.0, | |
| 12 | + "norm_eps": 1e-05, | |
| 13 | + "tie_embeddings": true, | |
| 14 | + "init_std": 0.02, | |
| 15 | + "use_workspace": false, | |
| 16 | + "w": 64, | |
| 17 | + "ffn_w": 192, | |
| 18 | + "n_slots": 16, | |
| 19 | + "n_editor_blocks": 2, | |
| 20 | + "n_editor_heads": 2, | |
| 21 | + "r_max": 8, | |
| 22 | + "eta_max": 0.25, | |
| 23 | + "clip_c": 1.0, | |
| 24 | + "clip_eps": 1e-06, | |
| 25 | + "readout_zero_init": true, | |
| 26 | + "controller_hidden": 0, | |
| 27 | + "lambda_cost": 0.0, | |
| 28 | + "kappa_risk": 0.0, | |
| 29 | + "dense_writes": false | |
| 30 | + }, | |
| 31 | + "saved_at": "2026-10-29T23:57:16-0400", | |
| 32 | + "step": 5000, | |
| 33 | + "run_id": "p4_tiny_backbone_v2_seed01_20261029233624", | |
| 34 | + "weights_sha256": "71927bfc6ee34ea898424f7c3c655f14e7c3257b12075f126003d51f5a1886f1" | |
| 35 | +} | |
| \ No newline at end of file | ||
modified
experiments/runs/p4_tiny_backbone_v2_seed01_20261029233624/manifest.json
+1 −1
@@ -9,7 +9,7 @@ | ||
| 9 | 9 | "python": "3.12.13", |
| 10 | 10 | "torch": "2.15.0", |
| 11 | 11 | "command": "/Users/simon-pierreboucher/Desktop/Recherche-UQO/ia/prisme-slm/training/train_editor.py --config configs/experiments/p4_tiny_backbone_v2.yaml --device mps", |
| 12 | − "state": "running", | |
| 12 | + "state": "stopped_overfitting_step9300_checkpoint_step5000_used", | |
| 13 | 13 | "wall_seconds": 0.02, |
| 14 | 14 | "device": "mps", |
| 15 | 15 | "backend": "mps", |
modified
experiments/runs/p4_tiny_backbone_v2_seed01_20261029233624/metrics.jsonl
+81 −0
@@ -23,3 +23,84 @@ | ||
| 23 | 23 | {"step": 2000, "loss": 0.25057756900787354, "lr": 0.000970626002548074, "grad_norm": 0.12103113532066345, "tokens": 14797545, "tok_per_s": 37036.8869671652, "t": 400.15} |
| 24 | 24 | {"step": 2100, "loss": 0.23892416059970856, "lr": 0.0009671125408057098, "grad_norm": 0.08961229771375656, "tokens": 15540833, "tok_per_s": 36457.20327366902, "t": 426.89} |
| 25 | 25 | {"step": 2200, "loss": 0.24898186326026917, "lr": 0.0009634085760746613, "grad_norm": 0.14835017919540405, "tokens": 16275558, "tok_per_s": 35731.45537336063, "t": 456.11} |
| 26 | +{"step": 2300, "loss": 0.23595838248729706, "lr": 0.00095951580002392, "grad_norm": 0.16141901910305023, "tokens": 17022061, "tok_per_s": 35139.94385487974, "t": 485.02} | |
| 27 | +{"step": 2400, "loss": 0.24106518924236298, "lr": 0.0009554359905560886, "grad_norm": 0.0851842537522316, "tokens": 17763312, "tok_per_s": 34671.051664215454, "t": 512.95} | |
| 28 | +{"step": 2500, "loss": 0.20480993390083313, "lr": 0.0009511710109953803, "grad_norm": 0.0695372149348259, "tokens": 18498245, "tok_per_s": 34175.30851428889, "t": 541.89} | |
| 29 | +{"step": 2600, "loss": 0.2770192623138428, "lr": 0.000946722809236604, "grad_norm": 0.1085411012172699, "tokens": 19238526, "tok_per_s": 33737.99756176212, "t": 570.84} | |
| 30 | +{"step": 2700, "loss": 0.22194169461727142, "lr": 0.0009420934168555253, "grad_norm": 0.07198011875152588, "tokens": 19973518, "tok_per_s": 33280.918869557994, "t": 600.76} | |
| 31 | +{"step": 2800, "loss": 0.24996443092823029, "lr": 0.0009372849481810073, "grad_norm": 0.11180724948644638, "tokens": 20706938, "tok_per_s": 32946.24092319232, "t": 629.12} | |
| 32 | +{"step": 2900, "loss": 0.25116056203842163, "lr": 0.0009322995993293593, "grad_norm": 0.08267257362604141, "tokens": 21455185, "tok_per_s": 32592.520323627323, "t": 658.9} | |
| 33 | +{"step": 2999, "val_loss_b0": 0.2490523274094323, "t": 688.03} | |
| 34 | +{"step": 3000, "loss": 0.2259361892938614, "lr": 0.0009271396472013282, "grad_norm": 0.08093008399009705, "tokens": 22195465, "tok_per_s": 32275.792960844774, "t": 688.29} | |
| 35 | +{"step": 3100, "loss": 0.23215486109256744, "lr": 0.0009218074484421977, "grad_norm": 0.09304545074701309, "tokens": 22927890, "tok_per_s": 32022.084999155453, "t": 716.61} | |
| 36 | +{"step": 3200, "loss": 0.26163896918296814, "lr": 0.0009163054383654632, "grad_norm": 0.10492544621229172, "tokens": 23665276, "tok_per_s": 31863.551082722563, "t": 743.32} | |
| 37 | +{"step": 3300, "loss": 0.25118446350097656, "lr": 0.0009106361298405805, "grad_norm": 0.09422911703586578, "tokens": 24408498, "tok_per_s": 32003.86730721813, "t": 763.28} | |
| 38 | +{"step": 3400, "loss": 0.239703968167305, "lr": 0.0009048021121452915, "grad_norm": 0.1178416982293129, "tokens": 25152353, "tok_per_s": 32149.98823824634, "t": 782.95} | |
| 39 | +{"step": 3500, "loss": 0.25807860493659973, "lr": 0.0008988060497830534, "grad_norm": 0.09707698225975037, "tokens": 25888139, "tok_per_s": 32197.485808180354, "t": 804.66} | |
| 40 | +{"step": 3600, "loss": 0.23271173238754272, "lr": 0.0008926506812661105, "grad_norm": 0.08569994568824768, "tokens": 26630664, "tok_per_s": 31958.49396409735, "t": 833.9} | |
| 41 | +{"step": 3700, "loss": 0.2531582713127136, "lr": 0.0008863388178647659, "grad_norm": 0.0903114601969719, "tokens": 27369665, "tok_per_s": 31740.88070223157, "t": 862.9} | |
| 42 | +{"step": 3800, "loss": 0.2413078397512436, "lr": 0.0008798733423234219, "grad_norm": 0.08409066498279572, "tokens": 28115168, "tok_per_s": 31492.4174954626, "t": 893.37} | |
| 43 | +{"step": 3900, "loss": 0.22927093505859375, "lr": 0.0008732572075439775, "grad_norm": 0.08642493188381195, "tokens": 28852349, "tok_per_s": 31122.18916815221, "t": 927.68} | |
| 44 | +{"step": 3999, "val_loss_b0": 0.24673577576941852, "t": 966.29} | |
| 45 | +{"step": 4000, "loss": 0.24888259172439575, "lr": 0.0008664934352371838, "grad_norm": 0.08086570352315903, "tokens": 29598638, "tok_per_s": 30637.600806213628, "t": 966.7} | |
| 46 | +{"step": 4100, "loss": 0.23915867507457733, "lr": 0.0008595851145425734, "grad_norm": 0.08897045254707336, "tokens": 30340887, "tok_per_s": 30066.126563382648, "t": 1009.75} | |
| 47 | +{"step": 4200, "loss": 0.2620323598384857, "lr": 0.0008525354006175926, "grad_norm": 0.09605882316827774, "tokens": 31075053, "tok_per_s": 29704.82556897383, "t": 1046.74} | |
| 48 | +{"step": 4300, "loss": 0.23395636677742004, "lr": 0.0008453475131965832, "grad_norm": 0.09766869246959686, "tokens": 31805156, "tok_per_s": 29540.262818070223, "t": 1077.28} | |
| 49 | +{"step": 4400, "loss": 0.21586599946022034, "lr": 0.0008380247351202708, "grad_norm": 0.08858782798051834, "tokens": 32546876, "tok_per_s": 29485.45496382066, "t": 1104.44} | |
| 50 | +{"step": 4500, "loss": 0.23905375599861145, "lr": 0.0008305704108364302, "grad_norm": 0.10618754476308823, "tokens": 33283782, "tok_per_s": 29487.24124656912, "t": 1129.36} | |
| 51 | +{"step": 4600, "loss": 0.24594713747501373, "lr": 0.0008229879448724149, "grad_norm": 0.09999126195907593, "tokens": 34028571, "tok_per_s": 29513.085509099164, "t": 1153.61} | |
| 52 | +{"step": 4700, "loss": 0.23214955627918243, "lr": 0.000815280800280247, "grad_norm": 0.08282716572284698, "tokens": 34766082, "tok_per_s": 29545.36412883303, "t": 1177.31} | |
| 53 | +{"step": 4800, "loss": 0.23158258199691772, "lr": 0.0008074524970549763, "grad_norm": 0.0798744335770607, "tokens": 35503604, "tok_per_s": 29570.258821686195, "t": 1201.27} | |
| 54 | +{"step": 4900, "loss": 0.22221843898296356, "lr": 0.0007995066105270354, "grad_norm": 0.08601359277963638, "tokens": 36246522, "tok_per_s": 29586.25869193613, "t": 1225.73} | |
| 55 | +{"step": 4999, "val_loss_b0": 0.2457874837291204, "t": 1251.74} | |
| 56 | +{"step": 4999, "checkpoint_sha256": "71927bfc6ee34ea898424f7c3c655f14e7c3257b12075f126003d51f5a1886f1", "t": 1252.0} | |
| 57 | +{"step": 5000, "loss": 0.22774429619312286, "lr": 0.0007914467697293183, "grad_norm": 0.0813710018992424, "tokens": 36994918, "tok_per_s": 29557.386772961494, "t": 1252.25} | |
| 58 | +{"step": 5100, "loss": 0.23766636848449707, "lr": 0.0007832766557397363, "grad_norm": 0.11291572451591492, "tokens": 37736785, "tok_per_s": 29557.424459246053, "t": 1277.34} | |
| 59 | +{"step": 5200, "loss": 0.24797874689102173, "lr": 0.0007750000000000001, "grad_norm": 0.09950819611549377, "tokens": 38482669, "tok_per_s": 29550.341683950977, "t": 1302.89} | |
| 60 | +{"step": 5300, "loss": 0.2394346445798874, "lr": 0.0007666205826114025, "grad_norm": 0.09214076399803162, "tokens": 39217877, "tok_per_s": 29552.652553165975, "t": 1327.68} | |
| 61 | +{"step": 5400, "loss": 0.22690193355083466, "lr": 0.000758142230608376, "grad_norm": 0.08511560410261154, "tokens": 39961932, "tok_per_s": 29588.87784254362, "t": 1351.19} | |
| 62 | +{"step": 5500, "loss": 0.22264301776885986, "lr": 0.000749568816210616, "grad_norm": 0.07027542591094971, "tokens": 40689343, "tok_per_s": 29511.284566847276, "t": 1379.39} | |
| 63 | +{"step": 5600, "loss": 0.2372397482395172, "lr": 0.0007409042550545667, "grad_norm": 0.0994800329208374, "tokens": 41419954, "tok_per_s": 29347.896095082524, "t": 1411.95} | |
| 64 | +{"step": 5700, "loss": 0.2575412094593048, "lr": 0.0007321525044050772, "grad_norm": 0.15150783956050873, "tokens": 42153008, "tok_per_s": 29307.818599319744, "t": 1438.9} | |
| 65 | +{"step": 5800, "loss": 0.2326875925064087, "lr": 0.0007233175613480459, "grad_norm": 0.09410254657268524, "tokens": 42891344, "tok_per_s": 29242.80011876678, "t": 1467.34} | |
| 66 | +{"step": 5900, "loss": 0.22659200429916382, "lr": 0.0007144034609648779, "grad_norm": 0.07550016790628433, "tokens": 43636103, "tok_per_s": 29178.531651642897, "t": 1496.1} | |
| 67 | +{"step": 5999, "val_loss_b0": 0.24712026692656713, "t": 1525.28} | |
| 68 | +{"step": 6000, "loss": 0.23550210893154144, "lr": 0.0007054142744895884, "grad_norm": 0.09134833514690399, "tokens": 44374883, "tok_per_s": 29099.076135675226, "t": 1525.57} | |
| 69 | +{"step": 6100, "loss": 0.24578362703323364, "lr": 0.0006963541074493962, "grad_norm": 0.09056496620178223, "tokens": 45111988, "tok_per_s": 29000.670551559306, "t": 1556.16} | |
| 70 | +{"step": 6200, "loss": 0.20576344430446625, "lr": 0.0006872270977896524, "grad_norm": 0.08477317541837692, "tokens": 45852797, "tok_per_s": 28935.58037167095, "t": 1585.26} | |
| 71 | +{"step": 6300, "loss": 0.22447314858436584, "lr": 0.0006780374139839646, "grad_norm": 0.09896229207515717, "tokens": 46588536, "tok_per_s": 28870.616629293163, "t": 1614.31} | |
| 72 | +{"step": 6400, "loss": 0.21796773374080658, "lr": 0.0006687892531303794, "grad_norm": 0.09591249376535416, "tokens": 47327551, "tok_per_s": 28820.224747576834, "t": 1642.78} | |
| 73 | +{"step": 6500, "loss": 0.22234489023685455, "lr": 0.000659486839034489, "grad_norm": 0.08444668352603912, "tokens": 48061930, "tok_per_s": 28780.156796154653, "t": 1670.58} | |
| 74 | +{"step": 6600, "loss": 0.22978655993938446, "lr": 0.0006501344202803415, "grad_norm": 0.09093228727579117, "tokens": 48799166, "tok_per_s": 28825.732926860233, "t": 1693.52} | |
| 75 | +{"step": 6700, "loss": 0.22826802730560303, "lr": 0.000640736268290035, "grad_norm": 0.0852239578962326, "tokens": 49542736, "tok_per_s": 28817.446488615966, "t": 1719.8} | |
| 76 | +{"step": 6800, "loss": 0.24896718561649323, "lr": 0.0006312966753728795, "grad_norm": 0.11489948630332947, "tokens": 50279703, "tok_per_s": 28817.993277229, "t": 1745.34} | |
| 77 | +{"step": 6900, "loss": 0.24561363458633423, "lr": 0.0006218199527650207, "grad_norm": 0.10542216151952744, "tokens": 51014086, "tok_per_s": 28883.15061642516, "t": 1766.83} | |
| 78 | +{"step": 6999, "val_loss_b0": 0.2523172920760197, "t": 1798.64} | |
| 79 | +{"step": 7000, "loss": 0.22365900874137878, "lr": 0.0006123104286604183, "grad_norm": 0.10404888540506363, "tokens": 51756321, "tok_per_s": 28779.93868386071, "t": 1798.96} | |
| 80 | +{"step": 7100, "loss": 0.21811366081237793, "lr": 0.0006027724462340809, "grad_norm": 0.09947623312473297, "tokens": 52501419, "tok_per_s": 28741.686862435763, "t": 1827.27} | |
| 81 | +{"step": 7200, "loss": 0.18835552036762238, "lr": 0.0005932103616584568, "grad_norm": 0.09692477434873581, "tokens": 53242348, "tok_per_s": 28672.112593973903, "t": 1857.55} | |
| 82 | +{"step": 7300, "loss": 0.222006157040596, "lr": 0.000583628542113891, "grad_norm": 0.08617004752159119, "tokens": 53980673, "tok_per_s": 28646.261278749655, "t": 1885.0} | |
| 83 | +{"step": 7400, "loss": 0.23116207122802734, "lr": 0.0005740313637940521, "grad_norm": 0.10457632690668106, "tokens": 54718992, "tok_per_s": 28581.463639404403, "t": 1915.1} | |
| 84 | +{"step": 7500, "loss": 0.21480606496334076, "lr": 0.0005644232099072449, "grad_norm": 0.09749004244804382, "tokens": 55464703, "tok_per_s": 28561.72064571933, "t": 1942.54} | |
| 85 | +{"step": 7600, "loss": 0.22075559198856354, "lr": 0.0005548084686745181, "grad_norm": 0.09191872179508209, "tokens": 56202787, "tok_per_s": 28584.31991083341, "t": 1966.83} | |
| 86 | +{"step": 7700, "loss": 0.233114093542099, "lr": 0.0005451915313254822, "grad_norm": 0.11089321970939636, "tokens": 56944732, "tok_per_s": 28587.51970203095, "t": 1992.55} | |
| 87 | +{"step": 7800, "loss": 0.20631960034370422, "lr": 0.0005355767900927552, "grad_norm": 0.10566965490579605, "tokens": 57687990, "tok_per_s": 28600.29852093074, "t": 2017.65} | |
| 88 | +{"step": 7900, "loss": 0.21017761528491974, "lr": 0.0005259686362059481, "grad_norm": 0.10289314389228821, "tokens": 58429571, "tok_per_s": 28637.497046198143, "t": 2040.93} | |
| 89 | +{"step": 7999, "val_loss_b0": 0.26403789397260957, "t": 2065.78} | |
| 90 | +{"step": 8000, "loss": 0.21417582035064697, "lr": 0.0005163714578861091, "grad_norm": 0.11148758232593536, "tokens": 59162425, "tok_per_s": 28641.94408279909, "t": 2066.2} | |
| 91 | +{"step": 8100, "loss": 0.1658760905265808, "lr": 0.0005067896383415433, "grad_norm": 0.08317413926124573, "tokens": 59903257, "tok_per_s": 28592.063785446953, "t": 2095.71} | |
| 92 | +{"step": 8200, "loss": 0.20030342042446136, "lr": 0.0004972275537659192, "grad_norm": 0.13063369691371918, "tokens": 60647639, "tok_per_s": 28560.088128883992, "t": 2124.12} | |
| 93 | +{"step": 8300, "loss": 0.20933914184570312, "lr": 0.0004876895713395816, "grad_norm": 0.11644300073385239, "tokens": 61390376, "tok_per_s": 28528.063007501823, "t": 2152.54} | |
| 94 | +{"step": 8400, "loss": 0.1932450383901596, "lr": 0.0004781800472349794, "grad_norm": 0.11099286377429962, "tokens": 62132384, "tok_per_s": 28485.408573198627, "t": 2181.81} | |
| 95 | +{"step": 8500, "loss": 0.20041947066783905, "lr": 0.0004687033246271206, "grad_norm": 0.11818403750658035, "tokens": 62871250, "tok_per_s": 28490.70425245547, "t": 2207.34} | |
| 96 | +{"step": 8600, "loss": 0.2051982879638672, "lr": 0.00045926373170996514, "grad_norm": 0.12751081585884094, "tokens": 63599816, "tok_per_s": 28484.227708673072, "t": 2233.42} | |
| 97 | +{"step": 8700, "loss": 0.2038002759218216, "lr": 0.0004498655797196586, "grad_norm": 0.11766581237316132, "tokens": 64339358, "tok_per_s": 28452.33818232927, "t": 2261.91} | |
| 98 | +{"step": 8800, "loss": 0.19860033690929413, "lr": 0.00044051316096551117, "grad_norm": 0.1309284120798111, "tokens": 65080898, "tok_per_s": 28413.644137607473, "t": 2291.09} | |
| 99 | +{"step": 8900, "loss": 0.20576757192611694, "lr": 0.0004312107468696206, "grad_norm": 0.12305575609207153, "tokens": 65823979, "tok_per_s": 28355.499522381953, "t": 2321.99} | |
| 100 | +{"step": 8999, "val_loss_b0": 0.28220341540577676, "t": 2351.33} | |
| 101 | +{"step": 9000, "loss": 0.19135911762714386, "lr": 0.00042196258601603546, "grad_norm": 0.14417314529418945, "tokens": 66572928, "tok_per_s": 28316.65131685448, "t": 2351.63} | |
| 102 | +{"step": 9100, "loss": 0.2039412409067154, "lr": 0.0004127729022103478, "grad_norm": 0.14502999186515808, "tokens": 67310597, "tok_per_s": 28278.401227138056, "t": 2380.89} | |
| 103 | +{"step": 9200, "loss": 0.19043566286563873, "lr": 0.0004036458925506039, "grad_norm": 0.1531902253627777, "tokens": 68048686, "tok_per_s": 28272.657269177755, "t": 2407.48} | |
| 104 | +{"step": 9300, "loss": 0.1848042607307434, "lr": 0.0003945857255104116, "grad_norm": 0.13151361048221588, "tokens": 68785548, "tok_per_s": 28289.174138224687, "t": 2432.13} | |
| 105 | +{"step": 9400, "loss": 0.16617880761623383, "lr": 0.00038559653903512224, "grad_norm": 0.13529454171657562, "tokens": 69518667, "tok_per_s": 28315.869980071766, "t": 2455.72} | |
| 106 | +{"step": 9500, "loss": 0.1821175515651703, "lr": 0.00037668243865195415, "grad_norm": 0.15534792840480804, "tokens": 70256594, "tok_per_s": 28380.86706972068, "t": 2476.1} | |
modified
literature/papers.bib
+9 −0
@@ -806,3 +806,12 @@ | ||
| 806 | 806 | url = {https://arxiv.org/abs/2203.15556}, |
| 807 | 807 | note = {Chinchilla. arXiv v1 2022-03-29. Venue (NeurIPS 2022) NOT verified from arXiv metadata.} |
| 808 | 808 | } |
| 809 | + | |
| 810 | +@article{penedo2024fineweb, | |
| 811 | + title = {The {FineWeb} Datasets: Decanting the Web for the Finest Text Data at Scale}, | |
| 812 | + author = {Penedo, Guilherme and Kydl{\'\i}{\v{c}}ek, Hynek and Ben allal, Loubna and Lozhkov, Anton and Mitchell, Margaret and Raffel, Colin and Von Werra, Leandro and Wolf, Thomas}, | |
| 813 | + year = {2024}, | |
| 814 | + journal = {arXiv preprint arXiv:2406.17557}, | |
| 815 | + url = {https://arxiv.org/abs/2406.17557}, | |
| 816 | + note = {Verified on the arXiv API 2026-10-30 (published 2024-06-25). FineWeb-Edu sample-10BT, ODC-By 1.0.} | |
| 817 | +} | |
added
paper/.latexmkrc
+5 −0
@@ -0,0 +1,5 @@ | ||
| 1 | +$pdf_mode = 1; | |
| 2 | +$pdflatex = 'pdflatex -interaction=nonstopmode -halt-on-error -synctex=1 %O %S'; | |
| 3 | +$bibtex_use = 2; | |
| 4 | +$clean_ext = 'synctex.gz run.xml bbl bcf fdb_latexmk fls log aux out toc lof lot blg'; | |
| 5 | +@default_files = ('main.tex'); | |
added
paper/Makefile
+11 −0
@@ -0,0 +1,11 @@ | ||
| 1 | +.PHONY: all clean distclean | |
| 2 | + | |
| 3 | +all: | |
| 4 | + latexmk | |
| 5 | + | |
| 6 | +clean: | |
| 7 | + latexmk -c | |
| 8 | + | |
| 9 | +distclean: | |
| 10 | + latexmk -C | |
| 11 | + rm -f *.synctex.gz *.run.xml *.bbl *.bcf | |
added
paper/appendix/appendix.tex
+18 −0
@@ -0,0 +1,18 @@ | ||
| 1 | +\begin{appendices} | |
| 2 | +\section{Notation}\label{app:notation} | |
| 3 | +\input{appendix/notation.tex} | |
| 4 | +\section{Proofs}\label{app:proofs} | |
| 5 | +\input{appendix/proofs.tex} | |
| 6 | +\section{Parameter reconciliation}\label{app:params} | |
| 7 | +\input{appendix/params.tex} | |
| 8 | +\section{Literature audit}\label{app:literature} | |
| 9 | +\input{appendix/literature.tex} | |
| 10 | +\section{Pre-registration and deviations}\label{app:prereg} | |
| 11 | +\input{appendix/prereg.tex} | |
| 12 | +\section{Hyper-parameters, commands and run index}\label{app:runs} | |
| 13 | +\input{appendix/runs.tex} | |
| 14 | +\section{Complete results}\label{app:results} | |
| 15 | +\input{appendix/results.tex} | |
| 16 | +\section{Failures}\label{app:failures} | |
| 17 | +\input{appendix/failures.tex} | |
| 18 | +\end{appendices} | |
added
paper/appendix/failures.tex
+21 −0
@@ -0,0 +1,21 @@ | ||
| 1 | +\subsection*{FAILURES.md --- failed hypotheses, tests and attempts (never deleted)}\label{failures.md-failed-hypotheses-tests-and-attempts-never-deleted} | |
| 2 | + | |
| 3 | +\begin{itemize} | |
| 4 | +\tightlist | |
| 5 | +\item | |
| 6 | + \textbf{date}: 2026-10-29; \textbf{what failed}: \texttt{test\_incremental\_equals\_full\_prefix\_forced}: cached decoding $\neq$ full-prefix forward.; \textbf{evidence}: pytest, atol 1e-4; \textbf{resolution}: Root cause: forced-action forward let edits resume after STOP; made STOP absorbing (D4). | |
| 7 | +\item | |
| 8 | + \textbf{date}: 2026-10-29; \textbf{what failed}: rsync exclusion \texttt{experiments} also dropped \texttt{configs/experiments/}; baseline launch on M3U96b failed twice (\texttt{FileNotFoundError}).; \textbf{evidence}: \texttt{experiments/logs/pretrain\_enlarged102m\_seed1.log} (first attempts); \textbf{resolution}: anchored pattern \texttt{/experiments}. | |
| 9 | +\item | |
| 10 | + \textbf{date}: 2026-10-29; \textbf{what failed}: Stage B bootstrap degeneracy (brief §13.1 confirmed): with U = 0 and lr 1e-3, 700 steps of forced mixed-budget training left val loss identical at every budget (0.4144 / 0.4143 / 0.4143 / 0.4146 / 0.4164 for budgets 0/1/2/4/8); grad-norm 0.01--0.02. Diagnosis: scale mismatch --- backbone residual rms 5.64 vs readout difference rms 0.06, so U must grow by $\times$100 before edits can affect logits.; \textbf{evidence}: \texttt{experiments/runs/p4\_editor\_v1\_seed01\_*/metrics.jsonl} (stopped at step \textasciitilde750); \textbf{resolution}: D9: fixed non-learned \texttt{readout\_scale} = measured backbone residual rms at bootstrap (keeps U = 0 init and exact identity; equivalent to a scale-aware learning rate). Variants with nonzero U init (std 0.02) and lr 3e-3 kept running on M2U64 / M4M64c as evidence. | |
| 11 | +\item | |
| 12 | + \textbf{date}: 2026-10-30; \textbf{what failed}: Editor collapse at lr 3e-3 (w = 128 mixed budgets): after 500 steps every budget gives exactly the budget-0 loss (0.4005) and the trainable gradient norm is 0.00 --- the adapter learned to make all edits inert (gate/U $\rightarrow$ 0) instead of useful.; \textbf{evidence}: M4M64a \texttt{experiments/logs/p4\_diag\_mixed\_w128\_lr3e3.log}, run \texttt{p4\_diag\_mixed\_w128\_lr3e3\_seed01\_*}; \textbf{resolution}: ``harmless collapse'' is a real failure mode of the bootstrap; lr 1e-3 retained. | |
| 13 | +\item | |
| 14 | + \textbf{date}: 2026-10-30; \textbf{what failed}: Global-recurrence baseline run crashed at the final exact-match evaluation (\texttt{AttributeError:\ \textquotesingle{}GlobalRecurrenceModel\textquotesingle{}\ object\ has\ no\ attribute\ \textquotesingle{}W\_K\textquotesingle{}}): the cached generator assumed PRISME caches. Training and the step-3000 checkpoint are intact.; \textbf{evidence}: M5M36 \texttt{experiments/logs/p4\_gr\_mixed\_w64.log}; \textbf{resolution}: added \texttt{generate\_uncached} fallback (full recomputation) for models without the bank; EM re-run from the checkpoint. | |
| 15 | +\item | |
| 16 | + \textbf{date}: 2026-10-30; \textbf{what failed}: Joint fine-tuning reference (backbone unfrozen, lr 3e-4, answer-span loss): validation loss rose from 0.4005 to 0.472 at every budget after 500 steps --- the backbone degraded under the narrower objective and the large learning rate.; \textbf{evidence}: M4M64a \texttt{experiments/logs/p4\_joint\_mixed\_w64.log}; \textbf{resolution}: Not a usable upper reference as run; a low-lr variant is deferred (not required by the pre-registered comparison). | |
| 17 | +\item | |
| 18 | + \textbf{date}: 2026-10-30; \textbf{what failed}: v2 tiny backbone (5.6M, 15k planned steps) overfit the 23k-item train split: validation loss (all tokens) 0.246 at step 6k $\rightarrow$ 0.282 at step 9k. Run stopped at step 9,300; the step-5,000 checkpoint (val 0.249) is used as the frozen v2 backbone.; \textbf{evidence}: \texttt{experiments/logs/p4\_tiny\_backbone\_v2.log}, run \texttt{p4\_tiny\_backbone\_v2\_seed01\_*} (state: stopped); \textbf{resolution}: 42 epochs of a small synthetic set is too many; schedule not re-run (the checkpoint is adequate as a frozen comparator). | |
| 19 | +\item | |
| 20 | + \textbf{date}: 2026-10-30; \textbf{what failed}: Killed to free nodes for the 5-seed matrix: the degraded joint-finetune reference (M4M64a) and the v1 dense-writes run on the slow m4mn node (superseded by the v2 matrix \texttt{p4v2\_dense}).; \textbf{evidence}: manifests remain in \texttt{experiments/runs/} on the nodes (state: running $\rightarrow$ killed) | |
| 21 | +\end{itemize} | |
added
paper/appendix/literature.tex
+22 −0
@@ -0,0 +1,22 @@ | ||
| 1 | +All 78 references of this paper were resolved through the arXiv Atom API or the publisher page at execution time | |
| 2 | +(2026-10-29/30); mechanisms of the closest neighbours were read from their full texts, and code availability and | |
| 3 | +licences from the official repositories (\texttt{literature/papers.bib}, \texttt{comparison\_matrix.csv}, | |
| 4 | +\texttt{novelty\_audit.md}). Table~\ref{tab:novelty} summarises the claim-by-claim audit. | |
| 5 | + | |
| 6 | +\begin{table}[h]\centering\small | |
| 7 | +\caption{Claim-by-claim novelty audit (2026-10-29).}\label{tab:novelty} | |
| 8 | +\begin{tabularx}{\textwidth}{p{3.1cm}p{3.4cm}XX}\toprule | |
| 9 | +Proposed claim & Closest sources & Actual distinction & Status\\\midrule | |
| 10 | +C1 selected local writes into a per-token slot workspace & RIMs, Shared Global Workspace, Perceiver AR, Slot Attention, SAM, ReFlux & no prior work combines a non-persistent per-token slot bank, a separate read-only cached source bank, and discrete 1--2-slot write actions & partially novel\\ | |
| 11 | +C2 action-conditional counterfactual gain supervision with logged propensities & Ouro stage II, TaH2 ``lookahead depth supervision'', RecurTrace oracle halting; Dud\'ik et al., Swaminathan \& Joachims & binary continue/stop versions exist; conditioning the label on which of 24 spatial actions is taken, with propensities and separate gain/cost/risk/harm heads, has no predecessor & partially novel\\ | |
| 12 | +C3 differential readout with exact zero-budget bypass & ReZero, Flamingo, ControlNet, Mixture-of-Depths, TaH/TaH2, ReFlux, ELR & a modest variant of zero-initialised identity insertion & not novel\\ | |
| 13 | +C4 the combination C1+C2+C3 & ReFlux, TaH2, RecurTrace, MoR & no work has all three; the contribution must be empirical & partially novel\\ | |
| 14 | +C5 catalogue of 16 singles + 8 fixed pairs + \Stop & top-$k$ expert/mechanism routing, DQN discrete action sets & a design choice; reported with sensitivity analysis & not novel\\ | |
| 15 | +\bottomrule\end{tabularx} | |
| 16 | +\end{table} | |
| 17 | + | |
| 18 | +Near neighbours published after June 2025 and re-checked at execution time include Ouro, Mixture-of-Recursions, | |
| 19 | +PonderLM, Inner Thinking Transformer, Encode--Think--Decode, Parallel Loop Transformer, Think-at-Hard and TaH2, | |
| 20 | +LoopFormer, AdaPonderLM, latent lookahead, LoopUS, the Latent Recurrent Transformer, LaTER, Depth-Attention, | |
| 21 | +DiscoLoop, Logan, RecurTrace, ReFlux and Elastic Latent Recurrence (all in \texttt{papers.bib}); the arXiv | |
| 22 | +listing should be re-checked the week before any submission. | |
added
paper/appendix/notation.tex
+16 −0
@@ -0,0 +1,16 @@ | ||
| 1 | +\begin{center}\small | |
| 2 | +\begin{tabular}{ll}\toprule | |
| 3 | +Symbol & Meaning\\\midrule | |
| 4 | +$x_{1:t}$, $V$ & token prefix; vocabulary size (32,768)\\ | |
| 5 | +$B_\phi$, $H_t$, $h_t$, $d$ & backbone, residual stream, its last row, backbone width\\ | |
| 6 | +$\hat H_t$, $K_t$, $V_t$, $z_t$, $w$ & bank-normalised stream, source-bank keys/values, summary, workspace width\\ | |
| 7 | +$M$, $e_j$, $S^{(r)}$ & number of slots (16), address embeddings, workspace after $r$ actions\\ | |
| 8 | +$\Act$, $I(a)$, $k(a)$, \Stop, $R_{\max}$ & nonterminal actions (24), written index set, group size, stop, cap (8)\\ | |
| 9 | +$L_w$, $f_w$ & editor blocks (2, shared), editor FFN width\\ | |
| 10 | +$\delta_j,\delta^c_j,\eta_j,c,\eta_{\max}$ & proposed change, clipped change, gate, clip threshold (1), max step (0.25)\\ | |
| 11 | +$r(S,z)$, $U$, $s$, $\Delta_t$ & attention read, readout matrix (zero-init), fixed scale, correction\\ | |
| 12 | +$g(s,a;y)$, $\hat g,\hat c,\hat\sigma,\hat\ell_{\mathrm{harm}}$ & intervention gain and its four predicted heads\\ | |
| 13 | +$Q(s,a)$, $\lambda$, $\kappa$, $c(a)$ & action score, cost and risk penalties, analytic cost of an action\\ | |
| 14 | +$n_t$, $C_t$ & executed edits at position $t$, realised relative cost\\ | |
| 15 | +\bottomrule\end{tabular} | |
| 16 | +\end{center} | |
added
paper/appendix/params.tex
+16 −0
@@ -0,0 +1,16 @@ | ||
| 1 | +The backbone counts of Table~\ref{tab:configs} are reproduced exactly by | |
| 2 | +$N_{\mathrm{bb}} = Vd + L\,(2d^2 + 2d\,d_{kv} + 3df + 2d) + d$ with $d_{kv}=n_{kv}\cdot 64$ (tied embeddings, two | |
| 3 | +RMSNorm vectors per block, one final norm). The proposal's workspace+controller figures (3,088,260 / 12,336,900 / | |
| 4 | +25,779,844) are reproduced exactly---for all three configurations---by the closed form | |
| 5 | +\[ | |
| 6 | +N^{\mathrm{prop}}_{\mathrm{ws}} = 4dw + Mw + 17w^2 + 6wf_w + 15.5\,w + 4, | |
| 7 | +\] | |
| 8 | +which we inferred algebraically from the three published values. The implemented design counts | |
| 9 | +\[ | |
| 10 | +N^{\mathrm{impl}}_{\mathrm{ws}} = d + 3dw + Mw + \underbrace{(6w^2 + 3wf_w + 3w)}_{\text{editor block}} + \underbrace{(4w + 2w + 1)}_{\text{ctx, gate}} | |
| 11 | + + \underbrace{(2w^2 + wd)}_{\text{readout}} + \underbrace{(2w^2 + 3w + w + 3wf_w + 2w + 4w + 4)}_{\text{controller}}, | |
| 12 | +\] | |
| 13 | +i.e.\ $10w^2$ square projections instead of $17w^2$ and a few extra vectors: the editor reads the cached bank | |
| 14 | +without re-projecting it (no $K/V$ projections, $-2w^2$), the readout uses raw slot values ($-w^2$), and the | |
| 15 | +controller shares one state MLP across actions ($-4w^2$). The difference is 457,087 (103M), 1,831,679 (459M) and | |
| 16 | +4,124,031 (1.18B) parameters. We did not add inert projections to match the proposal's figures. | |
added
paper/appendix/prereg.tex
+59 −0
@@ -0,0 +1,59 @@ | ||
| 1 | +\subsection*{\texorpdfstring{Pre-registration --- WP31 PRISME pilot (DRAFT v0.1, 2026-10-29; to be FROZEN before any \texttt{test\_*} split is read)}{Pre-registration --- WP31 PRISME pilot (DRAFT v0.1, 2026-10-29; to be FROZEN before any test\_* split is read)}}\label{pre-registration-wp31-prisme-pilot-draft-v0.1-2026-10-29-to-be-frozen-before-any-test_-split-is-read} | |
| 2 | + | |
| 3 | +Status: DRAFT. No file named \texttt{test*.jsonl} / \texttt{test.bin} has been opened by any evaluation script so far (\texttt{protocol/locked\_test\_manifest.json} records the split hashes and the unsealing timestamp once frozen). | |
| 4 | + | |
| 5 | +\subsubsection*{1. Primary benchmark suite}\label{primary-benchmark-suite} | |
| 6 | + | |
| 7 | +\begin{itemize} | |
| 8 | +\tightlist | |
| 9 | +\item | |
| 10 | + \textbf{Synthetic compositional suite v1} (\texttt{data/splits/synthetic\_v1}, generator \texttt{data/generators/synthetic.py}, manifest sha256 in \texttt{manifest.json}): families arith, bind, fsm, excl, sched (train/dev) + compose (held out). Task unit = one item (prompt + answer). Exclusion: items whose full text duplicates an earlier split (dropped at generation time; counts in manifest). Regimes: \texttt{*\_iid} (train templates \& depths, new seed), \texttt{*\_template} (held-out paraphrase templates 3--4), \texttt{*\_depth} (depths 5--6 \textgreater{} train max 4), \texttt{*\_compose} (held-out family). | |
| 11 | +\item | |
| 12 | + \textbf{General language}: FineWeb-Edu held-out \texttt{test.bin} (2.88M tokens, locked; \texttt{val.bin} is the development set). | |
| 13 | +\item | |
| 14 | + Independent benchmarks (GSM8K, EvalPlus) are OUT of the primary scope of the 103M pilot (a \textasciitilde200M-token backbone cannot solve them above chance); they are reported only if a run is executed, as secondary. | |
| 15 | +\end{itemize} | |
| 16 | + | |
| 17 | +\subsubsection*{2. Primary metrics}\label{primary-metrics} | |
| 18 | + | |
| 19 | +\begin{itemize} | |
| 20 | +\tightlist | |
| 21 | +\item | |
| 22 | + Quality (reasoning): exact-match accuracy of free greedy generation on the synthetic test regimes (pooled \texttt{test\_iid+test\_template+test\_depth}, each regime also reported). Quality (language): held-out cross-entropy on \texttt{test.bin} at each budget. | |
| 23 | +\item | |
| 24 | + Cost: (primary) measured median end-to-end decode latency per token at batch 1 on the reference device, with the backbone-only latency as the unit; (secondary) analytic ops/token (\texttt{prisme/flops.py}). | |
| 25 | +\item | |
| 26 | + Joint superiority: a configuration dominates if quality is higher at $\leq$ the comparator's latency, or latency is lower at non-inferior quality (paired CI excludes zero). | |
| 27 | +\end{itemize} | |
| 28 | + | |
| 29 | +\subsubsection*{3. Baselines and tuning budgets}\label{baselines-and-tuning-budgets} | |
| 30 | + | |
| 31 | +Same backbone checkpoint, same data exposure, same mixed-budget training recipe and step count: (a) frozen backbone (budget 0); (b) parameter-matched plain decoder trained from scratch with the backbone recipe; (c) dense 16-slot editor; (d) PRISME editor with random / uniform / fixed-pair routing at budgets 1, 2, 4, 8; (e) entropy and norm-change routers (thresholds tuned on calibration data); (f) global-recurrence adapter (Huginn/ELR-inspired, parameter-matched); (g) learned value controller ($\lambda$, $\kappa$ tuned on calibration rollouts only); (h) exhaustive one-step oracle and two-step oracle (diagnostic only, never plotted as attainable). Hyperparameter budget per adapter: one learning rate (1e-3 for adapters, chosen in Phase 4) and the shared schedule; no per-baseline tuning beyond the router thresholds. | |
| 32 | + | |
| 33 | +\subsubsection*{4. Minimum meaningful effect}\label{minimum-meaningful-effect} | |
| 34 | + | |
| 35 | +\begin{itemize} | |
| 36 | +\tightlist | |
| 37 | +\item | |
| 38 | + Reasoning: +3 percentage points pooled exact match at comparable latency ($\leq$ +10 \% of the comparator's), or --20 \% median latency at non-inferior accuracy (lower CI bound \textgreater{} --1 pp). | |
| 39 | +\item | |
| 40 | + Language: zero-budget cross-entropy within 1 \% (relative) of the frozen backbone (H5 preservation). | |
| 41 | +\item | |
| 42 | + Routing (H2/H3): learned controller loss at matched mean extra-ops within the CI of the best fixed/random schedule is a NULL for H3. | |
| 43 | +\end{itemize} | |
| 44 | + | |
| 45 | +\subsubsection*{5. Seeds}\label{seeds} | |
| 46 | + | |
| 47 | +Editor/controller stage: 5 independent seeds (1--5) at the Phase 4 scale; at 103M as many as the cluster permits within the session (target 3--5, actual count reported). The backbone is a single seed (declared limitation). | |
| 48 | + | |
| 49 | +\subsubsection*{6. Uncertainty}\label{uncertainty} | |
| 50 | + | |
| 51 | +Paired bootstrap over task items (10,000 resamples) for accuracy differences within a seed; across seeds, mean $\pm$ SD and a t-interval over seed-level differences. Tokens of a single run are never treated as independent replications of training. | |
| 52 | + | |
| 53 | +\subsubsection*{7. Stopping / resource rules}\label{stopping-resource-rules} | |
| 54 | + | |
| 55 | +Pilot compute cap: $\leq$ 2 nodes $\times$ 8 h for Stage A, $\leq$ 6 nodes $\times$ 6 h for stages B--D; if a stage cannot complete, the phase is reported as incomplete and downstream claims are withheld. | |
| 56 | + | |
| 57 | +\subsubsection*{8. Deviations}\label{deviations} | |
| 58 | + | |
| 59 | +Any change after freezing is recorded in \texttt{protocol/deviations.md} with justification and timestamp. | |
added
paper/appendix/proofs.tex
+46 −0
@@ -0,0 +1,46 @@ | ||
| 1 | +\begin{proof}[Proof of Proposition~\ref{prop:causal}] | |
| 2 | +By induction on the number of executed actions. Base case: $S^{(0)}_j=e_j+P\,\mathrm{RMSNorm}(h_t)$ with | |
| 3 | +$h_t=B_\phi(x_{1:t})[t]$, a function of $x_{1:t}$ by causality of $B_\phi$. Inductive step: if $S^{(r)}$ and the | |
| 4 | +controller features at step $r$ depend on $x_{1:t}$ only, then the action $a_{r}$ does (it is a function of those | |
| 5 | +features under Assumption~\ref{ass:causal}), the editor's inputs are $S^{(r)}$, the context scalars, and the bank | |
| 6 | +rows $\{(K_j,V_j)\}_{j\le t}$, each a function of $H_t[1{:}t]$ and hence of $x_{1:t}$; so $S^{(r+1)}$ depends on | |
| 7 | +$x_{1:t}$ only. The readout and head are deterministic functions of $(S^{(n_t)},S^{(0)},z_t,h_t)$. In the parallel | |
| 8 | +teacher-forced implementation the per-query mask $j\le t$ enforces the same restriction although all of $H_T$ is | |
| 9 | +resident in one tensor; the unit test perturbs $x_{t+1:T}$ and verifies that $\ell_{1:t}$ is unchanged. | |
| 10 | +\end{proof} | |
| 11 | + | |
| 12 | +\begin{proof}[Proof of Proposition~\ref{prop:identity}] | |
| 13 | +If $n_t=0$ then $S^{(n_t)}=S^{(0)}$ and $r(S^{(n_t)},z_t)-r(S^{(0)},z_t)=0$, hence $\Delta_t=sU0=0$ and | |
| 14 | +$h^{\mathrm{corr}}_t=h_t$. In the implementation two separate branches return the backbone head directly: | |
| 15 | +one when the requested budget is zero (before any slot is allocated or any bank row is read), and one for every | |
| 16 | +position whose edit count is zero after routing; both compute exactly the expression $E\,\mathrm{RMSNorm}(h_t)$ on | |
| 17 | +the same tensor $h_t$ and are therefore bitwise equal to the backbone-only forward on the same device with the same | |
| 18 | +kernels (tests 02--04 and the cache test). The statement about joint fine-tuning is immediate: $h_t$ is whatever | |
| 19 | +the current backbone produces. | |
| 20 | +\end{proof} | |
| 21 | + | |
| 22 | +\begin{proof}[Proof of Proposition~\ref{prop:locality}] | |
| 23 | +The new workspace is $S'=\mathbb{1}[m\in I]\odot C+\mathbb{1}[m\notin I]\odot S$ where | |
| 24 | +$C_m=\sum_{i} \mathbb{1}[\iota_i=m]\,\tilde S'_{\iota_i}$ sums the (at most two) lane results $\tilde S'$ over the | |
| 25 | +lane indices $\iota_i$ of the action. For $m\notin I$ the first term is zero and $S'_m=S_m$ by the identity | |
| 26 | +$0\cdot x + 1\cdot y = y$, which is exact in IEEE arithmetic for finite $y$. For a pair, the two lanes have | |
| 27 | +distinct indices, so $C_m$ is a single lane result plus exact zeros; both lanes read the same snapshot $\bar S$ and | |
| 28 | +neither reads the other's output, so permuting the lanes permutes the summands of an exact sum of one nonzero | |
| 29 | +term and zeros, leaving $C$ unchanged. | |
| 30 | +\end{proof} | |
| 31 | + | |
| 32 | +\begin{proof}[Proof of Proposition~\ref{prop:displacement}] | |
| 33 | +Each write into slot $j$ adds $\eta_j\delta^c_j$ with $0<\eta_j\le\eta_{\max}$ and $\rms(\delta^c_j)\le c$ by the | |
| 34 | +clipping rule (if $\rms(\delta_j)\le c$ the factor is $1$ and $\rms(\delta^c_j)=\rms(\delta_j)\le c$; otherwise | |
| 35 | +$\rms(\delta^c_j)=c\,\rms(\delta_j)/\rms(\delta_j)=c$ up to the $\epsilon$ guard, which can only shrink the step). | |
| 36 | +Since $\rms(\cdot)=\|\cdot\|_2/\sqrt{w}$ is a norm, the triangle inequality gives | |
| 37 | +$\rms(S^{(R)}_j-S^{(0)}_j)\le\sum_{\text{writes into }j}\eta_j\rms(\delta^c_j)\le n_j\eta_{\max}c$. | |
| 38 | +\end{proof} | |
| 39 | + | |
| 40 | +\begin{proof}[Proof of Proposition~\ref{prop:regret}] | |
| 41 | +Let $a^\star=\arg\max_a J(a)$ over $\Act\cup\{\Stop\}$ and $\hat a=\arg\max_a\hat J(a)$. Both $J$ and $\hat J$ | |
| 42 | +assign $0$ to \Stop. For every $a\in\Act$, $|\hat J(a)-J(a)|\le|\hat g-g|+\lambda|\hat c-c|\le\varepsilon+\lambda\delta=:\Delta$, | |
| 43 | +and $|\hat J(\Stop)-J(\Stop)|=0\le\Delta$. Then | |
| 44 | +$J(a^\star)-J(\hat a)=\big(J(a^\star)-\hat J(a^\star)\big)+\big(\hat J(a^\star)-\hat J(\hat a)\big)+\big(\hat J(\hat a)-J(\hat a)\big) | |
| 45 | +\le\Delta+0+\Delta=2\Delta$, the middle term being non-positive by the choice of $\hat a$. | |
| 46 | +\end{proof} | |
added
paper/appendix/results.tex
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +% to be generated/written | |
added
paper/appendix/runs.tex
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +% to be generated/written | |
added
paper/figures/architecture.tex
+36 −0
@@ -0,0 +1,36 @@ | ||
| 1 | +\begin{tikzpicture}[ | |
| 2 | + font=\footnotesize, node distance=4mm and 6mm, | |
| 3 | + box/.style={draw, rounded corners=2pt, minimum height=7mm, minimum width=18mm, align=center, fill=white}, | |
| 4 | + ro/.style={box, fill=gray!8}, | |
| 5 | + ws/.style={box, fill=NavyBlue!8}, | |
| 6 | + arr/.style={-{Latex[length=2mm]}, thick}, | |
| 7 | + warr/.style={-{Latex[length=2mm]}, thick, dashed, NavyBlue}, | |
| 8 | +] | |
| 9 | +\node[ro] (tok) {prefix $x_{1:t}$}; | |
| 10 | +\node[ro, right=of tok] (bb) {causal backbone $B_\phi$\\(KV cache)}; | |
| 11 | +\node[ro, right=of bb] (h) {$H_t$, $h_t$}; | |
| 12 | +\node[ro, below=9mm of h] (bank) {source bank $K_t,V_t$\\(rows $\le t$, cached)}; | |
| 13 | +\node[ro, right=of h] (z) {$z_t=P\hat h_t$}; | |
| 14 | +\node[ws, right=of z] (s0) {$S^{(0)}_j=e_j+z_t$\\$M=16$ slots}; | |
| 15 | +\node[ws, below=9mm of s0] (ed) {shared sparse editor\\$L_w=2$ blocks, lanes $k\le2$}; | |
| 16 | +\node[ws, right=of ed] (ctrl) {value controller\\$Q(s,a)=\hat g-\lambda c-\kappa\hat\sigma$}; | |
| 17 | +\node[ws, right=of s0] (sf) {$S^{(n_t)}$}; | |
| 18 | +\node[ro, right=of sf] (ro) {$\Delta_t=sU\big(r(S^{(n_t)})-r(S^{(0)})\big)$}; | |
| 19 | +\node[ro, below=9mm of ro] (out) {$\ell_t=E\,\mathrm{RMSNorm}(h_t+\Delta_t)$}; | |
| 20 | +\node[ro, below=9mm of ctrl, xshift=8mm] (bypass) {budget 0 / no edit:\\$\ell_t=E\,\mathrm{RMSNorm}(h_t)$ (bitwise)}; | |
| 21 | +\draw[arr] (tok) -- (bb); | |
| 22 | +\draw[arr] (bb) -- (h); | |
| 23 | +\draw[arr] (h) -- (bank); | |
| 24 | +\draw[arr] (h) -- (z); | |
| 25 | +\draw[arr] (z) -- (s0); | |
| 26 | +\draw[arr] (bank) -- (ed); | |
| 27 | +\draw[arr] (s0) -- (ed); | |
| 28 | +\draw[warr] (ed) -- node[right, pos=0.5, xshift=1mm] {write $I(a)$} (sf); | |
| 29 | +\draw[arr] (ctrl) -- node[above] {$a$} (ed); | |
| 30 | +\draw[arr] (sf) -- (ctrl); | |
| 31 | +\draw[arr] (sf) -- (ro); | |
| 32 | +\draw[arr] (s0) to[bend left=18] (ro); | |
| 33 | +\draw[arr] (ro) -- (out); | |
| 34 | +\draw[arr] (h) to[bend right=20] (out.west); | |
| 35 | +\draw[arr, gray] (h) to[bend right=30] (bypass.west); | |
| 36 | +\end{tikzpicture} | |
added
paper/main.log
+1558 −0
@@ -0,0 +1,1558 @@ | ||
| 1 | +This is pdfTeX, Version 3.141592653-2.6-1.40.29 (TeX Live 2026) (preloaded format=pdflatex 2026.3.22) 30 OCT 2026 00:24 | |
| 2 | +entering extended mode | |
| 3 | + restricted \write18 enabled. | |
| 4 | + %&-line parsing enabled. | |
| 5 | +**main.tex | |
| 6 | +(./main.tex | |
| 7 | +LaTeX2e <2025-11-01> | |
| 8 | +L3 programming layer <2026-01-19> | |
| 9 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/base/article.cls | |
| 10 | +Document Class: article 2025/01/22 v1.4n Standard LaTeX document class | |
| 11 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/base/size12.clo | |
| 12 | +File: size12.clo 2025/01/22 v1.4n Standard LaTeX file (size option) | |
| 13 | +) | |
| 14 | +\c@part=\count275 | |
| 15 | +\c@section=\count276 | |
| 16 | +\c@subsection=\count277 | |
| 17 | +\c@subsubsection=\count278 | |
| 18 | +\c@paragraph=\count279 | |
| 19 | +\c@subparagraph=\count280 | |
| 20 | +\c@figure=\count281 | |
| 21 | +\c@table=\count282 | |
| 22 | +\abovecaptionskip=\skip49 | |
| 23 | +\belowcaptionskip=\skip50 | |
| 24 | +\bibindent=\dimen148 | |
| 25 | +) | |
| 26 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/base/inputenc.sty | |
| 27 | +Package: inputenc 2024/02/08 v1.3d Input encoding file | |
| 28 | +\inpenc@prehook=\toks17 | |
| 29 | +\inpenc@posthook=\toks18 | |
| 30 | +) | |
| 31 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/base/fontenc.sty | |
| 32 | +Package: fontenc 2025/07/18 v2.1d Standard LaTeX package | |
| 33 | +) | |
| 34 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/babel/babel.sty | |
| 35 | +Package: babel 2026/03/08 v26.4 The multilingual framework for LuaLaTeX, pdfLaT | |
| 36 | +eX and XeLaTeX | |
| 37 | +\babel@savecnt=\count283 | |
| 38 | +LaTeX Encoding Info: Redeclaring text command \ij (encoding OT1) on input li | |
| 39 | +ne 2078. | |
| 40 | +LaTeX Encoding Info: Redeclaring text command \IJ (encoding OT1) on input li | |
| 41 | +ne 2080. | |
| 42 | +LaTeX Encoding Info: Redeclaring text command \ij (encoding T1) on input lin | |
| 43 | +e 2082. | |
| 44 | +LaTeX Encoding Info: Redeclaring text command \IJ (encoding T1) on input lin | |
| 45 | +e 2083. | |
| 46 | +LaTeX Encoding Info: Ignoring declaration for text command \ij (encoding ?) | |
| 47 | +on input line 2084. | |
| 48 | +LaTeX Encoding Info: Ignoring declaration for text command \IJ (encoding ?) | |
| 49 | +on input line 2086. | |
| 50 | +LaTeX Encoding Info: Ignoring declaration for text command \SS (encoding ?) | |
| 51 | +on input line 2111. | |
| 52 | +\U@D=\dimen149 | |
| 53 | +\l@unhyphenated=\language91 | |
| 54 | + | |
| 55 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/babel/txtbabel.def) | |
| 56 | +\bbl@readstream=\read2 | |
| 57 | +\bbl@dirlevel=\count284 | |
| 58 | + | |
| 59 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/babel/locale/en/babel-english.t | |
| 60 | +ex) | |
| 61 | +Package babel Info: Importing font and identification data for english | |
| 62 | +(babel) from babel-en.ini. Reported on input line 4330. | |
| 63 | + (/usr/local/texlive/2026/texmf-dist/tex/generic/babel-english/english.ldf | |
| 64 | +Language: english 2017/06/06 v3.3r English support from the babel system | |
| 65 | +Package babel Info: Hyphen rules for 'canadian' set to \l@english | |
| 66 | +(babel) (\language0). Reported on input line 102. | |
| 67 | +Package babel Info: Hyphen rules for 'australian' set to \l@ukenglish | |
| 68 | +(babel) (\language22). Reported on input line 105. | |
| 69 | +Package babel Info: Hyphen rules for 'newzealand' set to \l@ukenglish | |
| 70 | +(babel) (\language22). Reported on input line 108. | |
| 71 | +)) (/usr/local/texlive/2026/texmf-dist/tex/latex/geometry/geometry.sty | |
| 72 | +Package: geometry 2026/03/07 v6.0 Page Geometry | |
| 73 | + | |
| 74 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/keyval.sty | |
| 75 | +Package: keyval 2022/05/29 v1.15 key=value parser (DPC) | |
| 76 | +\KV@toks@=\toks19 | |
| 77 | +) | |
| 78 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/iftex/ifvtex.sty | |
| 79 | +Package: ifvtex 2019/10/25 v1.7 ifvtex legacy package. Use iftex instead. | |
| 80 | + | |
| 81 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/iftex/iftex.sty | |
| 82 | +Package: iftex 2024/12/12 v1.0g TeX engine tests | |
| 83 | +)) | |
| 84 | +\Gm@cnth=\count285 | |
| 85 | +\Gm@cntv=\count286 | |
| 86 | +\c@Gm@tempcnt=\count287 | |
| 87 | +\Gm@bindingoffset=\dimen150 | |
| 88 | +\Gm@wd@mp=\dimen151 | |
| 89 | +\Gm@odd@mp=\dimen152 | |
| 90 | +\Gm@even@mp=\dimen153 | |
| 91 | +\Gm@layoutwidth=\dimen154 | |
| 92 | +\Gm@layoutheight=\dimen155 | |
| 93 | +\Gm@layouthoffset=\dimen156 | |
| 94 | +\Gm@layoutvoffset=\dimen157 | |
| 95 | +\Gm@dimlist=\toks20 | |
| 96 | +) | |
| 97 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/mathptmx.sty | |
| 98 | +Package: mathptmx 2020/03/25 PSNFSS-v9.3 Times w/ Math, improved (SPQR, WaS) | |
| 99 | +LaTeX Font Info: Redeclaring symbol font `operators' on input line 28. | |
| 100 | +LaTeX Font Info: Overwriting symbol font `operators' in version `normal' | |
| 101 | +(Font) OT1/cmr/m/n --> OT1/ztmcm/m/n on input line 28. | |
| 102 | +LaTeX Font Info: Overwriting symbol font `operators' in version `bold' | |
| 103 | +(Font) OT1/cmr/bx/n --> OT1/ztmcm/m/n on input line 28. | |
| 104 | +LaTeX Font Info: Redeclaring symbol font `letters' on input line 29. | |
| 105 | +LaTeX Font Info: Overwriting symbol font `letters' in version `normal' | |
| 106 | +(Font) OML/cmm/m/it --> OML/ztmcm/m/it on input line 29. | |
| 107 | +LaTeX Font Info: Overwriting symbol font `letters' in version `bold' | |
| 108 | +(Font) OML/cmm/b/it --> OML/ztmcm/m/it on input line 29. | |
| 109 | +LaTeX Font Info: Redeclaring symbol font `symbols' on input line 30. | |
| 110 | +LaTeX Font Info: Overwriting symbol font `symbols' in version `normal' | |
| 111 | +(Font) OMS/cmsy/m/n --> OMS/ztmcm/m/n on input line 30. | |
| 112 | +LaTeX Font Info: Overwriting symbol font `symbols' in version `bold' | |
| 113 | +(Font) OMS/cmsy/b/n --> OMS/ztmcm/m/n on input line 30. | |
| 114 | +LaTeX Font Info: Redeclaring symbol font `largesymbols' on input line 31. | |
| 115 | +LaTeX Font Info: Overwriting symbol font `largesymbols' in version `normal' | |
| 116 | +(Font) OMX/cmex/m/n --> OMX/ztmcm/m/n on input line 31. | |
| 117 | +LaTeX Font Info: Overwriting symbol font `largesymbols' in version `bold' | |
| 118 | +(Font) OMX/cmex/m/n --> OMX/ztmcm/m/n on input line 31. | |
| 119 | +\symbold=\mathgroup4 | |
| 120 | +\symitalic=\mathgroup5 | |
| 121 | +LaTeX Font Info: Redeclaring math alphabet \mathbf on input line 34. | |
| 122 | +LaTeX Font Info: Overwriting math alphabet `\mathbf' in version `normal' | |
| 123 | +(Font) OT1/cmr/bx/n --> OT1/ptm/bx/n on input line 34. | |
| 124 | +LaTeX Font Info: Overwriting math alphabet `\mathbf' in version `bold' | |
| 125 | +(Font) OT1/cmr/bx/n --> OT1/ptm/bx/n on input line 34. | |
| 126 | +LaTeX Font Info: Redeclaring math alphabet \mathit on input line 35. | |
| 127 | +LaTeX Font Info: Overwriting math alphabet `\mathit' in version `normal' | |
| 128 | +(Font) OT1/cmr/m/it --> OT1/ptm/m/it on input line 35. | |
| 129 | +LaTeX Font Info: Overwriting math alphabet `\mathit' in version `bold' | |
| 130 | +(Font) OT1/cmr/bx/it --> OT1/ptm/m/it on input line 35. | |
| 131 | +LaTeX Info: Redefining \hbar on input line 50. | |
| 132 | +) | |
| 133 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype.sty | |
| 134 | +Package: microtype 2026/03/01 v3.2d Micro-typographical refinements (RS) | |
| 135 | + | |
| 136 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/etoolbox/etoolbox.sty | |
| 137 | +Package: etoolbox 2025/10/02 v2.5m e-TeX tools for LaTeX (JAW) | |
| 138 | +\etb@tempcnta=\count288 | |
| 139 | +) | |
| 140 | +\MT@toks=\toks21 | |
| 141 | +\MT@tempbox=\box53 | |
| 142 | +\MT@count=\count289 | |
| 143 | +LaTeX Info: Redefining \noprotrusionifhmode on input line 1084. | |
| 144 | +LaTeX Info: Redefining \leftprotrusion on input line 1085. | |
| 145 | +\MT@prot@toks=\toks22 | |
| 146 | +LaTeX Info: Redefining \rightprotrusion on input line 1104. | |
| 147 | +LaTeX Info: Redefining \textls on input line 1449. | |
| 148 | +\MT@outer@kern=\dimen158 | |
| 149 | +LaTeX Info: Redefining \microtypecontext on input line 2053. | |
| 150 | +LaTeX Info: Redefining \textmicrotypecontext on input line 2070. | |
| 151 | +\MT@listname@count=\count290 | |
| 152 | + | |
| 153 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype-pdftex.def | |
| 154 | +File: microtype-pdftex.def 2026/03/01 v3.2d Definitions specific to pdftex (RS) | |
| 155 | + | |
| 156 | +LaTeX Info: Redefining \lsstyle on input line 944. | |
| 157 | +LaTeX Info: Redefining \lslig on input line 944. | |
| 158 | +\MT@outer@space=\skip51 | |
| 159 | +) | |
| 160 | +Package microtype Info: Loading configuration file microtype.cfg. | |
| 161 | + | |
| 162 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/microtype.cfg | |
| 163 | +File: microtype.cfg 2026/03/01 v3.2d microtype main configuration file (RS) | |
| 164 | +) | |
| 165 | +LaTeX Info: Redefining \microtypesetup on input line 3065. | |
| 166 | +) | |
| 167 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/setspace/setspace.sty | |
| 168 | +Package: setspace 2022/12/04 v6.7b set line spacing | |
| 169 | +) | |
| 170 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsmath.sty | |
| 171 | +Package: amsmath 2025/07/09 v2.17z AMS math features | |
| 172 | +\@mathmargin=\skip52 | |
| 173 | + | |
| 174 | +For additional information on amsmath, use the `?' option. | |
| 175 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amstext.sty | |
| 176 | +Package: amstext 2024/11/17 v2.01 AMS text | |
| 177 | + | |
| 178 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsgen.sty | |
| 179 | +File: amsgen.sty 1999/11/30 v2.0 generic functions | |
| 180 | +\@emptytoks=\toks23 | |
| 181 | +\ex@=\dimen159 | |
| 182 | +)) | |
| 183 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsbsy.sty | |
| 184 | +Package: amsbsy 1999/11/29 v1.2d Bold Symbols | |
| 185 | +\pmbraise@=\dimen160 | |
| 186 | +) | |
| 187 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amsmath/amsopn.sty | |
| 188 | +Package: amsopn 2022/04/08 v2.04 operator names | |
| 189 | +) | |
| 190 | +\inf@bad=\count291 | |
| 191 | +LaTeX Info: Redefining \frac on input line 233. | |
| 192 | +\uproot@=\count292 | |
| 193 | +\leftroot@=\count293 | |
| 194 | +LaTeX Info: Redefining \overline on input line 398. | |
| 195 | +LaTeX Info: Redefining \colon on input line 409. | |
| 196 | +\classnum@=\count294 | |
| 197 | +\DOTSCASE@=\count295 | |
| 198 | +LaTeX Info: Redefining \ldots on input line 495. | |
| 199 | +LaTeX Info: Redefining \dots on input line 498. | |
| 200 | +LaTeX Info: Redefining \cdots on input line 619. | |
| 201 | +\Mathstrutbox@=\box54 | |
| 202 | +\strutbox@=\box55 | |
| 203 | +LaTeX Info: Redefining \big on input line 721. | |
| 204 | +LaTeX Info: Redefining \Big on input line 722. | |
| 205 | +LaTeX Info: Redefining \bigg on input line 723. | |
| 206 | +LaTeX Info: Redefining \Bigg on input line 724. | |
| 207 | +\big@size=\dimen161 | |
| 208 | +LaTeX Font Info: Redeclaring font encoding OML on input line 742. | |
| 209 | +LaTeX Font Info: Redeclaring font encoding OMS on input line 743. | |
| 210 | +\macc@depth=\count296 | |
| 211 | +LaTeX Info: Redefining \bmod on input line 904. | |
| 212 | +LaTeX Info: Redefining \pmod on input line 909. | |
| 213 | +LaTeX Info: Redefining \smash on input line 939. | |
| 214 | +LaTeX Info: Redefining \relbar on input line 969. | |
| 215 | +LaTeX Info: Redefining \Relbar on input line 970. | |
| 216 | +\c@MaxMatrixCols=\count297 | |
| 217 | +\dotsspace@=\muskip17 | |
| 218 | +\c@parentequation=\count298 | |
| 219 | +\dspbrk@lvl=\count299 | |
| 220 | +\tag@help=\toks24 | |
| 221 | +\row@=\count300 | |
| 222 | +\column@=\count301 | |
| 223 | +\maxfields@=\count302 | |
| 224 | +\andhelp@=\toks25 | |
| 225 | +\eqnshift@=\dimen162 | |
| 226 | +\alignsep@=\dimen163 | |
| 227 | +\tagshift@=\dimen164 | |
| 228 | +\tagwidth@=\dimen165 | |
| 229 | +\totwidth@=\dimen166 | |
| 230 | +\lineht@=\dimen167 | |
| 231 | +\@envbody=\toks26 | |
| 232 | +\multlinegap=\skip53 | |
| 233 | +\multlinetaggap=\skip54 | |
| 234 | +\mathdisplay@stack=\toks27 | |
| 235 | +LaTeX Info: Redefining \[ on input line 2950. | |
| 236 | +LaTeX Info: Redefining \] on input line 2951. | |
| 237 | +) | |
| 238 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amsfonts/amssymb.sty | |
| 239 | +Package: amssymb 2013/01/14 v3.01 AMS font symbols | |
| 240 | + | |
| 241 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amsfonts/amsfonts.sty | |
| 242 | +Package: amsfonts 2013/01/14 v3.01 Basic AMSFonts support | |
| 243 | +\symAMSa=\mathgroup6 | |
| 244 | +\symAMSb=\mathgroup7 | |
| 245 | +LaTeX Font Info: Overwriting math alphabet `\mathfrak' in version `bold' | |
| 246 | +(Font) U/euf/m/n --> U/euf/b/n on input line 106. | |
| 247 | +)) | |
| 248 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/amscls/amsthm.sty | |
| 249 | +Package: amsthm 2020/05/29 v2.20.6 | |
| 250 | +\thm@style=\toks28 | |
| 251 | +\thm@bodyfont=\toks29 | |
| 252 | +\thm@headfont=\toks30 | |
| 253 | +\thm@notefont=\toks31 | |
| 254 | +\thm@headpunct=\toks32 | |
| 255 | +\thm@preskip=\skip55 | |
| 256 | +\thm@postskip=\skip56 | |
| 257 | +\thm@headsep=\skip57 | |
| 258 | +\dth@everypar=\toks33 | |
| 259 | +) | |
| 260 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/tools/bm.sty | |
| 261 | +Package: bm 2025/10/21 v1.2g Bold Symbol Support (DPC/FMi) | |
| 262 | +Package bm Info: No bold for \OT1/ztmcm/m/n, using \pmb. | |
| 263 | +Package bm Info: No bold for \OML/ztmcm/m/it, using \pmb. | |
| 264 | +Package bm Info: No bold for \OMS/ztmcm/m/n, using \pmb. | |
| 265 | +Package bm Info: No bold for \OMX/ztmcm/m/n, using \pmb. | |
| 266 | +Package bm Info: No bold for \OT1/ptm/bx/n, using \pmb. | |
| 267 | +Package bm Info: No bold for \OT1/ptm/m/it, using \pmb. | |
| 268 | +Package bm Info: No bold for \U/msa/m/n, using \pmb. | |
| 269 | +Package bm Info: No bold for \U/msb/m/n, using \pmb. | |
| 270 | +) | |
| 271 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/booktabs/booktabs.sty | |
| 272 | +Package: booktabs 2020/01/12 v1.61803398 Publication quality tables | |
| 273 | +\heavyrulewidth=\dimen168 | |
| 274 | +\lightrulewidth=\dimen169 | |
| 275 | +\cmidrulewidth=\dimen170 | |
| 276 | +\belowrulesep=\dimen171 | |
| 277 | +\belowbottomsep=\dimen172 | |
| 278 | +\aboverulesep=\dimen173 | |
| 279 | +\abovetopsep=\dimen174 | |
| 280 | +\cmidrulesep=\dimen175 | |
| 281 | +\cmidrulekern=\dimen176 | |
| 282 | +\defaultaddspace=\dimen177 | |
| 283 | +\@cmidla=\count303 | |
| 284 | +\@cmidlb=\count304 | |
| 285 | +\@aboverulesep=\dimen178 | |
| 286 | +\@belowrulesep=\dimen179 | |
| 287 | +\@thisruleclass=\count305 | |
| 288 | +\@lastruleclass=\count306 | |
| 289 | +\@thisrulewidth=\dimen180 | |
| 290 | +) | |
| 291 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/tools/tabularx.sty | |
| 292 | +Package: tabularx 2023/12/11 v2.12a `tabularx' package (DPC) | |
| 293 | + | |
| 294 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/tools/array.sty | |
| 295 | +Package: array 2025/09/25 v2.6n Tabular extension package (FMi) | |
| 296 | +\col@sep=\dimen181 | |
| 297 | +\ar@mcellbox=\box56 | |
| 298 | +\extrarowheight=\dimen182 | |
| 299 | +\NC@list=\toks34 | |
| 300 | +\extratabsurround=\skip58 | |
| 301 | +\backup@length=\skip59 | |
| 302 | +\ar@cellbox=\box57 | |
| 303 | +) | |
| 304 | +\TX@col@width=\dimen183 | |
| 305 | +\TX@old@table=\dimen184 | |
| 306 | +\TX@old@col=\dimen185 | |
| 307 | +\TX@target=\dimen186 | |
| 308 | +\TX@delta=\dimen187 | |
| 309 | +\TX@cols=\count307 | |
| 310 | +\TX@ftn=\toks35 | |
| 311 | +) | |
| 312 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/multirow/multirow.sty | |
| 313 | +Package: multirow 2024/11/12 v2.9 Span multiple rows of a table | |
| 314 | +\multirow@colwidth=\skip60 | |
| 315 | +\multirow@cntb=\count308 | |
| 316 | +\multirow@dima=\skip61 | |
| 317 | +\bigstrutjot=\dimen188 | |
| 318 | +) | |
| 319 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/threeparttable/threeparttable.sty | |
| 320 | +Package: threeparttable 2003/06/13 v 3.0 | |
| 321 | +\@tempboxb=\box58 | |
| 322 | +) (/usr/local/texlive/2026/texmf-dist/tex/latex/tools/longtable.sty | |
| 323 | +Package: longtable 2025-10-13 v4.24 Multi-page Table package (DPC) | |
| 324 | +\LTleft=\skip62 | |
| 325 | +\LTright=\skip63 | |
| 326 | +\LTpre=\skip64 | |
| 327 | +\LTpost=\skip65 | |
| 328 | +\LTchunksize=\count309 | |
| 329 | +\LTcapwidth=\dimen189 | |
| 330 | +\LT@head=\box59 | |
| 331 | +\LT@firsthead=\box60 | |
| 332 | +\LT@foot=\box61 | |
| 333 | +\LT@lastfoot=\box62 | |
| 334 | +\LT@gbox=\box63 | |
| 335 | +\LT@cols=\count310 | |
| 336 | +\LT@rows=\count311 | |
| 337 | +\c@LT@tables=\count312 | |
| 338 | +\c@LT@chunks=\count313 | |
| 339 | +\LT@p@ftn=\toks36 | |
| 340 | +) | |
| 341 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/tools/calc.sty | |
| 342 | +Package: calc 2025/03/01 v4.3b Infix arithmetic (KKT,FJ) | |
| 343 | +\calc@Acount=\count314 | |
| 344 | +\calc@Bcount=\count315 | |
| 345 | +\calc@Adimen=\dimen190 | |
| 346 | +\calc@Bdimen=\dimen191 | |
| 347 | +\calc@Askip=\skip66 | |
| 348 | +\calc@Bskip=\skip67 | |
| 349 | +LaTeX Info: Redefining \setlength on input line 86. | |
| 350 | +LaTeX Info: Redefining \addtolength on input line 87. | |
| 351 | +\calc@Ccount=\count316 | |
| 352 | +\calc@Cskip=\skip68 | |
| 353 | +) | |
| 354 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/siunitx/siunitx.sty | |
| 355 | +Package: siunitx 2025-07-09 v3.4.14 A comprehensive (SI) units package | |
| 356 | +\l__siunitx_number_uncert_offset_int=\count317 | |
| 357 | +\l__siunitx_number_exponent_fixed_int=\count318 | |
| 358 | +\l__siunitx_number_min_decimal_int=\count319 | |
| 359 | +\l__siunitx_number_min_integer_int=\count320 | |
| 360 | +\l__siunitx_number_round_precision_int=\count321 | |
| 361 | +\l__siunitx_number_lower_threshold_int=\count322 | |
| 362 | +\l__siunitx_number_upper_threshold_int=\count323 | |
| 363 | +\l__siunitx_number_group_first_int=\count324 | |
| 364 | +\l__siunitx_number_group_size_int=\count325 | |
| 365 | +\l__siunitx_number_group_minimum_int=\count326 | |
| 366 | +\l__siunitx_angle_tmp_dim=\dimen192 | |
| 367 | +\l__siunitx_angle_marker_box=\box64 | |
| 368 | +\l__siunitx_angle_unit_box=\box65 | |
| 369 | +\l__siunitx_compound_count_int=\count327 | |
| 370 | + | |
| 371 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/translations/translations.sty | |
| 372 | +Package: translations 2022/02/05 v1.12 internationalization of LaTeX2e packages | |
| 373 | + (CN) | |
| 374 | + | |
| 375 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pdftexcmds/pdftexcmds.sty | |
| 376 | +Package: pdftexcmds 2020-06-27 v0.33 Utility functions of pdfTeX for LuaTeX (HO | |
| 377 | +) | |
| 378 | + | |
| 379 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/infwarerr/infwarerr.sty | |
| 380 | +Package: infwarerr 2019/12/03 v1.5 Providing info/warning/error messages (HO) | |
| 381 | +) | |
| 382 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/ltxcmds/ltxcmds.sty | |
| 383 | +Package: ltxcmds 2023-12-04 v1.26 LaTeX kernel commands for general use (HO) | |
| 384 | +) | |
| 385 | +Package pdftexcmds Info: \pdf@primitive is available. | |
| 386 | +Package pdftexcmds Info: \pdf@ifprimitive is available. | |
| 387 | +Package pdftexcmds Info: \pdfdraftmode found. | |
| 388 | +)) | |
| 389 | +\l__siunitx_table_tmp_box=\box66 | |
| 390 | +\l__siunitx_table_tmp_dim=\dimen193 | |
| 391 | +\l__siunitx_table_column_width_dim=\dimen194 | |
| 392 | +\l__siunitx_table_integer_box=\box67 | |
| 393 | +\l__siunitx_table_decimal_box=\box68 | |
| 394 | +\l__siunitx_table_uncert_box=\box69 | |
| 395 | +\l__siunitx_table_before_box=\box70 | |
| 396 | +\l__siunitx_table_after_box=\box71 | |
| 397 | +\l__siunitx_table_before_dim=\dimen195 | |
| 398 | +\l__siunitx_table_carry_dim=\dimen196 | |
| 399 | +\l__siunitx_unit_tmp_int=\count328 | |
| 400 | +\l__siunitx_unit_position_int=\count329 | |
| 401 | +\l__siunitx_unit_total_int=\count330 | |
| 402 | +) | |
| 403 | + | |
| 404 | +Package siunitx Info: Option "input-symbols" has been deprecated in this | |
| 405 | +(siunitx) release. | |
| 406 | +(siunitx) | |
| 407 | +(siunitx) Use "input-digits" as a replacement. | |
| 408 | + | |
| 409 | + | |
| 410 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/graphicx.sty | |
| 411 | +Package: graphicx 2024/12/31 v1.2e Enhanced LaTeX Graphics (DPC,SPQR) | |
| 412 | + | |
| 413 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/graphics.sty | |
| 414 | +Package: graphics 2024/08/06 v1.4g Standard LaTeX Graphics (DPC,SPQR) | |
| 415 | + | |
| 416 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/trig.sty | |
| 417 | +Package: trig 2023/12/02 v1.11 sin cos tan (DPC) | |
| 418 | +) | |
| 419 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics-cfg/graphics.cfg | |
| 420 | +File: graphics.cfg 2016/06/04 v1.11 sample graphics configuration | |
| 421 | +) | |
| 422 | +Package graphics Info: Driver file: pdftex.def on input line 106. | |
| 423 | + | |
| 424 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics-def/pdftex.def | |
| 425 | +File: pdftex.def 2025/09/29 v1.2d Graphics/color driver for pdftex | |
| 426 | +)) | |
| 427 | +\Gin@req@height=\dimen197 | |
| 428 | +\Gin@req@width=\dimen198 | |
| 429 | +) | |
| 430 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/caption/subcaption.sty | |
| 431 | +Package: subcaption 2023/07/28 v1.6b Sub-captions (AR) | |
| 432 | + | |
| 433 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/caption/caption.sty | |
| 434 | +Package: caption 2023/08/05 v3.6o Customizing captions (AR) | |
| 435 | + | |
| 436 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/caption/caption3.sty | |
| 437 | +Package: caption3 2023/07/31 v2.4d caption3 kernel (AR) | |
| 438 | +\caption@tempdima=\dimen199 | |
| 439 | +\captionmargin=\dimen256 | |
| 440 | +\caption@leftmargin=\dimen257 | |
| 441 | +\caption@rightmargin=\dimen258 | |
| 442 | +\caption@width=\dimen259 | |
| 443 | +\caption@indent=\dimen260 | |
| 444 | +\caption@parindent=\dimen261 | |
| 445 | +\caption@hangindent=\dimen262 | |
| 446 | +Package caption Info: Standard document class detected. | |
| 447 | +) | |
| 448 | +\c@caption@flags=\count331 | |
| 449 | +\c@continuedfloat=\count332 | |
| 450 | +Package caption Info: longtable package is loaded. | |
| 451 | + | |
| 452 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/caption/ltcaption.sty | |
| 453 | +Package: ltcaption 2021/01/08 v1.4c longtable captions (AR) | |
| 454 | +) | |
| 455 | +Package caption Info: threeparttable package is loaded. | |
| 456 | +) | |
| 457 | +Package caption Info: New subtype `subfigure' on input line 238. | |
| 458 | +\c@subfigure=\count333 | |
| 459 | +Package caption Info: New subtype `subtable' on input line 238. | |
| 460 | +\c@subtable=\count334 | |
| 461 | +) | |
| 462 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/float/float.sty | |
| 463 | +Package: float 2001/11/08 v1.3d Float enhancements (AL) | |
| 464 | +\c@float@type=\count335 | |
| 465 | +\float@exts=\toks37 | |
| 466 | +\float@box=\box72 | |
| 467 | +\@float@everytoks=\toks38 | |
| 468 | +\@floatcapt=\box73 | |
| 469 | +) | |
| 470 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/pgf/frontendlayer/tikz.sty | |
| 471 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/pgf/basiclayer/pgf.sty | |
| 472 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/pgf/utilities/pgfrcs.sty | |
| 473 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/utilities/pgfutil-common.te | |
| 474 | +x | |
| 475 | +\pgfutil@everybye=\toks39 | |
| 476 | +\pgfutil@tempdima=\dimen263 | |
| 477 | +\pgfutil@tempdimb=\dimen264 | |
| 478 | +) | |
| 479 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/utilities/pgfutil-latex.def | |
| 480 | +\pgfutil@abb=\box74 | |
| 481 | +) (/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/utilities/pgfrcs.code.tex | |
| 482 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/pgf.revision.tex) | |
| 483 | +Package: pgfrcs 2025-08-29 v3.1.11a (3.1.11a) | |
| 484 | +)) | |
| 485 | +Package: pgf 2025-08-29 v3.1.11a (3.1.11a) | |
| 486 | + | |
| 487 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/pgf/basiclayer/pgfcore.sty | |
| 488 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/pgf/systemlayer/pgfsys.sty | |
| 489 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/systemlayer/pgfsys.code.tex | |
| 490 | +Package: pgfsys 2025-08-29 v3.1.11a (3.1.11a) | |
| 491 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/utilities/pgfkeys.code.tex | |
| 492 | +\pgfkeys@pathtoks=\toks40 | |
| 493 | +\pgfkeys@temptoks=\toks41 | |
| 494 | + | |
| 495 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/utilities/pgfkeyslibraryfil | |
| 496 | +tered.code.tex | |
| 497 | +\pgfkeys@tmptoks=\toks42 | |
| 498 | +)) | |
| 499 | +\pgf@x=\dimen265 | |
| 500 | +\pgf@y=\dimen266 | |
| 501 | +\pgf@xa=\dimen267 | |
| 502 | +\pgf@ya=\dimen268 | |
| 503 | +\pgf@xb=\dimen269 | |
| 504 | +\pgf@yb=\dimen270 | |
| 505 | +\pgf@xc=\dimen271 | |
| 506 | +\pgf@yc=\dimen272 | |
| 507 | +\pgf@xd=\dimen273 | |
| 508 | +\pgf@yd=\dimen274 | |
| 509 | +\w@pgf@writea=\write3 | |
| 510 | +\r@pgf@reada=\read3 | |
| 511 | +\c@pgf@counta=\count336 | |
| 512 | +\c@pgf@countb=\count337 | |
| 513 | +\c@pgf@countc=\count338 | |
| 514 | +\c@pgf@countd=\count339 | |
| 515 | +\t@pgf@toka=\toks43 | |
| 516 | +\t@pgf@tokb=\toks44 | |
| 517 | +\t@pgf@tokc=\toks45 | |
| 518 | +\pgf@sys@id@count=\count340 | |
| 519 | + | |
| 520 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/systemlayer/pgf.cfg | |
| 521 | +File: pgf.cfg 2025-08-29 v3.1.11a (3.1.11a) | |
| 522 | +) | |
| 523 | +Driver file for pgf: pgfsys-pdftex.def | |
| 524 | + | |
| 525 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/systemlayer/pgfsys-pdftex.d | |
| 526 | +ef | |
| 527 | +File: pgfsys-pdftex.def 2025-08-29 v3.1.11a (3.1.11a) | |
| 528 | + | |
| 529 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/systemlayer/pgfsys-common-p | |
| 530 | +df.def | |
| 531 | +File: pgfsys-common-pdf.def 2025-08-29 v3.1.11a (3.1.11a) | |
| 532 | +))) | |
| 533 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/systemlayer/pgfsyssoftpath. | |
| 534 | +code.tex | |
| 535 | +File: pgfsyssoftpath.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 536 | +\pgfsyssoftpath@smallbuffer@items=\count341 | |
| 537 | +\pgfsyssoftpath@bigbuffer@items=\count342 | |
| 538 | +) | |
| 539 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/systemlayer/pgfsysprotocol. | |
| 540 | +code.tex | |
| 541 | +File: pgfsysprotocol.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 542 | +)) (/usr/local/texlive/2026/texmf-dist/tex/latex/xcolor/xcolor.sty | |
| 543 | +Package: xcolor 2024/09/29 v3.02 LaTeX color extensions (UK) | |
| 544 | + | |
| 545 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics-cfg/color.cfg | |
| 546 | +File: color.cfg 2016/01/02 v1.6 sample color configuration | |
| 547 | +) | |
| 548 | +Package xcolor Info: Driver file: pdftex.def on input line 274. | |
| 549 | + | |
| 550 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/mathcolor.ltx) | |
| 551 | +Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 1349. | |
| 552 | +Package xcolor Info: Model `hsb' substituted by `rgb' on input line 1353. | |
| 553 | +Package xcolor Info: Model `RGB' extended on input line 1365. | |
| 554 | +Package xcolor Info: Model `HTML' substituted by `rgb' on input line 1367. | |
| 555 | +Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 1368. | |
| 556 | +Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 1369. | |
| 557 | +Package xcolor Info: Model `HSB' substituted by `hsb' on input line 1370. | |
| 558 | +Package xcolor Info: Model `Gray' substituted by `gray' on input line 1371. | |
| 559 | +Package xcolor Info: Model `wave' substituted by `hsb' on input line 1372. | |
| 560 | +) | |
| 561 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcore.code.tex | |
| 562 | +Package: pgfcore 2025-08-29 v3.1.11a (3.1.11a) | |
| 563 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmath.code.tex | |
| 564 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmathutil.code.tex) | |
| 565 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmathparser.code.tex | |
| 566 | +\pgfmath@dimen=\dimen275 | |
| 567 | +\pgfmath@count=\count343 | |
| 568 | +\pgfmath@box=\box75 | |
| 569 | +\pgfmath@toks=\toks46 | |
| 570 | +\pgfmath@stack@operand=\toks47 | |
| 571 | +\pgfmath@stack@operation=\toks48 | |
| 572 | +) | |
| 573 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.code. | |
| 574 | +tex) | |
| 575 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.basic | |
| 576 | +.code.tex) | |
| 577 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.trigo | |
| 578 | +nometric.code.tex) | |
| 579 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.rando | |
| 580 | +m.code.tex) | |
| 581 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.compa | |
| 582 | +rison.code.tex) | |
| 583 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.base. | |
| 584 | +code.tex) | |
| 585 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.round | |
| 586 | +.code.tex) | |
| 587 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.misc. | |
| 588 | +code.tex) | |
| 589 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.integ | |
| 590 | +erarithmetics.code.tex) | |
| 591 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmathcalc.code.tex) | |
| 592 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmathfloat.code.tex | |
| 593 | +\c@pgfmathroundto@lastzeros=\count344 | |
| 594 | +)) (/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfint.code.tex) | |
| 595 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepoints.co | |
| 596 | +de.tex | |
| 597 | +File: pgfcorepoints.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 598 | +\pgf@picminx=\dimen276 | |
| 599 | +\pgf@picmaxx=\dimen277 | |
| 600 | +\pgf@picminy=\dimen278 | |
| 601 | +\pgf@picmaxy=\dimen279 | |
| 602 | +\pgf@pathminx=\dimen280 | |
| 603 | +\pgf@pathmaxx=\dimen281 | |
| 604 | +\pgf@pathminy=\dimen282 | |
| 605 | +\pgf@pathmaxy=\dimen283 | |
| 606 | +\pgf@xx=\dimen284 | |
| 607 | +\pgf@xy=\dimen285 | |
| 608 | +\pgf@yx=\dimen286 | |
| 609 | +\pgf@yy=\dimen287 | |
| 610 | +\pgf@zx=\dimen288 | |
| 611 | +\pgf@zy=\dimen289 | |
| 612 | +) | |
| 613 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepathconst | |
| 614 | +ruct.code.tex | |
| 615 | +File: pgfcorepathconstruct.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 616 | +\pgf@path@lastx=\dimen290 | |
| 617 | +\pgf@path@lasty=\dimen291 | |
| 618 | +) | |
| 619 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepathusage | |
| 620 | +.code.tex | |
| 621 | +File: pgfcorepathusage.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 622 | +\pgf@shorten@end@additional=\dimen292 | |
| 623 | +\pgf@shorten@start@additional=\dimen293 | |
| 624 | +) | |
| 625 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcorescopes.co | |
| 626 | +de.tex | |
| 627 | +File: pgfcorescopes.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 628 | +\pgfpic=\box76 | |
| 629 | +\pgf@hbox=\box77 | |
| 630 | +\pgf@layerbox@main=\box78 | |
| 631 | +\pgf@picture@serial@count=\count345 | |
| 632 | +) | |
| 633 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcoregraphicst | |
| 634 | +ate.code.tex | |
| 635 | +File: pgfcoregraphicstate.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 636 | +\pgflinewidth=\dimen294 | |
| 637 | +) | |
| 638 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcoretransform | |
| 639 | +ations.code.tex | |
| 640 | +File: pgfcoretransformations.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 641 | +\pgf@pt@x=\dimen295 | |
| 642 | +\pgf@pt@y=\dimen296 | |
| 643 | +\pgf@pt@temp=\dimen297 | |
| 644 | +) | |
| 645 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcorequick.cod | |
| 646 | +e.tex | |
| 647 | +File: pgfcorequick.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 648 | +) | |
| 649 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcoreobjects.c | |
| 650 | +ode.tex | |
| 651 | +File: pgfcoreobjects.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 652 | +) | |
| 653 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepathproce | |
| 654 | +ssing.code.tex | |
| 655 | +File: pgfcorepathprocessing.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 656 | +) | |
| 657 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcorearrows.co | |
| 658 | +de.tex | |
| 659 | +File: pgfcorearrows.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 660 | +\pgfarrowsep=\dimen298 | |
| 661 | +) | |
| 662 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcoreshade.cod | |
| 663 | +e.tex | |
| 664 | +File: pgfcoreshade.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 665 | +\pgf@max=\dimen299 | |
| 666 | +\pgf@sys@shading@range@num=\count346 | |
| 667 | +\pgf@shadingcount=\count347 | |
| 668 | +) | |
| 669 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcoreimage.cod | |
| 670 | +e.tex | |
| 671 | +File: pgfcoreimage.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 672 | +) | |
| 673 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcoreexternal. | |
| 674 | +code.tex | |
| 675 | +File: pgfcoreexternal.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 676 | +\pgfexternal@startupbox=\box79 | |
| 677 | +) | |
| 678 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcorelayers.co | |
| 679 | +de.tex | |
| 680 | +File: pgfcorelayers.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 681 | +) | |
| 682 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcoretranspare | |
| 683 | +ncy.code.tex | |
| 684 | +File: pgfcoretransparency.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 685 | +) | |
| 686 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepatterns. | |
| 687 | +code.tex | |
| 688 | +File: pgfcorepatterns.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 689 | +) | |
| 690 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/basiclayer/pgfcorerdf.code. | |
| 691 | +tex | |
| 692 | +File: pgfcorerdf.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 693 | +))) | |
| 694 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/modules/pgfmoduleshapes.cod | |
| 695 | +e.tex | |
| 696 | +File: pgfmoduleshapes.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 697 | +\pgfnodeparttextbox=\box80 | |
| 698 | +) | |
| 699 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/modules/pgfmoduleplot.code. | |
| 700 | +tex | |
| 701 | +File: pgfmoduleplot.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 702 | +) | |
| 703 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/pgf/compatibility/pgfcomp-version | |
| 704 | +-0-65.sty | |
| 705 | +Package: pgfcomp-version-0-65 2025-08-29 v3.1.11a (3.1.11a) | |
| 706 | +\pgf@nodesepstart=\dimen300 | |
| 707 | +\pgf@nodesepend=\dimen301 | |
| 708 | +) | |
| 709 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/pgf/compatibility/pgfcomp-version | |
| 710 | +-1-18.sty | |
| 711 | +Package: pgfcomp-version-1-18 2025-08-29 v3.1.11a (3.1.11a) | |
| 712 | +)) | |
| 713 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/pgf/utilities/pgffor.sty | |
| 714 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/pgf/utilities/pgfkeys.sty | |
| 715 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/utilities/pgfkeys.code.tex) | |
| 716 | +) (/usr/local/texlive/2026/texmf-dist/tex/latex/pgf/math/pgfmath.sty | |
| 717 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/math/pgfmath.code.tex)) | |
| 718 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/utilities/pgffor.code.tex | |
| 719 | +Package: pgffor 2025-08-29 v3.1.11a (3.1.11a) | |
| 720 | +\pgffor@iter=\dimen302 | |
| 721 | +\pgffor@skip=\dimen303 | |
| 722 | +\pgffor@stack=\toks49 | |
| 723 | +\pgffor@toks=\toks50 | |
| 724 | +)) | |
| 725 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/frontendlayer/tikz/tikz.cod | |
| 726 | +e.tex | |
| 727 | +Package: tikz 2025-08-29 v3.1.11a (3.1.11a) | |
| 728 | + | |
| 729 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/libraries/pgflibraryplothan | |
| 730 | +dlers.code.tex | |
| 731 | +File: pgflibraryplothandlers.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 732 | +\pgf@plot@mark@count=\count348 | |
| 733 | +\pgfplotmarksize=\dimen304 | |
| 734 | +) | |
| 735 | +\tikz@lastx=\dimen305 | |
| 736 | +\tikz@lasty=\dimen306 | |
| 737 | +\tikz@lastxsaved=\dimen307 | |
| 738 | +\tikz@lastysaved=\dimen308 | |
| 739 | +\tikz@lastmovetox=\dimen309 | |
| 740 | +\tikz@lastmovetoy=\dimen310 | |
| 741 | +\tikzleveldistance=\dimen311 | |
| 742 | +\tikzsiblingdistance=\dimen312 | |
| 743 | +\tikz@figbox=\box81 | |
| 744 | +\tikz@figbox@bg=\box82 | |
| 745 | +\tikz@tempbox=\box83 | |
| 746 | +\tikz@tempbox@bg=\box84 | |
| 747 | +\tikztreelevel=\count349 | |
| 748 | +\tikznumberofchildren=\count350 | |
| 749 | +\tikznumberofcurrentchild=\count351 | |
| 750 | +\tikz@fig@count=\count352 | |
| 751 | + | |
| 752 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/modules/pgfmodulematrix.cod | |
| 753 | +e.tex | |
| 754 | +File: pgfmodulematrix.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 755 | +\pgfmatrixcurrentrow=\count353 | |
| 756 | +\pgfmatrixcurrentcolumn=\count354 | |
| 757 | +\pgf@matrix@numberofcolumns=\count355 | |
| 758 | +) | |
| 759 | +\tikz@expandcount=\count356 | |
| 760 | + | |
| 761 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie | |
| 762 | +s/tikzlibrarytopaths.code.tex | |
| 763 | +File: tikzlibrarytopaths.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 764 | +))) | |
| 765 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/libraries/pgflibraryarrows. | |
| 766 | +meta.code.tex | |
| 767 | +File: pgflibraryarrows.meta.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 768 | +\pgfarrowinset=\dimen313 | |
| 769 | +\pgfarrowlength=\dimen314 | |
| 770 | +\pgfarrowwidth=\dimen315 | |
| 771 | +\pgfarrowlinewidth=\dimen316 | |
| 772 | +) | |
| 773 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie | |
| 774 | +s/tikzlibrarypositioning.code.tex | |
| 775 | +File: tikzlibrarypositioning.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 776 | +) | |
| 777 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie | |
| 778 | +s/tikzlibrarycalc.code.tex | |
| 779 | +File: tikzlibrarycalc.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 780 | +) | |
| 781 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie | |
| 782 | +s/tikzlibraryfit.code.tex | |
| 783 | +File: tikzlibraryfit.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 784 | +) | |
| 785 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie | |
| 786 | +s/tikzlibrarybackgrounds.code.tex | |
| 787 | +File: tikzlibrarybackgrounds.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 788 | +\pgf@layerbox@background=\box85 | |
| 789 | +\pgf@layerboxsaved@background=\box86 | |
| 790 | +) | |
| 791 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie | |
| 792 | +s/tikzlibraryshapes.misc.code.tex | |
| 793 | +File: tikzlibraryshapes.misc.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 794 | + | |
| 795 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pgf/libraries/shapes/pgflibrary | |
| 796 | +shapes.misc.code.tex | |
| 797 | +File: pgflibraryshapes.misc.code.tex 2025-08-29 v3.1.11a (3.1.11a) | |
| 798 | +)) | |
| 799 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/graphics/dvipsnam.def | |
| 800 | +File: dvipsnam.def 2016/06/17 v3.0m Driver-dependent file (DPC,SPQR) | |
| 801 | +) | |
| 802 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/hyperref.sty | |
| 803 | +Package: hyperref 2026-01-29 v7.01p Hypertext links for LaTeX | |
| 804 | + | |
| 805 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/kvsetkeys/kvsetkeys.sty | |
| 806 | +Package: kvsetkeys 2022-10-05 v1.19 Key value parser (HO) | |
| 807 | +) | |
| 808 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/kvdefinekeys/kvdefinekeys.sty | |
| 809 | +Package: kvdefinekeys 2019-12-19 v1.6 Define keys (HO) | |
| 810 | +) | |
| 811 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/pdfescape/pdfescape.sty | |
| 812 | +Package: pdfescape 2019/12/09 v1.15 Implements pdfTeX's escape features (HO) | |
| 813 | +) | |
| 814 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/hycolor/hycolor.sty | |
| 815 | +Package: hycolor 2020-01-27 v1.10 Color options for hyperref/bookmark (HO) | |
| 816 | +) | |
| 817 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/nameref.sty | |
| 818 | +Package: nameref 2026-01-29 v2.58 Cross-referencing by name of section | |
| 819 | + | |
| 820 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/refcount/refcount.sty | |
| 821 | +Package: refcount 2019/12/15 v3.6 Data extraction from label references (HO) | |
| 822 | +) | |
| 823 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/gettitlestring/gettitlestring.s | |
| 824 | +ty | |
| 825 | +Package: gettitlestring 2019/12/15 v1.6 Cleanup title references (HO) | |
| 826 | + (/usr/local/texlive/2026/texmf-dist/tex/latex/kvoptions/kvoptions.sty | |
| 827 | +Package: kvoptions 2022-06-15 v3.15 Key value format for package options (HO) | |
| 828 | +)) | |
| 829 | +\c@section@level=\count357 | |
| 830 | +) | |
| 831 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/stringenc/stringenc.sty | |
| 832 | +Package: stringenc 2019/11/29 v1.12 Convert strings between diff. encodings (HO | |
| 833 | +) | |
| 834 | +) | |
| 835 | +\@linkdim=\dimen317 | |
| 836 | +\Hy@linkcounter=\count358 | |
| 837 | +\Hy@pagecounter=\count359 | |
| 838 | + | |
| 839 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/pd1enc.def | |
| 840 | +File: pd1enc.def 2026-01-29 v7.01p Hyperref: PDFDocEncoding definition (HO) | |
| 841 | +Now handling font encoding PD1 ... | |
| 842 | +... no UTF-8 mapping file for font encoding PD1 | |
| 843 | +) | |
| 844 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/intcalc/intcalc.sty | |
| 845 | +Package: intcalc 2019/12/15 v1.3 Expandable calculations with integers (HO) | |
| 846 | +) | |
| 847 | +\Hy@SavedSpaceFactor=\count360 | |
| 848 | + | |
| 849 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/puenc.def | |
| 850 | +File: puenc.def 2026-01-29 v7.01p Hyperref: PDF Unicode definition (HO) | |
| 851 | +Now handling font encoding PU ... | |
| 852 | +... no UTF-8 mapping file for font encoding PU | |
| 853 | +) | |
| 854 | +Package hyperref Info: Option `colorlinks' set `true' on input line 4072. | |
| 855 | +Package hyperref Info: Option `breaklinks' set `true' on input line 4072. | |
| 856 | +Package hyperref Info: Hyper figures OFF on input line 4201. | |
| 857 | +Package hyperref Info: Link nesting OFF on input line 4206. | |
| 858 | +Package hyperref Info: Hyper index ON on input line 4209. | |
| 859 | +Package hyperref Info: Plain pages OFF on input line 4216. | |
| 860 | +Package hyperref Info: Backreferencing OFF on input line 4221. | |
| 861 | +Package hyperref Info: Implicit mode ON; LaTeX internals redefined. | |
| 862 | +Package hyperref Info: Bookmarks ON on input line 4468. | |
| 863 | +\c@Hy@tempcnt=\count361 | |
| 864 | + | |
| 865 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/url/url.sty | |
| 866 | +\Urlmuskip=\muskip18 | |
| 867 | +Package: url 2013/09/16 ver 3.4 Verb mode for urls, etc. | |
| 868 | +) | |
| 869 | +LaTeX Info: Redefining \url on input line 4807. | |
| 870 | +\XeTeXLinkMargin=\dimen318 | |
| 871 | + | |
| 872 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/bitset/bitset.sty | |
| 873 | +Package: bitset 2019/12/09 v1.3 Handle bit-vector datatype (HO) | |
| 874 | + | |
| 875 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/bigintcalc/bigintcalc.sty | |
| 876 | +Package: bigintcalc 2019/12/15 v1.5 Expandable calculations on big integers (HO | |
| 877 | +) | |
| 878 | +)) | |
| 879 | +\Fld@menulength=\count362 | |
| 880 | +\Field@Width=\dimen319 | |
| 881 | +\Fld@charsize=\dimen320 | |
| 882 | +Package hyperref Info: Hyper figures OFF on input line 6084. | |
| 883 | +Package hyperref Info: Link nesting OFF on input line 6089. | |
| 884 | +Package hyperref Info: Hyper index ON on input line 6092. | |
| 885 | +Package hyperref Info: backreferencing OFF on input line 6099. | |
| 886 | +Package hyperref Info: Link coloring ON on input line 6102. | |
| 887 | +Package hyperref Info: Link coloring with OCG OFF on input line 6109. | |
| 888 | +Package hyperref Info: PDF/A mode OFF on input line 6114. | |
| 889 | +\Hy@abspage=\count363 | |
| 890 | +\c@Item=\count364 | |
| 891 | +\c@Hfootnote=\count365 | |
| 892 | +) | |
| 893 | +Package hyperref Info: Driver (autodetected): hpdftex. | |
| 894 | + | |
| 895 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/hyperref/hpdftex.def | |
| 896 | +File: hpdftex.def 2026-01-29 v7.01p Hyperref driver for pdfTeX | |
| 897 | +\Fld@listcount=\count366 | |
| 898 | +\c@bookmark@seq@number=\count367 | |
| 899 | + | |
| 900 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/rerunfilecheck/rerunfilecheck.sty | |
| 901 | +Package: rerunfilecheck 2025-06-21 v1.11 Rerun checks for auxiliary files (HO) | |
| 902 | + | |
| 903 | +(/usr/local/texlive/2026/texmf-dist/tex/generic/uniquecounter/uniquecounter.sty | |
| 904 | +Package: uniquecounter 2019/12/15 v1.4 Provide unlimited unique counter (HO) | |
| 905 | +) | |
| 906 | +Package uniquecounter Info: New unique counter `rerunfilecheck' on input line 2 | |
| 907 | +84. | |
| 908 | +) | |
| 909 | +\Hy@SectionHShift=\skip69 | |
| 910 | +) (/usr/local/texlive/2026/texmf-dist/tex/latex/natbib/natbib.sty | |
| 911 | +Package: natbib 2010/09/13 8.31b (PWD, AO) | |
| 912 | +\bibhang=\skip70 | |
| 913 | +\bibsep=\skip71 | |
| 914 | +LaTeX Info: Redefining \cite on input line 694. | |
| 915 | +\c@NAT@ctr=\count368 | |
| 916 | +) | |
| 917 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/fancyhdr/fancyhdr.sty | |
| 918 | +Package: fancyhdr 2025/02/07 v5.2 Extensive control of page headers and footers | |
| 919 | + | |
| 920 | +\f@nch@headwidth=\skip72 | |
| 921 | +\f@nch@offset@elh=\skip73 | |
| 922 | +\f@nch@offset@erh=\skip74 | |
| 923 | +\f@nch@offset@olh=\skip75 | |
| 924 | +\f@nch@offset@orh=\skip76 | |
| 925 | +\f@nch@offset@elf=\skip77 | |
| 926 | +\f@nch@offset@erf=\skip78 | |
| 927 | +\f@nch@offset@olf=\skip79 | |
| 928 | +\f@nch@offset@orf=\skip80 | |
| 929 | +\f@nch@height=\skip81 | |
| 930 | +\f@nch@footalignment=\skip82 | |
| 931 | +\f@nch@widthL=\skip83 | |
| 932 | +\f@nch@widthC=\skip84 | |
| 933 | +\f@nch@widthR=\skip85 | |
| 934 | +\@temptokenb=\toks51 | |
| 935 | +) | |
| 936 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/titlesec/titlesec.sty | |
| 937 | +Package: titlesec 2025/01/04 v2.17 Sectioning titles | |
| 938 | +\ttl@box=\box87 | |
| 939 | +\beforetitleunit=\skip86 | |
| 940 | +\aftertitleunit=\skip87 | |
| 941 | +\ttl@plus=\dimen321 | |
| 942 | +\ttl@minus=\dimen322 | |
| 943 | +\ttl@toksa=\toks52 | |
| 944 | +\titlewidth=\dimen323 | |
| 945 | +\titlewidthlast=\dimen324 | |
| 946 | +\titlewidthfirst=\dimen325 | |
| 947 | +) | |
| 948 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/appendix/appendix.sty | |
| 949 | +Package: appendix 2020/02/08 v1.2c extra appendix facilities | |
| 950 | +\c@@pps=\count369 | |
| 951 | +\c@@ppsavesec=\count370 | |
| 952 | +\c@@ppsaveapp=\count371 | |
| 953 | +) | |
| 954 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/enumitem/enumitem.sty | |
| 955 | +Package: enumitem 2025/02/06 v3.11 Customized lists | |
| 956 | +\labelindent=\skip88 | |
| 957 | +\enit@outerparindent=\dimen326 | |
| 958 | +\enit@toks=\toks53 | |
| 959 | +\enit@inbox=\box88 | |
| 960 | +\enit@count@id=\count372 | |
| 961 | +\enitdp@description=\count373 | |
| 962 | +) | |
| 963 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/algorithms/algorithm.sty | |
| 964 | +Package: algorithm 2009/08/24 v0.1 Document Style `algorithm' - floating enviro | |
| 965 | +nment | |
| 966 | + | |
| 967 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/base/ifthen.sty | |
| 968 | +Package: ifthen 2024/03/16 v1.1e Standard LaTeX ifthen package (DPC) | |
| 969 | +) | |
| 970 | +\@float@every@algorithm=\toks54 | |
| 971 | +\c@algorithm=\count374 | |
| 972 | +) | |
| 973 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/algorithmicx/algpseudocode.sty | |
| 974 | +Package: algpseudocode | |
| 975 | + | |
| 976 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/algorithmicx/algorithmicx.sty | |
| 977 | +Package: algorithmicx 2005/04/27 v1.2 Algorithmicx | |
| 978 | + | |
| 979 | +Document Style algorithmicx 1.2 - a greatly improved `algorithmic' style | |
| 980 | +\c@ALG@line=\count375 | |
| 981 | +\c@ALG@rem=\count376 | |
| 982 | +\c@ALG@nested=\count377 | |
| 983 | +\ALG@tlm=\skip89 | |
| 984 | +\ALG@thistlm=\skip90 | |
| 985 | +\c@ALG@Lnr=\count378 | |
| 986 | +\c@ALG@blocknr=\count379 | |
| 987 | +\c@ALG@storecount=\count380 | |
| 988 | +\c@ALG@tmpcounter=\count381 | |
| 989 | +\ALG@tmplength=\skip91 | |
| 990 | +) | |
| 991 | +Document Style - pseudocode environments for use with the `algorithmicx' style | |
| 992 | +) (/usr/local/texlive/2026/texmf-dist/tex/latex/listings/listings.sty | |
| 993 | +\lst@mode=\count382 | |
| 994 | +\lst@gtempboxa=\box89 | |
| 995 | +\lst@token=\toks55 | |
| 996 | +\lst@length=\count383 | |
| 997 | +\lst@currlwidth=\dimen327 | |
| 998 | +\lst@column=\count384 | |
| 999 | +\lst@pos=\count385 | |
| 1000 | +\lst@lostspace=\dimen328 | |
| 1001 | +\lst@width=\dimen329 | |
| 1002 | +\lst@newlines=\count386 | |
| 1003 | +\lst@lineno=\count387 | |
| 1004 | +\lst@maxwidth=\dimen330 | |
| 1005 | + | |
| 1006 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/listings/lstpatch.sty | |
| 1007 | +File: lstpatch.sty 2025/11/14 1.11b (Carsten Heinz) | |
| 1008 | +) | |
| 1009 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/listings/lstmisc.sty | |
| 1010 | +File: lstmisc.sty 2025/11/14 1.11b (Carsten Heinz) | |
| 1011 | +\c@lstnumber=\count388 | |
| 1012 | +\lst@skipnumbers=\count389 | |
| 1013 | +\lst@framebox=\box90 | |
| 1014 | +) | |
| 1015 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/listings/listings.cfg | |
| 1016 | +File: listings.cfg 2025/11/14 1.11b listings configuration | |
| 1017 | +)) | |
| 1018 | +Package: listings 2025/11/14 1.11b (Carsten Heinz) | |
| 1019 | + | |
| 1020 | +==> First Aid for listings.sty no longer applied! | |
| 1021 | + Expected: | |
| 1022 | + 2024/09/23 1.10c (Carsten Heinz) | |
| 1023 | + but found: | |
| 1024 | + 2025/11/14 1.11b (Carsten Heinz) | |
| 1025 | + so I'm assuming it got fixed. | |
| 1026 | +\c@proposition=\count390 | |
| 1027 | +\c@lemma=\count391 | |
| 1028 | +\c@corollary=\count392 | |
| 1029 | +\c@definition=\count393 | |
| 1030 | +\c@assumption=\count394 | |
| 1031 | +\c@remark=\count395 | |
| 1032 | +(./tables/numbers.tex) | |
| 1033 | +LaTeX Font Info: Trying to load font information for T1+ptm on input line 85 | |
| 1034 | +. | |
| 1035 | + | |
| 1036 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/t1ptm.fd | |
| 1037 | +File: t1ptm.fd 2001/06/04 font definitions for T1/ptm. | |
| 1038 | +) | |
| 1039 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/l3backend/l3backend-pdftex.def | |
| 1040 | +File: l3backend-pdftex.def 2025-10-09 L3 backend support: PDF output (pdfTeX) | |
| 1041 | +\l__color_backend_stack_int=\count396 | |
| 1042 | +) | |
| 1043 | +(./main.aux) | |
| 1044 | +\openout1 = `main.aux'. | |
| 1045 | + | |
| 1046 | +LaTeX Font Info: Checking defaults for OML/cmm/m/it on input line 85. | |
| 1047 | +LaTeX Font Info: ... okay on input line 85. | |
| 1048 | +LaTeX Font Info: Checking defaults for OMS/cmsy/m/n on input line 85. | |
| 1049 | +LaTeX Font Info: ... okay on input line 85. | |
| 1050 | +LaTeX Font Info: Checking defaults for OT1/cmr/m/n on input line 85. | |
| 1051 | +LaTeX Font Info: ... okay on input line 85. | |
| 1052 | +LaTeX Font Info: Checking defaults for T1/cmr/m/n on input line 85. | |
| 1053 | +LaTeX Font Info: ... okay on input line 85. | |
| 1054 | +LaTeX Font Info: Checking defaults for TS1/cmr/m/n on input line 85. | |
| 1055 | +LaTeX Font Info: ... okay on input line 85. | |
| 1056 | +LaTeX Font Info: Checking defaults for OMX/cmex/m/n on input line 85. | |
| 1057 | +LaTeX Font Info: ... okay on input line 85. | |
| 1058 | +LaTeX Font Info: Checking defaults for U/cmr/m/n on input line 85. | |
| 1059 | +LaTeX Font Info: ... okay on input line 85. | |
| 1060 | +LaTeX Font Info: Checking defaults for PD1/pdf/m/n on input line 85. | |
| 1061 | +LaTeX Font Info: ... okay on input line 85. | |
| 1062 | +LaTeX Font Info: Checking defaults for PU/pdf/m/n on input line 85. | |
| 1063 | +LaTeX Font Info: ... okay on input line 85. | |
| 1064 | + | |
| 1065 | +*geometry* driver: auto-detecting | |
| 1066 | +*geometry* detected driver: pdftex | |
| 1067 | +*geometry* verbose mode - [ preamble ] result: | |
| 1068 | +* driver: pdftex | |
| 1069 | +* paper: letterpaper | |
| 1070 | +* layout: <same size as paper> | |
| 1071 | +* layoutoffset:(h,v)=(0.0pt,0.0pt) | |
| 1072 | +* modes: | |
| 1073 | +* h-part:(L,W,R)=(72.26999pt, 469.75502pt, 72.26999pt) | |
| 1074 | +* v-part:(T,H,B)=(72.26999pt, 650.43001pt, 72.26999pt) | |
| 1075 | +* \paperwidth=614.295pt | |
| 1076 | +* \paperheight=794.96999pt | |
| 1077 | +* \textwidth=469.75502pt | |
| 1078 | +* \textheight=650.43001pt | |
| 1079 | +* \oddsidemargin=0.0pt | |
| 1080 | +* \evensidemargin=0.0pt | |
| 1081 | +* \topmargin=-37.0pt | |
| 1082 | +* \headheight=12.0pt | |
| 1083 | +* \headsep=25.0pt | |
| 1084 | +* \topskip=12.0pt | |
| 1085 | +* \footskip=30.0pt | |
| 1086 | +* \marginparwidth=44.0pt | |
| 1087 | +* \marginparsep=10.0pt | |
| 1088 | +* \columnsep=10.0pt | |
| 1089 | +* \skip\footins=10.8pt plus 4.0pt minus 2.0pt | |
| 1090 | +* \hoffset=0.0pt | |
| 1091 | +* \voffset=0.0pt | |
| 1092 | +* \mag=1000 | |
| 1093 | +* \@twocolumnfalse | |
| 1094 | +* \@twosidefalse | |
| 1095 | +* \@mparswitchfalse | |
| 1096 | +* \@reversemarginfalse | |
| 1097 | +* (1in=72.27pt=25.4mm, 1cm=28.453pt) | |
| 1098 | + | |
| 1099 | +LaTeX Info: Redefining \microtypecontext on input line 85. | |
| 1100 | +Package microtype Info: Applying patch `item' on input line 85. | |
| 1101 | +Package microtype Info: Applying patch `toc' on input line 85. | |
| 1102 | +Package microtype Info: Applying patch `eqnum' on input line 85. | |
| 1103 | +Package microtype Info: Applying patch `footnote' on input line 85. | |
| 1104 | +Package microtype Info: Applying patch `verbatim' on input line 85. | |
| 1105 | +LaTeX Info: Redefining \microtypesetup on input line 85. | |
| 1106 | +Package microtype Info: Generating PDF output. | |
| 1107 | +Package microtype Info: Character protrusion enabled (level 2). | |
| 1108 | +Package microtype Info: Using default protrusion set `alltext'. | |
| 1109 | +Package microtype Info: Automatic font expansion enabled (level 2), | |
| 1110 | +(microtype) stretch: 20, shrink: 20, step: 1, non-selected. | |
| 1111 | +Package microtype Info: Using default expansion set `alltext-nott'. | |
| 1112 | +Package microtype Info: Patching command `\showhyphens'. | |
| 1113 | +Package microtype Info: No adjustment of tracking. | |
| 1114 | +Package microtype Info: No adjustment of interword spacing. | |
| 1115 | +Package microtype Info: No adjustment of character kerning. | |
| 1116 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-ptm.cfg | |
| 1117 | +File: mt-ptm.cfg 2006/04/20 v1.7 microtype config. file: Times (RS) | |
| 1118 | +) | |
| 1119 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/translations/translations-basic-d | |
| 1120 | +ictionary-english.trsl | |
| 1121 | +File: translations-basic-dictionary-english.trsl (english translation file `tra | |
| 1122 | +nslations-basic-dictionary') | |
| 1123 | +) | |
| 1124 | +Package translations Info: loading dictionary `translations-basic-dictionary' f | |
| 1125 | +or `english'. on input line 85. | |
| 1126 | + | |
| 1127 | +(/usr/local/texlive/2026/texmf-dist/tex/context/base/mkii/supp-pdf.mkii | |
| 1128 | +[Loading MPS to PDF converter (version 2006.09.02).] | |
| 1129 | +\scratchcounter=\count397 | |
| 1130 | +\scratchdimen=\dimen331 | |
| 1131 | +\scratchbox=\box91 | |
| 1132 | +\nofMPsegments=\count398 | |
| 1133 | +\nofMParguments=\count399 | |
| 1134 | +\everyMPshowfont=\toks56 | |
| 1135 | +\MPscratchCnt=\count400 | |
| 1136 | +\MPscratchDim=\dimen332 | |
| 1137 | +\MPnumerator=\count401 | |
| 1138 | +\makeMPintoPDFobject=\count402 | |
| 1139 | +\everyMPtoPDFconversion=\toks57 | |
| 1140 | +) (/usr/local/texlive/2026/texmf-dist/tex/latex/epstopdf-pkg/epstopdf-base.sty | |
| 1141 | +Package: epstopdf-base 2020-01-24 v2.11 Base part for package epstopdf | |
| 1142 | +Package epstopdf-base Info: Redefining graphics rule for `.eps' on input line 4 | |
| 1143 | +85. | |
| 1144 | + | |
| 1145 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg | |
| 1146 | +File: epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Liv | |
| 1147 | +e | |
| 1148 | +)) | |
| 1149 | +Package caption Info: Begin \AtBeginDocument code. | |
| 1150 | +Package caption Info: float package is loaded. | |
| 1151 | +Package caption Info: hyperref package is loaded. | |
| 1152 | +Package caption Info: listings package is loaded. | |
| 1153 | +Package caption Info: End \AtBeginDocument code. | |
| 1154 | +Package hyperref Info: Link coloring ON on input line 85. | |
| 1155 | + | |
| 1156 | +(./main.out) (./main.out) | |
| 1157 | +\@outlinefile=\write4 | |
| 1158 | +\openout4 = `main.out'. | |
| 1159 | + | |
| 1160 | +\c@lstlisting=\count403 | |
| 1161 | + (./sections/titlepage.tex | |
| 1162 | +<uq_logo.jpg, id=143, 722.7pt x 361.35pt> | |
| 1163 | +File: uq_logo.jpg Graphic file (type jpg) | |
| 1164 | +<use uq_logo.jpg> | |
| 1165 | +Package pdftex.def Info: uq_logo.jpg used on input line 9. | |
| 1166 | +(pdftex.def) Requested size: 99.58464pt x 49.78915pt. | |
| 1167 | + [1 | |
| 1168 | + | |
| 1169 | +{/usr/local/texlive/2026/texmf-var/fonts/map/pdftex/updmap/pdftex.map}{/usr/loc | |
| 1170 | +al/texlive/2026/texmf-dist/fonts/enc/dvips/base/8r.enc} <./uq_logo.jpg>] (./sec | |
| 1171 | +tions/abstract.tex) | |
| 1172 | +[2]) (./sections/introduction.tex) (./sections/related.tex | |
| 1173 | +Package microtype Info: Loading generic protrusion settings for font family | |
| 1174 | +(microtype) `cmtt' (encoding: T1). | |
| 1175 | +(microtype) For optimal results, create family-specific settings. | |
| 1176 | +(microtype) See the microtype manual for details. | |
| 1177 | + [3{/usr/local/texlive/2026/texmf-dist/fonts/enc/dvips/cm-super/cm-super-t1.enc | |
| 1178 | +}]) | |
| 1179 | +(./sections/problem.tex | |
| 1180 | +LaTeX Font Info: Trying to load font information for OT1+ztmcm on input line | |
| 1181 | + 4. | |
| 1182 | + | |
| 1183 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/ot1ztmcm.fd | |
| 1184 | +File: ot1ztmcm.fd 2000/01/03 Fontinst v1.801 font definitions for OT1/ztmcm. | |
| 1185 | +) | |
| 1186 | +Package microtype Info: Loading generic protrusion settings for font family | |
| 1187 | +(microtype) `ztmcm' (encoding: OT1). | |
| 1188 | +(microtype) For optimal results, create family-specific settings. | |
| 1189 | +(microtype) See the microtype manual for details. | |
| 1190 | +LaTeX Font Info: Trying to load font information for OML+ztmcm on input line | |
| 1191 | + 4. | |
| 1192 | + | |
| 1193 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omlztmcm.fd | |
| 1194 | +File: omlztmcm.fd 2000/01/03 Fontinst v1.801 font definitions for OML/ztmcm. | |
| 1195 | +) | |
| 1196 | +LaTeX Font Info: Trying to load font information for OMS+ztmcm on input line | |
| 1197 | + 4. | |
| 1198 | + | |
| 1199 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omsztmcm.fd | |
| 1200 | +File: omsztmcm.fd 2000/01/03 Fontinst v1.801 font definitions for OMS/ztmcm. | |
| 1201 | +) | |
| 1202 | +LaTeX Font Info: Trying to load font information for OMX+ztmcm on input line | |
| 1203 | + 4. | |
| 1204 | + | |
| 1205 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/omxztmcm.fd | |
| 1206 | +File: omxztmcm.fd 2000/01/03 Fontinst v1.801 font definitions for OMX/ztmcm. | |
| 1207 | +) | |
| 1208 | +LaTeX Font Info: Trying to load font information for OT1+ptm on input line 4 | |
| 1209 | +. | |
| 1210 | + | |
| 1211 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/ot1ptm.fd | |
| 1212 | +File: ot1ptm.fd 2001/06/04 font definitions for OT1/ptm. | |
| 1213 | +) | |
| 1214 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <12> not available | |
| 1215 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 4. | |
| 1216 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <9> not available | |
| 1217 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 4. | |
| 1218 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <7> not available | |
| 1219 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 4. | |
| 1220 | + | |
| 1221 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-msa.cfg | |
| 1222 | +File: mt-msa.cfg 2006/02/04 v1.1 microtype config. file: AMS symbols (a) (RS) | |
| 1223 | +) | |
| 1224 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/microtype/mt-msb.cfg | |
| 1225 | +File: mt-msb.cfg 2005/06/01 v1.0 microtype config. file: AMS symbols (b) (RS) | |
| 1226 | +) [4]) | |
| 1227 | +(./sections/architecture.tex (./figures/architecture.tex | |
| 1228 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <10> not available | |
| 1229 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 9. | |
| 1230 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <7.4> not available | |
| 1231 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 9. | |
| 1232 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <6> not available | |
| 1233 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 9. | |
| 1234 | +) | |
| 1235 | +Overfull \hbox (105.46294pt too wide) in paragraph at lines 36--13 | |
| 1236 | + [][] | |
| 1237 | + [] | |
| 1238 | + | |
| 1239 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <10.95> not available | |
| 1240 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 13. | |
| 1241 | +LaTeX Font Info: Font shape `OT1/ptm/bx/n' in size <8> not available | |
| 1242 | +(Font) Font shape `OT1/ptm/b/n' tried instead on input line 13. | |
| 1243 | + | |
| 1244 | +Overfull \hbox (0.70027pt too wide) detected at line 28 | |
| 1245 | +[][][][] \OT1/ztmcm/m/n/12 = [][](\OML/ztmcm/m/it/12 H[]\OT1/ztmcm/m/n/12 )\OML | |
| 1246 | +/ztmcm/m/it/12 ; K[] \OT1/ztmcm/m/n/12 = []\OML/ztmcm/m/it/12 W[]; V[] \OT1/z | |
| 1247 | +tmcm/m/n/12 = []\OML/ztmcm/m/it/12 W[]; z[] \OT1/ztmcm/m/n/12 = \OML/ztmcm/m/i | |
| 1248 | +t/12 P []; W[]; W[]; P[] \OMS/ztmcm/m/n/12 2 \U/msb/m/n/12 R[]\OML/ztmcm/m/it/ | |
| 1249 | +12 : | |
| 1250 | + [] | |
| 1251 | + | |
| 1252 | +[5] | |
| 1253 | +Overfull \hbox (8.35876pt too wide) detected at line 60 | |
| 1254 | +[] | |
| 1255 | + [] | |
| 1256 | + | |
| 1257 | +[6] | |
| 1258 | +Overfull \hbox (22.86838pt too wide) detected at line 74 | |
| 1259 | +[][][]\OML/ztmcm/m/it/12 ^^N[] \OT1/ztmcm/m/n/12 = [] \OMS/ztmcm/m/n/12 ^^@ \OM | |
| 1260 | +L/ztmcm/m/it/12 S[]; ^^N[] \OT1/ztmcm/m/n/12 = \OML/ztmcm/m/it/12 ^^N[] [] [] | |
| 1261 | +\OT1/ztmcm/m/n/12 1\OML/ztmcm/m/it/12 ; [][]; ^^Q[] \OT1/ztmcm/m/n/12 = \OML/z | |
| 1262 | +tmcm/m/it/12 ^^Q[] ^^[ []g\OT1/ztmcm/m/n/12 ([\OML/ztmcm/m/it/12 S[]\OT1/ztmcm/ | |
| 1263 | +m/n/12 ; []])[]\OML/ztmcm/m/it/12 ; S[] \OT1/ztmcm/m/n/12 = \OML/ztmcm/m/it/12 | |
| 1264 | + S[] \OT1/ztmcm/m/n/12 + \OML/ztmcm/m/it/12 ^^Q[]^^N[]; | |
| 1265 | + [] | |
| 1266 | + | |
| 1267 | + | |
| 1268 | +Overfull \hbox (2.33179pt too wide) in paragraph at lines 81--84 | |
| 1269 | +\T1/ptm/m/n/12 (-20) A sin-gle at-ten-tion read with a query that de-pends on $ | |
| 1270 | +\OML/ztmcm/m/it/12 z[]$ \T1/ptm/m/n/12 (-20) only, $\OML/ztmcm/m/it/12 r\OT1/zt | |
| 1271 | +mcm/m/n/12 (\OML/ztmcm/m/it/12 S; z[]\OT1/ztmcm/m/n/12 ) = [][] \OML/ztmcm/m/it | |
| 1272 | +/12 []S[]$\T1/ptm/m/n/12 (-20) , $\OML/ztmcm/m/it/12 \OT1/ztmcm/m/n/12 = [][ | |
| 1273 | +][](\OML/ztmcm/m/it/12 W[]z[]\OT1/ztmcm/m/n/12 ) \OMS/ztmcm/m/n/12 ^^A | |
| 1274 | + [] | |
| 1275 | + | |
| 1276 | +(./tables/configs.tex)) (./sections/policy.tex [7] | |
| 1277 | + | |
| 1278 | +LaTeX Warning: Reference `sec:results-oracle' on page 8 undefined on input line | |
| 1279 | + 29. | |
| 1280 | + | |
| 1281 | +) (./sections/learning.tex [8] | |
| 1282 | + | |
| 1283 | +LaTeX Warning: Reference `sec:results-reasoning' on page 9 undefined on input l | |
| 1284 | +ine 14. | |
| 1285 | + | |
| 1286 | + | |
| 1287 | +LaTeX Warning: Reference `sec:results-learnability' on page 9 undefined on inpu | |
| 1288 | +t line 37. | |
| 1289 | + | |
| 1290 | +[9]) (./sections/theory.tex [10] | |
| 1291 | + | |
| 1292 | +LaTeX Font Warning: Font shape `T1/ptm/m/scit' undefined | |
| 1293 | +(Font) using `T1/ptm/m/sc' instead on input line 37. | |
| 1294 | + | |
| 1295 | + | |
| 1296 | +LaTeX Warning: Reference `sec:results-oracle' on page 11 undefined on input lin | |
| 1297 | +e 46. | |
| 1298 | + | |
| 1299 | +) (./sections/implementation.tex [11] (./tables/ops.tex | |
| 1300 | +Underfull \hbox (badness 10000) in paragraph at lines 5--5 | |
| 1301 | +[]\T1/ptm/b/n/10.95 (+20) Table 2. |\T1/ptm/m/n/10.95 (+20) An-a-lytic op-er-a- | |
| 1302 | +tions per gen-er-ated to-ken, PRISME-459M, con-text $\OML/ztmcm/m/it/10.95 T \O | |
| 1303 | +T1/ztmcm/m/n/10.95 = 4096$ | |
| 1304 | + [] | |
| 1305 | + | |
| 1306 | +) | |
| 1307 | + | |
| 1308 | +LaTeX Warning: Reference `sec:results-latency' on page 12 undefined on input li | |
| 1309 | +ne 45. | |
| 1310 | + | |
| 1311 | +) (./sections/design.tex [12] | |
| 1312 | +Package microtype Info: Loading generic protrusion settings for font family | |
| 1313 | +(microtype) `cmss' (encoding: T1). | |
| 1314 | +(microtype) For optimal results, create family-specific settings. | |
| 1315 | +(microtype) See the microtype manual for details. | |
| 1316 | + (./tables/families.tex) | |
| 1317 | +Overfull \hbox (36.62411pt too wide) in paragraph at lines 19--24 | |
| 1318 | +\T1/ptm/m/n/12 (-20) 2.91M val-i-da-tion and 2.88M locked test to-kens (\T1/cmt | |
| 1319 | +t/m/n/12 data/processed/fineweb_edu_300m/manifest.json\T1/ptm/m/n/12 (-20) ). | |
| 1320 | + [] | |
| 1321 | + | |
| 1322 | +[13] | |
| 1323 | + | |
| 1324 | +LaTeX Warning: Reference `tab:controls' on page 14 undefined on input line 27. | |
| 1325 | + | |
| 1326 | +) (./sections/results.tex) (./sections/mechanistic.tex) | |
| 1327 | +(./sections/limitations.tex) (./sections/conclusion.tex) | |
| 1328 | +(./sections/reproducibility.tex) | |
| 1329 | +Overfull \hbox (71.70984pt too wide) in paragraph at lines 38--101 | |
| 1330 | +\T1/ptm/m/n/12 (-20) The pri-mary out-come, met-rics, de-ci-sion thresh-olds an | |
| 1331 | +d seed plan were writ-ten to \T1/cmtt/m/n/12 protocol/preregistration.md | |
| 1332 | + [] | |
| 1333 | + | |
| 1334 | +[14] (./main.bbl [15] [16] | |
| 1335 | +Underfull \hbox (badness 1297) in paragraph at lines 208--211 | |
| 1336 | +[]\T1/ptm/b/n/12 (+20) You, Yichen, Tianyu Fu, Aosong Feng, Xing-tai Lv, Xue-fe | |
| 1337 | +i Ning, Ning Ding, and | |
| 1338 | + [] | |
| 1339 | + | |
| 1340 | +[17]) (./appendix/appendix.tex (./appendix/notation.tex | |
| 1341 | +Overfull \hbox (12.19864pt too wide) in paragraph at lines 2--16 | |
| 1342 | + [] | |
| 1343 | + [] | |
| 1344 | + | |
| 1345 | +) (./appendix/proofs.tex [18]) (./appendix/params.tex [19] | |
| 1346 | +Overfull \hbox (92.32437pt too wide) detected at line 12 | |
| 1347 | +\OML/ztmcm/m/it/12 N[] \OT1/ztmcm/m/n/12 = \OML/ztmcm/m/it/12 d \OT1/ztmcm/m/n/ | |
| 1348 | +12 + 3\OML/ztmcm/m/it/12 dw \OT1/ztmcm/m/n/12 + \OML/ztmcm/m/it/12 Mw \OT1/ztmc | |
| 1349 | +m/m/n/12 + [] + [] + [] + []\OML/ztmcm/m/it/12 ; | |
| 1350 | + [] | |
| 1351 | + | |
| 1352 | +) (./appendix/literature.tex | |
| 1353 | +Underfull \hbox (badness 10000) in paragraph at lines 15--15 | |
| 1354 | +[]|\T1/ptm/m/n/10.95 (+20) C1 se-lected lo- | |
| 1355 | + [] | |
| 1356 | + | |
| 1357 | + | |
| 1358 | +Underfull \hbox (badness 10000) in paragraph at lines 15--15 | |
| 1359 | +\T1/ptm/m/n/10.95 (+20) cal writes into | |
| 1360 | + [] | |
| 1361 | + | |
| 1362 | + | |
| 1363 | +Underfull \hbox (badness 10000) in paragraph at lines 15--15 | |
| 1364 | +\T1/ptm/m/n/10.95 (+20) a per-token slot | |
| 1365 | + [] | |
| 1366 | + | |
| 1367 | + | |
| 1368 | +Underfull \hbox (badness 4543) in paragraph at lines 15--15 | |
| 1369 | +\T1/ptm/m/n/10.95 (+20) AR, Slot At-ten-tion, | |
| 1370 | + [] | |
| 1371 | + | |
| 1372 | + | |
| 1373 | +Underfull \hbox (badness 1320) in paragraph at lines 15--15 | |
| 1374 | +[]|\T1/ptm/m/n/10.95 (+20) no prior work com-bines | |
| 1375 | + [] | |
| 1376 | + | |
| 1377 | + | |
| 1378 | +Underfull \hbox (badness 10000) in paragraph at lines 15--15 | |
| 1379 | +[]|\T1/ptm/m/n/10.95 (+20) C2 action- | |
| 1380 | + [] | |
| 1381 | + | |
| 1382 | + | |
| 1383 | +Underfull \hbox (badness 10000) in paragraph at lines 15--15 | |
| 1384 | +\T1/ptm/m/n/10.95 (+20) conditional coun- | |
| 1385 | + [] | |
| 1386 | + | |
| 1387 | + | |
| 1388 | +Underfull \hbox (badness 10000) in paragraph at lines 15--15 | |
| 1389 | +\T1/ptm/m/n/10.95 (+20) ter-fac-tual gain | |
| 1390 | + [] | |
| 1391 | + | |
| 1392 | + | |
| 1393 | +Underfull \hbox (badness 10000) in paragraph at lines 15--15 | |
| 1394 | +\T1/ptm/m/n/10.95 (+20) su-per-vi-sion with | |
| 1395 | + [] | |
| 1396 | + | |
| 1397 | + | |
| 1398 | +Underfull \hbox (badness 10000) in paragraph at lines 15--15 | |
| 1399 | +\T1/ptm/m/n/10.95 (+20) ``looka-head depth | |
| 1400 | + [] | |
| 1401 | + | |
| 1402 | + | |
| 1403 | +Underfull \hbox (badness 3019) in paragraph at lines 15--15 | |
| 1404 | +\T1/ptm/m/n/10.95 (+20) su-per-vi-sion'', Re-cur- | |
| 1405 | + [] | |
| 1406 | + | |
| 1407 | + | |
| 1408 | +Underfull \hbox (badness 1681) in paragraph at lines 15--15 | |
| 1409 | +[]|\T1/ptm/m/n/10.95 (+20) binary con-tinue/stop ver- | |
| 1410 | + [] | |
| 1411 | + | |
| 1412 | + | |
| 1413 | +Underfull \hbox (badness 1221) in paragraph at lines 15--15 | |
| 1414 | +\T1/ptm/m/n/10.95 (+20) sions ex-ist; con-di-tion-ing | |
| 1415 | + [] | |
| 1416 | + | |
| 1417 | + | |
| 1418 | +Underfull \hbox (badness 3701) in paragraph at lines 15--15 | |
| 1419 | +\T1/ptm/m/n/10.95 (+20) spa-tial ac-tions is taken, | |
| 1420 | + [] | |
| 1421 | + | |
| 1422 | + | |
| 1423 | +Underfull \hbox (badness 10000) in paragraph at lines 15--15 | |
| 1424 | +\T1/ptm/m/n/10.95 (+20) a-rate gain/cost/risk/harm | |
| 1425 | + [] | |
| 1426 | + | |
| 1427 | + | |
| 1428 | +Underfull \hbox (badness 10000) in paragraph at lines 15--15 | |
| 1429 | +[]|\T1/ptm/m/n/10.95 (+20) ReZero, Flamingo, | |
| 1430 | + [] | |
| 1431 | + | |
| 1432 | + | |
| 1433 | +Underfull \hbox (badness 2762) in paragraph at lines 15--15 | |
| 1434 | +\T1/ptm/m/n/10.95 (+20) sin-gles + 8 fixed | |
| 1435 | + [] | |
| 1436 | + | |
| 1437 | + | |
| 1438 | +Underfull \hbox (badness 10000) in paragraph at lines 15--15 | |
| 1439 | +[]|\T1/ptm/m/n/10.95 (+20) top-$\OML/ztmcm/m/it/10.95 k$ \T1/ptm/m/n/10.95 (+20 | |
| 1440 | +) ex- | |
| 1441 | + [] | |
| 1442 | + | |
| 1443 | + | |
| 1444 | +Underfull \hbox (badness 10000) in paragraph at lines 15--15 | |
| 1445 | +\T1/ptm/m/n/10.95 (+20) pert/mechanism | |
| 1446 | + [] | |
| 1447 | + | |
| 1448 | + | |
| 1449 | +LaTeX Warning: `h' float specifier changed to `ht'. | |
| 1450 | + | |
| 1451 | +) (./appendix/prereg.tex | |
| 1452 | +LaTeX Font Info: Font shape `T1/cmtt/bx/n' in size <12> not available | |
| 1453 | +(Font) Font shape `T1/cmtt/m/n' tried instead on input line 1. | |
| 1454 | + [20] | |
| 1455 | +LaTeX Font Info: Trying to load font information for TS1+ptm on input line 8 | |
| 1456 | +. | |
| 1457 | + | |
| 1458 | +(/usr/local/texlive/2026/texmf-dist/tex/latex/psnfss/ts1ptm.fd | |
| 1459 | +File: ts1ptm.fd 2001/06/04 font definitions for TS1/ptm. | |
| 1460 | +) | |
| 1461 | +Overfull \hbox (80.09856pt too wide) in paragraph at lines 10--12 | |
| 1462 | +[]\T1/ptm/b/n/12 (-20) Synthetic com-po-si-tional suite v1 \T1/ptm/m/n/12 (-20) | |
| 1463 | + (\T1/cmtt/m/n/12 data/splits/synthetic_v1\T1/ptm/m/n/12 (-20) , gen-er-a-tor \ | |
| 1464 | +T1/cmtt/m/n/12 data/generators/synthetic.py\T1/ptm/m/n/12 (-20) , | |
| 1465 | + [] | |
| 1466 | + | |
| 1467 | +[21] [22]) (./appendix/runs.tex) (./appendix/results.tex) | |
| 1468 | +(./appendix/failures.tex | |
| 1469 | +Overfull \hbox (23.89659pt too wide) in paragraph at lines 8--10 | |
| 1470 | +[]\T1/ptm/b/n/12 (-20) date\T1/ptm/m/n/12 (-20) : 2026-10-29; \T1/ptm/b/n/12 (- | |
| 1471 | +20) what failed\T1/ptm/m/n/12 (-20) : rsync ex-clu-sion \T1/cmtt/m/n/12 experim | |
| 1472 | +ents \T1/ptm/m/n/12 (-20) also dropped \T1/cmtt/m/n/12 configs/experiments/\T1/ | |
| 1473 | +ptm/m/n/12 (-20) ; | |
| 1474 | + [] | |
| 1475 | + | |
| 1476 | + | |
| 1477 | +Overfull \hbox (197.64227pt too wide) in paragraph at lines 8--10 | |
| 1478 | +\T1/ptm/m/n/12 (-20) base-line launch on M3U96b failed twice (\T1/cmtt/m/n/12 F | |
| 1479 | +ileNotFoundError\T1/ptm/m/n/12 (-20) ).; \T1/ptm/b/n/12 (-20) ev-i-dence\T1/ptm | |
| 1480 | +/m/n/12 (-20) : \T1/cmtt/m/n/12 experiments/logs/pretrain_enlarged102m_seed1.lo | |
| 1481 | +g | |
| 1482 | + [] | |
| 1483 | + | |
| 1484 | + | |
| 1485 | +Overfull \hbox (49.9369pt too wide) in paragraph at lines 10--12 | |
| 1486 | +\T1/ptm/m/n/12 (-20) be-fore ed-its can af-fect log-its.; \T1/ptm/b/n/12 (-20) | |
| 1487 | +ev-i-dence\T1/ptm/m/n/12 (-20) : \T1/cmtt/m/n/12 experiments/runs/p4_editor_v1_ | |
| 1488 | +seed01_*/metrics.jsonl | |
| 1489 | + [] | |
| 1490 | + | |
| 1491 | +[23] | |
| 1492 | +Overfull \hbox (47.44559pt too wide) in paragraph at lines 12--14 | |
| 1493 | +\T1/cmtt/m/n/12 experiments/logs/p4_diag_mixed_w128_lr3e3.log\T1/ptm/m/n/12 (-2 | |
| 1494 | +0) , run \T1/cmtt/m/n/12 p4_diag_mixed_w128_lr3e3_seed01_*\T1/ptm/m/n/12 (-20) | |
| 1495 | +; | |
| 1496 | + [] | |
| 1497 | + | |
| 1498 | +Package microtype Info: Loading generic protrusion settings for font family | |
| 1499 | +(microtype) `cmtt' (encoding: TS1). | |
| 1500 | +(microtype) For optimal results, create family-specific settings. | |
| 1501 | +(microtype) See the microtype manual for details. | |
| 1502 | + | |
| 1503 | +Overfull \hbox (26.39578pt too wide) in paragraph at lines 14--16 | |
| 1504 | +\T1/ptm/m/n/12 (-20) eval-u-a-tion (\T1/cmtt/m/n/12 AttributeError: \TS1/cmtt/m | |
| 1505 | +/n/12 '\T1/cmtt/m/n/12 GlobalRecurrenceModel\TS1/cmtt/m/n/12 ' \T1/cmtt/m/n/12 | |
| 1506 | +object has no attribute \TS1/cmtt/m/n/12 '\T1/cmtt/m/n/12 W_K\TS1/cmtt/m/n/12 ' | |
| 1507 | +\T1/ptm/m/n/12 (-20) ): | |
| 1508 | + [] | |
| 1509 | + | |
| 1510 | + | |
| 1511 | +Overfull \hbox (12.79033pt too wide) in paragraph at lines 18--20 | |
| 1512 | +\T1/cmtt/m/n/12 experiments/logs/p4_tiny_backbone_v2.log\T1/ptm/m/n/12 (-20) , | |
| 1513 | +run \T1/cmtt/m/n/12 p4_tiny_backbone_v2_seed01_* \T1/ptm/m/n/12 (-20) (state: | |
| 1514 | + [] | |
| 1515 | + | |
| 1516 | +)) [24{/usr/local/texlive/2026/texmf-dist/fonts/enc/dvips/cm-super/cm-super-ts1 | |
| 1517 | +.enc}] (./main.aux) | |
| 1518 | + *********** | |
| 1519 | +LaTeX2e <2025-11-01> | |
| 1520 | +L3 programming layer <2026-01-19> | |
| 1521 | + *********** | |
| 1522 | + | |
| 1523 | + | |
| 1524 | +LaTeX Warning: There were undefined references. | |
| 1525 | + | |
| 1526 | +Package rerunfilecheck Info: File `main.out' has not changed. | |
| 1527 | +(rerunfilecheck) Checksum: 5DF33CC747756FF4B32FB47A63C0D5B8;5650. | |
| 1528 | + ) | |
| 1529 | +Here is how much of TeX's memory you used: | |
| 1530 | + 35658 strings out of 467525 | |
| 1531 | + 695072 string characters out of 5418982 | |
| 1532 | + 1097683 words of memory out of 5000000 | |
| 1533 | + 63228 multiletter control sequences out of 15000+600000 | |
| 1534 | + 780304 words of font info for 542 fonts, out of 8000000 for 9000 | |
| 1535 | + 1141 hyphenation exceptions out of 8191 | |
| 1536 | + 84i,13n,104p,1001b,1381s stack positions out of 10000i,1000n,20000p,200000b,200000s | |
| 1537 | +</usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmex10.pfb | |
| 1538 | +></usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi10.pfb> | |
| 1539 | +</usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pfb></ | |
| 1540 | +usr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/cm/cmsy10.pfb></u | |
| 1541 | +sr/local/texlive/2026/texmf-dist/fonts/type1/public/amsfonts/symbols/msbm10.pfb | |
| 1542 | +></usr/local/texlive/2026/texmf-dist/fonts/type1/public/rsfs/rsfs10.pfb></usr/l | |
| 1543 | +ocal/texlive/2026/texmf-dist/fonts/type1/public/cm-super/sfss1200.pfb></usr/loc | |
| 1544 | +al/texlive/2026/texmf-dist/fonts/type1/public/cm-super/sftt1095.pfb></usr/local | |
| 1545 | +/texlive/2026/texmf-dist/fonts/type1/public/cm-super/sftt1200.pfb></usr/local/t | |
| 1546 | +exlive/2026/texmf-dist/fonts/type1/urw/symbol/usyr.pfb></usr/local/texlive/2026 | |
| 1547 | +/texmf-dist/fonts/type1/urw/symbol/usyr.pfb></usr/local/texlive/2026/texmf-dist | |
| 1548 | +/fonts/type1/urw/times/utmb8a.pfb></usr/local/texlive/2026/texmf-dist/fonts/typ | |
| 1549 | +e1/urw/times/utmbi8a.pfb></usr/local/texlive/2026/texmf-dist/fonts/type1/urw/ti | |
| 1550 | +mes/utmr8a.pfb></usr/local/texlive/2026/texmf-dist/fonts/type1/urw/times/utmri8 | |
| 1551 | +a.pfb> | |
| 1552 | +Output written on main.pdf (24 pages, 337432 bytes). | |
| 1553 | +PDF statistics: | |
| 1554 | + 540 PDF objects out of 1000 (max. 8388607) | |
| 1555 | + 477 compressed objects within 5 object streams | |
| 1556 | + 117 named destinations out of 1000 (max. 500000) | |
| 1557 | + 133410 words of extra memory for PDF output out of 154059 (max. 10000000) | |
| 1558 | + | |
added
paper/main.pdf
+0 −0
Binary file not shown.
added
paper/main.tex
+104 −0
@@ -0,0 +1,104 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% UQO Working Paper 31 — PRISME | |
| 3 | +% ============================================================================ | |
| 4 | +\documentclass[12pt,letterpaper]{article} | |
| 5 | +\usepackage[utf8]{inputenc} | |
| 6 | +\usepackage[T1]{fontenc} | |
| 7 | +\usepackage[english]{babel} | |
| 8 | +\usepackage[letterpaper,top=1in,bottom=1in,left=1in,right=1in]{geometry} | |
| 9 | +\usepackage{mathptmx} | |
| 10 | +\usepackage{microtype} | |
| 11 | +\usepackage{setspace} | |
| 12 | +\onehalfspacing | |
| 13 | +\usepackage{amsmath,amssymb,amsthm} | |
| 14 | +\usepackage{bm} | |
| 15 | +\usepackage{booktabs} | |
| 16 | +\usepackage{tabularx} | |
| 17 | +\usepackage{multirow} | |
| 18 | +\usepackage{threeparttable} | |
| 19 | +\usepackage{longtable,array,calc} | |
| 20 | +\usepackage{siunitx} | |
| 21 | +\sisetup{input-symbols = {()}, table-format = -1.3, parse-numbers = false} | |
| 22 | +\usepackage{graphicx} | |
| 23 | +\usepackage{subcaption} | |
| 24 | +\usepackage{float} | |
| 25 | +\usepackage{tikz} | |
| 26 | +\usetikzlibrary{arrows.meta,positioning,calc,fit,backgrounds,shapes.misc} | |
| 27 | +\usepackage[dvipsnames]{xcolor} | |
| 28 | +\usepackage[colorlinks=true,linkcolor=NavyBlue,citecolor=NavyBlue,urlcolor=NavyBlue,breaklinks=true]{hyperref} | |
| 29 | +\usepackage[authoryear,round,semicolon]{natbib} | |
| 30 | +\bibliographystyle{aer} | |
| 31 | +\usepackage[font=small,labelfont=bf,labelsep=period,justification=justified,singlelinecheck=false]{caption} | |
| 32 | +\usepackage{fancyhdr} | |
| 33 | +\pagestyle{fancy} | |
| 34 | +\fancyhf{} | |
| 35 | +\renewcommand{\headrulewidth}{0pt} | |
| 36 | +\fancyfoot[C]{\thepage} | |
| 37 | +\usepackage{titlesec} | |
| 38 | +\titleformat{\section}{\large\bfseries}{\thesection.}{0.5em}{} | |
| 39 | +\titleformat{\subsection}{\normalsize\bfseries}{\thesubsection.}{0.5em}{} | |
| 40 | +\titleformat{\subsubsection}{\normalsize\itshape}{\thesubsubsection.}{0.5em}{} | |
| 41 | +\usepackage[toc,page]{appendix} | |
| 42 | +\usepackage{enumitem} | |
| 43 | +\setlist{nosep,leftmargin=*} | |
| 44 | +\usepackage{algorithm} | |
| 45 | +\usepackage{algpseudocode} | |
| 46 | +\usepackage{listings} | |
| 47 | + | |
| 48 | +\theoremstyle{plain} | |
| 49 | +\newtheorem{proposition}{Proposition} | |
| 50 | +\newtheorem{lemma}{Lemma} | |
| 51 | +\newtheorem{corollary}{Corollary} | |
| 52 | +\theoremstyle{definition} | |
| 53 | +\newtheorem{definition}{Definition} | |
| 54 | +\newtheorem{assumption}{Assumption} | |
| 55 | +\theoremstyle{remark} | |
| 56 | +\newtheorem{remark}{Remark} | |
| 57 | + | |
| 58 | +\newcommand{\R}{\mathbb{R}} | |
| 59 | +\newcommand{\E}{\mathbb{E}} | |
| 60 | +\newcommand{\Loss}{\mathcal{L}} | |
| 61 | +\newcommand{\Act}{\mathcal{A}} | |
| 62 | +\newcommand{\Stop}{\textsc{stop}} | |
| 63 | +\newcommand{\rms}{\operatorname{rms}} | |
| 64 | +\newcommand{\sg}{\operatorname{sg}} | |
| 65 | +\newcommand{\prisme}{\textsc{prisme}} | |
| 66 | +\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}} | |
| 67 | + | |
| 68 | +% Numbers generated by scripts (never typed by hand): results/tables/numbers.tex | |
| 69 | +\input{tables/numbers} | |
| 70 | + | |
| 71 | +% ============================================================================ | |
| 72 | +\newcommand{\WPnumber}{31} | |
| 73 | +\newcommand{\WPtitle}{PRISME: Intervention-Supervised Sparse Workspace Editing for Small Language Models} | |
| 74 | +\newcommand{\WPsubtitle}{Design, controlled evaluation, and negative and positive evidence at the 100M-parameter scale} | |
| 75 | +\newcommand{\WPdate}{October 2026} | |
| 76 | +\newcommand{\WPversion}{1.0} | |
| 77 | +\newcommand{\WPabstract}{\input{sections/abstract}} | |
| 78 | +\newcommand{\WPkeywords}{small language models; adaptive computation; latent workspace; counterfactual supervision; sparse routing; quality--compute frontier; reproducibility} | |
| 79 | +\newcommand{\WPjel}{C45, C63, C88} | |
| 80 | +\newcommand{\WPauthor}{Simon-Pierre Boucher} | |
| 81 | +\newcommand{\WPaffiliation}{Département des sciences administratives\\Université du Québec en Outaouais} | |
| 82 | +\newcommand{\WPemail}{simon-pierre.boucher@uqo.ca} | |
| 83 | +\newcommand{\WPaddress}{Gatineau -- Pavillon Alexandre-Taché\\283, boulevard Alexandre-Taché\\Gatineau, Québec, Canada J9A 1L8} | |
| 84 | + | |
| 85 | +\begin{document} | |
| 86 | +\input{sections/titlepage} | |
| 87 | +\input{sections/introduction} | |
| 88 | +\input{sections/related} | |
| 89 | +\input{sections/problem} | |
| 90 | +\input{sections/architecture} | |
| 91 | +\input{sections/policy} | |
| 92 | +\input{sections/learning} | |
| 93 | +\input{sections/theory} | |
| 94 | +\input{sections/implementation} | |
| 95 | +\input{sections/design} | |
| 96 | +\input{sections/results} | |
| 97 | +\input{sections/mechanistic} | |
| 98 | +\input{sections/limitations} | |
| 99 | +\input{sections/conclusion} | |
| 100 | +\input{sections/reproducibility} | |
| 101 | +\newpage | |
| 102 | +\bibliography{references} | |
| 103 | +\input{appendix/appendix} | |
| 104 | +\end{document} | |
added
paper/references.bib
+739 −0
@@ -0,0 +1,739 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% papers.bib -- UQO Working Paper 31 (PRISME), literature review | |
| 3 | +% Compiled 2026-10-29 by verifying every entry against primary sources: | |
| 4 | +% * arXiv Atom API (https://export.arxiv.org/api/query?id_list=...) for | |
| 5 | +% title, author list, first-submission date, latest version, comments/DOI; | |
| 6 | +% * GitHub REST API (api.github.com/repos/<owner>/<repo>) for code + licence; | |
| 7 | +% * JMLR / PMLR / project pages where cited. | |
| 8 | +% Convention: `year` = year of first arXiv submission unless a venue is given. | |
| 9 | +% Venues are stated ONLY when verified from arXiv metadata (comment/DOI field), | |
| 10 | +% a PMLR/JMLR page, or an official repository description. Where a venue is | |
| 11 | +% widely known but was NOT verified here, it is flagged in `note`. | |
| 12 | +% `note` also records code availability and licence (SPDX id from GitHub API). | |
| 13 | +% ============================================================================ | |
| 14 | + | |
| 15 | +% --------------------------------------------------------------------------- | |
| 16 | +% 1. Core references requested (12) | |
| 17 | +% --------------------------------------------------------------------------- | |
| 18 | + | |
| 19 | +@inproceedings{dehghani2019universal, | |
| 20 | + title = {Universal Transformers}, | |
| 21 | + author = {Dehghani, Mostafa and Gouws, Stephan and Vinyals, Oriol and Uszkoreit, Jakob and Kaiser, {\L}ukasz}, | |
| 22 | + booktitle = {International Conference on Learning Representations (ICLR)}, | |
| 23 | + year = {2019}, | |
| 24 | + eprint = {1807.03819}, | |
| 25 | + archivePrefix = {arXiv}, | |
| 26 | + url = {https://arxiv.org/abs/1807.03819}, | |
| 27 | +} | |
| 28 | + | |
| 29 | +@inproceedings{banino2021pondernet, | |
| 30 | + title = {PonderNet: Learning to Ponder}, | |
| 31 | + author = {Banino, Andrea and Balaguer, Jan and Blundell, Charles}, | |
| 32 | + booktitle = {8th ICML Workshop on Automated Machine Learning (AutoML)}, | |
| 33 | + year = {2021}, | |
| 34 | + eprint = {2107.05407}, | |
| 35 | + archivePrefix = {arXiv}, | |
| 36 | + url = {https://arxiv.org/abs/2107.05407}, | |
| 37 | +} | |
| 38 | + | |
| 39 | +@inproceedings{hawthorne2022perceiverar, | |
| 40 | + title = {General-purpose, long-context autoregressive modeling with {Perceiver AR}}, | |
| 41 | + author = {Hawthorne, Curtis and Jaegle, Andrew and Cangea, C{\u{a}}t{\u{a}}lina and Borgeaud, Sebastian and Nash, Charlie and Malinowski, Mateusz and Dieleman, Sander and Vinyals, Oriol and Botvinick, Matthew and Simon, Ian and Sheahan, Hannah and Zeghidour, Neil and Alayrac, Jean-Baptiste and Carreira, Jo{\~a}o and Engel, Jesse}, | |
| 42 | + booktitle = {International Conference on Machine Learning (ICML)}, | |
| 43 | + year = {2022}, | |
| 44 | + eprint = {2202.07765}, | |
| 45 | + archivePrefix = {arXiv}, | |
| 46 | + url = {https://arxiv.org/abs/2202.07765}, | |
| 47 | +} | |
| 48 | + | |
| 49 | +@article{goyal2019rims, | |
| 50 | + title = {Recurrent Independent Mechanisms}, | |
| 51 | + author = {Goyal, Anirudh and Lamb, Alex and Hoffmann, Jordan and Sodhani, Shagun and Levine, Sergey and Bengio, Yoshua and Sch{\"o}lkopf, Bernhard}, | |
| 52 | + journal = {arXiv preprint arXiv:1909.10893}, | |
| 53 | + year = {2019}, | |
| 54 | + url = {https://arxiv.org/abs/1909.10893}, | |
| 55 | +} | |
| 56 | + | |
| 57 | +@article{raposo2024mixtureofdepths, | |
| 58 | + title = {Mixture-of-Depths: Dynamically allocating compute in transformer-based language models}, | |
| 59 | + author = {Raposo, David and Ritter, Sam and Richards, Blake and Lillicrap, Timothy and Humphreys, Peter Conway and Santoro, Adam}, | |
| 60 | + journal = {arXiv preprint arXiv:2404.02258}, | |
| 61 | + year = {2024}, | |
| 62 | + url = {https://arxiv.org/abs/2404.02258}, | |
| 63 | +} | |
| 64 | + | |
| 65 | +@article{geiping2025recurrentdepth, | |
| 66 | + title = {Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach}, | |
| 67 | + author = {Geiping, Jonas and McLeish, Sean and Jain, Neel and Kirchenbauer, John and Singh, Siddharth and Bartoldson, Brian R. and Kailkhura, Bhavya and Bhatele, Abhinav and Goldstein, Tom}, | |
| 68 | + journal = {arXiv preprint arXiv:2502.05171}, | |
| 69 | + year = {2025}, | |
| 70 | + url = {https://arxiv.org/abs/2502.05171}, | |
| 71 | +} | |
| 72 | + | |
| 73 | +@inproceedings{bae2025mixtureofrecursions, | |
| 74 | + title = {Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation}, | |
| 75 | + author = {Bae, Sangmin and Kim, Yujin and Bayat, Reza and Kim, Sungnyun and Ha, Jiyoun and Schuster, Tal and Fisch, Adam and Harutyunyan, Hrayr and Ji, Ziwei and Courville, Aaron and Yun, Se-Young}, | |
| 76 | + booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, | |
| 77 | + year = {2025}, | |
| 78 | + eprint = {2507.10524}, | |
| 79 | + archivePrefix = {arXiv}, | |
| 80 | + url = {https://arxiv.org/abs/2507.10524}, | |
| 81 | +} | |
| 82 | + | |
| 83 | +@article{zhu2025ouro, | |
| 84 | + title = {Scaling Latent Reasoning via Looped Language Models}, | |
| 85 | + author = {Zhu, Rui-Jie and Wang, Zixuan and Hua, Kai and Zhang, Tianyu and Li, Ziniu and Que, Haoran and Wei, Boyi and Wen, Zixin and Yin, Fan and Xing, He and Li, Lu and Shi, Jiajun and Ma, Kaijing and Li, Shanda and Kergan, Taylor and Smith, Andrew and Qu, Xingwei and Hui, Mude and Wu, Bohong and Min, Qiyang and Huang, Hongzhi and Zhou, Xun and Ye, Wei and Liu, Jiaheng and Yang, Jian and Shi, Yunfeng and Lin, Chenghua and Zhao, Enduo and Cai, Tianle and Zhang, Ge and Huang, Wenhao and Bengio, Yoshua and Eshraghian, Jason}, | |
| 86 | + journal = {arXiv preprint arXiv:2510.25741}, | |
| 87 | + year = {2025}, | |
| 88 | + url = {https://arxiv.org/abs/2510.25741}, | |
| 89 | +} | |
| 90 | + | |
| 91 | +@article{wang2026recurtrace, | |
| 92 | + title = {RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory}, | |
| 93 | + author = {Wang, Yuxiang and Feng, Kunyu and Shen, Yingda and Xu, Haoning and Wang, Junyu and Wu, Zhizheng}, | |
| 94 | + journal = {arXiv preprint arXiv:2609.03379}, | |
| 95 | + year = {2026}, | |
| 96 | + url = {https://arxiv.org/abs/2609.03379}, | |
| 97 | +} | |
| 98 | + | |
| 99 | +@article{ladhak2026elastic, | |
| 100 | + title = {One Model to Loop Them All: Elastic Latent Recurrence for Language Models}, | |
| 101 | + author = {Ladhak, Faisal and Acharya, Shantanu and Akle Serano, Santiago and Sun, Simeng and Ginsburg, Boris and Puvvada, Krishna C.}, | |
| 102 | + journal = {arXiv preprint arXiv:2610.12582}, | |
| 103 | + year = {2026}, | |
| 104 | + url = {https://arxiv.org/abs/2610.12582}, | |
| 105 | +} | |
| 106 | + | |
| 107 | +@article{behrouz2024titans, | |
| 108 | + title = {Titans: Learning to Memorize at Test Time}, | |
| 109 | + author = {Behrouz, Ali and Zhong, Peilin and Mirrokni, Vahab}, | |
| 110 | + journal = {arXiv preprint arXiv:2501.00663}, | |
| 111 | + year = {2024}, | |
| 112 | + url = {https://arxiv.org/abs/2501.00663}, | |
| 113 | +} | |
| 114 | + | |
| 115 | +@inproceedings{elhoushi2024layerskip, | |
| 116 | + title = {LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding}, | |
| 117 | + author = {Elhoushi, Mostafa and Shrivastava, Akshat and Liskovich, Diana and Hosmer, Basil and Wasti, Bram and Lai, Liangzhen and Mahmoud, Anas and Acun, Bilge and Agarwal, Saurabh and Roman, Ahmed and Aly, Ahmed A. and Chen, Beidi and Wu, Carole-Jean}, | |
| 118 | + booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL)}, | |
| 119 | + year = {2024}, | |
| 120 | + doi = {10.18653/v1/2024.acl-long.681}, | |
| 121 | + eprint = {2404.16710}, | |
| 122 | + archivePrefix = {arXiv}, | |
| 123 | + url = {https://arxiv.org/abs/2404.16710}, | |
| 124 | +} | |
| 125 | + | |
| 126 | +% --------------------------------------------------------------------------- | |
| 127 | +% 2. Adaptive computation, looped / recurrent-depth, latent reasoning | |
| 128 | +% --------------------------------------------------------------------------- | |
| 129 | + | |
| 130 | +@article{graves2016act, | |
| 131 | + title = {Adaptive Computation Time for Recurrent Neural Networks}, | |
| 132 | + author = {Graves, Alex}, | |
| 133 | + journal = {arXiv preprint arXiv:1603.08983}, | |
| 134 | + year = {2016}, | |
| 135 | + url = {https://arxiv.org/abs/1603.08983}, | |
| 136 | +} | |
| 137 | + | |
| 138 | +@inproceedings{hao2024coconut, | |
| 139 | + title = {Training Large Language Models to Reason in a Continuous Latent Space}, | |
| 140 | + author = {Hao, Shibo and Sukhbaatar, Sainbayar and Su, DiJia and Li, Xian and Hu, Zhiting and Weston, Jason and Tian, Yuandong}, | |
| 141 | + booktitle = {Conference on Language Modeling (COLM)}, | |
| 142 | + year = {2025}, | |
| 143 | + eprint = {2412.06769}, | |
| 144 | + archivePrefix = {arXiv}, | |
| 145 | + url = {https://arxiv.org/abs/2412.06769}, | |
| 146 | +} | |
| 147 | + | |
| 148 | +@article{giannou2023looped, | |
| 149 | + title = {Looped Transformers as Programmable Computers}, | |
| 150 | + author = {Giannou, Angeliki and Rajput, Shashank and Sohn, Jy-yong and Lee, Kangwook and Lee, Jason D. and Papailiopoulos, Dimitris}, | |
| 151 | + journal = {arXiv preprint arXiv:2301.13196}, | |
| 152 | + year = {2023}, | |
| 153 | + url = {https://arxiv.org/abs/2301.13196}, | |
| 154 | +} | |
| 155 | + | |
| 156 | +@inproceedings{saunshi2025reasoning, | |
| 157 | + title = {Reasoning with Latent Thoughts: On the Power of Looped Transformers}, | |
| 158 | + author = {Saunshi, Nikunj and Dikkala, Nishanth and Li, Zhiyuan and Kumar, Sanjiv and Reddi, Sashank J.}, | |
| 159 | + booktitle = {International Conference on Learning Representations (ICLR)}, | |
| 160 | + year = {2025}, | |
| 161 | + eprint = {2502.17416}, | |
| 162 | + archivePrefix = {arXiv}, | |
| 163 | + url = {https://arxiv.org/abs/2502.17416}, | |
| 164 | +} | |
| 165 | + | |
| 166 | +@inproceedings{fan2025loopedlength, | |
| 167 | + title = {Looped Transformers for Length Generalization}, | |
| 168 | + author = {Fan, Ying and Du, Yilun and Ramchandran, Kannan and Lee, Kangwook}, | |
| 169 | + booktitle = {International Conference on Learning Representations (ICLR)}, | |
| 170 | + year = {2025}, | |
| 171 | + eprint = {2409.15647}, | |
| 172 | + archivePrefix = {arXiv}, | |
| 173 | + url = {https://arxiv.org/abs/2409.15647}, | |
| 174 | +} | |
| 175 | + | |
| 176 | +@inproceedings{bae2024relaxed, | |
| 177 | + title = {Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA}, | |
| 178 | + author = {Bae, Sangmin and Fisch, Adam and Harutyunyan, Hrayr and Ji, Ziwei and Kim, Seungyeon and Schuster, Tal}, | |
| 179 | + booktitle = {International Conference on Learning Representations (ICLR)}, | |
| 180 | + year = {2025}, | |
| 181 | + eprint = {2410.20672}, | |
| 182 | + archivePrefix = {arXiv}, | |
| 183 | + url = {https://arxiv.org/abs/2410.20672}, | |
| 184 | +} | |
| 185 | + | |
| 186 | +@inproceedings{goyal2023pause, | |
| 187 | + title = {Think before you speak: Training Language Models With Pause Tokens}, | |
| 188 | + author = {Goyal, Sachin and Ji, Ziwei and Rawat, Ankit Singh and Menon, Aditya Krishna and Kumar, Sanjiv and Nagarajan, Vaishnavh}, | |
| 189 | + booktitle = {International Conference on Learning Representations (ICLR)}, | |
| 190 | + year = {2024}, | |
| 191 | + eprint = {2310.02226}, | |
| 192 | + archivePrefix = {arXiv}, | |
| 193 | + url = {https://arxiv.org/abs/2310.02226}, | |
| 194 | +} | |
| 195 | + | |
| 196 | +@article{sun2025latenttokens, | |
| 197 | + title = {Enhancing Latent Computation in Transformers with Latent Tokens}, | |
| 198 | + author = {Sun, Yuchang and Chen, Yanxi and Li, Yaliang and Ding, Bolin}, | |
| 199 | + journal = {arXiv preprint arXiv:2505.12629}, | |
| 200 | + year = {2025}, | |
| 201 | + url = {https://arxiv.org/abs/2505.12629}, | |
| 202 | +} | |
| 203 | + | |
| 204 | +@inproceedings{zeng2025ponderlm, | |
| 205 | + title = {PonderLM: Pretraining Language Models to Ponder in Continuous Space}, | |
| 206 | + author = {Zeng, Boyi and Song, Shixiang and Huang, Siyuan and Wang, Yixuan and Li, He and He, Ziwei and Wang, Xinbing and Li, Zhiyu and Lin, Zhouhan}, | |
| 207 | + booktitle = {International Conference on Learning Representations (ICLR)}, | |
| 208 | + year = {2026}, | |
| 209 | + eprint = {2505.20674}, | |
| 210 | + archivePrefix = {arXiv}, | |
| 211 | + url = {https://arxiv.org/abs/2505.20674}, | |
| 212 | +} | |
| 213 | + | |
| 214 | +@article{chen2025innerthinking, | |
| 215 | + title = {Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking}, | |
| 216 | + author = {Chen, Yilong and Shang, Junyuan and Zhang, Zhenyu and Xie, Yanxi and Sheng, Jiawei and Liu, Tingwen and Wang, Shuohuan and Sun, Yu and Wu, Hua and Wang, Haifeng}, | |
| 217 | + journal = {arXiv preprint arXiv:2502.13842}, | |
| 218 | + year = {2025}, | |
| 219 | + url = {https://arxiv.org/abs/2502.13842}, | |
| 220 | +} | |
| 221 | + | |
| 222 | +@article{koishekenov2025etd, | |
| 223 | + title = {Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts}, | |
| 224 | + author = {Koishekenov, Yeskendir and Lipani, Aldo and Cancedda, Nicola}, | |
| 225 | + journal = {arXiv preprint arXiv:2510.07358}, | |
| 226 | + year = {2025}, | |
| 227 | + url = {https://arxiv.org/abs/2510.07358}, | |
| 228 | +} | |
| 229 | + | |
| 230 | +@article{wu2025parallelloop, | |
| 231 | + title = {Parallel Loop Transformer for Efficient Test-Time Computation Scaling}, | |
| 232 | + author = {Wu, Bohong and Chen, Mengzhao and Luo, Xiang and Yan, Shen and Yu, Qifan and Xia, Fan and Zhang, Tianqi and Zhan, Hongrui and Zhong, Zheng and Zhou, Xun and Qiao, Siyuan and Bin, Xingyan}, | |
| 233 | + journal = {arXiv preprint arXiv:2510.24824}, | |
| 234 | + year = {2025}, | |
| 235 | + url = {https://arxiv.org/abs/2510.24824}, | |
| 236 | +} | |
| 237 | + | |
| 238 | +@inproceedings{fu2026thinkathard, | |
| 239 | + title = {Think-at-Hard: Dynamic Looped Transformers for Improved Reasoning}, | |
| 240 | + author = {Fu, Tianyu and You, Yichen and Chen, Zekai and Dai, Guohao and Yang, Huazhong and Wang, Yu}, | |
| 241 | + booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306}, | |
| 242 | + pages = {32129--32154}, | |
| 243 | + year = {2026}, | |
| 244 | + eprint = {2511.08577}, | |
| 245 | + archivePrefix = {arXiv}, | |
| 246 | + url = {https://proceedings.mlr.press/v306/fu26t.html}, | |
| 247 | +} | |
| 248 | + | |
| 249 | +@article{you2026tah2, | |
| 250 | + title = {Improving Test-Time Scaling with Adaptive Looped Transformers}, | |
| 251 | + author = {You, Yichen and Fu, Tianyu and Feng, Aosong and Lv, Xingtai and Ning, Xuefei and Ding, Ning and Wang, Yu}, | |
| 252 | + journal = {arXiv preprint arXiv:2609.35748}, | |
| 253 | + year = {2026}, | |
| 254 | + url = {https://arxiv.org/abs/2609.35748}, | |
| 255 | +} | |
| 256 | + | |
| 257 | +@inproceedings{jeddi2026loopformer, | |
| 258 | + title = {LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation}, | |
| 259 | + author = {Jeddi, Ahmadreza and Ciccone, Marco and Taati, Babak}, | |
| 260 | + booktitle = {International Conference on Learning Representations (ICLR)}, | |
| 261 | + year = {2026}, | |
| 262 | + eprint = {2602.11451}, | |
| 263 | + archivePrefix = {arXiv}, | |
| 264 | + url = {https://arxiv.org/abs/2602.11451}, | |
| 265 | +} | |
| 266 | + | |
| 267 | +@article{song2026adaponderlm, | |
| 268 | + title = {AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth}, | |
| 269 | + author = {Song, Shixiang and Li, He and Wang, Zitong and Zeng, Boyi and Song, Feichen and Wang, Yixuan and Xu, Zhiqin John and He, Ziwei and Lin, Zhouhan}, | |
| 270 | + journal = {arXiv preprint arXiv:2603.01914}, | |
| 271 | + year = {2026}, | |
| 272 | + url = {https://arxiv.org/abs/2603.01914}, | |
| 273 | +} | |
| 274 | + | |
| 275 | +@article{noci2026lookahead, | |
| 276 | + title = {Thinking into the Future: Latent Lookahead Training for Transformers}, | |
| 277 | + author = {Noci, Lorenzo and Bachmann, Gregor and Moosavi-Dezfooli, Seyed-Mohsen and Nabi, Moin}, | |
| 278 | + journal = {arXiv preprint arXiv:2603.20219}, | |
| 279 | + year = {2026}, | |
| 280 | + url = {https://arxiv.org/abs/2603.20219}, | |
| 281 | +} | |
| 282 | + | |
| 283 | +@inproceedings{frey2026adaptiveloops, | |
| 284 | + title = {Adaptive Loops and Memory in Transformers: Think Harder or Know More?}, | |
| 285 | + author = {Frey, Markus and Shomali, Behzad and Bashir, Ali Hamza and Berghaus, David and Koehler, Joachim and Ali, Mehdi}, | |
| 286 | + booktitle = {Latent \& Implicit Thinking Workshop @ ICLR 2026}, | |
| 287 | + year = {2026}, | |
| 288 | + eprint = {2603.08391}, | |
| 289 | + archivePrefix = {arXiv}, | |
| 290 | + url = {https://arxiv.org/abs/2603.08391}, | |
| 291 | +} | |
| 292 | + | |
| 293 | +@article{park2026loopus, | |
| 294 | + title = {LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models}, | |
| 295 | + author = {Park, Taekhyun and Lee, Yongjae and Kim, Dohee and Bae, Hyerim}, | |
| 296 | + journal = {arXiv preprint arXiv:2605.11011}, | |
| 297 | + year = {2026}, | |
| 298 | + url = {https://arxiv.org/abs/2605.11011}, | |
| 299 | +} | |
| 300 | + | |
| 301 | +@article{huang2026lrt, | |
| 302 | + title = {Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior}, | |
| 303 | + author = {Huang, Zeyi and He, Xuehai and Ren, LiLiang and Wang, Yiping and Peng, Baolin and Cheng, Hao and Wang, Shuohang and He, Pengcheng and Gao, Jianfeng and Lee, Yong Jae and Shen, Yelong}, | |
| 304 | + journal = {arXiv preprint arXiv:2605.26797}, | |
| 305 | + year = {2026}, | |
| 306 | + url = {https://arxiv.org/abs/2605.26797}, | |
| 307 | +} | |
| 308 | + | |
| 309 | +@article{li2026later, | |
| 310 | + title = {LaTER: Efficient Test-Time Reasoning via Latent Exploration and Explicit Verification}, | |
| 311 | + author = {Li, Xuan and Wang, Yining and Liu, Yuchen and Liu, Guanjun and Qiu, Delai and Liu, Shengping and Liang, Jiaen and Huang, Wei and Yu, Jun and Zhu, Junnan}, | |
| 312 | + journal = {arXiv preprint arXiv:2605.07315}, | |
| 313 | + year = {2026}, | |
| 314 | + url = {https://arxiv.org/abs/2605.07315}, | |
| 315 | +} | |
| 316 | + | |
| 317 | +@article{sharma2026readout, | |
| 318 | + title = {Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models}, | |
| 319 | + author = {Sharma, Rituraj and Vu, Tu}, | |
| 320 | + journal = {arXiv preprint arXiv:2606.24898}, | |
| 321 | + year = {2026}, | |
| 322 | + url = {https://arxiv.org/abs/2606.24898}, | |
| 323 | +} | |
| 324 | + | |
| 325 | +@article{zeng2026depthattention, | |
| 326 | + title = {Depth-Attention: Cross-Layer Value Mixing for Language Models}, | |
| 327 | + author = {Zeng, Boyi and Hao, Yiqin and Wang, Zitong and Song, Shixiang and Li, He and Song, Feichen and Liu, Yifan and He, Ziwei and Wang, Xinbing and Lin, Zhouhan}, | |
| 328 | + journal = {arXiv preprint arXiv:2606.05014}, | |
| 329 | + year = {2026}, | |
| 330 | + url = {https://arxiv.org/abs/2606.05014}, | |
| 331 | +} | |
| 332 | + | |
| 333 | +@article{fu2026discoloop, | |
| 334 | + title = {DiscoLoop: Looping Discrete Embeddings and Continuous Hidden States for Multi-hop Reasoning}, | |
| 335 | + author = {Fu, Hengyu and Guo, Tianyu and Wang, Zixuan and Zhu, Hanlin and Lee, Jason D. and Jiao, Jiantao and Russell, Stuart and Mei, Song}, | |
| 336 | + journal = {arXiv preprint arXiv:2607.00341}, | |
| 337 | + year = {2026}, | |
| 338 | + url = {https://arxiv.org/abs/2607.00341}, | |
| 339 | +} | |
| 340 | + | |
| 341 | +@article{popescu2026adaptivedepth, | |
| 342 | + title = {Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readouts}, | |
| 343 | + author = {Popescu, Andrei Cristian and S{\'a}ez de Oc{\'a}riz Borde, Haitz and Li{\`o}, Pietro}, | |
| 344 | + journal = {arXiv preprint arXiv:2607.20519}, | |
| 345 | + year = {2026}, | |
| 346 | + url = {https://arxiv.org/abs/2607.20519}, | |
| 347 | +} | |
| 348 | + | |
| 349 | +@article{logan2026fixedpoint, | |
| 350 | + title = {Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers}, | |
| 351 | + author = {Logan, Joe}, | |
| 352 | + journal = {arXiv preprint arXiv:2607.14427}, | |
| 353 | + year = {2026}, | |
| 354 | + url = {https://arxiv.org/abs/2607.14427}, | |
| 355 | +} | |
| 356 | + | |
| 357 | +@article{dau2026beyonddepth, | |
| 358 | + title = {Beyond Depth Truncation: Controlled Evaluation of Depth Utilization in Recursive Language Models}, | |
| 359 | + author = {Dau, Ha Van and Khuat, Thanh Tung and Dung, Nguyen Thanh}, | |
| 360 | + journal = {arXiv preprint arXiv:2609.19934}, | |
| 361 | + year = {2026}, | |
| 362 | + url = {https://arxiv.org/abs/2609.19934}, | |
| 363 | +} | |
| 364 | + | |
| 365 | +@article{huang2026trading, | |
| 366 | + title = {Trading Depth for Time in Recurrent Transformers}, | |
| 367 | + author = {Huang, Zeyi and He, Xuehai and Shen, Yelong and Lee, Yong Jae}, | |
| 368 | + journal = {arXiv preprint arXiv:2609.21605}, | |
| 369 | + year = {2026}, | |
| 370 | + url = {https://arxiv.org/abs/2609.21605}, | |
| 371 | +} | |
| 372 | + | |
| 373 | +@article{ye2026reflux, | |
| 374 | + title = {Write Back the $\Delta$: Revisiting the Same Tokens with Fresh Representations}, | |
| 375 | + author = {Ye, Wencheng and Hu, Anning and Zhang, Xiangdong and Wang, Tianyi and Li, Yikang and Jin, Hengyu and Li, Bing and Yan, Junchi}, | |
| 376 | + journal = {arXiv preprint arXiv:2609.32457}, | |
| 377 | + year = {2026}, | |
| 378 | + url = {https://arxiv.org/abs/2609.32457}, | |
| 379 | +} | |
| 380 | + | |
| 381 | +@article{olszowiec2026monitoring, | |
| 382 | + title = {Monitoring latent reasoning in looped language models}, | |
| 383 | + author = {Olszowiec, Pawel and Tomaszewska, Paulina and {\.Z}elaszczyk, Maciej and Byra, Michal and Chung, Neo Christopher}, | |
| 384 | + journal = {arXiv preprint arXiv:2610.18127}, | |
| 385 | + year = {2026}, | |
| 386 | + url = {https://arxiv.org/abs/2610.18127}, | |
| 387 | +} | |
| 388 | + | |
| 389 | +% --------------------------------------------------------------------------- | |
| 390 | +% 3. Slots, workspaces, memory-augmented networks | |
| 391 | +% --------------------------------------------------------------------------- | |
| 392 | + | |
| 393 | +@inproceedings{locatello2020slot, | |
| 394 | + title = {Object-Centric Learning with Slot Attention}, | |
| 395 | + author = {Locatello, Francesco and Weissenborn, Dirk and Unterthiner, Thomas and Mahendran, Aravindh and Heigold, Georg and Uszkoreit, Jakob and Dosovitskiy, Alexey and Kipf, Thomas}, | |
| 396 | + booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, | |
| 397 | + year = {2020}, | |
| 398 | + eprint = {2006.15055}, | |
| 399 | + archivePrefix = {arXiv}, | |
| 400 | + url = {https://arxiv.org/abs/2006.15055}, | |
| 401 | +} | |
| 402 | + | |
| 403 | +@inproceedings{goyal2022workspace, | |
| 404 | + title = {Coordination Among Neural Modules Through a Shared Global Workspace}, | |
| 405 | + author = {Goyal, Anirudh and Didolkar, Aniket and Lamb, Alex and Badola, Kartikeya and Ke, Nan Rosemary and Rahaman, Nasim and Binas, Jonathan and Blundell, Charles and Mozer, Michael and Bengio, Yoshua}, | |
| 406 | + booktitle = {International Conference on Learning Representations (ICLR)}, | |
| 407 | + year = {2022}, | |
| 408 | + eprint = {2103.01197}, | |
| 409 | + archivePrefix = {arXiv}, | |
| 410 | + url = {https://arxiv.org/abs/2103.01197}, | |
| 411 | +} | |
| 412 | + | |
| 413 | +@inproceedings{jaegle2021perceiver, | |
| 414 | + title = {Perceiver: General Perception with Iterative Attention}, | |
| 415 | + author = {Jaegle, Andrew and Gimeno, Felix and Brock, Andrew and Zisserman, Andrew and Vinyals, Oriol and Carreira, Joao}, | |
| 416 | + booktitle = {International Conference on Machine Learning (ICML)}, | |
| 417 | + year = {2021}, | |
| 418 | + eprint = {2103.03206}, | |
| 419 | + archivePrefix = {arXiv}, | |
| 420 | + url = {https://arxiv.org/abs/2103.03206}, | |
| 421 | +} | |
| 422 | + | |
| 423 | +@article{graves2014ntm, | |
| 424 | + title = {Neural Turing Machines}, | |
| 425 | + author = {Graves, Alex and Wayne, Greg and Danihelka, Ivo}, | |
| 426 | + journal = {arXiv preprint arXiv:1410.5401}, | |
| 427 | + year = {2014}, | |
| 428 | + url = {https://arxiv.org/abs/1410.5401}, | |
| 429 | +} | |
| 430 | + | |
| 431 | +@article{weston2014memory, | |
| 432 | + title = {Memory Networks}, | |
| 433 | + author = {Weston, Jason and Chopra, Sumit and Bordes, Antoine}, | |
| 434 | + journal = {arXiv preprint arXiv:1410.3916}, | |
| 435 | + year = {2014}, | |
| 436 | + url = {https://arxiv.org/abs/1410.3916}, | |
| 437 | +} | |
| 438 | + | |
| 439 | +@inproceedings{rae2016sparse, | |
| 440 | + title = {Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes}, | |
| 441 | + author = {Rae, Jack W. and Hunt, Jonathan J. and Harley, Tim and Danihelka, Ivo and Senior, Andrew and Wayne, Greg and Graves, Alex and Lillicrap, Timothy P.}, | |
| 442 | + booktitle = {Advances in Neural Information Processing Systems (NIPS)}, | |
| 443 | + year = {2016}, | |
| 444 | + eprint = {1610.09027}, | |
| 445 | + archivePrefix = {arXiv}, | |
| 446 | + url = {https://arxiv.org/abs/1610.09027}, | |
| 447 | +} | |
| 448 | + | |
| 449 | +@article{wu2020memformer, | |
| 450 | + title = {Memformer: A Memory-Augmented Transformer for Sequence Modeling}, | |
| 451 | + author = {Wu, Qingyang and Lan, Zhenzhong and Qian, Kun and Gu, Jing and Geramifard, Alborz and Yu, Zhou}, | |
| 452 | + journal = {arXiv preprint arXiv:2010.06891}, | |
| 453 | + year = {2020}, | |
| 454 | + url = {https://arxiv.org/abs/2010.06891}, | |
| 455 | +} | |
| 456 | + | |
| 457 | +@inproceedings{bulatov2022rmt, | |
| 458 | + title = {Recurrent Memory Transformer}, | |
| 459 | + author = {Bulatov, Aydar and Kuratov, Yuri and Burtsev, Mikhail S.}, | |
| 460 | + booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, | |
| 461 | + year = {2022}, | |
| 462 | + eprint = {2207.06881}, | |
| 463 | + archivePrefix = {arXiv}, | |
| 464 | + url = {https://arxiv.org/abs/2207.06881}, | |
| 465 | +} | |
| 466 | + | |
| 467 | +@inproceedings{ryoo2023ttm, | |
| 468 | + title = {Token Turing Machines}, | |
| 469 | + author = {Ryoo, Michael S. and Gopalakrishnan, Keerthana and Kahatapitiya, Kumara and Xiao, Ted and Rao, Kanishka and Stone, Austin and Lu, Yao and Ibarz, Julian and Arnab, Anurag}, | |
| 470 | + booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, | |
| 471 | + year = {2023}, | |
| 472 | + eprint = {2211.09119}, | |
| 473 | + archivePrefix = {arXiv}, | |
| 474 | + url = {https://arxiv.org/abs/2211.09119}, | |
| 475 | +} | |
| 476 | + | |
| 477 | +@article{berges2024memorylayers, | |
| 478 | + title = {Memory Layers at Scale}, | |
| 479 | + author = {Berges, Vincent-Pierre and O{\u{g}}uz, Barlas and Haziza, Daniel and Yih, Wen-tau and Zettlemoyer, Luke and Ghosh, Gargi}, | |
| 480 | + journal = {arXiv preprint arXiv:2412.09764}, | |
| 481 | + year = {2024}, | |
| 482 | + url = {https://arxiv.org/abs/2412.09764}, | |
| 483 | +} | |
| 484 | + | |
| 485 | +@article{darcet2023registers, | |
| 486 | + title = {Vision Transformers Need Registers}, | |
| 487 | + author = {Darcet, Timoth{\'e}e and Oquab, Maxime and Mairal, Julien and Bojanowski, Piotr}, | |
| 488 | + journal = {arXiv preprint arXiv:2309.16588}, | |
| 489 | + year = {2023}, | |
| 490 | + url = {https://arxiv.org/abs/2309.16588}, | |
| 491 | +} | |
| 492 | + | |
| 493 | +% --------------------------------------------------------------------------- | |
| 494 | +% 4. Long-context / recurrence / linear-time baselines | |
| 495 | +% --------------------------------------------------------------------------- | |
| 496 | + | |
| 497 | +@inproceedings{dai2019transformerxl, | |
| 498 | + title = {Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context}, | |
| 499 | + author = {Dai, Zihang and Yang, Zhilin and Yang, Yiming and Carbonell, Jaime and Le, Quoc V. and Salakhutdinov, Ruslan}, | |
| 500 | + booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL)}, | |
| 501 | + year = {2019}, | |
| 502 | + eprint = {1901.02860}, | |
| 503 | + archivePrefix = {arXiv}, | |
| 504 | + url = {https://arxiv.org/abs/1901.02860}, | |
| 505 | +} | |
| 506 | + | |
| 507 | +@article{rae2019compressive, | |
| 508 | + title = {Compressive Transformers for Long-Range Sequence Modelling}, | |
| 509 | + author = {Rae, Jack W. and Potapenko, Anna and Jayakumar, Siddhant M. and Lillicrap, Timothy P.}, | |
| 510 | + journal = {arXiv preprint arXiv:1911.05507}, | |
| 511 | + year = {2019}, | |
| 512 | + url = {https://arxiv.org/abs/1911.05507}, | |
| 513 | +} | |
| 514 | + | |
| 515 | +@article{gu2023mamba, | |
| 516 | + title = {Mamba: Linear-Time Sequence Modeling with Selective State Spaces}, | |
| 517 | + author = {Gu, Albert and Dao, Tri}, | |
| 518 | + journal = {arXiv preprint arXiv:2312.00752}, | |
| 519 | + year = {2023}, | |
| 520 | + url = {https://arxiv.org/abs/2312.00752}, | |
| 521 | +} | |
| 522 | + | |
| 523 | +% --------------------------------------------------------------------------- | |
| 524 | +% 5. Sparse routing / mixture-of-experts | |
| 525 | +% --------------------------------------------------------------------------- | |
| 526 | + | |
| 527 | +@article{shazeer2017moe, | |
| 528 | + title = {Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer}, | |
| 529 | + author = {Shazeer, Noam and Mirhoseini, Azalia and Maziarz, Krzysztof and Davis, Andy and Le, Quoc and Hinton, Geoffrey and Dean, Jeff}, | |
| 530 | + journal = {arXiv preprint arXiv:1701.06538}, | |
| 531 | + year = {2017}, | |
| 532 | + url = {https://arxiv.org/abs/1701.06538}, | |
| 533 | +} | |
| 534 | + | |
| 535 | +@article{fedus2022switch, | |
| 536 | + title = {Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity}, | |
| 537 | + author = {Fedus, William and Zoph, Barret and Shazeer, Noam}, | |
| 538 | + journal = {Journal of Machine Learning Research}, | |
| 539 | + year = {2022}, | |
| 540 | + eprint = {2101.03961}, | |
| 541 | + archivePrefix = {arXiv}, | |
| 542 | + url = {https://arxiv.org/abs/2101.03961}, | |
| 543 | +} | |
| 544 | + | |
| 545 | +% --------------------------------------------------------------------------- | |
| 546 | +% 6. Value learning, off-policy / counterfactual evaluation | |
| 547 | +% --------------------------------------------------------------------------- | |
| 548 | + | |
| 549 | +@article{ernst2005tree, | |
| 550 | + title = {Tree-Based Batch Mode Reinforcement Learning}, | |
| 551 | + author = {Ernst, Damien and Geurts, Pierre and Wehenkel, Louis}, | |
| 552 | + journal = {Journal of Machine Learning Research}, | |
| 553 | + volume = {6}, | |
| 554 | + pages = {503--556}, | |
| 555 | + year = {2005}, | |
| 556 | + url = {https://www.jmlr.org/papers/v6/ernst05a.html}, | |
| 557 | +} | |
| 558 | + | |
| 559 | +@inproceedings{mnih2013atari, | |
| 560 | + title = {Playing Atari with Deep Reinforcement Learning}, | |
| 561 | + author = {Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Graves, Alex and Antonoglou, Ioannis and Wierstra, Daan and Riedmiller, Martin}, | |
| 562 | + booktitle = {NIPS Deep Learning Workshop}, | |
| 563 | + year = {2013}, | |
| 564 | + eprint = {1312.5602}, | |
| 565 | + archivePrefix = {arXiv}, | |
| 566 | + url = {https://arxiv.org/abs/1312.5602}, | |
| 567 | +} | |
| 568 | + | |
| 569 | +@inproceedings{dudik2011doublyrobust, | |
| 570 | + title = {Doubly Robust Policy Evaluation and Learning}, | |
| 571 | + author = {Dud{\'i}k, Miroslav and Langford, John and Li, Lihong}, | |
| 572 | + booktitle = {International Conference on Machine Learning (ICML)}, | |
| 573 | + year = {2011}, | |
| 574 | + eprint = {1103.4601}, | |
| 575 | + archivePrefix = {arXiv}, | |
| 576 | + url = {https://arxiv.org/abs/1103.4601}, | |
| 577 | +} | |
| 578 | + | |
| 579 | +@article{bottou2013counterfactual, | |
| 580 | + title = {Counterfactual Reasoning and Learning Systems}, | |
| 581 | + author = {Bottou, L{\'e}on and Peters, Jonas and Qui{\~n}onero-Candela, Joaquin and Charles, Denis X. and Chickering, D. Max and Portugaly, Elon and Ray, Dipankar and Simard, Patrice and Snelson, Ed}, | |
| 582 | + journal = {arXiv preprint arXiv:1209.2355}, | |
| 583 | + year = {2012}, | |
| 584 | + url = {https://arxiv.org/abs/1209.2355}, | |
| 585 | +} | |
| 586 | + | |
| 587 | +@article{swaminathan2015crm, | |
| 588 | + title = {Counterfactual Risk Minimization: Learning from Logged Bandit Feedback}, | |
| 589 | + author = {Swaminathan, Adith and Joachims, Thorsten}, | |
| 590 | + journal = {arXiv preprint arXiv:1502.02362}, | |
| 591 | + year = {2015}, | |
| 592 | + url = {https://arxiv.org/abs/1502.02362}, | |
| 593 | +} | |
| 594 | + | |
| 595 | +% --------------------------------------------------------------------------- | |
| 596 | +% 7. Early exit, speculative decoding, test-time compute | |
| 597 | +% --------------------------------------------------------------------------- | |
| 598 | + | |
| 599 | +@inproceedings{schuster2022calm, | |
| 600 | + title = {Confident Adaptive Language Modeling}, | |
| 601 | + author = {Schuster, Tal and Fisch, Adam and Gupta, Jai and Dehghani, Mostafa and Bahri, Dara and Tran, Vinh Q. and Tay, Yi and Metzler, Donald}, | |
| 602 | + booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, | |
| 603 | + year = {2022}, | |
| 604 | + eprint = {2207.07061}, | |
| 605 | + archivePrefix = {arXiv}, | |
| 606 | + url = {https://arxiv.org/abs/2207.07061}, | |
| 607 | +} | |
| 608 | + | |
| 609 | +@inproceedings{leviathan2023speculative, | |
| 610 | + title = {Fast Inference from Transformers via Speculative Decoding}, | |
| 611 | + author = {Leviathan, Yaniv and Kalman, Matan and Matias, Yossi}, | |
| 612 | + booktitle = {International Conference on Machine Learning (ICML)}, | |
| 613 | + year = {2023}, | |
| 614 | + eprint = {2211.17192}, | |
| 615 | + archivePrefix = {arXiv}, | |
| 616 | + url = {https://arxiv.org/abs/2211.17192}, | |
| 617 | +} | |
| 618 | + | |
| 619 | +@article{snell2024testtime, | |
| 620 | + title = {Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters}, | |
| 621 | + author = {Snell, Charlie and Lee, Jaehoon and Xu, Kelvin and Kumar, Aviral}, | |
| 622 | + journal = {arXiv preprint arXiv:2408.03314}, | |
| 623 | + year = {2024}, | |
| 624 | + url = {https://arxiv.org/abs/2408.03314}, | |
| 625 | +} | |
| 626 | + | |
| 627 | +@article{ye2024physics, | |
| 628 | + title = {Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process}, | |
| 629 | + author = {Ye, Tian and Xu, Zicheng and Li, Yuanzhi and Allen-Zhu, Zeyuan}, | |
| 630 | + journal = {arXiv preprint arXiv:2407.20311}, | |
| 631 | + year = {2024}, | |
| 632 | + url = {https://arxiv.org/abs/2407.20311}, | |
| 633 | +} | |
| 634 | + | |
| 635 | +% --------------------------------------------------------------------------- | |
| 636 | +% 8. Zero-initialised gates / exact-identity insertion (for C3) | |
| 637 | +% --------------------------------------------------------------------------- | |
| 638 | + | |
| 639 | +@article{bachlechner2020rezero, | |
| 640 | + title = {ReZero is All You Need: Fast Convergence at Large Depth}, | |
| 641 | + author = {Bachlechner, Thomas and Majumder, Bodhisattwa Prasad and Mao, Huanru Henry and Cottrell, Garrison W. and McAuley, Julian}, | |
| 642 | + journal = {arXiv preprint arXiv:2003.04887}, | |
| 643 | + year = {2020}, | |
| 644 | + url = {https://arxiv.org/abs/2003.04887}, | |
| 645 | +} | |
| 646 | + | |
| 647 | +@article{zhang2023controlnet, | |
| 648 | + title = {Adding Conditional Control to Text-to-Image Diffusion Models}, | |
| 649 | + author = {Zhang, Lvmin and Rao, Anyi and Agrawala, Maneesh}, | |
| 650 | + journal = {arXiv preprint arXiv:2302.05543}, | |
| 651 | + year = {2023}, | |
| 652 | + url = {https://arxiv.org/abs/2302.05543}, | |
| 653 | +} | |
| 654 | + | |
| 655 | +@inproceedings{alayrac2022flamingo, | |
| 656 | + title = {Flamingo: a Visual Language Model for Few-Shot Learning}, | |
| 657 | + author = {Alayrac, Jean-Baptiste and Donahue, Jeff and Luc, Pauline and Miech, Antoine and Barr, Iain and Hasson, Yana and Lenc, Karel and Mensch, Arthur and Millican, Katie and Reynolds, Malcolm and Ring, Roman and Rutherford, Eliza and Cabi, Serkan and Han, Tengda and Gong, Zhitao and Samangooei, Sina and Monteiro, Marianne and Menick, Jacob and Borgeaud, Sebastian and Brock, Andrew and Nematzadeh, Aida and Sharifzadeh, Sahand and Binkowski, Mikolaj and Barreira, Ricardo and Vinyals, Oriol and Zisserman, Andrew and Simonyan, Karen}, | |
| 658 | + booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, | |
| 659 | + year = {2022}, | |
| 660 | + eprint = {2204.14198}, | |
| 661 | + archivePrefix = {arXiv}, | |
| 662 | + url = {https://arxiv.org/abs/2204.14198}, | |
| 663 | +} | |
| 664 | + | |
| 665 | +% --------------------------------------------------------------------------- | |
| 666 | +% 9. Interpretability-style sparse interventions (for C1/C2 context) | |
| 667 | +% --------------------------------------------------------------------------- | |
| 668 | + | |
| 669 | +@article{ernst2026dsi, | |
| 670 | + title = {Distributed Sparse Interventions in Language Models}, | |
| 671 | + author = {Ernst, Maximilian S. and Linhardt, Lorenz and Peikert, Aaron and Eberle, Oliver}, | |
| 672 | + journal = {arXiv preprint arXiv:2607.07128}, | |
| 673 | + year = {2026}, | |
| 674 | + url = {https://arxiv.org/abs/2607.07128}, | |
| 675 | +} | |
| 676 | + | |
| 677 | +% --------------------------------------------------------------------------- | |
| 678 | +% 10. Backbone building blocks and scaling laws | |
| 679 | +% --------------------------------------------------------------------------- | |
| 680 | + | |
| 681 | +@article{shazeer2020glu, | |
| 682 | + title = {{GLU} Variants Improve Transformer}, | |
| 683 | + author = {Shazeer, Noam}, | |
| 684 | + journal = {arXiv preprint arXiv:2002.05202}, | |
| 685 | + year = {2020}, | |
| 686 | + url = {https://arxiv.org/abs/2002.05202}, | |
| 687 | +} | |
| 688 | + | |
| 689 | +@article{su2021roformer, | |
| 690 | + title = {RoFormer: Enhanced Transformer with Rotary Position Embedding}, | |
| 691 | + author = {Su, Jianlin and Lu, Yu and Pan, Shengfeng and Murtadha, Ahmed and Wen, Bo and Liu, Yunfeng}, | |
| 692 | + journal = {arXiv preprint arXiv:2104.09864}, | |
| 693 | + year = {2021}, | |
| 694 | + url = {https://arxiv.org/abs/2104.09864}, | |
| 695 | +} | |
| 696 | + | |
| 697 | +@inproceedings{ainslie2023gqa, | |
| 698 | + title = {{GQA}: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints}, | |
| 699 | + author = {Ainslie, Joshua and Lee-Thorp, James and de Jong, Michiel and Zemlyanskiy, Yury and Lebr{\'o}n, Federico and Sanghai, Sumit}, | |
| 700 | + booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP)}, | |
| 701 | + year = {2023}, | |
| 702 | + eprint = {2305.13245}, | |
| 703 | + archivePrefix = {arXiv}, | |
| 704 | + url = {https://arxiv.org/abs/2305.13245}, | |
| 705 | +} | |
| 706 | + | |
| 707 | +@inproceedings{zhang2019rmsnorm, | |
| 708 | + title = {Root Mean Square Layer Normalization}, | |
| 709 | + author = {Zhang, Biao and Sennrich, Rico}, | |
| 710 | + booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, | |
| 711 | + year = {2019}, | |
| 712 | + eprint = {1910.07467}, | |
| 713 | + archivePrefix = {arXiv}, | |
| 714 | + url = {https://arxiv.org/abs/1910.07467}, | |
| 715 | +} | |
| 716 | + | |
| 717 | +@article{kaplan2020scaling, | |
| 718 | + title = {Scaling Laws for Neural Language Models}, | |
| 719 | + author = {Kaplan, Jared and McCandlish, Sam and Henighan, Tom and Brown, Tom B. and Chess, Benjamin and Child, Rewon and Gray, Scott and Radford, Alec and Wu, Jeffrey and Amodei, Dario}, | |
| 720 | + journal = {arXiv preprint arXiv:2001.08361}, | |
| 721 | + year = {2020}, | |
| 722 | + url = {https://arxiv.org/abs/2001.08361}, | |
| 723 | +} | |
| 724 | + | |
| 725 | +@article{hoffmann2022chinchilla, | |
| 726 | + title = {Training Compute-Optimal Large Language Models}, | |
| 727 | + author = {Hoffmann, Jordan and Borgeaud, Sebastian and Mensch, Arthur and Buchatskaya, Elena and Cai, Trevor and Rutherford, Eliza and de Las Casas, Diego and Hendricks, Lisa Anne and Welbl, Johannes and Clark, Aidan and Hennigan, Tom and Noland, Eric and Millican, Katie and van den Driessche, George and Damoc, Bogdan and Guy, Aurelia and Osindero, Simon and Simonyan, Karen and Elsen, Erich and Rae, Jack W. and Vinyals, Oriol and Sifre, Laurent}, | |
| 728 | + journal = {arXiv preprint arXiv:2203.15556}, | |
| 729 | + year = {2022}, | |
| 730 | + url = {https://arxiv.org/abs/2203.15556}, | |
| 731 | +} | |
| 732 | + | |
| 733 | +@article{penedo2024fineweb, | |
| 734 | + title = {The {FineWeb} Datasets: Decanting the Web for the Finest Text Data at Scale}, | |
| 735 | + author = {Penedo, Guilherme and Kydl{\'\i}{\v{c}}ek, Hynek and Ben allal, Loubna and Lozhkov, Anton and Mitchell, Margaret and Raffel, Colin and Von Werra, Leandro and Wolf, Thomas}, | |
| 736 | + year = {2024}, | |
| 737 | + journal = {arXiv preprint arXiv:2406.17557}, | |
| 738 | + url = {https://arxiv.org/abs/2406.17557}, | |
| 739 | +} | |
added
paper/sections/abstract.tex
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +% to be written | |
added
paper/sections/architecture.tex
+97 −0
@@ -0,0 +1,97 @@ | ||
| 1 | +\section{The \prisme{} architecture}\label{sec:architecture} | |
| 2 | + | |
| 3 | +\prisme{} (Predictive Routing with Intervention-Supervised Memory Editing) adds to a standard decoder a read-only | |
| 4 | +\emph{source bank}, an ephemeral $M$-slot \emph{workspace}, a shared \emph{sparse editor}, a \emph{value controller} | |
| 5 | +and a \emph{differential readout}. Figure~\ref{fig:architecture} gives the data flow; Table~\ref{tab:configs} | |
| 6 | +the three dimensioned configurations. All notation is collected in Appendix~\ref{app:notation}. | |
| 7 | + | |
| 8 | +\begin{figure}[t]\centering | |
| 9 | +\input{figures/architecture.tex} | |
| 10 | +\caption{\prisme{} data flow for one position $t$. Solid arrows are read paths, the dashed arrow is the only write | |
| 11 | +path (into 1--2 of the $M$ slots per action). The backbone residual stream $H$, the source bank $(K,V)$ and the | |
| 12 | +summary $z_t$ are inputs to the editor and are never modified by it; the workspace is discarded after the token. | |
| 13 | +With zero executed edits the output is bitwise the backbone output (explicit bypass).} | |
| 14 | +\label{fig:architecture} | |
| 15 | +\end{figure} | |
| 16 | + | |
| 17 | +\subsection{Backbone and source bank}\label{sec:backbone} | |
| 18 | +The backbone $B_\phi$ is a pre-norm causal decoder with RMSNorm \citep{zhang2019rmsnorm}, rotary position | |
| 19 | +embeddings \citep{su2021roformer}, grouped-query attention \citep{ainslie2023gqa}, SwiGLU feed-forward blocks | |
| 20 | +\citep{shazeer2020glu}, tied input/output embeddings and one final normalisation. It returns the final residual | |
| 21 | +stream $H_t=B_\phi(x_{1:t})\in\R^{t\times d}$, with $h_t=H_t[t]$, and its own head is | |
| 22 | +$\ell^{\mathrm{bb}}_t = E\,\mathrm{RMSNorm}(h_t)$. The \emph{source bank} is a projection of the history to the | |
| 23 | +workspace width $w\ll d$: | |
| 24 | +\begin{equation} | |
| 25 | +\hat H_t=\mathrm{RMSNorm}_{\mathrm{bank}}(H_t),\qquad K_t=\hat H_t W_K,\qquad V_t=\hat H_t W_V,\qquad | |
| 26 | +z_t = P\,\hat h_t,\qquad W_K,W_V,P^{\!\top}\in\R^{d\times w}. | |
| 27 | +\label{eq:bank} | |
| 28 | +\end{equation} | |
| 29 | +The bank is distinct from the Transformer KV cache: it stores projected \emph{final} representations, it is | |
| 30 | +append-only, and a query at position $t$ may read rows $1,\dots,t$ only. | |
| 31 | + | |
| 32 | +\subsection{Ephemeral workspace} | |
| 33 | +For every position a fresh workspace $S^{(0)}\in\R^{M\times w}$ is formed from $M$ trainable address embeddings | |
| 34 | +$e_j$ and the current summary, | |
| 35 | +\begin{equation} | |
| 36 | +S^{(0)}_j = e_j + z_t,\qquad j=0,\dots,M-1 . | |
| 37 | +\label{eq:init} | |
| 38 | +\end{equation} | |
| 39 | +$e_j$ carries no imposed semantic role; it only makes slots distinguishable. No state of $S$ survives to position | |
| 40 | +$t+1$; history is carried exclusively by the autoregressive prefix and the two caches. | |
| 41 | + | |
| 42 | +\subsection{Action catalogue}\label{sec:actions} | |
| 43 | +The nonterminal actions are $\textsc{single}_j$ ($j<M$, edit one slot) and $\textsc{pair}_{2i,2i+1}$ | |
| 44 | +($i<M/2$, edit two fixed slots), plus \Stop. For $M=16$ this gives $|\Act|=24$ nonterminal actions and 25 in | |
| 45 | +total; the $\binom{16}{2}=120$ arbitrary pairs are \emph{not} available. We write $I(a)\subset\{0,\dots,M-1\}$ for | |
| 46 | +the written index set of action $a$ ($|I(a)|=k(a)\in\{1,2\}$, $I(\Stop)=\varnothing$). \Stop{} is absorbing and a | |
| 47 | +hard cap $R_{\max}=8$ bounds the number of actions per position. | |
| 48 | + | |
| 49 | +\subsection{Shared sparse editor}\label{sec:editor} | |
| 50 | +Given the current workspace $S$, an action $a$ with active set $I=I(a)$, the step index $r$ and the remaining budget | |
| 51 | +$\rho=R_{\max}-r-1$, the editor forms the active queries | |
| 52 | +$X = S_{I} + \mathrm{ctx}([r/R_{\max},\;\rho/R_{\max},\;k/2])\in\R^{k\times w}$ and applies $L_w=2$ \emph{shared-weight} | |
| 53 | +blocks. Each block computes, with pre-normalisation and residual connections, | |
| 54 | +\begin{align} | |
| 55 | +X &\leftarrow X + \mathrm{MHA}\big(q=X,\;k=K_t,\;v=V_t;\;\text{mask}_{\le t}\big) | |
| 56 | + && \text{(read the causal source bank)}\label{eq:blk1}\\ | |
| 57 | +X &\leftarrow X + \mathrm{MHA}\big(q=X,\;k=\bar S,\;v=\bar S\big) | |
| 58 | + && \text{(read all $M$ slots of the pre-write snapshot $\bar S=S$)}\label{eq:blk2}\\ | |
| 59 | +X &\leftarrow X + \mathrm{SwiGLU}(X). && \label{eq:blk3} | |
| 60 | +\end{align} | |
| 61 | +The bank attention uses query and output projections only (the bank is already in width $w$ and is cached), so the | |
| 62 | +bank is never re-projected per action. The slot attention re-projects the $M$ slot keys/values at every block | |
| 63 | +(reference implementation; an optimised kernel could cache the keys of unchanged slots). The two lanes of a pair | |
| 64 | +read the \emph{same} snapshot $\bar S$, so their writes are simultaneous by construction. | |
| 65 | + | |
| 66 | +\paragraph{Clipped, gated write.} | |
| 67 | +Let $\tilde S_j$ be the lane output for $j\in I$. The write is | |
| 68 | +\begin{equation} | |
| 69 | +\delta_j=\tilde S_j - S_j,\qquad | |
| 70 | +\delta^{c}_j=\delta_j\min\!\Big(1,\frac{c}{\max(\rms(\delta_j),\epsilon)}\Big),\qquad | |
| 71 | +\eta_j=\eta_{\max}\,\sigma\!\big(g([S_j;\tilde S_j])\big),\qquad | |
| 72 | +S'_j = S_j+\eta_j\delta^{c}_j, | |
| 73 | +\label{eq:write} | |
| 74 | +\end{equation} | |
| 75 | +and $S'_m=S_m$ \emph{exactly} for $m\notin I$ (the implementation materialises the new rows through a one-hot | |
| 76 | +scatter from the snapshot, so the result is independent of lane order; Proposition~\ref{prop:locality}). We use | |
| 77 | +$\eta_{\max}=0.25$, $c=1$: these are untested research hyper-parameters, not tuned values. The gate $g$ and the | |
| 78 | +context map are zero-initialised, so $\eta_j=\eta_{\max}/2$ at initialisation. | |
| 79 | + | |
| 80 | +\subsection{Differential readout and exact bypass}\label{sec:readout} | |
| 81 | +A single attention read with a query that depends on $z_t$ only, | |
| 82 | +$r(S,z_t)=\sum_j \alpha_j S_j$, $\alpha=\mathrm{softmax}_j\big((W_q z_t)\cdot (W_k S_j)/\sqrt{w}\big)$, is | |
| 83 | +evaluated at the final and at the initial workspace, and their difference is mapped back to the residual width: | |
| 84 | +\begin{equation} | |
| 85 | +\Delta_t = s\,U\big(r(S^{(n_t)},z_t)-r(S^{(0)},z_t)\big),\qquad | |
| 86 | +h^{\mathrm{corr}}_t=h_t+\Delta_t,\qquad | |
| 87 | +\ell_t = E\,\mathrm{RMSNorm}(h^{\mathrm{corr}}_t). | |
| 88 | +\label{eq:readout} | |
| 89 | +\end{equation} | |
| 90 | +$U\in\R^{d\times w}$ is initialised at zero and $s>0$ is a fixed scalar set once at the start of editor training to | |
| 91 | +the measured root-mean-square of the backbone residual stream (Section~\ref{sec:learning}); because the final | |
| 92 | +RMSNorm is scale invariant, $s$ only reparametrises the learning rate of $U$. When $n_t=0$, the term | |
| 93 | +\eqref{eq:readout} is algebraically zero; the implementation additionally branches out before allocating any slot | |
| 94 | +when the budget is zero, so the fast path performs no workspace computation at all (Proposition~\ref{prop:identity}). | |
| 95 | + | |
| 96 | +\subsection{Configurations and parameter accounting} | |
| 97 | +\input{tables/configs.tex} | |
added
paper/sections/conclusion.tex
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +% to be written | |
added
paper/sections/design.tex
+42 −0
@@ -0,0 +1,42 @@ | ||
| 1 | +\section{Experimental design}\label{sec:design} | |
| 2 | + | |
| 3 | +\subsection{Data} | |
| 4 | +\paragraph{Synthetic compositional suite.} Six generator families with programmatic ground truth | |
| 5 | +(Table~\ref{tab:families}): multi-step arithmetic (\textsf{arith}), variable binding with dependent substitution | |
| 6 | +and a distractor assignment (\textsf{bind}), register updates (\textsf{fsm}), conjunctive constraints with | |
| 7 | +distractor items (\textsf{excl}), rule-based selection under a budget and an exclusion (\textsf{sched}), and a | |
| 8 | +held-out composition of a binding chain feeding an arithmetic expression (\textsf{compose}). Each family has five | |
| 9 | +phrasing templates; training uses templates 0--2 at depths 1--4 and a dedicated seed stream. Evaluation regimes: | |
| 10 | +\emph{iid} (training templates and depths, new seed), \emph{template} (held-out paraphrases 3--4), | |
| 11 | +\emph{depth} (depths 5--6), and \emph{compose} (held-out family). Splits are fixed \emph{before} sampling by | |
| 12 | +(family, template, depth, seed); items whose full text duplicates an item of an earlier split are dropped and | |
| 13 | +counted (995 train, 256 dev-iid, 284 test-iid duplicates; \texttt{data/splits/synthetic\_v1/manifest.json}). | |
| 14 | +A separate \emph{calibration} stream (seed 3) is used only for controller calibration and penalty selection. | |
| 15 | +Test streams (seed 9) are locked: the evaluation driver refuses to read them until | |
| 16 | +\texttt{protocol/locked\_test\_manifest.json} records the unsealing. | |
| 17 | +\input{tables/families.tex} | |
| 18 | + | |
| 19 | +\paragraph{General language corpus.} For the 103M experiments we stream the first documents of | |
| 20 | +\textsc{FineWeb-Edu} (sample-10BT, ODC-By~1.0; \citealp{penedo2024fineweb}) in canonical shard order, deduplicate | |
| 21 | +exactly, assign documents to train/val/test by content hash (98/1/1\,\%), and train a byte-level BPE tokenizer | |
| 22 | +(32,768 symbols, digits split individually) on the train split only. The corpus holds 290.0M train, 2.91M | |
| 23 | +validation and 2.88M locked test tokens (\texttt{data/processed/fineweb\_edu\_300m/manifest.json}). | |
| 24 | + | |
| 25 | +\subsection{Systems and controls} | |
| 26 | +All systems in a comparison share the backbone checkpoint, tokenizer, data stream, step count and batch size. | |
| 27 | +Table~\ref{tab:controls} lists every system with its unique parameter count, training exposure and inference | |
| 28 | +regime. The parameter-matched plain decoder (feed-forward width 2144 instead of 2048, $+2.65$M parameters) is | |
| 29 | +trained from scratch with the backbone recipe. The dense-slot editor writes all $M$ slots per action with the same | |
| 30 | +weights. The global-recurrence adapter (Section~\ref{sec:related}) refines the whole top representation with one | |
| 31 | +shared block at width $w$, parameter-matched to the \prisme{} workspace, trained with the identical forced-budget | |
| 32 | +recipe and the identical symmetry-breaking device. Reference routers (random, uniform round-robin, fixed pair, | |
| 33 | +entropy threshold, norm-change threshold) reuse the trained \prisme{} editor and are therefore matched in | |
| 34 | +everything except the choice of action. The exhaustive one-step and two-step oracles read the target and are | |
| 35 | +reported as ceilings only. | |
| 36 | + | |
| 37 | +\subsection{Pre-registration, seeds and uncertainty} | |
| 38 | +The primary outcome, metrics, decision thresholds and seed plan were written to | |
| 39 | +\texttt{protocol/preregistration.md} before any test split was read (Appendix~\ref{app:prereg}). Differences | |
| 40 | +between systems on the same items use a paired bootstrap over items (10,000 resamples); differences across | |
| 41 | +training seeds use the per-seed statistic with a $t$-interval. Tokens of one training run are never treated as | |
| 42 | +independent replications of training. Latency is measured on the stated device with warm-up and synchronisation. | |
added
paper/sections/implementation.tex
+46 −0
@@ -0,0 +1,46 @@ | ||
| 1 | +\section{Implementation and compute model}\label{sec:compute} | |
| 2 | + | |
| 3 | +\subsection{Reference implementation} | |
| 4 | +The reference implementation is a self-contained PyTorch package (\texttt{prisme/}, about 1,100 lines) with the | |
| 5 | +backbone, bank, workspace, editor, controller, readout, caches, counterfactual labelling, oracles and the analytic | |
| 6 | +cost model, plus training, evaluation and baseline packages. In teacher-forced training all $T$ positions of a | |
| 7 | +batch are processed in parallel: the workspace is a tensor of shape $[B,T,M,w]$, the active queries a tensor | |
| 8 | +$[B,T,k,w]$ with $k=2$ lanes (the second lane is masked for single-slot actions), and the bank cross-attention of | |
| 9 | +\eqref{eq:blk1} is one scaled-dot-product call with a per-query causal mask over the $T$ bank rows. Incremental | |
| 10 | +decoding keeps two caches per request batch---the per-layer Transformer KV cache and the bank $(K,V)$---and is | |
| 11 | +tested to agree with full-prefix recomputation within $10^{-4}$ under forced budgets, learned routing, early | |
| 12 | +\Stop{} and mixed edited/unedited positions; the bypass path is bitwise identical to the backbone. | |
| 13 | + | |
| 14 | +\subsection{Analytic operation counts} | |
| 15 | +We count one multiply--add as two operations and ignore normalisations, activations and softmaxes. For a decode | |
| 16 | +step at context length $T$ the backbone costs $2N_{\mathrm{ne}}+4LTd+2dV$ operations ($N_{\mathrm{ne}}$ the | |
| 17 | +non-embedding parameters; the vocabulary head is included). One editor block with $k$ active lanes costs | |
| 18 | +\begin{equation} | |
| 19 | +F_{\mathrm{block}}(k,T)=\underbrace{4kw^2+4kTw}_{\text{bank read}}+\underbrace{4kw^2+4Mw^2+4kMw}_{\text{slot read}} | |
| 20 | ++\underbrace{6kwf_w}_{\text{FFN}}, | |
| 21 | +\label{eq:fblock} | |
| 22 | +\end{equation} | |
| 23 | +which is exactly the arithmetic model of the original proposal; the slot-key refresh $4Mw^2$ dominates for small | |
| 24 | +$k$ and is the obvious target for a cached kernel. One controller evaluation costs | |
| 25 | +$2w^2+4w+6wf_w+|\Act|(2w^2+8w)$ and the readout $2w^2+2(2Mw^2+4Mw)+2wd$. Table~\ref{tab:ops} reproduces the | |
| 26 | +reference table of the proposal for the 459M configuration at $T=4096$ and reports the implemented model's | |
| 27 | +accounting; the two agree within 1.3\,\% (the proposal's controller and readout terms were slightly smaller). | |
| 28 | +The per-action cost proxy used by the controller is $c(a)=\big(L_wF_{\mathrm{block}}(k(a),T)+8k(a)w+\text{one controller | |
| 29 | +evaluation}\big)/\text{backbone ops}$; at the 103M configuration and $T=512$ it is \SI{0.080}{} for a single and | |
| 30 | +\SI{0.100}{} for a pair, so eight pair edits cost about $+83\,\%$ of the backbone's operations and dense | |
| 31 | +sixteen-slot writes about $+310\,\%$. | |
| 32 | +\input{tables/ops.tex} | |
| 33 | + | |
| 34 | +\subsection{Memory} | |
| 35 | +Parameters at two bytes each: \SI{0.204}{GiB} (103M), \SI{0.852}{GiB} (459M), \SI{2.18}{GiB} (1.18B). Per request | |
| 36 | +at context $T$: Transformer KV cache $2\cdot L\cdot T\cdot d_{kv}\cdot 2$ bytes (\SI{96}{MiB} at 459M, $T=4096$), | |
| 37 | +bank $2\cdot T\cdot w\cdot 2$ bytes (\SI{8}{MiB}), one workspace $M\cdot w\cdot 2$ bytes (\SI{16}{KiB}). The bank | |
| 38 | +grows linearly with the context: \prisme{} v1 is not a constant-memory model, and the bank read in | |
| 39 | +\eqref{eq:fblock} is linear in $T$ per active lane. Training stores one workspace per position | |
| 40 | +($B\cdot T\cdot M\cdot w$ floats) per recurrence step; measured peak memory is reported with every run. | |
| 41 | + | |
| 42 | +\subsection{Latency protocol} | |
| 43 | +Latencies are measured for batch-1 and batch-8 greedy decoding with warm-up, device synchronisation after every | |
| 44 | +token, fp32, a stated prompt length and number of generated tokens, and the cache state recorded | |
| 45 | +(Section~\ref{sec:results-latency}). Dynamic routing introduces host-side control flow at every action; this | |
| 46 | +overhead is part of the measurement and is not removed. | |
added
paper/sections/introduction.tex
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +% to be written | |
added
paper/sections/learning.tex
+60 −0
@@ -0,0 +1,60 @@ | ||
| 1 | +\section{Learning from interventions}\label{sec:learning} | |
| 2 | + | |
| 3 | +\subsection{Paired intervention-value target} | |
| 4 | +For a training state $s$ (a position $t$ with workspace $S$ after $r$ executed actions), an action $a$ and the known | |
| 5 | +next token $y=x_{t+1}$, let $D(\cdot)$ denote the readout-plus-head map of \eqref{eq:readout} and $T_a$ the | |
| 6 | +editor's transition. The intervention gain is | |
| 7 | +\begin{equation} | |
| 8 | +g(s,a;y)=\Loss\big(D(s),y\big)-\Loss\big(D(T_a(s)),y\big), | |
| 9 | +\label{eq:gain} | |
| 10 | +\end{equation} | |
| 11 | +obtained by cloning $s$, executing $a$ with the same checkpoint, and evaluating both branches on the same $y$. | |
| 12 | +A positive gain means the edit reduced the teacher-forced loss on this example. This is an intervention on the | |
| 13 | +model's own computation, not a causal effect in the world, and a token-level gain does not imply that a complete | |
| 14 | +generated answer improves (Section~\ref{sec:results-reasoning}). The label $y$ enters \eqref{eq:gain} only; the | |
| 15 | +runtime features of \eqref{eq:controller} never contain it (verified by a unit test that perturbs $x_{t+1}$ and | |
| 16 | +checks that all earlier predictions are unchanged). | |
| 17 | + | |
| 18 | +\subsection{Logging policy and propensities} | |
| 19 | +On sampled training states we evaluate four candidates: the current controller's preferred action (or a random | |
| 20 | +single if no controller exists yet), a competitive alternative from the other group (a pair if the preferred action | |
| 21 | +is a single and vice versa), and two uniform draws (one single, one pair). The logging mixture's propensity of | |
| 22 | +each candidate is recorded, enabling inverse-propensity or doubly-robust corrections of selection bias | |
| 23 | +\citep{dudik2011doublyrobust,swaminathan2015crm}. On an audit subset every one of the 24 actions is executed, | |
| 24 | +giving exact one-step oracle values. Helpful and harmful gains are both kept; sibling branches of one state never | |
| 25 | +cross a partition boundary; every label file records the SHA-256 of the editor checkpoint that produced it, and the | |
| 26 | +controller trainer refuses labels from a different checkpoint. | |
| 27 | + | |
| 28 | +\subsection{Staged optimisation}\label{sec:stages} | |
| 29 | +\begin{description}[leftmargin=1.5em] | |
| 30 | +\item[Stage A (backbone).] A common backbone checkpoint is trained (or, for the synthetic experiments, trained from | |
| 31 | + scratch on the synthetic train split). Every comparison starts from the same weights, tokenizer and data. | |
| 32 | +\item[Stage B (editor bootstrap).] The backbone is frozen; the editor, bank projections, slot embeddings and | |
| 33 | + readout are trained with the language loss under \emph{forced} action sequences: each position draws a budget | |
| 34 | + $b\in\{0,1,2,4,8\}$ and $b$ uniformly random actions (singles and pairs balanced), later steps being \Stop. | |
| 35 | + Because $U=0$ blocks all gradient into the editor at initialisation, the bootstrap requires a symmetry-breaking | |
| 36 | + device; we study the zero initialisation, a non-zero initialisation and the scale $s$ of \eqref{eq:readout} | |
| 37 | + explicitly (Section~\ref{sec:results-learnability}). | |
| 38 | +\item[Stage C (counterfactual collection).] With the editor fixed, states are sampled by executing $r\sim U\{0,\dots,3\}$ | |
| 39 | + random actions, and labels \eqref{eq:gain} are collected as above. | |
| 40 | +\item[Stage D (controller fit).] The controller minimises | |
| 41 | + \begin{equation} | |
| 42 | + \Loss_{\mathrm{ctrl}}=\alpha\,\mathrm{Huber}_{\delta}\big(\hat g,\sg(g)\big) | |
| 43 | + +\beta\,\mathrm{BCE}\big(\hat\ell_{\mathrm{harm}},\mathbf 1[g<0]\big) | |
| 44 | + +\gamma\Big(\tfrac{(g-\sg(\hat g))^2}{2\hat\sigma^2}+\log\hat\sigma\Big), | |
| 45 | + \label{eq:ctrl-loss} | |
| 46 | + \end{equation} | |
| 47 | + on the training partition; sign accuracy, Brier score, expected calibration error and oracle regret are measured | |
| 48 | + on the calibration partition, where $(\lambda,\kappa)$ are also selected by actual rollouts. | |
| 49 | +\item[Stage E (lock).] The final checkpoint (editor + controller + selected penalties) is frozen before any locked | |
| 50 | + test split is read. Alternating editor/controller refinement is treated as a later ablation, not a dependency. | |
| 51 | +\end{description} | |
| 52 | +No teacher model and no distillation are used anywhere in this paper; the composite loss of the original proposal | |
| 53 | +reduces to the language loss in Stage B and \eqref{eq:ctrl-loss} in Stage D. | |
| 54 | + | |
| 55 | +\subsection{Delayed benefits} | |
| 56 | +An action may have negative immediate gain yet enable a later beneficial action. We therefore report, on audit | |
| 57 | +subsets, the exhaustive one-step oracle, the exhaustive two-action oracle ($|\Act|(|\Act|+1)$ branches per state) | |
| 58 | +and the ``greedy first, then best'' value, and decompose the gap between the learned policy and the two-step | |
| 59 | +oracle into an \emph{opportunity} term (two-step minus one-step oracle) and a \emph{prediction} term (one-step | |
| 60 | +oracle minus learned policy). | |
added
paper/sections/limitations.tex
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +% to be written | |
added
paper/sections/mechanistic.tex
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +% to be written | |
added
paper/sections/policy.tex
+32 −0
@@ -0,0 +1,32 @@ | ||
| 1 | +\section{Adaptive intervention policy}\label{sec:policy} | |
| 2 | + | |
| 3 | +The deployed router is a \emph{value controller} that scores every nonterminal action from target-free features | |
| 4 | +and never executes a candidate edit. At state $s=(z_t,S,r,\rho)$ it computes one shared state representation | |
| 5 | +$u = \mathrm{MLP}(W_z z_t + \mathrm{ctx}(r/R_{\max},\rho/R_{\max}))$ and, for each action $a$, | |
| 6 | +\begin{equation} | |
| 7 | +\bar s_a=\frac{1}{k(a)}\sum_{j\in I(a)} S_j,\qquad | |
| 8 | +\big(\hat g(s,a),\hat c(s,a),\log\hat\sigma(s,a),\hat\ell_{\mathrm{harm}}(s,a)\big) | |
| 9 | + = W_{\mathrm{out}}\,\mathrm{SiLU}\big(u + W_s\bar s_a + \kappa_{k(a)}\big)+b, | |
| 10 | +\label{eq:controller} | |
| 11 | +\end{equation} | |
| 12 | +so that scoring all $|\Act|$ actions costs one MLP plus $|\Act|$ projections of width $w$ | |
| 13 | +(Section~\ref{sec:compute}). The four heads estimate the \emph{intervention gain} $g(s,a)$ of | |
| 14 | +Section~\ref{sec:learning}, a cost proxy, a predictive scale, and the probability that the gain is negative. Actions | |
| 15 | +are chosen greedily, | |
| 16 | +\begin{equation} | |
| 17 | +Q(s,a)=\hat g(s,a)-\lambda\,c(a)-\kappa\,\hat\sigma(s,a),\qquad Q(s,\Stop)=0,\qquad | |
| 18 | +a^\star=\arg\max_{a\in\Act\cup\{\Stop\}} Q(s,a), | |
| 19 | +\label{eq:Q} | |
| 20 | +\end{equation} | |
| 21 | +where $c(a)$ is the \emph{known} analytic cost of the action (ops of one edit plus one controller evaluation, | |
| 22 | +relative to the backbone's ops per token; Section~\ref{sec:compute}); the learned $\hat c$ is kept only as a | |
| 23 | +diagnostic regressor. The state is recomputed after every executed edit, and the loop stops when \Stop{} wins or | |
| 24 | +the cap $R_{\max}$ is reached. The penalties $(\lambda,\kappa)$ are tuned on calibration rollouts only | |
| 25 | +(Section~\ref{sec:design}); sweeping $\lambda$ traces the controller's quality--cost curve. | |
| 26 | + | |
| 27 | +\paragraph{Known limitations of the rule.} | |
| 28 | +The policy is one-step greedy: it cannot represent an action whose value materialises only after a later action. | |
| 29 | +We quantify this myopia directly with a two-action lookahead oracle (Section~\ref{sec:results-oracle}) rather than | |
| 30 | +assume it away. The rule is also only as good as the calibration of $\hat g$; Proposition~\ref{prop:regret} bounds | |
| 31 | +the one-step loss of the greedy rule in terms of uniform estimation errors but says nothing about multi-step | |
| 32 | +optimality. | |
added
paper/sections/problem.tex
+30 −0
@@ -0,0 +1,30 @@ | ||
| 1 | +\section{Formal problem}\label{sec:problem} | |
| 2 | + | |
| 3 | +\paragraph{Autoregressive prediction under a computation budget.} | |
| 4 | +Let $\mathcal{V}$ be a vocabulary of size $V$ and $x_{1:T}\in\mathcal{V}^T$ a sequence. A causal language model | |
| 5 | +assigns $p_\theta(x_{t+1}\mid x_{1:t})$ and is trained to minimise the teacher-forced cross-entropy | |
| 6 | +$\Loss(\theta)=\E\big[-\log p_\theta(x_{t+1}\mid x_{1:t})\big]$. We consider models whose per-position computation | |
| 7 | +is \emph{variable}: for each position $t$ the model may execute a number $n_t\in\{0,\dots,R_{\max}\}$ of additional | |
| 8 | +\emph{actions} before emitting its distribution, each action having a known arithmetic cost. Writing $c(a)$ for the | |
| 9 | +cost of action $a$ (in units of the backbone's own operations per token), the realised cost of position $t$ is | |
| 10 | +$C_t = 1 + \sum_{r\le n_t} c(a_{t,r})$ plus the cost of deciding, and the quantities of interest are the pairs | |
| 11 | +\begin{equation} | |
| 12 | +\big(\;\text{quality}(\pi),\;\;\E_t[C_t\mid\pi]\;\big),\qquad | |
| 13 | +\big(\;\text{quality}(\pi),\;\;\operatorname{median}_t\,\text{latency}_t(\pi)\;\big), | |
| 14 | +\label{eq:frontier} | |
| 15 | +\end{equation} | |
| 16 | +traced out as the routing policy $\pi$ and its penalties vary. Quality is either the held-out cross-entropy or, for | |
| 17 | +tasks with programmatic ground truth, the exact-match accuracy of free greedy generation. The research question is | |
| 18 | +whether a policy that chooses \emph{where} to write (which of $M$ latent slots) using supervision from executed | |
| 19 | +counterfactual branches moves the frontier \eqref{eq:frontier} relative to policies that do not choose (uniform or | |
| 20 | +random slot selection, dense updates of all slots) and relative to global latent recurrence at equal cost. | |
| 21 | + | |
| 22 | +\paragraph{Causal prefix constraint.} | |
| 23 | +Every quantity that influences $p_\theta(x_{t+1}\mid x_{1:t})$---including the routing decision---must be a function | |
| 24 | +of $x_{1:t}$ only. In particular the label $x_{t+1}$ is available to the \emph{training} procedure that constructs | |
| 25 | +supervision, but never to the deployed router (Section~\ref{sec:policy}). | |
| 26 | + | |
| 27 | +\paragraph{What is \emph{not} studied.} | |
| 28 | +The model has no persistent memory across positions (the workspace is reset at every $t$), no retrieval, no tools, | |
| 29 | +no reinforcement learning, and no textual chain of thought. These exclusions are deliberate: they isolate the | |
| 30 | +single mechanism under test. | |
added
paper/sections/related.tex
+53 −0
@@ -0,0 +1,53 @@ | ||
| 1 | +\section{Related work and novelty positioning}\label{sec:related} | |
| 2 | + | |
| 3 | +We re-verified every reference at execution time from primary sources (arXiv metadata and full texts, official | |
| 4 | +repositories and licences; Appendix~\ref{app:literature} and \texttt{literature/novelty\_audit.md}). Four strands | |
| 5 | +are directly relevant. | |
| 6 | + | |
| 7 | +\paragraph{Adaptive depth and latent recurrence.} | |
| 8 | +Universal Transformers \citep{dehghani2019universal} and ACT \citep{graves2016act} introduced weight-shared | |
| 9 | +recurrence over depth with learned per-position halting; PonderNet \citep{banino2021pondernet} made halting | |
| 10 | +probabilistic. Mixture-of-Depths routes tokens around blocks with a fixed capacity \citep{raposo2024mixtureofdepths}, | |
| 11 | +LayerSkip trains early exits \citep{elhoushi2024layerskip}, and CALM calibrates confidence-based exits | |
| 12 | +\citep{schuster2022calm}. The recurrent-depth line scaled latent recurrence to language model pretraining: | |
| 13 | +Huginn \citep{geiping2025recurrentdepth}, Ouro \citep{zhu2025ouro}, Mixture-of-Recursions \citep{bae2025mixtureofrecursions}, | |
| 14 | +looped Transformers \citep{giannou2023looped,saunshi2025reasoning}, and in 2026 RecurTrace | |
| 15 | +\citep{wang2026recurtrace}, Elastic Latent Recurrence \citep{ladhak2026elastic}, Think-at-Hard | |
| 16 | +\citep{fu2026thinkathard,you2026tah2} and several contemporaries listed in Appendix~\ref{app:literature}. All of | |
| 17 | +these refine the \emph{entire} hidden state of a position (globally), and route depth per token or per sequence. | |
| 18 | + | |
| 19 | +\paragraph{Slots, workspaces and sparse writes.} | |
| 20 | +Recurrent Independent Mechanisms \citep{goyal2019rims} and the Shared Global Workspace | |
| 21 | +\citep{goyal2022workspace} update a small number of competing modules per step; Slot Attention | |
| 22 | +\citep{locatello2020slot} and Perceiver / Perceiver AR \citep{jaegle2021perceiver,hawthorne2022perceiverar} maintain | |
| 23 | +a small bank of latents that cross-attend to a long input; Sparse Access Memory \citep{rae2016sparse} writes | |
| 24 | +sparsely into an external memory. Most recently ReFlux \citep{ye2026reflux} writes router-selected sparse | |
| 25 | +deep-to-shallow deltas into a frozen backbone, with the backbone unchanged when no route is active. \prisme{}'s | |
| 26 | +workspace differs in being \emph{ephemeral} (reset per token), separate from a read-only cached bank, and edited by | |
| 27 | +discrete 1--2-slot actions rather than by competition among modules. | |
| 28 | + | |
| 29 | +\paragraph{Loss-gain supervision of routing.} | |
| 30 | +The supervision signal we use---the realised change in next-token loss of an executed extra branch---has become | |
| 31 | +standard for \emph{binary} continue/stop decisions in 2025--2026: Ouro's second stage fits its exit gate to observed | |
| 32 | +per-token loss improvements \citep{zhu2025ouro}; TaH2 calls the same construction ``lookahead depth supervision'' | |
| 33 | +\citep{you2026tah2}; RecurTrace trains a halting head on an oracle ``more depth still helps'' label | |
| 34 | +\citep{wang2026recurtrace}. Earlier, fitted value iteration and counterfactual risk minimisation established the | |
| 35 | +use of logged propensities for off-policy action evaluation \citep{ernst2005tree,dudik2011doublyrobust,swaminathan2015crm}. | |
| 36 | +\prisme{} makes the label \emph{action-conditional} over 24 spatial actions, logs propensities, and separates gain, | |
| 37 | +cost, uncertainty and harm heads. | |
| 38 | + | |
| 39 | +\paragraph{Zero-initialised identity insertion.} | |
| 40 | +Adding a zero-initialised branch so that a new module starts as the identity is a well-established device | |
| 41 | +\citep{bachlechner2020rezero,alayrac2022flamingo,zhang2023controlnet}; exact bypass of unrouted tokens exists in | |
| 42 | +Mixture-of-Depths and in TaH/TaH2, and ReFlux leaves the frozen backbone unchanged with no active route. The | |
| 43 | +differential readout of Section~\ref{sec:readout} is a modest variant of this device and is \emph{not} claimed as a | |
| 44 | +contribution. | |
| 45 | + | |
| 46 | +\paragraph{What is and is not new.} | |
| 47 | +After this audit, none of the three ingredients of \prisme{} is new on its own. What has no direct predecessor is | |
| 48 | +their combination: a per-token slot workspace with selected local writes, trained with action-conditional | |
| 49 | +counterfactual gains over a 25-action catalogue with logged propensities, behind an exact zero-budget bypass. | |
| 50 | +Accordingly, the contribution of this paper is empirical and methodological---a controlled test of whether spatial | |
| 51 | +selection supervised by counterfactual gains buys anything over uniform, dense or global alternatives at matched | |
| 52 | +cost---rather than architectural priority. Table~\ref{tab:novelty} in Appendix~\ref{app:literature} gives the | |
| 53 | +claim-by-claim comparison. | |
added
paper/sections/reproducibility.tex
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +% to be written | |
added
paper/sections/results.tex
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +% to be written | |
added
paper/sections/theory.tex
+48 −0
@@ -0,0 +1,48 @@ | ||
| 1 | +\section{Theoretical properties}\label{sec:theory} | |
| 2 | + | |
| 3 | +The statements below are structural properties of the computation graph, not performance guarantees. Proofs are | |
| 4 | +in Appendix~\ref{app:proofs}; each is accompanied by a numerical property test in \texttt{tests/unit/} that | |
| 5 | +exercises the implemented code on random inputs (tests are evidence about the implementation, not proofs). | |
| 6 | + | |
| 7 | +\begin{assumption}[prefix causality of the components]\label{ass:causal} | |
| 8 | +$B_\phi$ is causal; the bank visible to a query at position $t$ is $\{(K_j,V_j)\}_{j\le t}$; the controller features | |
| 9 | +at position $t$ are functions of $(z_t,S,r,\rho)$ only. | |
| 10 | +\end{assumption} | |
| 11 | + | |
| 12 | +\begin{proposition}[causality]\label{prop:causal} | |
| 13 | +Under Assumption~\ref{ass:causal}, for every executed action sequence the corrected logits $\ell_t$ are a function | |
| 14 | +of $x_{1:t}$ only, for all $t$ and for any number of edits $n_t\le R_{\max}$. | |
| 15 | +\end{proposition} | |
| 16 | + | |
| 17 | +\begin{proposition}[identity and exact bypass]\label{prop:identity} | |
| 18 | +If $n_t=0$ then $\Delta_t=0$ exactly and $\ell_t=\ell^{\mathrm{bb}}_t$ in exact arithmetic. In floating point, the | |
| 19 | +implementation returns the backbone head through an explicit branch that performs no workspace computation when | |
| 20 | +the budget is zero, and through the same branch for positions with no executed edit, so the equality is bitwise for | |
| 21 | +identical execution settings. After joint fine-tuning the identity holds relative to the \emph{current} backbone, | |
| 22 | +not to any earlier checkpoint. | |
| 23 | +\end{proposition} | |
| 24 | + | |
| 25 | +\begin{proposition}[exact locality and order independence]\label{prop:locality} | |
| 26 | +For an action with active set $I$, $S'_m=S_m$ bitwise for all $m\notin I$, and the result of a pair action does not | |
| 27 | +depend on the order in which its two lanes are processed. | |
| 28 | +\end{proposition} | |
| 29 | + | |
| 30 | +\begin{proposition}[bounded displacement]\label{prop:displacement} | |
| 31 | +With $\rms(v)=\|v\|_2/\sqrt{w}$, after $n_j$ writes into slot $j$, | |
| 32 | +$\rms\big(S^{(R)}_j-S^{(0)}_j\big)\le n_j\,\eta_{\max}\,c$. The bound is a consequence of clipping and of the | |
| 33 | +triangle inequality; it does not imply convergence of the edits or any decrease of the loss. | |
| 34 | +\end{proposition} | |
| 35 | + | |
| 36 | +\begin{proposition}[one-step selection regret]\label{prop:regret} | |
| 37 | +Fix a state $s$ and let $J(a)=g(s,a)-\lambda c(a)$ for $a\in\Act$ and $J(\Stop)=0$, with estimates | |
| 38 | +$\hat J(a)=\hat g(s,a)-\lambda\hat c(a)$. If $|\hat g(s,a)-g(s,a)|\le\varepsilon$ and $|\hat c(a)-c(a)|\le\delta$ | |
| 39 | +for all feasible $a$, then the greedy choice $\hat a=\arg\max\hat J$ satisfies | |
| 40 | +$\max_a J(a)-J(\hat a)\le 2(\varepsilon+\lambda\delta)$. With the known cost used in this paper, $\delta=0$. | |
| 41 | +Adding the risk term $-\kappa\hat\sigma$ changes the objective and the bound no longer applies as stated. | |
| 42 | +\end{proposition} | |
| 43 | + | |
| 44 | +\begin{remark} | |
| 45 | +None of the propositions bounds the multi-step value of a trajectory; the greedy rule can be arbitrarily | |
| 46 | +sub-optimal on states where the best first action has negative immediate gain. Section~\ref{sec:results-oracle} | |
| 47 | +measures that opportunity directly. | |
| 48 | +\end{remark} | |
added
paper/sections/titlepage.tex
+77 −0
@@ -0,0 +1,77 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% Title Page | |
| 3 | +% ============================================================================ | |
| 4 | +\thispagestyle{empty} | |
| 5 | + | |
| 6 | +\begin{center} | |
| 7 | + | |
| 8 | +% --- Logo --- | |
| 9 | +\includegraphics[width=3.5cm]{uq_logo.jpg} | |
| 10 | + | |
| 11 | +\vspace{0.6cm} | |
| 12 | + | |
| 13 | +{\footnotesize\textsc{Université du Québec en Outaouais}}\\[0.15cm] | |
| 14 | +{\footnotesize\textsc{Département des sciences administratives}} | |
| 15 | + | |
| 16 | +\vspace{0.8cm} | |
| 17 | + | |
| 18 | +{\footnotesize\textsc{Working Paper No.~\WPnumber}} | |
| 19 | + | |
| 20 | +\vspace{1.2cm} | |
| 21 | + | |
| 22 | +% --- Title --- | |
| 23 | +{\LARGE\bfseries \WPtitle\par} | |
| 24 | + | |
| 25 | +\ifx\WPsubtitle\empty\else | |
| 26 | + \vspace{0.3cm} | |
| 27 | + {\large\itshape \WPsubtitle\par} | |
| 28 | +\fi | |
| 29 | + | |
| 30 | +\vspace{1.2cm} | |
| 31 | + | |
| 32 | +% --- Author --- | |
| 33 | +{\large \WPauthor}\\[0.3cm] | |
| 34 | +{\normalsize \WPaffiliation}\\[0.15cm] | |
| 35 | +{\normalsize \href{mailto:\WPemail}{\WPemail}}\\[0.15cm] | |
| 36 | +{\small \WPaddress} | |
| 37 | + | |
| 38 | +\vspace{0.8cm} | |
| 39 | + | |
| 40 | +% --- Date & Version --- | |
| 41 | +{\normalsize \WPdate}\\[0.1cm] | |
| 42 | +{\small Version~\WPversion} | |
| 43 | + | |
| 44 | +\end{center} | |
| 45 | + | |
| 46 | +\vfill | |
| 47 | + | |
| 48 | +\newpage | |
| 49 | + | |
| 50 | +% ============================================================================ | |
| 51 | +% Abstract Page | |
| 52 | +% ============================================================================ | |
| 53 | +\thispagestyle{empty} | |
| 54 | + | |
| 55 | +\vspace*{1cm} | |
| 56 | + | |
| 57 | +\noindent\rule{\textwidth}{0.4pt} | |
| 58 | +\vspace{0.3cm} | |
| 59 | + | |
| 60 | +\noindent\textbf{Abstract} | |
| 61 | + | |
| 62 | +\vspace{0.15cm} | |
| 63 | + | |
| 64 | +\noindent\WPabstract | |
| 65 | + | |
| 66 | +\vspace{0.4cm} | |
| 67 | + | |
| 68 | +\noindent\textbf{Keywords:} \WPkeywords | |
| 69 | + | |
| 70 | +\vspace{0.15cm} | |
| 71 | + | |
| 72 | +\noindent\textbf{JEL Classification:} \WPjel | |
| 73 | + | |
| 74 | +\vspace{0.3cm} | |
| 75 | +\noindent\rule{\textwidth}{0.4pt} | |
| 76 | + | |
| 77 | +\newpage | |
added
paper/sections/titlepage_anon.tex
+53 −0
@@ -0,0 +1,53 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% ANONYMOUS TITLE PAGE — for double-anonymized journal submission | |
| 3 | +% Used by paper/main_anon.tex (same macros as main.tex, no author block, | |
| 4 | +% no institution, no logo). Abstract/keywords/JEL page identical to titlepage.tex. | |
| 5 | +% ============================================================================ | |
| 6 | +\thispagestyle{empty} | |
| 7 | +\begin{center} | |
| 8 | + | |
| 9 | +\vspace*{3cm} | |
| 10 | + | |
| 11 | +{\LARGE\bfseries \WPtitle\par} | |
| 12 | + | |
| 13 | +\ifx\WPsubtitle\empty\else | |
| 14 | + \vspace{0.3cm} | |
| 15 | + {\large\itshape \WPsubtitle\par} | |
| 16 | +\fi | |
| 17 | + | |
| 18 | +\vspace{1.5cm} | |
| 19 | + | |
| 20 | +{\normalsize [Author identity withheld for review]} | |
| 21 | + | |
| 22 | +\vspace{0.8cm} | |
| 23 | + | |
| 24 | +{\normalsize \WPdate} | |
| 25 | + | |
| 26 | +\end{center} | |
| 27 | + | |
| 28 | +\vfill | |
| 29 | + | |
| 30 | +\newpage | |
| 31 | +\thispagestyle{empty} | |
| 32 | +\vspace*{1cm} | |
| 33 | + | |
| 34 | +\noindent\rule{\textwidth}{0.4pt} | |
| 35 | +\vspace{0.3cm} | |
| 36 | + | |
| 37 | +\noindent\textbf{Abstract} | |
| 38 | +\vspace{0.15cm} | |
| 39 | + | |
| 40 | +\noindent\WPabstract | |
| 41 | + | |
| 42 | +\vspace{0.4cm} | |
| 43 | + | |
| 44 | +\noindent\textbf{Keywords:} \WPkeywords | |
| 45 | + | |
| 46 | +\vspace{0.15cm} | |
| 47 | + | |
| 48 | +\noindent\textbf{JEL Classification:} \WPjel | |
| 49 | + | |
| 50 | +\vspace{0.3cm} | |
| 51 | +\noindent\rule{\textwidth}{0.4pt} | |
| 52 | + | |
| 53 | +\newpage | |
added
paper/tables/configs.tex
+18 −0
@@ -0,0 +1,18 @@ | ||
| 1 | +\begin{table}[t]\centering\small | |
| 2 | +\caption{Dimensioned configurations and exact unique parameter counts (meta-device instantiation, | |
| 3 | +\texttt{scripts/count\_params.py}). Backbone counts equal the reference design exactly; workspace+controller counts | |
| 4 | +differ from the proposal's figures (reconciled in Appendix~\ref{app:params}).} | |
| 5 | +\label{tab:configs} | |
| 6 | +\begin{tabular}{lrrr}\toprule | |
| 7 | +Property & PRISME-103M & PRISME-459M & PRISME-1.18B\\\midrule | |
| 8 | +Backbone layers $L$ & 12 & 16 & 24\\ | |
| 9 | +Hidden $d$ / FFN & 768 / 2048 & 1536 / 4096 & 2048 / 5632\\ | |
| 10 | +Query / KV heads & 12 / 3 & 24 / 6 & 32 / 8\\ | |
| 11 | +Workspace $w$ / FFN$_w$ & 256 / 768 & 512 / 1536 & 768 / 2048\\ | |
| 12 | +Slots $M$, blocks $L_w$, cap $R_{\max}$ & 16, 2, 8 & 16, 2, 8 & 16, 2, 8\\ | |
| 13 | +\midrule | |
| 14 | +Backbone parameters & 99,502,848 & 446,744,064 & 1,149,339,648\\ | |
| 15 | +Workspace + controller & 2,631,173 & 10,505,221 & 21,655,813\\ | |
| 16 | +\textbf{Total} & \textbf{102,134,021} & \textbf{457,249,285} & \textbf{1,170,995,461}\\ | |
| 17 | +Proposal's workspace figure & 3,088,260 & 12,336,900 & 25,779,844\\ | |
| 18 | +\bottomrule\end{tabular}\end{table} | |
added
paper/tables/controller.tex
+8 −0
@@ -0,0 +1,8 @@ | ||
| 1 | +\begin{table}[t]\centering\small | |
| 2 | +\caption{Value-controller fit on logged counterfactual labels (tiny scale). Calibration-partition diagnostics: | |
| 3 | +Huber loss of $\hat g$, sign accuracy, Brier score and ECE of the harm head, one-step oracle opportunity and the | |
| 4 | +regret of the controller's greedy choice on the exhaustive-audit states (captured fraction $=1-$ regret/opportunity).} | |
| 5 | +\label{tab:controller} | |
| 6 | +\begin{tabular}{lrrrrrrr}\toprule | |
| 7 | +Controller features & labels & Huber & sign acc. & Brier & ECE & opportunity & captured\\\midrule | |
| 8 | +\bottomrule\end{tabular}\end{table} | |
added
paper/tables/controls.tex
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +% placeholder | |
added
paper/tables/families.tex
+17 −0
@@ -0,0 +1,17 @@ | ||
| 1 | +\begin{table}[t]\centering\small | |
| 2 | +\caption{Synthetic suite v1: splits fixed before sampling. $n$ = items after cross-split exact-duplicate removal; | |
| 3 | +``dropped'' = duplicates removed. Test splits are locked until pre-registration is frozen.} | |
| 4 | +\label{tab:families} | |
| 5 | +\begin{tabular}{llllrr}\toprule | |
| 6 | +Split & Families & Templates & Depths & $n$ & dropped\\\midrule | |
| 7 | +train & arith, bind, fsm, excl, sched & 0,1,2 & 1,2,3,4 & 23,005 & 995\\ | |
| 8 | +calib & arith, bind, fsm, excl, sched & 0,1,2 & 1,2,3,4 & 1,690 & 110\\ | |
| 9 | +dev\_iid & arith, bind, fsm, excl, sched & 0,1,2 & 1,2,3,4 & 3,344 & 256\\ | |
| 10 | +dev\_template & arith, bind, fsm, excl, sched & 3,4 & 1,2,3,4 & 2,373 & 27\\ | |
| 11 | +dev\_depth & arith, bind, fsm, excl, sched & 0,1,2 & 5,6 & 1,800 & 0\\ | |
| 12 | +dev\_compose & compose & 0,1,2 & 3,4,5 & 540 & 0\\ | |
| 13 | +test\_iid & arith, bind, fsm, excl, sched & 0,1,2 & 1,2,3,4 & 3,316 & 284\\ | |
| 14 | +test\_template & arith, bind, fsm, excl, sched & 3,4 & 1,2,3,4 & 2,328 & 72\\ | |
| 15 | +test\_depth & arith, bind, fsm, excl, sched & 0,1,2 & 5,6 & 1,800 & 0\\ | |
| 16 | +test\_compose & compose & 0,1,2 & 3,4,5 & 540 & 0\\ | |
| 17 | +\bottomrule\end{tabular}\end{table} | |
added
paper/tables/numbers.tex
+24 −0
@@ -0,0 +1,24 @@ | ||
| 1 | +% AUTO-GENERATED by scripts/make_tables.py — do not edit | |
| 2 | +\newcommand{\costpair}{0.100} | |
| 3 | +\newcommand{\costsingle}{0.080} | |
| 4 | +\newcommand{\labelscaliballneg}{30} | |
| 5 | +\newcommand{\labelscaliballpos}{38} | |
| 6 | +\newcommand{\labelscalibnstates}{1,308} | |
| 7 | +\newcommand{\labelscaliboracle}{0.0166} | |
| 8 | +\newcommand{\labelscalibrandom}{0.0004} | |
| 9 | +\newcommand{\labelscalibstoponly}{0.0098} | |
| 10 | +\newcommand{\labelscalibwithin}{11} | |
| 11 | +\newcommand{\labelstrainallneg}{29} | |
| 12 | +\newcommand{\labelstrainallpos}{37} | |
| 13 | +\newcommand{\labelstrainnstates}{4,778} | |
| 14 | +\newcommand{\labelstrainoracle}{0.0148} | |
| 15 | +\newcommand{\labelstrainrandom}{0.0049} | |
| 16 | +\newcommand{\labelstrainstoponly}{0.0083} | |
| 17 | +\newcommand{\labelstrainwithin}{14} | |
| 18 | +\newcommand{\ntrainsynth}{23,005} | |
| 19 | +\newcommand{\paramsLargeTotal}{1,170,995,461} | |
| 20 | +\newcommand{\paramsLargeWs}{21,655,813} | |
| 21 | +\newcommand{\paramsMediumTotal}{457,249,285} | |
| 22 | +\newcommand{\paramsMediumWs}{10,505,221} | |
| 23 | +\newcommand{\paramsSmallTotal}{102,134,021} | |
| 24 | +\newcommand{\paramsSmallWs}{2,631,173} | |
added
paper/tables/ops.tex
+14 −0
@@ -0,0 +1,14 @@ | ||
| 1 | +\begin{table}[t]\centering\small | |
| 2 | +\caption{Analytic operations per generated token, PRISME-459M, context $T=4096$ (\texttt{scripts/estimate\_flops.py}). | |
| 3 | +``Proposal'' is the reference table of the original brief; ``implemented'' is the accounting of | |
| 4 | +Section~\ref{sec:compute} with the controller evaluated once per executed action plus once to stop. These are | |
| 5 | +arithmetic counts, not measured timings.} | |
| 6 | +\label{tab:ops} | |
| 7 | +\begin{tabular}{lrrrrr}\toprule | |
| 8 | +Policy & Proposal ($10^9$) & Implemented ($10^9$) & editor & controller & readout\\\midrule | |
| 9 | +Backbone only & 1.296 & 1.296 & 0.000 & 0.000 & 0.000\\ | |
| 10 | +4 actions, 1 slot each & 1.645 & 1.666 & 0.256 & 0.090 & 0.019\\ | |
| 11 | +4 actions, 2 slots each & 1.767 & 1.787 & 0.378 & 0.090 & 0.019\\ | |
| 12 | +8 actions, 2 slots each & 2.195 & 2.219 & 0.756 & 0.143 & 0.019\\ | |
| 13 | +8 actions, dense 16-slot writes & 5.608 & 5.633 & 4.169 & 0.143 & 0.019\\ | |
| 14 | +\bottomrule\end{tabular}\end{table} | |
added
paper/uq_logo.jpg
+0 −0
Binary file not shown.
modified
prisme/config.py
+3 −0
@@ -38,9 +38,12 @@ class PrismeConfig: | ||
| 38 | 38 | clip_c: float = 1.0 |
| 39 | 39 | clip_eps: float = 1e-6 |
| 40 | 40 | readout_zero_init: bool = True |
| 41 | + readout_init_std: float = 0.02 # used only when readout_zero_init is False (bootstrap ablation) | |
| 41 | 42 | readout_scale: float = 1.0 # fixed (non-learned) multiplier on U(...); set to the backbone residual rms at bootstrap (D9) |
| 42 | 43 | # --- controller --------------------------------------------------------- |
| 43 | 44 | controller_hidden: int = 0 # 0 -> ffn_w |
| 45 | + ctrl_input: str = "z" # "z" (w-dim summary) or "h" (normalised backbone residual, d-dim; costs 2dw per token) | |
| 46 | + ctrl_use_entropy: bool = False # add the backbone's own predictive entropy at budget 0 (costs one head evaluation per token) | |
| 44 | 47 | lambda_cost: float = 0.0 |
| 45 | 48 | kappa_risk: float = 0.0 |
| 46 | 49 | # --- dense-workspace ablation (writes all slots) -------------------------- |
modified
prisme/controller.py
+11 −5
@@ -26,7 +26,8 @@ class ValueController(nn.Module): | ||
| 26 | 26 | self.cfg = cfg |
| 27 | 27 | hidden = cfg.controller_hidden or cfg.ffn_w |
| 28 | 28 | w = cfg.w |
| 29 | − self.z_proj = nn.Linear(w, w, bias=False) | |
| 29 | + self.z_proj = nn.Linear(cfg.d_model if cfg.ctrl_input == "h" else w, w, bias=False) | |
| 30 | + self.ent_proj = nn.Linear(2, w, bias=False) if cfg.ctrl_use_entropy else None | |
| 30 | 31 | self.ctx_proj = nn.Linear(self.N_CTX, w, bias=True) |
| 31 | 32 | self.norm = RMSNorm(w, cfg.norm_eps) |
| 32 | 33 | self.mlp = SwiGLU(w, hidden) |
@@ -36,9 +37,14 @@ class ValueController(nn.Module): | ||
| 36 | 37 | self.register_buffer("actions", action_table(cfg), persistent=False) |
| 37 | 38 | self.register_buffer("group_size", action_group_size(cfg), persistent=False) |
| 38 | 39 | |
| 39 | − def state_repr(self, z: torch.Tensor, step: torch.Tensor, remaining: torch.Tensor) -> torch.Tensor: | |
| 40 | + def state_repr(self, z: torch.Tensor, step: torch.Tensor, remaining: torch.Tensor, entropy=None) -> torch.Tensor: | |
| 41 | + """z: [B,Tq,w] summary (or [B,Tq,d] normalised residual when ctrl_input == "h"); entropy: [B,Tq] optional.""" | |
| 40 | 42 | ctx = torch.stack([step.to(z.dtype) / self.cfg.r_max, remaining.to(z.dtype) / self.cfg.r_max], -1) |
| 41 | 43 | u = self.z_proj(z) + self.ctx_proj(ctx) |
| 44 | + if self.ent_proj is not None: | |
| 45 | + assert entropy is not None, "ctrl_use_entropy requires the backbone entropy feature" | |
| 46 | + e = entropy.to(z.dtype).unsqueeze(-1) | |
| 47 | + u = u + self.ent_proj(torch.cat([e, torch.log(e + 1e-4)], -1)) | |
| 42 | 48 | return u + self.mlp(self.norm(u)) # [B,Tq,w] |
| 43 | 49 | |
| 44 | 50 | def slot_summaries(self, S: torch.Tensor) -> torch.Tensor: |
@@ -48,10 +54,10 @@ class ValueController(nn.Module): | ||
| 48 | 54 | sel = S[..., acts.clamp(min=0), :] # [B,Tq,A,2,w] |
| 49 | 55 | return (sel * amask.view(1, 1, A, 2, 1)).sum(-2) / amask.sum(-1).view(1, 1, A, 1) |
| 50 | 56 | |
| 51 | − def forward(self, z, S, step, remaining): | |
| 52 | − """z: [B,Tq,w]; S: [B,Tq,M,w]; step/remaining: [B,Tq] -> dict of [B,Tq,A] predictions.""" | |
| 57 | + def forward(self, z, S, step, remaining, entropy=None): | |
| 58 | + """z: controller input ([B,Tq,w] summary or [B,Tq,d] residual); S: [B,Tq,M,w]; step/remaining: [B,Tq].""" | |
| 53 | 59 | A = self.cfg.n_actions |
| 54 | − u = self.state_repr(z, step, remaining).unsqueeze(2) # [B,Tq,1,w] | |
| 60 | + u = self.state_repr(z, step, remaining, entropy).unsqueeze(2) # [B,Tq,1,w] | |
| 55 | 61 | s = self.s_proj(self.slot_summaries(S)) # [B,Tq,A,w] |
| 56 | 62 | kidx = (self.group_size[:A] - 1).clamp(min=0) |
| 57 | 63 | h = F.silu(u + s + self.k_embed[kidx].view(1, 1, A, -1)) |
modified
prisme/flops.py
+9 −2
@@ -64,11 +64,18 @@ def editor_block_ops(c: PrismeConfig, k: int, T: int) -> dict[str, float]: | ||
| 64 | 64 | |
| 65 | 65 | def controller_step_ops(c: PrismeConfig) -> float: |
| 66 | 66 | w, hidden, A = c.w, (c.controller_hidden or c.ffn_w), c.n_actions |
| 67 | − shared = 2 * w * w + 2 * 2 * w + 6 * w * hidden | |
| 67 | + din = c.d_model if c.ctrl_input == "h" else w | |
| 68 | + shared = 2 * din * w + 2 * 2 * w + 6 * w * hidden + (2 * 2 * w if c.ctrl_use_entropy else 0) | |
| 68 | 69 | per_action = 2 * w * w + 2 * 4 * w |
| 69 | 70 | return shared + A * per_action |
| 70 | 71 | |
| 71 | 72 | |
| 73 | +def controller_token_ops(c: PrismeConfig) -> float: | |
| 74 | + """Once-per-token controller features: the backbone entropy needs one head evaluation (budget 0 would need it | |
| 75 | + anyway only if the token stops immediately; we charge it fully, conservatively).""" | |
| 76 | + return 2 * c.d_model * c.vocab_size if c.ctrl_use_entropy else 0.0 | |
| 77 | + | |
| 78 | + | |
| 72 | 79 | def readout_ops(c: PrismeConfig) -> float: |
| 73 | 80 | w, d, M = c.w, c.d_model, c.n_slots |
| 74 | 81 | return 2 * w * w + 2 * (2 * M * w * w + 4 * M * w) + 2 * w * d |
@@ -90,7 +97,7 @@ def ops_per_token(c: PrismeConfig, T: int, n_actions: int, k: int, use_controlle | ||
| 90 | 97 | gate = n_actions * 2 * k * 2 * c.w |
| 91 | 98 | steps = (n_actions + 1 if controller_steps is None else controller_steps) if use_controller else 0 |
| 92 | 99 | steps = min(steps, c.r_max) |
| 93 | − ctrl = steps * controller_step_ops(c) | |
| 100 | + ctrl = steps * controller_step_ops(c) + controller_token_ops(c) | |
| 94 | 101 | return OpsBreakdown(bb_lin, bb_att, head, bank, nb * blk["source_attn"], nb * blk["slot_attn"], nb * blk["ffn"], |
| 95 | 102 | gate, ctrl, readout_ops(c)) |
| 96 | 103 | |
modified
prisme/generation.py
+39 −2
@@ -39,7 +39,12 @@ def step_logits(model: PrismeModel, tokens_new: torch.Tensor, cache: RequestBatc | ||
| 39 | 39 | bank_mask = key_pos.view(1, 1, Tb) <= qpos.unsqueeze(-1) |
| 40 | 40 | from .model import EncodedContext |
| 41 | 41 | from .workspace import init_workspace |
| 42 | − ctx = EncodedContext(H=H, K=cache.bank.K, V=cache.bank.V, z=z, bank_mask=bank_mask, S0=init_workspace(model.slot_embed, z)) | |
| 42 | + entropy = None | |
| 43 | + if cfg.ctrl_use_entropy: | |
| 44 | + p = torch.softmax(model.backbone.head(H).float(), -1) | |
| 45 | + entropy = -(p * torch.log(p.clamp_min(1e-12))).sum(-1) | |
| 46 | + ctx = EncodedContext(H=H, K=cache.bank.K, V=cache.bank.V, z=z, bank_mask=bank_mask, S0=init_workspace(model.slot_embed, z), | |
| 47 | + ctrl_in=(hb if cfg.ctrl_input == "h" else z), entropy=entropy) | |
| 43 | 48 | S = ctx.S0 |
| 44 | 49 | R = min(max_budget, cfg.r_max) |
| 45 | 50 | trace = torch.full((B, n, R), cfg.stop_action, dtype=torch.long, device=H.device) |
@@ -51,7 +56,7 @@ def step_logits(model: PrismeModel, tokens_new: torch.Tensor, cache: RequestBatc | ||
| 51 | 56 | if forced_actions is not None: |
| 52 | 57 | a = forced_actions[:, :, r] |
| 53 | 58 | else: |
| 54 | − preds = model.controller(z, S, step, remaining) | |
| 59 | + preds = model.controller(ctx.ctrl_in, S, step, remaining, ctx.entropy) | |
| 55 | 60 | a = model.controller.select(preds, cost, lambda_cost=lambda_cost, kappa_risk=kappa_risk) |
| 56 | 61 | a = torch.where(stopped, torch.full_like(a, cfg.stop_action), a) |
| 57 | 62 | stopped = stopped | (a == cfg.stop_action) |
@@ -64,6 +69,36 @@ def step_logits(model: PrismeModel, tokens_new: torch.Tensor, cache: RequestBatc | ||
| 64 | 69 | return logits, trace, n_edits |
| 65 | 70 | |
| 66 | 71 | |
| 72 | +@torch.no_grad() | |
| 73 | +def generate_uncached(model, prompt: torch.Tensor, max_new_tokens: int, max_budget: int = 0, temperature: float = 0.0, | |
| 74 | + eos_id=None, forced_budget_actions=None, generator=None, **_): | |
| 75 | + """Fallback for models without the PRISME caches (e.g. the global-recurrence baseline): recompute the full | |
| 76 | + sequence at every step with forced actions for all positions. O(T^2) per token; used only for evaluation.""" | |
| 77 | + cfg = model.cfg | |
| 78 | + B, T0 = prompt.shape | |
| 79 | + seq = prompt | |
| 80 | + done = torch.zeros(B, dtype=torch.bool, device=prompt.device) | |
| 81 | + edits = [] | |
| 82 | + for i in range(max_new_tokens): | |
| 83 | + T = seq.shape[1] | |
| 84 | + acts = forced_budget_actions(B, T) if (forced_budget_actions is not None and max_budget > 0) else None | |
| 85 | + out = model(seq, acts) | |
| 86 | + nxt = out["logits"][:, -1] | |
| 87 | + if out.get("n_edits") is not None: | |
| 88 | + edits.append(out["n_edits"][:, -1]) | |
| 89 | + if temperature <= 0: | |
| 90 | + tok = nxt.argmax(-1, keepdim=True) | |
| 91 | + else: | |
| 92 | + tok = torch.multinomial(torch.softmax(nxt / temperature, -1), 1, generator=generator) | |
| 93 | + if eos_id is not None: | |
| 94 | + tok = torch.where(done.unsqueeze(-1), torch.full_like(tok, eos_id), tok) | |
| 95 | + done = done | (tok.squeeze(-1) == eos_id) | |
| 96 | + seq = torch.cat([seq, tok], 1) | |
| 97 | + if bool(done.all()): | |
| 98 | + break | |
| 99 | + return {"tokens": seq, "n_edits": torch.stack(edits, 1) if edits else None, "cache_bytes": 0} | |
| 100 | + | |
| 101 | + | |
| 67 | 102 | @torch.no_grad() |
| 68 | 103 | def generate(model: PrismeModel, prompt: torch.Tensor, max_new_tokens: int, max_budget: int = 0, |
| 69 | 104 | cost: Optional[torch.Tensor] = None, temperature: float = 0.0, eos_id: Optional[int] = None, |
@@ -72,6 +107,8 @@ def generate(model: PrismeModel, prompt: torch.Tensor, max_new_tokens: int, max_ | ||
| 72 | 107 | """Greedy (temperature 0) or sampled decoding. prompt: [B, T0]. Returns dict(tokens [B,T0+new], n_edits [B,new], |
| 73 | 108 | actions list). Prompt positions are processed with the same per-token routing (prefill).""" |
| 74 | 109 | cfg = model.cfg |
| 110 | + if not hasattr(model, "W_K") and getattr(cfg, "use_workspace", False): | |
| 111 | + return generate_uncached(model, prompt, max_new_tokens, max_budget, temperature, eos_id, forced_budget_actions, generator) | |
| 75 | 112 | cache = RequestBatchCache(layers=model.backbone.new_cache()) |
| 76 | 113 | B, T0 = prompt.shape |
| 77 | 114 | fa = forced_budget_actions(B, T0) if forced_budget_actions is not None else None |
modified
prisme/model.py
+10 −2
@@ -34,6 +34,8 @@ class EncodedContext: | ||
| 34 | 34 | z: torch.Tensor # [B,Tq,w] |
| 35 | 35 | bank_mask: torch.Tensor # [B,Tq,Tb] bool |
| 36 | 36 | S0: torch.Tensor # [B,Tq,M,w] |
| 37 | + ctrl_in: Optional[torch.Tensor] = None # controller input ([B,Tq,w] z or [B,Tq,d] normalised H) | |
| 38 | + entropy: Optional[torch.Tensor] = None # [B,Tq] backbone predictive entropy at budget 0 (if cfg.ctrl_use_entropy) | |
| 37 | 39 | |
| 38 | 40 | |
| 39 | 41 | class PrismeModel(nn.Module): |
@@ -90,7 +92,13 @@ class PrismeModel(nn.Module): | ||
| 90 | 92 | if bank_len is not None: |
| 91 | 93 | bank_mask = bank_mask & (key_pos.view(1, 1, Tb) < bank_len.view(B, 1, 1)) |
| 92 | 94 | S0 = init_workspace(self.slot_embed, z) |
| 93 | − return EncodedContext(H=H_query, K=K, V=V, z=z, bank_mask=bank_mask, S0=S0) | |
| 95 | + hq = self.bank_norm(H_query) | |
| 96 | + ctrl_in = hq if self.cfg.ctrl_input == "h" else z | |
| 97 | + entropy = None | |
| 98 | + if self.cfg.ctrl_use_entropy: | |
| 99 | + p = torch.softmax(self.backbone.head(H_query).float(), -1) | |
| 100 | + entropy = -(p * torch.log(p.clamp_min(1e-12))).sum(-1) | |
| 101 | + return EncodedContext(H=H_query, K=K, V=V, z=z, bank_mask=bank_mask, S0=S0, ctrl_in=ctrl_in, entropy=entropy) | |
| 94 | 102 | |
| 95 | 103 | # ------------------------------------------------------------------ one action |
| 96 | 104 | def apply_action(self, S: torch.Tensor, ctx: EncodedContext, action: torch.Tensor, step, remaining): |
@@ -161,7 +169,7 @@ class PrismeModel(nn.Module): | ||
| 161 | 169 | for r in range(max_budget): |
| 162 | 170 | step = torch.full((B, T), r, device=tokens.device, dtype=torch.long) |
| 163 | 171 | remaining = torch.full((B, T), max_budget - r - 1, device=tokens.device, dtype=torch.long) |
| 164 | − preds = self.controller(ctx.z, S, step, remaining) | |
| 172 | + preds = self.controller(ctx.ctrl_in, S, step, remaining, ctx.entropy) | |
| 165 | 173 | a = self.controller.select(preds, cost, lambda_cost=lambda_cost, kappa_risk=kappa_risk) |
| 166 | 174 | a = torch.where(stopped, torch.full_like(a, cfg.stop_action), a) |
| 167 | 175 | stopped = stopped | (a == cfg.stop_action) |
modified
prisme/readout.py
+1 −1
@@ -19,7 +19,7 @@ class DifferentialReadout(nn.Module): | ||
| 19 | 19 | if cfg.readout_zero_init: |
| 20 | 20 | nn.init.zeros_(self.U.weight) |
| 21 | 21 | else: |
| 22 | − nn.init.normal_(self.U.weight, std=cfg.init_std) | |
| 22 | + nn.init.normal_(self.U.weight, std=cfg.readout_init_std) | |
| 23 | 23 | |
| 24 | 24 | def read(self, S: torch.Tensor, z: torch.Tensor) -> torch.Tensor: |
| 25 | 25 | """Attention read over slots with a query that depends only on z. S: [...,M,w]; z: [...,w] -> [...,w].""" |
added
reproducibility/environment.md
+12 −0
@@ -0,0 +1,12 @@ | ||
| 1 | +# Environment (WP31) | |
| 2 | + | |
| 3 | +* Laptop (development, Phase 4): Apple M5 Max, 18 CPU cores, 40 GPU cores, 48 GB unified memory, macOS 27.0, | |
| 4 | + Python 3.12 (uv), PyTorch 2.15.0 (MPS + CPU), MLX 0.32 available but not used by the reference implementation. | |
| 5 | +* MacLustr nodes used (existing passwordless SSH on the LAN; read-only audit in `hardware/device_inventory.json`): | |
| 6 | + M3U96a / M3U96b (Apple M3 Ultra, 32 cores, 60 GPU cores, 96 GB) for Stage A pretraining; M2U64 (M2 Ultra, 64 GB), | |
| 7 | + M4M64a/c (M4 Max, 64 GB) for editor/controller runs. Each node has its own venv built by `scripts/cluster_setup.sh` | |
| 8 | + (same lock file). No CUDA device was available; no cloud or paid service was used. | |
| 9 | +* Precision: fp32 everywhere (bf16 autocast gave no throughput gain on MPS: 15.3k vs 15.0k tok/s). | |
| 10 | +* Determinism: seeds fixed (`training.common.seed_all`); MPS kernels are not bit-deterministic across devices, so | |
| 11 | + cross-device equivalence tests use tolerances (1e-4) while bypass identity tests are bitwise on CPU. | |
| 12 | +* Dependency lock: `requirements-lock.txt` (uv pip freeze, 2026-10-29). | |
added
results/analyzed/labels_labels_p4_seed01_20261030000543.json
+236 −0
@@ -0,0 +1,236 @@ | ||
| 1 | +{ | |
| 2 | + "train": { | |
| 3 | + "n_labels": 80632, | |
| 4 | + "mean": 0.0041654990054667, | |
| 5 | + "std": 0.04521806538105011, | |
| 6 | + "median": 0.0, | |
| 7 | + "frac_pos": 0.4210983232463538, | |
| 8 | + "frac_neg": 0.3748635777358865, | |
| 9 | + "frac_abs_gt_0.01": 0.18173925984720707, | |
| 10 | + "frac_abs_gt_0.1": 0.02340261930747098, | |
| 11 | + "q05": -0.013343626633286476, | |
| 12 | + "q95": 0.03692528232932091, | |
| 13 | + "by_prefix_depth": { | |
| 14 | + "0": { | |
| 15 | + "mean": 0.012076183222234249, | |
| 16 | + "frac_pos": 0.5065833148926754, | |
| 17 | + "n": 18076 | |
| 18 | + }, | |
| 19 | + "1": { | |
| 20 | + "mean": 0.00342423003166914, | |
| 21 | + "frac_pos": 0.44935579781962337, | |
| 22 | + "n": 20180 | |
| 23 | + }, | |
| 24 | + "2": { | |
| 25 | + "mean": 0.0010541443480178714, | |
| 26 | + "frac_pos": 0.3993163724472136, | |
| 27 | + "n": 23112 | |
| 28 | + }, | |
| 29 | + "3": { | |
| 30 | + "mean": 0.001252029906027019, | |
| 31 | + "frac_pos": 0.3374169435215947, | |
| 32 | + "n": 19264 | |
| 33 | + } | |
| 34 | + }, | |
| 35 | + "audit": { | |
| 36 | + "n_states": 4778, | |
| 37 | + "n_actions": 24, | |
| 38 | + "var_total": 0.0025794953107833862, | |
| 39 | + "var_between_states": 0.0022299366537481546, | |
| 40 | + "var_within_state_actions": 0.0003495587734505534, | |
| 41 | + "within_share": 0.13551440834999084, | |
| 42 | + "oracle_one_step_mean": 0.014776400290429592, | |
| 43 | + "random_action_mean": 0.004916466772556305, | |
| 44 | + "stop_mean": 0.0, | |
| 45 | + "frac_states_any_positive": 0.46400167434072837, | |
| 46 | + "frac_states_all_positive": 0.37442444537463376, | |
| 47 | + "frac_states_all_negative": 0.2888237756383424, | |
| 48 | + "mean_rank_corr_between_actions": null, | |
| 49 | + "per_action_mean": [ | |
| 50 | + 0.004357533063739538, | |
| 51 | + 0.004356973338872194, | |
| 52 | + 0.004725269973278046, | |
| 53 | + 0.004466312006115913, | |
| 54 | + 0.004595508798956871, | |
| 55 | + 0.004644186235964298, | |
| 56 | + 0.004458694253116846, | |
| 57 | + 0.004305546171963215, | |
| 58 | + 0.004313195589929819, | |
| 59 | + 0.004399421159178019, | |
| 60 | + 0.004569229204207659, | |
| 61 | + 0.0045004794374108315, | |
| 62 | + 0.004307476803660393, | |
| 63 | + 0.004430056549608707, | |
| 64 | + 0.0043142689391970634, | |
| 65 | + 0.004313077311962843, | |
| 66 | + 0.0057276408188045025, | |
| 67 | + 0.006037583574652672, | |
| 68 | + 0.006085994653403759, | |
| 69 | + 0.005718533881008625, | |
| 70 | + 0.005754878278821707, | |
| 71 | + 0.006051815114915371, | |
| 72 | + 0.005772219970822334, | |
| 73 | + 0.005789478775113821 | |
| 74 | + ], | |
| 75 | + "per_action_frac_best": [ | |
| 76 | + 1685, | |
| 77 | + 107, | |
| 78 | + 157, | |
| 79 | + 43, | |
| 80 | + 105, | |
| 81 | + 96, | |
| 82 | + 69, | |
| 83 | + 43, | |
| 84 | + 131, | |
| 85 | + 207, | |
| 86 | + 68, | |
| 87 | + 27, | |
| 88 | + 58, | |
| 89 | + 116, | |
| 90 | + 33, | |
| 91 | + 45, | |
| 92 | + 270, | |
| 93 | + 250, | |
| 94 | + 476, | |
| 95 | + 211, | |
| 96 | + 184, | |
| 97 | + 201, | |
| 98 | + 98, | |
| 99 | + 98 | |
| 100 | + ], | |
| 101 | + "base_loss_mean": 0.2843226194381714, | |
| 102 | + "oracle_gain_relative_to_base": 0.05197054147720337, | |
| 103 | + "single_vs_pair_mean": { | |
| 104 | + "single": 0.004441073164343834, | |
| 105 | + "pair": 0.005867253057658672 | |
| 106 | + }, | |
| 107 | + "winners_curse_check_split_half": null, | |
| 108 | + "consecutive_state_best_action_agreement": 0.1149256855767218, | |
| 109 | + "oracle_state_level_mean": 0.008302818052470684, | |
| 110 | + "oracle_action_selection_increment": 0.006473582237958908, | |
| 111 | + "random_with_perfect_stop_mean": 0.008302818052470684, | |
| 112 | + "chance_agreement": 0.041666666666666664, | |
| 113 | + "oracle_gain_high_vs_low_base_loss": { | |
| 114 | + "high": 0.029565006494522095, | |
| 115 | + "low": 1.711727009023889e-07 | |
| 116 | + } | |
| 117 | + } | |
| 118 | + }, | |
| 119 | + "calib": { | |
| 120 | + "n_labels": 20040, | |
| 121 | + "mean": 0.0005741286440752447, | |
| 122 | + "std": 0.0787554383277893, | |
| 123 | + "median": 0.0, | |
| 124 | + "frac_pos": 0.4181137724550898, | |
| 125 | + "frac_neg": 0.37619760479041914, | |
| 126 | + "frac_abs_gt_0.01": 0.21656686626746507, | |
| 127 | + "frac_abs_gt_0.1": 0.04151696606786427, | |
| 128 | + "q05": -0.028652191162109375, | |
| 129 | + "q95": 0.04415319859981537, | |
| 130 | + "by_prefix_depth": { | |
| 131 | + "0": { | |
| 132 | + "mean": 0.008433588780462742, | |
| 133 | + "frac_pos": 0.48285619174434086, | |
| 134 | + "n": 6008 | |
| 135 | + }, | |
| 136 | + "1": { | |
| 137 | + "mean": -0.004564862698316574, | |
| 138 | + "frac_pos": 0.4384375, | |
| 139 | + "n": 6400 | |
| 140 | + }, | |
| 141 | + "2": { | |
| 142 | + "mean": -0.0007328253122977912, | |
| 143 | + "frac_pos": 0.3738558352402746, | |
| 144 | + "n": 3496 | |
| 145 | + }, | |
| 146 | + "3": { | |
| 147 | + "mean": -0.0017858755309134722, | |
| 148 | + "frac_pos": 0.33002901353965186, | |
| 149 | + "n": 4136 | |
| 150 | + } | |
| 151 | + }, | |
| 152 | + "audit": { | |
| 153 | + "n_states": 1308, | |
| 154 | + "n_actions": 24, | |
| 155 | + "var_total": 0.007274585776031017, | |
| 156 | + "var_between_states": 0.006477224640548229, | |
| 157 | + "var_within_state_actions": 0.0007973613683134317, | |
| 158 | + "within_share": 0.10960917919874191, | |
| 159 | + "oracle_one_step_mean": 0.01657850295305252, | |
| 160 | + "random_action_mean": 0.0003595371381379664, | |
| 161 | + "stop_mean": 0.0, | |
| 162 | + "frac_states_any_positive": 0.4709480122324159, | |
| 163 | + "frac_states_all_positive": 0.3769113149847095, | |
| 164 | + "frac_states_all_negative": 0.2958715596330275, | |
| 165 | + "mean_rank_corr_between_actions": null, | |
| 166 | + "per_action_mean": [ | |
| 167 | + -0.0005930604529567063, | |
| 168 | + 0.0007526387344114482, | |
| 169 | + 0.00037022720789536834, | |
| 170 | + 0.0010265340097248554, | |
| 171 | + 0.0013548287097364664, | |
| 172 | + 0.000778406101744622, | |
| 173 | + 0.001126126735471189, | |
| 174 | + 0.0014101429842412472, | |
| 175 | + 0.000936423079110682, | |
| 176 | + 0.000786297838203609, | |
| 177 | + 3.662590097519569e-05, | |
| 178 | + 4.5656248403247446e-05, | |
| 179 | + 0.00017277516599278897, | |
| 180 | + 0.0012526045320555568, | |
| 181 | + 0.0012178309261798859, | |
| 182 | + 0.0011301396880298853, | |
| 183 | + -0.0011659032898023725, | |
| 184 | + -0.0003716170904226601, | |
| 185 | + 5.648710794048384e-05, | |
| 186 | + 0.000264748465269804, | |
| 187 | + -0.00040692274342291057, | |
| 188 | + -0.0013585558626800776, | |
| 189 | + -0.0005096303066238761, | |
| 190 | + 0.00031610956648364663 | |
| 191 | + ], | |
| 192 | + "per_action_frac_best": [ | |
| 193 | + 451, | |
| 194 | + 31, | |
| 195 | + 24, | |
| 196 | + 7, | |
| 197 | + 30, | |
| 198 | + 23, | |
| 199 | + 20, | |
| 200 | + 13, | |
| 201 | + 51, | |
| 202 | + 57, | |
| 203 | + 8, | |
| 204 | + 13, | |
| 205 | + 14, | |
| 206 | + 38, | |
| 207 | + 10, | |
| 208 | + 14, | |
| 209 | + 69, | |
| 210 | + 72, | |
| 211 | + 143, | |
| 212 | + 46, | |
| 213 | + 55, | |
| 214 | + 60, | |
| 215 | + 29, | |
| 216 | + 30 | |
| 217 | + ], | |
| 218 | + "base_loss_mean": 0.442313551902771, | |
| 219 | + "oracle_gain_relative_to_base": 0.037481337785720825, | |
| 220 | + "single_vs_pair_mean": { | |
| 221 | + "single": 0.0007377619622275233, | |
| 222 | + "pair": -0.00039691125857643783 | |
| 223 | + }, | |
| 224 | + "winners_curse_check_split_half": null, | |
| 225 | + "consecutive_state_best_action_agreement": 0.117061973986228, | |
| 226 | + "oracle_state_level_mean": 0.009835593402385712, | |
| 227 | + "oracle_action_selection_increment": 0.006742909550666809, | |
| 228 | + "random_with_perfect_stop_mean": 0.009835593402385712, | |
| 229 | + "chance_agreement": 0.041666666666666664, | |
| 230 | + "oracle_gain_high_vs_low_base_loss": { | |
| 231 | + "high": 0.033156733959913254, | |
| 232 | + "low": 2.715730147428985e-07 | |
| 233 | + } | |
| 234 | + } | |
| 235 | + } | |
| 236 | +} | |
| \ No newline at end of file | ||
added
results/claim_registry.csv
+1 −0
@@ -0,0 +1 @@ | ||
| 1 | +claim,metric,value,benchmark,run_ids,artifact,analysis_script,status | |
added
results/figures/labels_analysis.pdf
+0 −0
Binary file not shown.
added
results/figures/labels_analysis.png
+0 −0
Binary file not shown.
added
results/figures/learnability_bootstrap.pdf
+0 −0
Binary file not shown.
added
results/figures/learnability_bootstrap.png
+0 −0
Binary file not shown.
added
results/raw/latency_p4_editor_v1_Ustd02_seed01_20261029235050_mps.json
+302 −0
@@ -0,0 +1,302 @@ | ||
| 1 | +{ | |
| 2 | + "config": "configs/experiments/latency_p4.yaml", | |
| 3 | + "checkpoint_sha256": "aabf646812efd9f1fdedbcccbf73004a44516484ad036798fdf3a7b80e5cefb0", | |
| 4 | + "device": "mps", | |
| 5 | + "torch": "2.15.0", | |
| 6 | + "precision": "fp32", | |
| 7 | + "prompt_len": 64, | |
| 8 | + "n_new": 32, | |
| 9 | + "code_commit": "a9d94e8a2b85c98bf5259c5029878fd47479c325", | |
| 10 | + "peak_memory_bytes": 102137856, | |
| 11 | + "analytic_ops_per_token": { | |
| 12 | + "budget0": 3493248, | |
| 13 | + "4_k1": 8504960.0, | |
| 14 | + "4_k2": 9587328.0, | |
| 15 | + "8_k2": 14722432.0, | |
| 16 | + "8_dense": 45028736.0 | |
| 17 | + }, | |
| 18 | + "records": [ | |
| 19 | + { | |
| 20 | + "policy": "budget0", | |
| 21 | + "batch": 1, | |
| 22 | + "budget": 0, | |
| 23 | + "median_ms": 2.8017294825986028, | |
| 24 | + "p95_ms": 3.1844482582528144, | |
| 25 | + "mean_ms": 2.759531239765541, | |
| 26 | + "n": 96, | |
| 27 | + "cache_bytes": 245760 | |
| 28 | + }, | |
| 29 | + { | |
| 30 | + "policy": "forced_single", | |
| 31 | + "batch": 1, | |
| 32 | + "budget": 1, | |
| 33 | + "forced_k": 1, | |
| 34 | + "median_ms": 5.252917006146163, | |
| 35 | + "p95_ms": 6.079749495256692, | |
| 36 | + "mean_ms": 5.123651044414146, | |
| 37 | + "n": 96, | |
| 38 | + "cache_bytes": 245760 | |
| 39 | + }, | |
| 40 | + { | |
| 41 | + "policy": "forced_pair", | |
| 42 | + "batch": 1, | |
| 43 | + "budget": 1, | |
| 44 | + "forced_k": 2, | |
| 45 | + "median_ms": 5.21864602342248, | |
| 46 | + "p95_ms": 6.357603764627129, | |
| 47 | + "mean_ms": 4.836453521420481, | |
| 48 | + "n": 96, | |
| 49 | + "cache_bytes": 245760 | |
| 50 | + }, | |
| 51 | + { | |
| 52 | + "policy": "forced_single", | |
| 53 | + "batch": 1, | |
| 54 | + "budget": 2, | |
| 55 | + "forced_k": 1, | |
| 56 | + "median_ms": 7.243604050017893, | |
| 57 | + "p95_ms": 9.018541488330811, | |
| 58 | + "mean_ms": 7.380335543227072, | |
| 59 | + "n": 96, | |
| 60 | + "cache_bytes": 245760 | |
| 61 | + }, | |
| 62 | + { | |
| 63 | + "policy": "forced_pair", | |
| 64 | + "batch": 1, | |
| 65 | + "budget": 2, | |
| 66 | + "forced_k": 2, | |
| 67 | + "median_ms": 7.464978960342705, | |
| 68 | + "p95_ms": 12.41436498821713, | |
| 69 | + "mean_ms": 8.055094188118042, | |
| 70 | + "n": 96, | |
| 71 | + "cache_bytes": 245760 | |
| 72 | + }, | |
| 73 | + { | |
| 74 | + "policy": "forced_single", | |
| 75 | + "batch": 1, | |
| 76 | + "budget": 4, | |
| 77 | + "forced_k": 1, | |
| 78 | + "median_ms": 11.287728964816779, | |
| 79 | + "p95_ms": 12.40296903415583, | |
| 80 | + "mean_ms": 10.915309430856723, | |
| 81 | + "n": 96, | |
| 82 | + "cache_bytes": 245760 | |
| 83 | + }, | |
| 84 | + { | |
| 85 | + "policy": "forced_pair", | |
| 86 | + "batch": 1, | |
| 87 | + "budget": 4, | |
| 88 | + "forced_k": 2, | |
| 89 | + "median_ms": 11.351562512572855, | |
| 90 | + "p95_ms": 12.37730251159519, | |
| 91 | + "mean_ms": 11.322401035916604, | |
| 92 | + "n": 96, | |
| 93 | + "cache_bytes": 245760 | |
| 94 | + }, | |
| 95 | + { | |
| 96 | + "policy": "forced_single", | |
| 97 | + "batch": 1, | |
| 98 | + "budget": 8, | |
| 99 | + "forced_k": 1, | |
| 100 | + "median_ms": 19.331770483404398, | |
| 101 | + "p95_ms": 21.22909395257011, | |
| 102 | + "mean_ms": 19.320005204159923, | |
| 103 | + "n": 96, | |
| 104 | + "cache_bytes": 245760 | |
| 105 | + }, | |
| 106 | + { | |
| 107 | + "policy": "forced_pair", | |
| 108 | + "batch": 1, | |
| 109 | + "budget": 8, | |
| 110 | + "forced_k": 2, | |
| 111 | + "median_ms": 18.950854428112507, | |
| 112 | + "p95_ms": 20.537573233013973, | |
| 113 | + "mean_ms": 19.09837671701098, | |
| 114 | + "n": 96, | |
| 115 | + "cache_bytes": 245760 | |
| 116 | + }, | |
| 117 | + { | |
| 118 | + "policy": "controller", | |
| 119 | + "batch": 1, | |
| 120 | + "budget": 8, | |
| 121 | + "median_ms": 5.7201660820283, | |
| 122 | + "p95_ms": 23.911051946925, | |
| 123 | + "mean_ms": 9.350690118784163, | |
| 124 | + "n": 96, | |
| 125 | + "cache_bytes": 245760 | |
| 126 | + }, | |
| 127 | + { | |
| 128 | + "policy": "dense_forced", | |
| 129 | + "batch": 1, | |
| 130 | + "budget": 1, | |
| 131 | + "forced_k": 1, | |
| 132 | + "median_ms": 4.747916478663683, | |
| 133 | + "p95_ms": 8.084010769380257, | |
| 134 | + "mean_ms": 4.76325996714877, | |
| 135 | + "n": 96, | |
| 136 | + "cache_bytes": 245760 | |
| 137 | + }, | |
| 138 | + { | |
| 139 | + "policy": "dense_forced", | |
| 140 | + "batch": 1, | |
| 141 | + "budget": 4, | |
| 142 | + "forced_k": 1, | |
| 143 | + "median_ms": 10.018958011642098, | |
| 144 | + "p95_ms": 11.016781529178843, | |
| 145 | + "mean_ms": 10.038378921308322, | |
| 146 | + "n": 96, | |
| 147 | + "cache_bytes": 245760 | |
| 148 | + }, | |
| 149 | + { | |
| 150 | + "policy": "dense_forced", | |
| 151 | + "batch": 1, | |
| 152 | + "budget": 8, | |
| 153 | + "forced_k": 1, | |
| 154 | + "median_ms": 17.32618745882064, | |
| 155 | + "p95_ms": 27.340614731656387, | |
| 156 | + "mean_ms": 17.98606334341457, | |
| 157 | + "n": 96, | |
| 158 | + "cache_bytes": 245760 | |
| 159 | + }, | |
| 160 | + { | |
| 161 | + "policy": "budget0", | |
| 162 | + "batch": 8, | |
| 163 | + "budget": 0, | |
| 164 | + "median_ms": 3.36672953562811, | |
| 165 | + "p95_ms": 3.923937038052827, | |
| 166 | + "mean_ms": 3.3720702607145845, | |
| 167 | + "n": 96, | |
| 168 | + "cache_bytes": 1966080 | |
| 169 | + }, | |
| 170 | + { | |
| 171 | + "policy": "forced_single", | |
| 172 | + "batch": 8, | |
| 173 | + "budget": 1, | |
| 174 | + "forced_k": 1, | |
| 175 | + "median_ms": 6.341208005324006, | |
| 176 | + "p95_ms": 10.401958308648318, | |
| 177 | + "mean_ms": 6.62630293663824, | |
| 178 | + "n": 96, | |
| 179 | + "cache_bytes": 1966080 | |
| 180 | + }, | |
| 181 | + { | |
| 182 | + "policy": "forced_pair", | |
| 183 | + "batch": 8, | |
| 184 | + "budget": 1, | |
| 185 | + "forced_k": 2, | |
| 186 | + "median_ms": 6.1652500298805535, | |
| 187 | + "p95_ms": 7.15568705345504, | |
| 188 | + "mean_ms": 6.102194831328234, | |
| 189 | + "n": 96, | |
| 190 | + "cache_bytes": 1966080 | |
| 191 | + }, | |
| 192 | + { | |
| 193 | + "policy": "forced_single", | |
| 194 | + "batch": 8, | |
| 195 | + "budget": 2, | |
| 196 | + "forced_k": 1, | |
| 197 | + "median_ms": 8.590396028012037, | |
| 198 | + "p95_ms": 14.236905990401283, | |
| 199 | + "mean_ms": 8.708206226098506, | |
| 200 | + "n": 96, | |
| 201 | + "cache_bytes": 1966080 | |
| 202 | + }, | |
| 203 | + { | |
| 204 | + "policy": "forced_pair", | |
| 205 | + "batch": 8, | |
| 206 | + "budget": 2, | |
| 207 | + "forced_k": 2, | |
| 208 | + "median_ms": 8.381562540307641, | |
| 209 | + "p95_ms": 9.846666041994467, | |
| 210 | + "mean_ms": 8.429156674537808, | |
| 211 | + "n": 96, | |
| 212 | + "cache_bytes": 1966080 | |
| 213 | + }, | |
| 214 | + { | |
| 215 | + "policy": "forced_single", | |
| 216 | + "batch": 8, | |
| 217 | + "budget": 4, | |
| 218 | + "forced_k": 1, | |
| 219 | + "median_ms": 12.963792018126696, | |
| 220 | + "p95_ms": 14.19885375071317, | |
| 221 | + "mean_ms": 12.95540061983047, | |
| 222 | + "n": 96, | |
| 223 | + "cache_bytes": 1966080 | |
| 224 | + }, | |
| 225 | + { | |
| 226 | + "policy": "forced_pair", | |
| 227 | + "batch": 8, | |
| 228 | + "budget": 4, | |
| 229 | + "forced_k": 2, | |
| 230 | + "median_ms": 13.345645973458886, | |
| 231 | + "p95_ms": 20.811073016375303, | |
| 232 | + "mean_ms": 13.279389810728995, | |
| 233 | + "n": 96, | |
| 234 | + "cache_bytes": 1966080 | |
| 235 | + }, | |
| 236 | + { | |
| 237 | + "policy": "forced_single", | |
| 238 | + "batch": 8, | |
| 239 | + "budget": 8, | |
| 240 | + "forced_k": 1, | |
| 241 | + "median_ms": 22.172499971929938, | |
| 242 | + "p95_ms": 26.029041764559224, | |
| 243 | + "mean_ms": 19.927763494100265, | |
| 244 | + "n": 96, | |
| 245 | + "cache_bytes": 1966080 | |
| 246 | + }, | |
| 247 | + { | |
| 248 | + "policy": "forced_pair", | |
| 249 | + "batch": 8, | |
| 250 | + "budget": 8, | |
| 251 | + "forced_k": 2, | |
| 252 | + "median_ms": 22.192937438376248, | |
| 253 | + "p95_ms": 24.253927753306925, | |
| 254 | + "mean_ms": 21.646448845179595, | |
| 255 | + "n": 96, | |
| 256 | + "cache_bytes": 1966080 | |
| 257 | + }, | |
| 258 | + { | |
| 259 | + "policy": "controller", | |
| 260 | + "batch": 8, | |
| 261 | + "budget": 8, | |
| 262 | + "median_ms": 24.369208491407335, | |
| 263 | + "p95_ms": 26.040082942927256, | |
| 264 | + "mean_ms": 21.67698220121868, | |
| 265 | + "n": 96, | |
| 266 | + "cache_bytes": 1966080 | |
| 267 | + }, | |
| 268 | + { | |
| 269 | + "policy": "dense_forced", | |
| 270 | + "batch": 8, | |
| 271 | + "budget": 1, | |
| 272 | + "forced_k": 1, | |
| 273 | + "median_ms": 5.198728991672397, | |
| 274 | + "p95_ms": 6.063812266802415, | |
| 275 | + "mean_ms": 5.117546953745962, | |
| 276 | + "n": 96, | |
| 277 | + "cache_bytes": 1966080 | |
| 278 | + }, | |
| 279 | + { | |
| 280 | + "policy": "dense_forced", | |
| 281 | + "batch": 8, | |
| 282 | + "budget": 4, | |
| 283 | + "forced_k": 1, | |
| 284 | + "median_ms": 10.355145495850593, | |
| 285 | + "p95_ms": 11.753489437978715, | |
| 286 | + "mean_ms": 10.392837262770627, | |
| 287 | + "n": 96, | |
| 288 | + "cache_bytes": 1966080 | |
| 289 | + }, | |
| 290 | + { | |
| 291 | + "policy": "dense_forced", | |
| 292 | + "batch": 8, | |
| 293 | + "budget": 8, | |
| 294 | + "forced_k": 1, | |
| 295 | + "median_ms": 17.042521038092673, | |
| 296 | + "p95_ms": 19.3497292639222, | |
| 297 | + "mean_ms": 16.37415804725606, | |
| 298 | + "n": 96, | |
| 299 | + "cache_bytes": 1966080 | |
| 300 | + } | |
| 301 | + ] | |
| 302 | +} | |
| \ No newline at end of file | ||
added
scripts/launch_103m.sh
+20 −0
@@ -0,0 +1,20 @@ | ||
| 1 | +#!/bin/bash | |
| 2 | +# Launch the 103M Stage B pilots once Stage A has completed on M3U96a. Usage: scripts/launch_103m.sh [steps] | |
| 3 | +set -e | |
| 4 | +cd "$(dirname "$0")/.." | |
| 5 | +RUN=$(ssh M3U96a 'ls -d ~/wp31/prisme-slm/experiments/runs/pretrain103m_seed01_*' | tail -1) | |
| 6 | +STATE=$(ssh M3U96a "grep -o '\"state\": \"[a-z_]*\"' $RUN/manifest.json") | |
| 7 | +echo "stage A run: $RUN ($STATE)" | |
| 8 | +[[ "$STATE" == *completed* ]] || { echo "Stage A not completed"; exit 1; } | |
| 9 | +REL=${RUN#*/prisme-slm/} | |
| 10 | +for c in editor_pilot_103m dense_pilot_103m gr_pilot_103m; do | |
| 11 | + sed -i '' "s|experiments/runs/PRETRAIN103M_RUN/checkpoint|$REL/checkpoint|" configs/experiments/$c.yaml | |
| 12 | +done | |
| 13 | +# copy the backbone checkpoint to the other pilot nodes | |
| 14 | +for n in M3U96b M2U64; do | |
| 15 | + ssh M3U96a "cd ~/wp31/prisme-slm && rsync -az --relative ./$REL/checkpoint $n:~/wp31/prisme-slm/" || rsync -az "M3U96a:$RUN/checkpoint" /tmp/ck103m/ && rsync -az --relative "/tmp/ck103m/." "$n:~/wp31/prisme-slm/$REL/" | |
| 16 | +done | |
| 17 | +for n in M3U96a M3U96b M2U64; do rsync -az --exclude .venv --exclude /data --exclude /experiments --exclude .git ./ $n:~/wp31/prisme-slm/; done | |
| 18 | +ssh M3U96a "cd ~/wp31/prisme-slm && nohup .venv/bin/python -m training.train_editor --config configs/experiments/editor_pilot_103m.yaml --device mps > experiments/logs/editor_pilot_103m_seed1.log 2>&1 < /dev/null & disown; echo launched editor" | |
| 19 | +ssh M3U96b "cd ~/wp31/prisme-slm && nohup .venv/bin/python -m training.train_editor --config configs/experiments/gr_pilot_103m.yaml --device mps > experiments/logs/gr_pilot_103m_seed1.log 2>&1 < /dev/null & disown; echo launched gr" | |
| 20 | +ssh M2U64 "cd ~/wp31/prisme-slm && nohup .venv/bin/python -m training.train_editor --config configs/experiments/dense_pilot_103m.yaml --device mps > experiments/logs/dense_pilot_103m_seed1.log 2>&1 < /dev/null & disown; echo launched dense" | |
added
scripts/launch_seed_matrix.sh
+35 −0
@@ -0,0 +1,35 @@ | ||
| 1 | +#!/bin/bash | |
| 2 | +# Launch the Phase 4 decisive seed matrix on the v2 tiny backbone: PRISME / global recurrence / dense, seeds 1..5. | |
| 3 | +# Usage: scripts/launch_seed_matrix.sh (edits configs in place from the v2 checkpoint, syncs, launches one run per node slot) | |
| 4 | +set -e | |
| 5 | +cd "$(dirname "$0")/.." | |
| 6 | +CK=$(ls -d experiments/runs/p4_tiny_backbone_v2_seed01_*/checkpoint | tail -1) | |
| 7 | +echo "backbone checkpoint: $CK" | |
| 8 | +.venv/bin/python - "$CK" <<'PY' | |
| 9 | +import sys, yaml | |
| 10 | +ck = sys.argv[1] | |
| 11 | +model = {"name": "tiny_p4v2", "vocab_size": 128, "d_model": 256, "n_layers": 6, "n_heads": 8, "n_kv_heads": 4, "ffn_width": 768, | |
| 12 | + "max_seq_len": 320, "w": 64, "ffn_w": 192, "n_slots": 16, "n_editor_heads": 2, "r_max": 8, | |
| 13 | + "readout_zero_init": False, "readout_init_std": 0.2} | |
| 14 | +base = {"seed": 1, "model": model, "init": {"backbone_checkpoint": ck, "freeze_backbone": True, "auto_readout_scale": True}, | |
| 15 | + "data": {"kind": "synthetic", "tokenizer": "char", "splits_dir": "data/splits/synthetic_v1", "loss_span": "answer", | |
| 16 | + "eval_splits": ["dev_iid", "dev_template", "dev_depth", "dev_compose"]}, | |
| 17 | + "budgets": [0, 1, 2, 4, 8], "budget_probs": [0.2] * 5, "budget_per": "position", | |
| 18 | + "train": {"batch_size": 32, "seq_len": 320, "steps": 3000, "lr": 1.0e-3, "warmup": 100, "weight_decay": 0.1, "grad_clip": 1.0, | |
| 19 | + "log_every": 50, "eval_every": 500, "eval_batches": 12, "ckpt_every": 3000}, | |
| 20 | + "em_budgets": [0, 1, 2, 8], "em_max_items": 1000} | |
| 21 | +for name, mode, extra in [("p4v2_prisme", "prisme", {}), ("p4v2_gr", "global_recurrence", {"baseline_kwargs": {"ffn_w": 640}}), ("p4v2_dense", "dense", {})]: | |
| 22 | + e = dict(base); e["name"] = name; e["mode"] = mode; e.update(extra) | |
| 23 | + yaml.safe_dump(e, open(f"configs/experiments/{name}.yaml", "w"), sort_keys=False) | |
| 24 | +print("configs written") | |
| 25 | +PY | |
| 26 | +# node slots: (node, config, seed) | |
| 27 | +SLOTS="M2U64:p4v2_prisme:1 M4M64c:p4v2_prisme:2 M4M64a:p4v2_prisme:3 M5M36:p4v2_prisme:4 m4mn:p4v2_prisme:5 M2U64:p4v2_gr:1 M4M64c:p4v2_gr:2 M4M64a:p4v2_gr:3 M5M36:p4v2_gr:4 m4mn:p4v2_gr:5 M2U64:p4v2_dense:1 M4M64c:p4v2_dense:2 M4M64a:p4v2_dense:3 M5M36:p4v2_dense:4 m4mn:p4v2_dense:5" | |
| 28 | +for n in M2U64 M4M64c M4M64a M5M36 m4mn; do | |
| 29 | + rsync -az --exclude .venv --exclude /data --exclude /experiments --exclude .git ./ $n:~/wp31/prisme-slm/ & | |
| 30 | +done; wait | |
| 31 | +for n in M2U64 M4M64c M4M64a M5M36 m4mn; do rsync -az --relative ./$CK $n:~/wp31/prisme-slm/ & done; wait | |
| 32 | +for slot in $SLOTS; do | |
| 33 | + IFS=: read -r node cfg seed <<< "$slot" | |
| 34 | + ssh $node "cd ~/wp31/prisme-slm && nohup .venv/bin/python -m training.train_editor --config configs/experiments/$cfg.yaml --seed $seed --device mps > experiments/logs/${cfg}_seed$seed.log 2>&1 < /dev/null & disown" && echo "launched $cfg seed $seed on $node" | |
| 35 | +done | |
added
scripts/make_results_tables.py
+174 −0
@@ -0,0 +1,174 @@ | ||
| 1 | +"""Results-dependent LaTeX tables + macros (called by scripts/make_tables.py). Reads run manifests/metrics under | |
| 2 | +experiments/runs (local copies), results/raw/*.json and results/analyzed/*.json. Every number is traceable to a run id.""" | |
| 3 | +from __future__ import annotations | |
| 4 | + | |
| 5 | +import glob, json, math | |
| 6 | +from collections import defaultdict | |
| 7 | +from pathlib import Path | |
| 8 | + | |
| 9 | +import numpy as np | |
| 10 | + | |
| 11 | +OUT = Path("paper/tables") | |
| 12 | +RUNS = "experiments/runs" | |
| 13 | + | |
| 14 | + | |
| 15 | +def _runs(prefix): | |
| 16 | + return sorted(d for d in glob.glob(f"{RUNS}/{prefix}_seed*") if Path(d, "manifest.json").exists()) | |
| 17 | + | |
| 18 | + | |
| 19 | +def _final_val(run): | |
| 20 | + rows = [json.loads(l) for l in open(Path(run) / "metrics.jsonl")] | |
| 21 | + last = [r for r in rows if any(k.startswith("val_loss_b") for k in r)] | |
| 22 | + return last[-1] if last else {} | |
| 23 | + | |
| 24 | + | |
| 25 | +def table_seed_matrix(macro): | |
| 26 | + """Phase 4 decisive matrix: PRISME vs global recurrence vs dense, 5 seeds, val loss by budget and exact match.""" | |
| 27 | + systems = [("p4v2_prisme", "PRISME sparse editor"), ("p4v2_gr", "Global recurrence (inspired)"), ("p4v2_dense", "Dense 16-slot writes")] | |
| 28 | + rows, raw = [], {} | |
| 29 | + for pre, lab in systems: | |
| 30 | + runs = [r for r in _runs(pre) if json.load(open(Path(r) / "manifest.json")).get("state") == "completed"] | |
| 31 | + if not runs: | |
| 32 | + continue | |
| 33 | + vals = defaultdict(list); ems = defaultdict(list) | |
| 34 | + for r in runs: | |
| 35 | + fv = _final_val(r) | |
| 36 | + for k, v in fv.items(): | |
| 37 | + if k.startswith("val_loss_b"): | |
| 38 | + vals[int(k[10:])].append(v) | |
| 39 | + fem = json.load(open(Path(r) / "manifest.json")).get("final_exact_match", {}) | |
| 40 | + for split, d in fem.items(): | |
| 41 | + for b, e in d.items(): | |
| 42 | + ems[(split, int(b))].append(e["accuracy"]) | |
| 43 | + raw[pre] = {"n_seeds": len(runs), "val": {b: v for b, v in vals.items()}, "em": {f"{s}_b{b}": v for (s, b), v in ems.items()}, "run_ids": [Path(r).name for r in runs]} | |
| 44 | + rows.append((lab, len(runs), vals, ems)) | |
| 45 | + if not rows: | |
| 46 | + return | |
| 47 | + Path("results/analyzed").mkdir(parents=True, exist_ok=True) | |
| 48 | + Path("results/analyzed/seed_matrix_v2.json").write_text(json.dumps(raw, indent=1)) | |
| 49 | + def ms(v): | |
| 50 | + v = np.asarray(v, float) | |
| 51 | + return f"{v.mean():.4f} $\\pm$ {v.std(ddof=1):.4f}" if len(v) > 1 else f"{v.mean():.4f}" | |
| 52 | + def pct(v): | |
| 53 | + v = 100 * np.asarray(v, float) | |
| 54 | + return f"{v.mean():.1f} $\\pm$ {v.std(ddof=1):.1f}" if len(v) > 1 else f"{v.mean():.1f}" | |
| 55 | + head = r"""\begin{table}[t]\centering\small | |
| 56 | +\caption{Phase 4 decisive comparison on the frozen 5.6M-parameter synthetic backbone (char-level): held-out | |
| 57 | +answer-span loss on dev-iid under random forced actions at fixed budgets, and free-generation exact match | |
| 58 | +(dev-iid, 1,000 stratified items). Mean $\pm$ SD over independent training seeds of the adapter; the backbone, | |
| 59 | +data, recipe and symmetry-breaking device are identical. Budget 0 is the frozen backbone for every system.} | |
| 60 | +\label{tab:seedmatrix} | |
| 61 | +\begin{tabular}{llrrrrr}\toprule | |
| 62 | +System & seeds & loss b=0 & loss b=1 & loss b=2 & loss b=4 & loss b=8\\\midrule | |
| 63 | +""" | |
| 64 | + body = "" | |
| 65 | + for lab, n, vals, ems in rows: | |
| 66 | + body += f"{lab} & {n} & " + " & ".join(ms(vals.get(b, [float('nan')])) for b in [0, 1, 2, 4, 8]) + r"\\" + "\n" | |
| 67 | + body += r"\midrule" + "\n" + r"\multicolumn{7}{l}{\emph{Exact match (\%), dev-iid}: b=0 / b=1 / b=2 / b=8}\\" + "\n" | |
| 68 | + for lab, n, vals, ems in rows: | |
| 69 | + cells = [pct(ems.get(("dev_iid", b), [float("nan")])) for b in [0, 1, 2, 8]] | |
| 70 | + body += f"{lab} & {n} & \\multicolumn{{5}}{{l}}{{{' / '.join(cells)}}}\\\\\n" | |
| 71 | + body += r"\multicolumn{7}{l}{\emph{Exact match (\%), held-out template / depth / compose at b=2}}\\" + "\n" | |
| 72 | + for lab, n, vals, ems in rows: | |
| 73 | + cells = [pct(ems.get((s, 2), [float("nan")])) for s in ["dev_template", "dev_depth", "dev_compose"]] | |
| 74 | + body += f"{lab} & {n} & \\multicolumn{{5}}{{l}}{{{' / '.join(cells)}}}\\\\\n" | |
| 75 | + (OUT / "seed_matrix.tex").write_text(head + body + r"\bottomrule\end{tabular}\end{table}" + "\n") | |
| 76 | + for pre, lab in systems: | |
| 77 | + if pre in raw: | |
| 78 | + v = raw[pre]["val"] | |
| 79 | + if 0 in v and 2 in v: | |
| 80 | + macro(f"{pre.replace('p4v2_', 'vtwo')}gainbtwo", f"{100 * (1 - np.mean(v[2]) / np.mean(v[0])):.2f}") | |
| 81 | + macro(f"{pre.replace('p4v2_', 'vtwo')}seeds", str(raw[pre]["n_seeds"])) | |
| 82 | + | |
| 83 | + | |
| 84 | +def table_routing(macro): | |
| 85 | + files = sorted(glob.glob("results/raw/routing_*.json")) | |
| 86 | + if not files: | |
| 87 | + return | |
| 88 | + d = json.load(open(files[-1])) | |
| 89 | + head = r"""\begin{table}[t]\centering\small | |
| 90 | +\caption{Routing comparison on dev-iid (tiny scale, editor checkpoint of the locked run): held-out answer-span loss, | |
| 91 | +executed edits per token and analytic extra operations per token (relative to the backbone). Oracles read the | |
| 92 | +target and are diagnostic ceilings, not deployable systems.} | |
| 93 | +\label{tab:routing} | |
| 94 | +\begin{tabular}{llrrr}\toprule | |
| 95 | +Policy & setting & loss & edits/token & extra ops\\\midrule | |
| 96 | +""" | |
| 97 | + body = "" | |
| 98 | + for r in d["records"]: | |
| 99 | + if r["policy"] == "oracle_two_step": | |
| 100 | + continue | |
| 101 | + setting = ", ".join(f"{k}={v}" for k, v in r.items() if k in ("budget", "threshold", "lam", "kap") and v is not None) | |
| 102 | + body += f"{r['policy'].replace('_', ' ')}{' (diag.)' if r.get('diagnostic') else ''} & {setting} & {r['loss']:.4f} & {r['edits_per_token']:.2f} & {r['extra_ops_ratio']:.3f}\\\\\n" | |
| 103 | + two = [r for r in d["records"] if r["policy"] == "oracle_two_step"] | |
| 104 | + if two: | |
| 105 | + t = two[0] | |
| 106 | + body += r"\midrule" + f"\\multicolumn{{5}}{{l}}{{Two-step oracle (diag.): one-step {t['one_step_gain']:.4f}, two-step {t['two_step_gain']:.4f}, greedy-then-best {t['greedy_then_best_gain']:.4f} nats/token}}\\\\\n" | |
| 107 | + macro("oracleonestep", f"{t['one_step_gain']:.4f}"); macro("oracletwostep", f"{t['two_step_gain']:.4f}") | |
| 108 | + (OUT / "routing.tex").write_text(head + body + r"\bottomrule\end{tabular}\end{table}" + "\n") | |
| 109 | + | |
| 110 | + | |
| 111 | +def table_controller(macro): | |
| 112 | + runs = sorted(glob.glob(f"{RUNS}/controller_p4*_seed*/manifest.json")) | |
| 113 | + if not runs: | |
| 114 | + return | |
| 115 | + head = r"""\begin{table}[t]\centering\small | |
| 116 | +\caption{Value-controller fit on logged counterfactual labels (tiny scale). Calibration-partition diagnostics: | |
| 117 | +Huber loss of $\hat g$, sign accuracy, Brier score and ECE of the harm head, one-step oracle opportunity and the | |
| 118 | +regret of the controller's greedy choice on the exhaustive-audit states (captured fraction $=1-$ regret/opportunity).} | |
| 119 | +\label{tab:controller} | |
| 120 | +\begin{tabular}{lrrrrrrr}\toprule | |
| 121 | +Controller features & labels & Huber & sign acc. & Brier & ECE & opportunity & captured\\\midrule | |
| 122 | +""" | |
| 123 | + body = "" | |
| 124 | + for mp in runs: | |
| 125 | + m = json.load(open(mp)) | |
| 126 | + if m.get("state") != "completed": | |
| 127 | + continue | |
| 128 | + ev = m.get("eval_calib", {}) | |
| 129 | + feats = "h + entropy" if "hent" in m["run_id"] else "z only" | |
| 130 | + body += f"{feats} & {ev.get('n_labels', 0):,} & {ev.get('huber', float('nan')):.4f} & {ev.get('sign_acc', float('nan')):.3f} & {ev.get('harm_brier', float('nan')):.3f} & {ev.get('harm_ece', float('nan')):.3f} & {ev.get('oracle_opportunity_per_state', float('nan')):.4f} & {ev.get('captured_fraction', float('nan')):.3f}\\\\\n" | |
| 131 | + macro("ctrl" + ("hent" if "hent" in m["run_id"] else "z") + "signacc", f"{100 * ev.get('sign_acc', float('nan')):.1f}") | |
| 132 | + (OUT / "controller.tex").write_text(head + body + r"\bottomrule\end{tabular}\end{table}" + "\n") | |
| 133 | + | |
| 134 | + | |
| 135 | +def table_labels(macro): | |
| 136 | + files = sorted(glob.glob("results/analyzed/labels_*.json")) | |
| 137 | + if not files: | |
| 138 | + return | |
| 139 | + d = json.load(open(files[-1])) | |
| 140 | + for part in ("train", "calib"): | |
| 141 | + if part in d and "audit" in d[part]: | |
| 142 | + a = d[part]["audit"] | |
| 143 | + macro(f"labels{part}within", f"{100 * a['within_share']:.0f}") | |
| 144 | + macro(f"labels{part}allpos", f"{100 * a['frac_states_all_positive']:.0f}") | |
| 145 | + macro(f"labels{part}allneg", f"{100 * a['frac_states_all_negative']:.0f}") | |
| 146 | + macro(f"labels{part}oracle", f"{a['oracle_one_step_mean']:.4f}") | |
| 147 | + macro(f"labels{part}stoponly", f"{a['oracle_state_level_mean']:.4f}") | |
| 148 | + macro(f"labels{part}random", f"{a['random_action_mean']:.4f}") | |
| 149 | + macro(f"labels{part}nstates", f"{a['n_states']:,}") | |
| 150 | + | |
| 151 | + | |
| 152 | +def table_latency(macro): | |
| 153 | + files = sorted(glob.glob("results/raw/latency_*.json")) | |
| 154 | + if not files: | |
| 155 | + return | |
| 156 | + d = json.load(open(files[-1])) | |
| 157 | + head = r"""\begin{table}[t]\centering\small | |
| 158 | +\caption{Measured decode latency per generated token (fp32, prompt %d tokens, %d generated, warm-up and | |
| 159 | +synchronisation), device %s. Median and p95 over generated tokens of three repetitions.} | |
| 160 | +\label{tab:latency} | |
| 161 | +\begin{tabular}{llrrrr}\toprule | |
| 162 | +Policy & budget & batch & median ms & p95 ms & cache bytes\\\midrule | |
| 163 | +""" % (d["prompt_len"], d["n_new"], d["device"].replace("_", " ")) | |
| 164 | + body = "" | |
| 165 | + for r in d["records"]: | |
| 166 | + body += f"{r['policy'].replace('_', ' ')} & {r['budget']} & {r['batch']} & {r['median_ms']:.2f} & {r['p95_ms']:.2f} & {r['cache_bytes']:,}\\\\\n" | |
| 167 | + (OUT / "latency.tex").write_text(head + body + r"\bottomrule\end{tabular}\end{table}" + "\n") | |
| 168 | + b0 = [r for r in d["records"] if r["policy"] == "budget0" and r["batch"] == 1] | |
| 169 | + if b0: | |
| 170 | + macro("latencybzero", f"{b0[0]['median_ms']:.2f}") | |
| 171 | + | |
| 172 | + | |
| 173 | +def make_all(macro): | |
| 174 | + table_seed_matrix(macro); table_routing(macro); table_controller(macro); table_labels(macro); table_latency(macro) | |
added
scripts/make_tables.py
+120 −0
@@ -0,0 +1,120 @@ | ||
| 1 | +#!/usr/bin/env python | |
| 2 | +"""Generate LaTeX table fragments and the numbers macro file from structured results (never typed by hand). | |
| 3 | +Usage: python scripts/make_tables.py -> paper/tables/*.tex, paper/tables/numbers.tex | |
| 4 | +""" | |
| 5 | +from __future__ import annotations | |
| 6 | + | |
| 7 | +import json, re, sys | |
| 8 | +from pathlib import Path | |
| 9 | + | |
| 10 | +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) | |
| 11 | +OUT = Path("paper/tables"); OUT.mkdir(parents=True, exist_ok=True) | |
| 12 | +NUM: dict[str, str] = {} | |
| 13 | + | |
| 14 | + | |
| 15 | +def macro(name: str, value) -> None: | |
| 16 | + name = re.sub(r"[^A-Za-z]", "", name) | |
| 17 | + NUM[name] = value if isinstance(value, str) else f"{value}" | |
| 18 | + | |
| 19 | + | |
| 20 | +def fmt_int(n) -> str: | |
| 21 | + return f"{int(n):,}" | |
| 22 | + | |
| 23 | + | |
| 24 | +def table_configs(): | |
| 25 | + pc = json.loads(Path("results/tables/param_counts.json").read_text()) | |
| 26 | + rows = [] | |
| 27 | + order = ["prisme_103m", "prisme_459m", "prisme_1180m"] | |
| 28 | + labels = {"prisme_103m": "PRISME-103M", "prisme_459m": "PRISME-459M", "prisme_1180m": "PRISME-1.18B"} | |
| 29 | + head = r"""\begin{table}[t]\centering\small | |
| 30 | +\caption{Dimensioned configurations and exact unique parameter counts (meta-device instantiation, | |
| 31 | +\texttt{scripts/count\_params.py}). Backbone counts equal the reference design exactly; workspace+controller counts | |
| 32 | +differ from the proposal's figures (reconciled in Appendix~\ref{app:params}).} | |
| 33 | +\label{tab:configs} | |
| 34 | +\begin{tabular}{lrrr}\toprule | |
| 35 | +Property & PRISME-103M & PRISME-459M & PRISME-1.18B\\\midrule | |
| 36 | +""" | |
| 37 | + cfgs = {} | |
| 38 | + for k in order: | |
| 39 | + c = json.loads(Path(f"configs/{k}.yaml").read_text().split("model:")[1].replace("\n", ",").replace(" ", "").replace(":", '":').replace(",", ',"').replace('{"', "{").replace(",}", "}") if False else "{}") | |
| 40 | + import yaml | |
| 41 | + cfgs = {k: yaml.safe_load(open(f"configs/{k}.yaml"))["model"] for k in order} | |
| 42 | + def row(name, f): | |
| 43 | + return name + " & " + " & ".join(f(k) for k in order) + r"\\" + "\n" | |
| 44 | + body = "" | |
| 45 | + body += row("Backbone layers $L$", lambda k: str(cfgs[k]["n_layers"])) | |
| 46 | + body += row("Hidden $d$ / FFN", lambda k: f"{cfgs[k]['d_model']} / {cfgs[k]['ffn_width']}") | |
| 47 | + body += row("Query / KV heads", lambda k: f"{cfgs[k]['n_heads']} / {cfgs[k]['n_kv_heads']}") | |
| 48 | + body += row("Workspace $w$ / FFN$_w$", lambda k: f"{cfgs[k]['w']} / {cfgs[k]['ffn_w']}") | |
| 49 | + body += row("Slots $M$, blocks $L_w$, cap $R_{\\max}$", lambda k: f"{cfgs[k]['n_slots']}, {cfgs[k]['n_editor_blocks']}, {cfgs[k]['r_max']}") | |
| 50 | + body += r"\midrule" + "\n" | |
| 51 | + body += row("Backbone parameters", lambda k: fmt_int(pc[k]["backbone"])) | |
| 52 | + body += row("Workspace + controller", lambda k: fmt_int(pc[k]["workspace_controller"])) | |
| 53 | + body += row("\\textbf{Total}", lambda k: "\\textbf{" + fmt_int(pc[k]["total"]) + "}") | |
| 54 | + body += row("Proposal's workspace figure", lambda k: fmt_int(pc[k]["reference_brief"]["workspace"])) | |
| 55 | + tail = r"\bottomrule\end{tabular}\end{table}" + "\n" | |
| 56 | + (OUT / "configs.tex").write_text(head + body + tail) | |
| 57 | + for k in order: | |
| 58 | + macro("params" + {"prisme_103m": "Small", "prisme_459m": "Medium", "prisme_1180m": "Large"}[k] + "Total", fmt_int(pc[k]["total"])) | |
| 59 | + macro("params" + {"prisme_103m": "Small", "prisme_459m": "Medium", "prisme_1180m": "Large"}[k] + "Ws", fmt_int(pc[k]["workspace_controller"])) | |
| 60 | + | |
| 61 | + | |
| 62 | +def table_ops(): | |
| 63 | + o = json.loads(Path("results/tables/ops_459m_T4096.json").read_text()) | |
| 64 | + names = {"backbone_only": "Backbone only", "4_actions_k1": "4 actions, 1 slot each", "4_actions_k2": "4 actions, 2 slots each", | |
| 65 | + "8_actions_k2": "8 actions, 2 slots each", "8_actions_dense16": "8 actions, dense 16-slot writes"} | |
| 66 | + head = r"""\begin{table}[t]\centering\small | |
| 67 | +\caption{Analytic operations per generated token, PRISME-459M, context $T=4096$ (\texttt{scripts/estimate\_flops.py}). | |
| 68 | +``Proposal'' is the reference table of the original brief; ``implemented'' is the accounting of | |
| 69 | +Section~\ref{sec:compute} with the controller evaluated once per executed action plus once to stop. These are | |
| 70 | +arithmetic counts, not measured timings.} | |
| 71 | +\label{tab:ops} | |
| 72 | +\begin{tabular}{lrrrrr}\toprule | |
| 73 | +Policy & Proposal ($10^9$) & Implemented ($10^9$) & editor & controller & readout\\\midrule | |
| 74 | +""" | |
| 75 | + body = "" | |
| 76 | + for k, lab in names.items(): | |
| 77 | + r = o["rows"][k] | |
| 78 | + ed = (r["editor_source_attn"] + r["editor_slot_attn"] + r["editor_ffn"] + r["editor_gate"]) / 1e9 | |
| 79 | + body += f"{lab} & {o['brief_reference_459m_T4096'][k]/1e9:.3f} & {r['total']/1e9:.3f} & {ed:.3f} & {r['controller']/1e9:.3f} & {r['readout']/1e9:.3f}\\\\\n" | |
| 80 | + (OUT / "ops.tex").write_text(head + body + r"\bottomrule\end{tabular}\end{table}" + "\n") | |
| 81 | + o103 = json.loads(Path("results/tables/ops_103m_T512.json").read_text()) | |
| 82 | + macro("costsingle", f"{o103['action_cost_proxy'][0]:.3f}"); macro("costpair", f"{o103['action_cost_proxy'][-1]:.3f}") | |
| 83 | + | |
| 84 | + | |
| 85 | +def table_families(): | |
| 86 | + man = json.loads(Path("data/splits/synthetic_v1/manifest.json").read_text()) | |
| 87 | + head = r"""\begin{table}[t]\centering\small | |
| 88 | +\caption{Synthetic suite v1: splits fixed before sampling. $n$ = items after cross-split exact-duplicate removal; | |
| 89 | +``dropped'' = duplicates removed. Test splits are locked until pre-registration is frozen.} | |
| 90 | +\label{tab:families} | |
| 91 | +\begin{tabular}{llllrr}\toprule | |
| 92 | +Split & Families & Templates & Depths & $n$ & dropped\\\midrule | |
| 93 | +""" | |
| 94 | + body = "" | |
| 95 | + for name, m in man.items(): | |
| 96 | + s = m["spec"] | |
| 97 | + fam = ", ".join(s["families"]) if len(s["families"]) > 1 else s["families"][0] | |
| 98 | + if len(s["families"]) == 5: | |
| 99 | + fam = "arith, bind, fsm, excl, sched" | |
| 100 | + body += f"{name.replace('_', '\\_')} & {fam} & {','.join(map(str, s['templates']))} & {','.join(map(str, s['depths']))} & {m['n']:,} & {m['duplicates_dropped']}\\\\\n" | |
| 101 | + (OUT / "families.tex").write_text(head + body + r"\bottomrule\end{tabular}\end{table}" + "\n") | |
| 102 | + macro("ntrainsynth", f"{man['train']['n']:,}") | |
| 103 | + | |
| 104 | + | |
| 105 | +def write_numbers(): | |
| 106 | + lines = ["% AUTO-GENERATED by scripts/make_tables.py — do not edit"] | |
| 107 | + for k, v in sorted(NUM.items()): | |
| 108 | + lines.append(f"\\newcommand{{\\{k}}}{{{v}}}") | |
| 109 | + (OUT / "numbers.tex").write_text("\n".join(lines) + "\n") | |
| 110 | + | |
| 111 | + | |
| 112 | +if __name__ == "__main__": | |
| 113 | + table_configs(); table_ops(); table_families() | |
| 114 | + try: | |
| 115 | + from scripts.make_results_tables import make_all # results-dependent tables (added later) | |
| 116 | + make_all(macro) | |
| 117 | + except Exception as e: # noqa: BLE001 | |
| 118 | + print("results tables skipped:", e) | |
| 119 | + write_numbers() | |
| 120 | + print("tables written:", sorted(p.name for p in OUT.glob("*.tex"))) | |
added
scripts/md2tex.py
+27 −0
@@ -0,0 +1,27 @@ | ||
| 1 | +#!/usr/bin/env python | |
| 2 | +"""Markdown -> LaTeX fragment via pandoc, with Unicode math symbols mapped to LaTeX (for appendix inclusion).""" | |
| 3 | +import subprocess, sys | |
| 4 | +src, dst = sys.argv[1], sys.argv[2] | |
| 5 | +# markdown pipe tables -> bullet lists (longtable output of pandoc clashes with the caption package) | |
| 6 | +lines, out, in_tab, header = open(src).read().splitlines(), [], False, None | |
| 7 | +for ln in lines: | |
| 8 | + if ln.startswith("|"): | |
| 9 | + cells = [c.strip() for c in ln.strip().strip("|").split("|")] | |
| 10 | + if all(set(c) <= set("-: ") for c in cells): | |
| 11 | + continue | |
| 12 | + if not in_tab: | |
| 13 | + in_tab, header = True, cells | |
| 14 | + continue | |
| 15 | + out.append("- " + "; ".join(f"**{h}**: {c}" for h, c in zip(header, cells) if c and c != "—")) | |
| 16 | + else: | |
| 17 | + in_tab = False | |
| 18 | + out.append(ln) | |
| 19 | +md = "\n".join(out) | |
| 20 | +tex = subprocess.run(["pandoc", "-f", "markdown", "-t", "latex", "--wrap=none"], input=md, capture_output=True, text=True).stdout | |
| 21 | +rep = {"≤": r"$\leq$", "≥": r"$\geq$", "→": r"$\rightarrow$", "←": r"$\leftarrow$", "×": r"$\times$", "−": "--", "≈": r"$\approx$", "±": r"$\pm$", | |
| 22 | + "σ": r"$\sigma$", "λ": r"$\lambda$", "κ": r"$\kappa$", "η": r"$\eta$", "Δ": r"$\Delta$", "δ": r"$\delta$", "ε": r"$\varepsilon$", "ĝ": r"$\hat g$", "ĉ": r"$\hat c$", "σ̂": r"$\hat\sigma$", | |
| 23 | + "⇒": r"$\Rightarrow$", "∈": r"$\in$", "≠": r"$\neq$", "·": r"$\cdot$", "²": r"$^2$", "…": r"\ldots{}", "—": "---", "–": "--", "’": "'", "“": "``", "”": "''", "ρ": r"$\rho$", "α": r"$\alpha$", "β": r"$\beta$", "γ": r"$\gamma$", "µ": r"$\mu$", "μ": r"$\mu$", "∅": r"$\varnothing$"} | |
| 24 | +for k, v in rep.items(): | |
| 25 | + tex = tex.replace(k, v) | |
| 26 | +tex = tex.replace(r"\section{", r"\subsection*{").replace(r"\subsection{", r"\subsubsection*{") | |
| 27 | +open(dst, "w").write(tex) | |
added
scripts/node_logs.sh
+6 −0
@@ -0,0 +1,6 @@ | ||
| 1 | +#!/bin/bash | |
| 2 | +# Print the last validation lines of every experiment log on the given nodes. Usage: scripts/node_logs.sh node1 node2 ... | |
| 3 | +for n in "$@"; do | |
| 4 | + echo "=== $n" | |
| 5 | + ssh -o ConnectTimeout=8 "$n" 'cd ~/wp31/prisme-slm/experiments/logs 2>/dev/null || exit 0; for f in $(ls -t *.log | head -6); do pid=$(pgrep -f "config configs/experiments/${f%.log}.yaml" | head -1); echo "-- $f $( [ -n "$pid" ] && echo RUNNING || echo stopped)"; grep "val loss\|exact match\|readout_scale\|^run " $f | tail -4; tail -1 $f; done' | |
| 6 | +done | |
added
scripts/plotstyle.py
+28 −0
@@ -0,0 +1,28 @@ | ||
| 1 | +"""Matplotlib style for WP31 figures: validated categorical palette (dataviz skill reference instance), thin marks, | |
| 2 | +recessive axes, one y-axis per panel, direct labels where few series.""" | |
| 3 | +import matplotlib as mpl | |
| 4 | +import matplotlib.pyplot as plt | |
| 5 | + | |
| 6 | +CAT = ["#2a78d6", "#eb6834", "#1baf7a", "#eda100", "#e87ba4", "#008300", "#6250d6", "#e34948"] # fixed order | |
| 7 | +SEQ_BLUE = ["#cde2fb", "#9ec5f4", "#6da7ec", "#3987e5", "#256abf", "#184f95", "#0d366b"] | |
| 8 | +TEXT, TEXT2, GRID = "#0b0b0b", "#52514e", "#e6e5e1" | |
| 9 | + | |
| 10 | + | |
| 11 | +def setup(): | |
| 12 | + mpl.rcParams.update({ | |
| 13 | + "figure.dpi": 150, "savefig.dpi": 300, "font.size": 8, "axes.titlesize": 8.5, "axes.labelsize": 8, | |
| 14 | + "legend.fontsize": 7, "xtick.labelsize": 7, "ytick.labelsize": 7, "axes.edgecolor": GRID, "axes.linewidth": 0.6, | |
| 15 | + "axes.grid": True, "grid.color": GRID, "grid.linewidth": 0.5, "axes.spines.top": False, "axes.spines.right": False, | |
| 16 | + "lines.linewidth": 1.4, "lines.markersize": 4, "text.color": TEXT, "axes.labelcolor": TEXT2, | |
| 17 | + "xtick.color": TEXT2, "ytick.color": TEXT2, "legend.frameon": False, "axes.prop_cycle": mpl.cycler(color=CAT), | |
| 18 | + "pdf.fonttype": 42, "font.family": "DejaVu Sans", | |
| 19 | + }) | |
| 20 | + | |
| 21 | + | |
| 22 | +def save(fig, name, outdir="results/figures"): | |
| 23 | + import os | |
| 24 | + os.makedirs(outdir, exist_ok=True) | |
| 25 | + for ext in ("pdf", "png"): | |
| 26 | + fig.savefig(f"{outdir}/{name}.{ext}", bbox_inches="tight") | |
| 27 | + plt.close(fig) | |
| 28 | + return f"{outdir}/{name}.pdf" | |
added
scripts/reproduce_figures.py
+186 −0
@@ -0,0 +1,186 @@ | ||
| 1 | +#!/usr/bin/env python | |
| 2 | +"""Regenerate every figure of the manuscript from structured results. Figures whose inputs are absent are skipped | |
| 3 | +and reported. Usage: python scripts/reproduce_figures.py --results results/raw/ [--runs experiments/runs] | |
| 4 | +""" | |
| 5 | +from __future__ import annotations | |
| 6 | + | |
| 7 | +import argparse, glob, json, sys | |
| 8 | +from collections import defaultdict | |
| 9 | +from pathlib import Path | |
| 10 | + | |
| 11 | +import numpy as np | |
| 12 | + | |
| 13 | +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) | |
| 14 | +from scripts.plotstyle import CAT, SEQ_BLUE, setup, save # noqa: E402 | |
| 15 | + | |
| 16 | +import matplotlib.pyplot as plt # noqa: E402 | |
| 17 | + | |
| 18 | +MADE: list[str] = [] | |
| 19 | +SKIPPED: list[str] = [] | |
| 20 | + | |
| 21 | + | |
| 22 | +def load_metrics(run_dir): | |
| 23 | + p = Path(run_dir) / "metrics.jsonl" | |
| 24 | + return [json.loads(l) for l in open(p)] if p.exists() else [] | |
| 25 | + | |
| 26 | + | |
| 27 | +def runs_matching(runs_root, prefix): | |
| 28 | + return sorted(d for d in glob.glob(f"{runs_root}/{prefix}_seed*") if Path(d, "manifest.json").exists()) | |
| 29 | + | |
| 30 | + | |
| 31 | +# ---------------------------------------------------------------- 1. learnability: val loss by budget vs step | |
| 32 | +def fig_learnability(runs_root, prefixes, name, title): | |
| 33 | + panels = [(p, runs_matching(runs_root, p)) for p in prefixes] | |
| 34 | + panels = [(p, r) for p, r in panels if r] | |
| 35 | + if not panels: | |
| 36 | + SKIPPED.append(name); return | |
| 37 | + fig, axes = plt.subplots(1, len(panels), figsize=(2.6 * len(panels), 2.4), sharey=True, squeeze=False) | |
| 38 | + for ax, (p, runs) in zip(axes[0], panels): | |
| 39 | + for run in runs[:1]: | |
| 40 | + rows = [r for r in load_metrics(run) if any(k.startswith("val_loss_b") for k in r)] | |
| 41 | + budgets = sorted({int(k[10:]) for r in rows for k in r if k.startswith("val_loss_b")}) | |
| 42 | + for i, b in enumerate(budgets): | |
| 43 | + xs = [r["step"] + 1 for r in rows if f"val_loss_b{b}" in r]; ys = [r[f"val_loss_b{b}"] for r in rows if f"val_loss_b{b}" in r] | |
| 44 | + ax.plot(xs, ys, marker="o", color=SEQ_BLUE[min(i + 1, 6)] if b else "#52514e", label=f"budget {b}") | |
| 45 | + ax.set_title(p.replace("_", " "), loc="left"); ax.set_xlabel("training step") | |
| 46 | + axes[0][0].set_ylabel("held-out answer-span loss (nats)"); axes[0][-1].legend(loc="upper right") | |
| 47 | + fig.suptitle(title, x=0.01, ha="left", fontsize=8.5) | |
| 48 | + MADE.append(save(fig, name)) | |
| 49 | + | |
| 50 | + | |
| 51 | +# ---------------------------------------------------------------- 2. labels: gain distribution + variance decomposition | |
| 52 | +def fig_labels(analyzed_glob): | |
| 53 | + files = sorted(glob.glob(analyzed_glob)) | |
| 54 | + if not files: | |
| 55 | + SKIPPED.append("labels"); return | |
| 56 | + d = json.loads(Path(files[-1]).read_text()) | |
| 57 | + part = d.get("calib") or d.get("train") | |
| 58 | + aud = part.get("audit", {}) | |
| 59 | + fig, axes = plt.subplots(1, 3, figsize=(7.6, 2.3)) | |
| 60 | + ax = axes[0] | |
| 61 | + depths = sorted(int(k) for k in part["by_prefix_depth"]) | |
| 62 | + ax.bar([str(k) for k in depths], [part["by_prefix_depth"][str(k)]["frac_pos"] for k in depths], color=CAT[0], width=0.6) | |
| 63 | + ax.axhline(0.5, color="#52514e", lw=0.8, ls="--"); ax.set_ylim(0, 0.7) | |
| 64 | + ax.set_xlabel("prefix depth r (edits already applied)"); ax.set_ylabel("fraction of gains > 0"); ax.set_title("Sign of logged gains", loc="left") | |
| 65 | + ax = axes[1] | |
| 66 | + labels = ["between states", "within state (action)"] | |
| 67 | + vals = [aud.get("var_between_states", 0), aud.get("var_within_state_actions", 0)] | |
| 68 | + tot = sum(vals) or 1 | |
| 69 | + ax.bar(labels, [v / tot for v in vals], color=[CAT[0], CAT[1]], width=0.6) | |
| 70 | + for i, v in enumerate(vals): | |
| 71 | + ax.text(i, v / tot + 0.02, f"{100*v/tot:.0f}%", ha="center", fontsize=7) | |
| 72 | + ax.set_ylim(0, 1.05); ax.set_ylabel("share of gain variance"); ax.set_title("Variance decomposition (24-action audit)", loc="left") | |
| 73 | + ax = axes[2] | |
| 74 | + names = ["random action", "perfect stop,\nrandom action", "one-step oracle"] | |
| 75 | + vals = [aud.get("random_action_mean", 0), aud.get("oracle_state_level_mean", 0), aud.get("oracle_one_step_mean", 0)] | |
| 76 | + ax.bar(names, vals, color=[CAT[2], CAT[3], CAT[0]], width=0.6) | |
| 77 | + for i, v in enumerate(vals): | |
| 78 | + ax.text(i, v + 0.0005, f"{v:.4f}", ha="center", fontsize=7) | |
| 79 | + ax.set_ylabel("mean gain per state (nats)"); ax.set_title("Oracle decomposition", loc="left") | |
| 80 | + MADE.append(save(fig, "labels_analysis")) | |
| 81 | + | |
| 82 | + | |
| 83 | +# ---------------------------------------------------------------- 3. routing Pareto: loss vs extra ops | |
| 84 | +def fig_routing(results_dir): | |
| 85 | + files = sorted(glob.glob(f"{results_dir}/routing_*.json")) | |
| 86 | + if not files: | |
| 87 | + SKIPPED.append("routing_pareto"); return | |
| 88 | + for f in files: | |
| 89 | + d = json.loads(Path(f).read_text()); recs = d["records"] | |
| 90 | + fig, ax = plt.subplots(figsize=(4.2, 2.8)) | |
| 91 | + groups = defaultdict(list) | |
| 92 | + for r in recs: | |
| 93 | + if r.get("policy") in ("oracle_two_step",): | |
| 94 | + continue | |
| 95 | + groups[r["policy"]].append(r) | |
| 96 | + order = ["budget0", "random_forced", "uniform", "fixed_pair", "entropy", "norm", "controller", "controller_locked", "oracle1_greedy"] | |
| 97 | + for i, pol in enumerate([p for p in order if p in groups]): | |
| 98 | + rs = sorted(groups[pol], key=lambda r: r["extra_ops_ratio"]) | |
| 99 | + xs = [r["extra_ops_ratio"] for r in rs]; ys = [r["loss"] for r in rs] | |
| 100 | + style = dict(marker="o", color=CAT[i % 8], ls="-" if len(rs) > 1 else "none") | |
| 101 | + if pol == "oracle1_greedy": | |
| 102 | + style.update(ls="--", color="#52514e") | |
| 103 | + if pol == "controller_locked": | |
| 104 | + style.update(marker="*", markersize=9, ls="none") | |
| 105 | + ax.plot(xs, ys, label=pol.replace("_", " ") + (" (diagnostic)" if "oracle" in pol else ""), **style) | |
| 106 | + ax.set_xlabel("extra operations per token (× backbone)"); ax.set_ylabel("held-out loss (nats)") | |
| 107 | + ax.set_title(f"Quality vs analytic cost — {d['split']}", loc="left"); ax.legend(ncol=2) | |
| 108 | + MADE.append(save(fig, "routing_" + Path(f).stem.replace("routing_", ""))) | |
| 109 | + | |
| 110 | + | |
| 111 | +# ---------------------------------------------------------------- 4. latency | |
| 112 | +def fig_latency(results_dir): | |
| 113 | + files = sorted(glob.glob(f"{results_dir}/latency_*.json")) | |
| 114 | + if not files: | |
| 115 | + SKIPPED.append("latency"); return | |
| 116 | + for f in files: | |
| 117 | + d = json.loads(Path(f).read_text()); recs = [r for r in d["records"] if r["batch"] == 1] | |
| 118 | + fig, ax = plt.subplots(figsize=(4.2, 2.6)) | |
| 119 | + groups = defaultdict(list) | |
| 120 | + for r in recs: | |
| 121 | + groups[r["policy"]].append(r) | |
| 122 | + for i, (pol, rs) in enumerate(groups.items()): | |
| 123 | + rs = sorted(rs, key=lambda r: r["budget"]) | |
| 124 | + ax.errorbar([r["budget"] for r in rs], [r["median_ms"] for r in rs], yerr=[[0] * len(rs), [r["p95_ms"] - r["median_ms"] for r in rs]], | |
| 125 | + marker="o", color=CAT[i % 8], label=pol.replace("_", " "), capsize=2, lw=1.2) | |
| 126 | + ax.set_xlabel("executed edits per token (budget)"); ax.set_ylabel("ms per token (median, +p95)") | |
| 127 | + ax.set_title(f"Batch-1 decode latency — {d['device']}", loc="left"); ax.legend() | |
| 128 | + MADE.append(save(fig, Path(f).stem)) | |
| 129 | + | |
| 130 | + | |
| 131 | +# ---------------------------------------------------------------- 5. exact match by regime/policy | |
| 132 | +def fig_exact_match(results_dir): | |
| 133 | + files = sorted(glob.glob(f"{results_dir}/eval_*_dev_*.json")) + sorted(glob.glob(f"{results_dir}/eval_*_test_*.json")) | |
| 134 | + files = [f for f in files if "exact_match" in Path(f).read_text()[:400] or True] | |
| 135 | + data = defaultdict(dict) | |
| 136 | + for f in files: | |
| 137 | + d = json.loads(Path(f).read_text()) | |
| 138 | + if "exact_match" not in d: | |
| 139 | + continue | |
| 140 | + for pol, em in d["exact_match"].items(): | |
| 141 | + data[(d["system"], pol)][d["split"]] = em["accuracy"] | |
| 142 | + if not data: | |
| 143 | + SKIPPED.append("exact_match"); return | |
| 144 | + splits = sorted({s for v in data.values() for s in v}) | |
| 145 | + fig, ax = plt.subplots(figsize=(6.4, 2.6)) | |
| 146 | + keys = sorted(data) | |
| 147 | + width = 0.8 / max(len(keys), 1) | |
| 148 | + for i, k in enumerate(keys): | |
| 149 | + xs = np.arange(len(splits)) + i * width | |
| 150 | + ax.bar(xs, [data[k].get(s, np.nan) for s in splits], width=width * 0.95, color=CAT[i % 8], label=f"{k[0]} / {k[1]}") | |
| 151 | + ax.set_xticks(np.arange(len(splits)) + 0.4 - width / 2); ax.set_xticklabels([s.replace("_", "\n") for s in splits]) | |
| 152 | + ax.set_ylabel("exact match"); ax.set_title("Free-generation exact match by regime", loc="left"); ax.legend(ncol=2, fontsize=6) | |
| 153 | + MADE.append(save(fig, "exact_match_regimes")) | |
| 154 | + | |
| 155 | + | |
| 156 | +# ---------------------------------------------------------------- 6. pretraining curves | |
| 157 | +def fig_pretrain(runs_root): | |
| 158 | + runs = runs_matching(runs_root, "pretrain103m") + runs_matching(runs_root, "pretrain_enlarged102m") | |
| 159 | + if not runs: | |
| 160 | + SKIPPED.append("pretrain"); return | |
| 161 | + fig, ax = plt.subplots(figsize=(4.2, 2.6)) | |
| 162 | + for i, run in enumerate(runs): | |
| 163 | + rows = load_metrics(run) | |
| 164 | + tr = [(r["tokens"], r["loss"]) for r in rows if "loss" in r]; va = [(r["tokens"], r["val_loss"]) for r in rows if "val_loss" in r] | |
| 165 | + name = Path(run).name.split("_seed")[0] | |
| 166 | + if tr: | |
| 167 | + ax.plot([t / 1e6 for t, _ in tr], [l for _, l in tr], color=CAT[i], alpha=0.35, lw=0.8) | |
| 168 | + if va: | |
| 169 | + ax.plot([t / 1e6 for t, _ in va], [l for _, l in va], color=CAT[i], marker="o", label=f"{name} (val)") | |
| 170 | + ax.set_xlabel("training tokens (M)"); ax.set_ylabel("cross-entropy (nats)"); ax.set_title("Stage A pretraining (FineWeb-Edu)", loc="left"); ax.legend() | |
| 171 | + MADE.append(save(fig, "pretrain_curves")) | |
| 172 | + | |
| 173 | + | |
| 174 | +def main(argv=None): | |
| 175 | + ap = argparse.ArgumentParser(description=__doc__) | |
| 176 | + ap.add_argument("--results", default="results/raw/"); ap.add_argument("--runs", default="experiments/runs"); ap.add_argument("--analyzed", default="results/analyzed") | |
| 177 | + args = ap.parse_args(argv); setup() | |
| 178 | + fig_learnability(args.runs, ["p4_editor_v1_scaled", "p4_editor_v1_Ustd02", "p4_editor_v1_Ustd05"], "learnability_bootstrap", "Stage B bootstrap on the frozen tiny backbone (dev-iid, random forced actions)") | |
| 179 | + fig_learnability(args.runs, ["p4v2_prisme", "p4v2_gr", "p4v2_dense"], "learnability_v2", "Stage B on the v2 tiny backbone: sparse editor vs global recurrence vs dense writes") | |
| 180 | + fig_labels(f"{args.analyzed}/labels_*.json") | |
| 181 | + fig_routing(args.results); fig_latency(args.results); fig_exact_match(args.results); fig_pretrain(args.runs) | |
| 182 | + print("made:", MADE); print("skipped (inputs absent):", SKIPPED) | |
| 183 | + | |
| 184 | + | |
| 185 | +if __name__ == "__main__": | |
| 186 | + main() | |
added
scripts/verify_manuscript_claims.py
+53 −0
@@ -0,0 +1,53 @@ | ||
| 1 | +#!/usr/bin/env python | |
| 2 | +"""Results-to-claim audit. Reads results/claim_registry.csv (claim, metric, value, benchmark, run_ids, artifact, | |
| 3 | +status) and verifies that (i) every artifact path exists, (ii) every run id has a manifest with state completed | |
| 4 | +(or the registry explicitly says 'partial'), (iii) every numeric macro used in the manuscript exists in | |
| 5 | +paper/tables/numbers.tex, and (iv) the manuscript does not use forbidden promotional words. | |
| 6 | +Exit code 1 on any failure. Usage: python scripts/verify_manuscript_claims.py | |
| 7 | +""" | |
| 8 | +from __future__ import annotations | |
| 9 | + | |
| 10 | +import csv, glob, json, re, sys | |
| 11 | +from pathlib import Path | |
| 12 | + | |
| 13 | +FORBIDDEN = ["revolutionary", "state-of-the-art", "breakthrough", "groundbreaking", "unprecedented", "novel architecture that outperforms"] | |
| 14 | + | |
| 15 | + | |
| 16 | +def main() -> int: | |
| 17 | + ok = True | |
| 18 | + reg = Path("results/claim_registry.csv") | |
| 19 | + if not reg.exists(): | |
| 20 | + print("FAIL: results/claim_registry.csv missing"); return 1 | |
| 21 | + rows = list(csv.DictReader(open(reg))) | |
| 22 | + print(f"{len(rows)} registered claims") | |
| 23 | + for r in rows: | |
| 24 | + for art in filter(None, [a.strip() for a in r["artifact"].split(";")]): | |
| 25 | + if not Path(art).exists(): | |
| 26 | + print(f"FAIL artifact missing for claim '{r['claim'][:60]}': {art}"); ok = False | |
| 27 | + for rid in filter(None, [x.strip() for x in r["run_ids"].split(";")]): | |
| 28 | + mans = glob.glob(f"experiments/runs/{rid}*/manifest.json") + glob.glob(f"experiments/runs/{rid}/manifest.json") | |
| 29 | + if not mans: | |
| 30 | + print(f"FAIL run manifest missing for '{r['claim'][:60]}': {rid}"); ok = False; continue | |
| 31 | + st = json.load(open(mans[0])).get("state", "") | |
| 32 | + if st != "completed" and r.get("status", "") not in ("partial", "stopped-run-used"): | |
| 33 | + print(f"FAIL run {rid} state={st} but claim status={r.get('status')}"); ok = False | |
| 34 | + nums = Path("paper/tables/numbers.tex").read_text() if Path("paper/tables/numbers.tex").exists() else "" | |
| 35 | + defined = set(re.findall(r"\\newcommand\{\\([A-Za-z]+)\}", nums)) | |
| 36 | + tex = "\n".join(Path(p).read_text() for p in glob.glob("paper/sections/*.tex") + glob.glob("paper/appendix/*.tex")) | |
| 37 | + used = set(re.findall(r"\\(params[A-Za-z]+|labels[A-Za-z]+|ctrl[A-Za-z]+|oracle[A-Za-z]+|vtwo[A-Za-z]+|latency[A-Za-z]+|cost[A-Za-z]+|ntrain[A-Za-z]+|pilot[A-Za-z]+|em[A-Z][A-Za-z]+)", tex)) | |
| 38 | + for u in sorted(used - defined): | |
| 39 | + print(f"FAIL macro \\{u} used in manuscript but not generated"); ok = False | |
| 40 | + low = tex.lower() | |
| 41 | + for w in FORBIDDEN: | |
| 42 | + if w in low: | |
| 43 | + print(f"FAIL forbidden phrase in manuscript: '{w}'"); ok = False | |
| 44 | + # unresolved placeholders | |
| 45 | + for p in glob.glob("paper/sections/*.tex") + glob.glob("paper/appendix/*.tex"): | |
| 46 | + if "% to be written" in Path(p).read_text() or "TODO" in Path(p).read_text(): | |
| 47 | + print(f"FAIL placeholder left in {p}"); ok = False | |
| 48 | + print("AUDIT", "PASS" if ok else "FAIL") | |
| 49 | + return 0 if ok else 1 | |
| 50 | + | |
| 51 | + | |
| 52 | +if __name__ == "__main__": | |
| 53 | + raise SystemExit(main()) | |
modified
tests/unit/test_invariants.py
+17 −0
@@ -222,3 +222,20 @@ def test_dense_writes_ablation_touches_all_slots(tokens): | ||
| 222 | 222 | assert (S1 != ctx.S0).any(-1).all() |
| 223 | 223 | S2, _ = m.apply_action(ctx.S0, ctx, torch.full((3, 12), cfg.stop_action), 0, 0) |
| 224 | 224 | assert torch.equal(S2, ctx.S0) |
| 225 | + | |
| 226 | + | |
| 227 | +def test_controller_h_entropy_features_target_free(tokens): | |
| 228 | + cfg = tiny_config(ctrl_input="h", ctrl_use_entropy=True) | |
| 229 | + torch.manual_seed(0) | |
| 230 | + m = PrismeModel(cfg).eval() | |
| 231 | + ctx = m.encode(tokens) | |
| 232 | + assert ctx.ctrl_in.shape[-1] == cfg.d_model and ctx.entropy.shape == tokens.shape | |
| 233 | + step = torch.zeros(3, 12, dtype=torch.long) | |
| 234 | + p1 = m.controller(ctx.ctrl_in, ctx.S0, step, step + 3, ctx.entropy) | |
| 235 | + t2 = tokens.clone(); t2[:, -1] = (t2[:, -1] + 1) % cfg.vocab_size | |
| 236 | + ctx2 = m.encode(t2) | |
| 237 | + p2 = m.controller(ctx2.ctrl_in, ctx2.S0, step, step + 3, ctx2.entropy) | |
| 238 | + assert torch.allclose(p1["g_hat"][:, :-1], p2["g_hat"][:, :-1], atol=1e-5) | |
| 239 | + cost = torch.full((cfg.n_actions,), 0.05) | |
| 240 | + out = m.forward_policy(tokens, 4, cost) | |
| 241 | + assert out["logits"].shape == (3, 12, cfg.vocab_size) | |
modified
training/collect_interventions.py
+1 −1
@@ -46,7 +46,7 @@ def collect(model, cfg, batches, n_batches, rng, device, max_prefix, audit_every | ||
| 46 | 46 | remaining = torch.full((B, T), R_total - r_pre - 1, device=device, dtype=torch.long) |
| 47 | 47 | pref = None |
| 48 | 48 | if use_policy: |
| 49 | − preds = model.controller(ctx.z, S, step, remaining) | |
| 49 | + preds = model.controller(ctx.ctrl_in, S, step, remaining, ctx.entropy) | |
| 50 | 50 | q = model.controller.q_values(preds, cost) |
| 51 | 51 | pref = q.argmax(-1) |
| 52 | 52 | gen = torch.Generator(device="cpu").manual_seed(int(rng.integers(0, 2**31))) |
modified
training/common.py
+7 −1
@@ -172,7 +172,13 @@ def load_checkpoint(path: str | Path, model: torch.nn.Module, strict: bool = Tru | ||
| 172 | 172 | from safetensors.torch import load_file |
| 173 | 173 | path = Path(path) |
| 174 | 174 | sd = load_file(str(path / "model.safetensors")) |
| 175 | + own = model.state_dict() | |
| 176 | + mismatched = [k for k, v in sd.items() if k in own and own[k].shape != v.shape] | |
| 177 | + if mismatched and strict: | |
| 178 | + raise RuntimeError(f"shape mismatch for {mismatched}") | |
| 179 | + for k in mismatched: # e.g. a controller re-shaped by a feature option: left at its fresh init | |
| 180 | + sd.pop(k) | |
| 175 | 181 | missing, unexpected = model.load_state_dict(sd, strict=strict) |
| 176 | 182 | meta = json.loads((path / "checkpoint.json").read_text()) |
| 177 | − meta["_missing"] = list(missing); meta["_unexpected"] = list(unexpected) | |
| 183 | + meta["_missing"] = list(missing) + mismatched; meta["_unexpected"] = list(unexpected); meta["_shape_mismatch"] = mismatched | |
| 178 | 184 | return meta |
modified
training/train_controller.py
+14 −3
@@ -35,7 +35,7 @@ def regenerate_state(model, cfg, rec, device): | ||
| 35 | 35 | |
| 36 | 36 | |
| 37 | 37 | def preds_for_candidates(model, ctx, S, step, remaining, cand): |
| 38 | − p = model.controller(ctx.z, S, step, remaining) # [B,T,A] | |
| 38 | + p = model.controller(ctx.ctrl_in, S, step, remaining, ctx.entropy) # [B,T,A] | |
| 39 | 39 | return {k: v.gather(-1, cand) for k, v in p.items()}, p |
| 40 | 40 | |
| 41 | 41 | |
@@ -83,8 +83,9 @@ def rollout_cost_quality(model, cfg, recs, device, cost, lam, kap, max_budget): | ||
| 83 | 83 | out = model.forward_policy(x, max_budget, cost, lambda_cost=lam, kappa_risk=kap) |
| 84 | 84 | nll = token_nll(out["logits"], y); v = (y != -100) |
| 85 | 85 | tot += float(nll[v].sum()); n += int(v.sum()); edits += float(out["n_edits"][v].float().sum()) |
| 86 | − a = out["actions"][v] # [n, R] | |
| 87 | − ops += float(torch.where(a < cfg.n_actions, cost[a.clamp(max=cfg.n_actions - 1)], torch.zeros_like(cost[0]).expand_as(a).float()).sum()) | |
| 86 | + if out["actions"] is not None: | |
| 87 | + a = out["actions"][v] # [n, R] | |
| 88 | + ops += float(torch.where(a < cfg.n_actions, cost[a.clamp(max=cfg.n_actions - 1)], torch.zeros_like(cost[0]).expand_as(a).float()).sum()) | |
| 88 | 89 | return {"loss": tot / max(n, 1), "edits_per_token": edits / max(n, 1), "extra_ops_ratio": ops / max(n, 1)} |
| 89 | 90 | |
| 90 | 91 | |
@@ -113,6 +114,9 @@ def main(argv=None) -> int: | ||
| 113 | 114 | recs = train["records"]; rng = np.random.default_rng(seed) |
| 114 | 115 | alpha, beta, gamma = tr.get("alpha", 1.0), tr.get("beta", 0.5), tr.get("gamma", 0.1) |
| 115 | 116 | model.train() |
| 117 | + best_state, best_score, best_step = None, -1e9, -1 | |
| 118 | + def score_of(ev): | |
| 119 | + return -ev["regret_per_state"] if "regret_per_state" in ev else ev["sign_acc"] | |
| 116 | 120 | for step in range(tr["steps"]): |
| 117 | 121 | rec = recs[int(rng.integers(0, len(recs)))] |
| 118 | 122 | with torch.no_grad(): |
@@ -129,6 +133,13 @@ def main(argv=None) -> int: | ||
| 129 | 133 | if (step + 1) % tr.get("eval_every", 500) == 0 or step == tr["steps"] - 1: |
| 130 | 134 | ev = evaluate_labels(model, cfg, calib["records"], device); model.train() |
| 131 | 135 | run.log(step=step, **{f"calib_{k}": v_ for k, v_ in ev.items()}); print(" calib:", {k: round(v_, 4) for k, v_ in ev.items()}, flush=True) |
| 136 | + if score_of(ev) > best_score: | |
| 137 | + best_score, best_step = score_of(ev), step + 1 | |
| 138 | + best_state = {k: v_.detach().cpu().clone() for k, v_ in model.controller.state_dict().items()} | |
| 139 | + if best_state is not None: # early stopping on the calibration partition (never on test) | |
| 140 | + model.controller.load_state_dict(best_state) | |
| 141 | + run.log(event="restored_best_calibration_checkpoint", step=best_step, score=best_score) | |
| 142 | + print(f"restored controller from step {best_step} (calibration score {best_score:.5f})", flush=True) | |
| 132 | 143 | model.eval() |
| 133 | 144 | ev_train = evaluate_labels(model, cfg, recs[: min(len(recs), 40)], device) |
| 134 | 145 | ev_calib = evaluate_labels(model, cfg, calib["records"], device) |
modified
training/train_editor.py
+8 −2
@@ -24,6 +24,8 @@ from data.synthetic_dataset import AnyTokenizer, SyntheticBatches, exact_match, | ||
| 24 | 24 | |
| 25 | 25 | def build_model(exp: dict): |
| 26 | 26 | m = exp["model"] |
| 27 | + if m == "from_checkpoint": # read the exact config saved with the checkpoint (incl. readout_scale) | |
| 28 | + m = json.loads((Path(exp["checkpoint"]) / "checkpoint.json").read_text())["config"] | |
| 27 | 29 | cfg = load_config(m) if isinstance(m, str) else PrismeConfig.from_dict(m) |
| 28 | 30 | for k, v in exp.get("model_overrides", {}).items(): |
| 29 | 31 | setattr(cfg, k, v) |
@@ -111,12 +113,16 @@ def main(argv=None) -> int: | ||
| 111 | 113 | else: |
| 112 | 114 | tok = AnyTokenizer(data.get("tokenizer", "char")) |
| 113 | 115 | assert tok.vocab_size <= cfg.vocab_size, (tok.vocab_size, cfg.vocab_size) |
| 114 | − train_set = SyntheticBatches(load_items(Path(data["splits_dir"]) / "train.jsonl"), tok, tr["seq_len"]) | |
| 116 | + train_items = load_items(Path(data["splits_dir"]) / "train.jsonl") | |
| 117 | + if data.get("max_train_items"): | |
| 118 | + k = len(train_items) / data["max_train_items"] | |
| 119 | + train_items = [train_items[int(i * k)] for i in range(data["max_train_items"])] | |
| 120 | + train_set = SyntheticBatches(train_items, tok, tr["seq_len"]) | |
| 115 | 121 | train_iter = train_set.random_batches(tr["batch_size"], seed, device, data.get("loss_span", "answer")) |
| 116 | 122 | val_sets = {name: SyntheticBatches(load_items(Path(data["splits_dir"]) / f"{name}.jsonl"), tok, tr["seq_len"]) |
| 117 | 123 | for name in data.get("eval_splits", ["dev_iid"])} |
| 118 | 124 | main_val = val_sets[data.get("eval_splits", ["dev_iid"])[0]] |
| 119 | − val_fn = lambda nb: main_val.sequential_batches(tr["batch_size"], device, data.get("loss_span", "answer"), nb * tr["batch_size"]) | |
| 125 | + val_fn = lambda nb: main_val.sequential_batches(tr["batch_size"], device, data.get("loss_span", "answer"), nb * tr["batch_size"]) # stratified subset (every k-th item) | |
| 120 | 126 | budgets = exp.get("budgets", [0, 1, 2, 4, 8]); probs = exp.get("budget_probs", [1 / len(budgets)] * len(budgets)) |
| 121 | 127 | probs = [p / sum(probs) for p in probs] |
| 122 | 128 | run = Run(exp.get("name", "editor"), {"experiment": exp, "model": cfg.to_dict(), "resolved_train": tr}, root=args.run_root, seed=seed) |
| 123 | 129 | |