200M cluster run: one epoch at micro-batch 8

8 x 32 x 1024 = 262144 tokens/step (effective batch unchanged, so the 3e-4 LR
still applies) x 1553 steps = one epoch over the 407,344,713-token corpus.
Micro-batch 8 rather than 16 halves activation memory, which was sitting at
65 of 77.8 GB working set.

Simon-Pierre Boucher committed 10 days ago (Jul 31, 2026) parent 6ec25b6

Showing 1 changed file with +8 and −8

modified configs/gpt-200m-cluster.json +8 −8

		`@@ -1,5 +1,5 @@`
1	1	{
2		− "_comment": "205.6M params for the M3U96a run (Apple M3 Ultra, 60-core GPU, 96 GB, 77.8 GB working set). vocab 4096 keeps the tied embedding cheap and matches the proven BPE pipeline; d_ff 3072 brings the total back over 200M. batch_size is the MICRO-batch: 16 x 16 x 1024 = 262144 tokens/step. Micro-batch 16 (not 4) because 96 GB affords it and it cuts sync points 4x. precision is parsed but not yet honored - all kernels are f32.",
	2	+ "_comment": "205.6M params, ONE EPOCH over the full TinyStories corpus on M3U96a (M3 Ultra, 60-core GPU, 96 GB). 8 x 32 x 1024 = 262144 tokens/step x 1553 steps = 407.1M of 407,344,713 tokens (99.94%). Micro-batch 8 halves activation memory vs 16; grad_accum 32 keeps the effective batch (and so the 3e-4 LR) unchanged. precision is parsed but not yet honored - all kernels are f32.",
3	3	"model": {
4	4	"name": "gpt-200m-cluster",
5	5	"n_layers": 16,
		`@@ -18,20 +18,20 @@`
18	18	"dropout": 0.0
19	19	},
20	20	"train": {
21		− "lr": 3e-4,
	21	+ "lr": 0.0003,
22	22	"min_lr_ratio": 0.1,
23		− "warmup_steps": 150,
24		− "max_steps": 1500,
	23	+ "warmup_steps": 155,
	24	+ "max_steps": 1553,
25	25	"beta1": 0.9,
26	26	"beta2": 0.95,
27	27	"eps": 1e-08,
28	28	"weight_decay": 0.1,
29	29	"grad_clip": 1.0,
30		− "batch_size": 16,
31		− "grad_accum_steps": 16,
	30	+ "batch_size": 8,
	31	+ "grad_accum_steps": 32,
32	32	"precision": "f32",
33		− "checkpoint_every": 50,
34		− "eval_every": 50,
	33	+ "checkpoint_every": 100,
	34	+ "eval_every": 100,
35	35	"eval_batches": 10,
36	36	"seed": 1337
37	37	}
38	38