SPB Git

spb/forge Public MIT

Forge — LLM training from scratch in pure C++20 + Metal on Apple Silicon.

llmtrainingcppmetalapple-silicon
C++ 61.2% C 23% Python 7.6% TeX 7.2% CMake 1.1%

History of configs/gpt-200m-cluster.json · clear filter

  1. 200M cluster run: one epoch at micro-batch 8
    …
    8 x 32 x 1024 = 262144 tokens/step (effective batch unchanged, so the 3e-4 LR
still applies) x 1553 steps = one epoch over the 407,344,713-token corpus.
Micro-batch 8 rather than 16 halves activation memory, which was sitting at
65 of 77.8 GB working set.
    Simon-Pierre Boucher committed 10 days ago (Jul 31, 2026) · 1 file changed +8 −8
  2. Add 205.6M-parameter config for the M3 Ultra cluster run
    …
    vocab 4096 (matches the proven BPE pipeline; the trainer's pair-count array is
O(V^2) and the encoder O(V*N), which does not scale to 16384 over 400M tokens),
with d_ff 3072 restoring the parameter count to 205.6M. Micro-batch 16 rather
than 4 since 96 GB affords it and it cuts sync points 4x.
    Simon-Pierre Boucher committed 10 days ago (Jul 31, 2026) · 1 file changed +38