spb/forge Public MIT
Forge — LLM training from scratch in pure C++20 + Metal on Apple Silicon.
C++ 61.2%
C 23%
Python 7.6%
TeX 7.2%
CMake 1.1%
-
Add architecture-variant waves 1+2: train Mistral/Qwen/Gemma/OLMo-class models by config
…
Wave 1 (module-level): - rope refactor: kernels read a host-precomputed inv-freq table; unlocks HF-"llama3" rope scaling (rope_scale_*), per-layer theta, and NoPE layers (nope_every, SmolLM3) - attention_bias (Qwen2.5 QKV bias), head_dim decoupled from d_model/n_heads (Qwen3), relu2 activation (nanoGPT-speedrun lineage), norm_placement pre|post|sandwich (OLMo2/Gemma) Wave 2 (attention kernels): - sliding_window + sliding_global_every (Mistral / Gemma3 local:global patterns) in the CPU reference, the unfused Metal kernels, and the fused scalar flash kernels — out-of-window KV blocks are skipped, so cost scales with the window; window > 0 auto-routes off the MMA kernel - attn_softcap (Gemma2): cap*tanh on scores pre-softmax, unfused path, exact tanh' chain in all backwards - rope_theta_global for dual-theta local/global layers (Gemma3) Parity suites cover every knob (fused + unfused paths); gradcheck and overfit stay green. New demo configs: gpt-50m-mistral, gpt-50m-gemma; ARCHITECTURES.md documents the per-family config matrix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>