spb/forge Public MIT
Forge — LLM training from scratch in pure C++20 + Metal on Apple Silicon.
C++ 61.2%
C 23%
Python 7.6%
TeX 7.2%
CMake 1.1%
-
Add configurable training modes: Muon optimizer and WSD schedule
…
- optimizer "adamw" | "muon": Newton-Schulz orthogonalized momentum on 2-D hidden matrices (composed from the existing matmul kernels on Metal), AdamW kept for embeddings/head/1-D params; muon_lr follows the lr schedule - schedule "cosine" | "wsd": warmup-stable-decay with 1-sqrt cooldown, extendable runs, wsd_decay_frac - gpt-50m base config + Muon+WSD and MobileLLM-style deep-and-thin variants Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>