spb/forge Public MIT
Forge — LLM training from scratch in pure C++20 + Metal on Apple Silicon.
C++ 61.2%
C 23%
Python 7.6%
TeX 7.2%
CMake 1.1%
-
Add wave 3: DeepSeek-V3-style MoE routing, all config-selected
…
- moe_scoring "softmax"|"sigmoid" (new sigmoid op, CPU+Metal+backward) - aux-loss-free balancing (V3 "noaux"): top-k selection ranks score+bias while gate values stay biasless; per-expert load counted on-GPU each forward and the balance bias nudged ±moe_bias_gamma after every optimizer step; bias is checkpointed as a grad-free parameter - moe_norm_topk (renormalize kept gates or keep raw sigmoid scores), routed_scaling_factor (V3: 2.5), moe_d_ff (per-expert width), first_k_dense (dense MLPs for the first k layers) - parity tests: biased/no-renorm topk, sigmoid, expert_counts, and a full V3-style model (sigmoid/noaux/scaled/first-dense) — CPU==GPU Remaining wave-3 items (MLA, MuonClip QK-clip, MTP) documented in ARCHITECTURES.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>