spb/forge Public MIT
Forge — LLM training from scratch in pure C++20 + Metal on Apple Silicon.
C++ 61.2%
C 23%
Python 7.6%
TeX 7.2%
CMake 1.1%
-
Add QAT, MoE, and architecture-variant knobs — all config-selected
…
- quant "int8"|"ternary": per-row fake-quant each forward (BitNet-style absmean for ternary), straight-through estimator backward, f32 masters; wired through the Linear quantization seam - n_experts/moe_top_k/n_shared_experts: softmax router, renormalized top-k gates (topk_renorm + row_scale ops, CPU+Metal), differentiable load-balance loss, DeepSeek-style always-active shared experts; v1 computes experts densely (correctness first) - qk_norm (Qwen3/Gemma3), final_softcap (Gemma2), scale_embeddings (Gemma) - new kernels: quant.metal, moe.metal, softcap in elementwise.metal - CPU references + parity tests for every new op and full-model variants (QAT int8/ternary, MoE 4+1shared, qk-norm+softcap+embed-scale) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>