Forge — LLM training from scratch in pure C++20 + Metal on Apple Silicon.
-
Wed, Aug 5, 2026 2
-
Add wave 3: DeepSeek-V3-style MoE routing, all config-selected
…
- moe_scoring "softmax"|"sigmoid" (new sigmoid op, CPU+Metal+backward) - aux-loss-free balancing (V3 "noaux"): top-k selection ranks score+bias while gate values stay biasless; per-expert load counted on-GPU each forward and the balance bias nudged ±moe_bias_gamma after every optimizer step; bias is checkpointed as a grad-free parameter - moe_norm_topk (renormalize kept gates or keep raw sigmoid scores), routed_scaling_factor (V3: 2.5), moe_d_ff (per-expert width), first_k_dense (dense MLPs for the first k layers) - parity tests: biased/no-renorm topk, sigmoid, expert_counts, and a full V3-style model (sigmoid/noaux/scaled/first-dense) — CPU==GPU Remaining wave-3 items (MLA, MuonClip QK-clip, MTP) documented in ARCHITECTURES.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
-
Add HF streaming data pipeline and three research reports
…
- tools/prepare_hf_data.py: stream any of 13 registered HF datasets (FineWeb-Edu, DCLM, Cosmopedia, FineMath, OpenWebMath, Wikipedia, C4, SmolTalk, ...) or weighted mixtures/presets (smollm-web, textbooks, decay-anneal) straight into train.bin/val.bin — no full downloads - SMALL_MODELS_RESEARCH.md: how sub-1B models get logical, useful text (data quality, deep-and-thin, distillation, test-time compute) - INFERENCE_RESEARCH.md: Apple Silicon inference speed playbook tied to the .forge format (bandwidth math, fused-dequant GEMV, KV cache, residency sets, warmup, quant layouts) with a prioritized roadmap - ARCHITECTURES.md: config matrix to train Llama/Mistral/Qwen/Gemma/ DeepSeek/Kimi-class variants, with a 3-wave implementation plan - README: training modes, .forge format, tools Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
-