spb/forge Public MIT
Forge — LLM training from scratch in pure C++20 + Metal on Apple Silicon.
C++ 61.2%
C 23%
Python 7.6%
TeX 7.2%
CMake 1.1%
-
Add HF streaming data pipeline and three research reports
…
- tools/prepare_hf_data.py: stream any of 13 registered HF datasets (FineWeb-Edu, DCLM, Cosmopedia, FineMath, OpenWebMath, Wikipedia, C4, SmolTalk, ...) or weighted mixtures/presets (smollm-web, textbooks, decay-anneal) straight into train.bin/val.bin — no full downloads - SMALL_MODELS_RESEARCH.md: how sub-1B models get logical, useful text (data quality, deep-and-thin, distillation, test-time compute) - INFERENCE_RESEARCH.md: Apple Silicon inference speed playbook tied to the .forge format (bandwidth math, fused-dequant GEMV, KV cache, residency sets, warmup, quant layouts) with a prioritized roadmap - ARCHITECTURES.md: config matrix to train Llama/Mistral/Qwen/Gemma/ DeepSeek/Kimi-class variants, with a 3-wave implementation plan - README: training modes, .forge format, tools Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>