spb/forge

Wed, Aug 5, 2026 1

1. Add HF streaming data pipeline and three research reports

   …

   - tools/prepare_hf_data.py: stream any of 13 registered HF datasets
     (FineWeb-Edu, DCLM, Cosmopedia, FineMath, OpenWebMath, Wikipedia, C4,
     SmolTalk, ...) or weighted mixtures/presets (smollm-web, textbooks,
     decay-anneal) straight into train.bin/val.bin — no full downloads
   - SMALL_MODELS_RESEARCH.md: how sub-1B models get logical, useful text
     (data quality, deep-and-thin, distillation, test-time compute)
   - INFERENCE_RESEARCH.md: Apple Silicon inference speed playbook tied to
     the .forge format (bandwidth math, fused-dequant GEMV, KV cache,
     residency sets, warmup, quant layouts) with a prioritized roadmap
   - ARCHITECTURES.md: config matrix to train Llama/Mistral/Qwen/Gemma/
     DeepSeek/Kimi-class variants, with a 3-wave implementation plan
   - README: training modes, .forge format, tools
   
   Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

   simon-pierre boucher 3 mo ago · 1 file +168
   6d77a01