https://ggrigorev.me/Recent content in About on George Grigorev BlogHugoen-usFri, 31 Oct 2025 22:00:00 +0000https://ggrigorev.me/posts/introduction-to-parallelism/Fri, 31 Oct 2025 22:00:00 +0000https://ggrigorev.me/posts/introduction-to-parallelism/<p>Training large models inevitably requires a solid understanding of parallelism techniques. In this post, I&rsquo;ll give a practical, in-depth overview of the most common approaches — DDP, FSDP, and TP — and how they&rsquo;re actually used in real PyTorch training setups.</p> <p>This article was inspired by the excellent “How to Scale Your Model” <a href="https://jax-ml.github.io/scaling-book/index">blog series</a>. While that series is clear and insightful, I felt it was missing some hands-on perspective and real-world lessons from someone who has trained models in the wild.</p>https://ggrigorev.me/posts/tokenizer-superbpe/Tue, 07 Oct 2025 00:00:00 +0000https://ggrigorev.me/posts/tokenizer-superbpe/Byte-level BPE from first principles: what matters for speed and quality, how to implement it cleanly, and why a SuperBPE variant can lift sample efficiency.