A Journey of Reinforcement Learning Begins with Miles
"A journey of a thousand miles is made one small step at a time."
At the forefront of innovation in reinforcement learning (RL) is Miles, a robust framework tailored for large-scale mixture of experts (MoE) training and production workloads. In this introductory chapter, we will embark on a series of tech blogs detailing the unique features, improvements, and roadmap of this exciting initiative.
The Genesis of Miles: Building on Slime
Miles is effectively a fork of slime, a lightweight RL framework that has quietly supported many of today’s post-training pipelines involving complex MoE setups. The foundation laid by slime serves as the springboard upon which Miles is built, promising teams an even more controlled, reliable, and scalable RL experience in real-world applications.
For those interested in exploring Miles, you can find the framework available on GitHub.
Slime: A Lightweight and Customizable RL Framework
Every significant advancement in technology begins with a solid starting point, and for Miles, that starting point is slime. Known for its lightweight design and high customizability, slime has gained popularity within the community, especially for large MoE training tasks. Notably, it has been successfully employed to train complex models, such as GLM-4.6.
More Read
Performance and Structure
-
Native Performance: Slime is developed with structured support for SGLang and Megatron’s optimization stack. This ensures it evolves in tandem with the rapid progress of inference and training frameworks, allowing for seamless integration and performance tuning.
-
Modular Design: The architecture of slime is inherently modular, with its core components—Algorithm, Data, Rollout, and Evaluation—fully decoupled. Users can easily integrate new agent types, reward functions, or sampling strategies without extensive modifications to the existing codebase.
-
Researcher-Friendly Abstractions: The framework is designed to be accessible for algorithm researchers, allowing for adjustments to crucial components like importance sampling, rollout logic, or loss dynamics. This flexibility comes without needing to dive into low-level code.
- Community-Driven Development: Slime has benefitted immensely from feedback within the LMSYS and SGLang communities. This open collaboration bridges the gap between research and engineering, leading to significant improvements based on real-world challenges.
Recent Advancements in Miles
While drawing from the strengths of slime, Miles has introduced several enhancements aimed at tackling the specific demands of large-scale MoE reinforcement learning and ensuring production-grade reliability.
True On-Policy Support
One of the most significant features added to Miles is true on-policy support. This design eliminates the disparity between training and inference by ensuring they align perfectly. Utilizing technologies such as Flash Attention 3 and DeepGEMM, Miles achieves zero mismatch, optimizing performance for complex training environments.
Memory Management Improvements
Miles introduces substantial updates aimed at optimizing GPU memory use without succumbing to out-of-memory (OOM) errors. Key updates include:
- Improved propagation strategies
- Implementing memory margins to resolve NCCL-related OOM issues
- Enhancements that cater to model parallel distributed training (FSDP)
These updates ensure that the framework operates smoothly even in resource-constrained environments, maximizing efficiency.
Speculative Training
Miles operates optimally by implementing online SFT (Self-Training Fine-Tuning) concurrently with reinforcement learning. This innovative method boosts rollout speed by over 25% compared to traditional methods, particularly during the later training stages.
Enhancements with Speculative Training:
- Support for multi-task planning (MTP) with sequence packing.
- Incremental gradient isolation for improved training outcomes.
Roadmap for Future Development
Looking ahead, Miles will prioritize enhancements that cater to the enterprise-level needs of RL training. Upcoming plans include:
- Extensive support for large-scale MoE RL on pioneering hardware, such as the GB300.
- Development of multi-modal training capabilities to enrich model versatility.
- Rollout accelerations that are compatible with SGLang specifications to enhance performance further.
- Robust resource allocation systems that balance training and serving effectively in larger asynchronous setups.
- Mechanisms to ensure computational resilience against GPU failures, enhancing system robustness.
Gratitude to Our Community
The foundation and ongoing advancements of Miles owe much to the insights and contributions of the slime authors and the broader SGLang RL community. Miles stands as a testament to the power of collaboration and innovation in overcoming real-world challenges in reinforcement learning.
We invite researchers, startups, and enterprise teams to delve into the capabilities of both slime and Miles, joining us on a path towards making reinforcement learning more efficient, dependable, and transformative. Together, we can shape the future of RL, step by step.
Inspired by: Source
