TL;DR
This guide develops Your RL Training Loop Is a Distributed System from its central engineering problem to practical design choices and limitations. It synthesizes the roadmap topics without claiming new experimental results. Reported findings belong to the cited sources.
Target searches: “RL training infrastructure”, “LLM RL system design”
Why This Topic Matters
Modern LLM and agent training turns familiar reinforcement-learning ideas into systems problems: actions may be tool calls, rewards may come from fallible graders, and trajectories may span many partially observed steps. The classical foundation is distributed RL; this article focuses on what changes in the newer setting.
The Core Ideas
1. The central fact: an RL post-training run is two workloads glued together — rollout generation (inference-shaped: KV cache, inference weight layout, vLLM/SGLang) and policy update (training-shaped: optimizer states, activations, FSDP/Megatron sharding)
2. Colocated vs disaggregated architectures — share the GPUs and swap, or split into separate pools
3. Weight synchronization, the problem nobody explains. After every update the trainer’s weights must reach the inference engine. Naive checkpoint-to-disk takes minutes per step. verl’s HybridEngine swaps layouts in place on the same devices. Walk through what “swapping layout” actually means
4. The step timeline, annotated — where the seconds actually go
5. Memory: rollout KV cache vs training optimizer state, and how they contend
6. Throughput math: what determines steps/hour
7. Where each framework sits on these axes
Practical Checklist
- Define the state, action, termination condition, and reward before choosing an optimizer.
- Log raw trajectories and every reward component, not only the aggregate score.
- Separate training measurements from held-out evaluation and adversarial tests.
- Treat generated code, tool calls, and environment actions as untrusted.
- Record model, framework, benchmark, and grader versions so comparisons remain reproducible.
Limitations
This area changes quickly. Framework support and benchmark leaderboards are time-sensitive, while many recent methods are preprints rather than settled practice. The article describes mechanisms and reported evidence; it does not present an original training run.
Key Learnings
- The central fact: an RL post-training run is two workloads glued together — rollout generation (inference-shaped: KV cache, inference weight layout, vLLM/SGLang) and policy update (training-shaped: optimizer states, activations, FSDP/Megatron sharding)
- Colocated vs disaggregated architectures — share the GPUs and swap, or split into separate pools
- Weight synchronization, the problem nobody explains. After every update the trainer’s weights must reach the inference engine. Naive checkpoint-to-disk takes minutes per step. verl’s HybridEngine swaps layouts in place on the same devices. Walk through what “swapping layout” actually means