A reinforcement-learning post-training run reports 85% GPU-busy across the cluster, and someone reads that as a saturated fleet. It is almost never that. In RL training the busy number is an average over two phases with completely different appetites for the hardware — a generation (rollout) phase that behaves like batched inference, and a policy-training phase that behaves like ordinary gradient-descent training. Average them together and you get a figure that looks healthy while one phase is idling the accelerators you are paying for. This is the specific failure that RL frameworks such as SLIME are built to attack. The interesting part is not the framework name — it is why the naive mental model of “one big training job, GPUs busy the whole time” breaks down the moment you put a rollout loop and a training loop on the same silicon. How does the SLIME RL framework work? SLIME is a framework for RL post-training of language models — the RLHF / RLVR class of workloads where you fine-tune a policy against a reward signal. Its central design decision is the one that matters for utilisation: it treats rollout generation and policy training as distinct workloads with their own resource profiles, rather than as one monolithic loop. Concretely, a step of RL post-training does two things. First it generates: the current policy produces completions for a batch of prompts, scored by a reward model or a verifiable check. Then it trains: those completions and rewards feed a gradient update to the policy weights. Generation is autoregressive decoding — memory-bandwidth-bound, KV-cache-heavy, exactly the regime an inference server like vLLM or SGLang is optimised for. Training is a dense backward pass — compute-bound, activation-memory-heavy, the regime PyTorch’s FSDP or DeepSpeed ZeRO is optimised for. These are not the same workload wearing two hats. They stress different parts of the GPU. The framework’s job is to let each phase run on a stack tuned for it, and to overlap them so that while the trainer is consuming one batch of rollouts, the rollout engine is already generating the next. If you have internalised how async RL rollouts change GPU inference cost, this is the same idea from the utilisation side rather than the cost side. How does an RL training pipeline split GPU work between rollout and policy training? There are three broad topologies, and the split you choose is the single largest determinant of idle time. Co-located. Rollout and training share the same GPUs, running sequentially. Simplest to build. Also the classic trap: during generation the training kernels are idle, and during the backward pass the inference engine sits on cold KV caches. You have provisioned for peak concurrent demand and get serial execution. Disaggregated. Rollout runs on one pool of GPUs, training on another. Each pool runs a stack tuned for its phase, and the two overlap in time. This is closer to how SGLang PD disaggregation separates prefill and decode — the same architectural instinct, applied to the rollout/train boundary instead of the prefill/decode boundary. Hybrid / partial overlap. A single pool time-shares but pipelines: rollout for step N+1 runs while training for step N completes. This recovers a lot of the idle time of the co-located design without doubling your GPU count, at the cost of tighter memory budgeting since both stacks must fit resident. The following table is the decision surface most teams actually need. Topology Idle-time risk GPU count Memory pressure Best when Co-located, serial High — one phase idles the other Lowest Low (one stack resident) Prototyping; small models; you are not yet cost-sensitive Disaggregated pools Low — phases overlap Highest Low per pool Large models; sustained runs; rollout is the bottleneck Hybrid pipelined Medium — depends on pipeline depth Medium High (both stacks resident) You want overlap without a second GPU pool There is no universally correct row. The right choice depends on which phase dominates your wall-clock, and you cannot know that without measuring — which is the whole point of the section below. Why is aggregate GPU-busy percentage misleading for reinforcement learning workloads? Because “busy” is a boolean per instant, and the aggregate hides which phase was busy and how well it used the silicon while busy. Two failure modes stack here. First, temporal averaging: if generation takes 60% of the step wall-clock and training 40%, a co-located setup shows GPUs busy most of the time even though the training kernels were idle for the whole generation window and vice versa. The percentage is high; the useful work is low. Second, the busy-flag lies about intensity. nvidia-smi reports a GPU as busy if any kernel is resident, so a bandwidth-bound decode step that leaves most of the tensor cores idle still reads as 100% utilisation. This is the same trap we describe in why GPU utilisation metrics can mislead you: the flag answers “is something running” when the question you care about is “is the expensive part of the chip doing useful work.” For RL specifically the two failure modes compound. Rollout is memory-bandwidth-bound and shows high busy-flag with low compute intensity; training is compute-bound. Averaging a bandwidth-bound phase and a compute-bound phase into one number produces a figure that is not interpretable as either. The operationally relevant measurement is utilisation per phase, reported separately, with the intensity dimension (SM occupancy, achieved FLOP rate) alongside the busy flag. Where do RL post-training pipelines most often leave GPU capacity idle? In our experience profiling GPU-bound pipelines, the idle time in RL post-training clusters in a handful of predictable places (observed pattern across engagements, not a benchmarked distribution): Phase-transition stalls. The handoff from generation to training — weight sync, batch reshaping, moving rollouts onto the trainer — is a serial gap where neither phase runs. In a co-located design this gap is pure idle. Rollout tail latency. A batch of completions finishes when the slowest sequence finishes. Long-tail generations leave most of the rollout GPUs idle waiting on a few stragglers, unless the scheduler continuously refills. Weight-sync barriers. After each training update the new policy weights must reach the rollout engine before the next generation can start on-policy. A synchronous barrier here idles the rollout pool for the duration of the transfer. Reward-model bottlenecks. If the reward model runs on the same GPUs and is not overlapped, scoring becomes a third serial phase nobody budgeted for. None of these is visible in an aggregate busy number. All of them are visible when you profile the two phases separately and watch the transitions. How do I measure GPU utilisation per phase in an RL framework like SLIME? Instrument the phase boundaries first, then attach the profiler to each phase. The framework already knows when it is generating versus training — emit a marker (an NVTX range or a simple timestamped log) at each phase entry and exit so the profiler timeline can be sliced by phase afterwards. A workable measurement recipe: Tag the phases. Wrap the rollout loop and the training loop in named NVTX ranges. Nsight Systems will then show you a timeline you can read phase by phase. Capture both dimensions per phase. Busy percentage answers “was a kernel resident”; SM occupancy and achieved FLOP rate (from Nsight Compute or DCGM) answer “was the compute doing work.” Record both for rollout and for training separately. Measure the transitions explicitly. Time the weight-sync and batch-reshape gaps as their own line items. This is where the recoverable idle usually hides. Attribute GPU-hours per phase. Multiply per-phase wall-clock by GPUs held during that phase. Now you have rollout GPU-hours and training GPU-hours as separate numbers. This is the same decomposition thinking behind the three pillars of observability applied to GPU utilisation — logs, metrics, and traces mapped onto the phases rather than onto the cluster as an undifferentiated whole. When we run a GPU performance audit on an RL pipeline, isolating rollout-phase versus training-phase idle time is the first deliverable, because it tells you which phase is wasting capacity before anyone talks about adding hardware. Should I add GPUs for RL post-training, or first profile the rollout/training utilisation gap? Profile first. Almost always. Adding GPUs to a co-located, serial pipeline often buys less than expected, because the bottleneck is phase serialisation, not raw capacity — more GPUs sitting idle during the wrong phase is still idle GPUs. The higher-leverage move is usually to restructure toward overlap: disaggregate the pools or pipeline the phases so that the rollout engine and the trainer are both doing useful work at the same wall-clock moment. Here is a worked example with explicit, illustrative assumptions (not a benchmark — the numbers exist to show the shape of the reasoning). Suppose a run holds 16 GPUs and each RL step is 60% generation, 40% training, run co-located and serial. The training GPUs are idle during generation and vice versa, so the effective useful utilisation is bounded well below the 90%+ busy flag you would read off the aggregate. Move to a hybrid pipelined design where step N+1 generation overlaps step N training, and the idle window shrinks to the phase-transition gaps. The GPU-hours consumed per completed run drop not because you added silicon, but because you stopped paying for two phases to take turns. This connects directly to the TCO-per-useful-FLOP framing on the GPU engineering practice page: the metric that matters is GPU-hours per completed run, not the instantaneous busy percentage. How do I estimate GPU-hours per completed RL run and reduce them by overlapping phases? Start from the phase decomposition you already measured. GPU-hours per run is the sum, across steps, of (GPUs held × wall-clock) for each phase, plus the transition gaps. Once that ledger exists, the reduction levers are concrete: Overlap the phases so rollout for the next step runs during training of the current one — this attacks the largest line item, serialisation idle. Refill the rollout scheduler continuously so long-tail generations do not idle the whole batch. Make weight sync asynchronous or pipelined so the rollout pool is not barred waiting for new weights. Right-size the split between rollout and training GPUs to the ratio your profile revealed, rather than a 50/50 guess. The estimate is only as good as the phase attribution underneath it, which is why the measurement step is not optional. FAQ What matters most about the SLIME RL framework in practice? SLIME is a framework for RL post-training of language models that treats rollout generation and policy training as distinct workloads with different resource profiles. Generation behaves like batched inference (memory-bandwidth-bound); training behaves like a dense backward pass (compute-bound). The framework lets each phase run on a stack tuned for it and overlaps them so the rollout engine generates the next batch while the trainer consumes the current one. How does an RL training pipeline split GPU work between rollout and policy training? Three topologies exist: co-located serial (both phases share GPUs and take turns, which idles one phase while the other runs), disaggregated pools (separate GPU pools per phase, overlapping in time), and hybrid pipelined (one pool time-shares but pipelines steps). The split is the single largest determinant of idle time, and the right choice depends on which phase dominates your wall-clock. Why is aggregate GPU-busy percentage misleading for reinforcement learning workloads? Because it averages two phases with different profiles and because the busy flag reports a GPU as busy if any kernel is resident, regardless of whether the expensive compute units are working. A bandwidth-bound decode step reads as 100% busy while most tensor cores idle. Averaging a bandwidth-bound rollout phase with a compute-bound training phase produces a number that is not interpretable as either. Where do RL post-training pipelines most often leave GPU capacity idle? At phase-transition stalls (weight sync and batch reshaping between generation and training), rollout tail latency (a batch finishes when the slowest sequence finishes), weight-sync barriers (rollout waits for new on-policy weights), and reward-model bottlenecks when scoring is not overlapped. None of these show up in an aggregate busy number; all become visible when you profile the two phases separately. How do I measure GPU utilisation per phase in an RL framework like SLIME? Tag the rollout and training loops with named NVTX ranges so a profiler timeline can be sliced by phase, then capture both busy percentage and intensity (SM occupancy, achieved FLOP rate via Nsight Compute or DCGM) for each phase separately. Time the transitions as their own line items, then attribute GPU-hours per phase by multiplying per-phase wall-clock by the GPUs held. Should I add GPUs for RL post-training, or first profile the rollout/training utilisation gap? Profile first. Adding GPUs to a co-located serial pipeline often buys less than expected because the bottleneck is phase serialisation, not capacity — you get more idle GPUs during the wrong phase. The higher-leverage move is restructuring toward overlap (disaggregation or pipelining) so both the rollout engine and trainer do useful work at the same moment. How do I estimate GPU-hours per completed RL run and reduce them by overlapping phases? Sum, across steps, the (GPUs held × wall-clock) for each phase plus the transition gaps. Then reduce it by overlapping phases, refilling the rollout scheduler continuously to avoid tail-latency idle, making weight sync asynchronous, and right-sizing the rollout/training GPU split to the ratio your profile revealed rather than a guess. The estimate is only as reliable as the phase attribution beneath it. The clean question to carry away is not “are my GPUs busy” — it is “which phase is idling them, and what does overlapping the two cost me in memory versus save me in GPU-hours.” A GPU performance audit that reports rollout-phase and training-phase idle time separately answers that before you sign a bigger cloud bill.
