Reach for a profiler on a reinforcement-learning post-training run and the first instinct is almost always wrong: tune the generation kernels, tune the training step, squeeze occupancy. SLIME, an open framework for scaling RL post-training of large models, exists because that instinct leaves most of the throughput on the table. It couples a rollout/generation engine to a training backend, and the interesting engineering is not inside either component — it is in how the two are decomposed, placed, and overlapped so neither sits idle waiting on the other. That distinction is the whole point of this article. RL post-training is not a single workload you can kernel-tune your way through. It is a structural composition problem, and the largest speedups come from getting the composition right. How does SLIME RL work in practice? At its core, a SLIME run alternates between two jobs with very different execution profiles. The rollout engine generates samples — it runs the current policy as an inference workload, producing sequences (and often reward signals) from prompts. The training backend then consumes those samples to update the policy weights, which is a gradient-computation workload with entirely different memory and compute characteristics. Generation is autoregressive and latency-bound: it lives on the same bottlenecks that shape any LLM inference path — KV-cache pressure, batching efficiency, decode-step throughput. Training is a dense forward-backward pass, bandwidth-hungry, often sharded across GPUs. These are not the same shape of work, and treating them as one workload is the naive framing SLIME is built to replace. Because the policy changes after every update, the two stages are coupled: fresh weights have to reach the rollout engine before the next batch of samples is meaningfully on-policy. That coupling is where the imbalance shows up, and it is the reason a faster attention kernel rarely moves the needle at the system level. The two components and how they run on GPUs A useful way to hold the picture is to separate what each stage is from where it runs. Stage Workload character Dominant GPU concern Typical bottleneck Rollout / generation Autoregressive inference Decode throughput, KV-cache capacity Under-batched decode, memory-bound steps Reward computation Model or rule-based scoring Varies (may be a second model) Serialized after generation Policy update (training) Dense forward-backward Interconnect bandwidth, sharded state Waiting on samples to arrive Weight sync Parameter transfer Interconnect, placement Trainer↔rollout handoff latency The rollout engine typically leans on an inference-serving stack — the same class of runtime you would profile when separating prefill and decode for GPU throughput. The trainer leans on a distributed-training stack, often something in the FSDP or Megatron family. Two runtimes, two sets of assumptions, one shared set of GPUs. How you allocate those GPUs between the two — and whether you run them on the same devices or split them — is the first real decision, and it is an algorithmic one, not a kernel one. Why throughput balance is a structural problem, not a kernel-tuning one Here is the mechanism that makes this stick. Suppose your rollout engine produces samples at some rate, and your trainer consumes them at another. If those rates do not match, one side stalls. A trainer that finishes an update and then waits 40% of wall-clock time for the next batch of rollouts is not slow because its kernels are slow — it is slow because the pipeline is imbalanced. No amount of kernel tuning inside the trainer fixes that. You could double the trainer’s step speed and simply increase its idle fraction; the constraint lives in the relationship between the two stages, not inside either one. This is the algorithmic-restructuring-versus-micro-optimization boundary applied directly to RL, and it is the same boundary that shows up whenever multi-GPU launch configuration is treated as algorithmic restructuring rather than a config afterthought. The three levers that actually change the balance are structural: Placement — co-located (rollout and trainer share the same GPUs, time-slicing) versus disaggregated (dedicated GPUs for each). Co-location saves weight-transfer cost but forces the two workloads to contend; disaggregation removes contention at the cost of a network hop and idle risk if the split is wrong. Async rollout batching — decoupling generation from the update loop so the rollout engine keeps producing samples while the trainer works, rather than a strict generate-then-train lockstep. This is what turns a stall into overlap. Sample staleness tolerance — how off-policy you allow rollouts to be. Tightening it improves learning per sample but re-couples the stages; loosening it buys overlap at an algorithmic cost that has to be reasoned about, not measured after the fact. None of these is a kernel. Each of them is a decision about how compute is decomposed, and in our experience with GPU-bound pipelines this is consistently where the recoverable throughput sits (observed pattern across engineering audits; not a benchmarked rate). When restructuring beats optimizing individual kernels The honest answer is: not always, but you can tell which regime you are in. If the GPUs are genuinely saturated on both stages and neither is waiting, then you are in kernel territory and a faster FlashAttention build or a fused reward-scoring kernel is the right move. That is the minority case. The majority case looks like this: one stage runs hot while the other idles, aggregate GPU utilization sits well below what the hardware can sustain, and the wall-clock time is dominated by handoffs rather than compute. When you see that, the payoff from restructuring — moving to disaggregated placement, or introducing async rollout batching — is on a different order than kernel work. Restructuring the pipeline to eliminate stalls can shift end-to-end utilization far more than tuning any single kernel, because it changes the fraction of time the hardware is doing useful work at all. A worked framing, with explicit assumptions: suppose a co-located run spends roughly 35% of wall-clock time in trainer idle while the rollout engine catches up, and both stages’ kernels are already reasonably efficient. Shaving 15% off the trainer’s kernel time reduces total wall-clock by a small single-digit percentage — because you are speeding up a stage that already has slack. Eliminating the 35% stall through async batching addresses the dominant term directly. This is illustrative arithmetic, not a benchmark result, but it captures why the ordering of interventions matters more than their individual difficulty. How to tell your RL stack has hit a kernel-tuning ceiling Use this as a diagnostic rubric. If you check most of these, algorithmic restructuring is the next move, not more kernel work: Aggregate GPU utilization is well below sustained peak despite both components being individually “optimized.” One stage’s idle-wait fraction is large — the trainer is blocked on rollouts, or the rollout engine is blocked waiting for fresh weights. Profiling shows time in synchronization and transfer, not in matmuls or attention. Kernel-level speedups stopped translating into end-to-end throughput — you made a kernel 20% faster and total training time barely moved. Scaling out added GPUs but not proportional throughput, which usually means the imbalance got worse, not the compute. Points 2 and 4 are the strongest signals. When a real kernel improvement does not propagate to wall-clock time, the bottleneck is definitionally somewhere else — and in an RL pipeline that somewhere is almost always the generation-versus-training balance. This is the same reasoning we apply when reading GPU utilisation across the three pillars of observability: utilization is a system property, not a kernel property. What metrics show whether restructuring worked The measurable outcome is end-to-end RL training throughput, and it decomposes into a small set of numbers worth tracking: Samples generated and consumed per GPU-hour — the top-line efficiency number. This is what improves when overlap improves. Idle-wait fraction per stage — the share of wall-clock each stage spends blocked on the other. This is the term you are directly attacking. Weight-sync latency — how long the trainer→rollout handoff takes; rises with certain disaggregation choices. Sample staleness — how off-policy consumed samples are, which guards against buying throughput by degrading learning. Track these before and after any placement or batching change. If samples-per-GPU-hour rose and staleness stayed within tolerance, the restructuring worked. If throughput rose but staleness blew out, you traded correctness for speed — a bad deal that only the staleness metric exposes. Framing the whole exercise around these numbers, rather than kernel occupancy, is what keeps an RL post-training effort honest. Our broader work on GPU performance engineering treats this generation-versus-training balance as a first-order concern precisely because it dominates the outcome. FAQ What should you know about SLIME RL in practice? SLIME is an open framework for scaling reinforcement-learning post-training of large models. It couples a rollout/generation engine — which runs the current policy as an inference workload to produce samples — with a training backend that consumes those samples to update the policy. In practice it means an RL run alternates between two workloads with very different GPU profiles, and the engineering payoff is in how you compose them, not in tuning either one in isolation. What are the main components of a SLIME RL training pipeline, and how do they run on GPUs? The two main components are the rollout/generation engine and the training backend. Generation is autoregressive, latency-bound inference dominated by decode throughput and KV-cache pressure; training is a dense forward-backward pass dominated by interconnect bandwidth and sharded state. They typically run on different runtime stacks, and how you allocate GPUs between them — co-located or disaggregated — is the first structural decision. Why is throughput balance between rollout generation and policy training a structural problem rather than a kernel-tuning one? Because the constraint lives in the relationship between the two stages, not inside either. If generation and consumption rates do not match, one side stalls waiting on the other, and a faster kernel in the idle stage simply increases its idle fraction. The imbalance is fixed by changing how compute is decomposed and overlapped, which is an algorithmic move. When does restructuring the RL pipeline give a bigger GPU speedup than optimizing individual kernels? When aggregate GPU utilization is well below sustained peak, one stage idles while the other runs hot, and wall-clock time is dominated by handoffs rather than compute. In that regime — the common one — disaggregated placement or async rollout batching addresses the dominant idle term directly, whereas kernel tuning only speeds up a stage that already has slack. How do I tell that my RL training stack has hit a kernel-tuning ceiling and needs algorithmic restructuring instead? The clearest signals are that kernel-level speedups stopped translating into end-to-end throughput, and that profiling shows time spent in synchronization and transfer rather than in matmuls or attention. If you made a kernel meaningfully faster and total training time barely moved, the bottleneck is elsewhere — almost always the generation-versus-training balance. What metrics should I track to measure whether pipeline restructuring improved end-to-end RL training throughput? Track samples generated and consumed per GPU-hour as the top-line number, the idle-wait fraction per stage as the term you are attacking, weight-sync latency for handoff cost, and sample staleness to guard against trading correctness for speed. Compare all four before and after a change: throughput must rise while staleness stays within tolerance. The uncomfortable part is that this reasoning cuts against habit. Profilers point at kernels because kernels are what profilers measure well; the pipeline imbalance is a property no single kernel trace reveals. Before the next round of occupancy tuning on an RL run, the question worth asking is whether the trainer and the rollout engine are actually feeding each other — or just taking turns waiting. A GPU performance audit that classifies each intervention as algorithmic or micro-level exists to force exactly that question before the tuning work begins.
