“sglang rl” is one search term that covers two entirely different jobs. One is using SGLang as a rollout backend to generate training data during reinforcement-learning post-training. The other is running SGLang as a production inference server that answers user requests. People type the same query for both, benchmark the same engine, and then quietly carry a number from one context into a decision that belongs to the other. That is where the money leaks. The naive move looks reasonable from the outside. You enable SGLang’s RL rollout mode, turn on its scheduling and batching features, watch tokens-per-second climb, and conclude the serving cost story got better. It did not — or rather, you have no idea whether it did, because you measured aggregate generation throughput and then reasoned about per-request margin. Those are different objectives that happen to run through the same code path. The claim of this article is narrow and worth stating plainly: a rollout tokens-per-second headline does not translate into production margin unless it was measured in cost-per-request units, at the p95 latency your product actually enforces, from the start. Everything else here is the mechanism behind that sentence. What does “sglang rl” actually mean in practice? SGLang is a serving engine for large language models. It became popular for two reasons that pull in opposite directions. First, its runtime is genuinely fast under load. RadixAttention caches shared prefix computation across requests, continuous batching keeps the GPU busy instead of stalling between requests, and the scheduler packs work aggressively. Those properties make it an excellent inference server. Second, those same properties make it an excellent rollout backend for reinforcement-learning post-training pipelines — RLHF, GRPO, and the family of methods where you repeatedly sample completions from a policy model, score them, and update weights. During training, you want to generate as many tokens as possible per GPU-hour, because rollout generation is often the throughput bottleneck of the whole training loop. Frameworks like veRL and OpenRLHF wire SGLang in specifically for this. So “sglang rl” resolves to SGLang used inside an RL training loop. That is a training-time throughput problem. It is not the same problem as serving your production traffic — and conflating the two is the root of the cost confusion. Why the RL-rollout objective diverges from the serving objective The RL rollout loop cares about one thing: total tokens generated per unit of wall-clock time during training, aggregated across all the samples in a batch. Latency of any individual completion is close to irrelevant. Nobody is waiting on a rollout. You can drive concurrency as high as memory allows, let per-request latency balloon, and it only helps — more in-flight requests means fuller batches means higher aggregate throughput. Production serving inverts almost every one of those assumptions. A real product has a latency contract, usually expressed as a p95 or p99 bound. Requests arrive on their own schedule, not in a tidy training batch. And the metric that decides whether the feature makes or loses money is cost-per-request (or cost-per-token) at that latency bound — not throughput in the abstract. Here is the trap. When you raise concurrency in SGLang, aggregate throughput usually goes up, which is exactly what the rollout use case wants. But tail latency goes up too, because more requests are queued and batched together, and each one waits longer for its turn on the decode step. Past a certain concurrency, you are still generating more tokens per second in total while every individual user is waiting past your p95. The config that “won” on rollout throughput can be underwater on per-request margin the moment a latency constraint is applied. The number only meant something because it was measured in the wrong units for the decision you are making. This is the same category of error we describe when spec-ing the compute behind a production AI feature on cost-per-request: a throughput figure with no latency bound and no cost denominator is not a serving metric. It is a training metric wearing a serving metric’s clothes. Which SGLang features move cost-per-request, and which only move throughput? Not every SGLang feature behaves the same way across the two objectives. Some genuinely reduce cost-per-request at a fixed latency; some only inflate aggregate throughput in a way that helps rollouts and can hurt serving. Knowing which is which is most of the skill. SGLang feature Effect on rollout throughput Effect on cost-per-request at fixed p95 How to read it RadixAttention (prefix cache) Higher when prompts share prefixes Lower cost when your traffic has shared system prompts / few-shot templates; neutral when prompts are unique Real serving win if your workload has prefix reuse — measure your actual prompt mix Continuous batching Higher Lower cost up to the latency knee, then the tail crosses p95 and effective cost rises Beneficial but bounded; the knee is workload-specific Aggressive scheduling / high max concurrency Higher, monotonically Helps until queueing pushes p95 past the bound, then hurts Rollout-favouring knob; must be capped by the latency contract in serving Prefill/decode disaggregation Modest for pure rollout Can lower cost when prefill and decode have very different resource profiles See the prefill/decode split walkthrough for when it pays Quantization (FP8 / INT8 weights) Higher Lower cost per token, with an accuracy trade-off to validate on your task Real win if quality holds; a quality regression is a hidden cost (Evidence class: observed-pattern — directional behaviour we see across serving-configuration engagements, not a single named benchmark. The direction of each effect is stable; the magnitude and the location of the latency knee are workload-specific and must be measured on your traffic.) The pattern to internalise: features that raise aggregate throughput without regard to tail latency (uncapped concurrency, maximally aggressive scheduling) are rollout-favouring. Features that reduce work per request (prefix caching on reused prompts, quantization) or reshape resource use (prefill/decode disaggregation) are the ones with a genuine cost-per-request story — and even those only pay off when your actual workload matches the assumption. How do you fix p95 latency so the benchmark maps to margin? The single most important discipline is this: pin the p95 latency target before you compare configs, and report cost-per-request at that fixed bound. Without it, every comparison is confounded, because different configs trade latency for throughput at different rates. Concretely: State the latency contract as a number. For example, p95 end-to-end latency ≤ 2.5s for a 400-token completion. This comes from the product, not from the benchmark. Sweep concurrency, not just one setting. For each candidate config, run a load sweep from low to high concurrency and record throughput and p95 latency at each level. Find the concurrency at which each config just meets the p95 bound. That is the config’s operating point. Beyond it, the config violates the contract; below it, you are leaving throughput on the table. Compute cost-per-request at that operating point. Cost is GPU-hour price divided by requests-per-hour achieved at the compliant concurrency. This is the number that maps to margin. A config that reaches 20k tokens/sec at concurrency 256 but blows past p95 there, and only sustains p95 at concurrency 96, is a concurrency-96 config for your purposes. Benchmark it there. The per-config utilisation and latency measurements this needs come from the same GPU profiling discipline covered in GPU profiling for cost-per-request-style instrumentation — you cannot compute a defensible cost denominator without knowing what the GPU was actually doing. What does a worked cost-per-request comparison between two SGLang configs look like? Assume a single H100 node at an illustrative on-demand price of $2.50/GPU-hour (substitute your real contracted rate). Two SGLang configurations of the same model, same 400-token average completion, latency contract p95 ≤ 2.5s. Metric Config A (rollout-tuned: uncapped concurrency) Config B (serving-tuned: concurrency capped at p95) Compliant concurrency (meets p95 ≤ 2.5s) 96 128 (prefix cache + FP8 let it hold p95 higher) Aggregate throughput at max concurrency ~22,000 tok/s at concurrency 256 ~18,000 tok/s at concurrency 128 Throughput at the compliant operating point ~11,500 tok/s ~16,200 tok/s Requests/hour at operating point ~103,500 ~145,800 Cost-per-request ~$0.0000242 ~$0.0000171 Gross-margin delta vs Config A baseline ~29% lower cost-per-request (Illustrative worked example with explicit assumptions, not a benchmark result. GPU price, model, and completion length are stated so you can re-run the arithmetic against your own numbers.) Read the table carefully. Config A wins the headline: 22,000 tok/s beats 18,000 tok/s, and if you were choosing a rollout backend you would pick A. But A only hits that number at a concurrency that violates the serving latency contract. At the compliant operating point, B is roughly 29% cheaper per request — because prefix caching and FP8 quantization let it hold p95 at higher concurrency, so it serves more compliant requests per GPU-hour. If you had carried A’s rollout headline into the serving decision, you would have shipped the more expensive config and never seen the gap. How do you turn SGLang benchmark results into a defensible config-selection decision? A config selection is defensible when a finance-literate reviewer can trace the choice from the latency contract to a cost-per-request number without any hand-waving. That means three things travel together in the decision: the p95 bound the config was measured against, the compliant operating point, and the cost-per-request at that point. A tokens-per-second figure with none of those attached is a training artifact, and it should not appear in a serving decision at all. The same rigour that keeps serving configs comparable on a level field applies here: hold the model, the completion-length distribution, the prompt mix, and the hardware fixed across configs, vary only the SGLang knobs, and report the cost-per-request delta. When those controls are in place, the config choice is no longer a matter of taste — it is arithmetic. Running this end-to-end against your deployed serving path is exactly what our [inference cost-cut engagement](Inference Cost-Cut Pack) does: it takes your current SGLang config as the baseline, sweeps candidate configs at your real p95, and hands back before/after cost-per-request numbers rather than a throughput headline. Teams building on this stack — the audience we work with across AI-infrastructure SaaS — usually find the gap sits in exactly the place this article describes: a config chosen on rollout or peak-throughput logic that was never re-measured under the serving contract. FAQ How does sglang rl work? “sglang rl” refers to using SGLang as a rollout backend inside a reinforcement-learning post-training loop (RLHF, GRPO, and similar), typically wired in through frameworks like veRL or OpenRLHF. In that role SGLang repeatedly samples completions from the policy model as fast as possible so the training loop is not bottlenecked on generation. In practice it means SGLang is being optimised for aggregate generation throughput during training — a different job from answering production requests. What is the difference between SGLang as an RL rollout backend and as a production inference server, and why does it change what you benchmark? As a rollout backend, SGLang optimises total tokens-per-second during training, where individual-request latency does not matter. As a production server, it must meet a p95 or p99 latency contract and the metric that decides profitability is cost-per-request at that bound. It changes what you benchmark because a config that maximises rollout throughput often violates the serving latency contract, so the throughput headline does not map to serving margin. Which SGLang serving features actually move cost-per-request, and which only move raw throughput? Features that reduce work per request or reshape resource use — RadixAttention prefix caching on reused prompts, FP8/INT8 quantization, and prefill/decode disaggregation — have a genuine cost-per-request story when your workload matches their assumptions. Uncapped concurrency and maximally aggressive scheduling mostly inflate aggregate throughput and can push tail latency past your p95, so they favour rollouts but not serving margin. The distinction is whether the feature helps at a fixed latency bound or only in the unbounded aggregate. How do you fix p95 latency when comparing SGLang configs so the benchmark maps to margin? Pin the p95 latency target as a number from the product, then sweep concurrency for each candidate config and record throughput and p95 at each level. For each config, find the concurrency at which it just meets the p95 bound — that is its operating point — and compute cost-per-request there. Comparing configs at their respective compliant operating points, rather than at their peak throughput, is what makes the number map to margin. How do batching and concurrency settings in SGLang change cost-per-token at a fixed latency target? Raising concurrency and batch aggressiveness increases aggregate throughput, which lowers cost-per-token up to a point. Past a workload-specific “knee,” queueing pushes tail latency beyond the fixed target, so the effective cost-per-token at the compliant operating point rises again. The optimum is the highest concurrency that still holds p95, not the highest concurrency the hardware can run. How do you turn SGLang benchmark results into a defensible production config-selection decision? Make three facts travel together with every candidate: the p95 bound it was measured against, its compliant operating point, and its cost-per-request at that point. Hold model, prompt mix, completion length, and hardware fixed across configs and vary only the SGLang knobs, then report the cost-per-request delta. A defensible decision is one a finance-literate reviewer can trace from latency contract to cost-per-request without hand-waving. The open question for most teams is not which SGLang feature is fastest — it is whether their current config was ever re-measured under the serving contract it actually operates against, or whether it inherited a number from a rollout benchmark that answered a different question entirely.
