A latency figure on a vendor page is not a property of gpt-oss. It is a property of gpt-oss served under one specific runtime, on one specific hardware configuration, with one specific batching policy. Read it as “the model’s number” and you are already comparing apples to oranges — because the same open weights, moved from one serving stack to another, can shift throughput, tokens-per-second, and cost-per-request by multiples. This matters the moment a procurement committee asks a question every committee eventually asks: “under what stack was this measured?” If the eval never pinned the serving runtime, there is no honest answer, and the whole exercise runs a second time. So before anyone signs off on a cost threshold for an open-weight model, it is worth understanding what a runtime like SGLang actually does when it serves gpt-oss, and why that changes the operational numbers you are about to defend. This article does not re-rank models. It makes the serving assumptions behind the numbers explicit. What does “sglang gpt-oss” actually mean in practice? The phrase looks like it names one thing. It names two, glued together by a decision. “gpt-oss” is a set of open weights — a model artifact you can download and run yourself. “SGLang” is a serving runtime: the software that loads those weights, holds them on the GPU, batches incoming requests, runs the attention and feed-forward math, and streams tokens back. The model is what you evaluate for quality. The runtime is what turns the model into a service with a latency, a throughput ceiling, and a cost. The naive mental model treats the two as inseparable — “gpt-oss is fast” or “gpt-oss is cheap.” Neither statement is complete. gpt-oss is fast when served a particular way, under a particular load. Change the runtime, the batching policy, or the quantization, and the operational metrics move, sometimes dramatically, while the weights on disk stay byte-for-byte identical. SGLang is one of several runtimes you might pick for that job (vLLM and TensorRT-LLM are the other two you will see most often). It matters here because it makes a few specific engineering choices — prefix caching, continuous batching, and constrained decoding — that map directly onto the operational side of a defensible metric set. Understanding those choices is what separates a reproducible cost figure from a number nobody can stand behind. What SGLang does when it serves an open-weight model Three mechanisms carry most of the operational weight. None of them changes what gpt-oss says; all of them change what it costs to say it. RadixAttention prefix caching. Transformer inference computes a key-value (KV) cache for every token in the prompt. When many requests share a common prefix — a long system prompt, a few-shot template, a fixed instruction block, a shared document in a retrieval-augmented pipeline — recomputing that prefix for every request is wasted work. SGLang organises cached prefixes in a radix tree so that overlapping prompts reuse the already-computed KV cache instead of recomputing it. For workloads with heavy prefix reuse, this is where the large throughput gains come from. Continuous batching. Static batching waits for a batch to fill, runs it to completion, then starts the next — which means fast requests wait for slow ones and the GPU idles between batches. Continuous batching (also called in-flight batching) lets new requests join and finished requests leave the batch at the token level, keeping the GPU saturated. This is the mechanism that turns raw hardware FLOPs into sustained tokens-per-second under realistic concurrent load rather than in a single-stream micro-benchmark. Structured / constrained output. When your application needs valid JSON, a specific schema, or a constrained grammar, SGLang can enforce that at decode time. This changes both the correctness profile and, sometimes, the speed — constrained decoding can skip tokens that are grammatically forced, but it also adds bookkeeping. Either way, whether constrained decoding was on is part of the run condition your eval needs to record. These are engineering choices, not magic. The reason to name them is that each one has a workload shape where it helps a lot and a workload shape where it does almost nothing — and a metric with no runtime attached hides which case you were actually in. Why the serving runtime moves the numbers a procurement eval reports Here is the core claim, stated plainly so it can be quoted on its own: the same gpt-oss weights served under different runtimes, or under the same runtime with different batching and caching settings, can differ by a large factor on throughput and cost-per-request, so an eval that does not pin the serving configuration is not measuring the model — it is measuring an unlabelled stack. Consider what feeds a cost-per-request number. It is roughly the cost of the GPU-hours consumed divided by the requests served in that time. Throughput is the denominator. Anything that raises sustained throughput at a fixed hardware cost lowers cost-per-request. Prefix caching raises it when prefixes repeat. Continuous batching raises it when concurrency is high. Quantization (say serving in a lower precision) raises it by fitting more work on the same silicon — at a quality cost you have to measure separately. None of these live in the model weights; all of them live in the serving decision. So two teams can benchmark the identical gpt-oss checkpoint, report cost-per-request figures that differ by a multiple, and both be telling the truth — about different serving stacks. This is exactly the divergence that a benchmark methodology has to control for. It is the same discipline we describe for laying out serving configs so comparisons are fair in structuring serving configs so cost-per-request comparisons are fair: fix the stack, or the number floats free of any decision it could inform. A companion walkthrough on serving gpt-oss with SGLang and what it means for cost-per-request carries the cost arithmetic further; this article stays on the why — why the runtime choice is a first-class variable in the eval, not an implementation detail to be filled in later. When does prefix caching actually help — and when is the gain negligible? RadixAttention is often quoted as a headline win, but its benefit is entirely a function of workload shape. Treat it as a conditional, not a constant. Workload shape Prefix reuse Expected caching benefit Long shared system prompt, short user turns (chatbot, assistant) High Large — the expensive prefix is computed once and reused across many requests Few-shot templates with a fixed instruction block High Large — the template prefix dominates and repeats RAG with a shared retrieved document across a session Moderate–high Meaningful when the same context recurs; drops as contexts diverge Unique long documents, one per request (summarization of distinct files) Low Negligible — nothing to reuse; each prefix is computed fresh Short, diverse prompts with no common prefix Very low Negligible — the cache lookup overhead can even cost a little The lesson is not “prefix caching is good.” It is that whether it helps your deployment depends on how much of your prompt volume is repeated structure. This is observed-pattern reasoning drawn from how these runtimes behave across workloads, not a single benchmarked ratio you should transplant onto your own numbers. If your eval workload does not resemble your production prompt distribution, the caching benefit you measure will not be the benefit you get — which is a broader trap in evaluation design, and one reason choosing which model metrics actually decide a serving config matters as much as choosing the model. How to pin the SGLang configuration so the metrics are reproducible A metric is only decision-grade if someone else can reproduce it from what you wrote down. For a gpt-oss-on-SGLang run, the fields below are the minimum you need to record for the operational numbers to mean anything to a committee six weeks later. Model artifact — exact gpt-oss checkpoint and revision, and the serving precision (the on-disk weights plus whatever quantization the runtime applies). Runtime and version — SGLang release, so the batching and caching behaviour is a known quantity rather than “whatever was current.” Batching policy — continuous batching on/off, max batch size, and max concurrent requests during the run. Prefix caching state — RadixAttention enabled or disabled, since this alone can move throughput by a large factor on repeated-prefix workloads. Decoding constraints — whether structured/constrained output was active, plus decoding parameters (temperature, max tokens) that affect tokens generated per request. Hardware — GPU model, count, and interconnect, because cost-per-request is GPU-hours over requests and the hardware sets the numerator. Load profile — request concurrency and the prompt/output length distribution, because throughput under load is not the same as single-stream latency. If those fields live in one place and travel with the numbers, a second eval round becomes unnecessary — the committee’s “under what stack?” is already answered on the page. That is the operational discipline our [production AI monitoring harness](Production AI Monitoring Harness) exists to enforce: the validation pack records the serving runtime and configuration alongside the operational metrics, so a cost figure is always attached to the stack that produced it. A model that looked cheap on a leaderboard but was benchmarked on an unrepresentative stack is a re-procurement waiting to happen, and this is the cheapest place to prevent it. FAQ What does working with sglang gpt-oss involve in practice? “gpt-oss” is a set of open weights; “SGLang” is the serving runtime that loads those weights and turns them into a service with a latency, throughput, and cost. In practice the phrase names a decision — which runtime serves the model — and that decision, not the weights alone, determines the operational numbers your eval reports. What does SGLang actually do when serving an open-weight model like gpt-oss? Three mechanisms carry most of the weight: RadixAttention prefix caching, which reuses the key-value cache for shared prompt prefixes instead of recomputing them; continuous batching, which keeps the GPU saturated by letting requests join and leave a batch at the token level; and structured/constrained output, which enforces valid JSON or a grammar at decode time. None change what the model says — all change what it costs to say it. Why does the serving runtime change the latency, throughput, and cost-per-request numbers a procurement eval reports? Cost-per-request is roughly GPU-hours divided by requests served, so anything that raises sustained throughput at fixed hardware cost lowers it — and prefix caching, continuous batching, and quantization all raise throughput without touching the model weights. The same gpt-oss checkpoint served under different runtimes can therefore report figures that differ by a multiple while both are truthful about different stacks. How do we pin the SGLang configuration so operational metrics are reproducible and comparable across candidate models? Record the exact checkpoint and serving precision, the SGLang version, the batching policy and concurrency, whether RadixAttention and constrained decoding were on, the hardware and interconnect, and the load profile. When those fields travel with the numbers, a second eval round becomes unnecessary because the “under what stack?” question is already answered. When does SGLang’s prefix caching materially help, and for which workload shapes is the gain negligible? It helps most when a large share of prompt volume is repeated structure — long shared system prompts, fixed few-shot templates, or a recurring retrieved document within a session. The gain is negligible when every request carries a unique long prompt or when short, diverse prompts share no common prefix, because there is nothing to reuse. How do we record the gpt-oss serving stack in the validation pack so the committee doesn’t ask for a second eval round? Capture the serving runtime and its full configuration in the same artifact as the operational metrics, so every latency and cost-per-request figure is attached to the stack that produced it. That way the committee’s inevitable “under what stack?” is answered on the page rather than by re-running the eval. The honest question to close on is not “which runtime is fastest” — it is which runtime, at which settings, matches the prompt distribution you will actually serve, because a caching win measured on the wrong workload shape is a number that will not survive contact with production. Pin the executor tuple — model checkpoint, runtime, batching, precision, hardware — before you read a single latency figure, and the figure becomes evidence instead of decoration.
