The moment a team decides to move gpt-oss off a metered API and onto its own SGLang serving stack, the first question everyone asks is the wrong one: “does it run?” It runs. That was never the hard part. SGLang loads the open-weight gpt-oss checkpoints, exposes an OpenAI-compatible endpoint, and starts returning tokens within an afternoon. The demo works, the tokens-per-second number looks healthy on a synthetic batch, and someone declares the migration a success. Then the first month’s GPU bill arrives, and the cost-per-request is higher than the API you left. This is the trap we see most often when teams pair SGLang — an open-source LLM serving runtime — with gpt-oss to escape per-token pricing. The serving path runs correctly and still loses on unit economics, because the number that decided the migration (peak throughput on a full batch) is not the number that governs the bill (cost per request at your real traffic shape). Self-hosting only wins when the serving path is tuned to your workload’s economics, not to a benchmark’s. How does SGLang serve gpt-oss, and what does it mean in practice? SGLang is a runtime built specifically for high-throughput LLM inference. It handles the request queue, packs concurrent requests into GPU-efficient batches, manages the KV cache, and streams tokens back. gpt-oss is an open-weight model family, so unlike a hosted API you are now responsible for the hardware it sits on and every scheduling decision the runtime makes on your behalf. In practice, “SGLang serves gpt-oss” means you have taken on the serving layer as an engineering surface. That is the whole point of leaving the API — you wanted control over cost. But control cuts both ways. The API vendor amortised idle time, batching inefficiency, and headroom across thousands of tenants and charged you a clean per-token rate. Once you self-host, every idle GPU-second and every half-full batch is your cost, and it lands in cost-per-request whether or not you measured it. The mechanism that makes this economically viable is SGLang’s scheduler. Two features do most of the work, and both are worth understanding before you trust any cost projection: RadixAttention prefix caching — SGLang stores KV-cache state for shared prompt prefixes in a radix tree, so requests that share a system prompt or few-shot preamble reuse that computation instead of recomputing it. For workloads with a heavy shared prefix (a fixed system prompt, a RAG template, an agent scaffold), the prefill cost that would otherwise dominate short requests collapses. Continuous batching — Rather than waiting to assemble a fixed batch, SGLang admits new requests into the running batch as slots free up. This keeps the GPU busy across requests of uneven length instead of stalling on the longest one in a static batch. Neither of these is a switch you flip once. Their benefit is entirely a function of what your traffic actually looks like, which is why a benchmark number rarely transfers. Which SGLang features move cost-per-request for gpt-oss? The honest answer: the features help exactly as much as your workload lets them, and no more. That sounds evasive until you look at where the cost actually comes from. Cost-per-request for a self-hosted deployment is, at its core, the GPU-hour cost of the instance divided by the number of requests it serves in that hour. Everything RadixAttention and continuous batching do is aimed at increasing the denominator — more requests served per GPU-hour — without pushing latency past your threshold. So the levers that matter are the ones that raise sustained useful throughput at your target latency, not peak throughput in isolation. RadixAttention is the clearest example. If your requests share a long prefix, cache-hit prefill is nearly free and cost-per-request drops sharply. If every request carries a unique prompt with no shared structure, the radix tree has nothing to reuse and the feature contributes little. We have seen the same SGLang configuration produce very different economics on two workloads that looked identical on a spec sheet — the difference was prefix-cache hit rate, which is a property of the traffic, not the runtime (observed across our serving engagements; not a published benchmark). This is the same reasoning covered in the companion piece on how serving-runtime choices shape your eval numbers for SGLang and gpt-oss — the runtime does not have a single “cost,” it has a cost function over your traffic. For deployments where prefill dominates and decode is comparatively light — or the reverse — separating the two stages onto differently-tuned resources can help further; the mechanics of that are worked through in how prefill/decode disaggregation cuts cost-per-request in SGLang. When does self-hosting gpt-oss on SGLang beat per-token API pricing? This is a crossover question, not a yes/no one. A metered API charges you strictly for tokens consumed, with zero cost when idle. Self-hosting charges you for GPU-hours whether you use them or not. So the comparison is a break-even on volume and utilisation, and it has a specific shape. Below some traffic threshold, the API almost always wins, because you cannot keep the GPU busy enough to amortise its rental cost. Above it, self-hosting wins by a widening margin, because the marginal cost of another request on an already-rented, well-batched GPU approaches zero. The crossover point is the number that should decide the migration — and it moves depending on your prefix-cache hit rate, your latency headroom, and how efficiently your concurrency settings keep the batch full. Decision surface: API versus self-hosted SGLang + gpt-oss Condition Leans toward metered API Leans toward self-hosted SGLang Sustained request volume Low or spiky, long idle windows High and steady enough to keep the GPU busy Prefix structure Mostly unique prompts, low cache reuse Heavy shared prefix (system prompt, RAG, agent scaffold) → high RadixAttention hit rate Latency headroom Tight p95 SLO, little room to batch Room to trade a few ms of p95 for fuller batches Traffic predictability Bursty, hard to provision for Predictable, so instances can run near target utilisation Operational appetite No platform team to own the serving stack ML-Platform team ready to own tuning and monitoring The mistake is deciding this on a single tokens-per-second benchmark. A configuration tuned for peak throughput on a synthetic full batch can still lose to the API on cost-per-request once idle GPU-time and p95 headroom are priced in. The LLM token calculator for estimating inference cost per request is a useful first pass on the token-cost side, but it only tells half the story — the other half is utilisation, and utilisation is a serving-config property. If you are sizing the compute behind this, the reasoning in spec-ing the compute behind a production AI feature walks through the same cost-per-request anchoring for a feature-level decision. The team profiling this crossover for you is the kind of work our [inference cost-cut engagement](Inference Cost-Cut Pack) is built around, and it maps directly onto the concerns of an AI-infrastructure SaaS platform trying to move off API margins. How do concurrency and batching settings change GPU utilisation and cost? Concurrency is the single most misread setting in SGLang deployments. Turning up the maximum number of concurrent requests raises throughput — until it doesn’t. Past the point where the KV cache saturates GPU memory, additional concurrency forces the scheduler to preempt or queue, and tail latency climbs while throughput plateaus. You paid for the higher concurrency in latency and got nothing in cost-per-request. The mechanism is memory, not compute. Each in-flight request holds KV-cache state proportional to its context length. gpt-oss at a given precision has a fixed model footprint; whatever HBM is left over is the budget for concurrent KV cache. Continuous batching keeps that budget working efficiently, but it cannot manufacture memory that isn’t there. So the right concurrency setting is the highest one that keeps p95 latency under your SLO with the cache comfortably within HBM — and that number is workload-specific. There is a broader point here about where cost hides in a serving stack. The GPU profiling that underwrites this per-request math — measuring where GPU-time actually goes rather than assuming it — is genuine engineering work, not a spec-sheet exercise. Getting that measurement right is what separates a real cost projection from a hopeful one. What p95 latency trade-offs come with tuning SGLang for throughput? Every throughput gain from batching is bought with latency variance. When SGLang packs more requests into a running batch, individual requests wait behind others in the decode loop, and the tail of the latency distribution stretches. Median latency can look fine while p95 quietly doubles. This matters because cost-per-request and p95 latency are coupled, not independent. You cannot report a low cost-per-request without stating the p95 it was measured at, because you can always push cost down by cramming the batch fuller and accepting worse tails. A cost number without a latency bound is meaningless — it is the serving equivalent of quoting a price without the product. The useful figure is cost-per-request at the p95 latency your product can tolerate, and the tuning job is finding the configuration that minimises the former subject to the latter. How do we measure cost-per-request for a self-hosted SGLang + gpt-oss deployment? Measure it on your real traffic, not a synthetic batch, and price in the parts that a throughput benchmark ignores. The metrics that actually decide the number are few: GPU-hour cost at target concurrency — the rental cost of the instance running at the concurrency you will actually operate at, not idle and not maxed. Requests served per GPU-hour — the denominator; this is what batching and caching are trying to raise. Prefix-cache hit rate — for RadixAttention workloads, this often dominates the economics; a low hit rate means you are recomputing prefill you thought you were caching. p95 latency at the operating point — the constraint the cost number must respect. Run those against a traffic replay that matches your real request-length distribution and prefix-sharing pattern. Synthetic full-batch benchmarks systematically flatter self-hosting, because they assume the GPU is always busy and every batch is full — the two conditions least likely to hold in production. The SGLang DeepSeek-V3 cost-per-request walkthrough shows this measurement discipline applied end-to-end for a different model, and the method transfers directly to gpt-oss. FAQ How does SGLang serve gpt-oss, and what does it mean in practice? SGLang is an open-source LLM serving runtime that loads gpt-oss’s open weights, exposes an OpenAI-compatible endpoint, and manages the request queue, batching, and KV cache. In practice it means you have taken ownership of the serving layer, so every idle GPU-second and half-full batch now lands in your cost-per-request rather than being amortised by an API vendor across many tenants. What features of SGLang (RadixAttention, continuous batching) affect cost-per-request for gpt-oss? RadixAttention caches shared prompt prefixes in a radix tree so requests reusing a system prompt or RAG template skip redundant prefill, and continuous batching admits new requests into the running batch as slots free instead of waiting for a static batch. Both raise the number of requests served per GPU-hour, but only as much as your traffic’s prefix-sharing and length distribution allow — a benchmark number rarely transfers. When does self-hosting gpt-oss on SGLang beat per-token API pricing on cost-per-request? Above a crossover traffic volume where you can keep the GPU busy enough to amortise its rental cost, and especially when your workload has a heavy shared prefix that lifts RadixAttention hit rate. Below that threshold, or with mostly unique prompts and tight p95 SLOs, a metered API usually wins because it charges nothing when idle while a self-hosted GPU is billed whether or not it is working. How do concurrency and batching settings change GPU utilisation and cost-per-request? Raising concurrency lifts throughput until the KV cache saturates HBM, after which the scheduler queues or preempts, tail latency climbs, and cost-per-request stops improving. The right setting is the highest concurrency that keeps p95 under your SLO with the cache within GPU memory — a workload-specific number, since KV-cache demand scales with context length. What p95 latency trade-offs come with tuning SGLang for throughput? Every throughput gain from fuller batches stretches the latency tail, so p95 can double while median stays flat. Cost-per-request and p95 are coupled — you can always cut cost by cramming the batch and accepting worse tails — so the only meaningful figure is cost-per-request measured at the p95 your product can tolerate. How do we measure cost-per-request for a self-hosted SGLang + gpt-oss deployment? Measure on a traffic replay matching your real request-length and prefix-sharing patterns, tracking GPU-hour cost at target concurrency, requests served per GPU-hour, prefix-cache hit rate, and p95 latency at the operating point. Synthetic full-batch benchmarks flatter self-hosting because they assume a always-busy, always-full GPU — the two conditions least likely to hold in production. Where the real risk actually sits The interesting question is never “does SGLang serve gpt-oss?” — it does. It is: at what traffic volume, prefix-sharing pattern, and p95 bound does your self-hosted serving path actually undercut the API you are trying to leave? That crossover is a property of your workload’s unit economics, and it is measurable before you commit a single GPU-month. Getting it wrong is the failure class we call migrating for control and paying for it in cost-per-request — and it is exactly what the profiling in our [inference cost-cut engagement](Inference Cost-Cut Pack) exists to catch before the bill does.
