A headline of “38,000 tokens per second” tells you the runtime is fast on the benchmark someone else ran. It tells you almost nothing about where your p95 latency goes, or which stage of SGLang’s serving path is quietly eating your GPU budget. That gap — between an advertised throughput number and an attributable one — is the whole reason profiling the serving runtime matters, and it is where most SGLang adoptions stop too early. SGLang’s OME is the serving/model-execution engine: the layer that takes incoming requests, decides how to batch them, manages the prefix cache, and drives the prefill and decode kernels on the GPU. When a team adopts it, the naive move is to trust the framework’s advertised throughput and assume the serving path is now optimal. It usually is not — not because SGLang is slow, but because its defaults were tuned for a workload shape that is not yours. How does SGLang OME actually work? Think of OME as three coupled decisions running continuously, not one throughput dial. First, scheduling: which pending requests get admitted into the next forward pass, and in what order. Second, continuous batching: instead of waiting to assemble a fixed batch, OME injects new sequences into the running batch as older ones finish decoding, so the GPU rarely sits idle between requests. Third, prefix caching via a radix tree: shared prompt prefixes (system prompts, few-shot examples, common instruction templates) are stored once and reused across requests, so their key-value cache does not get recomputed. Each of these is a real optimisation. Each also moves the bottleneck rather than removing it. Continuous batching keeps the GPU busy — but it turns queue admission into a latency variable, because a request’s wait depends on what else is in flight. Prefix caching cuts prefill cost dramatically when prompts overlap — but on a workload with unique prompts, the radix cache hit rate collapses and you pay full prefill every time. The runtime does not tell you which regime you are in. You have to measure it. That is the practical meaning of “how OME works”: it is a set of policies whose payoff is entirely workload-dependent. The same configuration that delivers a great benchmark on ShareGPT-style traffic can produce ugly p95 tails on long-context RAG traffic, because the two workloads stress the scheduler and the cache in opposite ways. What OME does differently from a naive model server A naive model server processes requests more or less one batch at a time: collect a batch, run prefill, run decode to completion, return, repeat. It is simple and it wastes the GPU. Two things dominate the waste — the decode phase runs at tiny batch sizes as sequences finish at different times, and every request recomputes its prompt from scratch. OME attacks both. Continuous batching (the technique vLLM popularised and SGLang implements with its own scheduler) keeps decode batches full by admitting new sequences mid-flight. RadixAttention keeps prefix key-value state resident so overlapping prompts skip prefill work. The result on a favourable workload is a large throughput gain — but the gain is not free of latency structure. It is redistributed. Understanding where it lands is the difference between “SGLang is fast” and a claim you can act on, like “22% of p95 is prefill queuing under continuous batching.” If you want the upstream framing of how a request travels through admission, batching, prefill, and decode, our walkthrough on mapping the serving path in a machine learning architecture diagram sets up the stages this article profiles. How do OME’s decisions show up in a prefill-vs-decode breakdown? This is where profiling earns its keep. A tokens-per-second headline is an aggregate; a per-stage latency breakdown is diagnostic. When you instrument the OME path and split latency by phase, the runtime’s decisions become visible as distinct signatures. The table below maps each OME behaviour to the profiler signal it produces and the number that tells you whether it is helping or hurting. OME behaviour Where it shows up Metric to read Healthy signal Continuous batching Prefill queue wait, batch occupancy Prefill p95 vs p50 spread; mean batch occupancy Occupancy high, p95/p50 spread modest Prefix / radix caching Prefill compute time Prefix-cache hit rate High hit rate on repetitive prompts Scheduling / admission Time-to-first-token variance TTFT p95, queue-admission delay Low variance under steady load Decode kernel execution Decode phase per-token time Decode p50/p95, inter-token latency Flat, low inter-token latency Scheduling gaps GPU utilisation between passes GPU idle-gap percentage Small idle gaps at target load The signatures are not subtle once you look. If prefill p95 is far above p50 while decode is flat, you have a queue-admission problem, not a model problem — the scheduler is the bottleneck. If prefill compute is high and roughly constant per request, your prefix-cache hit rate is low and RadixAttention is not earning its place on this traffic. If decode inter-token latency is jittery, the issue is kernel-level or memory-bandwidth-bound, which is a GPU-profiling question rather than a scheduler one. These are observed-pattern signatures we see repeatedly when auditing serving runtimes; they are diagnostic starting points, not universal thresholds. What metrics tell me whether OME is well-tuned for my workload? Four numbers do most of the work. None of them appear in the marketing throughput figure, and all of them are cheap to capture once the runtime is instrumented. Prefix-cache hit rate — the fraction of prompt tokens served from the radix cache rather than recomputed. High on templated or few-shot traffic; near zero on unique-prompt traffic. If it is low and your prompts should overlap, the cache is misconfigured or being evicted under memory pressure. Continuous-batching occupancy — the average number of active sequences per forward pass relative to the maximum the GPU can hold. Low occupancy means the GPU is under-fed and you are leaving throughput on the table; near-saturation with rising p95 means you are over-admitting and the queue is the tail. Prefill-vs-decode p50/p95 split — the two phases have different cost structures (prefill is compute-heavy and bursty; decode is memory-bandwidth-bound and steady). Attributing latency to the right phase tells you which lever to pull. GPU idle gaps introduced by scheduling — the percentage of GPU time lost between forward passes because the scheduler had nothing ready to run. Small gaps are normal; large ones point at admission logic or a data-loading stall. These four feed the audit baseline directly. Prefix-cache hit rate and batch occupancy tell you whether a configuration change captures the win; the prefill/decode split and idle gaps tell you whether the ceiling is the runtime or the hardware. When the idle-gap and decode-jitter signals dominate, the question crosses from scheduling into kernel and memory behaviour, which is measured with the same GPU profiling methodology applied one layer down. When does an OME config change capture the win versus needing a model swap? This is the decision the numbers exist to inform. A model swap — to a smaller model, a quantised variant, or a routed mix — is expensive: it touches evaluation, quality gates, and often the whole test suite. A runtime configuration change (cache sizing, max batch tokens, chunked-prefill settings, scheduling policy) is cheap and reversible. The audit sequence is to exhaust the cheap lever first. The rule of thumb from profiling serving runtimes: if your prefix-cache hit rate is low but addressable, if batch occupancy is well below saturation, or if idle gaps are large, the win is still inside OME configuration. You have not yet reached the runtime’s ceiling on this hardware. Only when occupancy is near saturation, the cache is hitting as well as the workload allows, idle gaps are small, and p95 is still over budget have you actually hit the wall — and then a model swap or a hardware change is the honest next move. This ordering is why we treat the runtime as a layer to profile rather than a solved problem. Getting the attribution right before spending on a model change is what our [inference cost audit](Inference Cost-Cut Pack) is built around, and SGLang OME is one of the serving runtimes it profiles stage by stage. The techniques compound with runtime-level levers covered in SGLang PD disaggregation and prefill/decode splitting and SGLang speculative decoding, both of which change the same numbers this article teaches you to read. Why the headline tokens-per-second number hides p95 tail problems Tokens-per-second is a throughput average taken at a saturating load, and averages are precisely the wrong tool for tail latency. Continuous batching optimises for aggregate throughput by keeping the GPU full — which is exactly the condition under which individual requests can wait longer in the admission queue. A runtime can post a superb tokens-per-second figure because it defers latency-sensitive requests behind a full batch. The mean improves; the p95 degrades. The benchmark celebrates the first and never reports the second. That is the structural reason to distrust a headline number for a latency-SLO workload. It is measuring a quantity you may not be optimising for. For a walkthrough of the same distinction on a concrete model, our note on what to profile when serving DeepSeek with SGLang shows the throughput-vs-tail gap on real traffic. FAQ How should you think about sglang ome in practice? SGLang’s OME is the serving engine that turns incoming requests into GPU work through three coupled decisions: scheduling (which requests to admit), continuous batching (injecting new sequences into a running batch to keep the GPU fed), and prefix caching (reusing shared prompt key-value state via a radix tree). In practice it means the runtime is a set of workload-dependent policies, not a fixed throughput dial — the same config can be excellent on one traffic shape and produce ugly tails on another. What does SGLang’s OME serving engine actually do differently from a naive model server? A naive server processes roughly one batch at a time, wasting GPU during decode and recomputing every prompt. OME uses continuous batching to keep decode batches full and RadixAttention to skip prefill on overlapping prompts. The gain is real but redistributed rather than eliminated — continuous batching turns queue admission into a latency variable, and prefix caching only pays off when prompts actually overlap. How do SGLang OME’s decisions show up in a profiler’s prefill-vs-decode latency breakdown? Each behaviour has a signature. A large prefill p95-vs-p50 spread with flat decode points to queue admission, not the model. High constant prefill compute means a low prefix-cache hit rate. Jittery decode inter-token latency indicates a kernel or memory-bandwidth issue rather than a scheduler one. Splitting latency by phase is what makes these attributable. What metrics tell me whether SGLang OME is well-tuned for my workload? Four: prefix-cache hit rate, continuous-batching occupancy, the prefill-vs-decode p50/p95 split, and GPU idle gaps introduced by scheduling. None appear in the advertised tokens-per-second figure. Together they tell you whether the GPU is being fed, whether the cache is earning its place, which phase owns the latency, and how much time the scheduler is wasting between passes. When does a SGLang OME configuration change capture the win versus needing a model swap? Exhaust the cheap lever first. If the prefix-cache hit rate is low but addressable, occupancy is below saturation, or idle gaps are large, the win is still inside OME configuration and you have not hit the runtime’s ceiling. Only when occupancy is near saturation, the cache hits as well as the workload allows, idle gaps are small, and p95 is still over budget does a model swap or hardware change become the honest next step. Why does SGLang’s headline tokens-per-second number hide p95 tail problems in the serving path? Tokens-per-second is a throughput average measured at saturating load, and continuous batching improves that average partly by deferring latency-sensitive requests behind a full batch. The mean rises while the p95 degrades — and the benchmark reports the first, not the second. For a latency-SLO workload it is measuring a quantity you may not be optimising for. How do I attribute cost-per-request to SGLang OME’s scheduling stages to see which step drives the bill? Split GPU time by OME stage — queue admission, prefill, decode, and scheduling idle gaps — then multiply each stage’s GPU-seconds by your instance cost to get per-stage cost-per-request. The stage that dominates GPU-seconds dominates the bill, and that is where a configuration change delivers the largest, cheapest saving before any model change is considered. What this means for the next engagement The runtime is not a black box that is either “fast” or “slow.” It is a stack of scheduling, batching, and caching decisions, each of which moves latency and cost somewhere specific. The honest question a serving audit answers is not “is SGLang fast?” — it plainly can be — but “which OME stage owns my p95, and is the cheapest lever a config change or have I actually reached the runtime’s ceiling on this hardware?” Answer that with numbers before you touch the model, and you will spend on the change that actually captures the win. That is the failure class the [inference cost audit](Inference Cost-Cut Pack) exists to prevent: paying for a model swap when the bottleneck was a cache you never measured.
