Open the source of a runtime called “mini sglang” and you get a clear reward: a serving loop small enough to read in an afternoon. The mistake is assuming that legibility equals efficiency — that a lean runtime is fast because it is small. It isn’t. A compact runtime obeys exactly the same inference constraints as a full serving stack, and reading its code tells you what it schedules, not where the time goes. Those are different questions, and only one of them is answered by measurement. That gap is worth naming up front, because it decides whether the next week of engineering pays off. Reading the code builds the mental model. Profiling tells you what to optimise. Confuse the two and you spend days tuning kernels that were never the bottleneck. How does mini sglang work in practice? The name “mini sglang” points at a minimal reimplementation of the SGLang serving model — the scheduler, the batching logic, and the KV-cache management that turn a stream of requests into GPU work. Strip away the production hardening (distributed serving, elaborate routing, the full API surface) and what remains is the core inference loop. That loop is the thing worth understanding, because it is shared, in spirit, with vLLM, TGI, and the full SGLang runtime. In practice, a lightweight runtime like this does four jobs, and every one of them shapes throughput: Prefill — it runs the prompt tokens through the model in one large, compute-heavy pass, producing the initial KV-cache entries. Decode — it generates output tokens one step at a time, each step reading the entire KV cache and doing comparatively little compute. Batching — it groups requests so the GPU does useful work per launch instead of processing one sequence at a time. KV cache — it stores and reuses the attention keys and values so decode steps don’t recompute the whole prompt. The claim that matters here: a compact serving runtime is governed by the same prefill/decode asymmetry that limits every autoregressive LLM server, and its small codebase does nothing to relax that constraint (observed pattern across the inference stacks we profile; not a benchmarked rate). Prefill is compute-bound. Decode is usually memory-bandwidth-bound. The runtime’s job is to keep the GPU busy across both phases despite that mismatch — and reading the code shows you how it tries, not whether it succeeds on your workload. What a lightweight SGLang runtime actually schedules — and why that shapes performance If you trace one request through the loop, the shape of the performance problem falls out on its own. A prompt arrives, the scheduler decides which requests to batch together, prefill runs, and then the request joins a decode group where it emits tokens until it hits a stop condition or length limit. The scheduler is continuously admitting new requests and evicting finished ones — continuous batching — so at any instant the GPU is processing a mix of sequences at different decode positions. This is where the naive reading breaks. Decode is dominated by reading the KV cache and the model weights from HBM, not by arithmetic. A single decode step for one sequence barely touches the GPU’s compute units; it moves a lot of memory and does very little math. So if the scheduler cannot assemble a wide enough decode batch — because requests are short, arrive unevenly, or the KV cache is fragmented — the GPU spends most of each step waiting on memory with idle compute. The kernels are efficient. The occupancy is not. The full SGLang project makes this concrete with techniques like RadixAttention for KV-cache reuse across shared prefixes and prefill/decode disaggregation that separates the two phases onto different resources. A mini runtime typically drops those optimisations to stay readable — which is exactly why it is a good teaching artifact and a poor production default. When you understand what it didn’t implement, you understand what the full stack is buying you. How does mini sglang differ from the full SGLang serving stack, and when is the minimal version enough? The honest answer is that “mini sglang” is a learning and prototyping runtime, not a serving target. Use the table below as a decision aid rather than a feature scorecard. Minimal runtime vs full serving stack — a decision table Dimension mini / lightweight runtime Full SGLang serving stack Codebase Small, readable in an afternoon Large, production-hardened Continuous batching Basic or simplified Mature, tuned scheduler KV-cache reuse Usually none RadixAttention prefix sharing Prefill/decode split Combined Optional PD disaggregation Multi-GPU / distributed Rarely Tensor + data parallel support Best for Learning the model, profiling experiments, single-GPU prototypes Production serving under real, uneven load The minimal version is enough when your goal is to understand the scheduling model, run controlled profiling experiments on a single GPU, or prototype a change before porting it into a heavier stack. It stops being enough the moment you have concurrent users with varied prompt lengths, shared prefixes worth caching, or a throughput target that a naive batcher can’t hit. For a broader look at how the mature stack behaves under load, our write-up on what profiling reveals about real SGLang and DeepSeek serving bottlenecks walks through the same measurement discipline on a production configuration, and the companion piece on nano SGLang and its GPU API implications covers the sibling minimal runtime. Why can reported GPU utilisation look high while throughput stays low? This is the trap that sends engineers down the wrong path. nvidia-smi reports a GPU utilisation figure that means “a kernel was resident on the device during the sampling window” — not “the compute units were saturated with useful work.” A decode step that is entirely memory-bound will still show high utilisation, because a kernel is running; it is simply stalled on HBM reads with the arithmetic units mostly idle. So a runtime can report 90-plus percent utilisation and still deliver a fraction of the tokens per second the GPU is capable of. The utilisation number is measuring presence, not efficiency. Treating the utilisation percentage as proof that the kernels are the limit is the single most common misread we see when a lean runtime disappoints on throughput (observed pattern; not a published benchmark). The number is high precisely because the GPU is busy waiting. This distinction is worth internalising beyond SGLang. We unpacked it more generally in the three pillars of observability applied to GPU utilisation — utilisation is a coverage signal, not a saturation signal, and the two diverge exactly when memory bandwidth is the constraint. How do I profile a mini sglang workload to find the real bottleneck? You stop reading the code and start measuring the trace. The runtime’s small size is an advantage here: there is less noise between your request and the kernels. The goal is to attribute time to one of four classes — compute, memory, host, or batching — and the tools for that are NVIDIA’s profilers. Profiling checklist — from trace to constraint Capture an end-to-end timeline with Nsight Systems. Look for gaps between kernel launches on the GPU stream. Contiguous kernels mean the GPU is fed; visible gaps mean it is starved by something upstream. Classify the gaps. If the gaps line up with CPU activity — tokenisation, scheduling, Python dispatch — the workload is host-bound. The fix is on the CPU side, not the kernel. Inspect the decode kernels with Nsight Compute. If the dominant kernels show high memory throughput and low compute throughput (a low arithmetic intensity, memory-bound roofline position), decode is memory-bound. Wider batches or KV-cache reuse help; kernel micro-tuning does not. Check batch occupancy. Log the effective batch size per decode step. If it is small and variable, the scheduler is the constraint — the GPU is under-fed because too few sequences are being processed together. Only then look at compute. If, and only if, the profiler shows the GPU compute-bound with a full batch, does kernel-level optimisation become the right lever. The order matters because the largest speedups come from targeting the actual constraint. In workloads like this, a meaningful share of decode steps often turn out to be batching- or host-bound rather than compute-bound (observed pattern; the exact fraction is workload-specific and only measurement on your trace will tell you). Improving batch occupancy on a batching-bound loop can move throughput far more than any kernel change — and blindly tuning kernels on a memory-bound decode loop moves nothing at all. What optimisation order gives the largest speedup once you’ve profiled? Fix the class the profiler names, in the order the profiler ranks them. If the trace is host-bound, the win is in reducing Python overhead, overlapping tokenisation, or moving scheduling off the critical path — CUDA changes are premature. If decode is memory-bound, the wins are structural: wider continuous batches, KV-cache reuse across shared prefixes (what RadixAttention does), or quantising the KV cache to move less memory per step. If the loop is genuinely batching-starved, the scheduler is the target. Kernel fusion, TensorRT, and low-level CUDA work belong last — they pay off only once the GPU is demonstrably compute-bound with a full batch. This ranked-remediation logic is exactly what a GPU performance audit produces on a real serving workload: it profiles your actual traffic pattern, classifies where the time goes, and returns an optimisation roadmap ordered by expected speedup rather than by whichever knob is easiest to reach. Reading mini sglang’s source is the fastest way to build the mental model that makes such an audit actionable — you arrive already knowing what the scheduler is trying to do. FAQ What’s worth understanding about mini sglang first? mini sglang is a stripped-down reimplementation of the SGLang serving loop: it runs prefill (a compute-heavy pass over the prompt), then decode (one token at a time), while a scheduler batches requests and a KV cache stores attention state for reuse. In practice it obeys the same prefill/decode asymmetry as any full LLM server — prefill is compute-bound, decode is usually memory-bandwidth-bound — and its small size makes it easier to read but does nothing to relax those constraints. What does a lightweight SGLang runtime actually schedule, and why does that shape performance? It schedules prefill passes, decode steps, request batching, and KV-cache allocation. Performance is shaped by decode: each decode step reads the whole KV cache and model weights from HBM while doing little arithmetic, so if the scheduler can’t assemble a wide batch, the GPU waits on memory with idle compute. The kernels can be efficient while occupancy is poor. How does mini sglang differ from the full SGLang serving stack, and when is the minimal version enough? The minimal runtime drops production features — RadixAttention KV-cache reuse, prefill/decode disaggregation, distributed serving — to stay readable. It is enough for learning the scheduling model, controlled profiling experiments, and single-GPU prototypes. It stops being enough once you have concurrent users with varied prompt lengths, shared prefixes worth caching, or throughput targets a naive batcher can’t hit. Why can the GPU utilisation a runtime like mini sglang reports look high while end-to-end throughput stays low? Because utilisation reports that a kernel was resident during the sampling window, not that the compute units were saturated. A memory-bound decode step still shows high utilisation while the arithmetic units sit idle waiting on HBM reads. So a runtime can report 90-plus percent utilisation and deliver a fraction of the GPU’s token throughput — the number measures presence, not efficiency. How do I profile a mini sglang workload to find whether the bottleneck is compute, memory, host, or batching? Capture an end-to-end timeline with Nsight Systems and look for gaps between kernel launches. Gaps aligned with CPU activity mean host-bound; use Nsight Compute to check whether decode kernels are memory-bound (high memory, low compute throughput); log effective batch size per decode step to detect batching starvation. Only conclude compute-bound when the GPU is saturated with a full batch. What optimisation order gives the largest speedup once I’ve profiled a mini sglang deployment? Fix the class the profiler names, in ranked order: host-bound loops need reduced Python overhead and overlapped scheduling; memory-bound decode needs wider batches, KV-cache reuse, or KV quantisation; batching-starved loops need scheduler work. Kernel fusion, TensorRT, and low-level CUDA tuning belong last — they pay off only once the loop is demonstrably compute-bound. The reason a lean runtime is such a good place to learn this is that it removes the excuses: with a small codebase, there is nowhere for the bottleneck to hide once you look at the trace. The open question for any given deployment is not is mini sglang fast — it is which of the four constraints is binding right now, and that answer changes with your prompts, your batch mix, and your hardware. Read the code to know what to look for; profile the workload to know what to fix.
