A team picks nano SGLang because it is small, installs cleanly, and runs a model in an afternoon. That is a serving convenience. It is also, quietly, a compute-API decision — and treating it as the former when it is the latter is where the hardware roadmap narrows. The naive read of “nano sglang” is that it is just a smaller SGLang: fewer dependencies, a trimmer install, the same runtime with the knobs removed. So you reach for it the way you reach for a lighter Docker base image — an operational preference that carries no strategic weight. That framing is where the trouble starts. A serving runtime is not a neutral wrapper around your model. Its kernels, its scheduler, and its attention implementation encode assumptions about the compute API underneath, and SGLang’s assumptions point at CUDA on NVIDIA silicon. Adopting a minimal build does not dilute that commitment. It hides it behind fewer knobs. What is nano SGLang, and what does it mean in practice? SGLang is a serving runtime for large language models built around structured generation, a fast scheduler, and prefix-cache reuse through RadixAttention’s KV-cache mechanics. Nano SGLang — like the closely related mini SGLang variant we cover separately — is a stripped-down distribution of that runtime: fewer optional dependencies, a smaller surface, easier to stand up on a single node for prototyping or a lightweight production deployment. In practice, “nano” is about install footprint and operational simplicity. It is not a different execution engine. The hot path — the batched attention kernels, the token scheduler, the memory allocator that manages the KV cache — is the same machinery SGLang uses, or a subset of it. That machinery was written to run fast on NVIDIA GPUs through CUDA, cuDNN, and in many configurations custom or FlashAttention-derived kernels compiled for that target. When you serve a model with nano SGLang, you are executing CUDA. The word “nano” changes what you have to configure. It does not change what the runtime asks of your hardware. This is the first thing to be honest about: a smaller runtime with fewer settings is more likely, not less, to bury the compute-API question, because there is no obvious knob that says “which GPU backend am I committing to.” The absence of the knob is not the absence of the commitment. What compute API does nano SGLang actually depend on? SGLang’s performance story is a CUDA story. The scheduler and the attention kernels are tuned against the NVIDIA execution model — warp scheduling, the memory hierarchy on Ampere and Hopper parts, the tensor-core paths for FP16 and FP8. Some of that reaches through PyTorch, which is itself CUDA-first for its highest-performance operators; some of it is compiled kernel code that assumes an NVIDIA target. There is active work in the broader ecosystem on AMD (ROCm/HIP) and other backends, but the sustained, well-trodden path — the one where the published throughput numbers come from — is CUDA on NVIDIA. That means the honest answer to “what does nano SGLang depend on” is: the same compute API you would be choosing if you selected CUDA directly. This is the core claim, and it is worth stating plainly so it can be quoted out of context: a serving runtime’s kernel and scheduler design carry an implicit compute-API commitment, and adopting a CUDA-first runtime like nano SGLang narrows your hardware roadmap exactly as choosing CUDA directly does (observed pattern across GPU inference-stack engagements; not a benchmarked portability score). The lock-in is not created by nano SGLang. It is inherited by it, and made harder to see. If you have read our treatment of porting an inference path off the CUDA lock-in with HIP, the mechanism is familiar: the cost of leaving CUDA is not one big wall, it is a thousand small kernel-level assumptions that all have to be re-validated on the new target. A serving runtime concentrates those assumptions into its hot path and hands you a clean API on top, which is precisely why the commitment is easy to miss. Does this create the same NVIDIA lock-in as choosing CUDA directly? Yes — with one qualification worth being precise about. Choosing CUDA directly is a decision your engineers make consciously; they know they are writing against cudaMalloc and warp intrinsics. Choosing a CUDA-first runtime is the same decision made implicitly, one abstraction level up. The portability consequence is identical. The visibility is worse. This is where runtime selection stops being an ops task and becomes an architecture decision. The moment your serving stack’s fast path is CUDA-bound, your three-year hardware roadmap is CUDA-bound too, whether or not anyone wrote that down. If a future budget cycle wants AMD Instinct parts or Intel accelerators — for price, for supply, for a specific memory-capacity advantage — the runtime is now the thing standing in the way, and it was chosen by whoever picked “the lightweight option.” Serving-runtime selection: convenience decision vs architecture decision Dimension Treated as ops convenience Treated as architecture decision Question asked “Which runtime installs cleanly and runs my model?” “Which compute API am I committing my roadmap to?” Portability cost Unquantified — discovered at migration time Estimated up front, scored against target hardware diversity Who decides Whoever stood up the prototype Named owner accountable for the 3-year hardware plan Reversal cost Kernel-level re-validation on a new backend, discovered late Known and budgeted, or deliberately accepted Metric attached “It was light” Tokens/sec at target batch, latency at target concurrency, cost to leave NVIDIA The right-hand column is not more bureaucracy. It is the difference between a choice you can defend in an audit and a choice you have to explain after it has already cost you. When is a lightweight runtime like nano SGLang the right call? Often. The point is not that CUDA-first serving is wrong — for a great many workloads it is the correct, pragmatic choice, and pretending otherwise would be its own kind of dishonesty. The point is that the choice should be made, not defaulted into. Nano SGLang is a strong fit when your hardware roadmap is already NVIDIA and expected to stay there, when you want SGLang’s structured-generation and prefix-caching behaviour without the full dependency surface, and when the workload is small enough that a trimmed single-node runtime covers it. It is a poor fit when portability across accelerator vendors is a stated requirement, when procurement wants leverage to negotiate against a single-vendor dependency, or when the deployment is large enough that a locked backend becomes a strategic exposure rather than a convenience. A useful diagnostic: if you cannot name the person who owns the consequence of the compute-API commitment, you have not made the decision — the runtime made it for you. What should I measure before committing? Before nano SGLang goes into anything you cannot easily unwind, measure the runtime against your workload, not against a published leaderboard. The relevant numbers are per-workload and operational. Sustained inference throughput in tokens/sec at your target batch size — not the peak burst number, the number the runtime holds under continuous load. Our note on what SGLang profiling reveals about real inference bottlenecks walks through why the sustained figure diverges from the headline one. Latency at target concurrency — p50 and p95 at the batch sizes and request rates you actually expect, since a runtime that looks fast at batch 1 can degrade sharply under real load. KV-cache hit behaviour for your prompt distribution, because RadixAttention’s benefit depends heavily on how much prefix your traffic actually shares. The concrete cost of leaving NVIDIA — estimated in engineering-weeks to re-validate the hot path on ROCm or another backend, expressed as a number you can put next to the throughput gain. If that number is unknown, it is not zero; it is unmeasured. All of these are benchmark-class figures when you run them against your own deployment and name the configuration. Treat anything you did not measure on your hardware as market-direction at best. This measurement discipline is exactly what a serving stack built on a well-understood compute-API decision on our GPU engineering practice is meant to make auditable. How does this fit into a GPU Performance Audit? Runtime selection is a downstream expression of the compute-API decision, so it surfaces naturally in an audit of an inference stack. A GPU Performance Audit does not stop at “is the GPU busy.” It traces the serving runtime back to the API it commits you to, quantifies the portability cost that commitment carries, and scores the choice against your target hardware diversity and roadmap. Nano SGLang’s CUDA assumptions are exactly the kind of structural constraint that audit is built to make visible — the sort of thing that is invisible in a dashboard and expensive in a migration. FAQ How does nano SGLang work? Nano SGLang is a stripped-down distribution of the SGLang serving runtime — fewer optional dependencies and a smaller install footprint, easier to stand up on a single node. The execution engine underneath is the same: the batched attention kernels, token scheduler, and KV-cache allocator that SGLang uses. “Nano” changes what you configure, not what the runtime asks of your hardware. What compute API does a nano SGLang serving stack actually depend on, and does that create the same NVIDIA lock-in as choosing CUDA directly? Its fast path is CUDA on NVIDIA — the tuned kernels and scheduler assume the NVIDIA execution model, reaching through PyTorch and compiled kernel code. That is the same commitment you would make by choosing CUDA directly. The portability consequence is identical; the difference is that a minimal runtime hides the decision behind fewer knobs, so the lock-in is inherited rather than chosen. When is a lightweight LLM-serving runtime like nano SGLang the right call versus a more portable serving path? It is a strong fit when your roadmap is already NVIDIA and staying there, and you want SGLang’s structured generation and prefix caching without the full dependency surface. It is a poor fit when cross-vendor portability is a stated requirement, when procurement wants leverage against single-vendor dependency, or when the deployment is large enough that a locked backend becomes a strategic exposure. The rule is to make the choice deliberately, not default into it. What inference-cost and throughput metrics should I measure before committing to nano SGLang for a workload? Measure sustained tokens/sec at your target batch size (not peak burst), p50/p95 latency at target concurrency, KV-cache hit behaviour for your prompt distribution, and the concrete engineering-week cost of leaving NVIDIA. Run these against your own deployment and name the configuration so they count as benchmark-class evidence. Anything you did not measure on your hardware is directional at best. If I adopt nano SGLang, how hard is it to later run the same models on AMD or Intel accelerators? The model weights are portable; the runtime’s hot path is not. Moving off NVIDIA means re-validating the kernel-level assumptions the CUDA-first path bakes in — a thousand small assumptions rather than one wall. Estimate that cost in engineering-weeks up front; if it is unknown, treat it as unmeasured, not as zero. How does the nano SGLang runtime decision fit into a GPU Performance Audit of my inference stack? A GPU Performance Audit traces the serving runtime back to the compute API it commits you to, quantifies the portability cost, and scores the choice against your target hardware diversity and roadmap. Nano SGLang’s CUDA assumptions are exactly the structural constraint the audit surfaces — invisible in a utilisation dashboard, expensive in a migration. The runtime you install in an afternoon can set the hardware you are allowed to buy for the next three years. Before nano SGLang becomes the default answer to “how do we serve this model,” ask which compute API it commits you to, and whether anyone on the team has agreed to own that commitment — because if runtime selection is treated as convenience rather than architecture, the portability cost is one you will pay without ever having quantified it.
