On an LLM serving path, the interpreter is rarely where the latency lives. A decode loop that emits hundreds of tokens amortises Python overhead across every step, while the GPU kernels, the KV-cache traffic and the request queue sit squarely on the critical path. That is the arithmetic a port decision on an LLM path has to face before anyone writes a line of C++.
This is one such decision walked end to end: the profiling pass, the attribution table it produced, and the call that came out of it. The method did not presuppose the answer. We have run the same sequence on other workloads and reached the opposite conclusion.
The situation
A serving team had a chat-style endpoint behind a Python application layer, model served on a single GPU, batching handled by the runtime. Two numbers were failing: p95 time-to-first-token was above the product target, and cost per million tokens was above what the pricing model tolerated. The proposal on the table was to rewrite the serving loop in C++ — request handling, tokenisation, and the decode loop — on the reasoning that “the Python layer is in the hot path for every token.”
That reasoning is not absurd. It is just untested. The Python layer is in the hot path for every token; the question is what fraction of each token’s wall-clock it owns.
How do you profile an LLM serving path without lumping everything as “Python is slow”?
You split the request, not the process. An LLM request has structurally distinct segments and they scale differently, so a single flame graph over the whole endpoint tells you almost nothing useful. The segments we instrument separately:
- Tokenisation — input encode and output decode, usually a Rust-backed tokeniser called from Python.
- Prefill — the forward pass over the full prompt, one large compute-bound burst that dominates time-to-first-token.
- Decode — the per-token autoregressive steps, memory-bandwidth-bound rather than compute-bound.
- KV-cache traffic — reads and writes of the attention cache, plus allocation and eviction behaviour under concurrency.
- Request-level queueing — time the request spends waiting for a batch slot, invisible in single-request microbenchmarks and often the largest term at load.
The instrumentation is unglamorous: timers around each segment in the Python layer, NVIDIA Nsight Systems traces to see where kernel time actually goes, torch.profiler with CUDA activity enabled to separate host from device time, and per-request tracing that records queue-entry and batch-dispatch timestamps. The important discipline is measuring at the batch size and concurrency the production path really sees. A profile taken at concurrency 1 will attribute close to zero to queueing and then mislead every downstream decision.
The measurement that decides a port is not “how slow is Python” but “what share of the latency budget does a language change touch at production concurrency?” On the LLM paths we have profiled, that share is consistently smaller than teams expect, because the interpreter cost per decode step is fixed while the GPU cost per decode step is not (observed pattern across TechnoLynx inference engagements; not a published benchmark).
The attribution table
This is the artifact the pass produces. Shares are of p95 wall-clock at the concurrency the endpoint was actually serving, and the last column is the honest question: could leaving Python reach this segment at all?
| Segment | Share of p95 wall-clock | Bound by | Reachable by a language port? |
|---|---|---|---|
| Queueing / batch admission | largest single term at load | scheduler policy, batch size | No — scheduling logic, not interpreter speed |
| Prefill | second-largest for long prompts | GPU compute (matmul kernels) | No — kernel time is language-independent |
| Decode steps | large in aggregate, small per step | memory bandwidth, KV-cache reads | No — bandwidth-bound |
| KV-cache handling | moderate, grows with concurrency | allocator behaviour, cache layout | Partially — allocation policy, not kernels |
| Tokenisation | small | native tokeniser library | Barely — already native under a Python binding |
| Python request/serving glue | small per token, non-trivial at TTFT | interpreter, async framework | Yes |
Two things fall out of a table like this immediately. First, the segments a C++ rewrite genuinely reaches are the small ones. Second, the largest term — queueing — is a policy problem: continuous batching, admission control, and how the runtime interleaves prefill with decode. Rewriting the glue in C++ leaves the batching policy exactly where it was.
Ranking the alternatives against the port
Once the attribution table exists, the port stops being a special case and becomes one candidate intervention among several, costed on the same axes: expected reach into the measured bottleneck, engineering weeks, and ongoing maintenance surface.
| Intervention | Segment it reaches | Engineering cost | Maintenance surface |
|---|---|---|---|
| Continuous batching / better admission policy | queueing, decode throughput | low — runtime configuration and load testing | small; config lives with the serving runtime |
| Serving-runtime change (e.g. vLLM, TensorRT-LLM) | queueing, KV-cache layout, decode | moderate — integration and re-validation | moderate; tracks upstream releases |
| Weight or KV-cache quantisation | prefill, decode, memory traffic | moderate — plus accuracy validation | model-version coupled |
| Kernel-level work (attention, FlashAttention-class kernels) | prefill, decode | high — specialist skills | high; CUDA and driver pinning |
| C++ port of the serving loop | Python glue only | high — dual implementation | high; second codebase, model-update path |
On this path the ranking was not close. The batching and admission change addressed the dominant term at a fraction of the cost, and KV-cache quantisation was queued behind it as the second move. The port was declined — not because ports never pay, but because the fraction of the budget it could reach was smaller than the fraction the first two interventions could reach, at higher cost. The reasoning behind that ordering is the same rubric we set out in when porting Python inference to C++ or WASM actually pays off, applied here to one concrete serving stack.
When does the same pass justify porting an LLM path?
It does, under conditions the table makes visible. If the model is small enough that per-token GPU work is genuinely tiny — a distilled classifier or a small embedding model behind a high-QPS endpoint — the fixed per-call host overhead stops being amortised and starts dominating. If the deployment surface forbids the Python runtime altogether (an embedded target, a browser sandbox, a strict container-image budget), the port is not a performance decision at all; it is a deployment constraint. And if the profile shows host-side work between kernel launches starving the GPU — visible as gaps in an Nsight timeline rather than as slow kernels — then interpreter overhead really is on the critical path.
None of those held here. All three have held elsewhere.
What the decision record has to contain
A port decision is only defensible for as long as its inputs are written down. Model versions move, prompt lengths change, and the latency target gets tightened by a product decision made in another room. When any of those shift, the arithmetic has to be re-run, not re-argued from memory.
The record we keep is short:
- The target being measured against — p50/p95 time-to-first-token, inter-token latency, tokens per second per GPU, cost per million tokens — with the concurrency and prompt-length distribution the numbers were taken at.
- The attribution table, with the tooling and build versions used to produce it.
- The share of wall-clock a language port could plausibly recover, stated as a range, not a point.
- The engineering estimate for the rewrite, including the dual-maintenance cost of keeping the Python path alive for training and experimentation.
- The ranked alternatives and why the chosen one won.
- The trigger conditions that would reopen the decision — a model swap, a step change in prompt length, or a target that moves by more than the recovery range in point 3.
That last item is the one most often skipped, and it is what turns a decision into an asset rather than a memo. This profiling-then-decide sequence is the port-decision step of our Inference Cost-Cut Pack, and the broader engineering context for it sits with our GPU and inference performance work. The serving-stack and cost-per-token targets these decisions are measured against are usually set at the AI-infrastructure layer, which is where the profiling numbers ultimately have to be argued.
The uncomfortable part of this method is that it produces a “no” more often than the teams commissioning it expect, and a “no” delivers value only if the avoided rewrite cost is counted as a result. That accounting is a management decision, not an engineering one — which may be why the port instinct survives so well on LLM paths where the arithmetic rarely supports it.
Frequently Asked Questions
What does a port decision applied to an LLM inference path mean in practice?
Should your LLM inference traffic route through port 443 alongside web requests, or does it warrant a dedicated port with isolated network policies? When applied to Port Decision Applied to an LLM, it means profiling the existing serving path against a stated latency or unit-cost target, attributing wall-clock time across tokenisation, prefill, decode, KV-cache traffic and queueing, and only then asking which of those segments a language change would touch. The output is a documented decision with a recovery estimate and a ranked list of alternatives — not a rewrite kickoff., less than intuition suggests, because interpreter cost per decode step is fixed while GPU and memory-bandwidth cost per step is not, so Python overhead is amortised across the loop. Express the recoverable share as a range taken from the profile at production concurrency, and treat any figure produced at concurrency 1 as unusable for this purpose.
Which interventions compete with a port for the same latency target, and how do we rank them?
Batching and admission policy, a serving-runtime change, weight or KV-cache quantisation, and kernel-level attention work all compete with a port. Rank them on three axes: how much of the measured bottleneck each reaches, engineering weeks, and ongoing maintenance surface — the port typically reaches the smallest segment at the highest cost.
What engineering and maintenance cost does a ported LLM inference path carry once model or runtime versions move?
A second implementation to keep in step with every model retrain, a build and toolchain surface with CUDA and driver pinning, and a narrower set of engineers who can safely change the path. If the Python path stays alive for training and experimentation, that dual burden is permanent and belongs in the original estimate.
How do we write down the resulting decision so it stays defensible when the latency target or model changes?
Record the target and the concurrency it was measured at, the attribution table with tooling versions, the recovery range, the fully loaded rewrite estimate, the ranked alternatives, and the trigger conditions that reopen the question. Without the triggers, the record ages into a memo instead of remaining a decision you can re-run.
Three constraints that dictate LLM porting outcomes
Token throughput, context window size, and quantization tolerance will shape every LLM port more than framework choice ever will. Everything else is detail.
