A team hits a latency wall on an inference service and the first instinct is almost always the same: the Python runtime must be the problem, so rewrite the hot path in C++ or compile it to WebAssembly. Sometimes that is right. Often it is not, and the only way to tell the difference before spending weeks on a port is to profile the existing Python path and attribute the wall-clock time to where it actually goes. That attribution is the whole decision. If the model compute dominates end-to-end latency, no language port moves it — the matrix multiplies run in the same BLAS or CUDA kernels whether Python or C++ dispatches them. If Python-layer overhead is a minority of the wall-clock time, porting the surrounding code buys you a fraction of a fraction. The port pays off only when the interpreter, the glue code, and the per-call overhead genuinely sit on the critical path. Profiling is how you find out which world you are in. What a Python inference profiling pass actually measures The purpose of the pass is not to produce a flame graph for its own sake. It is to split the end-to-end latency of one representative request into three buckets and put a number on each: Model compute — time spent inside the actual numerical kernels (GEMMs, convolutions, attention). On GPU this is kernel execution plus the launch overhead; on CPU it is the BLAS/oneDNN calls. Python-layer overhead — the interpreter loop, object allocation, tensor wrapping, framework dispatch, pre- and post-processing written in pure Python. IO and data movement — deserialization of the request, tokenisation, host-to-device copies, and the response path. A port to C++ or WASM primarily attacks the second bucket. It can help the third if the data-movement code is Python-heavy. It does essentially nothing for the first. So the ratio between these three buckets is the expected-gain estimate for a port. A path where Python overhead is, say, a fifth of wall-clock time has a hard ceiling on what any rewrite can return — you cannot recover more than that fifth, and in practice you recover less. This is the same discipline we describe in what “computationally expensive” means in an inference path: the cost lives somewhere specific, and the first job is to locate it rather than assume it. How do we attribute end-to-end latency across model compute, Python overhead, and IO? You need two complementary views, because no single tool tells the whole story. The first is a wall-clock timeline of a single representative request. Wrap the request handler with explicit timing spans — request parse, tokenise, model forward, decode/post-process, serialize — and record the elapsed time of each. This is coarse but it immediately shows whether the model forward is 90% of the request or 30%. If it is 90%, you can often stop here: the port is unlikely to be worth it, and the conversation shifts to the model itself. The second is a sampling or deterministic profiler on the Python process to see where interpreter time goes inside the non-compute spans. cProfile gives deterministic call counts and cumulative times, which is useful for finding a surprisingly hot pure-Python loop. py-spy samples the running process without instrumenting it, so it captures production-like behaviour without the deterministic-profiler overhead that itself distorts the numbers. For the compute bucket specifically, framework profilers matter: the PyTorch profiler (torch.profiler) and NVIDIA Nsight Systems attribute time to CUDA kernels versus host-side Python, and that host-versus-device split is exactly the line a port cares about. The trap here is the profiler’s own overhead. A deterministic profiler can inflate Python-heavy code far more than kernel-heavy code, which biases the attribution toward the conclusion that Python is the bottleneck — precisely the conclusion you are trying to test honestly. Cross-check cProfile findings against a low-overhead py-spy sample before trusting the ratio. A worked attribution example Assume a CPU inference service handling one request measures the following, averaged over a warm run of a few hundred requests (illustrative numbers, not a benchmark): Span Wall-clock (ms) Share Movable by a port? Request parse + validate 3 6% Partly (if pure Python) Tokenisation 8 16% Yes — often pure Python Model forward (oneDNN/BLAS) 34 68% No — kernels unchanged Post-process + serialize 5 10% Partly End-to-end 50 100% — The port ceiling here is at most the tokenisation plus half the parse and post-process spans — roughly 26% of wall-clock, and realistically less after accounting for the C++/WASM code still needing to call back into the same numerical libraries. A rewrite that costs three engineer-weeks to shave, optimistically, 10–12 ms off a 50 ms request is a different decision from one that shaves 30 ms. The attribution is what makes that decision defensible instead of a guess. Which profiling tools and measurement points give a trustworthy baseline? Trustworthy means reproducible and representative — the same properties we look for in benchmarking agentic inference before a port. A few measurement disciplines separate a real baseline from a misleading one. Measure a warm path, not a cold one. First-request latency includes model load, kernel autotuning, and cache population that will not recur; profiling the cold path attributes one-time costs to the steady state and skews everything. Run enough requests to get past warm-up, then measure. Fix the input distribution. Latency for a transformer scales with sequence length; a profile taken on 20-token inputs tells you nothing about a service that mostly sees 500-token inputs. Use a representative sample of real request shapes, and report the distribution, not just a mean. Separate host time from device time explicitly on GPU paths. A GPU can look busy in nvidia-smi while the real constraint is Python launching kernels one at a time with gaps between them — the same reason reading CPU GFLOPS as peak versus achieved throughput misleads. Nsight Systems shows those gaps as idle device time between kernels, which points at dispatch overhead rather than compute. That is one of the few cases where a port — or, more cheaply, CUDA graphs or torch.compile — actually addresses the bottleneck. Pin the measurement environment: same hardware, same library versions, same thread counts. A baseline taken on a laptop and a decision made for a server-class deployment share a slug but not a truth. What does the attribution result tell us about whether a port will move the bottleneck? The result maps onto a small decision space. This is the part worth being blunt about, because it is where the sunk-cost trap lives. Attribution result What a C++/WASM port does Recommended action Model compute dominates (>60%) Nothing — kernels are unchanged Optimise the model: quantisation, batching, a smaller architecture, or a faster kernel path Python dispatch overhead high, device idle between kernels Helps, but cheaper fixes exist first Try CUDA graphs / torch.compile / batching before porting Pure-Python pre/post-processing is a large share Port that specific code, not the whole path Rewrite the hot function in C++/Cython or move it to WASM; leave the rest IO/serialization dominates A language port rarely helps Fix the data path: streaming, zero-copy, protocol choice Footprint (not latency) is the driver WASM/native can shrink the runtime footprint Port for footprint, but state the footprint target explicitly Notice that only two rows genuinely favour a port, and one of them (footprint) is a different objective from latency entirely. The WASM case in particular is often about shipping inference into a browser or a constrained runtime rather than about speed — the trade-offs there are covered in what compiler flags do to Pyodide performance. If the driver is footprint, the profiling pass still matters, because it tells you whether the model weights or the Python runtime dominate the footprint you are trying to cut. Tying the numbers to a latency, footprint, or unit-cost target A profiling pass with no target is just curiosity. The number that makes it a decision is the gap between measured latency (or footprint, or cost-per-request) and the stated target. If the target is a 30 ms p95 and the path measures 50 ms with 68% in model compute, the arithmetic is unforgiving: even eliminating all Python overhead leaves you above target. That result redirects effort to the model — and it does so before anyone has written a line of C++. The same logic runs for unit cost. Cost-per-request on a GPU is a function of how much wall-clock time the request holds the device. If the device is idle between kernels because Python is dispatching serially, that idle time is billed cost you can recover; if the device is saturated by compute, the recovery path is a cheaper model or a cheaper accelerator, not a port. Making that distinction is precisely the baseline step in our [inference cost-cut work](Inference Cost-Cut Pack), and it is the same reasoning that runs across our broader GPU engineering practice. FAQ How does a Python inference profiling pass work, and what does it mean in practice for a port decision? A profiling pass instruments one representative, warmed-up request and splits its end-to-end latency into model compute, Python-layer overhead, and IO. In practice it converts a port decision from a hunch into arithmetic: the share of time in Python-layer overhead is the ceiling on what any C++ or WASM rewrite can recover, so the attribution tells you the expected gain before you commit engineering to it. How do we attribute end-to-end latency across model compute, Python overhead, and IO? Combine a coarse wall-clock timeline of the request handler’s spans (parse, tokenise, forward, post-process) with a profiler that separates host from device time. cProfile and py-spy locate hot pure-Python code, while torch.profiler and Nsight Systems attribute time to CUDA kernels versus host-side dispatch. Cross-check deterministic and sampling profilers, because deterministic profiling overhead biases the result toward blaming Python. Which profiling tools and measurement points give a trustworthy baseline for an inference path? Measure a warm path, not a cold one; fix the input distribution to match real request shapes; separate host time from device time explicitly on GPU; and pin the hardware and library versions. Tools include cProfile, py-spy, torch.profiler, and NVIDIA Nsight Systems — the discipline of warm, representative, environment-pinned measurement matters more than the tool choice. What does the attribution result tell us about whether a port will move the bottleneck? If model compute dominates, a port moves nothing and effort belongs on the model. If Python dispatch overhead is high with the device idle between kernels, cheaper fixes like CUDA graphs, torch.compile, or batching usually come first. A port genuinely pays off when a specific pure-Python pre/post-processing function is a large share of time, or when the objective is shrinking the runtime footprint rather than cutting latency. How do we tie the profiling numbers to a latency, footprint, or unit-cost target? Compare measured latency, footprint, or cost-per-request against the stated target and look at the movable share. If eliminating all Python overhead still leaves you above a latency target, the model is the constraint, not the language. For unit cost, idle device time between serial kernel launches is recoverable billed cost, whereas a compute-saturated device points to a cheaper model or accelerator rather than a port. When does profiling reveal that the model compute — not the language runtime — is the constraint? When the model-forward span holds the majority of wall-clock time and the device is saturated rather than idle between kernels. In that state the numerical kernels run in the same BLAS or CUDA code regardless of whether Python or C++ dispatches them, so no language port changes the number — the profiling pass surfaces this before the rewrite is sunk. Before any migration spend, run the pass and write down three numbers against one target. If the model compute owns the latency, the honest answer is that the port was never the lever — and finding that out cheaply is the point. For the broader picture of what moving a workload actually entails once the attribution justifies it, see what software porting really means when you move a workload to new hardware.
